2024-03-31T22:50:32.731 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-31T22:50:32.731 DEBUG:teuthology.run:Teuthology command: teuthology --description fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/fuse objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/suites/dbench}} --name teuthology-2024-03-31_21:24:02-fs-squid-distro-default-smithi --owner scheduled_teuthology@teuthology --archive /home/teuthworker/archive/teuthology-2024-03-31_21:24:02-fs-squid-distro-default-smithi/7633118 --verbose -- /home/teuthworker/archive/teuthology-2024-03-31_21:24:02-fs-squid-distro-default-smithi/7633118/orig.config.yaml 2024-03-31T22:50:32.790 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-31T22:50:32.879 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/teuthology-2024-03-31_21:24:02-fs-squid-distro-default-smithi/7633118 branch: squid description: fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/fuse objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/suites/dbench}} email: ceph-qa@ceph.com first_in_suite: false job_id: '7633118' kernel: kdb: true sha1: distro last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi name: teuthology-2024-03-31_21:24:02-fs-squid-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: centos os_version: 9.stream overrides: admin_socket: branch: squid ceph: cephfs: ec_profile: - m=2 - k=2 - crush-failure-domain=osd max_mds: 3 session_timeout: 300 standby_replay: true conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds bal fragment size max: 10000 mds bal merge size: 5 mds bal split bits: 3 mds bal split size: 100 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true mds_bal_replicate_threshold: 1 osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: cephsqlite lock renewal timeout: 900000 debug cephsqlite: 20 debug client: 20 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon down mkfs grace: 300 mon op complaint time: 120 osd: bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 osd shutdown pgref assert: true osd_max_omap_entries_per_request: 10 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - FS_DEGRADED - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - MDS_UP_LESS_THAN_MAX - POOL_APP_NOT_ENABLED - overall HEALTH_ - Replacing daemon - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - slow metadata IO - SLOW_OPS - slow request sha1: 384f31f3d3aee7f8ba3f60846b8844c049165583 subvols: create: 2 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs ceph-fuse: client.0: mount_subvol_num: 0 check-counter: counters: mds: - mds.exported - mds.imported - min: 3 name: mds_cache.dir_update - min: 3 name: mds_cache.dir_update_receipt - mds.root_rsnaps - mds_server.req_mksnap_latency.avgcount - mds_server.req_rmsnap_latency.avgcount install: ceph: flavor: default sha1: 384f31f3d3aee7f8ba3f60846b8844c049165583 kclient: client.0: mount_subvol_num: 1 selinux: allowlist: - scontext=system_u:system_r:logrotate_t:s0 - scontext=system_u:system_r:getty_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: squid sha1: 384f31f3d3aee7f8ba3f60846b8844c049165583 owner: scheduled_teuthology@teuthology priority: 100 repo: https://git.ceph.com/ceph.git roles: - - mon.a - mgr.x - mds.a - mds.d - mds.g - mds.j - osd.0 - osd.3 - osd.6 - osd.9 - client.0 - - mon.b - mgr.y - mds.b - mds.e - mds.h - mds.k - osd.1 - osd.4 - osd.7 - osd.10 - - mon.c - mgr.z - mds.c - mds.f - mds.i - mds.l - osd.2 - osd.5 - osd.8 - osd.11 seed: 8238 sha1: 384f31f3d3aee7f8ba3f60846b8844c049165583 sleep_before_teardown: 0 subset: 361/512 suite: fs suite_branch: squid suite_path: /home/teuthworker/src/git.ceph.com_ceph_384f31f3d3aee7f8ba3f60846b8844c049165583/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph.git suite_sha1: 384f31f3d3aee7f8ba3f60846b8844c049165583 syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi046.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKdZO7+tm/7WwtmA5s3xtAa9MJRB8l9mA76h6kRc/T2RfJWtZdopxOO1f0WThP2iPe2LvHj3cp16hOSeZKJr7Z0= smithi142.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPFrXvXXABKnguJpriXZPwIgvyK78LKrpd3e2av2k8DYf2gqPhIk/U4d/SMfnaPwoJlGSEiOk3KuNp3zh5O7w9Q= smithi162.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDlvehU7gK/q6QtQaxtb5B9Y0do8OvRq0Me6oeDQ5w1+zo2iivV6Ow2KFz+VHssbVkbn1zw4ACrcM6uAR5wZAW4= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - cephadm: roleless: false - cephadm.shell: mon.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: mon.a: - ceph fs dump - ceph osd dump - fs.ready: timeout: 300 - ceph-fuse: null - exec: mon.a: - ceph fs set cephfs balance_automate true - check-counter: {} - fwd_scrub: scrub_timeout: 900 sleep_between_iterations: 1 - exec: mon.a: - ceph mgr module enable snap_schedule - ceph config set mgr mgr/snap_schedule/allow_m_granularity true - ceph config set mgr mgr/snap_schedule/dump_on_update true - ceph fs snap-schedule add --fs=cephfs --path=/ --snap_schedule=1m - ceph fs snap-schedule retention add --fs=cephfs --path=/ --retention-spec-or-period=6m3h - ceph fs snap-schedule status --fs=cephfs --path=/ - ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true - date +%s > START_TIME - full_sequential_finally: - exec: mon.a: - date +%s > END_TIME - START_TIME=$(cat START_TIME); END_TIME=$(cat END_TIME); DIFF_TIME=$((600-(END_TIME-START_TIME))); if [ "$DIFF_TIME" -gt 0 ]; then sleep "$DIFF_TIME"; fi - ceph fs snap-schedule status --fs=cephfs --path=/ - ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true - background_exec: mon.a: - while sleep 13; do ceph tell mds.cephfs:0 flush journal; done - workunit: clients: all: - suites/dbench.sh teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return false end - local function is_kdistro() return false end - local function is_fuse() return true end - local function syntax_version() return '' end teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-03-31_21:24:02 tube: smithi user: teuthology verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-03-31T22:50:32.879 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph_384f31f3d3aee7f8ba3f60846b8844c049165583/qa; will attempt to use it 2024-03-31T22:50:32.880 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph_384f31f3d3aee7f8ba3f60846b8844c049165583/qa/tasks 2024-03-31T22:50:32.880 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-31T22:50:32.882 INFO:teuthology.task.internal:Checking packages... 2024-03-31T22:50:32.906 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '384f31f3d3aee7f8ba3f60846b8844c049165583' 2024-03-31T22:50:32.906 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-31T22:50:32.906 INFO:teuthology.packaging:ref: None 2024-03-31T22:50:32.906 INFO:teuthology.packaging:tag: None 2024-03-31T22:50:32.906 INFO:teuthology.packaging:branch: squid 2024-03-31T22:50:32.906 INFO:teuthology.packaging:sha1: 384f31f3d3aee7f8ba3f60846b8844c049165583 2024-03-31T22:50:32.906 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=squid 2024-03-31T22:50:33.087 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-1977.g384f31f3 2024-03-31T22:50:33.089 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-31T22:50:33.097 INFO:teuthology.task.internal:no buildpackages task found 2024-03-31T22:50:33.097 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-31T22:50:33.118 INFO:teuthology.task.internal:Saving configuration 2024-03-31T22:50:33.139 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-31T22:50:33.147 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-31T22:50:33.174 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi046.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-03-31_21:24:02-fs-squid-distro-default-smithi/7633118', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-31 22:45:23.367893', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKdZO7+tm/7WwtmA5s3xtAa9MJRB8l9mA76h6kRc/T2RfJWtZdopxOO1f0WThP2iPe2LvHj3cp16hOSeZKJr7Z0='} 2024-03-31T22:50:33.197 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi142.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-03-31_21:24:02-fs-squid-distro-default-smithi/7633118', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-31 22:45:23.366388', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPFrXvXXABKnguJpriXZPwIgvyK78LKrpd3e2av2k8DYf2gqPhIk/U4d/SMfnaPwoJlGSEiOk3KuNp3zh5O7w9Q='} 2024-03-31T22:50:33.224 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi162.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-03-31_21:24:02-fs-squid-distro-default-smithi/7633118', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-31 22:45:23.369609', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDlvehU7gK/q6QtQaxtb5B9Y0do8OvRq0Me6oeDQ5w1+zo2iivV6Ow2KFz+VHssbVkbn1zw4ACrcM6uAR5wZAW4='} 2024-03-31T22:50:33.224 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-31T22:50:33.232 INFO:teuthology.task.internal:roles: ubuntu@smithi046.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.d', 'mds.g', 'mds.j', 'osd.0', 'osd.3', 'osd.6', 'osd.9', 'client.0'] 2024-03-31T22:50:33.233 INFO:teuthology.task.internal:roles: ubuntu@smithi142.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'mds.b', 'mds.e', 'mds.h', 'mds.k', 'osd.1', 'osd.4', 'osd.7', 'osd.10'] 2024-03-31T22:50:33.233 INFO:teuthology.task.internal:roles: ubuntu@smithi162.front.sepia.ceph.com - ['mon.c', 'mgr.z', 'mds.c', 'mds.f', 'mds.i', 'mds.l', 'osd.2', 'osd.5', 'osd.8', 'osd.11'] 2024-03-31T22:50:33.233 INFO:teuthology.run_tasks:Running task console_log... 2024-03-31T22:50:33.349 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f135b35f160>, signals=[15]) 2024-03-31T22:50:33.350 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-31T22:50:33.360 INFO:teuthology.task.internal:Opening connections... 2024-03-31T22:50:33.360 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi046.front.sepia.ceph.com 2024-03-31T22:50:33.363 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi046.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-31T22:50:33.470 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi142.front.sepia.ceph.com 2024-03-31T22:50:33.470 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi142.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-31T22:50:33.560 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi162.front.sepia.ceph.com 2024-03-31T22:50:33.561 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi162.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-31T22:50:33.652 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-31T22:50:33.666 DEBUG:teuthology.orchestra.run.smithi046:> uname -m 2024-03-31T22:50:33.683 INFO:teuthology.orchestra.run.smithi046.stdout:x86_64 2024-03-31T22:50:33.684 DEBUG:teuthology.orchestra.run.smithi046:> cat /etc/os-release 2024-03-31T22:50:33.741 INFO:teuthology.orchestra.run.smithi046.stdout:NAME="CentOS Stream" 2024-03-31T22:50:33.741 INFO:teuthology.orchestra.run.smithi046.stdout:VERSION="9" 2024-03-31T22:50:33.741 INFO:teuthology.orchestra.run.smithi046.stdout:ID="centos" 2024-03-31T22:50:33.741 INFO:teuthology.orchestra.run.smithi046.stdout:ID_LIKE="rhel fedora" 2024-03-31T22:50:33.741 INFO:teuthology.orchestra.run.smithi046.stdout:VERSION_ID="9" 2024-03-31T22:50:33.741 INFO:teuthology.orchestra.run.smithi046.stdout:PLATFORM_ID="platform:el9" 2024-03-31T22:50:33.742 INFO:teuthology.orchestra.run.smithi046.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-31T22:50:33.742 INFO:teuthology.orchestra.run.smithi046.stdout:ANSI_COLOR="0;31" 2024-03-31T22:50:33.742 INFO:teuthology.orchestra.run.smithi046.stdout:LOGO="fedora-logo-icon" 2024-03-31T22:50:33.742 INFO:teuthology.orchestra.run.smithi046.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-31T22:50:33.742 INFO:teuthology.orchestra.run.smithi046.stdout:HOME_URL="https://centos.org/" 2024-03-31T22:50:33.742 INFO:teuthology.orchestra.run.smithi046.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-31T22:50:33.742 INFO:teuthology.orchestra.run.smithi046.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-31T22:50:33.742 INFO:teuthology.orchestra.run.smithi046.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-31T22:50:33.743 INFO:teuthology.lock.ops:Updating smithi046.front.sepia.ceph.com on lock server 2024-03-31T22:50:33.763 DEBUG:teuthology.orchestra.run.smithi142:> uname -m 2024-03-31T22:50:33.781 INFO:teuthology.orchestra.run.smithi142.stdout:x86_64 2024-03-31T22:50:33.781 DEBUG:teuthology.orchestra.run.smithi142:> cat /etc/os-release 2024-03-31T22:50:33.838 INFO:teuthology.orchestra.run.smithi142.stdout:NAME="CentOS Stream" 2024-03-31T22:50:33.839 INFO:teuthology.orchestra.run.smithi142.stdout:VERSION="9" 2024-03-31T22:50:33.839 INFO:teuthology.orchestra.run.smithi142.stdout:ID="centos" 2024-03-31T22:50:33.839 INFO:teuthology.orchestra.run.smithi142.stdout:ID_LIKE="rhel fedora" 2024-03-31T22:50:33.839 INFO:teuthology.orchestra.run.smithi142.stdout:VERSION_ID="9" 2024-03-31T22:50:33.839 INFO:teuthology.orchestra.run.smithi142.stdout:PLATFORM_ID="platform:el9" 2024-03-31T22:50:33.839 INFO:teuthology.orchestra.run.smithi142.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-31T22:50:33.839 INFO:teuthology.orchestra.run.smithi142.stdout:ANSI_COLOR="0;31" 2024-03-31T22:50:33.839 INFO:teuthology.orchestra.run.smithi142.stdout:LOGO="fedora-logo-icon" 2024-03-31T22:50:33.839 INFO:teuthology.orchestra.run.smithi142.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-31T22:50:33.839 INFO:teuthology.orchestra.run.smithi142.stdout:HOME_URL="https://centos.org/" 2024-03-31T22:50:33.839 INFO:teuthology.orchestra.run.smithi142.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-31T22:50:33.839 INFO:teuthology.orchestra.run.smithi142.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-31T22:50:33.839 INFO:teuthology.orchestra.run.smithi142.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-31T22:50:33.840 INFO:teuthology.lock.ops:Updating smithi142.front.sepia.ceph.com on lock server 2024-03-31T22:50:33.871 DEBUG:teuthology.orchestra.run.smithi162:> uname -m 2024-03-31T22:50:33.889 INFO:teuthology.orchestra.run.smithi162.stdout:x86_64 2024-03-31T22:50:33.890 DEBUG:teuthology.orchestra.run.smithi162:> cat /etc/os-release 2024-03-31T22:50:33.946 INFO:teuthology.orchestra.run.smithi162.stdout:NAME="CentOS Stream" 2024-03-31T22:50:33.947 INFO:teuthology.orchestra.run.smithi162.stdout:VERSION="9" 2024-03-31T22:50:33.947 INFO:teuthology.orchestra.run.smithi162.stdout:ID="centos" 2024-03-31T22:50:33.947 INFO:teuthology.orchestra.run.smithi162.stdout:ID_LIKE="rhel fedora" 2024-03-31T22:50:33.947 INFO:teuthology.orchestra.run.smithi162.stdout:VERSION_ID="9" 2024-03-31T22:50:33.947 INFO:teuthology.orchestra.run.smithi162.stdout:PLATFORM_ID="platform:el9" 2024-03-31T22:50:33.947 INFO:teuthology.orchestra.run.smithi162.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-31T22:50:33.947 INFO:teuthology.orchestra.run.smithi162.stdout:ANSI_COLOR="0;31" 2024-03-31T22:50:33.947 INFO:teuthology.orchestra.run.smithi162.stdout:LOGO="fedora-logo-icon" 2024-03-31T22:50:33.947 INFO:teuthology.orchestra.run.smithi162.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-31T22:50:33.947 INFO:teuthology.orchestra.run.smithi162.stdout:HOME_URL="https://centos.org/" 2024-03-31T22:50:33.947 INFO:teuthology.orchestra.run.smithi162.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-31T22:50:33.947 INFO:teuthology.orchestra.run.smithi162.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-31T22:50:33.948 INFO:teuthology.orchestra.run.smithi162.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-31T22:50:33.948 INFO:teuthology.lock.ops:Updating smithi162.front.sepia.ceph.com on lock server 2024-03-31T22:50:33.980 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-31T22:50:33.993 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-31T22:50:34.017 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-31T22:50:34.018 DEBUG:teuthology.orchestra.run.smithi046:> test '!' -e /home/ubuntu/cephtest 2024-03-31T22:50:34.020 DEBUG:teuthology.orchestra.run.smithi142:> test '!' -e /home/ubuntu/cephtest 2024-03-31T22:50:34.023 DEBUG:teuthology.orchestra.run.smithi162:> test '!' -e /home/ubuntu/cephtest 2024-03-31T22:50:34.038 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-31T22:50:34.049 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-31T22:50:34.049 DEBUG:teuthology.orchestra.run.smithi046:> test -z $(ls -A /var/lib/ceph) 2024-03-31T22:50:34.077 DEBUG:teuthology.orchestra.run.smithi142:> test -z $(ls -A /var/lib/ceph) 2024-03-31T22:50:34.079 DEBUG:teuthology.orchestra.run.smithi162:> test -z $(ls -A /var/lib/ceph) 2024-03-31T22:50:34.183 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-31T22:50:34.258 INFO:teuthology.run_tasks:Running task kernel... 2024-03-31T22:50:34.276 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-03-31T22:50:34.276 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}, 'mon.c': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-03-31T22:50:34.277 DEBUG:teuthology.orchestra.run.smithi046:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-31T22:50:34.277 DEBUG:teuthology.orchestra.run.smithi142:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-31T22:50:34.278 DEBUG:teuthology.orchestra.run.smithi162:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-31T22:50:34.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-31T22:50:34.294 DEBUG:teuthology.orchestra.run.smithi142:> uname -r 2024-03-31T22:50:34.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-31T22:50:34.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-31T22:50:34.296 DEBUG:teuthology.orchestra.run.smithi046:> uname -r 2024-03-31T22:50:34.296 DEBUG:teuthology.orchestra.run.smithi162:> uname -r 2024-03-31T22:50:34.351 INFO:teuthology.orchestra.run.smithi142.stdout:5.14.0-361.el9.x86_64 2024-03-31T22:50:34.351 INFO:teuthology.task.kernel:Running kernel on smithi142: 5.14.0-361.el9.x86_64 2024-03-31T22:50:34.351 DEBUG:teuthology.orchestra.run.smithi142:> sudo yum install -y kernel 2024-03-31T22:50:34.353 INFO:teuthology.orchestra.run.smithi162.stdout:5.14.0-361.el9.x86_64 2024-03-31T22:50:34.353 INFO:teuthology.task.kernel:Running kernel on smithi162: 5.14.0-361.el9.x86_64 2024-03-31T22:50:34.353 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum install -y kernel 2024-03-31T22:50:34.354 INFO:teuthology.orchestra.run.smithi046.stdout:5.14.0-361.el9.x86_64 2024-03-31T22:50:34.354 INFO:teuthology.task.kernel:Running kernel on smithi046: 5.14.0-361.el9.x86_64 2024-03-31T22:50:34.354 DEBUG:teuthology.orchestra.run.smithi046:> sudo yum install -y kernel 2024-03-31T22:50:36.538 INFO:teuthology.orchestra.run.smithi142.stdout:CentOS-9 - AppStream 13 MB/s | 6.7 MB 00:00 2024-03-31T22:50:36.574 INFO:teuthology.orchestra.run.smithi046.stdout:CentOS-9 - AppStream 12 MB/s | 6.7 MB 00:00 2024-03-31T22:50:36.827 INFO:teuthology.orchestra.run.smithi162.stdout:CentOS-9 - AppStream 9.4 MB/s | 6.7 MB 00:00 2024-03-31T22:50:38.016 INFO:teuthology.orchestra.run.smithi046.stdout:CentOS-9 - BaseOS 7.2 MB/s | 2.0 MB 00:00 2024-03-31T22:50:38.180 INFO:teuthology.orchestra.run.smithi142.stdout:CentOS-9 - BaseOS 4.0 MB/s | 2.0 MB 00:00 2024-03-31T22:50:38.227 INFO:teuthology.orchestra.run.smithi162.stdout:CentOS-9 - BaseOS 6.7 MB/s | 2.0 MB 00:00 2024-03-31T22:50:38.832 INFO:teuthology.orchestra.run.smithi046.stdout:CentOS Stream 9 - BaseOS 17 MB/s | 8.0 MB 00:00 2024-03-31T22:50:39.728 INFO:teuthology.orchestra.run.smithi162.stdout:CentOS Stream 9 - BaseOS 6.9 MB/s | 8.0 MB 00:01 2024-03-31T22:50:40.716 INFO:teuthology.orchestra.run.smithi046.stdout:CentOS Stream 9 - AppStream 25 MB/s | 19 MB 00:00 2024-03-31T22:50:41.562 INFO:teuthology.orchestra.run.smithi142.stdout:CentOS Stream 9 - BaseOS 2.6 MB/s | 8.0 MB 00:03 2024-03-31T22:50:43.229 INFO:teuthology.orchestra.run.smithi162.stdout:CentOS Stream 9 - AppStream 7.8 MB/s | 19 MB 00:02 2024-03-31T22:50:43.987 INFO:teuthology.orchestra.run.smithi142.stdout:CentOS Stream 9 - AppStream 14 MB/s | 19 MB 00:01 2024-03-31T22:50:45.761 INFO:teuthology.orchestra.run.smithi046.stdout:CentOS Stream 9 - CRB 4.6 MB/s | 6.1 MB 00:01 2024-03-31T22:50:47.394 INFO:teuthology.orchestra.run.smithi162.stdout:CentOS Stream 9 - CRB 9.5 MB/s | 6.1 MB 00:00 2024-03-31T22:50:47.443 INFO:teuthology.orchestra.run.smithi046.stdout:CentOS Stream 9 - Extras packages 32 kB/s | 16 kB 00:00 2024-03-31T22:50:48.232 INFO:teuthology.orchestra.run.smithi142.stdout:CentOS Stream 9 - CRB 9.1 MB/s | 6.1 MB 00:00 2024-03-31T22:50:48.760 INFO:teuthology.orchestra.run.smithi046.stdout:Extra Packages for Enterprise Linux 17 MB/s | 21 MB 00:01 2024-03-31T22:50:48.836 INFO:teuthology.orchestra.run.smithi162.stdout:CentOS Stream 9 - Extras packages 51 kB/s | 16 kB 00:00 2024-03-31T22:50:49.655 INFO:teuthology.orchestra.run.smithi142.stdout:CentOS Stream 9 - Extras packages 52 kB/s | 16 kB 00:00 2024-03-31T22:50:49.929 INFO:teuthology.orchestra.run.smithi162.stdout:Extra Packages for Enterprise Linux 21 MB/s | 21 MB 00:00 2024-03-31T22:50:50.915 INFO:teuthology.orchestra.run.smithi142.stdout:Extra Packages for Enterprise Linux 18 MB/s | 21 MB 00:01 2024-03-31T22:50:54.002 INFO:teuthology.orchestra.run.smithi046.stdout:lab-extras 31 kB/s | 1.7 kB 00:00 2024-03-31T22:50:54.962 INFO:teuthology.orchestra.run.smithi162.stdout:lab-extras 13 kB/s | 1.7 kB 00:00 2024-03-31T22:50:55.889 INFO:teuthology.orchestra.run.smithi142.stdout:lab-extras 14 kB/s | 1.7 kB 00:00 2024-03-31T22:50:56.385 INFO:teuthology.orchestra.run.smithi046.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-31T22:50:56.385 INFO:teuthology.orchestra.run.smithi046.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-31T22:50:56.484 INFO:teuthology.orchestra.run.smithi046.stdout:Dependencies resolved. 2024-03-31T22:50:56.487 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-03-31T22:50:56.488 INFO:teuthology.orchestra.run.smithi046.stdout: Package Arch Version Repository Size 2024-03-31T22:50:56.488 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-03-31T22:50:56.488 INFO:teuthology.orchestra.run.smithi046.stdout:Installing: 2024-03-31T22:50:56.488 INFO:teuthology.orchestra.run.smithi046.stdout: kernel x86_64 5.14.0-434.el9 CentOS-BaseOS 192 k 2024-03-31T22:50:56.488 INFO:teuthology.orchestra.run.smithi046.stdout:Installing dependencies: 2024-03-31T22:50:56.488 INFO:teuthology.orchestra.run.smithi046.stdout: kernel-core x86_64 5.14.0-434.el9 CentOS-BaseOS 15 M 2024-03-31T22:50:56.488 INFO:teuthology.orchestra.run.smithi046.stdout: kernel-modules x86_64 5.14.0-434.el9 CentOS-BaseOS 34 M 2024-03-31T22:50:56.488 INFO:teuthology.orchestra.run.smithi046.stdout: kernel-modules-core x86_64 5.14.0-434.el9 CentOS-BaseOS 28 M 2024-03-31T22:50:56.488 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-31T22:50:56.488 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction Summary 2024-03-31T22:50:56.488 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-03-31T22:50:56.488 INFO:teuthology.orchestra.run.smithi046.stdout:Install 4 Packages 2024-03-31T22:50:56.488 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-31T22:50:56.489 INFO:teuthology.orchestra.run.smithi046.stdout:Total download size: 76 M 2024-03-31T22:50:56.489 INFO:teuthology.orchestra.run.smithi046.stdout:Installed size: 123 M 2024-03-31T22:50:56.489 INFO:teuthology.orchestra.run.smithi046.stdout:Downloading Packages: 2024-03-31T22:50:56.645 INFO:teuthology.orchestra.run.smithi046.stdout:(1/4): kernel-5.14.0-434.el9.x86_64.rpm 1.2 MB/s | 192 kB 00:00 2024-03-31T22:50:57.013 INFO:teuthology.orchestra.run.smithi046.stdout:(2/4): kernel-core-5.14.0-434.el9.x86_64.rpm 29 MB/s | 15 MB 00:00 2024-03-31T22:50:57.252 INFO:teuthology.orchestra.run.smithi162.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-31T22:50:57.252 INFO:teuthology.orchestra.run.smithi162.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-31T22:50:57.353 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-03-31T22:50:57.364 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-03-31T22:50:57.365 INFO:teuthology.orchestra.run.smithi162.stdout: Package Arch Version Repository Size 2024-03-31T22:50:57.365 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-03-31T22:50:57.365 INFO:teuthology.orchestra.run.smithi162.stdout:Installing: 2024-03-31T22:50:57.365 INFO:teuthology.orchestra.run.smithi162.stdout: kernel x86_64 5.14.0-434.el9 CentOS-BaseOS 192 k 2024-03-31T22:50:57.365 INFO:teuthology.orchestra.run.smithi162.stdout:Installing dependencies: 2024-03-31T22:50:57.365 INFO:teuthology.orchestra.run.smithi162.stdout: kernel-core x86_64 5.14.0-434.el9 CentOS-BaseOS 15 M 2024-03-31T22:50:57.365 INFO:teuthology.orchestra.run.smithi162.stdout: kernel-modules x86_64 5.14.0-434.el9 CentOS-BaseOS 34 M 2024-03-31T22:50:57.365 INFO:teuthology.orchestra.run.smithi162.stdout: kernel-modules-core x86_64 5.14.0-434.el9 CentOS-BaseOS 28 M 2024-03-31T22:50:57.365 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-31T22:50:57.365 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction Summary 2024-03-31T22:50:57.365 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-03-31T22:50:57.365 INFO:teuthology.orchestra.run.smithi162.stdout:Install 4 Packages 2024-03-31T22:50:57.366 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-31T22:50:57.366 INFO:teuthology.orchestra.run.smithi162.stdout:Total download size: 76 M 2024-03-31T22:50:57.366 INFO:teuthology.orchestra.run.smithi162.stdout:Installed size: 123 M 2024-03-31T22:50:57.366 INFO:teuthology.orchestra.run.smithi162.stdout:Downloading Packages: 2024-03-31T22:50:57.529 INFO:teuthology.orchestra.run.smithi162.stdout:(1/4): kernel-5.14.0-434.el9.x86_64.rpm 1.1 MB/s | 192 kB 00:00 2024-03-31T22:50:57.537 INFO:teuthology.orchestra.run.smithi046.stdout:(3/4): kernel-modules-core-5.14.0-434.el9.x86_6 31 MB/s | 28 MB 00:00 2024-03-31T22:50:58.021 INFO:teuthology.orchestra.run.smithi162.stdout:(2/4): kernel-core-5.14.0-434.el9.x86_64.rpm 23 MB/s | 15 MB 00:00 2024-03-31T22:50:58.146 INFO:teuthology.orchestra.run.smithi142.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-31T22:50:58.146 INFO:teuthology.orchestra.run.smithi142.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-31T22:50:58.229 INFO:teuthology.orchestra.run.smithi142.stdout:Dependencies resolved. 2024-03-31T22:50:58.233 INFO:teuthology.orchestra.run.smithi142.stdout:================================================================================ 2024-03-31T22:50:58.233 INFO:teuthology.orchestra.run.smithi142.stdout: Package Arch Version Repository Size 2024-03-31T22:50:58.233 INFO:teuthology.orchestra.run.smithi142.stdout:================================================================================ 2024-03-31T22:50:58.233 INFO:teuthology.orchestra.run.smithi142.stdout:Installing: 2024-03-31T22:50:58.233 INFO:teuthology.orchestra.run.smithi142.stdout: kernel x86_64 5.14.0-434.el9 CentOS-BaseOS 192 k 2024-03-31T22:50:58.233 INFO:teuthology.orchestra.run.smithi142.stdout:Installing dependencies: 2024-03-31T22:50:58.233 INFO:teuthology.orchestra.run.smithi142.stdout: kernel-core x86_64 5.14.0-434.el9 CentOS-BaseOS 15 M 2024-03-31T22:50:58.233 INFO:teuthology.orchestra.run.smithi142.stdout: kernel-modules x86_64 5.14.0-434.el9 CentOS-BaseOS 34 M 2024-03-31T22:50:58.233 INFO:teuthology.orchestra.run.smithi142.stdout: kernel-modules-core x86_64 5.14.0-434.el9 CentOS-BaseOS 28 M 2024-03-31T22:50:58.233 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-03-31T22:50:58.233 INFO:teuthology.orchestra.run.smithi142.stdout:Transaction Summary 2024-03-31T22:50:58.234 INFO:teuthology.orchestra.run.smithi142.stdout:================================================================================ 2024-03-31T22:50:58.234 INFO:teuthology.orchestra.run.smithi142.stdout:Install 4 Packages 2024-03-31T22:50:58.234 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-03-31T22:50:58.234 INFO:teuthology.orchestra.run.smithi142.stdout:Total download size: 76 M 2024-03-31T22:50:58.234 INFO:teuthology.orchestra.run.smithi142.stdout:Installed size: 123 M 2024-03-31T22:50:58.234 INFO:teuthology.orchestra.run.smithi142.stdout:Downloading Packages: 2024-03-31T22:50:58.424 INFO:teuthology.orchestra.run.smithi142.stdout:(1/4): kernel-5.14.0-434.el9.x86_64.rpm 1.0 MB/s | 192 kB 00:00 2024-03-31T22:50:58.579 INFO:teuthology.orchestra.run.smithi046.stdout:(4/4): kernel-modules-5.14.0-434.el9.x86_64.rpm 16 MB/s | 34 MB 00:02 2024-03-31T22:50:58.582 INFO:teuthology.orchestra.run.smithi046.stdout:-------------------------------------------------------------------------------- 2024-03-31T22:50:58.583 INFO:teuthology.orchestra.run.smithi046.stdout:Total 37 MB/s | 76 MB 00:02 2024-03-31T22:50:58.583 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction check 2024-03-31T22:50:58.788 INFO:teuthology.orchestra.run.smithi162.stdout:(3/4): kernel-modules-5.14.0-434.el9.x86_64.rpm 24 MB/s | 34 MB 00:01 2024-03-31T22:50:59.007 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction check succeeded. 2024-03-31T22:50:59.008 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction test 2024-03-31T22:50:59.057 INFO:teuthology.orchestra.run.smithi142.stdout:(2/4): kernel-core-5.14.0-434.el9.x86_64.rpm 18 MB/s | 15 MB 00:00 2024-03-31T22:50:59.513 INFO:teuthology.orchestra.run.smithi162.stdout:(4/4): kernel-modules-core-5.14.0-434.el9.x86_6 14 MB/s | 28 MB 00:01 2024-03-31T22:50:59.516 INFO:teuthology.orchestra.run.smithi162.stdout:-------------------------------------------------------------------------------- 2024-03-31T22:50:59.516 INFO:teuthology.orchestra.run.smithi162.stdout:Total 36 MB/s | 76 MB 00:02 2024-03-31T22:50:59.517 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction check 2024-03-31T22:50:59.858 INFO:teuthology.orchestra.run.smithi142.stdout:(3/4): kernel-modules-5.14.0-434.el9.x86_64.rpm 21 MB/s | 34 MB 00:01 2024-03-31T22:50:59.880 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction test succeeded. 2024-03-31T22:50:59.880 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction 2024-03-31T22:50:59.881 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction check succeeded. 2024-03-31T22:50:59.881 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction test 2024-03-31T22:51:00.116 INFO:teuthology.orchestra.run.smithi142.stdout:(4/4): kernel-modules-core-5.14.0-434.el9.x86_6 16 MB/s | 28 MB 00:01 2024-03-31T22:51:00.119 INFO:teuthology.orchestra.run.smithi142.stdout:-------------------------------------------------------------------------------- 2024-03-31T22:51:00.119 INFO:teuthology.orchestra.run.smithi142.stdout:Total 41 MB/s | 76 MB 00:01 2024-03-31T22:51:00.120 INFO:teuthology.orchestra.run.smithi142.stdout:Running transaction check 2024-03-31T22:51:00.492 INFO:teuthology.orchestra.run.smithi142.stdout:Transaction check succeeded. 2024-03-31T22:51:00.492 INFO:teuthology.orchestra.run.smithi142.stdout:Running transaction test 2024-03-31T22:51:00.731 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction test succeeded. 2024-03-31T22:51:00.731 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction 2024-03-31T22:51:01.330 INFO:teuthology.orchestra.run.smithi142.stdout:Transaction test succeeded. 2024-03-31T22:51:01.331 INFO:teuthology.orchestra.run.smithi142.stdout:Running transaction 2024-03-31T22:51:02.303 INFO:teuthology.orchestra.run.smithi046.stdout: Preparing : 1/1 2024-03-31T22:51:02.934 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : kernel-modules-core-5.14.0-434.el9.x86_64 1/4 2024-03-31T22:51:03.021 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-31T22:51:03.041 INFO:teuthology.orchestra.run.smithi162.stdout: Preparing : 1/1 2024-03-31T22:51:03.620 INFO:teuthology.orchestra.run.smithi142.stdout: Preparing : 1/1 2024-03-31T22:51:03.635 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : kernel-modules-core-5.14.0-434.el9.x86_64 1/4 2024-03-31T22:51:03.746 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-31T22:51:04.204 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : kernel-modules-core-5.14.0-434.el9.x86_64 1/4 2024-03-31T22:51:04.299 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-31T22:51:05.947 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-31T22:51:06.079 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-31T22:51:06.507 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-31T22:51:06.646 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-31T22:51:07.045 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-31T22:51:07.174 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-31T22:51:11.190 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-31T22:51:11.229 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : kernel-5.14.0-434.el9.x86_64 4/4 2024-03-31T22:51:11.359 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-31T22:51:11.403 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : kernel-5.14.0-434.el9.x86_64 4/4 2024-03-31T22:51:11.874 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-31T22:51:12.027 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : kernel-5.14.0-434.el9.x86_64 4/4 2024-03-31T22:51:16.151 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: kernel-modules-core-5.14.0-434.el9.x86_64 4/4 2024-03-31T22:51:16.364 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: kernel-modules-core-5.14.0-434.el9.x86_64 4/4 2024-03-31T22:51:16.739 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: kernel-modules-core-5.14.0-434.el9.x86_64 4/4 2024-03-31T22:51:45.617 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: kernel-core-5.14.0-434.el9.x86_64 4/4 2024-03-31T22:51:45.617 INFO:teuthology.orchestra.run.smithi162.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-31T22:51:45.617 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-31T22:51:45.660 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: kernel-modules-5.14.0-434.el9.x86_64 4/4 2024-03-31T22:51:46.042 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: kernel-5.14.0-434.el9.x86_64 4/4 2024-03-31T22:51:46.043 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : kernel-5.14.0-434.el9.x86_64 1/4 2024-03-31T22:51:46.043 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-31T22:51:46.043 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-31T22:51:46.462 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : kernel-modules-core-5.14.0-434.el9.x86_64 4/4 2024-03-31T22:51:46.462 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-31T22:51:46.463 INFO:teuthology.orchestra.run.smithi162.stdout:Installed: 2024-03-31T22:51:46.463 INFO:teuthology.orchestra.run.smithi162.stdout: kernel-5.14.0-434.el9.x86_64 2024-03-31T22:51:46.463 INFO:teuthology.orchestra.run.smithi162.stdout: kernel-core-5.14.0-434.el9.x86_64 2024-03-31T22:51:46.463 INFO:teuthology.orchestra.run.smithi162.stdout: kernel-modules-5.14.0-434.el9.x86_64 2024-03-31T22:51:46.463 INFO:teuthology.orchestra.run.smithi162.stdout: kernel-modules-core-5.14.0-434.el9.x86_64 2024-03-31T22:51:46.463 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-31T22:51:46.463 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2024-03-31T22:51:46.514 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: kernel-core-5.14.0-434.el9.x86_64 4/4 2024-03-31T22:51:46.514 INFO:teuthology.orchestra.run.smithi142.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-31T22:51:46.514 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-03-31T22:51:46.560 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: kernel-modules-5.14.0-434.el9.x86_64 4/4 2024-03-31T22:51:46.776 DEBUG:teuthology.orchestra.run.smithi162:> rpm -q kernel | sort -rV | head -n 1 2024-03-31T22:51:46.875 INFO:teuthology.orchestra.run.smithi162.stdout:kernel-5.14.0-434.el9.x86_64 2024-03-31T22:51:46.875 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-434.el9.x86_64 2024-03-31T22:51:46.876 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-434.el9.x86_64 2024-03-31T22:51:46.876 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-31T22:51:46.876 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-31T22:51:46.876 INFO:teuthology.task.kernel:Installing distro kernel on mon.c... 2024-03-31T22:51:46.876 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi162.front.sepia.ceph.com, path=None, version=distro) 2024-03-31T22:51:46.876 DEBUG:teuthology.orchestra.run.smithi162:> rpm -q kernel | sort -rV | head -n 1 2024-03-31T22:51:46.956 INFO:teuthology.orchestra.run.smithi162.stdout:kernel-5.14.0-434.el9.x86_64 2024-03-31T22:51:46.957 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-434.el9.x86_64 2024-03-31T22:51:46.957 DEBUG:teuthology.orchestra.run.smithi162:> sudo rpm -qi grub2-tools 2024-03-31T22:51:46.964 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: kernel-5.14.0-434.el9.x86_64 4/4 2024-03-31T22:51:46.964 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : kernel-5.14.0-434.el9.x86_64 1/4 2024-03-31T22:51:46.964 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-31T22:51:46.964 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-31T22:51:47.000 INFO:teuthology.orchestra.run.smithi162.stdout:Name : grub2-tools 2024-03-31T22:51:47.000 INFO:teuthology.orchestra.run.smithi162.stdout:Epoch : 1 2024-03-31T22:51:47.000 INFO:teuthology.orchestra.run.smithi162.stdout:Version : 2.06 2024-03-31T22:51:47.000 INFO:teuthology.orchestra.run.smithi162.stdout:Release : 68.el9 2024-03-31T22:51:47.000 INFO:teuthology.orchestra.run.smithi162.stdout:Architecture: x86_64 2024-03-31T22:51:47.000 INFO:teuthology.orchestra.run.smithi162.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-31T22:51:47.000 INFO:teuthology.orchestra.run.smithi162.stdout:Group : Unspecified 2024-03-31T22:51:47.000 INFO:teuthology.orchestra.run.smithi162.stdout:Size : 8264254 2024-03-31T22:51:47.000 INFO:teuthology.orchestra.run.smithi162.stdout:License : GPLv3+ 2024-03-31T22:51:47.000 INFO:teuthology.orchestra.run.smithi162.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-31T22:51:47.000 INFO:teuthology.orchestra.run.smithi162.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-31T22:51:47.001 INFO:teuthology.orchestra.run.smithi162.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-31T22:51:47.001 INFO:teuthology.orchestra.run.smithi162.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-31T22:51:47.001 INFO:teuthology.orchestra.run.smithi162.stdout:Packager : builder@centos.org 2024-03-31T22:51:47.001 INFO:teuthology.orchestra.run.smithi162.stdout:Vendor : CentOS 2024-03-31T22:51:47.001 INFO:teuthology.orchestra.run.smithi162.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-31T22:51:47.001 INFO:teuthology.orchestra.run.smithi162.stdout:Summary : Support tools for GRUB. 2024-03-31T22:51:47.001 INFO:teuthology.orchestra.run.smithi162.stdout:Description : 2024-03-31T22:51:47.001 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-31T22:51:47.001 INFO:teuthology.orchestra.run.smithi162.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-31T22:51:47.001 INFO:teuthology.orchestra.run.smithi162.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-31T22:51:47.001 INFO:teuthology.orchestra.run.smithi162.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-31T22:51:47.001 INFO:teuthology.orchestra.run.smithi162.stdout:hardware devices. 2024-03-31T22:51:47.001 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-31T22:51:47.001 INFO:teuthology.orchestra.run.smithi162.stdout:This subpackage provides tools for support of all platforms. 2024-03-31T22:51:47.002 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-31T22:51:47.002 INFO:teuthology.task.kernel:Updating grub on smithi162 to boot 5.14.0-434.el9.x86_64 2024-03-31T22:51:47.003 DEBUG:teuthology.orchestra.run.smithi162:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-31T22:51:47.345 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: kernel-core-5.14.0-434.el9.x86_64 4/4 2024-03-31T22:51:47.345 INFO:teuthology.orchestra.run.smithi046.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-31T22:51:47.345 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-31T22:51:47.365 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : kernel-modules-core-5.14.0-434.el9.x86_64 4/4 2024-03-31T22:51:47.365 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-03-31T22:51:47.365 INFO:teuthology.orchestra.run.smithi142.stdout:Installed: 2024-03-31T22:51:47.365 INFO:teuthology.orchestra.run.smithi142.stdout: kernel-5.14.0-434.el9.x86_64 2024-03-31T22:51:47.365 INFO:teuthology.orchestra.run.smithi142.stdout: kernel-core-5.14.0-434.el9.x86_64 2024-03-31T22:51:47.365 INFO:teuthology.orchestra.run.smithi142.stdout: kernel-modules-5.14.0-434.el9.x86_64 2024-03-31T22:51:47.365 INFO:teuthology.orchestra.run.smithi142.stdout: kernel-modules-core-5.14.0-434.el9.x86_64 2024-03-31T22:51:47.366 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-03-31T22:51:47.366 INFO:teuthology.orchestra.run.smithi142.stdout:Complete! 2024-03-31T22:51:47.390 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: kernel-modules-5.14.0-434.el9.x86_64 4/4 2024-03-31T22:51:47.627 INFO:teuthology.orchestra.run.smithi162.stderr:Generating grub configuration file ... 2024-03-31T22:51:47.701 DEBUG:teuthology.orchestra.run.smithi142:> rpm -q kernel | sort -rV | head -n 1 2024-03-31T22:51:47.777 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: kernel-5.14.0-434.el9.x86_64 4/4 2024-03-31T22:51:47.777 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : kernel-5.14.0-434.el9.x86_64 1/4 2024-03-31T22:51:47.777 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-31T22:51:47.777 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-31T22:51:47.844 INFO:teuthology.orchestra.run.smithi142.stdout:kernel-5.14.0-434.el9.x86_64 2024-03-31T22:51:47.844 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-434.el9.x86_64 2024-03-31T22:51:47.844 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-434.el9.x86_64 2024-03-31T22:51:47.844 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-31T22:51:47.845 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-31T22:51:47.845 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-03-31T22:51:47.845 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi142.front.sepia.ceph.com, path=None, version=distro) 2024-03-31T22:51:47.845 DEBUG:teuthology.orchestra.run.smithi142:> rpm -q kernel | sort -rV | head -n 1 2024-03-31T22:51:47.924 INFO:teuthology.orchestra.run.smithi142.stdout:kernel-5.14.0-434.el9.x86_64 2024-03-31T22:51:47.925 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-434.el9.x86_64 2024-03-31T22:51:47.926 DEBUG:teuthology.orchestra.run.smithi142:> sudo rpm -qi grub2-tools 2024-03-31T22:51:47.960 INFO:teuthology.orchestra.run.smithi142.stdout:Name : grub2-tools 2024-03-31T22:51:47.960 INFO:teuthology.orchestra.run.smithi142.stdout:Epoch : 1 2024-03-31T22:51:47.960 INFO:teuthology.orchestra.run.smithi142.stdout:Version : 2.06 2024-03-31T22:51:47.960 INFO:teuthology.orchestra.run.smithi142.stdout:Release : 68.el9 2024-03-31T22:51:47.960 INFO:teuthology.orchestra.run.smithi142.stdout:Architecture: x86_64 2024-03-31T22:51:47.960 INFO:teuthology.orchestra.run.smithi142.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-31T22:51:47.961 INFO:teuthology.orchestra.run.smithi142.stdout:Group : Unspecified 2024-03-31T22:51:47.961 INFO:teuthology.orchestra.run.smithi142.stdout:Size : 8264254 2024-03-31T22:51:47.961 INFO:teuthology.orchestra.run.smithi142.stdout:License : GPLv3+ 2024-03-31T22:51:47.961 INFO:teuthology.orchestra.run.smithi142.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-31T22:51:47.961 INFO:teuthology.orchestra.run.smithi142.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-31T22:51:47.961 INFO:teuthology.orchestra.run.smithi142.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-31T22:51:47.961 INFO:teuthology.orchestra.run.smithi142.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-31T22:51:47.961 INFO:teuthology.orchestra.run.smithi142.stdout:Packager : builder@centos.org 2024-03-31T22:51:47.961 INFO:teuthology.orchestra.run.smithi142.stdout:Vendor : CentOS 2024-03-31T22:51:47.961 INFO:teuthology.orchestra.run.smithi142.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-31T22:51:47.961 INFO:teuthology.orchestra.run.smithi142.stdout:Summary : Support tools for GRUB. 2024-03-31T22:51:47.961 INFO:teuthology.orchestra.run.smithi142.stdout:Description : 2024-03-31T22:51:47.961 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-03-31T22:51:47.961 INFO:teuthology.orchestra.run.smithi142.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-31T22:51:47.962 INFO:teuthology.orchestra.run.smithi142.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-31T22:51:47.962 INFO:teuthology.orchestra.run.smithi142.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-31T22:51:47.962 INFO:teuthology.orchestra.run.smithi142.stdout:hardware devices. 2024-03-31T22:51:47.962 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-03-31T22:51:47.962 INFO:teuthology.orchestra.run.smithi142.stdout:This subpackage provides tools for support of all platforms. 2024-03-31T22:51:47.963 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-31T22:51:47.963 INFO:teuthology.task.kernel:Updating grub on smithi142 to boot 5.14.0-434.el9.x86_64 2024-03-31T22:51:47.963 DEBUG:teuthology.orchestra.run.smithi142:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-31T22:51:48.218 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : kernel-modules-core-5.14.0-434.el9.x86_64 4/4 2024-03-31T22:51:48.218 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-31T22:51:48.218 INFO:teuthology.orchestra.run.smithi046.stdout:Installed: 2024-03-31T22:51:48.219 INFO:teuthology.orchestra.run.smithi046.stdout: kernel-5.14.0-434.el9.x86_64 2024-03-31T22:51:48.219 INFO:teuthology.orchestra.run.smithi046.stdout: kernel-core-5.14.0-434.el9.x86_64 2024-03-31T22:51:48.219 INFO:teuthology.orchestra.run.smithi046.stdout: kernel-modules-5.14.0-434.el9.x86_64 2024-03-31T22:51:48.219 INFO:teuthology.orchestra.run.smithi046.stdout: kernel-modules-core-5.14.0-434.el9.x86_64 2024-03-31T22:51:48.219 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-31T22:51:48.219 INFO:teuthology.orchestra.run.smithi046.stdout:Complete! 2024-03-31T22:51:48.555 INFO:teuthology.orchestra.run.smithi142.stderr:Generating grub configuration file ... 2024-03-31T22:51:48.560 DEBUG:teuthology.orchestra.run.smithi046:> rpm -q kernel | sort -rV | head -n 1 2024-03-31T22:51:48.665 INFO:teuthology.orchestra.run.smithi046.stdout:kernel-5.14.0-434.el9.x86_64 2024-03-31T22:51:48.665 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-434.el9.x86_64 2024-03-31T22:51:48.665 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-434.el9.x86_64 2024-03-31T22:51:48.665 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-31T22:51:48.665 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-31T22:51:48.665 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-03-31T22:51:48.665 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi046.front.sepia.ceph.com, path=None, version=distro) 2024-03-31T22:51:48.666 DEBUG:teuthology.orchestra.run.smithi046:> rpm -q kernel | sort -rV | head -n 1 2024-03-31T22:51:48.749 INFO:teuthology.orchestra.run.smithi046.stdout:kernel-5.14.0-434.el9.x86_64 2024-03-31T22:51:48.749 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-434.el9.x86_64 2024-03-31T22:51:48.750 DEBUG:teuthology.orchestra.run.smithi046:> sudo rpm -qi grub2-tools 2024-03-31T22:51:48.795 INFO:teuthology.orchestra.run.smithi046.stdout:Name : grub2-tools 2024-03-31T22:51:48.795 INFO:teuthology.orchestra.run.smithi046.stdout:Epoch : 1 2024-03-31T22:51:48.796 INFO:teuthology.orchestra.run.smithi046.stdout:Version : 2.06 2024-03-31T22:51:48.796 INFO:teuthology.orchestra.run.smithi046.stdout:Release : 68.el9 2024-03-31T22:51:48.796 INFO:teuthology.orchestra.run.smithi046.stdout:Architecture: x86_64 2024-03-31T22:51:48.796 INFO:teuthology.orchestra.run.smithi046.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-31T22:51:48.796 INFO:teuthology.orchestra.run.smithi046.stdout:Group : Unspecified 2024-03-31T22:51:48.796 INFO:teuthology.orchestra.run.smithi046.stdout:Size : 8264254 2024-03-31T22:51:48.796 INFO:teuthology.orchestra.run.smithi046.stdout:License : GPLv3+ 2024-03-31T22:51:48.796 INFO:teuthology.orchestra.run.smithi046.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-31T22:51:48.796 INFO:teuthology.orchestra.run.smithi046.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-31T22:51:48.796 INFO:teuthology.orchestra.run.smithi046.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-31T22:51:48.796 INFO:teuthology.orchestra.run.smithi046.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-31T22:51:48.796 INFO:teuthology.orchestra.run.smithi046.stdout:Packager : builder@centos.org 2024-03-31T22:51:48.796 INFO:teuthology.orchestra.run.smithi046.stdout:Vendor : CentOS 2024-03-31T22:51:48.796 INFO:teuthology.orchestra.run.smithi046.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-31T22:51:48.796 INFO:teuthology.orchestra.run.smithi046.stdout:Summary : Support tools for GRUB. 2024-03-31T22:51:48.797 INFO:teuthology.orchestra.run.smithi046.stdout:Description : 2024-03-31T22:51:48.797 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-31T22:51:48.797 INFO:teuthology.orchestra.run.smithi046.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-31T22:51:48.797 INFO:teuthology.orchestra.run.smithi046.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-31T22:51:48.797 INFO:teuthology.orchestra.run.smithi046.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-31T22:51:48.797 INFO:teuthology.orchestra.run.smithi046.stdout:hardware devices. 2024-03-31T22:51:48.797 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-31T22:51:48.797 INFO:teuthology.orchestra.run.smithi046.stdout:This subpackage provides tools for support of all platforms. 2024-03-31T22:51:48.798 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-31T22:51:48.798 INFO:teuthology.task.kernel:Updating grub on smithi046 to boot 5.14.0-434.el9.x86_64 2024-03-31T22:51:48.798 DEBUG:teuthology.orchestra.run.smithi046:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-31T22:51:48.931 INFO:teuthology.orchestra.run.smithi162.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-31T22:51:48.970 INFO:teuthology.orchestra.run.smithi162.stderr:done 2024-03-31T22:51:48.972 DEBUG:teuthology.orchestra.run.smithi162:> mktemp 2024-03-31T22:51:48.987 INFO:teuthology.orchestra.run.smithi162.stdout:/tmp/tmp.P4fmbE5Iwy 2024-03-31T22:51:48.987 DEBUG:teuthology.orchestra.run.smithi162:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.P4fmbE5Iwy 2024-03-31T22:51:49.056 DEBUG:teuthology.orchestra.run.smithi162:> sudo chmod 0666 /tmp/tmp.P4fmbE5Iwy 2024-03-31T22:51:49.197 DEBUG:teuthology.orchestra.remote:smithi162:/tmp/tmp.P4fmbE5Iwy is 6KB 2024-03-31T22:51:49.217 DEBUG:teuthology.orchestra.run.smithi162:> rm -fr /tmp/tmp.P4fmbE5Iwy 2024-03-31T22:51:49.232 DEBUG:teuthology.orchestra.run.smithi162:> sudo /bin/ls /boot/loader/entries || true 2024-03-31T22:51:49.303 INFO:teuthology.orchestra.run.smithi162.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-31T22:51:49.303 INFO:teuthology.orchestra.run.smithi162.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-31T22:51:49.303 INFO:teuthology.orchestra.run.smithi162.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-31T22:51:49.303 INFO:teuthology.orchestra.run.smithi162.stdout:15949e3c88704300882482eddfd7baae-5.14.0-434.el9.x86_64.conf 2024-03-31T22:51:49.304 DEBUG:teuthology.orchestra.run.smithi162:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-434.el9.x86_64 2024-03-31T22:51:49.355 INFO:teuthology.orchestra.run.smithi046.stderr:Generating grub configuration file ... 2024-03-31T22:51:49.442 DEBUG:teuthology.orchestra.run.smithi162:> sudo shutdown -r now 2024-03-31T22:51:49.920 INFO:teuthology.orchestra.run.smithi142.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-31T22:51:49.965 INFO:teuthology.orchestra.run.smithi142.stderr:done 2024-03-31T22:51:49.967 DEBUG:teuthology.orchestra.run.smithi142:> mktemp 2024-03-31T22:51:49.982 INFO:teuthology.orchestra.run.smithi142.stdout:/tmp/tmp.te62EyQ7YX 2024-03-31T22:51:49.982 DEBUG:teuthology.orchestra.run.smithi142:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.te62EyQ7YX 2024-03-31T22:51:50.063 DEBUG:teuthology.orchestra.run.smithi142:> sudo chmod 0666 /tmp/tmp.te62EyQ7YX 2024-03-31T22:51:50.203 DEBUG:teuthology.orchestra.remote:smithi142:/tmp/tmp.te62EyQ7YX is 6KB 2024-03-31T22:51:50.251 DEBUG:teuthology.orchestra.run.smithi142:> rm -fr /tmp/tmp.te62EyQ7YX 2024-03-31T22:51:50.264 DEBUG:teuthology.orchestra.run.smithi142:> sudo /bin/ls /boot/loader/entries || true 2024-03-31T22:51:50.329 INFO:teuthology.orchestra.run.smithi142.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-31T22:51:50.329 INFO:teuthology.orchestra.run.smithi142.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-31T22:51:50.329 INFO:teuthology.orchestra.run.smithi142.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-31T22:51:50.329 INFO:teuthology.orchestra.run.smithi142.stdout:15949e3c88704300882482eddfd7baae-5.14.0-434.el9.x86_64.conf 2024-03-31T22:51:50.330 DEBUG:teuthology.orchestra.run.smithi142:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-434.el9.x86_64 2024-03-31T22:51:50.468 DEBUG:teuthology.orchestra.run.smithi142:> sudo shutdown -r now 2024-03-31T22:51:50.987 INFO:teuthology.orchestra.run.smithi046.stderr:File descriptor 3 (pipe:[28315]) leaked on vgs invocation. Parent PID 18170: grub2-probe 2024-03-31T22:51:50.987 INFO:teuthology.orchestra.run.smithi046.stderr:File descriptor 9 (pipe:[30485]) leaked on vgs invocation. Parent PID 18170: grub2-probe 2024-03-31T22:51:51.020 INFO:teuthology.orchestra.run.smithi046.stderr:File descriptor 3 (pipe:[28315]) leaked on vgs invocation. Parent PID 18170: grub2-probe 2024-03-31T22:51:51.021 INFO:teuthology.orchestra.run.smithi046.stderr:File descriptor 9 (pipe:[30485]) leaked on vgs invocation. Parent PID 18170: grub2-probe 2024-03-31T22:51:51.164 INFO:teuthology.orchestra.run.smithi046.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-31T22:51:51.180 INFO:teuthology.orchestra.run.smithi046.stderr:done 2024-03-31T22:51:51.182 DEBUG:teuthology.orchestra.run.smithi046:> mktemp 2024-03-31T22:51:51.197 INFO:teuthology.orchestra.run.smithi046.stdout:/tmp/tmp.5wR59LSGMb 2024-03-31T22:51:51.197 DEBUG:teuthology.orchestra.run.smithi046:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.5wR59LSGMb 2024-03-31T22:51:51.269 DEBUG:teuthology.orchestra.run.smithi046:> sudo chmod 0666 /tmp/tmp.5wR59LSGMb 2024-03-31T22:51:51.413 DEBUG:teuthology.orchestra.remote:smithi046:/tmp/tmp.5wR59LSGMb is 6KB 2024-03-31T22:51:51.463 DEBUG:teuthology.orchestra.run.smithi046:> rm -fr /tmp/tmp.5wR59LSGMb 2024-03-31T22:51:51.477 DEBUG:teuthology.orchestra.run.smithi046:> sudo /bin/ls /boot/loader/entries || true 2024-03-31T22:51:51.547 INFO:teuthology.orchestra.run.smithi046.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-31T22:51:51.547 INFO:teuthology.orchestra.run.smithi046.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-31T22:51:51.547 INFO:teuthology.orchestra.run.smithi046.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-31T22:51:51.548 INFO:teuthology.orchestra.run.smithi046.stdout:15949e3c88704300882482eddfd7baae-5.14.0-434.el9.x86_64.conf 2024-03-31T22:51:51.549 DEBUG:teuthology.orchestra.run.smithi046:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-434.el9.x86_64 2024-03-31T22:51:51.713 DEBUG:teuthology.orchestra.run.smithi046:> sudo shutdown -r now 2024-03-31T22:52:19.471 INFO:teuthology.task.kernel:Checking client mon.c for new kernel version... 2024-03-31T22:52:19.471 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi162.front.sepia.ceph.com' 2024-03-31T22:52:19.472 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi162.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-31T22:52:20.471 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-03-31T22:52:20.471 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi142.front.sepia.ceph.com' 2024-03-31T22:52:20.472 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi142.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-31T22:52:21.716 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-03-31T22:52:21.716 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi046.front.sepia.ceph.com' 2024-03-31T22:52:21.717 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi046.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-31T22:52:37.790 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.162 2024-03-31T22:52:38.814 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.142 2024-03-31T22:52:40.098 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.46 2024-03-31T22:52:46.793 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi162.front.sepia.ceph.com' 2024-03-31T22:52:46.794 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi162.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-31T22:52:47.815 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi142.front.sepia.ceph.com' 2024-03-31T22:52:47.816 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi142.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-31T22:52:49.101 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi046.front.sepia.ceph.com' 2024-03-31T22:52:49.101 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi046.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-31T22:52:49.854 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.162 2024-03-31T22:52:50.878 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.142 2024-03-31T22:52:52.158 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.46 2024-03-31T22:53:01.862 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi162.front.sepia.ceph.com' 2024-03-31T22:53:01.863 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi162.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-31T22:53:02.881 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi142.front.sepia.ceph.com' 2024-03-31T22:53:02.882 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi142.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-31T22:53:04.161 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi046.front.sepia.ceph.com' 2024-03-31T22:53:04.161 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi046.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-31T22:53:20.286 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.162 2024-03-31T22:53:22.590 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.46 2024-03-31T22:53:34.810 DEBUG:teuthology.orchestra.run.smithi142:> true 2024-03-31T22:53:35.212 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi142.front.sepia.ceph.com' 2024-03-31T22:53:35.212 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-434.el9.x86_64"... 2024-03-31T22:53:35.212 DEBUG:teuthology.orchestra.run.smithi142:> uname -r 2024-03-31T22:53:35.226 INFO:teuthology.orchestra.run.smithi142.stdout:5.14.0-434.el9.x86_64 2024-03-31T22:53:35.227 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-434.el9.x86_64 vs 5.14.0-434.el9.x86_64 2024-03-31T22:53:35.227 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-31T22:53:35.227 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-31T22:53:35.287 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi162.front.sepia.ceph.com' 2024-03-31T22:53:35.288 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi162.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-31T22:53:35.758 DEBUG:teuthology.orchestra.run.smithi162:> true 2024-03-31T22:53:36.150 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi162.front.sepia.ceph.com' 2024-03-31T22:53:36.150 INFO:teuthology.task.kernel:Checking kernel version of mon.c, want "5.14.0-434.el9.x86_64"... 2024-03-31T22:53:36.151 DEBUG:teuthology.orchestra.run.smithi162:> uname -r 2024-03-31T22:53:36.204 INFO:teuthology.orchestra.run.smithi162.stdout:5.14.0-434.el9.x86_64 2024-03-31T22:53:36.204 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-434.el9.x86_64 vs 5.14.0-434.el9.x86_64 2024-03-31T22:53:36.204 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-31T22:53:36.205 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-31T22:53:36.228 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-03-31T22:53:36.228 DEBUG:teuthology.orchestra.run.smithi142:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-31T22:53:36.294 INFO:teuthology.orchestra.run.smithi142.stdout:ttyS1 2024-03-31T22:53:36.348 DEBUG:teuthology.parallel:result is None 2024-03-31T22:53:37.205 INFO:teuthology.task.kernel:Enabling kdb on mon.c... 2024-03-31T22:53:37.205 DEBUG:teuthology.orchestra.run.smithi162:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-31T22:53:37.262 INFO:teuthology.orchestra.run.smithi162.stdout:ttyS1 2024-03-31T22:53:37.310 DEBUG:teuthology.parallel:result is None 2024-03-31T22:53:37.591 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi046.front.sepia.ceph.com' 2024-03-31T22:53:37.592 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi046.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-31T22:53:38.025 DEBUG:teuthology.orchestra.run.smithi046:> true 2024-03-31T22:53:38.524 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi046.front.sepia.ceph.com' 2024-03-31T22:53:38.524 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-434.el9.x86_64"... 2024-03-31T22:53:38.524 DEBUG:teuthology.orchestra.run.smithi046:> uname -r 2024-03-31T22:53:38.539 INFO:teuthology.orchestra.run.smithi046.stdout:5.14.0-434.el9.x86_64 2024-03-31T22:53:38.540 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-434.el9.x86_64 vs 5.14.0-434.el9.x86_64 2024-03-31T22:53:38.540 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-31T22:53:38.540 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-31T22:53:39.541 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-03-31T22:53:39.541 DEBUG:teuthology.orchestra.run.smithi046:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-31T22:53:39.604 INFO:teuthology.orchestra.run.smithi046.stdout:ttyS1 2024-03-31T22:53:39.647 DEBUG:teuthology.parallel:result is None 2024-03-31T22:53:39.647 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-31T22:53:39.654 INFO:teuthology.task.internal:Creating test directory... 2024-03-31T22:53:39.654 DEBUG:teuthology.orchestra.run.smithi046:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-31T22:53:39.656 DEBUG:teuthology.orchestra.run.smithi142:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-31T22:53:39.659 DEBUG:teuthology.orchestra.run.smithi162:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-31T22:53:39.673 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-31T22:53:39.680 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-31T22:53:39.690 INFO:teuthology.task.internal:Creating archive directory... 2024-03-31T22:53:39.690 DEBUG:teuthology.orchestra.run.smithi046:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-31T22:53:39.715 DEBUG:teuthology.orchestra.run.smithi142:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-31T22:53:39.717 DEBUG:teuthology.orchestra.run.smithi162:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-31T22:53:39.762 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-31T22:53:39.768 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-31T22:53:39.768 DEBUG:teuthology.orchestra.run.smithi046:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-31T22:53:39.774 DEBUG:teuthology.orchestra.run.smithi142:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-31T22:53:39.776 DEBUG:teuthology.orchestra.run.smithi162:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-31T22:53:39.806 INFO:teuthology.orchestra.run.smithi046.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-31T22:53:39.814 INFO:teuthology.orchestra.run.smithi142.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-31T22:53:39.819 INFO:teuthology.orchestra.run.smithi046.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-31T22:53:39.823 INFO:teuthology.orchestra.run.smithi162.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-31T22:53:39.828 INFO:teuthology.orchestra.run.smithi142.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-31T22:53:39.836 INFO:teuthology.orchestra.run.smithi162.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-31T22:53:39.838 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-31T22:53:39.845 INFO:teuthology.task.internal:Configuring sudo... 2024-03-31T22:53:39.845 DEBUG:teuthology.orchestra.run.smithi046:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-31T22:53:39.867 DEBUG:teuthology.orchestra.run.smithi142:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-31T22:53:39.872 DEBUG:teuthology.orchestra.run.smithi162:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-31T22:53:39.903 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-31T22:53:39.912 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-31T22:53:39.912 DEBUG:teuthology.orchestra.run.smithi046:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-31T22:53:39.938 DEBUG:teuthology.orchestra.run.smithi142:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-31T22:53:39.940 DEBUG:teuthology.orchestra.run.smithi162:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-31T22:53:39.955 DEBUG:teuthology.orchestra.run.smithi046:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-31T22:53:40.038 DEBUG:teuthology.orchestra.run.smithi046:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-31T22:53:40.090 DEBUG:teuthology.orchestra.run.smithi046:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-31T22:53:40.175 DEBUG:teuthology.orchestra.run.smithi046:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-31T22:53:40.219 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-03-31T22:53:40.220 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-31T22:53:40.290 DEBUG:teuthology.orchestra.run.smithi142:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-31T22:53:40.334 DEBUG:teuthology.orchestra.run.smithi142:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-31T22:53:40.384 DEBUG:teuthology.orchestra.run.smithi142:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-31T22:53:40.466 DEBUG:teuthology.orchestra.run.smithi142:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-31T22:53:40.510 DEBUG:teuthology.orchestra.run.smithi142:> set -ex 2024-03-31T22:53:40.511 DEBUG:teuthology.orchestra.run.smithi142:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-31T22:53:40.578 DEBUG:teuthology.orchestra.run.smithi162:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-31T22:53:40.623 DEBUG:teuthology.orchestra.run.smithi162:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-31T22:53:40.689 DEBUG:teuthology.orchestra.run.smithi162:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-31T22:53:40.743 DEBUG:teuthology.orchestra.run.smithi162:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-31T22:53:40.805 DEBUG:teuthology.orchestra.run.smithi162:> set -ex 2024-03-31T22:53:40.805 DEBUG:teuthology.orchestra.run.smithi162:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-31T22:53:40.851 DEBUG:teuthology.orchestra.run.smithi046:> sudo service rsyslog restart 2024-03-31T22:53:40.853 DEBUG:teuthology.orchestra.run.smithi142:> sudo service rsyslog restart 2024-03-31T22:53:40.855 DEBUG:teuthology.orchestra.run.smithi162:> sudo service rsyslog restart 2024-03-31T22:53:40.902 INFO:teuthology.orchestra.run.smithi046.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-31T22:53:40.910 INFO:teuthology.orchestra.run.smithi142.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-31T22:53:41.023 INFO:teuthology.orchestra.run.smithi162.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-31T22:53:41.412 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-31T22:53:41.421 INFO:teuthology.task.internal:Starting timer... 2024-03-31T22:53:41.421 INFO:teuthology.run_tasks:Running task pcp... 2024-03-31T22:53:41.494 INFO:teuthology.run_tasks:Running task selinux... 2024-03-31T22:53:41.504 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0', 'scontext=system_u:system_r:getty_t:s0']} 2024-03-31T22:53:41.504 DEBUG:teuthology.orchestra.run.smithi046:> sudo service auditd rotate 2024-03-31T22:53:41.558 INFO:teuthology.orchestra.run.smithi046.stdout:Rotating logs: 2024-03-31T22:53:41.560 DEBUG:teuthology.orchestra.run.smithi142:> sudo service auditd rotate 2024-03-31T22:53:41.606 INFO:teuthology.orchestra.run.smithi142.stdout:Rotating logs: 2024-03-31T22:53:41.608 DEBUG:teuthology.orchestra.run.smithi162:> sudo service auditd rotate 2024-03-31T22:53:41.671 INFO:teuthology.orchestra.run.smithi162.stdout:Rotating logs: 2024-03-31T22:53:41.673 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-31T22:53:41.673 DEBUG:teuthology.orchestra.run.smithi046:> /usr/sbin/getenforce 2024-03-31T22:53:41.712 INFO:teuthology.orchestra.run.smithi046.stdout:Permissive 2024-03-31T22:53:41.712 DEBUG:teuthology.orchestra.run.smithi142:> /usr/sbin/getenforce 2024-03-31T22:53:41.748 INFO:teuthology.orchestra.run.smithi142.stdout:Permissive 2024-03-31T22:53:41.748 DEBUG:teuthology.orchestra.run.smithi162:> /usr/sbin/getenforce 2024-03-31T22:53:41.772 INFO:teuthology.orchestra.run.smithi162.stdout:Permissive 2024-03-31T22:53:41.772 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi046.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi142.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi162.front.sepia.ceph.com': 'permissive'} 2024-03-31T22:53:41.772 DEBUG:teuthology.orchestra.run.smithi046:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-31T22:53:41.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-31T22:53:41.799 DEBUG:teuthology.orchestra.run.smithi142:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-31T22:53:41.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-31T22:53:41.828 DEBUG:teuthology.orchestra.run.smithi162:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-31T22:53:41.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-31T22:53:41.856 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-31T22:53:41.857 DEBUG:teuthology.orchestra.run.smithi046:> sudo /usr/sbin/setenforce permissive 2024-03-31T22:53:41.889 DEBUG:teuthology.orchestra.run.smithi142:> sudo /usr/sbin/setenforce permissive 2024-03-31T22:53:41.942 DEBUG:teuthology.orchestra.run.smithi162:> sudo /usr/sbin/setenforce permissive 2024-03-31T22:53:42.074 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-31T22:53:42.084 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-31T22:53:42.100 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-31T22:53:42.101 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi046.front.sepia.ceph.com,smithi142.front.sepia.ceph.com,smithi162.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-31T22:58:02.918 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi046.front.sepia.ceph.com'), Remote(name='ubuntu@smithi142.front.sepia.ceph.com'), Remote(name='ubuntu@smithi162.front.sepia.ceph.com')] 2024-03-31T22:58:02.920 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi046.front.sepia.ceph.com' 2024-03-31T22:58:02.921 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi046.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-31T22:58:03.014 DEBUG:teuthology.orchestra.run.smithi046:> true 2024-03-31T22:58:03.097 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi046.front.sepia.ceph.com' 2024-03-31T22:58:03.098 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi142.front.sepia.ceph.com' 2024-03-31T22:58:03.098 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi142.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-31T22:58:03.191 DEBUG:teuthology.orchestra.run.smithi142:> true 2024-03-31T22:58:03.276 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi142.front.sepia.ceph.com' 2024-03-31T22:58:03.277 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi162.front.sepia.ceph.com' 2024-03-31T22:58:03.278 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi162.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-31T22:58:03.369 DEBUG:teuthology.orchestra.run.smithi162:> true 2024-03-31T22:58:03.449 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi162.front.sepia.ceph.com' 2024-03-31T22:58:03.450 INFO:teuthology.run_tasks:Running task clock... 2024-03-31T22:58:03.460 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-31T22:58:03.460 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-31T22:58:03.460 DEBUG:teuthology.orchestra.run.smithi046:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-31T22:58:03.463 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-31T22:58:03.463 DEBUG:teuthology.orchestra.run.smithi142:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-31T22:58:03.465 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-31T22:58:03.465 DEBUG:teuthology.orchestra.run.smithi162:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-31T22:58:03.494 INFO:teuthology.orchestra.run.smithi142.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-31T22:58:03.496 INFO:teuthology.orchestra.run.smithi046.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-31T22:58:03.511 INFO:teuthology.orchestra.run.smithi142.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-31T22:58:03.514 INFO:teuthology.orchestra.run.smithi046.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-31T22:58:03.521 INFO:teuthology.orchestra.run.smithi162.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-31T22:58:03.538 INFO:teuthology.orchestra.run.smithi162.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-31T22:58:03.539 INFO:teuthology.orchestra.run.smithi142.stderr:sudo: ntpd: command not found 2024-03-31T22:58:03.545 INFO:teuthology.orchestra.run.smithi046.stderr:sudo: ntpd: command not found 2024-03-31T22:58:03.553 INFO:teuthology.orchestra.run.smithi142.stdout:506 Cannot talk to daemon 2024-03-31T22:58:03.559 INFO:teuthology.orchestra.run.smithi046.stdout:506 Cannot talk to daemon 2024-03-31T22:58:03.568 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: ntpd: command not found 2024-03-31T22:58:03.569 INFO:teuthology.orchestra.run.smithi142.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-31T22:58:03.577 INFO:teuthology.orchestra.run.smithi046.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-31T22:58:03.582 INFO:teuthology.orchestra.run.smithi162.stdout:506 Cannot talk to daemon 2024-03-31T22:58:03.585 INFO:teuthology.orchestra.run.smithi142.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-31T22:58:03.593 INFO:teuthology.orchestra.run.smithi046.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-31T22:58:03.598 INFO:teuthology.orchestra.run.smithi162.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-31T22:58:03.614 INFO:teuthology.orchestra.run.smithi162.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-31T22:58:03.644 INFO:teuthology.orchestra.run.smithi142.stderr:bash: line 1: ntpq: command not found 2024-03-31T22:58:03.644 INFO:teuthology.orchestra.run.smithi046.stderr:bash: line 1: ntpq: command not found 2024-03-31T22:58:03.646 INFO:teuthology.orchestra.run.smithi142.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-31T22:58:03.647 INFO:teuthology.orchestra.run.smithi142.stdout:=============================================================================== 2024-03-31T22:58:03.647 INFO:teuthology.orchestra.run.smithi142.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-31T22:58:03.647 INFO:teuthology.orchestra.run.smithi142.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-31T22:58:03.647 INFO:teuthology.orchestra.run.smithi142.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-31T22:58:03.647 INFO:teuthology.orchestra.run.smithi142.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-31T22:58:03.648 INFO:teuthology.orchestra.run.smithi046.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-31T22:58:03.648 INFO:teuthology.orchestra.run.smithi046.stdout:=============================================================================== 2024-03-31T22:58:03.648 INFO:teuthology.orchestra.run.smithi046.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-31T22:58:03.649 INFO:teuthology.orchestra.run.smithi046.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-31T22:58:03.649 INFO:teuthology.orchestra.run.smithi046.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-31T22:58:03.649 INFO:teuthology.orchestra.run.smithi046.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-31T22:58:03.664 INFO:teuthology.orchestra.run.smithi162.stderr:bash: line 1: ntpq: command not found 2024-03-31T22:58:03.668 INFO:teuthology.orchestra.run.smithi162.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-31T22:58:03.668 INFO:teuthology.orchestra.run.smithi162.stdout:=============================================================================== 2024-03-31T22:58:03.668 INFO:teuthology.orchestra.run.smithi162.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-31T22:58:03.668 INFO:teuthology.orchestra.run.smithi162.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-31T22:58:03.668 INFO:teuthology.orchestra.run.smithi162.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-31T22:58:03.668 INFO:teuthology.orchestra.run.smithi162.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-31T22:58:03.668 INFO:teuthology.run_tasks:Running task install... 2024-03-31T22:58:03.677 DEBUG:teuthology.task.install:project ceph 2024-03-31T22:58:03.677 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '384f31f3d3aee7f8ba3f60846b8844c049165583'}} 2024-03-31T22:58:03.677 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '384f31f3d3aee7f8ba3f60846b8844c049165583'} 2024-03-31T22:58:03.678 INFO:teuthology.task.install:Using flavor: default 2024-03-31T22:58:03.684 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-03-31T22:58:03.684 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']} 2024-03-31T22:58:03.684 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '384f31f3d3aee7f8ba3f60846b8844c049165583', 'tag': None, 'wait_for_package': False} 2024-03-31T22:58:03.685 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=384f31f3d3aee7f8ba3f60846b8844c049165583 2024-03-31T22:58:03.687 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '384f31f3d3aee7f8ba3f60846b8844c049165583', 'tag': None, 'wait_for_package': False} 2024-03-31T22:58:03.687 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=384f31f3d3aee7f8ba3f60846b8844c049165583 2024-03-31T22:58:03.688 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '384f31f3d3aee7f8ba3f60846b8844c049165583', 'tag': None, 'wait_for_package': False} 2024-03-31T22:58:03.688 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=384f31f3d3aee7f8ba3f60846b8844c049165583 2024-03-31T22:58:03.834 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/squid/384f31f3d3aee7f8ba3f60846b8844c049165583/centos/9/flavors/default/ 2024-03-31T22:58:03.834 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1977.g384f31f3 2024-03-31T22:58:03.900 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/squid/384f31f3d3aee7f8ba3f60846b8844c049165583/centos/9/flavors/default/ 2024-03-31T22:58:03.900 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1977.g384f31f3 2024-03-31T22:58:03.911 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/squid/384f31f3d3aee7f8ba3f60846b8844c049165583/centos/9/flavors/default/ 2024-03-31T22:58:03.911 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1977.g384f31f3 2024-03-31T22:58:04.010 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/squid/384f31f3d3aee7f8ba3f60846b8844c049165583/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/squid/384f31f3d3aee7f8ba3f60846b8844c049165583/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/squid/384f31f3d3aee7f8ba3f60846b8844c049165583/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-31T22:58:04.010 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-03-31T22:58:04.010 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-31T22:58:04.039 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-03-31T22:58:04.040 DEBUG:teuthology.orchestra.run.smithi046:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/384f31f3d3aee7f8ba3f60846b8844c049165583/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-31T22:58:04.071 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/squid/384f31f3d3aee7f8ba3f60846b8844c049165583/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/squid/384f31f3d3aee7f8ba3f60846b8844c049165583/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/squid/384f31f3d3aee7f8ba3f60846b8844c049165583/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-31T22:58:04.071 DEBUG:teuthology.orchestra.run.smithi142:> set -ex 2024-03-31T22:58:04.072 DEBUG:teuthology.orchestra.run.smithi142:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-31T22:58:04.080 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/squid/384f31f3d3aee7f8ba3f60846b8844c049165583/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/squid/384f31f3d3aee7f8ba3f60846b8844c049165583/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/squid/384f31f3d3aee7f8ba3f60846b8844c049165583/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-31T22:58:04.080 DEBUG:teuthology.orchestra.run.smithi162:> set -ex 2024-03-31T22:58:04.080 DEBUG:teuthology.orchestra.run.smithi162:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-31T22:58:04.101 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-03-31T22:58:04.101 DEBUG:teuthology.orchestra.run.smithi142:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/384f31f3d3aee7f8ba3f60846b8844c049165583/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-31T22:58:04.109 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-03-31T22:58:04.109 DEBUG:teuthology.orchestra.run.smithi162:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/384f31f3d3aee7f8ba3f60846b8844c049165583/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-31T22:58:04.110 DEBUG:teuthology.orchestra.run.smithi046:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-31T22:58:04.170 DEBUG:teuthology.orchestra.run.smithi142:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-31T22:58:04.177 DEBUG:teuthology.orchestra.run.smithi162:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-31T22:58:04.193 DEBUG:teuthology.orchestra.run.smithi046:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-31T22:58:04.253 DEBUG:teuthology.orchestra.run.smithi142:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-31T22:58:04.257 DEBUG:teuthology.orchestra.run.smithi162:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-31T22:58:04.261 INFO:teuthology.orchestra.run.smithi046.stdout:check_obsoletes = 1 2024-03-31T22:58:04.263 DEBUG:teuthology.orchestra.run.smithi046:> sudo yum clean all 2024-03-31T22:58:04.320 INFO:teuthology.orchestra.run.smithi142.stdout:check_obsoletes = 1 2024-03-31T22:58:04.322 DEBUG:teuthology.orchestra.run.smithi142:> sudo yum clean all 2024-03-31T22:58:04.323 INFO:teuthology.orchestra.run.smithi162.stdout:check_obsoletes = 1 2024-03-31T22:58:04.325 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum clean all 2024-03-31T22:58:04.534 INFO:teuthology.orchestra.run.smithi046.stdout:74 files removed 2024-03-31T22:58:04.559 INFO:teuthology.orchestra.run.smithi142.stdout:74 files removed 2024-03-31T22:58:04.561 DEBUG:teuthology.orchestra.run.smithi046:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-03-31T22:58:04.568 INFO:teuthology.orchestra.run.smithi162.stdout:74 files removed 2024-03-31T22:58:04.586 DEBUG:teuthology.orchestra.run.smithi142:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-03-31T22:58:04.595 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-03-31T22:58:05.187 INFO:teuthology.orchestra.run.smithi046.stdout:ceph packages for x86_64 260 kB/s | 83 kB 00:00 2024-03-31T22:58:05.245 INFO:teuthology.orchestra.run.smithi162.stdout:ceph packages for x86_64 240 kB/s | 83 kB 00:00 2024-03-31T22:58:05.359 INFO:teuthology.orchestra.run.smithi142.stdout:ceph packages for x86_64 176 kB/s | 83 kB 00:00 2024-03-31T22:58:05.580 INFO:teuthology.orchestra.run.smithi046.stdout:ceph noarch packages 34 kB/s | 11 kB 00:00 2024-03-31T22:58:05.662 INFO:teuthology.orchestra.run.smithi162.stdout:ceph noarch packages 31 kB/s | 11 kB 00:00 2024-03-31T22:58:05.717 INFO:teuthology.orchestra.run.smithi142.stdout:ceph noarch packages 37 kB/s | 11 kB 00:00 2024-03-31T22:58:05.997 INFO:teuthology.orchestra.run.smithi046.stdout:ceph source packages 5.0 kB/s | 1.9 kB 00:00 2024-03-31T22:58:06.088 INFO:teuthology.orchestra.run.smithi162.stdout:ceph source packages 4.9 kB/s | 1.9 kB 00:00 2024-03-31T22:58:06.126 INFO:teuthology.orchestra.run.smithi142.stdout:ceph source packages 5.1 kB/s | 1.9 kB 00:00 2024-03-31T22:58:06.491 INFO:teuthology.orchestra.run.smithi046.stdout:CentOS-9 - AppStream 15 MB/s | 6.7 MB 00:00 2024-03-31T22:58:06.498 INFO:teuthology.orchestra.run.smithi162.stdout:CentOS-9 - AppStream 18 MB/s | 6.7 MB 00:00 2024-03-31T22:58:06.553 INFO:teuthology.orchestra.run.smithi142.stdout:CentOS-9 - AppStream 17 MB/s | 6.7 MB 00:00 2024-03-31T22:58:07.929 INFO:teuthology.orchestra.run.smithi142.stdout:CentOS-9 - BaseOS 7.6 MB/s | 2.0 MB 00:00 2024-03-31T22:58:07.941 INFO:teuthology.orchestra.run.smithi046.stdout:CentOS-9 - BaseOS 7.2 MB/s | 2.0 MB 00:00 2024-03-31T22:58:08.065 INFO:teuthology.orchestra.run.smithi162.stdout:CentOS-9 - BaseOS 4.6 MB/s | 2.0 MB 00:00 2024-03-31T22:58:08.757 INFO:teuthology.orchestra.run.smithi046.stdout:CentOS Stream 9 - BaseOS 17 MB/s | 8.0 MB 00:00 2024-03-31T22:58:08.797 INFO:teuthology.orchestra.run.smithi162.stdout:CentOS Stream 9 - BaseOS 20 MB/s | 8.0 MB 00:00 2024-03-31T22:58:08.953 INFO:teuthology.orchestra.run.smithi142.stdout:CentOS Stream 9 - BaseOS 11 MB/s | 8.0 MB 00:00 2024-03-31T22:58:10.458 INFO:teuthology.orchestra.run.smithi046.stdout:CentOS Stream 9 - AppStream 32 MB/s | 19 MB 00:00 2024-03-31T22:58:13.087 INFO:teuthology.orchestra.run.smithi142.stdout:CentOS Stream 9 - AppStream 6.2 MB/s | 19 MB 00:03 2024-03-31T22:58:14.761 INFO:teuthology.orchestra.run.smithi046.stdout:CentOS Stream 9 - CRB 9.8 MB/s | 6.1 MB 00:00 2024-03-31T22:58:16.610 INFO:teuthology.orchestra.run.smithi046.stdout:CentOS Stream 9 - Extras packages 24 kB/s | 16 kB 00:00 2024-03-31T22:58:17.112 INFO:teuthology.orchestra.run.smithi142.stdout:CentOS Stream 9 - CRB 12 MB/s | 6.1 MB 00:00 2024-03-31T22:58:17.697 INFO:teuthology.orchestra.run.smithi046.stdout:Extra Packages for Enterprise Linux 22 MB/s | 21 MB 00:00 2024-03-31T22:58:19.613 INFO:teuthology.orchestra.run.smithi142.stdout:CentOS Stream 9 - Extras packages 14 kB/s | 16 kB 00:01 2024-03-31T22:58:22.820 INFO:teuthology.orchestra.run.smithi046.stdout:lab-extras 29 kB/s | 1.7 kB 00:00 2024-03-31T22:58:22.933 INFO:teuthology.orchestra.run.smithi142.stdout:Extra Packages for Enterprise Linux 6.5 MB/s | 21 MB 00:03 2024-03-31T22:58:25.468 INFO:teuthology.orchestra.run.smithi046.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-31T22:58:25.469 INFO:teuthology.orchestra.run.smithi046.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-31T22:58:25.481 INFO:teuthology.orchestra.run.smithi046.stdout:Package NetworkManager-1:1.44.0-2.el9.x86_64 is already installed. 2024-03-31T22:58:25.482 INFO:teuthology.orchestra.run.smithi046.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-03-31T22:58:25.483 INFO:teuthology.orchestra.run.smithi046.stdout:Package util-linux-2.37.4-15.el9.x86_64 is already installed. 2024-03-31T22:58:25.484 INFO:teuthology.orchestra.run.smithi046.stdout:Package libacl-devel-2.3.1-3.el9.x86_64 is already installed. 2024-03-31T22:58:25.486 INFO:teuthology.orchestra.run.smithi046.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-03-31T22:58:25.488 INFO:teuthology.orchestra.run.smithi046.stdout:Package xfsdump-3.1.12-3.el9.x86_64 is already installed. 2024-03-31T22:58:25.489 INFO:teuthology.orchestra.run.smithi046.stdout:Package xfsprogs-5.19.0-4.el9.x86_64 is already installed. 2024-03-31T22:58:25.574 INFO:teuthology.orchestra.run.smithi046.stdout:Dependencies resolved. 2024-03-31T22:58:25.595 INFO:teuthology.orchestra.run.smithi046.stdout:============================================================================================ 2024-03-31T22:58:25.595 INFO:teuthology.orchestra.run.smithi046.stdout: Package Arch Version Repository Size 2024-03-31T22:58:25.595 INFO:teuthology.orchestra.run.smithi046.stdout:============================================================================================ 2024-03-31T22:58:25.595 INFO:teuthology.orchestra.run.smithi046.stdout:Installing: 2024-03-31T22:58:25.595 INFO:teuthology.orchestra.run.smithi046.stdout: bison x86_64 3.7.4-5.el9 CentOS-AppStream 920 k 2024-03-31T22:58:25.596 INFO:teuthology.orchestra.run.smithi046.stdout: ceph x86_64 2:19.0.0-1977.g384f31f3.el9 ceph 6.6 k 2024-03-31T22:58:25.596 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-base x86_64 2:19.0.0-1977.g384f31f3.el9 ceph 5.3 M 2024-03-31T22:58:25.596 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-fuse x86_64 2:19.0.0-1977.g384f31f3.el9 ceph 889 k 2024-03-31T22:58:25.597 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1977.g384f31f3.el9 ceph 145 k 2024-03-31T22:58:25.597 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-mgr x86_64 2:19.0.0-1977.g384f31f3.el9 ceph 1.5 M 2024-03-31T22:58:25.597 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1977.g384f31f3.el9 ceph-noarch 139 k 2024-03-31T22:58:25.598 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1977.g384f31f3.el9 ceph-noarch 3.5 M 2024-03-31T22:58:25.598 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1977.g384f31f3.el9 ceph-noarch 7.4 M 2024-03-31T22:58:25.598 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-mgr-rook noarch 2:19.0.0-1977.g384f31f3.el9 ceph-noarch 49 k 2024-03-31T22:58:25.598 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-radosgw x86_64 2:19.0.0-1977.g384f31f3.el9 ceph 9.8 M 2024-03-31T22:58:25.598 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-test x86_64 2:19.0.0-1977.g384f31f3.el9 ceph 47 M 2024-03-31T22:58:25.598 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-volume noarch 2:19.0.0-1977.g384f31f3.el9 ceph-noarch 264 k 2024-03-31T22:58:25.598 INFO:teuthology.orchestra.run.smithi046.stdout: cephadm noarch 2:19.0.0-1977.g384f31f3.el9 ceph-noarch 748 k 2024-03-31T22:58:25.598 INFO:teuthology.orchestra.run.smithi046.stdout: cephfs-mirror x86_64 2:19.0.0-1977.g384f31f3.el9 ceph 221 k 2024-03-31T22:58:25.598 INFO:teuthology.orchestra.run.smithi046.stdout: cephfs-shell x86_64 2:19.0.0-1977.g384f31f3.el9 ceph 27 k 2024-03-31T22:58:25.599 INFO:teuthology.orchestra.run.smithi046.stdout: cephfs-top noarch 2:19.0.0-1977.g384f31f3.el9 ceph-noarch 18 k 2024-03-31T22:58:25.599 INFO:teuthology.orchestra.run.smithi046.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-03-31T22:58:25.599 INFO:teuthology.orchestra.run.smithi046.stdout: flex x86_64 2.6.4-9.el9 CentOS-AppStream 307 k 2024-03-31T22:58:25.599 INFO:teuthology.orchestra.run.smithi046.stdout: libaio-devel x86_64 0.3.111-13.el9 CentOS-AppStream 10 k 2024-03-31T22:58:25.599 INFO:teuthology.orchestra.run.smithi046.stdout: libcephfs-devel x86_64 2:19.0.0-1977.g384f31f3.el9 ceph 32 k 2024-03-31T22:58:25.599 INFO:teuthology.orchestra.run.smithi046.stdout: libcephfs2 x86_64 2:19.0.0-1977.g384f31f3.el9 ceph 746 k 2024-03-31T22:58:25.599 INFO:teuthology.orchestra.run.smithi046.stdout: librados-devel x86_64 2:19.0.0-1977.g384f31f3.el9 ceph 127 k 2024-03-31T22:58:25.599 INFO:teuthology.orchestra.run.smithi046.stdout: libtool x86_64 2.4.6-45.el9 CentOS-AppStream 578 k 2024-03-31T22:58:25.599 INFO:teuthology.orchestra.run.smithi046.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-03-31T22:58:25.599 INFO:teuthology.orchestra.run.smithi046.stdout: openssl-devel x86_64 1:3.0.7-27.el9 appstream 4.1 M 2024-03-31T22:58:25.599 INFO:teuthology.orchestra.run.smithi046.stdout: postgresql x86_64 13.14-1.el9 CentOS-AppStream 1.5 M 2024-03-31T22:58:25.599 INFO:teuthology.orchestra.run.smithi046.stdout: postgresql-contrib x86_64 13.14-1.el9 CentOS-AppStream 817 k 2024-03-31T22:58:25.599 INFO:teuthology.orchestra.run.smithi046.stdout: postgresql-server x86_64 13.14-1.el9 CentOS-AppStream 5.7 M 2024-03-31T22:58:25.599 INFO:teuthology.orchestra.run.smithi046.stdout: python3-cephfs x86_64 2:19.0.0-1977.g384f31f3.el9 ceph 161 k 2024-03-31T22:58:25.600 INFO:teuthology.orchestra.run.smithi046.stdout: python3-rados x86_64 2:19.0.0-1977.g384f31f3.el9 ceph 320 k 2024-03-31T22:58:25.600 INFO:teuthology.orchestra.run.smithi046.stdout: python3-rbd x86_64 2:19.0.0-1977.g384f31f3.el9 ceph 298 k 2024-03-31T22:58:25.600 INFO:teuthology.orchestra.run.smithi046.stdout: python3-rgw x86_64 2:19.0.0-1977.g384f31f3.el9 ceph 99 k 2024-03-31T22:58:25.600 INFO:teuthology.orchestra.run.smithi046.stdout: rbd-fuse x86_64 2:19.0.0-1977.g384f31f3.el9 ceph 85 k 2024-03-31T22:58:25.600 INFO:teuthology.orchestra.run.smithi046.stdout: rbd-mirror x86_64 2:19.0.0-1977.g384f31f3.el9 ceph 3.1 M 2024-03-31T22:58:25.600 INFO:teuthology.orchestra.run.smithi046.stdout: rbd-nbd x86_64 2:19.0.0-1977.g384f31f3.el9 ceph 171 k 2024-03-31T22:58:25.600 INFO:teuthology.orchestra.run.smithi046.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-03-31T22:58:25.600 INFO:teuthology.orchestra.run.smithi046.stdout:Upgrading: 2024-03-31T22:58:25.600 INFO:teuthology.orchestra.run.smithi046.stdout: NetworkManager x86_64 1:1.47.2-1.el9 CentOS-BaseOS 2.3 M 2024-03-31T22:58:25.600 INFO:teuthology.orchestra.run.smithi046.stdout: NetworkManager-libnm x86_64 1:1.47.2-1.el9 CentOS-BaseOS 1.8 M 2024-03-31T22:58:25.600 INFO:teuthology.orchestra.run.smithi046.stdout: NetworkManager-team x86_64 1:1.47.2-1.el9 CentOS-BaseOS 41 k 2024-03-31T22:58:25.600 INFO:teuthology.orchestra.run.smithi046.stdout: NetworkManager-tui x86_64 1:1.47.2-1.el9 CentOS-BaseOS 247 k 2024-03-31T22:58:25.600 INFO:teuthology.orchestra.run.smithi046.stdout: acl x86_64 2.3.1-4.el9 baseos 71 k 2024-03-31T22:58:25.601 INFO:teuthology.orchestra.run.smithi046.stdout: elfutils-debuginfod-client x86_64 0.190-2.el9 baseos 36 k 2024-03-31T22:58:25.601 INFO:teuthology.orchestra.run.smithi046.stdout: elfutils-libelf x86_64 0.190-2.el9 baseos 192 k 2024-03-31T22:58:25.601 INFO:teuthology.orchestra.run.smithi046.stdout: elfutils-libs x86_64 0.190-2.el9 baseos 256 k 2024-03-31T22:58:25.601 INFO:teuthology.orchestra.run.smithi046.stdout: libacl x86_64 2.3.1-4.el9 baseos 23 k 2024-03-31T22:58:25.601 INFO:teuthology.orchestra.run.smithi046.stdout: libacl-devel x86_64 2.3.1-4.el9 appstream 74 k 2024-03-31T22:58:25.601 INFO:teuthology.orchestra.run.smithi046.stdout: libblkid x86_64 2.37.4-18.el9 baseos 108 k 2024-03-31T22:58:25.601 INFO:teuthology.orchestra.run.smithi046.stdout: libfdisk x86_64 2.37.4-18.el9 baseos 155 k 2024-03-31T22:58:25.601 INFO:teuthology.orchestra.run.smithi046.stdout: libmount x86_64 2.37.4-18.el9 baseos 136 k 2024-03-31T22:58:25.601 INFO:teuthology.orchestra.run.smithi046.stdout: librados2 x86_64 2:19.0.0-1977.g384f31f3.el9 ceph 3.4 M 2024-03-31T22:58:25.601 INFO:teuthology.orchestra.run.smithi046.stdout: librbd1 x86_64 2:19.0.0-1977.g384f31f3.el9 ceph 3.1 M 2024-03-31T22:58:25.601 INFO:teuthology.orchestra.run.smithi046.stdout: libsmartcols x86_64 2.37.4-18.el9 baseos 64 k 2024-03-31T22:58:25.601 INFO:teuthology.orchestra.run.smithi046.stdout: libuuid x86_64 2.37.4-18.el9 baseos 29 k 2024-03-31T22:58:25.601 INFO:teuthology.orchestra.run.smithi046.stdout: openssl x86_64 1:3.0.7-27.el9 baseos 1.2 M 2024-03-31T22:58:25.601 INFO:teuthology.orchestra.run.smithi046.stdout: openssl-libs x86_64 1:3.0.7-27.el9 baseos 2.2 M 2024-03-31T22:58:25.602 INFO:teuthology.orchestra.run.smithi046.stdout: util-linux x86_64 2.37.4-18.el9 baseos 2.3 M 2024-03-31T22:58:25.602 INFO:teuthology.orchestra.run.smithi046.stdout: util-linux-core x86_64 2.37.4-18.el9 baseos 465 k 2024-03-31T22:58:25.602 INFO:teuthology.orchestra.run.smithi046.stdout: util-linux-user x86_64 2.37.4-18.el9 baseos 31 k 2024-03-31T22:58:25.602 INFO:teuthology.orchestra.run.smithi046.stdout: xfsdump x86_64 3.1.12-4.el9 CentOS-BaseOS 321 k 2024-03-31T22:58:25.602 INFO:teuthology.orchestra.run.smithi046.stdout: xfsprogs x86_64 6.3.0-1.el9 baseos 1.1 M 2024-03-31T22:58:25.602 INFO:teuthology.orchestra.run.smithi046.stdout:Installing dependencies: 2024-03-31T22:58:25.602 INFO:teuthology.orchestra.run.smithi046.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-31T22:58:25.602 INFO:teuthology.orchestra.run.smithi046.stdout: automake noarch 1.16.2-8.el9 CentOS-AppStream 662 k 2024-03-31T22:58:25.602 INFO:teuthology.orchestra.run.smithi046.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-31T22:58:25.603 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-common x86_64 2:19.0.0-1977.g384f31f3.el9 ceph 21 M 2024-03-31T22:58:25.603 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1977.g384f31f3.el9 ceph-noarch 31 k 2024-03-31T22:58:25.603 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-mds x86_64 2:19.0.0-1977.g384f31f3.el9 ceph 2.4 M 2024-03-31T22:58:25.603 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1977.g384f31f3.el9 ceph-noarch 248 k 2024-03-31T22:58:25.603 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-mon x86_64 2:19.0.0-1977.g384f31f3.el9 ceph 4.7 M 2024-03-31T22:58:25.603 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-osd x86_64 2:19.0.0-1977.g384f31f3.el9 ceph 17 M 2024-03-31T22:58:25.603 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1977.g384f31f3.el9 ceph-noarch 15 k 2024-03-31T22:58:25.604 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-selinux x86_64 2:19.0.0-1977.g384f31f3.el9 ceph 25 k 2024-03-31T22:58:25.604 INFO:teuthology.orchestra.run.smithi046.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-31T22:58:25.604 INFO:teuthology.orchestra.run.smithi046.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-31T22:58:25.604 INFO:teuthology.orchestra.run.smithi046.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-31T22:58:25.604 INFO:teuthology.orchestra.run.smithi046.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-31T22:58:25.604 INFO:teuthology.orchestra.run.smithi046.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-31T22:58:25.604 INFO:teuthology.orchestra.run.smithi046.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-31T22:58:25.604 INFO:teuthology.orchestra.run.smithi046.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-31T22:58:25.604 INFO:teuthology.orchestra.run.smithi046.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-31T22:58:25.604 INFO:teuthology.orchestra.run.smithi046.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-31T22:58:25.605 INFO:teuthology.orchestra.run.smithi046.stdout: libcephsqlite x86_64 2:19.0.0-1977.g384f31f3.el9 ceph 163 k 2024-03-31T22:58:25.605 INFO:teuthology.orchestra.run.smithi046.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-31T22:58:25.605 INFO:teuthology.orchestra.run.smithi046.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-31T22:58:25.605 INFO:teuthology.orchestra.run.smithi046.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-31T22:58:25.605 INFO:teuthology.orchestra.run.smithi046.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-31T22:58:25.605 INFO:teuthology.orchestra.run.smithi046.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-31T22:58:25.606 INFO:teuthology.orchestra.run.smithi046.stdout: libradosstriper1 x86_64 2:19.0.0-1977.g384f31f3.el9 ceph 501 k 2024-03-31T22:58:25.606 INFO:teuthology.orchestra.run.smithi046.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-31T22:58:25.606 INFO:teuthology.orchestra.run.smithi046.stdout: librgw2 x86_64 2:19.0.0-1977.g384f31f3.el9 ceph 5.1 M 2024-03-31T22:58:25.606 INFO:teuthology.orchestra.run.smithi046.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-31T22:58:25.606 INFO:teuthology.orchestra.run.smithi046.stdout: libzstd-devel x86_64 1.5.1-2.el9 CentOS-AppStream 46 k 2024-03-31T22:58:25.606 INFO:teuthology.orchestra.run.smithi046.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-31T22:58:25.606 INFO:teuthology.orchestra.run.smithi046.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-31T22:58:25.606 INFO:teuthology.orchestra.run.smithi046.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-31T22:58:25.606 INFO:teuthology.orchestra.run.smithi046.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-31T22:58:25.606 INFO:teuthology.orchestra.run.smithi046.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-31T22:58:25.606 INFO:teuthology.orchestra.run.smithi046.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-31T22:58:25.606 INFO:teuthology.orchestra.run.smithi046.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-31T22:58:25.607 INFO:teuthology.orchestra.run.smithi046.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-31T22:58:25.607 INFO:teuthology.orchestra.run.smithi046.stdout: postgresql-private-libs x86_64 13.14-1.el9 CentOS-AppStream 132 k 2024-03-31T22:58:25.607 INFO:teuthology.orchestra.run.smithi046.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-31T22:58:25.608 INFO:teuthology.orchestra.run.smithi046.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-31T22:58:25.608 INFO:teuthology.orchestra.run.smithi046.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-31T22:58:25.608 INFO:teuthology.orchestra.run.smithi046.stdout: python3-attrs noarch 20.3.0-7.el9 CentOS-AppStream 78 k 2024-03-31T22:58:25.608 INFO:teuthology.orchestra.run.smithi046.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-31T22:58:25.608 INFO:teuthology.orchestra.run.smithi046.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-31T22:58:25.608 INFO:teuthology.orchestra.run.smithi046.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-31T22:58:25.610 INFO:teuthology.orchestra.run.smithi046.stdout: python3-ceph-argparse x86_64 2:19.0.0-1977.g384f31f3.el9 ceph 45 k 2024-03-31T22:58:25.610 INFO:teuthology.orchestra.run.smithi046.stdout: python3-ceph-common x86_64 2:19.0.0-1977.g384f31f3.el9 ceph 131 k 2024-03-31T22:58:25.610 INFO:teuthology.orchestra.run.smithi046.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-31T22:58:25.610 INFO:teuthology.orchestra.run.smithi046.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-31T22:58:25.611 INFO:teuthology.orchestra.run.smithi046.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-31T22:58:25.611 INFO:teuthology.orchestra.run.smithi046.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-31T22:58:25.611 INFO:teuthology.orchestra.run.smithi046.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-03-31T22:58:25.611 INFO:teuthology.orchestra.run.smithi046.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-03-31T22:58:25.611 INFO:teuthology.orchestra.run.smithi046.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-31T22:58:25.611 INFO:teuthology.orchestra.run.smithi046.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-31T22:58:25.611 INFO:teuthology.orchestra.run.smithi046.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-31T22:58:25.612 INFO:teuthology.orchestra.run.smithi046.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-31T22:58:25.612 INFO:teuthology.orchestra.run.smithi046.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-31T22:58:25.612 INFO:teuthology.orchestra.run.smithi046.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-31T22:58:25.612 INFO:teuthology.orchestra.run.smithi046.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-31T22:58:25.612 INFO:teuthology.orchestra.run.smithi046.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-31T22:58:25.612 INFO:teuthology.orchestra.run.smithi046.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-31T22:58:25.613 INFO:teuthology.orchestra.run.smithi046.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-31T22:58:25.613 INFO:teuthology.orchestra.run.smithi046.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-31T22:58:25.613 INFO:teuthology.orchestra.run.smithi046.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-31T22:58:25.613 INFO:teuthology.orchestra.run.smithi046.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-31T22:58:25.613 INFO:teuthology.orchestra.run.smithi046.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-31T22:58:25.613 INFO:teuthology.orchestra.run.smithi046.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-31T22:58:25.613 INFO:teuthology.orchestra.run.smithi046.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-31T22:58:25.614 INFO:teuthology.orchestra.run.smithi046.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-31T22:58:25.614 INFO:teuthology.orchestra.run.smithi046.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-31T22:58:25.614 INFO:teuthology.orchestra.run.smithi046.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-31T22:58:25.614 INFO:teuthology.orchestra.run.smithi046.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-31T22:58:25.614 INFO:teuthology.orchestra.run.smithi046.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-31T22:58:25.615 INFO:teuthology.orchestra.run.smithi046.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-31T22:58:25.615 INFO:teuthology.orchestra.run.smithi046.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-31T22:58:25.615 INFO:teuthology.orchestra.run.smithi046.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-31T22:58:25.615 INFO:teuthology.orchestra.run.smithi046.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-31T22:58:25.615 INFO:teuthology.orchestra.run.smithi046.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-31T22:58:25.615 INFO:teuthology.orchestra.run.smithi046.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-31T22:58:25.616 INFO:teuthology.orchestra.run.smithi046.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-31T22:58:25.616 INFO:teuthology.orchestra.run.smithi046.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-31T22:58:25.616 INFO:teuthology.orchestra.run.smithi046.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-31T22:58:25.616 INFO:teuthology.orchestra.run.smithi046.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-31T22:58:25.617 INFO:teuthology.orchestra.run.smithi046.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-31T22:58:25.617 INFO:teuthology.orchestra.run.smithi046.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-03-31T22:58:25.617 INFO:teuthology.orchestra.run.smithi046.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-31T22:58:25.617 INFO:teuthology.orchestra.run.smithi046.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-31T22:58:25.617 INFO:teuthology.orchestra.run.smithi046.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-31T22:58:25.617 INFO:teuthology.orchestra.run.smithi046.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-31T22:58:25.617 INFO:teuthology.orchestra.run.smithi046.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-31T22:58:25.618 INFO:teuthology.orchestra.run.smithi046.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-31T22:58:25.618 INFO:teuthology.orchestra.run.smithi046.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-31T22:58:25.618 INFO:teuthology.orchestra.run.smithi046.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-31T22:58:25.618 INFO:teuthology.orchestra.run.smithi046.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-31T22:58:25.618 INFO:teuthology.orchestra.run.smithi046.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-31T22:58:25.618 INFO:teuthology.orchestra.run.smithi046.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-31T22:58:25.619 INFO:teuthology.orchestra.run.smithi046.stdout: python3-wcwidth noarch 0.2.5-8.el9 CentOS-AppStream 41 k 2024-03-31T22:58:25.619 INFO:teuthology.orchestra.run.smithi046.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-31T22:58:25.619 INFO:teuthology.orchestra.run.smithi046.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-31T22:58:25.619 INFO:teuthology.orchestra.run.smithi046.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-31T22:58:25.619 INFO:teuthology.orchestra.run.smithi046.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-31T22:58:25.619 INFO:teuthology.orchestra.run.smithi046.stdout: qatlib x86_64 24.02.0-1.el9 CentOS-AppStream 220 k 2024-03-31T22:58:25.620 INFO:teuthology.orchestra.run.smithi046.stdout: qatzip-libs x86_64 1.2.0-1.el9 CentOS-AppStream 46 k 2024-03-31T22:58:25.620 INFO:teuthology.orchestra.run.smithi046.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-31T22:58:25.620 INFO:teuthology.orchestra.run.smithi046.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-31T22:58:25.620 INFO:teuthology.orchestra.run.smithi046.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-31T22:58:25.620 INFO:teuthology.orchestra.run.smithi046.stdout: uuid x86_64 1.6.2-55.el9 CentOS-AppStream 56 k 2024-03-31T22:58:25.620 INFO:teuthology.orchestra.run.smithi046.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-31T22:58:25.620 INFO:teuthology.orchestra.run.smithi046.stdout: zlib-devel x86_64 1.2.11-40.el9 CentOS-AppStream 44 k 2024-03-31T22:58:25.621 INFO:teuthology.orchestra.run.smithi046.stdout:Installing weak dependencies: 2024-03-31T22:58:25.621 INFO:teuthology.orchestra.run.smithi046.stdout: qatlib-service x86_64 24.02.0-1.el9 CentOS-AppStream 35 k 2024-03-31T22:58:25.621 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-31T22:58:25.621 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction Summary 2024-03-31T22:58:25.621 INFO:teuthology.orchestra.run.smithi046.stdout:============================================================================================ 2024-03-31T22:58:25.622 INFO:teuthology.orchestra.run.smithi046.stdout:Install 150 Packages 2024-03-31T22:58:25.622 INFO:teuthology.orchestra.run.smithi046.stdout:Upgrade 24 Packages 2024-03-31T22:58:25.622 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-31T22:58:25.623 INFO:teuthology.orchestra.run.smithi046.stdout:Total download size: 226 M 2024-03-31T22:58:25.624 INFO:teuthology.orchestra.run.smithi046.stdout:Downloading Packages: 2024-03-31T22:58:26.113 INFO:teuthology.orchestra.run.smithi046.stdout:(1/174): ceph-19.0.0-1977.g384f31f3.el9.x86_64. 31 kB/s | 6.6 kB 00:00 2024-03-31T22:58:26.404 INFO:teuthology.orchestra.run.smithi046.stdout:(2/174): ceph-fuse-19.0.0-1977.g384f31f3.el9.x8 3.0 MB/s | 889 kB 00:00 2024-03-31T22:58:26.496 INFO:teuthology.orchestra.run.smithi046.stdout:(3/174): ceph-immutable-object-cache-19.0.0-197 1.5 MB/s | 145 kB 00:00 2024-03-31T22:58:26.772 INFO:teuthology.orchestra.run.smithi046.stdout:(4/174): ceph-base-19.0.0-1977.g384f31f3.el9.x8 6.1 MB/s | 5.3 MB 00:00 2024-03-31T22:58:26.822 INFO:teuthology.orchestra.run.smithi046.stdout:(5/174): ceph-mds-19.0.0-1977.g384f31f3.el9.x86 7.2 MB/s | 2.4 MB 00:00 2024-03-31T22:58:26.972 INFO:teuthology.orchestra.run.smithi046.stdout:(6/174): ceph-mgr-19.0.0-1977.g384f31f3.el9.x86 7.5 MB/s | 1.5 MB 00:00 2024-03-31T22:58:27.448 INFO:teuthology.orchestra.run.smithi046.stdout:(7/174): ceph-mon-19.0.0-1977.g384f31f3.el9.x86 7.5 MB/s | 4.7 MB 00:00 2024-03-31T22:58:27.922 INFO:teuthology.orchestra.run.smithi142.stdout:lab-extras 31 kB/s | 1.7 kB 00:00 2024-03-31T22:58:28.264 INFO:teuthology.orchestra.run.smithi046.stdout:(8/174): ceph-common-19.0.0-1977.g384f31f3.el9. 8.7 MB/s | 21 MB 00:02 2024-03-31T22:58:28.348 INFO:teuthology.orchestra.run.smithi046.stdout:(9/174): ceph-selinux-19.0.0-1977.g384f31f3.el9 303 kB/s | 25 kB 00:00 2024-03-31T22:58:28.657 INFO:teuthology.orchestra.run.smithi046.stdout:(10/174): ceph-radosgw-19.0.0-1977.g384f31f3.el 8.1 MB/s | 9.8 MB 00:01 2024-03-31T22:58:28.815 INFO:teuthology.orchestra.run.smithi046.stdout:(11/174): ceph-osd-19.0.0-1977.g384f31f3.el9.x8 9.1 MB/s | 17 MB 00:01 2024-03-31T22:58:28.840 INFO:teuthology.orchestra.run.smithi046.stdout:(12/174): cephfs-mirror-19.0.0-1977.g384f31f3.e 1.2 MB/s | 221 kB 00:00 2024-03-31T22:58:28.874 INFO:teuthology.orchestra.run.smithi046.stdout:(13/174): cephfs-shell-19.0.0-1977.g384f31f3.el 464 kB/s | 27 kB 00:00 2024-03-31T22:58:28.899 INFO:teuthology.orchestra.run.smithi046.stdout:(14/174): libcephfs-devel-19.0.0-1977.g384f31f3 541 kB/s | 32 kB 00:00 2024-03-31T22:58:28.974 INFO:teuthology.orchestra.run.smithi046.stdout:(15/174): libcephsqlite-19.0.0-1977.g384f31f3.e 2.1 MB/s | 163 kB 00:00 2024-03-31T22:58:29.008 INFO:teuthology.orchestra.run.smithi046.stdout:(16/174): libcephfs2-19.0.0-1977.g384f31f3.el9. 5.4 MB/s | 746 kB 00:00 2024-03-31T22:58:29.042 INFO:teuthology.orchestra.run.smithi046.stdout:(17/174): librados-devel-19.0.0-1977.g384f31f3. 1.8 MB/s | 127 kB 00:00 2024-03-31T22:58:29.100 INFO:teuthology.orchestra.run.smithi046.stdout:(18/174): libradosstriper1-19.0.0-1977.g384f31f 5.3 MB/s | 501 kB 00:00 2024-03-31T22:58:29.175 INFO:teuthology.orchestra.run.smithi046.stdout:(19/174): python3-ceph-argparse-19.0.0-1977.g38 600 kB/s | 45 kB 00:00 2024-03-31T22:58:29.259 INFO:teuthology.orchestra.run.smithi046.stdout:(20/174): python3-ceph-common-19.0.0-1977.g384f 1.5 MB/s | 131 kB 00:00 2024-03-31T22:58:29.343 INFO:teuthology.orchestra.run.smithi046.stdout:(21/174): python3-cephfs-19.0.0-1977.g384f31f3. 1.9 MB/s | 161 kB 00:00 2024-03-31T22:58:29.435 INFO:teuthology.orchestra.run.smithi046.stdout:(22/174): python3-rados-19.0.0-1977.g384f31f3.e 3.4 MB/s | 320 kB 00:00 2024-03-31T22:58:29.535 INFO:teuthology.orchestra.run.smithi046.stdout:(23/174): python3-rbd-19.0.0-1977.g384f31f3.el9 2.9 MB/s | 298 kB 00:00 2024-03-31T22:58:29.610 INFO:teuthology.orchestra.run.smithi046.stdout:(24/174): librgw2-19.0.0-1977.g384f31f3.el9.x86 9.0 MB/s | 5.1 MB 00:00 2024-03-31T22:58:29.636 INFO:teuthology.orchestra.run.smithi046.stdout:(25/174): python3-rgw-19.0.0-1977.g384f31f3.el9 990 kB/s | 99 kB 00:00 2024-03-31T22:58:29.694 INFO:teuthology.orchestra.run.smithi046.stdout:(26/174): rbd-fuse-19.0.0-1977.g384f31f3.el9.x8 1.0 MB/s | 85 kB 00:00 2024-03-31T22:58:29.786 INFO:teuthology.orchestra.run.smithi046.stdout:(27/174): rbd-nbd-19.0.0-1977.g384f31f3.el9.x86 1.8 MB/s | 171 kB 00:00 2024-03-31T22:58:29.878 INFO:teuthology.orchestra.run.smithi046.stdout:(28/174): ceph-grafana-dashboards-19.0.0-1977.g 344 kB/s | 31 kB 00:00 2024-03-31T22:58:29.937 INFO:teuthology.orchestra.run.smithi046.stdout:(29/174): rbd-mirror-19.0.0-1977.g384f31f3.el9. 10 MB/s | 3.1 MB 00:00 2024-03-31T22:58:29.962 INFO:teuthology.orchestra.run.smithi046.stdout:(30/174): ceph-mgr-cephadm-19.0.0-1977.g384f31f 1.6 MB/s | 139 kB 00:00 2024-03-31T22:58:30.379 INFO:teuthology.orchestra.run.smithi046.stdout:(31/174): ceph-mgr-dashboard-19.0.0-1977.g384f3 7.9 MB/s | 3.5 MB 00:00 2024-03-31T22:58:30.452 INFO:teuthology.orchestra.run.smithi142.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-31T22:58:30.453 INFO:teuthology.orchestra.run.smithi142.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-31T22:58:30.467 INFO:teuthology.orchestra.run.smithi142.stdout:Package NetworkManager-1:1.44.0-2.el9.x86_64 is already installed. 2024-03-31T22:58:30.468 INFO:teuthology.orchestra.run.smithi142.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-03-31T22:58:30.469 INFO:teuthology.orchestra.run.smithi142.stdout:Package util-linux-2.37.4-15.el9.x86_64 is already installed. 2024-03-31T22:58:30.470 INFO:teuthology.orchestra.run.smithi142.stdout:Package libacl-devel-2.3.1-3.el9.x86_64 is already installed. 2024-03-31T22:58:30.472 INFO:teuthology.orchestra.run.smithi142.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-03-31T22:58:30.475 INFO:teuthology.orchestra.run.smithi142.stdout:Package xfsdump-3.1.12-3.el9.x86_64 is already installed. 2024-03-31T22:58:30.476 INFO:teuthology.orchestra.run.smithi142.stdout:Package xfsprogs-5.19.0-4.el9.x86_64 is already installed. 2024-03-31T22:58:30.479 INFO:teuthology.orchestra.run.smithi046.stdout:(32/174): ceph-mgr-modules-core-19.0.0-1977.g38 2.4 MB/s | 248 kB 00:00 2024-03-31T22:58:30.556 INFO:teuthology.orchestra.run.smithi142.stdout:Dependencies resolved. 2024-03-31T22:58:30.571 INFO:teuthology.orchestra.run.smithi142.stdout:============================================================================================ 2024-03-31T22:58:30.571 INFO:teuthology.orchestra.run.smithi142.stdout: Package Arch Version Repository Size 2024-03-31T22:58:30.571 INFO:teuthology.orchestra.run.smithi142.stdout:============================================================================================ 2024-03-31T22:58:30.571 INFO:teuthology.orchestra.run.smithi142.stdout:Installing: 2024-03-31T22:58:30.571 INFO:teuthology.orchestra.run.smithi142.stdout: bison x86_64 3.7.4-5.el9 CentOS-AppStream 920 k 2024-03-31T22:58:30.571 INFO:teuthology.orchestra.run.smithi142.stdout: ceph x86_64 2:19.0.0-1977.g384f31f3.el9 ceph 6.6 k 2024-03-31T22:58:30.572 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-base x86_64 2:19.0.0-1977.g384f31f3.el9 ceph 5.3 M 2024-03-31T22:58:30.572 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-fuse x86_64 2:19.0.0-1977.g384f31f3.el9 ceph 889 k 2024-03-31T22:58:30.572 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1977.g384f31f3.el9 ceph 145 k 2024-03-31T22:58:30.572 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-mgr x86_64 2:19.0.0-1977.g384f31f3.el9 ceph 1.5 M 2024-03-31T22:58:30.572 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1977.g384f31f3.el9 ceph-noarch 139 k 2024-03-31T22:58:30.572 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1977.g384f31f3.el9 ceph-noarch 3.5 M 2024-03-31T22:58:30.572 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1977.g384f31f3.el9 ceph-noarch 7.4 M 2024-03-31T22:58:30.572 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-mgr-rook noarch 2:19.0.0-1977.g384f31f3.el9 ceph-noarch 49 k 2024-03-31T22:58:30.572 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-radosgw x86_64 2:19.0.0-1977.g384f31f3.el9 ceph 9.8 M 2024-03-31T22:58:30.572 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-test x86_64 2:19.0.0-1977.g384f31f3.el9 ceph 47 M 2024-03-31T22:58:30.572 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-volume noarch 2:19.0.0-1977.g384f31f3.el9 ceph-noarch 264 k 2024-03-31T22:58:30.572 INFO:teuthology.orchestra.run.smithi142.stdout: cephadm noarch 2:19.0.0-1977.g384f31f3.el9 ceph-noarch 748 k 2024-03-31T22:58:30.572 INFO:teuthology.orchestra.run.smithi142.stdout: cephfs-mirror x86_64 2:19.0.0-1977.g384f31f3.el9 ceph 221 k 2024-03-31T22:58:30.573 INFO:teuthology.orchestra.run.smithi142.stdout: cephfs-shell x86_64 2:19.0.0-1977.g384f31f3.el9 ceph 27 k 2024-03-31T22:58:30.573 INFO:teuthology.orchestra.run.smithi142.stdout: cephfs-top noarch 2:19.0.0-1977.g384f31f3.el9 ceph-noarch 18 k 2024-03-31T22:58:30.573 INFO:teuthology.orchestra.run.smithi142.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-03-31T22:58:30.573 INFO:teuthology.orchestra.run.smithi142.stdout: flex x86_64 2.6.4-9.el9 CentOS-AppStream 307 k 2024-03-31T22:58:30.573 INFO:teuthology.orchestra.run.smithi142.stdout: libaio-devel x86_64 0.3.111-13.el9 CentOS-AppStream 10 k 2024-03-31T22:58:30.573 INFO:teuthology.orchestra.run.smithi142.stdout: libcephfs-devel x86_64 2:19.0.0-1977.g384f31f3.el9 ceph 32 k 2024-03-31T22:58:30.573 INFO:teuthology.orchestra.run.smithi142.stdout: libcephfs2 x86_64 2:19.0.0-1977.g384f31f3.el9 ceph 746 k 2024-03-31T22:58:30.573 INFO:teuthology.orchestra.run.smithi142.stdout: librados-devel x86_64 2:19.0.0-1977.g384f31f3.el9 ceph 127 k 2024-03-31T22:58:30.573 INFO:teuthology.orchestra.run.smithi142.stdout: libtool x86_64 2.4.6-45.el9 CentOS-AppStream 578 k 2024-03-31T22:58:30.573 INFO:teuthology.orchestra.run.smithi142.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-03-31T22:58:30.573 INFO:teuthology.orchestra.run.smithi142.stdout: openssl-devel x86_64 1:3.0.7-27.el9 appstream 4.1 M 2024-03-31T22:58:30.573 INFO:teuthology.orchestra.run.smithi142.stdout: postgresql x86_64 13.14-1.el9 CentOS-AppStream 1.5 M 2024-03-31T22:58:30.573 INFO:teuthology.orchestra.run.smithi142.stdout: postgresql-contrib x86_64 13.14-1.el9 CentOS-AppStream 817 k 2024-03-31T22:58:30.573 INFO:teuthology.orchestra.run.smithi142.stdout: postgresql-server x86_64 13.14-1.el9 CentOS-AppStream 5.7 M 2024-03-31T22:58:30.574 INFO:teuthology.orchestra.run.smithi142.stdout: python3-cephfs x86_64 2:19.0.0-1977.g384f31f3.el9 ceph 161 k 2024-03-31T22:58:30.574 INFO:teuthology.orchestra.run.smithi142.stdout: python3-rados x86_64 2:19.0.0-1977.g384f31f3.el9 ceph 320 k 2024-03-31T22:58:30.574 INFO:teuthology.orchestra.run.smithi142.stdout: python3-rbd x86_64 2:19.0.0-1977.g384f31f3.el9 ceph 298 k 2024-03-31T22:58:30.574 INFO:teuthology.orchestra.run.smithi142.stdout: python3-rgw x86_64 2:19.0.0-1977.g384f31f3.el9 ceph 99 k 2024-03-31T22:58:30.574 INFO:teuthology.orchestra.run.smithi142.stdout: rbd-fuse x86_64 2:19.0.0-1977.g384f31f3.el9 ceph 85 k 2024-03-31T22:58:30.574 INFO:teuthology.orchestra.run.smithi142.stdout: rbd-mirror x86_64 2:19.0.0-1977.g384f31f3.el9 ceph 3.1 M 2024-03-31T22:58:30.574 INFO:teuthology.orchestra.run.smithi142.stdout: rbd-nbd x86_64 2:19.0.0-1977.g384f31f3.el9 ceph 171 k 2024-03-31T22:58:30.574 INFO:teuthology.orchestra.run.smithi142.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-03-31T22:58:30.575 INFO:teuthology.orchestra.run.smithi142.stdout:Upgrading: 2024-03-31T22:58:30.575 INFO:teuthology.orchestra.run.smithi142.stdout: NetworkManager x86_64 1:1.47.2-1.el9 CentOS-BaseOS 2.3 M 2024-03-31T22:58:30.575 INFO:teuthology.orchestra.run.smithi142.stdout: NetworkManager-libnm x86_64 1:1.47.2-1.el9 CentOS-BaseOS 1.8 M 2024-03-31T22:58:30.575 INFO:teuthology.orchestra.run.smithi142.stdout: NetworkManager-team x86_64 1:1.47.2-1.el9 CentOS-BaseOS 41 k 2024-03-31T22:58:30.575 INFO:teuthology.orchestra.run.smithi142.stdout: NetworkManager-tui x86_64 1:1.47.2-1.el9 CentOS-BaseOS 247 k 2024-03-31T22:58:30.575 INFO:teuthology.orchestra.run.smithi142.stdout: acl x86_64 2.3.1-4.el9 baseos 71 k 2024-03-31T22:58:30.575 INFO:teuthology.orchestra.run.smithi142.stdout: elfutils-debuginfod-client x86_64 0.190-2.el9 baseos 36 k 2024-03-31T22:58:30.576 INFO:teuthology.orchestra.run.smithi142.stdout: elfutils-libelf x86_64 0.190-2.el9 baseos 192 k 2024-03-31T22:58:30.576 INFO:teuthology.orchestra.run.smithi142.stdout: elfutils-libs x86_64 0.190-2.el9 baseos 256 k 2024-03-31T22:58:30.576 INFO:teuthology.orchestra.run.smithi142.stdout: libacl x86_64 2.3.1-4.el9 baseos 23 k 2024-03-31T22:58:30.576 INFO:teuthology.orchestra.run.smithi142.stdout: libacl-devel x86_64 2.3.1-4.el9 appstream 74 k 2024-03-31T22:58:30.576 INFO:teuthology.orchestra.run.smithi142.stdout: libblkid x86_64 2.37.4-18.el9 baseos 108 k 2024-03-31T22:58:30.576 INFO:teuthology.orchestra.run.smithi142.stdout: libfdisk x86_64 2.37.4-18.el9 baseos 155 k 2024-03-31T22:58:30.576 INFO:teuthology.orchestra.run.smithi142.stdout: libmount x86_64 2.37.4-18.el9 baseos 136 k 2024-03-31T22:58:30.576 INFO:teuthology.orchestra.run.smithi142.stdout: librados2 x86_64 2:19.0.0-1977.g384f31f3.el9 ceph 3.4 M 2024-03-31T22:58:30.576 INFO:teuthology.orchestra.run.smithi142.stdout: librbd1 x86_64 2:19.0.0-1977.g384f31f3.el9 ceph 3.1 M 2024-03-31T22:58:30.576 INFO:teuthology.orchestra.run.smithi142.stdout: libsmartcols x86_64 2.37.4-18.el9 baseos 64 k 2024-03-31T22:58:30.576 INFO:teuthology.orchestra.run.smithi142.stdout: libuuid x86_64 2.37.4-18.el9 baseos 29 k 2024-03-31T22:58:30.576 INFO:teuthology.orchestra.run.smithi142.stdout: openssl x86_64 1:3.0.7-27.el9 baseos 1.2 M 2024-03-31T22:58:30.577 INFO:teuthology.orchestra.run.smithi142.stdout: openssl-libs x86_64 1:3.0.7-27.el9 baseos 2.2 M 2024-03-31T22:58:30.577 INFO:teuthology.orchestra.run.smithi142.stdout: util-linux x86_64 2.37.4-18.el9 baseos 2.3 M 2024-03-31T22:58:30.577 INFO:teuthology.orchestra.run.smithi142.stdout: util-linux-core x86_64 2.37.4-18.el9 baseos 465 k 2024-03-31T22:58:30.577 INFO:teuthology.orchestra.run.smithi142.stdout: util-linux-user x86_64 2.37.4-18.el9 baseos 31 k 2024-03-31T22:58:30.577 INFO:teuthology.orchestra.run.smithi142.stdout: xfsdump x86_64 3.1.12-4.el9 CentOS-BaseOS 321 k 2024-03-31T22:58:30.577 INFO:teuthology.orchestra.run.smithi142.stdout: xfsprogs x86_64 6.3.0-1.el9 baseos 1.1 M 2024-03-31T22:58:30.577 INFO:teuthology.orchestra.run.smithi142.stdout:Installing dependencies: 2024-03-31T22:58:30.577 INFO:teuthology.orchestra.run.smithi142.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-31T22:58:30.578 INFO:teuthology.orchestra.run.smithi142.stdout: automake noarch 1.16.2-8.el9 CentOS-AppStream 662 k 2024-03-31T22:58:30.578 INFO:teuthology.orchestra.run.smithi142.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-31T22:58:30.578 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-common x86_64 2:19.0.0-1977.g384f31f3.el9 ceph 21 M 2024-03-31T22:58:30.578 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1977.g384f31f3.el9 ceph-noarch 31 k 2024-03-31T22:58:30.578 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-mds x86_64 2:19.0.0-1977.g384f31f3.el9 ceph 2.4 M 2024-03-31T22:58:30.578 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1977.g384f31f3.el9 ceph-noarch 248 k 2024-03-31T22:58:30.578 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-mon x86_64 2:19.0.0-1977.g384f31f3.el9 ceph 4.7 M 2024-03-31T22:58:30.578 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-osd x86_64 2:19.0.0-1977.g384f31f3.el9 ceph 17 M 2024-03-31T22:58:30.578 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1977.g384f31f3.el9 ceph-noarch 15 k 2024-03-31T22:58:30.578 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-selinux x86_64 2:19.0.0-1977.g384f31f3.el9 ceph 25 k 2024-03-31T22:58:30.578 INFO:teuthology.orchestra.run.smithi142.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-31T22:58:30.578 INFO:teuthology.orchestra.run.smithi142.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-31T22:58:30.579 INFO:teuthology.orchestra.run.smithi142.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-31T22:58:30.579 INFO:teuthology.orchestra.run.smithi142.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-31T22:58:30.579 INFO:teuthology.orchestra.run.smithi142.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-31T22:58:30.579 INFO:teuthology.orchestra.run.smithi142.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-31T22:58:30.579 INFO:teuthology.orchestra.run.smithi142.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-31T22:58:30.579 INFO:teuthology.orchestra.run.smithi142.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-31T22:58:30.579 INFO:teuthology.orchestra.run.smithi142.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-31T22:58:30.579 INFO:teuthology.orchestra.run.smithi142.stdout: libcephsqlite x86_64 2:19.0.0-1977.g384f31f3.el9 ceph 163 k 2024-03-31T22:58:30.579 INFO:teuthology.orchestra.run.smithi142.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-31T22:58:30.579 INFO:teuthology.orchestra.run.smithi142.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-31T22:58:30.579 INFO:teuthology.orchestra.run.smithi142.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-31T22:58:30.579 INFO:teuthology.orchestra.run.smithi142.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-31T22:58:30.580 INFO:teuthology.orchestra.run.smithi142.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-31T22:58:30.580 INFO:teuthology.orchestra.run.smithi142.stdout: libradosstriper1 x86_64 2:19.0.0-1977.g384f31f3.el9 ceph 501 k 2024-03-31T22:58:30.580 INFO:teuthology.orchestra.run.smithi142.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-31T22:58:30.580 INFO:teuthology.orchestra.run.smithi142.stdout: librgw2 x86_64 2:19.0.0-1977.g384f31f3.el9 ceph 5.1 M 2024-03-31T22:58:30.580 INFO:teuthology.orchestra.run.smithi142.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-31T22:58:30.580 INFO:teuthology.orchestra.run.smithi142.stdout: libzstd-devel x86_64 1.5.1-2.el9 CentOS-AppStream 46 k 2024-03-31T22:58:30.580 INFO:teuthology.orchestra.run.smithi142.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-31T22:58:30.580 INFO:teuthology.orchestra.run.smithi142.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-31T22:58:30.580 INFO:teuthology.orchestra.run.smithi142.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-31T22:58:30.580 INFO:teuthology.orchestra.run.smithi142.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-31T22:58:30.580 INFO:teuthology.orchestra.run.smithi142.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-31T22:58:30.580 INFO:teuthology.orchestra.run.smithi142.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-31T22:58:30.580 INFO:teuthology.orchestra.run.smithi142.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-31T22:58:30.581 INFO:teuthology.orchestra.run.smithi142.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-31T22:58:30.581 INFO:teuthology.orchestra.run.smithi142.stdout: postgresql-private-libs x86_64 13.14-1.el9 CentOS-AppStream 132 k 2024-03-31T22:58:30.581 INFO:teuthology.orchestra.run.smithi142.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-31T22:58:30.581 INFO:teuthology.orchestra.run.smithi142.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-31T22:58:30.581 INFO:teuthology.orchestra.run.smithi142.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-31T22:58:30.581 INFO:teuthology.orchestra.run.smithi142.stdout: python3-attrs noarch 20.3.0-7.el9 CentOS-AppStream 78 k 2024-03-31T22:58:30.581 INFO:teuthology.orchestra.run.smithi142.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-31T22:58:30.581 INFO:teuthology.orchestra.run.smithi142.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-31T22:58:30.581 INFO:teuthology.orchestra.run.smithi142.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-31T22:58:30.581 INFO:teuthology.orchestra.run.smithi142.stdout: python3-ceph-argparse x86_64 2:19.0.0-1977.g384f31f3.el9 ceph 45 k 2024-03-31T22:58:30.581 INFO:teuthology.orchestra.run.smithi142.stdout: python3-ceph-common x86_64 2:19.0.0-1977.g384f31f3.el9 ceph 131 k 2024-03-31T22:58:30.581 INFO:teuthology.orchestra.run.smithi142.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-31T22:58:30.581 INFO:teuthology.orchestra.run.smithi142.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-31T22:58:30.582 INFO:teuthology.orchestra.run.smithi142.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-31T22:58:30.582 INFO:teuthology.orchestra.run.smithi142.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-31T22:58:30.582 INFO:teuthology.orchestra.run.smithi142.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-03-31T22:58:30.582 INFO:teuthology.orchestra.run.smithi142.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-03-31T22:58:30.582 INFO:teuthology.orchestra.run.smithi142.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-31T22:58:30.582 INFO:teuthology.orchestra.run.smithi142.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-31T22:58:30.582 INFO:teuthology.orchestra.run.smithi142.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-31T22:58:30.582 INFO:teuthology.orchestra.run.smithi142.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-31T22:58:30.582 INFO:teuthology.orchestra.run.smithi142.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-31T22:58:30.582 INFO:teuthology.orchestra.run.smithi142.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-31T22:58:30.582 INFO:teuthology.orchestra.run.smithi142.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-31T22:58:30.582 INFO:teuthology.orchestra.run.smithi142.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-31T22:58:30.583 INFO:teuthology.orchestra.run.smithi142.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-31T22:58:30.583 INFO:teuthology.orchestra.run.smithi142.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-31T22:58:30.583 INFO:teuthology.orchestra.run.smithi142.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-31T22:58:30.583 INFO:teuthology.orchestra.run.smithi142.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-31T22:58:30.583 INFO:teuthology.orchestra.run.smithi142.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-31T22:58:30.583 INFO:teuthology.orchestra.run.smithi142.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-31T22:58:30.583 INFO:teuthology.orchestra.run.smithi142.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-31T22:58:30.583 INFO:teuthology.orchestra.run.smithi142.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-31T22:58:30.583 INFO:teuthology.orchestra.run.smithi142.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-31T22:58:30.583 INFO:teuthology.orchestra.run.smithi142.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-31T22:58:30.583 INFO:teuthology.orchestra.run.smithi142.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-31T22:58:30.583 INFO:teuthology.orchestra.run.smithi142.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-31T22:58:30.583 INFO:teuthology.orchestra.run.smithi142.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-31T22:58:30.583 INFO:teuthology.orchestra.run.smithi142.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-31T22:58:30.584 INFO:teuthology.orchestra.run.smithi142.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-31T22:58:30.584 INFO:teuthology.orchestra.run.smithi142.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-31T22:58:30.584 INFO:teuthology.orchestra.run.smithi142.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-31T22:58:30.584 INFO:teuthology.orchestra.run.smithi142.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-31T22:58:30.584 INFO:teuthology.orchestra.run.smithi142.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-31T22:58:30.584 INFO:teuthology.orchestra.run.smithi142.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-31T22:58:30.584 INFO:teuthology.orchestra.run.smithi142.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-31T22:58:30.584 INFO:teuthology.orchestra.run.smithi142.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-31T22:58:30.584 INFO:teuthology.orchestra.run.smithi142.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-31T22:58:30.584 INFO:teuthology.orchestra.run.smithi142.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-31T22:58:30.584 INFO:teuthology.orchestra.run.smithi142.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-03-31T22:58:30.584 INFO:teuthology.orchestra.run.smithi142.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-31T22:58:30.584 INFO:teuthology.orchestra.run.smithi142.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-31T22:58:30.584 INFO:teuthology.orchestra.run.smithi142.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-31T22:58:30.585 INFO:teuthology.orchestra.run.smithi142.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-31T22:58:30.585 INFO:teuthology.orchestra.run.smithi142.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-31T22:58:30.585 INFO:teuthology.orchestra.run.smithi142.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-31T22:58:30.585 INFO:teuthology.orchestra.run.smithi142.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-31T22:58:30.585 INFO:teuthology.orchestra.run.smithi142.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-31T22:58:30.585 INFO:teuthology.orchestra.run.smithi142.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-31T22:58:30.585 INFO:teuthology.orchestra.run.smithi142.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-31T22:58:30.585 INFO:teuthology.orchestra.run.smithi142.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-31T22:58:30.585 INFO:teuthology.orchestra.run.smithi142.stdout: python3-wcwidth noarch 0.2.5-8.el9 CentOS-AppStream 41 k 2024-03-31T22:58:30.585 INFO:teuthology.orchestra.run.smithi142.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-31T22:58:30.585 INFO:teuthology.orchestra.run.smithi142.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-31T22:58:30.585 INFO:teuthology.orchestra.run.smithi142.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-31T22:58:30.585 INFO:teuthology.orchestra.run.smithi142.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-31T22:58:30.586 INFO:teuthology.orchestra.run.smithi142.stdout: qatlib x86_64 24.02.0-1.el9 CentOS-AppStream 220 k 2024-03-31T22:58:30.586 INFO:teuthology.orchestra.run.smithi142.stdout: qatzip-libs x86_64 1.2.0-1.el9 CentOS-AppStream 46 k 2024-03-31T22:58:30.586 INFO:teuthology.orchestra.run.smithi142.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-31T22:58:30.586 INFO:teuthology.orchestra.run.smithi142.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-31T22:58:30.586 INFO:teuthology.orchestra.run.smithi142.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-31T22:58:30.586 INFO:teuthology.orchestra.run.smithi142.stdout: uuid x86_64 1.6.2-55.el9 CentOS-AppStream 56 k 2024-03-31T22:58:30.586 INFO:teuthology.orchestra.run.smithi142.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-31T22:58:30.586 INFO:teuthology.orchestra.run.smithi142.stdout: zlib-devel x86_64 1.2.11-40.el9 CentOS-AppStream 44 k 2024-03-31T22:58:30.586 INFO:teuthology.orchestra.run.smithi142.stdout:Installing weak dependencies: 2024-03-31T22:58:30.587 INFO:teuthology.orchestra.run.smithi142.stdout: qatlib-service x86_64 24.02.0-1.el9 CentOS-AppStream 35 k 2024-03-31T22:58:30.587 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-03-31T22:58:30.587 INFO:teuthology.orchestra.run.smithi142.stdout:Transaction Summary 2024-03-31T22:58:30.587 INFO:teuthology.orchestra.run.smithi142.stdout:============================================================================================ 2024-03-31T22:58:30.587 INFO:teuthology.orchestra.run.smithi142.stdout:Install 150 Packages 2024-03-31T22:58:30.587 INFO:teuthology.orchestra.run.smithi142.stdout:Upgrade 24 Packages 2024-03-31T22:58:30.587 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-03-31T22:58:30.588 INFO:teuthology.orchestra.run.smithi162.stdout:CentOS Stream 9 - AppStream 936 kB/s | 19 MB 00:20 2024-03-31T22:58:30.588 INFO:teuthology.orchestra.run.smithi046.stdout:(33/174): ceph-mgr-rook-19.0.0-1977.g384f31f3.e 539 kB/s | 49 kB 00:00 2024-03-31T22:58:30.589 INFO:teuthology.orchestra.run.smithi142.stdout:Total download size: 226 M 2024-03-31T22:58:30.589 INFO:teuthology.orchestra.run.smithi142.stdout:Downloading Packages: 2024-03-31T22:58:30.655 INFO:teuthology.orchestra.run.smithi046.stdout:(34/174): ceph-prometheus-alerts-19.0.0-1977.g3 184 kB/s | 15 kB 00:00 2024-03-31T22:58:30.813 INFO:teuthology.orchestra.run.smithi046.stdout:(35/174): ceph-mgr-diskprediction-local-19.0.0- 8.7 MB/s | 7.4 MB 00:00 2024-03-31T22:58:30.839 INFO:teuthology.orchestra.run.smithi046.stdout:(36/174): ceph-volume-19.0.0-1977.g384f31f3.el9 1.4 MB/s | 264 kB 00:00 2024-03-31T22:58:30.931 INFO:teuthology.orchestra.run.smithi046.stdout:(37/174): cephadm-19.0.0-1977.g384f31f3.el9.noa 6.3 MB/s | 748 kB 00:00 2024-03-31T22:58:30.956 INFO:teuthology.orchestra.run.smithi046.stdout:(38/174): cephfs-top-19.0.0-1977.g384f31f3.el9. 156 kB/s | 18 kB 00:00 2024-03-31T22:58:31.106 INFO:teuthology.orchestra.run.smithi046.stdout:(39/174): automake-1.16.2-8.el9.noarch.rpm 3.7 MB/s | 662 kB 00:00 2024-03-31T22:58:31.156 INFO:teuthology.orchestra.run.smithi046.stdout:(40/174): bison-3.7.4-5.el9.x86_64.rpm 4.5 MB/s | 920 kB 00:00 2024-03-31T22:58:31.182 INFO:teuthology.orchestra.run.smithi046.stdout:(41/174): boost-program-options-1.75.0-8.el9.x8 1.4 MB/s | 106 kB 00:00 2024-03-31T22:58:31.207 INFO:teuthology.orchestra.run.smithi046.stdout:(42/174): flex-2.6.4-9.el9.x86_64.rpm 6.0 MB/s | 307 kB 00:00 2024-03-31T22:58:31.232 INFO:teuthology.orchestra.run.smithi046.stdout:(43/174): flexiblas-3.0.4-8.el9.x86_64.rpm 635 kB/s | 32 kB 00:00 2024-03-31T22:58:31.357 INFO:teuthology.orchestra.run.smithi046.stdout:(44/174): flexiblas-netlib-3.0.4-8.el9.x86_64.r 20 MB/s | 3.0 MB 00:00 2024-03-31T22:58:31.382 INFO:teuthology.orchestra.run.smithi046.stdout:(45/174): flexiblas-openblas-openmp-3.0.4-8.el9 115 kB/s | 17 kB 00:00 2024-03-31T22:58:31.408 INFO:teuthology.orchestra.run.smithi046.stdout:(46/174): libaio-devel-0.3.111-13.el9.x86_64.rp 201 kB/s | 10 kB 00:00 2024-03-31T22:58:31.432 INFO:teuthology.orchestra.run.smithi142.stdout:(1/174): ceph-19.0.0-1977.g384f31f3.el9.x86_64. 22 kB/s | 6.6 kB 00:00 2024-03-31T22:58:31.433 INFO:teuthology.orchestra.run.smithi046.stdout:(47/174): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.1 MB/s | 159 kB 00:00 2024-03-31T22:58:31.458 INFO:teuthology.orchestra.run.smithi046.stdout:(48/174): librabbitmq-0.11.0-7.el9.x86_64.rpm 882 kB/s | 44 kB 00:00 2024-03-31T22:58:31.484 INFO:teuthology.orchestra.run.smithi046.stdout:(49/174): librdkafka-1.6.1-102.el9.x86_64.rpm 13 MB/s | 661 kB 00:00 2024-03-31T22:58:31.518 INFO:teuthology.orchestra.run.smithi046.stdout:(50/174): libtool-2.4.6-45.el9.x86_64.rpm 9.7 MB/s | 578 kB 00:00 2024-03-31T22:58:31.534 INFO:teuthology.orchestra.run.smithi046.stdout:(51/174): libzstd-devel-1.5.1-2.el9.x86_64.rpm 941 kB/s | 46 kB 00:00 2024-03-31T22:58:31.568 INFO:teuthology.orchestra.run.smithi046.stdout:(52/174): lttng-ust-2.12.0-6.el9.x86_64.rpm 5.5 MB/s | 282 kB 00:00 2024-03-31T22:58:31.593 INFO:teuthology.orchestra.run.smithi046.stdout:(53/174): lua-5.4.4-4.el9.x86_64.rpm 3.1 MB/s | 187 kB 00:00 2024-03-31T22:58:31.682 INFO:teuthology.orchestra.run.smithi142.stdout:(2/174): ceph-base-19.0.0-1977.g384f31f3.el9.x8 9.8 MB/s | 5.3 MB 00:00 2024-03-31T22:58:31.723 INFO:teuthology.orchestra.run.smithi142.stdout:(3/174): ceph-immutable-object-cache-19.0.0-197 3.4 MB/s | 145 kB 00:00 2024-03-31T22:58:31.891 INFO:teuthology.orchestra.run.smithi142.stdout:(4/174): ceph-mds-19.0.0-1977.g384f31f3.el9.x86 14 MB/s | 2.4 MB 00:00 2024-03-31T22:58:31.991 INFO:teuthology.orchestra.run.smithi142.stdout:(5/174): ceph-fuse-19.0.0-1977.g384f31f3.el9.x8 1.6 MB/s | 889 kB 00:00 2024-03-31T22:58:32.019 INFO:teuthology.orchestra.run.smithi046.stdout:(54/174): ceph-test-19.0.0-1977.g384f31f3.el9.x 13 MB/s | 47 MB 00:03 2024-03-31T22:58:32.032 INFO:teuthology.orchestra.run.smithi142.stdout:(6/174): ceph-mgr-19.0.0-1977.g384f31f3.el9.x86 11 MB/s | 1.5 MB 00:00 2024-03-31T22:58:32.044 INFO:teuthology.orchestra.run.smithi046.stdout:(55/174): oniguruma-6.9.6-1.el9.5.x86_64.rpm 456 kB/s | 217 kB 00:00 2024-03-31T22:58:32.061 INFO:teuthology.orchestra.run.smithi046.stdout:(56/174): openblas-0.3.21-2.el9.x86_64.rpm 73 kB/s | 34 kB 00:00 2024-03-31T22:58:32.169 INFO:teuthology.orchestra.run.smithi046.stdout:(57/174): openblas-openmp-0.3.21-2.el9.x86_64.r 32 MB/s | 4.7 MB 00:00 2024-03-31T22:58:32.211 INFO:teuthology.orchestra.run.smithi046.stdout:(58/174): postgresql-contrib-13.14-1.el9.x86_64 5.4 MB/s | 817 kB 00:00 2024-03-31T22:58:32.253 INFO:teuthology.orchestra.run.smithi046.stdout:(59/174): postgresql-13.14-1.el9.x86_64.rpm 8.1 MB/s | 1.5 MB 00:00 2024-03-31T22:58:32.278 INFO:teuthology.orchestra.run.smithi046.stdout:(60/174): postgresql-private-libs-13.14-1.el9.x 1.2 MB/s | 132 kB 00:00 2024-03-31T22:58:32.370 INFO:teuthology.orchestra.run.smithi046.stdout:(61/174): postgresql-server-13.14-1.el9.x86_64. 36 MB/s | 5.7 MB 00:00 2024-03-31T22:58:32.396 INFO:teuthology.orchestra.run.smithi046.stdout:(62/174): python3-attrs-20.3.0-7.el9.noarch.rpm 684 kB/s | 78 kB 00:00 2024-03-31T22:58:32.437 INFO:teuthology.orchestra.run.smithi046.stdout:(63/174): protobuf-3.14.0-13.el9.x86_64.rpm 6.5 MB/s | 1.0 MB 00:00 2024-03-31T22:58:32.521 INFO:teuthology.orchestra.run.smithi046.stdout:(64/174): python3-babel-2.9.1-2.el9.noarch.rpm 39 MB/s | 5.8 MB 00:00 2024-03-31T22:58:32.538 INFO:teuthology.orchestra.run.smithi046.stdout:(65/174): python3-jsonpatch-1.21-16.el9.noarch. 180 kB/s | 24 kB 00:00 2024-03-31T22:58:32.555 INFO:teuthology.orchestra.run.smithi046.stdout:(66/174): python3-jsonpointer-2.0-4.el9.noarch. 159 kB/s | 18 kB 00:00 2024-03-31T22:58:32.588 INFO:teuthology.orchestra.run.smithi046.stdout:(67/174): python3-mako-1.1.4-6.el9.noarch.rpm 2.1 MB/s | 144 kB 00:00 2024-03-31T22:58:32.605 INFO:teuthology.orchestra.run.smithi046.stdout:(68/174): python3-markupsafe-1.1.1-12.el9.x86_6 636 kB/s | 32 kB 00:00 2024-03-31T22:58:32.672 INFO:teuthology.orchestra.run.smithi046.stdout:(69/174): python3-numpy-1.20.1-5.el9.x86_64.rpm 41 MB/s | 4.8 MB 00:00 2024-03-31T22:58:32.698 INFO:teuthology.orchestra.run.smithi046.stdout:(70/174): python3-numpy-f2py-1.20.1-5.el9.x86_6 2.0 MB/s | 217 kB 00:00 2024-03-31T22:58:32.723 INFO:teuthology.orchestra.run.smithi046.stdout:(71/174): python3-oauthlib-3.1.1-5.el9.noarch.r 1.5 MB/s | 174 kB 00:00 2024-03-31T22:58:32.748 INFO:teuthology.orchestra.run.smithi046.stdout:(72/174): python3-packaging-20.9-5.el9.noarch.r 918 kB/s | 69 kB 00:00 2024-03-31T22:58:32.773 INFO:teuthology.orchestra.run.smithi046.stdout:(73/174): python3-prettytable-0.7.2-27.el9.noar 825 kB/s | 41 kB 00:00 2024-03-31T22:58:32.798 INFO:teuthology.orchestra.run.smithi046.stdout:(74/174): python3-protobuf-3.14.0-13.el9.noarch 3.1 MB/s | 238 kB 00:00 2024-03-31T22:58:32.815 INFO:teuthology.orchestra.run.smithi046.stdout:(75/174): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.9 MB/s | 132 kB 00:00 2024-03-31T22:58:32.841 INFO:teuthology.orchestra.run.smithi046.stdout:(76/174): python3-pytz-2021.1-5.el9.noarch.rpm 1.1 MB/s | 47 kB 00:00 2024-03-31T22:58:32.866 INFO:teuthology.orchestra.run.smithi046.stdout:(77/174): python3-pyasn1-modules-0.4.8-6.el9.no 3.1 MB/s | 211 kB 00:00 2024-03-31T22:58:32.891 INFO:teuthology.orchestra.run.smithi046.stdout:(78/174): python3-requests-oauthlib-1.3.0-12.el 569 kB/s | 43 kB 00:00 2024-03-31T22:58:33.008 INFO:teuthology.orchestra.run.smithi046.stdout:(79/174): python3-wcwidth-0.2.5-8.el9.noarch.rp 361 kB/s | 41 kB 00:00 2024-03-31T22:58:33.033 INFO:teuthology.orchestra.run.smithi046.stdout:(80/174): python3-toml-0.10.2-6.el9.noarch.rpm 278 kB/s | 39 kB 00:00 2024-03-31T22:58:33.066 INFO:teuthology.orchestra.run.smithi142.stdout:(7/174): ceph-osd-19.0.0-1977.g384f31f3.el9.x86 16 MB/s | 17 MB 00:01 2024-03-31T22:58:33.209 INFO:teuthology.orchestra.run.smithi046.stdout:(81/174): python3-scipy-1.6.2-8.el9.x86_64.rpm 43 MB/s | 16 MB 00:00 2024-03-31T22:58:33.234 INFO:teuthology.orchestra.run.smithi046.stdout:(82/174): qatlib-24.02.0-1.el9.x86_64.rpm 976 kB/s | 220 kB 00:00 2024-03-31T22:58:33.268 INFO:teuthology.orchestra.run.smithi046.stdout:(83/174): qatlib-service-24.02.0-1.el9.x86_64.r 150 kB/s | 35 kB 00:00 2024-03-31T22:58:33.293 INFO:teuthology.orchestra.run.smithi046.stdout:(84/174): qatzip-libs-1.2.0-1.el9.x86_64.rpm 556 kB/s | 46 kB 00:00 2024-03-31T22:58:33.318 INFO:teuthology.orchestra.run.smithi046.stdout:(85/174): uuid-1.6.2-55.el9.x86_64.rpm 1.1 MB/s | 56 kB 00:00 2024-03-31T22:58:33.343 INFO:teuthology.orchestra.run.smithi046.stdout:(86/174): socat-1.7.4.1-5.el9.x86_64.rpm 3.9 MB/s | 300 kB 00:00 2024-03-31T22:58:33.368 INFO:teuthology.orchestra.run.smithi046.stdout:(87/174): xmlstarlet-1.6.1-20.el9.x86_64.rpm 833 kB/s | 63 kB 00:00 2024-03-31T22:58:33.393 INFO:teuthology.orchestra.run.smithi046.stdout:(88/174): zlib-devel-1.2.11-40.el9.x86_64.rpm 591 kB/s | 44 kB 00:00 2024-03-31T22:58:33.419 INFO:teuthology.orchestra.run.smithi046.stdout:(89/174): fuse-2.9.9-15.el9.x86_64.rpm 1.5 MB/s | 79 kB 00:00 2024-03-31T22:58:33.444 INFO:teuthology.orchestra.run.smithi046.stdout:(90/174): python3-cffi-1.14.5-5.el9.x86_64.rpm 3.1 MB/s | 241 kB 00:00 2024-03-31T22:58:33.477 INFO:teuthology.orchestra.run.smithi046.stdout:(91/174): python3-cryptography-36.0.1-4.el9.x86 14 MB/s | 1.1 MB 00:00 2024-03-31T22:58:33.503 INFO:teuthology.orchestra.run.smithi046.stdout:(92/174): python3-ply-3.11-14.el9.noarch.rpm 1.8 MB/s | 103 kB 00:00 2024-03-31T22:58:33.520 INFO:teuthology.orchestra.run.smithi046.stdout:(93/174): python3-pycparser-2.20-6.el9.noarch.r 1.7 MB/s | 124 kB 00:00 2024-03-31T22:58:33.545 INFO:teuthology.orchestra.run.smithi046.stdout:(94/174): python3-urllib3-1.26.5-5.el9.noarch.r 2.7 MB/s | 187 kB 00:00 2024-03-31T22:58:33.670 INFO:teuthology.orchestra.run.smithi046.stdout:(95/174): jq-1.6-16.el9.x86_64.rpm 1.2 MB/s | 187 kB 00:00 2024-03-31T22:58:33.695 INFO:teuthology.orchestra.run.smithi046.stdout:(96/174): libquadmath-11.4.1-3.el9.x86_64.rpm 1.3 MB/s | 195 kB 00:00 2024-03-31T22:58:33.754 INFO:teuthology.orchestra.run.smithi046.stdout:(97/174): python3-requests-2.25.1-8.el9.noarch. 1.5 MB/s | 125 kB 00:00 2024-03-31T22:58:33.813 INFO:teuthology.orchestra.run.smithi046.stdout:(98/174): libgfortran-11.4.1-3.el9.x86_64.rpm 2.7 MB/s | 803 kB 00:00 2024-03-31T22:58:33.838 INFO:teuthology.orchestra.run.smithi046.stdout:(99/174): elfutils-libelf-devel-0.190-2.el9.x86 151 kB/s | 21 kB 00:00 2024-03-31T22:58:33.850 INFO:teuthology.orchestra.run.smithi142.stdout:(8/174): ceph-radosgw-19.0.0-1977.g384f31f3.el9 12 MB/s | 9.8 MB 00:00 2024-03-31T22:58:33.863 INFO:teuthology.orchestra.run.smithi046.stdout:(100/174): libuuid-devel-2.37.4-18.el9.x86_64.r 234 kB/s | 25 kB 00:00 2024-03-31T22:58:33.988 INFO:teuthology.orchestra.run.smithi046.stdout:(101/174): python3-devel-3.9.17-2.el9.x86_64.rp 1.9 MB/s | 245 kB 00:00 2024-03-31T22:58:34.022 INFO:teuthology.orchestra.run.smithi046.stdout:(102/174): python3-jinja2-2.11.3-5.el9.noarch.r 1.5 MB/s | 248 kB 00:00 2024-03-31T22:58:34.042 INFO:teuthology.orchestra.run.smithi142.stdout:(9/174): ceph-common-19.0.0-1977.g384f31f3.el9. 7.1 MB/s | 21 MB 00:02 2024-03-31T22:58:34.067 INFO:teuthology.orchestra.run.smithi142.stdout:(10/174): ceph-selinux-19.0.0-1977.g384f31f3.el 116 kB/s | 25 kB 00:00 2024-03-31T22:58:34.097 INFO:teuthology.orchestra.run.smithi046.stdout:(103/174): xfsprogs-devel-6.3.0-1.el9.x86_64.rp 865 kB/s | 94 kB 00:00 2024-03-31T22:58:34.122 INFO:teuthology.orchestra.run.smithi046.stdout:(104/174): lua-devel-5.4.4-4.el9.x86_64.rpm 223 kB/s | 22 kB 00:00 2024-03-31T22:58:34.134 INFO:teuthology.orchestra.run.smithi142.stdout:(11/174): ceph-mon-19.0.0-1977.g384f31f3.el9.x8 2.2 MB/s | 4.7 MB 00:02 2024-03-31T22:58:34.159 INFO:teuthology.orchestra.run.smithi142.stdout:(12/174): cephfs-mirror-19.0.0-1977.g384f31f3.e 2.4 MB/s | 221 kB 00:00 2024-03-31T22:58:34.201 INFO:teuthology.orchestra.run.smithi142.stdout:(13/174): cephfs-shell-19.0.0-1977.g384f31f3.el 405 kB/s | 27 kB 00:00 2024-03-31T22:58:34.226 INFO:teuthology.orchestra.run.smithi142.stdout:(14/174): libcephfs-devel-19.0.0-1977.g384f31f3 475 kB/s | 32 kB 00:00 2024-03-31T22:58:34.248 INFO:teuthology.orchestra.run.smithi046.stdout:(105/174): protobuf-compiler-3.14.0-13.el9.x86_ 5.6 MB/s | 863 kB 00:00 2024-03-31T22:58:34.285 INFO:teuthology.orchestra.run.smithi142.stdout:(15/174): libcephsqlite-19.0.0-1977.g384f31f3.e 2.7 MB/s | 163 kB 00:00 2024-03-31T22:58:34.343 INFO:teuthology.orchestra.run.smithi142.stdout:(16/174): librados-devel-19.0.0-1977.g384f31f3. 2.1 MB/s | 127 kB 00:00 2024-03-31T22:58:34.418 INFO:teuthology.orchestra.run.smithi142.stdout:(17/174): libradosstriper1-19.0.0-1977.g384f31f 6.5 MB/s | 501 kB 00:00 2024-03-31T22:58:34.452 INFO:teuthology.orchestra.run.smithi142.stdout:(18/174): libcephfs2-19.0.0-1977.g384f31f3.el9. 2.9 MB/s | 746 kB 00:00 2024-03-31T22:58:34.511 INFO:teuthology.orchestra.run.smithi142.stdout:(19/174): python3-ceph-argparse-19.0.0-1977.g38 775 kB/s | 45 kB 00:00 2024-03-31T22:58:34.556 INFO:teuthology.orchestra.run.smithi046.stdout:(106/174): openssl-devel-3.0.7-27.el9.x86_64.rp 5.6 MB/s | 4.1 MB 00:00 2024-03-31T22:58:34.569 INFO:teuthology.orchestra.run.smithi142.stdout:(20/174): python3-ceph-common-19.0.0-1977.g384f 2.2 MB/s | 131 kB 00:00 2024-03-31T22:58:34.598 INFO:teuthology.orchestra.run.smithi046.stdout:(107/174): abseil-cpp-20211102.0-3.el9.x86_64.r 1.1 MB/s | 548 kB 00:00 2024-03-31T22:58:34.636 INFO:teuthology.orchestra.run.smithi142.stdout:(21/174): python3-cephfs-19.0.0-1977.g384f31f3. 2.4 MB/s | 161 kB 00:00 2024-03-31T22:58:34.707 INFO:teuthology.orchestra.run.smithi046.stdout:(108/174): gperftools-libs-2.9.1-2.el9.x86_64.r 673 kB/s | 309 kB 00:00 2024-03-31T22:58:34.711 INFO:teuthology.orchestra.run.smithi142.stdout:(22/174): python3-rados-19.0.0-1977.g384f31f3.e 4.2 MB/s | 320 kB 00:00 2024-03-31T22:58:34.787 INFO:teuthology.orchestra.run.smithi142.stdout:(23/174): python3-rbd-19.0.0-1977.g384f31f3.el9 3.9 MB/s | 298 kB 00:00 2024-03-31T22:58:34.790 INFO:teuthology.orchestra.run.smithi046.stdout:(109/174): libarrow-doc-9.0.0-10.el9.noarch.rpm 316 kB/s | 26 kB 00:00 2024-03-31T22:58:34.832 INFO:teuthology.orchestra.run.smithi046.stdout:(110/174): grpc-data-1.46.7-10.el9.noarch.rpm 71 kB/s | 19 kB 00:00 2024-03-31T22:58:34.846 INFO:teuthology.orchestra.run.smithi142.stdout:(24/174): python3-rgw-19.0.0-1977.g384f31f3.el9 1.7 MB/s | 99 kB 00:00 2024-03-31T22:58:34.891 INFO:teuthology.orchestra.run.smithi046.stdout:(111/174): libarrow-9.0.0-10.el9.x86_64.rpm 15 MB/s | 4.4 MB 00:00 2024-03-31T22:58:34.904 INFO:teuthology.orchestra.run.smithi142.stdout:(25/174): rbd-fuse-19.0.0-1977.g384f31f3.el9.x8 1.4 MB/s | 85 kB 00:00 2024-03-31T22:58:34.916 INFO:teuthology.orchestra.run.smithi046.stdout:(112/174): liboath-2.6.7-2.el9.x86_64.rpm 387 kB/s | 49 kB 00:00 2024-03-31T22:58:34.966 INFO:teuthology.orchestra.run.smithi046.stdout:(113/174): luarocks-3.9.2-1.el9.noarch.rpm 2.0 MB/s | 151 kB 00:00 2024-03-31T22:58:34.991 INFO:teuthology.orchestra.run.smithi046.stdout:(114/174): libunwind-1.6.2-1.el9.x86_64.rpm 423 kB/s | 67 kB 00:00 2024-03-31T22:58:35.041 INFO:teuthology.orchestra.run.smithi046.stdout:(115/174): python3-asyncssh-2.13.2-5.el9.noarch 7.1 MB/s | 548 kB 00:00 2024-03-31T22:58:35.084 INFO:teuthology.orchestra.run.smithi046.stdout:(116/174): parquet-libs-9.0.0-10.el9.x86_64.rpm 4.9 MB/s | 838 kB 00:00 2024-03-31T22:58:35.109 INFO:teuthology.orchestra.run.smithi046.stdout:(117/174): python3-bcrypt-3.2.2-1.el9.x86_64.rp 370 kB/s | 43 kB 00:00 2024-03-31T22:58:35.134 INFO:teuthology.orchestra.run.smithi046.stdout:(118/174): python3-cachetools-4.2.4-1.el9.noarc 348 kB/s | 32 kB 00:00 2024-03-31T22:58:35.160 INFO:teuthology.orchestra.run.smithi046.stdout:(119/174): python3-certifi-2021.10.8-2.el9.noar 198 kB/s | 15 kB 00:00 2024-03-31T22:58:35.205 INFO:teuthology.orchestra.run.smithi142.stdout:(26/174): rbd-mirror-19.0.0-1977.g384f31f3.el9. 10 MB/s | 3.1 MB 00:00 2024-03-31T22:58:35.210 INFO:teuthology.orchestra.run.smithi046.stdout:(120/174): python3-cherrypy-18.6.1-2.el9.noarch 4.7 MB/s | 358 kB 00:00 2024-03-31T22:58:35.244 INFO:teuthology.orchestra.run.smithi046.stdout:(121/174): python3-cmd2-2.4.2-2.el9.noarch.rpm 3.5 MB/s | 303 kB 00:00 2024-03-31T22:58:35.263 INFO:teuthology.orchestra.run.smithi142.stdout:(27/174): rbd-nbd-19.0.0-1977.g384f31f3.el9.x86 2.9 MB/s | 171 kB 00:00 2024-03-31T22:58:35.269 INFO:teuthology.orchestra.run.smithi046.stdout:(122/174): python3-cheroot-8.6.0-4.el9.noarch.r 1.1 MB/s | 172 kB 00:00 2024-03-31T22:58:35.294 INFO:teuthology.orchestra.run.smithi046.stdout:(123/174): python3-colorama-0.4.4-12.el9.noarch 421 kB/s | 35 kB 00:00 2024-03-31T22:58:35.313 INFO:teuthology.orchestra.run.smithi142.stdout:(28/174): ceph-grafana-dashboards-19.0.0-1977.g 633 kB/s | 31 kB 00:00 2024-03-31T22:58:35.327 INFO:teuthology.orchestra.run.smithi046.stdout:(124/174): python3-google-auth-2.28.2-1.el9.noa 2.4 MB/s | 203 kB 00:00 2024-03-31T22:58:35.369 INFO:teuthology.orchestra.run.smithi046.stdout:(125/174): python3-grpcio-tools-1.46.7-10.el9.x 1.9 MB/s | 144 kB 00:00 2024-03-31T22:58:35.372 INFO:teuthology.orchestra.run.smithi142.stdout:(29/174): ceph-mgr-cephadm-19.0.0-1977.g384f31f 2.3 MB/s | 139 kB 00:00 2024-03-31T22:58:35.403 INFO:teuthology.orchestra.run.smithi046.stdout:(126/174): python3-jaraco-8.2.1-3.el9.noarch.rp 142 kB/s | 11 kB 00:00 2024-03-31T22:58:35.445 INFO:teuthology.orchestra.run.smithi046.stdout:(127/174): python3-jaraco-classes-3.2.1-5.el9.n 235 kB/s | 18 kB 00:00 2024-03-31T22:58:35.479 INFO:teuthology.orchestra.run.smithi046.stdout:(128/174): python3-jaraco-collections-3.0.0-8.e 307 kB/s | 23 kB 00:00 2024-03-31T22:58:35.512 INFO:teuthology.orchestra.run.smithi046.stdout:(129/174): python3-jaraco-functools-3.5.0-2.el9 291 kB/s | 19 kB 00:00 2024-03-31T22:58:35.562 INFO:teuthology.orchestra.run.smithi046.stdout:(130/174): python3-grpcio-1.46.7-10.el9.x86_64. 6.9 MB/s | 2.0 MB 00:00 2024-03-31T22:58:35.589 INFO:teuthology.orchestra.run.smithi046.stdout:(131/174): python3-jaraco-text-3.2.0-6.el9.noar 181 kB/s | 20 kB 00:00 2024-03-31T22:58:35.622 INFO:teuthology.orchestra.run.smithi046.stdout:(132/174): python3-kubernetes-26.1.0-2.el9.noar 9.4 MB/s | 1.0 MB 00:00 2024-03-31T22:58:35.680 INFO:teuthology.orchestra.run.smithi046.stdout:(133/174): python3-logutils-0.3.5-21.el9.noarch 395 kB/s | 46 kB 00:00 2024-03-31T22:58:35.680 INFO:teuthology.orchestra.run.smithi142.stdout:(30/174): ceph-mgr-dashboard-19.0.0-1977.g384f3 11 MB/s | 3.5 MB 00:00 2024-03-31T22:58:35.697 INFO:teuthology.orchestra.run.smithi046.stdout:(134/174): python3-natsort-7.1.1-5.el9.noarch.r 770 kB/s | 58 kB 00:00 2024-03-31T22:58:35.722 INFO:teuthology.orchestra.run.smithi046.stdout:(135/174): python3-more-itertools-8.12.0-2.el9. 588 kB/s | 79 kB 00:00 2024-03-31T22:58:35.756 INFO:teuthology.orchestra.run.smithi046.stdout:(136/174): python3-pecan-1.4.2-3.el9.noarch.rpm 3.5 MB/s | 272 kB 00:00 2024-03-31T22:58:35.773 INFO:teuthology.orchestra.run.smithi046.stdout:(137/174): python3-portend-3.1.0-2.el9.noarch.r 217 kB/s | 16 kB 00:00 2024-03-31T22:58:35.796 INFO:teuthology.orchestra.run.smithi162.stdout:CentOS Stream 9 - CRB 3.8 MB/s | 6.1 MB 00:01 2024-03-31T22:58:35.798 INFO:teuthology.orchestra.run.smithi046.stdout:(138/174): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.2 MB/s | 90 kB 00:00 2024-03-31T22:58:35.823 INFO:teuthology.orchestra.run.smithi046.stdout:(139/174): python3-pyperclip-1.8.2-2.el9.noarch 421 kB/s | 28 kB 00:00 2024-03-31T22:58:35.840 INFO:teuthology.orchestra.run.smithi046.stdout:(140/174): python3-repoze-lru-0.7-16.el9.noarch 459 kB/s | 31 kB 00:00 2024-03-31T22:58:35.873 INFO:teuthology.orchestra.run.smithi046.stdout:(141/174): python3-routes-2.5.1-5.el9.noarch.rp 2.4 MB/s | 188 kB 00:00 2024-03-31T22:58:35.899 INFO:teuthology.orchestra.run.smithi046.stdout:(142/174): python3-rsa-4.9-2.el9.noarch.rpm 783 kB/s | 59 kB 00:00 2024-03-31T22:58:35.949 INFO:teuthology.orchestra.run.smithi046.stdout:(143/174): python3-tempora-5.0.0-2.el9.noarch.r 329 kB/s | 36 kB 00:00 2024-03-31T22:58:35.974 INFO:teuthology.orchestra.run.smithi046.stdout:(144/174): python3-typing-extensions-4.4.0-2.el 507 kB/s | 51 kB 00:00 2024-03-31T22:58:36.000 INFO:teuthology.orchestra.run.smithi046.stdout:(145/174): python3-webob-1.8.7-6.el9.noarch.rpm 2.2 MB/s | 230 kB 00:00 2024-03-31T22:58:36.025 INFO:teuthology.orchestra.run.smithi046.stdout:(146/174): python3-websocket-client-1.2.3-2.el9 1.2 MB/s | 90 kB 00:00 2024-03-31T22:58:36.031 INFO:teuthology.orchestra.run.smithi142.stdout:(31/174): librgw2-19.0.0-1977.g384f31f3.el9.x86 3.2 MB/s | 5.1 MB 00:01 2024-03-31T22:58:36.067 INFO:teuthology.orchestra.run.smithi046.stdout:(147/174): python3-werkzeug-2.0.3-3.el9.1.noarc 4.5 MB/s | 427 kB 00:00 2024-03-31T22:58:36.092 INFO:teuthology.orchestra.run.smithi046.stdout:(148/174): python3-zc-lockfile-2.0-10.el9.noarc 217 kB/s | 20 kB 00:00 2024-03-31T22:58:36.126 INFO:teuthology.orchestra.run.smithi046.stdout:(149/174): re2-20211101-3.el9.x86_64.rpm 1.9 MB/s | 192 kB 00:00 2024-03-31T22:58:36.214 INFO:teuthology.orchestra.run.smithi142.stdout:(32/174): ceph-mgr-modules-core-19.0.0-1977.g38 1.3 MB/s | 248 kB 00:00 2024-03-31T22:58:36.218 INFO:teuthology.orchestra.run.smithi046.stdout:(150/174): thrift-0.15.0-2.el9.x86_64.rpm 11 MB/s | 1.6 MB 00:00 2024-03-31T22:58:36.302 INFO:teuthology.orchestra.run.smithi046.stdout:(151/174): NetworkManager-1.47.2-1.el9.x86_64.r 27 MB/s | 2.3 MB 00:00 2024-03-31T22:58:36.306 INFO:teuthology.orchestra.run.smithi142.stdout:(33/174): ceph-mgr-diskprediction-local-19.0.0- 12 MB/s | 7.4 MB 00:00 2024-03-31T22:58:36.323 INFO:teuthology.orchestra.run.smithi142.stdout:(34/174): ceph-mgr-rook-19.0.0-1977.g384f31f3.e 455 kB/s | 49 kB 00:00 2024-03-31T22:58:36.348 INFO:teuthology.orchestra.run.smithi142.stdout:(35/174): ceph-prometheus-alerts-19.0.0-1977.g3 370 kB/s | 15 kB 00:00 2024-03-31T22:58:36.385 INFO:teuthology.orchestra.run.smithi046.stdout:(152/174): NetworkManager-libnm-1.47.2-1.el9.x8 22 MB/s | 1.8 MB 00:00 2024-03-31T22:58:36.419 INFO:teuthology.orchestra.run.smithi046.stdout:(153/174): NetworkManager-team-1.47.2-1.el9.x86 1.2 MB/s | 41 kB 00:00 2024-03-31T22:58:36.448 INFO:teuthology.orchestra.run.smithi142.stdout:(36/174): cephadm-19.0.0-1977.g384f31f3.el9.noa 7.3 MB/s | 748 kB 00:00 2024-03-31T22:58:36.460 INFO:teuthology.orchestra.run.smithi046.stdout:(154/174): NetworkManager-tui-1.47.2-1.el9.x86_ 5.8 MB/s | 247 kB 00:00 2024-03-31T22:58:36.474 INFO:teuthology.orchestra.run.smithi142.stdout:(37/174): ceph-volume-19.0.0-1977.g384f31f3.el9 1.7 MB/s | 264 kB 00:00 2024-03-31T22:58:36.511 INFO:teuthology.orchestra.run.smithi046.stdout:(155/174): xfsdump-3.1.12-4.el9.x86_64.rpm 6.3 MB/s | 321 kB 00:00 2024-03-31T22:58:36.524 INFO:teuthology.orchestra.run.smithi142.stdout:(38/174): cephfs-top-19.0.0-1977.g384f31f3.el9. 242 kB/s | 18 kB 00:00 2024-03-31T22:58:36.594 INFO:teuthology.orchestra.run.smithi046.stdout:(156/174): acl-2.3.1-4.el9.x86_64.rpm 855 kB/s | 71 kB 00:00 2024-03-31T22:58:36.625 INFO:teuthology.orchestra.run.smithi142.stdout:(39/174): automake-1.16.2-8.el9.noarch.rpm 4.3 MB/s | 662 kB 00:00 2024-03-31T22:58:36.645 INFO:teuthology.orchestra.run.smithi046.stdout:(157/174): elfutils-debuginfod-client-0.190-2.e 730 kB/s | 36 kB 00:00 2024-03-31T22:58:36.658 INFO:teuthology.orchestra.run.smithi142.stdout:(40/174): bison-3.7.4-5.el9.x86_64.rpm 6.8 MB/s | 920 kB 00:00 2024-03-31T22:58:36.675 INFO:teuthology.orchestra.run.smithi142.stdout:(41/174): boost-program-options-1.75.0-8.el9.x8 2.1 MB/s | 106 kB 00:00 2024-03-31T22:58:36.700 INFO:teuthology.orchestra.run.smithi142.stdout:(42/174): flex-2.6.4-9.el9.x86_64.rpm 7.2 MB/s | 307 kB 00:00 2024-03-31T22:58:36.720 INFO:teuthology.orchestra.run.smithi046.stdout:(158/174): elfutils-libelf-0.190-2.el9.x86_64.r 2.5 MB/s | 192 kB 00:00 2024-03-31T22:58:36.725 INFO:teuthology.orchestra.run.smithi142.stdout:(43/174): flexiblas-3.0.4-8.el9.x86_64.rpm 636 kB/s | 32 kB 00:00 2024-03-31T22:58:36.770 INFO:teuthology.orchestra.run.smithi046.stdout:(159/174): elfutils-libs-0.190-2.el9.x86_64.rpm 5.0 MB/s | 256 kB 00:00 2024-03-31T22:58:36.801 INFO:teuthology.orchestra.run.smithi142.stdout:(44/174): flexiblas-netlib-3.0.4-8.el9.x86_64.r 30 MB/s | 3.0 MB 00:00 2024-03-31T22:58:36.812 INFO:teuthology.orchestra.run.smithi046.stdout:(160/174): libacl-2.3.1-4.el9.x86_64.rpm 554 kB/s | 23 kB 00:00 2024-03-31T22:58:36.826 INFO:teuthology.orchestra.run.smithi142.stdout:(45/174): flexiblas-openblas-openmp-3.0.4-8.el9 175 kB/s | 17 kB 00:00 2024-03-31T22:58:36.851 INFO:teuthology.orchestra.run.smithi142.stdout:(46/174): libaio-devel-0.3.111-13.el9.x86_64.rp 201 kB/s | 10 kB 00:00 2024-03-31T22:58:36.854 INFO:teuthology.orchestra.run.smithi046.stdout:(161/174): libblkid-2.37.4-18.el9.x86_64.rpm 2.5 MB/s | 108 kB 00:00 2024-03-31T22:58:36.868 INFO:teuthology.orchestra.run.smithi142.stdout:(47/174): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.7 MB/s | 159 kB 00:00 2024-03-31T22:58:36.893 INFO:teuthology.orchestra.run.smithi142.stdout:(48/174): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.0 MB/s | 44 kB 00:00 2024-03-31T22:58:36.913 INFO:teuthology.orchestra.run.smithi046.stdout:(162/174): librados2-19.0.0-1977.g384f31f3.el9. 4.1 MB/s | 3.4 MB 00:00 2024-03-31T22:58:36.918 INFO:teuthology.orchestra.run.smithi142.stdout:(49/174): librdkafka-1.6.1-102.el9.x86_64.rpm 13 MB/s | 661 kB 00:00 2024-03-31T22:58:36.938 INFO:teuthology.orchestra.run.smithi046.stdout:(163/174): libfdisk-2.37.4-18.el9.x86_64.rpm 1.8 MB/s | 155 kB 00:00 2024-03-31T22:58:36.952 INFO:teuthology.orchestra.run.smithi142.stdout:(50/174): libtool-2.4.6-45.el9.x86_64.rpm 9.9 MB/s | 578 kB 00:00 2024-03-31T22:58:36.978 INFO:teuthology.orchestra.run.smithi142.stdout:(51/174): libzstd-devel-1.5.1-2.el9.x86_64.rpm 800 kB/s | 46 kB 00:00 2024-03-31T22:58:36.988 INFO:teuthology.orchestra.run.smithi046.stdout:(164/174): librbd1-19.0.0-1977.g384f31f3.el9.x8 3.6 MB/s | 3.1 MB 00:00 2024-03-31T22:58:37.003 INFO:teuthology.orchestra.run.smithi142.stdout:(52/174): lttng-ust-2.12.0-6.el9.x86_64.rpm 5.5 MB/s | 282 kB 00:00 2024-03-31T22:58:37.014 INFO:teuthology.orchestra.run.smithi046.stdout:(165/174): libsmartcols-2.37.4-18.el9.x86_64.rp 849 kB/s | 64 kB 00:00 2024-03-31T22:58:37.028 INFO:teuthology.orchestra.run.smithi142.stdout:(53/174): lua-5.4.4-4.el9.x86_64.rpm 3.7 MB/s | 187 kB 00:00 2024-03-31T22:58:37.039 INFO:teuthology.orchestra.run.smithi046.stdout:(166/174): libmount-2.37.4-18.el9.x86_64.rpm 1.1 MB/s | 136 kB 00:00 2024-03-31T22:58:37.053 INFO:teuthology.orchestra.run.smithi142.stdout:(54/174): oniguruma-6.9.6-1.el9.5.x86_64.rpm 4.3 MB/s | 217 kB 00:00 2024-03-31T22:58:37.064 INFO:teuthology.orchestra.run.smithi046.stdout:(167/174): libuuid-2.37.4-18.el9.x86_64.rpm 380 kB/s | 29 kB 00:00 2024-03-31T22:58:37.079 INFO:teuthology.orchestra.run.smithi142.stdout:(55/174): openblas-0.3.21-2.el9.x86_64.rpm 688 kB/s | 34 kB 00:00 2024-03-31T22:58:37.139 INFO:teuthology.orchestra.run.smithi046.stdout:(168/174): openssl-3.0.7-27.el9.x86_64.rpm 9.7 MB/s | 1.2 MB 00:00 2024-03-31T22:58:37.198 INFO:teuthology.orchestra.run.smithi046.stdout:(169/174): openssl-libs-3.0.7-27.el9.x86_64.rpm 14 MB/s | 2.2 MB 00:00 2024-03-31T22:58:37.241 INFO:teuthology.orchestra.run.smithi046.stdout:(170/174): util-linux-user-2.37.4-18.el9.x86_64 752 kB/s | 31 kB 00:00 2024-03-31T22:58:37.282 INFO:teuthology.orchestra.run.smithi046.stdout:(171/174): util-linux-2.37.4-18.el9.x86_64.rpm 11 MB/s | 2.3 MB 00:00 2024-03-31T22:58:37.324 INFO:teuthology.orchestra.run.smithi046.stdout:(172/174): util-linux-core-2.37.4-18.el9.x86_64 2.5 MB/s | 465 kB 00:00 2024-03-31T22:58:37.339 INFO:teuthology.orchestra.run.smithi162.stdout:CentOS Stream 9 - Extras packages 94 kB/s | 16 kB 00:00 2024-03-31T22:58:37.357 INFO:teuthology.orchestra.run.smithi046.stdout:(173/174): xfsprogs-6.3.0-1.el9.x86_64.rpm 9.2 MB/s | 1.1 MB 00:00 2024-03-31T22:58:37.382 INFO:teuthology.orchestra.run.smithi046.stdout:(174/174): libacl-devel-2.3.1-4.el9.x86_64.rpm 741 kB/s | 74 kB 00:00 2024-03-31T22:58:37.390 INFO:teuthology.orchestra.run.smithi046.stdout:-------------------------------------------------------------------------------- 2024-03-31T22:58:37.390 INFO:teuthology.orchestra.run.smithi046.stdout:Total 19 MB/s | 226 MB 00:11 2024-03-31T22:58:37.563 INFO:teuthology.orchestra.run.smithi142.stdout:(56/174): ceph-test-19.0.0-1977.g384f31f3.el9.x 13 MB/s | 47 MB 00:03 2024-03-31T22:58:37.629 INFO:teuthology.orchestra.run.smithi142.stdout:(57/174): openblas-openmp-0.3.21-2.el9.x86_64.r 8.2 MB/s | 4.7 MB 00:00 2024-03-31T22:58:37.663 INFO:teuthology.orchestra.run.smithi142.stdout:(58/174): postgresql-13.14-1.el9.x86_64.rpm 2.6 MB/s | 1.5 MB 00:00 2024-03-31T22:58:37.696 INFO:teuthology.orchestra.run.smithi142.stdout:(59/174): postgresql-contrib-13.14-1.el9.x86_64 6.1 MB/s | 817 kB 00:00 2024-03-31T22:58:37.722 INFO:teuthology.orchestra.run.smithi142.stdout:(60/174): postgresql-private-libs-13.14-1.el9.x 1.4 MB/s | 132 kB 00:00 2024-03-31T22:58:37.755 INFO:teuthology.orchestra.run.smithi142.stdout:(61/174): protobuf-3.14.0-13.el9.x86_64.rpm 17 MB/s | 1.0 MB 00:00 2024-03-31T22:58:37.766 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction check 2024-03-31T22:58:37.872 INFO:teuthology.orchestra.run.smithi142.stdout:(62/174): postgresql-server-13.14-1.el9.x86_64. 33 MB/s | 5.7 MB 00:00 2024-03-31T22:58:37.897 INFO:teuthology.orchestra.run.smithi142.stdout:(63/174): python3-attrs-20.3.0-7.el9.noarch.rpm 445 kB/s | 78 kB 00:00 2024-03-31T22:58:37.981 INFO:teuthology.orchestra.run.smithi142.stdout:(64/174): python3-babel-2.9.1-2.el9.noarch.rpm 26 MB/s | 5.8 MB 00:00 2024-03-31T22:58:37.998 INFO:teuthology.orchestra.run.smithi142.stdout:(65/174): python3-jsonpatch-1.21-16.el9.noarch. 199 kB/s | 24 kB 00:00 2024-03-31T22:58:38.015 INFO:teuthology.orchestra.run.smithi142.stdout:(66/174): python3-jsonpointer-2.0-4.el9.noarch. 158 kB/s | 18 kB 00:00 2024-03-31T22:58:38.040 INFO:teuthology.orchestra.run.smithi142.stdout:(67/174): python3-mako-1.1.4-6.el9.noarch.rpm 2.4 MB/s | 144 kB 00:00 2024-03-31T22:58:38.057 INFO:teuthology.orchestra.run.smithi142.stdout:(68/174): python3-markupsafe-1.1.1-12.el9.x86_6 764 kB/s | 32 kB 00:00 2024-03-31T22:58:38.062 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction check succeeded. 2024-03-31T22:58:38.062 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction test 2024-03-31T22:58:38.157 INFO:teuthology.orchestra.run.smithi142.stdout:(69/174): python3-numpy-1.20.1-5.el9.x86_64.rpm 34 MB/s | 4.8 MB 00:00 2024-03-31T22:58:38.182 INFO:teuthology.orchestra.run.smithi142.stdout:(70/174): python3-numpy-f2py-1.20.1-5.el9.x86_6 1.5 MB/s | 217 kB 00:00 2024-03-31T22:58:38.207 INFO:teuthology.orchestra.run.smithi142.stdout:(71/174): python3-oauthlib-3.1.1-5.el9.noarch.r 1.2 MB/s | 174 kB 00:00 2024-03-31T22:58:38.233 INFO:teuthology.orchestra.run.smithi142.stdout:(72/174): python3-packaging-20.9-5.el9.noarch.r 921 kB/s | 69 kB 00:00 2024-03-31T22:58:38.249 INFO:teuthology.orchestra.run.smithi142.stdout:(73/174): python3-prettytable-0.7.2-27.el9.noar 993 kB/s | 41 kB 00:00 2024-03-31T22:58:38.275 INFO:teuthology.orchestra.run.smithi142.stdout:(74/174): python3-protobuf-3.14.0-13.el9.noarch 3.5 MB/s | 238 kB 00:00 2024-03-31T22:58:38.300 INFO:teuthology.orchestra.run.smithi142.stdout:(75/174): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.9 MB/s | 132 kB 00:00 2024-03-31T22:58:38.317 INFO:teuthology.orchestra.run.smithi142.stdout:(76/174): python3-pytz-2021.1-5.el9.noarch.rpm 1.1 MB/s | 47 kB 00:00 2024-03-31T22:58:38.342 INFO:teuthology.orchestra.run.smithi142.stdout:(77/174): python3-pyasn1-modules-0.4.8-6.el9.no 3.1 MB/s | 211 kB 00:00 2024-03-31T22:58:38.359 INFO:teuthology.orchestra.run.smithi142.stdout:(78/174): python3-requests-oauthlib-1.3.0-12.el 732 kB/s | 43 kB 00:00 2024-03-31T22:58:38.493 INFO:teuthology.orchestra.run.smithi142.stdout:(79/174): python3-toml-0.10.2-6.el9.noarch.rpm 295 kB/s | 39 kB 00:00 2024-03-31T22:58:38.509 INFO:teuthology.orchestra.run.smithi142.stdout:(80/174): python3-wcwidth-0.2.5-8.el9.noarch.rp 278 kB/s | 41 kB 00:00 2024-03-31T22:58:38.660 INFO:teuthology.orchestra.run.smithi142.stdout:(81/174): python3-scipy-1.6.2-8.el9.x86_64.rpm 46 MB/s | 16 MB 00:00 2024-03-31T22:58:38.685 INFO:teuthology.orchestra.run.smithi142.stdout:(82/174): qatlib-24.02.0-1.el9.x86_64.rpm 1.1 MB/s | 220 kB 00:00 2024-03-31T22:58:38.710 INFO:teuthology.orchestra.run.smithi142.stdout:(83/174): qatlib-service-24.02.0-1.el9.x86_64.r 176 kB/s | 35 kB 00:00 2024-03-31T22:58:38.737 INFO:teuthology.orchestra.run.smithi142.stdout:(84/174): qatzip-libs-1.2.0-1.el9.x86_64.rpm 923 kB/s | 46 kB 00:00 2024-03-31T22:58:38.760 INFO:teuthology.orchestra.run.smithi142.stdout:(85/174): socat-1.7.4.1-5.el9.x86_64.rpm 3.9 MB/s | 300 kB 00:00 2024-03-31T22:58:38.786 INFO:teuthology.orchestra.run.smithi142.stdout:(86/174): uuid-1.6.2-55.el9.x86_64.rpm 752 kB/s | 56 kB 00:00 2024-03-31T22:58:38.811 INFO:teuthology.orchestra.run.smithi142.stdout:(87/174): zlib-devel-1.2.11-40.el9.x86_64.rpm 876 kB/s | 44 kB 00:00 2024-03-31T22:58:38.836 INFO:teuthology.orchestra.run.smithi142.stdout:(88/174): xmlstarlet-1.6.1-20.el9.x86_64.rpm 829 kB/s | 63 kB 00:00 2024-03-31T22:58:38.862 INFO:teuthology.orchestra.run.smithi142.stdout:(89/174): fuse-2.9.9-15.el9.x86_64.rpm 1.0 MB/s | 79 kB 00:00 2024-03-31T22:58:38.887 INFO:teuthology.orchestra.run.smithi142.stdout:(90/174): python3-cffi-1.14.5-5.el9.x86_64.rpm 3.1 MB/s | 241 kB 00:00 2024-03-31T22:58:38.912 INFO:teuthology.orchestra.run.smithi142.stdout:(91/174): python3-ply-3.11-14.el9.noarch.rpm 2.0 MB/s | 103 kB 00:00 2024-03-31T22:58:38.946 INFO:teuthology.orchestra.run.smithi142.stdout:(92/174): python3-cryptography-36.0.1-4.el9.x86 14 MB/s | 1.1 MB 00:00 2024-03-31T22:58:38.971 INFO:teuthology.orchestra.run.smithi142.stdout:(93/174): python3-pycparser-2.20-6.el9.noarch.r 1.5 MB/s | 124 kB 00:00 2024-03-31T22:58:38.996 INFO:teuthology.orchestra.run.smithi142.stdout:(94/174): python3-urllib3-1.26.5-5.el9.noarch.r 2.2 MB/s | 187 kB 00:00 2024-03-31T22:58:39.188 INFO:teuthology.orchestra.run.smithi142.stdout:(95/174): libquadmath-11.4.1-3.el9.x86_64.rpm 1.0 MB/s | 195 kB 00:00 2024-03-31T22:58:39.213 INFO:teuthology.orchestra.run.smithi142.stdout:(96/174): jq-1.6-16.el9.x86_64.rpm 774 kB/s | 187 kB 00:00 2024-03-31T22:58:39.238 INFO:teuthology.orchestra.run.smithi142.stdout:(97/174): python3-requests-2.25.1-8.el9.noarch. 2.4 MB/s | 125 kB 00:00 2024-03-31T22:58:39.272 INFO:teuthology.orchestra.run.smithi142.stdout:(98/174): libgfortran-11.4.1-3.el9.x86_64.rpm 2.6 MB/s | 803 kB 00:00 2024-03-31T22:58:39.305 INFO:teuthology.orchestra.run.smithi142.stdout:(99/174): elfutils-libelf-devel-0.190-2.el9.x86 234 kB/s | 21 kB 00:00 2024-03-31T22:58:39.464 INFO:teuthology.orchestra.run.smithi142.stdout:(100/174): libuuid-devel-2.37.4-18.el9.x86_64.r 113 kB/s | 25 kB 00:00 2024-03-31T22:58:39.573 INFO:teuthology.orchestra.run.smithi142.stdout:(101/174): python3-devel-3.9.17-2.el9.x86_64.rp 919 kB/s | 245 kB 00:00 2024-03-31T22:58:39.640 INFO:teuthology.orchestra.run.smithi142.stdout:(102/174): openssl-devel-3.0.7-27.el9.x86_64.rp 11 MB/s | 4.1 MB 00:00 2024-03-31T22:58:39.665 INFO:teuthology.orchestra.run.smithi142.stdout:(103/174): python3-jinja2-2.11.3-5.el9.noarch.r 1.2 MB/s | 248 kB 00:00 2024-03-31T22:58:39.690 INFO:teuthology.orchestra.run.smithi142.stdout:(104/174): xfsprogs-devel-6.3.0-1.el9.x86_64.rp 800 kB/s | 94 kB 00:00 2024-03-31T22:58:39.734 INFO:teuthology.orchestra.run.smithi162.stdout:Extra Packages for Enterprise Linux 9.2 MB/s | 21 MB 00:02 2024-03-31T22:58:39.757 INFO:teuthology.orchestra.run.smithi142.stdout:(105/174): lua-devel-5.4.4-4.el9.x86_64.rpm 190 kB/s | 22 kB 00:00 2024-03-31T22:58:39.882 INFO:teuthology.orchestra.run.smithi142.stdout:(106/174): protobuf-compiler-3.14.0-13.el9.x86_ 3.9 MB/s | 863 kB 00:00 2024-03-31T22:58:40.124 INFO:teuthology.orchestra.run.smithi142.stdout:(107/174): grpc-data-1.46.7-10.el9.noarch.rpm 80 kB/s | 19 kB 00:00 2024-03-31T22:58:40.150 INFO:teuthology.orchestra.run.smithi142.stdout:(108/174): abseil-cpp-20211102.0-3.el9.x86_64.r 1.2 MB/s | 548 kB 00:00 2024-03-31T22:58:40.175 INFO:teuthology.orchestra.run.smithi142.stdout:(109/174): gperftools-libs-2.9.1-2.el9.x86_64.r 790 kB/s | 309 kB 00:00 2024-03-31T22:58:40.186 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction test succeeded. 2024-03-31T22:58:40.187 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction 2024-03-31T22:58:40.225 INFO:teuthology.orchestra.run.smithi142.stdout:(110/174): libarrow-doc-9.0.0-10.el9.noarch.rpm 345 kB/s | 26 kB 00:00 2024-03-31T22:58:40.259 INFO:teuthology.orchestra.run.smithi142.stdout:(111/174): liboath-2.6.7-2.el9.x86_64.rpm 582 kB/s | 49 kB 00:00 2024-03-31T22:58:40.300 INFO:teuthology.orchestra.run.smithi142.stdout:(112/174): libunwind-1.6.2-1.el9.x86_64.rpm 898 kB/s | 67 kB 00:00 2024-03-31T22:58:40.351 INFO:teuthology.orchestra.run.smithi142.stdout:(113/174): luarocks-3.9.2-1.el9.noarch.rpm 1.6 MB/s | 151 kB 00:00 2024-03-31T22:58:40.443 INFO:teuthology.orchestra.run.smithi142.stdout:(114/174): parquet-libs-9.0.0-10.el9.x86_64.rpm 5.8 MB/s | 838 kB 00:00 2024-03-31T22:58:40.510 INFO:teuthology.orchestra.run.smithi142.stdout:(115/174): python3-asyncssh-2.13.2-5.el9.noarch 3.4 MB/s | 548 kB 00:00 2024-03-31T22:58:40.535 INFO:teuthology.orchestra.run.smithi142.stdout:(116/174): python3-bcrypt-3.2.2-1.el9.x86_64.rp 474 kB/s | 43 kB 00:00 2024-03-31T22:58:40.618 INFO:teuthology.orchestra.run.smithi142.stdout:(117/174): libarrow-9.0.0-10.el9.x86_64.rpm 9.0 MB/s | 4.4 MB 00:00 2024-03-31T22:58:40.644 INFO:teuthology.orchestra.run.smithi142.stdout:(118/174): python3-certifi-2021.10.8-2.el9.noar 138 kB/s | 15 kB 00:00 2024-03-31T22:58:40.669 INFO:teuthology.orchestra.run.smithi142.stdout:(119/174): python3-cachetools-4.2.4-1.el9.noarc 203 kB/s | 32 kB 00:00 2024-03-31T22:58:40.696 INFO:teuthology.orchestra.run.smithi142.stdout:(120/174): python3-cheroot-8.6.0-4.el9.noarch.r 2.2 MB/s | 172 kB 00:00 2024-03-31T22:58:40.722 INFO:teuthology.orchestra.run.smithi142.stdout:(121/174): python3-cherrypy-18.6.1-2.el9.noarch 4.5 MB/s | 358 kB 00:00 2024-03-31T22:58:40.747 INFO:teuthology.orchestra.run.smithi142.stdout:(122/174): python3-cmd2-2.4.2-2.el9.noarch.rpm 3.8 MB/s | 303 kB 00:00 2024-03-31T22:58:40.772 INFO:teuthology.orchestra.run.smithi142.stdout:(123/174): python3-colorama-0.4.4-12.el9.noarch 469 kB/s | 35 kB 00:00 2024-03-31T22:58:40.806 INFO:teuthology.orchestra.run.smithi142.stdout:(124/174): python3-google-auth-2.28.2-1.el9.noa 2.4 MB/s | 203 kB 00:00 2024-03-31T22:58:40.831 INFO:teuthology.orchestra.run.smithi142.stdout:(125/174): python3-grpcio-tools-1.46.7-10.el9.x 2.4 MB/s | 144 kB 00:00 2024-03-31T22:58:40.873 INFO:teuthology.orchestra.run.smithi142.stdout:(126/174): python3-jaraco-8.2.1-3.el9.noarch.rp 160 kB/s | 11 kB 00:00 2024-03-31T22:58:40.915 INFO:teuthology.orchestra.run.smithi142.stdout:(127/174): python3-jaraco-classes-3.2.1-5.el9.n 211 kB/s | 18 kB 00:00 2024-03-31T22:58:40.940 INFO:teuthology.orchestra.run.smithi142.stdout:(128/174): python3-jaraco-collections-3.0.0-8.e 347 kB/s | 23 kB 00:00 2024-03-31T22:58:40.982 INFO:teuthology.orchestra.run.smithi142.stdout:(129/174): python3-grpcio-1.46.7-10.el9.x86_64. 8.7 MB/s | 2.0 MB 00:00 2024-03-31T22:58:41.007 INFO:teuthology.orchestra.run.smithi142.stdout:(130/174): python3-jaraco-functools-3.5.0-2.el9 211 kB/s | 19 kB 00:00 2024-03-31T22:58:41.032 INFO:teuthology.orchestra.run.smithi142.stdout:(131/174): python3-jaraco-text-3.2.0-6.el9.noar 214 kB/s | 20 kB 00:00 2024-03-31T22:58:41.091 INFO:teuthology.orchestra.run.smithi142.stdout:(132/174): python3-kubernetes-26.1.0-2.el9.noar 9.4 MB/s | 1.0 MB 00:00 2024-03-31T22:58:41.116 INFO:teuthology.orchestra.run.smithi142.stdout:(133/174): python3-logutils-0.3.5-21.el9.noarch 554 kB/s | 46 kB 00:00 2024-03-31T22:58:41.142 INFO:teuthology.orchestra.run.smithi142.stdout:(134/174): python3-more-itertools-8.12.0-2.el9. 726 kB/s | 79 kB 00:00 2024-03-31T22:58:41.175 INFO:teuthology.orchestra.run.smithi142.stdout:(135/174): python3-natsort-7.1.1-5.el9.noarch.r 689 kB/s | 58 kB 00:00 2024-03-31T22:58:41.200 INFO:teuthology.orchestra.run.smithi142.stdout:(136/174): python3-pecan-1.4.2-3.el9.noarch.rpm 3.2 MB/s | 272 kB 00:00 2024-03-31T22:58:41.226 INFO:teuthology.orchestra.run.smithi142.stdout:(137/174): python3-portend-3.1.0-2.el9.noarch.r 196 kB/s | 16 kB 00:00 2024-03-31T22:58:41.259 INFO:teuthology.orchestra.run.smithi142.stdout:(138/174): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.0 MB/s | 90 kB 00:00 2024-03-31T22:58:41.284 INFO:teuthology.orchestra.run.smithi142.stdout:(139/174): python3-pyperclip-1.8.2-2.el9.noarch 337 kB/s | 28 kB 00:00 2024-03-31T22:58:41.310 INFO:teuthology.orchestra.run.smithi142.stdout:(140/174): python3-repoze-lru-0.7-16.el9.noarch 367 kB/s | 31 kB 00:00 2024-03-31T22:58:41.337 INFO:teuthology.orchestra.run.smithi142.stdout:(141/174): python3-routes-2.5.1-5.el9.noarch.rp 2.4 MB/s | 188 kB 00:00 2024-03-31T22:58:41.360 INFO:teuthology.orchestra.run.smithi142.stdout:(142/174): python3-rsa-4.9-2.el9.noarch.rpm 783 kB/s | 59 kB 00:00 2024-03-31T22:58:41.385 INFO:teuthology.orchestra.run.smithi142.stdout:(143/174): python3-tempora-5.0.0-2.el9.noarch.r 474 kB/s | 36 kB 00:00 2024-03-31T22:58:41.419 INFO:teuthology.orchestra.run.smithi142.stdout:(144/174): python3-typing-extensions-4.4.0-2.el 609 kB/s | 51 kB 00:00 2024-03-31T22:58:41.444 INFO:teuthology.orchestra.run.smithi142.stdout:(145/174): python3-webob-1.8.7-6.el9.noarch.rpm 2.7 MB/s | 230 kB 00:00 2024-03-31T22:58:41.486 INFO:teuthology.orchestra.run.smithi142.stdout:(146/174): python3-websocket-client-1.2.3-2.el9 893 kB/s | 90 kB 00:00 2024-03-31T22:58:41.511 INFO:teuthology.orchestra.run.smithi142.stdout:(147/174): python3-zc-lockfile-2.0-10.el9.noarc 299 kB/s | 20 kB 00:00 2024-03-31T22:58:41.537 INFO:teuthology.orchestra.run.smithi142.stdout:(148/174): python3-werkzeug-2.0.3-3.el9.1.noarc 3.6 MB/s | 427 kB 00:00 2024-03-31T22:58:41.570 INFO:teuthology.orchestra.run.smithi142.stdout:(149/174): re2-20211101-3.el9.x86_64.rpm 2.2 MB/s | 192 kB 00:00 2024-03-31T22:58:41.671 INFO:teuthology.orchestra.run.smithi142.stdout:(150/174): thrift-0.15.0-2.el9.x86_64.rpm 10 MB/s | 1.6 MB 00:00 2024-03-31T22:58:41.838 INFO:teuthology.orchestra.run.smithi142.stdout:(151/174): NetworkManager-1.47.2-1.el9.x86_64.r 14 MB/s | 2.3 MB 00:00 2024-03-31T22:58:41.905 INFO:teuthology.orchestra.run.smithi142.stdout:(152/174): NetworkManager-libnm-1.47.2-1.el9.x8 27 MB/s | 1.8 MB 00:00 2024-03-31T22:58:41.938 INFO:teuthology.orchestra.run.smithi142.stdout:(153/174): NetworkManager-team-1.47.2-1.el9.x86 1.2 MB/s | 41 kB 00:00 2024-03-31T22:58:41.980 INFO:teuthology.orchestra.run.smithi142.stdout:(154/174): NetworkManager-tui-1.47.2-1.el9.x86_ 5.8 MB/s | 247 kB 00:00 2024-03-31T22:58:42.031 INFO:teuthology.orchestra.run.smithi142.stdout:(155/174): librbd1-19.0.0-1977.g384f31f3.el9.x8 6.8 MB/s | 3.1 MB 00:00 2024-03-31T22:58:42.056 INFO:teuthology.orchestra.run.smithi142.stdout:(156/174): xfsdump-3.1.12-4.el9.x86_64.rpm 4.2 MB/s | 321 kB 00:00 2024-03-31T22:58:42.164 INFO:teuthology.orchestra.run.smithi142.stdout:(157/174): acl-2.3.1-4.el9.x86_64.rpm 537 kB/s | 71 kB 00:00 2024-03-31T22:58:42.181 INFO:teuthology.orchestra.run.smithi142.stdout:(158/174): elfutils-debuginfod-client-0.190-2.e 291 kB/s | 36 kB 00:00 2024-03-31T22:58:42.265 INFO:teuthology.orchestra.run.smithi142.stdout:(159/174): elfutils-libelf-0.190-2.el9.x86_64.r 1.9 MB/s | 192 kB 00:00 2024-03-31T22:58:42.315 INFO:teuthology.orchestra.run.smithi142.stdout:(160/174): libacl-2.3.1-4.el9.x86_64.rpm 463 kB/s | 23 kB 00:00 2024-03-31T22:58:42.340 INFO:teuthology.orchestra.run.smithi142.stdout:(161/174): elfutils-libs-0.190-2.el9.x86_64.rpm 1.6 MB/s | 256 kB 00:00 2024-03-31T22:58:42.390 INFO:teuthology.orchestra.run.smithi142.stdout:(162/174): librados2-19.0.0-1977.g384f31f3.el9. 4.0 MB/s | 3.4 MB 00:00 2024-03-31T22:58:42.416 INFO:teuthology.orchestra.run.smithi142.stdout:(163/174): libblkid-2.37.4-18.el9.x86_64.rpm 1.1 MB/s | 108 kB 00:00 2024-03-31T22:58:42.433 INFO:teuthology.orchestra.run.smithi142.stdout:(164/174): libfdisk-2.37.4-18.el9.x86_64.rpm 1.6 MB/s | 155 kB 00:00 2024-03-31T22:58:42.475 INFO:teuthology.orchestra.run.smithi142.stdout:(165/174): libsmartcols-2.37.4-18.el9.x86_64.rp 1.1 MB/s | 64 kB 00:00 2024-03-31T22:58:42.501 INFO:teuthology.orchestra.run.smithi142.stdout:(166/174): libuuid-2.37.4-18.el9.x86_64.rpm 429 kB/s | 29 kB 00:00 2024-03-31T22:58:42.625 INFO:teuthology.orchestra.run.smithi142.stdout:(167/174): openssl-3.0.7-27.el9.x86_64.rpm 8.1 MB/s | 1.2 MB 00:00 2024-03-31T22:58:42.650 INFO:teuthology.orchestra.run.smithi142.stdout:(168/174): libmount-2.37.4-18.el9.x86_64.rpm 522 kB/s | 136 kB 00:00 2024-03-31T22:58:42.701 INFO:teuthology.orchestra.run.smithi142.stdout:(169/174): openssl-libs-3.0.7-27.el9.x86_64.rpm 11 MB/s | 2.2 MB 00:00 2024-03-31T22:58:42.767 INFO:teuthology.orchestra.run.smithi142.stdout:(170/174): util-linux-user-2.37.4-18.el9.x86_64 467 kB/s | 31 kB 00:00 2024-03-31T22:58:42.793 INFO:teuthology.orchestra.run.smithi142.stdout:(171/174): util-linux-core-2.37.4-18.el9.x86_64 3.2 MB/s | 465 kB 00:00 2024-03-31T22:58:42.843 INFO:teuthology.orchestra.run.smithi142.stdout:(172/174): util-linux-2.37.4-18.el9.x86_64.rpm 10 MB/s | 2.3 MB 00:00 2024-03-31T22:58:42.885 INFO:teuthology.orchestra.run.smithi142.stdout:(173/174): xfsprogs-6.3.0-1.el9.x86_64.rpm 9.2 MB/s | 1.1 MB 00:00 2024-03-31T22:58:43.068 INFO:teuthology.orchestra.run.smithi142.stdout:(174/174): libacl-devel-2.3.1-4.el9.x86_64.rpm 270 kB/s | 74 kB 00:00 2024-03-31T22:58:43.076 INFO:teuthology.orchestra.run.smithi142.stdout:-------------------------------------------------------------------------------- 2024-03-31T22:58:43.076 INFO:teuthology.orchestra.run.smithi142.stdout:Total 18 MB/s | 226 MB 00:12 2024-03-31T22:58:43.450 INFO:teuthology.orchestra.run.smithi142.stdout:Running transaction check 2024-03-31T22:58:43.725 INFO:teuthology.orchestra.run.smithi142.stdout:Transaction check succeeded. 2024-03-31T22:58:43.726 INFO:teuthology.orchestra.run.smithi142.stdout:Running transaction test 2024-03-31T22:58:44.765 INFO:teuthology.orchestra.run.smithi162.stdout:lab-extras 12 kB/s | 1.7 kB 00:00 2024-03-31T22:58:45.755 INFO:teuthology.orchestra.run.smithi142.stdout:Transaction test succeeded. 2024-03-31T22:58:45.755 INFO:teuthology.orchestra.run.smithi142.stdout:Running transaction 2024-03-31T22:58:47.317 INFO:teuthology.orchestra.run.smithi162.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-31T22:58:47.318 INFO:teuthology.orchestra.run.smithi162.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-31T22:58:47.329 INFO:teuthology.orchestra.run.smithi162.stdout:Package NetworkManager-1:1.44.0-2.el9.x86_64 is already installed. 2024-03-31T22:58:47.330 INFO:teuthology.orchestra.run.smithi162.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-03-31T22:58:47.331 INFO:teuthology.orchestra.run.smithi162.stdout:Package util-linux-2.37.4-15.el9.x86_64 is already installed. 2024-03-31T22:58:47.332 INFO:teuthology.orchestra.run.smithi162.stdout:Package libacl-devel-2.3.1-3.el9.x86_64 is already installed. 2024-03-31T22:58:47.334 INFO:teuthology.orchestra.run.smithi162.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-03-31T22:58:47.337 INFO:teuthology.orchestra.run.smithi162.stdout:Package xfsdump-3.1.12-3.el9.x86_64 is already installed. 2024-03-31T22:58:47.337 INFO:teuthology.orchestra.run.smithi162.stdout:Package xfsprogs-5.19.0-4.el9.x86_64 is already installed. 2024-03-31T22:58:47.416 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-03-31T22:58:47.438 INFO:teuthology.orchestra.run.smithi162.stdout:============================================================================================ 2024-03-31T22:58:47.438 INFO:teuthology.orchestra.run.smithi162.stdout: Package Arch Version Repository Size 2024-03-31T22:58:47.438 INFO:teuthology.orchestra.run.smithi162.stdout:============================================================================================ 2024-03-31T22:58:47.438 INFO:teuthology.orchestra.run.smithi162.stdout:Installing: 2024-03-31T22:58:47.438 INFO:teuthology.orchestra.run.smithi162.stdout: bison x86_64 3.7.4-5.el9 CentOS-AppStream 920 k 2024-03-31T22:58:47.438 INFO:teuthology.orchestra.run.smithi162.stdout: ceph x86_64 2:19.0.0-1977.g384f31f3.el9 ceph 6.6 k 2024-03-31T22:58:47.438 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-base x86_64 2:19.0.0-1977.g384f31f3.el9 ceph 5.3 M 2024-03-31T22:58:47.438 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-fuse x86_64 2:19.0.0-1977.g384f31f3.el9 ceph 889 k 2024-03-31T22:58:47.438 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1977.g384f31f3.el9 ceph 145 k 2024-03-31T22:58:47.438 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-mgr x86_64 2:19.0.0-1977.g384f31f3.el9 ceph 1.5 M 2024-03-31T22:58:47.438 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1977.g384f31f3.el9 ceph-noarch 139 k 2024-03-31T22:58:47.439 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1977.g384f31f3.el9 ceph-noarch 3.5 M 2024-03-31T22:58:47.439 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1977.g384f31f3.el9 ceph-noarch 7.4 M 2024-03-31T22:58:47.439 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-mgr-rook noarch 2:19.0.0-1977.g384f31f3.el9 ceph-noarch 49 k 2024-03-31T22:58:47.439 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-radosgw x86_64 2:19.0.0-1977.g384f31f3.el9 ceph 9.8 M 2024-03-31T22:58:47.439 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-test x86_64 2:19.0.0-1977.g384f31f3.el9 ceph 47 M 2024-03-31T22:58:47.439 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-volume noarch 2:19.0.0-1977.g384f31f3.el9 ceph-noarch 264 k 2024-03-31T22:58:47.439 INFO:teuthology.orchestra.run.smithi162.stdout: cephadm noarch 2:19.0.0-1977.g384f31f3.el9 ceph-noarch 748 k 2024-03-31T22:58:47.439 INFO:teuthology.orchestra.run.smithi162.stdout: cephfs-mirror x86_64 2:19.0.0-1977.g384f31f3.el9 ceph 221 k 2024-03-31T22:58:47.439 INFO:teuthology.orchestra.run.smithi162.stdout: cephfs-shell x86_64 2:19.0.0-1977.g384f31f3.el9 ceph 27 k 2024-03-31T22:58:47.439 INFO:teuthology.orchestra.run.smithi162.stdout: cephfs-top noarch 2:19.0.0-1977.g384f31f3.el9 ceph-noarch 18 k 2024-03-31T22:58:47.439 INFO:teuthology.orchestra.run.smithi162.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-03-31T22:58:47.439 INFO:teuthology.orchestra.run.smithi162.stdout: flex x86_64 2.6.4-9.el9 CentOS-AppStream 307 k 2024-03-31T22:58:47.439 INFO:teuthology.orchestra.run.smithi162.stdout: libaio-devel x86_64 0.3.111-13.el9 CentOS-AppStream 10 k 2024-03-31T22:58:47.440 INFO:teuthology.orchestra.run.smithi162.stdout: libcephfs-devel x86_64 2:19.0.0-1977.g384f31f3.el9 ceph 32 k 2024-03-31T22:58:47.440 INFO:teuthology.orchestra.run.smithi162.stdout: libcephfs2 x86_64 2:19.0.0-1977.g384f31f3.el9 ceph 746 k 2024-03-31T22:58:47.440 INFO:teuthology.orchestra.run.smithi162.stdout: librados-devel x86_64 2:19.0.0-1977.g384f31f3.el9 ceph 127 k 2024-03-31T22:58:47.440 INFO:teuthology.orchestra.run.smithi162.stdout: libtool x86_64 2.4.6-45.el9 CentOS-AppStream 578 k 2024-03-31T22:58:47.440 INFO:teuthology.orchestra.run.smithi162.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-03-31T22:58:47.440 INFO:teuthology.orchestra.run.smithi162.stdout: openssl-devel x86_64 1:3.0.7-27.el9 appstream 4.1 M 2024-03-31T22:58:47.440 INFO:teuthology.orchestra.run.smithi162.stdout: postgresql x86_64 13.14-1.el9 CentOS-AppStream 1.5 M 2024-03-31T22:58:47.440 INFO:teuthology.orchestra.run.smithi162.stdout: postgresql-contrib x86_64 13.14-1.el9 CentOS-AppStream 817 k 2024-03-31T22:58:47.440 INFO:teuthology.orchestra.run.smithi162.stdout: postgresql-server x86_64 13.14-1.el9 CentOS-AppStream 5.7 M 2024-03-31T22:58:47.440 INFO:teuthology.orchestra.run.smithi162.stdout: python3-cephfs x86_64 2:19.0.0-1977.g384f31f3.el9 ceph 161 k 2024-03-31T22:58:47.441 INFO:teuthology.orchestra.run.smithi162.stdout: python3-rados x86_64 2:19.0.0-1977.g384f31f3.el9 ceph 320 k 2024-03-31T22:58:47.441 INFO:teuthology.orchestra.run.smithi162.stdout: python3-rbd x86_64 2:19.0.0-1977.g384f31f3.el9 ceph 298 k 2024-03-31T22:58:47.441 INFO:teuthology.orchestra.run.smithi162.stdout: python3-rgw x86_64 2:19.0.0-1977.g384f31f3.el9 ceph 99 k 2024-03-31T22:58:47.441 INFO:teuthology.orchestra.run.smithi162.stdout: rbd-fuse x86_64 2:19.0.0-1977.g384f31f3.el9 ceph 85 k 2024-03-31T22:58:47.441 INFO:teuthology.orchestra.run.smithi162.stdout: rbd-mirror x86_64 2:19.0.0-1977.g384f31f3.el9 ceph 3.1 M 2024-03-31T22:58:47.441 INFO:teuthology.orchestra.run.smithi162.stdout: rbd-nbd x86_64 2:19.0.0-1977.g384f31f3.el9 ceph 171 k 2024-03-31T22:58:47.441 INFO:teuthology.orchestra.run.smithi162.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-03-31T22:58:47.442 INFO:teuthology.orchestra.run.smithi162.stdout:Upgrading: 2024-03-31T22:58:47.442 INFO:teuthology.orchestra.run.smithi162.stdout: NetworkManager x86_64 1:1.47.2-1.el9 CentOS-BaseOS 2.3 M 2024-03-31T22:58:47.442 INFO:teuthology.orchestra.run.smithi162.stdout: NetworkManager-libnm x86_64 1:1.47.2-1.el9 CentOS-BaseOS 1.8 M 2024-03-31T22:58:47.442 INFO:teuthology.orchestra.run.smithi162.stdout: NetworkManager-team x86_64 1:1.47.2-1.el9 CentOS-BaseOS 41 k 2024-03-31T22:58:47.442 INFO:teuthology.orchestra.run.smithi162.stdout: NetworkManager-tui x86_64 1:1.47.2-1.el9 CentOS-BaseOS 247 k 2024-03-31T22:58:47.442 INFO:teuthology.orchestra.run.smithi162.stdout: acl x86_64 2.3.1-4.el9 baseos 71 k 2024-03-31T22:58:47.442 INFO:teuthology.orchestra.run.smithi162.stdout: elfutils-debuginfod-client x86_64 0.190-2.el9 baseos 36 k 2024-03-31T22:58:47.442 INFO:teuthology.orchestra.run.smithi162.stdout: elfutils-libelf x86_64 0.190-2.el9 baseos 192 k 2024-03-31T22:58:47.442 INFO:teuthology.orchestra.run.smithi162.stdout: elfutils-libs x86_64 0.190-2.el9 baseos 256 k 2024-03-31T22:58:47.442 INFO:teuthology.orchestra.run.smithi162.stdout: libacl x86_64 2.3.1-4.el9 baseos 23 k 2024-03-31T22:58:47.442 INFO:teuthology.orchestra.run.smithi162.stdout: libacl-devel x86_64 2.3.1-4.el9 appstream 74 k 2024-03-31T22:58:47.442 INFO:teuthology.orchestra.run.smithi162.stdout: libblkid x86_64 2.37.4-18.el9 baseos 108 k 2024-03-31T22:58:47.443 INFO:teuthology.orchestra.run.smithi162.stdout: libfdisk x86_64 2.37.4-18.el9 baseos 155 k 2024-03-31T22:58:47.443 INFO:teuthology.orchestra.run.smithi162.stdout: libmount x86_64 2.37.4-18.el9 baseos 136 k 2024-03-31T22:58:47.443 INFO:teuthology.orchestra.run.smithi162.stdout: librados2 x86_64 2:19.0.0-1977.g384f31f3.el9 ceph 3.4 M 2024-03-31T22:58:47.443 INFO:teuthology.orchestra.run.smithi162.stdout: librbd1 x86_64 2:19.0.0-1977.g384f31f3.el9 ceph 3.1 M 2024-03-31T22:58:47.443 INFO:teuthology.orchestra.run.smithi162.stdout: libsmartcols x86_64 2.37.4-18.el9 baseos 64 k 2024-03-31T22:58:47.443 INFO:teuthology.orchestra.run.smithi162.stdout: libuuid x86_64 2.37.4-18.el9 baseos 29 k 2024-03-31T22:58:47.443 INFO:teuthology.orchestra.run.smithi162.stdout: openssl x86_64 1:3.0.7-27.el9 baseos 1.2 M 2024-03-31T22:58:47.443 INFO:teuthology.orchestra.run.smithi162.stdout: openssl-libs x86_64 1:3.0.7-27.el9 baseos 2.2 M 2024-03-31T22:58:47.443 INFO:teuthology.orchestra.run.smithi162.stdout: util-linux x86_64 2.37.4-18.el9 baseos 2.3 M 2024-03-31T22:58:47.443 INFO:teuthology.orchestra.run.smithi162.stdout: util-linux-core x86_64 2.37.4-18.el9 baseos 465 k 2024-03-31T22:58:47.443 INFO:teuthology.orchestra.run.smithi162.stdout: util-linux-user x86_64 2.37.4-18.el9 baseos 31 k 2024-03-31T22:58:47.443 INFO:teuthology.orchestra.run.smithi162.stdout: xfsdump x86_64 3.1.12-4.el9 CentOS-BaseOS 321 k 2024-03-31T22:58:47.443 INFO:teuthology.orchestra.run.smithi162.stdout: xfsprogs x86_64 6.3.0-1.el9 baseos 1.1 M 2024-03-31T22:58:47.443 INFO:teuthology.orchestra.run.smithi162.stdout:Installing dependencies: 2024-03-31T22:58:47.444 INFO:teuthology.orchestra.run.smithi162.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-31T22:58:47.444 INFO:teuthology.orchestra.run.smithi162.stdout: automake noarch 1.16.2-8.el9 CentOS-AppStream 662 k 2024-03-31T22:58:47.444 INFO:teuthology.orchestra.run.smithi162.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-31T22:58:47.444 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-common x86_64 2:19.0.0-1977.g384f31f3.el9 ceph 21 M 2024-03-31T22:58:47.444 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1977.g384f31f3.el9 ceph-noarch 31 k 2024-03-31T22:58:47.444 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-mds x86_64 2:19.0.0-1977.g384f31f3.el9 ceph 2.4 M 2024-03-31T22:58:47.444 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1977.g384f31f3.el9 ceph-noarch 248 k 2024-03-31T22:58:47.444 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-mon x86_64 2:19.0.0-1977.g384f31f3.el9 ceph 4.7 M 2024-03-31T22:58:47.444 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-osd x86_64 2:19.0.0-1977.g384f31f3.el9 ceph 17 M 2024-03-31T22:58:47.444 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1977.g384f31f3.el9 ceph-noarch 15 k 2024-03-31T22:58:47.444 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-selinux x86_64 2:19.0.0-1977.g384f31f3.el9 ceph 25 k 2024-03-31T22:58:47.444 INFO:teuthology.orchestra.run.smithi162.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-31T22:58:47.444 INFO:teuthology.orchestra.run.smithi162.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-31T22:58:47.444 INFO:teuthology.orchestra.run.smithi162.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-31T22:58:47.445 INFO:teuthology.orchestra.run.smithi162.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-31T22:58:47.445 INFO:teuthology.orchestra.run.smithi162.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-31T22:58:47.445 INFO:teuthology.orchestra.run.smithi162.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-31T22:58:47.445 INFO:teuthology.orchestra.run.smithi162.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-31T22:58:47.445 INFO:teuthology.orchestra.run.smithi162.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-31T22:58:47.445 INFO:teuthology.orchestra.run.smithi162.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-31T22:58:47.445 INFO:teuthology.orchestra.run.smithi162.stdout: libcephsqlite x86_64 2:19.0.0-1977.g384f31f3.el9 ceph 163 k 2024-03-31T22:58:47.445 INFO:teuthology.orchestra.run.smithi162.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-31T22:58:47.445 INFO:teuthology.orchestra.run.smithi162.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-31T22:58:47.445 INFO:teuthology.orchestra.run.smithi162.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-31T22:58:47.445 INFO:teuthology.orchestra.run.smithi162.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-31T22:58:47.445 INFO:teuthology.orchestra.run.smithi162.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-31T22:58:47.445 INFO:teuthology.orchestra.run.smithi162.stdout: libradosstriper1 x86_64 2:19.0.0-1977.g384f31f3.el9 ceph 501 k 2024-03-31T22:58:47.446 INFO:teuthology.orchestra.run.smithi162.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-31T22:58:47.446 INFO:teuthology.orchestra.run.smithi162.stdout: librgw2 x86_64 2:19.0.0-1977.g384f31f3.el9 ceph 5.1 M 2024-03-31T22:58:47.446 INFO:teuthology.orchestra.run.smithi162.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-31T22:58:47.446 INFO:teuthology.orchestra.run.smithi162.stdout: libzstd-devel x86_64 1.5.1-2.el9 CentOS-AppStream 46 k 2024-03-31T22:58:47.446 INFO:teuthology.orchestra.run.smithi162.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-31T22:58:47.446 INFO:teuthology.orchestra.run.smithi162.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-31T22:58:47.446 INFO:teuthology.orchestra.run.smithi162.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-31T22:58:47.446 INFO:teuthology.orchestra.run.smithi162.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-31T22:58:47.447 INFO:teuthology.orchestra.run.smithi162.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-31T22:58:47.447 INFO:teuthology.orchestra.run.smithi162.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-31T22:58:47.447 INFO:teuthology.orchestra.run.smithi162.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-31T22:58:47.447 INFO:teuthology.orchestra.run.smithi162.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-31T22:58:47.447 INFO:teuthology.orchestra.run.smithi162.stdout: postgresql-private-libs x86_64 13.14-1.el9 CentOS-AppStream 132 k 2024-03-31T22:58:47.447 INFO:teuthology.orchestra.run.smithi162.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-31T22:58:47.447 INFO:teuthology.orchestra.run.smithi162.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-31T22:58:47.447 INFO:teuthology.orchestra.run.smithi162.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-31T22:58:47.447 INFO:teuthology.orchestra.run.smithi162.stdout: python3-attrs noarch 20.3.0-7.el9 CentOS-AppStream 78 k 2024-03-31T22:58:47.447 INFO:teuthology.orchestra.run.smithi162.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-31T22:58:47.447 INFO:teuthology.orchestra.run.smithi162.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-31T22:58:47.448 INFO:teuthology.orchestra.run.smithi162.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-31T22:58:47.448 INFO:teuthology.orchestra.run.smithi162.stdout: python3-ceph-argparse x86_64 2:19.0.0-1977.g384f31f3.el9 ceph 45 k 2024-03-31T22:58:47.448 INFO:teuthology.orchestra.run.smithi162.stdout: python3-ceph-common x86_64 2:19.0.0-1977.g384f31f3.el9 ceph 131 k 2024-03-31T22:58:47.448 INFO:teuthology.orchestra.run.smithi162.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-31T22:58:47.448 INFO:teuthology.orchestra.run.smithi162.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-31T22:58:47.448 INFO:teuthology.orchestra.run.smithi162.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-31T22:58:47.448 INFO:teuthology.orchestra.run.smithi162.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-31T22:58:47.448 INFO:teuthology.orchestra.run.smithi162.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-03-31T22:58:47.448 INFO:teuthology.orchestra.run.smithi162.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-03-31T22:58:47.448 INFO:teuthology.orchestra.run.smithi162.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-31T22:58:47.448 INFO:teuthology.orchestra.run.smithi162.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-31T22:58:47.448 INFO:teuthology.orchestra.run.smithi162.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-31T22:58:47.448 INFO:teuthology.orchestra.run.smithi162.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-31T22:58:47.449 INFO:teuthology.orchestra.run.smithi162.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-31T22:58:47.449 INFO:teuthology.orchestra.run.smithi162.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-31T22:58:47.449 INFO:teuthology.orchestra.run.smithi162.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-31T22:58:47.449 INFO:teuthology.orchestra.run.smithi162.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-31T22:58:47.449 INFO:teuthology.orchestra.run.smithi162.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-31T22:58:47.449 INFO:teuthology.orchestra.run.smithi162.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-31T22:58:47.449 INFO:teuthology.orchestra.run.smithi162.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-31T22:58:47.449 INFO:teuthology.orchestra.run.smithi162.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-31T22:58:47.449 INFO:teuthology.orchestra.run.smithi162.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-31T22:58:47.449 INFO:teuthology.orchestra.run.smithi162.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-31T22:58:47.449 INFO:teuthology.orchestra.run.smithi162.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-31T22:58:47.449 INFO:teuthology.orchestra.run.smithi162.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-31T22:58:47.449 INFO:teuthology.orchestra.run.smithi162.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-31T22:58:47.450 INFO:teuthology.orchestra.run.smithi162.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-31T22:58:47.450 INFO:teuthology.orchestra.run.smithi162.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-31T22:58:47.450 INFO:teuthology.orchestra.run.smithi162.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-31T22:58:47.450 INFO:teuthology.orchestra.run.smithi162.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-31T22:58:47.450 INFO:teuthology.orchestra.run.smithi162.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-31T22:58:47.450 INFO:teuthology.orchestra.run.smithi162.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-31T22:58:47.450 INFO:teuthology.orchestra.run.smithi162.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-31T22:58:47.450 INFO:teuthology.orchestra.run.smithi162.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-31T22:58:47.450 INFO:teuthology.orchestra.run.smithi162.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-31T22:58:47.450 INFO:teuthology.orchestra.run.smithi162.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-31T22:58:47.450 INFO:teuthology.orchestra.run.smithi162.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-31T22:58:47.450 INFO:teuthology.orchestra.run.smithi162.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-31T22:58:47.450 INFO:teuthology.orchestra.run.smithi162.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-31T22:58:47.451 INFO:teuthology.orchestra.run.smithi162.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-31T22:58:47.451 INFO:teuthology.orchestra.run.smithi162.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-31T22:58:47.451 INFO:teuthology.orchestra.run.smithi162.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-03-31T22:58:47.451 INFO:teuthology.orchestra.run.smithi162.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-31T22:58:47.451 INFO:teuthology.orchestra.run.smithi162.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-31T22:58:47.451 INFO:teuthology.orchestra.run.smithi162.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-31T22:58:47.451 INFO:teuthology.orchestra.run.smithi162.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-31T22:58:47.451 INFO:teuthology.orchestra.run.smithi162.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-31T22:58:47.451 INFO:teuthology.orchestra.run.smithi162.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-31T22:58:47.451 INFO:teuthology.orchestra.run.smithi162.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-31T22:58:47.451 INFO:teuthology.orchestra.run.smithi162.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-31T22:58:47.451 INFO:teuthology.orchestra.run.smithi162.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-31T22:58:47.451 INFO:teuthology.orchestra.run.smithi162.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-31T22:58:47.452 INFO:teuthology.orchestra.run.smithi162.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-31T22:58:47.452 INFO:teuthology.orchestra.run.smithi162.stdout: python3-wcwidth noarch 0.2.5-8.el9 CentOS-AppStream 41 k 2024-03-31T22:58:47.452 INFO:teuthology.orchestra.run.smithi162.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-31T22:58:47.452 INFO:teuthology.orchestra.run.smithi162.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-31T22:58:47.452 INFO:teuthology.orchestra.run.smithi162.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-31T22:58:47.452 INFO:teuthology.orchestra.run.smithi162.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-31T22:58:47.452 INFO:teuthology.orchestra.run.smithi162.stdout: qatlib x86_64 24.02.0-1.el9 CentOS-AppStream 220 k 2024-03-31T22:58:47.452 INFO:teuthology.orchestra.run.smithi162.stdout: qatzip-libs x86_64 1.2.0-1.el9 CentOS-AppStream 46 k 2024-03-31T22:58:47.452 INFO:teuthology.orchestra.run.smithi162.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-31T22:58:47.452 INFO:teuthology.orchestra.run.smithi162.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-31T22:58:47.452 INFO:teuthology.orchestra.run.smithi162.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-31T22:58:47.452 INFO:teuthology.orchestra.run.smithi162.stdout: uuid x86_64 1.6.2-55.el9 CentOS-AppStream 56 k 2024-03-31T22:58:47.453 INFO:teuthology.orchestra.run.smithi162.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-31T22:58:47.453 INFO:teuthology.orchestra.run.smithi162.stdout: zlib-devel x86_64 1.2.11-40.el9 CentOS-AppStream 44 k 2024-03-31T22:58:47.453 INFO:teuthology.orchestra.run.smithi162.stdout:Installing weak dependencies: 2024-03-31T22:58:47.453 INFO:teuthology.orchestra.run.smithi162.stdout: qatlib-service x86_64 24.02.0-1.el9 CentOS-AppStream 35 k 2024-03-31T22:58:47.453 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-31T22:58:47.453 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction Summary 2024-03-31T22:58:47.453 INFO:teuthology.orchestra.run.smithi162.stdout:============================================================================================ 2024-03-31T22:58:47.453 INFO:teuthology.orchestra.run.smithi162.stdout:Install 150 Packages 2024-03-31T22:58:47.453 INFO:teuthology.orchestra.run.smithi162.stdout:Upgrade 24 Packages 2024-03-31T22:58:47.453 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-31T22:58:47.454 INFO:teuthology.orchestra.run.smithi162.stdout:Total download size: 226 M 2024-03-31T22:58:47.454 INFO:teuthology.orchestra.run.smithi162.stdout:Downloading Packages: 2024-03-31T22:58:47.489 INFO:teuthology.orchestra.run.smithi046.stdout: Preparing : 1/1 2024-03-31T22:58:47.571 INFO:teuthology.orchestra.run.smithi046.stdout: Upgrading : openssl-libs-1:3.0.7-27.el9.x86_64 1/198 2024-03-31T22:58:47.614 INFO:teuthology.orchestra.run.smithi046.stdout: Upgrading : libuuid-2.37.4-18.el9.x86_64 2/198 2024-03-31T22:58:47.652 INFO:teuthology.orchestra.run.smithi046.stdout: Upgrading : libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-31T22:58:47.661 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-31T22:58:47.714 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 4/198 2024-03-31T22:58:47.774 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 5/198 2024-03-31T22:58:47.816 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/198 2024-03-31T22:58:47.841 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 7/198 2024-03-31T22:58:47.897 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 8/198 2024-03-31T22:58:48.236 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 9/198 2024-03-31T22:58:48.291 INFO:teuthology.orchestra.run.smithi046.stdout: Upgrading : librados2-2:19.0.0-1977.g384f31f3.el9.x86_64 10/198 2024-03-31T22:58:48.399 INFO:teuthology.orchestra.run.smithi162.stdout:(1/174): ceph-19.0.0-1977.g384f31f3.el9.x86_64. 29 kB/s | 6.6 kB 00:00 2024-03-31T22:58:48.415 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: librados2-2:19.0.0-1977.g384f31f3.el9.x86_64 10/198 2024-03-31T22:58:48.567 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : libcephfs2-2:19.0.0-1977.g384f31f3.el9.x86_64 11/198 2024-03-31T22:58:48.659 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: libcephfs2-2:19.0.0-1977.g384f31f3.el9.x86_64 11/198 2024-03-31T22:58:48.688 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-rados-2:19.0.0-1977.g384f31f3.el9.x86_64 12/198 2024-03-31T22:58:48.732 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 13/198 2024-03-31T22:58:48.758 INFO:teuthology.orchestra.run.smithi162.stdout:(2/174): ceph-fuse-19.0.0-1977.g384f31f3.el9.x8 2.4 MB/s | 889 kB 00:00 2024-03-31T22:58:48.858 INFO:teuthology.orchestra.run.smithi162.stdout:(3/174): ceph-immutable-object-cache-19.0.0-197 1.4 MB/s | 145 kB 00:00 2024-03-31T22:58:48.931 INFO:teuthology.orchestra.run.smithi046.stdout: Upgrading : elfutils-libelf-0.190-2.el9.x86_64 14/198 2024-03-31T22:58:48.978 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : postgresql-13.14-1.el9.x86_64 15/198 2024-03-31T22:58:49.005 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : libcephsqlite-2:19.0.0-1977.g384f31f3.el9.x86_64 16/198 2024-03-31T22:58:49.016 INFO:teuthology.orchestra.run.smithi162.stdout:(4/174): ceph-base-19.0.0-1977.g384f31f3.el9.x8 6.4 MB/s | 5.3 MB 00:00 2024-03-31T22:58:49.110 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1977.g384f31f3.el9.x86_64 16/198 2024-03-31T22:58:49.140 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : libradosstriper1-2:19.0.0-1977.g384f31f3.el9.x86 17/198 2024-03-31T22:58:49.192 INFO:teuthology.orchestra.run.smithi162.stdout:(5/174): ceph-mgr-19.0.0-1977.g384f31f3.el9.x86 8.5 MB/s | 1.5 MB 00:00 2024-03-31T22:58:49.205 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1977.g384f31f3.el9.x86 17/198 2024-03-31T22:58:49.235 INFO:teuthology.orchestra.run.smithi046.stdout: Upgrading : libmount-2.37.4-18.el9.x86_64 18/198 2024-03-31T22:58:49.258 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 19/198 2024-03-31T22:58:49.273 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 19/198 2024-03-31T22:58:49.375 INFO:teuthology.orchestra.run.smithi162.stdout:(6/174): ceph-mds-19.0.0-1977.g384f31f3.el9.x86 4.6 MB/s | 2.4 MB 00:00 2024-03-31T22:58:49.416 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 20/198 2024-03-31T22:58:49.458 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 20/198 2024-03-31T22:58:49.496 INFO:teuthology.orchestra.run.smithi046.stdout: Upgrading : libsmartcols-2.37.4-18.el9.x86_64 21/198 2024-03-31T22:58:49.690 INFO:teuthology.orchestra.run.smithi046.stdout: Upgrading : libacl-2.3.1-4.el9.x86_64 22/198 2024-03-31T22:58:49.742 INFO:teuthology.orchestra.run.smithi162.stdout:(7/174): ceph-mon-19.0.0-1977.g384f31f3.el9.x86 8.5 MB/s | 4.7 MB 00:00 2024-03-31T22:58:49.814 INFO:teuthology.orchestra.run.smithi046.stdout: Upgrading : NetworkManager-libnm-1:1.47.2-1.el9.x86_64 23/198 2024-03-31T22:58:49.866 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.2-1.el9.x86_64 23/198 2024-03-31T22:58:49.954 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-31T22:58:50.001 INFO:teuthology.orchestra.run.smithi046.stdout: Upgrading : NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-31T22:58:50.208 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-31T22:58:50.242 INFO:teuthology.orchestra.run.smithi162.stdout:(8/174): ceph-common-19.0.0-1977.g384f31f3.el9. 10 MB/s | 21 MB 00:02 2024-03-31T22:58:50.317 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : re2-1:20211101-3.el9.x86_64 25/198 2024-03-31T22:58:50.326 INFO:teuthology.orchestra.run.smithi162.stdout:(9/174): ceph-selinux-19.0.0-1977.g384f31f3.el9 303 kB/s | 25 kB 00:00 2024-03-31T22:58:50.414 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 26/198 2024-03-31T22:58:50.447 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 27/198 2024-03-31T22:58:50.474 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : liboath-2.6.7-2.el9.x86_64 28/198 2024-03-31T22:58:50.510 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/198 2024-03-31T22:58:50.593 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/198 2024-03-31T22:58:50.764 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 31/198 2024-03-31T22:58:50.802 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 32/198 2024-03-31T22:58:50.875 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 33/198 2024-03-31T22:58:50.909 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 34/198 2024-03-31T22:58:50.952 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 35/198 2024-03-31T22:58:50.987 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 36/198 2024-03-31T22:58:51.013 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : lua-5.4.4-4.el9.x86_64 37/198 2024-03-31T22:58:51.075 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 38/198 2024-03-31T22:58:51.109 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-ceph-common-2:19.0.0-1977.g384f31f3.el9. 39/198 2024-03-31T22:58:51.152 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-ceph-argparse-2:19.0.0-1977.g384f31f3.el 40/198 2024-03-31T22:58:51.181 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-cephfs-2:19.0.0-1977.g384f31f3.el9.x86_6 41/198 2024-03-31T22:58:51.251 INFO:teuthology.orchestra.run.smithi162.stdout:(10/174): ceph-radosgw-19.0.0-1977.g384f31f3.el 6.5 MB/s | 9.8 MB 00:01 2024-03-31T22:58:51.253 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 42/198 2024-03-31T22:58:51.296 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : luarocks-3.9.2-1.el9.noarch 43/198 2024-03-31T22:58:51.372 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 44/198 2024-03-31T22:58:51.418 INFO:teuthology.orchestra.run.smithi162.stdout:(11/174): ceph-osd-19.0.0-1977.g384f31f3.el9.x8 8.2 MB/s | 17 MB 00:02 2024-03-31T22:58:51.443 INFO:teuthology.orchestra.run.smithi162.stdout:(12/174): cephfs-mirror-19.0.0-1977.g384f31f3.e 1.1 MB/s | 221 kB 00:00 2024-03-31T22:58:51.477 INFO:teuthology.orchestra.run.smithi162.stdout:(13/174): cephfs-shell-19.0.0-1977.g384f31f3.el 463 kB/s | 27 kB 00:00 2024-03-31T22:58:51.502 INFO:teuthology.orchestra.run.smithi162.stdout:(14/174): libcephfs-devel-19.0.0-1977.g384f31f3 544 kB/s | 32 kB 00:00 2024-03-31T22:58:51.508 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 45/198 2024-03-31T22:58:51.557 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 46/198 2024-03-31T22:58:51.586 INFO:teuthology.orchestra.run.smithi162.stdout:(15/174): libcephsqlite-19.0.0-1977.g384f31f3.e 1.9 MB/s | 163 kB 00:00 2024-03-31T22:58:51.620 INFO:teuthology.orchestra.run.smithi162.stdout:(16/174): libcephfs2-19.0.0-1977.g384f31f3.el9. 5.1 MB/s | 746 kB 00:00 2024-03-31T22:58:51.653 INFO:teuthology.orchestra.run.smithi162.stdout:(17/174): librados-devel-19.0.0-1977.g384f31f3. 1.9 MB/s | 127 kB 00:00 2024-03-31T22:58:51.720 INFO:teuthology.orchestra.run.smithi162.stdout:(18/174): libradosstriper1-19.0.0-1977.g384f31f 4.9 MB/s | 501 kB 00:00 2024-03-31T22:58:51.795 INFO:teuthology.orchestra.run.smithi162.stdout:(19/174): python3-ceph-argparse-19.0.0-1977.g38 603 kB/s | 45 kB 00:00 2024-03-31T22:58:51.879 INFO:teuthology.orchestra.run.smithi162.stdout:(20/174): python3-ceph-common-19.0.0-1977.g384f 1.5 MB/s | 131 kB 00:00 2024-03-31T22:58:51.962 INFO:teuthology.orchestra.run.smithi162.stdout:(21/174): python3-cephfs-19.0.0-1977.g384f31f3. 1.9 MB/s | 161 kB 00:00 2024-03-31T22:58:52.062 INFO:teuthology.orchestra.run.smithi162.stdout:(22/174): python3-rados-19.0.0-1977.g384f31f3.e 3.1 MB/s | 320 kB 00:00 2024-03-31T22:58:52.112 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-rsa-4.9-2.el9.noarch 47/198 2024-03-31T22:58:52.171 INFO:teuthology.orchestra.run.smithi162.stdout:(23/174): python3-rbd-19.0.0-1977.g384f31f3.el9 2.7 MB/s | 298 kB 00:00 2024-03-31T22:58:52.195 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 48/198 2024-03-31T22:58:52.246 INFO:teuthology.orchestra.run.smithi162.stdout:(24/174): python3-rgw-19.0.0-1977.g384f31f3.el9 1.3 MB/s | 99 kB 00:00 2024-03-31T22:58:52.283 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 49/198 2024-03-31T22:58:52.329 INFO:teuthology.orchestra.run.smithi046.stdout: Upgrading : util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-31T22:58:52.330 INFO:teuthology.orchestra.run.smithi162.stdout:(25/174): rbd-fuse-19.0.0-1977.g384f31f3.el9.x8 1.0 MB/s | 85 kB 00:00 2024-03-31T22:58:52.339 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-31T22:58:52.370 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 51/198 2024-03-31T22:58:52.414 INFO:teuthology.orchestra.run.smithi162.stdout:(26/174): librgw2-19.0.0-1977.g384f31f3.el9.x86 6.7 MB/s | 5.1 MB 00:00 2024-03-31T22:58:52.418 INFO:teuthology.orchestra.run.smithi046.stdout: Upgrading : elfutils-libs-0.190-2.el9.x86_64 52/198 2024-03-31T22:58:52.457 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : librados-devel-2:19.0.0-1977.g384f31f3.el9.x86_6 53/198 2024-03-31T22:58:52.487 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 54/198 2024-03-31T22:58:52.514 INFO:teuthology.orchestra.run.smithi162.stdout:(27/174): rbd-nbd-19.0.0-1977.g384f31f3.el9.x86 1.7 MB/s | 171 kB 00:00 2024-03-31T22:58:52.606 INFO:teuthology.orchestra.run.smithi162.stdout:(28/174): ceph-grafana-dashboards-19.0.0-1977.g 344 kB/s | 31 kB 00:00 2024-03-31T22:58:52.679 INFO:teuthology.orchestra.run.smithi142.stdout: Preparing : 1/1 2024-03-31T22:58:52.681 INFO:teuthology.orchestra.run.smithi162.stdout:(29/174): rbd-mirror-19.0.0-1977.g384f31f3.el9. 8.8 MB/s | 3.1 MB 00:00 2024-03-31T22:58:52.706 INFO:teuthology.orchestra.run.smithi162.stdout:(30/174): ceph-mgr-cephadm-19.0.0-1977.g384f31f 1.4 MB/s | 139 kB 00:00 2024-03-31T22:58:52.750 INFO:teuthology.orchestra.run.smithi142.stdout: Upgrading : openssl-libs-1:3.0.7-27.el9.x86_64 1/198 2024-03-31T22:58:52.756 INFO:teuthology.orchestra.run.smithi046.stdout: Upgrading : libfdisk-2.37.4-18.el9.x86_64 55/198 2024-03-31T22:58:52.783 INFO:teuthology.orchestra.run.smithi142.stdout: Upgrading : libuuid-2.37.4-18.el9.x86_64 2/198 2024-03-31T22:58:52.820 INFO:teuthology.orchestra.run.smithi142.stdout: Upgrading : libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-31T22:58:52.830 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-31T22:58:52.879 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 4/198 2024-03-31T22:58:52.914 INFO:teuthology.orchestra.run.smithi046.stdout: Upgrading : util-linux-2.37.4-18.el9.x86_64 56/198 2024-03-31T22:58:52.934 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 5/198 2024-03-31T22:58:52.974 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/198 2024-03-31T22:58:52.981 INFO:teuthology.orchestra.run.smithi046.stdout: Upgrading : xfsprogs-6.3.0-1.el9.x86_64 57/198 2024-03-31T22:58:52.999 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 7/198 2024-03-31T22:58:53.026 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 58/198 2024-03-31T22:58:53.091 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 8/198 2024-03-31T22:58:53.124 INFO:teuthology.orchestra.run.smithi162.stdout:(31/174): ceph-mgr-dashboard-19.0.0-1977.g384f3 7.9 MB/s | 3.5 MB 00:00 2024-03-31T22:58:53.248 INFO:teuthology.orchestra.run.smithi162.stdout:(32/174): ceph-mgr-modules-core-19.0.0-1977.g38 1.9 MB/s | 248 kB 00:00 2024-03-31T22:58:53.340 INFO:teuthology.orchestra.run.smithi162.stdout:(33/174): ceph-mgr-rook-19.0.0-1977.g384f31f3.e 538 kB/s | 49 kB 00:00 2024-03-31T22:58:53.516 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 9/198 2024-03-31T22:58:53.531 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 59/198 2024-03-31T22:58:53.567 INFO:teuthology.orchestra.run.smithi142.stdout: Upgrading : librados2-2:19.0.0-1977.g384f31f3.el9.x86_64 10/198 2024-03-31T22:58:53.605 INFO:teuthology.orchestra.run.smithi046.stdout: Upgrading : openssl-1:3.0.7-27.el9.x86_64 60/198 2024-03-31T22:58:53.662 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 61/198 2024-03-31T22:58:53.688 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: librados2-2:19.0.0-1977.g384f31f3.el9.x86_64 10/198 2024-03-31T22:58:53.718 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : libcephfs2-2:19.0.0-1977.g384f31f3.el9.x86_64 11/198 2024-03-31T22:58:53.735 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 62/198 2024-03-31T22:58:53.766 INFO:teuthology.orchestra.run.smithi162.stdout:(34/174): ceph-test-19.0.0-1977.g384f31f3.el9.x 14 MB/s | 47 MB 00:03 2024-03-31T22:58:53.779 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 63/198 2024-03-31T22:58:53.791 INFO:teuthology.orchestra.run.smithi162.stdout:(35/174): ceph-prometheus-alerts-19.0.0-1977.g3 34 kB/s | 15 kB 00:00 2024-03-31T22:58:53.809 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: libcephfs2-2:19.0.0-1977.g384f31f3.el9.x86_64 11/198 2024-03-31T22:58:53.833 INFO:teuthology.orchestra.run.smithi162.stdout:(36/174): ceph-volume-19.0.0-1977.g384f31f3.el9 3.9 MB/s | 264 kB 00:00 2024-03-31T22:58:53.833 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 64/198 2024-03-31T22:58:53.848 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-rados-2:19.0.0-1977.g384f31f3.el9.x86_64 12/198 2024-03-31T22:58:53.881 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 65/198 2024-03-31T22:58:53.895 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 13/198 2024-03-31T22:58:53.916 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 66/198 2024-03-31T22:58:53.917 INFO:teuthology.orchestra.run.smithi162.stdout:(37/174): ceph-mgr-diskprediction-local-19.0.0- 6.1 MB/s | 7.4 MB 00:01 2024-03-31T22:58:53.942 INFO:teuthology.orchestra.run.smithi162.stdout:(38/174): cephfs-top-19.0.0-1977.g384f31f3.el9. 168 kB/s | 18 kB 00:00 2024-03-31T22:58:53.960 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 67/198 2024-03-31T22:58:53.984 INFO:teuthology.orchestra.run.smithi162.stdout:(39/174): cephadm-19.0.0-1977.g384f31f3.el9.noa 3.8 MB/s | 748 kB 00:00 2024-03-31T22:58:54.001 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 68/198 2024-03-31T22:58:54.088 INFO:teuthology.orchestra.run.smithi142.stdout: Upgrading : elfutils-libelf-0.190-2.el9.x86_64 14/198 2024-03-31T22:58:54.131 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : postgresql-13.14-1.el9.x86_64 15/198 2024-03-31T22:58:54.140 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 69/198 2024-03-31T22:58:54.142 INFO:teuthology.orchestra.run.smithi162.stdout:(40/174): boost-program-options-1.75.0-8.el9.x8 998 kB/s | 106 kB 00:00 2024-03-31T22:58:54.159 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : libcephsqlite-2:19.0.0-1977.g384f31f3.el9.x86_64 16/198 2024-03-31T22:58:54.184 INFO:teuthology.orchestra.run.smithi162.stdout:(41/174): bison-3.7.4-5.el9.x86_64.rpm 3.7 MB/s | 920 kB 00:00 2024-03-31T22:58:54.190 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 70/198 2024-03-31T22:58:54.210 INFO:teuthology.orchestra.run.smithi162.stdout:(42/174): automake-1.16.2-8.el9.noarch.rpm 2.2 MB/s | 662 kB 00:00 2024-03-31T22:58:54.220 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 71/198 2024-03-31T22:58:54.235 INFO:teuthology.orchestra.run.smithi162.stdout:(43/174): flex-2.6.4-9.el9.x86_64.rpm 3.3 MB/s | 307 kB 00:00 2024-03-31T22:58:54.260 INFO:teuthology.orchestra.run.smithi162.stdout:(44/174): flexiblas-3.0.4-8.el9.x86_64.rpm 423 kB/s | 32 kB 00:00 2024-03-31T22:58:54.262 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 72/198 2024-03-31T22:58:54.263 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1977.g384f31f3.el9.x86_64 16/198 2024-03-31T22:58:54.285 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : libradosstriper1-2:19.0.0-1977.g384f31f3.el9.x86 17/198 2024-03-31T22:58:54.334 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1977.g384f31f3.el9.x86 17/198 2024-03-31T22:58:54.352 INFO:teuthology.orchestra.run.smithi162.stdout:(45/174): libaio-devel-0.3.111-13.el9.x86_64.rp 111 kB/s | 10 kB 00:00 2024-03-31T22:58:54.359 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 73/198 2024-03-31T22:58:54.372 INFO:teuthology.orchestra.run.smithi142.stdout: Upgrading : libmount-2.37.4-18.el9.x86_64 18/198 2024-03-31T22:58:54.377 INFO:teuthology.orchestra.run.smithi162.stdout:(46/174): flexiblas-openblas-openmp-3.0.4-8.el9 147 kB/s | 17 kB 00:00 2024-03-31T22:58:54.394 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 74/198 2024-03-31T22:58:54.395 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 19/198 2024-03-31T22:58:54.411 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 19/198 2024-03-31T22:58:54.426 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 75/198 2024-03-31T22:58:54.444 INFO:teuthology.orchestra.run.smithi162.stdout:(47/174): flexiblas-netlib-3.0.4-8.el9.x86_64.r 13 MB/s | 3.0 MB 00:00 2024-03-31T22:58:54.460 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 76/198 2024-03-31T22:58:54.469 INFO:teuthology.orchestra.run.smithi162.stdout:(48/174): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.3 MB/s | 159 kB 00:00 2024-03-31T22:58:54.485 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 77/198 2024-03-31T22:58:54.494 INFO:teuthology.orchestra.run.smithi162.stdout:(49/174): librabbitmq-0.11.0-7.el9.x86_64.rpm 380 kB/s | 44 kB 00:00 2024-03-31T22:58:54.553 INFO:teuthology.orchestra.run.smithi162.stdout:(50/174): librdkafka-1.6.1-102.el9.x86_64.rpm 7.7 MB/s | 661 kB 00:00 2024-03-31T22:58:54.564 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 78/198 2024-03-31T22:58:54.578 INFO:teuthology.orchestra.run.smithi162.stdout:(51/174): libtool-2.4.6-45.el9.x86_64.rpm 5.2 MB/s | 578 kB 00:00 2024-03-31T22:58:54.604 INFO:teuthology.orchestra.run.smithi162.stdout:(52/174): libzstd-devel-1.5.1-2.el9.x86_64.rpm 426 kB/s | 46 kB 00:00 2024-03-31T22:58:54.604 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 20/198 2024-03-31T22:58:54.629 INFO:teuthology.orchestra.run.smithi162.stdout:(53/174): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.7 MB/s | 282 kB 00:00 2024-03-31T22:58:54.635 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 20/198 2024-03-31T22:58:54.666 INFO:teuthology.orchestra.run.smithi142.stdout: Upgrading : libsmartcols-2.37.4-18.el9.x86_64 21/198 2024-03-31T22:58:54.671 INFO:teuthology.orchestra.run.smithi162.stdout:(54/174): oniguruma-6.9.6-1.el9.5.x86_64.rpm 3.2 MB/s | 217 kB 00:00 2024-03-31T22:58:54.696 INFO:teuthology.orchestra.run.smithi162.stdout:(55/174): lua-5.4.4-4.el9.x86_64.rpm 2.0 MB/s | 187 kB 00:00 2024-03-31T22:58:54.699 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 79/198 2024-03-31T22:58:54.713 INFO:teuthology.orchestra.run.smithi162.stdout:(56/174): openblas-0.3.21-2.el9.x86_64.rpm 408 kB/s | 34 kB 00:00 2024-03-31T22:58:54.809 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 80/198 2024-03-31T22:58:54.845 INFO:teuthology.orchestra.run.smithi142.stdout: Upgrading : libacl-2.3.1-4.el9.x86_64 22/198 2024-03-31T22:58:54.852 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 81/198 2024-03-31T22:58:54.888 INFO:teuthology.orchestra.run.smithi162.stdout:(57/174): openblas-openmp-0.3.21-2.el9.x86_64.r 22 MB/s | 4.7 MB 00:00 2024-03-31T22:58:54.902 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-ply-3.11-14.el9.noarch 82/198 2024-03-31T22:58:54.930 INFO:teuthology.orchestra.run.smithi162.stdout:(58/174): postgresql-13.14-1.el9.x86_64.rpm 7.1 MB/s | 1.5 MB 00:00 2024-03-31T22:58:54.955 INFO:teuthology.orchestra.run.smithi162.stdout:(59/174): postgresql-contrib-13.14-1.el9.x86_64 3.3 MB/s | 817 kB 00:00 2024-03-31T22:58:54.965 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 83/198 2024-03-31T22:58:54.968 INFO:teuthology.orchestra.run.smithi142.stdout: Upgrading : NetworkManager-libnm-1:1.47.2-1.el9.x86_64 23/198 2024-03-31T22:58:54.980 INFO:teuthology.orchestra.run.smithi162.stdout:(60/174): postgresql-private-libs-13.14-1.el9.x 1.4 MB/s | 132 kB 00:00 2024-03-31T22:58:55.028 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.2-1.el9.x86_64 23/198 2024-03-31T22:58:55.081 INFO:teuthology.orchestra.run.smithi162.stdout:(61/174): protobuf-3.14.0-13.el9.x86_64.rpm 8.1 MB/s | 1.0 MB 00:00 2024-03-31T22:58:55.124 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-31T22:58:55.157 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 84/198 2024-03-31T22:58:55.171 INFO:teuthology.orchestra.run.smithi142.stdout: Upgrading : NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-31T22:58:55.256 INFO:teuthology.orchestra.run.smithi162.stdout:(62/174): postgresql-server-13.14-1.el9.x86_64. 19 MB/s | 5.7 MB 00:00 2024-03-31T22:58:55.281 INFO:teuthology.orchestra.run.smithi162.stdout:(63/174): python3-attrs-20.3.0-7.el9.noarch.rpm 258 kB/s | 78 kB 00:00 2024-03-31T22:58:55.402 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-31T22:58:55.409 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 85/198 2024-03-31T22:58:55.440 INFO:teuthology.orchestra.run.smithi162.stdout:(64/174): python3-babel-2.9.1-2.el9.noarch.rpm 16 MB/s | 5.8 MB 00:00 2024-03-31T22:58:55.465 INFO:teuthology.orchestra.run.smithi162.stdout:(65/174): python3-jsonpatch-1.21-16.el9.noarch. 119 kB/s | 24 kB 00:00 2024-03-31T22:58:55.490 INFO:teuthology.orchestra.run.smithi162.stdout:(66/174): python3-jsonpointer-2.0-4.el9.noarch. 88 kB/s | 18 kB 00:00 2024-03-31T22:58:55.524 INFO:teuthology.orchestra.run.smithi162.stdout:(67/174): python3-markupsafe-1.1.1-12.el9.x86_6 544 kB/s | 32 kB 00:00 2024-03-31T22:58:55.524 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : re2-1:20211101-3.el9.x86_64 25/198 2024-03-31T22:58:55.549 INFO:teuthology.orchestra.run.smithi162.stdout:(68/174): python3-mako-1.1.4-6.el9.noarch.rpm 1.7 MB/s | 144 kB 00:00 2024-03-31T22:58:55.615 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 26/198 2024-03-31T22:58:55.652 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 27/198 2024-03-31T22:58:55.679 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : liboath-2.6.7-2.el9.x86_64 28/198 2024-03-31T22:58:55.714 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 86/198 2024-03-31T22:58:55.731 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/198 2024-03-31T22:58:55.746 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 87/198 2024-03-31T22:58:55.758 INFO:teuthology.orchestra.run.smithi162.stdout:(69/174): python3-numpy-1.20.1-5.el9.x86_64.rpm 18 MB/s | 4.8 MB 00:00 2024-03-31T22:58:55.783 INFO:teuthology.orchestra.run.smithi162.stdout:(70/174): python3-numpy-f2py-1.20.1-5.el9.x86_6 842 kB/s | 217 kB 00:00 2024-03-31T22:58:55.786 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 88/198 2024-03-31T22:58:55.805 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/198 2024-03-31T22:58:55.808 INFO:teuthology.orchestra.run.smithi162.stdout:(71/174): python3-oauthlib-3.1.1-5.el9.noarch.r 680 kB/s | 174 kB 00:00 2024-03-31T22:58:55.816 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : fuse-2.9.9-15.el9.x86_64 89/198 2024-03-31T22:58:55.844 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : zlib-devel-1.2.11-40.el9.x86_64 90/198 2024-03-31T22:58:55.859 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 31/198 2024-03-31T22:58:55.867 INFO:teuthology.orchestra.run.smithi162.stdout:(72/174): python3-packaging-20.9-5.el9.noarch.r 1.2 MB/s | 69 kB 00:00 2024-03-31T22:58:55.879 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 91/198 2024-03-31T22:58:55.892 INFO:teuthology.orchestra.run.smithi162.stdout:(73/174): python3-prettytable-0.7.2-27.el9.noar 496 kB/s | 41 kB 00:00 2024-03-31T22:58:55.897 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 32/198 2024-03-31T22:58:55.918 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : uuid-1.6.2-55.el9.x86_64 92/198 2024-03-31T22:58:55.925 INFO:teuthology.orchestra.run.smithi162.stdout:(74/174): python3-protobuf-3.14.0-13.el9.noarch 2.0 MB/s | 238 kB 00:00 2024-03-31T22:58:55.943 INFO:teuthology.orchestra.run.smithi162.stdout:(75/174): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.7 MB/s | 132 kB 00:00 2024-03-31T22:58:55.950 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 93/198 2024-03-31T22:58:55.962 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 33/198 2024-03-31T22:58:55.968 INFO:teuthology.orchestra.run.smithi162.stdout:(76/174): python3-pytz-2021.1-5.el9.noarch.rpm 1.1 MB/s | 47 kB 00:00 2024-03-31T22:58:55.980 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 94/198 2024-03-31T22:58:55.993 INFO:teuthology.orchestra.run.smithi162.stdout:(77/174): python3-pyasn1-modules-0.4.8-6.el9.no 3.1 MB/s | 211 kB 00:00 2024-03-31T22:58:55.996 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 34/198 2024-03-31T22:58:56.018 INFO:teuthology.orchestra.run.smithi162.stdout:(78/174): python3-requests-oauthlib-1.3.0-12.el 568 kB/s | 43 kB 00:00 2024-03-31T22:58:56.040 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 35/198 2024-03-31T22:58:56.067 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 95/198 2024-03-31T22:58:56.073 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 36/198 2024-03-31T22:58:56.098 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 96/198 2024-03-31T22:58:56.100 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : lua-5.4.4-4.el9.x86_64 37/198 2024-03-31T22:58:56.132 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 97/198 2024-03-31T22:58:56.169 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 38/198 2024-03-31T22:58:56.174 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 98/198 2024-03-31T22:58:56.205 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-ceph-common-2:19.0.0-1977.g384f31f3.el9. 39/198 2024-03-31T22:58:56.208 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 99/198 2024-03-31T22:58:56.248 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-ceph-argparse-2:19.0.0-1977.g384f31f3.el 40/198 2024-03-31T22:58:56.277 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-cephfs-2:19.0.0-1977.g384f31f3.el9.x86_6 41/198 2024-03-31T22:58:56.302 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 100/198 2024-03-31T22:58:56.345 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 42/198 2024-03-31T22:58:56.391 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 101/198 2024-03-31T22:58:56.392 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : luarocks-3.9.2-1.el9.noarch 43/198 2024-03-31T22:58:56.440 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 102/198 2024-03-31T22:58:56.465 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 44/198 2024-03-31T22:58:56.470 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 103/198 2024-03-31T22:58:56.518 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 104/198 2024-03-31T22:58:56.577 INFO:teuthology.orchestra.run.smithi162.stdout:(79/174): python3-wcwidth-0.2.5-8.el9.noarch.rp 73 kB/s | 41 kB 00:00 2024-03-31T22:58:56.601 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 45/198 2024-03-31T22:58:56.602 INFO:teuthology.orchestra.run.smithi162.stdout:(80/174): python3-toml-0.10.2-6.el9.noarch.rpm 66 kB/s | 39 kB 00:00 2024-03-31T22:58:56.618 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-packaging-20.9-5.el9.noarch 105/198 2024-03-31T22:58:56.652 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 46/198 2024-03-31T22:58:56.672 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 106/198 2024-03-31T22:58:56.844 INFO:teuthology.orchestra.run.smithi162.stdout:(81/174): python3-scipy-1.6.2-8.el9.x86_64.rpm 18 MB/s | 16 MB 00:00 2024-03-31T22:58:56.869 INFO:teuthology.orchestra.run.smithi162.stdout:(82/174): qatlib-24.02.0-1.el9.x86_64.rpm 753 kB/s | 220 kB 00:00 2024-03-31T22:58:56.894 INFO:teuthology.orchestra.run.smithi162.stdout:(83/174): qatlib-service-24.02.0-1.el9.x86_64.r 120 kB/s | 35 kB 00:00 2024-03-31T22:58:56.920 INFO:teuthology.orchestra.run.smithi162.stdout:(84/174): qatzip-libs-1.2.0-1.el9.x86_64.rpm 621 kB/s | 46 kB 00:00 2024-03-31T22:58:56.945 INFO:teuthology.orchestra.run.smithi162.stdout:(85/174): uuid-1.6.2-55.el9.x86_64.rpm 1.1 MB/s | 56 kB 00:00 2024-03-31T22:58:56.970 INFO:teuthology.orchestra.run.smithi162.stdout:(86/174): socat-1.7.4.1-5.el9.x86_64.rpm 3.9 MB/s | 300 kB 00:00 2024-03-31T22:58:56.995 INFO:teuthology.orchestra.run.smithi162.stdout:(87/174): xmlstarlet-1.6.1-20.el9.x86_64.rpm 835 kB/s | 63 kB 00:00 2024-03-31T22:58:57.021 INFO:teuthology.orchestra.run.smithi162.stdout:(88/174): zlib-devel-1.2.11-40.el9.x86_64.rpm 587 kB/s | 44 kB 00:00 2024-03-31T22:58:57.138 INFO:teuthology.orchestra.run.smithi162.stdout:(89/174): fuse-2.9.9-15.el9.x86_64.rpm 471 kB/s | 79 kB 00:00 2024-03-31T22:58:57.163 INFO:teuthology.orchestra.run.smithi162.stdout:(90/174): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.4 MB/s | 241 kB 00:00 2024-03-31T22:58:57.177 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-rsa-4.9-2.el9.noarch 47/198 2024-03-31T22:58:57.205 INFO:teuthology.orchestra.run.smithi162.stdout:(91/174): python3-cryptography-36.0.1-4.el9.x86 6.2 MB/s | 1.1 MB 00:00 2024-03-31T22:58:57.230 INFO:teuthology.orchestra.run.smithi162.stdout:(92/174): python3-ply-3.11-14.el9.noarch.rpm 1.6 MB/s | 103 kB 00:00 2024-03-31T22:58:57.255 INFO:teuthology.orchestra.run.smithi162.stdout:(93/174): python3-pycparser-2.20-6.el9.noarch.r 1.3 MB/s | 124 kB 00:00 2024-03-31T22:58:57.255 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 48/198 2024-03-31T22:58:57.280 INFO:teuthology.orchestra.run.smithi162.stdout:(94/174): python3-urllib3-1.26.5-5.el9.noarch.r 2.4 MB/s | 187 kB 00:00 2024-03-31T22:58:57.341 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 49/198 2024-03-31T22:58:57.384 INFO:teuthology.orchestra.run.smithi142.stdout: Upgrading : util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-31T22:58:57.394 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-31T22:58:57.425 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 51/198 2024-03-31T22:58:57.440 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 107/198 2024-03-31T22:58:57.465 INFO:teuthology.orchestra.run.smithi142.stdout: Upgrading : elfutils-libs-0.190-2.el9.x86_64 52/198 2024-03-31T22:58:57.487 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 108/198 2024-03-31T22:58:57.503 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : librados-devel-2:19.0.0-1977.g384f31f3.el9.x86_6 53/198 2024-03-31T22:58:57.521 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 109/198 2024-03-31T22:58:57.533 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 54/198 2024-03-31T22:58:57.564 INFO:teuthology.orchestra.run.smithi162.stdout:(95/174): jq-1.6-16.el9.x86_64.rpm 608 kB/s | 187 kB 00:00 2024-03-31T22:58:57.569 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 110/198 2024-03-31T22:58:57.623 INFO:teuthology.orchestra.run.smithi162.stdout:(96/174): libquadmath-11.4.1-3.el9.x86_64.rpm 572 kB/s | 195 kB 00:00 2024-03-31T22:58:57.648 INFO:teuthology.orchestra.run.smithi162.stdout:(97/174): python3-requests-2.25.1-8.el9.noarch. 1.5 MB/s | 125 kB 00:00 2024-03-31T22:58:57.659 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 111/198 2024-03-31T22:58:57.688 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 112/198 2024-03-31T22:58:57.690 INFO:teuthology.orchestra.run.smithi162.stdout:(98/174): libgfortran-11.4.1-3.el9.x86_64.rpm 1.8 MB/s | 803 kB 00:00 2024-03-31T22:58:57.794 INFO:teuthology.orchestra.run.smithi142.stdout: Upgrading : libfdisk-2.37.4-18.el9.x86_64 55/198 2024-03-31T22:58:57.865 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : openblas-0.3.21-2.el9.x86_64 113/198 2024-03-31T22:58:57.902 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 114/198 2024-03-31T22:58:57.957 INFO:teuthology.orchestra.run.smithi162.stdout:(99/174): elfutils-libelf-devel-0.190-2.el9.x86 64 kB/s | 21 kB 00:00 2024-03-31T22:58:57.974 INFO:teuthology.orchestra.run.smithi162.stdout:(100/174): libuuid-devel-2.37.4-18.el9.x86_64.r 78 kB/s | 25 kB 00:00 2024-03-31T22:58:57.982 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 115/198 2024-03-31T22:58:57.984 INFO:teuthology.orchestra.run.smithi142.stdout: Upgrading : util-linux-2.37.4-18.el9.x86_64 56/198 2024-03-31T22:58:58.060 INFO:teuthology.orchestra.run.smithi142.stdout: Upgrading : xfsprogs-6.3.0-1.el9.x86_64 57/198 2024-03-31T22:58:58.104 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 58/198 2024-03-31T22:58:58.250 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 59/198 2024-03-31T22:58:58.314 INFO:teuthology.orchestra.run.smithi142.stdout: Upgrading : openssl-1:3.0.7-27.el9.x86_64 60/198 2024-03-31T22:58:58.315 INFO:teuthology.orchestra.run.smithi162.stdout:(101/174): python3-devel-3.9.17-2.el9.x86_64.rp 684 kB/s | 245 kB 00:00 2024-03-31T22:58:58.341 INFO:teuthology.orchestra.run.smithi162.stdout:(102/174): python3-jinja2-2.11.3-5.el9.noarch.r 676 kB/s | 248 kB 00:00 2024-03-31T22:58:58.370 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 61/198 2024-03-31T22:58:58.442 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 62/198 2024-03-31T22:58:58.558 INFO:teuthology.orchestra.run.smithi162.stdout:(103/174): lua-devel-5.4.4-4.el9.x86_64.rpm 103 kB/s | 22 kB 00:00 2024-03-31T22:58:58.666 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 63/198 2024-03-31T22:58:58.669 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 116/198 2024-03-31T22:58:58.703 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 64/198 2024-03-31T22:58:58.751 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 65/198 2024-03-31T22:58:58.785 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 66/198 2024-03-31T22:58:58.797 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 117/198 2024-03-31T22:58:58.825 INFO:teuthology.orchestra.run.smithi162.stdout:(104/174): xfsprogs-devel-6.3.0-1.el9.x86_64.rp 184 kB/s | 94 kB 00:00 2024-03-31T22:58:58.828 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 67/198 2024-03-31T22:58:58.871 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 68/198 2024-03-31T22:58:58.958 INFO:teuthology.orchestra.run.smithi162.stdout:(105/174): openssl-devel-3.0.7-27.el9.x86_64.rp 3.3 MB/s | 4.1 MB 00:01 2024-03-31T22:58:59.014 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 69/198 2024-03-31T22:58:59.067 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 70/198 2024-03-31T22:58:59.097 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 71/198 2024-03-31T22:58:59.117 INFO:teuthology.orchestra.run.smithi162.stdout:(106/174): protobuf-compiler-3.14.0-13.el9.x86_ 1.5 MB/s | 863 kB 00:00 2024-03-31T22:58:59.139 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 72/198 2024-03-31T22:58:59.234 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 73/198 2024-03-31T22:58:59.272 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 74/198 2024-03-31T22:58:59.275 INFO:teuthology.orchestra.run.smithi162.stdout:(107/174): abseil-cpp-20211102.0-3.el9.x86_64.r 1.2 MB/s | 548 kB 00:00 2024-03-31T22:58:59.312 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 75/198 2024-03-31T22:58:59.334 INFO:teuthology.orchestra.run.smithi162.stdout:(108/174): gperftools-libs-2.9.1-2.el9.x86_64.r 825 kB/s | 309 kB 00:00 2024-03-31T22:58:59.338 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 76/198 2024-03-31T22:58:59.359 INFO:teuthology.orchestra.run.smithi162.stdout:(109/174): grpc-data-1.46.7-10.el9.noarch.rpm 80 kB/s | 19 kB 00:00 2024-03-31T22:58:59.362 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 77/198 2024-03-31T22:58:59.401 INFO:teuthology.orchestra.run.smithi162.stdout:(110/174): libarrow-doc-9.0.0-10.el9.noarch.rpm 389 kB/s | 26 kB 00:00 2024-03-31T22:58:59.441 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 78/198 2024-03-31T22:58:59.477 INFO:teuthology.orchestra.run.smithi162.stdout:(111/174): libunwind-1.6.2-1.el9.x86_64.rpm 897 kB/s | 67 kB 00:00 2024-03-31T22:58:59.502 INFO:teuthology.orchestra.run.smithi162.stdout:(112/174): liboath-2.6.7-2.el9.x86_64.rpm 341 kB/s | 49 kB 00:00 2024-03-31T22:58:59.565 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 79/198 2024-03-31T22:58:59.594 INFO:teuthology.orchestra.run.smithi162.stdout:(113/174): libarrow-9.0.0-10.el9.x86_64.rpm 14 MB/s | 4.4 MB 00:00 2024-03-31T22:58:59.619 INFO:teuthology.orchestra.run.smithi162.stdout:(114/174): luarocks-3.9.2-1.el9.noarch.rpm 1.0 MB/s | 151 kB 00:00 2024-03-31T22:58:59.669 INFO:teuthology.orchestra.run.smithi162.stdout:(115/174): python3-asyncssh-2.13.2-5.el9.noarch 7.1 MB/s | 548 kB 00:00 2024-03-31T22:58:59.681 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 80/198 2024-03-31T22:58:59.694 INFO:teuthology.orchestra.run.smithi162.stdout:(116/174): python3-bcrypt-3.2.2-1.el9.x86_64.rp 577 kB/s | 43 kB 00:00 2024-03-31T22:58:59.721 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 81/198 2024-03-31T22:58:59.736 INFO:teuthology.orchestra.run.smithi162.stdout:(117/174): python3-cachetools-4.2.4-1.el9.noarc 483 kB/s | 32 kB 00:00 2024-03-31T22:58:59.762 INFO:teuthology.orchestra.run.smithi162.stdout:(118/174): python3-certifi-2021.10.8-2.el9.noar 224 kB/s | 15 kB 00:00 2024-03-31T22:58:59.778 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-ply-3.11-14.el9.noarch 82/198 2024-03-31T22:58:59.804 INFO:teuthology.orchestra.run.smithi162.stdout:(119/174): parquet-libs-9.0.0-10.el9.x86_64.rpm 2.7 MB/s | 838 kB 00:00 2024-03-31T22:58:59.829 INFO:teuthology.orchestra.run.smithi162.stdout:(120/174): python3-cheroot-8.6.0-4.el9.noarch.r 1.8 MB/s | 172 kB 00:00 2024-03-31T22:58:59.833 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 83/198 2024-03-31T22:58:59.862 INFO:teuthology.orchestra.run.smithi162.stdout:(121/174): python3-cherrypy-18.6.1-2.el9.noarch 3.5 MB/s | 358 kB 00:00 2024-03-31T22:58:59.888 INFO:teuthology.orchestra.run.smithi162.stdout:(122/174): python3-cmd2-2.4.2-2.el9.noarch.rpm 3.5 MB/s | 303 kB 00:00 2024-03-31T22:58:59.905 INFO:teuthology.orchestra.run.smithi162.stdout:(123/174): python3-colorama-0.4.4-12.el9.noarch 468 kB/s | 35 kB 00:00 2024-03-31T22:58:59.963 INFO:teuthology.orchestra.run.smithi162.stdout:(124/174): python3-google-auth-2.28.2-1.el9.noa 2.0 MB/s | 203 kB 00:00 2024-03-31T22:59:00.014 INFO:teuthology.orchestra.run.smithi162.stdout:(125/174): python3-grpcio-tools-1.46.7-10.el9.x 1.3 MB/s | 144 kB 00:00 2024-03-31T22:59:00.016 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 84/198 2024-03-31T22:59:00.039 INFO:teuthology.orchestra.run.smithi162.stdout:(126/174): python3-jaraco-8.2.1-3.el9.noarch.rp 142 kB/s | 11 kB 00:00 2024-03-31T22:59:00.067 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 85/198 2024-03-31T22:59:00.088 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 118/198 2024-03-31T22:59:00.089 INFO:teuthology.orchestra.run.smithi162.stdout:(127/174): python3-grpcio-1.46.7-10.el9.x86_64. 10 MB/s | 2.0 MB 00:00 2024-03-31T22:59:00.114 INFO:teuthology.orchestra.run.smithi162.stdout:(128/174): python3-jaraco-classes-3.2.1-5.el9.n 176 kB/s | 18 kB 00:00 2024-03-31T22:59:00.140 INFO:teuthology.orchestra.run.smithi162.stdout:(129/174): python3-jaraco-collections-3.0.0-8.e 231 kB/s | 23 kB 00:00 2024-03-31T22:59:00.162 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 119/198 2024-03-31T22:59:00.165 INFO:teuthology.orchestra.run.smithi162.stdout:(130/174): python3-jaraco-functools-3.5.0-2.el9 258 kB/s | 19 kB 00:00 2024-03-31T22:59:00.193 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 120/198 2024-03-31T22:59:00.207 INFO:teuthology.orchestra.run.smithi162.stdout:(131/174): python3-jaraco-text-3.2.0-6.el9.noar 295 kB/s | 20 kB 00:00 2024-03-31T22:59:00.219 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 86/198 2024-03-31T22:59:00.224 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : jq-1.6-16.el9.x86_64 121/198 2024-03-31T22:59:00.232 INFO:teuthology.orchestra.run.smithi162.stdout:(132/174): python3-logutils-0.3.5-21.el9.noarch 692 kB/s | 46 kB 00:00 2024-03-31T22:59:00.252 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 122/198 2024-03-31T22:59:00.257 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 87/198 2024-03-31T22:59:00.274 INFO:teuthology.orchestra.run.smithi162.stdout:(133/174): python3-more-itertools-8.12.0-2.el9. 1.2 MB/s | 79 kB 00:00 2024-03-31T22:59:00.288 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 88/198 2024-03-31T22:59:00.308 INFO:teuthology.orchestra.run.smithi162.stdout:(134/174): python3-kubernetes-26.1.0-2.el9.noar 6.1 MB/s | 1.0 MB 00:00 2024-03-31T22:59:00.319 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : fuse-2.9.9-15.el9.x86_64 89/198 2024-03-31T22:59:00.341 INFO:teuthology.orchestra.run.smithi162.stdout:(135/174): python3-natsort-7.1.1-5.el9.noarch.r 531 kB/s | 58 kB 00:00 2024-03-31T22:59:00.347 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : zlib-devel-1.2.11-40.el9.x86_64 90/198 2024-03-31T22:59:00.375 INFO:teuthology.orchestra.run.smithi162.stdout:(136/174): python3-pecan-1.4.2-3.el9.noarch.rpm 2.6 MB/s | 272 kB 00:00 2024-03-31T22:59:00.381 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 91/198 2024-03-31T22:59:00.417 INFO:teuthology.orchestra.run.smithi162.stdout:(137/174): python3-portend-3.1.0-2.el9.noarch.r 150 kB/s | 16 kB 00:00 2024-03-31T22:59:00.429 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : uuid-1.6.2-55.el9.x86_64 92/198 2024-03-31T22:59:00.442 INFO:teuthology.orchestra.run.smithi162.stdout:(138/174): python3-pyOpenSSL-21.0.0-1.el9.noarc 892 kB/s | 90 kB 00:00 2024-03-31T22:59:00.460 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 93/198 2024-03-31T22:59:00.476 INFO:teuthology.orchestra.run.smithi162.stdout:(139/174): python3-pyperclip-1.8.2-2.el9.noarch 280 kB/s | 28 kB 00:00 2024-03-31T22:59:00.490 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 94/198 2024-03-31T22:59:00.501 INFO:teuthology.orchestra.run.smithi162.stdout:(140/174): python3-repoze-lru-0.7-16.el9.noarch 366 kB/s | 31 kB 00:00 2024-03-31T22:59:00.551 INFO:teuthology.orchestra.run.smithi162.stdout:(141/174): python3-routes-2.5.1-5.el9.noarch.rp 1.7 MB/s | 188 kB 00:00 2024-03-31T22:59:00.559 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 95/198 2024-03-31T22:59:00.577 INFO:teuthology.orchestra.run.smithi162.stdout:(142/174): python3-rsa-4.9-2.el9.noarch.rpm 588 kB/s | 59 kB 00:00 2024-03-31T22:59:00.580 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 123/198 2024-03-31T22:59:00.592 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 96/198 2024-03-31T22:59:00.602 INFO:teuthology.orchestra.run.smithi162.stdout:(143/174): python3-tempora-5.0.0-2.el9.noarch.r 356 kB/s | 36 kB 00:00 2024-03-31T22:59:00.615 INFO:teuthology.orchestra.run.smithi046.stdout: Upgrading : librbd1-2:19.0.0-1977.g384f31f3.el9.x86_64 124/198 2024-03-31T22:59:00.626 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 97/198 2024-03-31T22:59:00.644 INFO:teuthology.orchestra.run.smithi162.stdout:(144/174): python3-typing-extensions-4.4.0-2.el 553 kB/s | 51 kB 00:00 2024-03-31T22:59:00.667 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 98/198 2024-03-31T22:59:00.669 INFO:teuthology.orchestra.run.smithi162.stdout:(145/174): python3-webob-1.8.7-6.el9.noarch.rpm 2.4 MB/s | 230 kB 00:00 2024-03-31T22:59:00.694 INFO:teuthology.orchestra.run.smithi162.stdout:(146/174): python3-websocket-client-1.2.3-2.el9 973 kB/s | 90 kB 00:00 2024-03-31T22:59:00.697 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: librbd1-2:19.0.0-1977.g384f31f3.el9.x86_64 124/198 2024-03-31T22:59:00.701 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 99/198 2024-03-31T22:59:00.720 INFO:teuthology.orchestra.run.smithi162.stdout:(147/174): python3-werkzeug-2.0.3-3.el9.1.noarc 5.5 MB/s | 427 kB 00:00 2024-03-31T22:59:00.728 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-rbd-2:19.0.0-1977.g384f31f3.el9.x86_64 125/198 2024-03-31T22:59:00.745 INFO:teuthology.orchestra.run.smithi162.stdout:(148/174): python3-zc-lockfile-2.0-10.el9.noarc 265 kB/s | 20 kB 00:00 2024-03-31T22:59:00.778 INFO:teuthology.orchestra.run.smithi162.stdout:(149/174): re2-20211101-3.el9.x86_64.rpm 2.2 MB/s | 192 kB 00:00 2024-03-31T22:59:00.783 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 126/198 2024-03-31T22:59:00.792 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 100/198 2024-03-31T22:59:00.845 INFO:teuthology.orchestra.run.smithi162.stdout:(150/174): thrift-0.15.0-2.el9.x86_64.rpm 13 MB/s | 1.6 MB 00:00 2024-03-31T22:59:00.881 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 101/198 2024-03-31T22:59:00.924 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 102/198 2024-03-31T22:59:00.964 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 103/198 2024-03-31T22:59:01.011 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 104/198 2024-03-31T22:59:01.037 INFO:teuthology.orchestra.run.smithi162.stdout:(151/174): NetworkManager-1.47.2-1.el9.x86_64.r 14 MB/s | 2.3 MB 00:00 2024-03-31T22:59:01.104 INFO:teuthology.orchestra.run.smithi162.stdout:(152/174): NetworkManager-libnm-1.47.2-1.el9.x8 27 MB/s | 1.8 MB 00:00 2024-03-31T22:59:01.109 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-packaging-20.9-5.el9.noarch 105/198 2024-03-31T22:59:01.129 INFO:teuthology.orchestra.run.smithi162.stdout:(153/174): NetworkManager-team-1.47.2-1.el9.x86 1.6 MB/s | 41 kB 00:00 2024-03-31T22:59:01.168 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 106/198 2024-03-31T22:59:01.171 INFO:teuthology.orchestra.run.smithi162.stdout:(154/174): NetworkManager-tui-1.47.2-1.el9.x86_ 5.8 MB/s | 247 kB 00:00 2024-03-31T22:59:01.213 INFO:teuthology.orchestra.run.smithi162.stdout:(155/174): xfsdump-3.1.12-4.el9.x86_64.rpm 7.6 MB/s | 321 kB 00:00 2024-03-31T22:59:01.260 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 127/198 2024-03-31T22:59:01.284 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : librgw2-2:19.0.0-1977.g384f31f3.el9.x86_64 128/198 2024-03-31T22:59:01.305 INFO:teuthology.orchestra.run.smithi162.stdout:(156/174): acl-2.3.1-4.el9.x86_64.rpm 778 kB/s | 71 kB 00:00 2024-03-31T22:59:01.339 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: librgw2-2:19.0.0-1977.g384f31f3.el9.x86_64 128/198 2024-03-31T22:59:01.347 INFO:teuthology.orchestra.run.smithi162.stdout:(157/174): elfutils-debuginfod-client-0.190-2.e 872 kB/s | 36 kB 00:00 2024-03-31T22:59:01.369 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-rgw-2:19.0.0-1977.g384f31f3.el9.x86_64 129/198 2024-03-31T22:59:01.406 INFO:teuthology.orchestra.run.smithi162.stdout:(158/174): librados2-19.0.0-1977.g384f31f3.el9. 5.1 MB/s | 3.4 MB 00:00 2024-03-31T22:59:01.434 INFO:teuthology.orchestra.run.smithi162.stdout:(159/174): elfutils-libelf-0.190-2.el9.x86_64.r 2.2 MB/s | 192 kB 00:00 2024-03-31T22:59:01.481 INFO:teuthology.orchestra.run.smithi162.stdout:(160/174): libacl-2.3.1-4.el9.x86_64.rpm 462 kB/s | 23 kB 00:00 2024-03-31T22:59:01.532 INFO:teuthology.orchestra.run.smithi162.stdout:(161/174): librbd1-19.0.0-1977.g384f31f3.el9.x8 4.2 MB/s | 3.1 MB 00:00 2024-03-31T22:59:01.557 INFO:teuthology.orchestra.run.smithi162.stdout:(162/174): libblkid-2.37.4-18.el9.x86_64.rpm 1.4 MB/s | 108 kB 00:00 2024-03-31T22:59:01.582 INFO:teuthology.orchestra.run.smithi162.stdout:(163/174): elfutils-libs-0.190-2.el9.x86_64.rpm 1.4 MB/s | 256 kB 00:00 2024-03-31T22:59:01.607 INFO:teuthology.orchestra.run.smithi162.stdout:(164/174): libmount-2.37.4-18.el9.x86_64.rpm 2.6 MB/s | 136 kB 00:00 2024-03-31T22:59:01.633 INFO:teuthology.orchestra.run.smithi162.stdout:(165/174): libsmartcols-2.37.4-18.el9.x86_64.rp 1.2 MB/s | 64 kB 00:00 2024-03-31T22:59:01.674 INFO:teuthology.orchestra.run.smithi162.stdout:(166/174): libuuid-2.37.4-18.el9.x86_64.rpm 428 kB/s | 29 kB 00:00 2024-03-31T22:59:01.708 INFO:teuthology.orchestra.run.smithi162.stdout:(167/174): libfdisk-2.37.4-18.el9.x86_64.rpm 878 kB/s | 155 kB 00:00 2024-03-31T22:59:01.884 INFO:teuthology.orchestra.run.smithi162.stdout:(168/174): openssl-3.0.7-27.el9.x86_64.rpm 4.8 MB/s | 1.2 MB 00:00 2024-03-31T22:59:01.894 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 107/198 2024-03-31T22:59:01.939 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 108/198 2024-03-31T22:59:01.973 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 109/198 2024-03-31T22:59:02.021 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 110/198 2024-03-31T22:59:02.034 INFO:teuthology.orchestra.run.smithi162.stdout:(169/174): openssl-libs-3.0.7-27.el9.x86_64.rpm 6.0 MB/s | 2.2 MB 00:00 2024-03-31T22:59:02.067 INFO:teuthology.orchestra.run.smithi162.stdout:(170/174): util-linux-core-2.37.4-18.el9.x86_64 2.5 MB/s | 465 kB 00:00 2024-03-31T22:59:02.093 INFO:teuthology.orchestra.run.smithi162.stdout:(171/174): util-linux-user-2.37.4-18.el9.x86_64 531 kB/s | 31 kB 00:00 2024-03-31T22:59:02.108 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 111/198 2024-03-31T22:59:02.140 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 112/198 2024-03-31T22:59:02.176 INFO:teuthology.orchestra.run.smithi162.stdout:(172/174): xfsprogs-6.3.0-1.el9.x86_64.rpm 9.9 MB/s | 1.1 MB 00:00 2024-03-31T22:59:02.310 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : openblas-0.3.21-2.el9.x86_64 113/198 2024-03-31T22:59:02.347 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 114/198 2024-03-31T22:59:02.368 INFO:teuthology.orchestra.run.smithi162.stdout:(173/174): libacl-devel-2.3.1-4.el9.x86_64.rpm 270 kB/s | 74 kB 00:00 2024-03-31T22:59:02.418 INFO:teuthology.orchestra.run.smithi162.stdout:(174/174): util-linux-2.37.4-18.el9.x86_64.rpm 3.2 MB/s | 2.3 MB 00:00 2024-03-31T22:59:02.426 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 115/198 2024-03-31T22:59:02.426 INFO:teuthology.orchestra.run.smithi162.stdout:-------------------------------------------------------------------------------- 2024-03-31T22:59:02.426 INFO:teuthology.orchestra.run.smithi162.stdout:Total 15 MB/s | 226 MB 00:14 2024-03-31T22:59:02.793 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction check 2024-03-31T22:59:03.071 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 116/198 2024-03-31T22:59:03.075 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction check succeeded. 2024-03-31T22:59:03.075 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction test 2024-03-31T22:59:03.169 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 117/198 2024-03-31T22:59:03.725 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: ceph-common-2:19.0.0-1977.g384f31f3.el9.x86_64 130/198 2024-03-31T22:59:03.758 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-common-2:19.0.0-1977.g384f31f3.el9.x86_64 130/198 2024-03-31T22:59:04.328 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: ceph-common-2:19.0.0-1977.g384f31f3.el9.x86_64 130/198 2024-03-31T22:59:04.370 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-base-2:19.0.0-1977.g384f31f3.el9.x86_64 131/198 2024-03-31T22:59:04.382 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 118/198 2024-03-31T22:59:04.461 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: ceph-base-2:19.0.0-1977.g384f31f3.el9.x86_64 131/198 2024-03-31T22:59:04.462 INFO:teuthology.orchestra.run.smithi046.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-31T22:59:04.462 INFO:teuthology.orchestra.run.smithi046.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-31T22:59:04.462 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-31T22:59:04.463 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 119/198 2024-03-31T22:59:04.495 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 120/198 2024-03-31T22:59:04.495 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-selinux-2:19.0.0-1977.g384f31f3.el9.x86_64 132/198 2024-03-31T22:59:04.526 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : jq-1.6-16.el9.x86_64 121/198 2024-03-31T22:59:04.553 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 122/198 2024-03-31T22:59:04.866 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 123/198 2024-03-31T22:59:04.891 INFO:teuthology.orchestra.run.smithi142.stdout: Upgrading : librbd1-2:19.0.0-1977.g384f31f3.el9.x86_64 124/198 2024-03-31T22:59:04.982 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: librbd1-2:19.0.0-1977.g384f31f3.el9.x86_64 124/198 2024-03-31T22:59:05.013 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-rbd-2:19.0.0-1977.g384f31f3.el9.x86_64 125/198 2024-03-31T22:59:05.049 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 126/198 2024-03-31T22:59:05.138 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction test succeeded. 2024-03-31T22:59:05.138 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction 2024-03-31T22:59:05.551 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 127/198 2024-03-31T22:59:05.611 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : librgw2-2:19.0.0-1977.g384f31f3.el9.x86_64 128/198 2024-03-31T22:59:05.682 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: librgw2-2:19.0.0-1977.g384f31f3.el9.x86_64 128/198 2024-03-31T22:59:05.705 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-rgw-2:19.0.0-1977.g384f31f3.el9.x86_64 129/198 2024-03-31T22:59:07.911 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: ceph-common-2:19.0.0-1977.g384f31f3.el9.x86_64 130/198 2024-03-31T22:59:07.943 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : ceph-common-2:19.0.0-1977.g384f31f3.el9.x86_64 130/198 2024-03-31T22:59:08.485 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: ceph-common-2:19.0.0-1977.g384f31f3.el9.x86_64 130/198 2024-03-31T22:59:08.521 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : ceph-base-2:19.0.0-1977.g384f31f3.el9.x86_64 131/198 2024-03-31T22:59:08.620 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: ceph-base-2:19.0.0-1977.g384f31f3.el9.x86_64 131/198 2024-03-31T22:59:08.620 INFO:teuthology.orchestra.run.smithi142.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-31T22:59:08.620 INFO:teuthology.orchestra.run.smithi142.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-31T22:59:08.620 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-03-31T22:59:08.655 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : ceph-selinux-2:19.0.0-1977.g384f31f3.el9.x86_64 132/198 2024-03-31T22:59:12.368 INFO:teuthology.orchestra.run.smithi162.stdout: Preparing : 1/1 2024-03-31T22:59:12.481 INFO:teuthology.orchestra.run.smithi162.stdout: Upgrading : openssl-libs-1:3.0.7-27.el9.x86_64 1/198 2024-03-31T22:59:12.515 INFO:teuthology.orchestra.run.smithi162.stdout: Upgrading : libuuid-2.37.4-18.el9.x86_64 2/198 2024-03-31T22:59:12.553 INFO:teuthology.orchestra.run.smithi162.stdout: Upgrading : libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-31T22:59:12.563 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-31T22:59:12.608 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 4/198 2024-03-31T22:59:12.658 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 5/198 2024-03-31T22:59:12.698 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/198 2024-03-31T22:59:12.724 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 7/198 2024-03-31T22:59:12.775 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 8/198 2024-03-31T22:59:13.113 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 9/198 2024-03-31T22:59:13.165 INFO:teuthology.orchestra.run.smithi162.stdout: Upgrading : librados2-2:19.0.0-1977.g384f31f3.el9.x86_64 10/198 2024-03-31T22:59:13.292 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: librados2-2:19.0.0-1977.g384f31f3.el9.x86_64 10/198 2024-03-31T22:59:13.316 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : libcephfs2-2:19.0.0-1977.g384f31f3.el9.x86_64 11/198 2024-03-31T22:59:13.408 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: libcephfs2-2:19.0.0-1977.g384f31f3.el9.x86_64 11/198 2024-03-31T22:59:13.438 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-rados-2:19.0.0-1977.g384f31f3.el9.x86_64 12/198 2024-03-31T22:59:13.483 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 13/198 2024-03-31T22:59:13.660 INFO:teuthology.orchestra.run.smithi162.stdout: Upgrading : elfutils-libelf-0.190-2.el9.x86_64 14/198 2024-03-31T22:59:13.719 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : postgresql-13.14-1.el9.x86_64 15/198 2024-03-31T22:59:13.747 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : libcephsqlite-2:19.0.0-1977.g384f31f3.el9.x86_64 16/198 2024-03-31T22:59:13.878 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1977.g384f31f3.el9.x86_64 16/198 2024-03-31T22:59:13.906 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : libradosstriper1-2:19.0.0-1977.g384f31f3.el9.x86 17/198 2024-03-31T22:59:13.971 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1977.g384f31f3.el9.x86 17/198 2024-03-31T22:59:14.019 INFO:teuthology.orchestra.run.smithi162.stdout: Upgrading : libmount-2.37.4-18.el9.x86_64 18/198 2024-03-31T22:59:14.041 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 19/198 2024-03-31T22:59:14.056 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 19/198 2024-03-31T22:59:14.216 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 20/198 2024-03-31T22:59:14.248 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 20/198 2024-03-31T22:59:14.287 INFO:teuthology.orchestra.run.smithi162.stdout: Upgrading : libsmartcols-2.37.4-18.el9.x86_64 21/198 2024-03-31T22:59:14.484 INFO:teuthology.orchestra.run.smithi162.stdout: Upgrading : libacl-2.3.1-4.el9.x86_64 22/198 2024-03-31T22:59:14.624 INFO:teuthology.orchestra.run.smithi162.stdout: Upgrading : NetworkManager-libnm-1:1.47.2-1.el9.x86_64 23/198 2024-03-31T22:59:14.684 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.2-1.el9.x86_64 23/198 2024-03-31T22:59:14.770 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-31T22:59:14.836 INFO:teuthology.orchestra.run.smithi162.stdout: Upgrading : NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-31T22:59:15.050 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-31T22:59:15.165 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : re2-1:20211101-3.el9.x86_64 25/198 2024-03-31T22:59:15.261 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 26/198 2024-03-31T22:59:15.292 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 27/198 2024-03-31T22:59:15.318 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : liboath-2.6.7-2.el9.x86_64 28/198 2024-03-31T22:59:15.414 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/198 2024-03-31T22:59:15.657 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/198 2024-03-31T22:59:15.708 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 31/198 2024-03-31T22:59:15.754 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 32/198 2024-03-31T22:59:15.826 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 33/198 2024-03-31T22:59:15.893 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 34/198 2024-03-31T22:59:16.079 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 35/198 2024-03-31T22:59:16.294 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 36/198 2024-03-31T22:59:16.323 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : lua-5.4.4-4.el9.x86_64 37/198 2024-03-31T22:59:16.393 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 38/198 2024-03-31T22:59:16.428 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-ceph-common-2:19.0.0-1977.g384f31f3.el9. 39/198 2024-03-31T22:59:16.471 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-ceph-argparse-2:19.0.0-1977.g384f31f3.el 40/198 2024-03-31T22:59:16.500 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-cephfs-2:19.0.0-1977.g384f31f3.el9.x86_6 41/198 2024-03-31T22:59:16.519 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1977.g384f31f3.el9.x86_64 132/198 2024-03-31T22:59:16.519 INFO:teuthology.orchestra.run.smithi046.stdout:skipping the directory /sys 2024-03-31T22:59:16.519 INFO:teuthology.orchestra.run.smithi046.stdout:skipping the directory /proc 2024-03-31T22:59:16.519 INFO:teuthology.orchestra.run.smithi046.stdout:skipping the directory /mnt 2024-03-31T22:59:16.519 INFO:teuthology.orchestra.run.smithi046.stdout:skipping the directory /var/tmp 2024-03-31T22:59:16.520 INFO:teuthology.orchestra.run.smithi046.stdout:skipping the directory /home 2024-03-31T22:59:16.520 INFO:teuthology.orchestra.run.smithi046.stdout:skipping the directory /root 2024-03-31T22:59:16.520 INFO:teuthology.orchestra.run.smithi046.stdout:skipping the directory /tmp 2024-03-31T22:59:16.520 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-31T22:59:16.569 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 42/198 2024-03-31T22:59:16.623 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : luarocks-3.9.2-1.el9.noarch 43/198 2024-03-31T22:59:16.694 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-volume-2:19.0.0-1977.g384f31f3.el9.noarch 133/198 2024-03-31T22:59:16.696 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 44/198 2024-03-31T22:59:16.709 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: ceph-volume-2:19.0.0-1977.g384f31f3.el9.noarch 133/198 2024-03-31T22:59:16.709 INFO:teuthology.orchestra.run.smithi046.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-31T22:59:16.710 INFO:teuthology.orchestra.run.smithi046.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-31T22:59:16.710 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-31T22:59:16.829 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 45/198 2024-03-31T22:59:16.892 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 46/198 2024-03-31T22:59:17.411 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-rsa-4.9-2.el9.noarch 47/198 2024-03-31T22:59:17.496 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 48/198 2024-03-31T22:59:17.588 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 49/198 2024-03-31T22:59:17.640 INFO:teuthology.orchestra.run.smithi162.stdout: Upgrading : util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-31T22:59:17.650 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-31T22:59:17.687 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 51/198 2024-03-31T22:59:17.730 INFO:teuthology.orchestra.run.smithi162.stdout: Upgrading : elfutils-libs-0.190-2.el9.x86_64 52/198 2024-03-31T22:59:17.767 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : librados-devel-2:19.0.0-1977.g384f31f3.el9.x86_6 53/198 2024-03-31T22:59:17.806 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 54/198 2024-03-31T22:59:18.070 INFO:teuthology.orchestra.run.smithi162.stdout: Upgrading : libfdisk-2.37.4-18.el9.x86_64 55/198 2024-03-31T22:59:18.255 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-osd-2:19.0.0-1977.g384f31f3.el9.x86_64 134/198 2024-03-31T22:59:18.280 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: ceph-osd-2:19.0.0-1977.g384f31f3.el9.x86_64 134/198 2024-03-31T22:59:18.280 INFO:teuthology.orchestra.run.smithi046.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-31T22:59:18.280 INFO:teuthology.orchestra.run.smithi046.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-31T22:59:18.280 INFO:teuthology.orchestra.run.smithi046.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-31T22:59:18.280 INFO:teuthology.orchestra.run.smithi046.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-31T22:59:18.280 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-31T22:59:18.282 INFO:teuthology.orchestra.run.smithi162.stdout: Upgrading : util-linux-2.37.4-18.el9.x86_64 56/198 2024-03-31T22:59:18.358 INFO:teuthology.orchestra.run.smithi162.stdout: Upgrading : xfsprogs-6.3.0-1.el9.x86_64 57/198 2024-03-31T22:59:18.402 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 58/198 2024-03-31T22:59:18.513 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-mds-2:19.0.0-1977.g384f31f3.el9.x86_64 135/198 2024-03-31T22:59:18.536 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: ceph-mds-2:19.0.0-1977.g384f31f3.el9.x86_64 135/198 2024-03-31T22:59:18.536 INFO:teuthology.orchestra.run.smithi046.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-31T22:59:18.536 INFO:teuthology.orchestra.run.smithi046.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-31T22:59:18.536 INFO:teuthology.orchestra.run.smithi046.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-31T22:59:18.536 INFO:teuthology.orchestra.run.smithi046.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-31T22:59:18.536 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-31T22:59:18.564 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 59/198 2024-03-31T22:59:18.605 INFO:teuthology.orchestra.run.smithi162.stdout: Upgrading : openssl-1:3.0.7-27.el9.x86_64 60/198 2024-03-31T22:59:18.660 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 61/198 2024-03-31T22:59:18.733 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 62/198 2024-03-31T22:59:18.772 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 63/198 2024-03-31T22:59:18.809 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 64/198 2024-03-31T22:59:18.855 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 65/198 2024-03-31T22:59:18.892 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 66/198 2024-03-31T22:59:18.958 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 67/198 2024-03-31T22:59:18.998 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-mon-2:19.0.0-1977.g384f31f3.el9.x86_64 136/198 2024-03-31T22:59:19.002 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 68/198 2024-03-31T22:59:19.020 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: ceph-mon-2:19.0.0-1977.g384f31f3.el9.x86_64 136/198 2024-03-31T22:59:19.020 INFO:teuthology.orchestra.run.smithi046.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-31T22:59:19.020 INFO:teuthology.orchestra.run.smithi046.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-31T22:59:19.020 INFO:teuthology.orchestra.run.smithi046.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-31T22:59:19.020 INFO:teuthology.orchestra.run.smithi046.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-31T22:59:19.020 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-31T22:59:19.116 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : automake-1.16.2-8.el9.noarch 137/198 2024-03-31T22:59:19.159 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 69/198 2024-03-31T22:59:19.201 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 70/198 2024-03-31T22:59:19.232 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 71/198 2024-03-31T22:59:19.265 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 72/198 2024-03-31T22:59:19.358 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 73/198 2024-03-31T22:59:19.398 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 74/198 2024-03-31T22:59:19.438 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 75/198 2024-03-31T22:59:19.464 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 76/198 2024-03-31T22:59:19.489 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 77/198 2024-03-31T22:59:19.527 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: cephadm-2:19.0.0-1977.g384f31f3.el9.noarch 138/198 2024-03-31T22:59:19.556 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : cephadm-2:19.0.0-1977.g384f31f3.el9.noarch 138/198 2024-03-31T22:59:19.564 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 78/198 2024-03-31T22:59:19.588 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1977.g384f31f3.e 139/198 2024-03-31T22:59:19.650 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1977.g384f31f3. 140/198 2024-03-31T22:59:19.679 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1977.g384f31f3.el9.noa 141/198 2024-03-31T22:59:19.689 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 79/198 2024-03-31T22:59:19.790 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 80/198 2024-03-31T22:59:19.839 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 81/198 2024-03-31T22:59:19.888 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-ply-3.11-14.el9.noarch 82/198 2024-03-31T22:59:19.944 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 83/198 2024-03-31T22:59:20.127 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 84/198 2024-03-31T22:59:20.169 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 85/198 2024-03-31T22:59:20.306 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 86/198 2024-03-31T22:59:20.348 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 87/198 2024-03-31T22:59:20.382 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 88/198 2024-03-31T22:59:20.413 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : fuse-2.9.9-15.el9.x86_64 89/198 2024-03-31T22:59:20.441 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : zlib-devel-1.2.11-40.el9.x86_64 90/198 2024-03-31T22:59:20.478 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 91/198 2024-03-31T22:59:20.515 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : uuid-1.6.2-55.el9.x86_64 92/198 2024-03-31T22:59:20.547 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 93/198 2024-03-31T22:59:20.576 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 94/198 2024-03-31T22:59:20.586 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1977.g384f31f3.el9.x86_64 132/198 2024-03-31T22:59:20.586 INFO:teuthology.orchestra.run.smithi142.stdout:skipping the directory /sys 2024-03-31T22:59:20.587 INFO:teuthology.orchestra.run.smithi142.stdout:skipping the directory /proc 2024-03-31T22:59:20.587 INFO:teuthology.orchestra.run.smithi142.stdout:skipping the directory /mnt 2024-03-31T22:59:20.587 INFO:teuthology.orchestra.run.smithi142.stdout:skipping the directory /var/tmp 2024-03-31T22:59:20.587 INFO:teuthology.orchestra.run.smithi142.stdout:skipping the directory /home 2024-03-31T22:59:20.587 INFO:teuthology.orchestra.run.smithi142.stdout:skipping the directory /root 2024-03-31T22:59:20.587 INFO:teuthology.orchestra.run.smithi142.stdout:skipping the directory /tmp 2024-03-31T22:59:20.587 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-03-31T22:59:20.652 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 95/198 2024-03-31T22:59:20.687 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 96/198 2024-03-31T22:59:20.721 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 97/198 2024-03-31T22:59:20.754 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 98/198 2024-03-31T22:59:20.760 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : ceph-volume-2:19.0.0-1977.g384f31f3.el9.noarch 133/198 2024-03-31T22:59:20.768 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1977.g384f31f3.el9.noa 141/198 2024-03-31T22:59:20.775 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: ceph-volume-2:19.0.0-1977.g384f31f3.el9.noarch 133/198 2024-03-31T22:59:20.776 INFO:teuthology.orchestra.run.smithi142.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-31T22:59:20.776 INFO:teuthology.orchestra.run.smithi142.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-31T22:59:20.776 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-03-31T22:59:20.788 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 99/198 2024-03-31T22:59:20.797 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1977.g384f31f3.el9.n 142/198 2024-03-31T22:59:20.878 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 100/198 2024-03-31T22:59:20.976 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 101/198 2024-03-31T22:59:21.020 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 102/198 2024-03-31T22:59:21.051 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 103/198 2024-03-31T22:59:21.098 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 104/198 2024-03-31T22:59:21.196 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-packaging-20.9-5.el9.noarch 105/198 2024-03-31T22:59:21.245 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 106/198 2024-03-31T22:59:21.814 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1977.g384f31f3.el9.n 142/198 2024-03-31T22:59:21.840 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1977.g384 143/198 2024-03-31T22:59:21.948 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1977.g384 143/198 2024-03-31T22:59:21.967 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 107/198 2024-03-31T22:59:22.008 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 108/198 2024-03-31T22:59:22.042 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 109/198 2024-03-31T22:59:22.090 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 110/198 2024-03-31T22:59:22.137 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1977.g384f31f3.el 144/198 2024-03-31T22:59:22.167 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-mgr-2:19.0.0-1977.g384f31f3.el9.x86_64 145/198 2024-03-31T22:59:22.169 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 111/198 2024-03-31T22:59:22.189 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1977.g384f31f3.el9.x86_64 145/198 2024-03-31T22:59:22.190 INFO:teuthology.orchestra.run.smithi046.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-31T22:59:22.190 INFO:teuthology.orchestra.run.smithi046.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-31T22:59:22.190 INFO:teuthology.orchestra.run.smithi046.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-31T22:59:22.190 INFO:teuthology.orchestra.run.smithi046.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-31T22:59:22.190 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-31T22:59:22.211 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 112/198 2024-03-31T22:59:22.234 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-mgr-rook-2:19.0.0-1977.g384f31f3.el9.noarch 146/198 2024-03-31T22:59:22.245 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1977.g384f31f3.el9.noarch 146/198 2024-03-31T22:59:22.280 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : ceph-osd-2:19.0.0-1977.g384f31f3.el9.x86_64 134/198 2024-03-31T22:59:22.294 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-2:19.0.0-1977.g384f31f3.el9.x86_64 147/198 2024-03-31T22:59:22.305 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: ceph-osd-2:19.0.0-1977.g384f31f3.el9.x86_64 134/198 2024-03-31T22:59:22.306 INFO:teuthology.orchestra.run.smithi142.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-31T22:59:22.306 INFO:teuthology.orchestra.run.smithi142.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-31T22:59:22.306 INFO:teuthology.orchestra.run.smithi142.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-31T22:59:22.306 INFO:teuthology.orchestra.run.smithi142.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-31T22:59:22.306 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-03-31T22:59:22.387 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : openblas-0.3.21-2.el9.x86_64 113/198 2024-03-31T22:59:22.406 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 114/198 2024-03-31T22:59:22.483 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 115/198 2024-03-31T22:59:22.530 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : ceph-mds-2:19.0.0-1977.g384f31f3.el9.x86_64 135/198 2024-03-31T22:59:22.553 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: ceph-mds-2:19.0.0-1977.g384f31f3.el9.x86_64 135/198 2024-03-31T22:59:22.553 INFO:teuthology.orchestra.run.smithi142.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-31T22:59:22.553 INFO:teuthology.orchestra.run.smithi142.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-31T22:59:22.553 INFO:teuthology.orchestra.run.smithi142.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-31T22:59:22.553 INFO:teuthology.orchestra.run.smithi142.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-31T22:59:22.553 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-03-31T22:59:22.989 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : ceph-mon-2:19.0.0-1977.g384f31f3.el9.x86_64 136/198 2024-03-31T22:59:23.011 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: ceph-mon-2:19.0.0-1977.g384f31f3.el9.x86_64 136/198 2024-03-31T22:59:23.011 INFO:teuthology.orchestra.run.smithi142.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-31T22:59:23.012 INFO:teuthology.orchestra.run.smithi142.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-31T22:59:23.012 INFO:teuthology.orchestra.run.smithi142.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-31T22:59:23.012 INFO:teuthology.orchestra.run.smithi142.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-31T22:59:23.012 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-03-31T22:59:23.116 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : automake-1.16.2-8.el9.noarch 137/198 2024-03-31T22:59:23.129 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 116/198 2024-03-31T22:59:23.193 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : libtool-2.4.6-45.el9.x86_64 148/198 2024-03-31T22:59:23.220 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-radosgw-2:19.0.0-1977.g384f31f3.el9.x86_64 149/198 2024-03-31T22:59:23.238 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 117/198 2024-03-31T22:59:23.241 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1977.g384f31f3.el9.x86_64 149/198 2024-03-31T22:59:23.241 INFO:teuthology.orchestra.run.smithi046.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-31T22:59:23.242 INFO:teuthology.orchestra.run.smithi046.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-31T22:59:23.242 INFO:teuthology.orchestra.run.smithi046.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-31T22:59:23.242 INFO:teuthology.orchestra.run.smithi046.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-31T22:59:23.242 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-31T22:59:23.295 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1977.g384f3 150/198 2024-03-31T22:59:23.317 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1977.g384f3 150/198 2024-03-31T22:59:23.317 INFO:teuthology.orchestra.run.smithi046.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-31T22:59:23.317 INFO:teuthology.orchestra.run.smithi046.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-31T22:59:23.318 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-31T22:59:23.363 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : cephfs-mirror-2:19.0.0-1977.g384f31f3.el9.x86_64 151/198 2024-03-31T22:59:23.385 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-1977.g384f31f3.el9.x86_64 151/198 2024-03-31T22:59:23.385 INFO:teuthology.orchestra.run.smithi046.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-31T22:59:23.385 INFO:teuthology.orchestra.run.smithi046.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-03-31T22:59:23.385 INFO:teuthology.orchestra.run.smithi046.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-31T22:59:23.385 INFO:teuthology.orchestra.run.smithi046.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-31T22:59:23.385 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-31T22:59:23.569 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: cephadm-2:19.0.0-1977.g384f31f3.el9.noarch 138/198 2024-03-31T22:59:23.605 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : cephadm-2:19.0.0-1977.g384f31f3.el9.noarch 138/198 2024-03-31T22:59:23.637 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1977.g384f31f3.e 139/198 2024-03-31T22:59:23.697 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : rbd-mirror-2:19.0.0-1977.g384f31f3.el9.x86_64 152/198 2024-03-31T22:59:23.697 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1977.g384f31f3. 140/198 2024-03-31T22:59:23.718 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1977.g384f31f3.el9.x86_64 152/198 2024-03-31T22:59:23.718 INFO:teuthology.orchestra.run.smithi046.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-31T22:59:23.719 INFO:teuthology.orchestra.run.smithi046.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-31T22:59:23.719 INFO:teuthology.orchestra.run.smithi046.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-31T22:59:23.719 INFO:teuthology.orchestra.run.smithi046.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-31T22:59:23.719 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-31T22:59:23.720 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1977.g384f31f3.el9.noa 141/198 2024-03-31T22:59:24.467 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 118/198 2024-03-31T22:59:24.567 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 119/198 2024-03-31T22:59:24.598 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 120/198 2024-03-31T22:59:24.629 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : jq-1.6-16.el9.x86_64 121/198 2024-03-31T22:59:24.657 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 122/198 2024-03-31T22:59:24.748 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1977.g384f31f3.el9.noa 141/198 2024-03-31T22:59:24.789 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1977.g384f31f3.el9.n 142/198 2024-03-31T22:59:24.971 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 123/198 2024-03-31T22:59:24.995 INFO:teuthology.orchestra.run.smithi162.stdout: Upgrading : librbd1-2:19.0.0-1977.g384f31f3.el9.x86_64 124/198 2024-03-31T22:59:25.085 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: librbd1-2:19.0.0-1977.g384f31f3.el9.x86_64 124/198 2024-03-31T22:59:25.108 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-rbd-2:19.0.0-1977.g384f31f3.el9.x86_64 125/198 2024-03-31T22:59:25.137 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 126/198 2024-03-31T22:59:25.591 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 127/198 2024-03-31T22:59:25.623 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : librgw2-2:19.0.0-1977.g384f31f3.el9.x86_64 128/198 2024-03-31T22:59:25.677 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: librgw2-2:19.0.0-1977.g384f31f3.el9.x86_64 128/198 2024-03-31T22:59:25.700 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-rgw-2:19.0.0-1977.g384f31f3.el9.x86_64 129/198 2024-03-31T22:59:25.749 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1977.g384f31f3.el9.n 142/198 2024-03-31T22:59:25.772 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1977.g384 143/198 2024-03-31T22:59:25.872 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1977.g384 143/198 2024-03-31T22:59:26.038 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1977.g384f31f3.el 144/198 2024-03-31T22:59:26.066 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : ceph-mgr-2:19.0.0-1977.g384f31f3.el9.x86_64 145/198 2024-03-31T22:59:26.089 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1977.g384f31f3.el9.x86_64 145/198 2024-03-31T22:59:26.089 INFO:teuthology.orchestra.run.smithi142.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-31T22:59:26.089 INFO:teuthology.orchestra.run.smithi142.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-31T22:59:26.089 INFO:teuthology.orchestra.run.smithi142.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-31T22:59:26.089 INFO:teuthology.orchestra.run.smithi142.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-31T22:59:26.089 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-03-31T22:59:26.125 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : ceph-mgr-rook-2:19.0.0-1977.g384f31f3.el9.noarch 146/198 2024-03-31T22:59:26.136 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1977.g384f31f3.el9.noarch 146/198 2024-03-31T22:59:26.184 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : ceph-2:19.0.0-1977.g384f31f3.el9.x86_64 147/198 2024-03-31T22:59:27.036 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : libtool-2.4.6-45.el9.x86_64 148/198 2024-03-31T22:59:27.069 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : ceph-radosgw-2:19.0.0-1977.g384f31f3.el9.x86_64 149/198 2024-03-31T22:59:27.091 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1977.g384f31f3.el9.x86_64 149/198 2024-03-31T22:59:27.091 INFO:teuthology.orchestra.run.smithi142.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-31T22:59:27.092 INFO:teuthology.orchestra.run.smithi142.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-31T22:59:27.092 INFO:teuthology.orchestra.run.smithi142.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-31T22:59:27.092 INFO:teuthology.orchestra.run.smithi142.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-31T22:59:27.092 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-03-31T22:59:27.136 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1977.g384f3 150/198 2024-03-31T22:59:27.158 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1977.g384f3 150/198 2024-03-31T22:59:27.158 INFO:teuthology.orchestra.run.smithi142.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-31T22:59:27.158 INFO:teuthology.orchestra.run.smithi142.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-31T22:59:27.158 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-03-31T22:59:27.204 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : cephfs-mirror-2:19.0.0-1977.g384f31f3.el9.x86_64 151/198 2024-03-31T22:59:27.225 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-1977.g384f31f3.el9.x86_64 151/198 2024-03-31T22:59:27.226 INFO:teuthology.orchestra.run.smithi142.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-31T22:59:27.226 INFO:teuthology.orchestra.run.smithi142.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-03-31T22:59:27.226 INFO:teuthology.orchestra.run.smithi142.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-31T22:59:27.226 INFO:teuthology.orchestra.run.smithi142.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-31T22:59:27.226 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-03-31T22:59:27.529 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : rbd-mirror-2:19.0.0-1977.g384f31f3.el9.x86_64 152/198 2024-03-31T22:59:27.552 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1977.g384f31f3.el9.x86_64 152/198 2024-03-31T22:59:27.552 INFO:teuthology.orchestra.run.smithi142.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-31T22:59:27.552 INFO:teuthology.orchestra.run.smithi142.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-31T22:59:27.552 INFO:teuthology.orchestra.run.smithi142.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-31T22:59:27.552 INFO:teuthology.orchestra.run.smithi142.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-31T22:59:27.552 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-03-31T22:59:27.949 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: ceph-common-2:19.0.0-1977.g384f31f3.el9.x86_64 130/198 2024-03-31T22:59:27.980 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-common-2:19.0.0-1977.g384f31f3.el9.x86_64 130/198 2024-03-31T22:59:28.410 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-test-2:19.0.0-1977.g384f31f3.el9.x86_64 153/198 2024-03-31T22:59:28.452 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : rbd-fuse-2:19.0.0-1977.g384f31f3.el9.x86_64 154/198 2024-03-31T22:59:28.481 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : rbd-nbd-2:19.0.0-1977.g384f31f3.el9.x86_64 155/198 2024-03-31T22:59:28.508 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 156/198 2024-03-31T22:59:28.518 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: ceph-common-2:19.0.0-1977.g384f31f3.el9.x86_64 130/198 2024-03-31T22:59:28.559 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-base-2:19.0.0-1977.g384f31f3.el9.x86_64 131/198 2024-03-31T22:59:28.648 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: ceph-base-2:19.0.0-1977.g384f31f3.el9.x86_64 131/198 2024-03-31T22:59:28.648 INFO:teuthology.orchestra.run.smithi162.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-31T22:59:28.648 INFO:teuthology.orchestra.run.smithi162.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-31T22:59:28.648 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-31T22:59:28.685 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-selinux-2:19.0.0-1977.g384f31f3.el9.x86_64 132/198 2024-03-31T22:59:28.698 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : cephfs-shell-2:19.0.0-1977.g384f31f3.el9.x86_64 157/198 2024-03-31T22:59:28.812 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 158/198 2024-03-31T22:59:28.849 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-fuse-2:19.0.0-1977.g384f31f3.el9.x86_64 159/198 2024-03-31T22:59:28.910 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 160/198 2024-03-31T22:59:28.956 INFO:teuthology.orchestra.run.smithi046.stdout: Upgrading : xfsdump-3.1.12-4.el9.x86_64 161/198 2024-03-31T22:59:29.000 INFO:teuthology.orchestra.run.smithi046.stdout: Upgrading : util-linux-user-2.37.4-18.el9.x86_64 162/198 2024-03-31T22:59:29.024 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : libcephfs-devel-2:19.0.0-1977.g384f31f3.el9.x86_ 163/198 2024-03-31T22:59:29.056 INFO:teuthology.orchestra.run.smithi046.stdout: Upgrading : elfutils-debuginfod-client-0.190-2.el9.x86_64 164/198 2024-03-31T22:59:29.086 INFO:teuthology.orchestra.run.smithi046.stdout: Upgrading : NetworkManager-team-1:1.47.2-1.el9.x86_64 165/198 2024-03-31T22:59:29.145 INFO:teuthology.orchestra.run.smithi046.stdout: Upgrading : NetworkManager-tui-1:1.47.2-1.el9.x86_64 166/198 2024-03-31T22:59:29.207 INFO:teuthology.orchestra.run.smithi046.stdout: Upgrading : acl-2.3.1-4.el9.x86_64 167/198 2024-03-31T22:59:29.242 INFO:teuthology.orchestra.run.smithi046.stdout: Upgrading : libacl-devel-2.3.1-4.el9.x86_64 168/198 2024-03-31T22:59:29.863 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 169/198 2024-03-31T22:59:29.899 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 169/198 2024-03-31T22:59:29.919 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 169/198 2024-03-31T22:59:30.799 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : cephfs-top-2:19.0.0-1977.g384f31f3.el9.noarch 170/198 2024-03-31T22:59:30.877 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : openssl-devel-1:3.0.7-27.el9.x86_64 171/198 2024-03-31T22:59:30.919 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 172/198 2024-03-31T22:59:31.011 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : flex-2.6.4-9.el9.x86_64 173/198 2024-03-31T22:59:31.039 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : bison-3.7.4-5.el9.x86_64 174/198 2024-03-31T22:59:31.065 INFO:teuthology.orchestra.run.smithi046.stdout: Cleanup : NetworkManager-tui-1:1.44.0-2.el9.x86_64 175/198 2024-03-31T22:59:31.090 INFO:teuthology.orchestra.run.smithi046.stdout: Cleanup : elfutils-debuginfod-client-0.189-3.el9.x86_64 176/198 2024-03-31T22:59:31.117 INFO:teuthology.orchestra.run.smithi046.stdout: Cleanup : elfutils-libs-0.189-3.el9.x86_64 177/198 2024-03-31T22:59:31.117 INFO:teuthology.orchestra.run.smithi046.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-31T22:59:31.188 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-31T22:59:31.226 INFO:teuthology.orchestra.run.smithi046.stdout: Cleanup : openssl-1:3.0.7-24.el9.x86_64 179/198 2024-03-31T22:59:31.226 INFO:teuthology.orchestra.run.smithi046.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-31T22:59:31.302 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-31T22:59:31.328 INFO:teuthology.orchestra.run.smithi046.stdout: Cleanup : xfsdump-3.1.12-3.el9.x86_64 181/198 2024-03-31T22:59:31.361 INFO:teuthology.orchestra.run.smithi046.stdout: Cleanup : xfsprogs-5.19.0-4.el9.x86_64 182/198 2024-03-31T22:59:31.395 INFO:teuthology.orchestra.run.smithi046.stdout: Cleanup : acl-2.3.1-3.el9.x86_64 183/198 2024-03-31T22:59:31.420 INFO:teuthology.orchestra.run.smithi046.stdout: Cleanup : libacl-devel-2.3.1-3.el9.x86_64 184/198 2024-03-31T22:59:31.455 INFO:teuthology.orchestra.run.smithi046.stdout: Cleanup : util-linux-user-2.37.4-15.el9.x86_64 185/198 2024-03-31T22:59:31.511 INFO:teuthology.orchestra.run.smithi046.stdout: Cleanup : util-linux-2.37.4-15.el9.x86_64 186/198 2024-03-31T22:59:31.542 INFO:teuthology.orchestra.run.smithi046.stdout: Cleanup : util-linux-core-2.37.4-15.el9.x86_64 187/198 2024-03-31T22:59:31.567 INFO:teuthology.orchestra.run.smithi046.stdout: Cleanup : libfdisk-2.37.4-15.el9.x86_64 188/198 2024-03-31T22:59:31.592 INFO:teuthology.orchestra.run.smithi046.stdout: Cleanup : libmount-2.37.4-15.el9.x86_64 189/198 2024-03-31T22:59:31.617 INFO:teuthology.orchestra.run.smithi046.stdout: Cleanup : libblkid-2.37.4-15.el9.x86_64 190/198 2024-03-31T22:59:31.642 INFO:teuthology.orchestra.run.smithi046.stdout: Cleanup : NetworkManager-team-1:1.44.0-2.el9.x86_64 191/198 2024-03-31T22:59:31.649 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-31T22:59:31.649 INFO:teuthology.orchestra.run.smithi046.stdout: Cleanup : NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-31T22:59:31.837 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-31T22:59:31.837 INFO:teuthology.orchestra.run.smithi046.stdout: Cleanup : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-31T22:59:31.920 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-31T22:59:31.960 INFO:teuthology.orchestra.run.smithi046.stdout: Cleanup : libuuid-2.37.4-15.el9.x86_64 194/198 2024-03-31T22:59:31.983 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : ceph-test-2:19.0.0-1977.g384f31f3.el9.x86_64 153/198 2024-03-31T22:59:31.987 INFO:teuthology.orchestra.run.smithi046.stdout: Cleanup : libsmartcols-2.37.4-15.el9.x86_64 195/198 2024-03-31T22:59:32.024 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : rbd-fuse-2:19.0.0-1977.g384f31f3.el9.x86_64 154/198 2024-03-31T22:59:32.038 INFO:teuthology.orchestra.run.smithi046.stdout: Cleanup : libacl-2.3.1-3.el9.x86_64 196/198 2024-03-31T22:59:32.055 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : rbd-nbd-2:19.0.0-1977.g384f31f3.el9.x86_64 155/198 2024-03-31T22:59:32.063 INFO:teuthology.orchestra.run.smithi046.stdout: Cleanup : openssl-libs-1:3.0.7-24.el9.x86_64 197/198 2024-03-31T22:59:32.098 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 156/198 2024-03-31T22:59:32.132 INFO:teuthology.orchestra.run.smithi046.stdout: Cleanup : elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-31T22:59:32.278 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : cephfs-shell-2:19.0.0-1977.g384f31f3.el9.x86_64 157/198 2024-03-31T22:59:32.390 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 158/198 2024-03-31T22:59:32.429 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : ceph-fuse-2:19.0.0-1977.g384f31f3.el9.x86_64 159/198 2024-03-31T22:59:32.484 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 160/198 2024-03-31T22:59:32.546 INFO:teuthology.orchestra.run.smithi142.stdout: Upgrading : xfsdump-3.1.12-4.el9.x86_64 161/198 2024-03-31T22:59:32.590 INFO:teuthology.orchestra.run.smithi142.stdout: Upgrading : util-linux-user-2.37.4-18.el9.x86_64 162/198 2024-03-31T22:59:32.614 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : libcephfs-devel-2:19.0.0-1977.g384f31f3.el9.x86_ 163/198 2024-03-31T22:59:32.646 INFO:teuthology.orchestra.run.smithi142.stdout: Upgrading : elfutils-debuginfod-client-0.190-2.el9.x86_64 164/198 2024-03-31T22:59:32.676 INFO:teuthology.orchestra.run.smithi142.stdout: Upgrading : NetworkManager-team-1:1.47.2-1.el9.x86_64 165/198 2024-03-31T22:59:32.738 INFO:teuthology.orchestra.run.smithi142.stdout: Upgrading : NetworkManager-tui-1:1.47.2-1.el9.x86_64 166/198 2024-03-31T22:59:32.801 INFO:teuthology.orchestra.run.smithi142.stdout: Upgrading : acl-2.3.1-4.el9.x86_64 167/198 2024-03-31T22:59:32.841 INFO:teuthology.orchestra.run.smithi142.stdout: Upgrading : libacl-devel-2.3.1-4.el9.x86_64 168/198 2024-03-31T22:59:33.475 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 169/198 2024-03-31T22:59:33.522 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 169/198 2024-03-31T22:59:33.542 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 169/198 2024-03-31T22:59:34.351 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : cephfs-top-2:19.0.0-1977.g384f31f3.el9.noarch 170/198 2024-03-31T22:59:34.434 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : openssl-devel-1:3.0.7-27.el9.x86_64 171/198 2024-03-31T22:59:34.474 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 172/198 2024-03-31T22:59:34.559 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : flex-2.6.4-9.el9.x86_64 173/198 2024-03-31T22:59:34.595 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : bison-3.7.4-5.el9.x86_64 174/198 2024-03-31T22:59:34.620 INFO:teuthology.orchestra.run.smithi142.stdout: Cleanup : NetworkManager-tui-1:1.44.0-2.el9.x86_64 175/198 2024-03-31T22:59:34.646 INFO:teuthology.orchestra.run.smithi142.stdout: Cleanup : elfutils-debuginfod-client-0.189-3.el9.x86_64 176/198 2024-03-31T22:59:34.672 INFO:teuthology.orchestra.run.smithi142.stdout: Cleanup : elfutils-libs-0.189-3.el9.x86_64 177/198 2024-03-31T22:59:34.673 INFO:teuthology.orchestra.run.smithi142.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-31T22:59:34.752 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-31T22:59:34.790 INFO:teuthology.orchestra.run.smithi142.stdout: Cleanup : openssl-1:3.0.7-24.el9.x86_64 179/198 2024-03-31T22:59:34.790 INFO:teuthology.orchestra.run.smithi142.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-31T22:59:34.883 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-31T22:59:34.909 INFO:teuthology.orchestra.run.smithi142.stdout: Cleanup : xfsdump-3.1.12-3.el9.x86_64 181/198 2024-03-31T22:59:34.942 INFO:teuthology.orchestra.run.smithi142.stdout: Cleanup : xfsprogs-5.19.0-4.el9.x86_64 182/198 2024-03-31T22:59:34.967 INFO:teuthology.orchestra.run.smithi142.stdout: Cleanup : acl-2.3.1-3.el9.x86_64 183/198 2024-03-31T22:59:34.992 INFO:teuthology.orchestra.run.smithi142.stdout: Cleanup : libacl-devel-2.3.1-3.el9.x86_64 184/198 2024-03-31T22:59:35.027 INFO:teuthology.orchestra.run.smithi142.stdout: Cleanup : util-linux-user-2.37.4-15.el9.x86_64 185/198 2024-03-31T22:59:35.080 INFO:teuthology.orchestra.run.smithi142.stdout: Cleanup : util-linux-2.37.4-15.el9.x86_64 186/198 2024-03-31T22:59:35.114 INFO:teuthology.orchestra.run.smithi142.stdout: Cleanup : util-linux-core-2.37.4-15.el9.x86_64 187/198 2024-03-31T22:59:35.139 INFO:teuthology.orchestra.run.smithi142.stdout: Cleanup : libfdisk-2.37.4-15.el9.x86_64 188/198 2024-03-31T22:59:35.165 INFO:teuthology.orchestra.run.smithi142.stdout: Cleanup : libmount-2.37.4-15.el9.x86_64 189/198 2024-03-31T22:59:35.190 INFO:teuthology.orchestra.run.smithi142.stdout: Cleanup : libblkid-2.37.4-15.el9.x86_64 190/198 2024-03-31T22:59:35.215 INFO:teuthology.orchestra.run.smithi142.stdout: Cleanup : NetworkManager-team-1:1.44.0-2.el9.x86_64 191/198 2024-03-31T22:59:35.222 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-31T22:59:35.222 INFO:teuthology.orchestra.run.smithi142.stdout: Cleanup : NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-31T22:59:35.410 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-31T22:59:35.410 INFO:teuthology.orchestra.run.smithi142.stdout: Cleanup : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-31T22:59:35.493 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-31T22:59:35.525 INFO:teuthology.orchestra.run.smithi142.stdout: Cleanup : libuuid-2.37.4-15.el9.x86_64 194/198 2024-03-31T22:59:35.551 INFO:teuthology.orchestra.run.smithi142.stdout: Cleanup : libsmartcols-2.37.4-15.el9.x86_64 195/198 2024-03-31T22:59:35.577 INFO:teuthology.orchestra.run.smithi142.stdout: Cleanup : libacl-2.3.1-3.el9.x86_64 196/198 2024-03-31T22:59:35.602 INFO:teuthology.orchestra.run.smithi142.stdout: Cleanup : openssl-libs-1:3.0.7-24.el9.x86_64 197/198 2024-03-31T22:59:35.667 INFO:teuthology.orchestra.run.smithi142.stdout: Cleanup : elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-31T22:59:37.544 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-31T22:59:37.544 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-2:19.0.0-1977.g384f31f3.el9.x86_64 1/198 2024-03-31T22:59:37.545 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-base-2:19.0.0-1977.g384f31f3.el9.x86_64 2/198 2024-03-31T22:59:37.545 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-common-2:19.0.0-1977.g384f31f3.el9.x86_64 3/198 2024-03-31T22:59:37.545 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-fuse-2:19.0.0-1977.g384f31f3.el9.x86_64 4/198 2024-03-31T22:59:37.545 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1977.g384f3 5/198 2024-03-31T22:59:37.545 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-mds-2:19.0.0-1977.g384f31f3.el9.x86_64 6/198 2024-03-31T22:59:37.545 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-mgr-2:19.0.0-1977.g384f31f3.el9.x86_64 7/198 2024-03-31T22:59:37.545 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-mon-2:19.0.0-1977.g384f31f3.el9.x86_64 8/198 2024-03-31T22:59:37.545 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-osd-2:19.0.0-1977.g384f31f3.el9.x86_64 9/198 2024-03-31T22:59:37.545 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-radosgw-2:19.0.0-1977.g384f31f3.el9.x86_64 10/198 2024-03-31T22:59:37.545 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-selinux-2:19.0.0-1977.g384f31f3.el9.x86_64 11/198 2024-03-31T22:59:37.545 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-test-2:19.0.0-1977.g384f31f3.el9.x86_64 12/198 2024-03-31T22:59:37.545 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : cephfs-mirror-2:19.0.0-1977.g384f31f3.el9.x86_64 13/198 2024-03-31T22:59:37.546 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : cephfs-shell-2:19.0.0-1977.g384f31f3.el9.x86_64 14/198 2024-03-31T22:59:37.546 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : libcephfs-devel-2:19.0.0-1977.g384f31f3.el9.x86_ 15/198 2024-03-31T22:59:37.546 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : libcephfs2-2:19.0.0-1977.g384f31f3.el9.x86_64 16/198 2024-03-31T22:59:37.546 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : libcephsqlite-2:19.0.0-1977.g384f31f3.el9.x86_64 17/198 2024-03-31T22:59:37.546 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : librados-devel-2:19.0.0-1977.g384f31f3.el9.x86_6 18/198 2024-03-31T22:59:37.546 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : libradosstriper1-2:19.0.0-1977.g384f31f3.el9.x86 19/198 2024-03-31T22:59:37.546 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : librgw2-2:19.0.0-1977.g384f31f3.el9.x86_64 20/198 2024-03-31T22:59:37.546 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1977.g384f31f3.el 21/198 2024-03-31T22:59:37.546 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-ceph-common-2:19.0.0-1977.g384f31f3.el9. 22/198 2024-03-31T22:59:37.546 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-cephfs-2:19.0.0-1977.g384f31f3.el9.x86_6 23/198 2024-03-31T22:59:37.546 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-rados-2:19.0.0-1977.g384f31f3.el9.x86_64 24/198 2024-03-31T22:59:37.546 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-rbd-2:19.0.0-1977.g384f31f3.el9.x86_64 25/198 2024-03-31T22:59:37.546 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-rgw-2:19.0.0-1977.g384f31f3.el9.x86_64 26/198 2024-03-31T22:59:37.547 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : rbd-fuse-2:19.0.0-1977.g384f31f3.el9.x86_64 27/198 2024-03-31T22:59:37.547 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : rbd-mirror-2:19.0.0-1977.g384f31f3.el9.x86_64 28/198 2024-03-31T22:59:37.547 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : rbd-nbd-2:19.0.0-1977.g384f31f3.el9.x86_64 29/198 2024-03-31T22:59:37.547 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1977.g384f31f3. 30/198 2024-03-31T22:59:37.547 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1977.g384f31f3.el9.noa 31/198 2024-03-31T22:59:37.547 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1977.g384f31f3.el9.n 32/198 2024-03-31T22:59:37.547 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1977.g384 33/198 2024-03-31T22:59:37.547 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1977.g384f31f3.el 34/198 2024-03-31T22:59:37.547 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1977.g384f31f3.el9.noarch 35/198 2024-03-31T22:59:37.547 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1977.g384f31f3.e 36/198 2024-03-31T22:59:37.547 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-volume-2:19.0.0-1977.g384f31f3.el9.noarch 37/198 2024-03-31T22:59:37.547 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : cephadm-2:19.0.0-1977.g384f31f3.el9.noarch 38/198 2024-03-31T22:59:37.547 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : cephfs-top-2:19.0.0-1977.g384f31f3.el9.noarch 39/198 2024-03-31T22:59:37.548 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : automake-1.16.2-8.el9.noarch 40/198 2024-03-31T22:59:37.548 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : bison-3.7.4-5.el9.x86_64 41/198 2024-03-31T22:59:37.548 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 42/198 2024-03-31T22:59:37.548 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : flex-2.6.4-9.el9.x86_64 43/198 2024-03-31T22:59:37.548 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 44/198 2024-03-31T22:59:37.548 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 45/198 2024-03-31T22:59:37.548 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 46/198 2024-03-31T22:59:37.548 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 47/198 2024-03-31T22:59:37.548 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 48/198 2024-03-31T22:59:37.548 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 49/198 2024-03-31T22:59:37.548 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 50/198 2024-03-31T22:59:37.548 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 51/198 2024-03-31T22:59:37.548 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 52/198 2024-03-31T22:59:37.549 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 53/198 2024-03-31T22:59:37.549 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : lua-5.4.4-4.el9.x86_64 54/198 2024-03-31T22:59:37.549 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 55/198 2024-03-31T22:59:37.549 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 56/198 2024-03-31T22:59:37.549 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 57/198 2024-03-31T22:59:37.549 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : postgresql-13.14-1.el9.x86_64 58/198 2024-03-31T22:59:37.549 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 59/198 2024-03-31T22:59:37.549 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 60/198 2024-03-31T22:59:37.549 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 61/198 2024-03-31T22:59:37.549 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 62/198 2024-03-31T22:59:37.549 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 63/198 2024-03-31T22:59:37.549 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 64/198 2024-03-31T22:59:37.549 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 65/198 2024-03-31T22:59:37.549 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 66/198 2024-03-31T22:59:37.550 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 67/198 2024-03-31T22:59:37.550 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 68/198 2024-03-31T22:59:37.550 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 69/198 2024-03-31T22:59:37.550 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 70/198 2024-03-31T22:59:37.550 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 71/198 2024-03-31T22:59:37.550 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 72/198 2024-03-31T22:59:37.550 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 73/198 2024-03-31T22:59:37.550 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 74/198 2024-03-31T22:59:37.550 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 75/198 2024-03-31T22:59:37.550 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 76/198 2024-03-31T22:59:37.550 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 77/198 2024-03-31T22:59:37.550 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 78/198 2024-03-31T22:59:37.550 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 79/198 2024-03-31T22:59:37.551 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 80/198 2024-03-31T22:59:37.551 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 81/198 2024-03-31T22:59:37.551 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 82/198 2024-03-31T22:59:37.551 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 83/198 2024-03-31T22:59:37.551 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 84/198 2024-03-31T22:59:37.551 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 85/198 2024-03-31T22:59:37.551 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 86/198 2024-03-31T22:59:37.551 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 87/198 2024-03-31T22:59:37.551 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : zlib-devel-1.2.11-40.el9.x86_64 88/198 2024-03-31T22:59:37.551 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 89/198 2024-03-31T22:59:37.551 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 90/198 2024-03-31T22:59:37.551 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 91/198 2024-03-31T22:59:37.551 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-ply-3.11-14.el9.noarch 92/198 2024-03-31T22:59:37.552 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 93/198 2024-03-31T22:59:37.552 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 94/198 2024-03-31T22:59:37.552 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : jq-1.6-16.el9.x86_64 95/198 2024-03-31T22:59:37.552 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 96/198 2024-03-31T22:59:37.552 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 97/198 2024-03-31T22:59:37.552 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 98/198 2024-03-31T22:59:37.552 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 99/198 2024-03-31T22:59:37.552 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 100/198 2024-03-31T22:59:37.552 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : openssl-devel-1:3.0.7-27.el9.x86_64 101/198 2024-03-31T22:59:37.552 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 102/198 2024-03-31T22:59:37.552 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 103/198 2024-03-31T22:59:37.552 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 104/198 2024-03-31T22:59:37.552 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 105/198 2024-03-31T22:59:37.552 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 106/198 2024-03-31T22:59:37.553 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 107/198 2024-03-31T22:59:37.553 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 108/198 2024-03-31T22:59:37.553 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 109/198 2024-03-31T22:59:37.553 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 110/198 2024-03-31T22:59:37.553 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 111/198 2024-03-31T22:59:37.553 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 112/198 2024-03-31T22:59:37.553 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 113/198 2024-03-31T22:59:37.553 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 114/198 2024-03-31T22:59:37.553 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 115/198 2024-03-31T22:59:37.553 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 116/198 2024-03-31T22:59:37.553 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 117/198 2024-03-31T22:59:37.553 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 118/198 2024-03-31T22:59:37.553 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 119/198 2024-03-31T22:59:37.554 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 120/198 2024-03-31T22:59:37.554 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 121/198 2024-03-31T22:59:37.554 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 122/198 2024-03-31T22:59:37.554 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 123/198 2024-03-31T22:59:37.554 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 124/198 2024-03-31T22:59:37.554 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 125/198 2024-03-31T22:59:37.554 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 126/198 2024-03-31T22:59:37.554 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 127/198 2024-03-31T22:59:37.554 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 128/198 2024-03-31T22:59:37.554 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 129/198 2024-03-31T22:59:37.554 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 130/198 2024-03-31T22:59:37.554 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 131/198 2024-03-31T22:59:37.554 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 132/198 2024-03-31T22:59:37.555 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 133/198 2024-03-31T22:59:37.555 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 134/198 2024-03-31T22:59:37.555 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 135/198 2024-03-31T22:59:37.555 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 136/198 2024-03-31T22:59:37.555 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 137/198 2024-03-31T22:59:37.555 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 138/198 2024-03-31T22:59:37.555 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 139/198 2024-03-31T22:59:37.555 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 140/198 2024-03-31T22:59:37.555 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 141/198 2024-03-31T22:59:37.555 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 142/198 2024-03-31T22:59:37.555 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 143/198 2024-03-31T22:59:37.555 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 144/198 2024-03-31T22:59:37.555 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 145/198 2024-03-31T22:59:37.555 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 146/198 2024-03-31T22:59:37.556 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 147/198 2024-03-31T22:59:37.556 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 148/198 2024-03-31T22:59:37.556 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : re2-1:20211101-3.el9.x86_64 149/198 2024-03-31T22:59:37.556 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 150/198 2024-03-31T22:59:37.556 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : librados2-2:19.0.0-1977.g384f31f3.el9.x86_64 151/198 2024-03-31T22:59:37.556 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 152/198 2024-03-31T22:59:37.556 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : librbd1-2:19.0.0-1977.g384f31f3.el9.x86_64 153/198 2024-03-31T22:59:37.556 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 154/198 2024-03-31T22:59:37.556 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : NetworkManager-1:1.47.2-1.el9.x86_64 155/198 2024-03-31T22:59:37.556 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : NetworkManager-1:1.44.0-2.el9.x86_64 156/198 2024-03-31T22:59:37.556 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : NetworkManager-libnm-1:1.47.2-1.el9.x86_64 157/198 2024-03-31T22:59:37.556 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 158/198 2024-03-31T22:59:37.556 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : NetworkManager-team-1:1.47.2-1.el9.x86_64 159/198 2024-03-31T22:59:37.557 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : NetworkManager-team-1:1.44.0-2.el9.x86_64 160/198 2024-03-31T22:59:37.557 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : NetworkManager-tui-1:1.47.2-1.el9.x86_64 161/198 2024-03-31T22:59:37.557 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : NetworkManager-tui-1:1.44.0-2.el9.x86_64 162/198 2024-03-31T22:59:37.557 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : xfsdump-3.1.12-4.el9.x86_64 163/198 2024-03-31T22:59:37.557 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : xfsdump-3.1.12-3.el9.x86_64 164/198 2024-03-31T22:59:37.557 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : acl-2.3.1-4.el9.x86_64 165/198 2024-03-31T22:59:37.557 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : acl-2.3.1-3.el9.x86_64 166/198 2024-03-31T22:59:37.557 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 167/198 2024-03-31T22:59:37.557 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : elfutils-debuginfod-client-0.189-3.el9.x86_64 168/198 2024-03-31T22:59:37.557 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 169/198 2024-03-31T22:59:37.557 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : elfutils-libelf-0.189-3.el9.x86_64 170/198 2024-03-31T22:59:37.557 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 171/198 2024-03-31T22:59:37.557 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : elfutils-libs-0.189-3.el9.x86_64 172/198 2024-03-31T22:59:37.557 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : libacl-2.3.1-4.el9.x86_64 173/198 2024-03-31T22:59:37.558 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : libacl-2.3.1-3.el9.x86_64 174/198 2024-03-31T22:59:37.558 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : libblkid-2.37.4-18.el9.x86_64 175/198 2024-03-31T22:59:37.558 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : libblkid-2.37.4-15.el9.x86_64 176/198 2024-03-31T22:59:37.558 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : libfdisk-2.37.4-18.el9.x86_64 177/198 2024-03-31T22:59:37.558 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : libfdisk-2.37.4-15.el9.x86_64 178/198 2024-03-31T22:59:37.558 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : libmount-2.37.4-18.el9.x86_64 179/198 2024-03-31T22:59:37.558 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : libmount-2.37.4-15.el9.x86_64 180/198 2024-03-31T22:59:37.558 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : libsmartcols-2.37.4-18.el9.x86_64 181/198 2024-03-31T22:59:37.558 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : libsmartcols-2.37.4-15.el9.x86_64 182/198 2024-03-31T22:59:37.558 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : libuuid-2.37.4-18.el9.x86_64 183/198 2024-03-31T22:59:37.558 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : libuuid-2.37.4-15.el9.x86_64 184/198 2024-03-31T22:59:37.558 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 185/198 2024-03-31T22:59:37.558 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : openssl-1:3.0.7-24.el9.x86_64 186/198 2024-03-31T22:59:37.558 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 187/198 2024-03-31T22:59:37.559 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : openssl-libs-1:3.0.7-24.el9.x86_64 188/198 2024-03-31T22:59:37.559 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : util-linux-2.37.4-18.el9.x86_64 189/198 2024-03-31T22:59:37.559 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : util-linux-2.37.4-15.el9.x86_64 190/198 2024-03-31T22:59:37.559 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : util-linux-core-2.37.4-18.el9.x86_64 191/198 2024-03-31T22:59:37.559 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : util-linux-core-2.37.4-15.el9.x86_64 192/198 2024-03-31T22:59:37.559 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : util-linux-user-2.37.4-18.el9.x86_64 193/198 2024-03-31T22:59:37.559 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : util-linux-user-2.37.4-15.el9.x86_64 194/198 2024-03-31T22:59:37.559 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 195/198 2024-03-31T22:59:37.559 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : xfsprogs-5.19.0-4.el9.x86_64 196/198 2024-03-31T22:59:37.559 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : libacl-devel-2.3.1-4.el9.x86_64 197/198 2024-03-31T22:59:40.693 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1977.g384f31f3.el9.x86_64 132/198 2024-03-31T22:59:40.693 INFO:teuthology.orchestra.run.smithi162.stdout:skipping the directory /sys 2024-03-31T22:59:40.694 INFO:teuthology.orchestra.run.smithi162.stdout:skipping the directory /proc 2024-03-31T22:59:40.694 INFO:teuthology.orchestra.run.smithi162.stdout:skipping the directory /mnt 2024-03-31T22:59:40.694 INFO:teuthology.orchestra.run.smithi162.stdout:skipping the directory /var/tmp 2024-03-31T22:59:40.694 INFO:teuthology.orchestra.run.smithi162.stdout:skipping the directory /home 2024-03-31T22:59:40.694 INFO:teuthology.orchestra.run.smithi162.stdout:skipping the directory /root 2024-03-31T22:59:40.694 INFO:teuthology.orchestra.run.smithi162.stdout:skipping the directory /tmp 2024-03-31T22:59:40.694 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-31T22:59:40.730 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : libacl-devel-2.3.1-3.el9.x86_64 198/198 2024-03-31T22:59:40.730 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-31T22:59:40.730 INFO:teuthology.orchestra.run.smithi046.stdout:Upgraded: 2024-03-31T22:59:40.730 INFO:teuthology.orchestra.run.smithi046.stdout: NetworkManager-1:1.47.2-1.el9.x86_64 2024-03-31T22:59:40.730 INFO:teuthology.orchestra.run.smithi046.stdout: NetworkManager-libnm-1:1.47.2-1.el9.x86_64 2024-03-31T22:59:40.730 INFO:teuthology.orchestra.run.smithi046.stdout: NetworkManager-team-1:1.47.2-1.el9.x86_64 2024-03-31T22:59:40.730 INFO:teuthology.orchestra.run.smithi046.stdout: NetworkManager-tui-1:1.47.2-1.el9.x86_64 2024-03-31T22:59:40.730 INFO:teuthology.orchestra.run.smithi046.stdout: acl-2.3.1-4.el9.x86_64 2024-03-31T22:59:40.731 INFO:teuthology.orchestra.run.smithi046.stdout: elfutils-debuginfod-client-0.190-2.el9.x86_64 2024-03-31T22:59:40.731 INFO:teuthology.orchestra.run.smithi046.stdout: elfutils-libelf-0.190-2.el9.x86_64 2024-03-31T22:59:40.731 INFO:teuthology.orchestra.run.smithi046.stdout: elfutils-libs-0.190-2.el9.x86_64 2024-03-31T22:59:40.731 INFO:teuthology.orchestra.run.smithi046.stdout: libacl-2.3.1-4.el9.x86_64 2024-03-31T22:59:40.731 INFO:teuthology.orchestra.run.smithi046.stdout: libacl-devel-2.3.1-4.el9.x86_64 2024-03-31T22:59:40.731 INFO:teuthology.orchestra.run.smithi046.stdout: libblkid-2.37.4-18.el9.x86_64 2024-03-31T22:59:40.731 INFO:teuthology.orchestra.run.smithi046.stdout: libfdisk-2.37.4-18.el9.x86_64 2024-03-31T22:59:40.731 INFO:teuthology.orchestra.run.smithi046.stdout: libmount-2.37.4-18.el9.x86_64 2024-03-31T22:59:40.731 INFO:teuthology.orchestra.run.smithi046.stdout: librados2-2:19.0.0-1977.g384f31f3.el9.x86_64 2024-03-31T22:59:40.731 INFO:teuthology.orchestra.run.smithi046.stdout: librbd1-2:19.0.0-1977.g384f31f3.el9.x86_64 2024-03-31T22:59:40.731 INFO:teuthology.orchestra.run.smithi046.stdout: libsmartcols-2.37.4-18.el9.x86_64 2024-03-31T22:59:40.731 INFO:teuthology.orchestra.run.smithi046.stdout: libuuid-2.37.4-18.el9.x86_64 2024-03-31T22:59:40.731 INFO:teuthology.orchestra.run.smithi046.stdout: openssl-1:3.0.7-27.el9.x86_64 2024-03-31T22:59:40.732 INFO:teuthology.orchestra.run.smithi046.stdout: openssl-libs-1:3.0.7-27.el9.x86_64 2024-03-31T22:59:40.732 INFO:teuthology.orchestra.run.smithi046.stdout: util-linux-2.37.4-18.el9.x86_64 2024-03-31T22:59:40.732 INFO:teuthology.orchestra.run.smithi046.stdout: util-linux-core-2.37.4-18.el9.x86_64 2024-03-31T22:59:40.732 INFO:teuthology.orchestra.run.smithi046.stdout: util-linux-user-2.37.4-18.el9.x86_64 2024-03-31T22:59:40.732 INFO:teuthology.orchestra.run.smithi046.stdout: xfsdump-3.1.12-4.el9.x86_64 2024-03-31T22:59:40.732 INFO:teuthology.orchestra.run.smithi046.stdout: xfsprogs-6.3.0-1.el9.x86_64 2024-03-31T22:59:40.732 INFO:teuthology.orchestra.run.smithi046.stdout:Installed: 2024-03-31T22:59:40.732 INFO:teuthology.orchestra.run.smithi046.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-31T22:59:40.732 INFO:teuthology.orchestra.run.smithi046.stdout: automake-1.16.2-8.el9.noarch 2024-03-31T22:59:40.732 INFO:teuthology.orchestra.run.smithi046.stdout: bison-3.7.4-5.el9.x86_64 2024-03-31T22:59:40.732 INFO:teuthology.orchestra.run.smithi046.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-31T22:59:40.732 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-2:19.0.0-1977.g384f31f3.el9.x86_64 2024-03-31T22:59:40.733 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-base-2:19.0.0-1977.g384f31f3.el9.x86_64 2024-03-31T22:59:40.733 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-common-2:19.0.0-1977.g384f31f3.el9.x86_64 2024-03-31T22:59:40.733 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-fuse-2:19.0.0-1977.g384f31f3.el9.x86_64 2024-03-31T22:59:40.733 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-grafana-dashboards-2:19.0.0-1977.g384f31f3.el9.noarch 2024-03-31T22:59:40.733 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-immutable-object-cache-2:19.0.0-1977.g384f31f3.el9.x86_64 2024-03-31T22:59:40.733 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-mds-2:19.0.0-1977.g384f31f3.el9.x86_64 2024-03-31T22:59:40.733 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-mgr-2:19.0.0-1977.g384f31f3.el9.x86_64 2024-03-31T22:59:40.733 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-mgr-cephadm-2:19.0.0-1977.g384f31f3.el9.noarch 2024-03-31T22:59:40.733 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-mgr-dashboard-2:19.0.0-1977.g384f31f3.el9.noarch 2024-03-31T22:59:40.733 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1977.g384f31f3.el9.noarch 2024-03-31T22:59:40.734 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-mgr-modules-core-2:19.0.0-1977.g384f31f3.el9.noarch 2024-03-31T22:59:40.734 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-mgr-rook-2:19.0.0-1977.g384f31f3.el9.noarch 2024-03-31T22:59:40.734 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-mon-2:19.0.0-1977.g384f31f3.el9.x86_64 2024-03-31T22:59:40.734 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-osd-2:19.0.0-1977.g384f31f3.el9.x86_64 2024-03-31T22:59:40.734 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-prometheus-alerts-2:19.0.0-1977.g384f31f3.el9.noarch 2024-03-31T22:59:40.734 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-radosgw-2:19.0.0-1977.g384f31f3.el9.x86_64 2024-03-31T22:59:40.734 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-selinux-2:19.0.0-1977.g384f31f3.el9.x86_64 2024-03-31T22:59:40.734 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-test-2:19.0.0-1977.g384f31f3.el9.x86_64 2024-03-31T22:59:40.734 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-volume-2:19.0.0-1977.g384f31f3.el9.noarch 2024-03-31T22:59:40.734 INFO:teuthology.orchestra.run.smithi046.stdout: cephadm-2:19.0.0-1977.g384f31f3.el9.noarch 2024-03-31T22:59:40.734 INFO:teuthology.orchestra.run.smithi046.stdout: cephfs-mirror-2:19.0.0-1977.g384f31f3.el9.x86_64 2024-03-31T22:59:40.735 INFO:teuthology.orchestra.run.smithi046.stdout: cephfs-shell-2:19.0.0-1977.g384f31f3.el9.x86_64 2024-03-31T22:59:40.735 INFO:teuthology.orchestra.run.smithi046.stdout: cephfs-top-2:19.0.0-1977.g384f31f3.el9.noarch 2024-03-31T22:59:40.735 INFO:teuthology.orchestra.run.smithi046.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-03-31T22:59:40.735 INFO:teuthology.orchestra.run.smithi046.stdout: flex-2.6.4-9.el9.x86_64 2024-03-31T22:59:40.735 INFO:teuthology.orchestra.run.smithi046.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-31T22:59:40.735 INFO:teuthology.orchestra.run.smithi046.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-31T22:59:40.735 INFO:teuthology.orchestra.run.smithi046.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-31T22:59:40.735 INFO:teuthology.orchestra.run.smithi046.stdout: fuse-2.9.9-15.el9.x86_64 2024-03-31T22:59:40.735 INFO:teuthology.orchestra.run.smithi046.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-31T22:59:40.735 INFO:teuthology.orchestra.run.smithi046.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-31T22:59:40.736 INFO:teuthology.orchestra.run.smithi046.stdout: jq-1.6-16.el9.x86_64 2024-03-31T22:59:40.736 INFO:teuthology.orchestra.run.smithi046.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-03-31T22:59:40.736 INFO:teuthology.orchestra.run.smithi046.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-31T22:59:40.736 INFO:teuthology.orchestra.run.smithi046.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-31T22:59:40.736 INFO:teuthology.orchestra.run.smithi046.stdout: libcephfs-devel-2:19.0.0-1977.g384f31f3.el9.x86_64 2024-03-31T22:59:40.736 INFO:teuthology.orchestra.run.smithi046.stdout: libcephfs2-2:19.0.0-1977.g384f31f3.el9.x86_64 2024-03-31T22:59:40.736 INFO:teuthology.orchestra.run.smithi046.stdout: libcephsqlite-2:19.0.0-1977.g384f31f3.el9.x86_64 2024-03-31T22:59:40.736 INFO:teuthology.orchestra.run.smithi046.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-31T22:59:40.736 INFO:teuthology.orchestra.run.smithi046.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-31T22:59:40.736 INFO:teuthology.orchestra.run.smithi046.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-31T22:59:40.737 INFO:teuthology.orchestra.run.smithi046.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-31T22:59:40.737 INFO:teuthology.orchestra.run.smithi046.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-31T22:59:40.737 INFO:teuthology.orchestra.run.smithi046.stdout: librados-devel-2:19.0.0-1977.g384f31f3.el9.x86_64 2024-03-31T22:59:40.737 INFO:teuthology.orchestra.run.smithi046.stdout: libradosstriper1-2:19.0.0-1977.g384f31f3.el9.x86_64 2024-03-31T22:59:40.737 INFO:teuthology.orchestra.run.smithi046.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-31T22:59:40.737 INFO:teuthology.orchestra.run.smithi046.stdout: librgw2-2:19.0.0-1977.g384f31f3.el9.x86_64 2024-03-31T22:59:40.737 INFO:teuthology.orchestra.run.smithi046.stdout: libtool-2.4.6-45.el9.x86_64 2024-03-31T22:59:40.737 INFO:teuthology.orchestra.run.smithi046.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-31T22:59:40.738 INFO:teuthology.orchestra.run.smithi046.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-03-31T22:59:40.738 INFO:teuthology.orchestra.run.smithi046.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-03-31T22:59:40.738 INFO:teuthology.orchestra.run.smithi046.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-31T22:59:40.738 INFO:teuthology.orchestra.run.smithi046.stdout: lua-5.4.4-4.el9.x86_64 2024-03-31T22:59:40.738 INFO:teuthology.orchestra.run.smithi046.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-31T22:59:40.738 INFO:teuthology.orchestra.run.smithi046.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-31T22:59:40.738 INFO:teuthology.orchestra.run.smithi046.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-31T22:59:40.738 INFO:teuthology.orchestra.run.smithi046.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-31T22:59:40.738 INFO:teuthology.orchestra.run.smithi046.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-31T22:59:40.739 INFO:teuthology.orchestra.run.smithi046.stdout: openssl-devel-1:3.0.7-27.el9.x86_64 2024-03-31T22:59:40.739 INFO:teuthology.orchestra.run.smithi046.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-31T22:59:40.739 INFO:teuthology.orchestra.run.smithi046.stdout: postgresql-13.14-1.el9.x86_64 2024-03-31T22:59:40.739 INFO:teuthology.orchestra.run.smithi046.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-03-31T22:59:40.739 INFO:teuthology.orchestra.run.smithi046.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-03-31T22:59:40.739 INFO:teuthology.orchestra.run.smithi046.stdout: postgresql-server-13.14-1.el9.x86_64 2024-03-31T22:59:40.739 INFO:teuthology.orchestra.run.smithi046.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-31T22:59:40.739 INFO:teuthology.orchestra.run.smithi046.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-31T22:59:40.739 INFO:teuthology.orchestra.run.smithi046.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-31T22:59:40.739 INFO:teuthology.orchestra.run.smithi046.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-03-31T22:59:40.739 INFO:teuthology.orchestra.run.smithi046.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-31T22:59:40.739 INFO:teuthology.orchestra.run.smithi046.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-31T22:59:40.739 INFO:teuthology.orchestra.run.smithi046.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-31T22:59:40.739 INFO:teuthology.orchestra.run.smithi046.stdout: python3-ceph-argparse-2:19.0.0-1977.g384f31f3.el9.x86_64 2024-03-31T22:59:40.740 INFO:teuthology.orchestra.run.smithi046.stdout: python3-ceph-common-2:19.0.0-1977.g384f31f3.el9.x86_64 2024-03-31T22:59:40.740 INFO:teuthology.orchestra.run.smithi046.stdout: python3-cephfs-2:19.0.0-1977.g384f31f3.el9.x86_64 2024-03-31T22:59:40.740 INFO:teuthology.orchestra.run.smithi046.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-31T22:59:40.740 INFO:teuthology.orchestra.run.smithi046.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-31T22:59:40.740 INFO:teuthology.orchestra.run.smithi046.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-31T22:59:40.740 INFO:teuthology.orchestra.run.smithi046.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-31T22:59:40.740 INFO:teuthology.orchestra.run.smithi046.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-03-31T22:59:40.740 INFO:teuthology.orchestra.run.smithi046.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-03-31T22:59:40.740 INFO:teuthology.orchestra.run.smithi046.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-31T22:59:40.741 INFO:teuthology.orchestra.run.smithi046.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-31T22:59:40.741 INFO:teuthology.orchestra.run.smithi046.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-31T22:59:40.741 INFO:teuthology.orchestra.run.smithi046.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-31T22:59:40.741 INFO:teuthology.orchestra.run.smithi046.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-31T22:59:40.741 INFO:teuthology.orchestra.run.smithi046.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-31T22:59:40.741 INFO:teuthology.orchestra.run.smithi046.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-31T22:59:40.741 INFO:teuthology.orchestra.run.smithi046.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-31T22:59:40.741 INFO:teuthology.orchestra.run.smithi046.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-31T22:59:40.742 INFO:teuthology.orchestra.run.smithi046.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-31T22:59:40.742 INFO:teuthology.orchestra.run.smithi046.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-31T22:59:40.742 INFO:teuthology.orchestra.run.smithi046.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-31T22:59:40.742 INFO:teuthology.orchestra.run.smithi046.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-31T22:59:40.742 INFO:teuthology.orchestra.run.smithi046.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-31T22:59:40.742 INFO:teuthology.orchestra.run.smithi046.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-31T22:59:40.742 INFO:teuthology.orchestra.run.smithi046.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-31T22:59:40.742 INFO:teuthology.orchestra.run.smithi046.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-31T22:59:40.742 INFO:teuthology.orchestra.run.smithi046.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-31T22:59:40.742 INFO:teuthology.orchestra.run.smithi046.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-31T22:59:40.742 INFO:teuthology.orchestra.run.smithi046.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-31T22:59:40.742 INFO:teuthology.orchestra.run.smithi046.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-31T22:59:40.742 INFO:teuthology.orchestra.run.smithi046.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-31T22:59:40.743 INFO:teuthology.orchestra.run.smithi046.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-31T22:59:40.743 INFO:teuthology.orchestra.run.smithi046.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-31T22:59:40.743 INFO:teuthology.orchestra.run.smithi046.stdout: python3-ply-3.11-14.el9.noarch 2024-03-31T22:59:40.743 INFO:teuthology.orchestra.run.smithi046.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-31T22:59:40.743 INFO:teuthology.orchestra.run.smithi046.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-31T22:59:40.743 INFO:teuthology.orchestra.run.smithi046.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-31T22:59:40.743 INFO:teuthology.orchestra.run.smithi046.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-31T22:59:40.743 INFO:teuthology.orchestra.run.smithi046.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-31T22:59:40.743 INFO:teuthology.orchestra.run.smithi046.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-31T22:59:40.744 INFO:teuthology.orchestra.run.smithi046.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-31T22:59:40.744 INFO:teuthology.orchestra.run.smithi046.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-03-31T22:59:40.744 INFO:teuthology.orchestra.run.smithi046.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-31T22:59:40.744 INFO:teuthology.orchestra.run.smithi046.stdout: python3-rados-2:19.0.0-1977.g384f31f3.el9.x86_64 2024-03-31T22:59:40.744 INFO:teuthology.orchestra.run.smithi046.stdout: python3-rbd-2:19.0.0-1977.g384f31f3.el9.x86_64 2024-03-31T22:59:40.744 INFO:teuthology.orchestra.run.smithi046.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-31T22:59:40.744 INFO:teuthology.orchestra.run.smithi046.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-31T22:59:40.744 INFO:teuthology.orchestra.run.smithi046.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-31T22:59:40.744 INFO:teuthology.orchestra.run.smithi046.stdout: python3-rgw-2:19.0.0-1977.g384f31f3.el9.x86_64 2024-03-31T22:59:40.744 INFO:teuthology.orchestra.run.smithi046.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-31T22:59:40.744 INFO:teuthology.orchestra.run.smithi046.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-31T22:59:40.744 INFO:teuthology.orchestra.run.smithi046.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-31T22:59:40.744 INFO:teuthology.orchestra.run.smithi046.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-31T22:59:40.744 INFO:teuthology.orchestra.run.smithi046.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-31T22:59:40.745 INFO:teuthology.orchestra.run.smithi046.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-31T22:59:40.745 INFO:teuthology.orchestra.run.smithi046.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-31T22:59:40.745 INFO:teuthology.orchestra.run.smithi046.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-03-31T22:59:40.745 INFO:teuthology.orchestra.run.smithi046.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-31T22:59:40.745 INFO:teuthology.orchestra.run.smithi046.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-31T22:59:40.745 INFO:teuthology.orchestra.run.smithi046.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-31T22:59:40.745 INFO:teuthology.orchestra.run.smithi046.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-31T22:59:40.745 INFO:teuthology.orchestra.run.smithi046.stdout: qatlib-24.02.0-1.el9.x86_64 2024-03-31T22:59:40.745 INFO:teuthology.orchestra.run.smithi046.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-03-31T22:59:40.745 INFO:teuthology.orchestra.run.smithi046.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-03-31T22:59:40.745 INFO:teuthology.orchestra.run.smithi046.stdout: rbd-fuse-2:19.0.0-1977.g384f31f3.el9.x86_64 2024-03-31T22:59:40.745 INFO:teuthology.orchestra.run.smithi046.stdout: rbd-mirror-2:19.0.0-1977.g384f31f3.el9.x86_64 2024-03-31T22:59:40.745 INFO:teuthology.orchestra.run.smithi046.stdout: rbd-nbd-2:19.0.0-1977.g384f31f3.el9.x86_64 2024-03-31T22:59:40.746 INFO:teuthology.orchestra.run.smithi046.stdout: re2-1:20211101-3.el9.x86_64 2024-03-31T22:59:40.746 INFO:teuthology.orchestra.run.smithi046.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-31T22:59:40.746 INFO:teuthology.orchestra.run.smithi046.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-31T22:59:40.746 INFO:teuthology.orchestra.run.smithi046.stdout: uuid-1.6.2-55.el9.x86_64 2024-03-31T22:59:40.746 INFO:teuthology.orchestra.run.smithi046.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-03-31T22:59:40.746 INFO:teuthology.orchestra.run.smithi046.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-31T22:59:40.747 INFO:teuthology.orchestra.run.smithi046.stdout: zlib-devel-1.2.11-40.el9.x86_64 2024-03-31T22:59:40.747 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-31T22:59:40.747 INFO:teuthology.orchestra.run.smithi046.stdout:Complete! 2024-03-31T22:59:40.837 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-31T22:59:40.837 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : ceph-2:19.0.0-1977.g384f31f3.el9.x86_64 1/198 2024-03-31T22:59:40.837 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : ceph-base-2:19.0.0-1977.g384f31f3.el9.x86_64 2/198 2024-03-31T22:59:40.837 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : ceph-common-2:19.0.0-1977.g384f31f3.el9.x86_64 3/198 2024-03-31T22:59:40.837 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : ceph-fuse-2:19.0.0-1977.g384f31f3.el9.x86_64 4/198 2024-03-31T22:59:40.837 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1977.g384f3 5/198 2024-03-31T22:59:40.837 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : ceph-mds-2:19.0.0-1977.g384f31f3.el9.x86_64 6/198 2024-03-31T22:59:40.837 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : ceph-mgr-2:19.0.0-1977.g384f31f3.el9.x86_64 7/198 2024-03-31T22:59:40.837 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : ceph-mon-2:19.0.0-1977.g384f31f3.el9.x86_64 8/198 2024-03-31T22:59:40.837 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : ceph-osd-2:19.0.0-1977.g384f31f3.el9.x86_64 9/198 2024-03-31T22:59:40.838 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : ceph-radosgw-2:19.0.0-1977.g384f31f3.el9.x86_64 10/198 2024-03-31T22:59:40.838 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : ceph-selinux-2:19.0.0-1977.g384f31f3.el9.x86_64 11/198 2024-03-31T22:59:40.838 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : ceph-test-2:19.0.0-1977.g384f31f3.el9.x86_64 12/198 2024-03-31T22:59:40.838 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : cephfs-mirror-2:19.0.0-1977.g384f31f3.el9.x86_64 13/198 2024-03-31T22:59:40.838 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : cephfs-shell-2:19.0.0-1977.g384f31f3.el9.x86_64 14/198 2024-03-31T22:59:40.838 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : libcephfs-devel-2:19.0.0-1977.g384f31f3.el9.x86_ 15/198 2024-03-31T22:59:40.838 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : libcephfs2-2:19.0.0-1977.g384f31f3.el9.x86_64 16/198 2024-03-31T22:59:40.838 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : libcephsqlite-2:19.0.0-1977.g384f31f3.el9.x86_64 17/198 2024-03-31T22:59:40.838 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : librados-devel-2:19.0.0-1977.g384f31f3.el9.x86_6 18/198 2024-03-31T22:59:40.838 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : libradosstriper1-2:19.0.0-1977.g384f31f3.el9.x86 19/198 2024-03-31T22:59:40.838 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : librgw2-2:19.0.0-1977.g384f31f3.el9.x86_64 20/198 2024-03-31T22:59:40.838 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1977.g384f31f3.el 21/198 2024-03-31T22:59:40.839 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-ceph-common-2:19.0.0-1977.g384f31f3.el9. 22/198 2024-03-31T22:59:40.839 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-cephfs-2:19.0.0-1977.g384f31f3.el9.x86_6 23/198 2024-03-31T22:59:40.839 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-rados-2:19.0.0-1977.g384f31f3.el9.x86_64 24/198 2024-03-31T22:59:40.839 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-rbd-2:19.0.0-1977.g384f31f3.el9.x86_64 25/198 2024-03-31T22:59:40.839 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-rgw-2:19.0.0-1977.g384f31f3.el9.x86_64 26/198 2024-03-31T22:59:40.839 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : rbd-fuse-2:19.0.0-1977.g384f31f3.el9.x86_64 27/198 2024-03-31T22:59:40.839 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : rbd-mirror-2:19.0.0-1977.g384f31f3.el9.x86_64 28/198 2024-03-31T22:59:40.839 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : rbd-nbd-2:19.0.0-1977.g384f31f3.el9.x86_64 29/198 2024-03-31T22:59:40.839 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1977.g384f31f3. 30/198 2024-03-31T22:59:40.839 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1977.g384f31f3.el9.noa 31/198 2024-03-31T22:59:40.839 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1977.g384f31f3.el9.n 32/198 2024-03-31T22:59:40.856 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1977.g384 33/198 2024-03-31T22:59:40.856 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1977.g384f31f3.el 34/198 2024-03-31T22:59:40.856 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1977.g384f31f3.el9.noarch 35/198 2024-03-31T22:59:40.856 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1977.g384f31f3.e 36/198 2024-03-31T22:59:40.856 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : ceph-volume-2:19.0.0-1977.g384f31f3.el9.noarch 37/198 2024-03-31T22:59:40.856 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : cephadm-2:19.0.0-1977.g384f31f3.el9.noarch 38/198 2024-03-31T22:59:40.856 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : cephfs-top-2:19.0.0-1977.g384f31f3.el9.noarch 39/198 2024-03-31T22:59:40.856 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : automake-1.16.2-8.el9.noarch 40/198 2024-03-31T22:59:40.857 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : bison-3.7.4-5.el9.x86_64 41/198 2024-03-31T22:59:40.857 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 42/198 2024-03-31T22:59:40.857 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : flex-2.6.4-9.el9.x86_64 43/198 2024-03-31T22:59:40.857 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 44/198 2024-03-31T22:59:40.857 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 45/198 2024-03-31T22:59:40.857 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 46/198 2024-03-31T22:59:40.857 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 47/198 2024-03-31T22:59:40.857 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 48/198 2024-03-31T22:59:40.857 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 49/198 2024-03-31T22:59:40.857 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 50/198 2024-03-31T22:59:40.857 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 51/198 2024-03-31T22:59:40.857 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 52/198 2024-03-31T22:59:40.857 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 53/198 2024-03-31T22:59:40.857 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : lua-5.4.4-4.el9.x86_64 54/198 2024-03-31T22:59:40.858 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 55/198 2024-03-31T22:59:40.858 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 56/198 2024-03-31T22:59:40.858 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 57/198 2024-03-31T22:59:40.858 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : postgresql-13.14-1.el9.x86_64 58/198 2024-03-31T22:59:40.858 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 59/198 2024-03-31T22:59:40.858 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 60/198 2024-03-31T22:59:40.858 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 61/198 2024-03-31T22:59:40.858 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 62/198 2024-03-31T22:59:40.858 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 63/198 2024-03-31T22:59:40.858 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 64/198 2024-03-31T22:59:40.858 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 65/198 2024-03-31T22:59:40.858 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 66/198 2024-03-31T22:59:40.858 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 67/198 2024-03-31T22:59:40.859 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 68/198 2024-03-31T22:59:40.859 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 69/198 2024-03-31T22:59:40.859 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 70/198 2024-03-31T22:59:40.859 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 71/198 2024-03-31T22:59:40.859 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 72/198 2024-03-31T22:59:40.859 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 73/198 2024-03-31T22:59:40.859 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 74/198 2024-03-31T22:59:40.859 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 75/198 2024-03-31T22:59:40.859 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 76/198 2024-03-31T22:59:40.859 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 77/198 2024-03-31T22:59:40.859 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 78/198 2024-03-31T22:59:40.859 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 79/198 2024-03-31T22:59:40.859 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 80/198 2024-03-31T22:59:40.860 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 81/198 2024-03-31T22:59:40.860 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 82/198 2024-03-31T22:59:40.860 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 83/198 2024-03-31T22:59:40.860 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 84/198 2024-03-31T22:59:40.860 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 85/198 2024-03-31T22:59:40.860 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 86/198 2024-03-31T22:59:40.860 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 87/198 2024-03-31T22:59:40.860 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : zlib-devel-1.2.11-40.el9.x86_64 88/198 2024-03-31T22:59:40.860 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 89/198 2024-03-31T22:59:40.860 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 90/198 2024-03-31T22:59:40.861 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 91/198 2024-03-31T22:59:40.861 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-ply-3.11-14.el9.noarch 92/198 2024-03-31T22:59:40.861 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 93/198 2024-03-31T22:59:40.861 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 94/198 2024-03-31T22:59:40.861 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : jq-1.6-16.el9.x86_64 95/198 2024-03-31T22:59:40.861 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 96/198 2024-03-31T22:59:40.862 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 97/198 2024-03-31T22:59:40.862 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 98/198 2024-03-31T22:59:40.862 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 99/198 2024-03-31T22:59:40.862 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 100/198 2024-03-31T22:59:40.862 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : openssl-devel-1:3.0.7-27.el9.x86_64 101/198 2024-03-31T22:59:40.862 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 102/198 2024-03-31T22:59:40.862 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 103/198 2024-03-31T22:59:40.862 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 104/198 2024-03-31T22:59:40.862 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 105/198 2024-03-31T22:59:40.862 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 106/198 2024-03-31T22:59:40.862 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 107/198 2024-03-31T22:59:40.862 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 108/198 2024-03-31T22:59:40.863 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 109/198 2024-03-31T22:59:40.863 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 110/198 2024-03-31T22:59:40.863 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 111/198 2024-03-31T22:59:40.863 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 112/198 2024-03-31T22:59:40.863 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 113/198 2024-03-31T22:59:40.863 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 114/198 2024-03-31T22:59:40.863 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 115/198 2024-03-31T22:59:40.863 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 116/198 2024-03-31T22:59:40.863 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 117/198 2024-03-31T22:59:40.863 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 118/198 2024-03-31T22:59:40.863 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 119/198 2024-03-31T22:59:40.863 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 120/198 2024-03-31T22:59:40.863 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 121/198 2024-03-31T22:59:40.863 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 122/198 2024-03-31T22:59:40.864 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 123/198 2024-03-31T22:59:40.864 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 124/198 2024-03-31T22:59:40.864 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 125/198 2024-03-31T22:59:40.864 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 126/198 2024-03-31T22:59:40.864 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 127/198 2024-03-31T22:59:40.864 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 128/198 2024-03-31T22:59:40.864 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 129/198 2024-03-31T22:59:40.864 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 130/198 2024-03-31T22:59:40.864 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 131/198 2024-03-31T22:59:40.864 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 132/198 2024-03-31T22:59:40.865 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 133/198 2024-03-31T22:59:40.865 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 134/198 2024-03-31T22:59:40.865 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 135/198 2024-03-31T22:59:40.865 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 136/198 2024-03-31T22:59:40.865 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 137/198 2024-03-31T22:59:40.865 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 138/198 2024-03-31T22:59:40.865 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 139/198 2024-03-31T22:59:40.865 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 140/198 2024-03-31T22:59:40.865 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 141/198 2024-03-31T22:59:40.865 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 142/198 2024-03-31T22:59:40.865 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 143/198 2024-03-31T22:59:40.865 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 144/198 2024-03-31T22:59:40.865 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 145/198 2024-03-31T22:59:40.865 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 146/198 2024-03-31T22:59:40.866 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 147/198 2024-03-31T22:59:40.866 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 148/198 2024-03-31T22:59:40.866 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : re2-1:20211101-3.el9.x86_64 149/198 2024-03-31T22:59:40.866 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 150/198 2024-03-31T22:59:40.866 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : librados2-2:19.0.0-1977.g384f31f3.el9.x86_64 151/198 2024-03-31T22:59:40.866 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 152/198 2024-03-31T22:59:40.866 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : librbd1-2:19.0.0-1977.g384f31f3.el9.x86_64 153/198 2024-03-31T22:59:40.866 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 154/198 2024-03-31T22:59:40.866 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : NetworkManager-1:1.47.2-1.el9.x86_64 155/198 2024-03-31T22:59:40.866 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : NetworkManager-1:1.44.0-2.el9.x86_64 156/198 2024-03-31T22:59:40.866 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : NetworkManager-libnm-1:1.47.2-1.el9.x86_64 157/198 2024-03-31T22:59:40.866 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 158/198 2024-03-31T22:59:40.867 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : NetworkManager-team-1:1.47.2-1.el9.x86_64 159/198 2024-03-31T22:59:40.867 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : NetworkManager-team-1:1.44.0-2.el9.x86_64 160/198 2024-03-31T22:59:40.867 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : NetworkManager-tui-1:1.47.2-1.el9.x86_64 161/198 2024-03-31T22:59:40.867 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : NetworkManager-tui-1:1.44.0-2.el9.x86_64 162/198 2024-03-31T22:59:40.867 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : xfsdump-3.1.12-4.el9.x86_64 163/198 2024-03-31T22:59:40.867 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : xfsdump-3.1.12-3.el9.x86_64 164/198 2024-03-31T22:59:40.867 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : acl-2.3.1-4.el9.x86_64 165/198 2024-03-31T22:59:40.867 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : acl-2.3.1-3.el9.x86_64 166/198 2024-03-31T22:59:40.867 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 167/198 2024-03-31T22:59:40.867 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : elfutils-debuginfod-client-0.189-3.el9.x86_64 168/198 2024-03-31T22:59:40.867 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 169/198 2024-03-31T22:59:40.867 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : elfutils-libelf-0.189-3.el9.x86_64 170/198 2024-03-31T22:59:40.867 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 171/198 2024-03-31T22:59:40.867 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : elfutils-libs-0.189-3.el9.x86_64 172/198 2024-03-31T22:59:40.868 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : libacl-2.3.1-4.el9.x86_64 173/198 2024-03-31T22:59:40.868 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : libacl-2.3.1-3.el9.x86_64 174/198 2024-03-31T22:59:40.868 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : libblkid-2.37.4-18.el9.x86_64 175/198 2024-03-31T22:59:40.868 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : libblkid-2.37.4-15.el9.x86_64 176/198 2024-03-31T22:59:40.868 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : libfdisk-2.37.4-18.el9.x86_64 177/198 2024-03-31T22:59:40.868 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : libfdisk-2.37.4-15.el9.x86_64 178/198 2024-03-31T22:59:40.868 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : libmount-2.37.4-18.el9.x86_64 179/198 2024-03-31T22:59:40.868 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : libmount-2.37.4-15.el9.x86_64 180/198 2024-03-31T22:59:40.868 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : libsmartcols-2.37.4-18.el9.x86_64 181/198 2024-03-31T22:59:40.868 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : libsmartcols-2.37.4-15.el9.x86_64 182/198 2024-03-31T22:59:40.868 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : libuuid-2.37.4-18.el9.x86_64 183/198 2024-03-31T22:59:40.868 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : libuuid-2.37.4-15.el9.x86_64 184/198 2024-03-31T22:59:40.868 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 185/198 2024-03-31T22:59:40.869 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : openssl-1:3.0.7-24.el9.x86_64 186/198 2024-03-31T22:59:40.869 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 187/198 2024-03-31T22:59:40.869 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : openssl-libs-1:3.0.7-24.el9.x86_64 188/198 2024-03-31T22:59:40.869 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : util-linux-2.37.4-18.el9.x86_64 189/198 2024-03-31T22:59:40.869 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : util-linux-2.37.4-15.el9.x86_64 190/198 2024-03-31T22:59:40.869 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : util-linux-core-2.37.4-18.el9.x86_64 191/198 2024-03-31T22:59:40.869 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : util-linux-core-2.37.4-15.el9.x86_64 192/198 2024-03-31T22:59:40.869 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : util-linux-user-2.37.4-18.el9.x86_64 193/198 2024-03-31T22:59:40.869 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : util-linux-user-2.37.4-15.el9.x86_64 194/198 2024-03-31T22:59:40.869 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 195/198 2024-03-31T22:59:40.869 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : xfsprogs-5.19.0-4.el9.x86_64 196/198 2024-03-31T22:59:40.870 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : libacl-devel-2.3.1-4.el9.x86_64 197/198 2024-03-31T22:59:40.870 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-volume-2:19.0.0-1977.g384f31f3.el9.noarch 133/198 2024-03-31T22:59:40.884 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: ceph-volume-2:19.0.0-1977.g384f31f3.el9.noarch 133/198 2024-03-31T22:59:40.884 INFO:teuthology.orchestra.run.smithi162.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-31T22:59:40.884 INFO:teuthology.orchestra.run.smithi162.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-31T22:59:40.885 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-31T22:59:41.193 DEBUG:teuthology.parallel:result is None 2024-03-31T22:59:42.380 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-osd-2:19.0.0-1977.g384f31f3.el9.x86_64 134/198 2024-03-31T22:59:42.406 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: ceph-osd-2:19.0.0-1977.g384f31f3.el9.x86_64 134/198 2024-03-31T22:59:42.406 INFO:teuthology.orchestra.run.smithi162.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-31T22:59:42.406 INFO:teuthology.orchestra.run.smithi162.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-31T22:59:42.406 INFO:teuthology.orchestra.run.smithi162.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-31T22:59:42.406 INFO:teuthology.orchestra.run.smithi162.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-31T22:59:42.406 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-31T22:59:42.631 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-mds-2:19.0.0-1977.g384f31f3.el9.x86_64 135/198 2024-03-31T22:59:42.653 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: ceph-mds-2:19.0.0-1977.g384f31f3.el9.x86_64 135/198 2024-03-31T22:59:42.654 INFO:teuthology.orchestra.run.smithi162.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-31T22:59:42.654 INFO:teuthology.orchestra.run.smithi162.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-31T22:59:42.654 INFO:teuthology.orchestra.run.smithi162.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-31T22:59:42.654 INFO:teuthology.orchestra.run.smithi162.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-31T22:59:42.654 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-31T22:59:43.098 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-mon-2:19.0.0-1977.g384f31f3.el9.x86_64 136/198 2024-03-31T22:59:43.121 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: ceph-mon-2:19.0.0-1977.g384f31f3.el9.x86_64 136/198 2024-03-31T22:59:43.121 INFO:teuthology.orchestra.run.smithi162.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-31T22:59:43.121 INFO:teuthology.orchestra.run.smithi162.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-31T22:59:43.121 INFO:teuthology.orchestra.run.smithi162.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-31T22:59:43.122 INFO:teuthology.orchestra.run.smithi162.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-31T22:59:43.122 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-31T22:59:43.208 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : automake-1.16.2-8.el9.noarch 137/198 2024-03-31T22:59:43.627 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: cephadm-2:19.0.0-1977.g384f31f3.el9.noarch 138/198 2024-03-31T22:59:43.648 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : cephadm-2:19.0.0-1977.g384f31f3.el9.noarch 138/198 2024-03-31T22:59:43.680 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1977.g384f31f3.e 139/198 2024-03-31T22:59:43.741 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1977.g384f31f3. 140/198 2024-03-31T22:59:43.765 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : libacl-devel-2.3.1-3.el9.x86_64 198/198 2024-03-31T22:59:43.765 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-03-31T22:59:43.765 INFO:teuthology.orchestra.run.smithi142.stdout:Upgraded: 2024-03-31T22:59:43.765 INFO:teuthology.orchestra.run.smithi142.stdout: NetworkManager-1:1.47.2-1.el9.x86_64 2024-03-31T22:59:43.765 INFO:teuthology.orchestra.run.smithi142.stdout: NetworkManager-libnm-1:1.47.2-1.el9.x86_64 2024-03-31T22:59:43.765 INFO:teuthology.orchestra.run.smithi142.stdout: NetworkManager-team-1:1.47.2-1.el9.x86_64 2024-03-31T22:59:43.766 INFO:teuthology.orchestra.run.smithi142.stdout: NetworkManager-tui-1:1.47.2-1.el9.x86_64 2024-03-31T22:59:43.766 INFO:teuthology.orchestra.run.smithi142.stdout: acl-2.3.1-4.el9.x86_64 2024-03-31T22:59:43.766 INFO:teuthology.orchestra.run.smithi142.stdout: elfutils-debuginfod-client-0.190-2.el9.x86_64 2024-03-31T22:59:43.766 INFO:teuthology.orchestra.run.smithi142.stdout: elfutils-libelf-0.190-2.el9.x86_64 2024-03-31T22:59:43.766 INFO:teuthology.orchestra.run.smithi142.stdout: elfutils-libs-0.190-2.el9.x86_64 2024-03-31T22:59:43.766 INFO:teuthology.orchestra.run.smithi142.stdout: libacl-2.3.1-4.el9.x86_64 2024-03-31T22:59:43.766 INFO:teuthology.orchestra.run.smithi142.stdout: libacl-devel-2.3.1-4.el9.x86_64 2024-03-31T22:59:43.766 INFO:teuthology.orchestra.run.smithi142.stdout: libblkid-2.37.4-18.el9.x86_64 2024-03-31T22:59:43.766 INFO:teuthology.orchestra.run.smithi142.stdout: libfdisk-2.37.4-18.el9.x86_64 2024-03-31T22:59:43.766 INFO:teuthology.orchestra.run.smithi142.stdout: libmount-2.37.4-18.el9.x86_64 2024-03-31T22:59:43.766 INFO:teuthology.orchestra.run.smithi142.stdout: librados2-2:19.0.0-1977.g384f31f3.el9.x86_64 2024-03-31T22:59:43.766 INFO:teuthology.orchestra.run.smithi142.stdout: librbd1-2:19.0.0-1977.g384f31f3.el9.x86_64 2024-03-31T22:59:43.766 INFO:teuthology.orchestra.run.smithi142.stdout: libsmartcols-2.37.4-18.el9.x86_64 2024-03-31T22:59:43.767 INFO:teuthology.orchestra.run.smithi142.stdout: libuuid-2.37.4-18.el9.x86_64 2024-03-31T22:59:43.767 INFO:teuthology.orchestra.run.smithi142.stdout: openssl-1:3.0.7-27.el9.x86_64 2024-03-31T22:59:43.767 INFO:teuthology.orchestra.run.smithi142.stdout: openssl-libs-1:3.0.7-27.el9.x86_64 2024-03-31T22:59:43.767 INFO:teuthology.orchestra.run.smithi142.stdout: util-linux-2.37.4-18.el9.x86_64 2024-03-31T22:59:43.767 INFO:teuthology.orchestra.run.smithi142.stdout: util-linux-core-2.37.4-18.el9.x86_64 2024-03-31T22:59:43.767 INFO:teuthology.orchestra.run.smithi142.stdout: util-linux-user-2.37.4-18.el9.x86_64 2024-03-31T22:59:43.767 INFO:teuthology.orchestra.run.smithi142.stdout: xfsdump-3.1.12-4.el9.x86_64 2024-03-31T22:59:43.767 INFO:teuthology.orchestra.run.smithi142.stdout: xfsprogs-6.3.0-1.el9.x86_64 2024-03-31T22:59:43.767 INFO:teuthology.orchestra.run.smithi142.stdout:Installed: 2024-03-31T22:59:43.767 INFO:teuthology.orchestra.run.smithi142.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-31T22:59:43.767 INFO:teuthology.orchestra.run.smithi142.stdout: automake-1.16.2-8.el9.noarch 2024-03-31T22:59:43.767 INFO:teuthology.orchestra.run.smithi142.stdout: bison-3.7.4-5.el9.x86_64 2024-03-31T22:59:43.767 INFO:teuthology.orchestra.run.smithi142.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-31T22:59:43.768 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-2:19.0.0-1977.g384f31f3.el9.x86_64 2024-03-31T22:59:43.768 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-base-2:19.0.0-1977.g384f31f3.el9.x86_64 2024-03-31T22:59:43.768 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-common-2:19.0.0-1977.g384f31f3.el9.x86_64 2024-03-31T22:59:43.768 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-fuse-2:19.0.0-1977.g384f31f3.el9.x86_64 2024-03-31T22:59:43.768 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-grafana-dashboards-2:19.0.0-1977.g384f31f3.el9.noarch 2024-03-31T22:59:43.768 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-immutable-object-cache-2:19.0.0-1977.g384f31f3.el9.x86_64 2024-03-31T22:59:43.768 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-mds-2:19.0.0-1977.g384f31f3.el9.x86_64 2024-03-31T22:59:43.768 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-mgr-2:19.0.0-1977.g384f31f3.el9.x86_64 2024-03-31T22:59:43.768 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-mgr-cephadm-2:19.0.0-1977.g384f31f3.el9.noarch 2024-03-31T22:59:43.768 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-mgr-dashboard-2:19.0.0-1977.g384f31f3.el9.noarch 2024-03-31T22:59:43.768 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1977.g384f31f3.el9.noarch 2024-03-31T22:59:43.768 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-mgr-modules-core-2:19.0.0-1977.g384f31f3.el9.noarch 2024-03-31T22:59:43.768 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-mgr-rook-2:19.0.0-1977.g384f31f3.el9.noarch 2024-03-31T22:59:43.768 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-mon-2:19.0.0-1977.g384f31f3.el9.x86_64 2024-03-31T22:59:43.769 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-osd-2:19.0.0-1977.g384f31f3.el9.x86_64 2024-03-31T22:59:43.769 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-prometheus-alerts-2:19.0.0-1977.g384f31f3.el9.noarch 2024-03-31T22:59:43.769 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-radosgw-2:19.0.0-1977.g384f31f3.el9.x86_64 2024-03-31T22:59:43.769 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-selinux-2:19.0.0-1977.g384f31f3.el9.x86_64 2024-03-31T22:59:43.769 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-test-2:19.0.0-1977.g384f31f3.el9.x86_64 2024-03-31T22:59:43.769 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-volume-2:19.0.0-1977.g384f31f3.el9.noarch 2024-03-31T22:59:43.769 INFO:teuthology.orchestra.run.smithi142.stdout: cephadm-2:19.0.0-1977.g384f31f3.el9.noarch 2024-03-31T22:59:43.769 INFO:teuthology.orchestra.run.smithi142.stdout: cephfs-mirror-2:19.0.0-1977.g384f31f3.el9.x86_64 2024-03-31T22:59:43.769 INFO:teuthology.orchestra.run.smithi142.stdout: cephfs-shell-2:19.0.0-1977.g384f31f3.el9.x86_64 2024-03-31T22:59:43.769 INFO:teuthology.orchestra.run.smithi142.stdout: cephfs-top-2:19.0.0-1977.g384f31f3.el9.noarch 2024-03-31T22:59:43.769 INFO:teuthology.orchestra.run.smithi142.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-03-31T22:59:43.769 INFO:teuthology.orchestra.run.smithi142.stdout: flex-2.6.4-9.el9.x86_64 2024-03-31T22:59:43.769 INFO:teuthology.orchestra.run.smithi142.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-31T22:59:43.770 INFO:teuthology.orchestra.run.smithi142.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-31T22:59:43.770 INFO:teuthology.orchestra.run.smithi142.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-31T22:59:43.770 INFO:teuthology.orchestra.run.smithi142.stdout: fuse-2.9.9-15.el9.x86_64 2024-03-31T22:59:43.770 INFO:teuthology.orchestra.run.smithi142.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-31T22:59:43.770 INFO:teuthology.orchestra.run.smithi142.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-31T22:59:43.770 INFO:teuthology.orchestra.run.smithi142.stdout: jq-1.6-16.el9.x86_64 2024-03-31T22:59:43.770 INFO:teuthology.orchestra.run.smithi142.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-03-31T22:59:43.770 INFO:teuthology.orchestra.run.smithi142.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-31T22:59:43.770 INFO:teuthology.orchestra.run.smithi142.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-31T22:59:43.770 INFO:teuthology.orchestra.run.smithi142.stdout: libcephfs-devel-2:19.0.0-1977.g384f31f3.el9.x86_64 2024-03-31T22:59:43.770 INFO:teuthology.orchestra.run.smithi142.stdout: libcephfs2-2:19.0.0-1977.g384f31f3.el9.x86_64 2024-03-31T22:59:43.770 INFO:teuthology.orchestra.run.smithi142.stdout: libcephsqlite-2:19.0.0-1977.g384f31f3.el9.x86_64 2024-03-31T22:59:43.770 INFO:teuthology.orchestra.run.smithi142.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-31T22:59:43.771 INFO:teuthology.orchestra.run.smithi142.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-31T22:59:43.771 INFO:teuthology.orchestra.run.smithi142.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-31T22:59:43.771 INFO:teuthology.orchestra.run.smithi142.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-31T22:59:43.771 INFO:teuthology.orchestra.run.smithi142.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-31T22:59:43.771 INFO:teuthology.orchestra.run.smithi142.stdout: librados-devel-2:19.0.0-1977.g384f31f3.el9.x86_64 2024-03-31T22:59:43.771 INFO:teuthology.orchestra.run.smithi142.stdout: libradosstriper1-2:19.0.0-1977.g384f31f3.el9.x86_64 2024-03-31T22:59:43.771 INFO:teuthology.orchestra.run.smithi142.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-31T22:59:43.771 INFO:teuthology.orchestra.run.smithi142.stdout: librgw2-2:19.0.0-1977.g384f31f3.el9.x86_64 2024-03-31T22:59:43.771 INFO:teuthology.orchestra.run.smithi142.stdout: libtool-2.4.6-45.el9.x86_64 2024-03-31T22:59:43.771 INFO:teuthology.orchestra.run.smithi142.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-31T22:59:43.771 INFO:teuthology.orchestra.run.smithi142.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-03-31T22:59:43.771 INFO:teuthology.orchestra.run.smithi142.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-03-31T22:59:43.771 INFO:teuthology.orchestra.run.smithi142.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-31T22:59:43.771 INFO:teuthology.orchestra.run.smithi142.stdout: lua-5.4.4-4.el9.x86_64 2024-03-31T22:59:43.772 INFO:teuthology.orchestra.run.smithi142.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-31T22:59:43.772 INFO:teuthology.orchestra.run.smithi142.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-31T22:59:43.772 INFO:teuthology.orchestra.run.smithi142.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-31T22:59:43.772 INFO:teuthology.orchestra.run.smithi142.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-31T22:59:43.772 INFO:teuthology.orchestra.run.smithi142.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-31T22:59:43.772 INFO:teuthology.orchestra.run.smithi142.stdout: openssl-devel-1:3.0.7-27.el9.x86_64 2024-03-31T22:59:43.772 INFO:teuthology.orchestra.run.smithi142.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-31T22:59:43.772 INFO:teuthology.orchestra.run.smithi142.stdout: postgresql-13.14-1.el9.x86_64 2024-03-31T22:59:43.772 INFO:teuthology.orchestra.run.smithi142.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-03-31T22:59:43.772 INFO:teuthology.orchestra.run.smithi142.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-03-31T22:59:43.772 INFO:teuthology.orchestra.run.smithi142.stdout: postgresql-server-13.14-1.el9.x86_64 2024-03-31T22:59:43.773 INFO:teuthology.orchestra.run.smithi142.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-31T22:59:43.773 INFO:teuthology.orchestra.run.smithi142.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-31T22:59:43.773 INFO:teuthology.orchestra.run.smithi142.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-31T22:59:43.773 INFO:teuthology.orchestra.run.smithi142.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-03-31T22:59:43.773 INFO:teuthology.orchestra.run.smithi142.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-31T22:59:43.773 INFO:teuthology.orchestra.run.smithi142.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-31T22:59:43.773 INFO:teuthology.orchestra.run.smithi142.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-31T22:59:43.773 INFO:teuthology.orchestra.run.smithi142.stdout: python3-ceph-argparse-2:19.0.0-1977.g384f31f3.el9.x86_64 2024-03-31T22:59:43.773 INFO:teuthology.orchestra.run.smithi142.stdout: python3-ceph-common-2:19.0.0-1977.g384f31f3.el9.x86_64 2024-03-31T22:59:43.773 INFO:teuthology.orchestra.run.smithi142.stdout: python3-cephfs-2:19.0.0-1977.g384f31f3.el9.x86_64 2024-03-31T22:59:43.773 INFO:teuthology.orchestra.run.smithi142.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-31T22:59:43.773 INFO:teuthology.orchestra.run.smithi142.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-31T22:59:43.773 INFO:teuthology.orchestra.run.smithi142.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-31T22:59:43.773 INFO:teuthology.orchestra.run.smithi142.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-31T22:59:43.774 INFO:teuthology.orchestra.run.smithi142.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-03-31T22:59:43.774 INFO:teuthology.orchestra.run.smithi142.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-03-31T22:59:43.774 INFO:teuthology.orchestra.run.smithi142.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-31T22:59:43.774 INFO:teuthology.orchestra.run.smithi142.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-31T22:59:43.774 INFO:teuthology.orchestra.run.smithi142.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-31T22:59:43.774 INFO:teuthology.orchestra.run.smithi142.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-31T22:59:43.774 INFO:teuthology.orchestra.run.smithi142.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-31T22:59:43.774 INFO:teuthology.orchestra.run.smithi142.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-31T22:59:43.774 INFO:teuthology.orchestra.run.smithi142.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-31T22:59:43.774 INFO:teuthology.orchestra.run.smithi142.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-31T22:59:43.774 INFO:teuthology.orchestra.run.smithi142.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-31T22:59:43.774 INFO:teuthology.orchestra.run.smithi142.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-31T22:59:43.774 INFO:teuthology.orchestra.run.smithi142.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-31T22:59:43.775 INFO:teuthology.orchestra.run.smithi142.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-31T22:59:43.775 INFO:teuthology.orchestra.run.smithi142.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-31T22:59:43.775 INFO:teuthology.orchestra.run.smithi142.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-31T22:59:43.775 INFO:teuthology.orchestra.run.smithi142.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-31T22:59:43.775 INFO:teuthology.orchestra.run.smithi142.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-31T22:59:43.775 INFO:teuthology.orchestra.run.smithi142.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-31T22:59:43.775 INFO:teuthology.orchestra.run.smithi142.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-31T22:59:43.775 INFO:teuthology.orchestra.run.smithi142.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-31T22:59:43.775 INFO:teuthology.orchestra.run.smithi142.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-31T22:59:43.775 INFO:teuthology.orchestra.run.smithi142.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-31T22:59:43.775 INFO:teuthology.orchestra.run.smithi142.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-31T22:59:43.775 INFO:teuthology.orchestra.run.smithi142.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-31T22:59:43.775 INFO:teuthology.orchestra.run.smithi142.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-31T22:59:43.775 INFO:teuthology.orchestra.run.smithi142.stdout: python3-ply-3.11-14.el9.noarch 2024-03-31T22:59:43.776 INFO:teuthology.orchestra.run.smithi142.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-31T22:59:43.776 INFO:teuthology.orchestra.run.smithi142.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-31T22:59:43.776 INFO:teuthology.orchestra.run.smithi142.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-31T22:59:43.776 INFO:teuthology.orchestra.run.smithi142.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-31T22:59:43.776 INFO:teuthology.orchestra.run.smithi142.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-31T22:59:43.776 INFO:teuthology.orchestra.run.smithi142.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-31T22:59:43.776 INFO:teuthology.orchestra.run.smithi142.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-31T22:59:43.776 INFO:teuthology.orchestra.run.smithi142.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-03-31T22:59:43.776 INFO:teuthology.orchestra.run.smithi142.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-31T22:59:43.776 INFO:teuthology.orchestra.run.smithi142.stdout: python3-rados-2:19.0.0-1977.g384f31f3.el9.x86_64 2024-03-31T22:59:43.777 INFO:teuthology.orchestra.run.smithi142.stdout: python3-rbd-2:19.0.0-1977.g384f31f3.el9.x86_64 2024-03-31T22:59:43.777 INFO:teuthology.orchestra.run.smithi142.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-31T22:59:43.777 INFO:teuthology.orchestra.run.smithi142.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-31T22:59:43.777 INFO:teuthology.orchestra.run.smithi142.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-31T22:59:43.777 INFO:teuthology.orchestra.run.smithi142.stdout: python3-rgw-2:19.0.0-1977.g384f31f3.el9.x86_64 2024-03-31T22:59:43.777 INFO:teuthology.orchestra.run.smithi142.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-31T22:59:43.777 INFO:teuthology.orchestra.run.smithi142.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-31T22:59:43.777 INFO:teuthology.orchestra.run.smithi142.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-31T22:59:43.777 INFO:teuthology.orchestra.run.smithi142.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-31T22:59:43.777 INFO:teuthology.orchestra.run.smithi142.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-31T22:59:43.777 INFO:teuthology.orchestra.run.smithi142.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-31T22:59:43.777 INFO:teuthology.orchestra.run.smithi142.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-31T22:59:43.777 INFO:teuthology.orchestra.run.smithi142.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-03-31T22:59:43.777 INFO:teuthology.orchestra.run.smithi142.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-31T22:59:43.778 INFO:teuthology.orchestra.run.smithi142.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-31T22:59:43.778 INFO:teuthology.orchestra.run.smithi142.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-31T22:59:43.778 INFO:teuthology.orchestra.run.smithi142.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-31T22:59:43.778 INFO:teuthology.orchestra.run.smithi142.stdout: qatlib-24.02.0-1.el9.x86_64 2024-03-31T22:59:43.778 INFO:teuthology.orchestra.run.smithi142.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-03-31T22:59:43.778 INFO:teuthology.orchestra.run.smithi142.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-03-31T22:59:43.778 INFO:teuthology.orchestra.run.smithi142.stdout: rbd-fuse-2:19.0.0-1977.g384f31f3.el9.x86_64 2024-03-31T22:59:43.778 INFO:teuthology.orchestra.run.smithi142.stdout: rbd-mirror-2:19.0.0-1977.g384f31f3.el9.x86_64 2024-03-31T22:59:43.778 INFO:teuthology.orchestra.run.smithi142.stdout: rbd-nbd-2:19.0.0-1977.g384f31f3.el9.x86_64 2024-03-31T22:59:43.778 INFO:teuthology.orchestra.run.smithi142.stdout: re2-1:20211101-3.el9.x86_64 2024-03-31T22:59:43.778 INFO:teuthology.orchestra.run.smithi142.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-31T22:59:43.778 INFO:teuthology.orchestra.run.smithi142.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-31T22:59:43.778 INFO:teuthology.orchestra.run.smithi142.stdout: uuid-1.6.2-55.el9.x86_64 2024-03-31T22:59:43.778 INFO:teuthology.orchestra.run.smithi142.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-03-31T22:59:43.779 INFO:teuthology.orchestra.run.smithi142.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-31T22:59:43.779 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1977.g384f31f3.el9.noa 141/198 2024-03-31T22:59:43.779 INFO:teuthology.orchestra.run.smithi142.stdout: zlib-devel-1.2.11-40.el9.x86_64 2024-03-31T22:59:43.779 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-03-31T22:59:43.779 INFO:teuthology.orchestra.run.smithi142.stdout:Complete! 2024-03-31T22:59:44.242 DEBUG:teuthology.parallel:result is None 2024-03-31T22:59:44.808 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1977.g384f31f3.el9.noa 141/198 2024-03-31T22:59:44.840 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1977.g384f31f3.el9.n 142/198 2024-03-31T22:59:45.808 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1977.g384f31f3.el9.n 142/198 2024-03-31T22:59:45.832 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1977.g384 143/198 2024-03-31T22:59:45.934 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1977.g384 143/198 2024-03-31T22:59:46.136 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1977.g384f31f3.el 144/198 2024-03-31T22:59:46.160 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-mgr-2:19.0.0-1977.g384f31f3.el9.x86_64 145/198 2024-03-31T22:59:46.182 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1977.g384f31f3.el9.x86_64 145/198 2024-03-31T22:59:46.183 INFO:teuthology.orchestra.run.smithi162.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-31T22:59:46.183 INFO:teuthology.orchestra.run.smithi162.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-31T22:59:46.183 INFO:teuthology.orchestra.run.smithi162.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-31T22:59:46.183 INFO:teuthology.orchestra.run.smithi162.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-31T22:59:46.183 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-31T22:59:46.227 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-mgr-rook-2:19.0.0-1977.g384f31f3.el9.noarch 146/198 2024-03-31T22:59:46.238 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1977.g384f31f3.el9.noarch 146/198 2024-03-31T22:59:46.279 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-2:19.0.0-1977.g384f31f3.el9.x86_64 147/198 2024-03-31T22:59:47.135 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : libtool-2.4.6-45.el9.x86_64 148/198 2024-03-31T22:59:47.171 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-radosgw-2:19.0.0-1977.g384f31f3.el9.x86_64 149/198 2024-03-31T22:59:47.194 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1977.g384f31f3.el9.x86_64 149/198 2024-03-31T22:59:47.194 INFO:teuthology.orchestra.run.smithi162.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-31T22:59:47.194 INFO:teuthology.orchestra.run.smithi162.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-31T22:59:47.194 INFO:teuthology.orchestra.run.smithi162.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-31T22:59:47.194 INFO:teuthology.orchestra.run.smithi162.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-31T22:59:47.194 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-31T22:59:47.255 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1977.g384f3 150/198 2024-03-31T22:59:47.277 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1977.g384f3 150/198 2024-03-31T22:59:47.277 INFO:teuthology.orchestra.run.smithi162.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-31T22:59:47.277 INFO:teuthology.orchestra.run.smithi162.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-31T22:59:47.277 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-31T22:59:47.323 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : cephfs-mirror-2:19.0.0-1977.g384f31f3.el9.x86_64 151/198 2024-03-31T22:59:47.345 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-1977.g384f31f3.el9.x86_64 151/198 2024-03-31T22:59:47.345 INFO:teuthology.orchestra.run.smithi162.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-31T22:59:47.345 INFO:teuthology.orchestra.run.smithi162.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-03-31T22:59:47.345 INFO:teuthology.orchestra.run.smithi162.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-31T22:59:47.345 INFO:teuthology.orchestra.run.smithi162.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-31T22:59:47.345 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-31T22:59:47.657 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : rbd-mirror-2:19.0.0-1977.g384f31f3.el9.x86_64 152/198 2024-03-31T22:59:47.679 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1977.g384f31f3.el9.x86_64 152/198 2024-03-31T22:59:47.680 INFO:teuthology.orchestra.run.smithi162.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-31T22:59:47.680 INFO:teuthology.orchestra.run.smithi162.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-31T22:59:47.680 INFO:teuthology.orchestra.run.smithi162.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-31T22:59:47.680 INFO:teuthology.orchestra.run.smithi162.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-31T22:59:47.680 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-31T22:59:52.136 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-test-2:19.0.0-1977.g384f31f3.el9.x86_64 153/198 2024-03-31T22:59:52.177 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : rbd-fuse-2:19.0.0-1977.g384f31f3.el9.x86_64 154/198 2024-03-31T22:59:52.208 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : rbd-nbd-2:19.0.0-1977.g384f31f3.el9.x86_64 155/198 2024-03-31T22:59:52.243 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 156/198 2024-03-31T22:59:52.424 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : cephfs-shell-2:19.0.0-1977.g384f31f3.el9.x86_64 157/198 2024-03-31T22:59:52.546 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 158/198 2024-03-31T22:59:52.583 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-fuse-2:19.0.0-1977.g384f31f3.el9.x86_64 159/198 2024-03-31T22:59:52.637 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 160/198 2024-03-31T22:59:52.692 INFO:teuthology.orchestra.run.smithi162.stdout: Upgrading : xfsdump-3.1.12-4.el9.x86_64 161/198 2024-03-31T22:59:52.736 INFO:teuthology.orchestra.run.smithi162.stdout: Upgrading : util-linux-user-2.37.4-18.el9.x86_64 162/198 2024-03-31T22:59:52.768 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : libcephfs-devel-2:19.0.0-1977.g384f31f3.el9.x86_ 163/198 2024-03-31T22:59:52.809 INFO:teuthology.orchestra.run.smithi162.stdout: Upgrading : elfutils-debuginfod-client-0.190-2.el9.x86_64 164/198 2024-03-31T22:59:52.839 INFO:teuthology.orchestra.run.smithi162.stdout: Upgrading : NetworkManager-team-1:1.47.2-1.el9.x86_64 165/198 2024-03-31T22:59:52.903 INFO:teuthology.orchestra.run.smithi162.stdout: Upgrading : NetworkManager-tui-1:1.47.2-1.el9.x86_64 166/198 2024-03-31T22:59:52.972 INFO:teuthology.orchestra.run.smithi162.stdout: Upgrading : acl-2.3.1-4.el9.x86_64 167/198 2024-03-31T22:59:53.012 INFO:teuthology.orchestra.run.smithi162.stdout: Upgrading : libacl-devel-2.3.1-4.el9.x86_64 168/198 2024-03-31T22:59:53.596 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 169/198 2024-03-31T22:59:53.636 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 169/198 2024-03-31T22:59:53.655 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 169/198 2024-03-31T22:59:54.479 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : cephfs-top-2:19.0.0-1977.g384f31f3.el9.noarch 170/198 2024-03-31T22:59:54.632 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : openssl-devel-1:3.0.7-27.el9.x86_64 171/198 2024-03-31T22:59:54.673 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 172/198 2024-03-31T22:59:54.761 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : flex-2.6.4-9.el9.x86_64 173/198 2024-03-31T22:59:54.794 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : bison-3.7.4-5.el9.x86_64 174/198 2024-03-31T22:59:54.819 INFO:teuthology.orchestra.run.smithi162.stdout: Cleanup : NetworkManager-tui-1:1.44.0-2.el9.x86_64 175/198 2024-03-31T22:59:54.845 INFO:teuthology.orchestra.run.smithi162.stdout: Cleanup : elfutils-debuginfod-client-0.189-3.el9.x86_64 176/198 2024-03-31T22:59:54.871 INFO:teuthology.orchestra.run.smithi162.stdout: Cleanup : elfutils-libs-0.189-3.el9.x86_64 177/198 2024-03-31T22:59:54.872 INFO:teuthology.orchestra.run.smithi162.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-31T22:59:54.960 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-31T22:59:54.997 INFO:teuthology.orchestra.run.smithi162.stdout: Cleanup : openssl-1:3.0.7-24.el9.x86_64 179/198 2024-03-31T22:59:54.998 INFO:teuthology.orchestra.run.smithi162.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-31T22:59:55.082 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-31T22:59:55.109 INFO:teuthology.orchestra.run.smithi162.stdout: Cleanup : xfsdump-3.1.12-3.el9.x86_64 181/198 2024-03-31T22:59:55.142 INFO:teuthology.orchestra.run.smithi162.stdout: Cleanup : xfsprogs-5.19.0-4.el9.x86_64 182/198 2024-03-31T22:59:55.167 INFO:teuthology.orchestra.run.smithi162.stdout: Cleanup : acl-2.3.1-3.el9.x86_64 183/198 2024-03-31T22:59:55.192 INFO:teuthology.orchestra.run.smithi162.stdout: Cleanup : libacl-devel-2.3.1-3.el9.x86_64 184/198 2024-03-31T22:59:55.227 INFO:teuthology.orchestra.run.smithi162.stdout: Cleanup : util-linux-user-2.37.4-15.el9.x86_64 185/198 2024-03-31T22:59:55.293 INFO:teuthology.orchestra.run.smithi162.stdout: Cleanup : util-linux-2.37.4-15.el9.x86_64 186/198 2024-03-31T22:59:55.322 INFO:teuthology.orchestra.run.smithi162.stdout: Cleanup : util-linux-core-2.37.4-15.el9.x86_64 187/198 2024-03-31T22:59:55.348 INFO:teuthology.orchestra.run.smithi162.stdout: Cleanup : libfdisk-2.37.4-15.el9.x86_64 188/198 2024-03-31T22:59:55.373 INFO:teuthology.orchestra.run.smithi162.stdout: Cleanup : libmount-2.37.4-15.el9.x86_64 189/198 2024-03-31T22:59:55.399 INFO:teuthology.orchestra.run.smithi162.stdout: Cleanup : libblkid-2.37.4-15.el9.x86_64 190/198 2024-03-31T22:59:55.432 INFO:teuthology.orchestra.run.smithi162.stdout: Cleanup : NetworkManager-team-1:1.44.0-2.el9.x86_64 191/198 2024-03-31T22:59:55.439 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-31T22:59:55.439 INFO:teuthology.orchestra.run.smithi162.stdout: Cleanup : NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-31T22:59:55.618 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-31T22:59:55.619 INFO:teuthology.orchestra.run.smithi162.stdout: Cleanup : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-31T22:59:55.702 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-31T22:59:55.733 INFO:teuthology.orchestra.run.smithi162.stdout: Cleanup : libuuid-2.37.4-15.el9.x86_64 194/198 2024-03-31T22:59:55.760 INFO:teuthology.orchestra.run.smithi162.stdout: Cleanup : libsmartcols-2.37.4-15.el9.x86_64 195/198 2024-03-31T22:59:55.786 INFO:teuthology.orchestra.run.smithi162.stdout: Cleanup : libacl-2.3.1-3.el9.x86_64 196/198 2024-03-31T22:59:55.811 INFO:teuthology.orchestra.run.smithi162.stdout: Cleanup : openssl-libs-1:3.0.7-24.el9.x86_64 197/198 2024-03-31T22:59:55.876 INFO:teuthology.orchestra.run.smithi162.stdout: Cleanup : elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-31T23:00:01.412 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-31T23:00:01.412 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-2:19.0.0-1977.g384f31f3.el9.x86_64 1/198 2024-03-31T23:00:01.412 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-base-2:19.0.0-1977.g384f31f3.el9.x86_64 2/198 2024-03-31T23:00:01.412 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-common-2:19.0.0-1977.g384f31f3.el9.x86_64 3/198 2024-03-31T23:00:01.412 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-fuse-2:19.0.0-1977.g384f31f3.el9.x86_64 4/198 2024-03-31T23:00:01.412 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1977.g384f3 5/198 2024-03-31T23:00:01.412 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-mds-2:19.0.0-1977.g384f31f3.el9.x86_64 6/198 2024-03-31T23:00:01.412 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-mgr-2:19.0.0-1977.g384f31f3.el9.x86_64 7/198 2024-03-31T23:00:01.412 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-mon-2:19.0.0-1977.g384f31f3.el9.x86_64 8/198 2024-03-31T23:00:01.412 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-osd-2:19.0.0-1977.g384f31f3.el9.x86_64 9/198 2024-03-31T23:00:01.413 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-radosgw-2:19.0.0-1977.g384f31f3.el9.x86_64 10/198 2024-03-31T23:00:01.413 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-selinux-2:19.0.0-1977.g384f31f3.el9.x86_64 11/198 2024-03-31T23:00:01.413 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-test-2:19.0.0-1977.g384f31f3.el9.x86_64 12/198 2024-03-31T23:00:01.413 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : cephfs-mirror-2:19.0.0-1977.g384f31f3.el9.x86_64 13/198 2024-03-31T23:00:01.413 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : cephfs-shell-2:19.0.0-1977.g384f31f3.el9.x86_64 14/198 2024-03-31T23:00:01.413 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : libcephfs-devel-2:19.0.0-1977.g384f31f3.el9.x86_ 15/198 2024-03-31T23:00:01.413 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : libcephfs2-2:19.0.0-1977.g384f31f3.el9.x86_64 16/198 2024-03-31T23:00:01.413 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : libcephsqlite-2:19.0.0-1977.g384f31f3.el9.x86_64 17/198 2024-03-31T23:00:01.413 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : librados-devel-2:19.0.0-1977.g384f31f3.el9.x86_6 18/198 2024-03-31T23:00:01.413 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : libradosstriper1-2:19.0.0-1977.g384f31f3.el9.x86 19/198 2024-03-31T23:00:01.413 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : librgw2-2:19.0.0-1977.g384f31f3.el9.x86_64 20/198 2024-03-31T23:00:01.413 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1977.g384f31f3.el 21/198 2024-03-31T23:00:01.413 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-ceph-common-2:19.0.0-1977.g384f31f3.el9. 22/198 2024-03-31T23:00:01.414 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-cephfs-2:19.0.0-1977.g384f31f3.el9.x86_6 23/198 2024-03-31T23:00:01.414 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-rados-2:19.0.0-1977.g384f31f3.el9.x86_64 24/198 2024-03-31T23:00:01.414 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-rbd-2:19.0.0-1977.g384f31f3.el9.x86_64 25/198 2024-03-31T23:00:01.414 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-rgw-2:19.0.0-1977.g384f31f3.el9.x86_64 26/198 2024-03-31T23:00:01.414 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : rbd-fuse-2:19.0.0-1977.g384f31f3.el9.x86_64 27/198 2024-03-31T23:00:01.414 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : rbd-mirror-2:19.0.0-1977.g384f31f3.el9.x86_64 28/198 2024-03-31T23:00:01.414 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : rbd-nbd-2:19.0.0-1977.g384f31f3.el9.x86_64 29/198 2024-03-31T23:00:01.430 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1977.g384f31f3. 30/198 2024-03-31T23:00:01.430 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1977.g384f31f3.el9.noa 31/198 2024-03-31T23:00:01.430 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1977.g384f31f3.el9.n 32/198 2024-03-31T23:00:01.430 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1977.g384 33/198 2024-03-31T23:00:01.430 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1977.g384f31f3.el 34/198 2024-03-31T23:00:01.430 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1977.g384f31f3.el9.noarch 35/198 2024-03-31T23:00:01.430 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1977.g384f31f3.e 36/198 2024-03-31T23:00:01.430 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-volume-2:19.0.0-1977.g384f31f3.el9.noarch 37/198 2024-03-31T23:00:01.431 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : cephadm-2:19.0.0-1977.g384f31f3.el9.noarch 38/198 2024-03-31T23:00:01.431 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : cephfs-top-2:19.0.0-1977.g384f31f3.el9.noarch 39/198 2024-03-31T23:00:01.431 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : automake-1.16.2-8.el9.noarch 40/198 2024-03-31T23:00:01.431 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : bison-3.7.4-5.el9.x86_64 41/198 2024-03-31T23:00:01.431 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 42/198 2024-03-31T23:00:01.431 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : flex-2.6.4-9.el9.x86_64 43/198 2024-03-31T23:00:01.431 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 44/198 2024-03-31T23:00:01.431 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 45/198 2024-03-31T23:00:01.431 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 46/198 2024-03-31T23:00:01.431 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 47/198 2024-03-31T23:00:01.431 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 48/198 2024-03-31T23:00:01.431 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 49/198 2024-03-31T23:00:01.432 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 50/198 2024-03-31T23:00:01.432 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 51/198 2024-03-31T23:00:01.432 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 52/198 2024-03-31T23:00:01.432 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 53/198 2024-03-31T23:00:01.432 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : lua-5.4.4-4.el9.x86_64 54/198 2024-03-31T23:00:01.432 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 55/198 2024-03-31T23:00:01.432 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 56/198 2024-03-31T23:00:01.432 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 57/198 2024-03-31T23:00:01.432 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : postgresql-13.14-1.el9.x86_64 58/198 2024-03-31T23:00:01.432 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 59/198 2024-03-31T23:00:01.432 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 60/198 2024-03-31T23:00:01.432 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 61/198 2024-03-31T23:00:01.432 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 62/198 2024-03-31T23:00:01.433 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 63/198 2024-03-31T23:00:01.433 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 64/198 2024-03-31T23:00:01.433 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 65/198 2024-03-31T23:00:01.433 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 66/198 2024-03-31T23:00:01.433 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 67/198 2024-03-31T23:00:01.433 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 68/198 2024-03-31T23:00:01.433 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 69/198 2024-03-31T23:00:01.433 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 70/198 2024-03-31T23:00:01.433 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 71/198 2024-03-31T23:00:01.433 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 72/198 2024-03-31T23:00:01.433 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 73/198 2024-03-31T23:00:01.433 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 74/198 2024-03-31T23:00:01.433 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 75/198 2024-03-31T23:00:01.434 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 76/198 2024-03-31T23:00:01.434 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 77/198 2024-03-31T23:00:01.434 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 78/198 2024-03-31T23:00:01.434 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 79/198 2024-03-31T23:00:01.434 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 80/198 2024-03-31T23:00:01.434 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 81/198 2024-03-31T23:00:01.434 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 82/198 2024-03-31T23:00:01.434 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 83/198 2024-03-31T23:00:01.434 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 84/198 2024-03-31T23:00:01.434 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 85/198 2024-03-31T23:00:01.434 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 86/198 2024-03-31T23:00:01.434 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 87/198 2024-03-31T23:00:01.435 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : zlib-devel-1.2.11-40.el9.x86_64 88/198 2024-03-31T23:00:01.435 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 89/198 2024-03-31T23:00:01.435 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 90/198 2024-03-31T23:00:01.435 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 91/198 2024-03-31T23:00:01.435 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-ply-3.11-14.el9.noarch 92/198 2024-03-31T23:00:01.435 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 93/198 2024-03-31T23:00:01.435 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 94/198 2024-03-31T23:00:01.435 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : jq-1.6-16.el9.x86_64 95/198 2024-03-31T23:00:01.435 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 96/198 2024-03-31T23:00:01.435 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 97/198 2024-03-31T23:00:01.435 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 98/198 2024-03-31T23:00:01.435 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 99/198 2024-03-31T23:00:01.435 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 100/198 2024-03-31T23:00:01.435 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : openssl-devel-1:3.0.7-27.el9.x86_64 101/198 2024-03-31T23:00:01.436 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 102/198 2024-03-31T23:00:01.436 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 103/198 2024-03-31T23:00:01.436 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 104/198 2024-03-31T23:00:01.436 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 105/198 2024-03-31T23:00:01.436 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 106/198 2024-03-31T23:00:01.436 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 107/198 2024-03-31T23:00:01.436 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 108/198 2024-03-31T23:00:01.436 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 109/198 2024-03-31T23:00:01.436 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 110/198 2024-03-31T23:00:01.436 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 111/198 2024-03-31T23:00:01.436 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 112/198 2024-03-31T23:00:01.436 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 113/198 2024-03-31T23:00:01.436 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 114/198 2024-03-31T23:00:01.437 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 115/198 2024-03-31T23:00:01.437 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 116/198 2024-03-31T23:00:01.437 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 117/198 2024-03-31T23:00:01.437 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 118/198 2024-03-31T23:00:01.437 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 119/198 2024-03-31T23:00:01.437 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 120/198 2024-03-31T23:00:01.437 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 121/198 2024-03-31T23:00:01.437 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 122/198 2024-03-31T23:00:01.437 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 123/198 2024-03-31T23:00:01.437 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 124/198 2024-03-31T23:00:01.437 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 125/198 2024-03-31T23:00:01.437 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 126/198 2024-03-31T23:00:01.437 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 127/198 2024-03-31T23:00:01.437 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 128/198 2024-03-31T23:00:01.438 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 129/198 2024-03-31T23:00:01.438 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 130/198 2024-03-31T23:00:01.438 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 131/198 2024-03-31T23:00:01.438 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 132/198 2024-03-31T23:00:01.438 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 133/198 2024-03-31T23:00:01.438 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 134/198 2024-03-31T23:00:01.438 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 135/198 2024-03-31T23:00:01.438 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 136/198 2024-03-31T23:00:01.438 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 137/198 2024-03-31T23:00:01.438 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 138/198 2024-03-31T23:00:01.438 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 139/198 2024-03-31T23:00:01.438 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 140/198 2024-03-31T23:00:01.439 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 141/198 2024-03-31T23:00:01.439 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 142/198 2024-03-31T23:00:01.439 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 143/198 2024-03-31T23:00:01.439 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 144/198 2024-03-31T23:00:01.439 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 145/198 2024-03-31T23:00:01.439 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 146/198 2024-03-31T23:00:01.439 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 147/198 2024-03-31T23:00:01.439 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 148/198 2024-03-31T23:00:01.439 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : re2-1:20211101-3.el9.x86_64 149/198 2024-03-31T23:00:01.439 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 150/198 2024-03-31T23:00:01.439 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : librados2-2:19.0.0-1977.g384f31f3.el9.x86_64 151/198 2024-03-31T23:00:01.439 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 152/198 2024-03-31T23:00:01.439 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : librbd1-2:19.0.0-1977.g384f31f3.el9.x86_64 153/198 2024-03-31T23:00:01.440 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 154/198 2024-03-31T23:00:01.440 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : NetworkManager-1:1.47.2-1.el9.x86_64 155/198 2024-03-31T23:00:01.440 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : NetworkManager-1:1.44.0-2.el9.x86_64 156/198 2024-03-31T23:00:01.440 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : NetworkManager-libnm-1:1.47.2-1.el9.x86_64 157/198 2024-03-31T23:00:01.440 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 158/198 2024-03-31T23:00:01.440 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : NetworkManager-team-1:1.47.2-1.el9.x86_64 159/198 2024-03-31T23:00:01.440 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : NetworkManager-team-1:1.44.0-2.el9.x86_64 160/198 2024-03-31T23:00:01.440 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : NetworkManager-tui-1:1.47.2-1.el9.x86_64 161/198 2024-03-31T23:00:01.440 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : NetworkManager-tui-1:1.44.0-2.el9.x86_64 162/198 2024-03-31T23:00:01.440 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : xfsdump-3.1.12-4.el9.x86_64 163/198 2024-03-31T23:00:01.440 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : xfsdump-3.1.12-3.el9.x86_64 164/198 2024-03-31T23:00:01.440 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : acl-2.3.1-4.el9.x86_64 165/198 2024-03-31T23:00:01.440 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : acl-2.3.1-3.el9.x86_64 166/198 2024-03-31T23:00:01.441 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 167/198 2024-03-31T23:00:01.441 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : elfutils-debuginfod-client-0.189-3.el9.x86_64 168/198 2024-03-31T23:00:01.441 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 169/198 2024-03-31T23:00:01.441 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : elfutils-libelf-0.189-3.el9.x86_64 170/198 2024-03-31T23:00:01.441 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 171/198 2024-03-31T23:00:01.441 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : elfutils-libs-0.189-3.el9.x86_64 172/198 2024-03-31T23:00:01.441 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : libacl-2.3.1-4.el9.x86_64 173/198 2024-03-31T23:00:01.441 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : libacl-2.3.1-3.el9.x86_64 174/198 2024-03-31T23:00:01.441 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : libblkid-2.37.4-18.el9.x86_64 175/198 2024-03-31T23:00:01.441 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : libblkid-2.37.4-15.el9.x86_64 176/198 2024-03-31T23:00:01.441 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : libfdisk-2.37.4-18.el9.x86_64 177/198 2024-03-31T23:00:01.441 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : libfdisk-2.37.4-15.el9.x86_64 178/198 2024-03-31T23:00:01.441 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : libmount-2.37.4-18.el9.x86_64 179/198 2024-03-31T23:00:01.442 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : libmount-2.37.4-15.el9.x86_64 180/198 2024-03-31T23:00:01.442 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : libsmartcols-2.37.4-18.el9.x86_64 181/198 2024-03-31T23:00:01.442 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : libsmartcols-2.37.4-15.el9.x86_64 182/198 2024-03-31T23:00:01.442 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : libuuid-2.37.4-18.el9.x86_64 183/198 2024-03-31T23:00:01.442 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : libuuid-2.37.4-15.el9.x86_64 184/198 2024-03-31T23:00:01.442 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 185/198 2024-03-31T23:00:01.442 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : openssl-1:3.0.7-24.el9.x86_64 186/198 2024-03-31T23:00:01.442 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 187/198 2024-03-31T23:00:01.442 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : openssl-libs-1:3.0.7-24.el9.x86_64 188/198 2024-03-31T23:00:01.442 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : util-linux-2.37.4-18.el9.x86_64 189/198 2024-03-31T23:00:01.442 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : util-linux-2.37.4-15.el9.x86_64 190/198 2024-03-31T23:00:01.442 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : util-linux-core-2.37.4-18.el9.x86_64 191/198 2024-03-31T23:00:01.443 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : util-linux-core-2.37.4-15.el9.x86_64 192/198 2024-03-31T23:00:01.443 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : util-linux-user-2.37.4-18.el9.x86_64 193/198 2024-03-31T23:00:01.443 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : util-linux-user-2.37.4-15.el9.x86_64 194/198 2024-03-31T23:00:01.443 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 195/198 2024-03-31T23:00:01.443 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : xfsprogs-5.19.0-4.el9.x86_64 196/198 2024-03-31T23:00:01.443 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : libacl-devel-2.3.1-4.el9.x86_64 197/198 2024-03-31T23:00:04.359 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : libacl-devel-2.3.1-3.el9.x86_64 198/198 2024-03-31T23:00:04.359 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-31T23:00:04.359 INFO:teuthology.orchestra.run.smithi162.stdout:Upgraded: 2024-03-31T23:00:04.359 INFO:teuthology.orchestra.run.smithi162.stdout: NetworkManager-1:1.47.2-1.el9.x86_64 2024-03-31T23:00:04.359 INFO:teuthology.orchestra.run.smithi162.stdout: NetworkManager-libnm-1:1.47.2-1.el9.x86_64 2024-03-31T23:00:04.359 INFO:teuthology.orchestra.run.smithi162.stdout: NetworkManager-team-1:1.47.2-1.el9.x86_64 2024-03-31T23:00:04.359 INFO:teuthology.orchestra.run.smithi162.stdout: NetworkManager-tui-1:1.47.2-1.el9.x86_64 2024-03-31T23:00:04.359 INFO:teuthology.orchestra.run.smithi162.stdout: acl-2.3.1-4.el9.x86_64 2024-03-31T23:00:04.359 INFO:teuthology.orchestra.run.smithi162.stdout: elfutils-debuginfod-client-0.190-2.el9.x86_64 2024-03-31T23:00:04.359 INFO:teuthology.orchestra.run.smithi162.stdout: elfutils-libelf-0.190-2.el9.x86_64 2024-03-31T23:00:04.359 INFO:teuthology.orchestra.run.smithi162.stdout: elfutils-libs-0.190-2.el9.x86_64 2024-03-31T23:00:04.360 INFO:teuthology.orchestra.run.smithi162.stdout: libacl-2.3.1-4.el9.x86_64 2024-03-31T23:00:04.360 INFO:teuthology.orchestra.run.smithi162.stdout: libacl-devel-2.3.1-4.el9.x86_64 2024-03-31T23:00:04.360 INFO:teuthology.orchestra.run.smithi162.stdout: libblkid-2.37.4-18.el9.x86_64 2024-03-31T23:00:04.360 INFO:teuthology.orchestra.run.smithi162.stdout: libfdisk-2.37.4-18.el9.x86_64 2024-03-31T23:00:04.360 INFO:teuthology.orchestra.run.smithi162.stdout: libmount-2.37.4-18.el9.x86_64 2024-03-31T23:00:04.360 INFO:teuthology.orchestra.run.smithi162.stdout: librados2-2:19.0.0-1977.g384f31f3.el9.x86_64 2024-03-31T23:00:04.360 INFO:teuthology.orchestra.run.smithi162.stdout: librbd1-2:19.0.0-1977.g384f31f3.el9.x86_64 2024-03-31T23:00:04.360 INFO:teuthology.orchestra.run.smithi162.stdout: libsmartcols-2.37.4-18.el9.x86_64 2024-03-31T23:00:04.360 INFO:teuthology.orchestra.run.smithi162.stdout: libuuid-2.37.4-18.el9.x86_64 2024-03-31T23:00:04.360 INFO:teuthology.orchestra.run.smithi162.stdout: openssl-1:3.0.7-27.el9.x86_64 2024-03-31T23:00:04.360 INFO:teuthology.orchestra.run.smithi162.stdout: openssl-libs-1:3.0.7-27.el9.x86_64 2024-03-31T23:00:04.360 INFO:teuthology.orchestra.run.smithi162.stdout: util-linux-2.37.4-18.el9.x86_64 2024-03-31T23:00:04.361 INFO:teuthology.orchestra.run.smithi162.stdout: util-linux-core-2.37.4-18.el9.x86_64 2024-03-31T23:00:04.361 INFO:teuthology.orchestra.run.smithi162.stdout: util-linux-user-2.37.4-18.el9.x86_64 2024-03-31T23:00:04.361 INFO:teuthology.orchestra.run.smithi162.stdout: xfsdump-3.1.12-4.el9.x86_64 2024-03-31T23:00:04.361 INFO:teuthology.orchestra.run.smithi162.stdout: xfsprogs-6.3.0-1.el9.x86_64 2024-03-31T23:00:04.361 INFO:teuthology.orchestra.run.smithi162.stdout:Installed: 2024-03-31T23:00:04.361 INFO:teuthology.orchestra.run.smithi162.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-31T23:00:04.361 INFO:teuthology.orchestra.run.smithi162.stdout: automake-1.16.2-8.el9.noarch 2024-03-31T23:00:04.361 INFO:teuthology.orchestra.run.smithi162.stdout: bison-3.7.4-5.el9.x86_64 2024-03-31T23:00:04.361 INFO:teuthology.orchestra.run.smithi162.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-31T23:00:04.361 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-2:19.0.0-1977.g384f31f3.el9.x86_64 2024-03-31T23:00:04.361 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-base-2:19.0.0-1977.g384f31f3.el9.x86_64 2024-03-31T23:00:04.361 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-common-2:19.0.0-1977.g384f31f3.el9.x86_64 2024-03-31T23:00:04.361 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-fuse-2:19.0.0-1977.g384f31f3.el9.x86_64 2024-03-31T23:00:04.362 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-grafana-dashboards-2:19.0.0-1977.g384f31f3.el9.noarch 2024-03-31T23:00:04.362 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-immutable-object-cache-2:19.0.0-1977.g384f31f3.el9.x86_64 2024-03-31T23:00:04.362 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-mds-2:19.0.0-1977.g384f31f3.el9.x86_64 2024-03-31T23:00:04.362 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-mgr-2:19.0.0-1977.g384f31f3.el9.x86_64 2024-03-31T23:00:04.362 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-mgr-cephadm-2:19.0.0-1977.g384f31f3.el9.noarch 2024-03-31T23:00:04.362 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-mgr-dashboard-2:19.0.0-1977.g384f31f3.el9.noarch 2024-03-31T23:00:04.362 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1977.g384f31f3.el9.noarch 2024-03-31T23:00:04.362 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-mgr-modules-core-2:19.0.0-1977.g384f31f3.el9.noarch 2024-03-31T23:00:04.362 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-mgr-rook-2:19.0.0-1977.g384f31f3.el9.noarch 2024-03-31T23:00:04.362 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-mon-2:19.0.0-1977.g384f31f3.el9.x86_64 2024-03-31T23:00:04.362 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-osd-2:19.0.0-1977.g384f31f3.el9.x86_64 2024-03-31T23:00:04.362 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-prometheus-alerts-2:19.0.0-1977.g384f31f3.el9.noarch 2024-03-31T23:00:04.363 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-radosgw-2:19.0.0-1977.g384f31f3.el9.x86_64 2024-03-31T23:00:04.363 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-selinux-2:19.0.0-1977.g384f31f3.el9.x86_64 2024-03-31T23:00:04.363 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-test-2:19.0.0-1977.g384f31f3.el9.x86_64 2024-03-31T23:00:04.363 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-volume-2:19.0.0-1977.g384f31f3.el9.noarch 2024-03-31T23:00:04.363 INFO:teuthology.orchestra.run.smithi162.stdout: cephadm-2:19.0.0-1977.g384f31f3.el9.noarch 2024-03-31T23:00:04.363 INFO:teuthology.orchestra.run.smithi162.stdout: cephfs-mirror-2:19.0.0-1977.g384f31f3.el9.x86_64 2024-03-31T23:00:04.363 INFO:teuthology.orchestra.run.smithi162.stdout: cephfs-shell-2:19.0.0-1977.g384f31f3.el9.x86_64 2024-03-31T23:00:04.363 INFO:teuthology.orchestra.run.smithi162.stdout: cephfs-top-2:19.0.0-1977.g384f31f3.el9.noarch 2024-03-31T23:00:04.363 INFO:teuthology.orchestra.run.smithi162.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-03-31T23:00:04.363 INFO:teuthology.orchestra.run.smithi162.stdout: flex-2.6.4-9.el9.x86_64 2024-03-31T23:00:04.363 INFO:teuthology.orchestra.run.smithi162.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-31T23:00:04.363 INFO:teuthology.orchestra.run.smithi162.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-31T23:00:04.363 INFO:teuthology.orchestra.run.smithi162.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-31T23:00:04.363 INFO:teuthology.orchestra.run.smithi162.stdout: fuse-2.9.9-15.el9.x86_64 2024-03-31T23:00:04.364 INFO:teuthology.orchestra.run.smithi162.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-31T23:00:04.364 INFO:teuthology.orchestra.run.smithi162.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-31T23:00:04.364 INFO:teuthology.orchestra.run.smithi162.stdout: jq-1.6-16.el9.x86_64 2024-03-31T23:00:04.364 INFO:teuthology.orchestra.run.smithi162.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-03-31T23:00:04.364 INFO:teuthology.orchestra.run.smithi162.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-31T23:00:04.364 INFO:teuthology.orchestra.run.smithi162.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-31T23:00:04.364 INFO:teuthology.orchestra.run.smithi162.stdout: libcephfs-devel-2:19.0.0-1977.g384f31f3.el9.x86_64 2024-03-31T23:00:04.364 INFO:teuthology.orchestra.run.smithi162.stdout: libcephfs2-2:19.0.0-1977.g384f31f3.el9.x86_64 2024-03-31T23:00:04.364 INFO:teuthology.orchestra.run.smithi162.stdout: libcephsqlite-2:19.0.0-1977.g384f31f3.el9.x86_64 2024-03-31T23:00:04.364 INFO:teuthology.orchestra.run.smithi162.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-31T23:00:04.364 INFO:teuthology.orchestra.run.smithi162.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-31T23:00:04.364 INFO:teuthology.orchestra.run.smithi162.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-31T23:00:04.364 INFO:teuthology.orchestra.run.smithi162.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-31T23:00:04.365 INFO:teuthology.orchestra.run.smithi162.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-31T23:00:04.365 INFO:teuthology.orchestra.run.smithi162.stdout: librados-devel-2:19.0.0-1977.g384f31f3.el9.x86_64 2024-03-31T23:00:04.365 INFO:teuthology.orchestra.run.smithi162.stdout: libradosstriper1-2:19.0.0-1977.g384f31f3.el9.x86_64 2024-03-31T23:00:04.365 INFO:teuthology.orchestra.run.smithi162.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-31T23:00:04.365 INFO:teuthology.orchestra.run.smithi162.stdout: librgw2-2:19.0.0-1977.g384f31f3.el9.x86_64 2024-03-31T23:00:04.365 INFO:teuthology.orchestra.run.smithi162.stdout: libtool-2.4.6-45.el9.x86_64 2024-03-31T23:00:04.365 INFO:teuthology.orchestra.run.smithi162.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-31T23:00:04.365 INFO:teuthology.orchestra.run.smithi162.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-03-31T23:00:04.365 INFO:teuthology.orchestra.run.smithi162.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-03-31T23:00:04.365 INFO:teuthology.orchestra.run.smithi162.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-31T23:00:04.365 INFO:teuthology.orchestra.run.smithi162.stdout: lua-5.4.4-4.el9.x86_64 2024-03-31T23:00:04.365 INFO:teuthology.orchestra.run.smithi162.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-31T23:00:04.365 INFO:teuthology.orchestra.run.smithi162.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-31T23:00:04.366 INFO:teuthology.orchestra.run.smithi162.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-31T23:00:04.366 INFO:teuthology.orchestra.run.smithi162.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-31T23:00:04.366 INFO:teuthology.orchestra.run.smithi162.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-31T23:00:04.366 INFO:teuthology.orchestra.run.smithi162.stdout: openssl-devel-1:3.0.7-27.el9.x86_64 2024-03-31T23:00:04.366 INFO:teuthology.orchestra.run.smithi162.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-31T23:00:04.366 INFO:teuthology.orchestra.run.smithi162.stdout: postgresql-13.14-1.el9.x86_64 2024-03-31T23:00:04.366 INFO:teuthology.orchestra.run.smithi162.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-03-31T23:00:04.366 INFO:teuthology.orchestra.run.smithi162.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-03-31T23:00:04.366 INFO:teuthology.orchestra.run.smithi162.stdout: postgresql-server-13.14-1.el9.x86_64 2024-03-31T23:00:04.366 INFO:teuthology.orchestra.run.smithi162.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-31T23:00:04.366 INFO:teuthology.orchestra.run.smithi162.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-31T23:00:04.366 INFO:teuthology.orchestra.run.smithi162.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-31T23:00:04.366 INFO:teuthology.orchestra.run.smithi162.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-03-31T23:00:04.367 INFO:teuthology.orchestra.run.smithi162.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-31T23:00:04.367 INFO:teuthology.orchestra.run.smithi162.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-31T23:00:04.367 INFO:teuthology.orchestra.run.smithi162.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-31T23:00:04.367 INFO:teuthology.orchestra.run.smithi162.stdout: python3-ceph-argparse-2:19.0.0-1977.g384f31f3.el9.x86_64 2024-03-31T23:00:04.367 INFO:teuthology.orchestra.run.smithi162.stdout: python3-ceph-common-2:19.0.0-1977.g384f31f3.el9.x86_64 2024-03-31T23:00:04.367 INFO:teuthology.orchestra.run.smithi162.stdout: python3-cephfs-2:19.0.0-1977.g384f31f3.el9.x86_64 2024-03-31T23:00:04.367 INFO:teuthology.orchestra.run.smithi162.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-31T23:00:04.367 INFO:teuthology.orchestra.run.smithi162.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-31T23:00:04.367 INFO:teuthology.orchestra.run.smithi162.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-31T23:00:04.367 INFO:teuthology.orchestra.run.smithi162.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-31T23:00:04.367 INFO:teuthology.orchestra.run.smithi162.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-03-31T23:00:04.367 INFO:teuthology.orchestra.run.smithi162.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-03-31T23:00:04.367 INFO:teuthology.orchestra.run.smithi162.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-31T23:00:04.368 INFO:teuthology.orchestra.run.smithi162.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-31T23:00:04.368 INFO:teuthology.orchestra.run.smithi162.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-31T23:00:04.368 INFO:teuthology.orchestra.run.smithi162.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-31T23:00:04.368 INFO:teuthology.orchestra.run.smithi162.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-31T23:00:04.368 INFO:teuthology.orchestra.run.smithi162.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-31T23:00:04.368 INFO:teuthology.orchestra.run.smithi162.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-31T23:00:04.368 INFO:teuthology.orchestra.run.smithi162.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-31T23:00:04.368 INFO:teuthology.orchestra.run.smithi162.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-31T23:00:04.368 INFO:teuthology.orchestra.run.smithi162.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-31T23:00:04.368 INFO:teuthology.orchestra.run.smithi162.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-31T23:00:04.368 INFO:teuthology.orchestra.run.smithi162.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-31T23:00:04.368 INFO:teuthology.orchestra.run.smithi162.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-31T23:00:04.369 INFO:teuthology.orchestra.run.smithi162.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-31T23:00:04.369 INFO:teuthology.orchestra.run.smithi162.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-31T23:00:04.369 INFO:teuthology.orchestra.run.smithi162.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-31T23:00:04.369 INFO:teuthology.orchestra.run.smithi162.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-31T23:00:04.369 INFO:teuthology.orchestra.run.smithi162.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-31T23:00:04.369 INFO:teuthology.orchestra.run.smithi162.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-31T23:00:04.369 INFO:teuthology.orchestra.run.smithi162.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-31T23:00:04.369 INFO:teuthology.orchestra.run.smithi162.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-31T23:00:04.369 INFO:teuthology.orchestra.run.smithi162.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-31T23:00:04.369 INFO:teuthology.orchestra.run.smithi162.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-31T23:00:04.369 INFO:teuthology.orchestra.run.smithi162.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-31T23:00:04.369 INFO:teuthology.orchestra.run.smithi162.stdout: python3-ply-3.11-14.el9.noarch 2024-03-31T23:00:04.369 INFO:teuthology.orchestra.run.smithi162.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-31T23:00:04.370 INFO:teuthology.orchestra.run.smithi162.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-31T23:00:04.370 INFO:teuthology.orchestra.run.smithi162.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-31T23:00:04.370 INFO:teuthology.orchestra.run.smithi162.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-31T23:00:04.370 INFO:teuthology.orchestra.run.smithi162.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-31T23:00:04.370 INFO:teuthology.orchestra.run.smithi162.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-31T23:00:04.370 INFO:teuthology.orchestra.run.smithi162.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-31T23:00:04.370 INFO:teuthology.orchestra.run.smithi162.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-03-31T23:00:04.370 INFO:teuthology.orchestra.run.smithi162.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-31T23:00:04.370 INFO:teuthology.orchestra.run.smithi162.stdout: python3-rados-2:19.0.0-1977.g384f31f3.el9.x86_64 2024-03-31T23:00:04.370 INFO:teuthology.orchestra.run.smithi162.stdout: python3-rbd-2:19.0.0-1977.g384f31f3.el9.x86_64 2024-03-31T23:00:04.370 INFO:teuthology.orchestra.run.smithi162.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-31T23:00:04.370 INFO:teuthology.orchestra.run.smithi162.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-31T23:00:04.371 INFO:teuthology.orchestra.run.smithi162.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-31T23:00:04.371 INFO:teuthology.orchestra.run.smithi162.stdout: python3-rgw-2:19.0.0-1977.g384f31f3.el9.x86_64 2024-03-31T23:00:04.371 INFO:teuthology.orchestra.run.smithi162.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-31T23:00:04.371 INFO:teuthology.orchestra.run.smithi162.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-31T23:00:04.371 INFO:teuthology.orchestra.run.smithi162.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-31T23:00:04.371 INFO:teuthology.orchestra.run.smithi162.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-31T23:00:04.371 INFO:teuthology.orchestra.run.smithi162.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-31T23:00:04.371 INFO:teuthology.orchestra.run.smithi162.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-31T23:00:04.371 INFO:teuthology.orchestra.run.smithi162.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-31T23:00:04.371 INFO:teuthology.orchestra.run.smithi162.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-03-31T23:00:04.371 INFO:teuthology.orchestra.run.smithi162.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-31T23:00:04.371 INFO:teuthology.orchestra.run.smithi162.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-31T23:00:04.371 INFO:teuthology.orchestra.run.smithi162.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-31T23:00:04.372 INFO:teuthology.orchestra.run.smithi162.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-31T23:00:04.372 INFO:teuthology.orchestra.run.smithi162.stdout: qatlib-24.02.0-1.el9.x86_64 2024-03-31T23:00:04.372 INFO:teuthology.orchestra.run.smithi162.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-03-31T23:00:04.372 INFO:teuthology.orchestra.run.smithi162.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-03-31T23:00:04.372 INFO:teuthology.orchestra.run.smithi162.stdout: rbd-fuse-2:19.0.0-1977.g384f31f3.el9.x86_64 2024-03-31T23:00:04.372 INFO:teuthology.orchestra.run.smithi162.stdout: rbd-mirror-2:19.0.0-1977.g384f31f3.el9.x86_64 2024-03-31T23:00:04.372 INFO:teuthology.orchestra.run.smithi162.stdout: rbd-nbd-2:19.0.0-1977.g384f31f3.el9.x86_64 2024-03-31T23:00:04.372 INFO:teuthology.orchestra.run.smithi162.stdout: re2-1:20211101-3.el9.x86_64 2024-03-31T23:00:04.372 INFO:teuthology.orchestra.run.smithi162.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-31T23:00:04.372 INFO:teuthology.orchestra.run.smithi162.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-31T23:00:04.372 INFO:teuthology.orchestra.run.smithi162.stdout: uuid-1.6.2-55.el9.x86_64 2024-03-31T23:00:04.372 INFO:teuthology.orchestra.run.smithi162.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-03-31T23:00:04.372 INFO:teuthology.orchestra.run.smithi162.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-31T23:00:04.373 INFO:teuthology.orchestra.run.smithi162.stdout: zlib-devel-1.2.11-40.el9.x86_64 2024-03-31T23:00:04.373 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-31T23:00:04.373 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2024-03-31T23:00:04.833 DEBUG:teuthology.parallel:result is None 2024-03-31T23:00:04.834 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=384f31f3d3aee7f8ba3f60846b8844c049165583 2024-03-31T23:00:05.056 DEBUG:teuthology.orchestra.run.smithi046:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-31T23:00:05.081 INFO:teuthology.orchestra.run.smithi046.stdout:19.0.0-1977.g384f31f3.el9 2024-03-31T23:00:05.082 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1977.g384f31f3.el9 2024-03-31T23:00:05.082 INFO:teuthology.task.install:The correct ceph version 19.0.0-1977.g384f31f3 is installed. 2024-03-31T23:00:05.084 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=384f31f3d3aee7f8ba3f60846b8844c049165583 2024-03-31T23:00:05.301 DEBUG:teuthology.orchestra.run.smithi142:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-31T23:00:05.327 INFO:teuthology.orchestra.run.smithi142.stdout:19.0.0-1977.g384f31f3.el9 2024-03-31T23:00:05.327 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1977.g384f31f3.el9 2024-03-31T23:00:05.327 INFO:teuthology.task.install:The correct ceph version 19.0.0-1977.g384f31f3 is installed. 2024-03-31T23:00:05.329 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=384f31f3d3aee7f8ba3f60846b8844c049165583 2024-03-31T23:00:05.549 DEBUG:teuthology.orchestra.run.smithi162:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-31T23:00:05.575 INFO:teuthology.orchestra.run.smithi162.stdout:19.0.0-1977.g384f31f3.el9 2024-03-31T23:00:05.576 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1977.g384f31f3.el9 2024-03-31T23:00:05.576 INFO:teuthology.task.install:The correct ceph version 19.0.0-1977.g384f31f3 is installed. 2024-03-31T23:00:05.578 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-03-31T23:00:05.578 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-03-31T23:00:05.578 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-31T23:00:05.622 DEBUG:teuthology.orchestra.run.smithi142:> set -ex 2024-03-31T23:00:05.622 DEBUG:teuthology.orchestra.run.smithi142:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-31T23:00:05.669 DEBUG:teuthology.orchestra.run.smithi162:> set -ex 2024-03-31T23:00:05.669 DEBUG:teuthology.orchestra.run.smithi162:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-31T23:00:05.703 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-03-31T23:00:05.703 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-03-31T23:00:05.703 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd of=/usr/bin/daemon-helper 2024-03-31T23:00:05.734 DEBUG:teuthology.orchestra.run.smithi046:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-31T23:00:05.804 DEBUG:teuthology.orchestra.run.smithi142:> set -ex 2024-03-31T23:00:05.804 DEBUG:teuthology.orchestra.run.smithi142:> sudo dd of=/usr/bin/daemon-helper 2024-03-31T23:00:05.834 DEBUG:teuthology.orchestra.run.smithi142:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-31T23:00:05.905 DEBUG:teuthology.orchestra.run.smithi162:> set -ex 2024-03-31T23:00:05.905 DEBUG:teuthology.orchestra.run.smithi162:> sudo dd of=/usr/bin/daemon-helper 2024-03-31T23:00:05.937 DEBUG:teuthology.orchestra.run.smithi162:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-31T23:00:06.007 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-03-31T23:00:06.007 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-03-31T23:00:06.007 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-31T23:00:06.035 DEBUG:teuthology.orchestra.run.smithi046:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-31T23:00:06.101 DEBUG:teuthology.orchestra.run.smithi142:> set -ex 2024-03-31T23:00:06.102 DEBUG:teuthology.orchestra.run.smithi142:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-31T23:00:06.130 DEBUG:teuthology.orchestra.run.smithi142:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-31T23:00:06.196 DEBUG:teuthology.orchestra.run.smithi162:> set -ex 2024-03-31T23:00:06.196 DEBUG:teuthology.orchestra.run.smithi162:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-31T23:00:06.228 DEBUG:teuthology.orchestra.run.smithi162:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-31T23:00:06.299 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-03-31T23:00:06.300 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-03-31T23:00:06.300 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd of=/usr/bin/stdin-killer 2024-03-31T23:00:06.327 DEBUG:teuthology.orchestra.run.smithi046:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-31T23:00:06.395 DEBUG:teuthology.orchestra.run.smithi142:> set -ex 2024-03-31T23:00:06.395 DEBUG:teuthology.orchestra.run.smithi142:> sudo dd of=/usr/bin/stdin-killer 2024-03-31T23:00:06.422 DEBUG:teuthology.orchestra.run.smithi142:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-31T23:00:06.490 DEBUG:teuthology.orchestra.run.smithi162:> set -ex 2024-03-31T23:00:06.490 DEBUG:teuthology.orchestra.run.smithi162:> sudo dd of=/usr/bin/stdin-killer 2024-03-31T23:00:06.521 DEBUG:teuthology.orchestra.run.smithi162:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-31T23:00:06.592 INFO:teuthology.run_tasks:Running task cephadm... 2024-03-31T23:00:06.698 INFO:tasks.cephadm:Config: {'roleless': False, 'cephfs': {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd'], 'max_mds': 3, 'session_timeout': 300, 'standby_replay': True}, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'mds_bal_replicate_threshold': 1, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'cephsqlite lock renewal timeout': 900000, 'debug cephsqlite': 20, 'debug client': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180, 'osd shutdown pgref assert': True, 'osd_max_omap_entries_per_request': 10}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'slow metadata IO', 'SLOW_OPS', 'slow request'], 'sha1': '384f31f3d3aee7f8ba3f60846b8844c049165583', 'subvols': {'create': 2}} 2024-03-31T23:00:06.698 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:384f31f3d3aee7f8ba3f60846b8844c049165583 2024-03-31T23:00:06.699 INFO:tasks.cephadm:Cluster fsid is 6a9d8f08-efb2-11ee-b647-cb9ed24678a4 2024-03-31T23:00:06.699 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-03-31T23:00:06.699 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.46', 'mon.b': '172.21.15.142', 'mon.c': '172.21.15.162'} 2024-03-31T23:00:06.699 INFO:tasks.cephadm:First mon is mon.a on smithi046 2024-03-31T23:00:06.699 INFO:tasks.cephadm:First mgr is x 2024-03-31T23:00:06.699 INFO:tasks.cephadm:Normalizing hostnames... 2024-03-31T23:00:06.700 DEBUG:teuthology.orchestra.run.smithi046:> sudo hostname $(hostname -s) 2024-03-31T23:00:06.727 DEBUG:teuthology.orchestra.run.smithi142:> sudo hostname $(hostname -s) 2024-03-31T23:00:06.754 DEBUG:teuthology.orchestra.run.smithi162:> sudo hostname $(hostname -s) 2024-03-31T23:00:06.786 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra 2024-03-31T23:00:06.786 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=384f31f3d3aee7f8ba3f60846b8844c049165583 2024-03-31T23:00:07.007 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': '384f31f3d3aee7f8ba3f60846b8844c049165583', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/54774/', 'root_build_cause': 'SCMTRIGGER', 'version': '19.0.0-1977-g384f31f3', 'node_name': '172.21.2.1+braggi01', 'job_name': 'ceph-dev-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.0.0-1977.g384f31f3'}, 'url': 'https://3.chacra.ceph.com/r/ceph/squid/384f31f3d3aee7f8ba3f60846b8844c049165583/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-03-31 17:40:06.063697', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'squid', 'chacra_url': 'https://3.chacra.ceph.com/repos/ceph/squid/384f31f3d3aee7f8ba3f60846b8844c049165583/centos/9/flavors/default/', 'archs': ['x86_64', 'source'], 'distro': 'centos'}] 2024-03-31T23:00:07.167 INFO:tasks.util.chacra:got chacra host 3.chacra.ceph.com, ref squid, sha1 384f31f3d3aee7f8ba3f60846b8844c049165583 from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F9%2Fx86_64&flavor=default&sha1=384f31f3d3aee7f8ba3f60846b8844c049165583 2024-03-31T23:00:07.168 INFO:tasks.cephadm:Discovered cachra url: https://3.chacra.ceph.com/binaries/ceph/squid/384f31f3d3aee7f8ba3f60846b8844c049165583/centos/9/x86_64/flavors/default/cephadm 2024-03-31T23:00:07.169 DEBUG:teuthology.orchestra.run.smithi046:> curl --silent -L https://3.chacra.ceph.com/binaries/ceph/squid/384f31f3d3aee7f8ba3f60846b8844c049165583/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-31T23:00:07.601 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r--. 1 ubuntu ubuntu 767490 Mar 31 23:00 /home/ubuntu/cephtest/cephadm 2024-03-31T23:00:07.601 DEBUG:teuthology.orchestra.run.smithi142:> curl --silent -L https://3.chacra.ceph.com/binaries/ceph/squid/384f31f3d3aee7f8ba3f60846b8844c049165583/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-31T23:00:08.021 INFO:teuthology.orchestra.run.smithi142.stdout:-rw-r--r--. 1 ubuntu ubuntu 767490 Mar 31 23:00 /home/ubuntu/cephtest/cephadm 2024-03-31T23:00:08.021 DEBUG:teuthology.orchestra.run.smithi162:> curl --silent -L https://3.chacra.ceph.com/binaries/ceph/squid/384f31f3d3aee7f8ba3f60846b8844c049165583/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-31T23:00:08.453 INFO:teuthology.orchestra.run.smithi162.stdout:-rw-r--r--. 1 ubuntu ubuntu 767490 Mar 31 23:00 /home/ubuntu/cephtest/cephadm 2024-03-31T23:00:08.454 DEBUG:teuthology.orchestra.run.smithi046:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-31T23:00:08.473 DEBUG:teuthology.orchestra.run.smithi142:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-31T23:00:08.491 DEBUG:teuthology.orchestra.run.smithi162:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-31T23:00:08.528 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:384f31f3d3aee7f8ba3f60846b8844c049165583 on all hosts... 2024-03-31T23:00:08.528 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:384f31f3d3aee7f8ba3f60846b8844c049165583 pull 2024-03-31T23:00:08.530 DEBUG:teuthology.orchestra.run.smithi142:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:384f31f3d3aee7f8ba3f60846b8844c049165583 pull 2024-03-31T23:00:08.533 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:384f31f3d3aee7f8ba3f60846b8844c049165583 pull 2024-03-31T23:00:09.331 INFO:teuthology.orchestra.run.smithi046.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:384f31f3d3aee7f8ba3f60846b8844c049165583... 2024-03-31T23:00:09.424 INFO:teuthology.orchestra.run.smithi142.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:384f31f3d3aee7f8ba3f60846b8844c049165583... 2024-03-31T23:00:09.434 INFO:teuthology.orchestra.run.smithi162.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:384f31f3d3aee7f8ba3f60846b8844c049165583... 2024-03-31T23:00:36.954 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-03-31T23:00:36.954 INFO:teuthology.orchestra.run.smithi046.stdout: "ceph_version": "ceph version 19.0.0-1977-g384f31f3 (384f31f3d3aee7f8ba3f60846b8844c049165583) squid (dev)", 2024-03-31T23:00:36.954 INFO:teuthology.orchestra.run.smithi046.stdout: "image_id": "ec18d7baa60e8b1a0b126fbe5306acce11b1ed0ec5e3bcdddbf39b0a3c591791", 2024-03-31T23:00:36.954 INFO:teuthology.orchestra.run.smithi046.stdout: "repo_digests": [ 2024-03-31T23:00:36.954 INFO:teuthology.orchestra.run.smithi046.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175" 2024-03-31T23:00:36.955 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-03-31T23:00:36.955 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-03-31T23:01:03.336 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-03-31T23:01:03.336 INFO:teuthology.orchestra.run.smithi162.stdout: "ceph_version": "ceph version 19.0.0-1977-g384f31f3 (384f31f3d3aee7f8ba3f60846b8844c049165583) squid (dev)", 2024-03-31T23:01:03.336 INFO:teuthology.orchestra.run.smithi162.stdout: "image_id": "ec18d7baa60e8b1a0b126fbe5306acce11b1ed0ec5e3bcdddbf39b0a3c591791", 2024-03-31T23:01:03.336 INFO:teuthology.orchestra.run.smithi162.stdout: "repo_digests": [ 2024-03-31T23:01:03.336 INFO:teuthology.orchestra.run.smithi162.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175" 2024-03-31T23:01:03.337 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-03-31T23:01:03.337 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-03-31T23:01:06.186 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-03-31T23:01:06.186 INFO:teuthology.orchestra.run.smithi142.stdout: "ceph_version": "ceph version 19.0.0-1977-g384f31f3 (384f31f3d3aee7f8ba3f60846b8844c049165583) squid (dev)", 2024-03-31T23:01:06.186 INFO:teuthology.orchestra.run.smithi142.stdout: "image_id": "ec18d7baa60e8b1a0b126fbe5306acce11b1ed0ec5e3bcdddbf39b0a3c591791", 2024-03-31T23:01:06.186 INFO:teuthology.orchestra.run.smithi142.stdout: "repo_digests": [ 2024-03-31T23:01:06.187 INFO:teuthology.orchestra.run.smithi142.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175" 2024-03-31T23:01:06.187 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-03-31T23:01:06.187 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-03-31T23:01:06.214 DEBUG:teuthology.orchestra.run.smithi046:> sudo mkdir -p /etc/ceph 2024-03-31T23:01:06.279 DEBUG:teuthology.orchestra.run.smithi142:> sudo mkdir -p /etc/ceph 2024-03-31T23:01:06.311 DEBUG:teuthology.orchestra.run.smithi162:> sudo mkdir -p /etc/ceph 2024-03-31T23:01:06.347 DEBUG:teuthology.orchestra.run.smithi046:> sudo chmod 777 /etc/ceph 2024-03-31T23:01:06.379 DEBUG:teuthology.orchestra.run.smithi142:> sudo chmod 777 /etc/ceph 2024-03-31T23:01:06.407 DEBUG:teuthology.orchestra.run.smithi162:> sudo chmod 777 /etc/ceph 2024-03-31T23:01:06.437 INFO:tasks.cephadm:Writing seed config... 2024-03-31T23:01:06.438 INFO:tasks.cephadm: override: [client] client mount timeout = 600 2024-03-31T23:01:06.438 INFO:tasks.cephadm: override: [client] debug client = 20 2024-03-31T23:01:06.439 INFO:tasks.cephadm: override: [client] debug ms = 1 2024-03-31T23:01:06.439 INFO:tasks.cephadm: override: [client] rados mon op timeout = 900 2024-03-31T23:01:06.439 INFO:tasks.cephadm: override: [client] rados osd op timeout = 900 2024-03-31T23:01:06.439 INFO:tasks.cephadm: override: [mds] debug mds = 20 2024-03-31T23:01:06.439 INFO:tasks.cephadm: override: [mds] debug mds balancer = 20 2024-03-31T23:01:06.439 INFO:tasks.cephadm: override: [mds] debug ms = 1 2024-03-31T23:01:06.439 INFO:tasks.cephadm: override: [mds] mds bal fragment size max = 10000 2024-03-31T23:01:06.439 INFO:tasks.cephadm: override: [mds] mds bal merge size = 5 2024-03-31T23:01:06.439 INFO:tasks.cephadm: override: [mds] mds bal split bits = 3 2024-03-31T23:01:06.439 INFO:tasks.cephadm: override: [mds] mds bal split size = 100 2024-03-31T23:01:06.440 INFO:tasks.cephadm: override: [mds] mds debug frag = True 2024-03-31T23:01:06.440 INFO:tasks.cephadm: override: [mds] mds debug scatterstat = True 2024-03-31T23:01:06.440 INFO:tasks.cephadm: override: [mds] mds op complaint time = 180 2024-03-31T23:01:06.441 INFO:tasks.cephadm: override: [mds] mds verify scatter = True 2024-03-31T23:01:06.441 INFO:tasks.cephadm: override: [mds] mds_bal_replicate_threshold = 1 2024-03-31T23:01:06.441 INFO:tasks.cephadm: override: [mds] osd op complaint time = 180 2024-03-31T23:01:06.441 INFO:tasks.cephadm: override: [mds] rados mon op timeout = 900 2024-03-31T23:01:06.441 INFO:tasks.cephadm: override: [mds] rados osd op timeout = 900 2024-03-31T23:01:06.441 INFO:tasks.cephadm: override: [mgr] cephsqlite lock renewal timeout = 900000 2024-03-31T23:01:06.441 INFO:tasks.cephadm: override: [mgr] debug cephsqlite = 20 2024-03-31T23:01:06.441 INFO:tasks.cephadm: override: [mgr] debug client = 20 2024-03-31T23:01:06.441 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-03-31T23:01:06.441 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-03-31T23:01:06.441 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-03-31T23:01:06.441 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-03-31T23:01:06.441 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-03-31T23:01:06.441 INFO:tasks.cephadm: override: [mon] mon down mkfs grace = 300 2024-03-31T23:01:06.442 INFO:tasks.cephadm: override: [mon] mon op complaint time = 120 2024-03-31T23:01:06.442 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2024-03-31T23:01:06.442 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2024-03-31T23:01:06.442 INFO:tasks.cephadm: override: [osd] debug bluefs = 20 2024-03-31T23:01:06.442 INFO:tasks.cephadm: override: [osd] debug bluestore = 20 2024-03-31T23:01:06.442 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-03-31T23:01:06.442 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-03-31T23:01:06.442 INFO:tasks.cephadm: override: [osd] debug rocksdb = 10 2024-03-31T23:01:06.442 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2024-03-31T23:01:06.442 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2024-03-31T23:01:06.442 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2024-03-31T23:01:06.442 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2024-03-31T23:01:06.443 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2024-03-31T23:01:06.443 INFO:tasks.cephadm: override: [osd] osd op complaint time = 180 2024-03-31T23:01:06.443 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-03-31T23:01:06.443 INFO:tasks.cephadm: override: [osd] osd_max_omap_entries_per_request = 10 2024-03-31T23:01:06.444 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-03-31T23:01:06.444 DEBUG:teuthology.orchestra.run.smithi046:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-03-31T23:01:06.462 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 6a9d8f08-efb2-11ee-b647-cb9ed24678a4 [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true bluestore block size = 96636764160 bluestore fsck on mount = True debug bluefs = 20 debug bluestore = 20 debug ms = 1 debug osd = 20 debug rocksdb = 10 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd objectstore = bluestore osd op complaint time = 180 osd_max_omap_entries_per_request = 10 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false cephsqlite lock renewal timeout = 900000 debug cephsqlite = 20 debug client = 20 debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon down mkfs grace = 300 mon op complaint time = 120 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] client mount timeout = 600 debug client = 20 debug ms = 1 rados mon op timeout = 900 rados osd op timeout = 900 [mds] debug mds = 20 debug mds balancer = 20 debug ms = 1 mds bal fragment size max = 10000 mds bal merge size = 5 mds bal split bits = 3 mds bal split size = 100 mds debug frag = True mds debug scatterstat = True mds op complaint time = 180 mds verify scatter = True mds_bal_replicate_threshold = 1 osd op complaint time = 180 rados mon op timeout = 900 rados osd op timeout = 900 2024-03-31T23:01:06.463 DEBUG:teuthology.orchestra.run.smithi046:mon.a> sudo journalctl -f -n 0 -u ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4@mon.a.service 2024-03-31T23:01:06.505 DEBUG:teuthology.orchestra.run.smithi046:mgr.x> sudo journalctl -f -n 0 -u ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4@mgr.x.service 2024-03-31T23:01:06.549 INFO:tasks.cephadm:Bootstrapping... 2024-03-31T23:01:06.549 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:384f31f3d3aee7f8ba3f60846b8844c049165583 -v bootstrap --fsid 6a9d8f08-efb2-11ee-b647-cb9ed24678a4 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id x --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.46 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-03-31T23:01:06.751 INFO:teuthology.orchestra.run.smithi046.stdout:-------------------------------------------------------------------------------- 2024-03-31T23:01:06.751 INFO:teuthology.orchestra.run.smithi046.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:384f31f3d3aee7f8ba3f60846b8844c049165583', '-v', 'bootstrap', '--fsid', '6a9d8f08-efb2-11ee-b647-cb9ed24678a4', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'x', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.46', '--skip-admin-label'] 2024-03-31T23:01:06.751 INFO:teuthology.orchestra.run.smithi046.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-03-31T23:01:06.752 INFO:teuthology.orchestra.run.smithi046.stdout:Verifying podman|docker is present... 2024-03-31T23:01:06.778 INFO:teuthology.orchestra.run.smithi046.stdout:/bin/podman: stdout 4.9.4-dev 2024-03-31T23:01:06.778 INFO:teuthology.orchestra.run.smithi046.stdout:Verifying lvm2 is present... 2024-03-31T23:01:06.778 INFO:teuthology.orchestra.run.smithi046.stdout:Verifying time synchronization is in place... 2024-03-31T23:01:06.785 INFO:teuthology.orchestra.run.smithi046.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-03-31T23:01:06.786 INFO:teuthology.orchestra.run.smithi046.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-03-31T23:01:06.793 INFO:teuthology.orchestra.run.smithi046.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-03-31T23:01:06.794 INFO:teuthology.orchestra.run.smithi046.stdout:systemctl: stdout inactive 2024-03-31T23:01:06.798 INFO:teuthology.orchestra.run.smithi046.stdout:systemctl: stdout enabled 2024-03-31T23:01:06.806 INFO:teuthology.orchestra.run.smithi046.stdout:systemctl: stdout active 2024-03-31T23:01:06.806 INFO:teuthology.orchestra.run.smithi046.stdout:Unit chronyd.service is enabled and running 2024-03-31T23:01:06.806 INFO:teuthology.orchestra.run.smithi046.stdout:Repeating the final host check... 2024-03-31T23:01:06.828 INFO:teuthology.orchestra.run.smithi046.stdout:/bin/podman: stdout 4.9.4-dev 2024-03-31T23:01:06.828 INFO:teuthology.orchestra.run.smithi046.stdout:podman (/bin/podman) version 4.9.4 is present 2024-03-31T23:01:06.829 INFO:teuthology.orchestra.run.smithi046.stdout:systemctl is present 2024-03-31T23:01:06.829 INFO:teuthology.orchestra.run.smithi046.stdout:lvcreate is present 2024-03-31T23:01:06.835 INFO:teuthology.orchestra.run.smithi046.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-03-31T23:01:06.835 INFO:teuthology.orchestra.run.smithi046.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-03-31T23:01:06.843 INFO:teuthology.orchestra.run.smithi046.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-03-31T23:01:06.843 INFO:teuthology.orchestra.run.smithi046.stdout:systemctl: stdout inactive 2024-03-31T23:01:06.848 INFO:teuthology.orchestra.run.smithi046.stdout:systemctl: stdout enabled 2024-03-31T23:01:06.855 INFO:teuthology.orchestra.run.smithi046.stdout:systemctl: stdout active 2024-03-31T23:01:06.856 INFO:teuthology.orchestra.run.smithi046.stdout:Unit chronyd.service is enabled and running 2024-03-31T23:01:06.856 INFO:teuthology.orchestra.run.smithi046.stdout:Host looks OK 2024-03-31T23:01:06.856 INFO:teuthology.orchestra.run.smithi046.stdout:Cluster fsid: 6a9d8f08-efb2-11ee-b647-cb9ed24678a4 2024-03-31T23:01:06.856 INFO:teuthology.orchestra.run.smithi046.stdout:Acquiring lock 140676819495712 on /run/cephadm/6a9d8f08-efb2-11ee-b647-cb9ed24678a4.lock 2024-03-31T23:01:06.856 INFO:teuthology.orchestra.run.smithi046.stdout:Lock 140676819495712 acquired on /run/cephadm/6a9d8f08-efb2-11ee-b647-cb9ed24678a4.lock 2024-03-31T23:01:06.856 INFO:teuthology.orchestra.run.smithi046.stdout:Verifying IP 172.21.15.46 port 3300 ... 2024-03-31T23:01:06.856 INFO:teuthology.orchestra.run.smithi046.stdout:Verifying IP 172.21.15.46 port 6789 ... 2024-03-31T23:01:06.856 INFO:teuthology.orchestra.run.smithi046.stdout:Base mon IP(s) is [172.21.15.46:3300, 172.21.15.46:6789], mon addrv is [v2:172.21.15.46:3300,v1:172.21.15.46:6789] 2024-03-31T23:01:06.860 INFO:teuthology.orchestra.run.smithi046.stdout:/sbin/ip: stdout default via 172.21.15.254 dev ens1f0 proto dhcp src 172.21.15.46 metric 100 2024-03-31T23:01:06.860 INFO:teuthology.orchestra.run.smithi046.stdout:/sbin/ip: stdout 172.21.0.0/20 dev ens1f0 proto kernel scope link src 172.21.15.46 metric 100 2024-03-31T23:01:06.862 INFO:teuthology.orchestra.run.smithi046.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-03-31T23:01:06.862 INFO:teuthology.orchestra.run.smithi046.stdout:/sbin/ip: stdout fe80::/64 dev ens1f0 proto kernel metric 1024 pref medium 2024-03-31T23:01:06.862 INFO:teuthology.orchestra.run.smithi046.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev ens1f0 proto ra metric 100 pref medium 2024-03-31T23:01:06.865 INFO:teuthology.orchestra.run.smithi046.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-03-31T23:01:06.865 INFO:teuthology.orchestra.run.smithi046.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-03-31T23:01:06.865 INFO:teuthology.orchestra.run.smithi046.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-03-31T23:01:06.865 INFO:teuthology.orchestra.run.smithi046.stdout:/sbin/ip: stdout 4: ens1f0: mtu 1500 state UP qlen 1000 2024-03-31T23:01:06.865 INFO:teuthology.orchestra.run.smithi046.stdout:/sbin/ip: stdout inet6 fe80::ec4:7aff:febd:15f8/64 scope link noprefixroute 2024-03-31T23:01:06.865 INFO:teuthology.orchestra.run.smithi046.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-03-31T23:01:06.866 INFO:teuthology.orchestra.run.smithi046.stdout:Mon IP `172.21.15.46` is in CIDR network `172.21.0.0/20` 2024-03-31T23:01:06.866 INFO:teuthology.orchestra.run.smithi046.stdout:Mon IP `172.21.15.46` is in CIDR network `172.21.0.0/20` 2024-03-31T23:01:06.866 INFO:teuthology.orchestra.run.smithi046.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-03-31T23:01:06.866 INFO:teuthology.orchestra.run.smithi046.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-03-31T23:01:06.867 INFO:teuthology.orchestra.run.smithi046.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:384f31f3d3aee7f8ba3f60846b8844c049165583... 2024-03-31T23:01:07.332 INFO:teuthology.orchestra.run.smithi046.stdout:/bin/podman: stdout ec18d7baa60e8b1a0b126fbe5306acce11b1ed0ec5e3bcdddbf39b0a3c591791 2024-03-31T23:01:07.332 INFO:teuthology.orchestra.run.smithi046.stdout:/bin/podman: stderr Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:384f31f3d3aee7f8ba3f60846b8844c049165583... 2024-03-31T23:01:07.332 INFO:teuthology.orchestra.run.smithi046.stdout:/bin/podman: stderr Getting image source signatures 2024-03-31T23:01:07.332 INFO:teuthology.orchestra.run.smithi046.stdout:/bin/podman: stderr Copying blob sha256:11da15500eca1f826c328eb3265caa30b0a3730510e2a32023084d6fe9cee20c 2024-03-31T23:01:07.332 INFO:teuthology.orchestra.run.smithi046.stdout:/bin/podman: stderr Copying blob sha256:bbb14f026003c3be33a985e165b4ecff104c8c0e04a9fce5f3c6b6743271902a 2024-03-31T23:01:07.332 INFO:teuthology.orchestra.run.smithi046.stdout:/bin/podman: stderr Copying config sha256:ec18d7baa60e8b1a0b126fbe5306acce11b1ed0ec5e3bcdddbf39b0a3c591791 2024-03-31T23:01:07.332 INFO:teuthology.orchestra.run.smithi046.stdout:/bin/podman: stderr Writing manifest to image destination 2024-03-31T23:01:08.320 INFO:teuthology.orchestra.run.smithi046.stdout:ceph: stdout ceph version 19.0.0-1977-g384f31f3 (384f31f3d3aee7f8ba3f60846b8844c049165583) squid (dev) 2024-03-31T23:01:08.321 INFO:teuthology.orchestra.run.smithi046.stdout:Ceph version: ceph version 19.0.0-1977-g384f31f3 (384f31f3d3aee7f8ba3f60846b8844c049165583) squid (dev) 2024-03-31T23:01:08.321 INFO:teuthology.orchestra.run.smithi046.stdout:Extracting ceph user uid/gid from container image... 2024-03-31T23:01:09.327 INFO:teuthology.orchestra.run.smithi046.stdout:stat: stdout 167 167 2024-03-31T23:01:09.327 INFO:teuthology.orchestra.run.smithi046.stdout:Creating initial keys... 2024-03-31T23:01:10.394 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph-authtool: stdout AQA16wlmAid6JxAAFXdNGAWrhwTJ82pADf6DqQ== 2024-03-31T23:01:11.234 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph-authtool: stdout AQA26wlmSljnKhAADgMYw/gzgvLPKXsGDyR0aA== 2024-03-31T23:01:12.191 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph-authtool: stdout AQA36wlmXkvWIhAA3ClbwsPOjhH82XG3aSn6NQ== 2024-03-31T23:01:12.192 INFO:teuthology.orchestra.run.smithi046.stdout:Creating initial monmap... 2024-03-31T23:01:13.123 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-03-31T23:01:13.124 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = quincy 2024-03-31T23:01:13.124 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to 6a9d8f08-efb2-11ee-b647-cb9ed24678a4 2024-03-31T23:01:13.124 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-03-31T23:01:13.124 INFO:teuthology.orchestra.run.smithi046.stdout:monmaptool for a [v2:172.21.15.46:3300,v1:172.21.15.46:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-03-31T23:01:13.124 INFO:teuthology.orchestra.run.smithi046.stdout:setting min_mon_release = quincy 2024-03-31T23:01:13.124 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/monmaptool: set fsid to 6a9d8f08-efb2-11ee-b647-cb9ed24678a4 2024-03-31T23:01:13.124 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-03-31T23:01:13.125 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-31T23:01:13.125 INFO:teuthology.orchestra.run.smithi046.stdout:Creating mon... 2024-03-31T23:01:14.028 INFO:teuthology.orchestra.run.smithi046.stdout:create mon.a on 2024-03-31T23:01:14.243 INFO:teuthology.orchestra.run.smithi046.stdout:systemctl: stderr Removed "/etc/systemd/system/multi-user.target.wants/ceph.target". 2024-03-31T23:01:14.392 INFO:teuthology.orchestra.run.smithi046.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-03-31T23:01:14.566 INFO:teuthology.orchestra.run.smithi046.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4.target → /etc/systemd/system/ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4.target. 2024-03-31T23:01:14.566 INFO:teuthology.orchestra.run.smithi046.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4.target → /etc/systemd/system/ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4.target. 2024-03-31T23:01:14.797 INFO:teuthology.orchestra.run.smithi046.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4@mon.a 2024-03-31T23:01:14.797 INFO:teuthology.orchestra.run.smithi046.stdout:systemctl: stderr Failed to reset failed state of unit ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4@mon.a.service: Unit ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4@mon.a.service not loaded. 2024-03-31T23:01:14.954 INFO:teuthology.orchestra.run.smithi046.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4.target.wants/ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4@mon.a.service → /etc/systemd/system/ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4@.service. 2024-03-31T23:01:15.501 INFO:teuthology.orchestra.run.smithi046.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-31T23:01:15.501 INFO:teuthology.orchestra.run.smithi046.stdout:systemctl: stdout disabled 2024-03-31T23:01:15.508 INFO:teuthology.orchestra.run.smithi046.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-31T23:01:15.508 INFO:teuthology.orchestra.run.smithi046.stdout:systemctl: stdout inactive 2024-03-31T23:01:15.508 INFO:teuthology.orchestra.run.smithi046.stdout:firewalld.service is not enabled 2024-03-31T23:01:15.508 INFO:teuthology.orchestra.run.smithi046.stdout:Not possible to enable service . firewalld.service is not available 2024-03-31T23:01:15.508 INFO:teuthology.orchestra.run.smithi046.stdout:Waiting for mon to start... 2024-03-31T23:01:15.508 INFO:teuthology.orchestra.run.smithi046.stdout:Waiting for mon... 2024-03-31T23:01:16.577 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:16 smithi046 ceph-mon[32126]: from='client.? 172.21.15.46:0/3636721177' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-03-31T23:01:16.579 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout cluster: 2024-03-31T23:01:16.591 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout id: 6a9d8f08-efb2-11ee-b647-cb9ed24678a4 2024-03-31T23:01:16.591 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-03-31T23:01:16.591 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout 2024-03-31T23:01:16.591 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout services: 2024-03-31T23:01:16.592 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.540015s) 2024-03-31T23:01:16.592 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-03-31T23:01:16.592 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-03-31T23:01:16.592 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout 2024-03-31T23:01:16.592 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout data: 2024-03-31T23:01:16.592 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-03-31T23:01:16.593 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-03-31T23:01:16.593 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-03-31T23:01:16.593 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout pgs: 2024-03-31T23:01:16.593 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout 2024-03-31T23:01:16.593 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:15.922+0000 7f98f7ce3700 1 Processor -- start 2024-03-31T23:01:16.594 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:15.922+0000 7f98f7ce3700 1 -- start start 2024-03-31T23:01:16.594 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:15.922+0000 7f98f7ce3700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f98f0107a90 0x7f98f0107eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:01:16.594 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:15.922+0000 7f98f7ce3700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f98f0108480 con 0x7f98f0107a90 2024-03-31T23:01:16.594 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:15.923+0000 7f98f5a7f700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f98f0107a90 0x7f98f0107eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:01:16.594 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:15.923+0000 7f98f5a7f700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f98f0107a90 0x7f98f0107eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:42620/0 (socket says 172.21.15.46:42620) 2024-03-31T23:01:16.595 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:15.923+0000 7f98f5a7f700 1 -- 172.21.15.46:0/4094849848 learned_addr learned my addr 172.21.15.46:0/4094849848 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:01:16.595 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:15.923+0000 7f98f5a7f700 1 -- 172.21.15.46:0/4094849848 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f98f0108c90 con 0x7f98f0107a90 2024-03-31T23:01:16.595 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:15.923+0000 7f98f5a7f700 1 --2- 172.21.15.46:0/4094849848 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f98f0107a90 0x7f98f0107eb0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f98e80098a0 tx=0x7f98e8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d8eb681d2a7109d server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:01:16.595 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:15.923+0000 7f98f4a7d700 1 -- 172.21.15.46:0/4094849848 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f98e8004160 con 0x7f98f0107a90 2024-03-31T23:01:16.595 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:15.923+0000 7f98f4a7d700 1 -- 172.21.15.46:0/4094849848 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f98e8004300 con 0x7f98f0107a90 2024-03-31T23:01:16.595 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:15.923+0000 7f98f4a7d700 1 -- 172.21.15.46:0/4094849848 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f98e80036a0 con 0x7f98f0107a90 2024-03-31T23:01:16.596 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:15.924+0000 7f98f7ce3700 1 -- 172.21.15.46:0/4094849848 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f98f0107a90 msgr2=0x7f98f0107eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:01:16.596 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:15.924+0000 7f98f7ce3700 1 --2- 172.21.15.46:0/4094849848 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f98f0107a90 0x7f98f0107eb0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f98e80098a0 tx=0x7f98e8009bb0 comp rx=0 tx=0).stop 2024-03-31T23:01:16.596 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:15.924+0000 7f98f7ce3700 1 -- 172.21.15.46:0/4094849848 shutdown_connections 2024-03-31T23:01:16.596 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:15.924+0000 7f98f7ce3700 1 --2- 172.21.15.46:0/4094849848 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f98f0107a90 0x7f98f0107eb0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:01:16.596 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:15.924+0000 7f98f7ce3700 1 -- 172.21.15.46:0/4094849848 >> 172.21.15.46:0/4094849848 conn(0x7f98f0103030 msgr2=0x7f98f0105470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:01:16.596 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:15.924+0000 7f98f7ce3700 1 -- 172.21.15.46:0/4094849848 shutdown_connections 2024-03-31T23:01:16.597 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:15.924+0000 7f98f7ce3700 1 -- 172.21.15.46:0/4094849848 wait complete. 2024-03-31T23:01:16.597 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:15.925+0000 7f98f7ce3700 1 Processor -- start 2024-03-31T23:01:16.597 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:15.925+0000 7f98f7ce3700 1 -- start start 2024-03-31T23:01:16.597 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:15.925+0000 7f98f7ce3700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f98f0107a90 0x7f98f007db00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:01:16.597 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:15.925+0000 7f98f7ce3700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f98f01091c0 con 0x7f98f0107a90 2024-03-31T23:01:16.597 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:15.925+0000 7f98f5a7f700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f98f0107a90 0x7f98f007db00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:01:16.598 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:15.925+0000 7f98f5a7f700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f98f0107a90 0x7f98f007db00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:42626/0 (socket says 172.21.15.46:42626) 2024-03-31T23:01:16.598 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:15.925+0000 7f98f5a7f700 1 -- 172.21.15.46:0/3636721177 learned_addr learned my addr 172.21.15.46:0/3636721177 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:01:16.598 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:15.925+0000 7f98f5a7f700 1 -- 172.21.15.46:0/3636721177 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f98f007e040 con 0x7f98f0107a90 2024-03-31T23:01:16.598 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:15.925+0000 7f98f5a7f700 1 --2- 172.21.15.46:0/3636721177 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f98f0107a90 0x7f98f007db00 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f98f0108720 tx=0x7f98e8004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:01:16.598 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:15.925+0000 7f98eeffd700 1 -- 172.21.15.46:0/3636721177 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f98e8003db0 con 0x7f98f0107a90 2024-03-31T23:01:16.598 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:15.925+0000 7f98eeffd700 1 -- 172.21.15.46:0/3636721177 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f98e8011630 con 0x7f98f0107a90 2024-03-31T23:01:16.599 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:15.925+0000 7f98f7ce3700 1 -- 172.21.15.46:0/3636721177 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f98f007c230 con 0x7f98f0107a90 2024-03-31T23:01:16.599 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:15.925+0000 7f98eeffd700 1 -- 172.21.15.46:0/3636721177 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f98e8011930 con 0x7f98f0107a90 2024-03-31T23:01:16.599 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:15.926+0000 7f98f7ce3700 1 -- 172.21.15.46:0/3636721177 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f98f007c770 con 0x7f98f0107a90 2024-03-31T23:01:16.599 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:15.926+0000 7f98eeffd700 1 -- 172.21.15.46:0/3636721177 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f98e8024480 con 0x7f98f0107a90 2024-03-31T23:01:16.599 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:15.926+0000 7f98eeffd700 1 -- 172.21.15.46:0/3636721177 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f98e801a7a0 con 0x7f98f0107a90 2024-03-31T23:01:16.600 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:15.926+0000 7f98f7ce3700 1 -- 172.21.15.46:0/3636721177 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f98f0063870 con 0x7f98f0107a90 2024-03-31T23:01:16.600 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:15.929+0000 7f98eeffd700 1 -- 172.21.15.46:0/3636721177 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f98e801aa40 con 0x7f98f0107a90 2024-03-31T23:01:16.600 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:16.001+0000 7f98f7ce3700 1 -- 172.21.15.46:0/3636721177 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "status"} v 0) -- 0x7f98f0194990 con 0x7f98f0107a90 2024-03-31T23:01:16.600 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:16.001+0000 7f98eeffd700 1 -- 172.21.15.46:0/3636721177 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) ==== 54+0+317 (secure 0 0 0) 0x7f98e801ac20 con 0x7f98f0107a90 2024-03-31T23:01:16.600 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:16.002+0000 7f98f7ce3700 1 -- 172.21.15.46:0/3636721177 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f98f0107a90 msgr2=0x7f98f007db00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:01:16.600 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:16.002+0000 7f98f7ce3700 1 --2- 172.21.15.46:0/3636721177 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f98f0107a90 0x7f98f007db00 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f98f0108720 tx=0x7f98e8004750 comp rx=0 tx=0).stop 2024-03-31T23:01:16.601 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:16.002+0000 7f98f7ce3700 1 -- 172.21.15.46:0/3636721177 shutdown_connections 2024-03-31T23:01:16.601 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:16.002+0000 7f98f7ce3700 1 --2- 172.21.15.46:0/3636721177 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f98f0107a90 0x7f98f007db00 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:01:16.601 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:16.002+0000 7f98f7ce3700 1 -- 172.21.15.46:0/3636721177 >> 172.21.15.46:0/3636721177 conn(0x7f98f0103030 msgr2=0x7f98f0191df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:01:16.601 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:16.002+0000 7f98f7ce3700 1 -- 172.21.15.46:0/3636721177 shutdown_connections 2024-03-31T23:01:16.601 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:16.002+0000 7f98f7ce3700 1 -- 172.21.15.46:0/3636721177 wait complete. 2024-03-31T23:01:16.601 INFO:teuthology.orchestra.run.smithi046.stdout:mon is available 2024-03-31T23:01:16.602 INFO:teuthology.orchestra.run.smithi046.stdout:Assimilating anything we can from ceph.conf... 2024-03-31T23:01:17.584 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:17 smithi046 ceph-mon[32126]: from='client.? 172.21.15.46:0/2131477143' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-03-31T23:01:17.584 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:17 smithi046 ceph-mon[32126]: from='client.? 172.21.15.46:0/2131477143' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-03-31T23:01:17.593 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout 2024-03-31T23:01:17.593 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout [global] 2024-03-31T23:01:17.593 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout fsid = 6a9d8f08-efb2-11ee-b647-cb9ed24678a4 2024-03-31T23:01:17.593 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.46:3300,v1:172.21.15.46:6789] 2024-03-31T23:01:17.593 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-03-31T23:01:17.593 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-03-31T23:01:17.593 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-03-31T23:01:17.593 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-03-31T23:01:17.593 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout 2024-03-31T23:01:17.593 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout [mgr] 2024-03-31T23:01:17.594 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-03-31T23:01:17.594 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout 2024-03-31T23:01:17.594 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout [osd] 2024-03-31T23:01:17.594 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-03-31T23:01:17.594 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-03-31T23:01:17.594 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-03-31T23:01:17.594 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-03-31T23:01:17.594 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-03-31T23:01:17.594 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:16.966+0000 7f414b39e700 1 Processor -- start 2024-03-31T23:01:17.594 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:16.966+0000 7f414b39e700 1 -- start start 2024-03-31T23:01:17.594 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:16.966+0000 7f414b39e700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4144107910 0x7f4144107d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:01:17.594 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:16.966+0000 7f414b39e700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f4144108300 con 0x7f4144107910 2024-03-31T23:01:17.594 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:16.966+0000 7f414a39c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4144107910 0x7f4144107d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:01:17.594 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:16.966+0000 7f414a39c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4144107910 0x7f4144107d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:42632/0 (socket says 172.21.15.46:42632) 2024-03-31T23:01:17.595 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:16.966+0000 7f414a39c700 1 -- 172.21.15.46:0/4288858195 learned_addr learned my addr 172.21.15.46:0/4288858195 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:01:17.595 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:16.966+0000 7f414a39c700 1 -- 172.21.15.46:0/4288858195 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4144108b10 con 0x7f4144107910 2024-03-31T23:01:17.595 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:16.966+0000 7f414a39c700 1 --2- 172.21.15.46:0/4288858195 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4144107910 0x7f4144107d30 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f41340098a0 tx=0x7f4134009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a6885534e7057855 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:01:17.595 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:16.967+0000 7f414939a700 1 -- 172.21.15.46:0/4288858195 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4134004160 con 0x7f4144107910 2024-03-31T23:01:17.595 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:16.967+0000 7f414939a700 1 -- 172.21.15.46:0/4288858195 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4134004300 con 0x7f4144107910 2024-03-31T23:01:17.595 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:16.967+0000 7f414939a700 1 -- 172.21.15.46:0/4288858195 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f41340036a0 con 0x7f4144107910 2024-03-31T23:01:17.595 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:16.967+0000 7f414b39e700 1 -- 172.21.15.46:0/4288858195 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4144107910 msgr2=0x7f4144107d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:01:17.595 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:16.967+0000 7f414b39e700 1 --2- 172.21.15.46:0/4288858195 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4144107910 0x7f4144107d30 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f41340098a0 tx=0x7f4134009bb0 comp rx=0 tx=0).stop 2024-03-31T23:01:17.595 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:16.967+0000 7f414b39e700 1 -- 172.21.15.46:0/4288858195 shutdown_connections 2024-03-31T23:01:17.595 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:16.967+0000 7f414b39e700 1 --2- 172.21.15.46:0/4288858195 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4144107910 0x7f4144107d30 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:01:17.595 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:16.967+0000 7f414b39e700 1 -- 172.21.15.46:0/4288858195 >> 172.21.15.46:0/4288858195 conn(0x7f4144102ef0 msgr2=0x7f4144105310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:01:17.595 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:16.967+0000 7f414b39e700 1 -- 172.21.15.46:0/4288858195 shutdown_connections 2024-03-31T23:01:17.595 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:16.967+0000 7f414b39e700 1 -- 172.21.15.46:0/4288858195 wait complete. 2024-03-31T23:01:17.595 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:16.968+0000 7f414b39e700 1 Processor -- start 2024-03-31T23:01:17.596 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:16.968+0000 7f414b39e700 1 -- start start 2024-03-31T23:01:17.596 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:16.968+0000 7f414b39e700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4144107910 0x7f414419d210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:01:17.596 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:16.968+0000 7f414b39e700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f4144109040 con 0x7f4144107910 2024-03-31T23:01:17.596 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:16.968+0000 7f414a39c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4144107910 0x7f414419d210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:01:17.596 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:16.968+0000 7f414a39c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4144107910 0x7f414419d210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:42640/0 (socket says 172.21.15.46:42640) 2024-03-31T23:01:17.596 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:16.968+0000 7f414a39c700 1 -- 172.21.15.46:0/2131477143 learned_addr learned my addr 172.21.15.46:0/2131477143 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:01:17.596 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:16.969+0000 7f414a39c700 1 -- 172.21.15.46:0/2131477143 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f414419d750 con 0x7f4144107910 2024-03-31T23:01:17.596 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:16.969+0000 7f414a39c700 1 --2- 172.21.15.46:0/2131477143 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4144107910 0x7f414419d210 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f4134000c00 tx=0x7f4134004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:01:17.596 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:16.969+0000 7f413b7fe700 1 -- 172.21.15.46:0/2131477143 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4134003db0 con 0x7f4144107910 2024-03-31T23:01:17.596 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:16.969+0000 7f413b7fe700 1 -- 172.21.15.46:0/2131477143 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4134011630 con 0x7f4144107910 2024-03-31T23:01:17.596 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:16.969+0000 7f414b39e700 1 -- 172.21.15.46:0/2131477143 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f414419d9e0 con 0x7f4144107910 2024-03-31T23:01:17.596 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:16.969+0000 7f413b7fe700 1 -- 172.21.15.46:0/2131477143 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4134011950 con 0x7f4144107910 2024-03-31T23:01:17.596 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:16.969+0000 7f414b39e700 1 -- 172.21.15.46:0/2131477143 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f41441a06d0 con 0x7f4144107910 2024-03-31T23:01:17.596 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:16.969+0000 7f413b7fe700 1 -- 172.21.15.46:0/2131477143 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f4134011af0 con 0x7f4144107910 2024-03-31T23:01:17.597 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:16.969+0000 7f413b7fe700 1 -- 172.21.15.46:0/2131477143 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f413401c450 con 0x7f4144107910 2024-03-31T23:01:17.597 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:16.970+0000 7f414b39e700 1 -- 172.21.15.46:0/2131477143 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4144063870 con 0x7f4144107910 2024-03-31T23:01:17.597 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:16.972+0000 7f413b7fe700 1 -- 172.21.15.46:0/2131477143 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f4134004260 con 0x7f4144107910 2024-03-31T23:01:17.597 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:17.042+0000 7f414b39e700 1 -- 172.21.15.46:0/2131477143 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) -- 0x7f41441a08a0 con 0x7f4144107910 2024-03-31T23:01:17.597 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:17.045+0000 7f413b7fe700 1 -- 172.21.15.46:0/2131477143 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) ==== 70+0+429 (secure 0 0 0) 0x7f413401b450 con 0x7f4144107910 2024-03-31T23:01:17.597 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:17.046+0000 7f413b7fe700 1 -- 172.21.15.46:0/2131477143 <== mon.0 v2:172.21.15.46:3300/0 8 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7f413401b630 con 0x7f4144107910 2024-03-31T23:01:17.597 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:17.046+0000 7f414b39e700 1 -- 172.21.15.46:0/2131477143 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4144107910 msgr2=0x7f414419d210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:01:17.597 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:17.046+0000 7f414b39e700 1 --2- 172.21.15.46:0/2131477143 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4144107910 0x7f414419d210 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f4134000c00 tx=0x7f4134004750 comp rx=0 tx=0).stop 2024-03-31T23:01:17.597 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:17.046+0000 7f414b39e700 1 -- 172.21.15.46:0/2131477143 shutdown_connections 2024-03-31T23:01:17.597 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:17.046+0000 7f414b39e700 1 --2- 172.21.15.46:0/2131477143 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4144107910 0x7f414419d210 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:01:17.597 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:17.046+0000 7f414b39e700 1 -- 172.21.15.46:0/2131477143 >> 172.21.15.46:0/2131477143 conn(0x7f4144102ef0 msgr2=0x7f414407d270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:01:17.597 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:17.046+0000 7f414b39e700 1 -- 172.21.15.46:0/2131477143 shutdown_connections 2024-03-31T23:01:17.597 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:17.046+0000 7f414b39e700 1 -- 172.21.15.46:0/2131477143 wait complete. 2024-03-31T23:01:17.597 INFO:teuthology.orchestra.run.smithi046.stdout:Generating new minimal ceph.conf... 2024-03-31T23:01:18.649 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:18 smithi046 ceph-mon[32126]: from='client.? 172.21.15.46:0/2529930842' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:01:18.657 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:18.007+0000 7f250d3a2700 1 Processor -- start 2024-03-31T23:01:18.657 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:18.007+0000 7f250d3a2700 1 -- start start 2024-03-31T23:01:18.657 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:18.007+0000 7f250d3a2700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2508107910 0x7f2508107d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:01:18.657 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:18.007+0000 7f250d3a2700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f2508108300 con 0x7f2508107910 2024-03-31T23:01:18.657 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:18.007+0000 7f2507fff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2508107910 0x7f2508107d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:01:18.657 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:18.007+0000 7f2507fff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2508107910 0x7f2508107d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:42654/0 (socket says 172.21.15.46:42654) 2024-03-31T23:01:18.657 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:18.007+0000 7f2507fff700 1 -- 172.21.15.46:0/306055556 learned_addr learned my addr 172.21.15.46:0/306055556 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:01:18.657 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:18.008+0000 7f2507fff700 1 -- 172.21.15.46:0/306055556 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2508108b10 con 0x7f2508107910 2024-03-31T23:01:18.657 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:18.008+0000 7f2507fff700 1 --2- 172.21.15.46:0/306055556 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2508107910 0x7f2508107d30 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f24f80098a0 tx=0x7f24f8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=88446fd16902de61 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:01:18.657 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:18.008+0000 7f2506ffd700 1 -- 172.21.15.46:0/306055556 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f24f8004160 con 0x7f2508107910 2024-03-31T23:01:18.658 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:18.008+0000 7f2506ffd700 1 -- 172.21.15.46:0/306055556 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7f24f8004300 con 0x7f2508107910 2024-03-31T23:01:18.658 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:18.008+0000 7f2506ffd700 1 -- 172.21.15.46:0/306055556 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f24f8011a80 con 0x7f2508107910 2024-03-31T23:01:18.658 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:18.008+0000 7f250d3a2700 1 -- 172.21.15.46:0/306055556 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2508107910 msgr2=0x7f2508107d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:01:18.658 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:18.008+0000 7f250d3a2700 1 --2- 172.21.15.46:0/306055556 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2508107910 0x7f2508107d30 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f24f80098a0 tx=0x7f24f8009bb0 comp rx=0 tx=0).stop 2024-03-31T23:01:18.658 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:18.009+0000 7f250d3a2700 1 -- 172.21.15.46:0/306055556 shutdown_connections 2024-03-31T23:01:18.658 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:18.009+0000 7f250d3a2700 1 --2- 172.21.15.46:0/306055556 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2508107910 0x7f2508107d30 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:01:18.658 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:18.009+0000 7f250d3a2700 1 -- 172.21.15.46:0/306055556 >> 172.21.15.46:0/306055556 conn(0x7f2508102ef0 msgr2=0x7f2508105310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:01:18.658 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:18.009+0000 7f250d3a2700 1 -- 172.21.15.46:0/306055556 shutdown_connections 2024-03-31T23:01:18.658 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:18.009+0000 7f250d3a2700 1 -- 172.21.15.46:0/306055556 wait complete. 2024-03-31T23:01:18.658 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:18.009+0000 7f250d3a2700 1 Processor -- start 2024-03-31T23:01:18.658 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:18.009+0000 7f250d3a2700 1 -- start start 2024-03-31T23:01:18.658 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:18.010+0000 7f250d3a2700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2508107910 0x7f250819b880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:01:18.658 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:18.010+0000 7f250d3a2700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f250810bec0 con 0x7f2508107910 2024-03-31T23:01:18.659 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:18.010+0000 7f2507fff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2508107910 0x7f250819b880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:01:18.659 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:18.010+0000 7f2507fff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2508107910 0x7f250819b880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:42656/0 (socket says 172.21.15.46:42656) 2024-03-31T23:01:18.659 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:18.010+0000 7f2507fff700 1 -- 172.21.15.46:0/2529930842 learned_addr learned my addr 172.21.15.46:0/2529930842 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:01:18.659 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:18.010+0000 7f2507fff700 1 -- 172.21.15.46:0/2529930842 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f250819bdc0 con 0x7f2508107910 2024-03-31T23:01:18.659 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:18.010+0000 7f2507fff700 1 --2- 172.21.15.46:0/2529930842 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2508107910 0x7f250819b880 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f24f8000c00 tx=0x7f24f8011b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:01:18.659 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:18.010+0000 7f25057fa700 1 -- 172.21.15.46:0/2529930842 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f24f80039f0 con 0x7f2508107910 2024-03-31T23:01:18.659 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:18.010+0000 7f250d3a2700 1 -- 172.21.15.46:0/2529930842 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f250819c050 con 0x7f2508107910 2024-03-31T23:01:18.659 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:18.010+0000 7f25057fa700 1 -- 172.21.15.46:0/2529930842 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7f24f8003b90 con 0x7f2508107910 2024-03-31T23:01:18.659 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:18.010+0000 7f25057fa700 1 -- 172.21.15.46:0/2529930842 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f24f801a740 con 0x7f2508107910 2024-03-31T23:01:18.659 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:18.010+0000 7f250d3a2700 1 -- 172.21.15.46:0/2529930842 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f250819ed40 con 0x7f2508107910 2024-03-31T23:01:18.659 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:18.010+0000 7f25057fa700 1 -- 172.21.15.46:0/2529930842 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f24f802d3f0 con 0x7f2508107910 2024-03-31T23:01:18.659 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:18.011+0000 7f25057fa700 1 -- 172.21.15.46:0/2529930842 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f24f8019700 con 0x7f2508107910 2024-03-31T23:01:18.659 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:18.011+0000 7f250d3a2700 1 -- 172.21.15.46:0/2529930842 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2508063870 con 0x7f2508107910 2024-03-31T23:01:18.659 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:18.014+0000 7f25057fa700 1 -- 172.21.15.46:0/2529930842 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f24f8024af0 con 0x7f2508107910 2024-03-31T23:01:18.660 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:18.085+0000 7f250d3a2700 1 -- 172.21.15.46:0/2529930842 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) -- 0x7f250819f290 con 0x7f2508107910 2024-03-31T23:01:18.660 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:18.085+0000 7f25057fa700 1 -- 172.21.15.46:0/2529930842 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) ==== 76+0+175 (secure 0 0 0) 0x7f24f80199a0 con 0x7f2508107910 2024-03-31T23:01:18.660 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:18.086+0000 7f250d3a2700 1 -- 172.21.15.46:0/2529930842 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2508107910 msgr2=0x7f250819b880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:01:18.660 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:18.086+0000 7f250d3a2700 1 --2- 172.21.15.46:0/2529930842 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2508107910 0x7f250819b880 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f24f8000c00 tx=0x7f24f8011b00 comp rx=0 tx=0).stop 2024-03-31T23:01:18.660 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:18.086+0000 7f250d3a2700 1 -- 172.21.15.46:0/2529930842 shutdown_connections 2024-03-31T23:01:18.660 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:18.086+0000 7f250d3a2700 1 --2- 172.21.15.46:0/2529930842 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2508107910 0x7f250819b880 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:01:18.660 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:18.086+0000 7f250d3a2700 1 -- 172.21.15.46:0/2529930842 >> 172.21.15.46:0/2529930842 conn(0x7f2508102ef0 msgr2=0x7f2508103a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:01:18.660 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:18.086+0000 7f250d3a2700 1 -- 172.21.15.46:0/2529930842 shutdown_connections 2024-03-31T23:01:18.660 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:18.086+0000 7f250d3a2700 1 -- 172.21.15.46:0/2529930842 wait complete. 2024-03-31T23:01:18.660 INFO:teuthology.orchestra.run.smithi046.stdout:Restarting the monitor... 2024-03-31T23:01:18.911 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:18 smithi046 systemd[1]: Stopping Ceph mon.a for 6a9d8f08-efb2-11ee-b647-cb9ed24678a4... 2024-03-31T23:01:19.362 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:18 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mon-a[32122]: 2024-03-31T23:01:18.910+0000 7feea7da5700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-31T23:01:19.362 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:18 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mon-a[32122]: 2024-03-31T23:01:18.910+0000 7feea7da5700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-03-31T23:01:19.362 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:19 smithi046 podman[32372]: 2024-03-31 23:01:19.074701208 +0000 UTC m=+0.257042002 container died 750dc8bb858ca1d113414c8085850b188457a04204a96ee681483b5eb96abde1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:384f31f3d3aee7f8ba3f60846b8844c049165583, name=ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mon-a, org.label-schema.build-date=20240326, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=squid-384f31f, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5) 2024-03-31T23:01:19.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:19 smithi046 podman[32372]: 2024-03-31 23:01:19.502181092 +0000 UTC m=+0.684521886 container cleanup 750dc8bb858ca1d113414c8085850b188457a04204a96ee681483b5eb96abde1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:384f31f3d3aee7f8ba3f60846b8844c049165583, name=ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mon-a, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, RELEASE=squid-384f31f, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240326, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, ceph=True) 2024-03-31T23:01:19.862 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:19 smithi046 bash[32372]: ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mon-a 2024-03-31T23:01:19.862 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:19 smithi046 podman[32386]: 2024-03-31 23:01:19.669944904 +0000 UTC m=+0.598382537 container remove 750dc8bb858ca1d113414c8085850b188457a04204a96ee681483b5eb96abde1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:384f31f3d3aee7f8ba3f60846b8844c049165583, name=ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mon-a, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=squid-384f31f, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240326, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-31T23:01:20.185 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:19 smithi046 systemd[1]: ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4@mon.a.service: Deactivated successfully. 2024-03-31T23:01:20.185 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:19 smithi046 systemd[1]: Stopped Ceph mon.a for 6a9d8f08-efb2-11ee-b647-cb9ed24678a4. 2024-03-31T23:01:20.185 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:19 smithi046 systemd[1]: Starting Ceph mon.a for 6a9d8f08-efb2-11ee-b647-cb9ed24678a4... 2024-03-31T23:01:20.185 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:20 smithi046 podman[32479]: 2024-03-31 23:01:20.112393137 +0000 UTC m=+0.093032774 container create 857af8e9024c6c11705485611e9ddc07a77e0a6606eb97c45f49b8633751af71 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:384f31f3d3aee7f8ba3f60846b8844c049165583, name=ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mon-a, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240326, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, RELEASE=squid-384f31f, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-03-31T23:01:20.185 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:20 smithi046 podman[32479]: 2024-03-31 23:01:20.036772884 +0000 UTC m=+0.017412534 image pull ec18d7baa60e8b1a0b126fbe5306acce11b1ed0ec5e3bcdddbf39b0a3c591791 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:384f31f3d3aee7f8ba3f60846b8844c049165583 2024-03-31T23:01:20.335 INFO:teuthology.orchestra.run.smithi046.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-03-31T23:01:20.459 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:20 smithi046 podman[32479]: 2024-03-31 23:01:20.256254833 +0000 UTC m=+0.236894469 container init 857af8e9024c6c11705485611e9ddc07a77e0a6606eb97c45f49b8633751af71 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:384f31f3d3aee7f8ba3f60846b8844c049165583, name=ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mon-a, RELEASE=squid-384f31f, org.label-schema.build-date=20240326, ceph=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-03-31T23:01:20.459 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:20 smithi046 podman[32479]: 2024-03-31 23:01:20.259050788 +0000 UTC m=+0.239690427 container start 857af8e9024c6c11705485611e9ddc07a77e0a6606eb97c45f49b8633751af71 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:384f31f3d3aee7f8ba3f60846b8844c049165583, name=ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mon-a, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.build-date=20240326, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, RELEASE=squid-384f31f, ceph=True) 2024-03-31T23:01:20.459 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:20 smithi046 ceph-mon[32498]: set uid:gid to 167:167 (ceph:ceph) 2024-03-31T23:01:20.459 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:20 smithi046 ceph-mon[32498]: ceph version 19.0.0-1977-g384f31f3 (384f31f3d3aee7f8ba3f60846b8844c049165583) squid (dev), process ceph-mon, pid 2 2024-03-31T23:01:20.459 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:20 smithi046 ceph-mon[32498]: pidfile_write: ignore empty --pid-file 2024-03-31T23:01:20.459 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:20 smithi046 ceph-mon[32498]: load: jerasure load: lrc 2024-03-31T23:01:20.459 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:20 smithi046 ceph-mon[32498]: rocksdb: RocksDB version: 7.9.2 2024-03-31T23:01:20.459 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:20 smithi046 ceph-mon[32498]: rocksdb: Git sha 0 2024-03-31T23:01:20.460 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:20 smithi046 ceph-mon[32498]: rocksdb: Compile date 2024-03-31 16:17:26 2024-03-31T23:01:20.460 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:20 smithi046 ceph-mon[32498]: rocksdb: DB SUMMARY 2024-03-31T23:01:20.460 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:20 smithi046 ceph-mon[32498]: rocksdb: DB Session ID: I7UP2ZW29TPB6D34KA36 2024-03-31T23:01:20.460 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:20 smithi046 ceph-mon[32498]: rocksdb: CURRENT file: CURRENT 2024-03-31T23:01:20.460 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:20 smithi046 ceph-mon[32498]: rocksdb: IDENTITY file: IDENTITY 2024-03-31T23:01:20.460 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:20 smithi046 ceph-mon[32498]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2024-03-31T23:01:20.461 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:20 smithi046 ceph-mon[32498]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-03-31T23:01:20.461 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:20 smithi046 ceph-mon[32498]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 104178 ; 2024-03-31T23:01:20.461 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:20 smithi046 ceph-mon[32498]: rocksdb: Options.error_if_exists: 0 2024-03-31T23:01:20.461 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:20 smithi046 ceph-mon[32498]: rocksdb: Options.create_if_missing: 0 2024-03-31T23:01:20.461 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:20 smithi046 ceph-mon[32498]: rocksdb: Options.paranoid_checks: 1 2024-03-31T23:01:20.461 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:20 smithi046 ceph-mon[32498]: rocksdb: Options.flush_verify_memtable_count: 1 2024-03-31T23:01:20.461 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:20 smithi046 ceph-mon[32498]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-31T23:01:20.462 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:20 smithi046 ceph-mon[32498]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-03-31T23:01:20.462 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:20 smithi046 ceph-mon[32498]: rocksdb: Options.env: 0x5650d2a0cc60 2024-03-31T23:01:20.462 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:20 smithi046 ceph-mon[32498]: rocksdb: Options.fs: PosixFileSystem 2024-03-31T23:01:20.462 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:20 smithi046 ceph-mon[32498]: rocksdb: Options.info_log: 0x5650d357dae0 2024-03-31T23:01:20.462 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:20 smithi046 ceph-mon[32498]: rocksdb: Options.max_file_opening_threads: 16 2024-03-31T23:01:20.462 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:20 smithi046 ceph-mon[32498]: rocksdb: Options.statistics: (nil) 2024-03-31T23:01:20.462 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:20 smithi046 ceph-mon[32498]: rocksdb: Options.use_fsync: 0 2024-03-31T23:01:20.463 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:20 smithi046 ceph-mon[32498]: rocksdb: Options.max_log_file_size: 0 2024-03-31T23:01:20.463 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:20 smithi046 ceph-mon[32498]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-31T23:01:20.463 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:20 smithi046 ceph-mon[32498]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-31T23:01:20.463 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:20 smithi046 ceph-mon[32498]: rocksdb: Options.keep_log_file_num: 1000 2024-03-31T23:01:20.463 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:20 smithi046 ceph-mon[32498]: rocksdb: Options.recycle_log_file_num: 0 2024-03-31T23:01:20.463 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:20 smithi046 ceph-mon[32498]: rocksdb: Options.allow_fallocate: 1 2024-03-31T23:01:20.463 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:20 smithi046 ceph-mon[32498]: rocksdb: Options.allow_mmap_reads: 0 2024-03-31T23:01:20.463 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:20 smithi046 ceph-mon[32498]: rocksdb: Options.allow_mmap_writes: 0 2024-03-31T23:01:20.464 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:20 smithi046 ceph-mon[32498]: rocksdb: Options.use_direct_reads: 0 2024-03-31T23:01:20.464 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:20 smithi046 ceph-mon[32498]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-31T23:01:20.464 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:20 smithi046 ceph-mon[32498]: rocksdb: Options.create_missing_column_families: 0 2024-03-31T23:01:20.464 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:20 smithi046 ceph-mon[32498]: rocksdb: Options.db_log_dir: 2024-03-31T23:01:20.464 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:20 smithi046 ceph-mon[32498]: rocksdb: Options.wal_dir: 2024-03-31T23:01:20.464 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:20 smithi046 ceph-mon[32498]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-31T23:01:20.464 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:20 smithi046 ceph-mon[32498]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-31T23:01:20.464 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:20 smithi046 ceph-mon[32498]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-31T23:01:20.464 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:20 smithi046 ceph-mon[32498]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-31T23:01:20.465 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:20 smithi046 ceph-mon[32498]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-31T23:01:20.465 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:20 smithi046 ceph-mon[32498]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-31T23:01:20.465 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:20 smithi046 ceph-mon[32498]: rocksdb: Options.advise_random_on_open: 1 2024-03-31T23:01:20.466 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:20 smithi046 ceph-mon[32498]: rocksdb: Options.db_write_buffer_size: 0 2024-03-31T23:01:20.466 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:20 smithi046 ceph-mon[32498]: rocksdb: Options.write_buffer_manager: 0x5650d35501e0 2024-03-31T23:01:20.466 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:20 smithi046 ceph-mon[32498]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-31T23:01:20.466 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:20 smithi046 ceph-mon[32498]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-31T23:01:20.466 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:20 smithi046 ceph-mon[32498]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-31T23:01:20.466 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:20 smithi046 ceph-mon[32498]: rocksdb: Options.rate_limiter: (nil) 2024-03-31T23:01:20.467 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:20 smithi046 ceph-mon[32498]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-31T23:01:20.467 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:20 smithi046 ceph-mon[32498]: rocksdb: Options.wal_recovery_mode: 2 2024-03-31T23:01:20.467 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:20 smithi046 ceph-mon[32498]: rocksdb: Options.enable_thread_tracking: 0 2024-03-31T23:01:20.467 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:20 smithi046 ceph-mon[32498]: rocksdb: Options.enable_pipelined_write: 0 2024-03-31T23:01:20.467 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:20 smithi046 ceph-mon[32498]: rocksdb: Options.unordered_write: 0 2024-03-31T23:01:20.467 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:20 smithi046 ceph-mon[32498]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-31T23:01:20.467 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:20 smithi046 ceph-mon[32498]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-31T23:01:20.467 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:20 smithi046 ceph-mon[32498]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-31T23:01:20.467 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:20 smithi046 ceph-mon[32498]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-31T23:01:20.467 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:20 smithi046 ceph-mon[32498]: rocksdb: Options.row_cache: None 2024-03-31T23:01:20.467 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:20 smithi046 ceph-mon[32498]: rocksdb: Options.wal_filter: None 2024-03-31T23:01:20.467 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:20 smithi046 ceph-mon[32498]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-31T23:01:20.469 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:20 smithi046 ceph-mon[32498]: rocksdb: Options.allow_ingest_behind: 0 2024-03-31T23:01:20.470 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:20 smithi046 ceph-mon[32498]: rocksdb: Options.two_write_queues: 0 2024-03-31T23:01:20.470 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:20 smithi046 ceph-mon[32498]: rocksdb: Options.manual_wal_flush: 0 2024-03-31T23:01:20.470 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:20 smithi046 ceph-mon[32498]: rocksdb: Options.wal_compression: 0 2024-03-31T23:01:20.470 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:20 smithi046 ceph-mon[32498]: rocksdb: Options.atomic_flush: 0 2024-03-31T23:01:20.470 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:20 smithi046 ceph-mon[32498]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-31T23:01:20.470 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:20 smithi046 ceph-mon[32498]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-31T23:01:20.470 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:20 smithi046 ceph-mon[32498]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-31T23:01:20.470 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:20 smithi046 ceph-mon[32498]: rocksdb: Options.log_readahead_size: 0 2024-03-31T23:01:20.470 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:20 smithi046 ceph-mon[32498]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-31T23:01:20.470 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:20 smithi046 ceph-mon[32498]: rocksdb: Options.best_efforts_recovery: 0 2024-03-31T23:01:20.470 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:20 smithi046 ceph-mon[32498]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-31T23:01:20.470 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:20 smithi046 ceph-mon[32498]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-31T23:01:20.471 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:20 smithi046 ceph-mon[32498]: rocksdb: Options.allow_data_in_errors: 0 2024-03-31T23:01:20.471 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:20 smithi046 ceph-mon[32498]: rocksdb: Options.db_host_id: __hostname__ 2024-03-31T23:01:20.471 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:20 smithi046 ceph-mon[32498]: rocksdb: Options.enforce_single_del_contracts: true 2024-03-31T23:01:20.471 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:20 smithi046 ceph-mon[32498]: rocksdb: Options.max_background_jobs: 2 2024-03-31T23:01:20.471 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:20 smithi046 ceph-mon[32498]: rocksdb: Options.max_background_compactions: -1 2024-03-31T23:01:20.471 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:20 smithi046 ceph-mon[32498]: rocksdb: Options.max_subcompactions: 1 2024-03-31T23:01:20.471 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:20 smithi046 ceph-mon[32498]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-31T23:01:20.471 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:20 smithi046 ceph-mon[32498]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-31T23:01:20.471 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:20 smithi046 ceph-mon[32498]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-31T23:01:20.471 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:20 smithi046 ceph-mon[32498]: rocksdb: Options.max_total_wal_size: 0 2024-03-31T23:01:20.472 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:20 smithi046 ceph-mon[32498]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-31T23:01:20.472 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:20 smithi046 ceph-mon[32498]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-31T23:01:20.472 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:20 smithi046 ceph-mon[32498]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-31T23:01:20.472 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:20 smithi046 ceph-mon[32498]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-31T23:01:20.472 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:20 smithi046 ceph-mon[32498]: rocksdb: Options.max_open_files: -1 2024-03-31T23:01:20.472 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:20 smithi046 ceph-mon[32498]: rocksdb: Options.bytes_per_sync: 0 2024-03-31T23:01:20.472 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:20 smithi046 ceph-mon[32498]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-31T23:01:20.472 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:20 smithi046 ceph-mon[32498]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-31T23:01:20.473 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:20 smithi046 ceph-mon[32498]: rocksdb: Options.compaction_readahead_size: 0 2024-03-31T23:01:20.473 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:20 smithi046 ceph-mon[32498]: rocksdb: Options.max_background_flushes: -1 2024-03-31T23:01:20.473 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:20 smithi046 ceph-mon[32498]: rocksdb: Compression algorithms supported: 2024-03-31T23:01:20.473 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:20 smithi046 ceph-mon[32498]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-31T23:01:20.473 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:20 smithi046 ceph-mon[32498]: rocksdb: kZSTD supported: 0 2024-03-31T23:01:20.473 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:20 smithi046 ceph-mon[32498]: rocksdb: kXpressCompression supported: 0 2024-03-31T23:01:20.473 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:20 smithi046 ceph-mon[32498]: rocksdb: kLZ4HCCompression supported: 1 2024-03-31T23:01:20.473 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:20 smithi046 ceph-mon[32498]: rocksdb: kZlibCompression supported: 1 2024-03-31T23:01:20.473 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:20 smithi046 ceph-mon[32498]: rocksdb: kSnappyCompression supported: 1 2024-03-31T23:01:20.473 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:20 smithi046 ceph-mon[32498]: rocksdb: kLZ4Compression supported: 1 2024-03-31T23:01:20.473 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:20 smithi046 ceph-mon[32498]: rocksdb: kBZip2Compression supported: 0 2024-03-31T23:01:20.473 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:20 smithi046 ceph-mon[32498]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-31T23:01:20.474 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:20 smithi046 ceph-mon[32498]: rocksdb: DMutex implementation: pthread_mutex_t 2024-03-31T23:01:20.474 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:20 smithi046 ceph-mon[32498]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2024-03-31T23:01:20.474 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:20 smithi046 ceph-mon[32498]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-03-31T23:01:20.474 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:20 smithi046 ceph-mon[32498]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-31T23:01:20.474 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:20 smithi046 ceph-mon[32498]: rocksdb: Options.merge_operator: 2024-03-31T23:01:20.474 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:20 smithi046 ceph-mon[32498]: rocksdb: Options.compaction_filter: None 2024-03-31T23:01:20.474 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:20 smithi046 ceph-mon[32498]: rocksdb: Options.compaction_filter_factory: None 2024-03-31T23:01:20.474 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:20 smithi046 ceph-mon[32498]: rocksdb: Options.sst_partitioner_factory: None 2024-03-31T23:01:20.474 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:20 smithi046 ceph-mon[32498]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-31T23:01:20.474 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:20 smithi046 ceph-mon[32498]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-31T23:01:20.475 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:20 smithi046 ceph-mon[32498]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x5650d357dbe0) 2024-03-31T23:01:20.475 INFO:journalctl@ceph.mon.a.smithi046.stdout: cache_index_and_filter_blocks: 1 2024-03-31T23:01:20.475 INFO:journalctl@ceph.mon.a.smithi046.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-31T23:01:20.475 INFO:journalctl@ceph.mon.a.smithi046.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-31T23:01:20.475 INFO:journalctl@ceph.mon.a.smithi046.stdout: pin_top_level_index_and_filter: 1 2024-03-31T23:01:20.475 INFO:journalctl@ceph.mon.a.smithi046.stdout: index_type: 0 2024-03-31T23:01:20.475 INFO:journalctl@ceph.mon.a.smithi046.stdout: data_block_index_type: 0 2024-03-31T23:01:20.475 INFO:journalctl@ceph.mon.a.smithi046.stdout: index_shortening: 1 2024-03-31T23:01:20.475 INFO:journalctl@ceph.mon.a.smithi046.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-31T23:01:20.476 INFO:journalctl@ceph.mon.a.smithi046.stdout: checksum: 4 2024-03-31T23:01:20.476 INFO:journalctl@ceph.mon.a.smithi046.stdout: no_block_cache: 0 2024-03-31T23:01:20.476 INFO:journalctl@ceph.mon.a.smithi046.stdout: block_cache: 0x5650d3581090 2024-03-31T23:01:20.476 INFO:journalctl@ceph.mon.a.smithi046.stdout: block_cache_name: BinnedLRUCache 2024-03-31T23:01:20.476 INFO:journalctl@ceph.mon.a.smithi046.stdout: block_cache_options: 2024-03-31T23:01:20.476 INFO:journalctl@ceph.mon.a.smithi046.stdout: capacity : 536870912 2024-03-31T23:01:20.476 INFO:journalctl@ceph.mon.a.smithi046.stdout: num_shard_bits : 4 2024-03-31T23:01:20.476 INFO:journalctl@ceph.mon.a.smithi046.stdout: strict_capacity_limit : 0 2024-03-31T23:01:20.477 INFO:journalctl@ceph.mon.a.smithi046.stdout: high_pri_pool_ratio: 0.000 2024-03-31T23:01:20.477 INFO:journalctl@ceph.mon.a.smithi046.stdout: block_cache_compressed: (nil) 2024-03-31T23:01:20.477 INFO:journalctl@ceph.mon.a.smithi046.stdout: persistent_cache: (nil) 2024-03-31T23:01:20.477 INFO:journalctl@ceph.mon.a.smithi046.stdout: block_size: 4096 2024-03-31T23:01:20.477 INFO:journalctl@ceph.mon.a.smithi046.stdout: block_size_deviation: 10 2024-03-31T23:01:20.477 INFO:journalctl@ceph.mon.a.smithi046.stdout: block_restart_interval: 16 2024-03-31T23:01:20.477 INFO:journalctl@ceph.mon.a.smithi046.stdout: index_block_restart_interval: 1 2024-03-31T23:01:20.477 INFO:journalctl@ceph.mon.a.smithi046.stdout: metadata_block_size: 4096 2024-03-31T23:01:20.478 INFO:journalctl@ceph.mon.a.smithi046.stdout: partition_filters: 0 2024-03-31T23:01:20.478 INFO:journalctl@ceph.mon.a.smithi046.stdout: use_delta_encoding: 1 2024-03-31T23:01:20.478 INFO:journalctl@ceph.mon.a.smithi046.stdout: filter_policy: bloomfilter 2024-03-31T23:01:20.478 INFO:journalctl@ceph.mon.a.smithi046.stdout: whole_key_filtering: 1 2024-03-31T23:01:20.478 INFO:journalctl@ceph.mon.a.smithi046.stdout: verify_compression: 0 2024-03-31T23:01:20.478 INFO:journalctl@ceph.mon.a.smithi046.stdout: read_amp_bytes_per_bit: 0 2024-03-31T23:01:20.478 INFO:journalctl@ceph.mon.a.smithi046.stdout: format_version: 5 2024-03-31T23:01:20.479 INFO:journalctl@ceph.mon.a.smithi046.stdout: enable_index_compression: 1 2024-03-31T23:01:20.479 INFO:journalctl@ceph.mon.a.smithi046.stdout: block_align: 0 2024-03-31T23:01:20.479 INFO:journalctl@ceph.mon.a.smithi046.stdout: max_auto_readahead_size: 262144 2024-03-31T23:01:20.479 INFO:journalctl@ceph.mon.a.smithi046.stdout: prepopulate_block_cache: 0 2024-03-31T23:01:20.479 INFO:journalctl@ceph.mon.a.smithi046.stdout: initial_auto_readahead_size: 8192 2024-03-31T23:01:20.479 INFO:journalctl@ceph.mon.a.smithi046.stdout: num_file_reads_for_auto_readahead: 2 2024-03-31T23:01:20.479 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:20 smithi046 ceph-mon[32498]: rocksdb: Options.write_buffer_size: 33554432 2024-03-31T23:01:20.480 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:20 smithi046 ceph-mon[32498]: rocksdb: Options.max_write_buffer_number: 2 2024-03-31T23:01:20.480 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:20 smithi046 ceph-mon[32498]: rocksdb: Options.compression: NoCompression 2024-03-31T23:01:20.480 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:20 smithi046 ceph-mon[32498]: rocksdb: Options.bottommost_compression: Disabled 2024-03-31T23:01:20.480 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:20 smithi046 ceph-mon[32498]: rocksdb: Options.prefix_extractor: nullptr 2024-03-31T23:01:20.480 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:20 smithi046 ceph-mon[32498]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-31T23:01:20.481 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:20 smithi046 ceph-mon[32498]: rocksdb: Options.num_levels: 7 2024-03-31T23:01:20.481 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:20 smithi046 ceph-mon[32498]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-31T23:01:20.481 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:20 smithi046 ceph-mon[32498]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-31T23:01:20.481 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:20 smithi046 ceph-mon[32498]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-31T23:01:20.481 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:20 smithi046 ceph-mon[32498]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-31T23:01:20.481 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:20 smithi046 ceph-mon[32498]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-31T23:01:20.481 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:20 smithi046 ceph-mon[32498]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-31T23:01:20.481 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:20 smithi046 ceph-mon[32498]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-31T23:01:20.482 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:20 smithi046 ceph-mon[32498]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-31T23:01:20.482 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:20 smithi046 ceph-mon[32498]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-31T23:01:20.482 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:20 smithi046 ceph-mon[32498]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-31T23:01:20.482 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:20 smithi046 ceph-mon[32498]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-03-31T23:01:20.482 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:20 smithi046 ceph-mon[32498]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-03-31T23:01:20.482 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:20 smithi046 ceph-mon[32498]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-31T23:01:20.482 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:20 smithi046 ceph-mon[32498]: rocksdb: Options.compression_opts.level: 32767 2024-03-31T23:01:20.482 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:20 smithi046 ceph-mon[32498]: rocksdb: Options.compression_opts.strategy: 0 2024-03-31T23:01:20.482 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:20 smithi046 ceph-mon[32498]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-31T23:01:20.483 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:20 smithi046 ceph-mon[32498]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-31T23:01:20.483 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:20 smithi046 ceph-mon[32498]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-03-31T23:01:20.483 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:20 smithi046 ceph-mon[32498]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-31T23:01:20.483 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:20 smithi046 ceph-mon[32498]: rocksdb: Options.compression_opts.enabled: false 2024-03-31T23:01:20.483 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:20 smithi046 ceph-mon[32498]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-03-31T23:01:20.483 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:20 smithi046 ceph-mon[32498]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-31T23:01:20.483 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:20 smithi046 ceph-mon[32498]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-31T23:01:20.483 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:20 smithi046 ceph-mon[32498]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-31T23:01:20.484 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:20 smithi046 ceph-mon[32498]: rocksdb: Options.target_file_size_base: 67108864 2024-03-31T23:01:20.484 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:20 smithi046 ceph-mon[32498]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-31T23:01:20.484 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:20 smithi046 ceph-mon[32498]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-31T23:01:20.484 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:20 smithi046 ceph-mon[32498]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-31T23:01:20.484 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:20 smithi046 ceph-mon[32498]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-31T23:01:20.484 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:20 smithi046 ceph-mon[32498]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-31T23:01:20.484 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:20 smithi046 ceph-mon[32498]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-31T23:01:20.484 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:20 smithi046 ceph-mon[32498]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-31T23:01:20.485 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:20 smithi046 ceph-mon[32498]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-31T23:01:20.485 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:20 smithi046 ceph-mon[32498]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-31T23:01:20.485 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:20 smithi046 ceph-mon[32498]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-31T23:01:20.485 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:20 smithi046 ceph-mon[32498]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-31T23:01:20.485 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:20 smithi046 ceph-mon[32498]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-31T23:01:20.485 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:20 smithi046 ceph-mon[32498]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-31T23:01:20.485 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:20 smithi046 ceph-mon[32498]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-03-31T23:01:20.485 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:20 smithi046 ceph-mon[32498]: rocksdb: Options.arena_block_size: 1048576 2024-03-31T23:01:20.485 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:20 smithi046 ceph-mon[32498]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-31T23:01:20.486 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:20 smithi046 ceph-mon[32498]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-31T23:01:20.486 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:20 smithi046 ceph-mon[32498]: rocksdb: Options.disable_auto_compactions: 0 2024-03-31T23:01:20.486 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:20 smithi046 ceph-mon[32498]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-31T23:01:20.486 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:20 smithi046 ceph-mon[32498]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-31T23:01:20.486 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:20 smithi046 ceph-mon[32498]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-31T23:01:20.486 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:20 smithi046 ceph-mon[32498]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-31T23:01:20.486 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:20 smithi046 ceph-mon[32498]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-31T23:01:20.486 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:20 smithi046 ceph-mon[32498]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-31T23:01:20.487 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:20 smithi046 ceph-mon[32498]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-31T23:01:20.487 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:20 smithi046 ceph-mon[32498]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-31T23:01:20.487 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:20 smithi046 ceph-mon[32498]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-31T23:01:20.487 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:20 smithi046 ceph-mon[32498]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-31T23:01:20.487 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:20 smithi046 ceph-mon[32498]: rocksdb: Options.table_properties_collectors: 2024-03-31T23:01:20.487 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:20 smithi046 ceph-mon[32498]: rocksdb: Options.inplace_update_support: 0 2024-03-31T23:01:20.487 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:20 smithi046 ceph-mon[32498]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-31T23:01:20.487 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:20 smithi046 ceph-mon[32498]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-31T23:01:20.488 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:20 smithi046 ceph-mon[32498]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-31T23:01:20.488 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:20 smithi046 ceph-mon[32498]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-31T23:01:20.488 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:20 smithi046 ceph-mon[32498]: rocksdb: Options.bloom_locality: 0 2024-03-31T23:01:20.488 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:20 smithi046 ceph-mon[32498]: rocksdb: Options.max_successive_merges: 0 2024-03-31T23:01:20.488 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:20 smithi046 ceph-mon[32498]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-31T23:01:20.488 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:20 smithi046 ceph-mon[32498]: rocksdb: Options.paranoid_file_checks: 0 2024-03-31T23:01:20.488 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:20 smithi046 ceph-mon[32498]: rocksdb: Options.force_consistency_checks: 1 2024-03-31T23:01:20.488 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:20 smithi046 ceph-mon[32498]: rocksdb: Options.report_bg_io_stats: 0 2024-03-31T23:01:20.488 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:20 smithi046 ceph-mon[32498]: rocksdb: Options.ttl: 2592000 2024-03-31T23:01:20.489 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:20 smithi046 ceph-mon[32498]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-31T23:01:20.489 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:20 smithi046 ceph-mon[32498]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-03-31T23:01:20.489 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:20 smithi046 ceph-mon[32498]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-03-31T23:01:20.489 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:20 smithi046 ceph-mon[32498]: rocksdb: Options.enable_blob_files: false 2024-03-31T23:01:20.489 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:20 smithi046 ceph-mon[32498]: rocksdb: Options.min_blob_size: 0 2024-03-31T23:01:20.489 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:20 smithi046 ceph-mon[32498]: rocksdb: Options.blob_file_size: 268435456 2024-03-31T23:01:20.489 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:20 smithi046 ceph-mon[32498]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-31T23:01:20.490 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:20 smithi046 ceph-mon[32498]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-31T23:01:20.490 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:20 smithi046 ceph-mon[32498]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-31T23:01:20.490 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:20 smithi046 ceph-mon[32498]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-03-31T23:01:20.490 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:20 smithi046 ceph-mon[32498]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-03-31T23:01:20.490 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:20 smithi046 ceph-mon[32498]: rocksdb: Options.blob_file_starting_level: 0 2024-03-31T23:01:20.490 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:20 smithi046 ceph-mon[32498]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-03-31T23:01:20.490 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:20 smithi046 ceph-mon[32498]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2024-03-31T23:01:20.490 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:20 smithi046 ceph-mon[32498]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2024-03-31T23:01:20.491 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:20 smithi046 ceph-mon[32498]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 63e09539-10eb-48ab-94fc-34c024591f3f 2024-03-31T23:01:20.491 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:20 smithi046 ceph-mon[32498]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711926080287468, "job": 1, "event": "recovery_started", "wal_files": [9]} 2024-03-31T23:01:20.491 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:20 smithi046 ceph-mon[32498]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2024-03-31T23:01:20.491 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:20 smithi046 ceph-mon[32498]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711926080288403, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 99393, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 324, "table_properties": {"data_size": 97313, "index_size": 298, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 773, "raw_key_size": 14032, "raw_average_key_size": 49, "raw_value_size": 89641, "raw_average_value_size": 316, "num_data_blocks": 12, "num_entries": 283, "num_filter_entries": 283, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1711926080, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "63e09539-10eb-48ab-94fc-34c024591f3f", "db_session_id": "I7UP2ZW29TPB6D34KA36", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2024-03-31T23:01:20.491 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:20 smithi046 ceph-mon[32498]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711926080288491, "job": 1, "event": "recovery_finished"} 2024-03-31T23:01:20.491 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:20 smithi046 ceph-mon[32498]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2024-03-31T23:01:20.491 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:20 smithi046 ceph-mon[32498]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-31T23:01:20.492 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:20 smithi046 ceph-mon[32498]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x5650d3672000 2024-03-31T23:01:20.492 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:20 smithi046 ceph-mon[32498]: rocksdb: DB pointer 0x5650d365c000 2024-03-31T23:01:20.492 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:20 smithi046 ceph-mon[32498]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-03-31T23:01:20.492 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:20 smithi046 ceph-mon[32498]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-03-31T23:01:20.492 INFO:journalctl@ceph.mon.a.smithi046.stdout: ** DB Stats ** 2024-03-31T23:01:20.492 INFO:journalctl@ceph.mon.a.smithi046.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-31T23:01:20.492 INFO:journalctl@ceph.mon.a.smithi046.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-31T23:01:20.492 INFO:journalctl@ceph.mon.a.smithi046.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-31T23:01:20.492 INFO:journalctl@ceph.mon.a.smithi046.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-31T23:01:20.492 INFO:journalctl@ceph.mon.a.smithi046.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-31T23:01:20.493 INFO:journalctl@ceph.mon.a.smithi046.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-31T23:01:20.493 INFO:journalctl@ceph.mon.a.smithi046.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-31T23:01:20.493 INFO:journalctl@ceph.mon.a.smithi046.stdout: 2024-03-31T23:01:20.493 INFO:journalctl@ceph.mon.a.smithi046.stdout: ** Compaction Stats [default] ** 2024-03-31T23:01:20.493 INFO:journalctl@ceph.mon.a.smithi046.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-31T23:01:20.493 INFO:journalctl@ceph.mon.a.smithi046.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-03-31T23:01:20.493 INFO:journalctl@ceph.mon.a.smithi046.stdout: L0 2/0 98.90 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 160.9 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-31T23:01:20.493 INFO:journalctl@ceph.mon.a.smithi046.stdout: Sum 2/0 98.90 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 160.9 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-31T23:01:20.493 INFO:journalctl@ceph.mon.a.smithi046.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 160.9 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-31T23:01:20.493 INFO:journalctl@ceph.mon.a.smithi046.stdout: 2024-03-31T23:01:20.493 INFO:journalctl@ceph.mon.a.smithi046.stdout: ** Compaction Stats [default] ** 2024-03-31T23:01:20.494 INFO:journalctl@ceph.mon.a.smithi046.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-31T23:01:20.494 INFO:journalctl@ceph.mon.a.smithi046.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-31T23:01:20.494 INFO:journalctl@ceph.mon.a.smithi046.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 160.9 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-31T23:01:20.494 INFO:journalctl@ceph.mon.a.smithi046.stdout: 2024-03-31T23:01:20.494 INFO:journalctl@ceph.mon.a.smithi046.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-03-31T23:01:20.494 INFO:journalctl@ceph.mon.a.smithi046.stdout: 2024-03-31T23:01:20.494 INFO:journalctl@ceph.mon.a.smithi046.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-31T23:01:20.495 INFO:journalctl@ceph.mon.a.smithi046.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-31T23:01:20.495 INFO:journalctl@ceph.mon.a.smithi046.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-31T23:01:20.495 INFO:journalctl@ceph.mon.a.smithi046.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-31T23:01:20.495 INFO:journalctl@ceph.mon.a.smithi046.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-31T23:01:20.495 INFO:journalctl@ceph.mon.a.smithi046.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-31T23:01:20.495 INFO:journalctl@ceph.mon.a.smithi046.stdout: Cumulative compaction: 0.00 GB write, 31.84 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-31T23:01:20.495 INFO:journalctl@ceph.mon.a.smithi046.stdout: Interval compaction: 0.00 GB write, 31.84 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-31T23:01:20.496 INFO:journalctl@ceph.mon.a.smithi046.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-31T23:01:20.496 INFO:journalctl@ceph.mon.a.smithi046.stdout: Block cache BinnedLRUCache@0x5650d3581090#2 capacity: 512.00 MB usage: 1.52 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 8e-06 secs_since: 0 2024-03-31T23:01:20.496 INFO:journalctl@ceph.mon.a.smithi046.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,1.02 KB,0.000193715%) IndexBlock(2,0.50 KB,9.53674e-05%) Misc(1,0.00 KB,0%) 2024-03-31T23:01:20.496 INFO:journalctl@ceph.mon.a.smithi046.stdout: 2024-03-31T23:01:20.496 INFO:journalctl@ceph.mon.a.smithi046.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-31T23:01:20.496 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:20 smithi046 ceph-mon[32498]: starting mon.a rank 0 at public addrs [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] at bind addrs [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid 6a9d8f08-efb2-11ee-b647-cb9ed24678a4 2024-03-31T23:01:20.496 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:20 smithi046 ceph-mon[32498]: mon.a@-1(???) e1 preinit fsid 6a9d8f08-efb2-11ee-b647-cb9ed24678a4 2024-03-31T23:01:20.496 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:20 smithi046 ceph-mon[32498]: mon.a@-1(???).mds e1 new map 2024-03-31T23:01:20.496 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:20 smithi046 ceph-mon[32498]: mon.a@-1(???).mds e1 print_map 2024-03-31T23:01:20.496 INFO:journalctl@ceph.mon.a.smithi046.stdout: e1 2024-03-31T23:01:20.497 INFO:journalctl@ceph.mon.a.smithi046.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-31T23:01:20.497 INFO:journalctl@ceph.mon.a.smithi046.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-03-31T23:01:20.497 INFO:journalctl@ceph.mon.a.smithi046.stdout: legacy client fscid: -1 2024-03-31T23:01:20.497 INFO:journalctl@ceph.mon.a.smithi046.stdout: 2024-03-31T23:01:20.497 INFO:journalctl@ceph.mon.a.smithi046.stdout: No filesystems configured 2024-03-31T23:01:20.497 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:20 smithi046 ceph-mon[32498]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-03-31T23:01:20.497 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:20 smithi046 ceph-mon[32498]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-31T23:01:20.497 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:20 smithi046 ceph-mon[32498]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-31T23:01:20.497 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:20 smithi046 ceph-mon[32498]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-31T23:01:20.497 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:20 smithi046 ceph-mon[32498]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-03-31T23:01:20.497 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:20 smithi046 bash[32479]: 857af8e9024c6c11705485611e9ddc07a77e0a6606eb97c45f49b8633751af71 2024-03-31T23:01:20.497 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:20 smithi046 systemd[1]: Started Ceph mon.a for 6a9d8f08-efb2-11ee-b647-cb9ed24678a4. 2024-03-31T23:01:20.497 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:20 smithi046 ceph-mon[32498]: mon.a is new leader, mons a in quorum (ranks 0) 2024-03-31T23:01:20.498 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:20 smithi046 ceph-mon[32498]: monmap epoch 1 2024-03-31T23:01:20.498 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:20 smithi046 ceph-mon[32498]: fsid 6a9d8f08-efb2-11ee-b647-cb9ed24678a4 2024-03-31T23:01:20.498 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:20 smithi046 ceph-mon[32498]: last_changed 2024-03-31T23:01:12.507350+0000 2024-03-31T23:01:20.498 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:20 smithi046 ceph-mon[32498]: created 2024-03-31T23:01:12.507350+0000 2024-03-31T23:01:20.498 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:20 smithi046 ceph-mon[32498]: min_mon_release 19 (squid) 2024-03-31T23:01:20.498 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:20 smithi046 ceph-mon[32498]: election_strategy: 1 2024-03-31T23:01:20.498 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:20 smithi046 ceph-mon[32498]: 0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-03-31T23:01:20.498 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:20 smithi046 ceph-mon[32498]: fsmap 2024-03-31T23:01:20.498 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:20 smithi046 ceph-mon[32498]: osdmap e1: 0 total, 0 up, 0 in 2024-03-31T23:01:20.498 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:20 smithi046 ceph-mon[32498]: mgrmap e1: no daemons active 2024-03-31T23:01:21.462 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:20.862+0000 7fc9da488700 1 Processor -- start 2024-03-31T23:01:21.462 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:20.862+0000 7fc9da488700 1 -- start start 2024-03-31T23:01:21.462 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:20.862+0000 7fc9da488700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc9d4107b00 0x7fc9d4107f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:01:21.462 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:20.862+0000 7fc9da488700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fc9d41084f0 con 0x7fc9d4107b00 2024-03-31T23:01:21.462 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:20.863+0000 7fc9d3fff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc9d4107b00 0x7fc9d4107f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:01:21.462 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:20.863+0000 7fc9d3fff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc9d4107b00 0x7fc9d4107f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:42662/0 (socket says 172.21.15.46:42662) 2024-03-31T23:01:21.462 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:20.863+0000 7fc9d3fff700 1 -- 172.21.15.46:0/3829079096 learned_addr learned my addr 172.21.15.46:0/3829079096 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:01:21.463 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:20.863+0000 7fc9d3fff700 1 -- 172.21.15.46:0/3829079096 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9d4108d00 con 0x7fc9d4107b00 2024-03-31T23:01:21.463 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:20.863+0000 7fc9d3fff700 1 --2- 172.21.15.46:0/3829079096 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc9d4107b00 0x7fc9d4107f20 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fc9c40098a0 tx=0x7fc9c4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3af261ff9236d21e server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:01:21.463 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:20.863+0000 7fc9d2ffd700 1 -- 172.21.15.46:0/3829079096 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc9c4004160 con 0x7fc9d4107b00 2024-03-31T23:01:21.463 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:20.863+0000 7fc9d2ffd700 1 -- 172.21.15.46:0/3829079096 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7fc9c4004300 con 0x7fc9d4107b00 2024-03-31T23:01:21.463 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:20.863+0000 7fc9d2ffd700 1 -- 172.21.15.46:0/3829079096 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc9c4011a80 con 0x7fc9d4107b00 2024-03-31T23:01:21.463 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:20.864+0000 7fc9da488700 1 -- 172.21.15.46:0/3829079096 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc9d4107b00 msgr2=0x7fc9d4107f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:01:21.463 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:20.864+0000 7fc9da488700 1 --2- 172.21.15.46:0/3829079096 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc9d4107b00 0x7fc9d4107f20 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fc9c40098a0 tx=0x7fc9c4009bb0 comp rx=0 tx=0).stop 2024-03-31T23:01:21.463 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:20.864+0000 7fc9da488700 1 -- 172.21.15.46:0/3829079096 shutdown_connections 2024-03-31T23:01:21.463 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:20.864+0000 7fc9da488700 1 --2- 172.21.15.46:0/3829079096 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc9d4107b00 0x7fc9d4107f20 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:01:21.463 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:20.864+0000 7fc9da488700 1 -- 172.21.15.46:0/3829079096 >> 172.21.15.46:0/3829079096 conn(0x7fc9d41030a0 msgr2=0x7fc9d41054e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:01:21.463 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:20.864+0000 7fc9da488700 1 -- 172.21.15.46:0/3829079096 shutdown_connections 2024-03-31T23:01:21.463 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:20.864+0000 7fc9da488700 1 -- 172.21.15.46:0/3829079096 wait complete. 2024-03-31T23:01:21.463 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:20.864+0000 7fc9da488700 1 Processor -- start 2024-03-31T23:01:21.463 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:20.864+0000 7fc9da488700 1 -- start start 2024-03-31T23:01:21.464 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:20.865+0000 7fc9da488700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc9d4107b00 0x7fc9d407db00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:01:21.464 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:20.865+0000 7fc9da488700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fc9d410bf30 con 0x7fc9d4107b00 2024-03-31T23:01:21.464 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:20.865+0000 7fc9d3fff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc9d4107b00 0x7fc9d407db00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:01:21.464 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:20.865+0000 7fc9d3fff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc9d4107b00 0x7fc9d407db00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:42674/0 (socket says 172.21.15.46:42674) 2024-03-31T23:01:21.464 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:20.865+0000 7fc9d3fff700 1 -- 172.21.15.46:0/2144270991 learned_addr learned my addr 172.21.15.46:0/2144270991 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:01:21.464 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:20.865+0000 7fc9d3fff700 1 -- 172.21.15.46:0/2144270991 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9d407e040 con 0x7fc9d4107b00 2024-03-31T23:01:21.464 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:20.865+0000 7fc9d3fff700 1 --2- 172.21.15.46:0/2144270991 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc9d4107b00 0x7fc9d407db00 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fc9c4000c00 tx=0x7fc9c4004290 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:01:21.464 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:20.865+0000 7fc9d17fa700 1 -- 172.21.15.46:0/2144270991 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc9c40039c0 con 0x7fc9d4107b00 2024-03-31T23:01:21.464 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:20.865+0000 7fc9d17fa700 1 -- 172.21.15.46:0/2144270991 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7fc9c4003b60 con 0x7fc9d4107b00 2024-03-31T23:01:21.464 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:20.865+0000 7fc9da488700 1 -- 172.21.15.46:0/2144270991 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc9d407c230 con 0x7fc9d4107b00 2024-03-31T23:01:21.464 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:20.865+0000 7fc9d17fa700 1 -- 172.21.15.46:0/2144270991 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc9c401a6f0 con 0x7fc9d4107b00 2024-03-31T23:01:21.464 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:20.865+0000 7fc9da488700 1 -- 172.21.15.46:0/2144270991 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc9d407c770 con 0x7fc9d4107b00 2024-03-31T23:01:21.464 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:20.866+0000 7fc9d17fa700 1 -- 172.21.15.46:0/2144270991 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fc9c401ac20 con 0x7fc9d4107b00 2024-03-31T23:01:21.464 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:20.866+0000 7fc9d17fa700 1 -- 172.21.15.46:0/2144270991 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fc9c4019700 con 0x7fc9d4107b00 2024-03-31T23:01:21.465 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:20.866+0000 7fc9da488700 1 -- 172.21.15.46:0/2144270991 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc9d4063870 con 0x7fc9d4107b00 2024-03-31T23:01:21.465 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:20.869+0000 7fc9d17fa700 1 -- 172.21.15.46:0/2144270991 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7fc9c4004300 con 0x7fc9d4107b00 2024-03-31T23:01:21.465 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:20.938+0000 7fc9da488700 1 -- 172.21.15.46:0/2144270991 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) -- 0x7fc9d407c9f0 con 0x7fc9d4107b00 2024-03-31T23:01:21.465 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:20.940+0000 7fc9d17fa700 1 -- 172.21.15.46:0/2144270991 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) ==== 124+0+0 (secure 0 0 0) 0x7fc9c4024a40 con 0x7fc9d4107b00 2024-03-31T23:01:21.465 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:20.941+0000 7fc9da488700 1 -- 172.21.15.46:0/2144270991 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc9d4107b00 msgr2=0x7fc9d407db00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:01:21.465 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:20.941+0000 7fc9da488700 1 --2- 172.21.15.46:0/2144270991 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc9d4107b00 0x7fc9d407db00 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fc9c4000c00 tx=0x7fc9c4004290 comp rx=0 tx=0).stop 2024-03-31T23:01:21.465 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:20.941+0000 7fc9da488700 1 -- 172.21.15.46:0/2144270991 shutdown_connections 2024-03-31T23:01:21.465 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:20.941+0000 7fc9da488700 1 --2- 172.21.15.46:0/2144270991 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc9d4107b00 0x7fc9d407db00 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:01:21.465 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:20.941+0000 7fc9da488700 1 -- 172.21.15.46:0/2144270991 >> 172.21.15.46:0/2144270991 conn(0x7fc9d41030a0 msgr2=0x7fc9d4103be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:01:21.465 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:20.941+0000 7fc9da488700 1 -- 172.21.15.46:0/2144270991 shutdown_connections 2024-03-31T23:01:21.465 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:20.941+0000 7fc9da488700 1 -- 172.21.15.46:0/2144270991 wait complete. 2024-03-31T23:01:21.465 INFO:teuthology.orchestra.run.smithi046.stdout:Wrote config to /etc/ceph/ceph.conf 2024-03-31T23:01:21.479 INFO:teuthology.orchestra.run.smithi046.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-03-31T23:01:21.479 INFO:teuthology.orchestra.run.smithi046.stdout:Creating mgr... 2024-03-31T23:01:21.479 INFO:teuthology.orchestra.run.smithi046.stdout:Verifying port 0.0.0.0:9283 ... 2024-03-31T23:01:21.480 INFO:teuthology.orchestra.run.smithi046.stdout:Verifying port 0.0.0.0:8765 ... 2024-03-31T23:01:21.728 INFO:teuthology.orchestra.run.smithi046.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4@mgr.x 2024-03-31T23:01:21.728 INFO:teuthology.orchestra.run.smithi046.stdout:systemctl: stderr Failed to reset failed state of unit ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4@mgr.x.service: Unit ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4@mgr.x.service not loaded. 2024-03-31T23:01:21.887 INFO:teuthology.orchestra.run.smithi046.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4.target.wants/ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4@mgr.x.service → /etc/systemd/system/ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4@.service. 2024-03-31T23:01:22.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:21 smithi046 ceph-mon[32498]: from='client.? 172.21.15.46:0/2144270991' entity='client.admin' 2024-03-31T23:01:22.384 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 31 23:01:22 smithi046 podman[32712]: 2024-03-31 23:01:22.311564939 +0000 UTC m=+0.310148177 container start 2adfecce4e56bfe409ec19699477f75777c8d2397f49aceac6ba5bd2e14c8c22 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:384f31f3d3aee7f8ba3f60846b8844c049165583, name=ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mgr-x, GIT_CLEAN=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-384f31f, ceph=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240326, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-03-31T23:01:22.384 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 31 23:01:22 smithi046 bash[32712]: 2adfecce4e56bfe409ec19699477f75777c8d2397f49aceac6ba5bd2e14c8c22 2024-03-31T23:01:22.397 INFO:teuthology.orchestra.run.smithi046.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-31T23:01:22.397 INFO:teuthology.orchestra.run.smithi046.stdout:systemctl: stdout disabled 2024-03-31T23:01:22.404 INFO:teuthology.orchestra.run.smithi046.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-31T23:01:22.404 INFO:teuthology.orchestra.run.smithi046.stdout:systemctl: stdout inactive 2024-03-31T23:01:22.404 INFO:teuthology.orchestra.run.smithi046.stdout:firewalld.service is not enabled 2024-03-31T23:01:22.404 INFO:teuthology.orchestra.run.smithi046.stdout:Not possible to enable service . firewalld.service is not available 2024-03-31T23:01:22.412 INFO:teuthology.orchestra.run.smithi046.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-31T23:01:22.412 INFO:teuthology.orchestra.run.smithi046.stdout:systemctl: stdout disabled 2024-03-31T23:01:22.420 INFO:teuthology.orchestra.run.smithi046.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-31T23:01:22.420 INFO:teuthology.orchestra.run.smithi046.stdout:systemctl: stdout inactive 2024-03-31T23:01:22.420 INFO:teuthology.orchestra.run.smithi046.stdout:firewalld.service is not enabled 2024-03-31T23:01:22.420 INFO:teuthology.orchestra.run.smithi046.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2024-03-31T23:01:22.420 INFO:teuthology.orchestra.run.smithi046.stdout:Waiting for mgr to start... 2024-03-31T23:01:22.420 INFO:teuthology.orchestra.run.smithi046.stdout:Waiting for mgr... 2024-03-31T23:01:22.645 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 31 23:01:22 smithi046 systemd[1]: Started Ceph mgr.x for 6a9d8f08-efb2-11ee-b647-cb9ed24678a4. 2024-03-31T23:01:22.914 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 31 23:01:22 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mgr-x[32727]: 2024-03-31T23:01:22.883+0000 7fb20e97e200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-31T23:01:23.361 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:22 smithi046 ceph-mon[32498]: from='client.? 172.21.15.46:0/1686124842' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-31T23:01:23.362 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 31 23:01:22 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mgr-x[32727]: 2024-03-31T23:01:22.984+0000 7fb20e97e200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-31T23:01:23.476 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout 2024-03-31T23:01:23.479 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout { 2024-03-31T23:01:23.479 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "fsid": "6a9d8f08-efb2-11ee-b647-cb9ed24678a4", 2024-03-31T23:01:23.479 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "health": { 2024-03-31T23:01:23.479 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-31T23:01:23.479 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-31T23:01:23.479 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-31T23:01:23.480 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout }, 2024-03-31T23:01:23.480 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-31T23:01:23.480 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-31T23:01:23.480 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout 0 2024-03-31T23:01:23.480 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout ], 2024-03-31T23:01:23.480 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-31T23:01:23.480 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "a" 2024-03-31T23:01:23.480 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout ], 2024-03-31T23:01:23.480 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "quorum_age": 2, 2024-03-31T23:01:23.480 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-31T23:01:23.480 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-31T23:01:23.480 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-31T23:01:23.480 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-31T23:01:23.480 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout }, 2024-03-31T23:01:23.480 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-31T23:01:23.481 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-31T23:01:23.481 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-31T23:01:23.481 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-31T23:01:23.481 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-31T23:01:23.489 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-31T23:01:23.489 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-31T23:01:23.489 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-31T23:01:23.489 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout }, 2024-03-31T23:01:23.489 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-31T23:01:23.489 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-31T23:01:23.489 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-31T23:01:23.489 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-31T23:01:23.489 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-31T23:01:23.489 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-31T23:01:23.490 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-31T23:01:23.490 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-31T23:01:23.490 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-31T23:01:23.490 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout }, 2024-03-31T23:01:23.490 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-31T23:01:23.490 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-31T23:01:23.490 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-31T23:01:23.490 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-31T23:01:23.490 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout }, 2024-03-31T23:01:23.490 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-31T23:01:23.490 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-31T23:01:23.490 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-31T23:01:23.490 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-31T23:01:23.491 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-31T23:01:23.491 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-31T23:01:23.491 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "restful" 2024-03-31T23:01:23.491 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout ], 2024-03-31T23:01:23.491 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-31T23:01:23.491 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout }, 2024-03-31T23:01:23.491 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-31T23:01:23.491 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-31T23:01:23.491 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "modified": "2024-03-31T23:01:15.464154+0000", 2024-03-31T23:01:23.491 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-31T23:01:23.491 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout }, 2024-03-31T23:01:23.491 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-31T23:01:23.491 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout } 2024-03-31T23:01:23.491 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:22.821+0000 7fdf1e44d700 1 Processor -- start 2024-03-31T23:01:23.492 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:22.821+0000 7fdf1e44d700 1 -- start start 2024-03-31T23:01:23.492 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:22.821+0000 7fdf1e44d700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdf18109190 0x7fdf181095b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:01:23.492 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:22.821+0000 7fdf1e44d700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fdf181076d0 con 0x7fdf18109190 2024-03-31T23:01:23.492 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:22.821+0000 7fdf17fff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdf18109190 0x7fdf181095b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:01:23.492 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:22.822+0000 7fdf17fff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdf18109190 0x7fdf181095b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:47234/0 (socket says 172.21.15.46:47234) 2024-03-31T23:01:23.492 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:22.822+0000 7fdf17fff700 1 -- 172.21.15.46:0/2706428522 learned_addr learned my addr 172.21.15.46:0/2706428522 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:01:23.492 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:22.822+0000 7fdf17fff700 1 -- 172.21.15.46:0/2706428522 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf18107850 con 0x7fdf18109190 2024-03-31T23:01:23.492 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:22.822+0000 7fdf17fff700 1 --2- 172.21.15.46:0/2706428522 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdf18109190 0x7fdf181095b0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fdf000098a0 tx=0x7fdf00009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8dbeebd02bddf710 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:01:23.492 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:22.822+0000 7fdf16ffd700 1 -- 172.21.15.46:0/2706428522 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fdf00004160 con 0x7fdf18109190 2024-03-31T23:01:23.492 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:22.822+0000 7fdf16ffd700 1 -- 172.21.15.46:0/2706428522 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7fdf00004300 con 0x7fdf18109190 2024-03-31T23:01:23.492 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:22.822+0000 7fdf16ffd700 1 -- 172.21.15.46:0/2706428522 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fdf00011a80 con 0x7fdf18109190 2024-03-31T23:01:23.492 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:22.823+0000 7fdf1e44d700 1 -- 172.21.15.46:0/2706428522 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdf18109190 msgr2=0x7fdf181095b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:01:23.492 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:22.823+0000 7fdf1e44d700 1 --2- 172.21.15.46:0/2706428522 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdf18109190 0x7fdf181095b0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fdf000098a0 tx=0x7fdf00009bb0 comp rx=0 tx=0).stop 2024-03-31T23:01:23.492 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:22.823+0000 7fdf1e44d700 1 -- 172.21.15.46:0/2706428522 shutdown_connections 2024-03-31T23:01:23.492 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:22.823+0000 7fdf1e44d700 1 --2- 172.21.15.46:0/2706428522 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdf18109190 0x7fdf181095b0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:01:23.493 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:22.823+0000 7fdf1e44d700 1 -- 172.21.15.46:0/2706428522 >> 172.21.15.46:0/2706428522 conn(0x7fdf18103070 msgr2=0x7fdf18105490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:01:23.493 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:22.823+0000 7fdf1e44d700 1 -- 172.21.15.46:0/2706428522 shutdown_connections 2024-03-31T23:01:23.493 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:22.823+0000 7fdf1e44d700 1 -- 172.21.15.46:0/2706428522 wait complete. 2024-03-31T23:01:23.493 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:22.823+0000 7fdf1e44d700 1 Processor -- start 2024-03-31T23:01:23.493 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:22.824+0000 7fdf1e44d700 1 -- start start 2024-03-31T23:01:23.493 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:22.824+0000 7fdf1e44d700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdf18109190 0x7fdf1819bb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:01:23.493 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:22.824+0000 7fdf1e44d700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fdf1810c060 con 0x7fdf18109190 2024-03-31T23:01:23.493 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:22.824+0000 7fdf17fff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdf18109190 0x7fdf1819bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:01:23.493 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:22.824+0000 7fdf17fff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdf18109190 0x7fdf1819bb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:47248/0 (socket says 172.21.15.46:47248) 2024-03-31T23:01:23.493 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:22.824+0000 7fdf17fff700 1 -- 172.21.15.46:0/1686124842 learned_addr learned my addr 172.21.15.46:0/1686124842 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:01:23.493 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:22.824+0000 7fdf17fff700 1 -- 172.21.15.46:0/1686124842 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf1819c0b0 con 0x7fdf18109190 2024-03-31T23:01:23.493 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:22.824+0000 7fdf17fff700 1 --2- 172.21.15.46:0/1686124842 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdf18109190 0x7fdf1819bb70 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fdf00006af0 tx=0x7fdf00011b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:01:23.493 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:22.825+0000 7fdf157fa700 1 -- 172.21.15.46:0/1686124842 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fdf00003930 con 0x7fdf18109190 2024-03-31T23:01:23.493 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:22.825+0000 7fdf1e44d700 1 -- 172.21.15.46:0/1686124842 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdf1819c340 con 0x7fdf18109190 2024-03-31T23:01:23.494 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:22.825+0000 7fdf157fa700 1 -- 172.21.15.46:0/1686124842 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7fdf00003ad0 con 0x7fdf18109190 2024-03-31T23:01:23.494 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:22.825+0000 7fdf157fa700 1 -- 172.21.15.46:0/1686124842 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fdf000195e0 con 0x7fdf18109190 2024-03-31T23:01:23.494 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:22.825+0000 7fdf1e44d700 1 -- 172.21.15.46:0/1686124842 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdf1819f030 con 0x7fdf18109190 2024-03-31T23:01:23.494 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:22.825+0000 7fdf157fa700 1 -- 172.21.15.46:0/1686124842 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fdf00019c90 con 0x7fdf18109190 2024-03-31T23:01:23.494 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:22.825+0000 7fdf157fa700 1 -- 172.21.15.46:0/1686124842 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fdf00024b60 con 0x7fdf18109190 2024-03-31T23:01:23.494 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:22.826+0000 7fdf1e44d700 1 -- 172.21.15.46:0/1686124842 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdf1810c1e0 con 0x7fdf18109190 2024-03-31T23:01:23.494 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:22.828+0000 7fdf157fa700 1 -- 172.21.15.46:0/1686124842 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7fdf00043dd0 con 0x7fdf18109190 2024-03-31T23:01:23.494 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:22.913+0000 7fdf1e44d700 1 -- 172.21.15.46:0/1686124842 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7fdf1819c820 con 0x7fdf18109190 2024-03-31T23:01:23.494 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:22.913+0000 7fdf157fa700 1 -- 172.21.15.46:0/1686124842 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7fdf1819c820 con 0x7fdf18109190 2024-03-31T23:01:23.494 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:22.914+0000 7fdf1e44d700 1 -- 172.21.15.46:0/1686124842 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdf18109190 msgr2=0x7fdf1819bb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:01:23.494 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:22.914+0000 7fdf1e44d700 1 --2- 172.21.15.46:0/1686124842 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdf18109190 0x7fdf1819bb70 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fdf00006af0 tx=0x7fdf00011b70 comp rx=0 tx=0).stop 2024-03-31T23:01:23.498 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:22.915+0000 7fdf1e44d700 1 -- 172.21.15.46:0/1686124842 shutdown_connections 2024-03-31T23:01:23.498 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:22.915+0000 7fdf1e44d700 1 --2- 172.21.15.46:0/1686124842 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdf18109190 0x7fdf1819bb70 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:01:23.498 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:22.915+0000 7fdf1e44d700 1 -- 172.21.15.46:0/1686124842 >> 172.21.15.46:0/1686124842 conn(0x7fdf18103070 msgr2=0x7fdf18103bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:01:23.498 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:22.915+0000 7fdf1e44d700 1 -- 172.21.15.46:0/1686124842 shutdown_connections 2024-03-31T23:01:23.498 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:22.915+0000 7fdf1e44d700 1 -- 172.21.15.46:0/1686124842 wait complete. 2024-03-31T23:01:23.498 INFO:teuthology.orchestra.run.smithi046.stdout:mgr not available, waiting (1/15)... 2024-03-31T23:01:23.749 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 31 23:01:23 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mgr-x[32727]: 2024-03-31T23:01:23.654+0000 7fb20e97e200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-31T23:01:23.749 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 31 23:01:23 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mgr-x[32727]: 2024-03-31T23:01:23.748+0000 7fb20e97e200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-31T23:01:24.074 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 31 23:01:23 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mgr-x[32727]: 2024-03-31T23:01:23.891+0000 7fb20e97e200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-31T23:01:24.074 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 31 23:01:23 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mgr-x[32727]: 2024-03-31T23:01:23.988+0000 7fb20e97e200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-31T23:01:24.074 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 31 23:01:24 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mgr-x[32727]: 2024-03-31T23:01:24.073+0000 7fb20e97e200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-31T23:01:24.611 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 31 23:01:24 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mgr-x[32727]: 2024-03-31T23:01:24.339+0000 7fb20e97e200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-31T23:01:24.612 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 31 23:01:24 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mgr-x[32727]: 2024-03-31T23:01:24.451+0000 7fb20e97e200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-31T23:01:24.612 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 31 23:01:24 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mgr-x[32727]: 2024-03-31T23:01:24.540+0000 7fb20e97e200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-31T23:01:25.111 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 31 23:01:24 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mgr-x[32727]: 2024-03-31T23:01:24.637+0000 7fb20e97e200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-31T23:01:25.111 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 31 23:01:24 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mgr-x[32727]: 2024-03-31T23:01:24.814+0000 7fb20e97e200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-31T23:01:26.000 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 31 23:01:25 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mgr-x[32727]: 2024-03-31T23:01:25.825+0000 7fb20e97e200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-31T23:01:26.000 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 31 23:01:25 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mgr-x[32727]: 2024-03-31T23:01:25.964+0000 7fb20e97e200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-31T23:01:26.313 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:26 smithi046 ceph-mon[32498]: from='client.? 172.21.15.46:0/3689079371' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-31T23:01:26.542 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout 2024-03-31T23:01:26.543 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout { 2024-03-31T23:01:26.543 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "fsid": "6a9d8f08-efb2-11ee-b647-cb9ed24678a4", 2024-03-31T23:01:26.543 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "health": { 2024-03-31T23:01:26.543 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-31T23:01:26.543 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-31T23:01:26.543 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-31T23:01:26.543 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout }, 2024-03-31T23:01:26.543 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-31T23:01:26.543 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-31T23:01:26.543 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout 0 2024-03-31T23:01:26.543 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout ], 2024-03-31T23:01:26.544 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-31T23:01:26.544 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "a" 2024-03-31T23:01:26.544 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout ], 2024-03-31T23:01:26.544 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "quorum_age": 5, 2024-03-31T23:01:26.544 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-31T23:01:26.544 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-31T23:01:26.544 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-31T23:01:26.544 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-31T23:01:26.544 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout }, 2024-03-31T23:01:26.544 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-31T23:01:26.544 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-31T23:01:26.544 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-31T23:01:26.544 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-31T23:01:26.545 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-31T23:01:26.545 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-31T23:01:26.545 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-31T23:01:26.545 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-31T23:01:26.545 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout }, 2024-03-31T23:01:26.545 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-31T23:01:26.545 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-31T23:01:26.545 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-31T23:01:26.545 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-31T23:01:26.545 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-31T23:01:26.545 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-31T23:01:26.545 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-31T23:01:26.545 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-31T23:01:26.546 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-31T23:01:26.546 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout }, 2024-03-31T23:01:26.546 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-31T23:01:26.546 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-31T23:01:26.546 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-31T23:01:26.546 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-31T23:01:26.546 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout }, 2024-03-31T23:01:26.546 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-31T23:01:26.546 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-31T23:01:26.546 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-31T23:01:26.546 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-31T23:01:26.546 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-31T23:01:26.546 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-31T23:01:26.546 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "restful" 2024-03-31T23:01:26.547 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout ], 2024-03-31T23:01:26.547 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-31T23:01:26.547 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout }, 2024-03-31T23:01:26.547 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-31T23:01:26.547 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-31T23:01:26.547 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "modified": "2024-03-31T23:01:15.464154+0000", 2024-03-31T23:01:26.547 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-31T23:01:26.547 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout }, 2024-03-31T23:01:26.547 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-31T23:01:26.547 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout } 2024-03-31T23:01:26.547 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:25.917+0000 7ff891f0c700 1 Processor -- start 2024-03-31T23:01:26.547 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:25.917+0000 7ff891f0c700 1 -- start start 2024-03-31T23:01:26.547 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:25.917+0000 7ff891f0c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff88c07de70 0x7ff88c07c2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:01:26.548 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:25.917+0000 7ff891f0c700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7ff88c07c8a0 con 0x7ff88c07de70 2024-03-31T23:01:26.548 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:25.917+0000 7ff88b7fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff88c07de70 0x7ff88c07c2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:01:26.548 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:25.917+0000 7ff88b7fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff88c07de70 0x7ff88c07c2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:47264/0 (socket says 172.21.15.46:47264) 2024-03-31T23:01:26.548 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:25.917+0000 7ff88b7fe700 1 -- 172.21.15.46:0/3067213691 learned_addr learned my addr 172.21.15.46:0/3067213691 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:01:26.548 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:25.918+0000 7ff88b7fe700 1 -- 172.21.15.46:0/3067213691 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff88c07ca20 con 0x7ff88c07de70 2024-03-31T23:01:26.548 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:25.918+0000 7ff88b7fe700 1 --2- 172.21.15.46:0/3067213691 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff88c07de70 0x7ff88c07c2d0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7ff8740098a0 tx=0x7ff874009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c12dd0b2490d437 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:01:26.548 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:25.918+0000 7ff88a7fc700 1 -- 172.21.15.46:0/3067213691 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff874004160 con 0x7ff88c07de70 2024-03-31T23:01:26.548 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:25.918+0000 7ff88a7fc700 1 -- 172.21.15.46:0/3067213691 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7ff874004300 con 0x7ff88c07de70 2024-03-31T23:01:26.548 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:25.918+0000 7ff88a7fc700 1 -- 172.21.15.46:0/3067213691 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff874011a80 con 0x7ff88c07de70 2024-03-31T23:01:26.548 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:25.919+0000 7ff891f0c700 1 -- 172.21.15.46:0/3067213691 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff88c07de70 msgr2=0x7ff88c07c2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:01:26.548 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:25.919+0000 7ff891f0c700 1 --2- 172.21.15.46:0/3067213691 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff88c07de70 0x7ff88c07c2d0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7ff8740098a0 tx=0x7ff874009bb0 comp rx=0 tx=0).stop 2024-03-31T23:01:26.548 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:25.919+0000 7ff891f0c700 1 -- 172.21.15.46:0/3067213691 shutdown_connections 2024-03-31T23:01:26.548 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:25.919+0000 7ff891f0c700 1 --2- 172.21.15.46:0/3067213691 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff88c07de70 0x7ff88c07c2d0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:01:26.549 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:25.919+0000 7ff891f0c700 1 -- 172.21.15.46:0/3067213691 >> 172.21.15.46:0/3067213691 conn(0x7ff88c103030 msgr2=0x7ff88c105490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:01:26.549 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:25.919+0000 7ff891f0c700 1 -- 172.21.15.46:0/3067213691 shutdown_connections 2024-03-31T23:01:26.549 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:25.919+0000 7ff891f0c700 1 -- 172.21.15.46:0/3067213691 wait complete. 2024-03-31T23:01:26.549 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:25.920+0000 7ff891f0c700 1 Processor -- start 2024-03-31T23:01:26.549 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:25.920+0000 7ff891f0c700 1 -- start start 2024-03-31T23:01:26.549 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:25.920+0000 7ff891f0c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff88c07de70 0x7ff88c19dc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:01:26.549 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:25.920+0000 7ff891f0c700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7ff88c10e070 con 0x7ff88c07de70 2024-03-31T23:01:26.549 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:25.920+0000 7ff88b7fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff88c07de70 0x7ff88c19dc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:01:26.549 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:25.920+0000 7ff88b7fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff88c07de70 0x7ff88c19dc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:47280/0 (socket says 172.21.15.46:47280) 2024-03-31T23:01:26.549 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:25.920+0000 7ff88b7fe700 1 -- 172.21.15.46:0/3689079371 learned_addr learned my addr 172.21.15.46:0/3689079371 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:01:26.549 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:25.920+0000 7ff88b7fe700 1 -- 172.21.15.46:0/3689079371 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff88c19e1b0 con 0x7ff88c07de70 2024-03-31T23:01:26.549 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:25.921+0000 7ff88b7fe700 1 --2- 172.21.15.46:0/3689079371 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff88c07de70 0x7ff88c19dc70 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7ff874006af0 tx=0x7ff874011b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:01:26.549 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:25.921+0000 7ff888ff9700 1 -- 172.21.15.46:0/3689079371 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff874003930 con 0x7ff88c07de70 2024-03-31T23:01:26.550 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:25.921+0000 7ff888ff9700 1 -- 172.21.15.46:0/3689079371 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7ff874003ad0 con 0x7ff88c07de70 2024-03-31T23:01:26.550 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:25.921+0000 7ff891f0c700 1 -- 172.21.15.46:0/3689079371 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff88c19e440 con 0x7ff88c07de70 2024-03-31T23:01:26.550 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:25.921+0000 7ff888ff9700 1 -- 172.21.15.46:0/3689079371 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff8740195e0 con 0x7ff88c07de70 2024-03-31T23:01:26.550 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:25.921+0000 7ff891f0c700 1 -- 172.21.15.46:0/3689079371 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff88c1a1130 con 0x7ff88c07de70 2024-03-31T23:01:26.550 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:25.921+0000 7ff888ff9700 1 -- 172.21.15.46:0/3689079371 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7ff874019c90 con 0x7ff88c07de70 2024-03-31T23:01:26.550 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:25.921+0000 7ff888ff9700 1 -- 172.21.15.46:0/3689079371 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7ff874024730 con 0x7ff88c07de70 2024-03-31T23:01:26.550 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:25.922+0000 7ff891f0c700 1 -- 172.21.15.46:0/3689079371 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff88c063870 con 0x7ff88c07de70 2024-03-31T23:01:26.550 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:25.924+0000 7ff888ff9700 1 -- 172.21.15.46:0/3689079371 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7ff874043dd0 con 0x7ff88c07de70 2024-03-31T23:01:26.550 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:25.999+0000 7ff891f0c700 1 -- 172.21.15.46:0/3689079371 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7ff88c1a16f0 con 0x7ff88c07de70 2024-03-31T23:01:26.550 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:25.999+0000 7ff888ff9700 1 -- 172.21.15.46:0/3689079371 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7ff874032020 con 0x7ff88c07de70 2024-03-31T23:01:26.550 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:26.000+0000 7ff891f0c700 1 -- 172.21.15.46:0/3689079371 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff88c07de70 msgr2=0x7ff88c19dc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:01:26.550 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:26.000+0000 7ff891f0c700 1 --2- 172.21.15.46:0/3689079371 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff88c07de70 0x7ff88c19dc70 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7ff874006af0 tx=0x7ff874011b70 comp rx=0 tx=0).stop 2024-03-31T23:01:26.551 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:26.000+0000 7ff891f0c700 1 -- 172.21.15.46:0/3689079371 shutdown_connections 2024-03-31T23:01:26.551 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:26.000+0000 7ff891f0c700 1 --2- 172.21.15.46:0/3689079371 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff88c07de70 0x7ff88c19dc70 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:01:26.551 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:26.000+0000 7ff891f0c700 1 -- 172.21.15.46:0/3689079371 >> 172.21.15.46:0/3689079371 conn(0x7ff88c103030 msgr2=0x7ff88c103bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:01:26.551 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:26.000+0000 7ff891f0c700 1 -- 172.21.15.46:0/3689079371 shutdown_connections 2024-03-31T23:01:26.551 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:26.000+0000 7ff891f0c700 1 -- 172.21.15.46:0/3689079371 wait complete. 2024-03-31T23:01:26.551 INFO:teuthology.orchestra.run.smithi046.stdout:mgr not available, waiting (2/15)... 2024-03-31T23:01:26.607 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 31 23:01:26 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mgr-x[32727]: 2024-03-31T23:01:26.312+0000 7fb20e97e200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-31T23:01:26.607 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 31 23:01:26 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mgr-x[32727]: 2024-03-31T23:01:26.403+0000 7fb20e97e200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-31T23:01:26.607 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 31 23:01:26 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mgr-x[32727]: 2024-03-31T23:01:26.607+0000 7fb20e97e200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-31T23:01:26.861 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 31 23:01:26 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mgr-x[32727]: 2024-03-31T23:01:26.798+0000 7fb20e97e200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-31T23:01:27.353 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 31 23:01:27 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mgr-x[32727]: 2024-03-31T23:01:27.091+0000 7fb20e97e200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-31T23:01:27.353 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 31 23:01:27 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mgr-x[32727]: 2024-03-31T23:01:27.353+0000 7fb20e97e200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-31T23:01:27.611 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 31 23:01:27 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mgr-x[32727]: 2024-03-31T23:01:27.595+0000 7fb20e97e200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-31T23:01:28.766 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 31 23:01:28 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mgr-x[32727]: 2024-03-31T23:01:28.424+0000 7fb20e97e200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-31T23:01:29.349 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:29 smithi046 ceph-mon[32498]: from='client.? 172.21.15.46:0/328959659' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-31T23:01:29.350 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 31 23:01:29 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mgr-x[32727]: 2024-03-31T23:01:29.022+0000 7fb20e97e200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-31T23:01:29.350 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 31 23:01:29 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mgr-x[32727]: 2024-03-31T23:01:29.125+0000 7fb20e97e200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-31T23:01:29.350 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 31 23:01:29 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mgr-x[32727]: 2024-03-31T23:01:29.349+0000 7fb20e97e200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-31T23:01:29.651 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout 2024-03-31T23:01:29.652 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout { 2024-03-31T23:01:29.652 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "fsid": "6a9d8f08-efb2-11ee-b647-cb9ed24678a4", 2024-03-31T23:01:29.652 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "health": { 2024-03-31T23:01:29.652 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-31T23:01:29.652 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-31T23:01:29.652 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-31T23:01:29.652 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout }, 2024-03-31T23:01:29.652 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-31T23:01:29.652 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-31T23:01:29.652 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout 0 2024-03-31T23:01:29.652 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout ], 2024-03-31T23:01:29.653 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-31T23:01:29.653 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "a" 2024-03-31T23:01:29.653 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout ], 2024-03-31T23:01:29.653 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "quorum_age": 8, 2024-03-31T23:01:29.653 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-31T23:01:29.653 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-31T23:01:29.653 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-31T23:01:29.653 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-31T23:01:29.653 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout }, 2024-03-31T23:01:29.653 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-31T23:01:29.653 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-31T23:01:29.653 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-31T23:01:29.653 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-31T23:01:29.654 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-31T23:01:29.654 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-31T23:01:29.654 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-31T23:01:29.654 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-31T23:01:29.654 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout }, 2024-03-31T23:01:29.654 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-31T23:01:29.654 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-31T23:01:29.654 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-31T23:01:29.654 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-31T23:01:29.654 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-31T23:01:29.654 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-31T23:01:29.654 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-31T23:01:29.655 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-31T23:01:29.655 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-31T23:01:29.655 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout }, 2024-03-31T23:01:29.655 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-31T23:01:29.655 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-31T23:01:29.655 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-31T23:01:29.655 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-31T23:01:29.655 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout }, 2024-03-31T23:01:29.655 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-31T23:01:29.655 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-31T23:01:29.655 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-31T23:01:29.655 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-31T23:01:29.655 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-31T23:01:29.655 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-31T23:01:29.656 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "restful" 2024-03-31T23:01:29.656 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout ], 2024-03-31T23:01:29.656 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-31T23:01:29.656 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout }, 2024-03-31T23:01:29.656 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-31T23:01:29.656 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-31T23:01:29.656 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "modified": "2024-03-31T23:01:15.464154+0000", 2024-03-31T23:01:29.656 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-31T23:01:29.656 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout }, 2024-03-31T23:01:29.656 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-31T23:01:29.656 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout } 2024-03-31T23:01:29.656 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:28.947+0000 7fe2a1a84700 1 Processor -- start 2024-03-31T23:01:29.657 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:28.948+0000 7fe2a1a84700 1 -- start start 2024-03-31T23:01:29.657 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:28.948+0000 7fe2a1a84700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe29c108550 0x7fe29c10a940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:01:29.657 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:28.948+0000 7fe2a1a84700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fe29c07be20 con 0x7fe29c108550 2024-03-31T23:01:29.657 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:28.948+0000 7fe29b7fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe29c108550 0x7fe29c10a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:01:29.657 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:28.948+0000 7fe29b7fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe29c108550 0x7fe29c10a940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:47292/0 (socket says 172.21.15.46:47292) 2024-03-31T23:01:29.657 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:28.948+0000 7fe29b7fe700 1 -- 172.21.15.46:0/311078498 learned_addr learned my addr 172.21.15.46:0/311078498 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:01:29.657 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:28.948+0000 7fe29b7fe700 1 -- 172.21.15.46:0/311078498 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe29c10ae80 con 0x7fe29c108550 2024-03-31T23:01:29.657 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:28.948+0000 7fe29b7fe700 1 --2- 172.21.15.46:0/311078498 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe29c108550 0x7fe29c10a940 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fe28c0098a0 tx=0x7fe28c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=985999eae3ac26bf server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:01:29.657 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:28.949+0000 7fe29a7fc700 1 -- 172.21.15.46:0/311078498 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe28c004160 con 0x7fe29c108550 2024-03-31T23:01:29.657 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:28.949+0000 7fe29a7fc700 1 -- 172.21.15.46:0/311078498 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7fe28c004300 con 0x7fe29c108550 2024-03-31T23:01:29.658 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:28.949+0000 7fe29a7fc700 1 -- 172.21.15.46:0/311078498 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe28c011a80 con 0x7fe29c108550 2024-03-31T23:01:29.658 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:28.949+0000 7fe2a1a84700 1 -- 172.21.15.46:0/311078498 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe29c108550 msgr2=0x7fe29c10a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:01:29.658 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:28.949+0000 7fe2a1a84700 1 --2- 172.21.15.46:0/311078498 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe29c108550 0x7fe29c10a940 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fe28c0098a0 tx=0x7fe28c009bb0 comp rx=0 tx=0).stop 2024-03-31T23:01:29.658 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:28.949+0000 7fe2a1a84700 1 -- 172.21.15.46:0/311078498 shutdown_connections 2024-03-31T23:01:29.658 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:28.949+0000 7fe2a1a84700 1 --2- 172.21.15.46:0/311078498 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe29c108550 0x7fe29c10a940 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:01:29.658 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:28.949+0000 7fe2a1a84700 1 -- 172.21.15.46:0/311078498 >> 172.21.15.46:0/311078498 conn(0x7fe29c101f80 msgr2=0x7fe29c1043a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:01:29.658 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:28.949+0000 7fe2a1a84700 1 -- 172.21.15.46:0/311078498 shutdown_connections 2024-03-31T23:01:29.658 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:28.949+0000 7fe2a1a84700 1 -- 172.21.15.46:0/311078498 wait complete. 2024-03-31T23:01:29.658 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:28.950+0000 7fe2a1a84700 1 Processor -- start 2024-03-31T23:01:29.659 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:28.950+0000 7fe2a1a84700 1 -- start start 2024-03-31T23:01:29.659 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:28.950+0000 7fe2a1a84700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe29c108550 0x7fe29c107dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:01:29.659 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:28.950+0000 7fe2a1a84700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fe29c10bdd0 con 0x7fe29c108550 2024-03-31T23:01:29.659 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:28.950+0000 7fe29b7fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe29c108550 0x7fe29c107dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:01:29.659 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:28.950+0000 7fe29b7fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe29c108550 0x7fe29c107dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:47306/0 (socket says 172.21.15.46:47306) 2024-03-31T23:01:29.659 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:28.950+0000 7fe29b7fe700 1 -- 172.21.15.46:0/328959659 learned_addr learned my addr 172.21.15.46:0/328959659 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:01:29.659 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:28.951+0000 7fe29b7fe700 1 -- 172.21.15.46:0/328959659 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe29c108310 con 0x7fe29c108550 2024-03-31T23:01:29.660 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:28.951+0000 7fe29b7fe700 1 --2- 172.21.15.46:0/328959659 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe29c108550 0x7fe29c107dd0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fe28c011a50 tx=0x7fe28c011b30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:01:29.660 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:28.951+0000 7fe298ff9700 1 -- 172.21.15.46:0/328959659 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe28c003a80 con 0x7fe29c108550 2024-03-31T23:01:29.660 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:28.951+0000 7fe298ff9700 1 -- 172.21.15.46:0/328959659 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7fe28c003c20 con 0x7fe29c108550 2024-03-31T23:01:29.660 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:28.951+0000 7fe2a1a84700 1 -- 172.21.15.46:0/328959659 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe29c106500 con 0x7fe29c108550 2024-03-31T23:01:29.660 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:28.951+0000 7fe298ff9700 1 -- 172.21.15.46:0/328959659 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe28c01a650 con 0x7fe29c108550 2024-03-31T23:01:29.660 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:28.951+0000 7fe2a1a84700 1 -- 172.21.15.46:0/328959659 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe29c106a40 con 0x7fe29c108550 2024-03-31T23:01:29.661 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:28.951+0000 7fe298ff9700 1 -- 172.21.15.46:0/328959659 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fe28c01acb0 con 0x7fe29c108550 2024-03-31T23:01:29.661 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:28.951+0000 7fe298ff9700 1 -- 172.21.15.46:0/328959659 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fe28c019700 con 0x7fe29c108550 2024-03-31T23:01:29.661 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:28.952+0000 7fe2a1a84700 1 -- 172.21.15.46:0/328959659 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe29c063870 con 0x7fe29c108550 2024-03-31T23:01:29.661 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:28.956+0000 7fe298ff9700 1 -- 172.21.15.46:0/328959659 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7fe28c043dd0 con 0x7fe29c108550 2024-03-31T23:01:29.661 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:29.031+0000 7fe2a1a84700 1 -- 172.21.15.46:0/328959659 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7fe29c106da0 con 0x7fe29c108550 2024-03-31T23:01:29.661 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:29.031+0000 7fe298ff9700 1 -- 172.21.15.46:0/328959659 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7fe28c032070 con 0x7fe29c108550 2024-03-31T23:01:29.661 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:29.033+0000 7fe2a1a84700 1 -- 172.21.15.46:0/328959659 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe29c108550 msgr2=0x7fe29c107dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:01:29.661 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:29.033+0000 7fe2a1a84700 1 --2- 172.21.15.46:0/328959659 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe29c108550 0x7fe29c107dd0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fe28c011a50 tx=0x7fe28c011b30 comp rx=0 tx=0).stop 2024-03-31T23:01:29.661 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:29.033+0000 7fe2a1a84700 1 -- 172.21.15.46:0/328959659 shutdown_connections 2024-03-31T23:01:29.662 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:29.033+0000 7fe2a1a84700 1 --2- 172.21.15.46:0/328959659 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe29c108550 0x7fe29c107dd0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:01:29.662 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:29.033+0000 7fe2a1a84700 1 -- 172.21.15.46:0/328959659 >> 172.21.15.46:0/328959659 conn(0x7fe29c101f80 msgr2=0x7fe29c102a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:01:29.662 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:29.033+0000 7fe2a1a84700 1 -- 172.21.15.46:0/328959659 shutdown_connections 2024-03-31T23:01:29.662 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:29.033+0000 7fe2a1a84700 1 -- 172.21.15.46:0/328959659 wait complete. 2024-03-31T23:01:29.662 INFO:teuthology.orchestra.run.smithi046.stdout:mgr not available, waiting (3/15)... 2024-03-31T23:01:30.361 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:30 smithi046 ceph-mon[32498]: Activating manager daemon x 2024-03-31T23:01:30.362 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:30 smithi046 ceph-mon[32498]: mgrmap e2: x(active, starting, since 0.0030509s) 2024-03-31T23:01:30.362 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:30 smithi046 ceph-mon[32498]: from='mgr.14100 172.21.15.46:0/1725117308' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-31T23:01:30.362 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:30 smithi046 ceph-mon[32498]: from='mgr.14100 172.21.15.46:0/1725117308' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-31T23:01:30.362 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:30 smithi046 ceph-mon[32498]: from='mgr.14100 172.21.15.46:0/1725117308' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-31T23:01:30.362 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:30 smithi046 ceph-mon[32498]: from='mgr.14100 172.21.15.46:0/1725117308' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-31T23:01:30.362 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:30 smithi046 ceph-mon[32498]: from='mgr.14100 172.21.15.46:0/1725117308' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-31T23:01:30.362 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:30 smithi046 ceph-mon[32498]: Manager daemon x is now available 2024-03-31T23:01:30.363 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:30 smithi046 ceph-mon[32498]: from='mgr.14100 172.21.15.46:0/1725117308' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-31T23:01:30.363 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:30 smithi046 ceph-mon[32498]: from='mgr.14100 172.21.15.46:0/1725117308' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-31T23:01:30.363 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:30 smithi046 ceph-mon[32498]: from='mgr.14100 172.21.15.46:0/1725117308' entity='mgr.x' 2024-03-31T23:01:30.363 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:30 smithi046 ceph-mon[32498]: from='mgr.14100 172.21.15.46:0/1725117308' entity='mgr.x' 2024-03-31T23:01:30.363 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:30 smithi046 ceph-mon[32498]: from='mgr.14100 172.21.15.46:0/1725117308' entity='mgr.x' 2024-03-31T23:01:31.611 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:31 smithi046 ceph-mon[32498]: mgrmap e3: x(active, since 1.00682s) 2024-03-31T23:01:32.612 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:32 smithi046 ceph-mon[32498]: mgrmap e4: x(active, since 2s) 2024-03-31T23:01:32.612 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:32 smithi046 ceph-mon[32498]: from='client.? 172.21.15.46:0/1433868739' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-31T23:01:32.998 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout 2024-03-31T23:01:32.998 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout { 2024-03-31T23:01:32.998 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "fsid": "6a9d8f08-efb2-11ee-b647-cb9ed24678a4", 2024-03-31T23:01:32.998 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "health": { 2024-03-31T23:01:32.998 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-31T23:01:32.998 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-31T23:01:32.998 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-31T23:01:32.998 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout }, 2024-03-31T23:01:32.998 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-31T23:01:32.998 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-31T23:01:32.998 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout 0 2024-03-31T23:01:32.999 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout ], 2024-03-31T23:01:32.999 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-31T23:01:32.999 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "a" 2024-03-31T23:01:32.999 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout ], 2024-03-31T23:01:32.999 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "quorum_age": 12, 2024-03-31T23:01:32.999 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-31T23:01:32.999 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-31T23:01:32.999 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-31T23:01:32.999 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-31T23:01:32.999 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout }, 2024-03-31T23:01:32.999 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-31T23:01:32.999 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-31T23:01:32.999 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-31T23:01:32.999 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-31T23:01:32.999 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-31T23:01:33.009 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-31T23:01:33.009 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-31T23:01:33.009 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-31T23:01:33.009 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout }, 2024-03-31T23:01:33.009 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-31T23:01:33.010 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-31T23:01:33.010 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-31T23:01:33.010 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-31T23:01:33.010 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-31T23:01:33.010 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-31T23:01:33.010 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-31T23:01:33.010 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-31T23:01:33.010 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-31T23:01:33.010 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout }, 2024-03-31T23:01:33.010 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-31T23:01:33.010 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-31T23:01:33.010 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-31T23:01:33.010 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-31T23:01:33.011 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout }, 2024-03-31T23:01:33.011 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-31T23:01:33.011 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-31T23:01:33.011 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-31T23:01:33.011 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-31T23:01:33.011 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-31T23:01:33.011 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-31T23:01:33.011 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "restful" 2024-03-31T23:01:33.011 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout ], 2024-03-31T23:01:33.011 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-31T23:01:33.012 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout }, 2024-03-31T23:01:33.012 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-31T23:01:33.012 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-31T23:01:33.012 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "modified": "2024-03-31T23:01:15.464154+0000", 2024-03-31T23:01:33.012 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-31T23:01:33.012 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout }, 2024-03-31T23:01:33.012 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-31T23:01:33.012 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout } 2024-03-31T23:01:33.012 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:32.029+0000 7f8e5b96d700 1 Processor -- start 2024-03-31T23:01:33.012 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:32.029+0000 7f8e5b96d700 1 -- start start 2024-03-31T23:01:33.012 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:32.029+0000 7f8e5b96d700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8e54108530 0x7f8e5410a920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:01:33.012 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:32.029+0000 7f8e5b96d700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f8e5407be40 con 0x7f8e54108530 2024-03-31T23:01:33.012 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:32.029+0000 7f8e59709700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8e54108530 0x7f8e5410a920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:01:33.012 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:32.029+0000 7f8e59709700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8e54108530 0x7f8e5410a920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60696/0 (socket says 172.21.15.46:60696) 2024-03-31T23:01:33.013 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:32.029+0000 7f8e59709700 1 -- 172.21.15.46:0/1850088558 learned_addr learned my addr 172.21.15.46:0/1850088558 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:01:33.013 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:32.030+0000 7f8e59709700 1 -- 172.21.15.46:0/1850088558 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8e5410ae60 con 0x7f8e54108530 2024-03-31T23:01:33.013 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:32.030+0000 7f8e59709700 1 --2- 172.21.15.46:0/1850088558 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8e54108530 0x7f8e5410a920 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f8e500098a0 tx=0x7f8e50009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ec64825bc3d62bd server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:01:33.013 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:32.030+0000 7f8e4bfff700 1 -- 172.21.15.46:0/1850088558 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8e50004160 con 0x7f8e54108530 2024-03-31T23:01:33.013 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:32.030+0000 7f8e4bfff700 1 -- 172.21.15.46:0/1850088558 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7f8e50004300 con 0x7f8e54108530 2024-03-31T23:01:33.013 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:32.030+0000 7f8e4bfff700 1 -- 172.21.15.46:0/1850088558 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8e50011a80 con 0x7f8e54108530 2024-03-31T23:01:33.013 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:32.031+0000 7f8e5b96d700 1 -- 172.21.15.46:0/1850088558 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8e54108530 msgr2=0x7f8e5410a920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:01:33.013 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:32.031+0000 7f8e5b96d700 1 --2- 172.21.15.46:0/1850088558 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8e54108530 0x7f8e5410a920 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f8e500098a0 tx=0x7f8e50009bb0 comp rx=0 tx=0).stop 2024-03-31T23:01:33.013 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:32.031+0000 7f8e5b96d700 1 -- 172.21.15.46:0/1850088558 shutdown_connections 2024-03-31T23:01:33.013 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:32.031+0000 7f8e5b96d700 1 --2- 172.21.15.46:0/1850088558 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8e54108530 0x7f8e5410a920 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:01:33.013 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:32.031+0000 7f8e5b96d700 1 -- 172.21.15.46:0/1850088558 >> 172.21.15.46:0/1850088558 conn(0x7f8e54101f40 msgr2=0x7f8e541043a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:01:33.013 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:32.031+0000 7f8e5b96d700 1 -- 172.21.15.46:0/1850088558 shutdown_connections 2024-03-31T23:01:33.013 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:32.031+0000 7f8e5b96d700 1 -- 172.21.15.46:0/1850088558 wait complete. 2024-03-31T23:01:33.013 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:32.031+0000 7f8e5b96d700 1 Processor -- start 2024-03-31T23:01:33.014 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:32.031+0000 7f8e5b96d700 1 -- start start 2024-03-31T23:01:33.014 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:32.032+0000 7f8e5b96d700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8e54108530 0x7f8e54107db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:01:33.014 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:32.032+0000 7f8e5b96d700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f8e5410be30 con 0x7f8e54108530 2024-03-31T23:01:33.014 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:32.032+0000 7f8e59709700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8e54108530 0x7f8e54107db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:01:33.014 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:32.032+0000 7f8e59709700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8e54108530 0x7f8e54107db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60700/0 (socket says 172.21.15.46:60700) 2024-03-31T23:01:33.014 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:32.032+0000 7f8e59709700 1 -- 172.21.15.46:0/1433868739 learned_addr learned my addr 172.21.15.46:0/1433868739 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:01:33.014 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:32.032+0000 7f8e59709700 1 -- 172.21.15.46:0/1433868739 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8e541082f0 con 0x7f8e54108530 2024-03-31T23:01:33.014 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:32.032+0000 7f8e59709700 1 --2- 172.21.15.46:0/1433868739 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8e54108530 0x7f8e54107db0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f8e50006af0 tx=0x7f8e50011b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:01:33.014 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:32.032+0000 7f8e4a7fc700 1 -- 172.21.15.46:0/1433868739 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8e50003930 con 0x7f8e54108530 2024-03-31T23:01:33.014 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:32.032+0000 7f8e4a7fc700 1 -- 172.21.15.46:0/1433868739 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7f8e50003ad0 con 0x7f8e54108530 2024-03-31T23:01:33.014 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:32.032+0000 7f8e5b96d700 1 -- 172.21.15.46:0/1433868739 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8e541064e0 con 0x7f8e54108530 2024-03-31T23:01:33.014 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:32.032+0000 7f8e4a7fc700 1 -- 172.21.15.46:0/1433868739 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8e500195e0 con 0x7f8e54108530 2024-03-31T23:01:33.014 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:32.032+0000 7f8e5b96d700 1 -- 172.21.15.46:0/1433868739 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8e54106a20 con 0x7f8e54108530 2024-03-31T23:01:33.015 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:32.033+0000 7f8e4a7fc700 1 -- 172.21.15.46:0/1433868739 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 4) ==== 46105+0+0 (secure 0 0 0) 0x7f8e500198e0 con 0x7f8e54108530 2024-03-31T23:01:33.015 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:32.033+0000 7f8e4a7fc700 1 --2- 172.21.15.46:0/1433868739 >> [v2:172.21.15.46:6800/1515528215,v1:172.21.15.46:6801/1515528215] conn(0x7f8e40038d60 0x7f8e4003b220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:01:33.015 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:32.033+0000 7f8e4a7fc700 1 -- 172.21.15.46:0/1433868739 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f8e5004d150 con 0x7f8e54108530 2024-03-31T23:01:33.015 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:32.034+0000 7f8e58f08700 1 --2- 172.21.15.46:0/1433868739 >> [v2:172.21.15.46:6800/1515528215,v1:172.21.15.46:6801/1515528215] conn(0x7f8e40038d60 0x7f8e4003b220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:01:33.015 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:32.034+0000 7f8e5b96d700 1 -- 172.21.15.46:0/1433868739 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8e54063870 con 0x7f8e54108530 2024-03-31T23:01:33.015 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:32.034+0000 7f8e58f08700 1 --2- 172.21.15.46:0/1433868739 >> [v2:172.21.15.46:6800/1515528215,v1:172.21.15.46:6801/1515528215] conn(0x7f8e40038d60 0x7f8e4003b220 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f8e44006fd0 tx=0x7f8e44006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:01:33.015 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:32.039+0000 7f8e4a7fc700 1 -- 172.21.15.46:0/1433868739 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f8e50020240 con 0x7f8e54108530 2024-03-31T23:01:33.015 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:32.325+0000 7f8e5b96d700 1 -- 172.21.15.46:0/1433868739 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f8e541070d0 con 0x7f8e54108530 2024-03-31T23:01:33.015 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:32.325+0000 7f8e4a7fc700 1 -- 172.21.15.46:0/1433868739 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7f8e50029d50 con 0x7f8e54108530 2024-03-31T23:01:33.015 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:32.327+0000 7f8e5b96d700 1 -- 172.21.15.46:0/1433868739 >> [v2:172.21.15.46:6800/1515528215,v1:172.21.15.46:6801/1515528215] conn(0x7f8e40038d60 msgr2=0x7f8e4003b220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:01:33.015 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:32.327+0000 7f8e5b96d700 1 --2- 172.21.15.46:0/1433868739 >> [v2:172.21.15.46:6800/1515528215,v1:172.21.15.46:6801/1515528215] conn(0x7f8e40038d60 0x7f8e4003b220 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f8e44006fd0 tx=0x7f8e44006e40 comp rx=0 tx=0).stop 2024-03-31T23:01:33.015 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:32.327+0000 7f8e5b96d700 1 -- 172.21.15.46:0/1433868739 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8e54108530 msgr2=0x7f8e54107db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:01:33.015 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:32.327+0000 7f8e5b96d700 1 --2- 172.21.15.46:0/1433868739 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8e54108530 0x7f8e54107db0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f8e50006af0 tx=0x7f8e50011b70 comp rx=0 tx=0).stop 2024-03-31T23:01:33.015 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:32.327+0000 7f8e5b96d700 1 -- 172.21.15.46:0/1433868739 shutdown_connections 2024-03-31T23:01:33.016 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:32.328+0000 7f8e5b96d700 1 --2- 172.21.15.46:0/1433868739 >> [v2:172.21.15.46:6800/1515528215,v1:172.21.15.46:6801/1515528215] conn(0x7f8e40038d60 0x7f8e4003b220 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:01:33.016 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:32.328+0000 7f8e5b96d700 1 --2- 172.21.15.46:0/1433868739 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8e54108530 0x7f8e54107db0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:01:33.016 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:32.328+0000 7f8e5b96d700 1 -- 172.21.15.46:0/1433868739 >> 172.21.15.46:0/1433868739 conn(0x7f8e54101f40 msgr2=0x7f8e54102aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:01:33.016 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:32.328+0000 7f8e5b96d700 1 -- 172.21.15.46:0/1433868739 shutdown_connections 2024-03-31T23:01:33.016 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:32.328+0000 7f8e5b96d700 1 -- 172.21.15.46:0/1433868739 wait complete. 2024-03-31T23:01:33.016 INFO:teuthology.orchestra.run.smithi046.stdout:mgr is available 2024-03-31T23:01:33.907 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:33 smithi046 ceph-mon[32498]: from='client.? 172.21.15.46:0/3581817707' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-03-31T23:01:34.176 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout 2024-03-31T23:01:34.177 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout [global] 2024-03-31T23:01:34.177 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout fsid = 6a9d8f08-efb2-11ee-b647-cb9ed24678a4 2024-03-31T23:01:34.177 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-03-31T23:01:34.177 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-03-31T23:01:34.177 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-03-31T23:01:34.177 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-03-31T23:01:34.177 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout 2024-03-31T23:01:34.177 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout [mgr] 2024-03-31T23:01:34.177 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-03-31T23:01:34.177 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout 2024-03-31T23:01:34.177 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout [osd] 2024-03-31T23:01:34.178 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-03-31T23:01:34.178 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-03-31T23:01:34.178 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-03-31T23:01:34.178 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-03-31T23:01:34.178 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-03-31T23:01:34.178 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:33.401+0000 7f0d8972f700 1 Processor -- start 2024-03-31T23:01:34.178 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:33.402+0000 7f0d8972f700 1 -- start start 2024-03-31T23:01:34.178 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:33.402+0000 7f0d8972f700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0d8407de20 0x7f0d8407c2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:01:34.178 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:33.402+0000 7f0d8972f700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f0d8407c8a0 con 0x7f0d8407de20 2024-03-31T23:01:34.178 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:33.402+0000 7f0d82ffd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0d8407de20 0x7f0d8407c2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:01:34.178 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:33.402+0000 7f0d82ffd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0d8407de20 0x7f0d8407c2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60706/0 (socket says 172.21.15.46:60706) 2024-03-31T23:01:34.179 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:33.402+0000 7f0d82ffd700 1 -- 172.21.15.46:0/1604010731 learned_addr learned my addr 172.21.15.46:0/1604010731 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:01:34.179 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:33.402+0000 7f0d82ffd700 1 -- 172.21.15.46:0/1604010731 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d8407ca20 con 0x7f0d8407de20 2024-03-31T23:01:34.179 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:33.402+0000 7f0d82ffd700 1 --2- 172.21.15.46:0/1604010731 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0d8407de20 0x7f0d8407c2d0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f0d740098a0 tx=0x7f0d74009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=51874447490bad65 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:01:34.179 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:33.402+0000 7f0d81ffb700 1 -- 172.21.15.46:0/1604010731 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0d74004160 con 0x7f0d8407de20 2024-03-31T23:01:34.179 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:33.402+0000 7f0d81ffb700 1 -- 172.21.15.46:0/1604010731 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7f0d74004300 con 0x7f0d8407de20 2024-03-31T23:01:34.179 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:33.403+0000 7f0d81ffb700 1 -- 172.21.15.46:0/1604010731 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0d74011a80 con 0x7f0d8407de20 2024-03-31T23:01:34.179 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:33.403+0000 7f0d8972f700 1 -- 172.21.15.46:0/1604010731 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0d8407de20 msgr2=0x7f0d8407c2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:01:34.179 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:33.403+0000 7f0d8972f700 1 --2- 172.21.15.46:0/1604010731 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0d8407de20 0x7f0d8407c2d0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f0d740098a0 tx=0x7f0d74009bb0 comp rx=0 tx=0).stop 2024-03-31T23:01:34.179 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:33.403+0000 7f0d8972f700 1 -- 172.21.15.46:0/1604010731 shutdown_connections 2024-03-31T23:01:34.179 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:33.403+0000 7f0d8972f700 1 --2- 172.21.15.46:0/1604010731 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0d8407de20 0x7f0d8407c2d0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:01:34.179 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:33.403+0000 7f0d8972f700 1 -- 172.21.15.46:0/1604010731 >> 172.21.15.46:0/1604010731 conn(0x7f0d84103100 msgr2=0x7f0d84105540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:01:34.179 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:33.403+0000 7f0d8972f700 1 -- 172.21.15.46:0/1604010731 shutdown_connections 2024-03-31T23:01:34.180 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:33.403+0000 7f0d8972f700 1 -- 172.21.15.46:0/1604010731 wait complete. 2024-03-31T23:01:34.180 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:33.404+0000 7f0d8972f700 1 Processor -- start 2024-03-31T23:01:34.180 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:33.404+0000 7f0d8972f700 1 -- start start 2024-03-31T23:01:34.180 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:33.404+0000 7f0d8972f700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0d8407de20 0x7f0d8419dc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:01:34.180 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:33.404+0000 7f0d8972f700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f0d8410e070 con 0x7f0d8407de20 2024-03-31T23:01:34.180 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:33.404+0000 7f0d82ffd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0d8407de20 0x7f0d8419dc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:01:34.180 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:33.404+0000 7f0d82ffd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0d8407de20 0x7f0d8419dc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60722/0 (socket says 172.21.15.46:60722) 2024-03-31T23:01:34.180 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:33.404+0000 7f0d82ffd700 1 -- 172.21.15.46:0/3581817707 learned_addr learned my addr 172.21.15.46:0/3581817707 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:01:34.180 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:33.404+0000 7f0d82ffd700 1 -- 172.21.15.46:0/3581817707 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d8419e1a0 con 0x7f0d8407de20 2024-03-31T23:01:34.180 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:33.405+0000 7f0d82ffd700 1 --2- 172.21.15.46:0/3581817707 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0d8407de20 0x7f0d8419dc60 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f0d74011a50 tx=0x7f0d74004290 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:01:34.180 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:33.405+0000 7f0d7bfff700 1 -- 172.21.15.46:0/3581817707 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0d740039f0 con 0x7f0d8407de20 2024-03-31T23:01:34.180 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:33.405+0000 7f0d7bfff700 1 -- 172.21.15.46:0/3581817707 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7f0d74003b90 con 0x7f0d8407de20 2024-03-31T23:01:34.181 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:33.405+0000 7f0d8972f700 1 -- 172.21.15.46:0/3581817707 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0d8419e430 con 0x7f0d8407de20 2024-03-31T23:01:34.181 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:33.405+0000 7f0d7bfff700 1 -- 172.21.15.46:0/3581817707 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0d7401a600 con 0x7f0d8407de20 2024-03-31T23:01:34.181 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:33.405+0000 7f0d8972f700 1 -- 172.21.15.46:0/3581817707 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0d841a1120 con 0x7f0d8407de20 2024-03-31T23:01:34.181 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:33.406+0000 7f0d7bfff700 1 -- 172.21.15.46:0/3581817707 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 4) ==== 46105+0+0 (secure 0 0 0) 0x7f0d7401a900 con 0x7f0d8407de20 2024-03-31T23:01:34.181 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:33.406+0000 7f0d8972f700 1 -- 172.21.15.46:0/3581817707 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0d84063870 con 0x7f0d8407de20 2024-03-31T23:01:34.181 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:33.406+0000 7f0d7bfff700 1 --2- 172.21.15.46:0/3581817707 >> [v2:172.21.15.46:6800/1515528215,v1:172.21.15.46:6801/1515528215] conn(0x7f0d70038db0 0x7f0d7003b270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:01:34.181 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:33.406+0000 7f0d7bfff700 1 -- 172.21.15.46:0/3581817707 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f0d7404d060 con 0x7f0d8407de20 2024-03-31T23:01:34.181 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:33.406+0000 7f0d827fc700 1 --2- 172.21.15.46:0/3581817707 >> [v2:172.21.15.46:6800/1515528215,v1:172.21.15.46:6801/1515528215] conn(0x7f0d70038db0 0x7f0d7003b270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:01:34.181 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:33.407+0000 7f0d827fc700 1 --2- 172.21.15.46:0/3581817707 >> [v2:172.21.15.46:6800/1515528215,v1:172.21.15.46:6801/1515528215] conn(0x7f0d70038db0 0x7f0d7003b270 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f0d6c006fd0 tx=0x7f0d6c006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:01:34.181 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:33.411+0000 7f0d7bfff700 1 -- 172.21.15.46:0/3581817707 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f0d74024af0 con 0x7f0d8407de20 2024-03-31T23:01:34.181 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:33.611+0000 7f0d8972f700 1 -- 172.21.15.46:0/3581817707 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) -- 0x7f0d841a28e0 con 0x7f0d8407de20 2024-03-31T23:01:34.181 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:33.612+0000 7f0d7bfff700 1 -- 172.21.15.46:0/3581817707 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v3) ==== 70+0+373 (secure 0 0 0) 0x7f0d7404f1e0 con 0x7f0d8407de20 2024-03-31T23:01:34.182 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:33.614+0000 7f0d8972f700 1 -- 172.21.15.46:0/3581817707 >> [v2:172.21.15.46:6800/1515528215,v1:172.21.15.46:6801/1515528215] conn(0x7f0d70038db0 msgr2=0x7f0d7003b270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:01:34.182 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:33.614+0000 7f0d8972f700 1 --2- 172.21.15.46:0/3581817707 >> [v2:172.21.15.46:6800/1515528215,v1:172.21.15.46:6801/1515528215] conn(0x7f0d70038db0 0x7f0d7003b270 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f0d6c006fd0 tx=0x7f0d6c006e40 comp rx=0 tx=0).stop 2024-03-31T23:01:34.182 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:33.614+0000 7f0d8972f700 1 -- 172.21.15.46:0/3581817707 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0d8407de20 msgr2=0x7f0d8419dc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:01:34.182 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:33.614+0000 7f0d8972f700 1 --2- 172.21.15.46:0/3581817707 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0d8407de20 0x7f0d8419dc60 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f0d74011a50 tx=0x7f0d74004290 comp rx=0 tx=0).stop 2024-03-31T23:01:34.182 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:33.615+0000 7f0d8972f700 1 -- 172.21.15.46:0/3581817707 shutdown_connections 2024-03-31T23:01:34.182 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:33.615+0000 7f0d8972f700 1 --2- 172.21.15.46:0/3581817707 >> [v2:172.21.15.46:6800/1515528215,v1:172.21.15.46:6801/1515528215] conn(0x7f0d70038db0 0x7f0d7003b270 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:01:34.182 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:33.615+0000 7f0d8972f700 1 --2- 172.21.15.46:0/3581817707 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0d8407de20 0x7f0d8419dc60 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:01:34.182 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:33.615+0000 7f0d8972f700 1 -- 172.21.15.46:0/3581817707 >> 172.21.15.46:0/3581817707 conn(0x7f0d84103100 msgr2=0x7f0d84103c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:01:34.182 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:33.615+0000 7f0d8972f700 1 -- 172.21.15.46:0/3581817707 shutdown_connections 2024-03-31T23:01:34.182 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:33.615+0000 7f0d8972f700 1 -- 172.21.15.46:0/3581817707 wait complete. 2024-03-31T23:01:34.182 INFO:teuthology.orchestra.run.smithi046.stdout:Enabling cephadm module... 2024-03-31T23:01:35.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:34 smithi046 ceph-mon[32498]: from='client.? 172.21.15.46:0/2984024022' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-03-31T23:01:35.112 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 31 23:01:34 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mgr-x[32727]: ignoring --setuser ceph since I am not root 2024-03-31T23:01:35.112 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 31 23:01:34 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mgr-x[32727]: ignoring --setgroup ceph since I am not root 2024-03-31T23:01:35.663 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 31 23:01:35 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mgr-x[32727]: 2024-03-31T23:01:35.476+0000 7fa8f8bd3200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-31T23:01:35.663 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 31 23:01:35 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mgr-x[32727]: 2024-03-31T23:01:35.572+0000 7fa8f8bd3200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-31T23:01:35.675 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:34.583+0000 7fb630f01700 1 Processor -- start 2024-03-31T23:01:35.675 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:34.583+0000 7fb630f01700 1 -- start start 2024-03-31T23:01:35.675 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:34.583+0000 7fb630f01700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb62c108590 0x7fb62c10a980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:01:35.675 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:34.583+0000 7fb630f01700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fb62c07bde0 con 0x7fb62c108590 2024-03-31T23:01:35.675 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:34.584+0000 7fb62a59c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb62c108590 0x7fb62c10a980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:01:35.675 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:34.584+0000 7fb62a59c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb62c108590 0x7fb62c10a980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60736/0 (socket says 172.21.15.46:60736) 2024-03-31T23:01:35.675 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:34.584+0000 7fb62a59c700 1 -- 172.21.15.46:0/3674430922 learned_addr learned my addr 172.21.15.46:0/3674430922 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:01:35.676 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:34.584+0000 7fb62a59c700 1 -- 172.21.15.46:0/3674430922 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb62c10aec0 con 0x7fb62c108590 2024-03-31T23:01:35.676 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:34.584+0000 7fb62a59c700 1 --2- 172.21.15.46:0/3674430922 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb62c108590 0x7fb62c10a980 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fb624009e80 tx=0x7fb62400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6dd8654dfc65eb77 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:01:35.676 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:34.584+0000 7fb62959a700 1 -- 172.21.15.46:0/3674430922 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb624017070 con 0x7fb62c108590 2024-03-31T23:01:35.676 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:34.584+0000 7fb62959a700 1 -- 172.21.15.46:0/3674430922 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7fb62400b8c0 con 0x7fb62c108590 2024-03-31T23:01:35.676 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:34.584+0000 7fb62959a700 1 -- 172.21.15.46:0/3674430922 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb624004520 con 0x7fb62c108590 2024-03-31T23:01:35.676 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:34.585+0000 7fb630f01700 1 -- 172.21.15.46:0/3674430922 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb62c108590 msgr2=0x7fb62c10a980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:01:35.676 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:34.585+0000 7fb630f01700 1 --2- 172.21.15.46:0/3674430922 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb62c108590 0x7fb62c10a980 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fb624009e80 tx=0x7fb62400b330 comp rx=0 tx=0).stop 2024-03-31T23:01:35.676 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:34.585+0000 7fb630f01700 1 -- 172.21.15.46:0/3674430922 shutdown_connections 2024-03-31T23:01:35.676 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:34.585+0000 7fb630f01700 1 --2- 172.21.15.46:0/3674430922 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb62c108590 0x7fb62c10a980 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:01:35.676 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:34.585+0000 7fb630f01700 1 -- 172.21.15.46:0/3674430922 >> 172.21.15.46:0/3674430922 conn(0x7fb62c101f60 msgr2=0x7fb62c1043a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:01:35.676 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:34.585+0000 7fb630f01700 1 -- 172.21.15.46:0/3674430922 shutdown_connections 2024-03-31T23:01:35.676 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:34.585+0000 7fb630f01700 1 -- 172.21.15.46:0/3674430922 wait complete. 2024-03-31T23:01:35.676 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:34.586+0000 7fb630f01700 1 Processor -- start 2024-03-31T23:01:35.676 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:34.586+0000 7fb630f01700 1 -- start start 2024-03-31T23:01:35.677 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:34.586+0000 7fb630f01700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb62c108590 0x7fb62c107e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:01:35.677 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:34.586+0000 7fb630f01700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fb62c10be30 con 0x7fb62c108590 2024-03-31T23:01:35.677 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:34.586+0000 7fb62a59c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb62c108590 0x7fb62c107e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:01:35.677 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:34.586+0000 7fb62a59c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb62c108590 0x7fb62c107e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60738/0 (socket says 172.21.15.46:60738) 2024-03-31T23:01:35.677 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:34.586+0000 7fb62a59c700 1 -- 172.21.15.46:0/2984024022 learned_addr learned my addr 172.21.15.46:0/2984024022 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:01:35.677 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:34.586+0000 7fb62a59c700 1 -- 172.21.15.46:0/2984024022 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb62c108350 con 0x7fb62c108590 2024-03-31T23:01:35.677 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:34.586+0000 7fb62a59c700 1 --2- 172.21.15.46:0/2984024022 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb62c108590 0x7fb62c107e10 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fb62400b7b0 tx=0x7fb624004670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:01:35.677 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:34.586+0000 7fb6237fe700 1 -- 172.21.15.46:0/2984024022 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb624020020 con 0x7fb62c108590 2024-03-31T23:01:35.677 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:34.587+0000 7fb6237fe700 1 -- 172.21.15.46:0/2984024022 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7fb62400bba0 con 0x7fb62c108590 2024-03-31T23:01:35.677 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:34.587+0000 7fb630f01700 1 -- 172.21.15.46:0/2984024022 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb62c106540 con 0x7fb62c108590 2024-03-31T23:01:35.677 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:34.587+0000 7fb6237fe700 1 -- 172.21.15.46:0/2984024022 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb624017040 con 0x7fb62c108590 2024-03-31T23:01:35.677 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:34.587+0000 7fb630f01700 1 -- 172.21.15.46:0/2984024022 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb62c106a80 con 0x7fb62c108590 2024-03-31T23:01:35.677 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:34.587+0000 7fb6237fe700 1 -- 172.21.15.46:0/2984024022 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 4) ==== 46105+0+0 (secure 0 0 0) 0x7fb624023020 con 0x7fb62c108590 2024-03-31T23:01:35.678 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:34.588+0000 7fb6237fe700 1 --2- 172.21.15.46:0/2984024022 >> [v2:172.21.15.46:6800/1515528215,v1:172.21.15.46:6801/1515528215] conn(0x7fb618038db0 0x7fb61803b270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:01:35.678 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:34.588+0000 7fb6237fe700 1 -- 172.21.15.46:0/2984024022 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fb62400f050 con 0x7fb62c108590 2024-03-31T23:01:35.678 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:34.588+0000 7fb630f01700 1 -- 172.21.15.46:0/2984024022 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb62c063870 con 0x7fb62c108590 2024-03-31T23:01:35.678 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:34.588+0000 7fb629d9b700 1 --2- 172.21.15.46:0/2984024022 >> [v2:172.21.15.46:6800/1515528215,v1:172.21.15.46:6801/1515528215] conn(0x7fb618038db0 0x7fb61803b270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:01:35.678 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:34.588+0000 7fb629d9b700 1 --2- 172.21.15.46:0/2984024022 >> [v2:172.21.15.46:6800/1515528215,v1:172.21.15.46:6801/1515528215] conn(0x7fb618038db0 0x7fb61803b270 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fb614006fd0 tx=0x7fb614006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:01:35.678 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:34.593+0000 7fb6237fe700 1 -- 172.21.15.46:0/2984024022 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fb62402a020 con 0x7fb62c108590 2024-03-31T23:01:35.678 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:34.859+0000 7fb630f01700 1 -- 172.21.15.46:0/2984024022 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) -- 0x7fb62c106c50 con 0x7fb62c108590 2024-03-31T23:01:35.678 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:34.910+0000 7fb6237fe700 1 -- 172.21.15.46:0/2984024022 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) ==== 86+0+0 (secure 0 0 0) 0x7fb62404b220 con 0x7fb62c108590 2024-03-31T23:01:35.678 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:34.912+0000 7fb6237fe700 1 -- 172.21.15.46:0/2984024022 <== mon.0 v2:172.21.15.46:3300/0 8 ==== mgrmap(e 5) ==== 46116+0+0 (secure 0 0 0) 0x7fb624018bb0 con 0x7fb62c108590 2024-03-31T23:01:35.678 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:34.913+0000 7fb630f01700 1 -- 172.21.15.46:0/2984024022 >> [v2:172.21.15.46:6800/1515528215,v1:172.21.15.46:6801/1515528215] conn(0x7fb618038db0 msgr2=0x7fb61803b270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:01:35.678 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:34.913+0000 7fb630f01700 1 --2- 172.21.15.46:0/2984024022 >> [v2:172.21.15.46:6800/1515528215,v1:172.21.15.46:6801/1515528215] conn(0x7fb618038db0 0x7fb61803b270 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fb614006fd0 tx=0x7fb614006e40 comp rx=0 tx=0).stop 2024-03-31T23:01:35.679 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:34.913+0000 7fb630f01700 1 -- 172.21.15.46:0/2984024022 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb62c108590 msgr2=0x7fb62c107e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:01:35.679 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:34.913+0000 7fb630f01700 1 --2- 172.21.15.46:0/2984024022 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb62c108590 0x7fb62c107e10 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fb62400b7b0 tx=0x7fb624004670 comp rx=0 tx=0).stop 2024-03-31T23:01:35.679 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:34.913+0000 7fb630f01700 1 -- 172.21.15.46:0/2984024022 shutdown_connections 2024-03-31T23:01:35.679 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:34.913+0000 7fb630f01700 1 --2- 172.21.15.46:0/2984024022 >> [v2:172.21.15.46:6800/1515528215,v1:172.21.15.46:6801/1515528215] conn(0x7fb618038db0 0x7fb61803b270 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:01:35.679 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:34.913+0000 7fb630f01700 1 --2- 172.21.15.46:0/2984024022 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb62c108590 0x7fb62c107e10 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:01:35.679 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:34.913+0000 7fb630f01700 1 -- 172.21.15.46:0/2984024022 >> 172.21.15.46:0/2984024022 conn(0x7fb62c101f60 msgr2=0x7fb62c102a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:01:35.679 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:34.913+0000 7fb630f01700 1 -- 172.21.15.46:0/2984024022 shutdown_connections 2024-03-31T23:01:35.679 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:34.913+0000 7fb630f01700 1 -- 172.21.15.46:0/2984024022 wait complete. 2024-03-31T23:01:36.243 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:35 smithi046 ceph-mon[32498]: from='client.? 172.21.15.46:0/2984024022' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-03-31T23:01:36.244 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:35 smithi046 ceph-mon[32498]: mgrmap e5: x(active, since 5s) 2024-03-31T23:01:36.528 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 31 23:01:36 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mgr-x[32727]: 2024-03-31T23:01:36.243+0000 7fa8f8bd3200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-31T23:01:36.528 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 31 23:01:36 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mgr-x[32727]: 2024-03-31T23:01:36.342+0000 7fa8f8bd3200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-31T23:01:36.528 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 31 23:01:36 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mgr-x[32727]: 2024-03-31T23:01:36.491+0000 7fa8f8bd3200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-31T23:01:36.825 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 31 23:01:36 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mgr-x[32727]: 2024-03-31T23:01:36.594+0000 7fa8f8bd3200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-31T23:01:36.826 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 31 23:01:36 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mgr-x[32727]: 2024-03-31T23:01:36.683+0000 7fa8f8bd3200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-31T23:01:37.048 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout { 2024-03-31T23:01:37.048 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-03-31T23:01:37.049 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-31T23:01:37.049 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-03-31T23:01:37.049 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-03-31T23:01:37.049 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout } 2024-03-31T23:01:37.049 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:36.125+0000 7f6325c8c700 1 Processor -- start 2024-03-31T23:01:37.049 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:36.125+0000 7f6325c8c700 1 -- start start 2024-03-31T23:01:37.049 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:36.125+0000 7f6325c8c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6320108560 0x7f632010a950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:01:37.049 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:36.125+0000 7f6325c8c700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f632007bde0 con 0x7f6320108560 2024-03-31T23:01:37.049 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:36.125+0000 7f631f7fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6320108560 0x7f632010a950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:01:37.049 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:36.125+0000 7f631f7fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6320108560 0x7f632010a950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60764/0 (socket says 172.21.15.46:60764) 2024-03-31T23:01:37.049 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:36.125+0000 7f631f7fe700 1 -- 172.21.15.46:0/2723165392 learned_addr learned my addr 172.21.15.46:0/2723165392 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:01:37.049 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:36.126+0000 7f631f7fe700 1 -- 172.21.15.46:0/2723165392 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f632010ae90 con 0x7f6320108560 2024-03-31T23:01:37.049 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:36.126+0000 7f631f7fe700 1 --2- 172.21.15.46:0/2723165392 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6320108560 0x7f632010a950 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f6310009e80 tx=0x7f631000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a5eb59bf64f76597 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:01:37.050 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:36.126+0000 7f631e7fc700 1 -- 172.21.15.46:0/2723165392 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6310017070 con 0x7f6320108560 2024-03-31T23:01:37.050 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:36.126+0000 7f631e7fc700 1 -- 172.21.15.46:0/2723165392 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7f631000b8c0 con 0x7f6320108560 2024-03-31T23:01:37.050 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:36.126+0000 7f631e7fc700 1 -- 172.21.15.46:0/2723165392 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6310004520 con 0x7f6320108560 2024-03-31T23:01:37.050 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:36.127+0000 7f6325c8c700 1 -- 172.21.15.46:0/2723165392 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6320108560 msgr2=0x7f632010a950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:01:37.050 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:36.127+0000 7f6325c8c700 1 --2- 172.21.15.46:0/2723165392 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6320108560 0x7f632010a950 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f6310009e80 tx=0x7f631000b330 comp rx=0 tx=0).stop 2024-03-31T23:01:37.050 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:36.127+0000 7f6325c8c700 1 -- 172.21.15.46:0/2723165392 shutdown_connections 2024-03-31T23:01:37.050 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:36.127+0000 7f6325c8c700 1 --2- 172.21.15.46:0/2723165392 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6320108560 0x7f632010a950 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:01:37.050 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:36.127+0000 7f6325c8c700 1 -- 172.21.15.46:0/2723165392 >> 172.21.15.46:0/2723165392 conn(0x7f6320101f30 msgr2=0x7f6320104370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:01:37.050 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:36.127+0000 7f6325c8c700 1 -- 172.21.15.46:0/2723165392 shutdown_connections 2024-03-31T23:01:37.050 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:36.127+0000 7f6325c8c700 1 -- 172.21.15.46:0/2723165392 wait complete. 2024-03-31T23:01:37.050 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:36.127+0000 7f6325c8c700 1 Processor -- start 2024-03-31T23:01:37.050 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:36.128+0000 7f6325c8c700 1 -- start start 2024-03-31T23:01:37.050 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:36.128+0000 7f6325c8c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6320108560 0x7f632019b940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:01:37.051 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:36.128+0000 7f6325c8c700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f632010be00 con 0x7f6320108560 2024-03-31T23:01:37.051 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:36.128+0000 7f631f7fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6320108560 0x7f632019b940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:01:37.051 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:36.128+0000 7f631f7fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6320108560 0x7f632019b940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60778/0 (socket says 172.21.15.46:60778) 2024-03-31T23:01:37.051 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:36.128+0000 7f631f7fe700 1 -- 172.21.15.46:0/264756181 learned_addr learned my addr 172.21.15.46:0/264756181 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:01:37.051 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:36.128+0000 7f631f7fe700 1 -- 172.21.15.46:0/264756181 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f632019be80 con 0x7f6320108560 2024-03-31T23:01:37.051 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:36.128+0000 7f631f7fe700 1 --2- 172.21.15.46:0/264756181 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6320108560 0x7f632019b940 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f6310009e50 tx=0x7f6310004670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:01:37.051 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:36.128+0000 7f631cff9700 1 -- 172.21.15.46:0/264756181 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6310020020 con 0x7f6320108560 2024-03-31T23:01:37.051 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:36.129+0000 7f631cff9700 1 -- 172.21.15.46:0/264756181 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7f631000bba0 con 0x7f6320108560 2024-03-31T23:01:37.051 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:36.129+0000 7f6325c8c700 1 -- 172.21.15.46:0/264756181 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f632019c110 con 0x7f6320108560 2024-03-31T23:01:37.051 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:36.129+0000 7f631cff9700 1 -- 172.21.15.46:0/264756181 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6310017040 con 0x7f6320108560 2024-03-31T23:01:37.051 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:36.129+0000 7f6325c8c700 1 -- 172.21.15.46:0/264756181 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f632019ee00 con 0x7f6320108560 2024-03-31T23:01:37.051 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:36.129+0000 7f6325c8c700 1 -- 172.21.15.46:0/264756181 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6320063870 con 0x7f6320108560 2024-03-31T23:01:37.051 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:36.130+0000 7f631cff9700 1 -- 172.21.15.46:0/264756181 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 46116+0+0 (secure 0 0 0) 0x7f6310023020 con 0x7f6320108560 2024-03-31T23:01:37.051 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:36.130+0000 7f631cff9700 1 --2- 172.21.15.46:0/264756181 >> [v2:172.21.15.46:6800/1515528215,v1:172.21.15.46:6801/1515528215] conn(0x7f630c038e00 0x7f630c03b2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:01:37.052 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:36.130+0000 7f631cff9700 1 -- 172.21.15.46:0/264756181 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f631000f050 con 0x7f6320108560 2024-03-31T23:01:37.052 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:36.130+0000 7f631effd700 1 -- 172.21.15.46:0/264756181 >> [v2:172.21.15.46:6800/1515528215,v1:172.21.15.46:6801/1515528215] conn(0x7f630c038e00 msgr2=0x7f630c03b2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.46:6800/1515528215 2024-03-31T23:01:37.052 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:36.130+0000 7f631effd700 1 --2- 172.21.15.46:0/264756181 >> [v2:172.21.15.46:6800/1515528215,v1:172.21.15.46:6801/1515528215] conn(0x7f630c038e00 0x7f630c03b2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-31T23:01:37.052 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:36.139+0000 7f631cff9700 1 -- 172.21.15.46:0/264756181 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f631002a020 con 0x7f6320108560 2024-03-31T23:01:37.052 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:36.331+0000 7f631effd700 1 -- 172.21.15.46:0/264756181 >> [v2:172.21.15.46:6800/1515528215,v1:172.21.15.46:6801/1515528215] conn(0x7f630c038e00 msgr2=0x7f630c03b2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.46:6800/1515528215 2024-03-31T23:01:37.052 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:36.331+0000 7f631effd700 1 --2- 172.21.15.46:0/264756181 >> [v2:172.21.15.46:6800/1515528215,v1:172.21.15.46:6801/1515528215] conn(0x7f630c038e00 0x7f630c03b2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-31T23:01:37.052 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:36.402+0000 7f6325c8c700 1 -- 172.21.15.46:0/264756181 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) -- 0x7f632019f4f0 con 0x7f6320108560 2024-03-31T23:01:37.052 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:36.403+0000 7f631cff9700 1 -- 172.21.15.46:0/264756181 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) ==== 56+0+88 (secure 0 0 0) 0x7f6310018240 con 0x7f6320108560 2024-03-31T23:01:37.052 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:36.405+0000 7f6325c8c700 1 -- 172.21.15.46:0/264756181 >> [v2:172.21.15.46:6800/1515528215,v1:172.21.15.46:6801/1515528215] conn(0x7f630c038e00 msgr2=0x7f630c03b2c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-31T23:01:37.052 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:36.405+0000 7f6325c8c700 1 --2- 172.21.15.46:0/264756181 >> [v2:172.21.15.46:6800/1515528215,v1:172.21.15.46:6801/1515528215] conn(0x7f630c038e00 0x7f630c03b2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:01:37.052 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:36.405+0000 7f6325c8c700 1 -- 172.21.15.46:0/264756181 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6320108560 msgr2=0x7f632019b940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:01:37.052 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:36.405+0000 7f6325c8c700 1 --2- 172.21.15.46:0/264756181 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6320108560 0x7f632019b940 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f6310009e50 tx=0x7f6310004670 comp rx=0 tx=0).stop 2024-03-31T23:01:37.052 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:36.405+0000 7f6325c8c700 1 -- 172.21.15.46:0/264756181 shutdown_connections 2024-03-31T23:01:37.053 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:36.405+0000 7f6325c8c700 1 --2- 172.21.15.46:0/264756181 >> [v2:172.21.15.46:6800/1515528215,v1:172.21.15.46:6801/1515528215] conn(0x7f630c038e00 0x7f630c03b2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:01:37.053 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:36.405+0000 7f6325c8c700 1 --2- 172.21.15.46:0/264756181 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6320108560 0x7f632019b940 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:01:37.053 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:36.405+0000 7f6325c8c700 1 -- 172.21.15.46:0/264756181 >> 172.21.15.46:0/264756181 conn(0x7f6320101f30 msgr2=0x7f63201029d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:01:37.053 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:36.405+0000 7f6325c8c700 1 -- 172.21.15.46:0/264756181 shutdown_connections 2024-03-31T23:01:37.053 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:36.405+0000 7f6325c8c700 1 -- 172.21.15.46:0/264756181 wait complete. 2024-03-31T23:01:37.053 INFO:teuthology.orchestra.run.smithi046.stdout:Waiting for the mgr to restart... 2024-03-31T23:01:37.053 INFO:teuthology.orchestra.run.smithi046.stdout:Waiting for mgr epoch 5... 2024-03-31T23:01:37.077 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:36 smithi046 ceph-mon[32498]: from='client.? 172.21.15.46:0/264756181' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-03-31T23:01:37.077 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 31 23:01:36 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mgr-x[32727]: 2024-03-31T23:01:36.959+0000 7fa8f8bd3200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-31T23:01:37.077 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 31 23:01:37 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mgr-x[32727]: 2024-03-31T23:01:37.077+0000 7fa8f8bd3200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-31T23:01:37.361 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 31 23:01:37 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mgr-x[32727]: 2024-03-31T23:01:37.167+0000 7fa8f8bd3200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-31T23:01:37.361 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 31 23:01:37 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mgr-x[32727]: 2024-03-31T23:01:37.266+0000 7fa8f8bd3200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-31T23:01:37.709 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 31 23:01:37 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mgr-x[32727]: 2024-03-31T23:01:37.452+0000 7fa8f8bd3200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-31T23:01:38.861 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 31 23:01:38 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mgr-x[32727]: 2024-03-31T23:01:38.443+0000 7fa8f8bd3200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-31T23:01:38.861 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 31 23:01:38 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mgr-x[32727]: 2024-03-31T23:01:38.576+0000 7fa8f8bd3200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-31T23:01:39.205 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 31 23:01:38 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mgr-x[32727]: 2024-03-31T23:01:38.915+0000 7fa8f8bd3200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-31T23:01:39.206 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 31 23:01:39 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mgr-x[32727]: 2024-03-31T23:01:39.006+0000 7fa8f8bd3200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-31T23:01:39.206 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 31 23:01:39 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mgr-x[32727]: 2024-03-31T23:01:39.205+0000 7fa8f8bd3200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-31T23:01:39.611 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 31 23:01:39 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mgr-x[32727]: 2024-03-31T23:01:39.394+0000 7fa8f8bd3200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-31T23:01:39.944 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 31 23:01:39 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mgr-x[32727]: 2024-03-31T23:01:39.683+0000 7fa8f8bd3200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-31T23:01:40.361 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 31 23:01:39 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mgr-x[32727]: 2024-03-31T23:01:39.944+0000 7fa8f8bd3200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-31T23:01:40.361 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 31 23:01:40 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mgr-x[32727]: 2024-03-31T23:01:40.186+0000 7fa8f8bd3200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-31T23:01:41.361 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 31 23:01:41 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mgr-x[32727]: 2024-03-31T23:01:41.009+0000 7fa8f8bd3200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-31T23:01:41.861 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 31 23:01:41 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mgr-x[32727]: 2024-03-31T23:01:41.588+0000 7fa8f8bd3200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-31T23:01:41.861 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 31 23:01:41 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mgr-x[32727]: 2024-03-31T23:01:41.688+0000 7fa8f8bd3200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-31T23:01:42.361 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 31 23:01:41 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mgr-x[32727]: 2024-03-31T23:01:41.908+0000 7fa8f8bd3200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-31T23:01:42.362 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:41 smithi046 ceph-mon[32498]: Active manager daemon x restarted 2024-03-31T23:01:42.362 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:41 smithi046 ceph-mon[32498]: Activating manager daemon x 2024-03-31T23:01:42.362 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:41 smithi046 ceph-mon[32498]: osdmap e2: 0 total, 0 up, 0 in 2024-03-31T23:01:42.362 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:41 smithi046 ceph-mon[32498]: mgrmap e6: x(active, starting, since 0.00346695s) 2024-03-31T23:01:42.362 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:41 smithi046 ceph-mon[32498]: from='mgr.14120 172.21.15.46:0/1993682382' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-31T23:01:42.362 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:41 smithi046 ceph-mon[32498]: from='mgr.14120 172.21.15.46:0/1993682382' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-31T23:01:42.362 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:41 smithi046 ceph-mon[32498]: from='mgr.14120 172.21.15.46:0/1993682382' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-31T23:01:42.362 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:41 smithi046 ceph-mon[32498]: from='mgr.14120 172.21.15.46:0/1993682382' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-31T23:01:42.362 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:41 smithi046 ceph-mon[32498]: from='mgr.14120 172.21.15.46:0/1993682382' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-31T23:01:42.362 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:41 smithi046 ceph-mon[32498]: Manager daemon x is now available 2024-03-31T23:01:42.362 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:41 smithi046 ceph-mon[32498]: Found migration_current of "None". Setting to last migration. 2024-03-31T23:01:42.362 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:41 smithi046 ceph-mon[32498]: from='mgr.14120 172.21.15.46:0/1993682382' entity='mgr.x' 2024-03-31T23:01:42.363 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:41 smithi046 ceph-mon[32498]: from='mgr.14120 172.21.15.46:0/1993682382' entity='mgr.x' 2024-03-31T23:01:42.363 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:41 smithi046 ceph-mon[32498]: from='mgr.14120 172.21.15.46:0/1993682382' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-31T23:01:42.363 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:41 smithi046 ceph-mon[32498]: from='mgr.14120 172.21.15.46:0/1993682382' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-31T23:01:42.363 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:41 smithi046 ceph-mon[32498]: from='mgr.14120 172.21.15.46:0/1993682382' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-31T23:01:43.209 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:42 smithi046 ceph-mon[32498]: from='mgr.14120 172.21.15.46:0/1993682382' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-31T23:01:43.209 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:42 smithi046 ceph-mon[32498]: from='mgr.14120 172.21.15.46:0/1993682382' entity='mgr.x' 2024-03-31T23:01:43.209 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:42 smithi046 ceph-mon[32498]: from='mgr.14120 172.21.15.46:0/1993682382' entity='mgr.x' 2024-03-31T23:01:43.209 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:42 smithi046 ceph-mon[32498]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-03-31T23:01:43.209 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:42 smithi046 ceph-mon[32498]: mgrmap e7: x(active, since 1.00518s) 2024-03-31T23:01:43.209 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:42 smithi046 ceph-mon[32498]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-03-31T23:01:43.477 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout { 2024-03-31T23:01:43.478 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-03-31T23:01:43.478 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "initialized": true 2024-03-31T23:01:43.478 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout } 2024-03-31T23:01:43.478 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:37.503+0000 7f2e1aa28700 1 Processor -- start 2024-03-31T23:01:43.478 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:37.503+0000 7f2e1aa28700 1 -- start start 2024-03-31T23:01:43.478 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:37.503+0000 7f2e1aa28700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2e1407cd20 0x7f2e1407b180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:01:43.478 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:37.503+0000 7f2e1aa28700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f2e1407b750 con 0x7f2e1407cd20 2024-03-31T23:01:43.478 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:37.503+0000 7f2e13fff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2e1407cd20 0x7f2e1407b180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:01:43.478 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:37.503+0000 7f2e13fff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2e1407cd20 0x7f2e1407b180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60790/0 (socket says 172.21.15.46:60790) 2024-03-31T23:01:43.478 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:37.503+0000 7f2e13fff700 1 -- 172.21.15.46:0/2539018909 learned_addr learned my addr 172.21.15.46:0/2539018909 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:01:43.478 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:37.504+0000 7f2e13fff700 1 -- 172.21.15.46:0/2539018909 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2e1407b8d0 con 0x7f2e1407cd20 2024-03-31T23:01:43.479 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:37.504+0000 7f2e13fff700 1 --2- 172.21.15.46:0/2539018909 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2e1407cd20 0x7f2e1407b180 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f2e040098a0 tx=0x7f2e04009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=31696abf3a194e31 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:01:43.479 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:37.504+0000 7f2e12ffd700 1 -- 172.21.15.46:0/2539018909 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2e04004160 con 0x7f2e1407cd20 2024-03-31T23:01:43.479 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:37.504+0000 7f2e12ffd700 1 -- 172.21.15.46:0/2539018909 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7f2e04004300 con 0x7f2e1407cd20 2024-03-31T23:01:43.479 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:37.504+0000 7f2e12ffd700 1 -- 172.21.15.46:0/2539018909 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2e04011a80 con 0x7f2e1407cd20 2024-03-31T23:01:43.479 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:37.505+0000 7f2e1aa28700 1 -- 172.21.15.46:0/2539018909 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2e1407cd20 msgr2=0x7f2e1407b180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:01:43.479 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:37.505+0000 7f2e1aa28700 1 --2- 172.21.15.46:0/2539018909 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2e1407cd20 0x7f2e1407b180 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f2e040098a0 tx=0x7f2e04009bb0 comp rx=0 tx=0).stop 2024-03-31T23:01:43.479 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:37.505+0000 7f2e1aa28700 1 -- 172.21.15.46:0/2539018909 shutdown_connections 2024-03-31T23:01:43.480 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:37.505+0000 7f2e1aa28700 1 --2- 172.21.15.46:0/2539018909 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2e1407cd20 0x7f2e1407b180 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:01:43.480 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:37.505+0000 7f2e1aa28700 1 -- 172.21.15.46:0/2539018909 >> 172.21.15.46:0/2539018909 conn(0x7f2e14103040 msgr2=0x7f2e14105480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:01:43.480 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:37.505+0000 7f2e1aa28700 1 -- 172.21.15.46:0/2539018909 shutdown_connections 2024-03-31T23:01:43.480 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:37.505+0000 7f2e1aa28700 1 -- 172.21.15.46:0/2539018909 wait complete. 2024-03-31T23:01:43.480 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:37.506+0000 7f2e1aa28700 1 Processor -- start 2024-03-31T23:01:43.480 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:37.506+0000 7f2e1aa28700 1 -- start start 2024-03-31T23:01:43.480 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:37.506+0000 7f2e1aa28700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2e1407cd20 0x7f2e1419dbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:01:43.480 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:37.506+0000 7f2e1aa28700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f2e1410e050 con 0x7f2e1407cd20 2024-03-31T23:01:43.480 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:37.506+0000 7f2e13fff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2e1407cd20 0x7f2e1419dbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:01:43.481 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:37.506+0000 7f2e13fff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2e1407cd20 0x7f2e1419dbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60802/0 (socket says 172.21.15.46:60802) 2024-03-31T23:01:43.481 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:37.506+0000 7f2e13fff700 1 -- 172.21.15.46:0/1919166014 learned_addr learned my addr 172.21.15.46:0/1919166014 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:01:43.481 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:37.506+0000 7f2e13fff700 1 -- 172.21.15.46:0/1919166014 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2e1419e130 con 0x7f2e1407cd20 2024-03-31T23:01:43.481 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:37.507+0000 7f2e13fff700 1 --2- 172.21.15.46:0/1919166014 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2e1407cd20 0x7f2e1419dbf0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f2e04006af0 tx=0x7f2e04011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:01:43.481 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:37.507+0000 7f2e117fa700 1 -- 172.21.15.46:0/1919166014 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2e040039e0 con 0x7f2e1407cd20 2024-03-31T23:01:43.481 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:37.507+0000 7f2e1aa28700 1 -- 172.21.15.46:0/1919166014 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2e1419e3c0 con 0x7f2e1407cd20 2024-03-31T23:01:43.481 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:37.507+0000 7f2e117fa700 1 -- 172.21.15.46:0/1919166014 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7f2e04003b80 con 0x7f2e1407cd20 2024-03-31T23:01:43.481 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:37.507+0000 7f2e117fa700 1 -- 172.21.15.46:0/1919166014 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2e040196a0 con 0x7f2e1407cd20 2024-03-31T23:01:43.481 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:37.507+0000 7f2e1aa28700 1 -- 172.21.15.46:0/1919166014 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2e141a10b0 con 0x7f2e1407cd20 2024-03-31T23:01:43.481 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:37.508+0000 7f2e117fa700 1 -- 172.21.15.46:0/1919166014 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 46116+0+0 (secure 0 0 0) 0x7f2e0401a430 con 0x7f2e1407cd20 2024-03-31T23:01:43.481 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:37.508+0000 7f2e117fa700 1 --2- 172.21.15.46:0/1919166014 >> [v2:172.21.15.46:6800/1515528215,v1:172.21.15.46:6801/1515528215] conn(0x7f2dfc038db0 0x7f2dfc03b270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:01:43.481 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:37.508+0000 7f2e117fa700 1 -- 172.21.15.46:0/1919166014 --> [v2:172.21.15.46:6800/1515528215,v1:172.21.15.46:6801/1515528215] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f2dfc03b980 con 0x7f2dfc038db0 2024-03-31T23:01:43.481 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:37.508+0000 7f2e117fa700 1 -- 172.21.15.46:0/1919166014 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f2e0404d1f0 con 0x7f2e1407cd20 2024-03-31T23:01:43.481 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:37.508+0000 7f2e137fe700 1 -- 172.21.15.46:0/1919166014 >> [v2:172.21.15.46:6800/1515528215,v1:172.21.15.46:6801/1515528215] conn(0x7f2dfc038db0 msgr2=0x7f2dfc03b270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.46:6800/1515528215 2024-03-31T23:01:43.482 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:37.508+0000 7f2e137fe700 1 --2- 172.21.15.46:0/1919166014 >> [v2:172.21.15.46:6800/1515528215,v1:172.21.15.46:6801/1515528215] conn(0x7f2dfc038db0 0x7f2dfc03b270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-31T23:01:43.482 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:37.709+0000 7f2e137fe700 1 -- 172.21.15.46:0/1919166014 >> [v2:172.21.15.46:6800/1515528215,v1:172.21.15.46:6801/1515528215] conn(0x7f2dfc038db0 msgr2=0x7f2dfc03b270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.46:6800/1515528215 2024-03-31T23:01:43.482 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:37.709+0000 7f2e137fe700 1 --2- 172.21.15.46:0/1919166014 >> [v2:172.21.15.46:6800/1515528215,v1:172.21.15.46:6801/1515528215] conn(0x7f2dfc038db0 0x7f2dfc03b270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-31T23:01:43.482 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:38.110+0000 7f2e137fe700 1 -- 172.21.15.46:0/1919166014 >> [v2:172.21.15.46:6800/1515528215,v1:172.21.15.46:6801/1515528215] conn(0x7f2dfc038db0 msgr2=0x7f2dfc03b270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.46:6800/1515528215 2024-03-31T23:01:43.482 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:38.110+0000 7f2e137fe700 1 --2- 172.21.15.46:0/1919166014 >> [v2:172.21.15.46:6800/1515528215,v1:172.21.15.46:6801/1515528215] conn(0x7f2dfc038db0 0x7f2dfc03b270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-31T23:01:43.482 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:38.911+0000 7f2e137fe700 1 -- 172.21.15.46:0/1919166014 >> [v2:172.21.15.46:6800/1515528215,v1:172.21.15.46:6801/1515528215] conn(0x7f2dfc038db0 msgr2=0x7f2dfc03b270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.46:6800/1515528215 2024-03-31T23:01:43.482 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:38.911+0000 7f2e137fe700 1 --2- 172.21.15.46:0/1919166014 >> [v2:172.21.15.46:6800/1515528215,v1:172.21.15.46:6801/1515528215] conn(0x7f2dfc038db0 0x7f2dfc03b270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-31T23:01:43.482 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:40.512+0000 7f2e137fe700 1 -- 172.21.15.46:0/1919166014 >> [v2:172.21.15.46:6800/1515528215,v1:172.21.15.46:6801/1515528215] conn(0x7f2dfc038db0 msgr2=0x7f2dfc03b270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.46:6800/1515528215 2024-03-31T23:01:43.482 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:40.512+0000 7f2e137fe700 1 --2- 172.21.15.46:0/1919166014 >> [v2:172.21.15.46:6800/1515528215,v1:172.21.15.46:6801/1515528215] conn(0x7f2dfc038db0 0x7f2dfc03b270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-03-31T23:01:43.482 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:41.912+0000 7f2e117fa700 1 -- 172.21.15.46:0/1919166014 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mgrmap(e 6) ==== 45883+0+0 (secure 0 0 0) 0x7f2e040199a0 con 0x7f2e1407cd20 2024-03-31T23:01:43.482 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:41.912+0000 7f2e117fa700 1 -- 172.21.15.46:0/1919166014 >> [v2:172.21.15.46:6800/1515528215,v1:172.21.15.46:6801/1515528215] conn(0x7f2dfc038db0 msgr2=0x7f2dfc03b270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-31T23:01:43.483 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:41.912+0000 7f2e117fa700 1 --2- 172.21.15.46:0/1919166014 >> [v2:172.21.15.46:6800/1515528215,v1:172.21.15.46:6801/1515528215] conn(0x7f2dfc038db0 0x7f2dfc03b270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:01:43.483 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:42.912+0000 7f2e117fa700 1 -- 172.21.15.46:0/1919166014 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mgrmap(e 7) ==== 46010+0+0 (secure 0 0 0) 0x7f2e0404e920 con 0x7f2e1407cd20 2024-03-31T23:01:43.483 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:42.912+0000 7f2e117fa700 1 --2- 172.21.15.46:0/1919166014 >> [v2:172.21.15.46:6800/3681182647,v1:172.21.15.46:6801/3681182647] conn(0x7f2dfc03c800 0x7f2dfc03ebf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:01:43.483 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:42.912+0000 7f2e117fa700 1 -- 172.21.15.46:0/1919166014 --> [v2:172.21.15.46:6800/3681182647,v1:172.21.15.46:6801/3681182647] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f2e04008ea0 con 0x7f2dfc03c800 2024-03-31T23:01:43.483 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:42.912+0000 7f2e137fe700 1 --2- 172.21.15.46:0/1919166014 >> [v2:172.21.15.46:6800/3681182647,v1:172.21.15.46:6801/3681182647] conn(0x7f2dfc03c800 0x7f2dfc03ebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:01:43.483 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:42.913+0000 7f2e137fe700 1 --2- 172.21.15.46:0/1919166014 >> [v2:172.21.15.46:6800/3681182647,v1:172.21.15.46:6801/3681182647] conn(0x7f2dfc03c800 0x7f2dfc03ebf0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f2e08003a10 tx=0x7f2e08009290 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:01:43.483 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:42.913+0000 7f2e117fa700 1 -- 172.21.15.46:0/1919166014 <== mgr.14120 v2:172.21.15.46:6800/3681182647 1 ==== command_reply(tid 0: 0 ) ==== 8+0+7774 (secure 0 0 0) 0x7f2e04008ea0 con 0x7f2dfc03c800 2024-03-31T23:01:43.483 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:42.920+0000 7f2e1aa28700 1 -- 172.21.15.46:0/1919166014 --> [v2:172.21.15.46:6800/3681182647,v1:172.21.15.46:6801/3681182647] -- command(tid 1: {"prefix": "mgr_status"}) -- 0x7f2e14063870 con 0x7f2dfc03c800 2024-03-31T23:01:43.483 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:42.920+0000 7f2e117fa700 1 -- 172.21.15.46:0/1919166014 <== mgr.14120 v2:172.21.15.46:6800/3681182647 2 ==== command_reply(tid 1: 0 ) ==== 8+0+51 (secure 0 0 0) 0x7f2e14063870 con 0x7f2dfc03c800 2024-03-31T23:01:43.483 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:42.920+0000 7f2e1aa28700 1 -- 172.21.15.46:0/1919166014 >> [v2:172.21.15.46:6800/3681182647,v1:172.21.15.46:6801/3681182647] conn(0x7f2dfc03c800 msgr2=0x7f2dfc03ebf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:01:43.483 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:42.920+0000 7f2e1aa28700 1 --2- 172.21.15.46:0/1919166014 >> [v2:172.21.15.46:6800/3681182647,v1:172.21.15.46:6801/3681182647] conn(0x7f2dfc03c800 0x7f2dfc03ebf0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f2e08003a10 tx=0x7f2e08009290 comp rx=0 tx=0).stop 2024-03-31T23:01:43.484 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:42.920+0000 7f2e1aa28700 1 -- 172.21.15.46:0/1919166014 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2e1407cd20 msgr2=0x7f2e1419dbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:01:43.484 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:42.920+0000 7f2e1aa28700 1 --2- 172.21.15.46:0/1919166014 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2e1407cd20 0x7f2e1419dbf0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f2e04006af0 tx=0x7f2e04011c40 comp rx=0 tx=0).stop 2024-03-31T23:01:43.484 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:42.921+0000 7f2e1aa28700 1 -- 172.21.15.46:0/1919166014 shutdown_connections 2024-03-31T23:01:43.484 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:42.921+0000 7f2e1aa28700 1 --2- 172.21.15.46:0/1919166014 >> [v2:172.21.15.46:6800/3681182647,v1:172.21.15.46:6801/3681182647] conn(0x7f2dfc03c800 0x7f2dfc03ebf0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:01:43.484 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:42.921+0000 7f2e1aa28700 1 --2- 172.21.15.46:0/1919166014 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2e1407cd20 0x7f2e1419dbf0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:01:43.484 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:42.921+0000 7f2e1aa28700 1 --2- 172.21.15.46:0/1919166014 >> [v2:172.21.15.46:6800/1515528215,v1:172.21.15.46:6801/1515528215] conn(0x7f2dfc038db0 0x7f2dfc03b270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:01:43.484 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:42.921+0000 7f2e1aa28700 1 -- 172.21.15.46:0/1919166014 >> 172.21.15.46:0/1919166014 conn(0x7f2e14103040 msgr2=0x7f2e14103b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:01:43.484 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:42.921+0000 7f2e1aa28700 1 -- 172.21.15.46:0/1919166014 shutdown_connections 2024-03-31T23:01:43.484 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:42.921+0000 7f2e1aa28700 1 -- 172.21.15.46:0/1919166014 wait complete. 2024-03-31T23:01:43.484 INFO:teuthology.orchestra.run.smithi046.stdout:mgr epoch 5 is available 2024-03-31T23:01:43.485 INFO:teuthology.orchestra.run.smithi046.stdout:Setting orchestrator backend to cephadm... 2024-03-31T23:01:44.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:43 smithi046 ceph-mon[32498]: [31/Mar/2024:23:01:43] ENGINE Bus STARTING 2024-03-31T23:01:44.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:43 smithi046 ceph-mon[32498]: [31/Mar/2024:23:01:43] ENGINE Serving on https://172.21.15.46:7150 2024-03-31T23:01:44.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:43 smithi046 ceph-mon[32498]: [31/Mar/2024:23:01:43] ENGINE Serving on http://172.21.15.46:8765 2024-03-31T23:01:44.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:43 smithi046 ceph-mon[32498]: [31/Mar/2024:23:01:43] ENGINE Bus STARTED 2024-03-31T23:01:44.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:43 smithi046 ceph-mon[32498]: from='mgr.14120 172.21.15.46:0/1993682382' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-31T23:01:44.708 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:43.905+0000 7f4538480700 1 Processor -- start 2024-03-31T23:01:44.708 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:43.905+0000 7f4538480700 1 -- start start 2024-03-31T23:01:44.708 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:43.905+0000 7f4538480700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f45301069c0 0x7f4530106de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:01:44.708 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:43.905+0000 7f4538480700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f45301073b0 con 0x7f45301069c0 2024-03-31T23:01:44.708 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:43.905+0000 7f453621c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f45301069c0 0x7f4530106de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:01:44.709 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:43.905+0000 7f453621c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f45301069c0 0x7f4530106de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:36350/0 (socket says 172.21.15.46:36350) 2024-03-31T23:01:44.709 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:43.905+0000 7f453621c700 1 -- 172.21.15.46:0/1125924793 learned_addr learned my addr 172.21.15.46:0/1125924793 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:01:44.709 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:43.905+0000 7f453621c700 1 -- 172.21.15.46:0/1125924793 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4530107bc0 con 0x7f45301069c0 2024-03-31T23:01:44.709 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:43.906+0000 7f453621c700 1 --2- 172.21.15.46:0/1125924793 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f45301069c0 0x7f4530106de0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f452c0098a0 tx=0x7f452c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=64cd5d28ba0a56d9 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:01:44.709 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:43.906+0000 7f453521a700 1 -- 172.21.15.46:0/1125924793 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f452c004160 con 0x7f45301069c0 2024-03-31T23:01:44.709 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:43.906+0000 7f453521a700 1 -- 172.21.15.46:0/1125924793 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7f452c004300 con 0x7f45301069c0 2024-03-31T23:01:44.709 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:43.906+0000 7f453521a700 1 -- 172.21.15.46:0/1125924793 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f452c011a80 con 0x7f45301069c0 2024-03-31T23:01:44.709 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:43.906+0000 7f4538480700 1 -- 172.21.15.46:0/1125924793 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f45301069c0 msgr2=0x7f4530106de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:01:44.709 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:43.906+0000 7f4538480700 1 --2- 172.21.15.46:0/1125924793 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f45301069c0 0x7f4530106de0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f452c0098a0 tx=0x7f452c009bb0 comp rx=0 tx=0).stop 2024-03-31T23:01:44.709 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:43.907+0000 7f4538480700 1 -- 172.21.15.46:0/1125924793 shutdown_connections 2024-03-31T23:01:44.709 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:43.907+0000 7f4538480700 1 --2- 172.21.15.46:0/1125924793 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f45301069c0 0x7f4530106de0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:01:44.709 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:43.907+0000 7f4538480700 1 -- 172.21.15.46:0/1125924793 >> 172.21.15.46:0/1125924793 conn(0x7f4530101f60 msgr2=0x7f45301043a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:01:44.709 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:43.907+0000 7f4538480700 1 -- 172.21.15.46:0/1125924793 shutdown_connections 2024-03-31T23:01:44.709 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:43.907+0000 7f4538480700 1 -- 172.21.15.46:0/1125924793 wait complete. 2024-03-31T23:01:44.710 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:43.907+0000 7f4538480700 1 Processor -- start 2024-03-31T23:01:44.710 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:43.907+0000 7f4538480700 1 -- start start 2024-03-31T23:01:44.710 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:43.907+0000 7f4538480700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f45301069c0 0x7f453007ba40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:01:44.710 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:43.907+0000 7f4538480700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f453010af70 con 0x7f45301069c0 2024-03-31T23:01:44.710 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:43.907+0000 7f453621c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f45301069c0 0x7f453007ba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:01:44.710 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:43.907+0000 7f453621c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f45301069c0 0x7f453007ba40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:36360/0 (socket says 172.21.15.46:36360) 2024-03-31T23:01:44.710 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:43.907+0000 7f453621c700 1 -- 172.21.15.46:0/3062500391 learned_addr learned my addr 172.21.15.46:0/3062500391 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:01:44.710 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:43.908+0000 7f453621c700 1 -- 172.21.15.46:0/3062500391 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f453019e7b0 con 0x7f45301069c0 2024-03-31T23:01:44.710 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:43.908+0000 7f453621c700 1 --2- 172.21.15.46:0/3062500391 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f45301069c0 0x7f453007ba40 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f452c006af0 tx=0x7f452c011b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:01:44.710 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:43.908+0000 7f45277fe700 1 -- 172.21.15.46:0/3062500391 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f452c003930 con 0x7f45301069c0 2024-03-31T23:01:44.710 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:43.908+0000 7f45277fe700 1 -- 172.21.15.46:0/3062500391 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7f452c003ad0 con 0x7f45301069c0 2024-03-31T23:01:44.710 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:43.908+0000 7f4538480700 1 -- 172.21.15.46:0/3062500391 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f453019e980 con 0x7f45301069c0 2024-03-31T23:01:44.710 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:43.908+0000 7f45277fe700 1 -- 172.21.15.46:0/3062500391 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f452c0195e0 con 0x7f45301069c0 2024-03-31T23:01:44.711 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:43.908+0000 7f4538480700 1 -- 172.21.15.46:0/3062500391 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f453019eda0 con 0x7f45301069c0 2024-03-31T23:01:44.711 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:43.909+0000 7f45277fe700 1 -- 172.21.15.46:0/3062500391 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 7) ==== 46010+0+0 (secure 0 0 0) 0x7f452c019960 con 0x7f45301069c0 2024-03-31T23:01:44.711 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:43.909+0000 7f45277fe700 1 --2- 172.21.15.46:0/3062500391 >> [v2:172.21.15.46:6800/3681182647,v1:172.21.15.46:6801/3681182647] conn(0x7f451c038c80 0x7f451c03b140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:01:44.711 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:43.909+0000 7f45277fe700 1 -- 172.21.15.46:0/3062500391 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f452c04d1f0 con 0x7f45301069c0 2024-03-31T23:01:44.711 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:43.909+0000 7f4538480700 1 -- 172.21.15.46:0/3062500391 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4530063870 con 0x7f45301069c0 2024-03-31T23:01:44.711 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:43.909+0000 7f4535a1b700 1 --2- 172.21.15.46:0/3062500391 >> [v2:172.21.15.46:6800/3681182647,v1:172.21.15.46:6801/3681182647] conn(0x7f451c038c80 0x7f451c03b140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:01:44.711 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:43.910+0000 7f4535a1b700 1 --2- 172.21.15.46:0/3062500391 >> [v2:172.21.15.46:6800/3681182647,v1:172.21.15.46:6801/3681182647] conn(0x7f451c038c80 0x7f451c03b140 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f4520006fd0 tx=0x7f4520006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:01:44.711 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:43.914+0000 7f45277fe700 1 -- 172.21.15.46:0/3062500391 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f452c019ca0 con 0x7f45301069c0 2024-03-31T23:01:44.711 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:43.963+0000 7f45277fe700 1 -- 172.21.15.46:0/3062500391 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mgrmap(e 8) ==== 46116+0+0 (secure 0 0 0) 0x7f452c02d960 con 0x7f45301069c0 2024-03-31T23:01:44.711 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:44.135+0000 7f4538480700 1 -- 172.21.15.46:0/3062500391 --> [v2:172.21.15.46:6800/3681182647,v1:172.21.15.46:6801/3681182647] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) -- 0x7f4530103220 con 0x7f451c038c80 2024-03-31T23:01:44.711 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:44.141+0000 7f45277fe700 1 -- 172.21.15.46:0/3062500391 <== mgr.14120 v2:172.21.15.46:6800/3681182647 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f4530103220 con 0x7f451c038c80 2024-03-31T23:01:44.711 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:44.143+0000 7f4538480700 1 -- 172.21.15.46:0/3062500391 >> [v2:172.21.15.46:6800/3681182647,v1:172.21.15.46:6801/3681182647] conn(0x7f451c038c80 msgr2=0x7f451c03b140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:01:44.711 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:44.143+0000 7f4538480700 1 --2- 172.21.15.46:0/3062500391 >> [v2:172.21.15.46:6800/3681182647,v1:172.21.15.46:6801/3681182647] conn(0x7f451c038c80 0x7f451c03b140 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f4520006fd0 tx=0x7f4520006e40 comp rx=0 tx=0).stop 2024-03-31T23:01:44.711 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:44.143+0000 7f4538480700 1 -- 172.21.15.46:0/3062500391 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f45301069c0 msgr2=0x7f453007ba40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:01:44.711 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:44.143+0000 7f4538480700 1 --2- 172.21.15.46:0/3062500391 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f45301069c0 0x7f453007ba40 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f452c006af0 tx=0x7f452c011b70 comp rx=0 tx=0).stop 2024-03-31T23:01:44.712 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:44.143+0000 7f4538480700 1 -- 172.21.15.46:0/3062500391 shutdown_connections 2024-03-31T23:01:44.712 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:44.143+0000 7f4538480700 1 --2- 172.21.15.46:0/3062500391 >> [v2:172.21.15.46:6800/3681182647,v1:172.21.15.46:6801/3681182647] conn(0x7f451c038c80 0x7f451c03b140 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:01:44.712 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:44.143+0000 7f4538480700 1 --2- 172.21.15.46:0/3062500391 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f45301069c0 0x7f453007ba40 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:01:44.712 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:44.143+0000 7f4538480700 1 -- 172.21.15.46:0/3062500391 >> 172.21.15.46:0/3062500391 conn(0x7f4530101f60 msgr2=0x7f4530102b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:01:44.712 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:44.144+0000 7f4538480700 1 -- 172.21.15.46:0/3062500391 shutdown_connections 2024-03-31T23:01:44.712 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:44.144+0000 7f4538480700 1 -- 172.21.15.46:0/3062500391 wait complete. 2024-03-31T23:01:45.290 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:44 smithi046 ceph-mon[32498]: mgrmap e8: x(active, since 2s) 2024-03-31T23:01:45.290 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:44 smithi046 ceph-mon[32498]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-03-31T23:01:45.290 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:44 smithi046 ceph-mon[32498]: from='mgr.14120 172.21.15.46:0/1993682382' entity='mgr.x' 2024-03-31T23:01:45.290 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:44 smithi046 ceph-mon[32498]: from='mgr.14120 172.21.15.46:0/1993682382' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-31T23:01:45.848 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout value unchanged 2024-03-31T23:01:45.848 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:45.075+0000 7f193bd26700 1 Processor -- start 2024-03-31T23:01:45.848 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:45.075+0000 7f193bd26700 1 -- start start 2024-03-31T23:01:45.848 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:45.075+0000 7f193bd26700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1934106ed0 0x7f19341072f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:01:45.848 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:45.075+0000 7f193bd26700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f19341078c0 con 0x7f1934106ed0 2024-03-31T23:01:45.848 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:45.075+0000 7f1939ac2700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1934106ed0 0x7f19341072f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:01:45.848 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:45.075+0000 7f1939ac2700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1934106ed0 0x7f19341072f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:36370/0 (socket says 172.21.15.46:36370) 2024-03-31T23:01:45.848 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:45.075+0000 7f1939ac2700 1 -- 172.21.15.46:0/3583558325 learned_addr learned my addr 172.21.15.46:0/3583558325 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:01:45.848 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:45.075+0000 7f1939ac2700 1 -- 172.21.15.46:0/3583558325 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1934107bf0 con 0x7f1934106ed0 2024-03-31T23:01:45.848 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:45.076+0000 7f1939ac2700 1 --2- 172.21.15.46:0/3583558325 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1934106ed0 0x7f19341072f0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f19300098a0 tx=0x7f1930009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e938be42f3c709bb server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:01:45.848 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:45.076+0000 7f1938ac0700 1 -- 172.21.15.46:0/3583558325 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1930004160 con 0x7f1934106ed0 2024-03-31T23:01:45.849 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:45.076+0000 7f1938ac0700 1 -- 172.21.15.46:0/3583558325 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7f1930004300 con 0x7f1934106ed0 2024-03-31T23:01:45.849 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:45.076+0000 7f1938ac0700 1 -- 172.21.15.46:0/3583558325 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1930011a80 con 0x7f1934106ed0 2024-03-31T23:01:45.849 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:45.076+0000 7f193bd26700 1 -- 172.21.15.46:0/3583558325 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1934106ed0 msgr2=0x7f19341072f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:01:45.849 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:45.076+0000 7f193bd26700 1 --2- 172.21.15.46:0/3583558325 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1934106ed0 0x7f19341072f0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f19300098a0 tx=0x7f1930009bb0 comp rx=0 tx=0).stop 2024-03-31T23:01:45.849 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:45.076+0000 7f193bd26700 1 -- 172.21.15.46:0/3583558325 shutdown_connections 2024-03-31T23:01:45.849 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:45.076+0000 7f193bd26700 1 --2- 172.21.15.46:0/3583558325 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1934106ed0 0x7f19341072f0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:01:45.849 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:45.076+0000 7f193bd26700 1 -- 172.21.15.46:0/3583558325 >> 172.21.15.46:0/3583558325 conn(0x7f1934101730 msgr2=0x7f1934103b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:01:45.849 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:45.077+0000 7f193bd26700 1 -- 172.21.15.46:0/3583558325 shutdown_connections 2024-03-31T23:01:45.850 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:45.077+0000 7f193bd26700 1 -- 172.21.15.46:0/3583558325 wait complete. 2024-03-31T23:01:45.850 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:45.077+0000 7f193bd26700 1 Processor -- start 2024-03-31T23:01:45.850 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:45.077+0000 7f193bd26700 1 -- start start 2024-03-31T23:01:45.850 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:45.077+0000 7f193bd26700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1934106ed0 0x7f193419a910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:01:45.850 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:45.077+0000 7f193bd26700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f193410afa0 con 0x7f1934106ed0 2024-03-31T23:01:45.850 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:45.077+0000 7f1939ac2700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1934106ed0 0x7f193419a910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:01:45.850 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:45.077+0000 7f1939ac2700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1934106ed0 0x7f193419a910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:36376/0 (socket says 172.21.15.46:36376) 2024-03-31T23:01:45.850 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:45.077+0000 7f1939ac2700 1 -- 172.21.15.46:0/2550875750 learned_addr learned my addr 172.21.15.46:0/2550875750 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:01:45.850 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:45.078+0000 7f1939ac2700 1 -- 172.21.15.46:0/2550875750 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f193419ae50 con 0x7f1934106ed0 2024-03-31T23:01:45.850 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:45.078+0000 7f1939ac2700 1 --2- 172.21.15.46:0/2550875750 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1934106ed0 0x7f193419a910 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f1930011a50 tx=0x7f1930004290 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:01:45.851 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:45.078+0000 7f192affd700 1 -- 172.21.15.46:0/2550875750 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f19300039f0 con 0x7f1934106ed0 2024-03-31T23:01:45.851 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:45.078+0000 7f192affd700 1 -- 172.21.15.46:0/2550875750 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7f1930003b90 con 0x7f1934106ed0 2024-03-31T23:01:45.851 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:45.078+0000 7f192affd700 1 -- 172.21.15.46:0/2550875750 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f193001a600 con 0x7f1934106ed0 2024-03-31T23:01:45.851 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:45.078+0000 7f193bd26700 1 -- 172.21.15.46:0/2550875750 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f193419b0e0 con 0x7f1934106ed0 2024-03-31T23:01:45.851 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:45.078+0000 7f193bd26700 1 -- 172.21.15.46:0/2550875750 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f193419ddd0 con 0x7f1934106ed0 2024-03-31T23:01:45.851 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:45.079+0000 7f192affd700 1 -- 172.21.15.46:0/2550875750 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 8) ==== 46116+0+0 (secure 0 0 0) 0x7f1930024a40 con 0x7f1934106ed0 2024-03-31T23:01:45.851 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:45.079+0000 7f193bd26700 1 -- 172.21.15.46:0/2550875750 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1934063870 con 0x7f1934106ed0 2024-03-31T23:01:45.851 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:45.079+0000 7f192affd700 1 --2- 172.21.15.46:0/2550875750 >> [v2:172.21.15.46:6800/3681182647,v1:172.21.15.46:6801/3681182647] conn(0x7f1920038e50 0x7f192003b310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:01:45.851 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:45.079+0000 7f192affd700 1 -- 172.21.15.46:0/2550875750 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f193004d170 con 0x7f1934106ed0 2024-03-31T23:01:45.851 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:45.079+0000 7f19392c1700 1 --2- 172.21.15.46:0/2550875750 >> [v2:172.21.15.46:6800/3681182647,v1:172.21.15.46:6801/3681182647] conn(0x7f1920038e50 0x7f192003b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:01:45.851 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:45.079+0000 7f19392c1700 1 --2- 172.21.15.46:0/2550875750 >> [v2:172.21.15.46:6800/3681182647,v1:172.21.15.46:6801/3681182647] conn(0x7f1920038e50 0x7f192003b310 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f1924006fd0 tx=0x7f1924006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:01:45.851 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:45.084+0000 7f192affd700 1 -- 172.21.15.46:0/2550875750 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f193002d770 con 0x7f1934106ed0 2024-03-31T23:01:45.852 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:45.288+0000 7f193bd26700 1 -- 172.21.15.46:0/2550875750 --> [v2:172.21.15.46:6800/3681182647,v1:172.21.15.46:6801/3681182647] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) -- 0x7f19341028c0 con 0x7f1920038e50 2024-03-31T23:01:45.852 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:45.289+0000 7f192affd700 1 -- 172.21.15.46:0/2550875750 <== mgr.14120 v2:172.21.15.46:6800/3681182647 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+16 (secure 0 0 0) 0x7f19341028c0 con 0x7f1920038e50 2024-03-31T23:01:45.852 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:45.292+0000 7f193bd26700 1 -- 172.21.15.46:0/2550875750 >> [v2:172.21.15.46:6800/3681182647,v1:172.21.15.46:6801/3681182647] conn(0x7f1920038e50 msgr2=0x7f192003b310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:01:45.852 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:45.292+0000 7f193bd26700 1 --2- 172.21.15.46:0/2550875750 >> [v2:172.21.15.46:6800/3681182647,v1:172.21.15.46:6801/3681182647] conn(0x7f1920038e50 0x7f192003b310 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f1924006fd0 tx=0x7f1924006e40 comp rx=0 tx=0).stop 2024-03-31T23:01:45.852 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:45.292+0000 7f193bd26700 1 -- 172.21.15.46:0/2550875750 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1934106ed0 msgr2=0x7f193419a910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:01:45.852 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:45.292+0000 7f193bd26700 1 --2- 172.21.15.46:0/2550875750 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1934106ed0 0x7f193419a910 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f1930011a50 tx=0x7f1930004290 comp rx=0 tx=0).stop 2024-03-31T23:01:45.852 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:45.292+0000 7f193bd26700 1 -- 172.21.15.46:0/2550875750 shutdown_connections 2024-03-31T23:01:45.852 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:45.292+0000 7f193bd26700 1 --2- 172.21.15.46:0/2550875750 >> [v2:172.21.15.46:6800/3681182647,v1:172.21.15.46:6801/3681182647] conn(0x7f1920038e50 0x7f192003b310 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:01:45.852 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:45.292+0000 7f193bd26700 1 --2- 172.21.15.46:0/2550875750 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1934106ed0 0x7f193419a910 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:01:45.852 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:45.292+0000 7f193bd26700 1 -- 172.21.15.46:0/2550875750 >> 172.21.15.46:0/2550875750 conn(0x7f1934101730 msgr2=0x7f19341021b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:01:45.852 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:45.292+0000 7f193bd26700 1 -- 172.21.15.46:0/2550875750 shutdown_connections 2024-03-31T23:01:45.852 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:45.292+0000 7f193bd26700 1 -- 172.21.15.46:0/2550875750 wait complete. 2024-03-31T23:01:45.852 INFO:teuthology.orchestra.run.smithi046.stdout:Generating ssh key... 2024-03-31T23:01:46.225 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:45 smithi046 ceph-mon[32498]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-03-31T23:01:46.854 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 31 23:01:46 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mgr-x[32727]: Generating public/private rsa key pair. 2024-03-31T23:01:46.854 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 31 23:01:46 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mgr-x[32727]: Your identification has been saved in /tmp/tmpw8kd0fc5/key. 2024-03-31T23:01:46.854 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 31 23:01:46 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mgr-x[32727]: Your public key has been saved in /tmp/tmpw8kd0fc5/key.pub. 2024-03-31T23:01:46.854 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 31 23:01:46 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mgr-x[32727]: The key fingerprint is: 2024-03-31T23:01:46.854 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 31 23:01:46 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mgr-x[32727]: SHA256:mTJ0kecjG1ty9z3Hj/+rnBCKoFt2WZ+vYT8JIDH+8Mg ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4 2024-03-31T23:01:46.854 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 31 23:01:46 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mgr-x[32727]: The key's randomart image is: 2024-03-31T23:01:46.854 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 31 23:01:46 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mgr-x[32727]: +---[RSA 3072]----+ 2024-03-31T23:01:46.855 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 31 23:01:46 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mgr-x[32727]: | .. | 2024-03-31T23:01:46.855 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 31 23:01:46 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mgr-x[32727]: | o... | 2024-03-31T23:01:46.855 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 31 23:01:46 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mgr-x[32727]: | o +o | 2024-03-31T23:01:46.855 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 31 23:01:46 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mgr-x[32727]: | . =++= . | 2024-03-31T23:01:46.855 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 31 23:01:46 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mgr-x[32727]: | .+ SX.+ . o | 2024-03-31T23:01:46.855 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 31 23:01:46 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mgr-x[32727]: | . .E*oo.o ..+| 2024-03-31T23:01:46.855 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 31 23:01:46 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mgr-x[32727]: | . o + . *. ..+| 2024-03-31T23:01:46.855 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 31 23:01:46 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mgr-x[32727]: | + . . *oo .| 2024-03-31T23:01:46.855 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 31 23:01:46 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mgr-x[32727]: | . ..*oo=| 2024-03-31T23:01:46.855 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 31 23:01:46 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mgr-x[32727]: +----[SHA256]-----+ 2024-03-31T23:01:47.062 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:46.256+0000 7f10414d9700 1 Processor -- start 2024-03-31T23:01:47.062 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:46.256+0000 7f10414d9700 1 -- start start 2024-03-31T23:01:47.062 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:46.256+0000 7f10414d9700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f103c106350 0x7f103c106770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:01:47.062 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:46.256+0000 7f10414d9700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f103c07bdc0 con 0x7f103c106350 2024-03-31T23:01:47.062 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:46.257+0000 7f103affd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f103c106350 0x7f103c106770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:01:47.062 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:46.257+0000 7f103affd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f103c106350 0x7f103c106770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:36378/0 (socket says 172.21.15.46:36378) 2024-03-31T23:01:47.062 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:46.257+0000 7f103affd700 1 -- 172.21.15.46:0/3609513287 learned_addr learned my addr 172.21.15.46:0/3609513287 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:01:47.062 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:46.257+0000 7f103affd700 1 -- 172.21.15.46:0/3609513287 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f103c106cb0 con 0x7f103c106350 2024-03-31T23:01:47.062 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:46.257+0000 7f103affd700 1 --2- 172.21.15.46:0/3609513287 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f103c106350 0x7f103c106770 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f10340098a0 tx=0x7f1034009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1102b765cea28896 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:01:47.063 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:46.257+0000 7f1039ffb700 1 -- 172.21.15.46:0/3609513287 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1034004160 con 0x7f103c106350 2024-03-31T23:01:47.063 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:46.257+0000 7f1039ffb700 1 -- 172.21.15.46:0/3609513287 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7f1034004300 con 0x7f103c106350 2024-03-31T23:01:47.063 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:46.257+0000 7f1039ffb700 1 -- 172.21.15.46:0/3609513287 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1034011a80 con 0x7f103c106350 2024-03-31T23:01:47.063 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:46.258+0000 7f10414d9700 1 -- 172.21.15.46:0/3609513287 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f103c106350 msgr2=0x7f103c106770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:01:47.064 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:46.258+0000 7f10414d9700 1 --2- 172.21.15.46:0/3609513287 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f103c106350 0x7f103c106770 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f10340098a0 tx=0x7f1034009bb0 comp rx=0 tx=0).stop 2024-03-31T23:01:47.064 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:46.258+0000 7f10414d9700 1 -- 172.21.15.46:0/3609513287 shutdown_connections 2024-03-31T23:01:47.064 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:46.258+0000 7f10414d9700 1 --2- 172.21.15.46:0/3609513287 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f103c106350 0x7f103c106770 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:01:47.064 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:46.258+0000 7f10414d9700 1 -- 172.21.15.46:0/3609513287 >> 172.21.15.46:0/3609513287 conn(0x7f103c101ef0 msgr2=0x7f103c104350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:01:47.064 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:46.258+0000 7f10414d9700 1 -- 172.21.15.46:0/3609513287 shutdown_connections 2024-03-31T23:01:47.064 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:46.258+0000 7f10414d9700 1 -- 172.21.15.46:0/3609513287 wait complete. 2024-03-31T23:01:47.064 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:46.259+0000 7f10414d9700 1 Processor -- start 2024-03-31T23:01:47.064 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:46.259+0000 7f10414d9700 1 -- start start 2024-03-31T23:01:47.064 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:46.259+0000 7f10414d9700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f103c106350 0x7f103c19dba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:01:47.064 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:46.259+0000 7f10414d9700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f103c107ce0 con 0x7f103c106350 2024-03-31T23:01:47.064 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:46.259+0000 7f103affd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f103c106350 0x7f103c19dba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:01:47.064 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:46.259+0000 7f103affd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f103c106350 0x7f103c19dba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:36394/0 (socket says 172.21.15.46:36394) 2024-03-31T23:01:47.064 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:46.259+0000 7f103affd700 1 -- 172.21.15.46:0/2461114761 learned_addr learned my addr 172.21.15.46:0/2461114761 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:01:47.064 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:46.259+0000 7f103affd700 1 -- 172.21.15.46:0/2461114761 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f103c19e0e0 con 0x7f103c106350 2024-03-31T23:01:47.065 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:46.259+0000 7f103affd700 1 --2- 172.21.15.46:0/2461114761 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f103c106350 0x7f103c19dba0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f1034011a50 tx=0x7f1034011b30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:01:47.065 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:46.260+0000 7f1033fff700 1 -- 172.21.15.46:0/2461114761 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1034003a80 con 0x7f103c106350 2024-03-31T23:01:47.065 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:46.260+0000 7f1033fff700 1 -- 172.21.15.46:0/2461114761 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7f1034003c20 con 0x7f103c106350 2024-03-31T23:01:47.065 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:46.260+0000 7f10414d9700 1 -- 172.21.15.46:0/2461114761 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f103c19e370 con 0x7f103c106350 2024-03-31T23:01:47.065 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:46.260+0000 7f1033fff700 1 -- 172.21.15.46:0/2461114761 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f103401a650 con 0x7f103c106350 2024-03-31T23:01:47.065 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:46.260+0000 7f10414d9700 1 -- 172.21.15.46:0/2461114761 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f103c1a1060 con 0x7f103c106350 2024-03-31T23:01:47.065 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:46.261+0000 7f1033fff700 1 -- 172.21.15.46:0/2461114761 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 8) ==== 46116+0+0 (secure 0 0 0) 0x7f103401a950 con 0x7f103c106350 2024-03-31T23:01:47.065 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:46.261+0000 7f10414d9700 1 -- 172.21.15.46:0/2461114761 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f103c063870 con 0x7f103c106350 2024-03-31T23:01:47.065 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:46.261+0000 7f1033fff700 1 --2- 172.21.15.46:0/2461114761 >> [v2:172.21.15.46:6800/3681182647,v1:172.21.15.46:6801/3681182647] conn(0x7f1028038e00 0x7f102803b2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:01:47.065 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:46.261+0000 7f1033fff700 1 -- 172.21.15.46:0/2461114761 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f103404d210 con 0x7f103c106350 2024-03-31T23:01:47.065 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:46.261+0000 7f103a7fc700 1 --2- 172.21.15.46:0/2461114761 >> [v2:172.21.15.46:6800/3681182647,v1:172.21.15.46:6801/3681182647] conn(0x7f1028038e00 0x7f102803b2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:01:47.065 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:46.261+0000 7f103a7fc700 1 --2- 172.21.15.46:0/2461114761 >> [v2:172.21.15.46:6800/3681182647,v1:172.21.15.46:6801/3681182647] conn(0x7f1028038e00 0x7f102803b2c0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f1024006fd0 tx=0x7f1024006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:01:47.065 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:46.266+0000 7f1033fff700 1 -- 172.21.15.46:0/2461114761 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f1034016dc0 con 0x7f103c106350 2024-03-31T23:01:47.066 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:46.472+0000 7f10414d9700 1 -- 172.21.15.46:0/2461114761 --> [v2:172.21.15.46:6800/3681182647,v1:172.21.15.46:6801/3681182647] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) -- 0x7f103c0008d0 con 0x7f1028038e00 2024-03-31T23:01:47.066 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:46.569+0000 7f1033fff700 1 -- 172.21.15.46:0/2461114761 <== mgr.14120 v2:172.21.15.46:6800/3681182647 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f103c0008d0 con 0x7f1028038e00 2024-03-31T23:01:47.066 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:46.571+0000 7f10414d9700 1 -- 172.21.15.46:0/2461114761 >> [v2:172.21.15.46:6800/3681182647,v1:172.21.15.46:6801/3681182647] conn(0x7f1028038e00 msgr2=0x7f102803b2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:01:47.066 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:46.571+0000 7f10414d9700 1 --2- 172.21.15.46:0/2461114761 >> [v2:172.21.15.46:6800/3681182647,v1:172.21.15.46:6801/3681182647] conn(0x7f1028038e00 0x7f102803b2c0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f1024006fd0 tx=0x7f1024006e40 comp rx=0 tx=0).stop 2024-03-31T23:01:47.066 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:46.571+0000 7f10414d9700 1 -- 172.21.15.46:0/2461114761 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f103c106350 msgr2=0x7f103c19dba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:01:47.066 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:46.571+0000 7f10414d9700 1 --2- 172.21.15.46:0/2461114761 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f103c106350 0x7f103c19dba0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f1034011a50 tx=0x7f1034011b30 comp rx=0 tx=0).stop 2024-03-31T23:01:47.066 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:46.571+0000 7f10414d9700 1 -- 172.21.15.46:0/2461114761 shutdown_connections 2024-03-31T23:01:47.066 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:46.571+0000 7f10414d9700 1 --2- 172.21.15.46:0/2461114761 >> [v2:172.21.15.46:6800/3681182647,v1:172.21.15.46:6801/3681182647] conn(0x7f1028038e00 0x7f102803b2c0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:01:47.066 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:46.571+0000 7f10414d9700 1 --2- 172.21.15.46:0/2461114761 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f103c106350 0x7f103c19dba0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:01:47.066 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:46.571+0000 7f10414d9700 1 -- 172.21.15.46:0/2461114761 >> 172.21.15.46:0/2461114761 conn(0x7f103c101ef0 msgr2=0x7f103c194810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:01:47.066 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:46.571+0000 7f10414d9700 1 -- 172.21.15.46:0/2461114761 shutdown_connections 2024-03-31T23:01:47.066 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:46.571+0000 7f10414d9700 1 -- 172.21.15.46:0/2461114761 wait complete. 2024-03-31T23:01:47.723 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:47 smithi046 ceph-mon[32498]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-03-31T23:01:47.724 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:47 smithi046 ceph-mon[32498]: Generating ssh key... 2024-03-31T23:01:47.724 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:47 smithi046 ceph-mon[32498]: from='mgr.14120 172.21.15.46:0/1993682382' entity='mgr.x' 2024-03-31T23:01:47.724 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:47 smithi046 ceph-mon[32498]: from='mgr.14120 172.21.15.46:0/1993682382' entity='mgr.x' 2024-03-31T23:01:48.286 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCzkmkRKY8+8KOyWRDL0P4BaTstsGt0MM+rdgXSbH1jnOhELHHP3TiD3LoNsEHEEYqgsDtg7o2JhZYYQ4/b0y2krNB4uA/677oQBAbbejHzCdiBrGiDT/YO9+EYdMQJcBNB/fBn28xUJos7GAa8c28wumU0lGg6VxBjJ8nrFDEKMeaLamKUEc2UtL+KUxTqNPrljzc9ow0oOrY2j2WuGzsRbfBzAWPhHdj9LHT6bjUeVTEdoxpvwG6K3sBxqponiCgixvUf0D8P/yZoiUiybDq/7Pb2lIKSuuqqR1jaaZerjJKk4uW4OTN2Bxl3i2bwzINOtSCxTMwvuijSVlr7o88kEoyOX4YnRDb+ftXYAqLbECwrrQSv27ltcXg4mXHLfNyTJe3l4uwm0Awlur01Z7tjtw37BtuEcS2eAy76aVF2Da+QZb1It27c25YfKPKr4IcyXl8xjD+XR4SlbZ/mek8cTREagI9+fQmcrP8WRsdJ+7N2pP1qbePYZWD2xfJHUec= ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4 2024-03-31T23:01:48.286 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:47.472+0000 7fed297d1700 1 Processor -- start 2024-03-31T23:01:48.286 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:47.472+0000 7fed297d1700 1 -- start start 2024-03-31T23:01:48.286 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:47.472+0000 7fed297d1700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fed2407dda0 0x7fed2407c250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:01:48.286 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:47.472+0000 7fed297d1700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fed2407c790 con 0x7fed2407dda0 2024-03-31T23:01:48.286 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:47.472+0000 7fed22ffd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fed2407dda0 0x7fed2407c250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:01:48.286 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:47.472+0000 7fed22ffd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fed2407dda0 0x7fed2407c250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:36404/0 (socket says 172.21.15.46:36404) 2024-03-31T23:01:48.286 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:47.472+0000 7fed22ffd700 1 -- 172.21.15.46:0/2363199799 learned_addr learned my addr 172.21.15.46:0/2363199799 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:01:48.286 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:47.472+0000 7fed22ffd700 1 -- 172.21.15.46:0/2363199799 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed2407c910 con 0x7fed2407dda0 2024-03-31T23:01:48.287 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:47.473+0000 7fed22ffd700 1 --2- 172.21.15.46:0/2363199799 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fed2407dda0 0x7fed2407c250 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fed140098a0 tx=0x7fed14009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a1487f4eb81f7d0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:01:48.287 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:47.473+0000 7fed21ffb700 1 -- 172.21.15.46:0/2363199799 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fed14004160 con 0x7fed2407dda0 2024-03-31T23:01:48.287 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:47.473+0000 7fed21ffb700 1 -- 172.21.15.46:0/2363199799 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7fed14004300 con 0x7fed2407dda0 2024-03-31T23:01:48.287 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:47.473+0000 7fed21ffb700 1 -- 172.21.15.46:0/2363199799 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fed14011a80 con 0x7fed2407dda0 2024-03-31T23:01:48.287 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:47.473+0000 7fed297d1700 1 -- 172.21.15.46:0/2363199799 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fed2407dda0 msgr2=0x7fed2407c250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:01:48.287 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:47.473+0000 7fed297d1700 1 --2- 172.21.15.46:0/2363199799 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fed2407dda0 0x7fed2407c250 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fed140098a0 tx=0x7fed14009bb0 comp rx=0 tx=0).stop 2024-03-31T23:01:48.287 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:47.473+0000 7fed297d1700 1 -- 172.21.15.46:0/2363199799 shutdown_connections 2024-03-31T23:01:48.287 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:47.473+0000 7fed297d1700 1 --2- 172.21.15.46:0/2363199799 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fed2407dda0 0x7fed2407c250 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:01:48.287 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:47.473+0000 7fed297d1700 1 -- 172.21.15.46:0/2363199799 >> 172.21.15.46:0/2363199799 conn(0x7fed24103030 msgr2=0x7fed241054b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:01:48.287 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:47.473+0000 7fed297d1700 1 -- 172.21.15.46:0/2363199799 shutdown_connections 2024-03-31T23:01:48.287 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:47.474+0000 7fed297d1700 1 -- 172.21.15.46:0/2363199799 wait complete. 2024-03-31T23:01:48.287 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:47.474+0000 7fed297d1700 1 Processor -- start 2024-03-31T23:01:48.287 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:47.474+0000 7fed297d1700 1 -- start start 2024-03-31T23:01:48.287 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:47.474+0000 7fed297d1700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fed2407dda0 0x7fed2419dbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:01:48.288 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:47.474+0000 7fed297d1700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fed2410e060 con 0x7fed2407dda0 2024-03-31T23:01:48.288 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:47.474+0000 7fed22ffd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fed2407dda0 0x7fed2419dbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:01:48.288 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:47.474+0000 7fed22ffd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fed2407dda0 0x7fed2419dbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:36412/0 (socket says 172.21.15.46:36412) 2024-03-31T23:01:48.288 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:47.474+0000 7fed22ffd700 1 -- 172.21.15.46:0/645724942 learned_addr learned my addr 172.21.15.46:0/645724942 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:01:48.288 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:47.475+0000 7fed22ffd700 1 -- 172.21.15.46:0/645724942 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed2419e120 con 0x7fed2407dda0 2024-03-31T23:01:48.288 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:47.475+0000 7fed22ffd700 1 --2- 172.21.15.46:0/645724942 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fed2407dda0 0x7fed2419dbe0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fed14006af0 tx=0x7fed14011b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:01:48.288 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:47.475+0000 7fed0bfff700 1 -- 172.21.15.46:0/645724942 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fed14003930 con 0x7fed2407dda0 2024-03-31T23:01:48.288 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:47.475+0000 7fed297d1700 1 -- 172.21.15.46:0/645724942 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fed2419e3b0 con 0x7fed2407dda0 2024-03-31T23:01:48.288 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:47.475+0000 7fed0bfff700 1 -- 172.21.15.46:0/645724942 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7fed14003ad0 con 0x7fed2407dda0 2024-03-31T23:01:48.288 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:47.475+0000 7fed0bfff700 1 -- 172.21.15.46:0/645724942 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fed140195e0 con 0x7fed2407dda0 2024-03-31T23:01:48.288 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:47.475+0000 7fed297d1700 1 -- 172.21.15.46:0/645724942 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fed241a10a0 con 0x7fed2407dda0 2024-03-31T23:01:48.288 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:47.476+0000 7fed0bfff700 1 -- 172.21.15.46:0/645724942 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 8) ==== 46116+0+0 (secure 0 0 0) 0x7fed140198e0 con 0x7fed2407dda0 2024-03-31T23:01:48.288 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:47.476+0000 7fed297d1700 1 -- 172.21.15.46:0/645724942 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fed24063870 con 0x7fed2407dda0 2024-03-31T23:01:48.288 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:47.476+0000 7fed0bfff700 1 --2- 172.21.15.46:0/645724942 >> [v2:172.21.15.46:6800/3681182647,v1:172.21.15.46:6801/3681182647] conn(0x7fed0c038db0 0x7fed0c03b270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:01:48.289 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:47.476+0000 7fed0bfff700 1 -- 172.21.15.46:0/645724942 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fed1404d2e0 con 0x7fed2407dda0 2024-03-31T23:01:48.289 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:47.476+0000 7fed227fc700 1 --2- 172.21.15.46:0/645724942 >> [v2:172.21.15.46:6800/3681182647,v1:172.21.15.46:6801/3681182647] conn(0x7fed0c038db0 0x7fed0c03b270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:01:48.289 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:47.477+0000 7fed227fc700 1 --2- 172.21.15.46:0/645724942 >> [v2:172.21.15.46:6800/3681182647,v1:172.21.15.46:6801/3681182647] conn(0x7fed0c038db0 0x7fed0c03b270 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fed18006fd0 tx=0x7fed18006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:01:48.289 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:47.481+0000 7fed0bfff700 1 -- 172.21.15.46:0/645724942 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fed14020240 con 0x7fed2407dda0 2024-03-31T23:01:48.289 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:47.722+0000 7fed297d1700 1 -- 172.21.15.46:0/645724942 --> [v2:172.21.15.46:6800/3681182647,v1:172.21.15.46:6801/3681182647] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) -- 0x7fed241042e0 con 0x7fed0c038db0 2024-03-31T23:01:48.289 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:47.723+0000 7fed0bfff700 1 -- 172.21.15.46:0/645724942 <== mgr.14120 v2:172.21.15.46:6800/3681182647 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+595 (secure 0 0 0) 0x7fed241042e0 con 0x7fed0c038db0 2024-03-31T23:01:48.289 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:47.725+0000 7fed297d1700 1 -- 172.21.15.46:0/645724942 >> [v2:172.21.15.46:6800/3681182647,v1:172.21.15.46:6801/3681182647] conn(0x7fed0c038db0 msgr2=0x7fed0c03b270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:01:48.289 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:47.725+0000 7fed297d1700 1 --2- 172.21.15.46:0/645724942 >> [v2:172.21.15.46:6800/3681182647,v1:172.21.15.46:6801/3681182647] conn(0x7fed0c038db0 0x7fed0c03b270 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fed18006fd0 tx=0x7fed18006e40 comp rx=0 tx=0).stop 2024-03-31T23:01:48.289 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:47.725+0000 7fed297d1700 1 -- 172.21.15.46:0/645724942 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fed2407dda0 msgr2=0x7fed2419dbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:01:48.289 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:47.725+0000 7fed297d1700 1 --2- 172.21.15.46:0/645724942 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fed2407dda0 0x7fed2419dbe0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fed14006af0 tx=0x7fed14011b70 comp rx=0 tx=0).stop 2024-03-31T23:01:48.289 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:47.725+0000 7fed297d1700 1 -- 172.21.15.46:0/645724942 shutdown_connections 2024-03-31T23:01:48.289 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:47.725+0000 7fed297d1700 1 --2- 172.21.15.46:0/645724942 >> [v2:172.21.15.46:6800/3681182647,v1:172.21.15.46:6801/3681182647] conn(0x7fed0c038db0 0x7fed0c03b270 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:01:48.289 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:47.725+0000 7fed297d1700 1 --2- 172.21.15.46:0/645724942 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fed2407dda0 0x7fed2419dbe0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:01:48.289 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:47.725+0000 7fed297d1700 1 -- 172.21.15.46:0/645724942 >> 172.21.15.46:0/645724942 conn(0x7fed24103030 msgr2=0x7fed24103bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:01:48.290 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:47.725+0000 7fed297d1700 1 -- 172.21.15.46:0/645724942 shutdown_connections 2024-03-31T23:01:48.290 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:47.725+0000 7fed297d1700 1 -- 172.21.15.46:0/645724942 wait complete. 2024-03-31T23:01:48.290 INFO:teuthology.orchestra.run.smithi046.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-03-31T23:01:48.290 INFO:teuthology.orchestra.run.smithi046.stdout:Adding key to root@localhost authorized_keys... 2024-03-31T23:01:48.290 INFO:teuthology.orchestra.run.smithi046.stdout:Adding host smithi046... 2024-03-31T23:01:48.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:48 smithi046 ceph-mon[32498]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-03-31T23:01:49.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:49 smithi046 ceph-mon[32498]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi046", "addr": "172.21.15.46", "target": ["mon-mgr", ""]}]: dispatch 2024-03-31T23:01:50.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:50 smithi046 ceph-mon[32498]: Deploying cephadm binary to smithi046 2024-03-31T23:01:53.793 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout Added host 'smithi046' with addr '172.21.15.46' 2024-03-31T23:01:53.793 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:48.687+0000 7f4c00275700 1 Processor -- start 2024-03-31T23:01:53.793 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:48.687+0000 7f4c00275700 1 -- start start 2024-03-31T23:01:53.793 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:48.687+0000 7f4c00275700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4bf8107b00 0x7f4bf8107f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:01:53.793 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:48.687+0000 7f4c00275700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f4bf81084f0 con 0x7f4bf8107b00 2024-03-31T23:01:53.793 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:48.688+0000 7f4bfe011700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4bf8107b00 0x7f4bf8107f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:01:53.793 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:48.688+0000 7f4bfe011700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4bf8107b00 0x7f4bf8107f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:36424/0 (socket says 172.21.15.46:36424) 2024-03-31T23:01:53.794 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:48.688+0000 7f4bfe011700 1 -- 172.21.15.46:0/1964062639 learned_addr learned my addr 172.21.15.46:0/1964062639 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:01:53.794 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:48.688+0000 7f4bfe011700 1 -- 172.21.15.46:0/1964062639 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4bf8108d00 con 0x7f4bf8107b00 2024-03-31T23:01:53.794 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:48.688+0000 7f4bfe011700 1 --2- 172.21.15.46:0/1964062639 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4bf8107b00 0x7f4bf8107f20 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f4bf40098a0 tx=0x7f4bf4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bdb5cf075f817e74 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:01:53.794 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:48.688+0000 7f4bfd00f700 1 -- 172.21.15.46:0/1964062639 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4bf4004160 con 0x7f4bf8107b00 2024-03-31T23:01:53.794 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:48.688+0000 7f4bfd00f700 1 -- 172.21.15.46:0/1964062639 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7f4bf4004300 con 0x7f4bf8107b00 2024-03-31T23:01:53.794 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:48.688+0000 7f4bfd00f700 1 -- 172.21.15.46:0/1964062639 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4bf4011a80 con 0x7f4bf8107b00 2024-03-31T23:01:53.794 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:48.689+0000 7f4c00275700 1 -- 172.21.15.46:0/1964062639 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4bf8107b00 msgr2=0x7f4bf8107f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:01:53.794 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:48.689+0000 7f4c00275700 1 --2- 172.21.15.46:0/1964062639 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4bf8107b00 0x7f4bf8107f20 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f4bf40098a0 tx=0x7f4bf4009bb0 comp rx=0 tx=0).stop 2024-03-31T23:01:53.794 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:48.689+0000 7f4c00275700 1 -- 172.21.15.46:0/1964062639 shutdown_connections 2024-03-31T23:01:53.794 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:48.689+0000 7f4c00275700 1 --2- 172.21.15.46:0/1964062639 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4bf8107b00 0x7f4bf8107f20 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:01:53.794 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:48.689+0000 7f4c00275700 1 -- 172.21.15.46:0/1964062639 >> 172.21.15.46:0/1964062639 conn(0x7f4bf81030a0 msgr2=0x7f4bf81054e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:01:53.795 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:48.689+0000 7f4c00275700 1 -- 172.21.15.46:0/1964062639 shutdown_connections 2024-03-31T23:01:53.795 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:48.689+0000 7f4c00275700 1 -- 172.21.15.46:0/1964062639 wait complete. 2024-03-31T23:01:53.795 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:48.690+0000 7f4c00275700 1 Processor -- start 2024-03-31T23:01:53.795 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:48.690+0000 7f4c00275700 1 -- start start 2024-03-31T23:01:53.795 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:48.690+0000 7f4c00275700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4bf8107b00 0x7f4bf807db00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:01:53.795 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:48.690+0000 7f4c00275700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f4bf810bf30 con 0x7f4bf8107b00 2024-03-31T23:01:53.795 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:48.690+0000 7f4bfe011700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4bf8107b00 0x7f4bf807db00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:01:53.795 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:48.690+0000 7f4bfe011700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4bf8107b00 0x7f4bf807db00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:36432/0 (socket says 172.21.15.46:36432) 2024-03-31T23:01:53.795 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:48.690+0000 7f4bfe011700 1 -- 172.21.15.46:0/1806385268 learned_addr learned my addr 172.21.15.46:0/1806385268 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:01:53.795 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:48.690+0000 7f4bfe011700 1 -- 172.21.15.46:0/1806385268 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4bf807e040 con 0x7f4bf8107b00 2024-03-31T23:01:53.795 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:48.690+0000 7f4bfe011700 1 --2- 172.21.15.46:0/1806385268 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4bf8107b00 0x7f4bf807db00 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f4bf4000c00 tx=0x7f4bf4004290 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:01:53.795 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:48.690+0000 7f4bef7fe700 1 -- 172.21.15.46:0/1806385268 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4bf40039c0 con 0x7f4bf8107b00 2024-03-31T23:01:53.795 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:48.690+0000 7f4bef7fe700 1 -- 172.21.15.46:0/1806385268 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7f4bf4003b60 con 0x7f4bf8107b00 2024-03-31T23:01:53.795 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:48.691+0000 7f4bef7fe700 1 -- 172.21.15.46:0/1806385268 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4bf401a6f0 con 0x7f4bf8107b00 2024-03-31T23:01:53.796 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:48.691+0000 7f4c00275700 1 -- 172.21.15.46:0/1806385268 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4bf807c230 con 0x7f4bf8107b00 2024-03-31T23:01:53.796 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:48.691+0000 7f4c00275700 1 -- 172.21.15.46:0/1806385268 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4bf807c770 con 0x7f4bf8107b00 2024-03-31T23:01:53.796 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:48.691+0000 7f4bef7fe700 1 -- 172.21.15.46:0/1806385268 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 8) ==== 46116+0+0 (secure 0 0 0) 0x7f4bf4024a40 con 0x7f4bf8107b00 2024-03-31T23:01:53.796 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:48.692+0000 7f4bef7fe700 1 --2- 172.21.15.46:0/1806385268 >> [v2:172.21.15.46:6800/3681182647,v1:172.21.15.46:6801/3681182647] conn(0x7f4be4038e50 0x7f4be403b310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:01:53.796 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:48.692+0000 7f4c00275700 1 -- 172.21.15.46:0/1806385268 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4bf8063870 con 0x7f4bf8107b00 2024-03-31T23:01:53.796 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:48.692+0000 7f4bef7fe700 1 -- 172.21.15.46:0/1806385268 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f4bf404d000 con 0x7f4bf8107b00 2024-03-31T23:01:53.796 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:48.692+0000 7f4bfd810700 1 --2- 172.21.15.46:0/1806385268 >> [v2:172.21.15.46:6800/3681182647,v1:172.21.15.46:6801/3681182647] conn(0x7f4be4038e50 0x7f4be403b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:01:53.796 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:48.692+0000 7f4bfd810700 1 --2- 172.21.15.46:0/1806385268 >> [v2:172.21.15.46:6800/3681182647,v1:172.21.15.46:6801/3681182647] conn(0x7f4be4038e50 0x7f4be403b310 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f4be8006fd0 tx=0x7f4be8006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:01:53.796 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:48.697+0000 7f4bef7fe700 1 -- 172.21.15.46:0/1806385268 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f4bf400e260 con 0x7f4bf8107b00 2024-03-31T23:01:53.797 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:48.906+0000 7f4c00275700 1 -- 172.21.15.46:0/1806385268 --> [v2:172.21.15.46:6800/3681182647,v1:172.21.15.46:6801/3681182647] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi046", "addr": "172.21.15.46", "target": ["mon-mgr", ""]}) -- 0x7f4bf81042f0 con 0x7f4be4038e50 2024-03-31T23:01:53.797 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:53.220+0000 7f4bef7fe700 1 -- 172.21.15.46:0/1806385268 <== mgr.14120 v2:172.21.15.46:6800/3681182647 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+48 (secure 0 0 0) 0x7f4bf81042f0 con 0x7f4be4038e50 2024-03-31T23:01:53.797 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:53.223+0000 7f4c00275700 1 -- 172.21.15.46:0/1806385268 >> [v2:172.21.15.46:6800/3681182647,v1:172.21.15.46:6801/3681182647] conn(0x7f4be4038e50 msgr2=0x7f4be403b310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:01:53.797 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:53.223+0000 7f4c00275700 1 --2- 172.21.15.46:0/1806385268 >> [v2:172.21.15.46:6800/3681182647,v1:172.21.15.46:6801/3681182647] conn(0x7f4be4038e50 0x7f4be403b310 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f4be8006fd0 tx=0x7f4be8006e40 comp rx=0 tx=0).stop 2024-03-31T23:01:53.797 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:53.223+0000 7f4c00275700 1 -- 172.21.15.46:0/1806385268 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4bf8107b00 msgr2=0x7f4bf807db00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:01:53.797 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:53.223+0000 7f4c00275700 1 --2- 172.21.15.46:0/1806385268 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4bf8107b00 0x7f4bf807db00 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f4bf4000c00 tx=0x7f4bf4004290 comp rx=0 tx=0).stop 2024-03-31T23:01:53.797 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:53.223+0000 7f4c00275700 1 -- 172.21.15.46:0/1806385268 shutdown_connections 2024-03-31T23:01:53.797 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:53.223+0000 7f4c00275700 1 --2- 172.21.15.46:0/1806385268 >> [v2:172.21.15.46:6800/3681182647,v1:172.21.15.46:6801/3681182647] conn(0x7f4be4038e50 0x7f4be403b310 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:01:53.797 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:53.223+0000 7f4c00275700 1 --2- 172.21.15.46:0/1806385268 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4bf8107b00 0x7f4bf807db00 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:01:53.797 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:53.223+0000 7f4c00275700 1 -- 172.21.15.46:0/1806385268 >> 172.21.15.46:0/1806385268 conn(0x7f4bf81030a0 msgr2=0x7f4bf8103be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:01:53.797 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:53.223+0000 7f4c00275700 1 -- 172.21.15.46:0/1806385268 shutdown_connections 2024-03-31T23:01:53.797 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:53.223+0000 7f4c00275700 1 -- 172.21.15.46:0/1806385268 wait complete. 2024-03-31T23:01:53.798 INFO:teuthology.orchestra.run.smithi046.stdout:Deploying unmanaged mon service... 2024-03-31T23:01:54.432 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:54 smithi046 ceph-mon[32498]: from='mgr.14120 172.21.15.46:0/1993682382' entity='mgr.x' 2024-03-31T23:01:54.433 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:54 smithi046 ceph-mon[32498]: Added host smithi046 2024-03-31T23:01:54.433 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:54 smithi046 ceph-mon[32498]: from='mgr.14120 172.21.15.46:0/1993682382' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-31T23:01:55.061 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-03-31T23:01:55.061 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:54.216+0000 7fdb2c7e9700 1 Processor -- start 2024-03-31T23:01:55.061 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:54.216+0000 7fdb2c7e9700 1 -- start start 2024-03-31T23:01:55.061 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:54.216+0000 7fdb2c7e9700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdb2407cd20 0x7fdb2407b180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:01:55.061 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:54.216+0000 7fdb2c7e9700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fdb2407b750 con 0x7fdb2407cd20 2024-03-31T23:01:55.061 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:54.217+0000 7fdb2a585700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdb2407cd20 0x7fdb2407b180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:01:55.062 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:54.217+0000 7fdb2a585700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdb2407cd20 0x7fdb2407b180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:47304/0 (socket says 172.21.15.46:47304) 2024-03-31T23:01:55.062 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:54.217+0000 7fdb2a585700 1 -- 172.21.15.46:0/2173732293 learned_addr learned my addr 172.21.15.46:0/2173732293 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:01:55.062 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:54.217+0000 7fdb2a585700 1 -- 172.21.15.46:0/2173732293 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb2407b8d0 con 0x7fdb2407cd20 2024-03-31T23:01:55.062 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:54.217+0000 7fdb2a585700 1 --2- 172.21.15.46:0/2173732293 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdb2407cd20 0x7fdb2407b180 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fdb1c009e80 tx=0x7fdb1c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d5bd235391deccc server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:01:55.062 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:54.217+0000 7fdb29583700 1 -- 172.21.15.46:0/2173732293 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fdb1c017070 con 0x7fdb2407cd20 2024-03-31T23:01:55.062 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:54.217+0000 7fdb29583700 1 -- 172.21.15.46:0/2173732293 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7fdb1c00b8c0 con 0x7fdb2407cd20 2024-03-31T23:01:55.062 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:54.217+0000 7fdb29583700 1 -- 172.21.15.46:0/2173732293 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fdb1c004520 con 0x7fdb2407cd20 2024-03-31T23:01:55.062 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:54.218+0000 7fdb2c7e9700 1 -- 172.21.15.46:0/2173732293 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdb2407cd20 msgr2=0x7fdb2407b180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:01:55.062 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:54.218+0000 7fdb2c7e9700 1 --2- 172.21.15.46:0/2173732293 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdb2407cd20 0x7fdb2407b180 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fdb1c009e80 tx=0x7fdb1c00b330 comp rx=0 tx=0).stop 2024-03-31T23:01:55.062 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:54.218+0000 7fdb2c7e9700 1 -- 172.21.15.46:0/2173732293 shutdown_connections 2024-03-31T23:01:55.062 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:54.218+0000 7fdb2c7e9700 1 --2- 172.21.15.46:0/2173732293 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdb2407cd20 0x7fdb2407b180 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:01:55.062 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:54.218+0000 7fdb2c7e9700 1 -- 172.21.15.46:0/2173732293 >> 172.21.15.46:0/2173732293 conn(0x7fdb24103010 msgr2=0x7fdb24105470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:01:55.062 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:54.218+0000 7fdb2c7e9700 1 -- 172.21.15.46:0/2173732293 shutdown_connections 2024-03-31T23:01:55.063 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:54.218+0000 7fdb2c7e9700 1 -- 172.21.15.46:0/2173732293 wait complete. 2024-03-31T23:01:55.063 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:54.219+0000 7fdb2c7e9700 1 Processor -- start 2024-03-31T23:01:55.063 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:54.219+0000 7fdb2c7e9700 1 -- start start 2024-03-31T23:01:55.063 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:54.219+0000 7fdb2c7e9700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdb2407cd20 0x7fdb2419dbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:01:55.063 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:54.219+0000 7fdb2c7e9700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fdb2410e040 con 0x7fdb2407cd20 2024-03-31T23:01:55.063 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:54.219+0000 7fdb2a585700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdb2407cd20 0x7fdb2419dbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:01:55.063 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:54.219+0000 7fdb2a585700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdb2407cd20 0x7fdb2419dbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:47318/0 (socket says 172.21.15.46:47318) 2024-03-31T23:01:55.063 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:54.219+0000 7fdb2a585700 1 -- 172.21.15.46:0/2901674338 learned_addr learned my addr 172.21.15.46:0/2901674338 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:01:55.063 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:54.220+0000 7fdb2a585700 1 -- 172.21.15.46:0/2901674338 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb2419e100 con 0x7fdb2407cd20 2024-03-31T23:01:55.063 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:54.220+0000 7fdb2a585700 1 --2- 172.21.15.46:0/2901674338 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdb2407cd20 0x7fdb2419dbc0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fdb1c003f60 tx=0x7fdb1c004670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:01:55.063 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:54.221+0000 7fdb237fe700 1 -- 172.21.15.46:0/2901674338 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fdb1c020020 con 0x7fdb2407cd20 2024-03-31T23:01:55.063 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:54.221+0000 7fdb237fe700 1 -- 172.21.15.46:0/2901674338 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7fdb1c00bcb0 con 0x7fdb2407cd20 2024-03-31T23:01:55.063 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:54.221+0000 7fdb2c7e9700 1 -- 172.21.15.46:0/2901674338 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdb2419e390 con 0x7fdb2407cd20 2024-03-31T23:01:55.063 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:54.221+0000 7fdb237fe700 1 -- 172.21.15.46:0/2901674338 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fdb1c017040 con 0x7fdb2407cd20 2024-03-31T23:01:55.064 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:54.221+0000 7fdb2c7e9700 1 -- 172.21.15.46:0/2901674338 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdb241a1080 con 0x7fdb2407cd20 2024-03-31T23:01:55.064 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:54.222+0000 7fdb237fe700 1 -- 172.21.15.46:0/2901674338 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 8) ==== 46116+0+0 (secure 0 0 0) 0x7fdb1c023020 con 0x7fdb2407cd20 2024-03-31T23:01:55.064 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:54.222+0000 7fdb2c7e9700 1 -- 172.21.15.46:0/2901674338 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdb24063870 con 0x7fdb2407cd20 2024-03-31T23:01:55.064 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:54.222+0000 7fdb237fe700 1 --2- 172.21.15.46:0/2901674338 >> [v2:172.21.15.46:6800/3681182647,v1:172.21.15.46:6801/3681182647] conn(0x7fdb0c038e50 0x7fdb0c03b310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:01:55.064 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:54.222+0000 7fdb237fe700 1 -- 172.21.15.46:0/2901674338 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fdb1c00f050 con 0x7fdb2407cd20 2024-03-31T23:01:55.064 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:54.222+0000 7fdb29d84700 1 --2- 172.21.15.46:0/2901674338 >> [v2:172.21.15.46:6800/3681182647,v1:172.21.15.46:6801/3681182647] conn(0x7fdb0c038e50 0x7fdb0c03b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:01:55.064 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:54.222+0000 7fdb29d84700 1 --2- 172.21.15.46:0/2901674338 >> [v2:172.21.15.46:6800/3681182647,v1:172.21.15.46:6801/3681182647] conn(0x7fdb0c038e50 0x7fdb0c03b310 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fdb18006fd0 tx=0x7fdb18006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:01:55.064 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:54.227+0000 7fdb237fe700 1 -- 172.21.15.46:0/2901674338 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fdb1c02a020 con 0x7fdb2407cd20 2024-03-31T23:01:55.064 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:54.444+0000 7fdb2c7e9700 1 -- 172.21.15.46:0/2901674338 --> [v2:172.21.15.46:6800/3681182647,v1:172.21.15.46:6801/3681182647] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}) -- 0x7fdb24104230 con 0x7fdb0c038e50 2024-03-31T23:01:55.064 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:54.449+0000 7fdb237fe700 1 -- 172.21.15.46:0/2901674338 <== mgr.14120 v2:172.21.15.46:6800/3681182647 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7fdb24104230 con 0x7fdb0c038e50 2024-03-31T23:01:55.064 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:54.451+0000 7fdb2c7e9700 1 -- 172.21.15.46:0/2901674338 >> [v2:172.21.15.46:6800/3681182647,v1:172.21.15.46:6801/3681182647] conn(0x7fdb0c038e50 msgr2=0x7fdb0c03b310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:01:55.064 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:54.451+0000 7fdb2c7e9700 1 --2- 172.21.15.46:0/2901674338 >> [v2:172.21.15.46:6800/3681182647,v1:172.21.15.46:6801/3681182647] conn(0x7fdb0c038e50 0x7fdb0c03b310 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fdb18006fd0 tx=0x7fdb18006e40 comp rx=0 tx=0).stop 2024-03-31T23:01:55.064 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:54.451+0000 7fdb2c7e9700 1 -- 172.21.15.46:0/2901674338 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdb2407cd20 msgr2=0x7fdb2419dbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:01:55.065 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:54.451+0000 7fdb2c7e9700 1 --2- 172.21.15.46:0/2901674338 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdb2407cd20 0x7fdb2419dbc0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fdb1c003f60 tx=0x7fdb1c004670 comp rx=0 tx=0).stop 2024-03-31T23:01:55.065 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:54.451+0000 7fdb2c7e9700 1 -- 172.21.15.46:0/2901674338 shutdown_connections 2024-03-31T23:01:55.065 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:54.451+0000 7fdb2c7e9700 1 --2- 172.21.15.46:0/2901674338 >> [v2:172.21.15.46:6800/3681182647,v1:172.21.15.46:6801/3681182647] conn(0x7fdb0c038e50 0x7fdb0c03b310 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:01:55.065 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:54.451+0000 7fdb2c7e9700 1 --2- 172.21.15.46:0/2901674338 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdb2407cd20 0x7fdb2419dbc0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:01:55.065 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:54.451+0000 7fdb2c7e9700 1 -- 172.21.15.46:0/2901674338 >> 172.21.15.46:0/2901674338 conn(0x7fdb24103010 msgr2=0x7fdb24103b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:01:55.065 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:54.451+0000 7fdb2c7e9700 1 -- 172.21.15.46:0/2901674338 shutdown_connections 2024-03-31T23:01:55.065 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:54.451+0000 7fdb2c7e9700 1 -- 172.21.15.46:0/2901674338 wait complete. 2024-03-31T23:01:55.065 INFO:teuthology.orchestra.run.smithi046.stdout:Deploying unmanaged mgr service... 2024-03-31T23:01:55.695 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:55 smithi046 ceph-mon[32498]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-31T23:01:55.695 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:55 smithi046 ceph-mon[32498]: Saving service mon spec with placement count:5 2024-03-31T23:01:55.695 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:55 smithi046 ceph-mon[32498]: from='mgr.14120 172.21.15.46:0/1993682382' entity='mgr.x' 2024-03-31T23:01:56.921 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-03-31T23:01:56.921 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:55.495+0000 7f92c18d9700 1 Processor -- start 2024-03-31T23:01:56.921 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:55.495+0000 7f92c18d9700 1 -- start start 2024-03-31T23:01:56.922 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:55.495+0000 7f92c18d9700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f92bc107af0 0x7f92bc107f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:01:56.922 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:55.495+0000 7f92c18d9700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f92bc1084e0 con 0x7f92bc107af0 2024-03-31T23:01:56.922 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:55.495+0000 7f92baffd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f92bc107af0 0x7f92bc107f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:01:56.922 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:55.495+0000 7f92baffd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f92bc107af0 0x7f92bc107f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:47330/0 (socket says 172.21.15.46:47330) 2024-03-31T23:01:56.922 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:55.495+0000 7f92baffd700 1 -- 172.21.15.46:0/1569064681 learned_addr learned my addr 172.21.15.46:0/1569064681 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:01:56.922 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:55.495+0000 7f92baffd700 1 -- 172.21.15.46:0/1569064681 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f92bc108ca0 con 0x7f92bc107af0 2024-03-31T23:01:56.922 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:55.495+0000 7f92baffd700 1 --2- 172.21.15.46:0/1569064681 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f92bc107af0 0x7f92bc107f10 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f92ac0098a0 tx=0x7f92ac009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d5649f2a2ca1f95 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:01:56.922 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:55.496+0000 7f92b9ffb700 1 -- 172.21.15.46:0/1569064681 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f92ac004160 con 0x7f92bc107af0 2024-03-31T23:01:56.922 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:55.496+0000 7f92b9ffb700 1 -- 172.21.15.46:0/1569064681 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7f92ac004300 con 0x7f92bc107af0 2024-03-31T23:01:56.922 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:55.496+0000 7f92b9ffb700 1 -- 172.21.15.46:0/1569064681 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f92ac011a80 con 0x7f92bc107af0 2024-03-31T23:01:56.922 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:55.496+0000 7f92c18d9700 1 -- 172.21.15.46:0/1569064681 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f92bc107af0 msgr2=0x7f92bc107f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:01:56.922 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:55.496+0000 7f92c18d9700 1 --2- 172.21.15.46:0/1569064681 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f92bc107af0 0x7f92bc107f10 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f92ac0098a0 tx=0x7f92ac009bb0 comp rx=0 tx=0).stop 2024-03-31T23:01:56.922 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:55.496+0000 7f92c18d9700 1 -- 172.21.15.46:0/1569064681 shutdown_connections 2024-03-31T23:01:56.923 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:55.496+0000 7f92c18d9700 1 --2- 172.21.15.46:0/1569064681 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f92bc107af0 0x7f92bc107f10 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:01:56.923 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:55.496+0000 7f92c18d9700 1 -- 172.21.15.46:0/1569064681 >> 172.21.15.46:0/1569064681 conn(0x7f92bc103090 msgr2=0x7f92bc1054d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:01:56.923 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:55.496+0000 7f92c18d9700 1 -- 172.21.15.46:0/1569064681 shutdown_connections 2024-03-31T23:01:56.923 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:55.496+0000 7f92c18d9700 1 -- 172.21.15.46:0/1569064681 wait complete. 2024-03-31T23:01:56.923 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:55.497+0000 7f92c18d9700 1 Processor -- start 2024-03-31T23:01:56.923 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:55.497+0000 7f92c18d9700 1 -- start start 2024-03-31T23:01:56.923 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:55.497+0000 7f92c18d9700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f92bc107af0 0x7f92bc19b9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:01:56.923 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:55.497+0000 7f92c18d9700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f92bc10c050 con 0x7f92bc107af0 2024-03-31T23:01:56.923 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:55.497+0000 7f92baffd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f92bc107af0 0x7f92bc19b9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:01:56.923 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:55.497+0000 7f92baffd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f92bc107af0 0x7f92bc19b9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:47338/0 (socket says 172.21.15.46:47338) 2024-03-31T23:01:56.923 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:55.497+0000 7f92baffd700 1 -- 172.21.15.46:0/1117107011 learned_addr learned my addr 172.21.15.46:0/1117107011 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:01:56.923 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:55.498+0000 7f92baffd700 1 -- 172.21.15.46:0/1117107011 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f92bc19bf30 con 0x7f92bc107af0 2024-03-31T23:01:56.923 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:55.498+0000 7f92baffd700 1 --2- 172.21.15.46:0/1117107011 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f92bc107af0 0x7f92bc19b9f0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f92ac006af0 tx=0x7f92ac011b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:01:56.923 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:55.498+0000 7f92c08d7700 1 -- 172.21.15.46:0/1117107011 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f92ac003930 con 0x7f92bc107af0 2024-03-31T23:01:56.923 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:55.498+0000 7f92c18d9700 1 -- 172.21.15.46:0/1117107011 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f92bc19c1c0 con 0x7f92bc107af0 2024-03-31T23:01:56.924 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:55.498+0000 7f92c08d7700 1 -- 172.21.15.46:0/1117107011 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7f92ac003ad0 con 0x7f92bc107af0 2024-03-31T23:01:56.924 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:55.498+0000 7f92c08d7700 1 -- 172.21.15.46:0/1117107011 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f92ac0195e0 con 0x7f92bc107af0 2024-03-31T23:01:56.924 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:55.498+0000 7f92c18d9700 1 -- 172.21.15.46:0/1117107011 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f92bc19eeb0 con 0x7f92bc107af0 2024-03-31T23:01:56.924 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:55.499+0000 7f92c08d7700 1 -- 172.21.15.46:0/1117107011 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 8) ==== 46116+0+0 (secure 0 0 0) 0x7f92ac0198e0 con 0x7f92bc107af0 2024-03-31T23:01:56.924 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:55.499+0000 7f92c18d9700 1 -- 172.21.15.46:0/1117107011 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f92bc063870 con 0x7f92bc107af0 2024-03-31T23:01:56.924 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:55.499+0000 7f92c08d7700 1 --2- 172.21.15.46:0/1117107011 >> [v2:172.21.15.46:6800/3681182647,v1:172.21.15.46:6801/3681182647] conn(0x7f92a4038db0 0x7f92a403b270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:01:56.924 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:55.499+0000 7f92c08d7700 1 -- 172.21.15.46:0/1117107011 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f92ac04d3a0 con 0x7f92bc107af0 2024-03-31T23:01:56.924 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:55.499+0000 7f92ba7fc700 1 --2- 172.21.15.46:0/1117107011 >> [v2:172.21.15.46:6800/3681182647,v1:172.21.15.46:6801/3681182647] conn(0x7f92a4038db0 0x7f92a403b270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:01:56.924 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:55.500+0000 7f92ba7fc700 1 --2- 172.21.15.46:0/1117107011 >> [v2:172.21.15.46:6800/3681182647,v1:172.21.15.46:6801/3681182647] conn(0x7f92a4038db0 0x7f92a403b270 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f92b0006fd0 tx=0x7f92b0006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:01:56.924 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:55.504+0000 7f92c08d7700 1 -- 172.21.15.46:0/1117107011 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f92ac019cd0 con 0x7f92bc107af0 2024-03-31T23:01:56.925 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:55.723+0000 7f92c18d9700 1 -- 172.21.15.46:0/1117107011 --> [v2:172.21.15.46:6800/3681182647,v1:172.21.15.46:6801/3681182647] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}) -- 0x7f92bc1042e0 con 0x7f92a4038db0 2024-03-31T23:01:56.925 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:55.728+0000 7f92c08d7700 1 -- 172.21.15.46:0/1117107011 <== mgr.14120 v2:172.21.15.46:6800/3681182647 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f92bc1042e0 con 0x7f92a4038db0 2024-03-31T23:01:56.925 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:55.730+0000 7f92c18d9700 1 -- 172.21.15.46:0/1117107011 >> [v2:172.21.15.46:6800/3681182647,v1:172.21.15.46:6801/3681182647] conn(0x7f92a4038db0 msgr2=0x7f92a403b270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:01:56.925 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:55.730+0000 7f92c18d9700 1 --2- 172.21.15.46:0/1117107011 >> [v2:172.21.15.46:6800/3681182647,v1:172.21.15.46:6801/3681182647] conn(0x7f92a4038db0 0x7f92a403b270 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f92b0006fd0 tx=0x7f92b0006e40 comp rx=0 tx=0).stop 2024-03-31T23:01:56.925 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:55.730+0000 7f92c18d9700 1 -- 172.21.15.46:0/1117107011 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f92bc107af0 msgr2=0x7f92bc19b9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:01:56.925 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:55.730+0000 7f92c18d9700 1 --2- 172.21.15.46:0/1117107011 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f92bc107af0 0x7f92bc19b9f0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f92ac006af0 tx=0x7f92ac011b70 comp rx=0 tx=0).stop 2024-03-31T23:01:56.925 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:55.730+0000 7f92c18d9700 1 -- 172.21.15.46:0/1117107011 shutdown_connections 2024-03-31T23:01:56.925 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:55.730+0000 7f92c18d9700 1 --2- 172.21.15.46:0/1117107011 >> [v2:172.21.15.46:6800/3681182647,v1:172.21.15.46:6801/3681182647] conn(0x7f92a4038db0 0x7f92a403b270 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:01:56.925 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:55.730+0000 7f92c18d9700 1 --2- 172.21.15.46:0/1117107011 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f92bc107af0 0x7f92bc19b9f0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:01:56.925 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:55.730+0000 7f92c18d9700 1 -- 172.21.15.46:0/1117107011 >> 172.21.15.46:0/1117107011 conn(0x7f92bc103090 msgr2=0x7f92bc103bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:01:56.925 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:55.730+0000 7f92c18d9700 1 -- 172.21.15.46:0/1117107011 shutdown_connections 2024-03-31T23:01:56.925 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:55.730+0000 7f92c18d9700 1 -- 172.21.15.46:0/1117107011 wait complete. 2024-03-31T23:01:57.040 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:56 smithi046 ceph-mon[32498]: from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-31T23:01:57.040 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:56 smithi046 ceph-mon[32498]: Saving service mgr spec with placement count:2 2024-03-31T23:01:57.040 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:56 smithi046 ceph-mon[32498]: from='mgr.14120 172.21.15.46:0/1993682382' entity='mgr.x' 2024-03-31T23:01:57.040 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:56 smithi046 ceph-mon[32498]: from='mgr.14120 172.21.15.46:0/1993682382' entity='mgr.x' 2024-03-31T23:01:58.355 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:57 smithi046 ceph-mon[32498]: from='mgr.14120 172.21.15.46:0/1993682382' entity='mgr.x' 2024-03-31T23:01:58.355 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:57 smithi046 ceph-mon[32498]: from='client.? 172.21.15.46:0/1210440809' entity='client.admin' 2024-03-31T23:01:58.362 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:57.338+0000 7fde97409700 1 Processor -- start 2024-03-31T23:01:58.362 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:57.338+0000 7fde97409700 1 -- start start 2024-03-31T23:01:58.362 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:57.339+0000 7fde97409700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fde90108540 0x7fde9010a930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:01:58.362 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:57.339+0000 7fde97409700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fde9007bde0 con 0x7fde90108540 2024-03-31T23:01:58.362 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:57.339+0000 7fde951a5700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fde90108540 0x7fde9010a930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:01:58.362 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:57.339+0000 7fde951a5700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fde90108540 0x7fde9010a930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:47350/0 (socket says 172.21.15.46:47350) 2024-03-31T23:01:58.363 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:57.339+0000 7fde951a5700 1 -- 172.21.15.46:0/1538290806 learned_addr learned my addr 172.21.15.46:0/1538290806 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:01:58.363 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:57.339+0000 7fde951a5700 1 -- 172.21.15.46:0/1538290806 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde9010ae70 con 0x7fde90108540 2024-03-31T23:01:58.363 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:57.339+0000 7fde951a5700 1 --2- 172.21.15.46:0/1538290806 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fde90108540 0x7fde9010a930 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fde8c0098a0 tx=0x7fde8c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b5fb5b34ba57442e server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:01:58.363 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:57.339+0000 7fde83fff700 1 -- 172.21.15.46:0/1538290806 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fde8c004160 con 0x7fde90108540 2024-03-31T23:01:58.363 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:57.339+0000 7fde83fff700 1 -- 172.21.15.46:0/1538290806 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fde8c004300 con 0x7fde90108540 2024-03-31T23:01:58.363 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:57.339+0000 7fde83fff700 1 -- 172.21.15.46:0/1538290806 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fde8c011a50 con 0x7fde90108540 2024-03-31T23:01:58.363 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:57.340+0000 7fde97409700 1 -- 172.21.15.46:0/1538290806 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fde90108540 msgr2=0x7fde9010a930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:01:58.363 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:57.340+0000 7fde97409700 1 --2- 172.21.15.46:0/1538290806 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fde90108540 0x7fde9010a930 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fde8c0098a0 tx=0x7fde8c009bb0 comp rx=0 tx=0).stop 2024-03-31T23:01:58.363 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:57.340+0000 7fde97409700 1 -- 172.21.15.46:0/1538290806 shutdown_connections 2024-03-31T23:01:58.363 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:57.340+0000 7fde97409700 1 --2- 172.21.15.46:0/1538290806 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fde90108540 0x7fde9010a930 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:01:58.363 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:57.340+0000 7fde97409700 1 -- 172.21.15.46:0/1538290806 >> 172.21.15.46:0/1538290806 conn(0x7fde90101ef0 msgr2=0x7fde90104350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:01:58.363 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:57.340+0000 7fde97409700 1 -- 172.21.15.46:0/1538290806 shutdown_connections 2024-03-31T23:01:58.363 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:57.341+0000 7fde97409700 1 -- 172.21.15.46:0/1538290806 wait complete. 2024-03-31T23:01:58.364 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:57.341+0000 7fde97409700 1 Processor -- start 2024-03-31T23:01:58.364 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:57.341+0000 7fde97409700 1 -- start start 2024-03-31T23:01:58.364 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:57.341+0000 7fde97409700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fde90108540 0x7fde90107dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:01:58.364 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:57.341+0000 7fde97409700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fde9010be20 con 0x7fde90108540 2024-03-31T23:01:58.364 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:57.341+0000 7fde951a5700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fde90108540 0x7fde90107dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:01:58.364 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:57.341+0000 7fde951a5700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fde90108540 0x7fde90107dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:47360/0 (socket says 172.21.15.46:47360) 2024-03-31T23:01:58.364 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:57.341+0000 7fde951a5700 1 -- 172.21.15.46:0/1210440809 learned_addr learned my addr 172.21.15.46:0/1210440809 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:01:58.364 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:57.342+0000 7fde951a5700 1 -- 172.21.15.46:0/1210440809 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde90108300 con 0x7fde90108540 2024-03-31T23:01:58.364 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:57.342+0000 7fde951a5700 1 --2- 172.21.15.46:0/1210440809 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fde90108540 0x7fde90107dc0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fde8c011a20 tx=0x7fde8c011b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:01:58.364 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:57.342+0000 7fde827fc700 1 -- 172.21.15.46:0/1210440809 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fde8c003b70 con 0x7fde90108540 2024-03-31T23:01:58.364 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:57.342+0000 7fde827fc700 1 -- 172.21.15.46:0/1210440809 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fde8c01a8e0 con 0x7fde90108540 2024-03-31T23:01:58.364 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:57.342+0000 7fde97409700 1 -- 172.21.15.46:0/1210440809 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fde901064f0 con 0x7fde90108540 2024-03-31T23:01:58.364 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:57.342+0000 7fde827fc700 1 -- 172.21.15.46:0/1210440809 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fde8c024990 con 0x7fde90108540 2024-03-31T23:01:58.364 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:57.342+0000 7fde97409700 1 -- 172.21.15.46:0/1210440809 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fde90106a30 con 0x7fde90108540 2024-03-31T23:01:58.364 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:57.343+0000 7fde827fc700 1 -- 172.21.15.46:0/1210440809 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 8) ==== 46116+0+0 (secure 0 0 0) 0x7fde8c02c070 con 0x7fde90108540 2024-03-31T23:01:58.365 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:57.343+0000 7fde97409700 1 -- 172.21.15.46:0/1210440809 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fde90063870 con 0x7fde90108540 2024-03-31T23:01:58.365 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:57.343+0000 7fde827fc700 1 --2- 172.21.15.46:0/1210440809 >> [v2:172.21.15.46:6800/3681182647,v1:172.21.15.46:6801/3681182647] conn(0x7fde7c038db0 0x7fde7c03b270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:01:58.365 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:57.343+0000 7fde827fc700 1 -- 172.21.15.46:0/1210440809 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fde8c04d7b0 con 0x7fde90108540 2024-03-31T23:01:58.365 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:57.343+0000 7fde949a4700 1 --2- 172.21.15.46:0/1210440809 >> [v2:172.21.15.46:6800/3681182647,v1:172.21.15.46:6801/3681182647] conn(0x7fde7c038db0 0x7fde7c03b270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:01:58.365 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:57.344+0000 7fde949a4700 1 --2- 172.21.15.46:0/1210440809 >> [v2:172.21.15.46:6800/3681182647,v1:172.21.15.46:6801/3681182647] conn(0x7fde7c038db0 0x7fde7c03b270 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fde84006fd0 tx=0x7fde84006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:01:58.365 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:57.348+0000 7fde827fc700 1 -- 172.21.15.46:0/1210440809 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fde8c02cc60 con 0x7fde90108540 2024-03-31T23:01:58.365 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:57.559+0000 7fde97409700 1 -- 172.21.15.46:0/1210440809 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) -- 0x7fde901071a0 con 0x7fde90108540 2024-03-31T23:01:58.365 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:57.562+0000 7fde827fc700 1 -- 172.21.15.46:0/1210440809 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v7)=0 v7) ==== 142+0+0 (secure 0 0 0) 0x7fde8c0193c0 con 0x7fde90108540 2024-03-31T23:01:58.365 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:57.565+0000 7fde97409700 1 -- 172.21.15.46:0/1210440809 >> [v2:172.21.15.46:6800/3681182647,v1:172.21.15.46:6801/3681182647] conn(0x7fde7c038db0 msgr2=0x7fde7c03b270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:01:58.365 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:57.565+0000 7fde97409700 1 --2- 172.21.15.46:0/1210440809 >> [v2:172.21.15.46:6800/3681182647,v1:172.21.15.46:6801/3681182647] conn(0x7fde7c038db0 0x7fde7c03b270 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fde84006fd0 tx=0x7fde84006e40 comp rx=0 tx=0).stop 2024-03-31T23:01:58.365 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:57.565+0000 7fde97409700 1 -- 172.21.15.46:0/1210440809 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fde90108540 msgr2=0x7fde90107dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:01:58.365 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:57.565+0000 7fde97409700 1 --2- 172.21.15.46:0/1210440809 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fde90108540 0x7fde90107dc0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fde8c011a20 tx=0x7fde8c011b00 comp rx=0 tx=0).stop 2024-03-31T23:01:58.365 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:57.565+0000 7fde97409700 1 -- 172.21.15.46:0/1210440809 shutdown_connections 2024-03-31T23:01:58.366 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:57.565+0000 7fde97409700 1 --2- 172.21.15.46:0/1210440809 >> [v2:172.21.15.46:6800/3681182647,v1:172.21.15.46:6801/3681182647] conn(0x7fde7c038db0 0x7fde7c03b270 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:01:58.366 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:57.565+0000 7fde97409700 1 --2- 172.21.15.46:0/1210440809 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fde90108540 0x7fde90107dc0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:01:58.366 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:57.565+0000 7fde97409700 1 -- 172.21.15.46:0/1210440809 >> 172.21.15.46:0/1210440809 conn(0x7fde90101ef0 msgr2=0x7fde901029b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:01:58.366 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:57.565+0000 7fde97409700 1 -- 172.21.15.46:0/1210440809 shutdown_connections 2024-03-31T23:01:58.366 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:57.565+0000 7fde97409700 1 -- 172.21.15.46:0/1210440809 wait complete. 2024-03-31T23:02:00.014 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:59 smithi046 ceph-mon[32498]: from='mgr.14120 172.21.15.46:0/1993682382' entity='mgr.x' 2024-03-31T23:02:00.014 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:01:59 smithi046 ceph-mon[32498]: from='client.? 172.21.15.46:0/487913447' entity='client.admin' 2024-03-31T23:02:00.016 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:59.088+0000 7f33f5edc700 1 Processor -- start 2024-03-31T23:02:00.021 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:59.088+0000 7f33f5edc700 1 -- start start 2024-03-31T23:02:00.021 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:59.088+0000 7f33f5edc700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f33f0105d70 0x7f33f0106190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:02:00.021 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:59.088+0000 7f33f5edc700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f33f0106760 con 0x7f33f0105d70 2024-03-31T23:02:00.021 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:59.088+0000 7f33ef7fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f33f0105d70 0x7f33f0106190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:02:00.021 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:59.088+0000 7f33ef7fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f33f0105d70 0x7f33f0106190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:47362/0 (socket says 172.21.15.46:47362) 2024-03-31T23:02:00.021 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:59.088+0000 7f33ef7fe700 1 -- 172.21.15.46:0/738780144 learned_addr learned my addr 172.21.15.46:0/738780144 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:02:00.021 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:59.089+0000 7f33ef7fe700 1 -- 172.21.15.46:0/738780144 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f33f0108d00 con 0x7f33f0105d70 2024-03-31T23:02:00.021 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:59.089+0000 7f33ef7fe700 1 --2- 172.21.15.46:0/738780144 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f33f0105d70 0x7f33f0106190 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f33d80098a0 tx=0x7f33d8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=79d565e96ad7b736 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:02:00.021 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:59.089+0000 7f33ee7fc700 1 -- 172.21.15.46:0/738780144 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f33d8004160 con 0x7f33f0105d70 2024-03-31T23:02:00.021 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:59.089+0000 7f33ee7fc700 1 -- 172.21.15.46:0/738780144 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f33d8004300 con 0x7f33f0105d70 2024-03-31T23:02:00.022 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:59.089+0000 7f33ee7fc700 1 -- 172.21.15.46:0/738780144 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f33d8011a50 con 0x7f33f0105d70 2024-03-31T23:02:00.022 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:59.090+0000 7f33f5edc700 1 -- 172.21.15.46:0/738780144 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f33f0105d70 msgr2=0x7f33f0106190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:02:00.022 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:59.090+0000 7f33f5edc700 1 --2- 172.21.15.46:0/738780144 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f33f0105d70 0x7f33f0106190 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f33d80098a0 tx=0x7f33d8009bb0 comp rx=0 tx=0).stop 2024-03-31T23:02:00.022 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:59.090+0000 7f33f5edc700 1 -- 172.21.15.46:0/738780144 shutdown_connections 2024-03-31T23:02:00.022 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:59.090+0000 7f33f5edc700 1 --2- 172.21.15.46:0/738780144 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f33f0105d70 0x7f33f0106190 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:02:00.022 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:59.090+0000 7f33f5edc700 1 -- 172.21.15.46:0/738780144 >> 172.21.15.46:0/738780144 conn(0x7f33f0101720 msgr2=0x7f33f0103b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:02:00.022 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:59.090+0000 7f33f5edc700 1 -- 172.21.15.46:0/738780144 shutdown_connections 2024-03-31T23:02:00.022 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:59.090+0000 7f33f5edc700 1 -- 172.21.15.46:0/738780144 wait complete. 2024-03-31T23:02:00.022 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:59.090+0000 7f33f5edc700 1 Processor -- start 2024-03-31T23:02:00.022 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:59.091+0000 7f33f5edc700 1 -- start start 2024-03-31T23:02:00.022 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:59.091+0000 7f33f5edc700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f33f0105d70 0x7f33f019ba20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:02:00.022 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:59.091+0000 7f33f5edc700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f33f010c0b0 con 0x7f33f0105d70 2024-03-31T23:02:00.022 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:59.091+0000 7f33ef7fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f33f0105d70 0x7f33f019ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:02:00.023 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:59.091+0000 7f33ef7fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f33f0105d70 0x7f33f019ba20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:47378/0 (socket says 172.21.15.46:47378) 2024-03-31T23:02:00.023 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:59.091+0000 7f33ef7fe700 1 -- 172.21.15.46:0/487913447 learned_addr learned my addr 172.21.15.46:0/487913447 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:02:00.023 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:59.091+0000 7f33ef7fe700 1 -- 172.21.15.46:0/487913447 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f33f019bf60 con 0x7f33f0105d70 2024-03-31T23:02:00.023 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:59.091+0000 7f33ef7fe700 1 --2- 172.21.15.46:0/487913447 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f33f0105d70 0x7f33f019ba20 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f33d8011710 tx=0x7f33d8011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:02:00.023 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:59.092+0000 7f33ecff9700 1 -- 172.21.15.46:0/487913447 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f33d80039c0 con 0x7f33f0105d70 2024-03-31T23:02:00.023 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:59.092+0000 7f33f5edc700 1 -- 172.21.15.46:0/487913447 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f33f019c1f0 con 0x7f33f0105d70 2024-03-31T23:02:00.023 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:59.092+0000 7f33ecff9700 1 -- 172.21.15.46:0/487913447 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f33d8003b60 con 0x7f33f0105d70 2024-03-31T23:02:00.023 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:59.092+0000 7f33ecff9700 1 -- 172.21.15.46:0/487913447 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f33d801a690 con 0x7f33f0105d70 2024-03-31T23:02:00.023 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:59.092+0000 7f33f5edc700 1 -- 172.21.15.46:0/487913447 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f33f019eee0 con 0x7f33f0105d70 2024-03-31T23:02:00.023 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:59.093+0000 7f33f5edc700 1 -- 172.21.15.46:0/487913447 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f33f0063870 con 0x7f33f0105d70 2024-03-31T23:02:00.023 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:59.093+0000 7f33ecff9700 1 -- 172.21.15.46:0/487913447 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 8) ==== 46116+0+0 (secure 0 0 0) 0x7f33d8024430 con 0x7f33f0105d70 2024-03-31T23:02:00.023 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:59.093+0000 7f33ecff9700 1 --2- 172.21.15.46:0/487913447 >> [v2:172.21.15.46:6800/3681182647,v1:172.21.15.46:6801/3681182647] conn(0x7f33dc038db0 0x7f33dc03b270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:02:00.023 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:59.093+0000 7f33ecff9700 1 -- 172.21.15.46:0/487913447 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f33d804d360 con 0x7f33f0105d70 2024-03-31T23:02:00.024 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:59.093+0000 7f33eeffd700 1 --2- 172.21.15.46:0/487913447 >> [v2:172.21.15.46:6800/3681182647,v1:172.21.15.46:6801/3681182647] conn(0x7f33dc038db0 0x7f33dc03b270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:02:00.024 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:59.093+0000 7f33eeffd700 1 --2- 172.21.15.46:0/487913447 >> [v2:172.21.15.46:6800/3681182647,v1:172.21.15.46:6801/3681182647] conn(0x7f33dc038db0 0x7f33dc03b270 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f33e0006fd0 tx=0x7f33e0006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:02:00.024 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:59.098+0000 7f33ecff9700 1 -- 172.21.15.46:0/487913447 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f33d8024a70 con 0x7f33f0105d70 2024-03-31T23:02:00.024 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:59.302+0000 7f33f5edc700 1 -- 172.21.15.46:0/487913447 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) -- 0x7f33f019f0b0 con 0x7f33f0105d70 2024-03-31T23:02:00.024 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:59.305+0000 7f33ecff9700 1 -- 172.21.15.46:0/487913447 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v8)=0 v8) ==== 130+0+0 (secure 0 0 0) 0x7f33d8024720 con 0x7f33f0105d70 2024-03-31T23:02:00.024 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:59.307+0000 7f33f5edc700 1 -- 172.21.15.46:0/487913447 >> [v2:172.21.15.46:6800/3681182647,v1:172.21.15.46:6801/3681182647] conn(0x7f33dc038db0 msgr2=0x7f33dc03b270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:02:00.024 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:59.307+0000 7f33f5edc700 1 --2- 172.21.15.46:0/487913447 >> [v2:172.21.15.46:6800/3681182647,v1:172.21.15.46:6801/3681182647] conn(0x7f33dc038db0 0x7f33dc03b270 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f33e0006fd0 tx=0x7f33e0006e40 comp rx=0 tx=0).stop 2024-03-31T23:02:00.024 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:59.307+0000 7f33f5edc700 1 -- 172.21.15.46:0/487913447 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f33f0105d70 msgr2=0x7f33f019ba20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:02:00.024 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:59.307+0000 7f33f5edc700 1 --2- 172.21.15.46:0/487913447 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f33f0105d70 0x7f33f019ba20 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f33d8011710 tx=0x7f33d8011ad0 comp rx=0 tx=0).stop 2024-03-31T23:02:00.024 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:59.307+0000 7f33f5edc700 1 -- 172.21.15.46:0/487913447 shutdown_connections 2024-03-31T23:02:00.024 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:59.307+0000 7f33f5edc700 1 --2- 172.21.15.46:0/487913447 >> [v2:172.21.15.46:6800/3681182647,v1:172.21.15.46:6801/3681182647] conn(0x7f33dc038db0 0x7f33dc03b270 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:02:00.024 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:59.307+0000 7f33f5edc700 1 --2- 172.21.15.46:0/487913447 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f33f0105d70 0x7f33f019ba20 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:02:00.025 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:59.307+0000 7f33f5edc700 1 -- 172.21.15.46:0/487913447 >> 172.21.15.46:0/487913447 conn(0x7f33f0101720 msgr2=0x7f33f01021c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:02:00.025 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:59.307+0000 7f33f5edc700 1 -- 172.21.15.46:0/487913447 shutdown_connections 2024-03-31T23:02:00.025 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:01:59.307+0000 7f33f5edc700 1 -- 172.21.15.46:0/487913447 wait complete. 2024-03-31T23:02:00.025 INFO:teuthology.orchestra.run.smithi046.stdout:Enabling the dashboard module... 2024-03-31T23:02:01.611 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:02:01 smithi046 ceph-mon[32498]: from='mgr.14120 172.21.15.46:0/1993682382' entity='mgr.x' 2024-03-31T23:02:01.612 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:02:01 smithi046 ceph-mon[32498]: from='client.? 172.21.15.46:0/2421230013' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-03-31T23:02:01.612 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 31 23:02:01 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mgr-x[32727]: ignoring --setuser ceph since I am not root 2024-03-31T23:02:01.612 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 31 23:02:01 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mgr-x[32727]: ignoring --setgroup ceph since I am not root 2024-03-31T23:02:02.040 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:00.396+0000 7f78adfe1700 1 Processor -- start 2024-03-31T23:02:02.041 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:00.396+0000 7f78adfe1700 1 -- start start 2024-03-31T23:02:02.041 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:00.396+0000 7f78adfe1700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f78a8106310 0x7f78a8106730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:02:02.041 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:00.397+0000 7f78adfe1700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f78a807be00 con 0x7f78a8106310 2024-03-31T23:02:02.041 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:00.397+0000 7f78a77fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f78a8106310 0x7f78a8106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:02:02.041 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:00.397+0000 7f78a77fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f78a8106310 0x7f78a8106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:47380/0 (socket says 172.21.15.46:47380) 2024-03-31T23:02:02.041 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:00.397+0000 7f78a77fe700 1 -- 172.21.15.46:0/3962248099 learned_addr learned my addr 172.21.15.46:0/3962248099 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:02:02.041 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:00.397+0000 7f78a77fe700 1 -- 172.21.15.46:0/3962248099 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78a8106c70 con 0x7f78a8106310 2024-03-31T23:02:02.041 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:00.397+0000 7f78a77fe700 1 --2- 172.21.15.46:0/3962248099 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f78a8106310 0x7f78a8106730 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f789c0098a0 tx=0x7f789c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd19ccf0dd58666b server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:02:02.041 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:00.398+0000 7f78a67fc700 1 -- 172.21.15.46:0/3962248099 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f789c004160 con 0x7f78a8106310 2024-03-31T23:02:02.041 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:00.398+0000 7f78a67fc700 1 -- 172.21.15.46:0/3962248099 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f789c004300 con 0x7f78a8106310 2024-03-31T23:02:02.042 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:00.398+0000 7f78a67fc700 1 -- 172.21.15.46:0/3962248099 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f789c011a50 con 0x7f78a8106310 2024-03-31T23:02:02.042 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:00.398+0000 7f78adfe1700 1 -- 172.21.15.46:0/3962248099 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f78a8106310 msgr2=0x7f78a8106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:02:02.042 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:00.398+0000 7f78adfe1700 1 --2- 172.21.15.46:0/3962248099 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f78a8106310 0x7f78a8106730 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f789c0098a0 tx=0x7f789c009bb0 comp rx=0 tx=0).stop 2024-03-31T23:02:02.042 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:00.398+0000 7f78adfe1700 1 -- 172.21.15.46:0/3962248099 shutdown_connections 2024-03-31T23:02:02.042 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:00.398+0000 7f78adfe1700 1 --2- 172.21.15.46:0/3962248099 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f78a8106310 0x7f78a8106730 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:02:02.042 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:00.398+0000 7f78adfe1700 1 -- 172.21.15.46:0/3962248099 >> 172.21.15.46:0/3962248099 conn(0x7f78a8101ef0 msgr2=0x7f78a8104350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:02:02.042 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:00.398+0000 7f78adfe1700 1 -- 172.21.15.46:0/3962248099 shutdown_connections 2024-03-31T23:02:02.042 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:00.398+0000 7f78adfe1700 1 -- 172.21.15.46:0/3962248099 wait complete. 2024-03-31T23:02:02.042 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:00.399+0000 7f78adfe1700 1 Processor -- start 2024-03-31T23:02:02.042 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:00.399+0000 7f78adfe1700 1 -- start start 2024-03-31T23:02:02.042 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:00.399+0000 7f78adfe1700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f78a8106310 0x7f78a819db40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:02:02.042 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:00.399+0000 7f78adfe1700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f78a8107ce0 con 0x7f78a8106310 2024-03-31T23:02:02.042 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:00.400+0000 7f78a77fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f78a8106310 0x7f78a819db40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:02:02.043 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:00.400+0000 7f78a77fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f78a8106310 0x7f78a819db40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:47390/0 (socket says 172.21.15.46:47390) 2024-03-31T23:02:02.043 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:00.400+0000 7f78a77fe700 1 -- 172.21.15.46:0/2421230013 learned_addr learned my addr 172.21.15.46:0/2421230013 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:02:02.043 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:00.400+0000 7f78a77fe700 1 -- 172.21.15.46:0/2421230013 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78a819e080 con 0x7f78a8106310 2024-03-31T23:02:02.043 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:00.400+0000 7f78a77fe700 1 --2- 172.21.15.46:0/2421230013 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f78a8106310 0x7f78a819db40 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f789c011a20 tx=0x7f789c011b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:02:02.043 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:00.400+0000 7f78a4ff9700 1 -- 172.21.15.46:0/2421230013 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f789c003b70 con 0x7f78a8106310 2024-03-31T23:02:02.043 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:00.400+0000 7f78a4ff9700 1 -- 172.21.15.46:0/2421230013 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f789c01a8e0 con 0x7f78a8106310 2024-03-31T23:02:02.043 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:00.400+0000 7f78adfe1700 1 -- 172.21.15.46:0/2421230013 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f78a819e310 con 0x7f78a8106310 2024-03-31T23:02:02.043 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:00.400+0000 7f78a4ff9700 1 -- 172.21.15.46:0/2421230013 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f789c024990 con 0x7f78a8106310 2024-03-31T23:02:02.043 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:00.400+0000 7f78adfe1700 1 -- 172.21.15.46:0/2421230013 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f78a81a1000 con 0x7f78a8106310 2024-03-31T23:02:02.043 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:00.401+0000 7f78a4ff9700 1 -- 172.21.15.46:0/2421230013 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 8) ==== 46116+0+0 (secure 0 0 0) 0x7f789c022020 con 0x7f78a8106310 2024-03-31T23:02:02.043 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:00.401+0000 7f78adfe1700 1 -- 172.21.15.46:0/2421230013 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f78a8063870 con 0x7f78a8106310 2024-03-31T23:02:02.043 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:00.402+0000 7f78a4ff9700 1 --2- 172.21.15.46:0/2421230013 >> [v2:172.21.15.46:6800/3681182647,v1:172.21.15.46:6801/3681182647] conn(0x7f7890038e00 0x7f789003b2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:02:02.043 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:00.402+0000 7f78a4ff9700 1 -- 172.21.15.46:0/2421230013 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f789c04d700 con 0x7f78a8106310 2024-03-31T23:02:02.043 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:00.402+0000 7f78a6ffd700 1 --2- 172.21.15.46:0/2421230013 >> [v2:172.21.15.46:6800/3681182647,v1:172.21.15.46:6801/3681182647] conn(0x7f7890038e00 0x7f789003b2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:02:02.044 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:00.402+0000 7f78a6ffd700 1 --2- 172.21.15.46:0/2421230013 >> [v2:172.21.15.46:6800/3681182647,v1:172.21.15.46:6801/3681182647] conn(0x7f7890038e00 0x7f789003b2c0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f7898006fd0 tx=0x7f7898006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:02:02.044 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:00.407+0000 7f78a4ff9700 1 -- 172.21.15.46:0/2421230013 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f789c0172b0 con 0x7f78a8106310 2024-03-31T23:02:02.044 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:00.673+0000 7f78adfe1700 1 -- 172.21.15.46:0/2421230013 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) -- 0x7f78a802dd80 con 0x7f78a8106310 2024-03-31T23:02:02.044 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:01.446+0000 7f78a4ff9700 1 -- 172.21.15.46:0/2421230013 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v9) ==== 88+0+0 (secure 0 0 0) 0x7f789c00fdd0 con 0x7f78a8106310 2024-03-31T23:02:02.044 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:01.448+0000 7f78a4ff9700 1 -- 172.21.15.46:0/2421230013 <== mon.0 v2:172.21.15.46:3300/0 8 ==== mgrmap(e 9) ==== 46129+0+0 (secure 0 0 0) 0x7f789c02dc90 con 0x7f78a8106310 2024-03-31T23:02:02.044 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:01.450+0000 7f78adfe1700 1 -- 172.21.15.46:0/2421230013 >> [v2:172.21.15.46:6800/3681182647,v1:172.21.15.46:6801/3681182647] conn(0x7f7890038e00 msgr2=0x7f789003b2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:02:02.044 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:01.450+0000 7f78adfe1700 1 --2- 172.21.15.46:0/2421230013 >> [v2:172.21.15.46:6800/3681182647,v1:172.21.15.46:6801/3681182647] conn(0x7f7890038e00 0x7f789003b2c0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f7898006fd0 tx=0x7f7898006e40 comp rx=0 tx=0).stop 2024-03-31T23:02:02.044 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:01.450+0000 7f78adfe1700 1 -- 172.21.15.46:0/2421230013 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f78a8106310 msgr2=0x7f78a819db40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:02:02.044 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:01.450+0000 7f78adfe1700 1 --2- 172.21.15.46:0/2421230013 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f78a8106310 0x7f78a819db40 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f789c011a20 tx=0x7f789c011b00 comp rx=0 tx=0).stop 2024-03-31T23:02:02.044 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:01.450+0000 7f78adfe1700 1 -- 172.21.15.46:0/2421230013 shutdown_connections 2024-03-31T23:02:02.044 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:01.450+0000 7f78adfe1700 1 --2- 172.21.15.46:0/2421230013 >> [v2:172.21.15.46:6800/3681182647,v1:172.21.15.46:6801/3681182647] conn(0x7f7890038e00 0x7f789003b2c0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:02:02.044 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:01.450+0000 7f78adfe1700 1 --2- 172.21.15.46:0/2421230013 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f78a8106310 0x7f78a819db40 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:02:02.044 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:01.450+0000 7f78adfe1700 1 -- 172.21.15.46:0/2421230013 >> 172.21.15.46:0/2421230013 conn(0x7f78a8101ef0 msgr2=0x7f78a8194720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:02:02.044 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:01.450+0000 7f78adfe1700 1 -- 172.21.15.46:0/2421230013 shutdown_connections 2024-03-31T23:02:02.045 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:01.450+0000 7f78adfe1700 1 -- 172.21.15.46:0/2421230013 wait complete. 2024-03-31T23:02:02.111 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 31 23:02:02 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mgr-x[32727]: 2024-03-31T23:02:02.028+0000 7fbeaff39200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-31T23:02:02.381 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 31 23:02:02 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mgr-x[32727]: 2024-03-31T23:02:02.129+0000 7fbeaff39200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-31T23:02:02.685 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:02:02 smithi046 ceph-mon[32498]: from='client.? 172.21.15.46:0/2421230013' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-03-31T23:02:02.685 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:02:02 smithi046 ceph-mon[32498]: mgrmap e9: x(active, since 19s) 2024-03-31T23:02:03.043 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 31 23:02:02 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mgr-x[32727]: 2024-03-31T23:02:02.800+0000 7fbeaff39200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-31T23:02:03.043 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 31 23:02:02 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mgr-x[32727]: 2024-03-31T23:02:02.896+0000 7fbeaff39200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-31T23:02:03.043 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 31 23:02:03 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mgr-x[32727]: 2024-03-31T23:02:03.042+0000 7fbeaff39200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-31T23:02:03.361 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 31 23:02:03 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mgr-x[32727]: 2024-03-31T23:02:03.140+0000 7fbeaff39200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-31T23:02:03.361 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 31 23:02:03 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mgr-x[32727]: 2024-03-31T23:02:03.226+0000 7fbeaff39200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-31T23:02:03.440 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout { 2024-03-31T23:02:03.440 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-03-31T23:02:03.440 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-31T23:02:03.440 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-03-31T23:02:03.440 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-03-31T23:02:03.441 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout } 2024-03-31T23:02:03.441 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:02.478+0000 7fade01ee700 1 Processor -- start 2024-03-31T23:02:03.441 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:02.478+0000 7fade01ee700 1 -- start start 2024-03-31T23:02:03.441 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:02.478+0000 7fade01ee700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fadd807c2b0 0x7fadd807c6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:02:03.441 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:02.478+0000 7fade01ee700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fadd807dfe0 con 0x7fadd807c2b0 2024-03-31T23:02:03.441 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:02.478+0000 7fadddf8a700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fadd807c2b0 0x7fadd807c6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:02:03.441 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:02.478+0000 7fadddf8a700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fadd807c2b0 0x7fadd807c6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:49072/0 (socket says 172.21.15.46:49072) 2024-03-31T23:02:03.441 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:02.478+0000 7fadddf8a700 1 -- 172.21.15.46:0/357858220 learned_addr learned my addr 172.21.15.46:0/357858220 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:02:03.441 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:02.478+0000 7fadddf8a700 1 -- 172.21.15.46:0/357858220 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fadd807cc10 con 0x7fadd807c2b0 2024-03-31T23:02:03.441 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:02.479+0000 7fadddf8a700 1 --2- 172.21.15.46:0/357858220 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fadd807c2b0 0x7fadd807c6d0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fadd40098a0 tx=0x7fadd4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=980cdb1307d47400 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:02:03.441 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:02.479+0000 7faddcf88700 1 -- 172.21.15.46:0/357858220 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fadd4004160 con 0x7fadd807c2b0 2024-03-31T23:02:03.441 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:02.479+0000 7faddcf88700 1 -- 172.21.15.46:0/357858220 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fadd4004300 con 0x7fadd807c2b0 2024-03-31T23:02:03.441 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:02.479+0000 7faddcf88700 1 -- 172.21.15.46:0/357858220 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fadd4011a50 con 0x7fadd807c2b0 2024-03-31T23:02:03.441 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:02.479+0000 7fade01ee700 1 -- 172.21.15.46:0/357858220 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fadd807c2b0 msgr2=0x7fadd807c6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:02:03.442 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:02.479+0000 7fade01ee700 1 --2- 172.21.15.46:0/357858220 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fadd807c2b0 0x7fadd807c6d0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fadd40098a0 tx=0x7fadd4009bb0 comp rx=0 tx=0).stop 2024-03-31T23:02:03.442 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:02.480+0000 7fade01ee700 1 -- 172.21.15.46:0/357858220 shutdown_connections 2024-03-31T23:02:03.442 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:02.480+0000 7fade01ee700 1 --2- 172.21.15.46:0/357858220 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fadd807c2b0 0x7fadd807c6d0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:02:03.442 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:02.480+0000 7fade01ee700 1 -- 172.21.15.46:0/357858220 >> 172.21.15.46:0/357858220 conn(0x7fadd8103070 msgr2=0x7fadd8105490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:02:03.442 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:02.480+0000 7fade01ee700 1 -- 172.21.15.46:0/357858220 shutdown_connections 2024-03-31T23:02:03.442 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:02.480+0000 7fade01ee700 1 -- 172.21.15.46:0/357858220 wait complete. 2024-03-31T23:02:03.442 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:02.480+0000 7fade01ee700 1 Processor -- start 2024-03-31T23:02:03.442 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:02.480+0000 7fade01ee700 1 -- start start 2024-03-31T23:02:03.442 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:02.480+0000 7fade01ee700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fadd807c2b0 0x7fadd819dc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:02:03.442 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:02.480+0000 7fade01ee700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fadd810e150 con 0x7fadd807c2b0 2024-03-31T23:02:03.442 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:02.480+0000 7fadddf8a700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fadd807c2b0 0x7fadd819dc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:02:03.442 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:02.481+0000 7fadddf8a700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fadd807c2b0 0x7fadd819dc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:49074/0 (socket says 172.21.15.46:49074) 2024-03-31T23:02:03.442 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:02.481+0000 7fadddf8a700 1 -- 172.21.15.46:0/1766934912 learned_addr learned my addr 172.21.15.46:0/1766934912 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:02:03.442 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:02.481+0000 7fadddf8a700 1 -- 172.21.15.46:0/1766934912 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fadd819e1b0 con 0x7fadd807c2b0 2024-03-31T23:02:03.443 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:02.481+0000 7fadddf8a700 1 --2- 172.21.15.46:0/1766934912 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fadd807c2b0 0x7fadd819dc70 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fadd4011a20 tx=0x7fadd4011b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:02:03.443 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:02.481+0000 7fadceffd700 1 -- 172.21.15.46:0/1766934912 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fadd40039c0 con 0x7fadd807c2b0 2024-03-31T23:02:03.443 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:02.481+0000 7fadceffd700 1 -- 172.21.15.46:0/1766934912 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fadd4003b60 con 0x7fadd807c2b0 2024-03-31T23:02:03.443 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:02.481+0000 7fade01ee700 1 -- 172.21.15.46:0/1766934912 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fadd819e440 con 0x7fadd807c2b0 2024-03-31T23:02:03.443 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:02.481+0000 7fadceffd700 1 -- 172.21.15.46:0/1766934912 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fadd401a6b0 con 0x7fadd807c2b0 2024-03-31T23:02:03.443 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:02.481+0000 7fade01ee700 1 -- 172.21.15.46:0/1766934912 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fadd81a1130 con 0x7fadd807c2b0 2024-03-31T23:02:03.443 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:02.482+0000 7fadceffd700 1 -- 172.21.15.46:0/1766934912 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 9) ==== 46129+0+0 (secure 0 0 0) 0x7fadd401a9e0 con 0x7fadd807c2b0 2024-03-31T23:02:03.443 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:02.482+0000 7fade01ee700 1 -- 172.21.15.46:0/1766934912 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fadd8063870 con 0x7fadd807c2b0 2024-03-31T23:02:03.443 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:02.483+0000 7fadceffd700 1 --2- 172.21.15.46:0/1766934912 >> [v2:172.21.15.46:6800/3681182647,v1:172.21.15.46:6801/3681182647] conn(0x7fadc4038e60 0x7fadc403b320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:02:03.443 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:02.483+0000 7fadceffd700 1 -- 172.21.15.46:0/1766934912 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fadd404d580 con 0x7fadd807c2b0 2024-03-31T23:02:03.443 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:02.483+0000 7faddd789700 1 -- 172.21.15.46:0/1766934912 >> [v2:172.21.15.46:6800/3681182647,v1:172.21.15.46:6801/3681182647] conn(0x7fadc4038e60 msgr2=0x7fadc403b320 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.46:6800/3681182647 2024-03-31T23:02:03.443 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:02.483+0000 7faddd789700 1 --2- 172.21.15.46:0/1766934912 >> [v2:172.21.15.46:6800/3681182647,v1:172.21.15.46:6801/3681182647] conn(0x7fadc4038e60 0x7fadc403b320 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-31T23:02:03.443 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:02.487+0000 7fadceffd700 1 -- 172.21.15.46:0/1766934912 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fadd4024a70 con 0x7fadd807c2b0 2024-03-31T23:02:03.443 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:02.684+0000 7faddd789700 1 -- 172.21.15.46:0/1766934912 >> [v2:172.21.15.46:6800/3681182647,v1:172.21.15.46:6801/3681182647] conn(0x7fadc4038e60 msgr2=0x7fadc403b320 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.46:6800/3681182647 2024-03-31T23:02:03.444 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:02.684+0000 7faddd789700 1 --2- 172.21.15.46:0/1766934912 >> [v2:172.21.15.46:6800/3681182647,v1:172.21.15.46:6801/3681182647] conn(0x7fadc4038e60 0x7fadc403b320 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-31T23:02:03.444 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:02.754+0000 7fade01ee700 1 -- 172.21.15.46:0/1766934912 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) -- 0x7fadd81a15f0 con 0x7fadd807c2b0 2024-03-31T23:02:03.444 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:02.754+0000 7fadceffd700 1 -- 172.21.15.46:0/1766934912 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v9) ==== 56+0+88 (secure 0 0 0) 0x7fadd402d910 con 0x7fadd807c2b0 2024-03-31T23:02:03.444 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:02.756+0000 7fade01ee700 1 -- 172.21.15.46:0/1766934912 >> [v2:172.21.15.46:6800/3681182647,v1:172.21.15.46:6801/3681182647] conn(0x7fadc4038e60 msgr2=0x7fadc403b320 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-31T23:02:03.444 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:02.756+0000 7fade01ee700 1 --2- 172.21.15.46:0/1766934912 >> [v2:172.21.15.46:6800/3681182647,v1:172.21.15.46:6801/3681182647] conn(0x7fadc4038e60 0x7fadc403b320 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:02:03.444 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:02.756+0000 7fade01ee700 1 -- 172.21.15.46:0/1766934912 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fadd807c2b0 msgr2=0x7fadd819dc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:02:03.444 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:02.756+0000 7fade01ee700 1 --2- 172.21.15.46:0/1766934912 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fadd807c2b0 0x7fadd819dc70 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fadd4011a20 tx=0x7fadd4011b00 comp rx=0 tx=0).stop 2024-03-31T23:02:03.444 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:02.756+0000 7fade01ee700 1 -- 172.21.15.46:0/1766934912 shutdown_connections 2024-03-31T23:02:03.444 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:02.756+0000 7fade01ee700 1 --2- 172.21.15.46:0/1766934912 >> [v2:172.21.15.46:6800/3681182647,v1:172.21.15.46:6801/3681182647] conn(0x7fadc4038e60 0x7fadc403b320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:02:03.444 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:02.756+0000 7fade01ee700 1 --2- 172.21.15.46:0/1766934912 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fadd807c2b0 0x7fadd819dc70 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:02:03.444 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:02.756+0000 7fade01ee700 1 -- 172.21.15.46:0/1766934912 >> 172.21.15.46:0/1766934912 conn(0x7fadd8103070 msgr2=0x7fadd8103ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:02:03.444 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:02.757+0000 7fade01ee700 1 -- 172.21.15.46:0/1766934912 shutdown_connections 2024-03-31T23:02:03.444 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:02.757+0000 7fade01ee700 1 -- 172.21.15.46:0/1766934912 wait complete. 2024-03-31T23:02:03.444 INFO:teuthology.orchestra.run.smithi046.stdout:Waiting for the mgr to restart... 2024-03-31T23:02:03.444 INFO:teuthology.orchestra.run.smithi046.stdout:Waiting for mgr epoch 9... 2024-03-31T23:02:03.710 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:02:03 smithi046 ceph-mon[32498]: from='client.? 172.21.15.46:0/1766934912' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-03-31T23:02:03.711 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 31 23:02:03 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mgr-x[32727]: 2024-03-31T23:02:03.505+0000 7fbeaff39200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-31T23:02:03.711 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 31 23:02:03 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mgr-x[32727]: 2024-03-31T23:02:03.618+0000 7fbeaff39200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-31T23:02:03.711 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 31 23:02:03 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mgr-x[32727]: 2024-03-31T23:02:03.709+0000 7fbeaff39200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-31T23:02:03.991 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 31 23:02:03 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mgr-x[32727]: 2024-03-31T23:02:03.810+0000 7fbeaff39200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-31T23:02:03.992 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 31 23:02:03 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mgr-x[32727]: 2024-03-31T23:02:03.990+0000 7fbeaff39200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-31T23:02:05.361 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 31 23:02:04 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mgr-x[32727]: 2024-03-31T23:02:04.973+0000 7fbeaff39200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-31T23:02:05.361 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 31 23:02:05 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mgr-x[32727]: 2024-03-31T23:02:05.107+0000 7fbeaff39200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-31T23:02:05.743 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 31 23:02:05 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mgr-x[32727]: 2024-03-31T23:02:05.447+0000 7fbeaff39200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-31T23:02:05.743 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 31 23:02:05 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mgr-x[32727]: 2024-03-31T23:02:05.540+0000 7fbeaff39200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-31T23:02:05.743 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 31 23:02:05 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mgr-x[32727]: 2024-03-31T23:02:05.742+0000 7fbeaff39200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-31T23:02:06.111 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 31 23:02:05 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mgr-x[32727]: 2024-03-31T23:02:05.933+0000 7fbeaff39200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-31T23:02:06.490 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 31 23:02:06 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mgr-x[32727]: 2024-03-31T23:02:06.226+0000 7fbeaff39200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-31T23:02:06.822 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 31 23:02:06 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mgr-x[32727]: 2024-03-31T23:02:06.488+0000 7fbeaff39200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-31T23:02:06.822 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 31 23:02:06 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mgr-x[32727]: 2024-03-31T23:02:06.732+0000 7fbeaff39200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-31T23:02:07.862 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 31 23:02:07 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mgr-x[32727]: 2024-03-31T23:02:07.576+0000 7fbeaff39200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-31T23:02:08.492 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 31 23:02:08 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mgr-x[32727]: 2024-03-31T23:02:08.163+0000 7fbeaff39200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-31T23:02:08.492 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 31 23:02:08 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mgr-x[32727]: 2024-03-31T23:02:08.266+0000 7fbeaff39200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-31T23:02:08.492 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 31 23:02:08 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mgr-x[32727]: 2024-03-31T23:02:08.491+0000 7fbeaff39200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-31T23:02:08.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:02:08 smithi046 ceph-mon[32498]: Active manager daemon x restarted 2024-03-31T23:02:08.862 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:02:08 smithi046 ceph-mon[32498]: Activating manager daemon x 2024-03-31T23:02:08.862 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:02:08 smithi046 ceph-mon[32498]: osdmap e3: 0 total, 0 up, 0 in 2024-03-31T23:02:08.862 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:02:08 smithi046 ceph-mon[32498]: mgrmap e10: x(active, starting, since 0.00305759s) 2024-03-31T23:02:08.862 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:02:08 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-31T23:02:08.862 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:02:08 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-31T23:02:08.862 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:02:08 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-31T23:02:08.862 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:02:08 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-31T23:02:08.862 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:02:08 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-31T23:02:08.862 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:02:08 smithi046 ceph-mon[32498]: Manager daemon x is now available 2024-03-31T23:02:08.862 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:02:08 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-31T23:02:08.862 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:02:08 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-31T23:02:09.708 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:02:09 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-31T23:02:09.708 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:02:09 smithi046 ceph-mon[32498]: [31/Mar/2024:23:02:09] ENGINE Bus STARTING 2024-03-31T23:02:09.708 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:02:09 smithi046 ceph-mon[32498]: [31/Mar/2024:23:02:09] ENGINE Serving on http://172.21.15.46:8765 2024-03-31T23:02:09.709 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:02:09 smithi046 ceph-mon[32498]: [31/Mar/2024:23:02:09] ENGINE Serving on https://172.21.15.46:7150 2024-03-31T23:02:09.709 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:02:09 smithi046 ceph-mon[32498]: [31/Mar/2024:23:02:09] ENGINE Bus STARTED 2024-03-31T23:02:09.709 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:02:09 smithi046 ceph-mon[32498]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-03-31T23:02:09.709 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:02:09 smithi046 ceph-mon[32498]: mgrmap e11: x(active, since 1.00645s) 2024-03-31T23:02:09.709 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:02:09 smithi046 ceph-mon[32498]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-03-31T23:02:10.396 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout { 2024-03-31T23:02:10.396 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-03-31T23:02:10.396 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "initialized": true 2024-03-31T23:02:10.396 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout } 2024-03-31T23:02:10.396 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:03.811+0000 7ff27da87700 1 Processor -- start 2024-03-31T23:02:10.396 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:03.811+0000 7ff27da87700 1 -- start start 2024-03-31T23:02:10.396 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:03.812+0000 7ff27da87700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff27807c2b0 0x7ff27807c6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:02:10.396 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:03.812+0000 7ff27da87700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7ff27807dfe0 con 0x7ff27807c2b0 2024-03-31T23:02:10.396 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:03.812+0000 7ff2777fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff27807c2b0 0x7ff27807c6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:02:10.396 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:03.812+0000 7ff2777fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff27807c2b0 0x7ff27807c6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:49086/0 (socket says 172.21.15.46:49086) 2024-03-31T23:02:10.396 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:03.812+0000 7ff2777fe700 1 -- 172.21.15.46:0/1847058581 learned_addr learned my addr 172.21.15.46:0/1847058581 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:02:10.397 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:03.812+0000 7ff2777fe700 1 -- 172.21.15.46:0/1847058581 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff27807cc10 con 0x7ff27807c2b0 2024-03-31T23:02:10.397 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:03.812+0000 7ff2777fe700 1 --2- 172.21.15.46:0/1847058581 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff27807c2b0 0x7ff27807c6d0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7ff2680098a0 tx=0x7ff268009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b93551dbd260725 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:02:10.397 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:03.812+0000 7ff2767fc700 1 -- 172.21.15.46:0/1847058581 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff268004160 con 0x7ff27807c2b0 2024-03-31T23:02:10.397 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:03.812+0000 7ff2767fc700 1 -- 172.21.15.46:0/1847058581 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff268004300 con 0x7ff27807c2b0 2024-03-31T23:02:10.397 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:03.812+0000 7ff2767fc700 1 -- 172.21.15.46:0/1847058581 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff268011a50 con 0x7ff27807c2b0 2024-03-31T23:02:10.397 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:03.813+0000 7ff27da87700 1 -- 172.21.15.46:0/1847058581 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff27807c2b0 msgr2=0x7ff27807c6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:02:10.397 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:03.813+0000 7ff27da87700 1 --2- 172.21.15.46:0/1847058581 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff27807c2b0 0x7ff27807c6d0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7ff2680098a0 tx=0x7ff268009bb0 comp rx=0 tx=0).stop 2024-03-31T23:02:10.397 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:03.813+0000 7ff27da87700 1 -- 172.21.15.46:0/1847058581 shutdown_connections 2024-03-31T23:02:10.397 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:03.813+0000 7ff27da87700 1 --2- 172.21.15.46:0/1847058581 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff27807c2b0 0x7ff27807c6d0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:02:10.397 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:03.813+0000 7ff27da87700 1 -- 172.21.15.46:0/1847058581 >> 172.21.15.46:0/1847058581 conn(0x7ff278103070 msgr2=0x7ff278105490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:02:10.397 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:03.813+0000 7ff27da87700 1 -- 172.21.15.46:0/1847058581 shutdown_connections 2024-03-31T23:02:10.397 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:03.813+0000 7ff27da87700 1 -- 172.21.15.46:0/1847058581 wait complete. 2024-03-31T23:02:10.397 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:03.814+0000 7ff27da87700 1 Processor -- start 2024-03-31T23:02:10.398 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:03.814+0000 7ff27da87700 1 -- start start 2024-03-31T23:02:10.398 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:03.814+0000 7ff27da87700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff27807c2b0 0x7ff27819dc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:02:10.398 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:03.814+0000 7ff27da87700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7ff27810e150 con 0x7ff27807c2b0 2024-03-31T23:02:10.398 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:03.814+0000 7ff2777fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff27807c2b0 0x7ff27819dc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:02:10.398 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:03.814+0000 7ff2777fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff27807c2b0 0x7ff27819dc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:49090/0 (socket says 172.21.15.46:49090) 2024-03-31T23:02:10.398 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:03.814+0000 7ff2777fe700 1 -- 172.21.15.46:0/3647276000 learned_addr learned my addr 172.21.15.46:0/3647276000 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:02:10.398 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:03.814+0000 7ff2777fe700 1 -- 172.21.15.46:0/3647276000 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff27819e1b0 con 0x7ff27807c2b0 2024-03-31T23:02:10.398 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:03.815+0000 7ff2777fe700 1 --2- 172.21.15.46:0/3647276000 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff27807c2b0 0x7ff27819dc70 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7ff268011a20 tx=0x7ff268011b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:02:10.398 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:03.815+0000 7ff274ff9700 1 -- 172.21.15.46:0/3647276000 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff2680039c0 con 0x7ff27807c2b0 2024-03-31T23:02:10.398 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:03.815+0000 7ff27da87700 1 -- 172.21.15.46:0/3647276000 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff27819e440 con 0x7ff27807c2b0 2024-03-31T23:02:10.398 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:03.815+0000 7ff274ff9700 1 -- 172.21.15.46:0/3647276000 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff268003b60 con 0x7ff27807c2b0 2024-03-31T23:02:10.398 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:03.815+0000 7ff274ff9700 1 -- 172.21.15.46:0/3647276000 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff26801a6b0 con 0x7ff27807c2b0 2024-03-31T23:02:10.398 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:03.815+0000 7ff27da87700 1 -- 172.21.15.46:0/3647276000 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff2781a1130 con 0x7ff27807c2b0 2024-03-31T23:02:10.398 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:03.816+0000 7ff274ff9700 1 -- 172.21.15.46:0/3647276000 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 9) ==== 46129+0+0 (secure 0 0 0) 0x7ff26801aa30 con 0x7ff27807c2b0 2024-03-31T23:02:10.399 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:03.816+0000 7ff274ff9700 1 --2- 172.21.15.46:0/3647276000 >> [v2:172.21.15.46:6800/3681182647,v1:172.21.15.46:6801/3681182647] conn(0x7ff264038e00 0x7ff26403b2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:02:10.399 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:03.816+0000 7ff27da87700 1 -- 172.21.15.46:0/3647276000 --> [v2:172.21.15.46:6800/3681182647,v1:172.21.15.46:6801/3681182647] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7ff278063870 con 0x7ff264038e00 2024-03-31T23:02:10.399 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:03.816+0000 7ff276ffd700 1 -- 172.21.15.46:0/3647276000 >> [v2:172.21.15.46:6800/3681182647,v1:172.21.15.46:6801/3681182647] conn(0x7ff264038e00 msgr2=0x7ff26403b2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.46:6800/3681182647 2024-03-31T23:02:10.399 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:03.816+0000 7ff274ff9700 1 -- 172.21.15.46:0/3647276000 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7ff26804e260 con 0x7ff27807c2b0 2024-03-31T23:02:10.399 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:03.816+0000 7ff276ffd700 1 --2- 172.21.15.46:0/3647276000 >> [v2:172.21.15.46:6800/3681182647,v1:172.21.15.46:6801/3681182647] conn(0x7ff264038e00 0x7ff26403b2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-31T23:02:10.399 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:04.017+0000 7ff276ffd700 1 -- 172.21.15.46:0/3647276000 >> [v2:172.21.15.46:6800/3681182647,v1:172.21.15.46:6801/3681182647] conn(0x7ff264038e00 msgr2=0x7ff26403b2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.46:6800/3681182647 2024-03-31T23:02:10.399 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:04.017+0000 7ff276ffd700 1 --2- 172.21.15.46:0/3647276000 >> [v2:172.21.15.46:6800/3681182647,v1:172.21.15.46:6801/3681182647] conn(0x7ff264038e00 0x7ff26403b2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-31T23:02:10.399 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:04.418+0000 7ff276ffd700 1 -- 172.21.15.46:0/3647276000 >> [v2:172.21.15.46:6800/3681182647,v1:172.21.15.46:6801/3681182647] conn(0x7ff264038e00 msgr2=0x7ff26403b2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.46:6800/3681182647 2024-03-31T23:02:10.399 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:04.418+0000 7ff276ffd700 1 --2- 172.21.15.46:0/3647276000 >> [v2:172.21.15.46:6800/3681182647,v1:172.21.15.46:6801/3681182647] conn(0x7ff264038e00 0x7ff26403b2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-31T23:02:10.399 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:05.219+0000 7ff276ffd700 1 -- 172.21.15.46:0/3647276000 >> [v2:172.21.15.46:6800/3681182647,v1:172.21.15.46:6801/3681182647] conn(0x7ff264038e00 msgr2=0x7ff26403b2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.46:6800/3681182647 2024-03-31T23:02:10.399 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:05.219+0000 7ff276ffd700 1 --2- 172.21.15.46:0/3647276000 >> [v2:172.21.15.46:6800/3681182647,v1:172.21.15.46:6801/3681182647] conn(0x7ff264038e00 0x7ff26403b2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-31T23:02:10.399 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:06.820+0000 7ff276ffd700 1 -- 172.21.15.46:0/3647276000 >> [v2:172.21.15.46:6800/3681182647,v1:172.21.15.46:6801/3681182647] conn(0x7ff264038e00 msgr2=0x7ff26403b2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.46:6800/3681182647 2024-03-31T23:02:10.399 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:06.820+0000 7ff276ffd700 1 --2- 172.21.15.46:0/3647276000 >> [v2:172.21.15.46:6800/3681182647,v1:172.21.15.46:6801/3681182647] conn(0x7ff264038e00 0x7ff26403b2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-03-31T23:02:10.399 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:08.494+0000 7ff274ff9700 1 -- 172.21.15.46:0/3647276000 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mgrmap(e 10) ==== 45896+0+0 (secure 0 0 0) 0x7ff26802d910 con 0x7ff27807c2b0 2024-03-31T23:02:10.399 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:08.494+0000 7ff274ff9700 1 -- 172.21.15.46:0/3647276000 >> [v2:172.21.15.46:6800/3681182647,v1:172.21.15.46:6801/3681182647] conn(0x7ff264038e00 msgr2=0x7ff26403b2c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-31T23:02:10.400 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:08.494+0000 7ff274ff9700 1 --2- 172.21.15.46:0/3647276000 >> [v2:172.21.15.46:6800/3681182647,v1:172.21.15.46:6801/3681182647] conn(0x7ff264038e00 0x7ff26403b2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:02:10.400 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:09.496+0000 7ff274ff9700 1 -- 172.21.15.46:0/3647276000 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mgrmap(e 11) ==== 46023+0+0 (secure 0 0 0) 0x7ff268029cf0 con 0x7ff27807c2b0 2024-03-31T23:02:10.400 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:09.496+0000 7ff274ff9700 1 --2- 172.21.15.46:0/3647276000 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7ff26403c6d0 0x7ff26403eac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:02:10.400 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:09.496+0000 7ff274ff9700 1 -- 172.21.15.46:0/3647276000 --> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7ff278063870 con 0x7ff26403c6d0 2024-03-31T23:02:10.400 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:09.497+0000 7ff276ffd700 1 --2- 172.21.15.46:0/3647276000 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7ff26403c6d0 0x7ff26403eac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:02:10.400 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:09.497+0000 7ff276ffd700 1 --2- 172.21.15.46:0/3647276000 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7ff26403c6d0 0x7ff26403eac0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7ff260003a10 tx=0x7ff2600092b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:02:10.400 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:09.498+0000 7ff274ff9700 1 -- 172.21.15.46:0/3647276000 <== mgr.14152 v2:172.21.15.46:6800/162289569 1 ==== command_reply(tid 0: 0 ) ==== 8+0+7774 (secure 0 0 0) 0x7ff278063870 con 0x7ff26403c6d0 2024-03-31T23:02:10.400 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:09.505+0000 7ff27da87700 1 -- 172.21.15.46:0/3647276000 --> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] -- command(tid 1: {"prefix": "mgr_status"}) -- 0x7ff27819e920 con 0x7ff26403c6d0 2024-03-31T23:02:10.400 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:09.505+0000 7ff274ff9700 1 -- 172.21.15.46:0/3647276000 <== mgr.14152 v2:172.21.15.46:6800/162289569 2 ==== command_reply(tid 1: 0 ) ==== 8+0+52 (secure 0 0 0) 0x7ff27819e920 con 0x7ff26403c6d0 2024-03-31T23:02:10.400 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:09.505+0000 7ff27da87700 1 -- 172.21.15.46:0/3647276000 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7ff26403c6d0 msgr2=0x7ff26403eac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:02:10.400 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:09.505+0000 7ff27da87700 1 --2- 172.21.15.46:0/3647276000 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7ff26403c6d0 0x7ff26403eac0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7ff260003a10 tx=0x7ff2600092b0 comp rx=0 tx=0).stop 2024-03-31T23:02:10.400 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:09.505+0000 7ff27da87700 1 -- 172.21.15.46:0/3647276000 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff27807c2b0 msgr2=0x7ff27819dc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:02:10.400 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:09.505+0000 7ff27da87700 1 --2- 172.21.15.46:0/3647276000 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff27807c2b0 0x7ff27819dc70 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7ff268011a20 tx=0x7ff268011b00 comp rx=0 tx=0).stop 2024-03-31T23:02:10.400 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:09.505+0000 7ff27da87700 1 -- 172.21.15.46:0/3647276000 shutdown_connections 2024-03-31T23:02:10.401 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:09.505+0000 7ff27da87700 1 --2- 172.21.15.46:0/3647276000 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7ff26403c6d0 0x7ff26403eac0 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:02:10.401 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:09.505+0000 7ff27da87700 1 --2- 172.21.15.46:0/3647276000 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff27807c2b0 0x7ff27819dc70 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:02:10.401 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:09.505+0000 7ff27da87700 1 --2- 172.21.15.46:0/3647276000 >> [v2:172.21.15.46:6800/3681182647,v1:172.21.15.46:6801/3681182647] conn(0x7ff264038e00 0x7ff26403b2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:02:10.401 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:09.505+0000 7ff27da87700 1 -- 172.21.15.46:0/3647276000 >> 172.21.15.46:0/3647276000 conn(0x7ff278103070 msgr2=0x7ff278103ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:02:10.401 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:09.505+0000 7ff27da87700 1 -- 172.21.15.46:0/3647276000 shutdown_connections 2024-03-31T23:02:10.401 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:09.506+0000 7ff27da87700 1 -- 172.21.15.46:0/3647276000 wait complete. 2024-03-31T23:02:10.401 INFO:teuthology.orchestra.run.smithi046.stdout:mgr epoch 9 is available 2024-03-31T23:02:10.401 INFO:teuthology.orchestra.run.smithi046.stdout:Generating a dashboard self-signed certificate... 2024-03-31T23:02:11.082 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:02:11 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:02:11.696 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-03-31T23:02:11.696 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:10.823+0000 7f9b9a625700 1 Processor -- start 2024-03-31T23:02:11.696 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:10.823+0000 7f9b9a625700 1 -- start start 2024-03-31T23:02:11.696 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:10.823+0000 7f9b9a625700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9b9407cd00 0x7f9b9407b160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:02:11.696 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:10.823+0000 7f9b9a625700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f9b9407b730 con 0x7f9b9407cd00 2024-03-31T23:02:11.696 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:10.823+0000 7f9b93fff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9b9407cd00 0x7f9b9407b160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:02:11.696 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:10.823+0000 7f9b93fff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9b9407cd00 0x7f9b9407b160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:49166/0 (socket says 172.21.15.46:49166) 2024-03-31T23:02:11.696 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:10.823+0000 7f9b93fff700 1 -- 172.21.15.46:0/830105962 learned_addr learned my addr 172.21.15.46:0/830105962 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:02:11.696 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:10.823+0000 7f9b93fff700 1 -- 172.21.15.46:0/830105962 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9b9407b8b0 con 0x7f9b9407cd00 2024-03-31T23:02:11.696 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:10.824+0000 7f9b93fff700 1 --2- 172.21.15.46:0/830105962 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9b9407cd00 0x7f9b9407b160 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f9b840098a0 tx=0x7f9b84009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d1c6bab5550312d server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:02:11.696 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:10.824+0000 7f9b92ffd700 1 -- 172.21.15.46:0/830105962 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9b84004160 con 0x7f9b9407cd00 2024-03-31T23:02:11.697 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:10.824+0000 7f9b92ffd700 1 -- 172.21.15.46:0/830105962 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9b84004300 con 0x7f9b9407cd00 2024-03-31T23:02:11.697 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:10.824+0000 7f9b92ffd700 1 -- 172.21.15.46:0/830105962 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9b84011a50 con 0x7f9b9407cd00 2024-03-31T23:02:11.697 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:10.824+0000 7f9b9a625700 1 -- 172.21.15.46:0/830105962 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9b9407cd00 msgr2=0x7f9b9407b160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:02:11.697 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:10.824+0000 7f9b9a625700 1 --2- 172.21.15.46:0/830105962 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9b9407cd00 0x7f9b9407b160 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f9b840098a0 tx=0x7f9b84009bb0 comp rx=0 tx=0).stop 2024-03-31T23:02:11.697 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:10.824+0000 7f9b9a625700 1 -- 172.21.15.46:0/830105962 shutdown_connections 2024-03-31T23:02:11.697 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:10.824+0000 7f9b9a625700 1 --2- 172.21.15.46:0/830105962 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9b9407cd00 0x7f9b9407b160 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:02:11.697 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:10.824+0000 7f9b9a625700 1 -- 172.21.15.46:0/830105962 >> 172.21.15.46:0/830105962 conn(0x7f9b94103060 msgr2=0x7f9b94105480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:02:11.697 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:10.825+0000 7f9b9a625700 1 -- 172.21.15.46:0/830105962 shutdown_connections 2024-03-31T23:02:11.697 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:10.825+0000 7f9b9a625700 1 -- 172.21.15.46:0/830105962 wait complete. 2024-03-31T23:02:11.697 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:10.825+0000 7f9b9a625700 1 Processor -- start 2024-03-31T23:02:11.697 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:10.825+0000 7f9b9a625700 1 -- start start 2024-03-31T23:02:11.698 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:10.825+0000 7f9b9a625700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9b9407cd00 0x7f9b9419db30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:02:11.698 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:10.825+0000 7f9b9a625700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f9b9410e0c0 con 0x7f9b9407cd00 2024-03-31T23:02:11.698 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:10.826+0000 7f9b93fff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9b9407cd00 0x7f9b9419db30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:02:11.698 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:10.826+0000 7f9b93fff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9b9407cd00 0x7f9b9419db30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:49168/0 (socket says 172.21.15.46:49168) 2024-03-31T23:02:11.698 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:10.826+0000 7f9b93fff700 1 -- 172.21.15.46:0/3401164751 learned_addr learned my addr 172.21.15.46:0/3401164751 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:02:11.698 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:10.826+0000 7f9b93fff700 1 -- 172.21.15.46:0/3401164751 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9b9419e070 con 0x7f9b9407cd00 2024-03-31T23:02:11.698 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:10.826+0000 7f9b93fff700 1 --2- 172.21.15.46:0/3401164751 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9b9407cd00 0x7f9b9419db30 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f9b84006b20 tx=0x7f9b84011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:02:11.698 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:10.826+0000 7f9b917fa700 1 -- 172.21.15.46:0/3401164751 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9b84003b70 con 0x7f9b9407cd00 2024-03-31T23:02:11.698 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:10.826+0000 7f9b917fa700 1 -- 172.21.15.46:0/3401164751 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9b8401a8e0 con 0x7f9b9407cd00 2024-03-31T23:02:11.698 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:10.826+0000 7f9b9a625700 1 -- 172.21.15.46:0/3401164751 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9b9419e300 con 0x7f9b9407cd00 2024-03-31T23:02:11.698 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:10.826+0000 7f9b917fa700 1 -- 172.21.15.46:0/3401164751 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9b840249b0 con 0x7f9b9407cd00 2024-03-31T23:02:11.699 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:10.826+0000 7f9b9a625700 1 -- 172.21.15.46:0/3401164751 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9b941a0ff0 con 0x7f9b9407cd00 2024-03-31T23:02:11.699 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:10.827+0000 7f9b917fa700 1 -- 172.21.15.46:0/3401164751 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 11) ==== 46023+0+0 (secure 0 0 0) 0x7f9b84022020 con 0x7f9b9407cd00 2024-03-31T23:02:11.699 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:10.827+0000 7f9b917fa700 1 --2- 172.21.15.46:0/3401164751 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f9b7c038da0 0x7f9b7c03b260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:02:11.700 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:10.827+0000 7f9b917fa700 1 -- 172.21.15.46:0/3401164751 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f9b8404d560 con 0x7f9b9407cd00 2024-03-31T23:02:11.700 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:10.828+0000 7f9b937fe700 1 --2- 172.21.15.46:0/3401164751 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f9b7c038da0 0x7f9b7c03b260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:02:11.700 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:10.828+0000 7f9b9a625700 1 -- 172.21.15.46:0/3401164751 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9b94063870 con 0x7f9b9407cd00 2024-03-31T23:02:11.700 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:10.828+0000 7f9b937fe700 1 --2- 172.21.15.46:0/3401164751 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f9b7c038da0 0x7f9b7c03b260 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f9b88006fd0 tx=0x7f9b88006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:02:11.700 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:10.833+0000 7f9b917fa700 1 -- 172.21.15.46:0/3401164751 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f9b8400b3e0 con 0x7f9b9407cd00 2024-03-31T23:02:11.701 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:11.041+0000 7f9b9a625700 1 -- 172.21.15.46:0/3401164751 --> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) -- 0x7f9b94104160 con 0x7f9b7c038da0 2024-03-31T23:02:11.701 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:11.052+0000 7f9b917fa700 1 -- 172.21.15.46:0/3401164751 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mgrmap(e 12) ==== 46129+0+0 (secure 0 0 0) 0x7f9b8401aa90 con 0x7f9b9407cd00 2024-03-31T23:02:11.701 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:11.081+0000 7f9b917fa700 1 -- 172.21.15.46:0/3401164751 <== mgr.14152 v2:172.21.15.46:6800/162289569 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+32 (secure 0 0 0) 0x7f9b94104160 con 0x7f9b7c038da0 2024-03-31T23:02:11.701 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:11.083+0000 7f9b9a625700 1 -- 172.21.15.46:0/3401164751 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f9b7c038da0 msgr2=0x7f9b7c03b260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:02:11.701 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:11.083+0000 7f9b9a625700 1 --2- 172.21.15.46:0/3401164751 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f9b7c038da0 0x7f9b7c03b260 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f9b88006fd0 tx=0x7f9b88006e40 comp rx=0 tx=0).stop 2024-03-31T23:02:11.701 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:11.083+0000 7f9b9a625700 1 -- 172.21.15.46:0/3401164751 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9b9407cd00 msgr2=0x7f9b9419db30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:02:11.702 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:11.083+0000 7f9b9a625700 1 --2- 172.21.15.46:0/3401164751 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9b9407cd00 0x7f9b9419db30 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f9b84006b20 tx=0x7f9b84011ad0 comp rx=0 tx=0).stop 2024-03-31T23:02:11.702 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:11.083+0000 7f9b9a625700 1 -- 172.21.15.46:0/3401164751 shutdown_connections 2024-03-31T23:02:11.702 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:11.083+0000 7f9b9a625700 1 --2- 172.21.15.46:0/3401164751 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f9b7c038da0 0x7f9b7c03b260 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:02:11.702 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:11.083+0000 7f9b9a625700 1 --2- 172.21.15.46:0/3401164751 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9b9407cd00 0x7f9b9419db30 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:02:11.702 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:11.083+0000 7f9b9a625700 1 -- 172.21.15.46:0/3401164751 >> 172.21.15.46:0/3401164751 conn(0x7f9b94103060 msgr2=0x7f9b94103a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:02:11.702 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:11.084+0000 7f9b9a625700 1 -- 172.21.15.46:0/3401164751 shutdown_connections 2024-03-31T23:02:11.703 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:11.084+0000 7f9b9a625700 1 -- 172.21.15.46:0/3401164751 wait complete. 2024-03-31T23:02:11.703 INFO:teuthology.orchestra.run.smithi046.stdout:Creating initial admin user... 2024-03-31T23:02:12.320 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:02:12 smithi046 ceph-mon[32498]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-03-31T23:02:12.320 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:02:12 smithi046 ceph-mon[32498]: mgrmap e12: x(active, since 2s) 2024-03-31T23:02:12.320 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:02:12 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:02:12.320 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:02:12 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:02:12.320 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:02:12 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:02:13.320 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:02:13 smithi046 ceph-mon[32498]: from='client.14166 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-31T23:02:13.320 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:02:13 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:02:13.584 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$HsfzcI781rf7SIy60LZye.qv9MJnRvStzXrkPAlN6fVXBZwPNyADC", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1711926132, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-03-31T23:02:13.584 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:12.081+0000 7febd3932700 1 Processor -- start 2024-03-31T23:02:13.584 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:12.082+0000 7febd3932700 1 -- start start 2024-03-31T23:02:13.584 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:12.082+0000 7febd3932700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7febcc108500 0x7febcc10a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:02:13.584 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:12.082+0000 7febd3932700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7febcc075780 con 0x7febcc108500 2024-03-31T23:02:13.585 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:12.082+0000 7febd16ce700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7febcc108500 0x7febcc10a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:02:13.585 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:12.082+0000 7febd16ce700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7febcc108500 0x7febcc10a8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:50448/0 (socket says 172.21.15.46:50448) 2024-03-31T23:02:13.585 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:12.082+0000 7febd16ce700 1 -- 172.21.15.46:0/2215868573 learned_addr learned my addr 172.21.15.46:0/2215868573 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:02:13.585 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:12.082+0000 7febd16ce700 1 -- 172.21.15.46:0/2215868573 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7febcc10ae30 con 0x7febcc108500 2024-03-31T23:02:13.585 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:12.083+0000 7febd16ce700 1 --2- 172.21.15.46:0/2215868573 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7febcc108500 0x7febcc10a8f0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7febbc0098a0 tx=0x7febbc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3157a23b9b5a0d9c server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:02:13.585 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:12.083+0000 7febc3fff700 1 -- 172.21.15.46:0/2215868573 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7febbc004160 con 0x7febcc108500 2024-03-31T23:02:13.585 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:12.083+0000 7febc3fff700 1 -- 172.21.15.46:0/2215868573 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7febbc004300 con 0x7febcc108500 2024-03-31T23:02:13.585 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:12.083+0000 7febc3fff700 1 -- 172.21.15.46:0/2215868573 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7febbc011a50 con 0x7febcc108500 2024-03-31T23:02:13.585 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:12.084+0000 7febd3932700 1 -- 172.21.15.46:0/2215868573 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7febcc108500 msgr2=0x7febcc10a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:02:13.589 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:12.084+0000 7febd3932700 1 --2- 172.21.15.46:0/2215868573 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7febcc108500 0x7febcc10a8f0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7febbc0098a0 tx=0x7febbc009bb0 comp rx=0 tx=0).stop 2024-03-31T23:02:13.590 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:12.084+0000 7febd3932700 1 -- 172.21.15.46:0/2215868573 shutdown_connections 2024-03-31T23:02:13.590 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:12.084+0000 7febd3932700 1 --2- 172.21.15.46:0/2215868573 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7febcc108500 0x7febcc10a8f0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:02:13.590 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:12.084+0000 7febd3932700 1 -- 172.21.15.46:0/2215868573 >> 172.21.15.46:0/2215868573 conn(0x7febcc101f50 msgr2=0x7febcc104370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:02:13.590 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:12.084+0000 7febd3932700 1 -- 172.21.15.46:0/2215868573 shutdown_connections 2024-03-31T23:02:13.590 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:12.084+0000 7febd3932700 1 -- 172.21.15.46:0/2215868573 wait complete. 2024-03-31T23:02:13.590 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:12.084+0000 7febd3932700 1 Processor -- start 2024-03-31T23:02:13.590 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:12.084+0000 7febd3932700 1 -- start start 2024-03-31T23:02:13.590 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:12.085+0000 7febd3932700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7febcc108500 0x7febcc19b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:02:13.590 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:12.085+0000 7febd3932700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7febcc10be40 con 0x7febcc108500 2024-03-31T23:02:13.590 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:12.085+0000 7febd16ce700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7febcc108500 0x7febcc19b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:02:13.590 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:12.085+0000 7febd16ce700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7febcc108500 0x7febcc19b9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:50454/0 (socket says 172.21.15.46:50454) 2024-03-31T23:02:13.590 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:12.085+0000 7febd16ce700 1 -- 172.21.15.46:0/3767279573 learned_addr learned my addr 172.21.15.46:0/3767279573 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:02:13.591 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:12.085+0000 7febd16ce700 1 -- 172.21.15.46:0/3767279573 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7febcc19bf00 con 0x7febcc108500 2024-03-31T23:02:13.591 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:12.085+0000 7febd16ce700 1 --2- 172.21.15.46:0/3767279573 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7febcc108500 0x7febcc19b9c0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7febbc011a20 tx=0x7febbc011b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:02:13.591 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:12.086+0000 7febc27fc700 1 -- 172.21.15.46:0/3767279573 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7febbc0039c0 con 0x7febcc108500 2024-03-31T23:02:13.591 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:12.086+0000 7febc27fc700 1 -- 172.21.15.46:0/3767279573 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7febbc003b60 con 0x7febcc108500 2024-03-31T23:02:13.591 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:12.086+0000 7febc27fc700 1 -- 172.21.15.46:0/3767279573 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7febbc01a6b0 con 0x7febcc108500 2024-03-31T23:02:13.591 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:12.086+0000 7febd3932700 1 -- 172.21.15.46:0/3767279573 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7febcc19c190 con 0x7febcc108500 2024-03-31T23:02:13.591 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:12.086+0000 7febd3932700 1 -- 172.21.15.46:0/3767279573 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7febcc19ee80 con 0x7febcc108500 2024-03-31T23:02:13.591 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:12.087+0000 7febc27fc700 1 -- 172.21.15.46:0/3767279573 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 12) ==== 46129+0+0 (secure 0 0 0) 0x7febbc024430 con 0x7febcc108500 2024-03-31T23:02:13.591 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:12.087+0000 7febd3932700 1 -- 172.21.15.46:0/3767279573 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7febcc063870 con 0x7febcc108500 2024-03-31T23:02:13.591 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:12.087+0000 7febc27fc700 1 --2- 172.21.15.46:0/3767279573 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7febb8038e00 0x7febb803b2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:02:13.591 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:12.087+0000 7febc27fc700 1 -- 172.21.15.46:0/3767279573 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7febbc04d4f0 con 0x7febcc108500 2024-03-31T23:02:13.591 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:12.087+0000 7febd0ecd700 1 --2- 172.21.15.46:0/3767279573 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7febb8038e00 0x7febb803b2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:02:13.591 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:12.088+0000 7febd0ecd700 1 --2- 172.21.15.46:0/3767279573 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7febb8038e00 0x7febb803b2c0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7febc8006fd0 tx=0x7febc8006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:02:13.591 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:12.093+0000 7febc27fc700 1 -- 172.21.15.46:0/3767279573 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7febbc00f2e0 con 0x7febcc108500 2024-03-31T23:02:13.592 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:12.323+0000 7febd3932700 1 -- 172.21.15.46:0/3767279573 --> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) -- 0x7febcc19c670 con 0x7febb8038e00 2024-03-31T23:02:13.592 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:12.569+0000 7febc27fc700 1 -- 172.21.15.46:0/3767279573 <== mgr.14152 v2:172.21.15.46:6800/162289569 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+252 (secure 0 0 0) 0x7febcc19c670 con 0x7febb8038e00 2024-03-31T23:02:13.592 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:12.571+0000 7febd3932700 1 -- 172.21.15.46:0/3767279573 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7febb8038e00 msgr2=0x7febb803b2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:02:13.592 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:12.571+0000 7febd3932700 1 --2- 172.21.15.46:0/3767279573 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7febb8038e00 0x7febb803b2c0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7febc8006fd0 tx=0x7febc8006e40 comp rx=0 tx=0).stop 2024-03-31T23:02:13.592 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:12.571+0000 7febd3932700 1 -- 172.21.15.46:0/3767279573 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7febcc108500 msgr2=0x7febcc19b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:02:13.592 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:12.571+0000 7febd3932700 1 --2- 172.21.15.46:0/3767279573 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7febcc108500 0x7febcc19b9c0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7febbc011a20 tx=0x7febbc011b00 comp rx=0 tx=0).stop 2024-03-31T23:02:13.592 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:12.571+0000 7febd3932700 1 -- 172.21.15.46:0/3767279573 shutdown_connections 2024-03-31T23:02:13.592 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:12.571+0000 7febd3932700 1 --2- 172.21.15.46:0/3767279573 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7febb8038e00 0x7febb803b2c0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:02:13.592 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:12.571+0000 7febd3932700 1 --2- 172.21.15.46:0/3767279573 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7febcc108500 0x7febcc19b9c0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:02:13.592 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:12.571+0000 7febd3932700 1 -- 172.21.15.46:0/3767279573 >> 172.21.15.46:0/3767279573 conn(0x7febcc101f50 msgr2=0x7febcc102a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:02:13.592 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:12.571+0000 7febd3932700 1 -- 172.21.15.46:0/3767279573 shutdown_connections 2024-03-31T23:02:13.592 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:12.571+0000 7febd3932700 1 -- 172.21.15.46:0/3767279573 wait complete. 2024-03-31T23:02:13.592 INFO:teuthology.orchestra.run.smithi046.stdout:Fetching dashboard port number... 2024-03-31T23:02:14.611 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:02:14 smithi046 ceph-mon[32498]: from='client.? 172.21.15.46:0/3694922287' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-03-31T23:02:14.726 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout 8443 2024-03-31T23:02:14.726 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:14.008+0000 7f949d837700 1 Processor -- start 2024-03-31T23:02:14.726 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:14.008+0000 7f949d837700 1 -- start start 2024-03-31T23:02:14.726 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:14.008+0000 7f949d837700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f94981069c0 0x7f9498106de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:02:14.727 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:14.008+0000 7f949d837700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f94981073b0 con 0x7f94981069c0 2024-03-31T23:02:14.727 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:14.008+0000 7f9496ffd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f94981069c0 0x7f9498106de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:02:14.727 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:14.008+0000 7f9496ffd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f94981069c0 0x7f9498106de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:50458/0 (socket says 172.21.15.46:50458) 2024-03-31T23:02:14.727 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:14.008+0000 7f9496ffd700 1 -- 172.21.15.46:0/3016675219 learned_addr learned my addr 172.21.15.46:0/3016675219 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:02:14.727 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:14.009+0000 7f9496ffd700 1 -- 172.21.15.46:0/3016675219 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9498107bc0 con 0x7f94981069c0 2024-03-31T23:02:14.727 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:14.009+0000 7f9496ffd700 1 --2- 172.21.15.46:0/3016675219 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f94981069c0 0x7f9498106de0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f949000bb40 tx=0x7f949000be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=79fdf427b75ad0f1 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:02:14.727 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:14.009+0000 7f9495ffb700 1 -- 172.21.15.46:0/3016675219 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f949000dbe0 con 0x7f94981069c0 2024-03-31T23:02:14.727 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:14.009+0000 7f9495ffb700 1 -- 172.21.15.46:0/3016675219 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f94900044e0 con 0x7f94981069c0 2024-03-31T23:02:14.727 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:14.009+0000 7f9495ffb700 1 -- 172.21.15.46:0/3016675219 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9490010470 con 0x7f94981069c0 2024-03-31T23:02:14.727 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:14.009+0000 7f949d837700 1 -- 172.21.15.46:0/3016675219 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f94981069c0 msgr2=0x7f9498106de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:02:14.727 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:14.009+0000 7f949d837700 1 --2- 172.21.15.46:0/3016675219 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f94981069c0 0x7f9498106de0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f949000bb40 tx=0x7f949000be50 comp rx=0 tx=0).stop 2024-03-31T23:02:14.727 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:14.010+0000 7f949d837700 1 -- 172.21.15.46:0/3016675219 shutdown_connections 2024-03-31T23:02:14.727 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:14.010+0000 7f949d837700 1 --2- 172.21.15.46:0/3016675219 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f94981069c0 0x7f9498106de0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:02:14.728 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:14.010+0000 7f949d837700 1 -- 172.21.15.46:0/3016675219 >> 172.21.15.46:0/3016675219 conn(0x7f9498101f60 msgr2=0x7f94981043a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:02:14.728 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:14.010+0000 7f949d837700 1 -- 172.21.15.46:0/3016675219 shutdown_connections 2024-03-31T23:02:14.728 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:14.010+0000 7f949d837700 1 -- 172.21.15.46:0/3016675219 wait complete. 2024-03-31T23:02:14.728 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:14.010+0000 7f949d837700 1 Processor -- start 2024-03-31T23:02:14.728 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:14.010+0000 7f949d837700 1 -- start start 2024-03-31T23:02:14.728 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:14.011+0000 7f949d837700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f94981069c0 0x7f9498199800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:02:14.728 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:14.011+0000 7f949d837700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f949807af60 con 0x7f94981069c0 2024-03-31T23:02:14.728 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:14.011+0000 7f9496ffd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f94981069c0 0x7f9498199800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:02:14.728 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:14.011+0000 7f9496ffd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f94981069c0 0x7f9498199800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:50468/0 (socket says 172.21.15.46:50468) 2024-03-31T23:02:14.728 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:14.011+0000 7f9496ffd700 1 -- 172.21.15.46:0/3694922287 learned_addr learned my addr 172.21.15.46:0/3694922287 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:02:14.728 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:14.011+0000 7f9496ffd700 1 -- 172.21.15.46:0/3694922287 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9498199d40 con 0x7f94981069c0 2024-03-31T23:02:14.728 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:14.011+0000 7f9496ffd700 1 --2- 172.21.15.46:0/3694922287 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f94981069c0 0x7f9498199800 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f949000b340 tx=0x7f9490003fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:02:14.728 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:14.011+0000 7f949c835700 1 -- 172.21.15.46:0/3694922287 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9490027070 con 0x7f94981069c0 2024-03-31T23:02:14.728 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:14.011+0000 7f949d837700 1 -- 172.21.15.46:0/3694922287 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9498199fd0 con 0x7f94981069c0 2024-03-31T23:02:14.729 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:14.011+0000 7f949d837700 1 -- 172.21.15.46:0/3694922287 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f949819a430 con 0x7f94981069c0 2024-03-31T23:02:14.729 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:14.011+0000 7f949c835700 1 -- 172.21.15.46:0/3694922287 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9490004490 con 0x7f94981069c0 2024-03-31T23:02:14.729 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:14.011+0000 7f949c835700 1 -- 172.21.15.46:0/3694922287 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9490031060 con 0x7f94981069c0 2024-03-31T23:02:14.729 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:14.012+0000 7f949c835700 1 -- 172.21.15.46:0/3694922287 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 12) ==== 46129+0+0 (secure 0 0 0) 0x7f949002a070 con 0x7f94981069c0 2024-03-31T23:02:14.729 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:14.012+0000 7f949c835700 1 --2- 172.21.15.46:0/3694922287 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f9484038e50 0x7f948403b310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:02:14.729 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:14.012+0000 7f949d837700 1 -- 172.21.15.46:0/3694922287 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9478005360 con 0x7f94981069c0 2024-03-31T23:02:14.729 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:14.012+0000 7f949c835700 1 -- 172.21.15.46:0/3694922287 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f949001f070 con 0x7f94981069c0 2024-03-31T23:02:14.729 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:14.013+0000 7f94967fc700 1 --2- 172.21.15.46:0/3694922287 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f9484038e50 0x7f948403b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:02:14.729 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:14.013+0000 7f94967fc700 1 --2- 172.21.15.46:0/3694922287 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f9484038e50 0x7f948403b310 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f9480006fd0 tx=0x7f9480006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:02:14.729 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:14.017+0000 7f949c835700 1 -- 172.21.15.46:0/3694922287 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f9490015070 con 0x7f94981069c0 2024-03-31T23:02:14.729 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:14.220+0000 7f949d837700 1 -- 172.21.15.46:0/3694922287 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) -- 0x7f9478005650 con 0x7f94981069c0 2024-03-31T23:02:14.729 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:14.220+0000 7f949c835700 1 -- 172.21.15.46:0/3694922287 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v8) ==== 112+0+5 (secure 0 0 0) 0x7f949000f600 con 0x7f94981069c0 2024-03-31T23:02:14.729 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:14.222+0000 7f949d837700 1 -- 172.21.15.46:0/3694922287 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f9484038e50 msgr2=0x7f948403b310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:02:14.730 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:14.222+0000 7f949d837700 1 --2- 172.21.15.46:0/3694922287 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f9484038e50 0x7f948403b310 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f9480006fd0 tx=0x7f9480006e40 comp rx=0 tx=0).stop 2024-03-31T23:02:14.730 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:14.222+0000 7f949d837700 1 -- 172.21.15.46:0/3694922287 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f94981069c0 msgr2=0x7f9498199800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:02:14.730 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:14.222+0000 7f949d837700 1 --2- 172.21.15.46:0/3694922287 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f94981069c0 0x7f9498199800 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f949000b340 tx=0x7f9490003fb0 comp rx=0 tx=0).stop 2024-03-31T23:02:14.730 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:14.222+0000 7f949d837700 1 -- 172.21.15.46:0/3694922287 shutdown_connections 2024-03-31T23:02:14.730 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:14.222+0000 7f949d837700 1 --2- 172.21.15.46:0/3694922287 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f9484038e50 0x7f948403b310 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:02:14.730 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:14.222+0000 7f949d837700 1 --2- 172.21.15.46:0/3694922287 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f94981069c0 0x7f9498199800 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:02:14.730 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:14.222+0000 7f949d837700 1 -- 172.21.15.46:0/3694922287 >> 172.21.15.46:0/3694922287 conn(0x7f9498101f60 msgr2=0x7f949818fe40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:02:14.730 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:14.222+0000 7f949d837700 1 -- 172.21.15.46:0/3694922287 shutdown_connections 2024-03-31T23:02:14.730 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:14.222+0000 7f949d837700 1 -- 172.21.15.46:0/3694922287 wait complete. 2024-03-31T23:02:14.733 INFO:teuthology.orchestra.run.smithi046.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-31T23:02:14.733 INFO:teuthology.orchestra.run.smithi046.stdout:systemctl: stdout disabled 2024-03-31T23:02:14.740 INFO:teuthology.orchestra.run.smithi046.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-31T23:02:14.740 INFO:teuthology.orchestra.run.smithi046.stdout:systemctl: stdout inactive 2024-03-31T23:02:14.741 INFO:teuthology.orchestra.run.smithi046.stdout:firewalld.service is not enabled 2024-03-31T23:02:14.741 INFO:teuthology.orchestra.run.smithi046.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-03-31T23:02:14.742 INFO:teuthology.orchestra.run.smithi046.stdout:Ceph Dashboard is now available at: 2024-03-31T23:02:14.742 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-31T23:02:14.742 INFO:teuthology.orchestra.run.smithi046.stdout: URL: https://smithi046.front.sepia.ceph.com:8443/ 2024-03-31T23:02:14.742 INFO:teuthology.orchestra.run.smithi046.stdout: User: admin 2024-03-31T23:02:14.742 INFO:teuthology.orchestra.run.smithi046.stdout: Password: uxcrfyn2lx 2024-03-31T23:02:14.742 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-31T23:02:14.743 INFO:teuthology.orchestra.run.smithi046.stdout:Saving cluster configuration to /var/lib/ceph/6a9d8f08-efb2-11ee-b647-cb9ed24678a4/config directory 2024-03-31T23:02:14.743 INFO:teuthology.orchestra.run.smithi046.stdout:Enabling autotune for osd_memory_target 2024-03-31T23:02:15.874 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:15.175+0000 7f2c0128c700 1 Processor -- start 2024-03-31T23:02:15.875 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:15.175+0000 7f2c0128c700 1 -- start start 2024-03-31T23:02:15.875 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:15.175+0000 7f2c0128c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2bfc1063a0 0x7f2bfc1067c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:02:15.875 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:15.175+0000 7f2c0128c700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f2bfc07bdc0 con 0x7f2bfc1063a0 2024-03-31T23:02:15.875 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:15.175+0000 7f2bfaffd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2bfc1063a0 0x7f2bfc1067c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:02:15.875 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:15.175+0000 7f2bfaffd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2bfc1063a0 0x7f2bfc1067c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:50484/0 (socket says 172.21.15.46:50484) 2024-03-31T23:02:15.875 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:15.175+0000 7f2bfaffd700 1 -- 172.21.15.46:0/1800371286 learned_addr learned my addr 172.21.15.46:0/1800371286 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:02:15.875 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:15.176+0000 7f2bfaffd700 1 -- 172.21.15.46:0/1800371286 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2bfc106d00 con 0x7f2bfc1063a0 2024-03-31T23:02:15.875 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:15.176+0000 7f2bfaffd700 1 --2- 172.21.15.46:0/1800371286 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2bfc1063a0 0x7f2bfc1067c0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f2bf4009e80 tx=0x7f2bf400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=259cee5f2d582e57 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:02:15.875 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:15.176+0000 7f2bf9ffb700 1 -- 172.21.15.46:0/1800371286 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2bf4017070 con 0x7f2bfc1063a0 2024-03-31T23:02:15.875 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:15.176+0000 7f2bf9ffb700 1 -- 172.21.15.46:0/1800371286 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f2bf400b8c0 con 0x7f2bfc1063a0 2024-03-31T23:02:15.875 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:15.176+0000 7f2bf9ffb700 1 -- 172.21.15.46:0/1800371286 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2bf4004490 con 0x7f2bfc1063a0 2024-03-31T23:02:15.875 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:15.176+0000 7f2c0128c700 1 -- 172.21.15.46:0/1800371286 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2bfc1063a0 msgr2=0x7f2bfc1067c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:02:15.876 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:15.176+0000 7f2c0128c700 1 --2- 172.21.15.46:0/1800371286 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2bfc1063a0 0x7f2bfc1067c0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f2bf4009e80 tx=0x7f2bf400b330 comp rx=0 tx=0).stop 2024-03-31T23:02:15.876 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:15.176+0000 7f2c0128c700 1 -- 172.21.15.46:0/1800371286 shutdown_connections 2024-03-31T23:02:15.876 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:15.176+0000 7f2c0128c700 1 --2- 172.21.15.46:0/1800371286 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2bfc1063a0 0x7f2bfc1067c0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:02:15.876 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:15.177+0000 7f2c0128c700 1 -- 172.21.15.46:0/1800371286 >> 172.21.15.46:0/1800371286 conn(0x7f2bfc101f80 msgr2=0x7f2bfc1043a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:02:15.876 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:15.177+0000 7f2c0128c700 1 -- 172.21.15.46:0/1800371286 shutdown_connections 2024-03-31T23:02:15.876 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:15.177+0000 7f2c0128c700 1 -- 172.21.15.46:0/1800371286 wait complete. 2024-03-31T23:02:15.876 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:15.177+0000 7f2c0128c700 1 Processor -- start 2024-03-31T23:02:15.876 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:15.177+0000 7f2c0128c700 1 -- start start 2024-03-31T23:02:15.876 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:15.177+0000 7f2c0128c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2bfc1063a0 0x7f2bfc19dba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:02:15.876 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:15.178+0000 7f2c0128c700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f2bfc107d70 con 0x7f2bfc1063a0 2024-03-31T23:02:15.876 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:15.178+0000 7f2bfaffd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2bfc1063a0 0x7f2bfc19dba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:02:15.876 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:15.178+0000 7f2bfaffd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2bfc1063a0 0x7f2bfc19dba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:50500/0 (socket says 172.21.15.46:50500) 2024-03-31T23:02:15.876 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:15.178+0000 7f2bfaffd700 1 -- 172.21.15.46:0/2303526455 learned_addr learned my addr 172.21.15.46:0/2303526455 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:02:15.877 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:15.178+0000 7f2bfaffd700 1 -- 172.21.15.46:0/2303526455 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2bfc19e0e0 con 0x7f2bfc1063a0 2024-03-31T23:02:15.877 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:15.178+0000 7f2bfaffd700 1 --2- 172.21.15.46:0/2303526455 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2bfc1063a0 0x7f2bfc19dba0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f2bf4009e50 tx=0x7f2bf4003d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:02:15.877 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:15.178+0000 7f2bf3fff700 1 -- 172.21.15.46:0/2303526455 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2bf4020020 con 0x7f2bfc1063a0 2024-03-31T23:02:15.877 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:15.178+0000 7f2bf3fff700 1 -- 172.21.15.46:0/2303526455 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f2bf400bba0 con 0x7f2bfc1063a0 2024-03-31T23:02:15.877 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:15.178+0000 7f2c0128c700 1 -- 172.21.15.46:0/2303526455 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2bfc19e370 con 0x7f2bfc1063a0 2024-03-31T23:02:15.877 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:15.178+0000 7f2bf3fff700 1 -- 172.21.15.46:0/2303526455 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2bf4017040 con 0x7f2bfc1063a0 2024-03-31T23:02:15.877 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:15.178+0000 7f2c0128c700 1 -- 172.21.15.46:0/2303526455 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2bfc1a1060 con 0x7f2bfc1063a0 2024-03-31T23:02:15.877 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:15.179+0000 7f2bf3fff700 1 -- 172.21.15.46:0/2303526455 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 12) ==== 46129+0+0 (secure 0 0 0) 0x7f2bf4023020 con 0x7f2bfc1063a0 2024-03-31T23:02:15.877 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:15.179+0000 7f2bf3fff700 1 --2- 172.21.15.46:0/2303526455 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f2be8038e00 0x7f2be803b2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:02:15.877 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:15.179+0000 7f2bf3fff700 1 -- 172.21.15.46:0/2303526455 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f2bf400f050 con 0x7f2bfc1063a0 2024-03-31T23:02:15.877 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:15.179+0000 7f2c0128c700 1 -- 172.21.15.46:0/2303526455 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2bfc063870 con 0x7f2bfc1063a0 2024-03-31T23:02:15.877 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:15.179+0000 7f2bfa7fc700 1 --2- 172.21.15.46:0/2303526455 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f2be8038e00 0x7f2be803b2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:02:15.877 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:15.180+0000 7f2bfa7fc700 1 --2- 172.21.15.46:0/2303526455 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f2be8038e00 0x7f2be803b2c0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f2be4006fd0 tx=0x7f2be4006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:02:15.877 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:15.184+0000 7f2bf3fff700 1 -- 172.21.15.46:0/2303526455 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f2bf402a020 con 0x7f2bfc1063a0 2024-03-31T23:02:15.878 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:15.273+0000 7f2bf3fff700 1 -- 172.21.15.46:0/2303526455 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mgrmap(e 13) ==== 46172+0+0 (secure 0 0 0) 0x7f2bf4025400 con 0x7f2bfc1063a0 2024-03-31T23:02:15.878 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:15.389+0000 7f2c0128c700 1 -- 172.21.15.46:0/2303526455 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command([{prefix=config set, name=osd_memory_target_autotune}] v 0) -- 0x7f2bfc02ddb0 con 0x7f2bfc1063a0 2024-03-31T23:02:15.878 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:15.390+0000 7f2bf3fff700 1 -- 172.21.15.46:0/2303526455 <== mon.0 v2:172.21.15.46:3300/0 8 ==== mon_command_ack([{prefix=config set, name=osd_memory_target_autotune}]=0 v8)=0 v8) ==== 127+0+0 (secure 0 0 0) 0x7f2bf4018240 con 0x7f2bfc1063a0 2024-03-31T23:02:15.878 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:15.391+0000 7f2c0128c700 1 -- 172.21.15.46:0/2303526455 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f2be8038e00 msgr2=0x7f2be803b2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:02:15.878 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:15.392+0000 7f2c0128c700 1 --2- 172.21.15.46:0/2303526455 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f2be8038e00 0x7f2be803b2c0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f2be4006fd0 tx=0x7f2be4006e40 comp rx=0 tx=0).stop 2024-03-31T23:02:15.878 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:15.392+0000 7f2c0128c700 1 -- 172.21.15.46:0/2303526455 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2bfc1063a0 msgr2=0x7f2bfc19dba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:02:15.878 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:15.392+0000 7f2c0128c700 1 --2- 172.21.15.46:0/2303526455 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2bfc1063a0 0x7f2bfc19dba0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f2bf4009e50 tx=0x7f2bf4003d00 comp rx=0 tx=0).stop 2024-03-31T23:02:15.879 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:15.392+0000 7f2c0128c700 1 -- 172.21.15.46:0/2303526455 shutdown_connections 2024-03-31T23:02:15.879 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:15.392+0000 7f2c0128c700 1 --2- 172.21.15.46:0/2303526455 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f2be8038e00 0x7f2be803b2c0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:02:15.879 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:15.392+0000 7f2c0128c700 1 --2- 172.21.15.46:0/2303526455 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2bfc1063a0 0x7f2bfc19dba0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:02:15.879 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:15.392+0000 7f2c0128c700 1 -- 172.21.15.46:0/2303526455 >> 172.21.15.46:0/2303526455 conn(0x7f2bfc101f80 msgr2=0x7f2bfc1947c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:02:15.879 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:15.392+0000 7f2c0128c700 1 -- 172.21.15.46:0/2303526455 shutdown_connections 2024-03-31T23:02:15.879 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:15.392+0000 7f2c0128c700 1 -- 172.21.15.46:0/2303526455 wait complete. 2024-03-31T23:02:16.361 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:02:16 smithi046 ceph-mon[32498]: mgrmap e13: x(active, since 6s) 2024-03-31T23:02:17.298 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:16.276+0000 7fa8099ac700 1 Processor -- start 2024-03-31T23:02:17.298 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:16.276+0000 7fa8099ac700 1 -- start start 2024-03-31T23:02:17.298 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:16.277+0000 7fa8099ac700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa804105d70 0x7fa804106190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:02:17.298 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:16.277+0000 7fa8099ac700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fa804106760 con 0x7fa804105d70 2024-03-31T23:02:17.298 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:16.277+0000 7fa802ffd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa804105d70 0x7fa804106190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:02:17.298 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:16.277+0000 7fa802ffd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa804105d70 0x7fa804106190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:50502/0 (socket says 172.21.15.46:50502) 2024-03-31T23:02:17.299 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:16.277+0000 7fa802ffd700 1 -- 172.21.15.46:0/3111935256 learned_addr learned my addr 172.21.15.46:0/3111935256 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:02:17.299 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:16.277+0000 7fa802ffd700 1 -- 172.21.15.46:0/3111935256 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa804108d00 con 0x7fa804105d70 2024-03-31T23:02:17.299 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:16.277+0000 7fa802ffd700 1 --2- 172.21.15.46:0/3111935256 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa804105d70 0x7fa804106190 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fa7f40098a0 tx=0x7fa7f4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f0ecb0f4c2fc4f16 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:02:17.299 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:16.277+0000 7fa801ffb700 1 -- 172.21.15.46:0/3111935256 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa7f4004160 con 0x7fa804105d70 2024-03-31T23:02:17.299 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:16.277+0000 7fa801ffb700 1 -- 172.21.15.46:0/3111935256 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa7f4004300 con 0x7fa804105d70 2024-03-31T23:02:17.299 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:16.277+0000 7fa801ffb700 1 -- 172.21.15.46:0/3111935256 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa7f4011a50 con 0x7fa804105d70 2024-03-31T23:02:17.299 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:16.278+0000 7fa8099ac700 1 -- 172.21.15.46:0/3111935256 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa804105d70 msgr2=0x7fa804106190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:02:17.299 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:16.278+0000 7fa8099ac700 1 --2- 172.21.15.46:0/3111935256 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa804105d70 0x7fa804106190 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fa7f40098a0 tx=0x7fa7f4009bb0 comp rx=0 tx=0).stop 2024-03-31T23:02:17.299 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:16.278+0000 7fa8099ac700 1 -- 172.21.15.46:0/3111935256 shutdown_connections 2024-03-31T23:02:17.299 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:16.278+0000 7fa8099ac700 1 --2- 172.21.15.46:0/3111935256 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa804105d70 0x7fa804106190 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:02:17.299 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:16.278+0000 7fa8099ac700 1 -- 172.21.15.46:0/3111935256 >> 172.21.15.46:0/3111935256 conn(0x7fa804101720 msgr2=0x7fa804103b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:02:17.299 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:16.278+0000 7fa8099ac700 1 -- 172.21.15.46:0/3111935256 shutdown_connections 2024-03-31T23:02:17.299 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:16.278+0000 7fa8099ac700 1 -- 172.21.15.46:0/3111935256 wait complete. 2024-03-31T23:02:17.300 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:16.279+0000 7fa8099ac700 1 Processor -- start 2024-03-31T23:02:17.300 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:16.279+0000 7fa8099ac700 1 -- start start 2024-03-31T23:02:17.300 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:16.279+0000 7fa8099ac700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa804105d70 0x7fa804108580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:02:17.300 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:16.279+0000 7fa8099ac700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fa80410c0b0 con 0x7fa804105d70 2024-03-31T23:02:17.300 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:16.279+0000 7fa802ffd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa804105d70 0x7fa804108580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:02:17.300 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:16.279+0000 7fa802ffd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa804105d70 0x7fa804108580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:50508/0 (socket says 172.21.15.46:50508) 2024-03-31T23:02:17.300 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:16.279+0000 7fa802ffd700 1 -- 172.21.15.46:0/1583859213 learned_addr learned my addr 172.21.15.46:0/1583859213 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:02:17.300 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:16.279+0000 7fa802ffd700 1 -- 172.21.15.46:0/1583859213 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa804108ac0 con 0x7fa804105d70 2024-03-31T23:02:17.300 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:16.280+0000 7fa802ffd700 1 --2- 172.21.15.46:0/1583859213 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa804105d70 0x7fa804108580 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fa7f4011a20 tx=0x7fa7f4011b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:02:17.300 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:16.280+0000 7fa8089aa700 1 -- 172.21.15.46:0/1583859213 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa7f4003af0 con 0x7fa804105d70 2024-03-31T23:02:17.300 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:16.280+0000 7fa8089aa700 1 -- 172.21.15.46:0/1583859213 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa7f401a8e0 con 0x7fa804105d70 2024-03-31T23:02:17.300 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:16.280+0000 7fa8099ac700 1 -- 172.21.15.46:0/1583859213 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa804106cb0 con 0x7fa804105d70 2024-03-31T23:02:17.300 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:16.280+0000 7fa8089aa700 1 -- 172.21.15.46:0/1583859213 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa7f4024860 con 0x7fa804105d70 2024-03-31T23:02:17.300 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:16.280+0000 7fa8099ac700 1 -- 172.21.15.46:0/1583859213 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa8041071f0 con 0x7fa804105d70 2024-03-31T23:02:17.301 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:16.281+0000 7fa8089aa700 1 -- 172.21.15.46:0/1583859213 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 13) ==== 46172+0+0 (secure 0 0 0) 0x7fa7f4022020 con 0x7fa804105d70 2024-03-31T23:02:17.301 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:16.281+0000 7fa8089aa700 1 --2- 172.21.15.46:0/1583859213 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fa7ec038f50 0x7fa7ec03b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:02:17.301 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:16.281+0000 7fa8099ac700 1 -- 172.21.15.46:0/1583859213 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa804063870 con 0x7fa804105d70 2024-03-31T23:02:17.301 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:16.281+0000 7fa8089aa700 1 -- 172.21.15.46:0/1583859213 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fa7f404d1e0 con 0x7fa804105d70 2024-03-31T23:02:17.301 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:16.281+0000 7fa8027fc700 1 --2- 172.21.15.46:0/1583859213 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fa7ec038f50 0x7fa7ec03b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:02:17.301 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:16.281+0000 7fa8027fc700 1 --2- 172.21.15.46:0/1583859213 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fa7ec038f50 0x7fa7ec03b410 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fa7f8006fd0 tx=0x7fa7f8006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:02:17.301 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:16.286+0000 7fa8089aa700 1 -- 172.21.15.46:0/1583859213 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fa7f400b3e0 con 0x7fa804105d70 2024-03-31T23:02:17.301 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:16.602+0000 7fa8099ac700 1 -- 172.21.15.46:0/1583859213 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command([{prefix=config-key set, key=mgr/dashboard/cluster/status}] v 0) -- 0x7fa804107470 con 0x7fa804105d70 2024-03-31T23:02:17.301 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:16.604+0000 7fa8089aa700 1 -- 172.21.15.46:0/1583859213 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{prefix=config-key set, key=mgr/dashboard/cluster/status}]=0 set mgr/dashboard/cluster/status v28)=0 set mgr/dashboard/cluster/status v28) ==== 153+0+0 (secure 0 0 0) 0x7fa7f4016df0 con 0x7fa804105d70 2024-03-31T23:02:17.301 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:16.606+0000 7fa8099ac700 1 -- 172.21.15.46:0/1583859213 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fa7ec038f50 msgr2=0x7fa7ec03b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:02:17.301 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:16.606+0000 7fa8099ac700 1 --2- 172.21.15.46:0/1583859213 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fa7ec038f50 0x7fa7ec03b410 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fa7f8006fd0 tx=0x7fa7f8006e40 comp rx=0 tx=0).stop 2024-03-31T23:02:17.301 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:16.606+0000 7fa8099ac700 1 -- 172.21.15.46:0/1583859213 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa804105d70 msgr2=0x7fa804108580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:02:17.301 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:16.606+0000 7fa8099ac700 1 --2- 172.21.15.46:0/1583859213 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa804105d70 0x7fa804108580 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fa7f4011a20 tx=0x7fa7f4011b00 comp rx=0 tx=0).stop 2024-03-31T23:02:17.302 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:16.606+0000 7fa8099ac700 1 -- 172.21.15.46:0/1583859213 shutdown_connections 2024-03-31T23:02:17.302 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:16.606+0000 7fa8099ac700 1 --2- 172.21.15.46:0/1583859213 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fa7ec038f50 0x7fa7ec03b410 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:02:17.302 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:16.606+0000 7fa8099ac700 1 --2- 172.21.15.46:0/1583859213 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa804105d70 0x7fa804108580 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:02:17.302 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:16.606+0000 7fa8099ac700 1 -- 172.21.15.46:0/1583859213 >> 172.21.15.46:0/1583859213 conn(0x7fa804101720 msgr2=0x7fa8041021c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:02:17.302 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:16.606+0000 7fa8099ac700 1 -- 172.21.15.46:0/1583859213 shutdown_connections 2024-03-31T23:02:17.302 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-31T23:02:16.606+0000 7fa8099ac700 1 -- 172.21.15.46:0/1583859213 wait complete. 2024-03-31T23:02:17.302 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-03-31T23:02:17.302 INFO:teuthology.orchestra.run.smithi046.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-03-31T23:02:17.302 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-31T23:02:17.302 INFO:teuthology.orchestra.run.smithi046.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid 6a9d8f08-efb2-11ee-b647-cb9ed24678a4 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-03-31T23:02:17.302 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-31T23:02:17.302 INFO:teuthology.orchestra.run.smithi046.stdout:Or, if you are only running a single cluster on this host: 2024-03-31T23:02:17.302 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-31T23:02:17.302 INFO:teuthology.orchestra.run.smithi046.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-03-31T23:02:17.303 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-31T23:02:17.303 INFO:teuthology.orchestra.run.smithi046.stdout:Please consider enabling telemetry to help improve Ceph: 2024-03-31T23:02:17.303 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-31T23:02:17.303 INFO:teuthology.orchestra.run.smithi046.stdout: ceph telemetry on 2024-03-31T23:02:17.303 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-31T23:02:17.303 INFO:teuthology.orchestra.run.smithi046.stdout:For more information see: 2024-03-31T23:02:17.303 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-31T23:02:17.303 INFO:teuthology.orchestra.run.smithi046.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2024-03-31T23:02:17.303 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-31T23:02:17.303 INFO:teuthology.orchestra.run.smithi046.stdout:Bootstrap complete. 2024-03-31T23:02:17.346 INFO:tasks.cephadm:Fetching config... 2024-03-31T23:02:17.346 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-03-31T23:02:17.346 DEBUG:teuthology.orchestra.run.smithi046:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-03-31T23:02:17.363 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-03-31T23:02:17.363 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-03-31T23:02:17.363 DEBUG:teuthology.orchestra.run.smithi046:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-03-31T23:02:17.418 INFO:tasks.cephadm:Fetching mon keyring... 2024-03-31T23:02:17.418 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-03-31T23:02:17.419 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd if=/var/lib/ceph/6a9d8f08-efb2-11ee-b647-cb9ed24678a4/mon.a/keyring of=/dev/stdout 2024-03-31T23:02:17.486 INFO:tasks.cephadm:Fetching pub ssh key... 2024-03-31T23:02:17.487 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-03-31T23:02:17.487 DEBUG:teuthology.orchestra.run.smithi046:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-03-31T23:02:17.542 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-03-31T23:02:17.542 DEBUG:teuthology.orchestra.run.smithi046:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCzkmkRKY8+8KOyWRDL0P4BaTstsGt0MM+rdgXSbH1jnOhELHHP3TiD3LoNsEHEEYqgsDtg7o2JhZYYQ4/b0y2krNB4uA/677oQBAbbejHzCdiBrGiDT/YO9+EYdMQJcBNB/fBn28xUJos7GAa8c28wumU0lGg6VxBjJ8nrFDEKMeaLamKUEc2UtL+KUxTqNPrljzc9ow0oOrY2j2WuGzsRbfBzAWPhHdj9LHT6bjUeVTEdoxpvwG6K3sBxqponiCgixvUf0D8P/yZoiUiybDq/7Pb2lIKSuuqqR1jaaZerjJKk4uW4OTN2Bxl3i2bwzINOtSCxTMwvuijSVlr7o88kEoyOX4YnRDb+ftXYAqLbECwrrQSv27ltcXg4mXHLfNyTJe3l4uwm0Awlur01Z7tjtw37BtuEcS2eAy76aVF2Da+QZb1It27c25YfKPKr4IcyXl8xjD+XR4SlbZ/mek8cTREagI9+fQmcrP8WRsdJ+7N2pP1qbePYZWD2xfJHUec= ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-31T23:02:17.622 INFO:teuthology.orchestra.run.smithi046.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCzkmkRKY8+8KOyWRDL0P4BaTstsGt0MM+rdgXSbH1jnOhELHHP3TiD3LoNsEHEEYqgsDtg7o2JhZYYQ4/b0y2krNB4uA/677oQBAbbejHzCdiBrGiDT/YO9+EYdMQJcBNB/fBn28xUJos7GAa8c28wumU0lGg6VxBjJ8nrFDEKMeaLamKUEc2UtL+KUxTqNPrljzc9ow0oOrY2j2WuGzsRbfBzAWPhHdj9LHT6bjUeVTEdoxpvwG6K3sBxqponiCgixvUf0D8P/yZoiUiybDq/7Pb2lIKSuuqqR1jaaZerjJKk4uW4OTN2Bxl3i2bwzINOtSCxTMwvuijSVlr7o88kEoyOX4YnRDb+ftXYAqLbECwrrQSv27ltcXg4mXHLfNyTJe3l4uwm0Awlur01Z7tjtw37BtuEcS2eAy76aVF2Da+QZb1It27c25YfKPKr4IcyXl8xjD+XR4SlbZ/mek8cTREagI9+fQmcrP8WRsdJ+7N2pP1qbePYZWD2xfJHUec= ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4 2024-03-31T23:02:17.636 DEBUG:teuthology.orchestra.run.smithi142:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCzkmkRKY8+8KOyWRDL0P4BaTstsGt0MM+rdgXSbH1jnOhELHHP3TiD3LoNsEHEEYqgsDtg7o2JhZYYQ4/b0y2krNB4uA/677oQBAbbejHzCdiBrGiDT/YO9+EYdMQJcBNB/fBn28xUJos7GAa8c28wumU0lGg6VxBjJ8nrFDEKMeaLamKUEc2UtL+KUxTqNPrljzc9ow0oOrY2j2WuGzsRbfBzAWPhHdj9LHT6bjUeVTEdoxpvwG6K3sBxqponiCgixvUf0D8P/yZoiUiybDq/7Pb2lIKSuuqqR1jaaZerjJKk4uW4OTN2Bxl3i2bwzINOtSCxTMwvuijSVlr7o88kEoyOX4YnRDb+ftXYAqLbECwrrQSv27ltcXg4mXHLfNyTJe3l4uwm0Awlur01Z7tjtw37BtuEcS2eAy76aVF2Da+QZb1It27c25YfKPKr4IcyXl8xjD+XR4SlbZ/mek8cTREagI9+fQmcrP8WRsdJ+7N2pP1qbePYZWD2xfJHUec= ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-31T23:02:17.682 INFO:teuthology.orchestra.run.smithi142.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCzkmkRKY8+8KOyWRDL0P4BaTstsGt0MM+rdgXSbH1jnOhELHHP3TiD3LoNsEHEEYqgsDtg7o2JhZYYQ4/b0y2krNB4uA/677oQBAbbejHzCdiBrGiDT/YO9+EYdMQJcBNB/fBn28xUJos7GAa8c28wumU0lGg6VxBjJ8nrFDEKMeaLamKUEc2UtL+KUxTqNPrljzc9ow0oOrY2j2WuGzsRbfBzAWPhHdj9LHT6bjUeVTEdoxpvwG6K3sBxqponiCgixvUf0D8P/yZoiUiybDq/7Pb2lIKSuuqqR1jaaZerjJKk4uW4OTN2Bxl3i2bwzINOtSCxTMwvuijSVlr7o88kEoyOX4YnRDb+ftXYAqLbECwrrQSv27ltcXg4mXHLfNyTJe3l4uwm0Awlur01Z7tjtw37BtuEcS2eAy76aVF2Da+QZb1It27c25YfKPKr4IcyXl8xjD+XR4SlbZ/mek8cTREagI9+fQmcrP8WRsdJ+7N2pP1qbePYZWD2xfJHUec= ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4 2024-03-31T23:02:17.697 DEBUG:teuthology.orchestra.run.smithi162:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCzkmkRKY8+8KOyWRDL0P4BaTstsGt0MM+rdgXSbH1jnOhELHHP3TiD3LoNsEHEEYqgsDtg7o2JhZYYQ4/b0y2krNB4uA/677oQBAbbejHzCdiBrGiDT/YO9+EYdMQJcBNB/fBn28xUJos7GAa8c28wumU0lGg6VxBjJ8nrFDEKMeaLamKUEc2UtL+KUxTqNPrljzc9ow0oOrY2j2WuGzsRbfBzAWPhHdj9LHT6bjUeVTEdoxpvwG6K3sBxqponiCgixvUf0D8P/yZoiUiybDq/7Pb2lIKSuuqqR1jaaZerjJKk4uW4OTN2Bxl3i2bwzINOtSCxTMwvuijSVlr7o88kEoyOX4YnRDb+ftXYAqLbECwrrQSv27ltcXg4mXHLfNyTJe3l4uwm0Awlur01Z7tjtw37BtuEcS2eAy76aVF2Da+QZb1It27c25YfKPKr4IcyXl8xjD+XR4SlbZ/mek8cTREagI9+fQmcrP8WRsdJ+7N2pP1qbePYZWD2xfJHUec= ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-31T23:02:17.742 INFO:teuthology.orchestra.run.smithi162.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCzkmkRKY8+8KOyWRDL0P4BaTstsGt0MM+rdgXSbH1jnOhELHHP3TiD3LoNsEHEEYqgsDtg7o2JhZYYQ4/b0y2krNB4uA/677oQBAbbejHzCdiBrGiDT/YO9+EYdMQJcBNB/fBn28xUJos7GAa8c28wumU0lGg6VxBjJ8nrFDEKMeaLamKUEc2UtL+KUxTqNPrljzc9ow0oOrY2j2WuGzsRbfBzAWPhHdj9LHT6bjUeVTEdoxpvwG6K3sBxqponiCgixvUf0D8P/yZoiUiybDq/7Pb2lIKSuuqqR1jaaZerjJKk4uW4OTN2Bxl3i2bwzINOtSCxTMwvuijSVlr7o88kEoyOX4YnRDb+ftXYAqLbECwrrQSv27ltcXg4mXHLfNyTJe3l4uwm0Awlur01Z7tjtw37BtuEcS2eAy76aVF2Da+QZb1It27c25YfKPKr4IcyXl8xjD+XR4SlbZ/mek8cTREagI9+fQmcrP8WRsdJ+7N2pP1qbePYZWD2xfJHUec= ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4 2024-03-31T23:02:17.757 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:384f31f3d3aee7f8ba3f60846b8844c049165583 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6a9d8f08-efb2-11ee-b647-cb9ed24678a4 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-03-31T23:02:17.834 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:02:17 smithi046 ceph-mon[32498]: from='client.? 172.21.15.46:0/1583859213' entity='client.admin' 2024-03-31T23:02:17.999 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/6a9d8f08-efb2-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-31T23:02:20.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:02:20.103+0000 7f2e613e8700 1 -- 172.21.15.46:0/2809118566 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2e5c102780 msgr2=0x7f2e5c104b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:02:20.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:02:20.103+0000 7f2e613e8700 1 --2- 172.21.15.46:0/2809118566 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2e5c102780 0x7f2e5c104b70 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f2e4c009910 tx=0x7f2e4c009c20 comp rx=0 tx=0).stop 2024-03-31T23:02:20.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:02:20.103+0000 7f2e613e8700 1 -- 172.21.15.46:0/2809118566 shutdown_connections 2024-03-31T23:02:20.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:02:20.103+0000 7f2e613e8700 1 --2- 172.21.15.46:0/2809118566 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2e5c102780 0x7f2e5c104b70 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:02:20.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:02:20.103+0000 7f2e613e8700 1 -- 172.21.15.46:0/2809118566 >> 172.21.15.46:0/2809118566 conn(0x7f2e5c0fc150 msgr2=0x7f2e5c0fe570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:02:20.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:02:20.103+0000 7f2e613e8700 1 -- 172.21.15.46:0/2809118566 shutdown_connections 2024-03-31T23:02:20.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:02:20.103+0000 7f2e613e8700 1 -- 172.21.15.46:0/2809118566 wait complete. 2024-03-31T23:02:20.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:02:20.104+0000 7f2e613e8700 1 Processor -- start 2024-03-31T23:02:20.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:02:20.104+0000 7f2e613e8700 1 -- start start 2024-03-31T23:02:20.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:02:20.104+0000 7f2e613e8700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2e5c102780 0x7f2e5c1972f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:02:20.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:02:20.104+0000 7f2e613e8700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f2e5c107be0 con 0x7f2e5c102780 2024-03-31T23:02:20.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:02:20.104+0000 7f2e5affd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2e5c102780 0x7f2e5c1972f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:02:20.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:02:20.104+0000 7f2e5affd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2e5c102780 0x7f2e5c1972f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:50532/0 (socket says 172.21.15.46:50532) 2024-03-31T23:02:20.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:02:20.104+0000 7f2e5affd700 1 -- 172.21.15.46:0/2269449039 learned_addr learned my addr 172.21.15.46:0/2269449039 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:02:20.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:02:20.104+0000 7f2e5affd700 1 -- 172.21.15.46:0/2269449039 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2e5c197830 con 0x7f2e5c102780 2024-03-31T23:02:20.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:02:20.105+0000 7f2e5affd700 1 --2- 172.21.15.46:0/2269449039 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2e5c102780 0x7f2e5c1972f0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f2e4c005190 tx=0x7f2e4c004f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:02:20.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:02:20.105+0000 7f2e43fff700 1 -- 172.21.15.46:0/2269449039 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2e4c027070 con 0x7f2e5c102780 2024-03-31T23:02:20.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:02:20.105+0000 7f2e43fff700 1 -- 172.21.15.46:0/2269449039 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f2e4c022470 con 0x7f2e5c102780 2024-03-31T23:02:20.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:02:20.105+0000 7f2e613e8700 1 -- 172.21.15.46:0/2269449039 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2e5c197ac0 con 0x7f2e5c102780 2024-03-31T23:02:20.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:02:20.105+0000 7f2e43fff700 1 -- 172.21.15.46:0/2269449039 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2e4c00f5f0 con 0x7f2e5c102780 2024-03-31T23:02:20.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:02:20.105+0000 7f2e613e8700 1 -- 172.21.15.46:0/2269449039 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2e5c19a760 con 0x7f2e5c102780 2024-03-31T23:02:20.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:02:20.106+0000 7f2e43fff700 1 -- 172.21.15.46:0/2269449039 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 13) ==== 46172+0+0 (secure 0 0 0) 0x7f2e4c02a070 con 0x7f2e5c102780 2024-03-31T23:02:20.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:02:20.106+0000 7f2e613e8700 1 -- 172.21.15.46:0/2269449039 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2e5c108360 con 0x7f2e5c102780 2024-03-31T23:02:20.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:02:20.106+0000 7f2e43fff700 1 --2- 172.21.15.46:0/2269449039 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f2e44038ea0 0x7f2e4403b360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:02:20.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:02:20.106+0000 7f2e43fff700 1 -- 172.21.15.46:0/2269449039 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f2e4c01e030 con 0x7f2e5c102780 2024-03-31T23:02:20.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:02:20.106+0000 7f2e5a7fc700 1 --2- 172.21.15.46:0/2269449039 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f2e44038ea0 0x7f2e4403b360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:02:20.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:02:20.106+0000 7f2e5a7fc700 1 --2- 172.21.15.46:0/2269449039 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f2e44038ea0 0x7f2e4403b360 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f2e50006fd0 tx=0x7f2e50006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:02:20.112 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:02:20.111+0000 7f2e43fff700 1 -- 172.21.15.46:0/2269449039 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f2e4c013070 con 0x7f2e5c102780 2024-03-31T23:02:20.280 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:02:20 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:02:20.280 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:02:20 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:02:20.280 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:02:20 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi046", "name": "osd_memory_target"}]: dispatch 2024-03-31T23:02:20.280 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:02:20 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:02:20.280 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:02:20 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-31T23:02:20.280 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:02:20 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:02:20.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:02:20.319+0000 7f2e613e8700 1 -- 172.21.15.46:0/2269449039 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) -- 0x7f2e5c02dc40 con 0x7f2e5c102780 2024-03-31T23:02:20.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:02:20.322+0000 7f2e43fff700 1 -- 172.21.15.46:0/2269449039 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v9)=0 v9) ==== 125+0+0 (secure 0 0 0) 0x7f2e4c0310c0 con 0x7f2e5c102780 2024-03-31T23:02:20.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:02:20.324+0000 7f2e613e8700 1 -- 172.21.15.46:0/2269449039 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f2e44038ea0 msgr2=0x7f2e4403b360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:02:20.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:02:20.324+0000 7f2e613e8700 1 --2- 172.21.15.46:0/2269449039 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f2e44038ea0 0x7f2e4403b360 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f2e50006fd0 tx=0x7f2e50006e40 comp rx=0 tx=0).stop 2024-03-31T23:02:20.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:02:20.324+0000 7f2e613e8700 1 -- 172.21.15.46:0/2269449039 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2e5c102780 msgr2=0x7f2e5c1972f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:02:20.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:02:20.324+0000 7f2e613e8700 1 --2- 172.21.15.46:0/2269449039 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2e5c102780 0x7f2e5c1972f0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f2e4c005190 tx=0x7f2e4c004f80 comp rx=0 tx=0).stop 2024-03-31T23:02:20.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:02:20.324+0000 7f2e613e8700 1 -- 172.21.15.46:0/2269449039 shutdown_connections 2024-03-31T23:02:20.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:02:20.324+0000 7f2e613e8700 1 --2- 172.21.15.46:0/2269449039 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f2e44038ea0 0x7f2e4403b360 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:02:20.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:02:20.324+0000 7f2e613e8700 1 --2- 172.21.15.46:0/2269449039 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2e5c102780 0x7f2e5c1972f0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:02:20.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:02:20.324+0000 7f2e613e8700 1 -- 172.21.15.46:0/2269449039 >> 172.21.15.46:0/2269449039 conn(0x7f2e5c0fc150 msgr2=0x7f2e5c0fcbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:02:20.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:02:20.324+0000 7f2e613e8700 1 -- 172.21.15.46:0/2269449039 shutdown_connections 2024-03-31T23:02:20.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:02:20.324+0000 7f2e613e8700 1 -- 172.21.15.46:0/2269449039 wait complete. 2024-03-31T23:02:21.201 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-03-31T23:02:21.202 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:384f31f3d3aee7f8ba3f60846b8844c049165583 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6a9d8f08-efb2-11ee-b647-cb9ed24678a4 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-03-31T23:02:21.452 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:02:21 smithi046 ceph-mon[32498]: from='client.? 172.21.15.46:0/2269449039' entity='client.admin' 2024-03-31T23:02:21.452 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:02:21 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-31T23:02:21.452 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:02:21 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:02:21.771 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/6a9d8f08-efb2-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-31T23:02:23.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:02:23.199+0000 7f681cbcd700 1 -- 172.21.15.46:0/3931303877 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6818103370 msgr2=0x7f6818103790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:02:23.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:02:23.199+0000 7f681cbcd700 1 --2- 172.21.15.46:0/3931303877 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6818103370 0x7f6818103790 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f6808009910 tx=0x7f6808009c20 comp rx=0 tx=0).stop 2024-03-31T23:02:23.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:02:23.199+0000 7f681cbcd700 1 -- 172.21.15.46:0/3931303877 shutdown_connections 2024-03-31T23:02:23.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:02:23.199+0000 7f681cbcd700 1 --2- 172.21.15.46:0/3931303877 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6818103370 0x7f6818103790 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:02:23.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:02:23.199+0000 7f681cbcd700 1 -- 172.21.15.46:0/3931303877 >> 172.21.15.46:0/3931303877 conn(0x7f68180fd230 msgr2=0x7f68180ff670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:02:23.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:02:23.200+0000 7f681cbcd700 1 -- 172.21.15.46:0/3931303877 shutdown_connections 2024-03-31T23:02:23.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:02:23.200+0000 7f681cbcd700 1 -- 172.21.15.46:0/3931303877 wait complete. 2024-03-31T23:02:23.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:02:23.200+0000 7f681cbcd700 1 Processor -- start 2024-03-31T23:02:23.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:02:23.200+0000 7f681cbcd700 1 -- start start 2024-03-31T23:02:23.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:02:23.201+0000 7f681cbcd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6818103370 0x7f6818197460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:02:23.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:02:23.201+0000 7f681cbcd700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f6818107ce0 con 0x7f6818103370 2024-03-31T23:02:23.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:02:23.201+0000 7f681659c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6818103370 0x7f6818197460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:02:23.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:02:23.201+0000 7f681659c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6818103370 0x7f6818197460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:34462/0 (socket says 172.21.15.46:34462) 2024-03-31T23:02:23.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:02:23.201+0000 7f681659c700 1 -- 172.21.15.46:0/3349869213 learned_addr learned my addr 172.21.15.46:0/3349869213 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:02:23.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:02:23.201+0000 7f681659c700 1 -- 172.21.15.46:0/3349869213 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f68181979a0 con 0x7f6818103370 2024-03-31T23:02:23.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:02:23.201+0000 7f681659c700 1 --2- 172.21.15.46:0/3349869213 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6818103370 0x7f6818197460 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f6808005800 tx=0x7f6808009f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:02:23.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:02:23.201+0000 7f68077fe700 1 -- 172.21.15.46:0/3349869213 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6808027070 con 0x7f6818103370 2024-03-31T23:02:23.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:02:23.201+0000 7f68077fe700 1 -- 172.21.15.46:0/3349869213 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f680800be10 con 0x7f6818103370 2024-03-31T23:02:23.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:02:23.201+0000 7f681cbcd700 1 -- 172.21.15.46:0/3349869213 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6818197c30 con 0x7f6818103370 2024-03-31T23:02:23.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:02:23.201+0000 7f68077fe700 1 -- 172.21.15.46:0/3349869213 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f680800f460 con 0x7f6818103370 2024-03-31T23:02:23.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:02:23.201+0000 7f681cbcd700 1 -- 172.21.15.46:0/3349869213 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f681819a920 con 0x7f6818103370 2024-03-31T23:02:23.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:02:23.202+0000 7f68077fe700 1 -- 172.21.15.46:0/3349869213 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 13) ==== 46172+0+0 (secure 0 0 0) 0x7f680802a070 con 0x7f6818103370 2024-03-31T23:02:23.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:02:23.202+0000 7f681cbcd700 1 -- 172.21.15.46:0/3349869213 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6818108460 con 0x7f6818103370 2024-03-31T23:02:23.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:02:23.202+0000 7f68077fe700 1 --2- 172.21.15.46:0/3349869213 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f6800038f00 0x7f680003b3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:02:23.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:02:23.202+0000 7f68077fe700 1 -- 172.21.15.46:0/3349869213 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f680801e030 con 0x7f6818103370 2024-03-31T23:02:23.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:02:23.203+0000 7f6815d9b700 1 --2- 172.21.15.46:0/3349869213 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f6800038f00 0x7f680003b3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:02:23.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:02:23.203+0000 7f6815d9b700 1 --2- 172.21.15.46:0/3349869213 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f6800038f00 0x7f680003b3c0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f680c006fd0 tx=0x7f680c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:02:23.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:02:23.208+0000 7f68077fe700 1 -- 172.21.15.46:0/3349869213 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f6808013070 con 0x7f6818103370 2024-03-31T23:02:23.420 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:02:23.419+0000 7f681cbcd700 1 -- 172.21.15.46:0/3349869213 --> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] -- mgr_command(tid 0: {"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}) -- 0x7f6818062680 con 0x7f6800038f00 2024-03-31T23:02:23.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:02:23.423+0000 7f68077fe700 1 -- 172.21.15.46:0/3349869213 <== mgr.14152 v2:172.21.15.46:6800/162289569 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f6818062680 con 0x7f6800038f00 2024-03-31T23:02:23.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:02:23.425+0000 7f681cbcd700 1 -- 172.21.15.46:0/3349869213 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f6800038f00 msgr2=0x7f680003b3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:02:23.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:02:23.425+0000 7f681cbcd700 1 --2- 172.21.15.46:0/3349869213 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f6800038f00 0x7f680003b3c0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f680c006fd0 tx=0x7f680c006e40 comp rx=0 tx=0).stop 2024-03-31T23:02:23.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:02:23.425+0000 7f681cbcd700 1 -- 172.21.15.46:0/3349869213 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6818103370 msgr2=0x7f6818197460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:02:23.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:02:23.425+0000 7f681cbcd700 1 --2- 172.21.15.46:0/3349869213 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6818103370 0x7f6818197460 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f6808005800 tx=0x7f6808009f60 comp rx=0 tx=0).stop 2024-03-31T23:02:23.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:02:23.425+0000 7f681cbcd700 1 -- 172.21.15.46:0/3349869213 shutdown_connections 2024-03-31T23:02:23.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:02:23.425+0000 7f681cbcd700 1 --2- 172.21.15.46:0/3349869213 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f6800038f00 0x7f680003b3c0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:02:23.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:02:23.425+0000 7f681cbcd700 1 --2- 172.21.15.46:0/3349869213 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6818103370 0x7f6818197460 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:02:23.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:02:23.425+0000 7f681cbcd700 1 -- 172.21.15.46:0/3349869213 >> 172.21.15.46:0/3349869213 conn(0x7f68180fd230 msgr2=0x7f68180fde20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:02:23.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:02:23.425+0000 7f681cbcd700 1 -- 172.21.15.46:0/3349869213 shutdown_connections 2024-03-31T23:02:23.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:02:23.425+0000 7f681cbcd700 1 -- 172.21.15.46:0/3349869213 wait complete. 2024-03-31T23:02:24.162 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi142 2024-03-31T23:02:24.162 DEBUG:teuthology.orchestra.run.smithi142:> set -ex 2024-03-31T23:02:24.162 DEBUG:teuthology.orchestra.run.smithi142:> dd of=/etc/ceph/ceph.conf 2024-03-31T23:02:24.180 DEBUG:teuthology.orchestra.run.smithi142:> set -ex 2024-03-31T23:02:24.181 DEBUG:teuthology.orchestra.run.smithi142:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-31T23:02:24.238 INFO:tasks.cephadm:Adding host smithi142 to orchestrator... 2024-03-31T23:02:24.238 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:384f31f3d3aee7f8ba3f60846b8844c049165583 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6a9d8f08-efb2-11ee-b647-cb9ed24678a4 -- ceph orch host add smithi142 2024-03-31T23:02:24.490 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/6a9d8f08-efb2-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-31T23:02:24.733 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:02:24 smithi046 ceph-mon[32498]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-03-31T23:02:24.733 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:02:24 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:02:24.733 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:02:24 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-31T23:02:24.733 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:02:24 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:02:24.733 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:02:24 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-31T23:02:24.733 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:02:24 smithi046 ceph-mon[32498]: Updating smithi046:/etc/ceph/ceph.conf 2024-03-31T23:02:24.733 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:02:24 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:02:24.734 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:02:24 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:02:24.734 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:02:24 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:02:25.674 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:02:25 smithi046 ceph-mon[32498]: Updating smithi046:/var/lib/ceph/6a9d8f08-efb2-11ee-b647-cb9ed24678a4/config/ceph.conf 2024-03-31T23:02:25.674 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:02:25 smithi046 ceph-mon[32498]: Updating smithi046:/etc/ceph/ceph.client.admin.keyring 2024-03-31T23:02:25.674 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:02:25 smithi046 ceph-mon[32498]: Updating smithi046:/var/lib/ceph/6a9d8f08-efb2-11ee-b647-cb9ed24678a4/config/ceph.client.admin.keyring 2024-03-31T23:02:25.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:02:25.986+0000 7fb7f2b34700 1 -- 172.21.15.46:0/3222188643 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb7ec106bf0 msgr2=0x7fb7ec106fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:02:25.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:02:25.986+0000 7fb7f2b34700 1 --2- 172.21.15.46:0/3222188643 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb7ec106bf0 0x7fb7ec106fd0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fb7e0009910 tx=0x7fb7e0009c20 comp rx=0 tx=0).stop 2024-03-31T23:02:25.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:02:25.986+0000 7fb7f2b34700 1 -- 172.21.15.46:0/3222188643 shutdown_connections 2024-03-31T23:02:25.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:02:25.986+0000 7fb7f2b34700 1 --2- 172.21.15.46:0/3222188643 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb7ec106bf0 0x7fb7ec106fd0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:02:25.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:02:25.986+0000 7fb7f2b34700 1 -- 172.21.15.46:0/3222188643 >> 172.21.15.46:0/3222188643 conn(0x7fb7ec0fc610 msgr2=0x7fb7ec0fea30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:02:25.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:02:25.986+0000 7fb7f2b34700 1 -- 172.21.15.46:0/3222188643 shutdown_connections 2024-03-31T23:02:25.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:02:25.987+0000 7fb7f2b34700 1 -- 172.21.15.46:0/3222188643 wait complete. 2024-03-31T23:02:25.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:02:25.987+0000 7fb7f2b34700 1 Processor -- start 2024-03-31T23:02:25.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:02:25.987+0000 7fb7f2b34700 1 -- start start 2024-03-31T23:02:25.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:02:25.987+0000 7fb7f2b34700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb7ec106bf0 0x7fb7ec0fcac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:02:25.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:02:25.987+0000 7fb7f2b34700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fb7ec10b060 con 0x7fb7ec106bf0 2024-03-31T23:02:25.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:02:25.987+0000 7fb7f08d0700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb7ec106bf0 0x7fb7ec0fcac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:02:25.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:02:25.987+0000 7fb7f08d0700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb7ec106bf0 0x7fb7ec0fcac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:34476/0 (socket says 172.21.15.46:34476) 2024-03-31T23:02:25.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:02:25.987+0000 7fb7f08d0700 1 -- 172.21.15.46:0/930101086 learned_addr learned my addr 172.21.15.46:0/930101086 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:02:25.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:02:25.988+0000 7fb7f08d0700 1 -- 172.21.15.46:0/930101086 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb7ec0fe9b0 con 0x7fb7ec106bf0 2024-03-31T23:02:25.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:02:25.988+0000 7fb7f08d0700 1 --2- 172.21.15.46:0/930101086 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb7ec106bf0 0x7fb7ec0fcac0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fb7e0006010 tx=0x7fb7e0005780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:02:25.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:02:25.988+0000 7fb7e9ffb700 1 -- 172.21.15.46:0/930101086 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb7e0026070 con 0x7fb7ec106bf0 2024-03-31T23:02:25.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:02:25.988+0000 7fb7e9ffb700 1 -- 172.21.15.46:0/930101086 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb7e000bc00 con 0x7fb7ec106bf0 2024-03-31T23:02:25.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:02:25.988+0000 7fb7f2b34700 1 -- 172.21.15.46:0/930101086 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb7ec0fd000 con 0x7fb7ec106bf0 2024-03-31T23:02:25.991 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:02:25.988+0000 7fb7e9ffb700 1 -- 172.21.15.46:0/930101086 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb7e0003930 con 0x7fb7ec106bf0 2024-03-31T23:02:25.992 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:02:25.988+0000 7fb7f2b34700 1 -- 172.21.15.46:0/930101086 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb7ec0fd4c0 con 0x7fb7ec106bf0 2024-03-31T23:02:25.992 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:02:25.989+0000 7fb7e9ffb700 1 -- 172.21.15.46:0/930101086 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 13) ==== 46172+0+0 (secure 0 0 0) 0x7fb7e0029070 con 0x7fb7ec106bf0 2024-03-31T23:02:25.992 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:02:25.989+0000 7fb7f2b34700 1 -- 172.21.15.46:0/930101086 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb7ec1021f0 con 0x7fb7ec106bf0 2024-03-31T23:02:25.992 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:02:25.989+0000 7fb7e9ffb700 1 --2- 172.21.15.46:0/930101086 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fb7dc038f00 0x7fb7dc03b3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:02:25.992 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:02:25.989+0000 7fb7e9ffb700 1 -- 172.21.15.46:0/930101086 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fb7e001e070 con 0x7fb7ec106bf0 2024-03-31T23:02:25.992 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:02:25.989+0000 7fb7ebfff700 1 --2- 172.21.15.46:0/930101086 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fb7dc038f00 0x7fb7dc03b3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:02:25.992 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:02:25.990+0000 7fb7ebfff700 1 --2- 172.21.15.46:0/930101086 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fb7dc038f00 0x7fb7dc03b3c0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fb7d8006fd0 tx=0x7fb7d8006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:02:25.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:02:25.994+0000 7fb7e9ffb700 1 -- 172.21.15.46:0/930101086 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fb7e0053080 con 0x7fb7ec106bf0 2024-03-31T23:02:26.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:02:26.203+0000 7fb7f2b34700 1 -- 172.21.15.46:0/930101086 --> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi142", "target": ["mon-mgr", ""]}) -- 0x7fb7ec0fe060 con 0x7fb7dc038f00 2024-03-31T23:02:26.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:02:26 smithi046 ceph-mon[32498]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi142", "target": ["mon-mgr", ""]}]: dispatch 2024-03-31T23:02:27.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:02:27 smithi046 ceph-mon[32498]: Deploying cephadm binary to smithi142 2024-03-31T23:02:29.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:02:29 smithi046 ceph-mon[32498]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-31T23:02:30.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:02:30.526+0000 7fb7e9ffb700 1 -- 172.21.15.46:0/930101086 <== mgr.14152 v2:172.21.15.46:6800/162289569 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+49 (secure 0 0 0) 0x7fb7ec0fe060 con 0x7fb7dc038f00 2024-03-31T23:02:30.528 INFO:teuthology.orchestra.run.smithi046.stdout:Added host 'smithi142' with addr '172.21.15.142' 2024-03-31T23:02:30.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:02:30.528+0000 7fb7f2b34700 1 -- 172.21.15.46:0/930101086 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fb7dc038f00 msgr2=0x7fb7dc03b3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:02:30.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:02:30.528+0000 7fb7f2b34700 1 --2- 172.21.15.46:0/930101086 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fb7dc038f00 0x7fb7dc03b3c0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fb7d8006fd0 tx=0x7fb7d8006e40 comp rx=0 tx=0).stop 2024-03-31T23:02:30.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:02:30.528+0000 7fb7f2b34700 1 -- 172.21.15.46:0/930101086 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb7ec106bf0 msgr2=0x7fb7ec0fcac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:02:30.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:02:30.528+0000 7fb7f2b34700 1 --2- 172.21.15.46:0/930101086 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb7ec106bf0 0x7fb7ec0fcac0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fb7e0006010 tx=0x7fb7e0005780 comp rx=0 tx=0).stop 2024-03-31T23:02:30.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:02:30.528+0000 7fb7f2b34700 1 -- 172.21.15.46:0/930101086 shutdown_connections 2024-03-31T23:02:30.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:02:30.528+0000 7fb7f2b34700 1 --2- 172.21.15.46:0/930101086 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fb7dc038f00 0x7fb7dc03b3c0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:02:30.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:02:30.528+0000 7fb7f2b34700 1 --2- 172.21.15.46:0/930101086 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb7ec106bf0 0x7fb7ec0fcac0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:02:30.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:02:30.528+0000 7fb7f2b34700 1 -- 172.21.15.46:0/930101086 >> 172.21.15.46:0/930101086 conn(0x7fb7ec0fc610 msgr2=0x7fb7ec06a080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:02:30.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:02:30.529+0000 7fb7f2b34700 1 -- 172.21.15.46:0/930101086 shutdown_connections 2024-03-31T23:02:30.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:02:30.529+0000 7fb7f2b34700 1 -- 172.21.15.46:0/930101086 wait complete. 2024-03-31T23:02:31.129 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:384f31f3d3aee7f8ba3f60846b8844c049165583 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6a9d8f08-efb2-11ee-b647-cb9ed24678a4 -- ceph orch host ls --format=json 2024-03-31T23:02:31.370 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/6a9d8f08-efb2-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-31T23:02:31.774 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:02:31 smithi046 ceph-mon[32498]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-31T23:02:31.775 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:02:31 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:02:31.775 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:02:31 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-31T23:02:31.775 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:02:31 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:02:32.754 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:02:32 smithi046 ceph-mon[32498]: Added host smithi142 2024-03-31T23:02:32.754 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:02:32 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:02:32.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:02:32.853+0000 7f4c79ad0700 1 -- 172.21.15.46:0/864869925 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4c74101e80 msgr2=0x7f4c74102260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:02:32.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:02:32.853+0000 7f4c79ad0700 1 --2- 172.21.15.46:0/864869925 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4c74101e80 0x7f4c74102260 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f4c64009910 tx=0x7f4c64009c20 comp rx=0 tx=0).stop 2024-03-31T23:02:32.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:02:32.853+0000 7f4c79ad0700 1 -- 172.21.15.46:0/864869925 shutdown_connections 2024-03-31T23:02:32.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:02:32.853+0000 7f4c79ad0700 1 --2- 172.21.15.46:0/864869925 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4c74101e80 0x7f4c74102260 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:02:32.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:02:32.853+0000 7f4c79ad0700 1 -- 172.21.15.46:0/864869925 >> 172.21.15.46:0/864869925 conn(0x7f4c740fd750 msgr2=0x7f4c740ffb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:02:32.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:02:32.853+0000 7f4c79ad0700 1 -- 172.21.15.46:0/864869925 shutdown_connections 2024-03-31T23:02:32.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:02:32.853+0000 7f4c79ad0700 1 -- 172.21.15.46:0/864869925 wait complete. 2024-03-31T23:02:32.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:02:32.854+0000 7f4c79ad0700 1 Processor -- start 2024-03-31T23:02:32.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:02:32.854+0000 7f4c79ad0700 1 -- start start 2024-03-31T23:02:32.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:02:32.854+0000 7f4c79ad0700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4c74101e80 0x7f4c74074540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:02:32.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:02:32.854+0000 7f4c79ad0700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f4c7410c100 con 0x7f4c74101e80 2024-03-31T23:02:32.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:02:32.854+0000 7f4c737fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4c74101e80 0x7f4c74074540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:02:32.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:02:32.854+0000 7f4c737fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4c74101e80 0x7f4c74074540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40312/0 (socket says 172.21.15.46:40312) 2024-03-31T23:02:32.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:02:32.854+0000 7f4c737fe700 1 -- 172.21.15.46:0/1772121937 learned_addr learned my addr 172.21.15.46:0/1772121937 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:02:32.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:02:32.855+0000 7f4c737fe700 1 -- 172.21.15.46:0/1772121937 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4c74076430 con 0x7f4c74101e80 2024-03-31T23:02:32.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:02:32.855+0000 7f4c737fe700 1 --2- 172.21.15.46:0/1772121937 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4c74101e80 0x7f4c74074540 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f4c64006010 tx=0x7f4c64005780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:02:32.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:02:32.855+0000 7f4c70ff9700 1 -- 172.21.15.46:0/1772121937 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4c64026070 con 0x7f4c74101e80 2024-03-31T23:02:32.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:02:32.855+0000 7f4c79ad0700 1 -- 172.21.15.46:0/1772121937 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4c74074ae0 con 0x7f4c74101e80 2024-03-31T23:02:32.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:02:32.855+0000 7f4c70ff9700 1 -- 172.21.15.46:0/1772121937 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4c6400bc00 con 0x7f4c74101e80 2024-03-31T23:02:32.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:02:32.855+0000 7f4c70ff9700 1 -- 172.21.15.46:0/1772121937 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4c64003930 con 0x7f4c74101e80 2024-03-31T23:02:32.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:02:32.855+0000 7f4c79ad0700 1 -- 172.21.15.46:0/1772121937 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4c74074fa0 con 0x7f4c74101e80 2024-03-31T23:02:32.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:02:32.856+0000 7f4c70ff9700 1 -- 172.21.15.46:0/1772121937 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 13) ==== 46172+0+0 (secure 0 0 0) 0x7f4c64029070 con 0x7f4c74101e80 2024-03-31T23:02:32.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:02:32.856+0000 7f4c79ad0700 1 -- 172.21.15.46:0/1772121937 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4c74103360 con 0x7f4c74101e80 2024-03-31T23:02:32.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:02:32.856+0000 7f4c70ff9700 1 --2- 172.21.15.46:0/1772121937 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f4c5c038ea0 0x7f4c5c03b360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:02:32.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:02:32.856+0000 7f4c70ff9700 1 -- 172.21.15.46:0/1772121937 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f4c6401e070 con 0x7f4c74101e80 2024-03-31T23:02:32.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:02:32.856+0000 7f4c72ffd700 1 --2- 172.21.15.46:0/1772121937 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f4c5c038ea0 0x7f4c5c03b360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:02:32.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:02:32.857+0000 7f4c72ffd700 1 --2- 172.21.15.46:0/1772121937 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f4c5c038ea0 0x7f4c5c03b360 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f4c68006fd0 tx=0x7f4c68006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:02:32.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:02:32.861+0000 7f4c70ff9700 1 -- 172.21.15.46:0/1772121937 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f4c64053080 con 0x7f4c74101e80 2024-03-31T23:02:33.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:02:33.068+0000 7f4c79ad0700 1 -- 172.21.15.46:0/1772121937 --> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f4c74075230 con 0x7f4c5c038ea0 2024-03-31T23:02:33.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:02:33.070+0000 7f4c70ff9700 1 -- 172.21.15.46:0/1772121937 <== mgr.14152 v2:172.21.15.46:6800/162289569 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+160 (secure 0 0 0) 0x7f4c74075230 con 0x7f4c5c038ea0 2024-03-31T23:02:33.071 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-31T23:02:33.071 INFO:teuthology.orchestra.run.smithi046.stdout:[{"addr": "172.21.15.46", "hostname": "smithi046", "labels": [], "status": ""}, {"addr": "172.21.15.142", "hostname": "smithi142", "labels": [], "status": ""}] 2024-03-31T23:02:33.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:02:33.072+0000 7f4c79ad0700 1 -- 172.21.15.46:0/1772121937 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f4c5c038ea0 msgr2=0x7f4c5c03b360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:02:33.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:02:33.072+0000 7f4c79ad0700 1 --2- 172.21.15.46:0/1772121937 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f4c5c038ea0 0x7f4c5c03b360 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f4c68006fd0 tx=0x7f4c68006e40 comp rx=0 tx=0).stop 2024-03-31T23:02:33.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:02:33.072+0000 7f4c79ad0700 1 -- 172.21.15.46:0/1772121937 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4c74101e80 msgr2=0x7f4c74074540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:02:33.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:02:33.072+0000 7f4c79ad0700 1 --2- 172.21.15.46:0/1772121937 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4c74101e80 0x7f4c74074540 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f4c64006010 tx=0x7f4c64005780 comp rx=0 tx=0).stop 2024-03-31T23:02:33.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:02:33.072+0000 7f4c79ad0700 1 -- 172.21.15.46:0/1772121937 shutdown_connections 2024-03-31T23:02:33.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:02:33.072+0000 7f4c79ad0700 1 --2- 172.21.15.46:0/1772121937 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f4c5c038ea0 0x7f4c5c03b360 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:02:33.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:02:33.072+0000 7f4c79ad0700 1 --2- 172.21.15.46:0/1772121937 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4c74101e80 0x7f4c74074540 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:02:33.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:02:33.072+0000 7f4c79ad0700 1 -- 172.21.15.46:0/1772121937 >> 172.21.15.46:0/1772121937 conn(0x7f4c740fd750 msgr2=0x7f4c74103850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:02:33.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:02:33.072+0000 7f4c79ad0700 1 -- 172.21.15.46:0/1772121937 shutdown_connections 2024-03-31T23:02:33.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:02:33.072+0000 7f4c79ad0700 1 -- 172.21.15.46:0/1772121937 wait complete. 2024-03-31T23:02:33.569 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi162 2024-03-31T23:02:33.569 DEBUG:teuthology.orchestra.run.smithi162:> set -ex 2024-03-31T23:02:33.569 DEBUG:teuthology.orchestra.run.smithi162:> dd of=/etc/ceph/ceph.conf 2024-03-31T23:02:33.587 DEBUG:teuthology.orchestra.run.smithi162:> set -ex 2024-03-31T23:02:33.588 DEBUG:teuthology.orchestra.run.smithi162:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-31T23:02:33.645 INFO:tasks.cephadm:Adding host smithi162 to orchestrator... 2024-03-31T23:02:33.645 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:384f31f3d3aee7f8ba3f60846b8844c049165583 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6a9d8f08-efb2-11ee-b647-cb9ed24678a4 -- ceph orch host add smithi162 2024-03-31T23:02:33.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:02:33 smithi046 ceph-mon[32498]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-31T23:02:33.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:02:33 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:02:33.883 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/6a9d8f08-efb2-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-31T23:02:34.848 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:02:34 smithi046 ceph-mon[32498]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-31T23:02:35.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:02:35.297+0000 7fdc13914700 1 -- 172.21.15.46:0/3243981855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdc0c101e80 msgr2=0x7fdc0c102260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:02:35.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:02:35.297+0000 7fdc13914700 1 --2- 172.21.15.46:0/3243981855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdc0c101e80 0x7fdc0c102260 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fdc08009910 tx=0x7fdc08009c20 comp rx=0 tx=0).stop 2024-03-31T23:02:35.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:02:35.297+0000 7fdc13914700 1 -- 172.21.15.46:0/3243981855 shutdown_connections 2024-03-31T23:02:35.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:02:35.297+0000 7fdc13914700 1 --2- 172.21.15.46:0/3243981855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdc0c101e80 0x7fdc0c102260 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:02:35.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:02:35.297+0000 7fdc13914700 1 -- 172.21.15.46:0/3243981855 >> 172.21.15.46:0/3243981855 conn(0x7fdc0c0fd750 msgr2=0x7fdc0c0ffb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:02:35.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:02:35.298+0000 7fdc13914700 1 -- 172.21.15.46:0/3243981855 shutdown_connections 2024-03-31T23:02:35.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:02:35.298+0000 7fdc13914700 1 -- 172.21.15.46:0/3243981855 wait complete. 2024-03-31T23:02:35.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:02:35.298+0000 7fdc13914700 1 Processor -- start 2024-03-31T23:02:35.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:02:35.298+0000 7fdc13914700 1 -- start start 2024-03-31T23:02:35.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:02:35.298+0000 7fdc13914700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdc0c101e80 0x7fdc0c197690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:02:35.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:02:35.298+0000 7fdc13914700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fdc0c10c100 con 0x7fdc0c101e80 2024-03-31T23:02:35.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:02:35.298+0000 7fdc116b0700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdc0c101e80 0x7fdc0c197690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:02:35.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:02:35.299+0000 7fdc116b0700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdc0c101e80 0x7fdc0c197690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40326/0 (socket says 172.21.15.46:40326) 2024-03-31T23:02:35.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:02:35.299+0000 7fdc116b0700 1 -- 172.21.15.46:0/1947661768 learned_addr learned my addr 172.21.15.46:0/1947661768 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:02:35.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:02:35.299+0000 7fdc116b0700 1 -- 172.21.15.46:0/1947661768 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc0c197bd0 con 0x7fdc0c101e80 2024-03-31T23:02:35.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:02:35.299+0000 7fdc116b0700 1 --2- 172.21.15.46:0/1947661768 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdc0c101e80 0x7fdc0c197690 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fdc08006010 tx=0x7fdc08005780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:02:35.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:02:35.299+0000 7fdc027fc700 1 -- 172.21.15.46:0/1947661768 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fdc08026070 con 0x7fdc0c101e80 2024-03-31T23:02:35.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:02:35.299+0000 7fdc027fc700 1 -- 172.21.15.46:0/1947661768 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fdc0800bba0 con 0x7fdc0c101e80 2024-03-31T23:02:35.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:02:35.299+0000 7fdc027fc700 1 -- 172.21.15.46:0/1947661768 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fdc080038a0 con 0x7fdc0c101e80 2024-03-31T23:02:35.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:02:35.299+0000 7fdc13914700 1 -- 172.21.15.46:0/1947661768 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdc0c197ec0 con 0x7fdc0c101e80 2024-03-31T23:02:35.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:02:35.299+0000 7fdc13914700 1 -- 172.21.15.46:0/1947661768 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdc0c19bf60 con 0x7fdc0c101e80 2024-03-31T23:02:35.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:02:35.300+0000 7fdc13914700 1 -- 172.21.15.46:0/1947661768 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdc0c103360 con 0x7fdc0c101e80 2024-03-31T23:02:35.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:02:35.300+0000 7fdc027fc700 1 -- 172.21.15.46:0/1947661768 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 13) ==== 46172+0+0 (secure 0 0 0) 0x7fdc08029070 con 0x7fdc0c101e80 2024-03-31T23:02:35.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:02:35.301+0000 7fdc027fc700 1 --2- 172.21.15.46:0/1947661768 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fdbf8038f00 0x7fdbf803b3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:02:35.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:02:35.301+0000 7fdc027fc700 1 -- 172.21.15.46:0/1947661768 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fdc0801e070 con 0x7fdc0c101e80 2024-03-31T23:02:35.306 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:02:35.301+0000 7fdc10eaf700 1 --2- 172.21.15.46:0/1947661768 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fdbf8038f00 0x7fdbf803b3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:02:35.306 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:02:35.301+0000 7fdc10eaf700 1 --2- 172.21.15.46:0/1947661768 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fdbf8038f00 0x7fdbf803b3c0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fdbfc006fd0 tx=0x7fdbfc006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:02:35.306 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:02:35.306+0000 7fdc027fc700 1 -- 172.21.15.46:0/1947661768 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fdc08013070 con 0x7fdc0c101e80 2024-03-31T23:02:35.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:02:35.513+0000 7fdc13914700 1 -- 172.21.15.46:0/1947661768 --> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi162", "target": ["mon-mgr", ""]}) -- 0x7fdc0c1989f0 con 0x7fdbf8038f00 2024-03-31T23:02:35.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:02:35 smithi046 ceph-mon[32498]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-31T23:02:37.361 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:02:36 smithi046 ceph-mon[32498]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi162", "target": ["mon-mgr", ""]}]: dispatch 2024-03-31T23:02:37.362 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:02:36 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:02:37.362 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:02:36 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:02:37.362 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:02:36 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:02:37.362 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:02:36 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:02:37.362 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:02:36 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi142", "name": "osd_memory_target"}]: dispatch 2024-03-31T23:02:37.362 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:02:36 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:02:37.362 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:02:36 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-31T23:02:37.362 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:02:36 smithi046 ceph-mon[32498]: Updating smithi142:/etc/ceph/ceph.conf 2024-03-31T23:02:37.362 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:02:36 smithi046 ceph-mon[32498]: Updating smithi142:/var/lib/ceph/6a9d8f08-efb2-11ee-b647-cb9ed24678a4/config/ceph.conf 2024-03-31T23:02:37.362 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:02:36 smithi046 ceph-mon[32498]: Deploying cephadm binary to smithi162 2024-03-31T23:02:37.362 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:02:36 smithi046 ceph-mon[32498]: Updating smithi142:/etc/ceph/ceph.client.admin.keyring 2024-03-31T23:02:37.362 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:02:36 smithi046 ceph-mon[32498]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-31T23:02:38.361 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:02:37 smithi046 ceph-mon[32498]: Updating smithi142:/var/lib/ceph/6a9d8f08-efb2-11ee-b647-cb9ed24678a4/config/ceph.client.admin.keyring 2024-03-31T23:02:38.362 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:02:37 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:02:38.362 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:02:37 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:02:38.362 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:02:37 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:02:39.361 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:02:38 smithi046 ceph-mon[32498]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-31T23:02:40.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:02:40.059+0000 7fdc027fc700 1 -- 172.21.15.46:0/1947661768 <== mgr.14152 v2:172.21.15.46:6800/162289569 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+49 (secure 0 0 0) 0x7fdc0c1989f0 con 0x7fdbf8038f00 2024-03-31T23:02:40.062 INFO:teuthology.orchestra.run.smithi046.stdout:Added host 'smithi162' with addr '172.21.15.162' 2024-03-31T23:02:40.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:02:40.062+0000 7fdc13914700 1 -- 172.21.15.46:0/1947661768 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fdbf8038f00 msgr2=0x7fdbf803b3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:02:40.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:02:40.062+0000 7fdc13914700 1 --2- 172.21.15.46:0/1947661768 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fdbf8038f00 0x7fdbf803b3c0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fdbfc006fd0 tx=0x7fdbfc006e40 comp rx=0 tx=0).stop 2024-03-31T23:02:40.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:02:40.062+0000 7fdc13914700 1 -- 172.21.15.46:0/1947661768 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdc0c101e80 msgr2=0x7fdc0c197690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:02:40.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:02:40.062+0000 7fdc13914700 1 --2- 172.21.15.46:0/1947661768 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdc0c101e80 0x7fdc0c197690 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fdc08006010 tx=0x7fdc08005780 comp rx=0 tx=0).stop 2024-03-31T23:02:40.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:02:40.062+0000 7fdc13914700 1 -- 172.21.15.46:0/1947661768 shutdown_connections 2024-03-31T23:02:40.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:02:40.062+0000 7fdc13914700 1 --2- 172.21.15.46:0/1947661768 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fdbf8038f00 0x7fdbf803b3c0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:02:40.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:02:40.062+0000 7fdc13914700 1 --2- 172.21.15.46:0/1947661768 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdc0c101e80 0x7fdc0c197690 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:02:40.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:02:40.062+0000 7fdc13914700 1 -- 172.21.15.46:0/1947661768 >> 172.21.15.46:0/1947661768 conn(0x7fdc0c0fd750 msgr2=0x7fdc0c103850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:02:40.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:02:40.062+0000 7fdc13914700 1 -- 172.21.15.46:0/1947661768 shutdown_connections 2024-03-31T23:02:40.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:02:40.062+0000 7fdc13914700 1 -- 172.21.15.46:0/1947661768 wait complete. 2024-03-31T23:02:40.678 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:384f31f3d3aee7f8ba3f60846b8844c049165583 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6a9d8f08-efb2-11ee-b647-cb9ed24678a4 -- ceph orch host ls --format=json 2024-03-31T23:02:40.920 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/6a9d8f08-efb2-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-31T23:02:41.349 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:02:41 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:02:41.349 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:02:41 smithi046 ceph-mon[32498]: Added host smithi162 2024-03-31T23:02:41.349 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:02:41 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-31T23:02:41.349 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:02:41 smithi046 ceph-mon[32498]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-31T23:02:41.349 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:02:41 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:02:42.288 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:02:42 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:02:42.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:02:42.319+0000 7f981fb69700 1 -- 172.21.15.46:0/1250298607 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9818100bd0 msgr2=0x7f9818100fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:02:42.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:02:42.319+0000 7f981fb69700 1 --2- 172.21.15.46:0/1250298607 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9818100bd0 0x7f9818100fb0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f981000b0a0 tx=0x7f981000b3b0 comp rx=0 tx=0).stop 2024-03-31T23:02:42.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:02:42.319+0000 7f981fb69700 1 -- 172.21.15.46:0/1250298607 shutdown_connections 2024-03-31T23:02:42.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:02:42.319+0000 7f981fb69700 1 --2- 172.21.15.46:0/1250298607 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9818100bd0 0x7f9818100fb0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:02:42.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:02:42.319+0000 7f981fb69700 1 -- 172.21.15.46:0/1250298607 >> 172.21.15.46:0/1250298607 conn(0x7f98180fc630 msgr2=0x7f98180fea50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:02:42.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:02:42.319+0000 7f981fb69700 1 -- 172.21.15.46:0/1250298607 shutdown_connections 2024-03-31T23:02:42.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:02:42.319+0000 7f981fb69700 1 -- 172.21.15.46:0/1250298607 wait complete. 2024-03-31T23:02:42.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:02:42.319+0000 7f981fb69700 1 Processor -- start 2024-03-31T23:02:42.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:02:42.319+0000 7f981fb69700 1 -- start start 2024-03-31T23:02:42.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:02:42.320+0000 7f981fb69700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9818100bd0 0x7f98181a38e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:02:42.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:02:42.320+0000 7f981fb69700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f9818112780 con 0x7f9818100bd0 2024-03-31T23:02:42.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:02:42.320+0000 7f981d905700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9818100bd0 0x7f98181a38e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:02:42.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:02:42.320+0000 7f981d905700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9818100bd0 0x7f98181a38e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:46898/0 (socket says 172.21.15.46:46898) 2024-03-31T23:02:42.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:02:42.320+0000 7f981d905700 1 -- 172.21.15.46:0/437574958 learned_addr learned my addr 172.21.15.46:0/437574958 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:02:42.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:02:42.320+0000 7f981d905700 1 -- 172.21.15.46:0/437574958 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f98181a3e20 con 0x7f9818100bd0 2024-03-31T23:02:42.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:02:42.320+0000 7f981d905700 1 --2- 172.21.15.46:0/437574958 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9818100bd0 0x7f98181a38e0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f98100076a0 tx=0x7f9810009800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:02:42.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:02:42.320+0000 7f980effd700 1 -- 172.21.15.46:0/437574958 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9810017040 con 0x7f9818100bd0 2024-03-31T23:02:42.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:02:42.320+0000 7f981fb69700 1 -- 172.21.15.46:0/437574958 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f98181a40b0 con 0x7f9818100bd0 2024-03-31T23:02:42.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:02:42.320+0000 7f980effd700 1 -- 172.21.15.46:0/437574958 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9810004040 con 0x7f9818100bd0 2024-03-31T23:02:42.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:02:42.320+0000 7f980effd700 1 -- 172.21.15.46:0/437574958 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f981001d450 con 0x7f9818100bd0 2024-03-31T23:02:42.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:02:42.320+0000 7f981fb69700 1 -- 172.21.15.46:0/437574958 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f981819daa0 con 0x7f9818100bd0 2024-03-31T23:02:42.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:02:42.321+0000 7f980effd700 1 -- 172.21.15.46:0/437574958 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 13) ==== 46172+0+0 (secure 0 0 0) 0x7f9810024020 con 0x7f9818100bd0 2024-03-31T23:02:42.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:02:42.321+0000 7f981fb69700 1 -- 172.21.15.46:0/437574958 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f981806a980 con 0x7f9818100bd0 2024-03-31T23:02:42.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:02:42.321+0000 7f980effd700 1 --2- 172.21.15.46:0/437574958 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f9804038ea0 0x7f980403b360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:02:42.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:02:42.321+0000 7f980effd700 1 -- 172.21.15.46:0/437574958 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f9810022020 con 0x7f9818100bd0 2024-03-31T23:02:42.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:02:42.322+0000 7f981d104700 1 --2- 172.21.15.46:0/437574958 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f9804038ea0 0x7f980403b360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:02:42.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:02:42.322+0000 7f981d104700 1 --2- 172.21.15.46:0/437574958 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f9804038ea0 0x7f980403b360 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f9808006fd0 tx=0x7f9808006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:02:42.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:02:42.326+0000 7f980effd700 1 -- 172.21.15.46:0/437574958 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f981002b020 con 0x7f9818100bd0 2024-03-31T23:02:42.538 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:02:42.537+0000 7f981fb69700 1 -- 172.21.15.46:0/437574958 --> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f981819efd0 con 0x7f9804038ea0 2024-03-31T23:02:42.539 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:02:42.538+0000 7f980effd700 1 -- 172.21.15.46:0/437574958 <== mgr.14152 v2:172.21.15.46:6800/162289569 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+240 (secure 0 0 0) 0x7f981819efd0 con 0x7f9804038ea0 2024-03-31T23:02:42.539 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-31T23:02:42.540 INFO:teuthology.orchestra.run.smithi046.stdout:[{"addr": "172.21.15.46", "hostname": "smithi046", "labels": [], "status": ""}, {"addr": "172.21.15.142", "hostname": "smithi142", "labels": [], "status": ""}, {"addr": "172.21.15.162", "hostname": "smithi162", "labels": [], "status": ""}] 2024-03-31T23:02:42.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:02:42.540+0000 7f981fb69700 1 -- 172.21.15.46:0/437574958 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f9804038ea0 msgr2=0x7f980403b360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:02:42.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:02:42.540+0000 7f981fb69700 1 --2- 172.21.15.46:0/437574958 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f9804038ea0 0x7f980403b360 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f9808006fd0 tx=0x7f9808006e40 comp rx=0 tx=0).stop 2024-03-31T23:02:42.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:02:42.540+0000 7f981fb69700 1 -- 172.21.15.46:0/437574958 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9818100bd0 msgr2=0x7f98181a38e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:02:42.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:02:42.540+0000 7f981fb69700 1 --2- 172.21.15.46:0/437574958 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9818100bd0 0x7f98181a38e0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f98100076a0 tx=0x7f9810009800 comp rx=0 tx=0).stop 2024-03-31T23:02:42.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:02:42.540+0000 7f981fb69700 1 -- 172.21.15.46:0/437574958 shutdown_connections 2024-03-31T23:02:42.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:02:42.540+0000 7f981fb69700 1 --2- 172.21.15.46:0/437574958 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f9804038ea0 0x7f980403b360 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:02:42.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:02:42.540+0000 7f981fb69700 1 --2- 172.21.15.46:0/437574958 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9818100bd0 0x7f98181a38e0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:02:42.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:02:42.540+0000 7f981fb69700 1 -- 172.21.15.46:0/437574958 >> 172.21.15.46:0/437574958 conn(0x7f98180fc630 msgr2=0x7f98180fdf20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:02:42.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:02:42.541+0000 7f981fb69700 1 -- 172.21.15.46:0/437574958 shutdown_connections 2024-03-31T23:02:42.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:02:42.541+0000 7f981fb69700 1 -- 172.21.15.46:0/437574958 wait complete. 2024-03-31T23:02:43.121 INFO:tasks.cephadm:Setting crush tunables to default 2024-03-31T23:02:43.121 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:384f31f3d3aee7f8ba3f60846b8844c049165583 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6a9d8f08-efb2-11ee-b647-cb9ed24678a4 -- ceph osd crush tunables default 2024-03-31T23:02:43.360 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/6a9d8f08-efb2-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-31T23:02:43.361 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:02:43 smithi046 ceph-mon[32498]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-31T23:02:43.361 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:02:43 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:02:44.374 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:02:44 smithi046 ceph-mon[32498]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-31T23:02:44.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:02:44.833+0000 7f53c8b9a700 1 -- 172.21.15.46:0/1633825443 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f53c4106be0 msgr2=0x7f53c4106fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:02:44.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:02:44.833+0000 7f53c8b9a700 1 --2- 172.21.15.46:0/1633825443 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f53c4106be0 0x7f53c4106fc0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f53b8009910 tx=0x7f53b8009c20 comp rx=0 tx=0).stop 2024-03-31T23:02:44.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:02:44.833+0000 7f53c8b9a700 1 -- 172.21.15.46:0/1633825443 shutdown_connections 2024-03-31T23:02:44.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:02:44.833+0000 7f53c8b9a700 1 --2- 172.21.15.46:0/1633825443 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f53c4106be0 0x7f53c4106fc0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:02:44.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:02:44.833+0000 7f53c8b9a700 1 -- 172.21.15.46:0/1633825443 >> 172.21.15.46:0/1633825443 conn(0x7f53c4077780 msgr2=0x7f53c40fea50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:02:44.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:02:44.833+0000 7f53c8b9a700 1 -- 172.21.15.46:0/1633825443 shutdown_connections 2024-03-31T23:02:44.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:02:44.833+0000 7f53c8b9a700 1 -- 172.21.15.46:0/1633825443 wait complete. 2024-03-31T23:02:44.836 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:02:44.834+0000 7f53c8b9a700 1 Processor -- start 2024-03-31T23:02:44.836 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:02:44.834+0000 7f53c8b9a700 1 -- start start 2024-03-31T23:02:44.836 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:02:44.834+0000 7f53c8b9a700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f53c4106be0 0x7f53c4197640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:02:44.836 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:02:44.834+0000 7f53c8b9a700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f53c410b050 con 0x7f53c4106be0 2024-03-31T23:02:44.836 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:02:44.834+0000 7f53c259c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f53c4106be0 0x7f53c4197640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:02:44.836 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:02:44.834+0000 7f53c259c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f53c4106be0 0x7f53c4197640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:46912/0 (socket says 172.21.15.46:46912) 2024-03-31T23:02:44.836 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:02:44.834+0000 7f53c259c700 1 -- 172.21.15.46:0/3677361043 learned_addr learned my addr 172.21.15.46:0/3677361043 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:02:44.836 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:02:44.834+0000 7f53c259c700 1 -- 172.21.15.46:0/3677361043 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53c4197b80 con 0x7f53c4106be0 2024-03-31T23:02:44.836 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:02:44.834+0000 7f53c259c700 1 --2- 172.21.15.46:0/3677361043 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f53c4106be0 0x7f53c4197640 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f53b8006010 tx=0x7f53b8005780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:02:44.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:02:44.835+0000 7f53b37fe700 1 -- 172.21.15.46:0/3677361043 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f53b8026070 con 0x7f53c4106be0 2024-03-31T23:02:44.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:02:44.835+0000 7f53c8b9a700 1 -- 172.21.15.46:0/3677361043 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f53c4197e10 con 0x7f53c4106be0 2024-03-31T23:02:44.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:02:44.835+0000 7f53b37fe700 1 -- 172.21.15.46:0/3677361043 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f53b800bc00 con 0x7f53c4106be0 2024-03-31T23:02:44.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:02:44.835+0000 7f53b37fe700 1 -- 172.21.15.46:0/3677361043 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f53b8003930 con 0x7f53c4106be0 2024-03-31T23:02:44.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:02:44.835+0000 7f53c8b9a700 1 -- 172.21.15.46:0/3677361043 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f53c419bf10 con 0x7f53c4106be0 2024-03-31T23:02:44.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:02:44.836+0000 7f53b37fe700 1 -- 172.21.15.46:0/3677361043 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 13) ==== 46172+0+0 (secure 0 0 0) 0x7f53b8029070 con 0x7f53c4106be0 2024-03-31T23:02:44.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:02:44.836+0000 7f53c8b9a700 1 -- 172.21.15.46:0/3677361043 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f53c41021e0 con 0x7f53c4106be0 2024-03-31T23:02:44.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:02:44.836+0000 7f53b37fe700 1 --2- 172.21.15.46:0/3677361043 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f53ac038ea0 0x7f53ac03b360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:02:44.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:02:44.836+0000 7f53b37fe700 1 -- 172.21.15.46:0/3677361043 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f53b801e070 con 0x7f53c4106be0 2024-03-31T23:02:44.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:02:44.836+0000 7f53c1d9b700 1 --2- 172.21.15.46:0/3677361043 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f53ac038ea0 0x7f53ac03b360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:02:44.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:02:44.836+0000 7f53c1d9b700 1 --2- 172.21.15.46:0/3677361043 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f53ac038ea0 0x7f53ac03b360 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f53b4006fd0 tx=0x7f53b4006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:02:44.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:02:44.841+0000 7f53b37fe700 1 -- 172.21.15.46:0/3677361043 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f53b8052020 con 0x7f53c4106be0 2024-03-31T23:02:45.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:02:45.050+0000 7f53c8b9a700 1 -- 172.21.15.46:0/3677361043 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) -- 0x7f53c4063870 con 0x7f53c4106be0 2024-03-31T23:02:45.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:02:45.125+0000 7f53b37fe700 1 -- 172.21.15.46:0/3677361043 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v4) ==== 124+0+0 (secure 0 0 0) 0x7f53b8030090 con 0x7f53c4106be0 2024-03-31T23:02:45.130 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:02:45.128+0000 7f53c8b9a700 1 -- 172.21.15.46:0/3677361043 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f53ac038ea0 msgr2=0x7f53ac03b360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:02:45.130 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:02:45.128+0000 7f53c8b9a700 1 --2- 172.21.15.46:0/3677361043 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f53ac038ea0 0x7f53ac03b360 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f53b4006fd0 tx=0x7f53b4006e40 comp rx=0 tx=0).stop 2024-03-31T23:02:45.130 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:02:45.128+0000 7f53c8b9a700 1 -- 172.21.15.46:0/3677361043 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f53c4106be0 msgr2=0x7f53c4197640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:02:45.130 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:02:45.128+0000 7f53c8b9a700 1 --2- 172.21.15.46:0/3677361043 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f53c4106be0 0x7f53c4197640 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f53b8006010 tx=0x7f53b8005780 comp rx=0 tx=0).stop 2024-03-31T23:02:45.130 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:02:45.128+0000 7f53c8b9a700 1 -- 172.21.15.46:0/3677361043 shutdown_connections 2024-03-31T23:02:45.130 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:02:45.128+0000 7f53c8b9a700 1 --2- 172.21.15.46:0/3677361043 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f53ac038ea0 0x7f53ac03b360 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:02:45.130 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:02:45.128+0000 7f53c8b9a700 1 --2- 172.21.15.46:0/3677361043 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f53c4106be0 0x7f53c4197640 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:02:45.130 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:02:45.128+0000 7f53c8b9a700 1 -- 172.21.15.46:0/3677361043 >> 172.21.15.46:0/3677361043 conn(0x7f53c4077780 msgr2=0x7f53c4069630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:02:45.130 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:02:45.129+0000 7f53c8b9a700 1 -- 172.21.15.46:0/3677361043 shutdown_connections 2024-03-31T23:02:45.130 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:02:45.129+0000 7f53c8b9a700 1 -- 172.21.15.46:0/3677361043 wait complete. 2024-03-31T23:02:45.131 INFO:teuthology.orchestra.run.smithi046.stderr:adjusted tunables profile to default 2024-03-31T23:02:45.361 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:02:45 smithi046 ceph-mon[32498]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-31T23:02:45.361 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:02:45 smithi046 ceph-mon[32498]: from='client.? 172.21.15.46:0/3677361043' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-03-31T23:02:45.734 INFO:tasks.cephadm:Adding mon.a on smithi046 2024-03-31T23:02:45.734 INFO:tasks.cephadm:Adding mon.b on smithi142 2024-03-31T23:02:45.734 INFO:tasks.cephadm:Adding mon.c on smithi162 2024-03-31T23:02:45.734 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:384f31f3d3aee7f8ba3f60846b8844c049165583 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6a9d8f08-efb2-11ee-b647-cb9ed24678a4 -- ceph orch apply mon '3;smithi046:172.21.15.46=a;smithi142:172.21.15.142=b;smithi162:172.21.15.162=c' 2024-03-31T23:02:45.967 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-31T23:02:46.015 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-31T23:02:46.611 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:02:46 smithi046 ceph-mon[32498]: from='client.? 172.21.15.46:0/3677361043' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-03-31T23:02:46.612 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:02:46 smithi046 ceph-mon[32498]: osdmap e4: 0 total, 0 up, 0 in 2024-03-31T23:02:46.612 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:02:46 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:02:46.612 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:02:46 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:02:46.612 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:02:46 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:02:46.612 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:02:46 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:02:46.612 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:02:46 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi162", "name": "osd_memory_target"}]: dispatch 2024-03-31T23:02:46.612 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:02:46 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:02:46.612 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:02:46 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-31T23:02:47.502 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:02:47.501+0000 7f08679fd700 1 -- 172.21.15.162:0/3997399299 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0860106be0 msgr2=0x7f0860106fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:02:47.503 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:02:47.501+0000 7f08679fd700 1 --2- 172.21.15.162:0/3997399299 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0860106be0 0x7f0860106fc0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f085c009910 tx=0x7f085c009c20 comp rx=0 tx=0).stop 2024-03-31T23:02:47.503 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:02:47.501+0000 7f08679fd700 1 -- 172.21.15.162:0/3997399299 shutdown_connections 2024-03-31T23:02:47.503 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:02:47.501+0000 7f08679fd700 1 --2- 172.21.15.162:0/3997399299 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0860106be0 0x7f0860106fc0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:02:47.503 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:02:47.501+0000 7f08679fd700 1 -- 172.21.15.162:0/3997399299 >> 172.21.15.162:0/3997399299 conn(0x7f0860077780 msgr2=0x7f08600fea50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:02:47.503 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:02:47.501+0000 7f08679fd700 1 -- 172.21.15.162:0/3997399299 shutdown_connections 2024-03-31T23:02:47.503 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:02:47.501+0000 7f08679fd700 1 -- 172.21.15.162:0/3997399299 wait complete. 2024-03-31T23:02:47.504 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:02:47.501+0000 7f08679fd700 1 Processor -- start 2024-03-31T23:02:47.504 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:02:47.502+0000 7f08679fd700 1 -- start start 2024-03-31T23:02:47.505 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:02:47.502+0000 7f08679fd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0860106be0 0x7f0860198460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:02:47.505 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:02:47.502+0000 7f08679fd700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f086010b050 con 0x7f0860106be0 2024-03-31T23:02:47.505 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:02:47.502+0000 7f0865799700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0860106be0 0x7f0860198460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:02:47.505 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:02:47.502+0000 7f0865799700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0860106be0 0x7f0860198460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.162:43130/0 (socket says 172.21.15.162:43130) 2024-03-31T23:02:47.505 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:02:47.502+0000 7f0865799700 1 -- 172.21.15.162:0/2519878382 learned_addr learned my addr 172.21.15.162:0/2519878382 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-03-31T23:02:47.505 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:02:47.502+0000 7f0865799700 1 -- 172.21.15.162:0/2519878382 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f08601989a0 con 0x7f0860106be0 2024-03-31T23:02:47.505 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:02:47.503+0000 7f0865799700 1 --2- 172.21.15.162:0/2519878382 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0860106be0 0x7f0860198460 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f085c0056d0 tx=0x7f085c0057b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:02:47.505 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:02:47.503+0000 7f08567fc700 1 -- 172.21.15.162:0/2519878382 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f085c026070 con 0x7f0860106be0 2024-03-31T23:02:47.505 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:02:47.503+0000 7f08679fd700 1 -- 172.21.15.162:0/2519878382 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0860198c30 con 0x7f0860106be0 2024-03-31T23:02:47.505 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:02:47.503+0000 7f08567fc700 1 -- 172.21.15.162:0/2519878382 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f085c00bc00 con 0x7f0860106be0 2024-03-31T23:02:47.505 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:02:47.503+0000 7f08567fc700 1 -- 172.21.15.162:0/2519878382 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f085c0039b0 con 0x7f0860106be0 2024-03-31T23:02:47.505 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:02:47.503+0000 7f08679fd700 1 -- 172.21.15.162:0/2519878382 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0860199090 con 0x7f0860106be0 2024-03-31T23:02:47.506 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:02:47.504+0000 7f08679fd700 1 -- 172.21.15.162:0/2519878382 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f08601021e0 con 0x7f0860106be0 2024-03-31T23:02:47.506 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:02:47.504+0000 7f08567fc700 1 -- 172.21.15.162:0/2519878382 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 13) ==== 46172+0+0 (secure 0 0 0) 0x7f085c029070 con 0x7f0860106be0 2024-03-31T23:02:47.506 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:02:47.504+0000 7f08567fc700 1 --2- 172.21.15.162:0/2519878382 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f084c038f00 0x7f084c03b3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:02:47.506 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:02:47.504+0000 7f08567fc700 1 -- 172.21.15.162:0/2519878382 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f085c01e070 con 0x7f0860106be0 2024-03-31T23:02:47.507 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:02:47.504+0000 7f0864f98700 1 --2- 172.21.15.162:0/2519878382 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f084c038f00 0x7f084c03b3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:02:47.507 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:02:47.505+0000 7f0864f98700 1 --2- 172.21.15.162:0/2519878382 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f084c038f00 0x7f084c03b3c0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f0850006fd0 tx=0x7f0850006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:02:47.511 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:02:47.509+0000 7f08567fc700 1 -- 172.21.15.162:0/2519878382 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f085c013070 con 0x7f0860106be0 2024-03-31T23:02:47.612 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:02:47 smithi046 ceph-mon[32498]: Updating smithi162:/etc/ceph/ceph.conf 2024-03-31T23:02:47.612 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:02:47 smithi046 ceph-mon[32498]: Updating smithi162:/var/lib/ceph/6a9d8f08-efb2-11ee-b647-cb9ed24678a4/config/ceph.conf 2024-03-31T23:02:47.612 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:02:47 smithi046 ceph-mon[32498]: Updating smithi162:/etc/ceph/ceph.client.admin.keyring 2024-03-31T23:02:47.612 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:02:47 smithi046 ceph-mon[32498]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-31T23:02:47.612 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:02:47 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:02:47.612 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:02:47 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:02:47.613 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:02:47 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:02:47.715 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:02:47.713+0000 7f08679fd700 1 -- 172.21.15.162:0/2519878382 --> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi046:172.21.15.46=a;smithi142:172.21.15.142=b;smithi162:172.21.15.162=c", "target": ["mon-mgr", ""]}) -- 0x7f0860069c40 con 0x7f084c038f00 2024-03-31T23:02:47.721 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:02:47.721+0000 7f08567fc700 1 -- 172.21.15.162:0/2519878382 <== mgr.14152 v2:172.21.15.46:6800/162289569 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f0860069c40 con 0x7f084c038f00 2024-03-31T23:02:47.721 INFO:teuthology.orchestra.run.smithi162.stdout:Scheduled mon update... 2024-03-31T23:02:47.723 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:02:47.722+0000 7f08679fd700 1 -- 172.21.15.162:0/2519878382 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f084c038f00 msgr2=0x7f084c03b3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:02:47.723 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:02:47.722+0000 7f08679fd700 1 --2- 172.21.15.162:0/2519878382 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f084c038f00 0x7f084c03b3c0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f0850006fd0 tx=0x7f0850006e40 comp rx=0 tx=0).stop 2024-03-31T23:02:47.723 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:02:47.722+0000 7f08679fd700 1 -- 172.21.15.162:0/2519878382 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0860106be0 msgr2=0x7f0860198460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:02:47.724 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:02:47.722+0000 7f08679fd700 1 --2- 172.21.15.162:0/2519878382 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0860106be0 0x7f0860198460 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f085c0056d0 tx=0x7f085c0057b0 comp rx=0 tx=0).stop 2024-03-31T23:02:47.724 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:02:47.722+0000 7f08679fd700 1 -- 172.21.15.162:0/2519878382 shutdown_connections 2024-03-31T23:02:47.724 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:02:47.722+0000 7f08679fd700 1 --2- 172.21.15.162:0/2519878382 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f084c038f00 0x7f084c03b3c0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:02:47.724 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:02:47.722+0000 7f08679fd700 1 --2- 172.21.15.162:0/2519878382 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0860106be0 0x7f0860198460 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:02:47.724 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:02:47.722+0000 7f08679fd700 1 -- 172.21.15.162:0/2519878382 >> 172.21.15.162:0/2519878382 conn(0x7f0860077780 msgr2=0x7f08600fca40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:02:47.724 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:02:47.722+0000 7f08679fd700 1 -- 172.21.15.162:0/2519878382 shutdown_connections 2024-03-31T23:02:47.724 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:02:47.722+0000 7f08679fd700 1 -- 172.21.15.162:0/2519878382 wait complete. 2024-03-31T23:02:48.305 DEBUG:teuthology.orchestra.run.smithi142:mon.b> sudo journalctl -f -n 0 -u ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4@mon.b.service 2024-03-31T23:02:48.308 DEBUG:teuthology.orchestra.run.smithi162:mon.c> sudo journalctl -f -n 0 -u ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4@mon.c.service 2024-03-31T23:02:48.310 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-31T23:02:48.311 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:384f31f3d3aee7f8ba3f60846b8844c049165583 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6a9d8f08-efb2-11ee-b647-cb9ed24678a4 -- ceph mon dump -f json 2024-03-31T23:02:48.361 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:02:48 smithi046 ceph-mon[32498]: Updating smithi162:/var/lib/ceph/6a9d8f08-efb2-11ee-b647-cb9ed24678a4/config/ceph.client.admin.keyring 2024-03-31T23:02:48.361 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:02:48 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:02:48.361 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:02:48 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-31T23:02:48.362 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:02:48 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:02:48.362 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:02:48 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-31T23:02:48.362 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:02:48 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:02:48.362 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:02:48 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-31T23:02:48.362 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:02:48 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:02:49.004 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/6a9d8f08-efb2-11ee-b647-cb9ed24678a4/config/ceph.conf 2024-03-31T23:02:49.612 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:02:49 smithi046 ceph-mon[32498]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi046:172.21.15.46=a;smithi142:172.21.15.142=b;smithi162:172.21.15.162=c", "target": ["mon-mgr", ""]}]: dispatch 2024-03-31T23:02:49.612 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:02:49 smithi046 ceph-mon[32498]: Saving service mon spec with placement smithi046:172.21.15.46=a;smithi142:172.21.15.142=b;smithi162:172.21.15.162=c;count:3 2024-03-31T23:02:49.612 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:02:49 smithi046 ceph-mon[32498]: Deploying daemon mon.c on smithi162 2024-03-31T23:02:49.612 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:02:49 smithi046 ceph-mon[32498]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-31T23:02:50.356 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:02:50.354+0000 7f19031c4700 1 -- 172.21.15.162:0/200806024 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f18fc106b90 msgr2=0x7f18fc106f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:02:50.356 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:02:50.354+0000 7f19031c4700 1 --2- 172.21.15.162:0/200806024 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f18fc106b90 0x7f18fc106f70 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f18f0009910 tx=0x7f18f0009c20 comp rx=0 tx=0).stop 2024-03-31T23:02:50.356 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:02:50.355+0000 7f19031c4700 1 -- 172.21.15.162:0/200806024 shutdown_connections 2024-03-31T23:02:50.356 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:02:50.355+0000 7f19031c4700 1 --2- 172.21.15.162:0/200806024 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f18fc106b90 0x7f18fc106f70 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:02:50.356 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:02:50.355+0000 7f19031c4700 1 -- 172.21.15.162:0/200806024 >> 172.21.15.162:0/200806024 conn(0x7f18fc0fc5f0 msgr2=0x7f18fc0fea10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:02:50.356 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:02:50.355+0000 7f19031c4700 1 -- 172.21.15.162:0/200806024 shutdown_connections 2024-03-31T23:02:50.356 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:02:50.355+0000 7f19031c4700 1 -- 172.21.15.162:0/200806024 wait complete. 2024-03-31T23:02:50.356 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:02:50.355+0000 7f19031c4700 1 Processor -- start 2024-03-31T23:02:50.356 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:02:50.355+0000 7f19031c4700 1 -- start start 2024-03-31T23:02:50.357 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:02:50.356+0000 7f19031c4700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f18fc106b90 0x7f18fc198530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:02:50.357 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:02:50.356+0000 7f19031c4700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f18fc10aff0 con 0x7f18fc106b90 2024-03-31T23:02:50.358 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:02:50.356+0000 7f1900f60700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f18fc106b90 0x7f18fc198530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:02:50.358 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:02:50.356+0000 7f1900f60700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f18fc106b90 0x7f18fc198530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.162:58946/0 (socket says 172.21.15.162:58946) 2024-03-31T23:02:50.358 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:02:50.356+0000 7f1900f60700 1 -- 172.21.15.162:0/3618932929 learned_addr learned my addr 172.21.15.162:0/3618932929 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-03-31T23:02:50.358 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:02:50.356+0000 7f1900f60700 1 -- 172.21.15.162:0/3618932929 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f18fc198a70 con 0x7f18fc106b90 2024-03-31T23:02:50.358 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:02:50.357+0000 7f1900f60700 1 --2- 172.21.15.162:0/3618932929 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f18fc106b90 0x7f18fc198530 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f18f00056d0 tx=0x7f18f00057b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:02:50.358 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:02:50.357+0000 7f18f9ffb700 1 -- 172.21.15.162:0/3618932929 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f18f0026070 con 0x7f18fc106b90 2024-03-31T23:02:50.358 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:02:50.357+0000 7f19031c4700 1 -- 172.21.15.162:0/3618932929 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f18fc198d60 con 0x7f18fc106b90 2024-03-31T23:02:50.358 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:02:50.357+0000 7f18f9ffb700 1 -- 172.21.15.162:0/3618932929 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f18f000bba0 con 0x7f18fc106b90 2024-03-31T23:02:50.358 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:02:50.357+0000 7f18f9ffb700 1 -- 172.21.15.162:0/3618932929 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f18f00037f0 con 0x7f18fc106b90 2024-03-31T23:02:50.358 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:02:50.357+0000 7f19031c4700 1 -- 172.21.15.162:0/3618932929 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f18fc199220 con 0x7f18fc106b90 2024-03-31T23:02:50.359 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:02:50.358+0000 7f18f9ffb700 1 -- 172.21.15.162:0/3618932929 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 13) ==== 46172+0+0 (secure 0 0 0) 0x7f18f0029070 con 0x7f18fc106b90 2024-03-31T23:02:50.359 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:02:50.358+0000 7f19031c4700 1 -- 172.21.15.162:0/3618932929 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f18fc102200 con 0x7f18fc106b90 2024-03-31T23:02:50.359 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:02:50.358+0000 7f18f9ffb700 1 --2- 172.21.15.162:0/3618932929 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f18ec038ea0 0x7f18ec03b360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:02:50.359 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:02:50.358+0000 7f18f9ffb700 1 -- 172.21.15.162:0/3618932929 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f18f001e070 con 0x7f18fc106b90 2024-03-31T23:02:50.360 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:02:50.358+0000 7f18fbfff700 1 --2- 172.21.15.162:0/3618932929 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f18ec038ea0 0x7f18ec03b360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:02:50.360 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:02:50.359+0000 7f18fbfff700 1 --2- 172.21.15.162:0/3618932929 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f18ec038ea0 0x7f18ec03b360 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f18e8006fd0 tx=0x7f18e8006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:02:50.364 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:02:50.364+0000 7f18f9ffb700 1 -- 172.21.15.162:0/3618932929 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f18f0013070 con 0x7f18fc106b90 2024-03-31T23:02:50.642 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:02:50.641+0000 7f19031c4700 1 -- 172.21.15.162:0/3618932929 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7f18fc063870 con 0x7f18fc106b90 2024-03-31T23:02:50.642 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:02:50.642+0000 7f18f9ffb700 1 -- 172.21.15.162:0/3618932929 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) ==== 95+0+745 (secure 0 0 0) 0x7f18f0030090 con 0x7f18fc106b90 2024-03-31T23:02:50.642 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-31T23:02:50.642 INFO:teuthology.orchestra.run.smithi162.stdout:{"epoch":1,"fsid":"6a9d8f08-efb2-11ee-b647-cb9ed24678a4","modified":"2024-03-31T23:01:12.507350Z","created":"2024-03-31T23:01:12.507350Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3300","nonce":0},{"type":"v1","addr":"172.21.15.46:6789","nonce":0}]},"addr":"172.21.15.46:6789/0","public_addr":"172.21.15.46:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-31T23:02:50.644 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:02:50.644+0000 7f19031c4700 1 -- 172.21.15.162:0/3618932929 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f18ec038ea0 msgr2=0x7f18ec03b360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:02:50.645 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:02:50.644+0000 7f19031c4700 1 --2- 172.21.15.162:0/3618932929 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f18ec038ea0 0x7f18ec03b360 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f18e8006fd0 tx=0x7f18e8006e40 comp rx=0 tx=0).stop 2024-03-31T23:02:50.645 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:02:50.644+0000 7f19031c4700 1 -- 172.21.15.162:0/3618932929 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f18fc106b90 msgr2=0x7f18fc198530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:02:50.645 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:02:50.644+0000 7f19031c4700 1 --2- 172.21.15.162:0/3618932929 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f18fc106b90 0x7f18fc198530 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f18f00056d0 tx=0x7f18f00057b0 comp rx=0 tx=0).stop 2024-03-31T23:02:50.645 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:02:50.644+0000 7f19031c4700 1 -- 172.21.15.162:0/3618932929 shutdown_connections 2024-03-31T23:02:50.645 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:02:50.644+0000 7f19031c4700 1 --2- 172.21.15.162:0/3618932929 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f18ec038ea0 0x7f18ec03b360 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:02:50.645 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:02:50.644+0000 7f19031c4700 1 --2- 172.21.15.162:0/3618932929 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f18fc106b90 0x7f18fc198530 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:02:50.645 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:02:50.644+0000 7f19031c4700 1 -- 172.21.15.162:0/3618932929 >> 172.21.15.162:0/3618932929 conn(0x7f18fc0fc5f0 msgr2=0x7f18fc0fd380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:02:50.645 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:02:50.644+0000 7f19031c4700 1 -- 172.21.15.162:0/3618932929 shutdown_connections 2024-03-31T23:02:50.645 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:02:50.644+0000 7f19031c4700 1 -- 172.21.15.162:0/3618932929 wait complete. 2024-03-31T23:02:50.645 INFO:teuthology.orchestra.run.smithi162.stderr:dumped monmap epoch 1 2024-03-31T23:02:50.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:02:50 smithi046 ceph-mon[32498]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-31T23:02:51.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:02:51 smithi046 ceph-mon[32498]: from='client.? 172.21.15.162:0/3618932929' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-31T23:02:52.784 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-31T23:02:52.784 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:384f31f3d3aee7f8ba3f60846b8844c049165583 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6a9d8f08-efb2-11ee-b647-cb9ed24678a4 -- ceph mon dump -f json 2024-03-31T23:02:52.862 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:02:52 smithi046 ceph-mon[32498]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-31T23:02:53.144 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/6a9d8f08-efb2-11ee-b647-cb9ed24678a4/mon.c/config 2024-03-31T23:02:54.613 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:02:54.611+0000 7f6141db3700 1 -- 172.21.15.162:0/2624156293 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f613c100e60 msgr2=0x7f613c101240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:02:54.613 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:02:54.611+0000 7f6141db3700 1 --2- 172.21.15.162:0/2624156293 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f613c100e60 0x7f613c101240 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f612c00b0a0 tx=0x7f612c00b3b0 comp rx=0 tx=0).stop 2024-03-31T23:02:54.613 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:02:54.611+0000 7f6141db3700 1 -- 172.21.15.162:0/2624156293 shutdown_connections 2024-03-31T23:02:54.613 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:02:54.611+0000 7f6141db3700 1 --2- 172.21.15.162:0/2624156293 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f613c100e60 0x7f613c101240 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:02:54.613 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:02:54.611+0000 7f6141db3700 1 -- 172.21.15.162:0/2624156293 >> 172.21.15.162:0/2624156293 conn(0x7f613c0fc6f0 msgr2=0x7f613c0feb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:02:54.613 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:02:54.611+0000 7f6141db3700 1 -- 172.21.15.162:0/2624156293 shutdown_connections 2024-03-31T23:02:54.613 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:02:54.611+0000 7f6141db3700 1 -- 172.21.15.162:0/2624156293 wait complete. 2024-03-31T23:02:54.614 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:02:54.612+0000 7f6141db3700 1 Processor -- start 2024-03-31T23:02:54.614 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:02:54.612+0000 7f6141db3700 1 -- start start 2024-03-31T23:02:54.614 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:02:54.612+0000 7f6141db3700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f613c100e60 0x7f613c198550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:02:54.614 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:02:54.612+0000 7f6141db3700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f613c10b0e0 con 0x7f613c100e60 2024-03-31T23:02:54.614 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:02:54.613+0000 7f613b7fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f613c100e60 0x7f613c198550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:02:54.614 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:02:54.613+0000 7f613b7fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f613c100e60 0x7f613c198550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.162:58962/0 (socket says 172.21.15.162:58962) 2024-03-31T23:02:54.614 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:02:54.613+0000 7f613b7fe700 1 -- 172.21.15.162:0/1516844570 learned_addr learned my addr 172.21.15.162:0/1516844570 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-03-31T23:02:54.614 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:02:54.613+0000 7f613b7fe700 1 -- 172.21.15.162:0/1516844570 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f613c198a90 con 0x7f613c100e60 2024-03-31T23:02:54.615 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:02:54.613+0000 7f613b7fe700 1 --2- 172.21.15.162:0/1516844570 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f613c100e60 0x7f613c198550 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f612c0076a0 tx=0x7f612c003ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:02:54.616 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:02:54.613+0000 7f6138ff9700 1 -- 172.21.15.162:0/1516844570 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f612c017040 con 0x7f613c100e60 2024-03-31T23:02:54.616 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:02:54.613+0000 7f6141db3700 1 -- 172.21.15.162:0/1516844570 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f613c198d20 con 0x7f613c100e60 2024-03-31T23:02:54.616 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:02:54.613+0000 7f6138ff9700 1 -- 172.21.15.162:0/1516844570 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f612c022070 con 0x7f613c100e60 2024-03-31T23:02:54.616 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:02:54.613+0000 7f6138ff9700 1 -- 172.21.15.162:0/1516844570 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f612c01d5e0 con 0x7f613c100e60 2024-03-31T23:02:54.616 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:02:54.613+0000 7f6141db3700 1 -- 172.21.15.162:0/1516844570 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f613c1991e0 con 0x7f613c100e60 2024-03-31T23:02:54.616 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:02:54.614+0000 7f6141db3700 1 -- 172.21.15.162:0/1516844570 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f613c102340 con 0x7f613c100e60 2024-03-31T23:02:54.616 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:02:54.614+0000 7f6138ff9700 1 -- 172.21.15.162:0/1516844570 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 13) ==== 46172+0+0 (secure 0 0 0) 0x7f612c004580 con 0x7f613c100e60 2024-03-31T23:02:54.616 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:02:54.614+0000 7f6138ff9700 1 --2- 172.21.15.162:0/1516844570 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f6128038ea0 0x7f612803b360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:02:54.616 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:02:54.614+0000 7f6138ff9700 1 -- 172.21.15.162:0/1516844570 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f612c00f050 con 0x7f613c100e60 2024-03-31T23:02:54.616 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:02:54.615+0000 7f613affd700 1 --2- 172.21.15.162:0/1516844570 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f6128038ea0 0x7f612803b360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:02:54.616 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:02:54.615+0000 7f613affd700 1 --2- 172.21.15.162:0/1516844570 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f6128038ea0 0x7f612803b360 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f6124006fd0 tx=0x7f6124006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:02:54.620 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:02:54.620+0000 7f6138ff9700 1 -- 172.21.15.162:0/1516844570 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f612c01c3b0 con 0x7f613c100e60 2024-03-31T23:02:54.698 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 podman[36168]: 2024-03-31 23:02:54.673805621 +0000 UTC m=+0.961506590 container start ce17b19294be5c9adae1c26e57056358f993406384d7c6c6043bfbc26e4c9d83 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mon-c, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.build-date=20240326, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, RELEASE=squid-384f31f, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , ceph=True) 2024-03-31T23:02:54.698 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: set uid:gid to 167:167 (ceph:ceph) 2024-03-31T23:02:54.698 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: ceph version 19.0.0-1977-g384f31f3 (384f31f3d3aee7f8ba3f60846b8844c049165583) squid (dev), process ceph-mon, pid 2 2024-03-31T23:02:54.698 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: pidfile_write: ignore empty --pid-file 2024-03-31T23:02:54.731 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:02:54.730+0000 7f6138ff9700 1 -- 172.21.15.162:0/1516844570 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7f612c0093a0 con 0x7f613c100e60 2024-03-31T23:02:54.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:02:54 smithi046 ceph-mon[32498]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-31T23:02:54.909 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:02:54.908+0000 7f6141db3700 1 -- 172.21.15.162:0/1516844570 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7f613c063870 con 0x7f613c100e60 2024-03-31T23:02:55.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: load: jerasure load: lrc 2024-03-31T23:02:55.047 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: rocksdb: RocksDB version: 7.9.2 2024-03-31T23:02:55.047 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: rocksdb: Git sha 0 2024-03-31T23:02:55.047 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: rocksdb: Compile date 2024-03-31 16:17:26 2024-03-31T23:02:55.047 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: rocksdb: DB SUMMARY 2024-03-31T23:02:55.047 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: rocksdb: DB Session ID: 8WQZZ65HBIL7KD50E03L 2024-03-31T23:02:55.047 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: rocksdb: CURRENT file: CURRENT 2024-03-31T23:02:55.047 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: rocksdb: IDENTITY file: IDENTITY 2024-03-31T23:02:55.047 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-03-31T23:02:55.047 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: rocksdb: SST files in /var/lib/ceph/mon/ceph-c/store.db dir, Total Num: 0, files: 2024-03-31T23:02:55.047 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-c/store.db: 000004.log size: 511 ; 2024-03-31T23:02:55.047 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: rocksdb: Options.error_if_exists: 0 2024-03-31T23:02:55.047 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: rocksdb: Options.create_if_missing: 0 2024-03-31T23:02:55.047 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: rocksdb: Options.paranoid_checks: 1 2024-03-31T23:02:55.048 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: rocksdb: Options.flush_verify_memtable_count: 1 2024-03-31T23:02:55.048 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-31T23:02:55.048 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-03-31T23:02:55.048 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: rocksdb: Options.env: 0x5588ebfa4c60 2024-03-31T23:02:55.048 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: rocksdb: Options.fs: PosixFileSystem 2024-03-31T23:02:55.048 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: rocksdb: Options.info_log: 0x5588ee71fae0 2024-03-31T23:02:55.048 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: rocksdb: Options.max_file_opening_threads: 16 2024-03-31T23:02:55.048 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: rocksdb: Options.statistics: (nil) 2024-03-31T23:02:55.048 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: rocksdb: Options.use_fsync: 0 2024-03-31T23:02:55.048 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: rocksdb: Options.max_log_file_size: 0 2024-03-31T23:02:55.048 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-31T23:02:55.048 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-31T23:02:55.048 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: rocksdb: Options.keep_log_file_num: 1000 2024-03-31T23:02:55.048 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: rocksdb: Options.recycle_log_file_num: 0 2024-03-31T23:02:55.049 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: rocksdb: Options.allow_fallocate: 1 2024-03-31T23:02:55.049 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: rocksdb: Options.allow_mmap_reads: 0 2024-03-31T23:02:55.049 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: rocksdb: Options.allow_mmap_writes: 0 2024-03-31T23:02:55.049 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: rocksdb: Options.use_direct_reads: 0 2024-03-31T23:02:55.049 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-31T23:02:55.049 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: rocksdb: Options.create_missing_column_families: 0 2024-03-31T23:02:55.049 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: rocksdb: Options.db_log_dir: 2024-03-31T23:02:55.049 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: rocksdb: Options.wal_dir: 2024-03-31T23:02:55.049 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-31T23:02:55.049 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-31T23:02:55.049 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-31T23:02:55.049 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-31T23:02:55.049 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-31T23:02:55.050 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-31T23:02:55.050 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: rocksdb: Options.advise_random_on_open: 1 2024-03-31T23:02:55.050 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: rocksdb: Options.db_write_buffer_size: 0 2024-03-31T23:02:55.050 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: rocksdb: Options.write_buffer_manager: 0x5588ee6f21e0 2024-03-31T23:02:55.050 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-31T23:02:55.050 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-31T23:02:55.050 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-31T23:02:55.050 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: rocksdb: Options.rate_limiter: (nil) 2024-03-31T23:02:55.050 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-31T23:02:55.050 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: rocksdb: Options.wal_recovery_mode: 2 2024-03-31T23:02:55.050 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: rocksdb: Options.enable_thread_tracking: 0 2024-03-31T23:02:55.050 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: rocksdb: Options.enable_pipelined_write: 0 2024-03-31T23:02:55.050 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: rocksdb: Options.unordered_write: 0 2024-03-31T23:02:55.050 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-31T23:02:55.051 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-31T23:02:55.051 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-31T23:02:55.051 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-31T23:02:55.051 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: rocksdb: Options.row_cache: None 2024-03-31T23:02:55.051 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: rocksdb: Options.wal_filter: None 2024-03-31T23:02:55.051 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-31T23:02:55.051 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: rocksdb: Options.allow_ingest_behind: 0 2024-03-31T23:02:55.051 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: rocksdb: Options.two_write_queues: 0 2024-03-31T23:02:55.051 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: rocksdb: Options.manual_wal_flush: 0 2024-03-31T23:02:55.051 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: rocksdb: Options.wal_compression: 0 2024-03-31T23:02:55.051 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: rocksdb: Options.atomic_flush: 0 2024-03-31T23:02:55.051 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-31T23:02:55.051 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-31T23:02:55.051 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-31T23:02:55.051 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: rocksdb: Options.log_readahead_size: 0 2024-03-31T23:02:55.052 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-31T23:02:55.052 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: rocksdb: Options.best_efforts_recovery: 0 2024-03-31T23:02:55.052 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-31T23:02:55.052 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-31T23:02:55.052 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: rocksdb: Options.allow_data_in_errors: 0 2024-03-31T23:02:55.052 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: rocksdb: Options.db_host_id: __hostname__ 2024-03-31T23:02:55.052 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: rocksdb: Options.enforce_single_del_contracts: true 2024-03-31T23:02:55.052 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: rocksdb: Options.max_background_jobs: 2 2024-03-31T23:02:55.052 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: rocksdb: Options.max_background_compactions: -1 2024-03-31T23:02:55.052 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: rocksdb: Options.max_subcompactions: 1 2024-03-31T23:02:55.054 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-31T23:02:55.054 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-31T23:02:55.054 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-31T23:02:55.054 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: rocksdb: Options.max_total_wal_size: 0 2024-03-31T23:02:55.054 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-31T23:02:55.054 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-31T23:02:55.054 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-31T23:02:55.054 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-31T23:02:55.054 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: rocksdb: Options.max_open_files: -1 2024-03-31T23:02:55.054 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: rocksdb: Options.bytes_per_sync: 0 2024-03-31T23:02:55.054 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-31T23:02:55.055 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-31T23:02:55.055 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: rocksdb: Options.compaction_readahead_size: 0 2024-03-31T23:02:55.055 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: rocksdb: Options.max_background_flushes: -1 2024-03-31T23:02:55.055 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: rocksdb: Compression algorithms supported: 2024-03-31T23:02:55.055 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-31T23:02:55.055 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: rocksdb: kZSTD supported: 0 2024-03-31T23:02:55.055 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: rocksdb: kXpressCompression supported: 0 2024-03-31T23:02:55.055 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: rocksdb: kLZ4HCCompression supported: 1 2024-03-31T23:02:55.055 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: rocksdb: kZlibCompression supported: 1 2024-03-31T23:02:55.055 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: rocksdb: kSnappyCompression supported: 1 2024-03-31T23:02:55.055 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: rocksdb: kLZ4Compression supported: 1 2024-03-31T23:02:55.055 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: rocksdb: kBZip2Compression supported: 0 2024-03-31T23:02:55.055 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-31T23:02:55.055 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: rocksdb: DMutex implementation: pthread_mutex_t 2024-03-31T23:02:55.056 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 2024-03-31T23:02:55.056 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-03-31T23:02:55.056 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-31T23:02:55.056 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: rocksdb: Options.merge_operator: 2024-03-31T23:02:55.056 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: rocksdb: Options.compaction_filter: None 2024-03-31T23:02:55.056 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: rocksdb: Options.compaction_filter_factory: None 2024-03-31T23:02:55.056 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: rocksdb: Options.sst_partitioner_factory: None 2024-03-31T23:02:55.056 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-31T23:02:55.056 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-31T23:02:55.056 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x5588ee71fbe0) 2024-03-31T23:02:55.056 INFO:journalctl@ceph.mon.c.smithi162.stdout: cache_index_and_filter_blocks: 1 2024-03-31T23:02:55.056 INFO:journalctl@ceph.mon.c.smithi162.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-31T23:02:55.056 INFO:journalctl@ceph.mon.c.smithi162.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-31T23:02:55.056 INFO:journalctl@ceph.mon.c.smithi162.stdout: pin_top_level_index_and_filter: 1 2024-03-31T23:02:55.057 INFO:journalctl@ceph.mon.c.smithi162.stdout: index_type: 0 2024-03-31T23:02:55.057 INFO:journalctl@ceph.mon.c.smithi162.stdout: data_block_index_type: 0 2024-03-31T23:02:55.057 INFO:journalctl@ceph.mon.c.smithi162.stdout: index_shortening: 1 2024-03-31T23:02:55.057 INFO:journalctl@ceph.mon.c.smithi162.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-31T23:02:55.057 INFO:journalctl@ceph.mon.c.smithi162.stdout: checksum: 4 2024-03-31T23:02:55.057 INFO:journalctl@ceph.mon.c.smithi162.stdout: no_block_cache: 0 2024-03-31T23:02:55.057 INFO:journalctl@ceph.mon.c.smithi162.stdout: block_cache: 0x5588ee723090 2024-03-31T23:02:55.057 INFO:journalctl@ceph.mon.c.smithi162.stdout: block_cache_name: BinnedLRUCache 2024-03-31T23:02:55.057 INFO:journalctl@ceph.mon.c.smithi162.stdout: block_cache_options: 2024-03-31T23:02:55.057 INFO:journalctl@ceph.mon.c.smithi162.stdout: capacity : 536870912 2024-03-31T23:02:55.057 INFO:journalctl@ceph.mon.c.smithi162.stdout: num_shard_bits : 4 2024-03-31T23:02:55.057 INFO:journalctl@ceph.mon.c.smithi162.stdout: strict_capacity_limit : 0 2024-03-31T23:02:55.057 INFO:journalctl@ceph.mon.c.smithi162.stdout: high_pri_pool_ratio: 0.000 2024-03-31T23:02:55.057 INFO:journalctl@ceph.mon.c.smithi162.stdout: block_cache_compressed: (nil) 2024-03-31T23:02:55.058 INFO:journalctl@ceph.mon.c.smithi162.stdout: persistent_cache: (nil) 2024-03-31T23:02:55.058 INFO:journalctl@ceph.mon.c.smithi162.stdout: block_size: 4096 2024-03-31T23:02:55.058 INFO:journalctl@ceph.mon.c.smithi162.stdout: block_size_deviation: 10 2024-03-31T23:02:55.058 INFO:journalctl@ceph.mon.c.smithi162.stdout: block_restart_interval: 16 2024-03-31T23:02:55.058 INFO:journalctl@ceph.mon.c.smithi162.stdout: index_block_restart_interval: 1 2024-03-31T23:02:55.058 INFO:journalctl@ceph.mon.c.smithi162.stdout: metadata_block_size: 4096 2024-03-31T23:02:55.058 INFO:journalctl@ceph.mon.c.smithi162.stdout: partition_filters: 0 2024-03-31T23:02:55.058 INFO:journalctl@ceph.mon.c.smithi162.stdout: use_delta_encoding: 1 2024-03-31T23:02:55.058 INFO:journalctl@ceph.mon.c.smithi162.stdout: filter_policy: bloomfilter 2024-03-31T23:02:55.058 INFO:journalctl@ceph.mon.c.smithi162.stdout: whole_key_filtering: 1 2024-03-31T23:02:55.058 INFO:journalctl@ceph.mon.c.smithi162.stdout: verify_compression: 0 2024-03-31T23:02:55.058 INFO:journalctl@ceph.mon.c.smithi162.stdout: read_amp_bytes_per_bit: 0 2024-03-31T23:02:55.058 INFO:journalctl@ceph.mon.c.smithi162.stdout: format_version: 5 2024-03-31T23:02:55.058 INFO:journalctl@ceph.mon.c.smithi162.stdout: enable_index_compression: 1 2024-03-31T23:02:55.059 INFO:journalctl@ceph.mon.c.smithi162.stdout: block_align: 0 2024-03-31T23:02:55.059 INFO:journalctl@ceph.mon.c.smithi162.stdout: max_auto_readahead_size: 262144 2024-03-31T23:02:55.059 INFO:journalctl@ceph.mon.c.smithi162.stdout: prepopulate_block_cache: 0 2024-03-31T23:02:55.059 INFO:journalctl@ceph.mon.c.smithi162.stdout: initial_auto_readahead_size: 8192 2024-03-31T23:02:55.059 INFO:journalctl@ceph.mon.c.smithi162.stdout: num_file_reads_for_auto_readahead: 2 2024-03-31T23:02:55.059 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: rocksdb: Options.write_buffer_size: 33554432 2024-03-31T23:02:55.059 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: rocksdb: Options.max_write_buffer_number: 2 2024-03-31T23:02:55.059 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: rocksdb: Options.compression: NoCompression 2024-03-31T23:02:55.059 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: rocksdb: Options.bottommost_compression: Disabled 2024-03-31T23:02:55.059 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: rocksdb: Options.prefix_extractor: nullptr 2024-03-31T23:02:55.059 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-31T23:02:55.059 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: rocksdb: Options.num_levels: 7 2024-03-31T23:02:55.059 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-31T23:02:55.059 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-31T23:02:55.059 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-31T23:02:55.060 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-31T23:02:55.060 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-31T23:02:55.060 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-31T23:02:55.060 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-31T23:02:55.060 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-31T23:02:55.060 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-31T23:02:55.060 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-31T23:02:55.060 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-03-31T23:02:55.060 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-03-31T23:02:55.060 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-31T23:02:55.060 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: rocksdb: Options.compression_opts.level: 32767 2024-03-31T23:02:55.060 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: rocksdb: Options.compression_opts.strategy: 0 2024-03-31T23:02:55.060 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-31T23:02:55.060 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-31T23:02:55.061 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-03-31T23:02:55.061 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-31T23:02:55.061 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: rocksdb: Options.compression_opts.enabled: false 2024-03-31T23:02:55.061 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-03-31T23:02:55.061 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-31T23:02:55.061 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-31T23:02:55.061 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-31T23:02:55.061 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: rocksdb: Options.target_file_size_base: 67108864 2024-03-31T23:02:55.061 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-31T23:02:55.061 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-31T23:02:55.061 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-31T23:02:55.061 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-31T23:02:55.061 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-31T23:02:55.061 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-31T23:02:55.061 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-31T23:02:55.062 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-31T23:02:55.062 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-31T23:02:55.062 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-31T23:02:55.062 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-31T23:02:55.062 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-31T23:02:55.062 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-31T23:02:55.062 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-03-31T23:02:55.062 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: rocksdb: Options.arena_block_size: 1048576 2024-03-31T23:02:55.062 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-31T23:02:55.062 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-31T23:02:55.062 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: rocksdb: Options.disable_auto_compactions: 0 2024-03-31T23:02:55.062 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-31T23:02:55.062 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-31T23:02:55.062 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-31T23:02:55.063 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-31T23:02:55.063 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-31T23:02:55.063 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-31T23:02:55.063 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-31T23:02:55.063 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-31T23:02:55.063 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-31T23:02:55.063 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-31T23:02:55.063 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: rocksdb: Options.table_properties_collectors: 2024-03-31T23:02:55.063 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: rocksdb: Options.inplace_update_support: 0 2024-03-31T23:02:55.063 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-31T23:02:55.063 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-31T23:02:55.063 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-31T23:02:55.063 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-31T23:02:55.064 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: rocksdb: Options.bloom_locality: 0 2024-03-31T23:02:55.064 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: rocksdb: Options.max_successive_merges: 0 2024-03-31T23:02:55.064 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-31T23:02:55.064 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: rocksdb: Options.paranoid_file_checks: 0 2024-03-31T23:02:55.064 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: rocksdb: Options.force_consistency_checks: 1 2024-03-31T23:02:55.064 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: rocksdb: Options.report_bg_io_stats: 0 2024-03-31T23:02:55.064 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: rocksdb: Options.ttl: 2592000 2024-03-31T23:02:55.064 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-31T23:02:55.064 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-03-31T23:02:55.064 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-03-31T23:02:55.064 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: rocksdb: Options.enable_blob_files: false 2024-03-31T23:02:55.064 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: rocksdb: Options.min_blob_size: 0 2024-03-31T23:02:55.064 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: rocksdb: Options.blob_file_size: 268435456 2024-03-31T23:02:55.064 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-31T23:02:55.064 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-31T23:02:55.065 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-31T23:02:55.065 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-03-31T23:02:55.065 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-03-31T23:02:55.065 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: rocksdb: Options.blob_file_starting_level: 0 2024-03-31T23:02:55.065 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-03-31T23:02:55.065 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-03-31T23:02:55.065 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-03-31T23:02:55.065 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: cf2ab1bf-d800-45cb-8642-6aab4bc5b3db 2024-03-31T23:02:55.065 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711926174704426, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-03-31T23:02:55.065 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-03-31T23:02:55.065 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711926174704941, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1622, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 523, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 401, "raw_average_value_size": 80, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1711926174, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "cf2ab1bf-d800-45cb-8642-6aab4bc5b3db", "db_session_id": "8WQZZ65HBIL7KD50E03L", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-03-31T23:02:55.065 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711926174705041, "job": 1, "event": "recovery_finished"} 2024-03-31T23:02:55.065 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-03-31T23:02:55.065 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-c/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-31T23:02:55.066 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x5588ee814000 2024-03-31T23:02:55.066 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: rocksdb: DB pointer 0x5588ee7fe000 2024-03-31T23:02:55.066 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-03-31T23:02:55.066 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-03-31T23:02:55.066 INFO:journalctl@ceph.mon.c.smithi162.stdout: ** DB Stats ** 2024-03-31T23:02:55.066 INFO:journalctl@ceph.mon.c.smithi162.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-31T23:02:55.066 INFO:journalctl@ceph.mon.c.smithi162.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-31T23:02:55.066 INFO:journalctl@ceph.mon.c.smithi162.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-31T23:02:55.066 INFO:journalctl@ceph.mon.c.smithi162.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-31T23:02:55.066 INFO:journalctl@ceph.mon.c.smithi162.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-31T23:02:55.066 INFO:journalctl@ceph.mon.c.smithi162.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-31T23:02:55.066 INFO:journalctl@ceph.mon.c.smithi162.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-31T23:02:55.066 INFO:journalctl@ceph.mon.c.smithi162.stdout: 2024-03-31T23:02:55.067 INFO:journalctl@ceph.mon.c.smithi162.stdout: ** Compaction Stats [default] ** 2024-03-31T23:02:55.067 INFO:journalctl@ceph.mon.c.smithi162.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-31T23:02:55.067 INFO:journalctl@ceph.mon.c.smithi162.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-03-31T23:02:55.067 INFO:journalctl@ceph.mon.c.smithi162.stdout: L0 1/0 1.58 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.2 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-31T23:02:55.067 INFO:journalctl@ceph.mon.c.smithi162.stdout: Sum 1/0 1.58 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.2 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-31T23:02:55.067 INFO:journalctl@ceph.mon.c.smithi162.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.2 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-31T23:02:55.067 INFO:journalctl@ceph.mon.c.smithi162.stdout: 2024-03-31T23:02:55.067 INFO:journalctl@ceph.mon.c.smithi162.stdout: ** Compaction Stats [default] ** 2024-03-31T23:02:55.067 INFO:journalctl@ceph.mon.c.smithi162.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-31T23:02:55.067 INFO:journalctl@ceph.mon.c.smithi162.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-31T23:02:55.067 INFO:journalctl@ceph.mon.c.smithi162.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 3.2 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-31T23:02:55.067 INFO:journalctl@ceph.mon.c.smithi162.stdout: 2024-03-31T23:02:55.067 INFO:journalctl@ceph.mon.c.smithi162.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-03-31T23:02:55.067 INFO:journalctl@ceph.mon.c.smithi162.stdout: 2024-03-31T23:02:55.068 INFO:journalctl@ceph.mon.c.smithi162.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-31T23:02:55.068 INFO:journalctl@ceph.mon.c.smithi162.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-31T23:02:55.068 INFO:journalctl@ceph.mon.c.smithi162.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-31T23:02:55.068 INFO:journalctl@ceph.mon.c.smithi162.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-31T23:02:55.068 INFO:journalctl@ceph.mon.c.smithi162.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-31T23:02:55.068 INFO:journalctl@ceph.mon.c.smithi162.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-31T23:02:55.068 INFO:journalctl@ceph.mon.c.smithi162.stdout: Cumulative compaction: 0.00 GB write, 0.51 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-31T23:02:55.068 INFO:journalctl@ceph.mon.c.smithi162.stdout: Interval compaction: 0.00 GB write, 0.51 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-31T23:02:55.068 INFO:journalctl@ceph.mon.c.smithi162.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-31T23:02:55.068 INFO:journalctl@ceph.mon.c.smithi162.stdout: Block cache BinnedLRUCache@0x5588ee723090#2 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 8e-06 secs_since: 0 2024-03-31T23:02:55.068 INFO:journalctl@ceph.mon.c.smithi162.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-03-31T23:02:55.068 INFO:journalctl@ceph.mon.c.smithi162.stdout: 2024-03-31T23:02:55.068 INFO:journalctl@ceph.mon.c.smithi162.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-31T23:02:55.068 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: mon.c does not exist in monmap, will attempt to join an existing cluster 2024-03-31T23:02:55.068 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: using public_addr v2:172.21.15.162:0/0 -> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] 2024-03-31T23:02:55.069 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: starting mon.c rank -1 at public addrs [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] at bind addrs [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon_data /var/lib/ceph/mon/ceph-c fsid 6a9d8f08-efb2-11ee-b647-cb9ed24678a4 2024-03-31T23:02:55.069 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: mon.c@-1(???) e0 preinit fsid 6a9d8f08-efb2-11ee-b647-cb9ed24678a4 2024-03-31T23:02:55.069 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: mon.c@-1(synchronizing).mds e1 new map 2024-03-31T23:02:55.069 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: mon.c@-1(synchronizing).mds e1 print_map 2024-03-31T23:02:55.069 INFO:journalctl@ceph.mon.c.smithi162.stdout: e1 2024-03-31T23:02:55.069 INFO:journalctl@ceph.mon.c.smithi162.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-31T23:02:55.069 INFO:journalctl@ceph.mon.c.smithi162.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-03-31T23:02:55.069 INFO:journalctl@ceph.mon.c.smithi162.stdout: legacy client fscid: -1 2024-03-31T23:02:55.069 INFO:journalctl@ceph.mon.c.smithi162.stdout: 2024-03-31T23:02:55.069 INFO:journalctl@ceph.mon.c.smithi162.stdout: No filesystems configured 2024-03-31T23:02:55.069 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: mon.c@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-03-31T23:02:55.069 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: mon.c@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-03-31T23:02:55.069 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: mon.c@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-03-31T23:02:55.069 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: mon.c@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-03-31T23:02:55.070 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: mon.c@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-03-31T23:02:55.070 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: mon.c@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-03-31T23:02:55.070 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: mon.c@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-03-31T23:02:55.070 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-31T23:02:55.070 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-31T23:02:55.070 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-31T23:02:55.070 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-31T23:02:55.070 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:02:55.070 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-31T23:02:55.070 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-31T23:02:55.070 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi162", "target": ["mon-mgr", ""]}]: dispatch 2024-03-31T23:02:55.070 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:02:55.070 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:02:55.070 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:02:55.071 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:02:55.071 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi142", "name": "osd_memory_target"}]: dispatch 2024-03-31T23:02:55.071 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:02:55.071 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-31T23:02:55.071 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: Updating smithi142:/etc/ceph/ceph.conf 2024-03-31T23:02:55.071 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: Updating smithi142:/var/lib/ceph/6a9d8f08-efb2-11ee-b647-cb9ed24678a4/config/ceph.conf 2024-03-31T23:02:55.071 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: Deploying cephadm binary to smithi162 2024-03-31T23:02:55.071 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: Updating smithi142:/etc/ceph/ceph.client.admin.keyring 2024-03-31T23:02:55.071 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-31T23:02:55.071 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: Updating smithi142:/var/lib/ceph/6a9d8f08-efb2-11ee-b647-cb9ed24678a4/config/ceph.client.admin.keyring 2024-03-31T23:02:55.071 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:02:55.071 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:02:55.071 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:02:55.071 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-31T23:02:55.072 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:02:55.072 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: Added host smithi162 2024-03-31T23:02:55.072 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-31T23:02:55.072 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-31T23:02:55.072 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:02:55.072 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:02:55.072 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-31T23:02:55.072 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:02:55.072 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-31T23:02:55.072 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-31T23:02:55.072 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: from='client.? 172.21.15.46:0/3677361043' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-03-31T23:02:55.072 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: from='client.? 172.21.15.46:0/3677361043' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-03-31T23:02:55.072 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: osdmap e4: 0 total, 0 up, 0 in 2024-03-31T23:02:55.072 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:02:55.073 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:02:55.073 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:02:55.073 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:02:55.073 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi162", "name": "osd_memory_target"}]: dispatch 2024-03-31T23:02:55.073 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:02:55.073 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-31T23:02:55.073 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: Updating smithi162:/etc/ceph/ceph.conf 2024-03-31T23:02:55.073 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: Updating smithi162:/var/lib/ceph/6a9d8f08-efb2-11ee-b647-cb9ed24678a4/config/ceph.conf 2024-03-31T23:02:55.073 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: Updating smithi162:/etc/ceph/ceph.client.admin.keyring 2024-03-31T23:02:55.073 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-31T23:02:55.073 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:02:55.073 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:02:55.073 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:02:55.073 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: Updating smithi162:/var/lib/ceph/6a9d8f08-efb2-11ee-b647-cb9ed24678a4/config/ceph.client.admin.keyring 2024-03-31T23:02:55.074 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:02:55.074 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-31T23:02:55.074 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:02:55.074 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-31T23:02:55.074 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:02:55.074 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-31T23:02:55.074 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:02:55.074 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi046:172.21.15.46=a;smithi142:172.21.15.142=b;smithi162:172.21.15.162=c", "target": ["mon-mgr", ""]}]: dispatch 2024-03-31T23:02:55.074 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: Saving service mon spec with placement smithi046:172.21.15.46=a;smithi142:172.21.15.142=b;smithi162:172.21.15.162=c;count:3 2024-03-31T23:02:55.074 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: Deploying daemon mon.c on smithi162 2024-03-31T23:02:55.074 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-31T23:02:55.074 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-31T23:02:55.074 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: from='client.? 172.21.15.162:0/3618932929' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-31T23:02:55.074 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-31T23:02:55.075 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-31T23:02:55.075 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 ceph-mon[36237]: mon.c@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-03-31T23:02:55.075 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 bash[36168]: ce17b19294be5c9adae1c26e57056358f993406384d7c6c6043bfbc26e4c9d83 2024-03-31T23:02:55.075 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:54 smithi162 systemd[1]: Started Ceph mon.c for 6a9d8f08-efb2-11ee-b647-cb9ed24678a4. 2024-03-31T23:02:59.743 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:02:59.742+0000 7f6138ff9700 1 -- 172.21.15.162:0/1516844570 <== mon.0 v2:172.21.15.46:3300/0 8 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) ==== 95+0+1015 (secure 0 0 0) 0x7f612c0093a0 con 0x7f613c100e60 2024-03-31T23:02:59.743 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-31T23:02:59.744 INFO:teuthology.orchestra.run.smithi162.stdout:{"epoch":2,"fsid":"6a9d8f08-efb2-11ee-b647-cb9ed24678a4","modified":"2024-03-31T23:02:54.730244Z","created":"2024-03-31T23:01:12.507350Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3300","nonce":0},{"type":"v1","addr":"172.21.15.46:6789","nonce":0}]},"addr":"172.21.15.46:6789/0","public_addr":"172.21.15.46:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:3300","nonce":0},{"type":"v1","addr":"172.21.15.162:6789","nonce":0}]},"addr":"172.21.15.162:6789/0","public_addr":"172.21.15.162:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-03-31T23:02:59.746 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:02:59.746+0000 7f6141db3700 1 -- 172.21.15.162:0/1516844570 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f6128038ea0 msgr2=0x7f612803b360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:02:59.746 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:02:59.746+0000 7f6141db3700 1 --2- 172.21.15.162:0/1516844570 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f6128038ea0 0x7f612803b360 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f6124006fd0 tx=0x7f6124006e40 comp rx=0 tx=0).stop 2024-03-31T23:02:59.746 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:02:59.746+0000 7f6141db3700 1 -- 172.21.15.162:0/1516844570 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f613c100e60 msgr2=0x7f613c198550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:02:59.746 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:02:59.746+0000 7f6141db3700 1 --2- 172.21.15.162:0/1516844570 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f613c100e60 0x7f613c198550 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f612c0076a0 tx=0x7f612c003ce0 comp rx=0 tx=0).stop 2024-03-31T23:02:59.747 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:02:59.746+0000 7f6141db3700 1 -- 172.21.15.162:0/1516844570 shutdown_connections 2024-03-31T23:02:59.747 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:02:59.746+0000 7f6141db3700 1 --2- 172.21.15.162:0/1516844570 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f6128038ea0 0x7f612803b360 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:02:59.747 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:02:59.746+0000 7f6141db3700 1 --2- 172.21.15.162:0/1516844570 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f613c100e60 0x7f613c198550 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:02:59.747 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:02:59.746+0000 7f6141db3700 1 -- 172.21.15.162:0/1516844570 >> 172.21.15.162:0/1516844570 conn(0x7f613c0fc6f0 msgr2=0x7f613c102830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:02:59.747 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:02:59.746+0000 7f6141db3700 1 -- 172.21.15.162:0/1516844570 shutdown_connections 2024-03-31T23:02:59.747 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:02:59.746+0000 7f6141db3700 1 -- 172.21.15.162:0/1516844570 wait complete. 2024-03-31T23:02:59.747 INFO:teuthology.orchestra.run.smithi162.stderr:dumped monmap epoch 2 2024-03-31T23:03:00.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:59 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-31T23:03:00.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:59 smithi162 ceph-mon[36237]: mon.a calling monitor election 2024-03-31T23:03:00.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:59 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-31T23:03:00.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:59 smithi162 ceph-mon[36237]: from='client.? 172.21.15.162:0/1516844570' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-31T23:03:00.047 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:59 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-31T23:03:00.047 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:59 smithi162 ceph-mon[36237]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-31T23:03:00.047 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:59 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-31T23:03:00.047 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:59 smithi162 ceph-mon[36237]: mon.c calling monitor election 2024-03-31T23:03:00.047 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:59 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-31T23:03:00.047 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:59 smithi162 ceph-mon[36237]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-31T23:03:00.047 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:59 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-31T23:03:00.047 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:59 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-31T23:03:00.047 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:59 smithi162 ceph-mon[36237]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-31T23:03:00.047 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:59 smithi162 ceph-mon[36237]: monmap epoch 2 2024-03-31T23:03:00.047 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:59 smithi162 ceph-mon[36237]: fsid 6a9d8f08-efb2-11ee-b647-cb9ed24678a4 2024-03-31T23:03:00.047 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:59 smithi162 ceph-mon[36237]: last_changed 2024-03-31T23:02:54.730244+0000 2024-03-31T23:03:00.047 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:59 smithi162 ceph-mon[36237]: created 2024-03-31T23:01:12.507350+0000 2024-03-31T23:03:00.047 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:59 smithi162 ceph-mon[36237]: min_mon_release 19 (squid) 2024-03-31T23:03:00.048 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:59 smithi162 ceph-mon[36237]: election_strategy: 1 2024-03-31T23:03:00.048 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:59 smithi162 ceph-mon[36237]: 0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-03-31T23:03:00.048 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:59 smithi162 ceph-mon[36237]: 1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.c 2024-03-31T23:03:00.048 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:59 smithi162 ceph-mon[36237]: fsmap 2024-03-31T23:03:00.048 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:59 smithi162 ceph-mon[36237]: osdmap e4: 0 total, 0 up, 0 in 2024-03-31T23:03:00.048 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:59 smithi162 ceph-mon[36237]: mgrmap e13: x(active, since 51s) 2024-03-31T23:03:00.048 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:59 smithi162 ceph-mon[36237]: overall HEALTH_OK 2024-03-31T23:03:00.048 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:59 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:03:00.048 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:59 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:03:00.048 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:59 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:03:00.048 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:59 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-31T23:03:00.048 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:02:59 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:03:00.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:02:59 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-31T23:03:00.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:02:59 smithi046 ceph-mon[32498]: mon.a calling monitor election 2024-03-31T23:03:00.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:02:59 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-31T23:03:00.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:02:59 smithi046 ceph-mon[32498]: from='client.? 172.21.15.162:0/1516844570' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-31T23:03:00.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:02:59 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-31T23:03:00.113 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:02:59 smithi046 ceph-mon[32498]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-31T23:03:00.113 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:02:59 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-31T23:03:00.113 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:02:59 smithi046 ceph-mon[32498]: mon.c calling monitor election 2024-03-31T23:03:00.113 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:02:59 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-31T23:03:00.113 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:02:59 smithi046 ceph-mon[32498]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-31T23:03:00.113 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:02:59 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-31T23:03:00.113 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:02:59 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-31T23:03:00.113 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:02:59 smithi046 ceph-mon[32498]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-31T23:03:00.113 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:02:59 smithi046 ceph-mon[32498]: monmap epoch 2 2024-03-31T23:03:00.113 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:02:59 smithi046 ceph-mon[32498]: fsid 6a9d8f08-efb2-11ee-b647-cb9ed24678a4 2024-03-31T23:03:00.113 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:02:59 smithi046 ceph-mon[32498]: last_changed 2024-03-31T23:02:54.730244+0000 2024-03-31T23:03:00.113 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:02:59 smithi046 ceph-mon[32498]: created 2024-03-31T23:01:12.507350+0000 2024-03-31T23:03:00.113 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:02:59 smithi046 ceph-mon[32498]: min_mon_release 19 (squid) 2024-03-31T23:03:00.114 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:02:59 smithi046 ceph-mon[32498]: election_strategy: 1 2024-03-31T23:03:00.114 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:02:59 smithi046 ceph-mon[32498]: 0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-03-31T23:03:00.114 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:02:59 smithi046 ceph-mon[32498]: 1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.c 2024-03-31T23:03:00.114 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:02:59 smithi046 ceph-mon[32498]: fsmap 2024-03-31T23:03:00.114 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:02:59 smithi046 ceph-mon[32498]: osdmap e4: 0 total, 0 up, 0 in 2024-03-31T23:03:00.114 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:02:59 smithi046 ceph-mon[32498]: mgrmap e13: x(active, since 51s) 2024-03-31T23:03:00.114 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:02:59 smithi046 ceph-mon[32498]: overall HEALTH_OK 2024-03-31T23:03:00.114 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:02:59 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:03:00.114 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:02:59 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:03:00.114 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:02:59 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:03:00.114 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:02:59 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-31T23:03:00.114 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:02:59 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:03:01.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:03:00 smithi046 ceph-mon[32498]: Deploying daemon mon.b on smithi142 2024-03-31T23:03:01.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:03:00 smithi046 ceph-mon[32498]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-31T23:03:01.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:03:00 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-31T23:03:01.113 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:03:00 smithi162 ceph-mon[36237]: Deploying daemon mon.b on smithi142 2024-03-31T23:03:01.113 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:03:00 smithi162 ceph-mon[36237]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-31T23:03:01.114 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:03:00 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-31T23:03:01.361 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-31T23:03:01.361 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:384f31f3d3aee7f8ba3f60846b8844c049165583 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6a9d8f08-efb2-11ee-b647-cb9ed24678a4 -- ceph mon dump -f json 2024-03-31T23:03:01.593 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/6a9d8f08-efb2-11ee-b647-cb9ed24678a4/mon.c/config 2024-03-31T23:03:02.111 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 31 23:03:01 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mgr-x[32727]: 2024-03-31T23:03:01.730+0000 7fbe660e3700 -1 mgr.server handle_report got status from non-daemon mon.c 2024-03-31T23:03:02.795 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:03:02 smithi162 ceph-mon[36237]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-31T23:03:02.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:03:02 smithi046 ceph-mon[32498]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-31T23:03:02.991 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:03:02.989+0000 7f0d06e19700 1 -- 172.21.15.162:0/2126761069 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0d00101f80 msgr2=0x7f0d00102360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:03:02.991 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:03:02.989+0000 7f0d06e19700 1 --2- 172.21.15.162:0/2126761069 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0d00101f80 0x7f0d00102360 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f0cf4009910 tx=0x7f0cf4009c20 comp rx=0 tx=0).stop 2024-03-31T23:03:02.992 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:03:02.989+0000 7f0d06e19700 1 -- 172.21.15.162:0/2126761069 shutdown_connections 2024-03-31T23:03:02.992 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:03:02.989+0000 7f0d06e19700 1 --2- 172.21.15.162:0/2126761069 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0d00101f80 0x7f0d00102360 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:03:02.992 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:03:02.989+0000 7f0d06e19700 1 -- 172.21.15.162:0/2126761069 >> 172.21.15.162:0/2126761069 conn(0x7f0d000fd810 msgr2=0x7f0d000ffc30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:03:02.992 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:03:02.989+0000 7f0d06e19700 1 -- 172.21.15.162:0/2126761069 shutdown_connections 2024-03-31T23:03:02.992 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:03:02.989+0000 7f0d06e19700 1 -- 172.21.15.162:0/2126761069 wait complete. 2024-03-31T23:03:02.992 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:03:02.990+0000 7f0d06e19700 1 Processor -- start 2024-03-31T23:03:02.992 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:03:02.990+0000 7f0d06e19700 1 -- start start 2024-03-31T23:03:02.992 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:03:02.990+0000 7f0d06e19700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0d00101f80 0x7f0d0019d610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:03:02.992 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:03:02.990+0000 7f0d06e19700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0d0019db50 0x7f0d001976e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:03:02.993 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:03:02.990+0000 7f0d06e19700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f0d0010c200 con 0x7f0d00101f80 2024-03-31T23:03:02.993 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:03:02.990+0000 7f0d06e19700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f0d0010c080 con 0x7f0d0019db50 2024-03-31T23:03:02.993 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:03:02.990+0000 7f0cfffff700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0d0019db50 0x7f0d001976e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:03:02.993 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:03:02.990+0000 7f0cfffff700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0d0019db50 0x7f0d001976e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:41822/0 (socket says 172.21.15.162:41822) 2024-03-31T23:03:02.993 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:03:02.990+0000 7f0d04bb5700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0d00101f80 0x7f0d0019d610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:03:02.993 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:03:02.990+0000 7f0cfffff700 1 -- 172.21.15.162:0/2662807490 learned_addr learned my addr 172.21.15.162:0/2662807490 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-03-31T23:03:02.993 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:03:02.990+0000 7f0cfffff700 1 -- 172.21.15.162:0/2662807490 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0d0019db50 msgr2=0x7f0d001976e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-03-31T23:03:02.993 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:03:02.990+0000 7f0cfffff700 1 -- 172.21.15.162:0/2662807490 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0d0019db50 msgr2=0x7f0d001976e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-31T23:03:02.993 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:03:02.990+0000 7f0cfffff700 1 --2- 172.21.15.162:0/2662807490 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0d0019db50 0x7f0d001976e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-31T23:03:02.993 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:03:02.990+0000 7f0cfffff700 1 --2- 172.21.15.162:0/2662807490 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0d0019db50 0x7f0d001976e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-31T23:03:02.993 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:03:02.990+0000 7f0d04bb5700 1 -- 172.21.15.162:0/2662807490 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0d0019db50 msgr2=0x7f0d001976e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-31T23:03:02.993 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:03:02.990+0000 7f0d04bb5700 1 --2- 172.21.15.162:0/2662807490 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0d0019db50 0x7f0d001976e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:03:02.994 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:03:02.990+0000 7f0d04bb5700 1 -- 172.21.15.162:0/2662807490 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d0019e0d0 con 0x7f0d00101f80 2024-03-31T23:03:02.994 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:03:02.991+0000 7f0d04bb5700 1 --2- 172.21.15.162:0/2662807490 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0d00101f80 0x7f0d0019d610 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f0cf4014000 tx=0x7f0cf4005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:03:02.994 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:03:02.991+0000 7f0cfdffb700 1 -- 172.21.15.162:0/2662807490 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7f0cf4005dc0 con 0x7f0d00101f80 2024-03-31T23:03:02.994 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:03:02.991+0000 7f0d06e19700 1 -- 172.21.15.162:0/2662807490 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0d00197c80 con 0x7f0d00101f80 2024-03-31T23:03:02.994 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:03:02.991+0000 7f0cfdffb700 1 -- 172.21.15.162:0/2662807490 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0cf400b750 con 0x7f0d00101f80 2024-03-31T23:03:02.994 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:03:02.991+0000 7f0cfdffb700 1 -- 172.21.15.162:0/2662807490 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7f0cf40039a0 con 0x7f0d00101f80 2024-03-31T23:03:02.994 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:03:02.991+0000 7f0d06e19700 1 -- 172.21.15.162:0/2662807490 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0d00198210 con 0x7f0d00101f80 2024-03-31T23:03:02.995 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:03:02.992+0000 7f0d06e19700 1 -- 172.21.15.162:0/2662807490 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0d00103460 con 0x7f0d00101f80 2024-03-31T23:03:02.996 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:03:02.992+0000 7f0cfdffb700 1 -- 172.21.15.162:0/2662807490 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 13) ==== 46172+0+0 (secure 0 0 0) 0x7f0cf4029070 con 0x7f0d00101f80 2024-03-31T23:03:02.996 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:03:02.992+0000 7f0cfdffb700 1 --2- 172.21.15.162:0/2662807490 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f0cf0038c40 0x7f0cf003b100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:03:02.996 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:03:02.992+0000 7f0cfdffb700 1 -- 172.21.15.162:0/2662807490 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f0cf401e070 con 0x7f0d00101f80 2024-03-31T23:03:02.996 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:03:02.993+0000 7f0cfffff700 1 --2- 172.21.15.162:0/2662807490 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f0cf0038c40 0x7f0cf003b100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:03:02.996 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:03:02.993+0000 7f0cfffff700 1 --2- 172.21.15.162:0/2662807490 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f0cf0038c40 0x7f0cf003b100 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f0cec009180 tx=0x7f0cec008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:03:02.998 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:03:02.998+0000 7f0cfdffb700 1 -- 172.21.15.162:0/2662807490 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f0cf4013070 con 0x7f0d00101f80 2024-03-31T23:03:03.266 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:03:03.265+0000 7f0d06e19700 1 -- 172.21.15.162:0/2662807490 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7f0d00198580 con 0x7f0d00101f80 2024-03-31T23:03:03.266 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:03:03.266+0000 7f0cfdffb700 1 -- 172.21.15.162:0/2662807490 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) ==== 95+0+1015 (secure 0 0 0) 0x7f0cf4030090 con 0x7f0d00101f80 2024-03-31T23:03:03.267 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-31T23:03:03.267 INFO:teuthology.orchestra.run.smithi162.stdout:{"epoch":2,"fsid":"6a9d8f08-efb2-11ee-b647-cb9ed24678a4","modified":"2024-03-31T23:02:54.730244Z","created":"2024-03-31T23:01:12.507350Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3300","nonce":0},{"type":"v1","addr":"172.21.15.46:6789","nonce":0}]},"addr":"172.21.15.46:6789/0","public_addr":"172.21.15.46:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:3300","nonce":0},{"type":"v1","addr":"172.21.15.162:6789","nonce":0}]},"addr":"172.21.15.162:6789/0","public_addr":"172.21.15.162:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-03-31T23:03:03.269 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:03:03.268+0000 7f0d06e19700 1 -- 172.21.15.162:0/2662807490 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f0cf0038c40 msgr2=0x7f0cf003b100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:03:03.269 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:03:03.268+0000 7f0d06e19700 1 --2- 172.21.15.162:0/2662807490 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f0cf0038c40 0x7f0cf003b100 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f0cec009180 tx=0x7f0cec008040 comp rx=0 tx=0).stop 2024-03-31T23:03:03.269 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:03:03.268+0000 7f0d06e19700 1 -- 172.21.15.162:0/2662807490 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0d00101f80 msgr2=0x7f0d0019d610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:03:03.269 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:03:03.268+0000 7f0d06e19700 1 --2- 172.21.15.162:0/2662807490 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0d00101f80 0x7f0d0019d610 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f0cf4014000 tx=0x7f0cf4005030 comp rx=0 tx=0).stop 2024-03-31T23:03:03.269 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:03:03.268+0000 7f0d06e19700 1 -- 172.21.15.162:0/2662807490 shutdown_connections 2024-03-31T23:03:03.269 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:03:03.268+0000 7f0d06e19700 1 --2- 172.21.15.162:0/2662807490 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f0cf0038c40 0x7f0cf003b100 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:03:03.269 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:03:03.268+0000 7f0d06e19700 1 --2- 172.21.15.162:0/2662807490 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0d00101f80 0x7f0d0019d610 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:03:03.269 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:03:03.268+0000 7f0d06e19700 1 --2- 172.21.15.162:0/2662807490 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0d0019db50 0x7f0d001976e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:03:03.270 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:03:03.268+0000 7f0d06e19700 1 -- 172.21.15.162:0/2662807490 >> 172.21.15.162:0/2662807490 conn(0x7f0d000fd810 msgr2=0x7f0d00103950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:03:03.270 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:03:03.269+0000 7f0d06e19700 1 -- 172.21.15.162:0/2662807490 shutdown_connections 2024-03-31T23:03:03.270 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:03:03.269+0000 7f0d06e19700 1 -- 172.21.15.162:0/2662807490 wait complete. 2024-03-31T23:03:03.270 INFO:teuthology.orchestra.run.smithi162.stderr:dumped monmap epoch 2 2024-03-31T23:03:03.468 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:03 smithi142 ceph-mon[35271]: mon.b@-1(synchronizing).mds e1 print_map 2024-03-31T23:03:03.468 INFO:journalctl@ceph.mon.b.smithi142.stdout: e1 2024-03-31T23:03:03.468 INFO:journalctl@ceph.mon.b.smithi142.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-31T23:03:03.468 INFO:journalctl@ceph.mon.b.smithi142.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-03-31T23:03:03.469 INFO:journalctl@ceph.mon.b.smithi142.stdout: legacy client fscid: -1 2024-03-31T23:03:03.469 INFO:journalctl@ceph.mon.b.smithi142.stdout: 2024-03-31T23:03:03.469 INFO:journalctl@ceph.mon.b.smithi142.stdout: No filesystems configured 2024-03-31T23:03:03.469 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:03 smithi142 ceph-mon[35271]: mon.b@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-03-31T23:03:03.469 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:03 smithi142 ceph-mon[35271]: mon.b@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-03-31T23:03:03.469 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:03 smithi142 ceph-mon[35271]: mon.b@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-03-31T23:03:03.469 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:03 smithi142 ceph-mon[35271]: mon.b@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-03-31T23:03:03.469 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:03 smithi142 ceph-mon[35271]: mon.b@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-03-31T23:03:03.469 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:03 smithi142 ceph-mon[35271]: mon.b@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-03-31T23:03:03.944 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:03 smithi142 ceph-mon[35271]: mon.b@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-03-31T23:03:03.944 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:03 smithi142 ceph-mon[35271]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-31T23:03:03.944 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:03 smithi142 ceph-mon[35271]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-31T23:03:03.944 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:03 smithi142 ceph-mon[35271]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-31T23:03:03.944 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:03 smithi142 ceph-mon[35271]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-31T23:03:03.944 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:03 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:03:03.944 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:03 smithi142 ceph-mon[35271]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-31T23:03:03.944 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:03 smithi142 ceph-mon[35271]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-31T23:03:03.945 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:03 smithi142 ceph-mon[35271]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi162", "target": ["mon-mgr", ""]}]: dispatch 2024-03-31T23:03:03.945 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:03 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:03:03.945 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:03 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:03:03.945 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:03 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:03:03.945 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:03 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:03:03.945 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:03 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi142", "name": "osd_memory_target"}]: dispatch 2024-03-31T23:03:03.945 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:03 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:03:03.945 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:03 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-31T23:03:03.945 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:03 smithi142 ceph-mon[35271]: Updating smithi142:/etc/ceph/ceph.conf 2024-03-31T23:03:03.945 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:03 smithi142 ceph-mon[35271]: Updating smithi142:/var/lib/ceph/6a9d8f08-efb2-11ee-b647-cb9ed24678a4/config/ceph.conf 2024-03-31T23:03:03.945 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:03 smithi142 ceph-mon[35271]: Deploying cephadm binary to smithi162 2024-03-31T23:03:03.945 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:03 smithi142 ceph-mon[35271]: Updating smithi142:/etc/ceph/ceph.client.admin.keyring 2024-03-31T23:03:03.945 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:03 smithi142 ceph-mon[35271]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-31T23:03:03.946 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:03 smithi142 ceph-mon[35271]: Updating smithi142:/var/lib/ceph/6a9d8f08-efb2-11ee-b647-cb9ed24678a4/config/ceph.client.admin.keyring 2024-03-31T23:03:03.946 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:03 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:03:03.946 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:03 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:03:03.946 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:03 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:03:03.946 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:03 smithi142 ceph-mon[35271]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-31T23:03:03.946 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:03 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:03:03.946 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:03 smithi142 ceph-mon[35271]: Added host smithi162 2024-03-31T23:03:03.946 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:03 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-31T23:03:03.946 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:03 smithi142 ceph-mon[35271]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-31T23:03:03.946 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:03 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:03:03.946 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:03 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:03:03.946 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:03 smithi142 ceph-mon[35271]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-31T23:03:03.946 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:03 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:03:03.946 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:03 smithi142 ceph-mon[35271]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-31T23:03:03.947 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:03 smithi142 ceph-mon[35271]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-31T23:03:03.947 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:03 smithi142 ceph-mon[35271]: from='client.? 172.21.15.46:0/3677361043' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-03-31T23:03:03.947 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:03 smithi142 ceph-mon[35271]: from='client.? 172.21.15.46:0/3677361043' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-03-31T23:03:03.947 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:03 smithi142 ceph-mon[35271]: osdmap e4: 0 total, 0 up, 0 in 2024-03-31T23:03:03.947 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:03 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:03:03.947 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:03 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:03:03.947 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:03 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:03:03.947 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:03 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:03:03.947 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:03 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi162", "name": "osd_memory_target"}]: dispatch 2024-03-31T23:03:03.947 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:03 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:03:03.947 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:03 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-31T23:03:03.947 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:03 smithi142 ceph-mon[35271]: Updating smithi162:/etc/ceph/ceph.conf 2024-03-31T23:03:03.947 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:03 smithi142 ceph-mon[35271]: Updating smithi162:/var/lib/ceph/6a9d8f08-efb2-11ee-b647-cb9ed24678a4/config/ceph.conf 2024-03-31T23:03:03.947 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:03 smithi142 ceph-mon[35271]: Updating smithi162:/etc/ceph/ceph.client.admin.keyring 2024-03-31T23:03:03.947 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:03 smithi142 ceph-mon[35271]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-31T23:03:03.948 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:03 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:03:03.948 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:03 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:03:03.948 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:03 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:03:03.948 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:03 smithi142 ceph-mon[35271]: Updating smithi162:/var/lib/ceph/6a9d8f08-efb2-11ee-b647-cb9ed24678a4/config/ceph.client.admin.keyring 2024-03-31T23:03:03.948 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:03 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:03:03.948 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:03 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-31T23:03:03.948 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:03 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:03:03.948 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:03 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-31T23:03:03.948 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:03 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:03:03.948 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:03 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-31T23:03:03.948 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:03 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:03:03.948 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:03 smithi142 ceph-mon[35271]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi046:172.21.15.46=a;smithi142:172.21.15.142=b;smithi162:172.21.15.162=c", "target": ["mon-mgr", ""]}]: dispatch 2024-03-31T23:03:03.948 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:03 smithi142 ceph-mon[35271]: Saving service mon spec with placement smithi046:172.21.15.46=a;smithi142:172.21.15.142=b;smithi162:172.21.15.162=c;count:3 2024-03-31T23:03:03.948 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:03 smithi142 ceph-mon[35271]: Deploying daemon mon.c on smithi162 2024-03-31T23:03:03.948 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:03 smithi142 ceph-mon[35271]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-31T23:03:03.949 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:03 smithi142 ceph-mon[35271]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-31T23:03:03.949 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:03 smithi142 ceph-mon[35271]: from='client.? 172.21.15.162:0/3618932929' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-31T23:03:03.949 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:03 smithi142 ceph-mon[35271]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-31T23:03:03.949 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:03 smithi142 ceph-mon[35271]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-31T23:03:03.949 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:03 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-31T23:03:03.949 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:03 smithi142 ceph-mon[35271]: mon.a calling monitor election 2024-03-31T23:03:03.949 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:03 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-31T23:03:03.950 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:03 smithi142 ceph-mon[35271]: from='client.? 172.21.15.162:0/1516844570' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-31T23:03:03.950 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:03 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-31T23:03:03.950 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:03 smithi142 ceph-mon[35271]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-31T23:03:03.950 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:03 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-31T23:03:03.950 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:03 smithi142 ceph-mon[35271]: mon.c calling monitor election 2024-03-31T23:03:03.950 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:03 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-31T23:03:03.950 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:03 smithi142 ceph-mon[35271]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-31T23:03:03.950 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:03 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-31T23:03:03.950 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:03 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-31T23:03:03.950 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:03 smithi142 ceph-mon[35271]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-31T23:03:03.950 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:03 smithi142 ceph-mon[35271]: monmap epoch 2 2024-03-31T23:03:03.950 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:03 smithi142 ceph-mon[35271]: fsid 6a9d8f08-efb2-11ee-b647-cb9ed24678a4 2024-03-31T23:03:03.950 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:03 smithi142 ceph-mon[35271]: last_changed 2024-03-31T23:02:54.730244+0000 2024-03-31T23:03:03.950 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:03 smithi142 ceph-mon[35271]: created 2024-03-31T23:01:12.507350+0000 2024-03-31T23:03:03.950 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:03 smithi142 ceph-mon[35271]: min_mon_release 19 (squid) 2024-03-31T23:03:03.951 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:03 smithi142 ceph-mon[35271]: election_strategy: 1 2024-03-31T23:03:03.951 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:03 smithi142 ceph-mon[35271]: 0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-03-31T23:03:03.951 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:03 smithi142 ceph-mon[35271]: 1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.c 2024-03-31T23:03:03.951 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:03 smithi142 ceph-mon[35271]: fsmap 2024-03-31T23:03:03.951 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:03 smithi142 ceph-mon[35271]: osdmap e4: 0 total, 0 up, 0 in 2024-03-31T23:03:03.951 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:03 smithi142 ceph-mon[35271]: mgrmap e13: x(active, since 51s) 2024-03-31T23:03:03.951 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:03 smithi142 ceph-mon[35271]: overall HEALTH_OK 2024-03-31T23:03:03.951 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:03 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:03:03.951 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:03 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:03:03.951 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:03 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:03:03.951 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:03 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-31T23:03:03.951 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:03 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:03:03.951 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:03 smithi142 ceph-mon[35271]: Deploying daemon mon.b on smithi142 2024-03-31T23:03:03.951 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:03 smithi142 ceph-mon[35271]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-31T23:03:03.952 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:03 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-31T23:03:03.952 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:03 smithi142 ceph-mon[35271]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-31T23:03:03.952 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:03 smithi142 ceph-mon[35271]: mon.b@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-03-31T23:03:03.952 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:03 smithi142 bash[35252]: 5a36e49c19c9dad01033713a1e9f1692bf3201d01eedb8c3144be00d5aa0b724 2024-03-31T23:03:03.952 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:03 smithi142 systemd[1]: Started Ceph mon.b for 6a9d8f08-efb2-11ee-b647-cb9ed24678a4. 2024-03-31T23:03:04.828 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-31T23:03:04.829 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:384f31f3d3aee7f8ba3f60846b8844c049165583 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6a9d8f08-efb2-11ee-b647-cb9ed24678a4 -- ceph mon dump -f json 2024-03-31T23:03:05.073 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/6a9d8f08-efb2-11ee-b647-cb9ed24678a4/mon.c/config 2024-03-31T23:03:08.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:03:08 smithi162 ceph-mon[36237]: mon.c calling monitor election 2024-03-31T23:03:08.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:03:08 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-31T23:03:08.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:03:08 smithi162 ceph-mon[36237]: mon.a calling monitor election 2024-03-31T23:03:08.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:03:08 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-31T23:03:08.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:03:08 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-31T23:03:08.797 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:03:08 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-31T23:03:08.797 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:03:08 smithi162 ceph-mon[36237]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-31T23:03:08.797 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:03:08 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-31T23:03:08.797 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:03:08 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-31T23:03:08.797 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:03:08 smithi162 ceph-mon[36237]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-31T23:03:08.797 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:03:08 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-31T23:03:08.797 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:03:08 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-31T23:03:08.797 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:03:08 smithi162 ceph-mon[36237]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-31T23:03:08.797 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:03:08 smithi162 ceph-mon[36237]: monmap epoch 3 2024-03-31T23:03:08.797 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:03:08 smithi162 ceph-mon[36237]: fsid 6a9d8f08-efb2-11ee-b647-cb9ed24678a4 2024-03-31T23:03:08.797 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:03:08 smithi162 ceph-mon[36237]: last_changed 2024-03-31T23:03:03.473853+0000 2024-03-31T23:03:08.797 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:03:08 smithi162 ceph-mon[36237]: created 2024-03-31T23:01:12.507350+0000 2024-03-31T23:03:08.798 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:03:08 smithi162 ceph-mon[36237]: min_mon_release 19 (squid) 2024-03-31T23:03:08.798 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:03:08 smithi162 ceph-mon[36237]: election_strategy: 1 2024-03-31T23:03:08.798 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:03:08 smithi162 ceph-mon[36237]: 0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-03-31T23:03:08.798 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:03:08 smithi162 ceph-mon[36237]: 1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.c 2024-03-31T23:03:08.798 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:03:08 smithi162 ceph-mon[36237]: 2: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.b 2024-03-31T23:03:08.798 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:03:08 smithi162 ceph-mon[36237]: fsmap 2024-03-31T23:03:08.798 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:03:08 smithi162 ceph-mon[36237]: osdmap e4: 0 total, 0 up, 0 in 2024-03-31T23:03:08.798 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:03:08 smithi162 ceph-mon[36237]: mgrmap e13: x(active, since 59s) 2024-03-31T23:03:08.798 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:03:08 smithi162 ceph-mon[36237]: overall HEALTH_OK 2024-03-31T23:03:08.798 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:03:08 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:03:08.798 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:03:08 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:03:08.798 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:03:08 smithi162 ceph-mon[36237]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-31T23:03:08.798 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:03:08 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:03:08.798 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:03:08 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:03:08.862 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:03:08 smithi046 ceph-mon[32498]: mon.c calling monitor election 2024-03-31T23:03:08.862 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:03:08 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-31T23:03:08.862 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:03:08 smithi046 ceph-mon[32498]: mon.a calling monitor election 2024-03-31T23:03:08.862 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:03:08 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-31T23:03:08.862 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:03:08 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-31T23:03:08.863 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:03:08 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-31T23:03:08.863 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:03:08 smithi046 ceph-mon[32498]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-31T23:03:08.863 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:03:08 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-31T23:03:08.863 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:03:08 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-31T23:03:08.863 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:03:08 smithi046 ceph-mon[32498]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-31T23:03:08.863 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:03:08 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-31T23:03:08.863 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:03:08 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-31T23:03:08.863 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:03:08 smithi046 ceph-mon[32498]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-31T23:03:08.863 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:03:08 smithi046 ceph-mon[32498]: monmap epoch 3 2024-03-31T23:03:08.863 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:03:08 smithi046 ceph-mon[32498]: fsid 6a9d8f08-efb2-11ee-b647-cb9ed24678a4 2024-03-31T23:03:08.863 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:03:08 smithi046 ceph-mon[32498]: last_changed 2024-03-31T23:03:03.473853+0000 2024-03-31T23:03:08.863 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:03:08 smithi046 ceph-mon[32498]: created 2024-03-31T23:01:12.507350+0000 2024-03-31T23:03:08.863 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:03:08 smithi046 ceph-mon[32498]: min_mon_release 19 (squid) 2024-03-31T23:03:08.863 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:03:08 smithi046 ceph-mon[32498]: election_strategy: 1 2024-03-31T23:03:08.864 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:03:08 smithi046 ceph-mon[32498]: 0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-03-31T23:03:08.864 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:03:08 smithi046 ceph-mon[32498]: 1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.c 2024-03-31T23:03:08.864 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:03:08 smithi046 ceph-mon[32498]: 2: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.b 2024-03-31T23:03:08.864 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:03:08 smithi046 ceph-mon[32498]: fsmap 2024-03-31T23:03:08.864 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:03:08 smithi046 ceph-mon[32498]: osdmap e4: 0 total, 0 up, 0 in 2024-03-31T23:03:08.864 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:03:08 smithi046 ceph-mon[32498]: mgrmap e13: x(active, since 59s) 2024-03-31T23:03:08.864 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:03:08 smithi046 ceph-mon[32498]: overall HEALTH_OK 2024-03-31T23:03:08.864 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:03:08 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:03:08.864 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:03:08 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:03:08.864 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:03:08 smithi046 ceph-mon[32498]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-31T23:03:08.864 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:03:08 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:03:08.864 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:03:08 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:03:09.332 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:03:09.328+0000 7ff3f45d2700 1 -- 172.21.15.162:0/120275194 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff3d80035c0 msgr2=0x7ff3d8005a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:03:09.332 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:03:09.328+0000 7ff3f45d2700 1 --2- 172.21.15.162:0/120275194 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff3d80035c0 0x7ff3d8005a50 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7ff3ec101c50 tx=0x7ff3e800feb0 comp rx=0 tx=0).stop 2024-03-31T23:03:09.332 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:03:09.328+0000 7ff3f45d2700 1 -- 172.21.15.162:0/120275194 shutdown_connections 2024-03-31T23:03:09.332 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:03:09.328+0000 7ff3f45d2700 1 --2- 172.21.15.162:0/120275194 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff3d80035c0 0x7ff3d8005a50 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:03:09.332 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:03:09.328+0000 7ff3f45d2700 1 -- 172.21.15.162:0/120275194 >> 172.21.15.162:0/120275194 conn(0x7ff3ec0fd810 msgr2=0x7ff3ec0ffc30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:03:09.332 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:03:09.328+0000 7ff3f45d2700 1 -- 172.21.15.162:0/120275194 shutdown_connections 2024-03-31T23:03:09.332 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:03:09.328+0000 7ff3f45d2700 1 -- 172.21.15.162:0/120275194 wait complete. 2024-03-31T23:03:09.332 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:03:09.329+0000 7ff3f45d2700 1 Processor -- start 2024-03-31T23:03:09.332 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:03:09.329+0000 7ff3f45d2700 1 -- start start 2024-03-31T23:03:09.332 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:03:09.329+0000 7ff3f45d2700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff3d80035c0 0x7ff3ec074540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:03:09.332 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:03:09.329+0000 7ff3f45d2700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff3ec074a80 0x7ff3ec074f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:03:09.333 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:03:09.329+0000 7ff3f45d2700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff3ec1a3b40 0x7ff3ec1a5f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:03:09.333 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:03:09.329+0000 7ff3f45d2700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7ff3ec109c60 con 0x7ff3ec1a3b40 2024-03-31T23:03:09.333 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:03:09.329+0000 7ff3f45d2700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7ff3ec109ae0 con 0x7ff3ec074a80 2024-03-31T23:03:09.333 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:03:09.329+0000 7ff3f45d2700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7ff3ec109de0 con 0x7ff3d80035c0 2024-03-31T23:03:09.333 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:03:09.329+0000 7ff3f236e700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff3d80035c0 0x7ff3ec074540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:03:09.333 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:03:09.329+0000 7ff3f236e700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff3d80035c0 0x7ff3ec074540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:33970/0 (socket says 172.21.15.162:33970) 2024-03-31T23:03:09.333 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:03:09.329+0000 7ff3f236e700 1 -- 172.21.15.162:0/436431993 learned_addr learned my addr 172.21.15.162:0/436431993 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-03-31T23:03:09.333 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:03:09.329+0000 7ff3f2b6f700 1 --2- 172.21.15.162:0/436431993 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff3ec1a3b40 0x7ff3ec1a5f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:03:09.333 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:03:09.329+0000 7ff3f236e700 1 -- 172.21.15.162:0/436431993 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff3d80035c0 msgr2=0x7ff3ec074540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-03-31T23:03:09.333 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:03:09.329+0000 7ff3f236e700 1 -- 172.21.15.162:0/436431993 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff3d80035c0 msgr2=0x7ff3ec074540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-31T23:03:09.333 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:03:09.329+0000 7ff3f236e700 1 --2- 172.21.15.162:0/436431993 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff3d80035c0 0x7ff3ec074540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-31T23:03:09.333 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:03:09.329+0000 7ff3f236e700 1 --2- 172.21.15.162:0/436431993 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff3d80035c0 0x7ff3ec074540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-31T23:03:09.333 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:03:09.329+0000 7ff3f1b6d700 1 --2- 172.21.15.162:0/436431993 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff3ec074a80 0x7ff3ec074f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:03:09.333 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:03:09.329+0000 7ff3f2b6f700 1 -- 172.21.15.162:0/436431993 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff3d80035c0 msgr2=0x7ff3ec074540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-31T23:03:09.333 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:03:09.329+0000 7ff3f2b6f700 1 --2- 172.21.15.162:0/436431993 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff3d80035c0 0x7ff3ec074540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:03:09.334 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:03:09.329+0000 7ff3f2b6f700 1 -- 172.21.15.162:0/436431993 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff3ec074a80 msgr2=0x7ff3ec074f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:03:09.334 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:03:09.329+0000 7ff3f2b6f700 1 --2- 172.21.15.162:0/436431993 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff3ec074a80 0x7ff3ec074f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:03:09.334 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:03:09.329+0000 7ff3f2b6f700 1 -- 172.21.15.162:0/436431993 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff3ec1a65d0 con 0x7ff3ec1a3b40 2024-03-31T23:03:09.334 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:03:09.330+0000 7ff3f2b6f700 1 --2- 172.21.15.162:0/436431993 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff3ec1a3b40 0x7ff3ec1a5f30 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7ff3e400bd50 tx=0x7ff3e400be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:03:09.334 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:03:09.330+0000 7ff3e37fe700 1 -- 172.21.15.162:0/436431993 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3e4019070 con 0x7ff3ec1a3b40 2024-03-31T23:03:09.334 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:03:09.330+0000 7ff3f45d2700 1 -- 172.21.15.162:0/436431993 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff3ec1a68c0 con 0x7ff3ec1a3b40 2024-03-31T23:03:09.334 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:03:09.330+0000 7ff3e37fe700 1 -- 172.21.15.162:0/436431993 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff3e400cd10 con 0x7ff3ec1a3b40 2024-03-31T23:03:09.334 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:03:09.330+0000 7ff3e37fe700 1 -- 172.21.15.162:0/436431993 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3e4012700 con 0x7ff3ec1a3b40 2024-03-31T23:03:09.334 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:03:09.330+0000 7ff3f45d2700 1 -- 172.21.15.162:0/436431993 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff3ec1a6e00 con 0x7ff3ec1a3b40 2024-03-31T23:03:09.334 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:03:09.331+0000 7ff3e37fe700 1 -- 172.21.15.162:0/436431993 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 13) ==== 46172+0+0 (secure 0 0 0) 0x7ff3e401c070 con 0x7ff3ec1a3b40 2024-03-31T23:03:09.334 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:03:09.331+0000 7ff3f45d2700 1 -- 172.21.15.162:0/436431993 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff3ec063870 con 0x7ff3ec1a3b40 2024-03-31T23:03:09.334 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:03:09.331+0000 7ff3e37fe700 1 --2- 172.21.15.162:0/436431993 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7ff3d0039330 0x7ff3d003b7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:03:09.335 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:03:09.331+0000 7ff3e37fe700 1 -- 172.21.15.162:0/436431993 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7ff3e404b860 con 0x7ff3ec1a3b40 2024-03-31T23:03:09.335 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:03:09.331+0000 7ff3f236e700 1 --2- 172.21.15.162:0/436431993 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7ff3d0039330 0x7ff3d003b7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:03:09.335 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:03:09.332+0000 7ff3f236e700 1 --2- 172.21.15.162:0/436431993 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7ff3d0039330 0x7ff3d003b7f0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7ff3e800f470 tx=0x7ff3e800f3c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:03:09.336 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:03:09.336+0000 7ff3e37fe700 1 -- 172.21.15.162:0/436431993 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7ff3e4007290 con 0x7ff3ec1a3b40 2024-03-31T23:03:09.615 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:03:09 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:03:09.616 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:03:09 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-31T23:03:09.616 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:03:09 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-31T23:03:09.626 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:03:09.625+0000 7ff3f45d2700 1 -- 172.21.15.162:0/436431993 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7ff3ec02db30 con 0x7ff3ec1a3b40 2024-03-31T23:03:09.626 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:03:09.626+0000 7ff3e37fe700 1 -- 172.21.15.162:0/436431993 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 3 v3) ==== 95+0+1283 (secure 0 0 0) 0x7ff3e400e090 con 0x7ff3ec1a3b40 2024-03-31T23:03:09.627 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-31T23:03:09.627 INFO:teuthology.orchestra.run.smithi162.stdout:{"epoch":3,"fsid":"6a9d8f08-efb2-11ee-b647-cb9ed24678a4","modified":"2024-03-31T23:03:03.473853Z","created":"2024-03-31T23:01:12.507350Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3300","nonce":0},{"type":"v1","addr":"172.21.15.46:6789","nonce":0}]},"addr":"172.21.15.46:6789/0","public_addr":"172.21.15.46:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:3300","nonce":0},{"type":"v1","addr":"172.21.15.162:6789","nonce":0}]},"addr":"172.21.15.162:6789/0","public_addr":"172.21.15.162:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3300","nonce":0},{"type":"v1","addr":"172.21.15.142:6789","nonce":0}]},"addr":"172.21.15.142:6789/0","public_addr":"172.21.15.142:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-03-31T23:03:09.629 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:03:09.628+0000 7ff3f45d2700 1 -- 172.21.15.162:0/436431993 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7ff3d0039330 msgr2=0x7ff3d003b7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:03:09.629 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:03:09.628+0000 7ff3f45d2700 1 --2- 172.21.15.162:0/436431993 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7ff3d0039330 0x7ff3d003b7f0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7ff3e800f470 tx=0x7ff3e800f3c0 comp rx=0 tx=0).stop 2024-03-31T23:03:09.629 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:03:09.628+0000 7ff3f45d2700 1 -- 172.21.15.162:0/436431993 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff3ec1a3b40 msgr2=0x7ff3ec1a5f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:03:09.629 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:03:09.628+0000 7ff3f45d2700 1 --2- 172.21.15.162:0/436431993 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff3ec1a3b40 0x7ff3ec1a5f30 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7ff3e400bd50 tx=0x7ff3e400be30 comp rx=0 tx=0).stop 2024-03-31T23:03:09.629 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:03:09.628+0000 7ff3f45d2700 1 -- 172.21.15.162:0/436431993 shutdown_connections 2024-03-31T23:03:09.629 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:03:09.628+0000 7ff3f45d2700 1 --2- 172.21.15.162:0/436431993 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7ff3d0039330 0x7ff3d003b7f0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:03:09.629 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:03:09.628+0000 7ff3f45d2700 1 --2- 172.21.15.162:0/436431993 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff3ec1a3b40 0x7ff3ec1a5f30 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:03:09.630 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:03:09.628+0000 7ff3f45d2700 1 --2- 172.21.15.162:0/436431993 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff3d80035c0 0x7ff3ec074540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:03:09.630 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:03:09.628+0000 7ff3f45d2700 1 --2- 172.21.15.162:0/436431993 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff3ec074a80 0x7ff3ec074f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:03:09.630 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:03:09.628+0000 7ff3f45d2700 1 -- 172.21.15.162:0/436431993 >> 172.21.15.162:0/436431993 conn(0x7ff3ec0fd810 msgr2=0x7ff3ec0feaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:03:09.630 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:03:09.628+0000 7ff3f45d2700 1 -- 172.21.15.162:0/436431993 shutdown_connections 2024-03-31T23:03:09.630 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:03:09.628+0000 7ff3f45d2700 1 -- 172.21.15.162:0/436431993 wait complete. 2024-03-31T23:03:09.630 INFO:teuthology.orchestra.run.smithi162.stderr:dumped monmap epoch 3 2024-03-31T23:03:09.862 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:03:09 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:03:09.862 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:03:09 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-31T23:03:09.862 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:03:09 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-31T23:03:10.645 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-03-31T23:03:10.646 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:384f31f3d3aee7f8ba3f60846b8844c049165583 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6a9d8f08-efb2-11ee-b647-cb9ed24678a4 -- ceph config generate-minimal-conf 2024-03-31T23:03:10.897 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/6a9d8f08-efb2-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-31T23:03:10.944 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:10 smithi142 ceph-mon[35271]: mon.c calling monitor election 2024-03-31T23:03:10.944 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:10 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-31T23:03:10.944 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:10 smithi142 ceph-mon[35271]: mon.a calling monitor election 2024-03-31T23:03:10.944 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:10 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-31T23:03:10.944 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:10 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-31T23:03:10.944 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:10 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-31T23:03:10.944 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:10 smithi142 ceph-mon[35271]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-31T23:03:10.944 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:10 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-31T23:03:10.945 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:10 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-31T23:03:10.945 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:10 smithi142 ceph-mon[35271]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-31T23:03:10.945 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:10 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-31T23:03:10.945 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:10 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-31T23:03:10.945 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:10 smithi142 ceph-mon[35271]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-31T23:03:10.945 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:10 smithi142 ceph-mon[35271]: monmap epoch 3 2024-03-31T23:03:10.945 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:10 smithi142 ceph-mon[35271]: fsid 6a9d8f08-efb2-11ee-b647-cb9ed24678a4 2024-03-31T23:03:10.945 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:10 smithi142 ceph-mon[35271]: last_changed 2024-03-31T23:03:03.473853+0000 2024-03-31T23:03:10.945 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:10 smithi142 ceph-mon[35271]: created 2024-03-31T23:01:12.507350+0000 2024-03-31T23:03:10.945 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:10 smithi142 ceph-mon[35271]: min_mon_release 19 (squid) 2024-03-31T23:03:10.945 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:10 smithi142 ceph-mon[35271]: election_strategy: 1 2024-03-31T23:03:10.945 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:10 smithi142 ceph-mon[35271]: 0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-03-31T23:03:10.945 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:10 smithi142 ceph-mon[35271]: 1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.c 2024-03-31T23:03:10.945 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:10 smithi142 ceph-mon[35271]: 2: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.b 2024-03-31T23:03:10.946 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:10 smithi142 ceph-mon[35271]: fsmap 2024-03-31T23:03:10.946 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:10 smithi142 ceph-mon[35271]: osdmap e4: 0 total, 0 up, 0 in 2024-03-31T23:03:10.946 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:10 smithi142 ceph-mon[35271]: mgrmap e13: x(active, since 59s) 2024-03-31T23:03:10.946 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:10 smithi142 ceph-mon[35271]: overall HEALTH_OK 2024-03-31T23:03:10.946 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:10 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:03:10.946 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:10 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:03:10.946 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:10 smithi142 ceph-mon[35271]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-31T23:03:10.946 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:10 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:03:10.947 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:10 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:03:10.947 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:10 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:03:10.947 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:10 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-31T23:03:10.947 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:10 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-31T23:03:11.296 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:03:10 smithi162 ceph-mon[36237]: mon.b calling monitor election 2024-03-31T23:03:11.296 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:03:10 smithi162 ceph-mon[36237]: Updating smithi046:/etc/ceph/ceph.conf 2024-03-31T23:03:11.296 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:03:10 smithi162 ceph-mon[36237]: Updating smithi142:/etc/ceph/ceph.conf 2024-03-31T23:03:11.296 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:03:10 smithi162 ceph-mon[36237]: Updating smithi162:/etc/ceph/ceph.conf 2024-03-31T23:03:11.296 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:03:10 smithi162 ceph-mon[36237]: Updating smithi142:/var/lib/ceph/6a9d8f08-efb2-11ee-b647-cb9ed24678a4/config/ceph.conf 2024-03-31T23:03:11.296 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:03:10 smithi162 ceph-mon[36237]: Updating smithi046:/var/lib/ceph/6a9d8f08-efb2-11ee-b647-cb9ed24678a4/config/ceph.conf 2024-03-31T23:03:11.296 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:03:10 smithi162 ceph-mon[36237]: Updating smithi162:/var/lib/ceph/6a9d8f08-efb2-11ee-b647-cb9ed24678a4/config/ceph.conf 2024-03-31T23:03:11.296 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:03:10 smithi162 ceph-mon[36237]: mon.b calling monitor election 2024-03-31T23:03:11.297 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:03:10 smithi162 ceph-mon[36237]: mon.c calling monitor election 2024-03-31T23:03:11.297 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:03:10 smithi162 ceph-mon[36237]: mon.a calling monitor election 2024-03-31T23:03:11.297 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:03:10 smithi162 ceph-mon[36237]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-31T23:03:11.297 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:03:10 smithi162 ceph-mon[36237]: monmap epoch 3 2024-03-31T23:03:11.297 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:03:10 smithi162 ceph-mon[36237]: fsid 6a9d8f08-efb2-11ee-b647-cb9ed24678a4 2024-03-31T23:03:11.297 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:03:10 smithi162 ceph-mon[36237]: last_changed 2024-03-31T23:03:03.473853+0000 2024-03-31T23:03:11.297 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:03:10 smithi162 ceph-mon[36237]: created 2024-03-31T23:01:12.507350+0000 2024-03-31T23:03:11.297 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:03:10 smithi162 ceph-mon[36237]: min_mon_release 19 (squid) 2024-03-31T23:03:11.297 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:03:10 smithi162 ceph-mon[36237]: election_strategy: 1 2024-03-31T23:03:11.297 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:03:10 smithi162 ceph-mon[36237]: 0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-03-31T23:03:11.297 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:03:10 smithi162 ceph-mon[36237]: 1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.c 2024-03-31T23:03:11.297 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:03:10 smithi162 ceph-mon[36237]: 2: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.b 2024-03-31T23:03:11.297 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:03:10 smithi162 ceph-mon[36237]: fsmap 2024-03-31T23:03:11.297 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:03:10 smithi162 ceph-mon[36237]: osdmap e4: 0 total, 0 up, 0 in 2024-03-31T23:03:11.298 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:03:10 smithi162 ceph-mon[36237]: mgrmap e13: x(active, since 61s) 2024-03-31T23:03:11.298 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:03:10 smithi162 ceph-mon[36237]: overall HEALTH_OK 2024-03-31T23:03:11.298 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:03:10 smithi162 ceph-mon[36237]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-31T23:03:11.298 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:03:10 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:03:11.298 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:03:10 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:03:11.298 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:03:10 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:03:11.298 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:03:10 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:03:11.298 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:03:10 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:03:11.298 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:03:10 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:03:11.298 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:03:10 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:03:11.298 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:03:10 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:03:11.298 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:03:10 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:03:11.298 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:03:10 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:03:11.299 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:03:10 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:03:11.299 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:03:10 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-31T23:03:11.299 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:03:10 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-31T23:03:11.299 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:03:10 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:03:11.305 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:03:10 smithi046 ceph-mon[32498]: mon.b calling monitor election 2024-03-31T23:03:11.306 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:03:10 smithi046 ceph-mon[32498]: Updating smithi046:/etc/ceph/ceph.conf 2024-03-31T23:03:11.306 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:03:10 smithi046 ceph-mon[32498]: Updating smithi142:/etc/ceph/ceph.conf 2024-03-31T23:03:11.306 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:03:10 smithi046 ceph-mon[32498]: Updating smithi162:/etc/ceph/ceph.conf 2024-03-31T23:03:11.306 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:03:10 smithi046 ceph-mon[32498]: Updating smithi142:/var/lib/ceph/6a9d8f08-efb2-11ee-b647-cb9ed24678a4/config/ceph.conf 2024-03-31T23:03:11.306 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:03:10 smithi046 ceph-mon[32498]: Updating smithi046:/var/lib/ceph/6a9d8f08-efb2-11ee-b647-cb9ed24678a4/config/ceph.conf 2024-03-31T23:03:11.306 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:03:10 smithi046 ceph-mon[32498]: Updating smithi162:/var/lib/ceph/6a9d8f08-efb2-11ee-b647-cb9ed24678a4/config/ceph.conf 2024-03-31T23:03:11.306 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:03:10 smithi046 ceph-mon[32498]: mon.b calling monitor election 2024-03-31T23:03:11.306 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:03:10 smithi046 ceph-mon[32498]: mon.c calling monitor election 2024-03-31T23:03:11.306 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:03:10 smithi046 ceph-mon[32498]: mon.a calling monitor election 2024-03-31T23:03:11.306 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:03:10 smithi046 ceph-mon[32498]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-31T23:03:11.306 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:03:10 smithi046 ceph-mon[32498]: monmap epoch 3 2024-03-31T23:03:11.306 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:03:10 smithi046 ceph-mon[32498]: fsid 6a9d8f08-efb2-11ee-b647-cb9ed24678a4 2024-03-31T23:03:11.307 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:03:10 smithi046 ceph-mon[32498]: last_changed 2024-03-31T23:03:03.473853+0000 2024-03-31T23:03:11.307 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:03:10 smithi046 ceph-mon[32498]: created 2024-03-31T23:01:12.507350+0000 2024-03-31T23:03:11.307 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:03:10 smithi046 ceph-mon[32498]: min_mon_release 19 (squid) 2024-03-31T23:03:11.307 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:03:10 smithi046 ceph-mon[32498]: election_strategy: 1 2024-03-31T23:03:11.307 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:03:10 smithi046 ceph-mon[32498]: 0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-03-31T23:03:11.307 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:03:10 smithi046 ceph-mon[32498]: 1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.c 2024-03-31T23:03:11.307 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:03:10 smithi046 ceph-mon[32498]: 2: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.b 2024-03-31T23:03:11.307 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:03:10 smithi046 ceph-mon[32498]: fsmap 2024-03-31T23:03:11.307 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:03:10 smithi046 ceph-mon[32498]: osdmap e4: 0 total, 0 up, 0 in 2024-03-31T23:03:11.307 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:03:10 smithi046 ceph-mon[32498]: mgrmap e13: x(active, since 61s) 2024-03-31T23:03:11.308 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:03:10 smithi046 ceph-mon[32498]: overall HEALTH_OK 2024-03-31T23:03:11.308 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:03:10 smithi046 ceph-mon[32498]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-31T23:03:11.308 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:03:10 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:03:11.308 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:03:10 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:03:11.308 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:03:10 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:03:11.308 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:03:10 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:03:11.308 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:03:10 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:03:11.308 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:03:10 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:03:11.308 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:03:10 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:03:11.308 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:03:10 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:03:11.308 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:03:10 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:03:11.308 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:03:10 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:03:11.308 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:03:10 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:03:11.308 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:03:10 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-31T23:03:11.309 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:03:10 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-31T23:03:11.309 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:03:10 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:03:11.444 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:10 smithi142 ceph-mon[35271]: mon.b calling monitor election 2024-03-31T23:03:11.444 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:10 smithi142 ceph-mon[35271]: Updating smithi046:/etc/ceph/ceph.conf 2024-03-31T23:03:11.444 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:10 smithi142 ceph-mon[35271]: Updating smithi142:/etc/ceph/ceph.conf 2024-03-31T23:03:11.444 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:10 smithi142 ceph-mon[35271]: Updating smithi162:/etc/ceph/ceph.conf 2024-03-31T23:03:11.444 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:10 smithi142 ceph-mon[35271]: Updating smithi142:/var/lib/ceph/6a9d8f08-efb2-11ee-b647-cb9ed24678a4/config/ceph.conf 2024-03-31T23:03:11.444 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:10 smithi142 ceph-mon[35271]: Updating smithi046:/var/lib/ceph/6a9d8f08-efb2-11ee-b647-cb9ed24678a4/config/ceph.conf 2024-03-31T23:03:11.445 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:10 smithi142 ceph-mon[35271]: Updating smithi162:/var/lib/ceph/6a9d8f08-efb2-11ee-b647-cb9ed24678a4/config/ceph.conf 2024-03-31T23:03:11.445 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:10 smithi142 ceph-mon[35271]: mon.b calling monitor election 2024-03-31T23:03:11.445 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:10 smithi142 ceph-mon[35271]: mon.c calling monitor election 2024-03-31T23:03:11.445 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:10 smithi142 ceph-mon[35271]: mon.a calling monitor election 2024-03-31T23:03:11.445 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:10 smithi142 ceph-mon[35271]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-31T23:03:11.445 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:10 smithi142 ceph-mon[35271]: monmap epoch 3 2024-03-31T23:03:11.445 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:10 smithi142 ceph-mon[35271]: fsid 6a9d8f08-efb2-11ee-b647-cb9ed24678a4 2024-03-31T23:03:11.445 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:10 smithi142 ceph-mon[35271]: last_changed 2024-03-31T23:03:03.473853+0000 2024-03-31T23:03:11.445 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:10 smithi142 ceph-mon[35271]: created 2024-03-31T23:01:12.507350+0000 2024-03-31T23:03:11.445 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:10 smithi142 ceph-mon[35271]: min_mon_release 19 (squid) 2024-03-31T23:03:11.445 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:10 smithi142 ceph-mon[35271]: election_strategy: 1 2024-03-31T23:03:11.445 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:10 smithi142 ceph-mon[35271]: 0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-03-31T23:03:11.445 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:10 smithi142 ceph-mon[35271]: 1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.c 2024-03-31T23:03:11.445 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:10 smithi142 ceph-mon[35271]: 2: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.b 2024-03-31T23:03:11.445 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:10 smithi142 ceph-mon[35271]: fsmap 2024-03-31T23:03:11.446 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:10 smithi142 ceph-mon[35271]: osdmap e4: 0 total, 0 up, 0 in 2024-03-31T23:03:11.446 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:10 smithi142 ceph-mon[35271]: mgrmap e13: x(active, since 61s) 2024-03-31T23:03:11.446 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:10 smithi142 ceph-mon[35271]: overall HEALTH_OK 2024-03-31T23:03:11.446 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:10 smithi142 ceph-mon[35271]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-31T23:03:11.446 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:10 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:03:11.446 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:10 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:03:11.446 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:10 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:03:11.446 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:10 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:03:11.446 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:10 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:03:11.446 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:10 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:03:11.446 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:10 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:03:11.446 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:10 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:03:11.446 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:10 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:03:11.446 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:10 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:03:11.447 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:10 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:03:11.447 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:10 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-31T23:03:11.447 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:10 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-31T23:03:11.447 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:10 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:03:12.239 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:03:11 smithi046 ceph-mon[32498]: Reconfiguring mon.a (unknown last config time)... 2024-03-31T23:03:12.239 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:03:11 smithi046 ceph-mon[32498]: Reconfiguring daemon mon.a on smithi046 2024-03-31T23:03:12.239 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:03:11 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-31T23:03:12.295 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:03:11 smithi162 ceph-mon[36237]: Reconfiguring mon.a (unknown last config time)... 2024-03-31T23:03:12.296 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:03:11 smithi162 ceph-mon[36237]: Reconfiguring daemon mon.a on smithi046 2024-03-31T23:03:12.296 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:03:11 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-31T23:03:12.443 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:11 smithi142 ceph-mon[35271]: Reconfiguring mon.a (unknown last config time)... 2024-03-31T23:03:12.443 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:11 smithi142 ceph-mon[35271]: Reconfiguring daemon mon.a on smithi046 2024-03-31T23:03:12.443 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:11 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-31T23:03:13.243 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:03:12 smithi046 ceph-mon[32498]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-31T23:03:13.295 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:03:12 smithi162 ceph-mon[36237]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-31T23:03:13.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:03:13.346+0000 7f10f065b700 1 -- 172.21.15.46:0/1497432191 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f10e8102290 msgr2=0x7f10e8102710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:03:13.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:03:13.346+0000 7f10f065b700 1 --2- 172.21.15.46:0/1497432191 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f10e8102290 0x7f10e8102710 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f10dc0099c0 tx=0x7f10dc009cd0 comp rx=0 tx=0).stop 2024-03-31T23:03:13.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:03:13.346+0000 7f10f065b700 1 -- 172.21.15.46:0/1497432191 shutdown_connections 2024-03-31T23:03:13.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:03:13.346+0000 7f10f065b700 1 --2- 172.21.15.46:0/1497432191 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f10e8102c50 0x7f10e810f090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:03:13.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:03:13.346+0000 7f10f065b700 1 --2- 172.21.15.46:0/1497432191 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f10e8108270 0x7f10e8108650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:03:13.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:03:13.346+0000 7f10f065b700 1 --2- 172.21.15.46:0/1497432191 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f10e8102290 0x7f10e8102710 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:03:13.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:03:13.346+0000 7f10f065b700 1 -- 172.21.15.46:0/1497432191 >> 172.21.15.46:0/1497432191 conn(0x7f10e8079370 msgr2=0x7f10e8079780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:03:13.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:03:13.346+0000 7f10f065b700 1 -- 172.21.15.46:0/1497432191 shutdown_connections 2024-03-31T23:03:13.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:03:13.346+0000 7f10f065b700 1 -- 172.21.15.46:0/1497432191 wait complete. 2024-03-31T23:03:13.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:03:13.347+0000 7f10f065b700 1 Processor -- start 2024-03-31T23:03:13.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:03:13.347+0000 7f10f065b700 1 -- start start 2024-03-31T23:03:13.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:03:13.347+0000 7f10f065b700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f10e8102290 0x7f10e819e680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:03:13.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:03:13.347+0000 7f10f065b700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f10e8102c50 0x7f10e819ebc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:03:13.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:03:13.347+0000 7f10f065b700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f10e8108270 0x7f10e8198750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:03:13.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:03:13.347+0000 7f10f065b700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f10e81123d0 con 0x7f10e8102290 2024-03-31T23:03:13.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:03:13.347+0000 7f10f065b700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f10e8112250 con 0x7f10e8108270 2024-03-31T23:03:13.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:03:13.347+0000 7f10f065b700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f10e8112550 con 0x7f10e8102c50 2024-03-31T23:03:13.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:03:13.347+0000 7f10ee3f7700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f10e8102290 0x7f10e819e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:03:13.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:03:13.347+0000 7f10ee3f7700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f10e8102290 0x7f10e819e680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44368/0 (socket says 172.21.15.46:44368) 2024-03-31T23:03:13.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:03:13.347+0000 7f10ee3f7700 1 -- 172.21.15.46:0/1314555504 learned_addr learned my addr 172.21.15.46:0/1314555504 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:03:13.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:03:13.347+0000 7f10eebf8700 1 --2- 172.21.15.46:0/1314555504 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f10e8108270 0x7f10e8198750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:03:13.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:03:13.347+0000 7f10ee3f7700 1 -- 172.21.15.46:0/1314555504 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f10e8102c50 msgr2=0x7f10e819ebc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:03:13.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:03:13.347+0000 7f10edbf6700 1 --2- 172.21.15.46:0/1314555504 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f10e8102c50 0x7f10e819ebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:03:13.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:03:13.347+0000 7f10ee3f7700 1 --2- 172.21.15.46:0/1314555504 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f10e8102c50 0x7f10e819ebc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:03:13.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:03:13.347+0000 7f10ee3f7700 1 -- 172.21.15.46:0/1314555504 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f10e8108270 msgr2=0x7f10e8198750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:03:13.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:03:13.347+0000 7f10ee3f7700 1 --2- 172.21.15.46:0/1314555504 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f10e8108270 0x7f10e8198750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:03:13.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:03:13.348+0000 7f10ee3f7700 1 -- 172.21.15.46:0/1314555504 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f10e8199030 con 0x7f10e8102290 2024-03-31T23:03:13.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:03:13.348+0000 7f10ee3f7700 1 --2- 172.21.15.46:0/1314555504 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f10e8102290 0x7f10e819e680 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f10e000d980 tx=0x7f10e000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:03:13.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:03:13.348+0000 7f10e77fe700 1 -- 172.21.15.46:0/1314555504 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10e0016070 con 0x7f10e8102290 2024-03-31T23:03:13.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:03:13.348+0000 7f10e77fe700 1 -- 172.21.15.46:0/1314555504 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f10e0004830 con 0x7f10e8102290 2024-03-31T23:03:13.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:03:13.348+0000 7f10f065b700 1 -- 172.21.15.46:0/1314555504 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f10e8199320 con 0x7f10e8102290 2024-03-31T23:03:13.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:03:13.348+0000 7f10e77fe700 1 -- 172.21.15.46:0/1314555504 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10e00054b0 con 0x7f10e8102290 2024-03-31T23:03:13.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:03:13.348+0000 7f10f065b700 1 -- 172.21.15.46:0/1314555504 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f10e81a5600 con 0x7f10e8102290 2024-03-31T23:03:13.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:03:13.349+0000 7f10e77fe700 1 -- 172.21.15.46:0/1314555504 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 13) ==== 46172+0+0 (secure 0 0 0) 0x7f10e000b750 con 0x7f10e8102290 2024-03-31T23:03:13.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:03:13.349+0000 7f10e77fe700 1 --2- 172.21.15.46:0/1314555504 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f10d00390e0 0x7f10d003b5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:03:13.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:03:13.349+0000 7f10e77fe700 1 -- 172.21.15.46:0/1314555504 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f10e004bd20 con 0x7f10e8102290 2024-03-31T23:03:13.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:03:13.349+0000 7f10edbf6700 1 --2- 172.21.15.46:0/1314555504 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f10d00390e0 0x7f10d003b5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:03:13.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:03:13.349+0000 7f10f065b700 1 -- 172.21.15.46:0/1314555504 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f10e8103920 con 0x7f10e8102290 2024-03-31T23:03:13.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:03:13.349+0000 7f10edbf6700 1 --2- 172.21.15.46:0/1314555504 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f10d00390e0 0x7f10d003b5a0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f10dc006010 tx=0x7f10dc000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:03:13.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:03:13.354+0000 7f10e77fe700 1 -- 172.21.15.46:0/1314555504 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f10e001f020 con 0x7f10e8102290 2024-03-31T23:03:13.443 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:12 smithi142 ceph-mon[35271]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-31T23:03:13.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:03:13.558+0000 7f10f065b700 1 -- 172.21.15.46:0/1314555504 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) -- 0x7f10e8199c80 con 0x7f10e8102290 2024-03-31T23:03:13.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:03:13.558+0000 7f10e77fe700 1 -- 172.21.15.46:0/1314555504 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v9) ==== 76+0+275 (secure 0 0 0) 0x7f10e0004220 con 0x7f10e8102290 2024-03-31T23:03:13.560 INFO:teuthology.orchestra.run.smithi046.stdout:# minimal ceph.conf for 6a9d8f08-efb2-11ee-b647-cb9ed24678a4 2024-03-31T23:03:13.560 INFO:teuthology.orchestra.run.smithi046.stdout:[global] 2024-03-31T23:03:13.560 INFO:teuthology.orchestra.run.smithi046.stdout: fsid = 6a9d8f08-efb2-11ee-b647-cb9ed24678a4 2024-03-31T23:03:13.560 INFO:teuthology.orchestra.run.smithi046.stdout: mon_host = [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] 2024-03-31T23:03:13.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:03:13.561+0000 7f10f065b700 1 -- 172.21.15.46:0/1314555504 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f10d00390e0 msgr2=0x7f10d003b5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:03:13.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:03:13.561+0000 7f10f065b700 1 --2- 172.21.15.46:0/1314555504 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f10d00390e0 0x7f10d003b5a0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f10dc006010 tx=0x7f10dc000bc0 comp rx=0 tx=0).stop 2024-03-31T23:03:13.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:03:13.561+0000 7f10f065b700 1 -- 172.21.15.46:0/1314555504 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f10e8102290 msgr2=0x7f10e819e680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:03:13.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:03:13.561+0000 7f10f065b700 1 --2- 172.21.15.46:0/1314555504 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f10e8102290 0x7f10e819e680 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f10e000d980 tx=0x7f10e000dc90 comp rx=0 tx=0).stop 2024-03-31T23:03:13.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:03:13.561+0000 7f10f065b700 1 -- 172.21.15.46:0/1314555504 shutdown_connections 2024-03-31T23:03:13.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:03:13.561+0000 7f10f065b700 1 --2- 172.21.15.46:0/1314555504 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f10d00390e0 0x7f10d003b5a0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:03:13.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:03:13.561+0000 7f10f065b700 1 --2- 172.21.15.46:0/1314555504 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f10e8102290 0x7f10e819e680 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:03:13.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:03:13.561+0000 7f10f065b700 1 --2- 172.21.15.46:0/1314555504 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f10e8108270 0x7f10e8198750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:03:13.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:03:13.561+0000 7f10f065b700 1 --2- 172.21.15.46:0/1314555504 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f10e8102c50 0x7f10e819ebc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:03:13.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:03:13.561+0000 7f10f065b700 1 -- 172.21.15.46:0/1314555504 >> 172.21.15.46:0/1314555504 conn(0x7f10e8079370 msgr2=0x7f10e8101030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:03:13.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:03:13.561+0000 7f10f065b700 1 -- 172.21.15.46:0/1314555504 shutdown_connections 2024-03-31T23:03:13.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:03:13.561+0000 7f10f065b700 1 -- 172.21.15.46:0/1314555504 wait complete. 2024-03-31T23:03:14.149 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-03-31T23:03:14.150 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-03-31T23:03:14.150 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd of=/etc/ceph/ceph.conf 2024-03-31T23:03:14.179 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-03-31T23:03:14.179 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-31T23:03:14.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:14 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:03:14.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:14 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:03:14.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:14 smithi142 ceph-mon[35271]: Reconfiguring mon.b (monmap changed)... 2024-03-31T23:03:14.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:14 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-31T23:03:14.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:14 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-31T23:03:14.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:14 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:03:14.194 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:14 smithi142 ceph-mon[35271]: Reconfiguring daemon mon.b on smithi142 2024-03-31T23:03:14.194 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:14 smithi142 ceph-mon[35271]: from='client.? 172.21.15.46:0/1314555504' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:03:14.246 DEBUG:teuthology.orchestra.run.smithi142:> set -ex 2024-03-31T23:03:14.246 DEBUG:teuthology.orchestra.run.smithi142:> sudo dd of=/etc/ceph/ceph.conf 2024-03-31T23:03:14.277 DEBUG:teuthology.orchestra.run.smithi142:> set -ex 2024-03-31T23:03:14.277 DEBUG:teuthology.orchestra.run.smithi142:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-31T23:03:14.295 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:03:14 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:03:14.295 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:03:14 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:03:14.295 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:03:14 smithi162 ceph-mon[36237]: Reconfiguring mon.b (monmap changed)... 2024-03-31T23:03:14.296 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:03:14 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-31T23:03:14.296 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:03:14 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-31T23:03:14.296 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:03:14 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:03:14.296 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:03:14 smithi162 ceph-mon[36237]: Reconfiguring daemon mon.b on smithi142 2024-03-31T23:03:14.296 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:03:14 smithi162 ceph-mon[36237]: from='client.? 172.21.15.46:0/1314555504' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:03:14.345 DEBUG:teuthology.orchestra.run.smithi162:> set -ex 2024-03-31T23:03:14.345 DEBUG:teuthology.orchestra.run.smithi162:> sudo dd of=/etc/ceph/ceph.conf 2024-03-31T23:03:14.361 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:03:14 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:03:14.361 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:03:14 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:03:14.362 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:03:14 smithi046 ceph-mon[32498]: Reconfiguring mon.b (monmap changed)... 2024-03-31T23:03:14.362 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:03:14 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-31T23:03:14.362 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:03:14 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-31T23:03:14.362 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:03:14 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:03:14.362 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:03:14 smithi046 ceph-mon[32498]: Reconfiguring daemon mon.b on smithi142 2024-03-31T23:03:14.362 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:03:14 smithi046 ceph-mon[32498]: from='client.? 172.21.15.46:0/1314555504' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:03:14.380 DEBUG:teuthology.orchestra.run.smithi162:> set -ex 2024-03-31T23:03:14.380 DEBUG:teuthology.orchestra.run.smithi162:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-31T23:03:14.453 INFO:tasks.cephadm:Adding mgr.x on smithi046 2024-03-31T23:03:14.453 INFO:tasks.cephadm:Adding mgr.y on smithi142 2024-03-31T23:03:14.453 INFO:tasks.cephadm:Adding mgr.z on smithi162 2024-03-31T23:03:14.453 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:384f31f3d3aee7f8ba3f60846b8844c049165583 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6a9d8f08-efb2-11ee-b647-cb9ed24678a4 -- ceph orch apply mgr '3;smithi046=x;smithi142=y;smithi162=z' 2024-03-31T23:03:14.730 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/6a9d8f08-efb2-11ee-b647-cb9ed24678a4/mon.c/config 2024-03-31T23:03:15.693 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:15 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:03:15.693 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:15 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:03:15.693 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:15 smithi142 ceph-mon[35271]: Reconfiguring mon.c (monmap changed)... 2024-03-31T23:03:15.694 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:15 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-31T23:03:15.694 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:15 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-31T23:03:15.694 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:15 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:03:15.694 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:15 smithi142 ceph-mon[35271]: Reconfiguring daemon mon.c on smithi162 2024-03-31T23:03:15.694 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:15 smithi142 ceph-mon[35271]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-31T23:03:15.696 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:03:15 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:03:15.696 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:03:15 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:03:15.696 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:03:15 smithi162 ceph-mon[36237]: Reconfiguring mon.c (monmap changed)... 2024-03-31T23:03:15.696 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:03:15 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-31T23:03:15.696 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:03:15 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-31T23:03:15.696 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:03:15 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:03:15.697 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:03:15 smithi162 ceph-mon[36237]: Reconfiguring daemon mon.c on smithi162 2024-03-31T23:03:15.697 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:03:15 smithi162 ceph-mon[36237]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-31T23:03:15.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:03:15 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:03:15.862 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:03:15 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:03:15.862 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:03:15 smithi046 ceph-mon[32498]: Reconfiguring mon.c (monmap changed)... 2024-03-31T23:03:15.862 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:03:15 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-31T23:03:15.862 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:03:15 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-31T23:03:15.862 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:03:15 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:03:15.862 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:03:15 smithi046 ceph-mon[32498]: Reconfiguring daemon mon.c on smithi162 2024-03-31T23:03:15.862 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:03:15 smithi046 ceph-mon[32498]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-31T23:03:16.841 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:03:16 smithi162 ceph-mon[36237]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-31T23:03:16.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:03:16 smithi046 ceph-mon[32498]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-31T23:03:16.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:16 smithi142 ceph-mon[35271]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-31T23:03:16.950 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:03:16.943+0000 7fd55c879700 1 -- 172.21.15.162:0/1798585143 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd558102280 msgr2=0x7fd5581026a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:03:16.951 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:03:16.943+0000 7fd55c879700 1 --2- 172.21.15.162:0/1798585143 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd558102280 0x7fd5581026a0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fd548009910 tx=0x7fd548009c20 comp rx=0 tx=0).stop 2024-03-31T23:03:16.951 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:03:16.943+0000 7fd55c879700 1 -- 172.21.15.162:0/1798585143 shutdown_connections 2024-03-31T23:03:16.951 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:03:16.943+0000 7fd55c879700 1 --2- 172.21.15.162:0/1798585143 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd558102280 0x7fd5581026a0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:03:16.951 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:03:16.943+0000 7fd55c879700 1 -- 172.21.15.162:0/1798585143 >> 172.21.15.162:0/1798585143 conn(0x7fd5580fb990 msgr2=0x7fd5580fddd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:03:16.951 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:03:16.943+0000 7fd55c879700 1 -- 172.21.15.162:0/1798585143 shutdown_connections 2024-03-31T23:03:16.951 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:03:16.943+0000 7fd55c879700 1 -- 172.21.15.162:0/1798585143 wait complete. 2024-03-31T23:03:16.952 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:03:16.944+0000 7fd55c879700 1 Processor -- start 2024-03-31T23:03:16.952 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:03:16.944+0000 7fd55c879700 1 -- start start 2024-03-31T23:03:16.952 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:03:16.944+0000 7fd55c879700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd558102280 0x7fd558100950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:03:16.953 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:03:16.944+0000 7fd55c879700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd55819a240 0x7fd55819c630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:03:16.953 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:03:16.944+0000 7fd55c879700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd55819cbc0 0x7fd55819f060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:03:16.953 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:03:16.944+0000 7fd55c879700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fd558106bd0 con 0x7fd558102280 2024-03-31T23:03:16.953 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:03:16.944+0000 7fd55c879700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fd558106a50 con 0x7fd55819cbc0 2024-03-31T23:03:16.953 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:03:16.944+0000 7fd55c879700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fd558106d50 con 0x7fd55819a240 2024-03-31T23:03:16.953 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:03:16.944+0000 7fd54dd9b700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd55819a240 0x7fd55819c630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:03:16.954 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:03:16.944+0000 7fd55659c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd558102280 0x7fd558100950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:03:16.954 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:03:16.945+0000 7fd54dd9b700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd55819a240 0x7fd55819c630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:33974/0 (socket says 172.21.15.162:33974) 2024-03-31T23:03:16.954 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:03:16.945+0000 7fd54dd9b700 1 -- 172.21.15.162:0/1918219166 learned_addr learned my addr 172.21.15.162:0/1918219166 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-03-31T23:03:16.955 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:03:16.945+0000 7fd55659c700 1 --2- 172.21.15.162:0/1918219166 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd558102280 0x7fd558100950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.162:33118/0 (socket says 172.21.15.162:33118) 2024-03-31T23:03:16.955 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:03:16.945+0000 7fd556d9d700 1 --2- 172.21.15.162:0/1918219166 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd55819cbc0 0x7fd55819f060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:03:16.955 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:03:16.945+0000 7fd55659c700 1 -- 172.21.15.162:0/1918219166 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd55819a240 msgr2=0x7fd55819c630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:03:16.955 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:03:16.945+0000 7fd55659c700 1 --2- 172.21.15.162:0/1918219166 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd55819a240 0x7fd55819c630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:03:16.956 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:03:16.945+0000 7fd55659c700 1 -- 172.21.15.162:0/1918219166 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd55819cbc0 msgr2=0x7fd55819f060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:03:16.956 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:03:16.945+0000 7fd55659c700 1 --2- 172.21.15.162:0/1918219166 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd55819cbc0 0x7fd55819f060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:03:16.956 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:03:16.945+0000 7fd55659c700 1 -- 172.21.15.162:0/1918219166 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd55819f930 con 0x7fd558102280 2024-03-31T23:03:16.956 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:03:16.945+0000 7fd54dd9b700 1 --2- 172.21.15.162:0/1918219166 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd55819a240 0x7fd55819c630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:03:16.957 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:03:16.945+0000 7fd556d9d700 1 --2- 172.21.15.162:0/1918219166 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd55819cbc0 0x7fd55819f060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T23:03:16.957 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:03:16.945+0000 7fd55659c700 1 --2- 172.21.15.162:0/1918219166 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd558102280 0x7fd558100950 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fd548005e00 tx=0x7fd548004fa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:03:16.957 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:03:16.945+0000 7fd54ffff700 1 -- 172.21.15.162:0/1918219166 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd548026070 con 0x7fd558102280 2024-03-31T23:03:16.957 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:03:16.945+0000 7fd54ffff700 1 -- 172.21.15.162:0/1918219166 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd54800b750 con 0x7fd558102280 2024-03-31T23:03:16.957 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:03:16.945+0000 7fd55c879700 1 -- 172.21.15.162:0/1918219166 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd55819fbc0 con 0x7fd558102280 2024-03-31T23:03:16.957 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:03:16.945+0000 7fd54ffff700 1 -- 172.21.15.162:0/1918219166 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5480039e0 con 0x7fd558102280 2024-03-31T23:03:16.958 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:03:16.945+0000 7fd55c879700 1 -- 172.21.15.162:0/1918219166 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd5581a0070 con 0x7fd558102280 2024-03-31T23:03:16.958 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:03:16.946+0000 7fd54ffff700 1 -- 172.21.15.162:0/1918219166 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 13) ==== 46172+0+0 (secure 0 0 0) 0x7fd548029070 con 0x7fd558102280 2024-03-31T23:03:16.958 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:03:16.946+0000 7fd55c879700 1 -- 172.21.15.162:0/1918219166 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd558063870 con 0x7fd558102280 2024-03-31T23:03:16.958 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:03:16.946+0000 7fd54ffff700 1 --2- 172.21.15.162:0/1918219166 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fd544039330 0x7fd54403b7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:03:16.958 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:03:16.946+0000 7fd54ffff700 1 -- 172.21.15.162:0/1918219166 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7fd54804cec0 con 0x7fd558102280 2024-03-31T23:03:16.958 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:03:16.946+0000 7fd54dd9b700 1 --2- 172.21.15.162:0/1918219166 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fd544039330 0x7fd54403b7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:03:16.958 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:03:16.947+0000 7fd54dd9b700 1 --2- 172.21.15.162:0/1918219166 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fd544039330 0x7fd54403b7f0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fd558100eb0 tx=0x7fd5400092c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:03:16.959 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:03:16.952+0000 7fd54ffff700 1 -- 172.21.15.162:0/1918219166 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fd548013070 con 0x7fd558102280 2024-03-31T23:03:17.154 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:03:17.153+0000 7fd55c879700 1 -- 172.21.15.162:0/1918219166 --> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi046=x;smithi142=y;smithi162=z", "target": ["mon-mgr", ""]}) -- 0x7fd558062680 con 0x7fd544039330 2024-03-31T23:03:17.159 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:03:17.159+0000 7fd54ffff700 1 -- 172.21.15.162:0/1918219166 <== mgr.14152 v2:172.21.15.46:6800/162289569 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7fd558062680 con 0x7fd544039330 2024-03-31T23:03:17.160 INFO:teuthology.orchestra.run.smithi162.stdout:Scheduled mgr update... 2024-03-31T23:03:17.162 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:03:17.161+0000 7fd55c879700 1 -- 172.21.15.162:0/1918219166 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fd544039330 msgr2=0x7fd54403b7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:03:17.162 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:03:17.161+0000 7fd55c879700 1 --2- 172.21.15.162:0/1918219166 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fd544039330 0x7fd54403b7f0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fd558100eb0 tx=0x7fd5400092c0 comp rx=0 tx=0).stop 2024-03-31T23:03:17.162 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:03:17.161+0000 7fd55c879700 1 -- 172.21.15.162:0/1918219166 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd558102280 msgr2=0x7fd558100950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:03:17.163 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:03:17.161+0000 7fd55c879700 1 --2- 172.21.15.162:0/1918219166 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd558102280 0x7fd558100950 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fd548005e00 tx=0x7fd548004fa0 comp rx=0 tx=0).stop 2024-03-31T23:03:17.163 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:03:17.161+0000 7fd55c879700 1 -- 172.21.15.162:0/1918219166 shutdown_connections 2024-03-31T23:03:17.163 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:03:17.161+0000 7fd55c879700 1 --2- 172.21.15.162:0/1918219166 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fd544039330 0x7fd54403b7f0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:03:17.163 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:03:17.161+0000 7fd55c879700 1 --2- 172.21.15.162:0/1918219166 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd558102280 0x7fd558100950 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:03:17.163 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:03:17.161+0000 7fd55c879700 1 --2- 172.21.15.162:0/1918219166 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd55819cbc0 0x7fd55819f060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:03:17.163 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:03:17.161+0000 7fd55c879700 1 --2- 172.21.15.162:0/1918219166 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd55819a240 0x7fd55819c630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:03:17.163 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:03:17.161+0000 7fd55c879700 1 -- 172.21.15.162:0/1918219166 >> 172.21.15.162:0/1918219166 conn(0x7fd5580fb990 msgr2=0x7fd5580fc680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:03:17.163 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:03:17.161+0000 7fd55c879700 1 -- 172.21.15.162:0/1918219166 shutdown_connections 2024-03-31T23:03:17.163 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:03:17.161+0000 7fd55c879700 1 -- 172.21.15.162:0/1918219166 wait complete. 2024-03-31T23:03:17.748 DEBUG:teuthology.orchestra.run.smithi142:mgr.y> sudo journalctl -f -n 0 -u ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4@mgr.y.service 2024-03-31T23:03:17.750 DEBUG:teuthology.orchestra.run.smithi162:mgr.z> sudo journalctl -f -n 0 -u ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4@mgr.z.service 2024-03-31T23:03:17.753 INFO:tasks.cephadm:Deploying OSDs... 2024-03-31T23:03:17.753 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-03-31T23:03:17.753 DEBUG:teuthology.orchestra.run.smithi046:> dd if=/scratch_devs of=/dev/stdout 2024-03-31T23:03:17.770 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-31T23:03:17.770 DEBUG:teuthology.orchestra.run.smithi046:> stat /dev/vg_nvme/lv_1 2024-03-31T23:03:17.826 INFO:teuthology.orchestra.run.smithi046.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-31T23:03:17.826 INFO:teuthology.orchestra.run.smithi046.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-31T23:03:17.826 INFO:teuthology.orchestra.run.smithi046.stdout:Device: 5h/5d Inode: 804 Links: 1 2024-03-31T23:03:17.827 INFO:teuthology.orchestra.run.smithi046.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-31T23:03:17.827 INFO:teuthology.orchestra.run.smithi046.stdout:Context: system_u:object_r:device_t:s0 2024-03-31T23:03:17.827 INFO:teuthology.orchestra.run.smithi046.stdout:Access: 2024-03-31 23:02:20.884551642 +0000 2024-03-31T23:03:17.827 INFO:teuthology.orchestra.run.smithi046.stdout:Modify: 2024-03-31 22:57:40.869986260 +0000 2024-03-31T23:03:17.827 INFO:teuthology.orchestra.run.smithi046.stdout:Change: 2024-03-31 22:57:40.869986260 +0000 2024-03-31T23:03:17.827 INFO:teuthology.orchestra.run.smithi046.stdout: Birth: 2024-03-31 22:57:40.869986260 +0000 2024-03-31T23:03:17.827 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-31T23:03:17.894 INFO:teuthology.orchestra.run.smithi046.stderr:1+0 records in 2024-03-31T23:03:17.894 INFO:teuthology.orchestra.run.smithi046.stderr:1+0 records out 2024-03-31T23:03:17.894 INFO:teuthology.orchestra.run.smithi046.stderr:512 bytes copied, 9.6368e-05 s, 5.3 MB/s 2024-03-31T23:03:17.895 DEBUG:teuthology.orchestra.run.smithi046:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-31T23:03:17.952 DEBUG:teuthology.orchestra.run.smithi046:> stat /dev/vg_nvme/lv_2 2024-03-31T23:03:18.008 INFO:teuthology.orchestra.run.smithi046.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-31T23:03:18.008 INFO:teuthology.orchestra.run.smithi046.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-31T23:03:18.008 INFO:teuthology.orchestra.run.smithi046.stdout:Device: 5h/5d Inode: 825 Links: 1 2024-03-31T23:03:18.008 INFO:teuthology.orchestra.run.smithi046.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-31T23:03:18.009 INFO:teuthology.orchestra.run.smithi046.stdout:Context: system_u:object_r:device_t:s0 2024-03-31T23:03:18.009 INFO:teuthology.orchestra.run.smithi046.stdout:Access: 2024-03-31 23:02:20.921551206 +0000 2024-03-31T23:03:18.009 INFO:teuthology.orchestra.run.smithi046.stdout:Modify: 2024-03-31 22:57:41.258981656 +0000 2024-03-31T23:03:18.009 INFO:teuthology.orchestra.run.smithi046.stdout:Change: 2024-03-31 22:57:41.258981656 +0000 2024-03-31T23:03:18.009 INFO:teuthology.orchestra.run.smithi046.stdout: Birth: 2024-03-31 22:57:41.258981656 +0000 2024-03-31T23:03:18.009 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-31T23:03:18.026 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:03:17 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:03:18.026 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:03:17 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:03:18.026 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:03:17 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-31T23:03:18.026 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:03:17 smithi162 ceph-mon[36237]: from='client.14214 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi046=x;smithi142=y;smithi162=z", "target": ["mon-mgr", ""]}]: dispatch 2024-03-31T23:03:18.026 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:03:17 smithi162 ceph-mon[36237]: Saving service mgr spec with placement smithi046=x;smithi142=y;smithi162=z;count:3 2024-03-31T23:03:18.026 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:03:17 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:03:18.026 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:03:17 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:03:18.026 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:03:17 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-31T23:03:18.026 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:03:17 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:03:18.027 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:03:17 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-31T23:03:18.027 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:03:17 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-31T23:03:18.027 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:03:17 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-31T23:03:18.027 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:03:17 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:03:18.027 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:03:17 smithi162 ceph-mon[36237]: Deploying daemon mgr.z on smithi162 2024-03-31T23:03:18.075 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:03:17 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:03:18.075 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:03:17 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:03:18.076 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:03:17 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-31T23:03:18.076 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:03:17 smithi046 ceph-mon[32498]: from='client.14214 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi046=x;smithi142=y;smithi162=z", "target": ["mon-mgr", ""]}]: dispatch 2024-03-31T23:03:18.076 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:03:17 smithi046 ceph-mon[32498]: Saving service mgr spec with placement smithi046=x;smithi142=y;smithi162=z;count:3 2024-03-31T23:03:18.076 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:03:17 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:03:18.076 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:03:17 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:03:18.076 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:03:17 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-31T23:03:18.076 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:03:17 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:03:18.076 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:03:17 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-31T23:03:18.076 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:03:17 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-31T23:03:18.076 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:03:17 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-31T23:03:18.076 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:03:17 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:03:18.076 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:03:17 smithi046 ceph-mon[32498]: Deploying daemon mgr.z on smithi162 2024-03-31T23:03:18.077 INFO:teuthology.orchestra.run.smithi046.stderr:1+0 records in 2024-03-31T23:03:18.078 INFO:teuthology.orchestra.run.smithi046.stderr:1+0 records out 2024-03-31T23:03:18.078 INFO:teuthology.orchestra.run.smithi046.stderr:512 bytes copied, 0.000108221 s, 4.7 MB/s 2024-03-31T23:03:18.078 DEBUG:teuthology.orchestra.run.smithi046:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-31T23:03:18.134 DEBUG:teuthology.orchestra.run.smithi046:> stat /dev/vg_nvme/lv_3 2024-03-31T23:03:18.192 INFO:teuthology.orchestra.run.smithi046.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-31T23:03:18.193 INFO:teuthology.orchestra.run.smithi046.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-31T23:03:18.193 INFO:teuthology.orchestra.run.smithi046.stdout:Device: 5h/5d Inode: 843 Links: 1 2024-03-31T23:03:18.193 INFO:teuthology.orchestra.run.smithi046.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-31T23:03:18.193 INFO:teuthology.orchestra.run.smithi046.stdout:Context: system_u:object_r:device_t:s0 2024-03-31T23:03:18.193 INFO:teuthology.orchestra.run.smithi046.stdout:Access: 2024-03-31 23:02:20.956550793 +0000 2024-03-31T23:03:18.193 INFO:teuthology.orchestra.run.smithi046.stdout:Modify: 2024-03-31 22:57:41.578977869 +0000 2024-03-31T23:03:18.193 INFO:teuthology.orchestra.run.smithi046.stdout:Change: 2024-03-31 22:57:41.578977869 +0000 2024-03-31T23:03:18.193 INFO:teuthology.orchestra.run.smithi046.stdout: Birth: 2024-03-31 22:57:41.578977869 +0000 2024-03-31T23:03:18.193 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-31T23:03:18.194 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:17 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:03:18.194 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:17 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:03:18.194 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:17 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-31T23:03:18.194 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:17 smithi142 ceph-mon[35271]: from='client.14214 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi046=x;smithi142=y;smithi162=z", "target": ["mon-mgr", ""]}]: dispatch 2024-03-31T23:03:18.195 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:17 smithi142 ceph-mon[35271]: Saving service mgr spec with placement smithi046=x;smithi142=y;smithi162=z;count:3 2024-03-31T23:03:18.195 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:17 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:03:18.195 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:17 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:03:18.195 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:17 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-31T23:03:18.195 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:17 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:03:18.195 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:17 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-31T23:03:18.195 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:17 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-31T23:03:18.195 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:17 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-31T23:03:18.195 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:17 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:03:18.195 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:17 smithi142 ceph-mon[35271]: Deploying daemon mgr.z on smithi162 2024-03-31T23:03:18.259 INFO:teuthology.orchestra.run.smithi046.stderr:1+0 records in 2024-03-31T23:03:18.259 INFO:teuthology.orchestra.run.smithi046.stderr:1+0 records out 2024-03-31T23:03:18.259 INFO:teuthology.orchestra.run.smithi046.stderr:512 bytes copied, 9.4693e-05 s, 5.4 MB/s 2024-03-31T23:03:18.260 DEBUG:teuthology.orchestra.run.smithi046:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-31T23:03:18.317 DEBUG:teuthology.orchestra.run.smithi046:> stat /dev/vg_nvme/lv_4 2024-03-31T23:03:18.373 INFO:teuthology.orchestra.run.smithi046.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-31T23:03:18.373 INFO:teuthology.orchestra.run.smithi046.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-31T23:03:18.373 INFO:teuthology.orchestra.run.smithi046.stdout:Device: 5h/5d Inode: 860 Links: 1 2024-03-31T23:03:18.373 INFO:teuthology.orchestra.run.smithi046.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-31T23:03:18.373 INFO:teuthology.orchestra.run.smithi046.stdout:Context: system_u:object_r:device_t:s0 2024-03-31T23:03:18.373 INFO:teuthology.orchestra.run.smithi046.stdout:Access: 2024-03-31 23:02:20.990550393 +0000 2024-03-31T23:03:18.373 INFO:teuthology.orchestra.run.smithi046.stdout:Modify: 2024-03-31 22:57:41.909973951 +0000 2024-03-31T23:03:18.373 INFO:teuthology.orchestra.run.smithi046.stdout:Change: 2024-03-31 22:57:41.909973951 +0000 2024-03-31T23:03:18.373 INFO:teuthology.orchestra.run.smithi046.stdout: Birth: 2024-03-31 22:57:41.909973951 +0000 2024-03-31T23:03:18.374 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-31T23:03:18.438 INFO:teuthology.orchestra.run.smithi046.stderr:1+0 records in 2024-03-31T23:03:18.439 INFO:teuthology.orchestra.run.smithi046.stderr:1+0 records out 2024-03-31T23:03:18.439 INFO:teuthology.orchestra.run.smithi046.stderr:512 bytes copied, 0.00010463 s, 4.9 MB/s 2024-03-31T23:03:18.440 DEBUG:teuthology.orchestra.run.smithi046:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-31T23:03:18.497 DEBUG:teuthology.orchestra.run.smithi142:> set -ex 2024-03-31T23:03:18.497 DEBUG:teuthology.orchestra.run.smithi142:> dd if=/scratch_devs of=/dev/stdout 2024-03-31T23:03:18.516 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-31T23:03:18.516 DEBUG:teuthology.orchestra.run.smithi142:> stat /dev/vg_nvme/lv_1 2024-03-31T23:03:18.574 INFO:teuthology.orchestra.run.smithi142.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-31T23:03:18.575 INFO:teuthology.orchestra.run.smithi142.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-31T23:03:18.575 INFO:teuthology.orchestra.run.smithi142.stdout:Device: 5h/5d Inode: 665 Links: 1 2024-03-31T23:03:18.575 INFO:teuthology.orchestra.run.smithi142.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-31T23:03:18.576 INFO:teuthology.orchestra.run.smithi142.stdout:Context: system_u:object_r:device_t:s0 2024-03-31T23:03:18.576 INFO:teuthology.orchestra.run.smithi142.stdout:Access: 2024-03-31 23:03:02.919660146 +0000 2024-03-31T23:03:18.576 INFO:teuthology.orchestra.run.smithi142.stdout:Modify: 2024-03-31 22:57:32.225205314 +0000 2024-03-31T23:03:18.576 INFO:teuthology.orchestra.run.smithi142.stdout:Change: 2024-03-31 22:57:32.225205314 +0000 2024-03-31T23:03:18.576 INFO:teuthology.orchestra.run.smithi142.stdout: Birth: 2024-03-31 22:57:32.225205314 +0000 2024-03-31T23:03:18.576 DEBUG:teuthology.orchestra.run.smithi142:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-31T23:03:18.644 INFO:teuthology.orchestra.run.smithi142.stderr:1+0 records in 2024-03-31T23:03:18.645 INFO:teuthology.orchestra.run.smithi142.stderr:1+0 records out 2024-03-31T23:03:18.645 INFO:teuthology.orchestra.run.smithi142.stderr:512 bytes copied, 0.000211935 s, 2.4 MB/s 2024-03-31T23:03:18.646 DEBUG:teuthology.orchestra.run.smithi142:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-31T23:03:18.704 DEBUG:teuthology.orchestra.run.smithi142:> stat /dev/vg_nvme/lv_2 2024-03-31T23:03:18.761 INFO:teuthology.orchestra.run.smithi142.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-31T23:03:18.762 INFO:teuthology.orchestra.run.smithi142.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-31T23:03:18.762 INFO:teuthology.orchestra.run.smithi142.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-03-31T23:03:18.762 INFO:teuthology.orchestra.run.smithi142.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-31T23:03:18.762 INFO:teuthology.orchestra.run.smithi142.stdout:Context: system_u:object_r:device_t:s0 2024-03-31T23:03:18.762 INFO:teuthology.orchestra.run.smithi142.stdout:Access: 2024-03-31 23:03:02.920660126 +0000 2024-03-31T23:03:18.762 INFO:teuthology.orchestra.run.smithi142.stdout:Modify: 2024-03-31 22:57:32.572198552 +0000 2024-03-31T23:03:18.762 INFO:teuthology.orchestra.run.smithi142.stdout:Change: 2024-03-31 22:57:32.572198552 +0000 2024-03-31T23:03:18.762 INFO:teuthology.orchestra.run.smithi142.stdout: Birth: 2024-03-31 22:57:32.572198552 +0000 2024-03-31T23:03:18.763 DEBUG:teuthology.orchestra.run.smithi142:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-31T23:03:18.830 INFO:teuthology.orchestra.run.smithi142.stderr:1+0 records in 2024-03-31T23:03:18.831 INFO:teuthology.orchestra.run.smithi142.stderr:1+0 records out 2024-03-31T23:03:18.831 INFO:teuthology.orchestra.run.smithi142.stderr:512 bytes copied, 0.000227768 s, 2.2 MB/s 2024-03-31T23:03:18.832 DEBUG:teuthology.orchestra.run.smithi142:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-31T23:03:18.892 DEBUG:teuthology.orchestra.run.smithi142:> stat /dev/vg_nvme/lv_3 2024-03-31T23:03:18.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:18 smithi142 ceph-mon[35271]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-31T23:03:18.948 INFO:teuthology.orchestra.run.smithi142.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-31T23:03:18.948 INFO:teuthology.orchestra.run.smithi142.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-31T23:03:18.948 INFO:teuthology.orchestra.run.smithi142.stdout:Device: 5h/5d Inode: 703 Links: 1 2024-03-31T23:03:18.948 INFO:teuthology.orchestra.run.smithi142.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-31T23:03:18.948 INFO:teuthology.orchestra.run.smithi142.stdout:Context: system_u:object_r:device_t:s0 2024-03-31T23:03:18.948 INFO:teuthology.orchestra.run.smithi142.stdout:Access: 2024-03-31 23:03:02.920660126 +0000 2024-03-31T23:03:18.948 INFO:teuthology.orchestra.run.smithi142.stdout:Modify: 2024-03-31 22:57:32.914191888 +0000 2024-03-31T23:03:18.948 INFO:teuthology.orchestra.run.smithi142.stdout:Change: 2024-03-31 22:57:32.914191888 +0000 2024-03-31T23:03:18.948 INFO:teuthology.orchestra.run.smithi142.stdout: Birth: 2024-03-31 22:57:32.914191888 +0000 2024-03-31T23:03:18.948 DEBUG:teuthology.orchestra.run.smithi142:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-31T23:03:18.956 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:03:18 smithi162 ceph-mon[36237]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-31T23:03:19.014 INFO:teuthology.orchestra.run.smithi142.stderr:1+0 records in 2024-03-31T23:03:19.014 INFO:teuthology.orchestra.run.smithi142.stderr:1+0 records out 2024-03-31T23:03:19.014 INFO:teuthology.orchestra.run.smithi142.stderr:512 bytes copied, 0.000193699 s, 2.6 MB/s 2024-03-31T23:03:19.016 DEBUG:teuthology.orchestra.run.smithi142:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-31T23:03:19.071 DEBUG:teuthology.orchestra.run.smithi142:> stat /dev/vg_nvme/lv_4 2024-03-31T23:03:19.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:03:18 smithi046 ceph-mon[32498]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-31T23:03:19.127 INFO:teuthology.orchestra.run.smithi142.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-31T23:03:19.127 INFO:teuthology.orchestra.run.smithi142.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-31T23:03:19.127 INFO:teuthology.orchestra.run.smithi142.stdout:Device: 5h/5d Inode: 724 Links: 1 2024-03-31T23:03:19.127 INFO:teuthology.orchestra.run.smithi142.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-31T23:03:19.127 INFO:teuthology.orchestra.run.smithi142.stdout:Context: system_u:object_r:device_t:s0 2024-03-31T23:03:19.127 INFO:teuthology.orchestra.run.smithi142.stdout:Access: 2024-03-31 23:03:02.920660126 +0000 2024-03-31T23:03:19.127 INFO:teuthology.orchestra.run.smithi142.stdout:Modify: 2024-03-31 22:57:33.251185322 +0000 2024-03-31T23:03:19.127 INFO:teuthology.orchestra.run.smithi142.stdout:Change: 2024-03-31 22:57:33.251185322 +0000 2024-03-31T23:03:19.127 INFO:teuthology.orchestra.run.smithi142.stdout: Birth: 2024-03-31 22:57:33.251185322 +0000 2024-03-31T23:03:19.128 DEBUG:teuthology.orchestra.run.smithi142:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-31T23:03:19.193 INFO:teuthology.orchestra.run.smithi142.stderr:1+0 records in 2024-03-31T23:03:19.193 INFO:teuthology.orchestra.run.smithi142.stderr:1+0 records out 2024-03-31T23:03:19.193 INFO:teuthology.orchestra.run.smithi142.stderr:512 bytes copied, 0.000235583 s, 2.2 MB/s 2024-03-31T23:03:19.194 DEBUG:teuthology.orchestra.run.smithi142:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-31T23:03:19.251 DEBUG:teuthology.orchestra.run.smithi162:> set -ex 2024-03-31T23:03:19.251 DEBUG:teuthology.orchestra.run.smithi162:> dd if=/scratch_devs of=/dev/stdout 2024-03-31T23:03:19.267 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-31T23:03:19.267 DEBUG:teuthology.orchestra.run.smithi162:> stat /dev/vg_nvme/lv_1 2024-03-31T23:03:19.296 INFO:journalctl@ceph.mgr.z.smithi162.stdout:Mar 31 23:03:19 smithi162 systemd[1]: Starting Ceph mgr.z for 6a9d8f08-efb2-11ee-b647-cb9ed24678a4... 2024-03-31T23:03:19.310 INFO:teuthology.orchestra.run.smithi162.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-31T23:03:19.310 INFO:teuthology.orchestra.run.smithi162.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-31T23:03:19.310 INFO:teuthology.orchestra.run.smithi162.stdout:Device: 5h/5d Inode: 668 Links: 1 2024-03-31T23:03:19.310 INFO:teuthology.orchestra.run.smithi162.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-31T23:03:19.310 INFO:teuthology.orchestra.run.smithi162.stdout:Context: system_u:object_r:device_t:s0 2024-03-31T23:03:19.310 INFO:teuthology.orchestra.run.smithi162.stdout:Access: 2024-03-31 23:03:19.060992574 +0000 2024-03-31T23:03:19.311 INFO:teuthology.orchestra.run.smithi162.stdout:Modify: 2024-03-31 22:57:32.215400803 +0000 2024-03-31T23:03:19.311 INFO:teuthology.orchestra.run.smithi162.stdout:Change: 2024-03-31 22:57:32.215400803 +0000 2024-03-31T23:03:19.311 INFO:teuthology.orchestra.run.smithi162.stdout: Birth: 2024-03-31 22:57:32.215400803 +0000 2024-03-31T23:03:19.311 DEBUG:teuthology.orchestra.run.smithi162:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-31T23:03:19.379 INFO:teuthology.orchestra.run.smithi162.stderr:1+0 records in 2024-03-31T23:03:19.503 INFO:teuthology.orchestra.run.smithi162.stderr:1+0 records out 2024-03-31T23:03:19.503 INFO:teuthology.orchestra.run.smithi162.stderr:512 bytes copied, 0.000196138 s, 2.6 MB/s 2024-03-31T23:03:19.504 DEBUG:teuthology.orchestra.run.smithi162:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-31T23:03:19.560 DEBUG:teuthology.orchestra.run.smithi162:> stat /dev/vg_nvme/lv_2 2024-03-31T23:03:19.597 INFO:journalctl@ceph.mgr.z.smithi162.stdout:Mar 31 23:03:19 smithi162 podman[37856]: 2024-03-31 23:03:19.24299789 +0000 UTC m=+0.017992975 image pull ec18d7baa60e8b1a0b126fbe5306acce11b1ed0ec5e3bcdddbf39b0a3c591791 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175 2024-03-31T23:03:19.597 INFO:journalctl@ceph.mgr.z.smithi162.stdout:Mar 31 23:03:19 smithi162 podman[37856]: 2024-03-31 23:03:19.379866242 +0000 UTC m=+0.154861324 container create e44587aca31bac3080bf444a7f7973106a2a3ffbb3436f17e0af74bade9ca78e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mgr-z, RELEASE=squid-384f31f, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240326, GIT_BRANCH=HEAD) 2024-03-31T23:03:19.597 INFO:journalctl@ceph.mgr.z.smithi162.stdout:Mar 31 23:03:19 smithi162 podman[37856]: 2024-03-31 23:03:19.596722366 +0000 UTC m=+0.371717449 container init e44587aca31bac3080bf444a7f7973106a2a3ffbb3436f17e0af74bade9ca78e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mgr-z, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240326, RELEASE=squid-384f31f, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0) 2024-03-31T23:03:19.613 INFO:teuthology.orchestra.run.smithi162.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-31T23:03:19.613 INFO:teuthology.orchestra.run.smithi162.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-31T23:03:19.613 INFO:teuthology.orchestra.run.smithi162.stdout:Device: 5h/5d Inode: 686 Links: 1 2024-03-31T23:03:19.613 INFO:teuthology.orchestra.run.smithi162.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-31T23:03:19.613 INFO:teuthology.orchestra.run.smithi162.stdout:Context: system_u:object_r:device_t:s0 2024-03-31T23:03:19.613 INFO:teuthology.orchestra.run.smithi162.stdout:Access: 2024-03-31 23:03:19.061992555 +0000 2024-03-31T23:03:19.613 INFO:teuthology.orchestra.run.smithi162.stdout:Modify: 2024-03-31 22:57:32.564393879 +0000 2024-03-31T23:03:19.613 INFO:teuthology.orchestra.run.smithi162.stdout:Change: 2024-03-31 22:57:32.564393879 +0000 2024-03-31T23:03:19.613 INFO:teuthology.orchestra.run.smithi162.stdout: Birth: 2024-03-31 22:57:32.564393879 +0000 2024-03-31T23:03:19.614 DEBUG:teuthology.orchestra.run.smithi162:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-31T23:03:19.681 INFO:teuthology.orchestra.run.smithi162.stderr:1+0 records in 2024-03-31T23:03:19.681 INFO:teuthology.orchestra.run.smithi162.stderr:1+0 records out 2024-03-31T23:03:19.681 INFO:teuthology.orchestra.run.smithi162.stderr:512 bytes copied, 0.000187574 s, 2.7 MB/s 2024-03-31T23:03:19.682 DEBUG:teuthology.orchestra.run.smithi162:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-31T23:03:19.738 DEBUG:teuthology.orchestra.run.smithi162:> stat /dev/vg_nvme/lv_3 2024-03-31T23:03:19.793 INFO:teuthology.orchestra.run.smithi162.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-31T23:03:19.794 INFO:teuthology.orchestra.run.smithi162.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-31T23:03:19.794 INFO:teuthology.orchestra.run.smithi162.stdout:Device: 5h/5d Inode: 706 Links: 1 2024-03-31T23:03:19.794 INFO:teuthology.orchestra.run.smithi162.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-31T23:03:19.794 INFO:teuthology.orchestra.run.smithi162.stdout:Context: system_u:object_r:device_t:s0 2024-03-31T23:03:19.794 INFO:teuthology.orchestra.run.smithi162.stdout:Access: 2024-03-31 23:03:19.061992555 +0000 2024-03-31T23:03:19.794 INFO:teuthology.orchestra.run.smithi162.stdout:Modify: 2024-03-31 22:57:32.906387092 +0000 2024-03-31T23:03:19.794 INFO:teuthology.orchestra.run.smithi162.stdout:Change: 2024-03-31 22:57:32.906387092 +0000 2024-03-31T23:03:19.794 INFO:teuthology.orchestra.run.smithi162.stdout: Birth: 2024-03-31 22:57:32.906387092 +0000 2024-03-31T23:03:19.794 DEBUG:teuthology.orchestra.run.smithi162:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-31T23:03:19.858 INFO:journalctl@ceph.mgr.z.smithi162.stdout:Mar 31 23:03:19 smithi162 podman[37856]: 2024-03-31 23:03:19.600017149 +0000 UTC m=+0.375012215 container start e44587aca31bac3080bf444a7f7973106a2a3ffbb3436f17e0af74bade9ca78e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mgr-z, RELEASE=squid-384f31f, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240326, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, ceph=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_CLEAN=True) 2024-03-31T23:03:19.858 INFO:journalctl@ceph.mgr.z.smithi162.stdout:Mar 31 23:03:19 smithi162 ceph-mgr[37953]: -- 172.21.15.162:0/467626593 <== mon.1 v2:172.21.15.162:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55b300a945a0 con 0x55b300a68800 2024-03-31T23:03:19.858 INFO:journalctl@ceph.mgr.z.smithi162.stdout:Mar 31 23:03:19 smithi162 bash[37856]: e44587aca31bac3080bf444a7f7973106a2a3ffbb3436f17e0af74bade9ca78e 2024-03-31T23:03:19.858 INFO:journalctl@ceph.mgr.z.smithi162.stdout:Mar 31 23:03:19 smithi162 systemd[1]: Started Ceph mgr.z for 6a9d8f08-efb2-11ee-b647-cb9ed24678a4. 2024-03-31T23:03:19.860 INFO:teuthology.orchestra.run.smithi162.stderr:1+0 records in 2024-03-31T23:03:19.860 INFO:teuthology.orchestra.run.smithi162.stderr:1+0 records out 2024-03-31T23:03:19.861 INFO:teuthology.orchestra.run.smithi162.stderr:512 bytes copied, 0.000173176 s, 3.0 MB/s 2024-03-31T23:03:19.862 DEBUG:teuthology.orchestra.run.smithi162:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-31T23:03:19.918 DEBUG:teuthology.orchestra.run.smithi162:> stat /dev/vg_nvme/lv_4 2024-03-31T23:03:19.975 INFO:teuthology.orchestra.run.smithi162.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-31T23:03:19.975 INFO:teuthology.orchestra.run.smithi162.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-31T23:03:19.975 INFO:teuthology.orchestra.run.smithi162.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-03-31T23:03:19.975 INFO:teuthology.orchestra.run.smithi162.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-31T23:03:19.975 INFO:teuthology.orchestra.run.smithi162.stdout:Context: system_u:object_r:device_t:s0 2024-03-31T23:03:19.975 INFO:teuthology.orchestra.run.smithi162.stdout:Access: 2024-03-31 23:03:19.061992555 +0000 2024-03-31T23:03:19.975 INFO:teuthology.orchestra.run.smithi162.stdout:Modify: 2024-03-31 22:57:33.241380444 +0000 2024-03-31T23:03:19.975 INFO:teuthology.orchestra.run.smithi162.stdout:Change: 2024-03-31 22:57:33.241380444 +0000 2024-03-31T23:03:19.975 INFO:teuthology.orchestra.run.smithi162.stdout: Birth: 2024-03-31 22:57:33.241380444 +0000 2024-03-31T23:03:19.976 DEBUG:teuthology.orchestra.run.smithi162:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-31T23:03:20.041 INFO:teuthology.orchestra.run.smithi162.stderr:1+0 records in 2024-03-31T23:03:20.041 INFO:teuthology.orchestra.run.smithi162.stderr:1+0 records out 2024-03-31T23:03:20.041 INFO:teuthology.orchestra.run.smithi162.stderr:512 bytes copied, 0.000192306 s, 2.7 MB/s 2024-03-31T23:03:20.042 DEBUG:teuthology.orchestra.run.smithi162:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-31T23:03:20.099 INFO:tasks.cephadm:Deploying osd.0 on smithi046 with /dev/vg_nvme/lv_4... 2024-03-31T23:03:20.099 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:384f31f3d3aee7f8ba3f60846b8844c049165583 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6a9d8f08-efb2-11ee-b647-cb9ed24678a4 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-03-31T23:03:20.346 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/6a9d8f08-efb2-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-31T23:03:20.545 INFO:journalctl@ceph.mgr.z.smithi162.stdout:Mar 31 23:03:20 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mgr-z[37924]: 2024-03-31T23:03:20.187+0000 7f1d2d356200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-31T23:03:20.546 INFO:journalctl@ceph.mgr.z.smithi162.stdout:Mar 31 23:03:20 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mgr-z[37924]: 2024-03-31T23:03:20.284+0000 7f1d2d356200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-31T23:03:21.046 INFO:journalctl@ceph.mgr.z.smithi162.stdout:Mar 31 23:03:20 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mgr-z[37924]: 2024-03-31T23:03:20.933+0000 7f1d2d356200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-31T23:03:21.046 INFO:journalctl@ceph.mgr.z.smithi162.stdout:Mar 31 23:03:21 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mgr-z[37924]: 2024-03-31T23:03:21.025+0000 7f1d2d356200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-31T23:03:21.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:03:20 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:03:21.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:03:20 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:03:21.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:03:20 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:03:21.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:03:20 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-31T23:03:21.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:03:20 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-31T23:03:21.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:03:20 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-31T23:03:21.047 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:03:20 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:03:21.047 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:03:20 smithi162 ceph-mon[36237]: Deploying daemon mgr.y on smithi142 2024-03-31T23:03:21.047 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:03:20 smithi162 ceph-mon[36237]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-31T23:03:21.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:03:20 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:03:21.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:03:20 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:03:21.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:03:20 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:03:21.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:03:20 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-31T23:03:21.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:03:20 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-31T23:03:21.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:03:20 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-31T23:03:21.113 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:03:20 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:03:21.113 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:03:20 smithi046 ceph-mon[32498]: Deploying daemon mgr.y on smithi142 2024-03-31T23:03:21.113 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:03:20 smithi046 ceph-mon[32498]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-31T23:03:21.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:20 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:03:21.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:20 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:03:21.194 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:20 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:03:21.194 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:20 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-31T23:03:21.194 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:20 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-31T23:03:21.194 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:20 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-31T23:03:21.194 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:20 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:03:21.194 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:20 smithi142 ceph-mon[35271]: Deploying daemon mgr.y on smithi142 2024-03-31T23:03:21.194 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:20 smithi142 ceph-mon[35271]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-31T23:03:21.546 INFO:journalctl@ceph.mgr.z.smithi162.stdout:Mar 31 23:03:21 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mgr-z[37924]: 2024-03-31T23:03:21.166+0000 7f1d2d356200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-31T23:03:21.546 INFO:journalctl@ceph.mgr.z.smithi162.stdout:Mar 31 23:03:21 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mgr-z[37924]: 2024-03-31T23:03:21.261+0000 7f1d2d356200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-31T23:03:21.546 INFO:journalctl@ceph.mgr.z.smithi162.stdout:Mar 31 23:03:21 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mgr-z[37924]: 2024-03-31T23:03:21.343+0000 7f1d2d356200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-31T23:03:21.836 INFO:journalctl@ceph.mgr.y.smithi142.stdout:Mar 31 23:03:21 smithi142 systemd[1]: Starting Ceph mgr.y for 6a9d8f08-efb2-11ee-b647-cb9ed24678a4... 2024-03-31T23:03:21.896 INFO:journalctl@ceph.mgr.z.smithi162.stdout:Mar 31 23:03:21 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mgr-z[37924]: 2024-03-31T23:03:21.604+0000 7f1d2d356200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-31T23:03:21.896 INFO:journalctl@ceph.mgr.z.smithi162.stdout:Mar 31 23:03:21 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mgr-z[37924]: 2024-03-31T23:03:21.714+0000 7f1d2d356200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-31T23:03:21.896 INFO:journalctl@ceph.mgr.z.smithi162.stdout:Mar 31 23:03:21 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mgr-z[37924]: 2024-03-31T23:03:21.800+0000 7f1d2d356200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-31T23:03:21.896 INFO:journalctl@ceph.mgr.z.smithi162.stdout:Mar 31 23:03:21 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mgr-z[37924]: 2024-03-31T23:03:21.894+0000 7f1d2d356200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-31T23:03:22.194 INFO:journalctl@ceph.mgr.y.smithi142.stdout:Mar 31 23:03:21 smithi142 podman[36648]: 2024-03-31 23:03:21.738249882 +0000 UTC m=+0.017995255 image pull ec18d7baa60e8b1a0b126fbe5306acce11b1ed0ec5e3bcdddbf39b0a3c591791 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175 2024-03-31T23:03:22.194 INFO:journalctl@ceph.mgr.y.smithi142.stdout:Mar 31 23:03:21 smithi142 podman[36648]: 2024-03-31 23:03:21.839132521 +0000 UTC m=+0.118877879 container create 1db2604d0e85c52c4ab3e717432e55e0c0f0fdf99161a4d456c4cd9dcfcc4e29 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mgr-y, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240326, org.label-schema.vendor=CentOS, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=squid-384f31f, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, io.buildah.version=1.33.5) 2024-03-31T23:03:22.194 INFO:journalctl@ceph.mgr.y.smithi142.stdout:Mar 31 23:03:22 smithi142 podman[36648]: 2024-03-31 23:03:22.008618223 +0000 UTC m=+0.288363589 container init 1db2604d0e85c52c4ab3e717432e55e0c0f0fdf99161a4d456c4cd9dcfcc4e29 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mgr-y, CEPH_POINT_RELEASE=, RELEASE=squid-384f31f, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240326, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-03-31T23:03:22.194 INFO:journalctl@ceph.mgr.y.smithi142.stdout:Mar 31 23:03:22 smithi142 podman[36648]: 2024-03-31 23:03:22.011036554 +0000 UTC m=+0.290781920 container start 1db2604d0e85c52c4ab3e717432e55e0c0f0fdf99161a4d456c4cd9dcfcc4e29 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mgr-y, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240326, RELEASE=squid-384f31f, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True) 2024-03-31T23:03:22.194 INFO:journalctl@ceph.mgr.y.smithi142.stdout:Mar 31 23:03:22 smithi142 ceph-mgr[36667]: -- 172.21.15.142:0/441016055 <== mon.2 v2:172.21.15.142:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55fe17e5a5a0 con 0x55fe17e2e800 2024-03-31T23:03:22.194 INFO:journalctl@ceph.mgr.y.smithi142.stdout:Mar 31 23:03:22 smithi142 bash[36648]: 1db2604d0e85c52c4ab3e717432e55e0c0f0fdf99161a4d456c4cd9dcfcc4e29 2024-03-31T23:03:22.194 INFO:journalctl@ceph.mgr.y.smithi142.stdout:Mar 31 23:03:22 smithi142 systemd[1]: Started Ceph mgr.y for 6a9d8f08-efb2-11ee-b647-cb9ed24678a4. 2024-03-31T23:03:22.257 INFO:teuthology.orchestra.run.smithi046.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-03-31T23:03:22.257 INFO:teuthology.orchestra.run.smithi046.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-03-31T23:03:22.257 INFO:teuthology.orchestra.run.smithi046.stderr: stderr: 10+0 records in 2024-03-31T23:03:22.257 INFO:teuthology.orchestra.run.smithi046.stderr:10+0 records out 2024-03-31T23:03:22.257 INFO:teuthology.orchestra.run.smithi046.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.00815731 s, 1.3 GB/s 2024-03-31T23:03:22.257 INFO:teuthology.orchestra.run.smithi046.stderr:--> Zapping successful for: 2024-03-31T23:03:22.295 INFO:journalctl@ceph.mgr.z.smithi162.stdout:Mar 31 23:03:22 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mgr-z[37924]: 2024-03-31T23:03:22.070+0000 7f1d2d356200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-31T23:03:22.943 INFO:journalctl@ceph.mgr.y.smithi142.stdout:Mar 31 23:03:22 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mgr-y[36663]: 2024-03-31T23:03:22.565+0000 7f9b9ed00200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-31T23:03:22.943 INFO:journalctl@ceph.mgr.y.smithi142.stdout:Mar 31 23:03:22 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mgr-y[36663]: 2024-03-31T23:03:22.660+0000 7f9b9ed00200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-31T23:03:22.979 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:384f31f3d3aee7f8ba3f60846b8844c049165583 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6a9d8f08-efb2-11ee-b647-cb9ed24678a4 -- ceph orch daemon add osd smithi046:vg_nvme/lv_4 2024-03-31T23:03:23.222 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/6a9d8f08-efb2-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-31T23:03:23.334 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:03:23 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:03:23.334 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:03:23 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:03:23.335 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:03:23 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:03:23.335 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:03:23 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:03:23.335 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:03:23 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-31T23:03:23.335 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:03:23 smithi046 ceph-mon[32498]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-31T23:03:23.433 INFO:journalctl@ceph.mgr.y.smithi142.stdout:Mar 31 23:03:23 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mgr-y[36663]: 2024-03-31T23:03:23.340+0000 7f9b9ed00200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-31T23:03:23.434 INFO:journalctl@ceph.mgr.y.smithi142.stdout:Mar 31 23:03:23 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mgr-y[36663]: 2024-03-31T23:03:23.432+0000 7f9b9ed00200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-31T23:03:23.434 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:23 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:03:23.434 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:23 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:03:23.434 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:23 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:03:23.434 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:23 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:03:23.434 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:23 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-31T23:03:23.435 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:23 smithi142 ceph-mon[35271]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-31T23:03:23.441 INFO:journalctl@ceph.mgr.z.smithi162.stdout:Mar 31 23:03:23 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mgr-z[37924]: 2024-03-31T23:03:23.101+0000 7f1d2d356200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-31T23:03:23.441 INFO:journalctl@ceph.mgr.z.smithi162.stdout:Mar 31 23:03:23 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mgr-z[37924]: 2024-03-31T23:03:23.236+0000 7f1d2d356200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-31T23:03:23.441 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:03:23 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:03:23.441 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:03:23 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:03:23.442 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:03:23 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:03:23.442 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:03:23 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:03:23.442 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:03:23 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-31T23:03:23.442 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:03:23 smithi162 ceph-mon[36237]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-31T23:03:23.693 INFO:journalctl@ceph.mgr.y.smithi142.stdout:Mar 31 23:03:23 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mgr-y[36663]: 2024-03-31T23:03:23.582+0000 7f9b9ed00200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-31T23:03:23.693 INFO:journalctl@ceph.mgr.y.smithi142.stdout:Mar 31 23:03:23 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mgr-y[36663]: 2024-03-31T23:03:23.675+0000 7f9b9ed00200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-31T23:03:23.799 INFO:journalctl@ceph.mgr.z.smithi162.stdout:Mar 31 23:03:23 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mgr-z[37924]: 2024-03-31T23:03:23.577+0000 7f1d2d356200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-31T23:03:23.799 INFO:journalctl@ceph.mgr.z.smithi162.stdout:Mar 31 23:03:23 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mgr-z[37924]: 2024-03-31T23:03:23.669+0000 7f1d2d356200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-31T23:03:24.023 INFO:journalctl@ceph.mgr.y.smithi142.stdout:Mar 31 23:03:23 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mgr-y[36663]: 2024-03-31T23:03:23.759+0000 7f9b9ed00200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-31T23:03:24.023 INFO:journalctl@ceph.mgr.y.smithi142.stdout:Mar 31 23:03:24 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mgr-y[36663]: 2024-03-31T23:03:24.022+0000 7f9b9ed00200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-31T23:03:24.296 INFO:journalctl@ceph.mgr.z.smithi162.stdout:Mar 31 23:03:23 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mgr-z[37924]: 2024-03-31T23:03:23.870+0000 7f1d2d356200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-31T23:03:24.296 INFO:journalctl@ceph.mgr.z.smithi162.stdout:Mar 31 23:03:24 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mgr-z[37924]: 2024-03-31T23:03:24.051+0000 7f1d2d356200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-31T23:03:24.309 INFO:journalctl@ceph.mgr.y.smithi142.stdout:Mar 31 23:03:24 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mgr-y[36663]: 2024-03-31T23:03:24.131+0000 7f9b9ed00200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-31T23:03:24.309 INFO:journalctl@ceph.mgr.y.smithi142.stdout:Mar 31 23:03:24 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mgr-y[36663]: 2024-03-31T23:03:24.215+0000 7f9b9ed00200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-31T23:03:24.309 INFO:journalctl@ceph.mgr.y.smithi142.stdout:Mar 31 23:03:24 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mgr-y[36663]: 2024-03-31T23:03:24.308+0000 7f9b9ed00200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-31T23:03:24.587 INFO:journalctl@ceph.mgr.z.smithi162.stdout:Mar 31 23:03:24 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mgr-z[37924]: 2024-03-31T23:03:24.327+0000 7f1d2d356200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-31T23:03:24.587 INFO:journalctl@ceph.mgr.z.smithi162.stdout:Mar 31 23:03:24 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mgr-z[37924]: 2024-03-31T23:03:24.585+0000 7f1d2d356200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-31T23:03:24.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:03:24 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:03:24.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:03:24 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:03:24.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:03:24 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:03:24.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:03:24 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:03:24.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:03:24 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:03:24.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:03:24 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:03:24.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:03:24 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-31T23:03:24.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:03:24 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:03:24.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:03:24 smithi162 ceph-mon[36237]: Reconfiguring mgr.x (unknown last config time)... 2024-03-31T23:03:24.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:03:24 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-31T23:03:24.589 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:03:24 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-31T23:03:24.589 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:03:24 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:03:24.589 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:03:24 smithi162 ceph-mon[36237]: Reconfiguring daemon mgr.x on smithi046 2024-03-31T23:03:24.589 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:03:24 smithi162 ceph-mon[36237]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-31T23:03:24.624 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:03:24.623+0000 7f7bdbcb0700 1 -- 172.21.15.46:0/1215209010 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7bd4102270 msgr2=0x7f7bd4102690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:03:24.624 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:03:24.623+0000 7f7bdbcb0700 1 --2- 172.21.15.46:0/1215209010 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7bd4102270 0x7f7bd4102690 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f7bcc0099c0 tx=0x7f7bcc009cd0 comp rx=0 tx=0).stop 2024-03-31T23:03:24.624 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:03:24.623+0000 7f7bdbcb0700 1 -- 172.21.15.46:0/1215209010 shutdown_connections 2024-03-31T23:03:24.625 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:03:24.623+0000 7f7bdbcb0700 1 --2- 172.21.15.46:0/1215209010 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7bd4102270 0x7f7bd4102690 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:03:24.625 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:03:24.623+0000 7f7bdbcb0700 1 --2- 172.21.15.46:0/1215209010 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7bd4103e10 0x7f7bd410a6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:03:24.625 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:03:24.623+0000 7f7bdbcb0700 1 --2- 172.21.15.46:0/1215209010 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7bd4103470 0x7f7bd41038d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:03:24.625 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:03:24.623+0000 7f7bdbcb0700 1 -- 172.21.15.46:0/1215209010 >> 172.21.15.46:0/1215209010 conn(0x7f7bd40fd850 msgr2=0x7f7bd40ffc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:03:24.625 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:03:24.623+0000 7f7bdbcb0700 1 -- 172.21.15.46:0/1215209010 shutdown_connections 2024-03-31T23:03:24.625 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:03:24.623+0000 7f7bdbcb0700 1 -- 172.21.15.46:0/1215209010 wait complete. 2024-03-31T23:03:24.626 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:03:24.623+0000 7f7bdbcb0700 1 Processor -- start 2024-03-31T23:03:24.627 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:03:24.624+0000 7f7bdbcb0700 1 -- start start 2024-03-31T23:03:24.627 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:03:24.624+0000 7f7bdbcb0700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7bd4102270 0x7f7bd4198470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:03:24.627 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:03:24.624+0000 7f7bdbcb0700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7bd4103470 0x7f7bd41989b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:03:24.627 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:03:24.624+0000 7f7bdbcb0700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7bd4103e10 0x7f7bd419fa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:03:24.627 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:03:24.624+0000 7f7bdbcb0700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f7bd410dd50 con 0x7f7bd4102270 2024-03-31T23:03:24.627 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:03:24.624+0000 7f7bdbcb0700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f7bd410dbd0 con 0x7f7bd4103470 2024-03-31T23:03:24.627 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:03:24.624+0000 7f7bdbcb0700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f7bd410ded0 con 0x7f7bd4103e10 2024-03-31T23:03:24.627 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:03:24.624+0000 7f7bd9a4c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7bd4102270 0x7f7bd4198470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:03:24.627 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:03:24.624+0000 7f7bd9a4c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7bd4102270 0x7f7bd4198470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:35586/0 (socket says 172.21.15.46:35586) 2024-03-31T23:03:24.627 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:03:24.624+0000 7f7bd9a4c700 1 -- 172.21.15.46:0/1674504741 learned_addr learned my addr 172.21.15.46:0/1674504741 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:03:24.627 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:03:24.624+0000 7f7bd924b700 1 --2- 172.21.15.46:0/1674504741 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7bd4103470 0x7f7bd41989b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:03:24.627 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:03:24.624+0000 7f7bda24d700 1 --2- 172.21.15.46:0/1674504741 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7bd4103e10 0x7f7bd419fa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:03:24.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:03:24.624+0000 7f7bd9a4c700 1 -- 172.21.15.46:0/1674504741 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7bd4103e10 msgr2=0x7f7bd419fa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:03:24.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:03:24.624+0000 7f7bd9a4c700 1 --2- 172.21.15.46:0/1674504741 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7bd4103e10 0x7f7bd419fa30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:03:24.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:03:24.624+0000 7f7bd9a4c700 1 -- 172.21.15.46:0/1674504741 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7bd4103470 msgr2=0x7f7bd41989b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:03:24.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:03:24.624+0000 7f7bd9a4c700 1 --2- 172.21.15.46:0/1674504741 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7bd4103470 0x7f7bd41989b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:03:24.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:03:24.624+0000 7f7bd9a4c700 1 -- 172.21.15.46:0/1674504741 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7bd4074470 con 0x7f7bd4102270 2024-03-31T23:03:24.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:03:24.624+0000 7f7bd9a4c700 1 --2- 172.21.15.46:0/1674504741 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7bd4102270 0x7f7bd4198470 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f7bcc00fa00 tx=0x7f7bcc00fa30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:03:24.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:03:24.624+0000 7f7bd2ffd700 1 -- 172.21.15.46:0/1674504741 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bcc027070 con 0x7f7bd4102270 2024-03-31T23:03:24.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:03:24.625+0000 7f7bd2ffd700 1 -- 172.21.15.46:0/1674504741 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7bcc022920 con 0x7f7bd4102270 2024-03-31T23:03:24.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:03:24.625+0000 7f7bdbcb0700 1 -- 172.21.15.46:0/1674504741 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7bd4074700 con 0x7f7bd4102270 2024-03-31T23:03:24.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:03:24.625+0000 7f7bd2ffd700 1 -- 172.21.15.46:0/1674504741 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bcc00fe60 con 0x7f7bd4102270 2024-03-31T23:03:24.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:03:24.625+0000 7f7bdbcb0700 1 -- 172.21.15.46:0/1674504741 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7bd4074c30 con 0x7f7bd4102270 2024-03-31T23:03:24.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:03:24.625+0000 7f7bd2ffd700 1 -- 172.21.15.46:0/1674504741 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 13) ==== 46172+0+0 (secure 0 0 0) 0x7f7bcc02a070 con 0x7f7bd4102270 2024-03-31T23:03:24.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:03:24.626+0000 7f7bd2ffd700 1 --2- 172.21.15.46:0/1674504741 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f7bbc0390e0 0x7f7bbc03b5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:03:24.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:03:24.626+0000 7f7bd2ffd700 1 -- 172.21.15.46:0/1674504741 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f7bcc04cc70 con 0x7f7bd4102270 2024-03-31T23:03:24.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:03:24.626+0000 7f7bdbcb0700 1 -- 172.21.15.46:0/1674504741 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7bd4068f00 con 0x7f7bd4102270 2024-03-31T23:03:24.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:03:24.626+0000 7f7bd924b700 1 --2- 172.21.15.46:0/1674504741 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f7bbc0390e0 0x7f7bbc03b5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:03:24.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:03:24.626+0000 7f7bd924b700 1 --2- 172.21.15.46:0/1674504741 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f7bbc0390e0 0x7f7bbc03b5a0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f7bc8009c60 tx=0x7f7bc8009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:03:24.632 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:03:24.631+0000 7f7bd2ffd700 1 -- 172.21.15.46:0/1674504741 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f7bcc013070 con 0x7f7bd4102270 2024-03-31T23:03:24.693 INFO:journalctl@ceph.mgr.y.smithi142.stdout:Mar 31 23:03:24 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mgr-y[36663]: 2024-03-31T23:03:24.483+0000 7f9b9ed00200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-31T23:03:24.693 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:24 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:03:24.694 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:24 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:03:24.694 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:24 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:03:24.694 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:24 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:03:24.694 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:24 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:03:24.694 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:24 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:03:24.694 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:24 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-31T23:03:24.694 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:24 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:03:24.694 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:24 smithi142 ceph-mon[35271]: Reconfiguring mgr.x (unknown last config time)... 2024-03-31T23:03:24.694 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:24 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-31T23:03:24.694 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:24 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-31T23:03:24.694 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:24 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:03:24.694 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:24 smithi142 ceph-mon[35271]: Reconfiguring daemon mgr.x on smithi046 2024-03-31T23:03:24.694 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:24 smithi142 ceph-mon[35271]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-31T23:03:24.718 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:03:24 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:03:24.718 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:03:24 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:03:24.718 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:03:24 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:03:24.719 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:03:24 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:03:24.719 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:03:24 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:03:24.719 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:03:24 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:03:24.719 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:03:24 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-31T23:03:24.719 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:03:24 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:03:24.719 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:03:24 smithi046 ceph-mon[32498]: Reconfiguring mgr.x (unknown last config time)... 2024-03-31T23:03:24.719 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:03:24 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-31T23:03:24.719 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:03:24 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-31T23:03:24.719 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:03:24 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:03:24.720 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:03:24 smithi046 ceph-mon[32498]: Reconfiguring daemon mgr.x on smithi046 2024-03-31T23:03:24.720 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:03:24 smithi046 ceph-mon[32498]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-31T23:03:24.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:03:24.850+0000 7f7bdbcb0700 1 -- 172.21.15.46:0/1674504741 --> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi046:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7f7bd4062680 con 0x7f7bbc0390e0 2024-03-31T23:03:25.046 INFO:journalctl@ceph.mgr.z.smithi162.stdout:Mar 31 23:03:24 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mgr-z[37924]: 2024-03-31T23:03:24.819+0000 7f1d2d356200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-31T23:03:25.693 INFO:journalctl@ceph.mgr.y.smithi142.stdout:Mar 31 23:03:25 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mgr-y[36663]: 2024-03-31T23:03:25.440+0000 7f9b9ed00200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-31T23:03:25.693 INFO:journalctl@ceph.mgr.y.smithi142.stdout:Mar 31 23:03:25 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mgr-y[36663]: 2024-03-31T23:03:25.565+0000 7f9b9ed00200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-31T23:03:25.987 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:25 smithi142 ceph-mon[35271]: from='client.14232 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi046:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-31T23:03:25.987 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:25 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-31T23:03:25.987 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:25 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-31T23:03:25.987 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:25 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:03:25.988 INFO:journalctl@ceph.mgr.y.smithi142.stdout:Mar 31 23:03:25 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mgr-y[36663]: 2024-03-31T23:03:25.900+0000 7f9b9ed00200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-31T23:03:25.988 INFO:journalctl@ceph.mgr.y.smithi142.stdout:Mar 31 23:03:25 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mgr-y[36663]: 2024-03-31T23:03:25.986+0000 7f9b9ed00200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-31T23:03:25.999 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:03:25 smithi046 ceph-mon[32498]: from='client.14232 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi046:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-31T23:03:25.999 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:03:25 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-31T23:03:25.999 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:03:25 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-31T23:03:25.999 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:03:25 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:03:26.046 INFO:journalctl@ceph.mgr.z.smithi162.stdout:Mar 31 23:03:25 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mgr-z[37924]: 2024-03-31T23:03:25.606+0000 7f1d2d356200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-31T23:03:26.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:03:25 smithi162 ceph-mon[36237]: from='client.14232 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi046:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-31T23:03:26.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:03:25 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-31T23:03:26.047 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:03:25 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-31T23:03:26.047 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:03:25 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:03:26.356 INFO:journalctl@ceph.mgr.y.smithi142.stdout:Mar 31 23:03:26 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mgr-y[36663]: 2024-03-31T23:03:26.176+0000 7f9b9ed00200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-31T23:03:26.356 INFO:journalctl@ceph.mgr.y.smithi142.stdout:Mar 31 23:03:26 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mgr-y[36663]: 2024-03-31T23:03:26.355+0000 7f9b9ed00200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-31T23:03:26.480 INFO:journalctl@ceph.mgr.z.smithi162.stdout:Mar 31 23:03:26 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mgr-z[37924]: 2024-03-31T23:03:26.173+0000 7f1d2d356200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-31T23:03:26.480 INFO:journalctl@ceph.mgr.z.smithi162.stdout:Mar 31 23:03:26 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mgr-z[37924]: 2024-03-31T23:03:26.268+0000 7f1d2d356200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-31T23:03:26.480 INFO:journalctl@ceph.mgr.z.smithi162.stdout:Mar 31 23:03:26 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mgr-z[37924]: 2024-03-31T23:03:26.478+0000 7f1d2d356200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-31T23:03:26.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:03:26.821+0000 7f7bd2ffd700 1 -- 172.21.15.46:0/1674504741 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mgrmap(e 14) ==== 91718+0+0 (secure 0 0 0) 0x7f7bcc017650 con 0x7f7bd4102270 2024-03-31T23:03:26.893 INFO:journalctl@ceph.mgr.y.smithi142.stdout:Mar 31 23:03:26 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mgr-y[36663]: 2024-03-31T23:03:26.631+0000 7f9b9ed00200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-31T23:03:26.893 INFO:journalctl@ceph.mgr.y.smithi142.stdout:Mar 31 23:03:26 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mgr-y[36663]: 2024-03-31T23:03:26.891+0000 7f9b9ed00200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-31T23:03:26.893 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:26 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:03:26.893 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:26 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:03:26.893 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:26 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-31T23:03:26.893 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:26 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:03:26.893 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:26 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-31T23:03:26.893 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:26 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:03:26.894 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:26 smithi142 ceph-mon[35271]: Standby manager daemon z started 2024-03-31T23:03:26.894 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:26 smithi142 ceph-mon[35271]: from='mgr.? 172.21.15.162:0/2060553565' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-31T23:03:26.894 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:26 smithi142 ceph-mon[35271]: from='mgr.? 172.21.15.162:0/2060553565' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-31T23:03:26.894 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:26 smithi142 ceph-mon[35271]: from='mgr.? 172.21.15.162:0/2060553565' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-31T23:03:26.894 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:26 smithi142 ceph-mon[35271]: from='mgr.? 172.21.15.162:0/2060553565' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-31T23:03:26.894 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:26 smithi142 ceph-mon[35271]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-31T23:03:27.042 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:03:26 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:03:27.042 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:03:26 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:03:27.042 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:03:26 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-31T23:03:27.042 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:03:26 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:03:27.042 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:03:26 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-31T23:03:27.043 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:03:26 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:03:27.043 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:03:26 smithi046 ceph-mon[32498]: Standby manager daemon z started 2024-03-31T23:03:27.043 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:03:26 smithi046 ceph-mon[32498]: from='mgr.? 172.21.15.162:0/2060553565' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-31T23:03:27.043 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:03:26 smithi046 ceph-mon[32498]: from='mgr.? 172.21.15.162:0/2060553565' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-31T23:03:27.043 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:03:26 smithi046 ceph-mon[32498]: from='mgr.? 172.21.15.162:0/2060553565' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-31T23:03:27.043 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:03:26 smithi046 ceph-mon[32498]: from='mgr.? 172.21.15.162:0/2060553565' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-31T23:03:27.043 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:03:26 smithi046 ceph-mon[32498]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-31T23:03:27.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:03:26 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:03:27.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:03:26 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:03:27.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:03:26 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-31T23:03:27.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:03:26 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:03:27.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:03:26 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-31T23:03:27.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:03:26 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:03:27.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:03:26 smithi162 ceph-mon[36237]: Standby manager daemon z started 2024-03-31T23:03:27.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:03:26 smithi162 ceph-mon[36237]: from='mgr.? 172.21.15.162:0/2060553565' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-31T23:03:27.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:03:26 smithi162 ceph-mon[36237]: from='mgr.? 172.21.15.162:0/2060553565' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-31T23:03:27.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:03:26 smithi162 ceph-mon[36237]: from='mgr.? 172.21.15.162:0/2060553565' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-31T23:03:27.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:03:26 smithi162 ceph-mon[36237]: from='mgr.? 172.21.15.162:0/2060553565' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-31T23:03:27.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:03:26 smithi162 ceph-mon[36237]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-31T23:03:27.193 INFO:journalctl@ceph.mgr.y.smithi142.stdout:Mar 31 23:03:27 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mgr-y[36663]: 2024-03-31T23:03:27.123+0000 7f9b9ed00200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-31T23:03:28.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:03:27 smithi046 ceph-mon[32498]: mgrmap e14: x(active, since 78s), standbys: z 2024-03-31T23:03:28.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:03:27 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-31T23:03:28.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:27 smithi142 ceph-mon[35271]: mgrmap e14: x(active, since 78s), standbys: z 2024-03-31T23:03:28.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:27 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-31T23:03:28.193 INFO:journalctl@ceph.mgr.y.smithi142.stdout:Mar 31 23:03:27 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mgr-y[36663]: 2024-03-31T23:03:27.905+0000 7f9b9ed00200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-31T23:03:28.296 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:03:27 smithi162 ceph-mon[36237]: mgrmap e14: x(active, since 78s), standbys: z 2024-03-31T23:03:28.296 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:03:27 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-31T23:03:28.770 INFO:journalctl@ceph.mgr.y.smithi142.stdout:Mar 31 23:03:28 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mgr-y[36663]: 2024-03-31T23:03:28.464+0000 7f9b9ed00200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-31T23:03:28.771 INFO:journalctl@ceph.mgr.y.smithi142.stdout:Mar 31 23:03:28 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mgr-y[36663]: 2024-03-31T23:03:28.560+0000 7f9b9ed00200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-31T23:03:28.867 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:03:28.866+0000 7f7bd2ffd700 1 -- 172.21.15.46:0/1674504741 <== mon.0 v2:172.21.15.46:3300/0 8 ==== mgrmap(e 15) ==== 137264+0+0 (secure 0 0 0) 0x7f7bcc02abf0 con 0x7f7bd4102270 2024-03-31T23:03:29.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:03:28 smithi046 ceph-mon[32498]: from='client.? 172.21.15.46:0/1236567618' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "47429095-3f46-4ed7-b7c4-504464719815"}]: dispatch 2024-03-31T23:03:29.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:03:28 smithi046 ceph-mon[32498]: from='client.? 172.21.15.46:0/1236567618' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "47429095-3f46-4ed7-b7c4-504464719815"}]': finished 2024-03-31T23:03:29.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:03:28 smithi046 ceph-mon[32498]: osdmap e5: 1 total, 0 up, 1 in 2024-03-31T23:03:29.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:03:28 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-31T23:03:29.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:03:28 smithi046 ceph-mon[32498]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-31T23:03:29.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:03:28 smithi046 ceph-mon[32498]: from='client.? 172.21.15.46:0/3151630995' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-31T23:03:29.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:03:28 smithi046 ceph-mon[32498]: Standby manager daemon y started 2024-03-31T23:03:29.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:03:28 smithi046 ceph-mon[32498]: from='mgr.? 172.21.15.142:0/1761466707' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-31T23:03:29.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:03:28 smithi046 ceph-mon[32498]: from='mgr.? 172.21.15.142:0/1761466707' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-31T23:03:29.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:03:28 smithi046 ceph-mon[32498]: from='mgr.? 172.21.15.142:0/1761466707' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-31T23:03:29.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:03:28 smithi046 ceph-mon[32498]: from='mgr.? 172.21.15.142:0/1761466707' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-31T23:03:29.193 INFO:journalctl@ceph.mgr.y.smithi142.stdout:Mar 31 23:03:28 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mgr-y[36663]: 2024-03-31T23:03:28.769+0000 7f9b9ed00200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-31T23:03:29.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:28 smithi142 ceph-mon[35271]: from='client.? 172.21.15.46:0/1236567618' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "47429095-3f46-4ed7-b7c4-504464719815"}]: dispatch 2024-03-31T23:03:29.194 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:28 smithi142 ceph-mon[35271]: from='client.? 172.21.15.46:0/1236567618' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "47429095-3f46-4ed7-b7c4-504464719815"}]': finished 2024-03-31T23:03:29.194 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:28 smithi142 ceph-mon[35271]: osdmap e5: 1 total, 0 up, 1 in 2024-03-31T23:03:29.194 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:28 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-31T23:03:29.194 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:28 smithi142 ceph-mon[35271]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-31T23:03:29.194 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:28 smithi142 ceph-mon[35271]: from='client.? 172.21.15.46:0/3151630995' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-31T23:03:29.194 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:28 smithi142 ceph-mon[35271]: Standby manager daemon y started 2024-03-31T23:03:29.194 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:28 smithi142 ceph-mon[35271]: from='mgr.? 172.21.15.142:0/1761466707' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-31T23:03:29.194 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:28 smithi142 ceph-mon[35271]: from='mgr.? 172.21.15.142:0/1761466707' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-31T23:03:29.194 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:28 smithi142 ceph-mon[35271]: from='mgr.? 172.21.15.142:0/1761466707' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-31T23:03:29.194 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:28 smithi142 ceph-mon[35271]: from='mgr.? 172.21.15.142:0/1761466707' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-31T23:03:29.296 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:03:28 smithi162 ceph-mon[36237]: from='client.? 172.21.15.46:0/1236567618' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "47429095-3f46-4ed7-b7c4-504464719815"}]: dispatch 2024-03-31T23:03:29.296 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:03:28 smithi162 ceph-mon[36237]: from='client.? 172.21.15.46:0/1236567618' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "47429095-3f46-4ed7-b7c4-504464719815"}]': finished 2024-03-31T23:03:29.296 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:03:28 smithi162 ceph-mon[36237]: osdmap e5: 1 total, 0 up, 1 in 2024-03-31T23:03:29.296 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:03:28 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-31T23:03:29.296 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:03:28 smithi162 ceph-mon[36237]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-31T23:03:29.296 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:03:28 smithi162 ceph-mon[36237]: from='client.? 172.21.15.46:0/3151630995' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-31T23:03:29.296 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:03:28 smithi162 ceph-mon[36237]: Standby manager daemon y started 2024-03-31T23:03:29.296 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:03:28 smithi162 ceph-mon[36237]: from='mgr.? 172.21.15.142:0/1761466707' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-31T23:03:29.296 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:03:28 smithi162 ceph-mon[36237]: from='mgr.? 172.21.15.142:0/1761466707' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-31T23:03:29.296 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:03:28 smithi162 ceph-mon[36237]: from='mgr.? 172.21.15.142:0/1761466707' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-31T23:03:29.297 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:03:28 smithi162 ceph-mon[36237]: from='mgr.? 172.21.15.142:0/1761466707' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-31T23:03:29.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:29 smithi142 ceph-mon[35271]: mgrmap e15: x(active, since 80s), standbys: y, z 2024-03-31T23:03:29.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:29 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-31T23:03:30.296 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:03:29 smithi162 ceph-mon[36237]: mgrmap e15: x(active, since 80s), standbys: y, z 2024-03-31T23:03:30.296 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:03:29 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-31T23:03:30.361 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:03:29 smithi046 ceph-mon[32498]: mgrmap e15: x(active, since 80s), standbys: y, z 2024-03-31T23:03:30.362 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:03:29 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-31T23:03:31.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:30 smithi142 ceph-mon[35271]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-31T23:03:31.295 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:03:30 smithi162 ceph-mon[36237]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-31T23:03:31.361 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:03:30 smithi046 ceph-mon[32498]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-31T23:03:32.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:03:32 smithi046 ceph-mon[32498]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-31T23:03:32.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:32 smithi142 ceph-mon[35271]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-31T23:03:33.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:03:32 smithi162 ceph-mon[36237]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-31T23:03:34.612 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:03:34 smithi046 ceph-mon[32498]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-31T23:03:34.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:34 smithi142 ceph-mon[35271]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-31T23:03:35.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:03:34 smithi162 ceph-mon[36237]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-31T23:03:35.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:03:35 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-31T23:03:35.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:03:35 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:03:35.862 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:03:35 smithi046 ceph-mon[32498]: Deploying daemon osd.0 on smithi046 2024-03-31T23:03:35.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:35 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-31T23:03:35.944 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:35 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:03:35.944 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:35 smithi142 ceph-mon[35271]: Deploying daemon osd.0 on smithi046 2024-03-31T23:03:36.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:03:35 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-31T23:03:36.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:03:35 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:03:36.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:03:35 smithi162 ceph-mon[36237]: Deploying daemon osd.0 on smithi046 2024-03-31T23:03:36.710 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:03:36 smithi046 ceph-mon[32498]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-31T23:03:36.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:36 smithi142 ceph-mon[35271]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-31T23:03:37.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:03:36 smithi162 ceph-mon[36237]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-31T23:03:38.658 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:03:38 smithi046 ceph-mon[32498]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-31T23:03:38.946 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:38 smithi142 ceph-mon[35271]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-31T23:03:39.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:03:38 smithi162 ceph-mon[36237]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-31T23:03:41.409 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:03:41 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:03:41.409 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:03:41 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-31T23:03:41.410 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:03:41 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:03:41.410 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:03:41 smithi046 ceph-mon[32498]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-31T23:03:41.443 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:41 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:03:41.443 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:41 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-31T23:03:41.443 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:41 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:03:41.444 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:41 smithi142 ceph-mon[35271]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-31T23:03:41.546 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:03:41 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:03:41.546 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:03:41 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-31T23:03:41.546 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:03:41 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:03:41.546 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:03:41 smithi162 ceph-mon[36237]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-31T23:03:42.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:03:42 smithi046 ceph-mon[32498]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-31T23:03:42.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:42 smithi142 ceph-mon[35271]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-31T23:03:43.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:03:42 smithi162 ceph-mon[36237]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-31T23:03:43.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:03:43.420+0000 7f7bd2ffd700 1 -- 172.21.15.46:0/1674504741 <== mgr.14152 v2:172.21.15.46:6800/162289569 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f7bd4062680 con 0x7f7bbc0390e0 2024-03-31T23:03:43.421 INFO:teuthology.orchestra.run.smithi046.stdout:Created osd(s) 0 on host 'smithi046' 2024-03-31T23:03:43.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:03:43.422+0000 7f7bdbcb0700 1 -- 172.21.15.46:0/1674504741 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f7bbc0390e0 msgr2=0x7f7bbc03b5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:03:43.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:03:43.422+0000 7f7bdbcb0700 1 --2- 172.21.15.46:0/1674504741 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f7bbc0390e0 0x7f7bbc03b5a0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f7bc8009c60 tx=0x7f7bc8009260 comp rx=0 tx=0).stop 2024-03-31T23:03:43.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:03:43.422+0000 7f7bdbcb0700 1 -- 172.21.15.46:0/1674504741 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7bd4102270 msgr2=0x7f7bd4198470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:03:43.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:03:43.422+0000 7f7bdbcb0700 1 --2- 172.21.15.46:0/1674504741 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7bd4102270 0x7f7bd4198470 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f7bcc00fa00 tx=0x7f7bcc00fa30 comp rx=0 tx=0).stop 2024-03-31T23:03:43.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:03:43.422+0000 7f7bdbcb0700 1 -- 172.21.15.46:0/1674504741 shutdown_connections 2024-03-31T23:03:43.424 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:03:43.422+0000 7f7bdbcb0700 1 --2- 172.21.15.46:0/1674504741 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f7bbc0390e0 0x7f7bbc03b5a0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:03:43.424 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:03:43.422+0000 7f7bdbcb0700 1 --2- 172.21.15.46:0/1674504741 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7bd4102270 0x7f7bd4198470 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:03:43.424 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:03:43.422+0000 7f7bdbcb0700 1 --2- 172.21.15.46:0/1674504741 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7bd4103e10 0x7f7bd419fa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:03:43.424 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:03:43.422+0000 7f7bdbcb0700 1 --2- 172.21.15.46:0/1674504741 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7bd4103470 0x7f7bd41989b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:03:43.424 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:03:43.422+0000 7f7bdbcb0700 1 -- 172.21.15.46:0/1674504741 >> 172.21.15.46:0/1674504741 conn(0x7f7bd40fd850 msgr2=0x7f7bd4106890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:03:43.424 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:03:43.422+0000 7f7bdbcb0700 1 -- 172.21.15.46:0/1674504741 shutdown_connections 2024-03-31T23:03:43.424 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:03:43.422+0000 7f7bdbcb0700 1 -- 172.21.15.46:0/1674504741 wait complete. 2024-03-31T23:03:43.945 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:03:43 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:03:43.945 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:03:43 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:03:43.945 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:03:43 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:03:43.945 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:03:43 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-31T23:03:43.945 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:03:43 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:03:43.945 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:03:43 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-31T23:03:43.945 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:03:43 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:03:43.945 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:03:43 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:03:43.968 DEBUG:teuthology.orchestra.run.smithi046:osd.0> sudo journalctl -f -n 0 -u ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4@osd.0.service 2024-03-31T23:03:43.971 INFO:tasks.cephadm:Deploying osd.1 on smithi142 with /dev/vg_nvme/lv_4... 2024-03-31T23:03:43.971 DEBUG:teuthology.orchestra.run.smithi142:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:384f31f3d3aee7f8ba3f60846b8844c049165583 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6a9d8f08-efb2-11ee-b647-cb9ed24678a4 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-03-31T23:03:44.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:43 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:03:44.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:43 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:03:44.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:43 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:03:44.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:43 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-31T23:03:44.194 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:43 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:03:44.194 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:43 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-31T23:03:44.194 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:43 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:03:44.194 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:43 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:03:44.214 INFO:teuthology.orchestra.run.smithi142.stderr:Inferring config /var/lib/ceph/6a9d8f08-efb2-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-31T23:03:44.270 INFO:journalctl@ceph.osd.0.smithi046.stdout:Mar 31 23:03:44 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-0[47281]: 2024-03-31T23:03:44.026+0000 7f9fcb1587c0 -1 osd.0 0 log_to_monitors true 2024-03-31T23:03:44.296 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:03:43 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:03:44.296 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:03:43 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:03:44.296 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:03:43 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:03:44.296 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:03:43 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-31T23:03:44.297 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:03:43 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:03:44.297 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:03:43 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-31T23:03:44.297 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:03:43 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:03:44.297 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:03:43 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:03:44.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:44 smithi142 ceph-mon[35271]: from='osd.0 [v2:172.21.15.46:6802/1824772695,v1:172.21.15.46:6803/1824772695]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-31T23:03:44.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:44 smithi142 ceph-mon[35271]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-31T23:03:45.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:03:44 smithi046 ceph-mon[32498]: from='osd.0 [v2:172.21.15.46:6802/1824772695,v1:172.21.15.46:6803/1824772695]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-31T23:03:45.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:03:44 smithi046 ceph-mon[32498]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-31T23:03:45.296 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:03:44 smithi162 ceph-mon[36237]: from='osd.0 [v2:172.21.15.46:6802/1824772695,v1:172.21.15.46:6803/1824772695]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-31T23:03:45.296 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:03:44 smithi162 ceph-mon[36237]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-31T23:03:45.902 INFO:teuthology.orchestra.run.smithi142.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-03-31T23:03:45.902 INFO:teuthology.orchestra.run.smithi142.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-03-31T23:03:45.902 INFO:teuthology.orchestra.run.smithi142.stderr: stderr: 10+0 records in 2024-03-31T23:03:45.902 INFO:teuthology.orchestra.run.smithi142.stderr:10+0 records out 2024-03-31T23:03:45.902 INFO:teuthology.orchestra.run.smithi142.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0112647 s, 931 MB/s 2024-03-31T23:03:45.903 INFO:teuthology.orchestra.run.smithi142.stderr:--> Zapping successful for: 2024-03-31T23:03:46.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:03:45 smithi046 ceph-mon[32498]: from='osd.0 [v2:172.21.15.46:6802/1824772695,v1:172.21.15.46:6803/1824772695]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-31T23:03:46.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:03:45 smithi046 ceph-mon[32498]: osdmap e6: 1 total, 0 up, 1 in 2024-03-31T23:03:46.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:03:45 smithi046 ceph-mon[32498]: from='osd.0 [v2:172.21.15.46:6802/1824772695,v1:172.21.15.46:6803/1824772695]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi046", "root=default"]}]: dispatch 2024-03-31T23:03:46.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:03:45 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-31T23:03:46.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:45 smithi142 ceph-mon[35271]: from='osd.0 [v2:172.21.15.46:6802/1824772695,v1:172.21.15.46:6803/1824772695]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-31T23:03:46.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:45 smithi142 ceph-mon[35271]: osdmap e6: 1 total, 0 up, 1 in 2024-03-31T23:03:46.194 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:45 smithi142 ceph-mon[35271]: from='osd.0 [v2:172.21.15.46:6802/1824772695,v1:172.21.15.46:6803/1824772695]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi046", "root=default"]}]: dispatch 2024-03-31T23:03:46.194 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:45 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-31T23:03:46.295 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:03:45 smithi162 ceph-mon[36237]: from='osd.0 [v2:172.21.15.46:6802/1824772695,v1:172.21.15.46:6803/1824772695]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-31T23:03:46.296 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:03:45 smithi162 ceph-mon[36237]: osdmap e6: 1 total, 0 up, 1 in 2024-03-31T23:03:46.296 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:03:45 smithi162 ceph-mon[36237]: from='osd.0 [v2:172.21.15.46:6802/1824772695,v1:172.21.15.46:6803/1824772695]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi046", "root=default"]}]: dispatch 2024-03-31T23:03:46.296 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:03:45 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-31T23:03:46.465 DEBUG:teuthology.orchestra.run.smithi142:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:384f31f3d3aee7f8ba3f60846b8844c049165583 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6a9d8f08-efb2-11ee-b647-cb9ed24678a4 -- ceph orch daemon add osd smithi142:vg_nvme/lv_4 2024-03-31T23:03:46.712 INFO:teuthology.orchestra.run.smithi142.stderr:Inferring config /var/lib/ceph/6a9d8f08-efb2-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-31T23:03:46.773 INFO:journalctl@ceph.osd.0.smithi046.stdout:Mar 31 23:03:46 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-0[47281]: 2024-03-31T23:03:46.425+0000 7f9fc07bd700 -1 osd.0 0 waiting for initial osdmap 2024-03-31T23:03:46.773 INFO:journalctl@ceph.osd.0.smithi046.stdout:Mar 31 23:03:46 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-0[47281]: 2024-03-31T23:03:46.429+0000 7f9fb85a8700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-31T23:03:47.061 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:46 smithi142 ceph-mon[35271]: from='osd.0 [v2:172.21.15.46:6802/1824772695,v1:172.21.15.46:6803/1824772695]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi046", "root=default"]}]': finished 2024-03-31T23:03:47.061 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:46 smithi142 ceph-mon[35271]: osdmap e7: 1 total, 0 up, 1 in 2024-03-31T23:03:47.062 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:46 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-31T23:03:47.062 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:46 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-31T23:03:47.062 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:46 smithi142 ceph-mon[35271]: from='osd.0 [v2:172.21.15.46:6802/1824772695,v1:172.21.15.46:6803/1824772695]' entity='osd.0' 2024-03-31T23:03:47.062 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:46 smithi142 ceph-mon[35271]: pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-31T23:03:47.062 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:46 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-31T23:03:47.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:03:46 smithi046 ceph-mon[32498]: from='osd.0 [v2:172.21.15.46:6802/1824772695,v1:172.21.15.46:6803/1824772695]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi046", "root=default"]}]': finished 2024-03-31T23:03:47.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:03:46 smithi046 ceph-mon[32498]: osdmap e7: 1 total, 0 up, 1 in 2024-03-31T23:03:47.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:03:46 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-31T23:03:47.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:03:46 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-31T23:03:47.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:03:46 smithi046 ceph-mon[32498]: from='osd.0 [v2:172.21.15.46:6802/1824772695,v1:172.21.15.46:6803/1824772695]' entity='osd.0' 2024-03-31T23:03:47.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:03:46 smithi046 ceph-mon[32498]: pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-31T23:03:47.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:03:46 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-31T23:03:47.296 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:03:46 smithi162 ceph-mon[36237]: from='osd.0 [v2:172.21.15.46:6802/1824772695,v1:172.21.15.46:6803/1824772695]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi046", "root=default"]}]': finished 2024-03-31T23:03:47.296 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:03:46 smithi162 ceph-mon[36237]: osdmap e7: 1 total, 0 up, 1 in 2024-03-31T23:03:47.296 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:03:46 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-31T23:03:47.296 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:03:46 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-31T23:03:47.297 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:03:46 smithi162 ceph-mon[36237]: from='osd.0 [v2:172.21.15.46:6802/1824772695,v1:172.21.15.46:6803/1824772695]' entity='osd.0' 2024-03-31T23:03:47.297 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:03:46 smithi162 ceph-mon[36237]: pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-31T23:03:47.297 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:03:46 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-31T23:03:48.049 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:47 smithi142 ceph-mon[35271]: purged_snaps scrub starts 2024-03-31T23:03:48.049 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:47 smithi142 ceph-mon[35271]: purged_snaps scrub ok 2024-03-31T23:03:48.049 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:47 smithi142 ceph-mon[35271]: Detected new or changed devices on smithi046 2024-03-31T23:03:48.049 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:47 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:03:48.050 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:47 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:03:48.050 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:47 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-31T23:03:48.050 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:47 smithi142 ceph-mon[35271]: Adjusting osd_memory_target on smithi046 to 17093M 2024-03-31T23:03:48.050 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:47 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:03:48.050 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:47 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:03:48.050 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:47 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-31T23:03:48.050 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:47 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:03:48.050 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:47 smithi142 ceph-mon[35271]: osd.0 [v2:172.21.15.46:6802/1824772695,v1:172.21.15.46:6803/1824772695] boot 2024-03-31T23:03:48.050 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:47 smithi142 ceph-mon[35271]: osdmap e8: 1 total, 1 up, 1 in 2024-03-31T23:03:48.050 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:47 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-31T23:03:48.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:03:47 smithi046 ceph-mon[32498]: purged_snaps scrub starts 2024-03-31T23:03:48.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:03:47 smithi046 ceph-mon[32498]: purged_snaps scrub ok 2024-03-31T23:03:48.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:03:47 smithi046 ceph-mon[32498]: Detected new or changed devices on smithi046 2024-03-31T23:03:48.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:03:47 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:03:48.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:03:47 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:03:48.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:03:47 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-31T23:03:48.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:03:47 smithi046 ceph-mon[32498]: Adjusting osd_memory_target on smithi046 to 17093M 2024-03-31T23:03:48.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:03:47 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:03:48.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:03:47 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:03:48.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:03:47 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-31T23:03:48.113 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:03:47 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:03:48.113 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:03:47 smithi046 ceph-mon[32498]: osd.0 [v2:172.21.15.46:6802/1824772695,v1:172.21.15.46:6803/1824772695] boot 2024-03-31T23:03:48.113 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:03:47 smithi046 ceph-mon[32498]: osdmap e8: 1 total, 1 up, 1 in 2024-03-31T23:03:48.113 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:03:47 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-31T23:03:48.197 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:03:47 smithi162 ceph-mon[36237]: purged_snaps scrub starts 2024-03-31T23:03:48.197 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:03:47 smithi162 ceph-mon[36237]: purged_snaps scrub ok 2024-03-31T23:03:48.197 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:03:47 smithi162 ceph-mon[36237]: Detected new or changed devices on smithi046 2024-03-31T23:03:48.198 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:03:47 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:03:48.198 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:03:47 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:03:48.198 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:03:47 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-31T23:03:48.198 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:03:47 smithi162 ceph-mon[36237]: Adjusting osd_memory_target on smithi046 to 17093M 2024-03-31T23:03:48.198 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:03:47 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:03:48.198 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:03:47 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:03:48.198 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:03:47 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-31T23:03:48.198 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:03:47 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:03:48.198 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:03:47 smithi162 ceph-mon[36237]: osd.0 [v2:172.21.15.46:6802/1824772695,v1:172.21.15.46:6803/1824772695] boot 2024-03-31T23:03:48.198 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:03:47 smithi162 ceph-mon[36237]: osdmap e8: 1 total, 1 up, 1 in 2024-03-31T23:03:48.198 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:03:47 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-31T23:03:48.240 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-31T23:03:48.236+0000 7f1b66094700 1 -- 172.21.15.142:0/555156461 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1b60102370 msgr2=0x7f1b601027d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:03:48.240 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-31T23:03:48.236+0000 7f1b66094700 1 --2- 172.21.15.142:0/555156461 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1b60102370 0x7f1b601027d0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f1b480099c0 tx=0x7f1b48009cd0 comp rx=0 tx=0).stop 2024-03-31T23:03:48.241 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-31T23:03:48.236+0000 7f1b66094700 1 -- 172.21.15.142:0/555156461 shutdown_connections 2024-03-31T23:03:48.241 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-31T23:03:48.236+0000 7f1b66094700 1 --2- 172.21.15.142:0/555156461 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1b60102d10 0x7f1b601095c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:03:48.241 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-31T23:03:48.236+0000 7f1b66094700 1 --2- 172.21.15.142:0/555156461 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1b60101170 0x7f1b60101590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:03:48.241 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-31T23:03:48.236+0000 7f1b66094700 1 --2- 172.21.15.142:0/555156461 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1b60102370 0x7f1b601027d0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:03:48.241 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-31T23:03:48.236+0000 7f1b66094700 1 -- 172.21.15.142:0/555156461 >> 172.21.15.142:0/555156461 conn(0x7f1b600fc710 msgr2=0x7f1b600feb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:03:48.241 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-31T23:03:48.236+0000 7f1b66094700 1 -- 172.21.15.142:0/555156461 shutdown_connections 2024-03-31T23:03:48.242 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-31T23:03:48.236+0000 7f1b66094700 1 -- 172.21.15.142:0/555156461 wait complete. 2024-03-31T23:03:48.242 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-31T23:03:48.236+0000 7f1b66094700 1 Processor -- start 2024-03-31T23:03:48.242 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-31T23:03:48.237+0000 7f1b66094700 1 -- start start 2024-03-31T23:03:48.242 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-31T23:03:48.237+0000 7f1b66094700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1b60101170 0x7f1b601973f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:03:48.242 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-31T23:03:48.237+0000 7f1b66094700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1b60102370 0x7f1b60197930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:03:48.243 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-31T23:03:48.237+0000 7f1b66094700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1b60102d10 0x7f1b6019e9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:03:48.243 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-31T23:03:48.237+0000 7f1b66094700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f1b6010cce0 con 0x7f1b60101170 2024-03-31T23:03:48.243 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-31T23:03:48.237+0000 7f1b66094700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f1b6010cb60 con 0x7f1b60102370 2024-03-31T23:03:48.243 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-31T23:03:48.237+0000 7f1b66094700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f1b6010ce60 con 0x7f1b60102d10 2024-03-31T23:03:48.243 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-31T23:03:48.237+0000 7f1b56ffd700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1b60102370 0x7f1b60197930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:03:48.243 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-31T23:03:48.237+0000 7f1b56ffd700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1b60102370 0x7f1b60197930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:55838/0 (socket says 172.21.15.142:55838) 2024-03-31T23:03:48.243 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-31T23:03:48.237+0000 7f1b56ffd700 1 -- 172.21.15.142:0/1631422312 learned_addr learned my addr 172.21.15.142:0/1631422312 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-03-31T23:03:48.244 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-31T23:03:48.237+0000 7f1b5f7fe700 1 --2- 172.21.15.142:0/1631422312 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1b60101170 0x7f1b601973f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:03:48.244 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-31T23:03:48.237+0000 7f1b5ffff700 1 --2- 172.21.15.142:0/1631422312 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1b60102d10 0x7f1b6019e9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:03:48.244 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-31T23:03:48.237+0000 7f1b56ffd700 1 -- 172.21.15.142:0/1631422312 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1b60102d10 msgr2=0x7f1b6019e9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:03:48.244 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-31T23:03:48.237+0000 7f1b56ffd700 1 --2- 172.21.15.142:0/1631422312 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1b60102d10 0x7f1b6019e9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:03:48.244 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-31T23:03:48.237+0000 7f1b56ffd700 1 -- 172.21.15.142:0/1631422312 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1b60101170 msgr2=0x7f1b601973f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:03:48.244 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-31T23:03:48.237+0000 7f1b56ffd700 1 --2- 172.21.15.142:0/1631422312 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1b60101170 0x7f1b601973f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:03:48.245 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-31T23:03:48.237+0000 7f1b56ffd700 1 -- 172.21.15.142:0/1631422312 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b6019f050 con 0x7f1b60102370 2024-03-31T23:03:48.245 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-31T23:03:48.237+0000 7f1b5f7fe700 1 --2- 172.21.15.142:0/1631422312 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1b60101170 0x7f1b601973f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T23:03:48.245 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-31T23:03:48.237+0000 7f1b56ffd700 1 --2- 172.21.15.142:0/1631422312 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1b60102370 0x7f1b60197930 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f1b48000c00 tx=0x7f1b48005e80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:03:48.245 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-31T23:03:48.238+0000 7f1b5d7fa700 1 -- 172.21.15.142:0/1631422312 <== mon.2 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b48027070 con 0x7f1b60102370 2024-03-31T23:03:48.245 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-31T23:03:48.238+0000 7f1b66094700 1 -- 172.21.15.142:0/1631422312 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1b6019f2e0 con 0x7f1b60102370 2024-03-31T23:03:48.245 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-31T23:03:48.238+0000 7f1b5d7fa700 1 -- 172.21.15.142:0/1631422312 <== mon.2 v2:172.21.15.142:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1b48004190 con 0x7f1b60102370 2024-03-31T23:03:48.246 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-31T23:03:48.238+0000 7f1b5d7fa700 1 -- 172.21.15.142:0/1631422312 <== mon.2 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b48022da0 con 0x7f1b60102370 2024-03-31T23:03:48.246 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-31T23:03:48.238+0000 7f1b66094700 1 -- 172.21.15.142:0/1631422312 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1b6019f7b0 con 0x7f1b60102370 2024-03-31T23:03:48.246 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-31T23:03:48.239+0000 7f1b66094700 1 -- 172.21.15.142:0/1631422312 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1b60068f00 con 0x7f1b60102370 2024-03-31T23:03:48.248 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-31T23:03:48.240+0000 7f1b5d7fa700 1 -- 172.21.15.142:0/1631422312 <== mon.2 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 15) ==== 137264+0+0 (secure 0 0 0) 0x7f1b4802a070 con 0x7f1b60102370 2024-03-31T23:03:48.248 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-31T23:03:48.240+0000 7f1b5d7fa700 1 --2- 172.21.15.142:0/1631422312 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f1b4c0a24a0 0x7f1b4c0a4960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:03:48.248 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-31T23:03:48.240+0000 7f1b5d7fa700 1 -- 172.21.15.142:0/1631422312 <== mon.2 v2:172.21.15.142:3300/0 5 ==== osd_map(8..8 src has 1..8) ==== 1767+0+0 (secure 0 0 0) 0x7f1b480cdcf0 con 0x7f1b60102370 2024-03-31T23:03:48.248 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-31T23:03:48.241+0000 7f1b5f7fe700 1 --2- 172.21.15.142:0/1631422312 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f1b4c0a24a0 0x7f1b4c0a4960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:03:48.248 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-31T23:03:48.241+0000 7f1b5f7fe700 1 --2- 172.21.15.142:0/1631422312 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f1b4c0a24a0 0x7f1b4c0a4960 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f1b50005950 tx=0x7f1b50005f50 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:03:48.249 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-31T23:03:48.245+0000 7f1b5d7fa700 1 -- 172.21.15.142:0/1631422312 <== mon.2 v2:172.21.15.142:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f1b4805e280 con 0x7f1b60102370 2024-03-31T23:03:48.449 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-31T23:03:48.448+0000 7f1b66094700 1 -- 172.21.15.142:0/1631422312 --> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi142:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7f1b60062680 con 0x7f1b4c0a24a0 2024-03-31T23:03:49.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:03:48 smithi046 ceph-mon[32498]: from='client.24146 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi142:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-31T23:03:49.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:03:48 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-31T23:03:49.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:03:48 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-31T23:03:49.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:03:48 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:03:49.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:03:48 smithi046 ceph-mon[32498]: pgmap v48: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-03-31T23:03:49.145 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:48 smithi142 ceph-mon[35271]: from='client.24146 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi142:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-31T23:03:49.145 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:48 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-31T23:03:49.145 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:48 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-31T23:03:49.145 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:48 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:03:49.145 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:48 smithi142 ceph-mon[35271]: pgmap v48: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-03-31T23:03:49.209 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:03:48 smithi162 ceph-mon[36237]: from='client.24146 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi142:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-31T23:03:49.210 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:03:48 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-31T23:03:49.210 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:03:48 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-31T23:03:49.210 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:03:48 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:03:49.210 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:03:48 smithi162 ceph-mon[36237]: pgmap v48: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-03-31T23:03:49.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:49 smithi142 ceph-mon[35271]: osdmap e9: 1 total, 1 up, 1 in 2024-03-31T23:03:50.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:03:49 smithi046 ceph-mon[32498]: osdmap e9: 1 total, 1 up, 1 in 2024-03-31T23:03:50.296 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:03:49 smithi162 ceph-mon[36237]: osdmap e9: 1 total, 1 up, 1 in 2024-03-31T23:03:51.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:50 smithi142 ceph-mon[35271]: pgmap v50: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-03-31T23:03:51.296 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:03:50 smithi162 ceph-mon[36237]: pgmap v50: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-03-31T23:03:51.361 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:03:50 smithi046 ceph-mon[32498]: pgmap v50: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-03-31T23:03:52.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:51 smithi142 ceph-mon[35271]: from='client.? 172.21.15.142:0/3332513451' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "be56f883-411e-4dbe-b50e-a158134df4a6"}]: dispatch 2024-03-31T23:03:52.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:51 smithi142 ceph-mon[35271]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "be56f883-411e-4dbe-b50e-a158134df4a6"}]: dispatch 2024-03-31T23:03:52.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:51 smithi142 ceph-mon[35271]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "be56f883-411e-4dbe-b50e-a158134df4a6"}]': finished 2024-03-31T23:03:52.194 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:51 smithi142 ceph-mon[35271]: osdmap e10: 2 total, 1 up, 2 in 2024-03-31T23:03:52.194 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:51 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-31T23:03:52.194 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:51 smithi142 ceph-mon[35271]: from='client.? 172.21.15.142:0/3818388368' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-31T23:03:52.295 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:03:51 smithi162 ceph-mon[36237]: from='client.? 172.21.15.142:0/3332513451' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "be56f883-411e-4dbe-b50e-a158134df4a6"}]: dispatch 2024-03-31T23:03:52.296 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:03:51 smithi162 ceph-mon[36237]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "be56f883-411e-4dbe-b50e-a158134df4a6"}]: dispatch 2024-03-31T23:03:52.296 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:03:51 smithi162 ceph-mon[36237]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "be56f883-411e-4dbe-b50e-a158134df4a6"}]': finished 2024-03-31T23:03:52.296 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:03:51 smithi162 ceph-mon[36237]: osdmap e10: 2 total, 1 up, 2 in 2024-03-31T23:03:52.296 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:03:51 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-31T23:03:52.296 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:03:51 smithi162 ceph-mon[36237]: from='client.? 172.21.15.142:0/3818388368' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-31T23:03:52.361 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:03:51 smithi046 ceph-mon[32498]: from='client.? 172.21.15.142:0/3332513451' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "be56f883-411e-4dbe-b50e-a158134df4a6"}]: dispatch 2024-03-31T23:03:52.361 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:03:51 smithi046 ceph-mon[32498]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "be56f883-411e-4dbe-b50e-a158134df4a6"}]: dispatch 2024-03-31T23:03:52.362 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:03:51 smithi046 ceph-mon[32498]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "be56f883-411e-4dbe-b50e-a158134df4a6"}]': finished 2024-03-31T23:03:52.362 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:03:51 smithi046 ceph-mon[32498]: osdmap e10: 2 total, 1 up, 2 in 2024-03-31T23:03:52.362 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:03:51 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-31T23:03:52.362 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:03:51 smithi046 ceph-mon[32498]: from='client.? 172.21.15.142:0/3818388368' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-31T23:03:53.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:52 smithi142 ceph-mon[35271]: pgmap v52: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-03-31T23:03:53.296 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:03:52 smithi162 ceph-mon[36237]: pgmap v52: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-03-31T23:03:53.361 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:03:52 smithi046 ceph-mon[32498]: pgmap v52: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-03-31T23:03:54.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:03:54 smithi046 ceph-mon[32498]: pgmap v53: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-03-31T23:03:54.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:54 smithi142 ceph-mon[35271]: pgmap v53: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-03-31T23:03:55.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:03:54 smithi162 ceph-mon[36237]: pgmap v53: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-03-31T23:03:56.762 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:56 smithi142 ceph-mon[35271]: pgmap v54: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-03-31T23:03:56.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:03:56 smithi046 ceph-mon[32498]: pgmap v54: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-03-31T23:03:57.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:03:56 smithi162 ceph-mon[36237]: pgmap v54: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-03-31T23:03:58.443 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:58 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-31T23:03:58.443 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:58 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:03:58.545 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:03:58 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-31T23:03:58.546 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:03:58 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:03:58.611 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:03:58 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-31T23:03:58.611 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:03:58 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:03:59.443 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:59 smithi142 ceph-mon[35271]: Deploying daemon osd.1 on smithi142 2024-03-31T23:03:59.444 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:03:59 smithi142 ceph-mon[35271]: pgmap v55: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-03-31T23:03:59.545 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:03:59 smithi162 ceph-mon[36237]: Deploying daemon osd.1 on smithi142 2024-03-31T23:03:59.546 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:03:59 smithi162 ceph-mon[36237]: pgmap v55: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-03-31T23:03:59.612 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:03:59 smithi046 ceph-mon[32498]: Deploying daemon osd.1 on smithi142 2024-03-31T23:03:59.612 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:03:59 smithi046 ceph-mon[32498]: pgmap v55: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-03-31T23:04:00.693 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:04:00 smithi142 ceph-mon[35271]: pgmap v56: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-03-31T23:04:00.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:04:00 smithi162 ceph-mon[36237]: pgmap v56: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-03-31T23:04:00.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:04:00 smithi046 ceph-mon[32498]: pgmap v56: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-03-31T23:04:02.862 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:04:02 smithi046 ceph-mon[32498]: pgmap v57: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-03-31T23:04:02.881 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:04:02 smithi142 ceph-mon[35271]: pgmap v57: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-03-31T23:04:03.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:04:02 smithi162 ceph-mon[36237]: pgmap v57: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-03-31T23:04:04.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:04:04 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:04:04.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:04:04 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:04:04.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:04:04 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-31T23:04:04.361 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:04:04 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:04:04.362 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:04:04 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:04:04.362 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:04:04 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-31T23:04:04.546 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:04:04 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:04:04.547 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:04:04 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:04:04.547 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:04:04 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-31T23:04:05.361 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:04:05 smithi046 ceph-mon[32498]: pgmap v58: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-03-31T23:04:05.397 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:04:05 smithi142 ceph-mon[35271]: pgmap v58: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-03-31T23:04:05.545 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:04:05 smithi162 ceph-mon[36237]: pgmap v58: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-03-31T23:04:06.514 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-31T23:04:06.512+0000 7f1b5d7fa700 1 -- 172.21.15.142:0/1631422312 <== mgr.14152 v2:172.21.15.46:6800/162289569 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f1b60062680 con 0x7f1b4c0a24a0 2024-03-31T23:04:06.514 INFO:teuthology.orchestra.run.smithi142.stdout:Created osd(s) 1 on host 'smithi142' 2024-03-31T23:04:06.516 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-31T23:04:06.515+0000 7f1b66094700 1 -- 172.21.15.142:0/1631422312 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f1b4c0a24a0 msgr2=0x7f1b4c0a4960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:04:06.516 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-31T23:04:06.515+0000 7f1b66094700 1 --2- 172.21.15.142:0/1631422312 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f1b4c0a24a0 0x7f1b4c0a4960 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f1b50005950 tx=0x7f1b50005f50 comp rx=0 tx=0).stop 2024-03-31T23:04:06.516 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-31T23:04:06.515+0000 7f1b66094700 1 -- 172.21.15.142:0/1631422312 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1b60102370 msgr2=0x7f1b60197930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:04:06.516 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-31T23:04:06.515+0000 7f1b66094700 1 --2- 172.21.15.142:0/1631422312 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1b60102370 0x7f1b60197930 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f1b48000c00 tx=0x7f1b48005e80 comp rx=0 tx=0).stop 2024-03-31T23:04:06.517 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-31T23:04:06.515+0000 7f1b66094700 1 -- 172.21.15.142:0/1631422312 shutdown_connections 2024-03-31T23:04:06.517 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-31T23:04:06.515+0000 7f1b66094700 1 --2- 172.21.15.142:0/1631422312 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f1b4c0a24a0 0x7f1b4c0a4960 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:04:06.517 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-31T23:04:06.515+0000 7f1b66094700 1 --2- 172.21.15.142:0/1631422312 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1b60101170 0x7f1b601973f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:04:06.517 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-31T23:04:06.515+0000 7f1b66094700 1 --2- 172.21.15.142:0/1631422312 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1b60102d10 0x7f1b6019e9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:04:06.517 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-31T23:04:06.515+0000 7f1b66094700 1 --2- 172.21.15.142:0/1631422312 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1b60102370 0x7f1b60197930 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:04:06.517 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-31T23:04:06.515+0000 7f1b66094700 1 -- 172.21.15.142:0/1631422312 >> 172.21.15.142:0/1631422312 conn(0x7f1b600fc710 msgr2=0x7f1b60105790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:04:06.517 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-31T23:04:06.515+0000 7f1b66094700 1 -- 172.21.15.142:0/1631422312 shutdown_connections 2024-03-31T23:04:06.517 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-31T23:04:06.515+0000 7f1b66094700 1 -- 172.21.15.142:0/1631422312 wait complete. 2024-03-31T23:04:06.811 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:04:06 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:04:06.811 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:04:06 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:04:06.811 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:04:06 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:04:06.811 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:04:06 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-31T23:04:06.811 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:04:06 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:04:06.811 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:04:06 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:04:06.811 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:04:06 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:04:06.811 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:04:06 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-31T23:04:06.811 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:04:06 smithi142 ceph-mon[35271]: pgmap v59: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-03-31T23:04:07.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:04:06 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:04:07.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:04:06 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:04:07.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:04:06 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:04:07.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:04:06 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-31T23:04:07.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:04:06 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:04:07.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:04:06 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:04:07.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:04:06 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:04:07.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:04:06 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-31T23:04:07.047 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:04:06 smithi162 ceph-mon[36237]: pgmap v59: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-03-31T23:04:07.086 DEBUG:teuthology.orchestra.run.smithi142:osd.1> sudo journalctl -f -n 0 -u ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4@osd.1.service 2024-03-31T23:04:07.089 INFO:tasks.cephadm:Deploying osd.2 on smithi162 with /dev/vg_nvme/lv_4... 2024-03-31T23:04:07.089 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:384f31f3d3aee7f8ba3f60846b8844c049165583 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6a9d8f08-efb2-11ee-b647-cb9ed24678a4 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-03-31T23:04:07.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:04:06 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:04:07.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:04:06 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:04:07.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:04:06 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:04:07.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:04:06 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-31T23:04:07.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:04:06 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:04:07.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:04:06 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:04:07.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:04:06 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:04:07.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:04:06 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-31T23:04:07.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:04:06 smithi046 ceph-mon[32498]: pgmap v59: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-03-31T23:04:07.334 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/6a9d8f08-efb2-11ee-b647-cb9ed24678a4/mon.c/config 2024-03-31T23:04:07.693 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:04:07 smithi142 ceph-mon[35271]: from='osd.1 [v2:172.21.15.142:6800/973693922,v1:172.21.15.142:6801/973693922]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-31T23:04:07.693 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:04:07 smithi142 ceph-mon[35271]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-31T23:04:07.747 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:04:07 smithi162 ceph-mon[36237]: from='osd.1 [v2:172.21.15.142:6800/973693922,v1:172.21.15.142:6801/973693922]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-31T23:04:07.747 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:04:07 smithi162 ceph-mon[36237]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-31T23:04:08.013 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:04:07 smithi046 ceph-mon[32498]: from='osd.1 [v2:172.21.15.142:6800/973693922,v1:172.21.15.142:6801/973693922]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-31T23:04:08.013 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:04:07 smithi046 ceph-mon[32498]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-31T23:04:08.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:04:08 smithi162 ceph-mon[36237]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-31T23:04:08.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:04:08 smithi162 ceph-mon[36237]: from='osd.1 [v2:172.21.15.142:6800/973693922,v1:172.21.15.142:6801/973693922]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi142", "root=default"]}]: dispatch 2024-03-31T23:04:08.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:04:08 smithi162 ceph-mon[36237]: osdmap e11: 2 total, 1 up, 2 in 2024-03-31T23:04:08.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:04:08 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-31T23:04:08.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:04:08 smithi162 ceph-mon[36237]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi142", "root=default"]}]: dispatch 2024-03-31T23:04:08.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:04:08 smithi162 ceph-mon[36237]: pgmap v61: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-03-31T23:04:08.944 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:04:08 smithi142 ceph-mon[35271]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-31T23:04:08.944 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:04:08 smithi142 ceph-mon[35271]: from='osd.1 [v2:172.21.15.142:6800/973693922,v1:172.21.15.142:6801/973693922]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi142", "root=default"]}]: dispatch 2024-03-31T23:04:08.944 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:04:08 smithi142 ceph-mon[35271]: osdmap e11: 2 total, 1 up, 2 in 2024-03-31T23:04:08.944 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:04:08 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-31T23:04:08.944 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:04:08 smithi142 ceph-mon[35271]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi142", "root=default"]}]: dispatch 2024-03-31T23:04:08.944 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:04:08 smithi142 ceph-mon[35271]: pgmap v61: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-03-31T23:04:08.944 INFO:journalctl@ceph.osd.1.smithi142.stdout:Mar 31 23:04:08 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-1[39861]: 2024-03-31T23:04:08.641+0000 7f23ae1c6700 -1 osd.1 0 waiting for initial osdmap 2024-03-31T23:04:08.944 INFO:journalctl@ceph.osd.1.smithi142.stdout:Mar 31 23:04:08 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-1[39861]: 2024-03-31T23:04:08.644+0000 7f23a9fb9700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-31T23:04:09.006 INFO:teuthology.orchestra.run.smithi162.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-03-31T23:04:09.006 INFO:teuthology.orchestra.run.smithi162.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-03-31T23:04:09.007 INFO:teuthology.orchestra.run.smithi162.stderr: stderr: 10+0 records in 2024-03-31T23:04:09.007 INFO:teuthology.orchestra.run.smithi162.stderr:10+0 records out 2024-03-31T23:04:09.007 INFO:teuthology.orchestra.run.smithi162.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0122256 s, 858 MB/s 2024-03-31T23:04:09.007 INFO:teuthology.orchestra.run.smithi162.stderr:--> Zapping successful for: 2024-03-31T23:04:09.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:04:08 smithi046 ceph-mon[32498]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-31T23:04:09.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:04:08 smithi046 ceph-mon[32498]: from='osd.1 [v2:172.21.15.142:6800/973693922,v1:172.21.15.142:6801/973693922]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi142", "root=default"]}]: dispatch 2024-03-31T23:04:09.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:04:08 smithi046 ceph-mon[32498]: osdmap e11: 2 total, 1 up, 2 in 2024-03-31T23:04:09.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:04:08 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-31T23:04:09.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:04:08 smithi046 ceph-mon[32498]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi142", "root=default"]}]: dispatch 2024-03-31T23:04:09.113 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:04:08 smithi046 ceph-mon[32498]: pgmap v61: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-03-31T23:04:09.494 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:384f31f3d3aee7f8ba3f60846b8844c049165583 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6a9d8f08-efb2-11ee-b647-cb9ed24678a4 -- ceph orch daemon add osd smithi162:vg_nvme/lv_4 2024-03-31T23:04:09.736 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/6a9d8f08-efb2-11ee-b647-cb9ed24678a4/mon.c/config 2024-03-31T23:04:09.926 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:04:09 smithi142 ceph-mon[35271]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi142", "root=default"]}]': finished 2024-03-31T23:04:09.926 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:04:09 smithi142 ceph-mon[35271]: osdmap e12: 2 total, 1 up, 2 in 2024-03-31T23:04:09.926 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:04:09 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-31T23:04:09.926 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:04:09 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-31T23:04:09.940 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:04:09 smithi162 ceph-mon[36237]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi142", "root=default"]}]': finished 2024-03-31T23:04:09.940 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:04:09 smithi162 ceph-mon[36237]: osdmap e12: 2 total, 1 up, 2 in 2024-03-31T23:04:09.940 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:04:09 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-31T23:04:09.941 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:04:09 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-31T23:04:10.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:04:09 smithi046 ceph-mon[32498]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi142", "root=default"]}]': finished 2024-03-31T23:04:10.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:04:09 smithi046 ceph-mon[32498]: osdmap e12: 2 total, 1 up, 2 in 2024-03-31T23:04:10.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:04:09 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-31T23:04:10.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:04:09 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-31T23:04:10.922 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:04:10 smithi162 ceph-mon[36237]: purged_snaps scrub starts 2024-03-31T23:04:10.922 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:04:10 smithi162 ceph-mon[36237]: purged_snaps scrub ok 2024-03-31T23:04:10.922 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:04:10 smithi162 ceph-mon[36237]: osd.1 [v2:172.21.15.142:6800/973693922,v1:172.21.15.142:6801/973693922] boot 2024-03-31T23:04:10.922 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:04:10 smithi162 ceph-mon[36237]: osdmap e13: 2 total, 2 up, 2 in 2024-03-31T23:04:10.922 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:04:10 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-31T23:04:10.922 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:04:10 smithi162 ceph-mon[36237]: Detected new or changed devices on smithi142 2024-03-31T23:04:10.922 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:04:10 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:04:10.923 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:04:10 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:04:10.923 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:04:10 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-31T23:04:10.923 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:04:10 smithi162 ceph-mon[36237]: Adjusting osd_memory_target on smithi142 to 17088M 2024-03-31T23:04:10.923 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:04:10 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:04:10.923 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:04:10 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:04:10.923 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:04:10 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-31T23:04:10.923 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:04:10 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:04:10.923 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:04:10 smithi162 ceph-mon[36237]: pgmap v64: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-03-31T23:04:10.944 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:04:10 smithi142 ceph-mon[35271]: purged_snaps scrub starts 2024-03-31T23:04:10.944 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:04:10 smithi142 ceph-mon[35271]: purged_snaps scrub ok 2024-03-31T23:04:10.944 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:04:10 smithi142 ceph-mon[35271]: osd.1 [v2:172.21.15.142:6800/973693922,v1:172.21.15.142:6801/973693922] boot 2024-03-31T23:04:10.944 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:04:10 smithi142 ceph-mon[35271]: osdmap e13: 2 total, 2 up, 2 in 2024-03-31T23:04:10.944 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:04:10 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-31T23:04:10.944 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:04:10 smithi142 ceph-mon[35271]: Detected new or changed devices on smithi142 2024-03-31T23:04:10.944 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:04:10 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:04:10.944 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:04:10 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:04:10.944 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:04:10 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-31T23:04:10.944 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:04:10 smithi142 ceph-mon[35271]: Adjusting osd_memory_target on smithi142 to 17088M 2024-03-31T23:04:10.944 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:04:10 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:04:10.944 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:04:10 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:04:10.945 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:04:10 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-31T23:04:10.945 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:04:10 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:04:10.945 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:04:10 smithi142 ceph-mon[35271]: pgmap v64: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-03-31T23:04:11.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:04:10 smithi046 ceph-mon[32498]: purged_snaps scrub starts 2024-03-31T23:04:11.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:04:10 smithi046 ceph-mon[32498]: purged_snaps scrub ok 2024-03-31T23:04:11.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:04:10 smithi046 ceph-mon[32498]: osd.1 [v2:172.21.15.142:6800/973693922,v1:172.21.15.142:6801/973693922] boot 2024-03-31T23:04:11.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:04:10 smithi046 ceph-mon[32498]: osdmap e13: 2 total, 2 up, 2 in 2024-03-31T23:04:11.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:04:10 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-31T23:04:11.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:04:10 smithi046 ceph-mon[32498]: Detected new or changed devices on smithi142 2024-03-31T23:04:11.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:04:10 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:04:11.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:04:10 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:04:11.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:04:10 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-31T23:04:11.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:04:10 smithi046 ceph-mon[32498]: Adjusting osd_memory_target on smithi142 to 17088M 2024-03-31T23:04:11.113 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:04:10 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:04:11.113 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:04:10 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:04:11.113 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:04:10 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-31T23:04:11.113 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:04:10 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:04:11.113 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:04:10 smithi046 ceph-mon[32498]: pgmap v64: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-03-31T23:04:11.222 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:04:11.219+0000 7fbc76848700 1 -- 172.21.15.162:0/1808108049 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbc70102d60 msgr2=0x7fbc70105150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:04:11.222 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:04:11.219+0000 7fbc76848700 1 --2- 172.21.15.162:0/1808108049 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbc70102d60 0x7fbc70105150 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fbc60009990 tx=0x7fbc60009ca0 comp rx=0 tx=0).stop 2024-03-31T23:04:11.222 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:04:11.219+0000 7fbc76848700 1 -- 172.21.15.162:0/1808108049 shutdown_connections 2024-03-31T23:04:11.222 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:04:11.219+0000 7fbc76848700 1 --2- 172.21.15.162:0/1808108049 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbc70107fc0 0x7fbc7010a3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:04:11.222 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:04:11.219+0000 7fbc76848700 1 --2- 172.21.15.162:0/1808108049 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbc70102d60 0x7fbc70105150 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:04:11.222 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:04:11.219+0000 7fbc76848700 1 --2- 172.21.15.162:0/1808108049 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbc70105690 0x7fbc70107a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:04:11.222 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:04:11.219+0000 7fbc76848700 1 -- 172.21.15.162:0/1808108049 >> 172.21.15.162:0/1808108049 conn(0x7fbc700fc730 msgr2=0x7fbc700feb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:04:11.222 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:04:11.219+0000 7fbc76848700 1 -- 172.21.15.162:0/1808108049 shutdown_connections 2024-03-31T23:04:11.222 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:04:11.219+0000 7fbc76848700 1 -- 172.21.15.162:0/1808108049 wait complete. 2024-03-31T23:04:11.222 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:04:11.219+0000 7fbc76848700 1 Processor -- start 2024-03-31T23:04:11.223 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:04:11.220+0000 7fbc76848700 1 -- start start 2024-03-31T23:04:11.223 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:04:11.220+0000 7fbc76848700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbc70102d60 0x7fbc701983a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:04:11.223 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:04:11.220+0000 7fbc76848700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbc70105690 0x7fbc701988e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:04:11.223 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:04:11.220+0000 7fbc76848700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbc70107fc0 0x7fbc7019f960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:04:11.223 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:04:11.220+0000 7fbc76848700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fbc7010dd50 con 0x7fbc70102d60 2024-03-31T23:04:11.223 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:04:11.220+0000 7fbc76848700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fbc7010dbd0 con 0x7fbc70107fc0 2024-03-31T23:04:11.223 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:04:11.220+0000 7fbc76848700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fbc7010ded0 con 0x7fbc70105690 2024-03-31T23:04:11.223 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:04:11.220+0000 7fbc6f7fe700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbc70105690 0x7fbc701988e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:04:11.223 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:04:11.220+0000 7fbc6f7fe700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbc70105690 0x7fbc701988e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:59090/0 (socket says 172.21.15.162:59090) 2024-03-31T23:04:11.223 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:04:11.220+0000 7fbc6f7fe700 1 -- 172.21.15.162:0/306881056 learned_addr learned my addr 172.21.15.162:0/306881056 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-03-31T23:04:11.223 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:04:11.220+0000 7fbc74de5700 1 --2- 172.21.15.162:0/306881056 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbc70107fc0 0x7fbc7019f960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:04:11.223 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:04:11.220+0000 7fbc6ffff700 1 --2- 172.21.15.162:0/306881056 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbc70102d60 0x7fbc701983a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:04:11.223 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:04:11.220+0000 7fbc6f7fe700 1 -- 172.21.15.162:0/306881056 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbc70107fc0 msgr2=0x7fbc7019f960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:04:11.224 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:04:11.220+0000 7fbc6f7fe700 1 --2- 172.21.15.162:0/306881056 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbc70107fc0 0x7fbc7019f960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:04:11.224 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:04:11.220+0000 7fbc6f7fe700 1 -- 172.21.15.162:0/306881056 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbc70102d60 msgr2=0x7fbc701983a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:04:11.224 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:04:11.220+0000 7fbc6f7fe700 1 --2- 172.21.15.162:0/306881056 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbc70102d60 0x7fbc701983a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:04:11.224 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:04:11.220+0000 7fbc6f7fe700 1 -- 172.21.15.162:0/306881056 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc701a0000 con 0x7fbc70105690 2024-03-31T23:04:11.224 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:04:11.221+0000 7fbc6f7fe700 1 --2- 172.21.15.162:0/306881056 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbc70105690 0x7fbc701988e0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fbc6400e9d0 tx=0x7fbc6400ece0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:04:11.225 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:04:11.221+0000 7fbc6d7fa700 1 -- 172.21.15.162:0/306881056 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc6400cbd0 con 0x7fbc70105690 2024-03-31T23:04:11.225 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:04:11.221+0000 7fbc6d7fa700 1 -- 172.21.15.162:0/306881056 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fbc6400ee70 con 0x7fbc70105690 2024-03-31T23:04:11.225 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:04:11.221+0000 7fbc6d7fa700 1 -- 172.21.15.162:0/306881056 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc64018680 con 0x7fbc70105690 2024-03-31T23:04:11.225 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:04:11.221+0000 7fbc76848700 1 -- 172.21.15.162:0/306881056 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbc701a02f0 con 0x7fbc70105690 2024-03-31T23:04:11.225 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:04:11.221+0000 7fbc76848700 1 -- 172.21.15.162:0/306881056 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbc701a0880 con 0x7fbc70105690 2024-03-31T23:04:11.225 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:04:11.222+0000 7fbc76848700 1 -- 172.21.15.162:0/306881056 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbc70068f00 con 0x7fbc70105690 2024-03-31T23:04:11.225 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:04:11.223+0000 7fbc6d7fa700 1 -- 172.21.15.162:0/306881056 <== mon.1 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 15) ==== 137264+0+0 (secure 0 0 0) 0x7fbc64010430 con 0x7fbc70105690 2024-03-31T23:04:11.226 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:04:11.223+0000 7fbc6d7fa700 1 --2- 172.21.15.162:0/306881056 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fbc580a2520 0x7fbc580a49e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:04:11.226 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:04:11.223+0000 7fbc6d7fa700 1 -- 172.21.15.162:0/306881056 <== mon.1 v2:172.21.15.162:3300/0 5 ==== osd_map(14..14 src has 1..14) ==== 2309+0+0 (secure 0 0 0) 0x7fbc640ccfe0 con 0x7fbc70105690 2024-03-31T23:04:11.226 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:04:11.224+0000 7fbc6ffff700 1 --2- 172.21.15.162:0/306881056 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fbc580a2520 0x7fbc580a49e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:04:11.226 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:04:11.224+0000 7fbc6ffff700 1 --2- 172.21.15.162:0/306881056 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fbc580a2520 0x7fbc580a49e0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fbc6000b670 tx=0x7fbc60005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:04:11.229 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:04:11.228+0000 7fbc6d7fa700 1 -- 172.21.15.162:0/306881056 <== mon.1 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fbc6409b9f0 con 0x7fbc70105690 2024-03-31T23:04:11.428 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:04:11.426+0000 7fbc76848700 1 -- 172.21.15.162:0/306881056 --> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi162:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7fbc70062740 con 0x7fbc580a2520 2024-03-31T23:04:11.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:04:11 smithi142 ceph-mon[35271]: osdmap e14: 2 total, 2 up, 2 in 2024-03-31T23:04:11.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:04:11 smithi142 ceph-mon[35271]: from='client.24157 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi162:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-31T23:04:11.944 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:04:11 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-31T23:04:11.944 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:04:11 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-31T23:04:11.944 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:04:11 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:04:12.031 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:04:11 smithi162 ceph-mon[36237]: osdmap e14: 2 total, 2 up, 2 in 2024-03-31T23:04:12.031 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:04:11 smithi162 ceph-mon[36237]: from='client.24157 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi162:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-31T23:04:12.031 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:04:11 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-31T23:04:12.032 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:04:11 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-31T23:04:12.032 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:04:11 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:04:12.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:04:11 smithi046 ceph-mon[32498]: osdmap e14: 2 total, 2 up, 2 in 2024-03-31T23:04:12.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:04:11 smithi046 ceph-mon[32498]: from='client.24157 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi162:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-31T23:04:12.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:04:11 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-31T23:04:12.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:04:11 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-31T23:04:12.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:04:11 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:04:12.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:04:12 smithi142 ceph-mon[35271]: pgmap v66: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-31T23:04:13.037 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:04:12 smithi162 ceph-mon[36237]: pgmap v66: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-31T23:04:13.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:04:12 smithi046 ceph-mon[32498]: pgmap v66: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-31T23:04:14.296 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:04:13 smithi162 ceph-mon[36237]: from='client.? 172.21.15.162:0/4076720760' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c99a8794-ab19-4260-9a8d-4bc0bd2c8175"}]: dispatch 2024-03-31T23:04:14.296 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:04:14 smithi162 ceph-mon[36237]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c99a8794-ab19-4260-9a8d-4bc0bd2c8175"}]: dispatch 2024-03-31T23:04:14.296 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:04:14 smithi162 ceph-mon[36237]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c99a8794-ab19-4260-9a8d-4bc0bd2c8175"}]': finished 2024-03-31T23:04:14.296 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:04:14 smithi162 ceph-mon[36237]: osdmap e15: 3 total, 2 up, 3 in 2024-03-31T23:04:14.296 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:04:14 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-31T23:04:14.362 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:04:13 smithi046 ceph-mon[32498]: from='client.? 172.21.15.162:0/4076720760' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c99a8794-ab19-4260-9a8d-4bc0bd2c8175"}]: dispatch 2024-03-31T23:04:14.362 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:04:13 smithi046 ceph-mon[32498]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c99a8794-ab19-4260-9a8d-4bc0bd2c8175"}]: dispatch 2024-03-31T23:04:14.362 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:04:13 smithi046 ceph-mon[32498]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c99a8794-ab19-4260-9a8d-4bc0bd2c8175"}]': finished 2024-03-31T23:04:14.362 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:04:13 smithi046 ceph-mon[32498]: osdmap e15: 3 total, 2 up, 3 in 2024-03-31T23:04:14.363 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:04:13 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-31T23:04:14.443 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:04:13 smithi142 ceph-mon[35271]: from='client.? 172.21.15.162:0/4076720760' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c99a8794-ab19-4260-9a8d-4bc0bd2c8175"}]: dispatch 2024-03-31T23:04:14.443 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:04:13 smithi142 ceph-mon[35271]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c99a8794-ab19-4260-9a8d-4bc0bd2c8175"}]: dispatch 2024-03-31T23:04:14.444 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:04:13 smithi142 ceph-mon[35271]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c99a8794-ab19-4260-9a8d-4bc0bd2c8175"}]': finished 2024-03-31T23:04:14.444 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:04:13 smithi142 ceph-mon[35271]: osdmap e15: 3 total, 2 up, 3 in 2024-03-31T23:04:14.444 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:04:13 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-31T23:04:15.296 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:04:15 smithi162 ceph-mon[36237]: pgmap v68: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-31T23:04:15.296 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:04:15 smithi162 ceph-mon[36237]: from='client.? 172.21.15.162:0/941091848' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-31T23:04:15.361 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:04:15 smithi046 ceph-mon[32498]: pgmap v68: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-31T23:04:15.362 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:04:15 smithi046 ceph-mon[32498]: from='client.? 172.21.15.162:0/941091848' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-31T23:04:15.443 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:04:15 smithi142 ceph-mon[35271]: pgmap v68: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-31T23:04:15.443 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:04:15 smithi142 ceph-mon[35271]: from='client.? 172.21.15.162:0/941091848' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-31T23:04:16.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:04:16 smithi046 ceph-mon[32498]: pgmap v69: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-31T23:04:16.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:04:16 smithi142 ceph-mon[35271]: pgmap v69: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-31T23:04:17.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:04:16 smithi162 ceph-mon[36237]: pgmap v69: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-31T23:04:18.863 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:04:18 smithi046 ceph-mon[32498]: pgmap v70: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-31T23:04:18.872 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:04:18 smithi162 ceph-mon[36237]: pgmap v70: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-31T23:04:18.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:04:18 smithi142 ceph-mon[35271]: pgmap v70: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-31T23:04:20.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:04:20 smithi162 ceph-mon[36237]: pgmap v71: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-31T23:04:20.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:04:20 smithi046 ceph-mon[32498]: pgmap v71: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-31T23:04:20.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:04:20 smithi142 ceph-mon[35271]: pgmap v71: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-31T23:04:21.846 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:04:21 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-31T23:04:21.934 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:04:21 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:04:21.934 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:04:21 smithi162 ceph-mon[36237]: Deploying daemon osd.2 on smithi162 2024-03-31T23:04:21.935 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:04:21 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-31T23:04:21.935 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:04:21 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:04:21.935 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:04:21 smithi046 ceph-mon[32498]: Deploying daemon osd.2 on smithi162 2024-03-31T23:04:21.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:04:21 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-31T23:04:21.944 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:04:21 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:04:21.944 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:04:21 smithi142 ceph-mon[35271]: Deploying daemon osd.2 on smithi162 2024-03-31T23:04:22.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:04:22 smithi046 ceph-mon[32498]: pgmap v72: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-31T23:04:22.937 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:04:22 smithi162 ceph-mon[36237]: pgmap v72: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-31T23:04:22.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:04:22 smithi142 ceph-mon[35271]: pgmap v72: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-31T23:04:24.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:04:24 smithi162 ceph-mon[36237]: pgmap v73: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-31T23:04:24.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:04:24 smithi046 ceph-mon[32498]: pgmap v73: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-31T23:04:24.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:04:24 smithi142 ceph-mon[35271]: pgmap v73: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-31T23:04:27.296 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:04:27 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:04:27.296 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:04:27 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:04:27.296 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:04:27 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-31T23:04:27.296 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:04:27 smithi162 ceph-mon[36237]: pgmap v74: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-31T23:04:27.362 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:04:27 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:04:27.362 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:04:27 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:04:27.362 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:04:27 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-31T23:04:27.362 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:04:27 smithi046 ceph-mon[32498]: pgmap v74: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-31T23:04:27.443 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:04:27 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:04:27.443 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:04:27 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:04:27.443 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:04:27 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-31T23:04:27.444 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:04:27 smithi142 ceph-mon[35271]: pgmap v74: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-31T23:04:29.295 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:04:29.295+0000 7fbc6d7fa700 1 -- 172.21.15.162:0/306881056 <== mgr.14152 v2:172.21.15.46:6800/162289569 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fbc70062740 con 0x7fbc580a2520 2024-03-31T23:04:29.296 INFO:teuthology.orchestra.run.smithi162.stdout:Created osd(s) 2 on host 'smithi162' 2024-03-31T23:04:29.298 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:04:29.296+0000 7fbc76848700 1 -- 172.21.15.162:0/306881056 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fbc580a2520 msgr2=0x7fbc580a49e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:04:29.298 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:04:29.296+0000 7fbc76848700 1 --2- 172.21.15.162:0/306881056 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fbc580a2520 0x7fbc580a49e0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fbc6000b670 tx=0x7fbc60005fd0 comp rx=0 tx=0).stop 2024-03-31T23:04:29.298 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:04:29.296+0000 7fbc76848700 1 -- 172.21.15.162:0/306881056 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbc70105690 msgr2=0x7fbc701988e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:04:29.298 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:04:29.296+0000 7fbc76848700 1 --2- 172.21.15.162:0/306881056 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbc70105690 0x7fbc701988e0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fbc6400e9d0 tx=0x7fbc6400ece0 comp rx=0 tx=0).stop 2024-03-31T23:04:29.298 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:04:29.296+0000 7fbc76848700 1 -- 172.21.15.162:0/306881056 shutdown_connections 2024-03-31T23:04:29.298 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:04:29.296+0000 7fbc76848700 1 --2- 172.21.15.162:0/306881056 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fbc580a2520 0x7fbc580a49e0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:04:29.298 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:04:29.296+0000 7fbc76848700 1 --2- 172.21.15.162:0/306881056 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbc70102d60 0x7fbc701983a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:04:29.298 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:04:29.296+0000 7fbc76848700 1 --2- 172.21.15.162:0/306881056 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbc70107fc0 0x7fbc7019f960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:04:29.299 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:04:29.296+0000 7fbc76848700 1 --2- 172.21.15.162:0/306881056 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbc70105690 0x7fbc701988e0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:04:29.299 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:04:29.296+0000 7fbc76848700 1 -- 172.21.15.162:0/306881056 >> 172.21.15.162:0/306881056 conn(0x7fbc700fc730 msgr2=0x7fbc701036a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:04:29.299 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:04:29.296+0000 7fbc76848700 1 -- 172.21.15.162:0/306881056 shutdown_connections 2024-03-31T23:04:29.299 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:04:29.297+0000 7fbc76848700 1 -- 172.21.15.162:0/306881056 wait complete. 2024-03-31T23:04:29.592 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:04:29 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:04:29.592 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:04:29 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:04:29.592 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:04:29 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:04:29.593 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:04:29 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-31T23:04:29.593 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:04:29 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:04:29.593 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:04:29 smithi162 ceph-mon[36237]: pgmap v75: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-31T23:04:29.593 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:04:29 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:04:29.593 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:04:29 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:04:29.593 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:04:29 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-31T23:04:29.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:04:29 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:04:29.862 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:04:29 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:04:29.862 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:04:29 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:04:29.862 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:04:29 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-31T23:04:29.862 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:04:29 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:04:29.862 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:04:29 smithi046 ceph-mon[32498]: pgmap v75: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-31T23:04:29.862 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:04:29 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:04:29.862 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:04:29 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:04:29.862 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:04:29 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-31T23:04:29.911 DEBUG:teuthology.orchestra.run.smithi162:osd.2> sudo journalctl -f -n 0 -u ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4@osd.2.service 2024-03-31T23:04:29.914 INFO:tasks.cephadm:Deploying osd.3 on smithi046 with /dev/vg_nvme/lv_3... 2024-03-31T23:04:29.914 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:384f31f3d3aee7f8ba3f60846b8844c049165583 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6a9d8f08-efb2-11ee-b647-cb9ed24678a4 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-03-31T23:04:29.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:04:29 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:04:29.944 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:04:29 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:04:29.944 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:04:29 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:04:29.944 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:04:29 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-31T23:04:29.944 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:04:29 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:04:29.944 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:04:29 smithi142 ceph-mon[35271]: pgmap v75: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-31T23:04:29.944 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:04:29 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:04:29.944 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:04:29 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:04:29.944 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:04:29 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-31T23:04:30.164 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/6a9d8f08-efb2-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-31T23:04:30.612 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:04:30 smithi046 ceph-mon[32498]: from='osd.2 [v2:172.21.15.162:6800/4084261813,v1:172.21.15.162:6801/4084261813]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-31T23:04:30.612 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:04:30 smithi046 ceph-mon[32498]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-31T23:04:30.795 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:04:30 smithi162 ceph-mon[36237]: from='osd.2 [v2:172.21.15.162:6800/4084261813,v1:172.21.15.162:6801/4084261813]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-31T23:04:30.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:04:30 smithi162 ceph-mon[36237]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-31T23:04:30.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:04:30 smithi142 ceph-mon[35271]: from='osd.2 [v2:172.21.15.162:6800/4084261813,v1:172.21.15.162:6801/4084261813]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-31T23:04:30.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:04:30 smithi142 ceph-mon[35271]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-31T23:04:31.614 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:04:31 smithi046 ceph-mon[32498]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-31T23:04:31.614 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:04:31 smithi046 ceph-mon[32498]: osdmap e16: 3 total, 2 up, 3 in 2024-03-31T23:04:31.614 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:04:31 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-31T23:04:31.615 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:04:31 smithi046 ceph-mon[32498]: from='osd.2 [v2:172.21.15.162:6800/4084261813,v1:172.21.15.162:6801/4084261813]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]: dispatch 2024-03-31T23:04:31.615 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:04:31 smithi046 ceph-mon[32498]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]: dispatch 2024-03-31T23:04:31.615 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:04:31 smithi046 ceph-mon[32498]: pgmap v77: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-31T23:04:31.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:04:31 smithi162 ceph-mon[36237]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-31T23:04:31.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:04:31 smithi162 ceph-mon[36237]: osdmap e16: 3 total, 2 up, 3 in 2024-03-31T23:04:31.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:04:31 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-31T23:04:31.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:04:31 smithi162 ceph-mon[36237]: from='osd.2 [v2:172.21.15.162:6800/4084261813,v1:172.21.15.162:6801/4084261813]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]: dispatch 2024-03-31T23:04:31.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:04:31 smithi162 ceph-mon[36237]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]: dispatch 2024-03-31T23:04:31.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:04:31 smithi162 ceph-mon[36237]: pgmap v77: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-31T23:04:31.797 INFO:journalctl@ceph.osd.2.smithi162.stdout:Mar 31 23:04:31 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-2[41385]: 2024-03-31T23:04:31.504+0000 7f9869269700 -1 osd.2 0 waiting for initial osdmap 2024-03-31T23:04:31.797 INFO:journalctl@ceph.osd.2.smithi162.stdout:Mar 31 23:04:31 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-2[41385]: 2024-03-31T23:04:31.508+0000 7f986505c700 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-31T23:04:31.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:04:31 smithi142 ceph-mon[35271]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-31T23:04:31.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:04:31 smithi142 ceph-mon[35271]: osdmap e16: 3 total, 2 up, 3 in 2024-03-31T23:04:31.944 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:04:31 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-31T23:04:31.944 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:04:31 smithi142 ceph-mon[35271]: from='osd.2 [v2:172.21.15.162:6800/4084261813,v1:172.21.15.162:6801/4084261813]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]: dispatch 2024-03-31T23:04:31.944 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:04:31 smithi142 ceph-mon[35271]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]: dispatch 2024-03-31T23:04:31.944 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:04:31 smithi142 ceph-mon[35271]: pgmap v77: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-31T23:04:32.069 INFO:teuthology.orchestra.run.smithi046.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-03-31T23:04:32.069 INFO:teuthology.orchestra.run.smithi046.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-03-31T23:04:32.069 INFO:teuthology.orchestra.run.smithi046.stderr: stderr: 10+0 records in 2024-03-31T23:04:32.069 INFO:teuthology.orchestra.run.smithi046.stderr:10+0 records out 2024-03-31T23:04:32.069 INFO:teuthology.orchestra.run.smithi046.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.00821507 s, 1.3 GB/s 2024-03-31T23:04:32.069 INFO:teuthology.orchestra.run.smithi046.stderr:--> Zapping successful for: 2024-03-31T23:04:32.665 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:04:32 smithi162 ceph-mon[36237]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]': finished 2024-03-31T23:04:32.665 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:04:32 smithi162 ceph-mon[36237]: osdmap e17: 3 total, 2 up, 3 in 2024-03-31T23:04:32.665 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:04:32 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-31T23:04:32.665 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:04:32 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-31T23:04:32.668 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:04:32 smithi046 ceph-mon[32498]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]': finished 2024-03-31T23:04:32.668 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:04:32 smithi046 ceph-mon[32498]: osdmap e17: 3 total, 2 up, 3 in 2024-03-31T23:04:32.668 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:04:32 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-31T23:04:32.668 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:04:32 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-31T23:04:32.696 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:384f31f3d3aee7f8ba3f60846b8844c049165583 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6a9d8f08-efb2-11ee-b647-cb9ed24678a4 -- ceph orch daemon add osd smithi046:vg_nvme/lv_3 2024-03-31T23:04:32.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:04:32 smithi142 ceph-mon[35271]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]': finished 2024-03-31T23:04:32.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:04:32 smithi142 ceph-mon[35271]: osdmap e17: 3 total, 2 up, 3 in 2024-03-31T23:04:32.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:04:32 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-31T23:04:32.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:04:32 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-31T23:04:32.947 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/6a9d8f08-efb2-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-31T23:04:33.612 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:04:33 smithi046 ceph-mon[32498]: purged_snaps scrub starts 2024-03-31T23:04:33.612 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:04:33 smithi046 ceph-mon[32498]: purged_snaps scrub ok 2024-03-31T23:04:33.612 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:04:33 smithi046 ceph-mon[32498]: osd.2 [v2:172.21.15.162:6800/4084261813,v1:172.21.15.162:6801/4084261813] boot 2024-03-31T23:04:33.612 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:04:33 smithi046 ceph-mon[32498]: osdmap e18: 3 total, 3 up, 3 in 2024-03-31T23:04:33.612 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:04:33 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-31T23:04:33.612 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:04:33 smithi046 ceph-mon[32498]: pgmap v80: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-31T23:04:33.612 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:04:33 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-31T23:04:33.612 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:04:33 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:04:33.612 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:04:33 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:04:33.613 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:04:33 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-31T23:04:33.613 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:04:33 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:04:33.613 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:04:33 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:04:33.613 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:04:33 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-31T23:04:33.613 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:04:33 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:04:33.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:04:33 smithi162 ceph-mon[36237]: purged_snaps scrub starts 2024-03-31T23:04:33.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:04:33 smithi162 ceph-mon[36237]: purged_snaps scrub ok 2024-03-31T23:04:33.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:04:33 smithi162 ceph-mon[36237]: osd.2 [v2:172.21.15.162:6800/4084261813,v1:172.21.15.162:6801/4084261813] boot 2024-03-31T23:04:33.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:04:33 smithi162 ceph-mon[36237]: osdmap e18: 3 total, 3 up, 3 in 2024-03-31T23:04:33.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:04:33 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-31T23:04:33.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:04:33 smithi162 ceph-mon[36237]: pgmap v80: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-31T23:04:33.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:04:33 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-31T23:04:33.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:04:33 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:04:33.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:04:33 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:04:33.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:04:33 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-31T23:04:33.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:04:33 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:04:33.797 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:04:33 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:04:33.797 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:04:33 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-31T23:04:33.797 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:04:33 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:04:33.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:04:33 smithi142 ceph-mon[35271]: purged_snaps scrub starts 2024-03-31T23:04:33.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:04:33 smithi142 ceph-mon[35271]: purged_snaps scrub ok 2024-03-31T23:04:33.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:04:33 smithi142 ceph-mon[35271]: osd.2 [v2:172.21.15.162:6800/4084261813,v1:172.21.15.162:6801/4084261813] boot 2024-03-31T23:04:33.944 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:04:33 smithi142 ceph-mon[35271]: osdmap e18: 3 total, 3 up, 3 in 2024-03-31T23:04:33.944 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:04:33 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-31T23:04:33.944 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:04:33 smithi142 ceph-mon[35271]: pgmap v80: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-31T23:04:33.944 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:04:33 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-31T23:04:33.944 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:04:33 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:04:33.944 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:04:33 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:04:33.944 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:04:33 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-31T23:04:33.944 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:04:33 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:04:33.944 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:04:33 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:04:33.944 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:04:33 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-31T23:04:33.944 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:04:33 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:04:34.550 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:04:34.548+0000 7f0d8e09d700 1 -- 172.21.15.46:0/1879434752 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0d88103470 msgr2=0x7f0d881038d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:04:34.550 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:04:34.548+0000 7f0d8e09d700 1 --2- 172.21.15.46:0/1879434752 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0d88103470 0x7f0d881038d0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f0d7c0099c0 tx=0x7f0d7c009cd0 comp rx=0 tx=0).stop 2024-03-31T23:04:34.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:04:34.548+0000 7f0d8e09d700 1 -- 172.21.15.46:0/1879434752 shutdown_connections 2024-03-31T23:04:34.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:04:34.548+0000 7f0d8e09d700 1 --2- 172.21.15.46:0/1879434752 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0d88103e10 0x7f0d8810a6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:04:34.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:04:34.548+0000 7f0d8e09d700 1 --2- 172.21.15.46:0/1879434752 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0d88102270 0x7f0d88102690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:04:34.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:04:34.548+0000 7f0d8e09d700 1 --2- 172.21.15.46:0/1879434752 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0d88103470 0x7f0d881038d0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:04:34.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:04:34.548+0000 7f0d8e09d700 1 -- 172.21.15.46:0/1879434752 >> 172.21.15.46:0/1879434752 conn(0x7f0d880fd7f0 msgr2=0x7f0d880ffc50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:04:34.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:04:34.548+0000 7f0d8e09d700 1 -- 172.21.15.46:0/1879434752 shutdown_connections 2024-03-31T23:04:34.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:04:34.548+0000 7f0d8e09d700 1 -- 172.21.15.46:0/1879434752 wait complete. 2024-03-31T23:04:34.552 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:04:34 smithi046 ceph-mon[32498]: Detected new or changed devices on smithi162 2024-03-31T23:04:34.552 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:04:34 smithi046 ceph-mon[32498]: Adjusting osd_memory_target on smithi162 to 17088M 2024-03-31T23:04:34.552 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:04:34 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-31T23:04:34.552 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:04:34 smithi046 ceph-mon[32498]: osdmap e19: 3 total, 3 up, 3 in 2024-03-31T23:04:34.553 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:04:34 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-31T23:04:34.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:04:34.549+0000 7f0d8e09d700 1 Processor -- start 2024-03-31T23:04:34.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:04:34.549+0000 7f0d8e09d700 1 -- start start 2024-03-31T23:04:34.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:04:34.550+0000 7f0d8e09d700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0d88102270 0x7f0d88198410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:04:34.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:04:34.550+0000 7f0d8e09d700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0d88103470 0x7f0d88198950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:04:34.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:04:34.550+0000 7f0d8e09d700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0d88103e10 0x7f0d8819f980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:04:34.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:04:34.550+0000 7f0d8e09d700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f0d8810dd10 con 0x7f0d88102270 2024-03-31T23:04:34.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:04:34.550+0000 7f0d8e09d700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f0d8810db90 con 0x7f0d88103470 2024-03-31T23:04:34.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:04:34.550+0000 7f0d8e09d700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f0d8810de90 con 0x7f0d88103e10 2024-03-31T23:04:34.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:04:34.550+0000 7f0d877fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0d88102270 0x7f0d88198410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:04:34.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:04:34.550+0000 7f0d86ffd700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0d88103470 0x7f0d88198950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:04:34.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:04:34.550+0000 7f0d877fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0d88102270 0x7f0d88198410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39942/0 (socket says 172.21.15.46:39942) 2024-03-31T23:04:34.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:04:34.550+0000 7f0d877fe700 1 -- 172.21.15.46:0/2185802971 learned_addr learned my addr 172.21.15.46:0/2185802971 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:04:34.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:04:34.550+0000 7f0d87fff700 1 --2- 172.21.15.46:0/2185802971 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0d88103e10 0x7f0d8819f980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:04:34.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:04:34.550+0000 7f0d877fe700 1 -- 172.21.15.46:0/2185802971 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0d88103e10 msgr2=0x7f0d8819f980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:04:34.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:04:34.550+0000 7f0d877fe700 1 --2- 172.21.15.46:0/2185802971 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0d88103e10 0x7f0d8819f980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:04:34.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:04:34.550+0000 7f0d877fe700 1 -- 172.21.15.46:0/2185802971 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0d88103470 msgr2=0x7f0d88198950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:04:34.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:04:34.550+0000 7f0d877fe700 1 --2- 172.21.15.46:0/2185802971 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0d88103470 0x7f0d88198950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:04:34.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:04:34.550+0000 7f0d877fe700 1 -- 172.21.15.46:0/2185802971 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d881a0020 con 0x7f0d88102270 2024-03-31T23:04:34.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:04:34.550+0000 7f0d86ffd700 1 --2- 172.21.15.46:0/2185802971 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0d88103470 0x7f0d88198950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:04:34.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:04:34.551+0000 7f0d877fe700 1 --2- 172.21.15.46:0/2185802971 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0d88102270 0x7f0d88198410 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f0d7800e980 tx=0x7f0d7800ec90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:04:34.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:04:34.551+0000 7f0d84ff9700 1 -- 172.21.15.46:0/2185802971 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d78004230 con 0x7f0d88102270 2024-03-31T23:04:34.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:04:34.551+0000 7f0d8e09d700 1 -- 172.21.15.46:0/2185802971 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0d880744d0 con 0x7f0d88102270 2024-03-31T23:04:34.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:04:34.551+0000 7f0d84ff9700 1 -- 172.21.15.46:0/2185802971 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0d780043d0 con 0x7f0d88102270 2024-03-31T23:04:34.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:04:34.551+0000 7f0d8e09d700 1 -- 172.21.15.46:0/2185802971 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0d880749b0 con 0x7f0d88102270 2024-03-31T23:04:34.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:04:34.551+0000 7f0d84ff9700 1 -- 172.21.15.46:0/2185802971 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d78003c90 con 0x7f0d88102270 2024-03-31T23:04:34.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:04:34.552+0000 7f0d8e09d700 1 -- 172.21.15.46:0/2185802971 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0d88068f00 con 0x7f0d88102270 2024-03-31T23:04:34.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:04:34.554+0000 7f0d84ff9700 1 -- 172.21.15.46:0/2185802971 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 15) ==== 137264+0+0 (secure 0 0 0) 0x7f0d7800c810 con 0x7f0d88102270 2024-03-31T23:04:34.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:04:34.554+0000 7f0d84ff9700 1 --2- 172.21.15.46:0/2185802971 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f0d700a2520 0x7f0d700a49e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:04:34.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:04:34.554+0000 7f0d86ffd700 1 --2- 172.21.15.46:0/2185802971 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f0d700a2520 0x7f0d700a49e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:04:34.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:04:34.555+0000 7f0d84ff9700 1 -- 172.21.15.46:0/2185802971 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 3237+0+0 (secure 0 0 0) 0x7f0d780cd4c0 con 0x7f0d88102270 2024-03-31T23:04:34.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:04:34.555+0000 7f0d86ffd700 1 --2- 172.21.15.46:0/2185802971 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f0d700a2520 0x7f0d700a49e0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f0d7c000c00 tx=0x7f0d7c005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:04:34.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:04:34.559+0000 7f0d84ff9700 1 -- 172.21.15.46:0/2185802971 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f0d7809bb30 con 0x7f0d88102270 2024-03-31T23:04:34.780 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:04:34.779+0000 7f0d8e09d700 1 -- 172.21.15.46:0/2185802971 --> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi046:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7f0d88075eb0 con 0x7f0d700a2520 2024-03-31T23:04:34.795 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:04:34 smithi162 ceph-mon[36237]: Detected new or changed devices on smithi162 2024-03-31T23:04:34.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:04:34 smithi162 ceph-mon[36237]: Adjusting osd_memory_target on smithi162 to 17088M 2024-03-31T23:04:34.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:04:34 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-31T23:04:34.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:04:34 smithi162 ceph-mon[36237]: osdmap e19: 3 total, 3 up, 3 in 2024-03-31T23:04:34.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:04:34 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-31T23:04:34.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:04:34 smithi142 ceph-mon[35271]: Detected new or changed devices on smithi162 2024-03-31T23:04:34.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:04:34 smithi142 ceph-mon[35271]: Adjusting osd_memory_target on smithi162 to 17088M 2024-03-31T23:04:34.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:04:34 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-31T23:04:34.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:04:34 smithi142 ceph-mon[35271]: osdmap e19: 3 total, 3 up, 3 in 2024-03-31T23:04:34.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:04:34 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-31T23:04:35.129 INFO:journalctl@ceph.osd.0.smithi046.stdout:Mar 31 23:04:34 smithi046 sudo[52035]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-31T23:04:35.130 INFO:journalctl@ceph.osd.0.smithi046.stdout:Mar 31 23:04:34 smithi046 sudo[52035]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-31T23:04:35.130 INFO:journalctl@ceph.osd.0.smithi046.stdout:Mar 31 23:04:34 smithi046 sudo[52035]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-31T23:04:35.130 INFO:journalctl@ceph.osd.0.smithi046.stdout:Mar 31 23:04:34 smithi046 sudo[52035]: pam_unix(sudo:session): session closed for user root 2024-03-31T23:04:35.397 INFO:journalctl@ceph.osd.0.smithi046.stdout:Mar 31 23:04:35 smithi046 sudo[52038]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-31T23:04:35.397 INFO:journalctl@ceph.osd.0.smithi046.stdout:Mar 31 23:04:35 smithi046 sudo[52038]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-31T23:04:35.397 INFO:journalctl@ceph.osd.0.smithi046.stdout:Mar 31 23:04:35 smithi046 sudo[52038]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-31T23:04:35.397 INFO:journalctl@ceph.osd.0.smithi046.stdout:Mar 31 23:04:35 smithi046 sudo[52038]: pam_unix(sudo:session): session closed for user root 2024-03-31T23:04:35.649 INFO:journalctl@ceph.osd.1.smithi142.stdout:Mar 31 23:04:35 smithi142 sudo[43682]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-31T23:04:35.649 INFO:journalctl@ceph.osd.1.smithi142.stdout:Mar 31 23:04:35 smithi142 sudo[43682]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-31T23:04:35.649 INFO:journalctl@ceph.osd.1.smithi142.stdout:Mar 31 23:04:35 smithi142 sudo[43682]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-31T23:04:35.649 INFO:journalctl@ceph.osd.1.smithi142.stdout:Mar 31 23:04:35 smithi142 sudo[43682]: pam_unix(sudo:session): session closed for user root 2024-03-31T23:04:35.649 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:04:35 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-31T23:04:35.649 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:04:35 smithi142 ceph-mon[35271]: osdmap e20: 3 total, 3 up, 3 in 2024-03-31T23:04:35.649 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:04:35 smithi142 ceph-mon[35271]: pgmap v83: 1 pgs: 1 unknown; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-31T23:04:35.650 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:04:35 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-31T23:04:35.650 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:04:35 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-31T23:04:35.650 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:04:35 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:04:35.650 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:04:35 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:04:35.650 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:04:35 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-31T23:04:35.650 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:04:35 smithi046 ceph-mon[32498]: osdmap e20: 3 total, 3 up, 3 in 2024-03-31T23:04:35.650 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:04:35 smithi046 ceph-mon[32498]: pgmap v83: 1 pgs: 1 unknown; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-31T23:04:35.650 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:04:35 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-31T23:04:35.650 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:04:35 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-31T23:04:35.651 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:04:35 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:04:35.651 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:04:35 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:04:35.795 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:04:35 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-31T23:04:35.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:04:35 smithi162 ceph-mon[36237]: osdmap e20: 3 total, 3 up, 3 in 2024-03-31T23:04:35.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:04:35 smithi162 ceph-mon[36237]: pgmap v83: 1 pgs: 1 unknown; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-31T23:04:35.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:04:35 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-31T23:04:35.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:04:35 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-31T23:04:35.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:04:35 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:04:35.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:04:35 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:04:35.943 INFO:journalctl@ceph.osd.1.smithi142.stdout:Mar 31 23:04:35 smithi142 sudo[43685]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-31T23:04:35.943 INFO:journalctl@ceph.osd.1.smithi142.stdout:Mar 31 23:04:35 smithi142 sudo[43685]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-31T23:04:35.943 INFO:journalctl@ceph.osd.1.smithi142.stdout:Mar 31 23:04:35 smithi142 sudo[43685]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-31T23:04:35.944 INFO:journalctl@ceph.osd.1.smithi142.stdout:Mar 31 23:04:35 smithi142 sudo[43685]: pam_unix(sudo:session): session closed for user root 2024-03-31T23:04:36.172 INFO:journalctl@ceph.osd.2.smithi162.stdout:Mar 31 23:04:35 smithi162 sudo[45293]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-31T23:04:36.172 INFO:journalctl@ceph.osd.2.smithi162.stdout:Mar 31 23:04:35 smithi162 sudo[45293]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-31T23:04:36.172 INFO:journalctl@ceph.osd.2.smithi162.stdout:Mar 31 23:04:35 smithi162 sudo[45293]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-31T23:04:36.172 INFO:journalctl@ceph.osd.2.smithi162.stdout:Mar 31 23:04:35 smithi162 sudo[45293]: pam_unix(sudo:session): session closed for user root 2024-03-31T23:04:36.546 INFO:journalctl@ceph.osd.2.smithi162.stdout:Mar 31 23:04:36 smithi162 sudo[45296]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-31T23:04:36.546 INFO:journalctl@ceph.osd.2.smithi162.stdout:Mar 31 23:04:36 smithi162 sudo[45296]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-31T23:04:36.546 INFO:journalctl@ceph.osd.2.smithi162.stdout:Mar 31 23:04:36 smithi162 sudo[45296]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-31T23:04:36.546 INFO:journalctl@ceph.osd.2.smithi162.stdout:Mar 31 23:04:36 smithi162 sudo[45296]: pam_unix(sudo:session): session closed for user root 2024-03-31T23:04:36.609 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:04:36 smithi046 sudo[52142]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-31T23:04:36.609 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:04:36 smithi046 sudo[52142]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-31T23:04:36.609 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:04:36 smithi046 sudo[52142]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-31T23:04:36.609 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:04:36 smithi046 sudo[52142]: pam_unix(sudo:session): session closed for user root 2024-03-31T23:04:36.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:04:36 smithi046 sudo[52150]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-31T23:04:36.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:04:36 smithi046 sudo[52150]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-31T23:04:36.862 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:04:36 smithi046 sudo[52150]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-31T23:04:36.862 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:04:36 smithi046 sudo[52150]: pam_unix(sudo:session): session closed for user root 2024-03-31T23:04:36.862 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:04:36 smithi046 ceph-mon[32498]: from='client.14286 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi046:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-31T23:04:36.862 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:04:36 smithi046 ceph-mon[32498]: osdmap e21: 3 total, 3 up, 3 in 2024-03-31T23:04:36.862 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:04:36 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:04:36.862 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:04:36 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:04:36.862 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:04:36 smithi046 ceph-mon[32498]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-31T23:04:36.988 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:04:36 smithi162 ceph-mon[36237]: from='client.14286 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi046:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-31T23:04:36.988 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:04:36 smithi162 ceph-mon[36237]: osdmap e21: 3 total, 3 up, 3 in 2024-03-31T23:04:36.988 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:04:36 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:04:36.988 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:04:36 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:04:36.988 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:04:36 smithi162 ceph-mon[36237]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-31T23:04:36.988 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:04:36 smithi162 sudo[45299]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-31T23:04:36.988 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:04:36 smithi162 sudo[45299]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-31T23:04:36.988 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:04:36 smithi162 sudo[45299]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-31T23:04:36.988 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:04:36 smithi162 sudo[45299]: pam_unix(sudo:session): session closed for user root 2024-03-31T23:04:37.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:04:36 smithi142 ceph-mon[35271]: from='client.14286 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi046:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-31T23:04:37.194 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:04:36 smithi142 ceph-mon[35271]: osdmap e21: 3 total, 3 up, 3 in 2024-03-31T23:04:37.194 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:04:36 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:04:37.194 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:04:36 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:04:37.194 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:04:36 smithi142 ceph-mon[35271]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-31T23:04:37.298 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:04:37 smithi162 sudo[45302]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-31T23:04:37.298 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:04:37 smithi162 sudo[45302]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-31T23:04:37.298 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:04:37 smithi162 sudo[45302]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-31T23:04:37.298 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:04:37 smithi162 sudo[45302]: pam_unix(sudo:session): session closed for user root 2024-03-31T23:04:37.719 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:04:37 smithi142 sudo[43688]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-31T23:04:37.719 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:04:37 smithi142 sudo[43688]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-31T23:04:37.719 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:04:37 smithi142 sudo[43688]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-31T23:04:37.719 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:04:37 smithi142 sudo[43688]: pam_unix(sudo:session): session closed for user root 2024-03-31T23:04:37.719 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:04:37 smithi142 sudo[43691]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-31T23:04:37.719 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:04:37 smithi142 sudo[43691]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-31T23:04:37.719 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:04:37 smithi142 sudo[43691]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-31T23:04:37.719 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:04:37 smithi142 sudo[43691]: pam_unix(sudo:session): session closed for user root 2024-03-31T23:04:37.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:04:37.728+0000 7f0d84ff9700 1 -- 172.21.15.46:0/2185802971 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mgrmap(e 16) ==== 137320+0+0 (secure 0 0 0) 0x7f0d78097aa0 con 0x7f0d88102270 2024-03-31T23:04:38.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:04:37 smithi162 ceph-mon[36237]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 89 MiB used, 268 GiB / 268 GiB avail 2024-03-31T23:04:38.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:04:37 smithi162 ceph-mon[36237]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-31T23:04:38.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:04:37 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-31T23:04:38.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:04:37 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-31T23:04:38.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:04:37 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-31T23:04:38.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:04:37 smithi162 ceph-mon[36237]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-31T23:04:38.047 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:04:37 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-31T23:04:38.047 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:04:37 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-31T23:04:38.047 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:04:37 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-31T23:04:38.047 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:04:37 smithi162 ceph-mon[36237]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-31T23:04:38.047 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:04:37 smithi162 ceph-mon[36237]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-31T23:04:38.047 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:04:37 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-31T23:04:38.048 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:04:37 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-31T23:04:38.048 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:04:37 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-31T23:04:38.048 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:04:37 smithi162 ceph-mon[36237]: from='client.? 172.21.15.46:0/3781588857' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f877814e-9ef6-4872-a081-5609bb9ebc06"}]: dispatch 2024-03-31T23:04:38.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:04:37 smithi046 ceph-mon[32498]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 89 MiB used, 268 GiB / 268 GiB avail 2024-03-31T23:04:38.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:04:37 smithi046 ceph-mon[32498]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-31T23:04:38.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:04:37 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-31T23:04:38.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:04:37 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-31T23:04:38.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:04:37 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-31T23:04:38.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:04:37 smithi046 ceph-mon[32498]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-31T23:04:38.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:04:37 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-31T23:04:38.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:04:37 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-31T23:04:38.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:04:37 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-31T23:04:38.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:04:37 smithi046 ceph-mon[32498]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-31T23:04:38.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:04:37 smithi046 ceph-mon[32498]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-31T23:04:38.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:04:37 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-31T23:04:38.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:04:37 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-31T23:04:38.113 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:04:37 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-31T23:04:38.113 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:04:37 smithi046 ceph-mon[32498]: from='client.? 172.21.15.46:0/3781588857' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f877814e-9ef6-4872-a081-5609bb9ebc06"}]: dispatch 2024-03-31T23:04:38.194 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:04:37 smithi142 ceph-mon[35271]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 89 MiB used, 268 GiB / 268 GiB avail 2024-03-31T23:04:38.194 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:04:37 smithi142 ceph-mon[35271]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-31T23:04:38.194 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:04:37 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-31T23:04:38.194 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:04:37 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-31T23:04:38.194 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:04:37 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-31T23:04:38.194 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:04:37 smithi142 ceph-mon[35271]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-31T23:04:38.194 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:04:37 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-31T23:04:38.194 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:04:37 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-31T23:04:38.194 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:04:37 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-31T23:04:38.195 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:04:37 smithi142 ceph-mon[35271]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-31T23:04:38.195 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:04:37 smithi142 ceph-mon[35271]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-31T23:04:38.195 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:04:37 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-31T23:04:38.195 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:04:37 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-31T23:04:38.195 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:04:37 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-31T23:04:38.195 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:04:37 smithi142 ceph-mon[35271]: from='client.? 172.21.15.46:0/3781588857' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f877814e-9ef6-4872-a081-5609bb9ebc06"}]: dispatch 2024-03-31T23:04:39.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:04:38 smithi162 ceph-mon[36237]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-31T23:04:39.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:04:38 smithi162 ceph-mon[36237]: mgrmap e16: x(active, since 2m), standbys: y, z 2024-03-31T23:04:39.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:04:38 smithi162 ceph-mon[36237]: from='client.? 172.21.15.46:0/3781588857' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f877814e-9ef6-4872-a081-5609bb9ebc06"}]': finished 2024-03-31T23:04:39.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:04:38 smithi162 ceph-mon[36237]: osdmap e22: 4 total, 3 up, 4 in 2024-03-31T23:04:39.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:04:38 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-31T23:04:39.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:04:38 smithi162 ceph-mon[36237]: from='client.? 172.21.15.46:0/1097247544' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-31T23:04:39.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:04:38 smithi162 ceph-mon[36237]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-31T23:04:39.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:04:38 smithi046 ceph-mon[32498]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-31T23:04:39.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:04:38 smithi046 ceph-mon[32498]: mgrmap e16: x(active, since 2m), standbys: y, z 2024-03-31T23:04:39.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:04:38 smithi046 ceph-mon[32498]: from='client.? 172.21.15.46:0/3781588857' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f877814e-9ef6-4872-a081-5609bb9ebc06"}]': finished 2024-03-31T23:04:39.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:04:38 smithi046 ceph-mon[32498]: osdmap e22: 4 total, 3 up, 4 in 2024-03-31T23:04:39.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:04:38 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-31T23:04:39.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:04:38 smithi046 ceph-mon[32498]: from='client.? 172.21.15.46:0/1097247544' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-31T23:04:39.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:04:38 smithi046 ceph-mon[32498]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-31T23:04:39.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:04:38 smithi142 ceph-mon[35271]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-31T23:04:39.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:04:38 smithi142 ceph-mon[35271]: mgrmap e16: x(active, since 2m), standbys: y, z 2024-03-31T23:04:39.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:04:38 smithi142 ceph-mon[35271]: from='client.? 172.21.15.46:0/3781588857' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f877814e-9ef6-4872-a081-5609bb9ebc06"}]': finished 2024-03-31T23:04:39.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:04:38 smithi142 ceph-mon[35271]: osdmap e22: 4 total, 3 up, 4 in 2024-03-31T23:04:39.194 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:04:38 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-31T23:04:39.194 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:04:38 smithi142 ceph-mon[35271]: from='client.? 172.21.15.46:0/1097247544' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-31T23:04:39.194 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:04:38 smithi142 ceph-mon[35271]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-31T23:04:40.864 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:04:40 smithi046 ceph-mon[32498]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-31T23:04:40.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:04:40 smithi142 ceph-mon[35271]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-31T23:04:41.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:04:40 smithi162 ceph-mon[36237]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-31T23:04:42.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:04:42 smithi046 ceph-mon[32498]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 5.0 KiB/s rd, 218 KiB/s wr, 15 op/s 2024-03-31T23:04:42.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:04:42 smithi142 ceph-mon[35271]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 5.0 KiB/s rd, 218 KiB/s wr, 15 op/s 2024-03-31T23:04:43.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:04:42 smithi162 ceph-mon[36237]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 5.0 KiB/s rd, 218 KiB/s wr, 15 op/s 2024-03-31T23:04:44.611 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:04:44 smithi046 ceph-mon[32498]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail; 3.7 KiB/s rd, 163 KiB/s wr, 11 op/s 2024-03-31T23:04:44.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:04:44 smithi142 ceph-mon[35271]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail; 3.7 KiB/s rd, 163 KiB/s wr, 11 op/s 2024-03-31T23:04:45.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:04:44 smithi162 ceph-mon[36237]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail; 3.7 KiB/s rd, 163 KiB/s wr, 11 op/s 2024-03-31T23:04:45.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:04:45 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-31T23:04:45.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:04:45 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:04:45.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:04:45 smithi046 ceph-mon[32498]: Deploying daemon osd.3 on smithi046 2024-03-31T23:04:45.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:04:45 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-31T23:04:45.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:04:45 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:04:45.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:04:45 smithi142 ceph-mon[35271]: Deploying daemon osd.3 on smithi046 2024-03-31T23:04:46.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:04:45 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-31T23:04:46.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:04:45 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:04:46.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:04:45 smithi162 ceph-mon[36237]: Deploying daemon osd.3 on smithi046 2024-03-31T23:04:46.862 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:04:46 smithi046 ceph-mon[32498]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail; 3.0 KiB/s rd, 131 KiB/s wr, 9 op/s 2024-03-31T23:04:46.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:04:46 smithi142 ceph-mon[35271]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail; 3.0 KiB/s rd, 131 KiB/s wr, 9 op/s 2024-03-31T23:04:47.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:04:46 smithi162 ceph-mon[36237]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail; 3.0 KiB/s rd, 131 KiB/s wr, 9 op/s 2024-03-31T23:04:48.845 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:04:48 smithi046 ceph-mon[32498]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail; 2.8 KiB/s rd, 121 KiB/s wr, 8 op/s 2024-03-31T23:04:48.848 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:04:48 smithi162 ceph-mon[36237]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail; 2.8 KiB/s rd, 121 KiB/s wr, 8 op/s 2024-03-31T23:04:48.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:04:48 smithi142 ceph-mon[35271]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail; 2.8 KiB/s rd, 121 KiB/s wr, 8 op/s 2024-03-31T23:04:50.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:04:50 smithi046 ceph-mon[32498]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail; 2.5 KiB/s rd, 109 KiB/s wr, 7 op/s 2024-03-31T23:04:50.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:04:50 smithi142 ceph-mon[35271]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail; 2.5 KiB/s rd, 109 KiB/s wr, 7 op/s 2024-03-31T23:04:51.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:04:50 smithi162 ceph-mon[36237]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail; 2.5 KiB/s rd, 109 KiB/s wr, 7 op/s 2024-03-31T23:04:51.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:04:51 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-31T23:04:51.862 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:04:51 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:04:51.862 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:04:51 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:04:51.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:04:51 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-31T23:04:51.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:04:51 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:04:51.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:04:51 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:04:52.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:04:51 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-31T23:04:52.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:04:51 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:04:52.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:04:51 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:04:52.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:04:52 smithi046 ceph-mon[32498]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail; 2.5 KiB/s rd, 109 KiB/s wr, 7 op/s 2024-03-31T23:04:52.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:04:52 smithi142 ceph-mon[35271]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail; 2.5 KiB/s rd, 109 KiB/s wr, 7 op/s 2024-03-31T23:04:53.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:04:52 smithi162 ceph-mon[36237]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail; 2.5 KiB/s rd, 109 KiB/s wr, 7 op/s 2024-03-31T23:04:54.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:04:54.338+0000 7f0d84ff9700 1 -- 172.21.15.46:0/2185802971 <== mgr.14152 v2:172.21.15.46:6800/162289569 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f0d88075eb0 con 0x7f0d700a2520 2024-03-31T23:04:54.339 INFO:teuthology.orchestra.run.smithi046.stdout:Created osd(s) 3 on host 'smithi046' 2024-03-31T23:04:54.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:04:54.340+0000 7f0d8e09d700 1 -- 172.21.15.46:0/2185802971 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f0d700a2520 msgr2=0x7f0d700a49e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:04:54.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:04:54.340+0000 7f0d8e09d700 1 --2- 172.21.15.46:0/2185802971 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f0d700a2520 0x7f0d700a49e0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f0d7c000c00 tx=0x7f0d7c005fb0 comp rx=0 tx=0).stop 2024-03-31T23:04:54.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:04:54.340+0000 7f0d8e09d700 1 -- 172.21.15.46:0/2185802971 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0d88102270 msgr2=0x7f0d88198410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:04:54.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:04:54.340+0000 7f0d8e09d700 1 --2- 172.21.15.46:0/2185802971 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0d88102270 0x7f0d88198410 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f0d7800e980 tx=0x7f0d7800ec90 comp rx=0 tx=0).stop 2024-03-31T23:04:54.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:04:54.340+0000 7f0d8e09d700 1 -- 172.21.15.46:0/2185802971 shutdown_connections 2024-03-31T23:04:54.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:04:54.340+0000 7f0d8e09d700 1 --2- 172.21.15.46:0/2185802971 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f0d700a2520 0x7f0d700a49e0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:04:54.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:04:54.340+0000 7f0d8e09d700 1 --2- 172.21.15.46:0/2185802971 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0d88102270 0x7f0d88198410 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:04:54.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:04:54.340+0000 7f0d8e09d700 1 --2- 172.21.15.46:0/2185802971 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0d88103e10 0x7f0d8819f980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:04:54.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:04:54.340+0000 7f0d8e09d700 1 --2- 172.21.15.46:0/2185802971 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0d88103470 0x7f0d88198950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:04:54.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:04:54.340+0000 7f0d8e09d700 1 -- 172.21.15.46:0/2185802971 >> 172.21.15.46:0/2185802971 conn(0x7f0d880fd7f0 msgr2=0x7f0d881067c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:04:54.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:04:54.340+0000 7f0d8e09d700 1 -- 172.21.15.46:0/2185802971 shutdown_connections 2024-03-31T23:04:54.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:04:54.340+0000 7f0d8e09d700 1 -- 172.21.15.46:0/2185802971 wait complete. 2024-03-31T23:04:54.543 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:04:54 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:04:54.544 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:04:54 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:04:54.544 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:04:54 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:04:54.544 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:04:54 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-31T23:04:54.544 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:04:54 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:04:54.544 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:04:54 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:04:54.544 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:04:54 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:04:54.544 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:04:54 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-31T23:04:54.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:04:54 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:04:54.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:04:54 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:04:54.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:04:54 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:04:54.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:04:54 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-31T23:04:54.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:04:54 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:04:54.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:04:54 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:04:54.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:04:54 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:04:54.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:04:54 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-31T23:04:54.946 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:04:54 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:04:54.946 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:04:54 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:04:54.947 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:04:54 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:04:54.947 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:04:54 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-31T23:04:54.947 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:04:54 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:04:54.947 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:04:54 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:04:54.947 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:04:54 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:04:54.947 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:04:54 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-31T23:04:54.954 DEBUG:teuthology.orchestra.run.smithi046:osd.3> sudo journalctl -f -n 0 -u ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4@osd.3.service 2024-03-31T23:04:54.956 INFO:tasks.cephadm:Deploying osd.4 on smithi142 with /dev/vg_nvme/lv_3... 2024-03-31T23:04:54.957 DEBUG:teuthology.orchestra.run.smithi142:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:384f31f3d3aee7f8ba3f60846b8844c049165583 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6a9d8f08-efb2-11ee-b647-cb9ed24678a4 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-03-31T23:04:55.202 INFO:teuthology.orchestra.run.smithi142.stderr:Inferring config /var/lib/ceph/6a9d8f08-efb2-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-31T23:04:55.625 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:04:55 smithi142 ceph-mon[35271]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail 2024-03-31T23:04:55.625 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:04:55 smithi142 ceph-mon[35271]: from='osd.3 [v2:172.21.15.46:6810/3496317970,v1:172.21.15.46:6811/3496317970]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-31T23:04:55.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:04:55 smithi162 ceph-mon[36237]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail 2024-03-31T23:04:55.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:04:55 smithi162 ceph-mon[36237]: from='osd.3 [v2:172.21.15.46:6810/3496317970,v1:172.21.15.46:6811/3496317970]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-31T23:04:55.826 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:04:55 smithi046 ceph-mon[32498]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail 2024-03-31T23:04:55.826 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:04:55 smithi046 ceph-mon[32498]: from='osd.3 [v2:172.21.15.46:6810/3496317970,v1:172.21.15.46:6811/3496317970]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-31T23:04:56.556 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:04:56 smithi142 ceph-mon[35271]: from='osd.3 [v2:172.21.15.46:6810/3496317970,v1:172.21.15.46:6811/3496317970]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-31T23:04:56.556 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:04:56 smithi142 ceph-mon[35271]: osdmap e23: 4 total, 3 up, 4 in 2024-03-31T23:04:56.556 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:04:56 smithi142 ceph-mon[35271]: from='osd.3 [v2:172.21.15.46:6810/3496317970,v1:172.21.15.46:6811/3496317970]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi046", "root=default"]}]: dispatch 2024-03-31T23:04:56.556 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:04:56 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-31T23:04:56.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:04:56 smithi162 ceph-mon[36237]: from='osd.3 [v2:172.21.15.46:6810/3496317970,v1:172.21.15.46:6811/3496317970]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-31T23:04:56.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:04:56 smithi162 ceph-mon[36237]: osdmap e23: 4 total, 3 up, 4 in 2024-03-31T23:04:56.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:04:56 smithi162 ceph-mon[36237]: from='osd.3 [v2:172.21.15.46:6810/3496317970,v1:172.21.15.46:6811/3496317970]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi046", "root=default"]}]: dispatch 2024-03-31T23:04:56.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:04:56 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-31T23:04:56.862 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:04:56 smithi046 ceph-mon[32498]: from='osd.3 [v2:172.21.15.46:6810/3496317970,v1:172.21.15.46:6811/3496317970]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-31T23:04:56.862 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:04:56 smithi046 ceph-mon[32498]: osdmap e23: 4 total, 3 up, 4 in 2024-03-31T23:04:56.862 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:04:56 smithi046 ceph-mon[32498]: from='osd.3 [v2:172.21.15.46:6810/3496317970,v1:172.21.15.46:6811/3496317970]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi046", "root=default"]}]: dispatch 2024-03-31T23:04:56.862 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:04:56 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-31T23:04:56.862 INFO:journalctl@ceph.osd.3.smithi046.stdout:Mar 31 23:04:56 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-3[54227]: 2024-03-31T23:04:56.530+0000 7f3d981a5700 -1 osd.3 0 waiting for initial osdmap 2024-03-31T23:04:56.862 INFO:journalctl@ceph.osd.3.smithi046.stdout:Mar 31 23:04:56 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-3[54227]: 2024-03-31T23:04:56.536+0000 7f3d93f98700 -1 osd.3 24 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-31T23:04:57.145 INFO:teuthology.orchestra.run.smithi142.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-03-31T23:04:57.145 INFO:teuthology.orchestra.run.smithi142.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-03-31T23:04:57.145 INFO:teuthology.orchestra.run.smithi142.stderr: stderr: 10+0 records in 2024-03-31T23:04:57.145 INFO:teuthology.orchestra.run.smithi142.stderr:10+0 records out 2024-03-31T23:04:57.146 INFO:teuthology.orchestra.run.smithi142.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0114805 s, 913 MB/s 2024-03-31T23:04:57.146 INFO:teuthology.orchestra.run.smithi142.stderr:--> Zapping successful for: 2024-03-31T23:04:57.684 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:04:57 smithi046 ceph-mon[32498]: from='osd.3 [v2:172.21.15.46:6810/3496317970,v1:172.21.15.46:6811/3496317970]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi046", "root=default"]}]': finished 2024-03-31T23:04:57.684 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:04:57 smithi046 ceph-mon[32498]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail 2024-03-31T23:04:57.684 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:04:57 smithi046 ceph-mon[32498]: osdmap e24: 4 total, 3 up, 4 in 2024-03-31T23:04:57.684 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:04:57 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-31T23:04:57.684 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:04:57 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-31T23:04:57.761 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:04:57 smithi142 ceph-mon[35271]: from='osd.3 [v2:172.21.15.46:6810/3496317970,v1:172.21.15.46:6811/3496317970]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi046", "root=default"]}]': finished 2024-03-31T23:04:57.761 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:04:57 smithi142 ceph-mon[35271]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail 2024-03-31T23:04:57.761 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:04:57 smithi142 ceph-mon[35271]: osdmap e24: 4 total, 3 up, 4 in 2024-03-31T23:04:57.762 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:04:57 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-31T23:04:57.762 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:04:57 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-31T23:04:57.794 DEBUG:teuthology.orchestra.run.smithi142:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:384f31f3d3aee7f8ba3f60846b8844c049165583 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6a9d8f08-efb2-11ee-b647-cb9ed24678a4 -- ceph orch daemon add osd smithi142:vg_nvme/lv_3 2024-03-31T23:04:57.795 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:04:57 smithi162 ceph-mon[36237]: from='osd.3 [v2:172.21.15.46:6810/3496317970,v1:172.21.15.46:6811/3496317970]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi046", "root=default"]}]': finished 2024-03-31T23:04:57.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:04:57 smithi162 ceph-mon[36237]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail 2024-03-31T23:04:57.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:04:57 smithi162 ceph-mon[36237]: osdmap e24: 4 total, 3 up, 4 in 2024-03-31T23:04:57.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:04:57 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-31T23:04:57.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:04:57 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-31T23:04:58.032 INFO:teuthology.orchestra.run.smithi142.stderr:Inferring config /var/lib/ceph/6a9d8f08-efb2-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-31T23:04:58.792 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:04:58 smithi142 ceph-mon[35271]: purged_snaps scrub starts 2024-03-31T23:04:58.792 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:04:58 smithi142 ceph-mon[35271]: purged_snaps scrub ok 2024-03-31T23:04:58.792 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:04:58 smithi142 ceph-mon[35271]: osd.3 [v2:172.21.15.46:6810/3496317970,v1:172.21.15.46:6811/3496317970] boot 2024-03-31T23:04:58.792 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:04:58 smithi142 ceph-mon[35271]: osdmap e25: 4 total, 4 up, 4 in 2024-03-31T23:04:58.792 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:04:58 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-31T23:04:58.792 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:04:58 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:04:58.793 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:04:58 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:04:58.793 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:04:58 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-31T23:04:58.793 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:04:58 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-31T23:04:58.793 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:04:58 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:04:58.793 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:04:58 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:04:58.793 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:04:58 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-31T23:04:58.793 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:04:58 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:04:58.795 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:04:58 smithi162 ceph-mon[36237]: purged_snaps scrub starts 2024-03-31T23:04:58.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:04:58 smithi162 ceph-mon[36237]: purged_snaps scrub ok 2024-03-31T23:04:58.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:04:58 smithi162 ceph-mon[36237]: osd.3 [v2:172.21.15.46:6810/3496317970,v1:172.21.15.46:6811/3496317970] boot 2024-03-31T23:04:58.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:04:58 smithi162 ceph-mon[36237]: osdmap e25: 4 total, 4 up, 4 in 2024-03-31T23:04:58.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:04:58 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-31T23:04:58.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:04:58 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:04:58.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:04:58 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:04:58.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:04:58 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-31T23:04:58.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:04:58 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-31T23:04:58.797 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:04:58 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:04:58.797 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:04:58 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:04:58.797 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:04:58 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-31T23:04:58.797 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:04:58 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:04:58.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:04:58 smithi046 ceph-mon[32498]: purged_snaps scrub starts 2024-03-31T23:04:58.862 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:04:58 smithi046 ceph-mon[32498]: purged_snaps scrub ok 2024-03-31T23:04:58.862 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:04:58 smithi046 ceph-mon[32498]: osd.3 [v2:172.21.15.46:6810/3496317970,v1:172.21.15.46:6811/3496317970] boot 2024-03-31T23:04:58.862 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:04:58 smithi046 ceph-mon[32498]: osdmap e25: 4 total, 4 up, 4 in 2024-03-31T23:04:58.862 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:04:58 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-31T23:04:58.862 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:04:58 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:04:58.862 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:04:58 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:04:58.862 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:04:58 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-31T23:04:58.862 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:04:58 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-31T23:04:58.862 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:04:58 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:04:58.863 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:04:58 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:04:58.863 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:04:58 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-31T23:04:58.863 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:04:58 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:04:59.577 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-31T23:04:59.574+0000 7f780dfd1700 1 -- 172.21.15.142:0/981261035 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7808102410 msgr2=0x7f7808102870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:04:59.577 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-31T23:04:59.574+0000 7f780dfd1700 1 --2- 172.21.15.142:0/981261035 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7808102410 0x7f7808102870 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f77f00099c0 tx=0x7f77f0009cd0 comp rx=0 tx=0).stop 2024-03-31T23:04:59.578 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-31T23:04:59.574+0000 7f780dfd1700 1 -- 172.21.15.142:0/981261035 shutdown_connections 2024-03-31T23:04:59.578 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-31T23:04:59.574+0000 7f780dfd1700 1 --2- 172.21.15.142:0/981261035 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7808101210 0x7f7808101630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:04:59.578 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-31T23:04:59.574+0000 7f780dfd1700 1 --2- 172.21.15.142:0/981261035 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7808102db0 0x7f7808109660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:04:59.578 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-31T23:04:59.574+0000 7f780dfd1700 1 --2- 172.21.15.142:0/981261035 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7808102410 0x7f7808102870 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:04:59.578 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-31T23:04:59.574+0000 7f780dfd1700 1 -- 172.21.15.142:0/981261035 >> 172.21.15.142:0/981261035 conn(0x7f78080fc7d0 msgr2=0x7f78080febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:04:59.578 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-31T23:04:59.574+0000 7f780dfd1700 1 -- 172.21.15.142:0/981261035 shutdown_connections 2024-03-31T23:04:59.578 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-31T23:04:59.574+0000 7f780dfd1700 1 -- 172.21.15.142:0/981261035 wait complete. 2024-03-31T23:04:59.578 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-31T23:04:59.575+0000 7f780dfd1700 1 Processor -- start 2024-03-31T23:04:59.578 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-31T23:04:59.575+0000 7f780dfd1700 1 -- start start 2024-03-31T23:04:59.578 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-31T23:04:59.575+0000 7f780dfd1700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7808101210 0x7f780806b4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:04:59.578 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-31T23:04:59.575+0000 7f780dfd1700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7808102410 0x7f780819f530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:04:59.578 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-31T23:04:59.575+0000 7f780dfd1700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7808102db0 0x7f78081a1a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:04:59.579 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-31T23:04:59.575+0000 7f780dfd1700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f780810cc40 con 0x7f7808102db0 2024-03-31T23:04:59.579 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-31T23:04:59.575+0000 7f780dfd1700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f780810cac0 con 0x7f7808102410 2024-03-31T23:04:59.579 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-31T23:04:59.575+0000 7f780dfd1700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f780810cdc0 con 0x7f7808101210 2024-03-31T23:04:59.579 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-31T23:04:59.575+0000 7f77feffd700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7808102410 0x7f780819f530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:04:59.579 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-31T23:04:59.575+0000 7f77feffd700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7808102410 0x7f780819f530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:36370/0 (socket says 172.21.15.142:36370) 2024-03-31T23:04:59.579 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-31T23:04:59.575+0000 7f78077fe700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7808101210 0x7f780806b4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:04:59.579 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-31T23:04:59.575+0000 7f77feffd700 1 -- 172.21.15.142:0/3716392621 learned_addr learned my addr 172.21.15.142:0/3716392621 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-03-31T23:04:59.579 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-31T23:04:59.576+0000 7f7807fff700 1 --2- 172.21.15.142:0/3716392621 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7808102db0 0x7f78081a1a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:04:59.579 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-31T23:04:59.576+0000 7f77feffd700 1 -- 172.21.15.142:0/3716392621 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7808101210 msgr2=0x7f780806b4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:04:59.579 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-31T23:04:59.576+0000 7f77feffd700 1 --2- 172.21.15.142:0/3716392621 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7808101210 0x7f780806b4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:04:59.579 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-31T23:04:59.576+0000 7f77feffd700 1 -- 172.21.15.142:0/3716392621 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7808102db0 msgr2=0x7f78081a1a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:04:59.579 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-31T23:04:59.576+0000 7f77feffd700 1 --2- 172.21.15.142:0/3716392621 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7808102db0 0x7f78081a1a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:04:59.580 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-31T23:04:59.576+0000 7f77feffd700 1 -- 172.21.15.142:0/3716392621 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78081a2260 con 0x7f7808102410 2024-03-31T23:04:59.580 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-31T23:04:59.576+0000 7f77feffd700 1 --2- 172.21.15.142:0/3716392621 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7808102410 0x7f780819f530 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f77f0005910 tx=0x7f77f000fa00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:04:59.580 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-31T23:04:59.576+0000 7f78057fa700 1 -- 172.21.15.142:0/3716392621 <== mon.2 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77f0027070 con 0x7f7808102410 2024-03-31T23:04:59.580 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-31T23:04:59.576+0000 7f78057fa700 1 -- 172.21.15.142:0/3716392621 <== mon.2 v2:172.21.15.142:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f77f0022920 con 0x7f7808102410 2024-03-31T23:04:59.580 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-31T23:04:59.576+0000 7f78057fa700 1 -- 172.21.15.142:0/3716392621 <== mon.2 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77f0009e60 con 0x7f7808102410 2024-03-31T23:04:59.580 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-31T23:04:59.576+0000 7f780dfd1700 1 -- 172.21.15.142:0/3716392621 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f78081a24f0 con 0x7f7808102410 2024-03-31T23:04:59.580 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-31T23:04:59.576+0000 7f780dfd1700 1 -- 172.21.15.142:0/3716392621 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f78081a29a0 con 0x7f7808102410 2024-03-31T23:04:59.581 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-31T23:04:59.577+0000 7f780dfd1700 1 -- 172.21.15.142:0/3716392621 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7808068f00 con 0x7f7808102410 2024-03-31T23:04:59.581 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-31T23:04:59.578+0000 7f78057fa700 1 -- 172.21.15.142:0/3716392621 <== mon.2 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 16) ==== 137320+0+0 (secure 0 0 0) 0x7f77f002a070 con 0x7f7808102410 2024-03-31T23:04:59.581 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-31T23:04:59.579+0000 7f78057fa700 1 --2- 172.21.15.142:0/3716392621 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f77f40a25b0 0x7f77f40a4a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:04:59.581 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-31T23:04:59.579+0000 7f78077fe700 1 --2- 172.21.15.142:0/3716392621 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f77f40a25b0 0x7f77f40a4a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:04:59.581 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-31T23:04:59.579+0000 7f78057fa700 1 -- 172.21.15.142:0/3716392621 <== mon.2 v2:172.21.15.142:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 3698+0+0 (secure 0 0 0) 0x7f77f00ce5f0 con 0x7f7808102410 2024-03-31T23:04:59.581 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-31T23:04:59.580+0000 7f78077fe700 1 --2- 172.21.15.142:0/3716392621 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f77f40a25b0 0x7f77f40a4a70 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f7808102270 tx=0x7f77f8009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:04:59.583 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-31T23:04:59.583+0000 7f78057fa700 1 -- 172.21.15.142:0/3716392621 <== mon.2 v2:172.21.15.142:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f77f0013070 con 0x7f7808102410 2024-03-31T23:04:59.693 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:04:59 smithi142 ceph-mon[35271]: Detected new or changed devices on smithi046 2024-03-31T23:04:59.693 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:04:59 smithi142 ceph-mon[35271]: Adjusting osd_memory_target on smithi046 to 8546M 2024-03-31T23:04:59.693 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:04:59 smithi142 ceph-mon[35271]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-31T23:04:59.693 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:04:59 smithi142 ceph-mon[35271]: osdmap e26: 4 total, 4 up, 4 in 2024-03-31T23:04:59.788 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-31T23:04:59.787+0000 7f780dfd1700 1 -- 172.21.15.142:0/3716392621 --> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi142:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7f7808062740 con 0x7f77f40a25b0 2024-03-31T23:04:59.795 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:04:59 smithi162 ceph-mon[36237]: Detected new or changed devices on smithi046 2024-03-31T23:04:59.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:04:59 smithi162 ceph-mon[36237]: Adjusting osd_memory_target on smithi046 to 8546M 2024-03-31T23:04:59.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:04:59 smithi162 ceph-mon[36237]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-31T23:04:59.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:04:59 smithi162 ceph-mon[36237]: osdmap e26: 4 total, 4 up, 4 in 2024-03-31T23:04:59.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:04:59 smithi046 ceph-mon[32498]: Detected new or changed devices on smithi046 2024-03-31T23:04:59.862 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:04:59 smithi046 ceph-mon[32498]: Adjusting osd_memory_target on smithi046 to 8546M 2024-03-31T23:04:59.862 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:04:59 smithi046 ceph-mon[32498]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-31T23:04:59.862 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:04:59 smithi046 ceph-mon[32498]: osdmap e26: 4 total, 4 up, 4 in 2024-03-31T23:05:00.666 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:05:00 smithi142 ceph-mon[35271]: osdmap e27: 4 total, 4 up, 4 in 2024-03-31T23:05:00.666 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:05:00 smithi142 ceph-mon[35271]: from='client.24227 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi142:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-31T23:05:00.666 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:05:00 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-31T23:05:00.666 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:05:00 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-31T23:05:00.666 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:05:00 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:05:00.666 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:05:00 smithi142 ceph-mon[35271]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 117 MiB used, 357 GiB / 358 GiB avail 2024-03-31T23:05:00.862 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:05:00 smithi046 ceph-mon[32498]: osdmap e27: 4 total, 4 up, 4 in 2024-03-31T23:05:00.862 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:05:00 smithi046 ceph-mon[32498]: from='client.24227 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi142:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-31T23:05:00.862 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:05:00 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-31T23:05:00.862 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:05:00 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-31T23:05:00.862 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:05:00 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:05:00.862 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:05:00 smithi046 ceph-mon[32498]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 117 MiB used, 357 GiB / 358 GiB avail 2024-03-31T23:05:01.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:05:00 smithi162 ceph-mon[36237]: osdmap e27: 4 total, 4 up, 4 in 2024-03-31T23:05:01.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:05:00 smithi162 ceph-mon[36237]: from='client.24227 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi142:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-31T23:05:01.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:05:00 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-31T23:05:01.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:05:00 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-31T23:05:01.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:05:00 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:05:01.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:05:00 smithi162 ceph-mon[36237]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 117 MiB used, 357 GiB / 358 GiB avail 2024-03-31T23:05:01.682 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:05:01 smithi142 ceph-mon[35271]: osdmap e28: 4 total, 4 up, 4 in 2024-03-31T23:05:01.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:05:01 smithi046 ceph-mon[32498]: osdmap e28: 4 total, 4 up, 4 in 2024-03-31T23:05:02.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:05:01 smithi162 ceph-mon[36237]: osdmap e28: 4 total, 4 up, 4 in 2024-03-31T23:05:02.857 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:05:02 smithi142 ceph-mon[35271]: osdmap e29: 4 total, 4 up, 4 in 2024-03-31T23:05:02.857 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:05:02 smithi142 ceph-mon[35271]: from='client.? 172.21.15.142:0/1763742668' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "beecec0a-8280-44fe-8f5f-8498a35eca8f"}]: dispatch 2024-03-31T23:05:02.857 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:05:02 smithi142 ceph-mon[35271]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "beecec0a-8280-44fe-8f5f-8498a35eca8f"}]: dispatch 2024-03-31T23:05:02.857 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:05:02 smithi142 ceph-mon[35271]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "beecec0a-8280-44fe-8f5f-8498a35eca8f"}]': finished 2024-03-31T23:05:02.857 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:05:02 smithi142 ceph-mon[35271]: osdmap e30: 5 total, 4 up, 5 in 2024-03-31T23:05:02.857 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:05:02 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-31T23:05:02.857 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:05:02 smithi142 ceph-mon[35271]: pgmap v107: 1 pgs: 1 peering; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-31T23:05:02.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:05:02 smithi046 ceph-mon[32498]: osdmap e29: 4 total, 4 up, 4 in 2024-03-31T23:05:02.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:05:02 smithi046 ceph-mon[32498]: from='client.? 172.21.15.142:0/1763742668' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "beecec0a-8280-44fe-8f5f-8498a35eca8f"}]: dispatch 2024-03-31T23:05:02.862 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:05:02 smithi046 ceph-mon[32498]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "beecec0a-8280-44fe-8f5f-8498a35eca8f"}]: dispatch 2024-03-31T23:05:02.862 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:05:02 smithi046 ceph-mon[32498]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "beecec0a-8280-44fe-8f5f-8498a35eca8f"}]': finished 2024-03-31T23:05:02.862 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:05:02 smithi046 ceph-mon[32498]: osdmap e30: 5 total, 4 up, 5 in 2024-03-31T23:05:02.862 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:05:02 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-31T23:05:02.862 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:05:02 smithi046 ceph-mon[32498]: pgmap v107: 1 pgs: 1 peering; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-31T23:05:03.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:05:02 smithi162 ceph-mon[36237]: osdmap e29: 4 total, 4 up, 4 in 2024-03-31T23:05:03.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:05:02 smithi162 ceph-mon[36237]: from='client.? 172.21.15.142:0/1763742668' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "beecec0a-8280-44fe-8f5f-8498a35eca8f"}]: dispatch 2024-03-31T23:05:03.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:05:02 smithi162 ceph-mon[36237]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "beecec0a-8280-44fe-8f5f-8498a35eca8f"}]: dispatch 2024-03-31T23:05:03.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:05:02 smithi162 ceph-mon[36237]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "beecec0a-8280-44fe-8f5f-8498a35eca8f"}]': finished 2024-03-31T23:05:03.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:05:02 smithi162 ceph-mon[36237]: osdmap e30: 5 total, 4 up, 5 in 2024-03-31T23:05:03.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:05:02 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-31T23:05:03.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:05:02 smithi162 ceph-mon[36237]: pgmap v107: 1 pgs: 1 peering; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-31T23:05:03.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:05:03 smithi046 ceph-mon[32498]: from='client.? 172.21.15.142:0/4123784500' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-31T23:05:03.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:05:03 smithi142 ceph-mon[35271]: from='client.? 172.21.15.142:0/4123784500' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-31T23:05:04.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:05:03 smithi162 ceph-mon[36237]: from='client.? 172.21.15.142:0/4123784500' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-31T23:05:04.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:05:04 smithi046 ceph-mon[32498]: pgmap v108: 1 pgs: 1 peering; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail; 116 KiB/s, 0 objects/s recovering 2024-03-31T23:05:04.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:05:04 smithi142 ceph-mon[35271]: pgmap v108: 1 pgs: 1 peering; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail; 116 KiB/s, 0 objects/s recovering 2024-03-31T23:05:05.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:05:04 smithi162 ceph-mon[36237]: pgmap v108: 1 pgs: 1 peering; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail; 116 KiB/s, 0 objects/s recovering 2024-03-31T23:05:06.770 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:05:06 smithi142 ceph-mon[35271]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-31T23:05:06.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:05:06 smithi046 ceph-mon[32498]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-31T23:05:07.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:05:06 smithi162 ceph-mon[36237]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-31T23:05:08.693 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:05:08 smithi142 ceph-mon[35271]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-31T23:05:08.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:05:08 smithi046 ceph-mon[32498]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-31T23:05:09.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:05:08 smithi162 ceph-mon[36237]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-31T23:05:09.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:05:09 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-31T23:05:09.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:05:09 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:05:09.949 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:05:09 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-31T23:05:09.949 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:05:09 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:05:10.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:05:09 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-31T23:05:10.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:05:09 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:05:10.830 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:05:10 smithi142 ceph-mon[35271]: Deploying daemon osd.4 on smithi142 2024-03-31T23:05:10.831 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:05:10 smithi142 ceph-mon[35271]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-31T23:05:11.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:05:10 smithi162 ceph-mon[36237]: Deploying daemon osd.4 on smithi142 2024-03-31T23:05:11.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:05:10 smithi162 ceph-mon[36237]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-31T23:05:11.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:05:10 smithi046 ceph-mon[32498]: Deploying daemon osd.4 on smithi142 2024-03-31T23:05:11.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:05:10 smithi046 ceph-mon[32498]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-31T23:05:12.693 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:05:12 smithi142 ceph-mon[35271]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-31T23:05:12.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:05:12 smithi046 ceph-mon[32498]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-31T23:05:13.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:05:12 smithi162 ceph-mon[36237]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-31T23:05:14.693 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:05:14 smithi142 ceph-mon[35271]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-31T23:05:14.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:05:14 smithi046 ceph-mon[32498]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-31T23:05:15.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:05:14 smithi162 ceph-mon[36237]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-31T23:05:16.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:05:15 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:05:16.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:05:15 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-31T23:05:16.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:05:15 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:05:16.296 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:05:16 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:05:16.296 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:05:16 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-31T23:05:16.296 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:05:16 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:05:16.361 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:05:15 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:05:16.362 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:05:15 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-31T23:05:16.362 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:05:15 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:05:17.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:05:17 smithi142 ceph-mon[35271]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-31T23:05:17.296 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:05:17 smithi162 ceph-mon[36237]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-31T23:05:17.361 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:05:17 smithi046 ceph-mon[32498]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-31T23:05:18.152 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-31T23:05:18.151+0000 7f78057fa700 1 -- 172.21.15.142:0/3716392621 <== mgr.14152 v2:172.21.15.46:6800/162289569 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f7808062740 con 0x7f77f40a25b0 2024-03-31T23:05:18.152 INFO:teuthology.orchestra.run.smithi142.stdout:Created osd(s) 4 on host 'smithi142' 2024-03-31T23:05:18.155 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-31T23:05:18.154+0000 7f780dfd1700 1 -- 172.21.15.142:0/3716392621 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f77f40a25b0 msgr2=0x7f77f40a4a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:05:18.155 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-31T23:05:18.154+0000 7f780dfd1700 1 --2- 172.21.15.142:0/3716392621 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f77f40a25b0 0x7f77f40a4a70 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f7808102270 tx=0x7f77f8009260 comp rx=0 tx=0).stop 2024-03-31T23:05:18.155 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-31T23:05:18.154+0000 7f780dfd1700 1 -- 172.21.15.142:0/3716392621 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7808102410 msgr2=0x7f780819f530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:05:18.155 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-31T23:05:18.154+0000 7f780dfd1700 1 --2- 172.21.15.142:0/3716392621 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7808102410 0x7f780819f530 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f77f0005910 tx=0x7f77f000fa00 comp rx=0 tx=0).stop 2024-03-31T23:05:18.156 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-31T23:05:18.154+0000 7f780dfd1700 1 -- 172.21.15.142:0/3716392621 shutdown_connections 2024-03-31T23:05:18.156 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-31T23:05:18.154+0000 7f780dfd1700 1 --2- 172.21.15.142:0/3716392621 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f77f40a25b0 0x7f77f40a4a70 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:05:18.156 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-31T23:05:18.154+0000 7f780dfd1700 1 --2- 172.21.15.142:0/3716392621 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7808102db0 0x7f78081a1a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:05:18.156 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-31T23:05:18.154+0000 7f780dfd1700 1 --2- 172.21.15.142:0/3716392621 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7808101210 0x7f780806b4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:05:18.156 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-31T23:05:18.154+0000 7f780dfd1700 1 --2- 172.21.15.142:0/3716392621 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7808102410 0x7f780819f530 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:05:18.156 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-31T23:05:18.154+0000 7f780dfd1700 1 -- 172.21.15.142:0/3716392621 >> 172.21.15.142:0/3716392621 conn(0x7f78080fc7d0 msgr2=0x7f78081057e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:05:18.156 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-31T23:05:18.154+0000 7f780dfd1700 1 -- 172.21.15.142:0/3716392621 shutdown_connections 2024-03-31T23:05:18.156 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-31T23:05:18.154+0000 7f780dfd1700 1 -- 172.21.15.142:0/3716392621 wait complete. 2024-03-31T23:05:19.091 DEBUG:teuthology.orchestra.run.smithi142:osd.4> sudo journalctl -f -n 0 -u ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4@osd.4.service 2024-03-31T23:05:19.094 INFO:tasks.cephadm:Deploying osd.5 on smithi162 with /dev/vg_nvme/lv_3... 2024-03-31T23:05:19.094 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:384f31f3d3aee7f8ba3f60846b8844c049165583 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6a9d8f08-efb2-11ee-b647-cb9ed24678a4 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-03-31T23:05:19.339 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/6a9d8f08-efb2-11ee-b647-cb9ed24678a4/mon.c/config 2024-03-31T23:05:19.377 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:05:19 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:05:19.377 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:05:19 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:05:19.377 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:05:19 smithi142 ceph-mon[35271]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-31T23:05:19.377 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:05:19 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:05:19.377 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:05:19 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:05:19.377 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:05:19 smithi142 ceph-mon[35271]: from='osd.4 [v2:172.21.15.142:6808/2996266929,v1:172.21.15.142:6809/2996266929]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-31T23:05:19.378 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:05:19 smithi142 ceph-mon[35271]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-31T23:05:19.459 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:05:19 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:05:19.459 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:05:19 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:05:19.459 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:05:19 smithi162 ceph-mon[36237]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-31T23:05:19.459 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:05:19 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:05:19.460 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:05:19 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:05:19.460 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:05:19 smithi162 ceph-mon[36237]: from='osd.4 [v2:172.21.15.142:6808/2996266929,v1:172.21.15.142:6809/2996266929]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-31T23:05:19.460 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:05:19 smithi162 ceph-mon[36237]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-31T23:05:19.612 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:05:19 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:05:19.612 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:05:19 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:05:19.612 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:05:19 smithi046 ceph-mon[32498]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-31T23:05:19.612 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:05:19 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:05:19.612 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:05:19 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:05:19.612 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:05:19 smithi046 ceph-mon[32498]: from='osd.4 [v2:172.21.15.142:6808/2996266929,v1:172.21.15.142:6809/2996266929]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-31T23:05:19.612 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:05:19 smithi046 ceph-mon[32498]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-31T23:05:20.944 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:05:20 smithi142 ceph-mon[35271]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-31T23:05:20.944 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:05:20 smithi142 ceph-mon[35271]: osdmap e31: 5 total, 4 up, 5 in 2024-03-31T23:05:20.944 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:05:20 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-31T23:05:20.944 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:05:20 smithi142 ceph-mon[35271]: from='osd.4 [v2:172.21.15.142:6808/2996266929,v1:172.21.15.142:6809/2996266929]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi142", "root=default"]}]: dispatch 2024-03-31T23:05:20.944 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:05:20 smithi142 ceph-mon[35271]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi142", "root=default"]}]: dispatch 2024-03-31T23:05:20.944 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:05:20 smithi142 ceph-mon[35271]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-31T23:05:21.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:05:20 smithi162 ceph-mon[36237]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-31T23:05:21.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:05:20 smithi162 ceph-mon[36237]: osdmap e31: 5 total, 4 up, 5 in 2024-03-31T23:05:21.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:05:20 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-31T23:05:21.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:05:20 smithi162 ceph-mon[36237]: from='osd.4 [v2:172.21.15.142:6808/2996266929,v1:172.21.15.142:6809/2996266929]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi142", "root=default"]}]: dispatch 2024-03-31T23:05:21.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:05:20 smithi162 ceph-mon[36237]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi142", "root=default"]}]: dispatch 2024-03-31T23:05:21.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:05:20 smithi162 ceph-mon[36237]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-31T23:05:21.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:05:20 smithi046 ceph-mon[32498]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-31T23:05:21.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:05:20 smithi046 ceph-mon[32498]: osdmap e31: 5 total, 4 up, 5 in 2024-03-31T23:05:21.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:05:20 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-31T23:05:21.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:05:20 smithi046 ceph-mon[32498]: from='osd.4 [v2:172.21.15.142:6808/2996266929,v1:172.21.15.142:6809/2996266929]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi142", "root=default"]}]: dispatch 2024-03-31T23:05:21.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:05:20 smithi046 ceph-mon[32498]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi142", "root=default"]}]: dispatch 2024-03-31T23:05:21.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:05:20 smithi046 ceph-mon[32498]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-31T23:05:21.237 INFO:teuthology.orchestra.run.smithi162.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-03-31T23:05:21.237 INFO:teuthology.orchestra.run.smithi162.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-03-31T23:05:21.237 INFO:teuthology.orchestra.run.smithi162.stderr: stderr: 10+0 records in 2024-03-31T23:05:21.237 INFO:teuthology.orchestra.run.smithi162.stderr:10+0 records out 2024-03-31T23:05:21.237 INFO:teuthology.orchestra.run.smithi162.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0126144 s, 831 MB/s 2024-03-31T23:05:21.237 INFO:teuthology.orchestra.run.smithi162.stderr:--> Zapping successful for: 2024-03-31T23:05:21.793 INFO:journalctl@ceph.osd.4.smithi142.stdout:Mar 31 23:05:21 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-4[46503]: 2024-03-31T23:05:21.513+0000 7f6359832700 -1 osd.4 0 waiting for initial osdmap 2024-03-31T23:05:21.793 INFO:journalctl@ceph.osd.4.smithi142.stdout:Mar 31 23:05:21 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-4[46503]: 2024-03-31T23:05:21.519+0000 7f635161d700 -1 osd.4 32 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-31T23:05:21.880 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:05:21 smithi162 ceph-mon[36237]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi142", "root=default"]}]': finished 2024-03-31T23:05:21.880 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:05:21 smithi162 ceph-mon[36237]: osdmap e32: 5 total, 4 up, 5 in 2024-03-31T23:05:21.880 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:05:21 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-31T23:05:21.880 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:05:21 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-31T23:05:21.880 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:05:21 smithi162 ceph-mon[36237]: from='osd.4 ' entity='osd.4' 2024-03-31T23:05:21.910 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:384f31f3d3aee7f8ba3f60846b8844c049165583 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6a9d8f08-efb2-11ee-b647-cb9ed24678a4 -- ceph orch daemon add osd smithi162:vg_nvme/lv_3 2024-03-31T23:05:22.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:05:21 smithi046 ceph-mon[32498]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi142", "root=default"]}]': finished 2024-03-31T23:05:22.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:05:21 smithi046 ceph-mon[32498]: osdmap e32: 5 total, 4 up, 5 in 2024-03-31T23:05:22.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:05:21 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-31T23:05:22.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:05:21 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-31T23:05:22.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:05:21 smithi046 ceph-mon[32498]: from='osd.4 ' entity='osd.4' 2024-03-31T23:05:22.148 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/6a9d8f08-efb2-11ee-b647-cb9ed24678a4/mon.c/config 2024-03-31T23:05:22.169 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:05:21 smithi142 ceph-mon[35271]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi142", "root=default"]}]': finished 2024-03-31T23:05:22.169 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:05:21 smithi142 ceph-mon[35271]: osdmap e32: 5 total, 4 up, 5 in 2024-03-31T23:05:22.169 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:05:21 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-31T23:05:22.170 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:05:21 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-31T23:05:22.170 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:05:21 smithi142 ceph-mon[35271]: from='osd.4 ' entity='osd.4' 2024-03-31T23:05:23.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:05:22 smithi162 ceph-mon[36237]: purged_snaps scrub starts 2024-03-31T23:05:23.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:05:22 smithi162 ceph-mon[36237]: purged_snaps scrub ok 2024-03-31T23:05:23.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:05:22 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-31T23:05:23.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:05:22 smithi162 ceph-mon[36237]: Detected new or changed devices on smithi142 2024-03-31T23:05:23.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:05:22 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:05:23.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:05:22 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:05:23.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:05:22 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-31T23:05:23.047 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:05:22 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-31T23:05:23.047 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:05:22 smithi162 ceph-mon[36237]: Adjusting osd_memory_target on smithi142 to 8544M 2024-03-31T23:05:23.047 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:05:22 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:05:23.047 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:05:22 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:05:23.047 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:05:22 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-31T23:05:23.047 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:05:22 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:05:23.047 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:05:22 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-31T23:05:23.047 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:05:22 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:05:23.047 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:05:22 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-31T23:05:23.047 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:05:22 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:05:23.047 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:05:22 smithi162 ceph-mon[36237]: osd.4 [v2:172.21.15.142:6808/2996266929,v1:172.21.15.142:6809/2996266929] boot 2024-03-31T23:05:23.047 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:05:22 smithi162 ceph-mon[36237]: osdmap e33: 5 total, 5 up, 5 in 2024-03-31T23:05:23.047 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:05:22 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-31T23:05:23.047 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:05:22 smithi162 ceph-mon[36237]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-31T23:05:23.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:05:22 smithi046 ceph-mon[32498]: purged_snaps scrub starts 2024-03-31T23:05:23.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:05:22 smithi046 ceph-mon[32498]: purged_snaps scrub ok 2024-03-31T23:05:23.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:05:22 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-31T23:05:23.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:05:22 smithi046 ceph-mon[32498]: Detected new or changed devices on smithi142 2024-03-31T23:05:23.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:05:22 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:05:23.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:05:22 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:05:23.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:05:22 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-31T23:05:23.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:05:22 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-31T23:05:23.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:05:22 smithi046 ceph-mon[32498]: Adjusting osd_memory_target on smithi142 to 8544M 2024-03-31T23:05:23.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:05:22 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:05:23.113 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:05:22 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:05:23.113 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:05:22 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-31T23:05:23.113 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:05:22 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:05:23.113 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:05:22 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-31T23:05:23.113 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:05:22 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:05:23.113 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:05:22 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-31T23:05:23.113 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:05:22 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:05:23.113 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:05:22 smithi046 ceph-mon[32498]: osd.4 [v2:172.21.15.142:6808/2996266929,v1:172.21.15.142:6809/2996266929] boot 2024-03-31T23:05:23.113 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:05:22 smithi046 ceph-mon[32498]: osdmap e33: 5 total, 5 up, 5 in 2024-03-31T23:05:23.113 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:05:22 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-31T23:05:23.113 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:05:22 smithi046 ceph-mon[32498]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-31T23:05:23.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:05:22 smithi142 ceph-mon[35271]: purged_snaps scrub starts 2024-03-31T23:05:23.194 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:05:22 smithi142 ceph-mon[35271]: purged_snaps scrub ok 2024-03-31T23:05:23.194 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:05:22 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-31T23:05:23.194 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:05:22 smithi142 ceph-mon[35271]: Detected new or changed devices on smithi142 2024-03-31T23:05:23.194 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:05:22 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:05:23.194 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:05:22 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:05:23.194 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:05:22 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-31T23:05:23.194 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:05:22 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-31T23:05:23.194 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:05:22 smithi142 ceph-mon[35271]: Adjusting osd_memory_target on smithi142 to 8544M 2024-03-31T23:05:23.194 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:05:22 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:05:23.194 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:05:22 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:05:23.195 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:05:22 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-31T23:05:23.195 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:05:22 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:05:23.195 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:05:22 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-31T23:05:23.195 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:05:22 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:05:23.195 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:05:22 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-31T23:05:23.195 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:05:22 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:05:23.195 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:05:22 smithi142 ceph-mon[35271]: osd.4 [v2:172.21.15.142:6808/2996266929,v1:172.21.15.142:6809/2996266929] boot 2024-03-31T23:05:23.195 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:05:22 smithi142 ceph-mon[35271]: osdmap e33: 5 total, 5 up, 5 in 2024-03-31T23:05:23.195 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:05:22 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-31T23:05:23.195 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:05:22 smithi142 ceph-mon[35271]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-31T23:05:23.628 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:05:23.625+0000 7f0d94181700 1 -- 172.21.15.162:0/3424411217 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0d8c103e00 msgr2=0x7f0d8c10a6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:05:23.628 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:05:23.625+0000 7f0d94181700 1 --2- 172.21.15.162:0/3424411217 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0d8c103e00 0x7f0d8c10a6b0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f0d8400b0a0 tx=0x7f0d8400b3b0 comp rx=0 tx=0).stop 2024-03-31T23:05:23.628 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:05:23.625+0000 7f0d94181700 1 -- 172.21.15.162:0/3424411217 shutdown_connections 2024-03-31T23:05:23.628 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:05:23.625+0000 7f0d94181700 1 --2- 172.21.15.162:0/3424411217 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0d8c103e00 0x7f0d8c10a6b0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:05:23.629 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:05:23.625+0000 7f0d94181700 1 --2- 172.21.15.162:0/3424411217 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0d8c102260 0x7f0d8c102680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:05:23.629 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:05:23.625+0000 7f0d94181700 1 --2- 172.21.15.162:0/3424411217 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0d8c103460 0x7f0d8c1038c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:05:23.629 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:05:23.625+0000 7f0d94181700 1 -- 172.21.15.162:0/3424411217 >> 172.21.15.162:0/3424411217 conn(0x7f0d8c0fd820 msgr2=0x7f0d8c0ffc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:05:23.629 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:05:23.625+0000 7f0d94181700 1 -- 172.21.15.162:0/3424411217 shutdown_connections 2024-03-31T23:05:23.629 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:05:23.625+0000 7f0d94181700 1 -- 172.21.15.162:0/3424411217 wait complete. 2024-03-31T23:05:23.629 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:05:23.626+0000 7f0d94181700 1 Processor -- start 2024-03-31T23:05:23.629 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:05:23.626+0000 7f0d94181700 1 -- start start 2024-03-31T23:05:23.629 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:05:23.626+0000 7f0d94181700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0d8c102260 0x7f0d8c198300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:05:23.629 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:05:23.626+0000 7f0d94181700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0d8c103460 0x7f0d8c198840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:05:23.629 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:05:23.626+0000 7f0d94181700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0d8c103e00 0x7f0d8c19f8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:05:23.629 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:05:23.626+0000 7f0d94181700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f0d8c10dcf0 con 0x7f0d8c103e00 2024-03-31T23:05:23.629 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:05:23.626+0000 7f0d94181700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f0d8c10db70 con 0x7f0d8c102260 2024-03-31T23:05:23.629 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:05:23.626+0000 7f0d94181700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f0d8c10de70 con 0x7f0d8c103460 2024-03-31T23:05:23.630 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:05:23.626+0000 7f0d9171c700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0d8c103460 0x7f0d8c198840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:05:23.630 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:05:23.626+0000 7f0d9171c700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0d8c103460 0x7f0d8c198840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:36204/0 (socket says 172.21.15.162:36204) 2024-03-31T23:05:23.630 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:05:23.626+0000 7f0d9171c700 1 -- 172.21.15.162:0/1321756897 learned_addr learned my addr 172.21.15.162:0/1321756897 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-03-31T23:05:23.630 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:05:23.626+0000 7f0d91f1d700 1 --2- 172.21.15.162:0/1321756897 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0d8c102260 0x7f0d8c198300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:05:23.630 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:05:23.626+0000 7f0d9271e700 1 --2- 172.21.15.162:0/1321756897 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0d8c103e00 0x7f0d8c19f8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:05:23.630 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:05:23.627+0000 7f0d9171c700 1 -- 172.21.15.162:0/1321756897 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0d8c102260 msgr2=0x7f0d8c198300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:05:23.630 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:05:23.627+0000 7f0d9171c700 1 --2- 172.21.15.162:0/1321756897 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0d8c102260 0x7f0d8c198300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:05:23.630 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:05:23.627+0000 7f0d9171c700 1 -- 172.21.15.162:0/1321756897 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0d8c103e00 msgr2=0x7f0d8c19f8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:05:23.630 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:05:23.627+0000 7f0d9171c700 1 --2- 172.21.15.162:0/1321756897 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0d8c103e00 0x7f0d8c19f8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:05:23.630 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:05:23.627+0000 7f0d9171c700 1 -- 172.21.15.162:0/1321756897 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d8c19ff60 con 0x7f0d8c103460 2024-03-31T23:05:23.630 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:05:23.627+0000 7f0d9171c700 1 --2- 172.21.15.162:0/1321756897 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0d8c103460 0x7f0d8c198840 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f0d7c00c950 tx=0x7f0d7c00cc60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:05:23.630 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:05:23.627+0000 7f0d82ffd700 1 -- 172.21.15.162:0/1321756897 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d7c007ac0 con 0x7f0d8c103460 2024-03-31T23:05:23.630 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:05:23.627+0000 7f0d94181700 1 -- 172.21.15.162:0/1321756897 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0d8c1a0250 con 0x7f0d8c103460 2024-03-31T23:05:23.630 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:05:23.627+0000 7f0d82ffd700 1 -- 172.21.15.162:0/1321756897 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0d7c004ce0 con 0x7f0d8c103460 2024-03-31T23:05:23.631 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:05:23.627+0000 7f0d82ffd700 1 -- 172.21.15.162:0/1321756897 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d7c0054c0 con 0x7f0d8c103460 2024-03-31T23:05:23.631 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:05:23.627+0000 7f0d94181700 1 -- 172.21.15.162:0/1321756897 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0d8c1a0790 con 0x7f0d8c103460 2024-03-31T23:05:23.632 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:05:23.628+0000 7f0d94181700 1 -- 172.21.15.162:0/1321756897 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0d8c068f00 con 0x7f0d8c103460 2024-03-31T23:05:23.632 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:05:23.630+0000 7f0d82ffd700 1 -- 172.21.15.162:0/1321756897 <== mon.1 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 16) ==== 137320+0+0 (secure 0 0 0) 0x7f0d7c004750 con 0x7f0d8c103460 2024-03-31T23:05:23.632 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:05:23.630+0000 7f0d82ffd700 1 --2- 172.21.15.162:0/1321756897 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f0d780a2530 0x7f0d780a49f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:05:23.632 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:05:23.630+0000 7f0d82ffd700 1 -- 172.21.15.162:0/1321756897 <== mon.1 v2:172.21.15.162:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 4101+0+0 (secure 0 0 0) 0x7f0d7c0cd6d0 con 0x7f0d8c103460 2024-03-31T23:05:23.632 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:05:23.631+0000 7f0d91f1d700 1 --2- 172.21.15.162:0/1321756897 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f0d780a2530 0x7f0d780a49f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:05:23.632 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:05:23.631+0000 7f0d91f1d700 1 --2- 172.21.15.162:0/1321756897 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f0d780a2530 0x7f0d780a49f0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f0d8c1032c0 tx=0x7f0d8800b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:05:23.635 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:05:23.634+0000 7f0d82ffd700 1 -- 172.21.15.162:0/1321756897 <== mon.1 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f0d7c014030 con 0x7f0d8c103460 2024-03-31T23:05:23.841 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:05:23.839+0000 7f0d94181700 1 -- 172.21.15.162:0/1321756897 --> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi162:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7f0d8c062680 con 0x7f0d780a2530 2024-03-31T23:05:25.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:05:24 smithi162 ceph-mon[36237]: osdmap e34: 5 total, 5 up, 5 in 2024-03-31T23:05:25.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:05:24 smithi162 ceph-mon[36237]: from='client.24226 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi162:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-31T23:05:25.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:05:24 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-31T23:05:25.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:05:24 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-31T23:05:25.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:05:24 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:05:25.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:05:24 smithi162 ceph-mon[36237]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-03-31T23:05:25.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:05:24 smithi046 ceph-mon[32498]: osdmap e34: 5 total, 5 up, 5 in 2024-03-31T23:05:25.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:05:24 smithi046 ceph-mon[32498]: from='client.24226 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi162:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-31T23:05:25.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:05:24 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-31T23:05:25.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:05:24 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-31T23:05:25.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:05:24 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:05:25.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:05:24 smithi046 ceph-mon[32498]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-03-31T23:05:25.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:05:24 smithi142 ceph-mon[35271]: osdmap e34: 5 total, 5 up, 5 in 2024-03-31T23:05:25.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:05:24 smithi142 ceph-mon[35271]: from='client.24226 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi162:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-31T23:05:25.194 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:05:24 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-31T23:05:25.194 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:05:24 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-31T23:05:25.194 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:05:24 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:05:25.194 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:05:24 smithi142 ceph-mon[35271]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-03-31T23:05:26.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:05:26 smithi162 ceph-mon[36237]: from='client.? 172.21.15.162:0/1963001951' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9da7ff75-f658-45ea-a02f-1943fdda76b9"}]: dispatch 2024-03-31T23:05:26.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:05:26 smithi162 ceph-mon[36237]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9da7ff75-f658-45ea-a02f-1943fdda76b9"}]: dispatch 2024-03-31T23:05:26.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:05:26 smithi162 ceph-mon[36237]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9da7ff75-f658-45ea-a02f-1943fdda76b9"}]': finished 2024-03-31T23:05:26.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:05:26 smithi162 ceph-mon[36237]: osdmap e35: 6 total, 5 up, 6 in 2024-03-31T23:05:26.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:05:26 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-31T23:05:26.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:05:26 smithi162 ceph-mon[36237]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-03-31T23:05:26.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:05:26 smithi046 ceph-mon[32498]: from='client.? 172.21.15.162:0/1963001951' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9da7ff75-f658-45ea-a02f-1943fdda76b9"}]: dispatch 2024-03-31T23:05:26.862 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:05:26 smithi046 ceph-mon[32498]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9da7ff75-f658-45ea-a02f-1943fdda76b9"}]: dispatch 2024-03-31T23:05:26.862 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:05:26 smithi046 ceph-mon[32498]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9da7ff75-f658-45ea-a02f-1943fdda76b9"}]': finished 2024-03-31T23:05:26.862 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:05:26 smithi046 ceph-mon[32498]: osdmap e35: 6 total, 5 up, 6 in 2024-03-31T23:05:26.862 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:05:26 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-31T23:05:26.862 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:05:26 smithi046 ceph-mon[32498]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-03-31T23:05:26.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:05:26 smithi142 ceph-mon[35271]: from='client.? 172.21.15.162:0/1963001951' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9da7ff75-f658-45ea-a02f-1943fdda76b9"}]: dispatch 2024-03-31T23:05:26.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:05:26 smithi142 ceph-mon[35271]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9da7ff75-f658-45ea-a02f-1943fdda76b9"}]: dispatch 2024-03-31T23:05:26.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:05:26 smithi142 ceph-mon[35271]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9da7ff75-f658-45ea-a02f-1943fdda76b9"}]': finished 2024-03-31T23:05:26.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:05:26 smithi142 ceph-mon[35271]: osdmap e35: 6 total, 5 up, 6 in 2024-03-31T23:05:26.944 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:05:26 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-31T23:05:26.944 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:05:26 smithi142 ceph-mon[35271]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-03-31T23:05:27.797 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:05:27 smithi162 ceph-mon[36237]: from='client.? 172.21.15.162:0/714059835' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-31T23:05:27.862 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:05:27 smithi046 ceph-mon[32498]: from='client.? 172.21.15.162:0/714059835' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-31T23:05:27.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:05:27 smithi142 ceph-mon[35271]: from='client.? 172.21.15.162:0/714059835' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-31T23:05:28.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:05:28 smithi046 ceph-mon[32498]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-03-31T23:05:28.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:05:28 smithi142 ceph-mon[35271]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-03-31T23:05:29.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:05:28 smithi162 ceph-mon[36237]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-03-31T23:05:30.693 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:05:30 smithi142 ceph-mon[35271]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-03-31T23:05:30.789 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:05:30 smithi162 ceph-mon[36237]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-03-31T23:05:30.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:05:30 smithi046 ceph-mon[32498]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-03-31T23:05:32.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:05:32 smithi046 ceph-mon[32498]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-03-31T23:05:32.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:05:32 smithi142 ceph-mon[35271]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-03-31T23:05:32.995 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:05:32 smithi162 ceph-mon[36237]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-03-31T23:05:34.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:05:33 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-31T23:05:34.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:05:33 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:05:34.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:05:33 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-31T23:05:34.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:05:33 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:05:34.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:05:33 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-31T23:05:34.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:05:33 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:05:34.791 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:05:34 smithi162 ceph-mon[36237]: Deploying daemon osd.5 on smithi162 2024-03-31T23:05:34.791 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:05:34 smithi162 ceph-mon[36237]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-03-31T23:05:35.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:05:34 smithi046 ceph-mon[32498]: Deploying daemon osd.5 on smithi162 2024-03-31T23:05:35.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:05:34 smithi046 ceph-mon[32498]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-03-31T23:05:35.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:05:34 smithi142 ceph-mon[35271]: Deploying daemon osd.5 on smithi162 2024-03-31T23:05:35.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:05:34 smithi142 ceph-mon[35271]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-03-31T23:05:36.712 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:05:36 smithi162 ceph-mon[36237]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-03-31T23:05:36.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:05:36 smithi046 ceph-mon[32498]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-03-31T23:05:36.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:05:36 smithi142 ceph-mon[35271]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-03-31T23:05:38.709 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:05:38 smithi162 ceph-mon[36237]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-03-31T23:05:38.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:05:38 smithi046 ceph-mon[32498]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-03-31T23:05:38.944 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:05:38 smithi142 ceph-mon[35271]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-03-31T23:05:40.296 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:05:40 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:05:40.296 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:05:40 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:05:40.296 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:05:40 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-31T23:05:40.361 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:05:40 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:05:40.361 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:05:40 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:05:40.362 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:05:40 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-31T23:05:40.443 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:05:40 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:05:40.443 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:05:40 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:05:40.443 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:05:40 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-31T23:05:41.295 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:05:41 smithi162 ceph-mon[36237]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-03-31T23:05:41.361 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:05:41 smithi046 ceph-mon[32498]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-03-31T23:05:41.443 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:05:41 smithi142 ceph-mon[35271]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-03-31T23:05:42.757 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:05:42.756+0000 7f0d82ffd700 1 -- 172.21.15.162:0/1321756897 <== mgr.14152 v2:172.21.15.46:6800/162289569 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f0d8c062680 con 0x7f0d780a2530 2024-03-31T23:05:42.757 INFO:teuthology.orchestra.run.smithi162.stdout:Created osd(s) 5 on host 'smithi162' 2024-03-31T23:05:42.759 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:05:42.758+0000 7f0d94181700 1 -- 172.21.15.162:0/1321756897 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f0d780a2530 msgr2=0x7f0d780a49f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:05:42.759 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:05:42.758+0000 7f0d94181700 1 --2- 172.21.15.162:0/1321756897 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f0d780a2530 0x7f0d780a49f0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f0d8c1032c0 tx=0x7f0d8800b410 comp rx=0 tx=0).stop 2024-03-31T23:05:42.759 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:05:42.759+0000 7f0d94181700 1 -- 172.21.15.162:0/1321756897 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0d8c103460 msgr2=0x7f0d8c198840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:05:42.759 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:05:42.759+0000 7f0d94181700 1 --2- 172.21.15.162:0/1321756897 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0d8c103460 0x7f0d8c198840 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f0d7c00c950 tx=0x7f0d7c00cc60 comp rx=0 tx=0).stop 2024-03-31T23:05:42.759 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:05:42.759+0000 7f0d94181700 1 -- 172.21.15.162:0/1321756897 shutdown_connections 2024-03-31T23:05:42.759 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:05:42.759+0000 7f0d94181700 1 --2- 172.21.15.162:0/1321756897 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f0d780a2530 0x7f0d780a49f0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:05:42.759 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:05:42.759+0000 7f0d94181700 1 --2- 172.21.15.162:0/1321756897 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0d8c103e00 0x7f0d8c19f8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:05:42.760 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:05:42.759+0000 7f0d94181700 1 --2- 172.21.15.162:0/1321756897 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0d8c102260 0x7f0d8c198300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:05:42.760 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:05:42.759+0000 7f0d94181700 1 --2- 172.21.15.162:0/1321756897 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0d8c103460 0x7f0d8c198840 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:05:42.760 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:05:42.759+0000 7f0d94181700 1 -- 172.21.15.162:0/1321756897 >> 172.21.15.162:0/1321756897 conn(0x7f0d8c0fd820 msgr2=0x7f0d8c106800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:05:42.760 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:05:42.759+0000 7f0d94181700 1 -- 172.21.15.162:0/1321756897 shutdown_connections 2024-03-31T23:05:42.760 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:05:42.759+0000 7f0d94181700 1 -- 172.21.15.162:0/1321756897 wait complete. 2024-03-31T23:05:43.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:05:42 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:05:43.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:05:42 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:05:43.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:05:42 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:05:43.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:05:42 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-31T23:05:43.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:05:42 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:05:43.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:05:42 smithi162 ceph-mon[36237]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-03-31T23:05:43.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:05:42 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:05:43.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:05:42 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:05:43.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:05:42 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-31T23:05:43.047 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:05:42 smithi162 ceph-mon[36237]: from='osd.5 [v2:172.21.15.162:6808/914208293,v1:172.21.15.162:6809/914208293]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-31T23:05:43.047 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:05:42 smithi162 ceph-mon[36237]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-31T23:05:43.361 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:05:42 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:05:43.362 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:05:42 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:05:43.362 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:05:42 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:05:43.362 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:05:42 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-31T23:05:43.362 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:05:42 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:05:43.362 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:05:42 smithi046 ceph-mon[32498]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-03-31T23:05:43.362 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:05:42 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:05:43.362 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:05:42 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:05:43.362 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:05:42 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-31T23:05:43.362 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:05:42 smithi046 ceph-mon[32498]: from='osd.5 [v2:172.21.15.162:6808/914208293,v1:172.21.15.162:6809/914208293]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-31T23:05:43.363 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:05:42 smithi046 ceph-mon[32498]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-31T23:05:43.395 DEBUG:teuthology.orchestra.run.smithi162:osd.5> sudo journalctl -f -n 0 -u ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4@osd.5.service 2024-03-31T23:05:43.397 INFO:tasks.cephadm:Deploying osd.6 on smithi046 with /dev/vg_nvme/lv_2... 2024-03-31T23:05:43.398 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:384f31f3d3aee7f8ba3f60846b8844c049165583 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6a9d8f08-efb2-11ee-b647-cb9ed24678a4 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-03-31T23:05:43.443 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:05:42 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:05:43.444 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:05:42 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:05:43.444 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:05:42 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:05:43.444 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:05:42 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-31T23:05:43.444 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:05:42 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:05:43.444 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:05:42 smithi142 ceph-mon[35271]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-03-31T23:05:43.444 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:05:42 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:05:43.444 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:05:42 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:05:43.444 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:05:42 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-31T23:05:43.444 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:05:42 smithi142 ceph-mon[35271]: from='osd.5 [v2:172.21.15.162:6808/914208293,v1:172.21.15.162:6809/914208293]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-31T23:05:43.444 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:05:42 smithi142 ceph-mon[35271]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-31T23:05:43.651 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/6a9d8f08-efb2-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-31T23:05:44.892 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:05:44 smithi162 ceph-mon[36237]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-31T23:05:44.892 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:05:44 smithi162 ceph-mon[36237]: osdmap e36: 6 total, 5 up, 6 in 2024-03-31T23:05:44.892 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:05:44 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-31T23:05:44.892 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:05:44 smithi162 ceph-mon[36237]: from='osd.5 [v2:172.21.15.162:6808/914208293,v1:172.21.15.162:6809/914208293]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]: dispatch 2024-03-31T23:05:44.892 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:05:44 smithi162 ceph-mon[36237]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]: dispatch 2024-03-31T23:05:44.893 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:05:44 smithi162 ceph-mon[36237]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-03-31T23:05:45.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:05:44 smithi046 ceph-mon[32498]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-31T23:05:45.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:05:44 smithi046 ceph-mon[32498]: osdmap e36: 6 total, 5 up, 6 in 2024-03-31T23:05:45.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:05:44 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-31T23:05:45.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:05:44 smithi046 ceph-mon[32498]: from='osd.5 [v2:172.21.15.162:6808/914208293,v1:172.21.15.162:6809/914208293]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]: dispatch 2024-03-31T23:05:45.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:05:44 smithi046 ceph-mon[32498]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]: dispatch 2024-03-31T23:05:45.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:05:44 smithi046 ceph-mon[32498]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-03-31T23:05:45.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:05:44 smithi142 ceph-mon[35271]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-31T23:05:45.194 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:05:44 smithi142 ceph-mon[35271]: osdmap e36: 6 total, 5 up, 6 in 2024-03-31T23:05:45.194 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:05:44 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-31T23:05:45.194 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:05:44 smithi142 ceph-mon[35271]: from='osd.5 [v2:172.21.15.162:6808/914208293,v1:172.21.15.162:6809/914208293]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]: dispatch 2024-03-31T23:05:45.194 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:05:44 smithi142 ceph-mon[35271]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]: dispatch 2024-03-31T23:05:45.194 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:05:44 smithi142 ceph-mon[35271]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-03-31T23:05:45.771 INFO:journalctl@ceph.osd.5.smithi162.stdout:Mar 31 23:05:45 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-5[48185]: 2024-03-31T23:05:45.491+0000 7f4344320700 -1 osd.5 0 waiting for initial osdmap 2024-03-31T23:05:45.771 INFO:journalctl@ceph.osd.5.smithi162.stdout:Mar 31 23:05:45 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-5[48185]: 2024-03-31T23:05:45.499+0000 7f433c10b700 -1 osd.5 38 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-31T23:05:45.839 INFO:teuthology.orchestra.run.smithi046.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-03-31T23:05:45.839 INFO:teuthology.orchestra.run.smithi046.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-03-31T23:05:45.839 INFO:teuthology.orchestra.run.smithi046.stderr: stderr: 10+0 records in 2024-03-31T23:05:45.839 INFO:teuthology.orchestra.run.smithi046.stderr:10+0 records out 2024-03-31T23:05:45.839 INFO:teuthology.orchestra.run.smithi046.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.00854834 s, 1.2 GB/s 2024-03-31T23:05:45.839 INFO:teuthology.orchestra.run.smithi046.stderr:--> Zapping successful for: 2024-03-31T23:05:46.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:05:45 smithi162 ceph-mon[36237]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]': finished 2024-03-31T23:05:46.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:05:45 smithi162 ceph-mon[36237]: osdmap e37: 6 total, 5 up, 6 in 2024-03-31T23:05:46.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:05:45 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-31T23:05:46.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:05:45 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-31T23:05:46.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:05:45 smithi162 ceph-mon[36237]: osdmap e38: 6 total, 5 up, 6 in 2024-03-31T23:05:46.047 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:05:45 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-31T23:05:46.047 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:05:45 smithi162 ceph-mon[36237]: from='osd.5 ' entity='osd.5' 2024-03-31T23:05:46.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:05:45 smithi046 ceph-mon[32498]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]': finished 2024-03-31T23:05:46.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:05:45 smithi046 ceph-mon[32498]: osdmap e37: 6 total, 5 up, 6 in 2024-03-31T23:05:46.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:05:45 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-31T23:05:46.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:05:45 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-31T23:05:46.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:05:45 smithi046 ceph-mon[32498]: osdmap e38: 6 total, 5 up, 6 in 2024-03-31T23:05:46.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:05:45 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-31T23:05:46.113 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:05:45 smithi046 ceph-mon[32498]: from='osd.5 ' entity='osd.5' 2024-03-31T23:05:46.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:05:45 smithi142 ceph-mon[35271]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]': finished 2024-03-31T23:05:46.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:05:45 smithi142 ceph-mon[35271]: osdmap e37: 6 total, 5 up, 6 in 2024-03-31T23:05:46.194 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:05:45 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-31T23:05:46.194 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:05:45 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-31T23:05:46.194 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:05:45 smithi142 ceph-mon[35271]: osdmap e38: 6 total, 5 up, 6 in 2024-03-31T23:05:46.194 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:05:45 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-31T23:05:46.194 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:05:45 smithi142 ceph-mon[35271]: from='osd.5 ' entity='osd.5' 2024-03-31T23:05:46.449 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:384f31f3d3aee7f8ba3f60846b8844c049165583 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6a9d8f08-efb2-11ee-b647-cb9ed24678a4 -- ceph orch daemon add osd smithi046:vg_nvme/lv_2 2024-03-31T23:05:46.687 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/6a9d8f08-efb2-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-31T23:05:47.026 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:05:46 smithi046 ceph-mon[32498]: purged_snaps scrub starts 2024-03-31T23:05:47.026 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:05:46 smithi046 ceph-mon[32498]: purged_snaps scrub ok 2024-03-31T23:05:47.026 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:05:46 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-31T23:05:47.026 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:05:46 smithi046 ceph-mon[32498]: osd.5 [v2:172.21.15.162:6808/914208293,v1:172.21.15.162:6809/914208293] boot 2024-03-31T23:05:47.026 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:05:46 smithi046 ceph-mon[32498]: osdmap e39: 6 total, 6 up, 6 in 2024-03-31T23:05:47.026 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:05:46 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-31T23:05:47.026 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:05:46 smithi046 ceph-mon[32498]: Detected new or changed devices on smithi162 2024-03-31T23:05:47.026 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:05:46 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:05:47.027 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:05:46 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:05:47.027 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:05:46 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-31T23:05:47.027 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:05:46 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-31T23:05:47.027 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:05:46 smithi046 ceph-mon[32498]: Adjusting osd_memory_target on smithi162 to 8544M 2024-03-31T23:05:47.027 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:05:46 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:05:47.027 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:05:46 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:05:47.027 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:05:46 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-31T23:05:47.027 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:05:46 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:05:47.027 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:05:46 smithi046 ceph-mon[32498]: pgmap v138: 1 pgs: 1 remapped+peering; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-03-31T23:05:47.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:05:46 smithi162 ceph-mon[36237]: purged_snaps scrub starts 2024-03-31T23:05:47.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:05:46 smithi162 ceph-mon[36237]: purged_snaps scrub ok 2024-03-31T23:05:47.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:05:46 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-31T23:05:47.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:05:46 smithi162 ceph-mon[36237]: osd.5 [v2:172.21.15.162:6808/914208293,v1:172.21.15.162:6809/914208293] boot 2024-03-31T23:05:47.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:05:46 smithi162 ceph-mon[36237]: osdmap e39: 6 total, 6 up, 6 in 2024-03-31T23:05:47.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:05:46 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-31T23:05:47.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:05:46 smithi162 ceph-mon[36237]: Detected new or changed devices on smithi162 2024-03-31T23:05:47.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:05:46 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:05:47.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:05:46 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:05:47.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:05:46 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-31T23:05:47.047 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:05:46 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-31T23:05:47.047 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:05:46 smithi162 ceph-mon[36237]: Adjusting osd_memory_target on smithi162 to 8544M 2024-03-31T23:05:47.047 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:05:46 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:05:47.047 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:05:46 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:05:47.047 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:05:46 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-31T23:05:47.047 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:05:46 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:05:47.047 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:05:46 smithi162 ceph-mon[36237]: pgmap v138: 1 pgs: 1 remapped+peering; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-03-31T23:05:47.194 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:05:46 smithi142 ceph-mon[35271]: purged_snaps scrub starts 2024-03-31T23:05:47.194 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:05:46 smithi142 ceph-mon[35271]: purged_snaps scrub ok 2024-03-31T23:05:47.194 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:05:46 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-31T23:05:47.194 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:05:46 smithi142 ceph-mon[35271]: osd.5 [v2:172.21.15.162:6808/914208293,v1:172.21.15.162:6809/914208293] boot 2024-03-31T23:05:47.194 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:05:46 smithi142 ceph-mon[35271]: osdmap e39: 6 total, 6 up, 6 in 2024-03-31T23:05:47.194 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:05:46 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-31T23:05:47.194 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:05:46 smithi142 ceph-mon[35271]: Detected new or changed devices on smithi162 2024-03-31T23:05:47.194 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:05:46 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:05:47.194 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:05:46 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:05:47.194 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:05:46 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-31T23:05:47.194 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:05:46 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-31T23:05:47.195 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:05:46 smithi142 ceph-mon[35271]: Adjusting osd_memory_target on smithi162 to 8544M 2024-03-31T23:05:47.195 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:05:46 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:05:47.195 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:05:46 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:05:47.195 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:05:46 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-31T23:05:47.195 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:05:46 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:05:47.195 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:05:46 smithi142 ceph-mon[35271]: pgmap v138: 1 pgs: 1 remapped+peering; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-03-31T23:05:48.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:05:48.322+0000 7fc5083bc700 1 -- 172.21.15.46:0/2786285457 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc500101160 msgr2=0x7fc500101580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:05:48.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:05:48.322+0000 7fc5083bc700 1 --2- 172.21.15.46:0/2786285457 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc500101160 0x7fc500101580 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fc4fc0099c0 tx=0x7fc4fc009cd0 comp rx=0 tx=0).stop 2024-03-31T23:05:48.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:05:48.322+0000 7fc5083bc700 1 -- 172.21.15.46:0/2786285457 shutdown_connections 2024-03-31T23:05:48.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:05:48.322+0000 7fc5083bc700 1 --2- 172.21.15.46:0/2786285457 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc500101160 0x7fc500101580 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:05:48.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:05:48.322+0000 7fc5083bc700 1 --2- 172.21.15.46:0/2786285457 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc500102d00 0x7fc5001095b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:05:48.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:05:48.322+0000 7fc5083bc700 1 --2- 172.21.15.46:0/2786285457 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc500102360 0x7fc5001027c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:05:48.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:05:48.322+0000 7fc5083bc700 1 -- 172.21.15.46:0/2786285457 >> 172.21.15.46:0/2786285457 conn(0x7fc5000fc740 msgr2=0x7fc5000feb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:05:48.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:05:48.322+0000 7fc5083bc700 1 -- 172.21.15.46:0/2786285457 shutdown_connections 2024-03-31T23:05:48.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:05:48.322+0000 7fc5083bc700 1 -- 172.21.15.46:0/2786285457 wait complete. 2024-03-31T23:05:48.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:05:48.323+0000 7fc5083bc700 1 Processor -- start 2024-03-31T23:05:48.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:05:48.323+0000 7fc5083bc700 1 -- start start 2024-03-31T23:05:48.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:05:48.323+0000 7fc5083bc700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc500101160 0x7fc50006b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:05:48.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:05:48.323+0000 7fc5083bc700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc500102360 0x7fc50019d280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:05:48.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:05:48.323+0000 7fc5083bc700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc500102d00 0x7fc50019f8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:05:48.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:05:48.323+0000 7fc5083bc700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fc50010ccd0 con 0x7fc500101160 2024-03-31T23:05:48.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:05:48.323+0000 7fc5083bc700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fc50010cb50 con 0x7fc500102360 2024-03-31T23:05:48.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:05:48.323+0000 7fc5083bc700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fc50010ce50 con 0x7fc500102d00 2024-03-31T23:05:48.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:05:48.323+0000 7fc506158700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc500101160 0x7fc50006b4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:05:48.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:05:48.323+0000 7fc506158700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc500101160 0x7fc50006b4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:42396/0 (socket says 172.21.15.46:42396) 2024-03-31T23:05:48.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:05:48.323+0000 7fc506158700 1 -- 172.21.15.46:0/4096898829 learned_addr learned my addr 172.21.15.46:0/4096898829 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:05:48.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:05:48.323+0000 7fc506959700 1 --2- 172.21.15.46:0/4096898829 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc500102d00 0x7fc50019f8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:05:48.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:05:48.323+0000 7fc505957700 1 --2- 172.21.15.46:0/4096898829 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc500102360 0x7fc50019d280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:05:48.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:05:48.323+0000 7fc506158700 1 -- 172.21.15.46:0/4096898829 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc500102d00 msgr2=0x7fc50019f8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:05:48.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:05:48.323+0000 7fc506158700 1 --2- 172.21.15.46:0/4096898829 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc500102d00 0x7fc50019f8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:05:48.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:05:48.323+0000 7fc506158700 1 -- 172.21.15.46:0/4096898829 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc500102360 msgr2=0x7fc50019d280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:05:48.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:05:48.323+0000 7fc506158700 1 --2- 172.21.15.46:0/4096898829 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc500102360 0x7fc50019d280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:05:48.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:05:48.323+0000 7fc506158700 1 -- 172.21.15.46:0/4096898829 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5001a0190 con 0x7fc500101160 2024-03-31T23:05:48.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:05:48.324+0000 7fc506959700 1 --2- 172.21.15.46:0/4096898829 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc500102d00 0x7fc50019f8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T23:05:48.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:05:48.324+0000 7fc505957700 1 --2- 172.21.15.46:0/4096898829 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc500102360 0x7fc50019d280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T23:05:48.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:05:48.324+0000 7fc506158700 1 --2- 172.21.15.46:0/4096898829 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc500101160 0x7fc50006b4e0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fc4fc00c010 tx=0x7fc4fc005e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:05:48.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:05:48.324+0000 7fc4f77fe700 1 -- 172.21.15.46:0/4096898829 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4fc027070 con 0x7fc500101160 2024-03-31T23:05:48.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:05:48.324+0000 7fc4f77fe700 1 -- 172.21.15.46:0/4096898829 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc4fc022470 con 0x7fc500101160 2024-03-31T23:05:48.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:05:48.324+0000 7fc4f77fe700 1 -- 172.21.15.46:0/4096898829 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4fc00f6c0 con 0x7fc500101160 2024-03-31T23:05:48.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:05:48.324+0000 7fc5083bc700 1 -- 172.21.15.46:0/4096898829 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc5001a0420 con 0x7fc500101160 2024-03-31T23:05:48.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:05:48.324+0000 7fc5083bc700 1 -- 172.21.15.46:0/4096898829 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc5001a0980 con 0x7fc500101160 2024-03-31T23:05:48.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:05:48.325+0000 7fc5083bc700 1 -- 172.21.15.46:0/4096898829 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc500068f00 con 0x7fc500101160 2024-03-31T23:05:48.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:05:48.326+0000 7fc4f77fe700 1 -- 172.21.15.46:0/4096898829 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 16) ==== 137320+0+0 (secure 0 0 0) 0x7fc4fc02a070 con 0x7fc500101160 2024-03-31T23:05:48.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:05:48.327+0000 7fc4f77fe700 1 --2- 172.21.15.46:0/4096898829 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fc4ec0a2530 0x7fc4ec0a49f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:05:48.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:05:48.327+0000 7fc505957700 1 --2- 172.21.15.46:0/4096898829 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fc4ec0a2530 0x7fc4ec0a49f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:05:48.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:05:48.327+0000 7fc4f77fe700 1 -- 172.21.15.46:0/4096898829 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(40..40 src has 1..40) ==== 4562+0+0 (secure 0 0 0) 0x7fc4fc0ce9d0 con 0x7fc500101160 2024-03-31T23:05:48.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:05:48.327+0000 7fc505957700 1 --2- 172.21.15.46:0/4096898829 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fc4ec0a2530 0x7fc4ec0a49f0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fc50006ba60 tx=0x7fc4f0008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:05:48.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:05:48.330+0000 7fc4f77fe700 1 -- 172.21.15.46:0/4096898829 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fc4fc013070 con 0x7fc500101160 2024-03-31T23:05:48.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:05:48.546+0000 7fc5083bc700 1 -- 172.21.15.46:0/4096898829 --> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi046:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7fc500062680 con 0x7fc4ec0a2530 2024-03-31T23:05:48.546 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:05:48 smithi046 ceph-mon[32498]: osdmap e40: 6 total, 6 up, 6 in 2024-03-31T23:05:48.693 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:05:48 smithi142 ceph-mon[35271]: osdmap e40: 6 total, 6 up, 6 in 2024-03-31T23:05:48.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:05:48 smithi162 ceph-mon[36237]: osdmap e40: 6 total, 6 up, 6 in 2024-03-31T23:05:49.411 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:05:49 smithi046 ceph-mon[32498]: osdmap e41: 6 total, 6 up, 6 in 2024-03-31T23:05:49.411 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:05:49 smithi046 ceph-mon[32498]: pgmap v141: 1 pgs: 1 remapped+peering; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-03-31T23:05:49.411 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:05:49 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-31T23:05:49.411 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:05:49 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-31T23:05:49.411 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:05:49 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:05:49.694 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:05:49 smithi142 ceph-mon[35271]: osdmap e41: 6 total, 6 up, 6 in 2024-03-31T23:05:49.694 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:05:49 smithi142 ceph-mon[35271]: pgmap v141: 1 pgs: 1 remapped+peering; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-03-31T23:05:49.694 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:05:49 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-31T23:05:49.695 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:05:49 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-31T23:05:49.695 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:05:49 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:05:49.697 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:05:49 smithi162 ceph-mon[36237]: osdmap e41: 6 total, 6 up, 6 in 2024-03-31T23:05:49.697 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:05:49 smithi162 ceph-mon[36237]: pgmap v141: 1 pgs: 1 remapped+peering; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-03-31T23:05:49.698 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:05:49 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-31T23:05:49.698 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:05:49 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-31T23:05:49.698 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:05:49 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:05:50.518 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:05:50 smithi046 ceph-mon[32498]: from='client.14355 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi046:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-31T23:05:50.518 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:05:50 smithi046 ceph-mon[32498]: osdmap e42: 6 total, 6 up, 6 in 2024-03-31T23:05:50.693 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:05:50 smithi142 ceph-mon[35271]: from='client.14355 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi046:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-31T23:05:50.693 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:05:50 smithi142 ceph-mon[35271]: osdmap e42: 6 total, 6 up, 6 in 2024-03-31T23:05:50.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:05:50 smithi162 ceph-mon[36237]: from='client.14355 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi046:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-31T23:05:50.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:05:50 smithi162 ceph-mon[36237]: osdmap e42: 6 total, 6 up, 6 in 2024-03-31T23:05:51.611 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:05:51 smithi046 ceph-mon[32498]: osdmap e43: 6 total, 6 up, 6 in 2024-03-31T23:05:51.612 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:05:51 smithi046 ceph-mon[32498]: pgmap v144: 1 pgs: 1 remapped+peering; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-03-31T23:05:51.693 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:05:51 smithi142 ceph-mon[35271]: osdmap e43: 6 total, 6 up, 6 in 2024-03-31T23:05:51.693 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:05:51 smithi142 ceph-mon[35271]: pgmap v144: 1 pgs: 1 remapped+peering; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-03-31T23:05:51.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:05:51 smithi162 ceph-mon[36237]: osdmap e43: 6 total, 6 up, 6 in 2024-03-31T23:05:51.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:05:51 smithi162 ceph-mon[36237]: pgmap v144: 1 pgs: 1 remapped+peering; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-03-31T23:05:52.611 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:05:52 smithi046 ceph-mon[32498]: from='client.? 172.21.15.46:0/3941795296' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8fa11224-cb26-4d3f-87d1-6a8f31de2a19"}]: dispatch 2024-03-31T23:05:52.611 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:05:52 smithi046 ceph-mon[32498]: from='client.? 172.21.15.46:0/3941795296' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8fa11224-cb26-4d3f-87d1-6a8f31de2a19"}]': finished 2024-03-31T23:05:52.612 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:05:52 smithi046 ceph-mon[32498]: osdmap e44: 7 total, 6 up, 7 in 2024-03-31T23:05:52.612 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:05:52 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-31T23:05:52.612 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:05:52 smithi046 ceph-mon[32498]: from='client.? 172.21.15.46:0/902325371' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-31T23:05:52.693 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:05:52 smithi142 ceph-mon[35271]: from='client.? 172.21.15.46:0/3941795296' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8fa11224-cb26-4d3f-87d1-6a8f31de2a19"}]: dispatch 2024-03-31T23:05:52.693 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:05:52 smithi142 ceph-mon[35271]: from='client.? 172.21.15.46:0/3941795296' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8fa11224-cb26-4d3f-87d1-6a8f31de2a19"}]': finished 2024-03-31T23:05:52.693 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:05:52 smithi142 ceph-mon[35271]: osdmap e44: 7 total, 6 up, 7 in 2024-03-31T23:05:52.693 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:05:52 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-31T23:05:52.693 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:05:52 smithi142 ceph-mon[35271]: from='client.? 172.21.15.46:0/902325371' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-31T23:05:52.798 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:05:52 smithi162 ceph-mon[36237]: from='client.? 172.21.15.46:0/3941795296' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8fa11224-cb26-4d3f-87d1-6a8f31de2a19"}]: dispatch 2024-03-31T23:05:52.798 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:05:52 smithi162 ceph-mon[36237]: from='client.? 172.21.15.46:0/3941795296' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8fa11224-cb26-4d3f-87d1-6a8f31de2a19"}]': finished 2024-03-31T23:05:52.798 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:05:52 smithi162 ceph-mon[36237]: osdmap e44: 7 total, 6 up, 7 in 2024-03-31T23:05:52.798 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:05:52 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-31T23:05:52.798 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:05:52 smithi162 ceph-mon[36237]: from='client.? 172.21.15.46:0/902325371' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-31T23:05:53.693 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:05:53 smithi142 ceph-mon[35271]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail; 137 KiB/s, 0 objects/s recovering 2024-03-31T23:05:53.795 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:05:53 smithi162 ceph-mon[36237]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail; 137 KiB/s, 0 objects/s recovering 2024-03-31T23:05:53.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:05:53 smithi046 ceph-mon[32498]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail; 137 KiB/s, 0 objects/s recovering 2024-03-31T23:05:54.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:05:54 smithi046 ceph-mon[32498]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-31T23:05:54.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:05:54 smithi142 ceph-mon[35271]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-31T23:05:55.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:05:54 smithi162 ceph-mon[36237]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-31T23:05:56.611 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:05:56 smithi046 ceph-mon[32498]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail; 80 KiB/s, 0 objects/s recovering 2024-03-31T23:05:56.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:05:56 smithi142 ceph-mon[35271]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail; 80 KiB/s, 0 objects/s recovering 2024-03-31T23:05:57.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:05:56 smithi162 ceph-mon[36237]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail; 80 KiB/s, 0 objects/s recovering 2024-03-31T23:05:58.681 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:05:58 smithi046 ceph-mon[32498]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail; 70 KiB/s, 0 objects/s recovering 2024-03-31T23:05:58.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:05:58 smithi142 ceph-mon[35271]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail; 70 KiB/s, 0 objects/s recovering 2024-03-31T23:05:59.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:05:58 smithi162 ceph-mon[36237]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail; 70 KiB/s, 0 objects/s recovering 2024-03-31T23:05:59.693 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:05:59 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-31T23:05:59.693 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:05:59 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:05:59.693 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:05:59 smithi142 ceph-mon[35271]: Deploying daemon osd.6 on smithi046 2024-03-31T23:05:59.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:05:59 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-31T23:05:59.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:05:59 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:05:59.862 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:05:59 smithi046 ceph-mon[32498]: Deploying daemon osd.6 on smithi046 2024-03-31T23:06:00.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:05:59 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-31T23:06:00.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:05:59 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:06:00.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:05:59 smithi162 ceph-mon[36237]: Deploying daemon osd.6 on smithi046 2024-03-31T23:06:00.795 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:06:00 smithi162 ceph-mon[36237]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-31T23:06:00.862 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:06:00 smithi046 ceph-mon[32498]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-31T23:06:00.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:06:00 smithi142 ceph-mon[35271]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-31T23:06:02.862 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:06:02 smithi046 ceph-mon[32498]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-03-31T23:06:02.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:06:02 smithi142 ceph-mon[35271]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-03-31T23:06:03.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:06:02 smithi162 ceph-mon[36237]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-03-31T23:06:05.361 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:06:05 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:06:05.362 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:06:05 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:06:05.362 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:06:05 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-31T23:06:05.362 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:06:05 smithi046 ceph-mon[32498]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-03-31T23:06:05.443 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:06:05 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:06:05.444 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:06:05 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:06:05.444 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:06:05 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-31T23:06:05.444 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:06:05 smithi142 ceph-mon[35271]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-03-31T23:06:05.545 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:06:05 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:06:05.546 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:06:05 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:06:05.546 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:06:05 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-31T23:06:05.546 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:06:05 smithi162 ceph-mon[36237]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-03-31T23:06:06.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:06:06 smithi046 ceph-mon[32498]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-03-31T23:06:06.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:06:06 smithi142 ceph-mon[35271]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-03-31T23:06:07.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:06:06 smithi162 ceph-mon[36237]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-03-31T23:06:08.070 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:06:08.068+0000 7fc4f77fe700 1 -- 172.21.15.46:0/4096898829 <== mgr.14152 v2:172.21.15.46:6800/162289569 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fc500062680 con 0x7fc4ec0a2530 2024-03-31T23:06:08.070 INFO:teuthology.orchestra.run.smithi046.stdout:Created osd(s) 6 on host 'smithi046' 2024-03-31T23:06:08.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:06:08.070+0000 7fc5083bc700 1 -- 172.21.15.46:0/4096898829 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fc4ec0a2530 msgr2=0x7fc4ec0a49f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:06:08.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:06:08.070+0000 7fc5083bc700 1 --2- 172.21.15.46:0/4096898829 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fc4ec0a2530 0x7fc4ec0a49f0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fc50006ba60 tx=0x7fc4f0008040 comp rx=0 tx=0).stop 2024-03-31T23:06:08.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:06:08.070+0000 7fc5083bc700 1 -- 172.21.15.46:0/4096898829 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc500101160 msgr2=0x7fc50006b4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:06:08.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:06:08.070+0000 7fc5083bc700 1 --2- 172.21.15.46:0/4096898829 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc500101160 0x7fc50006b4e0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fc4fc00c010 tx=0x7fc4fc005e80 comp rx=0 tx=0).stop 2024-03-31T23:06:08.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:06:08.071+0000 7fc5083bc700 1 -- 172.21.15.46:0/4096898829 shutdown_connections 2024-03-31T23:06:08.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:06:08.071+0000 7fc5083bc700 1 --2- 172.21.15.46:0/4096898829 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fc4ec0a2530 0x7fc4ec0a49f0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:06:08.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:06:08.071+0000 7fc5083bc700 1 --2- 172.21.15.46:0/4096898829 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc500101160 0x7fc50006b4e0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:06:08.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:06:08.071+0000 7fc5083bc700 1 --2- 172.21.15.46:0/4096898829 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc500102d00 0x7fc50019f8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:06:08.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:06:08.071+0000 7fc5083bc700 1 --2- 172.21.15.46:0/4096898829 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc500102360 0x7fc50019d280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:06:08.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:06:08.071+0000 7fc5083bc700 1 -- 172.21.15.46:0/4096898829 >> 172.21.15.46:0/4096898829 conn(0x7fc5000fc740 msgr2=0x7fc500105870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:06:08.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:06:08.071+0000 7fc5083bc700 1 -- 172.21.15.46:0/4096898829 shutdown_connections 2024-03-31T23:06:08.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:06:08.071+0000 7fc5083bc700 1 -- 172.21.15.46:0/4096898829 wait complete. 2024-03-31T23:06:08.362 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:06:08 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:06:08.362 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:06:08 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:06:08.362 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:06:08 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:06:08.362 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:06:08 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-31T23:06:08.362 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:06:08 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:06:08.362 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:06:08 smithi046 ceph-mon[32498]: from='osd.6 [v2:172.21.15.46:6818/56291966,v1:172.21.15.46:6819/56291966]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-31T23:06:08.363 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:06:08 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:06:08.363 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:06:08 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:06:08.363 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:06:08 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-31T23:06:08.443 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:06:08 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:06:08.444 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:06:08 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:06:08.444 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:06:08 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:06:08.444 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:06:08 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-31T23:06:08.444 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:06:08 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:06:08.444 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:06:08 smithi142 ceph-mon[35271]: from='osd.6 [v2:172.21.15.46:6818/56291966,v1:172.21.15.46:6819/56291966]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-31T23:06:08.444 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:06:08 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:06:08.444 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:06:08 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:06:08.444 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:06:08 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-31T23:06:08.546 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:06:08 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:06:08.546 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:06:08 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:06:08.546 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:06:08 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:06:08.546 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:06:08 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-31T23:06:08.546 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:06:08 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:06:08.546 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:06:08 smithi162 ceph-mon[36237]: from='osd.6 [v2:172.21.15.46:6818/56291966,v1:172.21.15.46:6819/56291966]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-31T23:06:08.546 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:06:08 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:06:08.546 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:06:08 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:06:08.547 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:06:08 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-31T23:06:08.796 DEBUG:teuthology.orchestra.run.smithi046:osd.6> sudo journalctl -f -n 0 -u ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4@osd.6.service 2024-03-31T23:06:08.799 INFO:tasks.cephadm:Deploying osd.7 on smithi142 with /dev/vg_nvme/lv_2... 2024-03-31T23:06:08.799 DEBUG:teuthology.orchestra.run.smithi142:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:384f31f3d3aee7f8ba3f60846b8844c049165583 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6a9d8f08-efb2-11ee-b647-cb9ed24678a4 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-03-31T23:06:09.046 INFO:teuthology.orchestra.run.smithi142.stderr:Inferring config /var/lib/ceph/6a9d8f08-efb2-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-31T23:06:09.272 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:06:09 smithi046 ceph-mon[32498]: from='osd.6 [v2:172.21.15.46:6818/56291966,v1:172.21.15.46:6819/56291966]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-31T23:06:09.272 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:06:09 smithi046 ceph-mon[32498]: osdmap e45: 7 total, 6 up, 7 in 2024-03-31T23:06:09.272 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:06:09 smithi046 ceph-mon[32498]: from='osd.6 [v2:172.21.15.46:6818/56291966,v1:172.21.15.46:6819/56291966]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi046", "root=default"]}]: dispatch 2024-03-31T23:06:09.272 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:06:09 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-31T23:06:09.272 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:06:09 smithi046 ceph-mon[32498]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-03-31T23:06:09.273 INFO:journalctl@ceph.osd.6.smithi046.stdout:Mar 31 23:06:09 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-6[60760]: 2024-03-31T23:06:09.099+0000 7f8b7f6ec700 -1 osd.6 0 waiting for initial osdmap 2024-03-31T23:06:09.273 INFO:journalctl@ceph.osd.6.smithi046.stdout:Mar 31 23:06:09 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-6[60760]: 2024-03-31T23:06:09.107+0000 7f8b7b4df700 -1 osd.6 46 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-31T23:06:09.386 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:06:09 smithi142 ceph-mon[35271]: from='osd.6 [v2:172.21.15.46:6818/56291966,v1:172.21.15.46:6819/56291966]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-31T23:06:09.386 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:06:09 smithi142 ceph-mon[35271]: osdmap e45: 7 total, 6 up, 7 in 2024-03-31T23:06:09.386 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:06:09 smithi142 ceph-mon[35271]: from='osd.6 [v2:172.21.15.46:6818/56291966,v1:172.21.15.46:6819/56291966]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi046", "root=default"]}]: dispatch 2024-03-31T23:06:09.386 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:06:09 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-31T23:06:09.386 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:06:09 smithi142 ceph-mon[35271]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-03-31T23:06:09.546 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:06:09 smithi162 ceph-mon[36237]: from='osd.6 [v2:172.21.15.46:6818/56291966,v1:172.21.15.46:6819/56291966]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-31T23:06:09.547 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:06:09 smithi162 ceph-mon[36237]: osdmap e45: 7 total, 6 up, 7 in 2024-03-31T23:06:09.547 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:06:09 smithi162 ceph-mon[36237]: from='osd.6 [v2:172.21.15.46:6818/56291966,v1:172.21.15.46:6819/56291966]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi046", "root=default"]}]: dispatch 2024-03-31T23:06:09.547 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:06:09 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-31T23:06:09.547 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:06:09 smithi162 ceph-mon[36237]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-03-31T23:06:10.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:06:10 smithi142 ceph-mon[35271]: from='osd.6 [v2:172.21.15.46:6818/56291966,v1:172.21.15.46:6819/56291966]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi046", "root=default"]}]': finished 2024-03-31T23:06:10.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:06:10 smithi142 ceph-mon[35271]: osdmap e46: 7 total, 6 up, 7 in 2024-03-31T23:06:10.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:06:10 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-31T23:06:10.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:06:10 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-31T23:06:10.362 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:06:10 smithi046 ceph-mon[32498]: from='osd.6 [v2:172.21.15.46:6818/56291966,v1:172.21.15.46:6819/56291966]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi046", "root=default"]}]': finished 2024-03-31T23:06:10.362 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:06:10 smithi046 ceph-mon[32498]: osdmap e46: 7 total, 6 up, 7 in 2024-03-31T23:06:10.362 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:06:10 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-31T23:06:10.362 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:06:10 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-31T23:06:10.546 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:06:10 smithi162 ceph-mon[36237]: from='osd.6 [v2:172.21.15.46:6818/56291966,v1:172.21.15.46:6819/56291966]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi046", "root=default"]}]': finished 2024-03-31T23:06:10.546 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:06:10 smithi162 ceph-mon[36237]: osdmap e46: 7 total, 6 up, 7 in 2024-03-31T23:06:10.546 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:06:10 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-31T23:06:10.546 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:06:10 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-31T23:06:11.261 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:06:11 smithi142 ceph-mon[35271]: purged_snaps scrub starts 2024-03-31T23:06:11.261 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:06:11 smithi142 ceph-mon[35271]: purged_snaps scrub ok 2024-03-31T23:06:11.261 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:06:11 smithi142 ceph-mon[35271]: osd.6 [v2:172.21.15.46:6818/56291966,v1:172.21.15.46:6819/56291966] boot 2024-03-31T23:06:11.261 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:06:11 smithi142 ceph-mon[35271]: osdmap e47: 7 total, 7 up, 7 in 2024-03-31T23:06:11.261 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:06:11 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-31T23:06:11.262 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:06:11 smithi142 ceph-mon[35271]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 205 MiB used, 626 GiB / 626 GiB avail 2024-03-31T23:06:11.262 INFO:teuthology.orchestra.run.smithi142.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-03-31T23:06:11.262 INFO:teuthology.orchestra.run.smithi142.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-03-31T23:06:11.262 INFO:teuthology.orchestra.run.smithi142.stderr: stderr: 10+0 records in 2024-03-31T23:06:11.262 INFO:teuthology.orchestra.run.smithi142.stderr:10+0 records out 2024-03-31T23:06:11.262 INFO:teuthology.orchestra.run.smithi142.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0114556 s, 915 MB/s 2024-03-31T23:06:11.262 INFO:teuthology.orchestra.run.smithi142.stderr:--> Zapping successful for: 2024-03-31T23:06:11.362 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:06:11 smithi046 ceph-mon[32498]: purged_snaps scrub starts 2024-03-31T23:06:11.362 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:06:11 smithi046 ceph-mon[32498]: purged_snaps scrub ok 2024-03-31T23:06:11.362 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:06:11 smithi046 ceph-mon[32498]: osd.6 [v2:172.21.15.46:6818/56291966,v1:172.21.15.46:6819/56291966] boot 2024-03-31T23:06:11.362 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:06:11 smithi046 ceph-mon[32498]: osdmap e47: 7 total, 7 up, 7 in 2024-03-31T23:06:11.362 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:06:11 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-31T23:06:11.362 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:06:11 smithi046 ceph-mon[32498]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 205 MiB used, 626 GiB / 626 GiB avail 2024-03-31T23:06:11.546 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:06:11 smithi162 ceph-mon[36237]: purged_snaps scrub starts 2024-03-31T23:06:11.546 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:06:11 smithi162 ceph-mon[36237]: purged_snaps scrub ok 2024-03-31T23:06:11.546 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:06:11 smithi162 ceph-mon[36237]: osd.6 [v2:172.21.15.46:6818/56291966,v1:172.21.15.46:6819/56291966] boot 2024-03-31T23:06:11.546 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:06:11 smithi162 ceph-mon[36237]: osdmap e47: 7 total, 7 up, 7 in 2024-03-31T23:06:11.546 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:06:11 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-31T23:06:11.546 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:06:11 smithi162 ceph-mon[36237]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 205 MiB used, 626 GiB / 626 GiB avail 2024-03-31T23:06:11.893 DEBUG:teuthology.orchestra.run.smithi142:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:384f31f3d3aee7f8ba3f60846b8844c049165583 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6a9d8f08-efb2-11ee-b647-cb9ed24678a4 -- ceph orch daemon add osd smithi142:vg_nvme/lv_2 2024-03-31T23:06:12.147 INFO:teuthology.orchestra.run.smithi142.stderr:Inferring config /var/lib/ceph/6a9d8f08-efb2-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-31T23:06:13.189 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:06:13 smithi142 ceph-mon[35271]: Detected new or changed devices on smithi046 2024-03-31T23:06:13.189 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:06:13 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:06:13.189 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:06:13 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:06:13.189 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:06:13 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-31T23:06:13.189 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:06:13 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-31T23:06:13.189 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:06:13 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-31T23:06:13.189 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:06:13 smithi142 ceph-mon[35271]: Adjusting osd_memory_target on smithi046 to 5697M 2024-03-31T23:06:13.189 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:06:13 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:06:13.189 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:06:13 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:06:13.189 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:06:13 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-31T23:06:13.190 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:06:13 smithi142 ceph-mon[35271]: osdmap e48: 7 total, 7 up, 7 in 2024-03-31T23:06:13.190 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:06:13 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:06:13.190 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:06:13 smithi142 ceph-mon[35271]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 205 MiB used, 626 GiB / 626 GiB avail 2024-03-31T23:06:13.361 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:06:13 smithi046 ceph-mon[32498]: Detected new or changed devices on smithi046 2024-03-31T23:06:13.362 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:06:13 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:06:13.362 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:06:13 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:06:13.362 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:06:13 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-31T23:06:13.362 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:06:13 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-31T23:06:13.362 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:06:13 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-31T23:06:13.362 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:06:13 smithi046 ceph-mon[32498]: Adjusting osd_memory_target on smithi046 to 5697M 2024-03-31T23:06:13.362 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:06:13 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:06:13.362 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:06:13 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:06:13.362 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:06:13 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-31T23:06:13.362 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:06:13 smithi046 ceph-mon[32498]: osdmap e48: 7 total, 7 up, 7 in 2024-03-31T23:06:13.362 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:06:13 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:06:13.362 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:06:13 smithi046 ceph-mon[32498]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 205 MiB used, 626 GiB / 626 GiB avail 2024-03-31T23:06:13.546 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:06:13 smithi162 ceph-mon[36237]: Detected new or changed devices on smithi046 2024-03-31T23:06:13.546 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:06:13 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:06:13.546 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:06:13 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:06:13.546 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:06:13 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-31T23:06:13.546 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:06:13 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-31T23:06:13.546 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:06:13 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-31T23:06:13.546 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:06:13 smithi162 ceph-mon[36237]: Adjusting osd_memory_target on smithi046 to 5697M 2024-03-31T23:06:13.546 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:06:13 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:06:13.546 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:06:13 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:06:13.546 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:06:13 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-31T23:06:13.547 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:06:13 smithi162 ceph-mon[36237]: osdmap e48: 7 total, 7 up, 7 in 2024-03-31T23:06:13.547 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:06:13 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:06:13.547 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:06:13 smithi162 ceph-mon[36237]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 205 MiB used, 626 GiB / 626 GiB avail 2024-03-31T23:06:13.680 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-31T23:06:13.679+0000 7fbca8b29700 1 -- 172.21.15.142:0/230370567 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbca41024b0 msgr2=0x7fbca4108d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:06:13.681 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-31T23:06:13.679+0000 7fbca8b29700 1 --2- 172.21.15.142:0/230370567 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbca41024b0 0x7fbca4108d60 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fbc9c00b0d0 tx=0x7fbc9c00b3e0 comp rx=0 tx=0).stop 2024-03-31T23:06:13.681 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-31T23:06:13.679+0000 7fbca8b29700 1 -- 172.21.15.142:0/230370567 shutdown_connections 2024-03-31T23:06:13.681 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-31T23:06:13.679+0000 7fbca8b29700 1 --2- 172.21.15.142:0/230370567 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbca4100910 0x7fbca4100d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:06:13.681 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-31T23:06:13.679+0000 7fbca8b29700 1 --2- 172.21.15.142:0/230370567 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbca41024b0 0x7fbca4108d60 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:06:13.681 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-31T23:06:13.679+0000 7fbca8b29700 1 --2- 172.21.15.142:0/230370567 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbca4101b10 0x7fbca4101f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:06:13.681 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-31T23:06:13.679+0000 7fbca8b29700 1 -- 172.21.15.142:0/230370567 >> 172.21.15.142:0/230370567 conn(0x7fbca40fbe90 msgr2=0x7fbca40fe2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:06:13.681 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-31T23:06:13.679+0000 7fbca8b29700 1 -- 172.21.15.142:0/230370567 shutdown_connections 2024-03-31T23:06:13.681 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-31T23:06:13.679+0000 7fbca8b29700 1 -- 172.21.15.142:0/230370567 wait complete. 2024-03-31T23:06:13.683 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-31T23:06:13.680+0000 7fbca8b29700 1 Processor -- start 2024-03-31T23:06:13.683 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-31T23:06:13.680+0000 7fbca8b29700 1 -- start start 2024-03-31T23:06:13.683 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-31T23:06:13.681+0000 7fbca8b29700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbca4100910 0x7fbca4197240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:06:13.683 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-31T23:06:13.681+0000 7fbca8b29700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbca4101b10 0x7fbca4197780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:06:13.683 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-31T23:06:13.681+0000 7fbca8b29700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbca41024b0 0x7fbca419e800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:06:13.683 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-31T23:06:13.681+0000 7fbca8b29700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fbca410cc30 con 0x7fbca4101b10 2024-03-31T23:06:13.684 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-31T23:06:13.681+0000 7fbca8b29700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fbca410cab0 con 0x7fbca4100910 2024-03-31T23:06:13.684 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-31T23:06:13.681+0000 7fbca8b29700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fbca410cdb0 con 0x7fbca41024b0 2024-03-31T23:06:13.684 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-31T23:06:13.681+0000 7fbca259c700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbca4100910 0x7fbca4197240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:06:13.684 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-31T23:06:13.681+0000 7fbca259c700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbca4100910 0x7fbca4197240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:42636/0 (socket says 172.21.15.142:42636) 2024-03-31T23:06:13.684 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-31T23:06:13.681+0000 7fbca259c700 1 -- 172.21.15.142:0/2032022630 learned_addr learned my addr 172.21.15.142:0/2032022630 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-03-31T23:06:13.684 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-31T23:06:13.681+0000 7fbca2d9d700 1 --2- 172.21.15.142:0/2032022630 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbca41024b0 0x7fbca419e800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:06:13.684 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-31T23:06:13.681+0000 7fbca1d9b700 1 --2- 172.21.15.142:0/2032022630 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbca4101b10 0x7fbca4197780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:06:13.684 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-31T23:06:13.681+0000 7fbca259c700 1 -- 172.21.15.142:0/2032022630 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbca41024b0 msgr2=0x7fbca419e800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:06:13.684 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-31T23:06:13.681+0000 7fbca259c700 1 --2- 172.21.15.142:0/2032022630 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbca41024b0 0x7fbca419e800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:06:13.684 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-31T23:06:13.681+0000 7fbca259c700 1 -- 172.21.15.142:0/2032022630 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbca4101b10 msgr2=0x7fbca4197780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:06:13.684 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-31T23:06:13.681+0000 7fbca259c700 1 --2- 172.21.15.142:0/2032022630 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbca4101b10 0x7fbca4197780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:06:13.684 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-31T23:06:13.681+0000 7fbca259c700 1 -- 172.21.15.142:0/2032022630 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbca419eea0 con 0x7fbca4100910 2024-03-31T23:06:13.684 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-31T23:06:13.681+0000 7fbca1d9b700 1 --2- 172.21.15.142:0/2032022630 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbca4101b10 0x7fbca4197780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T23:06:13.685 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-31T23:06:13.681+0000 7fbca259c700 1 --2- 172.21.15.142:0/2032022630 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbca4100910 0x7fbca4197240 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fbc9400d980 tx=0x7fbc9400dc90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:06:13.685 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-31T23:06:13.681+0000 7fbc937fe700 1 -- 172.21.15.142:0/2032022630 <== mon.2 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc94016070 con 0x7fbca4100910 2024-03-31T23:06:13.685 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-31T23:06:13.681+0000 7fbca8b29700 1 -- 172.21.15.142:0/2032022630 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbca419f190 con 0x7fbca4100910 2024-03-31T23:06:13.685 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-31T23:06:13.681+0000 7fbc937fe700 1 -- 172.21.15.142:0/2032022630 <== mon.2 v2:172.21.15.142:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fbc9400be00 con 0x7fbca4100910 2024-03-31T23:06:13.685 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-31T23:06:13.681+0000 7fbc937fe700 1 -- 172.21.15.142:0/2032022630 <== mon.2 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc9400f3c0 con 0x7fbca4100910 2024-03-31T23:06:13.685 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-31T23:06:13.681+0000 7fbca8b29700 1 -- 172.21.15.142:0/2032022630 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbca419f6d0 con 0x7fbca4100910 2024-03-31T23:06:13.685 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-31T23:06:13.683+0000 7fbca8b29700 1 -- 172.21.15.142:0/2032022630 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbca4068f00 con 0x7fbca4100910 2024-03-31T23:06:13.686 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-31T23:06:13.683+0000 7fbc937fe700 1 -- 172.21.15.142:0/2032022630 <== mon.2 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 16) ==== 137320+0+0 (secure 0 0 0) 0x7fbc94019070 con 0x7fbca4100910 2024-03-31T23:06:13.686 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-31T23:06:13.684+0000 7fbc937fe700 1 --2- 172.21.15.142:0/2032022630 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fbc8c0a2530 0x7fbc8c0a49f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:06:13.686 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-31T23:06:13.684+0000 7fbc937fe700 1 -- 172.21.15.142:0/2032022630 <== mon.2 v2:172.21.15.142:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 4965+0+0 (secure 0 0 0) 0x7fbc940cd8c0 con 0x7fbca4100910 2024-03-31T23:06:13.686 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-31T23:06:13.684+0000 7fbca1d9b700 1 --2- 172.21.15.142:0/2032022630 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fbc8c0a2530 0x7fbc8c0a49f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:06:13.686 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-31T23:06:13.685+0000 7fbca1d9b700 1 --2- 172.21.15.142:0/2032022630 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fbc8c0a2530 0x7fbc8c0a49f0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fbc98005fd0 tx=0x7fbc98005d40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:06:13.689 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-31T23:06:13.689+0000 7fbc937fe700 1 -- 172.21.15.142:0/2032022630 <== mon.2 v2:172.21.15.142:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fbc94020020 con 0x7fbca4100910 2024-03-31T23:06:13.891 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-31T23:06:13.890+0000 7fbca8b29700 1 -- 172.21.15.142:0/2032022630 --> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi142:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7fbca4062740 con 0x7fbc8c0a2530 2024-03-31T23:06:14.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:06:14 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-31T23:06:14.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:06:14 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-31T23:06:14.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:06:14 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:06:14.546 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:06:14 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-31T23:06:14.546 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:06:14 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-31T23:06:14.546 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:06:14 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:06:14.612 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:06:14 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-31T23:06:14.612 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:06:14 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-31T23:06:14.612 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:06:14 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:06:15.443 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:06:15 smithi142 ceph-mon[35271]: from='client.24290 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi142:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-31T23:06:15.443 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:06:15 smithi142 ceph-mon[35271]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-03-31T23:06:15.545 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:06:15 smithi162 ceph-mon[36237]: from='client.24290 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi142:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-31T23:06:15.546 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:06:15 smithi162 ceph-mon[36237]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-03-31T23:06:15.611 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:06:15 smithi046 ceph-mon[32498]: from='client.24290 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi142:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-31T23:06:15.611 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:06:15 smithi046 ceph-mon[32498]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-03-31T23:06:16.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:06:16 smithi046 ceph-mon[32498]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-03-31T23:06:16.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:06:16 smithi142 ceph-mon[35271]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-03-31T23:06:17.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:06:16 smithi162 ceph-mon[36237]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-03-31T23:06:17.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:06:17 smithi046 ceph-mon[32498]: from='client.? 172.21.15.142:0/1514146252' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0d0338b7-4d0a-4bb0-b152-8f5224871d9f"}]: dispatch 2024-03-31T23:06:17.862 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:06:17 smithi046 ceph-mon[32498]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0d0338b7-4d0a-4bb0-b152-8f5224871d9f"}]: dispatch 2024-03-31T23:06:17.862 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:06:17 smithi046 ceph-mon[32498]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0d0338b7-4d0a-4bb0-b152-8f5224871d9f"}]': finished 2024-03-31T23:06:17.862 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:06:17 smithi046 ceph-mon[32498]: osdmap e49: 8 total, 7 up, 8 in 2024-03-31T23:06:17.862 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:06:17 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-31T23:06:17.862 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:06:17 smithi046 ceph-mon[32498]: from='client.? 172.21.15.142:0/768299603' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-31T23:06:17.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:06:17 smithi142 ceph-mon[35271]: from='client.? 172.21.15.142:0/1514146252' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0d0338b7-4d0a-4bb0-b152-8f5224871d9f"}]: dispatch 2024-03-31T23:06:17.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:06:17 smithi142 ceph-mon[35271]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0d0338b7-4d0a-4bb0-b152-8f5224871d9f"}]: dispatch 2024-03-31T23:06:17.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:06:17 smithi142 ceph-mon[35271]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0d0338b7-4d0a-4bb0-b152-8f5224871d9f"}]': finished 2024-03-31T23:06:17.944 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:06:17 smithi142 ceph-mon[35271]: osdmap e49: 8 total, 7 up, 8 in 2024-03-31T23:06:17.944 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:06:17 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-31T23:06:17.944 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:06:17 smithi142 ceph-mon[35271]: from='client.? 172.21.15.142:0/768299603' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-31T23:06:18.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:06:17 smithi162 ceph-mon[36237]: from='client.? 172.21.15.142:0/1514146252' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0d0338b7-4d0a-4bb0-b152-8f5224871d9f"}]: dispatch 2024-03-31T23:06:18.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:06:17 smithi162 ceph-mon[36237]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0d0338b7-4d0a-4bb0-b152-8f5224871d9f"}]: dispatch 2024-03-31T23:06:18.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:06:17 smithi162 ceph-mon[36237]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0d0338b7-4d0a-4bb0-b152-8f5224871d9f"}]': finished 2024-03-31T23:06:18.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:06:17 smithi162 ceph-mon[36237]: osdmap e49: 8 total, 7 up, 8 in 2024-03-31T23:06:18.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:06:17 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-31T23:06:18.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:06:17 smithi162 ceph-mon[36237]: from='client.? 172.21.15.142:0/768299603' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-31T23:06:18.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:06:18 smithi046 ceph-mon[32498]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-03-31T23:06:18.863 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:06:18 smithi142 ceph-mon[35271]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-03-31T23:06:19.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:06:18 smithi162 ceph-mon[36237]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-03-31T23:06:20.848 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:06:20 smithi162 ceph-mon[36237]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-03-31T23:06:20.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:06:20 smithi046 ceph-mon[32498]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-03-31T23:06:20.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:06:20 smithi142 ceph-mon[35271]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-03-31T23:06:22.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:06:22 smithi046 ceph-mon[32498]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-03-31T23:06:22.913 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:06:22 smithi142 ceph-mon[35271]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-03-31T23:06:23.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:06:22 smithi162 ceph-mon[36237]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-03-31T23:06:24.443 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:06:24 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-31T23:06:24.443 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:06:24 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:06:24.545 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:06:24 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-31T23:06:24.546 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:06:24 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:06:24.611 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:06:24 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-31T23:06:24.611 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:06:24 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:06:25.443 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:06:25 smithi142 ceph-mon[35271]: Deploying daemon osd.7 on smithi142 2024-03-31T23:06:25.443 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:06:25 smithi142 ceph-mon[35271]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-03-31T23:06:25.545 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:06:25 smithi162 ceph-mon[36237]: Deploying daemon osd.7 on smithi142 2024-03-31T23:06:25.545 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:06:25 smithi162 ceph-mon[36237]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-03-31T23:06:25.611 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:06:25 smithi046 ceph-mon[32498]: Deploying daemon osd.7 on smithi142 2024-03-31T23:06:25.612 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:06:25 smithi046 ceph-mon[32498]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-03-31T23:06:26.693 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:06:26 smithi142 ceph-mon[35271]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-03-31T23:06:26.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:06:26 smithi046 ceph-mon[32498]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-03-31T23:06:27.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:06:26 smithi162 ceph-mon[36237]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-03-31T23:06:28.838 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:06:28 smithi142 ceph-mon[35271]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-03-31T23:06:28.862 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:06:28 smithi046 ceph-mon[32498]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-03-31T23:06:29.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:06:28 smithi162 ceph-mon[36237]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-03-31T23:06:31.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:06:30 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:06:31.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:06:30 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:06:31.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:06:30 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-31T23:06:31.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:06:30 smithi162 ceph-mon[36237]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-03-31T23:06:31.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:06:30 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:06:31.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:06:30 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:06:31.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:06:30 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-31T23:06:31.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:06:30 smithi046 ceph-mon[32498]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-03-31T23:06:31.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:06:30 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:06:31.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:06:30 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:06:31.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:06:30 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-31T23:06:31.194 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:06:30 smithi142 ceph-mon[35271]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-03-31T23:06:32.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:06:32 smithi046 ceph-mon[32498]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-03-31T23:06:32.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:06:32 smithi142 ceph-mon[35271]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-03-31T23:06:33.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:06:32 smithi162 ceph-mon[36237]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-03-31T23:06:33.709 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-31T23:06:33.708+0000 7fbc937fe700 1 -- 172.21.15.142:0/2032022630 <== mgr.14152 v2:172.21.15.46:6800/162289569 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fbca4062740 con 0x7fbc8c0a2530 2024-03-31T23:06:33.710 INFO:teuthology.orchestra.run.smithi142.stdout:Created osd(s) 7 on host 'smithi142' 2024-03-31T23:06:33.711 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-31T23:06:33.710+0000 7fbca8b29700 1 -- 172.21.15.142:0/2032022630 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fbc8c0a2530 msgr2=0x7fbc8c0a49f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:06:33.711 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-31T23:06:33.710+0000 7fbca8b29700 1 --2- 172.21.15.142:0/2032022630 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fbc8c0a2530 0x7fbc8c0a49f0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fbc98005fd0 tx=0x7fbc98005d40 comp rx=0 tx=0).stop 2024-03-31T23:06:33.712 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-31T23:06:33.710+0000 7fbca8b29700 1 -- 172.21.15.142:0/2032022630 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbca4100910 msgr2=0x7fbca4197240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:06:33.712 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-31T23:06:33.710+0000 7fbca8b29700 1 --2- 172.21.15.142:0/2032022630 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbca4100910 0x7fbca4197240 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fbc9400d980 tx=0x7fbc9400dc90 comp rx=0 tx=0).stop 2024-03-31T23:06:33.712 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-31T23:06:33.710+0000 7fbca8b29700 1 -- 172.21.15.142:0/2032022630 shutdown_connections 2024-03-31T23:06:33.712 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-31T23:06:33.710+0000 7fbca8b29700 1 --2- 172.21.15.142:0/2032022630 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fbc8c0a2530 0x7fbc8c0a49f0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:06:33.712 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-31T23:06:33.710+0000 7fbca8b29700 1 --2- 172.21.15.142:0/2032022630 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbca41024b0 0x7fbca419e800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:06:33.712 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-31T23:06:33.710+0000 7fbca8b29700 1 --2- 172.21.15.142:0/2032022630 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbca4100910 0x7fbca4197240 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:06:33.712 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-31T23:06:33.710+0000 7fbca8b29700 1 --2- 172.21.15.142:0/2032022630 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbca4101b10 0x7fbca4197780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:06:33.712 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-31T23:06:33.711+0000 7fbca8b29700 1 -- 172.21.15.142:0/2032022630 >> 172.21.15.142:0/2032022630 conn(0x7fbca40fbe90 msgr2=0x7fbca4104e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:06:33.712 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-31T23:06:33.711+0000 7fbca8b29700 1 -- 172.21.15.142:0/2032022630 shutdown_connections 2024-03-31T23:06:33.712 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-31T23:06:33.711+0000 7fbca8b29700 1 -- 172.21.15.142:0/2032022630 wait complete. 2024-03-31T23:06:34.080 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:06:33 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:06:34.080 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:06:33 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:06:34.080 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:06:33 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:06:34.080 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:06:33 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-31T23:06:34.080 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:06:33 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:06:34.080 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:06:33 smithi142 ceph-mon[35271]: from='osd.7 [v2:172.21.15.142:6816/2146598446,v1:172.21.15.142:6817/2146598446]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-31T23:06:34.080 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:06:33 smithi142 ceph-mon[35271]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-31T23:06:34.080 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:06:33 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:06:34.080 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:06:33 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:06:34.080 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:06:33 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-31T23:06:34.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:06:33 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:06:34.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:06:33 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:06:34.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:06:33 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:06:34.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:06:33 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-31T23:06:34.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:06:33 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:06:34.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:06:33 smithi046 ceph-mon[32498]: from='osd.7 [v2:172.21.15.142:6816/2146598446,v1:172.21.15.142:6817/2146598446]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-31T23:06:34.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:06:33 smithi046 ceph-mon[32498]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-31T23:06:34.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:06:33 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:06:34.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:06:33 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:06:34.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:06:33 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-31T23:06:34.296 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:06:33 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:06:34.296 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:06:33 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:06:34.296 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:06:33 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:06:34.296 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:06:33 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-31T23:06:34.296 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:06:33 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:06:34.296 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:06:33 smithi162 ceph-mon[36237]: from='osd.7 [v2:172.21.15.142:6816/2146598446,v1:172.21.15.142:6817/2146598446]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-31T23:06:34.296 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:06:33 smithi162 ceph-mon[36237]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-31T23:06:34.296 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:06:33 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:06:34.297 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:06:33 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:06:34.297 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:06:33 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-31T23:06:34.399 DEBUG:teuthology.orchestra.run.smithi142:osd.7> sudo journalctl -f -n 0 -u ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4@osd.7.service 2024-03-31T23:06:34.401 INFO:tasks.cephadm:Deploying osd.8 on smithi162 with /dev/vg_nvme/lv_2... 2024-03-31T23:06:34.402 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:384f31f3d3aee7f8ba3f60846b8844c049165583 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6a9d8f08-efb2-11ee-b647-cb9ed24678a4 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-03-31T23:06:34.648 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/6a9d8f08-efb2-11ee-b647-cb9ed24678a4/mon.c/config 2024-03-31T23:06:34.944 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:06:34 smithi142 ceph-mon[35271]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-31T23:06:34.944 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:06:34 smithi142 ceph-mon[35271]: osdmap e50: 8 total, 7 up, 8 in 2024-03-31T23:06:34.944 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:06:34 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-31T23:06:34.944 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:06:34 smithi142 ceph-mon[35271]: from='osd.7 [v2:172.21.15.142:6816/2146598446,v1:172.21.15.142:6817/2146598446]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi142", "root=default"]}]: dispatch 2024-03-31T23:06:34.944 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:06:34 smithi142 ceph-mon[35271]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi142", "root=default"]}]: dispatch 2024-03-31T23:06:34.944 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:06:34 smithi142 ceph-mon[35271]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-03-31T23:06:34.944 INFO:journalctl@ceph.osd.7.smithi142.stdout:Mar 31 23:06:34 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-7[53065]: 2024-03-31T23:06:34.818+0000 7fa41a098700 -1 osd.7 0 waiting for initial osdmap 2024-03-31T23:06:34.944 INFO:journalctl@ceph.osd.7.smithi142.stdout:Mar 31 23:06:34 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-7[53065]: 2024-03-31T23:06:34.826+0000 7fa415e8b700 -1 osd.7 51 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-31T23:06:35.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:06:34 smithi162 ceph-mon[36237]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-31T23:06:35.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:06:34 smithi162 ceph-mon[36237]: osdmap e50: 8 total, 7 up, 8 in 2024-03-31T23:06:35.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:06:34 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-31T23:06:35.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:06:34 smithi162 ceph-mon[36237]: from='osd.7 [v2:172.21.15.142:6816/2146598446,v1:172.21.15.142:6817/2146598446]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi142", "root=default"]}]: dispatch 2024-03-31T23:06:35.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:06:34 smithi162 ceph-mon[36237]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi142", "root=default"]}]: dispatch 2024-03-31T23:06:35.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:06:34 smithi162 ceph-mon[36237]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-03-31T23:06:35.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:06:34 smithi046 ceph-mon[32498]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-31T23:06:35.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:06:34 smithi046 ceph-mon[32498]: osdmap e50: 8 total, 7 up, 8 in 2024-03-31T23:06:35.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:06:34 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-31T23:06:35.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:06:34 smithi046 ceph-mon[32498]: from='osd.7 [v2:172.21.15.142:6816/2146598446,v1:172.21.15.142:6817/2146598446]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi142", "root=default"]}]: dispatch 2024-03-31T23:06:35.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:06:34 smithi046 ceph-mon[32498]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi142", "root=default"]}]: dispatch 2024-03-31T23:06:35.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:06:34 smithi046 ceph-mon[32498]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-03-31T23:06:35.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:06:35 smithi142 ceph-mon[35271]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi142", "root=default"]}]': finished 2024-03-31T23:06:35.944 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:06:35 smithi142 ceph-mon[35271]: osdmap e51: 8 total, 7 up, 8 in 2024-03-31T23:06:35.944 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:06:35 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-31T23:06:35.944 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:06:35 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-31T23:06:35.944 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:06:35 smithi142 ceph-mon[35271]: osd.7 [v2:172.21.15.142:6816/2146598446,v1:172.21.15.142:6817/2146598446] boot 2024-03-31T23:06:35.944 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:06:35 smithi142 ceph-mon[35271]: osdmap e52: 8 total, 8 up, 8 in 2024-03-31T23:06:35.944 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:06:35 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-31T23:06:36.093 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:06:35 smithi162 ceph-mon[36237]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi142", "root=default"]}]': finished 2024-03-31T23:06:36.093 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:06:35 smithi162 ceph-mon[36237]: osdmap e51: 8 total, 7 up, 8 in 2024-03-31T23:06:36.093 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:06:35 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-31T23:06:36.093 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:06:35 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-31T23:06:36.093 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:06:35 smithi162 ceph-mon[36237]: osd.7 [v2:172.21.15.142:6816/2146598446,v1:172.21.15.142:6817/2146598446] boot 2024-03-31T23:06:36.093 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:06:35 smithi162 ceph-mon[36237]: osdmap e52: 8 total, 8 up, 8 in 2024-03-31T23:06:36.093 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:06:35 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-31T23:06:36.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:06:35 smithi046 ceph-mon[32498]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi142", "root=default"]}]': finished 2024-03-31T23:06:36.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:06:35 smithi046 ceph-mon[32498]: osdmap e51: 8 total, 7 up, 8 in 2024-03-31T23:06:36.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:06:35 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-31T23:06:36.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:06:35 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-31T23:06:36.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:06:35 smithi046 ceph-mon[32498]: osd.7 [v2:172.21.15.142:6816/2146598446,v1:172.21.15.142:6817/2146598446] boot 2024-03-31T23:06:36.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:06:35 smithi046 ceph-mon[32498]: osdmap e52: 8 total, 8 up, 8 in 2024-03-31T23:06:36.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:06:35 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-31T23:06:36.693 INFO:teuthology.orchestra.run.smithi162.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-03-31T23:06:36.693 INFO:teuthology.orchestra.run.smithi162.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-03-31T23:06:36.694 INFO:teuthology.orchestra.run.smithi162.stderr: stderr: 10+0 records in 2024-03-31T23:06:36.694 INFO:teuthology.orchestra.run.smithi162.stderr:10+0 records out 2024-03-31T23:06:36.694 INFO:teuthology.orchestra.run.smithi162.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0122303 s, 857 MB/s 2024-03-31T23:06:36.694 INFO:teuthology.orchestra.run.smithi162.stderr:--> Zapping successful for: 2024-03-31T23:06:37.284 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:384f31f3d3aee7f8ba3f60846b8844c049165583 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6a9d8f08-efb2-11ee-b647-cb9ed24678a4 -- ceph orch daemon add osd smithi162:vg_nvme/lv_2 2024-03-31T23:06:37.443 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:06:37 smithi142 ceph-mon[35271]: purged_snaps scrub starts 2024-03-31T23:06:37.443 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:06:37 smithi142 ceph-mon[35271]: purged_snaps scrub ok 2024-03-31T23:06:37.444 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:06:37 smithi142 ceph-mon[35271]: osdmap e53: 8 total, 8 up, 8 in 2024-03-31T23:06:37.444 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:06:37 smithi142 ceph-mon[35271]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-03-31T23:06:37.530 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/6a9d8f08-efb2-11ee-b647-cb9ed24678a4/mon.c/config 2024-03-31T23:06:37.545 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:06:37 smithi162 ceph-mon[36237]: purged_snaps scrub starts 2024-03-31T23:06:37.545 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:06:37 smithi162 ceph-mon[36237]: purged_snaps scrub ok 2024-03-31T23:06:37.546 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:06:37 smithi162 ceph-mon[36237]: osdmap e53: 8 total, 8 up, 8 in 2024-03-31T23:06:37.546 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:06:37 smithi162 ceph-mon[36237]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-03-31T23:06:37.611 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:06:37 smithi046 ceph-mon[32498]: purged_snaps scrub starts 2024-03-31T23:06:37.612 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:06:37 smithi046 ceph-mon[32498]: purged_snaps scrub ok 2024-03-31T23:06:37.612 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:06:37 smithi046 ceph-mon[32498]: osdmap e53: 8 total, 8 up, 8 in 2024-03-31T23:06:37.612 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:06:37 smithi046 ceph-mon[32498]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-03-31T23:06:38.612 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:06:38 smithi046 ceph-mon[32498]: osdmap e54: 8 total, 8 up, 8 in 2024-03-31T23:06:38.612 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:06:38 smithi046 ceph-mon[32498]: Detected new or changed devices on smithi142 2024-03-31T23:06:38.612 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:06:38 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:06:38.612 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:06:38 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:06:38.612 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:06:38 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-31T23:06:38.612 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:06:38 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-31T23:06:38.612 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:06:38 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-31T23:06:38.612 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:06:38 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:06:38.612 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:06:38 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:06:38.612 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:06:38 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-31T23:06:38.612 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:06:38 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:06:38.678 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:06:38 smithi162 ceph-mon[36237]: osdmap e54: 8 total, 8 up, 8 in 2024-03-31T23:06:38.679 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:06:38 smithi162 ceph-mon[36237]: Detected new or changed devices on smithi142 2024-03-31T23:06:38.679 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:06:38 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:06:38.679 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:06:38 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:06:38.679 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:06:38 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-31T23:06:38.679 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:06:38 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-31T23:06:38.679 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:06:38 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-31T23:06:38.679 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:06:38 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:06:38.679 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:06:38 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:06:38.679 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:06:38 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-31T23:06:38.679 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:06:38 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:06:38.693 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:06:38 smithi142 ceph-mon[35271]: osdmap e54: 8 total, 8 up, 8 in 2024-03-31T23:06:38.693 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:06:38 smithi142 ceph-mon[35271]: Detected new or changed devices on smithi142 2024-03-31T23:06:38.693 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:06:38 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:06:38.694 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:06:38 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:06:38.694 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:06:38 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-31T23:06:38.694 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:06:38 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-31T23:06:38.694 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:06:38 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-31T23:06:38.694 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:06:38 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:06:38.694 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:06:38 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:06:38.694 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:06:38 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-31T23:06:38.694 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:06:38 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:06:39.202 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:06:39.200+0000 7f415660b700 1 -- 172.21.15.162:0/849581530 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4150103e10 msgr2=0x7f415010a6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:06:39.202 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:06:39.200+0000 7f415660b700 1 --2- 172.21.15.162:0/849581530 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4150103e10 0x7f415010a6c0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f414800b0a0 tx=0x7f414800b3b0 comp rx=0 tx=0).stop 2024-03-31T23:06:39.203 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:06:39.200+0000 7f415660b700 1 -- 172.21.15.162:0/849581530 shutdown_connections 2024-03-31T23:06:39.203 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:06:39.200+0000 7f415660b700 1 --2- 172.21.15.162:0/849581530 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4150102270 0x7f4150102690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:06:39.203 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:06:39.200+0000 7f415660b700 1 --2- 172.21.15.162:0/849581530 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4150103e10 0x7f415010a6c0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:06:39.203 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:06:39.200+0000 7f415660b700 1 --2- 172.21.15.162:0/849581530 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4150103470 0x7f41501038d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:06:39.203 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:06:39.200+0000 7f415660b700 1 -- 172.21.15.162:0/849581530 >> 172.21.15.162:0/849581530 conn(0x7f41500fd850 msgr2=0x7f41500ffc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:06:39.203 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:06:39.200+0000 7f415660b700 1 -- 172.21.15.162:0/849581530 shutdown_connections 2024-03-31T23:06:39.203 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:06:39.200+0000 7f415660b700 1 -- 172.21.15.162:0/849581530 wait complete. 2024-03-31T23:06:39.203 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:06:39.200+0000 7f415660b700 1 Processor -- start 2024-03-31T23:06:39.203 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:06:39.200+0000 7f415660b700 1 -- start start 2024-03-31T23:06:39.203 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:06:39.201+0000 7f415660b700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4150102270 0x7f4150198340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:06:39.203 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:06:39.201+0000 7f415660b700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4150103470 0x7f4150198880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:06:39.203 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:06:39.201+0000 7f415660b700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4150103e10 0x7f415019f900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:06:39.203 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:06:39.201+0000 7f415660b700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f415010dd00 con 0x7f4150103e10 2024-03-31T23:06:39.204 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:06:39.201+0000 7f415660b700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f415010db80 con 0x7f4150103470 2024-03-31T23:06:39.204 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:06:39.201+0000 7f415660b700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f415010de80 con 0x7f4150102270 2024-03-31T23:06:39.204 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:06:39.201+0000 7f414ffff700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4150102270 0x7f4150198340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:06:39.204 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:06:39.201+0000 7f414ffff700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4150102270 0x7f4150198340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:49838/0 (socket says 172.21.15.162:49838) 2024-03-31T23:06:39.204 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:06:39.201+0000 7f414ffff700 1 -- 172.21.15.162:0/4045279590 learned_addr learned my addr 172.21.15.162:0/4045279590 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-03-31T23:06:39.204 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:06:39.201+0000 7f4154ba8700 1 --2- 172.21.15.162:0/4045279590 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4150103e10 0x7f415019f900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:06:39.204 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:06:39.201+0000 7f414f7fe700 1 --2- 172.21.15.162:0/4045279590 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4150103470 0x7f4150198880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:06:39.204 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:06:39.201+0000 7f414ffff700 1 -- 172.21.15.162:0/4045279590 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4150103470 msgr2=0x7f4150198880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:06:39.204 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:06:39.201+0000 7f414ffff700 1 --2- 172.21.15.162:0/4045279590 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4150103470 0x7f4150198880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:06:39.204 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:06:39.201+0000 7f414ffff700 1 -- 172.21.15.162:0/4045279590 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4150103e10 msgr2=0x7f415019f900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:06:39.204 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:06:39.201+0000 7f414ffff700 1 --2- 172.21.15.162:0/4045279590 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4150103e10 0x7f415019f900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:06:39.204 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:06:39.201+0000 7f414ffff700 1 -- 172.21.15.162:0/4045279590 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f415019ffa0 con 0x7f4150102270 2024-03-31T23:06:39.204 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:06:39.201+0000 7f414f7fe700 1 --2- 172.21.15.162:0/4045279590 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4150103470 0x7f4150198880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T23:06:39.204 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:06:39.201+0000 7f4154ba8700 1 --2- 172.21.15.162:0/4045279590 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4150103e10 0x7f415019f900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:06:39.205 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:06:39.201+0000 7f414ffff700 1 --2- 172.21.15.162:0/4045279590 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4150102270 0x7f4150198340 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f414400e9a0 tx=0x7f414400ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:06:39.205 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:06:39.201+0000 7f414d7fa700 1 -- 172.21.15.162:0/4045279590 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f414400cbd0 con 0x7f4150102270 2024-03-31T23:06:39.205 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:06:39.201+0000 7f415660b700 1 -- 172.21.15.162:0/4045279590 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f41501a0290 con 0x7f4150102270 2024-03-31T23:06:39.205 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:06:39.201+0000 7f414d7fa700 1 -- 172.21.15.162:0/4045279590 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f41440044d0 con 0x7f4150102270 2024-03-31T23:06:39.205 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:06:39.201+0000 7f414d7fa700 1 -- 172.21.15.162:0/4045279590 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4144003e30 con 0x7f4150102270 2024-03-31T23:06:39.205 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:06:39.202+0000 7f415660b700 1 -- 172.21.15.162:0/4045279590 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f41501a07d0 con 0x7f4150102270 2024-03-31T23:06:39.206 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:06:39.202+0000 7f415660b700 1 -- 172.21.15.162:0/4045279590 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4150068f00 con 0x7f4150102270 2024-03-31T23:06:39.206 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:06:39.204+0000 7f414d7fa700 1 -- 172.21.15.162:0/4045279590 <== mon.1 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 16) ==== 137320+0+0 (secure 0 0 0) 0x7f4144004000 con 0x7f4150102270 2024-03-31T23:06:39.206 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:06:39.204+0000 7f414d7fa700 1 --2- 172.21.15.162:0/4045279590 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f41380a2530 0x7f41380a49f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:06:39.206 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:06:39.204+0000 7f414d7fa700 1 -- 172.21.15.162:0/4045279590 <== mon.1 v2:172.21.15.162:3300/0 5 ==== osd_map(55..55 src has 1..55) ==== 5397+0+0 (secure 0 0 0) 0x7f41440cdf10 con 0x7f4150102270 2024-03-31T23:06:39.206 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:06:39.204+0000 7f414f7fe700 1 --2- 172.21.15.162:0/4045279590 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f41380a2530 0x7f41380a49f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:06:39.206 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:06:39.205+0000 7f414f7fe700 1 --2- 172.21.15.162:0/4045279590 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f41380a2530 0x7f41380a49f0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f4140009cb0 tx=0x7f4140009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:06:39.208 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:06:39.208+0000 7f414d7fa700 1 -- 172.21.15.162:0/4045279590 <== mon.1 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f41440042f0 con 0x7f4150102270 2024-03-31T23:06:39.412 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:06:39.411+0000 7f415660b700 1 -- 172.21.15.162:0/4045279590 --> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi162:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7f4150062680 con 0x7f41380a2530 2024-03-31T23:06:39.612 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:06:39 smithi046 ceph-mon[32498]: Adjusting osd_memory_target on smithi142 to 5696M 2024-03-31T23:06:39.612 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:06:39 smithi046 ceph-mon[32498]: pgmap v179: 1 pgs: 1 remapped+peering; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-03-31T23:06:39.612 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:06:39 smithi046 ceph-mon[32498]: osdmap e55: 8 total, 8 up, 8 in 2024-03-31T23:06:39.693 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:06:39 smithi142 ceph-mon[35271]: Adjusting osd_memory_target on smithi142 to 5696M 2024-03-31T23:06:39.693 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:06:39 smithi142 ceph-mon[35271]: pgmap v179: 1 pgs: 1 remapped+peering; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-03-31T23:06:39.693 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:06:39 smithi142 ceph-mon[35271]: osdmap e55: 8 total, 8 up, 8 in 2024-03-31T23:06:39.795 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:06:39 smithi162 ceph-mon[36237]: Adjusting osd_memory_target on smithi142 to 5696M 2024-03-31T23:06:39.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:06:39 smithi162 ceph-mon[36237]: pgmap v179: 1 pgs: 1 remapped+peering; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-03-31T23:06:39.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:06:39 smithi162 ceph-mon[36237]: osdmap e55: 8 total, 8 up, 8 in 2024-03-31T23:06:40.546 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:06:40 smithi162 ceph-mon[36237]: from='client.24289 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi162:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-31T23:06:40.546 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:06:40 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-31T23:06:40.546 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:06:40 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-31T23:06:40.546 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:06:40 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:06:40.546 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:06:40 smithi162 ceph-mon[36237]: osdmap e56: 8 total, 8 up, 8 in 2024-03-31T23:06:40.611 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:06:40 smithi046 ceph-mon[32498]: from='client.24289 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi162:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-31T23:06:40.612 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:06:40 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-31T23:06:40.612 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:06:40 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-31T23:06:40.612 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:06:40 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:06:40.612 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:06:40 smithi046 ceph-mon[32498]: osdmap e56: 8 total, 8 up, 8 in 2024-03-31T23:06:40.693 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:06:40 smithi142 ceph-mon[35271]: from='client.24289 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi162:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-31T23:06:40.693 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:06:40 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-31T23:06:40.693 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:06:40 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-31T23:06:40.693 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:06:40 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:06:40.693 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:06:40 smithi142 ceph-mon[35271]: osdmap e56: 8 total, 8 up, 8 in 2024-03-31T23:06:41.546 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:06:41 smithi162 ceph-mon[36237]: pgmap v182: 1 pgs: 1 remapped+peering; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-03-31T23:06:41.693 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:06:41 smithi142 ceph-mon[35271]: pgmap v182: 1 pgs: 1 remapped+peering; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-03-31T23:06:41.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:06:41 smithi046 ceph-mon[32498]: pgmap v182: 1 pgs: 1 remapped+peering; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-03-31T23:06:42.693 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:06:42 smithi142 ceph-mon[35271]: from='client.? 172.21.15.162:0/1397107711' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8b6f6037-892a-416c-babb-7552648d0b3a"}]: dispatch 2024-03-31T23:06:42.693 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:06:42 smithi142 ceph-mon[35271]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8b6f6037-892a-416c-babb-7552648d0b3a"}]: dispatch 2024-03-31T23:06:42.694 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:06:42 smithi142 ceph-mon[35271]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8b6f6037-892a-416c-babb-7552648d0b3a"}]': finished 2024-03-31T23:06:42.694 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:06:42 smithi142 ceph-mon[35271]: osdmap e57: 9 total, 8 up, 9 in 2024-03-31T23:06:42.694 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:06:42 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-31T23:06:42.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:06:42 smithi162 ceph-mon[36237]: from='client.? 172.21.15.162:0/1397107711' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8b6f6037-892a-416c-babb-7552648d0b3a"}]: dispatch 2024-03-31T23:06:42.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:06:42 smithi162 ceph-mon[36237]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8b6f6037-892a-416c-babb-7552648d0b3a"}]: dispatch 2024-03-31T23:06:42.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:06:42 smithi162 ceph-mon[36237]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8b6f6037-892a-416c-babb-7552648d0b3a"}]': finished 2024-03-31T23:06:42.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:06:42 smithi162 ceph-mon[36237]: osdmap e57: 9 total, 8 up, 9 in 2024-03-31T23:06:42.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:06:42 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-31T23:06:42.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:06:42 smithi046 ceph-mon[32498]: from='client.? 172.21.15.162:0/1397107711' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8b6f6037-892a-416c-babb-7552648d0b3a"}]: dispatch 2024-03-31T23:06:42.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:06:42 smithi046 ceph-mon[32498]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8b6f6037-892a-416c-babb-7552648d0b3a"}]: dispatch 2024-03-31T23:06:42.862 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:06:42 smithi046 ceph-mon[32498]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8b6f6037-892a-416c-babb-7552648d0b3a"}]': finished 2024-03-31T23:06:42.862 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:06:42 smithi046 ceph-mon[32498]: osdmap e57: 9 total, 8 up, 9 in 2024-03-31T23:06:42.862 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:06:42 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-31T23:06:43.693 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:06:43 smithi142 ceph-mon[35271]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 234 MiB used, 715 GiB / 715 GiB avail 2024-03-31T23:06:43.693 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:06:43 smithi142 ceph-mon[35271]: from='client.? 172.21.15.162:0/251293496' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-31T23:06:43.795 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:06:43 smithi162 ceph-mon[36237]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 234 MiB used, 715 GiB / 715 GiB avail 2024-03-31T23:06:43.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:06:43 smithi162 ceph-mon[36237]: from='client.? 172.21.15.162:0/251293496' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-31T23:06:43.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:06:43 smithi046 ceph-mon[32498]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 234 MiB used, 715 GiB / 715 GiB avail 2024-03-31T23:06:43.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:06:43 smithi046 ceph-mon[32498]: from='client.? 172.21.15.162:0/251293496' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-31T23:06:44.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:06:44 smithi046 ceph-mon[32498]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 234 MiB used, 715 GiB / 715 GiB avail 2024-03-31T23:06:44.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:06:44 smithi142 ceph-mon[35271]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 234 MiB used, 715 GiB / 715 GiB avail 2024-03-31T23:06:45.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:06:44 smithi162 ceph-mon[36237]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 234 MiB used, 715 GiB / 715 GiB avail 2024-03-31T23:06:46.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:06:46 smithi046 ceph-mon[32498]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-03-31T23:06:46.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:06:46 smithi142 ceph-mon[35271]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-03-31T23:06:46.959 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:06:46 smithi162 ceph-mon[36237]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-03-31T23:06:48.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:06:48 smithi046 ceph-mon[32498]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-03-31T23:06:48.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:06:48 smithi142 ceph-mon[35271]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-03-31T23:06:49.018 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:06:48 smithi162 ceph-mon[36237]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-03-31T23:06:49.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:06:49 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-03-31T23:06:49.862 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:06:49 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:06:49.862 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:06:49 smithi046 ceph-mon[32498]: Deploying daemon osd.8 on smithi162 2024-03-31T23:06:49.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:06:49 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-03-31T23:06:49.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:06:49 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:06:49.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:06:49 smithi142 ceph-mon[35271]: Deploying daemon osd.8 on smithi162 2024-03-31T23:06:49.970 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:06:49 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-03-31T23:06:49.970 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:06:49 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:06:49.970 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:06:49 smithi162 ceph-mon[36237]: Deploying daemon osd.8 on smithi162 2024-03-31T23:06:50.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:06:50 smithi162 ceph-mon[36237]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-03-31T23:06:50.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:06:50 smithi046 ceph-mon[32498]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-03-31T23:06:50.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:06:50 smithi142 ceph-mon[35271]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-03-31T23:06:52.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:06:52 smithi046 ceph-mon[32498]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-03-31T23:06:52.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:06:52 smithi142 ceph-mon[35271]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-03-31T23:06:53.011 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:06:52 smithi162 ceph-mon[36237]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-03-31T23:06:54.770 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:06:54 smithi162 ceph-mon[36237]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-03-31T23:06:54.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:06:54 smithi046 ceph-mon[32498]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-03-31T23:06:54.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:06:54 smithi142 ceph-mon[35271]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-03-31T23:06:56.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:06:55 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:06:56.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:06:55 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:06:56.194 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:06:55 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-31T23:06:56.295 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:06:55 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:06:56.296 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:06:55 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:06:56.296 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:06:55 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-31T23:06:56.361 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:06:55 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:06:56.362 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:06:55 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:06:56.362 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:06:55 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-31T23:06:57.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:06:56 smithi142 ceph-mon[35271]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-03-31T23:06:57.223 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:06:56 smithi162 ceph-mon[36237]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-03-31T23:06:57.361 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:06:56 smithi046 ceph-mon[32498]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-03-31T23:06:58.728 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:06:58.726+0000 7f414d7fa700 1 -- 172.21.15.162:0/4045279590 <== mgr.14152 v2:172.21.15.46:6800/162289569 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f4150062680 con 0x7f41380a2530 2024-03-31T23:06:58.728 INFO:teuthology.orchestra.run.smithi162.stdout:Created osd(s) 8 on host 'smithi162' 2024-03-31T23:06:58.730 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:06:58.729+0000 7f415660b700 1 -- 172.21.15.162:0/4045279590 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f41380a2530 msgr2=0x7f41380a49f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:06:58.730 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:06:58.729+0000 7f415660b700 1 --2- 172.21.15.162:0/4045279590 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f41380a2530 0x7f41380a49f0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f4140009cb0 tx=0x7f4140009260 comp rx=0 tx=0).stop 2024-03-31T23:06:58.730 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:06:58.729+0000 7f415660b700 1 -- 172.21.15.162:0/4045279590 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4150102270 msgr2=0x7f4150198340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:06:58.730 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:06:58.729+0000 7f415660b700 1 --2- 172.21.15.162:0/4045279590 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4150102270 0x7f4150198340 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f414400e9a0 tx=0x7f414400ecb0 comp rx=0 tx=0).stop 2024-03-31T23:06:58.730 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:06:58.729+0000 7f415660b700 1 -- 172.21.15.162:0/4045279590 shutdown_connections 2024-03-31T23:06:58.731 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:06:58.729+0000 7f415660b700 1 --2- 172.21.15.162:0/4045279590 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f41380a2530 0x7f41380a49f0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:06:58.731 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:06:58.729+0000 7f415660b700 1 --2- 172.21.15.162:0/4045279590 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4150103e10 0x7f415019f900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:06:58.731 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:06:58.729+0000 7f415660b700 1 --2- 172.21.15.162:0/4045279590 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4150102270 0x7f4150198340 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:06:58.731 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:06:58.729+0000 7f415660b700 1 --2- 172.21.15.162:0/4045279590 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4150103470 0x7f4150198880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:06:58.731 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:06:58.729+0000 7f415660b700 1 -- 172.21.15.162:0/4045279590 >> 172.21.15.162:0/4045279590 conn(0x7f41500fd850 msgr2=0x7f4150106840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:06:58.731 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:06:58.729+0000 7f415660b700 1 -- 172.21.15.162:0/4045279590 shutdown_connections 2024-03-31T23:06:58.731 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:06:58.729+0000 7f415660b700 1 -- 172.21.15.162:0/4045279590 wait complete. 2024-03-31T23:06:59.025 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:06:58 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:06:59.026 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:06:58 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:06:59.026 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:06:58 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:06:59.026 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:06:58 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-31T23:06:59.026 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:06:58 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:06:59.026 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:06:58 smithi162 ceph-mon[36237]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-03-31T23:06:59.026 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:06:58 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:06:59.026 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:06:58 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:06:59.026 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:06:58 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-31T23:06:59.026 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:06:58 smithi162 ceph-mon[36237]: from='osd.8 [v2:172.21.15.162:6816/1173078318,v1:172.21.15.162:6817/1173078318]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-31T23:06:59.026 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:06:58 smithi162 ceph-mon[36237]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-31T23:06:59.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:06:58 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:06:59.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:06:58 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:06:59.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:06:58 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:06:59.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:06:58 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-31T23:06:59.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:06:58 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:06:59.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:06:58 smithi046 ceph-mon[32498]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-03-31T23:06:59.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:06:58 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:06:59.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:06:58 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:06:59.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:06:58 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-31T23:06:59.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:06:58 smithi046 ceph-mon[32498]: from='osd.8 [v2:172.21.15.162:6816/1173078318,v1:172.21.15.162:6817/1173078318]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-31T23:06:59.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:06:58 smithi046 ceph-mon[32498]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-31T23:06:59.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:06:58 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:06:59.194 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:06:58 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:06:59.194 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:06:58 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:06:59.194 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:06:58 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-31T23:06:59.194 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:06:58 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:06:59.194 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:06:58 smithi142 ceph-mon[35271]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-03-31T23:06:59.194 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:06:58 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:06:59.194 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:06:58 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:06:59.194 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:06:58 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-31T23:06:59.194 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:06:58 smithi142 ceph-mon[35271]: from='osd.8 [v2:172.21.15.162:6816/1173078318,v1:172.21.15.162:6817/1173078318]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-31T23:06:59.194 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:06:58 smithi142 ceph-mon[35271]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-31T23:06:59.386 DEBUG:teuthology.orchestra.run.smithi162:osd.8> sudo journalctl -f -n 0 -u ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4@osd.8.service 2024-03-31T23:06:59.389 INFO:tasks.cephadm:Deploying osd.9 on smithi046 with /dev/vg_nvme/lv_1... 2024-03-31T23:06:59.389 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:384f31f3d3aee7f8ba3f60846b8844c049165583 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6a9d8f08-efb2-11ee-b647-cb9ed24678a4 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-03-31T23:06:59.644 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/6a9d8f08-efb2-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-31T23:07:00.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:07:00 smithi162 ceph-mon[36237]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-03-31T23:07:00.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:07:00 smithi162 ceph-mon[36237]: osdmap e58: 9 total, 8 up, 9 in 2024-03-31T23:07:00.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:07:00 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-31T23:07:00.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:07:00 smithi162 ceph-mon[36237]: from='osd.8 [v2:172.21.15.162:6816/1173078318,v1:172.21.15.162:6817/1173078318]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]: dispatch 2024-03-31T23:07:00.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:07:00 smithi162 ceph-mon[36237]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]: dispatch 2024-03-31T23:07:00.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:07:00 smithi162 ceph-mon[36237]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-03-31T23:07:00.797 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 31 23:07:00 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-8[55211]: 2024-03-31T23:07:00.733+0000 7fee87e73700 -1 osd.8 0 waiting for initial osdmap 2024-03-31T23:07:00.797 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 31 23:07:00 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-8[55211]: 2024-03-31T23:07:00.743+0000 7fee83c66700 -1 osd.8 59 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-31T23:07:01.081 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:07:00 smithi046 ceph-mon[32498]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-03-31T23:07:01.081 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:07:00 smithi046 ceph-mon[32498]: osdmap e58: 9 total, 8 up, 9 in 2024-03-31T23:07:01.081 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:07:00 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-31T23:07:01.082 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:07:00 smithi046 ceph-mon[32498]: from='osd.8 [v2:172.21.15.162:6816/1173078318,v1:172.21.15.162:6817/1173078318]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]: dispatch 2024-03-31T23:07:01.082 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:07:00 smithi046 ceph-mon[32498]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]: dispatch 2024-03-31T23:07:01.082 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:07:00 smithi046 ceph-mon[32498]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-03-31T23:07:01.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:07:00 smithi142 ceph-mon[35271]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-03-31T23:07:01.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:07:00 smithi142 ceph-mon[35271]: osdmap e58: 9 total, 8 up, 9 in 2024-03-31T23:07:01.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:07:00 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-31T23:07:01.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:07:00 smithi142 ceph-mon[35271]: from='osd.8 [v2:172.21.15.162:6816/1173078318,v1:172.21.15.162:6817/1173078318]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]: dispatch 2024-03-31T23:07:01.194 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:07:00 smithi142 ceph-mon[35271]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]: dispatch 2024-03-31T23:07:01.194 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:07:00 smithi142 ceph-mon[35271]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-03-31T23:07:01.787 INFO:teuthology.orchestra.run.smithi046.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-03-31T23:07:01.788 INFO:teuthology.orchestra.run.smithi046.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-03-31T23:07:01.788 INFO:teuthology.orchestra.run.smithi046.stderr: stderr: 10+0 records in 2024-03-31T23:07:01.788 INFO:teuthology.orchestra.run.smithi046.stderr:10+0 records out 2024-03-31T23:07:01.788 INFO:teuthology.orchestra.run.smithi046.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.00850533 s, 1.2 GB/s 2024-03-31T23:07:01.788 INFO:teuthology.orchestra.run.smithi046.stderr: stderr: 2024-03-31T23:07:01.788 INFO:teuthology.orchestra.run.smithi046.stderr:--> Zapping successful for: 2024-03-31T23:07:01.795 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:07:01 smithi162 ceph-mon[36237]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]': finished 2024-03-31T23:07:01.795 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:07:01 smithi162 ceph-mon[36237]: osdmap e59: 9 total, 8 up, 9 in 2024-03-31T23:07:01.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:07:01 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-31T23:07:01.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:07:01 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-31T23:07:02.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:07:01 smithi046 ceph-mon[32498]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]': finished 2024-03-31T23:07:02.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:07:01 smithi046 ceph-mon[32498]: osdmap e59: 9 total, 8 up, 9 in 2024-03-31T23:07:02.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:07:01 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-31T23:07:02.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:07:01 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-31T23:07:02.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:07:01 smithi142 ceph-mon[35271]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]': finished 2024-03-31T23:07:02.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:07:01 smithi142 ceph-mon[35271]: osdmap e59: 9 total, 8 up, 9 in 2024-03-31T23:07:02.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:07:01 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-31T23:07:02.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:07:01 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-31T23:07:02.680 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:384f31f3d3aee7f8ba3f60846b8844c049165583 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6a9d8f08-efb2-11ee-b647-cb9ed24678a4 -- ceph orch daemon add osd smithi046:vg_nvme/lv_1 2024-03-31T23:07:02.937 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/6a9d8f08-efb2-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-31T23:07:03.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:07:02 smithi162 ceph-mon[36237]: purged_snaps scrub starts 2024-03-31T23:07:03.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:07:02 smithi162 ceph-mon[36237]: purged_snaps scrub ok 2024-03-31T23:07:03.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:07:02 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-31T23:07:03.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:07:02 smithi162 ceph-mon[36237]: osd.8 [v2:172.21.15.162:6816/1173078318,v1:172.21.15.162:6817/1173078318] boot 2024-03-31T23:07:03.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:07:02 smithi162 ceph-mon[36237]: osdmap e60: 9 total, 9 up, 9 in 2024-03-31T23:07:03.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:07:02 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-31T23:07:03.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:07:02 smithi162 ceph-mon[36237]: Detected new or changed devices on smithi162 2024-03-31T23:07:03.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:07:02 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:07:03.047 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:07:02 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:07:03.047 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:07:02 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-31T23:07:03.047 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:07:02 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-31T23:07:03.047 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:07:02 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-31T23:07:03.047 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:07:02 smithi162 ceph-mon[36237]: Adjusting osd_memory_target on smithi162 to 5696M 2024-03-31T23:07:03.047 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:07:02 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:07:03.047 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:07:02 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:07:03.047 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:07:02 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-31T23:07:03.047 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:07:02 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:07:03.047 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:07:02 smithi162 ceph-mon[36237]: pgmap v197: 1 pgs: 1 peering; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-03-31T23:07:03.059 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:07:02 smithi046 ceph-mon[32498]: purged_snaps scrub starts 2024-03-31T23:07:03.059 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:07:02 smithi046 ceph-mon[32498]: purged_snaps scrub ok 2024-03-31T23:07:03.059 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:07:02 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-31T23:07:03.059 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:07:02 smithi046 ceph-mon[32498]: osd.8 [v2:172.21.15.162:6816/1173078318,v1:172.21.15.162:6817/1173078318] boot 2024-03-31T23:07:03.059 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:07:02 smithi046 ceph-mon[32498]: osdmap e60: 9 total, 9 up, 9 in 2024-03-31T23:07:03.059 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:07:02 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-31T23:07:03.059 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:07:02 smithi046 ceph-mon[32498]: Detected new or changed devices on smithi162 2024-03-31T23:07:03.060 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:07:02 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:07:03.060 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:07:02 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:07:03.060 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:07:02 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-31T23:07:03.060 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:07:02 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-31T23:07:03.060 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:07:02 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-31T23:07:03.060 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:07:02 smithi046 ceph-mon[32498]: Adjusting osd_memory_target on smithi162 to 5696M 2024-03-31T23:07:03.060 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:07:02 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:07:03.060 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:07:02 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:07:03.060 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:07:02 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-31T23:07:03.060 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:07:02 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:07:03.060 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:07:02 smithi046 ceph-mon[32498]: pgmap v197: 1 pgs: 1 peering; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-03-31T23:07:03.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:07:02 smithi142 ceph-mon[35271]: purged_snaps scrub starts 2024-03-31T23:07:03.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:07:02 smithi142 ceph-mon[35271]: purged_snaps scrub ok 2024-03-31T23:07:03.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:07:02 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-31T23:07:03.194 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:07:02 smithi142 ceph-mon[35271]: osd.8 [v2:172.21.15.162:6816/1173078318,v1:172.21.15.162:6817/1173078318] boot 2024-03-31T23:07:03.194 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:07:02 smithi142 ceph-mon[35271]: osdmap e60: 9 total, 9 up, 9 in 2024-03-31T23:07:03.194 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:07:02 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-31T23:07:03.194 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:07:02 smithi142 ceph-mon[35271]: Detected new or changed devices on smithi162 2024-03-31T23:07:03.194 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:07:02 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:07:03.194 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:07:02 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:07:03.194 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:07:02 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-31T23:07:03.194 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:07:02 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-31T23:07:03.194 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:07:02 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-31T23:07:03.194 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:07:02 smithi142 ceph-mon[35271]: Adjusting osd_memory_target on smithi162 to 5696M 2024-03-31T23:07:03.194 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:07:02 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:07:03.194 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:07:02 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:07:03.194 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:07:02 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-31T23:07:03.195 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:07:02 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:07:03.195 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:07:02 smithi142 ceph-mon[35271]: pgmap v197: 1 pgs: 1 peering; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-03-31T23:07:04.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:07:03 smithi162 ceph-mon[36237]: osdmap e61: 9 total, 9 up, 9 in 2024-03-31T23:07:04.065 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:07:03 smithi046 ceph-mon[32498]: osdmap e61: 9 total, 9 up, 9 in 2024-03-31T23:07:04.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:07:03 smithi142 ceph-mon[35271]: osdmap e61: 9 total, 9 up, 9 in 2024-03-31T23:07:04.582 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:07:04.581+0000 7fbcfef27700 1 -- 172.21.15.46:0/3772229757 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbcf8108000 msgr2=0x7fbcf810a3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:07:04.582 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:07:04.581+0000 7fbcfef27700 1 --2- 172.21.15.46:0/3772229757 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbcf8108000 0x7fbcf810a3f0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fbcf000b0a0 tx=0x7fbcf000b3b0 comp rx=0 tx=0).stop 2024-03-31T23:07:04.583 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:07:04.581+0000 7fbcfef27700 1 -- 172.21.15.46:0/3772229757 shutdown_connections 2024-03-31T23:07:04.583 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:07:04.581+0000 7fbcfef27700 1 --2- 172.21.15.46:0/3772229757 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbcf8108000 0x7fbcf810a3f0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:07:04.583 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:07:04.581+0000 7fbcfef27700 1 --2- 172.21.15.46:0/3772229757 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbcf8102da0 0x7fbcf8105190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:07:04.583 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:07:04.581+0000 7fbcfef27700 1 --2- 172.21.15.46:0/3772229757 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbcf81056d0 0x7fbcf8107ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:07:04.583 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:07:04.581+0000 7fbcfef27700 1 -- 172.21.15.46:0/3772229757 >> 172.21.15.46:0/3772229757 conn(0x7fbcf80fc710 msgr2=0x7fbcf80feb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:07:04.583 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:07:04.581+0000 7fbcfef27700 1 -- 172.21.15.46:0/3772229757 shutdown_connections 2024-03-31T23:07:04.583 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:07:04.581+0000 7fbcfef27700 1 -- 172.21.15.46:0/3772229757 wait complete. 2024-03-31T23:07:04.585 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:07:04.582+0000 7fbcfef27700 1 Processor -- start 2024-03-31T23:07:04.585 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:07:04.582+0000 7fbcfef27700 1 -- start start 2024-03-31T23:07:04.585 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:07:04.582+0000 7fbcfef27700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbcf8102da0 0x7fbcf8198310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:07:04.585 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:07:04.582+0000 7fbcfef27700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbcf81056d0 0x7fbcf8198850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:07:04.585 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:07:04.582+0000 7fbcfef27700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbcf8108000 0x7fbcf819f8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:07:04.585 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:07:04.582+0000 7fbcfef27700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fbcf810dd00 con 0x7fbcf8102da0 2024-03-31T23:07:04.585 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:07:04.582+0000 7fbcfef27700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fbcf810db80 con 0x7fbcf81056d0 2024-03-31T23:07:04.586 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:07:04.582+0000 7fbcfef27700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fbcf810de80 con 0x7fbcf8108000 2024-03-31T23:07:04.586 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:07:04.582+0000 7fbcfccc3700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbcf8102da0 0x7fbcf8198310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:07:04.586 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:07:04.582+0000 7fbcfccc3700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbcf8102da0 0x7fbcf8198310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:35858/0 (socket says 172.21.15.46:35858) 2024-03-31T23:07:04.586 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:07:04.582+0000 7fbcfccc3700 1 -- 172.21.15.46:0/3575261631 learned_addr learned my addr 172.21.15.46:0/3575261631 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:07:04.586 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:07:04.582+0000 7fbcfd4c4700 1 --2- 172.21.15.46:0/3575261631 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbcf8108000 0x7fbcf819f8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:07:04.586 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:07:04.582+0000 7fbcfccc3700 1 -- 172.21.15.46:0/3575261631 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbcf8108000 msgr2=0x7fbcf819f8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:07:04.586 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:07:04.582+0000 7fbcf7fff700 1 --2- 172.21.15.46:0/3575261631 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbcf81056d0 0x7fbcf8198850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:07:04.586 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:07:04.582+0000 7fbcfccc3700 1 --2- 172.21.15.46:0/3575261631 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbcf8108000 0x7fbcf819f8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:07:04.586 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:07:04.582+0000 7fbcfccc3700 1 -- 172.21.15.46:0/3575261631 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbcf81056d0 msgr2=0x7fbcf8198850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:07:04.586 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:07:04.582+0000 7fbcfccc3700 1 --2- 172.21.15.46:0/3575261631 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbcf81056d0 0x7fbcf8198850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:07:04.586 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:07:04.582+0000 7fbcfccc3700 1 -- 172.21.15.46:0/3575261631 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbcf819ff70 con 0x7fbcf8102da0 2024-03-31T23:07:04.586 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:07:04.582+0000 7fbcfd4c4700 1 --2- 172.21.15.46:0/3575261631 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbcf8108000 0x7fbcf819f8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T23:07:04.586 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:07:04.582+0000 7fbcf7fff700 1 --2- 172.21.15.46:0/3575261631 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbcf81056d0 0x7fbcf8198850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T23:07:04.586 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:07:04.583+0000 7fbcfccc3700 1 --2- 172.21.15.46:0/3575261631 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbcf8102da0 0x7fbcf8198310 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fbce800c9e0 tx=0x7fbce800cda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:07:04.587 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:07:04.583+0000 7fbcf5ffb700 1 -- 172.21.15.46:0/3575261631 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbce8007ac0 con 0x7fbcf8102da0 2024-03-31T23:07:04.587 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:07:04.583+0000 7fbcf5ffb700 1 -- 172.21.15.46:0/3575261631 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fbce8007c60 con 0x7fbcf8102da0 2024-03-31T23:07:04.587 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:07:04.583+0000 7fbcfef27700 1 -- 172.21.15.46:0/3575261631 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbcf81a0260 con 0x7fbcf8102da0 2024-03-31T23:07:04.587 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:07:04.583+0000 7fbcf5ffb700 1 -- 172.21.15.46:0/3575261631 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbce8018660 con 0x7fbcf8102da0 2024-03-31T23:07:04.587 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:07:04.583+0000 7fbcfef27700 1 -- 172.21.15.46:0/3575261631 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbcf81a07f0 con 0x7fbcf8102da0 2024-03-31T23:07:04.587 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:07:04.584+0000 7fbcfef27700 1 -- 172.21.15.46:0/3575261631 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbcf8068f00 con 0x7fbcf8102da0 2024-03-31T23:07:04.588 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:07:04.585+0000 7fbcf5ffb700 1 -- 172.21.15.46:0/3575261631 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 16) ==== 137320+0+0 (secure 0 0 0) 0x7fbce800f450 con 0x7fbcf8102da0 2024-03-31T23:07:04.588 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:07:04.586+0000 7fbcf5ffb700 1 --2- 172.21.15.46:0/3575261631 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fbce00a25b0 0x7fbce00a4a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:07:04.588 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:07:04.586+0000 7fbcf7fff700 1 --2- 172.21.15.46:0/3575261631 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fbce00a25b0 0x7fbce00a4a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:07:04.588 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:07:04.586+0000 7fbcf5ffb700 1 -- 172.21.15.46:0/3575261631 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(62..62 src has 1..62) ==== 5858+0+0 (secure 0 0 0) 0x7fbce80cddd0 con 0x7fbcf8102da0 2024-03-31T23:07:04.588 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:07:04.586+0000 7fbcf7fff700 1 --2- 172.21.15.46:0/3575261631 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fbce00a25b0 0x7fbce00a4a70 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fbcec005fd0 tx=0x7fbcec005ee0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:07:04.592 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:07:04.591+0000 7fbcf5ffb700 1 -- 172.21.15.46:0/3575261631 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fbce8014030 con 0x7fbcf8102da0 2024-03-31T23:07:04.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:07:04.806+0000 7fbcfef27700 1 -- 172.21.15.46:0/3575261631 --> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi046:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7fbcf8062680 con 0x7fbce00a25b0 2024-03-31T23:07:05.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:07:04 smithi162 ceph-mon[36237]: osdmap e62: 9 total, 9 up, 9 in 2024-03-31T23:07:05.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:07:04 smithi162 ceph-mon[36237]: pgmap v200: 1 pgs: 1 peering; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail 2024-03-31T23:07:05.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:07:04 smithi046 ceph-mon[32498]: osdmap e62: 9 total, 9 up, 9 in 2024-03-31T23:07:05.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:07:04 smithi046 ceph-mon[32498]: pgmap v200: 1 pgs: 1 peering; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail 2024-03-31T23:07:05.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:07:04 smithi142 ceph-mon[35271]: osdmap e62: 9 total, 9 up, 9 in 2024-03-31T23:07:05.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:07:04 smithi142 ceph-mon[35271]: pgmap v200: 1 pgs: 1 peering; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail 2024-03-31T23:07:06.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:07:05 smithi162 ceph-mon[36237]: osdmap e63: 9 total, 9 up, 9 in 2024-03-31T23:07:06.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:07:05 smithi162 ceph-mon[36237]: from='client.14418 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi046:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-31T23:07:06.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:07:05 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-31T23:07:06.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:07:05 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-31T23:07:06.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:07:05 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:07:06.113 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:07:05 smithi046 ceph-mon[32498]: osdmap e63: 9 total, 9 up, 9 in 2024-03-31T23:07:06.114 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:07:05 smithi046 ceph-mon[32498]: from='client.14418 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi046:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-31T23:07:06.114 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:07:05 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-31T23:07:06.114 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:07:05 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-31T23:07:06.114 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:07:05 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:07:06.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:07:05 smithi142 ceph-mon[35271]: osdmap e63: 9 total, 9 up, 9 in 2024-03-31T23:07:06.227 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:07:05 smithi142 ceph-mon[35271]: from='client.14418 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi046:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-31T23:07:06.227 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:07:05 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-31T23:07:06.227 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:07:05 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-31T23:07:06.228 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:07:05 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:07:07.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:07:06 smithi162 ceph-mon[36237]: osdmap e64: 9 total, 9 up, 9 in 2024-03-31T23:07:07.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:07:06 smithi162 ceph-mon[36237]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-31T23:07:07.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:07:06 smithi046 ceph-mon[32498]: osdmap e64: 9 total, 9 up, 9 in 2024-03-31T23:07:07.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:07:06 smithi046 ceph-mon[32498]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-31T23:07:07.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:07:06 smithi142 ceph-mon[35271]: osdmap e64: 9 total, 9 up, 9 in 2024-03-31T23:07:07.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:07:06 smithi142 ceph-mon[35271]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-31T23:07:08.022 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:07:07 smithi046 ceph-mon[32498]: from='client.? 172.21.15.46:0/2220067767' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "faccce25-6bf2-4220-8858-cda82b72be68"}]: dispatch 2024-03-31T23:07:08.022 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:07:07 smithi046 ceph-mon[32498]: from='client.? 172.21.15.46:0/2220067767' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "faccce25-6bf2-4220-8858-cda82b72be68"}]': finished 2024-03-31T23:07:08.022 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:07:07 smithi046 ceph-mon[32498]: osdmap e65: 10 total, 9 up, 10 in 2024-03-31T23:07:08.022 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:07:07 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-31T23:07:08.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:07:07 smithi162 ceph-mon[36237]: from='client.? 172.21.15.46:0/2220067767' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "faccce25-6bf2-4220-8858-cda82b72be68"}]: dispatch 2024-03-31T23:07:08.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:07:07 smithi162 ceph-mon[36237]: from='client.? 172.21.15.46:0/2220067767' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "faccce25-6bf2-4220-8858-cda82b72be68"}]': finished 2024-03-31T23:07:08.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:07:07 smithi162 ceph-mon[36237]: osdmap e65: 10 total, 9 up, 10 in 2024-03-31T23:07:08.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:07:07 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-31T23:07:08.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:07:07 smithi142 ceph-mon[35271]: from='client.? 172.21.15.46:0/2220067767' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "faccce25-6bf2-4220-8858-cda82b72be68"}]: dispatch 2024-03-31T23:07:08.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:07:07 smithi142 ceph-mon[35271]: from='client.? 172.21.15.46:0/2220067767' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "faccce25-6bf2-4220-8858-cda82b72be68"}]': finished 2024-03-31T23:07:08.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:07:07 smithi142 ceph-mon[35271]: osdmap e65: 10 total, 9 up, 10 in 2024-03-31T23:07:08.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:07:07 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-31T23:07:09.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:07:08 smithi162 ceph-mon[36237]: from='client.? 172.21.15.46:0/2282405727' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-31T23:07:09.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:07:08 smithi162 ceph-mon[36237]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail; 120 KiB/s, 0 objects/s recovering 2024-03-31T23:07:09.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:07:08 smithi046 ceph-mon[32498]: from='client.? 172.21.15.46:0/2282405727' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-31T23:07:09.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:07:08 smithi046 ceph-mon[32498]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail; 120 KiB/s, 0 objects/s recovering 2024-03-31T23:07:09.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:07:08 smithi142 ceph-mon[35271]: from='client.? 172.21.15.46:0/2282405727' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-31T23:07:09.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:07:08 smithi142 ceph-mon[35271]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail; 120 KiB/s, 0 objects/s recovering 2024-03-31T23:07:10.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:07:10 smithi046 ceph-mon[32498]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-31T23:07:10.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:07:10 smithi142 ceph-mon[35271]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-31T23:07:11.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:07:10 smithi162 ceph-mon[36237]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-31T23:07:12.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:07:12 smithi046 ceph-mon[32498]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail; 74 KiB/s, 0 objects/s recovering 2024-03-31T23:07:12.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:07:12 smithi142 ceph-mon[35271]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail; 74 KiB/s, 0 objects/s recovering 2024-03-31T23:07:13.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:07:12 smithi162 ceph-mon[36237]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail; 74 KiB/s, 0 objects/s recovering 2024-03-31T23:07:14.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:07:14 smithi046 ceph-mon[32498]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail; 66 KiB/s, 0 objects/s recovering 2024-03-31T23:07:14.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:07:14 smithi142 ceph-mon[35271]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail; 66 KiB/s, 0 objects/s recovering 2024-03-31T23:07:15.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:07:14 smithi162 ceph-mon[36237]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail; 66 KiB/s, 0 objects/s recovering 2024-03-31T23:07:15.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:07:15 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-03-31T23:07:15.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:07:15 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:07:15.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:07:15 smithi046 ceph-mon[32498]: Deploying daemon osd.9 on smithi046 2024-03-31T23:07:15.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:07:15 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-03-31T23:07:15.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:07:15 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:07:15.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:07:15 smithi142 ceph-mon[35271]: Deploying daemon osd.9 on smithi046 2024-03-31T23:07:16.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:07:15 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-03-31T23:07:16.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:07:15 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:07:16.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:07:15 smithi162 ceph-mon[36237]: Deploying daemon osd.9 on smithi046 2024-03-31T23:07:16.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:07:16 smithi142 ceph-mon[35271]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail 2024-03-31T23:07:16.978 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:07:16 smithi046 ceph-mon[32498]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail 2024-03-31T23:07:17.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:07:16 smithi162 ceph-mon[36237]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail 2024-03-31T23:07:18.804 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:07:18 smithi046 ceph-mon[32498]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail 2024-03-31T23:07:18.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:07:18 smithi142 ceph-mon[35271]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail 2024-03-31T23:07:19.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:07:18 smithi162 ceph-mon[36237]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail 2024-03-31T23:07:20.861 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:07:20 smithi162 ceph-mon[36237]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail 2024-03-31T23:07:20.862 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:07:20 smithi046 ceph-mon[32498]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail 2024-03-31T23:07:20.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:07:20 smithi142 ceph-mon[35271]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail 2024-03-31T23:07:22.546 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:07:22 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:07:22.546 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:07:22 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:07:22.546 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:07:22 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-31T23:07:22.612 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:07:22 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:07:22.612 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:07:22 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:07:22.612 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:07:22 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-31T23:07:22.693 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:07:22 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:07:22.693 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:07:22 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:07:22.693 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:07:22 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-31T23:07:23.361 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:07:23 smithi046 ceph-mon[32498]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail 2024-03-31T23:07:23.693 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:07:23 smithi142 ceph-mon[35271]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail 2024-03-31T23:07:23.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:07:23 smithi162 ceph-mon[36237]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail 2024-03-31T23:07:25.523 INFO:teuthology.orchestra.run.smithi046.stdout:Created osd(s) 9 on host 'smithi046' 2024-03-31T23:07:25.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:07:25.521+0000 7fbcf5ffb700 1 -- 172.21.15.46:0/3575261631 <== mgr.14152 v2:172.21.15.46:6800/162289569 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fbcf8062680 con 0x7fbce00a25b0 2024-03-31T23:07:25.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:07:25.524+0000 7fbcfef27700 1 -- 172.21.15.46:0/3575261631 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fbce00a25b0 msgr2=0x7fbce00a4a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:07:25.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:07:25.524+0000 7fbcfef27700 1 --2- 172.21.15.46:0/3575261631 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fbce00a25b0 0x7fbce00a4a70 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fbcec005fd0 tx=0x7fbcec005ee0 comp rx=0 tx=0).stop 2024-03-31T23:07:25.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:07:25.524+0000 7fbcfef27700 1 -- 172.21.15.46:0/3575261631 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbcf8102da0 msgr2=0x7fbcf8198310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:07:25.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:07:25.524+0000 7fbcfef27700 1 --2- 172.21.15.46:0/3575261631 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbcf8102da0 0x7fbcf8198310 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fbce800c9e0 tx=0x7fbce800cda0 comp rx=0 tx=0).stop 2024-03-31T23:07:25.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:07:25.524+0000 7fbcfef27700 1 -- 172.21.15.46:0/3575261631 shutdown_connections 2024-03-31T23:07:25.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:07:25.524+0000 7fbcfef27700 1 --2- 172.21.15.46:0/3575261631 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fbce00a25b0 0x7fbce00a4a70 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:07:25.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:07:25.524+0000 7fbcfef27700 1 --2- 172.21.15.46:0/3575261631 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbcf8102da0 0x7fbcf8198310 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:07:25.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:07:25.524+0000 7fbcfef27700 1 --2- 172.21.15.46:0/3575261631 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbcf8108000 0x7fbcf819f8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:07:25.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:07:25.524+0000 7fbcfef27700 1 --2- 172.21.15.46:0/3575261631 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbcf81056d0 0x7fbcf8198850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:07:25.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:07:25.524+0000 7fbcfef27700 1 -- 172.21.15.46:0/3575261631 >> 172.21.15.46:0/3575261631 conn(0x7fbcf80fc710 msgr2=0x7fbcf81037c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:07:25.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:07:25.524+0000 7fbcfef27700 1 -- 172.21.15.46:0/3575261631 shutdown_connections 2024-03-31T23:07:25.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:07:25.524+0000 7fbcfef27700 1 -- 172.21.15.46:0/3575261631 wait complete. 2024-03-31T23:07:25.839 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:07:25 smithi046 ceph-mon[32498]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail 2024-03-31T23:07:25.839 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:07:25 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:07:25.839 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:07:25 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:07:25.839 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:07:25 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:07:25.839 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:07:25 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-31T23:07:25.839 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:07:25 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:07:25.839 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:07:25 smithi046 ceph-mon[32498]: from='osd.9 [v2:172.21.15.46:6826/532708881,v1:172.21.15.46:6827/532708881]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-03-31T23:07:25.840 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:07:25 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:07:25.840 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:07:25 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:07:25.840 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:07:25 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-31T23:07:25.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:07:25 smithi142 ceph-mon[35271]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail 2024-03-31T23:07:25.944 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:07:25 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:07:25.944 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:07:25 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:07:25.944 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:07:25 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:07:25.944 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:07:25 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-31T23:07:25.944 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:07:25 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:07:25.944 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:07:25 smithi142 ceph-mon[35271]: from='osd.9 [v2:172.21.15.46:6826/532708881,v1:172.21.15.46:6827/532708881]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-03-31T23:07:25.944 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:07:25 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:07:25.944 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:07:25 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:07:25.944 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:07:25 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-31T23:07:26.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:07:25 smithi162 ceph-mon[36237]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail 2024-03-31T23:07:26.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:07:25 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:07:26.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:07:25 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:07:26.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:07:25 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:07:26.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:07:25 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-31T23:07:26.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:07:25 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:07:26.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:07:25 smithi162 ceph-mon[36237]: from='osd.9 [v2:172.21.15.46:6826/532708881,v1:172.21.15.46:6827/532708881]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-03-31T23:07:26.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:07:25 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:07:26.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:07:25 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:07:26.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:07:25 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-31T23:07:26.171 DEBUG:teuthology.orchestra.run.smithi046:osd.9> sudo journalctl -f -n 0 -u ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4@osd.9.service 2024-03-31T23:07:26.173 INFO:tasks.cephadm:Deploying osd.10 on smithi142 with /dev/vg_nvme/lv_1... 2024-03-31T23:07:26.173 DEBUG:teuthology.orchestra.run.smithi142:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:384f31f3d3aee7f8ba3f60846b8844c049165583 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6a9d8f08-efb2-11ee-b647-cb9ed24678a4 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-03-31T23:07:26.419 INFO:teuthology.orchestra.run.smithi142.stderr:Inferring config /var/lib/ceph/6a9d8f08-efb2-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-31T23:07:26.862 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:07:26 smithi046 ceph-mon[32498]: from='osd.9 [v2:172.21.15.46:6826/532708881,v1:172.21.15.46:6827/532708881]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-03-31T23:07:26.862 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:07:26 smithi046 ceph-mon[32498]: osdmap e66: 10 total, 9 up, 10 in 2024-03-31T23:07:26.862 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:07:26 smithi046 ceph-mon[32498]: from='osd.9 [v2:172.21.15.46:6826/532708881,v1:172.21.15.46:6827/532708881]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi046", "root=default"]}]: dispatch 2024-03-31T23:07:26.862 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:07:26 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-31T23:07:26.862 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:07:26 smithi046 ceph-mon[32498]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail 2024-03-31T23:07:26.863 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 31 23:07:26 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-9[67499]: 2024-03-31T23:07:26.614+0000 7fcc0a01f700 -1 osd.9 0 waiting for initial osdmap 2024-03-31T23:07:26.863 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 31 23:07:26 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-9[67499]: 2024-03-31T23:07:26.625+0000 7fcc05e12700 -1 osd.9 67 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-31T23:07:26.896 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:07:26 smithi142 ceph-mon[35271]: from='osd.9 [v2:172.21.15.46:6826/532708881,v1:172.21.15.46:6827/532708881]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-03-31T23:07:26.896 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:07:26 smithi142 ceph-mon[35271]: osdmap e66: 10 total, 9 up, 10 in 2024-03-31T23:07:26.896 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:07:26 smithi142 ceph-mon[35271]: from='osd.9 [v2:172.21.15.46:6826/532708881,v1:172.21.15.46:6827/532708881]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi046", "root=default"]}]: dispatch 2024-03-31T23:07:26.896 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:07:26 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-31T23:07:26.896 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:07:26 smithi142 ceph-mon[35271]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail 2024-03-31T23:07:27.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:07:26 smithi162 ceph-mon[36237]: from='osd.9 [v2:172.21.15.46:6826/532708881,v1:172.21.15.46:6827/532708881]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-03-31T23:07:27.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:07:26 smithi162 ceph-mon[36237]: osdmap e66: 10 total, 9 up, 10 in 2024-03-31T23:07:27.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:07:26 smithi162 ceph-mon[36237]: from='osd.9 [v2:172.21.15.46:6826/532708881,v1:172.21.15.46:6827/532708881]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi046", "root=default"]}]: dispatch 2024-03-31T23:07:27.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:07:26 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-31T23:07:27.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:07:26 smithi162 ceph-mon[36237]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail 2024-03-31T23:07:27.859 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:07:27 smithi142 ceph-mon[35271]: from='osd.9 [v2:172.21.15.46:6826/532708881,v1:172.21.15.46:6827/532708881]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi046", "root=default"]}]': finished 2024-03-31T23:07:27.859 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:07:27 smithi142 ceph-mon[35271]: osdmap e67: 10 total, 9 up, 10 in 2024-03-31T23:07:27.859 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:07:27 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-31T23:07:27.859 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:07:27 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-31T23:07:27.859 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:07:27 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-31T23:07:28.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:07:27 smithi162 ceph-mon[36237]: from='osd.9 [v2:172.21.15.46:6826/532708881,v1:172.21.15.46:6827/532708881]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi046", "root=default"]}]': finished 2024-03-31T23:07:28.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:07:27 smithi162 ceph-mon[36237]: osdmap e67: 10 total, 9 up, 10 in 2024-03-31T23:07:28.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:07:27 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-31T23:07:28.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:07:27 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-31T23:07:28.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:07:27 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-31T23:07:28.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:07:27 smithi046 ceph-mon[32498]: from='osd.9 [v2:172.21.15.46:6826/532708881,v1:172.21.15.46:6827/532708881]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi046", "root=default"]}]': finished 2024-03-31T23:07:28.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:07:27 smithi046 ceph-mon[32498]: osdmap e67: 10 total, 9 up, 10 in 2024-03-31T23:07:28.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:07:27 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-31T23:07:28.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:07:27 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-31T23:07:28.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:07:27 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-31T23:07:28.453 INFO:teuthology.orchestra.run.smithi142.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-03-31T23:07:28.453 INFO:teuthology.orchestra.run.smithi142.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-03-31T23:07:28.453 INFO:teuthology.orchestra.run.smithi142.stderr: stderr: 10+0 records in 2024-03-31T23:07:28.453 INFO:teuthology.orchestra.run.smithi142.stderr:10+0 records out 2024-03-31T23:07:28.453 INFO:teuthology.orchestra.run.smithi142.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0156398 s, 670 MB/s 2024-03-31T23:07:28.453 INFO:teuthology.orchestra.run.smithi142.stderr:--> Zapping successful for: 2024-03-31T23:07:28.806 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:07:28 smithi046 ceph-mon[32498]: purged_snaps scrub starts 2024-03-31T23:07:28.807 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:07:28 smithi046 ceph-mon[32498]: purged_snaps scrub ok 2024-03-31T23:07:28.807 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:07:28 smithi046 ceph-mon[32498]: osd.9 [v2:172.21.15.46:6826/532708881,v1:172.21.15.46:6827/532708881] boot 2024-03-31T23:07:28.807 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:07:28 smithi046 ceph-mon[32498]: osdmap e68: 10 total, 10 up, 10 in 2024-03-31T23:07:28.807 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:07:28 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-31T23:07:28.807 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:07:28 smithi046 ceph-mon[32498]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail 2024-03-31T23:07:28.832 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:07:28 smithi142 ceph-mon[35271]: purged_snaps scrub starts 2024-03-31T23:07:28.832 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:07:28 smithi142 ceph-mon[35271]: purged_snaps scrub ok 2024-03-31T23:07:28.832 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:07:28 smithi142 ceph-mon[35271]: osd.9 [v2:172.21.15.46:6826/532708881,v1:172.21.15.46:6827/532708881] boot 2024-03-31T23:07:28.832 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:07:28 smithi142 ceph-mon[35271]: osdmap e68: 10 total, 10 up, 10 in 2024-03-31T23:07:28.832 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:07:28 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-31T23:07:28.832 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:07:28 smithi142 ceph-mon[35271]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail 2024-03-31T23:07:29.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:07:28 smithi162 ceph-mon[36237]: purged_snaps scrub starts 2024-03-31T23:07:29.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:07:28 smithi162 ceph-mon[36237]: purged_snaps scrub ok 2024-03-31T23:07:29.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:07:28 smithi162 ceph-mon[36237]: osd.9 [v2:172.21.15.46:6826/532708881,v1:172.21.15.46:6827/532708881] boot 2024-03-31T23:07:29.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:07:28 smithi162 ceph-mon[36237]: osdmap e68: 10 total, 10 up, 10 in 2024-03-31T23:07:29.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:07:28 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-31T23:07:29.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:07:28 smithi162 ceph-mon[36237]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail 2024-03-31T23:07:29.183 DEBUG:teuthology.orchestra.run.smithi142:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:384f31f3d3aee7f8ba3f60846b8844c049165583 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6a9d8f08-efb2-11ee-b647-cb9ed24678a4 -- ceph orch daemon add osd smithi142:vg_nvme/lv_1 2024-03-31T23:07:29.430 INFO:teuthology.orchestra.run.smithi142.stderr:Inferring config /var/lib/ceph/6a9d8f08-efb2-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-31T23:07:29.810 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:07:29 smithi142 ceph-mon[35271]: osdmap e69: 10 total, 10 up, 10 in 2024-03-31T23:07:29.810 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:07:29 smithi142 ceph-mon[35271]: Detected new or changed devices on smithi046 2024-03-31T23:07:29.810 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:07:29 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:07:29.810 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:07:29 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:07:29.811 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:07:29 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-31T23:07:29.811 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:07:29 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-31T23:07:29.811 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:07:29 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-31T23:07:29.811 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:07:29 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-31T23:07:29.811 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:07:29 smithi142 ceph-mon[35271]: Adjusting osd_memory_target on smithi046 to 4273M 2024-03-31T23:07:29.811 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:07:29 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:07:29.811 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:07:29 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:07:29.811 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:07:29 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-31T23:07:29.811 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:07:29 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:07:30.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:07:29 smithi162 ceph-mon[36237]: osdmap e69: 10 total, 10 up, 10 in 2024-03-31T23:07:30.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:07:29 smithi162 ceph-mon[36237]: Detected new or changed devices on smithi046 2024-03-31T23:07:30.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:07:29 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:07:30.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:07:29 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:07:30.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:07:29 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-31T23:07:30.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:07:29 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-31T23:07:30.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:07:29 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-31T23:07:30.047 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:07:29 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-31T23:07:30.047 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:07:29 smithi162 ceph-mon[36237]: Adjusting osd_memory_target on smithi046 to 4273M 2024-03-31T23:07:30.047 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:07:29 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:07:30.047 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:07:29 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:07:30.047 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:07:29 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-31T23:07:30.047 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:07:29 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:07:30.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:07:29 smithi046 ceph-mon[32498]: osdmap e69: 10 total, 10 up, 10 in 2024-03-31T23:07:30.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:07:29 smithi046 ceph-mon[32498]: Detected new or changed devices on smithi046 2024-03-31T23:07:30.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:07:29 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:07:30.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:07:29 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:07:30.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:07:29 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-31T23:07:30.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:07:29 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-31T23:07:30.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:07:29 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-31T23:07:30.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:07:29 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-31T23:07:30.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:07:29 smithi046 ceph-mon[32498]: Adjusting osd_memory_target on smithi046 to 4273M 2024-03-31T23:07:30.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:07:29 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:07:30.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:07:29 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:07:30.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:07:29 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-31T23:07:30.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:07:29 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:07:30.893 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:07:30 smithi142 ceph-mon[35271]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 288 MiB used, 894 GiB / 894 GiB avail 2024-03-31T23:07:31.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:07:30 smithi162 ceph-mon[36237]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 288 MiB used, 894 GiB / 894 GiB avail 2024-03-31T23:07:31.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:07:30 smithi046 ceph-mon[32498]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 288 MiB used, 894 GiB / 894 GiB avail 2024-03-31T23:07:31.141 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-31T23:07:31.140+0000 7f14d9616700 1 -- 172.21.15.142:0/3014097826 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f14d4103490 msgr2=0x7f14d41038f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:07:31.141 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-31T23:07:31.140+0000 7f14d9616700 1 --2- 172.21.15.142:0/3014097826 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f14d4103490 0x7f14d41038f0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f14bc0099c0 tx=0x7f14bc009cd0 comp rx=0 tx=0).stop 2024-03-31T23:07:31.141 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-31T23:07:31.140+0000 7f14d9616700 1 -- 172.21.15.142:0/3014097826 shutdown_connections 2024-03-31T23:07:31.141 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-31T23:07:31.140+0000 7f14d9616700 1 --2- 172.21.15.142:0/3014097826 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f14d4103e30 0x7f14d410a6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:07:31.141 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-31T23:07:31.140+0000 7f14d9616700 1 --2- 172.21.15.142:0/3014097826 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f14d4102290 0x7f14d41026b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:07:31.141 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-31T23:07:31.140+0000 7f14d9616700 1 --2- 172.21.15.142:0/3014097826 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f14d4103490 0x7f14d41038f0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:07:31.141 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-31T23:07:31.140+0000 7f14d9616700 1 -- 172.21.15.142:0/3014097826 >> 172.21.15.142:0/3014097826 conn(0x7f14d40fd7f0 msgr2=0x7f14d40ffc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:07:31.141 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-31T23:07:31.140+0000 7f14d9616700 1 -- 172.21.15.142:0/3014097826 shutdown_connections 2024-03-31T23:07:31.141 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-31T23:07:31.140+0000 7f14d9616700 1 -- 172.21.15.142:0/3014097826 wait complete. 2024-03-31T23:07:31.143 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-31T23:07:31.140+0000 7f14d9616700 1 Processor -- start 2024-03-31T23:07:31.143 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-31T23:07:31.141+0000 7f14d9616700 1 -- start start 2024-03-31T23:07:31.144 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-31T23:07:31.141+0000 7f14d9616700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f14d4102290 0x7f14d4198400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:07:31.144 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-31T23:07:31.141+0000 7f14d9616700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f14d4103490 0x7f14d4198940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:07:31.144 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-31T23:07:31.141+0000 7f14d9616700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f14d4103e30 0x7f14d419f9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:07:31.144 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-31T23:07:31.141+0000 7f14d9616700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f14d410dd80 con 0x7f14d4103490 2024-03-31T23:07:31.144 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-31T23:07:31.141+0000 7f14d9616700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f14d410dc00 con 0x7f14d4103e30 2024-03-31T23:07:31.144 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-31T23:07:31.141+0000 7f14d9616700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f14d410df00 con 0x7f14d4102290 2024-03-31T23:07:31.144 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-31T23:07:31.141+0000 7f14d37fe700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f14d4103e30 0x7f14d419f9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:07:31.144 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-31T23:07:31.141+0000 7f14d37fe700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f14d4103e30 0x7f14d419f9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:56670/0 (socket says 172.21.15.142:56670) 2024-03-31T23:07:31.144 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-31T23:07:31.141+0000 7f14d37fe700 1 -- 172.21.15.142:0/1401967410 learned_addr learned my addr 172.21.15.142:0/1401967410 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-03-31T23:07:31.144 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-31T23:07:31.141+0000 7f14d2ffd700 1 --2- 172.21.15.142:0/1401967410 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f14d4102290 0x7f14d4198400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:07:31.144 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-31T23:07:31.141+0000 7f14ca7fc700 1 --2- 172.21.15.142:0/1401967410 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f14d4103490 0x7f14d4198940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:07:31.144 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-31T23:07:31.141+0000 7f14d37fe700 1 -- 172.21.15.142:0/1401967410 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f14d4102290 msgr2=0x7f14d4198400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:07:31.144 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-31T23:07:31.141+0000 7f14d37fe700 1 --2- 172.21.15.142:0/1401967410 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f14d4102290 0x7f14d4198400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:07:31.145 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-31T23:07:31.141+0000 7f14d37fe700 1 -- 172.21.15.142:0/1401967410 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f14d4103490 msgr2=0x7f14d4198940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:07:31.145 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-31T23:07:31.141+0000 7f14d37fe700 1 --2- 172.21.15.142:0/1401967410 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f14d4103490 0x7f14d4198940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:07:31.145 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-31T23:07:31.141+0000 7f14d37fe700 1 -- 172.21.15.142:0/1401967410 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f14d41a0060 con 0x7f14d4103e30 2024-03-31T23:07:31.145 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-31T23:07:31.141+0000 7f14d37fe700 1 --2- 172.21.15.142:0/1401967410 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f14d4103e30 0x7f14d419f9c0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f14cc00d980 tx=0x7f14cc00dc90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:07:31.145 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-31T23:07:31.142+0000 7f14d0ff9700 1 -- 172.21.15.142:0/1401967410 <== mon.2 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14cc017070 con 0x7f14d4103e30 2024-03-31T23:07:31.145 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-31T23:07:31.142+0000 7f14d0ff9700 1 -- 172.21.15.142:0/1401967410 <== mon.2 v2:172.21.15.142:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f14cc005020 con 0x7f14d4103e30 2024-03-31T23:07:31.145 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-31T23:07:31.142+0000 7f14d9616700 1 -- 172.21.15.142:0/1401967410 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f14d41a0350 con 0x7f14d4103e30 2024-03-31T23:07:31.145 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-31T23:07:31.142+0000 7f14d0ff9700 1 -- 172.21.15.142:0/1401967410 <== mon.2 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14cc00fa90 con 0x7f14d4103e30 2024-03-31T23:07:31.145 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-31T23:07:31.142+0000 7f14d9616700 1 -- 172.21.15.142:0/1401967410 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f14d41a0890 con 0x7f14d4103e30 2024-03-31T23:07:31.145 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-31T23:07:31.143+0000 7f14d9616700 1 -- 172.21.15.142:0/1401967410 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f14d4068f00 con 0x7f14d4103e30 2024-03-31T23:07:31.146 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-31T23:07:31.144+0000 7f14d0ff9700 1 -- 172.21.15.142:0/1401967410 <== mon.2 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 16) ==== 137320+0+0 (secure 0 0 0) 0x7f14cc0051d0 con 0x7f14d4103e30 2024-03-31T23:07:31.146 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-31T23:07:31.145+0000 7f14d0ff9700 1 --2- 172.21.15.142:0/1401967410 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f14c00a25b0 0x7f14c00a4a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:07:31.146 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-31T23:07:31.145+0000 7f14d0ff9700 1 -- 172.21.15.142:0/1401967410 <== mon.2 v2:172.21.15.142:3300/0 5 ==== osd_map(69..69 src has 1..69) ==== 6261+0+0 (secure 0 0 0) 0x7f14cc0ce080 con 0x7f14d4103e30 2024-03-31T23:07:31.146 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-31T23:07:31.145+0000 7f14d2ffd700 1 --2- 172.21.15.142:0/1401967410 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f14c00a25b0 0x7f14c00a4a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:07:31.147 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-31T23:07:31.146+0000 7f14d2ffd700 1 --2- 172.21.15.142:0/1401967410 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f14c00a25b0 0x7f14c00a4a70 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f14d41032f0 tx=0x7f14c4009180 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:07:31.149 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-31T23:07:31.149+0000 7f14d0ff9700 1 -- 172.21.15.142:0/1401967410 <== mon.2 v2:172.21.15.142:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f14cc020020 con 0x7f14d4103e30 2024-03-31T23:07:31.351 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-31T23:07:31.350+0000 7f14d9616700 1 -- 172.21.15.142:0/1401967410 --> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi142:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7f14d4062740 con 0x7f14c00a25b0 2024-03-31T23:07:31.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:07:31 smithi142 ceph-mon[35271]: from='client.24335 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi142:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-31T23:07:31.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:07:31 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-31T23:07:31.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:07:31 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-31T23:07:31.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:07:31 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:07:32.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:07:31 smithi162 ceph-mon[36237]: from='client.24335 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi142:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-31T23:07:32.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:07:31 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-31T23:07:32.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:07:31 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-31T23:07:32.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:07:31 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:07:32.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:07:31 smithi046 ceph-mon[32498]: from='client.24335 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi142:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-31T23:07:32.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:07:31 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-31T23:07:32.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:07:31 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-31T23:07:32.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:07:31 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:07:32.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:07:32 smithi142 ceph-mon[35271]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 289 MiB used, 894 GiB / 894 GiB avail 2024-03-31T23:07:33.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:07:32 smithi162 ceph-mon[36237]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 289 MiB used, 894 GiB / 894 GiB avail 2024-03-31T23:07:33.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:07:32 smithi046 ceph-mon[32498]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 289 MiB used, 894 GiB / 894 GiB avail 2024-03-31T23:07:34.546 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:07:34 smithi162 ceph-mon[36237]: from='client.? 172.21.15.142:0/2896332982' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cdd356db-f9eb-4080-95ab-db5d8223e572"}]: dispatch 2024-03-31T23:07:34.546 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:07:34 smithi162 ceph-mon[36237]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cdd356db-f9eb-4080-95ab-db5d8223e572"}]: dispatch 2024-03-31T23:07:34.546 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:07:34 smithi162 ceph-mon[36237]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "cdd356db-f9eb-4080-95ab-db5d8223e572"}]': finished 2024-03-31T23:07:34.546 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:07:34 smithi162 ceph-mon[36237]: osdmap e70: 11 total, 10 up, 11 in 2024-03-31T23:07:34.546 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:07:34 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-31T23:07:34.553 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:07:34 smithi142 ceph-mon[35271]: from='client.? 172.21.15.142:0/2896332982' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cdd356db-f9eb-4080-95ab-db5d8223e572"}]: dispatch 2024-03-31T23:07:34.553 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:07:34 smithi142 ceph-mon[35271]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cdd356db-f9eb-4080-95ab-db5d8223e572"}]: dispatch 2024-03-31T23:07:34.554 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:07:34 smithi142 ceph-mon[35271]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "cdd356db-f9eb-4080-95ab-db5d8223e572"}]': finished 2024-03-31T23:07:34.554 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:07:34 smithi142 ceph-mon[35271]: osdmap e70: 11 total, 10 up, 11 in 2024-03-31T23:07:34.554 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:07:34 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-31T23:07:34.611 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:07:34 smithi046 ceph-mon[32498]: from='client.? 172.21.15.142:0/2896332982' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cdd356db-f9eb-4080-95ab-db5d8223e572"}]: dispatch 2024-03-31T23:07:34.612 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:07:34 smithi046 ceph-mon[32498]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cdd356db-f9eb-4080-95ab-db5d8223e572"}]: dispatch 2024-03-31T23:07:34.612 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:07:34 smithi046 ceph-mon[32498]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "cdd356db-f9eb-4080-95ab-db5d8223e572"}]': finished 2024-03-31T23:07:34.612 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:07:34 smithi046 ceph-mon[32498]: osdmap e70: 11 total, 10 up, 11 in 2024-03-31T23:07:34.612 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:07:34 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-31T23:07:35.546 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:07:35 smithi162 ceph-mon[36237]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 289 MiB used, 894 GiB / 894 GiB avail 2024-03-31T23:07:35.546 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:07:35 smithi162 ceph-mon[36237]: from='client.? 172.21.15.142:0/3781415887' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-31T23:07:35.611 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:07:35 smithi046 ceph-mon[32498]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 289 MiB used, 894 GiB / 894 GiB avail 2024-03-31T23:07:35.611 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:07:35 smithi046 ceph-mon[32498]: from='client.? 172.21.15.142:0/3781415887' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-31T23:07:35.693 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:07:35 smithi142 ceph-mon[35271]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 289 MiB used, 894 GiB / 894 GiB avail 2024-03-31T23:07:35.693 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:07:35 smithi142 ceph-mon[35271]: from='client.? 172.21.15.142:0/3781415887' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-31T23:07:36.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:07:36 smithi142 ceph-mon[35271]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 289 MiB used, 894 GiB / 894 GiB avail 2024-03-31T23:07:37.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:07:36 smithi162 ceph-mon[36237]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 289 MiB used, 894 GiB / 894 GiB avail 2024-03-31T23:07:37.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:07:36 smithi046 ceph-mon[32498]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 289 MiB used, 894 GiB / 894 GiB avail 2024-03-31T23:07:38.914 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:07:38 smithi046 ceph-mon[32498]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 289 MiB used, 894 GiB / 894 GiB avail 2024-03-31T23:07:38.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:07:38 smithi142 ceph-mon[35271]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 289 MiB used, 894 GiB / 894 GiB avail 2024-03-31T23:07:39.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:07:38 smithi162 ceph-mon[36237]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 289 MiB used, 894 GiB / 894 GiB avail 2024-03-31T23:07:40.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:07:40 smithi142 ceph-mon[35271]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 289 MiB used, 894 GiB / 894 GiB avail 2024-03-31T23:07:41.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:07:40 smithi162 ceph-mon[36237]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 289 MiB used, 894 GiB / 894 GiB avail 2024-03-31T23:07:41.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:07:40 smithi046 ceph-mon[32498]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 289 MiB used, 894 GiB / 894 GiB avail 2024-03-31T23:07:41.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:07:41 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-03-31T23:07:41.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:07:41 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:07:42.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:07:41 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-03-31T23:07:42.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:07:41 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:07:42.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:07:41 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-03-31T23:07:42.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:07:41 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:07:42.946 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:07:42 smithi142 ceph-mon[35271]: Deploying daemon osd.10 on smithi142 2024-03-31T23:07:42.946 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:07:42 smithi142 ceph-mon[35271]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 289 MiB used, 894 GiB / 894 GiB avail 2024-03-31T23:07:43.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:07:42 smithi162 ceph-mon[36237]: Deploying daemon osd.10 on smithi142 2024-03-31T23:07:43.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:07:42 smithi162 ceph-mon[36237]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 289 MiB used, 894 GiB / 894 GiB avail 2024-03-31T23:07:43.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:07:42 smithi046 ceph-mon[32498]: Deploying daemon osd.10 on smithi142 2024-03-31T23:07:43.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:07:42 smithi046 ceph-mon[32498]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 289 MiB used, 894 GiB / 894 GiB avail 2024-03-31T23:07:44.722 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:07:44 smithi142 ceph-mon[35271]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 289 MiB used, 894 GiB / 894 GiB avail 2024-03-31T23:07:45.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:07:44 smithi162 ceph-mon[36237]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 289 MiB used, 894 GiB / 894 GiB avail 2024-03-31T23:07:45.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:07:44 smithi046 ceph-mon[32498]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 289 MiB used, 894 GiB / 894 GiB avail 2024-03-31T23:07:48.645 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:07:48 smithi142 ceph-mon[35271]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 289 MiB used, 894 GiB / 894 GiB avail 2024-03-31T23:07:48.645 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:07:48 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:07:48.645 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:07:48 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:07:48.645 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:07:48 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-31T23:07:48.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:07:48 smithi162 ceph-mon[36237]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 289 MiB used, 894 GiB / 894 GiB avail 2024-03-31T23:07:48.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:07:48 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:07:48.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:07:48 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:07:48.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:07:48 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-31T23:07:48.862 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:07:48 smithi046 ceph-mon[32498]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 289 MiB used, 894 GiB / 894 GiB avail 2024-03-31T23:07:48.862 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:07:48 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:07:48.862 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:07:48 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:07:48.862 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:07:48 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-31T23:07:50.611 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:07:50 smithi046 ceph-mon[32498]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 289 MiB used, 894 GiB / 894 GiB avail 2024-03-31T23:07:50.693 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:07:50 smithi142 ceph-mon[35271]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 289 MiB used, 894 GiB / 894 GiB avail 2024-03-31T23:07:50.795 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:07:50 smithi162 ceph-mon[36237]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 289 MiB used, 894 GiB / 894 GiB avail 2024-03-31T23:07:50.918 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-31T23:07:50.915+0000 7f14d0ff9700 1 -- 172.21.15.142:0/1401967410 <== mgr.14152 v2:172.21.15.46:6800/162289569 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+38 (secure 0 0 0) 0x7f14d4062740 con 0x7f14c00a25b0 2024-03-31T23:07:50.918 INFO:teuthology.orchestra.run.smithi142.stdout:Created osd(s) 10 on host 'smithi142' 2024-03-31T23:07:50.919 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-31T23:07:50.918+0000 7f14d9616700 1 -- 172.21.15.142:0/1401967410 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f14c00a25b0 msgr2=0x7f14c00a4a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:07:50.919 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-31T23:07:50.918+0000 7f14d9616700 1 --2- 172.21.15.142:0/1401967410 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f14c00a25b0 0x7f14c00a4a70 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f14d41032f0 tx=0x7f14c4009180 comp rx=0 tx=0).stop 2024-03-31T23:07:50.919 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-31T23:07:50.918+0000 7f14d9616700 1 -- 172.21.15.142:0/1401967410 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f14d4103e30 msgr2=0x7f14d419f9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:07:50.919 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-31T23:07:50.918+0000 7f14d9616700 1 --2- 172.21.15.142:0/1401967410 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f14d4103e30 0x7f14d419f9c0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f14cc00d980 tx=0x7f14cc00dc90 comp rx=0 tx=0).stop 2024-03-31T23:07:50.919 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-31T23:07:50.918+0000 7f14d9616700 1 -- 172.21.15.142:0/1401967410 shutdown_connections 2024-03-31T23:07:50.919 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-31T23:07:50.918+0000 7f14d9616700 1 --2- 172.21.15.142:0/1401967410 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f14c00a25b0 0x7f14c00a4a70 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:07:50.919 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-31T23:07:50.918+0000 7f14d9616700 1 --2- 172.21.15.142:0/1401967410 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f14d4103e30 0x7f14d419f9c0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:07:50.919 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-31T23:07:50.918+0000 7f14d9616700 1 --2- 172.21.15.142:0/1401967410 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f14d4102290 0x7f14d4198400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:07:50.919 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-31T23:07:50.918+0000 7f14d9616700 1 --2- 172.21.15.142:0/1401967410 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f14d4103490 0x7f14d4198940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:07:50.920 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-31T23:07:50.918+0000 7f14d9616700 1 -- 172.21.15.142:0/1401967410 >> 172.21.15.142:0/1401967410 conn(0x7f14d40fd7f0 msgr2=0x7f14d4106810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:07:50.920 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-31T23:07:50.918+0000 7f14d9616700 1 -- 172.21.15.142:0/1401967410 shutdown_connections 2024-03-31T23:07:50.920 INFO:teuthology.orchestra.run.smithi142.stderr:2024-03-31T23:07:50.918+0000 7f14d9616700 1 -- 172.21.15.142:0/1401967410 wait complete. 2024-03-31T23:07:51.591 DEBUG:teuthology.orchestra.run.smithi142:osd.10> sudo journalctl -f -n 0 -u ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4@osd.10.service 2024-03-31T23:07:51.594 INFO:tasks.cephadm:Deploying osd.11 on smithi162 with /dev/vg_nvme/lv_1... 2024-03-31T23:07:51.594 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:384f31f3d3aee7f8ba3f60846b8844c049165583 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6a9d8f08-efb2-11ee-b647-cb9ed24678a4 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-03-31T23:07:51.837 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/6a9d8f08-efb2-11ee-b647-cb9ed24678a4/mon.c/config 2024-03-31T23:07:52.171 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:07:51 smithi162 ceph-mon[36237]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 289 MiB used, 894 GiB / 894 GiB avail 2024-03-31T23:07:52.171 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:07:51 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:07:52.171 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:07:51 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:07:52.172 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:07:51 smithi162 ceph-mon[36237]: from='osd.10 [v2:172.21.15.142:6824/2845222366,v1:172.21.15.142:6825/2845222366]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-31T23:07:52.172 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:07:51 smithi162 ceph-mon[36237]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-31T23:07:52.172 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:07:51 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:07:52.172 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:07:51 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:07:52.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:07:51 smithi142 ceph-mon[35271]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 289 MiB used, 894 GiB / 894 GiB avail 2024-03-31T23:07:52.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:07:51 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:07:52.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:07:51 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:07:52.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:07:51 smithi142 ceph-mon[35271]: from='osd.10 [v2:172.21.15.142:6824/2845222366,v1:172.21.15.142:6825/2845222366]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-31T23:07:52.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:07:51 smithi142 ceph-mon[35271]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-31T23:07:52.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:07:51 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:07:52.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:07:51 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:07:52.361 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:07:51 smithi046 ceph-mon[32498]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 289 MiB used, 894 GiB / 894 GiB avail 2024-03-31T23:07:52.362 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:07:51 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:07:52.362 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:07:51 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:07:52.362 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:07:51 smithi046 ceph-mon[32498]: from='osd.10 [v2:172.21.15.142:6824/2845222366,v1:172.21.15.142:6825/2845222366]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-31T23:07:52.362 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:07:51 smithi046 ceph-mon[32498]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-31T23:07:52.362 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:07:51 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:07:52.362 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:07:51 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:07:53.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:07:52 smithi142 ceph-mon[35271]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-03-31T23:07:53.194 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:07:52 smithi142 ceph-mon[35271]: osdmap e71: 11 total, 10 up, 11 in 2024-03-31T23:07:53.194 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:07:52 smithi142 ceph-mon[35271]: from='osd.10 [v2:172.21.15.142:6824/2845222366,v1:172.21.15.142:6825/2845222366]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi142", "root=default"]}]: dispatch 2024-03-31T23:07:53.194 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:07:52 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-31T23:07:53.194 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:07:52 smithi142 ceph-mon[35271]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi142", "root=default"]}]: dispatch 2024-03-31T23:07:53.194 INFO:journalctl@ceph.osd.10.smithi142.stdout:Mar 31 23:07:52 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-10[60116]: 2024-03-31T23:07:52.893+0000 7f64c2eb2700 -1 osd.10 0 waiting for initial osdmap 2024-03-31T23:07:53.194 INFO:journalctl@ceph.osd.10.smithi142.stdout:Mar 31 23:07:52 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-10[60116]: 2024-03-31T23:07:52.908+0000 7f64beca5700 -1 osd.10 72 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-31T23:07:53.195 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:07:52 smithi162 ceph-mon[36237]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-03-31T23:07:53.195 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:07:52 smithi162 ceph-mon[36237]: osdmap e71: 11 total, 10 up, 11 in 2024-03-31T23:07:53.195 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:07:52 smithi162 ceph-mon[36237]: from='osd.10 [v2:172.21.15.142:6824/2845222366,v1:172.21.15.142:6825/2845222366]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi142", "root=default"]}]: dispatch 2024-03-31T23:07:53.196 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:07:52 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-31T23:07:53.196 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:07:52 smithi162 ceph-mon[36237]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi142", "root=default"]}]: dispatch 2024-03-31T23:07:53.361 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:07:52 smithi046 ceph-mon[32498]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-03-31T23:07:53.362 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:07:52 smithi046 ceph-mon[32498]: osdmap e71: 11 total, 10 up, 11 in 2024-03-31T23:07:53.362 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:07:52 smithi046 ceph-mon[32498]: from='osd.10 [v2:172.21.15.142:6824/2845222366,v1:172.21.15.142:6825/2845222366]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi142", "root=default"]}]: dispatch 2024-03-31T23:07:53.362 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:07:52 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-31T23:07:53.362 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:07:52 smithi046 ceph-mon[32498]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi142", "root=default"]}]: dispatch 2024-03-31T23:07:53.787 INFO:teuthology.orchestra.run.smithi162.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-03-31T23:07:53.787 INFO:teuthology.orchestra.run.smithi162.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-03-31T23:07:53.787 INFO:teuthology.orchestra.run.smithi162.stderr: stderr: 10+0 records in 2024-03-31T23:07:53.787 INFO:teuthology.orchestra.run.smithi162.stderr:10+0 records out 2024-03-31T23:07:53.787 INFO:teuthology.orchestra.run.smithi162.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0130997 s, 800 MB/s 2024-03-31T23:07:53.787 INFO:teuthology.orchestra.run.smithi162.stderr:--> Zapping successful for: 2024-03-31T23:07:54.124 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:07:53 smithi162 ceph-mon[36237]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 289 MiB used, 894 GiB / 894 GiB avail 2024-03-31T23:07:54.124 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:07:53 smithi162 ceph-mon[36237]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi142", "root=default"]}]': finished 2024-03-31T23:07:54.125 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:07:53 smithi162 ceph-mon[36237]: osdmap e72: 11 total, 10 up, 11 in 2024-03-31T23:07:54.125 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:07:53 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-31T23:07:54.125 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:07:53 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-31T23:07:54.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:07:53 smithi142 ceph-mon[35271]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 289 MiB used, 894 GiB / 894 GiB avail 2024-03-31T23:07:54.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:07:53 smithi142 ceph-mon[35271]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi142", "root=default"]}]': finished 2024-03-31T23:07:54.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:07:53 smithi142 ceph-mon[35271]: osdmap e72: 11 total, 10 up, 11 in 2024-03-31T23:07:54.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:07:53 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-31T23:07:54.194 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:07:53 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-31T23:07:54.361 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:07:53 smithi046 ceph-mon[32498]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 289 MiB used, 894 GiB / 894 GiB avail 2024-03-31T23:07:54.362 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:07:53 smithi046 ceph-mon[32498]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi142", "root=default"]}]': finished 2024-03-31T23:07:54.362 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:07:53 smithi046 ceph-mon[32498]: osdmap e72: 11 total, 10 up, 11 in 2024-03-31T23:07:54.362 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:07:53 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-31T23:07:54.362 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:07:53 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-31T23:07:54.558 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:384f31f3d3aee7f8ba3f60846b8844c049165583 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6a9d8f08-efb2-11ee-b647-cb9ed24678a4 -- ceph orch daemon add osd smithi162:vg_nvme/lv_1 2024-03-31T23:07:54.801 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/6a9d8f08-efb2-11ee-b647-cb9ed24678a4/mon.c/config 2024-03-31T23:07:55.166 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:07:54 smithi142 ceph-mon[35271]: purged_snaps scrub starts 2024-03-31T23:07:55.166 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:07:54 smithi142 ceph-mon[35271]: purged_snaps scrub ok 2024-03-31T23:07:55.166 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:07:54 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-31T23:07:55.167 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:07:54 smithi142 ceph-mon[35271]: osd.10 [v2:172.21.15.142:6824/2845222366,v1:172.21.15.142:6825/2845222366] boot 2024-03-31T23:07:55.167 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:07:54 smithi142 ceph-mon[35271]: osdmap e73: 11 total, 11 up, 11 in 2024-03-31T23:07:55.167 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:07:54 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-31T23:07:55.179 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:07:54 smithi162 ceph-mon[36237]: purged_snaps scrub starts 2024-03-31T23:07:55.179 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:07:54 smithi162 ceph-mon[36237]: purged_snaps scrub ok 2024-03-31T23:07:55.179 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:07:54 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-31T23:07:55.179 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:07:54 smithi162 ceph-mon[36237]: osd.10 [v2:172.21.15.142:6824/2845222366,v1:172.21.15.142:6825/2845222366] boot 2024-03-31T23:07:55.179 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:07:54 smithi162 ceph-mon[36237]: osdmap e73: 11 total, 11 up, 11 in 2024-03-31T23:07:55.179 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:07:54 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-31T23:07:55.361 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:07:54 smithi046 ceph-mon[32498]: purged_snaps scrub starts 2024-03-31T23:07:55.362 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:07:54 smithi046 ceph-mon[32498]: purged_snaps scrub ok 2024-03-31T23:07:55.362 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:07:54 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-31T23:07:55.362 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:07:54 smithi046 ceph-mon[32498]: osd.10 [v2:172.21.15.142:6824/2845222366,v1:172.21.15.142:6825/2845222366] boot 2024-03-31T23:07:55.362 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:07:54 smithi046 ceph-mon[32498]: osdmap e73: 11 total, 11 up, 11 in 2024-03-31T23:07:55.362 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:07:54 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-31T23:07:56.178 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:07:55 smithi162 ceph-mon[36237]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 289 MiB used, 894 GiB / 894 GiB avail 2024-03-31T23:07:56.178 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:07:55 smithi162 ceph-mon[36237]: osdmap e74: 11 total, 11 up, 11 in 2024-03-31T23:07:56.178 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:07:55 smithi162 ceph-mon[36237]: Detected new or changed devices on smithi142 2024-03-31T23:07:56.179 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:07:55 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:07:56.179 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:07:55 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:07:56.179 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:07:55 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-31T23:07:56.179 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:07:55 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-31T23:07:56.179 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:07:55 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-31T23:07:56.179 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:07:55 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-31T23:07:56.179 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:07:55 smithi162 ceph-mon[36237]: Adjusting osd_memory_target on smithi142 to 4272M 2024-03-31T23:07:56.179 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:07:55 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:07:56.179 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:07:55 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:07:56.179 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:07:55 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-31T23:07:56.179 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:07:55 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:07:56.179 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:07:55 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-31T23:07:56.179 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:07:55 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:07:56.179 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:07:55 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-31T23:07:56.179 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:07:55 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:07:56.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:07:55 smithi142 ceph-mon[35271]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 289 MiB used, 894 GiB / 894 GiB avail 2024-03-31T23:07:56.194 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:07:55 smithi142 ceph-mon[35271]: osdmap e74: 11 total, 11 up, 11 in 2024-03-31T23:07:56.194 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:07:55 smithi142 ceph-mon[35271]: Detected new or changed devices on smithi142 2024-03-31T23:07:56.194 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:07:55 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:07:56.194 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:07:55 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:07:56.194 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:07:55 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-31T23:07:56.194 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:07:55 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-31T23:07:56.194 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:07:55 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-31T23:07:56.194 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:07:55 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-31T23:07:56.194 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:07:55 smithi142 ceph-mon[35271]: Adjusting osd_memory_target on smithi142 to 4272M 2024-03-31T23:07:56.194 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:07:55 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:07:56.194 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:07:55 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:07:56.194 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:07:55 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-31T23:07:56.194 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:07:55 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:07:56.195 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:07:55 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-31T23:07:56.195 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:07:55 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:07:56.195 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:07:55 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-31T23:07:56.195 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:07:55 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:07:56.362 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:07:55 smithi046 ceph-mon[32498]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 289 MiB used, 894 GiB / 894 GiB avail 2024-03-31T23:07:56.362 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:07:55 smithi046 ceph-mon[32498]: osdmap e74: 11 total, 11 up, 11 in 2024-03-31T23:07:56.362 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:07:55 smithi046 ceph-mon[32498]: Detected new or changed devices on smithi142 2024-03-31T23:07:56.362 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:07:55 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:07:56.362 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:07:55 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:07:56.362 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:07:55 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-31T23:07:56.362 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:07:55 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-31T23:07:56.362 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:07:55 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-31T23:07:56.362 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:07:55 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-31T23:07:56.363 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:07:55 smithi046 ceph-mon[32498]: Adjusting osd_memory_target on smithi142 to 4272M 2024-03-31T23:07:56.363 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:07:55 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:07:56.363 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:07:55 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:07:56.363 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:07:55 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-31T23:07:56.363 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:07:55 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:07:56.363 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:07:55 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-31T23:07:56.363 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:07:55 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:07:56.363 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:07:55 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-31T23:07:56.363 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:07:55 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:07:56.542 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:07:56.538+0000 7f51b673e700 1 -- 172.21.15.162:0/3623855929 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f51b0103e00 msgr2=0x7f51b010a6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:07:56.542 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:07:56.538+0000 7f51b673e700 1 --2- 172.21.15.162:0/3623855929 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f51b0103e00 0x7f51b010a6b0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f51a800b0a0 tx=0x7f51a800b3b0 comp rx=0 tx=0).stop 2024-03-31T23:07:56.542 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:07:56.538+0000 7f51b673e700 1 -- 172.21.15.162:0/3623855929 shutdown_connections 2024-03-31T23:07:56.542 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:07:56.538+0000 7f51b673e700 1 --2- 172.21.15.162:0/3623855929 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f51b0102260 0x7f51b0102680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:07:56.542 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:07:56.538+0000 7f51b673e700 1 --2- 172.21.15.162:0/3623855929 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f51b0103e00 0x7f51b010a6b0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:07:56.542 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:07:56.538+0000 7f51b673e700 1 --2- 172.21.15.162:0/3623855929 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f51b0103460 0x7f51b01038c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:07:56.542 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:07:56.538+0000 7f51b673e700 1 -- 172.21.15.162:0/3623855929 >> 172.21.15.162:0/3623855929 conn(0x7f51b00fd820 msgr2=0x7f51b00ffc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:07:56.542 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:07:56.538+0000 7f51b673e700 1 -- 172.21.15.162:0/3623855929 shutdown_connections 2024-03-31T23:07:56.542 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:07:56.538+0000 7f51b673e700 1 -- 172.21.15.162:0/3623855929 wait complete. 2024-03-31T23:07:56.543 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:07:56.539+0000 7f51b673e700 1 Processor -- start 2024-03-31T23:07:56.543 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:07:56.539+0000 7f51b673e700 1 -- start start 2024-03-31T23:07:56.543 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:07:56.539+0000 7f51b673e700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f51b0102260 0x7f51b01984f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:07:56.543 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:07:56.539+0000 7f51b673e700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f51b0103460 0x7f51b0198a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:07:56.543 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:07:56.539+0000 7f51b673e700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f51b0103e00 0x7f51b019fab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:07:56.543 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:07:56.539+0000 7f51b673e700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f51b010ddf0 con 0x7f51b0103e00 2024-03-31T23:07:56.543 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:07:56.539+0000 7f51b673e700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f51b010dc70 con 0x7f51b0102260 2024-03-31T23:07:56.543 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:07:56.539+0000 7f51b673e700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f51b010df70 con 0x7f51b0103460 2024-03-31T23:07:56.543 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:07:56.539+0000 7f51af7fe700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f51b0103460 0x7f51b0198a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:07:56.543 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:07:56.539+0000 7f51af7fe700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f51b0103460 0x7f51b0198a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:40090/0 (socket says 172.21.15.162:40090) 2024-03-31T23:07:56.543 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:07:56.539+0000 7f51af7fe700 1 -- 172.21.15.162:0/1713703050 learned_addr learned my addr 172.21.15.162:0/1713703050 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-03-31T23:07:56.543 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:07:56.540+0000 7f51affff700 1 --2- 172.21.15.162:0/1713703050 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f51b0102260 0x7f51b01984f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:07:56.543 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:07:56.540+0000 7f51b4cdb700 1 --2- 172.21.15.162:0/1713703050 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f51b0103e00 0x7f51b019fab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:07:56.544 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:07:56.540+0000 7f51af7fe700 1 -- 172.21.15.162:0/1713703050 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f51b0102260 msgr2=0x7f51b01984f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:07:56.544 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:07:56.540+0000 7f51af7fe700 1 --2- 172.21.15.162:0/1713703050 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f51b0102260 0x7f51b01984f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:07:56.544 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:07:56.540+0000 7f51af7fe700 1 -- 172.21.15.162:0/1713703050 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f51b0103e00 msgr2=0x7f51b019fab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:07:56.544 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:07:56.540+0000 7f51af7fe700 1 --2- 172.21.15.162:0/1713703050 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f51b0103e00 0x7f51b019fab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:07:56.544 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:07:56.540+0000 7f51af7fe700 1 -- 172.21.15.162:0/1713703050 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f51b01a0150 con 0x7f51b0103460 2024-03-31T23:07:56.544 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:07:56.540+0000 7f51af7fe700 1 --2- 172.21.15.162:0/1713703050 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f51b0103460 0x7f51b0198a30 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f51a400b560 tx=0x7f51a400b870 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:07:56.544 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:07:56.540+0000 7f51ad7fa700 1 -- 172.21.15.162:0/1713703050 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51a4015020 con 0x7f51b0103460 2024-03-31T23:07:56.544 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:07:56.540+0000 7f51ad7fa700 1 -- 172.21.15.162:0/1713703050 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f51a4011ce0 con 0x7f51b0103460 2024-03-31T23:07:56.544 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:07:56.540+0000 7f51b673e700 1 -- 172.21.15.162:0/1713703050 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f51b00744d0 con 0x7f51b0103460 2024-03-31T23:07:56.544 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:07:56.540+0000 7f51ad7fa700 1 -- 172.21.15.162:0/1713703050 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51a4010d60 con 0x7f51b0103460 2024-03-31T23:07:56.544 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:07:56.540+0000 7f51b673e700 1 -- 172.21.15.162:0/1713703050 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f51b0074a90 con 0x7f51b0103460 2024-03-31T23:07:56.545 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:07:56.541+0000 7f51b673e700 1 -- 172.21.15.162:0/1713703050 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f51b0068f00 con 0x7f51b0103460 2024-03-31T23:07:56.545 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:07:56.542+0000 7f51ad7fa700 1 -- 172.21.15.162:0/1713703050 <== mon.1 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 16) ==== 137320+0+0 (secure 0 0 0) 0x7f51a4018020 con 0x7f51b0103460 2024-03-31T23:07:56.545 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:07:56.543+0000 7f51ad7fa700 1 --2- 172.21.15.162:0/1713703050 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f51980a2530 0x7f51980a49f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:07:56.545 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:07:56.543+0000 7f51ad7fa700 1 -- 172.21.15.162:0/1713703050 <== mon.1 v2:172.21.15.162:3300/0 5 ==== osd_map(75..75 src has 1..75) ==== 6723+0+0 (secure 0 0 0) 0x7f51a40cd8d0 con 0x7f51b0103460 2024-03-31T23:07:56.545 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:07:56.543+0000 7f51affff700 1 --2- 172.21.15.162:0/1713703050 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f51980a2530 0x7f51980a49f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:07:56.545 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:07:56.544+0000 7f51affff700 1 --2- 172.21.15.162:0/1713703050 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f51980a2530 0x7f51980a49f0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f51b01032c0 tx=0x7f51a0009190 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:07:56.548 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:07:56.547+0000 7f51ad7fa700 1 -- 172.21.15.162:0/1713703050 <== mon.1 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f51a401f020 con 0x7f51b0103460 2024-03-31T23:07:56.750 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:07:56.748+0000 7f51b673e700 1 -- 172.21.15.162:0/1713703050 --> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi162:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7f51b0062680 con 0x7f51980a2530 2024-03-31T23:07:57.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:07:56 smithi162 ceph-mon[36237]: osdmap e75: 11 total, 11 up, 11 in 2024-03-31T23:07:57.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:07:56 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-31T23:07:57.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:07:56 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-31T23:07:57.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:07:56 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:07:57.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:07:56 smithi142 ceph-mon[35271]: osdmap e75: 11 total, 11 up, 11 in 2024-03-31T23:07:57.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:07:56 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-31T23:07:57.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:07:56 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-31T23:07:57.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:07:56 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:07:57.361 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:07:56 smithi046 ceph-mon[32498]: osdmap e75: 11 total, 11 up, 11 in 2024-03-31T23:07:57.362 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:07:56 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-31T23:07:57.362 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:07:56 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-31T23:07:57.362 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:07:56 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:07:58.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:07:57 smithi162 ceph-mon[36237]: pgmap v239: 1 pgs: 1 remapped+peering; 577 KiB data, 316 MiB used, 983 GiB / 983 GiB avail 2024-03-31T23:07:58.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:07:57 smithi162 ceph-mon[36237]: from='client.24349 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi162:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-31T23:07:58.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:07:57 smithi162 ceph-mon[36237]: osdmap e76: 11 total, 11 up, 11 in 2024-03-31T23:07:58.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:07:57 smithi142 ceph-mon[35271]: pgmap v239: 1 pgs: 1 remapped+peering; 577 KiB data, 316 MiB used, 983 GiB / 983 GiB avail 2024-03-31T23:07:58.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:07:57 smithi142 ceph-mon[35271]: from='client.24349 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi162:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-31T23:07:58.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:07:57 smithi142 ceph-mon[35271]: osdmap e76: 11 total, 11 up, 11 in 2024-03-31T23:07:58.361 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:07:57 smithi046 ceph-mon[32498]: pgmap v239: 1 pgs: 1 remapped+peering; 577 KiB data, 316 MiB used, 983 GiB / 983 GiB avail 2024-03-31T23:07:58.362 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:07:57 smithi046 ceph-mon[32498]: from='client.24349 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi162:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-31T23:07:58.362 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:07:57 smithi046 ceph-mon[32498]: osdmap e76: 11 total, 11 up, 11 in 2024-03-31T23:07:59.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:07:58 smithi142 ceph-mon[35271]: osdmap e77: 11 total, 11 up, 11 in 2024-03-31T23:07:59.295 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:07:58 smithi162 ceph-mon[36237]: osdmap e77: 11 total, 11 up, 11 in 2024-03-31T23:07:59.361 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:07:58 smithi046 ceph-mon[32498]: osdmap e77: 11 total, 11 up, 11 in 2024-03-31T23:08:00.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:07:59 smithi142 ceph-mon[35271]: pgmap v242: 1 pgs: 1 remapped+peering; 577 KiB data, 316 MiB used, 983 GiB / 983 GiB avail 2024-03-31T23:08:00.194 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:07:59 smithi142 ceph-mon[35271]: from='client.? 172.21.15.162:0/802629304' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "72104d1c-3ea2-45ad-bc0b-0500f146c06b"}]: dispatch 2024-03-31T23:08:00.194 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:07:59 smithi142 ceph-mon[35271]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "72104d1c-3ea2-45ad-bc0b-0500f146c06b"}]: dispatch 2024-03-31T23:08:00.194 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:07:59 smithi142 ceph-mon[35271]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "72104d1c-3ea2-45ad-bc0b-0500f146c06b"}]': finished 2024-03-31T23:08:00.194 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:07:59 smithi142 ceph-mon[35271]: osdmap e78: 12 total, 11 up, 12 in 2024-03-31T23:08:00.194 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:07:59 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-31T23:08:00.296 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:07:59 smithi162 ceph-mon[36237]: pgmap v242: 1 pgs: 1 remapped+peering; 577 KiB data, 316 MiB used, 983 GiB / 983 GiB avail 2024-03-31T23:08:00.296 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:07:59 smithi162 ceph-mon[36237]: from='client.? 172.21.15.162:0/802629304' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "72104d1c-3ea2-45ad-bc0b-0500f146c06b"}]: dispatch 2024-03-31T23:08:00.296 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:07:59 smithi162 ceph-mon[36237]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "72104d1c-3ea2-45ad-bc0b-0500f146c06b"}]: dispatch 2024-03-31T23:08:00.296 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:07:59 smithi162 ceph-mon[36237]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "72104d1c-3ea2-45ad-bc0b-0500f146c06b"}]': finished 2024-03-31T23:08:00.296 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:07:59 smithi162 ceph-mon[36237]: osdmap e78: 12 total, 11 up, 12 in 2024-03-31T23:08:00.296 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:07:59 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-31T23:08:00.361 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:07:59 smithi046 ceph-mon[32498]: pgmap v242: 1 pgs: 1 remapped+peering; 577 KiB data, 316 MiB used, 983 GiB / 983 GiB avail 2024-03-31T23:08:00.362 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:07:59 smithi046 ceph-mon[32498]: from='client.? 172.21.15.162:0/802629304' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "72104d1c-3ea2-45ad-bc0b-0500f146c06b"}]: dispatch 2024-03-31T23:08:00.362 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:07:59 smithi046 ceph-mon[32498]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "72104d1c-3ea2-45ad-bc0b-0500f146c06b"}]: dispatch 2024-03-31T23:08:00.362 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:07:59 smithi046 ceph-mon[32498]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "72104d1c-3ea2-45ad-bc0b-0500f146c06b"}]': finished 2024-03-31T23:08:00.362 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:07:59 smithi046 ceph-mon[32498]: osdmap e78: 12 total, 11 up, 12 in 2024-03-31T23:08:00.362 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:07:59 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-31T23:08:01.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:08:00 smithi142 ceph-mon[35271]: from='client.? 172.21.15.162:0/2374388176' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-31T23:08:01.295 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:08:00 smithi162 ceph-mon[36237]: from='client.? 172.21.15.162:0/2374388176' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-31T23:08:01.361 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:08:00 smithi046 ceph-mon[32498]: from='client.? 172.21.15.162:0/2374388176' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-31T23:08:02.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:08:01 smithi142 ceph-mon[35271]: pgmap v244: 1 pgs: 1 remapped+peering; 577 KiB data, 317 MiB used, 983 GiB / 983 GiB avail 2024-03-31T23:08:02.295 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:08:01 smithi162 ceph-mon[36237]: pgmap v244: 1 pgs: 1 remapped+peering; 577 KiB data, 317 MiB used, 983 GiB / 983 GiB avail 2024-03-31T23:08:02.361 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:08:01 smithi046 ceph-mon[32498]: pgmap v244: 1 pgs: 1 remapped+peering; 577 KiB data, 317 MiB used, 983 GiB / 983 GiB avail 2024-03-31T23:08:04.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:08:03 smithi142 ceph-mon[35271]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 983 GiB / 983 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-31T23:08:04.295 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:08:03 smithi162 ceph-mon[36237]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 983 GiB / 983 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-31T23:08:04.361 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:08:03 smithi046 ceph-mon[32498]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 983 GiB / 983 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-31T23:08:06.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:08:05 smithi142 ceph-mon[35271]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 983 GiB / 983 GiB avail; 75 KiB/s, 0 objects/s recovering 2024-03-31T23:08:06.296 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:08:05 smithi162 ceph-mon[36237]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 983 GiB / 983 GiB avail; 75 KiB/s, 0 objects/s recovering 2024-03-31T23:08:06.361 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:08:05 smithi046 ceph-mon[32498]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 983 GiB / 983 GiB avail; 75 KiB/s, 0 objects/s recovering 2024-03-31T23:08:08.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:08:07 smithi142 ceph-mon[35271]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 983 GiB / 983 GiB avail; 67 KiB/s, 0 objects/s recovering 2024-03-31T23:08:08.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:08:07 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-03-31T23:08:08.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:08:07 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:08:08.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:08:07 smithi142 ceph-mon[35271]: Deploying daemon osd.11 on smithi162 2024-03-31T23:08:08.226 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:08:07 smithi162 ceph-mon[36237]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 983 GiB / 983 GiB avail; 67 KiB/s, 0 objects/s recovering 2024-03-31T23:08:08.226 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:08:07 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-03-31T23:08:08.226 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:08:07 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:08:08.226 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:08:07 smithi162 ceph-mon[36237]: Deploying daemon osd.11 on smithi162 2024-03-31T23:08:08.361 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:08:07 smithi046 ceph-mon[32498]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 983 GiB / 983 GiB avail; 67 KiB/s, 0 objects/s recovering 2024-03-31T23:08:08.361 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:08:07 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-03-31T23:08:08.362 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:08:07 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:08:08.362 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:08:07 smithi046 ceph-mon[32498]: Deploying daemon osd.11 on smithi162 2024-03-31T23:08:10.245 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:08:09 smithi162 ceph-mon[36237]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-31T23:08:10.361 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:08:09 smithi046 ceph-mon[32498]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-31T23:08:10.443 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:08:09 smithi142 ceph-mon[35271]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-31T23:08:12.289 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:08:11 smithi162 ceph-mon[36237]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 983 GiB / 983 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-03-31T23:08:12.361 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:08:11 smithi046 ceph-mon[32498]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 983 GiB / 983 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-03-31T23:08:12.443 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:08:11 smithi142 ceph-mon[35271]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 983 GiB / 983 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-03-31T23:08:13.216 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:08:12 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-31T23:08:13.217 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:08:12 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:08:13.217 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:08:12 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:08:13.361 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:08:12 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-31T23:08:13.362 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:08:12 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:08:13.362 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:08:12 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:08:13.443 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:08:12 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-31T23:08:13.443 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:08:12 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:08:13.443 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:08:12 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:08:14.854 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:08:14 smithi162 ceph-mon[36237]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 983 GiB / 983 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-03-31T23:08:14.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:08:14 smithi046 ceph-mon[32498]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 983 GiB / 983 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-03-31T23:08:14.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:08:14 smithi142 ceph-mon[35271]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 983 GiB / 983 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-03-31T23:08:16.424 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:08:16.423+0000 7f51ad7fa700 1 -- 172.21.15.162:0/1713703050 <== mgr.14152 v2:172.21.15.46:6800/162289569 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+38 (secure 0 0 0) 0x7f51b0062680 con 0x7f51980a2530 2024-03-31T23:08:16.424 INFO:teuthology.orchestra.run.smithi162.stdout:Created osd(s) 11 on host 'smithi162' 2024-03-31T23:08:16.426 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:08:16.426+0000 7f51b673e700 1 -- 172.21.15.162:0/1713703050 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f51980a2530 msgr2=0x7f51980a49f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:08:16.426 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:08:16.426+0000 7f51b673e700 1 --2- 172.21.15.162:0/1713703050 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f51980a2530 0x7f51980a49f0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f51b01032c0 tx=0x7f51a0009190 comp rx=0 tx=0).stop 2024-03-31T23:08:16.427 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:08:16.426+0000 7f51b673e700 1 -- 172.21.15.162:0/1713703050 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f51b0103460 msgr2=0x7f51b0198a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:08:16.427 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:08:16.426+0000 7f51b673e700 1 --2- 172.21.15.162:0/1713703050 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f51b0103460 0x7f51b0198a30 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f51a400b560 tx=0x7f51a400b870 comp rx=0 tx=0).stop 2024-03-31T23:08:16.427 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:08:16.426+0000 7f51b673e700 1 -- 172.21.15.162:0/1713703050 shutdown_connections 2024-03-31T23:08:16.427 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:08:16.426+0000 7f51b673e700 1 --2- 172.21.15.162:0/1713703050 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f51980a2530 0x7f51980a49f0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:08:16.427 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:08:16.426+0000 7f51b673e700 1 --2- 172.21.15.162:0/1713703050 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f51b0103e00 0x7f51b019fab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:08:16.427 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:08:16.426+0000 7f51b673e700 1 --2- 172.21.15.162:0/1713703050 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f51b0102260 0x7f51b01984f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:08:16.427 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:08:16.426+0000 7f51b673e700 1 --2- 172.21.15.162:0/1713703050 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f51b0103460 0x7f51b0198a30 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:08:16.427 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:08:16.426+0000 7f51b673e700 1 -- 172.21.15.162:0/1713703050 >> 172.21.15.162:0/1713703050 conn(0x7f51b00fd820 msgr2=0x7f51b01067d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:08:16.427 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:08:16.426+0000 7f51b673e700 1 -- 172.21.15.162:0/1713703050 shutdown_connections 2024-03-31T23:08:16.427 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:08:16.426+0000 7f51b673e700 1 -- 172.21.15.162:0/1713703050 wait complete. 2024-03-31T23:08:16.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:08:16 smithi046 ceph-mon[32498]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 983 GiB / 983 GiB avail 2024-03-31T23:08:16.862 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:08:16 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:08:16.862 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:08:16 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:08:16.862 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:08:16 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:08:16.862 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:08:16 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-31T23:08:16.862 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:08:16 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:08:16.862 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:08:16 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:08:16.862 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:08:16 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:08:16.862 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:08:16 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-31T23:08:16.862 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:08:16 smithi046 ceph-mon[32498]: from='osd.11 [v2:172.21.15.162:6824/1986323778,v1:172.21.15.162:6825/1986323778]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-31T23:08:16.862 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:08:16 smithi046 ceph-mon[32498]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-31T23:08:16.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:08:16 smithi142 ceph-mon[35271]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 983 GiB / 983 GiB avail 2024-03-31T23:08:16.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:08:16 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:08:16.944 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:08:16 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:08:16.944 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:08:16 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:08:16.944 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:08:16 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-31T23:08:16.944 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:08:16 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:08:16.944 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:08:16 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:08:16.944 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:08:16 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:08:16.944 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:08:16 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-31T23:08:16.944 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:08:16 smithi142 ceph-mon[35271]: from='osd.11 [v2:172.21.15.162:6824/1986323778,v1:172.21.15.162:6825/1986323778]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-31T23:08:16.944 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:08:16 smithi142 ceph-mon[35271]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-31T23:08:17.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:08:16 smithi162 ceph-mon[36237]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 983 GiB / 983 GiB avail 2024-03-31T23:08:17.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:08:16 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:08:17.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:08:16 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:08:17.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:08:16 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:08:17.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:08:16 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-31T23:08:17.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:08:16 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:08:17.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:08:16 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:08:17.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:08:16 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:08:17.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:08:16 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-31T23:08:17.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:08:16 smithi162 ceph-mon[36237]: from='osd.11 [v2:172.21.15.162:6824/1986323778,v1:172.21.15.162:6825/1986323778]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-31T23:08:17.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:08:16 smithi162 ceph-mon[36237]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-31T23:08:17.074 DEBUG:teuthology.orchestra.run.smithi162:osd.11> sudo journalctl -f -n 0 -u ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4@osd.11.service 2024-03-31T23:08:17.076 INFO:tasks.cephadm:Waiting for 12 OSDs to come up... 2024-03-31T23:08:17.077 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:384f31f3d3aee7f8ba3f60846b8844c049165583 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6a9d8f08-efb2-11ee-b647-cb9ed24678a4 -- ceph osd stat -f json 2024-03-31T23:08:17.327 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/6a9d8f08-efb2-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-31T23:08:18.546 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:08:18 smithi162 ceph-mon[36237]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 983 GiB / 983 GiB avail 2024-03-31T23:08:18.546 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:08:18 smithi162 ceph-mon[36237]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-03-31T23:08:18.546 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:08:18 smithi162 ceph-mon[36237]: osdmap e79: 12 total, 11 up, 12 in 2024-03-31T23:08:18.546 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:08:18 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-31T23:08:18.546 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:08:18 smithi162 ceph-mon[36237]: from='osd.11 [v2:172.21.15.162:6824/1986323778,v1:172.21.15.162:6825/1986323778]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]: dispatch 2024-03-31T23:08:18.546 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:08:18 smithi162 ceph-mon[36237]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]: dispatch 2024-03-31T23:08:18.694 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:08:18 smithi142 ceph-mon[35271]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 983 GiB / 983 GiB avail 2024-03-31T23:08:18.694 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:08:18 smithi142 ceph-mon[35271]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-03-31T23:08:18.694 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:08:18 smithi142 ceph-mon[35271]: osdmap e79: 12 total, 11 up, 12 in 2024-03-31T23:08:18.694 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:08:18 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-31T23:08:18.694 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:08:18 smithi142 ceph-mon[35271]: from='osd.11 [v2:172.21.15.162:6824/1986323778,v1:172.21.15.162:6825/1986323778]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]: dispatch 2024-03-31T23:08:18.695 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:08:18 smithi142 ceph-mon[35271]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]: dispatch 2024-03-31T23:08:18.701 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:08:18 smithi046 ceph-mon[32498]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 983 GiB / 983 GiB avail 2024-03-31T23:08:18.701 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:08:18 smithi046 ceph-mon[32498]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-03-31T23:08:18.701 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:08:18 smithi046 ceph-mon[32498]: osdmap e79: 12 total, 11 up, 12 in 2024-03-31T23:08:18.701 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:08:18 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-31T23:08:18.701 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:08:18 smithi046 ceph-mon[32498]: from='osd.11 [v2:172.21.15.162:6824/1986323778,v1:172.21.15.162:6825/1986323778]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]: dispatch 2024-03-31T23:08:18.701 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:08:18 smithi046 ceph-mon[32498]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]: dispatch 2024-03-31T23:08:19.281 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:19.277+0000 7f6b2073e700 1 -- 172.21.15.46:0/1057755234 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6b18103e10 msgr2=0x7f6b1810a6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:08:19.281 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:19.277+0000 7f6b2073e700 1 --2- 172.21.15.46:0/1057755234 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6b18103e10 0x7f6b1810a6c0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f6b1000b0d0 tx=0x7f6b1000b3e0 comp rx=0 tx=0).stop 2024-03-31T23:08:19.281 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:19.277+0000 7f6b2073e700 1 -- 172.21.15.46:0/1057755234 shutdown_connections 2024-03-31T23:08:19.281 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:19.277+0000 7f6b2073e700 1 --2- 172.21.15.46:0/1057755234 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6b18103e10 0x7f6b1810a6c0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:08:19.281 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:19.277+0000 7f6b2073e700 1 --2- 172.21.15.46:0/1057755234 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6b18102270 0x7f6b18102690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:08:19.281 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:19.277+0000 7f6b2073e700 1 --2- 172.21.15.46:0/1057755234 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6b18103470 0x7f6b181038d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:08:19.281 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:19.277+0000 7f6b2073e700 1 -- 172.21.15.46:0/1057755234 >> 172.21.15.46:0/1057755234 conn(0x7f6b180fd7f0 msgr2=0x7f6b180ffc50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:08:19.281 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:19.277+0000 7f6b2073e700 1 -- 172.21.15.46:0/1057755234 shutdown_connections 2024-03-31T23:08:19.282 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:19.277+0000 7f6b2073e700 1 -- 172.21.15.46:0/1057755234 wait complete. 2024-03-31T23:08:19.282 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:19.278+0000 7f6b2073e700 1 Processor -- start 2024-03-31T23:08:19.282 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:19.278+0000 7f6b2073e700 1 -- start start 2024-03-31T23:08:19.282 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:19.278+0000 7f6b2073e700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6b18102270 0x7f6b18075fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:08:19.282 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:19.278+0000 7f6b2073e700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6b18103470 0x7f6b18074630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:08:19.282 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:19.278+0000 7f6b2073e700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6b18103e10 0x7f6b18074b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:08:19.282 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:19.278+0000 7f6b2073e700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f6b1810dd00 con 0x7f6b18102270 2024-03-31T23:08:19.282 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:19.278+0000 7f6b2073e700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f6b1810db80 con 0x7f6b18103e10 2024-03-31T23:08:19.282 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:19.278+0000 7f6b2073e700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f6b1810de80 con 0x7f6b18103470 2024-03-31T23:08:19.282 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:19.278+0000 7f6b1e4da700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6b18102270 0x7f6b18075fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:08:19.282 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:19.278+0000 7f6b1e4da700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6b18102270 0x7f6b18075fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:43190/0 (socket says 172.21.15.46:43190) 2024-03-31T23:08:19.282 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:19.278+0000 7f6b1e4da700 1 -- 172.21.15.46:0/1887299085 learned_addr learned my addr 172.21.15.46:0/1887299085 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:08:19.282 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:19.278+0000 7f6b1dcd9700 1 --2- 172.21.15.46:0/1887299085 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6b18103470 0x7f6b18074630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:08:19.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:19.279+0000 7f6b1ecdb700 1 --2- 172.21.15.46:0/1887299085 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6b18103e10 0x7f6b18074b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:08:19.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:19.279+0000 7f6b1e4da700 1 -- 172.21.15.46:0/1887299085 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6b18103470 msgr2=0x7f6b18074630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:08:19.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:19.279+0000 7f6b1e4da700 1 --2- 172.21.15.46:0/1887299085 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6b18103470 0x7f6b18074630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:08:19.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:19.279+0000 7f6b1e4da700 1 -- 172.21.15.46:0/1887299085 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6b18103e10 msgr2=0x7f6b18074b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:08:19.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:19.279+0000 7f6b1e4da700 1 --2- 172.21.15.46:0/1887299085 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6b18103e10 0x7f6b18074b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:08:19.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:19.279+0000 7f6b1e4da700 1 -- 172.21.15.46:0/1887299085 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b18075420 con 0x7f6b18102270 2024-03-31T23:08:19.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:19.279+0000 7f6b1ecdb700 1 --2- 172.21.15.46:0/1887299085 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6b18103e10 0x7f6b18074b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T23:08:19.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:19.279+0000 7f6b1e4da700 1 --2- 172.21.15.46:0/1887299085 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6b18102270 0x7f6b18075fe0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f6b0800e950 tx=0x7f6b0800ec60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:08:19.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:19.279+0000 7f6b0f7fe700 1 -- 172.21.15.46:0/1887299085 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b0800eea0 con 0x7f6b18102270 2024-03-31T23:08:19.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:19.279+0000 7f6b0f7fe700 1 -- 172.21.15.46:0/1887299085 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6b080044e0 con 0x7f6b18102270 2024-03-31T23:08:19.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:19.279+0000 7f6b0f7fe700 1 -- 172.21.15.46:0/1887299085 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b08003e10 con 0x7f6b18102270 2024-03-31T23:08:19.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:19.279+0000 7f6b2073e700 1 -- 172.21.15.46:0/1887299085 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6b181a2440 con 0x7f6b18102270 2024-03-31T23:08:19.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:19.279+0000 7f6b2073e700 1 -- 172.21.15.46:0/1887299085 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6b181a2900 con 0x7f6b18102270 2024-03-31T23:08:19.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:19.280+0000 7f6b2073e700 1 -- 172.21.15.46:0/1887299085 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6b18068f00 con 0x7f6b18102270 2024-03-31T23:08:19.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:19.282+0000 7f6b0f7fe700 1 -- 172.21.15.46:0/1887299085 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 16) ==== 137320+0+0 (secure 0 0 0) 0x7f6b08004000 con 0x7f6b18102270 2024-03-31T23:08:19.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:19.282+0000 7f6b0f7fe700 1 --2- 172.21.15.46:0/1887299085 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f6b040a2530 0x7f6b040a49f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:08:19.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:19.283+0000 7f6b1dcd9700 1 --2- 172.21.15.46:0/1887299085 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f6b040a2530 0x7f6b040a49f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:08:19.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:19.283+0000 7f6b0f7fe700 1 -- 172.21.15.46:0/1887299085 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 6847+0+0 (secure 0 0 0) 0x7f6b080ce800 con 0x7f6b18102270 2024-03-31T23:08:19.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:19.283+0000 7f6b1dcd9700 1 --2- 172.21.15.46:0/1887299085 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f6b040a2530 0x7f6b040a49f0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f6b1400ac50 tx=0x7f6b1400a380 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:08:19.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:19.286+0000 7f6b0f7fe700 1 -- 172.21.15.46:0/1887299085 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f6b08016370 con 0x7f6b18102270 2024-03-31T23:08:19.456 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:08:19 smithi162 ceph-mon[36237]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]': finished 2024-03-31T23:08:19.456 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:08:19 smithi162 ceph-mon[36237]: osdmap e80: 12 total, 11 up, 12 in 2024-03-31T23:08:19.457 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:08:19 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-31T23:08:19.457 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:08:19 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-31T23:08:19.457 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:08:19 smithi162 ceph-mon[36237]: from='osd.11 ' entity='osd.11' 2024-03-31T23:08:19.457 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:08:19 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:08:19.197+0000 7f98d6d4a700 -1 osd.11 0 waiting for initial osdmap 2024-03-31T23:08:19.457 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:08:19 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:08:19.209+0000 7f98ceb35700 -1 osd.11 80 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-31T23:08:19.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:19.496+0000 7f6b2073e700 1 -- 172.21.15.46:0/1887299085 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) -- 0x7f6b18062e50 con 0x7f6b18102270 2024-03-31T23:08:19.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:19.496+0000 7f6b0f7fe700 1 -- 172.21.15.46:0/1887299085 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v80) ==== 74+0+133 (secure 0 0 0) 0x7f6b0809bf80 con 0x7f6b18102270 2024-03-31T23:08:19.498 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-31T23:08:19.500 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:19.498+0000 7f6b2073e700 1 -- 172.21.15.46:0/1887299085 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f6b040a2530 msgr2=0x7f6b040a49f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:08:19.500 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:19.498+0000 7f6b2073e700 1 --2- 172.21.15.46:0/1887299085 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f6b040a2530 0x7f6b040a49f0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f6b1400ac50 tx=0x7f6b1400a380 comp rx=0 tx=0).stop 2024-03-31T23:08:19.500 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:19.498+0000 7f6b2073e700 1 -- 172.21.15.46:0/1887299085 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6b18102270 msgr2=0x7f6b18075fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:08:19.500 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:19.498+0000 7f6b2073e700 1 --2- 172.21.15.46:0/1887299085 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6b18102270 0x7f6b18075fe0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f6b0800e950 tx=0x7f6b0800ec60 comp rx=0 tx=0).stop 2024-03-31T23:08:19.500 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:19.499+0000 7f6b2073e700 1 -- 172.21.15.46:0/1887299085 shutdown_connections 2024-03-31T23:08:19.500 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:19.499+0000 7f6b2073e700 1 --2- 172.21.15.46:0/1887299085 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f6b040a2530 0x7f6b040a49f0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:08:19.500 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:19.499+0000 7f6b2073e700 1 --2- 172.21.15.46:0/1887299085 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6b18102270 0x7f6b18075fe0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:08:19.500 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:19.499+0000 7f6b2073e700 1 --2- 172.21.15.46:0/1887299085 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6b18103e10 0x7f6b18074b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:08:19.500 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:19.499+0000 7f6b2073e700 1 --2- 172.21.15.46:0/1887299085 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6b18103470 0x7f6b18074630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:08:19.500 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:19.499+0000 7f6b2073e700 1 -- 172.21.15.46:0/1887299085 >> 172.21.15.46:0/1887299085 conn(0x7f6b180fd7f0 msgr2=0x7f6b18106710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:08:19.501 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:19.499+0000 7f6b2073e700 1 -- 172.21.15.46:0/1887299085 shutdown_connections 2024-03-31T23:08:19.501 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:19.499+0000 7f6b2073e700 1 -- 172.21.15.46:0/1887299085 wait complete. 2024-03-31T23:08:19.611 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:08:19 smithi046 ceph-mon[32498]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]': finished 2024-03-31T23:08:19.611 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:08:19 smithi046 ceph-mon[32498]: osdmap e80: 12 total, 11 up, 12 in 2024-03-31T23:08:19.611 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:08:19 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-31T23:08:19.612 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:08:19 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-31T23:08:19.612 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:08:19 smithi046 ceph-mon[32498]: from='osd.11 ' entity='osd.11' 2024-03-31T23:08:19.693 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:08:19 smithi142 ceph-mon[35271]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]': finished 2024-03-31T23:08:19.693 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:08:19 smithi142 ceph-mon[35271]: osdmap e80: 12 total, 11 up, 12 in 2024-03-31T23:08:19.693 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:08:19 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-31T23:08:19.693 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:08:19 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-31T23:08:19.693 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:08:19 smithi142 ceph-mon[35271]: from='osd.11 ' entity='osd.11' 2024-03-31T23:08:20.228 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":80,"num_osds":12,"num_up_osds":11,"osd_up_since":1711926473,"num_in_osds":12,"osd_in_since":1711926479,"num_remapped_pgs":0} 2024-03-31T23:08:20.611 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:08:20 smithi046 ceph-mon[32498]: purged_snaps scrub starts 2024-03-31T23:08:20.611 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:08:20 smithi046 ceph-mon[32498]: purged_snaps scrub ok 2024-03-31T23:08:20.612 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:08:20 smithi046 ceph-mon[32498]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 983 GiB / 983 GiB avail 2024-03-31T23:08:20.612 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:08:20 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-31T23:08:20.612 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:08:20 smithi046 ceph-mon[32498]: from='client.? 172.21.15.46:0/1887299085' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-31T23:08:20.612 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:08:20 smithi046 ceph-mon[32498]: osd.11 [v2:172.21.15.162:6824/1986323778,v1:172.21.15.162:6825/1986323778] boot 2024-03-31T23:08:20.612 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:08:20 smithi046 ceph-mon[32498]: osdmap e81: 12 total, 12 up, 12 in 2024-03-31T23:08:20.612 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:08:20 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-31T23:08:20.693 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:08:20 smithi142 ceph-mon[35271]: purged_snaps scrub starts 2024-03-31T23:08:20.693 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:08:20 smithi142 ceph-mon[35271]: purged_snaps scrub ok 2024-03-31T23:08:20.693 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:08:20 smithi142 ceph-mon[35271]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 983 GiB / 983 GiB avail 2024-03-31T23:08:20.693 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:08:20 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-31T23:08:20.693 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:08:20 smithi142 ceph-mon[35271]: from='client.? 172.21.15.46:0/1887299085' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-31T23:08:20.693 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:08:20 smithi142 ceph-mon[35271]: osd.11 [v2:172.21.15.162:6824/1986323778,v1:172.21.15.162:6825/1986323778] boot 2024-03-31T23:08:20.693 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:08:20 smithi142 ceph-mon[35271]: osdmap e81: 12 total, 12 up, 12 in 2024-03-31T23:08:20.694 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:08:20 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-31T23:08:20.795 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:08:20 smithi162 ceph-mon[36237]: purged_snaps scrub starts 2024-03-31T23:08:20.795 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:08:20 smithi162 ceph-mon[36237]: purged_snaps scrub ok 2024-03-31T23:08:20.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:08:20 smithi162 ceph-mon[36237]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 983 GiB / 983 GiB avail 2024-03-31T23:08:20.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:08:20 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-31T23:08:20.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:08:20 smithi162 ceph-mon[36237]: from='client.? 172.21.15.46:0/1887299085' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-31T23:08:20.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:08:20 smithi162 ceph-mon[36237]: osd.11 [v2:172.21.15.162:6824/1986323778,v1:172.21.15.162:6825/1986323778] boot 2024-03-31T23:08:20.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:08:20 smithi162 ceph-mon[36237]: osdmap e81: 12 total, 12 up, 12 in 2024-03-31T23:08:20.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:08:20 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-31T23:08:21.229 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:384f31f3d3aee7f8ba3f60846b8844c049165583 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6a9d8f08-efb2-11ee-b647-cb9ed24678a4 -- ceph osd stat -f json 2024-03-31T23:08:21.471 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/6a9d8f08-efb2-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-31T23:08:21.593 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:08:21 smithi046 ceph-mon[32498]: Detected new or changed devices on smithi162 2024-03-31T23:08:21.593 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:08:21 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:08:21.594 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:08:21 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:08:21.594 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:08:21 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-31T23:08:21.594 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:08:21 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-31T23:08:21.594 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:08:21 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-31T23:08:21.594 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:08:21 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-31T23:08:21.594 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:08:21 smithi046 ceph-mon[32498]: Adjusting osd_memory_target on smithi162 to 4272M 2024-03-31T23:08:21.594 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:08:21 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:08:21.594 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:08:21 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:08:21.594 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:08:21 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-31T23:08:21.594 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:08:21 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:08:21.594 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:08:21 smithi046 ceph-mon[32498]: osdmap e82: 12 total, 12 up, 12 in 2024-03-31T23:08:21.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:08:21 smithi162 ceph-mon[36237]: Detected new or changed devices on smithi162 2024-03-31T23:08:21.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:08:21 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:08:21.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:08:21 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:08:21.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:08:21 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-31T23:08:21.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:08:21 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-31T23:08:21.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:08:21 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-31T23:08:21.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:08:21 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-31T23:08:21.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:08:21 smithi162 ceph-mon[36237]: Adjusting osd_memory_target on smithi162 to 4272M 2024-03-31T23:08:21.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:08:21 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:08:21.797 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:08:21 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:08:21.797 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:08:21 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-31T23:08:21.797 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:08:21 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:08:21.797 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:08:21 smithi162 ceph-mon[36237]: osdmap e82: 12 total, 12 up, 12 in 2024-03-31T23:08:21.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:08:21 smithi142 ceph-mon[35271]: Detected new or changed devices on smithi162 2024-03-31T23:08:21.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:08:21 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:08:21.944 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:08:21 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:08:21.944 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:08:21 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-31T23:08:21.944 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:08:21 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-31T23:08:21.944 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:08:21 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-31T23:08:21.944 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:08:21 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-31T23:08:21.944 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:08:21 smithi142 ceph-mon[35271]: Adjusting osd_memory_target on smithi162 to 4272M 2024-03-31T23:08:21.944 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:08:21 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:08:21.944 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:08:21 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:08:21.944 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:08:21 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-31T23:08:21.944 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:08:21 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:08:21.944 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:08:21 smithi142 ceph-mon[35271]: osdmap e82: 12 total, 12 up, 12 in 2024-03-31T23:08:22.760 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:08:22 smithi046 ceph-mon[32498]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-31T23:08:22.795 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:08:22 smithi162 ceph-mon[36237]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-31T23:08:22.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:08:22 smithi142 ceph-mon[35271]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-31T23:08:23.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:23.327+0000 7f30f7221700 1 -- 172.21.15.46:0/1960543583 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f30f0102370 msgr2=0x7f30f01027d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:08:23.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:23.327+0000 7f30f7221700 1 --2- 172.21.15.46:0/1960543583 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f30f0102370 0x7f30f01027d0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f30dc0099c0 tx=0x7f30dc009cd0 comp rx=0 tx=0).stop 2024-03-31T23:08:23.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:23.327+0000 7f30f7221700 1 -- 172.21.15.46:0/1960543583 shutdown_connections 2024-03-31T23:08:23.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:23.327+0000 7f30f7221700 1 --2- 172.21.15.46:0/1960543583 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f30f0102d10 0x7f30f01095c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:08:23.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:23.327+0000 7f30f7221700 1 --2- 172.21.15.46:0/1960543583 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f30f0101170 0x7f30f0101590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:08:23.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:23.327+0000 7f30f7221700 1 --2- 172.21.15.46:0/1960543583 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f30f0102370 0x7f30f01027d0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:08:23.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:23.327+0000 7f30f7221700 1 -- 172.21.15.46:0/1960543583 >> 172.21.15.46:0/1960543583 conn(0x7f30f00fc710 msgr2=0x7f30f00feb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:08:23.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:23.327+0000 7f30f7221700 1 -- 172.21.15.46:0/1960543583 shutdown_connections 2024-03-31T23:08:23.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:23.327+0000 7f30f7221700 1 -- 172.21.15.46:0/1960543583 wait complete. 2024-03-31T23:08:23.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:23.327+0000 7f30f7221700 1 Processor -- start 2024-03-31T23:08:23.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:23.328+0000 7f30f7221700 1 -- start start 2024-03-31T23:08:23.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:23.328+0000 7f30f7221700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f30f0101170 0x7f30f0198390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:08:23.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:23.328+0000 7f30f7221700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f30f0102370 0x7f30f01988d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:08:23.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:23.328+0000 7f30f7221700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f30f0102d10 0x7f30f019f900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:08:23.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:23.328+0000 7f30f7221700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f30f010cbf0 con 0x7f30f0102d10 2024-03-31T23:08:23.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:23.328+0000 7f30f7221700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f30f010ca70 con 0x7f30f0102370 2024-03-31T23:08:23.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:23.328+0000 7f30f7221700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f30f010cd70 con 0x7f30f0101170 2024-03-31T23:08:23.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:23.328+0000 7f30f57be700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f30f0102d10 0x7f30f019f900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:08:23.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:23.328+0000 7f30f57be700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f30f0102d10 0x7f30f019f900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44392/0 (socket says 172.21.15.46:44392) 2024-03-31T23:08:23.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:23.328+0000 7f30f57be700 1 -- 172.21.15.46:0/807539841 learned_addr learned my addr 172.21.15.46:0/807539841 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:08:23.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:23.328+0000 7f30effff700 1 --2- 172.21.15.46:0/807539841 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f30f0102370 0x7f30f01988d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:08:23.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:23.328+0000 7f30f4fbd700 1 --2- 172.21.15.46:0/807539841 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f30f0101170 0x7f30f0198390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:08:23.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:23.328+0000 7f30f57be700 1 -- 172.21.15.46:0/807539841 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f30f0101170 msgr2=0x7f30f0198390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:08:23.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:23.328+0000 7f30f57be700 1 --2- 172.21.15.46:0/807539841 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f30f0101170 0x7f30f0198390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:08:23.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:23.328+0000 7f30f57be700 1 -- 172.21.15.46:0/807539841 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f30f0102370 msgr2=0x7f30f01988d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:08:23.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:23.328+0000 7f30f57be700 1 --2- 172.21.15.46:0/807539841 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f30f0102370 0x7f30f01988d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:08:23.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:23.328+0000 7f30f57be700 1 -- 172.21.15.46:0/807539841 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f30f019ffa0 con 0x7f30f0102d10 2024-03-31T23:08:23.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:23.328+0000 7f30f57be700 1 --2- 172.21.15.46:0/807539841 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f30f0102d10 0x7f30f019f900 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f30e800ef60 tx=0x7f30e800ef90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:08:23.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:23.329+0000 7f30edffb700 1 -- 172.21.15.46:0/807539841 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30e80092e0 con 0x7f30f0102d10 2024-03-31T23:08:23.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:23.329+0000 7f30edffb700 1 -- 172.21.15.46:0/807539841 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f30e8004bb0 con 0x7f30f0102d10 2024-03-31T23:08:23.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:23.329+0000 7f30f7221700 1 -- 172.21.15.46:0/807539841 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f30f01a0230 con 0x7f30f0102d10 2024-03-31T23:08:23.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:23.329+0000 7f30edffb700 1 -- 172.21.15.46:0/807539841 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30e8004280 con 0x7f30f0102d10 2024-03-31T23:08:23.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:23.329+0000 7f30f7221700 1 -- 172.21.15.46:0/807539841 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f30f01a07c0 con 0x7f30f0102d10 2024-03-31T23:08:23.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:23.330+0000 7f30f7221700 1 -- 172.21.15.46:0/807539841 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f30f0068f00 con 0x7f30f0102d10 2024-03-31T23:08:23.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:23.331+0000 7f30edffb700 1 -- 172.21.15.46:0/807539841 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 16) ==== 137320+0+0 (secure 0 0 0) 0x7f30e801b030 con 0x7f30f0102d10 2024-03-31T23:08:23.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:23.331+0000 7f30edffb700 1 --2- 172.21.15.46:0/807539841 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f30e00a25b0 0x7f30e00a4a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:08:23.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:23.331+0000 7f30f4fbd700 1 --2- 172.21.15.46:0/807539841 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f30e00a25b0 0x7f30e00a4a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:08:23.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:23.331+0000 7f30edffb700 1 -- 172.21.15.46:0/807539841 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f30e80cdcc0 con 0x7f30f0102d10 2024-03-31T23:08:23.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:23.332+0000 7f30f4fbd700 1 --2- 172.21.15.46:0/807539841 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f30e00a25b0 0x7f30e00a4a70 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f30e4005fd0 tx=0x7f30e4005e50 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:08:23.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:23.336+0000 7f30edffb700 1 -- 172.21.15.46:0/807539841 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f30e8022020 con 0x7f30f0102d10 2024-03-31T23:08:23.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:23.541+0000 7f30f7221700 1 -- 172.21.15.46:0/807539841 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) -- 0x7f30f01a0bb0 con 0x7f30f0102d10 2024-03-31T23:08:23.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:23.542+0000 7f30edffb700 1 -- 172.21.15.46:0/807539841 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v82) ==== 74+0+133 (secure 0 0 0) 0x7f30e8010040 con 0x7f30f0102d10 2024-03-31T23:08:23.543 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-31T23:08:23.545 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:23.544+0000 7f30f7221700 1 -- 172.21.15.46:0/807539841 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f30e00a25b0 msgr2=0x7f30e00a4a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:08:23.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:23.544+0000 7f30f7221700 1 --2- 172.21.15.46:0/807539841 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f30e00a25b0 0x7f30e00a4a70 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f30e4005fd0 tx=0x7f30e4005e50 comp rx=0 tx=0).stop 2024-03-31T23:08:23.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:23.544+0000 7f30f7221700 1 -- 172.21.15.46:0/807539841 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f30f0102d10 msgr2=0x7f30f019f900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:08:23.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:23.544+0000 7f30f7221700 1 --2- 172.21.15.46:0/807539841 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f30f0102d10 0x7f30f019f900 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f30e800ef60 tx=0x7f30e800ef90 comp rx=0 tx=0).stop 2024-03-31T23:08:23.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:23.544+0000 7f30f7221700 1 -- 172.21.15.46:0/807539841 shutdown_connections 2024-03-31T23:08:23.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:23.544+0000 7f30f7221700 1 --2- 172.21.15.46:0/807539841 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f30e00a25b0 0x7f30e00a4a70 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:08:23.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:23.544+0000 7f30f7221700 1 --2- 172.21.15.46:0/807539841 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f30f0102d10 0x7f30f019f900 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:08:23.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:23.544+0000 7f30f7221700 1 --2- 172.21.15.46:0/807539841 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f30f0101170 0x7f30f0198390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:08:23.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:23.544+0000 7f30f7221700 1 --2- 172.21.15.46:0/807539841 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f30f0102370 0x7f30f01988d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:08:23.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:23.544+0000 7f30f7221700 1 -- 172.21.15.46:0/807539841 >> 172.21.15.46:0/807539841 conn(0x7f30f00fc710 msgr2=0x7f30f00fe1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:08:23.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:23.544+0000 7f30f7221700 1 -- 172.21.15.46:0/807539841 shutdown_connections 2024-03-31T23:08:23.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:23.544+0000 7f30f7221700 1 -- 172.21.15.46:0/807539841 wait complete. 2024-03-31T23:08:24.335 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":82,"num_osds":12,"num_up_osds":12,"osd_up_since":1711926500,"num_in_osds":12,"osd_in_since":1711926479,"num_remapped_pgs":0} 2024-03-31T23:08:24.335 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:384f31f3d3aee7f8ba3f60846b8844c049165583 shell --fsid 6a9d8f08-efb2-11ee-b647-cb9ed24678a4 -- ceph osd dump --format=json 2024-03-31T23:08:24.583 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/6a9d8f08-efb2-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-31T23:08:24.611 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:08:24 smithi046 ceph-mon[32498]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-31T23:08:24.611 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:08:24 smithi046 ceph-mon[32498]: from='client.? 172.21.15.46:0/807539841' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-31T23:08:24.795 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:08:24 smithi162 ceph-mon[36237]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-31T23:08:24.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:08:24 smithi162 ceph-mon[36237]: from='client.? 172.21.15.46:0/807539841' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-31T23:08:24.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:08:24 smithi142 ceph-mon[35271]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-31T23:08:24.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:08:24 smithi142 ceph-mon[35271]: from='client.? 172.21.15.46:0/807539841' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-31T23:08:26.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:26.505+0000 7fa4482d7700 1 -- 172.21.15.46:0/380630078 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa440101c80 msgr2=0x7fa440102100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:08:26.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:26.505+0000 7fa4482d7700 1 --2- 172.21.15.46:0/380630078 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa440101c80 0x7fa440102100 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fa43c009990 tx=0x7fa43c009ca0 comp rx=0 tx=0).stop 2024-03-31T23:08:26.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:26.505+0000 7fa4482d7700 1 -- 172.21.15.46:0/380630078 shutdown_connections 2024-03-31T23:08:26.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:26.505+0000 7fa4482d7700 1 --2- 172.21.15.46:0/380630078 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa440102640 0x7fa44010e010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:08:26.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:26.505+0000 7fa4482d7700 1 --2- 172.21.15.46:0/380630078 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa440106170 0x7fa440106550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:08:26.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:26.505+0000 7fa4482d7700 1 --2- 172.21.15.46:0/380630078 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa440101c80 0x7fa440102100 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:08:26.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:26.505+0000 7fa4482d7700 1 -- 172.21.15.46:0/380630078 >> 172.21.15.46:0/380630078 conn(0x7fa440076470 msgr2=0x7fa440076880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:08:26.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:26.505+0000 7fa4482d7700 1 -- 172.21.15.46:0/380630078 shutdown_connections 2024-03-31T23:08:26.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:26.505+0000 7fa4482d7700 1 -- 172.21.15.46:0/380630078 wait complete. 2024-03-31T23:08:26.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:26.506+0000 7fa4482d7700 1 Processor -- start 2024-03-31T23:08:26.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:26.506+0000 7fa4482d7700 1 -- start start 2024-03-31T23:08:26.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:26.506+0000 7fa4482d7700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa440101c80 0x7fa440102e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:08:26.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:26.506+0000 7fa4482d7700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa440102640 0x7fa4401033b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:08:26.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:26.506+0000 7fa4482d7700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa440106170 0x7fa4401038f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:08:26.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:26.506+0000 7fa4482d7700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fa4401113a0 con 0x7fa440106170 2024-03-31T23:08:26.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:26.506+0000 7fa4482d7700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fa440111220 con 0x7fa440101c80 2024-03-31T23:08:26.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:26.506+0000 7fa4482d7700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fa440111520 con 0x7fa440102640 2024-03-31T23:08:26.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:26.506+0000 7fa446874700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa440106170 0x7fa4401038f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:08:26.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:26.506+0000 7fa446874700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa440106170 0x7fa4401038f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44414/0 (socket says 172.21.15.46:44414) 2024-03-31T23:08:26.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:26.506+0000 7fa446874700 1 -- 172.21.15.46:0/2460664332 learned_addr learned my addr 172.21.15.46:0/2460664332 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:08:26.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:26.507+0000 7fa445872700 1 --2- 172.21.15.46:0/2460664332 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa440102640 0x7fa4401033b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:08:26.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:26.507+0000 7fa446073700 1 --2- 172.21.15.46:0/2460664332 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa440101c80 0x7fa440102e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:08:26.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:26.507+0000 7fa446874700 1 -- 172.21.15.46:0/2460664332 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa440102640 msgr2=0x7fa4401033b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:08:26.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:26.507+0000 7fa446874700 1 --2- 172.21.15.46:0/2460664332 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa440102640 0x7fa4401033b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:08:26.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:26.507+0000 7fa446874700 1 -- 172.21.15.46:0/2460664332 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa440101c80 msgr2=0x7fa440102e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:08:26.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:26.507+0000 7fa446874700 1 --2- 172.21.15.46:0/2460664332 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa440101c80 0x7fa440102e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:08:26.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:26.507+0000 7fa446874700 1 -- 172.21.15.46:0/2460664332 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4401a6f00 con 0x7fa440106170 2024-03-31T23:08:26.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:26.507+0000 7fa446073700 1 --2- 172.21.15.46:0/2460664332 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa440101c80 0x7fa440102e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T23:08:26.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:26.507+0000 7fa446874700 1 --2- 172.21.15.46:0/2460664332 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa440106170 0x7fa4401038f0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fa438007ee0 tx=0x7fa43800d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:08:26.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:26.507+0000 7fa4377fe700 1 -- 172.21.15.46:0/2460664332 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa43800dbd0 con 0x7fa440106170 2024-03-31T23:08:26.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:26.507+0000 7fa4482d7700 1 -- 172.21.15.46:0/2460664332 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa4401a71f0 con 0x7fa440106170 2024-03-31T23:08:26.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:26.507+0000 7fa4377fe700 1 -- 172.21.15.46:0/2460664332 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa43800dd70 con 0x7fa440106170 2024-03-31T23:08:26.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:26.507+0000 7fa4377fe700 1 -- 172.21.15.46:0/2460664332 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa438005990 con 0x7fa440106170 2024-03-31T23:08:26.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:26.507+0000 7fa4482d7700 1 -- 172.21.15.46:0/2460664332 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa4401a7780 con 0x7fa440106170 2024-03-31T23:08:26.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:26.508+0000 7fa4482d7700 1 -- 172.21.15.46:0/2460664332 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa440068f00 con 0x7fa440106170 2024-03-31T23:08:26.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:26.509+0000 7fa4377fe700 1 -- 172.21.15.46:0/2460664332 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 16) ==== 137320+0+0 (secure 0 0 0) 0x7fa43801a020 con 0x7fa440106170 2024-03-31T23:08:26.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:26.510+0000 7fa4377fe700 1 --2- 172.21.15.46:0/2460664332 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fa42c0a25b0 0x7fa42c0a4a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:08:26.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:26.510+0000 7fa446073700 1 --2- 172.21.15.46:0/2460664332 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fa42c0a25b0 0x7fa42c0a4a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:08:26.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:26.510+0000 7fa4377fe700 1 -- 172.21.15.46:0/2460664332 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7fa4380cd9e0 con 0x7fa440106170 2024-03-31T23:08:26.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:26.510+0000 7fa446073700 1 --2- 172.21.15.46:0/2460664332 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fa42c0a25b0 0x7fa42c0a4a70 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fa430005fd0 tx=0x7fa430005db0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:08:26.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:26.514+0000 7fa4377fe700 1 -- 172.21.15.46:0/2460664332 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fa438010070 con 0x7fa440106170 2024-03-31T23:08:26.718 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:08:26 smithi046 ceph-mon[32498]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-31T23:08:26.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:26.716+0000 7fa4482d7700 1 -- 172.21.15.46:0/2460664332 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fa44002db30 con 0x7fa440106170 2024-03-31T23:08:26.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:26.718+0000 7fa4377fe700 1 -- 172.21.15.46:0/2460664332 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v82) ==== 74+0+18433 (secure 0 0 0) 0x7fa43809afa0 con 0x7fa440106170 2024-03-31T23:08:26.719 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-31T23:08:26.720 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":82,"fsid":"6a9d8f08-efb2-11ee-b647-cb9ed24678a4","created":"2024-03-31T23:01:15.463711+0000","modified":"2024-03-31T23:08:21.201474+0000","last_up_change":"2024-03-31T23:08:20.199888+0000","last_in_change":"2024-03-31T23:07:59.438035+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-31T23:04:32.823751+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"47429095-3f46-4ed7-b7c4-504464719815","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":1824772695},{"type":"v1","addr":"172.21.15.46:6803","nonce":1824772695}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":1824772695},{"type":"v1","addr":"172.21.15.46:6805","nonce":1824772695}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":1824772695},{"type":"v1","addr":"172.21.15.46:6809","nonce":1824772695}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":1824772695},{"type":"v1","addr":"172.21.15.46:6807","nonce":1824772695}]},"public_addr":"172.21.15.46:6803/1824772695","cluster_addr":"172.21.15.46:6805/1824772695","heartbeat_back_addr":"172.21.15.46:6809/1824772695","heartbeat_front_addr":"172.21.15.46:6807/1824772695","state":["exists","up"]},{"osd":1,"uuid":"be56f883-411e-4dbe-b50e-a158134df4a6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6800","nonce":973693922},{"type":"v1","addr":"172.21.15.142:6801","nonce":973693922}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6802","nonce":973693922},{"type":"v1","addr":"172.21.15.142:6803","nonce":973693922}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6806","nonce":973693922},{"type":"v1","addr":"172.21.15.142:6807","nonce":973693922}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6804","nonce":973693922},{"type":"v1","addr":"172.21.15.142:6805","nonce":973693922}]},"public_addr":"172.21.15.142:6801/973693922","cluster_addr":"172.21.15.142:6803/973693922","heartbeat_back_addr":"172.21.15.142:6807/973693922","heartbeat_front_addr":"172.21.15.142:6805/973693922","state":["exists","up"]},{"osd":2,"uuid":"c99a8794-ab19-4260-9a8d-4bc0bd2c8175","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6800","nonce":4084261813},{"type":"v1","addr":"172.21.15.162:6801","nonce":4084261813}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6802","nonce":4084261813},{"type":"v1","addr":"172.21.15.162:6803","nonce":4084261813}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6806","nonce":4084261813},{"type":"v1","addr":"172.21.15.162:6807","nonce":4084261813}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6804","nonce":4084261813},{"type":"v1","addr":"172.21.15.162:6805","nonce":4084261813}]},"public_addr":"172.21.15.162:6801/4084261813","cluster_addr":"172.21.15.162:6803/4084261813","heartbeat_back_addr":"172.21.15.162:6807/4084261813","heartbeat_front_addr":"172.21.15.162:6805/4084261813","state":["exists","up"]},{"osd":3,"uuid":"f877814e-9ef6-4872-a081-5609bb9ebc06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":3496317970},{"type":"v1","addr":"172.21.15.46:6811","nonce":3496317970}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":3496317970},{"type":"v1","addr":"172.21.15.46:6813","nonce":3496317970}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":3496317970},{"type":"v1","addr":"172.21.15.46:6817","nonce":3496317970}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":3496317970},{"type":"v1","addr":"172.21.15.46:6815","nonce":3496317970}]},"public_addr":"172.21.15.46:6811/3496317970","cluster_addr":"172.21.15.46:6813/3496317970","heartbeat_back_addr":"172.21.15.46:6817/3496317970","heartbeat_front_addr":"172.21.15.46:6815/3496317970","state":["exists","up"]},{"osd":4,"uuid":"beecec0a-8280-44fe-8f5f-8498a35eca8f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6808","nonce":2996266929},{"type":"v1","addr":"172.21.15.142:6809","nonce":2996266929}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6810","nonce":2996266929},{"type":"v1","addr":"172.21.15.142:6811","nonce":2996266929}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6814","nonce":2996266929},{"type":"v1","addr":"172.21.15.142:6815","nonce":2996266929}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6812","nonce":2996266929},{"type":"v1","addr":"172.21.15.142:6813","nonce":2996266929}]},"public_addr":"172.21.15.142:6809/2996266929","cluster_addr":"172.21.15.142:6811/2996266929","heartbeat_back_addr":"172.21.15.142:6815/2996266929","heartbeat_front_addr":"172.21.15.142:6813/2996266929","state":["exists","up"]},{"osd":5,"uuid":"9da7ff75-f658-45ea-a02f-1943fdda76b9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6808","nonce":914208293},{"type":"v1","addr":"172.21.15.162:6809","nonce":914208293}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6810","nonce":914208293},{"type":"v1","addr":"172.21.15.162:6811","nonce":914208293}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6814","nonce":914208293},{"type":"v1","addr":"172.21.15.162:6815","nonce":914208293}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6812","nonce":914208293},{"type":"v1","addr":"172.21.15.162:6813","nonce":914208293}]},"public_addr":"172.21.15.162:6809/914208293","cluster_addr":"172.21.15.162:6811/914208293","heartbeat_back_addr":"172.21.15.162:6815/914208293","heartbeat_front_addr":"172.21.15.162:6813/914208293","state":["exists","up"]},{"osd":6,"uuid":"8fa11224-cb26-4d3f-87d1-6a8f31de2a19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":56291966},{"type":"v1","addr":"172.21.15.46:6819","nonce":56291966}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":56291966},{"type":"v1","addr":"172.21.15.46:6821","nonce":56291966}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6824","nonce":56291966},{"type":"v1","addr":"172.21.15.46:6825","nonce":56291966}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":56291966},{"type":"v1","addr":"172.21.15.46:6823","nonce":56291966}]},"public_addr":"172.21.15.46:6819/56291966","cluster_addr":"172.21.15.46:6821/56291966","heartbeat_back_addr":"172.21.15.46:6825/56291966","heartbeat_front_addr":"172.21.15.46:6823/56291966","state":["exists","up"]},{"osd":7,"uuid":"0d0338b7-4d0a-4bb0-b152-8f5224871d9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6816","nonce":2146598446},{"type":"v1","addr":"172.21.15.142:6817","nonce":2146598446}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6818","nonce":2146598446},{"type":"v1","addr":"172.21.15.142:6819","nonce":2146598446}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6822","nonce":2146598446},{"type":"v1","addr":"172.21.15.142:6823","nonce":2146598446}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6820","nonce":2146598446},{"type":"v1","addr":"172.21.15.142:6821","nonce":2146598446}]},"public_addr":"172.21.15.142:6817/2146598446","cluster_addr":"172.21.15.142:6819/2146598446","heartbeat_back_addr":"172.21.15.142:6823/2146598446","heartbeat_front_addr":"172.21.15.142:6821/2146598446","state":["exists","up"]},{"osd":8,"uuid":"8b6f6037-892a-416c-babb-7552648d0b3a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6816","nonce":1173078318},{"type":"v1","addr":"172.21.15.162:6817","nonce":1173078318}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6818","nonce":1173078318},{"type":"v1","addr":"172.21.15.162:6819","nonce":1173078318}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6822","nonce":1173078318},{"type":"v1","addr":"172.21.15.162:6823","nonce":1173078318}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6820","nonce":1173078318},{"type":"v1","addr":"172.21.15.162:6821","nonce":1173078318}]},"public_addr":"172.21.15.162:6817/1173078318","cluster_addr":"172.21.15.162:6819/1173078318","heartbeat_back_addr":"172.21.15.162:6823/1173078318","heartbeat_front_addr":"172.21.15.162:6821/1173078318","state":["exists","up"]},{"osd":9,"uuid":"faccce25-6bf2-4220-8858-cda82b72be68","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":532708881},{"type":"v1","addr":"172.21.15.46:6827","nonce":532708881}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6828","nonce":532708881},{"type":"v1","addr":"172.21.15.46:6829","nonce":532708881}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":532708881},{"type":"v1","addr":"172.21.15.46:6833","nonce":532708881}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":532708881},{"type":"v1","addr":"172.21.15.46:6831","nonce":532708881}]},"public_addr":"172.21.15.46:6827/532708881","cluster_addr":"172.21.15.46:6829/532708881","heartbeat_back_addr":"172.21.15.46:6833/532708881","heartbeat_front_addr":"172.21.15.46:6831/532708881","state":["exists","up"]},{"osd":10,"uuid":"cdd356db-f9eb-4080-95ab-db5d8223e572","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6824","nonce":2845222366},{"type":"v1","addr":"172.21.15.142:6825","nonce":2845222366}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6826","nonce":2845222366},{"type":"v1","addr":"172.21.15.142:6827","nonce":2845222366}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6830","nonce":2845222366},{"type":"v1","addr":"172.21.15.142:6831","nonce":2845222366}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6828","nonce":2845222366},{"type":"v1","addr":"172.21.15.142:6829","nonce":2845222366}]},"public_addr":"172.21.15.142:6825/2845222366","cluster_addr":"172.21.15.142:6827/2845222366","heartbeat_back_addr":"172.21.15.142:6831/2845222366","heartbeat_front_addr":"172.21.15.142:6829/2845222366","state":["exists","up"]},{"osd":11,"uuid":"72104d1c-3ea2-45ad-bc0b-0500f146c06b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6824","nonce":1986323778},{"type":"v1","addr":"172.21.15.162:6825","nonce":1986323778}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6826","nonce":1986323778},{"type":"v1","addr":"172.21.15.162:6827","nonce":1986323778}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6830","nonce":1986323778},{"type":"v1","addr":"172.21.15.162:6831","nonce":1986323778}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6828","nonce":1986323778},{"type":"v1","addr":"172.21.15.162:6829","nonce":1986323778}]},"public_addr":"172.21.15.162:6825/1986323778","cluster_addr":"172.21.15.162:6827/1986323778","heartbeat_back_addr":"172.21.15.162:6831/1986323778","heartbeat_front_addr":"172.21.15.162:6829/1986323778","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:03:44.996331+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:04:07.994738+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:04:30.942255+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:04:55.588884+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:05:19.900600+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:05:43.807442+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:06:09.026976+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:06:34.586784+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:06:59.751605+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:07:26.144495+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:07:52.326909+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:08:17.423042+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.46:0/4021039857":"2024-04-01T23:02:08.493619+0000","172.21.15.46:6801/3681182647":"2024-04-01T23:02:08.493619+0000","172.21.15.46:0/4022214806":"2024-04-01T23:02:08.493619+0000","172.21.15.46:6800/1515528215":"2024-04-01T23:01:41.910162+0000","172.21.15.46:6800/3681182647":"2024-04-01T23:02:08.493619+0000","172.21.15.46:0/143885707":"2024-04-01T23:01:41.910162+0000","172.21.15.46:6801/1515528215":"2024-04-01T23:01:41.910162+0000","172.21.15.46:0/1680389778":"2024-04-01T23:01:41.910162+0000","172.21.15.46:0/1782133966":"2024-04-01T23:01:41.910162+0000","172.21.15.46:0/3925209376":"2024-04-01T23:02:08.493619+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-31T23:08:26.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:26.720+0000 7fa4482d7700 1 -- 172.21.15.46:0/2460664332 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fa42c0a25b0 msgr2=0x7fa42c0a4a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:08:26.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:26.720+0000 7fa4482d7700 1 --2- 172.21.15.46:0/2460664332 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fa42c0a25b0 0x7fa42c0a4a70 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fa430005fd0 tx=0x7fa430005db0 comp rx=0 tx=0).stop 2024-03-31T23:08:26.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:26.720+0000 7fa4482d7700 1 -- 172.21.15.46:0/2460664332 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa440106170 msgr2=0x7fa4401038f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:08:26.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:26.720+0000 7fa4482d7700 1 --2- 172.21.15.46:0/2460664332 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa440106170 0x7fa4401038f0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fa438007ee0 tx=0x7fa43800d3b0 comp rx=0 tx=0).stop 2024-03-31T23:08:26.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:26.720+0000 7fa4482d7700 1 -- 172.21.15.46:0/2460664332 shutdown_connections 2024-03-31T23:08:26.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:26.720+0000 7fa4482d7700 1 --2- 172.21.15.46:0/2460664332 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fa42c0a25b0 0x7fa42c0a4a70 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:08:26.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:26.720+0000 7fa4482d7700 1 --2- 172.21.15.46:0/2460664332 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa440106170 0x7fa4401038f0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:08:26.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:26.720+0000 7fa4482d7700 1 --2- 172.21.15.46:0/2460664332 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa440101c80 0x7fa440102e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:08:26.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:26.720+0000 7fa4482d7700 1 --2- 172.21.15.46:0/2460664332 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa440102640 0x7fa4401033b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:08:26.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:26.720+0000 7fa4482d7700 1 -- 172.21.15.46:0/2460664332 >> 172.21.15.46:0/2460664332 conn(0x7fa440076470 msgr2=0x7fa440100d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:08:26.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:26.720+0000 7fa4482d7700 1 -- 172.21.15.46:0/2460664332 shutdown_connections 2024-03-31T23:08:26.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:26.720+0000 7fa4482d7700 1 -- 172.21.15.46:0/2460664332 wait complete. 2024-03-31T23:08:26.795 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:08:26 smithi162 ceph-mon[36237]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-31T23:08:26.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:08:26 smithi142 ceph-mon[35271]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-31T23:08:27.502 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-31T23:04:32.823751+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-31T23:08:27.503 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:384f31f3d3aee7f8ba3f60846b8844c049165583 shell --fsid 6a9d8f08-efb2-11ee-b647-cb9ed24678a4 -- ceph osd pool get .mgr pg_num 2024-03-31T23:08:27.747 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/6a9d8f08-efb2-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-31T23:08:27.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:08:27 smithi162 ceph-mon[36237]: from='client.? 172.21.15.46:0/2460664332' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-31T23:08:27.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:08:27 smithi046 ceph-mon[32498]: from='client.? 172.21.15.46:0/2460664332' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-31T23:08:27.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:08:27 smithi142 ceph-mon[35271]: from='client.? 172.21.15.46:0/2460664332' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-31T23:08:28.795 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:08:28 smithi162 ceph-mon[36237]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-31T23:08:28.846 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:08:28 smithi046 ceph-mon[32498]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-31T23:08:28.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:08:28 smithi142 ceph-mon[35271]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-31T23:08:29.384 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:29.379+0000 7fcdc387a700 1 -- 172.21.15.46:0/764973594 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcdbc107550 msgr2=0x7fcdbc10f120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:08:29.384 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:29.379+0000 7fcdc387a700 1 --2- 172.21.15.46:0/764973594 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcdbc107550 0x7fcdbc10f120 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fcdb4009990 tx=0x7fcdb4009ca0 comp rx=0 tx=0).stop 2024-03-31T23:08:29.384 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:29.379+0000 7fcdc387a700 1 -- 172.21.15.46:0/764973594 shutdown_connections 2024-03-31T23:08:29.384 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:29.379+0000 7fcdc387a700 1 --2- 172.21.15.46:0/764973594 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcdbc107550 0x7fcdbc10f120 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:08:29.384 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:29.379+0000 7fcdc387a700 1 --2- 172.21.15.46:0/764973594 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcdbc102520 0x7fcdbc102900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:08:29.384 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:29.379+0000 7fcdc387a700 1 --2- 172.21.15.46:0/764973594 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcdbc102ed0 0x7fcdbc106e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:08:29.384 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:29.379+0000 7fcdc387a700 1 -- 172.21.15.46:0/764973594 >> 172.21.15.46:0/764973594 conn(0x7fcdbc0792d0 msgr2=0x7fcdbc0796e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:08:29.384 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:29.379+0000 7fcdc387a700 1 -- 172.21.15.46:0/764973594 shutdown_connections 2024-03-31T23:08:29.384 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:29.379+0000 7fcdc387a700 1 -- 172.21.15.46:0/764973594 wait complete. 2024-03-31T23:08:29.384 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:29.380+0000 7fcdc387a700 1 Processor -- start 2024-03-31T23:08:29.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:29.380+0000 7fcdc387a700 1 -- start start 2024-03-31T23:08:29.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:29.380+0000 7fcdc387a700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcdbc102520 0x7fcdbc074540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:08:29.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:29.380+0000 7fcdc387a700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcdbc102ed0 0x7fcdbc074a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:08:29.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:29.380+0000 7fcdc387a700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcdbc107550 0x7fcdbc074fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:08:29.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:29.380+0000 7fcdc387a700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fcdbc1123e0 con 0x7fcdbc102ed0 2024-03-31T23:08:29.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:29.380+0000 7fcdc387a700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fcdbc112260 con 0x7fcdbc107550 2024-03-31T23:08:29.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:29.380+0000 7fcdc387a700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fcdbc112560 con 0x7fcdbc102520 2024-03-31T23:08:29.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:29.380+0000 7fcdc0e15700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcdbc102ed0 0x7fcdbc074a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:08:29.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:29.380+0000 7fcdc0e15700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcdbc102ed0 0x7fcdbc074a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44426/0 (socket says 172.21.15.46:44426) 2024-03-31T23:08:29.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:29.380+0000 7fcdc0e15700 1 -- 172.21.15.46:0/2500990899 learned_addr learned my addr 172.21.15.46:0/2500990899 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:08:29.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:29.380+0000 7fcdc1e17700 1 --2- 172.21.15.46:0/2500990899 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcdbc107550 0x7fcdbc074fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:08:29.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:29.380+0000 7fcdc1616700 1 --2- 172.21.15.46:0/2500990899 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcdbc102520 0x7fcdbc074540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:08:29.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:29.380+0000 7fcdc0e15700 1 -- 172.21.15.46:0/2500990899 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcdbc102520 msgr2=0x7fcdbc074540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:08:29.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:29.380+0000 7fcdc0e15700 1 --2- 172.21.15.46:0/2500990899 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcdbc102520 0x7fcdbc074540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:08:29.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:29.380+0000 7fcdc0e15700 1 -- 172.21.15.46:0/2500990899 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcdbc107550 msgr2=0x7fcdbc074fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:08:29.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:29.380+0000 7fcdc0e15700 1 --2- 172.21.15.46:0/2500990899 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcdbc107550 0x7fcdbc074fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:08:29.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:29.380+0000 7fcdc0e15700 1 -- 172.21.15.46:0/2500990899 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcdbc1a6c50 con 0x7fcdbc102ed0 2024-03-31T23:08:29.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:29.380+0000 7fcdc1e17700 1 --2- 172.21.15.46:0/2500990899 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcdbc107550 0x7fcdbc074fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T23:08:29.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:29.380+0000 7fcdc0e15700 1 --2- 172.21.15.46:0/2500990899 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcdbc102ed0 0x7fcdbc074a80 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fcdb000c9e0 tx=0x7fcdb000cda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:08:29.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:29.380+0000 7fcdba7fc700 1 -- 172.21.15.46:0/2500990899 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdb0007aa0 con 0x7fcdbc102ed0 2024-03-31T23:08:29.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:29.381+0000 7fcdc387a700 1 -- 172.21.15.46:0/2500990899 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcdbc1a6f40 con 0x7fcdbc102ed0 2024-03-31T23:08:29.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:29.381+0000 7fcdba7fc700 1 -- 172.21.15.46:0/2500990899 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fcdb0007c40 con 0x7fcdbc102ed0 2024-03-31T23:08:29.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:29.381+0000 7fcdba7fc700 1 -- 172.21.15.46:0/2500990899 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdb0018660 con 0x7fcdbc102ed0 2024-03-31T23:08:29.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:29.381+0000 7fcdc387a700 1 -- 172.21.15.46:0/2500990899 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcdbc1a73f0 con 0x7fcdbc102ed0 2024-03-31T23:08:29.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:29.382+0000 7fcdc387a700 1 -- 172.21.15.46:0/2500990899 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcdbc068f00 con 0x7fcdbc102ed0 2024-03-31T23:08:29.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:29.383+0000 7fcdba7fc700 1 -- 172.21.15.46:0/2500990899 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 16) ==== 137320+0+0 (secure 0 0 0) 0x7fcdb000f450 con 0x7fcdbc102ed0 2024-03-31T23:08:29.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:29.383+0000 7fcdba7fc700 1 --2- 172.21.15.46:0/2500990899 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fcda40a2530 0x7fcda40a49f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:08:29.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:29.383+0000 7fcdc1616700 1 --2- 172.21.15.46:0/2500990899 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fcda40a2530 0x7fcda40a49f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:08:29.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:29.383+0000 7fcdba7fc700 1 -- 172.21.15.46:0/2500990899 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7fcdb00ce360 con 0x7fcdbc102ed0 2024-03-31T23:08:29.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:29.384+0000 7fcdc1616700 1 --2- 172.21.15.46:0/2500990899 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fcda40a2530 0x7fcda40a49f0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fcdac00d260 tx=0x7fcdac009040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:08:29.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:29.387+0000 7fcdba7fc700 1 -- 172.21.15.46:0/2500990899 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fcdb0014030 con 0x7fcdbc102ed0 2024-03-31T23:08:29.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:29.592+0000 7fcdc387a700 1 -- 172.21.15.46:0/2500990899 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fcdbc02dbd0 con 0x7fcdbc102ed0 2024-03-31T23:08:29.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:29.593+0000 7fcdba7fc700 1 -- 172.21.15.46:0/2500990899 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v82) ==== 93+0+10 (secure 0 0 0) 0x7fcdb009b9c0 con 0x7fcdbc102ed0 2024-03-31T23:08:29.594 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 1 2024-03-31T23:08:29.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:29.595+0000 7fcdc387a700 1 -- 172.21.15.46:0/2500990899 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fcda40a2530 msgr2=0x7fcda40a49f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:08:29.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:29.595+0000 7fcdc387a700 1 --2- 172.21.15.46:0/2500990899 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fcda40a2530 0x7fcda40a49f0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fcdac00d260 tx=0x7fcdac009040 comp rx=0 tx=0).stop 2024-03-31T23:08:29.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:29.595+0000 7fcdc387a700 1 -- 172.21.15.46:0/2500990899 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcdbc102ed0 msgr2=0x7fcdbc074a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:08:29.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:29.595+0000 7fcdc387a700 1 --2- 172.21.15.46:0/2500990899 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcdbc102ed0 0x7fcdbc074a80 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fcdb000c9e0 tx=0x7fcdb000cda0 comp rx=0 tx=0).stop 2024-03-31T23:08:29.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:29.595+0000 7fcdc387a700 1 -- 172.21.15.46:0/2500990899 shutdown_connections 2024-03-31T23:08:29.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:29.595+0000 7fcdc387a700 1 --2- 172.21.15.46:0/2500990899 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fcda40a2530 0x7fcda40a49f0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:08:29.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:29.595+0000 7fcdc387a700 1 --2- 172.21.15.46:0/2500990899 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcdbc107550 0x7fcdbc074fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:08:29.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:29.595+0000 7fcdc387a700 1 --2- 172.21.15.46:0/2500990899 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcdbc102520 0x7fcdbc074540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:08:29.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:29.595+0000 7fcdc387a700 1 --2- 172.21.15.46:0/2500990899 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcdbc102ed0 0x7fcdbc074a80 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:08:29.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:29.595+0000 7fcdc387a700 1 -- 172.21.15.46:0/2500990899 >> 172.21.15.46:0/2500990899 conn(0x7fcdbc0792d0 msgr2=0x7fcdbc100270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:08:29.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:29.595+0000 7fcdc387a700 1 -- 172.21.15.46:0/2500990899 shutdown_connections 2024-03-31T23:08:29.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:29.595+0000 7fcdc387a700 1 -- 172.21.15.46:0/2500990899 wait complete. 2024-03-31T23:08:30.339 INFO:tasks.cephadm:Adding mds.a on smithi046 2024-03-31T23:08:30.340 INFO:tasks.cephadm:Adding mds.d on smithi046 2024-03-31T23:08:30.340 INFO:tasks.cephadm:Adding mds.g on smithi046 2024-03-31T23:08:30.340 INFO:tasks.cephadm:Adding mds.j on smithi046 2024-03-31T23:08:30.340 INFO:tasks.cephadm:Adding mds.b on smithi142 2024-03-31T23:08:30.340 INFO:tasks.cephadm:Adding mds.e on smithi142 2024-03-31T23:08:30.340 INFO:tasks.cephadm:Adding mds.h on smithi142 2024-03-31T23:08:30.340 INFO:tasks.cephadm:Adding mds.k on smithi142 2024-03-31T23:08:30.340 INFO:tasks.cephadm:Adding mds.c on smithi162 2024-03-31T23:08:30.340 INFO:tasks.cephadm:Adding mds.f on smithi162 2024-03-31T23:08:30.340 INFO:tasks.cephadm:Adding mds.i on smithi162 2024-03-31T23:08:30.340 INFO:tasks.cephadm:Adding mds.l on smithi162 2024-03-31T23:08:30.341 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:384f31f3d3aee7f8ba3f60846b8844c049165583 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6a9d8f08-efb2-11ee-b647-cb9ed24678a4 -- ceph orch apply mds all '12;smithi046=a;smithi046=d;smithi046=g;smithi046=j;smithi142=b;smithi142=e;smithi142=h;smithi142=k;smithi162=c;smithi162=f;smithi162=i;smithi162=l' 2024-03-31T23:08:30.593 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/6a9d8f08-efb2-11ee-b647-cb9ed24678a4/mon.c/config 2024-03-31T23:08:30.611 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:08:30 smithi046 ceph-mon[32498]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-31T23:08:30.611 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:08:30 smithi046 ceph-mon[32498]: from='client.? 172.21.15.46:0/2500990899' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-31T23:08:30.693 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:08:30 smithi142 ceph-mon[35271]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-31T23:08:30.693 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:08:30 smithi142 ceph-mon[35271]: from='client.? 172.21.15.46:0/2500990899' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-31T23:08:30.716 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:08:30 smithi162 ceph-mon[36237]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-31T23:08:30.716 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:08:30 smithi162 ceph-mon[36237]: from='client.? 172.21.15.46:0/2500990899' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-31T23:08:32.206 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:08:32.202+0000 7f2b85809700 1 -- 172.21.15.162:0/3131112847 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2b80103460 msgr2=0x7f2b801038c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:08:32.206 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:08:32.202+0000 7f2b85809700 1 --2- 172.21.15.162:0/3131112847 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2b80103460 0x7f2b801038c0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f2b74009990 tx=0x7f2b74009ca0 comp rx=0 tx=0).stop 2024-03-31T23:08:32.206 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:08:32.202+0000 7f2b85809700 1 -- 172.21.15.162:0/3131112847 shutdown_connections 2024-03-31T23:08:32.206 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:08:32.202+0000 7f2b85809700 1 --2- 172.21.15.162:0/3131112847 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2b80103e00 0x7f2b8010a6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:08:32.206 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:08:32.202+0000 7f2b85809700 1 --2- 172.21.15.162:0/3131112847 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2b80102260 0x7f2b80102680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:08:32.206 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:08:32.202+0000 7f2b85809700 1 --2- 172.21.15.162:0/3131112847 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2b80103460 0x7f2b801038c0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:08:32.207 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:08:32.202+0000 7f2b85809700 1 -- 172.21.15.162:0/3131112847 >> 172.21.15.162:0/3131112847 conn(0x7f2b800fd820 msgr2=0x7f2b800ffc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:08:32.207 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:08:32.202+0000 7f2b85809700 1 -- 172.21.15.162:0/3131112847 shutdown_connections 2024-03-31T23:08:32.207 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:08:32.202+0000 7f2b85809700 1 -- 172.21.15.162:0/3131112847 wait complete. 2024-03-31T23:08:32.207 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:08:32.203+0000 7f2b85809700 1 Processor -- start 2024-03-31T23:08:32.207 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:08:32.203+0000 7f2b85809700 1 -- start start 2024-03-31T23:08:32.207 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:08:32.203+0000 7f2b85809700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2b80102260 0x7f2b8019a520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:08:32.207 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:08:32.203+0000 7f2b85809700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2b80103460 0x7f2b8019aa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:08:32.207 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:08:32.203+0000 7f2b85809700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2b80103e00 0x7f2b801a1ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:08:32.207 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:08:32.203+0000 7f2b85809700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f2b8010dd00 con 0x7f2b80103e00 2024-03-31T23:08:32.207 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:08:32.203+0000 7f2b85809700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f2b8010db80 con 0x7f2b80102260 2024-03-31T23:08:32.207 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:08:32.203+0000 7f2b85809700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f2b8010de80 con 0x7f2b80103460 2024-03-31T23:08:32.207 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:08:32.203+0000 7f2b7e7fc700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2b80103460 0x7f2b8019aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:08:32.207 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:08:32.203+0000 7f2b7e7fc700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2b80103460 0x7f2b8019aa60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:42048/0 (socket says 172.21.15.162:42048) 2024-03-31T23:08:32.208 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:08:32.203+0000 7f2b7e7fc700 1 -- 172.21.15.162:0/3310561234 learned_addr learned my addr 172.21.15.162:0/3310561234 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-03-31T23:08:32.208 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:08:32.204+0000 7f2b7effd700 1 --2- 172.21.15.162:0/3310561234 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2b80102260 0x7f2b8019a520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:08:32.208 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:08:32.204+0000 7f2b7f7fe700 1 --2- 172.21.15.162:0/3310561234 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2b80103e00 0x7f2b801a1ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:08:32.208 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:08:32.204+0000 7f2b7e7fc700 1 -- 172.21.15.162:0/3310561234 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2b80102260 msgr2=0x7f2b8019a520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:08:32.208 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:08:32.204+0000 7f2b7e7fc700 1 --2- 172.21.15.162:0/3310561234 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2b80102260 0x7f2b8019a520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:08:32.208 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:08:32.204+0000 7f2b7e7fc700 1 -- 172.21.15.162:0/3310561234 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2b80103e00 msgr2=0x7f2b801a1ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:08:32.208 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:08:32.204+0000 7f2b7e7fc700 1 --2- 172.21.15.162:0/3310561234 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2b80103e00 0x7f2b801a1ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:08:32.208 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:08:32.204+0000 7f2b7e7fc700 1 -- 172.21.15.162:0/3310561234 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b801a2180 con 0x7f2b80103460 2024-03-31T23:08:32.208 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:08:32.204+0000 7f2b7f7fe700 1 --2- 172.21.15.162:0/3310561234 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2b80103e00 0x7f2b801a1ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T23:08:32.208 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:08:32.204+0000 7f2b7e7fc700 1 --2- 172.21.15.162:0/3310561234 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2b80103460 0x7f2b8019aa60 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f2b74000c00 tx=0x7f2b7400bc70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:08:32.208 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:08:32.204+0000 7f2b84807700 1 -- 172.21.15.162:0/3310561234 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b74026070 con 0x7f2b80103460 2024-03-31T23:08:32.208 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:08:32.204+0000 7f2b84807700 1 -- 172.21.15.162:0/3310561234 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f2b7400be50 con 0x7f2b80103460 2024-03-31T23:08:32.208 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:08:32.204+0000 7f2b85809700 1 -- 172.21.15.162:0/3310561234 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2b801a2410 con 0x7f2b80103460 2024-03-31T23:08:32.208 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:08:32.204+0000 7f2b84807700 1 -- 172.21.15.162:0/3310561234 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b74021670 con 0x7f2b80103460 2024-03-31T23:08:32.209 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:08:32.204+0000 7f2b85809700 1 -- 172.21.15.162:0/3310561234 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2b801a2860 con 0x7f2b80103460 2024-03-31T23:08:32.210 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:08:32.205+0000 7f2b85809700 1 -- 172.21.15.162:0/3310561234 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2b80068f00 con 0x7f2b80103460 2024-03-31T23:08:32.210 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:08:32.206+0000 7f2b84807700 1 -- 172.21.15.162:0/3310561234 <== mon.1 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 16) ==== 137320+0+0 (secure 0 0 0) 0x7f2b74029070 con 0x7f2b80103460 2024-03-31T23:08:32.210 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:08:32.207+0000 7f2b84807700 1 --2- 172.21.15.162:0/3310561234 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f2b680a2530 0x7f2b680a49f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:08:32.210 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:08:32.207+0000 7f2b84807700 1 -- 172.21.15.162:0/3310561234 <== mon.1 v2:172.21.15.162:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f2b740cf110 con 0x7f2b80103460 2024-03-31T23:08:32.210 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:08:32.207+0000 7f2b7effd700 1 --2- 172.21.15.162:0/3310561234 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f2b680a2530 0x7f2b680a49f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:08:32.210 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:08:32.208+0000 7f2b7effd700 1 --2- 172.21.15.162:0/3310561234 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f2b680a2530 0x7f2b680a49f0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f2b801032c0 tx=0x7f2b70009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:08:32.213 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:08:32.211+0000 7f2b84807700 1 -- 172.21.15.162:0/3310561234 <== mon.1 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f2b74013070 con 0x7f2b80103460 2024-03-31T23:08:32.423 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:08:32.422+0000 7f2b85809700 1 -- 172.21.15.162:0/3310561234 --> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] -- mgr_command(tid 0: {"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi046=a;smithi046=d;smithi046=g;smithi046=j;smithi142=b;smithi142=e;smithi142=h;smithi142=k;smithi162=c;smithi162=f;smithi162=i;smithi162=l", "target": ["mon-mgr", ""]}) -- 0x7f2b80075c90 con 0x7f2b680a2530 2024-03-31T23:08:32.431 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:08:32.430+0000 7f2b84807700 1 -- 172.21.15.162:0/3310561234 <== mgr.14152 v2:172.21.15.46:6800/162289569 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+28 (secure 0 0 0) 0x7f2b80075c90 con 0x7f2b680a2530 2024-03-31T23:08:32.431 INFO:teuthology.orchestra.run.smithi162.stdout:Scheduled mds.all update... 2024-03-31T23:08:32.434 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:08:32.432+0000 7f2b85809700 1 -- 172.21.15.162:0/3310561234 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f2b680a2530 msgr2=0x7f2b680a49f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:08:32.434 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:08:32.432+0000 7f2b85809700 1 --2- 172.21.15.162:0/3310561234 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f2b680a2530 0x7f2b680a49f0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f2b801032c0 tx=0x7f2b70009260 comp rx=0 tx=0).stop 2024-03-31T23:08:32.434 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:08:32.432+0000 7f2b85809700 1 -- 172.21.15.162:0/3310561234 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2b80103460 msgr2=0x7f2b8019aa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:08:32.434 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:08:32.432+0000 7f2b85809700 1 --2- 172.21.15.162:0/3310561234 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2b80103460 0x7f2b8019aa60 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f2b74000c00 tx=0x7f2b7400bc70 comp rx=0 tx=0).stop 2024-03-31T23:08:32.434 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:08:32.432+0000 7f2b85809700 1 -- 172.21.15.162:0/3310561234 shutdown_connections 2024-03-31T23:08:32.434 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:08:32.432+0000 7f2b85809700 1 --2- 172.21.15.162:0/3310561234 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f2b680a2530 0x7f2b680a49f0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:08:32.434 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:08:32.432+0000 7f2b85809700 1 --2- 172.21.15.162:0/3310561234 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2b80103e00 0x7f2b801a1ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:08:32.434 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:08:32.432+0000 7f2b85809700 1 --2- 172.21.15.162:0/3310561234 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2b80102260 0x7f2b8019a520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:08:32.434 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:08:32.432+0000 7f2b85809700 1 --2- 172.21.15.162:0/3310561234 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2b80103460 0x7f2b8019aa60 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:08:32.434 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:08:32.432+0000 7f2b85809700 1 -- 172.21.15.162:0/3310561234 >> 172.21.15.162:0/3310561234 conn(0x7f2b800fd820 msgr2=0x7f2b80074470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:08:32.434 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:08:32.432+0000 7f2b85809700 1 -- 172.21.15.162:0/3310561234 shutdown_connections 2024-03-31T23:08:32.434 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-31T23:08:32.432+0000 7f2b85809700 1 -- 172.21.15.162:0/3310561234 wait complete. 2024-03-31T23:08:32.795 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:08:32 smithi162 ceph-mon[36237]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-31T23:08:32.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:08:32 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:08:32.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:08:32 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-31T23:08:32.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:08:32 smithi046 ceph-mon[32498]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-31T23:08:32.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:08:32 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:08:32.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:08:32 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-31T23:08:32.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:08:32 smithi142 ceph-mon[35271]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-31T23:08:32.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:08:32 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:08:32.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:08:32 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-31T23:08:33.055 DEBUG:teuthology.orchestra.run.smithi046:mds.a> sudo journalctl -f -n 0 -u ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4@mds.a.service 2024-03-31T23:08:33.058 DEBUG:teuthology.orchestra.run.smithi046:mds.d> sudo journalctl -f -n 0 -u ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4@mds.d.service 2024-03-31T23:08:33.100 DEBUG:teuthology.orchestra.run.smithi046:mds.g> sudo journalctl -f -n 0 -u ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4@mds.g.service 2024-03-31T23:08:33.144 DEBUG:teuthology.orchestra.run.smithi046:mds.j> sudo journalctl -f -n 0 -u ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4@mds.j.service 2024-03-31T23:08:33.187 DEBUG:teuthology.orchestra.run.smithi142:mds.b> sudo journalctl -f -n 0 -u ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4@mds.b.service 2024-03-31T23:08:33.238 DEBUG:teuthology.orchestra.run.smithi142:mds.e> sudo journalctl -f -n 0 -u ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4@mds.e.service 2024-03-31T23:08:33.281 DEBUG:teuthology.orchestra.run.smithi142:mds.h> sudo journalctl -f -n 0 -u ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4@mds.h.service 2024-03-31T23:08:33.324 DEBUG:teuthology.orchestra.run.smithi142:mds.k> sudo journalctl -f -n 0 -u ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4@mds.k.service 2024-03-31T23:08:33.367 DEBUG:teuthology.orchestra.run.smithi162:mds.c> sudo journalctl -f -n 0 -u ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4@mds.c.service 2024-03-31T23:08:33.370 DEBUG:teuthology.orchestra.run.smithi162:mds.f> sudo journalctl -f -n 0 -u ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4@mds.f.service 2024-03-31T23:08:33.413 DEBUG:teuthology.orchestra.run.smithi162:mds.i> sudo journalctl -f -n 0 -u ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4@mds.i.service 2024-03-31T23:08:33.456 DEBUG:teuthology.orchestra.run.smithi162:mds.l> sudo journalctl -f -n 0 -u ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4@mds.l.service 2024-03-31T23:08:33.499 INFO:tasks.cephadm:Setting up CephFS filesystem(s)... 2024-03-31T23:08:33.500 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-31T23:08:33.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:33.612+0000 7faeb35f2640 1 -- 172.21.15.46:0/274595811 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faeac104b20 msgr2=0x7faeac106f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:08:33.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:33.612+0000 7faeb35f2640 1 --2- 172.21.15.46:0/274595811 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faeac104b20 0x7faeac106f10 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7faea0009900 tx=0x7faea002bc30 comp rx=0 tx=0).stop 2024-03-31T23:08:33.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:33.612+0000 7faeb35f2640 1 -- 172.21.15.46:0/274595811 shutdown_connections 2024-03-31T23:08:33.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:33.612+0000 7faeb35f2640 1 --2- 172.21.15.46:0/274595811 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7faeac107450 0x7faeac1098a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:08:33.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:33.612+0000 7faeb35f2640 1 --2- 172.21.15.46:0/274595811 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faeac104b20 0x7faeac106f10 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:08:33.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:33.612+0000 7faeb35f2640 1 --2- 172.21.15.46:0/274595811 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7faeac1021f0 0x7faeac1045e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:08:33.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:33.612+0000 7faeb35f2640 1 -- 172.21.15.46:0/274595811 >> 172.21.15.46:0/274595811 conn(0x7faeac0fbbe0 msgr2=0x7faeac0fe020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:08:33.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:33.612+0000 7faeb35f2640 1 -- 172.21.15.46:0/274595811 shutdown_connections 2024-03-31T23:08:33.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:33.612+0000 7faeb35f2640 1 -- 172.21.15.46:0/274595811 wait complete. 2024-03-31T23:08:33.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:33.612+0000 7faeb35f2640 1 Processor -- start 2024-03-31T23:08:33.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:33.612+0000 7faeb35f2640 1 -- start start 2024-03-31T23:08:33.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:33.612+0000 7faeb35f2640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faeac1021f0 0x7faeac1979d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:08:33.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:33.612+0000 7faeb35f2640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7faeac104b20 0x7faeac197f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:08:33.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:33.612+0000 7faeb35f2640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7faeac107450 0x7faeac19ef80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:08:33.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:33.612+0000 7faeb35f2640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7faeac10d2a0 con 0x7faeac1021f0 2024-03-31T23:08:33.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:33.612+0000 7faeb35f2640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7faeac10d120 con 0x7faeac107450 2024-03-31T23:08:33.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:33.612+0000 7faeb35f2640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7faeac10d420 con 0x7faeac104b20 2024-03-31T23:08:33.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:33.613+0000 7faeb138e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faeac1021f0 0x7faeac1979d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:08:33.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:33.613+0000 7faeb138e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faeac1021f0 0x7faeac1979d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:54278/0 (socket says 172.21.15.46:54278) 2024-03-31T23:08:33.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:33.613+0000 7faeb138e640 1 -- 172.21.15.46:0/3624970080 learned_addr learned my addr 172.21.15.46:0/3624970080 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:08:33.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:33.613+0000 7faeb0b8d640 1 --2- 172.21.15.46:0/3624970080 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7faeac104b20 0x7faeac197f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:08:33.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:33.613+0000 7faeb1b8f640 1 --2- 172.21.15.46:0/3624970080 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7faeac107450 0x7faeac19ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:08:33.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:33.613+0000 7faeb138e640 1 -- 172.21.15.46:0/3624970080 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7faeac104b20 msgr2=0x7faeac197f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:08:33.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:33.613+0000 7faeb138e640 1 --2- 172.21.15.46:0/3624970080 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7faeac104b20 0x7faeac197f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:08:33.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:33.613+0000 7faeb138e640 1 -- 172.21.15.46:0/3624970080 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7faeac107450 msgr2=0x7faeac19ef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:08:33.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:33.613+0000 7faeb138e640 1 --2- 172.21.15.46:0/3624970080 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7faeac107450 0x7faeac19ef80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:08:33.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:33.613+0000 7faeb138e640 1 -- 172.21.15.46:0/3624970080 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faeac19f5f0 con 0x7faeac1021f0 2024-03-31T23:08:33.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:33.613+0000 7faeb1b8f640 1 --2- 172.21.15.46:0/3624970080 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7faeac107450 0x7faeac19ef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T23:08:33.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:33.613+0000 7faeb138e640 1 --2- 172.21.15.46:0/3624970080 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faeac1021f0 0x7faeac1979d0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7faea400d950 tx=0x7faea400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:08:33.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:33.613+0000 7faeaa7fc640 1 -- 172.21.15.46:0/3624970080 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faea4017070 con 0x7faeac1021f0 2024-03-31T23:08:33.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:33.613+0000 7faeaa7fc640 1 -- 172.21.15.46:0/3624970080 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7faea4009c30 con 0x7faeac1021f0 2024-03-31T23:08:33.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:33.613+0000 7faeb35f2640 1 -- 172.21.15.46:0/3624970080 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faeac19f8e0 con 0x7faeac1021f0 2024-03-31T23:08:33.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:33.613+0000 7faeaa7fc640 1 -- 172.21.15.46:0/3624970080 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faea400f620 con 0x7faeac1021f0 2024-03-31T23:08:33.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:33.613+0000 7faeb35f2640 1 -- 172.21.15.46:0/3624970080 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faeac19fe50 con 0x7faeac1021f0 2024-03-31T23:08:33.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:33.614+0000 7faeb35f2640 1 -- 172.21.15.46:0/3624970080 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faeac0fd6d0 con 0x7faeac1021f0 2024-03-31T23:08:33.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:33.615+0000 7faeaa7fc640 1 -- 172.21.15.46:0/3624970080 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 16) ==== 137320+0+0 (secure 0 0 0) 0x7faea40109a0 con 0x7faeac1021f0 2024-03-31T23:08:33.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:33.616+0000 7faeaa7fc640 1 --2- 172.21.15.46:0/3624970080 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fae940a2520 0x7fae940a49e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:08:33.619 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:33.616+0000 7faeb0b8d640 1 --2- 172.21.15.46:0/3624970080 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fae940a2520 0x7fae940a49e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:08:33.619 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:33.616+0000 7faeaa7fc640 1 -- 172.21.15.46:0/3624970080 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7faea40cde80 con 0x7faeac1021f0 2024-03-31T23:08:33.619 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:33.616+0000 7faeb0b8d640 1 --2- 172.21.15.46:0/3624970080 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fae940a2520 0x7fae940a49e0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7faea0005ab0 tx=0x7faea0005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:08:33.620 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:33.620+0000 7faeaa7fc640 1 -- 172.21.15.46:0/3624970080 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7faea4009de0 con 0x7faeac1021f0 2024-03-31T23:08:33.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:08:33 smithi162 ceph-mon[36237]: from='client.24391 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi046=a;smithi046=d;smithi046=g;smithi046=j;smithi142=b;smithi142=e;smithi142=h;smithi142=k;smithi162=c;smithi162=f;smithi162=i;smithi162=l", "target": ["mon-mgr", ""]}]: dispatch 2024-03-31T23:08:33.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:08:33 smithi162 ceph-mon[36237]: Saving service mds.all spec with placement smithi046=a;smithi046=d;smithi046=g;smithi046=j;smithi142=b;smithi142=e;smithi142=h;smithi142=k;smithi162=c;smithi162=f;smithi162=i;smithi162=l;count:12 2024-03-31T23:08:33.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:08:33 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:08:33.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:08:33 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-31T23:08:33.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:08:33 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:08:33.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:08:33 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-31T23:08:33.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:08:33 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-31T23:08:33.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:08:33 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:08:33.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:33.806+0000 7faeb35f2640 1 -- 172.21.15.46:0/3624970080 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7faeac02d420 con 0x7faeac1021f0 2024-03-31T23:08:33.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:33.807+0000 7faeaa7fc640 1 -- 172.21.15.46:0/3624970080 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v82) ==== 74+0+18433 (secure 0 0 0) 0x7faea409b4e0 con 0x7faeac1021f0 2024-03-31T23:08:33.807 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-31T23:08:33.807 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":82,"fsid":"6a9d8f08-efb2-11ee-b647-cb9ed24678a4","created":"2024-03-31T23:01:15.463711+0000","modified":"2024-03-31T23:08:21.201474+0000","last_up_change":"2024-03-31T23:08:20.199888+0000","last_in_change":"2024-03-31T23:07:59.438035+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-31T23:04:32.823751+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"47429095-3f46-4ed7-b7c4-504464719815","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":1824772695},{"type":"v1","addr":"172.21.15.46:6803","nonce":1824772695}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":1824772695},{"type":"v1","addr":"172.21.15.46:6805","nonce":1824772695}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":1824772695},{"type":"v1","addr":"172.21.15.46:6809","nonce":1824772695}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":1824772695},{"type":"v1","addr":"172.21.15.46:6807","nonce":1824772695}]},"public_addr":"172.21.15.46:6803/1824772695","cluster_addr":"172.21.15.46:6805/1824772695","heartbeat_back_addr":"172.21.15.46:6809/1824772695","heartbeat_front_addr":"172.21.15.46:6807/1824772695","state":["exists","up"]},{"osd":1,"uuid":"be56f883-411e-4dbe-b50e-a158134df4a6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6800","nonce":973693922},{"type":"v1","addr":"172.21.15.142:6801","nonce":973693922}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6802","nonce":973693922},{"type":"v1","addr":"172.21.15.142:6803","nonce":973693922}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6806","nonce":973693922},{"type":"v1","addr":"172.21.15.142:6807","nonce":973693922}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6804","nonce":973693922},{"type":"v1","addr":"172.21.15.142:6805","nonce":973693922}]},"public_addr":"172.21.15.142:6801/973693922","cluster_addr":"172.21.15.142:6803/973693922","heartbeat_back_addr":"172.21.15.142:6807/973693922","heartbeat_front_addr":"172.21.15.142:6805/973693922","state":["exists","up"]},{"osd":2,"uuid":"c99a8794-ab19-4260-9a8d-4bc0bd2c8175","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6800","nonce":4084261813},{"type":"v1","addr":"172.21.15.162:6801","nonce":4084261813}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6802","nonce":4084261813},{"type":"v1","addr":"172.21.15.162:6803","nonce":4084261813}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6806","nonce":4084261813},{"type":"v1","addr":"172.21.15.162:6807","nonce":4084261813}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6804","nonce":4084261813},{"type":"v1","addr":"172.21.15.162:6805","nonce":4084261813}]},"public_addr":"172.21.15.162:6801/4084261813","cluster_addr":"172.21.15.162:6803/4084261813","heartbeat_back_addr":"172.21.15.162:6807/4084261813","heartbeat_front_addr":"172.21.15.162:6805/4084261813","state":["exists","up"]},{"osd":3,"uuid":"f877814e-9ef6-4872-a081-5609bb9ebc06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":3496317970},{"type":"v1","addr":"172.21.15.46:6811","nonce":3496317970}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":3496317970},{"type":"v1","addr":"172.21.15.46:6813","nonce":3496317970}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":3496317970},{"type":"v1","addr":"172.21.15.46:6817","nonce":3496317970}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":3496317970},{"type":"v1","addr":"172.21.15.46:6815","nonce":3496317970}]},"public_addr":"172.21.15.46:6811/3496317970","cluster_addr":"172.21.15.46:6813/3496317970","heartbeat_back_addr":"172.21.15.46:6817/3496317970","heartbeat_front_addr":"172.21.15.46:6815/3496317970","state":["exists","up"]},{"osd":4,"uuid":"beecec0a-8280-44fe-8f5f-8498a35eca8f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6808","nonce":2996266929},{"type":"v1","addr":"172.21.15.142:6809","nonce":2996266929}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6810","nonce":2996266929},{"type":"v1","addr":"172.21.15.142:6811","nonce":2996266929}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6814","nonce":2996266929},{"type":"v1","addr":"172.21.15.142:6815","nonce":2996266929}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6812","nonce":2996266929},{"type":"v1","addr":"172.21.15.142:6813","nonce":2996266929}]},"public_addr":"172.21.15.142:6809/2996266929","cluster_addr":"172.21.15.142:6811/2996266929","heartbeat_back_addr":"172.21.15.142:6815/2996266929","heartbeat_front_addr":"172.21.15.142:6813/2996266929","state":["exists","up"]},{"osd":5,"uuid":"9da7ff75-f658-45ea-a02f-1943fdda76b9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6808","nonce":914208293},{"type":"v1","addr":"172.21.15.162:6809","nonce":914208293}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6810","nonce":914208293},{"type":"v1","addr":"172.21.15.162:6811","nonce":914208293}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6814","nonce":914208293},{"type":"v1","addr":"172.21.15.162:6815","nonce":914208293}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6812","nonce":914208293},{"type":"v1","addr":"172.21.15.162:6813","nonce":914208293}]},"public_addr":"172.21.15.162:6809/914208293","cluster_addr":"172.21.15.162:6811/914208293","heartbeat_back_addr":"172.21.15.162:6815/914208293","heartbeat_front_addr":"172.21.15.162:6813/914208293","state":["exists","up"]},{"osd":6,"uuid":"8fa11224-cb26-4d3f-87d1-6a8f31de2a19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":56291966},{"type":"v1","addr":"172.21.15.46:6819","nonce":56291966}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":56291966},{"type":"v1","addr":"172.21.15.46:6821","nonce":56291966}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6824","nonce":56291966},{"type":"v1","addr":"172.21.15.46:6825","nonce":56291966}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":56291966},{"type":"v1","addr":"172.21.15.46:6823","nonce":56291966}]},"public_addr":"172.21.15.46:6819/56291966","cluster_addr":"172.21.15.46:6821/56291966","heartbeat_back_addr":"172.21.15.46:6825/56291966","heartbeat_front_addr":"172.21.15.46:6823/56291966","state":["exists","up"]},{"osd":7,"uuid":"0d0338b7-4d0a-4bb0-b152-8f5224871d9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6816","nonce":2146598446},{"type":"v1","addr":"172.21.15.142:6817","nonce":2146598446}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6818","nonce":2146598446},{"type":"v1","addr":"172.21.15.142:6819","nonce":2146598446}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6822","nonce":2146598446},{"type":"v1","addr":"172.21.15.142:6823","nonce":2146598446}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6820","nonce":2146598446},{"type":"v1","addr":"172.21.15.142:6821","nonce":2146598446}]},"public_addr":"172.21.15.142:6817/2146598446","cluster_addr":"172.21.15.142:6819/2146598446","heartbeat_back_addr":"172.21.15.142:6823/2146598446","heartbeat_front_addr":"172.21.15.142:6821/2146598446","state":["exists","up"]},{"osd":8,"uuid":"8b6f6037-892a-416c-babb-7552648d0b3a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6816","nonce":1173078318},{"type":"v1","addr":"172.21.15.162:6817","nonce":1173078318}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6818","nonce":1173078318},{"type":"v1","addr":"172.21.15.162:6819","nonce":1173078318}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6822","nonce":1173078318},{"type":"v1","addr":"172.21.15.162:6823","nonce":1173078318}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6820","nonce":1173078318},{"type":"v1","addr":"172.21.15.162:6821","nonce":1173078318}]},"public_addr":"172.21.15.162:6817/1173078318","cluster_addr":"172.21.15.162:6819/1173078318","heartbeat_back_addr":"172.21.15.162:6823/1173078318","heartbeat_front_addr":"172.21.15.162:6821/1173078318","state":["exists","up"]},{"osd":9,"uuid":"faccce25-6bf2-4220-8858-cda82b72be68","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":532708881},{"type":"v1","addr":"172.21.15.46:6827","nonce":532708881}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6828","nonce":532708881},{"type":"v1","addr":"172.21.15.46:6829","nonce":532708881}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":532708881},{"type":"v1","addr":"172.21.15.46:6833","nonce":532708881}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":532708881},{"type":"v1","addr":"172.21.15.46:6831","nonce":532708881}]},"public_addr":"172.21.15.46:6827/532708881","cluster_addr":"172.21.15.46:6829/532708881","heartbeat_back_addr":"172.21.15.46:6833/532708881","heartbeat_front_addr":"172.21.15.46:6831/532708881","state":["exists","up"]},{"osd":10,"uuid":"cdd356db-f9eb-4080-95ab-db5d8223e572","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6824","nonce":2845222366},{"type":"v1","addr":"172.21.15.142:6825","nonce":2845222366}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6826","nonce":2845222366},{"type":"v1","addr":"172.21.15.142:6827","nonce":2845222366}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6830","nonce":2845222366},{"type":"v1","addr":"172.21.15.142:6831","nonce":2845222366}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6828","nonce":2845222366},{"type":"v1","addr":"172.21.15.142:6829","nonce":2845222366}]},"public_addr":"172.21.15.142:6825/2845222366","cluster_addr":"172.21.15.142:6827/2845222366","heartbeat_back_addr":"172.21.15.142:6831/2845222366","heartbeat_front_addr":"172.21.15.142:6829/2845222366","state":["exists","up"]},{"osd":11,"uuid":"72104d1c-3ea2-45ad-bc0b-0500f146c06b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6824","nonce":1986323778},{"type":"v1","addr":"172.21.15.162:6825","nonce":1986323778}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6826","nonce":1986323778},{"type":"v1","addr":"172.21.15.162:6827","nonce":1986323778}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6830","nonce":1986323778},{"type":"v1","addr":"172.21.15.162:6831","nonce":1986323778}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6828","nonce":1986323778},{"type":"v1","addr":"172.21.15.162:6829","nonce":1986323778}]},"public_addr":"172.21.15.162:6825/1986323778","cluster_addr":"172.21.15.162:6827/1986323778","heartbeat_back_addr":"172.21.15.162:6831/1986323778","heartbeat_front_addr":"172.21.15.162:6829/1986323778","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:03:44.996331+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:04:07.994738+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:04:30.942255+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:04:55.588884+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:05:19.900600+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:05:43.807442+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:06:09.026976+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:06:34.586784+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:06:59.751605+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:07:26.144495+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:07:52.326909+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:08:17.423042+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.46:0/4021039857":"2024-04-01T23:02:08.493619+0000","172.21.15.46:6801/3681182647":"2024-04-01T23:02:08.493619+0000","172.21.15.46:0/4022214806":"2024-04-01T23:02:08.493619+0000","172.21.15.46:6800/1515528215":"2024-04-01T23:01:41.910162+0000","172.21.15.46:6800/3681182647":"2024-04-01T23:02:08.493619+0000","172.21.15.46:0/143885707":"2024-04-01T23:01:41.910162+0000","172.21.15.46:6801/1515528215":"2024-04-01T23:01:41.910162+0000","172.21.15.46:0/1680389778":"2024-04-01T23:01:41.910162+0000","172.21.15.46:0/1782133966":"2024-04-01T23:01:41.910162+0000","172.21.15.46:0/3925209376":"2024-04-01T23:02:08.493619+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-31T23:08:33.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:33.809+0000 7faeb35f2640 1 -- 172.21.15.46:0/3624970080 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fae940a2520 msgr2=0x7fae940a49e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:08:33.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:33.809+0000 7faeb35f2640 1 --2- 172.21.15.46:0/3624970080 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fae940a2520 0x7fae940a49e0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7faea0005ab0 tx=0x7faea0005fd0 comp rx=0 tx=0).stop 2024-03-31T23:08:33.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:33.809+0000 7faeb35f2640 1 -- 172.21.15.46:0/3624970080 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faeac1021f0 msgr2=0x7faeac1979d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:08:33.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:33.809+0000 7faeb35f2640 1 --2- 172.21.15.46:0/3624970080 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faeac1021f0 0x7faeac1979d0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7faea400d950 tx=0x7faea400de10 comp rx=0 tx=0).stop 2024-03-31T23:08:33.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:33.809+0000 7faeb35f2640 1 -- 172.21.15.46:0/3624970080 shutdown_connections 2024-03-31T23:08:33.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:33.809+0000 7faeb35f2640 1 --2- 172.21.15.46:0/3624970080 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fae940a2520 0x7fae940a49e0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:08:33.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:33.809+0000 7faeb35f2640 1 --2- 172.21.15.46:0/3624970080 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7faeac107450 0x7faeac19ef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:08:33.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:33.809+0000 7faeb35f2640 1 --2- 172.21.15.46:0/3624970080 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7faeac104b20 0x7faeac197f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:08:33.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:33.809+0000 7faeb35f2640 1 --2- 172.21.15.46:0/3624970080 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faeac1021f0 0x7faeac1979d0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:08:33.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:33.809+0000 7faeb35f2640 1 -- 172.21.15.46:0/3624970080 >> 172.21.15.46:0/3624970080 conn(0x7faeac0fbbe0 msgr2=0x7faeac102cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:08:33.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:33.809+0000 7faeb35f2640 1 -- 172.21.15.46:0/3624970080 shutdown_connections 2024-03-31T23:08:33.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:33.809+0000 7faeb35f2640 1 -- 172.21.15.46:0/3624970080 wait complete. 2024-03-31T23:08:33.819 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:08:33 smithi046 ceph-mon[32498]: from='client.24391 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi046=a;smithi046=d;smithi046=g;smithi046=j;smithi142=b;smithi142=e;smithi142=h;smithi142=k;smithi162=c;smithi162=f;smithi162=i;smithi162=l", "target": ["mon-mgr", ""]}]: dispatch 2024-03-31T23:08:33.819 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:08:33 smithi046 ceph-mon[32498]: Saving service mds.all spec with placement smithi046=a;smithi046=d;smithi046=g;smithi046=j;smithi142=b;smithi142=e;smithi142=h;smithi142=k;smithi162=c;smithi162=f;smithi162=i;smithi162=l;count:12 2024-03-31T23:08:33.819 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:08:33 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:08:33.819 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:08:33 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-31T23:08:33.819 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:08:33 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:08:33.819 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:08:33 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-31T23:08:33.819 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:08:33 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-31T23:08:33.819 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:08:33 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:08:33.820 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-31T23:04:32.823751+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-31T23:08:33.821 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-31T23:08:33.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:33.927+0000 7f79354aa640 1 -- 172.21.15.46:0/2772010345 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7930103480 msgr2=0x7f7930109d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:08:33.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:33.927+0000 7f79354aa640 1 --2- 172.21.15.46:0/2772010345 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7930103480 0x7f7930109d10 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f7928009cd0 tx=0x7f792802c0f0 comp rx=0 tx=0).stop 2024-03-31T23:08:33.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:33.927+0000 7f79354aa640 1 -- 172.21.15.46:0/2772010345 shutdown_connections 2024-03-31T23:08:33.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:33.927+0000 7f79354aa640 1 --2- 172.21.15.46:0/2772010345 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7930103480 0x7f7930109d10 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:08:33.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:33.927+0000 7f79354aa640 1 --2- 172.21.15.46:0/2772010345 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7930102ac0 0x7f7930102f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:08:33.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:33.927+0000 7f79354aa640 1 --2- 172.21.15.46:0/2772010345 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f79300ffa50 0x7f79300ffe70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:08:33.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:33.927+0000 7f79354aa640 1 -- 172.21.15.46:0/2772010345 >> 172.21.15.46:0/2772010345 conn(0x7f79300fb360 msgr2=0x7f79300fd7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:08:33.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:33.927+0000 7f79354aa640 1 -- 172.21.15.46:0/2772010345 shutdown_connections 2024-03-31T23:08:33.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:33.927+0000 7f79354aa640 1 -- 172.21.15.46:0/2772010345 wait complete. 2024-03-31T23:08:33.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:33.927+0000 7f79354aa640 1 Processor -- start 2024-03-31T23:08:33.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:33.928+0000 7f79354aa640 1 -- start start 2024-03-31T23:08:33.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:33.928+0000 7f79354aa640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f79300ffa50 0x7f79301978f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:08:33.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:33.928+0000 7f79354aa640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7930102ac0 0x7f7930197e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:08:33.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:33.928+0000 7f79354aa640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7930103480 0x7f793019eea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:08:33.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:33.928+0000 7f79354aa640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f793010d2a0 con 0x7f7930103480 2024-03-31T23:08:33.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:33.928+0000 7f79354aa640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f793010d120 con 0x7f7930102ac0 2024-03-31T23:08:33.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:33.928+0000 7f79354aa640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f793010d420 con 0x7f79300ffa50 2024-03-31T23:08:33.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:33.928+0000 7f792f7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7930103480 0x7f793019eea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:08:33.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:33.928+0000 7f792f7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7930103480 0x7f793019eea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:54306/0 (socket says 172.21.15.46:54306) 2024-03-31T23:08:33.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:33.928+0000 7f792f7fe640 1 -- 172.21.15.46:0/2755684748 learned_addr learned my addr 172.21.15.46:0/2755684748 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:08:33.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:33.928+0000 7f792effd640 1 --2- 172.21.15.46:0/2755684748 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f79300ffa50 0x7f79301978f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:08:33.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:33.928+0000 7f792e7fc640 1 --2- 172.21.15.46:0/2755684748 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7930102ac0 0x7f7930197e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:08:33.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:33.928+0000 7f792f7fe640 1 -- 172.21.15.46:0/2755684748 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f79300ffa50 msgr2=0x7f79301978f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:08:33.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:33.928+0000 7f792f7fe640 1 --2- 172.21.15.46:0/2755684748 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f79300ffa50 0x7f79301978f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:08:33.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:33.928+0000 7f792f7fe640 1 -- 172.21.15.46:0/2755684748 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7930102ac0 msgr2=0x7f7930197e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:08:33.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:33.928+0000 7f792f7fe640 1 --2- 172.21.15.46:0/2755684748 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7930102ac0 0x7f7930197e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:08:33.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:33.928+0000 7f792f7fe640 1 -- 172.21.15.46:0/2755684748 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f793019f510 con 0x7f7930103480 2024-03-31T23:08:33.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:33.928+0000 7f792f7fe640 1 --2- 172.21.15.46:0/2755684748 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7930103480 0x7f793019eea0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f792802c630 tx=0x7f79280328a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:08:33.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:33.928+0000 7f7917fff640 1 -- 172.21.15.46:0/2755684748 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7928041020 con 0x7f7930103480 2024-03-31T23:08:33.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:33.928+0000 7f7917fff640 1 -- 172.21.15.46:0/2755684748 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7928003e20 con 0x7f7930103480 2024-03-31T23:08:33.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:33.928+0000 7f79354aa640 1 -- 172.21.15.46:0/2755684748 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f793019f7a0 con 0x7f7930103480 2024-03-31T23:08:33.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:33.928+0000 7f7917fff640 1 -- 172.21.15.46:0/2755684748 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7928039040 con 0x7f7930103480 2024-03-31T23:08:33.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:33.929+0000 7f79354aa640 1 -- 172.21.15.46:0/2755684748 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f793019fc90 con 0x7f7930103480 2024-03-31T23:08:33.932 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:33.930+0000 7f79354aa640 1 -- 172.21.15.46:0/2755684748 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7930108bf0 con 0x7f7930103480 2024-03-31T23:08:33.932 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:33.931+0000 7f7917fff640 1 -- 172.21.15.46:0/2755684748 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 16) ==== 137320+0+0 (secure 0 0 0) 0x7f7928044020 con 0x7f7930103480 2024-03-31T23:08:33.932 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:33.931+0000 7f7917fff640 1 --2- 172.21.15.46:0/2755684748 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f79180a24a0 0x7f79180a4960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:08:33.932 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:33.931+0000 7f792effd640 1 --2- 172.21.15.46:0/2755684748 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f79180a24a0 0x7f79180a4960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:08:33.932 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:33.932+0000 7f7917fff640 1 -- 172.21.15.46:0/2755684748 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f79280f06d0 con 0x7f7930103480 2024-03-31T23:08:33.932 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:33.932+0000 7f792effd640 1 --2- 172.21.15.46:0/2755684748 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f79180a24a0 0x7f79180a4960 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f7920005fd0 tx=0x7f792000b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:08:33.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:33.936+0000 7f7917fff640 1 -- 172.21.15.46:0/2755684748 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f792804b020 con 0x7f7930103480 2024-03-31T23:08:33.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:08:33 smithi142 ceph-mon[35271]: from='client.24391 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi046=a;smithi046=d;smithi046=g;smithi046=j;smithi142=b;smithi142=e;smithi142=h;smithi142=k;smithi162=c;smithi162=f;smithi162=i;smithi162=l", "target": ["mon-mgr", ""]}]: dispatch 2024-03-31T23:08:33.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:08:33 smithi142 ceph-mon[35271]: Saving service mds.all spec with placement smithi046=a;smithi046=d;smithi046=g;smithi046=j;smithi142=b;smithi142=e;smithi142=h;smithi142=k;smithi162=c;smithi162=f;smithi162=i;smithi162=l;count:12 2024-03-31T23:08:33.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:08:33 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:08:33.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:08:33 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-31T23:08:33.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:08:33 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:08:33.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:08:33 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-31T23:08:33.944 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:08:33 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-31T23:08:33.944 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:08:33 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:08:34.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:34.116+0000 7f79354aa640 1 -- 172.21.15.46:0/2755684748 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f7930062350 con 0x7f7930103480 2024-03-31T23:08:34.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:34.117+0000 7f7917fff640 1 -- 172.21.15.46:0/2755684748 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v82) ==== 93+0+10 (secure 0 0 0) 0x7f79280bdd30 con 0x7f7930103480 2024-03-31T23:08:34.117 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 1 2024-03-31T23:08:34.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:34.119+0000 7f79354aa640 1 -- 172.21.15.46:0/2755684748 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f79180a24a0 msgr2=0x7f79180a4960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:08:34.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:34.119+0000 7f79354aa640 1 --2- 172.21.15.46:0/2755684748 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f79180a24a0 0x7f79180a4960 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f7920005fd0 tx=0x7f792000b390 comp rx=0 tx=0).stop 2024-03-31T23:08:34.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:34.119+0000 7f79354aa640 1 -- 172.21.15.46:0/2755684748 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7930103480 msgr2=0x7f793019eea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:08:34.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:34.119+0000 7f79354aa640 1 --2- 172.21.15.46:0/2755684748 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7930103480 0x7f793019eea0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f792802c630 tx=0x7f79280328a0 comp rx=0 tx=0).stop 2024-03-31T23:08:34.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:34.119+0000 7f79354aa640 1 -- 172.21.15.46:0/2755684748 shutdown_connections 2024-03-31T23:08:34.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:34.119+0000 7f79354aa640 1 --2- 172.21.15.46:0/2755684748 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f79180a24a0 0x7f79180a4960 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:08:34.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:34.119+0000 7f79354aa640 1 --2- 172.21.15.46:0/2755684748 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7930103480 0x7f793019eea0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:08:34.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:34.119+0000 7f79354aa640 1 --2- 172.21.15.46:0/2755684748 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7930102ac0 0x7f7930197e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:08:34.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:34.119+0000 7f79354aa640 1 --2- 172.21.15.46:0/2755684748 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f79300ffa50 0x7f79301978f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:08:34.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:34.119+0000 7f79354aa640 1 -- 172.21.15.46:0/2755684748 >> 172.21.15.46:0/2755684748 conn(0x7f79300fb360 msgr2=0x7f7930105dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:08:34.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:34.119+0000 7f79354aa640 1 -- 172.21.15.46:0/2755684748 shutdown_connections 2024-03-31T23:08:34.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:34.119+0000 7f79354aa640 1 -- 172.21.15.46:0/2755684748 wait complete. 2024-03-31T23:08:34.130 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-31T23:08:34.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:34.232+0000 7f668e6f4640 1 -- 172.21.15.46:0/693529412 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6688102220 msgr2=0x7f6688108ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:08:34.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:34.232+0000 7f668e6f4640 1 --2- 172.21.15.46:0/693529412 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6688102220 0x7f6688108ab0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f6680009ca0 tx=0x7f668002c0f0 comp rx=0 tx=0).stop 2024-03-31T23:08:34.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:34.233+0000 7f668e6f4640 1 -- 172.21.15.46:0/693529412 shutdown_connections 2024-03-31T23:08:34.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:34.233+0000 7f668e6f4640 1 --2- 172.21.15.46:0/693529412 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6688102220 0x7f6688108ab0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:08:34.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:34.233+0000 7f668e6f4640 1 --2- 172.21.15.46:0/693529412 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6688101860 0x7f6688101ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:08:34.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:34.233+0000 7f668e6f4640 1 --2- 172.21.15.46:0/693529412 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6688100670 0x7f6688100a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:08:34.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:34.233+0000 7f668e6f4640 1 -- 172.21.15.46:0/693529412 >> 172.21.15.46:0/693529412 conn(0x7f66880fbbe0 msgr2=0x7f66880fe020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:08:34.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:34.233+0000 7f668e6f4640 1 -- 172.21.15.46:0/693529412 shutdown_connections 2024-03-31T23:08:34.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:34.233+0000 7f668e6f4640 1 -- 172.21.15.46:0/693529412 wait complete. 2024-03-31T23:08:34.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:34.233+0000 7f668e6f4640 1 Processor -- start 2024-03-31T23:08:34.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:34.233+0000 7f668e6f4640 1 -- start start 2024-03-31T23:08:34.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:34.233+0000 7f668e6f4640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6688100670 0x7f6688197970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:08:34.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:34.234+0000 7f668e6f4640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6688101860 0x7f6688197eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:08:34.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:34.234+0000 7f668e6f4640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6688102220 0x7f668819ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:08:34.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:34.234+0000 7f668e6f4640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f668810c1a0 con 0x7f6688101860 2024-03-31T23:08:34.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:34.234+0000 7f668e6f4640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f668810c020 con 0x7f6688102220 2024-03-31T23:08:34.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:34.234+0000 7f668e6f4640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f668810c320 con 0x7f6688100670 2024-03-31T23:08:34.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:34.234+0000 7f66877fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6688101860 0x7f6688197eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:08:34.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:34.234+0000 7f66877fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6688101860 0x7f6688197eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:54336/0 (socket says 172.21.15.46:54336) 2024-03-31T23:08:34.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:34.234+0000 7f66877fe640 1 -- 172.21.15.46:0/2684222967 learned_addr learned my addr 172.21.15.46:0/2684222967 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:08:34.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:34.234+0000 7f668cc91640 1 --2- 172.21.15.46:0/2684222967 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6688102220 0x7f668819ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:08:34.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:34.234+0000 7f6687fff640 1 --2- 172.21.15.46:0/2684222967 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6688100670 0x7f6688197970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:08:34.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:34.234+0000 7f66877fe640 1 -- 172.21.15.46:0/2684222967 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6688100670 msgr2=0x7f6688197970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:08:34.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:34.234+0000 7f66877fe640 1 --2- 172.21.15.46:0/2684222967 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6688100670 0x7f6688197970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:08:34.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:34.234+0000 7f66877fe640 1 -- 172.21.15.46:0/2684222967 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6688102220 msgr2=0x7f668819ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:08:34.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:34.234+0000 7f66877fe640 1 --2- 172.21.15.46:0/2684222967 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6688102220 0x7f668819ef20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:08:34.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:34.234+0000 7f66877fe640 1 -- 172.21.15.46:0/2684222967 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f668819f590 con 0x7f6688101860 2024-03-31T23:08:34.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:34.234+0000 7f66877fe640 1 --2- 172.21.15.46:0/2684222967 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6688101860 0x7f6688197eb0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f667000d950 tx=0x7f667000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:08:34.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:34.234+0000 7f66857fa640 1 -- 172.21.15.46:0/2684222967 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6670017070 con 0x7f6688101860 2024-03-31T23:08:34.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:34.234+0000 7f66857fa640 1 -- 172.21.15.46:0/2684222967 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6670009c30 con 0x7f6688101860 2024-03-31T23:08:34.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:34.234+0000 7f668e6f4640 1 -- 172.21.15.46:0/2684222967 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f668819f880 con 0x7f6688101860 2024-03-31T23:08:34.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:34.234+0000 7f66857fa640 1 -- 172.21.15.46:0/2684222967 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f667000f620 con 0x7f6688101860 2024-03-31T23:08:34.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:34.234+0000 7f668e6f4640 1 -- 172.21.15.46:0/2684222967 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f668819fdd0 con 0x7f6688101860 2024-03-31T23:08:34.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:34.235+0000 7f668e6f4640 1 -- 172.21.15.46:0/2684222967 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f66881a0030 con 0x7f6688101860 2024-03-31T23:08:34.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:34.237+0000 7f66857fa640 1 -- 172.21.15.46:0/2684222967 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 16) ==== 137320+0+0 (secure 0 0 0) 0x7f66700109a0 con 0x7f6688101860 2024-03-31T23:08:34.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:34.238+0000 7f66857fa640 1 --2- 172.21.15.46:0/2684222967 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f66740a24a0 0x7f66740a4960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:08:34.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:34.238+0000 7f6687fff640 1 --2- 172.21.15.46:0/2684222967 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f66740a24a0 0x7f66740a4960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:08:34.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:34.238+0000 7f66857fa640 1 -- 172.21.15.46:0/2684222967 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f66700cdf60 con 0x7f6688101860 2024-03-31T23:08:34.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:34.238+0000 7f6687fff640 1 --2- 172.21.15.46:0/2684222967 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f66740a24a0 0x7f66740a4960 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f66881016c0 tx=0x7f6678005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:08:34.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:34.241+0000 7f66857fa640 1 -- 172.21.15.46:0/2684222967 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f6670009de0 con 0x7f6688101860 2024-03-31T23:08:34.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:34.461+0000 7f668e6f4640 1 -- 172.21.15.46:0/2684222967 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f66881a0200 con 0x7f6688101860 2024-03-31T23:08:34.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:34.462+0000 7f66857fa640 1 -- 172.21.15.46:0/2684222967 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 1 v1) ==== 93+0+569 (secure 0 0 0) 0x7f667009b5c0 con 0x7f6688101860 2024-03-31T23:08:34.462 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-31T23:08:34.462 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":1,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[]} 2024-03-31T23:08:34.462 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 1 2024-03-31T23:08:34.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:34.464+0000 7f668e6f4640 1 -- 172.21.15.46:0/2684222967 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f66740a24a0 msgr2=0x7f66740a4960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:08:34.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:34.464+0000 7f668e6f4640 1 --2- 172.21.15.46:0/2684222967 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f66740a24a0 0x7f66740a4960 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f66881016c0 tx=0x7f6678005f90 comp rx=0 tx=0).stop 2024-03-31T23:08:34.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:34.464+0000 7f668e6f4640 1 -- 172.21.15.46:0/2684222967 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6688101860 msgr2=0x7f6688197eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:08:34.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:34.464+0000 7f668e6f4640 1 --2- 172.21.15.46:0/2684222967 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6688101860 0x7f6688197eb0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f667000d950 tx=0x7f667000de10 comp rx=0 tx=0).stop 2024-03-31T23:08:34.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:34.464+0000 7f668e6f4640 1 -- 172.21.15.46:0/2684222967 shutdown_connections 2024-03-31T23:08:34.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:34.464+0000 7f668e6f4640 1 --2- 172.21.15.46:0/2684222967 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f66740a24a0 0x7f66740a4960 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:08:34.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:34.464+0000 7f668e6f4640 1 --2- 172.21.15.46:0/2684222967 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6688102220 0x7f668819ef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:08:34.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:34.464+0000 7f668e6f4640 1 --2- 172.21.15.46:0/2684222967 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6688101860 0x7f6688197eb0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:08:34.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:34.464+0000 7f668e6f4640 1 --2- 172.21.15.46:0/2684222967 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6688100670 0x7f6688197970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:08:34.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:34.464+0000 7f668e6f4640 1 -- 172.21.15.46:0/2684222967 >> 172.21.15.46:0/2684222967 conn(0x7f66880fbbe0 msgr2=0x7f66880fd6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:08:34.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:34.464+0000 7f668e6f4640 1 -- 172.21.15.46:0/2684222967 shutdown_connections 2024-03-31T23:08:34.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:34.464+0000 7f668e6f4640 1 -- 172.21.15.46:0/2684222967 wait complete. 2024-03-31T23:08:34.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:08:34 smithi162 ceph-mon[36237]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-31T23:08:34.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:08:34 smithi162 ceph-mon[36237]: Deploying daemon mds.l on smithi162 2024-03-31T23:08:34.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:08:34 smithi162 ceph-mon[36237]: from='client.? 172.21.15.46:0/3624970080' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-31T23:08:34.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:08:34 smithi162 ceph-mon[36237]: from='client.? 172.21.15.46:0/2755684748' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-31T23:08:34.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:08:34 smithi162 ceph-mon[36237]: from='client.? 172.21.15.46:0/2684222967' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-31T23:08:34.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:08:34 smithi046 ceph-mon[32498]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-31T23:08:34.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:08:34 smithi046 ceph-mon[32498]: Deploying daemon mds.l on smithi162 2024-03-31T23:08:34.862 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:08:34 smithi046 ceph-mon[32498]: from='client.? 172.21.15.46:0/3624970080' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-31T23:08:34.862 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:08:34 smithi046 ceph-mon[32498]: from='client.? 172.21.15.46:0/2755684748' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-31T23:08:34.862 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:08:34 smithi046 ceph-mon[32498]: from='client.? 172.21.15.46:0/2684222967' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-31T23:08:34.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:08:34 smithi142 ceph-mon[35271]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-31T23:08:34.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:08:34 smithi142 ceph-mon[35271]: Deploying daemon mds.l on smithi162 2024-03-31T23:08:34.944 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:08:34 smithi142 ceph-mon[35271]: from='client.? 172.21.15.46:0/3624970080' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-31T23:08:34.944 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:08:34 smithi142 ceph-mon[35271]: from='client.? 172.21.15.46:0/2755684748' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-31T23:08:34.944 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:08:34 smithi142 ceph-mon[35271]: from='client.? 172.21.15.46:0/2684222967' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-31T23:08:35.518 INFO:journalctl@ceph.mds.l.smithi162.stdout:Mar 31 23:08:35 smithi162 systemd[1]: Starting Ceph mds.l for 6a9d8f08-efb2-11ee-b647-cb9ed24678a4... 2024-03-31T23:08:35.786 INFO:journalctl@ceph.mds.l.smithi162.stdout:Mar 31 23:08:35 smithi162 podman[66429]: 2024-03-31 23:08:35.42052486 +0000 UTC m=+0.017782836 image pull ec18d7baa60e8b1a0b126fbe5306acce11b1ed0ec5e3bcdddbf39b0a3c591791 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175 2024-03-31T23:08:35.786 INFO:journalctl@ceph.mds.l.smithi162.stdout:Mar 31 23:08:35 smithi162 podman[66429]: 2024-03-31 23:08:35.549194868 +0000 UTC m=+0.146452832 container create dad2ed6cb60ba505d629e02b8f167a0456ceb3006caeacf01b3343199a32a4f5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mds-l, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, RELEASE=squid-384f31f, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240326, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0) 2024-03-31T23:08:35.786 INFO:journalctl@ceph.mds.l.smithi162.stdout:Mar 31 23:08:35 smithi162 podman[66429]: 2024-03-31 23:08:35.785635401 +0000 UTC m=+0.382893378 container init dad2ed6cb60ba505d629e02b8f167a0456ceb3006caeacf01b3343199a32a4f5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mds-l, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, RELEASE=squid-384f31f, org.label-schema.build-date=20240326, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, ceph=True) 2024-03-31T23:08:36.046 INFO:journalctl@ceph.mds.l.smithi162.stdout:Mar 31 23:08:35 smithi162 podman[66429]: 2024-03-31 23:08:35.788797193 +0000 UTC m=+0.386055164 container start dad2ed6cb60ba505d629e02b8f167a0456ceb3006caeacf01b3343199a32a4f5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mds-l, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=squid-384f31f, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.build-date=20240326, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-03-31T23:08:36.046 INFO:journalctl@ceph.mds.l.smithi162.stdout:Mar 31 23:08:35 smithi162 ceph-mds[66448]: -- 172.21.15.162:0/1145286018 <== mon.1 v2:172.21.15.162:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55da4ad1fa40 con 0x55da4acf2800 2024-03-31T23:08:36.046 INFO:journalctl@ceph.mds.l.smithi162.stdout:Mar 31 23:08:35 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mds-l[66444]: starting mds.l at 2024-03-31T23:08:36.046 INFO:journalctl@ceph.mds.l.smithi162.stdout:Mar 31 23:08:35 smithi162 bash[66429]: dad2ed6cb60ba505d629e02b8f167a0456ceb3006caeacf01b3343199a32a4f5 2024-03-31T23:08:36.046 INFO:journalctl@ceph.mds.l.smithi162.stdout:Mar 31 23:08:35 smithi162 systemd[1]: Started Ceph mds.l for 6a9d8f08-efb2-11ee-b647-cb9ed24678a4. 2024-03-31T23:08:36.475 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-31T23:08:36.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:36.591+0000 7f9d7468f640 1 -- 172.21.15.46:0/1340543824 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9d6c1022a0 msgr2=0x7f9d6c108b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:08:36.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:36.591+0000 7f9d7468f640 1 --2- 172.21.15.46:0/1340543824 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9d6c1022a0 0x7f9d6c108b30 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f9d64009900 tx=0x7f9d6402bc30 comp rx=0 tx=0).stop 2024-03-31T23:08:36.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:36.591+0000 7f9d7468f640 1 -- 172.21.15.46:0/1340543824 shutdown_connections 2024-03-31T23:08:36.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:36.591+0000 7f9d7468f640 1 --2- 172.21.15.46:0/1340543824 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9d6c1022a0 0x7f9d6c108b30 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:08:36.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:36.591+0000 7f9d7468f640 1 --2- 172.21.15.46:0/1340543824 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9d6c1018e0 0x7f9d6c101d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:08:36.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:36.591+0000 7f9d7468f640 1 --2- 172.21.15.46:0/1340543824 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9d6c1006f0 0x7f9d6c100b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:08:36.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:36.591+0000 7f9d7468f640 1 -- 172.21.15.46:0/1340543824 >> 172.21.15.46:0/1340543824 conn(0x7f9d6c0fbc80 msgr2=0x7f9d6c0fe0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:08:36.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:36.591+0000 7f9d7468f640 1 -- 172.21.15.46:0/1340543824 shutdown_connections 2024-03-31T23:08:36.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:36.591+0000 7f9d7468f640 1 -- 172.21.15.46:0/1340543824 wait complete. 2024-03-31T23:08:36.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:36.592+0000 7f9d7468f640 1 Processor -- start 2024-03-31T23:08:36.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:36.592+0000 7f9d7468f640 1 -- start start 2024-03-31T23:08:36.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:36.592+0000 7f9d7468f640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9d6c1006f0 0x7f9d6c104630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:08:36.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:36.592+0000 7f9d7468f640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9d6c1018e0 0x7f9d6c102cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:08:36.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:36.592+0000 7f9d7468f640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9d6c1022a0 0x7f9d6c103200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:08:36.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:36.592+0000 7f9d7468f640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f9d6c10c270 con 0x7f9d6c1006f0 2024-03-31T23:08:36.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:36.592+0000 7f9d7468f640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f9d6c10c0f0 con 0x7f9d6c1022a0 2024-03-31T23:08:36.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:36.592+0000 7f9d7468f640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f9d6c10c3f0 con 0x7f9d6c1018e0 2024-03-31T23:08:36.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:36.592+0000 7f9d7242b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9d6c1006f0 0x7f9d6c104630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:08:36.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:36.592+0000 7f9d7242b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9d6c1006f0 0x7f9d6c104630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:54340/0 (socket says 172.21.15.46:54340) 2024-03-31T23:08:36.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:36.592+0000 7f9d7242b640 1 -- 172.21.15.46:0/536084969 learned_addr learned my addr 172.21.15.46:0/536084969 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:08:36.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:36.592+0000 7f9d71c2a640 1 --2- 172.21.15.46:0/536084969 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9d6c1018e0 0x7f9d6c102cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:08:36.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:36.592+0000 7f9d72c2c640 1 --2- 172.21.15.46:0/536084969 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9d6c1022a0 0x7f9d6c103200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:08:36.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:36.592+0000 7f9d7242b640 1 -- 172.21.15.46:0/536084969 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9d6c1018e0 msgr2=0x7f9d6c102cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:08:36.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:36.592+0000 7f9d7242b640 1 --2- 172.21.15.46:0/536084969 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9d6c1018e0 0x7f9d6c102cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:08:36.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:36.592+0000 7f9d7242b640 1 -- 172.21.15.46:0/536084969 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9d6c1022a0 msgr2=0x7f9d6c103200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:08:36.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:36.592+0000 7f9d7242b640 1 --2- 172.21.15.46:0/536084969 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9d6c1022a0 0x7f9d6c103200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:08:36.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:36.592+0000 7f9d7242b640 1 -- 172.21.15.46:0/536084969 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9d6c103ac0 con 0x7f9d6c1006f0 2024-03-31T23:08:36.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:36.592+0000 7f9d71c2a640 1 --2- 172.21.15.46:0/536084969 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9d6c1018e0 0x7f9d6c102cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T23:08:36.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:36.592+0000 7f9d72c2c640 1 --2- 172.21.15.46:0/536084969 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9d6c1022a0 0x7f9d6c103200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T23:08:36.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:36.593+0000 7f9d7242b640 1 --2- 172.21.15.46:0/536084969 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9d6c1006f0 0x7f9d6c104630 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f9d5c00b9c0 tx=0x7f9d5c00be80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:08:36.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:36.593+0000 7f9d6b7fe640 1 -- 172.21.15.46:0/536084969 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d5c019070 con 0x7f9d6c1006f0 2024-03-31T23:08:36.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:36.593+0000 7f9d6b7fe640 1 -- 172.21.15.46:0/536084969 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9d5c00cba0 con 0x7f9d6c1006f0 2024-03-31T23:08:36.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:36.593+0000 7f9d7468f640 1 -- 172.21.15.46:0/536084969 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9d6c1a1b30 con 0x7f9d6c1006f0 2024-03-31T23:08:36.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:36.593+0000 7f9d6b7fe640 1 -- 172.21.15.46:0/536084969 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d5c0125a0 con 0x7f9d6c1006f0 2024-03-31T23:08:36.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:36.593+0000 7f9d7468f640 1 -- 172.21.15.46:0/536084969 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9d6c1a2050 con 0x7f9d6c1006f0 2024-03-31T23:08:36.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:36.594+0000 7f9d7468f640 1 -- 172.21.15.46:0/536084969 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9d6c1a2280 con 0x7f9d6c1006f0 2024-03-31T23:08:36.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:36.595+0000 7f9d6b7fe640 1 -- 172.21.15.46:0/536084969 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 16) ==== 137320+0+0 (secure 0 0 0) 0x7f9d5c01c070 con 0x7f9d6c1006f0 2024-03-31T23:08:36.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:36.595+0000 7f9d6b7fe640 1 --2- 172.21.15.46:0/536084969 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f9d540a2520 0x7f9d540a49e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:08:36.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:36.596+0000 7f9d71c2a640 1 --2- 172.21.15.46:0/536084969 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f9d540a2520 0x7f9d540a49e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:08:36.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:36.596+0000 7f9d6b7fe640 1 -- 172.21.15.46:0/536084969 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f9d5c0cdf40 con 0x7f9d6c1006f0 2024-03-31T23:08:36.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:36.596+0000 7f9d71c2a640 1 --2- 172.21.15.46:0/536084969 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f9d540a2520 0x7f9d540a49e0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f9d600058c0 tx=0x7f9d60005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:08:36.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:36.599+0000 7f9d6b7fe640 1 -- 172.21.15.46:0/536084969 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f9d5c09b500 con 0x7f9d6c1006f0 2024-03-31T23:08:36.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:08:36 smithi162 ceph-mon[36237]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-31T23:08:36.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:08:36 smithi162 ceph-mon[36237]: mds.? [v2:172.21.15.162:6832/1746950966,v1:172.21.15.162:6833/1746950966] up:boot 2024-03-31T23:08:36.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:08:36 smithi162 ceph-mon[36237]: fsmap 1 up:standby 2024-03-31T23:08:36.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:08:36 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-31T23:08:36.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:08:36 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:08:36.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:08:36 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:08:36.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:08:36 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:08:36.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:08:36 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-31T23:08:36.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:08:36 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-31T23:08:36.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:08:36 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:08:36.800 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:08:36 smithi142 ceph-mon[35271]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-31T23:08:36.800 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:08:36 smithi142 ceph-mon[35271]: mds.? [v2:172.21.15.162:6832/1746950966,v1:172.21.15.162:6833/1746950966] up:boot 2024-03-31T23:08:36.800 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:08:36 smithi142 ceph-mon[35271]: fsmap 1 up:standby 2024-03-31T23:08:36.800 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:08:36 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-31T23:08:36.800 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:08:36 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:08:36.800 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:08:36 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:08:36.800 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:08:36 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:08:36.800 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:08:36 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-31T23:08:36.801 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:08:36 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-31T23:08:36.801 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:08:36 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:08:36.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:36.820+0000 7f9d7468f640 1 -- 172.21.15.46:0/536084969 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9d6c1a2450 con 0x7f9d6c1006f0 2024-03-31T23:08:36.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:36.820+0000 7f9d6b7fe640 1 -- 172.21.15.46:0/536084969 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) ==== 93+0+1396 (secure 0 0 0) 0x7f9d5c09ae00 con 0x7f9d6c1006f0 2024-03-31T23:08:36.820 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-31T23:08:36.820 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":2,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24415,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6833/1746950966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6832","nonce":1746950966},{"type":"v1","addr":"172.21.15.162:6833","nonce":1746950966}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-03-31T23:08:36.821 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 2 2024-03-31T23:08:36.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:36.822+0000 7f9d7468f640 1 -- 172.21.15.46:0/536084969 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f9d540a2520 msgr2=0x7f9d540a49e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:08:36.823 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:36.822+0000 7f9d7468f640 1 --2- 172.21.15.46:0/536084969 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f9d540a2520 0x7f9d540a49e0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f9d600058c0 tx=0x7f9d60005850 comp rx=0 tx=0).stop 2024-03-31T23:08:36.823 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:36.822+0000 7f9d7468f640 1 -- 172.21.15.46:0/536084969 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9d6c1006f0 msgr2=0x7f9d6c104630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:08:36.823 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:36.822+0000 7f9d7468f640 1 --2- 172.21.15.46:0/536084969 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9d6c1006f0 0x7f9d6c104630 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f9d5c00b9c0 tx=0x7f9d5c00be80 comp rx=0 tx=0).stop 2024-03-31T23:08:36.823 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:36.822+0000 7f9d7468f640 1 -- 172.21.15.46:0/536084969 shutdown_connections 2024-03-31T23:08:36.823 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:36.822+0000 7f9d7468f640 1 --2- 172.21.15.46:0/536084969 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f9d540a2520 0x7f9d540a49e0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:08:36.823 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:36.822+0000 7f9d7468f640 1 --2- 172.21.15.46:0/536084969 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9d6c1022a0 0x7f9d6c103200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:08:36.823 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:36.822+0000 7f9d7468f640 1 --2- 172.21.15.46:0/536084969 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9d6c1018e0 0x7f9d6c102cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:08:36.823 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:36.822+0000 7f9d7468f640 1 --2- 172.21.15.46:0/536084969 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9d6c1006f0 0x7f9d6c104630 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:08:36.823 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:36.822+0000 7f9d7468f640 1 -- 172.21.15.46:0/536084969 >> 172.21.15.46:0/536084969 conn(0x7f9d6c0fbc80 msgr2=0x7f9d6c0fd790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:08:36.823 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:36.822+0000 7f9d7468f640 1 -- 172.21.15.46:0/536084969 shutdown_connections 2024-03-31T23:08:36.823 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:36.822+0000 7f9d7468f640 1 -- 172.21.15.46:0/536084969 wait complete. 2024-03-31T23:08:36.832 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:08:36 smithi046 ceph-mon[32498]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-31T23:08:36.832 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:08:36 smithi046 ceph-mon[32498]: mds.? [v2:172.21.15.162:6832/1746950966,v1:172.21.15.162:6833/1746950966] up:boot 2024-03-31T23:08:36.832 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:08:36 smithi046 ceph-mon[32498]: fsmap 1 up:standby 2024-03-31T23:08:36.832 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:08:36 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-31T23:08:36.832 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:08:36 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:08:36.832 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:08:36 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:08:36.832 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:08:36 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:08:36.832 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:08:36 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-31T23:08:36.833 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:08:36 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-31T23:08:36.833 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:08:36 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:08:37.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:08:37 smithi162 ceph-mon[36237]: Deploying daemon mds.k on smithi142 2024-03-31T23:08:37.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:08:37 smithi162 ceph-mon[36237]: from='client.? 172.21.15.46:0/536084969' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-31T23:08:37.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:08:37 smithi046 ceph-mon[32498]: Deploying daemon mds.k on smithi142 2024-03-31T23:08:37.862 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:08:37 smithi046 ceph-mon[32498]: from='client.? 172.21.15.46:0/536084969' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-31T23:08:37.934 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:08:37 smithi142 ceph-mon[35271]: Deploying daemon mds.k on smithi142 2024-03-31T23:08:37.935 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:08:37 smithi142 ceph-mon[35271]: from='client.? 172.21.15.46:0/536084969' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-31T23:08:38.193 INFO:journalctl@ceph.mds.k.smithi142.stdout:Mar 31 23:08:38 smithi142 systemd[1]: Starting Ceph mds.k for 6a9d8f08-efb2-11ee-b647-cb9ed24678a4... 2024-03-31T23:08:38.585 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:08:38 smithi142 ceph-mon[35271]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-31T23:08:38.585 INFO:journalctl@ceph.mds.k.smithi142.stdout:Mar 31 23:08:38 smithi142 podman[64840]: 2024-03-31 23:08:38.222674116 +0000 UTC m=+0.017719637 image pull ec18d7baa60e8b1a0b126fbe5306acce11b1ed0ec5e3bcdddbf39b0a3c591791 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175 2024-03-31T23:08:38.586 INFO:journalctl@ceph.mds.k.smithi142.stdout:Mar 31 23:08:38 smithi142 podman[64840]: 2024-03-31 23:08:38.340670814 +0000 UTC m=+0.135716362 container create 7f5af54f1631112124998a5118513482bd5651538b1a1d1b6996134446f5a123 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mds-k, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, RELEASE=squid-384f31f, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.build-date=20240326, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.license=GPLv2, ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=) 2024-03-31T23:08:38.586 INFO:journalctl@ceph.mds.k.smithi142.stdout:Mar 31 23:08:38 smithi142 podman[64840]: 2024-03-31 23:08:38.559971213 +0000 UTC m=+0.355016715 container init 7f5af54f1631112124998a5118513482bd5651538b1a1d1b6996134446f5a123 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mds-k, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True, RELEASE=squid-384f31f, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240326, GIT_BRANCH=HEAD, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0) 2024-03-31T23:08:38.586 INFO:journalctl@ceph.mds.k.smithi142.stdout:Mar 31 23:08:38 smithi142 podman[64840]: 2024-03-31 23:08:38.562707414 +0000 UTC m=+0.357752918 container start 7f5af54f1631112124998a5118513482bd5651538b1a1d1b6996134446f5a123 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mds-k, org.label-schema.license=GPLv2, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240326, RELEASE=squid-384f31f) 2024-03-31T23:08:38.795 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:08:38 smithi162 ceph-mon[36237]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-31T23:08:38.834 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-31T23:08:38.859 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:08:38 smithi046 ceph-mon[32498]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-31T23:08:38.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:38.941+0000 7f14aded4640 1 -- 172.21.15.46:0/161590940 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f14a81017a0 msgr2=0x7f14a8101bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:08:38.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:38.941+0000 7f14aded4640 1 --2- 172.21.15.46:0/161590940 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f14a81017a0 0x7f14a8101bc0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f1498009900 tx=0x7f149802bc30 comp rx=0 tx=0).stop 2024-03-31T23:08:38.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:38.941+0000 7f14aded4640 1 -- 172.21.15.46:0/161590940 shutdown_connections 2024-03-31T23:08:38.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:38.941+0000 7f14aded4640 1 --2- 172.21.15.46:0/161590940 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f14a8103350 0x7f14a8109be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:08:38.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:38.941+0000 7f14aded4640 1 --2- 172.21.15.46:0/161590940 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f14a8102990 0x7f14a8102e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:08:38.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:38.941+0000 7f14aded4640 1 --2- 172.21.15.46:0/161590940 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f14a81017a0 0x7f14a8101bc0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:08:38.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:38.941+0000 7f14aded4640 1 -- 172.21.15.46:0/161590940 >> 172.21.15.46:0/161590940 conn(0x7f14a80fccf0 msgr2=0x7f14a80ff150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:08:38.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:38.941+0000 7f14aded4640 1 -- 172.21.15.46:0/161590940 shutdown_connections 2024-03-31T23:08:38.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:38.941+0000 7f14aded4640 1 -- 172.21.15.46:0/161590940 wait complete. 2024-03-31T23:08:38.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:38.941+0000 7f14aded4640 1 Processor -- start 2024-03-31T23:08:38.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:38.941+0000 7f14aded4640 1 -- start start 2024-03-31T23:08:38.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:38.942+0000 7f14aded4640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f14a81017a0 0x7f14a8197a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:08:38.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:38.942+0000 7f14aded4640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f14a8102990 0x7f14a8197f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:08:38.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:38.942+0000 7f14aded4640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f14a8103350 0x7f14a819efd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:08:38.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:38.942+0000 7f14aded4640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f14a810d2d0 con 0x7f14a8103350 2024-03-31T23:08:38.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:38.942+0000 7f14aded4640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f14a810d150 con 0x7f14a8102990 2024-03-31T23:08:38.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:38.942+0000 7f14aded4640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f14a810d450 con 0x7f14a81017a0 2024-03-31T23:08:38.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:38.942+0000 7f14a7fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f14a8103350 0x7f14a819efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:08:38.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:38.942+0000 7f14a7fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f14a8103350 0x7f14a819efd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:54368/0 (socket says 172.21.15.46:54368) 2024-03-31T23:08:38.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:38.942+0000 7f14a7fff640 1 -- 172.21.15.46:0/3498468015 learned_addr learned my addr 172.21.15.46:0/3498468015 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:08:38.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:38.942+0000 7f14a6ffd640 1 --2- 172.21.15.46:0/3498468015 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f14a8102990 0x7f14a8197f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:08:38.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:38.942+0000 7f14a77fe640 1 --2- 172.21.15.46:0/3498468015 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f14a81017a0 0x7f14a8197a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:08:38.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:38.942+0000 7f14a7fff640 1 -- 172.21.15.46:0/3498468015 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f14a81017a0 msgr2=0x7f14a8197a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:08:38.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:38.942+0000 7f14a7fff640 1 --2- 172.21.15.46:0/3498468015 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f14a81017a0 0x7f14a8197a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:08:38.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:38.942+0000 7f14a7fff640 1 -- 172.21.15.46:0/3498468015 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f14a8102990 msgr2=0x7f14a8197f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:08:38.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:38.942+0000 7f14a7fff640 1 --2- 172.21.15.46:0/3498468015 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f14a8102990 0x7f14a8197f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:08:38.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:38.942+0000 7f14a7fff640 1 -- 172.21.15.46:0/3498468015 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f14a819f640 con 0x7f14a8103350 2024-03-31T23:08:38.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:38.942+0000 7f14a77fe640 1 --2- 172.21.15.46:0/3498468015 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f14a81017a0 0x7f14a8197a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T23:08:38.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:38.942+0000 7f14a7fff640 1 --2- 172.21.15.46:0/3498468015 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f14a8103350 0x7f14a819efd0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f14a000cad0 tx=0x7f14a000cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:08:38.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:38.942+0000 7f14a4ff9640 1 -- 172.21.15.46:0/3498468015 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14a000ae20 con 0x7f14a8103350 2024-03-31T23:08:38.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:38.943+0000 7f14a4ff9640 1 -- 172.21.15.46:0/3498468015 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f14a0005020 con 0x7f14a8103350 2024-03-31T23:08:38.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:38.943+0000 7f14aded4640 1 -- 172.21.15.46:0/3498468015 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f14a819f930 con 0x7f14a8103350 2024-03-31T23:08:38.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:38.943+0000 7f14a4ff9640 1 -- 172.21.15.46:0/3498468015 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14a000f6b0 con 0x7f14a8103350 2024-03-31T23:08:38.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:38.943+0000 7f14aded4640 1 -- 172.21.15.46:0/3498468015 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f14a819fe80 con 0x7f14a8103350 2024-03-31T23:08:38.946 INFO:journalctl@ceph.mds.k.smithi142.stdout:Mar 31 23:08:38 smithi142 ceph-mds[64859]: -- 172.21.15.142:0/2655143290 <== mon.2 v2:172.21.15.142:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x555bdbf4ba40 con 0x555bdbf1e800 2024-03-31T23:08:38.946 INFO:journalctl@ceph.mds.k.smithi142.stdout:Mar 31 23:08:38 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mds-k[64855]: starting mds.k at 2024-03-31T23:08:38.946 INFO:journalctl@ceph.mds.k.smithi142.stdout:Mar 31 23:08:38 smithi142 bash[64840]: 7f5af54f1631112124998a5118513482bd5651538b1a1d1b6996134446f5a123 2024-03-31T23:08:38.946 INFO:journalctl@ceph.mds.k.smithi142.stdout:Mar 31 23:08:38 smithi142 systemd[1]: Started Ceph mds.k for 6a9d8f08-efb2-11ee-b647-cb9ed24678a4. 2024-03-31T23:08:38.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:38.944+0000 7f14aded4640 1 -- 172.21.15.46:0/3498468015 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f14a8108b60 con 0x7f14a8103350 2024-03-31T23:08:38.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:38.946+0000 7f14a4ff9640 1 -- 172.21.15.46:0/3498468015 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 16) ==== 137320+0+0 (secure 0 0 0) 0x7f14a00051d0 con 0x7f14a8103350 2024-03-31T23:08:38.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:38.946+0000 7f14a4ff9640 1 --2- 172.21.15.46:0/3498468015 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f14900a2520 0x7f14900a49e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:08:38.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:38.946+0000 7f14a4ff9640 1 -- 172.21.15.46:0/3498468015 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f14a00ce640 con 0x7f14a8103350 2024-03-31T23:08:38.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:38.946+0000 7f14a77fe640 1 --2- 172.21.15.46:0/3498468015 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f14900a2520 0x7f14900a49e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:08:38.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:38.947+0000 7f14a77fe640 1 --2- 172.21.15.46:0/3498468015 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f14900a2520 0x7f14900a49e0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f1498006010 tx=0x7f149802c7e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:08:38.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:38.949+0000 7f14a4ff9640 1 -- 172.21.15.46:0/3498468015 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f14a0014030 con 0x7f14a8103350 2024-03-31T23:08:39.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:39.173+0000 7f14aded4640 1 -- 172.21.15.46:0/3498468015 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f14a802d320 con 0x7f14a8103350 2024-03-31T23:08:39.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:39.173+0000 7f14a4ff9640 1 -- 172.21.15.46:0/3498468015 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) ==== 93+0+1396 (secure 0 0 0) 0x7f14a009bd20 con 0x7f14a8103350 2024-03-31T23:08:39.173 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-31T23:08:39.173 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":2,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24415,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6833/1746950966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6832","nonce":1746950966},{"type":"v1","addr":"172.21.15.162:6833","nonce":1746950966}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-03-31T23:08:39.174 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 2 2024-03-31T23:08:39.176 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:39.175+0000 7f14aded4640 1 -- 172.21.15.46:0/3498468015 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f14900a2520 msgr2=0x7f14900a49e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:08:39.176 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:39.175+0000 7f14aded4640 1 --2- 172.21.15.46:0/3498468015 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f14900a2520 0x7f14900a49e0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f1498006010 tx=0x7f149802c7e0 comp rx=0 tx=0).stop 2024-03-31T23:08:39.176 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:39.175+0000 7f14aded4640 1 -- 172.21.15.46:0/3498468015 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f14a8103350 msgr2=0x7f14a819efd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:08:39.176 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:39.175+0000 7f14aded4640 1 --2- 172.21.15.46:0/3498468015 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f14a8103350 0x7f14a819efd0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f14a000cad0 tx=0x7f14a000cf90 comp rx=0 tx=0).stop 2024-03-31T23:08:39.176 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:39.175+0000 7f14aded4640 1 -- 172.21.15.46:0/3498468015 shutdown_connections 2024-03-31T23:08:39.176 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:39.175+0000 7f14aded4640 1 --2- 172.21.15.46:0/3498468015 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f14900a2520 0x7f14900a49e0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:08:39.176 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:39.175+0000 7f14aded4640 1 --2- 172.21.15.46:0/3498468015 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f14a8103350 0x7f14a819efd0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:08:39.176 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:39.175+0000 7f14aded4640 1 --2- 172.21.15.46:0/3498468015 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f14a8102990 0x7f14a8197f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:08:39.176 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:39.175+0000 7f14aded4640 1 --2- 172.21.15.46:0/3498468015 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f14a81017a0 0x7f14a8197a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:08:39.176 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:39.175+0000 7f14aded4640 1 -- 172.21.15.46:0/3498468015 >> 172.21.15.46:0/3498468015 conn(0x7f14a80fccf0 msgr2=0x7f14a8105d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:08:39.176 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:39.175+0000 7f14aded4640 1 -- 172.21.15.46:0/3498468015 shutdown_connections 2024-03-31T23:08:39.177 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:39.175+0000 7f14aded4640 1 -- 172.21.15.46:0/3498468015 wait complete. 2024-03-31T23:08:39.862 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:08:39 smithi046 ceph-mon[32498]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-31T23:08:39.862 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:08:39 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:08:39.862 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:08:39 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:08:39.862 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:08:39 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:08:39.863 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:08:39 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-31T23:08:39.863 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:08:39 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-31T23:08:39.863 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:08:39 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:08:39.863 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:08:39 smithi046 ceph-mon[32498]: Deploying daemon mds.j on smithi046 2024-03-31T23:08:39.863 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:08:39 smithi046 ceph-mon[32498]: from='client.? 172.21.15.46:0/3498468015' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-31T23:08:39.863 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:08:39 smithi046 ceph-mon[32498]: mds.? [v2:172.21.15.142:6832/3394739181,v1:172.21.15.142:6833/3394739181] up:boot 2024-03-31T23:08:39.863 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:08:39 smithi046 ceph-mon[32498]: fsmap 2 up:standby 2024-03-31T23:08:39.863 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:08:39 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-31T23:08:39.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:08:39 smithi142 ceph-mon[35271]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-31T23:08:39.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:08:39 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:08:39.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:08:39 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:08:39.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:08:39 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:08:39.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:08:39 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-31T23:08:39.944 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:08:39 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-31T23:08:39.944 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:08:39 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:08:39.944 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:08:39 smithi142 ceph-mon[35271]: Deploying daemon mds.j on smithi046 2024-03-31T23:08:39.944 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:08:39 smithi142 ceph-mon[35271]: from='client.? 172.21.15.46:0/3498468015' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-31T23:08:39.944 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:08:39 smithi142 ceph-mon[35271]: mds.? [v2:172.21.15.142:6832/3394739181,v1:172.21.15.142:6833/3394739181] up:boot 2024-03-31T23:08:39.944 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:08:39 smithi142 ceph-mon[35271]: fsmap 2 up:standby 2024-03-31T23:08:39.944 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:08:39 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-31T23:08:40.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:08:39 smithi162 ceph-mon[36237]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-31T23:08:40.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:08:39 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:08:40.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:08:39 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:08:40.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:08:39 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:08:40.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:08:39 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-31T23:08:40.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:08:39 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-31T23:08:40.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:08:39 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:08:40.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:08:39 smithi162 ceph-mon[36237]: Deploying daemon mds.j on smithi046 2024-03-31T23:08:40.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:08:39 smithi162 ceph-mon[36237]: from='client.? 172.21.15.46:0/3498468015' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-31T23:08:40.047 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:08:39 smithi162 ceph-mon[36237]: mds.? [v2:172.21.15.142:6832/3394739181,v1:172.21.15.142:6833/3394739181] up:boot 2024-03-31T23:08:40.047 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:08:39 smithi162 ceph-mon[36237]: fsmap 2 up:standby 2024-03-31T23:08:40.047 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:08:39 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-31T23:08:41.111 INFO:journalctl@ceph.mds.j.smithi046.stdout:Mar 31 23:08:40 smithi046 systemd[1]: Starting Ceph mds.j for 6a9d8f08-efb2-11ee-b647-cb9ed24678a4... 2024-03-31T23:08:41.189 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-31T23:08:41.298 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:41.295+0000 7f8f0f343640 1 -- 172.21.15.46:0/3835987230 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8f08100690 msgr2=0x7f8f08100ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:08:41.298 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:41.295+0000 7f8f0f343640 1 --2- 172.21.15.46:0/3835987230 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8f08100690 0x7f8f08100ab0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f8f00009900 tx=0x7f8f0002bc30 comp rx=0 tx=0).stop 2024-03-31T23:08:41.298 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:41.295+0000 7f8f0f343640 1 -- 172.21.15.46:0/3835987230 shutdown_connections 2024-03-31T23:08:41.298 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:41.295+0000 7f8f0f343640 1 --2- 172.21.15.46:0/3835987230 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8f08102240 0x7f8f08108ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:08:41.298 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:41.295+0000 7f8f0f343640 1 --2- 172.21.15.46:0/3835987230 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8f08101880 0x7f8f08101d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:08:41.298 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:41.295+0000 7f8f0f343640 1 --2- 172.21.15.46:0/3835987230 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8f08100690 0x7f8f08100ab0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:08:41.298 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:41.295+0000 7f8f0f343640 1 -- 172.21.15.46:0/3835987230 >> 172.21.15.46:0/3835987230 conn(0x7f8f080fbc00 msgr2=0x7f8f080fe040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:08:41.298 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:41.296+0000 7f8f0f343640 1 -- 172.21.15.46:0/3835987230 shutdown_connections 2024-03-31T23:08:41.298 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:41.296+0000 7f8f0f343640 1 -- 172.21.15.46:0/3835987230 wait complete. 2024-03-31T23:08:41.298 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:41.296+0000 7f8f0f343640 1 Processor -- start 2024-03-31T23:08:41.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:41.296+0000 7f8f0f343640 1 -- start start 2024-03-31T23:08:41.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:41.296+0000 7f8f0f343640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8f08100690 0x7f8f0806a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:08:41.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:41.296+0000 7f8f0f343640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8f08101880 0x7f8f0819c6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:08:41.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:41.296+0000 7f8f0f343640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8f08102240 0x7f8f0819ecd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:08:41.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:41.296+0000 7f8f0f343640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f8f0810c1e0 con 0x7f8f08102240 2024-03-31T23:08:41.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:41.296+0000 7f8f0f343640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f8f0810c060 con 0x7f8f08101880 2024-03-31T23:08:41.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:41.296+0000 7f8f0f343640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f8f0810c360 con 0x7f8f08100690 2024-03-31T23:08:41.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:41.296+0000 7f8f0d8e0640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8f08102240 0x7f8f0819ecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:08:41.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:41.297+0000 7f8f0d8e0640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8f08102240 0x7f8f0819ecd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:54378/0 (socket says 172.21.15.46:54378) 2024-03-31T23:08:41.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:41.297+0000 7f8f0d8e0640 1 -- 172.21.15.46:0/3277281668 learned_addr learned my addr 172.21.15.46:0/3277281668 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:08:41.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:41.297+0000 7f8f0c8de640 1 --2- 172.21.15.46:0/3277281668 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8f08101880 0x7f8f0819c6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:08:41.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:41.297+0000 7f8f0d0df640 1 --2- 172.21.15.46:0/3277281668 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8f08100690 0x7f8f0806a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:08:41.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:41.297+0000 7f8f0d8e0640 1 -- 172.21.15.46:0/3277281668 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8f08100690 msgr2=0x7f8f0806a8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:08:41.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:41.297+0000 7f8f0d8e0640 1 --2- 172.21.15.46:0/3277281668 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8f08100690 0x7f8f0806a8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:08:41.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:41.297+0000 7f8f0d8e0640 1 -- 172.21.15.46:0/3277281668 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8f08101880 msgr2=0x7f8f0819c6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:08:41.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:41.297+0000 7f8f0d8e0640 1 --2- 172.21.15.46:0/3277281668 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8f08101880 0x7f8f0819c6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:08:41.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:41.297+0000 7f8f0d8e0640 1 -- 172.21.15.46:0/3277281668 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8f0819f5e0 con 0x7f8f08102240 2024-03-31T23:08:41.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:41.297+0000 7f8f0d8e0640 1 --2- 172.21.15.46:0/3277281668 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8f08102240 0x7f8f0819ecd0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f8ef800eaa0 tx=0x7f8ef800eea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:08:41.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:41.297+0000 7f8f067fc640 1 -- 172.21.15.46:0/3277281668 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ef8004d20 con 0x7f8f08102240 2024-03-31T23:08:41.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:41.297+0000 7f8f067fc640 1 -- 172.21.15.46:0/3277281668 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8ef8005020 con 0x7f8f08102240 2024-03-31T23:08:41.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:41.297+0000 7f8f0f343640 1 -- 172.21.15.46:0/3277281668 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8f0819f870 con 0x7f8f08102240 2024-03-31T23:08:41.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:41.297+0000 7f8f067fc640 1 -- 172.21.15.46:0/3277281668 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ef80116b0 con 0x7f8f08102240 2024-03-31T23:08:41.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:41.297+0000 7f8f0f343640 1 -- 172.21.15.46:0/3277281668 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8f0819fdc0 con 0x7f8f08102240 2024-03-31T23:08:41.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:41.298+0000 7f8f0f343640 1 -- 172.21.15.46:0/3277281668 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8f081079b0 con 0x7f8f08102240 2024-03-31T23:08:41.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:41.299+0000 7f8f067fc640 1 -- 172.21.15.46:0/3277281668 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 16) ==== 137320+0+0 (secure 0 0 0) 0x7f8ef80051d0 con 0x7f8f08102240 2024-03-31T23:08:41.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:41.300+0000 7f8f067fc640 1 --2- 172.21.15.46:0/3277281668 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f8ef00a2520 0x7f8ef00a49e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:08:41.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:41.300+0000 7f8f0d0df640 1 --2- 172.21.15.46:0/3277281668 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f8ef00a2520 0x7f8ef00a49e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:08:41.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:41.300+0000 7f8f067fc640 1 -- 172.21.15.46:0/3277281668 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f8ef80ce520 con 0x7f8f08102240 2024-03-31T23:08:41.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:41.300+0000 7f8f0d0df640 1 --2- 172.21.15.46:0/3277281668 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f8ef00a2520 0x7f8ef00a49e0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f8f000096c0 tx=0x7f8f00005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:08:41.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:41.305+0000 7f8f067fc640 1 -- 172.21.15.46:0/3277281668 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f8ef8016030 con 0x7f8f08102240 2024-03-31T23:08:41.445 INFO:journalctl@ceph.mds.j.smithi046.stdout:Mar 31 23:08:41 smithi046 podman[72684]: 2024-03-31 23:08:41.080788788 +0000 UTC m=+0.017846791 image pull ec18d7baa60e8b1a0b126fbe5306acce11b1ed0ec5e3bcdddbf39b0a3c591791 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175 2024-03-31T23:08:41.446 INFO:journalctl@ceph.mds.j.smithi046.stdout:Mar 31 23:08:41 smithi046 podman[72684]: 2024-03-31 23:08:41.200870291 +0000 UTC m=+0.137928277 container create 85b09f805600e0c06cc942a5c6e2f23ecb5eb992bc83785e7914324991fb49e4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mds-j, RELEASE=squid-384f31f, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240326, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5) 2024-03-31T23:08:41.446 INFO:journalctl@ceph.mds.j.smithi046.stdout:Mar 31 23:08:41 smithi046 podman[72684]: 2024-03-31 23:08:41.445378221 +0000 UTC m=+0.382436221 container init 85b09f805600e0c06cc942a5c6e2f23ecb5eb992bc83785e7914324991fb49e4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mds-j, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-384f31f, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240326, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5) 2024-03-31T23:08:41.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:41.533+0000 7f8f0f343640 1 -- 172.21.15.46:0/3277281668 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8f08062380 con 0x7f8f08102240 2024-03-31T23:08:41.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:41.533+0000 7f8f067fc640 1 -- 172.21.15.46:0/3277281668 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) ==== 93+0+2224 (secure 0 0 0) 0x7f8ef809bae0 con 0x7f8f08102240 2024-03-31T23:08:41.534 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-31T23:08:41.534 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24415,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6833/1746950966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6832","nonce":1746950966},{"type":"v1","addr":"172.21.15.162:6833","nonce":1746950966}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24422,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6833/3394739181","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6832","nonce":3394739181},{"type":"v1","addr":"172.21.15.142:6833","nonce":3394739181}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[]} 2024-03-31T23:08:41.534 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 3 2024-03-31T23:08:41.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:41.536+0000 7f8f0f343640 1 -- 172.21.15.46:0/3277281668 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f8ef00a2520 msgr2=0x7f8ef00a49e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:08:41.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:41.536+0000 7f8f0f343640 1 --2- 172.21.15.46:0/3277281668 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f8ef00a2520 0x7f8ef00a49e0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f8f000096c0 tx=0x7f8f00005fd0 comp rx=0 tx=0).stop 2024-03-31T23:08:41.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:41.536+0000 7f8f0f343640 1 -- 172.21.15.46:0/3277281668 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8f08102240 msgr2=0x7f8f0819ecd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:08:41.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:41.536+0000 7f8f0f343640 1 --2- 172.21.15.46:0/3277281668 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8f08102240 0x7f8f0819ecd0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f8ef800eaa0 tx=0x7f8ef800eea0 comp rx=0 tx=0).stop 2024-03-31T23:08:41.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:41.536+0000 7f8f0f343640 1 -- 172.21.15.46:0/3277281668 shutdown_connections 2024-03-31T23:08:41.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:41.536+0000 7f8f0f343640 1 --2- 172.21.15.46:0/3277281668 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f8ef00a2520 0x7f8ef00a49e0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:08:41.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:41.536+0000 7f8f0f343640 1 --2- 172.21.15.46:0/3277281668 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8f08102240 0x7f8f0819ecd0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:08:41.537 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:41.536+0000 7f8f0f343640 1 --2- 172.21.15.46:0/3277281668 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8f08101880 0x7f8f0819c6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:08:41.537 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:41.536+0000 7f8f0f343640 1 --2- 172.21.15.46:0/3277281668 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8f08100690 0x7f8f0806a8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:08:41.537 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:41.536+0000 7f8f0f343640 1 -- 172.21.15.46:0/3277281668 >> 172.21.15.46:0/3277281668 conn(0x7f8f080fbc00 msgr2=0x7f8f08104b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:08:41.537 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:41.536+0000 7f8f0f343640 1 -- 172.21.15.46:0/3277281668 shutdown_connections 2024-03-31T23:08:41.537 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:41.536+0000 7f8f0f343640 1 -- 172.21.15.46:0/3277281668 wait complete. 2024-03-31T23:08:41.698 INFO:journalctl@ceph.mds.j.smithi046.stdout:Mar 31 23:08:41 smithi046 podman[72684]: 2024-03-31 23:08:41.448418208 +0000 UTC m=+0.385476195 container start 85b09f805600e0c06cc942a5c6e2f23ecb5eb992bc83785e7914324991fb49e4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mds-j, RELEASE=squid-384f31f, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240326, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, ceph=True) 2024-03-31T23:08:41.698 INFO:journalctl@ceph.mds.j.smithi046.stdout:Mar 31 23:08:41 smithi046 ceph-mds[72742]: -- 172.21.15.46:0/706493056 <== mon.0 v2:172.21.15.46:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x559e0b87da40 con 0x559e0b850800 2024-03-31T23:08:41.698 INFO:journalctl@ceph.mds.j.smithi046.stdout:Mar 31 23:08:41 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mds-j[72738]: starting mds.j at 2024-03-31T23:08:41.699 INFO:journalctl@ceph.mds.j.smithi046.stdout:Mar 31 23:08:41 smithi046 bash[72684]: 85b09f805600e0c06cc942a5c6e2f23ecb5eb992bc83785e7914324991fb49e4 2024-03-31T23:08:41.699 INFO:journalctl@ceph.mds.j.smithi046.stdout:Mar 31 23:08:41 smithi046 systemd[1]: Started Ceph mds.j for 6a9d8f08-efb2-11ee-b647-cb9ed24678a4. 2024-03-31T23:08:42.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:08:41 smithi162 ceph-mon[36237]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-31T23:08:42.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:08:41 smithi162 ceph-mon[36237]: from='client.? 172.21.15.46:0/3277281668' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-31T23:08:42.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:08:41 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:08:42.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:08:41 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:08:42.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:08:41 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:08:42.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:08:41 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-31T23:08:42.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:08:41 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-31T23:08:42.047 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:08:41 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:08:42.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:08:41 smithi046 ceph-mon[32498]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-31T23:08:42.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:08:41 smithi046 ceph-mon[32498]: from='client.? 172.21.15.46:0/3277281668' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-31T23:08:42.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:08:41 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:08:42.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:08:41 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:08:42.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:08:41 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:08:42.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:08:41 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-31T23:08:42.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:08:41 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-31T23:08:42.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:08:41 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:08:42.194 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:08:41 smithi142 ceph-mon[35271]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-31T23:08:42.194 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:08:41 smithi142 ceph-mon[35271]: from='client.? 172.21.15.46:0/3277281668' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-31T23:08:42.194 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:08:41 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:08:42.194 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:08:41 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:08:42.194 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:08:41 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:08:42.194 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:08:41 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-31T23:08:42.194 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:08:41 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-31T23:08:42.194 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:08:41 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:08:42.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:08:42 smithi046 ceph-mon[32498]: Deploying daemon mds.d on smithi046 2024-03-31T23:08:42.862 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:08:42 smithi046 ceph-mon[32498]: mds.? [v2:172.21.15.46:6834/2169018547,v1:172.21.15.46:6835/2169018547] up:boot 2024-03-31T23:08:42.862 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:08:42 smithi046 ceph-mon[32498]: fsmap 3 up:standby 2024-03-31T23:08:42.862 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:08:42 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-31T23:08:43.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:08:42 smithi162 ceph-mon[36237]: Deploying daemon mds.d on smithi046 2024-03-31T23:08:43.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:08:42 smithi162 ceph-mon[36237]: mds.? [v2:172.21.15.46:6834/2169018547,v1:172.21.15.46:6835/2169018547] up:boot 2024-03-31T23:08:43.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:08:42 smithi162 ceph-mon[36237]: fsmap 3 up:standby 2024-03-31T23:08:43.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:08:42 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-31T23:08:43.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:08:42 smithi142 ceph-mon[35271]: Deploying daemon mds.d on smithi046 2024-03-31T23:08:43.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:08:42 smithi142 ceph-mon[35271]: mds.? [v2:172.21.15.46:6834/2169018547,v1:172.21.15.46:6835/2169018547] up:boot 2024-03-31T23:08:43.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:08:42 smithi142 ceph-mon[35271]: fsmap 3 up:standby 2024-03-31T23:08:43.194 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:08:42 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-31T23:08:43.547 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-31T23:08:43.736 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:43.733+0000 7f7d6064e640 1 -- 172.21.15.46:0/1526893601 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7d58104ba0 msgr2=0x7f7d58106f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:08:43.736 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:43.733+0000 7f7d6064e640 1 --2- 172.21.15.46:0/1526893601 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7d58104ba0 0x7f7d58106f90 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f7d48009900 tx=0x7f7d4802bc30 comp rx=0 tx=0).stop 2024-03-31T23:08:43.736 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:43.733+0000 7f7d6064e640 1 -- 172.21.15.46:0/1526893601 shutdown_connections 2024-03-31T23:08:43.736 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:43.733+0000 7f7d6064e640 1 --2- 172.21.15.46:0/1526893601 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7d581074d0 0x7f7d581098c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:08:43.736 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:43.733+0000 7f7d6064e640 1 --2- 172.21.15.46:0/1526893601 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7d58104ba0 0x7f7d58106f90 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:08:43.736 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:43.733+0000 7f7d6064e640 1 --2- 172.21.15.46:0/1526893601 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7d58102270 0x7f7d58104660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:08:43.736 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:43.733+0000 7f7d6064e640 1 -- 172.21.15.46:0/1526893601 >> 172.21.15.46:0/1526893601 conn(0x7f7d580fbbe0 msgr2=0x7f7d580fe020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:08:43.736 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:43.734+0000 7f7d6064e640 1 -- 172.21.15.46:0/1526893601 shutdown_connections 2024-03-31T23:08:43.736 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:43.734+0000 7f7d6064e640 1 -- 172.21.15.46:0/1526893601 wait complete. 2024-03-31T23:08:43.736 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:43.734+0000 7f7d6064e640 1 Processor -- start 2024-03-31T23:08:43.736 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:43.734+0000 7f7d6064e640 1 -- start start 2024-03-31T23:08:43.736 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:43.734+0000 7f7d6064e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7d58102270 0x7f7d581979a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:08:43.737 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:43.734+0000 7f7d6064e640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7d58104ba0 0x7f7d58197ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:08:43.737 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:43.734+0000 7f7d6064e640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7d581074d0 0x7f7d5819ef50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:08:43.737 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:43.734+0000 7f7d6064e640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f7d5810d270 con 0x7f7d58102270 2024-03-31T23:08:43.737 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:43.734+0000 7f7d6064e640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f7d5810d0f0 con 0x7f7d581074d0 2024-03-31T23:08:43.737 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:43.734+0000 7f7d6064e640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f7d5810d3f0 con 0x7f7d58104ba0 2024-03-31T23:08:43.737 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:43.734+0000 7f7d5e3ea640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7d58102270 0x7f7d581979a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:08:43.737 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:43.734+0000 7f7d5e3ea640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7d58102270 0x7f7d581979a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:33512/0 (socket says 172.21.15.46:33512) 2024-03-31T23:08:43.737 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:43.734+0000 7f7d5e3ea640 1 -- 172.21.15.46:0/1685959976 learned_addr learned my addr 172.21.15.46:0/1685959976 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:08:43.737 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:43.735+0000 7f7d5ebeb640 1 --2- 172.21.15.46:0/1685959976 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7d581074d0 0x7f7d5819ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:08:43.737 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:43.735+0000 7f7d5dbe9640 1 --2- 172.21.15.46:0/1685959976 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7d58104ba0 0x7f7d58197ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:08:43.737 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:43.735+0000 7f7d5e3ea640 1 -- 172.21.15.46:0/1685959976 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7d58104ba0 msgr2=0x7f7d58197ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:08:43.737 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:43.735+0000 7f7d5e3ea640 1 --2- 172.21.15.46:0/1685959976 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7d58104ba0 0x7f7d58197ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:08:43.737 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:43.735+0000 7f7d5e3ea640 1 -- 172.21.15.46:0/1685959976 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7d581074d0 msgr2=0x7f7d5819ef50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:08:43.737 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:43.735+0000 7f7d5e3ea640 1 --2- 172.21.15.46:0/1685959976 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7d581074d0 0x7f7d5819ef50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:08:43.738 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:43.735+0000 7f7d5e3ea640 1 -- 172.21.15.46:0/1685959976 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d5819f5c0 con 0x7f7d58102270 2024-03-31T23:08:43.738 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:43.735+0000 7f7d5e3ea640 1 --2- 172.21.15.46:0/1685959976 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7d58102270 0x7f7d581979a0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f7d5400c890 tx=0x7f7d5400cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:08:43.738 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:43.735+0000 7f7d4f7fe640 1 -- 172.21.15.46:0/1685959976 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d54004220 con 0x7f7d58102270 2024-03-31T23:08:43.738 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:43.735+0000 7f7d6064e640 1 -- 172.21.15.46:0/1685959976 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7d5819f8b0 con 0x7f7d58102270 2024-03-31T23:08:43.738 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:43.735+0000 7f7d4f7fe640 1 -- 172.21.15.46:0/1685959976 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7d540043c0 con 0x7f7d58102270 2024-03-31T23:08:43.738 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:43.735+0000 7f7d4f7fe640 1 -- 172.21.15.46:0/1685959976 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d54004a00 con 0x7f7d58102270 2024-03-31T23:08:43.738 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:43.735+0000 7f7d6064e640 1 -- 172.21.15.46:0/1685959976 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7d5819fe50 con 0x7f7d58102270 2024-03-31T23:08:43.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:43.736+0000 7f7d6064e640 1 -- 172.21.15.46:0/1685959976 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7d580fd6d0 con 0x7f7d58102270 2024-03-31T23:08:43.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:43.737+0000 7f7d4f7fe640 1 -- 172.21.15.46:0/1685959976 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 16) ==== 137320+0+0 (secure 0 0 0) 0x7f7d54004ba0 con 0x7f7d58102270 2024-03-31T23:08:43.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:43.738+0000 7f7d4f7fe640 1 --2- 172.21.15.46:0/1685959976 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f7d440a24a0 0x7f7d440a4960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:08:43.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:43.738+0000 7f7d5dbe9640 1 --2- 172.21.15.46:0/1685959976 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f7d440a24a0 0x7f7d440a4960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:08:43.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:43.738+0000 7f7d4f7fe640 1 -- 172.21.15.46:0/1685959976 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f7d540cdfe0 con 0x7f7d58102270 2024-03-31T23:08:43.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:43.738+0000 7f7d5dbe9640 1 --2- 172.21.15.46:0/1685959976 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f7d440a24a0 0x7f7d440a4960 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f7d4802c840 tx=0x7f7d48000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:08:43.742 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:43.742+0000 7f7d4f7fe640 1 -- 172.21.15.46:0/1685959976 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f7d54014030 con 0x7f7d58102270 2024-03-31T23:08:43.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:43.972+0000 7f7d6064e640 1 -- 172.21.15.46:0/1685959976 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7d5802d400 con 0x7f7d58102270 2024-03-31T23:08:43.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:43.972+0000 7f7d4f7fe640 1 -- 172.21.15.46:0/1685959976 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 4 v4) ==== 93+0+3049 (secure 0 0 0) 0x7f7d5409b640 con 0x7f7d58102270 2024-03-31T23:08:43.973 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-31T23:08:43.973 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":4,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/2169018547","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":2169018547},{"type":"v1","addr":"172.21.15.46:6835","nonce":2169018547}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24415,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6833/1746950966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6832","nonce":1746950966},{"type":"v1","addr":"172.21.15.162:6833","nonce":1746950966}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24422,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6833/3394739181","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6832","nonce":3394739181},{"type":"v1","addr":"172.21.15.142:6833","nonce":3394739181}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[]} 2024-03-31T23:08:43.973 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 4 2024-03-31T23:08:43.975 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:43.974+0000 7f7d6064e640 1 -- 172.21.15.46:0/1685959976 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f7d440a24a0 msgr2=0x7f7d440a4960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:08:43.975 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:43.974+0000 7f7d6064e640 1 --2- 172.21.15.46:0/1685959976 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f7d440a24a0 0x7f7d440a4960 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f7d4802c840 tx=0x7f7d48000bc0 comp rx=0 tx=0).stop 2024-03-31T23:08:43.975 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:43.974+0000 7f7d6064e640 1 -- 172.21.15.46:0/1685959976 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7d58102270 msgr2=0x7f7d581979a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:08:43.975 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:43.974+0000 7f7d6064e640 1 --2- 172.21.15.46:0/1685959976 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7d58102270 0x7f7d581979a0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f7d5400c890 tx=0x7f7d5400cd50 comp rx=0 tx=0).stop 2024-03-31T23:08:43.976 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:43.974+0000 7f7d6064e640 1 -- 172.21.15.46:0/1685959976 shutdown_connections 2024-03-31T23:08:43.976 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:43.974+0000 7f7d6064e640 1 --2- 172.21.15.46:0/1685959976 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f7d440a24a0 0x7f7d440a4960 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:08:43.976 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:43.974+0000 7f7d6064e640 1 --2- 172.21.15.46:0/1685959976 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7d581074d0 0x7f7d5819ef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:08:43.976 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:43.974+0000 7f7d6064e640 1 --2- 172.21.15.46:0/1685959976 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7d58104ba0 0x7f7d58197ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:08:43.976 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:43.974+0000 7f7d6064e640 1 --2- 172.21.15.46:0/1685959976 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7d58102270 0x7f7d581979a0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:08:43.976 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:43.975+0000 7f7d6064e640 1 -- 172.21.15.46:0/1685959976 >> 172.21.15.46:0/1685959976 conn(0x7f7d580fbbe0 msgr2=0x7f7d58102d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:08:43.976 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:43.975+0000 7f7d6064e640 1 -- 172.21.15.46:0/1685959976 shutdown_connections 2024-03-31T23:08:43.976 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:43.975+0000 7f7d6064e640 1 -- 172.21.15.46:0/1685959976 wait complete. 2024-03-31T23:08:43.987 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:08:43 smithi046 ceph-mon[32498]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-31T23:08:43.987 INFO:journalctl@ceph.mds.d.smithi046.stdout:Mar 31 23:08:43 smithi046 systemd[1]: Starting Ceph mds.d for 6a9d8f08-efb2-11ee-b647-cb9ed24678a4... 2024-03-31T23:08:44.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:08:43 smithi162 ceph-mon[36237]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-31T23:08:44.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:08:43 smithi142 ceph-mon[35271]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-31T23:08:44.243 INFO:journalctl@ceph.mds.d.smithi046.stdout:Mar 31 23:08:44 smithi046 podman[73071]: 2024-03-31 23:08:43.975925876 +0000 UTC m=+0.018114303 image pull ec18d7baa60e8b1a0b126fbe5306acce11b1ed0ec5e3bcdddbf39b0a3c591791 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175 2024-03-31T23:08:44.243 INFO:journalctl@ceph.mds.d.smithi046.stdout:Mar 31 23:08:44 smithi046 podman[73071]: 2024-03-31 23:08:44.107153596 +0000 UTC m=+0.149342009 container create 70ad99574a650e8671f156263068ada2ba0b90982c33115aba4922ef80e522a9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mds-d, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=squid-384f31f, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240326, io.buildah.version=1.33.5) 2024-03-31T23:08:44.611 INFO:journalctl@ceph.mds.d.smithi046.stdout:Mar 31 23:08:44 smithi046 podman[73071]: 2024-03-31 23:08:44.359970906 +0000 UTC m=+0.402159335 container init 70ad99574a650e8671f156263068ada2ba0b90982c33115aba4922ef80e522a9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mds-d, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=squid-384f31f, ceph=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240326, maintainer=Guillaume Abrioux , GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-03-31T23:08:44.612 INFO:journalctl@ceph.mds.d.smithi046.stdout:Mar 31 23:08:44 smithi046 podman[73071]: 2024-03-31 23:08:44.362579185 +0000 UTC m=+0.404767608 container start 70ad99574a650e8671f156263068ada2ba0b90982c33115aba4922ef80e522a9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mds-d, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240326, RELEASE=squid-384f31f, maintainer=Guillaume Abrioux ) 2024-03-31T23:08:44.612 INFO:journalctl@ceph.mds.d.smithi046.stdout:Mar 31 23:08:44 smithi046 ceph-mds[73092]: -- 172.21.15.46:0/2454844584 <== mon.0 v2:172.21.15.46:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5609cd215a40 con 0x5609cd1e8800 2024-03-31T23:08:44.612 INFO:journalctl@ceph.mds.d.smithi046.stdout:Mar 31 23:08:44 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mds-d[73088]: starting mds.d at 2024-03-31T23:08:44.612 INFO:journalctl@ceph.mds.d.smithi046.stdout:Mar 31 23:08:44 smithi046 bash[73071]: 70ad99574a650e8671f156263068ada2ba0b90982c33115aba4922ef80e522a9 2024-03-31T23:08:44.612 INFO:journalctl@ceph.mds.d.smithi046.stdout:Mar 31 23:08:44 smithi046 systemd[1]: Started Ceph mds.d for 6a9d8f08-efb2-11ee-b647-cb9ed24678a4. 2024-03-31T23:08:45.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:08:44 smithi162 ceph-mon[36237]: from='client.? 172.21.15.46:0/1685959976' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-31T23:08:45.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:08:44 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:08:45.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:08:44 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:08:45.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:08:44 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:08:45.047 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:08:44 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-31T23:08:45.047 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:08:44 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-31T23:08:45.047 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:08:44 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:08:45.047 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:08:44 smithi162 ceph-mon[36237]: Deploying daemon mds.e on smithi142 2024-03-31T23:08:45.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:08:44 smithi046 ceph-mon[32498]: from='client.? 172.21.15.46:0/1685959976' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-31T23:08:45.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:08:44 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:08:45.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:08:44 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:08:45.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:08:44 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:08:45.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:08:44 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-31T23:08:45.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:08:44 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-31T23:08:45.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:08:44 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:08:45.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:08:44 smithi046 ceph-mon[32498]: Deploying daemon mds.e on smithi142 2024-03-31T23:08:45.130 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:08:44 smithi142 ceph-mon[35271]: from='client.? 172.21.15.46:0/1685959976' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-31T23:08:45.130 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:08:44 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:08:45.130 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:08:44 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:08:45.130 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:08:44 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:08:45.130 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:08:44 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-31T23:08:45.130 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:08:44 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-31T23:08:45.130 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:08:44 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:08:45.131 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:08:44 smithi142 ceph-mon[35271]: Deploying daemon mds.e on smithi142 2024-03-31T23:08:45.990 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-31T23:08:46.022 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:08:45 smithi046 ceph-mon[32498]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-31T23:08:46.022 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:08:45 smithi046 ceph-mon[32498]: mds.? [v2:172.21.15.46:6836/1069387756,v1:172.21.15.46:6837/1069387756] up:boot 2024-03-31T23:08:46.022 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:08:45 smithi046 ceph-mon[32498]: fsmap 4 up:standby 2024-03-31T23:08:46.022 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:08:45 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-31T23:08:46.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:08:45 smithi162 ceph-mon[36237]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-31T23:08:46.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:08:45 smithi162 ceph-mon[36237]: mds.? [v2:172.21.15.46:6836/1069387756,v1:172.21.15.46:6837/1069387756] up:boot 2024-03-31T23:08:46.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:08:45 smithi162 ceph-mon[36237]: fsmap 4 up:standby 2024-03-31T23:08:46.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:08:45 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-31T23:08:46.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:46.110+0000 7f5a178c9640 1 -- 172.21.15.46:0/2829109873 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5a10102950 msgr2=0x7f5a10104dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:08:46.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:46.110+0000 7f5a178c9640 1 --2- 172.21.15.46:0/2829109873 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5a10102950 0x7f5a10104dc0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f5a00009900 tx=0x7f5a0002bc30 comp rx=0 tx=0).stop 2024-03-31T23:08:46.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:46.110+0000 7f5a178c9640 1 -- 172.21.15.46:0/2829109873 shutdown_connections 2024-03-31T23:08:46.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:46.110+0000 7f5a178c9640 1 --2- 172.21.15.46:0/2829109873 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5a10105300 0x7f5a10105780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:08:46.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:46.110+0000 7f5a178c9640 1 --2- 172.21.15.46:0/2829109873 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5a10102950 0x7f5a10104dc0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:08:46.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:46.110+0000 7f5a178c9640 1 --2- 172.21.15.46:0/2829109873 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5a100fffe0 0x7f5a10102410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:08:46.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:46.110+0000 7f5a178c9640 1 -- 172.21.15.46:0/2829109873 >> 172.21.15.46:0/2829109873 conn(0x7f5a100fbbe0 msgr2=0x7f5a100fe020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:08:46.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:46.110+0000 7f5a178c9640 1 -- 172.21.15.46:0/2829109873 shutdown_connections 2024-03-31T23:08:46.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:46.110+0000 7f5a178c9640 1 -- 172.21.15.46:0/2829109873 wait complete. 2024-03-31T23:08:46.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:46.111+0000 7f5a178c9640 1 Processor -- start 2024-03-31T23:08:46.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:46.111+0000 7f5a178c9640 1 -- start start 2024-03-31T23:08:46.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:46.111+0000 7f5a178c9640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5a100fffe0 0x7f5a1019dff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:08:46.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:46.111+0000 7f5a178c9640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5a10102950 0x7f5a1019e530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:08:46.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:46.111+0000 7f5a178c9640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5a10105300 0x7f5a101a55a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:08:46.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:46.111+0000 7f5a178c9640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f5a101138f0 con 0x7f5a10102950 2024-03-31T23:08:46.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:46.111+0000 7f5a178c9640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f5a10113770 con 0x7f5a10105300 2024-03-31T23:08:46.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:46.111+0000 7f5a178c9640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f5a10113a70 con 0x7f5a100fffe0 2024-03-31T23:08:46.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:46.111+0000 7f5a14e64640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5a10102950 0x7f5a1019e530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:08:46.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:46.111+0000 7f5a14e64640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5a10102950 0x7f5a1019e530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:33560/0 (socket says 172.21.15.46:33560) 2024-03-31T23:08:46.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:46.111+0000 7f5a14e64640 1 -- 172.21.15.46:0/3917176136 learned_addr learned my addr 172.21.15.46:0/3917176136 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:08:46.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:46.111+0000 7f5a15665640 1 --2- 172.21.15.46:0/3917176136 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5a100fffe0 0x7f5a1019dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:08:46.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:46.111+0000 7f5a15e66640 1 --2- 172.21.15.46:0/3917176136 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5a10105300 0x7f5a101a55a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:08:46.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:46.111+0000 7f5a14e64640 1 -- 172.21.15.46:0/3917176136 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5a100fffe0 msgr2=0x7f5a1019dff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:08:46.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:46.111+0000 7f5a14e64640 1 --2- 172.21.15.46:0/3917176136 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5a100fffe0 0x7f5a1019dff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:08:46.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:46.111+0000 7f5a14e64640 1 -- 172.21.15.46:0/3917176136 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5a10105300 msgr2=0x7f5a101a55a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:08:46.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:46.111+0000 7f5a14e64640 1 --2- 172.21.15.46:0/3917176136 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5a10105300 0x7f5a101a55a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:08:46.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:46.111+0000 7f5a14e64640 1 -- 172.21.15.46:0/3917176136 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a101a5c10 con 0x7f5a10102950 2024-03-31T23:08:46.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:46.111+0000 7f5a15665640 1 --2- 172.21.15.46:0/3917176136 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5a100fffe0 0x7f5a1019dff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T23:08:46.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:46.111+0000 7f5a15e66640 1 --2- 172.21.15.46:0/3917176136 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5a10105300 0x7f5a101a55a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T23:08:46.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:46.112+0000 7f5a14e64640 1 --2- 172.21.15.46:0/3917176136 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5a10102950 0x7f5a1019e530 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f5a00006010 tx=0x7f5a000042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:08:46.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:46.112+0000 7f5a067fc640 1 -- 172.21.15.46:0/3917176136 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a00043020 con 0x7f5a10102950 2024-03-31T23:08:46.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:46.112+0000 7f5a067fc640 1 -- 172.21.15.46:0/3917176136 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5a00004aa0 con 0x7f5a10102950 2024-03-31T23:08:46.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:46.112+0000 7f5a178c9640 1 -- 172.21.15.46:0/3917176136 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5a101a5ea0 con 0x7f5a10102950 2024-03-31T23:08:46.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:46.112+0000 7f5a067fc640 1 -- 172.21.15.46:0/3917176136 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a00003c60 con 0x7f5a10102950 2024-03-31T23:08:46.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:46.112+0000 7f5a178c9640 1 -- 172.21.15.46:0/3917176136 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5a101a63e0 con 0x7f5a10102950 2024-03-31T23:08:46.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:46.113+0000 7f5a178c9640 1 -- 172.21.15.46:0/3917176136 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5a100fd6d0 con 0x7f5a10102950 2024-03-31T23:08:46.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:46.114+0000 7f5a067fc640 1 -- 172.21.15.46:0/3917176136 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 16) ==== 137320+0+0 (secure 0 0 0) 0x7f5a00046020 con 0x7f5a10102950 2024-03-31T23:08:46.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:46.115+0000 7f5a067fc640 1 --2- 172.21.15.46:0/3917176136 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f59fc0a24a0 0x7f59fc0a4960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:08:46.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:46.115+0000 7f5a15665640 1 --2- 172.21.15.46:0/3917176136 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f59fc0a24a0 0x7f59fc0a4960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:08:46.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:46.115+0000 7f5a067fc640 1 -- 172.21.15.46:0/3917176136 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f5a000f0cd0 con 0x7f5a10102950 2024-03-31T23:08:46.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:46.115+0000 7f5a15665640 1 --2- 172.21.15.46:0/3917176136 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f59fc0a24a0 0x7f59fc0a4960 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f5a0c0058c0 tx=0x7f5a0c005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:08:46.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:46.118+0000 7f5a067fc640 1 -- 172.21.15.46:0/3917176136 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f5a0004d020 con 0x7f5a10102950 2024-03-31T23:08:46.155 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:08:45 smithi142 ceph-mon[35271]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-31T23:08:46.156 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:08:45 smithi142 ceph-mon[35271]: mds.? [v2:172.21.15.46:6836/1069387756,v1:172.21.15.46:6837/1069387756] up:boot 2024-03-31T23:08:46.156 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:08:45 smithi142 ceph-mon[35271]: fsmap 4 up:standby 2024-03-31T23:08:46.156 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:08:45 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-31T23:08:46.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:46.342+0000 7f5a178c9640 1 -- 172.21.15.46:0/3917176136 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5a1002d370 con 0x7f5a10102950 2024-03-31T23:08:46.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:46.342+0000 7f5a067fc640 1 -- 172.21.15.46:0/3917176136 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 5 v5) ==== 93+0+3874 (secure 0 0 0) 0x7f5a000be330 con 0x7f5a10102950 2024-03-31T23:08:46.343 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-31T23:08:46.343 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":5,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/2169018547","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":2169018547},{"type":"v1","addr":"172.21.15.46:6835","nonce":2169018547}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6837/1069387756","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6836","nonce":1069387756},{"type":"v1","addr":"172.21.15.46:6837","nonce":1069387756}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24415,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6833/1746950966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6832","nonce":1746950966},{"type":"v1","addr":"172.21.15.162:6833","nonce":1746950966}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24422,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6833/3394739181","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6832","nonce":3394739181},{"type":"v1","addr":"172.21.15.142:6833","nonce":3394739181}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[]} 2024-03-31T23:08:46.343 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 5 2024-03-31T23:08:46.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:46.345+0000 7f5a178c9640 1 -- 172.21.15.46:0/3917176136 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f59fc0a24a0 msgr2=0x7f59fc0a4960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:08:46.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:46.345+0000 7f5a178c9640 1 --2- 172.21.15.46:0/3917176136 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f59fc0a24a0 0x7f59fc0a4960 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f5a0c0058c0 tx=0x7f5a0c005850 comp rx=0 tx=0).stop 2024-03-31T23:08:46.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:46.345+0000 7f5a178c9640 1 -- 172.21.15.46:0/3917176136 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5a10102950 msgr2=0x7f5a1019e530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:08:46.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:46.345+0000 7f5a178c9640 1 --2- 172.21.15.46:0/3917176136 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5a10102950 0x7f5a1019e530 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f5a00006010 tx=0x7f5a000042f0 comp rx=0 tx=0).stop 2024-03-31T23:08:46.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:46.345+0000 7f5a178c9640 1 -- 172.21.15.46:0/3917176136 shutdown_connections 2024-03-31T23:08:46.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:46.345+0000 7f5a178c9640 1 --2- 172.21.15.46:0/3917176136 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f59fc0a24a0 0x7f59fc0a4960 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:08:46.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:46.345+0000 7f5a178c9640 1 --2- 172.21.15.46:0/3917176136 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5a10105300 0x7f5a101a55a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:08:46.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:46.345+0000 7f5a178c9640 1 --2- 172.21.15.46:0/3917176136 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5a10102950 0x7f5a1019e530 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:08:46.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:46.345+0000 7f5a178c9640 1 --2- 172.21.15.46:0/3917176136 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5a100fffe0 0x7f5a1019dff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:08:46.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:46.345+0000 7f5a178c9640 1 -- 172.21.15.46:0/3917176136 >> 172.21.15.46:0/3917176136 conn(0x7f5a100fbbe0 msgr2=0x7f5a10100a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:08:46.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:46.345+0000 7f5a178c9640 1 -- 172.21.15.46:0/3917176136 shutdown_connections 2024-03-31T23:08:46.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:46.345+0000 7f5a178c9640 1 -- 172.21.15.46:0/3917176136 wait complete. 2024-03-31T23:08:46.880 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:08:46 smithi142 ceph-mon[35271]: from='client.? 172.21.15.46:0/3917176136' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-31T23:08:47.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:08:46 smithi162 ceph-mon[36237]: from='client.? 172.21.15.46:0/3917176136' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-31T23:08:47.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:08:46 smithi046 ceph-mon[32498]: from='client.? 172.21.15.46:0/3917176136' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-31T23:08:47.443 INFO:journalctl@ceph.mds.e.smithi142.stdout:Mar 31 23:08:47 smithi142 systemd[1]: Started Ceph mds.e for 6a9d8f08-efb2-11ee-b647-cb9ed24678a4. 2024-03-31T23:08:48.357 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-31T23:08:48.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:48.471+0000 7f5c531e5640 1 -- 172.21.15.46:0/60645472 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5c4c102220 msgr2=0x7f5c4c108ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:08:48.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:48.471+0000 7f5c531e5640 1 --2- 172.21.15.46:0/60645472 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5c4c102220 0x7f5c4c108ab0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f5c44009ca0 tx=0x7f5c4402c0f0 comp rx=0 tx=0).stop 2024-03-31T23:08:48.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:48.471+0000 7f5c531e5640 1 -- 172.21.15.46:0/60645472 shutdown_connections 2024-03-31T23:08:48.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:48.471+0000 7f5c531e5640 1 --2- 172.21.15.46:0/60645472 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5c4c102220 0x7f5c4c108ab0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:08:48.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:48.471+0000 7f5c531e5640 1 --2- 172.21.15.46:0/60645472 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5c4c101860 0x7f5c4c101ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:08:48.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:48.471+0000 7f5c531e5640 1 --2- 172.21.15.46:0/60645472 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5c4c100670 0x7f5c4c100a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:08:48.475 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:48.471+0000 7f5c531e5640 1 -- 172.21.15.46:0/60645472 >> 172.21.15.46:0/60645472 conn(0x7f5c4c0fbbe0 msgr2=0x7f5c4c0fe020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:08:48.475 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:48.471+0000 7f5c531e5640 1 -- 172.21.15.46:0/60645472 shutdown_connections 2024-03-31T23:08:48.475 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:48.471+0000 7f5c531e5640 1 -- 172.21.15.46:0/60645472 wait complete. 2024-03-31T23:08:48.475 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:48.471+0000 7f5c531e5640 1 Processor -- start 2024-03-31T23:08:48.475 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:48.471+0000 7f5c531e5640 1 -- start start 2024-03-31T23:08:48.475 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:48.472+0000 7f5c531e5640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5c4c100670 0x7f5c4c196870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:08:48.475 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:48.472+0000 7f5c531e5640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5c4c101860 0x7f5c4c196db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:08:48.475 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:48.472+0000 7f5c531e5640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5c4c102220 0x7f5c4c19de20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:08:48.475 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:48.472+0000 7f5c531e5640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f5c4c10c1a0 con 0x7f5c4c102220 2024-03-31T23:08:48.475 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:48.472+0000 7f5c531e5640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f5c4c10c020 con 0x7f5c4c100670 2024-03-31T23:08:48.475 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:48.472+0000 7f5c531e5640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f5c4c10c320 con 0x7f5c4c101860 2024-03-31T23:08:48.475 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:48.472+0000 7f5c51782640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5c4c102220 0x7f5c4c19de20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:08:48.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:48.472+0000 7f5c51782640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5c4c102220 0x7f5c4c19de20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:33582/0 (socket says 172.21.15.46:33582) 2024-03-31T23:08:48.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:48.472+0000 7f5c51782640 1 -- 172.21.15.46:0/1309496812 learned_addr learned my addr 172.21.15.46:0/1309496812 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:08:48.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:48.472+0000 7f5c4bfff640 1 --2- 172.21.15.46:0/1309496812 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5c4c101860 0x7f5c4c196db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:08:48.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:48.472+0000 7f5c50f81640 1 --2- 172.21.15.46:0/1309496812 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5c4c100670 0x7f5c4c196870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:08:48.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:48.472+0000 7f5c51782640 1 -- 172.21.15.46:0/1309496812 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5c4c101860 msgr2=0x7f5c4c196db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:08:48.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:48.472+0000 7f5c51782640 1 --2- 172.21.15.46:0/1309496812 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5c4c101860 0x7f5c4c196db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:08:48.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:48.472+0000 7f5c51782640 1 -- 172.21.15.46:0/1309496812 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5c4c100670 msgr2=0x7f5c4c196870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:08:48.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:48.472+0000 7f5c51782640 1 --2- 172.21.15.46:0/1309496812 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5c4c100670 0x7f5c4c196870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:08:48.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:48.472+0000 7f5c51782640 1 -- 172.21.15.46:0/1309496812 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5c4c19e490 con 0x7f5c4c102220 2024-03-31T23:08:48.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:48.472+0000 7f5c4bfff640 1 --2- 172.21.15.46:0/1309496812 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5c4c101860 0x7f5c4c196db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T23:08:48.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:48.472+0000 7f5c51782640 1 --2- 172.21.15.46:0/1309496812 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5c4c102220 0x7f5c4c19de20 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f5c4402c700 tx=0x7f5c440328a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:08:48.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:48.472+0000 7f5c49ffb640 1 -- 172.21.15.46:0/1309496812 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c44041020 con 0x7f5c4c102220 2024-03-31T23:08:48.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:48.472+0000 7f5c49ffb640 1 -- 172.21.15.46:0/1309496812 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5c44003e20 con 0x7f5c4c102220 2024-03-31T23:08:48.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:48.472+0000 7f5c531e5640 1 -- 172.21.15.46:0/1309496812 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5c4c19e720 con 0x7f5c4c102220 2024-03-31T23:08:48.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:48.472+0000 7f5c49ffb640 1 -- 172.21.15.46:0/1309496812 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c44039040 con 0x7f5c4c102220 2024-03-31T23:08:48.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:48.473+0000 7f5c531e5640 1 -- 172.21.15.46:0/1309496812 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5c4c19ec10 con 0x7f5c4c102220 2024-03-31T23:08:48.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:48.474+0000 7f5c531e5640 1 -- 172.21.15.46:0/1309496812 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5c4c1079e0 con 0x7f5c4c102220 2024-03-31T23:08:48.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:48.475+0000 7f5c49ffb640 1 -- 172.21.15.46:0/1309496812 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 16) ==== 137320+0+0 (secure 0 0 0) 0x7f5c44044020 con 0x7f5c4c102220 2024-03-31T23:08:48.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:48.475+0000 7f5c49ffb640 1 --2- 172.21.15.46:0/1309496812 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f5c3c0a24a0 0x7f5c3c0a4960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:08:48.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:48.475+0000 7f5c50f81640 1 --2- 172.21.15.46:0/1309496812 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f5c3c0a24a0 0x7f5c3c0a4960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:08:48.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:48.475+0000 7f5c49ffb640 1 -- 172.21.15.46:0/1309496812 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f5c440f0770 con 0x7f5c4c102220 2024-03-31T23:08:48.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:48.476+0000 7f5c50f81640 1 --2- 172.21.15.46:0/1309496812 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f5c3c0a24a0 0x7f5c3c0a4960 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f5c40005fd0 tx=0x7f5c40005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:08:48.481 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:48.480+0000 7f5c49ffb640 1 -- 172.21.15.46:0/1309496812 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f5c4404b020 con 0x7f5c4c102220 2024-03-31T23:08:48.546 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:08:48 smithi162 ceph-mon[36237]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-31T23:08:48.546 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:08:48 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:08:48.546 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:08:48 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:08:48.546 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:08:48 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:08:48.546 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:08:48 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-31T23:08:48.546 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:08:48 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-31T23:08:48.547 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:08:48 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:08:48.547 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:08:48 smithi162 ceph-mon[36237]: Deploying daemon mds.f on smithi162 2024-03-31T23:08:48.547 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:08:48 smithi162 ceph-mon[36237]: mds.? [v2:172.21.15.142:6834/212637431,v1:172.21.15.142:6835/212637431] up:boot 2024-03-31T23:08:48.547 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:08:48 smithi162 ceph-mon[36237]: fsmap 5 up:standby 2024-03-31T23:08:48.547 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:08:48 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-31T23:08:48.611 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:08:48 smithi046 ceph-mon[32498]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-31T23:08:48.611 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:08:48 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:08:48.612 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:08:48 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:08:48.612 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:08:48 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:08:48.612 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:08:48 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-31T23:08:48.612 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:08:48 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-31T23:08:48.612 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:08:48 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:08:48.612 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:08:48 smithi046 ceph-mon[32498]: Deploying daemon mds.f on smithi162 2024-03-31T23:08:48.612 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:08:48 smithi046 ceph-mon[32498]: mds.? [v2:172.21.15.142:6834/212637431,v1:172.21.15.142:6835/212637431] up:boot 2024-03-31T23:08:48.612 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:08:48 smithi046 ceph-mon[32498]: fsmap 5 up:standby 2024-03-31T23:08:48.612 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:08:48 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-31T23:08:48.693 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:08:48 smithi142 ceph-mon[35271]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-31T23:08:48.694 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:08:48 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:08:48.694 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:08:48 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:08:48.694 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:08:48 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:08:48.694 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:08:48 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-31T23:08:48.694 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:08:48 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-31T23:08:48.694 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:08:48 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:08:48.694 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:08:48 smithi142 ceph-mon[35271]: Deploying daemon mds.f on smithi162 2024-03-31T23:08:48.694 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:08:48 smithi142 ceph-mon[35271]: mds.? [v2:172.21.15.142:6834/212637431,v1:172.21.15.142:6835/212637431] up:boot 2024-03-31T23:08:48.694 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:08:48 smithi142 ceph-mon[35271]: fsmap 5 up:standby 2024-03-31T23:08:48.694 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:08:48 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-31T23:08:48.705 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:48.704+0000 7f5c531e5640 1 -- 172.21.15.46:0/1309496812 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5c4c062350 con 0x7f5c4c102220 2024-03-31T23:08:48.705 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:48.704+0000 7f5c49ffb640 1 -- 172.21.15.46:0/1309496812 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) ==== 93+0+4699 (secure 0 0 0) 0x7f5c4403b2f0 con 0x7f5c4c102220 2024-03-31T23:08:48.706 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-31T23:08:48.706 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":6,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/2169018547","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":2169018547},{"type":"v1","addr":"172.21.15.46:6835","nonce":2169018547}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6837/1069387756","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6836","nonce":1069387756},{"type":"v1","addr":"172.21.15.46:6837","nonce":1069387756}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24415,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6833/1746950966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6832","nonce":1746950966},{"type":"v1","addr":"172.21.15.162:6833","nonce":1746950966}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24422,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6833/3394739181","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6832","nonce":3394739181},{"type":"v1","addr":"172.21.15.142:6833","nonce":3394739181}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24455,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6835/212637431","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6834","nonce":212637431},{"type":"v1","addr":"172.21.15.142:6835","nonce":212637431}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[]} 2024-03-31T23:08:48.706 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 6 2024-03-31T23:08:48.708 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:48.706+0000 7f5c531e5640 1 -- 172.21.15.46:0/1309496812 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f5c3c0a24a0 msgr2=0x7f5c3c0a4960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:08:48.708 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:48.706+0000 7f5c531e5640 1 --2- 172.21.15.46:0/1309496812 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f5c3c0a24a0 0x7f5c3c0a4960 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f5c40005fd0 tx=0x7f5c40005850 comp rx=0 tx=0).stop 2024-03-31T23:08:48.708 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:48.706+0000 7f5c531e5640 1 -- 172.21.15.46:0/1309496812 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5c4c102220 msgr2=0x7f5c4c19de20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:08:48.708 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:48.706+0000 7f5c531e5640 1 --2- 172.21.15.46:0/1309496812 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5c4c102220 0x7f5c4c19de20 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f5c4402c700 tx=0x7f5c440328a0 comp rx=0 tx=0).stop 2024-03-31T23:08:48.708 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:48.706+0000 7f5c531e5640 1 -- 172.21.15.46:0/1309496812 shutdown_connections 2024-03-31T23:08:48.708 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:48.706+0000 7f5c531e5640 1 --2- 172.21.15.46:0/1309496812 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f5c3c0a24a0 0x7f5c3c0a4960 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:08:48.708 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:48.707+0000 7f5c531e5640 1 --2- 172.21.15.46:0/1309496812 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5c4c102220 0x7f5c4c19de20 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:08:48.708 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:48.707+0000 7f5c531e5640 1 --2- 172.21.15.46:0/1309496812 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5c4c101860 0x7f5c4c196db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:08:48.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:48.707+0000 7f5c531e5640 1 --2- 172.21.15.46:0/1309496812 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5c4c100670 0x7f5c4c196870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:08:48.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:48.707+0000 7f5c531e5640 1 -- 172.21.15.46:0/1309496812 >> 172.21.15.46:0/1309496812 conn(0x7f5c4c0fbbe0 msgr2=0x7f5c4c104bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:08:48.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:48.707+0000 7f5c531e5640 1 -- 172.21.15.46:0/1309496812 shutdown_connections 2024-03-31T23:08:48.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:48.707+0000 7f5c531e5640 1 -- 172.21.15.46:0/1309496812 wait complete. 2024-03-31T23:08:49.546 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:08:49 smithi162 ceph-mon[36237]: from='client.? 172.21.15.46:0/1309496812' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-31T23:08:49.546 INFO:journalctl@ceph.mds.f.smithi162.stdout:Mar 31 23:08:49 smithi162 systemd[1]: Starting Ceph mds.f for 6a9d8f08-efb2-11ee-b647-cb9ed24678a4... 2024-03-31T23:08:49.611 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:08:49 smithi046 ceph-mon[32498]: from='client.? 172.21.15.46:0/1309496812' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-31T23:08:49.693 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:08:49 smithi142 ceph-mon[35271]: from='client.? 172.21.15.46:0/1309496812' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-31T23:08:49.855 INFO:journalctl@ceph.mds.f.smithi162.stdout:Mar 31 23:08:49 smithi162 podman[66737]: 2024-03-31 23:08:49.476240375 +0000 UTC m=+0.018098522 image pull ec18d7baa60e8b1a0b126fbe5306acce11b1ed0ec5e3bcdddbf39b0a3c591791 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175 2024-03-31T23:08:49.855 INFO:journalctl@ceph.mds.f.smithi162.stdout:Mar 31 23:08:49 smithi162 podman[66737]: 2024-03-31 23:08:49.594714751 +0000 UTC m=+0.136572923 container create b67d463eb8fee0a0edec3829c2ad68283d3970498bb5c32956d5eece780a6166 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mds-f, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.build-date=20240326, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=squid-384f31f, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-31T23:08:49.856 INFO:journalctl@ceph.mds.f.smithi162.stdout:Mar 31 23:08:49 smithi162 podman[66737]: 2024-03-31 23:08:49.855667983 +0000 UTC m=+0.397526115 container init b67d463eb8fee0a0edec3829c2ad68283d3970498bb5c32956d5eece780a6166 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mds-f, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-384f31f, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240326, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5) 2024-03-31T23:08:50.295 INFO:journalctl@ceph.mds.f.smithi162.stdout:Mar 31 23:08:49 smithi162 podman[66737]: 2024-03-31 23:08:49.858459437 +0000 UTC m=+0.400317576 container start b67d463eb8fee0a0edec3829c2ad68283d3970498bb5c32956d5eece780a6166 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mds-f, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, maintainer=Guillaume Abrioux , RELEASE=squid-384f31f, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240326, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-31T23:08:50.296 INFO:journalctl@ceph.mds.f.smithi162.stdout:Mar 31 23:08:49 smithi162 ceph-mds[66756]: -- 172.21.15.162:0/2951844894 <== mon.1 v2:172.21.15.162:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55aaa78eda40 con 0x55aaa78c0c00 2024-03-31T23:08:50.296 INFO:journalctl@ceph.mds.f.smithi162.stdout:Mar 31 23:08:49 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mds-f[66752]: starting mds.f at 2024-03-31T23:08:50.296 INFO:journalctl@ceph.mds.f.smithi162.stdout:Mar 31 23:08:49 smithi162 bash[66737]: b67d463eb8fee0a0edec3829c2ad68283d3970498bb5c32956d5eece780a6166 2024-03-31T23:08:50.296 INFO:journalctl@ceph.mds.f.smithi162.stdout:Mar 31 23:08:49 smithi162 systemd[1]: Started Ceph mds.f for 6a9d8f08-efb2-11ee-b647-cb9ed24678a4. 2024-03-31T23:08:50.545 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:08:50 smithi162 ceph-mon[36237]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-31T23:08:50.546 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:08:50 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:08:50.546 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:08:50 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:08:50.546 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:08:50 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:08:50.546 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:08:50 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-31T23:08:50.546 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:08:50 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-31T23:08:50.546 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:08:50 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:08:50.611 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:08:50 smithi046 ceph-mon[32498]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-31T23:08:50.611 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:08:50 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:08:50.612 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:08:50 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:08:50.612 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:08:50 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:08:50.612 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:08:50 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-31T23:08:50.612 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:08:50 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-31T23:08:50.612 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:08:50 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:08:50.693 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:08:50 smithi142 ceph-mon[35271]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-31T23:08:50.693 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:08:50 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:08:50.694 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:08:50 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:08:50.694 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:08:50 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:08:50.694 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:08:50 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-31T23:08:50.694 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:08:50 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-31T23:08:50.694 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:08:50 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:08:50.720 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-31T23:08:50.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:50.834+0000 7fba0cb08640 1 -- 172.21.15.46:0/4254868766 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fba08101c90 msgr2=0x7fba081020b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:08:50.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:50.834+0000 7fba0cb08640 1 --2- 172.21.15.46:0/4254868766 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fba08101c90 0x7fba081020b0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fb9f8009900 tx=0x7fb9f802bc30 comp rx=0 tx=0).stop 2024-03-31T23:08:50.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:50.834+0000 7fba0cb08640 1 -- 172.21.15.46:0/4254868766 shutdown_connections 2024-03-31T23:08:50.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:50.834+0000 7fba0cb08640 1 --2- 172.21.15.46:0/4254868766 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fba081012a0 0x7fba08108b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:08:50.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:50.834+0000 7fba0cb08640 1 --2- 172.21.15.46:0/4254868766 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fba081008e0 0x7fba08100d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:08:50.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:50.834+0000 7fba0cb08640 1 --2- 172.21.15.46:0/4254868766 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fba08101c90 0x7fba081020b0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:08:50.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:50.834+0000 7fba0cb08640 1 -- 172.21.15.46:0/4254868766 >> 172.21.15.46:0/4254868766 conn(0x7fba080fb370 msgr2=0x7fba080fd7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:08:50.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:50.834+0000 7fba0cb08640 1 -- 172.21.15.46:0/4254868766 shutdown_connections 2024-03-31T23:08:50.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:50.834+0000 7fba0cb08640 1 -- 172.21.15.46:0/4254868766 wait complete. 2024-03-31T23:08:50.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:50.835+0000 7fba0cb08640 1 Processor -- start 2024-03-31T23:08:50.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:50.835+0000 7fba0cb08640 1 -- start start 2024-03-31T23:08:50.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:50.835+0000 7fba0cb08640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fba081008e0 0x7fba08197990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:08:50.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:50.835+0000 7fba0cb08640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fba081012a0 0x7fba08197ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:08:50.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:50.835+0000 7fba0cb08640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fba08101c90 0x7fba0819ef40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:08:50.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:50.835+0000 7fba0cb08640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fba0810c1a0 con 0x7fba08101c90 2024-03-31T23:08:50.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:50.835+0000 7fba0cb08640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fba0810c020 con 0x7fba081008e0 2024-03-31T23:08:50.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:50.835+0000 7fba0cb08640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fba0810c320 con 0x7fba081012a0 2024-03-31T23:08:50.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:50.835+0000 7fba06d9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fba08101c90 0x7fba0819ef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:08:50.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:50.835+0000 7fba06d9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fba08101c90 0x7fba0819ef40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:33592/0 (socket says 172.21.15.46:33592) 2024-03-31T23:08:50.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:50.835+0000 7fba06d9d640 1 -- 172.21.15.46:0/1640139731 learned_addr learned my addr 172.21.15.46:0/1640139731 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:08:50.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:50.835+0000 7fba05d9b640 1 --2- 172.21.15.46:0/1640139731 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fba081012a0 0x7fba08197ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:08:50.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:50.835+0000 7fba0659c640 1 --2- 172.21.15.46:0/1640139731 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fba081008e0 0x7fba08197990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:08:50.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:50.835+0000 7fba06d9d640 1 -- 172.21.15.46:0/1640139731 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fba081012a0 msgr2=0x7fba08197ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:08:50.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:50.835+0000 7fba06d9d640 1 --2- 172.21.15.46:0/1640139731 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fba081012a0 0x7fba08197ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:08:50.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:50.835+0000 7fba06d9d640 1 -- 172.21.15.46:0/1640139731 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fba081008e0 msgr2=0x7fba08197990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:08:50.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:50.835+0000 7fba06d9d640 1 --2- 172.21.15.46:0/1640139731 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fba081008e0 0x7fba08197990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:08:50.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:50.835+0000 7fba06d9d640 1 -- 172.21.15.46:0/1640139731 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fba0819f5b0 con 0x7fba08101c90 2024-03-31T23:08:50.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:50.836+0000 7fba06d9d640 1 --2- 172.21.15.46:0/1640139731 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fba08101c90 0x7fba0819ef40 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fba0000cce0 tx=0x7fba0000a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:08:50.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:50.836+0000 7fb9f77fe640 1 -- 172.21.15.46:0/1640139731 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba0000f450 con 0x7fba08101c90 2024-03-31T23:08:50.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:50.836+0000 7fb9f77fe640 1 -- 172.21.15.46:0/1640139731 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fba0000faa0 con 0x7fba08101c90 2024-03-31T23:08:50.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:50.836+0000 7fba0cb08640 1 -- 172.21.15.46:0/1640139731 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fba0819f8a0 con 0x7fba08101c90 2024-03-31T23:08:50.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:50.836+0000 7fb9f77fe640 1 -- 172.21.15.46:0/1640139731 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba00018b70 con 0x7fba08101c90 2024-03-31T23:08:50.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:50.836+0000 7fba0cb08640 1 -- 172.21.15.46:0/1640139731 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fba0819fdf0 con 0x7fba08101c90 2024-03-31T23:08:50.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:50.837+0000 7fba0cb08640 1 -- 172.21.15.46:0/1640139731 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fba081a0050 con 0x7fba08101c90 2024-03-31T23:08:50.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:50.838+0000 7fb9f77fe640 1 -- 172.21.15.46:0/1640139731 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 16) ==== 137320+0+0 (secure 0 0 0) 0x7fba00018d10 con 0x7fba08101c90 2024-03-31T23:08:50.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:50.838+0000 7fb9f77fe640 1 --2- 172.21.15.46:0/1640139731 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fb9f00a2520 0x7fb9f00a49e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:08:50.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:50.839+0000 7fba0659c640 1 --2- 172.21.15.46:0/1640139731 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fb9f00a2520 0x7fb9f00a49e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:08:50.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:50.839+0000 7fb9f77fe640 1 -- 172.21.15.46:0/1640139731 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7fba000ce750 con 0x7fba08101c90 2024-03-31T23:08:50.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:50.839+0000 7fba0659c640 1 --2- 172.21.15.46:0/1640139731 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fb9f00a2520 0x7fb9f00a49e0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fb9f8006010 tx=0x7fb9f802c800 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:08:50.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:50.842+0000 7fb9f77fe640 1 -- 172.21.15.46:0/1640139731 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fba00014030 con 0x7fba08101c90 2024-03-31T23:08:51.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:51.066+0000 7fba0cb08640 1 -- 172.21.15.46:0/1640139731 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fba081a0220 con 0x7fba08101c90 2024-03-31T23:08:51.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:51.066+0000 7fb9f77fe640 1 -- 172.21.15.46:0/1640139731 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) ==== 93+0+5527 (secure 0 0 0) 0x7fba0009bec0 con 0x7fba08101c90 2024-03-31T23:08:51.067 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-31T23:08:51.067 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":7,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/2169018547","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":2169018547},{"type":"v1","addr":"172.21.15.46:6835","nonce":2169018547}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6837/1069387756","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6836","nonce":1069387756},{"type":"v1","addr":"172.21.15.46:6837","nonce":1069387756}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24415,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6833/1746950966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6832","nonce":1746950966},{"type":"v1","addr":"172.21.15.162:6833","nonce":1746950966}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24422,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6833/3394739181","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6832","nonce":3394739181},{"type":"v1","addr":"172.21.15.142:6833","nonce":3394739181}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24442,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6835/1957208984","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6834","nonce":1957208984},{"type":"v1","addr":"172.21.15.162:6835","nonce":1957208984}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24455,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6835/212637431","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6834","nonce":212637431},{"type":"v1","addr":"172.21.15.142:6835","nonce":212637431}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[]} 2024-03-31T23:08:51.068 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 7 2024-03-31T23:08:51.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:51.068+0000 7fba0cb08640 1 -- 172.21.15.46:0/1640139731 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fb9f00a2520 msgr2=0x7fb9f00a49e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:08:51.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:51.068+0000 7fba0cb08640 1 --2- 172.21.15.46:0/1640139731 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fb9f00a2520 0x7fb9f00a49e0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fb9f8006010 tx=0x7fb9f802c800 comp rx=0 tx=0).stop 2024-03-31T23:08:51.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:51.068+0000 7fba0cb08640 1 -- 172.21.15.46:0/1640139731 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fba08101c90 msgr2=0x7fba0819ef40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:08:51.070 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:51.068+0000 7fba0cb08640 1 --2- 172.21.15.46:0/1640139731 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fba08101c90 0x7fba0819ef40 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fba0000cce0 tx=0x7fba0000a620 comp rx=0 tx=0).stop 2024-03-31T23:08:51.070 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:51.068+0000 7fba0cb08640 1 -- 172.21.15.46:0/1640139731 shutdown_connections 2024-03-31T23:08:51.070 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:51.068+0000 7fba0cb08640 1 --2- 172.21.15.46:0/1640139731 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fb9f00a2520 0x7fb9f00a49e0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:08:51.070 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:51.068+0000 7fba0cb08640 1 --2- 172.21.15.46:0/1640139731 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fba08101c90 0x7fba0819ef40 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:08:51.070 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:51.068+0000 7fba0cb08640 1 --2- 172.21.15.46:0/1640139731 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fba081012a0 0x7fba08197ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:08:51.070 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:51.068+0000 7fba0cb08640 1 --2- 172.21.15.46:0/1640139731 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fba081008e0 0x7fba08197990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:08:51.070 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:51.068+0000 7fba0cb08640 1 -- 172.21.15.46:0/1640139731 >> 172.21.15.46:0/1640139731 conn(0x7fba080fb370 msgr2=0x7fba080fce90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:08:51.070 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:51.068+0000 7fba0cb08640 1 -- 172.21.15.46:0/1640139731 shutdown_connections 2024-03-31T23:08:51.070 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:51.068+0000 7fba0cb08640 1 -- 172.21.15.46:0/1640139731 wait complete. 2024-03-31T23:08:51.361 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:08:51 smithi046 ceph-mon[32498]: Deploying daemon mds.c on smithi162 2024-03-31T23:08:51.362 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:08:51 smithi046 ceph-mon[32498]: mds.? [v2:172.21.15.162:6834/1957208984,v1:172.21.15.162:6835/1957208984] up:boot 2024-03-31T23:08:51.362 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:08:51 smithi046 ceph-mon[32498]: fsmap 6 up:standby 2024-03-31T23:08:51.362 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:08:51 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-31T23:08:51.362 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:08:51 smithi046 ceph-mon[32498]: from='client.? 172.21.15.46:0/1640139731' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-31T23:08:51.693 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:08:51 smithi142 ceph-mon[35271]: Deploying daemon mds.c on smithi162 2024-03-31T23:08:51.693 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:08:51 smithi142 ceph-mon[35271]: mds.? [v2:172.21.15.162:6834/1957208984,v1:172.21.15.162:6835/1957208984] up:boot 2024-03-31T23:08:51.693 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:08:51 smithi142 ceph-mon[35271]: fsmap 6 up:standby 2024-03-31T23:08:51.693 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:08:51 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-31T23:08:51.694 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:08:51 smithi142 ceph-mon[35271]: from='client.? 172.21.15.46:0/1640139731' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-31T23:08:51.694 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:08:51 smithi162 ceph-mon[36237]: Deploying daemon mds.c on smithi162 2024-03-31T23:08:51.694 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:08:51 smithi162 ceph-mon[36237]: mds.? [v2:172.21.15.162:6834/1957208984,v1:172.21.15.162:6835/1957208984] up:boot 2024-03-31T23:08:51.694 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:08:51 smithi162 ceph-mon[36237]: fsmap 6 up:standby 2024-03-31T23:08:51.694 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:08:51 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-31T23:08:51.694 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:08:51 smithi162 ceph-mon[36237]: from='client.? 172.21.15.46:0/1640139731' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-31T23:08:52.255 INFO:journalctl@ceph.mds.c.smithi162.stdout:Mar 31 23:08:52 smithi162 systemd[1]: Starting Ceph mds.c for 6a9d8f08-efb2-11ee-b647-cb9ed24678a4... 2024-03-31T23:08:52.546 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:08:52 smithi162 ceph-mon[36237]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-31T23:08:52.546 INFO:journalctl@ceph.mds.c.smithi162.stdout:Mar 31 23:08:52 smithi162 podman[67044]: 2024-03-31 23:08:52.325531782 +0000 UTC m=+0.019947468 image pull ec18d7baa60e8b1a0b126fbe5306acce11b1ed0ec5e3bcdddbf39b0a3c591791 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175 2024-03-31T23:08:52.546 INFO:journalctl@ceph.mds.c.smithi162.stdout:Mar 31 23:08:52 smithi162 podman[67044]: 2024-03-31 23:08:52.449553731 +0000 UTC m=+0.143969399 container create 6c86ca0bcc43acac46a1a8136c7b9e481e6b83fddedbe03afe393f331be9d4ac (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mds-c, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.build-date=20240326, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, RELEASE=squid-384f31f, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-31T23:08:52.611 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:08:52 smithi046 ceph-mon[32498]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-31T23:08:52.693 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:08:52 smithi142 ceph-mon[35271]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-31T23:08:53.046 INFO:journalctl@ceph.mds.c.smithi162.stdout:Mar 31 23:08:52 smithi162 podman[67044]: 2024-03-31 23:08:52.685951269 +0000 UTC m=+0.380366948 container init 6c86ca0bcc43acac46a1a8136c7b9e481e6b83fddedbe03afe393f331be9d4ac (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mds-c, GIT_CLEAN=True, ceph=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.build-date=20240326, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-384f31f, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-03-31T23:08:53.046 INFO:journalctl@ceph.mds.c.smithi162.stdout:Mar 31 23:08:52 smithi162 podman[67044]: 2024-03-31 23:08:52.68901501 +0000 UTC m=+0.383430682 container start 6c86ca0bcc43acac46a1a8136c7b9e481e6b83fddedbe03afe393f331be9d4ac (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mds-c, ceph=True, org.label-schema.build-date=20240326, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, RELEASE=squid-384f31f, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5) 2024-03-31T23:08:53.046 INFO:journalctl@ceph.mds.c.smithi162.stdout:Mar 31 23:08:52 smithi162 ceph-mds[67073]: -- 172.21.15.162:0/3405498961 <== mon.1 v2:172.21.15.162:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55c95a7e3a40 con 0x55c95a7b6800 2024-03-31T23:08:53.046 INFO:journalctl@ceph.mds.c.smithi162.stdout:Mar 31 23:08:52 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mds-c[67067]: starting mds.c at 2024-03-31T23:08:53.046 INFO:journalctl@ceph.mds.c.smithi162.stdout:Mar 31 23:08:52 smithi162 bash[67044]: 6c86ca0bcc43acac46a1a8136c7b9e481e6b83fddedbe03afe393f331be9d4ac 2024-03-31T23:08:53.046 INFO:journalctl@ceph.mds.c.smithi162.stdout:Mar 31 23:08:52 smithi162 systemd[1]: Started Ceph mds.c for 6a9d8f08-efb2-11ee-b647-cb9ed24678a4. 2024-03-31T23:08:53.081 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-31T23:08:53.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:53.195+0000 7fdcc5b48640 1 -- 172.21.15.46:0/1152049953 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdcc0104b20 msgr2=0x7fdcc0106f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:08:53.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:53.195+0000 7fdcc5b48640 1 --2- 172.21.15.46:0/1152049953 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdcc0104b20 0x7fdcc0106f10 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fdcb4009900 tx=0x7fdcb402bc30 comp rx=0 tx=0).stop 2024-03-31T23:08:53.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:53.195+0000 7fdcc5b48640 1 -- 172.21.15.46:0/1152049953 shutdown_connections 2024-03-31T23:08:53.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:53.195+0000 7fdcc5b48640 1 --2- 172.21.15.46:0/1152049953 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdcc0107450 0x7fdcc01098a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:08:53.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:53.195+0000 7fdcc5b48640 1 --2- 172.21.15.46:0/1152049953 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdcc0104b20 0x7fdcc0106f10 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:08:53.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:53.195+0000 7fdcc5b48640 1 --2- 172.21.15.46:0/1152049953 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdcc01021f0 0x7fdcc01045e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:08:53.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:53.195+0000 7fdcc5b48640 1 -- 172.21.15.46:0/1152049953 >> 172.21.15.46:0/1152049953 conn(0x7fdcc00fbbe0 msgr2=0x7fdcc00fe020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:08:53.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:53.195+0000 7fdcc5b48640 1 -- 172.21.15.46:0/1152049953 shutdown_connections 2024-03-31T23:08:53.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:53.195+0000 7fdcc5b48640 1 -- 172.21.15.46:0/1152049953 wait complete. 2024-03-31T23:08:53.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:53.195+0000 7fdcc5b48640 1 Processor -- start 2024-03-31T23:08:53.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:53.196+0000 7fdcc5b48640 1 -- start start 2024-03-31T23:08:53.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:53.196+0000 7fdcc5b48640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdcc01021f0 0x7fdcc0197a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:08:53.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:53.196+0000 7fdcc5b48640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdcc0104b20 0x7fdcc0197f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:08:53.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:53.196+0000 7fdcc5b48640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdcc0107450 0x7fdcc019efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:08:53.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:53.196+0000 7fdcc5b48640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fdcc010d300 con 0x7fdcc0104b20 2024-03-31T23:08:53.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:53.196+0000 7fdcc5b48640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fdcc010d180 con 0x7fdcc01021f0 2024-03-31T23:08:53.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:53.196+0000 7fdcc5b48640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fdcc010d480 con 0x7fdcc0107450 2024-03-31T23:08:53.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:53.196+0000 7fdcbeffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdcc0104b20 0x7fdcc0197f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:08:53.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:53.196+0000 7fdcbeffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdcc0104b20 0x7fdcc0197f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:43718/0 (socket says 172.21.15.46:43718) 2024-03-31T23:08:53.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:53.196+0000 7fdcbeffd640 1 -- 172.21.15.46:0/211459919 learned_addr learned my addr 172.21.15.46:0/211459919 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:08:53.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:53.196+0000 7fdcbf7fe640 1 --2- 172.21.15.46:0/211459919 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdcc01021f0 0x7fdcc0197a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:08:53.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:53.196+0000 7fdcbffff640 1 --2- 172.21.15.46:0/211459919 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdcc0107450 0x7fdcc019efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:08:53.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:53.196+0000 7fdcbeffd640 1 -- 172.21.15.46:0/211459919 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdcc0107450 msgr2=0x7fdcc019efb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:08:53.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:53.196+0000 7fdcbeffd640 1 --2- 172.21.15.46:0/211459919 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdcc0107450 0x7fdcc019efb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:08:53.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:53.196+0000 7fdcbeffd640 1 -- 172.21.15.46:0/211459919 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdcc01021f0 msgr2=0x7fdcc0197a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:08:53.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:53.196+0000 7fdcbeffd640 1 --2- 172.21.15.46:0/211459919 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdcc01021f0 0x7fdcc0197a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:08:53.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:53.196+0000 7fdcbeffd640 1 -- 172.21.15.46:0/211459919 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdcc019f620 con 0x7fdcc0104b20 2024-03-31T23:08:53.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:53.196+0000 7fdcbeffd640 1 --2- 172.21.15.46:0/211459919 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdcc0104b20 0x7fdcc0197f40 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fdcb4005ab0 tx=0x7fdcb40042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:08:53.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:53.196+0000 7fdcbcff9640 1 -- 172.21.15.46:0/211459919 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdcb4043020 con 0x7fdcc0104b20 2024-03-31T23:08:53.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:53.196+0000 7fdcbcff9640 1 -- 172.21.15.46:0/211459919 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fdcb40054c0 con 0x7fdcc0104b20 2024-03-31T23:08:53.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:53.196+0000 7fdcc5b48640 1 -- 172.21.15.46:0/211459919 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdcc019f8b0 con 0x7fdcc0104b20 2024-03-31T23:08:53.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:53.196+0000 7fdcbcff9640 1 -- 172.21.15.46:0/211459919 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdcb4003d10 con 0x7fdcc0104b20 2024-03-31T23:08:53.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:53.197+0000 7fdcc5b48640 1 -- 172.21.15.46:0/211459919 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdcc019fe00 con 0x7fdcc0104b20 2024-03-31T23:08:53.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:53.197+0000 7fdcc5b48640 1 -- 172.21.15.46:0/211459919 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdcc00fd6d0 con 0x7fdcc0104b20 2024-03-31T23:08:53.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:53.199+0000 7fdcbcff9640 1 -- 172.21.15.46:0/211459919 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 16) ==== 137320+0+0 (secure 0 0 0) 0x7fdcb4046020 con 0x7fdcc0104b20 2024-03-31T23:08:53.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:53.199+0000 7fdcbcff9640 1 --2- 172.21.15.46:0/211459919 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fdca80a24a0 0x7fdca80a4960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:08:53.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:53.199+0000 7fdcbf7fe640 1 --2- 172.21.15.46:0/211459919 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fdca80a24a0 0x7fdca80a4960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:08:53.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:53.199+0000 7fdcbcff9640 1 -- 172.21.15.46:0/211459919 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7fdcb40f0500 con 0x7fdcc0104b20 2024-03-31T23:08:53.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:53.200+0000 7fdcbf7fe640 1 --2- 172.21.15.46:0/211459919 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fdca80a24a0 0x7fdca80a4960 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fdcb0005fd0 tx=0x7fdcb000b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:08:53.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:53.204+0000 7fdcbcff9640 1 -- 172.21.15.46:0/211459919 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fdcb404d020 con 0x7fdcc0104b20 2024-03-31T23:08:53.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:53.426+0000 7fdcc5b48640 1 -- 172.21.15.46:0/211459919 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fdcc002d3c0 con 0x7fdcc0104b20 2024-03-31T23:08:53.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:53.426+0000 7fdcbcff9640 1 -- 172.21.15.46:0/211459919 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+6355 (secure 0 0 0) 0x7fdcb40bdb60 con 0x7fdcc0104b20 2024-03-31T23:08:53.428 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-31T23:08:53.428 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":8,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/2169018547","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":2169018547},{"type":"v1","addr":"172.21.15.46:6835","nonce":2169018547}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6837/1069387756","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6836","nonce":1069387756},{"type":"v1","addr":"172.21.15.46:6837","nonce":1069387756}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24415,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6833/1746950966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6832","nonce":1746950966},{"type":"v1","addr":"172.21.15.162:6833","nonce":1746950966}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24422,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6833/3394739181","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6832","nonce":3394739181},{"type":"v1","addr":"172.21.15.142:6833","nonce":3394739181}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24442,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6835/1957208984","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6834","nonce":1957208984},{"type":"v1","addr":"172.21.15.162:6835","nonce":1957208984}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24451,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6837/1371886876","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6836","nonce":1371886876},{"type":"v1","addr":"172.21.15.162:6837","nonce":1371886876}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24455,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6835/212637431","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6834","nonce":212637431},{"type":"v1","addr":"172.21.15.142:6835","nonce":212637431}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[]} 2024-03-31T23:08:53.428 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 8 2024-03-31T23:08:53.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:53.429+0000 7fdcc5b48640 1 -- 172.21.15.46:0/211459919 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fdca80a24a0 msgr2=0x7fdca80a4960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:08:53.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:53.429+0000 7fdcc5b48640 1 --2- 172.21.15.46:0/211459919 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fdca80a24a0 0x7fdca80a4960 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fdcb0005fd0 tx=0x7fdcb000b390 comp rx=0 tx=0).stop 2024-03-31T23:08:53.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:53.429+0000 7fdcc5b48640 1 -- 172.21.15.46:0/211459919 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdcc0104b20 msgr2=0x7fdcc0197f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:08:53.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:53.429+0000 7fdcc5b48640 1 --2- 172.21.15.46:0/211459919 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdcc0104b20 0x7fdcc0197f40 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fdcb4005ab0 tx=0x7fdcb40042f0 comp rx=0 tx=0).stop 2024-03-31T23:08:53.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:53.429+0000 7fdcc5b48640 1 -- 172.21.15.46:0/211459919 shutdown_connections 2024-03-31T23:08:53.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:53.429+0000 7fdcc5b48640 1 --2- 172.21.15.46:0/211459919 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fdca80a24a0 0x7fdca80a4960 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:08:53.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:53.429+0000 7fdcc5b48640 1 --2- 172.21.15.46:0/211459919 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdcc0107450 0x7fdcc019efb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:08:53.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:53.429+0000 7fdcc5b48640 1 --2- 172.21.15.46:0/211459919 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdcc0104b20 0x7fdcc0197f40 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:08:53.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:53.429+0000 7fdcc5b48640 1 --2- 172.21.15.46:0/211459919 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdcc01021f0 0x7fdcc0197a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:08:53.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:53.429+0000 7fdcc5b48640 1 -- 172.21.15.46:0/211459919 >> 172.21.15.46:0/211459919 conn(0x7fdcc00fbbe0 msgr2=0x7fdcc0102c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:08:53.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:53.429+0000 7fdcc5b48640 1 -- 172.21.15.46:0/211459919 shutdown_connections 2024-03-31T23:08:53.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:53.429+0000 7fdcc5b48640 1 -- 172.21.15.46:0/211459919 wait complete. 2024-03-31T23:08:54.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:08:53 smithi046 ceph-mon[32498]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-31T23:08:54.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:08:53 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:08:54.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:08:53 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:08:54.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:08:53 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:08:54.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:08:53 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-31T23:08:54.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:08:53 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-31T23:08:54.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:08:53 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:08:54.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:08:53 smithi046 ceph-mon[32498]: Deploying daemon mds.h on smithi142 2024-03-31T23:08:54.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:08:53 smithi046 ceph-mon[32498]: mds.? [v2:172.21.15.162:6836/1371886876,v1:172.21.15.162:6837/1371886876] up:boot 2024-03-31T23:08:54.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:08:53 smithi046 ceph-mon[32498]: fsmap 7 up:standby 2024-03-31T23:08:54.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:08:53 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-31T23:08:54.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:08:53 smithi046 ceph-mon[32498]: from='client.? 172.21.15.46:0/211459919' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-31T23:08:54.190 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:08:53 smithi142 ceph-mon[35271]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-31T23:08:54.190 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:08:53 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:08:54.191 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:08:53 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:08:54.191 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:08:53 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:08:54.191 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:08:53 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-31T23:08:54.191 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:08:53 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-31T23:08:54.191 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:08:53 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:08:54.191 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:08:53 smithi142 ceph-mon[35271]: Deploying daemon mds.h on smithi142 2024-03-31T23:08:54.191 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:08:53 smithi142 ceph-mon[35271]: mds.? [v2:172.21.15.162:6836/1371886876,v1:172.21.15.162:6837/1371886876] up:boot 2024-03-31T23:08:54.191 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:08:53 smithi142 ceph-mon[35271]: fsmap 7 up:standby 2024-03-31T23:08:54.191 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:08:53 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-31T23:08:54.191 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:08:53 smithi142 ceph-mon[35271]: from='client.? 172.21.15.46:0/211459919' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-31T23:08:54.296 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:08:53 smithi162 ceph-mon[36237]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-31T23:08:54.296 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:08:53 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:08:54.296 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:08:53 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:08:54.296 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:08:53 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:08:54.296 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:08:53 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-31T23:08:54.296 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:08:53 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-31T23:08:54.296 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:08:53 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:08:54.296 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:08:53 smithi162 ceph-mon[36237]: Deploying daemon mds.h on smithi142 2024-03-31T23:08:54.296 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:08:53 smithi162 ceph-mon[36237]: mds.? [v2:172.21.15.162:6836/1371886876,v1:172.21.15.162:6837/1371886876] up:boot 2024-03-31T23:08:54.296 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:08:53 smithi162 ceph-mon[36237]: fsmap 7 up:standby 2024-03-31T23:08:54.296 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:08:53 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-31T23:08:54.296 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:08:53 smithi162 ceph-mon[36237]: from='client.? 172.21.15.46:0/211459919' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-31T23:08:55.193 INFO:journalctl@ceph.mds.h.smithi142.stdout:Mar 31 23:08:55 smithi142 systemd[1]: Starting Ceph mds.h for 6a9d8f08-efb2-11ee-b647-cb9ed24678a4... 2024-03-31T23:08:55.442 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-31T23:08:55.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:55.557+0000 7f49e4a6d640 1 -- 172.21.15.46:0/2468749999 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f49e0102290 msgr2=0x7f49e0104680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:08:55.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:55.557+0000 7f49e4a6d640 1 --2- 172.21.15.46:0/2468749999 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f49e0102290 0x7f49e0104680 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f49d0009900 tx=0x7f49d002bc30 comp rx=0 tx=0).stop 2024-03-31T23:08:55.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:55.557+0000 7f49e4a6d640 1 -- 172.21.15.46:0/2468749999 shutdown_connections 2024-03-31T23:08:55.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:55.557+0000 7f49e4a6d640 1 --2- 172.21.15.46:0/2468749999 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f49e01074f0 0x7f49e01098e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:08:55.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:55.557+0000 7f49e4a6d640 1 --2- 172.21.15.46:0/2468749999 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f49e0104bc0 0x7f49e0106fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:08:55.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:55.557+0000 7f49e4a6d640 1 --2- 172.21.15.46:0/2468749999 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f49e0102290 0x7f49e0104680 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:08:55.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:55.557+0000 7f49e4a6d640 1 -- 172.21.15.46:0/2468749999 >> 172.21.15.46:0/2468749999 conn(0x7f49e00fbbe0 msgr2=0x7f49e00fe020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:08:55.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:55.557+0000 7f49e4a6d640 1 -- 172.21.15.46:0/2468749999 shutdown_connections 2024-03-31T23:08:55.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:55.557+0000 7f49e4a6d640 1 -- 172.21.15.46:0/2468749999 wait complete. 2024-03-31T23:08:55.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:55.558+0000 7f49e4a6d640 1 Processor -- start 2024-03-31T23:08:55.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:55.558+0000 7f49e4a6d640 1 -- start start 2024-03-31T23:08:55.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:55.558+0000 7f49e4a6d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f49e0102290 0x7f49e0101c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:08:55.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:55.558+0000 7f49e4a6d640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f49e0104bc0 0x7f49e0100240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:08:55.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:55.558+0000 7f49e4a6d640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f49e01074f0 0x7f49e0100780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:08:55.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:55.558+0000 7f49e4a6d640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f49e010d2b0 con 0x7f49e0102290 2024-03-31T23:08:55.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:55.558+0000 7f49e4a6d640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f49e010d130 con 0x7f49e0104bc0 2024-03-31T23:08:55.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:55.558+0000 7f49e4a6d640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f49e010d430 con 0x7f49e01074f0 2024-03-31T23:08:55.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:55.558+0000 7f49de59c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f49e0102290 0x7f49e0101c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:08:55.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:55.558+0000 7f49de59c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f49e0102290 0x7f49e0101c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:43746/0 (socket says 172.21.15.46:43746) 2024-03-31T23:08:55.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:55.558+0000 7f49de59c640 1 -- 172.21.15.46:0/4036734589 learned_addr learned my addr 172.21.15.46:0/4036734589 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:08:55.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:55.558+0000 7f49ded9d640 1 --2- 172.21.15.46:0/4036734589 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f49e01074f0 0x7f49e0100780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:08:55.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:55.558+0000 7f49ddd9b640 1 --2- 172.21.15.46:0/4036734589 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f49e0104bc0 0x7f49e0100240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:08:55.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:55.558+0000 7f49de59c640 1 -- 172.21.15.46:0/4036734589 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f49e01074f0 msgr2=0x7f49e0100780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:08:55.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:55.558+0000 7f49de59c640 1 --2- 172.21.15.46:0/4036734589 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f49e01074f0 0x7f49e0100780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:08:55.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:55.558+0000 7f49de59c640 1 -- 172.21.15.46:0/4036734589 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f49e0104bc0 msgr2=0x7f49e0100240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:08:55.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:55.558+0000 7f49de59c640 1 --2- 172.21.15.46:0/4036734589 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f49e0104bc0 0x7f49e0100240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:08:55.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:55.558+0000 7f49de59c640 1 -- 172.21.15.46:0/4036734589 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f49e0101010 con 0x7f49e0102290 2024-03-31T23:08:55.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:55.558+0000 7f49ddd9b640 1 --2- 172.21.15.46:0/4036734589 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f49e0104bc0 0x7f49e0100240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T23:08:55.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:55.558+0000 7f49ded9d640 1 --2- 172.21.15.46:0/4036734589 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f49e01074f0 0x7f49e0100780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T23:08:55.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:55.558+0000 7f49de59c640 1 --2- 172.21.15.46:0/4036734589 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f49e0102290 0x7f49e0101c00 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f49d0006010 tx=0x7f49d00042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:08:55.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:55.558+0000 7f49cf7fe640 1 -- 172.21.15.46:0/4036734589 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49d0043020 con 0x7f49e0102290 2024-03-31T23:08:55.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:55.559+0000 7f49e4a6d640 1 -- 172.21.15.46:0/4036734589 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f49e01a1b00 con 0x7f49e0102290 2024-03-31T23:08:55.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:55.559+0000 7f49cf7fe640 1 -- 172.21.15.46:0/4036734589 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f49d0004aa0 con 0x7f49e0102290 2024-03-31T23:08:55.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:55.559+0000 7f49cf7fe640 1 -- 172.21.15.46:0/4036734589 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49d0003c60 con 0x7f49e0102290 2024-03-31T23:08:55.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:55.559+0000 7f49e4a6d640 1 -- 172.21.15.46:0/4036734589 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f49e01a2050 con 0x7f49e0102290 2024-03-31T23:08:55.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:55.560+0000 7f49e4a6d640 1 -- 172.21.15.46:0/4036734589 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f49e00fd6d0 con 0x7f49e0102290 2024-03-31T23:08:55.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:55.561+0000 7f49cf7fe640 1 -- 172.21.15.46:0/4036734589 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 16) ==== 137320+0+0 (secure 0 0 0) 0x7f49d0046020 con 0x7f49e0102290 2024-03-31T23:08:55.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:55.561+0000 7f49cf7fe640 1 --2- 172.21.15.46:0/4036734589 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f49c80a24a0 0x7f49c80a4960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:08:55.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:55.562+0000 7f49ddd9b640 1 --2- 172.21.15.46:0/4036734589 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f49c80a24a0 0x7f49c80a4960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:08:55.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:55.562+0000 7f49cf7fe640 1 -- 172.21.15.46:0/4036734589 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f49d00f0e30 con 0x7f49e0102290 2024-03-31T23:08:55.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:55.562+0000 7f49ddd9b640 1 --2- 172.21.15.46:0/4036734589 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f49c80a24a0 0x7f49c80a4960 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f49e0101970 tx=0x7f49d4005ec0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:08:55.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:55.565+0000 7f49cf7fe640 1 -- 172.21.15.46:0/4036734589 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f49d004d020 con 0x7f49e0102290 2024-03-31T23:08:55.579 INFO:journalctl@ceph.mds.h.smithi142.stdout:Mar 31 23:08:55 smithi142 podman[65470]: 2024-03-31 23:08:55.188381275 +0000 UTC m=+0.018319873 image pull ec18d7baa60e8b1a0b126fbe5306acce11b1ed0ec5e3bcdddbf39b0a3c591791 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175 2024-03-31T23:08:55.580 INFO:journalctl@ceph.mds.h.smithi142.stdout:Mar 31 23:08:55 smithi142 podman[65470]: 2024-03-31 23:08:55.318903378 +0000 UTC m=+0.148842010 container create eeb232fe0e9b59716285ca3939bc12186a76ea51e560c71ecc84a77f46f3487c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mds-h, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.build-date=20240326, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=squid-384f31f, org.label-schema.schema-version=1.0) 2024-03-31T23:08:55.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:55.790+0000 7f49e4a6d640 1 -- 172.21.15.46:0/4036734589 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f49e002d3c0 con 0x7f49e0102290 2024-03-31T23:08:55.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:55.790+0000 7f49cf7fe640 1 -- 172.21.15.46:0/4036734589 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+6355 (secure 0 0 0) 0x7f49d00be3f0 con 0x7f49e0102290 2024-03-31T23:08:55.791 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-31T23:08:55.791 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":8,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/2169018547","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":2169018547},{"type":"v1","addr":"172.21.15.46:6835","nonce":2169018547}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6837/1069387756","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6836","nonce":1069387756},{"type":"v1","addr":"172.21.15.46:6837","nonce":1069387756}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24415,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6833/1746950966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6832","nonce":1746950966},{"type":"v1","addr":"172.21.15.162:6833","nonce":1746950966}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24422,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6833/3394739181","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6832","nonce":3394739181},{"type":"v1","addr":"172.21.15.142:6833","nonce":3394739181}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24442,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6835/1957208984","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6834","nonce":1957208984},{"type":"v1","addr":"172.21.15.162:6835","nonce":1957208984}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24451,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6837/1371886876","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6836","nonce":1371886876},{"type":"v1","addr":"172.21.15.162:6837","nonce":1371886876}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24455,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6835/212637431","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6834","nonce":212637431},{"type":"v1","addr":"172.21.15.142:6835","nonce":212637431}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[]} 2024-03-31T23:08:55.792 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 8 2024-03-31T23:08:55.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:55.792+0000 7f49e4a6d640 1 -- 172.21.15.46:0/4036734589 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f49c80a24a0 msgr2=0x7f49c80a4960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:08:55.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:55.792+0000 7f49e4a6d640 1 --2- 172.21.15.46:0/4036734589 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f49c80a24a0 0x7f49c80a4960 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f49e0101970 tx=0x7f49d4005ec0 comp rx=0 tx=0).stop 2024-03-31T23:08:55.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:55.792+0000 7f49e4a6d640 1 -- 172.21.15.46:0/4036734589 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f49e0102290 msgr2=0x7f49e0101c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:08:55.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:55.792+0000 7f49e4a6d640 1 --2- 172.21.15.46:0/4036734589 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f49e0102290 0x7f49e0101c00 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f49d0006010 tx=0x7f49d00042f0 comp rx=0 tx=0).stop 2024-03-31T23:08:55.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:55.792+0000 7f49e4a6d640 1 -- 172.21.15.46:0/4036734589 shutdown_connections 2024-03-31T23:08:55.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:55.792+0000 7f49e4a6d640 1 --2- 172.21.15.46:0/4036734589 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f49c80a24a0 0x7f49c80a4960 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:08:55.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:55.792+0000 7f49e4a6d640 1 --2- 172.21.15.46:0/4036734589 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f49e01074f0 0x7f49e0100780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:08:55.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:55.792+0000 7f49e4a6d640 1 --2- 172.21.15.46:0/4036734589 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f49e0104bc0 0x7f49e0100240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:08:55.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:55.792+0000 7f49e4a6d640 1 --2- 172.21.15.46:0/4036734589 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f49e0102290 0x7f49e0101c00 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:08:55.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:55.792+0000 7f49e4a6d640 1 -- 172.21.15.46:0/4036734589 >> 172.21.15.46:0/4036734589 conn(0x7f49e00fbbe0 msgr2=0x7f49e0102cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:08:55.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:55.792+0000 7f49e4a6d640 1 -- 172.21.15.46:0/4036734589 shutdown_connections 2024-03-31T23:08:55.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:55.792+0000 7f49e4a6d640 1 -- 172.21.15.46:0/4036734589 wait complete. 2024-03-31T23:08:55.845 INFO:journalctl@ceph.mds.h.smithi142.stdout:Mar 31 23:08:55 smithi142 podman[65470]: 2024-03-31 23:08:55.580133537 +0000 UTC m=+0.410072125 container init eeb232fe0e9b59716285ca3939bc12186a76ea51e560c71ecc84a77f46f3487c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mds-h, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240326, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.33.5, RELEASE=squid-384f31f) 2024-03-31T23:08:55.845 INFO:journalctl@ceph.mds.h.smithi142.stdout:Mar 31 23:08:55 smithi142 podman[65470]: 2024-03-31 23:08:55.583123307 +0000 UTC m=+0.413061900 container start eeb232fe0e9b59716285ca3939bc12186a76ea51e560c71ecc84a77f46f3487c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mds-h, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, maintainer=Guillaume Abrioux , ceph=True, RELEASE=squid-384f31f, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.build-date=20240326, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-03-31T23:08:55.846 INFO:journalctl@ceph.mds.h.smithi142.stdout:Mar 31 23:08:55 smithi142 ceph-mds[65489]: -- 172.21.15.142:0/4085069731 <== mon.2 v2:172.21.15.142:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x559402253a40 con 0x559402227000 2024-03-31T23:08:55.846 INFO:journalctl@ceph.mds.h.smithi142.stdout:Mar 31 23:08:55 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mds-h[65485]: starting mds.h at 2024-03-31T23:08:55.846 INFO:journalctl@ceph.mds.h.smithi142.stdout:Mar 31 23:08:55 smithi142 bash[65470]: eeb232fe0e9b59716285ca3939bc12186a76ea51e560c71ecc84a77f46f3487c 2024-03-31T23:08:55.846 INFO:journalctl@ceph.mds.h.smithi142.stdout:Mar 31 23:08:55 smithi142 systemd[1]: Started Ceph mds.h for 6a9d8f08-efb2-11ee-b647-cb9ed24678a4. 2024-03-31T23:08:56.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:08:55 smithi046 ceph-mon[32498]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-31T23:08:56.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:08:55 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:08:56.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:08:55 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:08:56.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:08:55 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:08:56.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:08:55 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-31T23:08:56.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:08:55 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-31T23:08:56.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:08:55 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:08:56.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:08:55 smithi046 ceph-mon[32498]: from='client.? 172.21.15.46:0/4036734589' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-31T23:08:56.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:08:55 smithi142 ceph-mon[35271]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-31T23:08:56.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:08:55 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:08:56.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:08:55 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:08:56.194 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:08:55 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:08:56.194 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:08:55 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-31T23:08:56.194 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:08:55 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-31T23:08:56.194 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:08:55 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:08:56.194 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:08:55 smithi142 ceph-mon[35271]: from='client.? 172.21.15.46:0/4036734589' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-31T23:08:56.295 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:08:55 smithi162 ceph-mon[36237]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-31T23:08:56.296 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:08:55 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:08:56.296 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:08:55 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:08:56.296 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:08:55 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:08:56.296 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:08:55 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-31T23:08:56.296 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:08:55 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-31T23:08:56.296 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:08:55 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:08:56.296 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:08:55 smithi162 ceph-mon[36237]: from='client.? 172.21.15.46:0/4036734589' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-31T23:08:57.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:08:56 smithi162 ceph-mon[36237]: Deploying daemon mds.i on smithi162 2024-03-31T23:08:57.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:08:56 smithi162 ceph-mon[36237]: mds.? [v2:172.21.15.142:6836/2559258399,v1:172.21.15.142:6837/2559258399] up:boot 2024-03-31T23:08:57.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:08:56 smithi162 ceph-mon[36237]: fsmap 8 up:standby 2024-03-31T23:08:57.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:08:56 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-31T23:08:57.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:08:56 smithi046 ceph-mon[32498]: Deploying daemon mds.i on smithi162 2024-03-31T23:08:57.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:08:56 smithi046 ceph-mon[32498]: mds.? [v2:172.21.15.142:6836/2559258399,v1:172.21.15.142:6837/2559258399] up:boot 2024-03-31T23:08:57.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:08:56 smithi046 ceph-mon[32498]: fsmap 8 up:standby 2024-03-31T23:08:57.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:08:56 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-31T23:08:57.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:08:56 smithi142 ceph-mon[35271]: Deploying daemon mds.i on smithi162 2024-03-31T23:08:57.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:08:56 smithi142 ceph-mon[35271]: mds.? [v2:172.21.15.142:6836/2559258399,v1:172.21.15.142:6837/2559258399] up:boot 2024-03-31T23:08:57.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:08:56 smithi142 ceph-mon[35271]: fsmap 8 up:standby 2024-03-31T23:08:57.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:08:56 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-31T23:08:57.805 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-31T23:08:57.921 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:57.919+0000 7f72d4b81640 1 -- 172.21.15.46:0/2917344573 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f72d0102240 msgr2=0x7f72d0108ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:08:57.921 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:57.919+0000 7f72d4b81640 1 --2- 172.21.15.46:0/2917344573 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f72d0102240 0x7f72d0108ad0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f72c8009ca0 tx=0x7f72c802c0f0 comp rx=0 tx=0).stop 2024-03-31T23:08:57.921 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:57.919+0000 7f72d4b81640 1 -- 172.21.15.46:0/2917344573 shutdown_connections 2024-03-31T23:08:57.921 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:57.919+0000 7f72d4b81640 1 --2- 172.21.15.46:0/2917344573 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f72d0102240 0x7f72d0108ad0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:08:57.921 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:57.919+0000 7f72d4b81640 1 --2- 172.21.15.46:0/2917344573 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f72d0101880 0x7f72d0101d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:08:57.921 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:57.919+0000 7f72d4b81640 1 --2- 172.21.15.46:0/2917344573 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f72d0100690 0x7f72d0100ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:08:57.921 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:57.919+0000 7f72d4b81640 1 -- 172.21.15.46:0/2917344573 >> 172.21.15.46:0/2917344573 conn(0x7f72d00fbc00 msgr2=0x7f72d00fe040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:08:57.921 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:57.919+0000 7f72d4b81640 1 -- 172.21.15.46:0/2917344573 shutdown_connections 2024-03-31T23:08:57.921 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:57.919+0000 7f72d4b81640 1 -- 172.21.15.46:0/2917344573 wait complete. 2024-03-31T23:08:57.921 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:57.920+0000 7f72d4b81640 1 Processor -- start 2024-03-31T23:08:57.921 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:57.920+0000 7f72d4b81640 1 -- start start 2024-03-31T23:08:57.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:57.920+0000 7f72d4b81640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f72d0100690 0x7f72d01979b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:08:57.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:57.920+0000 7f72d4b81640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f72d0101880 0x7f72d0197ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:08:57.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:57.920+0000 7f72d4b81640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f72d0102240 0x7f72d019ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:08:57.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:57.920+0000 7f72d4b81640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f72d010d2e0 con 0x7f72d0101880 2024-03-31T23:08:57.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:57.920+0000 7f72d4b81640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f72d010d160 con 0x7f72d0102240 2024-03-31T23:08:57.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:57.920+0000 7f72d4b81640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f72d010d460 con 0x7f72d0100690 2024-03-31T23:08:57.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:57.920+0000 7f72cdd9b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f72d0101880 0x7f72d0197ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:08:57.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:57.920+0000 7f72cdd9b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f72d0101880 0x7f72d0197ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:43752/0 (socket says 172.21.15.46:43752) 2024-03-31T23:08:57.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:57.920+0000 7f72cdd9b640 1 -- 172.21.15.46:0/3702128864 learned_addr learned my addr 172.21.15.46:0/3702128864 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:08:57.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:57.920+0000 7f72ce59c640 1 --2- 172.21.15.46:0/3702128864 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f72d0100690 0x7f72d01979b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:08:57.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:57.921+0000 7f72ced9d640 1 --2- 172.21.15.46:0/3702128864 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f72d0102240 0x7f72d019ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:08:57.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:57.921+0000 7f72cdd9b640 1 -- 172.21.15.46:0/3702128864 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f72d0100690 msgr2=0x7f72d01979b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:08:57.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:57.921+0000 7f72cdd9b640 1 --2- 172.21.15.46:0/3702128864 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f72d0100690 0x7f72d01979b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:08:57.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:57.921+0000 7f72cdd9b640 1 -- 172.21.15.46:0/3702128864 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f72d0102240 msgr2=0x7f72d019ef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:08:57.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:57.921+0000 7f72cdd9b640 1 --2- 172.21.15.46:0/3702128864 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f72d0102240 0x7f72d019ef60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:08:57.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:57.921+0000 7f72cdd9b640 1 -- 172.21.15.46:0/3702128864 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72d019f5d0 con 0x7f72d0101880 2024-03-31T23:08:57.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:57.921+0000 7f72ced9d640 1 --2- 172.21.15.46:0/3702128864 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f72d0102240 0x7f72d019ef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T23:08:57.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:57.921+0000 7f72cdd9b640 1 --2- 172.21.15.46:0/3702128864 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f72d0101880 0x7f72d0197ef0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f72c400e9f0 tx=0x7f72c400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:08:57.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:57.921+0000 7f72bf7fe640 1 -- 172.21.15.46:0/3702128864 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72c400cd80 con 0x7f72d0101880 2024-03-31T23:08:57.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:57.921+0000 7f72bf7fe640 1 -- 172.21.15.46:0/3702128864 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f72c40054d0 con 0x7f72d0101880 2024-03-31T23:08:57.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:57.921+0000 7f72d4b81640 1 -- 172.21.15.46:0/3702128864 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f72d019f8c0 con 0x7f72d0101880 2024-03-31T23:08:57.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:57.921+0000 7f72bf7fe640 1 -- 172.21.15.46:0/3702128864 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72c4010490 con 0x7f72d0101880 2024-03-31T23:08:57.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:57.921+0000 7f72d4b81640 1 -- 172.21.15.46:0/3702128864 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f72d019fdc0 con 0x7f72d0101880 2024-03-31T23:08:57.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:57.922+0000 7f72d4b81640 1 -- 172.21.15.46:0/3702128864 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f72d01079b0 con 0x7f72d0101880 2024-03-31T23:08:57.926 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:57.923+0000 7f72bf7fe640 1 -- 172.21.15.46:0/3702128864 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 16) ==== 137320+0+0 (secure 0 0 0) 0x7f72c4005020 con 0x7f72d0101880 2024-03-31T23:08:57.926 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:57.924+0000 7f72bf7fe640 1 --2- 172.21.15.46:0/3702128864 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f72b80a24a0 0x7f72b80a4960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:08:57.926 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:57.924+0000 7f72ce59c640 1 --2- 172.21.15.46:0/3702128864 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f72b80a24a0 0x7f72b80a4960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:08:57.926 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:57.924+0000 7f72bf7fe640 1 -- 172.21.15.46:0/3702128864 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f72c40ce6a0 con 0x7f72d0101880 2024-03-31T23:08:57.926 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:57.924+0000 7f72ce59c640 1 --2- 172.21.15.46:0/3702128864 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f72b80a24a0 0x7f72b80a4960 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f72d01016e0 tx=0x7f72c000a2d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:08:57.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:57.928+0000 7f72bf7fe640 1 -- 172.21.15.46:0/3702128864 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f72c409bd00 con 0x7f72d0101880 2024-03-31T23:08:58.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:08:57 smithi162 ceph-mon[36237]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-31T23:08:58.046 INFO:journalctl@ceph.mds.i.smithi162.stdout:Mar 31 23:08:57 smithi162 systemd[1]: Starting Ceph mds.i for 6a9d8f08-efb2-11ee-b647-cb9ed24678a4... 2024-03-31T23:08:58.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:08:57 smithi046 ceph-mon[32498]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-31T23:08:58.154 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:58.153+0000 7f72d4b81640 1 -- 172.21.15.46:0/3702128864 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f72d002d170 con 0x7f72d0101880 2024-03-31T23:08:58.154 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:58.154+0000 7f72bf7fe640 1 -- 172.21.15.46:0/3702128864 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) ==== 93+0+7183 (secure 0 0 0) 0x7f72c409b600 con 0x7f72d0101880 2024-03-31T23:08:58.155 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-31T23:08:58.155 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":9,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/2169018547","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":2169018547},{"type":"v1","addr":"172.21.15.46:6835","nonce":2169018547}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6837/1069387756","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6836","nonce":1069387756},{"type":"v1","addr":"172.21.15.46:6837","nonce":1069387756}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24415,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6833/1746950966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6832","nonce":1746950966},{"type":"v1","addr":"172.21.15.162:6833","nonce":1746950966}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24422,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6833/3394739181","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6832","nonce":3394739181},{"type":"v1","addr":"172.21.15.142:6833","nonce":3394739181}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24442,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6835/1957208984","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6834","nonce":1957208984},{"type":"v1","addr":"172.21.15.162:6835","nonce":1957208984}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24451,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6837/1371886876","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6836","nonce":1371886876},{"type":"v1","addr":"172.21.15.162:6837","nonce":1371886876}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24455,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6835/212637431","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6834","nonce":212637431},{"type":"v1","addr":"172.21.15.142:6835","nonce":212637431}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24482,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6837/2559258399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6836","nonce":2559258399},{"type":"v1","addr":"172.21.15.142:6837","nonce":2559258399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9}],"filesystems":[]} 2024-03-31T23:08:58.155 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 9 2024-03-31T23:08:58.157 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:58.156+0000 7f72d4b81640 1 -- 172.21.15.46:0/3702128864 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f72b80a24a0 msgr2=0x7f72b80a4960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:08:58.158 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:58.156+0000 7f72d4b81640 1 --2- 172.21.15.46:0/3702128864 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f72b80a24a0 0x7f72b80a4960 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f72d01016e0 tx=0x7f72c000a2d0 comp rx=0 tx=0).stop 2024-03-31T23:08:58.158 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:58.156+0000 7f72d4b81640 1 -- 172.21.15.46:0/3702128864 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f72d0101880 msgr2=0x7f72d0197ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:08:58.158 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:58.156+0000 7f72d4b81640 1 --2- 172.21.15.46:0/3702128864 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f72d0101880 0x7f72d0197ef0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f72c400e9f0 tx=0x7f72c400edf0 comp rx=0 tx=0).stop 2024-03-31T23:08:58.158 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:58.156+0000 7f72d4b81640 1 -- 172.21.15.46:0/3702128864 shutdown_connections 2024-03-31T23:08:58.158 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:58.156+0000 7f72d4b81640 1 --2- 172.21.15.46:0/3702128864 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f72b80a24a0 0x7f72b80a4960 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:08:58.158 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:58.156+0000 7f72d4b81640 1 --2- 172.21.15.46:0/3702128864 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f72d0102240 0x7f72d019ef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:08:58.158 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:58.156+0000 7f72d4b81640 1 --2- 172.21.15.46:0/3702128864 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f72d0101880 0x7f72d0197ef0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:08:58.158 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:58.156+0000 7f72d4b81640 1 --2- 172.21.15.46:0/3702128864 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f72d0100690 0x7f72d01979b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:08:58.158 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:58.156+0000 7f72d4b81640 1 -- 172.21.15.46:0/3702128864 >> 172.21.15.46:0/3702128864 conn(0x7f72d00fbc00 msgr2=0x7f72d0104b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:08:58.158 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:58.156+0000 7f72d4b81640 1 -- 172.21.15.46:0/3702128864 shutdown_connections 2024-03-31T23:08:58.158 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:08:58.156+0000 7f72d4b81640 1 -- 172.21.15.46:0/3702128864 wait complete. 2024-03-31T23:08:58.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:08:57 smithi142 ceph-mon[35271]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-31T23:08:58.374 INFO:journalctl@ceph.mds.i.smithi162.stdout:Mar 31 23:08:58 smithi162 podman[67363]: 2024-03-31 23:08:57.972171228 +0000 UTC m=+0.018129135 image pull ec18d7baa60e8b1a0b126fbe5306acce11b1ed0ec5e3bcdddbf39b0a3c591791 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175 2024-03-31T23:08:58.374 INFO:journalctl@ceph.mds.i.smithi162.stdout:Mar 31 23:08:58 smithi162 podman[67363]: 2024-03-31 23:08:58.121048746 +0000 UTC m=+0.167006667 container create 608a1c3cee32562aa8619764769eccfbeaf3a7427e69faed2e6ad97595d517a0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mds-i, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-384f31f, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.build-date=20240326, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5) 2024-03-31T23:08:58.374 INFO:journalctl@ceph.mds.i.smithi162.stdout:Mar 31 23:08:58 smithi162 podman[67363]: 2024-03-31 23:08:58.373907531 +0000 UTC m=+0.419865430 container init 608a1c3cee32562aa8619764769eccfbeaf3a7427e69faed2e6ad97595d517a0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mds-i, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240326, GIT_CLEAN=True, ceph=True, RELEASE=squid-384f31f, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-03-31T23:08:58.796 INFO:journalctl@ceph.mds.i.smithi162.stdout:Mar 31 23:08:58 smithi162 podman[67363]: 2024-03-31 23:08:58.376617133 +0000 UTC m=+0.422575015 container start 608a1c3cee32562aa8619764769eccfbeaf3a7427e69faed2e6ad97595d517a0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mds-i, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS, RELEASE=squid-384f31f, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.build-date=20240326) 2024-03-31T23:08:58.796 INFO:journalctl@ceph.mds.i.smithi162.stdout:Mar 31 23:08:58 smithi162 ceph-mds[67382]: -- 172.21.15.162:0/648703384 <== mon.1 v2:172.21.15.162:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55825186fa40 con 0x558251843000 2024-03-31T23:08:58.796 INFO:journalctl@ceph.mds.i.smithi162.stdout:Mar 31 23:08:58 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mds-i[67378]: starting mds.i at 2024-03-31T23:08:58.796 INFO:journalctl@ceph.mds.i.smithi162.stdout:Mar 31 23:08:58 smithi162 bash[67363]: 608a1c3cee32562aa8619764769eccfbeaf3a7427e69faed2e6ad97595d517a0 2024-03-31T23:08:58.796 INFO:journalctl@ceph.mds.i.smithi162.stdout:Mar 31 23:08:58 smithi162 systemd[1]: Started Ceph mds.i for 6a9d8f08-efb2-11ee-b647-cb9ed24678a4. 2024-03-31T23:08:59.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:08:58 smithi046 ceph-mon[32498]: from='client.? 172.21.15.46:0/3702128864' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-31T23:08:59.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:08:58 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:08:59.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:08:58 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:08:59.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:08:58 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:08:59.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:08:58 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-31T23:08:59.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:08:58 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-31T23:08:59.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:08:58 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:08:59.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:08:58 smithi046 ceph-mon[32498]: Deploying daemon mds.g on smithi046 2024-03-31T23:08:59.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:08:58 smithi142 ceph-mon[35271]: from='client.? 172.21.15.46:0/3702128864' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-31T23:08:59.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:08:58 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:08:59.194 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:08:58 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:08:59.194 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:08:58 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:08:59.194 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:08:58 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-31T23:08:59.194 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:08:58 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-31T23:08:59.194 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:08:58 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:08:59.194 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:08:58 smithi142 ceph-mon[35271]: Deploying daemon mds.g on smithi046 2024-03-31T23:08:59.295 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:08:58 smithi162 ceph-mon[36237]: from='client.? 172.21.15.46:0/3702128864' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-31T23:08:59.296 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:08:58 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:08:59.296 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:08:58 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:08:59.296 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:08:58 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:08:59.296 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:08:58 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-31T23:08:59.296 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:08:58 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-31T23:08:59.296 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:08:58 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:08:59.296 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:08:58 smithi162 ceph-mon[36237]: Deploying daemon mds.g on smithi046 2024-03-31T23:09:00.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:08:59 smithi046 ceph-mon[32498]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-31T23:09:00.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:08:59 smithi046 ceph-mon[32498]: mds.? [v2:172.21.15.162:6838/1976275826,v1:172.21.15.162:6839/1976275826] up:boot 2024-03-31T23:09:00.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:08:59 smithi046 ceph-mon[32498]: fsmap 9 up:standby 2024-03-31T23:09:00.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:08:59 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-31T23:09:00.168 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-31T23:09:00.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:08:59 smithi142 ceph-mon[35271]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-31T23:09:00.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:08:59 smithi142 ceph-mon[35271]: mds.? [v2:172.21.15.162:6838/1976275826,v1:172.21.15.162:6839/1976275826] up:boot 2024-03-31T23:09:00.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:08:59 smithi142 ceph-mon[35271]: fsmap 9 up:standby 2024-03-31T23:09:00.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:08:59 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-31T23:09:00.282 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:00.279+0000 7fd1fabeb640 1 -- 172.21.15.46:0/573927925 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd1f4102af0 msgr2=0x7fd1f4102f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:09:00.282 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:00.279+0000 7fd1fabeb640 1 --2- 172.21.15.46:0/573927925 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd1f4102af0 0x7fd1f4102f70 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fd1e0009930 tx=0x7fd1e002bc30 comp rx=0 tx=0).stop 2024-03-31T23:09:00.282 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:00.279+0000 7fd1fabeb640 1 -- 172.21.15.46:0/573927925 shutdown_connections 2024-03-31T23:09:00.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:00.279+0000 7fd1fabeb640 1 --2- 172.21.15.46:0/573927925 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd1f41034b0 0x7fd1f4109d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:00.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:00.279+0000 7fd1fabeb640 1 --2- 172.21.15.46:0/573927925 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd1f4102af0 0x7fd1f4102f70 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:00.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:00.279+0000 7fd1fabeb640 1 --2- 172.21.15.46:0/573927925 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd1f40ffa80 0x7fd1f40ffea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:00.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:00.279+0000 7fd1fabeb640 1 -- 172.21.15.46:0/573927925 >> 172.21.15.46:0/573927925 conn(0x7fd1f40fb390 msgr2=0x7fd1f40fd7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:09:00.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:00.279+0000 7fd1fabeb640 1 -- 172.21.15.46:0/573927925 shutdown_connections 2024-03-31T23:09:00.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:00.279+0000 7fd1fabeb640 1 -- 172.21.15.46:0/573927925 wait complete. 2024-03-31T23:09:00.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:00.279+0000 7fd1fabeb640 1 Processor -- start 2024-03-31T23:09:00.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:00.280+0000 7fd1fabeb640 1 -- start start 2024-03-31T23:09:00.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:00.280+0000 7fd1fabeb640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd1f40ffa80 0x7fd1f41979d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:09:00.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:00.280+0000 7fd1fabeb640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd1f4102af0 0x7fd1f4197f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:09:00.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:00.280+0000 7fd1fabeb640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd1f41034b0 0x7fd1f419ef80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:09:00.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:00.280+0000 7fd1fabeb640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fd1f410d330 con 0x7fd1f41034b0 2024-03-31T23:09:00.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:00.280+0000 7fd1fabeb640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fd1f410d1b0 con 0x7fd1f40ffa80 2024-03-31T23:09:00.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:00.280+0000 7fd1fabeb640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fd1f410d4b0 con 0x7fd1f4102af0 2024-03-31T23:09:00.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:00.280+0000 7fd1f9188640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd1f41034b0 0x7fd1f419ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:09:00.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:00.280+0000 7fd1f9188640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd1f41034b0 0x7fd1f419ef80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:43766/0 (socket says 172.21.15.46:43766) 2024-03-31T23:09:00.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:00.280+0000 7fd1f9188640 1 -- 172.21.15.46:0/4098097287 learned_addr learned my addr 172.21.15.46:0/4098097287 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:09:00.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:00.280+0000 7fd1f3fff640 1 --2- 172.21.15.46:0/4098097287 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd1f4102af0 0x7fd1f4197f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:09:00.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:00.280+0000 7fd1f8987640 1 --2- 172.21.15.46:0/4098097287 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd1f40ffa80 0x7fd1f41979d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:09:00.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:00.280+0000 7fd1f9188640 1 -- 172.21.15.46:0/4098097287 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd1f4102af0 msgr2=0x7fd1f4197f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:09:00.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:00.280+0000 7fd1f9188640 1 --2- 172.21.15.46:0/4098097287 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd1f4102af0 0x7fd1f4197f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:00.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:00.280+0000 7fd1f9188640 1 -- 172.21.15.46:0/4098097287 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd1f40ffa80 msgr2=0x7fd1f41979d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:09:00.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:00.280+0000 7fd1f9188640 1 --2- 172.21.15.46:0/4098097287 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd1f40ffa80 0x7fd1f41979d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:00.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:00.280+0000 7fd1f9188640 1 -- 172.21.15.46:0/4098097287 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd1f419f5f0 con 0x7fd1f41034b0 2024-03-31T23:09:00.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:00.280+0000 7fd1f9188640 1 --2- 172.21.15.46:0/4098097287 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd1f41034b0 0x7fd1f419ef80 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fd1ec00eaa0 tx=0x7fd1ec00ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:09:00.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:00.280+0000 7fd1f1ffb640 1 -- 172.21.15.46:0/4098097287 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1ec00cdb0 con 0x7fd1f41034b0 2024-03-31T23:09:00.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:00.280+0000 7fd1f1ffb640 1 -- 172.21.15.46:0/4098097287 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd1ec005020 con 0x7fd1f41034b0 2024-03-31T23:09:00.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:00.280+0000 7fd1f1ffb640 1 -- 172.21.15.46:0/4098097287 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1ec0116d0 con 0x7fd1f41034b0 2024-03-31T23:09:00.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:00.280+0000 7fd1fabeb640 1 -- 172.21.15.46:0/4098097287 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd1f419f8e0 con 0x7fd1f41034b0 2024-03-31T23:09:00.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:00.281+0000 7fd1fabeb640 1 -- 172.21.15.46:0/4098097287 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd1f419fe00 con 0x7fd1f41034b0 2024-03-31T23:09:00.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:00.282+0000 7fd1fabeb640 1 -- 172.21.15.46:0/4098097287 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd1f4108c20 con 0x7fd1f41034b0 2024-03-31T23:09:00.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:00.283+0000 7fd1f1ffb640 1 -- 172.21.15.46:0/4098097287 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 16) ==== 137320+0+0 (secure 0 0 0) 0x7fd1ec0051d0 con 0x7fd1f41034b0 2024-03-31T23:09:00.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:00.283+0000 7fd1f1ffb640 1 --2- 172.21.15.46:0/4098097287 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fd1e40a24a0 0x7fd1e40a4960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:09:00.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:00.283+0000 7fd1f8987640 1 --2- 172.21.15.46:0/4098097287 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fd1e40a24a0 0x7fd1e40a4960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:09:00.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:00.283+0000 7fd1f1ffb640 1 -- 172.21.15.46:0/4098097287 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7fd1ec0ce3a0 con 0x7fd1f41034b0 2024-03-31T23:09:00.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:00.284+0000 7fd1f8987640 1 --2- 172.21.15.46:0/4098097287 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fd1e40a24a0 0x7fd1e40a4960 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fd1e8005d00 tx=0x7fd1e8005c50 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:09:00.288 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:00.288+0000 7fd1f1ffb640 1 -- 172.21.15.46:0/4098097287 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fd1ec016030 con 0x7fd1f41034b0 2024-03-31T23:09:00.295 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:08:59 smithi162 ceph-mon[36237]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-31T23:09:00.295 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:08:59 smithi162 ceph-mon[36237]: mds.? [v2:172.21.15.162:6838/1976275826,v1:172.21.15.162:6839/1976275826] up:boot 2024-03-31T23:09:00.296 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:08:59 smithi162 ceph-mon[36237]: fsmap 9 up:standby 2024-03-31T23:09:00.296 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:08:59 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-31T23:09:00.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:00.516+0000 7fd1fabeb640 1 -- 172.21.15.46:0/4098097287 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd1f4062350 con 0x7fd1f41034b0 2024-03-31T23:09:00.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:00.517+0000 7fd1f1ffb640 1 -- 172.21.15.46:0/4098097287 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) ==== 94+0+8013 (secure 0 0 0) 0x7fd1ec09ba00 con 0x7fd1f41034b0 2024-03-31T23:09:00.518 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-31T23:09:00.519 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":10,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/2169018547","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":2169018547},{"type":"v1","addr":"172.21.15.46:6835","nonce":2169018547}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6837/1069387756","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6836","nonce":1069387756},{"type":"v1","addr":"172.21.15.46:6837","nonce":1069387756}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24415,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6833/1746950966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6832","nonce":1746950966},{"type":"v1","addr":"172.21.15.162:6833","nonce":1746950966}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24422,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6833/3394739181","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6832","nonce":3394739181},{"type":"v1","addr":"172.21.15.142:6833","nonce":3394739181}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24442,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6835/1957208984","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6834","nonce":1957208984},{"type":"v1","addr":"172.21.15.162:6835","nonce":1957208984}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24451,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6837/1371886876","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6836","nonce":1371886876},{"type":"v1","addr":"172.21.15.162:6837","nonce":1371886876}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24455,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6835/212637431","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6834","nonce":212637431},{"type":"v1","addr":"172.21.15.142:6835","nonce":212637431}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24478,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6839/1976275826","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6838","nonce":1976275826},{"type":"v1","addr":"172.21.15.162:6839","nonce":1976275826}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24482,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6837/2559258399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6836","nonce":2559258399},{"type":"v1","addr":"172.21.15.142:6837","nonce":2559258399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9}],"filesystems":[]} 2024-03-31T23:09:00.519 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 10 2024-03-31T23:09:00.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:00.519+0000 7fd1fabeb640 1 -- 172.21.15.46:0/4098097287 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fd1e40a24a0 msgr2=0x7fd1e40a4960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:09:00.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:00.519+0000 7fd1fabeb640 1 --2- 172.21.15.46:0/4098097287 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fd1e40a24a0 0x7fd1e40a4960 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fd1e8005d00 tx=0x7fd1e8005c50 comp rx=0 tx=0).stop 2024-03-31T23:09:00.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:00.519+0000 7fd1fabeb640 1 -- 172.21.15.46:0/4098097287 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd1f41034b0 msgr2=0x7fd1f419ef80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:09:00.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:00.519+0000 7fd1fabeb640 1 --2- 172.21.15.46:0/4098097287 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd1f41034b0 0x7fd1f419ef80 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fd1ec00eaa0 tx=0x7fd1ec00ef60 comp rx=0 tx=0).stop 2024-03-31T23:09:00.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:00.520+0000 7fd1fabeb640 1 -- 172.21.15.46:0/4098097287 shutdown_connections 2024-03-31T23:09:00.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:00.520+0000 7fd1fabeb640 1 --2- 172.21.15.46:0/4098097287 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fd1e40a24a0 0x7fd1e40a4960 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:00.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:00.520+0000 7fd1fabeb640 1 --2- 172.21.15.46:0/4098097287 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd1f41034b0 0x7fd1f419ef80 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:00.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:00.520+0000 7fd1fabeb640 1 --2- 172.21.15.46:0/4098097287 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd1f4102af0 0x7fd1f4197f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:00.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:00.520+0000 7fd1fabeb640 1 --2- 172.21.15.46:0/4098097287 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd1f40ffa80 0x7fd1f41979d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:00.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:00.520+0000 7fd1fabeb640 1 -- 172.21.15.46:0/4098097287 >> 172.21.15.46:0/4098097287 conn(0x7fd1f40fb390 msgr2=0x7fd1f4105df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:09:00.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:00.520+0000 7fd1fabeb640 1 -- 172.21.15.46:0/4098097287 shutdown_connections 2024-03-31T23:09:00.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:00.520+0000 7fd1fabeb640 1 -- 172.21.15.46:0/4098097287 wait complete. 2024-03-31T23:09:01.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:09:00 smithi046 ceph-mon[32498]: from='client.? 172.21.15.46:0/4098097287' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-31T23:09:01.112 INFO:journalctl@ceph.mds.g.smithi046.stdout:Mar 31 23:09:01 smithi046 podman[73669]: 2024-03-31 23:09:01.058313139 +0000 UTC m=+0.147373477 container create b4331229633e8ef56dc7c1666bed6b78cabde29bdf8fe3bda0f4fd8f459732cd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mds-g, ceph=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.build-date=20240326, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-384f31f) 2024-03-31T23:09:01.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:09:00 smithi142 ceph-mon[35271]: from='client.? 172.21.15.46:0/4098097287' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-31T23:09:01.296 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:09:00 smithi162 ceph-mon[36237]: from='client.? 172.21.15.46:0/4098097287' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-31T23:09:01.611 INFO:journalctl@ceph.mds.g.smithi046.stdout:Mar 31 23:09:01 smithi046 podman[73669]: 2024-03-31 23:09:01.319790492 +0000 UTC m=+0.408850829 container init b4331229633e8ef56dc7c1666bed6b78cabde29bdf8fe3bda0f4fd8f459732cd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mds-g, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, RELEASE=squid-384f31f, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240326, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-31T23:09:01.612 INFO:journalctl@ceph.mds.g.smithi046.stdout:Mar 31 23:09:01 smithi046 podman[73669]: 2024-03-31 23:09:01.322618591 +0000 UTC m=+0.411678933 container start b4331229633e8ef56dc7c1666bed6b78cabde29bdf8fe3bda0f4fd8f459732cd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mds-g, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, RELEASE=squid-384f31f, org.label-schema.license=GPLv2, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.build-date=20240326, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-31T23:09:01.612 INFO:journalctl@ceph.mds.g.smithi046.stdout:Mar 31 23:09:01 smithi046 ceph-mds[73688]: -- 172.21.15.46:0/171850533 <== mon.0 v2:172.21.15.46:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55ad3ab09a40 con 0x55ad3aadd000 2024-03-31T23:09:01.612 INFO:journalctl@ceph.mds.g.smithi046.stdout:Mar 31 23:09:01 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mds-g[73684]: starting mds.g at 2024-03-31T23:09:01.612 INFO:journalctl@ceph.mds.g.smithi046.stdout:Mar 31 23:09:01 smithi046 bash[73669]: b4331229633e8ef56dc7c1666bed6b78cabde29bdf8fe3bda0f4fd8f459732cd 2024-03-31T23:09:01.612 INFO:journalctl@ceph.mds.g.smithi046.stdout:Mar 31 23:09:01 smithi046 systemd[1]: Started Ceph mds.g for 6a9d8f08-efb2-11ee-b647-cb9ed24678a4. 2024-03-31T23:09:02.533 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-31T23:09:02.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:02.642+0000 7facf5e9a640 1 -- 172.21.15.46:0/2517108897 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7facf0102220 msgr2=0x7facf0108ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:09:02.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:02.642+0000 7facf5e9a640 1 --2- 172.21.15.46:0/2517108897 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7facf0102220 0x7facf0108ab0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7face8009ca0 tx=0x7face802c0f0 comp rx=0 tx=0).stop 2024-03-31T23:09:02.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:02.642+0000 7facf5e9a640 1 -- 172.21.15.46:0/2517108897 shutdown_connections 2024-03-31T23:09:02.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:02.642+0000 7facf5e9a640 1 --2- 172.21.15.46:0/2517108897 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7facf0102220 0x7facf0108ab0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:02.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:02.642+0000 7facf5e9a640 1 --2- 172.21.15.46:0/2517108897 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7facf0101860 0x7facf0101ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:02.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:02.642+0000 7facf5e9a640 1 --2- 172.21.15.46:0/2517108897 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7facf0100670 0x7facf0100a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:02.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:02.642+0000 7facf5e9a640 1 -- 172.21.15.46:0/2517108897 >> 172.21.15.46:0/2517108897 conn(0x7facf00fbbe0 msgr2=0x7facf00fe020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:09:02.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:02.643+0000 7facf5e9a640 1 -- 172.21.15.46:0/2517108897 shutdown_connections 2024-03-31T23:09:02.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:02.643+0000 7facf5e9a640 1 -- 172.21.15.46:0/2517108897 wait complete. 2024-03-31T23:09:02.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:02.643+0000 7facf5e9a640 1 Processor -- start 2024-03-31T23:09:02.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:02.643+0000 7facf5e9a640 1 -- start start 2024-03-31T23:09:02.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:02.643+0000 7facf5e9a640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7facf0100670 0x7facf0069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:09:02.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:02.643+0000 7facf5e9a640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7facf0101860 0x7facf019c6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:09:02.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:02.643+0000 7facf5e9a640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7facf0102220 0x7facf019ece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:09:02.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:02.643+0000 7facf5e9a640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7facf010c1a0 con 0x7facf0101860 2024-03-31T23:09:02.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:02.643+0000 7facf5e9a640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7facf010c020 con 0x7facf0100670 2024-03-31T23:09:02.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:02.643+0000 7facf5e9a640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7facf010c320 con 0x7facf0102220 2024-03-31T23:09:02.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:02.644+0000 7faceeffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7facf0101860 0x7facf019c6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:09:02.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:02.644+0000 7faceeffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7facf0101860 0x7facf019c6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:48950/0 (socket says 172.21.15.46:48950) 2024-03-31T23:09:02.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:02.644+0000 7faceeffd640 1 -- 172.21.15.46:0/1682746451 learned_addr learned my addr 172.21.15.46:0/1682746451 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:09:02.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:02.644+0000 7facef7fe640 1 --2- 172.21.15.46:0/1682746451 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7facf0100670 0x7facf0069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:09:02.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:02.644+0000 7faceffff640 1 --2- 172.21.15.46:0/1682746451 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7facf0102220 0x7facf019ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:09:02.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:02.644+0000 7faceeffd640 1 -- 172.21.15.46:0/1682746451 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7facf0102220 msgr2=0x7facf019ece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:09:02.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:02.644+0000 7faceeffd640 1 --2- 172.21.15.46:0/1682746451 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7facf0102220 0x7facf019ece0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:02.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:02.644+0000 7faceeffd640 1 -- 172.21.15.46:0/1682746451 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7facf0100670 msgr2=0x7facf0069420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:09:02.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:02.644+0000 7faceeffd640 1 --2- 172.21.15.46:0/1682746451 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7facf0100670 0x7facf0069420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:02.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:02.644+0000 7faceeffd640 1 -- 172.21.15.46:0/1682746451 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7facf019f5f0 con 0x7facf0101860 2024-03-31T23:09:02.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:02.644+0000 7faceffff640 1 --2- 172.21.15.46:0/1682746451 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7facf0102220 0x7facf019ece0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T23:09:02.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:02.644+0000 7facef7fe640 1 --2- 172.21.15.46:0/1682746451 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7facf0100670 0x7facf0069420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T23:09:02.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:02.644+0000 7faceeffd640 1 --2- 172.21.15.46:0/1682746451 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7facf0101860 0x7facf019c6c0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7face000c890 tx=0x7face000cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:09:02.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:02.644+0000 7facecff9640 1 -- 172.21.15.46:0/1682746451 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7face0004220 con 0x7facf0101860 2024-03-31T23:09:02.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:02.644+0000 7facecff9640 1 -- 172.21.15.46:0/1682746451 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7face00043c0 con 0x7facf0101860 2024-03-31T23:09:02.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:02.644+0000 7facecff9640 1 -- 172.21.15.46:0/1682746451 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7face0004a00 con 0x7facf0101860 2024-03-31T23:09:02.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:02.644+0000 7facf5e9a640 1 -- 172.21.15.46:0/1682746451 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7facf019f880 con 0x7facf0101860 2024-03-31T23:09:02.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:02.644+0000 7facf5e9a640 1 -- 172.21.15.46:0/1682746451 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7facf019fdd0 con 0x7facf0101860 2024-03-31T23:09:02.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:02.646+0000 7facf5e9a640 1 -- 172.21.15.46:0/1682746451 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7facf01079e0 con 0x7facf0101860 2024-03-31T23:09:02.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:02.647+0000 7facecff9640 1 -- 172.21.15.46:0/1682746451 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 16) ==== 137320+0+0 (secure 0 0 0) 0x7face0004ba0 con 0x7facf0101860 2024-03-31T23:09:02.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:02.648+0000 7facecff9640 1 --2- 172.21.15.46:0/1682746451 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7facd80a24a0 0x7facd80a4960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:09:02.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:02.648+0000 7facef7fe640 1 --2- 172.21.15.46:0/1682746451 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7facd80a24a0 0x7facd80a4960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:09:02.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:02.648+0000 7facecff9640 1 -- 172.21.15.46:0/1682746451 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7face00cdfe0 con 0x7facf0101860 2024-03-31T23:09:02.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:02.649+0000 7facef7fe640 1 --2- 172.21.15.46:0/1682746451 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7facd80a24a0 0x7facd80a4960 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7face40058c0 tx=0x7face4005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:09:02.654 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:02.653+0000 7facecff9640 1 -- 172.21.15.46:0/1682746451 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7face0014030 con 0x7facf0101860 2024-03-31T23:09:02.702 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:09:02 smithi046 ceph-mon[32498]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-31T23:09:02.702 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:09:02 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:09:02.702 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:09:02 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:09:02.702 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:09:02 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:09:02.702 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:09:02 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-31T23:09:02.702 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:09:02 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-31T23:09:02.703 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:09:02 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:09:02.703 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:09:02 smithi046 ceph-mon[32498]: Deploying daemon mds.a on smithi046 2024-03-31T23:09:02.703 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:09:02 smithi046 ceph-mon[32498]: mds.? [v2:172.21.15.46:6838/3915337844,v1:172.21.15.46:6839/3915337844] up:boot 2024-03-31T23:09:02.703 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:09:02 smithi046 ceph-mon[32498]: fsmap 10 up:standby 2024-03-31T23:09:02.703 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:09:02 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-31T23:09:02.709 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:09:02 smithi162 ceph-mon[36237]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-31T23:09:02.710 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:09:02 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:09:02.710 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:09:02 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:09:02.710 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:09:02 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:09:02.710 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:09:02 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-31T23:09:02.710 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:09:02 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-31T23:09:02.710 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:09:02 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:09:02.710 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:09:02 smithi162 ceph-mon[36237]: Deploying daemon mds.a on smithi046 2024-03-31T23:09:02.710 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:09:02 smithi162 ceph-mon[36237]: mds.? [v2:172.21.15.46:6838/3915337844,v1:172.21.15.46:6839/3915337844] up:boot 2024-03-31T23:09:02.710 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:09:02 smithi162 ceph-mon[36237]: fsmap 10 up:standby 2024-03-31T23:09:02.710 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:09:02 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-31T23:09:02.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:02.880+0000 7facf5e9a640 1 -- 172.21.15.46:0/1682746451 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7facf0062350 con 0x7facf0101860 2024-03-31T23:09:02.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:02.881+0000 7facecff9640 1 -- 172.21.15.46:0/1682746451 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 11 v11) ==== 94+0+8839 (secure 0 0 0) 0x7face009b640 con 0x7facf0101860 2024-03-31T23:09:02.882 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-31T23:09:02.882 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":11,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/2169018547","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":2169018547},{"type":"v1","addr":"172.21.15.46:6835","nonce":2169018547}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6837/1069387756","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6836","nonce":1069387756},{"type":"v1","addr":"172.21.15.46:6837","nonce":1069387756}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14634,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6839/3915337844","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6838","nonce":3915337844},{"type":"v1","addr":"172.21.15.46:6839","nonce":3915337844}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24415,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6833/1746950966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6832","nonce":1746950966},{"type":"v1","addr":"172.21.15.162:6833","nonce":1746950966}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24422,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6833/3394739181","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6832","nonce":3394739181},{"type":"v1","addr":"172.21.15.142:6833","nonce":3394739181}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24442,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6835/1957208984","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6834","nonce":1957208984},{"type":"v1","addr":"172.21.15.162:6835","nonce":1957208984}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24451,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6837/1371886876","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6836","nonce":1371886876},{"type":"v1","addr":"172.21.15.162:6837","nonce":1371886876}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24455,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6835/212637431","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6834","nonce":212637431},{"type":"v1","addr":"172.21.15.142:6835","nonce":212637431}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24478,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6839/1976275826","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6838","nonce":1976275826},{"type":"v1","addr":"172.21.15.162:6839","nonce":1976275826}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24482,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6837/2559258399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6836","nonce":2559258399},{"type":"v1","addr":"172.21.15.142:6837","nonce":2559258399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9}],"filesystems":[]} 2024-03-31T23:09:02.883 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 11 2024-03-31T23:09:02.884 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:02.883+0000 7facf5e9a640 1 -- 172.21.15.46:0/1682746451 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7facd80a24a0 msgr2=0x7facd80a4960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:09:02.884 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:02.883+0000 7facf5e9a640 1 --2- 172.21.15.46:0/1682746451 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7facd80a24a0 0x7facd80a4960 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7face40058c0 tx=0x7face4005850 comp rx=0 tx=0).stop 2024-03-31T23:09:02.884 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:02.883+0000 7facf5e9a640 1 -- 172.21.15.46:0/1682746451 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7facf0101860 msgr2=0x7facf019c6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:09:02.884 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:02.883+0000 7facf5e9a640 1 --2- 172.21.15.46:0/1682746451 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7facf0101860 0x7facf019c6c0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7face000c890 tx=0x7face000cd50 comp rx=0 tx=0).stop 2024-03-31T23:09:02.884 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:02.883+0000 7facf5e9a640 1 -- 172.21.15.46:0/1682746451 shutdown_connections 2024-03-31T23:09:02.884 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:02.883+0000 7facf5e9a640 1 --2- 172.21.15.46:0/1682746451 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7facd80a24a0 0x7facd80a4960 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:02.884 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:02.883+0000 7facf5e9a640 1 --2- 172.21.15.46:0/1682746451 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7facf0102220 0x7facf019ece0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:02.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:02.883+0000 7facf5e9a640 1 --2- 172.21.15.46:0/1682746451 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7facf0101860 0x7facf019c6c0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:02.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:02.883+0000 7facf5e9a640 1 --2- 172.21.15.46:0/1682746451 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7facf0100670 0x7facf0069420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:02.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:02.883+0000 7facf5e9a640 1 -- 172.21.15.46:0/1682746451 >> 172.21.15.46:0/1682746451 conn(0x7facf00fbbe0 msgr2=0x7facf0104bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:09:02.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:02.883+0000 7facf5e9a640 1 -- 172.21.15.46:0/1682746451 shutdown_connections 2024-03-31T23:09:02.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:02.883+0000 7facf5e9a640 1 -- 172.21.15.46:0/1682746451 wait complete. 2024-03-31T23:09:02.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:09:02 smithi142 ceph-mon[35271]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-31T23:09:02.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:09:02 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:09:02.944 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:09:02 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:09:02.944 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:09:02 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:09:02.944 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:09:02 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-31T23:09:02.944 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:09:02 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-31T23:09:02.944 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:09:02 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:09:02.944 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:09:02 smithi142 ceph-mon[35271]: Deploying daemon mds.a on smithi046 2024-03-31T23:09:02.944 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:09:02 smithi142 ceph-mon[35271]: mds.? [v2:172.21.15.46:6838/3915337844,v1:172.21.15.46:6839/3915337844] up:boot 2024-03-31T23:09:02.944 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:09:02 smithi142 ceph-mon[35271]: fsmap 10 up:standby 2024-03-31T23:09:02.944 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:09:02 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-31T23:09:03.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:09:03 smithi162 ceph-mon[36237]: from='client.? 172.21.15.46:0/1682746451' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-31T23:09:03.824 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:09:03 smithi046 ceph-mon[32498]: from='client.? 172.21.15.46:0/1682746451' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-31T23:09:03.824 INFO:journalctl@ceph.mds.a.smithi046.stdout:Mar 31 23:09:03 smithi046 systemd[1]: Starting Ceph mds.a for 6a9d8f08-efb2-11ee-b647-cb9ed24678a4... 2024-03-31T23:09:03.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:09:03 smithi142 ceph-mon[35271]: from='client.? 172.21.15.46:0/1682746451' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-31T23:09:04.111 INFO:journalctl@ceph.mds.a.smithi046.stdout:Mar 31 23:09:03 smithi046 podman[74031]: 2024-03-31 23:09:03.726432229 +0000 UTC m=+0.017456168 image pull ec18d7baa60e8b1a0b126fbe5306acce11b1ed0ec5e3bcdddbf39b0a3c591791 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175 2024-03-31T23:09:04.112 INFO:journalctl@ceph.mds.a.smithi046.stdout:Mar 31 23:09:03 smithi046 podman[74031]: 2024-03-31 23:09:03.839733177 +0000 UTC m=+0.130757096 container create c6deca2be3eca8883012d6efae149642a0986a2f39ea83312bc465081a3e7b7c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mds-a, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-384f31f, org.label-schema.build-date=20240326, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True) 2024-03-31T23:09:04.469 INFO:journalctl@ceph.mds.a.smithi046.stdout:Mar 31 23:09:04 smithi046 podman[74031]: 2024-03-31 23:09:04.309731583 +0000 UTC m=+0.600755523 container init c6deca2be3eca8883012d6efae149642a0986a2f39ea83312bc465081a3e7b7c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mds-a, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=squid-384f31f, org.label-schema.build-date=20240326, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, ceph=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-03-31T23:09:04.469 INFO:journalctl@ceph.mds.a.smithi046.stdout:Mar 31 23:09:04 smithi046 podman[74031]: 2024-03-31 23:09:04.312402528 +0000 UTC m=+0.603426456 container start c6deca2be3eca8883012d6efae149642a0986a2f39ea83312bc465081a3e7b7c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mds-a, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , RELEASE=squid-384f31f, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240326, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, ceph=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0) 2024-03-31T23:09:04.469 INFO:journalctl@ceph.mds.a.smithi046.stdout:Mar 31 23:09:04 smithi046 ceph-mds[74051]: -- 172.21.15.46:0/2606346066 <== mon.0 v2:172.21.15.46:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55bb8b8d3a40 con 0x55bb8b8a6c00 2024-03-31T23:09:04.469 INFO:journalctl@ceph.mds.a.smithi046.stdout:Mar 31 23:09:04 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mds-a[74047]: starting mds.a at 2024-03-31T23:09:04.469 INFO:journalctl@ceph.mds.a.smithi046.stdout:Mar 31 23:09:04 smithi046 bash[74031]: c6deca2be3eca8883012d6efae149642a0986a2f39ea83312bc465081a3e7b7c 2024-03-31T23:09:04.469 INFO:journalctl@ceph.mds.a.smithi046.stdout:Mar 31 23:09:04 smithi046 systemd[1]: Started Ceph mds.a for 6a9d8f08-efb2-11ee-b647-cb9ed24678a4. 2024-03-31T23:09:04.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:09:04 smithi162 ceph-mon[36237]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-31T23:09:04.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:09:04 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:09:04.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:09:04 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:09:04.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:09:04 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:09:04.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:09:04 smithi046 ceph-mon[32498]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-31T23:09:04.862 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:09:04 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:09:04.862 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:09:04 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:09:04.862 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:09:04 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:09:04.896 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-31T23:09:04.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:09:04 smithi142 ceph-mon[35271]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-31T23:09:04.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:09:04 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:09:04.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:09:04 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:09:04.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:09:04 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:09:05.010 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:05.009+0000 7f73e8f25640 1 -- 172.21.15.46:0/3084468813 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f73e4102970 msgr2=0x7f73e4102df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:09:05.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:05.009+0000 7f73e8f25640 1 --2- 172.21.15.46:0/3084468813 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f73e4102970 0x7f73e4102df0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f73cc009900 tx=0x7f73cc02bc30 comp rx=0 tx=0).stop 2024-03-31T23:09:05.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:05.009+0000 7f73e8f25640 1 -- 172.21.15.46:0/3084468813 shutdown_connections 2024-03-31T23:09:05.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:05.009+0000 7f73e8f25640 1 --2- 172.21.15.46:0/3084468813 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f73e4103330 0x7f73e4109bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:05.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:05.009+0000 7f73e8f25640 1 --2- 172.21.15.46:0/3084468813 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f73e4102970 0x7f73e4102df0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:05.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:05.009+0000 7f73e8f25640 1 --2- 172.21.15.46:0/3084468813 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f73e4101780 0x7f73e4101ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:05.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:05.009+0000 7f73e8f25640 1 -- 172.21.15.46:0/3084468813 >> 172.21.15.46:0/3084468813 conn(0x7f73e40fcd30 msgr2=0x7f73e40ff150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:09:05.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:05.009+0000 7f73e8f25640 1 -- 172.21.15.46:0/3084468813 shutdown_connections 2024-03-31T23:09:05.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:05.009+0000 7f73e8f25640 1 -- 172.21.15.46:0/3084468813 wait complete. 2024-03-31T23:09:05.014 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:05.010+0000 7f73e8f25640 1 Processor -- start 2024-03-31T23:09:05.014 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:05.010+0000 7f73e8f25640 1 -- start start 2024-03-31T23:09:05.014 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:05.011+0000 7f73e8f25640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f73e4101780 0x7f73e4075460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:09:05.014 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:05.011+0000 7f73e8f25640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f73e4102970 0x7f73e4073b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:09:05.014 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:05.011+0000 7f73e8f25640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f73e4103330 0x7f73e4074080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:09:05.014 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:05.011+0000 7f73e8f25640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f73e410d3a0 con 0x7f73e4102970 2024-03-31T23:09:05.014 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:05.011+0000 7f73e8f25640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f73e410d220 con 0x7f73e4103330 2024-03-31T23:09:05.014 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:05.011+0000 7f73e8f25640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f73e410d520 con 0x7f73e4101780 2024-03-31T23:09:05.014 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:05.011+0000 7f73d9d9b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f73e4102970 0x7f73e4073b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:09:05.014 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:05.011+0000 7f73d9d9b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f73e4102970 0x7f73e4073b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:48990/0 (socket says 172.21.15.46:48990) 2024-03-31T23:09:05.015 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:05.011+0000 7f73d9d9b640 1 -- 172.21.15.46:0/4156966031 learned_addr learned my addr 172.21.15.46:0/4156966031 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:09:05.015 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:05.011+0000 7f73e2d9d640 1 --2- 172.21.15.46:0/4156966031 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f73e4103330 0x7f73e4074080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:09:05.015 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:05.011+0000 7f73e259c640 1 --2- 172.21.15.46:0/4156966031 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f73e4101780 0x7f73e4075460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:09:05.015 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:05.011+0000 7f73d9d9b640 1 -- 172.21.15.46:0/4156966031 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f73e4101780 msgr2=0x7f73e4075460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:09:05.015 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:05.011+0000 7f73d9d9b640 1 --2- 172.21.15.46:0/4156966031 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f73e4101780 0x7f73e4075460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:05.015 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:05.011+0000 7f73d9d9b640 1 -- 172.21.15.46:0/4156966031 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f73e4103330 msgr2=0x7f73e4074080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:09:05.015 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:05.011+0000 7f73d9d9b640 1 --2- 172.21.15.46:0/4156966031 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f73e4103330 0x7f73e4074080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:05.015 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:05.011+0000 7f73d9d9b640 1 -- 172.21.15.46:0/4156966031 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f73e4074880 con 0x7f73e4102970 2024-03-31T23:09:05.015 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:05.011+0000 7f73d9d9b640 1 --2- 172.21.15.46:0/4156966031 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f73e4102970 0x7f73e4073b40 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f73cc036040 tx=0x7f73cc0042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:09:05.015 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:05.011+0000 7f73dbfff640 1 -- 172.21.15.46:0/4156966031 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73cc043020 con 0x7f73e4102970 2024-03-31T23:09:05.015 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:05.011+0000 7f73dbfff640 1 -- 172.21.15.46:0/4156966031 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f73cc0054c0 con 0x7f73e4102970 2024-03-31T23:09:05.015 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:05.011+0000 7f73e8f25640 1 -- 172.21.15.46:0/4156966031 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f73e41a1c40 con 0x7f73e4102970 2024-03-31T23:09:05.015 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:05.011+0000 7f73dbfff640 1 -- 172.21.15.46:0/4156966031 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73cc003d70 con 0x7f73e4102970 2024-03-31T23:09:05.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:05.011+0000 7f73e8f25640 1 -- 172.21.15.46:0/4156966031 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f73e41a2160 con 0x7f73e4102970 2024-03-31T23:09:05.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:05.012+0000 7f73e8f25640 1 -- 172.21.15.46:0/4156966031 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f73e4108bc0 con 0x7f73e4102970 2024-03-31T23:09:05.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:05.013+0000 7f73dbfff640 1 -- 172.21.15.46:0/4156966031 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 16) ==== 137320+0+0 (secure 0 0 0) 0x7f73cc046020 con 0x7f73e4102970 2024-03-31T23:09:05.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:05.014+0000 7f73dbfff640 1 --2- 172.21.15.46:0/4156966031 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f73d00a24a0 0x7f73d00a4960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:09:05.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:05.014+0000 7f73e259c640 1 --2- 172.21.15.46:0/4156966031 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f73d00a24a0 0x7f73d00a4960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:09:05.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:05.014+0000 7f73dbfff640 1 -- 172.21.15.46:0/4156966031 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f73cc0f0730 con 0x7f73e4102970 2024-03-31T23:09:05.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:05.014+0000 7f73e259c640 1 --2- 172.21.15.46:0/4156966031 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f73d00a24a0 0x7f73d00a4960 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f73e41027d0 tx=0x7f73d4008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:09:05.019 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:05.018+0000 7f73dbfff640 1 -- 172.21.15.46:0/4156966031 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f73cc04d020 con 0x7f73e4102970 2024-03-31T23:09:05.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:05.241+0000 7f73e8f25640 1 -- 172.21.15.46:0/4156966031 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f73e402d3e0 con 0x7f73e4102970 2024-03-31T23:09:05.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:05.241+0000 7f73dbfff640 1 -- 172.21.15.46:0/4156966031 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) ==== 94+0+9662 (secure 0 0 0) 0x7f73cc0bdd90 con 0x7f73e4102970 2024-03-31T23:09:05.243 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-31T23:09:05.243 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":12,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/2169018547","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":2169018547},{"type":"v1","addr":"172.21.15.46:6835","nonce":2169018547}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6837/1069387756","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6836","nonce":1069387756},{"type":"v1","addr":"172.21.15.46:6837","nonce":1069387756}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14634,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6839/3915337844","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6838","nonce":3915337844},{"type":"v1","addr":"172.21.15.46:6839","nonce":3915337844}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14646,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6841/897767145","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6840","nonce":897767145},{"type":"v1","addr":"172.21.15.46:6841","nonce":897767145}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24415,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6833/1746950966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6832","nonce":1746950966},{"type":"v1","addr":"172.21.15.162:6833","nonce":1746950966}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24422,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6833/3394739181","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6832","nonce":3394739181},{"type":"v1","addr":"172.21.15.142:6833","nonce":3394739181}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24442,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6835/1957208984","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6834","nonce":1957208984},{"type":"v1","addr":"172.21.15.162:6835","nonce":1957208984}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24451,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6837/1371886876","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6836","nonce":1371886876},{"type":"v1","addr":"172.21.15.162:6837","nonce":1371886876}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24455,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6835/212637431","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6834","nonce":212637431},{"type":"v1","addr":"172.21.15.142:6835","nonce":212637431}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24478,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6839/1976275826","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6838","nonce":1976275826},{"type":"v1","addr":"172.21.15.162:6839","nonce":1976275826}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24482,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6837/2559258399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6836","nonce":2559258399},{"type":"v1","addr":"172.21.15.142:6837","nonce":2559258399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9}],"filesystems":[]} 2024-03-31T23:09:05.243 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 12 2024-03-31T23:09:05.245 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:05.244+0000 7f73e8f25640 1 -- 172.21.15.46:0/4156966031 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f73d00a24a0 msgr2=0x7f73d00a4960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:09:05.245 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:05.244+0000 7f73e8f25640 1 --2- 172.21.15.46:0/4156966031 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f73d00a24a0 0x7f73d00a4960 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f73e41027d0 tx=0x7f73d4008040 comp rx=0 tx=0).stop 2024-03-31T23:09:05.245 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:05.244+0000 7f73e8f25640 1 -- 172.21.15.46:0/4156966031 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f73e4102970 msgr2=0x7f73e4073b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:09:05.245 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:05.244+0000 7f73e8f25640 1 --2- 172.21.15.46:0/4156966031 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f73e4102970 0x7f73e4073b40 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f73cc036040 tx=0x7f73cc0042f0 comp rx=0 tx=0).stop 2024-03-31T23:09:05.245 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:05.244+0000 7f73e8f25640 1 -- 172.21.15.46:0/4156966031 shutdown_connections 2024-03-31T23:09:05.245 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:05.244+0000 7f73e8f25640 1 --2- 172.21.15.46:0/4156966031 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f73d00a24a0 0x7f73d00a4960 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:05.245 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:05.244+0000 7f73e8f25640 1 --2- 172.21.15.46:0/4156966031 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f73e4103330 0x7f73e4074080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:05.245 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:05.244+0000 7f73e8f25640 1 --2- 172.21.15.46:0/4156966031 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f73e4102970 0x7f73e4073b40 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:05.245 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:05.244+0000 7f73e8f25640 1 --2- 172.21.15.46:0/4156966031 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f73e4101780 0x7f73e4075460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:05.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:05.244+0000 7f73e8f25640 1 -- 172.21.15.46:0/4156966031 >> 172.21.15.46:0/4156966031 conn(0x7f73e40fcd30 msgr2=0x7f73e4105d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:09:05.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:05.244+0000 7f73e8f25640 1 -- 172.21.15.46:0/4156966031 shutdown_connections 2024-03-31T23:09:05.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:05.244+0000 7f73e8f25640 1 -- 172.21.15.46:0/4156966031 wait complete. 2024-03-31T23:09:05.612 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:09:05 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-31T23:09:05.612 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:09:05 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-31T23:09:05.612 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:09:05 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:09:05.612 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:09:05 smithi046 ceph-mon[32498]: Deploying daemon mds.b on smithi142 2024-03-31T23:09:05.612 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:09:05 smithi046 ceph-mon[32498]: mds.? [v2:172.21.15.46:6840/897767145,v1:172.21.15.46:6841/897767145] up:boot 2024-03-31T23:09:05.612 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:09:05 smithi046 ceph-mon[32498]: fsmap 11 up:standby 2024-03-31T23:09:05.612 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:09:05 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-31T23:09:05.612 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:09:05 smithi046 ceph-mon[32498]: from='client.? 172.21.15.46:0/4156966031' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-31T23:09:05.693 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:09:05 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-31T23:09:05.693 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:09:05 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-31T23:09:05.694 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:09:05 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:09:05.694 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:09:05 smithi142 ceph-mon[35271]: Deploying daemon mds.b on smithi142 2024-03-31T23:09:05.694 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:09:05 smithi142 ceph-mon[35271]: mds.? [v2:172.21.15.46:6840/897767145,v1:172.21.15.46:6841/897767145] up:boot 2024-03-31T23:09:05.694 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:09:05 smithi142 ceph-mon[35271]: fsmap 11 up:standby 2024-03-31T23:09:05.694 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:09:05 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-31T23:09:05.694 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:09:05 smithi142 ceph-mon[35271]: from='client.? 172.21.15.46:0/4156966031' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-31T23:09:05.795 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:09:05 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-31T23:09:05.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:09:05 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-31T23:09:05.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:09:05 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:09:05.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:09:05 smithi162 ceph-mon[36237]: Deploying daemon mds.b on smithi142 2024-03-31T23:09:05.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:09:05 smithi162 ceph-mon[36237]: mds.? [v2:172.21.15.46:6840/897767145,v1:172.21.15.46:6841/897767145] up:boot 2024-03-31T23:09:05.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:09:05 smithi162 ceph-mon[36237]: fsmap 11 up:standby 2024-03-31T23:09:05.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:09:05 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-31T23:09:05.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:09:05 smithi162 ceph-mon[36237]: from='client.? 172.21.15.46:0/4156966031' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-31T23:09:06.693 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:09:06 smithi142 ceph-mon[35271]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-31T23:09:06.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:09:06 smithi162 ceph-mon[36237]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-31T23:09:06.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:09:06 smithi046 ceph-mon[32498]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-31T23:09:06.999 INFO:journalctl@ceph.mds.b.smithi142.stdout:Mar 31 23:09:06 smithi142 podman[65787]: 2024-03-31 23:09:06.868899412 +0000 UTC m=+0.018299661 image pull ec18d7baa60e8b1a0b126fbe5306acce11b1ed0ec5e3bcdddbf39b0a3c591791 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175 2024-03-31T23:09:06.999 INFO:journalctl@ceph.mds.b.smithi142.stdout:Mar 31 23:09:06 smithi142 podman[65787]: 2024-03-31 23:09:06.998470826 +0000 UTC m=+0.147871098 container create 162eacae90f31163644e60f2d674f1568d00a26f04ace8df61f9c48322607d7c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mds-b, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240326, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2, RELEASE=squid-384f31f, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2024-03-31T23:09:07.257 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-31T23:09:07.260 INFO:journalctl@ceph.mds.b.smithi142.stdout:Mar 31 23:09:07 smithi142 podman[65787]: 2024-03-31 23:09:07.234388614 +0000 UTC m=+0.383788852 container init 162eacae90f31163644e60f2d674f1568d00a26f04ace8df61f9c48322607d7c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mds-b, GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, RELEASE=squid-384f31f, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240326, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux ) 2024-03-31T23:09:07.260 INFO:journalctl@ceph.mds.b.smithi142.stdout:Mar 31 23:09:07 smithi142 podman[65787]: 2024-03-31 23:09:07.237100114 +0000 UTC m=+0.386500342 container start 162eacae90f31163644e60f2d674f1568d00a26f04ace8df61f9c48322607d7c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mds-b, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=squid-384f31f, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240326, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0) 2024-03-31T23:09:07.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:07.372+0000 7fcd4b6ed640 1 -- 172.21.15.46:0/2936325774 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcd44100000 msgr2=0x7fcd44100420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:09:07.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:07.372+0000 7fcd4b6ed640 1 --2- 172.21.15.46:0/2936325774 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcd44100000 0x7fcd44100420 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fcd40009900 tx=0x7fcd4002bc30 comp rx=0 tx=0).stop 2024-03-31T23:09:07.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:07.372+0000 7fcd4b6ed640 1 -- 172.21.15.46:0/2936325774 shutdown_connections 2024-03-31T23:09:07.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:07.372+0000 7fcd4b6ed640 1 --2- 172.21.15.46:0/2936325774 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcd44101300 0x7fcd44101780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:07.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:07.372+0000 7fcd4b6ed640 1 --2- 172.21.15.46:0/2936325774 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcd44100960 0x7fcd44100dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:07.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:07.372+0000 7fcd4b6ed640 1 --2- 172.21.15.46:0/2936325774 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcd44100000 0x7fcd44100420 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:07.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:07.372+0000 7fcd4b6ed640 1 -- 172.21.15.46:0/2936325774 >> 172.21.15.46:0/2936325774 conn(0x7fcd440fbc00 msgr2=0x7fcd440fe040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:09:07.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:07.372+0000 7fcd4b6ed640 1 -- 172.21.15.46:0/2936325774 shutdown_connections 2024-03-31T23:09:07.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:07.372+0000 7fcd4b6ed640 1 -- 172.21.15.46:0/2936325774 wait complete. 2024-03-31T23:09:07.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:07.373+0000 7fcd4b6ed640 1 Processor -- start 2024-03-31T23:09:07.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:07.373+0000 7fcd4b6ed640 1 -- start start 2024-03-31T23:09:07.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:07.373+0000 7fcd4b6ed640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcd44100000 0x7fcd44199ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:09:07.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:07.373+0000 7fcd4b6ed640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcd44100960 0x7fcd4419a0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:09:07.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:07.373+0000 7fcd4b6ed640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcd44101300 0x7fcd441a1150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:09:07.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:07.373+0000 7fcd4b6ed640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fcd4410f4a0 con 0x7fcd44100000 2024-03-31T23:09:07.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:07.373+0000 7fcd4b6ed640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fcd4410f320 con 0x7fcd44101300 2024-03-31T23:09:07.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:07.373+0000 7fcd4b6ed640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fcd4410f620 con 0x7fcd44100960 2024-03-31T23:09:07.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:07.373+0000 7fcd49489640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcd44100000 0x7fcd44199ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:09:07.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:07.373+0000 7fcd49489640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcd44100000 0x7fcd44199ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:48998/0 (socket says 172.21.15.46:48998) 2024-03-31T23:09:07.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:07.373+0000 7fcd49489640 1 -- 172.21.15.46:0/3789494717 learned_addr learned my addr 172.21.15.46:0/3789494717 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:09:07.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:07.373+0000 7fcd48c88640 1 --2- 172.21.15.46:0/3789494717 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcd44100960 0x7fcd4419a0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:09:07.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:07.373+0000 7fcd49c8a640 1 --2- 172.21.15.46:0/3789494717 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcd44101300 0x7fcd441a1150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:09:07.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:07.373+0000 7fcd49489640 1 -- 172.21.15.46:0/3789494717 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcd44100960 msgr2=0x7fcd4419a0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:09:07.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:07.373+0000 7fcd49489640 1 --2- 172.21.15.46:0/3789494717 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcd44100960 0x7fcd4419a0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:07.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:07.373+0000 7fcd49489640 1 -- 172.21.15.46:0/3789494717 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcd44101300 msgr2=0x7fcd441a1150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:09:07.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:07.373+0000 7fcd49489640 1 --2- 172.21.15.46:0/3789494717 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcd44101300 0x7fcd441a1150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:07.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:07.373+0000 7fcd49489640 1 -- 172.21.15.46:0/3789494717 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcd441a17c0 con 0x7fcd44100000 2024-03-31T23:09:07.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:07.373+0000 7fcd49c8a640 1 --2- 172.21.15.46:0/3789494717 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcd44101300 0x7fcd441a1150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T23:09:07.379 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:07.374+0000 7fcd49489640 1 --2- 172.21.15.46:0/3789494717 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcd44100000 0x7fcd44199ba0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fcd40006010 tx=0x7fcd40004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:09:07.379 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:07.374+0000 7fcd3a7fc640 1 -- 172.21.15.46:0/3789494717 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd40043020 con 0x7fcd44100000 2024-03-31T23:09:07.379 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:07.374+0000 7fcd4b6ed640 1 -- 172.21.15.46:0/3789494717 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcd441a1a50 con 0x7fcd44100000 2024-03-31T23:09:07.379 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:07.374+0000 7fcd3a7fc640 1 -- 172.21.15.46:0/3789494717 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fcd40004440 con 0x7fcd44100000 2024-03-31T23:09:07.379 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:07.374+0000 7fcd3a7fc640 1 -- 172.21.15.46:0/3789494717 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd40003bf0 con 0x7fcd44100000 2024-03-31T23:09:07.379 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:07.374+0000 7fcd4b6ed640 1 -- 172.21.15.46:0/3789494717 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcd441a1fa0 con 0x7fcd44100000 2024-03-31T23:09:07.379 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:07.375+0000 7fcd4b6ed640 1 -- 172.21.15.46:0/3789494717 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcd4410af80 con 0x7fcd44100000 2024-03-31T23:09:07.380 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:07.376+0000 7fcd3a7fc640 1 -- 172.21.15.46:0/3789494717 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 16) ==== 137320+0+0 (secure 0 0 0) 0x7fcd40046020 con 0x7fcd44100000 2024-03-31T23:09:07.380 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:07.377+0000 7fcd3a7fc640 1 --2- 172.21.15.46:0/3789494717 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fcd300a24a0 0x7fcd300a4960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:09:07.380 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:07.377+0000 7fcd48c88640 1 --2- 172.21.15.46:0/3789494717 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fcd300a24a0 0x7fcd300a4960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:09:07.380 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:07.377+0000 7fcd3a7fc640 1 -- 172.21.15.46:0/3789494717 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7fcd400f0680 con 0x7fcd44100000 2024-03-31T23:09:07.380 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:07.377+0000 7fcd48c88640 1 --2- 172.21.15.46:0/3789494717 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fcd300a24a0 0x7fcd300a4960 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fcd4419b0b0 tx=0x7fcd3400b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:09:07.382 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:07.381+0000 7fcd3a7fc640 1 -- 172.21.15.46:0/3789494717 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fcd4004d020 con 0x7fcd44100000 2024-03-31T23:09:07.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:07.613+0000 7fcd4b6ed640 1 -- 172.21.15.46:0/3789494717 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fcd4402d3c0 con 0x7fcd44100000 2024-03-31T23:09:07.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:07.614+0000 7fcd3a7fc640 1 -- 172.21.15.46:0/3789494717 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) ==== 94+0+10491 (secure 0 0 0) 0x7fcd400bdce0 con 0x7fcd44100000 2024-03-31T23:09:07.615 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-31T23:09:07.615 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":13,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/2169018547","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":2169018547},{"type":"v1","addr":"172.21.15.46:6835","nonce":2169018547}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6837/1069387756","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6836","nonce":1069387756},{"type":"v1","addr":"172.21.15.46:6837","nonce":1069387756}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14634,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6839/3915337844","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6838","nonce":3915337844},{"type":"v1","addr":"172.21.15.46:6839","nonce":3915337844}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14646,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6841/897767145","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6840","nonce":897767145},{"type":"v1","addr":"172.21.15.46:6841","nonce":897767145}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24415,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6833/1746950966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6832","nonce":1746950966},{"type":"v1","addr":"172.21.15.162:6833","nonce":1746950966}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24422,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6833/3394739181","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6832","nonce":3394739181},{"type":"v1","addr":"172.21.15.142:6833","nonce":3394739181}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24442,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6835/1957208984","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6834","nonce":1957208984},{"type":"v1","addr":"172.21.15.162:6835","nonce":1957208984}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24451,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6837/1371886876","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6836","nonce":1371886876},{"type":"v1","addr":"172.21.15.162:6837","nonce":1371886876}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24455,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6835/212637431","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6834","nonce":212637431},{"type":"v1","addr":"172.21.15.142:6835","nonce":212637431}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24478,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6839/1976275826","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6838","nonce":1976275826},{"type":"v1","addr":"172.21.15.162:6839","nonce":1976275826}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24482,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6837/2559258399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6836","nonce":2559258399},{"type":"v1","addr":"172.21.15.142:6837","nonce":2559258399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24515,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6839/2519143918","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6838","nonce":2519143918},{"type":"v1","addr":"172.21.15.142:6839","nonce":2519143918}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13}],"filesystems":[]} 2024-03-31T23:09:07.616 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 13 2024-03-31T23:09:07.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:07.616+0000 7fcd4b6ed640 1 -- 172.21.15.46:0/3789494717 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fcd300a24a0 msgr2=0x7fcd300a4960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:09:07.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:07.616+0000 7fcd4b6ed640 1 --2- 172.21.15.46:0/3789494717 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fcd300a24a0 0x7fcd300a4960 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fcd4419b0b0 tx=0x7fcd3400b390 comp rx=0 tx=0).stop 2024-03-31T23:09:07.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:07.616+0000 7fcd4b6ed640 1 -- 172.21.15.46:0/3789494717 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcd44100000 msgr2=0x7fcd44199ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:09:07.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:07.616+0000 7fcd4b6ed640 1 --2- 172.21.15.46:0/3789494717 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcd44100000 0x7fcd44199ba0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fcd40006010 tx=0x7fcd40004040 comp rx=0 tx=0).stop 2024-03-31T23:09:07.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:07.616+0000 7fcd4b6ed640 1 -- 172.21.15.46:0/3789494717 shutdown_connections 2024-03-31T23:09:07.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:07.616+0000 7fcd4b6ed640 1 --2- 172.21.15.46:0/3789494717 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fcd300a24a0 0x7fcd300a4960 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:07.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:07.616+0000 7fcd4b6ed640 1 --2- 172.21.15.46:0/3789494717 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcd44101300 0x7fcd441a1150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:07.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:07.616+0000 7fcd4b6ed640 1 --2- 172.21.15.46:0/3789494717 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcd44100960 0x7fcd4419a0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:07.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:07.616+0000 7fcd4b6ed640 1 --2- 172.21.15.46:0/3789494717 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcd44100000 0x7fcd44199ba0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:07.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:07.616+0000 7fcd4b6ed640 1 -- 172.21.15.46:0/3789494717 >> 172.21.15.46:0/3789494717 conn(0x7fcd440fbc00 msgr2=0x7fcd440fd3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:09:07.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:07.616+0000 7fcd4b6ed640 1 -- 172.21.15.46:0/3789494717 shutdown_connections 2024-03-31T23:09:07.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:07.616+0000 7fcd4b6ed640 1 -- 172.21.15.46:0/3789494717 wait complete. 2024-03-31T23:09:07.630 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-31T23:09:07.693 INFO:journalctl@ceph.mds.b.smithi142.stdout:Mar 31 23:09:07 smithi142 ceph-mds[65805]: -- 172.21.15.142:0/2045959502 <== mon.2 v2:172.21.15.142:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5624d0185a40 con 0x5624d0158c00 2024-03-31T23:09:07.693 INFO:journalctl@ceph.mds.b.smithi142.stdout:Mar 31 23:09:07 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mds-b[65801]: starting mds.b at 2024-03-31T23:09:07.693 INFO:journalctl@ceph.mds.b.smithi142.stdout:Mar 31 23:09:07 smithi142 bash[65787]: 162eacae90f31163644e60f2d674f1568d00a26f04ace8df61f9c48322607d7c 2024-03-31T23:09:07.693 INFO:journalctl@ceph.mds.b.smithi142.stdout:Mar 31 23:09:07 smithi142 systemd[1]: Started Ceph mds.b for 6a9d8f08-efb2-11ee-b647-cb9ed24678a4. 2024-03-31T23:09:07.736 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:07.733+0000 7f1fb87e2640 1 -- 172.21.15.46:0/1430154836 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1fb01018e0 msgr2=0x7f1fb0101d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:09:07.736 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:07.733+0000 7f1fb87e2640 1 --2- 172.21.15.46:0/1430154836 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1fb01018e0 0x7f1fb0101d60 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f1fa0009900 tx=0x7f1fa002bc30 comp rx=0 tx=0).stop 2024-03-31T23:09:07.736 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:07.733+0000 7f1fb87e2640 1 -- 172.21.15.46:0/1430154836 shutdown_connections 2024-03-31T23:09:07.737 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:07.733+0000 7f1fb87e2640 1 --2- 172.21.15.46:0/1430154836 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1fb01022a0 0x7f1fb0108b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:07.737 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:07.733+0000 7f1fb87e2640 1 --2- 172.21.15.46:0/1430154836 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1fb01018e0 0x7f1fb0101d60 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:07.737 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:07.733+0000 7f1fb87e2640 1 --2- 172.21.15.46:0/1430154836 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1fb01006f0 0x7f1fb0100b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:07.738 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:07.733+0000 7f1fb87e2640 1 -- 172.21.15.46:0/1430154836 >> 172.21.15.46:0/1430154836 conn(0x7f1fb00fbc80 msgr2=0x7f1fb00fe0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:09:07.738 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:07.733+0000 7f1fb87e2640 1 -- 172.21.15.46:0/1430154836 shutdown_connections 2024-03-31T23:09:07.738 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:07.733+0000 7f1fb87e2640 1 -- 172.21.15.46:0/1430154836 wait complete. 2024-03-31T23:09:07.738 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:07.734+0000 7f1fb87e2640 1 Processor -- start 2024-03-31T23:09:07.738 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:07.734+0000 7f1fb87e2640 1 -- start start 2024-03-31T23:09:07.738 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:07.734+0000 7f1fb87e2640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1fb01006f0 0x7f1fb0104680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:09:07.738 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:07.734+0000 7f1fb87e2640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1fb01018e0 0x7f1fb0102cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:09:07.738 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:07.734+0000 7f1fb87e2640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1fb01022a0 0x7f1fb0103200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:09:07.738 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:07.734+0000 7f1fb87e2640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f1fb010c190 con 0x7f1fb01006f0 2024-03-31T23:09:07.738 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:07.734+0000 7f1fb87e2640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f1fb010c010 con 0x7f1fb01018e0 2024-03-31T23:09:07.738 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:07.734+0000 7f1fb87e2640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f1fb010c310 con 0x7f1fb01022a0 2024-03-31T23:09:07.738 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:07.734+0000 7f1fb657e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1fb01006f0 0x7f1fb0104680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:09:07.738 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:07.734+0000 7f1fb657e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1fb01006f0 0x7f1fb0104680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:49018/0 (socket says 172.21.15.46:49018) 2024-03-31T23:09:07.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:07.734+0000 7f1fb657e640 1 -- 172.21.15.46:0/286376108 learned_addr learned my addr 172.21.15.46:0/286376108 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:09:07.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:07.734+0000 7f1fb5d7d640 1 --2- 172.21.15.46:0/286376108 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1fb01018e0 0x7f1fb0102cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:09:07.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:07.734+0000 7f1fb6d7f640 1 --2- 172.21.15.46:0/286376108 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1fb01022a0 0x7f1fb0103200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:09:07.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:07.735+0000 7f1fb657e640 1 -- 172.21.15.46:0/286376108 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1fb01022a0 msgr2=0x7f1fb0103200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:09:07.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:07.735+0000 7f1fb657e640 1 --2- 172.21.15.46:0/286376108 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1fb01022a0 0x7f1fb0103200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:07.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:07.735+0000 7f1fb657e640 1 -- 172.21.15.46:0/286376108 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1fb01018e0 msgr2=0x7f1fb0102cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:09:07.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:07.735+0000 7f1fb657e640 1 --2- 172.21.15.46:0/286376108 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1fb01018e0 0x7f1fb0102cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:07.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:07.735+0000 7f1fb657e640 1 -- 172.21.15.46:0/286376108 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1fb0103a90 con 0x7f1fb01006f0 2024-03-31T23:09:07.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:07.735+0000 7f1fb6d7f640 1 --2- 172.21.15.46:0/286376108 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1fb01022a0 0x7f1fb0103200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T23:09:07.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:07.735+0000 7f1fb657e640 1 --2- 172.21.15.46:0/286376108 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1fb01006f0 0x7f1fb0104680 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f1fac00d770 tx=0x7f1fac00dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:09:07.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:07.735+0000 7f1fa77fe640 1 -- 172.21.15.46:0/286376108 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fac017070 con 0x7f1fb01006f0 2024-03-31T23:09:07.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:07.735+0000 7f1fa77fe640 1 -- 172.21.15.46:0/286376108 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1fac009c30 con 0x7f1fb01006f0 2024-03-31T23:09:07.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:07.735+0000 7f1fb87e2640 1 -- 172.21.15.46:0/286376108 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1fb01a1ae0 con 0x7f1fb01006f0 2024-03-31T23:09:07.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:07.735+0000 7f1fa77fe640 1 -- 172.21.15.46:0/286376108 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fac003d00 con 0x7f1fb01006f0 2024-03-31T23:09:07.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:07.735+0000 7f1fb87e2640 1 -- 172.21.15.46:0/286376108 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1fb01a2000 con 0x7f1fb01006f0 2024-03-31T23:09:07.741 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:07.736+0000 7f1fb87e2640 1 -- 172.21.15.46:0/286376108 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1fb01a21d0 con 0x7f1fb01006f0 2024-03-31T23:09:07.741 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:07.737+0000 7f1fa77fe640 1 -- 172.21.15.46:0/286376108 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 16) ==== 137320+0+0 (secure 0 0 0) 0x7f1fac003ea0 con 0x7f1fb01006f0 2024-03-31T23:09:07.741 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:07.738+0000 7f1fa77fe640 1 --2- 172.21.15.46:0/286376108 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f1f9c0a24a0 0x7f1f9c0a4960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:09:07.741 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:07.738+0000 7f1fa77fe640 1 -- 172.21.15.46:0/286376108 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f1fac0cdfc0 con 0x7f1fb01006f0 2024-03-31T23:09:07.741 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:07.738+0000 7f1fb5d7d640 1 --2- 172.21.15.46:0/286376108 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f1f9c0a24a0 0x7f1f9c0a4960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:09:07.741 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:07.738+0000 7f1fb5d7d640 1 --2- 172.21.15.46:0/286376108 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f1f9c0a24a0 0x7f1f9c0a4960 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f1fa0009230 tx=0x7f1fa0005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:09:07.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:07.742+0000 7f1fa77fe640 1 -- 172.21.15.46:0/286376108 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f1fac09b620 con 0x7f1fb01006f0 2024-03-31T23:09:07.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:07.928+0000 7f1fb87e2640 1 -- 172.21.15.46:0/286376108 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f1fb01a2560 con 0x7f1fb01006f0 2024-03-31T23:09:07.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:07.929+0000 7f1fa77fe640 1 -- 172.21.15.46:0/286376108 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v82) ==== 74+0+18433 (secure 0 0 0) 0x7f1fac009de0 con 0x7f1fb01006f0 2024-03-31T23:09:07.931 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-31T23:09:07.931 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":82,"fsid":"6a9d8f08-efb2-11ee-b647-cb9ed24678a4","created":"2024-03-31T23:01:15.463711+0000","modified":"2024-03-31T23:08:21.201474+0000","last_up_change":"2024-03-31T23:08:20.199888+0000","last_in_change":"2024-03-31T23:07:59.438035+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-31T23:04:32.823751+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"47429095-3f46-4ed7-b7c4-504464719815","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":1824772695},{"type":"v1","addr":"172.21.15.46:6803","nonce":1824772695}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":1824772695},{"type":"v1","addr":"172.21.15.46:6805","nonce":1824772695}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":1824772695},{"type":"v1","addr":"172.21.15.46:6809","nonce":1824772695}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":1824772695},{"type":"v1","addr":"172.21.15.46:6807","nonce":1824772695}]},"public_addr":"172.21.15.46:6803/1824772695","cluster_addr":"172.21.15.46:6805/1824772695","heartbeat_back_addr":"172.21.15.46:6809/1824772695","heartbeat_front_addr":"172.21.15.46:6807/1824772695","state":["exists","up"]},{"osd":1,"uuid":"be56f883-411e-4dbe-b50e-a158134df4a6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6800","nonce":973693922},{"type":"v1","addr":"172.21.15.142:6801","nonce":973693922}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6802","nonce":973693922},{"type":"v1","addr":"172.21.15.142:6803","nonce":973693922}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6806","nonce":973693922},{"type":"v1","addr":"172.21.15.142:6807","nonce":973693922}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6804","nonce":973693922},{"type":"v1","addr":"172.21.15.142:6805","nonce":973693922}]},"public_addr":"172.21.15.142:6801/973693922","cluster_addr":"172.21.15.142:6803/973693922","heartbeat_back_addr":"172.21.15.142:6807/973693922","heartbeat_front_addr":"172.21.15.142:6805/973693922","state":["exists","up"]},{"osd":2,"uuid":"c99a8794-ab19-4260-9a8d-4bc0bd2c8175","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6800","nonce":4084261813},{"type":"v1","addr":"172.21.15.162:6801","nonce":4084261813}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6802","nonce":4084261813},{"type":"v1","addr":"172.21.15.162:6803","nonce":4084261813}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6806","nonce":4084261813},{"type":"v1","addr":"172.21.15.162:6807","nonce":4084261813}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6804","nonce":4084261813},{"type":"v1","addr":"172.21.15.162:6805","nonce":4084261813}]},"public_addr":"172.21.15.162:6801/4084261813","cluster_addr":"172.21.15.162:6803/4084261813","heartbeat_back_addr":"172.21.15.162:6807/4084261813","heartbeat_front_addr":"172.21.15.162:6805/4084261813","state":["exists","up"]},{"osd":3,"uuid":"f877814e-9ef6-4872-a081-5609bb9ebc06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":3496317970},{"type":"v1","addr":"172.21.15.46:6811","nonce":3496317970}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":3496317970},{"type":"v1","addr":"172.21.15.46:6813","nonce":3496317970}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":3496317970},{"type":"v1","addr":"172.21.15.46:6817","nonce":3496317970}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":3496317970},{"type":"v1","addr":"172.21.15.46:6815","nonce":3496317970}]},"public_addr":"172.21.15.46:6811/3496317970","cluster_addr":"172.21.15.46:6813/3496317970","heartbeat_back_addr":"172.21.15.46:6817/3496317970","heartbeat_front_addr":"172.21.15.46:6815/3496317970","state":["exists","up"]},{"osd":4,"uuid":"beecec0a-8280-44fe-8f5f-8498a35eca8f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6808","nonce":2996266929},{"type":"v1","addr":"172.21.15.142:6809","nonce":2996266929}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6810","nonce":2996266929},{"type":"v1","addr":"172.21.15.142:6811","nonce":2996266929}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6814","nonce":2996266929},{"type":"v1","addr":"172.21.15.142:6815","nonce":2996266929}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6812","nonce":2996266929},{"type":"v1","addr":"172.21.15.142:6813","nonce":2996266929}]},"public_addr":"172.21.15.142:6809/2996266929","cluster_addr":"172.21.15.142:6811/2996266929","heartbeat_back_addr":"172.21.15.142:6815/2996266929","heartbeat_front_addr":"172.21.15.142:6813/2996266929","state":["exists","up"]},{"osd":5,"uuid":"9da7ff75-f658-45ea-a02f-1943fdda76b9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6808","nonce":914208293},{"type":"v1","addr":"172.21.15.162:6809","nonce":914208293}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6810","nonce":914208293},{"type":"v1","addr":"172.21.15.162:6811","nonce":914208293}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6814","nonce":914208293},{"type":"v1","addr":"172.21.15.162:6815","nonce":914208293}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6812","nonce":914208293},{"type":"v1","addr":"172.21.15.162:6813","nonce":914208293}]},"public_addr":"172.21.15.162:6809/914208293","cluster_addr":"172.21.15.162:6811/914208293","heartbeat_back_addr":"172.21.15.162:6815/914208293","heartbeat_front_addr":"172.21.15.162:6813/914208293","state":["exists","up"]},{"osd":6,"uuid":"8fa11224-cb26-4d3f-87d1-6a8f31de2a19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":56291966},{"type":"v1","addr":"172.21.15.46:6819","nonce":56291966}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":56291966},{"type":"v1","addr":"172.21.15.46:6821","nonce":56291966}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6824","nonce":56291966},{"type":"v1","addr":"172.21.15.46:6825","nonce":56291966}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":56291966},{"type":"v1","addr":"172.21.15.46:6823","nonce":56291966}]},"public_addr":"172.21.15.46:6819/56291966","cluster_addr":"172.21.15.46:6821/56291966","heartbeat_back_addr":"172.21.15.46:6825/56291966","heartbeat_front_addr":"172.21.15.46:6823/56291966","state":["exists","up"]},{"osd":7,"uuid":"0d0338b7-4d0a-4bb0-b152-8f5224871d9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6816","nonce":2146598446},{"type":"v1","addr":"172.21.15.142:6817","nonce":2146598446}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6818","nonce":2146598446},{"type":"v1","addr":"172.21.15.142:6819","nonce":2146598446}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6822","nonce":2146598446},{"type":"v1","addr":"172.21.15.142:6823","nonce":2146598446}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6820","nonce":2146598446},{"type":"v1","addr":"172.21.15.142:6821","nonce":2146598446}]},"public_addr":"172.21.15.142:6817/2146598446","cluster_addr":"172.21.15.142:6819/2146598446","heartbeat_back_addr":"172.21.15.142:6823/2146598446","heartbeat_front_addr":"172.21.15.142:6821/2146598446","state":["exists","up"]},{"osd":8,"uuid":"8b6f6037-892a-416c-babb-7552648d0b3a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6816","nonce":1173078318},{"type":"v1","addr":"172.21.15.162:6817","nonce":1173078318}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6818","nonce":1173078318},{"type":"v1","addr":"172.21.15.162:6819","nonce":1173078318}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6822","nonce":1173078318},{"type":"v1","addr":"172.21.15.162:6823","nonce":1173078318}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6820","nonce":1173078318},{"type":"v1","addr":"172.21.15.162:6821","nonce":1173078318}]},"public_addr":"172.21.15.162:6817/1173078318","cluster_addr":"172.21.15.162:6819/1173078318","heartbeat_back_addr":"172.21.15.162:6823/1173078318","heartbeat_front_addr":"172.21.15.162:6821/1173078318","state":["exists","up"]},{"osd":9,"uuid":"faccce25-6bf2-4220-8858-cda82b72be68","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":532708881},{"type":"v1","addr":"172.21.15.46:6827","nonce":532708881}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6828","nonce":532708881},{"type":"v1","addr":"172.21.15.46:6829","nonce":532708881}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":532708881},{"type":"v1","addr":"172.21.15.46:6833","nonce":532708881}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":532708881},{"type":"v1","addr":"172.21.15.46:6831","nonce":532708881}]},"public_addr":"172.21.15.46:6827/532708881","cluster_addr":"172.21.15.46:6829/532708881","heartbeat_back_addr":"172.21.15.46:6833/532708881","heartbeat_front_addr":"172.21.15.46:6831/532708881","state":["exists","up"]},{"osd":10,"uuid":"cdd356db-f9eb-4080-95ab-db5d8223e572","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6824","nonce":2845222366},{"type":"v1","addr":"172.21.15.142:6825","nonce":2845222366}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6826","nonce":2845222366},{"type":"v1","addr":"172.21.15.142:6827","nonce":2845222366}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6830","nonce":2845222366},{"type":"v1","addr":"172.21.15.142:6831","nonce":2845222366}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6828","nonce":2845222366},{"type":"v1","addr":"172.21.15.142:6829","nonce":2845222366}]},"public_addr":"172.21.15.142:6825/2845222366","cluster_addr":"172.21.15.142:6827/2845222366","heartbeat_back_addr":"172.21.15.142:6831/2845222366","heartbeat_front_addr":"172.21.15.142:6829/2845222366","state":["exists","up"]},{"osd":11,"uuid":"72104d1c-3ea2-45ad-bc0b-0500f146c06b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6824","nonce":1986323778},{"type":"v1","addr":"172.21.15.162:6825","nonce":1986323778}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6826","nonce":1986323778},{"type":"v1","addr":"172.21.15.162:6827","nonce":1986323778}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6830","nonce":1986323778},{"type":"v1","addr":"172.21.15.162:6831","nonce":1986323778}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6828","nonce":1986323778},{"type":"v1","addr":"172.21.15.162:6829","nonce":1986323778}]},"public_addr":"172.21.15.162:6825/1986323778","cluster_addr":"172.21.15.162:6827/1986323778","heartbeat_back_addr":"172.21.15.162:6831/1986323778","heartbeat_front_addr":"172.21.15.162:6829/1986323778","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:03:44.996331+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:04:07.994738+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:04:30.942255+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:04:55.588884+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:05:19.900600+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:05:43.807442+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:06:09.026976+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:06:34.586784+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:06:59.751605+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:07:26.144495+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:07:52.326909+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:08:17.423042+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.46:0/4021039857":"2024-04-01T23:02:08.493619+0000","172.21.15.46:6801/3681182647":"2024-04-01T23:02:08.493619+0000","172.21.15.46:0/4022214806":"2024-04-01T23:02:08.493619+0000","172.21.15.46:6800/1515528215":"2024-04-01T23:01:41.910162+0000","172.21.15.46:6800/3681182647":"2024-04-01T23:02:08.493619+0000","172.21.15.46:0/143885707":"2024-04-01T23:01:41.910162+0000","172.21.15.46:6801/1515528215":"2024-04-01T23:01:41.910162+0000","172.21.15.46:0/1680389778":"2024-04-01T23:01:41.910162+0000","172.21.15.46:0/1782133966":"2024-04-01T23:01:41.910162+0000","172.21.15.46:0/3925209376":"2024-04-01T23:02:08.493619+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-31T23:09:07.933 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:07.932+0000 7f1fb87e2640 1 -- 172.21.15.46:0/286376108 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f1f9c0a24a0 msgr2=0x7f1f9c0a4960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:09:07.933 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:07.932+0000 7f1fb87e2640 1 --2- 172.21.15.46:0/286376108 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f1f9c0a24a0 0x7f1f9c0a4960 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f1fa0009230 tx=0x7f1fa0005fd0 comp rx=0 tx=0).stop 2024-03-31T23:09:07.933 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:07.932+0000 7f1fb87e2640 1 -- 172.21.15.46:0/286376108 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1fb01006f0 msgr2=0x7f1fb0104680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:09:07.933 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:07.932+0000 7f1fb87e2640 1 --2- 172.21.15.46:0/286376108 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1fb01006f0 0x7f1fb0104680 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f1fac00d770 tx=0x7f1fac00dc30 comp rx=0 tx=0).stop 2024-03-31T23:09:07.933 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:07.932+0000 7f1fb87e2640 1 -- 172.21.15.46:0/286376108 shutdown_connections 2024-03-31T23:09:07.933 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:07.932+0000 7f1fb87e2640 1 --2- 172.21.15.46:0/286376108 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f1f9c0a24a0 0x7f1f9c0a4960 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:07.933 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:07.932+0000 7f1fb87e2640 1 --2- 172.21.15.46:0/286376108 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1fb01022a0 0x7f1fb0103200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:07.933 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:07.932+0000 7f1fb87e2640 1 --2- 172.21.15.46:0/286376108 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1fb01018e0 0x7f1fb0102cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:07.933 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:07.932+0000 7f1fb87e2640 1 --2- 172.21.15.46:0/286376108 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1fb01006f0 0x7f1fb0104680 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:07.933 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:07.932+0000 7f1fb87e2640 1 -- 172.21.15.46:0/286376108 >> 172.21.15.46:0/286376108 conn(0x7f1fb00fbc80 msgr2=0x7f1fb0069d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:09:07.934 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:07.932+0000 7f1fb87e2640 1 -- 172.21.15.46:0/286376108 shutdown_connections 2024-03-31T23:09:07.934 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:07.932+0000 7f1fb87e2640 1 -- 172.21.15.46:0/286376108 wait complete. 2024-03-31T23:09:07.944 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-31T23:04:32.823751+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-31T23:09:07.944 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-31T23:09:08.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:08.051+0000 7ff88c725640 1 -- 172.21.15.46:0/147661408 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff8841022d0 msgr2=0x7ff8841046c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:09:08.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:08.051+0000 7ff88c725640 1 --2- 172.21.15.46:0/147661408 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff8841022d0 0x7ff8841046c0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7ff880009900 tx=0x7ff88002bc30 comp rx=0 tx=0).stop 2024-03-31T23:09:08.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:08.051+0000 7ff88c725640 1 -- 172.21.15.46:0/147661408 shutdown_connections 2024-03-31T23:09:08.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:08.051+0000 7ff88c725640 1 --2- 172.21.15.46:0/147661408 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff884107530 0x7ff884109920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:08.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:08.051+0000 7ff88c725640 1 --2- 172.21.15.46:0/147661408 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff884104c00 0x7ff884106ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:08.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:08.051+0000 7ff88c725640 1 --2- 172.21.15.46:0/147661408 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff8841022d0 0x7ff8841046c0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:08.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:08.051+0000 7ff88c725640 1 -- 172.21.15.46:0/147661408 >> 172.21.15.46:0/147661408 conn(0x7ff8840fbc80 msgr2=0x7ff8840fe0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:09:08.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:08.051+0000 7ff88c725640 1 -- 172.21.15.46:0/147661408 shutdown_connections 2024-03-31T23:09:08.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:08.051+0000 7ff88c725640 1 -- 172.21.15.46:0/147661408 wait complete. 2024-03-31T23:09:08.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:08.052+0000 7ff88c725640 1 Processor -- start 2024-03-31T23:09:08.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:08.052+0000 7ff88c725640 1 -- start start 2024-03-31T23:09:08.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:08.052+0000 7ff88c725640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff8841022d0 0x7ff8841979a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:09:08.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:08.052+0000 7ff88c725640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff884104c00 0x7ff884197ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:09:08.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:08.052+0000 7ff88c725640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff884107530 0x7ff88419ef50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:09:08.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:08.052+0000 7ff88c725640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7ff88410d2a0 con 0x7ff8841022d0 2024-03-31T23:09:08.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:08.052+0000 7ff88c725640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7ff88410d120 con 0x7ff884104c00 2024-03-31T23:09:08.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:08.052+0000 7ff88c725640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7ff88410d420 con 0x7ff884107530 2024-03-31T23:09:08.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:08.052+0000 7ff88a4c1640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff8841022d0 0x7ff8841979a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:09:08.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:08.052+0000 7ff88a4c1640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff8841022d0 0x7ff8841979a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:49032/0 (socket says 172.21.15.46:49032) 2024-03-31T23:09:08.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:08.052+0000 7ff88a4c1640 1 -- 172.21.15.46:0/1027302507 learned_addr learned my addr 172.21.15.46:0/1027302507 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:09:08.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:08.052+0000 7ff88acc2640 1 --2- 172.21.15.46:0/1027302507 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff884107530 0x7ff88419ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:09:08.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:08.052+0000 7ff889cc0640 1 --2- 172.21.15.46:0/1027302507 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff884104c00 0x7ff884197ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:09:08.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:08.052+0000 7ff88a4c1640 1 -- 172.21.15.46:0/1027302507 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff884107530 msgr2=0x7ff88419ef50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:09:08.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:08.052+0000 7ff88a4c1640 1 --2- 172.21.15.46:0/1027302507 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff884107530 0x7ff88419ef50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:08.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:08.052+0000 7ff88a4c1640 1 -- 172.21.15.46:0/1027302507 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff884104c00 msgr2=0x7ff884197ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:09:08.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:08.052+0000 7ff88a4c1640 1 --2- 172.21.15.46:0/1027302507 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff884104c00 0x7ff884197ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:08.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:08.052+0000 7ff88a4c1640 1 -- 172.21.15.46:0/1027302507 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff88419f5c0 con 0x7ff8841022d0 2024-03-31T23:09:08.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:08.052+0000 7ff88a4c1640 1 --2- 172.21.15.46:0/1027302507 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff8841022d0 0x7ff8841979a0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7ff880006010 tx=0x7ff880004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:09:08.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:08.053+0000 7ff87b7fe640 1 -- 172.21.15.46:0/1027302507 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff880043020 con 0x7ff8841022d0 2024-03-31T23:09:08.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:08.053+0000 7ff87b7fe640 1 -- 172.21.15.46:0/1027302507 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff880004440 con 0x7ff8841022d0 2024-03-31T23:09:08.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:08.053+0000 7ff88c725640 1 -- 172.21.15.46:0/1027302507 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff88419f850 con 0x7ff8841022d0 2024-03-31T23:09:08.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:08.053+0000 7ff87b7fe640 1 -- 172.21.15.46:0/1027302507 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff880003bf0 con 0x7ff8841022d0 2024-03-31T23:09:08.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:08.053+0000 7ff88c725640 1 -- 172.21.15.46:0/1027302507 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff88419fda0 con 0x7ff8841022d0 2024-03-31T23:09:08.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:08.054+0000 7ff88c725640 1 -- 172.21.15.46:0/1027302507 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff8840fd770 con 0x7ff8841022d0 2024-03-31T23:09:08.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:08.055+0000 7ff87b7fe640 1 -- 172.21.15.46:0/1027302507 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 16) ==== 137320+0+0 (secure 0 0 0) 0x7ff880046020 con 0x7ff8841022d0 2024-03-31T23:09:08.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:08.055+0000 7ff87b7fe640 1 --2- 172.21.15.46:0/1027302507 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7ff8700a24a0 0x7ff8700a4960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:09:08.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:08.056+0000 7ff889cc0640 1 --2- 172.21.15.46:0/1027302507 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7ff8700a24a0 0x7ff8700a4960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:09:08.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:08.056+0000 7ff87b7fe640 1 -- 172.21.15.46:0/1027302507 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7ff8800f0570 con 0x7ff8841022d0 2024-03-31T23:09:08.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:08.056+0000 7ff889cc0640 1 --2- 172.21.15.46:0/1027302507 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7ff8700a24a0 0x7ff8700a4960 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7ff8740095f0 tx=0x7ff874005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:09:08.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:08.061+0000 7ff87b7fe640 1 -- 172.21.15.46:0/1027302507 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7ff88004d020 con 0x7ff8841022d0 2024-03-31T23:09:08.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:08.259+0000 7ff88c725640 1 -- 172.21.15.46:0/1027302507 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7ff884062350 con 0x7ff8841022d0 2024-03-31T23:09:08.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:08.259+0000 7ff87b7fe640 1 -- 172.21.15.46:0/1027302507 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v82) ==== 93+0+10 (secure 0 0 0) 0x7ff8800bdbd0 con 0x7ff8841022d0 2024-03-31T23:09:08.261 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 1 2024-03-31T23:09:08.263 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:08.261+0000 7ff88c725640 1 -- 172.21.15.46:0/1027302507 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7ff8700a24a0 msgr2=0x7ff8700a4960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:09:08.263 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:08.261+0000 7ff88c725640 1 --2- 172.21.15.46:0/1027302507 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7ff8700a24a0 0x7ff8700a4960 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7ff8740095f0 tx=0x7ff874005f90 comp rx=0 tx=0).stop 2024-03-31T23:09:08.263 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:08.261+0000 7ff88c725640 1 -- 172.21.15.46:0/1027302507 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff8841022d0 msgr2=0x7ff8841979a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:09:08.263 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:08.261+0000 7ff88c725640 1 --2- 172.21.15.46:0/1027302507 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff8841022d0 0x7ff8841979a0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7ff880006010 tx=0x7ff880004040 comp rx=0 tx=0).stop 2024-03-31T23:09:08.263 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:08.261+0000 7ff88c725640 1 -- 172.21.15.46:0/1027302507 shutdown_connections 2024-03-31T23:09:08.263 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:08.261+0000 7ff88c725640 1 --2- 172.21.15.46:0/1027302507 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7ff8700a24a0 0x7ff8700a4960 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:08.263 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:08.261+0000 7ff88c725640 1 --2- 172.21.15.46:0/1027302507 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff884107530 0x7ff88419ef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:08.263 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:08.261+0000 7ff88c725640 1 --2- 172.21.15.46:0/1027302507 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff884104c00 0x7ff884197ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:08.263 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:08.261+0000 7ff88c725640 1 --2- 172.21.15.46:0/1027302507 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff8841022d0 0x7ff8841979a0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:08.263 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:08.261+0000 7ff88c725640 1 -- 172.21.15.46:0/1027302507 >> 172.21.15.46:0/1027302507 conn(0x7ff8840fbc80 msgr2=0x7ff884102d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:09:08.263 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:08.261+0000 7ff88c725640 1 -- 172.21.15.46:0/1027302507 shutdown_connections 2024-03-31T23:09:08.263 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:08.261+0000 7ff88c725640 1 -- 172.21.15.46:0/1027302507 wait complete. 2024-03-31T23:09:08.273 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-31T23:09:08.379 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:08.376+0000 7f9a6e7a1640 1 -- 172.21.15.46:0/3474490381 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9a681021f0 msgr2=0x7f9a68108a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:09:08.379 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:08.376+0000 7f9a6e7a1640 1 --2- 172.21.15.46:0/3474490381 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9a681021f0 0x7f9a68108a80 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f9a60009ca0 tx=0x7f9a6002c0f0 comp rx=0 tx=0).stop 2024-03-31T23:09:08.379 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:08.376+0000 7f9a6e7a1640 1 -- 172.21.15.46:0/3474490381 shutdown_connections 2024-03-31T23:09:08.379 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:08.376+0000 7f9a6e7a1640 1 --2- 172.21.15.46:0/3474490381 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9a681021f0 0x7f9a68108a80 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:08.380 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:08.376+0000 7f9a6e7a1640 1 --2- 172.21.15.46:0/3474490381 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9a68101830 0x7f9a68101cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:08.380 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:08.376+0000 7f9a6e7a1640 1 --2- 172.21.15.46:0/3474490381 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9a68100640 0x7f9a68100a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:08.380 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:08.376+0000 7f9a6e7a1640 1 -- 172.21.15.46:0/3474490381 >> 172.21.15.46:0/3474490381 conn(0x7f9a680fbbb0 msgr2=0x7f9a680fdff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:09:08.380 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:08.376+0000 7f9a6e7a1640 1 -- 172.21.15.46:0/3474490381 shutdown_connections 2024-03-31T23:09:08.380 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:08.376+0000 7f9a6e7a1640 1 -- 172.21.15.46:0/3474490381 wait complete. 2024-03-31T23:09:08.380 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:08.377+0000 7f9a6e7a1640 1 Processor -- start 2024-03-31T23:09:08.380 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:08.377+0000 7f9a6e7a1640 1 -- start start 2024-03-31T23:09:08.380 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:08.377+0000 7f9a6e7a1640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9a68100640 0x7f9a68196790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:09:08.380 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:08.377+0000 7f9a6e7a1640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9a68101830 0x7f9a68196cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:09:08.380 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:08.377+0000 7f9a6e7a1640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9a681021f0 0x7f9a6819dd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:09:08.380 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:08.377+0000 7f9a6e7a1640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f9a6810c180 con 0x7f9a68101830 2024-03-31T23:09:08.380 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:08.377+0000 7f9a6e7a1640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f9a6810c000 con 0x7f9a681021f0 2024-03-31T23:09:08.380 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:08.377+0000 7f9a6e7a1640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f9a6810c300 con 0x7f9a68100640 2024-03-31T23:09:08.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:08.377+0000 7f9a677fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9a68101830 0x7f9a68196cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:09:08.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:08.377+0000 7f9a67fff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9a68100640 0x7f9a68196790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:09:08.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:08.377+0000 7f9a677fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9a68101830 0x7f9a68196cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:49046/0 (socket says 172.21.15.46:49046) 2024-03-31T23:09:08.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:08.377+0000 7f9a677fe640 1 -- 172.21.15.46:0/3145996804 learned_addr learned my addr 172.21.15.46:0/3145996804 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:09:08.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:08.377+0000 7f9a6cd3e640 1 --2- 172.21.15.46:0/3145996804 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9a681021f0 0x7f9a6819dd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:09:08.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:08.377+0000 7f9a677fe640 1 -- 172.21.15.46:0/3145996804 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9a68100640 msgr2=0x7f9a68196790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:09:08.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:08.377+0000 7f9a677fe640 1 --2- 172.21.15.46:0/3145996804 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9a68100640 0x7f9a68196790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:08.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:08.377+0000 7f9a677fe640 1 -- 172.21.15.46:0/3145996804 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9a681021f0 msgr2=0x7f9a6819dd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:09:08.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:08.377+0000 7f9a677fe640 1 --2- 172.21.15.46:0/3145996804 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9a681021f0 0x7f9a6819dd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:08.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:08.377+0000 7f9a677fe640 1 -- 172.21.15.46:0/3145996804 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a6819e3b0 con 0x7f9a68101830 2024-03-31T23:09:08.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:08.377+0000 7f9a677fe640 1 --2- 172.21.15.46:0/3145996804 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9a68101830 0x7f9a68196cd0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f9a5c00e9f0 tx=0x7f9a5c00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:09:08.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:08.378+0000 7f9a657fa640 1 -- 172.21.15.46:0/3145996804 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a5c00cd80 con 0x7f9a68101830 2024-03-31T23:09:08.382 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:08.378+0000 7f9a657fa640 1 -- 172.21.15.46:0/3145996804 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9a5c0054d0 con 0x7f9a68101830 2024-03-31T23:09:08.382 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:08.378+0000 7f9a6e7a1640 1 -- 172.21.15.46:0/3145996804 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9a6819e6a0 con 0x7f9a68101830 2024-03-31T23:09:08.382 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:08.378+0000 7f9a657fa640 1 -- 172.21.15.46:0/3145996804 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a5c010490 con 0x7f9a68101830 2024-03-31T23:09:08.382 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:08.378+0000 7f9a6e7a1640 1 -- 172.21.15.46:0/3145996804 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9a6819eba0 con 0x7f9a68101830 2024-03-31T23:09:08.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:08.379+0000 7f9a6e7a1640 1 -- 172.21.15.46:0/3145996804 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9a68107960 con 0x7f9a68101830 2024-03-31T23:09:08.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:08.380+0000 7f9a657fa640 1 -- 172.21.15.46:0/3145996804 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 16) ==== 137320+0+0 (secure 0 0 0) 0x7f9a5c005020 con 0x7f9a68101830 2024-03-31T23:09:08.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:08.380+0000 7f9a657fa640 1 --2- 172.21.15.46:0/3145996804 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f9a500a24a0 0x7f9a500a4960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:09:08.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:08.380+0000 7f9a67fff640 1 --2- 172.21.15.46:0/3145996804 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f9a500a24a0 0x7f9a500a4960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:09:08.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:08.381+0000 7f9a657fa640 1 -- 172.21.15.46:0/3145996804 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f9a5c0ce690 con 0x7f9a68101830 2024-03-31T23:09:08.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:08.381+0000 7f9a67fff640 1 --2- 172.21.15.46:0/3145996804 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f9a500a24a0 0x7f9a500a4960 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f9a68101690 tx=0x7f9a58005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:09:08.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:08.385+0000 7f9a657fa640 1 -- 172.21.15.46:0/3145996804 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f9a5c09bc50 con 0x7f9a68101830 2024-03-31T23:09:08.565 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:09:08 smithi046 ceph-mon[32498]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-31T23:09:08.565 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:09:08 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:09:08.565 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:09:08 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:09:08.566 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:09:08 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:09:08.566 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:09:08 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:09:08.566 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:09:08 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:09:08.566 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:09:08 smithi046 ceph-mon[32498]: mds.? [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] up:boot 2024-03-31T23:09:08.566 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:09:08 smithi046 ceph-mon[32498]: fsmap 12 up:standby 2024-03-31T23:09:08.566 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:09:08 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-31T23:09:08.566 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:09:08 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-31T23:09:08.566 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:09:08 smithi046 ceph-mon[32498]: from='client.? 172.21.15.46:0/3789494717' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-31T23:09:08.566 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:09:08 smithi046 ceph-mon[32498]: from='client.? 172.21.15.46:0/286376108' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-31T23:09:08.566 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:09:08 smithi046 ceph-mon[32498]: from='client.? 172.21.15.46:0/1027302507' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-31T23:09:08.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:08.576+0000 7f9a6e7a1640 1 -- 172.21.15.46:0/3145996804 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7f9a68062350 con 0x7f9a68101830 2024-03-31T23:09:08.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:08.576+0000 7f9a657fa640 1 -- 172.21.15.46:0/3145996804 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v82) ==== 84+0+65 (secure 0 0 0) 0x7f9a5c09b550 con 0x7f9a68101830 2024-03-31T23:09:08.577 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-31T23:09:08.577 INFO:teuthology.orchestra.run.smithi046.stdout:[ 2024-03-31T23:09:08.578 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-03-31T23:09:08.578 INFO:teuthology.orchestra.run.smithi046.stdout: "poolnum": 1, 2024-03-31T23:09:08.578 INFO:teuthology.orchestra.run.smithi046.stdout: "poolname": ".mgr" 2024-03-31T23:09:08.578 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-31T23:09:08.578 INFO:teuthology.orchestra.run.smithi046.stdout:] 2024-03-31T23:09:08.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:08.579+0000 7f9a6e7a1640 1 -- 172.21.15.46:0/3145996804 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f9a500a24a0 msgr2=0x7f9a500a4960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:09:08.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:08.579+0000 7f9a6e7a1640 1 --2- 172.21.15.46:0/3145996804 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f9a500a24a0 0x7f9a500a4960 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f9a68101690 tx=0x7f9a58005f90 comp rx=0 tx=0).stop 2024-03-31T23:09:08.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:08.579+0000 7f9a6e7a1640 1 -- 172.21.15.46:0/3145996804 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9a68101830 msgr2=0x7f9a68196cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:09:08.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:08.579+0000 7f9a6e7a1640 1 --2- 172.21.15.46:0/3145996804 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9a68101830 0x7f9a68196cd0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f9a5c00e9f0 tx=0x7f9a5c00edf0 comp rx=0 tx=0).stop 2024-03-31T23:09:08.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:08.580+0000 7f9a6e7a1640 1 -- 172.21.15.46:0/3145996804 shutdown_connections 2024-03-31T23:09:08.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:08.580+0000 7f9a6e7a1640 1 --2- 172.21.15.46:0/3145996804 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f9a500a24a0 0x7f9a500a4960 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:08.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:08.580+0000 7f9a6e7a1640 1 --2- 172.21.15.46:0/3145996804 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9a681021f0 0x7f9a6819dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:08.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:08.580+0000 7f9a6e7a1640 1 --2- 172.21.15.46:0/3145996804 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9a68101830 0x7f9a68196cd0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:08.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:08.580+0000 7f9a6e7a1640 1 --2- 172.21.15.46:0/3145996804 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9a68100640 0x7f9a68196790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:08.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:08.580+0000 7f9a6e7a1640 1 -- 172.21.15.46:0/3145996804 >> 172.21.15.46:0/3145996804 conn(0x7f9a680fbbb0 msgr2=0x7f9a68104b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:09:08.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:08.580+0000 7f9a6e7a1640 1 -- 172.21.15.46:0/3145996804 shutdown_connections 2024-03-31T23:09:08.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:08.580+0000 7f9a6e7a1640 1 -- 172.21.15.46:0/3145996804 wait complete. 2024-03-31T23:09:08.594 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-31T23:09:08.594 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-31T23:09:08.671 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:09:08 smithi142 ceph-mon[35271]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-31T23:09:08.671 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:09:08 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:09:08.672 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:09:08 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:09:08.672 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:09:08 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:09:08.672 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:09:08 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:09:08.672 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:09:08 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:09:08.672 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:09:08 smithi142 ceph-mon[35271]: mds.? [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] up:boot 2024-03-31T23:09:08.672 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:09:08 smithi142 ceph-mon[35271]: fsmap 12 up:standby 2024-03-31T23:09:08.672 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:09:08 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-31T23:09:08.672 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:09:08 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-31T23:09:08.672 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:09:08 smithi142 ceph-mon[35271]: from='client.? 172.21.15.46:0/3789494717' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-31T23:09:08.672 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:09:08 smithi142 ceph-mon[35271]: from='client.? 172.21.15.46:0/286376108' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-31T23:09:08.672 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:09:08 smithi142 ceph-mon[35271]: from='client.? 172.21.15.46:0/1027302507' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-31T23:09:08.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:08.701+0000 7fc7ba03b640 1 -- 172.21.15.46:0/2571998486 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc7b4100650 msgr2=0x7fc7b4100a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:09:08.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:08.701+0000 7fc7ba03b640 1 --2- 172.21.15.46:0/2571998486 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc7b4100650 0x7fc7b4100a70 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fc7a8009900 tx=0x7fc7a802bc30 comp rx=0 tx=0).stop 2024-03-31T23:09:08.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:08.701+0000 7fc7ba03b640 1 -- 172.21.15.46:0/2571998486 shutdown_connections 2024-03-31T23:09:08.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:08.701+0000 7fc7ba03b640 1 --2- 172.21.15.46:0/2571998486 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc7b4102200 0x7fc7b4108a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:08.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:08.701+0000 7fc7ba03b640 1 --2- 172.21.15.46:0/2571998486 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc7b4101840 0x7fc7b4101cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:08.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:08.701+0000 7fc7ba03b640 1 --2- 172.21.15.46:0/2571998486 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc7b4100650 0x7fc7b4100a70 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:08.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:08.701+0000 7fc7ba03b640 1 -- 172.21.15.46:0/2571998486 >> 172.21.15.46:0/2571998486 conn(0x7fc7b40fbbe0 msgr2=0x7fc7b40fe000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:09:08.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:08.701+0000 7fc7ba03b640 1 -- 172.21.15.46:0/2571998486 shutdown_connections 2024-03-31T23:09:08.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:08.701+0000 7fc7ba03b640 1 -- 172.21.15.46:0/2571998486 wait complete. 2024-03-31T23:09:08.705 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:08.701+0000 7fc7ba03b640 1 Processor -- start 2024-03-31T23:09:08.705 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:08.702+0000 7fc7ba03b640 1 -- start start 2024-03-31T23:09:08.705 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:08.702+0000 7fc7ba03b640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc7b4100650 0x7fc7b41045e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:09:08.705 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:08.702+0000 7fc7ba03b640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc7b4101840 0x7fc7b4102c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:09:08.705 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:08.702+0000 7fc7ba03b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc7b4102200 0x7fc7b4103160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:09:08.705 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:08.702+0000 7fc7ba03b640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fc7b410c1d0 con 0x7fc7b4102200 2024-03-31T23:09:08.705 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:08.702+0000 7fc7ba03b640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fc7b410c050 con 0x7fc7b4100650 2024-03-31T23:09:08.705 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:08.702+0000 7fc7ba03b640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fc7b410c350 con 0x7fc7b4101840 2024-03-31T23:09:08.705 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:08.702+0000 7fc7b3fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc7b4102200 0x7fc7b4103160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:09:08.705 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:08.702+0000 7fc7b3fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc7b4102200 0x7fc7b4103160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:49058/0 (socket says 172.21.15.46:49058) 2024-03-31T23:09:08.705 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:08.702+0000 7fc7b3fff640 1 -- 172.21.15.46:0/1474954948 learned_addr learned my addr 172.21.15.46:0/1474954948 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:09:08.705 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:08.702+0000 7fc7b2ffd640 1 --2- 172.21.15.46:0/1474954948 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc7b4101840 0x7fc7b4102c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:09:08.705 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:08.702+0000 7fc7b37fe640 1 --2- 172.21.15.46:0/1474954948 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc7b4100650 0x7fc7b41045e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:09:08.706 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:08.702+0000 7fc7b3fff640 1 -- 172.21.15.46:0/1474954948 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc7b4101840 msgr2=0x7fc7b4102c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:09:08.706 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:08.702+0000 7fc7b3fff640 1 --2- 172.21.15.46:0/1474954948 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc7b4101840 0x7fc7b4102c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:08.706 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:08.702+0000 7fc7b3fff640 1 -- 172.21.15.46:0/1474954948 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc7b4100650 msgr2=0x7fc7b41045e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:09:08.706 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:08.702+0000 7fc7b3fff640 1 --2- 172.21.15.46:0/1474954948 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc7b4100650 0x7fc7b41045e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:08.706 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:08.702+0000 7fc7b3fff640 1 -- 172.21.15.46:0/1474954948 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc7b41039f0 con 0x7fc7b4102200 2024-03-31T23:09:08.706 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:08.702+0000 7fc7b2ffd640 1 --2- 172.21.15.46:0/1474954948 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc7b4101840 0x7fc7b4102c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T23:09:08.706 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:08.702+0000 7fc7b3fff640 1 --2- 172.21.15.46:0/1474954948 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc7b4102200 0x7fc7b4103160 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fc7ac00eaa0 tx=0x7fc7ac00ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:09:08.706 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:08.702+0000 7fc7b0ff9640 1 -- 172.21.15.46:0/1474954948 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7ac00cdb0 con 0x7fc7b4102200 2024-03-31T23:09:08.706 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:08.703+0000 7fc7b0ff9640 1 -- 172.21.15.46:0/1474954948 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc7ac005020 con 0x7fc7b4102200 2024-03-31T23:09:08.706 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:08.703+0000 7fc7ba03b640 1 -- 172.21.15.46:0/1474954948 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc7b41a1b10 con 0x7fc7b4102200 2024-03-31T23:09:08.706 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:08.703+0000 7fc7b0ff9640 1 -- 172.21.15.46:0/1474954948 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7ac0116d0 con 0x7fc7b4102200 2024-03-31T23:09:08.706 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:08.703+0000 7fc7ba03b640 1 -- 172.21.15.46:0/1474954948 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc7b41a2060 con 0x7fc7b4102200 2024-03-31T23:09:08.706 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:08.704+0000 7fc7ba03b640 1 -- 172.21.15.46:0/1474954948 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc7b41a2290 con 0x7fc7b4102200 2024-03-31T23:09:08.707 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:08.705+0000 7fc7b0ff9640 1 -- 172.21.15.46:0/1474954948 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 16) ==== 137320+0+0 (secure 0 0 0) 0x7fc7ac0051d0 con 0x7fc7b4102200 2024-03-31T23:09:08.707 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:08.705+0000 7fc7b0ff9640 1 --2- 172.21.15.46:0/1474954948 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fc79c0a24a0 0x7fc79c0a4960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:09:08.707 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:08.705+0000 7fc7b37fe640 1 --2- 172.21.15.46:0/1474954948 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fc79c0a24a0 0x7fc79c0a4960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:09:08.707 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:08.705+0000 7fc7b0ff9640 1 -- 172.21.15.46:0/1474954948 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7fc7ac0ce480 con 0x7fc7b4102200 2024-03-31T23:09:08.708 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:08.706+0000 7fc7b37fe640 1 --2- 172.21.15.46:0/1474954948 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fc79c0a24a0 0x7fc79c0a4960 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fc7a802c240 tx=0x7fc7a802c7e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:09:08.710 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:09:08 smithi162 ceph-mon[36237]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-31T23:09:08.710 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:09:08 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:09:08.710 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:09:08 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:09:08.710 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:09:08 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:09:08.711 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:09:08 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:09:08.711 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:09:08 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:09:08.711 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:09:08 smithi162 ceph-mon[36237]: mds.? [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] up:boot 2024-03-31T23:09:08.711 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:09:08 smithi162 ceph-mon[36237]: fsmap 12 up:standby 2024-03-31T23:09:08.711 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:09:08 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-31T23:09:08.711 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:09:08 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-31T23:09:08.711 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:09:08 smithi162 ceph-mon[36237]: from='client.? 172.21.15.46:0/3789494717' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-31T23:09:08.711 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:09:08 smithi162 ceph-mon[36237]: from='client.? 172.21.15.46:0/286376108' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-31T23:09:08.711 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:09:08 smithi162 ceph-mon[36237]: from='client.? 172.21.15.46:0/1027302507' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-31T23:09:08.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:08.710+0000 7fc7b0ff9640 1 -- 172.21.15.46:0/1474954948 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fc7ac016030 con 0x7fc7b4102200 2024-03-31T23:09:08.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:08.893+0000 7fc7ba03b640 1 -- 172.21.15.46:0/1474954948 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7fc7b41a2500 con 0x7fc7b4102200 2024-03-31T23:09:09.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:09.569+0000 7fc7b0ff9640 1 -- 172.21.15.46:0/1474954948 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v83) ==== 138+0+0 (secure 0 0 0) 0x7fc7ac09bae0 con 0x7fc7b4102200 2024-03-31T23:09:09.570 INFO:teuthology.orchestra.run.smithi046.stderr:pool 'cephfs_metadata' created 2024-03-31T23:09:09.574 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:09.572+0000 7fc7ba03b640 1 -- 172.21.15.46:0/1474954948 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fc79c0a24a0 msgr2=0x7fc79c0a4960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:09:09.574 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:09.572+0000 7fc7ba03b640 1 --2- 172.21.15.46:0/1474954948 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fc79c0a24a0 0x7fc79c0a4960 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fc7a802c240 tx=0x7fc7a802c7e0 comp rx=0 tx=0).stop 2024-03-31T23:09:09.574 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:09.572+0000 7fc7ba03b640 1 -- 172.21.15.46:0/1474954948 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc7b4102200 msgr2=0x7fc7b4103160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:09:09.574 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:09.572+0000 7fc7ba03b640 1 --2- 172.21.15.46:0/1474954948 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc7b4102200 0x7fc7b4103160 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fc7ac00eaa0 tx=0x7fc7ac00ef60 comp rx=0 tx=0).stop 2024-03-31T23:09:09.574 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:09.573+0000 7fc7ba03b640 1 -- 172.21.15.46:0/1474954948 shutdown_connections 2024-03-31T23:09:09.574 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:09.573+0000 7fc7ba03b640 1 --2- 172.21.15.46:0/1474954948 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fc79c0a24a0 0x7fc79c0a4960 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:09.574 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:09.573+0000 7fc7ba03b640 1 --2- 172.21.15.46:0/1474954948 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc7b4102200 0x7fc7b4103160 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:09.574 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:09.573+0000 7fc7ba03b640 1 --2- 172.21.15.46:0/1474954948 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc7b4101840 0x7fc7b4102c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:09.574 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:09.573+0000 7fc7ba03b640 1 --2- 172.21.15.46:0/1474954948 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc7b4100650 0x7fc7b41045e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:09.574 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:09.573+0000 7fc7ba03b640 1 -- 172.21.15.46:0/1474954948 >> 172.21.15.46:0/1474954948 conn(0x7fc7b40fbbe0 msgr2=0x7fc7b40fd6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:09:09.575 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:09.573+0000 7fc7ba03b640 1 -- 172.21.15.46:0/1474954948 shutdown_connections 2024-03-31T23:09:09.575 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:09.573+0000 7fc7ba03b640 1 -- 172.21.15.46:0/1474954948 wait complete. 2024-03-31T23:09:09.588 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.05 2024-03-31T23:09:09.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:09.699+0000 7fa68b7e9640 1 -- 172.21.15.46:0/2964040665 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa684101840 msgr2=0x7fa684101cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:09:09.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:09.699+0000 7fa68b7e9640 1 --2- 172.21.15.46:0/2964040665 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa684101840 0x7fa684101cc0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fa678009ca0 tx=0x7fa67802c0f0 comp rx=0 tx=0).stop 2024-03-31T23:09:09.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:09.699+0000 7fa68b7e9640 1 -- 172.21.15.46:0/2964040665 shutdown_connections 2024-03-31T23:09:09.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:09.699+0000 7fa68b7e9640 1 --2- 172.21.15.46:0/2964040665 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa684102200 0x7fa684108a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:09.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:09.699+0000 7fa68b7e9640 1 --2- 172.21.15.46:0/2964040665 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa684101840 0x7fa684101cc0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:09.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:09.699+0000 7fa68b7e9640 1 --2- 172.21.15.46:0/2964040665 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa684100650 0x7fa684100a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:09.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:09.699+0000 7fa68b7e9640 1 -- 172.21.15.46:0/2964040665 >> 172.21.15.46:0/2964040665 conn(0x7fa6840fbbe0 msgr2=0x7fa6840fe000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:09:09.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:09.699+0000 7fa68b7e9640 1 -- 172.21.15.46:0/2964040665 shutdown_connections 2024-03-31T23:09:09.703 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:09.699+0000 7fa68b7e9640 1 -- 172.21.15.46:0/2964040665 wait complete. 2024-03-31T23:09:09.703 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:09.699+0000 7fa68b7e9640 1 Processor -- start 2024-03-31T23:09:09.703 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:09.700+0000 7fa68b7e9640 1 -- start start 2024-03-31T23:09:09.703 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:09.700+0000 7fa68b7e9640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa684100650 0x7fa684104590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:09:09.703 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:09.700+0000 7fa68b7e9640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa684101840 0x7fa684102c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:09:09.703 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:09.700+0000 7fa68b7e9640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa684102200 0x7fa684103160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:09:09.703 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:09.700+0000 7fa68b7e9640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fa68410c190 con 0x7fa684101840 2024-03-31T23:09:09.703 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:09.700+0000 7fa68b7e9640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fa68410c010 con 0x7fa684102200 2024-03-31T23:09:09.703 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:09.700+0000 7fa68b7e9640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fa68410c310 con 0x7fa684100650 2024-03-31T23:09:09.703 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:09.700+0000 7fa688d84640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa684101840 0x7fa684102c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:09:09.703 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:09.700+0000 7fa688d84640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa684101840 0x7fa684102c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:49070/0 (socket says 172.21.15.46:49070) 2024-03-31T23:09:09.703 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:09.700+0000 7fa688d84640 1 -- 172.21.15.46:0/735353621 learned_addr learned my addr 172.21.15.46:0/735353621 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:09:09.703 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:09.700+0000 7fa689585640 1 --2- 172.21.15.46:0/735353621 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa684100650 0x7fa684104590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:09:09.703 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:09.700+0000 7fa689d86640 1 --2- 172.21.15.46:0/735353621 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa684102200 0x7fa684103160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:09:09.704 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:09.700+0000 7fa688d84640 1 -- 172.21.15.46:0/735353621 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa684100650 msgr2=0x7fa684104590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:09:09.704 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:09.700+0000 7fa688d84640 1 --2- 172.21.15.46:0/735353621 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa684100650 0x7fa684104590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:09.704 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:09.700+0000 7fa688d84640 1 -- 172.21.15.46:0/735353621 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa684102200 msgr2=0x7fa684103160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:09:09.704 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:09.700+0000 7fa688d84640 1 --2- 172.21.15.46:0/735353621 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa684102200 0x7fa684103160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:09.704 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:09.700+0000 7fa688d84640 1 -- 172.21.15.46:0/735353621 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa684103a20 con 0x7fa684101840 2024-03-31T23:09:09.704 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:09.700+0000 7fa688d84640 1 --2- 172.21.15.46:0/735353621 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa684101840 0x7fa684102c20 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fa678009dc0 tx=0x7fa6780328a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:09:09.704 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:09.700+0000 7fa6827fc640 1 -- 172.21.15.46:0/735353621 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa678041020 con 0x7fa684101840 2024-03-31T23:09:09.704 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:09.700+0000 7fa6827fc640 1 -- 172.21.15.46:0/735353621 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa678003e20 con 0x7fa684101840 2024-03-31T23:09:09.704 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:09.700+0000 7fa68b7e9640 1 -- 172.21.15.46:0/735353621 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa6841a1ae0 con 0x7fa684101840 2024-03-31T23:09:09.704 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:09.701+0000 7fa6827fc640 1 -- 172.21.15.46:0/735353621 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa678039040 con 0x7fa684101840 2024-03-31T23:09:09.704 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:09.701+0000 7fa68b7e9640 1 -- 172.21.15.46:0/735353621 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa6841a1ff0 con 0x7fa684101840 2024-03-31T23:09:09.705 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:09.702+0000 7fa68b7e9640 1 -- 172.21.15.46:0/735353621 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa684068540 con 0x7fa684101840 2024-03-31T23:09:09.705 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:09.703+0000 7fa6827fc640 1 -- 172.21.15.46:0/735353621 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 16) ==== 137320+0+0 (secure 0 0 0) 0x7fa678044020 con 0x7fa684101840 2024-03-31T23:09:09.705 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:09.703+0000 7fa6827fc640 1 --2- 172.21.15.46:0/735353621 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fa66c0a2520 0x7fa66c0a49e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:09:09.706 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:09.703+0000 7fa689585640 1 --2- 172.21.15.46:0/735353621 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fa66c0a2520 0x7fa66c0a49e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:09:09.706 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:09.703+0000 7fa6827fc640 1 -- 172.21.15.46:0/735353621 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7497+0+0 (secure 0 0 0) 0x7fa6780f0880 con 0x7fa684101840 2024-03-31T23:09:09.706 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:09.704+0000 7fa689585640 1 --2- 172.21.15.46:0/735353621 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fa66c0a2520 0x7fa66c0a49e0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fa6841016a0 tx=0x7fa67c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:09:09.708 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:09.707+0000 7fa6827fc640 1 -- 172.21.15.46:0/735353621 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fa67804b020 con 0x7fa684101840 2024-03-31T23:09:09.862 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:09:09 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:09:09.862 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:09:09 smithi046 ceph-mon[32498]: from='client.? 172.21.15.46:0/3145996804' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-03-31T23:09:09.862 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:09:09 smithi046 ceph-mon[32498]: from='client.? 172.21.15.46:0/1474954948' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-03-31T23:09:09.862 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:09:09 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:09:09.862 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:09:09 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:09:09.862 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:09:09 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:09:09.862 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:09:09 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:09:09.862 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:09:09 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:09:09.862 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:09:09 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:09:09.862 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:09:09 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:09:09.862 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:09:09 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-31T23:09:09.863 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:09:09 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:09:09.863 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:09:09 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-31T23:09:09.892 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:09.891+0000 7fa68b7e9640 1 -- 172.21.15.46:0/735353621 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05} v 0) -- 0x7fa6841a22e0 con 0x7fa684101840 2024-03-31T23:09:09.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:09:09 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:09:09.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:09:09 smithi142 ceph-mon[35271]: from='client.? 172.21.15.46:0/3145996804' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-03-31T23:09:09.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:09:09 smithi142 ceph-mon[35271]: from='client.? 172.21.15.46:0/1474954948' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-03-31T23:09:09.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:09:09 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:09:09.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:09:09 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:09:09.944 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:09:09 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:09:09.944 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:09:09 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:09:09.944 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:09:09 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:09:09.944 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:09:09 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:09:09.944 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:09:09 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:09:09.944 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:09:09 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-31T23:09:09.944 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:09:09 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:09:09.944 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:09:09 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-31T23:09:10.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:09:09 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:09:10.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:09:09 smithi162 ceph-mon[36237]: from='client.? 172.21.15.46:0/3145996804' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-03-31T23:09:10.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:09:09 smithi162 ceph-mon[36237]: from='client.? 172.21.15.46:0/1474954948' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-03-31T23:09:10.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:09:09 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:09:10.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:09:09 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:09:10.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:09:09 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:09:10.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:09:09 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:09:10.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:09:09 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:09:10.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:09:09 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:09:10.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:09:09 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:09:10.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:09:09 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-31T23:09:10.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:09:09 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:09:10.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:09:09 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-31T23:09:10.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:10.571+0000 7fa6827fc640 1 -- 172.21.15.46:0/735353621 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]=0 pool 'cephfs_data' created v84) ==== 171+0+0 (secure 0 0 0) 0x7fa6780bdcd0 con 0x7fa684101840 2024-03-31T23:09:10.572 INFO:teuthology.orchestra.run.smithi046.stderr:pool 'cephfs_data' created 2024-03-31T23:09:10.575 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:10.574+0000 7fa68b7e9640 1 -- 172.21.15.46:0/735353621 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fa66c0a2520 msgr2=0x7fa66c0a49e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:09:10.575 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:10.574+0000 7fa68b7e9640 1 --2- 172.21.15.46:0/735353621 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fa66c0a2520 0x7fa66c0a49e0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fa6841016a0 tx=0x7fa67c005f90 comp rx=0 tx=0).stop 2024-03-31T23:09:10.575 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:10.574+0000 7fa68b7e9640 1 -- 172.21.15.46:0/735353621 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa684101840 msgr2=0x7fa684102c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:09:10.575 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:10.574+0000 7fa68b7e9640 1 --2- 172.21.15.46:0/735353621 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa684101840 0x7fa684102c20 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fa678009dc0 tx=0x7fa6780328a0 comp rx=0 tx=0).stop 2024-03-31T23:09:10.575 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:10.574+0000 7fa68b7e9640 1 -- 172.21.15.46:0/735353621 shutdown_connections 2024-03-31T23:09:10.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:10.574+0000 7fa68b7e9640 1 --2- 172.21.15.46:0/735353621 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fa66c0a2520 0x7fa66c0a49e0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:10.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:10.574+0000 7fa68b7e9640 1 --2- 172.21.15.46:0/735353621 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa684102200 0x7fa684103160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:10.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:10.574+0000 7fa68b7e9640 1 --2- 172.21.15.46:0/735353621 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa684101840 0x7fa684102c20 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:10.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:10.574+0000 7fa68b7e9640 1 --2- 172.21.15.46:0/735353621 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa684100650 0x7fa684104590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:10.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:10.574+0000 7fa68b7e9640 1 -- 172.21.15.46:0/735353621 >> 172.21.15.46:0/735353621 conn(0x7fa6840fbbe0 msgr2=0x7fa6840688b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:09:10.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:10.575+0000 7fa68b7e9640 1 -- 172.21.15.46:0/735353621 shutdown_connections 2024-03-31T23:09:10.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:10.575+0000 7fa68b7e9640 1 -- 172.21.15.46:0/735353621 wait complete. 2024-03-31T23:09:10.600 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-31T23:09:10.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:10.707+0000 7f4b06b5e640 1 -- 172.21.15.46:0/2495576705 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4b00103330 msgr2=0x7f4b00109bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:09:10.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:10.707+0000 7f4b06b5e640 1 --2- 172.21.15.46:0/2495576705 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4b00103330 0x7f4b00109bc0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f4af8009ca0 tx=0x7f4af802c0f0 comp rx=0 tx=0).stop 2024-03-31T23:09:10.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:10.707+0000 7f4b06b5e640 1 -- 172.21.15.46:0/2495576705 shutdown_connections 2024-03-31T23:09:10.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:10.707+0000 7f4b06b5e640 1 --2- 172.21.15.46:0/2495576705 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4b00103330 0x7f4b00109bc0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:10.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:10.707+0000 7f4b06b5e640 1 --2- 172.21.15.46:0/2495576705 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4b00102970 0x7f4b00102df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:10.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:10.707+0000 7f4b06b5e640 1 --2- 172.21.15.46:0/2495576705 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4b00101780 0x7f4b00101ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:10.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:10.707+0000 7f4b06b5e640 1 -- 172.21.15.46:0/2495576705 >> 172.21.15.46:0/2495576705 conn(0x7f4b000fccf0 msgr2=0x7f4b000ff130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:09:10.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:10.707+0000 7f4b06b5e640 1 -- 172.21.15.46:0/2495576705 shutdown_connections 2024-03-31T23:09:10.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:10.707+0000 7f4b06b5e640 1 -- 172.21.15.46:0/2495576705 wait complete. 2024-03-31T23:09:10.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:10.707+0000 7f4b06b5e640 1 Processor -- start 2024-03-31T23:09:10.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:10.707+0000 7f4b06b5e640 1 -- start start 2024-03-31T23:09:10.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:10.707+0000 7f4b06b5e640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4b00101780 0x7f4b00197970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:09:10.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:10.707+0000 7f4b06b5e640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4b00102970 0x7f4b00197eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:09:10.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:10.707+0000 7f4b06b5e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4b00103330 0x7f4b0019ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:09:10.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:10.707+0000 7f4b06b5e640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f4b0010d290 con 0x7f4b00103330 2024-03-31T23:09:10.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:10.707+0000 7f4b06b5e640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f4b0010d110 con 0x7f4b00101780 2024-03-31T23:09:10.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:10.707+0000 7f4b06b5e640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f4b0010d410 con 0x7f4b00102970 2024-03-31T23:09:10.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:10.708+0000 7f4b050fb640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4b00103330 0x7f4b0019ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:09:10.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:10.708+0000 7f4b050fb640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4b00103330 0x7f4b0019ef20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:49092/0 (socket says 172.21.15.46:49092) 2024-03-31T23:09:10.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:10.708+0000 7f4b050fb640 1 -- 172.21.15.46:0/4149552256 learned_addr learned my addr 172.21.15.46:0/4149552256 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:09:10.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:10.708+0000 7f4b048fa640 1 --2- 172.21.15.46:0/4149552256 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4b00101780 0x7f4b00197970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:09:10.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:10.708+0000 7f4afffff640 1 --2- 172.21.15.46:0/4149552256 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4b00102970 0x7f4b00197eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:09:10.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:10.708+0000 7f4b050fb640 1 -- 172.21.15.46:0/4149552256 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4b00102970 msgr2=0x7f4b00197eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:09:10.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:10.708+0000 7f4b050fb640 1 --2- 172.21.15.46:0/4149552256 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4b00102970 0x7f4b00197eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:10.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:10.708+0000 7f4b050fb640 1 -- 172.21.15.46:0/4149552256 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4b00101780 msgr2=0x7f4b00197970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:09:10.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:10.708+0000 7f4b050fb640 1 --2- 172.21.15.46:0/4149552256 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4b00101780 0x7f4b00197970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:10.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:10.708+0000 7f4b050fb640 1 -- 172.21.15.46:0/4149552256 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b0019f590 con 0x7f4b00103330 2024-03-31T23:09:10.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:10.708+0000 7f4b050fb640 1 --2- 172.21.15.46:0/4149552256 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4b00103330 0x7f4b0019ef20 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f4af802c700 tx=0x7f4af80328a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:09:10.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:10.708+0000 7f4afdffb640 1 -- 172.21.15.46:0/4149552256 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4af8041020 con 0x7f4b00103330 2024-03-31T23:09:10.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:10.708+0000 7f4b06b5e640 1 -- 172.21.15.46:0/4149552256 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4b0019f820 con 0x7f4b00103330 2024-03-31T23:09:10.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:10.708+0000 7f4afdffb640 1 -- 172.21.15.46:0/4149552256 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4af8003e20 con 0x7f4b00103330 2024-03-31T23:09:10.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:10.708+0000 7f4afdffb640 1 -- 172.21.15.46:0/4149552256 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4af8039040 con 0x7f4b00103330 2024-03-31T23:09:10.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:10.708+0000 7f4b06b5e640 1 -- 172.21.15.46:0/4149552256 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4b0019fd10 con 0x7f4b00103330 2024-03-31T23:09:10.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:10.709+0000 7f4b06b5e640 1 -- 172.21.15.46:0/4149552256 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4b00108ac0 con 0x7f4b00103330 2024-03-31T23:09:10.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:10.710+0000 7f4afdffb640 1 -- 172.21.15.46:0/4149552256 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 16) ==== 137320+0+0 (secure 0 0 0) 0x7f4af8044020 con 0x7f4b00103330 2024-03-31T23:09:10.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:10.711+0000 7f4afdffb640 1 --2- 172.21.15.46:0/4149552256 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f4af00a24a0 0x7f4af00a4960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:09:10.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:10.711+0000 7f4afdffb640 1 -- 172.21.15.46:0/4149552256 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 7879+0+0 (secure 0 0 0) 0x7f4af80f0b70 con 0x7f4b00103330 2024-03-31T23:09:10.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:10.711+0000 7f4b048fa640 1 --2- 172.21.15.46:0/4149552256 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f4af00a24a0 0x7f4af00a4960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:09:10.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:10.712+0000 7f4b048fa640 1 --2- 172.21.15.46:0/4149552256 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f4af00a24a0 0x7f4af00a4960 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f4af4005fd0 tx=0x7f4af400b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:09:10.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:10.715+0000 7f4afdffb640 1 -- 172.21.15.46:0/4149552256 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f4af804b020 con 0x7f4b00103330 2024-03-31T23:09:10.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:09:10 smithi046 ceph-mon[32498]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-31T23:09:10.862 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:09:10 smithi046 ceph-mon[32498]: from='client.? 172.21.15.46:0/1474954948' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-03-31T23:09:10.862 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:09:10 smithi046 ceph-mon[32498]: osdmap e83: 12 total, 12 up, 12 in 2024-03-31T23:09:10.862 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:09:10 smithi046 ceph-mon[32498]: from='client.? 172.21.15.46:0/735353621' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-03-31T23:09:10.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:09:10 smithi142 ceph-mon[35271]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-31T23:09:10.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:09:10 smithi142 ceph-mon[35271]: from='client.? 172.21.15.46:0/1474954948' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-03-31T23:09:10.944 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:09:10 smithi142 ceph-mon[35271]: osdmap e83: 12 total, 12 up, 12 in 2024-03-31T23:09:10.944 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:09:10 smithi142 ceph-mon[35271]: from='client.? 172.21.15.46:0/735353621' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-03-31T23:09:11.008 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:11.007+0000 7f4b06b5e640 1 -- 172.21.15.46:0/4149552256 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7f4b000623c0 con 0x7f4b00103330 2024-03-31T23:09:11.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:09:10 smithi162 ceph-mon[36237]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-31T23:09:11.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:09:10 smithi162 ceph-mon[36237]: from='client.? 172.21.15.46:0/1474954948' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-03-31T23:09:11.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:09:10 smithi162 ceph-mon[36237]: osdmap e83: 12 total, 12 up, 12 in 2024-03-31T23:09:11.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:09:10 smithi162 ceph-mon[36237]: from='client.? 172.21.15.46:0/735353621' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-03-31T23:09:11.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:11.594+0000 7f4afdffb640 1 -- 172.21.15.46:0/4149552256 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v14) ==== 172+0+0 (secure 0 0 0) 0x7f4af80bdee0 con 0x7f4b00103330 2024-03-31T23:09:11.595 INFO:teuthology.orchestra.run.smithi046.stderr:new fs with metadata pool 2 and data pool 3 2024-03-31T23:09:11.598 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:11.596+0000 7f4b06b5e640 1 -- 172.21.15.46:0/4149552256 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f4af00a24a0 msgr2=0x7f4af00a4960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:09:11.598 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:11.596+0000 7f4b06b5e640 1 --2- 172.21.15.46:0/4149552256 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f4af00a24a0 0x7f4af00a4960 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f4af4005fd0 tx=0x7f4af400b410 comp rx=0 tx=0).stop 2024-03-31T23:09:11.598 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:11.596+0000 7f4b06b5e640 1 -- 172.21.15.46:0/4149552256 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4b00103330 msgr2=0x7f4b0019ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:09:11.598 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:11.596+0000 7f4b06b5e640 1 --2- 172.21.15.46:0/4149552256 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4b00103330 0x7f4b0019ef20 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f4af802c700 tx=0x7f4af80328a0 comp rx=0 tx=0).stop 2024-03-31T23:09:11.598 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:11.596+0000 7f4b06b5e640 1 -- 172.21.15.46:0/4149552256 shutdown_connections 2024-03-31T23:09:11.598 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:11.596+0000 7f4b06b5e640 1 --2- 172.21.15.46:0/4149552256 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f4af00a24a0 0x7f4af00a4960 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:11.598 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:11.596+0000 7f4b06b5e640 1 --2- 172.21.15.46:0/4149552256 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4b00103330 0x7f4b0019ef20 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:11.598 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:11.596+0000 7f4b06b5e640 1 --2- 172.21.15.46:0/4149552256 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4b00102970 0x7f4b00197eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:11.598 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:11.596+0000 7f4b06b5e640 1 --2- 172.21.15.46:0/4149552256 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4b00101780 0x7f4b00197970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:11.598 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:11.596+0000 7f4b06b5e640 1 -- 172.21.15.46:0/4149552256 >> 172.21.15.46:0/4149552256 conn(0x7f4b000fccf0 msgr2=0x7f4b00105c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:09:11.598 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:11.596+0000 7f4b06b5e640 1 -- 172.21.15.46:0/4149552256 shutdown_connections 2024-03-31T23:09:11.598 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:11.596+0000 7f4b06b5e640 1 -- 172.21.15.46:0/4149552256 wait complete. 2024-03-31T23:09:11.613 DEBUG:tasks.cephfs.filesystem:EC profile is ['m=2', 'k=2', 'crush-failure-domain=osd'] 2024-03-31T23:09:11.613 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd erasure-code-profile set cephfs_data_ec m=2 k=2 crush-failure-domain=osd 2024-03-31T23:09:11.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:11.722+0000 7f7028cda640 1 -- 172.21.15.46:0/2270152281 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7024103330 msgr2=0x7f7024109bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:09:11.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:11.722+0000 7f7028cda640 1 --2- 172.21.15.46:0/2270152281 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7024103330 0x7f7024109bc0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f701c009ca0 tx=0x7f701c02c0f0 comp rx=0 tx=0).stop 2024-03-31T23:09:11.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:11.722+0000 7f7028cda640 1 -- 172.21.15.46:0/2270152281 shutdown_connections 2024-03-31T23:09:11.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:11.722+0000 7f7028cda640 1 --2- 172.21.15.46:0/2270152281 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7024103330 0x7f7024109bc0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:11.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:11.722+0000 7f7028cda640 1 --2- 172.21.15.46:0/2270152281 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7024102970 0x7f7024102df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:11.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:11.722+0000 7f7028cda640 1 --2- 172.21.15.46:0/2270152281 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7024101780 0x7f7024101ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:11.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:11.722+0000 7f7028cda640 1 -- 172.21.15.46:0/2270152281 >> 172.21.15.46:0/2270152281 conn(0x7f70240fcd10 msgr2=0x7f70240ff130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:09:11.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:11.722+0000 7f7028cda640 1 -- 172.21.15.46:0/2270152281 shutdown_connections 2024-03-31T23:09:11.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:11.722+0000 7f7028cda640 1 -- 172.21.15.46:0/2270152281 wait complete. 2024-03-31T23:09:11.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:11.722+0000 7f7028cda640 1 Processor -- start 2024-03-31T23:09:11.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:11.723+0000 7f7028cda640 1 -- start start 2024-03-31T23:09:11.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:11.723+0000 7f7028cda640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7024101780 0x7f7024197980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:09:11.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:11.723+0000 7f7028cda640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7024102970 0x7f7024197ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:09:11.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:11.723+0000 7f7028cda640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7024103330 0x7f702419ef30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:09:11.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:11.723+0000 7f7028cda640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f702410d310 con 0x7f7024101780 2024-03-31T23:09:11.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:11.723+0000 7f7028cda640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f702410d190 con 0x7f7024102970 2024-03-31T23:09:11.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:11.723+0000 7f7028cda640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f702410d490 con 0x7f7024103330 2024-03-31T23:09:11.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:11.723+0000 7f702259c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7024101780 0x7f7024197980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:09:11.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:11.723+0000 7f702259c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7024101780 0x7f7024197980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41104/0 (socket says 172.21.15.46:41104) 2024-03-31T23:09:11.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:11.723+0000 7f702259c640 1 -- 172.21.15.46:0/2588971857 learned_addr learned my addr 172.21.15.46:0/2588971857 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:09:11.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:11.723+0000 7f7022d9d640 1 --2- 172.21.15.46:0/2588971857 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7024103330 0x7f702419ef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:09:11.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:11.723+0000 7f7021d9b640 1 --2- 172.21.15.46:0/2588971857 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7024102970 0x7f7024197ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:09:11.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:11.723+0000 7f702259c640 1 -- 172.21.15.46:0/2588971857 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7024103330 msgr2=0x7f702419ef30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:09:11.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:11.723+0000 7f702259c640 1 --2- 172.21.15.46:0/2588971857 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7024103330 0x7f702419ef30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:11.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:11.723+0000 7f702259c640 1 -- 172.21.15.46:0/2588971857 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7024102970 msgr2=0x7f7024197ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:09:11.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:11.723+0000 7f702259c640 1 --2- 172.21.15.46:0/2588971857 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7024102970 0x7f7024197ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:11.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:11.723+0000 7f702259c640 1 -- 172.21.15.46:0/2588971857 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f702419f5a0 con 0x7f7024101780 2024-03-31T23:09:11.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:11.723+0000 7f702259c640 1 --2- 172.21.15.46:0/2588971857 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7024101780 0x7f7024197980 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f701400d770 tx=0x7f701400dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:09:11.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:11.723+0000 7f70137fe640 1 -- 172.21.15.46:0/2588971857 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7014017070 con 0x7f7024101780 2024-03-31T23:09:11.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:11.723+0000 7f70137fe640 1 -- 172.21.15.46:0/2588971857 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7014009c30 con 0x7f7024101780 2024-03-31T23:09:11.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:11.723+0000 7f7028cda640 1 -- 172.21.15.46:0/2588971857 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f702419f890 con 0x7f7024101780 2024-03-31T23:09:11.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:11.723+0000 7f70137fe640 1 -- 172.21.15.46:0/2588971857 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7014003d00 con 0x7f7024101780 2024-03-31T23:09:11.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:11.723+0000 7f7028cda640 1 -- 172.21.15.46:0/2588971857 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f702419fe60 con 0x7f7024101780 2024-03-31T23:09:11.728 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:11.724+0000 7f7028cda640 1 -- 172.21.15.46:0/2588971857 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7024108aa0 con 0x7f7024101780 2024-03-31T23:09:11.728 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:11.725+0000 7f70137fe640 1 -- 172.21.15.46:0/2588971857 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 16) ==== 137320+0+0 (secure 0 0 0) 0x7f7014003ea0 con 0x7f7024101780 2024-03-31T23:09:11.728 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:11.726+0000 7f70137fe640 1 --2- 172.21.15.46:0/2588971857 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f700c0a24a0 0x7f700c0a4960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:09:11.728 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:11.726+0000 7f7021d9b640 1 --2- 172.21.15.46:0/2588971857 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f700c0a24a0 0x7f700c0a4960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:09:11.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:11.726+0000 7f70137fe640 1 -- 172.21.15.46:0/2588971857 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7f70140ce3c0 con 0x7f7024101780 2024-03-31T23:09:11.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:11.726+0000 7f7021d9b640 1 --2- 172.21.15.46:0/2588971857 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f700c0a24a0 0x7f700c0a4960 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f7024198e90 tx=0x7f7018008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:09:11.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:11.731+0000 7f70137fe640 1 -- 172.21.15.46:0/2588971857 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f701409b6d0 con 0x7f7024101780 2024-03-31T23:09:11.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:09:11 smithi046 ceph-mon[32498]: from='client.? 172.21.15.46:0/735353621' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2024-03-31T23:09:11.862 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:09:11 smithi046 ceph-mon[32498]: pgmap v285: 97 pgs: 6 creating+peering, 1 active+clean, 90 unknown; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-31T23:09:11.862 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:09:11 smithi046 ceph-mon[32498]: osdmap e84: 12 total, 12 up, 12 in 2024-03-31T23:09:11.862 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:09:11 smithi046 ceph-mon[32498]: from='client.? 172.21.15.46:0/4149552256' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-03-31T23:09:11.862 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:09:11 smithi046 ceph-mon[32498]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-03-31T23:09:11.862 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:09:11 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:09:11.862 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:09:11 smithi046 ceph-mon[32498]: osdmap e85: 12 total, 12 up, 12 in 2024-03-31T23:09:11.862 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:09:11 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:09:11.862 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:09:11 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:09:11.862 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:09:11 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:09:11.862 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:09:11 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:09:11.862 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:09:11 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:09:11.862 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:09:11 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:09:11.863 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:09:11 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-31T23:09:11.863 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:09:11 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:09:11.863 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:09:11 smithi046 ceph-mon[32498]: Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-31T23:09:11.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:11.924+0000 7f7028cda640 1 -- 172.21.15.46:0/2588971857 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]} v 0) -- 0x7f702402d170 con 0x7f7024101780 2024-03-31T23:09:11.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:09:11 smithi142 ceph-mon[35271]: from='client.? 172.21.15.46:0/735353621' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2024-03-31T23:09:11.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:09:11 smithi142 ceph-mon[35271]: pgmap v285: 97 pgs: 6 creating+peering, 1 active+clean, 90 unknown; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-31T23:09:11.944 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:09:11 smithi142 ceph-mon[35271]: osdmap e84: 12 total, 12 up, 12 in 2024-03-31T23:09:11.944 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:09:11 smithi142 ceph-mon[35271]: from='client.? 172.21.15.46:0/4149552256' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-03-31T23:09:11.944 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:09:11 smithi142 ceph-mon[35271]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-03-31T23:09:11.944 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:09:11 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:09:11.944 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:09:11 smithi142 ceph-mon[35271]: osdmap e85: 12 total, 12 up, 12 in 2024-03-31T23:09:11.944 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:09:11 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:09:11.944 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:09:11 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:09:11.944 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:09:11 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:09:11.944 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:09:11 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:09:11.944 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:09:11 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:09:11.944 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:09:11 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:09:11.944 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:09:11 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-31T23:09:11.944 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:09:11 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:09:11.944 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:09:11 smithi142 ceph-mon[35271]: Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-31T23:09:12.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:09:11 smithi162 ceph-mon[36237]: from='client.? 172.21.15.46:0/735353621' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2024-03-31T23:09:12.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:09:11 smithi162 ceph-mon[36237]: pgmap v285: 97 pgs: 6 creating+peering, 1 active+clean, 90 unknown; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-31T23:09:12.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:09:11 smithi162 ceph-mon[36237]: osdmap e84: 12 total, 12 up, 12 in 2024-03-31T23:09:12.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:09:11 smithi162 ceph-mon[36237]: from='client.? 172.21.15.46:0/4149552256' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-03-31T23:09:12.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:09:11 smithi162 ceph-mon[36237]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-03-31T23:09:12.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:09:11 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:09:12.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:09:11 smithi162 ceph-mon[36237]: osdmap e85: 12 total, 12 up, 12 in 2024-03-31T23:09:12.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:09:11 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:09:12.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:09:11 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:09:12.047 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:09:11 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:09:12.047 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:09:11 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:09:12.047 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:09:11 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:09:12.047 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:09:11 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:09:12.047 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:09:11 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-31T23:09:12.047 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:09:11 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:09:12.047 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:09:11 smithi162 ceph-mon[36237]: Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-31T23:09:12.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:12.164+0000 7f70137fe640 1 -- 172.21.15.46:0/2588971857 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]=0 v86) ==== 157+0+0 (secure 0 0 0) 0x7f7014009de0 con 0x7f7024101780 2024-03-31T23:09:12.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:12.166+0000 7f7028cda640 1 -- 172.21.15.46:0/2588971857 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f700c0a24a0 msgr2=0x7f700c0a4960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:09:12.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:12.166+0000 7f7028cda640 1 --2- 172.21.15.46:0/2588971857 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f700c0a24a0 0x7f700c0a4960 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f7024198e90 tx=0x7f7018008040 comp rx=0 tx=0).stop 2024-03-31T23:09:12.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:12.166+0000 7f7028cda640 1 -- 172.21.15.46:0/2588971857 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7024101780 msgr2=0x7f7024197980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:09:12.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:12.166+0000 7f7028cda640 1 --2- 172.21.15.46:0/2588971857 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7024101780 0x7f7024197980 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f701400d770 tx=0x7f701400dc30 comp rx=0 tx=0).stop 2024-03-31T23:09:12.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:12.166+0000 7f7028cda640 1 -- 172.21.15.46:0/2588971857 shutdown_connections 2024-03-31T23:09:12.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:12.166+0000 7f7028cda640 1 --2- 172.21.15.46:0/2588971857 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f700c0a24a0 0x7f700c0a4960 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:12.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:12.166+0000 7f7028cda640 1 --2- 172.21.15.46:0/2588971857 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7024103330 0x7f702419ef30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:12.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:12.166+0000 7f7028cda640 1 --2- 172.21.15.46:0/2588971857 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7024102970 0x7f7024197ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:12.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:12.166+0000 7f7028cda640 1 --2- 172.21.15.46:0/2588971857 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7024101780 0x7f7024197980 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:12.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:12.167+0000 7f7028cda640 1 -- 172.21.15.46:0/2588971857 >> 172.21.15.46:0/2588971857 conn(0x7f70240fcd10 msgr2=0x7f7024105c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:09:12.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:12.167+0000 7f7028cda640 1 -- 172.21.15.46:0/2588971857 shutdown_connections 2024-03-31T23:09:12.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:12.167+0000 7f7028cda640 1 -- 172.21.15.46:0/2588971857 wait complete. 2024-03-31T23:09:12.179 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data_ec erasure cephfs_data_ec --pg_num_min 64 --target_size_ratio 0.9 2024-03-31T23:09:12.294 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:12.290+0000 7f7e72eb5640 1 -- 172.21.15.46:0/3472557482 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7e6c103320 msgr2=0x7f7e6c109bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:09:12.294 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:12.290+0000 7f7e72eb5640 1 --2- 172.21.15.46:0/3472557482 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7e6c103320 0x7f7e6c109bb0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f7e64009ca0 tx=0x7f7e6402c0f0 comp rx=0 tx=0).stop 2024-03-31T23:09:12.294 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:12.290+0000 7f7e72eb5640 1 -- 172.21.15.46:0/3472557482 shutdown_connections 2024-03-31T23:09:12.294 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:12.290+0000 7f7e72eb5640 1 --2- 172.21.15.46:0/3472557482 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7e6c103320 0x7f7e6c109bb0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:12.294 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:12.290+0000 7f7e72eb5640 1 --2- 172.21.15.46:0/3472557482 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7e6c102960 0x7f7e6c102de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:12.294 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:12.290+0000 7f7e72eb5640 1 --2- 172.21.15.46:0/3472557482 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7e6c101770 0x7f7e6c101b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:12.294 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:12.290+0000 7f7e72eb5640 1 -- 172.21.15.46:0/3472557482 >> 172.21.15.46:0/3472557482 conn(0x7f7e6c0fcd00 msgr2=0x7f7e6c0ff120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:09:12.294 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:12.290+0000 7f7e72eb5640 1 -- 172.21.15.46:0/3472557482 shutdown_connections 2024-03-31T23:09:12.294 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:12.290+0000 7f7e72eb5640 1 -- 172.21.15.46:0/3472557482 wait complete. 2024-03-31T23:09:12.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:12.290+0000 7f7e72eb5640 1 Processor -- start 2024-03-31T23:09:12.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:12.291+0000 7f7e72eb5640 1 -- start start 2024-03-31T23:09:12.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:12.291+0000 7f7e72eb5640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7e6c101770 0x7f7e6c1978f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:09:12.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:12.291+0000 7f7e72eb5640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7e6c102960 0x7f7e6c197e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:09:12.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:12.291+0000 7f7e72eb5640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7e6c103320 0x7f7e6c19eea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:09:12.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:12.291+0000 7f7e72eb5640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f7e6c10d280 con 0x7f7e6c102960 2024-03-31T23:09:12.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:12.291+0000 7f7e72eb5640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f7e6c10d100 con 0x7f7e6c101770 2024-03-31T23:09:12.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:12.291+0000 7f7e72eb5640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f7e6c10d400 con 0x7f7e6c103320 2024-03-31T23:09:12.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:12.291+0000 7f7e6bfff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7e6c102960 0x7f7e6c197e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:09:12.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:12.291+0000 7f7e6bfff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7e6c102960 0x7f7e6c197e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41122/0 (socket says 172.21.15.46:41122) 2024-03-31T23:09:12.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:12.291+0000 7f7e6bfff640 1 -- 172.21.15.46:0/4257473789 learned_addr learned my addr 172.21.15.46:0/4257473789 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:09:12.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:12.291+0000 7f7e71452640 1 --2- 172.21.15.46:0/4257473789 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7e6c103320 0x7f7e6c19eea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:09:12.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:12.291+0000 7f7e70c51640 1 --2- 172.21.15.46:0/4257473789 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7e6c101770 0x7f7e6c1978f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:09:12.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:12.291+0000 7f7e6bfff640 1 -- 172.21.15.46:0/4257473789 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7e6c103320 msgr2=0x7f7e6c19eea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:09:12.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:12.291+0000 7f7e6bfff640 1 --2- 172.21.15.46:0/4257473789 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7e6c103320 0x7f7e6c19eea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:12.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:12.291+0000 7f7e6bfff640 1 -- 172.21.15.46:0/4257473789 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7e6c101770 msgr2=0x7f7e6c1978f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:09:12.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:12.291+0000 7f7e6bfff640 1 --2- 172.21.15.46:0/4257473789 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7e6c101770 0x7f7e6c1978f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:12.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:12.291+0000 7f7e6bfff640 1 -- 172.21.15.46:0/4257473789 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e6c19f510 con 0x7f7e6c102960 2024-03-31T23:09:12.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:12.291+0000 7f7e70c51640 1 --2- 172.21.15.46:0/4257473789 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7e6c101770 0x7f7e6c1978f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T23:09:12.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:12.291+0000 7f7e6bfff640 1 --2- 172.21.15.46:0/4257473789 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7e6c102960 0x7f7e6c197e30 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f7e5800c920 tx=0x7f7e5800cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:09:12.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:12.291+0000 7f7e69ffb640 1 -- 172.21.15.46:0/4257473789 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e58007c70 con 0x7f7e6c102960 2024-03-31T23:09:12.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:12.291+0000 7f7e69ffb640 1 -- 172.21.15.46:0/4257473789 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7e58007e10 con 0x7f7e6c102960 2024-03-31T23:09:12.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:12.291+0000 7f7e69ffb640 1 -- 172.21.15.46:0/4257473789 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e58018820 con 0x7f7e6c102960 2024-03-31T23:09:12.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:12.291+0000 7f7e72eb5640 1 -- 172.21.15.46:0/4257473789 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7e6c19f800 con 0x7f7e6c102960 2024-03-31T23:09:12.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:12.292+0000 7f7e72eb5640 1 -- 172.21.15.46:0/4257473789 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7e6c19fd00 con 0x7f7e6c102960 2024-03-31T23:09:12.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:12.293+0000 7f7e72eb5640 1 -- 172.21.15.46:0/4257473789 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7e6c108a90 con 0x7f7e6c102960 2024-03-31T23:09:12.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:12.293+0000 7f7e69ffb640 1 -- 172.21.15.46:0/4257473789 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 16) ==== 137320+0+0 (secure 0 0 0) 0x7f7e580189c0 con 0x7f7e6c102960 2024-03-31T23:09:12.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:12.294+0000 7f7e69ffb640 1 --2- 172.21.15.46:0/4257473789 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f7e5c0a2520 0x7f7e5c0a49e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:09:12.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:12.294+0000 7f7e69ffb640 1 -- 172.21.15.46:0/4257473789 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 8277+0+0 (secure 0 0 0) 0x7f7e580ce930 con 0x7f7e6c102960 2024-03-31T23:09:12.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:12.294+0000 7f7e70c51640 1 --2- 172.21.15.46:0/4257473789 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f7e5c0a2520 0x7f7e5c0a49e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:09:12.298 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:12.295+0000 7f7e70c51640 1 --2- 172.21.15.46:0/4257473789 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f7e5c0a2520 0x7f7e5c0a49e0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f7e6c1027c0 tx=0x7f7e60005e70 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:09:12.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:12.298+0000 7f7e69ffb640 1 -- 172.21.15.46:0/4257473789 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f7e58014030 con 0x7f7e6c102960 2024-03-31T23:09:12.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:12.481+0000 7f7e72eb5640 1 -- 172.21.15.46:0/4257473789 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7f7e6c02d170 con 0x7f7e6c102960 2024-03-31T23:09:12.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:09:12 smithi046 ceph-mon[32498]: from='client.? 172.21.15.46:0/4149552256' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-03-31T23:09:12.862 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:09:12 smithi046 ceph-mon[32498]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-03-31T23:09:12.862 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:09:12 smithi046 ceph-mon[32498]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-03-31T23:09:12.862 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:09:12 smithi046 ceph-mon[32498]: from='client.? 172.21.15.46:0/2588971857' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-03-31T23:09:12.862 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:09:12 smithi046 ceph-mon[32498]: from='client.? 172.21.15.46:0/2588971857' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2024-03-31T23:09:12.862 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:09:12 smithi046 ceph-mon[32498]: osdmap e86: 12 total, 12 up, 12 in 2024-03-31T23:09:12.862 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:09:12 smithi046 ceph-mon[32498]: from='client.? 172.21.15.46:0/4257473789' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-03-31T23:09:12.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:09:12 smithi142 ceph-mon[35271]: from='client.? 172.21.15.46:0/4149552256' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-03-31T23:09:12.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:09:12 smithi142 ceph-mon[35271]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-03-31T23:09:12.944 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:09:12 smithi142 ceph-mon[35271]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-03-31T23:09:12.944 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:09:12 smithi142 ceph-mon[35271]: from='client.? 172.21.15.46:0/2588971857' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-03-31T23:09:12.944 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:09:12 smithi142 ceph-mon[35271]: from='client.? 172.21.15.46:0/2588971857' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2024-03-31T23:09:12.944 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:09:12 smithi142 ceph-mon[35271]: osdmap e86: 12 total, 12 up, 12 in 2024-03-31T23:09:12.944 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:09:12 smithi142 ceph-mon[35271]: from='client.? 172.21.15.46:0/4257473789' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-03-31T23:09:13.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:09:12 smithi162 ceph-mon[36237]: from='client.? 172.21.15.46:0/4149552256' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-03-31T23:09:13.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:09:12 smithi162 ceph-mon[36237]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-03-31T23:09:13.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:09:12 smithi162 ceph-mon[36237]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-03-31T23:09:13.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:09:12 smithi162 ceph-mon[36237]: from='client.? 172.21.15.46:0/2588971857' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-03-31T23:09:13.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:09:12 smithi162 ceph-mon[36237]: from='client.? 172.21.15.46:0/2588971857' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2024-03-31T23:09:13.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:09:12 smithi162 ceph-mon[36237]: osdmap e86: 12 total, 12 up, 12 in 2024-03-31T23:09:13.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:09:12 smithi162 ceph-mon[36237]: from='client.? 172.21.15.46:0/4257473789' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-03-31T23:09:13.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:09:13 smithi046 ceph-mon[32498]: pgmap v288: 97 pgs: 15 active+clean, 20 creating+peering, 62 unknown; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 511 B/s wr, 1 op/s 2024-03-31T23:09:13.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:09:13 smithi046 ceph-mon[32498]: mds.? [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] up:active 2024-03-31T23:09:13.862 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:09:13 smithi046 ceph-mon[32498]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-31T23:09:13.862 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:09:13 smithi046 ceph-mon[32498]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-31T23:09:13.862 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:09:13 smithi046 ceph-mon[32498]: Health check cleared: POOL_APP_NOT_ENABLED (was: 2 pool(s) do not have an application enabled) 2024-03-31T23:09:13.862 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:09:13 smithi046 ceph-mon[32498]: Cluster is now healthy 2024-03-31T23:09:13.862 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:09:13 smithi046 ceph-mon[32498]: osdmap e87: 12 total, 12 up, 12 in 2024-03-31T23:09:13.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:09:13 smithi142 ceph-mon[35271]: pgmap v288: 97 pgs: 15 active+clean, 20 creating+peering, 62 unknown; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 511 B/s wr, 1 op/s 2024-03-31T23:09:13.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:09:13 smithi142 ceph-mon[35271]: mds.? [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] up:active 2024-03-31T23:09:13.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:09:13 smithi142 ceph-mon[35271]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-31T23:09:13.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:09:13 smithi142 ceph-mon[35271]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-31T23:09:13.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:09:13 smithi142 ceph-mon[35271]: Health check cleared: POOL_APP_NOT_ENABLED (was: 2 pool(s) do not have an application enabled) 2024-03-31T23:09:13.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:09:13 smithi142 ceph-mon[35271]: Cluster is now healthy 2024-03-31T23:09:13.944 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:09:13 smithi142 ceph-mon[35271]: osdmap e87: 12 total, 12 up, 12 in 2024-03-31T23:09:14.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:09:13 smithi162 ceph-mon[36237]: pgmap v288: 97 pgs: 15 active+clean, 20 creating+peering, 62 unknown; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 511 B/s wr, 1 op/s 2024-03-31T23:09:14.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:09:13 smithi162 ceph-mon[36237]: mds.? [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] up:active 2024-03-31T23:09:14.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:09:13 smithi162 ceph-mon[36237]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-31T23:09:14.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:09:13 smithi162 ceph-mon[36237]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-31T23:09:14.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:09:13 smithi162 ceph-mon[36237]: Health check cleared: POOL_APP_NOT_ENABLED (was: 2 pool(s) do not have an application enabled) 2024-03-31T23:09:14.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:09:13 smithi162 ceph-mon[36237]: Cluster is now healthy 2024-03-31T23:09:14.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:09:13 smithi162 ceph-mon[36237]: osdmap e87: 12 total, 12 up, 12 in 2024-03-31T23:09:14.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:14.172+0000 7f7e69ffb640 1 -- 172.21.15.46:0/4257473789 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data_ec' created v88) ==== 228+0+0 (secure 0 0 0) 0x7f7e5809bac0 con 0x7f7e6c102960 2024-03-31T23:09:14.173 INFO:teuthology.orchestra.run.smithi046.stderr:pool 'cephfs_data_ec' created 2024-03-31T23:09:14.177 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:14.175+0000 7f7e72eb5640 1 -- 172.21.15.46:0/4257473789 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f7e5c0a2520 msgr2=0x7f7e5c0a49e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:09:14.177 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:14.175+0000 7f7e72eb5640 1 --2- 172.21.15.46:0/4257473789 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f7e5c0a2520 0x7f7e5c0a49e0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f7e6c1027c0 tx=0x7f7e60005e70 comp rx=0 tx=0).stop 2024-03-31T23:09:14.177 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:14.175+0000 7f7e72eb5640 1 -- 172.21.15.46:0/4257473789 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7e6c102960 msgr2=0x7f7e6c197e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:09:14.177 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:14.175+0000 7f7e72eb5640 1 --2- 172.21.15.46:0/4257473789 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7e6c102960 0x7f7e6c197e30 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f7e5800c920 tx=0x7f7e5800cde0 comp rx=0 tx=0).stop 2024-03-31T23:09:14.177 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:14.175+0000 7f7e72eb5640 1 -- 172.21.15.46:0/4257473789 shutdown_connections 2024-03-31T23:09:14.177 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:14.175+0000 7f7e72eb5640 1 --2- 172.21.15.46:0/4257473789 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f7e5c0a2520 0x7f7e5c0a49e0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:14.177 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:14.175+0000 7f7e72eb5640 1 --2- 172.21.15.46:0/4257473789 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7e6c103320 0x7f7e6c19eea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:14.177 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:14.175+0000 7f7e72eb5640 1 --2- 172.21.15.46:0/4257473789 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7e6c102960 0x7f7e6c197e30 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:14.177 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:14.175+0000 7f7e72eb5640 1 --2- 172.21.15.46:0/4257473789 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7e6c101770 0x7f7e6c1978f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:14.177 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:14.175+0000 7f7e72eb5640 1 -- 172.21.15.46:0/4257473789 >> 172.21.15.46:0/4257473789 conn(0x7f7e6c0fcd00 msgr2=0x7f7e6c105c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:09:14.177 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:14.176+0000 7f7e72eb5640 1 -- 172.21.15.46:0/4257473789 shutdown_connections 2024-03-31T23:09:14.177 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:14.176+0000 7f7e72eb5640 1 -- 172.21.15.46:0/4257473789 wait complete. 2024-03-31T23:09:14.195 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool set cephfs_data_ec allow_ec_overwrites true 2024-03-31T23:09:14.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:14.307+0000 7fb3a9ad4640 1 -- 172.21.15.46:0/1626962396 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb3a4100660 msgr2=0x7fb3a4100a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:09:14.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:14.307+0000 7fb3a9ad4640 1 --2- 172.21.15.46:0/1626962396 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb3a4100660 0x7fb3a4100a80 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fb394009900 tx=0x7fb39402bc30 comp rx=0 tx=0).stop 2024-03-31T23:09:14.311 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:14.307+0000 7fb3a9ad4640 1 -- 172.21.15.46:0/1626962396 shutdown_connections 2024-03-31T23:09:14.311 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:14.307+0000 7fb3a9ad4640 1 --2- 172.21.15.46:0/1626962396 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb3a4102210 0x7fb3a4108aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:14.311 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:14.307+0000 7fb3a9ad4640 1 --2- 172.21.15.46:0/1626962396 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb3a4101850 0x7fb3a4101cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:14.311 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:14.307+0000 7fb3a9ad4640 1 --2- 172.21.15.46:0/1626962396 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb3a4100660 0x7fb3a4100a80 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:14.311 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:14.307+0000 7fb3a9ad4640 1 -- 172.21.15.46:0/1626962396 >> 172.21.15.46:0/1626962396 conn(0x7fb3a40fbbf0 msgr2=0x7fb3a40fe010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:09:14.311 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:14.307+0000 7fb3a9ad4640 1 -- 172.21.15.46:0/1626962396 shutdown_connections 2024-03-31T23:09:14.311 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:14.307+0000 7fb3a9ad4640 1 -- 172.21.15.46:0/1626962396 wait complete. 2024-03-31T23:09:14.311 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:14.307+0000 7fb3a9ad4640 1 Processor -- start 2024-03-31T23:09:14.311 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:14.308+0000 7fb3a9ad4640 1 -- start start 2024-03-31T23:09:14.311 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:14.308+0000 7fb3a9ad4640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb3a4100660 0x7fb3a4197a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:09:14.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:14.308+0000 7fb3a9ad4640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb3a4101850 0x7fb3a4197f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:09:14.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:14.308+0000 7fb3a9ad4640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb3a4102210 0x7fb3a419efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:09:14.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:14.308+0000 7fb3a9ad4640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fb3a410c190 con 0x7fb3a4102210 2024-03-31T23:09:14.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:14.308+0000 7fb3a9ad4640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fb3a410c010 con 0x7fb3a4101850 2024-03-31T23:09:14.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:14.308+0000 7fb3a9ad4640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fb3a410c310 con 0x7fb3a4100660 2024-03-31T23:09:14.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:14.308+0000 7fb3a3fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb3a4102210 0x7fb3a419efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:09:14.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:14.308+0000 7fb3a3fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb3a4102210 0x7fb3a419efb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41140/0 (socket says 172.21.15.46:41140) 2024-03-31T23:09:14.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:14.308+0000 7fb3a3fff640 1 -- 172.21.15.46:0/2447156371 learned_addr learned my addr 172.21.15.46:0/2447156371 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:09:14.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:14.308+0000 7fb3a2ffd640 1 --2- 172.21.15.46:0/2447156371 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb3a4101850 0x7fb3a4197f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:09:14.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:14.308+0000 7fb3a37fe640 1 --2- 172.21.15.46:0/2447156371 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb3a4100660 0x7fb3a4197a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:09:14.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:14.308+0000 7fb3a3fff640 1 -- 172.21.15.46:0/2447156371 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb3a4100660 msgr2=0x7fb3a4197a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:09:14.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:14.308+0000 7fb3a3fff640 1 --2- 172.21.15.46:0/2447156371 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb3a4100660 0x7fb3a4197a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:14.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:14.308+0000 7fb3a3fff640 1 -- 172.21.15.46:0/2447156371 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb3a4101850 msgr2=0x7fb3a4197f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:09:14.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:14.308+0000 7fb3a3fff640 1 --2- 172.21.15.46:0/2447156371 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb3a4101850 0x7fb3a4197f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:14.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:14.308+0000 7fb3a3fff640 1 -- 172.21.15.46:0/2447156371 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb3a419f620 con 0x7fb3a4102210 2024-03-31T23:09:14.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:14.308+0000 7fb3a37fe640 1 --2- 172.21.15.46:0/2447156371 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb3a4100660 0x7fb3a4197a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T23:09:14.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:14.308+0000 7fb3a3fff640 1 --2- 172.21.15.46:0/2447156371 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb3a4102210 0x7fb3a419efb0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fb39c00cad0 tx=0x7fb39c00cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:09:14.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:14.309+0000 7fb3a0ff9640 1 -- 172.21.15.46:0/2447156371 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb39c00ae20 con 0x7fb3a4102210 2024-03-31T23:09:14.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:14.309+0000 7fb3a0ff9640 1 -- 172.21.15.46:0/2447156371 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb39c005020 con 0x7fb3a4102210 2024-03-31T23:09:14.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:14.309+0000 7fb3a0ff9640 1 -- 172.21.15.46:0/2447156371 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb39c00f6b0 con 0x7fb3a4102210 2024-03-31T23:09:14.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:14.309+0000 7fb3a9ad4640 1 -- 172.21.15.46:0/2447156371 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb3a419f910 con 0x7fb3a4102210 2024-03-31T23:09:14.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:14.309+0000 7fb3a9ad4640 1 -- 172.21.15.46:0/2447156371 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb3a419fe60 con 0x7fb3a4102210 2024-03-31T23:09:14.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:14.310+0000 7fb3a9ad4640 1 -- 172.21.15.46:0/2447156371 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb3a41a00f0 con 0x7fb3a4102210 2024-03-31T23:09:14.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:14.311+0000 7fb3a0ff9640 1 -- 172.21.15.46:0/2447156371 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 16) ==== 137320+0+0 (secure 0 0 0) 0x7fb39c0051d0 con 0x7fb3a4102210 2024-03-31T23:09:14.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:14.311+0000 7fb3a0ff9640 1 --2- 172.21.15.46:0/2447156371 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fb3900a24a0 0x7fb3900a4960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:09:14.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:14.312+0000 7fb3a37fe640 1 --2- 172.21.15.46:0/2447156371 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fb3900a24a0 0x7fb3900a4960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:09:14.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:14.312+0000 7fb3a0ff9640 1 -- 172.21.15.46:0/2447156371 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8770+0+0 (secure 0 0 0) 0x7fb39c0ced90 con 0x7fb3a4102210 2024-03-31T23:09:14.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:14.312+0000 7fb3a37fe640 1 --2- 172.21.15.46:0/2447156371 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fb3900a24a0 0x7fb3900a4960 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fb394006010 tx=0x7fb39402c7e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:09:14.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:14.315+0000 7fb3a0ff9640 1 -- 172.21.15.46:0/2447156371 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fb39c014030 con 0x7fb3a4102210 2024-03-31T23:09:14.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:14.497+0000 7fb3a9ad4640 1 -- 172.21.15.46:0/2447156371 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"} v 0) -- 0x7fb3a41a0550 con 0x7fb3a4102210 2024-03-31T23:09:15.180 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:15.179+0000 7fb3a0ff9640 1 -- 172.21.15.46:0/2447156371 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]=0 set pool 4 allow_ec_overwrites to true v89) ==== 169+0+0 (secure 0 0 0) 0x7fb39c09bcd0 con 0x7fb3a4102210 2024-03-31T23:09:15.180 INFO:teuthology.orchestra.run.smithi046.stderr:set pool 4 allow_ec_overwrites to true 2024-03-31T23:09:15.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:15.183+0000 7fb3a9ad4640 1 -- 172.21.15.46:0/2447156371 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fb3900a24a0 msgr2=0x7fb3900a4960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:09:15.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:15.183+0000 7fb3a9ad4640 1 --2- 172.21.15.46:0/2447156371 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fb3900a24a0 0x7fb3900a4960 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fb394006010 tx=0x7fb39402c7e0 comp rx=0 tx=0).stop 2024-03-31T23:09:15.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:15.183+0000 7fb3a9ad4640 1 -- 172.21.15.46:0/2447156371 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb3a4102210 msgr2=0x7fb3a419efb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:09:15.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:15.183+0000 7fb3a9ad4640 1 --2- 172.21.15.46:0/2447156371 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb3a4102210 0x7fb3a419efb0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fb39c00cad0 tx=0x7fb39c00cf90 comp rx=0 tx=0).stop 2024-03-31T23:09:15.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:15.183+0000 7fb3a9ad4640 1 -- 172.21.15.46:0/2447156371 shutdown_connections 2024-03-31T23:09:15.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:15.183+0000 7fb3a9ad4640 1 --2- 172.21.15.46:0/2447156371 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fb3900a24a0 0x7fb3900a4960 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:15.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:15.183+0000 7fb3a9ad4640 1 --2- 172.21.15.46:0/2447156371 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb3a4102210 0x7fb3a419efb0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:15.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:15.183+0000 7fb3a9ad4640 1 --2- 172.21.15.46:0/2447156371 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb3a4101850 0x7fb3a4197f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:15.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:15.183+0000 7fb3a9ad4640 1 --2- 172.21.15.46:0/2447156371 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb3a4100660 0x7fb3a4197a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:15.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:15.183+0000 7fb3a9ad4640 1 -- 172.21.15.46:0/2447156371 >> 172.21.15.46:0/2447156371 conn(0x7fb3a40fbbf0 msgr2=0x7fb3a4069d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:09:15.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:15.183+0000 7fb3a9ad4640 1 -- 172.21.15.46:0/2447156371 shutdown_connections 2024-03-31T23:09:15.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:15.183+0000 7fb3a9ad4640 1 -- 172.21.15.46:0/2447156371 wait complete. 2024-03-31T23:09:15.206 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs add_data_pool cephfs cephfs_data_ec 2024-03-31T23:09:15.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:15.315+0000 7f57e3040640 1 -- 172.21.15.46:0/1487303994 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f57dc1033c0 msgr2=0x7f57dc109c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:09:15.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:15.315+0000 7f57e3040640 1 --2- 172.21.15.46:0/1487303994 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f57dc1033c0 0x7f57dc109c50 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f57cc009930 tx=0x7f57cc02bc30 comp rx=0 tx=0).stop 2024-03-31T23:09:15.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:15.315+0000 7f57e3040640 1 -- 172.21.15.46:0/1487303994 shutdown_connections 2024-03-31T23:09:15.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:15.315+0000 7f57e3040640 1 --2- 172.21.15.46:0/1487303994 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f57dc1033c0 0x7f57dc109c50 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:15.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:15.315+0000 7f57e3040640 1 --2- 172.21.15.46:0/1487303994 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f57dc102a00 0x7f57dc102e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:15.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:15.315+0000 7f57e3040640 1 --2- 172.21.15.46:0/1487303994 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f57dc0ffa30 0x7f57dc0ffe50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:15.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:15.315+0000 7f57e3040640 1 -- 172.21.15.46:0/1487303994 >> 172.21.15.46:0/1487303994 conn(0x7f57dc0fb360 msgr2=0x7f57dc0fd7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:09:15.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:15.315+0000 7f57e3040640 1 -- 172.21.15.46:0/1487303994 shutdown_connections 2024-03-31T23:09:15.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:15.315+0000 7f57e3040640 1 -- 172.21.15.46:0/1487303994 wait complete. 2024-03-31T23:09:15.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:15.316+0000 7f57e3040640 1 Processor -- start 2024-03-31T23:09:15.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:15.316+0000 7f57e3040640 1 -- start start 2024-03-31T23:09:15.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:15.316+0000 7f57e3040640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f57dc0ffa30 0x7f57dc197940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:09:15.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:15.316+0000 7f57e3040640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f57dc102a00 0x7f57dc197e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:09:15.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:15.316+0000 7f57e3040640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f57dc1033c0 0x7f57dc19eef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:09:15.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:15.316+0000 7f57e3040640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f57dc10d2b0 con 0x7f57dc1033c0 2024-03-31T23:09:15.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:15.316+0000 7f57e3040640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f57dc10d130 con 0x7f57dc102a00 2024-03-31T23:09:15.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:15.316+0000 7f57e3040640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f57dc10d430 con 0x7f57dc0ffa30 2024-03-31T23:09:15.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:15.316+0000 7f57e15dd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f57dc1033c0 0x7f57dc19eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:09:15.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:15.316+0000 7f57e15dd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f57dc1033c0 0x7f57dc19eef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41168/0 (socket says 172.21.15.46:41168) 2024-03-31T23:09:15.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:15.316+0000 7f57e15dd640 1 -- 172.21.15.46:0/2264946362 learned_addr learned my addr 172.21.15.46:0/2264946362 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:09:15.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:15.316+0000 7f57dbfff640 1 --2- 172.21.15.46:0/2264946362 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f57dc102a00 0x7f57dc197e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:09:15.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:15.316+0000 7f57e0ddc640 1 --2- 172.21.15.46:0/2264946362 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f57dc0ffa30 0x7f57dc197940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:09:15.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:15.316+0000 7f57e15dd640 1 -- 172.21.15.46:0/2264946362 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f57dc0ffa30 msgr2=0x7f57dc197940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:09:15.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:15.316+0000 7f57e15dd640 1 --2- 172.21.15.46:0/2264946362 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f57dc0ffa30 0x7f57dc197940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:15.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:15.316+0000 7f57e15dd640 1 -- 172.21.15.46:0/2264946362 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f57dc102a00 msgr2=0x7f57dc197e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:09:15.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:15.316+0000 7f57e15dd640 1 --2- 172.21.15.46:0/2264946362 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f57dc102a00 0x7f57dc197e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:15.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:15.316+0000 7f57e15dd640 1 -- 172.21.15.46:0/2264946362 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f57dc19f560 con 0x7f57dc1033c0 2024-03-31T23:09:15.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:15.317+0000 7f57e15dd640 1 --2- 172.21.15.46:0/2264946362 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f57dc1033c0 0x7f57dc19eef0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f57cc009900 tx=0x7f57cc0040c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:09:15.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:15.317+0000 7f57d9ffb640 1 -- 172.21.15.46:0/2264946362 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57cc043020 con 0x7f57dc1033c0 2024-03-31T23:09:15.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:15.317+0000 7f57e3040640 1 -- 172.21.15.46:0/2264946362 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f57dc19f7f0 con 0x7f57dc1033c0 2024-03-31T23:09:15.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:15.317+0000 7f57d9ffb640 1 -- 172.21.15.46:0/2264946362 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f57cc0044c0 con 0x7f57dc1033c0 2024-03-31T23:09:15.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:15.317+0000 7f57d9ffb640 1 -- 172.21.15.46:0/2264946362 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57cc003c90 con 0x7f57dc1033c0 2024-03-31T23:09:15.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:15.317+0000 7f57e3040640 1 -- 172.21.15.46:0/2264946362 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f57dc19fd40 con 0x7f57dc1033c0 2024-03-31T23:09:15.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:15.318+0000 7f57e3040640 1 -- 172.21.15.46:0/2264946362 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f57dc108c00 con 0x7f57dc1033c0 2024-03-31T23:09:15.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:15.319+0000 7f57d9ffb640 1 -- 172.21.15.46:0/2264946362 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 16) ==== 137320+0+0 (secure 0 0 0) 0x7f57cc046020 con 0x7f57dc1033c0 2024-03-31T23:09:15.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:15.320+0000 7f57d9ffb640 1 --2- 172.21.15.46:0/2264946362 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f57c80a24a0 0x7f57c80a4960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:09:15.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:15.320+0000 7f57e0ddc640 1 --2- 172.21.15.46:0/2264946362 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f57c80a24a0 0x7f57c80a4960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:09:15.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:15.320+0000 7f57d9ffb640 1 -- 172.21.15.46:0/2264946362 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8770+0+0 (secure 0 0 0) 0x7f57cc0f1080 con 0x7f57dc1033c0 2024-03-31T23:09:15.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:15.320+0000 7f57e0ddc640 1 --2- 172.21.15.46:0/2264946362 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f57c80a24a0 0x7f57c80a4960 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f57dc069b40 tx=0x7f57d4009680 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:09:15.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:15.323+0000 7f57d9ffb640 1 -- 172.21.15.46:0/2264946362 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f57cc04d020 con 0x7f57dc1033c0 2024-03-31T23:09:15.443 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:09:15 smithi142 ceph-mon[35271]: from='client.? 172.21.15.46:0/4257473789' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-03-31T23:09:15.443 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:09:15 smithi142 ceph-mon[35271]: osdmap e88: 12 total, 12 up, 12 in 2024-03-31T23:09:15.443 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:09:15 smithi142 ceph-mon[35271]: from='client.? 172.21.15.46:0/2447156371' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-03-31T23:09:15.545 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:09:15 smithi162 ceph-mon[36237]: from='client.? 172.21.15.46:0/4257473789' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-03-31T23:09:15.546 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:09:15 smithi162 ceph-mon[36237]: osdmap e88: 12 total, 12 up, 12 in 2024-03-31T23:09:15.546 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:09:15 smithi162 ceph-mon[36237]: from='client.? 172.21.15.46:0/2447156371' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-03-31T23:09:15.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:15.555+0000 7f57e3040640 1 -- 172.21.15.46:0/2264946362 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"} v 0) -- 0x7f57dc02d170 con 0x7f57dc1033c0 2024-03-31T23:09:15.611 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:09:15 smithi046 ceph-mon[32498]: from='client.? 172.21.15.46:0/4257473789' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-03-31T23:09:15.611 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:09:15 smithi046 ceph-mon[32498]: osdmap e88: 12 total, 12 up, 12 in 2024-03-31T23:09:15.611 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:09:15 smithi046 ceph-mon[32498]: from='client.? 172.21.15.46:0/2447156371' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-03-31T23:09:16.380 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:16.379+0000 7f57d9ffb640 1 -- 172.21.15.46:0/2264946362 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]=0 added data pool 4 to fsmap v17) ==== 137+0+0 (secure 0 0 0) 0x7f57cc0bdfc0 con 0x7f57dc1033c0 2024-03-31T23:09:16.381 INFO:teuthology.orchestra.run.smithi046.stderr:added data pool 4 to fsmap 2024-03-31T23:09:16.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:16.382+0000 7f57e3040640 1 -- 172.21.15.46:0/2264946362 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f57c80a24a0 msgr2=0x7f57c80a4960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:09:16.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:16.382+0000 7f57e3040640 1 --2- 172.21.15.46:0/2264946362 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f57c80a24a0 0x7f57c80a4960 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f57dc069b40 tx=0x7f57d4009680 comp rx=0 tx=0).stop 2024-03-31T23:09:16.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:16.382+0000 7f57e3040640 1 -- 172.21.15.46:0/2264946362 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f57dc1033c0 msgr2=0x7f57dc19eef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:09:16.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:16.382+0000 7f57e3040640 1 --2- 172.21.15.46:0/2264946362 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f57dc1033c0 0x7f57dc19eef0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f57cc009900 tx=0x7f57cc0040c0 comp rx=0 tx=0).stop 2024-03-31T23:09:16.384 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:16.382+0000 7f57e3040640 1 -- 172.21.15.46:0/2264946362 shutdown_connections 2024-03-31T23:09:16.384 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:16.382+0000 7f57e3040640 1 --2- 172.21.15.46:0/2264946362 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f57c80a24a0 0x7f57c80a4960 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:16.384 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:16.382+0000 7f57e3040640 1 --2- 172.21.15.46:0/2264946362 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f57dc1033c0 0x7f57dc19eef0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:16.384 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:16.382+0000 7f57e3040640 1 --2- 172.21.15.46:0/2264946362 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f57dc102a00 0x7f57dc197e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:16.384 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:16.382+0000 7f57e3040640 1 --2- 172.21.15.46:0/2264946362 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f57dc0ffa30 0x7f57dc197940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:16.384 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:16.382+0000 7f57e3040640 1 -- 172.21.15.46:0/2264946362 >> 172.21.15.46:0/2264946362 conn(0x7f57dc0fb360 msgr2=0x7f57dc105dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:09:16.384 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:16.382+0000 7f57e3040640 1 -- 172.21.15.46:0/2264946362 shutdown_connections 2024-03-31T23:09:16.384 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:16.382+0000 7f57e3040640 1 -- 172.21.15.46:0/2264946362 wait complete. 2024-03-31T23:09:16.397 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-31T23:09:16.443 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:09:16 smithi142 ceph-mon[35271]: pgmap v291: 129 pgs: 60 active+clean, 20 creating+peering, 49 unknown; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s wr, 8 op/s 2024-03-31T23:09:16.443 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:09:16 smithi142 ceph-mon[35271]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-31T23:09:16.443 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:09:16 smithi142 ceph-mon[35271]: from='client.? 172.21.15.46:0/2447156371' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2024-03-31T23:09:16.443 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:09:16 smithi142 ceph-mon[35271]: osdmap e89: 12 total, 12 up, 12 in 2024-03-31T23:09:16.443 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:09:16 smithi142 ceph-mon[35271]: from='client.? 172.21.15.46:0/2264946362' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-03-31T23:09:16.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:16.504+0000 7f8c85bb5640 1 -- 172.21.15.46:0/3172377143 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8c80101860 msgr2=0x7f8c80101ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:09:16.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:16.504+0000 7f8c85bb5640 1 --2- 172.21.15.46:0/3172377143 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8c80101860 0x7f8c80101ce0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f8c74009900 tx=0x7f8c7402bc30 comp rx=0 tx=0).stop 2024-03-31T23:09:16.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:16.504+0000 7f8c85bb5640 1 -- 172.21.15.46:0/3172377143 shutdown_connections 2024-03-31T23:09:16.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:16.504+0000 7f8c85bb5640 1 --2- 172.21.15.46:0/3172377143 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8c80102220 0x7f8c80108ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:16.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:16.504+0000 7f8c85bb5640 1 --2- 172.21.15.46:0/3172377143 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8c80101860 0x7f8c80101ce0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:16.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:16.504+0000 7f8c85bb5640 1 --2- 172.21.15.46:0/3172377143 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8c80100670 0x7f8c80100a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:16.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:16.504+0000 7f8c85bb5640 1 -- 172.21.15.46:0/3172377143 >> 172.21.15.46:0/3172377143 conn(0x7f8c800fbbe0 msgr2=0x7f8c800fe020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:09:16.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:16.504+0000 7f8c85bb5640 1 -- 172.21.15.46:0/3172377143 shutdown_connections 2024-03-31T23:09:16.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:16.505+0000 7f8c85bb5640 1 -- 172.21.15.46:0/3172377143 wait complete. 2024-03-31T23:09:16.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:16.505+0000 7f8c85bb5640 1 Processor -- start 2024-03-31T23:09:16.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:16.505+0000 7f8c85bb5640 1 -- start start 2024-03-31T23:09:16.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:16.505+0000 7f8c85bb5640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8c80100670 0x7f8c80069440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:09:16.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:16.505+0000 7f8c85bb5640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8c80101860 0x7f8c8019c710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:09:16.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:16.505+0000 7f8c85bb5640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8c80102220 0x7f8c8019ed50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:09:16.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:16.505+0000 7f8c85bb5640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f8c8010c1b0 con 0x7f8c80100670 2024-03-31T23:09:16.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:16.505+0000 7f8c85bb5640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f8c8010c030 con 0x7f8c80101860 2024-03-31T23:09:16.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:16.505+0000 7f8c85bb5640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f8c8010c330 con 0x7f8c80102220 2024-03-31T23:09:16.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:16.505+0000 7f8c7f7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8c80100670 0x7f8c80069440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:09:16.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:16.506+0000 7f8c7f7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8c80100670 0x7f8c80069440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41194/0 (socket says 172.21.15.46:41194) 2024-03-31T23:09:16.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:16.506+0000 7f8c7f7fe640 1 -- 172.21.15.46:0/367195758 learned_addr learned my addr 172.21.15.46:0/367195758 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:09:16.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:16.506+0000 7f8c7effd640 1 --2- 172.21.15.46:0/367195758 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8c80101860 0x7f8c8019c710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:09:16.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:16.506+0000 7f8c7ffff640 1 --2- 172.21.15.46:0/367195758 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8c80102220 0x7f8c8019ed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:09:16.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:16.506+0000 7f8c7f7fe640 1 -- 172.21.15.46:0/367195758 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8c80102220 msgr2=0x7f8c8019ed50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:09:16.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:16.506+0000 7f8c7f7fe640 1 --2- 172.21.15.46:0/367195758 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8c80102220 0x7f8c8019ed50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:16.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:16.506+0000 7f8c7f7fe640 1 -- 172.21.15.46:0/367195758 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8c80101860 msgr2=0x7f8c8019c710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:09:16.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:16.506+0000 7f8c7f7fe640 1 --2- 172.21.15.46:0/367195758 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8c80101860 0x7f8c8019c710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:16.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:16.506+0000 7f8c7f7fe640 1 -- 172.21.15.46:0/367195758 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8c8019f610 con 0x7f8c80100670 2024-03-31T23:09:16.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:16.506+0000 7f8c7f7fe640 1 --2- 172.21.15.46:0/367195758 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8c80100670 0x7f8c80069440 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f8c7000e9c0 tx=0x7f8c7000ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:09:16.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:16.506+0000 7f8c7cff9640 1 -- 172.21.15.46:0/367195758 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c7000cd80 con 0x7f8c80100670 2024-03-31T23:09:16.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:16.506+0000 7f8c7cff9640 1 -- 172.21.15.46:0/367195758 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8c700035f0 con 0x7f8c80100670 2024-03-31T23:09:16.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:16.506+0000 7f8c85bb5640 1 -- 172.21.15.46:0/367195758 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8c8019f900 con 0x7f8c80100670 2024-03-31T23:09:16.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:16.506+0000 7f8c7cff9640 1 -- 172.21.15.46:0/367195758 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c70010690 con 0x7f8c80100670 2024-03-31T23:09:16.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:16.506+0000 7f8c85bb5640 1 -- 172.21.15.46:0/367195758 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8c8019fe20 con 0x7f8c80100670 2024-03-31T23:09:16.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:16.507+0000 7f8c85bb5640 1 -- 172.21.15.46:0/367195758 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8c801079b0 con 0x7f8c80100670 2024-03-31T23:09:16.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:16.508+0000 7f8c7cff9640 1 -- 172.21.15.46:0/367195758 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 16) ==== 137320+0+0 (secure 0 0 0) 0x7f8c700037a0 con 0x7f8c80100670 2024-03-31T23:09:16.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:16.509+0000 7f8c7cff9640 1 --2- 172.21.15.46:0/367195758 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f8c680a24a0 0x7f8c680a4960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:09:16.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:16.509+0000 7f8c7effd640 1 --2- 172.21.15.46:0/367195758 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f8c680a24a0 0x7f8c680a4960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:09:16.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:16.509+0000 7f8c7cff9640 1 -- 172.21.15.46:0/367195758 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7f8c700ced90 con 0x7f8c80100670 2024-03-31T23:09:16.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:16.510+0000 7f8c7effd640 1 --2- 172.21.15.46:0/367195758 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f8c680a24a0 0x7f8c680a4960 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f8c74009230 tx=0x7f8c74005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:09:16.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:16.514+0000 7f8c7cff9640 1 -- 172.21.15.46:0/367195758 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f8c7009bd10 con 0x7f8c80100670 2024-03-31T23:09:16.546 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:09:16 smithi162 ceph-mon[36237]: pgmap v291: 129 pgs: 60 active+clean, 20 creating+peering, 49 unknown; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s wr, 8 op/s 2024-03-31T23:09:16.546 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:09:16 smithi162 ceph-mon[36237]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-31T23:09:16.546 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:09:16 smithi162 ceph-mon[36237]: from='client.? 172.21.15.46:0/2447156371' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2024-03-31T23:09:16.546 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:09:16 smithi162 ceph-mon[36237]: osdmap e89: 12 total, 12 up, 12 in 2024-03-31T23:09:16.546 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:09:16 smithi162 ceph-mon[36237]: from='client.? 172.21.15.46:0/2264946362' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-03-31T23:09:16.611 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:09:16 smithi046 ceph-mon[32498]: pgmap v291: 129 pgs: 60 active+clean, 20 creating+peering, 49 unknown; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s wr, 8 op/s 2024-03-31T23:09:16.611 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:09:16 smithi046 ceph-mon[32498]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-31T23:09:16.611 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:09:16 smithi046 ceph-mon[32498]: from='client.? 172.21.15.46:0/2447156371' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2024-03-31T23:09:16.612 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:09:16 smithi046 ceph-mon[32498]: osdmap e89: 12 total, 12 up, 12 in 2024-03-31T23:09:16.612 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:09:16 smithi046 ceph-mon[32498]: from='client.? 172.21.15.46:0/2264946362' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-03-31T23:09:16.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:16.735+0000 7f8c85bb5640 1 -- 172.21.15.46:0/367195758 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8c80062350 con 0x7f8c80100670 2024-03-31T23:09:16.736 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:16.735+0000 7f8c7cff9640 1 -- 172.21.15.46:0/367195758 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) ==== 94+0+11809 (secure 0 0 0) 0x7f8c7009b610 con 0x7f8c80100670 2024-03-31T23:09:16.736 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-31T23:09:16.737 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":17,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/2169018547","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":2169018547},{"type":"v1","addr":"172.21.15.46:6835","nonce":2169018547}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6837/1069387756","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6836","nonce":1069387756},{"type":"v1","addr":"172.21.15.46:6837","nonce":1069387756}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14634,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6839/3915337844","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6838","nonce":3915337844},{"type":"v1","addr":"172.21.15.46:6839","nonce":3915337844}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14646,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6841/897767145","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6840","nonce":897767145},{"type":"v1","addr":"172.21.15.46:6841","nonce":897767145}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24415,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6833/1746950966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6832","nonce":1746950966},{"type":"v1","addr":"172.21.15.162:6833","nonce":1746950966}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24422,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6833/3394739181","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6832","nonce":3394739181},{"type":"v1","addr":"172.21.15.142:6833","nonce":3394739181}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24442,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6835/1957208984","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6834","nonce":1957208984},{"type":"v1","addr":"172.21.15.162:6835","nonce":1957208984}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24451,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6837/1371886876","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6836","nonce":1371886876},{"type":"v1","addr":"172.21.15.162:6837","nonce":1371886876}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24455,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6835/212637431","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6834","nonce":212637431},{"type":"v1","addr":"172.21.15.142:6835","nonce":212637431}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24478,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6839/1976275826","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6838","nonce":1976275826},{"type":"v1","addr":"172.21.15.162:6839","nonce":1976275826}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24482,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6837/2559258399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6836","nonce":2559258399},{"type":"v1","addr":"172.21.15.142:6837","nonce":2559258399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-31T23:09:11.009090+0000","modified":"2024-03-31T23:09:15.556861+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24515},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24515":{"gid":24515,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.142:6839/2519143918","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6838","nonce":2519143918},{"type":"v1","addr":"172.21.15.142:6839","nonce":2519143918}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":24515,"qdb_cluster":[24515]},"id":1}]} 2024-03-31T23:09:16.737 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 17 2024-03-31T23:09:16.738 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:16.737+0000 7f8c85bb5640 1 -- 172.21.15.46:0/367195758 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f8c680a24a0 msgr2=0x7f8c680a4960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:09:16.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:16.737+0000 7f8c85bb5640 1 --2- 172.21.15.46:0/367195758 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f8c680a24a0 0x7f8c680a4960 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f8c74009230 tx=0x7f8c74005fd0 comp rx=0 tx=0).stop 2024-03-31T23:09:16.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:16.737+0000 7f8c85bb5640 1 -- 172.21.15.46:0/367195758 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8c80100670 msgr2=0x7f8c80069440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:09:16.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:16.737+0000 7f8c85bb5640 1 --2- 172.21.15.46:0/367195758 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8c80100670 0x7f8c80069440 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f8c7000e9c0 tx=0x7f8c7000ee80 comp rx=0 tx=0).stop 2024-03-31T23:09:16.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:16.737+0000 7f8c85bb5640 1 -- 172.21.15.46:0/367195758 shutdown_connections 2024-03-31T23:09:16.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:16.737+0000 7f8c85bb5640 1 --2- 172.21.15.46:0/367195758 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f8c680a24a0 0x7f8c680a4960 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:16.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:16.737+0000 7f8c85bb5640 1 --2- 172.21.15.46:0/367195758 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8c80102220 0x7f8c8019ed50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:16.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:16.737+0000 7f8c85bb5640 1 --2- 172.21.15.46:0/367195758 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8c80101860 0x7f8c8019c710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:16.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:16.737+0000 7f8c85bb5640 1 --2- 172.21.15.46:0/367195758 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8c80100670 0x7f8c80069440 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:16.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:16.737+0000 7f8c85bb5640 1 -- 172.21.15.46:0/367195758 >> 172.21.15.46:0/367195758 conn(0x7f8c800fbbe0 msgr2=0x7f8c80104b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:09:16.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:16.737+0000 7f8c85bb5640 1 -- 172.21.15.46:0/367195758 shutdown_connections 2024-03-31T23:09:16.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:16.737+0000 7f8c85bb5640 1 -- 172.21.15.46:0/367195758 wait complete. 2024-03-31T23:09:16.750 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-31T23:09:16.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:16.851+0000 7f65315c7640 1 -- 172.21.15.46:0/3337188442 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f652c101880 msgr2=0x7f652c101d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:09:16.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:16.851+0000 7f65315c7640 1 --2- 172.21.15.46:0/3337188442 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f652c101880 0x7f652c101d00 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f6514009900 tx=0x7f651402bc30 comp rx=0 tx=0).stop 2024-03-31T23:09:16.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:16.851+0000 7f65315c7640 1 -- 172.21.15.46:0/3337188442 shutdown_connections 2024-03-31T23:09:16.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:16.851+0000 7f65315c7640 1 --2- 172.21.15.46:0/3337188442 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f652c102240 0x7f652c108ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:16.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:16.851+0000 7f65315c7640 1 --2- 172.21.15.46:0/3337188442 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f652c101880 0x7f652c101d00 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:16.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:16.852+0000 7f65315c7640 1 --2- 172.21.15.46:0/3337188442 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f652c100690 0x7f652c100ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:16.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:16.852+0000 7f65315c7640 1 -- 172.21.15.46:0/3337188442 >> 172.21.15.46:0/3337188442 conn(0x7f652c0fbc00 msgr2=0x7f652c0fe040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:09:16.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:16.852+0000 7f65315c7640 1 -- 172.21.15.46:0/3337188442 shutdown_connections 2024-03-31T23:09:16.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:16.852+0000 7f65315c7640 1 -- 172.21.15.46:0/3337188442 wait complete. 2024-03-31T23:09:16.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:16.852+0000 7f65315c7640 1 Processor -- start 2024-03-31T23:09:16.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:16.852+0000 7f65315c7640 1 -- start start 2024-03-31T23:09:16.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:16.852+0000 7f65315c7640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f652c100690 0x7f652c10b010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:09:16.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:16.852+0000 7f65315c7640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f652c101880 0x7f652c109650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:09:16.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:16.852+0000 7f65315c7640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f652c102240 0x7f652c109b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:09:16.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:16.852+0000 7f65315c7640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f652c10d2e0 con 0x7f652c102240 2024-03-31T23:09:16.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:16.852+0000 7f65315c7640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f652c10d160 con 0x7f652c100690 2024-03-31T23:09:16.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:16.852+0000 7f65315c7640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f652c10d460 con 0x7f652c101880 2024-03-31T23:09:16.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:16.852+0000 7f652b7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f652c102240 0x7f652c109b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:09:16.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:16.852+0000 7f652b7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f652c102240 0x7f652c109b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41208/0 (socket says 172.21.15.46:41208) 2024-03-31T23:09:16.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:16.852+0000 7f652b7fe640 1 -- 172.21.15.46:0/3599472486 learned_addr learned my addr 172.21.15.46:0/3599472486 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:09:16.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:16.852+0000 7f652affd640 1 --2- 172.21.15.46:0/3599472486 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f652c100690 0x7f652c10b010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:09:16.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:16.853+0000 7f6523fff640 1 --2- 172.21.15.46:0/3599472486 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f652c101880 0x7f652c109650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:09:16.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:16.853+0000 7f652b7fe640 1 -- 172.21.15.46:0/3599472486 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f652c101880 msgr2=0x7f652c109650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:09:16.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:16.853+0000 7f652b7fe640 1 --2- 172.21.15.46:0/3599472486 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f652c101880 0x7f652c109650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:16.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:16.853+0000 7f652b7fe640 1 -- 172.21.15.46:0/3599472486 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f652c100690 msgr2=0x7f652c10b010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:09:16.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:16.853+0000 7f652b7fe640 1 --2- 172.21.15.46:0/3599472486 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f652c100690 0x7f652c10b010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:16.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:16.853+0000 7f652b7fe640 1 -- 172.21.15.46:0/3599472486 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f652c10a420 con 0x7f652c102240 2024-03-31T23:09:16.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:16.853+0000 7f652b7fe640 1 --2- 172.21.15.46:0/3599472486 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f652c102240 0x7f652c109b90 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f6524009fd0 tx=0x7f652400b260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:09:16.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:16.853+0000 7f6528ff9640 1 -- 172.21.15.46:0/3599472486 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6524017070 con 0x7f652c102240 2024-03-31T23:09:16.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:16.853+0000 7f6528ff9640 1 -- 172.21.15.46:0/3599472486 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6524003e20 con 0x7f652c102240 2024-03-31T23:09:16.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:16.853+0000 7f65315c7640 1 -- 172.21.15.46:0/3599472486 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f652c1a1ac0 con 0x7f652c102240 2024-03-31T23:09:16.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:16.853+0000 7f6528ff9640 1 -- 172.21.15.46:0/3599472486 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65240106c0 con 0x7f652c102240 2024-03-31T23:09:16.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:16.853+0000 7f65315c7640 1 -- 172.21.15.46:0/3599472486 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f652c1a1fe0 con 0x7f652c102240 2024-03-31T23:09:16.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:16.854+0000 7f65315c7640 1 -- 172.21.15.46:0/3599472486 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f652c107a30 con 0x7f652c102240 2024-03-31T23:09:16.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:16.855+0000 7f6528ff9640 1 -- 172.21.15.46:0/3599472486 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 16) ==== 137320+0+0 (secure 0 0 0) 0x7f6524003970 con 0x7f652c102240 2024-03-31T23:09:16.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:16.856+0000 7f6528ff9640 1 --2- 172.21.15.46:0/3599472486 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f65180a2520 0x7f65180a49e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:09:16.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:16.856+0000 7f6528ff9640 1 -- 172.21.15.46:0/3599472486 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7f65240ce550 con 0x7f652c102240 2024-03-31T23:09:16.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:16.856+0000 7f652affd640 1 --2- 172.21.15.46:0/3599472486 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f65180a2520 0x7f65180a49e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:09:16.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:16.856+0000 7f652affd640 1 --2- 172.21.15.46:0/3599472486 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f65180a2520 0x7f65180a49e0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f652c1016e0 tx=0x7f651c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:09:16.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:16.860+0000 7f6528ff9640 1 -- 172.21.15.46:0/3599472486 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f65240047e0 con 0x7f652c102240 2024-03-31T23:09:17.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:17.045+0000 7f65315c7640 1 -- 172.21.15.46:0/3599472486 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f652c02d1c0 con 0x7f652c102240 2024-03-31T23:09:17.047 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:17.046+0000 7f6528ff9640 1 -- 172.21.15.46:0/3599472486 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) ==== 74+0+24118 (secure 0 0 0) 0x7f652409ae80 con 0x7f652c102240 2024-03-31T23:09:17.048 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-31T23:09:17.048 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":90,"fsid":"6a9d8f08-efb2-11ee-b647-cb9ed24678a4","created":"2024-03-31T23:01:15.463711+0000","modified":"2024-03-31T23:09:15.556670+0000","last_up_change":"2024-03-31T23:08:20.199888+0000","last_in_change":"2024-03-31T23:07:59.438035+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-31T23:04:32.823751+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-31T23:09:09.145711+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-31T23:09:10.148579+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-31T23:09:13.419519+0000","flags":32773,"flags_names":"hashpspool,ec_overwrites,creating","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"47429095-3f46-4ed7-b7c4-504464719815","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":1824772695},{"type":"v1","addr":"172.21.15.46:6803","nonce":1824772695}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":1824772695},{"type":"v1","addr":"172.21.15.46:6805","nonce":1824772695}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":1824772695},{"type":"v1","addr":"172.21.15.46:6809","nonce":1824772695}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":1824772695},{"type":"v1","addr":"172.21.15.46:6807","nonce":1824772695}]},"public_addr":"172.21.15.46:6803/1824772695","cluster_addr":"172.21.15.46:6805/1824772695","heartbeat_back_addr":"172.21.15.46:6809/1824772695","heartbeat_front_addr":"172.21.15.46:6807/1824772695","state":["exists","up"]},{"osd":1,"uuid":"be56f883-411e-4dbe-b50e-a158134df4a6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6800","nonce":973693922},{"type":"v1","addr":"172.21.15.142:6801","nonce":973693922}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6802","nonce":973693922},{"type":"v1","addr":"172.21.15.142:6803","nonce":973693922}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6806","nonce":973693922},{"type":"v1","addr":"172.21.15.142:6807","nonce":973693922}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6804","nonce":973693922},{"type":"v1","addr":"172.21.15.142:6805","nonce":973693922}]},"public_addr":"172.21.15.142:6801/973693922","cluster_addr":"172.21.15.142:6803/973693922","heartbeat_back_addr":"172.21.15.142:6807/973693922","heartbeat_front_addr":"172.21.15.142:6805/973693922","state":["exists","up"]},{"osd":2,"uuid":"c99a8794-ab19-4260-9a8d-4bc0bd2c8175","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6800","nonce":4084261813},{"type":"v1","addr":"172.21.15.162:6801","nonce":4084261813}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6802","nonce":4084261813},{"type":"v1","addr":"172.21.15.162:6803","nonce":4084261813}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6806","nonce":4084261813},{"type":"v1","addr":"172.21.15.162:6807","nonce":4084261813}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6804","nonce":4084261813},{"type":"v1","addr":"172.21.15.162:6805","nonce":4084261813}]},"public_addr":"172.21.15.162:6801/4084261813","cluster_addr":"172.21.15.162:6803/4084261813","heartbeat_back_addr":"172.21.15.162:6807/4084261813","heartbeat_front_addr":"172.21.15.162:6805/4084261813","state":["exists","up"]},{"osd":3,"uuid":"f877814e-9ef6-4872-a081-5609bb9ebc06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":3496317970},{"type":"v1","addr":"172.21.15.46:6811","nonce":3496317970}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":3496317970},{"type":"v1","addr":"172.21.15.46:6813","nonce":3496317970}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":3496317970},{"type":"v1","addr":"172.21.15.46:6817","nonce":3496317970}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":3496317970},{"type":"v1","addr":"172.21.15.46:6815","nonce":3496317970}]},"public_addr":"172.21.15.46:6811/3496317970","cluster_addr":"172.21.15.46:6813/3496317970","heartbeat_back_addr":"172.21.15.46:6817/3496317970","heartbeat_front_addr":"172.21.15.46:6815/3496317970","state":["exists","up"]},{"osd":4,"uuid":"beecec0a-8280-44fe-8f5f-8498a35eca8f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6808","nonce":2996266929},{"type":"v1","addr":"172.21.15.142:6809","nonce":2996266929}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6810","nonce":2996266929},{"type":"v1","addr":"172.21.15.142:6811","nonce":2996266929}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6814","nonce":2996266929},{"type":"v1","addr":"172.21.15.142:6815","nonce":2996266929}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6812","nonce":2996266929},{"type":"v1","addr":"172.21.15.142:6813","nonce":2996266929}]},"public_addr":"172.21.15.142:6809/2996266929","cluster_addr":"172.21.15.142:6811/2996266929","heartbeat_back_addr":"172.21.15.142:6815/2996266929","heartbeat_front_addr":"172.21.15.142:6813/2996266929","state":["exists","up"]},{"osd":5,"uuid":"9da7ff75-f658-45ea-a02f-1943fdda76b9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6808","nonce":914208293},{"type":"v1","addr":"172.21.15.162:6809","nonce":914208293}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6810","nonce":914208293},{"type":"v1","addr":"172.21.15.162:6811","nonce":914208293}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6814","nonce":914208293},{"type":"v1","addr":"172.21.15.162:6815","nonce":914208293}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6812","nonce":914208293},{"type":"v1","addr":"172.21.15.162:6813","nonce":914208293}]},"public_addr":"172.21.15.162:6809/914208293","cluster_addr":"172.21.15.162:6811/914208293","heartbeat_back_addr":"172.21.15.162:6815/914208293","heartbeat_front_addr":"172.21.15.162:6813/914208293","state":["exists","up"]},{"osd":6,"uuid":"8fa11224-cb26-4d3f-87d1-6a8f31de2a19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":56291966},{"type":"v1","addr":"172.21.15.46:6819","nonce":56291966}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":56291966},{"type":"v1","addr":"172.21.15.46:6821","nonce":56291966}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6824","nonce":56291966},{"type":"v1","addr":"172.21.15.46:6825","nonce":56291966}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":56291966},{"type":"v1","addr":"172.21.15.46:6823","nonce":56291966}]},"public_addr":"172.21.15.46:6819/56291966","cluster_addr":"172.21.15.46:6821/56291966","heartbeat_back_addr":"172.21.15.46:6825/56291966","heartbeat_front_addr":"172.21.15.46:6823/56291966","state":["exists","up"]},{"osd":7,"uuid":"0d0338b7-4d0a-4bb0-b152-8f5224871d9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6816","nonce":2146598446},{"type":"v1","addr":"172.21.15.142:6817","nonce":2146598446}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6818","nonce":2146598446},{"type":"v1","addr":"172.21.15.142:6819","nonce":2146598446}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6822","nonce":2146598446},{"type":"v1","addr":"172.21.15.142:6823","nonce":2146598446}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6820","nonce":2146598446},{"type":"v1","addr":"172.21.15.142:6821","nonce":2146598446}]},"public_addr":"172.21.15.142:6817/2146598446","cluster_addr":"172.21.15.142:6819/2146598446","heartbeat_back_addr":"172.21.15.142:6823/2146598446","heartbeat_front_addr":"172.21.15.142:6821/2146598446","state":["exists","up"]},{"osd":8,"uuid":"8b6f6037-892a-416c-babb-7552648d0b3a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6816","nonce":1173078318},{"type":"v1","addr":"172.21.15.162:6817","nonce":1173078318}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6818","nonce":1173078318},{"type":"v1","addr":"172.21.15.162:6819","nonce":1173078318}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6822","nonce":1173078318},{"type":"v1","addr":"172.21.15.162:6823","nonce":1173078318}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6820","nonce":1173078318},{"type":"v1","addr":"172.21.15.162:6821","nonce":1173078318}]},"public_addr":"172.21.15.162:6817/1173078318","cluster_addr":"172.21.15.162:6819/1173078318","heartbeat_back_addr":"172.21.15.162:6823/1173078318","heartbeat_front_addr":"172.21.15.162:6821/1173078318","state":["exists","up"]},{"osd":9,"uuid":"faccce25-6bf2-4220-8858-cda82b72be68","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":532708881},{"type":"v1","addr":"172.21.15.46:6827","nonce":532708881}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6828","nonce":532708881},{"type":"v1","addr":"172.21.15.46:6829","nonce":532708881}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":532708881},{"type":"v1","addr":"172.21.15.46:6833","nonce":532708881}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":532708881},{"type":"v1","addr":"172.21.15.46:6831","nonce":532708881}]},"public_addr":"172.21.15.46:6827/532708881","cluster_addr":"172.21.15.46:6829/532708881","heartbeat_back_addr":"172.21.15.46:6833/532708881","heartbeat_front_addr":"172.21.15.46:6831/532708881","state":["exists","up"]},{"osd":10,"uuid":"cdd356db-f9eb-4080-95ab-db5d8223e572","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6824","nonce":2845222366},{"type":"v1","addr":"172.21.15.142:6825","nonce":2845222366}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6826","nonce":2845222366},{"type":"v1","addr":"172.21.15.142:6827","nonce":2845222366}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6830","nonce":2845222366},{"type":"v1","addr":"172.21.15.142:6831","nonce":2845222366}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6828","nonce":2845222366},{"type":"v1","addr":"172.21.15.142:6829","nonce":2845222366}]},"public_addr":"172.21.15.142:6825/2845222366","cluster_addr":"172.21.15.142:6827/2845222366","heartbeat_back_addr":"172.21.15.142:6831/2845222366","heartbeat_front_addr":"172.21.15.142:6829/2845222366","state":["exists","up"]},{"osd":11,"uuid":"72104d1c-3ea2-45ad-bc0b-0500f146c06b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6824","nonce":1986323778},{"type":"v1","addr":"172.21.15.162:6825","nonce":1986323778}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6826","nonce":1986323778},{"type":"v1","addr":"172.21.15.162:6827","nonce":1986323778}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6830","nonce":1986323778},{"type":"v1","addr":"172.21.15.162:6831","nonce":1986323778}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6828","nonce":1986323778},{"type":"v1","addr":"172.21.15.162:6829","nonce":1986323778}]},"public_addr":"172.21.15.162:6825/1986323778","cluster_addr":"172.21.15.162:6827/1986323778","heartbeat_back_addr":"172.21.15.162:6831/1986323778","heartbeat_front_addr":"172.21.15.162:6829/1986323778","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:03:44.996331+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:04:07.994738+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:04:30.942255+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:04:55.588884+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:05:19.900600+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:05:43.807442+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:06:09.026976+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:06:34.586784+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:06:59.751605+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:07:26.144495+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:07:52.326909+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:08:17.423042+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.46:0/4021039857":"2024-04-01T23:02:08.493619+0000","172.21.15.46:6801/3681182647":"2024-04-01T23:02:08.493619+0000","172.21.15.46:0/4022214806":"2024-04-01T23:02:08.493619+0000","172.21.15.46:6800/1515528215":"2024-04-01T23:01:41.910162+0000","172.21.15.46:6800/3681182647":"2024-04-01T23:02:08.493619+0000","172.21.15.46:0/143885707":"2024-04-01T23:01:41.910162+0000","172.21.15.46:6801/1515528215":"2024-04-01T23:01:41.910162+0000","172.21.15.46:0/1680389778":"2024-04-01T23:01:41.910162+0000","172.21.15.46:0/1782133966":"2024-04-01T23:01:41.910162+0000","172.21.15.46:0/3925209376":"2024-04-01T23:02:08.493619+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-31T23:09:17.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:17.048+0000 7f65315c7640 1 -- 172.21.15.46:0/3599472486 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f65180a2520 msgr2=0x7f65180a49e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:09:17.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:17.048+0000 7f65315c7640 1 --2- 172.21.15.46:0/3599472486 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f65180a2520 0x7f65180a49e0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f652c1016e0 tx=0x7f651c005f90 comp rx=0 tx=0).stop 2024-03-31T23:09:17.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:17.048+0000 7f65315c7640 1 -- 172.21.15.46:0/3599472486 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f652c102240 msgr2=0x7f652c109b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:09:17.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:17.048+0000 7f65315c7640 1 --2- 172.21.15.46:0/3599472486 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f652c102240 0x7f652c109b90 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f6524009fd0 tx=0x7f652400b260 comp rx=0 tx=0).stop 2024-03-31T23:09:17.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:17.048+0000 7f65315c7640 1 -- 172.21.15.46:0/3599472486 shutdown_connections 2024-03-31T23:09:17.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:17.048+0000 7f65315c7640 1 --2- 172.21.15.46:0/3599472486 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f65180a2520 0x7f65180a49e0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:17.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:17.048+0000 7f65315c7640 1 --2- 172.21.15.46:0/3599472486 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f652c102240 0x7f652c109b90 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:17.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:17.048+0000 7f65315c7640 1 --2- 172.21.15.46:0/3599472486 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f652c101880 0x7f652c109650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:17.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:17.048+0000 7f65315c7640 1 --2- 172.21.15.46:0/3599472486 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f652c100690 0x7f652c10b010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:17.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:17.048+0000 7f65315c7640 1 -- 172.21.15.46:0/3599472486 >> 172.21.15.46:0/3599472486 conn(0x7f652c0fbc00 msgr2=0x7f652c104c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:09:17.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:17.048+0000 7f65315c7640 1 -- 172.21.15.46:0/3599472486 shutdown_connections 2024-03-31T23:09:17.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:17.048+0000 7f65315c7640 1 -- 172.21.15.46:0/3599472486 wait complete. 2024-03-31T23:09:17.061 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-31T23:09:17.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:17.162+0000 7f8b26180640 1 -- 172.21.15.46:0/3835508114 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8b20102a30 msgr2=0x7f8b20102eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:09:17.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:17.162+0000 7f8b26180640 1 --2- 172.21.15.46:0/3835508114 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8b20102a30 0x7f8b20102eb0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f8b14009930 tx=0x7f8b1402bc30 comp rx=0 tx=0).stop 2024-03-31T23:09:17.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:17.162+0000 7f8b26180640 1 -- 172.21.15.46:0/3835508114 shutdown_connections 2024-03-31T23:09:17.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:17.162+0000 7f8b26180640 1 --2- 172.21.15.46:0/3835508114 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8b201033f0 0x7f8b20109c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:17.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:17.162+0000 7f8b26180640 1 --2- 172.21.15.46:0/3835508114 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8b20102a30 0x7f8b20102eb0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:17.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:17.162+0000 7f8b26180640 1 --2- 172.21.15.46:0/3835508114 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8b200ffa60 0x7f8b200ffe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:17.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:17.162+0000 7f8b26180640 1 -- 172.21.15.46:0/3835508114 >> 172.21.15.46:0/3835508114 conn(0x7f8b200fb390 msgr2=0x7f8b200fd7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:09:17.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:17.162+0000 7f8b26180640 1 -- 172.21.15.46:0/3835508114 shutdown_connections 2024-03-31T23:09:17.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:17.162+0000 7f8b26180640 1 -- 172.21.15.46:0/3835508114 wait complete. 2024-03-31T23:09:17.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:17.163+0000 7f8b26180640 1 Processor -- start 2024-03-31T23:09:17.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:17.163+0000 7f8b26180640 1 -- start start 2024-03-31T23:09:17.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:17.163+0000 7f8b26180640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8b200ffa60 0x7f8b201979d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:09:17.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:17.163+0000 7f8b26180640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8b20102a30 0x7f8b20197f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:09:17.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:17.163+0000 7f8b26180640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8b201033f0 0x7f8b2019ef80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:09:17.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:17.163+0000 7f8b26180640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f8b2010d2d0 con 0x7f8b201033f0 2024-03-31T23:09:17.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:17.163+0000 7f8b26180640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f8b2010d150 con 0x7f8b20102a30 2024-03-31T23:09:17.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:17.163+0000 7f8b26180640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f8b2010d450 con 0x7f8b200ffa60 2024-03-31T23:09:17.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:17.163+0000 7f8b1ffff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8b201033f0 0x7f8b2019ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:09:17.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:17.163+0000 7f8b1ffff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8b201033f0 0x7f8b2019ef80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41232/0 (socket says 172.21.15.46:41232) 2024-03-31T23:09:17.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:17.163+0000 7f8b1ffff640 1 -- 172.21.15.46:0/511558404 learned_addr learned my addr 172.21.15.46:0/511558404 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:09:17.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:17.163+0000 7f8b1effd640 1 --2- 172.21.15.46:0/511558404 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8b20102a30 0x7f8b20197f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:09:17.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:17.163+0000 7f8b1f7fe640 1 --2- 172.21.15.46:0/511558404 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8b200ffa60 0x7f8b201979d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:09:17.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:17.163+0000 7f8b1ffff640 1 -- 172.21.15.46:0/511558404 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8b200ffa60 msgr2=0x7f8b201979d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:09:17.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:17.163+0000 7f8b1ffff640 1 --2- 172.21.15.46:0/511558404 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8b200ffa60 0x7f8b201979d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:17.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:17.163+0000 7f8b1ffff640 1 -- 172.21.15.46:0/511558404 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8b20102a30 msgr2=0x7f8b20197f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:09:17.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:17.163+0000 7f8b1ffff640 1 --2- 172.21.15.46:0/511558404 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8b20102a30 0x7f8b20197f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:17.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:17.163+0000 7f8b1ffff640 1 -- 172.21.15.46:0/511558404 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b2019f5f0 con 0x7f8b201033f0 2024-03-31T23:09:17.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:17.163+0000 7f8b1f7fe640 1 --2- 172.21.15.46:0/511558404 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8b200ffa60 0x7f8b201979d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T23:09:17.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:17.164+0000 7f8b1ffff640 1 --2- 172.21.15.46:0/511558404 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8b201033f0 0x7f8b2019ef80 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f8b1800eb10 tx=0x7f8b1800c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:09:17.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:17.164+0000 7f8b1cff9640 1 -- 172.21.15.46:0/511558404 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b18004b20 con 0x7f8b201033f0 2024-03-31T23:09:17.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:17.164+0000 7f8b26180640 1 -- 172.21.15.46:0/511558404 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8b2019f8e0 con 0x7f8b201033f0 2024-03-31T23:09:17.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:17.164+0000 7f8b1cff9640 1 -- 172.21.15.46:0/511558404 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8b18004cc0 con 0x7f8b201033f0 2024-03-31T23:09:17.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:17.164+0000 7f8b1cff9640 1 -- 172.21.15.46:0/511558404 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b18010690 con 0x7f8b201033f0 2024-03-31T23:09:17.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:17.164+0000 7f8b26180640 1 -- 172.21.15.46:0/511558404 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8b2019fe00 con 0x7f8b201033f0 2024-03-31T23:09:17.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:17.165+0000 7f8b26180640 1 -- 172.21.15.46:0/511558404 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8b20108c30 con 0x7f8b201033f0 2024-03-31T23:09:17.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:17.166+0000 7f8b1cff9640 1 -- 172.21.15.46:0/511558404 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 16) ==== 137320+0+0 (secure 0 0 0) 0x7f8b18005000 con 0x7f8b201033f0 2024-03-31T23:09:17.169 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:17.166+0000 7f8b1cff9640 1 --2- 172.21.15.46:0/511558404 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f8b080a24a0 0x7f8b080a4960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:09:17.169 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:17.166+0000 7f8b1cff9640 1 -- 172.21.15.46:0/511558404 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7f8b180cf190 con 0x7f8b201033f0 2024-03-31T23:09:17.169 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:17.166+0000 7f8b1f7fe640 1 --2- 172.21.15.46:0/511558404 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f8b080a24a0 0x7f8b080a4960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:09:17.169 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:17.167+0000 7f8b1f7fe640 1 --2- 172.21.15.46:0/511558404 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f8b080a24a0 0x7f8b080a4960 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f8b20069ba0 tx=0x7f8b10005d30 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:09:17.171 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:17.170+0000 7f8b1cff9640 1 -- 172.21.15.46:0/511558404 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f8b18053280 con 0x7f8b201033f0 2024-03-31T23:09:17.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:17.356+0000 7f8b26180640 1 -- 172.21.15.46:0/511558404 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f8b2002d170 con 0x7f8b201033f0 2024-03-31T23:09:17.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:17.357+0000 7f8b1cff9640 1 -- 172.21.15.46:0/511558404 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) ==== 74+0+24105 (secure 0 0 0) 0x7f8b1809c110 con 0x7f8b201033f0 2024-03-31T23:09:17.359 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-31T23:09:17.359 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":91,"fsid":"6a9d8f08-efb2-11ee-b647-cb9ed24678a4","created":"2024-03-31T23:01:15.463711+0000","modified":"2024-03-31T23:09:17.181787+0000","last_up_change":"2024-03-31T23:08:20.199888+0000","last_in_change":"2024-03-31T23:07:59.438035+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-31T23:04:32.823751+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-31T23:09:09.145711+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-31T23:09:10.148579+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-31T23:09:13.419519+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"91","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"47429095-3f46-4ed7-b7c4-504464719815","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":1824772695},{"type":"v1","addr":"172.21.15.46:6803","nonce":1824772695}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":1824772695},{"type":"v1","addr":"172.21.15.46:6805","nonce":1824772695}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":1824772695},{"type":"v1","addr":"172.21.15.46:6809","nonce":1824772695}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":1824772695},{"type":"v1","addr":"172.21.15.46:6807","nonce":1824772695}]},"public_addr":"172.21.15.46:6803/1824772695","cluster_addr":"172.21.15.46:6805/1824772695","heartbeat_back_addr":"172.21.15.46:6809/1824772695","heartbeat_front_addr":"172.21.15.46:6807/1824772695","state":["exists","up"]},{"osd":1,"uuid":"be56f883-411e-4dbe-b50e-a158134df4a6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6800","nonce":973693922},{"type":"v1","addr":"172.21.15.142:6801","nonce":973693922}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6802","nonce":973693922},{"type":"v1","addr":"172.21.15.142:6803","nonce":973693922}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6806","nonce":973693922},{"type":"v1","addr":"172.21.15.142:6807","nonce":973693922}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6804","nonce":973693922},{"type":"v1","addr":"172.21.15.142:6805","nonce":973693922}]},"public_addr":"172.21.15.142:6801/973693922","cluster_addr":"172.21.15.142:6803/973693922","heartbeat_back_addr":"172.21.15.142:6807/973693922","heartbeat_front_addr":"172.21.15.142:6805/973693922","state":["exists","up"]},{"osd":2,"uuid":"c99a8794-ab19-4260-9a8d-4bc0bd2c8175","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6800","nonce":4084261813},{"type":"v1","addr":"172.21.15.162:6801","nonce":4084261813}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6802","nonce":4084261813},{"type":"v1","addr":"172.21.15.162:6803","nonce":4084261813}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6806","nonce":4084261813},{"type":"v1","addr":"172.21.15.162:6807","nonce":4084261813}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6804","nonce":4084261813},{"type":"v1","addr":"172.21.15.162:6805","nonce":4084261813}]},"public_addr":"172.21.15.162:6801/4084261813","cluster_addr":"172.21.15.162:6803/4084261813","heartbeat_back_addr":"172.21.15.162:6807/4084261813","heartbeat_front_addr":"172.21.15.162:6805/4084261813","state":["exists","up"]},{"osd":3,"uuid":"f877814e-9ef6-4872-a081-5609bb9ebc06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":3496317970},{"type":"v1","addr":"172.21.15.46:6811","nonce":3496317970}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":3496317970},{"type":"v1","addr":"172.21.15.46:6813","nonce":3496317970}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":3496317970},{"type":"v1","addr":"172.21.15.46:6817","nonce":3496317970}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":3496317970},{"type":"v1","addr":"172.21.15.46:6815","nonce":3496317970}]},"public_addr":"172.21.15.46:6811/3496317970","cluster_addr":"172.21.15.46:6813/3496317970","heartbeat_back_addr":"172.21.15.46:6817/3496317970","heartbeat_front_addr":"172.21.15.46:6815/3496317970","state":["exists","up"]},{"osd":4,"uuid":"beecec0a-8280-44fe-8f5f-8498a35eca8f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6808","nonce":2996266929},{"type":"v1","addr":"172.21.15.142:6809","nonce":2996266929}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6810","nonce":2996266929},{"type":"v1","addr":"172.21.15.142:6811","nonce":2996266929}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6814","nonce":2996266929},{"type":"v1","addr":"172.21.15.142:6815","nonce":2996266929}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6812","nonce":2996266929},{"type":"v1","addr":"172.21.15.142:6813","nonce":2996266929}]},"public_addr":"172.21.15.142:6809/2996266929","cluster_addr":"172.21.15.142:6811/2996266929","heartbeat_back_addr":"172.21.15.142:6815/2996266929","heartbeat_front_addr":"172.21.15.142:6813/2996266929","state":["exists","up"]},{"osd":5,"uuid":"9da7ff75-f658-45ea-a02f-1943fdda76b9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6808","nonce":914208293},{"type":"v1","addr":"172.21.15.162:6809","nonce":914208293}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6810","nonce":914208293},{"type":"v1","addr":"172.21.15.162:6811","nonce":914208293}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6814","nonce":914208293},{"type":"v1","addr":"172.21.15.162:6815","nonce":914208293}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6812","nonce":914208293},{"type":"v1","addr":"172.21.15.162:6813","nonce":914208293}]},"public_addr":"172.21.15.162:6809/914208293","cluster_addr":"172.21.15.162:6811/914208293","heartbeat_back_addr":"172.21.15.162:6815/914208293","heartbeat_front_addr":"172.21.15.162:6813/914208293","state":["exists","up"]},{"osd":6,"uuid":"8fa11224-cb26-4d3f-87d1-6a8f31de2a19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":56291966},{"type":"v1","addr":"172.21.15.46:6819","nonce":56291966}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":56291966},{"type":"v1","addr":"172.21.15.46:6821","nonce":56291966}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6824","nonce":56291966},{"type":"v1","addr":"172.21.15.46:6825","nonce":56291966}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":56291966},{"type":"v1","addr":"172.21.15.46:6823","nonce":56291966}]},"public_addr":"172.21.15.46:6819/56291966","cluster_addr":"172.21.15.46:6821/56291966","heartbeat_back_addr":"172.21.15.46:6825/56291966","heartbeat_front_addr":"172.21.15.46:6823/56291966","state":["exists","up"]},{"osd":7,"uuid":"0d0338b7-4d0a-4bb0-b152-8f5224871d9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6816","nonce":2146598446},{"type":"v1","addr":"172.21.15.142:6817","nonce":2146598446}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6818","nonce":2146598446},{"type":"v1","addr":"172.21.15.142:6819","nonce":2146598446}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6822","nonce":2146598446},{"type":"v1","addr":"172.21.15.142:6823","nonce":2146598446}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6820","nonce":2146598446},{"type":"v1","addr":"172.21.15.142:6821","nonce":2146598446}]},"public_addr":"172.21.15.142:6817/2146598446","cluster_addr":"172.21.15.142:6819/2146598446","heartbeat_back_addr":"172.21.15.142:6823/2146598446","heartbeat_front_addr":"172.21.15.142:6821/2146598446","state":["exists","up"]},{"osd":8,"uuid":"8b6f6037-892a-416c-babb-7552648d0b3a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6816","nonce":1173078318},{"type":"v1","addr":"172.21.15.162:6817","nonce":1173078318}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6818","nonce":1173078318},{"type":"v1","addr":"172.21.15.162:6819","nonce":1173078318}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6822","nonce":1173078318},{"type":"v1","addr":"172.21.15.162:6823","nonce":1173078318}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6820","nonce":1173078318},{"type":"v1","addr":"172.21.15.162:6821","nonce":1173078318}]},"public_addr":"172.21.15.162:6817/1173078318","cluster_addr":"172.21.15.162:6819/1173078318","heartbeat_back_addr":"172.21.15.162:6823/1173078318","heartbeat_front_addr":"172.21.15.162:6821/1173078318","state":["exists","up"]},{"osd":9,"uuid":"faccce25-6bf2-4220-8858-cda82b72be68","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":532708881},{"type":"v1","addr":"172.21.15.46:6827","nonce":532708881}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6828","nonce":532708881},{"type":"v1","addr":"172.21.15.46:6829","nonce":532708881}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":532708881},{"type":"v1","addr":"172.21.15.46:6833","nonce":532708881}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":532708881},{"type":"v1","addr":"172.21.15.46:6831","nonce":532708881}]},"public_addr":"172.21.15.46:6827/532708881","cluster_addr":"172.21.15.46:6829/532708881","heartbeat_back_addr":"172.21.15.46:6833/532708881","heartbeat_front_addr":"172.21.15.46:6831/532708881","state":["exists","up"]},{"osd":10,"uuid":"cdd356db-f9eb-4080-95ab-db5d8223e572","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6824","nonce":2845222366},{"type":"v1","addr":"172.21.15.142:6825","nonce":2845222366}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6826","nonce":2845222366},{"type":"v1","addr":"172.21.15.142:6827","nonce":2845222366}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6830","nonce":2845222366},{"type":"v1","addr":"172.21.15.142:6831","nonce":2845222366}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6828","nonce":2845222366},{"type":"v1","addr":"172.21.15.142:6829","nonce":2845222366}]},"public_addr":"172.21.15.142:6825/2845222366","cluster_addr":"172.21.15.142:6827/2845222366","heartbeat_back_addr":"172.21.15.142:6831/2845222366","heartbeat_front_addr":"172.21.15.142:6829/2845222366","state":["exists","up"]},{"osd":11,"uuid":"72104d1c-3ea2-45ad-bc0b-0500f146c06b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6824","nonce":1986323778},{"type":"v1","addr":"172.21.15.162:6825","nonce":1986323778}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6826","nonce":1986323778},{"type":"v1","addr":"172.21.15.162:6827","nonce":1986323778}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6830","nonce":1986323778},{"type":"v1","addr":"172.21.15.162:6831","nonce":1986323778}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6828","nonce":1986323778},{"type":"v1","addr":"172.21.15.162:6829","nonce":1986323778}]},"public_addr":"172.21.15.162:6825/1986323778","cluster_addr":"172.21.15.162:6827/1986323778","heartbeat_back_addr":"172.21.15.162:6831/1986323778","heartbeat_front_addr":"172.21.15.162:6829/1986323778","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:03:44.996331+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:04:07.994738+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:04:30.942255+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:04:55.588884+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:05:19.900600+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:05:43.807442+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:06:09.026976+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:06:34.586784+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:06:59.751605+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:07:26.144495+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:07:52.326909+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:08:17.423042+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.46:0/4021039857":"2024-04-01T23:02:08.493619+0000","172.21.15.46:6801/3681182647":"2024-04-01T23:02:08.493619+0000","172.21.15.46:0/4022214806":"2024-04-01T23:02:08.493619+0000","172.21.15.46:6800/1515528215":"2024-04-01T23:01:41.910162+0000","172.21.15.46:6800/3681182647":"2024-04-01T23:02:08.493619+0000","172.21.15.46:0/143885707":"2024-04-01T23:01:41.910162+0000","172.21.15.46:6801/1515528215":"2024-04-01T23:01:41.910162+0000","172.21.15.46:0/1680389778":"2024-04-01T23:01:41.910162+0000","172.21.15.46:0/1782133966":"2024-04-01T23:01:41.910162+0000","172.21.15.46:0/3925209376":"2024-04-01T23:02:08.493619+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-31T23:09:17.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:17.360+0000 7f8b26180640 1 -- 172.21.15.46:0/511558404 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f8b080a24a0 msgr2=0x7f8b080a4960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:09:17.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:17.360+0000 7f8b26180640 1 --2- 172.21.15.46:0/511558404 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f8b080a24a0 0x7f8b080a4960 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f8b20069ba0 tx=0x7f8b10005d30 comp rx=0 tx=0).stop 2024-03-31T23:09:17.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:17.360+0000 7f8b26180640 1 -- 172.21.15.46:0/511558404 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8b201033f0 msgr2=0x7f8b2019ef80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:09:17.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:17.360+0000 7f8b26180640 1 --2- 172.21.15.46:0/511558404 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8b201033f0 0x7f8b2019ef80 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f8b1800eb10 tx=0x7f8b1800c2d0 comp rx=0 tx=0).stop 2024-03-31T23:09:17.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:17.360+0000 7f8b26180640 1 -- 172.21.15.46:0/511558404 shutdown_connections 2024-03-31T23:09:17.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:17.360+0000 7f8b26180640 1 --2- 172.21.15.46:0/511558404 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f8b080a24a0 0x7f8b080a4960 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:17.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:17.360+0000 7f8b26180640 1 --2- 172.21.15.46:0/511558404 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8b201033f0 0x7f8b2019ef80 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:17.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:17.360+0000 7f8b26180640 1 --2- 172.21.15.46:0/511558404 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8b20102a30 0x7f8b20197f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:17.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:17.360+0000 7f8b26180640 1 --2- 172.21.15.46:0/511558404 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8b200ffa60 0x7f8b201979d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:17.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:17.360+0000 7f8b26180640 1 -- 172.21.15.46:0/511558404 >> 172.21.15.46:0/511558404 conn(0x7f8b200fb390 msgr2=0x7f8b20105e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:09:17.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:17.360+0000 7f8b26180640 1 -- 172.21.15.46:0/511558404 shutdown_connections 2024-03-31T23:09:17.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:17.360+0000 7f8b26180640 1 -- 172.21.15.46:0/511558404 wait complete. 2024-03-31T23:09:17.362 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:09:17 smithi046 ceph-mon[32498]: osdmap e90: 12 total, 12 up, 12 in 2024-03-31T23:09:17.362 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:09:17 smithi046 ceph-mon[32498]: from='client.? 172.21.15.46:0/2264946362' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2024-03-31T23:09:17.363 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:09:17 smithi046 ceph-mon[32498]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-31T23:09:17.363 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:09:17 smithi046 ceph-mon[32498]: from='client.? 172.21.15.46:0/367195758' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-31T23:09:17.363 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:09:17 smithi046 ceph-mon[32498]: from='client.? 172.21.15.46:0/3599472486' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-31T23:09:17.388 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-31T23:09:17.443 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:09:17 smithi142 ceph-mon[35271]: osdmap e90: 12 total, 12 up, 12 in 2024-03-31T23:09:17.443 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:09:17 smithi142 ceph-mon[35271]: from='client.? 172.21.15.46:0/2264946362' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2024-03-31T23:09:17.443 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:09:17 smithi142 ceph-mon[35271]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-31T23:09:17.443 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:09:17 smithi142 ceph-mon[35271]: from='client.? 172.21.15.46:0/367195758' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-31T23:09:17.443 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:09:17 smithi142 ceph-mon[35271]: from='client.? 172.21.15.46:0/3599472486' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-31T23:09:17.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:17.489+0000 7feae04eb640 1 -- 172.21.15.46:0/914669978 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fead8102220 msgr2=0x7fead8108ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:09:17.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:17.489+0000 7feae04eb640 1 --2- 172.21.15.46:0/914669978 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fead8102220 0x7fead8108ab0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fead0009ca0 tx=0x7fead002c0f0 comp rx=0 tx=0).stop 2024-03-31T23:09:17.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:17.489+0000 7feae04eb640 1 -- 172.21.15.46:0/914669978 shutdown_connections 2024-03-31T23:09:17.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:17.489+0000 7feae04eb640 1 --2- 172.21.15.46:0/914669978 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fead8102220 0x7fead8108ab0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:17.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:17.489+0000 7feae04eb640 1 --2- 172.21.15.46:0/914669978 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fead8101860 0x7fead8101ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:17.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:17.489+0000 7feae04eb640 1 --2- 172.21.15.46:0/914669978 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fead8100670 0x7fead8100a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:17.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:17.489+0000 7feae04eb640 1 -- 172.21.15.46:0/914669978 >> 172.21.15.46:0/914669978 conn(0x7fead80fbc00 msgr2=0x7fead80fe020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:09:17.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:17.489+0000 7feae04eb640 1 -- 172.21.15.46:0/914669978 shutdown_connections 2024-03-31T23:09:17.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:17.489+0000 7feae04eb640 1 -- 172.21.15.46:0/914669978 wait complete. 2024-03-31T23:09:17.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:17.490+0000 7feae04eb640 1 Processor -- start 2024-03-31T23:09:17.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:17.490+0000 7feae04eb640 1 -- start start 2024-03-31T23:09:17.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:17.490+0000 7feae04eb640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fead8100670 0x7fead81979e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:09:17.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:17.490+0000 7feae04eb640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fead8101860 0x7fead8197f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:09:17.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:17.490+0000 7feae04eb640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fead8102220 0x7fead819ef90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:09:17.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:17.490+0000 7feae04eb640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fead810c200 con 0x7fead8102220 2024-03-31T23:09:17.494 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:17.490+0000 7feae04eb640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fead810c080 con 0x7fead8101860 2024-03-31T23:09:17.494 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:17.490+0000 7feae04eb640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fead810c380 con 0x7fead8100670 2024-03-31T23:09:17.494 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:17.490+0000 7feadea88640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fead8102220 0x7fead819ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:09:17.494 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:17.490+0000 7feadda86640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fead8101860 0x7fead8197f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:09:17.494 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:17.490+0000 7feadea88640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fead8102220 0x7fead819ef90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41242/0 (socket says 172.21.15.46:41242) 2024-03-31T23:09:17.494 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:17.490+0000 7feadea88640 1 -- 172.21.15.46:0/3973573710 learned_addr learned my addr 172.21.15.46:0/3973573710 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:09:17.494 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:17.490+0000 7feade287640 1 --2- 172.21.15.46:0/3973573710 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fead8100670 0x7fead81979e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:09:17.494 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:17.490+0000 7feadea88640 1 -- 172.21.15.46:0/3973573710 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fead8100670 msgr2=0x7fead81979e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:09:17.494 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:17.490+0000 7feadea88640 1 --2- 172.21.15.46:0/3973573710 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fead8100670 0x7fead81979e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:17.494 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:17.490+0000 7feadea88640 1 -- 172.21.15.46:0/3973573710 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fead8101860 msgr2=0x7fead8197f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:09:17.494 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:17.490+0000 7feadea88640 1 --2- 172.21.15.46:0/3973573710 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fead8101860 0x7fead8197f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:17.494 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:17.490+0000 7feadea88640 1 -- 172.21.15.46:0/3973573710 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fead819f600 con 0x7fead8102220 2024-03-31T23:09:17.494 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:17.490+0000 7feade287640 1 --2- 172.21.15.46:0/3973573710 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fead8100670 0x7fead81979e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T23:09:17.494 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:17.490+0000 7feadda86640 1 --2- 172.21.15.46:0/3973573710 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fead8101860 0x7fead8197f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:09:17.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:17.490+0000 7feadea88640 1 --2- 172.21.15.46:0/3973573710 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fead8102220 0x7fead819ef90 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fead002c700 tx=0x7fead00328a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:09:17.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:17.490+0000 7feacf7fe640 1 -- 172.21.15.46:0/3973573710 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fead0041020 con 0x7fead8102220 2024-03-31T23:09:17.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:17.491+0000 7feacf7fe640 1 -- 172.21.15.46:0/3973573710 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fead0003e20 con 0x7fead8102220 2024-03-31T23:09:17.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:17.491+0000 7feae04eb640 1 -- 172.21.15.46:0/3973573710 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fead819f890 con 0x7fead8102220 2024-03-31T23:09:17.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:17.491+0000 7feacf7fe640 1 -- 172.21.15.46:0/3973573710 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fead0039040 con 0x7fead8102220 2024-03-31T23:09:17.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:17.491+0000 7feae04eb640 1 -- 172.21.15.46:0/3973573710 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fead819fe50 con 0x7fead8102220 2024-03-31T23:09:17.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:17.491+0000 7feae04eb640 1 -- 172.21.15.46:0/3973573710 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fead81a0080 con 0x7fead8102220 2024-03-31T23:09:17.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:17.492+0000 7feacf7fe640 1 -- 172.21.15.46:0/3973573710 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 16) ==== 137320+0+0 (secure 0 0 0) 0x7fead0044020 con 0x7fead8102220 2024-03-31T23:09:17.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:17.493+0000 7feacf7fe640 1 --2- 172.21.15.46:0/3973573710 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7feac40a2520 0x7feac40a49e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:09:17.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:17.493+0000 7feade287640 1 --2- 172.21.15.46:0/3973573710 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7feac40a2520 0x7feac40a49e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:09:17.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:17.493+0000 7feacf7fe640 1 -- 172.21.15.46:0/3973573710 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8802+0+0 (secure 0 0 0) 0x7fead00f0bd0 con 0x7fead8102220 2024-03-31T23:09:17.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:17.494+0000 7feade287640 1 --2- 172.21.15.46:0/3973573710 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7feac40a2520 0x7feac40a49e0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fead81016c0 tx=0x7fead4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:09:17.499 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:17.498+0000 7feacf7fe640 1 -- 172.21.15.46:0/3973573710 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fead004b020 con 0x7fead8102220 2024-03-31T23:09:17.545 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:09:17 smithi162 ceph-mon[36237]: osdmap e90: 12 total, 12 up, 12 in 2024-03-31T23:09:17.546 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:09:17 smithi162 ceph-mon[36237]: from='client.? 172.21.15.46:0/2264946362' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2024-03-31T23:09:17.546 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:09:17 smithi162 ceph-mon[36237]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-31T23:09:17.546 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:09:17 smithi162 ceph-mon[36237]: from='client.? 172.21.15.46:0/367195758' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-31T23:09:17.546 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:09:17 smithi162 ceph-mon[36237]: from='client.? 172.21.15.46:0/3599472486' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-31T23:09:17.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:17.719+0000 7feae04eb640 1 -- 172.21.15.46:0/3973573710 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fead81a0250 con 0x7fead8102220 2024-03-31T23:09:17.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:17.720+0000 7feacf7fe640 1 -- 172.21.15.46:0/3973573710 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) ==== 94+0+11809 (secure 0 0 0) 0x7fead003b2f0 con 0x7fead8102220 2024-03-31T23:09:17.721 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-31T23:09:17.721 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":17,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/2169018547","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":2169018547},{"type":"v1","addr":"172.21.15.46:6835","nonce":2169018547}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6837/1069387756","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6836","nonce":1069387756},{"type":"v1","addr":"172.21.15.46:6837","nonce":1069387756}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14634,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6839/3915337844","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6838","nonce":3915337844},{"type":"v1","addr":"172.21.15.46:6839","nonce":3915337844}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14646,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6841/897767145","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6840","nonce":897767145},{"type":"v1","addr":"172.21.15.46:6841","nonce":897767145}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24415,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6833/1746950966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6832","nonce":1746950966},{"type":"v1","addr":"172.21.15.162:6833","nonce":1746950966}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24422,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6833/3394739181","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6832","nonce":3394739181},{"type":"v1","addr":"172.21.15.142:6833","nonce":3394739181}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24442,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6835/1957208984","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6834","nonce":1957208984},{"type":"v1","addr":"172.21.15.162:6835","nonce":1957208984}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24451,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6837/1371886876","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6836","nonce":1371886876},{"type":"v1","addr":"172.21.15.162:6837","nonce":1371886876}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24455,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6835/212637431","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6834","nonce":212637431},{"type":"v1","addr":"172.21.15.142:6835","nonce":212637431}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24478,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6839/1976275826","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6838","nonce":1976275826},{"type":"v1","addr":"172.21.15.162:6839","nonce":1976275826}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24482,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6837/2559258399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6836","nonce":2559258399},{"type":"v1","addr":"172.21.15.142:6837","nonce":2559258399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-31T23:09:11.009090+0000","modified":"2024-03-31T23:09:15.556861+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24515},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24515":{"gid":24515,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.142:6839/2519143918","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6838","nonce":2519143918},{"type":"v1","addr":"172.21.15.142:6839","nonce":2519143918}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":24515,"qdb_cluster":[24515]},"id":1}]} 2024-03-31T23:09:17.722 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 17 2024-03-31T23:09:17.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:17.722+0000 7feae04eb640 1 -- 172.21.15.46:0/3973573710 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7feac40a2520 msgr2=0x7feac40a49e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:09:17.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:17.723+0000 7feae04eb640 1 --2- 172.21.15.46:0/3973573710 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7feac40a2520 0x7feac40a49e0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fead81016c0 tx=0x7fead4005f90 comp rx=0 tx=0).stop 2024-03-31T23:09:17.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:17.723+0000 7feae04eb640 1 -- 172.21.15.46:0/3973573710 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fead8102220 msgr2=0x7fead819ef90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:09:17.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:17.723+0000 7feae04eb640 1 --2- 172.21.15.46:0/3973573710 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fead8102220 0x7fead819ef90 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fead002c700 tx=0x7fead00328a0 comp rx=0 tx=0).stop 2024-03-31T23:09:17.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:17.723+0000 7feae04eb640 1 -- 172.21.15.46:0/3973573710 shutdown_connections 2024-03-31T23:09:17.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:17.723+0000 7feae04eb640 1 --2- 172.21.15.46:0/3973573710 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7feac40a2520 0x7feac40a49e0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:17.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:17.723+0000 7feae04eb640 1 --2- 172.21.15.46:0/3973573710 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fead8102220 0x7fead819ef90 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:17.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:17.723+0000 7feae04eb640 1 --2- 172.21.15.46:0/3973573710 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fead8101860 0x7fead8197f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:17.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:17.723+0000 7feae04eb640 1 --2- 172.21.15.46:0/3973573710 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fead8100670 0x7fead81979e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:17.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:17.723+0000 7feae04eb640 1 -- 172.21.15.46:0/3973573710 >> 172.21.15.46:0/3973573710 conn(0x7fead80fbc00 msgr2=0x7fead80fd6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:09:17.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:17.723+0000 7feae04eb640 1 -- 172.21.15.46:0/3973573710 shutdown_connections 2024-03-31T23:09:17.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:17.723+0000 7feae04eb640 1 -- 172.21.15.46:0/3973573710 wait complete. 2024-03-31T23:09:17.735 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-31T23:09:17.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:17.836+0000 7f218a48f640 1 -- 172.21.15.46:0/3315077470 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2184102220 msgr2=0x7f2184108ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:09:17.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:17.836+0000 7f218a48f640 1 --2- 172.21.15.46:0/3315077470 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2184102220 0x7f2184108ab0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f217c009ca0 tx=0x7f217c02c0f0 comp rx=0 tx=0).stop 2024-03-31T23:09:17.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:17.837+0000 7f218a48f640 1 -- 172.21.15.46:0/3315077470 shutdown_connections 2024-03-31T23:09:17.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:17.837+0000 7f218a48f640 1 --2- 172.21.15.46:0/3315077470 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2184102220 0x7f2184108ab0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:17.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:17.837+0000 7f218a48f640 1 --2- 172.21.15.46:0/3315077470 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2184101860 0x7f2184101ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:17.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:17.837+0000 7f218a48f640 1 --2- 172.21.15.46:0/3315077470 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2184100670 0x7f2184100a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:17.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:17.837+0000 7f218a48f640 1 -- 172.21.15.46:0/3315077470 >> 172.21.15.46:0/3315077470 conn(0x7f21840fbc00 msgr2=0x7f21840fe020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:09:17.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:17.837+0000 7f218a48f640 1 -- 172.21.15.46:0/3315077470 shutdown_connections 2024-03-31T23:09:17.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:17.837+0000 7f218a48f640 1 -- 172.21.15.46:0/3315077470 wait complete. 2024-03-31T23:09:17.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:17.837+0000 7f218a48f640 1 Processor -- start 2024-03-31T23:09:17.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:17.837+0000 7f218a48f640 1 -- start start 2024-03-31T23:09:17.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:17.837+0000 7f218a48f640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2184100670 0x7f21841978f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:09:17.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:17.837+0000 7f218a48f640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2184101860 0x7f2184197e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:09:17.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:17.837+0000 7f218a48f640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2184102220 0x7f218419eea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:09:17.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:17.837+0000 7f218a48f640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f218410c180 con 0x7f2184101860 2024-03-31T23:09:17.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:17.837+0000 7f218a48f640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f218410c000 con 0x7f2184100670 2024-03-31T23:09:17.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:17.837+0000 7f218a48f640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f218410c300 con 0x7f2184102220 2024-03-31T23:09:17.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:17.838+0000 7f21837fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2184101860 0x7f2184197e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:09:17.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:17.838+0000 7f21837fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2184101860 0x7f2184197e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41252/0 (socket says 172.21.15.46:41252) 2024-03-31T23:09:17.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:17.838+0000 7f2183fff640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2184100670 0x7f21841978f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:09:17.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:17.838+0000 7f21837fe640 1 -- 172.21.15.46:0/2093683791 learned_addr learned my addr 172.21.15.46:0/2093683791 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:09:17.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:17.838+0000 7f2188a2c640 1 --2- 172.21.15.46:0/2093683791 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2184102220 0x7f218419eea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:09:17.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:17.838+0000 7f21837fe640 1 -- 172.21.15.46:0/2093683791 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2184102220 msgr2=0x7f218419eea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:09:17.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:17.838+0000 7f21837fe640 1 --2- 172.21.15.46:0/2093683791 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2184102220 0x7f218419eea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:17.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:17.838+0000 7f21837fe640 1 -- 172.21.15.46:0/2093683791 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2184100670 msgr2=0x7f21841978f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:09:17.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:17.838+0000 7f21837fe640 1 --2- 172.21.15.46:0/2093683791 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2184100670 0x7f21841978f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:17.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:17.838+0000 7f21837fe640 1 -- 172.21.15.46:0/2093683791 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f218419f510 con 0x7f2184101860 2024-03-31T23:09:17.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:17.838+0000 7f2183fff640 1 --2- 172.21.15.46:0/2093683791 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2184100670 0x7f21841978f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:09:17.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:17.838+0000 7f21837fe640 1 --2- 172.21.15.46:0/2093683791 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2184101860 0x7f2184197e30 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f216c00c920 tx=0x7f216c00cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:09:17.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:17.838+0000 7f21817fa640 1 -- 172.21.15.46:0/2093683791 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f216c007c70 con 0x7f2184101860 2024-03-31T23:09:17.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:17.838+0000 7f218a48f640 1 -- 172.21.15.46:0/2093683791 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f218419f800 con 0x7f2184101860 2024-03-31T23:09:17.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:17.838+0000 7f21817fa640 1 -- 172.21.15.46:0/2093683791 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f216c007e10 con 0x7f2184101860 2024-03-31T23:09:17.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:17.838+0000 7f21817fa640 1 -- 172.21.15.46:0/2093683791 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f216c018820 con 0x7f2184101860 2024-03-31T23:09:17.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:17.838+0000 7f218a48f640 1 -- 172.21.15.46:0/2093683791 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f218419fd50 con 0x7f2184101860 2024-03-31T23:09:17.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:17.839+0000 7f218a48f640 1 -- 172.21.15.46:0/2093683791 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f218419ffb0 con 0x7f2184101860 2024-03-31T23:09:17.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:17.841+0000 7f21817fa640 1 -- 172.21.15.46:0/2093683791 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 16) ==== 137320+0+0 (secure 0 0 0) 0x7f216c0189c0 con 0x7f2184101860 2024-03-31T23:09:17.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:17.841+0000 7f21817fa640 1 --2- 172.21.15.46:0/2093683791 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f21700a24a0 0x7f21700a4960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:09:17.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:17.842+0000 7f2183fff640 1 --2- 172.21.15.46:0/2093683791 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f21700a24a0 0x7f21700a4960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:09:17.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:17.842+0000 7f21817fa640 1 -- 172.21.15.46:0/2093683791 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8802+0+0 (secure 0 0 0) 0x7f216c0cedc0 con 0x7f2184101860 2024-03-31T23:09:17.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:17.842+0000 7f2183fff640 1 --2- 172.21.15.46:0/2093683791 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f21700a24a0 0x7f21700a4960 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f21841016c0 tx=0x7f2174005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:09:17.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:17.845+0000 7f21817fa640 1 -- 172.21.15.46:0/2093683791 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f216c014030 con 0x7f2184101860 2024-03-31T23:09:17.943 INFO:journalctl@ceph.mds.b.smithi142.stdout:Mar 31 23:09:17 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mds-b[65801]: 2024-03-31T23:09:17.485+0000 7fb486a60700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-31T23:09:18.028 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:18.027+0000 7f218a48f640 1 -- 172.21.15.46:0/2093683791 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f21841a0180 con 0x7f2184101860 2024-03-31T23:09:18.030 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:18.029+0000 7f21817fa640 1 -- 172.21.15.46:0/2093683791 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) ==== 74+0+24105 (secure 0 0 0) 0x7f216c09bd70 con 0x7f2184101860 2024-03-31T23:09:18.030 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-31T23:09:18.030 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":91,"fsid":"6a9d8f08-efb2-11ee-b647-cb9ed24678a4","created":"2024-03-31T23:01:15.463711+0000","modified":"2024-03-31T23:09:17.181787+0000","last_up_change":"2024-03-31T23:08:20.199888+0000","last_in_change":"2024-03-31T23:07:59.438035+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-31T23:04:32.823751+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-31T23:09:09.145711+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-31T23:09:10.148579+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-31T23:09:13.419519+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"91","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"47429095-3f46-4ed7-b7c4-504464719815","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":1824772695},{"type":"v1","addr":"172.21.15.46:6803","nonce":1824772695}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":1824772695},{"type":"v1","addr":"172.21.15.46:6805","nonce":1824772695}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":1824772695},{"type":"v1","addr":"172.21.15.46:6809","nonce":1824772695}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":1824772695},{"type":"v1","addr":"172.21.15.46:6807","nonce":1824772695}]},"public_addr":"172.21.15.46:6803/1824772695","cluster_addr":"172.21.15.46:6805/1824772695","heartbeat_back_addr":"172.21.15.46:6809/1824772695","heartbeat_front_addr":"172.21.15.46:6807/1824772695","state":["exists","up"]},{"osd":1,"uuid":"be56f883-411e-4dbe-b50e-a158134df4a6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6800","nonce":973693922},{"type":"v1","addr":"172.21.15.142:6801","nonce":973693922}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6802","nonce":973693922},{"type":"v1","addr":"172.21.15.142:6803","nonce":973693922}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6806","nonce":973693922},{"type":"v1","addr":"172.21.15.142:6807","nonce":973693922}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6804","nonce":973693922},{"type":"v1","addr":"172.21.15.142:6805","nonce":973693922}]},"public_addr":"172.21.15.142:6801/973693922","cluster_addr":"172.21.15.142:6803/973693922","heartbeat_back_addr":"172.21.15.142:6807/973693922","heartbeat_front_addr":"172.21.15.142:6805/973693922","state":["exists","up"]},{"osd":2,"uuid":"c99a8794-ab19-4260-9a8d-4bc0bd2c8175","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6800","nonce":4084261813},{"type":"v1","addr":"172.21.15.162:6801","nonce":4084261813}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6802","nonce":4084261813},{"type":"v1","addr":"172.21.15.162:6803","nonce":4084261813}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6806","nonce":4084261813},{"type":"v1","addr":"172.21.15.162:6807","nonce":4084261813}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6804","nonce":4084261813},{"type":"v1","addr":"172.21.15.162:6805","nonce":4084261813}]},"public_addr":"172.21.15.162:6801/4084261813","cluster_addr":"172.21.15.162:6803/4084261813","heartbeat_back_addr":"172.21.15.162:6807/4084261813","heartbeat_front_addr":"172.21.15.162:6805/4084261813","state":["exists","up"]},{"osd":3,"uuid":"f877814e-9ef6-4872-a081-5609bb9ebc06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":3496317970},{"type":"v1","addr":"172.21.15.46:6811","nonce":3496317970}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":3496317970},{"type":"v1","addr":"172.21.15.46:6813","nonce":3496317970}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":3496317970},{"type":"v1","addr":"172.21.15.46:6817","nonce":3496317970}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":3496317970},{"type":"v1","addr":"172.21.15.46:6815","nonce":3496317970}]},"public_addr":"172.21.15.46:6811/3496317970","cluster_addr":"172.21.15.46:6813/3496317970","heartbeat_back_addr":"172.21.15.46:6817/3496317970","heartbeat_front_addr":"172.21.15.46:6815/3496317970","state":["exists","up"]},{"osd":4,"uuid":"beecec0a-8280-44fe-8f5f-8498a35eca8f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6808","nonce":2996266929},{"type":"v1","addr":"172.21.15.142:6809","nonce":2996266929}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6810","nonce":2996266929},{"type":"v1","addr":"172.21.15.142:6811","nonce":2996266929}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6814","nonce":2996266929},{"type":"v1","addr":"172.21.15.142:6815","nonce":2996266929}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6812","nonce":2996266929},{"type":"v1","addr":"172.21.15.142:6813","nonce":2996266929}]},"public_addr":"172.21.15.142:6809/2996266929","cluster_addr":"172.21.15.142:6811/2996266929","heartbeat_back_addr":"172.21.15.142:6815/2996266929","heartbeat_front_addr":"172.21.15.142:6813/2996266929","state":["exists","up"]},{"osd":5,"uuid":"9da7ff75-f658-45ea-a02f-1943fdda76b9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6808","nonce":914208293},{"type":"v1","addr":"172.21.15.162:6809","nonce":914208293}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6810","nonce":914208293},{"type":"v1","addr":"172.21.15.162:6811","nonce":914208293}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6814","nonce":914208293},{"type":"v1","addr":"172.21.15.162:6815","nonce":914208293}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6812","nonce":914208293},{"type":"v1","addr":"172.21.15.162:6813","nonce":914208293}]},"public_addr":"172.21.15.162:6809/914208293","cluster_addr":"172.21.15.162:6811/914208293","heartbeat_back_addr":"172.21.15.162:6815/914208293","heartbeat_front_addr":"172.21.15.162:6813/914208293","state":["exists","up"]},{"osd":6,"uuid":"8fa11224-cb26-4d3f-87d1-6a8f31de2a19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":56291966},{"type":"v1","addr":"172.21.15.46:6819","nonce":56291966}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":56291966},{"type":"v1","addr":"172.21.15.46:6821","nonce":56291966}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6824","nonce":56291966},{"type":"v1","addr":"172.21.15.46:6825","nonce":56291966}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":56291966},{"type":"v1","addr":"172.21.15.46:6823","nonce":56291966}]},"public_addr":"172.21.15.46:6819/56291966","cluster_addr":"172.21.15.46:6821/56291966","heartbeat_back_addr":"172.21.15.46:6825/56291966","heartbeat_front_addr":"172.21.15.46:6823/56291966","state":["exists","up"]},{"osd":7,"uuid":"0d0338b7-4d0a-4bb0-b152-8f5224871d9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6816","nonce":2146598446},{"type":"v1","addr":"172.21.15.142:6817","nonce":2146598446}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6818","nonce":2146598446},{"type":"v1","addr":"172.21.15.142:6819","nonce":2146598446}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6822","nonce":2146598446},{"type":"v1","addr":"172.21.15.142:6823","nonce":2146598446}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6820","nonce":2146598446},{"type":"v1","addr":"172.21.15.142:6821","nonce":2146598446}]},"public_addr":"172.21.15.142:6817/2146598446","cluster_addr":"172.21.15.142:6819/2146598446","heartbeat_back_addr":"172.21.15.142:6823/2146598446","heartbeat_front_addr":"172.21.15.142:6821/2146598446","state":["exists","up"]},{"osd":8,"uuid":"8b6f6037-892a-416c-babb-7552648d0b3a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6816","nonce":1173078318},{"type":"v1","addr":"172.21.15.162:6817","nonce":1173078318}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6818","nonce":1173078318},{"type":"v1","addr":"172.21.15.162:6819","nonce":1173078318}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6822","nonce":1173078318},{"type":"v1","addr":"172.21.15.162:6823","nonce":1173078318}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6820","nonce":1173078318},{"type":"v1","addr":"172.21.15.162:6821","nonce":1173078318}]},"public_addr":"172.21.15.162:6817/1173078318","cluster_addr":"172.21.15.162:6819/1173078318","heartbeat_back_addr":"172.21.15.162:6823/1173078318","heartbeat_front_addr":"172.21.15.162:6821/1173078318","state":["exists","up"]},{"osd":9,"uuid":"faccce25-6bf2-4220-8858-cda82b72be68","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":532708881},{"type":"v1","addr":"172.21.15.46:6827","nonce":532708881}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6828","nonce":532708881},{"type":"v1","addr":"172.21.15.46:6829","nonce":532708881}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":532708881},{"type":"v1","addr":"172.21.15.46:6833","nonce":532708881}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":532708881},{"type":"v1","addr":"172.21.15.46:6831","nonce":532708881}]},"public_addr":"172.21.15.46:6827/532708881","cluster_addr":"172.21.15.46:6829/532708881","heartbeat_back_addr":"172.21.15.46:6833/532708881","heartbeat_front_addr":"172.21.15.46:6831/532708881","state":["exists","up"]},{"osd":10,"uuid":"cdd356db-f9eb-4080-95ab-db5d8223e572","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6824","nonce":2845222366},{"type":"v1","addr":"172.21.15.142:6825","nonce":2845222366}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6826","nonce":2845222366},{"type":"v1","addr":"172.21.15.142:6827","nonce":2845222366}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6830","nonce":2845222366},{"type":"v1","addr":"172.21.15.142:6831","nonce":2845222366}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6828","nonce":2845222366},{"type":"v1","addr":"172.21.15.142:6829","nonce":2845222366}]},"public_addr":"172.21.15.142:6825/2845222366","cluster_addr":"172.21.15.142:6827/2845222366","heartbeat_back_addr":"172.21.15.142:6831/2845222366","heartbeat_front_addr":"172.21.15.142:6829/2845222366","state":["exists","up"]},{"osd":11,"uuid":"72104d1c-3ea2-45ad-bc0b-0500f146c06b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6824","nonce":1986323778},{"type":"v1","addr":"172.21.15.162:6825","nonce":1986323778}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6826","nonce":1986323778},{"type":"v1","addr":"172.21.15.162:6827","nonce":1986323778}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6830","nonce":1986323778},{"type":"v1","addr":"172.21.15.162:6831","nonce":1986323778}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6828","nonce":1986323778},{"type":"v1","addr":"172.21.15.162:6829","nonce":1986323778}]},"public_addr":"172.21.15.162:6825/1986323778","cluster_addr":"172.21.15.162:6827/1986323778","heartbeat_back_addr":"172.21.15.162:6831/1986323778","heartbeat_front_addr":"172.21.15.162:6829/1986323778","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:03:44.996331+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:04:07.994738+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:04:30.942255+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:04:55.588884+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:05:19.900600+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:05:43.807442+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:06:09.026976+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:06:34.586784+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:06:59.751605+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:07:26.144495+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:07:52.326909+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:08:17.423042+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.46:0/4021039857":"2024-04-01T23:02:08.493619+0000","172.21.15.46:6801/3681182647":"2024-04-01T23:02:08.493619+0000","172.21.15.46:0/4022214806":"2024-04-01T23:02:08.493619+0000","172.21.15.46:6800/1515528215":"2024-04-01T23:01:41.910162+0000","172.21.15.46:6800/3681182647":"2024-04-01T23:02:08.493619+0000","172.21.15.46:0/143885707":"2024-04-01T23:01:41.910162+0000","172.21.15.46:6801/1515528215":"2024-04-01T23:01:41.910162+0000","172.21.15.46:0/1680389778":"2024-04-01T23:01:41.910162+0000","172.21.15.46:0/1782133966":"2024-04-01T23:01:41.910162+0000","172.21.15.46:0/3925209376":"2024-04-01T23:02:08.493619+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-31T23:09:18.032 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:18.031+0000 7f218a48f640 1 -- 172.21.15.46:0/2093683791 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f21700a24a0 msgr2=0x7f21700a4960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:09:18.032 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:18.031+0000 7f218a48f640 1 --2- 172.21.15.46:0/2093683791 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f21700a24a0 0x7f21700a4960 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f21841016c0 tx=0x7f2174005f90 comp rx=0 tx=0).stop 2024-03-31T23:09:18.032 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:18.031+0000 7f218a48f640 1 -- 172.21.15.46:0/2093683791 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2184101860 msgr2=0x7f2184197e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:09:18.032 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:18.031+0000 7f218a48f640 1 --2- 172.21.15.46:0/2093683791 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2184101860 0x7f2184197e30 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f216c00c920 tx=0x7f216c00cde0 comp rx=0 tx=0).stop 2024-03-31T23:09:18.032 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:18.031+0000 7f218a48f640 1 -- 172.21.15.46:0/2093683791 shutdown_connections 2024-03-31T23:09:18.032 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:18.031+0000 7f218a48f640 1 --2- 172.21.15.46:0/2093683791 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f21700a24a0 0x7f21700a4960 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:18.032 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:18.031+0000 7f218a48f640 1 --2- 172.21.15.46:0/2093683791 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2184102220 0x7f218419eea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:18.032 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:18.031+0000 7f218a48f640 1 --2- 172.21.15.46:0/2093683791 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2184101860 0x7f2184197e30 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:18.032 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:18.031+0000 7f218a48f640 1 --2- 172.21.15.46:0/2093683791 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2184100670 0x7f21841978f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:18.032 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:18.031+0000 7f218a48f640 1 -- 172.21.15.46:0/2093683791 >> 172.21.15.46:0/2093683791 conn(0x7f21840fbc00 msgr2=0x7f21840fd650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:09:18.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:18.031+0000 7f218a48f640 1 -- 172.21.15.46:0/2093683791 shutdown_connections 2024-03-31T23:09:18.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:18.031+0000 7f218a48f640 1 -- 172.21.15.46:0/2093683791 wait complete. 2024-03-31T23:09:18.045 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 17, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-31T23:09:11.009090+0000', 'modified': '2024-03-31T23:09:15.556861+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 24515}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24515': {'gid': 24515, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.142:6839/2519143918', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.142:6838', 'nonce': 2519143918}, {'type': 'v1', 'addr': '172.21.15.142:6839', 'nonce': 2519143918}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 1, 'qdb_leader': 24515, 'qdb_cluster': [24515]} 2024-03-31T23:09:18.045 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-31T23:09:18.045 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-03-31T23:09:18.151 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:18.148+0000 7fddda030640 1 -- 172.21.15.46:0/1097121838 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fddd4100ff0 msgr2=0x7fddd4101470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:09:18.151 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:18.148+0000 7fddda030640 1 --2- 172.21.15.46:0/1097121838 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fddd4100ff0 0x7fddd4101470 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fddc800aa10 tx=0x7fddc802cf70 comp rx=0 tx=0).stop 2024-03-31T23:09:18.151 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:18.148+0000 7fddda030640 1 -- 172.21.15.46:0/1097121838 shutdown_connections 2024-03-31T23:09:18.151 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:18.148+0000 7fddda030640 1 --2- 172.21.15.46:0/1097121838 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fddd41019b0 0x7fddd4108240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:18.151 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:18.148+0000 7fddda030640 1 --2- 172.21.15.46:0/1097121838 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fddd4100ff0 0x7fddd4101470 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:18.151 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:18.148+0000 7fddda030640 1 --2- 172.21.15.46:0/1097121838 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fddd40ffe00 0x7fddd4100220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:18.152 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:18.148+0000 7fddda030640 1 -- 172.21.15.46:0/1097121838 >> 172.21.15.46:0/1097121838 conn(0x7fddd40fb350 msgr2=0x7fddd40fd7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:09:18.152 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:18.148+0000 7fddda030640 1 -- 172.21.15.46:0/1097121838 shutdown_connections 2024-03-31T23:09:18.152 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:18.148+0000 7fddda030640 1 -- 172.21.15.46:0/1097121838 wait complete. 2024-03-31T23:09:18.152 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:18.148+0000 7fddda030640 1 Processor -- start 2024-03-31T23:09:18.152 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:18.148+0000 7fddda030640 1 -- start start 2024-03-31T23:09:18.152 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:18.148+0000 7fddda030640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fddd40ffe00 0x7fddd41979e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:09:18.152 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:18.149+0000 7fddda030640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fddd4100ff0 0x7fddd4197f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:09:18.152 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:18.149+0000 7fddda030640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fddd41019b0 0x7fddd419ef90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:09:18.152 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:18.149+0000 7fddda030640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fddd410b0a0 con 0x7fddd41019b0 2024-03-31T23:09:18.152 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:18.149+0000 7fddda030640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fddd410af20 con 0x7fddd40ffe00 2024-03-31T23:09:18.153 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:18.149+0000 7fddda030640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fddd410b220 con 0x7fddd4100ff0 2024-03-31T23:09:18.153 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:18.149+0000 7fddd3fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fddd41019b0 0x7fddd419ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:09:18.153 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:18.149+0000 7fddd3fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fddd41019b0 0x7fddd419ef90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41260/0 (socket says 172.21.15.46:41260) 2024-03-31T23:09:18.153 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:18.149+0000 7fddd3fff640 1 -- 172.21.15.46:0/811719635 learned_addr learned my addr 172.21.15.46:0/811719635 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:09:18.153 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:18.149+0000 7fddd37fe640 1 --2- 172.21.15.46:0/811719635 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fddd40ffe00 0x7fddd41979e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:09:18.153 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:18.149+0000 7fddd2ffd640 1 --2- 172.21.15.46:0/811719635 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fddd4100ff0 0x7fddd4197f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:09:18.153 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:18.149+0000 7fddd3fff640 1 -- 172.21.15.46:0/811719635 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fddd4100ff0 msgr2=0x7fddd4197f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:09:18.153 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:18.149+0000 7fddd3fff640 1 --2- 172.21.15.46:0/811719635 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fddd4100ff0 0x7fddd4197f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:18.153 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:18.149+0000 7fddd3fff640 1 -- 172.21.15.46:0/811719635 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fddd40ffe00 msgr2=0x7fddd41979e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:09:18.153 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:18.149+0000 7fddd3fff640 1 --2- 172.21.15.46:0/811719635 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fddd40ffe00 0x7fddd41979e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:18.153 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:18.149+0000 7fddd3fff640 1 -- 172.21.15.46:0/811719635 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fddd419f600 con 0x7fddd41019b0 2024-03-31T23:09:18.154 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:18.149+0000 7fddd3fff640 1 --2- 172.21.15.46:0/811719635 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fddd41019b0 0x7fddd419ef90 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fddcc00eaa0 tx=0x7fddcc00ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:09:18.154 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:18.149+0000 7fddd0ff9640 1 -- 172.21.15.46:0/811719635 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fddcc00cdb0 con 0x7fddd41019b0 2024-03-31T23:09:18.154 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:18.149+0000 7fddd0ff9640 1 -- 172.21.15.46:0/811719635 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fddcc005020 con 0x7fddd41019b0 2024-03-31T23:09:18.154 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:18.149+0000 7fddda030640 1 -- 172.21.15.46:0/811719635 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fddd419f8f0 con 0x7fddd41019b0 2024-03-31T23:09:18.154 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:18.149+0000 7fddd0ff9640 1 -- 172.21.15.46:0/811719635 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fddcc0116d0 con 0x7fddd41019b0 2024-03-31T23:09:18.154 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:18.149+0000 7fddda030640 1 -- 172.21.15.46:0/811719635 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fddd419fe40 con 0x7fddd41019b0 2024-03-31T23:09:18.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:18.151+0000 7fddd0ff9640 1 -- 172.21.15.46:0/811719635 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 16) ==== 137320+0+0 (secure 0 0 0) 0x7fddcc0051d0 con 0x7fddd41019b0 2024-03-31T23:09:18.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:18.152+0000 7fdddcc0b740 1 Processor -- start 2024-03-31T23:09:18.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:18.152+0000 7fdddcc0b740 1 -- start start 2024-03-31T23:09:18.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:18.152+0000 7fdddcc0b740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5643b8c22610 0x5643b8c22a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:09:18.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:18.152+0000 7fdddcc0b740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5643b8c237e0 0x5643b8c23c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:09:18.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:18.152+0000 7fdddcc0b740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5643b8c241a0 0x5643b8c2aa30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:09:18.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:18.152+0000 7fddd0ff9640 1 --2- 172.21.15.46:0/811719635 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fddbc0a24a0 0x7fddbc0a4960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:09:18.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:18.152+0000 7fdddcc0b740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x5643b8b86af0 con 0x5643b8c22610 2024-03-31T23:09:18.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:18.152+0000 7fdddcc0b740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x5643b8b2f9d0 con 0x5643b8c241a0 2024-03-31T23:09:18.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:18.152+0000 7fdddcc0b740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5643b8b82b90 con 0x5643b8c237e0 2024-03-31T23:09:18.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:18.152+0000 7fddd2ffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5643b8c22610 0x5643b8c22a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:09:18.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:18.152+0000 7fddd2ffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5643b8c22610 0x5643b8c22a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41264/0 (socket says 172.21.15.46:41264) 2024-03-31T23:09:18.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:18.152+0000 7fddd2ffd640 1 -- 172.21.15.46:0/2480399720 learned_addr learned my addr 172.21.15.46:0/2480399720 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:09:18.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:18.152+0000 7fddd2ffd640 1 --2- 172.21.15.46:0/811719635 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fddbc0a24a0 0x7fddbc0a4960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:09:18.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:18.152+0000 7fddd3fff640 1 --2- 172.21.15.46:0/2480399720 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5643b8c241a0 0x5643b8c2aa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:09:18.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:18.152+0000 7fddd0ff9640 1 -- 172.21.15.46:0/811719635 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8802+0+0 (secure 0 0 0) 0x7fddcc0ceae0 con 0x7fddd41019b0 2024-03-31T23:09:18.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:18.152+0000 7fddd37fe640 1 --2- 172.21.15.46:0/2480399720 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5643b8c237e0 0x5643b8c23c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:09:18.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:18.152+0000 7fddd2ffd640 1 -- 172.21.15.46:0/2480399720 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5643b8c237e0 msgr2=0x5643b8c23c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:09:18.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:18.152+0000 7fddd2ffd640 1 --2- 172.21.15.46:0/2480399720 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5643b8c237e0 0x5643b8c23c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:18.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:18.152+0000 7fddd2ffd640 1 -- 172.21.15.46:0/2480399720 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5643b8c241a0 msgr2=0x5643b8c2aa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:09:18.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:18.153+0000 7fddd2ffd640 1 --2- 172.21.15.46:0/2480399720 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5643b8c241a0 0x5643b8c2aa30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:18.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:18.153+0000 7fddd2ffd640 1 -- 172.21.15.46:0/2480399720 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5643b889b2d0 con 0x5643b8c22610 2024-03-31T23:09:18.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:18.153+0000 7fddd2ffd640 1 --2- 172.21.15.46:0/2480399720 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5643b8c22610 0x5643b8c22a30 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fddc800a7d0 tx=0x7fddc802def0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c7b4db51c6857765 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:09:18.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:18.153+0000 7fddd2ffd640 1 --2- 172.21.15.46:0/811719635 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fddbc0a24a0 0x7fddbc0a4960 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fddc8004d40 tx=0x7fddc8008000 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:09:18.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:18.153+0000 7fddb9ffb640 1 -- 172.21.15.46:0/2480399720 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fddc8051070 con 0x5643b8c22610 2024-03-31T23:09:18.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:18.153+0000 7fddb9ffb640 1 -- 172.21.15.46:0/2480399720 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fddc8035670 con 0x5643b8c22610 2024-03-31T23:09:18.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:18.153+0000 7fddb9ffb640 1 -- 172.21.15.46:0/2480399720 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fddc8003ac0 con 0x5643b8c22610 2024-03-31T23:09:18.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:18.154+0000 7fdddcc0b740 1 -- 172.21.15.46:0/2480399720 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5643b8c22610 msgr2=0x5643b8c22a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:09:18.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:18.154+0000 7fdddcc0b740 1 --2- 172.21.15.46:0/2480399720 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5643b8c22610 0x5643b8c22a30 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fddc800a7d0 tx=0x7fddc802def0 comp rx=0 tx=0).stop 2024-03-31T23:09:18.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:18.154+0000 7fdddcc0b740 1 -- 172.21.15.46:0/2480399720 shutdown_connections 2024-03-31T23:09:18.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:18.154+0000 7fdddcc0b740 1 --2- 172.21.15.46:0/2480399720 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5643b8c241a0 0x5643b8c2aa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:18.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:18.154+0000 7fdddcc0b740 1 --2- 172.21.15.46:0/2480399720 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5643b8c237e0 0x5643b8c23c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:18.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:18.154+0000 7fdddcc0b740 1 --2- 172.21.15.46:0/2480399720 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5643b8c22610 0x5643b8c22a30 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:18.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:18.154+0000 7fdddcc0b740 1 -- 172.21.15.46:0/2480399720 >> 172.21.15.46:0/2480399720 conn(0x5643b8c1da00 msgr2=0x5643b8c1de00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:09:18.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:18.154+0000 7fdddcc0b740 1 -- 172.21.15.46:0/2480399720 shutdown_connections 2024-03-31T23:09:18.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:18.154+0000 7fdddcc0b740 1 -- 172.21.15.46:0/2480399720 wait complete. 2024-03-31T23:09:18.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:18.154+0000 7fdddcc0b740 1 -- start start 2024-03-31T23:09:18.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:18.154+0000 7fdddcc0b740 1 Processor -- start 2024-03-31T23:09:18.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:18.154+0000 7fdddcc0b740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5643b8c22610 0x5643b8c2c920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:09:18.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:18.154+0000 7fdddcc0b740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5643b8c237e0 0x5643b8c2ce60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:09:18.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:18.154+0000 7fdddcc0b740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5643b8c241a0 0x5643b8c33ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:09:18.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:18.154+0000 7fdddcc0b740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x5643b8c2b150 con 0x5643b8c241a0 2024-03-31T23:09:18.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:18.154+0000 7fdddcc0b740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x5643b8b42f70 con 0x5643b8c22610 2024-03-31T23:09:18.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:18.154+0000 7fdddcc0b740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5643b8c2b2d0 con 0x5643b8c237e0 2024-03-31T23:09:18.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:18.154+0000 7fdddcc0b740 10 client.0 _ll_register_callbacks cb 0x5643b889aac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T23:09:18.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:18.154+0000 7fddd3fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5643b8c241a0 0x5643b8c33ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:09:18.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:18.155+0000 7fddd3fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5643b8c241a0 0x5643b8c33ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41276/0 (socket says 172.21.15.46:41276) 2024-03-31T23:09:18.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:18.155+0000 7fddd37fe640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5643b8c22610 0x5643b8c2c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:09:18.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:18.155+0000 7fddd3fff640 1 -- 172.21.15.46:0/2821434779 learned_addr learned my addr 172.21.15.46:0/2821434779 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:09:18.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:18.155+0000 7fddd2ffd640 1 --2- 172.21.15.46:0/2821434779 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5643b8c237e0 0x5643b8c2ce60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:09:18.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:18.155+0000 7fddd3fff640 1 -- 172.21.15.46:0/2821434779 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5643b8c237e0 msgr2=0x5643b8c2ce60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:09:18.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:18.155+0000 7fddd3fff640 1 --2- 172.21.15.46:0/2821434779 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5643b8c237e0 0x5643b8c2ce60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:18.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:18.155+0000 7fddd3fff640 1 -- 172.21.15.46:0/2821434779 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5643b8c22610 msgr2=0x5643b8c2c920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:09:18.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:18.155+0000 7fddd3fff640 1 --2- 172.21.15.46:0/2821434779 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5643b8c22610 0x5643b8c2c920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:18.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:18.155+0000 7fddd3fff640 1 -- 172.21.15.46:0/2821434779 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5643b89656b0 con 0x5643b8c241a0 2024-03-31T23:09:18.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:18.155+0000 7fddd3fff640 1 --2- 172.21.15.46:0/2821434779 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5643b8c241a0 0x5643b8c33ea0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fddcc0977c0 tx=0x7fddcc00af00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a0e4eaff266f1cf0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:09:18.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:18.155+0000 7fddb2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-31T23:09:18.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:18.155+0000 7fddb2ffd640 1 -- 172.21.15.46:0/2821434779 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fddcc09b360 con 0x5643b8c241a0 2024-03-31T23:09:18.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:18.155+0000 7fddb2ffd640 1 -- 172.21.15.46:0/2821434779 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8802+0+0 (secure 0 0 0) 0x7fddcc0bb0c0 con 0x5643b8c241a0 2024-03-31T23:09:18.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:18.155+0000 7fdddcc0b740 1 -- 172.21.15.46:0/2821434779 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fddc8035670 con 0x5643b8c241a0 2024-03-31T23:09:18.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:18.155+0000 7fddb2ffd640 1 -- 172.21.15.46:0/2821434779 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) ==== 24+0+0 (secure 0 0 0) 0x7fddcc09f4b0 con 0x5643b8c241a0 2024-03-31T23:09:18.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:18.155+0000 7fdddcc0b740 10 client.14766 fetch_fsmap learned FSMap version 17 2024-03-31T23:09:18.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:18.155+0000 7fdddcc0b740 1 -- 172.21.15.46:0/2821434779 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=4+}) -- 0x5643b8c36ef0 con 0x5643b8c241a0 2024-03-31T23:09:18.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:18.155+0000 7fddb2ffd640 1 -- 172.21.15.46:0/2821434779 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fddc8035670 con 0x5643b8c241a0 2024-03-31T23:09:18.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:18.155+0000 7fddb2ffd640 1 -- 172.21.15.46:0/2821434779 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 17) ==== 8342+0+0 (secure 0 0 0) 0x7fddcc00f040 con 0x5643b8c241a0 2024-03-31T23:09:18.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:18.156+0000 7fdddcc0b740 10 client.14766 fetch_fsmap finished waiting for FSMap version 17 2024-03-31T23:09:18.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:18.156+0000 7fdddcc0b740 10 client.14766 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-31T23:09:18.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:18.156+0000 7fdddcc0b740 20 client.14766 populate_metadata read hostname 'smithi046' 2024-03-31T23:09:18.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:18.156+0000 7fdddcc0b740 1 --2- 172.21.15.46:0/2821434779 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x5643b8c395e0 0x5643b8c3aa50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:09:18.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:18.156+0000 7fdddcc0b740 4 client.14766 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-31T23:09:18.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:18.156+0000 7fdddcc0b740 1 -- 172.21.15.46:0/2821434779 --> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5643b8b94030 con 0x5643b8c395e0 2024-03-31T23:09:18.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:18.156+0000 7fddd37fe640 1 --2- 172.21.15.46:0/2821434779 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x5643b8c395e0 0x5643b8c3aa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:09:18.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:18.156+0000 7fddb2ffd640 10 client.14766 ms_handle_connect on v2:172.21.15.142:6838/2519143918 2024-03-31T23:09:18.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:18.156+0000 7fddd37fe640 1 --2- 172.21.15.46:0/2821434779 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x5643b8c395e0 0x5643b8c3aa50 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=10cfa45f1d22735b server_cookie=f31b27482af6fe50 in_seq=0 out_seq=0 2024-03-31T23:09:18.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:18.157+0000 7fddb2ffd640 1 -- 172.21.15.46:0/2821434779 <== mds.0 v2:172.21.15.142:6838/2519143918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5643b8b94030 con 0x5643b8c395e0 2024-03-31T23:09:18.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:18.157+0000 7fddb2ffd640 10 client.14766 handle_command_reply: tid=0 2024-03-31T23:09:18.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:18.157+0000 7fdddcc0b740 1 client.14766 shutdown 2024-03-31T23:09:18.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:18.157+0000 7fdddcc0b740 1 -- 172.21.15.46:0/2821434779 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5643b8c241a0 msgr2=0x5643b8c33ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:09:18.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:18.157+0000 7fdddcc0b740 1 --2- 172.21.15.46:0/2821434779 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5643b8c241a0 0x5643b8c33ea0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fddcc0977c0 tx=0x7fddcc00af00 comp rx=0 tx=0).stop 2024-03-31T23:09:18.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:18.157+0000 7fdddcc0b740 1 -- 172.21.15.46:0/2821434779 shutdown_connections 2024-03-31T23:09:18.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:18.157+0000 7fdddcc0b740 1 --2- 172.21.15.46:0/2821434779 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x5643b8c395e0 0x5643b8c3aa50 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:18.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:18.157+0000 7fdddcc0b740 1 --2- 172.21.15.46:0/2821434779 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5643b8c241a0 0x5643b8c33ea0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:18.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:18.157+0000 7fdddcc0b740 1 --2- 172.21.15.46:0/2821434779 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5643b8c237e0 0x5643b8c2ce60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:18.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:18.157+0000 7fddb2ffd640 0 client.14766 ms_handle_reset on v2:172.21.15.142:6838/2519143918 2024-03-31T23:09:18.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:18.157+0000 7fdddcc0b740 1 --2- 172.21.15.46:0/2821434779 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5643b8c22610 0x5643b8c2c920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:18.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:18.157+0000 7fdddcc0b740 1 -- 172.21.15.46:0/2821434779 >> 172.21.15.46:0/2821434779 conn(0x5643b8c1da00 msgr2=0x5643b8c1cb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:09:18.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:18.157+0000 7fdddcc0b740 1 -- 172.21.15.46:0/2821434779 shutdown_connections 2024-03-31T23:09:18.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:18.157+0000 7fdddcc0b740 1 -- 172.21.15.46:0/2821434779 wait complete. 2024-03-31T23:09:18.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:18.157+0000 7fdddcc0b740 20 client.14766 trim_cache size 0 max 16384 2024-03-31T23:09:18.182 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:18.175+0000 7fdddcc0b740 1 Processor -- start 2024-03-31T23:09:18.182 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:18.175+0000 7fdddcc0b740 1 -- start start 2024-03-31T23:09:18.182 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:18.176+0000 7fdddcc0b740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5643b8c22610 0x5643b8c2bf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:09:18.182 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:18.176+0000 7fdddcc0b740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5643b8c237e0 0x5643b8c33e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:09:18.182 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:18.176+0000 7fdddcc0b740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5643b8c241a0 0x5643b8c2cc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:09:18.182 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:18.176+0000 7fdddcc0b740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x5643b8c38d90 con 0x5643b8c241a0 2024-03-31T23:09:18.182 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:18.176+0000 7fdddcc0b740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x5643b8c39460 con 0x5643b8c237e0 2024-03-31T23:09:18.182 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:18.176+0000 7fdddcc0b740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5643b8c39b30 con 0x5643b8c22610 2024-03-31T23:09:18.182 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:18.176+0000 7fddd3fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5643b8c241a0 0x5643b8c2cc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:09:18.182 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:18.176+0000 7fddd3fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5643b8c241a0 0x5643b8c2cc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41284/0 (socket says 172.21.15.46:41284) 2024-03-31T23:09:18.183 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:18.176+0000 7fddd3fff640 1 -- 172.21.15.46:0/1990521553 learned_addr learned my addr 172.21.15.46:0/1990521553 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:09:18.183 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:18.176+0000 7fddd2ffd640 1 --2- 172.21.15.46:0/1990521553 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5643b8c237e0 0x5643b8c33e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:09:18.183 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:18.176+0000 7fddd3fff640 1 -- 172.21.15.46:0/1990521553 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5643b8c22610 msgr2=0x5643b8c2bf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:09:18.183 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:18.176+0000 7fddd3fff640 1 --2- 172.21.15.46:0/1990521553 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5643b8c22610 0x5643b8c2bf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:18.183 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:18.176+0000 7fddd3fff640 1 -- 172.21.15.46:0/1990521553 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5643b8c237e0 msgr2=0x5643b8c33e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:09:18.183 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:18.176+0000 7fddd3fff640 1 --2- 172.21.15.46:0/1990521553 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5643b8c237e0 0x5643b8c33e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:18.183 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:18.176+0000 7fddd3fff640 1 -- 172.21.15.46:0/1990521553 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5643b8c220d0 con 0x5643b8c241a0 2024-03-31T23:09:18.183 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:18.176+0000 7fddd3fff640 1 --2- 172.21.15.46:0/1990521553 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5643b8c241a0 0x5643b8c2cc40 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fddcc097d30 tx=0x7fddcc0cee90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d84cfbf5eb332578 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:09:18.183 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:18.176+0000 7fddb97fa640 1 -- 172.21.15.46:0/1990521553 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fddcc09b360 con 0x5643b8c241a0 2024-03-31T23:09:18.183 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:18.176+0000 7fddb97fa640 1 -- 172.21.15.46:0/1990521553 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fddcc0071c0 con 0x5643b8c241a0 2024-03-31T23:09:18.183 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:18.176+0000 7fddb97fa640 1 -- 172.21.15.46:0/1990521553 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fddcc01edf0 con 0x5643b8c241a0 2024-03-31T23:09:18.183 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:18.177+0000 7fdddcc0b740 1 -- 172.21.15.46:0/1990521553 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5643b8c241a0 msgr2=0x5643b8c2cc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:09:18.183 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:18.177+0000 7fdddcc0b740 1 --2- 172.21.15.46:0/1990521553 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5643b8c241a0 0x5643b8c2cc40 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fddcc097d30 tx=0x7fddcc0cee90 comp rx=0 tx=0).stop 2024-03-31T23:09:18.183 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:18.177+0000 7fdddcc0b740 1 -- 172.21.15.46:0/1990521553 shutdown_connections 2024-03-31T23:09:18.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:18.177+0000 7fdddcc0b740 1 --2- 172.21.15.46:0/1990521553 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5643b8c241a0 0x5643b8c2cc40 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:18.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:18.177+0000 7fdddcc0b740 1 --2- 172.21.15.46:0/1990521553 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5643b8c237e0 0x5643b8c33e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:18.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:18.177+0000 7fdddcc0b740 1 --2- 172.21.15.46:0/1990521553 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5643b8c22610 0x5643b8c2bf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:18.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:18.177+0000 7fdddcc0b740 1 -- 172.21.15.46:0/1990521553 >> 172.21.15.46:0/1990521553 conn(0x5643b8c1da00 msgr2=0x5643b8c2c530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:09:18.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:18.177+0000 7fdddcc0b740 1 -- 172.21.15.46:0/1990521553 shutdown_connections 2024-03-31T23:09:18.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:18.177+0000 7fdddcc0b740 1 -- 172.21.15.46:0/1990521553 wait complete. 2024-03-31T23:09:18.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:18.177+0000 7fdddcc0b740 1 -- start start 2024-03-31T23:09:18.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:18.177+0000 7fdddcc0b740 1 Processor -- start 2024-03-31T23:09:18.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:18.178+0000 7fdddcc0b740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5643b8c22610 0x5643b8c30090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:09:18.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:18.178+0000 7fdddcc0b740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5643b8c237e0 0x5643b8c30630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:09:18.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:18.178+0000 7fdddcc0b740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5643b8c241a0 0x5643b8c33320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:09:18.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:18.178+0000 7fdddcc0b740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x5643b8c3a8d0 con 0x5643b8c237e0 2024-03-31T23:09:18.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:18.178+0000 7fdddcc0b740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x5643b8c3a200 con 0x5643b8c22610 2024-03-31T23:09:18.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:18.178+0000 7fdddcc0b740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5643b8c3afa0 con 0x5643b8c241a0 2024-03-31T23:09:18.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:18.178+0000 7fdddcc0b740 10 client.0 _ll_register_callbacks cb 0x7fddcc0cede0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T23:09:18.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:18.178+0000 7fddd2ffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5643b8c237e0 0x5643b8c30630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:09:18.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:18.178+0000 7fddd2ffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5643b8c237e0 0x5643b8c30630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41292/0 (socket says 172.21.15.46:41292) 2024-03-31T23:09:18.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:18.178+0000 7fddd2ffd640 1 -- 172.21.15.46:0/169789430 learned_addr learned my addr 172.21.15.46:0/169789430 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:09:18.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:18.178+0000 7fddd37fe640 1 --2- 172.21.15.46:0/169789430 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5643b8c22610 0x5643b8c30090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:09:18.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:18.178+0000 7fddd3fff640 1 --2- 172.21.15.46:0/169789430 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5643b8c241a0 0x5643b8c33320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:09:18.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:18.178+0000 7fddd2ffd640 1 -- 172.21.15.46:0/169789430 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5643b8c241a0 msgr2=0x5643b8c33320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:09:18.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:18.178+0000 7fddd2ffd640 1 --2- 172.21.15.46:0/169789430 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5643b8c241a0 0x5643b8c33320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:18.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:18.178+0000 7fddd2ffd640 1 -- 172.21.15.46:0/169789430 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5643b8c22610 msgr2=0x5643b8c30090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:09:18.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:18.178+0000 7fddd2ffd640 1 --2- 172.21.15.46:0/169789430 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5643b8c22610 0x5643b8c30090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:18.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:18.178+0000 7fddd2ffd640 1 -- 172.21.15.46:0/169789430 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5643b8c33930 con 0x5643b8c237e0 2024-03-31T23:09:18.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:18.178+0000 7fddd2ffd640 1 --2- 172.21.15.46:0/169789430 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5643b8c237e0 0x5643b8c30630 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fddc802d4b0 tx=0x7fddc802de80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4dd4465f1f604ae4 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:09:18.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:18.178+0000 7fddb2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-31T23:09:18.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:18.178+0000 7fddb2ffd640 1 -- 172.21.15.46:0/169789430 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fddc8059020 con 0x5643b8c237e0 2024-03-31T23:09:18.186 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:18.178+0000 7fdddcc0b740 1 -- 172.21.15.46:0/169789430 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fddcc0071c0 con 0x5643b8c237e0 2024-03-31T23:09:18.186 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:18.178+0000 7fddb2ffd640 1 -- 172.21.15.46:0/169789430 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8802+0+0 (secure 0 0 0) 0x7fddc8003d00 con 0x5643b8c237e0 2024-03-31T23:09:18.186 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:18.178+0000 7fddb2ffd640 1 -- 172.21.15.46:0/169789430 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) ==== 24+0+0 (secure 0 0 0) 0x7fddc8051040 con 0x5643b8c237e0 2024-03-31T23:09:18.186 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:18.178+0000 7fdddcc0b740 10 client.14772 fetch_fsmap learned FSMap version 17 2024-03-31T23:09:18.186 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:18.178+0000 7fdddcc0b740 1 -- 172.21.15.46:0/169789430 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=4+}) -- 0x5643b8c9fb90 con 0x5643b8c237e0 2024-03-31T23:09:18.186 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:18.179+0000 7fddb2ffd640 1 -- 172.21.15.46:0/169789430 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fddcc0071c0 con 0x5643b8c237e0 2024-03-31T23:09:18.186 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:18.179+0000 7fddb2ffd640 1 -- 172.21.15.46:0/169789430 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 17) ==== 8342+0+0 (secure 0 0 0) 0x7fddc804b070 con 0x5643b8c237e0 2024-03-31T23:09:18.186 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:18.179+0000 7fdddcc0b740 10 client.14772 fetch_fsmap finished waiting for FSMap version 17 2024-03-31T23:09:18.186 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:18.179+0000 7fdddcc0b740 10 client.14772 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-31T23:09:18.186 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:18.179+0000 7fdddcc0b740 20 client.14772 populate_metadata read hostname 'smithi046' 2024-03-31T23:09:18.186 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:18.179+0000 7fdddcc0b740 1 --2- 172.21.15.46:0/169789430 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x5643b8ca4350 0x5643b8ca3060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:09:18.186 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:18.179+0000 7fdddcc0b740 4 client.14772 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-31T23:09:18.186 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:18.179+0000 7fdddcc0b740 1 -- 172.21.15.46:0/169789430 --> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] -- command(tid 0: {"prefix": "status"}) -- 0x5643b8b8e380 con 0x5643b8ca4350 2024-03-31T23:09:18.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:18.179+0000 7fddd37fe640 1 --2- 172.21.15.46:0/169789430 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x5643b8ca4350 0x5643b8ca3060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:09:18.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:18.179+0000 7fddd37fe640 1 --2- 172.21.15.46:0/169789430 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x5643b8ca4350 0x5643b8ca3060 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=441d10060f9b0033 server_cookie=708419449f1f16f5 in_seq=0 out_seq=0 2024-03-31T23:09:18.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:18.179+0000 7fddb2ffd640 10 client.14772 ms_handle_connect on v2:172.21.15.142:6838/2519143918 2024-03-31T23:09:18.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:18.180+0000 7fddb2ffd640 1 -- 172.21.15.46:0/169789430 <== mds.0 v2:172.21.15.142:6838/2519143918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+326 (crc 0 0 0) 0x5643b8b8e380 con 0x5643b8ca4350 2024-03-31T23:09:18.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:18.180+0000 7fddb2ffd640 10 client.14772 handle_command_reply: tid=0 2024-03-31T23:09:18.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:18.180+0000 7fdddcc0b740 1 client.14772 shutdown 2024-03-31T23:09:18.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:18.180+0000 7fdddcc0b740 1 -- 172.21.15.46:0/169789430 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5643b8c237e0 msgr2=0x5643b8c30630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:09:18.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:18.180+0000 7fdddcc0b740 1 --2- 172.21.15.46:0/169789430 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5643b8c237e0 0x5643b8c30630 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fddc802d4b0 tx=0x7fddc802de80 comp rx=0 tx=0).stop 2024-03-31T23:09:18.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:18.180+0000 7fdddcc0b740 1 -- 172.21.15.46:0/169789430 shutdown_connections 2024-03-31T23:09:18.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:18.180+0000 7fdddcc0b740 1 --2- 172.21.15.46:0/169789430 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x5643b8ca4350 0x5643b8ca3060 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:18.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:18.180+0000 7fdddcc0b740 1 --2- 172.21.15.46:0/169789430 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5643b8c241a0 0x5643b8c33320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:18.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:18.180+0000 7fdddcc0b740 1 --2- 172.21.15.46:0/169789430 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5643b8c237e0 0x5643b8c30630 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:18.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:18.180+0000 7fdddcc0b740 1 --2- 172.21.15.46:0/169789430 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5643b8c22610 0x5643b8c30090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:18.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:18.180+0000 7fddb2ffd640 0 client.14772 ms_handle_reset on v2:172.21.15.142:6838/2519143918 2024-03-31T23:09:18.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:18.180+0000 7fdddcc0b740 1 -- 172.21.15.46:0/169789430 >> 172.21.15.46:0/169789430 conn(0x5643b8c1da00 msgr2=0x5643b8c20c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:09:18.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:18.180+0000 7fdddcc0b740 1 -- 172.21.15.46:0/169789430 shutdown_connections 2024-03-31T23:09:18.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:18.180+0000 7fdddcc0b740 1 -- 172.21.15.46:0/169789430 wait complete. 2024-03-31T23:09:18.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:18.180+0000 7fdddcc0b740 20 client.14772 trim_cache size 0 max 16384 2024-03-31T23:09:18.188 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-03-31T23:09:18.188 INFO:teuthology.orchestra.run.smithi046.stdout: "cluster_fsid": "6a9d8f08-efb2-11ee-b647-cb9ed24678a4", 2024-03-31T23:09:18.188 INFO:teuthology.orchestra.run.smithi046.stdout: "whoami": 0, 2024-03-31T23:09:18.188 INFO:teuthology.orchestra.run.smithi046.stdout: "id": 24515, 2024-03-31T23:09:18.188 INFO:teuthology.orchestra.run.smithi046.stdout: "want_state": "up:active", 2024-03-31T23:09:18.188 INFO:teuthology.orchestra.run.smithi046.stdout: "state": "up:active", 2024-03-31T23:09:18.188 INFO:teuthology.orchestra.run.smithi046.stdout: "fs_name": "cephfs", 2024-03-31T23:09:18.189 INFO:teuthology.orchestra.run.smithi046.stdout: "rank_uptime": 6.5841721240000002, 2024-03-31T23:09:18.189 INFO:teuthology.orchestra.run.smithi046.stdout: "mdsmap_epoch": 17, 2024-03-31T23:09:18.189 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap_epoch": 91, 2024-03-31T23:09:18.189 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap_epoch_barrier": 90, 2024-03-31T23:09:18.189 INFO:teuthology.orchestra.run.smithi046.stdout: "uptime": 10.918646462 2024-03-31T23:09:18.189 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-03-31T23:09:18.189 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:18.180+0000 7fddda030640 1 -- 172.21.15.46:0/811719635 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fddbc0a24a0 msgr2=0x7fddbc0a4960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:09:18.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:18.180+0000 7fddda030640 1 --2- 172.21.15.46:0/811719635 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fddbc0a24a0 0x7fddbc0a4960 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fddc8004d40 tx=0x7fddc8008000 comp rx=0 tx=0).stop 2024-03-31T23:09:18.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:18.180+0000 7fddda030640 1 -- 172.21.15.46:0/811719635 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fddd41019b0 msgr2=0x7fddd419ef90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:09:18.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:18.180+0000 7fddda030640 1 --2- 172.21.15.46:0/811719635 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fddd41019b0 0x7fddd419ef90 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fddcc00eaa0 tx=0x7fddcc00ef60 comp rx=0 tx=0).stop 2024-03-31T23:09:18.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:18.180+0000 7fddda030640 1 -- 172.21.15.46:0/811719635 shutdown_connections 2024-03-31T23:09:18.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:18.180+0000 7fddda030640 1 --2- 172.21.15.46:0/811719635 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fddbc0a24a0 0x7fddbc0a4960 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:18.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:18.180+0000 7fddda030640 1 --2- 172.21.15.46:0/811719635 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fddd41019b0 0x7fddd419ef90 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:18.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:18.180+0000 7fddda030640 1 --2- 172.21.15.46:0/811719635 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fddd4100ff0 0x7fddd4197f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:18.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:18.180+0000 7fddda030640 1 --2- 172.21.15.46:0/811719635 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fddd40ffe00 0x7fddd41979e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:18.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:18.180+0000 7fddda030640 1 -- 172.21.15.46:0/811719635 >> 172.21.15.46:0/811719635 conn(0x7fddd40fb350 msgr2=0x7fddd4106530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:09:18.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:18.181+0000 7fddda030640 1 -- 172.21.15.46:0/811719635 shutdown_connections 2024-03-31T23:09:18.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:18.181+0000 7fddda030640 1 -- 172.21.15.46:0/811719635 wait complete. 2024-03-31T23:09:18.193 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-03-31T23:09:18.193 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2024-03-31T23:09:18.193 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:384f31f3d3aee7f8ba3f60846b8844c049165583 shell --fsid 6a9d8f08-efb2-11ee-b647-cb9ed24678a4 -- ceph fs ls 2024-03-31T23:09:18.316 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:09:18 smithi046 ceph-mon[32498]: pgmap v294: 129 pgs: 102 active+clean, 3 creating+peering, 24 unknown; 579 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s wr, 9 op/s 2024-03-31T23:09:18.316 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:09:18 smithi046 ceph-mon[32498]: osdmap e91: 12 total, 12 up, 12 in 2024-03-31T23:09:18.316 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:09:18 smithi046 ceph-mon[32498]: from='client.? 172.21.15.46:0/511558404' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-31T23:09:18.316 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:09:18 smithi046 ceph-mon[32498]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-31T23:09:18.316 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:09:18 smithi046 ceph-mon[32498]: Cluster is now healthy 2024-03-31T23:09:18.316 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:09:18 smithi046 ceph-mon[32498]: from='client.? 172.21.15.46:0/3973573710' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-31T23:09:18.317 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:09:18 smithi046 ceph-mon[32498]: from='client.? 172.21.15.46:0/2093683791' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-31T23:09:18.439 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/6a9d8f08-efb2-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-31T23:09:18.443 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:09:18 smithi142 ceph-mon[35271]: pgmap v294: 129 pgs: 102 active+clean, 3 creating+peering, 24 unknown; 579 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s wr, 9 op/s 2024-03-31T23:09:18.443 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:09:18 smithi142 ceph-mon[35271]: osdmap e91: 12 total, 12 up, 12 in 2024-03-31T23:09:18.443 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:09:18 smithi142 ceph-mon[35271]: from='client.? 172.21.15.46:0/511558404' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-31T23:09:18.443 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:09:18 smithi142 ceph-mon[35271]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-31T23:09:18.443 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:09:18 smithi142 ceph-mon[35271]: Cluster is now healthy 2024-03-31T23:09:18.443 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:09:18 smithi142 ceph-mon[35271]: from='client.? 172.21.15.46:0/3973573710' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-31T23:09:18.443 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:09:18 smithi142 ceph-mon[35271]: from='client.? 172.21.15.46:0/2093683791' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-31T23:09:18.546 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:09:18 smithi162 ceph-mon[36237]: pgmap v294: 129 pgs: 102 active+clean, 3 creating+peering, 24 unknown; 579 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s wr, 9 op/s 2024-03-31T23:09:18.546 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:09:18 smithi162 ceph-mon[36237]: osdmap e91: 12 total, 12 up, 12 in 2024-03-31T23:09:18.546 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:09:18 smithi162 ceph-mon[36237]: from='client.? 172.21.15.46:0/511558404' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-31T23:09:18.546 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:09:18 smithi162 ceph-mon[36237]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-31T23:09:18.546 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:09:18 smithi162 ceph-mon[36237]: Cluster is now healthy 2024-03-31T23:09:18.546 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:09:18 smithi162 ceph-mon[36237]: from='client.? 172.21.15.46:0/3973573710' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-31T23:09:18.546 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:09:18 smithi162 ceph-mon[36237]: from='client.? 172.21.15.46:0/2093683791' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-31T23:09:20.210 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:09:20 smithi046 ceph-mon[32498]: pgmap v296: 129 pgs: 122 active+clean, 3 creating+peering, 4 unknown; 579 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 697 B/s wr, 2 op/s 2024-03-31T23:09:20.218 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:20.214+0000 7f2b93bf6700 1 -- 172.21.15.46:0/2803296191 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2b8c102c50 msgr2=0x7f2b8c10f090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:09:20.218 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:20.214+0000 7f2b93bf6700 1 --2- 172.21.15.46:0/2803296191 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2b8c102c50 0x7f2b8c10f090 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f2b8400b0d0 tx=0x7f2b8400b3e0 comp rx=0 tx=0).stop 2024-03-31T23:09:20.218 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:20.214+0000 7f2b93bf6700 1 -- 172.21.15.46:0/2803296191 shutdown_connections 2024-03-31T23:09:20.218 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:20.214+0000 7f2b93bf6700 1 --2- 172.21.15.46:0/2803296191 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2b8c102c50 0x7f2b8c10f090 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:20.218 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:20.214+0000 7f2b93bf6700 1 --2- 172.21.15.46:0/2803296191 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2b8c108270 0x7f2b8c108650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:20.218 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:20.214+0000 7f2b93bf6700 1 --2- 172.21.15.46:0/2803296191 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2b8c102290 0x7f2b8c102710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:20.218 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:20.215+0000 7f2b93bf6700 1 -- 172.21.15.46:0/2803296191 >> 172.21.15.46:0/2803296191 conn(0x7f2b8c079370 msgr2=0x7f2b8c079780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:09:20.218 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:20.215+0000 7f2b93bf6700 1 -- 172.21.15.46:0/2803296191 shutdown_connections 2024-03-31T23:09:20.218 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:20.215+0000 7f2b93bf6700 1 -- 172.21.15.46:0/2803296191 wait complete. 2024-03-31T23:09:20.218 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:20.215+0000 7f2b93bf6700 1 Processor -- start 2024-03-31T23:09:20.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:20.215+0000 7f2b93bf6700 1 -- start start 2024-03-31T23:09:20.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:20.215+0000 7f2b93bf6700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2b8c102290 0x7f2b8c19e5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:09:20.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:20.215+0000 7f2b93bf6700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2b8c102c50 0x7f2b8c19eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:09:20.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:20.215+0000 7f2b93bf6700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2b8c108270 0x7f2b8c1986c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:09:20.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:20.215+0000 7f2b93bf6700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f2b8c1123a0 con 0x7f2b8c102290 2024-03-31T23:09:20.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:20.215+0000 7f2b93bf6700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f2b8c112220 con 0x7f2b8c108270 2024-03-31T23:09:20.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:20.215+0000 7f2b93bf6700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f2b8c112520 con 0x7f2b8c102c50 2024-03-31T23:09:20.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:20.216+0000 7f2b91992700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2b8c102290 0x7f2b8c19e5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:09:20.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:20.216+0000 7f2b91992700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2b8c102290 0x7f2b8c19e5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41310/0 (socket says 172.21.15.46:41310) 2024-03-31T23:09:20.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:20.216+0000 7f2b91992700 1 -- 172.21.15.46:0/2690683906 learned_addr learned my addr 172.21.15.46:0/2690683906 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:09:20.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:20.216+0000 7f2b92193700 1 --2- 172.21.15.46:0/2690683906 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2b8c108270 0x7f2b8c1986c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:09:20.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:20.216+0000 7f2b91191700 1 --2- 172.21.15.46:0/2690683906 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2b8c102c50 0x7f2b8c19eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:09:20.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:20.216+0000 7f2b91992700 1 -- 172.21.15.46:0/2690683906 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2b8c102c50 msgr2=0x7f2b8c19eb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:09:20.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:20.216+0000 7f2b91992700 1 --2- 172.21.15.46:0/2690683906 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2b8c102c50 0x7f2b8c19eb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:20.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:20.216+0000 7f2b91992700 1 -- 172.21.15.46:0/2690683906 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2b8c108270 msgr2=0x7f2b8c1986c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:09:20.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:20.216+0000 7f2b91992700 1 --2- 172.21.15.46:0/2690683906 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2b8c108270 0x7f2b8c1986c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:20.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:20.216+0000 7f2b91992700 1 -- 172.21.15.46:0/2690683906 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b8c198fa0 con 0x7f2b8c102290 2024-03-31T23:09:20.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:20.216+0000 7f2b91992700 1 --2- 172.21.15.46:0/2690683906 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2b8c102290 0x7f2b8c19e5f0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f2b8800e9a0 tx=0x7f2b8800ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:09:20.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:20.216+0000 7f2b82ffd700 1 -- 172.21.15.46:0/2690683906 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b8800cbd0 con 0x7f2b8c102290 2024-03-31T23:09:20.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:20.216+0000 7f2b82ffd700 1 -- 172.21.15.46:0/2690683906 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f2b8800cd70 con 0x7f2b8c102290 2024-03-31T23:09:20.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:20.216+0000 7f2b93bf6700 1 -- 172.21.15.46:0/2690683906 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2b8c199290 con 0x7f2b8c102290 2024-03-31T23:09:20.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:20.216+0000 7f2b82ffd700 1 -- 172.21.15.46:0/2690683906 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b88018680 con 0x7f2b8c102290 2024-03-31T23:09:20.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:20.216+0000 7f2b93bf6700 1 -- 172.21.15.46:0/2690683906 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2b8c1a5570 con 0x7f2b8c102290 2024-03-31T23:09:20.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:20.217+0000 7f2b93bf6700 1 -- 172.21.15.46:0/2690683906 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2b8c103920 con 0x7f2b8c102290 2024-03-31T23:09:20.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:20.218+0000 7f2b82ffd700 1 -- 172.21.15.46:0/2690683906 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 16) ==== 137320+0+0 (secure 0 0 0) 0x7f2b88010430 con 0x7f2b8c102290 2024-03-31T23:09:20.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:20.219+0000 7f2b82ffd700 1 --2- 172.21.15.46:0/2690683906 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f2b780a25b0 0x7f2b780a4a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:09:20.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:20.219+0000 7f2b91191700 1 --2- 172.21.15.46:0/2690683906 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f2b780a25b0 0x7f2b780a4a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:09:20.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:20.219+0000 7f2b82ffd700 1 -- 172.21.15.46:0/2690683906 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8802+0+0 (secure 0 0 0) 0x7f2b880cef60 con 0x7f2b8c102290 2024-03-31T23:09:20.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:20.219+0000 7f2b91191700 1 --2- 172.21.15.46:0/2690683906 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f2b780a25b0 0x7f2b780a4a70 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f2b7c005950 tx=0x7f2b7c0058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:09:20.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:20.223+0000 7f2b82ffd700 1 -- 172.21.15.46:0/2690683906 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f2b8809bee0 con 0x7f2b8c102290 2024-03-31T23:09:20.443 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:09:20 smithi142 ceph-mon[35271]: pgmap v296: 129 pgs: 122 active+clean, 3 creating+peering, 4 unknown; 579 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 697 B/s wr, 2 op/s 2024-03-31T23:09:20.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:20.473+0000 7f2b93bf6700 1 -- 172.21.15.46:0/2690683906 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f2b8c199bf0 con 0x7f2b8c102290 2024-03-31T23:09:20.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:20.473+0000 7f2b82ffd700 1 -- 172.21.15.46:0/2690683906 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v17) ==== 53+0+88 (secure 0 0 0) 0x7f2b8809b7e0 con 0x7f2b8c102290 2024-03-31T23:09:20.474 INFO:teuthology.orchestra.run.smithi046.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-03-31T23:09:20.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:20.476+0000 7f2b93bf6700 1 -- 172.21.15.46:0/2690683906 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f2b780a25b0 msgr2=0x7f2b780a4a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:09:20.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:20.476+0000 7f2b93bf6700 1 --2- 172.21.15.46:0/2690683906 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f2b780a25b0 0x7f2b780a4a70 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f2b7c005950 tx=0x7f2b7c0058e0 comp rx=0 tx=0).stop 2024-03-31T23:09:20.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:20.476+0000 7f2b93bf6700 1 -- 172.21.15.46:0/2690683906 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2b8c102290 msgr2=0x7f2b8c19e5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:09:20.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:20.476+0000 7f2b93bf6700 1 --2- 172.21.15.46:0/2690683906 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2b8c102290 0x7f2b8c19e5f0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f2b8800e9a0 tx=0x7f2b8800ecb0 comp rx=0 tx=0).stop 2024-03-31T23:09:20.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:20.476+0000 7f2b93bf6700 1 -- 172.21.15.46:0/2690683906 shutdown_connections 2024-03-31T23:09:20.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:20.476+0000 7f2b93bf6700 1 --2- 172.21.15.46:0/2690683906 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f2b780a25b0 0x7f2b780a4a70 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:20.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:20.476+0000 7f2b93bf6700 1 --2- 172.21.15.46:0/2690683906 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2b8c102290 0x7f2b8c19e5f0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:20.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:20.476+0000 7f2b93bf6700 1 --2- 172.21.15.46:0/2690683906 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2b8c108270 0x7f2b8c1986c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:20.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:20.476+0000 7f2b93bf6700 1 --2- 172.21.15.46:0/2690683906 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2b8c102c50 0x7f2b8c19eb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:20.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:20.476+0000 7f2b93bf6700 1 -- 172.21.15.46:0/2690683906 >> 172.21.15.46:0/2690683906 conn(0x7f2b8c079370 msgr2=0x7f2b8c100fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:09:20.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:20.477+0000 7f2b93bf6700 1 -- 172.21.15.46:0/2690683906 shutdown_connections 2024-03-31T23:09:20.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:20.477+0000 7f2b93bf6700 1 -- 172.21.15.46:0/2690683906 wait complete. 2024-03-31T23:09:20.545 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:09:20 smithi162 ceph-mon[36237]: pgmap v296: 129 pgs: 122 active+clean, 3 creating+peering, 4 unknown; 579 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 697 B/s wr, 2 op/s 2024-03-31T23:09:21.145 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-03-31T23:09:21.145 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-31T23:09:21.145 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi046.front.sepia.ceph.com 2024-03-31T23:09:21.145 INFO:tasks.cephfs.mount:self.client.name = client.admin 2024-03-31T23:09:21.145 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2024-03-31T23:09:21.146 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-03-31T23:09:21.146 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-31T23:09:21.146 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-31T23:09:21.146 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.admin' 2024-03-31T23:09:21.146 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T23:09:21.146 DEBUG:teuthology.orchestra.run.smithi046:> ip addr 2024-03-31T23:09:21.163 INFO:teuthology.orchestra.run.smithi046.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-31T23:09:21.163 INFO:teuthology.orchestra.run.smithi046.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-31T23:09:21.163 INFO:teuthology.orchestra.run.smithi046.stdout: inet 127.0.0.1/8 scope host lo 2024-03-31T23:09:21.164 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-03-31T23:09:21.164 INFO:teuthology.orchestra.run.smithi046.stdout: inet6 ::1/128 scope host 2024-03-31T23:09:21.164 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-03-31T23:09:21.164 INFO:teuthology.orchestra.run.smithi046.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-31T23:09:21.164 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 0c:c4:7a:6b:f6:10 brd ff:ff:ff:ff:ff:ff 2024-03-31T23:09:21.164 INFO:teuthology.orchestra.run.smithi046.stdout: altname enp7s0f0 2024-03-31T23:09:21.164 INFO:teuthology.orchestra.run.smithi046.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-31T23:09:21.164 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 0c:c4:7a:6b:f6:11 brd ff:ff:ff:ff:ff:ff 2024-03-31T23:09:21.164 INFO:teuthology.orchestra.run.smithi046.stdout: altname enp7s0f1 2024-03-31T23:09:21.164 INFO:teuthology.orchestra.run.smithi046.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-31T23:09:21.164 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 0c:c4:7a:bd:15:f8 brd ff:ff:ff:ff:ff:ff 2024-03-31T23:09:21.164 INFO:teuthology.orchestra.run.smithi046.stdout: altname enp1s0f0 2024-03-31T23:09:21.164 INFO:teuthology.orchestra.run.smithi046.stdout: inet 172.21.15.46/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-03-31T23:09:21.164 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft 42250sec preferred_lft 42250sec 2024-03-31T23:09:21.165 INFO:teuthology.orchestra.run.smithi046.stdout: inet6 fe80::ec4:7aff:febd:15f8/64 scope link noprefixroute 2024-03-31T23:09:21.165 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-03-31T23:09:21.165 INFO:teuthology.orchestra.run.smithi046.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-31T23:09:21.165 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 0c:c4:7a:bd:15:f9 brd ff:ff:ff:ff:ff:ff 2024-03-31T23:09:21.165 INFO:teuthology.orchestra.run.smithi046.stdout: altname enp1s0f1 2024-03-31T23:09:21.165 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-03-31T23:09:21.166 DEBUG:teuthology.orchestra.run.smithi046:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-31T23:09:21.166 DEBUG:teuthology.orchestra.run.smithi046:> set -e 2024-03-31T23:09:21.166 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip link add name ceph-brx type bridge 2024-03-31T23:09:21.166 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip addr flush dev ceph-brx 2024-03-31T23:09:21.166 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip link set ceph-brx up 2024-03-31T23:09:21.166 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-03-31T23:09:21.166 DEBUG:teuthology.orchestra.run.smithi046:> ') 2024-03-31T23:09:21.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:21 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-31T23:09:21.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:21 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-31T23:09:21.414 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T23:09:21.414 DEBUG:teuthology.orchestra.run.smithi046:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-31T23:09:21.442 INFO:teuthology.orchestra.run.smithi046.stdout:1 2024-03-31T23:09:21.444 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T23:09:21.444 DEBUG:teuthology.orchestra.run.smithi046:> route 2024-03-31T23:09:21.533 INFO:teuthology.orchestra.run.smithi046.stdout:Kernel IP routing table 2024-03-31T23:09:21.533 INFO:teuthology.orchestra.run.smithi046.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-31T23:09:21.533 INFO:teuthology.orchestra.run.smithi046.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-03-31T23:09:21.534 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-03-31T23:09:21.534 INFO:teuthology.orchestra.run.smithi046.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-31T23:09:21.534 DEBUG:teuthology.orchestra.run.smithi046:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-31T23:09:21.534 DEBUG:teuthology.orchestra.run.smithi046:> set -e 2024-03-31T23:09:21.534 DEBUG:teuthology.orchestra.run.smithi046:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-03-31T23:09:21.534 DEBUG:teuthology.orchestra.run.smithi046:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-03-31T23:09:21.535 DEBUG:teuthology.orchestra.run.smithi046:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-03-31T23:09:21.535 DEBUG:teuthology.orchestra.run.smithi046:> ') 2024-03-31T23:09:21.583 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:21 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-31T23:09:21.690 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:09:21 smithi046 ceph-mon[32498]: from='client.? 172.21.15.46:0/2690683906' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-31T23:09:21.693 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:09:21 smithi142 ceph-mon[35271]: from='client.? 172.21.15.46:0/2690683906' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-31T23:09:21.772 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:21 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-31T23:09:21.778 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T23:09:21.778 DEBUG:teuthology.orchestra.run.smithi046:> ip netns list 2024-03-31T23:09:21.793 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T23:09:21.793 DEBUG:teuthology.orchestra.run.smithi046:> ip netns list-id 2024-03-31T23:09:21.795 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:09:21 smithi162 ceph-mon[36237]: from='client.? 172.21.15.46:0/2690683906' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-31T23:09:21.847 DEBUG:teuthology.orchestra.run.smithi046:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-31T23:09:21.848 DEBUG:teuthology.orchestra.run.smithi046:> set -e 2024-03-31T23:09:21.848 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-31T23:09:21.848 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.admin 0 2024-03-31T23:09:21.848 DEBUG:teuthology.orchestra.run.smithi046:> ') 2024-03-31T23:09:21.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:21 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-31T23:09:21.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:21 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-31T23:09:21.971 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.admin' with 192.168.0.1/16 2024-03-31T23:09:21.971 DEBUG:teuthology.orchestra.run.smithi046:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-31T23:09:21.971 DEBUG:teuthology.orchestra.run.smithi046:> set -e 2024-03-31T23:09:21.971 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.admin type veth peer name brx.0 2024-03-31T23:09:21.971 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-03-31T23:09:21.971 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set veth0 up 2024-03-31T23:09:21.971 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set lo up 2024-03-31T23:09:21.972 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip route add default via 192.168.255.254 2024-03-31T23:09:21.972 DEBUG:teuthology.orchestra.run.smithi046:> ') 2024-03-31T23:09:22.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:22 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-31T23:09:22.100 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:22 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-31T23:09:22.106 DEBUG:teuthology.orchestra.run.smithi046:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-31T23:09:22.106 DEBUG:teuthology.orchestra.run.smithi046:> set -e 2024-03-31T23:09:22.106 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip link set brx.0 up 2024-03-31T23:09:22.106 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip link set dev brx.0 master ceph-brx 2024-03-31T23:09:22.106 DEBUG:teuthology.orchestra.run.smithi046:> ') 2024-03-31T23:09:22.154 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:22 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-31T23:09:22.182 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:22 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-31T23:09:22.188 INFO:tasks.cephfs.fuse_mount:Client client.admin config is {} 2024-03-31T23:09:22.188 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-31T23:09:22.188 DEBUG:teuthology.orchestra.run.smithi046:> mkdir -p -v /home/ubuntu/cephtest/mnt.admin 2024-03-31T23:09:22.203 INFO:teuthology.orchestra.run.smithi046.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.admin' 2024-03-31T23:09:22.203 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-31T23:09:22.203 DEBUG:teuthology.orchestra.run.smithi046:> chmod 0000 /home/ubuntu/cephtest/mnt.admin 2024-03-31T23:09:22.257 DEBUG:teuthology.orchestra.run.smithi046:> sudo modprobe fuse 2024-03-31T23:09:22.324 DEBUG:teuthology.orchestra.run.smithi046:> cat /proc/self/mounts | awk '{print $2}' 2024-03-31T23:09:22.380 INFO:teuthology.orchestra.run.smithi046.stdout:/proc 2024-03-31T23:09:22.380 INFO:teuthology.orchestra.run.smithi046.stdout:/sys 2024-03-31T23:09:22.380 INFO:teuthology.orchestra.run.smithi046.stdout:/dev 2024-03-31T23:09:22.380 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/security 2024-03-31T23:09:22.380 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/shm 2024-03-31T23:09:22.380 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/pts 2024-03-31T23:09:22.380 INFO:teuthology.orchestra.run.smithi046.stdout:/run 2024-03-31T23:09:22.380 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/cgroup 2024-03-31T23:09:22.380 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/pstore 2024-03-31T23:09:22.381 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/bpf 2024-03-31T23:09:22.381 INFO:teuthology.orchestra.run.smithi046.stdout:/ 2024-03-31T23:09:22.381 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/selinux 2024-03-31T23:09:22.381 INFO:teuthology.orchestra.run.smithi046.stdout:/proc/sys/fs/binfmt_misc 2024-03-31T23:09:22.381 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/mqueue 2024-03-31T23:09:22.381 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/hugepages 2024-03-31T23:09:22.381 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/debug 2024-03-31T23:09:22.381 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/tracing 2024-03-31T23:09:22.381 INFO:teuthology.orchestra.run.smithi046.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-31T23:09:22.381 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/fuse/connections 2024-03-31T23:09:22.381 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/config 2024-03-31T23:09:22.381 INFO:teuthology.orchestra.run.smithi046.stdout:/run/credentials/systemd-sysctl.service 2024-03-31T23:09:22.381 INFO:teuthology.orchestra.run.smithi046.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-31T23:09:22.381 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/nfs/rpc_pipefs 2024-03-31T23:09:22.382 INFO:teuthology.orchestra.run.smithi046.stdout:/run/user/1000 2024-03-31T23:09:22.382 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph 2024-03-31T23:09:22.382 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/containers/storage/overlay 2024-03-31T23:09:22.382 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/containers/storage/overlay/af6e3dee319c5cd951091dcc321002d5c94992669c333da4f71a8811f5553b4d/merged 2024-03-31T23:09:22.382 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/containers/storage/overlay/6aceac46eafe077575119569412871a784cac219bbecd4a816eb0dd6e31cb766/merged 2024-03-31T23:09:22.382 INFO:teuthology.orchestra.run.smithi046.stdout:/run/user/0 2024-03-31T23:09:22.382 INFO:teuthology.orchestra.run.smithi046.stdout:/proc/sys/fs/binfmt_misc 2024-03-31T23:09:22.382 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/containers/storage/overlay/9732c8b2bef193a8ef899edc437102cc83b87c3c2b9aeede866239f014160834/merged 2024-03-31T23:09:22.382 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/containers/storage/overlay/164dbd5bc6ff0e68c1ce7493f0d76208f294fcb7e4baf61acb26eb069c007959/merged 2024-03-31T23:09:22.382 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/containers/storage/overlay/3ac5e44512b35cb8a3802d7963e396a049e4c945b1c971c39f38030cdbde1024/merged 2024-03-31T23:09:22.382 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/containers/storage/overlay/d90685ef54c2be4027d89ad1c931156b41c2ccfa453582969c652fb1f066e1cd/merged 2024-03-31T23:09:22.382 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/containers/storage/overlay/65432fa8477e0ccf6f4c64e0305f8fa786782d7091f0c38d3a93b23978a7be8f/merged 2024-03-31T23:09:22.383 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/containers/storage/overlay/d41b0d141c8e4fb859226d0b5cb5ff4152c520b9aa66eb4b49dac96f28e032d0/merged 2024-03-31T23:09:22.383 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/containers/storage/overlay/6691bede7b8bc82e883758a6ef749bc93e4df469e71f598744e9970488ce50e9/merged 2024-03-31T23:09:22.383 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/containers/storage/overlay/8e0bc49805369f19f18f5f8d0ac9abfc75e13ffb949d5193d70953f236d7a403/merged 2024-03-31T23:09:22.383 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns 2024-03-31T23:09:22.383 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-31T23:09:22.383 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-31T23:09:22.383 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T23:09:22.383 DEBUG:teuthology.orchestra.run.smithi046:> ls /sys/fs/fuse/connections 2024-03-31T23:09:22.435 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-03-31T23:09:22.436 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.admin --id admin --client_fs=cephfs) 2024-03-31T23:09:22.478 DEBUG:teuthology.orchestra.run.smithi046:> sudo modprobe fuse 2024-03-31T23:09:22.504 DEBUG:teuthology.orchestra.run.smithi046:> cat /proc/self/mounts | awk '{print $2}' 2024-03-31T23:09:22.560 INFO:teuthology.orchestra.run.smithi046.stdout:/proc 2024-03-31T23:09:22.560 INFO:teuthology.orchestra.run.smithi046.stdout:/sys 2024-03-31T23:09:22.560 INFO:teuthology.orchestra.run.smithi046.stdout:/dev 2024-03-31T23:09:22.561 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/security 2024-03-31T23:09:22.561 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/shm 2024-03-31T23:09:22.561 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/pts 2024-03-31T23:09:22.561 INFO:teuthology.orchestra.run.smithi046.stdout:/run 2024-03-31T23:09:22.561 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/cgroup 2024-03-31T23:09:22.561 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/pstore 2024-03-31T23:09:22.561 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/bpf 2024-03-31T23:09:22.561 INFO:teuthology.orchestra.run.smithi046.stdout:/ 2024-03-31T23:09:22.561 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/selinux 2024-03-31T23:09:22.561 INFO:teuthology.orchestra.run.smithi046.stdout:/proc/sys/fs/binfmt_misc 2024-03-31T23:09:22.561 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/mqueue 2024-03-31T23:09:22.561 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/hugepages 2024-03-31T23:09:22.561 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/debug 2024-03-31T23:09:22.562 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/tracing 2024-03-31T23:09:22.562 INFO:teuthology.orchestra.run.smithi046.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-31T23:09:22.562 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/fuse/connections 2024-03-31T23:09:22.562 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/config 2024-03-31T23:09:22.562 INFO:teuthology.orchestra.run.smithi046.stdout:/run/credentials/systemd-sysctl.service 2024-03-31T23:09:22.562 INFO:teuthology.orchestra.run.smithi046.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-31T23:09:22.562 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/nfs/rpc_pipefs 2024-03-31T23:09:22.562 INFO:teuthology.orchestra.run.smithi046.stdout:/run/user/1000 2024-03-31T23:09:22.562 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph 2024-03-31T23:09:22.562 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/containers/storage/overlay 2024-03-31T23:09:22.562 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/containers/storage/overlay/af6e3dee319c5cd951091dcc321002d5c94992669c333da4f71a8811f5553b4d/merged 2024-03-31T23:09:22.562 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/containers/storage/overlay/6aceac46eafe077575119569412871a784cac219bbecd4a816eb0dd6e31cb766/merged 2024-03-31T23:09:22.562 INFO:teuthology.orchestra.run.smithi046.stdout:/run/user/0 2024-03-31T23:09:22.563 INFO:teuthology.orchestra.run.smithi046.stdout:/proc/sys/fs/binfmt_misc 2024-03-31T23:09:22.563 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/containers/storage/overlay/9732c8b2bef193a8ef899edc437102cc83b87c3c2b9aeede866239f014160834/merged 2024-03-31T23:09:22.563 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/containers/storage/overlay/164dbd5bc6ff0e68c1ce7493f0d76208f294fcb7e4baf61acb26eb069c007959/merged 2024-03-31T23:09:22.564 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/containers/storage/overlay/3ac5e44512b35cb8a3802d7963e396a049e4c945b1c971c39f38030cdbde1024/merged 2024-03-31T23:09:22.564 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/containers/storage/overlay/d90685ef54c2be4027d89ad1c931156b41c2ccfa453582969c652fb1f066e1cd/merged 2024-03-31T23:09:22.564 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/containers/storage/overlay/65432fa8477e0ccf6f4c64e0305f8fa786782d7091f0c38d3a93b23978a7be8f/merged 2024-03-31T23:09:22.564 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/containers/storage/overlay/d41b0d141c8e4fb859226d0b5cb5ff4152c520b9aa66eb4b49dac96f28e032d0/merged 2024-03-31T23:09:22.564 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/containers/storage/overlay/6691bede7b8bc82e883758a6ef749bc93e4df469e71f598744e9970488ce50e9/merged 2024-03-31T23:09:22.564 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/containers/storage/overlay/8e0bc49805369f19f18f5f8d0ac9abfc75e13ffb949d5193d70953f236d7a403/merged 2024-03-31T23:09:22.564 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns 2024-03-31T23:09:22.564 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-31T23:09:22.564 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-31T23:09:22.565 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T23:09:22.565 DEBUG:teuthology.orchestra.run.smithi046:> ls /sys/fs/fuse/connections 2024-03-31T23:09:22.573 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi046.stderr:2024-03-31T23:09:22.572+0000 7f31c23f9480 -1 init, newargv = 0x55a3a5447d20 newargc=15 2024-03-31T23:09:22.573 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi046.stderr:ceph-fuse[76163]: starting ceph client 2024-03-31T23:09:22.579 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi046.stderr:ceph-fuse[76163]: starting fuse 2024-03-31T23:09:22.588 INFO:teuthology.orchestra.run.smithi046.stdout:119 2024-03-31T23:09:22.588 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [119] 2024-03-31T23:09:22.588 DEBUG:teuthology.orchestra.run.smithi046:> sudo stdin-killer -- python3 -c ' 2024-03-31T23:09:22.588 DEBUG:teuthology.orchestra.run.smithi046:> import glob 2024-03-31T23:09:22.588 DEBUG:teuthology.orchestra.run.smithi046:> import re 2024-03-31T23:09:22.589 DEBUG:teuthology.orchestra.run.smithi046:> import os 2024-03-31T23:09:22.589 DEBUG:teuthology.orchestra.run.smithi046:> import subprocess 2024-03-31T23:09:22.589 DEBUG:teuthology.orchestra.run.smithi046:> 2024-03-31T23:09:22.589 DEBUG:teuthology.orchestra.run.smithi046:> def _find_admin_socket(client_name): 2024-03-31T23:09:22.589 DEBUG:teuthology.orchestra.run.smithi046:> asok_path = "/var/run/ceph/ceph-client.admin.*.asok" 2024-03-31T23:09:22.589 DEBUG:teuthology.orchestra.run.smithi046:> files = glob.glob(asok_path) 2024-03-31T23:09:22.589 DEBUG:teuthology.orchestra.run.smithi046:> mountpoint = "/home/ubuntu/cephtest/mnt.admin" 2024-03-31T23:09:22.589 DEBUG:teuthology.orchestra.run.smithi046:> 2024-03-31T23:09:22.589 DEBUG:teuthology.orchestra.run.smithi046:> # Given a non-glob path, it better be there 2024-03-31T23:09:22.589 DEBUG:teuthology.orchestra.run.smithi046:> if "*" not in asok_path: 2024-03-31T23:09:22.589 DEBUG:teuthology.orchestra.run.smithi046:> assert(len(files) == 1) 2024-03-31T23:09:22.589 DEBUG:teuthology.orchestra.run.smithi046:> return files[0] 2024-03-31T23:09:22.589 DEBUG:teuthology.orchestra.run.smithi046:> 2024-03-31T23:09:22.589 DEBUG:teuthology.orchestra.run.smithi046:> for f in files: 2024-03-31T23:09:22.590 DEBUG:teuthology.orchestra.run.smithi046:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-31T23:09:22.590 DEBUG:teuthology.orchestra.run.smithi046:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-31T23:09:22.590 DEBUG:teuthology.orchestra.run.smithi046:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-31T23:09:22.590 DEBUG:teuthology.orchestra.run.smithi046:> contents = proc_f.read() 2024-03-31T23:09:22.590 DEBUG:teuthology.orchestra.run.smithi046:> if mountpoint in contents: 2024-03-31T23:09:22.590 DEBUG:teuthology.orchestra.run.smithi046:> return f 2024-03-31T23:09:22.590 DEBUG:teuthology.orchestra.run.smithi046:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-31T23:09:22.590 DEBUG:teuthology.orchestra.run.smithi046:> 2024-03-31T23:09:22.590 DEBUG:teuthology.orchestra.run.smithi046:> print(_find_admin_socket("client.admin")) 2024-03-31T23:09:22.590 DEBUG:teuthology.orchestra.run.smithi046:> ' 2024-03-31T23:09:22.611 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:09:22 smithi046 ceph-mon[32498]: pgmap v297: 129 pgs: 129 active+clean; 579 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 511 B/s wr, 1 op/s 2024-03-31T23:09:22.691 INFO:teuthology.orchestra.run.smithi046.stdout:/var/run/ceph/ceph-client.admin.76163.asok 2024-03-31T23:09:22.693 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:09:22 smithi142 ceph-mon[35271]: pgmap v297: 129 pgs: 129 active+clean; 579 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 511 B/s wr, 1 op/s 2024-03-31T23:09:22.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:22 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-31T23:09:22.702 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.admin.76163.asok 2024-03-31T23:09:22.702 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T23:09:22.702 DEBUG:teuthology.orchestra.run.smithi046:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.admin.76163.asok status 2024-03-31T23:09:22.795 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:09:22 smithi162 ceph-mon[36237]: pgmap v297: 129 pgs: 129 active+clean; 579 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 511 B/s wr, 1 op/s 2024-03-31T23:09:22.839 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-03-31T23:09:22.840 INFO:teuthology.orchestra.run.smithi046.stdout: "metadata": { 2024-03-31T23:09:22.840 INFO:teuthology.orchestra.run.smithi046.stdout: "ceph_sha1": "384f31f3d3aee7f8ba3f60846b8844c049165583", 2024-03-31T23:09:22.840 INFO:teuthology.orchestra.run.smithi046.stdout: "ceph_version": "ceph version 19.0.0-1977-g384f31f3 (384f31f3d3aee7f8ba3f60846b8844c049165583) squid (dev)", 2024-03-31T23:09:22.840 INFO:teuthology.orchestra.run.smithi046.stdout: "entity_id": "admin", 2024-03-31T23:09:22.840 INFO:teuthology.orchestra.run.smithi046.stdout: "hostname": "smithi046", 2024-03-31T23:09:22.840 INFO:teuthology.orchestra.run.smithi046.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.admin", 2024-03-31T23:09:22.840 INFO:teuthology.orchestra.run.smithi046.stdout: "pid": "76163", 2024-03-31T23:09:22.840 INFO:teuthology.orchestra.run.smithi046.stdout: "root": "/" 2024-03-31T23:09:22.840 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:09:22.840 INFO:teuthology.orchestra.run.smithi046.stdout: "dentry_count": 0, 2024-03-31T23:09:22.840 INFO:teuthology.orchestra.run.smithi046.stdout: "dentry_pinned_count": 0, 2024-03-31T23:09:22.840 INFO:teuthology.orchestra.run.smithi046.stdout: "id": 14784, 2024-03-31T23:09:22.840 INFO:teuthology.orchestra.run.smithi046.stdout: "inst": { 2024-03-31T23:09:22.840 INFO:teuthology.orchestra.run.smithi046.stdout: "name": { 2024-03-31T23:09:22.841 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "client", 2024-03-31T23:09:22.841 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 14784 2024-03-31T23:09:22.841 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:09:22.841 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": { 2024-03-31T23:09:22.841 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-03-31T23:09:22.841 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "192.168.0.1:0", 2024-03-31T23:09:22.841 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 2527094143 2024-03-31T23:09:22.841 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-31T23:09:22.841 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:09:22.841 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": { 2024-03-31T23:09:22.841 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-03-31T23:09:22.841 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "192.168.0.1:0", 2024-03-31T23:09:22.841 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 2527094143 2024-03-31T23:09:22.841 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:09:22.842 INFO:teuthology.orchestra.run.smithi046.stdout: "inst_str": "client.14784 192.168.0.1:0/2527094143", 2024-03-31T23:09:22.842 INFO:teuthology.orchestra.run.smithi046.stdout: "addr_str": "192.168.0.1:0/2527094143", 2024-03-31T23:09:22.842 INFO:teuthology.orchestra.run.smithi046.stdout: "inode_count": 1, 2024-03-31T23:09:22.842 INFO:teuthology.orchestra.run.smithi046.stdout: "mds_epoch": 17, 2024-03-31T23:09:22.842 INFO:teuthology.orchestra.run.smithi046.stdout: "osd_epoch": 91, 2024-03-31T23:09:22.842 INFO:teuthology.orchestra.run.smithi046.stdout: "osd_epoch_barrier": 0, 2024-03-31T23:09:22.842 INFO:teuthology.orchestra.run.smithi046.stdout: "blocklisted": false, 2024-03-31T23:09:22.842 INFO:teuthology.orchestra.run.smithi046.stdout: "fs_name": "cephfs" 2024-03-31T23:09:22.842 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-03-31T23:09:22.848 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T23:09:22.848 DEBUG:teuthology.orchestra.run.smithi046:> stat --file-system '--printf=%T 2024-03-31T23:09:22.848 DEBUG:teuthology.orchestra.run.smithi046:> ' -- /home/ubuntu/cephtest/mnt.admin 2024-03-31T23:09:22.863 INFO:teuthology.orchestra.run.smithi046.stdout:fuseblk 2024-03-31T23:09:22.863 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.admin 2024-03-31T23:09:22.863 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T23:09:22.863 DEBUG:teuthology.orchestra.run.smithi046:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.admin 2024-03-31T23:09:22.932 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.admin && exec stdin-killer --timeout=900 -- bash -c 'setfattr -n ceph.dir.layout.pool -v cephfs_data_ec . && getfattr -n ceph.dir.layout .') 2024-03-31T23:09:23.023 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:23 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-03-31T23:09:23.035 INFO:teuthology.orchestra.run.smithi046.stdout:# file: . 2024-03-31T23:09:23.035 INFO:teuthology.orchestra.run.smithi046.stdout:ceph.dir.layout="stripe_unit=4194304 stripe_count=1 object_size=4194304 pool=cephfs_data_ec" 2024-03-31T23:09:23.035 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-31T23:09:23.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:23 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-31T23:09:23.041 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-03-31T23:09:23.041 DEBUG:teuthology.orchestra.run.smithi046:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-31T23:09:23.057 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-31T23:09:23.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:23.199+0000 7f59c7d02640 1 -- 172.21.15.46:0/3036965479 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f59c0102970 msgr2=0x7f59c0102df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:09:23.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:23.199+0000 7f59c7d02640 1 --2- 172.21.15.46:0/3036965479 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f59c0102970 0x7f59c0102df0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f59b0009900 tx=0x7f59b002bc30 comp rx=0 tx=0).stop 2024-03-31T23:09:23.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:23.199+0000 7f59c7d02640 1 -- 172.21.15.46:0/3036965479 shutdown_connections 2024-03-31T23:09:23.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:23.199+0000 7f59c7d02640 1 --2- 172.21.15.46:0/3036965479 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f59c0103330 0x7f59c0109bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:23.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:23.199+0000 7f59c7d02640 1 --2- 172.21.15.46:0/3036965479 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f59c0102970 0x7f59c0102df0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:23.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:23.199+0000 7f59c7d02640 1 --2- 172.21.15.46:0/3036965479 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f59c0101780 0x7f59c0101ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:23.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:23.199+0000 7f59c7d02640 1 -- 172.21.15.46:0/3036965479 >> 172.21.15.46:0/3036965479 conn(0x7f59c00fccf0 msgr2=0x7f59c00ff130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:09:23.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:23.200+0000 7f59c7d02640 1 -- 172.21.15.46:0/3036965479 shutdown_connections 2024-03-31T23:09:23.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:23.200+0000 7f59c7d02640 1 -- 172.21.15.46:0/3036965479 wait complete. 2024-03-31T23:09:23.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:23.200+0000 7f59c7d02640 1 Processor -- start 2024-03-31T23:09:23.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:23.200+0000 7f59c7d02640 1 -- start start 2024-03-31T23:09:23.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:23.200+0000 7f59c7d02640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f59c0101780 0x7f59c0197990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:09:23.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:23.200+0000 7f59c7d02640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f59c0102970 0x7f59c0197ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:09:23.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:23.200+0000 7f59c7d02640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f59c0103330 0x7f59c019ef40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:09:23.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:23.200+0000 7f59c7d02640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f59c010d290 con 0x7f59c0101780 2024-03-31T23:09:23.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:23.200+0000 7f59c7d02640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f59c010d110 con 0x7f59c0102970 2024-03-31T23:09:23.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:23.200+0000 7f59c7d02640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f59c010d410 con 0x7f59c0103330 2024-03-31T23:09:23.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:23.201+0000 7f59c5a9e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f59c0101780 0x7f59c0197990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:09:23.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:23.201+0000 7f59c529d640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f59c0102970 0x7f59c0197ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:09:23.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:23.201+0000 7f59c5a9e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f59c0101780 0x7f59c0197990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44734/0 (socket says 172.21.15.46:44734) 2024-03-31T23:09:23.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:23.201+0000 7f59c5a9e640 1 -- 172.21.15.46:0/3744734253 learned_addr learned my addr 172.21.15.46:0/3744734253 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:09:23.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:23.201+0000 7f59c629f640 1 --2- 172.21.15.46:0/3744734253 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f59c0103330 0x7f59c019ef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:09:23.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:23.201+0000 7f59c5a9e640 1 -- 172.21.15.46:0/3744734253 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f59c0103330 msgr2=0x7f59c019ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:09:23.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:23.201+0000 7f59c5a9e640 1 --2- 172.21.15.46:0/3744734253 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f59c0103330 0x7f59c019ef40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:23.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:23.201+0000 7f59c5a9e640 1 -- 172.21.15.46:0/3744734253 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f59c0102970 msgr2=0x7f59c0197ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:09:23.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:23.201+0000 7f59c5a9e640 1 --2- 172.21.15.46:0/3744734253 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f59c0102970 0x7f59c0197ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:23.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:23.201+0000 7f59c5a9e640 1 -- 172.21.15.46:0/3744734253 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59c019f5b0 con 0x7f59c0101780 2024-03-31T23:09:23.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:23.201+0000 7f59c529d640 1 --2- 172.21.15.46:0/3744734253 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f59c0102970 0x7f59c0197ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:09:23.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:23.201+0000 7f59c5a9e640 1 --2- 172.21.15.46:0/3744734253 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f59c0101780 0x7f59c0197990 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f59bc00e9c0 tx=0x7f59bc00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:09:23.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:23.201+0000 7f59b6ffd640 1 -- 172.21.15.46:0/3744734253 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59bc00cde0 con 0x7f59c0101780 2024-03-31T23:09:23.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:23.201+0000 7f59c7d02640 1 -- 172.21.15.46:0/3744734253 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f59c019f8a0 con 0x7f59c0101780 2024-03-31T23:09:23.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:23.201+0000 7f59b6ffd640 1 -- 172.21.15.46:0/3744734253 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f59bc0108e0 con 0x7f59c0101780 2024-03-31T23:09:23.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:23.201+0000 7f59b6ffd640 1 -- 172.21.15.46:0/3744734253 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59bc0188e0 con 0x7f59c0101780 2024-03-31T23:09:23.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:23.201+0000 7f59c7d02640 1 -- 172.21.15.46:0/3744734253 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f59c019fdc0 con 0x7f59c0101780 2024-03-31T23:09:23.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:23.202+0000 7f59c7d02640 1 -- 172.21.15.46:0/3744734253 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f59c0108b70 con 0x7f59c0101780 2024-03-31T23:09:23.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:23.204+0000 7f59b6ffd640 1 -- 172.21.15.46:0/3744734253 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 16) ==== 137320+0+0 (secure 0 0 0) 0x7f59bc01f020 con 0x7f59c0101780 2024-03-31T23:09:23.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:23.205+0000 7f59b6ffd640 1 --2- 172.21.15.46:0/3744734253 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f59ac0a24a0 0x7f59ac0a4960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:09:23.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:23.205+0000 7f59b6ffd640 1 -- 172.21.15.46:0/3744734253 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8802+0+0 (secure 0 0 0) 0x7f59bc0cf140 con 0x7f59c0101780 2024-03-31T23:09:23.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:23.205+0000 7f59c529d640 1 --2- 172.21.15.46:0/3744734253 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f59ac0a24a0 0x7f59ac0a4960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:09:23.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:23.205+0000 7f59c529d640 1 --2- 172.21.15.46:0/3744734253 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f59ac0a24a0 0x7f59ac0a4960 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f59b0009230 tx=0x7f59b0005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:09:23.209 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:23.209+0000 7f59b6ffd640 1 -- 172.21.15.46:0/3744734253 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f59bc09c0c0 con 0x7f59c0101780 2024-03-31T23:09:23.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:23.394+0000 7f59c7d02640 1 -- 172.21.15.46:0/3744734253 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f59c002d170 con 0x7f59c0101780 2024-03-31T23:09:23.396 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:23.395+0000 7f59b6ffd640 1 -- 172.21.15.46:0/3744734253 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) ==== 74+0+24105 (secure 0 0 0) 0x7f59bc09b9c0 con 0x7f59c0101780 2024-03-31T23:09:23.396 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-31T23:09:23.396 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":91,"fsid":"6a9d8f08-efb2-11ee-b647-cb9ed24678a4","created":"2024-03-31T23:01:15.463711+0000","modified":"2024-03-31T23:09:17.181787+0000","last_up_change":"2024-03-31T23:08:20.199888+0000","last_in_change":"2024-03-31T23:07:59.438035+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-31T23:04:32.823751+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-31T23:09:09.145711+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-31T23:09:10.148579+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-31T23:09:13.419519+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"91","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"47429095-3f46-4ed7-b7c4-504464719815","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":1824772695},{"type":"v1","addr":"172.21.15.46:6803","nonce":1824772695}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":1824772695},{"type":"v1","addr":"172.21.15.46:6805","nonce":1824772695}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":1824772695},{"type":"v1","addr":"172.21.15.46:6809","nonce":1824772695}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":1824772695},{"type":"v1","addr":"172.21.15.46:6807","nonce":1824772695}]},"public_addr":"172.21.15.46:6803/1824772695","cluster_addr":"172.21.15.46:6805/1824772695","heartbeat_back_addr":"172.21.15.46:6809/1824772695","heartbeat_front_addr":"172.21.15.46:6807/1824772695","state":["exists","up"]},{"osd":1,"uuid":"be56f883-411e-4dbe-b50e-a158134df4a6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6800","nonce":973693922},{"type":"v1","addr":"172.21.15.142:6801","nonce":973693922}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6802","nonce":973693922},{"type":"v1","addr":"172.21.15.142:6803","nonce":973693922}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6806","nonce":973693922},{"type":"v1","addr":"172.21.15.142:6807","nonce":973693922}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6804","nonce":973693922},{"type":"v1","addr":"172.21.15.142:6805","nonce":973693922}]},"public_addr":"172.21.15.142:6801/973693922","cluster_addr":"172.21.15.142:6803/973693922","heartbeat_back_addr":"172.21.15.142:6807/973693922","heartbeat_front_addr":"172.21.15.142:6805/973693922","state":["exists","up"]},{"osd":2,"uuid":"c99a8794-ab19-4260-9a8d-4bc0bd2c8175","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6800","nonce":4084261813},{"type":"v1","addr":"172.21.15.162:6801","nonce":4084261813}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6802","nonce":4084261813},{"type":"v1","addr":"172.21.15.162:6803","nonce":4084261813}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6806","nonce":4084261813},{"type":"v1","addr":"172.21.15.162:6807","nonce":4084261813}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6804","nonce":4084261813},{"type":"v1","addr":"172.21.15.162:6805","nonce":4084261813}]},"public_addr":"172.21.15.162:6801/4084261813","cluster_addr":"172.21.15.162:6803/4084261813","heartbeat_back_addr":"172.21.15.162:6807/4084261813","heartbeat_front_addr":"172.21.15.162:6805/4084261813","state":["exists","up"]},{"osd":3,"uuid":"f877814e-9ef6-4872-a081-5609bb9ebc06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":3496317970},{"type":"v1","addr":"172.21.15.46:6811","nonce":3496317970}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":3496317970},{"type":"v1","addr":"172.21.15.46:6813","nonce":3496317970}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":3496317970},{"type":"v1","addr":"172.21.15.46:6817","nonce":3496317970}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":3496317970},{"type":"v1","addr":"172.21.15.46:6815","nonce":3496317970}]},"public_addr":"172.21.15.46:6811/3496317970","cluster_addr":"172.21.15.46:6813/3496317970","heartbeat_back_addr":"172.21.15.46:6817/3496317970","heartbeat_front_addr":"172.21.15.46:6815/3496317970","state":["exists","up"]},{"osd":4,"uuid":"beecec0a-8280-44fe-8f5f-8498a35eca8f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6808","nonce":2996266929},{"type":"v1","addr":"172.21.15.142:6809","nonce":2996266929}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6810","nonce":2996266929},{"type":"v1","addr":"172.21.15.142:6811","nonce":2996266929}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6814","nonce":2996266929},{"type":"v1","addr":"172.21.15.142:6815","nonce":2996266929}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6812","nonce":2996266929},{"type":"v1","addr":"172.21.15.142:6813","nonce":2996266929}]},"public_addr":"172.21.15.142:6809/2996266929","cluster_addr":"172.21.15.142:6811/2996266929","heartbeat_back_addr":"172.21.15.142:6815/2996266929","heartbeat_front_addr":"172.21.15.142:6813/2996266929","state":["exists","up"]},{"osd":5,"uuid":"9da7ff75-f658-45ea-a02f-1943fdda76b9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6808","nonce":914208293},{"type":"v1","addr":"172.21.15.162:6809","nonce":914208293}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6810","nonce":914208293},{"type":"v1","addr":"172.21.15.162:6811","nonce":914208293}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6814","nonce":914208293},{"type":"v1","addr":"172.21.15.162:6815","nonce":914208293}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6812","nonce":914208293},{"type":"v1","addr":"172.21.15.162:6813","nonce":914208293}]},"public_addr":"172.21.15.162:6809/914208293","cluster_addr":"172.21.15.162:6811/914208293","heartbeat_back_addr":"172.21.15.162:6815/914208293","heartbeat_front_addr":"172.21.15.162:6813/914208293","state":["exists","up"]},{"osd":6,"uuid":"8fa11224-cb26-4d3f-87d1-6a8f31de2a19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":56291966},{"type":"v1","addr":"172.21.15.46:6819","nonce":56291966}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":56291966},{"type":"v1","addr":"172.21.15.46:6821","nonce":56291966}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6824","nonce":56291966},{"type":"v1","addr":"172.21.15.46:6825","nonce":56291966}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":56291966},{"type":"v1","addr":"172.21.15.46:6823","nonce":56291966}]},"public_addr":"172.21.15.46:6819/56291966","cluster_addr":"172.21.15.46:6821/56291966","heartbeat_back_addr":"172.21.15.46:6825/56291966","heartbeat_front_addr":"172.21.15.46:6823/56291966","state":["exists","up"]},{"osd":7,"uuid":"0d0338b7-4d0a-4bb0-b152-8f5224871d9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6816","nonce":2146598446},{"type":"v1","addr":"172.21.15.142:6817","nonce":2146598446}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6818","nonce":2146598446},{"type":"v1","addr":"172.21.15.142:6819","nonce":2146598446}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6822","nonce":2146598446},{"type":"v1","addr":"172.21.15.142:6823","nonce":2146598446}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6820","nonce":2146598446},{"type":"v1","addr":"172.21.15.142:6821","nonce":2146598446}]},"public_addr":"172.21.15.142:6817/2146598446","cluster_addr":"172.21.15.142:6819/2146598446","heartbeat_back_addr":"172.21.15.142:6823/2146598446","heartbeat_front_addr":"172.21.15.142:6821/2146598446","state":["exists","up"]},{"osd":8,"uuid":"8b6f6037-892a-416c-babb-7552648d0b3a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6816","nonce":1173078318},{"type":"v1","addr":"172.21.15.162:6817","nonce":1173078318}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6818","nonce":1173078318},{"type":"v1","addr":"172.21.15.162:6819","nonce":1173078318}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6822","nonce":1173078318},{"type":"v1","addr":"172.21.15.162:6823","nonce":1173078318}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6820","nonce":1173078318},{"type":"v1","addr":"172.21.15.162:6821","nonce":1173078318}]},"public_addr":"172.21.15.162:6817/1173078318","cluster_addr":"172.21.15.162:6819/1173078318","heartbeat_back_addr":"172.21.15.162:6823/1173078318","heartbeat_front_addr":"172.21.15.162:6821/1173078318","state":["exists","up"]},{"osd":9,"uuid":"faccce25-6bf2-4220-8858-cda82b72be68","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":532708881},{"type":"v1","addr":"172.21.15.46:6827","nonce":532708881}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6828","nonce":532708881},{"type":"v1","addr":"172.21.15.46:6829","nonce":532708881}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":532708881},{"type":"v1","addr":"172.21.15.46:6833","nonce":532708881}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":532708881},{"type":"v1","addr":"172.21.15.46:6831","nonce":532708881}]},"public_addr":"172.21.15.46:6827/532708881","cluster_addr":"172.21.15.46:6829/532708881","heartbeat_back_addr":"172.21.15.46:6833/532708881","heartbeat_front_addr":"172.21.15.46:6831/532708881","state":["exists","up"]},{"osd":10,"uuid":"cdd356db-f9eb-4080-95ab-db5d8223e572","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6824","nonce":2845222366},{"type":"v1","addr":"172.21.15.142:6825","nonce":2845222366}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6826","nonce":2845222366},{"type":"v1","addr":"172.21.15.142:6827","nonce":2845222366}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6830","nonce":2845222366},{"type":"v1","addr":"172.21.15.142:6831","nonce":2845222366}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6828","nonce":2845222366},{"type":"v1","addr":"172.21.15.142:6829","nonce":2845222366}]},"public_addr":"172.21.15.142:6825/2845222366","cluster_addr":"172.21.15.142:6827/2845222366","heartbeat_back_addr":"172.21.15.142:6831/2845222366","heartbeat_front_addr":"172.21.15.142:6829/2845222366","state":["exists","up"]},{"osd":11,"uuid":"72104d1c-3ea2-45ad-bc0b-0500f146c06b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6824","nonce":1986323778},{"type":"v1","addr":"172.21.15.162:6825","nonce":1986323778}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6826","nonce":1986323778},{"type":"v1","addr":"172.21.15.162:6827","nonce":1986323778}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6830","nonce":1986323778},{"type":"v1","addr":"172.21.15.162:6831","nonce":1986323778}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6828","nonce":1986323778},{"type":"v1","addr":"172.21.15.162:6829","nonce":1986323778}]},"public_addr":"172.21.15.162:6825/1986323778","cluster_addr":"172.21.15.162:6827/1986323778","heartbeat_back_addr":"172.21.15.162:6831/1986323778","heartbeat_front_addr":"172.21.15.162:6829/1986323778","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:03:44.996331+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:04:07.994738+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:04:30.942255+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:04:55.588884+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:05:19.900600+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:05:43.807442+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:06:09.026976+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:06:34.586784+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:06:59.751605+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:07:26.144495+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:07:52.326909+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:08:17.423042+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.46:0/4021039857":"2024-04-01T23:02:08.493619+0000","172.21.15.46:6801/3681182647":"2024-04-01T23:02:08.493619+0000","172.21.15.46:0/4022214806":"2024-04-01T23:02:08.493619+0000","172.21.15.46:6800/1515528215":"2024-04-01T23:01:41.910162+0000","172.21.15.46:6800/3681182647":"2024-04-01T23:02:08.493619+0000","172.21.15.46:0/143885707":"2024-04-01T23:01:41.910162+0000","172.21.15.46:6801/1515528215":"2024-04-01T23:01:41.910162+0000","172.21.15.46:0/1680389778":"2024-04-01T23:01:41.910162+0000","172.21.15.46:0/1782133966":"2024-04-01T23:01:41.910162+0000","172.21.15.46:0/3925209376":"2024-04-01T23:02:08.493619+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-31T23:09:23.398 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:23.397+0000 7f59c7d02640 1 -- 172.21.15.46:0/3744734253 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f59ac0a24a0 msgr2=0x7f59ac0a4960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:09:23.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:23.397+0000 7f59c7d02640 1 --2- 172.21.15.46:0/3744734253 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f59ac0a24a0 0x7f59ac0a4960 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f59b0009230 tx=0x7f59b0005fd0 comp rx=0 tx=0).stop 2024-03-31T23:09:23.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:23.397+0000 7f59c7d02640 1 -- 172.21.15.46:0/3744734253 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f59c0101780 msgr2=0x7f59c0197990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:09:23.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:23.397+0000 7f59c7d02640 1 --2- 172.21.15.46:0/3744734253 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f59c0101780 0x7f59c0197990 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f59bc00e9c0 tx=0x7f59bc00ee80 comp rx=0 tx=0).stop 2024-03-31T23:09:23.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:23.397+0000 7f59c7d02640 1 -- 172.21.15.46:0/3744734253 shutdown_connections 2024-03-31T23:09:23.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:23.397+0000 7f59c7d02640 1 --2- 172.21.15.46:0/3744734253 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f59ac0a24a0 0x7f59ac0a4960 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:23.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:23.397+0000 7f59c7d02640 1 --2- 172.21.15.46:0/3744734253 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f59c0103330 0x7f59c019ef40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:23.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:23.397+0000 7f59c7d02640 1 --2- 172.21.15.46:0/3744734253 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f59c0102970 0x7f59c0197ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:23.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:23.397+0000 7f59c7d02640 1 --2- 172.21.15.46:0/3744734253 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f59c0101780 0x7f59c0197990 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:23.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:23.397+0000 7f59c7d02640 1 -- 172.21.15.46:0/3744734253 >> 172.21.15.46:0/3744734253 conn(0x7f59c00fccf0 msgr2=0x7f59c0105d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:09:23.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:23.397+0000 7f59c7d02640 1 -- 172.21.15.46:0/3744734253 shutdown_connections 2024-03-31T23:09:23.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:23.397+0000 7f59c7d02640 1 -- 172.21.15.46:0/3744734253 wait complete. 2024-03-31T23:09:23.409 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-31T23:04:32.823751+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-31T23:09:09.145711+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-31T23:09:10.148579+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-31T23:09:13.419519+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '91', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-31T23:09:23.410 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-31T23:09:23.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:23.514+0000 7f9971b7a640 1 -- 172.21.15.46:0/1689873739 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f996c100650 msgr2=0x7f996c100a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:09:23.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:23.514+0000 7f9971b7a640 1 --2- 172.21.15.46:0/1689873739 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f996c100650 0x7f996c100a70 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f995c009900 tx=0x7f995c02bc30 comp rx=0 tx=0).stop 2024-03-31T23:09:23.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:23.514+0000 7f9971b7a640 1 -- 172.21.15.46:0/1689873739 shutdown_connections 2024-03-31T23:09:23.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:23.514+0000 7f9971b7a640 1 --2- 172.21.15.46:0/1689873739 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f996c102200 0x7f996c108a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:23.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:23.514+0000 7f9971b7a640 1 --2- 172.21.15.46:0/1689873739 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f996c101840 0x7f996c101cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:23.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:23.514+0000 7f9971b7a640 1 --2- 172.21.15.46:0/1689873739 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f996c100650 0x7f996c100a70 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:23.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:23.514+0000 7f9971b7a640 1 -- 172.21.15.46:0/1689873739 >> 172.21.15.46:0/1689873739 conn(0x7f996c0fbbe0 msgr2=0x7f996c0fe000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:09:23.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:23.514+0000 7f9971b7a640 1 -- 172.21.15.46:0/1689873739 shutdown_connections 2024-03-31T23:09:23.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:23.514+0000 7f9971b7a640 1 -- 172.21.15.46:0/1689873739 wait complete. 2024-03-31T23:09:23.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:23.515+0000 7f9971b7a640 1 Processor -- start 2024-03-31T23:09:23.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:23.515+0000 7f9971b7a640 1 -- start start 2024-03-31T23:09:23.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:23.515+0000 7f9971b7a640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f996c100650 0x7f996c104590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:09:23.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:23.515+0000 7f9971b7a640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f996c101840 0x7f996c102c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:09:23.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:23.515+0000 7f9971b7a640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f996c102200 0x7f996c103160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:09:23.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:23.515+0000 7f9971b7a640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f996c10c170 con 0x7f996c102200 2024-03-31T23:09:23.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:23.515+0000 7f9971b7a640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f996c10bff0 con 0x7f996c100650 2024-03-31T23:09:23.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:23.515+0000 7f9971b7a640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f996c10c2f0 con 0x7f996c101840 2024-03-31T23:09:23.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:23.515+0000 7f996bfff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f996c102200 0x7f996c103160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:09:23.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:23.515+0000 7f996bfff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f996c102200 0x7f996c103160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44758/0 (socket says 172.21.15.46:44758) 2024-03-31T23:09:23.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:23.515+0000 7f996bfff640 1 -- 172.21.15.46:0/3526089716 learned_addr learned my addr 172.21.15.46:0/3526089716 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:09:23.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:23.515+0000 7f996b7fe640 1 --2- 172.21.15.46:0/3526089716 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f996c100650 0x7f996c104590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:09:23.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:23.515+0000 7f996affd640 1 --2- 172.21.15.46:0/3526089716 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f996c101840 0x7f996c102c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:09:23.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:23.516+0000 7f996bfff640 1 -- 172.21.15.46:0/3526089716 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f996c101840 msgr2=0x7f996c102c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:09:23.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:23.516+0000 7f996bfff640 1 --2- 172.21.15.46:0/3526089716 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f996c101840 0x7f996c102c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:23.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:23.516+0000 7f996bfff640 1 -- 172.21.15.46:0/3526089716 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f996c100650 msgr2=0x7f996c104590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:09:23.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:23.516+0000 7f996bfff640 1 --2- 172.21.15.46:0/3526089716 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f996c100650 0x7f996c104590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:23.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:23.516+0000 7f996bfff640 1 -- 172.21.15.46:0/3526089716 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f996c103a20 con 0x7f996c102200 2024-03-31T23:09:23.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:23.516+0000 7f996affd640 1 --2- 172.21.15.46:0/3526089716 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f996c101840 0x7f996c102c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T23:09:23.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:23.516+0000 7f996bfff640 1 --2- 172.21.15.46:0/3526089716 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f996c102200 0x7f996c103160 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f9964009fd0 tx=0x7f996400b260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:09:23.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:23.516+0000 7f9968ff9640 1 -- 172.21.15.46:0/3526089716 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9964017070 con 0x7f996c102200 2024-03-31T23:09:23.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:23.516+0000 7f9968ff9640 1 -- 172.21.15.46:0/3526089716 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9964003e20 con 0x7f996c102200 2024-03-31T23:09:23.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:23.516+0000 7f9971b7a640 1 -- 172.21.15.46:0/3526089716 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f996c1a1b50 con 0x7f996c102200 2024-03-31T23:09:23.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:23.516+0000 7f9968ff9640 1 -- 172.21.15.46:0/3526089716 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9964010460 con 0x7f996c102200 2024-03-31T23:09:23.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:23.516+0000 7f9971b7a640 1 -- 172.21.15.46:0/3526089716 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f996c1a2070 con 0x7f996c102200 2024-03-31T23:09:23.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:23.517+0000 7f9971b7a640 1 -- 172.21.15.46:0/3526089716 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f996c1a2240 con 0x7f996c102200 2024-03-31T23:09:23.522 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:23.518+0000 7f9968ff9640 1 -- 172.21.15.46:0/3526089716 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 16) ==== 137320+0+0 (secure 0 0 0) 0x7f9964003970 con 0x7f996c102200 2024-03-31T23:09:23.522 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:23.518+0000 7f9968ff9640 1 --2- 172.21.15.46:0/3526089716 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f99540a2520 0x7f99540a49e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:09:23.522 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:23.519+0000 7f996b7fe640 1 --2- 172.21.15.46:0/3526089716 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f99540a2520 0x7f99540a49e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:09:23.522 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:23.519+0000 7f9968ff9640 1 -- 172.21.15.46:0/3526089716 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8802+0+0 (secure 0 0 0) 0x7f99640ce740 con 0x7f996c102200 2024-03-31T23:09:23.522 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:23.519+0000 7f996b7fe640 1 --2- 172.21.15.46:0/3526089716 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f99540a2520 0x7f99540a49e0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f995c005ab0 tx=0x7f995c005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:09:23.524 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:23.523+0000 7f9968ff9640 1 -- 172.21.15.46:0/3526089716 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f996409b660 con 0x7f996c102200 2024-03-31T23:09:23.707 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:23.706+0000 7f9971b7a640 1 -- 172.21.15.46:0/3526089716 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f996c1a24b0 con 0x7f996c102200 2024-03-31T23:09:23.707 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:23.706+0000 7f9968ff9640 1 -- 172.21.15.46:0/3526089716 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v91) ==== 93+0+10 (secure 0 0 0) 0x7f996409af60 con 0x7f996c102200 2024-03-31T23:09:23.707 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 1 2024-03-31T23:09:23.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:23.708+0000 7f9971b7a640 1 -- 172.21.15.46:0/3526089716 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f99540a2520 msgr2=0x7f99540a49e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:09:23.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:23.708+0000 7f9971b7a640 1 --2- 172.21.15.46:0/3526089716 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f99540a2520 0x7f99540a49e0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f995c005ab0 tx=0x7f995c005fd0 comp rx=0 tx=0).stop 2024-03-31T23:09:23.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:23.708+0000 7f9971b7a640 1 -- 172.21.15.46:0/3526089716 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f996c102200 msgr2=0x7f996c103160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:09:23.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:23.708+0000 7f9971b7a640 1 --2- 172.21.15.46:0/3526089716 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f996c102200 0x7f996c103160 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f9964009fd0 tx=0x7f996400b260 comp rx=0 tx=0).stop 2024-03-31T23:09:23.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:23.708+0000 7f9971b7a640 1 -- 172.21.15.46:0/3526089716 shutdown_connections 2024-03-31T23:09:23.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:23.708+0000 7f9971b7a640 1 --2- 172.21.15.46:0/3526089716 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f99540a2520 0x7f99540a49e0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:23.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:23.708+0000 7f9971b7a640 1 --2- 172.21.15.46:0/3526089716 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f996c102200 0x7f996c103160 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:23.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:23.708+0000 7f9971b7a640 1 --2- 172.21.15.46:0/3526089716 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f996c101840 0x7f996c102c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:23.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:23.708+0000 7f9971b7a640 1 --2- 172.21.15.46:0/3526089716 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f996c100650 0x7f996c104590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:23.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:23.708+0000 7f9971b7a640 1 -- 172.21.15.46:0/3526089716 >> 172.21.15.46:0/3526089716 conn(0x7f996c0fbbe0 msgr2=0x7f996c069e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:09:23.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:23.708+0000 7f9971b7a640 1 -- 172.21.15.46:0/3526089716 shutdown_connections 2024-03-31T23:09:23.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:23.708+0000 7f9971b7a640 1 -- 172.21.15.46:0/3526089716 wait complete. 2024-03-31T23:09:23.720 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-31T23:09:23.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:23.825+0000 7ff1d6bae640 1 -- 172.21.15.46:0/1586259760 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff1d01001b0 msgr2=0x7ff1d0109d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:09:23.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:23.825+0000 7ff1d6bae640 1 --2- 172.21.15.46:0/1586259760 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff1d01001b0 0x7ff1d0109d60 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7ff1c8009ca0 tx=0x7ff1c802c0f0 comp rx=0 tx=0).stop 2024-03-31T23:09:23.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:23.825+0000 7ff1d6bae640 1 -- 172.21.15.46:0/1586259760 shutdown_connections 2024-03-31T23:09:23.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:23.825+0000 7ff1d6bae640 1 --2- 172.21.15.46:0/1586259760 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff1d01001b0 0x7ff1d0109d60 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:23.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:23.825+0000 7ff1d6bae640 1 --2- 172.21.15.46:0/1586259760 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff1d00ff7f0 0x7ff1d00ffc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:23.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:23.825+0000 7ff1d6bae640 1 --2- 172.21.15.46:0/1586259760 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff1d0102d80 0x7ff1d01031a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:23.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:23.825+0000 7ff1d6bae640 1 -- 172.21.15.46:0/1586259760 >> 172.21.15.46:0/1586259760 conn(0x7ff1d00fb350 msgr2=0x7ff1d00fd7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:09:23.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:23.825+0000 7ff1d6bae640 1 -- 172.21.15.46:0/1586259760 shutdown_connections 2024-03-31T23:09:23.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:23.825+0000 7ff1d6bae640 1 -- 172.21.15.46:0/1586259760 wait complete. 2024-03-31T23:09:23.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:23.826+0000 7ff1d6bae640 1 Processor -- start 2024-03-31T23:09:23.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:23.826+0000 7ff1d6bae640 1 -- start start 2024-03-31T23:09:23.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:23.826+0000 7ff1d6bae640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff1d00ff7f0 0x7ff1d01978a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:09:23.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:23.826+0000 7ff1d6bae640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff1d01001b0 0x7ff1d0197de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:09:23.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:23.826+0000 7ff1d6bae640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff1d0102d80 0x7ff1d019ee50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:09:23.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:23.826+0000 7ff1d6bae640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7ff1d010d290 con 0x7ff1d01001b0 2024-03-31T23:09:23.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:23.826+0000 7ff1d6bae640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7ff1d010d110 con 0x7ff1d0102d80 2024-03-31T23:09:23.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:23.826+0000 7ff1d6bae640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7ff1d010d410 con 0x7ff1d00ff7f0 2024-03-31T23:09:23.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:23.826+0000 7ff1cffff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff1d01001b0 0x7ff1d0197de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:09:23.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:23.826+0000 7ff1cffff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff1d01001b0 0x7ff1d0197de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44778/0 (socket says 172.21.15.46:44778) 2024-03-31T23:09:23.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:23.826+0000 7ff1cffff640 1 -- 172.21.15.46:0/3186356477 learned_addr learned my addr 172.21.15.46:0/3186356477 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:09:23.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:23.826+0000 7ff1d514b640 1 --2- 172.21.15.46:0/3186356477 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff1d0102d80 0x7ff1d019ee50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:09:23.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:23.826+0000 7ff1d494a640 1 --2- 172.21.15.46:0/3186356477 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff1d00ff7f0 0x7ff1d01978a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:09:23.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:23.826+0000 7ff1cffff640 1 -- 172.21.15.46:0/3186356477 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff1d00ff7f0 msgr2=0x7ff1d01978a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:09:23.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:23.826+0000 7ff1cffff640 1 --2- 172.21.15.46:0/3186356477 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff1d00ff7f0 0x7ff1d01978a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:23.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:23.826+0000 7ff1cffff640 1 -- 172.21.15.46:0/3186356477 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff1d0102d80 msgr2=0x7ff1d019ee50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:09:23.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:23.826+0000 7ff1cffff640 1 --2- 172.21.15.46:0/3186356477 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff1d0102d80 0x7ff1d019ee50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:23.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:23.826+0000 7ff1cffff640 1 -- 172.21.15.46:0/3186356477 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1d019f4c0 con 0x7ff1d01001b0 2024-03-31T23:09:23.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:23.826+0000 7ff1d494a640 1 --2- 172.21.15.46:0/3186356477 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff1d00ff7f0 0x7ff1d01978a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T23:09:23.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:23.826+0000 7ff1cffff640 1 --2- 172.21.15.46:0/3186356477 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff1d01001b0 0x7ff1d0197de0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7ff1bc00c920 tx=0x7ff1bc00cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:09:23.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:23.827+0000 7ff1cdffb640 1 -- 172.21.15.46:0/3186356477 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1bc007c70 con 0x7ff1d01001b0 2024-03-31T23:09:23.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:23.827+0000 7ff1d6bae640 1 -- 172.21.15.46:0/3186356477 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff1d019f7b0 con 0x7ff1d01001b0 2024-03-31T23:09:23.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:23.827+0000 7ff1cdffb640 1 -- 172.21.15.46:0/3186356477 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff1bc007e10 con 0x7ff1d01001b0 2024-03-31T23:09:23.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:23.827+0000 7ff1cdffb640 1 -- 172.21.15.46:0/3186356477 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1bc018820 con 0x7ff1d01001b0 2024-03-31T23:09:23.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:23.827+0000 7ff1d6bae640 1 -- 172.21.15.46:0/3186356477 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff1d019fd50 con 0x7ff1d01001b0 2024-03-31T23:09:23.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:23.828+0000 7ff1d6bae640 1 -- 172.21.15.46:0/3186356477 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff1d0108c40 con 0x7ff1d01001b0 2024-03-31T23:09:23.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:23.829+0000 7ff1cdffb640 1 -- 172.21.15.46:0/3186356477 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 16) ==== 137320+0+0 (secure 0 0 0) 0x7ff1bc0189c0 con 0x7ff1d01001b0 2024-03-31T23:09:23.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:23.829+0000 7ff1cdffb640 1 --2- 172.21.15.46:0/3186356477 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7ff1c00a24a0 0x7ff1c00a4960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:09:23.832 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:23.829+0000 7ff1d494a640 1 --2- 172.21.15.46:0/3186356477 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7ff1c00a24a0 0x7ff1c00a4960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:09:23.832 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:23.830+0000 7ff1cdffb640 1 -- 172.21.15.46:0/3186356477 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8802+0+0 (secure 0 0 0) 0x7ff1bc0cea60 con 0x7ff1d01001b0 2024-03-31T23:09:23.832 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:23.830+0000 7ff1d494a640 1 --2- 172.21.15.46:0/3186356477 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7ff1c00a24a0 0x7ff1c00a4960 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7ff1d0069a60 tx=0x7ff1c4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:09:23.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:23.834+0000 7ff1cdffb640 1 -- 172.21.15.46:0/3186356477 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7ff1bc014030 con 0x7ff1d01001b0 2024-03-31T23:09:24.015 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:24.014+0000 7ff1d6bae640 1 -- 172.21.15.46:0/3186356477 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7ff1d0062350 con 0x7ff1d01001b0 2024-03-31T23:09:24.015 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:24.014+0000 7ff1cdffb640 1 -- 172.21.15.46:0/3186356477 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v91) ==== 104+0+11 (secure 0 0 0) 0x7ff1bc09b9e0 con 0x7ff1d01001b0 2024-03-31T23:09:24.015 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 32 2024-03-31T23:09:24.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:24.016+0000 7ff1d6bae640 1 -- 172.21.15.46:0/3186356477 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7ff1c00a24a0 msgr2=0x7ff1c00a4960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:09:24.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:24.016+0000 7ff1d6bae640 1 --2- 172.21.15.46:0/3186356477 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7ff1c00a24a0 0x7ff1c00a4960 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7ff1d0069a60 tx=0x7ff1c4005f90 comp rx=0 tx=0).stop 2024-03-31T23:09:24.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:24.016+0000 7ff1d6bae640 1 -- 172.21.15.46:0/3186356477 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff1d01001b0 msgr2=0x7ff1d0197de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:09:24.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:24.016+0000 7ff1d6bae640 1 --2- 172.21.15.46:0/3186356477 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff1d01001b0 0x7ff1d0197de0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7ff1bc00c920 tx=0x7ff1bc00cde0 comp rx=0 tx=0).stop 2024-03-31T23:09:24.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:24.016+0000 7ff1d6bae640 1 -- 172.21.15.46:0/3186356477 shutdown_connections 2024-03-31T23:09:24.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:24.016+0000 7ff1d6bae640 1 --2- 172.21.15.46:0/3186356477 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7ff1c00a24a0 0x7ff1c00a4960 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:24.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:24.016+0000 7ff1d6bae640 1 --2- 172.21.15.46:0/3186356477 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff1d0102d80 0x7ff1d019ee50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:24.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:24.016+0000 7ff1d6bae640 1 --2- 172.21.15.46:0/3186356477 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff1d01001b0 0x7ff1d0197de0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:24.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:24.016+0000 7ff1d6bae640 1 --2- 172.21.15.46:0/3186356477 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff1d00ff7f0 0x7ff1d01978a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:24.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:24.016+0000 7ff1d6bae640 1 -- 172.21.15.46:0/3186356477 >> 172.21.15.46:0/3186356477 conn(0x7ff1d00fb350 msgr2=0x7ff1d0105e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:09:24.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:24.016+0000 7ff1d6bae640 1 -- 172.21.15.46:0/3186356477 shutdown_connections 2024-03-31T23:09:24.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:24.016+0000 7ff1d6bae640 1 -- 172.21.15.46:0/3186356477 wait complete. 2024-03-31T23:09:24.028 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-31T23:09:24.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:24.133+0000 7fb44aeaa640 1 -- 172.21.15.46:0/3843435215 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb4441074d0 msgr2=0x7fb4441098c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:09:24.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:24.133+0000 7fb44aeaa640 1 --2- 172.21.15.46:0/3843435215 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb4441074d0 0x7fb4441098c0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fb43c009ca0 tx=0x7fb43c02c0f0 comp rx=0 tx=0).stop 2024-03-31T23:09:24.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:24.133+0000 7fb44aeaa640 1 -- 172.21.15.46:0/3843435215 shutdown_connections 2024-03-31T23:09:24.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:24.133+0000 7fb44aeaa640 1 --2- 172.21.15.46:0/3843435215 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb4441074d0 0x7fb4441098c0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:24.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:24.133+0000 7fb44aeaa640 1 --2- 172.21.15.46:0/3843435215 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb444104ba0 0x7fb444106f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:24.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:24.133+0000 7fb44aeaa640 1 --2- 172.21.15.46:0/3843435215 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb444102270 0x7fb444104660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:24.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:24.133+0000 7fb44aeaa640 1 -- 172.21.15.46:0/3843435215 >> 172.21.15.46:0/3843435215 conn(0x7fb4440fbbc0 msgr2=0x7fb4440fe000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:09:24.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:24.133+0000 7fb44aeaa640 1 -- 172.21.15.46:0/3843435215 shutdown_connections 2024-03-31T23:09:24.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:24.133+0000 7fb44aeaa640 1 -- 172.21.15.46:0/3843435215 wait complete. 2024-03-31T23:09:24.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:24.133+0000 7fb44aeaa640 1 Processor -- start 2024-03-31T23:09:24.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:24.133+0000 7fb44aeaa640 1 -- start start 2024-03-31T23:09:24.137 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:24.133+0000 7fb44aeaa640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb444102270 0x7fb444101be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:09:24.137 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:24.133+0000 7fb44aeaa640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb444104ba0 0x7fb444100220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:09:24.137 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:24.133+0000 7fb44aeaa640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb4441074d0 0x7fb444100760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:09:24.137 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:24.133+0000 7fb44aeaa640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fb44410d280 con 0x7fb4441074d0 2024-03-31T23:09:24.137 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:24.133+0000 7fb44aeaa640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fb44410d100 con 0x7fb444102270 2024-03-31T23:09:24.137 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:24.133+0000 7fb44aeaa640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fb44410d400 con 0x7fb444104ba0 2024-03-31T23:09:24.137 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:24.134+0000 7fb449447640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb4441074d0 0x7fb444100760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:09:24.137 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:24.134+0000 7fb449447640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb4441074d0 0x7fb444100760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44788/0 (socket says 172.21.15.46:44788) 2024-03-31T23:09:24.137 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:24.134+0000 7fb449447640 1 -- 172.21.15.46:0/1404783437 learned_addr learned my addr 172.21.15.46:0/1404783437 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:09:24.137 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:24.134+0000 7fb448c46640 1 --2- 172.21.15.46:0/1404783437 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb444102270 0x7fb444101be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:09:24.137 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:24.134+0000 7fb443fff640 1 --2- 172.21.15.46:0/1404783437 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb444104ba0 0x7fb444100220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:09:24.137 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:24.134+0000 7fb449447640 1 -- 172.21.15.46:0/1404783437 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb444104ba0 msgr2=0x7fb444100220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:09:24.137 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:24.134+0000 7fb449447640 1 --2- 172.21.15.46:0/1404783437 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb444104ba0 0x7fb444100220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:24.137 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:24.134+0000 7fb449447640 1 -- 172.21.15.46:0/1404783437 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb444102270 msgr2=0x7fb444101be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:09:24.138 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:24.134+0000 7fb449447640 1 --2- 172.21.15.46:0/1404783437 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb444102270 0x7fb444101be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:24.138 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:24.134+0000 7fb449447640 1 -- 172.21.15.46:0/1404783437 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb444100ff0 con 0x7fb4441074d0 2024-03-31T23:09:24.138 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:24.134+0000 7fb449447640 1 --2- 172.21.15.46:0/1404783437 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb4441074d0 0x7fb444100760 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fb43c032380 tx=0x7fb43c032820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:09:24.138 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:24.134+0000 7fb441ffb640 1 -- 172.21.15.46:0/1404783437 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb43c041020 con 0x7fb4441074d0 2024-03-31T23:09:24.138 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:24.134+0000 7fb441ffb640 1 -- 172.21.15.46:0/1404783437 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb43c003e20 con 0x7fb4441074d0 2024-03-31T23:09:24.138 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:24.134+0000 7fb44aeaa640 1 -- 172.21.15.46:0/1404783437 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb4441a1a10 con 0x7fb4441074d0 2024-03-31T23:09:24.138 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:24.134+0000 7fb441ffb640 1 -- 172.21.15.46:0/1404783437 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb43c039040 con 0x7fb4441074d0 2024-03-31T23:09:24.138 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:24.134+0000 7fb44aeaa640 1 -- 172.21.15.46:0/1404783437 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb4441a1f60 con 0x7fb4441074d0 2024-03-31T23:09:24.139 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:24.136+0000 7fb44aeaa640 1 -- 172.21.15.46:0/1404783437 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb444068540 con 0x7fb4441074d0 2024-03-31T23:09:24.139 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:24.136+0000 7fb441ffb640 1 -- 172.21.15.46:0/1404783437 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 16) ==== 137320+0+0 (secure 0 0 0) 0x7fb43c044020 con 0x7fb4441074d0 2024-03-31T23:09:24.139 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:24.137+0000 7fb441ffb640 1 --2- 172.21.15.46:0/1404783437 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fb4340a24a0 0x7fb4340a4960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:09:24.139 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:24.137+0000 7fb448c46640 1 --2- 172.21.15.46:0/1404783437 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fb4340a24a0 0x7fb4340a4960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:09:24.139 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:24.137+0000 7fb441ffb640 1 -- 172.21.15.46:0/1404783437 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8802+0+0 (secure 0 0 0) 0x7fb43c0f0b70 con 0x7fb4441074d0 2024-03-31T23:09:24.139 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:24.138+0000 7fb448c46640 1 --2- 172.21.15.46:0/1404783437 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fb4340a24a0 0x7fb4340a4960 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fb4380095f0 tx=0x7fb438005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:09:24.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:24.142+0000 7fb441ffb640 1 -- 172.21.15.46:0/1404783437 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fb43c04b020 con 0x7fb4441074d0 2024-03-31T23:09:24.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:24.321+0000 7fb44aeaa640 1 -- 172.21.15.46:0/1404783437 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fb444062350 con 0x7fb4441074d0 2024-03-31T23:09:24.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:24.321+0000 7fb441ffb640 1 -- 172.21.15.46:0/1404783437 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v91) ==== 100+0+11 (secure 0 0 0) 0x7fb43c0bda90 con 0x7fb4441074d0 2024-03-31T23:09:24.322 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 64 2024-03-31T23:09:24.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:24.323+0000 7fb44aeaa640 1 -- 172.21.15.46:0/1404783437 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fb4340a24a0 msgr2=0x7fb4340a4960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:09:24.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:24.323+0000 7fb44aeaa640 1 --2- 172.21.15.46:0/1404783437 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fb4340a24a0 0x7fb4340a4960 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fb4380095f0 tx=0x7fb438005f90 comp rx=0 tx=0).stop 2024-03-31T23:09:24.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:24.323+0000 7fb44aeaa640 1 -- 172.21.15.46:0/1404783437 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb4441074d0 msgr2=0x7fb444100760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:09:24.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:24.323+0000 7fb44aeaa640 1 --2- 172.21.15.46:0/1404783437 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb4441074d0 0x7fb444100760 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fb43c032380 tx=0x7fb43c032820 comp rx=0 tx=0).stop 2024-03-31T23:09:24.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:24.323+0000 7fb44aeaa640 1 -- 172.21.15.46:0/1404783437 shutdown_connections 2024-03-31T23:09:24.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:24.323+0000 7fb44aeaa640 1 --2- 172.21.15.46:0/1404783437 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fb4340a24a0 0x7fb4340a4960 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:24.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:24.323+0000 7fb44aeaa640 1 --2- 172.21.15.46:0/1404783437 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb4441074d0 0x7fb444100760 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:24.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:24.323+0000 7fb44aeaa640 1 --2- 172.21.15.46:0/1404783437 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb444104ba0 0x7fb444100220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:24.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:24.323+0000 7fb44aeaa640 1 --2- 172.21.15.46:0/1404783437 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb444102270 0x7fb444101be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:24.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:24.323+0000 7fb44aeaa640 1 -- 172.21.15.46:0/1404783437 >> 172.21.15.46:0/1404783437 conn(0x7fb4440fbbc0 msgr2=0x7fb444102c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:09:24.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:24.323+0000 7fb44aeaa640 1 -- 172.21.15.46:0/1404783437 shutdown_connections 2024-03-31T23:09:24.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:24.323+0000 7fb44aeaa640 1 -- 172.21.15.46:0/1404783437 wait complete. 2024-03-31T23:09:24.335 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-31T23:09:24.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:24.440+0000 7f6357348640 1 -- 172.21.15.46:0/1434012133 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f63501033c0 msgr2=0x7f6350109c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:09:24.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:24.440+0000 7f6357348640 1 --2- 172.21.15.46:0/1434012133 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f63501033c0 0x7f6350109c50 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f6348009cd0 tx=0x7f634802c0f0 comp rx=0 tx=0).stop 2024-03-31T23:09:24.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:24.440+0000 7f6357348640 1 -- 172.21.15.46:0/1434012133 shutdown_connections 2024-03-31T23:09:24.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:24.440+0000 7f6357348640 1 --2- 172.21.15.46:0/1434012133 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f63501033c0 0x7f6350109c50 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:24.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:24.440+0000 7f6357348640 1 --2- 172.21.15.46:0/1434012133 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6350102a00 0x7f6350102e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:24.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:24.440+0000 7f6357348640 1 --2- 172.21.15.46:0/1434012133 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f63500ffa30 0x7f63500ffe50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:24.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:24.440+0000 7f6357348640 1 -- 172.21.15.46:0/1434012133 >> 172.21.15.46:0/1434012133 conn(0x7f63500fb340 msgr2=0x7f63500fd7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:09:24.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:24.440+0000 7f6357348640 1 -- 172.21.15.46:0/1434012133 shutdown_connections 2024-03-31T23:09:24.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:24.440+0000 7f6357348640 1 -- 172.21.15.46:0/1434012133 wait complete. 2024-03-31T23:09:24.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:24.441+0000 7f6357348640 1 Processor -- start 2024-03-31T23:09:24.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:24.441+0000 7f6357348640 1 -- start start 2024-03-31T23:09:24.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:24.441+0000 7f6357348640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f63500ffa30 0x7f6350102370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:09:24.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:24.441+0000 7f6357348640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6350102a00 0x7f63501009b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:09:24.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:24.441+0000 7f6357348640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f63501033c0 0x7f6350100ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:09:24.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:24.441+0000 7f6357348640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f635010d2c0 con 0x7f6350102a00 2024-03-31T23:09:24.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:24.441+0000 7f6357348640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f635010d140 con 0x7f63500ffa30 2024-03-31T23:09:24.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:24.441+0000 7f6357348640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f635010d440 con 0x7f63501033c0 2024-03-31T23:09:24.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:24.441+0000 7f63548e3640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6350102a00 0x7f63501009b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:09:24.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:24.441+0000 7f63548e3640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6350102a00 0x7f63501009b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44796/0 (socket says 172.21.15.46:44796) 2024-03-31T23:09:24.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:24.441+0000 7f63548e3640 1 -- 172.21.15.46:0/2266026602 learned_addr learned my addr 172.21.15.46:0/2266026602 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:09:24.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:24.441+0000 7f63550e4640 1 --2- 172.21.15.46:0/2266026602 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f63500ffa30 0x7f6350102370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:09:24.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:24.441+0000 7f63558e5640 1 --2- 172.21.15.46:0/2266026602 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f63501033c0 0x7f6350100ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:09:24.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:24.441+0000 7f63548e3640 1 -- 172.21.15.46:0/2266026602 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f63501033c0 msgr2=0x7f6350100ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:09:24.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:24.441+0000 7f63548e3640 1 --2- 172.21.15.46:0/2266026602 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f63501033c0 0x7f6350100ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:24.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:24.441+0000 7f63548e3640 1 -- 172.21.15.46:0/2266026602 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f63500ffa30 msgr2=0x7f6350102370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:09:24.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:24.441+0000 7f63548e3640 1 --2- 172.21.15.46:0/2266026602 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f63500ffa30 0x7f6350102370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:24.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:24.441+0000 7f63548e3640 1 -- 172.21.15.46:0/2266026602 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6350101780 con 0x7f6350102a00 2024-03-31T23:09:24.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:24.441+0000 7f63558e5640 1 --2- 172.21.15.46:0/2266026602 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f63501033c0 0x7f6350100ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T23:09:24.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:24.441+0000 7f63550e4640 1 --2- 172.21.15.46:0/2266026602 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f63500ffa30 0x7f6350102370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:09:24.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:24.441+0000 7f63548e3640 1 --2- 172.21.15.46:0/2266026602 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6350102a00 0x7f63501009b0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f634000c970 tx=0x7f634000ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:09:24.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:24.442+0000 7f63467fc640 1 -- 172.21.15.46:0/2266026602 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6340007bb0 con 0x7f6350102a00 2024-03-31T23:09:24.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:24.442+0000 7f63467fc640 1 -- 172.21.15.46:0/2266026602 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6340007d50 con 0x7f6350102a00 2024-03-31T23:09:24.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:24.442+0000 7f6357348640 1 -- 172.21.15.46:0/2266026602 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f63501a1a80 con 0x7f6350102a00 2024-03-31T23:09:24.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:24.442+0000 7f63467fc640 1 -- 172.21.15.46:0/2266026602 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f634000f450 con 0x7f6350102a00 2024-03-31T23:09:24.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:24.442+0000 7f6357348640 1 -- 172.21.15.46:0/2266026602 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f63501a1fd0 con 0x7f6350102a00 2024-03-31T23:09:24.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:24.443+0000 7f6357348640 1 -- 172.21.15.46:0/2266026602 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6350108b30 con 0x7f6350102a00 2024-03-31T23:09:24.466 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:24.444+0000 7f63467fc640 1 -- 172.21.15.46:0/2266026602 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 16) ==== 137320+0+0 (secure 0 0 0) 0x7f6340004f00 con 0x7f6350102a00 2024-03-31T23:09:24.466 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:24.444+0000 7f63467fc640 1 --2- 172.21.15.46:0/2266026602 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f633c0a2520 0x7f633c0a49e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:09:24.466 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:24.444+0000 7f63550e4640 1 --2- 172.21.15.46:0/2266026602 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f633c0a2520 0x7f633c0a49e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:09:24.466 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:24.444+0000 7f63467fc640 1 -- 172.21.15.46:0/2266026602 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8802+0+0 (secure 0 0 0) 0x7f63400cea50 con 0x7f6350102a00 2024-03-31T23:09:24.466 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:24.445+0000 7f63550e4640 1 --2- 172.21.15.46:0/2266026602 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f633c0a2520 0x7f633c0a49e0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f6350069b20 tx=0x7f634c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:09:24.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:24.448+0000 7f63467fc640 1 -- 172.21.15.46:0/2266026602 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f6340014030 con 0x7f6350102a00 2024-03-31T23:09:24.611 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:09:24 smithi046 ceph-mon[32498]: pgmap v298: 129 pgs: 129 active+clean; 579 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-31T23:09:24.611 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:09:24 smithi046 ceph-mon[32498]: from='client.? 172.21.15.46:0/3744734253' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-31T23:09:24.611 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:09:24 smithi046 ceph-mon[32498]: from='client.? 172.21.15.46:0/3526089716' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-31T23:09:24.612 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:09:24 smithi046 ceph-mon[32498]: from='client.? 172.21.15.46:0/3186356477' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-31T23:09:24.612 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:09:24 smithi046 ceph-mon[32498]: from='client.? 172.21.15.46:0/1404783437' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-31T23:09:24.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:24.629+0000 7f6357348640 1 -- 172.21.15.46:0/2266026602 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f635002d300 con 0x7f6350102a00 2024-03-31T23:09:24.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:24.629+0000 7f63467fc640 1 -- 172.21.15.46:0/2266026602 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v91) ==== 103+0+11 (secure 0 0 0) 0x7f634009ba50 con 0x7f6350102a00 2024-03-31T23:09:24.630 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 32 2024-03-31T23:09:24.632 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:24.631+0000 7f6357348640 1 -- 172.21.15.46:0/2266026602 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f633c0a2520 msgr2=0x7f633c0a49e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:09:24.632 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:24.631+0000 7f6357348640 1 --2- 172.21.15.46:0/2266026602 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f633c0a2520 0x7f633c0a49e0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f6350069b20 tx=0x7f634c005f90 comp rx=0 tx=0).stop 2024-03-31T23:09:24.632 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:24.631+0000 7f6357348640 1 -- 172.21.15.46:0/2266026602 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6350102a00 msgr2=0x7f63501009b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:09:24.633 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:24.631+0000 7f6357348640 1 --2- 172.21.15.46:0/2266026602 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6350102a00 0x7f63501009b0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f634000c970 tx=0x7f634000ce30 comp rx=0 tx=0).stop 2024-03-31T23:09:24.633 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:24.631+0000 7f6357348640 1 -- 172.21.15.46:0/2266026602 shutdown_connections 2024-03-31T23:09:24.633 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:24.631+0000 7f6357348640 1 --2- 172.21.15.46:0/2266026602 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f633c0a2520 0x7f633c0a49e0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:24.633 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:24.631+0000 7f6357348640 1 --2- 172.21.15.46:0/2266026602 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f63501033c0 0x7f6350100ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:24.633 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:24.631+0000 7f6357348640 1 --2- 172.21.15.46:0/2266026602 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6350102a00 0x7f63501009b0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:24.633 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:24.631+0000 7f6357348640 1 --2- 172.21.15.46:0/2266026602 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f63500ffa30 0x7f6350102370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:24.633 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:24.631+0000 7f6357348640 1 -- 172.21.15.46:0/2266026602 >> 172.21.15.46:0/2266026602 conn(0x7f63500fb340 msgr2=0x7f6350105d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:09:24.633 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:24.631+0000 7f6357348640 1 -- 172.21.15.46:0/2266026602 shutdown_connections 2024-03-31T23:09:24.633 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:24.631+0000 7f6357348640 1 -- 172.21.15.46:0/2266026602 wait complete. 2024-03-31T23:09:24.644 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-31T23:09:24.693 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:09:24 smithi142 ceph-mon[35271]: pgmap v298: 129 pgs: 129 active+clean; 579 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-31T23:09:24.693 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:09:24 smithi142 ceph-mon[35271]: from='client.? 172.21.15.46:0/3744734253' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-31T23:09:24.693 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:09:24 smithi142 ceph-mon[35271]: from='client.? 172.21.15.46:0/3526089716' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-31T23:09:24.693 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:09:24 smithi142 ceph-mon[35271]: from='client.? 172.21.15.46:0/3186356477' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-31T23:09:24.693 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:09:24 smithi142 ceph-mon[35271]: from='client.? 172.21.15.46:0/1404783437' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-31T23:09:24.749 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:24.746+0000 7ff48cb25640 1 -- 172.21.15.46:0/4188644948 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff488101c00 msgr2=0x7ff488102020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:09:24.750 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:24.746+0000 7ff48cb25640 1 --2- 172.21.15.46:0/4188644948 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff488101c00 0x7ff488102020 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7ff480009900 tx=0x7ff48002bc30 comp rx=0 tx=0).stop 2024-03-31T23:09:24.750 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:24.746+0000 7ff48cb25640 1 -- 172.21.15.46:0/4188644948 shutdown_connections 2024-03-31T23:09:24.750 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:24.746+0000 7ff48cb25640 1 --2- 172.21.15.46:0/4188644948 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff4881012f0 0x7ff488108b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:24.750 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:24.746+0000 7ff48cb25640 1 --2- 172.21.15.46:0/4188644948 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff488100930 0x7ff488100db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:24.750 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:24.746+0000 7ff48cb25640 1 --2- 172.21.15.46:0/4188644948 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff488101c00 0x7ff488102020 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:24.750 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:24.746+0000 7ff48cb25640 1 -- 172.21.15.46:0/4188644948 >> 172.21.15.46:0/4188644948 conn(0x7ff4880fb340 msgr2=0x7ff4880fd7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:09:24.750 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:24.746+0000 7ff48cb25640 1 -- 172.21.15.46:0/4188644948 shutdown_connections 2024-03-31T23:09:24.750 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:24.746+0000 7ff48cb25640 1 -- 172.21.15.46:0/4188644948 wait complete. 2024-03-31T23:09:24.750 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:24.747+0000 7ff48cb25640 1 Processor -- start 2024-03-31T23:09:24.750 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:24.747+0000 7ff48cb25640 1 -- start start 2024-03-31T23:09:24.751 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:24.747+0000 7ff48cb25640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff488100930 0x7ff4881003b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:09:24.751 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:24.747+0000 7ff48cb25640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff4881012f0 0x7ff48819c740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:09:24.751 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:24.747+0000 7ff48cb25640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff488101c00 0x7ff48819ed80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:09:24.751 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:24.747+0000 7ff48cb25640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7ff48810c190 con 0x7ff488100930 2024-03-31T23:09:24.751 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:24.747+0000 7ff48cb25640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7ff48810c010 con 0x7ff4881012f0 2024-03-31T23:09:24.751 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:24.747+0000 7ff48cb25640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7ff48810c310 con 0x7ff488101c00 2024-03-31T23:09:24.751 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:24.747+0000 7ff48659c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff488100930 0x7ff4881003b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:09:24.751 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:24.747+0000 7ff48659c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff488100930 0x7ff4881003b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44804/0 (socket says 172.21.15.46:44804) 2024-03-31T23:09:24.751 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:24.747+0000 7ff48659c640 1 -- 172.21.15.46:0/2402447570 learned_addr learned my addr 172.21.15.46:0/2402447570 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:09:24.751 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:24.748+0000 7ff485d9b640 1 --2- 172.21.15.46:0/2402447570 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff4881012f0 0x7ff48819c740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:09:24.751 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:24.748+0000 7ff48659c640 1 -- 172.21.15.46:0/2402447570 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff488101c00 msgr2=0x7ff48819ed80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:09:24.751 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:24.748+0000 7ff48659c640 1 --2- 172.21.15.46:0/2402447570 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff488101c00 0x7ff48819ed80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:24.751 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:24.748+0000 7ff48659c640 1 -- 172.21.15.46:0/2402447570 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff4881012f0 msgr2=0x7ff48819c740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:09:24.751 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:24.748+0000 7ff48659c640 1 --2- 172.21.15.46:0/2402447570 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff4881012f0 0x7ff48819c740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:24.752 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:24.748+0000 7ff48659c640 1 -- 172.21.15.46:0/2402447570 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff48819f640 con 0x7ff488100930 2024-03-31T23:09:24.752 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:24.748+0000 7ff48659c640 1 --2- 172.21.15.46:0/2402447570 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff488100930 0x7ff4881003b0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7ff480000c00 tx=0x7ff480004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:09:24.752 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:24.748+0000 7ff4777fe640 1 -- 172.21.15.46:0/2402447570 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff480043020 con 0x7ff488100930 2024-03-31T23:09:24.752 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:24.748+0000 7ff4777fe640 1 -- 172.21.15.46:0/2402447570 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff480004aa0 con 0x7ff488100930 2024-03-31T23:09:24.752 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:24.748+0000 7ff48cb25640 1 -- 172.21.15.46:0/2402447570 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff48819f8d0 con 0x7ff488100930 2024-03-31T23:09:24.752 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:24.748+0000 7ff4777fe640 1 -- 172.21.15.46:0/2402447570 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff480003c60 con 0x7ff488100930 2024-03-31T23:09:24.752 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:24.748+0000 7ff48cb25640 1 -- 172.21.15.46:0/2402447570 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff48819fe70 con 0x7ff488100930 2024-03-31T23:09:24.753 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:24.749+0000 7ff48cb25640 1 -- 172.21.15.46:0/2402447570 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff488107b30 con 0x7ff488100930 2024-03-31T23:09:24.753 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:24.751+0000 7ff4777fe640 1 -- 172.21.15.46:0/2402447570 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 16) ==== 137320+0+0 (secure 0 0 0) 0x7ff480046020 con 0x7ff488100930 2024-03-31T23:09:24.753 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:24.751+0000 7ff4777fe640 1 --2- 172.21.15.46:0/2402447570 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7ff4700a2520 0x7ff4700a49e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:09:24.753 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:24.751+0000 7ff485d9b640 1 --2- 172.21.15.46:0/2402447570 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7ff4700a2520 0x7ff4700a49e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:09:24.753 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:24.751+0000 7ff4777fe640 1 -- 172.21.15.46:0/2402447570 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8802+0+0 (secure 0 0 0) 0x7ff4800f1170 con 0x7ff488100930 2024-03-31T23:09:24.753 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:24.752+0000 7ff485d9b640 1 --2- 172.21.15.46:0/2402447570 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7ff4700a2520 0x7ff4700a49e0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7ff488100120 tx=0x7ff47c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:09:24.756 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:24.755+0000 7ff4777fe640 1 -- 172.21.15.46:0/2402447570 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7ff48004d020 con 0x7ff488100930 2024-03-31T23:09:24.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:09:24 smithi162 ceph-mon[36237]: pgmap v298: 129 pgs: 129 active+clean; 579 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-31T23:09:24.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:09:24 smithi162 ceph-mon[36237]: from='client.? 172.21.15.46:0/3744734253' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-31T23:09:24.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:09:24 smithi162 ceph-mon[36237]: from='client.? 172.21.15.46:0/3526089716' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-31T23:09:24.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:09:24 smithi162 ceph-mon[36237]: from='client.? 172.21.15.46:0/3186356477' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-31T23:09:24.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:09:24 smithi162 ceph-mon[36237]: from='client.? 172.21.15.46:0/1404783437' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-31T23:09:24.951 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:24.950+0000 7ff48cb25640 1 -- 172.21.15.46:0/2402447570 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7ff48802d3c0 con 0x7ff488100930 2024-03-31T23:09:24.951 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:24.950+0000 7ff4777fe640 1 -- 172.21.15.46:0/2402447570 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v91) ==== 81+0+591 (secure 0 0 0) 0x7ff4800be0f0 con 0x7ff488100930 2024-03-31T23:09:24.952 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:0/4021039857 2024-04-01T23:02:08.493619+0000 2024-03-31T23:09:24.952 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6801/3681182647 2024-04-01T23:02:08.493619+0000 2024-03-31T23:09:24.952 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:0/4022214806 2024-04-01T23:02:08.493619+0000 2024-03-31T23:09:24.952 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6800/1515528215 2024-04-01T23:01:41.910162+0000 2024-03-31T23:09:24.952 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6800/3681182647 2024-04-01T23:02:08.493619+0000 2024-03-31T23:09:24.952 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:0/143885707 2024-04-01T23:01:41.910162+0000 2024-03-31T23:09:24.952 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6801/1515528215 2024-04-01T23:01:41.910162+0000 2024-03-31T23:09:24.952 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:0/1680389778 2024-04-01T23:01:41.910162+0000 2024-03-31T23:09:24.952 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:0/1782133966 2024-04-01T23:01:41.910162+0000 2024-03-31T23:09:24.952 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:0/3925209376 2024-04-01T23:02:08.493619+0000 2024-03-31T23:09:24.952 INFO:teuthology.orchestra.run.smithi046.stderr:listed 10 entries 2024-03-31T23:09:24.953 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:24.952+0000 7ff48cb25640 1 -- 172.21.15.46:0/2402447570 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7ff4700a2520 msgr2=0x7ff4700a49e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:09:24.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:24.952+0000 7ff48cb25640 1 --2- 172.21.15.46:0/2402447570 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7ff4700a2520 0x7ff4700a49e0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7ff488100120 tx=0x7ff47c005f90 comp rx=0 tx=0).stop 2024-03-31T23:09:24.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:24.952+0000 7ff48cb25640 1 -- 172.21.15.46:0/2402447570 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff488100930 msgr2=0x7ff4881003b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:09:24.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:24.952+0000 7ff48cb25640 1 --2- 172.21.15.46:0/2402447570 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff488100930 0x7ff4881003b0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7ff480000c00 tx=0x7ff480004270 comp rx=0 tx=0).stop 2024-03-31T23:09:24.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:24.952+0000 7ff48cb25640 1 -- 172.21.15.46:0/2402447570 shutdown_connections 2024-03-31T23:09:24.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:24.952+0000 7ff48cb25640 1 --2- 172.21.15.46:0/2402447570 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7ff4700a2520 0x7ff4700a49e0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:24.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:24.952+0000 7ff48cb25640 1 --2- 172.21.15.46:0/2402447570 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff488101c00 0x7ff48819ed80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:24.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:24.952+0000 7ff48cb25640 1 --2- 172.21.15.46:0/2402447570 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff4881012f0 0x7ff48819c740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:24.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:24.952+0000 7ff48cb25640 1 --2- 172.21.15.46:0/2402447570 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff488100930 0x7ff4881003b0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:24.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:24.952+0000 7ff48cb25640 1 -- 172.21.15.46:0/2402447570 >> 172.21.15.46:0/2402447570 conn(0x7ff4880fb340 msgr2=0x7ff488104d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:09:24.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:24.952+0000 7ff48cb25640 1 -- 172.21.15.46:0/2402447570 shutdown_connections 2024-03-31T23:09:24.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:24.952+0000 7ff48cb25640 1 -- 172.21.15.46:0/2402447570 wait complete. 2024-03-31T23:09:24.964 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-03-31T23:09:24.964 DEBUG:teuthology.orchestra.run.smithi046:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-31T23:09:24.979 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-31T23:09:25.125 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:25.121+0000 7f518e9ae640 1 -- 172.21.15.46:0/2640505015 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f51881033f0 msgr2=0x7f5188109c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:09:25.125 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:25.121+0000 7f518e9ae640 1 --2- 172.21.15.46:0/2640505015 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f51881033f0 0x7f5188109c80 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f5180009cd0 tx=0x7f518002c0f0 comp rx=0 tx=0).stop 2024-03-31T23:09:25.125 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:25.121+0000 7f518e9ae640 1 -- 172.21.15.46:0/2640505015 shutdown_connections 2024-03-31T23:09:25.125 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:25.121+0000 7f518e9ae640 1 --2- 172.21.15.46:0/2640505015 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f51881033f0 0x7f5188109c80 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:25.125 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:25.121+0000 7f518e9ae640 1 --2- 172.21.15.46:0/2640505015 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5188102a30 0x7f5188102eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:25.125 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:25.121+0000 7f518e9ae640 1 --2- 172.21.15.46:0/2640505015 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f51880ffa60 0x7f51880ffe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:25.125 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:25.121+0000 7f518e9ae640 1 -- 172.21.15.46:0/2640505015 >> 172.21.15.46:0/2640505015 conn(0x7f51880fb390 msgr2=0x7f51880fd7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:09:25.125 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:25.121+0000 7f518e9ae640 1 -- 172.21.15.46:0/2640505015 shutdown_connections 2024-03-31T23:09:25.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:25.121+0000 7f518e9ae640 1 -- 172.21.15.46:0/2640505015 wait complete. 2024-03-31T23:09:25.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:25.122+0000 7f518e9ae640 1 Processor -- start 2024-03-31T23:09:25.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:25.122+0000 7f518e9ae640 1 -- start start 2024-03-31T23:09:25.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:25.122+0000 7f518e9ae640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f51880ffa60 0x7f5188199ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:09:25.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:25.122+0000 7f518e9ae640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5188102a30 0x7f518819a0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:09:25.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:25.122+0000 7f518e9ae640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f51881033f0 0x7f51881a1150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:09:25.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:25.122+0000 7f518e9ae640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f518810d2e0 con 0x7f51881033f0 2024-03-31T23:09:25.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:25.122+0000 7f518e9ae640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f518810d160 con 0x7f51880ffa60 2024-03-31T23:09:25.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:25.122+0000 7f518e9ae640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f518810d460 con 0x7f5188102a30 2024-03-31T23:09:25.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:25.122+0000 7f518cf4b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f51881033f0 0x7f51881a1150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:09:25.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:25.122+0000 7f518cf4b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f51881033f0 0x7f51881a1150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44808/0 (socket says 172.21.15.46:44808) 2024-03-31T23:09:25.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:25.122+0000 7f518cf4b640 1 -- 172.21.15.46:0/606870757 learned_addr learned my addr 172.21.15.46:0/606870757 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:09:25.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:25.122+0000 7f5187fff640 1 --2- 172.21.15.46:0/606870757 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f51880ffa60 0x7f5188199ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:09:25.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:25.123+0000 7f518cf4b640 1 -- 172.21.15.46:0/606870757 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5188102a30 msgr2=0x7f518819a0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:09:25.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:25.123+0000 7f518cf4b640 1 --2- 172.21.15.46:0/606870757 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5188102a30 0x7f518819a0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:25.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:25.123+0000 7f518cf4b640 1 -- 172.21.15.46:0/606870757 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f51880ffa60 msgr2=0x7f5188199ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:09:25.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:25.123+0000 7f518cf4b640 1 --2- 172.21.15.46:0/606870757 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f51880ffa60 0x7f5188199ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:25.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:25.123+0000 7f518cf4b640 1 -- 172.21.15.46:0/606870757 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f51881a17c0 con 0x7f51881033f0 2024-03-31T23:09:25.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:25.123+0000 7f5187fff640 1 --2- 172.21.15.46:0/606870757 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f51880ffa60 0x7f5188199ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T23:09:25.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:25.123+0000 7f518cf4b640 1 --2- 172.21.15.46:0/606870757 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f51881033f0 0x7f51881a1150 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f518002c630 tx=0x7f51800328a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:09:25.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:25.123+0000 7f51857fa640 1 -- 172.21.15.46:0/606870757 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5180041020 con 0x7f51881033f0 2024-03-31T23:09:25.128 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:25.123+0000 7f518e9ae640 1 -- 172.21.15.46:0/606870757 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f51881a1a50 con 0x7f51881033f0 2024-03-31T23:09:25.128 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:25.123+0000 7f51857fa640 1 -- 172.21.15.46:0/606870757 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5180003e20 con 0x7f51881033f0 2024-03-31T23:09:25.128 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:25.123+0000 7f51857fa640 1 -- 172.21.15.46:0/606870757 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5180039040 con 0x7f51881033f0 2024-03-31T23:09:25.128 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:25.123+0000 7f518e9ae640 1 -- 172.21.15.46:0/606870757 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f51881a1f40 con 0x7f51881033f0 2024-03-31T23:09:25.128 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:25.124+0000 7f518e9ae640 1 -- 172.21.15.46:0/606870757 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f51881a2170 con 0x7f51881033f0 2024-03-31T23:09:25.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:25.125+0000 7f51857fa640 1 -- 172.21.15.46:0/606870757 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 16) ==== 137320+0+0 (secure 0 0 0) 0x7f5180044020 con 0x7f51881033f0 2024-03-31T23:09:25.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:25.126+0000 7f51857fa640 1 --2- 172.21.15.46:0/606870757 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f51740a24a0 0x7f51740a4960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:09:25.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:25.126+0000 7f5187fff640 1 --2- 172.21.15.46:0/606870757 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f51740a24a0 0x7f51740a4960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:09:25.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:25.126+0000 7f51857fa640 1 -- 172.21.15.46:0/606870757 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8802+0+0 (secure 0 0 0) 0x7f51800f1000 con 0x7f51881033f0 2024-03-31T23:09:25.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:25.126+0000 7f5187fff640 1 --2- 172.21.15.46:0/606870757 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f51740a24a0 0x7f51740a4960 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f51780058c0 tx=0x7f5178005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:09:25.130 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:25.129+0000 7f51857fa640 1 -- 172.21.15.46:0/606870757 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f518004b020 con 0x7f51881033f0 2024-03-31T23:09:25.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:25.312+0000 7f518e9ae640 1 -- 172.21.15.46:0/606870757 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f51881a2460 con 0x7f51881033f0 2024-03-31T23:09:25.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:25.314+0000 7f51857fa640 1 -- 172.21.15.46:0/606870757 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) ==== 74+0+24105 (secure 0 0 0) 0x7f518004e350 con 0x7f51881033f0 2024-03-31T23:09:25.315 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-31T23:09:25.315 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":91,"fsid":"6a9d8f08-efb2-11ee-b647-cb9ed24678a4","created":"2024-03-31T23:01:15.463711+0000","modified":"2024-03-31T23:09:17.181787+0000","last_up_change":"2024-03-31T23:08:20.199888+0000","last_in_change":"2024-03-31T23:07:59.438035+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-31T23:04:32.823751+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-31T23:09:09.145711+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-31T23:09:10.148579+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-31T23:09:13.419519+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"91","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"47429095-3f46-4ed7-b7c4-504464719815","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":1824772695},{"type":"v1","addr":"172.21.15.46:6803","nonce":1824772695}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":1824772695},{"type":"v1","addr":"172.21.15.46:6805","nonce":1824772695}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":1824772695},{"type":"v1","addr":"172.21.15.46:6809","nonce":1824772695}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":1824772695},{"type":"v1","addr":"172.21.15.46:6807","nonce":1824772695}]},"public_addr":"172.21.15.46:6803/1824772695","cluster_addr":"172.21.15.46:6805/1824772695","heartbeat_back_addr":"172.21.15.46:6809/1824772695","heartbeat_front_addr":"172.21.15.46:6807/1824772695","state":["exists","up"]},{"osd":1,"uuid":"be56f883-411e-4dbe-b50e-a158134df4a6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6800","nonce":973693922},{"type":"v1","addr":"172.21.15.142:6801","nonce":973693922}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6802","nonce":973693922},{"type":"v1","addr":"172.21.15.142:6803","nonce":973693922}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6806","nonce":973693922},{"type":"v1","addr":"172.21.15.142:6807","nonce":973693922}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6804","nonce":973693922},{"type":"v1","addr":"172.21.15.142:6805","nonce":973693922}]},"public_addr":"172.21.15.142:6801/973693922","cluster_addr":"172.21.15.142:6803/973693922","heartbeat_back_addr":"172.21.15.142:6807/973693922","heartbeat_front_addr":"172.21.15.142:6805/973693922","state":["exists","up"]},{"osd":2,"uuid":"c99a8794-ab19-4260-9a8d-4bc0bd2c8175","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6800","nonce":4084261813},{"type":"v1","addr":"172.21.15.162:6801","nonce":4084261813}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6802","nonce":4084261813},{"type":"v1","addr":"172.21.15.162:6803","nonce":4084261813}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6806","nonce":4084261813},{"type":"v1","addr":"172.21.15.162:6807","nonce":4084261813}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6804","nonce":4084261813},{"type":"v1","addr":"172.21.15.162:6805","nonce":4084261813}]},"public_addr":"172.21.15.162:6801/4084261813","cluster_addr":"172.21.15.162:6803/4084261813","heartbeat_back_addr":"172.21.15.162:6807/4084261813","heartbeat_front_addr":"172.21.15.162:6805/4084261813","state":["exists","up"]},{"osd":3,"uuid":"f877814e-9ef6-4872-a081-5609bb9ebc06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":3496317970},{"type":"v1","addr":"172.21.15.46:6811","nonce":3496317970}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":3496317970},{"type":"v1","addr":"172.21.15.46:6813","nonce":3496317970}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":3496317970},{"type":"v1","addr":"172.21.15.46:6817","nonce":3496317970}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":3496317970},{"type":"v1","addr":"172.21.15.46:6815","nonce":3496317970}]},"public_addr":"172.21.15.46:6811/3496317970","cluster_addr":"172.21.15.46:6813/3496317970","heartbeat_back_addr":"172.21.15.46:6817/3496317970","heartbeat_front_addr":"172.21.15.46:6815/3496317970","state":["exists","up"]},{"osd":4,"uuid":"beecec0a-8280-44fe-8f5f-8498a35eca8f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6808","nonce":2996266929},{"type":"v1","addr":"172.21.15.142:6809","nonce":2996266929}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6810","nonce":2996266929},{"type":"v1","addr":"172.21.15.142:6811","nonce":2996266929}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6814","nonce":2996266929},{"type":"v1","addr":"172.21.15.142:6815","nonce":2996266929}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6812","nonce":2996266929},{"type":"v1","addr":"172.21.15.142:6813","nonce":2996266929}]},"public_addr":"172.21.15.142:6809/2996266929","cluster_addr":"172.21.15.142:6811/2996266929","heartbeat_back_addr":"172.21.15.142:6815/2996266929","heartbeat_front_addr":"172.21.15.142:6813/2996266929","state":["exists","up"]},{"osd":5,"uuid":"9da7ff75-f658-45ea-a02f-1943fdda76b9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6808","nonce":914208293},{"type":"v1","addr":"172.21.15.162:6809","nonce":914208293}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6810","nonce":914208293},{"type":"v1","addr":"172.21.15.162:6811","nonce":914208293}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6814","nonce":914208293},{"type":"v1","addr":"172.21.15.162:6815","nonce":914208293}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6812","nonce":914208293},{"type":"v1","addr":"172.21.15.162:6813","nonce":914208293}]},"public_addr":"172.21.15.162:6809/914208293","cluster_addr":"172.21.15.162:6811/914208293","heartbeat_back_addr":"172.21.15.162:6815/914208293","heartbeat_front_addr":"172.21.15.162:6813/914208293","state":["exists","up"]},{"osd":6,"uuid":"8fa11224-cb26-4d3f-87d1-6a8f31de2a19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":56291966},{"type":"v1","addr":"172.21.15.46:6819","nonce":56291966}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":56291966},{"type":"v1","addr":"172.21.15.46:6821","nonce":56291966}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6824","nonce":56291966},{"type":"v1","addr":"172.21.15.46:6825","nonce":56291966}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":56291966},{"type":"v1","addr":"172.21.15.46:6823","nonce":56291966}]},"public_addr":"172.21.15.46:6819/56291966","cluster_addr":"172.21.15.46:6821/56291966","heartbeat_back_addr":"172.21.15.46:6825/56291966","heartbeat_front_addr":"172.21.15.46:6823/56291966","state":["exists","up"]},{"osd":7,"uuid":"0d0338b7-4d0a-4bb0-b152-8f5224871d9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6816","nonce":2146598446},{"type":"v1","addr":"172.21.15.142:6817","nonce":2146598446}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6818","nonce":2146598446},{"type":"v1","addr":"172.21.15.142:6819","nonce":2146598446}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6822","nonce":2146598446},{"type":"v1","addr":"172.21.15.142:6823","nonce":2146598446}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6820","nonce":2146598446},{"type":"v1","addr":"172.21.15.142:6821","nonce":2146598446}]},"public_addr":"172.21.15.142:6817/2146598446","cluster_addr":"172.21.15.142:6819/2146598446","heartbeat_back_addr":"172.21.15.142:6823/2146598446","heartbeat_front_addr":"172.21.15.142:6821/2146598446","state":["exists","up"]},{"osd":8,"uuid":"8b6f6037-892a-416c-babb-7552648d0b3a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6816","nonce":1173078318},{"type":"v1","addr":"172.21.15.162:6817","nonce":1173078318}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6818","nonce":1173078318},{"type":"v1","addr":"172.21.15.162:6819","nonce":1173078318}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6822","nonce":1173078318},{"type":"v1","addr":"172.21.15.162:6823","nonce":1173078318}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6820","nonce":1173078318},{"type":"v1","addr":"172.21.15.162:6821","nonce":1173078318}]},"public_addr":"172.21.15.162:6817/1173078318","cluster_addr":"172.21.15.162:6819/1173078318","heartbeat_back_addr":"172.21.15.162:6823/1173078318","heartbeat_front_addr":"172.21.15.162:6821/1173078318","state":["exists","up"]},{"osd":9,"uuid":"faccce25-6bf2-4220-8858-cda82b72be68","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":532708881},{"type":"v1","addr":"172.21.15.46:6827","nonce":532708881}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6828","nonce":532708881},{"type":"v1","addr":"172.21.15.46:6829","nonce":532708881}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":532708881},{"type":"v1","addr":"172.21.15.46:6833","nonce":532708881}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":532708881},{"type":"v1","addr":"172.21.15.46:6831","nonce":532708881}]},"public_addr":"172.21.15.46:6827/532708881","cluster_addr":"172.21.15.46:6829/532708881","heartbeat_back_addr":"172.21.15.46:6833/532708881","heartbeat_front_addr":"172.21.15.46:6831/532708881","state":["exists","up"]},{"osd":10,"uuid":"cdd356db-f9eb-4080-95ab-db5d8223e572","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6824","nonce":2845222366},{"type":"v1","addr":"172.21.15.142:6825","nonce":2845222366}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6826","nonce":2845222366},{"type":"v1","addr":"172.21.15.142:6827","nonce":2845222366}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6830","nonce":2845222366},{"type":"v1","addr":"172.21.15.142:6831","nonce":2845222366}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6828","nonce":2845222366},{"type":"v1","addr":"172.21.15.142:6829","nonce":2845222366}]},"public_addr":"172.21.15.142:6825/2845222366","cluster_addr":"172.21.15.142:6827/2845222366","heartbeat_back_addr":"172.21.15.142:6831/2845222366","heartbeat_front_addr":"172.21.15.142:6829/2845222366","state":["exists","up"]},{"osd":11,"uuid":"72104d1c-3ea2-45ad-bc0b-0500f146c06b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6824","nonce":1986323778},{"type":"v1","addr":"172.21.15.162:6825","nonce":1986323778}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6826","nonce":1986323778},{"type":"v1","addr":"172.21.15.162:6827","nonce":1986323778}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6830","nonce":1986323778},{"type":"v1","addr":"172.21.15.162:6831","nonce":1986323778}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6828","nonce":1986323778},{"type":"v1","addr":"172.21.15.162:6829","nonce":1986323778}]},"public_addr":"172.21.15.162:6825/1986323778","cluster_addr":"172.21.15.162:6827/1986323778","heartbeat_back_addr":"172.21.15.162:6831/1986323778","heartbeat_front_addr":"172.21.15.162:6829/1986323778","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:03:44.996331+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:04:07.994738+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:04:30.942255+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:04:55.588884+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:05:19.900600+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:05:43.807442+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:06:09.026976+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:06:34.586784+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:06:59.751605+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:07:26.144495+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:07:52.326909+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:08:17.423042+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.46:0/4021039857":"2024-04-01T23:02:08.493619+0000","172.21.15.46:6801/3681182647":"2024-04-01T23:02:08.493619+0000","172.21.15.46:0/4022214806":"2024-04-01T23:02:08.493619+0000","172.21.15.46:6800/1515528215":"2024-04-01T23:01:41.910162+0000","172.21.15.46:6800/3681182647":"2024-04-01T23:02:08.493619+0000","172.21.15.46:0/143885707":"2024-04-01T23:01:41.910162+0000","172.21.15.46:6801/1515528215":"2024-04-01T23:01:41.910162+0000","172.21.15.46:0/1680389778":"2024-04-01T23:01:41.910162+0000","172.21.15.46:0/1782133966":"2024-04-01T23:01:41.910162+0000","172.21.15.46:0/3925209376":"2024-04-01T23:02:08.493619+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-31T23:09:25.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:25.316+0000 7f518e9ae640 1 -- 172.21.15.46:0/606870757 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f51740a24a0 msgr2=0x7f51740a4960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:09:25.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:25.316+0000 7f518e9ae640 1 --2- 172.21.15.46:0/606870757 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f51740a24a0 0x7f51740a4960 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f51780058c0 tx=0x7f5178005850 comp rx=0 tx=0).stop 2024-03-31T23:09:25.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:25.316+0000 7f518e9ae640 1 -- 172.21.15.46:0/606870757 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f51881033f0 msgr2=0x7f51881a1150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:09:25.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:25.316+0000 7f518e9ae640 1 --2- 172.21.15.46:0/606870757 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f51881033f0 0x7f51881a1150 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f518002c630 tx=0x7f51800328a0 comp rx=0 tx=0).stop 2024-03-31T23:09:25.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:25.316+0000 7f518e9ae640 1 -- 172.21.15.46:0/606870757 shutdown_connections 2024-03-31T23:09:25.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:25.316+0000 7f518e9ae640 1 --2- 172.21.15.46:0/606870757 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f51740a24a0 0x7f51740a4960 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:25.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:25.316+0000 7f518e9ae640 1 --2- 172.21.15.46:0/606870757 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f51881033f0 0x7f51881a1150 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:25.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:25.316+0000 7f518e9ae640 1 --2- 172.21.15.46:0/606870757 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5188102a30 0x7f518819a0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:25.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:25.316+0000 7f518e9ae640 1 --2- 172.21.15.46:0/606870757 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f51880ffa60 0x7f5188199ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:25.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:25.316+0000 7f518e9ae640 1 -- 172.21.15.46:0/606870757 >> 172.21.15.46:0/606870757 conn(0x7f51880fb390 msgr2=0x7f5188100820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:09:25.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:25.316+0000 7f518e9ae640 1 -- 172.21.15.46:0/606870757 shutdown_connections 2024-03-31T23:09:25.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:25.316+0000 7f518e9ae640 1 -- 172.21.15.46:0/606870757 wait complete. 2024-03-31T23:09:25.328 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-31T23:04:32.823751+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-31T23:09:09.145711+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-31T23:09:10.148579+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-31T23:09:13.419519+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '91', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-31T23:09:25.329 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-31T23:09:25.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:25.434+0000 7f957113e640 1 -- 172.21.15.46:0/2951699808 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f956c102200 msgr2=0x7f956c108a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:09:25.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:25.434+0000 7f957113e640 1 --2- 172.21.15.46:0/2951699808 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f956c102200 0x7f956c108a90 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f9564009ca0 tx=0x7f956402c0f0 comp rx=0 tx=0).stop 2024-03-31T23:09:25.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:25.434+0000 7f957113e640 1 -- 172.21.15.46:0/2951699808 shutdown_connections 2024-03-31T23:09:25.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:25.434+0000 7f957113e640 1 --2- 172.21.15.46:0/2951699808 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f956c102200 0x7f956c108a90 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:25.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:25.434+0000 7f957113e640 1 --2- 172.21.15.46:0/2951699808 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f956c101840 0x7f956c101cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:25.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:25.434+0000 7f957113e640 1 --2- 172.21.15.46:0/2951699808 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f956c100650 0x7f956c100a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:25.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:25.434+0000 7f957113e640 1 -- 172.21.15.46:0/2951699808 >> 172.21.15.46:0/2951699808 conn(0x7f956c0fbbe0 msgr2=0x7f956c0fe000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:09:25.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:25.434+0000 7f957113e640 1 -- 172.21.15.46:0/2951699808 shutdown_connections 2024-03-31T23:09:25.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:25.434+0000 7f957113e640 1 -- 172.21.15.46:0/2951699808 wait complete. 2024-03-31T23:09:25.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:25.435+0000 7f957113e640 1 Processor -- start 2024-03-31T23:09:25.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:25.435+0000 7f957113e640 1 -- start start 2024-03-31T23:09:25.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:25.435+0000 7f957113e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f956c100650 0x7f956c1978c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:09:25.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:25.435+0000 7f957113e640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f956c101840 0x7f956c197e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:09:25.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:25.435+0000 7f957113e640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f956c102200 0x7f956c19ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:09:25.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:25.435+0000 7f957113e640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f956c10c160 con 0x7f956c100650 2024-03-31T23:09:25.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:25.435+0000 7f957113e640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f956c10bfe0 con 0x7f956c102200 2024-03-31T23:09:25.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:25.435+0000 7f957113e640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f956c10c2e0 con 0x7f956c101840 2024-03-31T23:09:25.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:25.435+0000 7f956ad9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f956c100650 0x7f956c1978c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:09:25.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:25.435+0000 7f956ad9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f956c100650 0x7f956c1978c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44824/0 (socket says 172.21.15.46:44824) 2024-03-31T23:09:25.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:25.435+0000 7f956b59e640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f956c102200 0x7f956c19ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:09:25.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:25.435+0000 7f956ad9d640 1 -- 172.21.15.46:0/2962655238 learned_addr learned my addr 172.21.15.46:0/2962655238 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:09:25.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:25.435+0000 7f956259c640 1 --2- 172.21.15.46:0/2962655238 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f956c101840 0x7f956c197e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:09:25.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:25.435+0000 7f956ad9d640 1 -- 172.21.15.46:0/2962655238 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f956c101840 msgr2=0x7f956c197e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:09:25.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:25.435+0000 7f956ad9d640 1 --2- 172.21.15.46:0/2962655238 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f956c101840 0x7f956c197e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:25.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:25.436+0000 7f956ad9d640 1 -- 172.21.15.46:0/2962655238 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f956c102200 msgr2=0x7f956c19ee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:09:25.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:25.436+0000 7f956ad9d640 1 --2- 172.21.15.46:0/2962655238 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f956c102200 0x7f956c19ee70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:25.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:25.436+0000 7f956ad9d640 1 -- 172.21.15.46:0/2962655238 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f956c19f4e0 con 0x7f956c100650 2024-03-31T23:09:25.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:25.436+0000 7f956b59e640 1 --2- 172.21.15.46:0/2962655238 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f956c102200 0x7f956c19ee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:09:25.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:25.436+0000 7f956ad9d640 1 --2- 172.21.15.46:0/2962655238 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f956c100650 0x7f956c1978c0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f955c00c970 tx=0x7f955c00ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:09:25.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:25.436+0000 7f9568d99640 1 -- 172.21.15.46:0/2962655238 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f955c007c10 con 0x7f956c100650 2024-03-31T23:09:25.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:25.436+0000 7f9568d99640 1 -- 172.21.15.46:0/2962655238 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f955c007db0 con 0x7f956c100650 2024-03-31T23:09:25.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:25.436+0000 7f957113e640 1 -- 172.21.15.46:0/2962655238 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f956c19f7d0 con 0x7f956c100650 2024-03-31T23:09:25.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:25.436+0000 7f9568d99640 1 -- 172.21.15.46:0/2962655238 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f955c00f870 con 0x7f956c100650 2024-03-31T23:09:25.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:25.436+0000 7f957113e640 1 -- 172.21.15.46:0/2962655238 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f956c19fd20 con 0x7f956c100650 2024-03-31T23:09:25.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:25.437+0000 7f957113e640 1 -- 172.21.15.46:0/2962655238 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f956c19ffe0 con 0x7f956c100650 2024-03-31T23:09:25.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:25.438+0000 7f9568d99640 1 -- 172.21.15.46:0/2962655238 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 16) ==== 137320+0+0 (secure 0 0 0) 0x7f955c00fa10 con 0x7f956c100650 2024-03-31T23:09:25.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:25.438+0000 7f9568d99640 1 --2- 172.21.15.46:0/2962655238 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f95580a24a0 0x7f95580a4960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:09:25.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:25.439+0000 7f956259c640 1 --2- 172.21.15.46:0/2962655238 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f95580a24a0 0x7f95580a4960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:09:25.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:25.439+0000 7f9568d99640 1 -- 172.21.15.46:0/2962655238 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8802+0+0 (secure 0 0 0) 0x7f955c0cec20 con 0x7f956c100650 2024-03-31T23:09:25.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:25.439+0000 7f956259c640 1 --2- 172.21.15.46:0/2962655238 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f95580a24a0 0x7f95580a4960 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f956c198dd0 tx=0x7f9554005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:09:25.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:25.443+0000 7f9568d99640 1 -- 172.21.15.46:0/2962655238 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f955c09bc30 con 0x7f956c100650 2024-03-31T23:09:25.611 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:09:25 smithi046 ceph-mon[32498]: from='client.? 172.21.15.46:0/2266026602' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-31T23:09:25.611 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:09:25 smithi046 ceph-mon[32498]: from='client.? 172.21.15.46:0/2402447570' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-31T23:09:25.612 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:09:25 smithi046 ceph-mon[32498]: from='client.? 172.21.15.46:0/606870757' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-31T23:09:25.626 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:25.624+0000 7f957113e640 1 -- 172.21.15.46:0/2962655238 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f956c1a01b0 con 0x7f956c100650 2024-03-31T23:09:25.626 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:25.625+0000 7f9568d99640 1 -- 172.21.15.46:0/2962655238 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v91) ==== 93+0+10 (secure 0 0 0) 0x7f955c09b530 con 0x7f956c100650 2024-03-31T23:09:25.626 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 1 2024-03-31T23:09:25.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:25.627+0000 7f957113e640 1 -- 172.21.15.46:0/2962655238 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f95580a24a0 msgr2=0x7f95580a4960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:09:25.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:25.627+0000 7f957113e640 1 --2- 172.21.15.46:0/2962655238 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f95580a24a0 0x7f95580a4960 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f956c198dd0 tx=0x7f9554005f90 comp rx=0 tx=0).stop 2024-03-31T23:09:25.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:25.627+0000 7f957113e640 1 -- 172.21.15.46:0/2962655238 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f956c100650 msgr2=0x7f956c1978c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:09:25.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:25.627+0000 7f957113e640 1 --2- 172.21.15.46:0/2962655238 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f956c100650 0x7f956c1978c0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f955c00c970 tx=0x7f955c00ce30 comp rx=0 tx=0).stop 2024-03-31T23:09:25.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:25.627+0000 7f957113e640 1 -- 172.21.15.46:0/2962655238 shutdown_connections 2024-03-31T23:09:25.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:25.627+0000 7f957113e640 1 --2- 172.21.15.46:0/2962655238 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f95580a24a0 0x7f95580a4960 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:25.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:25.627+0000 7f957113e640 1 --2- 172.21.15.46:0/2962655238 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f956c102200 0x7f956c19ee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:25.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:25.627+0000 7f957113e640 1 --2- 172.21.15.46:0/2962655238 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f956c101840 0x7f956c197e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:25.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:25.627+0000 7f957113e640 1 --2- 172.21.15.46:0/2962655238 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f956c100650 0x7f956c1978c0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:25.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:25.627+0000 7f957113e640 1 -- 172.21.15.46:0/2962655238 >> 172.21.15.46:0/2962655238 conn(0x7f956c0fbbe0 msgr2=0x7f956c0fd5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:09:25.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:25.627+0000 7f957113e640 1 -- 172.21.15.46:0/2962655238 shutdown_connections 2024-03-31T23:09:25.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:25.627+0000 7f957113e640 1 -- 172.21.15.46:0/2962655238 wait complete. 2024-03-31T23:09:25.639 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-31T23:09:25.693 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:09:25 smithi142 ceph-mon[35271]: from='client.? 172.21.15.46:0/2266026602' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-31T23:09:25.693 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:09:25 smithi142 ceph-mon[35271]: from='client.? 172.21.15.46:0/2402447570' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-31T23:09:25.693 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:09:25 smithi142 ceph-mon[35271]: from='client.? 172.21.15.46:0/606870757' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-31T23:09:25.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:25.743+0000 7fcb61888640 1 -- 172.21.15.46:0/1207075002 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcb5c102a10 msgr2=0x7fcb5c102e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:09:25.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:25.743+0000 7fcb61888640 1 --2- 172.21.15.46:0/1207075002 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcb5c102a10 0x7fcb5c102e90 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fcb44009860 tx=0x7fcb4402bb70 comp rx=0 tx=0).stop 2024-03-31T23:09:25.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:25.743+0000 7fcb61888640 1 -- 172.21.15.46:0/1207075002 shutdown_connections 2024-03-31T23:09:25.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:25.743+0000 7fcb61888640 1 --2- 172.21.15.46:0/1207075002 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcb5c1033d0 0x7fcb5c109c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:25.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:25.743+0000 7fcb61888640 1 --2- 172.21.15.46:0/1207075002 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcb5c102a10 0x7fcb5c102e90 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:25.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:25.743+0000 7fcb61888640 1 --2- 172.21.15.46:0/1207075002 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcb5c0ffa40 0x7fcb5c0ffe60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:25.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:25.743+0000 7fcb61888640 1 -- 172.21.15.46:0/1207075002 >> 172.21.15.46:0/1207075002 conn(0x7fcb5c0fb350 msgr2=0x7fcb5c0fd7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:09:25.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:25.743+0000 7fcb61888640 1 -- 172.21.15.46:0/1207075002 shutdown_connections 2024-03-31T23:09:25.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:25.743+0000 7fcb61888640 1 -- 172.21.15.46:0/1207075002 wait complete. 2024-03-31T23:09:25.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:25.743+0000 7fcb61888640 1 Processor -- start 2024-03-31T23:09:25.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:25.744+0000 7fcb61888640 1 -- start start 2024-03-31T23:09:25.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:25.744+0000 7fcb61888640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcb5c0ffa40 0x7fcb5c1979d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:09:25.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:25.744+0000 7fcb61888640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcb5c102a10 0x7fcb5c197f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:09:25.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:25.744+0000 7fcb61888640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcb5c1033d0 0x7fcb5c19ef80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:09:25.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:25.744+0000 7fcb61888640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fcb5c10d2b0 con 0x7fcb5c102a10 2024-03-31T23:09:25.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:25.744+0000 7fcb61888640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fcb5c10d130 con 0x7fcb5c0ffa40 2024-03-31T23:09:25.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:25.744+0000 7fcb61888640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fcb5c10d430 con 0x7fcb5c1033d0 2024-03-31T23:09:25.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:25.744+0000 7fcb5a7fc640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcb5c102a10 0x7fcb5c197f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:09:25.748 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:25.744+0000 7fcb5a7fc640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcb5c102a10 0x7fcb5c197f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44852/0 (socket says 172.21.15.46:44852) 2024-03-31T23:09:25.748 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:25.744+0000 7fcb5a7fc640 1 -- 172.21.15.46:0/2670883742 learned_addr learned my addr 172.21.15.46:0/2670883742 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:09:25.748 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:25.744+0000 7fcb5affd640 1 --2- 172.21.15.46:0/2670883742 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcb5c0ffa40 0x7fcb5c1979d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:09:25.748 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:25.744+0000 7fcb5b7fe640 1 --2- 172.21.15.46:0/2670883742 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcb5c1033d0 0x7fcb5c19ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:09:25.748 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:25.744+0000 7fcb5a7fc640 1 -- 172.21.15.46:0/2670883742 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcb5c1033d0 msgr2=0x7fcb5c19ef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:09:25.748 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:25.744+0000 7fcb5a7fc640 1 --2- 172.21.15.46:0/2670883742 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcb5c1033d0 0x7fcb5c19ef80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:25.748 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:25.744+0000 7fcb5a7fc640 1 -- 172.21.15.46:0/2670883742 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcb5c0ffa40 msgr2=0x7fcb5c1979d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:09:25.748 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:25.744+0000 7fcb5a7fc640 1 --2- 172.21.15.46:0/2670883742 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcb5c0ffa40 0x7fcb5c1979d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:25.748 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:25.744+0000 7fcb5a7fc640 1 -- 172.21.15.46:0/2670883742 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcb5c19f5f0 con 0x7fcb5c102a10 2024-03-31T23:09:25.748 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:25.744+0000 7fcb5b7fe640 1 --2- 172.21.15.46:0/2670883742 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcb5c1033d0 0x7fcb5c19ef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T23:09:25.748 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:25.744+0000 7fcb5a7fc640 1 --2- 172.21.15.46:0/2670883742 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcb5c102a10 0x7fcb5c197f10 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fcb44005940 tx=0x7fcb440042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:09:25.748 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:25.744+0000 7fcb60886640 1 -- 172.21.15.46:0/2670883742 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb44041020 con 0x7fcb5c102a10 2024-03-31T23:09:25.748 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:25.744+0000 7fcb61888640 1 -- 172.21.15.46:0/2670883742 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcb5c19f880 con 0x7fcb5c102a10 2024-03-31T23:09:25.749 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:25.744+0000 7fcb60886640 1 -- 172.21.15.46:0/2670883742 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fcb440035f0 con 0x7fcb5c102a10 2024-03-31T23:09:25.749 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:25.744+0000 7fcb60886640 1 -- 172.21.15.46:0/2670883742 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb44039040 con 0x7fcb5c102a10 2024-03-31T23:09:25.749 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:25.745+0000 7fcb61888640 1 -- 172.21.15.46:0/2670883742 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcb5c19fd70 con 0x7fcb5c102a10 2024-03-31T23:09:25.750 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:25.746+0000 7fcb61888640 1 -- 172.21.15.46:0/2670883742 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcb5c108c10 con 0x7fcb5c102a10 2024-03-31T23:09:25.750 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:25.746+0000 7fcb60886640 1 -- 172.21.15.46:0/2670883742 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 16) ==== 137320+0+0 (secure 0 0 0) 0x7fcb44044020 con 0x7fcb5c102a10 2024-03-31T23:09:25.750 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:25.747+0000 7fcb60886640 1 --2- 172.21.15.46:0/2670883742 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fcb480a24a0 0x7fcb480a4960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:09:25.750 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:25.747+0000 7fcb5affd640 1 --2- 172.21.15.46:0/2670883742 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fcb480a24a0 0x7fcb480a4960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:09:25.750 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:25.747+0000 7fcb60886640 1 -- 172.21.15.46:0/2670883742 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8802+0+0 (secure 0 0 0) 0x7fcb440f0460 con 0x7fcb5c102a10 2024-03-31T23:09:25.750 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:25.748+0000 7fcb5affd640 1 --2- 172.21.15.46:0/2670883742 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fcb480a24a0 0x7fcb480a4960 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fcb5c069b60 tx=0x7fcb5400a2d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:09:25.753 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:25.752+0000 7fcb60886640 1 -- 172.21.15.46:0/2670883742 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fcb4404b020 con 0x7fcb5c102a10 2024-03-31T23:09:25.795 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:09:25 smithi162 ceph-mon[36237]: from='client.? 172.21.15.46:0/2266026602' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-31T23:09:25.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:09:25 smithi162 ceph-mon[36237]: from='client.? 172.21.15.46:0/2402447570' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-31T23:09:25.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:09:25 smithi162 ceph-mon[36237]: from='client.? 172.21.15.46:0/606870757' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-31T23:09:25.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:25.937+0000 7fcb61888640 1 -- 172.21.15.46:0/2670883742 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fcb5c062350 con 0x7fcb5c102a10 2024-03-31T23:09:25.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:25.937+0000 7fcb60886640 1 -- 172.21.15.46:0/2670883742 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v91) ==== 104+0+11 (secure 0 0 0) 0x7fcb440bd3e0 con 0x7fcb5c102a10 2024-03-31T23:09:25.939 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 32 2024-03-31T23:09:25.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:25.940+0000 7fcb61888640 1 -- 172.21.15.46:0/2670883742 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fcb480a24a0 msgr2=0x7fcb480a4960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:09:25.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:25.940+0000 7fcb61888640 1 --2- 172.21.15.46:0/2670883742 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fcb480a24a0 0x7fcb480a4960 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fcb5c069b60 tx=0x7fcb5400a2d0 comp rx=0 tx=0).stop 2024-03-31T23:09:25.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:25.940+0000 7fcb61888640 1 -- 172.21.15.46:0/2670883742 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcb5c102a10 msgr2=0x7fcb5c197f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:09:25.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:25.940+0000 7fcb61888640 1 --2- 172.21.15.46:0/2670883742 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcb5c102a10 0x7fcb5c197f10 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fcb44005940 tx=0x7fcb440042f0 comp rx=0 tx=0).stop 2024-03-31T23:09:25.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:25.940+0000 7fcb61888640 1 -- 172.21.15.46:0/2670883742 shutdown_connections 2024-03-31T23:09:25.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:25.940+0000 7fcb61888640 1 --2- 172.21.15.46:0/2670883742 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fcb480a24a0 0x7fcb480a4960 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:25.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:25.940+0000 7fcb61888640 1 --2- 172.21.15.46:0/2670883742 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcb5c1033d0 0x7fcb5c19ef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:25.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:25.940+0000 7fcb61888640 1 --2- 172.21.15.46:0/2670883742 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcb5c102a10 0x7fcb5c197f10 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:25.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:25.940+0000 7fcb61888640 1 --2- 172.21.15.46:0/2670883742 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcb5c0ffa40 0x7fcb5c1979d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:25.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:25.940+0000 7fcb61888640 1 -- 172.21.15.46:0/2670883742 >> 172.21.15.46:0/2670883742 conn(0x7fcb5c0fb350 msgr2=0x7fcb5c105de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:09:25.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:25.940+0000 7fcb61888640 1 -- 172.21.15.46:0/2670883742 shutdown_connections 2024-03-31T23:09:25.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:25.940+0000 7fcb61888640 1 -- 172.21.15.46:0/2670883742 wait complete. 2024-03-31T23:09:25.951 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-31T23:09:26.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:26.056+0000 7f093f223640 1 -- 172.21.15.46:0/1566129269 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0938101840 msgr2=0x7f0938101cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:09:26.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:26.056+0000 7f093f223640 1 --2- 172.21.15.46:0/1566129269 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0938101840 0x7f0938101cc0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f0920009900 tx=0x7f092002bc30 comp rx=0 tx=0).stop 2024-03-31T23:09:26.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:26.056+0000 7f093f223640 1 -- 172.21.15.46:0/1566129269 shutdown_connections 2024-03-31T23:09:26.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:26.056+0000 7f093f223640 1 --2- 172.21.15.46:0/1566129269 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0938102200 0x7f0938108a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:26.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:26.056+0000 7f093f223640 1 --2- 172.21.15.46:0/1566129269 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0938101840 0x7f0938101cc0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:26.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:26.056+0000 7f093f223640 1 --2- 172.21.15.46:0/1566129269 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0938100650 0x7f0938100a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:26.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:26.056+0000 7f093f223640 1 -- 172.21.15.46:0/1566129269 >> 172.21.15.46:0/1566129269 conn(0x7f09380fbbe0 msgr2=0x7f09380fe000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:09:26.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:26.056+0000 7f093f223640 1 -- 172.21.15.46:0/1566129269 shutdown_connections 2024-03-31T23:09:26.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:26.056+0000 7f093f223640 1 -- 172.21.15.46:0/1566129269 wait complete. 2024-03-31T23:09:26.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:26.056+0000 7f093f223640 1 Processor -- start 2024-03-31T23:09:26.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:26.056+0000 7f093f223640 1 -- start start 2024-03-31T23:09:26.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:26.057+0000 7f093f223640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0938100650 0x7f093806a910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:09:26.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:26.057+0000 7f093f223640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0938101840 0x7f093819c6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:09:26.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:26.057+0000 7f093f223640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0938102200 0x7f093819ece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:09:26.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:26.057+0000 7f093f223640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f093810c190 con 0x7f0938101840 2024-03-31T23:09:26.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:26.057+0000 7f093f223640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f093810c010 con 0x7f0938100650 2024-03-31T23:09:26.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:26.057+0000 7f093f223640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f093810c310 con 0x7f0938102200 2024-03-31T23:09:26.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:26.057+0000 7f0937fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0938101840 0x7f093819c6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:09:26.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:26.057+0000 7f0937fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0938101840 0x7f093819c6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44868/0 (socket says 172.21.15.46:44868) 2024-03-31T23:09:26.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:26.057+0000 7f0937fff640 1 -- 172.21.15.46:0/2986304129 learned_addr learned my addr 172.21.15.46:0/2986304129 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:09:26.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:26.057+0000 7f093cfbf640 1 --2- 172.21.15.46:0/2986304129 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0938100650 0x7f093806a910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:09:26.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:26.057+0000 7f093d7c0640 1 --2- 172.21.15.46:0/2986304129 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0938102200 0x7f093819ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:09:26.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:26.057+0000 7f0937fff640 1 -- 172.21.15.46:0/2986304129 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0938102200 msgr2=0x7f093819ece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:09:26.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:26.057+0000 7f0937fff640 1 --2- 172.21.15.46:0/2986304129 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0938102200 0x7f093819ece0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:26.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:26.057+0000 7f0937fff640 1 -- 172.21.15.46:0/2986304129 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0938100650 msgr2=0x7f093806a910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:09:26.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:26.057+0000 7f0937fff640 1 --2- 172.21.15.46:0/2986304129 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0938100650 0x7f093806a910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:26.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:26.057+0000 7f0937fff640 1 -- 172.21.15.46:0/2986304129 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f093819f5a0 con 0x7f0938101840 2024-03-31T23:09:26.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:26.057+0000 7f093d7c0640 1 --2- 172.21.15.46:0/2986304129 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0938102200 0x7f093819ece0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T23:09:26.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:26.057+0000 7f0937fff640 1 --2- 172.21.15.46:0/2986304129 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0938101840 0x7f093819c6a0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f0920006010 tx=0x7f0920004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:09:26.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:26.057+0000 7f0935ffb640 1 -- 172.21.15.46:0/2986304129 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0920043020 con 0x7f0938101840 2024-03-31T23:09:26.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:26.057+0000 7f0935ffb640 1 -- 172.21.15.46:0/2986304129 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0920004440 con 0x7f0938101840 2024-03-31T23:09:26.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:26.057+0000 7f093f223640 1 -- 172.21.15.46:0/2986304129 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f093819f830 con 0x7f0938101840 2024-03-31T23:09:26.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:26.057+0000 7f0935ffb640 1 -- 172.21.15.46:0/2986304129 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0920003bf0 con 0x7f0938101840 2024-03-31T23:09:26.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:26.058+0000 7f093f223640 1 -- 172.21.15.46:0/2986304129 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f093819fdd0 con 0x7f0938101840 2024-03-31T23:09:26.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:26.058+0000 7f093f223640 1 -- 172.21.15.46:0/2986304129 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f09381079f0 con 0x7f0938101840 2024-03-31T23:09:26.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:26.059+0000 7f0935ffb640 1 -- 172.21.15.46:0/2986304129 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 16) ==== 137320+0+0 (secure 0 0 0) 0x7f0920046020 con 0x7f0938101840 2024-03-31T23:09:26.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:26.060+0000 7f0935ffb640 1 --2- 172.21.15.46:0/2986304129 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f09240a2520 0x7f09240a49e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:09:26.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:26.060+0000 7f093cfbf640 1 --2- 172.21.15.46:0/2986304129 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f09240a2520 0x7f09240a49e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:09:26.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:26.060+0000 7f0935ffb640 1 -- 172.21.15.46:0/2986304129 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8802+0+0 (secure 0 0 0) 0x7f09200f0ce0 con 0x7f0938101840 2024-03-31T23:09:26.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:26.060+0000 7f093cfbf640 1 --2- 172.21.15.46:0/2986304129 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f09240a2520 0x7f09240a49e0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f09300099a0 tx=0x7f0930009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:09:26.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:26.064+0000 7f0935ffb640 1 -- 172.21.15.46:0/2986304129 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f092004d020 con 0x7f0938101840 2024-03-31T23:09:26.248 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:26.246+0000 7f093f223640 1 -- 172.21.15.46:0/2986304129 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f0938062350 con 0x7f0938101840 2024-03-31T23:09:26.248 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:26.247+0000 7f0935ffb640 1 -- 172.21.15.46:0/2986304129 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v91) ==== 100+0+11 (secure 0 0 0) 0x7f09200bdc00 con 0x7f0938101840 2024-03-31T23:09:26.248 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 64 2024-03-31T23:09:26.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:26.249+0000 7f093f223640 1 -- 172.21.15.46:0/2986304129 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f09240a2520 msgr2=0x7f09240a49e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:09:26.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:26.249+0000 7f093f223640 1 --2- 172.21.15.46:0/2986304129 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f09240a2520 0x7f09240a49e0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f09300099a0 tx=0x7f0930009500 comp rx=0 tx=0).stop 2024-03-31T23:09:26.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:26.249+0000 7f093f223640 1 -- 172.21.15.46:0/2986304129 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0938101840 msgr2=0x7f093819c6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:09:26.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:26.249+0000 7f093f223640 1 --2- 172.21.15.46:0/2986304129 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0938101840 0x7f093819c6a0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f0920006010 tx=0x7f0920004040 comp rx=0 tx=0).stop 2024-03-31T23:09:26.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:26.249+0000 7f093f223640 1 -- 172.21.15.46:0/2986304129 shutdown_connections 2024-03-31T23:09:26.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:26.249+0000 7f093f223640 1 --2- 172.21.15.46:0/2986304129 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f09240a2520 0x7f09240a49e0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:26.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:26.249+0000 7f093f223640 1 --2- 172.21.15.46:0/2986304129 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0938102200 0x7f093819ece0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:26.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:26.249+0000 7f093f223640 1 --2- 172.21.15.46:0/2986304129 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0938101840 0x7f093819c6a0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:26.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:26.249+0000 7f093f223640 1 --2- 172.21.15.46:0/2986304129 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0938100650 0x7f093806a910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:26.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:26.249+0000 7f093f223640 1 -- 172.21.15.46:0/2986304129 >> 172.21.15.46:0/2986304129 conn(0x7f09380fbbe0 msgr2=0x7f0938104bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:09:26.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:26.249+0000 7f093f223640 1 -- 172.21.15.46:0/2986304129 shutdown_connections 2024-03-31T23:09:26.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:26.249+0000 7f093f223640 1 -- 172.21.15.46:0/2986304129 wait complete. 2024-03-31T23:09:26.261 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-31T23:09:26.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:26.365+0000 7f9fba30d640 1 -- 172.21.15.46:0/876753312 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9fb4100650 msgr2=0x7f9fb4100a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:09:26.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:26.365+0000 7f9fba30d640 1 --2- 172.21.15.46:0/876753312 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9fb4100650 0x7f9fb4100a70 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f9fa4009900 tx=0x7f9fa402bc30 comp rx=0 tx=0).stop 2024-03-31T23:09:26.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:26.365+0000 7f9fba30d640 1 -- 172.21.15.46:0/876753312 shutdown_connections 2024-03-31T23:09:26.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:26.365+0000 7f9fba30d640 1 --2- 172.21.15.46:0/876753312 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9fb4102200 0x7f9fb4108a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:26.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:26.365+0000 7f9fba30d640 1 --2- 172.21.15.46:0/876753312 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9fb4101840 0x7f9fb4101cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:26.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:26.365+0000 7f9fba30d640 1 --2- 172.21.15.46:0/876753312 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9fb4100650 0x7f9fb4100a70 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:26.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:26.365+0000 7f9fba30d640 1 -- 172.21.15.46:0/876753312 >> 172.21.15.46:0/876753312 conn(0x7f9fb40fbbe0 msgr2=0x7f9fb40fe000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:09:26.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:26.365+0000 7f9fba30d640 1 -- 172.21.15.46:0/876753312 shutdown_connections 2024-03-31T23:09:26.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:26.365+0000 7f9fba30d640 1 -- 172.21.15.46:0/876753312 wait complete. 2024-03-31T23:09:26.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:26.366+0000 7f9fba30d640 1 Processor -- start 2024-03-31T23:09:26.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:26.366+0000 7f9fba30d640 1 -- start start 2024-03-31T23:09:26.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:26.366+0000 7f9fba30d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9fb4100650 0x7f9fb4196930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:09:26.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:26.366+0000 7f9fba30d640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9fb4101840 0x7f9fb4196e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:09:26.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:26.366+0000 7f9fba30d640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9fb4102200 0x7f9fb419dee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:09:26.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:26.366+0000 7f9fba30d640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f9fb410c170 con 0x7f9fb4100650 2024-03-31T23:09:26.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:26.366+0000 7f9fba30d640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f9fb410bff0 con 0x7f9fb4101840 2024-03-31T23:09:26.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:26.366+0000 7f9fba30d640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f9fb410c2f0 con 0x7f9fb4102200 2024-03-31T23:09:26.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:26.366+0000 7f9fb3fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9fb4100650 0x7f9fb4196930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:09:26.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:26.366+0000 7f9fb3fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9fb4100650 0x7f9fb4196930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44878/0 (socket says 172.21.15.46:44878) 2024-03-31T23:09:26.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:26.366+0000 7f9fb3fff640 1 -- 172.21.15.46:0/1095399710 learned_addr learned my addr 172.21.15.46:0/1095399710 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:09:26.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:26.367+0000 7f9fb37fe640 1 --2- 172.21.15.46:0/1095399710 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9fb4101840 0x7f9fb4196e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:09:26.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:26.367+0000 7f9fb88aa640 1 --2- 172.21.15.46:0/1095399710 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9fb4102200 0x7f9fb419dee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:09:26.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:26.367+0000 7f9fb3fff640 1 -- 172.21.15.46:0/1095399710 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9fb4102200 msgr2=0x7f9fb419dee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:09:26.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:26.367+0000 7f9fb3fff640 1 --2- 172.21.15.46:0/1095399710 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9fb4102200 0x7f9fb419dee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:26.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:26.367+0000 7f9fb3fff640 1 -- 172.21.15.46:0/1095399710 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9fb4101840 msgr2=0x7f9fb4196e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:09:26.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:26.367+0000 7f9fb3fff640 1 --2- 172.21.15.46:0/1095399710 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9fb4101840 0x7f9fb4196e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:26.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:26.367+0000 7f9fb3fff640 1 -- 172.21.15.46:0/1095399710 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9fb419e550 con 0x7f9fb4100650 2024-03-31T23:09:26.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:26.367+0000 7f9fb88aa640 1 --2- 172.21.15.46:0/1095399710 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9fb4102200 0x7f9fb419dee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T23:09:26.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:26.367+0000 7f9fb3fff640 1 --2- 172.21.15.46:0/1095399710 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9fb4100650 0x7f9fb4196930 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f9fa4000c00 tx=0x7f9fa4004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:09:26.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:26.367+0000 7f9fb17fa640 1 -- 172.21.15.46:0/1095399710 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fa4043020 con 0x7f9fb4100650 2024-03-31T23:09:26.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:26.367+0000 7f9fb17fa640 1 -- 172.21.15.46:0/1095399710 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9fa4004440 con 0x7f9fb4100650 2024-03-31T23:09:26.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:26.367+0000 7f9fba30d640 1 -- 172.21.15.46:0/1095399710 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9fb419e7e0 con 0x7f9fb4100650 2024-03-31T23:09:26.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:26.367+0000 7f9fb17fa640 1 -- 172.21.15.46:0/1095399710 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fa4003c90 con 0x7f9fb4100650 2024-03-31T23:09:26.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:26.367+0000 7f9fba30d640 1 -- 172.21.15.46:0/1095399710 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9fb419ed80 con 0x7f9fb4100650 2024-03-31T23:09:26.372 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:26.368+0000 7f9fba30d640 1 -- 172.21.15.46:0/1095399710 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9fb4107a80 con 0x7f9fb4100650 2024-03-31T23:09:26.372 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:26.369+0000 7f9fb17fa640 1 -- 172.21.15.46:0/1095399710 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 16) ==== 137320+0+0 (secure 0 0 0) 0x7f9fa4046020 con 0x7f9fb4100650 2024-03-31T23:09:26.372 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:26.370+0000 7f9fb17fa640 1 --2- 172.21.15.46:0/1095399710 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f9f9c0a2520 0x7f9f9c0a49e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:09:26.372 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:26.370+0000 7f9fb37fe640 1 --2- 172.21.15.46:0/1095399710 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f9f9c0a2520 0x7f9f9c0a49e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:09:26.372 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:26.370+0000 7f9fb17fa640 1 -- 172.21.15.46:0/1095399710 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8802+0+0 (secure 0 0 0) 0x7f9fa40f0bc0 con 0x7f9fb4100650 2024-03-31T23:09:26.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:26.370+0000 7f9fb37fe640 1 --2- 172.21.15.46:0/1095399710 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f9f9c0a2520 0x7f9f9c0a49e0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f9fb4197e40 tx=0x7f9fa800b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:09:26.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:26.374+0000 7f9fb17fa640 1 -- 172.21.15.46:0/1095399710 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f9fa404d020 con 0x7f9fb4100650 2024-03-31T23:09:26.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:26.556+0000 7f9fba30d640 1 -- 172.21.15.46:0/1095399710 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f9fb402d170 con 0x7f9fb4100650 2024-03-31T23:09:26.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:26.556+0000 7f9fb17fa640 1 -- 172.21.15.46:0/1095399710 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v91) ==== 103+0+11 (secure 0 0 0) 0x7f9fa40bdae0 con 0x7f9fb4100650 2024-03-31T23:09:26.557 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 32 2024-03-31T23:09:26.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:26.558+0000 7f9fba30d640 1 -- 172.21.15.46:0/1095399710 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f9f9c0a2520 msgr2=0x7f9f9c0a49e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:09:26.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:26.558+0000 7f9fba30d640 1 --2- 172.21.15.46:0/1095399710 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f9f9c0a2520 0x7f9f9c0a49e0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f9fb4197e40 tx=0x7f9fa800b390 comp rx=0 tx=0).stop 2024-03-31T23:09:26.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:26.558+0000 7f9fba30d640 1 -- 172.21.15.46:0/1095399710 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9fb4100650 msgr2=0x7f9fb4196930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:09:26.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:26.558+0000 7f9fba30d640 1 --2- 172.21.15.46:0/1095399710 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9fb4100650 0x7f9fb4196930 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f9fa4000c00 tx=0x7f9fa4004040 comp rx=0 tx=0).stop 2024-03-31T23:09:26.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:26.558+0000 7f9fba30d640 1 -- 172.21.15.46:0/1095399710 shutdown_connections 2024-03-31T23:09:26.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:26.558+0000 7f9fba30d640 1 --2- 172.21.15.46:0/1095399710 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f9f9c0a2520 0x7f9f9c0a49e0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:26.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:26.558+0000 7f9fba30d640 1 --2- 172.21.15.46:0/1095399710 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9fb4102200 0x7f9fb419dee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:26.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:26.558+0000 7f9fba30d640 1 --2- 172.21.15.46:0/1095399710 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9fb4101840 0x7f9fb4196e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:26.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:26.558+0000 7f9fba30d640 1 --2- 172.21.15.46:0/1095399710 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9fb4100650 0x7f9fb4196930 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:26.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:26.558+0000 7f9fba30d640 1 -- 172.21.15.46:0/1095399710 >> 172.21.15.46:0/1095399710 conn(0x7f9fb40fbbe0 msgr2=0x7f9fb4104c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:09:26.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:26.558+0000 7f9fba30d640 1 -- 172.21.15.46:0/1095399710 shutdown_connections 2024-03-31T23:09:26.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:26.558+0000 7f9fba30d640 1 -- 172.21.15.46:0/1095399710 wait complete. 2024-03-31T23:09:26.569 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:09:26 smithi046 ceph-mon[32498]: pgmap v299: 129 pgs: 129 active+clean; 579 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-31T23:09:26.569 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:09:26 smithi046 ceph-mon[32498]: from='client.? 172.21.15.46:0/2962655238' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-31T23:09:26.569 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:09:26 smithi046 ceph-mon[32498]: from='client.? 172.21.15.46:0/2670883742' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-31T23:09:26.569 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:09:26 smithi046 ceph-mon[32498]: from='client.? 172.21.15.46:0/2986304129' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-31T23:09:26.570 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-31T23:09:26.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:26.674+0000 7f7f43d7a640 1 -- 172.21.15.46:0/666744085 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7f3c0ffa30 msgr2=0x7f7f3c0ffe50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:09:26.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:26.674+0000 7f7f43d7a640 1 --2- 172.21.15.46:0/666744085 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7f3c0ffa30 0x7f7f3c0ffe50 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f7f38009930 tx=0x7f7f3802bc30 comp rx=0 tx=0).stop 2024-03-31T23:09:26.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:26.674+0000 7f7f43d7a640 1 -- 172.21.15.46:0/666744085 shutdown_connections 2024-03-31T23:09:26.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:26.674+0000 7f7f43d7a640 1 --2- 172.21.15.46:0/666744085 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7f3c1033c0 0x7f7f3c109c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:26.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:26.674+0000 7f7f43d7a640 1 --2- 172.21.15.46:0/666744085 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7f3c102a00 0x7f7f3c102e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:26.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:26.674+0000 7f7f43d7a640 1 --2- 172.21.15.46:0/666744085 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7f3c0ffa30 0x7f7f3c0ffe50 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:26.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:26.674+0000 7f7f43d7a640 1 -- 172.21.15.46:0/666744085 >> 172.21.15.46:0/666744085 conn(0x7f7f3c0fb340 msgr2=0x7f7f3c0fd7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:09:26.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:26.674+0000 7f7f43d7a640 1 -- 172.21.15.46:0/666744085 shutdown_connections 2024-03-31T23:09:26.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:26.674+0000 7f7f43d7a640 1 -- 172.21.15.46:0/666744085 wait complete. 2024-03-31T23:09:26.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:26.675+0000 7f7f43d7a640 1 Processor -- start 2024-03-31T23:09:26.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:26.675+0000 7f7f43d7a640 1 -- start start 2024-03-31T23:09:26.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:26.675+0000 7f7f43d7a640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7f3c0ffa30 0x7f7f3c197a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:09:26.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:26.675+0000 7f7f43d7a640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7f3c102a00 0x7f7f3c197f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:09:26.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:26.675+0000 7f7f43d7a640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7f3c1033c0 0x7f7f3c19efe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:09:26.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:26.675+0000 7f7f43d7a640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f7f3c10d2b0 con 0x7f7f3c102a00 2024-03-31T23:09:26.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:26.675+0000 7f7f43d7a640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f7f3c10d130 con 0x7f7f3c1033c0 2024-03-31T23:09:26.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:26.675+0000 7f7f43d7a640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f7f3c10d430 con 0x7f7f3c0ffa30 2024-03-31T23:09:26.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:26.675+0000 7f7f41315640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7f3c102a00 0x7f7f3c197f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:09:26.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:26.675+0000 7f7f41315640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7f3c102a00 0x7f7f3c197f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44888/0 (socket says 172.21.15.46:44888) 2024-03-31T23:09:26.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:26.675+0000 7f7f41315640 1 -- 172.21.15.46:0/1362418804 learned_addr learned my addr 172.21.15.46:0/1362418804 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:09:26.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:26.675+0000 7f7f42317640 1 --2- 172.21.15.46:0/1362418804 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7f3c1033c0 0x7f7f3c19efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:09:26.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:26.675+0000 7f7f41315640 1 -- 172.21.15.46:0/1362418804 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7f3c0ffa30 msgr2=0x7f7f3c197a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:09:26.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:26.675+0000 7f7f41315640 1 --2- 172.21.15.46:0/1362418804 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7f3c0ffa30 0x7f7f3c197a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:26.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:26.675+0000 7f7f41315640 1 -- 172.21.15.46:0/1362418804 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7f3c1033c0 msgr2=0x7f7f3c19efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:09:26.680 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:26.675+0000 7f7f41315640 1 --2- 172.21.15.46:0/1362418804 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7f3c1033c0 0x7f7f3c19efe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:26.680 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:26.676+0000 7f7f41315640 1 -- 172.21.15.46:0/1362418804 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f3c19f650 con 0x7f7f3c102a00 2024-03-31T23:09:26.680 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:26.676+0000 7f7f42317640 1 --2- 172.21.15.46:0/1362418804 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7f3c1033c0 0x7f7f3c19efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T23:09:26.680 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:26.676+0000 7f7f41315640 1 --2- 172.21.15.46:0/1362418804 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7f3c102a00 0x7f7f3c197f70 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f7f2c00d770 tx=0x7f7f2c00dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:09:26.680 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:26.676+0000 7f7f32ffd640 1 -- 172.21.15.46:0/1362418804 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f2c017070 con 0x7f7f3c102a00 2024-03-31T23:09:26.680 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:26.676+0000 7f7f32ffd640 1 -- 172.21.15.46:0/1362418804 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7f2c0054d0 con 0x7f7f3c102a00 2024-03-31T23:09:26.680 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:26.676+0000 7f7f43d7a640 1 -- 172.21.15.46:0/1362418804 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7f3c19f940 con 0x7f7f3c102a00 2024-03-31T23:09:26.680 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:26.676+0000 7f7f32ffd640 1 -- 172.21.15.46:0/1362418804 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f2c004cf0 con 0x7f7f3c102a00 2024-03-31T23:09:26.680 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:26.676+0000 7f7f43d7a640 1 -- 172.21.15.46:0/1362418804 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7f3c19fe90 con 0x7f7f3c102a00 2024-03-31T23:09:26.680 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:26.677+0000 7f7f43d7a640 1 -- 172.21.15.46:0/1362418804 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7f3c108c50 con 0x7f7f3c102a00 2024-03-31T23:09:26.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:26.678+0000 7f7f32ffd640 1 -- 172.21.15.46:0/1362418804 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 16) ==== 137320+0+0 (secure 0 0 0) 0x7f7f2c004e90 con 0x7f7f3c102a00 2024-03-31T23:09:26.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:26.678+0000 7f7f32ffd640 1 --2- 172.21.15.46:0/1362418804 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f7f280a2520 0x7f7f280a49e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:09:26.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:26.678+0000 7f7f41b16640 1 --2- 172.21.15.46:0/1362418804 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f7f280a2520 0x7f7f280a49e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:09:26.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:26.678+0000 7f7f32ffd640 1 -- 172.21.15.46:0/1362418804 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8802+0+0 (secure 0 0 0) 0x7f7f2c0ce3f0 con 0x7f7f3c102a00 2024-03-31T23:09:26.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:26.679+0000 7f7f41b16640 1 --2- 172.21.15.46:0/1362418804 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f7f280a2520 0x7f7f280a49e0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f7f380096c0 tx=0x7f7f38000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:09:26.684 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:26.683+0000 7f7f32ffd640 1 -- 172.21.15.46:0/1362418804 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f7f2c09b370 con 0x7f7f3c102a00 2024-03-31T23:09:26.693 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:09:26 smithi142 ceph-mon[35271]: pgmap v299: 129 pgs: 129 active+clean; 579 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-31T23:09:26.693 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:09:26 smithi142 ceph-mon[35271]: from='client.? 172.21.15.46:0/2962655238' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-31T23:09:26.693 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:09:26 smithi142 ceph-mon[35271]: from='client.? 172.21.15.46:0/2670883742' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-31T23:09:26.693 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:09:26 smithi142 ceph-mon[35271]: from='client.? 172.21.15.46:0/2986304129' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-31T23:09:26.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:09:26 smithi162 ceph-mon[36237]: pgmap v299: 129 pgs: 129 active+clean; 579 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-31T23:09:26.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:09:26 smithi162 ceph-mon[36237]: from='client.? 172.21.15.46:0/2962655238' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-31T23:09:26.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:09:26 smithi162 ceph-mon[36237]: from='client.? 172.21.15.46:0/2670883742' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-31T23:09:26.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:09:26 smithi162 ceph-mon[36237]: from='client.? 172.21.15.46:0/2986304129' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-31T23:09:26.880 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:26.879+0000 7f7f43d7a640 1 -- 172.21.15.46:0/1362418804 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f7f3c02d170 con 0x7f7f3c102a00 2024-03-31T23:09:26.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:26.880+0000 7f7f32ffd640 1 -- 172.21.15.46:0/1362418804 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v91) ==== 81+0+591 (secure 0 0 0) 0x7f7f2c09ac70 con 0x7f7f3c102a00 2024-03-31T23:09:26.881 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:0/4021039857 2024-04-01T23:02:08.493619+0000 2024-03-31T23:09:26.881 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6801/3681182647 2024-04-01T23:02:08.493619+0000 2024-03-31T23:09:26.881 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:0/4022214806 2024-04-01T23:02:08.493619+0000 2024-03-31T23:09:26.881 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6800/1515528215 2024-04-01T23:01:41.910162+0000 2024-03-31T23:09:26.881 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6800/3681182647 2024-04-01T23:02:08.493619+0000 2024-03-31T23:09:26.881 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:0/143885707 2024-04-01T23:01:41.910162+0000 2024-03-31T23:09:26.881 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6801/1515528215 2024-04-01T23:01:41.910162+0000 2024-03-31T23:09:26.881 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:0/1680389778 2024-04-01T23:01:41.910162+0000 2024-03-31T23:09:26.882 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:0/1782133966 2024-04-01T23:01:41.910162+0000 2024-03-31T23:09:26.882 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:0/3925209376 2024-04-01T23:02:08.493619+0000 2024-03-31T23:09:26.882 INFO:teuthology.orchestra.run.smithi046.stderr:listed 10 entries 2024-03-31T23:09:26.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:26.882+0000 7f7f43d7a640 1 -- 172.21.15.46:0/1362418804 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f7f280a2520 msgr2=0x7f7f280a49e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:09:26.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:26.882+0000 7f7f43d7a640 1 --2- 172.21.15.46:0/1362418804 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f7f280a2520 0x7f7f280a49e0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f7f380096c0 tx=0x7f7f38000bc0 comp rx=0 tx=0).stop 2024-03-31T23:09:26.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:26.882+0000 7f7f43d7a640 1 -- 172.21.15.46:0/1362418804 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7f3c102a00 msgr2=0x7f7f3c197f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:09:26.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:26.882+0000 7f7f43d7a640 1 --2- 172.21.15.46:0/1362418804 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7f3c102a00 0x7f7f3c197f70 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f7f2c00d770 tx=0x7f7f2c00dc30 comp rx=0 tx=0).stop 2024-03-31T23:09:26.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:26.882+0000 7f7f43d7a640 1 -- 172.21.15.46:0/1362418804 shutdown_connections 2024-03-31T23:09:26.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:26.882+0000 7f7f43d7a640 1 --2- 172.21.15.46:0/1362418804 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f7f280a2520 0x7f7f280a49e0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:26.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:26.882+0000 7f7f43d7a640 1 --2- 172.21.15.46:0/1362418804 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7f3c1033c0 0x7f7f3c19efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:26.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:26.882+0000 7f7f43d7a640 1 --2- 172.21.15.46:0/1362418804 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7f3c102a00 0x7f7f3c197f70 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:26.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:26.882+0000 7f7f43d7a640 1 --2- 172.21.15.46:0/1362418804 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7f3c0ffa30 0x7f7f3c197a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:26.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:26.882+0000 7f7f43d7a640 1 -- 172.21.15.46:0/1362418804 >> 172.21.15.46:0/1362418804 conn(0x7f7f3c0fb340 msgr2=0x7f7f3c105e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:09:26.884 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:26.882+0000 7f7f43d7a640 1 -- 172.21.15.46:0/1362418804 shutdown_connections 2024-03-31T23:09:26.884 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:26.882+0000 7f7f43d7a640 1 -- 172.21.15.46:0/1362418804 wait complete. 2024-03-31T23:09:26.893 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi046.front.sepia.ceph.com... 2024-03-31T23:09:26.894 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T23:09:26.894 DEBUG:teuthology.orchestra.run.smithi046:> sudo fusermount -u /home/ubuntu/cephtest/mnt.admin 2024-03-31T23:09:26.919 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi046.stderr:ceph-fuse[76163]: fuse finished with error 0 and tester_r 0 2024-03-31T23:09:26.921 INFO:teuthology.orchestra.run:waiting for 300 2024-03-31T23:09:26.958 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi046.front.sepia.ceph.com 2024-03-31T23:09:26.958 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T23:09:26.959 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.admin) 2024-03-31T23:09:27.110 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-31T23:09:27.217 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:27.214+0000 7f1798fbc640 1 -- 172.21.15.46:0/1983102299 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1794102da0 msgr2=0x7f17941031c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:09:27.217 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:27.214+0000 7f1798fbc640 1 --2- 172.21.15.46:0/1983102299 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1794102da0 0x7f17941031c0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f1784009900 tx=0x7f178402bc30 comp rx=0 tx=0).stop 2024-03-31T23:09:27.218 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:27.214+0000 7f1798fbc640 1 -- 172.21.15.46:0/1983102299 shutdown_connections 2024-03-31T23:09:27.218 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:27.214+0000 7f1798fbc640 1 --2- 172.21.15.46:0/1983102299 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f17941001d0 0x7f1794109d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:27.218 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:27.214+0000 7f1798fbc640 1 --2- 172.21.15.46:0/1983102299 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f17940ff810 0x7f17940ffc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:27.218 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:27.214+0000 7f1798fbc640 1 --2- 172.21.15.46:0/1983102299 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1794102da0 0x7f17941031c0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:27.218 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:27.214+0000 7f1798fbc640 1 -- 172.21.15.46:0/1983102299 >> 172.21.15.46:0/1983102299 conn(0x7f17940fb390 msgr2=0x7f17940fd7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:09:27.218 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:27.214+0000 7f1798fbc640 1 -- 172.21.15.46:0/1983102299 shutdown_connections 2024-03-31T23:09:27.218 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:27.214+0000 7f1798fbc640 1 -- 172.21.15.46:0/1983102299 wait complete. 2024-03-31T23:09:27.218 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:27.215+0000 7f1798fbc640 1 Processor -- start 2024-03-31T23:09:27.218 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:27.215+0000 7f1798fbc640 1 -- start start 2024-03-31T23:09:27.218 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:27.215+0000 7f1798fbc640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f17940ff810 0x7f17941979e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:09:27.218 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:27.215+0000 7f1798fbc640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f17941001d0 0x7f1794197f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:09:27.218 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:27.215+0000 7f1798fbc640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1794102da0 0x7f179419ef90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:09:27.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:27.215+0000 7f1798fbc640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f179410d2b0 con 0x7f17941001d0 2024-03-31T23:09:27.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:27.215+0000 7f1798fbc640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f179410d130 con 0x7f1794102da0 2024-03-31T23:09:27.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:27.215+0000 7f1798fbc640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f179410d430 con 0x7f17940ff810 2024-03-31T23:09:27.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:27.215+0000 7f1791d9b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f17941001d0 0x7f1794197f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:09:27.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:27.215+0000 7f1791d9b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f17941001d0 0x7f1794197f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44908/0 (socket says 172.21.15.46:44908) 2024-03-31T23:09:27.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:27.215+0000 7f1791d9b640 1 -- 172.21.15.46:0/627857365 learned_addr learned my addr 172.21.15.46:0/627857365 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:09:27.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:27.215+0000 7f1792d9d640 1 --2- 172.21.15.46:0/627857365 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1794102da0 0x7f179419ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:09:27.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:27.215+0000 7f179259c640 1 --2- 172.21.15.46:0/627857365 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f17940ff810 0x7f17941979e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:09:27.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:27.215+0000 7f1791d9b640 1 -- 172.21.15.46:0/627857365 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f17940ff810 msgr2=0x7f17941979e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:09:27.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:27.215+0000 7f1791d9b640 1 --2- 172.21.15.46:0/627857365 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f17940ff810 0x7f17941979e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:27.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:27.215+0000 7f1791d9b640 1 -- 172.21.15.46:0/627857365 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1794102da0 msgr2=0x7f179419ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:09:27.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:27.215+0000 7f1791d9b640 1 --2- 172.21.15.46:0/627857365 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1794102da0 0x7f179419ef90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:27.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:27.215+0000 7f1791d9b640 1 -- 172.21.15.46:0/627857365 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f179419f600 con 0x7f17941001d0 2024-03-31T23:09:27.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:27.215+0000 7f179259c640 1 --2- 172.21.15.46:0/627857365 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f17940ff810 0x7f17941979e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T23:09:27.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:27.215+0000 7f1791d9b640 1 --2- 172.21.15.46:0/627857365 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f17941001d0 0x7f1794197f20 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f178800d950 tx=0x7f178800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:09:27.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:27.216+0000 7f17837fe640 1 -- 172.21.15.46:0/627857365 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1788017030 con 0x7f17941001d0 2024-03-31T23:09:27.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:27.216+0000 7f17837fe640 1 -- 172.21.15.46:0/627857365 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1788004020 con 0x7f17941001d0 2024-03-31T23:09:27.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:27.216+0000 7f1798fbc640 1 -- 172.21.15.46:0/627857365 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f179419f8f0 con 0x7f17941001d0 2024-03-31T23:09:27.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:27.216+0000 7f17837fe640 1 -- 172.21.15.46:0/627857365 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17880047a0 con 0x7f17941001d0 2024-03-31T23:09:27.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:27.216+0000 7f1798fbc640 1 -- 172.21.15.46:0/627857365 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f179419fe90 con 0x7f17941001d0 2024-03-31T23:09:27.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:27.217+0000 7f1798fbc640 1 -- 172.21.15.46:0/627857365 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1794108d30 con 0x7f17941001d0 2024-03-31T23:09:27.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:27.218+0000 7f17837fe640 1 -- 172.21.15.46:0/627857365 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 16) ==== 137320+0+0 (secure 0 0 0) 0x7f17880041d0 con 0x7f17941001d0 2024-03-31T23:09:27.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:27.218+0000 7f17837fe640 1 --2- 172.21.15.46:0/627857365 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f177c0a2520 0x7f177c0a49e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:09:27.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:27.219+0000 7f17837fe640 1 -- 172.21.15.46:0/627857365 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8802+0+0 (secure 0 0 0) 0x7f17880ce3e0 con 0x7f17941001d0 2024-03-31T23:09:27.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:27.219+0000 7f179259c640 1 --2- 172.21.15.46:0/627857365 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f177c0a2520 0x7f177c0a49e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:09:27.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:27.219+0000 7f179259c640 1 --2- 172.21.15.46:0/627857365 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f177c0a2520 0x7f177c0a49e0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f17840096c0 tx=0x7f1784000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:09:27.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:27.223+0000 7f17837fe640 1 -- 172.21.15.46:0/627857365 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f178809b360 con 0x7f17941001d0 2024-03-31T23:09:27.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:27.412+0000 7f1798fbc640 1 -- 172.21.15.46:0/627857365 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f1794062350 con 0x7f17941001d0 2024-03-31T23:09:27.414 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:27.413+0000 7f17837fe640 1 -- 172.21.15.46:0/627857365 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) ==== 74+0+24105 (secure 0 0 0) 0x7f178809ac60 con 0x7f17941001d0 2024-03-31T23:09:27.414 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-31T23:09:27.415 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":91,"fsid":"6a9d8f08-efb2-11ee-b647-cb9ed24678a4","created":"2024-03-31T23:01:15.463711+0000","modified":"2024-03-31T23:09:17.181787+0000","last_up_change":"2024-03-31T23:08:20.199888+0000","last_in_change":"2024-03-31T23:07:59.438035+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-31T23:04:32.823751+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-31T23:09:09.145711+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-31T23:09:10.148579+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-31T23:09:13.419519+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"91","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"47429095-3f46-4ed7-b7c4-504464719815","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":1824772695},{"type":"v1","addr":"172.21.15.46:6803","nonce":1824772695}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":1824772695},{"type":"v1","addr":"172.21.15.46:6805","nonce":1824772695}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":1824772695},{"type":"v1","addr":"172.21.15.46:6809","nonce":1824772695}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":1824772695},{"type":"v1","addr":"172.21.15.46:6807","nonce":1824772695}]},"public_addr":"172.21.15.46:6803/1824772695","cluster_addr":"172.21.15.46:6805/1824772695","heartbeat_back_addr":"172.21.15.46:6809/1824772695","heartbeat_front_addr":"172.21.15.46:6807/1824772695","state":["exists","up"]},{"osd":1,"uuid":"be56f883-411e-4dbe-b50e-a158134df4a6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6800","nonce":973693922},{"type":"v1","addr":"172.21.15.142:6801","nonce":973693922}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6802","nonce":973693922},{"type":"v1","addr":"172.21.15.142:6803","nonce":973693922}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6806","nonce":973693922},{"type":"v1","addr":"172.21.15.142:6807","nonce":973693922}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6804","nonce":973693922},{"type":"v1","addr":"172.21.15.142:6805","nonce":973693922}]},"public_addr":"172.21.15.142:6801/973693922","cluster_addr":"172.21.15.142:6803/973693922","heartbeat_back_addr":"172.21.15.142:6807/973693922","heartbeat_front_addr":"172.21.15.142:6805/973693922","state":["exists","up"]},{"osd":2,"uuid":"c99a8794-ab19-4260-9a8d-4bc0bd2c8175","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6800","nonce":4084261813},{"type":"v1","addr":"172.21.15.162:6801","nonce":4084261813}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6802","nonce":4084261813},{"type":"v1","addr":"172.21.15.162:6803","nonce":4084261813}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6806","nonce":4084261813},{"type":"v1","addr":"172.21.15.162:6807","nonce":4084261813}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6804","nonce":4084261813},{"type":"v1","addr":"172.21.15.162:6805","nonce":4084261813}]},"public_addr":"172.21.15.162:6801/4084261813","cluster_addr":"172.21.15.162:6803/4084261813","heartbeat_back_addr":"172.21.15.162:6807/4084261813","heartbeat_front_addr":"172.21.15.162:6805/4084261813","state":["exists","up"]},{"osd":3,"uuid":"f877814e-9ef6-4872-a081-5609bb9ebc06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":3496317970},{"type":"v1","addr":"172.21.15.46:6811","nonce":3496317970}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":3496317970},{"type":"v1","addr":"172.21.15.46:6813","nonce":3496317970}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":3496317970},{"type":"v1","addr":"172.21.15.46:6817","nonce":3496317970}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":3496317970},{"type":"v1","addr":"172.21.15.46:6815","nonce":3496317970}]},"public_addr":"172.21.15.46:6811/3496317970","cluster_addr":"172.21.15.46:6813/3496317970","heartbeat_back_addr":"172.21.15.46:6817/3496317970","heartbeat_front_addr":"172.21.15.46:6815/3496317970","state":["exists","up"]},{"osd":4,"uuid":"beecec0a-8280-44fe-8f5f-8498a35eca8f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6808","nonce":2996266929},{"type":"v1","addr":"172.21.15.142:6809","nonce":2996266929}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6810","nonce":2996266929},{"type":"v1","addr":"172.21.15.142:6811","nonce":2996266929}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6814","nonce":2996266929},{"type":"v1","addr":"172.21.15.142:6815","nonce":2996266929}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6812","nonce":2996266929},{"type":"v1","addr":"172.21.15.142:6813","nonce":2996266929}]},"public_addr":"172.21.15.142:6809/2996266929","cluster_addr":"172.21.15.142:6811/2996266929","heartbeat_back_addr":"172.21.15.142:6815/2996266929","heartbeat_front_addr":"172.21.15.142:6813/2996266929","state":["exists","up"]},{"osd":5,"uuid":"9da7ff75-f658-45ea-a02f-1943fdda76b9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6808","nonce":914208293},{"type":"v1","addr":"172.21.15.162:6809","nonce":914208293}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6810","nonce":914208293},{"type":"v1","addr":"172.21.15.162:6811","nonce":914208293}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6814","nonce":914208293},{"type":"v1","addr":"172.21.15.162:6815","nonce":914208293}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6812","nonce":914208293},{"type":"v1","addr":"172.21.15.162:6813","nonce":914208293}]},"public_addr":"172.21.15.162:6809/914208293","cluster_addr":"172.21.15.162:6811/914208293","heartbeat_back_addr":"172.21.15.162:6815/914208293","heartbeat_front_addr":"172.21.15.162:6813/914208293","state":["exists","up"]},{"osd":6,"uuid":"8fa11224-cb26-4d3f-87d1-6a8f31de2a19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":56291966},{"type":"v1","addr":"172.21.15.46:6819","nonce":56291966}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":56291966},{"type":"v1","addr":"172.21.15.46:6821","nonce":56291966}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6824","nonce":56291966},{"type":"v1","addr":"172.21.15.46:6825","nonce":56291966}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":56291966},{"type":"v1","addr":"172.21.15.46:6823","nonce":56291966}]},"public_addr":"172.21.15.46:6819/56291966","cluster_addr":"172.21.15.46:6821/56291966","heartbeat_back_addr":"172.21.15.46:6825/56291966","heartbeat_front_addr":"172.21.15.46:6823/56291966","state":["exists","up"]},{"osd":7,"uuid":"0d0338b7-4d0a-4bb0-b152-8f5224871d9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6816","nonce":2146598446},{"type":"v1","addr":"172.21.15.142:6817","nonce":2146598446}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6818","nonce":2146598446},{"type":"v1","addr":"172.21.15.142:6819","nonce":2146598446}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6822","nonce":2146598446},{"type":"v1","addr":"172.21.15.142:6823","nonce":2146598446}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6820","nonce":2146598446},{"type":"v1","addr":"172.21.15.142:6821","nonce":2146598446}]},"public_addr":"172.21.15.142:6817/2146598446","cluster_addr":"172.21.15.142:6819/2146598446","heartbeat_back_addr":"172.21.15.142:6823/2146598446","heartbeat_front_addr":"172.21.15.142:6821/2146598446","state":["exists","up"]},{"osd":8,"uuid":"8b6f6037-892a-416c-babb-7552648d0b3a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6816","nonce":1173078318},{"type":"v1","addr":"172.21.15.162:6817","nonce":1173078318}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6818","nonce":1173078318},{"type":"v1","addr":"172.21.15.162:6819","nonce":1173078318}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6822","nonce":1173078318},{"type":"v1","addr":"172.21.15.162:6823","nonce":1173078318}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6820","nonce":1173078318},{"type":"v1","addr":"172.21.15.162:6821","nonce":1173078318}]},"public_addr":"172.21.15.162:6817/1173078318","cluster_addr":"172.21.15.162:6819/1173078318","heartbeat_back_addr":"172.21.15.162:6823/1173078318","heartbeat_front_addr":"172.21.15.162:6821/1173078318","state":["exists","up"]},{"osd":9,"uuid":"faccce25-6bf2-4220-8858-cda82b72be68","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":532708881},{"type":"v1","addr":"172.21.15.46:6827","nonce":532708881}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6828","nonce":532708881},{"type":"v1","addr":"172.21.15.46:6829","nonce":532708881}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":532708881},{"type":"v1","addr":"172.21.15.46:6833","nonce":532708881}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":532708881},{"type":"v1","addr":"172.21.15.46:6831","nonce":532708881}]},"public_addr":"172.21.15.46:6827/532708881","cluster_addr":"172.21.15.46:6829/532708881","heartbeat_back_addr":"172.21.15.46:6833/532708881","heartbeat_front_addr":"172.21.15.46:6831/532708881","state":["exists","up"]},{"osd":10,"uuid":"cdd356db-f9eb-4080-95ab-db5d8223e572","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6824","nonce":2845222366},{"type":"v1","addr":"172.21.15.142:6825","nonce":2845222366}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6826","nonce":2845222366},{"type":"v1","addr":"172.21.15.142:6827","nonce":2845222366}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6830","nonce":2845222366},{"type":"v1","addr":"172.21.15.142:6831","nonce":2845222366}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6828","nonce":2845222366},{"type":"v1","addr":"172.21.15.142:6829","nonce":2845222366}]},"public_addr":"172.21.15.142:6825/2845222366","cluster_addr":"172.21.15.142:6827/2845222366","heartbeat_back_addr":"172.21.15.142:6831/2845222366","heartbeat_front_addr":"172.21.15.142:6829/2845222366","state":["exists","up"]},{"osd":11,"uuid":"72104d1c-3ea2-45ad-bc0b-0500f146c06b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6824","nonce":1986323778},{"type":"v1","addr":"172.21.15.162:6825","nonce":1986323778}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6826","nonce":1986323778},{"type":"v1","addr":"172.21.15.162:6827","nonce":1986323778}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6830","nonce":1986323778},{"type":"v1","addr":"172.21.15.162:6831","nonce":1986323778}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6828","nonce":1986323778},{"type":"v1","addr":"172.21.15.162:6829","nonce":1986323778}]},"public_addr":"172.21.15.162:6825/1986323778","cluster_addr":"172.21.15.162:6827/1986323778","heartbeat_back_addr":"172.21.15.162:6831/1986323778","heartbeat_front_addr":"172.21.15.162:6829/1986323778","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:03:44.996331+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:04:07.994738+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:04:30.942255+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:04:55.588884+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:05:19.900600+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:05:43.807442+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:06:09.026976+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:06:34.586784+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:06:59.751605+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:07:26.144495+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:07:52.326909+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:08:17.423042+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.46:0/4021039857":"2024-04-01T23:02:08.493619+0000","172.21.15.46:6801/3681182647":"2024-04-01T23:02:08.493619+0000","172.21.15.46:0/4022214806":"2024-04-01T23:02:08.493619+0000","172.21.15.46:6800/1515528215":"2024-04-01T23:01:41.910162+0000","172.21.15.46:6800/3681182647":"2024-04-01T23:02:08.493619+0000","172.21.15.46:0/143885707":"2024-04-01T23:01:41.910162+0000","172.21.15.46:6801/1515528215":"2024-04-01T23:01:41.910162+0000","172.21.15.46:0/1680389778":"2024-04-01T23:01:41.910162+0000","172.21.15.46:0/1782133966":"2024-04-01T23:01:41.910162+0000","172.21.15.46:0/3925209376":"2024-04-01T23:02:08.493619+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-31T23:09:27.416 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:27.415+0000 7f1798fbc640 1 -- 172.21.15.46:0/627857365 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f177c0a2520 msgr2=0x7f177c0a49e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:09:27.417 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:27.415+0000 7f1798fbc640 1 --2- 172.21.15.46:0/627857365 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f177c0a2520 0x7f177c0a49e0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f17840096c0 tx=0x7f1784000bc0 comp rx=0 tx=0).stop 2024-03-31T23:09:27.417 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:27.415+0000 7f1798fbc640 1 -- 172.21.15.46:0/627857365 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f17941001d0 msgr2=0x7f1794197f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:09:27.417 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:27.415+0000 7f1798fbc640 1 --2- 172.21.15.46:0/627857365 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f17941001d0 0x7f1794197f20 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f178800d950 tx=0x7f178800de10 comp rx=0 tx=0).stop 2024-03-31T23:09:27.417 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:27.415+0000 7f1798fbc640 1 -- 172.21.15.46:0/627857365 shutdown_connections 2024-03-31T23:09:27.417 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:27.415+0000 7f1798fbc640 1 --2- 172.21.15.46:0/627857365 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f177c0a2520 0x7f177c0a49e0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:27.417 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:27.415+0000 7f1798fbc640 1 --2- 172.21.15.46:0/627857365 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1794102da0 0x7f179419ef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:27.417 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:27.415+0000 7f1798fbc640 1 --2- 172.21.15.46:0/627857365 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f17941001d0 0x7f1794197f20 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:27.417 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:27.415+0000 7f1798fbc640 1 --2- 172.21.15.46:0/627857365 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f17940ff810 0x7f17941979e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:27.417 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:27.415+0000 7f1798fbc640 1 -- 172.21.15.46:0/627857365 >> 172.21.15.46:0/627857365 conn(0x7f17940fb390 msgr2=0x7f1794105f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:09:27.417 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:27.416+0000 7f1798fbc640 1 -- 172.21.15.46:0/627857365 shutdown_connections 2024-03-31T23:09:27.417 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:27.416+0000 7f1798fbc640 1 -- 172.21.15.46:0/627857365 wait complete. 2024-03-31T23:09:27.428 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-31T23:09:27.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:27.530+0000 7f1ebd902640 1 -- 172.21.15.46:0/500531954 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1eb81018e0 msgr2=0x7f1eb8101d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:09:27.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:27.530+0000 7f1ebd902640 1 --2- 172.21.15.46:0/500531954 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1eb81018e0 0x7f1eb8101d60 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f1eac009900 tx=0x7f1eac02bc30 comp rx=0 tx=0).stop 2024-03-31T23:09:27.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:27.530+0000 7f1ebd902640 1 -- 172.21.15.46:0/500531954 shutdown_connections 2024-03-31T23:09:27.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:27.530+0000 7f1ebd902640 1 --2- 172.21.15.46:0/500531954 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1eb81022a0 0x7f1eb8108b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:27.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:27.530+0000 7f1ebd902640 1 --2- 172.21.15.46:0/500531954 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1eb81018e0 0x7f1eb8101d60 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:27.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:27.530+0000 7f1ebd902640 1 --2- 172.21.15.46:0/500531954 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1eb81006f0 0x7f1eb8100b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:27.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:27.530+0000 7f1ebd902640 1 -- 172.21.15.46:0/500531954 >> 172.21.15.46:0/500531954 conn(0x7f1eb80fbc80 msgr2=0x7f1eb80fe0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:09:27.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:27.530+0000 7f1ebd902640 1 -- 172.21.15.46:0/500531954 shutdown_connections 2024-03-31T23:09:27.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:27.530+0000 7f1ebd902640 1 -- 172.21.15.46:0/500531954 wait complete. 2024-03-31T23:09:27.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:27.530+0000 7f1ebd902640 1 Processor -- start 2024-03-31T23:09:27.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:27.530+0000 7f1ebd902640 1 -- start start 2024-03-31T23:09:27.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:27.531+0000 7f1ebd902640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1eb81006f0 0x7f1eb806a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:09:27.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:27.531+0000 7f1ebd902640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1eb81018e0 0x7f1eb819c720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:09:27.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:27.531+0000 7f1ebd902640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1eb81022a0 0x7f1eb819ed40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:09:27.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:27.531+0000 7f1ebd902640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f1eb810c1b0 con 0x7f1eb81022a0 2024-03-31T23:09:27.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:27.531+0000 7f1ebd902640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f1eb810c030 con 0x7f1eb81018e0 2024-03-31T23:09:27.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:27.531+0000 7f1ebd902640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f1eb810c330 con 0x7f1eb81006f0 2024-03-31T23:09:27.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:27.531+0000 7f1eb77fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1eb81022a0 0x7f1eb819ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:09:27.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:27.531+0000 7f1eb77fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1eb81022a0 0x7f1eb819ed40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44916/0 (socket says 172.21.15.46:44916) 2024-03-31T23:09:27.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:27.531+0000 7f1eb77fe640 1 -- 172.21.15.46:0/3849590012 learned_addr learned my addr 172.21.15.46:0/3849590012 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:09:27.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:27.531+0000 7f1eb67fc640 1 --2- 172.21.15.46:0/3849590012 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1eb81018e0 0x7f1eb819c720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:09:27.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:27.531+0000 7f1eb6ffd640 1 --2- 172.21.15.46:0/3849590012 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1eb81006f0 0x7f1eb806a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:09:27.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:27.531+0000 7f1eb77fe640 1 -- 172.21.15.46:0/3849590012 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1eb81006f0 msgr2=0x7f1eb806a8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:09:27.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:27.531+0000 7f1eb77fe640 1 --2- 172.21.15.46:0/3849590012 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1eb81006f0 0x7f1eb806a8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:27.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:27.531+0000 7f1eb77fe640 1 -- 172.21.15.46:0/3849590012 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1eb81018e0 msgr2=0x7f1eb819c720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:09:27.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:27.531+0000 7f1eb77fe640 1 --2- 172.21.15.46:0/3849590012 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1eb81018e0 0x7f1eb819c720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:27.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:27.531+0000 7f1eb77fe640 1 -- 172.21.15.46:0/3849590012 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1eb819f650 con 0x7f1eb81022a0 2024-03-31T23:09:27.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:27.531+0000 7f1eb77fe640 1 --2- 172.21.15.46:0/3849590012 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1eb81022a0 0x7f1eb819ed40 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f1ea800eb10 tx=0x7f1ea800c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:09:27.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:27.531+0000 7f1ebc900640 1 -- 172.21.15.46:0/3849590012 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ea8004b20 con 0x7f1eb81022a0 2024-03-31T23:09:27.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:27.531+0000 7f1ebc900640 1 -- 172.21.15.46:0/3849590012 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1ea8004cc0 con 0x7f1eb81022a0 2024-03-31T23:09:27.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:27.531+0000 7f1ebd902640 1 -- 172.21.15.46:0/3849590012 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1eb819f8e0 con 0x7f1eb81022a0 2024-03-31T23:09:27.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:27.531+0000 7f1ebc900640 1 -- 172.21.15.46:0/3849590012 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ea8010690 con 0x7f1eb81022a0 2024-03-31T23:09:27.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:27.531+0000 7f1ebd902640 1 -- 172.21.15.46:0/3849590012 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1eb819fdb0 con 0x7f1eb81022a0 2024-03-31T23:09:27.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:27.533+0000 7f1ebd902640 1 -- 172.21.15.46:0/3849590012 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1eb8107b30 con 0x7f1eb81022a0 2024-03-31T23:09:27.537 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:27.533+0000 7f1ebc900640 1 -- 172.21.15.46:0/3849590012 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 16) ==== 137320+0+0 (secure 0 0 0) 0x7f1ea8005020 con 0x7f1eb81022a0 2024-03-31T23:09:27.537 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:27.534+0000 7f1ebc900640 1 --2- 172.21.15.46:0/3849590012 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f1ea00a24a0 0x7f1ea00a4960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:09:27.537 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:27.534+0000 7f1eb6ffd640 1 --2- 172.21.15.46:0/3849590012 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f1ea00a24a0 0x7f1ea00a4960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:09:27.537 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:27.534+0000 7f1ebc900640 1 -- 172.21.15.46:0/3849590012 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8802+0+0 (secure 0 0 0) 0x7f1ea80cefa0 con 0x7f1eb81022a0 2024-03-31T23:09:27.537 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:27.534+0000 7f1eb6ffd640 1 --2- 172.21.15.46:0/3849590012 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f1ea00a24a0 0x7f1ea00a4960 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f1eb0005fd0 tx=0x7f1eb0005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:09:27.539 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:27.538+0000 7f1ebc900640 1 -- 172.21.15.46:0/3849590012 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f1ea809bf20 con 0x7f1eb81022a0 2024-03-31T23:09:27.693 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:09:27 smithi142 ceph-mon[35271]: from='client.? 172.21.15.46:0/1095399710' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-31T23:09:27.693 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:09:27 smithi142 ceph-mon[35271]: from='client.? 172.21.15.46:0/1362418804' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-31T23:09:27.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:27.722+0000 7f1ebd902640 1 -- 172.21.15.46:0/3849590012 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f1eb8062350 con 0x7f1eb81022a0 2024-03-31T23:09:27.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:27.724+0000 7f1ebc900640 1 -- 172.21.15.46:0/3849590012 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) ==== 74+0+24105 (secure 0 0 0) 0x7f1ea809b820 con 0x7f1eb81022a0 2024-03-31T23:09:27.725 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-31T23:09:27.725 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":91,"fsid":"6a9d8f08-efb2-11ee-b647-cb9ed24678a4","created":"2024-03-31T23:01:15.463711+0000","modified":"2024-03-31T23:09:17.181787+0000","last_up_change":"2024-03-31T23:08:20.199888+0000","last_in_change":"2024-03-31T23:07:59.438035+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-31T23:04:32.823751+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-31T23:09:09.145711+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-31T23:09:10.148579+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-31T23:09:13.419519+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"91","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"47429095-3f46-4ed7-b7c4-504464719815","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":1824772695},{"type":"v1","addr":"172.21.15.46:6803","nonce":1824772695}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":1824772695},{"type":"v1","addr":"172.21.15.46:6805","nonce":1824772695}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":1824772695},{"type":"v1","addr":"172.21.15.46:6809","nonce":1824772695}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":1824772695},{"type":"v1","addr":"172.21.15.46:6807","nonce":1824772695}]},"public_addr":"172.21.15.46:6803/1824772695","cluster_addr":"172.21.15.46:6805/1824772695","heartbeat_back_addr":"172.21.15.46:6809/1824772695","heartbeat_front_addr":"172.21.15.46:6807/1824772695","state":["exists","up"]},{"osd":1,"uuid":"be56f883-411e-4dbe-b50e-a158134df4a6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6800","nonce":973693922},{"type":"v1","addr":"172.21.15.142:6801","nonce":973693922}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6802","nonce":973693922},{"type":"v1","addr":"172.21.15.142:6803","nonce":973693922}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6806","nonce":973693922},{"type":"v1","addr":"172.21.15.142:6807","nonce":973693922}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6804","nonce":973693922},{"type":"v1","addr":"172.21.15.142:6805","nonce":973693922}]},"public_addr":"172.21.15.142:6801/973693922","cluster_addr":"172.21.15.142:6803/973693922","heartbeat_back_addr":"172.21.15.142:6807/973693922","heartbeat_front_addr":"172.21.15.142:6805/973693922","state":["exists","up"]},{"osd":2,"uuid":"c99a8794-ab19-4260-9a8d-4bc0bd2c8175","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6800","nonce":4084261813},{"type":"v1","addr":"172.21.15.162:6801","nonce":4084261813}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6802","nonce":4084261813},{"type":"v1","addr":"172.21.15.162:6803","nonce":4084261813}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6806","nonce":4084261813},{"type":"v1","addr":"172.21.15.162:6807","nonce":4084261813}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6804","nonce":4084261813},{"type":"v1","addr":"172.21.15.162:6805","nonce":4084261813}]},"public_addr":"172.21.15.162:6801/4084261813","cluster_addr":"172.21.15.162:6803/4084261813","heartbeat_back_addr":"172.21.15.162:6807/4084261813","heartbeat_front_addr":"172.21.15.162:6805/4084261813","state":["exists","up"]},{"osd":3,"uuid":"f877814e-9ef6-4872-a081-5609bb9ebc06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":3496317970},{"type":"v1","addr":"172.21.15.46:6811","nonce":3496317970}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":3496317970},{"type":"v1","addr":"172.21.15.46:6813","nonce":3496317970}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":3496317970},{"type":"v1","addr":"172.21.15.46:6817","nonce":3496317970}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":3496317970},{"type":"v1","addr":"172.21.15.46:6815","nonce":3496317970}]},"public_addr":"172.21.15.46:6811/3496317970","cluster_addr":"172.21.15.46:6813/3496317970","heartbeat_back_addr":"172.21.15.46:6817/3496317970","heartbeat_front_addr":"172.21.15.46:6815/3496317970","state":["exists","up"]},{"osd":4,"uuid":"beecec0a-8280-44fe-8f5f-8498a35eca8f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6808","nonce":2996266929},{"type":"v1","addr":"172.21.15.142:6809","nonce":2996266929}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6810","nonce":2996266929},{"type":"v1","addr":"172.21.15.142:6811","nonce":2996266929}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6814","nonce":2996266929},{"type":"v1","addr":"172.21.15.142:6815","nonce":2996266929}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6812","nonce":2996266929},{"type":"v1","addr":"172.21.15.142:6813","nonce":2996266929}]},"public_addr":"172.21.15.142:6809/2996266929","cluster_addr":"172.21.15.142:6811/2996266929","heartbeat_back_addr":"172.21.15.142:6815/2996266929","heartbeat_front_addr":"172.21.15.142:6813/2996266929","state":["exists","up"]},{"osd":5,"uuid":"9da7ff75-f658-45ea-a02f-1943fdda76b9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6808","nonce":914208293},{"type":"v1","addr":"172.21.15.162:6809","nonce":914208293}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6810","nonce":914208293},{"type":"v1","addr":"172.21.15.162:6811","nonce":914208293}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6814","nonce":914208293},{"type":"v1","addr":"172.21.15.162:6815","nonce":914208293}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6812","nonce":914208293},{"type":"v1","addr":"172.21.15.162:6813","nonce":914208293}]},"public_addr":"172.21.15.162:6809/914208293","cluster_addr":"172.21.15.162:6811/914208293","heartbeat_back_addr":"172.21.15.162:6815/914208293","heartbeat_front_addr":"172.21.15.162:6813/914208293","state":["exists","up"]},{"osd":6,"uuid":"8fa11224-cb26-4d3f-87d1-6a8f31de2a19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":56291966},{"type":"v1","addr":"172.21.15.46:6819","nonce":56291966}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":56291966},{"type":"v1","addr":"172.21.15.46:6821","nonce":56291966}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6824","nonce":56291966},{"type":"v1","addr":"172.21.15.46:6825","nonce":56291966}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":56291966},{"type":"v1","addr":"172.21.15.46:6823","nonce":56291966}]},"public_addr":"172.21.15.46:6819/56291966","cluster_addr":"172.21.15.46:6821/56291966","heartbeat_back_addr":"172.21.15.46:6825/56291966","heartbeat_front_addr":"172.21.15.46:6823/56291966","state":["exists","up"]},{"osd":7,"uuid":"0d0338b7-4d0a-4bb0-b152-8f5224871d9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6816","nonce":2146598446},{"type":"v1","addr":"172.21.15.142:6817","nonce":2146598446}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6818","nonce":2146598446},{"type":"v1","addr":"172.21.15.142:6819","nonce":2146598446}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6822","nonce":2146598446},{"type":"v1","addr":"172.21.15.142:6823","nonce":2146598446}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6820","nonce":2146598446},{"type":"v1","addr":"172.21.15.142:6821","nonce":2146598446}]},"public_addr":"172.21.15.142:6817/2146598446","cluster_addr":"172.21.15.142:6819/2146598446","heartbeat_back_addr":"172.21.15.142:6823/2146598446","heartbeat_front_addr":"172.21.15.142:6821/2146598446","state":["exists","up"]},{"osd":8,"uuid":"8b6f6037-892a-416c-babb-7552648d0b3a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6816","nonce":1173078318},{"type":"v1","addr":"172.21.15.162:6817","nonce":1173078318}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6818","nonce":1173078318},{"type":"v1","addr":"172.21.15.162:6819","nonce":1173078318}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6822","nonce":1173078318},{"type":"v1","addr":"172.21.15.162:6823","nonce":1173078318}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6820","nonce":1173078318},{"type":"v1","addr":"172.21.15.162:6821","nonce":1173078318}]},"public_addr":"172.21.15.162:6817/1173078318","cluster_addr":"172.21.15.162:6819/1173078318","heartbeat_back_addr":"172.21.15.162:6823/1173078318","heartbeat_front_addr":"172.21.15.162:6821/1173078318","state":["exists","up"]},{"osd":9,"uuid":"faccce25-6bf2-4220-8858-cda82b72be68","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":532708881},{"type":"v1","addr":"172.21.15.46:6827","nonce":532708881}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6828","nonce":532708881},{"type":"v1","addr":"172.21.15.46:6829","nonce":532708881}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":532708881},{"type":"v1","addr":"172.21.15.46:6833","nonce":532708881}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":532708881},{"type":"v1","addr":"172.21.15.46:6831","nonce":532708881}]},"public_addr":"172.21.15.46:6827/532708881","cluster_addr":"172.21.15.46:6829/532708881","heartbeat_back_addr":"172.21.15.46:6833/532708881","heartbeat_front_addr":"172.21.15.46:6831/532708881","state":["exists","up"]},{"osd":10,"uuid":"cdd356db-f9eb-4080-95ab-db5d8223e572","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6824","nonce":2845222366},{"type":"v1","addr":"172.21.15.142:6825","nonce":2845222366}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6826","nonce":2845222366},{"type":"v1","addr":"172.21.15.142:6827","nonce":2845222366}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6830","nonce":2845222366},{"type":"v1","addr":"172.21.15.142:6831","nonce":2845222366}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6828","nonce":2845222366},{"type":"v1","addr":"172.21.15.142:6829","nonce":2845222366}]},"public_addr":"172.21.15.142:6825/2845222366","cluster_addr":"172.21.15.142:6827/2845222366","heartbeat_back_addr":"172.21.15.142:6831/2845222366","heartbeat_front_addr":"172.21.15.142:6829/2845222366","state":["exists","up"]},{"osd":11,"uuid":"72104d1c-3ea2-45ad-bc0b-0500f146c06b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6824","nonce":1986323778},{"type":"v1","addr":"172.21.15.162:6825","nonce":1986323778}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6826","nonce":1986323778},{"type":"v1","addr":"172.21.15.162:6827","nonce":1986323778}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6830","nonce":1986323778},{"type":"v1","addr":"172.21.15.162:6831","nonce":1986323778}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6828","nonce":1986323778},{"type":"v1","addr":"172.21.15.162:6829","nonce":1986323778}]},"public_addr":"172.21.15.162:6825/1986323778","cluster_addr":"172.21.15.162:6827/1986323778","heartbeat_back_addr":"172.21.15.162:6831/1986323778","heartbeat_front_addr":"172.21.15.162:6829/1986323778","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:03:44.996331+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:04:07.994738+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:04:30.942255+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:04:55.588884+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:05:19.900600+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:05:43.807442+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:06:09.026976+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:06:34.586784+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:06:59.751605+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:07:26.144495+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:07:52.326909+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:08:17.423042+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.46:0/4021039857":"2024-04-01T23:02:08.493619+0000","172.21.15.46:6801/3681182647":"2024-04-01T23:02:08.493619+0000","172.21.15.46:0/4022214806":"2024-04-01T23:02:08.493619+0000","172.21.15.46:6800/1515528215":"2024-04-01T23:01:41.910162+0000","172.21.15.46:6800/3681182647":"2024-04-01T23:02:08.493619+0000","172.21.15.46:0/143885707":"2024-04-01T23:01:41.910162+0000","172.21.15.46:6801/1515528215":"2024-04-01T23:01:41.910162+0000","172.21.15.46:0/1680389778":"2024-04-01T23:01:41.910162+0000","172.21.15.46:0/1782133966":"2024-04-01T23:01:41.910162+0000","172.21.15.46:0/3925209376":"2024-04-01T23:02:08.493619+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-31T23:09:27.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:27.726+0000 7f1ebd902640 1 -- 172.21.15.46:0/3849590012 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f1ea00a24a0 msgr2=0x7f1ea00a4960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:09:27.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:27.726+0000 7f1ebd902640 1 --2- 172.21.15.46:0/3849590012 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f1ea00a24a0 0x7f1ea00a4960 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f1eb0005fd0 tx=0x7f1eb0005850 comp rx=0 tx=0).stop 2024-03-31T23:09:27.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:27.726+0000 7f1ebd902640 1 -- 172.21.15.46:0/3849590012 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1eb81022a0 msgr2=0x7f1eb819ed40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:09:27.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:27.726+0000 7f1ebd902640 1 --2- 172.21.15.46:0/3849590012 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1eb81022a0 0x7f1eb819ed40 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f1ea800eb10 tx=0x7f1ea800c2d0 comp rx=0 tx=0).stop 2024-03-31T23:09:27.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:27.726+0000 7f1ebd902640 1 -- 172.21.15.46:0/3849590012 shutdown_connections 2024-03-31T23:09:27.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:27.726+0000 7f1ebd902640 1 --2- 172.21.15.46:0/3849590012 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f1ea00a24a0 0x7f1ea00a4960 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:27.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:27.726+0000 7f1ebd902640 1 --2- 172.21.15.46:0/3849590012 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1eb81022a0 0x7f1eb819ed40 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:27.728 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:27.726+0000 7f1ebd902640 1 --2- 172.21.15.46:0/3849590012 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1eb81018e0 0x7f1eb819c720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:27.728 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:27.726+0000 7f1ebd902640 1 --2- 172.21.15.46:0/3849590012 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1eb81006f0 0x7f1eb806a8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:27.728 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:27.726+0000 7f1ebd902640 1 -- 172.21.15.46:0/3849590012 >> 172.21.15.46:0/3849590012 conn(0x7f1eb80fbc80 msgr2=0x7f1eb8104d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:09:27.728 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:27.726+0000 7f1ebd902640 1 -- 172.21.15.46:0/3849590012 shutdown_connections 2024-03-31T23:09:27.728 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:27.726+0000 7f1ebd902640 1 -- 172.21.15.46:0/3849590012 wait complete. 2024-03-31T23:09:27.737 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:09:27 smithi046 ceph-mon[32498]: from='client.? 172.21.15.46:0/1095399710' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-31T23:09:27.737 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:09:27 smithi046 ceph-mon[32498]: from='client.? 172.21.15.46:0/1362418804' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-31T23:09:27.738 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-31T23:09:27.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:09:27 smithi162 ceph-mon[36237]: from='client.? 172.21.15.46:0/1095399710' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-31T23:09:27.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:09:27 smithi162 ceph-mon[36237]: from='client.? 172.21.15.46:0/1362418804' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-31T23:09:27.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:27.844+0000 7fb140fe9640 1 -- 172.21.15.46:0/3430323524 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb13c107450 msgr2=0x7fb13c1098a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:09:27.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:27.844+0000 7fb140fe9640 1 --2- 172.21.15.46:0/3430323524 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb13c107450 0x7fb13c1098a0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fb134009ca0 tx=0x7fb13402c0f0 comp rx=0 tx=0).stop 2024-03-31T23:09:27.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:27.844+0000 7fb140fe9640 1 -- 172.21.15.46:0/3430323524 shutdown_connections 2024-03-31T23:09:27.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:27.844+0000 7fb140fe9640 1 --2- 172.21.15.46:0/3430323524 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb13c107450 0x7fb13c1098a0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:27.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:27.844+0000 7fb140fe9640 1 --2- 172.21.15.46:0/3430323524 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb13c104b20 0x7fb13c106f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:27.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:27.844+0000 7fb140fe9640 1 --2- 172.21.15.46:0/3430323524 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb13c1021f0 0x7fb13c1045e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:27.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:27.844+0000 7fb140fe9640 1 -- 172.21.15.46:0/3430323524 >> 172.21.15.46:0/3430323524 conn(0x7fb13c0fbc00 msgr2=0x7fb13c0fe020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:09:27.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:27.844+0000 7fb140fe9640 1 -- 172.21.15.46:0/3430323524 shutdown_connections 2024-03-31T23:09:27.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:27.844+0000 7fb140fe9640 1 -- 172.21.15.46:0/3430323524 wait complete. 2024-03-31T23:09:27.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:27.845+0000 7fb140fe9640 1 Processor -- start 2024-03-31T23:09:27.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:27.845+0000 7fb140fe9640 1 -- start start 2024-03-31T23:09:27.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:27.845+0000 7fb140fe9640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb13c1021f0 0x7fb13c101b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:09:27.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:27.845+0000 7fb140fe9640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb13c104b20 0x7fb13c1001a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:09:27.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:27.845+0000 7fb140fe9640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb13c107450 0x7fb13c1006e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:09:27.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:27.845+0000 7fb140fe9640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fb13c10d310 con 0x7fb13c107450 2024-03-31T23:09:27.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:27.845+0000 7fb140fe9640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fb13c10d190 con 0x7fb13c1021f0 2024-03-31T23:09:27.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:27.845+0000 7fb140fe9640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fb13c10d490 con 0x7fb13c104b20 2024-03-31T23:09:27.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:27.845+0000 7fb13ad9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb13c107450 0x7fb13c1006e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:09:27.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:27.845+0000 7fb13ad9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb13c107450 0x7fb13c1006e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44930/0 (socket says 172.21.15.46:44930) 2024-03-31T23:09:27.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:27.845+0000 7fb13ad9d640 1 -- 172.21.15.46:0/2831254963 learned_addr learned my addr 172.21.15.46:0/2831254963 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:09:27.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:27.845+0000 7fb13a59c640 1 --2- 172.21.15.46:0/2831254963 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb13c1021f0 0x7fb13c101b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:09:27.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:27.845+0000 7fb13ad9d640 1 -- 172.21.15.46:0/2831254963 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb13c104b20 msgr2=0x7fb13c1001a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:09:27.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:27.845+0000 7fb13ad9d640 1 --2- 172.21.15.46:0/2831254963 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb13c104b20 0x7fb13c1001a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:27.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:27.845+0000 7fb13ad9d640 1 -- 172.21.15.46:0/2831254963 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb13c1021f0 msgr2=0x7fb13c101b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:09:27.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:27.845+0000 7fb13ad9d640 1 --2- 172.21.15.46:0/2831254963 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb13c1021f0 0x7fb13c101b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:27.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:27.845+0000 7fb13ad9d640 1 -- 172.21.15.46:0/2831254963 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb13c100f70 con 0x7fb13c107450 2024-03-31T23:09:27.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:27.845+0000 7fb13ad9d640 1 --2- 172.21.15.46:0/2831254963 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb13c107450 0x7fb13c1006e0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fb13402c700 tx=0x7fb1340328a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:09:27.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:27.845+0000 7fb1337fe640 1 -- 172.21.15.46:0/2831254963 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb134041020 con 0x7fb13c107450 2024-03-31T23:09:27.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:27.845+0000 7fb1337fe640 1 -- 172.21.15.46:0/2831254963 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb134003e20 con 0x7fb13c107450 2024-03-31T23:09:27.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:27.845+0000 7fb140fe9640 1 -- 172.21.15.46:0/2831254963 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb13c1a1a80 con 0x7fb13c107450 2024-03-31T23:09:27.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:27.845+0000 7fb1337fe640 1 -- 172.21.15.46:0/2831254963 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb134039040 con 0x7fb13c107450 2024-03-31T23:09:27.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:27.845+0000 7fb140fe9640 1 -- 172.21.15.46:0/2831254963 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb13c1a2050 con 0x7fb13c107450 2024-03-31T23:09:27.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:27.846+0000 7fb140fe9640 1 -- 172.21.15.46:0/2831254963 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb13c0fd6f0 con 0x7fb13c107450 2024-03-31T23:09:27.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:27.848+0000 7fb1337fe640 1 -- 172.21.15.46:0/2831254963 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 16) ==== 137320+0+0 (secure 0 0 0) 0x7fb134044020 con 0x7fb13c107450 2024-03-31T23:09:27.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:27.848+0000 7fb1337fe640 1 --2- 172.21.15.46:0/2831254963 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fb1280a24a0 0x7fb1280a4960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:09:27.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:27.848+0000 7fb13a59c640 1 --2- 172.21.15.46:0/2831254963 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fb1280a24a0 0x7fb1280a4960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:09:27.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:27.848+0000 7fb1337fe640 1 -- 172.21.15.46:0/2831254963 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8802+0+0 (secure 0 0 0) 0x7fb1340f1020 con 0x7fb13c107450 2024-03-31T23:09:27.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:27.849+0000 7fb13a59c640 1 --2- 172.21.15.46:0/2831254963 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fb1280a24a0 0x7fb1280a4960 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fb12c009990 tx=0x7fb12c009110 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:09:27.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:27.852+0000 7fb1337fe640 1 -- 172.21.15.46:0/2831254963 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fb13404b020 con 0x7fb13c107450 2024-03-31T23:09:28.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:28.072+0000 7fb140fe9640 1 -- 172.21.15.46:0/2831254963 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7fb13c02d170 con 0x7fb13c107450 2024-03-31T23:09:28.402 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:28.401+0000 7fb1337fe640 1 -- 172.21.15.46:0/2831254963 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v18) ==== 118+0+0 (secure 0 0 0) 0x7fb1340bdfa0 con 0x7fb13c107450 2024-03-31T23:09:28.405 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:28.404+0000 7fb140fe9640 1 -- 172.21.15.46:0/2831254963 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fb1280a24a0 msgr2=0x7fb1280a4960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:09:28.405 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:28.404+0000 7fb140fe9640 1 --2- 172.21.15.46:0/2831254963 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fb1280a24a0 0x7fb1280a4960 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fb12c009990 tx=0x7fb12c009110 comp rx=0 tx=0).stop 2024-03-31T23:09:28.405 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:28.404+0000 7fb140fe9640 1 -- 172.21.15.46:0/2831254963 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb13c107450 msgr2=0x7fb13c1006e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:09:28.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:28.404+0000 7fb140fe9640 1 --2- 172.21.15.46:0/2831254963 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb13c107450 0x7fb13c1006e0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fb13402c700 tx=0x7fb1340328a0 comp rx=0 tx=0).stop 2024-03-31T23:09:28.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:28.404+0000 7fb140fe9640 1 -- 172.21.15.46:0/2831254963 shutdown_connections 2024-03-31T23:09:28.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:28.404+0000 7fb140fe9640 1 --2- 172.21.15.46:0/2831254963 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fb1280a24a0 0x7fb1280a4960 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:28.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:28.404+0000 7fb140fe9640 1 --2- 172.21.15.46:0/2831254963 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb13c107450 0x7fb13c1006e0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:28.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:28.404+0000 7fb140fe9640 1 --2- 172.21.15.46:0/2831254963 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb13c104b20 0x7fb13c1001a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:28.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:28.404+0000 7fb140fe9640 1 --2- 172.21.15.46:0/2831254963 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb13c1021f0 0x7fb13c101b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:28.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:28.404+0000 7fb140fe9640 1 -- 172.21.15.46:0/2831254963 >> 172.21.15.46:0/2831254963 conn(0x7fb13c0fbc00 msgr2=0x7fb13c102bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:09:28.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:28.404+0000 7fb140fe9640 1 -- 172.21.15.46:0/2831254963 shutdown_connections 2024-03-31T23:09:28.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:28.404+0000 7fb140fe9640 1 -- 172.21.15.46:0/2831254963 wait complete. 2024-03-31T23:09:28.417 DEBUG:tasks.cephfs.filesystem:fs_config: {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd'], 'max_mds': 3, 'session_timeout': 300, 'standby_replay': True, 'subvols': {'create': 2}} 2024-03-31T23:09:28.418 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 3 2024-03-31T23:09:28.529 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:28.525+0000 7fe5b653e640 1 -- 172.21.15.46:0/2122339102 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe5b0102200 msgr2=0x7fe5b0108a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:09:28.529 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:28.525+0000 7fe5b653e640 1 --2- 172.21.15.46:0/2122339102 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe5b0102200 0x7fe5b0108a90 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fe5a8009ca0 tx=0x7fe5a802c0f0 comp rx=0 tx=0).stop 2024-03-31T23:09:28.529 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:28.525+0000 7fe5b653e640 1 -- 172.21.15.46:0/2122339102 shutdown_connections 2024-03-31T23:09:28.529 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:28.525+0000 7fe5b653e640 1 --2- 172.21.15.46:0/2122339102 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe5b0102200 0x7fe5b0108a90 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:28.529 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:28.525+0000 7fe5b653e640 1 --2- 172.21.15.46:0/2122339102 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe5b0101840 0x7fe5b0101cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:28.529 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:28.525+0000 7fe5b653e640 1 --2- 172.21.15.46:0/2122339102 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe5b0100650 0x7fe5b0100a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:28.529 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:28.525+0000 7fe5b653e640 1 -- 172.21.15.46:0/2122339102 >> 172.21.15.46:0/2122339102 conn(0x7fe5b00fbbe0 msgr2=0x7fe5b00fe000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:09:28.529 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:28.526+0000 7fe5b653e640 1 -- 172.21.15.46:0/2122339102 shutdown_connections 2024-03-31T23:09:28.529 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:28.526+0000 7fe5b653e640 1 -- 172.21.15.46:0/2122339102 wait complete. 2024-03-31T23:09:28.529 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:28.526+0000 7fe5b653e640 1 Processor -- start 2024-03-31T23:09:28.529 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:28.526+0000 7fe5b653e640 1 -- start start 2024-03-31T23:09:28.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:28.526+0000 7fe5b653e640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe5b0100650 0x7fe5b0104590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:09:28.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:28.526+0000 7fe5b653e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe5b0101840 0x7fe5b0102c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:09:28.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:28.526+0000 7fe5b653e640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe5b0102200 0x7fe5b0103160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:09:28.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:28.526+0000 7fe5b653e640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fe5b010c180 con 0x7fe5b0101840 2024-03-31T23:09:28.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:28.526+0000 7fe5b653e640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fe5b010c000 con 0x7fe5b0102200 2024-03-31T23:09:28.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:28.526+0000 7fe5b653e640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fe5b010c300 con 0x7fe5b0100650 2024-03-31T23:09:28.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:28.527+0000 7fe5af7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe5b0101840 0x7fe5b0102c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:09:28.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:28.527+0000 7fe5af7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe5b0101840 0x7fe5b0102c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44958/0 (socket says 172.21.15.46:44958) 2024-03-31T23:09:28.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:28.527+0000 7fe5af7fe640 1 -- 172.21.15.46:0/3140945768 learned_addr learned my addr 172.21.15.46:0/3140945768 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:09:28.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:28.527+0000 7fe5b4adb640 1 --2- 172.21.15.46:0/3140945768 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe5b0102200 0x7fe5b0103160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:09:28.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:28.527+0000 7fe5affff640 1 --2- 172.21.15.46:0/3140945768 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe5b0100650 0x7fe5b0104590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:09:28.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:28.527+0000 7fe5af7fe640 1 -- 172.21.15.46:0/3140945768 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe5b0100650 msgr2=0x7fe5b0104590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:09:28.531 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:28.527+0000 7fe5af7fe640 1 --2- 172.21.15.46:0/3140945768 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe5b0100650 0x7fe5b0104590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:28.531 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:28.527+0000 7fe5af7fe640 1 -- 172.21.15.46:0/3140945768 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe5b0102200 msgr2=0x7fe5b0103160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:09:28.531 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:28.527+0000 7fe5af7fe640 1 --2- 172.21.15.46:0/3140945768 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe5b0102200 0x7fe5b0103160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:28.531 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:28.527+0000 7fe5af7fe640 1 -- 172.21.15.46:0/3140945768 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe5b0103a20 con 0x7fe5b0101840 2024-03-31T23:09:28.531 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:28.527+0000 7fe5b4adb640 1 --2- 172.21.15.46:0/3140945768 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe5b0102200 0x7fe5b0103160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:09:28.531 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:28.527+0000 7fe5af7fe640 1 --2- 172.21.15.46:0/3140945768 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe5b0101840 0x7fe5b0102c20 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fe5a400e9f0 tx=0x7fe5a400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:09:28.531 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:28.527+0000 7fe5ad7fa640 1 -- 172.21.15.46:0/3140945768 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5a400cd80 con 0x7fe5b0101840 2024-03-31T23:09:28.531 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:28.527+0000 7fe5b653e640 1 -- 172.21.15.46:0/3140945768 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe5b01a1b20 con 0x7fe5b0101840 2024-03-31T23:09:28.531 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:28.527+0000 7fe5ad7fa640 1 -- 172.21.15.46:0/3140945768 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe5a40108e0 con 0x7fe5b0101840 2024-03-31T23:09:28.531 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:28.527+0000 7fe5ad7fa640 1 -- 172.21.15.46:0/3140945768 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5a4018920 con 0x7fe5b0101840 2024-03-31T23:09:28.531 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:28.527+0000 7fe5b653e640 1 -- 172.21.15.46:0/3140945768 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe5b01a2090 con 0x7fe5b0101840 2024-03-31T23:09:28.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:28.528+0000 7fe5b653e640 1 -- 172.21.15.46:0/3140945768 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe5b01a2260 con 0x7fe5b0101840 2024-03-31T23:09:28.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:28.529+0000 7fe5ad7fa640 1 -- 172.21.15.46:0/3140945768 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 16) ==== 137320+0+0 (secure 0 0 0) 0x7fe5a4010a90 con 0x7fe5b0101840 2024-03-31T23:09:28.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:28.530+0000 7fe5ad7fa640 1 --2- 172.21.15.46:0/3140945768 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fe5980a24a0 0x7fe5980a4960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:09:28.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:28.530+0000 7fe5ad7fa640 1 -- 172.21.15.46:0/3140945768 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8802+0+0 (secure 0 0 0) 0x7fe5a40cec90 con 0x7fe5b0101840 2024-03-31T23:09:28.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:28.530+0000 7fe5affff640 1 --2- 172.21.15.46:0/3140945768 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fe5980a24a0 0x7fe5980a4960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:09:28.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:28.530+0000 7fe5affff640 1 --2- 172.21.15.46:0/3140945768 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fe5980a24a0 0x7fe5980a4960 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fe5a0005fd0 tx=0x7fe5a000b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:09:28.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:28.534+0000 7fe5ad7fa640 1 -- 172.21.15.46:0/3140945768 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fe5a409b560 con 0x7fe5b0101840 2024-03-31T23:09:28.693 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:09:28 smithi142 ceph-mon[35271]: pgmap v300: 129 pgs: 129 active+clean; 581 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 307 B/s wr, 0 op/s 2024-03-31T23:09:28.693 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:09:28 smithi142 ceph-mon[35271]: from='client.? 172.21.15.46:0/627857365' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-31T23:09:28.693 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:09:28 smithi142 ceph-mon[35271]: from='client.? 172.21.15.46:0/3849590012' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-31T23:09:28.693 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:09:28 smithi142 ceph-mon[35271]: from='client.? 172.21.15.46:0/2831254963' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-03-31T23:09:28.758 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:28.757+0000 7fe5b653e640 1 -- 172.21.15.46:0/3140945768 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"} v 0) -- 0x7fe5b01a2430 con 0x7fe5b0101840 2024-03-31T23:09:28.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:09:28 smithi162 ceph-mon[36237]: pgmap v300: 129 pgs: 129 active+clean; 581 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 307 B/s wr, 0 op/s 2024-03-31T23:09:28.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:09:28 smithi162 ceph-mon[36237]: from='client.? 172.21.15.46:0/627857365' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-31T23:09:28.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:09:28 smithi162 ceph-mon[36237]: from='client.? 172.21.15.46:0/3849590012' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-31T23:09:28.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:09:28 smithi162 ceph-mon[36237]: from='client.? 172.21.15.46:0/2831254963' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-03-31T23:09:28.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:09:28 smithi046 ceph-mon[32498]: pgmap v300: 129 pgs: 129 active+clean; 581 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 307 B/s wr, 0 op/s 2024-03-31T23:09:28.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:09:28 smithi046 ceph-mon[32498]: from='client.? 172.21.15.46:0/627857365' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-31T23:09:28.862 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:09:28 smithi046 ceph-mon[32498]: from='client.? 172.21.15.46:0/3849590012' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-31T23:09:28.862 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:09:28 smithi046 ceph-mon[32498]: from='client.? 172.21.15.46:0/2831254963' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-03-31T23:09:29.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:29.407+0000 7fe5ad7fa640 1 -- 172.21.15.46:0/3140945768 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]=0 v19) ==== 105+0+0 (secure 0 0 0) 0x7fe5a409f600 con 0x7fe5b0101840 2024-03-31T23:09:29.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:29.410+0000 7fe5b653e640 1 -- 172.21.15.46:0/3140945768 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fe5980a24a0 msgr2=0x7fe5980a4960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:09:29.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:29.410+0000 7fe5b653e640 1 --2- 172.21.15.46:0/3140945768 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fe5980a24a0 0x7fe5980a4960 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fe5a0005fd0 tx=0x7fe5a000b390 comp rx=0 tx=0).stop 2024-03-31T23:09:29.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:29.410+0000 7fe5b653e640 1 -- 172.21.15.46:0/3140945768 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe5b0101840 msgr2=0x7fe5b0102c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:09:29.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:29.410+0000 7fe5b653e640 1 --2- 172.21.15.46:0/3140945768 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe5b0101840 0x7fe5b0102c20 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fe5a400e9f0 tx=0x7fe5a400eeb0 comp rx=0 tx=0).stop 2024-03-31T23:09:29.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:29.410+0000 7fe5b653e640 1 -- 172.21.15.46:0/3140945768 shutdown_connections 2024-03-31T23:09:29.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:29.410+0000 7fe5b653e640 1 --2- 172.21.15.46:0/3140945768 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fe5980a24a0 0x7fe5980a4960 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:29.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:29.410+0000 7fe5b653e640 1 --2- 172.21.15.46:0/3140945768 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe5b0102200 0x7fe5b0103160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:29.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:29.410+0000 7fe5b653e640 1 --2- 172.21.15.46:0/3140945768 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe5b0101840 0x7fe5b0102c20 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:29.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:29.410+0000 7fe5b653e640 1 --2- 172.21.15.46:0/3140945768 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe5b0100650 0x7fe5b0104590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:29.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:29.410+0000 7fe5b653e640 1 -- 172.21.15.46:0/3140945768 >> 172.21.15.46:0/3140945768 conn(0x7fe5b00fbbe0 msgr2=0x7fe5b00fd6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:09:29.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:29.410+0000 7fe5b653e640 1 -- 172.21.15.46:0/3140945768 shutdown_connections 2024-03-31T23:09:29.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:29.410+0000 7fe5b653e640 1 -- 172.21.15.46:0/3140945768 wait complete. 2024-03-31T23:09:29.428 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay true 2024-03-31T23:09:29.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:29.539+0000 7fe51c8eb640 1 -- 172.21.15.46:0/586565520 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe518103320 msgr2=0x7fe518109bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:09:29.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:29.539+0000 7fe51c8eb640 1 --2- 172.21.15.46:0/586565520 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe518103320 0x7fe518109bb0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fe510009ca0 tx=0x7fe51002c0f0 comp rx=0 tx=0).stop 2024-03-31T23:09:29.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:29.539+0000 7fe51c8eb640 1 -- 172.21.15.46:0/586565520 shutdown_connections 2024-03-31T23:09:29.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:29.539+0000 7fe51c8eb640 1 --2- 172.21.15.46:0/586565520 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe518103320 0x7fe518109bb0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:29.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:29.539+0000 7fe51c8eb640 1 --2- 172.21.15.46:0/586565520 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe518102960 0x7fe518102de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:29.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:29.539+0000 7fe51c8eb640 1 --2- 172.21.15.46:0/586565520 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe518101770 0x7fe518101b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:29.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:29.539+0000 7fe51c8eb640 1 -- 172.21.15.46:0/586565520 >> 172.21.15.46:0/586565520 conn(0x7fe5180fccc0 msgr2=0x7fe5180ff120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:09:29.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:29.539+0000 7fe51c8eb640 1 -- 172.21.15.46:0/586565520 shutdown_connections 2024-03-31T23:09:29.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:29.539+0000 7fe51c8eb640 1 -- 172.21.15.46:0/586565520 wait complete. 2024-03-31T23:09:29.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:29.540+0000 7fe51c8eb640 1 Processor -- start 2024-03-31T23:09:29.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:29.540+0000 7fe51c8eb640 1 -- start start 2024-03-31T23:09:29.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:29.540+0000 7fe51c8eb640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe518101770 0x7fe5181979a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:09:29.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:29.540+0000 7fe51c8eb640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe518102960 0x7fe518197ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:09:29.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:29.540+0000 7fe51c8eb640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe518103320 0x7fe51819ef50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:09:29.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:29.540+0000 7fe51c8eb640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fe51810d300 con 0x7fe518103320 2024-03-31T23:09:29.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:29.540+0000 7fe51c8eb640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fe51810d180 con 0x7fe518101770 2024-03-31T23:09:29.545 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:29.540+0000 7fe51c8eb640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fe51810d480 con 0x7fe518102960 2024-03-31T23:09:29.545 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:29.540+0000 7fe516d9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe518103320 0x7fe51819ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:09:29.545 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:29.540+0000 7fe516d9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe518103320 0x7fe51819ef50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44970/0 (socket says 172.21.15.46:44970) 2024-03-31T23:09:29.545 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:29.540+0000 7fe516d9d640 1 -- 172.21.15.46:0/4235894080 learned_addr learned my addr 172.21.15.46:0/4235894080 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:09:29.545 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:29.540+0000 7fe51659c640 1 --2- 172.21.15.46:0/4235894080 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe518101770 0x7fe5181979a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:09:29.545 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:29.540+0000 7fe515d9b640 1 --2- 172.21.15.46:0/4235894080 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe518102960 0x7fe518197ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:09:29.545 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:29.540+0000 7fe516d9d640 1 -- 172.21.15.46:0/4235894080 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe518102960 msgr2=0x7fe518197ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:09:29.545 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:29.540+0000 7fe516d9d640 1 --2- 172.21.15.46:0/4235894080 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe518102960 0x7fe518197ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:29.545 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:29.540+0000 7fe516d9d640 1 -- 172.21.15.46:0/4235894080 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe518101770 msgr2=0x7fe5181979a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:09:29.545 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:29.540+0000 7fe516d9d640 1 --2- 172.21.15.46:0/4235894080 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe518101770 0x7fe5181979a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:29.545 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:29.540+0000 7fe516d9d640 1 -- 172.21.15.46:0/4235894080 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe51819f5c0 con 0x7fe518103320 2024-03-31T23:09:29.545 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:29.541+0000 7fe516d9d640 1 --2- 172.21.15.46:0/4235894080 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe518103320 0x7fe51819ef50 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fe51002c700 tx=0x7fe5100328a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:09:29.545 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:29.541+0000 7fe5077fe640 1 -- 172.21.15.46:0/4235894080 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe510041020 con 0x7fe518103320 2024-03-31T23:09:29.545 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:29.541+0000 7fe5077fe640 1 -- 172.21.15.46:0/4235894080 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe510003e20 con 0x7fe518103320 2024-03-31T23:09:29.545 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:29.541+0000 7fe51c8eb640 1 -- 172.21.15.46:0/4235894080 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe51819f850 con 0x7fe518103320 2024-03-31T23:09:29.545 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:29.541+0000 7fe5077fe640 1 -- 172.21.15.46:0/4235894080 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe510039040 con 0x7fe518103320 2024-03-31T23:09:29.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:29.541+0000 7fe51c8eb640 1 -- 172.21.15.46:0/4235894080 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe51819fd40 con 0x7fe518103320 2024-03-31T23:09:29.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:29.542+0000 7fe51c8eb640 1 -- 172.21.15.46:0/4235894080 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe518108a90 con 0x7fe518103320 2024-03-31T23:09:29.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:29.543+0000 7fe5077fe640 1 -- 172.21.15.46:0/4235894080 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 16) ==== 137320+0+0 (secure 0 0 0) 0x7fe510044020 con 0x7fe518103320 2024-03-31T23:09:29.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:29.543+0000 7fe5077fe640 1 --2- 172.21.15.46:0/4235894080 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fe5000a24a0 0x7fe5000a4960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:09:29.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:29.544+0000 7fe51659c640 1 --2- 172.21.15.46:0/4235894080 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fe5000a24a0 0x7fe5000a4960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:09:29.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:29.544+0000 7fe5077fe640 1 -- 172.21.15.46:0/4235894080 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8802+0+0 (secure 0 0 0) 0x7fe5100f0d90 con 0x7fe518103320 2024-03-31T23:09:29.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:29.544+0000 7fe51659c640 1 --2- 172.21.15.46:0/4235894080 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fe5000a24a0 0x7fe5000a4960 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fe508005fd0 tx=0x7fe50800b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:09:29.549 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:29.548+0000 7fe5077fe640 1 -- 172.21.15.46:0/4235894080 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fe51004b020 con 0x7fe518103320 2024-03-31T23:09:29.693 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:09:29 smithi142 ceph-mon[35271]: from='client.? 172.21.15.46:0/2831254963' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-03-31T23:09:29.693 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:09:29 smithi142 ceph-mon[35271]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-31T23:09:29.693 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:09:29 smithi142 ceph-mon[35271]: from='client.? 172.21.15.46:0/3140945768' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2024-03-31T23:09:29.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:29.769+0000 7fe51c8eb640 1 -- 172.21.15.46:0/4235894080 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"} v 0) -- 0x7fe518062350 con 0x7fe518103320 2024-03-31T23:09:29.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:09:29 smithi162 ceph-mon[36237]: from='client.? 172.21.15.46:0/2831254963' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-03-31T23:09:29.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:09:29 smithi162 ceph-mon[36237]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-31T23:09:29.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:09:29 smithi162 ceph-mon[36237]: from='client.? 172.21.15.46:0/3140945768' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2024-03-31T23:09:29.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:09:29 smithi046 ceph-mon[32498]: from='client.? 172.21.15.46:0/2831254963' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-03-31T23:09:29.862 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:09:29 smithi046 ceph-mon[32498]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-31T23:09:29.862 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:09:29 smithi046 ceph-mon[32498]: from='client.? 172.21.15.46:0/3140945768' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2024-03-31T23:09:30.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:30.380+0000 7fe5077fe640 1 -- 172.21.15.46:0/4235894080 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]=0 v21) ==== 121+0+0 (secure 0 0 0) 0x7fe5100bdcb0 con 0x7fe518103320 2024-03-31T23:09:30.384 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:30.383+0000 7fe51c8eb640 1 -- 172.21.15.46:0/4235894080 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fe5000a24a0 msgr2=0x7fe5000a4960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:09:30.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:30.383+0000 7fe51c8eb640 1 --2- 172.21.15.46:0/4235894080 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fe5000a24a0 0x7fe5000a4960 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fe508005fd0 tx=0x7fe50800b410 comp rx=0 tx=0).stop 2024-03-31T23:09:30.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:30.383+0000 7fe51c8eb640 1 -- 172.21.15.46:0/4235894080 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe518103320 msgr2=0x7fe51819ef50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:09:30.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:30.383+0000 7fe51c8eb640 1 --2- 172.21.15.46:0/4235894080 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe518103320 0x7fe51819ef50 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fe51002c700 tx=0x7fe5100328a0 comp rx=0 tx=0).stop 2024-03-31T23:09:30.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:30.383+0000 7fe51c8eb640 1 -- 172.21.15.46:0/4235894080 shutdown_connections 2024-03-31T23:09:30.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:30.383+0000 7fe51c8eb640 1 --2- 172.21.15.46:0/4235894080 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fe5000a24a0 0x7fe5000a4960 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:30.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:30.383+0000 7fe51c8eb640 1 --2- 172.21.15.46:0/4235894080 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe518103320 0x7fe51819ef50 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:30.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:30.383+0000 7fe51c8eb640 1 --2- 172.21.15.46:0/4235894080 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe518102960 0x7fe518197ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:30.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:30.383+0000 7fe51c8eb640 1 --2- 172.21.15.46:0/4235894080 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe518101770 0x7fe5181979a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:30.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:30.383+0000 7fe51c8eb640 1 -- 172.21.15.46:0/4235894080 >> 172.21.15.46:0/4235894080 conn(0x7fe5180fccc0 msgr2=0x7fe518105c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:09:30.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:30.384+0000 7fe51c8eb640 1 -- 172.21.15.46:0/4235894080 shutdown_connections 2024-03-31T23:09:30.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:30.384+0000 7fe51c8eb640 1 -- 172.21.15.46:0/4235894080 wait complete. 2024-03-31T23:09:30.402 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs session_timeout 300 2024-03-31T23:09:30.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:30.513+0000 7f8d147e9640 1 -- 172.21.15.46:0/373498835 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8d0c100670 msgr2=0x7f8d0c100a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:09:30.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:30.513+0000 7f8d147e9640 1 --2- 172.21.15.46:0/373498835 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8d0c100670 0x7f8d0c100a90 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f8d08009900 tx=0x7f8d0802bc30 comp rx=0 tx=0).stop 2024-03-31T23:09:30.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:30.513+0000 7f8d147e9640 1 -- 172.21.15.46:0/373498835 shutdown_connections 2024-03-31T23:09:30.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:30.513+0000 7f8d147e9640 1 --2- 172.21.15.46:0/373498835 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8d0c102220 0x7f8d0c108ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:30.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:30.513+0000 7f8d147e9640 1 --2- 172.21.15.46:0/373498835 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8d0c101860 0x7f8d0c101ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:30.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:30.513+0000 7f8d147e9640 1 --2- 172.21.15.46:0/373498835 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8d0c100670 0x7f8d0c100a90 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:30.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:30.513+0000 7f8d147e9640 1 -- 172.21.15.46:0/373498835 >> 172.21.15.46:0/373498835 conn(0x7f8d0c0fbc00 msgr2=0x7f8d0c0fe020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:09:30.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:30.513+0000 7f8d147e9640 1 -- 172.21.15.46:0/373498835 shutdown_connections 2024-03-31T23:09:30.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:30.513+0000 7f8d147e9640 1 -- 172.21.15.46:0/373498835 wait complete. 2024-03-31T23:09:30.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:30.514+0000 7f8d147e9640 1 Processor -- start 2024-03-31T23:09:30.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:30.514+0000 7f8d147e9640 1 -- start start 2024-03-31T23:09:30.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:30.514+0000 7f8d147e9640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8d0c100670 0x7f8d0c1968f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:09:30.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:30.514+0000 7f8d147e9640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8d0c101860 0x7f8d0c196e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:09:30.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:30.514+0000 7f8d147e9640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8d0c102220 0x7f8d0c19dea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:09:30.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:30.514+0000 7f8d147e9640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f8d0c10c1a0 con 0x7f8d0c101860 2024-03-31T23:09:30.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:30.514+0000 7f8d147e9640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f8d0c10c020 con 0x7f8d0c100670 2024-03-31T23:09:30.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:30.514+0000 7f8d147e9640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f8d0c10c320 con 0x7f8d0c102220 2024-03-31T23:09:30.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:30.514+0000 7f8d11d84640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8d0c101860 0x7f8d0c196e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:09:30.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:30.514+0000 7f8d11d84640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8d0c101860 0x7f8d0c196e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44984/0 (socket says 172.21.15.46:44984) 2024-03-31T23:09:30.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:30.514+0000 7f8d11d84640 1 -- 172.21.15.46:0/3825995404 learned_addr learned my addr 172.21.15.46:0/3825995404 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:09:30.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:30.514+0000 7f8d12585640 1 --2- 172.21.15.46:0/3825995404 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8d0c100670 0x7f8d0c1968f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:09:30.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:30.514+0000 7f8d12d86640 1 --2- 172.21.15.46:0/3825995404 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8d0c102220 0x7f8d0c19dea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:09:30.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:30.514+0000 7f8d11d84640 1 -- 172.21.15.46:0/3825995404 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8d0c102220 msgr2=0x7f8d0c19dea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:09:30.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:30.514+0000 7f8d11d84640 1 --2- 172.21.15.46:0/3825995404 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8d0c102220 0x7f8d0c19dea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:30.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:30.514+0000 7f8d11d84640 1 -- 172.21.15.46:0/3825995404 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8d0c100670 msgr2=0x7f8d0c1968f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:09:30.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:30.514+0000 7f8d11d84640 1 --2- 172.21.15.46:0/3825995404 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8d0c100670 0x7f8d0c1968f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:30.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:30.514+0000 7f8d11d84640 1 -- 172.21.15.46:0/3825995404 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8d0c19e510 con 0x7f8d0c101860 2024-03-31T23:09:30.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:30.515+0000 7f8d11d84640 1 --2- 172.21.15.46:0/3825995404 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8d0c101860 0x7f8d0c196e30 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f8cfc00c970 tx=0x7f8cfc00ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:09:30.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:30.515+0000 7f8d037fe640 1 -- 172.21.15.46:0/3825995404 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8cfc007bb0 con 0x7f8d0c101860 2024-03-31T23:09:30.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:30.515+0000 7f8d037fe640 1 -- 172.21.15.46:0/3825995404 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8cfc007d50 con 0x7f8d0c101860 2024-03-31T23:09:30.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:30.515+0000 7f8d147e9640 1 -- 172.21.15.46:0/3825995404 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8d0c19e800 con 0x7f8d0c101860 2024-03-31T23:09:30.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:30.515+0000 7f8d037fe640 1 -- 172.21.15.46:0/3825995404 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8cfc00f450 con 0x7f8d0c101860 2024-03-31T23:09:30.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:30.515+0000 7f8d147e9640 1 -- 172.21.15.46:0/3825995404 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8d0c19ed00 con 0x7f8d0c101860 2024-03-31T23:09:30.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:30.516+0000 7f8d147e9640 1 -- 172.21.15.46:0/3825995404 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8d0c107ab0 con 0x7f8d0c101860 2024-03-31T23:09:30.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:30.517+0000 7f8d037fe640 1 -- 172.21.15.46:0/3825995404 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 16) ==== 137320+0+0 (secure 0 0 0) 0x7f8cfc004f00 con 0x7f8d0c101860 2024-03-31T23:09:30.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:30.518+0000 7f8d037fe640 1 --2- 172.21.15.46:0/3825995404 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f8cf80a24a0 0x7f8cf80a4960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:09:30.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:30.518+0000 7f8d12585640 1 --2- 172.21.15.46:0/3825995404 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f8cf80a24a0 0x7f8cf80a4960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:09:30.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:30.518+0000 7f8d037fe640 1 -- 172.21.15.46:0/3825995404 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8802+0+0 (secure 0 0 0) 0x7f8cfc0ceab0 con 0x7f8d0c101860 2024-03-31T23:09:30.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:30.519+0000 7f8d12585640 1 --2- 172.21.15.46:0/3825995404 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f8cf80a24a0 0x7f8cf80a4960 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f8d08005ab0 tx=0x7f8d08005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:09:30.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:30.522+0000 7f8d037fe640 1 -- 172.21.15.46:0/3825995404 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f8cfc014030 con 0x7f8d0c101860 2024-03-31T23:09:30.693 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:09:30 smithi142 ceph-mon[35271]: pgmap v301: 129 pgs: 129 active+clean; 581 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 269 B/s wr, 0 op/s 2024-03-31T23:09:30.694 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:09:30 smithi142 ceph-mon[35271]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-03-31T23:09:30.694 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:09:30 smithi142 ceph-mon[35271]: from='client.? 172.21.15.46:0/3140945768' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]': finished 2024-03-31T23:09:30.694 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:09:30 smithi142 ceph-mon[35271]: daemon mds.h assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-03-31T23:09:30.694 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:09:30 smithi142 ceph-mon[35271]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-31T23:09:30.694 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:09:30 smithi142 ceph-mon[35271]: fsmap cephfs:2 {0=b=up:active,1=h=up:creating} 10 up:standby 2024-03-31T23:09:30.694 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:09:30 smithi142 ceph-mon[35271]: daemon mds.h is now active in filesystem cephfs as rank 1 2024-03-31T23:09:30.694 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:09:30 smithi142 ceph-mon[35271]: from='client.? 172.21.15.46:0/4235894080' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-03-31T23:09:30.694 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:09:30 smithi142 ceph-mon[35271]: from='client.? 172.21.15.46:0/4235894080' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-03-31T23:09:30.694 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:09:30 smithi142 ceph-mon[35271]: mds.? [v2:172.21.15.142:6836/2559258399,v1:172.21.15.142:6837/2559258399] up:active 2024-03-31T23:09:30.694 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:09:30 smithi142 ceph-mon[35271]: daemon mds.c assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-03-31T23:09:30.694 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:09:30 smithi142 ceph-mon[35271]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-03-31T23:09:30.694 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:09:30 smithi142 ceph-mon[35271]: Cluster is now healthy 2024-03-31T23:09:30.695 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:09:30 smithi142 ceph-mon[35271]: fsmap cephfs:2 {0=b=up:active,1=h=up:active} 2 up:standby-replay 8 up:standby 2024-03-31T23:09:30.695 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:09:30 smithi142 ceph-mon[35271]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=c=up:creating} 2 up:standby-replay 7 up:standby 2024-03-31T23:09:30.695 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:09:30 smithi142 ceph-mon[35271]: daemon mds.c is now active in filesystem cephfs as rank 2 2024-03-31T23:09:30.744 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:30.743+0000 7f8d147e9640 1 -- 172.21.15.46:0/3825995404 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"} v 0) -- 0x7f8d0c02d170 con 0x7f8d0c101860 2024-03-31T23:09:30.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:09:30 smithi162 ceph-mon[36237]: pgmap v301: 129 pgs: 129 active+clean; 581 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 269 B/s wr, 0 op/s 2024-03-31T23:09:30.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:09:30 smithi162 ceph-mon[36237]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-03-31T23:09:30.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:09:30 smithi162 ceph-mon[36237]: from='client.? 172.21.15.46:0/3140945768' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]': finished 2024-03-31T23:09:30.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:09:30 smithi162 ceph-mon[36237]: daemon mds.h assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-03-31T23:09:30.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:09:30 smithi162 ceph-mon[36237]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-31T23:09:30.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:09:30 smithi162 ceph-mon[36237]: fsmap cephfs:2 {0=b=up:active,1=h=up:creating} 10 up:standby 2024-03-31T23:09:30.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:09:30 smithi162 ceph-mon[36237]: daemon mds.h is now active in filesystem cephfs as rank 1 2024-03-31T23:09:30.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:09:30 smithi162 ceph-mon[36237]: from='client.? 172.21.15.46:0/4235894080' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-03-31T23:09:30.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:09:30 smithi162 ceph-mon[36237]: from='client.? 172.21.15.46:0/4235894080' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-03-31T23:09:30.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:09:30 smithi162 ceph-mon[36237]: mds.? [v2:172.21.15.142:6836/2559258399,v1:172.21.15.142:6837/2559258399] up:active 2024-03-31T23:09:30.797 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:09:30 smithi162 ceph-mon[36237]: daemon mds.c assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-03-31T23:09:30.797 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:09:30 smithi162 ceph-mon[36237]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-03-31T23:09:30.797 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:09:30 smithi162 ceph-mon[36237]: Cluster is now healthy 2024-03-31T23:09:30.797 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:09:30 smithi162 ceph-mon[36237]: fsmap cephfs:2 {0=b=up:active,1=h=up:active} 2 up:standby-replay 8 up:standby 2024-03-31T23:09:30.797 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:09:30 smithi162 ceph-mon[36237]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=c=up:creating} 2 up:standby-replay 7 up:standby 2024-03-31T23:09:30.797 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:09:30 smithi162 ceph-mon[36237]: daemon mds.c is now active in filesystem cephfs as rank 2 2024-03-31T23:09:30.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:09:30 smithi046 ceph-mon[32498]: pgmap v301: 129 pgs: 129 active+clean; 581 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 269 B/s wr, 0 op/s 2024-03-31T23:09:30.862 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:09:30 smithi046 ceph-mon[32498]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-03-31T23:09:30.862 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:09:30 smithi046 ceph-mon[32498]: from='client.? 172.21.15.46:0/3140945768' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]': finished 2024-03-31T23:09:30.862 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:09:30 smithi046 ceph-mon[32498]: daemon mds.h assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-03-31T23:09:30.862 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:09:30 smithi046 ceph-mon[32498]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-31T23:09:30.862 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:09:30 smithi046 ceph-mon[32498]: fsmap cephfs:2 {0=b=up:active,1=h=up:creating} 10 up:standby 2024-03-31T23:09:30.862 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:09:30 smithi046 ceph-mon[32498]: daemon mds.h is now active in filesystem cephfs as rank 1 2024-03-31T23:09:30.862 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:09:30 smithi046 ceph-mon[32498]: from='client.? 172.21.15.46:0/4235894080' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-03-31T23:09:30.862 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:09:30 smithi046 ceph-mon[32498]: from='client.? 172.21.15.46:0/4235894080' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-03-31T23:09:30.862 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:09:30 smithi046 ceph-mon[32498]: mds.? [v2:172.21.15.142:6836/2559258399,v1:172.21.15.142:6837/2559258399] up:active 2024-03-31T23:09:30.862 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:09:30 smithi046 ceph-mon[32498]: daemon mds.c assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-03-31T23:09:30.862 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:09:30 smithi046 ceph-mon[32498]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-03-31T23:09:30.863 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:09:30 smithi046 ceph-mon[32498]: Cluster is now healthy 2024-03-31T23:09:30.863 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:09:30 smithi046 ceph-mon[32498]: fsmap cephfs:2 {0=b=up:active,1=h=up:active} 2 up:standby-replay 8 up:standby 2024-03-31T23:09:30.863 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:09:30 smithi046 ceph-mon[32498]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=c=up:creating} 2 up:standby-replay 7 up:standby 2024-03-31T23:09:30.863 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:09:30 smithi046 ceph-mon[32498]: daemon mds.c is now active in filesystem cephfs as rank 2 2024-03-31T23:09:31.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:31.387+0000 7f8d037fe640 1 -- 172.21.15.46:0/3825995404 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]=0 v23) ==== 115+0+0 (secure 0 0 0) 0x7f8cfc09ba30 con 0x7f8d0c101860 2024-03-31T23:09:31.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:31.390+0000 7f8d147e9640 1 -- 172.21.15.46:0/3825995404 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f8cf80a24a0 msgr2=0x7f8cf80a4960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:09:31.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:31.390+0000 7f8d147e9640 1 --2- 172.21.15.46:0/3825995404 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f8cf80a24a0 0x7f8cf80a4960 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f8d08005ab0 tx=0x7f8d08005fd0 comp rx=0 tx=0).stop 2024-03-31T23:09:31.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:31.390+0000 7f8d147e9640 1 -- 172.21.15.46:0/3825995404 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8d0c101860 msgr2=0x7f8d0c196e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:09:31.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:31.390+0000 7f8d147e9640 1 --2- 172.21.15.46:0/3825995404 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8d0c101860 0x7f8d0c196e30 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f8cfc00c970 tx=0x7f8cfc00ce30 comp rx=0 tx=0).stop 2024-03-31T23:09:31.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:31.390+0000 7f8d147e9640 1 -- 172.21.15.46:0/3825995404 shutdown_connections 2024-03-31T23:09:31.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:31.390+0000 7f8d147e9640 1 --2- 172.21.15.46:0/3825995404 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f8cf80a24a0 0x7f8cf80a4960 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:31.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:31.390+0000 7f8d147e9640 1 --2- 172.21.15.46:0/3825995404 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8d0c102220 0x7f8d0c19dea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:31.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:31.390+0000 7f8d147e9640 1 --2- 172.21.15.46:0/3825995404 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8d0c101860 0x7f8d0c196e30 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:31.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:31.390+0000 7f8d147e9640 1 --2- 172.21.15.46:0/3825995404 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8d0c100670 0x7f8d0c1968f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:31.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:31.390+0000 7f8d147e9640 1 -- 172.21.15.46:0/3825995404 >> 172.21.15.46:0/3825995404 conn(0x7f8d0c0fbc00 msgr2=0x7f8d0c104c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:09:31.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:31.390+0000 7f8d147e9640 1 -- 172.21.15.46:0/3825995404 shutdown_connections 2024-03-31T23:09:31.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:31.390+0000 7f8d147e9640 1 -- 172.21.15.46:0/3825995404 wait complete. 2024-03-31T23:09:31.404 DEBUG:tasks.cephfs.filesystem:Creating {'create': 2} subvols for filesystem 'cephfs' 2024-03-31T23:09:31.405 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolumegroup create cephfs qa 2024-03-31T23:09:31.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:31.514+0000 7f62a58da640 1 -- 172.21.15.46:0/3682538673 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f62a0100670 msgr2=0x7f62a0100a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:09:31.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:31.514+0000 7f62a58da640 1 --2- 172.21.15.46:0/3682538673 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f62a0100670 0x7f62a0100a90 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f6290009900 tx=0x7f629002bc30 comp rx=0 tx=0).stop 2024-03-31T23:09:31.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:31.514+0000 7f62a58da640 1 -- 172.21.15.46:0/3682538673 shutdown_connections 2024-03-31T23:09:31.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:31.514+0000 7f62a58da640 1 --2- 172.21.15.46:0/3682538673 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f62a0102220 0x7f62a0108ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:31.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:31.514+0000 7f62a58da640 1 --2- 172.21.15.46:0/3682538673 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f62a0101860 0x7f62a0101ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:31.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:31.514+0000 7f62a58da640 1 --2- 172.21.15.46:0/3682538673 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f62a0100670 0x7f62a0100a90 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:31.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:31.514+0000 7f62a58da640 1 -- 172.21.15.46:0/3682538673 >> 172.21.15.46:0/3682538673 conn(0x7f62a00fbc20 msgr2=0x7f62a00fe040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:09:31.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:31.514+0000 7f62a58da640 1 -- 172.21.15.46:0/3682538673 shutdown_connections 2024-03-31T23:09:31.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:31.514+0000 7f62a58da640 1 -- 172.21.15.46:0/3682538673 wait complete. 2024-03-31T23:09:31.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:31.514+0000 7f62a58da640 1 Processor -- start 2024-03-31T23:09:31.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:31.515+0000 7f62a58da640 1 -- start start 2024-03-31T23:09:31.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:31.515+0000 7f62a58da640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f62a0100670 0x7f62a0197a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:09:31.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:31.515+0000 7f62a58da640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f62a0101860 0x7f62a0197f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:09:31.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:31.515+0000 7f62a58da640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f62a0102220 0x7f62a019ef80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:09:31.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:31.515+0000 7f62a58da640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f62a010c1a0 con 0x7f62a0101860 2024-03-31T23:09:31.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:31.515+0000 7f62a58da640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f62a010c020 con 0x7f62a0102220 2024-03-31T23:09:31.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:31.515+0000 7f62a58da640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f62a010c320 con 0x7f62a0100670 2024-03-31T23:09:31.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:31.515+0000 7f629e7fc640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f62a0101860 0x7f62a0197f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:09:31.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:31.515+0000 7f629e7fc640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f62a0101860 0x7f62a0197f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:56978/0 (socket says 172.21.15.46:56978) 2024-03-31T23:09:31.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:31.515+0000 7f629e7fc640 1 -- 172.21.15.46:0/2568769624 learned_addr learned my addr 172.21.15.46:0/2568769624 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:09:31.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:31.515+0000 7f629effd640 1 --2- 172.21.15.46:0/2568769624 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f62a0100670 0x7f62a0197a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:09:31.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:31.515+0000 7f629f7fe640 1 --2- 172.21.15.46:0/2568769624 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f62a0102220 0x7f62a019ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:09:31.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:31.515+0000 7f629e7fc640 1 -- 172.21.15.46:0/2568769624 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f62a0100670 msgr2=0x7f62a0197a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:09:31.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:31.515+0000 7f629e7fc640 1 --2- 172.21.15.46:0/2568769624 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f62a0100670 0x7f62a0197a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:31.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:31.515+0000 7f629e7fc640 1 -- 172.21.15.46:0/2568769624 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f62a0102220 msgr2=0x7f62a019ef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:09:31.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:31.515+0000 7f629e7fc640 1 --2- 172.21.15.46:0/2568769624 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f62a0102220 0x7f62a019ef80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:31.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:31.515+0000 7f629e7fc640 1 -- 172.21.15.46:0/2568769624 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f62a019f5f0 con 0x7f62a0101860 2024-03-31T23:09:31.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:31.515+0000 7f629effd640 1 --2- 172.21.15.46:0/2568769624 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f62a0100670 0x7f62a0197a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:09:31.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:31.515+0000 7f629e7fc640 1 --2- 172.21.15.46:0/2568769624 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f62a0101860 0x7f62a0197f60 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f629400e9f0 tx=0x7f629400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:09:31.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:31.516+0000 7f62a48d8640 1 -- 172.21.15.46:0/2568769624 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f629400ccc0 con 0x7f62a0101860 2024-03-31T23:09:31.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:31.516+0000 7f62a48d8640 1 -- 172.21.15.46:0/2568769624 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f629400ce60 con 0x7f62a0101860 2024-03-31T23:09:31.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:31.516+0000 7f62a58da640 1 -- 172.21.15.46:0/2568769624 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f62a019f8e0 con 0x7f62a0101860 2024-03-31T23:09:31.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:31.516+0000 7f62a48d8640 1 -- 172.21.15.46:0/2568769624 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6294010430 con 0x7f62a0101860 2024-03-31T23:09:31.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:31.516+0000 7f62a58da640 1 -- 172.21.15.46:0/2568769624 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f62a019fde0 con 0x7f62a0101860 2024-03-31T23:09:31.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:31.517+0000 7f62a58da640 1 -- 172.21.15.46:0/2568769624 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f62a01a0040 con 0x7f62a0101860 2024-03-31T23:09:31.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:31.518+0000 7f62a48d8640 1 -- 172.21.15.46:0/2568769624 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 16) ==== 137320+0+0 (secure 0 0 0) 0x7f6294005020 con 0x7f62a0101860 2024-03-31T23:09:31.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:31.518+0000 7f62a48d8640 1 --2- 172.21.15.46:0/2568769624 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f62880a24a0 0x7f62880a4960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:09:31.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:31.518+0000 7f629effd640 1 --2- 172.21.15.46:0/2568769624 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f62880a24a0 0x7f62880a4960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:09:31.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:31.518+0000 7f62a48d8640 1 -- 172.21.15.46:0/2568769624 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8802+0+0 (secure 0 0 0) 0x7f62940ced40 con 0x7f62a0101860 2024-03-31T23:09:31.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:31.519+0000 7f629effd640 1 --2- 172.21.15.46:0/2568769624 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f62880a24a0 0x7f62880a4960 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f6290005ab0 tx=0x7f6290005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:09:31.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:31.523+0000 7f62a48d8640 1 -- 172.21.15.46:0/2568769624 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f629409bcc0 con 0x7f62a0101860 2024-03-31T23:09:31.693 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:09:31 smithi142 ceph-mon[35271]: from='client.? 172.21.15.46:0/3825995404' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-03-31T23:09:31.693 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:09:31 smithi142 ceph-mon[35271]: from='client.? 172.21.15.46:0/3825995404' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-03-31T23:09:31.693 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:09:31 smithi142 ceph-mon[35271]: mds.? [v2:172.21.15.162:6836/1371886876,v1:172.21.15.162:6837/1371886876] up:active 2024-03-31T23:09:31.693 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:09:31 smithi142 ceph-mon[35271]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=c=up:active} 2 up:standby-replay 7 up:standby 2024-03-31T23:09:31.693 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:09:31 smithi142 ceph-mon[35271]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-03-31T23:09:31.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:31.732+0000 7f62a58da640 1 -- 172.21.15.46:0/2568769624 --> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7f62a01a0210 con 0x7f62880a24a0 2024-03-31T23:09:31.739 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:09:31 smithi046 ceph-mon[32498]: from='client.? 172.21.15.46:0/3825995404' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-03-31T23:09:31.739 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:09:31 smithi046 ceph-mon[32498]: from='client.? 172.21.15.46:0/3825995404' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-03-31T23:09:31.739 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:09:31 smithi046 ceph-mon[32498]: mds.? [v2:172.21.15.162:6836/1371886876,v1:172.21.15.162:6837/1371886876] up:active 2024-03-31T23:09:31.739 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:09:31 smithi046 ceph-mon[32498]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=c=up:active} 2 up:standby-replay 7 up:standby 2024-03-31T23:09:31.739 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:09:31 smithi046 ceph-mon[32498]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-03-31T23:09:31.759 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:31.759+0000 7f62a48d8640 1 -- 172.21.15.46:0/2568769624 <== mgr.14152 v2:172.21.15.46:6800/162289569 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f62a01a0210 con 0x7f62880a24a0 2024-03-31T23:09:31.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:31.761+0000 7f62a58da640 1 -- 172.21.15.46:0/2568769624 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f62880a24a0 msgr2=0x7f62880a4960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:09:31.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:31.761+0000 7f62a58da640 1 --2- 172.21.15.46:0/2568769624 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f62880a24a0 0x7f62880a4960 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f6290005ab0 tx=0x7f6290005fd0 comp rx=0 tx=0).stop 2024-03-31T23:09:31.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:31.761+0000 7f62a58da640 1 -- 172.21.15.46:0/2568769624 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f62a0101860 msgr2=0x7f62a0197f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:09:31.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:31.761+0000 7f62a58da640 1 --2- 172.21.15.46:0/2568769624 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f62a0101860 0x7f62a0197f60 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f629400e9f0 tx=0x7f629400edf0 comp rx=0 tx=0).stop 2024-03-31T23:09:31.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:31.761+0000 7f62a58da640 1 -- 172.21.15.46:0/2568769624 shutdown_connections 2024-03-31T23:09:31.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:31.761+0000 7f62a58da640 1 --2- 172.21.15.46:0/2568769624 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f62880a24a0 0x7f62880a4960 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:31.763 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:31.761+0000 7f62a58da640 1 --2- 172.21.15.46:0/2568769624 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f62a0102220 0x7f62a019ef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:31.763 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:31.761+0000 7f62a58da640 1 --2- 172.21.15.46:0/2568769624 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f62a0101860 0x7f62a0197f60 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:31.763 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:31.761+0000 7f62a58da640 1 --2- 172.21.15.46:0/2568769624 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f62a0100670 0x7f62a0197a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:31.763 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:31.761+0000 7f62a58da640 1 -- 172.21.15.46:0/2568769624 >> 172.21.15.46:0/2568769624 conn(0x7f62a00fbc20 msgr2=0x7f62a00fd730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:09:31.763 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:31.761+0000 7f62a58da640 1 -- 172.21.15.46:0/2568769624 shutdown_connections 2024-03-31T23:09:31.763 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:31.761+0000 7f62a58da640 1 -- 172.21.15.46:0/2568769624 wait complete. 2024-03-31T23:09:31.773 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv_0 --group_name qa 2024-03-31T23:09:31.795 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:09:31 smithi162 ceph-mon[36237]: from='client.? 172.21.15.46:0/3825995404' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-03-31T23:09:31.795 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:09:31 smithi162 ceph-mon[36237]: from='client.? 172.21.15.46:0/3825995404' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-03-31T23:09:31.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:09:31 smithi162 ceph-mon[36237]: mds.? [v2:172.21.15.162:6836/1371886876,v1:172.21.15.162:6837/1371886876] up:active 2024-03-31T23:09:31.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:09:31 smithi162 ceph-mon[36237]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=c=up:active} 2 up:standby-replay 7 up:standby 2024-03-31T23:09:31.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:09:31 smithi162 ceph-mon[36237]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-03-31T23:09:31.884 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:31.880+0000 7f2aeef19640 1 -- 172.21.15.46:0/2647526627 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2ae8103310 msgr2=0x7f2ae8109ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:09:31.884 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:31.880+0000 7f2aeef19640 1 --2- 172.21.15.46:0/2647526627 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2ae8103310 0x7f2ae8109ba0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f2ae0009ca0 tx=0x7f2ae002c0f0 comp rx=0 tx=0).stop 2024-03-31T23:09:31.884 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:31.880+0000 7f2aeef19640 1 -- 172.21.15.46:0/2647526627 shutdown_connections 2024-03-31T23:09:31.884 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:31.880+0000 7f2aeef19640 1 --2- 172.21.15.46:0/2647526627 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2ae8103310 0x7f2ae8109ba0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:31.884 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:31.880+0000 7f2aeef19640 1 --2- 172.21.15.46:0/2647526627 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2ae8102950 0x7f2ae8102dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:31.884 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:31.880+0000 7f2aeef19640 1 --2- 172.21.15.46:0/2647526627 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2ae8101760 0x7f2ae8101b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:31.884 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:31.880+0000 7f2aeef19640 1 -- 172.21.15.46:0/2647526627 >> 172.21.15.46:0/2647526627 conn(0x7f2ae80fccf0 msgr2=0x7f2ae80ff110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:09:31.884 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:31.880+0000 7f2aeef19640 1 -- 172.21.15.46:0/2647526627 shutdown_connections 2024-03-31T23:09:31.884 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:31.880+0000 7f2aeef19640 1 -- 172.21.15.46:0/2647526627 wait complete. 2024-03-31T23:09:31.884 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:31.881+0000 7f2aeef19640 1 Processor -- start 2024-03-31T23:09:31.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:31.881+0000 7f2aeef19640 1 -- start start 2024-03-31T23:09:31.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:31.881+0000 7f2aeef19640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2ae8101760 0x7f2ae8075500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:09:31.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:31.881+0000 7f2aeef19640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2ae8102950 0x7f2ae8073b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:09:31.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:31.881+0000 7f2aeef19640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2ae8103310 0x7f2ae8074080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:09:31.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:31.881+0000 7f2aeef19640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f2ae810d290 con 0x7f2ae8102950 2024-03-31T23:09:31.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:31.881+0000 7f2aeef19640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f2ae810d110 con 0x7f2ae8101760 2024-03-31T23:09:31.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:31.881+0000 7f2aeef19640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f2ae810d410 con 0x7f2ae8103310 2024-03-31T23:09:31.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:31.881+0000 7f2ae7fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2ae8102950 0x7f2ae8073b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:09:31.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:31.881+0000 7f2ae7fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2ae8102950 0x7f2ae8073b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:57002/0 (socket says 172.21.15.46:57002) 2024-03-31T23:09:31.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:31.881+0000 7f2ae7fff640 1 -- 172.21.15.46:0/389787559 learned_addr learned my addr 172.21.15.46:0/389787559 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:09:31.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:31.881+0000 7f2aeccb5640 1 --2- 172.21.15.46:0/389787559 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2ae8101760 0x7f2ae8075500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:09:31.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:31.881+0000 7f2aed4b6640 1 --2- 172.21.15.46:0/389787559 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2ae8103310 0x7f2ae8074080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:09:31.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:31.881+0000 7f2ae7fff640 1 -- 172.21.15.46:0/389787559 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2ae8103310 msgr2=0x7f2ae8074080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:09:31.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:31.881+0000 7f2ae7fff640 1 --2- 172.21.15.46:0/389787559 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2ae8103310 0x7f2ae8074080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:31.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:31.881+0000 7f2ae7fff640 1 -- 172.21.15.46:0/389787559 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2ae8101760 msgr2=0x7f2ae8075500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:09:31.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:31.881+0000 7f2ae7fff640 1 --2- 172.21.15.46:0/389787559 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2ae8101760 0x7f2ae8075500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:31.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:31.881+0000 7f2ae7fff640 1 -- 172.21.15.46:0/389787559 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ae8074910 con 0x7f2ae8102950 2024-03-31T23:09:31.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:31.881+0000 7f2ae7fff640 1 --2- 172.21.15.46:0/389787559 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2ae8102950 0x7f2ae8073b40 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f2ad400e9f0 tx=0x7f2ad400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:09:31.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:31.881+0000 7f2ae5ffb640 1 -- 172.21.15.46:0/389787559 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ad400cd80 con 0x7f2ae8102950 2024-03-31T23:09:31.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:31.882+0000 7f2aeef19640 1 -- 172.21.15.46:0/389787559 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2ae81a1a80 con 0x7f2ae8102950 2024-03-31T23:09:31.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:31.882+0000 7f2ae5ffb640 1 -- 172.21.15.46:0/389787559 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f2ad40108e0 con 0x7f2ae8102950 2024-03-31T23:09:31.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:31.882+0000 7f2ae5ffb640 1 -- 172.21.15.46:0/389787559 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ad4018920 con 0x7f2ae8102950 2024-03-31T23:09:31.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:31.882+0000 7f2aeef19640 1 -- 172.21.15.46:0/389787559 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2ae81a1fd0 con 0x7f2ae8102950 2024-03-31T23:09:31.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:31.883+0000 7f2aeef19640 1 -- 172.21.15.46:0/389787559 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2ae8108ba0 con 0x7f2ae8102950 2024-03-31T23:09:31.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:31.884+0000 7f2ae5ffb640 1 -- 172.21.15.46:0/389787559 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 16) ==== 137320+0+0 (secure 0 0 0) 0x7f2ad4010a90 con 0x7f2ae8102950 2024-03-31T23:09:31.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:31.885+0000 7f2ae5ffb640 1 --2- 172.21.15.46:0/389787559 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f2ad80a24a0 0x7f2ad80a4960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:09:31.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:31.885+0000 7f2aeccb5640 1 --2- 172.21.15.46:0/389787559 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f2ad80a24a0 0x7f2ad80a4960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:09:31.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:31.885+0000 7f2ae5ffb640 1 -- 172.21.15.46:0/389787559 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8802+0+0 (secure 0 0 0) 0x7f2ad40cec90 con 0x7f2ae8102950 2024-03-31T23:09:31.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:31.885+0000 7f2aeccb5640 1 --2- 172.21.15.46:0/389787559 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f2ad80a24a0 0x7f2ad80a4960 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f2ae81027b0 tx=0x7f2adc005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:09:31.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:31.889+0000 7f2ae5ffb640 1 -- 172.21.15.46:0/389787559 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f2ad409b560 con 0x7f2ae8102950 2024-03-31T23:09:32.101 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:32.100+0000 7f2aeef19640 1 -- 172.21.15.46:0/389787559 --> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7f2ae8061b90 con 0x7f2ad80a24a0 2024-03-31T23:09:32.111 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 31 23:09:31 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mgr-x[32727]: 2024-03-31T23:09:31.737+0000 7fbe12415700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-31T23:09:32.111 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 31 23:09:31 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mgr-x[32727]: 2024-03-31T23:09:31.737+0000 7fbe12415700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-31T23:09:32.112 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 31 23:09:31 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mgr-x[32727]: 2024-03-31T23:09:31.737+0000 7fbe12415700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-31T23:09:32.112 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 31 23:09:31 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mgr-x[32727]: 2024-03-31T23:09:31.737+0000 7fbe12415700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-31T23:09:32.112 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 31 23:09:31 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mgr-x[32727]: 2024-03-31T23:09:31.737+0000 7fbe12415700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-31T23:09:32.124 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:32.123+0000 7f2ae5ffb640 1 -- 172.21.15.46:0/389787559 <== mgr.14152 v2:172.21.15.46:6800/162289569 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f2ae8061b90 con 0x7f2ad80a24a0 2024-03-31T23:09:32.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:32.125+0000 7f2aeef19640 1 -- 172.21.15.46:0/389787559 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f2ad80a24a0 msgr2=0x7f2ad80a4960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:09:32.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:32.125+0000 7f2aeef19640 1 --2- 172.21.15.46:0/389787559 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f2ad80a24a0 0x7f2ad80a4960 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f2ae81027b0 tx=0x7f2adc005f90 comp rx=0 tx=0).stop 2024-03-31T23:09:32.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:32.125+0000 7f2aeef19640 1 -- 172.21.15.46:0/389787559 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2ae8102950 msgr2=0x7f2ae8073b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:09:32.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:32.125+0000 7f2aeef19640 1 --2- 172.21.15.46:0/389787559 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2ae8102950 0x7f2ae8073b40 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f2ad400e9f0 tx=0x7f2ad400eeb0 comp rx=0 tx=0).stop 2024-03-31T23:09:32.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:32.125+0000 7f2aeef19640 1 -- 172.21.15.46:0/389787559 shutdown_connections 2024-03-31T23:09:32.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:32.125+0000 7f2aeef19640 1 --2- 172.21.15.46:0/389787559 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f2ad80a24a0 0x7f2ad80a4960 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:32.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:32.125+0000 7f2aeef19640 1 --2- 172.21.15.46:0/389787559 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2ae8103310 0x7f2ae8074080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:32.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:32.125+0000 7f2aeef19640 1 --2- 172.21.15.46:0/389787559 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2ae8102950 0x7f2ae8073b40 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:32.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:32.125+0000 7f2aeef19640 1 --2- 172.21.15.46:0/389787559 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2ae8101760 0x7f2ae8075500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:32.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:32.125+0000 7f2aeef19640 1 -- 172.21.15.46:0/389787559 >> 172.21.15.46:0/389787559 conn(0x7f2ae80fccf0 msgr2=0x7f2ae8105d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:09:32.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:32.125+0000 7f2aeef19640 1 -- 172.21.15.46:0/389787559 shutdown_connections 2024-03-31T23:09:32.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:32.125+0000 7f2aeef19640 1 -- 172.21.15.46:0/389787559 wait complete. 2024-03-31T23:09:32.137 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_0 2024-03-31T23:09:32.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:32.244+0000 7f075ec0f640 1 -- 172.21.15.46:0/1189022539 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0758102260 msgr2=0x7f0758104650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:09:32.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:32.244+0000 7f075ec0f640 1 --2- 172.21.15.46:0/1189022539 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0758102260 0x7f0758104650 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f074c009900 tx=0x7f074c02bc30 comp rx=0 tx=0).stop 2024-03-31T23:09:32.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:32.244+0000 7f075ec0f640 1 -- 172.21.15.46:0/1189022539 shutdown_connections 2024-03-31T23:09:32.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:32.244+0000 7f075ec0f640 1 --2- 172.21.15.46:0/1189022539 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f07581074c0 0x7f07581098b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:32.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:32.244+0000 7f075ec0f640 1 --2- 172.21.15.46:0/1189022539 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0758104b90 0x7f0758106f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:32.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:32.244+0000 7f075ec0f640 1 --2- 172.21.15.46:0/1189022539 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0758102260 0x7f0758104650 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:32.248 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:32.244+0000 7f075ec0f640 1 -- 172.21.15.46:0/1189022539 >> 172.21.15.46:0/1189022539 conn(0x7f07580fbbf0 msgr2=0x7f07580fe010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:09:32.248 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:32.244+0000 7f075ec0f640 1 -- 172.21.15.46:0/1189022539 shutdown_connections 2024-03-31T23:09:32.248 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:32.244+0000 7f075ec0f640 1 -- 172.21.15.46:0/1189022539 wait complete. 2024-03-31T23:09:32.248 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:32.244+0000 7f075ec0f640 1 Processor -- start 2024-03-31T23:09:32.248 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:32.244+0000 7f075ec0f640 1 -- start start 2024-03-31T23:09:32.248 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:32.245+0000 7f075ec0f640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0758102260 0x7f0758197a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:09:32.248 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:32.245+0000 7f075ec0f640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0758104b90 0x7f0758197f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:09:32.248 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:32.245+0000 7f075ec0f640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f07581074c0 0x7f075819efd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:09:32.248 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:32.245+0000 7f075ec0f640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f075810d2b0 con 0x7f07581074c0 2024-03-31T23:09:32.248 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:32.245+0000 7f075ec0f640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f075810d130 con 0x7f0758104b90 2024-03-31T23:09:32.248 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:32.245+0000 7f075ec0f640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f075810d430 con 0x7f0758102260 2024-03-31T23:09:32.248 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:32.245+0000 7f075d1ac640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f07581074c0 0x7f075819efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:09:32.248 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:32.245+0000 7f075c9ab640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0758102260 0x7f0758197a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:09:32.248 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:32.245+0000 7f075d1ac640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f07581074c0 0x7f075819efd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:57008/0 (socket says 172.21.15.46:57008) 2024-03-31T23:09:32.249 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:32.245+0000 7f075d1ac640 1 -- 172.21.15.46:0/2361753018 learned_addr learned my addr 172.21.15.46:0/2361753018 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:09:32.249 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:32.245+0000 7f0757fff640 1 --2- 172.21.15.46:0/2361753018 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0758104b90 0x7f0758197f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:09:32.249 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:32.245+0000 7f075d1ac640 1 -- 172.21.15.46:0/2361753018 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0758102260 msgr2=0x7f0758197a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:09:32.249 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:32.245+0000 7f075d1ac640 1 --2- 172.21.15.46:0/2361753018 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0758102260 0x7f0758197a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:32.249 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:32.245+0000 7f075d1ac640 1 -- 172.21.15.46:0/2361753018 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0758104b90 msgr2=0x7f0758197f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:09:32.249 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:32.245+0000 7f075d1ac640 1 --2- 172.21.15.46:0/2361753018 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0758104b90 0x7f0758197f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:32.249 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:32.245+0000 7f075d1ac640 1 -- 172.21.15.46:0/2361753018 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f075819f640 con 0x7f07581074c0 2024-03-31T23:09:32.249 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:32.245+0000 7f075d1ac640 1 --2- 172.21.15.46:0/2361753018 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f07581074c0 0x7f075819efd0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f075000eb10 tx=0x7f075000c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:09:32.249 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:32.245+0000 7f0755ffb640 1 -- 172.21.15.46:0/2361753018 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0750004b20 con 0x7f07581074c0 2024-03-31T23:09:32.249 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:32.245+0000 7f0755ffb640 1 -- 172.21.15.46:0/2361753018 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0750004cc0 con 0x7f07581074c0 2024-03-31T23:09:32.249 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:32.245+0000 7f075ec0f640 1 -- 172.21.15.46:0/2361753018 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f075819f930 con 0x7f07581074c0 2024-03-31T23:09:32.249 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:32.245+0000 7f0755ffb640 1 -- 172.21.15.46:0/2361753018 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0750010690 con 0x7f07581074c0 2024-03-31T23:09:32.249 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:32.246+0000 7f075ec0f640 1 -- 172.21.15.46:0/2361753018 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f075819fe80 con 0x7f07581074c0 2024-03-31T23:09:32.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:32.247+0000 7f075ec0f640 1 -- 172.21.15.46:0/2361753018 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f07580fd6e0 con 0x7f07581074c0 2024-03-31T23:09:32.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:32.248+0000 7f0755ffb640 1 -- 172.21.15.46:0/2361753018 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 16) ==== 137320+0+0 (secure 0 0 0) 0x7f0750005020 con 0x7f07581074c0 2024-03-31T23:09:32.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:32.248+0000 7f0755ffb640 1 --2- 172.21.15.46:0/2361753018 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f07480a2520 0x7f07480a49e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:09:32.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:32.248+0000 7f0755ffb640 1 -- 172.21.15.46:0/2361753018 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8802+0+0 (secure 0 0 0) 0x7f07500cee70 con 0x7f07581074c0 2024-03-31T23:09:32.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:32.248+0000 7f075c9ab640 1 --2- 172.21.15.46:0/2361753018 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f07480a2520 0x7f07480a49e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:09:32.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:32.249+0000 7f075c9ab640 1 --2- 172.21.15.46:0/2361753018 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f07480a2520 0x7f07480a49e0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f074c009230 tx=0x7f074c005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:09:32.254 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:32.253+0000 7f0755ffb640 1 -- 172.21.15.46:0/2361753018 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f075009bdf0 con 0x7f07581074c0 2024-03-31T23:09:32.466 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:32.465+0000 7f075ec0f640 1 -- 172.21.15.46:0/2361753018 --> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}) -- 0x7f0758061b90 con 0x7f07480a2520 2024-03-31T23:09:32.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:32.470+0000 7f0755ffb640 1 -- 172.21.15.46:0/2361753018 <== mgr.14152 v2:172.21.15.46:6800/162289569 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+54 (secure 0 0 0) 0x7f0758061b90 con 0x7f07480a2520 2024-03-31T23:09:32.471 INFO:teuthology.orchestra.run.smithi046.stdout:/volumes/qa/sv_0/e990713a-13d9-4c54-877d-f0f60b19e88e 2024-03-31T23:09:32.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:32.472+0000 7f075ec0f640 1 -- 172.21.15.46:0/2361753018 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f07480a2520 msgr2=0x7f07480a49e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:09:32.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:32.472+0000 7f075ec0f640 1 --2- 172.21.15.46:0/2361753018 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f07480a2520 0x7f07480a49e0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f074c009230 tx=0x7f074c005fd0 comp rx=0 tx=0).stop 2024-03-31T23:09:32.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:32.472+0000 7f075ec0f640 1 -- 172.21.15.46:0/2361753018 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f07581074c0 msgr2=0x7f075819efd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:09:32.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:32.472+0000 7f075ec0f640 1 --2- 172.21.15.46:0/2361753018 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f07581074c0 0x7f075819efd0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f075000eb10 tx=0x7f075000c2d0 comp rx=0 tx=0).stop 2024-03-31T23:09:32.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:32.472+0000 7f075ec0f640 1 -- 172.21.15.46:0/2361753018 shutdown_connections 2024-03-31T23:09:32.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:32.472+0000 7f075ec0f640 1 --2- 172.21.15.46:0/2361753018 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f07480a2520 0x7f07480a49e0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:32.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:32.472+0000 7f075ec0f640 1 --2- 172.21.15.46:0/2361753018 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f07581074c0 0x7f075819efd0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:32.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:32.472+0000 7f075ec0f640 1 --2- 172.21.15.46:0/2361753018 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0758104b90 0x7f0758197f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:32.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:32.472+0000 7f075ec0f640 1 --2- 172.21.15.46:0/2361753018 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0758102260 0x7f0758197a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:32.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:32.472+0000 7f075ec0f640 1 -- 172.21.15.46:0/2361753018 >> 172.21.15.46:0/2361753018 conn(0x7f07580fbbf0 msgr2=0x7f0758102da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:09:32.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:32.472+0000 7f075ec0f640 1 -- 172.21.15.46:0/2361753018 shutdown_connections 2024-03-31T23:09:32.475 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:32.472+0000 7f075ec0f640 1 -- 172.21.15.46:0/2361753018 wait complete. 2024-03-31T23:09:32.484 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv_1 --group_name qa 2024-03-31T23:09:32.592 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:32.590+0000 7f04d9ec0640 1 -- 172.21.15.46:0/1950666550 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f04d4100650 msgr2=0x7f04d4100a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:09:32.592 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:32.590+0000 7f04d9ec0640 1 --2- 172.21.15.46:0/1950666550 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f04d4100650 0x7f04d4100a70 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f04c4009900 tx=0x7f04c402bc30 comp rx=0 tx=0).stop 2024-03-31T23:09:32.592 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:32.591+0000 7f04d9ec0640 1 -- 172.21.15.46:0/1950666550 shutdown_connections 2024-03-31T23:09:32.592 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:32.591+0000 7f04d9ec0640 1 --2- 172.21.15.46:0/1950666550 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f04d4102200 0x7f04d4108a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:32.592 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:32.591+0000 7f04d9ec0640 1 --2- 172.21.15.46:0/1950666550 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f04d4101840 0x7f04d4101cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:32.592 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:32.591+0000 7f04d9ec0640 1 --2- 172.21.15.46:0/1950666550 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f04d4100650 0x7f04d4100a70 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:32.592 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:32.591+0000 7f04d9ec0640 1 -- 172.21.15.46:0/1950666550 >> 172.21.15.46:0/1950666550 conn(0x7f04d40fbbe0 msgr2=0x7f04d40fe000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:09:32.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:32.591+0000 7f04d9ec0640 1 -- 172.21.15.46:0/1950666550 shutdown_connections 2024-03-31T23:09:32.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:32.591+0000 7f04d9ec0640 1 -- 172.21.15.46:0/1950666550 wait complete. 2024-03-31T23:09:32.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:32.591+0000 7f04d9ec0640 1 Processor -- start 2024-03-31T23:09:32.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:32.591+0000 7f04d9ec0640 1 -- start start 2024-03-31T23:09:32.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:32.591+0000 7f04d9ec0640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f04d4100650 0x7f04d4197a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:09:32.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:32.592+0000 7f04d9ec0640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f04d4101840 0x7f04d4197f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:09:32.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:32.592+0000 7f04d9ec0640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f04d4102200 0x7f04d419eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:09:32.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:32.592+0000 7f04d9ec0640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f04d410c180 con 0x7f04d4100650 2024-03-31T23:09:32.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:32.592+0000 7f04d9ec0640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f04d410c000 con 0x7f04d4102200 2024-03-31T23:09:32.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:32.592+0000 7f04d9ec0640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f04d410c300 con 0x7f04d4101840 2024-03-31T23:09:32.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:32.592+0000 7f04d37fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f04d4100650 0x7f04d4197a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:09:32.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:32.592+0000 7f04d37fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f04d4100650 0x7f04d4197a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:57030/0 (socket says 172.21.15.46:57030) 2024-03-31T23:09:32.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:32.592+0000 7f04d37fe640 1 -- 172.21.15.46:0/1446854365 learned_addr learned my addr 172.21.15.46:0/1446854365 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:09:32.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:32.592+0000 7f04d2ffd640 1 --2- 172.21.15.46:0/1446854365 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f04d4101840 0x7f04d4197f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:09:32.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:32.592+0000 7f04d3fff640 1 --2- 172.21.15.46:0/1446854365 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f04d4102200 0x7f04d419eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:09:32.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:32.592+0000 7f04d37fe640 1 -- 172.21.15.46:0/1446854365 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f04d4101840 msgr2=0x7f04d4197f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:09:32.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:32.592+0000 7f04d37fe640 1 --2- 172.21.15.46:0/1446854365 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f04d4101840 0x7f04d4197f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:32.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:32.592+0000 7f04d37fe640 1 -- 172.21.15.46:0/1446854365 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f04d4102200 msgr2=0x7f04d419eff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:09:32.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:32.592+0000 7f04d37fe640 1 --2- 172.21.15.46:0/1446854365 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f04d4102200 0x7f04d419eff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:32.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:32.592+0000 7f04d37fe640 1 -- 172.21.15.46:0/1446854365 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f04d419f660 con 0x7f04d4100650 2024-03-31T23:09:32.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:32.592+0000 7f04d3fff640 1 --2- 172.21.15.46:0/1446854365 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f04d4102200 0x7f04d419eff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T23:09:32.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:32.592+0000 7f04d37fe640 1 --2- 172.21.15.46:0/1446854365 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f04d4100650 0x7f04d4197a40 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f04c4000c00 tx=0x7f04c4005dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:09:32.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:32.592+0000 7f04d0ff9640 1 -- 172.21.15.46:0/1446854365 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04c4043020 con 0x7f04d4100650 2024-03-31T23:09:32.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:32.592+0000 7f04d0ff9640 1 -- 172.21.15.46:0/1446854365 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f04c4004350 con 0x7f04d4100650 2024-03-31T23:09:32.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:32.592+0000 7f04d9ec0640 1 -- 172.21.15.46:0/1446854365 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f04d419f8f0 con 0x7f04d4100650 2024-03-31T23:09:32.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:32.592+0000 7f04d0ff9640 1 -- 172.21.15.46:0/1446854365 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04c4003b80 con 0x7f04d4100650 2024-03-31T23:09:32.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:32.592+0000 7f04d9ec0640 1 -- 172.21.15.46:0/1446854365 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f04d419fe40 con 0x7f04d4100650 2024-03-31T23:09:32.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:32.593+0000 7f04d9ec0640 1 -- 172.21.15.46:0/1446854365 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f04d4068540 con 0x7f04d4100650 2024-03-31T23:09:32.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:32.595+0000 7f04d0ff9640 1 -- 172.21.15.46:0/1446854365 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 16) ==== 137320+0+0 (secure 0 0 0) 0x7f04c4046020 con 0x7f04d4100650 2024-03-31T23:09:32.598 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:32.596+0000 7f04d0ff9640 1 --2- 172.21.15.46:0/1446854365 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f04bc0a24a0 0x7f04bc0a4960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:09:32.598 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:32.596+0000 7f04d2ffd640 1 --2- 172.21.15.46:0/1446854365 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f04bc0a24a0 0x7f04bc0a4960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:09:32.598 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:32.596+0000 7f04d0ff9640 1 -- 172.21.15.46:0/1446854365 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8802+0+0 (secure 0 0 0) 0x7f04c40f0fc0 con 0x7f04d4100650 2024-03-31T23:09:32.598 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:32.596+0000 7f04d2ffd640 1 --2- 172.21.15.46:0/1446854365 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f04bc0a24a0 0x7f04bc0a4960 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f04d4198f50 tx=0x7f04c8005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:09:32.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:32.599+0000 7f04d0ff9640 1 -- 172.21.15.46:0/1446854365 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f04c404d020 con 0x7f04d4100650 2024-03-31T23:09:32.693 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:09:32 smithi142 ceph-mon[35271]: pgmap v302: 129 pgs: 129 active+clean; 581 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 511 B/s wr, 0 op/s 2024-03-31T23:09:32.693 INFO:journalctl@ceph.mds.b.smithi142.stdout:Mar 31 23:09:32 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mds-b[65801]: 2024-03-31T23:09:32.486+0000 7fb486a60700 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-03-31T23:09:32.694 INFO:journalctl@ceph.mds.b.smithi142.stdout:Mar 31 23:09:32 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mds-b[65801]: 2024-03-31T23:09:32.486+0000 7fb486a60700 -1 mds.pinger is_rank_lagging: rank=2 was never sent ping request. 2024-03-31T23:09:32.795 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:09:32 smithi162 ceph-mon[36237]: pgmap v302: 129 pgs: 129 active+clean; 581 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 511 B/s wr, 0 op/s 2024-03-31T23:09:32.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:32.808+0000 7f04d9ec0640 1 -- 172.21.15.46:0/1446854365 --> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7f04d4069070 con 0x7f04bc0a24a0 2024-03-31T23:09:32.836 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:32.835+0000 7f04d0ff9640 1 -- 172.21.15.46:0/1446854365 <== mgr.14152 v2:172.21.15.46:6800/162289569 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f04d4069070 con 0x7f04bc0a24a0 2024-03-31T23:09:32.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:32.837+0000 7f04d9ec0640 1 -- 172.21.15.46:0/1446854365 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f04bc0a24a0 msgr2=0x7f04bc0a4960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:09:32.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:32.837+0000 7f04d9ec0640 1 --2- 172.21.15.46:0/1446854365 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f04bc0a24a0 0x7f04bc0a4960 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f04d4198f50 tx=0x7f04c8005f90 comp rx=0 tx=0).stop 2024-03-31T23:09:32.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:32.837+0000 7f04d9ec0640 1 -- 172.21.15.46:0/1446854365 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f04d4100650 msgr2=0x7f04d4197a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:09:32.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:32.837+0000 7f04d9ec0640 1 --2- 172.21.15.46:0/1446854365 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f04d4100650 0x7f04d4197a40 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f04c4000c00 tx=0x7f04c4005dc0 comp rx=0 tx=0).stop 2024-03-31T23:09:32.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:32.837+0000 7f04d9ec0640 1 -- 172.21.15.46:0/1446854365 shutdown_connections 2024-03-31T23:09:32.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:32.837+0000 7f04d9ec0640 1 --2- 172.21.15.46:0/1446854365 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f04bc0a24a0 0x7f04bc0a4960 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:32.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:32.837+0000 7f04d9ec0640 1 --2- 172.21.15.46:0/1446854365 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f04d4102200 0x7f04d419eff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:32.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:32.837+0000 7f04d9ec0640 1 --2- 172.21.15.46:0/1446854365 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f04d4101840 0x7f04d4197f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:32.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:32.837+0000 7f04d9ec0640 1 --2- 172.21.15.46:0/1446854365 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f04d4100650 0x7f04d4197a40 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:32.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:32.837+0000 7f04d9ec0640 1 -- 172.21.15.46:0/1446854365 >> 172.21.15.46:0/1446854365 conn(0x7f04d40fbbe0 msgr2=0x7f04d4068960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:09:32.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:32.837+0000 7f04d9ec0640 1 -- 172.21.15.46:0/1446854365 shutdown_connections 2024-03-31T23:09:32.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:32.837+0000 7f04d9ec0640 1 -- 172.21.15.46:0/1446854365 wait complete. 2024-03-31T23:09:32.848 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:09:32 smithi046 ceph-mon[32498]: pgmap v302: 129 pgs: 129 active+clean; 581 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 511 B/s wr, 0 op/s 2024-03-31T23:09:32.849 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_1 2024-03-31T23:09:32.958 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:32.955+0000 7f6d0f0af640 1 -- 172.21.15.46:0/566290088 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6d08104b10 msgr2=0x7f6d08106f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:09:32.958 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:32.955+0000 7f6d0f0af640 1 --2- 172.21.15.46:0/566290088 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6d08104b10 0x7f6d08106f00 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f6cfc009900 tx=0x7f6cfc02bc30 comp rx=0 tx=0).stop 2024-03-31T23:09:32.958 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:32.955+0000 7f6d0f0af640 1 -- 172.21.15.46:0/566290088 shutdown_connections 2024-03-31T23:09:32.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:32.955+0000 7f6d0f0af640 1 --2- 172.21.15.46:0/566290088 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6d08107440 0x7f6d08109890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:32.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:32.955+0000 7f6d0f0af640 1 --2- 172.21.15.46:0/566290088 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6d08104b10 0x7f6d08106f00 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:32.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:32.955+0000 7f6d0f0af640 1 --2- 172.21.15.46:0/566290088 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6d081021e0 0x7f6d081045d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:32.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:32.955+0000 7f6d0f0af640 1 -- 172.21.15.46:0/566290088 >> 172.21.15.46:0/566290088 conn(0x7f6d080fbbf0 msgr2=0x7f6d080fe010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:09:32.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:32.955+0000 7f6d0f0af640 1 -- 172.21.15.46:0/566290088 shutdown_connections 2024-03-31T23:09:32.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:32.955+0000 7f6d0f0af640 1 -- 172.21.15.46:0/566290088 wait complete. 2024-03-31T23:09:32.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:32.956+0000 7f6d0f0af640 1 Processor -- start 2024-03-31T23:09:32.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:32.956+0000 7f6d0f0af640 1 -- start start 2024-03-31T23:09:32.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:32.956+0000 7f6d0f0af640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6d081021e0 0x7f6d08197a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:09:32.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:32.956+0000 7f6d0f0af640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6d08104b10 0x7f6d08197f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:09:32.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:32.956+0000 7f6d0f0af640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6d08107440 0x7f6d0819efd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:09:32.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:32.956+0000 7f6d0f0af640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f6d0810d2a0 con 0x7f6d081021e0 2024-03-31T23:09:32.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:32.956+0000 7f6d0f0af640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f6d0810d120 con 0x7f6d08104b10 2024-03-31T23:09:32.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:32.956+0000 7f6d0f0af640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f6d0810d420 con 0x7f6d08107440 2024-03-31T23:09:32.960 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:32.956+0000 7f6d0ce4b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6d081021e0 0x7f6d08197a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:09:32.960 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:32.956+0000 7f6d0ce4b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6d081021e0 0x7f6d08197a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:57050/0 (socket says 172.21.15.46:57050) 2024-03-31T23:09:32.960 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:32.956+0000 7f6d0ce4b640 1 -- 172.21.15.46:0/3909749980 learned_addr learned my addr 172.21.15.46:0/3909749980 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:09:32.960 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:32.956+0000 7f6d07fff640 1 --2- 172.21.15.46:0/3909749980 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6d08104b10 0x7f6d08197f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:09:32.960 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:32.956+0000 7f6d0d64c640 1 --2- 172.21.15.46:0/3909749980 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6d08107440 0x7f6d0819efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:09:32.960 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:32.956+0000 7f6d0ce4b640 1 -- 172.21.15.46:0/3909749980 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6d08107440 msgr2=0x7f6d0819efd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:09:32.960 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:32.957+0000 7f6d0ce4b640 1 --2- 172.21.15.46:0/3909749980 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6d08107440 0x7f6d0819efd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:32.960 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:32.957+0000 7f6d0ce4b640 1 -- 172.21.15.46:0/3909749980 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6d08104b10 msgr2=0x7f6d08197f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:09:32.960 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:32.957+0000 7f6d0ce4b640 1 --2- 172.21.15.46:0/3909749980 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6d08104b10 0x7f6d08197f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:32.960 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:32.957+0000 7f6d0ce4b640 1 -- 172.21.15.46:0/3909749980 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d0819f640 con 0x7f6d081021e0 2024-03-31T23:09:32.960 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:32.957+0000 7f6d0ce4b640 1 --2- 172.21.15.46:0/3909749980 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6d081021e0 0x7f6d08197a20 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f6d0000eb90 tx=0x7f6d0000c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:09:32.960 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:32.957+0000 7f6d05ffb640 1 -- 172.21.15.46:0/3909749980 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d00004d20 con 0x7f6d081021e0 2024-03-31T23:09:32.960 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:32.957+0000 7f6d05ffb640 1 -- 172.21.15.46:0/3909749980 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6d0000ee90 con 0x7f6d081021e0 2024-03-31T23:09:32.960 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:32.957+0000 7f6d05ffb640 1 -- 172.21.15.46:0/3909749980 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d00010670 con 0x7f6d081021e0 2024-03-31T23:09:32.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:32.957+0000 7f6d0f0af640 1 -- 172.21.15.46:0/3909749980 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6d0819f930 con 0x7f6d081021e0 2024-03-31T23:09:32.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:32.957+0000 7f6d0f0af640 1 -- 172.21.15.46:0/3909749980 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6d0819fe00 con 0x7f6d081021e0 2024-03-31T23:09:32.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:32.958+0000 7f6d0f0af640 1 -- 172.21.15.46:0/3909749980 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6d080fd6e0 con 0x7f6d081021e0 2024-03-31T23:09:32.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:32.959+0000 7f6d05ffb640 1 -- 172.21.15.46:0/3909749980 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 16) ==== 137320+0+0 (secure 0 0 0) 0x7f6d00005020 con 0x7f6d081021e0 2024-03-31T23:09:32.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:32.959+0000 7f6d05ffb640 1 --2- 172.21.15.46:0/3909749980 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f6cf00a2520 0x7f6cf00a49e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:09:32.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:32.960+0000 7f6d05ffb640 1 -- 172.21.15.46:0/3909749980 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8802+0+0 (secure 0 0 0) 0x7f6d000cef30 con 0x7f6d081021e0 2024-03-31T23:09:32.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:32.960+0000 7f6d07fff640 1 --2- 172.21.15.46:0/3909749980 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f6cf00a2520 0x7f6cf00a49e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:09:32.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:32.960+0000 7f6d07fff640 1 --2- 172.21.15.46:0/3909749980 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f6cf00a2520 0x7f6cf00a49e0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f6cfc005ab0 tx=0x7f6cfc005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:09:32.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:32.964+0000 7f6d05ffb640 1 -- 172.21.15.46:0/3909749980 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f6d0009beb0 con 0x7f6d081021e0 2024-03-31T23:09:33.177 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:33.176+0000 7f6d0f0af640 1 -- 172.21.15.46:0/3909749980 --> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}) -- 0x7f6d08061b90 con 0x7f6cf00a2520 2024-03-31T23:09:33.181 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:33.181+0000 7f6d05ffb640 1 -- 172.21.15.46:0/3909749980 <== mgr.14152 v2:172.21.15.46:6800/162289569 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+54 (secure 0 0 0) 0x7f6d08061b90 con 0x7f6cf00a2520 2024-03-31T23:09:33.182 INFO:teuthology.orchestra.run.smithi046.stdout:/volumes/qa/sv_1/69d74bb8-19e3-449e-b10a-7c03870f34da 2024-03-31T23:09:33.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:33.182+0000 7f6d0f0af640 1 -- 172.21.15.46:0/3909749980 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f6cf00a2520 msgr2=0x7f6cf00a49e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:09:33.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:33.182+0000 7f6d0f0af640 1 --2- 172.21.15.46:0/3909749980 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f6cf00a2520 0x7f6cf00a49e0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f6cfc005ab0 tx=0x7f6cfc005fd0 comp rx=0 tx=0).stop 2024-03-31T23:09:33.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:33.182+0000 7f6d0f0af640 1 -- 172.21.15.46:0/3909749980 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6d081021e0 msgr2=0x7f6d08197a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:09:33.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:33.182+0000 7f6d0f0af640 1 --2- 172.21.15.46:0/3909749980 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6d081021e0 0x7f6d08197a20 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f6d0000eb90 tx=0x7f6d0000c6a0 comp rx=0 tx=0).stop 2024-03-31T23:09:33.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:33.182+0000 7f6d0f0af640 1 -- 172.21.15.46:0/3909749980 shutdown_connections 2024-03-31T23:09:33.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:33.183+0000 7f6d0f0af640 1 --2- 172.21.15.46:0/3909749980 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f6cf00a2520 0x7f6cf00a49e0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:33.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:33.183+0000 7f6d0f0af640 1 --2- 172.21.15.46:0/3909749980 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6d08107440 0x7f6d0819efd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:33.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:33.183+0000 7f6d0f0af640 1 --2- 172.21.15.46:0/3909749980 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6d08104b10 0x7f6d08197f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:33.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:33.183+0000 7f6d0f0af640 1 --2- 172.21.15.46:0/3909749980 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6d081021e0 0x7f6d08197a20 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:33.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:33.183+0000 7f6d0f0af640 1 -- 172.21.15.46:0/3909749980 >> 172.21.15.46:0/3909749980 conn(0x7f6d080fbbf0 msgr2=0x7f6d08102c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:09:33.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:33.183+0000 7f6d0f0af640 1 -- 172.21.15.46:0/3909749980 shutdown_connections 2024-03-31T23:09:33.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:33.183+0000 7f6d0f0af640 1 -- 172.21.15.46:0/3909749980 wait complete. 2024-03-31T23:09:33.194 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-31T23:09:33.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:33.296+0000 7f5bbdf2c640 1 -- 172.21.15.46:0/1405595336 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5bb8103320 msgr2=0x7f5bb8109bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:09:33.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:33.296+0000 7f5bbdf2c640 1 --2- 172.21.15.46:0/1405595336 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5bb8103320 0x7f5bb8109bb0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f5bb0009ca0 tx=0x7f5bb002c0f0 comp rx=0 tx=0).stop 2024-03-31T23:09:33.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:33.296+0000 7f5bbdf2c640 1 -- 172.21.15.46:0/1405595336 shutdown_connections 2024-03-31T23:09:33.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:33.296+0000 7f5bbdf2c640 1 --2- 172.21.15.46:0/1405595336 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5bb8103320 0x7f5bb8109bb0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:33.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:33.296+0000 7f5bbdf2c640 1 --2- 172.21.15.46:0/1405595336 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5bb8102960 0x7f5bb8102de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:33.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:33.296+0000 7f5bbdf2c640 1 --2- 172.21.15.46:0/1405595336 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5bb8101770 0x7f5bb8101b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:33.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:33.296+0000 7f5bbdf2c640 1 -- 172.21.15.46:0/1405595336 >> 172.21.15.46:0/1405595336 conn(0x7f5bb80fcd00 msgr2=0x7f5bb80ff120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:09:33.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:33.296+0000 7f5bbdf2c640 1 -- 172.21.15.46:0/1405595336 shutdown_connections 2024-03-31T23:09:33.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:33.296+0000 7f5bbdf2c640 1 -- 172.21.15.46:0/1405595336 wait complete. 2024-03-31T23:09:33.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:33.297+0000 7f5bbdf2c640 1 Processor -- start 2024-03-31T23:09:33.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:33.297+0000 7f5bbdf2c640 1 -- start start 2024-03-31T23:09:33.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:33.297+0000 7f5bbdf2c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5bb8101770 0x7f5bb8075500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:09:33.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:33.297+0000 7f5bbdf2c640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5bb8102960 0x7f5bb8073b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:09:33.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:33.297+0000 7f5bbdf2c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5bb8103320 0x7f5bb8074080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:09:33.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:33.297+0000 7f5bbdf2c640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f5bb810d280 con 0x7f5bb8103320 2024-03-31T23:09:33.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:33.297+0000 7f5bbdf2c640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f5bb810d100 con 0x7f5bb8102960 2024-03-31T23:09:33.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:33.297+0000 7f5bbdf2c640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f5bb810d400 con 0x7f5bb8101770 2024-03-31T23:09:33.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:33.297+0000 7f5bb7fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5bb8103320 0x7f5bb8074080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:09:33.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:33.297+0000 7f5bb7fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5bb8103320 0x7f5bb8074080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:57074/0 (socket says 172.21.15.46:57074) 2024-03-31T23:09:33.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:33.297+0000 7f5bb7fff640 1 -- 172.21.15.46:0/2481721173 learned_addr learned my addr 172.21.15.46:0/2481721173 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:09:33.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:33.297+0000 7f5bb6ffd640 1 --2- 172.21.15.46:0/2481721173 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5bb8102960 0x7f5bb8073b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:09:33.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:33.297+0000 7f5bb77fe640 1 --2- 172.21.15.46:0/2481721173 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5bb8101770 0x7f5bb8075500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:09:33.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:33.298+0000 7f5bb7fff640 1 -- 172.21.15.46:0/2481721173 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5bb8101770 msgr2=0x7f5bb8075500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:09:33.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:33.298+0000 7f5bb7fff640 1 --2- 172.21.15.46:0/2481721173 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5bb8101770 0x7f5bb8075500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:33.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:33.298+0000 7f5bb7fff640 1 -- 172.21.15.46:0/2481721173 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5bb8102960 msgr2=0x7f5bb8073b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:09:33.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:33.298+0000 7f5bb7fff640 1 --2- 172.21.15.46:0/2481721173 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5bb8102960 0x7f5bb8073b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:33.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:33.298+0000 7f5bb7fff640 1 -- 172.21.15.46:0/2481721173 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5bb80745c0 con 0x7f5bb8103320 2024-03-31T23:09:33.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:33.298+0000 7f5bb7fff640 1 --2- 172.21.15.46:0/2481721173 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5bb8103320 0x7f5bb8074080 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f5bb002c700 tx=0x7f5bb00328a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:09:33.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:33.298+0000 7f5bb4ff9640 1 -- 172.21.15.46:0/2481721173 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bb0041020 con 0x7f5bb8103320 2024-03-31T23:09:33.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:33.298+0000 7f5bb4ff9640 1 -- 172.21.15.46:0/2481721173 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5bb0003e20 con 0x7f5bb8103320 2024-03-31T23:09:33.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:33.298+0000 7f5bbdf2c640 1 -- 172.21.15.46:0/2481721173 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5bb8074850 con 0x7f5bb8103320 2024-03-31T23:09:33.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:33.298+0000 7f5bb4ff9640 1 -- 172.21.15.46:0/2481721173 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bb0039040 con 0x7f5bb8103320 2024-03-31T23:09:33.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:33.298+0000 7f5bbdf2c640 1 -- 172.21.15.46:0/2481721173 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5bb81a6350 con 0x7f5bb8103320 2024-03-31T23:09:33.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:33.299+0000 7f5bbdf2c640 1 -- 172.21.15.46:0/2481721173 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5bb8108a90 con 0x7f5bb8103320 2024-03-31T23:09:33.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:33.300+0000 7f5bb4ff9640 1 -- 172.21.15.46:0/2481721173 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 16) ==== 137320+0+0 (secure 0 0 0) 0x7f5bb0044020 con 0x7f5bb8103320 2024-03-31T23:09:33.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:33.301+0000 7f5bb4ff9640 1 --2- 172.21.15.46:0/2481721173 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f5ba00a24a0 0x7f5ba00a4960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:09:33.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:33.301+0000 7f5bb77fe640 1 --2- 172.21.15.46:0/2481721173 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f5ba00a24a0 0x7f5ba00a4960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:09:33.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:33.301+0000 7f5bb4ff9640 1 -- 172.21.15.46:0/2481721173 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8802+0+0 (secure 0 0 0) 0x7f5bb00f0ee0 con 0x7f5bb8103320 2024-03-31T23:09:33.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:33.301+0000 7f5bb77fe640 1 --2- 172.21.15.46:0/2481721173 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f5ba00a24a0 0x7f5ba00a4960 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f5ba80095f0 tx=0x7f5ba8005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:09:33.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:33.304+0000 7f5bb4ff9640 1 -- 172.21.15.46:0/2481721173 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f5bb004b020 con 0x7f5bb8103320 2024-03-31T23:09:33.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:33.432+0000 7f5bb4ff9640 1 -- 172.21.15.46:0/2481721173 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mgrmap(e 17) ==== 137371+0+0 (secure 0 0 0) 0x7f5bb00b9dd0 con 0x7f5bb8103320 2024-03-31T23:09:33.531 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:33.530+0000 7f5bbdf2c640 1 -- 172.21.15.46:0/2481721173 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5bb802d170 con 0x7f5bb8103320 2024-03-31T23:09:33.531 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:33.530+0000 7f5bb4ff9640 1 -- 172.21.15.46:0/2481721173 <== mon.0 v2:172.21.15.46:3300/0 8 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+11882 (secure 0 0 0) 0x7f5bb00bde60 con 0x7f5bb8103320 2024-03-31T23:09:33.531 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-31T23:09:33.532 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/2169018547","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":2169018547},{"type":"v1","addr":"172.21.15.46:6835","nonce":2169018547}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6837/1069387756","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6836","nonce":1069387756},{"type":"v1","addr":"172.21.15.46:6837","nonce":1069387756}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14634,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6839/3915337844","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6838","nonce":3915337844},{"type":"v1","addr":"172.21.15.46:6839","nonce":3915337844}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14646,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6841/897767145","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6840","nonce":897767145},{"type":"v1","addr":"172.21.15.46:6841","nonce":897767145}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24415,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6833/1746950966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6832","nonce":1746950966},{"type":"v1","addr":"172.21.15.162:6833","nonce":1746950966}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24422,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6833/3394739181","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6832","nonce":3394739181},{"type":"v1","addr":"172.21.15.142:6833","nonce":3394739181}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":24,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-31T23:09:11.009090+0000","modified":"2024-03-31T23:09:31.392282+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24515,"mds_1":24482,"mds_2":24451},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24442":{"gid":24442,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.162:6835/1957208984","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6834","nonce":1957208984},{"type":"v1","addr":"172.21.15.162:6835","nonce":1957208984}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24451":{"gid":24451,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":11,"addr":"172.21.15.162:6837/1371886876","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6836","nonce":1371886876},{"type":"v1","addr":"172.21.15.162:6837","nonce":1371886876}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24455":{"gid":24455,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.142:6835/212637431","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6834","nonce":212637431},{"type":"v1","addr":"172.21.15.142:6835","nonce":212637431}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24478":{"gid":24478,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.162:6839/1976275826","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6838","nonce":1976275826},{"type":"v1","addr":"172.21.15.162:6839","nonce":1976275826}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24482":{"gid":24482,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":10,"addr":"172.21.15.142:6837/2559258399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6836","nonce":2559258399},{"type":"v1","addr":"172.21.15.142:6837","nonce":2559258399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24515":{"gid":24515,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.142:6839/2519143918","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6838","nonce":2519143918},{"type":"v1","addr":"172.21.15.142:6839","nonce":2519143918}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24515,"qdb_cluster":[24451,24482,24515]},"id":1}]} 2024-03-31T23:09:33.532 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 24 2024-03-31T23:09:33.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:33.532+0000 7f5bbdf2c640 1 -- 172.21.15.46:0/2481721173 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f5ba00a24a0 msgr2=0x7f5ba00a4960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:09:33.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:33.532+0000 7f5bbdf2c640 1 --2- 172.21.15.46:0/2481721173 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f5ba00a24a0 0x7f5ba00a4960 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f5ba80095f0 tx=0x7f5ba8005f90 comp rx=0 tx=0).stop 2024-03-31T23:09:33.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:33.532+0000 7f5bbdf2c640 1 -- 172.21.15.46:0/2481721173 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5bb8103320 msgr2=0x7f5bb8074080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:09:33.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:33.532+0000 7f5bbdf2c640 1 --2- 172.21.15.46:0/2481721173 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5bb8103320 0x7f5bb8074080 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f5bb002c700 tx=0x7f5bb00328a0 comp rx=0 tx=0).stop 2024-03-31T23:09:33.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:33.532+0000 7f5bbdf2c640 1 -- 172.21.15.46:0/2481721173 shutdown_connections 2024-03-31T23:09:33.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:33.533+0000 7f5bbdf2c640 1 --2- 172.21.15.46:0/2481721173 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f5ba00a24a0 0x7f5ba00a4960 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:33.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:33.533+0000 7f5bbdf2c640 1 --2- 172.21.15.46:0/2481721173 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5bb8103320 0x7f5bb8074080 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:33.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:33.533+0000 7f5bbdf2c640 1 --2- 172.21.15.46:0/2481721173 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5bb8102960 0x7f5bb8073b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:33.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:33.533+0000 7f5bbdf2c640 1 --2- 172.21.15.46:0/2481721173 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5bb8101770 0x7f5bb8075500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:33.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:33.533+0000 7f5bbdf2c640 1 -- 172.21.15.46:0/2481721173 >> 172.21.15.46:0/2481721173 conn(0x7f5bb80fcd00 msgr2=0x7f5bb8105c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:09:33.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:33.533+0000 7f5bbdf2c640 1 -- 172.21.15.46:0/2481721173 shutdown_connections 2024-03-31T23:09:33.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:33.533+0000 7f5bbdf2c640 1 -- 172.21.15.46:0/2481721173 wait complete. 2024-03-31T23:09:33.543 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:09:33 smithi046 ceph-mon[32498]: from='client.14898 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-31T23:09:33.543 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:09:33 smithi046 ceph-mon[32498]: from='client.14910 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-31T23:09:33.544 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:09:33 smithi046 ceph-mon[32498]: from='client.14916 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-03-31T23:09:33.545 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-31T23:09:33.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:33.646+0000 7f2fda53e640 1 -- 172.21.15.46:0/2579777798 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2fd4102220 msgr2=0x7f2fd4108ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:09:33.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:33.646+0000 7f2fda53e640 1 --2- 172.21.15.46:0/2579777798 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2fd4102220 0x7f2fd4108ab0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f2fcc009ca0 tx=0x7f2fcc02c0f0 comp rx=0 tx=0).stop 2024-03-31T23:09:33.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:33.646+0000 7f2fda53e640 1 -- 172.21.15.46:0/2579777798 shutdown_connections 2024-03-31T23:09:33.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:33.646+0000 7f2fda53e640 1 --2- 172.21.15.46:0/2579777798 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2fd4102220 0x7f2fd4108ab0 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:33.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:33.646+0000 7f2fda53e640 1 --2- 172.21.15.46:0/2579777798 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2fd4101860 0x7f2fd4101ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:33.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:33.646+0000 7f2fda53e640 1 --2- 172.21.15.46:0/2579777798 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2fd4100670 0x7f2fd4100a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:33.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:33.646+0000 7f2fda53e640 1 -- 172.21.15.46:0/2579777798 >> 172.21.15.46:0/2579777798 conn(0x7f2fd40fbbe0 msgr2=0x7f2fd40fe020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:09:33.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:33.646+0000 7f2fda53e640 1 -- 172.21.15.46:0/2579777798 shutdown_connections 2024-03-31T23:09:33.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:33.646+0000 7f2fda53e640 1 -- 172.21.15.46:0/2579777798 wait complete. 2024-03-31T23:09:33.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:33.647+0000 7f2fda53e640 1 Processor -- start 2024-03-31T23:09:33.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:33.647+0000 7f2fda53e640 1 -- start start 2024-03-31T23:09:33.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:33.647+0000 7f2fda53e640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2fd4100670 0x7f2fd4196830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:09:33.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:33.647+0000 7f2fda53e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2fd4101860 0x7f2fd4196d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:09:33.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:33.647+0000 7f2fda53e640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2fd4102220 0x7f2fd419dde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:09:33.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:33.647+0000 7f2fda53e640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f2fd410c180 con 0x7f2fd4101860 2024-03-31T23:09:33.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:33.647+0000 7f2fda53e640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f2fd410c000 con 0x7f2fd4102220 2024-03-31T23:09:33.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:33.647+0000 7f2fda53e640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f2fd410c300 con 0x7f2fd4100670 2024-03-31T23:09:33.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:33.647+0000 7f2fd37fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2fd4101860 0x7f2fd4196d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:09:33.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:33.647+0000 7f2fd37fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2fd4101860 0x7f2fd4196d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:57094/0 (socket says 172.21.15.46:57094) 2024-03-31T23:09:33.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:33.647+0000 7f2fd37fe640 1 -- 172.21.15.46:0/2013748687 learned_addr learned my addr 172.21.15.46:0/2013748687 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:09:33.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:33.648+0000 7f2fd8adb640 1 --2- 172.21.15.46:0/2013748687 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2fd4102220 0x7f2fd419dde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:09:33.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:33.648+0000 7f2fd3fff640 1 --2- 172.21.15.46:0/2013748687 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2fd4100670 0x7f2fd4196830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:09:33.652 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:33.648+0000 7f2fd37fe640 1 -- 172.21.15.46:0/2013748687 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2fd4100670 msgr2=0x7f2fd4196830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:09:33.652 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:33.648+0000 7f2fd37fe640 1 --2- 172.21.15.46:0/2013748687 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2fd4100670 0x7f2fd4196830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:33.652 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:33.648+0000 7f2fd37fe640 1 -- 172.21.15.46:0/2013748687 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2fd4102220 msgr2=0x7f2fd419dde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:09:33.652 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:33.648+0000 7f2fd37fe640 1 --2- 172.21.15.46:0/2013748687 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2fd4102220 0x7f2fd419dde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:33.652 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:33.648+0000 7f2fd37fe640 1 -- 172.21.15.46:0/2013748687 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2fd419e450 con 0x7f2fd4101860 2024-03-31T23:09:33.652 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:33.648+0000 7f2fd37fe640 1 --2- 172.21.15.46:0/2013748687 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2fd4101860 0x7f2fd4196d70 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f2fbc00e9f0 tx=0x7f2fbc00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:09:33.652 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:33.648+0000 7f2fd17fa640 1 -- 172.21.15.46:0/2013748687 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fbc00cd80 con 0x7f2fd4101860 2024-03-31T23:09:33.652 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:33.648+0000 7f2fd17fa640 1 -- 172.21.15.46:0/2013748687 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f2fbc0108e0 con 0x7f2fd4101860 2024-03-31T23:09:33.652 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:33.648+0000 7f2fda53e640 1 -- 172.21.15.46:0/2013748687 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2fd419e740 con 0x7f2fd4101860 2024-03-31T23:09:33.652 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:33.648+0000 7f2fd17fa640 1 -- 172.21.15.46:0/2013748687 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fbc018920 con 0x7f2fd4101860 2024-03-31T23:09:33.652 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:33.648+0000 7f2fda53e640 1 -- 172.21.15.46:0/2013748687 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2fd419ece0 con 0x7f2fd4101860 2024-03-31T23:09:33.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:33.649+0000 7f2fda53e640 1 -- 172.21.15.46:0/2013748687 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2fd4107990 con 0x7f2fd4101860 2024-03-31T23:09:33.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:33.650+0000 7f2fd17fa640 1 -- 172.21.15.46:0/2013748687 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 17) ==== 137371+0+0 (secure 0 0 0) 0x7f2fbc010a90 con 0x7f2fd4101860 2024-03-31T23:09:33.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:33.651+0000 7f2fd17fa640 1 --2- 172.21.15.46:0/2013748687 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f2fc00a25b0 0x7f2fc00a4a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:09:33.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:33.651+0000 7f2fd3fff640 1 --2- 172.21.15.46:0/2013748687 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f2fc00a25b0 0x7f2fc00a4a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:09:33.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:33.651+0000 7f2fd17fa640 1 -- 172.21.15.46:0/2013748687 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8802+0+0 (secure 0 0 0) 0x7f2fbc0cedb0 con 0x7f2fd4101860 2024-03-31T23:09:33.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:33.651+0000 7f2fd3fff640 1 --2- 172.21.15.46:0/2013748687 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f2fc00a25b0 0x7f2fc00a4a70 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f2fc40095f0 tx=0x7f2fc4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:09:33.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:33.655+0000 7f2fd17fa640 1 -- 172.21.15.46:0/2013748687 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f2fbc0d3070 con 0x7f2fd4101860 2024-03-31T23:09:33.693 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:09:33 smithi142 ceph-mon[35271]: from='client.14898 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-31T23:09:33.693 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:09:33 smithi142 ceph-mon[35271]: from='client.14910 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-31T23:09:33.693 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:09:33 smithi142 ceph-mon[35271]: from='client.14916 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-03-31T23:09:33.795 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:09:33 smithi162 ceph-mon[36237]: from='client.14898 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-31T23:09:33.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:09:33 smithi162 ceph-mon[36237]: from='client.14910 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-31T23:09:33.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:09:33 smithi162 ceph-mon[36237]: from='client.14916 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-03-31T23:09:33.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:33.853+0000 7f2fda53e640 1 -- 172.21.15.46:0/2013748687 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f2fd402d2f0 con 0x7f2fd4101860 2024-03-31T23:09:33.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:33.855+0000 7f2fd17fa640 1 -- 172.21.15.46:0/2013748687 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) ==== 74+0+24105 (secure 0 0 0) 0x7f2fbc09b520 con 0x7f2fd4101860 2024-03-31T23:09:33.856 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-31T23:09:33.857 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":91,"fsid":"6a9d8f08-efb2-11ee-b647-cb9ed24678a4","created":"2024-03-31T23:01:15.463711+0000","modified":"2024-03-31T23:09:17.181787+0000","last_up_change":"2024-03-31T23:08:20.199888+0000","last_in_change":"2024-03-31T23:07:59.438035+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-31T23:04:32.823751+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-31T23:09:09.145711+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-31T23:09:10.148579+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-31T23:09:13.419519+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"91","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"47429095-3f46-4ed7-b7c4-504464719815","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":1824772695},{"type":"v1","addr":"172.21.15.46:6803","nonce":1824772695}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":1824772695},{"type":"v1","addr":"172.21.15.46:6805","nonce":1824772695}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":1824772695},{"type":"v1","addr":"172.21.15.46:6809","nonce":1824772695}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":1824772695},{"type":"v1","addr":"172.21.15.46:6807","nonce":1824772695}]},"public_addr":"172.21.15.46:6803/1824772695","cluster_addr":"172.21.15.46:6805/1824772695","heartbeat_back_addr":"172.21.15.46:6809/1824772695","heartbeat_front_addr":"172.21.15.46:6807/1824772695","state":["exists","up"]},{"osd":1,"uuid":"be56f883-411e-4dbe-b50e-a158134df4a6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6800","nonce":973693922},{"type":"v1","addr":"172.21.15.142:6801","nonce":973693922}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6802","nonce":973693922},{"type":"v1","addr":"172.21.15.142:6803","nonce":973693922}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6806","nonce":973693922},{"type":"v1","addr":"172.21.15.142:6807","nonce":973693922}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6804","nonce":973693922},{"type":"v1","addr":"172.21.15.142:6805","nonce":973693922}]},"public_addr":"172.21.15.142:6801/973693922","cluster_addr":"172.21.15.142:6803/973693922","heartbeat_back_addr":"172.21.15.142:6807/973693922","heartbeat_front_addr":"172.21.15.142:6805/973693922","state":["exists","up"]},{"osd":2,"uuid":"c99a8794-ab19-4260-9a8d-4bc0bd2c8175","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6800","nonce":4084261813},{"type":"v1","addr":"172.21.15.162:6801","nonce":4084261813}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6802","nonce":4084261813},{"type":"v1","addr":"172.21.15.162:6803","nonce":4084261813}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6806","nonce":4084261813},{"type":"v1","addr":"172.21.15.162:6807","nonce":4084261813}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6804","nonce":4084261813},{"type":"v1","addr":"172.21.15.162:6805","nonce":4084261813}]},"public_addr":"172.21.15.162:6801/4084261813","cluster_addr":"172.21.15.162:6803/4084261813","heartbeat_back_addr":"172.21.15.162:6807/4084261813","heartbeat_front_addr":"172.21.15.162:6805/4084261813","state":["exists","up"]},{"osd":3,"uuid":"f877814e-9ef6-4872-a081-5609bb9ebc06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":3496317970},{"type":"v1","addr":"172.21.15.46:6811","nonce":3496317970}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":3496317970},{"type":"v1","addr":"172.21.15.46:6813","nonce":3496317970}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":3496317970},{"type":"v1","addr":"172.21.15.46:6817","nonce":3496317970}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":3496317970},{"type":"v1","addr":"172.21.15.46:6815","nonce":3496317970}]},"public_addr":"172.21.15.46:6811/3496317970","cluster_addr":"172.21.15.46:6813/3496317970","heartbeat_back_addr":"172.21.15.46:6817/3496317970","heartbeat_front_addr":"172.21.15.46:6815/3496317970","state":["exists","up"]},{"osd":4,"uuid":"beecec0a-8280-44fe-8f5f-8498a35eca8f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6808","nonce":2996266929},{"type":"v1","addr":"172.21.15.142:6809","nonce":2996266929}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6810","nonce":2996266929},{"type":"v1","addr":"172.21.15.142:6811","nonce":2996266929}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6814","nonce":2996266929},{"type":"v1","addr":"172.21.15.142:6815","nonce":2996266929}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6812","nonce":2996266929},{"type":"v1","addr":"172.21.15.142:6813","nonce":2996266929}]},"public_addr":"172.21.15.142:6809/2996266929","cluster_addr":"172.21.15.142:6811/2996266929","heartbeat_back_addr":"172.21.15.142:6815/2996266929","heartbeat_front_addr":"172.21.15.142:6813/2996266929","state":["exists","up"]},{"osd":5,"uuid":"9da7ff75-f658-45ea-a02f-1943fdda76b9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6808","nonce":914208293},{"type":"v1","addr":"172.21.15.162:6809","nonce":914208293}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6810","nonce":914208293},{"type":"v1","addr":"172.21.15.162:6811","nonce":914208293}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6814","nonce":914208293},{"type":"v1","addr":"172.21.15.162:6815","nonce":914208293}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6812","nonce":914208293},{"type":"v1","addr":"172.21.15.162:6813","nonce":914208293}]},"public_addr":"172.21.15.162:6809/914208293","cluster_addr":"172.21.15.162:6811/914208293","heartbeat_back_addr":"172.21.15.162:6815/914208293","heartbeat_front_addr":"172.21.15.162:6813/914208293","state":["exists","up"]},{"osd":6,"uuid":"8fa11224-cb26-4d3f-87d1-6a8f31de2a19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":56291966},{"type":"v1","addr":"172.21.15.46:6819","nonce":56291966}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":56291966},{"type":"v1","addr":"172.21.15.46:6821","nonce":56291966}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6824","nonce":56291966},{"type":"v1","addr":"172.21.15.46:6825","nonce":56291966}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":56291966},{"type":"v1","addr":"172.21.15.46:6823","nonce":56291966}]},"public_addr":"172.21.15.46:6819/56291966","cluster_addr":"172.21.15.46:6821/56291966","heartbeat_back_addr":"172.21.15.46:6825/56291966","heartbeat_front_addr":"172.21.15.46:6823/56291966","state":["exists","up"]},{"osd":7,"uuid":"0d0338b7-4d0a-4bb0-b152-8f5224871d9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6816","nonce":2146598446},{"type":"v1","addr":"172.21.15.142:6817","nonce":2146598446}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6818","nonce":2146598446},{"type":"v1","addr":"172.21.15.142:6819","nonce":2146598446}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6822","nonce":2146598446},{"type":"v1","addr":"172.21.15.142:6823","nonce":2146598446}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6820","nonce":2146598446},{"type":"v1","addr":"172.21.15.142:6821","nonce":2146598446}]},"public_addr":"172.21.15.142:6817/2146598446","cluster_addr":"172.21.15.142:6819/2146598446","heartbeat_back_addr":"172.21.15.142:6823/2146598446","heartbeat_front_addr":"172.21.15.142:6821/2146598446","state":["exists","up"]},{"osd":8,"uuid":"8b6f6037-892a-416c-babb-7552648d0b3a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6816","nonce":1173078318},{"type":"v1","addr":"172.21.15.162:6817","nonce":1173078318}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6818","nonce":1173078318},{"type":"v1","addr":"172.21.15.162:6819","nonce":1173078318}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6822","nonce":1173078318},{"type":"v1","addr":"172.21.15.162:6823","nonce":1173078318}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6820","nonce":1173078318},{"type":"v1","addr":"172.21.15.162:6821","nonce":1173078318}]},"public_addr":"172.21.15.162:6817/1173078318","cluster_addr":"172.21.15.162:6819/1173078318","heartbeat_back_addr":"172.21.15.162:6823/1173078318","heartbeat_front_addr":"172.21.15.162:6821/1173078318","state":["exists","up"]},{"osd":9,"uuid":"faccce25-6bf2-4220-8858-cda82b72be68","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":532708881},{"type":"v1","addr":"172.21.15.46:6827","nonce":532708881}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6828","nonce":532708881},{"type":"v1","addr":"172.21.15.46:6829","nonce":532708881}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":532708881},{"type":"v1","addr":"172.21.15.46:6833","nonce":532708881}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":532708881},{"type":"v1","addr":"172.21.15.46:6831","nonce":532708881}]},"public_addr":"172.21.15.46:6827/532708881","cluster_addr":"172.21.15.46:6829/532708881","heartbeat_back_addr":"172.21.15.46:6833/532708881","heartbeat_front_addr":"172.21.15.46:6831/532708881","state":["exists","up"]},{"osd":10,"uuid":"cdd356db-f9eb-4080-95ab-db5d8223e572","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6824","nonce":2845222366},{"type":"v1","addr":"172.21.15.142:6825","nonce":2845222366}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6826","nonce":2845222366},{"type":"v1","addr":"172.21.15.142:6827","nonce":2845222366}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6830","nonce":2845222366},{"type":"v1","addr":"172.21.15.142:6831","nonce":2845222366}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6828","nonce":2845222366},{"type":"v1","addr":"172.21.15.142:6829","nonce":2845222366}]},"public_addr":"172.21.15.142:6825/2845222366","cluster_addr":"172.21.15.142:6827/2845222366","heartbeat_back_addr":"172.21.15.142:6831/2845222366","heartbeat_front_addr":"172.21.15.142:6829/2845222366","state":["exists","up"]},{"osd":11,"uuid":"72104d1c-3ea2-45ad-bc0b-0500f146c06b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6824","nonce":1986323778},{"type":"v1","addr":"172.21.15.162:6825","nonce":1986323778}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6826","nonce":1986323778},{"type":"v1","addr":"172.21.15.162:6827","nonce":1986323778}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6830","nonce":1986323778},{"type":"v1","addr":"172.21.15.162:6831","nonce":1986323778}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6828","nonce":1986323778},{"type":"v1","addr":"172.21.15.162:6829","nonce":1986323778}]},"public_addr":"172.21.15.162:6825/1986323778","cluster_addr":"172.21.15.162:6827/1986323778","heartbeat_back_addr":"172.21.15.162:6831/1986323778","heartbeat_front_addr":"172.21.15.162:6829/1986323778","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:03:44.996331+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:04:07.994738+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:04:30.942255+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:04:55.588884+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:05:19.900600+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:05:43.807442+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:06:09.026976+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:06:34.586784+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:06:59.751605+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:07:26.144495+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:07:52.326909+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:08:17.423042+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.46:0/4021039857":"2024-04-01T23:02:08.493619+0000","172.21.15.46:6801/3681182647":"2024-04-01T23:02:08.493619+0000","172.21.15.46:0/4022214806":"2024-04-01T23:02:08.493619+0000","172.21.15.46:6800/1515528215":"2024-04-01T23:01:41.910162+0000","172.21.15.46:6800/3681182647":"2024-04-01T23:02:08.493619+0000","172.21.15.46:0/143885707":"2024-04-01T23:01:41.910162+0000","172.21.15.46:6801/1515528215":"2024-04-01T23:01:41.910162+0000","172.21.15.46:0/1680389778":"2024-04-01T23:01:41.910162+0000","172.21.15.46:0/1782133966":"2024-04-01T23:01:41.910162+0000","172.21.15.46:0/3925209376":"2024-04-01T23:02:08.493619+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-31T23:09:33.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:33.857+0000 7f2fda53e640 1 -- 172.21.15.46:0/2013748687 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f2fc00a25b0 msgr2=0x7f2fc00a4a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:09:33.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:33.857+0000 7f2fda53e640 1 --2- 172.21.15.46:0/2013748687 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f2fc00a25b0 0x7f2fc00a4a70 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f2fc40095f0 tx=0x7f2fc4005f90 comp rx=0 tx=0).stop 2024-03-31T23:09:33.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:33.857+0000 7f2fda53e640 1 -- 172.21.15.46:0/2013748687 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2fd4101860 msgr2=0x7f2fd4196d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:09:33.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:33.857+0000 7f2fda53e640 1 --2- 172.21.15.46:0/2013748687 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2fd4101860 0x7f2fd4196d70 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f2fbc00e9f0 tx=0x7f2fbc00eeb0 comp rx=0 tx=0).stop 2024-03-31T23:09:33.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:33.857+0000 7f2fda53e640 1 -- 172.21.15.46:0/2013748687 shutdown_connections 2024-03-31T23:09:33.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:33.857+0000 7f2fda53e640 1 --2- 172.21.15.46:0/2013748687 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f2fc00a25b0 0x7f2fc00a4a70 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:33.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:33.857+0000 7f2fda53e640 1 --2- 172.21.15.46:0/2013748687 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2fd4102220 0x7f2fd419dde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:33.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:33.857+0000 7f2fda53e640 1 --2- 172.21.15.46:0/2013748687 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2fd4101860 0x7f2fd4196d70 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:33.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:33.857+0000 7f2fda53e640 1 --2- 172.21.15.46:0/2013748687 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2fd4100670 0x7f2fd4196830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:33.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:33.857+0000 7f2fda53e640 1 -- 172.21.15.46:0/2013748687 >> 172.21.15.46:0/2013748687 conn(0x7f2fd40fbbe0 msgr2=0x7f2fd4104b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:09:33.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:33.857+0000 7f2fda53e640 1 -- 172.21.15.46:0/2013748687 shutdown_connections 2024-03-31T23:09:33.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:33.857+0000 7f2fda53e640 1 -- 172.21.15.46:0/2013748687 wait complete. 2024-03-31T23:09:33.870 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-31T23:09:33.870 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-31T23:09:33.975 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:33.971+0000 7fbc1db3e640 1 -- 172.21.15.46:0/4240309404 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbc18102220 msgr2=0x7fbc18108ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:09:33.975 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:33.971+0000 7fbc1db3e640 1 --2- 172.21.15.46:0/4240309404 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbc18102220 0x7fbc18108ab0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fbc10009ca0 tx=0x7fbc1002c0f0 comp rx=0 tx=0).stop 2024-03-31T23:09:33.975 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:33.971+0000 7fbc1db3e640 1 -- 172.21.15.46:0/4240309404 shutdown_connections 2024-03-31T23:09:33.975 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:33.972+0000 7fbc1db3e640 1 --2- 172.21.15.46:0/4240309404 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbc18102220 0x7fbc18108ab0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:33.975 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:33.972+0000 7fbc1db3e640 1 --2- 172.21.15.46:0/4240309404 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbc18101860 0x7fbc18101ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:33.975 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:33.972+0000 7fbc1db3e640 1 --2- 172.21.15.46:0/4240309404 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbc18100670 0x7fbc18100a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:33.975 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:33.972+0000 7fbc1db3e640 1 -- 172.21.15.46:0/4240309404 >> 172.21.15.46:0/4240309404 conn(0x7fbc180fbbe0 msgr2=0x7fbc180fe020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:09:33.975 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:33.972+0000 7fbc1db3e640 1 -- 172.21.15.46:0/4240309404 shutdown_connections 2024-03-31T23:09:33.975 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:33.972+0000 7fbc1db3e640 1 -- 172.21.15.46:0/4240309404 wait complete. 2024-03-31T23:09:33.975 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:33.972+0000 7fbc1db3e640 1 Processor -- start 2024-03-31T23:09:33.975 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:33.972+0000 7fbc1db3e640 1 -- start start 2024-03-31T23:09:33.976 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:33.972+0000 7fbc1db3e640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbc18100670 0x7fbc18197970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:09:33.976 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:33.972+0000 7fbc1db3e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbc18101860 0x7fbc18197eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:09:33.976 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:33.972+0000 7fbc1db3e640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbc18102220 0x7fbc1819ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:09:33.976 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:33.972+0000 7fbc1db3e640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fbc1810c1b0 con 0x7fbc18101860 2024-03-31T23:09:33.976 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:33.972+0000 7fbc1db3e640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fbc1810c030 con 0x7fbc18100670 2024-03-31T23:09:33.976 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:33.972+0000 7fbc1db3e640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fbc1810c330 con 0x7fbc18102220 2024-03-31T23:09:33.976 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:33.973+0000 7fbc16ffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbc18101860 0x7fbc18197eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:09:33.976 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:33.973+0000 7fbc16ffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbc18101860 0x7fbc18197eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:57106/0 (socket says 172.21.15.46:57106) 2024-03-31T23:09:33.976 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:33.973+0000 7fbc16ffd640 1 -- 172.21.15.46:0/749011050 learned_addr learned my addr 172.21.15.46:0/749011050 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:09:33.976 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:33.973+0000 7fbc177fe640 1 --2- 172.21.15.46:0/749011050 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbc18100670 0x7fbc18197970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:09:33.976 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:33.973+0000 7fbc17fff640 1 --2- 172.21.15.46:0/749011050 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbc18102220 0x7fbc1819ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:09:33.976 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:33.973+0000 7fbc16ffd640 1 -- 172.21.15.46:0/749011050 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbc18102220 msgr2=0x7fbc1819ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:09:33.976 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:33.973+0000 7fbc16ffd640 1 --2- 172.21.15.46:0/749011050 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbc18102220 0x7fbc1819ef20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:33.976 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:33.973+0000 7fbc16ffd640 1 -- 172.21.15.46:0/749011050 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbc18100670 msgr2=0x7fbc18197970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:09:33.977 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:33.973+0000 7fbc16ffd640 1 --2- 172.21.15.46:0/749011050 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbc18100670 0x7fbc18197970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:33.977 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:33.973+0000 7fbc16ffd640 1 -- 172.21.15.46:0/749011050 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc1819f590 con 0x7fbc18101860 2024-03-31T23:09:33.977 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:33.973+0000 7fbc16ffd640 1 --2- 172.21.15.46:0/749011050 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbc18101860 0x7fbc18197eb0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fbc0c00e9f0 tx=0x7fbc0c00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:09:33.977 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:33.973+0000 7fbc14ff9640 1 -- 172.21.15.46:0/749011050 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc0c00cd80 con 0x7fbc18101860 2024-03-31T23:09:33.977 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:33.973+0000 7fbc14ff9640 1 -- 172.21.15.46:0/749011050 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fbc0c0054d0 con 0x7fbc18101860 2024-03-31T23:09:33.977 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:33.973+0000 7fbc1db3e640 1 -- 172.21.15.46:0/749011050 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbc1819f880 con 0x7fbc18101860 2024-03-31T23:09:33.977 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:33.973+0000 7fbc14ff9640 1 -- 172.21.15.46:0/749011050 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc0c010490 con 0x7fbc18101860 2024-03-31T23:09:33.977 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:33.973+0000 7fbc1db3e640 1 -- 172.21.15.46:0/749011050 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbc1819fd80 con 0x7fbc18101860 2024-03-31T23:09:33.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:33.974+0000 7fbc1db3e640 1 -- 172.21.15.46:0/749011050 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbc1819ffe0 con 0x7fbc18101860 2024-03-31T23:09:33.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:33.975+0000 7fbc14ff9640 1 -- 172.21.15.46:0/749011050 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 17) ==== 137371+0+0 (secure 0 0 0) 0x7fbc0c005020 con 0x7fbc18101860 2024-03-31T23:09:33.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:33.976+0000 7fbc14ff9640 1 --2- 172.21.15.46:0/749011050 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fbc000a2530 0x7fbc000a49f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:09:33.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:33.976+0000 7fbc177fe640 1 --2- 172.21.15.46:0/749011050 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fbc000a2530 0x7fbc000a49f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:09:33.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:33.976+0000 7fbc14ff9640 1 -- 172.21.15.46:0/749011050 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8802+0+0 (secure 0 0 0) 0x7fbc0c0ceec0 con 0x7fbc18101860 2024-03-31T23:09:33.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:33.976+0000 7fbc177fe640 1 --2- 172.21.15.46:0/749011050 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fbc000a2530 0x7fbc000a49f0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fbc181016c0 tx=0x7fbc08005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:09:33.981 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:33.980+0000 7fbc14ff9640 1 -- 172.21.15.46:0/749011050 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fbc0c09bc80 con 0x7fbc18101860 2024-03-31T23:09:34.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:34.159+0000 7fbc1db3e640 1 -- 172.21.15.46:0/749011050 --> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fbc181a01b0 con 0x7fbc000a2530 2024-03-31T23:09:34.170 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:34.169+0000 7fbc14ff9640 1 -- 172.21.15.46:0/749011050 <== mgr.14152 v2:172.21.15.46:6800/162289569 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340221 (secure 0 0 0) 0x7fbc181a01b0 con 0x7fbc000a2530 2024-03-31T23:09:34.175 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-31T23:09:34.177 INFO:teuthology.orchestra.run.smithi046.stderr:dumped all 2024-03-31T23:09:34.178 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:34.173+0000 7fbc1db3e640 1 -- 172.21.15.46:0/749011050 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fbc000a2530 msgr2=0x7fbc000a49f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:09:34.178 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:34.173+0000 7fbc1db3e640 1 --2- 172.21.15.46:0/749011050 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fbc000a2530 0x7fbc000a49f0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fbc181016c0 tx=0x7fbc08005f90 comp rx=0 tx=0).stop 2024-03-31T23:09:34.179 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:34.173+0000 7fbc1db3e640 1 -- 172.21.15.46:0/749011050 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbc18101860 msgr2=0x7fbc18197eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:09:34.179 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:34.173+0000 7fbc1db3e640 1 --2- 172.21.15.46:0/749011050 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbc18101860 0x7fbc18197eb0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fbc0c00e9f0 tx=0x7fbc0c00edf0 comp rx=0 tx=0).stop 2024-03-31T23:09:34.179 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:34.173+0000 7fbc1db3e640 1 -- 172.21.15.46:0/749011050 shutdown_connections 2024-03-31T23:09:34.179 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:34.173+0000 7fbc1db3e640 1 --2- 172.21.15.46:0/749011050 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fbc000a2530 0x7fbc000a49f0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:34.179 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:34.173+0000 7fbc1db3e640 1 --2- 172.21.15.46:0/749011050 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbc18102220 0x7fbc1819ef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:34.179 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:34.173+0000 7fbc1db3e640 1 --2- 172.21.15.46:0/749011050 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbc18101860 0x7fbc18197eb0 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:34.179 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:34.173+0000 7fbc1db3e640 1 --2- 172.21.15.46:0/749011050 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbc18100670 0x7fbc18197970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:34.179 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:34.173+0000 7fbc1db3e640 1 -- 172.21.15.46:0/749011050 >> 172.21.15.46:0/749011050 conn(0x7fbc180fbbe0 msgr2=0x7fbc180688b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:09:34.179 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:34.173+0000 7fbc1db3e640 1 -- 172.21.15.46:0/749011050 shutdown_connections 2024-03-31T23:09:34.179 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:34.173+0000 7fbc1db3e640 1 -- 172.21.15.46:0/749011050 wait complete. 2024-03-31T23:09:34.185 INFO:teuthology.orchestra.run.smithi046.stdout:{"pg_ready":true,"pg_map":{"version":303,"stamp":"2024-03-31T23:09:32.581099+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596693,"num_objects":34,"num_object_clones":0,"num_object_copies":102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34,"num_whiteouts":0,"num_read":142,"num_read_kb":123,"num_write":253,"num_write_kb":2962,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":20,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":204,"ondisk_log_size":204,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":367068,"kb_used_data":13720,"kb_used_omap":18,"kb_used_meta":353197,"kb_avail":1124574756,"statfs":{"total":1151940427776,"available":1151564550144,"internally_reserved":0,"allocated":14049280,"data_stored":7680920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19140,"internal_metadata":361674044},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":3915,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":27,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003009"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":23,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-31T23:09:17.536388+0000","last_change":"2024-03-31T23:09:11.177502+0000","last_active":"2024-03-31T23:09:17.536388+0000","last_peered":"2024-03-31T23:09:17.536388+0000","last_clean":"2024-03-31T23:09:17.536388+0000","last_became_active":"2024-03-31T23:09:11.177134+0000","last_became_peered":"2024-03-31T23:09:11.177134+0000","last_unstale":"2024-03-31T23:09:17.536388+0000","last_undegraded":"2024-03-31T23:09:17.536388+0000","last_fullsized":"2024-03-31T23:09:17.536388+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T10:29:36.508229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":23,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-31T23:09:17.770764+0000","last_change":"2024-03-31T23:09:11.172990+0000","last_active":"2024-03-31T23:09:17.770764+0000","last_peered":"2024-03-31T23:09:17.770764+0000","last_clean":"2024-03-31T23:09:17.770764+0000","last_became_active":"2024-03-31T23:09:11.172684+0000","last_became_peered":"2024-03-31T23:09:11.172684+0000","last_unstale":"2024-03-31T23:09:17.770764+0000","last_undegraded":"2024-03-31T23:09:17.770764+0000","last_fullsized":"2024-03-31T23:09:17.770764+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T07:02:59.991757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":23,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-31T23:09:17.770500+0000","last_change":"2024-03-31T23:09:11.176805+0000","last_active":"2024-03-31T23:09:17.770500+0000","last_peered":"2024-03-31T23:09:17.770500+0000","last_clean":"2024-03-31T23:09:17.770500+0000","last_became_active":"2024-03-31T23:09:11.176468+0000","last_became_peered":"2024-03-31T23:09:11.176468+0000","last_unstale":"2024-03-31T23:09:17.770500+0000","last_undegraded":"2024-03-31T23:09:17.770500+0000","last_fullsized":"2024-03-31T23:09:17.770500+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T08:23:41.872558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":23,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-31T23:09:17.185909+0000","last_change":"2024-03-31T23:09:11.182385+0000","last_active":"2024-03-31T23:09:17.185909+0000","last_peered":"2024-03-31T23:09:17.185909+0000","last_clean":"2024-03-31T23:09:17.185909+0000","last_became_active":"2024-03-31T23:09:11.181908+0000","last_became_peered":"2024-03-31T23:09:11.181908+0000","last_unstale":"2024-03-31T23:09:17.185909+0000","last_undegraded":"2024-03-31T23:09:17.185909+0000","last_fullsized":"2024-03-31T23:09:17.185909+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:52:24.066360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":23,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-31T23:09:17.536719+0000","last_change":"2024-03-31T23:09:11.181377+0000","last_active":"2024-03-31T23:09:17.536719+0000","last_peered":"2024-03-31T23:09:17.536719+0000","last_clean":"2024-03-31T23:09:17.536719+0000","last_became_active":"2024-03-31T23:09:11.181064+0000","last_became_peered":"2024-03-31T23:09:11.181064+0000","last_unstale":"2024-03-31T23:09:17.536719+0000","last_undegraded":"2024-03-31T23:09:17.536719+0000","last_fullsized":"2024-03-31T23:09:17.536719+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:28:40.766541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":23,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-31T23:09:17.535946+0000","last_change":"2024-03-31T23:09:11.176087+0000","last_active":"2024-03-31T23:09:17.535946+0000","last_peered":"2024-03-31T23:09:17.535946+0000","last_clean":"2024-03-31T23:09:17.535946+0000","last_became_active":"2024-03-31T23:09:11.175569+0000","last_became_peered":"2024-03-31T23:09:11.175569+0000","last_unstale":"2024-03-31T23:09:17.535946+0000","last_undegraded":"2024-03-31T23:09:17.535946+0000","last_fullsized":"2024-03-31T23:09:17.535946+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T06:33:19.189909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":23,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-31T23:09:17.535508+0000","last_change":"2024-03-31T23:09:11.170289+0000","last_active":"2024-03-31T23:09:17.535508+0000","last_peered":"2024-03-31T23:09:17.535508+0000","last_clean":"2024-03-31T23:09:17.535508+0000","last_became_active":"2024-03-31T23:09:11.169959+0000","last_became_peered":"2024-03-31T23:09:11.169959+0000","last_unstale":"2024-03-31T23:09:17.535508+0000","last_undegraded":"2024-03-31T23:09:17.535508+0000","last_fullsized":"2024-03-31T23:09:17.535508+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:10:44.614767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":23,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-31T23:09:17.388094+0000","last_change":"2024-03-31T23:09:11.182447+0000","last_active":"2024-03-31T23:09:17.388094+0000","last_peered":"2024-03-31T23:09:17.388094+0000","last_clean":"2024-03-31T23:09:17.388094+0000","last_became_active":"2024-03-31T23:09:11.181629+0000","last_became_peered":"2024-03-31T23:09:11.181629+0000","last_unstale":"2024-03-31T23:09:17.388094+0000","last_undegraded":"2024-03-31T23:09:17.388094+0000","last_fullsized":"2024-03-31T23:09:17.388094+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:18:26.269451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":23,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-31T23:09:17.536065+0000","last_change":"2024-03-31T23:09:11.176853+0000","last_active":"2024-03-31T23:09:17.536065+0000","last_peered":"2024-03-31T23:09:17.536065+0000","last_clean":"2024-03-31T23:09:17.536065+0000","last_became_active":"2024-03-31T23:09:11.176413+0000","last_became_peered":"2024-03-31T23:09:11.176413+0000","last_unstale":"2024-03-31T23:09:17.536065+0000","last_undegraded":"2024-03-31T23:09:17.536065+0000","last_fullsized":"2024-03-31T23:09:17.536065+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:23:40.486947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":23,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-31T23:09:17.536433+0000","last_change":"2024-03-31T23:09:11.174902+0000","last_active":"2024-03-31T23:09:17.536433+0000","last_peered":"2024-03-31T23:09:17.536433+0000","last_clean":"2024-03-31T23:09:17.536433+0000","last_became_active":"2024-03-31T23:09:11.174385+0000","last_became_peered":"2024-03-31T23:09:11.174385+0000","last_unstale":"2024-03-31T23:09:17.536433+0000","last_undegraded":"2024-03-31T23:09:17.536433+0000","last_fullsized":"2024-03-31T23:09:17.536433+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:07:58.649222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":23,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-31T23:09:17.535686+0000","last_change":"2024-03-31T23:09:11.179536+0000","last_active":"2024-03-31T23:09:17.535686+0000","last_peered":"2024-03-31T23:09:17.535686+0000","last_clean":"2024-03-31T23:09:17.535686+0000","last_became_active":"2024-03-31T23:09:11.178521+0000","last_became_peered":"2024-03-31T23:09:11.178521+0000","last_unstale":"2024-03-31T23:09:17.535686+0000","last_undegraded":"2024-03-31T23:09:17.535686+0000","last_fullsized":"2024-03-31T23:09:17.535686+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T09:29:23.128832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":23,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-31T23:09:17.185822+0000","last_change":"2024-03-31T23:09:11.182548+0000","last_active":"2024-03-31T23:09:17.185822+0000","last_peered":"2024-03-31T23:09:17.185822+0000","last_clean":"2024-03-31T23:09:17.185822+0000","last_became_active":"2024-03-31T23:09:11.181966+0000","last_became_peered":"2024-03-31T23:09:11.181966+0000","last_unstale":"2024-03-31T23:09:17.185822+0000","last_undegraded":"2024-03-31T23:09:17.185822+0000","last_fullsized":"2024-03-31T23:09:17.185822+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:43:26.061594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":23,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-31T23:09:17.535331+0000","last_change":"2024-03-31T23:09:11.171254+0000","last_active":"2024-03-31T23:09:17.535331+0000","last_peered":"2024-03-31T23:09:17.535331+0000","last_clean":"2024-03-31T23:09:17.535331+0000","last_became_active":"2024-03-31T23:09:11.170684+0000","last_became_peered":"2024-03-31T23:09:11.170684+0000","last_unstale":"2024-03-31T23:09:17.535331+0000","last_undegraded":"2024-03-31T23:09:17.535331+0000","last_fullsized":"2024-03-31T23:09:17.535331+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:03:10.231149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":23,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-31T23:09:17.770231+0000","last_change":"2024-03-31T23:09:11.168784+0000","last_active":"2024-03-31T23:09:17.770231+0000","last_peered":"2024-03-31T23:09:17.770231+0000","last_clean":"2024-03-31T23:09:17.770231+0000","last_became_active":"2024-03-31T23:09:11.168442+0000","last_became_peered":"2024-03-31T23:09:11.168442+0000","last_unstale":"2024-03-31T23:09:17.770231+0000","last_undegraded":"2024-03-31T23:09:17.770231+0000","last_fullsized":"2024-03-31T23:09:17.770231+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T09:37:43.568122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":23,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-31T23:09:17.770411+0000","last_change":"2024-03-31T23:09:11.176017+0000","last_active":"2024-03-31T23:09:17.770411+0000","last_peered":"2024-03-31T23:09:17.770411+0000","last_clean":"2024-03-31T23:09:17.770411+0000","last_became_active":"2024-03-31T23:09:11.175690+0000","last_became_peered":"2024-03-31T23:09:11.175690+0000","last_unstale":"2024-03-31T23:09:17.770411+0000","last_undegraded":"2024-03-31T23:09:17.770411+0000","last_fullsized":"2024-03-31T23:09:17.770411+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:14:13.259895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":23,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-31T23:09:17.535080+0000","last_change":"2024-03-31T23:09:11.171926+0000","last_active":"2024-03-31T23:09:17.535080+0000","last_peered":"2024-03-31T23:09:17.535080+0000","last_clean":"2024-03-31T23:09:17.535080+0000","last_became_active":"2024-03-31T23:09:11.171625+0000","last_became_peered":"2024-03-31T23:09:11.171625+0000","last_unstale":"2024-03-31T23:09:17.535080+0000","last_undegraded":"2024-03-31T23:09:17.535080+0000","last_fullsized":"2024-03-31T23:09:17.535080+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T06:20:20.433802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":23,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-31T23:09:17.536147+0000","last_change":"2024-03-31T23:09:11.183805+0000","last_active":"2024-03-31T23:09:17.536147+0000","last_peered":"2024-03-31T23:09:17.536147+0000","last_clean":"2024-03-31T23:09:17.536147+0000","last_became_active":"2024-03-31T23:09:11.183464+0000","last_became_peered":"2024-03-31T23:09:11.183464+0000","last_unstale":"2024-03-31T23:09:17.536147+0000","last_undegraded":"2024-03-31T23:09:17.536147+0000","last_fullsized":"2024-03-31T23:09:17.536147+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T06:02:33.286563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":23,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-31T23:09:17.184894+0000","last_change":"2024-03-31T23:09:11.170095+0000","last_active":"2024-03-31T23:09:17.184894+0000","last_peered":"2024-03-31T23:09:17.184894+0000","last_clean":"2024-03-31T23:09:17.184894+0000","last_became_active":"2024-03-31T23:09:11.169683+0000","last_became_peered":"2024-03-31T23:09:11.169683+0000","last_unstale":"2024-03-31T23:09:17.184894+0000","last_undegraded":"2024-03-31T23:09:17.184894+0000","last_fullsized":"2024-03-31T23:09:17.184894+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T08:19:47.101431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":23,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-31T23:09:17.186475+0000","last_change":"2024-03-31T23:09:11.181347+0000","last_active":"2024-03-31T23:09:17.186475+0000","last_peered":"2024-03-31T23:09:17.186475+0000","last_clean":"2024-03-31T23:09:17.186475+0000","last_became_active":"2024-03-31T23:09:11.180510+0000","last_became_peered":"2024-03-31T23:09:11.180510+0000","last_unstale":"2024-03-31T23:09:17.186475+0000","last_undegraded":"2024-03-31T23:09:17.186475+0000","last_fullsized":"2024-03-31T23:09:17.186475+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:25:51.083446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-31T23:09:17.186233+0000","last_change":"2024-03-31T23:09:15.205059+0000","last_active":"2024-03-31T23:09:17.186233+0000","last_peered":"2024-03-31T23:09:17.186233+0000","last_clean":"2024-03-31T23:09:17.186233+0000","last_became_active":"2024-03-31T23:09:15.204612+0000","last_became_peered":"2024-03-31T23:09:15.204612+0000","last_unstale":"2024-03-31T23:09:17.186233+0000","last_undegraded":"2024-03-31T23:09:17.186233+0000","last_fullsized":"2024-03-31T23:09:17.186233+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_clean_scrub_stamp":"2024-03-31T23:09:14.171251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:53:33.998026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"85'4","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-31T23:09:17.185332+0000","last_change":"2024-03-31T23:09:10.594147+0000","last_active":"2024-03-31T23:09:17.185332+0000","last_peered":"2024-03-31T23:09:17.185332+0000","last_clean":"2024-03-31T23:09:17.185332+0000","last_became_active":"2024-03-31T23:09:10.592047+0000","last_became_peered":"2024-03-31T23:09:10.592047+0000","last_unstale":"2024-03-31T23:09:17.185332+0000","last_undegraded":"2024-03-31T23:09:17.185332+0000","last_fullsized":"2024-03-31T23:09:17.185332+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_clean_scrub_stamp":"2024-03-31T23:09:09.564263+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:04:34.548282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":23,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-31T23:09:17.388462+0000","last_change":"2024-03-31T23:09:11.182292+0000","last_active":"2024-03-31T23:09:17.388462+0000","last_peered":"2024-03-31T23:09:17.388462+0000","last_clean":"2024-03-31T23:09:17.388462+0000","last_became_active":"2024-03-31T23:09:11.181885+0000","last_became_peered":"2024-03-31T23:09:11.181885+0000","last_unstale":"2024-03-31T23:09:17.388462+0000","last_undegraded":"2024-03-31T23:09:17.388462+0000","last_fullsized":"2024-03-31T23:09:17.388462+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T10:38:17.550625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-31T23:09:17.185543+0000","last_change":"2024-03-31T23:09:15.209034+0000","last_active":"2024-03-31T23:09:17.185543+0000","last_peered":"2024-03-31T23:09:17.185543+0000","last_clean":"2024-03-31T23:09:17.185543+0000","last_became_active":"2024-03-31T23:09:15.208615+0000","last_became_peered":"2024-03-31T23:09:15.208615+0000","last_unstale":"2024-03-31T23:09:17.185543+0000","last_undegraded":"2024-03-31T23:09:17.185543+0000","last_fullsized":"2024-03-31T23:09:17.185543+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_clean_scrub_stamp":"2024-03-31T23:09:14.171251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T07:59:19.538784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"85'1","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-31T23:09:17.388648+0000","last_change":"2024-03-31T23:09:10.598375+0000","last_active":"2024-03-31T23:09:17.388648+0000","last_peered":"2024-03-31T23:09:17.388648+0000","last_clean":"2024-03-31T23:09:17.388648+0000","last_became_active":"2024-03-31T23:09:10.598045+0000","last_became_peered":"2024-03-31T23:09:10.598045+0000","last_unstale":"2024-03-31T23:09:17.388648+0000","last_undegraded":"2024-03-31T23:09:17.388648+0000","last_fullsized":"2024-03-31T23:09:17.388648+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_clean_scrub_stamp":"2024-03-31T23:09:09.564263+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:31:27.113317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":23,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-31T23:09:17.770536+0000","last_change":"2024-03-31T23:09:11.183458+0000","last_active":"2024-03-31T23:09:17.770536+0000","last_peered":"2024-03-31T23:09:17.770536+0000","last_clean":"2024-03-31T23:09:17.770536+0000","last_became_active":"2024-03-31T23:09:11.183100+0000","last_became_peered":"2024-03-31T23:09:11.183100+0000","last_unstale":"2024-03-31T23:09:17.770536+0000","last_undegraded":"2024-03-31T23:09:17.770536+0000","last_fullsized":"2024-03-31T23:09:17.770536+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T10:55:35.512065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-31T23:09:17.185727+0000","last_change":"2024-03-31T23:09:15.208103+0000","last_active":"2024-03-31T23:09:17.185727+0000","last_peered":"2024-03-31T23:09:17.185727+0000","last_clean":"2024-03-31T23:09:17.185727+0000","last_became_active":"2024-03-31T23:09:15.207723+0000","last_became_peered":"2024-03-31T23:09:15.207723+0000","last_unstale":"2024-03-31T23:09:17.185727+0000","last_undegraded":"2024-03-31T23:09:17.185727+0000","last_fullsized":"2024-03-31T23:09:17.185727+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_clean_scrub_stamp":"2024-03-31T23:09:14.171251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T07:55:01.303693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"85'2","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-31T23:09:17.537116+0000","last_change":"2024-03-31T23:09:10.597471+0000","last_active":"2024-03-31T23:09:17.537116+0000","last_peered":"2024-03-31T23:09:17.537116+0000","last_clean":"2024-03-31T23:09:17.537116+0000","last_became_active":"2024-03-31T23:09:10.597165+0000","last_became_peered":"2024-03-31T23:09:10.597165+0000","last_unstale":"2024-03-31T23:09:17.537116+0000","last_undegraded":"2024-03-31T23:09:17.537116+0000","last_fullsized":"2024-03-31T23:09:17.537116+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_clean_scrub_stamp":"2024-03-31T23:09:09.564263+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:10:26.826525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":23,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-31T23:09:17.536710+0000","last_change":"2024-03-31T23:09:11.170650+0000","last_active":"2024-03-31T23:09:17.536710+0000","last_peered":"2024-03-31T23:09:17.536710+0000","last_clean":"2024-03-31T23:09:17.536710+0000","last_became_active":"2024-03-31T23:09:11.170291+0000","last_became_peered":"2024-03-31T23:09:11.170291+0000","last_unstale":"2024-03-31T23:09:17.536710+0000","last_undegraded":"2024-03-31T23:09:17.536710+0000","last_fullsized":"2024-03-31T23:09:17.536710+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:29:54.904377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-31T23:09:17.185586+0000","last_change":"2024-03-31T23:09:15.204857+0000","last_active":"2024-03-31T23:09:17.185586+0000","last_peered":"2024-03-31T23:09:17.185586+0000","last_clean":"2024-03-31T23:09:17.185586+0000","last_became_active":"2024-03-31T23:09:15.204484+0000","last_became_peered":"2024-03-31T23:09:15.204484+0000","last_unstale":"2024-03-31T23:09:17.185586+0000","last_undegraded":"2024-03-31T23:09:17.185586+0000","last_fullsized":"2024-03-31T23:09:17.185586+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_clean_scrub_stamp":"2024-03-31T23:09:14.171251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T09:54:24.646918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"85'2","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-31T23:09:17.536910+0000","last_change":"2024-03-31T23:09:10.589922+0000","last_active":"2024-03-31T23:09:17.536910+0000","last_peered":"2024-03-31T23:09:17.536910+0000","last_clean":"2024-03-31T23:09:17.536910+0000","last_became_active":"2024-03-31T23:09:10.589569+0000","last_became_peered":"2024-03-31T23:09:10.589569+0000","last_unstale":"2024-03-31T23:09:17.536910+0000","last_undegraded":"2024-03-31T23:09:17.536910+0000","last_fullsized":"2024-03-31T23:09:17.536910+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_clean_scrub_stamp":"2024-03-31T23:09:09.564263+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:57:05.347344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":23,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-31T23:09:17.537194+0000","last_change":"2024-03-31T23:09:11.174204+0000","last_active":"2024-03-31T23:09:17.537194+0000","last_peered":"2024-03-31T23:09:17.537194+0000","last_clean":"2024-03-31T23:09:17.537194+0000","last_became_active":"2024-03-31T23:09:11.173903+0000","last_became_peered":"2024-03-31T23:09:11.173903+0000","last_unstale":"2024-03-31T23:09:17.537194+0000","last_undegraded":"2024-03-31T23:09:17.537194+0000","last_fullsized":"2024-03-31T23:09:17.537194+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:20:23.038437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-31T23:09:17.186498+0000","last_change":"2024-03-31T23:09:15.193096+0000","last_active":"2024-03-31T23:09:17.186498+0000","last_peered":"2024-03-31T23:09:17.186498+0000","last_clean":"2024-03-31T23:09:17.186498+0000","last_became_active":"2024-03-31T23:09:15.192624+0000","last_became_peered":"2024-03-31T23:09:15.192624+0000","last_unstale":"2024-03-31T23:09:17.186498+0000","last_undegraded":"2024-03-31T23:09:17.186498+0000","last_fullsized":"2024-03-31T23:09:17.186498+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_clean_scrub_stamp":"2024-03-31T23:09:14.171251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:22:24.921346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"91'6","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-31T23:09:31.391419+0000","last_change":"2024-03-31T23:09:10.587209+0000","last_active":"2024-03-31T23:09:31.391419+0000","last_peered":"2024-03-31T23:09:31.391419+0000","last_clean":"2024-03-31T23:09:31.391419+0000","last_became_active":"2024-03-31T23:09:10.586486+0000","last_became_peered":"2024-03-31T23:09:10.586486+0000","last_unstale":"2024-03-31T23:09:31.391419+0000","last_undegraded":"2024-03-31T23:09:31.391419+0000","last_fullsized":"2024-03-31T23:09:31.391419+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_clean_scrub_stamp":"2024-03-31T23:09:09.564263+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:27:56.046414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1013,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":23,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-31T23:09:17.537109+0000","last_change":"2024-03-31T23:09:11.175458+0000","last_active":"2024-03-31T23:09:17.537109+0000","last_peered":"2024-03-31T23:09:17.537109+0000","last_clean":"2024-03-31T23:09:17.537109+0000","last_became_active":"2024-03-31T23:09:11.175047+0000","last_became_peered":"2024-03-31T23:09:11.175047+0000","last_unstale":"2024-03-31T23:09:17.537109+0000","last_undegraded":"2024-03-31T23:09:17.537109+0000","last_fullsized":"2024-03-31T23:09:17.537109+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T06:21:22.733625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-31T23:09:17.186391+0000","last_change":"2024-03-31T23:09:15.194717+0000","last_active":"2024-03-31T23:09:17.186391+0000","last_peered":"2024-03-31T23:09:17.186391+0000","last_clean":"2024-03-31T23:09:17.186391+0000","last_became_active":"2024-03-31T23:09:15.194232+0000","last_became_peered":"2024-03-31T23:09:15.194232+0000","last_unstale":"2024-03-31T23:09:17.186391+0000","last_undegraded":"2024-03-31T23:09:17.186391+0000","last_fullsized":"2024-03-31T23:09:17.186391+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_clean_scrub_stamp":"2024-03-31T23:09:14.171251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:21:00.445232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"85'1","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-31T23:09:17.535893+0000","last_change":"2024-03-31T23:09:10.586346+0000","last_active":"2024-03-31T23:09:17.535893+0000","last_peered":"2024-03-31T23:09:17.535893+0000","last_clean":"2024-03-31T23:09:17.535893+0000","last_became_active":"2024-03-31T23:09:10.586043+0000","last_became_peered":"2024-03-31T23:09:10.586043+0000","last_unstale":"2024-03-31T23:09:17.535893+0000","last_undegraded":"2024-03-31T23:09:17.535893+0000","last_fullsized":"2024-03-31T23:09:17.535893+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_clean_scrub_stamp":"2024-03-31T23:09:09.564263+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:17:27.387786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":23,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-31T23:09:17.536602+0000","last_change":"2024-03-31T23:09:11.181543+0000","last_active":"2024-03-31T23:09:17.536602+0000","last_peered":"2024-03-31T23:09:17.536602+0000","last_clean":"2024-03-31T23:09:17.536602+0000","last_became_active":"2024-03-31T23:09:11.181234+0000","last_became_peered":"2024-03-31T23:09:11.181234+0000","last_unstale":"2024-03-31T23:09:17.536602+0000","last_undegraded":"2024-03-31T23:09:17.536602+0000","last_fullsized":"2024-03-31T23:09:17.536602+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T09:50:14.712348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-31T23:09:17.536138+0000","last_change":"2024-03-31T23:09:15.205855+0000","last_active":"2024-03-31T23:09:17.536138+0000","last_peered":"2024-03-31T23:09:17.536138+0000","last_clean":"2024-03-31T23:09:17.536138+0000","last_became_active":"2024-03-31T23:09:15.205359+0000","last_became_peered":"2024-03-31T23:09:15.205359+0000","last_unstale":"2024-03-31T23:09:17.536138+0000","last_undegraded":"2024-03-31T23:09:17.536138+0000","last_fullsized":"2024-03-31T23:09:17.536138+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_clean_scrub_stamp":"2024-03-31T23:09:14.171251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T09:39:32.461011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-31T23:09:17.535211+0000","last_change":"2024-03-31T23:09:10.583274+0000","last_active":"2024-03-31T23:09:17.535211+0000","last_peered":"2024-03-31T23:09:17.535211+0000","last_clean":"2024-03-31T23:09:17.535211+0000","last_became_active":"2024-03-31T23:09:10.582910+0000","last_became_peered":"2024-03-31T23:09:10.582910+0000","last_unstale":"2024-03-31T23:09:17.535211+0000","last_undegraded":"2024-03-31T23:09:17.535211+0000","last_fullsized":"2024-03-31T23:09:17.535211+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_clean_scrub_stamp":"2024-03-31T23:09:09.564263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:16:32.251169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":23,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-31T23:09:17.388219+0000","last_change":"2024-03-31T23:09:11.178619+0000","last_active":"2024-03-31T23:09:17.388219+0000","last_peered":"2024-03-31T23:09:17.388219+0000","last_clean":"2024-03-31T23:09:17.388219+0000","last_became_active":"2024-03-31T23:09:11.178239+0000","last_became_peered":"2024-03-31T23:09:11.178239+0000","last_unstale":"2024-03-31T23:09:17.388219+0000","last_undegraded":"2024-03-31T23:09:17.388219+0000","last_fullsized":"2024-03-31T23:09:17.388219+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:28:58.863199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-31T23:09:17.770679+0000","last_change":"2024-03-31T23:09:15.205864+0000","last_active":"2024-03-31T23:09:17.770679+0000","last_peered":"2024-03-31T23:09:17.770679+0000","last_clean":"2024-03-31T23:09:17.770679+0000","last_became_active":"2024-03-31T23:09:15.204976+0000","last_became_peered":"2024-03-31T23:09:15.204976+0000","last_unstale":"2024-03-31T23:09:17.770679+0000","last_undegraded":"2024-03-31T23:09:17.770679+0000","last_fullsized":"2024-03-31T23:09:17.770679+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_clean_scrub_stamp":"2024-03-31T23:09:14.171251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:39:08.537827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"91'1","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-31T23:09:17.535993+0000","last_change":"2024-03-31T23:09:10.588407+0000","last_active":"2024-03-31T23:09:17.535993+0000","last_peered":"2024-03-31T23:09:17.535993+0000","last_clean":"2024-03-31T23:09:17.535993+0000","last_became_active":"2024-03-31T23:09:10.588109+0000","last_became_peered":"2024-03-31T23:09:10.588109+0000","last_unstale":"2024-03-31T23:09:17.535993+0000","last_undegraded":"2024-03-31T23:09:17.535993+0000","last_fullsized":"2024-03-31T23:09:17.535993+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_clean_scrub_stamp":"2024-03-31T23:09:09.564263+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:01:25.858834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":23,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-31T23:09:17.184363+0000","last_change":"2024-03-31T23:09:11.180452+0000","last_active":"2024-03-31T23:09:17.184363+0000","last_peered":"2024-03-31T23:09:17.184363+0000","last_clean":"2024-03-31T23:09:17.184363+0000","last_became_active":"2024-03-31T23:09:11.180117+0000","last_became_peered":"2024-03-31T23:09:11.180117+0000","last_unstale":"2024-03-31T23:09:17.184363+0000","last_undegraded":"2024-03-31T23:09:17.184363+0000","last_fullsized":"2024-03-31T23:09:17.184363+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:04:58.288981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-31T23:09:17.388557+0000","last_change":"2024-03-31T23:09:15.199669+0000","last_active":"2024-03-31T23:09:17.388557+0000","last_peered":"2024-03-31T23:09:17.388557+0000","last_clean":"2024-03-31T23:09:17.388557+0000","last_became_active":"2024-03-31T23:09:15.199262+0000","last_became_peered":"2024-03-31T23:09:15.199262+0000","last_unstale":"2024-03-31T23:09:17.388557+0000","last_undegraded":"2024-03-31T23:09:17.388557+0000","last_fullsized":"2024-03-31T23:09:17.388557+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_clean_scrub_stamp":"2024-03-31T23:09:14.171251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:44:02.975000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"91'3","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-31T23:09:29.421862+0000","last_change":"2024-03-31T23:09:10.583476+0000","last_active":"2024-03-31T23:09:29.421862+0000","last_peered":"2024-03-31T23:09:29.421862+0000","last_clean":"2024-03-31T23:09:29.421862+0000","last_became_active":"2024-03-31T23:09:10.582789+0000","last_became_peered":"2024-03-31T23:09:10.582789+0000","last_unstale":"2024-03-31T23:09:29.421862+0000","last_undegraded":"2024-03-31T23:09:29.421862+0000","last_fullsized":"2024-03-31T23:09:29.421862+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_clean_scrub_stamp":"2024-03-31T23:09:09.564263+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T10:52:35.478325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":23,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-31T23:09:17.535626+0000","last_change":"2024-03-31T23:09:11.171074+0000","last_active":"2024-03-31T23:09:17.535626+0000","last_peered":"2024-03-31T23:09:17.535626+0000","last_clean":"2024-03-31T23:09:17.535626+0000","last_became_active":"2024-03-31T23:09:11.169572+0000","last_became_peered":"2024-03-31T23:09:11.169572+0000","last_unstale":"2024-03-31T23:09:17.535626+0000","last_undegraded":"2024-03-31T23:09:17.535626+0000","last_fullsized":"2024-03-31T23:09:17.535626+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T07:15:34.833388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-31T23:09:17.184460+0000","last_change":"2024-03-31T23:09:15.203835+0000","last_active":"2024-03-31T23:09:17.184460+0000","last_peered":"2024-03-31T23:09:17.184460+0000","last_clean":"2024-03-31T23:09:17.184460+0000","last_became_active":"2024-03-31T23:09:15.200861+0000","last_became_peered":"2024-03-31T23:09:15.200861+0000","last_unstale":"2024-03-31T23:09:17.184460+0000","last_undegraded":"2024-03-31T23:09:17.184460+0000","last_fullsized":"2024-03-31T23:09:17.184460+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_clean_scrub_stamp":"2024-03-31T23:09:14.171251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:56:19.664708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"91'2","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-31T23:09:29.423479+0000","last_change":"2024-03-31T23:09:10.592601+0000","last_active":"2024-03-31T23:09:29.423479+0000","last_peered":"2024-03-31T23:09:29.423479+0000","last_clean":"2024-03-31T23:09:29.423479+0000","last_became_active":"2024-03-31T23:09:10.592345+0000","last_became_peered":"2024-03-31T23:09:10.592345+0000","last_unstale":"2024-03-31T23:09:29.423479+0000","last_undegraded":"2024-03-31T23:09:29.423479+0000","last_fullsized":"2024-03-31T23:09:29.423479+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_clean_scrub_stamp":"2024-03-31T23:09:09.564263+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:10:21.195544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":23,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-31T23:09:17.770411+0000","last_change":"2024-03-31T23:09:11.175305+0000","last_active":"2024-03-31T23:09:17.770411+0000","last_peered":"2024-03-31T23:09:17.770411+0000","last_clean":"2024-03-31T23:09:17.770411+0000","last_became_active":"2024-03-31T23:09:11.174970+0000","last_became_peered":"2024-03-31T23:09:11.174970+0000","last_unstale":"2024-03-31T23:09:17.770411+0000","last_undegraded":"2024-03-31T23:09:17.770411+0000","last_fullsized":"2024-03-31T23:09:17.770411+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:29:39.718127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-31T23:09:17.186161+0000","last_change":"2024-03-31T23:09:15.200365+0000","last_active":"2024-03-31T23:09:17.186161+0000","last_peered":"2024-03-31T23:09:17.186161+0000","last_clean":"2024-03-31T23:09:17.186161+0000","last_became_active":"2024-03-31T23:09:15.199830+0000","last_became_peered":"2024-03-31T23:09:15.199830+0000","last_unstale":"2024-03-31T23:09:17.186161+0000","last_undegraded":"2024-03-31T23:09:17.186161+0000","last_fullsized":"2024-03-31T23:09:17.186161+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_clean_scrub_stamp":"2024-03-31T23:09:14.171251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:37:30.048859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"91'4","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-31T23:09:29.420719+0000","last_change":"2024-03-31T23:09:10.582654+0000","last_active":"2024-03-31T23:09:29.420719+0000","last_peered":"2024-03-31T23:09:29.420719+0000","last_clean":"2024-03-31T23:09:29.420719+0000","last_became_active":"2024-03-31T23:09:10.582314+0000","last_became_peered":"2024-03-31T23:09:10.582314+0000","last_unstale":"2024-03-31T23:09:29.420719+0000","last_undegraded":"2024-03-31T23:09:29.420719+0000","last_fullsized":"2024-03-31T23:09:29.420719+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_clean_scrub_stamp":"2024-03-31T23:09:09.564263+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:51:34.236575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":23,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-31T23:09:17.536765+0000","last_change":"2024-03-31T23:09:11.172811+0000","last_active":"2024-03-31T23:09:17.536765+0000","last_peered":"2024-03-31T23:09:17.536765+0000","last_clean":"2024-03-31T23:09:17.536765+0000","last_became_active":"2024-03-31T23:09:11.172478+0000","last_became_peered":"2024-03-31T23:09:11.172478+0000","last_unstale":"2024-03-31T23:09:17.536765+0000","last_undegraded":"2024-03-31T23:09:17.536765+0000","last_fullsized":"2024-03-31T23:09:17.536765+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T10:00:09.825254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-31T23:09:17.184780+0000","last_change":"2024-03-31T23:09:15.205494+0000","last_active":"2024-03-31T23:09:17.184780+0000","last_peered":"2024-03-31T23:09:17.184780+0000","last_clean":"2024-03-31T23:09:17.184780+0000","last_became_active":"2024-03-31T23:09:15.205101+0000","last_became_peered":"2024-03-31T23:09:15.205101+0000","last_unstale":"2024-03-31T23:09:17.184780+0000","last_undegraded":"2024-03-31T23:09:17.184780+0000","last_fullsized":"2024-03-31T23:09:17.184780+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_clean_scrub_stamp":"2024-03-31T23:09:14.171251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:52:47.472296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"91'12","reported_seq":37,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-31T23:09:29.420328+0000","last_change":"2024-03-31T23:09:10.585611+0000","last_active":"2024-03-31T23:09:29.420328+0000","last_peered":"2024-03-31T23:09:29.420328+0000","last_clean":"2024-03-31T23:09:29.420328+0000","last_became_active":"2024-03-31T23:09:10.585287+0000","last_became_peered":"2024-03-31T23:09:10.585287+0000","last_unstale":"2024-03-31T23:09:29.420328+0000","last_undegraded":"2024-03-31T23:09:29.420328+0000","last_fullsized":"2024-03-31T23:09:29.420328+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_clean_scrub_stamp":"2024-03-31T23:09:09.564263+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:47:44.063831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3842,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":23,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-31T23:09:17.536930+0000","last_change":"2024-03-31T23:09:11.175812+0000","last_active":"2024-03-31T23:09:17.536930+0000","last_peered":"2024-03-31T23:09:17.536930+0000","last_clean":"2024-03-31T23:09:17.536930+0000","last_became_active":"2024-03-31T23:09:11.174972+0000","last_became_peered":"2024-03-31T23:09:11.174972+0000","last_unstale":"2024-03-31T23:09:17.536930+0000","last_undegraded":"2024-03-31T23:09:17.536930+0000","last_fullsized":"2024-03-31T23:09:17.536930+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:17:18.870340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-31T23:09:17.537653+0000","last_change":"2024-03-31T23:09:15.205018+0000","last_active":"2024-03-31T23:09:17.537653+0000","last_peered":"2024-03-31T23:09:17.537653+0000","last_clean":"2024-03-31T23:09:17.537653+0000","last_became_active":"2024-03-31T23:09:15.204510+0000","last_became_peered":"2024-03-31T23:09:15.204510+0000","last_unstale":"2024-03-31T23:09:17.537653+0000","last_undegraded":"2024-03-31T23:09:17.537653+0000","last_fullsized":"2024-03-31T23:09:17.537653+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_clean_scrub_stamp":"2024-03-31T23:09:14.171251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T07:48:59.819937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"85'2","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-31T23:09:17.186385+0000","last_change":"2024-03-31T23:09:10.590233+0000","last_active":"2024-03-31T23:09:17.186385+0000","last_peered":"2024-03-31T23:09:17.186385+0000","last_clean":"2024-03-31T23:09:17.186385+0000","last_became_active":"2024-03-31T23:09:10.589853+0000","last_became_peered":"2024-03-31T23:09:10.589853+0000","last_unstale":"2024-03-31T23:09:17.186385+0000","last_undegraded":"2024-03-31T23:09:17.186385+0000","last_fullsized":"2024-03-31T23:09:17.186385+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_clean_scrub_stamp":"2024-03-31T23:09:09.564263+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:10:54.980394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":23,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-31T23:09:17.535338+0000","last_change":"2024-03-31T23:09:11.177871+0000","last_active":"2024-03-31T23:09:17.535338+0000","last_peered":"2024-03-31T23:09:17.535338+0000","last_clean":"2024-03-31T23:09:17.535338+0000","last_became_active":"2024-03-31T23:09:11.176504+0000","last_became_peered":"2024-03-31T23:09:11.176504+0000","last_unstale":"2024-03-31T23:09:17.535338+0000","last_undegraded":"2024-03-31T23:09:17.535338+0000","last_fullsized":"2024-03-31T23:09:17.535338+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T06:09:08.797695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-31T23:09:17.535914+0000","last_change":"2024-03-31T23:09:15.207336+0000","last_active":"2024-03-31T23:09:17.535914+0000","last_peered":"2024-03-31T23:09:17.535914+0000","last_clean":"2024-03-31T23:09:17.535914+0000","last_became_active":"2024-03-31T23:09:15.206893+0000","last_became_peered":"2024-03-31T23:09:15.206893+0000","last_unstale":"2024-03-31T23:09:17.535914+0000","last_undegraded":"2024-03-31T23:09:17.535914+0000","last_fullsized":"2024-03-31T23:09:17.535914+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_clean_scrub_stamp":"2024-03-31T23:09:14.171251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:24:59.766421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"91'3","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-31T23:09:29.423986+0000","last_change":"2024-03-31T23:09:10.587099+0000","last_active":"2024-03-31T23:09:29.423986+0000","last_peered":"2024-03-31T23:09:29.423986+0000","last_clean":"2024-03-31T23:09:29.423986+0000","last_became_active":"2024-03-31T23:09:10.586429+0000","last_became_peered":"2024-03-31T23:09:10.586429+0000","last_unstale":"2024-03-31T23:09:29.423986+0000","last_undegraded":"2024-03-31T23:09:29.423986+0000","last_fullsized":"2024-03-31T23:09:29.423986+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_clean_scrub_stamp":"2024-03-31T23:09:09.564263+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:55:09.307378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":23,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-31T23:09:17.536335+0000","last_change":"2024-03-31T23:09:11.177661+0000","last_active":"2024-03-31T23:09:17.536335+0000","last_peered":"2024-03-31T23:09:17.536335+0000","last_clean":"2024-03-31T23:09:17.536335+0000","last_became_active":"2024-03-31T23:09:11.177314+0000","last_became_peered":"2024-03-31T23:09:11.177314+0000","last_unstale":"2024-03-31T23:09:17.536335+0000","last_undegraded":"2024-03-31T23:09:17.536335+0000","last_fullsized":"2024-03-31T23:09:17.536335+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T09:26:50.556837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-31T23:09:17.184783+0000","last_change":"2024-03-31T23:09:15.198653+0000","last_active":"2024-03-31T23:09:17.184783+0000","last_peered":"2024-03-31T23:09:17.184783+0000","last_clean":"2024-03-31T23:09:17.184783+0000","last_became_active":"2024-03-31T23:09:15.198277+0000","last_became_peered":"2024-03-31T23:09:15.198277+0000","last_unstale":"2024-03-31T23:09:17.184783+0000","last_undegraded":"2024-03-31T23:09:17.184783+0000","last_fullsized":"2024-03-31T23:09:17.184783+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_clean_scrub_stamp":"2024-03-31T23:09:14.171251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:27:31.391784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-31T23:09:17.185458+0000","last_change":"2024-03-31T23:09:10.586570+0000","last_active":"2024-03-31T23:09:17.185458+0000","last_peered":"2024-03-31T23:09:17.185458+0000","last_clean":"2024-03-31T23:09:17.185458+0000","last_became_active":"2024-03-31T23:09:10.586239+0000","last_became_peered":"2024-03-31T23:09:10.586239+0000","last_unstale":"2024-03-31T23:09:17.185458+0000","last_undegraded":"2024-03-31T23:09:17.185458+0000","last_fullsized":"2024-03-31T23:09:17.185458+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_clean_scrub_stamp":"2024-03-31T23:09:09.564263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:23:37.182954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":23,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-31T23:09:17.536044+0000","last_change":"2024-03-31T23:09:11.173294+0000","last_active":"2024-03-31T23:09:17.536044+0000","last_peered":"2024-03-31T23:09:17.536044+0000","last_clean":"2024-03-31T23:09:17.536044+0000","last_became_active":"2024-03-31T23:09:11.172926+0000","last_became_peered":"2024-03-31T23:09:11.172926+0000","last_unstale":"2024-03-31T23:09:17.536044+0000","last_undegraded":"2024-03-31T23:09:17.536044+0000","last_fullsized":"2024-03-31T23:09:17.536044+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T06:21:46.603381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-31T23:09:17.535654+0000","last_change":"2024-03-31T23:09:15.199893+0000","last_active":"2024-03-31T23:09:17.535654+0000","last_peered":"2024-03-31T23:09:17.535654+0000","last_clean":"2024-03-31T23:09:17.535654+0000","last_became_active":"2024-03-31T23:09:15.199517+0000","last_became_peered":"2024-03-31T23:09:15.199517+0000","last_unstale":"2024-03-31T23:09:17.535654+0000","last_undegraded":"2024-03-31T23:09:17.535654+0000","last_fullsized":"2024-03-31T23:09:17.535654+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_clean_scrub_stamp":"2024-03-31T23:09:14.171251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T07:48:35.261108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"85'8","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-31T23:09:17.536545+0000","last_change":"2024-03-31T23:09:10.588213+0000","last_active":"2024-03-31T23:09:17.536545+0000","last_peered":"2024-03-31T23:09:17.536545+0000","last_clean":"2024-03-31T23:09:17.536545+0000","last_became_active":"2024-03-31T23:09:10.587579+0000","last_became_peered":"2024-03-31T23:09:10.587579+0000","last_unstale":"2024-03-31T23:09:17.536545+0000","last_undegraded":"2024-03-31T23:09:17.536545+0000","last_fullsized":"2024-03-31T23:09:17.536545+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_clean_scrub_stamp":"2024-03-31T23:09:09.564263+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:00:17.883435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":23,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-31T23:09:17.536667+0000","last_change":"2024-03-31T23:09:11.182427+0000","last_active":"2024-03-31T23:09:17.536667+0000","last_peered":"2024-03-31T23:09:17.536667+0000","last_clean":"2024-03-31T23:09:17.536667+0000","last_became_active":"2024-03-31T23:09:11.182120+0000","last_became_peered":"2024-03-31T23:09:11.182120+0000","last_unstale":"2024-03-31T23:09:17.536667+0000","last_undegraded":"2024-03-31T23:09:17.536667+0000","last_fullsized":"2024-03-31T23:09:17.536667+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:17:48.728424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-31T23:09:17.536553+0000","last_change":"2024-03-31T23:09:15.207831+0000","last_active":"2024-03-31T23:09:17.536553+0000","last_peered":"2024-03-31T23:09:17.536553+0000","last_clean":"2024-03-31T23:09:17.536553+0000","last_became_active":"2024-03-31T23:09:15.207432+0000","last_became_peered":"2024-03-31T23:09:15.207432+0000","last_unstale":"2024-03-31T23:09:17.536553+0000","last_undegraded":"2024-03-31T23:09:17.536553+0000","last_fullsized":"2024-03-31T23:09:17.536553+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_clean_scrub_stamp":"2024-03-31T23:09:14.171251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:05:27.072054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"85'1","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-31T23:09:17.537551+0000","last_change":"2024-03-31T23:09:10.591483+0000","last_active":"2024-03-31T23:09:17.537551+0000","last_peered":"2024-03-31T23:09:17.537551+0000","last_clean":"2024-03-31T23:09:17.537551+0000","last_became_active":"2024-03-31T23:09:10.591173+0000","last_became_peered":"2024-03-31T23:09:10.591173+0000","last_unstale":"2024-03-31T23:09:17.537551+0000","last_undegraded":"2024-03-31T23:09:17.537551+0000","last_fullsized":"2024-03-31T23:09:17.537551+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_clean_scrub_stamp":"2024-03-31T23:09:09.564263+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T10:04:14.275875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":23,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-31T23:09:17.536341+0000","last_change":"2024-03-31T23:09:11.181171+0000","last_active":"2024-03-31T23:09:17.536341+0000","last_peered":"2024-03-31T23:09:17.536341+0000","last_clean":"2024-03-31T23:09:17.536341+0000","last_became_active":"2024-03-31T23:09:11.180847+0000","last_became_peered":"2024-03-31T23:09:11.180847+0000","last_unstale":"2024-03-31T23:09:17.536341+0000","last_undegraded":"2024-03-31T23:09:17.536341+0000","last_fullsized":"2024-03-31T23:09:17.536341+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:39:29.429009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-31T23:09:17.537438+0000","last_change":"2024-03-31T23:09:15.746941+0000","last_active":"2024-03-31T23:09:17.537438+0000","last_peered":"2024-03-31T23:09:17.537438+0000","last_clean":"2024-03-31T23:09:17.537438+0000","last_became_active":"2024-03-31T23:09:15.746352+0000","last_became_peered":"2024-03-31T23:09:15.746352+0000","last_unstale":"2024-03-31T23:09:17.537438+0000","last_undegraded":"2024-03-31T23:09:17.537438+0000","last_fullsized":"2024-03-31T23:09:17.537438+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_clean_scrub_stamp":"2024-03-31T23:09:14.171251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T06:21:43.482575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-31T23:09:17.184670+0000","last_change":"2024-03-31T23:09:10.593029+0000","last_active":"2024-03-31T23:09:17.184670+0000","last_peered":"2024-03-31T23:09:17.184670+0000","last_clean":"2024-03-31T23:09:17.184670+0000","last_became_active":"2024-03-31T23:09:10.592709+0000","last_became_peered":"2024-03-31T23:09:10.592709+0000","last_unstale":"2024-03-31T23:09:17.184670+0000","last_undegraded":"2024-03-31T23:09:17.184670+0000","last_fullsized":"2024-03-31T23:09:17.184670+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_clean_scrub_stamp":"2024-03-31T23:09:09.564263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:46:45.700207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":23,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-31T23:09:17.535767+0000","last_change":"2024-03-31T23:09:11.177857+0000","last_active":"2024-03-31T23:09:17.535767+0000","last_peered":"2024-03-31T23:09:17.535767+0000","last_clean":"2024-03-31T23:09:17.535767+0000","last_became_active":"2024-03-31T23:09:11.177513+0000","last_became_peered":"2024-03-31T23:09:11.177513+0000","last_unstale":"2024-03-31T23:09:17.535767+0000","last_undegraded":"2024-03-31T23:09:17.535767+0000","last_fullsized":"2024-03-31T23:09:17.535767+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T06:09:23.579208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-31T23:09:17.535426+0000","last_change":"2024-03-31T23:09:15.203412+0000","last_active":"2024-03-31T23:09:17.535426+0000","last_peered":"2024-03-31T23:09:17.535426+0000","last_clean":"2024-03-31T23:09:17.535426+0000","last_became_active":"2024-03-31T23:09:15.202983+0000","last_became_peered":"2024-03-31T23:09:15.202983+0000","last_unstale":"2024-03-31T23:09:17.535426+0000","last_undegraded":"2024-03-31T23:09:17.535426+0000","last_fullsized":"2024-03-31T23:09:17.535426+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_clean_scrub_stamp":"2024-03-31T23:09:14.171251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:57:46.357378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"91'1","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-31T23:09:17.536372+0000","last_change":"2024-03-31T23:09:10.593299+0000","last_active":"2024-03-31T23:09:17.536372+0000","last_peered":"2024-03-31T23:09:17.536372+0000","last_clean":"2024-03-31T23:09:17.536372+0000","last_became_active":"2024-03-31T23:09:10.593007+0000","last_became_peered":"2024-03-31T23:09:10.593007+0000","last_unstale":"2024-03-31T23:09:17.536372+0000","last_undegraded":"2024-03-31T23:09:17.536372+0000","last_fullsized":"2024-03-31T23:09:17.536372+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_clean_scrub_stamp":"2024-03-31T23:09:09.564263+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:15:07.199645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":23,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-31T23:09:17.535583+0000","last_change":"2024-03-31T23:09:11.176846+0000","last_active":"2024-03-31T23:09:17.535583+0000","last_peered":"2024-03-31T23:09:17.535583+0000","last_clean":"2024-03-31T23:09:17.535583+0000","last_became_active":"2024-03-31T23:09:11.176554+0000","last_became_peered":"2024-03-31T23:09:11.176554+0000","last_unstale":"2024-03-31T23:09:17.535583+0000","last_undegraded":"2024-03-31T23:09:17.535583+0000","last_fullsized":"2024-03-31T23:09:17.535583+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:25:23.639093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-31T23:09:17.535659+0000","last_change":"2024-03-31T23:09:15.746599+0000","last_active":"2024-03-31T23:09:17.535659+0000","last_peered":"2024-03-31T23:09:17.535659+0000","last_clean":"2024-03-31T23:09:17.535659+0000","last_became_active":"2024-03-31T23:09:15.745879+0000","last_became_peered":"2024-03-31T23:09:15.745879+0000","last_unstale":"2024-03-31T23:09:17.535659+0000","last_undegraded":"2024-03-31T23:09:17.535659+0000","last_fullsized":"2024-03-31T23:09:17.535659+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_clean_scrub_stamp":"2024-03-31T23:09:14.171251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T08:02:25.198610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":23,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-31T23:09:17.536136+0000","last_change":"2024-03-31T23:09:11.175546+0000","last_active":"2024-03-31T23:09:17.536136+0000","last_peered":"2024-03-31T23:09:17.536136+0000","last_clean":"2024-03-31T23:09:17.536136+0000","last_became_active":"2024-03-31T23:09:11.175253+0000","last_became_peered":"2024-03-31T23:09:11.175253+0000","last_unstale":"2024-03-31T23:09:17.536136+0000","last_undegraded":"2024-03-31T23:09:17.536136+0000","last_fullsized":"2024-03-31T23:09:17.536136+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T10:27:31.753102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"91'1","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-31T23:09:17.535146+0000","last_change":"2024-03-31T23:09:10.580888+0000","last_active":"2024-03-31T23:09:17.535146+0000","last_peered":"2024-03-31T23:09:17.535146+0000","last_clean":"2024-03-31T23:09:17.535146+0000","last_became_active":"2024-03-31T23:09:10.580557+0000","last_became_peered":"2024-03-31T23:09:10.580557+0000","last_unstale":"2024-03-31T23:09:17.535146+0000","last_undegraded":"2024-03-31T23:09:17.535146+0000","last_fullsized":"2024-03-31T23:09:17.535146+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_clean_scrub_stamp":"2024-03-31T23:09:09.564263+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:17:38.878570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-31T23:09:17.535437+0000","last_change":"2024-03-31T23:09:15.205282+0000","last_active":"2024-03-31T23:09:17.535437+0000","last_peered":"2024-03-31T23:09:17.535437+0000","last_clean":"2024-03-31T23:09:17.535437+0000","last_became_active":"2024-03-31T23:09:15.204873+0000","last_became_peered":"2024-03-31T23:09:15.204873+0000","last_unstale":"2024-03-31T23:09:17.535437+0000","last_undegraded":"2024-03-31T23:09:17.535437+0000","last_fullsized":"2024-03-31T23:09:17.535437+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_clean_scrub_stamp":"2024-03-31T23:09:14.171251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:56:05.697685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":23,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-31T23:09:17.388276+0000","last_change":"2024-03-31T23:09:11.182155+0000","last_active":"2024-03-31T23:09:17.388276+0000","last_peered":"2024-03-31T23:09:17.388276+0000","last_clean":"2024-03-31T23:09:17.388276+0000","last_became_active":"2024-03-31T23:09:11.181829+0000","last_became_peered":"2024-03-31T23:09:11.181829+0000","last_unstale":"2024-03-31T23:09:17.388276+0000","last_undegraded":"2024-03-31T23:09:17.388276+0000","last_fullsized":"2024-03-31T23:09:17.388276+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:19:22.832440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-31T23:09:17.185653+0000","last_change":"2024-03-31T23:09:10.590451+0000","last_active":"2024-03-31T23:09:17.185653+0000","last_peered":"2024-03-31T23:09:17.185653+0000","last_clean":"2024-03-31T23:09:17.185653+0000","last_became_active":"2024-03-31T23:09:10.590123+0000","last_became_peered":"2024-03-31T23:09:10.590123+0000","last_unstale":"2024-03-31T23:09:17.185653+0000","last_undegraded":"2024-03-31T23:09:17.185653+0000","last_fullsized":"2024-03-31T23:09:17.185653+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_clean_scrub_stamp":"2024-03-31T23:09:09.564263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:27:02.724456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":360,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-31T23:09:17.535950+0000","last_change":"2024-03-31T23:07:57.920942+0000","last_active":"2024-03-31T23:09:17.535950+0000","last_peered":"2024-03-31T23:09:17.535950+0000","last_clean":"2024-03-31T23:09:17.535950+0000","last_became_active":"2024-03-31T23:07:57.920381+0000","last_became_peered":"2024-03-31T23:07:57.920381+0000","last_unstale":"2024-03-31T23:09:17.535950+0000","last_undegraded":"2024-03-31T23:09:17.535950+0000","last_fullsized":"2024-03-31T23:09:17.535950+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:04:33.508180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:04:33.508180+0000","last_clean_scrub_stamp":"2024-03-31T23:04:33.508180+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:20:11.211403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-31T23:09:17.537121+0000","last_change":"2024-03-31T23:09:15.746822+0000","last_active":"2024-03-31T23:09:17.537121+0000","last_peered":"2024-03-31T23:09:17.537121+0000","last_clean":"2024-03-31T23:09:17.537121+0000","last_became_active":"2024-03-31T23:09:15.746368+0000","last_became_peered":"2024-03-31T23:09:15.746368+0000","last_unstale":"2024-03-31T23:09:17.537121+0000","last_undegraded":"2024-03-31T23:09:17.537121+0000","last_fullsized":"2024-03-31T23:09:17.537121+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_clean_scrub_stamp":"2024-03-31T23:09:14.171251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T06:36:35.658042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":23,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-31T23:09:17.535071+0000","last_change":"2024-03-31T23:09:11.175336+0000","last_active":"2024-03-31T23:09:17.535071+0000","last_peered":"2024-03-31T23:09:17.535071+0000","last_clean":"2024-03-31T23:09:17.535071+0000","last_became_active":"2024-03-31T23:09:11.174768+0000","last_became_peered":"2024-03-31T23:09:11.174768+0000","last_unstale":"2024-03-31T23:09:17.535071+0000","last_undegraded":"2024-03-31T23:09:17.535071+0000","last_fullsized":"2024-03-31T23:09:17.535071+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:09:11.678744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"91'3","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-31T23:09:30.397261+0000","last_change":"2024-03-31T23:09:10.591807+0000","last_active":"2024-03-31T23:09:30.397261+0000","last_peered":"2024-03-31T23:09:30.397261+0000","last_clean":"2024-03-31T23:09:30.397261+0000","last_became_active":"2024-03-31T23:09:10.591407+0000","last_became_peered":"2024-03-31T23:09:10.591407+0000","last_unstale":"2024-03-31T23:09:30.397261+0000","last_undegraded":"2024-03-31T23:09:30.397261+0000","last_fullsized":"2024-03-31T23:09:30.397261+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_clean_scrub_stamp":"2024-03-31T23:09:09.564263+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T08:46:33.660717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-31T23:09:17.535906+0000","last_change":"2024-03-31T23:09:15.203879+0000","last_active":"2024-03-31T23:09:17.535906+0000","last_peered":"2024-03-31T23:09:17.535906+0000","last_clean":"2024-03-31T23:09:17.535906+0000","last_became_active":"2024-03-31T23:09:15.203412+0000","last_became_peered":"2024-03-31T23:09:15.203412+0000","last_unstale":"2024-03-31T23:09:17.535906+0000","last_undegraded":"2024-03-31T23:09:17.535906+0000","last_fullsized":"2024-03-31T23:09:17.535906+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_clean_scrub_stamp":"2024-03-31T23:09:14.171251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:03:54.486038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-31T23:09:17.536951+0000","last_change":"2024-03-31T23:09:10.596404+0000","last_active":"2024-03-31T23:09:17.536951+0000","last_peered":"2024-03-31T23:09:17.536951+0000","last_clean":"2024-03-31T23:09:17.536951+0000","last_became_active":"2024-03-31T23:09:10.596031+0000","last_became_peered":"2024-03-31T23:09:10.596031+0000","last_unstale":"2024-03-31T23:09:17.536951+0000","last_undegraded":"2024-03-31T23:09:17.536951+0000","last_fullsized":"2024-03-31T23:09:17.536951+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_clean_scrub_stamp":"2024-03-31T23:09:09.564263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:30:31.904241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":23,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-31T23:09:17.535531+0000","last_change":"2024-03-31T23:09:11.164300+0000","last_active":"2024-03-31T23:09:17.535531+0000","last_peered":"2024-03-31T23:09:17.535531+0000","last_clean":"2024-03-31T23:09:17.535531+0000","last_became_active":"2024-03-31T23:09:11.163960+0000","last_became_peered":"2024-03-31T23:09:11.163960+0000","last_unstale":"2024-03-31T23:09:17.535531+0000","last_undegraded":"2024-03-31T23:09:17.535531+0000","last_fullsized":"2024-03-31T23:09:17.535531+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T10:37:30.512216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-31T23:09:17.388290+0000","last_change":"2024-03-31T23:09:15.746426+0000","last_active":"2024-03-31T23:09:17.388290+0000","last_peered":"2024-03-31T23:09:17.388290+0000","last_clean":"2024-03-31T23:09:17.388290+0000","last_became_active":"2024-03-31T23:09:15.745887+0000","last_became_peered":"2024-03-31T23:09:15.745887+0000","last_unstale":"2024-03-31T23:09:17.388290+0000","last_undegraded":"2024-03-31T23:09:17.388290+0000","last_fullsized":"2024-03-31T23:09:17.388290+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_clean_scrub_stamp":"2024-03-31T23:09:14.171251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:41:52.005300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"87'1","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-31T23:09:17.184977+0000","last_change":"2024-03-31T23:09:10.590631+0000","last_active":"2024-03-31T23:09:17.184977+0000","last_peered":"2024-03-31T23:09:17.184977+0000","last_clean":"2024-03-31T23:09:17.184977+0000","last_became_active":"2024-03-31T23:09:10.590211+0000","last_became_peered":"2024-03-31T23:09:10.590211+0000","last_unstale":"2024-03-31T23:09:17.184977+0000","last_undegraded":"2024-03-31T23:09:17.184977+0000","last_fullsized":"2024-03-31T23:09:17.184977+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_clean_scrub_stamp":"2024-03-31T23:09:09.564263+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T08:51:24.978892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":23,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-31T23:09:17.388575+0000","last_change":"2024-03-31T23:09:11.177353+0000","last_active":"2024-03-31T23:09:17.388575+0000","last_peered":"2024-03-31T23:09:17.388575+0000","last_clean":"2024-03-31T23:09:17.388575+0000","last_became_active":"2024-03-31T23:09:11.176996+0000","last_became_peered":"2024-03-31T23:09:11.176996+0000","last_unstale":"2024-03-31T23:09:17.388575+0000","last_undegraded":"2024-03-31T23:09:17.388575+0000","last_fullsized":"2024-03-31T23:09:17.388575+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:20:52.486281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-31T23:09:17.186103+0000","last_change":"2024-03-31T23:09:15.198528+0000","last_active":"2024-03-31T23:09:17.186103+0000","last_peered":"2024-03-31T23:09:17.186103+0000","last_clean":"2024-03-31T23:09:17.186103+0000","last_became_active":"2024-03-31T23:09:15.197938+0000","last_became_peered":"2024-03-31T23:09:15.197938+0000","last_unstale":"2024-03-31T23:09:17.186103+0000","last_undegraded":"2024-03-31T23:09:17.186103+0000","last_fullsized":"2024-03-31T23:09:17.186103+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_clean_scrub_stamp":"2024-03-31T23:09:14.171251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:45:46.567268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"85'2","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-31T23:09:17.185980+0000","last_change":"2024-03-31T23:09:10.584779+0000","last_active":"2024-03-31T23:09:17.185980+0000","last_peered":"2024-03-31T23:09:17.185980+0000","last_clean":"2024-03-31T23:09:17.185980+0000","last_became_active":"2024-03-31T23:09:10.584400+0000","last_became_peered":"2024-03-31T23:09:10.584400+0000","last_unstale":"2024-03-31T23:09:17.185980+0000","last_undegraded":"2024-03-31T23:09:17.185980+0000","last_fullsized":"2024-03-31T23:09:17.185980+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_clean_scrub_stamp":"2024-03-31T23:09:09.564263+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:10:10.095704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":23,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-31T23:09:17.770617+0000","last_change":"2024-03-31T23:09:11.177152+0000","last_active":"2024-03-31T23:09:17.770617+0000","last_peered":"2024-03-31T23:09:17.770617+0000","last_clean":"2024-03-31T23:09:17.770617+0000","last_became_active":"2024-03-31T23:09:11.176797+0000","last_became_peered":"2024-03-31T23:09:11.176797+0000","last_unstale":"2024-03-31T23:09:17.770617+0000","last_undegraded":"2024-03-31T23:09:17.770617+0000","last_fullsized":"2024-03-31T23:09:17.770617+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:19:49.070930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-31T23:09:17.185904+0000","last_change":"2024-03-31T23:09:15.746525+0000","last_active":"2024-03-31T23:09:17.185904+0000","last_peered":"2024-03-31T23:09:17.185904+0000","last_clean":"2024-03-31T23:09:17.185904+0000","last_became_active":"2024-03-31T23:09:15.746126+0000","last_became_peered":"2024-03-31T23:09:15.746126+0000","last_unstale":"2024-03-31T23:09:17.185904+0000","last_undegraded":"2024-03-31T23:09:17.185904+0000","last_fullsized":"2024-03-31T23:09:17.185904+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_clean_scrub_stamp":"2024-03-31T23:09:14.171251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:04:57.313615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"91'2","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-31T23:09:29.423929+0000","last_change":"2024-03-31T23:09:10.591469+0000","last_active":"2024-03-31T23:09:29.423929+0000","last_peered":"2024-03-31T23:09:29.423929+0000","last_clean":"2024-03-31T23:09:29.423929+0000","last_became_active":"2024-03-31T23:09:10.591166+0000","last_became_peered":"2024-03-31T23:09:10.591166+0000","last_unstale":"2024-03-31T23:09:29.423929+0000","last_undegraded":"2024-03-31T23:09:29.423929+0000","last_fullsized":"2024-03-31T23:09:29.423929+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_clean_scrub_stamp":"2024-03-31T23:09:09.564263+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T10:30:40.279041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":23,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-31T23:09:17.536582+0000","last_change":"2024-03-31T23:09:11.174471+0000","last_active":"2024-03-31T23:09:17.536582+0000","last_peered":"2024-03-31T23:09:17.536582+0000","last_clean":"2024-03-31T23:09:17.536582+0000","last_became_active":"2024-03-31T23:09:11.174154+0000","last_became_peered":"2024-03-31T23:09:11.174154+0000","last_unstale":"2024-03-31T23:09:17.536582+0000","last_undegraded":"2024-03-31T23:09:17.536582+0000","last_fullsized":"2024-03-31T23:09:17.536582+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T06:10:14.727009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-31T23:09:17.185327+0000","last_change":"2024-03-31T23:09:15.201416+0000","last_active":"2024-03-31T23:09:17.185327+0000","last_peered":"2024-03-31T23:09:17.185327+0000","last_clean":"2024-03-31T23:09:17.185327+0000","last_became_active":"2024-03-31T23:09:15.200987+0000","last_became_peered":"2024-03-31T23:09:15.200987+0000","last_unstale":"2024-03-31T23:09:17.185327+0000","last_undegraded":"2024-03-31T23:09:17.185327+0000","last_fullsized":"2024-03-31T23:09:17.185327+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_clean_scrub_stamp":"2024-03-31T23:09:14.171251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T10:56:54.872988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-31T23:09:17.388235+0000","last_change":"2024-03-31T23:09:10.594035+0000","last_active":"2024-03-31T23:09:17.388235+0000","last_peered":"2024-03-31T23:09:17.388235+0000","last_clean":"2024-03-31T23:09:17.388235+0000","last_became_active":"2024-03-31T23:09:10.593710+0000","last_became_peered":"2024-03-31T23:09:10.593710+0000","last_unstale":"2024-03-31T23:09:17.388235+0000","last_undegraded":"2024-03-31T23:09:17.388235+0000","last_fullsized":"2024-03-31T23:09:17.388235+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_clean_scrub_stamp":"2024-03-31T23:09:09.564263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:46:48.580244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":23,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-31T23:09:17.535932+0000","last_change":"2024-03-31T23:09:11.170501+0000","last_active":"2024-03-31T23:09:17.535932+0000","last_peered":"2024-03-31T23:09:17.535932+0000","last_clean":"2024-03-31T23:09:17.535932+0000","last_became_active":"2024-03-31T23:09:11.170068+0000","last_became_peered":"2024-03-31T23:09:11.170068+0000","last_unstale":"2024-03-31T23:09:17.535932+0000","last_undegraded":"2024-03-31T23:09:17.535932+0000","last_fullsized":"2024-03-31T23:09:17.535932+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T06:05:53.488957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-31T23:09:17.185840+0000","last_change":"2024-03-31T23:09:15.208092+0000","last_active":"2024-03-31T23:09:17.185840+0000","last_peered":"2024-03-31T23:09:17.185840+0000","last_clean":"2024-03-31T23:09:17.185840+0000","last_became_active":"2024-03-31T23:09:15.207750+0000","last_became_peered":"2024-03-31T23:09:15.207750+0000","last_unstale":"2024-03-31T23:09:17.185840+0000","last_undegraded":"2024-03-31T23:09:17.185840+0000","last_fullsized":"2024-03-31T23:09:17.185840+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_clean_scrub_stamp":"2024-03-31T23:09:14.171251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:24:39.649569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-31T23:09:17.186198+0000","last_change":"2024-03-31T23:09:10.591275+0000","last_active":"2024-03-31T23:09:17.186198+0000","last_peered":"2024-03-31T23:09:17.186198+0000","last_clean":"2024-03-31T23:09:17.186198+0000","last_became_active":"2024-03-31T23:09:10.590652+0000","last_became_peered":"2024-03-31T23:09:10.590652+0000","last_unstale":"2024-03-31T23:09:17.186198+0000","last_undegraded":"2024-03-31T23:09:17.186198+0000","last_fullsized":"2024-03-31T23:09:17.186198+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_clean_scrub_stamp":"2024-03-31T23:09:09.564263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:58:39.037196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":23,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-31T23:09:17.388365+0000","last_change":"2024-03-31T23:09:11.175169+0000","last_active":"2024-03-31T23:09:17.388365+0000","last_peered":"2024-03-31T23:09:17.388365+0000","last_clean":"2024-03-31T23:09:17.388365+0000","last_became_active":"2024-03-31T23:09:11.174849+0000","last_became_peered":"2024-03-31T23:09:11.174849+0000","last_unstale":"2024-03-31T23:09:17.388365+0000","last_undegraded":"2024-03-31T23:09:17.388365+0000","last_fullsized":"2024-03-31T23:09:17.388365+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:58:06.814034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-31T23:09:17.185805+0000","last_change":"2024-03-31T23:09:15.746423+0000","last_active":"2024-03-31T23:09:17.185805+0000","last_peered":"2024-03-31T23:09:17.185805+0000","last_clean":"2024-03-31T23:09:17.185805+0000","last_became_active":"2024-03-31T23:09:15.745942+0000","last_became_peered":"2024-03-31T23:09:15.745942+0000","last_unstale":"2024-03-31T23:09:17.185805+0000","last_undegraded":"2024-03-31T23:09:17.185805+0000","last_fullsized":"2024-03-31T23:09:17.185805+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_clean_scrub_stamp":"2024-03-31T23:09:14.171251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:56:02.903212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"85'2","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-31T23:09:17.536825+0000","last_change":"2024-03-31T23:09:10.597524+0000","last_active":"2024-03-31T23:09:17.536825+0000","last_peered":"2024-03-31T23:09:17.536825+0000","last_clean":"2024-03-31T23:09:17.536825+0000","last_became_active":"2024-03-31T23:09:10.597222+0000","last_became_peered":"2024-03-31T23:09:10.597222+0000","last_unstale":"2024-03-31T23:09:17.536825+0000","last_undegraded":"2024-03-31T23:09:17.536825+0000","last_fullsized":"2024-03-31T23:09:17.536825+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_clean_scrub_stamp":"2024-03-31T23:09:09.564263+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T09:24:20.160978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":23,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-31T23:09:17.536417+0000","last_change":"2024-03-31T23:09:11.170440+0000","last_active":"2024-03-31T23:09:17.536417+0000","last_peered":"2024-03-31T23:09:17.536417+0000","last_clean":"2024-03-31T23:09:17.536417+0000","last_became_active":"2024-03-31T23:09:11.170081+0000","last_became_peered":"2024-03-31T23:09:11.170081+0000","last_unstale":"2024-03-31T23:09:17.536417+0000","last_undegraded":"2024-03-31T23:09:17.536417+0000","last_fullsized":"2024-03-31T23:09:17.536417+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:53:31.429822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-31T23:09:17.536228+0000","last_change":"2024-03-31T23:09:15.207146+0000","last_active":"2024-03-31T23:09:17.536228+0000","last_peered":"2024-03-31T23:09:17.536228+0000","last_clean":"2024-03-31T23:09:17.536228+0000","last_became_active":"2024-03-31T23:09:15.206769+0000","last_became_peered":"2024-03-31T23:09:15.206769+0000","last_unstale":"2024-03-31T23:09:17.536228+0000","last_undegraded":"2024-03-31T23:09:17.536228+0000","last_fullsized":"2024-03-31T23:09:17.536228+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_clean_scrub_stamp":"2024-03-31T23:09:14.171251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T07:42:12.961109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-31T23:09:17.186427+0000","last_change":"2024-03-31T23:09:10.587291+0000","last_active":"2024-03-31T23:09:17.186427+0000","last_peered":"2024-03-31T23:09:17.186427+0000","last_clean":"2024-03-31T23:09:17.186427+0000","last_became_active":"2024-03-31T23:09:10.585992+0000","last_became_peered":"2024-03-31T23:09:10.585992+0000","last_unstale":"2024-03-31T23:09:17.186427+0000","last_undegraded":"2024-03-31T23:09:17.186427+0000","last_fullsized":"2024-03-31T23:09:17.186427+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_clean_scrub_stamp":"2024-03-31T23:09:09.564263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:28:00.786268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":23,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-31T23:09:17.535786+0000","last_change":"2024-03-31T23:09:11.179369+0000","last_active":"2024-03-31T23:09:17.535786+0000","last_peered":"2024-03-31T23:09:17.535786+0000","last_clean":"2024-03-31T23:09:17.535786+0000","last_became_active":"2024-03-31T23:09:11.178776+0000","last_became_peered":"2024-03-31T23:09:11.178776+0000","last_unstale":"2024-03-31T23:09:17.535786+0000","last_undegraded":"2024-03-31T23:09:17.535786+0000","last_fullsized":"2024-03-31T23:09:17.535786+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T07:56:05.201338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-31T23:09:17.770547+0000","last_change":"2024-03-31T23:09:15.201462+0000","last_active":"2024-03-31T23:09:17.770547+0000","last_peered":"2024-03-31T23:09:17.770547+0000","last_clean":"2024-03-31T23:09:17.770547+0000","last_became_active":"2024-03-31T23:09:15.201067+0000","last_became_peered":"2024-03-31T23:09:15.201067+0000","last_unstale":"2024-03-31T23:09:17.770547+0000","last_undegraded":"2024-03-31T23:09:17.770547+0000","last_fullsized":"2024-03-31T23:09:17.770547+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_clean_scrub_stamp":"2024-03-31T23:09:14.171251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:54:29.339456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"85'1","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-31T23:09:17.537351+0000","last_change":"2024-03-31T23:09:10.584752+0000","last_active":"2024-03-31T23:09:17.537351+0000","last_peered":"2024-03-31T23:09:17.537351+0000","last_clean":"2024-03-31T23:09:17.537351+0000","last_became_active":"2024-03-31T23:09:10.584432+0000","last_became_peered":"2024-03-31T23:09:10.584432+0000","last_unstale":"2024-03-31T23:09:17.537351+0000","last_undegraded":"2024-03-31T23:09:17.537351+0000","last_fullsized":"2024-03-31T23:09:17.537351+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_clean_scrub_stamp":"2024-03-31T23:09:09.564263+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T10:50:53.889719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":23,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-31T23:09:17.536462+0000","last_change":"2024-03-31T23:09:11.178540+0000","last_active":"2024-03-31T23:09:17.536462+0000","last_peered":"2024-03-31T23:09:17.536462+0000","last_clean":"2024-03-31T23:09:17.536462+0000","last_became_active":"2024-03-31T23:09:11.178171+0000","last_became_peered":"2024-03-31T23:09:11.178171+0000","last_unstale":"2024-03-31T23:09:17.536462+0000","last_undegraded":"2024-03-31T23:09:17.536462+0000","last_fullsized":"2024-03-31T23:09:17.536462+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:28:46.287915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-31T23:09:17.535754+0000","last_change":"2024-03-31T23:09:15.197271+0000","last_active":"2024-03-31T23:09:17.535754+0000","last_peered":"2024-03-31T23:09:17.535754+0000","last_clean":"2024-03-31T23:09:17.535754+0000","last_became_active":"2024-03-31T23:09:15.196339+0000","last_became_peered":"2024-03-31T23:09:15.196339+0000","last_unstale":"2024-03-31T23:09:17.535754+0000","last_undegraded":"2024-03-31T23:09:17.535754+0000","last_fullsized":"2024-03-31T23:09:17.535754+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_clean_scrub_stamp":"2024-03-31T23:09:14.171251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:44:13.044278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"85'2","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-31T23:09:17.535586+0000","last_change":"2024-03-31T23:09:10.595131+0000","last_active":"2024-03-31T23:09:17.535586+0000","last_peered":"2024-03-31T23:09:17.535586+0000","last_clean":"2024-03-31T23:09:17.535586+0000","last_became_active":"2024-03-31T23:09:10.594740+0000","last_became_peered":"2024-03-31T23:09:10.594740+0000","last_unstale":"2024-03-31T23:09:17.535586+0000","last_undegraded":"2024-03-31T23:09:17.535586+0000","last_fullsized":"2024-03-31T23:09:17.535586+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_clean_scrub_stamp":"2024-03-31T23:09:09.564263+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:52:41.849621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":23,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-31T23:09:17.535797+0000","last_change":"2024-03-31T23:09:11.177530+0000","last_active":"2024-03-31T23:09:17.535797+0000","last_peered":"2024-03-31T23:09:17.535797+0000","last_clean":"2024-03-31T23:09:17.535797+0000","last_became_active":"2024-03-31T23:09:11.176972+0000","last_became_peered":"2024-03-31T23:09:11.176972+0000","last_unstale":"2024-03-31T23:09:17.535797+0000","last_undegraded":"2024-03-31T23:09:17.535797+0000","last_fullsized":"2024-03-31T23:09:17.535797+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T07:57:21.094911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":23,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-31T23:09:17.536072+0000","last_change":"2024-03-31T23:09:11.172792+0000","last_active":"2024-03-31T23:09:17.536072+0000","last_peered":"2024-03-31T23:09:17.536072+0000","last_clean":"2024-03-31T23:09:17.536072+0000","last_became_active":"2024-03-31T23:09:11.172463+0000","last_became_peered":"2024-03-31T23:09:11.172463+0000","last_unstale":"2024-03-31T23:09:17.536072+0000","last_undegraded":"2024-03-31T23:09:17.536072+0000","last_fullsized":"2024-03-31T23:09:17.536072+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:07:28.789783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":23,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-31T23:09:17.186302+0000","last_change":"2024-03-31T23:09:11.174331+0000","last_active":"2024-03-31T23:09:17.186302+0000","last_peered":"2024-03-31T23:09:17.186302+0000","last_clean":"2024-03-31T23:09:17.186302+0000","last_became_active":"2024-03-31T23:09:11.174001+0000","last_became_peered":"2024-03-31T23:09:11.174001+0000","last_unstale":"2024-03-31T23:09:17.186302+0000","last_undegraded":"2024-03-31T23:09:17.186302+0000","last_fullsized":"2024-03-31T23:09:17.186302+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:26:28.005406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":23,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-31T23:09:17.185924+0000","last_change":"2024-03-31T23:09:11.180047+0000","last_active":"2024-03-31T23:09:17.185924+0000","last_peered":"2024-03-31T23:09:17.185924+0000","last_clean":"2024-03-31T23:09:17.185924+0000","last_became_active":"2024-03-31T23:09:11.179691+0000","last_became_peered":"2024-03-31T23:09:11.179691+0000","last_unstale":"2024-03-31T23:09:17.185924+0000","last_undegraded":"2024-03-31T23:09:17.185924+0000","last_fullsized":"2024-03-31T23:09:17.185924+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:28:05.083710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":23,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-31T23:09:17.770788+0000","last_change":"2024-03-31T23:09:11.177642+0000","last_active":"2024-03-31T23:09:17.770788+0000","last_peered":"2024-03-31T23:09:17.770788+0000","last_clean":"2024-03-31T23:09:17.770788+0000","last_became_active":"2024-03-31T23:09:11.177261+0000","last_became_peered":"2024-03-31T23:09:11.177261+0000","last_unstale":"2024-03-31T23:09:17.770788+0000","last_undegraded":"2024-03-31T23:09:17.770788+0000","last_fullsized":"2024-03-31T23:09:17.770788+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T07:10:43.616863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":23,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-31T23:09:17.535424+0000","last_change":"2024-03-31T23:09:11.173606+0000","last_active":"2024-03-31T23:09:17.535424+0000","last_peered":"2024-03-31T23:09:17.535424+0000","last_clean":"2024-03-31T23:09:17.535424+0000","last_became_active":"2024-03-31T23:09:11.173304+0000","last_became_peered":"2024-03-31T23:09:11.173304+0000","last_unstale":"2024-03-31T23:09:17.535424+0000","last_undegraded":"2024-03-31T23:09:17.535424+0000","last_fullsized":"2024-03-31T23:09:17.535424+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:53:06.112181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":23,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-31T23:09:17.536187+0000","last_change":"2024-03-31T23:09:11.176628+0000","last_active":"2024-03-31T23:09:17.536187+0000","last_peered":"2024-03-31T23:09:17.536187+0000","last_clean":"2024-03-31T23:09:17.536187+0000","last_became_active":"2024-03-31T23:09:11.176318+0000","last_became_peered":"2024-03-31T23:09:11.176318+0000","last_unstale":"2024-03-31T23:09:17.536187+0000","last_undegraded":"2024-03-31T23:09:17.536187+0000","last_fullsized":"2024-03-31T23:09:17.536187+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:06:40.723675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":23,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-31T23:09:17.186737+0000","last_change":"2024-03-31T23:09:11.169318+0000","last_active":"2024-03-31T23:09:17.186737+0000","last_peered":"2024-03-31T23:09:17.186737+0000","last_clean":"2024-03-31T23:09:17.186737+0000","last_became_active":"2024-03-31T23:09:11.169026+0000","last_became_peered":"2024-03-31T23:09:11.169026+0000","last_unstale":"2024-03-31T23:09:17.186737+0000","last_undegraded":"2024-03-31T23:09:17.186737+0000","last_fullsized":"2024-03-31T23:09:17.186737+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:02:18.562854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":23,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-31T23:09:17.388686+0000","last_change":"2024-03-31T23:09:11.184670+0000","last_active":"2024-03-31T23:09:17.388686+0000","last_peered":"2024-03-31T23:09:17.388686+0000","last_clean":"2024-03-31T23:09:17.388686+0000","last_became_active":"2024-03-31T23:09:11.184354+0000","last_became_peered":"2024-03-31T23:09:11.184354+0000","last_unstale":"2024-03-31T23:09:17.388686+0000","last_undegraded":"2024-03-31T23:09:17.388686+0000","last_fullsized":"2024-03-31T23:09:17.388686+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:16:14.354947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":23,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-31T23:09:17.536042+0000","last_change":"2024-03-31T23:09:11.180991+0000","last_active":"2024-03-31T23:09:17.536042+0000","last_peered":"2024-03-31T23:09:17.536042+0000","last_clean":"2024-03-31T23:09:17.536042+0000","last_became_active":"2024-03-31T23:09:11.180667+0000","last_became_peered":"2024-03-31T23:09:11.180667+0000","last_unstale":"2024-03-31T23:09:17.536042+0000","last_undegraded":"2024-03-31T23:09:17.536042+0000","last_fullsized":"2024-03-31T23:09:17.536042+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T06:35:33.157176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":23,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-31T23:09:17.388440+0000","last_change":"2024-03-31T23:09:11.182025+0000","last_active":"2024-03-31T23:09:17.388440+0000","last_peered":"2024-03-31T23:09:17.388440+0000","last_clean":"2024-03-31T23:09:17.388440+0000","last_became_active":"2024-03-31T23:09:11.179310+0000","last_became_peered":"2024-03-31T23:09:11.179310+0000","last_unstale":"2024-03-31T23:09:17.388440+0000","last_undegraded":"2024-03-31T23:09:17.388440+0000","last_fullsized":"2024-03-31T23:09:17.388440+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T10:21:40.784738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":23,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-31T23:09:17.536489+0000","last_change":"2024-03-31T23:09:11.176594+0000","last_active":"2024-03-31T23:09:17.536489+0000","last_peered":"2024-03-31T23:09:17.536489+0000","last_clean":"2024-03-31T23:09:17.536489+0000","last_became_active":"2024-03-31T23:09:11.176306+0000","last_became_peered":"2024-03-31T23:09:11.176306+0000","last_unstale":"2024-03-31T23:09:17.536489+0000","last_undegraded":"2024-03-31T23:09:17.536489+0000","last_fullsized":"2024-03-31T23:09:17.536489+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T09:08:50.736985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":23,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-31T23:09:17.535382+0000","last_change":"2024-03-31T23:09:11.172585+0000","last_active":"2024-03-31T23:09:17.535382+0000","last_peered":"2024-03-31T23:09:17.535382+0000","last_clean":"2024-03-31T23:09:17.535382+0000","last_became_active":"2024-03-31T23:09:11.172266+0000","last_became_peered":"2024-03-31T23:09:11.172266+0000","last_unstale":"2024-03-31T23:09:17.535382+0000","last_undegraded":"2024-03-31T23:09:17.535382+0000","last_fullsized":"2024-03-31T23:09:17.535382+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:38:26.570175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":23,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-31T23:09:17.536165+0000","last_change":"2024-03-31T23:09:11.181434+0000","last_active":"2024-03-31T23:09:17.536165+0000","last_peered":"2024-03-31T23:09:17.536165+0000","last_clean":"2024-03-31T23:09:17.536165+0000","last_became_active":"2024-03-31T23:09:11.181140+0000","last_became_peered":"2024-03-31T23:09:11.181140+0000","last_unstale":"2024-03-31T23:09:17.536165+0000","last_undegraded":"2024-03-31T23:09:17.536165+0000","last_fullsized":"2024-03-31T23:09:17.536165+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:55:06.379696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4157440,"data_stored":4132576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":6325,"num_objects":32,"num_object_clones":0,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":72,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":20,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":143360,"data_stored":18159,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":67,"ondisk_log_size":67,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":81,"seq":347892350993,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36324,"kb_used_data":988,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708828,"statfs":{"total":95995035648,"available":95957839872,"internally_reserved":0,"allocated":1011712,"data_stored":491116,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":36174276},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":73,"seq":313532612630,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28608,"kb_used_data":1592,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716544,"statfs":{"total":95995035648,"available":95965741056,"internally_reserved":0,"allocated":1630208,"data_stored":1085520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":68,"seq":292057776155,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28012,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717140,"statfs":{"total":95995035648,"available":95966351360,"internally_reserved":0,"allocated":1015808,"data_stored":491698,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":60,"seq":257698037791,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28016,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717136,"statfs":{"total":95995035648,"available":95966347264,"internally_reserved":0,"allocated":1019904,"data_stored":491720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299429,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28600,"kb_used_data":1584,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716552,"statfs":{"total":95995035648,"available":95965749248,"internally_reserved":0,"allocated":1622016,"data_stored":1083276,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1588,"internal_metadata":27654604},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863462954,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28004,"kb_used_data":988,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717148,"statfs":{"total":95995035648,"available":95966359552,"internally_reserved":0,"allocated":1011712,"data_stored":491116,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574914,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28016,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717136,"statfs":{"total":95995035648,"available":95966347264,"internally_reserved":0,"allocated":1024000,"data_stored":494958,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738439,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36340,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708812,"statfs":{"total":95995035648,"available":95957823488,"internally_reserved":0,"allocated":1015808,"data_stored":491206,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411390,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28020,"kb_used_data":1004,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717132,"statfs":{"total":95995035648,"available":95966343168,"internally_reserved":0,"allocated":1028096,"data_stored":494992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182457,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28020,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717132,"statfs":{"total":95995035648,"available":95966343168,"internally_reserved":0,"allocated":1019904,"data_stored":491220,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920820,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32176,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712976,"statfs":{"total":95995035648,"available":95962087424,"internally_reserved":0,"allocated":1015808,"data_stored":491138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724592,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36932,"kb_used_data":1596,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708220,"statfs":{"total":95995035648,"available":95957217280,"internally_reserved":0,"allocated":1634304,"data_stored":1082960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":3876,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1867,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":4798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-31T23:09:34.191 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-31T23:09:34.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:34.328+0000 7f91ea61c640 1 -- 172.21.15.46:0/847747261 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f91e4101780 msgr2=0x7f91e4101ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:09:34.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:34.328+0000 7f91ea61c640 1 --2- 172.21.15.46:0/847747261 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f91e4101780 0x7f91e4101ba0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f91d4009900 tx=0x7f91d402bc30 comp rx=0 tx=0).stop 2024-03-31T23:09:34.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:34.328+0000 7f91ea61c640 1 -- 172.21.15.46:0/847747261 shutdown_connections 2024-03-31T23:09:34.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:34.328+0000 7f91ea61c640 1 --2- 172.21.15.46:0/847747261 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f91e4103330 0x7f91e4109bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:34.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:34.328+0000 7f91ea61c640 1 --2- 172.21.15.46:0/847747261 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f91e4102970 0x7f91e4102df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:34.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:34.328+0000 7f91ea61c640 1 --2- 172.21.15.46:0/847747261 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f91e4101780 0x7f91e4101ba0 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:34.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:34.328+0000 7f91ea61c640 1 -- 172.21.15.46:0/847747261 >> 172.21.15.46:0/847747261 conn(0x7f91e40fcd30 msgr2=0x7f91e40ff150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:09:34.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:34.328+0000 7f91ea61c640 1 -- 172.21.15.46:0/847747261 shutdown_connections 2024-03-31T23:09:34.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:34.328+0000 7f91ea61c640 1 -- 172.21.15.46:0/847747261 wait complete. 2024-03-31T23:09:34.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:34.328+0000 7f91ea61c640 1 Processor -- start 2024-03-31T23:09:34.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:34.329+0000 7f91ea61c640 1 -- start start 2024-03-31T23:09:34.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:34.329+0000 7f91ea61c640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f91e4101780 0x7f91e4075500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:09:34.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:34.329+0000 7f91ea61c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f91e4102970 0x7f91e4073b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:09:34.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:34.329+0000 7f91ea61c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f91e4103330 0x7f91e4074080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:09:34.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:34.329+0000 7f91ea61c640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f91e410d280 con 0x7f91e4102970 2024-03-31T23:09:34.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:34.329+0000 7f91ea61c640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f91e410d100 con 0x7f91e4101780 2024-03-31T23:09:34.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:34.329+0000 7f91ea61c640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f91e410d400 con 0x7f91e4103330 2024-03-31T23:09:34.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:34.329+0000 7f91e37fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f91e4102970 0x7f91e4073b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:09:34.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:34.329+0000 7f91e3fff640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f91e4101780 0x7f91e4075500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:09:34.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:34.329+0000 7f91e37fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f91e4102970 0x7f91e4073b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:57118/0 (socket says 172.21.15.46:57118) 2024-03-31T23:09:34.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:34.329+0000 7f91e37fe640 1 -- 172.21.15.46:0/1667845929 learned_addr learned my addr 172.21.15.46:0/1667845929 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:09:34.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:34.329+0000 7f91e8bb9640 1 --2- 172.21.15.46:0/1667845929 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f91e4103330 0x7f91e4074080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:09:34.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:34.329+0000 7f91e37fe640 1 -- 172.21.15.46:0/1667845929 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f91e4103330 msgr2=0x7f91e4074080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:09:34.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:34.329+0000 7f91e37fe640 1 --2- 172.21.15.46:0/1667845929 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f91e4103330 0x7f91e4074080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:34.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:34.329+0000 7f91e37fe640 1 -- 172.21.15.46:0/1667845929 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f91e4101780 msgr2=0x7f91e4075500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:09:34.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:34.329+0000 7f91e37fe640 1 --2- 172.21.15.46:0/1667845929 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f91e4101780 0x7f91e4075500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:34.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:34.329+0000 7f91e37fe640 1 -- 172.21.15.46:0/1667845929 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f91e4074910 con 0x7f91e4102970 2024-03-31T23:09:34.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:34.329+0000 7f91e3fff640 1 --2- 172.21.15.46:0/1667845929 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f91e4101780 0x7f91e4075500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:09:34.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:34.329+0000 7f91e37fe640 1 --2- 172.21.15.46:0/1667845929 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f91e4102970 0x7f91e4073b40 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f91d800ea60 tx=0x7f91d800ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:09:34.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:34.330+0000 7f91e17fa640 1 -- 172.21.15.46:0/1667845929 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91d800cde0 con 0x7f91e4102970 2024-03-31T23:09:34.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:34.330+0000 7f91ea61c640 1 -- 172.21.15.46:0/1667845929 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f91e41a1ae0 con 0x7f91e4102970 2024-03-31T23:09:34.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:34.330+0000 7f91e17fa640 1 -- 172.21.15.46:0/1667845929 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f91d80035f0 con 0x7f91e4102970 2024-03-31T23:09:34.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:34.330+0000 7f91e17fa640 1 -- 172.21.15.46:0/1667845929 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91d8010690 con 0x7f91e4102970 2024-03-31T23:09:34.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:34.330+0000 7f91ea61c640 1 -- 172.21.15.46:0/1667845929 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f91e41a2030 con 0x7f91e4102970 2024-03-31T23:09:34.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:34.331+0000 7f91ea61c640 1 -- 172.21.15.46:0/1667845929 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f91e4108bc0 con 0x7f91e4102970 2024-03-31T23:09:34.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:34.332+0000 7f91e17fa640 1 -- 172.21.15.46:0/1667845929 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 17) ==== 137371+0+0 (secure 0 0 0) 0x7f91d80037a0 con 0x7f91e4102970 2024-03-31T23:09:34.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:34.332+0000 7f91e17fa640 1 --2- 172.21.15.46:0/1667845929 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f91cc0a2530 0x7f91cc0a49f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:09:34.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:34.332+0000 7f91e3fff640 1 --2- 172.21.15.46:0/1667845929 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f91cc0a2530 0x7f91cc0a49f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:09:34.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:34.332+0000 7f91e17fa640 1 -- 172.21.15.46:0/1667845929 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8802+0+0 (secure 0 0 0) 0x7f91d80cee10 con 0x7f91e4102970 2024-03-31T23:09:34.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:34.333+0000 7f91e3fff640 1 --2- 172.21.15.46:0/1667845929 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f91cc0a2530 0x7f91cc0a49f0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f91d4005ab0 tx=0x7f91d4005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:09:34.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:34.337+0000 7f91e17fa640 1 -- 172.21.15.46:0/1667845929 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f91d80d2030 con 0x7f91e4102970 2024-03-31T23:09:34.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:34.516+0000 7f91ea61c640 1 -- 172.21.15.46:0/1667845929 --> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f91e4061b90 con 0x7f91cc0a2530 2024-03-31T23:09:34.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:34.526+0000 7f91e17fa640 1 -- 172.21.15.46:0/1667845929 <== mgr.14152 v2:172.21.15.46:6800/162289569 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340221 (secure 0 0 0) 0x7f91e4061b90 con 0x7f91cc0a2530 2024-03-31T23:09:34.530 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-31T23:09:34.532 INFO:teuthology.orchestra.run.smithi046.stderr:dumped all 2024-03-31T23:09:34.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:34.529+0000 7f91ea61c640 1 -- 172.21.15.46:0/1667845929 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f91cc0a2530 msgr2=0x7f91cc0a49f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:09:34.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:34.529+0000 7f91ea61c640 1 --2- 172.21.15.46:0/1667845929 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f91cc0a2530 0x7f91cc0a49f0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f91d4005ab0 tx=0x7f91d4005fd0 comp rx=0 tx=0).stop 2024-03-31T23:09:34.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:34.529+0000 7f91ea61c640 1 -- 172.21.15.46:0/1667845929 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f91e4102970 msgr2=0x7f91e4073b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:09:34.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:34.529+0000 7f91ea61c640 1 --2- 172.21.15.46:0/1667845929 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f91e4102970 0x7f91e4073b40 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f91d800ea60 tx=0x7f91d800ef20 comp rx=0 tx=0).stop 2024-03-31T23:09:34.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:34.529+0000 7f91ea61c640 1 -- 172.21.15.46:0/1667845929 shutdown_connections 2024-03-31T23:09:34.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:34.529+0000 7f91ea61c640 1 --2- 172.21.15.46:0/1667845929 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f91cc0a2530 0x7f91cc0a49f0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:34.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:34.529+0000 7f91ea61c640 1 --2- 172.21.15.46:0/1667845929 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f91e4103330 0x7f91e4074080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:34.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:34.529+0000 7f91ea61c640 1 --2- 172.21.15.46:0/1667845929 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f91e4102970 0x7f91e4073b40 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:34.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:34.529+0000 7f91ea61c640 1 --2- 172.21.15.46:0/1667845929 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f91e4101780 0x7f91e4075500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:34.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:34.529+0000 7f91ea61c640 1 -- 172.21.15.46:0/1667845929 >> 172.21.15.46:0/1667845929 conn(0x7f91e40fcd30 msgr2=0x7f91e4105d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:09:34.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:34.529+0000 7f91ea61c640 1 -- 172.21.15.46:0/1667845929 shutdown_connections 2024-03-31T23:09:34.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:34.529+0000 7f91ea61c640 1 -- 172.21.15.46:0/1667845929 wait complete. 2024-03-31T23:09:34.540 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:09:34 smithi046 ceph-mon[32498]: pgmap v303: 129 pgs: 129 active+clean; 583 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s rd, 853 B/s wr, 2 op/s 2024-03-31T23:09:34.540 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:09:34 smithi046 ceph-mon[32498]: from='client.14922 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-31T23:09:34.540 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:09:34 smithi046 ceph-mon[32498]: from='client.14928 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-31T23:09:34.540 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:09:34 smithi046 ceph-mon[32498]: mgrmap e17: x(active, since 7m), standbys: y, z 2024-03-31T23:09:34.540 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:09:34 smithi046 ceph-mon[32498]: from='client.? 172.21.15.46:0/2481721173' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-31T23:09:34.540 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:09:34 smithi046 ceph-mon[32498]: from='client.? 172.21.15.46:0/2013748687' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-31T23:09:34.541 INFO:teuthology.orchestra.run.smithi046.stdout:{"pg_ready":true,"pg_map":{"version":303,"stamp":"2024-03-31T23:09:32.581099+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596693,"num_objects":34,"num_object_clones":0,"num_object_copies":102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34,"num_whiteouts":0,"num_read":142,"num_read_kb":123,"num_write":253,"num_write_kb":2962,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":20,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":204,"ondisk_log_size":204,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":367068,"kb_used_data":13720,"kb_used_omap":18,"kb_used_meta":353197,"kb_avail":1124574756,"statfs":{"total":1151940427776,"available":1151564550144,"internally_reserved":0,"allocated":14049280,"data_stored":7680920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19140,"internal_metadata":361674044},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":3915,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":27,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003009"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":23,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-31T23:09:17.536388+0000","last_change":"2024-03-31T23:09:11.177502+0000","last_active":"2024-03-31T23:09:17.536388+0000","last_peered":"2024-03-31T23:09:17.536388+0000","last_clean":"2024-03-31T23:09:17.536388+0000","last_became_active":"2024-03-31T23:09:11.177134+0000","last_became_peered":"2024-03-31T23:09:11.177134+0000","last_unstale":"2024-03-31T23:09:17.536388+0000","last_undegraded":"2024-03-31T23:09:17.536388+0000","last_fullsized":"2024-03-31T23:09:17.536388+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T10:29:36.508229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":23,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-31T23:09:17.770764+0000","last_change":"2024-03-31T23:09:11.172990+0000","last_active":"2024-03-31T23:09:17.770764+0000","last_peered":"2024-03-31T23:09:17.770764+0000","last_clean":"2024-03-31T23:09:17.770764+0000","last_became_active":"2024-03-31T23:09:11.172684+0000","last_became_peered":"2024-03-31T23:09:11.172684+0000","last_unstale":"2024-03-31T23:09:17.770764+0000","last_undegraded":"2024-03-31T23:09:17.770764+0000","last_fullsized":"2024-03-31T23:09:17.770764+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T07:02:59.991757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":23,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-31T23:09:17.770500+0000","last_change":"2024-03-31T23:09:11.176805+0000","last_active":"2024-03-31T23:09:17.770500+0000","last_peered":"2024-03-31T23:09:17.770500+0000","last_clean":"2024-03-31T23:09:17.770500+0000","last_became_active":"2024-03-31T23:09:11.176468+0000","last_became_peered":"2024-03-31T23:09:11.176468+0000","last_unstale":"2024-03-31T23:09:17.770500+0000","last_undegraded":"2024-03-31T23:09:17.770500+0000","last_fullsized":"2024-03-31T23:09:17.770500+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T08:23:41.872558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":23,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-31T23:09:17.185909+0000","last_change":"2024-03-31T23:09:11.182385+0000","last_active":"2024-03-31T23:09:17.185909+0000","last_peered":"2024-03-31T23:09:17.185909+0000","last_clean":"2024-03-31T23:09:17.185909+0000","last_became_active":"2024-03-31T23:09:11.181908+0000","last_became_peered":"2024-03-31T23:09:11.181908+0000","last_unstale":"2024-03-31T23:09:17.185909+0000","last_undegraded":"2024-03-31T23:09:17.185909+0000","last_fullsized":"2024-03-31T23:09:17.185909+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:52:24.066360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":23,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-31T23:09:17.536719+0000","last_change":"2024-03-31T23:09:11.181377+0000","last_active":"2024-03-31T23:09:17.536719+0000","last_peered":"2024-03-31T23:09:17.536719+0000","last_clean":"2024-03-31T23:09:17.536719+0000","last_became_active":"2024-03-31T23:09:11.181064+0000","last_became_peered":"2024-03-31T23:09:11.181064+0000","last_unstale":"2024-03-31T23:09:17.536719+0000","last_undegraded":"2024-03-31T23:09:17.536719+0000","last_fullsized":"2024-03-31T23:09:17.536719+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:28:40.766541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":23,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-31T23:09:17.535946+0000","last_change":"2024-03-31T23:09:11.176087+0000","last_active":"2024-03-31T23:09:17.535946+0000","last_peered":"2024-03-31T23:09:17.535946+0000","last_clean":"2024-03-31T23:09:17.535946+0000","last_became_active":"2024-03-31T23:09:11.175569+0000","last_became_peered":"2024-03-31T23:09:11.175569+0000","last_unstale":"2024-03-31T23:09:17.535946+0000","last_undegraded":"2024-03-31T23:09:17.535946+0000","last_fullsized":"2024-03-31T23:09:17.535946+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T06:33:19.189909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":23,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-31T23:09:17.535508+0000","last_change":"2024-03-31T23:09:11.170289+0000","last_active":"2024-03-31T23:09:17.535508+0000","last_peered":"2024-03-31T23:09:17.535508+0000","last_clean":"2024-03-31T23:09:17.535508+0000","last_became_active":"2024-03-31T23:09:11.169959+0000","last_became_peered":"2024-03-31T23:09:11.169959+0000","last_unstale":"2024-03-31T23:09:17.535508+0000","last_undegraded":"2024-03-31T23:09:17.535508+0000","last_fullsized":"2024-03-31T23:09:17.535508+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:10:44.614767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":23,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-31T23:09:17.388094+0000","last_change":"2024-03-31T23:09:11.182447+0000","last_active":"2024-03-31T23:09:17.388094+0000","last_peered":"2024-03-31T23:09:17.388094+0000","last_clean":"2024-03-31T23:09:17.388094+0000","last_became_active":"2024-03-31T23:09:11.181629+0000","last_became_peered":"2024-03-31T23:09:11.181629+0000","last_unstale":"2024-03-31T23:09:17.388094+0000","last_undegraded":"2024-03-31T23:09:17.388094+0000","last_fullsized":"2024-03-31T23:09:17.388094+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:18:26.269451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":23,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-31T23:09:17.536065+0000","last_change":"2024-03-31T23:09:11.176853+0000","last_active":"2024-03-31T23:09:17.536065+0000","last_peered":"2024-03-31T23:09:17.536065+0000","last_clean":"2024-03-31T23:09:17.536065+0000","last_became_active":"2024-03-31T23:09:11.176413+0000","last_became_peered":"2024-03-31T23:09:11.176413+0000","last_unstale":"2024-03-31T23:09:17.536065+0000","last_undegraded":"2024-03-31T23:09:17.536065+0000","last_fullsized":"2024-03-31T23:09:17.536065+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:23:40.486947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":23,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-31T23:09:17.536433+0000","last_change":"2024-03-31T23:09:11.174902+0000","last_active":"2024-03-31T23:09:17.536433+0000","last_peered":"2024-03-31T23:09:17.536433+0000","last_clean":"2024-03-31T23:09:17.536433+0000","last_became_active":"2024-03-31T23:09:11.174385+0000","last_became_peered":"2024-03-31T23:09:11.174385+0000","last_unstale":"2024-03-31T23:09:17.536433+0000","last_undegraded":"2024-03-31T23:09:17.536433+0000","last_fullsized":"2024-03-31T23:09:17.536433+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:07:58.649222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":23,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-31T23:09:17.535686+0000","last_change":"2024-03-31T23:09:11.179536+0000","last_active":"2024-03-31T23:09:17.535686+0000","last_peered":"2024-03-31T23:09:17.535686+0000","last_clean":"2024-03-31T23:09:17.535686+0000","last_became_active":"2024-03-31T23:09:11.178521+0000","last_became_peered":"2024-03-31T23:09:11.178521+0000","last_unstale":"2024-03-31T23:09:17.535686+0000","last_undegraded":"2024-03-31T23:09:17.535686+0000","last_fullsized":"2024-03-31T23:09:17.535686+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T09:29:23.128832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":23,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-31T23:09:17.185822+0000","last_change":"2024-03-31T23:09:11.182548+0000","last_active":"2024-03-31T23:09:17.185822+0000","last_peered":"2024-03-31T23:09:17.185822+0000","last_clean":"2024-03-31T23:09:17.185822+0000","last_became_active":"2024-03-31T23:09:11.181966+0000","last_became_peered":"2024-03-31T23:09:11.181966+0000","last_unstale":"2024-03-31T23:09:17.185822+0000","last_undegraded":"2024-03-31T23:09:17.185822+0000","last_fullsized":"2024-03-31T23:09:17.185822+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:43:26.061594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":23,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-31T23:09:17.535331+0000","last_change":"2024-03-31T23:09:11.171254+0000","last_active":"2024-03-31T23:09:17.535331+0000","last_peered":"2024-03-31T23:09:17.535331+0000","last_clean":"2024-03-31T23:09:17.535331+0000","last_became_active":"2024-03-31T23:09:11.170684+0000","last_became_peered":"2024-03-31T23:09:11.170684+0000","last_unstale":"2024-03-31T23:09:17.535331+0000","last_undegraded":"2024-03-31T23:09:17.535331+0000","last_fullsized":"2024-03-31T23:09:17.535331+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:03:10.231149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":23,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-31T23:09:17.770231+0000","last_change":"2024-03-31T23:09:11.168784+0000","last_active":"2024-03-31T23:09:17.770231+0000","last_peered":"2024-03-31T23:09:17.770231+0000","last_clean":"2024-03-31T23:09:17.770231+0000","last_became_active":"2024-03-31T23:09:11.168442+0000","last_became_peered":"2024-03-31T23:09:11.168442+0000","last_unstale":"2024-03-31T23:09:17.770231+0000","last_undegraded":"2024-03-31T23:09:17.770231+0000","last_fullsized":"2024-03-31T23:09:17.770231+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T09:37:43.568122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":23,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-31T23:09:17.770411+0000","last_change":"2024-03-31T23:09:11.176017+0000","last_active":"2024-03-31T23:09:17.770411+0000","last_peered":"2024-03-31T23:09:17.770411+0000","last_clean":"2024-03-31T23:09:17.770411+0000","last_became_active":"2024-03-31T23:09:11.175690+0000","last_became_peered":"2024-03-31T23:09:11.175690+0000","last_unstale":"2024-03-31T23:09:17.770411+0000","last_undegraded":"2024-03-31T23:09:17.770411+0000","last_fullsized":"2024-03-31T23:09:17.770411+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:14:13.259895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":23,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-31T23:09:17.535080+0000","last_change":"2024-03-31T23:09:11.171926+0000","last_active":"2024-03-31T23:09:17.535080+0000","last_peered":"2024-03-31T23:09:17.535080+0000","last_clean":"2024-03-31T23:09:17.535080+0000","last_became_active":"2024-03-31T23:09:11.171625+0000","last_became_peered":"2024-03-31T23:09:11.171625+0000","last_unstale":"2024-03-31T23:09:17.535080+0000","last_undegraded":"2024-03-31T23:09:17.535080+0000","last_fullsized":"2024-03-31T23:09:17.535080+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T06:20:20.433802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":23,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-31T23:09:17.536147+0000","last_change":"2024-03-31T23:09:11.183805+0000","last_active":"2024-03-31T23:09:17.536147+0000","last_peered":"2024-03-31T23:09:17.536147+0000","last_clean":"2024-03-31T23:09:17.536147+0000","last_became_active":"2024-03-31T23:09:11.183464+0000","last_became_peered":"2024-03-31T23:09:11.183464+0000","last_unstale":"2024-03-31T23:09:17.536147+0000","last_undegraded":"2024-03-31T23:09:17.536147+0000","last_fullsized":"2024-03-31T23:09:17.536147+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T06:02:33.286563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":23,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-31T23:09:17.184894+0000","last_change":"2024-03-31T23:09:11.170095+0000","last_active":"2024-03-31T23:09:17.184894+0000","last_peered":"2024-03-31T23:09:17.184894+0000","last_clean":"2024-03-31T23:09:17.184894+0000","last_became_active":"2024-03-31T23:09:11.169683+0000","last_became_peered":"2024-03-31T23:09:11.169683+0000","last_unstale":"2024-03-31T23:09:17.184894+0000","last_undegraded":"2024-03-31T23:09:17.184894+0000","last_fullsized":"2024-03-31T23:09:17.184894+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T08:19:47.101431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":23,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-31T23:09:17.186475+0000","last_change":"2024-03-31T23:09:11.181347+0000","last_active":"2024-03-31T23:09:17.186475+0000","last_peered":"2024-03-31T23:09:17.186475+0000","last_clean":"2024-03-31T23:09:17.186475+0000","last_became_active":"2024-03-31T23:09:11.180510+0000","last_became_peered":"2024-03-31T23:09:11.180510+0000","last_unstale":"2024-03-31T23:09:17.186475+0000","last_undegraded":"2024-03-31T23:09:17.186475+0000","last_fullsized":"2024-03-31T23:09:17.186475+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:25:51.083446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-31T23:09:17.186233+0000","last_change":"2024-03-31T23:09:15.205059+0000","last_active":"2024-03-31T23:09:17.186233+0000","last_peered":"2024-03-31T23:09:17.186233+0000","last_clean":"2024-03-31T23:09:17.186233+0000","last_became_active":"2024-03-31T23:09:15.204612+0000","last_became_peered":"2024-03-31T23:09:15.204612+0000","last_unstale":"2024-03-31T23:09:17.186233+0000","last_undegraded":"2024-03-31T23:09:17.186233+0000","last_fullsized":"2024-03-31T23:09:17.186233+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_clean_scrub_stamp":"2024-03-31T23:09:14.171251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:53:33.998026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"85'4","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-31T23:09:17.185332+0000","last_change":"2024-03-31T23:09:10.594147+0000","last_active":"2024-03-31T23:09:17.185332+0000","last_peered":"2024-03-31T23:09:17.185332+0000","last_clean":"2024-03-31T23:09:17.185332+0000","last_became_active":"2024-03-31T23:09:10.592047+0000","last_became_peered":"2024-03-31T23:09:10.592047+0000","last_unstale":"2024-03-31T23:09:17.185332+0000","last_undegraded":"2024-03-31T23:09:17.185332+0000","last_fullsized":"2024-03-31T23:09:17.185332+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_clean_scrub_stamp":"2024-03-31T23:09:09.564263+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:04:34.548282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":23,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-31T23:09:17.388462+0000","last_change":"2024-03-31T23:09:11.182292+0000","last_active":"2024-03-31T23:09:17.388462+0000","last_peered":"2024-03-31T23:09:17.388462+0000","last_clean":"2024-03-31T23:09:17.388462+0000","last_became_active":"2024-03-31T23:09:11.181885+0000","last_became_peered":"2024-03-31T23:09:11.181885+0000","last_unstale":"2024-03-31T23:09:17.388462+0000","last_undegraded":"2024-03-31T23:09:17.388462+0000","last_fullsized":"2024-03-31T23:09:17.388462+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T10:38:17.550625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-31T23:09:17.185543+0000","last_change":"2024-03-31T23:09:15.209034+0000","last_active":"2024-03-31T23:09:17.185543+0000","last_peered":"2024-03-31T23:09:17.185543+0000","last_clean":"2024-03-31T23:09:17.185543+0000","last_became_active":"2024-03-31T23:09:15.208615+0000","last_became_peered":"2024-03-31T23:09:15.208615+0000","last_unstale":"2024-03-31T23:09:17.185543+0000","last_undegraded":"2024-03-31T23:09:17.185543+0000","last_fullsized":"2024-03-31T23:09:17.185543+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_clean_scrub_stamp":"2024-03-31T23:09:14.171251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T07:59:19.538784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"85'1","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-31T23:09:17.388648+0000","last_change":"2024-03-31T23:09:10.598375+0000","last_active":"2024-03-31T23:09:17.388648+0000","last_peered":"2024-03-31T23:09:17.388648+0000","last_clean":"2024-03-31T23:09:17.388648+0000","last_became_active":"2024-03-31T23:09:10.598045+0000","last_became_peered":"2024-03-31T23:09:10.598045+0000","last_unstale":"2024-03-31T23:09:17.388648+0000","last_undegraded":"2024-03-31T23:09:17.388648+0000","last_fullsized":"2024-03-31T23:09:17.388648+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_clean_scrub_stamp":"2024-03-31T23:09:09.564263+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:31:27.113317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":23,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-31T23:09:17.770536+0000","last_change":"2024-03-31T23:09:11.183458+0000","last_active":"2024-03-31T23:09:17.770536+0000","last_peered":"2024-03-31T23:09:17.770536+0000","last_clean":"2024-03-31T23:09:17.770536+0000","last_became_active":"2024-03-31T23:09:11.183100+0000","last_became_peered":"2024-03-31T23:09:11.183100+0000","last_unstale":"2024-03-31T23:09:17.770536+0000","last_undegraded":"2024-03-31T23:09:17.770536+0000","last_fullsized":"2024-03-31T23:09:17.770536+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T10:55:35.512065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-31T23:09:17.185727+0000","last_change":"2024-03-31T23:09:15.208103+0000","last_active":"2024-03-31T23:09:17.185727+0000","last_peered":"2024-03-31T23:09:17.185727+0000","last_clean":"2024-03-31T23:09:17.185727+0000","last_became_active":"2024-03-31T23:09:15.207723+0000","last_became_peered":"2024-03-31T23:09:15.207723+0000","last_unstale":"2024-03-31T23:09:17.185727+0000","last_undegraded":"2024-03-31T23:09:17.185727+0000","last_fullsized":"2024-03-31T23:09:17.185727+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_clean_scrub_stamp":"2024-03-31T23:09:14.171251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T07:55:01.303693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"85'2","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-31T23:09:17.537116+0000","last_change":"2024-03-31T23:09:10.597471+0000","last_active":"2024-03-31T23:09:17.537116+0000","last_peered":"2024-03-31T23:09:17.537116+0000","last_clean":"2024-03-31T23:09:17.537116+0000","last_became_active":"2024-03-31T23:09:10.597165+0000","last_became_peered":"2024-03-31T23:09:10.597165+0000","last_unstale":"2024-03-31T23:09:17.537116+0000","last_undegraded":"2024-03-31T23:09:17.537116+0000","last_fullsized":"2024-03-31T23:09:17.537116+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_clean_scrub_stamp":"2024-03-31T23:09:09.564263+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:10:26.826525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":23,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-31T23:09:17.536710+0000","last_change":"2024-03-31T23:09:11.170650+0000","last_active":"2024-03-31T23:09:17.536710+0000","last_peered":"2024-03-31T23:09:17.536710+0000","last_clean":"2024-03-31T23:09:17.536710+0000","last_became_active":"2024-03-31T23:09:11.170291+0000","last_became_peered":"2024-03-31T23:09:11.170291+0000","last_unstale":"2024-03-31T23:09:17.536710+0000","last_undegraded":"2024-03-31T23:09:17.536710+0000","last_fullsized":"2024-03-31T23:09:17.536710+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:29:54.904377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-31T23:09:17.185586+0000","last_change":"2024-03-31T23:09:15.204857+0000","last_active":"2024-03-31T23:09:17.185586+0000","last_peered":"2024-03-31T23:09:17.185586+0000","last_clean":"2024-03-31T23:09:17.185586+0000","last_became_active":"2024-03-31T23:09:15.204484+0000","last_became_peered":"2024-03-31T23:09:15.204484+0000","last_unstale":"2024-03-31T23:09:17.185586+0000","last_undegraded":"2024-03-31T23:09:17.185586+0000","last_fullsized":"2024-03-31T23:09:17.185586+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_clean_scrub_stamp":"2024-03-31T23:09:14.171251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T09:54:24.646918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"85'2","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-31T23:09:17.536910+0000","last_change":"2024-03-31T23:09:10.589922+0000","last_active":"2024-03-31T23:09:17.536910+0000","last_peered":"2024-03-31T23:09:17.536910+0000","last_clean":"2024-03-31T23:09:17.536910+0000","last_became_active":"2024-03-31T23:09:10.589569+0000","last_became_peered":"2024-03-31T23:09:10.589569+0000","last_unstale":"2024-03-31T23:09:17.536910+0000","last_undegraded":"2024-03-31T23:09:17.536910+0000","last_fullsized":"2024-03-31T23:09:17.536910+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_clean_scrub_stamp":"2024-03-31T23:09:09.564263+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:57:05.347344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":23,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-31T23:09:17.537194+0000","last_change":"2024-03-31T23:09:11.174204+0000","last_active":"2024-03-31T23:09:17.537194+0000","last_peered":"2024-03-31T23:09:17.537194+0000","last_clean":"2024-03-31T23:09:17.537194+0000","last_became_active":"2024-03-31T23:09:11.173903+0000","last_became_peered":"2024-03-31T23:09:11.173903+0000","last_unstale":"2024-03-31T23:09:17.537194+0000","last_undegraded":"2024-03-31T23:09:17.537194+0000","last_fullsized":"2024-03-31T23:09:17.537194+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:20:23.038437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-31T23:09:17.186498+0000","last_change":"2024-03-31T23:09:15.193096+0000","last_active":"2024-03-31T23:09:17.186498+0000","last_peered":"2024-03-31T23:09:17.186498+0000","last_clean":"2024-03-31T23:09:17.186498+0000","last_became_active":"2024-03-31T23:09:15.192624+0000","last_became_peered":"2024-03-31T23:09:15.192624+0000","last_unstale":"2024-03-31T23:09:17.186498+0000","last_undegraded":"2024-03-31T23:09:17.186498+0000","last_fullsized":"2024-03-31T23:09:17.186498+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_clean_scrub_stamp":"2024-03-31T23:09:14.171251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:22:24.921346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"91'6","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-31T23:09:31.391419+0000","last_change":"2024-03-31T23:09:10.587209+0000","last_active":"2024-03-31T23:09:31.391419+0000","last_peered":"2024-03-31T23:09:31.391419+0000","last_clean":"2024-03-31T23:09:31.391419+0000","last_became_active":"2024-03-31T23:09:10.586486+0000","last_became_peered":"2024-03-31T23:09:10.586486+0000","last_unstale":"2024-03-31T23:09:31.391419+0000","last_undegraded":"2024-03-31T23:09:31.391419+0000","last_fullsized":"2024-03-31T23:09:31.391419+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_clean_scrub_stamp":"2024-03-31T23:09:09.564263+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:27:56.046414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1013,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":23,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-31T23:09:17.537109+0000","last_change":"2024-03-31T23:09:11.175458+0000","last_active":"2024-03-31T23:09:17.537109+0000","last_peered":"2024-03-31T23:09:17.537109+0000","last_clean":"2024-03-31T23:09:17.537109+0000","last_became_active":"2024-03-31T23:09:11.175047+0000","last_became_peered":"2024-03-31T23:09:11.175047+0000","last_unstale":"2024-03-31T23:09:17.537109+0000","last_undegraded":"2024-03-31T23:09:17.537109+0000","last_fullsized":"2024-03-31T23:09:17.537109+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T06:21:22.733625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-31T23:09:17.186391+0000","last_change":"2024-03-31T23:09:15.194717+0000","last_active":"2024-03-31T23:09:17.186391+0000","last_peered":"2024-03-31T23:09:17.186391+0000","last_clean":"2024-03-31T23:09:17.186391+0000","last_became_active":"2024-03-31T23:09:15.194232+0000","last_became_peered":"2024-03-31T23:09:15.194232+0000","last_unstale":"2024-03-31T23:09:17.186391+0000","last_undegraded":"2024-03-31T23:09:17.186391+0000","last_fullsized":"2024-03-31T23:09:17.186391+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_clean_scrub_stamp":"2024-03-31T23:09:14.171251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:21:00.445232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"85'1","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-31T23:09:17.535893+0000","last_change":"2024-03-31T23:09:10.586346+0000","last_active":"2024-03-31T23:09:17.535893+0000","last_peered":"2024-03-31T23:09:17.535893+0000","last_clean":"2024-03-31T23:09:17.535893+0000","last_became_active":"2024-03-31T23:09:10.586043+0000","last_became_peered":"2024-03-31T23:09:10.586043+0000","last_unstale":"2024-03-31T23:09:17.535893+0000","last_undegraded":"2024-03-31T23:09:17.535893+0000","last_fullsized":"2024-03-31T23:09:17.535893+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_clean_scrub_stamp":"2024-03-31T23:09:09.564263+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:17:27.387786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":23,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-31T23:09:17.536602+0000","last_change":"2024-03-31T23:09:11.181543+0000","last_active":"2024-03-31T23:09:17.536602+0000","last_peered":"2024-03-31T23:09:17.536602+0000","last_clean":"2024-03-31T23:09:17.536602+0000","last_became_active":"2024-03-31T23:09:11.181234+0000","last_became_peered":"2024-03-31T23:09:11.181234+0000","last_unstale":"2024-03-31T23:09:17.536602+0000","last_undegraded":"2024-03-31T23:09:17.536602+0000","last_fullsized":"2024-03-31T23:09:17.536602+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T09:50:14.712348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-31T23:09:17.536138+0000","last_change":"2024-03-31T23:09:15.205855+0000","last_active":"2024-03-31T23:09:17.536138+0000","last_peered":"2024-03-31T23:09:17.536138+0000","last_clean":"2024-03-31T23:09:17.536138+0000","last_became_active":"2024-03-31T23:09:15.205359+0000","last_became_peered":"2024-03-31T23:09:15.205359+0000","last_unstale":"2024-03-31T23:09:17.536138+0000","last_undegraded":"2024-03-31T23:09:17.536138+0000","last_fullsized":"2024-03-31T23:09:17.536138+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_clean_scrub_stamp":"2024-03-31T23:09:14.171251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T09:39:32.461011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-31T23:09:17.535211+0000","last_change":"2024-03-31T23:09:10.583274+0000","last_active":"2024-03-31T23:09:17.535211+0000","last_peered":"2024-03-31T23:09:17.535211+0000","last_clean":"2024-03-31T23:09:17.535211+0000","last_became_active":"2024-03-31T23:09:10.582910+0000","last_became_peered":"2024-03-31T23:09:10.582910+0000","last_unstale":"2024-03-31T23:09:17.535211+0000","last_undegraded":"2024-03-31T23:09:17.535211+0000","last_fullsized":"2024-03-31T23:09:17.535211+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_clean_scrub_stamp":"2024-03-31T23:09:09.564263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:16:32.251169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":23,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-31T23:09:17.388219+0000","last_change":"2024-03-31T23:09:11.178619+0000","last_active":"2024-03-31T23:09:17.388219+0000","last_peered":"2024-03-31T23:09:17.388219+0000","last_clean":"2024-03-31T23:09:17.388219+0000","last_became_active":"2024-03-31T23:09:11.178239+0000","last_became_peered":"2024-03-31T23:09:11.178239+0000","last_unstale":"2024-03-31T23:09:17.388219+0000","last_undegraded":"2024-03-31T23:09:17.388219+0000","last_fullsized":"2024-03-31T23:09:17.388219+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:28:58.863199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-31T23:09:17.770679+0000","last_change":"2024-03-31T23:09:15.205864+0000","last_active":"2024-03-31T23:09:17.770679+0000","last_peered":"2024-03-31T23:09:17.770679+0000","last_clean":"2024-03-31T23:09:17.770679+0000","last_became_active":"2024-03-31T23:09:15.204976+0000","last_became_peered":"2024-03-31T23:09:15.204976+0000","last_unstale":"2024-03-31T23:09:17.770679+0000","last_undegraded":"2024-03-31T23:09:17.770679+0000","last_fullsized":"2024-03-31T23:09:17.770679+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_clean_scrub_stamp":"2024-03-31T23:09:14.171251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:39:08.537827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"91'1","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-31T23:09:17.535993+0000","last_change":"2024-03-31T23:09:10.588407+0000","last_active":"2024-03-31T23:09:17.535993+0000","last_peered":"2024-03-31T23:09:17.535993+0000","last_clean":"2024-03-31T23:09:17.535993+0000","last_became_active":"2024-03-31T23:09:10.588109+0000","last_became_peered":"2024-03-31T23:09:10.588109+0000","last_unstale":"2024-03-31T23:09:17.535993+0000","last_undegraded":"2024-03-31T23:09:17.535993+0000","last_fullsized":"2024-03-31T23:09:17.535993+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_clean_scrub_stamp":"2024-03-31T23:09:09.564263+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:01:25.858834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":23,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-31T23:09:17.184363+0000","last_change":"2024-03-31T23:09:11.180452+0000","last_active":"2024-03-31T23:09:17.184363+0000","last_peered":"2024-03-31T23:09:17.184363+0000","last_clean":"2024-03-31T23:09:17.184363+0000","last_became_active":"2024-03-31T23:09:11.180117+0000","last_became_peered":"2024-03-31T23:09:11.180117+0000","last_unstale":"2024-03-31T23:09:17.184363+0000","last_undegraded":"2024-03-31T23:09:17.184363+0000","last_fullsized":"2024-03-31T23:09:17.184363+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:04:58.288981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-31T23:09:17.388557+0000","last_change":"2024-03-31T23:09:15.199669+0000","last_active":"2024-03-31T23:09:17.388557+0000","last_peered":"2024-03-31T23:09:17.388557+0000","last_clean":"2024-03-31T23:09:17.388557+0000","last_became_active":"2024-03-31T23:09:15.199262+0000","last_became_peered":"2024-03-31T23:09:15.199262+0000","last_unstale":"2024-03-31T23:09:17.388557+0000","last_undegraded":"2024-03-31T23:09:17.388557+0000","last_fullsized":"2024-03-31T23:09:17.388557+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_clean_scrub_stamp":"2024-03-31T23:09:14.171251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:44:02.975000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"91'3","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-31T23:09:29.421862+0000","last_change":"2024-03-31T23:09:10.583476+0000","last_active":"2024-03-31T23:09:29.421862+0000","last_peered":"2024-03-31T23:09:29.421862+0000","last_clean":"2024-03-31T23:09:29.421862+0000","last_became_active":"2024-03-31T23:09:10.582789+0000","last_became_peered":"2024-03-31T23:09:10.582789+0000","last_unstale":"2024-03-31T23:09:29.421862+0000","last_undegraded":"2024-03-31T23:09:29.421862+0000","last_fullsized":"2024-03-31T23:09:29.421862+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_clean_scrub_stamp":"2024-03-31T23:09:09.564263+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T10:52:35.478325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":23,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-31T23:09:17.535626+0000","last_change":"2024-03-31T23:09:11.171074+0000","last_active":"2024-03-31T23:09:17.535626+0000","last_peered":"2024-03-31T23:09:17.535626+0000","last_clean":"2024-03-31T23:09:17.535626+0000","last_became_active":"2024-03-31T23:09:11.169572+0000","last_became_peered":"2024-03-31T23:09:11.169572+0000","last_unstale":"2024-03-31T23:09:17.535626+0000","last_undegraded":"2024-03-31T23:09:17.535626+0000","last_fullsized":"2024-03-31T23:09:17.535626+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T07:15:34.833388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-31T23:09:17.184460+0000","last_change":"2024-03-31T23:09:15.203835+0000","last_active":"2024-03-31T23:09:17.184460+0000","last_peered":"2024-03-31T23:09:17.184460+0000","last_clean":"2024-03-31T23:09:17.184460+0000","last_became_active":"2024-03-31T23:09:15.200861+0000","last_became_peered":"2024-03-31T23:09:15.200861+0000","last_unstale":"2024-03-31T23:09:17.184460+0000","last_undegraded":"2024-03-31T23:09:17.184460+0000","last_fullsized":"2024-03-31T23:09:17.184460+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_clean_scrub_stamp":"2024-03-31T23:09:14.171251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:56:19.664708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"91'2","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-31T23:09:29.423479+0000","last_change":"2024-03-31T23:09:10.592601+0000","last_active":"2024-03-31T23:09:29.423479+0000","last_peered":"2024-03-31T23:09:29.423479+0000","last_clean":"2024-03-31T23:09:29.423479+0000","last_became_active":"2024-03-31T23:09:10.592345+0000","last_became_peered":"2024-03-31T23:09:10.592345+0000","last_unstale":"2024-03-31T23:09:29.423479+0000","last_undegraded":"2024-03-31T23:09:29.423479+0000","last_fullsized":"2024-03-31T23:09:29.423479+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_clean_scrub_stamp":"2024-03-31T23:09:09.564263+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:10:21.195544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":23,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-31T23:09:17.770411+0000","last_change":"2024-03-31T23:09:11.175305+0000","last_active":"2024-03-31T23:09:17.770411+0000","last_peered":"2024-03-31T23:09:17.770411+0000","last_clean":"2024-03-31T23:09:17.770411+0000","last_became_active":"2024-03-31T23:09:11.174970+0000","last_became_peered":"2024-03-31T23:09:11.174970+0000","last_unstale":"2024-03-31T23:09:17.770411+0000","last_undegraded":"2024-03-31T23:09:17.770411+0000","last_fullsized":"2024-03-31T23:09:17.770411+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:29:39.718127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-31T23:09:17.186161+0000","last_change":"2024-03-31T23:09:15.200365+0000","last_active":"2024-03-31T23:09:17.186161+0000","last_peered":"2024-03-31T23:09:17.186161+0000","last_clean":"2024-03-31T23:09:17.186161+0000","last_became_active":"2024-03-31T23:09:15.199830+0000","last_became_peered":"2024-03-31T23:09:15.199830+0000","last_unstale":"2024-03-31T23:09:17.186161+0000","last_undegraded":"2024-03-31T23:09:17.186161+0000","last_fullsized":"2024-03-31T23:09:17.186161+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_clean_scrub_stamp":"2024-03-31T23:09:14.171251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:37:30.048859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"91'4","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-31T23:09:29.420719+0000","last_change":"2024-03-31T23:09:10.582654+0000","last_active":"2024-03-31T23:09:29.420719+0000","last_peered":"2024-03-31T23:09:29.420719+0000","last_clean":"2024-03-31T23:09:29.420719+0000","last_became_active":"2024-03-31T23:09:10.582314+0000","last_became_peered":"2024-03-31T23:09:10.582314+0000","last_unstale":"2024-03-31T23:09:29.420719+0000","last_undegraded":"2024-03-31T23:09:29.420719+0000","last_fullsized":"2024-03-31T23:09:29.420719+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_clean_scrub_stamp":"2024-03-31T23:09:09.564263+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:51:34.236575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":23,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-31T23:09:17.536765+0000","last_change":"2024-03-31T23:09:11.172811+0000","last_active":"2024-03-31T23:09:17.536765+0000","last_peered":"2024-03-31T23:09:17.536765+0000","last_clean":"2024-03-31T23:09:17.536765+0000","last_became_active":"2024-03-31T23:09:11.172478+0000","last_became_peered":"2024-03-31T23:09:11.172478+0000","last_unstale":"2024-03-31T23:09:17.536765+0000","last_undegraded":"2024-03-31T23:09:17.536765+0000","last_fullsized":"2024-03-31T23:09:17.536765+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T10:00:09.825254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-31T23:09:17.184780+0000","last_change":"2024-03-31T23:09:15.205494+0000","last_active":"2024-03-31T23:09:17.184780+0000","last_peered":"2024-03-31T23:09:17.184780+0000","last_clean":"2024-03-31T23:09:17.184780+0000","last_became_active":"2024-03-31T23:09:15.205101+0000","last_became_peered":"2024-03-31T23:09:15.205101+0000","last_unstale":"2024-03-31T23:09:17.184780+0000","last_undegraded":"2024-03-31T23:09:17.184780+0000","last_fullsized":"2024-03-31T23:09:17.184780+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_clean_scrub_stamp":"2024-03-31T23:09:14.171251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:52:47.472296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"91'12","reported_seq":37,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-31T23:09:29.420328+0000","last_change":"2024-03-31T23:09:10.585611+0000","last_active":"2024-03-31T23:09:29.420328+0000","last_peered":"2024-03-31T23:09:29.420328+0000","last_clean":"2024-03-31T23:09:29.420328+0000","last_became_active":"2024-03-31T23:09:10.585287+0000","last_became_peered":"2024-03-31T23:09:10.585287+0000","last_unstale":"2024-03-31T23:09:29.420328+0000","last_undegraded":"2024-03-31T23:09:29.420328+0000","last_fullsized":"2024-03-31T23:09:29.420328+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_clean_scrub_stamp":"2024-03-31T23:09:09.564263+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:47:44.063831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3842,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":23,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-31T23:09:17.536930+0000","last_change":"2024-03-31T23:09:11.175812+0000","last_active":"2024-03-31T23:09:17.536930+0000","last_peered":"2024-03-31T23:09:17.536930+0000","last_clean":"2024-03-31T23:09:17.536930+0000","last_became_active":"2024-03-31T23:09:11.174972+0000","last_became_peered":"2024-03-31T23:09:11.174972+0000","last_unstale":"2024-03-31T23:09:17.536930+0000","last_undegraded":"2024-03-31T23:09:17.536930+0000","last_fullsized":"2024-03-31T23:09:17.536930+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:17:18.870340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-31T23:09:17.537653+0000","last_change":"2024-03-31T23:09:15.205018+0000","last_active":"2024-03-31T23:09:17.537653+0000","last_peered":"2024-03-31T23:09:17.537653+0000","last_clean":"2024-03-31T23:09:17.537653+0000","last_became_active":"2024-03-31T23:09:15.204510+0000","last_became_peered":"2024-03-31T23:09:15.204510+0000","last_unstale":"2024-03-31T23:09:17.537653+0000","last_undegraded":"2024-03-31T23:09:17.537653+0000","last_fullsized":"2024-03-31T23:09:17.537653+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_clean_scrub_stamp":"2024-03-31T23:09:14.171251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T07:48:59.819937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"85'2","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-31T23:09:17.186385+0000","last_change":"2024-03-31T23:09:10.590233+0000","last_active":"2024-03-31T23:09:17.186385+0000","last_peered":"2024-03-31T23:09:17.186385+0000","last_clean":"2024-03-31T23:09:17.186385+0000","last_became_active":"2024-03-31T23:09:10.589853+0000","last_became_peered":"2024-03-31T23:09:10.589853+0000","last_unstale":"2024-03-31T23:09:17.186385+0000","last_undegraded":"2024-03-31T23:09:17.186385+0000","last_fullsized":"2024-03-31T23:09:17.186385+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_clean_scrub_stamp":"2024-03-31T23:09:09.564263+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:10:54.980394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":23,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-31T23:09:17.535338+0000","last_change":"2024-03-31T23:09:11.177871+0000","last_active":"2024-03-31T23:09:17.535338+0000","last_peered":"2024-03-31T23:09:17.535338+0000","last_clean":"2024-03-31T23:09:17.535338+0000","last_became_active":"2024-03-31T23:09:11.176504+0000","last_became_peered":"2024-03-31T23:09:11.176504+0000","last_unstale":"2024-03-31T23:09:17.535338+0000","last_undegraded":"2024-03-31T23:09:17.535338+0000","last_fullsized":"2024-03-31T23:09:17.535338+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T06:09:08.797695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-31T23:09:17.535914+0000","last_change":"2024-03-31T23:09:15.207336+0000","last_active":"2024-03-31T23:09:17.535914+0000","last_peered":"2024-03-31T23:09:17.535914+0000","last_clean":"2024-03-31T23:09:17.535914+0000","last_became_active":"2024-03-31T23:09:15.206893+0000","last_became_peered":"2024-03-31T23:09:15.206893+0000","last_unstale":"2024-03-31T23:09:17.535914+0000","last_undegraded":"2024-03-31T23:09:17.535914+0000","last_fullsized":"2024-03-31T23:09:17.535914+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_clean_scrub_stamp":"2024-03-31T23:09:14.171251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:24:59.766421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"91'3","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-31T23:09:29.423986+0000","last_change":"2024-03-31T23:09:10.587099+0000","last_active":"2024-03-31T23:09:29.423986+0000","last_peered":"2024-03-31T23:09:29.423986+0000","last_clean":"2024-03-31T23:09:29.423986+0000","last_became_active":"2024-03-31T23:09:10.586429+0000","last_became_peered":"2024-03-31T23:09:10.586429+0000","last_unstale":"2024-03-31T23:09:29.423986+0000","last_undegraded":"2024-03-31T23:09:29.423986+0000","last_fullsized":"2024-03-31T23:09:29.423986+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_clean_scrub_stamp":"2024-03-31T23:09:09.564263+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:55:09.307378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":23,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-31T23:09:17.536335+0000","last_change":"2024-03-31T23:09:11.177661+0000","last_active":"2024-03-31T23:09:17.536335+0000","last_peered":"2024-03-31T23:09:17.536335+0000","last_clean":"2024-03-31T23:09:17.536335+0000","last_became_active":"2024-03-31T23:09:11.177314+0000","last_became_peered":"2024-03-31T23:09:11.177314+0000","last_unstale":"2024-03-31T23:09:17.536335+0000","last_undegraded":"2024-03-31T23:09:17.536335+0000","last_fullsized":"2024-03-31T23:09:17.536335+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T09:26:50.556837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-31T23:09:17.184783+0000","last_change":"2024-03-31T23:09:15.198653+0000","last_active":"2024-03-31T23:09:17.184783+0000","last_peered":"2024-03-31T23:09:17.184783+0000","last_clean":"2024-03-31T23:09:17.184783+0000","last_became_active":"2024-03-31T23:09:15.198277+0000","last_became_peered":"2024-03-31T23:09:15.198277+0000","last_unstale":"2024-03-31T23:09:17.184783+0000","last_undegraded":"2024-03-31T23:09:17.184783+0000","last_fullsized":"2024-03-31T23:09:17.184783+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_clean_scrub_stamp":"2024-03-31T23:09:14.171251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:27:31.391784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-31T23:09:17.185458+0000","last_change":"2024-03-31T23:09:10.586570+0000","last_active":"2024-03-31T23:09:17.185458+0000","last_peered":"2024-03-31T23:09:17.185458+0000","last_clean":"2024-03-31T23:09:17.185458+0000","last_became_active":"2024-03-31T23:09:10.586239+0000","last_became_peered":"2024-03-31T23:09:10.586239+0000","last_unstale":"2024-03-31T23:09:17.185458+0000","last_undegraded":"2024-03-31T23:09:17.185458+0000","last_fullsized":"2024-03-31T23:09:17.185458+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_clean_scrub_stamp":"2024-03-31T23:09:09.564263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:23:37.182954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":23,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-31T23:09:17.536044+0000","last_change":"2024-03-31T23:09:11.173294+0000","last_active":"2024-03-31T23:09:17.536044+0000","last_peered":"2024-03-31T23:09:17.536044+0000","last_clean":"2024-03-31T23:09:17.536044+0000","last_became_active":"2024-03-31T23:09:11.172926+0000","last_became_peered":"2024-03-31T23:09:11.172926+0000","last_unstale":"2024-03-31T23:09:17.536044+0000","last_undegraded":"2024-03-31T23:09:17.536044+0000","last_fullsized":"2024-03-31T23:09:17.536044+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T06:21:46.603381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-31T23:09:17.535654+0000","last_change":"2024-03-31T23:09:15.199893+0000","last_active":"2024-03-31T23:09:17.535654+0000","last_peered":"2024-03-31T23:09:17.535654+0000","last_clean":"2024-03-31T23:09:17.535654+0000","last_became_active":"2024-03-31T23:09:15.199517+0000","last_became_peered":"2024-03-31T23:09:15.199517+0000","last_unstale":"2024-03-31T23:09:17.535654+0000","last_undegraded":"2024-03-31T23:09:17.535654+0000","last_fullsized":"2024-03-31T23:09:17.535654+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_clean_scrub_stamp":"2024-03-31T23:09:14.171251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T07:48:35.261108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"85'8","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-31T23:09:17.536545+0000","last_change":"2024-03-31T23:09:10.588213+0000","last_active":"2024-03-31T23:09:17.536545+0000","last_peered":"2024-03-31T23:09:17.536545+0000","last_clean":"2024-03-31T23:09:17.536545+0000","last_became_active":"2024-03-31T23:09:10.587579+0000","last_became_peered":"2024-03-31T23:09:10.587579+0000","last_unstale":"2024-03-31T23:09:17.536545+0000","last_undegraded":"2024-03-31T23:09:17.536545+0000","last_fullsized":"2024-03-31T23:09:17.536545+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_clean_scrub_stamp":"2024-03-31T23:09:09.564263+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:00:17.883435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":23,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-31T23:09:17.536667+0000","last_change":"2024-03-31T23:09:11.182427+0000","last_active":"2024-03-31T23:09:17.536667+0000","last_peered":"2024-03-31T23:09:17.536667+0000","last_clean":"2024-03-31T23:09:17.536667+0000","last_became_active":"2024-03-31T23:09:11.182120+0000","last_became_peered":"2024-03-31T23:09:11.182120+0000","last_unstale":"2024-03-31T23:09:17.536667+0000","last_undegraded":"2024-03-31T23:09:17.536667+0000","last_fullsized":"2024-03-31T23:09:17.536667+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:17:48.728424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-31T23:09:17.536553+0000","last_change":"2024-03-31T23:09:15.207831+0000","last_active":"2024-03-31T23:09:17.536553+0000","last_peered":"2024-03-31T23:09:17.536553+0000","last_clean":"2024-03-31T23:09:17.536553+0000","last_became_active":"2024-03-31T23:09:15.207432+0000","last_became_peered":"2024-03-31T23:09:15.207432+0000","last_unstale":"2024-03-31T23:09:17.536553+0000","last_undegraded":"2024-03-31T23:09:17.536553+0000","last_fullsized":"2024-03-31T23:09:17.536553+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_clean_scrub_stamp":"2024-03-31T23:09:14.171251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:05:27.072054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"85'1","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-31T23:09:17.537551+0000","last_change":"2024-03-31T23:09:10.591483+0000","last_active":"2024-03-31T23:09:17.537551+0000","last_peered":"2024-03-31T23:09:17.537551+0000","last_clean":"2024-03-31T23:09:17.537551+0000","last_became_active":"2024-03-31T23:09:10.591173+0000","last_became_peered":"2024-03-31T23:09:10.591173+0000","last_unstale":"2024-03-31T23:09:17.537551+0000","last_undegraded":"2024-03-31T23:09:17.537551+0000","last_fullsized":"2024-03-31T23:09:17.537551+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_clean_scrub_stamp":"2024-03-31T23:09:09.564263+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T10:04:14.275875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":23,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-31T23:09:17.536341+0000","last_change":"2024-03-31T23:09:11.181171+0000","last_active":"2024-03-31T23:09:17.536341+0000","last_peered":"2024-03-31T23:09:17.536341+0000","last_clean":"2024-03-31T23:09:17.536341+0000","last_became_active":"2024-03-31T23:09:11.180847+0000","last_became_peered":"2024-03-31T23:09:11.180847+0000","last_unstale":"2024-03-31T23:09:17.536341+0000","last_undegraded":"2024-03-31T23:09:17.536341+0000","last_fullsized":"2024-03-31T23:09:17.536341+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:39:29.429009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-31T23:09:17.537438+0000","last_change":"2024-03-31T23:09:15.746941+0000","last_active":"2024-03-31T23:09:17.537438+0000","last_peered":"2024-03-31T23:09:17.537438+0000","last_clean":"2024-03-31T23:09:17.537438+0000","last_became_active":"2024-03-31T23:09:15.746352+0000","last_became_peered":"2024-03-31T23:09:15.746352+0000","last_unstale":"2024-03-31T23:09:17.537438+0000","last_undegraded":"2024-03-31T23:09:17.537438+0000","last_fullsized":"2024-03-31T23:09:17.537438+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_clean_scrub_stamp":"2024-03-31T23:09:14.171251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T06:21:43.482575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-31T23:09:17.184670+0000","last_change":"2024-03-31T23:09:10.593029+0000","last_active":"2024-03-31T23:09:17.184670+0000","last_peered":"2024-03-31T23:09:17.184670+0000","last_clean":"2024-03-31T23:09:17.184670+0000","last_became_active":"2024-03-31T23:09:10.592709+0000","last_became_peered":"2024-03-31T23:09:10.592709+0000","last_unstale":"2024-03-31T23:09:17.184670+0000","last_undegraded":"2024-03-31T23:09:17.184670+0000","last_fullsized":"2024-03-31T23:09:17.184670+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_clean_scrub_stamp":"2024-03-31T23:09:09.564263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:46:45.700207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":23,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-31T23:09:17.535767+0000","last_change":"2024-03-31T23:09:11.177857+0000","last_active":"2024-03-31T23:09:17.535767+0000","last_peered":"2024-03-31T23:09:17.535767+0000","last_clean":"2024-03-31T23:09:17.535767+0000","last_became_active":"2024-03-31T23:09:11.177513+0000","last_became_peered":"2024-03-31T23:09:11.177513+0000","last_unstale":"2024-03-31T23:09:17.535767+0000","last_undegraded":"2024-03-31T23:09:17.535767+0000","last_fullsized":"2024-03-31T23:09:17.535767+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T06:09:23.579208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-31T23:09:17.535426+0000","last_change":"2024-03-31T23:09:15.203412+0000","last_active":"2024-03-31T23:09:17.535426+0000","last_peered":"2024-03-31T23:09:17.535426+0000","last_clean":"2024-03-31T23:09:17.535426+0000","last_became_active":"2024-03-31T23:09:15.202983+0000","last_became_peered":"2024-03-31T23:09:15.202983+0000","last_unstale":"2024-03-31T23:09:17.535426+0000","last_undegraded":"2024-03-31T23:09:17.535426+0000","last_fullsized":"2024-03-31T23:09:17.535426+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_clean_scrub_stamp":"2024-03-31T23:09:14.171251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:57:46.357378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"91'1","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-31T23:09:17.536372+0000","last_change":"2024-03-31T23:09:10.593299+0000","last_active":"2024-03-31T23:09:17.536372+0000","last_peered":"2024-03-31T23:09:17.536372+0000","last_clean":"2024-03-31T23:09:17.536372+0000","last_became_active":"2024-03-31T23:09:10.593007+0000","last_became_peered":"2024-03-31T23:09:10.593007+0000","last_unstale":"2024-03-31T23:09:17.536372+0000","last_undegraded":"2024-03-31T23:09:17.536372+0000","last_fullsized":"2024-03-31T23:09:17.536372+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_clean_scrub_stamp":"2024-03-31T23:09:09.564263+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:15:07.199645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":23,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-31T23:09:17.535583+0000","last_change":"2024-03-31T23:09:11.176846+0000","last_active":"2024-03-31T23:09:17.535583+0000","last_peered":"2024-03-31T23:09:17.535583+0000","last_clean":"2024-03-31T23:09:17.535583+0000","last_became_active":"2024-03-31T23:09:11.176554+0000","last_became_peered":"2024-03-31T23:09:11.176554+0000","last_unstale":"2024-03-31T23:09:17.535583+0000","last_undegraded":"2024-03-31T23:09:17.535583+0000","last_fullsized":"2024-03-31T23:09:17.535583+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:25:23.639093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-31T23:09:17.535659+0000","last_change":"2024-03-31T23:09:15.746599+0000","last_active":"2024-03-31T23:09:17.535659+0000","last_peered":"2024-03-31T23:09:17.535659+0000","last_clean":"2024-03-31T23:09:17.535659+0000","last_became_active":"2024-03-31T23:09:15.745879+0000","last_became_peered":"2024-03-31T23:09:15.745879+0000","last_unstale":"2024-03-31T23:09:17.535659+0000","last_undegraded":"2024-03-31T23:09:17.535659+0000","last_fullsized":"2024-03-31T23:09:17.535659+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_clean_scrub_stamp":"2024-03-31T23:09:14.171251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T08:02:25.198610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":23,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-31T23:09:17.536136+0000","last_change":"2024-03-31T23:09:11.175546+0000","last_active":"2024-03-31T23:09:17.536136+0000","last_peered":"2024-03-31T23:09:17.536136+0000","last_clean":"2024-03-31T23:09:17.536136+0000","last_became_active":"2024-03-31T23:09:11.175253+0000","last_became_peered":"2024-03-31T23:09:11.175253+0000","last_unstale":"2024-03-31T23:09:17.536136+0000","last_undegraded":"2024-03-31T23:09:17.536136+0000","last_fullsized":"2024-03-31T23:09:17.536136+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T10:27:31.753102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"91'1","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-31T23:09:17.535146+0000","last_change":"2024-03-31T23:09:10.580888+0000","last_active":"2024-03-31T23:09:17.535146+0000","last_peered":"2024-03-31T23:09:17.535146+0000","last_clean":"2024-03-31T23:09:17.535146+0000","last_became_active":"2024-03-31T23:09:10.580557+0000","last_became_peered":"2024-03-31T23:09:10.580557+0000","last_unstale":"2024-03-31T23:09:17.535146+0000","last_undegraded":"2024-03-31T23:09:17.535146+0000","last_fullsized":"2024-03-31T23:09:17.535146+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_clean_scrub_stamp":"2024-03-31T23:09:09.564263+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:17:38.878570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-31T23:09:17.535437+0000","last_change":"2024-03-31T23:09:15.205282+0000","last_active":"2024-03-31T23:09:17.535437+0000","last_peered":"2024-03-31T23:09:17.535437+0000","last_clean":"2024-03-31T23:09:17.535437+0000","last_became_active":"2024-03-31T23:09:15.204873+0000","last_became_peered":"2024-03-31T23:09:15.204873+0000","last_unstale":"2024-03-31T23:09:17.535437+0000","last_undegraded":"2024-03-31T23:09:17.535437+0000","last_fullsized":"2024-03-31T23:09:17.535437+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_clean_scrub_stamp":"2024-03-31T23:09:14.171251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:56:05.697685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":23,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-31T23:09:17.388276+0000","last_change":"2024-03-31T23:09:11.182155+0000","last_active":"2024-03-31T23:09:17.388276+0000","last_peered":"2024-03-31T23:09:17.388276+0000","last_clean":"2024-03-31T23:09:17.388276+0000","last_became_active":"2024-03-31T23:09:11.181829+0000","last_became_peered":"2024-03-31T23:09:11.181829+0000","last_unstale":"2024-03-31T23:09:17.388276+0000","last_undegraded":"2024-03-31T23:09:17.388276+0000","last_fullsized":"2024-03-31T23:09:17.388276+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:19:22.832440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-31T23:09:17.185653+0000","last_change":"2024-03-31T23:09:10.590451+0000","last_active":"2024-03-31T23:09:17.185653+0000","last_peered":"2024-03-31T23:09:17.185653+0000","last_clean":"2024-03-31T23:09:17.185653+0000","last_became_active":"2024-03-31T23:09:10.590123+0000","last_became_peered":"2024-03-31T23:09:10.590123+0000","last_unstale":"2024-03-31T23:09:17.185653+0000","last_undegraded":"2024-03-31T23:09:17.185653+0000","last_fullsized":"2024-03-31T23:09:17.185653+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_clean_scrub_stamp":"2024-03-31T23:09:09.564263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:27:02.724456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":360,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-31T23:09:17.535950+0000","last_change":"2024-03-31T23:07:57.920942+0000","last_active":"2024-03-31T23:09:17.535950+0000","last_peered":"2024-03-31T23:09:17.535950+0000","last_clean":"2024-03-31T23:09:17.535950+0000","last_became_active":"2024-03-31T23:07:57.920381+0000","last_became_peered":"2024-03-31T23:07:57.920381+0000","last_unstale":"2024-03-31T23:09:17.535950+0000","last_undegraded":"2024-03-31T23:09:17.535950+0000","last_fullsized":"2024-03-31T23:09:17.535950+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:04:33.508180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:04:33.508180+0000","last_clean_scrub_stamp":"2024-03-31T23:04:33.508180+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:20:11.211403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-31T23:09:17.537121+0000","last_change":"2024-03-31T23:09:15.746822+0000","last_active":"2024-03-31T23:09:17.537121+0000","last_peered":"2024-03-31T23:09:17.537121+0000","last_clean":"2024-03-31T23:09:17.537121+0000","last_became_active":"2024-03-31T23:09:15.746368+0000","last_became_peered":"2024-03-31T23:09:15.746368+0000","last_unstale":"2024-03-31T23:09:17.537121+0000","last_undegraded":"2024-03-31T23:09:17.537121+0000","last_fullsized":"2024-03-31T23:09:17.537121+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_clean_scrub_stamp":"2024-03-31T23:09:14.171251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T06:36:35.658042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":23,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-31T23:09:17.535071+0000","last_change":"2024-03-31T23:09:11.175336+0000","last_active":"2024-03-31T23:09:17.535071+0000","last_peered":"2024-03-31T23:09:17.535071+0000","last_clean":"2024-03-31T23:09:17.535071+0000","last_became_active":"2024-03-31T23:09:11.174768+0000","last_became_peered":"2024-03-31T23:09:11.174768+0000","last_unstale":"2024-03-31T23:09:17.535071+0000","last_undegraded":"2024-03-31T23:09:17.535071+0000","last_fullsized":"2024-03-31T23:09:17.535071+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:09:11.678744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"91'3","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-31T23:09:30.397261+0000","last_change":"2024-03-31T23:09:10.591807+0000","last_active":"2024-03-31T23:09:30.397261+0000","last_peered":"2024-03-31T23:09:30.397261+0000","last_clean":"2024-03-31T23:09:30.397261+0000","last_became_active":"2024-03-31T23:09:10.591407+0000","last_became_peered":"2024-03-31T23:09:10.591407+0000","last_unstale":"2024-03-31T23:09:30.397261+0000","last_undegraded":"2024-03-31T23:09:30.397261+0000","last_fullsized":"2024-03-31T23:09:30.397261+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_clean_scrub_stamp":"2024-03-31T23:09:09.564263+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T08:46:33.660717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-31T23:09:17.535906+0000","last_change":"2024-03-31T23:09:15.203879+0000","last_active":"2024-03-31T23:09:17.535906+0000","last_peered":"2024-03-31T23:09:17.535906+0000","last_clean":"2024-03-31T23:09:17.535906+0000","last_became_active":"2024-03-31T23:09:15.203412+0000","last_became_peered":"2024-03-31T23:09:15.203412+0000","last_unstale":"2024-03-31T23:09:17.535906+0000","last_undegraded":"2024-03-31T23:09:17.535906+0000","last_fullsized":"2024-03-31T23:09:17.535906+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_clean_scrub_stamp":"2024-03-31T23:09:14.171251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:03:54.486038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-31T23:09:17.536951+0000","last_change":"2024-03-31T23:09:10.596404+0000","last_active":"2024-03-31T23:09:17.536951+0000","last_peered":"2024-03-31T23:09:17.536951+0000","last_clean":"2024-03-31T23:09:17.536951+0000","last_became_active":"2024-03-31T23:09:10.596031+0000","last_became_peered":"2024-03-31T23:09:10.596031+0000","last_unstale":"2024-03-31T23:09:17.536951+0000","last_undegraded":"2024-03-31T23:09:17.536951+0000","last_fullsized":"2024-03-31T23:09:17.536951+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_clean_scrub_stamp":"2024-03-31T23:09:09.564263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:30:31.904241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":23,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-31T23:09:17.535531+0000","last_change":"2024-03-31T23:09:11.164300+0000","last_active":"2024-03-31T23:09:17.535531+0000","last_peered":"2024-03-31T23:09:17.535531+0000","last_clean":"2024-03-31T23:09:17.535531+0000","last_became_active":"2024-03-31T23:09:11.163960+0000","last_became_peered":"2024-03-31T23:09:11.163960+0000","last_unstale":"2024-03-31T23:09:17.535531+0000","last_undegraded":"2024-03-31T23:09:17.535531+0000","last_fullsized":"2024-03-31T23:09:17.535531+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T10:37:30.512216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-31T23:09:17.388290+0000","last_change":"2024-03-31T23:09:15.746426+0000","last_active":"2024-03-31T23:09:17.388290+0000","last_peered":"2024-03-31T23:09:17.388290+0000","last_clean":"2024-03-31T23:09:17.388290+0000","last_became_active":"2024-03-31T23:09:15.745887+0000","last_became_peered":"2024-03-31T23:09:15.745887+0000","last_unstale":"2024-03-31T23:09:17.388290+0000","last_undegraded":"2024-03-31T23:09:17.388290+0000","last_fullsized":"2024-03-31T23:09:17.388290+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_clean_scrub_stamp":"2024-03-31T23:09:14.171251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:41:52.005300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"87'1","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-31T23:09:17.184977+0000","last_change":"2024-03-31T23:09:10.590631+0000","last_active":"2024-03-31T23:09:17.184977+0000","last_peered":"2024-03-31T23:09:17.184977+0000","last_clean":"2024-03-31T23:09:17.184977+0000","last_became_active":"2024-03-31T23:09:10.590211+0000","last_became_peered":"2024-03-31T23:09:10.590211+0000","last_unstale":"2024-03-31T23:09:17.184977+0000","last_undegraded":"2024-03-31T23:09:17.184977+0000","last_fullsized":"2024-03-31T23:09:17.184977+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_clean_scrub_stamp":"2024-03-31T23:09:09.564263+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T08:51:24.978892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":23,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-31T23:09:17.388575+0000","last_change":"2024-03-31T23:09:11.177353+0000","last_active":"2024-03-31T23:09:17.388575+0000","last_peered":"2024-03-31T23:09:17.388575+0000","last_clean":"2024-03-31T23:09:17.388575+0000","last_became_active":"2024-03-31T23:09:11.176996+0000","last_became_peered":"2024-03-31T23:09:11.176996+0000","last_unstale":"2024-03-31T23:09:17.388575+0000","last_undegraded":"2024-03-31T23:09:17.388575+0000","last_fullsized":"2024-03-31T23:09:17.388575+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:20:52.486281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-31T23:09:17.186103+0000","last_change":"2024-03-31T23:09:15.198528+0000","last_active":"2024-03-31T23:09:17.186103+0000","last_peered":"2024-03-31T23:09:17.186103+0000","last_clean":"2024-03-31T23:09:17.186103+0000","last_became_active":"2024-03-31T23:09:15.197938+0000","last_became_peered":"2024-03-31T23:09:15.197938+0000","last_unstale":"2024-03-31T23:09:17.186103+0000","last_undegraded":"2024-03-31T23:09:17.186103+0000","last_fullsized":"2024-03-31T23:09:17.186103+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_clean_scrub_stamp":"2024-03-31T23:09:14.171251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:45:46.567268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"85'2","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-31T23:09:17.185980+0000","last_change":"2024-03-31T23:09:10.584779+0000","last_active":"2024-03-31T23:09:17.185980+0000","last_peered":"2024-03-31T23:09:17.185980+0000","last_clean":"2024-03-31T23:09:17.185980+0000","last_became_active":"2024-03-31T23:09:10.584400+0000","last_became_peered":"2024-03-31T23:09:10.584400+0000","last_unstale":"2024-03-31T23:09:17.185980+0000","last_undegraded":"2024-03-31T23:09:17.185980+0000","last_fullsized":"2024-03-31T23:09:17.185980+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_clean_scrub_stamp":"2024-03-31T23:09:09.564263+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:10:10.095704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":23,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-31T23:09:17.770617+0000","last_change":"2024-03-31T23:09:11.177152+0000","last_active":"2024-03-31T23:09:17.770617+0000","last_peered":"2024-03-31T23:09:17.770617+0000","last_clean":"2024-03-31T23:09:17.770617+0000","last_became_active":"2024-03-31T23:09:11.176797+0000","last_became_peered":"2024-03-31T23:09:11.176797+0000","last_unstale":"2024-03-31T23:09:17.770617+0000","last_undegraded":"2024-03-31T23:09:17.770617+0000","last_fullsized":"2024-03-31T23:09:17.770617+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:19:49.070930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-31T23:09:17.185904+0000","last_change":"2024-03-31T23:09:15.746525+0000","last_active":"2024-03-31T23:09:17.185904+0000","last_peered":"2024-03-31T23:09:17.185904+0000","last_clean":"2024-03-31T23:09:17.185904+0000","last_became_active":"2024-03-31T23:09:15.746126+0000","last_became_peered":"2024-03-31T23:09:15.746126+0000","last_unstale":"2024-03-31T23:09:17.185904+0000","last_undegraded":"2024-03-31T23:09:17.185904+0000","last_fullsized":"2024-03-31T23:09:17.185904+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_clean_scrub_stamp":"2024-03-31T23:09:14.171251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:04:57.313615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"91'2","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-31T23:09:29.423929+0000","last_change":"2024-03-31T23:09:10.591469+0000","last_active":"2024-03-31T23:09:29.423929+0000","last_peered":"2024-03-31T23:09:29.423929+0000","last_clean":"2024-03-31T23:09:29.423929+0000","last_became_active":"2024-03-31T23:09:10.591166+0000","last_became_peered":"2024-03-31T23:09:10.591166+0000","last_unstale":"2024-03-31T23:09:29.423929+0000","last_undegraded":"2024-03-31T23:09:29.423929+0000","last_fullsized":"2024-03-31T23:09:29.423929+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_clean_scrub_stamp":"2024-03-31T23:09:09.564263+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T10:30:40.279041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":23,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-31T23:09:17.536582+0000","last_change":"2024-03-31T23:09:11.174471+0000","last_active":"2024-03-31T23:09:17.536582+0000","last_peered":"2024-03-31T23:09:17.536582+0000","last_clean":"2024-03-31T23:09:17.536582+0000","last_became_active":"2024-03-31T23:09:11.174154+0000","last_became_peered":"2024-03-31T23:09:11.174154+0000","last_unstale":"2024-03-31T23:09:17.536582+0000","last_undegraded":"2024-03-31T23:09:17.536582+0000","last_fullsized":"2024-03-31T23:09:17.536582+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T06:10:14.727009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-31T23:09:17.185327+0000","last_change":"2024-03-31T23:09:15.201416+0000","last_active":"2024-03-31T23:09:17.185327+0000","last_peered":"2024-03-31T23:09:17.185327+0000","last_clean":"2024-03-31T23:09:17.185327+0000","last_became_active":"2024-03-31T23:09:15.200987+0000","last_became_peered":"2024-03-31T23:09:15.200987+0000","last_unstale":"2024-03-31T23:09:17.185327+0000","last_undegraded":"2024-03-31T23:09:17.185327+0000","last_fullsized":"2024-03-31T23:09:17.185327+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_clean_scrub_stamp":"2024-03-31T23:09:14.171251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T10:56:54.872988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-31T23:09:17.388235+0000","last_change":"2024-03-31T23:09:10.594035+0000","last_active":"2024-03-31T23:09:17.388235+0000","last_peered":"2024-03-31T23:09:17.388235+0000","last_clean":"2024-03-31T23:09:17.388235+0000","last_became_active":"2024-03-31T23:09:10.593710+0000","last_became_peered":"2024-03-31T23:09:10.593710+0000","last_unstale":"2024-03-31T23:09:17.388235+0000","last_undegraded":"2024-03-31T23:09:17.388235+0000","last_fullsized":"2024-03-31T23:09:17.388235+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_clean_scrub_stamp":"2024-03-31T23:09:09.564263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:46:48.580244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":23,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-31T23:09:17.535932+0000","last_change":"2024-03-31T23:09:11.170501+0000","last_active":"2024-03-31T23:09:17.535932+0000","last_peered":"2024-03-31T23:09:17.535932+0000","last_clean":"2024-03-31T23:09:17.535932+0000","last_became_active":"2024-03-31T23:09:11.170068+0000","last_became_peered":"2024-03-31T23:09:11.170068+0000","last_unstale":"2024-03-31T23:09:17.535932+0000","last_undegraded":"2024-03-31T23:09:17.535932+0000","last_fullsized":"2024-03-31T23:09:17.535932+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T06:05:53.488957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-31T23:09:17.185840+0000","last_change":"2024-03-31T23:09:15.208092+0000","last_active":"2024-03-31T23:09:17.185840+0000","last_peered":"2024-03-31T23:09:17.185840+0000","last_clean":"2024-03-31T23:09:17.185840+0000","last_became_active":"2024-03-31T23:09:15.207750+0000","last_became_peered":"2024-03-31T23:09:15.207750+0000","last_unstale":"2024-03-31T23:09:17.185840+0000","last_undegraded":"2024-03-31T23:09:17.185840+0000","last_fullsized":"2024-03-31T23:09:17.185840+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_clean_scrub_stamp":"2024-03-31T23:09:14.171251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:24:39.649569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-31T23:09:17.186198+0000","last_change":"2024-03-31T23:09:10.591275+0000","last_active":"2024-03-31T23:09:17.186198+0000","last_peered":"2024-03-31T23:09:17.186198+0000","last_clean":"2024-03-31T23:09:17.186198+0000","last_became_active":"2024-03-31T23:09:10.590652+0000","last_became_peered":"2024-03-31T23:09:10.590652+0000","last_unstale":"2024-03-31T23:09:17.186198+0000","last_undegraded":"2024-03-31T23:09:17.186198+0000","last_fullsized":"2024-03-31T23:09:17.186198+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_clean_scrub_stamp":"2024-03-31T23:09:09.564263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:58:39.037196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":23,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-31T23:09:17.388365+0000","last_change":"2024-03-31T23:09:11.175169+0000","last_active":"2024-03-31T23:09:17.388365+0000","last_peered":"2024-03-31T23:09:17.388365+0000","last_clean":"2024-03-31T23:09:17.388365+0000","last_became_active":"2024-03-31T23:09:11.174849+0000","last_became_peered":"2024-03-31T23:09:11.174849+0000","last_unstale":"2024-03-31T23:09:17.388365+0000","last_undegraded":"2024-03-31T23:09:17.388365+0000","last_fullsized":"2024-03-31T23:09:17.388365+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:58:06.814034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-31T23:09:17.185805+0000","last_change":"2024-03-31T23:09:15.746423+0000","last_active":"2024-03-31T23:09:17.185805+0000","last_peered":"2024-03-31T23:09:17.185805+0000","last_clean":"2024-03-31T23:09:17.185805+0000","last_became_active":"2024-03-31T23:09:15.745942+0000","last_became_peered":"2024-03-31T23:09:15.745942+0000","last_unstale":"2024-03-31T23:09:17.185805+0000","last_undegraded":"2024-03-31T23:09:17.185805+0000","last_fullsized":"2024-03-31T23:09:17.185805+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_clean_scrub_stamp":"2024-03-31T23:09:14.171251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:56:02.903212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"85'2","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-31T23:09:17.536825+0000","last_change":"2024-03-31T23:09:10.597524+0000","last_active":"2024-03-31T23:09:17.536825+0000","last_peered":"2024-03-31T23:09:17.536825+0000","last_clean":"2024-03-31T23:09:17.536825+0000","last_became_active":"2024-03-31T23:09:10.597222+0000","last_became_peered":"2024-03-31T23:09:10.597222+0000","last_unstale":"2024-03-31T23:09:17.536825+0000","last_undegraded":"2024-03-31T23:09:17.536825+0000","last_fullsized":"2024-03-31T23:09:17.536825+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_clean_scrub_stamp":"2024-03-31T23:09:09.564263+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T09:24:20.160978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":23,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-31T23:09:17.536417+0000","last_change":"2024-03-31T23:09:11.170440+0000","last_active":"2024-03-31T23:09:17.536417+0000","last_peered":"2024-03-31T23:09:17.536417+0000","last_clean":"2024-03-31T23:09:17.536417+0000","last_became_active":"2024-03-31T23:09:11.170081+0000","last_became_peered":"2024-03-31T23:09:11.170081+0000","last_unstale":"2024-03-31T23:09:17.536417+0000","last_undegraded":"2024-03-31T23:09:17.536417+0000","last_fullsized":"2024-03-31T23:09:17.536417+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:53:31.429822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-31T23:09:17.536228+0000","last_change":"2024-03-31T23:09:15.207146+0000","last_active":"2024-03-31T23:09:17.536228+0000","last_peered":"2024-03-31T23:09:17.536228+0000","last_clean":"2024-03-31T23:09:17.536228+0000","last_became_active":"2024-03-31T23:09:15.206769+0000","last_became_peered":"2024-03-31T23:09:15.206769+0000","last_unstale":"2024-03-31T23:09:17.536228+0000","last_undegraded":"2024-03-31T23:09:17.536228+0000","last_fullsized":"2024-03-31T23:09:17.536228+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_clean_scrub_stamp":"2024-03-31T23:09:14.171251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T07:42:12.961109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-31T23:09:17.186427+0000","last_change":"2024-03-31T23:09:10.587291+0000","last_active":"2024-03-31T23:09:17.186427+0000","last_peered":"2024-03-31T23:09:17.186427+0000","last_clean":"2024-03-31T23:09:17.186427+0000","last_became_active":"2024-03-31T23:09:10.585992+0000","last_became_peered":"2024-03-31T23:09:10.585992+0000","last_unstale":"2024-03-31T23:09:17.186427+0000","last_undegraded":"2024-03-31T23:09:17.186427+0000","last_fullsized":"2024-03-31T23:09:17.186427+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_clean_scrub_stamp":"2024-03-31T23:09:09.564263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:28:00.786268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":23,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-31T23:09:17.535786+0000","last_change":"2024-03-31T23:09:11.179369+0000","last_active":"2024-03-31T23:09:17.535786+0000","last_peered":"2024-03-31T23:09:17.535786+0000","last_clean":"2024-03-31T23:09:17.535786+0000","last_became_active":"2024-03-31T23:09:11.178776+0000","last_became_peered":"2024-03-31T23:09:11.178776+0000","last_unstale":"2024-03-31T23:09:17.535786+0000","last_undegraded":"2024-03-31T23:09:17.535786+0000","last_fullsized":"2024-03-31T23:09:17.535786+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T07:56:05.201338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-31T23:09:17.770547+0000","last_change":"2024-03-31T23:09:15.201462+0000","last_active":"2024-03-31T23:09:17.770547+0000","last_peered":"2024-03-31T23:09:17.770547+0000","last_clean":"2024-03-31T23:09:17.770547+0000","last_became_active":"2024-03-31T23:09:15.201067+0000","last_became_peered":"2024-03-31T23:09:15.201067+0000","last_unstale":"2024-03-31T23:09:17.770547+0000","last_undegraded":"2024-03-31T23:09:17.770547+0000","last_fullsized":"2024-03-31T23:09:17.770547+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_clean_scrub_stamp":"2024-03-31T23:09:14.171251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:54:29.339456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"85'1","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-31T23:09:17.537351+0000","last_change":"2024-03-31T23:09:10.584752+0000","last_active":"2024-03-31T23:09:17.537351+0000","last_peered":"2024-03-31T23:09:17.537351+0000","last_clean":"2024-03-31T23:09:17.537351+0000","last_became_active":"2024-03-31T23:09:10.584432+0000","last_became_peered":"2024-03-31T23:09:10.584432+0000","last_unstale":"2024-03-31T23:09:17.537351+0000","last_undegraded":"2024-03-31T23:09:17.537351+0000","last_fullsized":"2024-03-31T23:09:17.537351+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_clean_scrub_stamp":"2024-03-31T23:09:09.564263+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T10:50:53.889719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":23,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-31T23:09:17.536462+0000","last_change":"2024-03-31T23:09:11.178540+0000","last_active":"2024-03-31T23:09:17.536462+0000","last_peered":"2024-03-31T23:09:17.536462+0000","last_clean":"2024-03-31T23:09:17.536462+0000","last_became_active":"2024-03-31T23:09:11.178171+0000","last_became_peered":"2024-03-31T23:09:11.178171+0000","last_unstale":"2024-03-31T23:09:17.536462+0000","last_undegraded":"2024-03-31T23:09:17.536462+0000","last_fullsized":"2024-03-31T23:09:17.536462+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:28:46.287915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-31T23:09:17.535754+0000","last_change":"2024-03-31T23:09:15.197271+0000","last_active":"2024-03-31T23:09:17.535754+0000","last_peered":"2024-03-31T23:09:17.535754+0000","last_clean":"2024-03-31T23:09:17.535754+0000","last_became_active":"2024-03-31T23:09:15.196339+0000","last_became_peered":"2024-03-31T23:09:15.196339+0000","last_unstale":"2024-03-31T23:09:17.535754+0000","last_undegraded":"2024-03-31T23:09:17.535754+0000","last_fullsized":"2024-03-31T23:09:17.535754+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_clean_scrub_stamp":"2024-03-31T23:09:14.171251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:44:13.044278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"85'2","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-31T23:09:17.535586+0000","last_change":"2024-03-31T23:09:10.595131+0000","last_active":"2024-03-31T23:09:17.535586+0000","last_peered":"2024-03-31T23:09:17.535586+0000","last_clean":"2024-03-31T23:09:17.535586+0000","last_became_active":"2024-03-31T23:09:10.594740+0000","last_became_peered":"2024-03-31T23:09:10.594740+0000","last_unstale":"2024-03-31T23:09:17.535586+0000","last_undegraded":"2024-03-31T23:09:17.535586+0000","last_fullsized":"2024-03-31T23:09:17.535586+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_clean_scrub_stamp":"2024-03-31T23:09:09.564263+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:52:41.849621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":23,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-31T23:09:17.535797+0000","last_change":"2024-03-31T23:09:11.177530+0000","last_active":"2024-03-31T23:09:17.535797+0000","last_peered":"2024-03-31T23:09:17.535797+0000","last_clean":"2024-03-31T23:09:17.535797+0000","last_became_active":"2024-03-31T23:09:11.176972+0000","last_became_peered":"2024-03-31T23:09:11.176972+0000","last_unstale":"2024-03-31T23:09:17.535797+0000","last_undegraded":"2024-03-31T23:09:17.535797+0000","last_fullsized":"2024-03-31T23:09:17.535797+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T07:57:21.094911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":23,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-31T23:09:17.536072+0000","last_change":"2024-03-31T23:09:11.172792+0000","last_active":"2024-03-31T23:09:17.536072+0000","last_peered":"2024-03-31T23:09:17.536072+0000","last_clean":"2024-03-31T23:09:17.536072+0000","last_became_active":"2024-03-31T23:09:11.172463+0000","last_became_peered":"2024-03-31T23:09:11.172463+0000","last_unstale":"2024-03-31T23:09:17.536072+0000","last_undegraded":"2024-03-31T23:09:17.536072+0000","last_fullsized":"2024-03-31T23:09:17.536072+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:07:28.789783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":23,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-31T23:09:17.186302+0000","last_change":"2024-03-31T23:09:11.174331+0000","last_active":"2024-03-31T23:09:17.186302+0000","last_peered":"2024-03-31T23:09:17.186302+0000","last_clean":"2024-03-31T23:09:17.186302+0000","last_became_active":"2024-03-31T23:09:11.174001+0000","last_became_peered":"2024-03-31T23:09:11.174001+0000","last_unstale":"2024-03-31T23:09:17.186302+0000","last_undegraded":"2024-03-31T23:09:17.186302+0000","last_fullsized":"2024-03-31T23:09:17.186302+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:26:28.005406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":23,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-31T23:09:17.185924+0000","last_change":"2024-03-31T23:09:11.180047+0000","last_active":"2024-03-31T23:09:17.185924+0000","last_peered":"2024-03-31T23:09:17.185924+0000","last_clean":"2024-03-31T23:09:17.185924+0000","last_became_active":"2024-03-31T23:09:11.179691+0000","last_became_peered":"2024-03-31T23:09:11.179691+0000","last_unstale":"2024-03-31T23:09:17.185924+0000","last_undegraded":"2024-03-31T23:09:17.185924+0000","last_fullsized":"2024-03-31T23:09:17.185924+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:28:05.083710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":23,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-31T23:09:17.770788+0000","last_change":"2024-03-31T23:09:11.177642+0000","last_active":"2024-03-31T23:09:17.770788+0000","last_peered":"2024-03-31T23:09:17.770788+0000","last_clean":"2024-03-31T23:09:17.770788+0000","last_became_active":"2024-03-31T23:09:11.177261+0000","last_became_peered":"2024-03-31T23:09:11.177261+0000","last_unstale":"2024-03-31T23:09:17.770788+0000","last_undegraded":"2024-03-31T23:09:17.770788+0000","last_fullsized":"2024-03-31T23:09:17.770788+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T07:10:43.616863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":23,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-31T23:09:17.535424+0000","last_change":"2024-03-31T23:09:11.173606+0000","last_active":"2024-03-31T23:09:17.535424+0000","last_peered":"2024-03-31T23:09:17.535424+0000","last_clean":"2024-03-31T23:09:17.535424+0000","last_became_active":"2024-03-31T23:09:11.173304+0000","last_became_peered":"2024-03-31T23:09:11.173304+0000","last_unstale":"2024-03-31T23:09:17.535424+0000","last_undegraded":"2024-03-31T23:09:17.535424+0000","last_fullsized":"2024-03-31T23:09:17.535424+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:53:06.112181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":23,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-31T23:09:17.536187+0000","last_change":"2024-03-31T23:09:11.176628+0000","last_active":"2024-03-31T23:09:17.536187+0000","last_peered":"2024-03-31T23:09:17.536187+0000","last_clean":"2024-03-31T23:09:17.536187+0000","last_became_active":"2024-03-31T23:09:11.176318+0000","last_became_peered":"2024-03-31T23:09:11.176318+0000","last_unstale":"2024-03-31T23:09:17.536187+0000","last_undegraded":"2024-03-31T23:09:17.536187+0000","last_fullsized":"2024-03-31T23:09:17.536187+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:06:40.723675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":23,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-31T23:09:17.186737+0000","last_change":"2024-03-31T23:09:11.169318+0000","last_active":"2024-03-31T23:09:17.186737+0000","last_peered":"2024-03-31T23:09:17.186737+0000","last_clean":"2024-03-31T23:09:17.186737+0000","last_became_active":"2024-03-31T23:09:11.169026+0000","last_became_peered":"2024-03-31T23:09:11.169026+0000","last_unstale":"2024-03-31T23:09:17.186737+0000","last_undegraded":"2024-03-31T23:09:17.186737+0000","last_fullsized":"2024-03-31T23:09:17.186737+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:02:18.562854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":23,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-31T23:09:17.388686+0000","last_change":"2024-03-31T23:09:11.184670+0000","last_active":"2024-03-31T23:09:17.388686+0000","last_peered":"2024-03-31T23:09:17.388686+0000","last_clean":"2024-03-31T23:09:17.388686+0000","last_became_active":"2024-03-31T23:09:11.184354+0000","last_became_peered":"2024-03-31T23:09:11.184354+0000","last_unstale":"2024-03-31T23:09:17.388686+0000","last_undegraded":"2024-03-31T23:09:17.388686+0000","last_fullsized":"2024-03-31T23:09:17.388686+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:16:14.354947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":23,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-31T23:09:17.536042+0000","last_change":"2024-03-31T23:09:11.180991+0000","last_active":"2024-03-31T23:09:17.536042+0000","last_peered":"2024-03-31T23:09:17.536042+0000","last_clean":"2024-03-31T23:09:17.536042+0000","last_became_active":"2024-03-31T23:09:11.180667+0000","last_became_peered":"2024-03-31T23:09:11.180667+0000","last_unstale":"2024-03-31T23:09:17.536042+0000","last_undegraded":"2024-03-31T23:09:17.536042+0000","last_fullsized":"2024-03-31T23:09:17.536042+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T06:35:33.157176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":23,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-31T23:09:17.388440+0000","last_change":"2024-03-31T23:09:11.182025+0000","last_active":"2024-03-31T23:09:17.388440+0000","last_peered":"2024-03-31T23:09:17.388440+0000","last_clean":"2024-03-31T23:09:17.388440+0000","last_became_active":"2024-03-31T23:09:11.179310+0000","last_became_peered":"2024-03-31T23:09:11.179310+0000","last_unstale":"2024-03-31T23:09:17.388440+0000","last_undegraded":"2024-03-31T23:09:17.388440+0000","last_fullsized":"2024-03-31T23:09:17.388440+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T10:21:40.784738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":23,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-31T23:09:17.536489+0000","last_change":"2024-03-31T23:09:11.176594+0000","last_active":"2024-03-31T23:09:17.536489+0000","last_peered":"2024-03-31T23:09:17.536489+0000","last_clean":"2024-03-31T23:09:17.536489+0000","last_became_active":"2024-03-31T23:09:11.176306+0000","last_became_peered":"2024-03-31T23:09:11.176306+0000","last_unstale":"2024-03-31T23:09:17.536489+0000","last_undegraded":"2024-03-31T23:09:17.536489+0000","last_fullsized":"2024-03-31T23:09:17.536489+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T09:08:50.736985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":23,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-31T23:09:17.535382+0000","last_change":"2024-03-31T23:09:11.172585+0000","last_active":"2024-03-31T23:09:17.535382+0000","last_peered":"2024-03-31T23:09:17.535382+0000","last_clean":"2024-03-31T23:09:17.535382+0000","last_became_active":"2024-03-31T23:09:11.172266+0000","last_became_peered":"2024-03-31T23:09:11.172266+0000","last_unstale":"2024-03-31T23:09:17.535382+0000","last_undegraded":"2024-03-31T23:09:17.535382+0000","last_fullsized":"2024-03-31T23:09:17.535382+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:38:26.570175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":23,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-31T23:09:17.536165+0000","last_change":"2024-03-31T23:09:11.181434+0000","last_active":"2024-03-31T23:09:17.536165+0000","last_peered":"2024-03-31T23:09:17.536165+0000","last_clean":"2024-03-31T23:09:17.536165+0000","last_became_active":"2024-03-31T23:09:11.181140+0000","last_became_peered":"2024-03-31T23:09:11.181140+0000","last_unstale":"2024-03-31T23:09:17.536165+0000","last_undegraded":"2024-03-31T23:09:17.536165+0000","last_fullsized":"2024-03-31T23:09:17.536165+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:55:06.379696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4157440,"data_stored":4132576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":6325,"num_objects":32,"num_object_clones":0,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":72,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":20,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":143360,"data_stored":18159,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":67,"ondisk_log_size":67,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":81,"seq":347892350993,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36324,"kb_used_data":988,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708828,"statfs":{"total":95995035648,"available":95957839872,"internally_reserved":0,"allocated":1011712,"data_stored":491116,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":36174276},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":73,"seq":313532612630,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28608,"kb_used_data":1592,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716544,"statfs":{"total":95995035648,"available":95965741056,"internally_reserved":0,"allocated":1630208,"data_stored":1085520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":68,"seq":292057776155,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28012,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717140,"statfs":{"total":95995035648,"available":95966351360,"internally_reserved":0,"allocated":1015808,"data_stored":491698,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":60,"seq":257698037791,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28016,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717136,"statfs":{"total":95995035648,"available":95966347264,"internally_reserved":0,"allocated":1019904,"data_stored":491720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299429,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28600,"kb_used_data":1584,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716552,"statfs":{"total":95995035648,"available":95965749248,"internally_reserved":0,"allocated":1622016,"data_stored":1083276,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1588,"internal_metadata":27654604},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863462954,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28004,"kb_used_data":988,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717148,"statfs":{"total":95995035648,"available":95966359552,"internally_reserved":0,"allocated":1011712,"data_stored":491116,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574914,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28016,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717136,"statfs":{"total":95995035648,"available":95966347264,"internally_reserved":0,"allocated":1024000,"data_stored":494958,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738439,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36340,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708812,"statfs":{"total":95995035648,"available":95957823488,"internally_reserved":0,"allocated":1015808,"data_stored":491206,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411390,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28020,"kb_used_data":1004,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717132,"statfs":{"total":95995035648,"available":95966343168,"internally_reserved":0,"allocated":1028096,"data_stored":494992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182457,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28020,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717132,"statfs":{"total":95995035648,"available":95966343168,"internally_reserved":0,"allocated":1019904,"data_stored":491220,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920820,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32176,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712976,"statfs":{"total":95995035648,"available":95962087424,"internally_reserved":0,"allocated":1015808,"data_stored":491138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724592,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36932,"kb_used_data":1596,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708220,"statfs":{"total":95995035648,"available":95957217280,"internally_reserved":0,"allocated":1634304,"data_stored":1082960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":3876,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1867,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":4798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-31T23:09:34.546 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-31T23:09:34.546 INFO:tasks.cephadm:Setting up client nodes... 2024-03-31T23:09:34.547 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:384f31f3d3aee7f8ba3f60846b8844c049165583 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6a9d8f08-efb2-11ee-b647-cb9ed24678a4 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-03-31T23:09:34.693 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:09:34 smithi142 ceph-mon[35271]: pgmap v303: 129 pgs: 129 active+clean; 583 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s rd, 853 B/s wr, 2 op/s 2024-03-31T23:09:34.694 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:09:34 smithi142 ceph-mon[35271]: from='client.14922 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-31T23:09:34.694 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:09:34 smithi142 ceph-mon[35271]: from='client.14928 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-31T23:09:34.694 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:09:34 smithi142 ceph-mon[35271]: mgrmap e17: x(active, since 7m), standbys: y, z 2024-03-31T23:09:34.694 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:09:34 smithi142 ceph-mon[35271]: from='client.? 172.21.15.46:0/2481721173' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-31T23:09:34.694 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:09:34 smithi142 ceph-mon[35271]: from='client.? 172.21.15.46:0/2013748687' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-31T23:09:34.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:09:34 smithi162 ceph-mon[36237]: pgmap v303: 129 pgs: 129 active+clean; 583 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s rd, 853 B/s wr, 2 op/s 2024-03-31T23:09:34.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:09:34 smithi162 ceph-mon[36237]: from='client.14922 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-31T23:09:34.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:09:34 smithi162 ceph-mon[36237]: from='client.14928 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-31T23:09:34.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:09:34 smithi162 ceph-mon[36237]: mgrmap e17: x(active, since 7m), standbys: y, z 2024-03-31T23:09:34.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:09:34 smithi162 ceph-mon[36237]: from='client.? 172.21.15.46:0/2481721173' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-31T23:09:34.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:09:34 smithi162 ceph-mon[36237]: from='client.? 172.21.15.46:0/2013748687' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-31T23:09:34.826 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/6a9d8f08-efb2-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-31T23:09:35.612 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:09:35 smithi046 ceph-mon[32498]: from='client.14946 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-31T23:09:35.612 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:09:35 smithi046 ceph-mon[32498]: from='client.14952 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-31T23:09:35.693 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:09:35 smithi142 ceph-mon[35271]: from='client.14946 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-31T23:09:35.693 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:09:35 smithi142 ceph-mon[35271]: from='client.14952 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-31T23:09:35.795 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:09:35 smithi162 ceph-mon[36237]: from='client.14946 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-31T23:09:35.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:09:35 smithi162 ceph-mon[36237]: from='client.14952 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-31T23:09:36.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:36.555+0000 7f73234f0700 1 -- 172.21.15.46:0/4108103665 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f731c103490 msgr2=0x7f731c1038f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:09:36.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:36.555+0000 7f73234f0700 1 --2- 172.21.15.46:0/4108103665 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f731c103490 0x7f731c1038f0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f730c0099c0 tx=0x7f730c009cd0 comp rx=0 tx=0).stop 2024-03-31T23:09:36.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:36.555+0000 7f73234f0700 1 -- 172.21.15.46:0/4108103665 shutdown_connections 2024-03-31T23:09:36.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:36.555+0000 7f73234f0700 1 --2- 172.21.15.46:0/4108103665 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f731c103e30 0x7f731c10a6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:36.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:36.555+0000 7f73234f0700 1 --2- 172.21.15.46:0/4108103665 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f731c102290 0x7f731c1026b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:36.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:36.555+0000 7f73234f0700 1 --2- 172.21.15.46:0/4108103665 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f731c103490 0x7f731c1038f0 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:36.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:36.555+0000 7f73234f0700 1 -- 172.21.15.46:0/4108103665 >> 172.21.15.46:0/4108103665 conn(0x7f731c0fd850 msgr2=0x7f731c0ffc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:09:36.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:36.555+0000 7f73234f0700 1 -- 172.21.15.46:0/4108103665 shutdown_connections 2024-03-31T23:09:36.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:36.555+0000 7f73234f0700 1 -- 172.21.15.46:0/4108103665 wait complete. 2024-03-31T23:09:36.558 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:09:36 smithi046 ceph-mon[32498]: pgmap v304: 129 pgs: 129 active+clean; 584 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 2.7 KiB/s wr, 8 op/s 2024-03-31T23:09:36.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:36.556+0000 7f73234f0700 1 Processor -- start 2024-03-31T23:09:36.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:36.556+0000 7f73234f0700 1 -- start start 2024-03-31T23:09:36.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:36.556+0000 7f73234f0700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f731c102290 0x7f731c075fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:09:36.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:36.556+0000 7f73234f0700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f731c103490 0x7f731c074630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:09:36.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:36.556+0000 7f73234f0700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f731c103e30 0x7f731c074b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:09:36.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:36.557+0000 7f7320a8b700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f731c103490 0x7f731c074630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:09:36.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:36.557+0000 7f7320a8b700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f731c103490 0x7f731c074630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.46:41074/0 (socket says 172.21.15.46:41074) 2024-03-31T23:09:36.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:36.557+0000 7f732128c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f731c102290 0x7f731c075fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:09:36.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:36.557+0000 7f732128c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f731c102290 0x7f731c075fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:57132/0 (socket says 172.21.15.46:57132) 2024-03-31T23:09:36.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:36.557+0000 7f7320a8b700 1 -- 172.21.15.46:0/2391619456 learned_addr learned my addr 172.21.15.46:0/2391619456 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:09:36.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:36.557+0000 7f73234f0700 1 -- 172.21.15.46:0/2391619456 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f731c10dd20 con 0x7f731c102290 2024-03-31T23:09:36.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:36.557+0000 7f73234f0700 1 -- 172.21.15.46:0/2391619456 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f731c10dba0 con 0x7f731c103e30 2024-03-31T23:09:36.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:36.557+0000 7f73234f0700 1 -- 172.21.15.46:0/2391619456 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f731c10dea0 con 0x7f731c103490 2024-03-31T23:09:36.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:36.557+0000 7f7321a8d700 1 --2- 172.21.15.46:0/2391619456 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f731c103e30 0x7f731c074b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:09:36.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:36.557+0000 7f7320a8b700 1 -- 172.21.15.46:0/2391619456 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f731c103e30 msgr2=0x7f731c074b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:09:36.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:36.557+0000 7f7320a8b700 1 --2- 172.21.15.46:0/2391619456 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f731c103e30 0x7f731c074b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:36.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:36.557+0000 7f7320a8b700 1 -- 172.21.15.46:0/2391619456 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f731c102290 msgr2=0x7f731c075fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:09:36.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:36.557+0000 7f7320a8b700 1 --2- 172.21.15.46:0/2391619456 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f731c102290 0x7f731c075fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:36.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:36.557+0000 7f7320a8b700 1 -- 172.21.15.46:0/2391619456 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f731c075420 con 0x7f731c103490 2024-03-31T23:09:36.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:36.557+0000 7f732128c700 1 --2- 172.21.15.46:0/2391619456 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f731c102290 0x7f731c075fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:09:36.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:36.557+0000 7f7321a8d700 1 --2- 172.21.15.46:0/2391619456 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f731c103e30 0x7f731c074b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T23:09:36.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:36.557+0000 7f7320a8b700 1 --2- 172.21.15.46:0/2391619456 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f731c103490 0x7f731c074630 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f730c0036f0 tx=0x7f730c00f690 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:09:36.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:36.557+0000 7f73127fc700 1 -- 172.21.15.46:0/2391619456 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f730c009e60 con 0x7f731c103490 2024-03-31T23:09:36.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:36.558+0000 7f73127fc700 1 -- 172.21.15.46:0/2391619456 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f730c003d70 con 0x7f731c103490 2024-03-31T23:09:36.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:36.558+0000 7f73234f0700 1 -- 172.21.15.46:0/2391619456 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f731c1a2410 con 0x7f731c103490 2024-03-31T23:09:36.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:36.558+0000 7f73127fc700 1 -- 172.21.15.46:0/2391619456 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f730c0056a0 con 0x7f731c103490 2024-03-31T23:09:36.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:36.558+0000 7f73234f0700 1 -- 172.21.15.46:0/2391619456 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f731c1a2860 con 0x7f731c103490 2024-03-31T23:09:36.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:36.559+0000 7f73234f0700 1 -- 172.21.15.46:0/2391619456 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f731c050750 con 0x7f731c103490 2024-03-31T23:09:36.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:36.559+0000 7f73127fc700 1 -- 172.21.15.46:0/2391619456 <== mon.1 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 17) ==== 137371+0+0 (secure 0 0 0) 0x7f730c0038c0 con 0x7f731c103490 2024-03-31T23:09:36.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:36.560+0000 7f73127fc700 1 --2- 172.21.15.46:0/2391619456 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f73080a2670 0x7f73080a4b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:09:36.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:36.560+0000 7f732128c700 1 --2- 172.21.15.46:0/2391619456 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f73080a2670 0x7f73080a4b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:09:36.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:36.560+0000 7f73127fc700 1 -- 172.21.15.46:0/2391619456 <== mon.1 v2:172.21.15.162:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8802+0+0 (secure 0 0 0) 0x7f730c0cf910 con 0x7f731c103490 2024-03-31T23:09:36.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:36.561+0000 7f732128c700 1 --2- 172.21.15.46:0/2391619456 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f73080a2670 0x7f73080a4b30 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f731c1032f0 tx=0x7f7318009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:09:36.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:36.565+0000 7f73127fc700 1 -- 172.21.15.46:0/2391619456 <== mon.1 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f730c013070 con 0x7f731c103490 2024-03-31T23:09:36.693 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:09:36 smithi142 ceph-mon[35271]: pgmap v304: 129 pgs: 129 active+clean; 584 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 2.7 KiB/s wr, 8 op/s 2024-03-31T23:09:36.795 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:09:36 smithi162 ceph-mon[36237]: pgmap v304: 129 pgs: 129 active+clean; 584 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 2.7 KiB/s wr, 8 op/s 2024-03-31T23:09:36.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:36.854+0000 7f73234f0700 1 -- 172.21.15.46:0/2391619456 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) -- 0x7f731c062e50 con 0x7f731c103490 2024-03-31T23:09:36.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:36.861+0000 7f73127fc700 1 -- 172.21.15.46:0/2391619456 <== mon.1 v2:172.21.15.162:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v32) ==== 170+0+59 (secure 0 0 0) 0x7f730c09c7c0 con 0x7f731c103490 2024-03-31T23:09:36.862 INFO:teuthology.orchestra.run.smithi046.stdout:[client.0] 2024-03-31T23:09:36.862 INFO:teuthology.orchestra.run.smithi046.stdout: key = AQAw7Qlm+RQLMxAACthcv3de6mwMvpnF7vtkkw== 2024-03-31T23:09:36.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:36.863+0000 7f73234f0700 1 -- 172.21.15.46:0/2391619456 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f73080a2670 msgr2=0x7f73080a4b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:09:36.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:36.863+0000 7f73234f0700 1 --2- 172.21.15.46:0/2391619456 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f73080a2670 0x7f73080a4b30 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f731c1032f0 tx=0x7f7318009260 comp rx=0 tx=0).stop 2024-03-31T23:09:36.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:36.863+0000 7f73234f0700 1 -- 172.21.15.46:0/2391619456 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f731c103490 msgr2=0x7f731c074630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:09:36.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:36.863+0000 7f73234f0700 1 --2- 172.21.15.46:0/2391619456 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f731c103490 0x7f731c074630 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f730c0036f0 tx=0x7f730c00f690 comp rx=0 tx=0).stop 2024-03-31T23:09:36.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:36.863+0000 7f73234f0700 1 -- 172.21.15.46:0/2391619456 shutdown_connections 2024-03-31T23:09:36.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:36.863+0000 7f73234f0700 1 --2- 172.21.15.46:0/2391619456 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f73080a2670 0x7f73080a4b30 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:36.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:36.863+0000 7f73234f0700 1 --2- 172.21.15.46:0/2391619456 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f731c102290 0x7f731c075fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:36.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:36.863+0000 7f73234f0700 1 --2- 172.21.15.46:0/2391619456 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f731c103e30 0x7f731c074b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:36.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:36.863+0000 7f73234f0700 1 --2- 172.21.15.46:0/2391619456 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f731c103490 0x7f731c074630 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:36.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:36.863+0000 7f73234f0700 1 -- 172.21.15.46:0/2391619456 >> 172.21.15.46:0/2391619456 conn(0x7f731c0fd850 msgr2=0x7f731c106790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:09:36.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:36.863+0000 7f73234f0700 1 -- 172.21.15.46:0/2391619456 shutdown_connections 2024-03-31T23:09:36.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:36.863+0000 7f73234f0700 1 -- 172.21.15.46:0/2391619456 wait complete. 2024-03-31T23:09:37.512 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:09:37 smithi046 ceph-mon[32498]: from='client.? 172.21.15.46:0/2391619456' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-31T23:09:37.512 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:09:37 smithi046 ceph-mon[32498]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-31T23:09:37.512 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:09:37 smithi046 ceph-mon[32498]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-31T23:09:37.539 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-03-31T23:09:37.539 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-03-31T23:09:37.540 DEBUG:teuthology.orchestra.run.smithi046:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-03-31T23:09:37.584 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-03-31T23:09:37.584 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-03-31T23:09:37.584 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:384f31f3d3aee7f8ba3f60846b8844c049165583 shell --fsid 6a9d8f08-efb2-11ee-b647-cb9ed24678a4 -- ceph mgr dump --format=json 2024-03-31T23:09:37.693 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:09:37 smithi142 ceph-mon[35271]: from='client.? 172.21.15.46:0/2391619456' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-31T23:09:37.693 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:09:37 smithi142 ceph-mon[35271]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-31T23:09:37.693 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:09:37 smithi142 ceph-mon[35271]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-31T23:09:37.795 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:09:37 smithi162 ceph-mon[36237]: from='client.? 172.21.15.46:0/2391619456' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-31T23:09:37.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:09:37 smithi162 ceph-mon[36237]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-31T23:09:37.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:09:37 smithi162 ceph-mon[36237]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-31T23:09:37.875 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/6a9d8f08-efb2-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-31T23:09:38.693 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:09:38 smithi142 ceph-mon[35271]: pgmap v305: 129 pgs: 129 active+clean; 620 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 6.2 KiB/s wr, 12 op/s 2024-03-31T23:09:38.745 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:09:38 smithi046 ceph-mon[32498]: pgmap v305: 129 pgs: 129 active+clean; 620 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 6.2 KiB/s wr, 12 op/s 2024-03-31T23:09:38.795 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:09:38 smithi162 ceph-mon[36237]: pgmap v305: 129 pgs: 129 active+clean; 620 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 6.2 KiB/s wr, 12 op/s 2024-03-31T23:09:39.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:39.593+0000 7f24ab091700 1 -- 172.21.15.46:0/554218793 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f24a4101d80 msgr2=0x7f24a4105d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:09:39.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:39.593+0000 7f24ab091700 1 --2- 172.21.15.46:0/554218793 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f24a4101d80 0x7f24a4105d40 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f24980099c0 tx=0x7f2498009cd0 comp rx=0 tx=0).stop 2024-03-31T23:09:39.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:39.593+0000 7f24ab091700 1 -- 172.21.15.46:0/554218793 shutdown_connections 2024-03-31T23:09:39.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:39.593+0000 7f24ab091700 1 --2- 172.21.15.46:0/554218793 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f24a4106420 0x7f24a410dff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:39.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:39.593+0000 7f24ab091700 1 --2- 172.21.15.46:0/554218793 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f24a41013d0 0x7f24a41017b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:39.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:39.593+0000 7f24ab091700 1 --2- 172.21.15.46:0/554218793 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f24a4101d80 0x7f24a4105d40 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:39.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:39.593+0000 7f24ab091700 1 -- 172.21.15.46:0/554218793 >> 172.21.15.46:0/554218793 conn(0x7f24a4077190 msgr2=0x7f24a40775a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:09:39.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:39.594+0000 7f24ab091700 1 -- 172.21.15.46:0/554218793 shutdown_connections 2024-03-31T23:09:39.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:39.594+0000 7f24ab091700 1 -- 172.21.15.46:0/554218793 wait complete. 2024-03-31T23:09:39.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:39.594+0000 7f24ab091700 1 Processor -- start 2024-03-31T23:09:39.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:39.594+0000 7f24ab091700 1 -- start start 2024-03-31T23:09:39.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:39.594+0000 7f24ab091700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f24a41013d0 0x7f24a41a0780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:09:39.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:39.594+0000 7f24ab091700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f24a4101d80 0x7f24a41a0cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:09:39.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:39.594+0000 7f24ab091700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f24a4106420 0x7f24a419a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:09:39.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:39.594+0000 7f24ab091700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f24a41112b0 con 0x7f24a4101d80 2024-03-31T23:09:39.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:39.594+0000 7f24ab091700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f24a4111130 con 0x7f24a4106420 2024-03-31T23:09:39.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:39.594+0000 7f24ab091700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f24a4111430 con 0x7f24a41013d0 2024-03-31T23:09:39.598 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:39.595+0000 7f24a3fff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f24a4101d80 0x7f24a41a0cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:09:39.598 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:39.595+0000 7f24a3fff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f24a4101d80 0x7f24a41a0cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:57152/0 (socket says 172.21.15.46:57152) 2024-03-31T23:09:39.598 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:39.595+0000 7f24a3fff700 1 -- 172.21.15.46:0/3974690568 learned_addr learned my addr 172.21.15.46:0/3974690568 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:09:39.598 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:39.595+0000 7f24a8e2d700 1 --2- 172.21.15.46:0/3974690568 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f24a41013d0 0x7f24a41a0780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:09:39.598 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:39.595+0000 7f24a962e700 1 --2- 172.21.15.46:0/3974690568 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f24a4106420 0x7f24a419a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:09:39.598 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:39.595+0000 7f24a3fff700 1 -- 172.21.15.46:0/3974690568 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f24a41013d0 msgr2=0x7f24a41a0780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:09:39.598 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:39.595+0000 7f24a3fff700 1 --2- 172.21.15.46:0/3974690568 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f24a41013d0 0x7f24a41a0780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:39.598 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:39.595+0000 7f24a3fff700 1 -- 172.21.15.46:0/3974690568 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f24a4106420 msgr2=0x7f24a419a940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:09:39.598 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:39.595+0000 7f24a3fff700 1 --2- 172.21.15.46:0/3974690568 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f24a4106420 0x7f24a419a940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:39.598 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:39.595+0000 7f24a3fff700 1 -- 172.21.15.46:0/3974690568 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f24a419b0a0 con 0x7f24a4101d80 2024-03-31T23:09:39.598 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:39.595+0000 7f24a962e700 1 --2- 172.21.15.46:0/3974690568 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f24a4106420 0x7f24a419a940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T23:09:39.598 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:39.595+0000 7f24a3fff700 1 --2- 172.21.15.46:0/3974690568 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f24a4101d80 0x7f24a41a0cc0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f2498005f50 tx=0x7f2498004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:09:39.598 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:39.595+0000 7f24a1ffb700 1 -- 172.21.15.46:0/3974690568 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2498026070 con 0x7f24a4101d80 2024-03-31T23:09:39.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:39.595+0000 7f24a1ffb700 1 -- 172.21.15.46:0/3974690568 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f2498003c70 con 0x7f24a4101d80 2024-03-31T23:09:39.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:39.595+0000 7f24ab091700 1 -- 172.21.15.46:0/3974690568 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f24a419b330 con 0x7f24a4101d80 2024-03-31T23:09:39.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:39.595+0000 7f24a1ffb700 1 -- 172.21.15.46:0/3974690568 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f249800fad0 con 0x7f24a4101d80 2024-03-31T23:09:39.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:39.595+0000 7f24ab091700 1 -- 172.21.15.46:0/3974690568 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f24a41a7650 con 0x7f24a4101d80 2024-03-31T23:09:39.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:39.597+0000 7f24ab091700 1 -- 172.21.15.46:0/3974690568 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f24a4068f00 con 0x7f24a4101d80 2024-03-31T23:09:39.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:39.598+0000 7f24a1ffb700 1 -- 172.21.15.46:0/3974690568 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 17) ==== 137371+0+0 (secure 0 0 0) 0x7f2498029070 con 0x7f24a4101d80 2024-03-31T23:09:39.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:39.598+0000 7f24a1ffb700 1 --2- 172.21.15.46:0/3974690568 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f248c0a2640 0x7f248c0a4b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:09:39.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:39.598+0000 7f24a8e2d700 1 --2- 172.21.15.46:0/3974690568 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f248c0a2640 0x7f248c0a4b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:09:39.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:39.599+0000 7f24a1ffb700 1 -- 172.21.15.46:0/3974690568 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8802+0+0 (secure 0 0 0) 0x7f24980cf250 con 0x7f24a4101d80 2024-03-31T23:09:39.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:39.599+0000 7f24a8e2d700 1 --2- 172.21.15.46:0/3974690568 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f248c0a2640 0x7f248c0a4b00 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f24a419ba80 tx=0x7f2494009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:09:39.604 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:39.603+0000 7f24a1ffb700 1 -- 172.21.15.46:0/3974690568 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f249805d280 con 0x7f24a4101d80 2024-03-31T23:09:39.867 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:39.866+0000 7f24ab091700 1 -- 172.21.15.46:0/3974690568 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) -- 0x7f24a402db30 con 0x7f24a4101d80 2024-03-31T23:09:39.872 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:39.871+0000 7f24a1ffb700 1 -- 172.21.15.46:0/3974690568 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v17) ==== 74+0+262355 (secure 0 0 0) 0x7f2498013070 con 0x7f24a4101d80 2024-03-31T23:09:39.872 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-31T23:09:39.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:39.874+0000 7f24ab091700 1 -- 172.21.15.46:0/3974690568 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f248c0a2640 msgr2=0x7f248c0a4b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:09:39.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:39.874+0000 7f24ab091700 1 --2- 172.21.15.46:0/3974690568 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f248c0a2640 0x7f248c0a4b00 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f24a419ba80 tx=0x7f2494009260 comp rx=0 tx=0).stop 2024-03-31T23:09:39.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:39.874+0000 7f24ab091700 1 -- 172.21.15.46:0/3974690568 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f24a4101d80 msgr2=0x7f24a41a0cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:09:39.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:39.874+0000 7f24ab091700 1 --2- 172.21.15.46:0/3974690568 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f24a4101d80 0x7f24a41a0cc0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f2498005f50 tx=0x7f2498004270 comp rx=0 tx=0).stop 2024-03-31T23:09:39.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:39.874+0000 7f24ab091700 1 -- 172.21.15.46:0/3974690568 shutdown_connections 2024-03-31T23:09:39.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:39.874+0000 7f24ab091700 1 --2- 172.21.15.46:0/3974690568 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f248c0a2640 0x7f248c0a4b00 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:39.884 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:39.874+0000 7f24ab091700 1 --2- 172.21.15.46:0/3974690568 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f24a4106420 0x7f24a419a940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:39.884 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:39.874+0000 7f24ab091700 1 --2- 172.21.15.46:0/3974690568 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f24a41013d0 0x7f24a41a0780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:39.884 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:39.874+0000 7f24ab091700 1 --2- 172.21.15.46:0/3974690568 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f24a4101d80 0x7f24a41a0cc0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:39.884 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:39.874+0000 7f24ab091700 1 -- 172.21.15.46:0/3974690568 >> 172.21.15.46:0/3974690568 conn(0x7f24a4077190 msgr2=0x7f24a40ffd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:09:39.884 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:39.874+0000 7f24ab091700 1 -- 172.21.15.46:0/3974690568 shutdown_connections 2024-03-31T23:09:39.884 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:39.874+0000 7f24ab091700 1 -- 172.21.15.46:0/3974690568 wait complete. 2024-03-31T23:09:40.518 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:09:40 smithi046 ceph-mon[32498]: pgmap v306: 129 pgs: 129 active+clean; 621 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 6.1 KiB/s wr, 13 op/s 2024-03-31T23:09:40.518 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:09:40 smithi046 ceph-mon[32498]: from='client.? 172.21.15.46:0/3974690568' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-31T23:09:40.544 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":17,"flags":0,"active_gid":14152,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":162289569},{"type":"v1","addr":"172.21.15.46:6801","nonce":162289569}]},"active_addr":"172.21.15.46:6801/162289569","active_change":"2024-03-31T23:02:08.493729+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":14226,"name":"y","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]},{"gid":24109,"name":"z","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.46:8443/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":3,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.46:0","nonce":1146542737}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.46:0","nonce":3306820105}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.46:0","nonce":34745898}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.46:0","nonce":249968062}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.46:0","nonce":4188763190}]}]} 2024-03-31T23:09:40.549 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-03-31T23:09:40.549 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-03-31T23:09:40.549 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:384f31f3d3aee7f8ba3f60846b8844c049165583 shell --fsid 6a9d8f08-efb2-11ee-b647-cb9ed24678a4 -- ceph osd dump --format=json 2024-03-31T23:09:40.693 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:09:40 smithi142 ceph-mon[35271]: pgmap v306: 129 pgs: 129 active+clean; 621 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 6.1 KiB/s wr, 13 op/s 2024-03-31T23:09:40.693 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:09:40 smithi142 ceph-mon[35271]: from='client.? 172.21.15.46:0/3974690568' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-31T23:09:40.795 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:09:40 smithi162 ceph-mon[36237]: pgmap v306: 129 pgs: 129 active+clean; 621 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 6.1 KiB/s wr, 13 op/s 2024-03-31T23:09:40.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:09:40 smithi162 ceph-mon[36237]: from='client.? 172.21.15.46:0/3974690568' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-31T23:09:40.796 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/6a9d8f08-efb2-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-31T23:09:42.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:42.670+0000 7f84b153c700 1 -- 172.21.15.46:0/2296326417 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f84ac101780 msgr2=0x7f84ac106ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:09:42.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:42.670+0000 7f84b153c700 1 --2- 172.21.15.46:0/2296326417 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f84ac101780 0x7f84ac106ec0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f8494009990 tx=0x7f8494009ca0 comp rx=0 tx=0).stop 2024-03-31T23:09:42.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:42.670+0000 7f84b153c700 1 -- 172.21.15.46:0/2296326417 shutdown_connections 2024-03-31T23:09:42.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:42.670+0000 7f84b153c700 1 --2- 172.21.15.46:0/2296326417 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f84ac1075f0 0x7f84ac10f170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:42.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:42.670+0000 7f84b153c700 1 --2- 172.21.15.46:0/2296326417 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f84ac100dd0 0x7f84ac1011b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:42.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:42.670+0000 7f84b153c700 1 --2- 172.21.15.46:0/2296326417 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f84ac101780 0x7f84ac106ec0 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:42.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:42.670+0000 7f84b153c700 1 -- 172.21.15.46:0/2296326417 >> 172.21.15.46:0/2296326417 conn(0x7f84ac076470 msgr2=0x7f84ac076880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:09:42.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:42.670+0000 7f84b153c700 1 -- 172.21.15.46:0/2296326417 shutdown_connections 2024-03-31T23:09:42.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:42.670+0000 7f84b153c700 1 -- 172.21.15.46:0/2296326417 wait complete. 2024-03-31T23:09:42.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:42.671+0000 7f84b153c700 1 Processor -- start 2024-03-31T23:09:42.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:42.671+0000 7f84b153c700 1 -- start start 2024-03-31T23:09:42.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:42.671+0000 7f84b153c700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f84ac100dd0 0x7f84ac101d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:09:42.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:42.671+0000 7f84b153c700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f84ac101780 0x7f84ac102290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:09:42.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:42.671+0000 7f84b153c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f84ac1075f0 0x7f84ac1027d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:09:42.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:42.671+0000 7f84b153c700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f84ac1123f0 con 0x7f84ac1075f0 2024-03-31T23:09:42.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:42.671+0000 7f84b153c700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f84ac112270 con 0x7f84ac100dd0 2024-03-31T23:09:42.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:42.671+0000 7f84b153c700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f84ac112570 con 0x7f84ac101780 2024-03-31T23:09:42.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:42.671+0000 7f84a3fff700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f84ac101780 0x7f84ac102290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:09:42.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:42.671+0000 7f84ab7fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f84ac1075f0 0x7f84ac1027d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:09:42.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:42.671+0000 7f84aaffd700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f84ac100dd0 0x7f84ac101d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:09:42.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:42.671+0000 7f84a3fff700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f84ac101780 0x7f84ac102290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.46:43158/0 (socket says 172.21.15.46:43158) 2024-03-31T23:09:42.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:42.671+0000 7f84a3fff700 1 -- 172.21.15.46:0/4065216548 learned_addr learned my addr 172.21.15.46:0/4065216548 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:09:42.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:42.671+0000 7f84ab7fe700 1 --2- 172.21.15.46:0/4065216548 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f84ac1075f0 0x7f84ac1027d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:45798/0 (socket says 172.21.15.46:45798) 2024-03-31T23:09:42.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:42.671+0000 7f84ab7fe700 1 -- 172.21.15.46:0/4065216548 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f84ac101780 msgr2=0x7f84ac102290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:09:42.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:42.671+0000 7f84ab7fe700 1 --2- 172.21.15.46:0/4065216548 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f84ac101780 0x7f84ac102290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:42.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:42.671+0000 7f84ab7fe700 1 -- 172.21.15.46:0/4065216548 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f84ac100dd0 msgr2=0x7f84ac101d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:09:42.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:42.671+0000 7f84ab7fe700 1 --2- 172.21.15.46:0/4065216548 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f84ac100dd0 0x7f84ac101d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:42.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:42.671+0000 7f84ab7fe700 1 -- 172.21.15.46:0/4065216548 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f84ac1a6f10 con 0x7f84ac1075f0 2024-03-31T23:09:42.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:42.671+0000 7f84a3fff700 1 --2- 172.21.15.46:0/4065216548 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f84ac101780 0x7f84ac102290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:09:42.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:42.671+0000 7f84aaffd700 1 --2- 172.21.15.46:0/4065216548 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f84ac100dd0 0x7f84ac101d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:09:42.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:42.672+0000 7f84ab7fe700 1 --2- 172.21.15.46:0/4065216548 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f84ac1075f0 0x7f84ac1027d0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f84a400efb0 tx=0x7f84a400c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:09:42.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:42.672+0000 7f84a8ff9700 1 -- 172.21.15.46:0/4065216548 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84a4003a00 con 0x7f84ac1075f0 2024-03-31T23:09:42.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:42.672+0000 7f84a8ff9700 1 -- 172.21.15.46:0/4065216548 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f84a400ce50 con 0x7f84ac1075f0 2024-03-31T23:09:42.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:42.672+0000 7f84b153c700 1 -- 172.21.15.46:0/4065216548 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f84ac1a7200 con 0x7f84ac1075f0 2024-03-31T23:09:42.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:42.672+0000 7f84a8ff9700 1 -- 172.21.15.46:0/4065216548 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84a40047c0 con 0x7f84ac1075f0 2024-03-31T23:09:42.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:42.672+0000 7f84b153c700 1 -- 172.21.15.46:0/4065216548 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f84ac1a76f0 con 0x7f84ac1075f0 2024-03-31T23:09:42.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:42.673+0000 7f84b153c700 1 -- 172.21.15.46:0/4065216548 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f84ac068f00 con 0x7f84ac1075f0 2024-03-31T23:09:42.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:42.674+0000 7f84a8ff9700 1 -- 172.21.15.46:0/4065216548 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 17) ==== 137371+0+0 (secure 0 0 0) 0x7f84a4012050 con 0x7f84ac1075f0 2024-03-31T23:09:42.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:42.674+0000 7f84a8ff9700 1 --2- 172.21.15.46:0/4065216548 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f84980a25c0 0x7f84980a4a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:09:42.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:42.675+0000 7f84aaffd700 1 --2- 172.21.15.46:0/4065216548 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f84980a25c0 0x7f84980a4a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:09:42.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:42.675+0000 7f84a8ff9700 1 -- 172.21.15.46:0/4065216548 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8802+0+0 (secure 0 0 0) 0x7f84a40ce300 con 0x7f84ac1075f0 2024-03-31T23:09:42.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:42.675+0000 7f84aaffd700 1 --2- 172.21.15.46:0/4065216548 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f84980a25c0 0x7f84980a4a80 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f849c011490 tx=0x7f849c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:09:42.680 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:42.679+0000 7f84a8ff9700 1 -- 172.21.15.46:0/4065216548 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f84a4022060 con 0x7f84ac1075f0 2024-03-31T23:09:42.693 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:09:42 smithi142 ceph-mon[35271]: pgmap v307: 129 pgs: 129 active+clean; 629 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 8.3 KiB/s rd, 7.2 KiB/s wr, 15 op/s 2024-03-31T23:09:42.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:09:42 smithi162 ceph-mon[36237]: pgmap v307: 129 pgs: 129 active+clean; 629 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 8.3 KiB/s rd, 7.2 KiB/s wr, 15 op/s 2024-03-31T23:09:42.862 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:09:42 smithi046 ceph-mon[32498]: pgmap v307: 129 pgs: 129 active+clean; 629 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 8.3 KiB/s rd, 7.2 KiB/s wr, 15 op/s 2024-03-31T23:09:42.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:42.882+0000 7f84b153c700 1 -- 172.21.15.46:0/4065216548 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f84ac103280 con 0x7f84ac1075f0 2024-03-31T23:09:42.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:42.884+0000 7f84a8ff9700 1 -- 172.21.15.46:0/4065216548 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) ==== 74+0+24105 (secure 0 0 0) 0x7f84a4010040 con 0x7f84ac1075f0 2024-03-31T23:09:42.885 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-31T23:09:42.885 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":91,"fsid":"6a9d8f08-efb2-11ee-b647-cb9ed24678a4","created":"2024-03-31T23:01:15.463711+0000","modified":"2024-03-31T23:09:17.181787+0000","last_up_change":"2024-03-31T23:08:20.199888+0000","last_in_change":"2024-03-31T23:07:59.438035+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-31T23:04:32.823751+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-31T23:09:09.145711+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-31T23:09:10.148579+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-31T23:09:13.419519+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"91","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"47429095-3f46-4ed7-b7c4-504464719815","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":1824772695},{"type":"v1","addr":"172.21.15.46:6803","nonce":1824772695}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":1824772695},{"type":"v1","addr":"172.21.15.46:6805","nonce":1824772695}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":1824772695},{"type":"v1","addr":"172.21.15.46:6809","nonce":1824772695}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":1824772695},{"type":"v1","addr":"172.21.15.46:6807","nonce":1824772695}]},"public_addr":"172.21.15.46:6803/1824772695","cluster_addr":"172.21.15.46:6805/1824772695","heartbeat_back_addr":"172.21.15.46:6809/1824772695","heartbeat_front_addr":"172.21.15.46:6807/1824772695","state":["exists","up"]},{"osd":1,"uuid":"be56f883-411e-4dbe-b50e-a158134df4a6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6800","nonce":973693922},{"type":"v1","addr":"172.21.15.142:6801","nonce":973693922}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6802","nonce":973693922},{"type":"v1","addr":"172.21.15.142:6803","nonce":973693922}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6806","nonce":973693922},{"type":"v1","addr":"172.21.15.142:6807","nonce":973693922}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6804","nonce":973693922},{"type":"v1","addr":"172.21.15.142:6805","nonce":973693922}]},"public_addr":"172.21.15.142:6801/973693922","cluster_addr":"172.21.15.142:6803/973693922","heartbeat_back_addr":"172.21.15.142:6807/973693922","heartbeat_front_addr":"172.21.15.142:6805/973693922","state":["exists","up"]},{"osd":2,"uuid":"c99a8794-ab19-4260-9a8d-4bc0bd2c8175","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6800","nonce":4084261813},{"type":"v1","addr":"172.21.15.162:6801","nonce":4084261813}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6802","nonce":4084261813},{"type":"v1","addr":"172.21.15.162:6803","nonce":4084261813}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6806","nonce":4084261813},{"type":"v1","addr":"172.21.15.162:6807","nonce":4084261813}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6804","nonce":4084261813},{"type":"v1","addr":"172.21.15.162:6805","nonce":4084261813}]},"public_addr":"172.21.15.162:6801/4084261813","cluster_addr":"172.21.15.162:6803/4084261813","heartbeat_back_addr":"172.21.15.162:6807/4084261813","heartbeat_front_addr":"172.21.15.162:6805/4084261813","state":["exists","up"]},{"osd":3,"uuid":"f877814e-9ef6-4872-a081-5609bb9ebc06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":3496317970},{"type":"v1","addr":"172.21.15.46:6811","nonce":3496317970}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":3496317970},{"type":"v1","addr":"172.21.15.46:6813","nonce":3496317970}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":3496317970},{"type":"v1","addr":"172.21.15.46:6817","nonce":3496317970}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":3496317970},{"type":"v1","addr":"172.21.15.46:6815","nonce":3496317970}]},"public_addr":"172.21.15.46:6811/3496317970","cluster_addr":"172.21.15.46:6813/3496317970","heartbeat_back_addr":"172.21.15.46:6817/3496317970","heartbeat_front_addr":"172.21.15.46:6815/3496317970","state":["exists","up"]},{"osd":4,"uuid":"beecec0a-8280-44fe-8f5f-8498a35eca8f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6808","nonce":2996266929},{"type":"v1","addr":"172.21.15.142:6809","nonce":2996266929}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6810","nonce":2996266929},{"type":"v1","addr":"172.21.15.142:6811","nonce":2996266929}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6814","nonce":2996266929},{"type":"v1","addr":"172.21.15.142:6815","nonce":2996266929}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6812","nonce":2996266929},{"type":"v1","addr":"172.21.15.142:6813","nonce":2996266929}]},"public_addr":"172.21.15.142:6809/2996266929","cluster_addr":"172.21.15.142:6811/2996266929","heartbeat_back_addr":"172.21.15.142:6815/2996266929","heartbeat_front_addr":"172.21.15.142:6813/2996266929","state":["exists","up"]},{"osd":5,"uuid":"9da7ff75-f658-45ea-a02f-1943fdda76b9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6808","nonce":914208293},{"type":"v1","addr":"172.21.15.162:6809","nonce":914208293}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6810","nonce":914208293},{"type":"v1","addr":"172.21.15.162:6811","nonce":914208293}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6814","nonce":914208293},{"type":"v1","addr":"172.21.15.162:6815","nonce":914208293}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6812","nonce":914208293},{"type":"v1","addr":"172.21.15.162:6813","nonce":914208293}]},"public_addr":"172.21.15.162:6809/914208293","cluster_addr":"172.21.15.162:6811/914208293","heartbeat_back_addr":"172.21.15.162:6815/914208293","heartbeat_front_addr":"172.21.15.162:6813/914208293","state":["exists","up"]},{"osd":6,"uuid":"8fa11224-cb26-4d3f-87d1-6a8f31de2a19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":56291966},{"type":"v1","addr":"172.21.15.46:6819","nonce":56291966}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":56291966},{"type":"v1","addr":"172.21.15.46:6821","nonce":56291966}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6824","nonce":56291966},{"type":"v1","addr":"172.21.15.46:6825","nonce":56291966}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":56291966},{"type":"v1","addr":"172.21.15.46:6823","nonce":56291966}]},"public_addr":"172.21.15.46:6819/56291966","cluster_addr":"172.21.15.46:6821/56291966","heartbeat_back_addr":"172.21.15.46:6825/56291966","heartbeat_front_addr":"172.21.15.46:6823/56291966","state":["exists","up"]},{"osd":7,"uuid":"0d0338b7-4d0a-4bb0-b152-8f5224871d9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6816","nonce":2146598446},{"type":"v1","addr":"172.21.15.142:6817","nonce":2146598446}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6818","nonce":2146598446},{"type":"v1","addr":"172.21.15.142:6819","nonce":2146598446}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6822","nonce":2146598446},{"type":"v1","addr":"172.21.15.142:6823","nonce":2146598446}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6820","nonce":2146598446},{"type":"v1","addr":"172.21.15.142:6821","nonce":2146598446}]},"public_addr":"172.21.15.142:6817/2146598446","cluster_addr":"172.21.15.142:6819/2146598446","heartbeat_back_addr":"172.21.15.142:6823/2146598446","heartbeat_front_addr":"172.21.15.142:6821/2146598446","state":["exists","up"]},{"osd":8,"uuid":"8b6f6037-892a-416c-babb-7552648d0b3a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6816","nonce":1173078318},{"type":"v1","addr":"172.21.15.162:6817","nonce":1173078318}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6818","nonce":1173078318},{"type":"v1","addr":"172.21.15.162:6819","nonce":1173078318}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6822","nonce":1173078318},{"type":"v1","addr":"172.21.15.162:6823","nonce":1173078318}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6820","nonce":1173078318},{"type":"v1","addr":"172.21.15.162:6821","nonce":1173078318}]},"public_addr":"172.21.15.162:6817/1173078318","cluster_addr":"172.21.15.162:6819/1173078318","heartbeat_back_addr":"172.21.15.162:6823/1173078318","heartbeat_front_addr":"172.21.15.162:6821/1173078318","state":["exists","up"]},{"osd":9,"uuid":"faccce25-6bf2-4220-8858-cda82b72be68","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":532708881},{"type":"v1","addr":"172.21.15.46:6827","nonce":532708881}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6828","nonce":532708881},{"type":"v1","addr":"172.21.15.46:6829","nonce":532708881}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":532708881},{"type":"v1","addr":"172.21.15.46:6833","nonce":532708881}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":532708881},{"type":"v1","addr":"172.21.15.46:6831","nonce":532708881}]},"public_addr":"172.21.15.46:6827/532708881","cluster_addr":"172.21.15.46:6829/532708881","heartbeat_back_addr":"172.21.15.46:6833/532708881","heartbeat_front_addr":"172.21.15.46:6831/532708881","state":["exists","up"]},{"osd":10,"uuid":"cdd356db-f9eb-4080-95ab-db5d8223e572","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6824","nonce":2845222366},{"type":"v1","addr":"172.21.15.142:6825","nonce":2845222366}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6826","nonce":2845222366},{"type":"v1","addr":"172.21.15.142:6827","nonce":2845222366}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6830","nonce":2845222366},{"type":"v1","addr":"172.21.15.142:6831","nonce":2845222366}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6828","nonce":2845222366},{"type":"v1","addr":"172.21.15.142:6829","nonce":2845222366}]},"public_addr":"172.21.15.142:6825/2845222366","cluster_addr":"172.21.15.142:6827/2845222366","heartbeat_back_addr":"172.21.15.142:6831/2845222366","heartbeat_front_addr":"172.21.15.142:6829/2845222366","state":["exists","up"]},{"osd":11,"uuid":"72104d1c-3ea2-45ad-bc0b-0500f146c06b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6824","nonce":1986323778},{"type":"v1","addr":"172.21.15.162:6825","nonce":1986323778}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6826","nonce":1986323778},{"type":"v1","addr":"172.21.15.162:6827","nonce":1986323778}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6830","nonce":1986323778},{"type":"v1","addr":"172.21.15.162:6831","nonce":1986323778}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6828","nonce":1986323778},{"type":"v1","addr":"172.21.15.162:6829","nonce":1986323778}]},"public_addr":"172.21.15.162:6825/1986323778","cluster_addr":"172.21.15.162:6827/1986323778","heartbeat_back_addr":"172.21.15.162:6831/1986323778","heartbeat_front_addr":"172.21.15.162:6829/1986323778","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:03:44.996331+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:04:07.994738+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:04:30.942255+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:04:55.588884+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:05:19.900600+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:05:43.807442+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:06:09.026976+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:06:34.586784+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:06:59.751605+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:07:26.144495+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:07:52.326909+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:08:17.423042+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.46:0/4021039857":"2024-04-01T23:02:08.493619+0000","172.21.15.46:6801/3681182647":"2024-04-01T23:02:08.493619+0000","172.21.15.46:0/4022214806":"2024-04-01T23:02:08.493619+0000","172.21.15.46:6800/1515528215":"2024-04-01T23:01:41.910162+0000","172.21.15.46:6800/3681182647":"2024-04-01T23:02:08.493619+0000","172.21.15.46:0/143885707":"2024-04-01T23:01:41.910162+0000","172.21.15.46:6801/1515528215":"2024-04-01T23:01:41.910162+0000","172.21.15.46:0/1680389778":"2024-04-01T23:01:41.910162+0000","172.21.15.46:0/1782133966":"2024-04-01T23:01:41.910162+0000","172.21.15.46:0/3925209376":"2024-04-01T23:02:08.493619+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-31T23:09:42.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:42.886+0000 7f84b153c700 1 -- 172.21.15.46:0/4065216548 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f84980a25c0 msgr2=0x7f84980a4a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:09:42.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:42.886+0000 7f84b153c700 1 --2- 172.21.15.46:0/4065216548 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f84980a25c0 0x7f84980a4a80 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f849c011490 tx=0x7f849c009260 comp rx=0 tx=0).stop 2024-03-31T23:09:42.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:42.886+0000 7f84b153c700 1 -- 172.21.15.46:0/4065216548 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f84ac1075f0 msgr2=0x7f84ac1027d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:09:42.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:42.886+0000 7f84b153c700 1 --2- 172.21.15.46:0/4065216548 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f84ac1075f0 0x7f84ac1027d0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f84a400efb0 tx=0x7f84a400c3a0 comp rx=0 tx=0).stop 2024-03-31T23:09:42.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:42.886+0000 7f84b153c700 1 -- 172.21.15.46:0/4065216548 shutdown_connections 2024-03-31T23:09:42.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:42.886+0000 7f84b153c700 1 --2- 172.21.15.46:0/4065216548 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f84980a25c0 0x7f84980a4a80 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:42.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:42.886+0000 7f84b153c700 1 --2- 172.21.15.46:0/4065216548 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f84ac1075f0 0x7f84ac1027d0 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:42.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:42.886+0000 7f84b153c700 1 --2- 172.21.15.46:0/4065216548 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f84ac100dd0 0x7f84ac101d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:42.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:42.886+0000 7f84b153c700 1 --2- 172.21.15.46:0/4065216548 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f84ac101780 0x7f84ac102290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:42.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:42.886+0000 7f84b153c700 1 -- 172.21.15.46:0/4065216548 >> 172.21.15.46:0/4065216548 conn(0x7f84ac076470 msgr2=0x7f84ac0fef70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:09:42.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:42.886+0000 7f84b153c700 1 -- 172.21.15.46:0/4065216548 shutdown_connections 2024-03-31T23:09:42.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:42.886+0000 7f84b153c700 1 -- 172.21.15.46:0/4065216548 wait complete. 2024-03-31T23:09:43.611 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:09:43 smithi046 ceph-mon[32498]: from='client.? 172.21.15.46:0/4065216548' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-31T23:09:43.693 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:09:43 smithi142 ceph-mon[35271]: from='client.? 172.21.15.46:0/4065216548' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-31T23:09:43.795 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:09:43 smithi162 ceph-mon[36237]: from='client.? 172.21.15.46:0/4065216548' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-31T23:09:43.817 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-03-31T23:09:43.817 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:384f31f3d3aee7f8ba3f60846b8844c049165583 shell --fsid 6a9d8f08-efb2-11ee-b647-cb9ed24678a4 -- ceph osd dump --format=json 2024-03-31T23:09:44.064 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/6a9d8f08-efb2-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-31T23:09:44.795 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:09:44 smithi162 ceph-mon[36237]: pgmap v308: 129 pgs: 129 active+clean; 629 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 8.3 KiB/s rd, 6.9 KiB/s wr, 15 op/s 2024-03-31T23:09:44.862 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:09:44 smithi046 ceph-mon[32498]: pgmap v308: 129 pgs: 129 active+clean; 629 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 8.3 KiB/s rd, 6.9 KiB/s wr, 15 op/s 2024-03-31T23:09:44.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:09:44 smithi142 ceph-mon[35271]: pgmap v308: 129 pgs: 129 active+clean; 629 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 8.3 KiB/s rd, 6.9 KiB/s wr, 15 op/s 2024-03-31T23:09:45.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:45.905+0000 7f1b6c40e700 1 -- 172.21.15.46:0/3836708052 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1b6406a940 msgr2=0x7f1b6406ada0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:09:45.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:45.905+0000 7f1b6c40e700 1 --2- 172.21.15.46:0/3836708052 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1b6406a940 0x7f1b6406ada0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f1b600099c0 tx=0x7f1b60009cd0 comp rx=0 tx=0).stop 2024-03-31T23:09:45.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:45.905+0000 7f1b6c40e700 1 -- 172.21.15.46:0/3836708052 shutdown_connections 2024-03-31T23:09:45.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:45.905+0000 7f1b6c40e700 1 --2- 172.21.15.46:0/3836708052 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1b6406b2e0 0x7f1b6410f100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:45.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:45.905+0000 7f1b6c40e700 1 --2- 172.21.15.46:0/3836708052 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1b641082c0 0x7f1b641086a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:45.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:45.905+0000 7f1b6c40e700 1 --2- 172.21.15.46:0/3836708052 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1b6406a940 0x7f1b6406ada0 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:45.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:45.905+0000 7f1b6c40e700 1 -- 172.21.15.46:0/3836708052 >> 172.21.15.46:0/3836708052 conn(0x7f1b64077160 msgr2=0x7f1b64077570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:09:45.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:45.905+0000 7f1b6c40e700 1 -- 172.21.15.46:0/3836708052 shutdown_connections 2024-03-31T23:09:45.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:45.905+0000 7f1b6c40e700 1 -- 172.21.15.46:0/3836708052 wait complete. 2024-03-31T23:09:45.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:45.905+0000 7f1b6c40e700 1 Processor -- start 2024-03-31T23:09:45.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:45.905+0000 7f1b6c40e700 1 -- start start 2024-03-31T23:09:45.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:45.906+0000 7f1b6c40e700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1b6406a940 0x7f1b641a0790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:09:45.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:45.906+0000 7f1b6c40e700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1b6406b2e0 0x7f1b641a0cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:09:45.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:45.906+0000 7f1b6c40e700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1b641082c0 0x7f1b6419a950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:09:45.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:45.906+0000 7f1b6c40e700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f1b641123f0 con 0x7f1b641082c0 2024-03-31T23:09:45.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:45.906+0000 7f1b6c40e700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f1b64112270 con 0x7f1b6406a940 2024-03-31T23:09:45.910 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:45.906+0000 7f1b6c40e700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f1b64112570 con 0x7f1b6406b2e0 2024-03-31T23:09:45.910 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:45.906+0000 7f1b6a9ab700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1b641082c0 0x7f1b6419a950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:09:45.910 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:45.906+0000 7f1b6a1aa700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1b6406a940 0x7f1b641a0790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:09:45.910 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:45.906+0000 7f1b6a9ab700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1b641082c0 0x7f1b6419a950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:45820/0 (socket says 172.21.15.46:45820) 2024-03-31T23:09:45.910 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:45.906+0000 7f1b6a9ab700 1 -- 172.21.15.46:0/4062383298 learned_addr learned my addr 172.21.15.46:0/4062383298 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:09:45.910 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:45.906+0000 7f1b699a9700 1 --2- 172.21.15.46:0/4062383298 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1b6406b2e0 0x7f1b641a0cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:09:45.910 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:45.906+0000 7f1b6a9ab700 1 -- 172.21.15.46:0/4062383298 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1b6406b2e0 msgr2=0x7f1b641a0cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:09:45.910 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:45.906+0000 7f1b6a9ab700 1 --2- 172.21.15.46:0/4062383298 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1b6406b2e0 0x7f1b641a0cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:45.910 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:45.906+0000 7f1b6a9ab700 1 -- 172.21.15.46:0/4062383298 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1b6406a940 msgr2=0x7f1b641a0790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:09:45.910 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:45.906+0000 7f1b6a9ab700 1 --2- 172.21.15.46:0/4062383298 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1b6406a940 0x7f1b641a0790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:45.910 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:45.906+0000 7f1b6a9ab700 1 -- 172.21.15.46:0/4062383298 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b6419b0b0 con 0x7f1b641082c0 2024-03-31T23:09:45.910 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:45.906+0000 7f1b6a1aa700 1 --2- 172.21.15.46:0/4062383298 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1b6406a940 0x7f1b641a0790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:09:45.911 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:45.906+0000 7f1b6a9ab700 1 --2- 172.21.15.46:0/4062383298 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1b641082c0 0x7f1b6419a950 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f1b5c00ef30 tx=0x7f1b5c00c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:09:45.911 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:45.906+0000 7f1b5b7fe700 1 -- 172.21.15.46:0/4062383298 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b5c0092e0 con 0x7f1b641082c0 2024-03-31T23:09:45.911 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:45.906+0000 7f1b5b7fe700 1 -- 172.21.15.46:0/4062383298 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1b5c004bb0 con 0x7f1b641082c0 2024-03-31T23:09:45.911 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:45.906+0000 7f1b6c40e700 1 -- 172.21.15.46:0/4062383298 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1b6419b340 con 0x7f1b641082c0 2024-03-31T23:09:45.911 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:45.907+0000 7f1b5b7fe700 1 -- 172.21.15.46:0/4062383298 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b5c004020 con 0x7f1b641082c0 2024-03-31T23:09:45.911 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:45.907+0000 7f1b6c40e700 1 -- 172.21.15.46:0/4062383298 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1b641a7660 con 0x7f1b641082c0 2024-03-31T23:09:45.912 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:45.908+0000 7f1b6c40e700 1 -- 172.21.15.46:0/4062383298 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1b64068f00 con 0x7f1b641082c0 2024-03-31T23:09:45.912 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:45.909+0000 7f1b5b7fe700 1 -- 172.21.15.46:0/4062383298 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 17) ==== 137371+0+0 (secure 0 0 0) 0x7f1b5c01b030 con 0x7f1b641082c0 2024-03-31T23:09:45.912 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:45.909+0000 7f1b5b7fe700 1 --2- 172.21.15.46:0/4062383298 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f1b500a2640 0x7f1b500a4b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:09:45.912 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:45.909+0000 7f1b6a1aa700 1 --2- 172.21.15.46:0/4062383298 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f1b500a2640 0x7f1b500a4b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:09:45.912 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:45.909+0000 7f1b5b7fe700 1 -- 172.21.15.46:0/4062383298 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8802+0+0 (secure 0 0 0) 0x7f1b5c0ce520 con 0x7f1b641082c0 2024-03-31T23:09:45.912 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:45.910+0000 7f1b6a1aa700 1 --2- 172.21.15.46:0/4062383298 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f1b500a2640 0x7f1b500a4b00 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f1b6419ba90 tx=0x7f1b54009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:09:45.915 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:45.914+0000 7f1b5b7fe700 1 -- 172.21.15.46:0/4062383298 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f1b5c022020 con 0x7f1b641082c0 2024-03-31T23:09:46.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:46.118+0000 7f1b6c40e700 1 -- 172.21.15.46:0/4062383298 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f1b641a7830 con 0x7f1b641082c0 2024-03-31T23:09:46.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:46.119+0000 7f1b5b7fe700 1 -- 172.21.15.46:0/4062383298 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) ==== 74+0+24105 (secure 0 0 0) 0x7f1b5c010040 con 0x7f1b641082c0 2024-03-31T23:09:46.120 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-31T23:09:46.121 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":91,"fsid":"6a9d8f08-efb2-11ee-b647-cb9ed24678a4","created":"2024-03-31T23:01:15.463711+0000","modified":"2024-03-31T23:09:17.181787+0000","last_up_change":"2024-03-31T23:08:20.199888+0000","last_in_change":"2024-03-31T23:07:59.438035+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-31T23:04:32.823751+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-31T23:09:09.145711+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-31T23:09:10.148579+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-31T23:09:13.419519+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"91","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"47429095-3f46-4ed7-b7c4-504464719815","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":1824772695},{"type":"v1","addr":"172.21.15.46:6803","nonce":1824772695}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":1824772695},{"type":"v1","addr":"172.21.15.46:6805","nonce":1824772695}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":1824772695},{"type":"v1","addr":"172.21.15.46:6809","nonce":1824772695}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":1824772695},{"type":"v1","addr":"172.21.15.46:6807","nonce":1824772695}]},"public_addr":"172.21.15.46:6803/1824772695","cluster_addr":"172.21.15.46:6805/1824772695","heartbeat_back_addr":"172.21.15.46:6809/1824772695","heartbeat_front_addr":"172.21.15.46:6807/1824772695","state":["exists","up"]},{"osd":1,"uuid":"be56f883-411e-4dbe-b50e-a158134df4a6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6800","nonce":973693922},{"type":"v1","addr":"172.21.15.142:6801","nonce":973693922}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6802","nonce":973693922},{"type":"v1","addr":"172.21.15.142:6803","nonce":973693922}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6806","nonce":973693922},{"type":"v1","addr":"172.21.15.142:6807","nonce":973693922}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6804","nonce":973693922},{"type":"v1","addr":"172.21.15.142:6805","nonce":973693922}]},"public_addr":"172.21.15.142:6801/973693922","cluster_addr":"172.21.15.142:6803/973693922","heartbeat_back_addr":"172.21.15.142:6807/973693922","heartbeat_front_addr":"172.21.15.142:6805/973693922","state":["exists","up"]},{"osd":2,"uuid":"c99a8794-ab19-4260-9a8d-4bc0bd2c8175","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6800","nonce":4084261813},{"type":"v1","addr":"172.21.15.162:6801","nonce":4084261813}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6802","nonce":4084261813},{"type":"v1","addr":"172.21.15.162:6803","nonce":4084261813}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6806","nonce":4084261813},{"type":"v1","addr":"172.21.15.162:6807","nonce":4084261813}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6804","nonce":4084261813},{"type":"v1","addr":"172.21.15.162:6805","nonce":4084261813}]},"public_addr":"172.21.15.162:6801/4084261813","cluster_addr":"172.21.15.162:6803/4084261813","heartbeat_back_addr":"172.21.15.162:6807/4084261813","heartbeat_front_addr":"172.21.15.162:6805/4084261813","state":["exists","up"]},{"osd":3,"uuid":"f877814e-9ef6-4872-a081-5609bb9ebc06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":3496317970},{"type":"v1","addr":"172.21.15.46:6811","nonce":3496317970}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":3496317970},{"type":"v1","addr":"172.21.15.46:6813","nonce":3496317970}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":3496317970},{"type":"v1","addr":"172.21.15.46:6817","nonce":3496317970}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":3496317970},{"type":"v1","addr":"172.21.15.46:6815","nonce":3496317970}]},"public_addr":"172.21.15.46:6811/3496317970","cluster_addr":"172.21.15.46:6813/3496317970","heartbeat_back_addr":"172.21.15.46:6817/3496317970","heartbeat_front_addr":"172.21.15.46:6815/3496317970","state":["exists","up"]},{"osd":4,"uuid":"beecec0a-8280-44fe-8f5f-8498a35eca8f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6808","nonce":2996266929},{"type":"v1","addr":"172.21.15.142:6809","nonce":2996266929}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6810","nonce":2996266929},{"type":"v1","addr":"172.21.15.142:6811","nonce":2996266929}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6814","nonce":2996266929},{"type":"v1","addr":"172.21.15.142:6815","nonce":2996266929}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6812","nonce":2996266929},{"type":"v1","addr":"172.21.15.142:6813","nonce":2996266929}]},"public_addr":"172.21.15.142:6809/2996266929","cluster_addr":"172.21.15.142:6811/2996266929","heartbeat_back_addr":"172.21.15.142:6815/2996266929","heartbeat_front_addr":"172.21.15.142:6813/2996266929","state":["exists","up"]},{"osd":5,"uuid":"9da7ff75-f658-45ea-a02f-1943fdda76b9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6808","nonce":914208293},{"type":"v1","addr":"172.21.15.162:6809","nonce":914208293}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6810","nonce":914208293},{"type":"v1","addr":"172.21.15.162:6811","nonce":914208293}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6814","nonce":914208293},{"type":"v1","addr":"172.21.15.162:6815","nonce":914208293}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6812","nonce":914208293},{"type":"v1","addr":"172.21.15.162:6813","nonce":914208293}]},"public_addr":"172.21.15.162:6809/914208293","cluster_addr":"172.21.15.162:6811/914208293","heartbeat_back_addr":"172.21.15.162:6815/914208293","heartbeat_front_addr":"172.21.15.162:6813/914208293","state":["exists","up"]},{"osd":6,"uuid":"8fa11224-cb26-4d3f-87d1-6a8f31de2a19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":56291966},{"type":"v1","addr":"172.21.15.46:6819","nonce":56291966}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":56291966},{"type":"v1","addr":"172.21.15.46:6821","nonce":56291966}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6824","nonce":56291966},{"type":"v1","addr":"172.21.15.46:6825","nonce":56291966}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":56291966},{"type":"v1","addr":"172.21.15.46:6823","nonce":56291966}]},"public_addr":"172.21.15.46:6819/56291966","cluster_addr":"172.21.15.46:6821/56291966","heartbeat_back_addr":"172.21.15.46:6825/56291966","heartbeat_front_addr":"172.21.15.46:6823/56291966","state":["exists","up"]},{"osd":7,"uuid":"0d0338b7-4d0a-4bb0-b152-8f5224871d9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6816","nonce":2146598446},{"type":"v1","addr":"172.21.15.142:6817","nonce":2146598446}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6818","nonce":2146598446},{"type":"v1","addr":"172.21.15.142:6819","nonce":2146598446}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6822","nonce":2146598446},{"type":"v1","addr":"172.21.15.142:6823","nonce":2146598446}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6820","nonce":2146598446},{"type":"v1","addr":"172.21.15.142:6821","nonce":2146598446}]},"public_addr":"172.21.15.142:6817/2146598446","cluster_addr":"172.21.15.142:6819/2146598446","heartbeat_back_addr":"172.21.15.142:6823/2146598446","heartbeat_front_addr":"172.21.15.142:6821/2146598446","state":["exists","up"]},{"osd":8,"uuid":"8b6f6037-892a-416c-babb-7552648d0b3a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6816","nonce":1173078318},{"type":"v1","addr":"172.21.15.162:6817","nonce":1173078318}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6818","nonce":1173078318},{"type":"v1","addr":"172.21.15.162:6819","nonce":1173078318}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6822","nonce":1173078318},{"type":"v1","addr":"172.21.15.162:6823","nonce":1173078318}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6820","nonce":1173078318},{"type":"v1","addr":"172.21.15.162:6821","nonce":1173078318}]},"public_addr":"172.21.15.162:6817/1173078318","cluster_addr":"172.21.15.162:6819/1173078318","heartbeat_back_addr":"172.21.15.162:6823/1173078318","heartbeat_front_addr":"172.21.15.162:6821/1173078318","state":["exists","up"]},{"osd":9,"uuid":"faccce25-6bf2-4220-8858-cda82b72be68","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":532708881},{"type":"v1","addr":"172.21.15.46:6827","nonce":532708881}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6828","nonce":532708881},{"type":"v1","addr":"172.21.15.46:6829","nonce":532708881}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":532708881},{"type":"v1","addr":"172.21.15.46:6833","nonce":532708881}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":532708881},{"type":"v1","addr":"172.21.15.46:6831","nonce":532708881}]},"public_addr":"172.21.15.46:6827/532708881","cluster_addr":"172.21.15.46:6829/532708881","heartbeat_back_addr":"172.21.15.46:6833/532708881","heartbeat_front_addr":"172.21.15.46:6831/532708881","state":["exists","up"]},{"osd":10,"uuid":"cdd356db-f9eb-4080-95ab-db5d8223e572","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6824","nonce":2845222366},{"type":"v1","addr":"172.21.15.142:6825","nonce":2845222366}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6826","nonce":2845222366},{"type":"v1","addr":"172.21.15.142:6827","nonce":2845222366}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6830","nonce":2845222366},{"type":"v1","addr":"172.21.15.142:6831","nonce":2845222366}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6828","nonce":2845222366},{"type":"v1","addr":"172.21.15.142:6829","nonce":2845222366}]},"public_addr":"172.21.15.142:6825/2845222366","cluster_addr":"172.21.15.142:6827/2845222366","heartbeat_back_addr":"172.21.15.142:6831/2845222366","heartbeat_front_addr":"172.21.15.142:6829/2845222366","state":["exists","up"]},{"osd":11,"uuid":"72104d1c-3ea2-45ad-bc0b-0500f146c06b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6824","nonce":1986323778},{"type":"v1","addr":"172.21.15.162:6825","nonce":1986323778}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6826","nonce":1986323778},{"type":"v1","addr":"172.21.15.162:6827","nonce":1986323778}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6830","nonce":1986323778},{"type":"v1","addr":"172.21.15.162:6831","nonce":1986323778}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6828","nonce":1986323778},{"type":"v1","addr":"172.21.15.162:6829","nonce":1986323778}]},"public_addr":"172.21.15.162:6825/1986323778","cluster_addr":"172.21.15.162:6827/1986323778","heartbeat_back_addr":"172.21.15.162:6831/1986323778","heartbeat_front_addr":"172.21.15.162:6829/1986323778","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:03:44.996331+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:04:07.994738+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:04:30.942255+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:04:55.588884+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:05:19.900600+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:05:43.807442+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:06:09.026976+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:06:34.586784+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:06:59.751605+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:07:26.144495+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:07:52.326909+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:08:17.423042+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.46:0/4021039857":"2024-04-01T23:02:08.493619+0000","172.21.15.46:6801/3681182647":"2024-04-01T23:02:08.493619+0000","172.21.15.46:0/4022214806":"2024-04-01T23:02:08.493619+0000","172.21.15.46:6800/1515528215":"2024-04-01T23:01:41.910162+0000","172.21.15.46:6800/3681182647":"2024-04-01T23:02:08.493619+0000","172.21.15.46:0/143885707":"2024-04-01T23:01:41.910162+0000","172.21.15.46:6801/1515528215":"2024-04-01T23:01:41.910162+0000","172.21.15.46:0/1680389778":"2024-04-01T23:01:41.910162+0000","172.21.15.46:0/1782133966":"2024-04-01T23:01:41.910162+0000","172.21.15.46:0/3925209376":"2024-04-01T23:02:08.493619+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-31T23:09:46.122 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:46.121+0000 7f1b6c40e700 1 -- 172.21.15.46:0/4062383298 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f1b500a2640 msgr2=0x7f1b500a4b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:09:46.122 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:46.121+0000 7f1b6c40e700 1 --2- 172.21.15.46:0/4062383298 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f1b500a2640 0x7f1b500a4b00 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f1b6419ba90 tx=0x7f1b54009260 comp rx=0 tx=0).stop 2024-03-31T23:09:46.122 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:46.121+0000 7f1b6c40e700 1 -- 172.21.15.46:0/4062383298 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1b641082c0 msgr2=0x7f1b6419a950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:09:46.123 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:46.121+0000 7f1b6c40e700 1 --2- 172.21.15.46:0/4062383298 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1b641082c0 0x7f1b6419a950 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f1b5c00ef30 tx=0x7f1b5c00c3a0 comp rx=0 tx=0).stop 2024-03-31T23:09:46.123 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:46.121+0000 7f1b6c40e700 1 -- 172.21.15.46:0/4062383298 shutdown_connections 2024-03-31T23:09:46.123 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:46.121+0000 7f1b6c40e700 1 --2- 172.21.15.46:0/4062383298 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f1b500a2640 0x7f1b500a4b00 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:46.123 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:46.121+0000 7f1b6c40e700 1 --2- 172.21.15.46:0/4062383298 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1b641082c0 0x7f1b6419a950 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:46.123 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:46.121+0000 7f1b6c40e700 1 --2- 172.21.15.46:0/4062383298 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1b6406a940 0x7f1b641a0790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:46.123 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:46.121+0000 7f1b6c40e700 1 --2- 172.21.15.46:0/4062383298 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1b6406b2e0 0x7f1b641a0cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:46.123 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:46.121+0000 7f1b6c40e700 1 -- 172.21.15.46:0/4062383298 >> 172.21.15.46:0/4062383298 conn(0x7f1b64077160 msgr2=0x7f1b6410d450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:09:46.123 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:46.121+0000 7f1b6c40e700 1 -- 172.21.15.46:0/4062383298 shutdown_connections 2024-03-31T23:09:46.123 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:46.121+0000 7f1b6c40e700 1 -- 172.21.15.46:0/4062383298 wait complete. 2024-03-31T23:09:46.795 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:09:46 smithi162 ceph-mon[36237]: pgmap v309: 129 pgs: 129 active+clean; 629 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 8.0 KiB/s rd, 6.6 KiB/s wr, 14 op/s 2024-03-31T23:09:46.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:09:46 smithi162 ceph-mon[36237]: from='client.? 172.21.15.46:0/4062383298' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-31T23:09:46.797 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:09:46 smithi046 ceph-mon[32498]: pgmap v309: 129 pgs: 129 active+clean; 629 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 8.0 KiB/s rd, 6.6 KiB/s wr, 14 op/s 2024-03-31T23:09:46.798 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:09:46 smithi046 ceph-mon[32498]: from='client.? 172.21.15.46:0/4062383298' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-31T23:09:46.833 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:384f31f3d3aee7f8ba3f60846b8844c049165583 shell --fsid 6a9d8f08-efb2-11ee-b647-cb9ed24678a4 -- ceph tell osd.0 flush_pg_stats 2024-03-31T23:09:46.833 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:384f31f3d3aee7f8ba3f60846b8844c049165583 shell --fsid 6a9d8f08-efb2-11ee-b647-cb9ed24678a4 -- ceph tell osd.1 flush_pg_stats 2024-03-31T23:09:46.834 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:384f31f3d3aee7f8ba3f60846b8844c049165583 shell --fsid 6a9d8f08-efb2-11ee-b647-cb9ed24678a4 -- ceph tell osd.2 flush_pg_stats 2024-03-31T23:09:46.834 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:384f31f3d3aee7f8ba3f60846b8844c049165583 shell --fsid 6a9d8f08-efb2-11ee-b647-cb9ed24678a4 -- ceph tell osd.3 flush_pg_stats 2024-03-31T23:09:46.834 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:384f31f3d3aee7f8ba3f60846b8844c049165583 shell --fsid 6a9d8f08-efb2-11ee-b647-cb9ed24678a4 -- ceph tell osd.4 flush_pg_stats 2024-03-31T23:09:46.835 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:384f31f3d3aee7f8ba3f60846b8844c049165583 shell --fsid 6a9d8f08-efb2-11ee-b647-cb9ed24678a4 -- ceph tell osd.5 flush_pg_stats 2024-03-31T23:09:46.835 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:384f31f3d3aee7f8ba3f60846b8844c049165583 shell --fsid 6a9d8f08-efb2-11ee-b647-cb9ed24678a4 -- ceph tell osd.6 flush_pg_stats 2024-03-31T23:09:46.836 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:384f31f3d3aee7f8ba3f60846b8844c049165583 shell --fsid 6a9d8f08-efb2-11ee-b647-cb9ed24678a4 -- ceph tell osd.7 flush_pg_stats 2024-03-31T23:09:46.836 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:384f31f3d3aee7f8ba3f60846b8844c049165583 shell --fsid 6a9d8f08-efb2-11ee-b647-cb9ed24678a4 -- ceph tell osd.8 flush_pg_stats 2024-03-31T23:09:46.836 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:384f31f3d3aee7f8ba3f60846b8844c049165583 shell --fsid 6a9d8f08-efb2-11ee-b647-cb9ed24678a4 -- ceph tell osd.9 flush_pg_stats 2024-03-31T23:09:46.837 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:384f31f3d3aee7f8ba3f60846b8844c049165583 shell --fsid 6a9d8f08-efb2-11ee-b647-cb9ed24678a4 -- ceph tell osd.10 flush_pg_stats 2024-03-31T23:09:46.837 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:384f31f3d3aee7f8ba3f60846b8844c049165583 shell --fsid 6a9d8f08-efb2-11ee-b647-cb9ed24678a4 -- ceph tell osd.11 flush_pg_stats 2024-03-31T23:09:46.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:09:46 smithi142 ceph-mon[35271]: pgmap v309: 129 pgs: 129 active+clean; 629 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 8.0 KiB/s rd, 6.6 KiB/s wr, 14 op/s 2024-03-31T23:09:46.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:09:46 smithi142 ceph-mon[35271]: from='client.? 172.21.15.46:0/4062383298' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-31T23:09:47.622 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/6a9d8f08-efb2-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-31T23:09:47.626 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/6a9d8f08-efb2-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-31T23:09:47.631 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/6a9d8f08-efb2-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-31T23:09:47.632 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/6a9d8f08-efb2-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-31T23:09:47.632 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/6a9d8f08-efb2-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-31T23:09:47.632 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/6a9d8f08-efb2-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-31T23:09:47.633 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/6a9d8f08-efb2-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-31T23:09:47.633 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/6a9d8f08-efb2-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-31T23:09:47.633 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/6a9d8f08-efb2-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-31T23:09:47.634 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/6a9d8f08-efb2-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-31T23:09:47.636 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/6a9d8f08-efb2-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-31T23:09:47.641 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/6a9d8f08-efb2-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-31T23:09:48.712 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:09:48 smithi046 ceph-mon[32498]: pgmap v310: 129 pgs: 129 active+clean; 638 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 5.6 KiB/s wr, 10 op/s 2024-03-31T23:09:48.745 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:09:48 smithi142 ceph-mon[35271]: pgmap v310: 129 pgs: 129 active+clean; 638 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 5.6 KiB/s wr, 10 op/s 2024-03-31T23:09:48.795 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:09:48 smithi162 ceph-mon[36237]: pgmap v310: 129 pgs: 129 active+clean; 638 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 5.6 KiB/s wr, 10 op/s 2024-03-31T23:09:50.735 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:09:50 smithi046 ceph-mon[32498]: pgmap v311: 129 pgs: 129 active+clean; 638 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 2.0 KiB/s wr, 7 op/s 2024-03-31T23:09:50.795 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:09:50 smithi162 ceph-mon[36237]: pgmap v311: 129 pgs: 129 active+clean; 638 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 2.0 KiB/s wr, 7 op/s 2024-03-31T23:09:50.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:09:50 smithi142 ceph-mon[35271]: pgmap v311: 129 pgs: 129 active+clean; 638 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 2.0 KiB/s wr, 7 op/s 2024-03-31T23:09:52.459 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:09:52 smithi046 ceph-mon[32498]: pgmap v312: 129 pgs: 129 active+clean; 640 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 2.2 KiB/s wr, 8 op/s 2024-03-31T23:09:52.795 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:09:52 smithi162 ceph-mon[36237]: pgmap v312: 129 pgs: 129 active+clean; 640 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 2.2 KiB/s wr, 8 op/s 2024-03-31T23:09:52.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:09:52 smithi142 ceph-mon[35271]: pgmap v312: 129 pgs: 129 active+clean; 640 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 2.2 KiB/s wr, 8 op/s 2024-03-31T23:09:54.611 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:09:54 smithi046 ceph-mon[32498]: pgmap v313: 129 pgs: 129 active+clean; 640 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 1.2 KiB/s wr, 6 op/s 2024-03-31T23:09:54.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:54.716+0000 7fbf23adc700 1 -- 172.21.15.46:0/2163879843 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbf1c100830 msgr2=0x7fbf1c100cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:09:54.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:54.716+0000 7fbf23adc700 1 --2- 172.21.15.46:0/2163879843 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbf1c100830 0x7fbf1c100cb0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fbf1000b0a0 tx=0x7fbf1000b3b0 comp rx=0 tx=0).stop 2024-03-31T23:09:54.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:54.716+0000 7fbf23adc700 1 -- 172.21.15.46:0/2163879843 shutdown_connections 2024-03-31T23:09:54.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:54.716+0000 7fbf23adc700 1 --2- 172.21.15.46:0/2163879843 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbf1c1011f0 0x7fbf1c10f150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:54.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:54.716+0000 7fbf23adc700 1 --2- 172.21.15.46:0/2163879843 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbf1c1072b0 0x7fbf1c107690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:54.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:54.716+0000 7fbf23adc700 1 --2- 172.21.15.46:0/2163879843 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbf1c100830 0x7fbf1c100cb0 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:54.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:54.716+0000 7fbf23adc700 1 -- 172.21.15.46:0/2163879843 >> 172.21.15.46:0/2163879843 conn(0x7fbf1c077740 msgr2=0x7fbf1c0feb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:09:54.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:54.716+0000 7fbf23adc700 1 -- 172.21.15.46:0/2163879843 shutdown_connections 2024-03-31T23:09:54.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:54.716+0000 7fbf23adc700 1 -- 172.21.15.46:0/2163879843 wait complete. 2024-03-31T23:09:54.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:54.717+0000 7fbf23adc700 1 Processor -- start 2024-03-31T23:09:54.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:54.717+0000 7fbf23adc700 1 -- start start 2024-03-31T23:09:54.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:54.717+0000 7fbf23adc700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbf1c100830 0x7fbf1c101a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:09:54.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:54.717+0000 7fbf23adc700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbf1c1011f0 0x7fbf1c101f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:09:54.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:54.717+0000 7fbf23adc700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbf1c1072b0 0x7fbf1c102490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:09:54.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:54.717+0000 7fbf23adc700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fbf1c112540 con 0x7fbf1c1011f0 2024-03-31T23:09:54.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:54.717+0000 7fbf23adc700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fbf1c1123c0 con 0x7fbf1c1072b0 2024-03-31T23:09:54.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:54.717+0000 7fbf23adc700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fbf1c1126c0 con 0x7fbf1c100830 2024-03-31T23:09:54.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:54.717+0000 7fbf21077700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbf1c1011f0 0x7fbf1c101f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:09:54.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:54.717+0000 7fbf21878700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbf1c100830 0x7fbf1c101a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:09:54.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:54.717+0000 7fbf22079700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbf1c1072b0 0x7fbf1c102490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:09:54.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:54.717+0000 7fbf21077700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbf1c1011f0 0x7fbf1c101f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39292/0 (socket says 172.21.15.46:39292) 2024-03-31T23:09:54.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:54.717+0000 7fbf21077700 1 -- 172.21.15.46:0/3542691691 learned_addr learned my addr 172.21.15.46:0/3542691691 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:09:54.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:54.717+0000 7fbf21077700 1 -- 172.21.15.46:0/3542691691 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbf1c100830 msgr2=0x7fbf1c101a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:09:54.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:54.717+0000 7fbf21077700 1 --2- 172.21.15.46:0/3542691691 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbf1c100830 0x7fbf1c101a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:54.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:54.717+0000 7fbf21077700 1 -- 172.21.15.46:0/3542691691 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbf1c1072b0 msgr2=0x7fbf1c102490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:09:54.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:54.717+0000 7fbf21077700 1 --2- 172.21.15.46:0/3542691691 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbf1c1072b0 0x7fbf1c102490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:54.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:54.717+0000 7fbf21077700 1 -- 172.21.15.46:0/3542691691 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbf1c1a6fc0 con 0x7fbf1c1011f0 2024-03-31T23:09:54.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:54.717+0000 7fbf21077700 1 --2- 172.21.15.46:0/3542691691 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbf1c1011f0 0x7fbf1c101f50 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fbf1000b070 tx=0x7fbf10004d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:09:54.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:54.718+0000 7fbf1affd700 1 -- 172.21.15.46:0/3542691691 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf10021070 con 0x7fbf1c1011f0 2024-03-31T23:09:54.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:54.718+0000 7fbf23adc700 1 -- 172.21.15.46:0/3542691691 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbf1c1a7250 con 0x7fbf1c1011f0 2024-03-31T23:09:54.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:54.718+0000 7fbf1affd700 1 -- 172.21.15.46:0/3542691691 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fbf10007e30 con 0x7fbf1c1011f0 2024-03-31T23:09:54.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:54.718+0000 7fbf1affd700 1 -- 172.21.15.46:0/3542691691 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf100046c0 con 0x7fbf1c1011f0 2024-03-31T23:09:54.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:54.718+0000 7fbf23adc700 1 -- 172.21.15.46:0/3542691691 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbf1c1a7750 con 0x7fbf1c1011f0 2024-03-31T23:09:54.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:54.718+0000 7fbf23adc700 1 -- 172.21.15.46:0/3542691691 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fbf1c062950 con 0x7fbf1c1011f0 2024-03-31T23:09:54.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:54.720+0000 7fbf1affd700 1 -- 172.21.15.46:0/3542691691 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 17) ==== 137371+0+0 (secure 0 0 0) 0x7fbf10007980 con 0x7fbf1c1011f0 2024-03-31T23:09:54.728 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:54.720+0000 7fbf1affd700 1 --2- 172.21.15.46:0/3542691691 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fbf040a25c0 0x7fbf040a4a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:09:54.728 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:54.720+0000 7fbf21878700 1 --2- 172.21.15.46:0/3542691691 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fbf040a25c0 0x7fbf040a4a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:09:54.728 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:54.720+0000 7fbf1affd700 1 -- 172.21.15.46:0/3542691691 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8802+0+0 (secure 0 0 0) 0x7fbf1000f020 con 0x7fbf1c1011f0 2024-03-31T23:09:54.728 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:54.721+0000 7fbf1affd700 1 --2- 172.21.15.46:0/3542691691 >> [v2:172.21.15.162:6816/1173078318,v1:172.21.15.162:6817/1173078318] conn(0x7fbf040ad280 0x7fbf040af6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:09:54.728 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:54.721+0000 7fbf1affd700 1 -- 172.21.15.46:0/3542691691 --> [v2:172.21.15.162:6816/1173078318,v1:172.21.15.162:6817/1173078318] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fbf040afd90 con 0x7fbf040ad280 2024-03-31T23:09:54.728 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:54.721+0000 7fbf1affd700 1 -- 172.21.15.46:0/3542691691 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7fbf100d0530 con 0x7fbf1c1011f0 2024-03-31T23:09:54.728 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:54.721+0000 7fbf21878700 1 --2- 172.21.15.46:0/3542691691 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fbf040a25c0 0x7fbf040a4a80 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fbf14009be0 tx=0x7fbf14009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:09:54.728 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:54.721+0000 7fbf22079700 1 --2- 172.21.15.46:0/3542691691 >> [v2:172.21.15.162:6816/1173078318,v1:172.21.15.162:6817/1173078318] conn(0x7fbf040ad280 0x7fbf040af6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:09:54.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:54.721+0000 7fbf22079700 1 --2- 172.21.15.46:0/3542691691 >> [v2:172.21.15.162:6816/1173078318,v1:172.21.15.162:6817/1173078318] conn(0x7fbf040ad280 0x7fbf040af6c0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:09:54.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:54.722+0000 7fbf1affd700 1 -- 172.21.15.46:0/3542691691 <== osd.8 v2:172.21.15.162:6816/1173078318 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fbf040afd90 con 0x7fbf040ad280 2024-03-31T23:09:54.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:54.745+0000 7fbf23adc700 1 -- 172.21.15.46:0/3542691691 --> [v2:172.21.15.162:6816/1173078318,v1:172.21.15.162:6817/1173078318] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fbf1c062d50 con 0x7fbf040ad280 2024-03-31T23:09:54.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:54.746+0000 7fbf1affd700 1 -- 172.21.15.46:0/3542691691 <== osd.8 v2:172.21.15.162:6816/1173078318 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fbf1c062d50 con 0x7fbf040ad280 2024-03-31T23:09:54.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:54.746+0000 7fbf23adc700 1 -- 172.21.15.46:0/3542691691 >> [v2:172.21.15.162:6816/1173078318,v1:172.21.15.162:6817/1173078318] conn(0x7fbf040ad280 msgr2=0x7fbf040af6c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:09:54.748 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:54.746+0000 7fbf23adc700 1 --2- 172.21.15.46:0/3542691691 >> [v2:172.21.15.162:6816/1173078318,v1:172.21.15.162:6817/1173078318] conn(0x7fbf040ad280 0x7fbf040af6c0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:54.748 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:54.746+0000 7fbf23adc700 1 -- 172.21.15.46:0/3542691691 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fbf040a25c0 msgr2=0x7fbf040a4a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:09:54.748 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:54.746+0000 7fbf23adc700 1 --2- 172.21.15.46:0/3542691691 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fbf040a25c0 0x7fbf040a4a80 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fbf14009be0 tx=0x7fbf14009260 comp rx=0 tx=0).stop 2024-03-31T23:09:54.748 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:54.746+0000 7fbf23adc700 1 -- 172.21.15.46:0/3542691691 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbf1c1011f0 msgr2=0x7fbf1c101f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:09:54.748 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:54.746+0000 7fbf23adc700 1 --2- 172.21.15.46:0/3542691691 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbf1c1011f0 0x7fbf1c101f50 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fbf1000b070 tx=0x7fbf10004d00 comp rx=0 tx=0).stop 2024-03-31T23:09:54.748 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:54.746+0000 7fbf22079700 1 -- 172.21.15.46:0/3542691691 reap_dead start 2024-03-31T23:09:54.748 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:54.747+0000 7fbf23adc700 1 -- 172.21.15.46:0/3542691691 shutdown_connections 2024-03-31T23:09:54.748 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:54.747+0000 7fbf23adc700 1 -- 172.21.15.46:0/3542691691 >> 172.21.15.46:0/3542691691 conn(0x7fbf1c077740 msgr2=0x7fbf1c105550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:09:54.748 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:54.747+0000 7fbf23adc700 1 -- 172.21.15.46:0/3542691691 shutdown_connections 2024-03-31T23:09:54.748 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:54.747+0000 7fbf23adc700 1 -- 172.21.15.46:0/3542691691 wait complete. 2024-03-31T23:09:54.795 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:09:54 smithi162 ceph-mon[36237]: pgmap v313: 129 pgs: 129 active+clean; 640 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 1.2 KiB/s wr, 6 op/s 2024-03-31T23:09:54.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:09:54 smithi142 ceph-mon[35271]: pgmap v313: 129 pgs: 129 active+clean; 640 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 1.2 KiB/s wr, 6 op/s 2024-03-31T23:09:56.795 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:09:56 smithi162 ceph-mon[36237]: pgmap v314: 129 pgs: 129 active+clean; 640 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 1.2 KiB/s wr, 6 op/s 2024-03-31T23:09:56.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:09:56 smithi046 ceph-mon[32498]: pgmap v314: 129 pgs: 129 active+clean; 640 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 1.2 KiB/s wr, 6 op/s 2024-03-31T23:09:56.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:09:56 smithi142 ceph-mon[35271]: pgmap v314: 129 pgs: 129 active+clean; 640 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 1.2 KiB/s wr, 6 op/s 2024-03-31T23:09:58.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:57.533+0000 7f06976d8700 1 -- 172.21.15.46:0/2849152290 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0690106420 msgr2=0x7f069010dff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:09:58.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:57.533+0000 7f06976d8700 1 --2- 172.21.15.46:0/2849152290 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0690106420 0x7f069010dff0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f068000b0d0 tx=0x7f068000b3e0 comp rx=0 tx=0).stop 2024-03-31T23:09:58.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:57.533+0000 7f06976d8700 1 -- 172.21.15.46:0/2849152290 shutdown_connections 2024-03-31T23:09:58.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:57.533+0000 7f06976d8700 1 --2- 172.21.15.46:0/2849152290 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0690106420 0x7f069010dff0 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:58.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:57.533+0000 7f06976d8700 1 --2- 172.21.15.46:0/2849152290 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f06901013d0 0x7f06901017b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:58.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:57.533+0000 7f06976d8700 1 --2- 172.21.15.46:0/2849152290 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0690101d80 0x7f0690105d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:58.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:57.533+0000 7f06976d8700 1 -- 172.21.15.46:0/2849152290 >> 172.21.15.46:0/2849152290 conn(0x7f0690077190 msgr2=0x7f06900775a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:09:58.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:57.533+0000 7f06976d8700 1 -- 172.21.15.46:0/2849152290 shutdown_connections 2024-03-31T23:09:58.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:57.533+0000 7f06976d8700 1 -- 172.21.15.46:0/2849152290 wait complete. 2024-03-31T23:09:58.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:57.534+0000 7f06976d8700 1 Processor -- start 2024-03-31T23:09:58.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:57.534+0000 7f06976d8700 1 -- start start 2024-03-31T23:09:58.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:57.534+0000 7f06976d8700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f06901013d0 0x7f069019e600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:09:58.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:57.534+0000 7f06976d8700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0690101d80 0x7f069019eb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:09:58.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:57.534+0000 7f06976d8700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0690106420 0x7f06901986d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:09:58.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:57.534+0000 7f06976d8700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f06901112c0 con 0x7f0690101d80 2024-03-31T23:09:58.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:57.534+0000 7f06976d8700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f0690111140 con 0x7f0690106420 2024-03-31T23:09:58.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:57.534+0000 7f06976d8700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f0690111440 con 0x7f06901013d0 2024-03-31T23:09:58.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:57.534+0000 7f0694c73700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0690101d80 0x7f069019eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:09:58.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:57.534+0000 7f0695474700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f06901013d0 0x7f069019e600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:09:58.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:57.534+0000 7f0694c73700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0690101d80 0x7f069019eb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39306/0 (socket says 172.21.15.46:39306) 2024-03-31T23:09:58.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:57.534+0000 7f0694c73700 1 -- 172.21.15.46:0/2538719174 learned_addr learned my addr 172.21.15.46:0/2538719174 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:09:58.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:57.534+0000 7f0695c75700 1 --2- 172.21.15.46:0/2538719174 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0690106420 0x7f06901986d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:09:58.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:57.535+0000 7f0694c73700 1 -- 172.21.15.46:0/2538719174 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f06901013d0 msgr2=0x7f069019e600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:09:58.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:57.535+0000 7f0694c73700 1 --2- 172.21.15.46:0/2538719174 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f06901013d0 0x7f069019e600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:58.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:57.535+0000 7f0694c73700 1 -- 172.21.15.46:0/2538719174 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0690106420 msgr2=0x7f06901986d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:09:58.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:57.535+0000 7f0694c73700 1 --2- 172.21.15.46:0/2538719174 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0690106420 0x7f06901986d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:58.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:57.535+0000 7f0694c73700 1 -- 172.21.15.46:0/2538719174 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0690198fb0 con 0x7f0690101d80 2024-03-31T23:09:58.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:57.535+0000 7f0695474700 1 --2- 172.21.15.46:0/2538719174 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f06901013d0 0x7f069019e600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:09:58.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:57.535+0000 7f0695c75700 1 --2- 172.21.15.46:0/2538719174 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0690106420 0x7f06901986d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:09:58.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:57.535+0000 7f0694c73700 1 --2- 172.21.15.46:0/2538719174 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0690101d80 0x7f069019eb40 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f068400e970 tx=0x7f068400ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:09:58.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:57.535+0000 7f068e7fc700 1 -- 172.21.15.46:0/2538719174 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0684004a30 con 0x7f0690101d80 2024-03-31T23:09:58.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:57.535+0000 7f068e7fc700 1 -- 172.21.15.46:0/2538719174 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f068400cc90 con 0x7f0690101d80 2024-03-31T23:09:58.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:57.535+0000 7f06976d8700 1 -- 172.21.15.46:0/2538719174 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f06901992a0 con 0x7f0690101d80 2024-03-31T23:09:58.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:57.535+0000 7f068e7fc700 1 -- 172.21.15.46:0/2538719174 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06840055e0 con 0x7f0690101d80 2024-03-31T23:09:58.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:57.535+0000 7f06976d8700 1 -- 172.21.15.46:0/2538719174 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f06901a5580 con 0x7f0690101d80 2024-03-31T23:09:58.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:57.536+0000 7f06976d8700 1 -- 172.21.15.46:0/2538719174 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f0690062990 con 0x7f0690101d80 2024-03-31T23:09:58.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:57.537+0000 7f068e7fc700 1 -- 172.21.15.46:0/2538719174 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 17) ==== 137371+0+0 (secure 0 0 0) 0x7f068400c7e0 con 0x7f0690101d80 2024-03-31T23:09:58.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:57.538+0000 7f068e7fc700 1 --2- 172.21.15.46:0/2538719174 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f06780a2640 0x7f06780a4b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:09:58.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:57.538+0000 7f0695474700 1 --2- 172.21.15.46:0/2538719174 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f06780a2640 0x7f06780a4b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:09:58.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:57.538+0000 7f068e7fc700 1 -- 172.21.15.46:0/2538719174 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8802+0+0 (secure 0 0 0) 0x7f06840ceff0 con 0x7f0690101d80 2024-03-31T23:09:58.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:57.538+0000 7f068e7fc700 1 --2- 172.21.15.46:0/2538719174 >> [v2:172.21.15.142:6800/973693922,v1:172.21.15.142:6801/973693922] conn(0x7f06780ad300 0x7f06780af740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:09:58.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:57.538+0000 7f068e7fc700 1 -- 172.21.15.46:0/2538719174 --> [v2:172.21.15.142:6800/973693922,v1:172.21.15.142:6801/973693922] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f06780afe10 con 0x7f06780ad300 2024-03-31T23:09:58.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:57.538+0000 7f068e7fc700 1 -- 172.21.15.46:0/2538719174 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7f06840cf440 con 0x7f0690101d80 2024-03-31T23:09:58.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:57.538+0000 7f0695474700 1 --2- 172.21.15.46:0/2538719174 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f06780a2640 0x7f06780a4b00 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f0688011490 tx=0x7f0688009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:09:58.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:57.538+0000 7f0695c75700 1 --2- 172.21.15.46:0/2538719174 >> [v2:172.21.15.142:6800/973693922,v1:172.21.15.142:6801/973693922] conn(0x7f06780ad300 0x7f06780af740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:09:58.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:57.539+0000 7f0695c75700 1 --2- 172.21.15.46:0/2538719174 >> [v2:172.21.15.142:6800/973693922,v1:172.21.15.142:6801/973693922] conn(0x7f06780ad300 0x7f06780af740 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:09:58.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:57.539+0000 7f068e7fc700 1 -- 172.21.15.46:0/2538719174 <== osd.1 v2:172.21.15.142:6800/973693922 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f06780afe10 con 0x7f06780ad300 2024-03-31T23:09:58.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:57.555+0000 7f06976d8700 1 -- 172.21.15.46:0/2538719174 --> [v2:172.21.15.142:6800/973693922,v1:172.21.15.142:6801/973693922] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f0690062d90 con 0x7f06780ad300 2024-03-31T23:09:58.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:57.556+0000 7f068e7fc700 1 -- 172.21.15.46:0/2538719174 <== osd.1 v2:172.21.15.142:6800/973693922 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f0690062d90 con 0x7f06780ad300 2024-03-31T23:09:58.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:57.556+0000 7f06976d8700 1 -- 172.21.15.46:0/2538719174 >> [v2:172.21.15.142:6800/973693922,v1:172.21.15.142:6801/973693922] conn(0x7f06780ad300 msgr2=0x7f06780af740 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:09:58.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:57.556+0000 7f06976d8700 1 --2- 172.21.15.46:0/2538719174 >> [v2:172.21.15.142:6800/973693922,v1:172.21.15.142:6801/973693922] conn(0x7f06780ad300 0x7f06780af740 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:58.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:57.556+0000 7f06976d8700 1 -- 172.21.15.46:0/2538719174 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f06780a2640 msgr2=0x7f06780a4b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:09:58.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:57.556+0000 7f06976d8700 1 --2- 172.21.15.46:0/2538719174 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f06780a2640 0x7f06780a4b00 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f0688011490 tx=0x7f0688009260 comp rx=0 tx=0).stop 2024-03-31T23:09:58.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:57.556+0000 7f06976d8700 1 -- 172.21.15.46:0/2538719174 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0690101d80 msgr2=0x7f069019eb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:09:58.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:57.556+0000 7f06976d8700 1 --2- 172.21.15.46:0/2538719174 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0690101d80 0x7f069019eb40 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f068400e970 tx=0x7f068400ec80 comp rx=0 tx=0).stop 2024-03-31T23:09:58.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:57.556+0000 7f0695c75700 1 -- 172.21.15.46:0/2538719174 reap_dead start 2024-03-31T23:09:58.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:57.556+0000 7f06976d8700 1 -- 172.21.15.46:0/2538719174 shutdown_connections 2024-03-31T23:09:58.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:57.556+0000 7f06976d8700 1 -- 172.21.15.46:0/2538719174 >> 172.21.15.46:0/2538719174 conn(0x7f0690077190 msgr2=0x7f069006aa90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:09:58.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:57.556+0000 7f06976d8700 1 -- 172.21.15.46:0/2538719174 shutdown_connections 2024-03-31T23:09:58.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:57.556+0000 7f06976d8700 1 -- 172.21.15.46:0/2538719174 wait complete. 2024-03-31T23:09:58.790 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:09:58 smithi046 ceph-mon[32498]: pgmap v315: 129 pgs: 129 active+clean; 640 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1.2 KiB/s wr, 7 op/s 2024-03-31T23:09:58.795 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:09:58 smithi162 ceph-mon[36237]: pgmap v315: 129 pgs: 129 active+clean; 640 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1.2 KiB/s wr, 7 op/s 2024-03-31T23:09:58.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:09:58 smithi142 ceph-mon[35271]: pgmap v315: 129 pgs: 129 active+clean; 640 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1.2 KiB/s wr, 7 op/s 2024-03-31T23:09:59.306 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:59.305+0000 7f0005587700 1 -- 172.21.15.46:0/4132016946 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f000006abc0 msgr2=0x7f000006afa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:09:59.306 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:59.305+0000 7f0005587700 1 --2- 172.21.15.46:0/4132016946 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f000006abc0 0x7f000006afa0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7efff00099c0 tx=0x7efff0009cd0 comp rx=0 tx=0).stop 2024-03-31T23:09:59.306 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:59.305+0000 7f0005587700 1 -- 172.21.15.46:0/4132016946 shutdown_connections 2024-03-31T23:09:59.306 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:59.305+0000 7f0005587700 1 --2- 172.21.15.46:0/4132016946 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f000006abc0 0x7f000006afa0 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:59.306 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:59.305+0000 7f0005587700 1 --2- 172.21.15.46:0/4132016946 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f00001075d0 0x7f000010f170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:59.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:59.305+0000 7f0005587700 1 --2- 172.21.15.46:0/4132016946 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f000006b570 0x7f0000106ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:59.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:59.305+0000 7f0005587700 1 -- 172.21.15.46:0/4132016946 >> 172.21.15.46:0/4132016946 conn(0x7f0000078000 msgr2=0x7f0000078410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:09:59.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:59.305+0000 7f0005587700 1 -- 172.21.15.46:0/4132016946 shutdown_connections 2024-03-31T23:09:59.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:59.305+0000 7f0005587700 1 -- 172.21.15.46:0/4132016946 wait complete. 2024-03-31T23:09:59.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:59.306+0000 7f0005587700 1 Processor -- start 2024-03-31T23:09:59.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:59.306+0000 7f0005587700 1 -- start start 2024-03-31T23:09:59.308 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:59.306+0000 7f0005587700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f000006abc0 0x7f00001014f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:09:59.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:59.306+0000 7f0005587700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f000006b570 0x7f0000101a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:09:59.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:59.306+0000 7f0005587700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f00001075d0 0x7f0000101f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:09:59.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:59.306+0000 7f0005587700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f0000112460 con 0x7f00001075d0 2024-03-31T23:09:59.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:59.306+0000 7f0005587700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f00001122e0 con 0x7f000006b570 2024-03-31T23:09:59.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:59.306+0000 7f0005587700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f00001125e0 con 0x7f000006abc0 2024-03-31T23:09:59.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:59.307+0000 7effff7fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f00001075d0 0x7f0000101f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:09:59.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:59.307+0000 7efffe7fc700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f000006b570 0x7f0000101a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:09:59.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:59.307+0000 7efffe7fc700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f000006b570 0x7f0000101a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.46:49916/0 (socket says 172.21.15.46:49916) 2024-03-31T23:09:59.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:59.307+0000 7efffe7fc700 1 -- 172.21.15.46:0/1664827949 learned_addr learned my addr 172.21.15.46:0/1664827949 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:09:59.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:59.307+0000 7effff7fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f00001075d0 0x7f0000101f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39330/0 (socket says 172.21.15.46:39330) 2024-03-31T23:09:59.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:59.307+0000 7efffeffd700 1 --2- 172.21.15.46:0/1664827949 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f000006abc0 0x7f00001014f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:09:59.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:59.307+0000 7efffe7fc700 1 -- 172.21.15.46:0/1664827949 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f000006abc0 msgr2=0x7f00001014f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:09:59.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:59.307+0000 7efffe7fc700 1 --2- 172.21.15.46:0/1664827949 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f000006abc0 0x7f00001014f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:59.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:59.307+0000 7efffe7fc700 1 -- 172.21.15.46:0/1664827949 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f00001075d0 msgr2=0x7f0000101f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:09:59.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:59.307+0000 7efffe7fc700 1 --2- 172.21.15.46:0/1664827949 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f00001075d0 0x7f0000101f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:59.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:59.307+0000 7efffe7fc700 1 -- 172.21.15.46:0/1664827949 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f00001a6e60 con 0x7f000006b570 2024-03-31T23:09:59.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:59.307+0000 7effff7fe700 1 --2- 172.21.15.46:0/1664827949 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f00001075d0 0x7f0000101f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:09:59.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:59.307+0000 7efffe7fc700 1 --2- 172.21.15.46:0/1664827949 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f000006b570 0x7f0000101a30 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7efff400e940 tx=0x7efff400ec50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:09:59.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:59.307+0000 7effe7fff700 1 -- 172.21.15.46:0/1664827949 <== mon.2 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efff400c9e0 con 0x7f000006b570 2024-03-31T23:09:59.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:59.307+0000 7f0005587700 1 -- 172.21.15.46:0/1664827949 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f00001a7150 con 0x7f000006b570 2024-03-31T23:09:59.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:59.307+0000 7effe7fff700 1 -- 172.21.15.46:0/1664827949 <== mon.2 v2:172.21.15.142:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7efff400cb80 con 0x7f000006b570 2024-03-31T23:09:59.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:59.307+0000 7effe7fff700 1 -- 172.21.15.46:0/1664827949 <== mon.2 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efff4018680 con 0x7f000006b570 2024-03-31T23:09:59.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:59.307+0000 7f0005587700 1 -- 172.21.15.46:0/1664827949 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f00001a76e0 con 0x7f000006b570 2024-03-31T23:09:59.311 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:59.308+0000 7f0005587700 1 -- 172.21.15.46:0/1664827949 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f0000062910 con 0x7f000006b570 2024-03-31T23:09:59.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:59.310+0000 7effe7fff700 1 -- 172.21.15.46:0/1664827949 <== mon.2 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 17) ==== 137371+0+0 (secure 0 0 0) 0x7efff4010430 con 0x7f000006b570 2024-03-31T23:09:59.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:59.310+0000 7effe7fff700 1 --2- 172.21.15.46:0/1664827949 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7effe80a25c0 0x7effe80a4a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:09:59.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:59.310+0000 7efffeffd700 1 --2- 172.21.15.46:0/1664827949 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7effe80a25c0 0x7effe80a4a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:09:59.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:59.310+0000 7effe7fff700 1 -- 172.21.15.46:0/1664827949 <== mon.2 v2:172.21.15.142:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8802+0+0 (secure 0 0 0) 0x7efff40cf1e0 con 0x7f000006b570 2024-03-31T23:09:59.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:59.311+0000 7effe7fff700 1 --2- 172.21.15.46:0/1664827949 >> [v2:172.21.15.162:6808/914208293,v1:172.21.15.162:6809/914208293] conn(0x7effe80ad280 0x7effe80af6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:09:59.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:59.311+0000 7effe7fff700 1 -- 172.21.15.46:0/1664827949 --> [v2:172.21.15.162:6808/914208293,v1:172.21.15.162:6809/914208293] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7efff4001d30 con 0x7effe80ad280 2024-03-31T23:09:59.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:59.311+0000 7effe7fff700 1 -- 172.21.15.46:0/1664827949 <== mon.2 v2:172.21.15.142:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7efff40cf590 con 0x7f000006b570 2024-03-31T23:09:59.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:59.311+0000 7efffeffd700 1 --2- 172.21.15.46:0/1664827949 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7effe80a25c0 0x7effe80a4a80 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7efff0005b40 tx=0x7efff0005a90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:09:59.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:59.311+0000 7effff7fe700 1 --2- 172.21.15.46:0/1664827949 >> [v2:172.21.15.162:6808/914208293,v1:172.21.15.162:6809/914208293] conn(0x7effe80ad280 0x7effe80af6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:09:59.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:59.311+0000 7effff7fe700 1 --2- 172.21.15.46:0/1664827949 >> [v2:172.21.15.162:6808/914208293,v1:172.21.15.162:6809/914208293] conn(0x7effe80ad280 0x7effe80af6c0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:09:59.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:59.312+0000 7effe7fff700 1 -- 172.21.15.46:0/1664827949 <== osd.5 v2:172.21.15.162:6808/914208293 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7efff4001d30 con 0x7effe80ad280 2024-03-31T23:09:59.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:59.330+0000 7f0005587700 1 -- 172.21.15.46:0/1664827949 --> [v2:172.21.15.162:6808/914208293,v1:172.21.15.162:6809/914208293] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f0000062d10 con 0x7effe80ad280 2024-03-31T23:09:59.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:59.330+0000 7effe7fff700 1 -- 172.21.15.46:0/1664827949 <== osd.5 v2:172.21.15.162:6808/914208293 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f0000062d10 con 0x7effe80ad280 2024-03-31T23:09:59.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:59.330+0000 7f0005587700 1 -- 172.21.15.46:0/1664827949 >> [v2:172.21.15.162:6808/914208293,v1:172.21.15.162:6809/914208293] conn(0x7effe80ad280 msgr2=0x7effe80af6c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:09:59.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:59.330+0000 7f0005587700 1 --2- 172.21.15.46:0/1664827949 >> [v2:172.21.15.162:6808/914208293,v1:172.21.15.162:6809/914208293] conn(0x7effe80ad280 0x7effe80af6c0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:09:59.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:59.330+0000 7f0005587700 1 -- 172.21.15.46:0/1664827949 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7effe80a25c0 msgr2=0x7effe80a4a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:09:59.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:59.330+0000 7f0005587700 1 --2- 172.21.15.46:0/1664827949 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7effe80a25c0 0x7effe80a4a80 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7efff0005b40 tx=0x7efff0005a90 comp rx=0 tx=0).stop 2024-03-31T23:09:59.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:59.330+0000 7f0005587700 1 -- 172.21.15.46:0/1664827949 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f000006b570 msgr2=0x7f0000101a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:09:59.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:59.330+0000 7f0005587700 1 --2- 172.21.15.46:0/1664827949 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f000006b570 0x7f0000101a30 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7efff400e940 tx=0x7efff400ec50 comp rx=0 tx=0).stop 2024-03-31T23:09:59.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:59.330+0000 7effff7fe700 1 -- 172.21.15.46:0/1664827949 reap_dead start 2024-03-31T23:09:59.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:59.331+0000 7f0005587700 1 -- 172.21.15.46:0/1664827949 shutdown_connections 2024-03-31T23:09:59.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:59.331+0000 7f0005587700 1 -- 172.21.15.46:0/1664827949 >> 172.21.15.46:0/1664827949 conn(0x7f0000078000 msgr2=0x7f00000ff6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:09:59.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:59.331+0000 7f0005587700 1 -- 172.21.15.46:0/1664827949 shutdown_connections 2024-03-31T23:09:59.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:09:59.331+0000 7f0005587700 1 -- 172.21.15.46:0/1664827949 wait complete. 2024-03-31T23:10:00.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:10:00 smithi162 ceph-mon[36237]: pgmap v316: 129 pgs: 129 active+clean; 640 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 341 B/s wr, 6 op/s 2024-03-31T23:10:00.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:10:00 smithi162 ceph-mon[36237]: overall HEALTH_OK 2024-03-31T23:10:00.849 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:10:00 smithi046 ceph-mon[32498]: pgmap v316: 129 pgs: 129 active+clean; 640 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 341 B/s wr, 6 op/s 2024-03-31T23:10:00.849 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:10:00 smithi046 ceph-mon[32498]: overall HEALTH_OK 2024-03-31T23:10:00.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:10:00 smithi142 ceph-mon[35271]: pgmap v316: 129 pgs: 129 active+clean; 640 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 341 B/s wr, 6 op/s 2024-03-31T23:10:00.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:10:00 smithi142 ceph-mon[35271]: overall HEALTH_OK 2024-03-31T23:10:02.689 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:10:02 smithi046 ceph-mon[32498]: pgmap v317: 129 pgs: 129 active+clean; 642 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 597 B/s wr, 7 op/s 2024-03-31T23:10:02.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:02.695+0000 7f3e7bf84700 1 -- 172.21.15.46:0/195064083 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3e74101140 msgr2=0x7f3e741015a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:10:02.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:02.695+0000 7f3e7bf84700 1 --2- 172.21.15.46:0/195064083 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3e74101140 0x7f3e741015a0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f3e64009990 tx=0x7f3e64009ca0 comp rx=0 tx=0).stop 2024-03-31T23:10:02.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:02.695+0000 7f3e7bf84700 1 -- 172.21.15.46:0/195064083 shutdown_connections 2024-03-31T23:10:02.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:02.695+0000 7f3e7bf84700 1 --2- 172.21.15.46:0/195064083 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3e74101ae0 0x7f3e7410dfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:02.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:02.695+0000 7f3e7bf84700 1 --2- 172.21.15.46:0/195064083 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3e74107180 0x7f3e74107560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:02.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:02.695+0000 7f3e7bf84700 1 --2- 172.21.15.46:0/195064083 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3e74101140 0x7f3e741015a0 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:02.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:02.695+0000 7f3e7bf84700 1 -- 172.21.15.46:0/195064083 >> 172.21.15.46:0/195064083 conn(0x7f3e74078070 msgr2=0x7f3e74078480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:10:02.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:02.695+0000 7f3e7bf84700 1 -- 172.21.15.46:0/195064083 shutdown_connections 2024-03-31T23:10:02.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:02.695+0000 7f3e7bf84700 1 -- 172.21.15.46:0/195064083 wait complete. 2024-03-31T23:10:02.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:02.696+0000 7f3e7bf84700 1 Processor -- start 2024-03-31T23:10:02.698 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:02.696+0000 7f3e7bf84700 1 -- start start 2024-03-31T23:10:02.698 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:02.696+0000 7f3e7bf84700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3e74101140 0x7f3e7419e640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:10:02.698 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:02.696+0000 7f3e7bf84700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3e74101ae0 0x7f3e7419eb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:10:02.698 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:02.696+0000 7f3e7bf84700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3e74107180 0x7f3e74198760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:10:02.698 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:02.696+0000 7f3e7bf84700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f3e741112b0 con 0x7f3e74101140 2024-03-31T23:10:02.698 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:02.696+0000 7f3e7bf84700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f3e74111130 con 0x7f3e74101ae0 2024-03-31T23:10:02.698 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:02.696+0000 7f3e7bf84700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f3e74111430 con 0x7f3e74107180 2024-03-31T23:10:02.698 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:02.696+0000 7f3e79d20700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3e74101140 0x7f3e7419e640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:10:02.698 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:02.696+0000 7f3e79d20700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3e74101140 0x7f3e7419e640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:38308/0 (socket says 172.21.15.46:38308) 2024-03-31T23:10:02.698 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:02.696+0000 7f3e79d20700 1 -- 172.21.15.46:0/3758658541 learned_addr learned my addr 172.21.15.46:0/3758658541 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:10:02.698 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:02.696+0000 7f3e7951f700 1 --2- 172.21.15.46:0/3758658541 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3e74101ae0 0x7f3e7419eb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:10:02.698 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:02.696+0000 7f3e7a521700 1 --2- 172.21.15.46:0/3758658541 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3e74107180 0x7f3e74198760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:10:02.698 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:02.696+0000 7f3e7a521700 1 -- 172.21.15.46:0/3758658541 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3e74101ae0 msgr2=0x7f3e7419eb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:10:02.698 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:02.696+0000 7f3e7a521700 1 --2- 172.21.15.46:0/3758658541 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3e74101ae0 0x7f3e7419eb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:02.698 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:02.696+0000 7f3e7a521700 1 -- 172.21.15.46:0/3758658541 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3e74101140 msgr2=0x7f3e7419e640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:10:02.699 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:02.696+0000 7f3e7a521700 1 --2- 172.21.15.46:0/3758658541 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3e74101140 0x7f3e7419e640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:02.699 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:02.696+0000 7f3e7a521700 1 -- 172.21.15.46:0/3758658541 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e74199040 con 0x7f3e74107180 2024-03-31T23:10:02.699 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:02.696+0000 7f3e79d20700 1 --2- 172.21.15.46:0/3758658541 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3e74101140 0x7f3e7419e640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:10:02.699 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:02.697+0000 7f3e7a521700 1 --2- 172.21.15.46:0/3758658541 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3e74107180 0x7f3e74198760 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f3e6c00efc0 tx=0x7f3e6c00c3a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:10:02.699 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:02.697+0000 7f3e6affd700 1 -- 172.21.15.46:0/3758658541 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e6c0092e0 con 0x7f3e74107180 2024-03-31T23:10:02.699 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:02.697+0000 7f3e6affd700 1 -- 172.21.15.46:0/3758658541 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3e6c004bb0 con 0x7f3e74107180 2024-03-31T23:10:02.699 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:02.697+0000 7f3e6affd700 1 -- 172.21.15.46:0/3758658541 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e6c004020 con 0x7f3e74107180 2024-03-31T23:10:02.699 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:02.697+0000 7f3e7bf84700 1 -- 172.21.15.46:0/3758658541 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3e741992d0 con 0x7f3e74107180 2024-03-31T23:10:02.699 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:02.697+0000 7f3e7bf84700 1 -- 172.21.15.46:0/3758658541 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3e741a55c0 con 0x7f3e74107180 2024-03-31T23:10:02.700 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:02.698+0000 7f3e7bf84700 1 -- 172.21.15.46:0/3758658541 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f3e74106290 con 0x7f3e74107180 2024-03-31T23:10:02.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:02.699+0000 7f3e6affd700 1 -- 172.21.15.46:0/3758658541 <== mon.1 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 17) ==== 137371+0+0 (secure 0 0 0) 0x7f3e6c01b030 con 0x7f3e74107180 2024-03-31T23:10:02.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:02.699+0000 7f3e6affd700 1 --2- 172.21.15.46:0/3758658541 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f3e600a2640 0x7f3e600a4b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:10:02.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:02.699+0000 7f3e79d20700 1 --2- 172.21.15.46:0/3758658541 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f3e600a2640 0x7f3e600a4b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:10:02.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:02.700+0000 7f3e6affd700 1 -- 172.21.15.46:0/3758658541 <== mon.1 v2:172.21.15.162:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8802+0+0 (secure 0 0 0) 0x7f3e6c0ce6a0 con 0x7f3e74107180 2024-03-31T23:10:02.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:02.700+0000 7f3e6affd700 1 --2- 172.21.15.46:0/3758658541 >> [v2:172.21.15.46:6810/3496317970,v1:172.21.15.46:6811/3496317970] conn(0x7f3e600ad300 0x7f3e600af740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:10:02.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:02.700+0000 7f3e6affd700 1 -- 172.21.15.46:0/3758658541 --> [v2:172.21.15.46:6810/3496317970,v1:172.21.15.46:6811/3496317970] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f3e6c021e70 con 0x7f3e600ad300 2024-03-31T23:10:02.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:02.700+0000 7f3e6affd700 1 -- 172.21.15.46:0/3758658541 <== mon.1 v2:172.21.15.162:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7f3e6c0ceaf0 con 0x7f3e74107180 2024-03-31T23:10:02.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:02.700+0000 7f3e7951f700 1 --2- 172.21.15.46:0/3758658541 >> [v2:172.21.15.46:6810/3496317970,v1:172.21.15.46:6811/3496317970] conn(0x7f3e600ad300 0x7f3e600af740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:10:02.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:02.700+0000 7f3e79d20700 1 --2- 172.21.15.46:0/3758658541 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f3e600a2640 0x7f3e600a4b00 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f3e70011550 tx=0x7f3e70009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:10:02.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:02.700+0000 7f3e7951f700 1 --2- 172.21.15.46:0/3758658541 >> [v2:172.21.15.46:6810/3496317970,v1:172.21.15.46:6811/3496317970] conn(0x7f3e600ad300 0x7f3e600af740 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:10:02.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:02.701+0000 7f3e6affd700 1 -- 172.21.15.46:0/3758658541 <== osd.3 v2:172.21.15.46:6810/3496317970 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f3e6c021e70 con 0x7f3e600ad300 2024-03-31T23:10:02.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:02.718+0000 7f3e7bf84700 1 -- 172.21.15.46:0/3758658541 --> [v2:172.21.15.46:6810/3496317970,v1:172.21.15.46:6811/3496317970] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f3e74102830 con 0x7f3e600ad300 2024-03-31T23:10:02.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:02.718+0000 7f3e6affd700 1 -- 172.21.15.46:0/3758658541 <== osd.3 v2:172.21.15.46:6810/3496317970 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f3e74102830 con 0x7f3e600ad300 2024-03-31T23:10:02.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:02.718+0000 7f3e7bf84700 1 -- 172.21.15.46:0/3758658541 >> [v2:172.21.15.46:6810/3496317970,v1:172.21.15.46:6811/3496317970] conn(0x7f3e600ad300 msgr2=0x7f3e600af740 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:10:02.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:02.718+0000 7f3e7bf84700 1 --2- 172.21.15.46:0/3758658541 >> [v2:172.21.15.46:6810/3496317970,v1:172.21.15.46:6811/3496317970] conn(0x7f3e600ad300 0x7f3e600af740 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:02.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:02.718+0000 7f3e7bf84700 1 -- 172.21.15.46:0/3758658541 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f3e600a2640 msgr2=0x7f3e600a4b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:10:02.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:02.718+0000 7f3e7bf84700 1 --2- 172.21.15.46:0/3758658541 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f3e600a2640 0x7f3e600a4b00 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f3e70011550 tx=0x7f3e70009260 comp rx=0 tx=0).stop 2024-03-31T23:10:02.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:02.718+0000 7f3e7bf84700 1 -- 172.21.15.46:0/3758658541 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3e74107180 msgr2=0x7f3e74198760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:10:02.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:02.718+0000 7f3e7bf84700 1 --2- 172.21.15.46:0/3758658541 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3e74107180 0x7f3e74198760 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f3e6c00efc0 tx=0x7f3e6c00c3a0 comp rx=0 tx=0).stop 2024-03-31T23:10:02.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:02.718+0000 7f3e7a521700 1 -- 172.21.15.46:0/3758658541 reap_dead start 2024-03-31T23:10:02.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:02.719+0000 7f3e7bf84700 1 -- 172.21.15.46:0/3758658541 shutdown_connections 2024-03-31T23:10:02.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:02.719+0000 7f3e7bf84700 1 -- 172.21.15.46:0/3758658541 >> 172.21.15.46:0/3758658541 conn(0x7f3e74078070 msgr2=0x7f3e7406a100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:10:02.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:02.719+0000 7f3e7bf84700 1 -- 172.21.15.46:0/3758658541 shutdown_connections 2024-03-31T23:10:02.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:02.719+0000 7f3e7bf84700 1 -- 172.21.15.46:0/3758658541 wait complete. 2024-03-31T23:10:02.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:10:02 smithi162 ceph-mon[36237]: pgmap v317: 129 pgs: 129 active+clean; 642 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 597 B/s wr, 7 op/s 2024-03-31T23:10:02.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:10:02 smithi142 ceph-mon[35271]: pgmap v317: 129 pgs: 129 active+clean; 642 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 597 B/s wr, 7 op/s 2024-03-31T23:10:03.165 INFO:teuthology.orchestra.run.smithi046.stdout:167503724598 2024-03-31T23:10:03.166 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:384f31f3d3aee7f8ba3f60846b8844c049165583 shell --fsid 6a9d8f08-efb2-11ee-b647-cb9ed24678a4 -- ceph osd last-stat-seq osd.5 2024-03-31T23:10:03.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:03.795+0000 7f48bd88b700 1 -- 172.21.15.46:0/182897020 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f48b8107180 msgr2=0x7f48b8107560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:10:03.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:03.795+0000 7f48bd88b700 1 --2- 172.21.15.46:0/182897020 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f48b8107180 0x7f48b8107560 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f48b000b0a0 tx=0x7f48b000b3b0 comp rx=0 tx=0).stop 2024-03-31T23:10:03.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:03.795+0000 7f48bd88b700 1 -- 172.21.15.46:0/182897020 shutdown_connections 2024-03-31T23:10:03.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:03.795+0000 7f48bd88b700 1 --2- 172.21.15.46:0/182897020 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f48b8107180 0x7f48b8107560 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:03.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:03.795+0000 7f48bd88b700 1 --2- 172.21.15.46:0/182897020 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f48b8101ae0 0x7f48b810dfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:03.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:03.795+0000 7f48bd88b700 1 --2- 172.21.15.46:0/182897020 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f48b8101140 0x7f48b81015a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:03.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:03.795+0000 7f48bd88b700 1 -- 172.21.15.46:0/182897020 >> 172.21.15.46:0/182897020 conn(0x7f48b8078070 msgr2=0x7f48b8078480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:10:03.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:03.795+0000 7f48bd88b700 1 -- 172.21.15.46:0/182897020 shutdown_connections 2024-03-31T23:10:03.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:03.795+0000 7f48bd88b700 1 -- 172.21.15.46:0/182897020 wait complete. 2024-03-31T23:10:03.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:03.795+0000 7f48bd88b700 1 Processor -- start 2024-03-31T23:10:03.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:03.796+0000 7f48bd88b700 1 -- start start 2024-03-31T23:10:03.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:03.796+0000 7f48bd88b700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f48b8101140 0x7f48b819e600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:10:03.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:03.796+0000 7f48bd88b700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f48b8101ae0 0x7f48b819eb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:10:03.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:03.796+0000 7f48bd88b700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f48b8107180 0x7f48b81986d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:10:03.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:03.796+0000 7f48bd88b700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f48b8111290 con 0x7f48b8107180 2024-03-31T23:10:03.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:03.796+0000 7f48bd88b700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f48b8111110 con 0x7f48b8101ae0 2024-03-31T23:10:03.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:03.796+0000 7f48bd88b700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f48b8111410 con 0x7f48b8101140 2024-03-31T23:10:03.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:03.796+0000 7f48b77fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f48b8107180 0x7f48b81986d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:10:03.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:03.796+0000 7f48b6ffd700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f48b8101140 0x7f48b819e600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:10:03.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:03.796+0000 7f48b67fc700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f48b8101ae0 0x7f48b819eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:10:03.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:03.796+0000 7f48b77fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f48b8107180 0x7f48b81986d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:38334/0 (socket says 172.21.15.46:38334) 2024-03-31T23:10:03.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:03.796+0000 7f48b77fe700 1 -- 172.21.15.46:0/4230555918 learned_addr learned my addr 172.21.15.46:0/4230555918 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:10:03.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:03.796+0000 7f48b67fc700 1 -- 172.21.15.46:0/4230555918 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f48b8101140 msgr2=0x7f48b819e600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:10:03.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:03.796+0000 7f48b67fc700 1 --2- 172.21.15.46:0/4230555918 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f48b8101140 0x7f48b819e600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:03.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:03.796+0000 7f48b67fc700 1 -- 172.21.15.46:0/4230555918 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f48b8107180 msgr2=0x7f48b81986d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:10:03.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:03.796+0000 7f48b67fc700 1 --2- 172.21.15.46:0/4230555918 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f48b8107180 0x7f48b81986d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:03.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:03.796+0000 7f48b67fc700 1 -- 172.21.15.46:0/4230555918 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f48b8198fb0 con 0x7f48b8101ae0 2024-03-31T23:10:03.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:03.796+0000 7f48b77fe700 1 --2- 172.21.15.46:0/4230555918 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f48b8107180 0x7f48b81986d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:10:03.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:03.796+0000 7f48b6ffd700 1 --2- 172.21.15.46:0/4230555918 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f48b8101140 0x7f48b819e600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:10:03.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:03.797+0000 7f48b67fc700 1 --2- 172.21.15.46:0/4230555918 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f48b8101ae0 0x7f48b819eb40 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f48a000e9f0 tx=0x7f48a000ed00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:10:03.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:03.797+0000 7f48bc889700 1 -- 172.21.15.46:0/4230555918 <== mon.2 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48a0004220 con 0x7f48b8101ae0 2024-03-31T23:10:03.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:03.797+0000 7f48bd88b700 1 -- 172.21.15.46:0/4230555918 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f48b81992a0 con 0x7f48b8101ae0 2024-03-31T23:10:03.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:03.797+0000 7f48bc889700 1 -- 172.21.15.46:0/4230555918 <== mon.2 v2:172.21.15.142:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f48a00043c0 con 0x7f48b8101ae0 2024-03-31T23:10:03.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:03.797+0000 7f48bc889700 1 -- 172.21.15.46:0/4230555918 <== mon.2 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48a0005020 con 0x7f48b8101ae0 2024-03-31T23:10:03.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:03.797+0000 7f48bd88b700 1 -- 172.21.15.46:0/4230555918 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f48b81a5580 con 0x7f48b8101ae0 2024-03-31T23:10:03.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:03.797+0000 7f48bd88b700 1 -- 172.21.15.46:0/4230555918 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f48b8106250 con 0x7f48b8101ae0 2024-03-31T23:10:03.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:03.799+0000 7f48bc889700 1 -- 172.21.15.46:0/4230555918 <== mon.2 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 17) ==== 137371+0+0 (secure 0 0 0) 0x7f48a0004750 con 0x7f48b8101ae0 2024-03-31T23:10:03.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:03.799+0000 7f48bc889700 1 --2- 172.21.15.46:0/4230555918 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f48a40a25c0 0x7f48a40a4a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:10:03.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:03.800+0000 7f48b6ffd700 1 --2- 172.21.15.46:0/4230555918 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f48a40a25c0 0x7f48a40a4a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:10:03.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:03.800+0000 7f48bc889700 1 -- 172.21.15.46:0/4230555918 <== mon.2 v2:172.21.15.142:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8802+0+0 (secure 0 0 0) 0x7f48a00ced50 con 0x7f48b8101ae0 2024-03-31T23:10:03.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:03.800+0000 7f48bc889700 1 --2- 172.21.15.46:0/4230555918 >> [v2:172.21.15.142:6816/2146598446,v1:172.21.15.142:6817/2146598446] conn(0x7f48a40ad280 0x7f48a40af6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:10:03.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:03.800+0000 7f48bc889700 1 -- 172.21.15.46:0/4230555918 --> [v2:172.21.15.142:6816/2146598446,v1:172.21.15.142:6817/2146598446] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f48a40afd90 con 0x7f48a40ad280 2024-03-31T23:10:03.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:03.800+0000 7f48bc889700 1 -- 172.21.15.46:0/4230555918 <== mon.2 v2:172.21.15.142:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7f48a00cf100 con 0x7f48b8101ae0 2024-03-31T23:10:03.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:03.800+0000 7f48b6ffd700 1 --2- 172.21.15.46:0/4230555918 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f48a40a25c0 0x7f48a40a4a80 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f48b0009fd0 tx=0x7f48b000b700 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:10:03.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:03.800+0000 7f48b77fe700 1 --2- 172.21.15.46:0/4230555918 >> [v2:172.21.15.142:6816/2146598446,v1:172.21.15.142:6817/2146598446] conn(0x7f48a40ad280 0x7f48a40af6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:10:03.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:03.800+0000 7f48b77fe700 1 --2- 172.21.15.46:0/4230555918 >> [v2:172.21.15.142:6816/2146598446,v1:172.21.15.142:6817/2146598446] conn(0x7f48a40ad280 0x7f48a40af6c0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:10:03.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:03.801+0000 7f48bc889700 1 -- 172.21.15.46:0/4230555918 <== osd.7 v2:172.21.15.142:6816/2146598446 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f48a40afd90 con 0x7f48a40ad280 2024-03-31T23:10:03.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:03.818+0000 7f48bd88b700 1 -- 172.21.15.46:0/4230555918 --> [v2:172.21.15.142:6816/2146598446,v1:172.21.15.142:6817/2146598446] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f48b8102830 con 0x7f48a40ad280 2024-03-31T23:10:03.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:03.818+0000 7f48bc889700 1 -- 172.21.15.46:0/4230555918 <== osd.7 v2:172.21.15.142:6816/2146598446 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f48b8102830 con 0x7f48a40ad280 2024-03-31T23:10:03.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:03.819+0000 7f48bd88b700 1 -- 172.21.15.46:0/4230555918 >> [v2:172.21.15.142:6816/2146598446,v1:172.21.15.142:6817/2146598446] conn(0x7f48a40ad280 msgr2=0x7f48a40af6c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:10:03.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:03.819+0000 7f48bd88b700 1 --2- 172.21.15.46:0/4230555918 >> [v2:172.21.15.142:6816/2146598446,v1:172.21.15.142:6817/2146598446] conn(0x7f48a40ad280 0x7f48a40af6c0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:03.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:03.819+0000 7f48bd88b700 1 -- 172.21.15.46:0/4230555918 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f48a40a25c0 msgr2=0x7f48a40a4a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:10:03.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:03.819+0000 7f48bd88b700 1 --2- 172.21.15.46:0/4230555918 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f48a40a25c0 0x7f48a40a4a80 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f48b0009fd0 tx=0x7f48b000b700 comp rx=0 tx=0).stop 2024-03-31T23:10:03.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:03.819+0000 7f48bd88b700 1 -- 172.21.15.46:0/4230555918 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f48b8101ae0 msgr2=0x7f48b819eb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:10:03.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:03.819+0000 7f48bd88b700 1 --2- 172.21.15.46:0/4230555918 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f48b8101ae0 0x7f48b819eb40 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f48a000e9f0 tx=0x7f48a000ed00 comp rx=0 tx=0).stop 2024-03-31T23:10:03.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:03.819+0000 7f48b77fe700 1 -- 172.21.15.46:0/4230555918 reap_dead start 2024-03-31T23:10:03.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:03.819+0000 7f48bd88b700 1 -- 172.21.15.46:0/4230555918 shutdown_connections 2024-03-31T23:10:03.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:03.819+0000 7f48bd88b700 1 -- 172.21.15.46:0/4230555918 >> 172.21.15.46:0/4230555918 conn(0x7f48b8078070 msgr2=0x7f48b806a130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:10:03.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:03.819+0000 7f48bd88b700 1 -- 172.21.15.46:0/4230555918 shutdown_connections 2024-03-31T23:10:03.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:03.819+0000 7f48bd88b700 1 -- 172.21.15.46:0/4230555918 wait complete. 2024-03-31T23:10:04.172 INFO:teuthology.orchestra.run.smithi046.stdout:257698037797 2024-03-31T23:10:04.173 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:384f31f3d3aee7f8ba3f60846b8844c049165583 shell --fsid 6a9d8f08-efb2-11ee-b647-cb9ed24678a4 -- ceph osd last-stat-seq osd.8 2024-03-31T23:10:04.771 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:10:04 smithi046 ceph-mon[32498]: pgmap v318: 129 pgs: 129 active+clean; 642 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 255 B/s wr, 5 op/s 2024-03-31T23:10:04.795 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:10:04 smithi162 ceph-mon[36237]: pgmap v318: 129 pgs: 129 active+clean; 642 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 255 B/s wr, 5 op/s 2024-03-31T23:10:04.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:10:04 smithi142 ceph-mon[35271]: pgmap v318: 129 pgs: 129 active+clean; 642 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 255 B/s wr, 5 op/s 2024-03-31T23:10:05.014 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:05.012+0000 7faf32303700 1 -- 172.21.15.46:0/1852454667 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faf2c101ae0 msgr2=0x7faf2c10dfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:10:05.014 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:05.012+0000 7faf32303700 1 --2- 172.21.15.46:0/1852454667 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faf2c101ae0 0x7faf2c10dfc0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7faf2400b0a0 tx=0x7faf2400b3b0 comp rx=0 tx=0).stop 2024-03-31T23:10:05.014 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:05.012+0000 7faf32303700 1 -- 172.21.15.46:0/1852454667 shutdown_connections 2024-03-31T23:10:05.014 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:05.012+0000 7faf32303700 1 --2- 172.21.15.46:0/1852454667 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faf2c101ae0 0x7faf2c10dfc0 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:05.014 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:05.012+0000 7faf32303700 1 --2- 172.21.15.46:0/1852454667 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7faf2c107180 0x7faf2c107560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:05.014 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:05.012+0000 7faf32303700 1 --2- 172.21.15.46:0/1852454667 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7faf2c101140 0x7faf2c1015a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:05.014 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:05.012+0000 7faf32303700 1 -- 172.21.15.46:0/1852454667 >> 172.21.15.46:0/1852454667 conn(0x7faf2c078070 msgr2=0x7faf2c078480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:10:05.014 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:05.012+0000 7faf32303700 1 -- 172.21.15.46:0/1852454667 shutdown_connections 2024-03-31T23:10:05.014 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:05.012+0000 7faf32303700 1 -- 172.21.15.46:0/1852454667 wait complete. 2024-03-31T23:10:05.014 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:05.012+0000 7faf32303700 1 Processor -- start 2024-03-31T23:10:05.014 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:05.012+0000 7faf32303700 1 -- start start 2024-03-31T23:10:05.015 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:05.013+0000 7faf32303700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7faf2c101140 0x7faf2c1996e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:10:05.015 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:05.013+0000 7faf32303700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7faf2c101ae0 0x7faf2c199c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:10:05.015 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:05.013+0000 7faf32303700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faf2c107180 0x7faf2c1a4760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:10:05.015 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:05.013+0000 7faf32303700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7faf2c111280 con 0x7faf2c107180 2024-03-31T23:10:05.015 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:05.013+0000 7faf32303700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7faf2c111100 con 0x7faf2c101140 2024-03-31T23:10:05.015 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:05.013+0000 7faf32303700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7faf2c111400 con 0x7faf2c101ae0 2024-03-31T23:10:05.015 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:05.013+0000 7faf308a0700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faf2c107180 0x7faf2c1a4760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:10:05.015 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:05.013+0000 7faf2bfff700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7faf2c101140 0x7faf2c1996e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:10:05.015 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:05.013+0000 7faf308a0700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faf2c107180 0x7faf2c1a4760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:38348/0 (socket says 172.21.15.46:38348) 2024-03-31T23:10:05.015 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:05.013+0000 7faf308a0700 1 -- 172.21.15.46:0/2670206988 learned_addr learned my addr 172.21.15.46:0/2670206988 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:10:05.015 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:05.013+0000 7faf2b7fe700 1 --2- 172.21.15.46:0/2670206988 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7faf2c101ae0 0x7faf2c199c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:10:05.015 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:05.013+0000 7faf2bfff700 1 -- 172.21.15.46:0/2670206988 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7faf2c101ae0 msgr2=0x7faf2c199c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:10:05.015 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:05.013+0000 7faf2bfff700 1 --2- 172.21.15.46:0/2670206988 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7faf2c101ae0 0x7faf2c199c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:05.015 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:05.013+0000 7faf2bfff700 1 -- 172.21.15.46:0/2670206988 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faf2c107180 msgr2=0x7faf2c1a4760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:10:05.015 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:05.013+0000 7faf2bfff700 1 --2- 172.21.15.46:0/2670206988 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faf2c107180 0x7faf2c1a4760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:05.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:05.013+0000 7faf2bfff700 1 -- 172.21.15.46:0/2670206988 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faf2c1a4ca0 con 0x7faf2c101140 2024-03-31T23:10:05.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:05.013+0000 7faf308a0700 1 --2- 172.21.15.46:0/2670206988 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faf2c107180 0x7faf2c1a4760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:10:05.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:05.013+0000 7faf2b7fe700 1 --2- 172.21.15.46:0/2670206988 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7faf2c101ae0 0x7faf2c199c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:10:05.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:05.013+0000 7faf2bfff700 1 --2- 172.21.15.46:0/2670206988 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7faf2c101140 0x7faf2c1996e0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7faf1c010d00 tx=0x7faf1c00b5a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:10:05.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:05.014+0000 7faf297fa700 1 -- 172.21.15.46:0/2670206988 <== mon.2 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf1c010e50 con 0x7faf2c101140 2024-03-31T23:10:05.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:05.014+0000 7faf32303700 1 -- 172.21.15.46:0/2670206988 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faf2c197530 con 0x7faf2c101140 2024-03-31T23:10:05.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:05.014+0000 7faf297fa700 1 -- 172.21.15.46:0/2670206988 <== mon.2 v2:172.21.15.142:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7faf1c0044d0 con 0x7faf2c101140 2024-03-31T23:10:05.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:05.014+0000 7faf297fa700 1 -- 172.21.15.46:0/2670206988 <== mon.2 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf1c003e10 con 0x7faf2c101140 2024-03-31T23:10:05.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:05.014+0000 7faf32303700 1 -- 172.21.15.46:0/2670206988 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faf2c197ac0 con 0x7faf2c101140 2024-03-31T23:10:05.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:05.015+0000 7faf32303700 1 -- 172.21.15.46:0/2670206988 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7faf2c0628d0 con 0x7faf2c101140 2024-03-31T23:10:05.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:05.016+0000 7faf297fa700 1 -- 172.21.15.46:0/2670206988 <== mon.2 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 17) ==== 137371+0+0 (secure 0 0 0) 0x7faf1c003fb0 con 0x7faf2c101140 2024-03-31T23:10:05.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:05.016+0000 7faf297fa700 1 --2- 172.21.15.46:0/2670206988 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7faf140a2760 0x7faf140a4c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:10:05.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:05.016+0000 7faf2b7fe700 1 --2- 172.21.15.46:0/2670206988 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7faf140a2760 0x7faf140a4c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:10:05.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:05.017+0000 7faf297fa700 1 -- 172.21.15.46:0/2670206988 <== mon.2 v2:172.21.15.142:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8802+0+0 (secure 0 0 0) 0x7faf1c0d2940 con 0x7faf2c101140 2024-03-31T23:10:05.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:05.017+0000 7faf297fa700 1 --2- 172.21.15.46:0/2670206988 >> [v2:172.21.15.46:6802/1824772695,v1:172.21.15.46:6803/1824772695] conn(0x7faf140ad420 0x7faf140af860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:10:05.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:05.017+0000 7faf297fa700 1 -- 172.21.15.46:0/2670206988 --> [v2:172.21.15.46:6802/1824772695,v1:172.21.15.46:6803/1824772695] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7faf1c01de70 con 0x7faf140ad420 2024-03-31T23:10:05.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:05.017+0000 7faf297fa700 1 -- 172.21.15.46:0/2670206988 <== mon.2 v2:172.21.15.142:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7faf1c0d2d90 con 0x7faf2c101140 2024-03-31T23:10:05.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:05.017+0000 7faf308a0700 1 --2- 172.21.15.46:0/2670206988 >> [v2:172.21.15.46:6802/1824772695,v1:172.21.15.46:6803/1824772695] conn(0x7faf140ad420 0x7faf140af860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:10:05.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:05.017+0000 7faf2b7fe700 1 --2- 172.21.15.46:0/2670206988 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7faf140a2760 0x7faf140a4c20 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7faf2c078790 tx=0x7faf20009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:10:05.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:05.017+0000 7faf308a0700 1 --2- 172.21.15.46:0/2670206988 >> [v2:172.21.15.46:6802/1824772695,v1:172.21.15.46:6803/1824772695] conn(0x7faf140ad420 0x7faf140af860 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:10:05.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:05.018+0000 7faf297fa700 1 -- 172.21.15.46:0/2670206988 <== osd.0 v2:172.21.15.46:6802/1824772695 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7faf1c01de70 con 0x7faf140ad420 2024-03-31T23:10:05.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:05.034+0000 7faf32303700 1 -- 172.21.15.46:0/2670206988 --> [v2:172.21.15.46:6802/1824772695,v1:172.21.15.46:6803/1824772695] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7faf2c062d10 con 0x7faf140ad420 2024-03-31T23:10:05.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:05.035+0000 7faf297fa700 1 -- 172.21.15.46:0/2670206988 <== osd.0 v2:172.21.15.46:6802/1824772695 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7faf2c062d10 con 0x7faf140ad420 2024-03-31T23:10:05.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:05.035+0000 7faf32303700 1 -- 172.21.15.46:0/2670206988 >> [v2:172.21.15.46:6802/1824772695,v1:172.21.15.46:6803/1824772695] conn(0x7faf140ad420 msgr2=0x7faf140af860 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:10:05.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:05.035+0000 7faf32303700 1 --2- 172.21.15.46:0/2670206988 >> [v2:172.21.15.46:6802/1824772695,v1:172.21.15.46:6803/1824772695] conn(0x7faf140ad420 0x7faf140af860 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:05.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:05.035+0000 7faf32303700 1 -- 172.21.15.46:0/2670206988 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7faf140a2760 msgr2=0x7faf140a4c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:10:05.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:05.035+0000 7faf32303700 1 --2- 172.21.15.46:0/2670206988 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7faf140a2760 0x7faf140a4c20 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7faf2c078790 tx=0x7faf20009260 comp rx=0 tx=0).stop 2024-03-31T23:10:05.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:05.035+0000 7faf32303700 1 -- 172.21.15.46:0/2670206988 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7faf2c101140 msgr2=0x7faf2c1996e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:10:05.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:05.035+0000 7faf32303700 1 --2- 172.21.15.46:0/2670206988 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7faf2c101140 0x7faf2c1996e0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7faf1c010d00 tx=0x7faf1c00b5a0 comp rx=0 tx=0).stop 2024-03-31T23:10:05.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:05.035+0000 7faf308a0700 1 -- 172.21.15.46:0/2670206988 reap_dead start 2024-03-31T23:10:05.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:05.035+0000 7faf32303700 1 -- 172.21.15.46:0/2670206988 shutdown_connections 2024-03-31T23:10:05.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:05.035+0000 7faf32303700 1 -- 172.21.15.46:0/2670206988 >> 172.21.15.46:0/2670206988 conn(0x7faf2c078070 msgr2=0x7faf2c0fff70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:10:05.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:05.035+0000 7faf32303700 1 -- 172.21.15.46:0/2670206988 shutdown_connections 2024-03-31T23:10:05.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:05.035+0000 7faf32303700 1 -- 172.21.15.46:0/2670206988 wait complete. 2024-03-31T23:10:05.245 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:05.238+0000 7fcad51f2700 1 -- 172.21.15.46:0/2286931480 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcad006a9e0 msgr2=0x7fcad006ae60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:10:05.245 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:05.238+0000 7fcad51f2700 1 --2- 172.21.15.46:0/2286931480 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcad006a9e0 0x7fcad006ae60 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fcac4009990 tx=0x7fcac4009ca0 comp rx=0 tx=0).stop 2024-03-31T23:10:05.245 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:05.238+0000 7fcad51f2700 1 -- 172.21.15.46:0/2286931480 shutdown_connections 2024-03-31T23:10:05.245 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:05.238+0000 7fcad51f2700 1 --2- 172.21.15.46:0/2286931480 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcad006b3a0 0x7fcad010f270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:05.245 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:05.238+0000 7fcad51f2700 1 --2- 172.21.15.46:0/2286931480 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcad0104210 0x7fcad01045f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:05.245 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:05.238+0000 7fcad51f2700 1 --2- 172.21.15.46:0/2286931480 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcad006a9e0 0x7fcad006ae60 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:05.245 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:05.238+0000 7fcad51f2700 1 -- 172.21.15.46:0/2286931480 >> 172.21.15.46:0/2286931480 conn(0x7fcad0077190 msgr2=0x7fcad00775a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:10:05.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:05.238+0000 7fcad51f2700 1 -- 172.21.15.46:0/2286931480 shutdown_connections 2024-03-31T23:10:05.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:05.238+0000 7fcad51f2700 1 -- 172.21.15.46:0/2286931480 wait complete. 2024-03-31T23:10:05.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:05.239+0000 7fcad51f2700 1 Processor -- start 2024-03-31T23:10:05.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:05.239+0000 7fcad51f2700 1 -- start start 2024-03-31T23:10:05.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:05.239+0000 7fcad51f2700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcad006a9e0 0x7fcad01a0840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:10:05.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:05.239+0000 7fcad51f2700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcad006b3a0 0x7fcad01a0d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:10:05.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:05.239+0000 7fcad51f2700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcad0104210 0x7fcad019aa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:10:05.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:05.239+0000 7fcad51f2700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fcad0112530 con 0x7fcad006a9e0 2024-03-31T23:10:05.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:05.239+0000 7fcad51f2700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fcad01123b0 con 0x7fcad0104210 2024-03-31T23:10:05.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:05.239+0000 7fcad51f2700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fcad01126b0 con 0x7fcad006b3a0 2024-03-31T23:10:05.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:05.239+0000 7fcaced9d700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcad006a9e0 0x7fcad01a0840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:10:05.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:05.239+0000 7fcaced9d700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcad006a9e0 0x7fcad01a0840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:38364/0 (socket says 172.21.15.46:38364) 2024-03-31T23:10:05.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:05.239+0000 7fcaced9d700 1 -- 172.21.15.46:0/2635630230 learned_addr learned my addr 172.21.15.46:0/2635630230 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:10:05.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:05.239+0000 7fcacf59e700 1 --2- 172.21.15.46:0/2635630230 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcad0104210 0x7fcad019aa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:10:05.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:05.239+0000 7fcace59c700 1 --2- 172.21.15.46:0/2635630230 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcad006b3a0 0x7fcad01a0d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:10:05.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:05.240+0000 7fcaced9d700 1 -- 172.21.15.46:0/2635630230 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcad006b3a0 msgr2=0x7fcad01a0d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:10:05.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:05.240+0000 7fcaced9d700 1 --2- 172.21.15.46:0/2635630230 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcad006b3a0 0x7fcad01a0d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:05.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:05.240+0000 7fcaced9d700 1 -- 172.21.15.46:0/2635630230 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcad0104210 msgr2=0x7fcad019aa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:10:05.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:05.240+0000 7fcaced9d700 1 --2- 172.21.15.46:0/2635630230 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcad0104210 0x7fcad019aa00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:05.248 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:05.240+0000 7fcaced9d700 1 -- 172.21.15.46:0/2635630230 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcad019b160 con 0x7fcad006a9e0 2024-03-31T23:10:05.248 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:05.240+0000 7fcaced9d700 1 --2- 172.21.15.46:0/2635630230 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcad006a9e0 0x7fcad01a0840 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fcac000eb30 tx=0x7fcac000ee40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:10:05.248 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:05.240+0000 7fcab7fff700 1 -- 172.21.15.46:0/2635630230 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcac000cbd0 con 0x7fcad006a9e0 2024-03-31T23:10:05.248 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:05.240+0000 7fcad51f2700 1 -- 172.21.15.46:0/2635630230 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcad019b450 con 0x7fcad006a9e0 2024-03-31T23:10:05.248 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:05.240+0000 7fcab7fff700 1 -- 172.21.15.46:0/2635630230 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fcac00044d0 con 0x7fcad006a9e0 2024-03-31T23:10:05.248 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:05.240+0000 7fcab7fff700 1 -- 172.21.15.46:0/2635630230 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcac0003e30 con 0x7fcad006a9e0 2024-03-31T23:10:05.248 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:05.240+0000 7fcad51f2700 1 -- 172.21.15.46:0/2635630230 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcad01a7710 con 0x7fcad006a9e0 2024-03-31T23:10:05.248 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:05.241+0000 7fcad51f2700 1 -- 172.21.15.46:0/2635630230 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fcad0062950 con 0x7fcad006a9e0 2024-03-31T23:10:05.248 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:05.242+0000 7fcab7fff700 1 -- 172.21.15.46:0/2635630230 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 17) ==== 137371+0+0 (secure 0 0 0) 0x7fcac0004000 con 0x7fcad006a9e0 2024-03-31T23:10:05.248 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:05.243+0000 7fcab7fff700 1 --2- 172.21.15.46:0/2635630230 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fcab80a25c0 0x7fcab80a4a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:10:05.248 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:05.243+0000 7fcace59c700 1 --2- 172.21.15.46:0/2635630230 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fcab80a25c0 0x7fcab80a4a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:10:05.248 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:05.243+0000 7fcab7fff700 1 -- 172.21.15.46:0/2635630230 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8802+0+0 (secure 0 0 0) 0x7fcac00cf4d0 con 0x7fcad006a9e0 2024-03-31T23:10:05.249 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:05.243+0000 7fcab7fff700 1 --2- 172.21.15.46:0/2635630230 >> [v2:172.21.15.162:6800/4084261813,v1:172.21.15.162:6801/4084261813] conn(0x7fcab80ad280 0x7fcab80af6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:10:05.249 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:05.243+0000 7fcab7fff700 1 -- 172.21.15.46:0/2635630230 --> [v2:172.21.15.162:6800/4084261813,v1:172.21.15.162:6801/4084261813] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fcac001aec0 con 0x7fcab80ad280 2024-03-31T23:10:05.249 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:05.243+0000 7fcab7fff700 1 -- 172.21.15.46:0/2635630230 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7fcac00cf920 con 0x7fcad006a9e0 2024-03-31T23:10:05.249 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:05.243+0000 7fcacf59e700 1 --2- 172.21.15.46:0/2635630230 >> [v2:172.21.15.162:6800/4084261813,v1:172.21.15.162:6801/4084261813] conn(0x7fcab80ad280 0x7fcab80af6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:10:05.249 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:05.243+0000 7fcace59c700 1 --2- 172.21.15.46:0/2635630230 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fcab80a25c0 0x7fcab80a4a80 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fcac40098e0 tx=0x7fcac4005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:10:05.249 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:05.244+0000 7fcacf59e700 1 --2- 172.21.15.46:0/2635630230 >> [v2:172.21.15.162:6800/4084261813,v1:172.21.15.162:6801/4084261813] conn(0x7fcab80ad280 0x7fcab80af6c0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:10:05.249 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:05.244+0000 7fcab7fff700 1 -- 172.21.15.46:0/2635630230 <== osd.2 v2:172.21.15.162:6800/4084261813 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fcac001aec0 con 0x7fcab80ad280 2024-03-31T23:10:05.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:05.261+0000 7fcad51f2700 1 -- 172.21.15.46:0/2635630230 --> [v2:172.21.15.162:6800/4084261813,v1:172.21.15.162:6801/4084261813] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fcad0062dd0 con 0x7fcab80ad280 2024-03-31T23:10:05.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:05.261+0000 7fcab7fff700 1 -- 172.21.15.46:0/2635630230 <== osd.2 v2:172.21.15.162:6800/4084261813 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fcad0062dd0 con 0x7fcab80ad280 2024-03-31T23:10:05.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:05.261+0000 7fcad51f2700 1 -- 172.21.15.46:0/2635630230 >> [v2:172.21.15.162:6800/4084261813,v1:172.21.15.162:6801/4084261813] conn(0x7fcab80ad280 msgr2=0x7fcab80af6c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:10:05.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:05.261+0000 7fcad51f2700 1 --2- 172.21.15.46:0/2635630230 >> [v2:172.21.15.162:6800/4084261813,v1:172.21.15.162:6801/4084261813] conn(0x7fcab80ad280 0x7fcab80af6c0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:05.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:05.261+0000 7fcad51f2700 1 -- 172.21.15.46:0/2635630230 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fcab80a25c0 msgr2=0x7fcab80a4a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:10:05.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:05.261+0000 7fcad51f2700 1 --2- 172.21.15.46:0/2635630230 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fcab80a25c0 0x7fcab80a4a80 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fcac40098e0 tx=0x7fcac4005960 comp rx=0 tx=0).stop 2024-03-31T23:10:05.263 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:05.262+0000 7fcad51f2700 1 -- 172.21.15.46:0/2635630230 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcad006a9e0 msgr2=0x7fcad01a0840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:10:05.263 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:05.262+0000 7fcad51f2700 1 --2- 172.21.15.46:0/2635630230 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcad006a9e0 0x7fcad01a0840 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fcac000eb30 tx=0x7fcac000ee40 comp rx=0 tx=0).stop 2024-03-31T23:10:05.263 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:05.262+0000 7fcacf59e700 1 -- 172.21.15.46:0/2635630230 reap_dead start 2024-03-31T23:10:05.263 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:05.262+0000 7fcad51f2700 1 -- 172.21.15.46:0/2635630230 shutdown_connections 2024-03-31T23:10:05.263 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:05.262+0000 7fcad51f2700 1 -- 172.21.15.46:0/2635630230 >> 172.21.15.46:0/2635630230 conn(0x7fcad0077190 msgr2=0x7fcad010d7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:10:05.263 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:05.262+0000 7fcad51f2700 1 -- 172.21.15.46:0/2635630230 shutdown_connections 2024-03-31T23:10:05.263 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:05.262+0000 7fcad51f2700 1 -- 172.21.15.46:0/2635630230 wait complete. 2024-03-31T23:10:06.364 INFO:teuthology.orchestra.run.smithi046.stdout:55834574920 2024-03-31T23:10:06.364 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:384f31f3d3aee7f8ba3f60846b8844c049165583 shell --fsid 6a9d8f08-efb2-11ee-b647-cb9ed24678a4 -- ceph osd last-stat-seq osd.1 2024-03-31T23:10:06.611 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:10:06 smithi046 ceph-mon[32498]: pgmap v319: 129 pgs: 129 active+clean; 642 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 255 B/s wr, 5 op/s 2024-03-31T23:10:06.795 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:10:06 smithi162 ceph-mon[36237]: pgmap v319: 129 pgs: 129 active+clean; 642 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 255 B/s wr, 5 op/s 2024-03-31T23:10:06.899 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:06.898+0000 7fc8a0c5e700 1 -- 172.21.15.46:0/191255444 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc89c101de0 msgr2=0x7fc89c10e000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:10:06.899 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:06.898+0000 7fc8a0c5e700 1 --2- 172.21.15.46:0/191255444 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc89c101de0 0x7fc89c10e000 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fc89400b0a0 tx=0x7fc89400b3b0 comp rx=0 tx=0).stop 2024-03-31T23:10:06.899 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:06.898+0000 7fc8a0c5e700 1 -- 172.21.15.46:0/191255444 shutdown_connections 2024-03-31T23:10:06.899 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:06.898+0000 7fc8a0c5e700 1 --2- 172.21.15.46:0/191255444 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc89c101de0 0x7fc89c10e000 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:06.899 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:06.898+0000 7fc8a0c5e700 1 --2- 172.21.15.46:0/191255444 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc89c107270 0x7fc89c107650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:06.899 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:06.898+0000 7fc8a0c5e700 1 --2- 172.21.15.46:0/191255444 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc89c101420 0x7fc89c1018a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:06.899 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:06.898+0000 7fc8a0c5e700 1 -- 172.21.15.46:0/191255444 >> 172.21.15.46:0/191255444 conn(0x7fc89c078000 msgr2=0x7fc89c078410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:10:06.899 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:06.898+0000 7fc8a0c5e700 1 -- 172.21.15.46:0/191255444 shutdown_connections 2024-03-31T23:10:06.899 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:06.898+0000 7fc8a0c5e700 1 -- 172.21.15.46:0/191255444 wait complete. 2024-03-31T23:10:06.901 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:06.899+0000 7fc8a0c5e700 1 Processor -- start 2024-03-31T23:10:06.901 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:06.899+0000 7fc8a0c5e700 1 -- start start 2024-03-31T23:10:06.901 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:06.899+0000 7fc8a0c5e700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc89c101420 0x7fc89c10e910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:10:06.901 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:06.899+0000 7fc8a0c5e700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc89c101de0 0x7fc89c10ee50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:10:06.901 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:06.899+0000 7fc8a0c5e700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc89c107270 0x7fc89c10f390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:10:06.901 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:06.899+0000 7fc8a0c5e700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fc89c1124e0 con 0x7fc89c101420 2024-03-31T23:10:06.901 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:06.899+0000 7fc8a0c5e700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fc89c112360 con 0x7fc89c107270 2024-03-31T23:10:06.901 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:06.899+0000 7fc8a0c5e700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fc89c112660 con 0x7fc89c101de0 2024-03-31T23:10:06.902 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:06.900+0000 7fc89ad9d700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc89c107270 0x7fc89c10f390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:10:06.902 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:06.900+0000 7fc89ad9d700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc89c107270 0x7fc89c10f390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.46:58168/0 (socket says 172.21.15.46:58168) 2024-03-31T23:10:06.902 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:06.900+0000 7fc89ad9d700 1 -- 172.21.15.46:0/3879857905 learned_addr learned my addr 172.21.15.46:0/3879857905 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:10:06.902 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:06.900+0000 7fc899d9b700 1 --2- 172.21.15.46:0/3879857905 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc89c101de0 0x7fc89c10ee50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:10:06.902 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:06.900+0000 7fc89a59c700 1 --2- 172.21.15.46:0/3879857905 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc89c101420 0x7fc89c10e910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:10:06.902 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:06.900+0000 7fc89ad9d700 1 -- 172.21.15.46:0/3879857905 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc89c101de0 msgr2=0x7fc89c10ee50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:10:06.902 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:06.900+0000 7fc89ad9d700 1 --2- 172.21.15.46:0/3879857905 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc89c101de0 0x7fc89c10ee50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:06.902 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:06.900+0000 7fc89ad9d700 1 -- 172.21.15.46:0/3879857905 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc89c101420 msgr2=0x7fc89c10e910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:10:06.902 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:06.900+0000 7fc89ad9d700 1 --2- 172.21.15.46:0/3879857905 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc89c101420 0x7fc89c10e910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:06.902 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:06.900+0000 7fc89ad9d700 1 -- 172.21.15.46:0/3879857905 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc89c1a6ee0 con 0x7fc89c107270 2024-03-31T23:10:06.902 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:06.900+0000 7fc89ad9d700 1 --2- 172.21.15.46:0/3879857905 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc89c107270 0x7fc89c10f390 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fc89400b070 tx=0x7fc894004c80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:10:06.902 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:06.900+0000 7fc8937fe700 1 -- 172.21.15.46:0/3879857905 <== mon.2 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc894021070 con 0x7fc89c107270 2024-03-31T23:10:06.902 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:06.900+0000 7fc8a0c5e700 1 -- 172.21.15.46:0/3879857905 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc89c1a7170 con 0x7fc89c107270 2024-03-31T23:10:06.903 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:06.900+0000 7fc8a0c5e700 1 -- 172.21.15.46:0/3879857905 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc89c1a7670 con 0x7fc89c107270 2024-03-31T23:10:06.903 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:06.900+0000 7fc8937fe700 1 -- 172.21.15.46:0/3879857905 <== mon.2 v2:172.21.15.142:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc894007e30 con 0x7fc89c107270 2024-03-31T23:10:06.903 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:06.900+0000 7fc8937fe700 1 -- 172.21.15.46:0/3879857905 <== mon.2 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc894004400 con 0x7fc89c107270 2024-03-31T23:10:06.903 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:06.901+0000 7fc8a0c5e700 1 -- 172.21.15.46:0/3879857905 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fc89c062950 con 0x7fc89c107270 2024-03-31T23:10:06.904 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:06.902+0000 7fc8937fe700 1 -- 172.21.15.46:0/3879857905 <== mon.2 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 17) ==== 137371+0+0 (secure 0 0 0) 0x7fc894003d00 con 0x7fc89c107270 2024-03-31T23:10:06.904 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:06.903+0000 7fc8937fe700 1 --2- 172.21.15.46:0/3879857905 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fc8880a25c0 0x7fc8880a4a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:10:06.904 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:06.903+0000 7fc89a59c700 1 --2- 172.21.15.46:0/3879857905 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fc8880a25c0 0x7fc8880a4a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:10:06.904 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:06.903+0000 7fc8937fe700 1 -- 172.21.15.46:0/3879857905 <== mon.2 v2:172.21.15.142:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8802+0+0 (secure 0 0 0) 0x7fc8940cfd20 con 0x7fc89c107270 2024-03-31T23:10:06.904 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:06.903+0000 7fc8937fe700 1 --2- 172.21.15.46:0/3879857905 >> [v2:172.21.15.142:6808/2996266929,v1:172.21.15.142:6809/2996266929] conn(0x7fc8880ad280 0x7fc8880af6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:10:06.904 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:06.903+0000 7fc8937fe700 1 -- 172.21.15.46:0/3879857905 --> [v2:172.21.15.142:6808/2996266929,v1:172.21.15.142:6809/2996266929] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fc8940020b0 con 0x7fc8880ad280 2024-03-31T23:10:06.904 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:06.903+0000 7fc8937fe700 1 -- 172.21.15.46:0/3879857905 <== mon.2 v2:172.21.15.142:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7fc8940d00d0 con 0x7fc89c107270 2024-03-31T23:10:06.904 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:06.904+0000 7fc899d9b700 1 --2- 172.21.15.46:0/3879857905 >> [v2:172.21.15.142:6808/2996266929,v1:172.21.15.142:6809/2996266929] conn(0x7fc8880ad280 0x7fc8880af6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:10:06.905 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:06.904+0000 7fc89a59c700 1 --2- 172.21.15.46:0/3879857905 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fc8880a25c0 0x7fc8880a4a80 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fc88c009bd0 tx=0x7fc88c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:10:06.905 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:06.904+0000 7fc899d9b700 1 --2- 172.21.15.46:0/3879857905 >> [v2:172.21.15.142:6808/2996266929,v1:172.21.15.142:6809/2996266929] conn(0x7fc8880ad280 0x7fc8880af6c0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:10:06.905 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:06.905+0000 7fc8937fe700 1 -- 172.21.15.46:0/3879857905 <== osd.4 v2:172.21.15.142:6808/2996266929 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fc8940020b0 con 0x7fc8880ad280 2024-03-31T23:10:06.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:06.921+0000 7fc8a0c5e700 1 -- 172.21.15.46:0/3879857905 --> [v2:172.21.15.142:6808/2996266929,v1:172.21.15.142:6809/2996266929] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fc89c062d50 con 0x7fc8880ad280 2024-03-31T23:10:06.921 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:06.921+0000 7fc8937fe700 1 -- 172.21.15.46:0/3879857905 <== osd.4 v2:172.21.15.142:6808/2996266929 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fc89c062d50 con 0x7fc8880ad280 2024-03-31T23:10:06.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:06.921+0000 7fc8a0c5e700 1 -- 172.21.15.46:0/3879857905 >> [v2:172.21.15.142:6808/2996266929,v1:172.21.15.142:6809/2996266929] conn(0x7fc8880ad280 msgr2=0x7fc8880af6c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:10:06.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:06.921+0000 7fc8a0c5e700 1 --2- 172.21.15.46:0/3879857905 >> [v2:172.21.15.142:6808/2996266929,v1:172.21.15.142:6809/2996266929] conn(0x7fc8880ad280 0x7fc8880af6c0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:06.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:06.921+0000 7fc8a0c5e700 1 -- 172.21.15.46:0/3879857905 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fc8880a25c0 msgr2=0x7fc8880a4a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:10:06.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:06.921+0000 7fc8a0c5e700 1 --2- 172.21.15.46:0/3879857905 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fc8880a25c0 0x7fc8880a4a80 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fc88c009bd0 tx=0x7fc88c009260 comp rx=0 tx=0).stop 2024-03-31T23:10:06.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:06.921+0000 7fc8a0c5e700 1 -- 172.21.15.46:0/3879857905 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc89c107270 msgr2=0x7fc89c10f390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:10:06.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:06.921+0000 7fc8a0c5e700 1 --2- 172.21.15.46:0/3879857905 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc89c107270 0x7fc89c10f390 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fc89400b070 tx=0x7fc894004c80 comp rx=0 tx=0).stop 2024-03-31T23:10:06.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:06.921+0000 7fc89ad9d700 1 -- 172.21.15.46:0/3879857905 reap_dead start 2024-03-31T23:10:06.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:06.922+0000 7fc8a0c5e700 1 -- 172.21.15.46:0/3879857905 shutdown_connections 2024-03-31T23:10:06.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:06.922+0000 7fc8a0c5e700 1 -- 172.21.15.46:0/3879857905 >> 172.21.15.46:0/3879857905 conn(0x7fc89c078000 msgr2=0x7fc89c100110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:10:06.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:06.922+0000 7fc8a0c5e700 1 -- 172.21.15.46:0/3879857905 shutdown_connections 2024-03-31T23:10:06.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:06.922+0000 7fc8a0c5e700 1 -- 172.21.15.46:0/3879857905 wait complete. 2024-03-31T23:10:06.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:10:06 smithi142 ceph-mon[35271]: pgmap v319: 129 pgs: 129 active+clean; 642 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 255 B/s wr, 5 op/s 2024-03-31T23:10:07.899 INFO:teuthology.orchestra.run.smithi046.stdout:223338299436 2024-03-31T23:10:07.900 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:384f31f3d3aee7f8ba3f60846b8844c049165583 shell --fsid 6a9d8f08-efb2-11ee-b647-cb9ed24678a4 -- ceph osd last-stat-seq osd.7 2024-03-31T23:10:08.675 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:10:08 smithi046 ceph-mon[32498]: pgmap v320: 129 pgs: 129 active+clean; 642 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 255 B/s wr, 6 op/s 2024-03-31T23:10:08.702 INFO:teuthology.orchestra.run.smithi046.stdout:141733920828 2024-03-31T23:10:08.702 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:384f31f3d3aee7f8ba3f60846b8844c049165583 shell --fsid 6a9d8f08-efb2-11ee-b647-cb9ed24678a4 -- ceph osd last-stat-seq osd.4 2024-03-31T23:10:08.795 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:10:08 smithi162 ceph-mon[36237]: pgmap v320: 129 pgs: 129 active+clean; 642 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 255 B/s wr, 6 op/s 2024-03-31T23:10:08.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:08.884+0000 7f0e56974700 1 -- 172.21.15.46:0/4025495843 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0e5006a9e0 msgr2=0x7f0e5006adc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:10:08.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:08.884+0000 7f0e56974700 1 --2- 172.21.15.46:0/4025495843 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0e5006a9e0 0x7f0e5006adc0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f0e400099c0 tx=0x7f0e40009cd0 comp rx=0 tx=0).stop 2024-03-31T23:10:08.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:08.885+0000 7f0e56974700 1 -- 172.21.15.46:0/4025495843 shutdown_connections 2024-03-31T23:10:08.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:08.885+0000 7f0e56974700 1 --2- 172.21.15.46:0/4025495843 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0e5006a9e0 0x7f0e5006adc0 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:08.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:08.885+0000 7f0e56974700 1 --2- 172.21.15.46:0/4025495843 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0e50107520 0x7f0e5010f0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:08.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:08.885+0000 7f0e56974700 1 --2- 172.21.15.46:0/4025495843 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0e5006b390 0x7f0e50106e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:08.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:08.885+0000 7f0e56974700 1 -- 172.21.15.46:0/4025495843 >> 172.21.15.46:0/4025495843 conn(0x7f0e50077190 msgr2=0x7f0e500775a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:10:08.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:08.885+0000 7f0e56974700 1 -- 172.21.15.46:0/4025495843 shutdown_connections 2024-03-31T23:10:08.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:08.885+0000 7f0e56974700 1 -- 172.21.15.46:0/4025495843 wait complete. 2024-03-31T23:10:08.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:08.886+0000 7f0e56974700 1 Processor -- start 2024-03-31T23:10:08.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:08.886+0000 7f0e56974700 1 -- start start 2024-03-31T23:10:08.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:08.886+0000 7f0e56974700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0e5006a9e0 0x7f0e5019e640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:10:08.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:08.886+0000 7f0e56974700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0e5006b390 0x7f0e5019eb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:10:08.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:08.886+0000 7f0e56974700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0e50107520 0x7f0e50198710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:10:08.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:08.886+0000 7f0e56974700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f0e501123d0 con 0x7f0e50107520 2024-03-31T23:10:08.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:08.886+0000 7f0e56974700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f0e50112250 con 0x7f0e5006a9e0 2024-03-31T23:10:08.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:08.886+0000 7f0e56974700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f0e50112550 con 0x7f0e5006b390 2024-03-31T23:10:08.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:08.886+0000 7f0e4ffff700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0e5006a9e0 0x7f0e5019e640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:10:08.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:08.886+0000 7f0e54f11700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0e50107520 0x7f0e50198710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:10:08.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:08.886+0000 7f0e4ffff700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0e5006a9e0 0x7f0e5019e640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.46:58192/0 (socket says 172.21.15.46:58192) 2024-03-31T23:10:08.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:08.886+0000 7f0e54f11700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0e50107520 0x7f0e50198710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:38406/0 (socket says 172.21.15.46:38406) 2024-03-31T23:10:08.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:08.886+0000 7f0e4ffff700 1 -- 172.21.15.46:0/1394310700 learned_addr learned my addr 172.21.15.46:0/1394310700 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:10:08.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:08.886+0000 7f0e4f7fe700 1 --2- 172.21.15.46:0/1394310700 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0e5006b390 0x7f0e5019eb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:10:08.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:08.887+0000 7f0e4ffff700 1 -- 172.21.15.46:0/1394310700 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0e5006b390 msgr2=0x7f0e5019eb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:10:08.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:08.887+0000 7f0e4ffff700 1 --2- 172.21.15.46:0/1394310700 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0e5006b390 0x7f0e5019eb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:08.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:08.887+0000 7f0e4ffff700 1 -- 172.21.15.46:0/1394310700 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0e50107520 msgr2=0x7f0e50198710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:10:08.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:08.887+0000 7f0e4ffff700 1 --2- 172.21.15.46:0/1394310700 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0e50107520 0x7f0e50198710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:08.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:08.887+0000 7f0e4ffff700 1 -- 172.21.15.46:0/1394310700 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0e50198ff0 con 0x7f0e5006a9e0 2024-03-31T23:10:08.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:08.887+0000 7f0e4ffff700 1 --2- 172.21.15.46:0/1394310700 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0e5006a9e0 0x7f0e5019e640 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f0e40006010 tx=0x7f0e40004270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:10:08.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:08.887+0000 7f0e4d7fa700 1 -- 172.21.15.46:0/1394310700 <== mon.2 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e40026070 con 0x7f0e5006a9e0 2024-03-31T23:10:08.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:08.887+0000 7f0e56974700 1 -- 172.21.15.46:0/1394310700 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0e50199280 con 0x7f0e5006a9e0 2024-03-31T23:10:08.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:08.887+0000 7f0e4d7fa700 1 -- 172.21.15.46:0/1394310700 <== mon.2 v2:172.21.15.142:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0e40004630 con 0x7f0e5006a9e0 2024-03-31T23:10:08.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:08.887+0000 7f0e4d7fa700 1 -- 172.21.15.46:0/1394310700 <== mon.2 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e4000fb20 con 0x7f0e5006a9e0 2024-03-31T23:10:08.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:08.887+0000 7f0e56974700 1 -- 172.21.15.46:0/1394310700 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0e501a55c0 con 0x7f0e5006a9e0 2024-03-31T23:10:08.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:08.888+0000 7f0e56974700 1 -- 172.21.15.46:0/1394310700 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f0e50062990 con 0x7f0e5006a9e0 2024-03-31T23:10:08.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:08.889+0000 7f0e4d7fa700 1 -- 172.21.15.46:0/1394310700 <== mon.2 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 17) ==== 137371+0+0 (secure 0 0 0) 0x7f0e40029070 con 0x7f0e5006a9e0 2024-03-31T23:10:08.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:08.890+0000 7f0e4d7fa700 1 --2- 172.21.15.46:0/1394310700 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f0e380a2640 0x7f0e380a4b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:10:08.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:08.890+0000 7f0e4d7fa700 1 -- 172.21.15.46:0/1394310700 <== mon.2 v2:172.21.15.142:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8802+0+0 (secure 0 0 0) 0x7f0e400cf510 con 0x7f0e5006a9e0 2024-03-31T23:10:08.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:08.890+0000 7f0e4f7fe700 1 --2- 172.21.15.46:0/1394310700 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f0e380a2640 0x7f0e380a4b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:10:08.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:08.890+0000 7f0e4d7fa700 1 --2- 172.21.15.46:0/1394310700 >> [v2:172.21.15.142:6824/2845222366,v1:172.21.15.142:6825/2845222366] conn(0x7f0e380ad300 0x7f0e380af740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:10:08.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:08.890+0000 7f0e4d7fa700 1 -- 172.21.15.46:0/1394310700 --> [v2:172.21.15.142:6824/2845222366,v1:172.21.15.142:6825/2845222366] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f0e40001d30 con 0x7f0e380ad300 2024-03-31T23:10:08.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:08.890+0000 7f0e4d7fa700 1 -- 172.21.15.46:0/1394310700 <== mon.2 v2:172.21.15.142:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7f0e400cf8c0 con 0x7f0e5006a9e0 2024-03-31T23:10:08.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:08.890+0000 7f0e54f11700 1 --2- 172.21.15.46:0/1394310700 >> [v2:172.21.15.142:6824/2845222366,v1:172.21.15.142:6825/2845222366] conn(0x7f0e380ad300 0x7f0e380af740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:10:08.892 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:08.891+0000 7f0e4f7fe700 1 --2- 172.21.15.46:0/1394310700 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f0e380a2640 0x7f0e380a4b00 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f0e50199ed0 tx=0x7f0e44009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:10:08.892 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:08.891+0000 7f0e54f11700 1 --2- 172.21.15.46:0/1394310700 >> [v2:172.21.15.142:6824/2845222366,v1:172.21.15.142:6825/2845222366] conn(0x7f0e380ad300 0x7f0e380af740 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.10 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:10:08.892 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:08.892+0000 7f0e4d7fa700 1 -- 172.21.15.46:0/1394310700 <== osd.10 v2:172.21.15.142:6824/2845222366 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f0e40001d30 con 0x7f0e380ad300 2024-03-31T23:10:08.915 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:08.915+0000 7f0e56974700 1 -- 172.21.15.46:0/1394310700 --> [v2:172.21.15.142:6824/2845222366,v1:172.21.15.142:6825/2845222366] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f0e50062d90 con 0x7f0e380ad300 2024-03-31T23:10:08.915 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:08.915+0000 7f0e4d7fa700 1 -- 172.21.15.46:0/1394310700 <== osd.10 v2:172.21.15.142:6824/2845222366 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f0e50062d90 con 0x7f0e380ad300 2024-03-31T23:10:08.916 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:08.916+0000 7f0e56974700 1 -- 172.21.15.46:0/1394310700 >> [v2:172.21.15.142:6824/2845222366,v1:172.21.15.142:6825/2845222366] conn(0x7f0e380ad300 msgr2=0x7f0e380af740 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:10:08.916 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:08.916+0000 7f0e56974700 1 --2- 172.21.15.46:0/1394310700 >> [v2:172.21.15.142:6824/2845222366,v1:172.21.15.142:6825/2845222366] conn(0x7f0e380ad300 0x7f0e380af740 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:08.917 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:08.916+0000 7f0e56974700 1 -- 172.21.15.46:0/1394310700 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f0e380a2640 msgr2=0x7f0e380a4b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:10:08.917 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:08.916+0000 7f0e56974700 1 --2- 172.21.15.46:0/1394310700 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f0e380a2640 0x7f0e380a4b00 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f0e50199ed0 tx=0x7f0e44009260 comp rx=0 tx=0).stop 2024-03-31T23:10:08.917 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:08.916+0000 7f0e56974700 1 -- 172.21.15.46:0/1394310700 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0e5006a9e0 msgr2=0x7f0e5019e640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:10:08.917 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:08.916+0000 7f0e56974700 1 --2- 172.21.15.46:0/1394310700 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0e5006a9e0 0x7f0e5019e640 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f0e40006010 tx=0x7f0e40004270 comp rx=0 tx=0).stop 2024-03-31T23:10:08.917 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:08.916+0000 7f0e54f11700 1 -- 172.21.15.46:0/1394310700 reap_dead start 2024-03-31T23:10:08.917 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:08.916+0000 7f0e56974700 1 -- 172.21.15.46:0/1394310700 shutdown_connections 2024-03-31T23:10:08.917 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:08.916+0000 7f0e56974700 1 -- 172.21.15.46:0/1394310700 >> 172.21.15.46:0/1394310700 conn(0x7f0e50077190 msgr2=0x7f0e501061c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:10:08.917 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:08.916+0000 7f0e56974700 1 -- 172.21.15.46:0/1394310700 shutdown_connections 2024-03-31T23:10:08.917 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:08.916+0000 7f0e56974700 1 -- 172.21.15.46:0/1394310700 wait complete. 2024-03-31T23:10:08.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:10:08 smithi142 ceph-mon[35271]: pgmap v320: 129 pgs: 129 active+clean; 642 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 255 B/s wr, 6 op/s 2024-03-31T23:10:09.480 INFO:teuthology.orchestra.run.smithi046.stdout:107374182464 2024-03-31T23:10:09.480 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:384f31f3d3aee7f8ba3f60846b8844c049165583 shell --fsid 6a9d8f08-efb2-11ee-b647-cb9ed24678a4 -- ceph osd last-stat-seq osd.3 2024-03-31T23:10:09.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:10:09 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.6", "id": [4, 7]}]: dispatch 2024-03-31T23:10:09.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:10:09 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.c", "id": [3, 9]}]: dispatch 2024-03-31T23:10:09.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:10:09 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1a", "id": [5, 6]}]: dispatch 2024-03-31T23:10:09.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:10:09 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [4, 8]}]: dispatch 2024-03-31T23:10:09.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:10:09 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1d", "id": [4, 8]}]: dispatch 2024-03-31T23:10:09.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:10:09 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2a", "id": [3, 6]}]: dispatch 2024-03-31T23:10:09.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:10:09 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.33", "id": [5, 9]}]: dispatch 2024-03-31T23:10:09.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:10:09 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.36", "id": [4, 9]}]: dispatch 2024-03-31T23:10:09.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:10:09 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [9, 8]}]: dispatch 2024-03-31T23:10:09.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:10:09 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [9, 8]}]: dispatch 2024-03-31T23:10:09.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:10:09 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.6", "id": [4, 7]}]: dispatch 2024-03-31T23:10:09.862 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:10:09 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.c", "id": [3, 9]}]: dispatch 2024-03-31T23:10:09.862 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:10:09 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1a", "id": [5, 6]}]: dispatch 2024-03-31T23:10:09.862 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:10:09 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [4, 8]}]: dispatch 2024-03-31T23:10:09.862 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:10:09 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1d", "id": [4, 8]}]: dispatch 2024-03-31T23:10:09.862 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:10:09 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2a", "id": [3, 6]}]: dispatch 2024-03-31T23:10:09.862 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:10:09 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.33", "id": [5, 9]}]: dispatch 2024-03-31T23:10:09.862 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:10:09 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.36", "id": [4, 9]}]: dispatch 2024-03-31T23:10:09.862 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:10:09 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [9, 8]}]: dispatch 2024-03-31T23:10:09.862 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:10:09 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [9, 8]}]: dispatch 2024-03-31T23:10:09.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:10:09 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.6", "id": [4, 7]}]: dispatch 2024-03-31T23:10:09.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:10:09 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.c", "id": [3, 9]}]: dispatch 2024-03-31T23:10:09.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:10:09 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1a", "id": [5, 6]}]: dispatch 2024-03-31T23:10:09.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:10:09 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [4, 8]}]: dispatch 2024-03-31T23:10:09.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:10:09 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1d", "id": [4, 8]}]: dispatch 2024-03-31T23:10:09.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:10:09 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2a", "id": [3, 6]}]: dispatch 2024-03-31T23:10:09.944 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:10:09 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.33", "id": [5, 9]}]: dispatch 2024-03-31T23:10:09.944 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:10:09 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.36", "id": [4, 9]}]: dispatch 2024-03-31T23:10:09.944 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:10:09 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [9, 8]}]: dispatch 2024-03-31T23:10:09.944 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:10:09 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [9, 8]}]: dispatch 2024-03-31T23:10:10.555 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:10:10 smithi046 ceph-mon[32498]: pgmap v321: 129 pgs: 129 active+clean; 642 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 255 B/s wr, 6 op/s 2024-03-31T23:10:10.556 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:10:10 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.6", "id": [4, 7]}]': finished 2024-03-31T23:10:10.556 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:10:10 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.c", "id": [3, 9]}]': finished 2024-03-31T23:10:10.556 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:10:10 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1a", "id": [5, 6]}]': finished 2024-03-31T23:10:10.556 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:10:10 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [4, 8]}]': finished 2024-03-31T23:10:10.556 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:10:10 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1d", "id": [4, 8]}]': finished 2024-03-31T23:10:10.556 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:10:10 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2a", "id": [3, 6]}]': finished 2024-03-31T23:10:10.556 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:10:10 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.33", "id": [5, 9]}]': finished 2024-03-31T23:10:10.556 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:10:10 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.36", "id": [4, 9]}]': finished 2024-03-31T23:10:10.556 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:10:10 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [9, 8]}]': finished 2024-03-31T23:10:10.556 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:10:10 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [9, 8]}]': finished 2024-03-31T23:10:10.556 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:10:10 smithi046 ceph-mon[32498]: osdmap e92: 12 total, 12 up, 12 in 2024-03-31T23:10:10.582 INFO:teuthology.orchestra.run.smithi046.stdout:34359738447 2024-03-31T23:10:10.582 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:384f31f3d3aee7f8ba3f60846b8844c049165583 shell --fsid 6a9d8f08-efb2-11ee-b647-cb9ed24678a4 -- ceph osd last-stat-seq osd.0 2024-03-31T23:10:10.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:10:10 smithi162 ceph-mon[36237]: pgmap v321: 129 pgs: 129 active+clean; 642 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 255 B/s wr, 6 op/s 2024-03-31T23:10:10.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:10:10 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.6", "id": [4, 7]}]': finished 2024-03-31T23:10:10.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:10:10 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.c", "id": [3, 9]}]': finished 2024-03-31T23:10:10.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:10:10 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1a", "id": [5, 6]}]': finished 2024-03-31T23:10:10.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:10:10 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [4, 8]}]': finished 2024-03-31T23:10:10.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:10:10 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1d", "id": [4, 8]}]': finished 2024-03-31T23:10:10.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:10:10 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2a", "id": [3, 6]}]': finished 2024-03-31T23:10:10.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:10:10 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.33", "id": [5, 9]}]': finished 2024-03-31T23:10:10.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:10:10 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.36", "id": [4, 9]}]': finished 2024-03-31T23:10:10.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:10:10 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [9, 8]}]': finished 2024-03-31T23:10:10.797 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:10:10 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [9, 8]}]': finished 2024-03-31T23:10:10.797 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:10:10 smithi162 ceph-mon[36237]: osdmap e92: 12 total, 12 up, 12 in 2024-03-31T23:10:10.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:10.350+0000 7fb5dd5d7700 1 -- 172.21.15.46:0/2647543469 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb5d8101170 msgr2=0x7fb5d81015f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:10:10.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:10.350+0000 7fb5dd5d7700 1 --2- 172.21.15.46:0/2647543469 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb5d8101170 0x7fb5d81015f0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fb5cc009990 tx=0x7fb5cc009ca0 comp rx=0 tx=0).stop 2024-03-31T23:10:10.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:10.350+0000 7fb5dd5d7700 1 -- 172.21.15.46:0/2647543469 shutdown_connections 2024-03-31T23:10:10.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:10.350+0000 7fb5dd5d7700 1 --2- 172.21.15.46:0/2647543469 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb5d8101b30 0x7fb5d810e0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:10.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:10.350+0000 7fb5dd5d7700 1 --2- 172.21.15.46:0/2647543469 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb5d81030d0 0x7fb5d81034b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:10.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:10.350+0000 7fb5dd5d7700 1 --2- 172.21.15.46:0/2647543469 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb5d8101170 0x7fb5d81015f0 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:10.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:10.351+0000 7fb5dd5d7700 1 -- 172.21.15.46:0/2647543469 >> 172.21.15.46:0/2647543469 conn(0x7fb5d8077160 msgr2=0x7fb5d8077570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:10:10.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:10.351+0000 7fb5dd5d7700 1 -- 172.21.15.46:0/2647543469 shutdown_connections 2024-03-31T23:10:10.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:10.351+0000 7fb5dd5d7700 1 -- 172.21.15.46:0/2647543469 wait complete. 2024-03-31T23:10:10.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:10.351+0000 7fb5dd5d7700 1 Processor -- start 2024-03-31T23:10:10.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:10.351+0000 7fb5dd5d7700 1 -- start start 2024-03-31T23:10:10.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:10.351+0000 7fb5dd5d7700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb5d8101170 0x7fb5d819e610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:10:10.884 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:10.351+0000 7fb5dd5d7700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb5d8101b30 0x7fb5d819eb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:10:10.884 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:10.351+0000 7fb5dd5d7700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb5d81030d0 0x7fb5d8198790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:10:10.884 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:10.351+0000 7fb5dd5d7700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fb5d8112530 con 0x7fb5d81030d0 2024-03-31T23:10:10.884 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:10.351+0000 7fb5dd5d7700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fb5d81123b0 con 0x7fb5d8101170 2024-03-31T23:10:10.884 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:10.352+0000 7fb5dd5d7700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fb5d81126b0 con 0x7fb5d8101b30 2024-03-31T23:10:10.884 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:10.352+0000 7fb5d77fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb5d81030d0 0x7fb5d8198790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:10:10.884 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:10.352+0000 7fb5d6ffd700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb5d8101170 0x7fb5d819e610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:10:10.884 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:10.352+0000 7fb5d67fc700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb5d8101b30 0x7fb5d819eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:10:10.884 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:10.352+0000 7fb5d77fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb5d81030d0 0x7fb5d8198790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:38424/0 (socket says 172.21.15.46:38424) 2024-03-31T23:10:10.884 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:10.352+0000 7fb5d77fe700 1 -- 172.21.15.46:0/383090517 learned_addr learned my addr 172.21.15.46:0/383090517 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:10:10.884 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:10.352+0000 7fb5d6ffd700 1 --2- 172.21.15.46:0/383090517 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb5d8101170 0x7fb5d819e610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.46:58204/0 (socket says 172.21.15.46:58204) 2024-03-31T23:10:10.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:10.352+0000 7fb5d77fe700 1 -- 172.21.15.46:0/383090517 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb5d8101b30 msgr2=0x7fb5d819eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:10:10.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:10.352+0000 7fb5d77fe700 1 --2- 172.21.15.46:0/383090517 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb5d8101b30 0x7fb5d819eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:10.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:10.352+0000 7fb5d77fe700 1 -- 172.21.15.46:0/383090517 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb5d8101170 msgr2=0x7fb5d819e610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:10:10.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:10.352+0000 7fb5d77fe700 1 --2- 172.21.15.46:0/383090517 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb5d8101170 0x7fb5d819e610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:10.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:10.352+0000 7fb5d77fe700 1 -- 172.21.15.46:0/383090517 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb5d8198f70 con 0x7fb5d81030d0 2024-03-31T23:10:10.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:10.352+0000 7fb5d6ffd700 1 --2- 172.21.15.46:0/383090517 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb5d8101170 0x7fb5d819e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:10:10.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:10.352+0000 7fb5d77fe700 1 --2- 172.21.15.46:0/383090517 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb5d81030d0 0x7fb5d8198790 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fb5d000ef30 tx=0x7fb5d000c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:10:10.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:10.352+0000 7fb5bffff700 1 -- 172.21.15.46:0/383090517 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5d00092e0 con 0x7fb5d81030d0 2024-03-31T23:10:10.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:10.352+0000 7fb5dd5d7700 1 -- 172.21.15.46:0/383090517 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb5d8199200 con 0x7fb5d81030d0 2024-03-31T23:10:10.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:10.352+0000 7fb5dd5d7700 1 -- 172.21.15.46:0/383090517 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb5d81a5470 con 0x7fb5d81030d0 2024-03-31T23:10:10.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:10.352+0000 7fb5bffff700 1 -- 172.21.15.46:0/383090517 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb5d0004bb0 con 0x7fb5d81030d0 2024-03-31T23:10:10.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:10.352+0000 7fb5bffff700 1 -- 172.21.15.46:0/383090517 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5d0004020 con 0x7fb5d81030d0 2024-03-31T23:10:10.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:10.353+0000 7fb5dd5d7700 1 -- 172.21.15.46:0/383090517 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fb5c4001030 con 0x7fb5d81030d0 2024-03-31T23:10:10.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:10.354+0000 7fb5bffff700 1 -- 172.21.15.46:0/383090517 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 17) ==== 137371+0+0 (secure 0 0 0) 0x7fb5d001b030 con 0x7fb5d81030d0 2024-03-31T23:10:10.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:10.355+0000 7fb5bffff700 1 --2- 172.21.15.46:0/383090517 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fb5c00a25c0 0x7fb5c00a4a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:10:10.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:10.355+0000 7fb5bffff700 1 -- 172.21.15.46:0/383090517 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9092+0+0 (secure 0 0 0) 0x7fb5d00cef10 con 0x7fb5d81030d0 2024-03-31T23:10:10.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:10.355+0000 7fb5d6ffd700 1 --2- 172.21.15.46:0/383090517 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fb5c00a25c0 0x7fb5c00a4a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:10:10.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:10.355+0000 7fb5bffff700 1 --2- 172.21.15.46:0/383090517 >> [v2:172.21.15.46:6818/56291966,v1:172.21.15.46:6819/56291966] conn(0x7fb5c00ad720 0x7fb5c00afbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:10:10.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:10.355+0000 7fb5bffff700 1 -- 172.21.15.46:0/383090517 --> [v2:172.21.15.46:6818/56291966,v1:172.21.15.46:6819/56291966] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fb5d000be50 con 0x7fb5c00ad720 2024-03-31T23:10:10.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:10.355+0000 7fb5bffff700 1 -- 172.21.15.46:0/383090517 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_get_version_reply(handle=1 version=92) ==== 24+0+0 (secure 0 0 0) 0x7fb5d00cf360 con 0x7fb5d81030d0 2024-03-31T23:10:10.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:10.355+0000 7fb5d67fc700 1 --2- 172.21.15.46:0/383090517 >> [v2:172.21.15.46:6818/56291966,v1:172.21.15.46:6819/56291966] conn(0x7fb5c00ad720 0x7fb5c00afbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:10:10.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:10.355+0000 7fb5d6ffd700 1 --2- 172.21.15.46:0/383090517 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fb5c00a25c0 0x7fb5c00a4a80 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fb5c8011490 tx=0x7fb5c8009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:10:10.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:10.356+0000 7fb5d67fc700 1 --2- 172.21.15.46:0/383090517 >> [v2:172.21.15.46:6818/56291966,v1:172.21.15.46:6819/56291966] conn(0x7fb5c00ad720 0x7fb5c00afbc0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:10:10.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:10.356+0000 7fb5bffff700 1 -- 172.21.15.46:0/383090517 <== osd.6 v2:172.21.15.46:6818/56291966 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fb5d000be50 con 0x7fb5c00ad720 2024-03-31T23:10:10.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:10.373+0000 7fb5dd5d7700 1 -- 172.21.15.46:0/383090517 --> [v2:172.21.15.46:6818/56291966,v1:172.21.15.46:6819/56291966] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fb5c4002e60 con 0x7fb5c00ad720 2024-03-31T23:10:10.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:10.373+0000 7fb5bffff700 1 -- 172.21.15.46:0/383090517 <== osd.6 v2:172.21.15.46:6818/56291966 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fb5c4002e60 con 0x7fb5c00ad720 2024-03-31T23:10:10.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:10.373+0000 7fb5dd5d7700 1 -- 172.21.15.46:0/383090517 >> [v2:172.21.15.46:6818/56291966,v1:172.21.15.46:6819/56291966] conn(0x7fb5c00ad720 msgr2=0x7fb5c00afbc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:10:10.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:10.373+0000 7fb5dd5d7700 1 --2- 172.21.15.46:0/383090517 >> [v2:172.21.15.46:6818/56291966,v1:172.21.15.46:6819/56291966] conn(0x7fb5c00ad720 0x7fb5c00afbc0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:10.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:10.373+0000 7fb5dd5d7700 1 -- 172.21.15.46:0/383090517 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fb5c00a25c0 msgr2=0x7fb5c00a4a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:10:10.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:10.373+0000 7fb5dd5d7700 1 --2- 172.21.15.46:0/383090517 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fb5c00a25c0 0x7fb5c00a4a80 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fb5c8011490 tx=0x7fb5c8009260 comp rx=0 tx=0).stop 2024-03-31T23:10:10.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:10.373+0000 7fb5dd5d7700 1 -- 172.21.15.46:0/383090517 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb5d81030d0 msgr2=0x7fb5d8198790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:10:10.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:10.373+0000 7fb5dd5d7700 1 --2- 172.21.15.46:0/383090517 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb5d81030d0 0x7fb5d8198790 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fb5d000ef30 tx=0x7fb5d000c3a0 comp rx=0 tx=0).stop 2024-03-31T23:10:10.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:10.373+0000 7fb5d77fe700 1 -- 172.21.15.46:0/383090517 reap_dead start 2024-03-31T23:10:10.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:10.374+0000 7fb5dd5d7700 1 -- 172.21.15.46:0/383090517 shutdown_connections 2024-03-31T23:10:10.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:10.374+0000 7fb5dd5d7700 1 -- 172.21.15.46:0/383090517 >> 172.21.15.46:0/383090517 conn(0x7fb5d8077160 msgr2=0x7fb5d80ffea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:10:10.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:10.374+0000 7fb5dd5d7700 1 -- 172.21.15.46:0/383090517 shutdown_connections 2024-03-31T23:10:10.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:10.374+0000 7fb5dd5d7700 1 -- 172.21.15.46:0/383090517 wait complete. 2024-03-31T23:10:10.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:10:10 smithi142 ceph-mon[35271]: pgmap v321: 129 pgs: 129 active+clean; 642 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 255 B/s wr, 6 op/s 2024-03-31T23:10:10.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:10:10 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.6", "id": [4, 7]}]': finished 2024-03-31T23:10:10.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:10:10 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.c", "id": [3, 9]}]': finished 2024-03-31T23:10:10.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:10:10 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1a", "id": [5, 6]}]': finished 2024-03-31T23:10:10.944 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:10:10 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [4, 8]}]': finished 2024-03-31T23:10:10.944 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:10:10 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1d", "id": [4, 8]}]': finished 2024-03-31T23:10:10.944 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:10:10 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2a", "id": [3, 6]}]': finished 2024-03-31T23:10:10.944 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:10:10 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.33", "id": [5, 9]}]': finished 2024-03-31T23:10:10.944 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:10:10 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.36", "id": [4, 9]}]': finished 2024-03-31T23:10:10.944 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:10:10 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [9, 8]}]': finished 2024-03-31T23:10:10.944 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:10:10 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [9, 8]}]': finished 2024-03-31T23:10:10.944 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:10:10 smithi142 ceph-mon[35271]: osdmap e92: 12 total, 12 up, 12 in 2024-03-31T23:10:11.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:11.087+0000 7f2a5406c700 1 -- 172.21.15.46:0/1105548194 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2a4c102290 msgr2=0x7f2a4c102710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:10:11.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:11.087+0000 7f2a5406c700 1 --2- 172.21.15.46:0/1105548194 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2a4c102290 0x7f2a4c102710 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f2a3c0099c0 tx=0x7f2a3c009cd0 comp rx=0 tx=0).stop 2024-03-31T23:10:11.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:11.087+0000 7f2a5406c700 1 -- 172.21.15.46:0/1105548194 shutdown_connections 2024-03-31T23:10:11.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:11.087+0000 7f2a5406c700 1 --2- 172.21.15.46:0/1105548194 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2a4c102c50 0x7f2a4c10f090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:11.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:11.087+0000 7f2a5406c700 1 --2- 172.21.15.46:0/1105548194 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2a4c108270 0x7f2a4c108650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:11.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:11.087+0000 7f2a5406c700 1 --2- 172.21.15.46:0/1105548194 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2a4c102290 0x7f2a4c102710 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:11.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:11.087+0000 7f2a5406c700 1 -- 172.21.15.46:0/1105548194 >> 172.21.15.46:0/1105548194 conn(0x7f2a4c079370 msgr2=0x7f2a4c079780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:10:11.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:11.087+0000 7f2a5406c700 1 -- 172.21.15.46:0/1105548194 shutdown_connections 2024-03-31T23:10:11.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:11.087+0000 7f2a5406c700 1 -- 172.21.15.46:0/1105548194 wait complete. 2024-03-31T23:10:11.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:11.088+0000 7f2a5406c700 1 Processor -- start 2024-03-31T23:10:11.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:11.088+0000 7f2a5406c700 1 -- start start 2024-03-31T23:10:11.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:11.088+0000 7f2a5406c700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2a4c102290 0x7f2a4c074540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:10:11.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:11.088+0000 7f2a5406c700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2a4c102c50 0x7f2a4c074a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:10:11.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:11.088+0000 7f2a5406c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2a4c108270 0x7f2a4c074fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:10:11.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:11.088+0000 7f2a5406c700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f2a4c1123d0 con 0x7f2a4c108270 2024-03-31T23:10:11.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:11.088+0000 7f2a5406c700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f2a4c112250 con 0x7f2a4c102c50 2024-03-31T23:10:11.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:11.088+0000 7f2a5406c700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f2a4c112550 con 0x7f2a4c102290 2024-03-31T23:10:11.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:11.088+0000 7f2a52609700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2a4c108270 0x7f2a4c074fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:10:11.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:11.088+0000 7f2a52609700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2a4c108270 0x7f2a4c074fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:38450/0 (socket says 172.21.15.46:38450) 2024-03-31T23:10:11.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:11.088+0000 7f2a52609700 1 -- 172.21.15.46:0/3379928337 learned_addr learned my addr 172.21.15.46:0/3379928337 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:10:11.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:11.088+0000 7f2a51607700 1 --2- 172.21.15.46:0/3379928337 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2a4c102c50 0x7f2a4c074a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:10:11.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:11.088+0000 7f2a51e08700 1 --2- 172.21.15.46:0/3379928337 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2a4c102290 0x7f2a4c074540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:10:11.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:11.089+0000 7f2a52609700 1 -- 172.21.15.46:0/3379928337 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2a4c102290 msgr2=0x7f2a4c074540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:10:11.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:11.089+0000 7f2a52609700 1 --2- 172.21.15.46:0/3379928337 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2a4c102290 0x7f2a4c074540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:11.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:11.089+0000 7f2a52609700 1 -- 172.21.15.46:0/3379928337 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2a4c102c50 msgr2=0x7f2a4c074a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:10:11.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:11.089+0000 7f2a52609700 1 --2- 172.21.15.46:0/3379928337 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2a4c102c50 0x7f2a4c074a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:11.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:11.089+0000 7f2a52609700 1 -- 172.21.15.46:0/3379928337 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a4c1a6e70 con 0x7f2a4c108270 2024-03-31T23:10:11.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:11.089+0000 7f2a52609700 1 --2- 172.21.15.46:0/3379928337 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2a4c108270 0x7f2a4c074fc0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f2a4400ef30 tx=0x7f2a4400c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:10:11.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:11.089+0000 7f2a42ffd700 1 -- 172.21.15.46:0/3379928337 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a440092e0 con 0x7f2a4c108270 2024-03-31T23:10:11.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:11.089+0000 7f2a42ffd700 1 -- 172.21.15.46:0/3379928337 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f2a44004bb0 con 0x7f2a4c108270 2024-03-31T23:10:11.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:11.089+0000 7f2a5406c700 1 -- 172.21.15.46:0/3379928337 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2a4c1a7100 con 0x7f2a4c108270 2024-03-31T23:10:11.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:11.089+0000 7f2a5406c700 1 -- 172.21.15.46:0/3379928337 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2a4c1a7610 con 0x7f2a4c108270 2024-03-31T23:10:11.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:11.089+0000 7f2a42ffd700 1 -- 172.21.15.46:0/3379928337 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a44004020 con 0x7f2a4c108270 2024-03-31T23:10:11.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:11.090+0000 7f2a5406c700 1 -- 172.21.15.46:0/3379928337 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f2a4c062910 con 0x7f2a4c108270 2024-03-31T23:10:11.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:11.091+0000 7f2a42ffd700 1 -- 172.21.15.46:0/3379928337 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 17) ==== 137371+0+0 (secure 0 0 0) 0x7f2a4401b030 con 0x7f2a4c108270 2024-03-31T23:10:11.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:11.092+0000 7f2a42ffd700 1 --2- 172.21.15.46:0/3379928337 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f2a380a25c0 0x7f2a380a4a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:10:11.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:11.092+0000 7f2a42ffd700 1 -- 172.21.15.46:0/3379928337 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7f2a440cefc0 con 0x7f2a4c108270 2024-03-31T23:10:11.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:11.092+0000 7f2a51e08700 1 --2- 172.21.15.46:0/3379928337 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f2a380a25c0 0x7f2a380a4a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:10:11.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:11.092+0000 7f2a42ffd700 1 --2- 172.21.15.46:0/3379928337 >> [v2:172.21.15.162:6824/1986323778,v1:172.21.15.162:6825/1986323778] conn(0x7f2a380ad720 0x7f2a380afbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:10:11.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:11.092+0000 7f2a42ffd700 1 -- 172.21.15.46:0/3379928337 --> [v2:172.21.15.162:6824/1986323778,v1:172.21.15.162:6825/1986323778] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f2a4401fe50 con 0x7f2a380ad720 2024-03-31T23:10:11.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:11.092+0000 7f2a42ffd700 1 -- 172.21.15.46:0/3379928337 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_get_version_reply(handle=1 version=93) ==== 24+0+0 (secure 0 0 0) 0x7f2a440cf410 con 0x7f2a4c108270 2024-03-31T23:10:11.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:11.092+0000 7f2a51607700 1 --2- 172.21.15.46:0/3379928337 >> [v2:172.21.15.162:6824/1986323778,v1:172.21.15.162:6825/1986323778] conn(0x7f2a380ad720 0x7f2a380afbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:10:11.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:11.092+0000 7f2a51e08700 1 --2- 172.21.15.46:0/3379928337 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f2a380a25c0 0x7f2a380a4a80 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f2a48009be0 tx=0x7f2a48009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:10:11.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:11.093+0000 7f2a51607700 1 --2- 172.21.15.46:0/3379928337 >> [v2:172.21.15.162:6824/1986323778,v1:172.21.15.162:6825/1986323778] conn(0x7f2a380ad720 0x7f2a380afbc0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.11 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:10:11.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:11.094+0000 7f2a42ffd700 1 -- 172.21.15.46:0/3379928337 <== osd.11 v2:172.21.15.162:6824/1986323778 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f2a4401fe50 con 0x7f2a380ad720 2024-03-31T23:10:11.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:11.110+0000 7f2a5406c700 1 -- 172.21.15.46:0/3379928337 --> [v2:172.21.15.162:6824/1986323778,v1:172.21.15.162:6825/1986323778] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f2a4c062d90 con 0x7f2a380ad720 2024-03-31T23:10:11.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:11.111+0000 7f2a42ffd700 1 -- 172.21.15.46:0/3379928337 <== osd.11 v2:172.21.15.162:6824/1986323778 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f2a4c062d90 con 0x7f2a380ad720 2024-03-31T23:10:11.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:11.111+0000 7f2a5406c700 1 -- 172.21.15.46:0/3379928337 >> [v2:172.21.15.162:6824/1986323778,v1:172.21.15.162:6825/1986323778] conn(0x7f2a380ad720 msgr2=0x7f2a380afbc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:10:11.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:11.111+0000 7f2a5406c700 1 --2- 172.21.15.46:0/3379928337 >> [v2:172.21.15.162:6824/1986323778,v1:172.21.15.162:6825/1986323778] conn(0x7f2a380ad720 0x7f2a380afbc0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:11.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:11.111+0000 7f2a5406c700 1 -- 172.21.15.46:0/3379928337 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f2a380a25c0 msgr2=0x7f2a380a4a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:10:11.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:11.111+0000 7f2a5406c700 1 --2- 172.21.15.46:0/3379928337 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f2a380a25c0 0x7f2a380a4a80 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f2a48009be0 tx=0x7f2a48009260 comp rx=0 tx=0).stop 2024-03-31T23:10:11.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:11.111+0000 7f2a5406c700 1 -- 172.21.15.46:0/3379928337 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2a4c108270 msgr2=0x7f2a4c074fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:10:11.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:11.111+0000 7f2a5406c700 1 --2- 172.21.15.46:0/3379928337 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2a4c108270 0x7f2a4c074fc0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f2a4400ef30 tx=0x7f2a4400c3a0 comp rx=0 tx=0).stop 2024-03-31T23:10:11.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:11.111+0000 7f2a52609700 1 -- 172.21.15.46:0/3379928337 reap_dead start 2024-03-31T23:10:11.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:11.111+0000 7f2a5406c700 1 -- 172.21.15.46:0/3379928337 shutdown_connections 2024-03-31T23:10:11.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:11.111+0000 7f2a5406c700 1 -- 172.21.15.46:0/3379928337 >> 172.21.15.46:0/3379928337 conn(0x7f2a4c079370 msgr2=0x7f2a4c101030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:10:11.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:11.111+0000 7f2a5406c700 1 -- 172.21.15.46:0/3379928337 shutdown_connections 2024-03-31T23:10:11.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:11.111+0000 7f2a5406c700 1 -- 172.21.15.46:0/3379928337 wait complete. 2024-03-31T23:10:11.611 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:10:11 smithi046 ceph-mon[32498]: osdmap e93: 12 total, 12 up, 12 in 2024-03-31T23:10:11.611 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:10:11 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-31T23:10:11.772 INFO:teuthology.orchestra.run.smithi046.stdout:77309411398 2024-03-31T23:10:11.773 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:384f31f3d3aee7f8ba3f60846b8844c049165583 shell --fsid 6a9d8f08-efb2-11ee-b647-cb9ed24678a4 -- ceph osd last-stat-seq osd.2 2024-03-31T23:10:11.795 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:10:11 smithi162 ceph-mon[36237]: osdmap e93: 12 total, 12 up, 12 in 2024-03-31T23:10:11.795 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:10:11 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-31T23:10:11.842 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:10:11 smithi142 ceph-mon[35271]: osdmap e93: 12 total, 12 up, 12 in 2024-03-31T23:10:11.842 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:10:11 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-31T23:10:12.519 INFO:teuthology.orchestra.run.smithi046.stdout:313532612638 2024-03-31T23:10:12.520 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:384f31f3d3aee7f8ba3f60846b8844c049165583 shell --fsid 6a9d8f08-efb2-11ee-b647-cb9ed24678a4 -- ceph osd last-stat-seq osd.10 2024-03-31T23:10:12.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:10:12 smithi162 ceph-mon[36237]: pgmap v324: 129 pgs: 2 peering, 127 active+clean; 642 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-03-31T23:10:12.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:10:12 smithi046 ceph-mon[32498]: pgmap v324: 129 pgs: 2 peering, 127 active+clean; 642 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-03-31T23:10:12.914 INFO:teuthology.orchestra.run.smithi046.stdout:347892351002 2024-03-31T23:10:12.915 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:384f31f3d3aee7f8ba3f60846b8844c049165583 shell --fsid 6a9d8f08-efb2-11ee-b647-cb9ed24678a4 -- ceph osd last-stat-seq osd.11 2024-03-31T23:10:12.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:10:12 smithi142 ceph-mon[35271]: pgmap v324: 129 pgs: 2 peering, 127 active+clean; 642 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-03-31T23:10:13.414 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:13.412+0000 7f714d3ee700 1 -- 172.21.15.46:0/1080189237 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f71481075c0 msgr2=0x7f714810f140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:10:13.414 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:13.412+0000 7f714d3ee700 1 --2- 172.21.15.46:0/1080189237 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f71481075c0 0x7f714810f140 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f71380099c0 tx=0x7f7138009cd0 comp rx=0 tx=0).stop 2024-03-31T23:10:13.414 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:13.412+0000 7f714d3ee700 1 -- 172.21.15.46:0/1080189237 shutdown_connections 2024-03-31T23:10:13.414 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:13.412+0000 7f714d3ee700 1 --2- 172.21.15.46:0/1080189237 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f71481075c0 0x7f714810f140 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:13.414 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:13.412+0000 7f714d3ee700 1 --2- 172.21.15.46:0/1080189237 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f714806a9e0 0x7f714806adc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:13.414 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:13.412+0000 7f714d3ee700 1 --2- 172.21.15.46:0/1080189237 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f714806b390 0x7f7148106e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:13.414 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:13.412+0000 7f714d3ee700 1 -- 172.21.15.46:0/1080189237 >> 172.21.15.46:0/1080189237 conn(0x7f7148077190 msgr2=0x7f71480775a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:10:13.414 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:13.412+0000 7f714d3ee700 1 -- 172.21.15.46:0/1080189237 shutdown_connections 2024-03-31T23:10:13.414 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:13.412+0000 7f714d3ee700 1 -- 172.21.15.46:0/1080189237 wait complete. 2024-03-31T23:10:13.414 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:13.413+0000 7f714d3ee700 1 Processor -- start 2024-03-31T23:10:13.414 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:13.413+0000 7f714d3ee700 1 -- start start 2024-03-31T23:10:13.416 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:13.413+0000 7f714d3ee700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f714806a9e0 0x7f714819e620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:10:13.416 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:13.413+0000 7f714d3ee700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f714806b390 0x7f714819eb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:10:13.416 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:13.413+0000 7f714d3ee700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f71481075c0 0x7f71481986f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:10:13.416 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:13.413+0000 7f714d3ee700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f7148112430 con 0x7f71481075c0 2024-03-31T23:10:13.416 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:13.413+0000 7f714d3ee700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f71481122b0 con 0x7f714806a9e0 2024-03-31T23:10:13.416 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:13.413+0000 7f714d3ee700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f71481125b0 con 0x7f714806b390 2024-03-31T23:10:13.416 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:13.413+0000 7f71477fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f71481075c0 0x7f71481986f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:10:13.417 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:13.413+0000 7f71477fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f71481075c0 0x7f71481986f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:33232/0 (socket says 172.21.15.46:33232) 2024-03-31T23:10:13.417 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:13.413+0000 7f71477fe700 1 -- 172.21.15.46:0/3464566402 learned_addr learned my addr 172.21.15.46:0/3464566402 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:10:13.417 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:13.413+0000 7f71467fc700 1 --2- 172.21.15.46:0/3464566402 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f714806b390 0x7f714819eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:10:13.417 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:13.413+0000 7f7146ffd700 1 --2- 172.21.15.46:0/3464566402 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f714806a9e0 0x7f714819e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:10:13.417 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:13.413+0000 7f71477fe700 1 -- 172.21.15.46:0/3464566402 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f714806b390 msgr2=0x7f714819eb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:10:13.417 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:13.413+0000 7f71477fe700 1 --2- 172.21.15.46:0/3464566402 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f714806b390 0x7f714819eb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:13.417 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:13.413+0000 7f71477fe700 1 -- 172.21.15.46:0/3464566402 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f714806a9e0 msgr2=0x7f714819e620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:10:13.417 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:13.413+0000 7f71477fe700 1 --2- 172.21.15.46:0/3464566402 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f714806a9e0 0x7f714819e620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:13.417 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:13.413+0000 7f71477fe700 1 -- 172.21.15.46:0/3464566402 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7148198fd0 con 0x7f71481075c0 2024-03-31T23:10:13.417 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:13.414+0000 7f71477fe700 1 --2- 172.21.15.46:0/3464566402 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f71481075c0 0x7f71481986f0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f71380059d0 tx=0x7f7138004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:10:13.417 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:13.414+0000 7f713ffff700 1 -- 172.21.15.46:0/3464566402 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7138026070 con 0x7f71481075c0 2024-03-31T23:10:13.417 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:13.414+0000 7f713ffff700 1 -- 172.21.15.46:0/3464566402 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7138003c70 con 0x7f71481075c0 2024-03-31T23:10:13.417 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:13.414+0000 7f713ffff700 1 -- 172.21.15.46:0/3464566402 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f713800fba0 con 0x7f71481075c0 2024-03-31T23:10:13.417 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:13.414+0000 7f714d3ee700 1 -- 172.21.15.46:0/3464566402 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7148199260 con 0x7f71481075c0 2024-03-31T23:10:13.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:13.414+0000 7f714d3ee700 1 -- 172.21.15.46:0/3464566402 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f71481a53d0 con 0x7f71481075c0 2024-03-31T23:10:13.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:13.415+0000 7f714d3ee700 1 -- 172.21.15.46:0/3464566402 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f71480629d0 con 0x7f71481075c0 2024-03-31T23:10:13.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:13.416+0000 7f713ffff700 1 -- 172.21.15.46:0/3464566402 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 17) ==== 137371+0+0 (secure 0 0 0) 0x7f7138029070 con 0x7f71481075c0 2024-03-31T23:10:13.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:13.417+0000 7f713ffff700 1 --2- 172.21.15.46:0/3464566402 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f71340a25c0 0x7f71340a4a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:10:13.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:13.417+0000 7f7146ffd700 1 --2- 172.21.15.46:0/3464566402 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f71340a25c0 0x7f71340a4a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:10:13.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:13.417+0000 7f713ffff700 1 -- 172.21.15.46:0/3464566402 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7f71380cfa50 con 0x7f71481075c0 2024-03-31T23:10:13.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:13.417+0000 7f713ffff700 1 --2- 172.21.15.46:0/3464566402 >> [v2:172.21.15.46:6826/532708881,v1:172.21.15.46:6827/532708881] conn(0x7f71340ad720 0x7f71340afbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:10:13.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:13.417+0000 7f713ffff700 1 -- 172.21.15.46:0/3464566402 --> [v2:172.21.15.46:6826/532708881,v1:172.21.15.46:6827/532708881] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f71340b0290 con 0x7f71340ad720 2024-03-31T23:10:13.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:13.417+0000 7f713ffff700 1 -- 172.21.15.46:0/3464566402 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_get_version_reply(handle=1 version=93) ==== 24+0+0 (secure 0 0 0) 0x7f71380cfea0 con 0x7f71481075c0 2024-03-31T23:10:13.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:13.417+0000 7f71467fc700 1 --2- 172.21.15.46:0/3464566402 >> [v2:172.21.15.46:6826/532708881,v1:172.21.15.46:6827/532708881] conn(0x7f71340ad720 0x7f71340afbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:10:13.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:13.417+0000 7f7146ffd700 1 --2- 172.21.15.46:0/3464566402 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f71340a25c0 0x7f71340a4a80 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f714000f540 tx=0x7f714000b040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:10:13.420 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:13.417+0000 7f71467fc700 1 --2- 172.21.15.46:0/3464566402 >> [v2:172.21.15.46:6826/532708881,v1:172.21.15.46:6827/532708881] conn(0x7f71340ad720 0x7f71340afbc0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.9 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:10:13.420 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:13.418+0000 7f713ffff700 1 -- 172.21.15.46:0/3464566402 <== osd.9 v2:172.21.15.46:6826/532708881 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f71340b0290 con 0x7f71340ad720 2024-03-31T23:10:13.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:13.434+0000 7f714d3ee700 1 -- 172.21.15.46:0/3464566402 --> [v2:172.21.15.46:6826/532708881,v1:172.21.15.46:6827/532708881] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f7148062e50 con 0x7f71340ad720 2024-03-31T23:10:13.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:13.435+0000 7f713ffff700 1 -- 172.21.15.46:0/3464566402 <== osd.9 v2:172.21.15.46:6826/532708881 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f7148062e50 con 0x7f71340ad720 2024-03-31T23:10:13.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:13.435+0000 7f714d3ee700 1 -- 172.21.15.46:0/3464566402 >> [v2:172.21.15.46:6826/532708881,v1:172.21.15.46:6827/532708881] conn(0x7f71340ad720 msgr2=0x7f71340afbc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:10:13.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:13.435+0000 7f714d3ee700 1 --2- 172.21.15.46:0/3464566402 >> [v2:172.21.15.46:6826/532708881,v1:172.21.15.46:6827/532708881] conn(0x7f71340ad720 0x7f71340afbc0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:13.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:13.435+0000 7f714d3ee700 1 -- 172.21.15.46:0/3464566402 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f71340a25c0 msgr2=0x7f71340a4a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:10:13.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:13.435+0000 7f714d3ee700 1 --2- 172.21.15.46:0/3464566402 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f71340a25c0 0x7f71340a4a80 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f714000f540 tx=0x7f714000b040 comp rx=0 tx=0).stop 2024-03-31T23:10:13.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:13.435+0000 7f714d3ee700 1 -- 172.21.15.46:0/3464566402 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f71481075c0 msgr2=0x7f71481986f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:10:13.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:13.435+0000 7f714d3ee700 1 --2- 172.21.15.46:0/3464566402 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f71481075c0 0x7f71481986f0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f71380059d0 tx=0x7f7138004270 comp rx=0 tx=0).stop 2024-03-31T23:10:13.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:13.435+0000 7f71477fe700 1 -- 172.21.15.46:0/3464566402 reap_dead start 2024-03-31T23:10:13.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:13.435+0000 7f714d3ee700 1 -- 172.21.15.46:0/3464566402 shutdown_connections 2024-03-31T23:10:13.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:13.435+0000 7f714d3ee700 1 -- 172.21.15.46:0/3464566402 >> 172.21.15.46:0/3464566402 conn(0x7f7148077190 msgr2=0x7f71480ff500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:10:13.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:13.435+0000 7f714d3ee700 1 -- 172.21.15.46:0/3464566402 shutdown_connections 2024-03-31T23:10:13.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:13.435+0000 7f714d3ee700 1 -- 172.21.15.46:0/3464566402 wait complete. 2024-03-31T23:10:14.099 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:10:14 smithi046 ceph-mon[32498]: pgmap v325: 129 pgs: 2 peering, 127 active+clean; 642 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-03-31T23:10:14.099 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:10:14 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:10:14.099 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:10:14 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:10:14.099 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:10:14 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:10:14.100 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:10:14 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:10:14.125 INFO:teuthology.orchestra.run.smithi046.stdout:292057776164 2024-03-31T23:10:14.125 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:384f31f3d3aee7f8ba3f60846b8844c049165583 shell --fsid 6a9d8f08-efb2-11ee-b647-cb9ed24678a4 -- ceph osd last-stat-seq osd.9 2024-03-31T23:10:14.296 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:10:14 smithi162 ceph-mon[36237]: pgmap v325: 129 pgs: 2 peering, 127 active+clean; 642 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-03-31T23:10:14.296 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:10:14 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:10:14.296 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:10:14 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:10:14.296 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:10:14 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:10:14.296 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:10:14 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:10:14.443 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:10:14 smithi142 ceph-mon[35271]: pgmap v325: 129 pgs: 2 peering, 127 active+clean; 642 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-03-31T23:10:14.443 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:10:14 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:10:14.443 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:10:14 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:10:14.444 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:10:14 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:10:14.444 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:10:14 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:10:14.605 INFO:teuthology.orchestra.run.smithi046.stdout:201863462963 2024-03-31T23:10:14.605 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:384f31f3d3aee7f8ba3f60846b8844c049165583 shell --fsid 6a9d8f08-efb2-11ee-b647-cb9ed24678a4 -- ceph osd last-stat-seq osd.6 2024-03-31T23:10:14.663 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/6a9d8f08-efb2-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-31T23:10:14.687 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/6a9d8f08-efb2-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-31T23:10:14.687 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/6a9d8f08-efb2-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-31T23:10:14.687 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/6a9d8f08-efb2-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-31T23:10:14.694 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/6a9d8f08-efb2-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-31T23:10:14.711 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/6a9d8f08-efb2-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-31T23:10:14.715 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/6a9d8f08-efb2-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-31T23:10:16.295 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:10:16 smithi162 ceph-mon[36237]: pgmap v326: 129 pgs: 2 peering, 127 active+clean; 642 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 4 op/s 2024-03-31T23:10:16.361 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:10:16 smithi046 ceph-mon[32498]: pgmap v326: 129 pgs: 2 peering, 127 active+clean; 642 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 4 op/s 2024-03-31T23:10:16.443 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:10:16 smithi142 ceph-mon[35271]: pgmap v326: 129 pgs: 2 peering, 127 active+clean; 642 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 4 op/s 2024-03-31T23:10:17.894 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/6a9d8f08-efb2-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-31T23:10:17.895 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/6a9d8f08-efb2-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-31T23:10:17.896 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/6a9d8f08-efb2-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-31T23:10:18.097 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:10:18 smithi046 ceph-mon[32498]: pgmap v327: 129 pgs: 129 active+clean; 642 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-03-31T23:10:18.296 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:10:18 smithi162 ceph-mon[36237]: pgmap v327: 129 pgs: 129 active+clean; 642 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-03-31T23:10:18.443 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:10:18 smithi142 ceph-mon[35271]: pgmap v327: 129 pgs: 129 active+clean; 642 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-03-31T23:10:20.204 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:10:20 smithi046 ceph-mon[32498]: pgmap v328: 129 pgs: 129 active+clean; 642 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 5 op/s 2024-03-31T23:10:20.295 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:10:20 smithi162 ceph-mon[36237]: pgmap v328: 129 pgs: 129 active+clean; 642 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 5 op/s 2024-03-31T23:10:20.443 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:10:20 smithi142 ceph-mon[35271]: pgmap v328: 129 pgs: 129 active+clean; 642 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 5 op/s 2024-03-31T23:10:22.295 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:10:22 smithi162 ceph-mon[36237]: pgmap v329: 129 pgs: 129 active+clean; 642 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-31T23:10:22.362 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:10:22 smithi046 ceph-mon[32498]: pgmap v329: 129 pgs: 129 active+clean; 642 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-31T23:10:22.443 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:10:22 smithi142 ceph-mon[35271]: pgmap v329: 129 pgs: 129 active+clean; 642 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-31T23:10:23.024 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:23.023+0000 7f783adb1700 1 -- 172.21.15.46:0/967962517 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7834069ef0 msgr2=0x7f783406a350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:10:23.025 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:23.023+0000 7f783adb1700 1 --2- 172.21.15.46:0/967962517 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7834069ef0 0x7f783406a350 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f78200099c0 tx=0x7f7820009cd0 comp rx=0 tx=0).stop 2024-03-31T23:10:23.025 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:23.023+0000 7f783adb1700 1 -- 172.21.15.46:0/967962517 shutdown_connections 2024-03-31T23:10:23.025 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:23.023+0000 7f783adb1700 1 --2- 172.21.15.46:0/967962517 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f783406a890 0x7f783410f0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:23.025 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:23.023+0000 7f783adb1700 1 --2- 172.21.15.46:0/967962517 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7834108290 0x7f7834108670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:23.025 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:23.023+0000 7f783adb1700 1 --2- 172.21.15.46:0/967962517 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7834069ef0 0x7f783406a350 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:23.025 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:23.023+0000 7f783adb1700 1 -- 172.21.15.46:0/967962517 >> 172.21.15.46:0/967962517 conn(0x7f7834078070 msgr2=0x7f7834078480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:10:23.025 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:23.023+0000 7f783adb1700 1 -- 172.21.15.46:0/967962517 shutdown_connections 2024-03-31T23:10:23.025 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:23.023+0000 7f783adb1700 1 -- 172.21.15.46:0/967962517 wait complete. 2024-03-31T23:10:23.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:23.024+0000 7f783adb1700 1 Processor -- start 2024-03-31T23:10:23.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:23.024+0000 7f783adb1700 1 -- start start 2024-03-31T23:10:23.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:23.024+0000 7f783adb1700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7834069ef0 0x7f783419e610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:10:23.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:23.024+0000 7f783adb1700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f783406a890 0x7f783419eb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:10:23.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:23.024+0000 7f783adb1700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7834108290 0x7f78341986e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:10:23.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:23.025+0000 7f783adb1700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f78341123a0 con 0x7f7834108290 2024-03-31T23:10:23.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:23.025+0000 7f783adb1700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f7834112220 con 0x7f783406a890 2024-03-31T23:10:23.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:23.025+0000 7f783adb1700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f7834112520 con 0x7f7834069ef0 2024-03-31T23:10:23.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:23.025+0000 7f783934e700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7834108290 0x7f78341986e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:10:23.028 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:23.025+0000 7f783934e700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7834108290 0x7f78341986e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59400/0 (socket says 172.21.15.46:59400) 2024-03-31T23:10:23.028 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:23.025+0000 7f7838b4d700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7834069ef0 0x7f783419e610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:10:23.028 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:23.025+0000 7f783934e700 1 -- 172.21.15.46:0/1567775657 learned_addr learned my addr 172.21.15.46:0/1567775657 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:10:23.028 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:23.025+0000 7f7833fff700 1 --2- 172.21.15.46:0/1567775657 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f783406a890 0x7f783419eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:10:23.028 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:23.025+0000 7f783934e700 1 -- 172.21.15.46:0/1567775657 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7834069ef0 msgr2=0x7f783419e610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:10:23.028 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:23.025+0000 7f783934e700 1 --2- 172.21.15.46:0/1567775657 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7834069ef0 0x7f783419e610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:23.028 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:23.025+0000 7f783934e700 1 -- 172.21.15.46:0/1567775657 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f783406a890 msgr2=0x7f783419eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:10:23.028 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:23.025+0000 7f783934e700 1 --2- 172.21.15.46:0/1567775657 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f783406a890 0x7f783419eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:23.028 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:23.025+0000 7f783934e700 1 -- 172.21.15.46:0/1567775657 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7834198fc0 con 0x7f7834108290 2024-03-31T23:10:23.028 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:23.025+0000 7f7838b4d700 1 --2- 172.21.15.46:0/1567775657 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7834069ef0 0x7f783419e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:10:23.028 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:23.025+0000 7f7833fff700 1 --2- 172.21.15.46:0/1567775657 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f783406a890 0x7f783419eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:10:23.028 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:23.025+0000 7f783934e700 1 --2- 172.21.15.46:0/1567775657 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7834108290 0x7f78341986e0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f782c00ef30 tx=0x7f782c00c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:10:23.029 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:23.025+0000 7f7831ffb700 1 -- 172.21.15.46:0/1567775657 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f782c0092e0 con 0x7f7834108290 2024-03-31T23:10:23.029 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:23.025+0000 7f7831ffb700 1 -- 172.21.15.46:0/1567775657 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f782c004bb0 con 0x7f7834108290 2024-03-31T23:10:23.029 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:23.025+0000 7f783adb1700 1 -- 172.21.15.46:0/1567775657 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7834199250 con 0x7f7834108290 2024-03-31T23:10:23.029 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:23.025+0000 7f7831ffb700 1 -- 172.21.15.46:0/1567775657 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f782c004020 con 0x7f7834108290 2024-03-31T23:10:23.029 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:23.026+0000 7f783adb1700 1 -- 172.21.15.46:0/1567775657 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f78341a5590 con 0x7f7834108290 2024-03-31T23:10:23.030 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:23.026+0000 7f783adb1700 1 -- 172.21.15.46:0/1567775657 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7834103940 con 0x7f7834108290 2024-03-31T23:10:23.030 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:23.027+0000 7f7831ffb700 1 -- 172.21.15.46:0/1567775657 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 17) ==== 137371+0+0 (secure 0 0 0) 0x7f782c01b030 con 0x7f7834108290 2024-03-31T23:10:23.030 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:23.028+0000 7f7831ffb700 1 --2- 172.21.15.46:0/1567775657 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f78240a2640 0x7f78240a4b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:10:23.030 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:23.028+0000 7f7838b4d700 1 --2- 172.21.15.46:0/1567775657 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f78240a2640 0x7f78240a4b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:10:23.030 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:23.028+0000 7f7831ffb700 1 -- 172.21.15.46:0/1567775657 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7f782c0ce640 con 0x7f7834108290 2024-03-31T23:10:23.030 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:23.029+0000 7f7838b4d700 1 --2- 172.21.15.46:0/1567775657 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f78240a2640 0x7f78240a4b00 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f7828009fd0 tx=0x7f7828009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:10:23.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:23.033+0000 7f7831ffb700 1 -- 172.21.15.46:0/1567775657 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f782c022020 con 0x7f7834108290 2024-03-31T23:10:23.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:23.244+0000 7f783adb1700 1 -- 172.21.15.46:0/1567775657 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7f7834199c10 con 0x7f7834108290 2024-03-31T23:10:23.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:23.245+0000 7f7831ffb700 1 -- 172.21.15.46:0/1567775657 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f782c010040 con 0x7f7834108290 2024-03-31T23:10:23.246 INFO:teuthology.orchestra.run.smithi046.stdout:34359738450 2024-03-31T23:10:23.248 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:23.247+0000 7f783adb1700 1 -- 172.21.15.46:0/1567775657 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f78240a2640 msgr2=0x7f78240a4b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:10:23.248 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:23.247+0000 7f783adb1700 1 --2- 172.21.15.46:0/1567775657 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f78240a2640 0x7f78240a4b00 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f7828009fd0 tx=0x7f7828009260 comp rx=0 tx=0).stop 2024-03-31T23:10:23.248 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:23.247+0000 7f783adb1700 1 -- 172.21.15.46:0/1567775657 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7834108290 msgr2=0x7f78341986e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:10:23.248 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:23.247+0000 7f783adb1700 1 --2- 172.21.15.46:0/1567775657 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7834108290 0x7f78341986e0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f782c00ef30 tx=0x7f782c00c3a0 comp rx=0 tx=0).stop 2024-03-31T23:10:23.248 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:23.247+0000 7f783adb1700 1 -- 172.21.15.46:0/1567775657 shutdown_connections 2024-03-31T23:10:23.249 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:23.247+0000 7f783adb1700 1 --2- 172.21.15.46:0/1567775657 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f78240a2640 0x7f78240a4b00 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:23.249 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:23.247+0000 7f783adb1700 1 --2- 172.21.15.46:0/1567775657 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7834108290 0x7f78341986e0 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:23.249 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:23.247+0000 7f783adb1700 1 --2- 172.21.15.46:0/1567775657 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7834069ef0 0x7f783419e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:23.249 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:23.247+0000 7f783adb1700 1 --2- 172.21.15.46:0/1567775657 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f783406a890 0x7f783419eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:23.249 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:23.247+0000 7f783adb1700 1 -- 172.21.15.46:0/1567775657 >> 172.21.15.46:0/1567775657 conn(0x7f7834078070 msgr2=0x7f78340fff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:10:23.249 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:23.247+0000 7f783adb1700 1 -- 172.21.15.46:0/1567775657 shutdown_connections 2024-03-31T23:10:23.249 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:23.247+0000 7f783adb1700 1 -- 172.21.15.46:0/1567775657 wait complete. 2024-03-31T23:10:24.295 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:10:24 smithi162 ceph-mon[36237]: pgmap v330: 129 pgs: 129 active+clean; 642 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-31T23:10:24.296 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:10:24 smithi162 ceph-mon[36237]: from='client.? 172.21.15.46:0/1567775657' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-31T23:10:24.337 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:10:24 smithi046 ceph-mon[32498]: pgmap v330: 129 pgs: 129 active+clean; 642 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-31T23:10:24.337 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:10:24 smithi046 ceph-mon[32498]: from='client.? 172.21.15.46:0/1567775657' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-31T23:10:24.443 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:10:24 smithi142 ceph-mon[35271]: pgmap v330: 129 pgs: 129 active+clean; 642 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-31T23:10:24.443 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:10:24 smithi142 ceph-mon[35271]: from='client.? 172.21.15.46:0/1567775657' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-31T23:10:26.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:26.059+0000 7f1d4792c700 1 -- 172.21.15.46:0/1828415766 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1d4006b2a0 msgr2=0x7f1d4010f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:10:26.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:26.059+0000 7f1d4792c700 1 --2- 172.21.15.46:0/1828415766 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1d4006b2a0 0x7f1d4010f0e0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f1d30009990 tx=0x7f1d30009ca0 comp rx=0 tx=0).stop 2024-03-31T23:10:26.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:26.059+0000 7f1d4792c700 1 -- 172.21.15.46:0/1828415766 shutdown_connections 2024-03-31T23:10:26.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:26.059+0000 7f1d4792c700 1 --2- 172.21.15.46:0/1828415766 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1d4006b2a0 0x7f1d4010f0e0 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:26.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:26.059+0000 7f1d4792c700 1 --2- 172.21.15.46:0/1828415766 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1d40108330 0x7f1d40108710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:26.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:26.059+0000 7f1d4792c700 1 --2- 172.21.15.46:0/1828415766 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1d4006a8e0 0x7f1d4006ad60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:26.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:26.059+0000 7f1d4792c700 1 -- 172.21.15.46:0/1828415766 >> 172.21.15.46:0/1828415766 conn(0x7f1d40078000 msgr2=0x7f1d40078410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:10:26.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:26.059+0000 7f1d4792c700 1 -- 172.21.15.46:0/1828415766 shutdown_connections 2024-03-31T23:10:26.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:26.059+0000 7f1d4792c700 1 -- 172.21.15.46:0/1828415766 wait complete. 2024-03-31T23:10:26.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:26.060+0000 7f1d4792c700 1 Processor -- start 2024-03-31T23:10:26.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:26.060+0000 7f1d4792c700 1 -- start start 2024-03-31T23:10:26.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:26.060+0000 7f1d4792c700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1d4006a8e0 0x7f1d401014f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:10:26.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:26.060+0000 7f1d4792c700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1d4006b2a0 0x7f1d40101a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:10:26.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:26.060+0000 7f1d4792c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1d40108330 0x7f1d40101f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:10:26.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:26.060+0000 7f1d4792c700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f1d40112400 con 0x7f1d40108330 2024-03-31T23:10:26.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:26.060+0000 7f1d4792c700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f1d40112280 con 0x7f1d4006b2a0 2024-03-31T23:10:26.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:26.060+0000 7f1d4792c700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f1d40112580 con 0x7f1d4006a8e0 2024-03-31T23:10:26.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:26.060+0000 7f1d45ec9700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1d40108330 0x7f1d40101f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:10:26.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:26.060+0000 7f1d44ec7700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1d4006b2a0 0x7f1d40101a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:10:26.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:26.060+0000 7f1d45ec9700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1d40108330 0x7f1d40101f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59422/0 (socket says 172.21.15.46:59422) 2024-03-31T23:10:26.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:26.060+0000 7f1d45ec9700 1 -- 172.21.15.46:0/3143267175 learned_addr learned my addr 172.21.15.46:0/3143267175 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:10:26.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:26.060+0000 7f1d456c8700 1 --2- 172.21.15.46:0/3143267175 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1d4006a8e0 0x7f1d401014f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:10:26.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:26.061+0000 7f1d45ec9700 1 -- 172.21.15.46:0/3143267175 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1d4006a8e0 msgr2=0x7f1d401014f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:10:26.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:26.061+0000 7f1d45ec9700 1 --2- 172.21.15.46:0/3143267175 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1d4006a8e0 0x7f1d401014f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:26.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:26.061+0000 7f1d45ec9700 1 -- 172.21.15.46:0/3143267175 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1d4006b2a0 msgr2=0x7f1d40101a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:10:26.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:26.061+0000 7f1d45ec9700 1 --2- 172.21.15.46:0/3143267175 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1d4006b2a0 0x7f1d40101a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:26.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:26.061+0000 7f1d45ec9700 1 -- 172.21.15.46:0/3143267175 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1d401a6db0 con 0x7f1d40108330 2024-03-31T23:10:26.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:26.061+0000 7f1d456c8700 1 --2- 172.21.15.46:0/3143267175 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1d4006a8e0 0x7f1d401014f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:10:26.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:26.061+0000 7f1d45ec9700 1 --2- 172.21.15.46:0/3143267175 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1d40108330 0x7f1d40101f70 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f1d300047b0 tx=0x7f1d3000bf70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:10:26.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:26.061+0000 7f1d3e7fc700 1 -- 172.21.15.46:0/3143267175 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d30027070 con 0x7f1d40108330 2024-03-31T23:10:26.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:26.061+0000 7f1d3e7fc700 1 -- 172.21.15.46:0/3143267175 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1d30022470 con 0x7f1d40108330 2024-03-31T23:10:26.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:26.061+0000 7f1d4792c700 1 -- 172.21.15.46:0/3143267175 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1d401a7040 con 0x7f1d40108330 2024-03-31T23:10:26.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:26.061+0000 7f1d3e7fc700 1 -- 172.21.15.46:0/3143267175 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d3000b610 con 0x7f1d40108330 2024-03-31T23:10:26.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:26.061+0000 7f1d4792c700 1 -- 172.21.15.46:0/3143267175 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1d401a7570 con 0x7f1d40108330 2024-03-31T23:10:26.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:26.062+0000 7f1d4792c700 1 -- 172.21.15.46:0/3143267175 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1d40068f00 con 0x7f1d40108330 2024-03-31T23:10:26.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:26.063+0000 7f1d3e7fc700 1 -- 172.21.15.46:0/3143267175 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 17) ==== 137371+0+0 (secure 0 0 0) 0x7f1d3002a070 con 0x7f1d40108330 2024-03-31T23:10:26.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:26.064+0000 7f1d3e7fc700 1 --2- 172.21.15.46:0/3143267175 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f1d280a2640 0x7f1d280a4b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:10:26.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:26.064+0000 7f1d456c8700 1 --2- 172.21.15.46:0/3143267175 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f1d280a2640 0x7f1d280a4b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:10:26.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:26.064+0000 7f1d3e7fc700 1 -- 172.21.15.46:0/3143267175 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7f1d300cfc50 con 0x7f1d40108330 2024-03-31T23:10:26.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:26.064+0000 7f1d456c8700 1 --2- 172.21.15.46:0/3143267175 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f1d280a2640 0x7f1d280a4b00 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f1d38009480 tx=0x7f1d3800b040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:10:26.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:26.068+0000 7f1d3e7fc700 1 -- 172.21.15.46:0/3143267175 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f1d30013070 con 0x7f1d40108330 2024-03-31T23:10:26.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:10:26 smithi046 ceph-mon[32498]: pgmap v331: 129 pgs: 129 active+clean; 642 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-31T23:10:26.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:26.285+0000 7f1d4792c700 1 -- 172.21.15.46:0/3143267175 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7f1d4002db30 con 0x7f1d40108330 2024-03-31T23:10:26.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:26.286+0000 7f1d3e7fc700 1 -- 172.21.15.46:0/3143267175 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f1d3009c980 con 0x7f1d40108330 2024-03-31T23:10:26.287 INFO:teuthology.orchestra.run.smithi046.stdout:141733920832 2024-03-31T23:10:26.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:26.288+0000 7f1d4792c700 1 -- 172.21.15.46:0/3143267175 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f1d280a2640 msgr2=0x7f1d280a4b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:10:26.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:26.288+0000 7f1d4792c700 1 --2- 172.21.15.46:0/3143267175 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f1d280a2640 0x7f1d280a4b00 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f1d38009480 tx=0x7f1d3800b040 comp rx=0 tx=0).stop 2024-03-31T23:10:26.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:26.288+0000 7f1d4792c700 1 -- 172.21.15.46:0/3143267175 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1d40108330 msgr2=0x7f1d40101f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:10:26.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:26.288+0000 7f1d4792c700 1 --2- 172.21.15.46:0/3143267175 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1d40108330 0x7f1d40101f70 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f1d300047b0 tx=0x7f1d3000bf70 comp rx=0 tx=0).stop 2024-03-31T23:10:26.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:26.288+0000 7f1d4792c700 1 -- 172.21.15.46:0/3143267175 shutdown_connections 2024-03-31T23:10:26.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:26.288+0000 7f1d4792c700 1 --2- 172.21.15.46:0/3143267175 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f1d280a2640 0x7f1d280a4b00 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:26.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:26.288+0000 7f1d4792c700 1 --2- 172.21.15.46:0/3143267175 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1d40108330 0x7f1d40101f70 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:26.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:26.288+0000 7f1d4792c700 1 --2- 172.21.15.46:0/3143267175 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1d4006a8e0 0x7f1d401014f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:26.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:26.288+0000 7f1d4792c700 1 --2- 172.21.15.46:0/3143267175 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1d4006b2a0 0x7f1d40101a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:26.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:26.288+0000 7f1d4792c700 1 -- 172.21.15.46:0/3143267175 >> 172.21.15.46:0/3143267175 conn(0x7f1d40078000 msgr2=0x7f1d401001a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:10:26.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:26.288+0000 7f1d4792c700 1 -- 172.21.15.46:0/3143267175 shutdown_connections 2024-03-31T23:10:26.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:26.288+0000 7f1d4792c700 1 -- 172.21.15.46:0/3143267175 wait complete. 2024-03-31T23:10:26.295 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:10:26 smithi162 ceph-mon[36237]: pgmap v331: 129 pgs: 129 active+clean; 642 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-31T23:10:26.443 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:10:26 smithi142 ceph-mon[35271]: pgmap v331: 129 pgs: 129 active+clean; 642 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-31T23:10:26.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:26.828+0000 7fc80a3c7700 1 -- 172.21.15.46:0/62022929 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc804101ae0 msgr2=0x7fc80410dfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:10:26.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:26.828+0000 7fc80a3c7700 1 --2- 172.21.15.46:0/62022929 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc804101ae0 0x7fc80410dfc0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fc7fc00b0a0 tx=0x7fc7fc00b3b0 comp rx=0 tx=0).stop 2024-03-31T23:10:26.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:26.828+0000 7fc80a3c7700 1 -- 172.21.15.46:0/62022929 shutdown_connections 2024-03-31T23:10:26.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:26.828+0000 7fc80a3c7700 1 --2- 172.21.15.46:0/62022929 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc804101ae0 0x7fc80410dfc0 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:26.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:26.828+0000 7fc80a3c7700 1 --2- 172.21.15.46:0/62022929 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc804107180 0x7fc804107560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:26.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:26.828+0000 7fc80a3c7700 1 --2- 172.21.15.46:0/62022929 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc804101140 0x7fc8041015a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:26.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:26.828+0000 7fc80a3c7700 1 -- 172.21.15.46:0/62022929 >> 172.21.15.46:0/62022929 conn(0x7fc804078070 msgr2=0x7fc804078480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:10:26.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:26.828+0000 7fc80a3c7700 1 -- 172.21.15.46:0/62022929 shutdown_connections 2024-03-31T23:10:26.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:26.828+0000 7fc80a3c7700 1 -- 172.21.15.46:0/62022929 wait complete. 2024-03-31T23:10:26.832 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:26.829+0000 7fc80a3c7700 1 Processor -- start 2024-03-31T23:10:26.832 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:26.829+0000 7fc80a3c7700 1 -- start start 2024-03-31T23:10:26.832 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:26.829+0000 7fc80a3c7700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc804101140 0x7fc80419d3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:10:26.832 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:26.829+0000 7fc80a3c7700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc804101ae0 0x7fc80419d910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:10:26.832 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:26.829+0000 7fc80a3c7700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc804107180 0x7fc804197590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:10:26.832 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:26.830+0000 7fc80a3c7700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fc8041112b0 con 0x7fc804101ae0 2024-03-31T23:10:26.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:26.830+0000 7fc80a3c7700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fc804111130 con 0x7fc804107180 2024-03-31T23:10:26.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:26.830+0000 7fc80a3c7700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fc804111430 con 0x7fc804101140 2024-03-31T23:10:26.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:26.830+0000 7fc8037fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc804101ae0 0x7fc80419d910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:10:26.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:26.830+0000 7fc8037fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc804101ae0 0x7fc80419d910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59454/0 (socket says 172.21.15.46:59454) 2024-03-31T23:10:26.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:26.830+0000 7fc8037fe700 1 -- 172.21.15.46:0/3638849729 learned_addr learned my addr 172.21.15.46:0/3638849729 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:10:26.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:26.830+0000 7fc803fff700 1 --2- 172.21.15.46:0/3638849729 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc804101140 0x7fc80419d3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:10:26.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:26.830+0000 7fc808964700 1 --2- 172.21.15.46:0/3638849729 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc804107180 0x7fc804197590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:10:26.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:26.830+0000 7fc8037fe700 1 -- 172.21.15.46:0/3638849729 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc804101140 msgr2=0x7fc80419d3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:10:26.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:26.830+0000 7fc8037fe700 1 --2- 172.21.15.46:0/3638849729 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc804101140 0x7fc80419d3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:26.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:26.830+0000 7fc8037fe700 1 -- 172.21.15.46:0/3638849729 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc804107180 msgr2=0x7fc804197590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:10:26.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:26.830+0000 7fc8037fe700 1 --2- 172.21.15.46:0/3638849729 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc804107180 0x7fc804197590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:26.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:26.830+0000 7fc8037fe700 1 -- 172.21.15.46:0/3638849729 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc804197cf0 con 0x7fc804101ae0 2024-03-31T23:10:26.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:26.830+0000 7fc8037fe700 1 --2- 172.21.15.46:0/3638849729 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc804101ae0 0x7fc80419d910 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fc7f800e9a0 tx=0x7fc7f800ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:10:26.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:26.830+0000 7fc8017fa700 1 -- 172.21.15.46:0/3638849729 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7f800ca10 con 0x7fc804101ae0 2024-03-31T23:10:26.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:26.830+0000 7fc8017fa700 1 -- 172.21.15.46:0/3638849729 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc7f800cbb0 con 0x7fc804101ae0 2024-03-31T23:10:26.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:26.830+0000 7fc80a3c7700 1 -- 172.21.15.46:0/3638849729 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc804197fe0 con 0x7fc804101ae0 2024-03-31T23:10:26.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:26.830+0000 7fc8017fa700 1 -- 172.21.15.46:0/3638849729 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7f8018680 con 0x7fc804101ae0 2024-03-31T23:10:26.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:26.830+0000 7fc80a3c7700 1 -- 172.21.15.46:0/3638849729 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc804069ef0 con 0x7fc804101ae0 2024-03-31T23:10:26.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:26.831+0000 7fc80a3c7700 1 -- 172.21.15.46:0/3638849729 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc804068f00 con 0x7fc804101ae0 2024-03-31T23:10:26.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:26.832+0000 7fc8017fa700 1 -- 172.21.15.46:0/3638849729 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 17) ==== 137371+0+0 (secure 0 0 0) 0x7fc7f8010430 con 0x7fc804101ae0 2024-03-31T23:10:26.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:26.833+0000 7fc8017fa700 1 --2- 172.21.15.46:0/3638849729 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fc7ec0a2640 0x7fc7ec0a4b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:10:26.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:26.833+0000 7fc803fff700 1 --2- 172.21.15.46:0/3638849729 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fc7ec0a2640 0x7fc7ec0a4b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:10:26.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:26.833+0000 7fc8017fa700 1 -- 172.21.15.46:0/3638849729 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7fc7f80ced50 con 0x7fc804101ae0 2024-03-31T23:10:26.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:26.833+0000 7fc803fff700 1 --2- 172.21.15.46:0/3638849729 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fc7ec0a2640 0x7fc7ec0a4b00 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fc7f4009be0 tx=0x7fc7f4009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:10:26.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:26.837+0000 7fc8017fa700 1 -- 172.21.15.46:0/3638849729 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fc7f809ba50 con 0x7fc804101ae0 2024-03-31T23:10:27.030 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:27.029+0000 7efdc61a7700 1 -- 172.21.15.46:0/2812473454 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efdc0107270 msgr2=0x7efdc0107650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:10:27.031 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:27.029+0000 7efdc61a7700 1 --2- 172.21.15.46:0/2812473454 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efdc0107270 0x7efdc0107650 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7efdb00099c0 tx=0x7efdb0009cd0 comp rx=0 tx=0).stop 2024-03-31T23:10:27.031 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:27.029+0000 7efdc61a7700 1 -- 172.21.15.46:0/2812473454 shutdown_connections 2024-03-31T23:10:27.031 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:27.029+0000 7efdc61a7700 1 --2- 172.21.15.46:0/2812473454 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efdc0107270 0x7efdc0107650 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:27.031 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:27.029+0000 7efdc61a7700 1 --2- 172.21.15.46:0/2812473454 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7efdc0101de0 0x7efdc010e000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:27.031 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:27.029+0000 7efdc61a7700 1 --2- 172.21.15.46:0/2812473454 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7efdc0101420 0x7efdc01018a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:27.031 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:27.029+0000 7efdc61a7700 1 -- 172.21.15.46:0/2812473454 >> 172.21.15.46:0/2812473454 conn(0x7efdc0078000 msgr2=0x7efdc0078410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:10:27.031 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:27.029+0000 7efdc61a7700 1 -- 172.21.15.46:0/2812473454 shutdown_connections 2024-03-31T23:10:27.031 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:27.029+0000 7efdc61a7700 1 -- 172.21.15.46:0/2812473454 wait complete. 2024-03-31T23:10:27.031 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:27.030+0000 7efdc61a7700 1 Processor -- start 2024-03-31T23:10:27.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:27.030+0000 7efdc61a7700 1 -- start start 2024-03-31T23:10:27.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:27.030+0000 7efdc61a7700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7efdc0101420 0x7efdc0102dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:10:27.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:27.030+0000 7efdc61a7700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efdc0101de0 0x7efdc0103310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:10:27.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:27.030+0000 7efdc61a7700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7efdc0107270 0x7efdc0103850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:10:27.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:27.030+0000 7efdc61a7700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7efdc01112d0 con 0x7efdc0101de0 2024-03-31T23:10:27.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:27.030+0000 7efdc61a7700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7efdc0111150 con 0x7efdc0101420 2024-03-31T23:10:27.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:27.030+0000 7efdc61a7700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7efdc0111450 con 0x7efdc0107270 2024-03-31T23:10:27.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:27.030+0000 7efdbeffd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efdc0101de0 0x7efdc0103310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:10:27.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:27.030+0000 7efdbf7fe700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7efdc0101420 0x7efdc0102dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:10:27.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:27.030+0000 7efdbf7fe700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7efdc0101420 0x7efdc0102dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.46:33506/0 (socket says 172.21.15.46:33506) 2024-03-31T23:10:27.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:27.030+0000 7efdbf7fe700 1 -- 172.21.15.46:0/183418106 learned_addr learned my addr 172.21.15.46:0/183418106 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:10:27.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:27.030+0000 7efdbffff700 1 --2- 172.21.15.46:0/183418106 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7efdc0107270 0x7efdc0103850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:10:27.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:27.031+0000 7efdbeffd700 1 -- 172.21.15.46:0/183418106 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7efdc0107270 msgr2=0x7efdc0103850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:10:27.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:27.031+0000 7efdbeffd700 1 --2- 172.21.15.46:0/183418106 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7efdc0107270 0x7efdc0103850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:27.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:27.031+0000 7efdbeffd700 1 -- 172.21.15.46:0/183418106 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7efdc0101420 msgr2=0x7efdc0102dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:10:27.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:27.031+0000 7efdbeffd700 1 --2- 172.21.15.46:0/183418106 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7efdc0101420 0x7efdc0102dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:27.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:27.031+0000 7efdbeffd700 1 -- 172.21.15.46:0/183418106 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efdc01a6e50 con 0x7efdc0101de0 2024-03-31T23:10:27.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:27.031+0000 7efdbf7fe700 1 --2- 172.21.15.46:0/183418106 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7efdc0101420 0x7efdc0102dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:10:27.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:27.031+0000 7efdbeffd700 1 --2- 172.21.15.46:0/183418106 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efdc0101de0 0x7efdc0103310 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7efdb400e9f0 tx=0x7efdb400ed00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:10:27.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:27.031+0000 7efdbcff9700 1 -- 172.21.15.46:0/183418106 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdb400c9c0 con 0x7efdc0101de0 2024-03-31T23:10:27.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:27.031+0000 7efdbcff9700 1 -- 172.21.15.46:0/183418106 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7efdb400cb60 con 0x7efdc0101de0 2024-03-31T23:10:27.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:27.031+0000 7efdbcff9700 1 -- 172.21.15.46:0/183418106 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdb4018680 con 0x7efdc0101de0 2024-03-31T23:10:27.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:27.031+0000 7efdc61a7700 1 -- 172.21.15.46:0/183418106 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efdc01a7140 con 0x7efdc0101de0 2024-03-31T23:10:27.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:27.031+0000 7efdc61a7700 1 -- 172.21.15.46:0/183418106 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efdc01a7680 con 0x7efdc0101de0 2024-03-31T23:10:27.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:27.032+0000 7efdc61a7700 1 -- 172.21.15.46:0/183418106 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efdc0068f00 con 0x7efdc0101de0 2024-03-31T23:10:27.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:27.033+0000 7efdbcff9700 1 -- 172.21.15.46:0/183418106 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 17) ==== 137371+0+0 (secure 0 0 0) 0x7efdb4010430 con 0x7efdc0101de0 2024-03-31T23:10:27.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:27.034+0000 7efdbcff9700 1 --2- 172.21.15.46:0/183418106 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7efda80a25c0 0x7efda80a4a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:10:27.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:27.034+0000 7efdbf7fe700 1 --2- 172.21.15.46:0/183418106 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7efda80a25c0 0x7efda80a4a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:10:27.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:27.034+0000 7efdbcff9700 1 -- 172.21.15.46:0/183418106 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7efdb40cf040 con 0x7efdc0101de0 2024-03-31T23:10:27.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:27.035+0000 7efdbf7fe700 1 --2- 172.21.15.46:0/183418106 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7efda80a25c0 0x7efda80a4a80 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7efdb0009990 tx=0x7efdb0005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:10:27.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:27.038+0000 7efdbcff9700 1 -- 172.21.15.46:0/183418106 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7efdb409bd40 con 0x7efdc0101de0 2024-03-31T23:10:27.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:27.050+0000 7fc80a3c7700 1 -- 172.21.15.46:0/3638849729 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7fc80406a8f0 con 0x7fc804101ae0 2024-03-31T23:10:27.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:27.050+0000 7fc8017fa700 1 -- 172.21.15.46:0/3638849729 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7fc7f809b350 con 0x7fc804101ae0 2024-03-31T23:10:27.051 INFO:teuthology.orchestra.run.smithi046.stdout:167503724603 2024-03-31T23:10:27.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:27.052+0000 7fc80a3c7700 1 -- 172.21.15.46:0/3638849729 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fc7ec0a2640 msgr2=0x7fc7ec0a4b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:10:27.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:27.052+0000 7fc80a3c7700 1 --2- 172.21.15.46:0/3638849729 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fc7ec0a2640 0x7fc7ec0a4b00 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fc7f4009be0 tx=0x7fc7f4009260 comp rx=0 tx=0).stop 2024-03-31T23:10:27.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:27.052+0000 7fc80a3c7700 1 -- 172.21.15.46:0/3638849729 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc804101ae0 msgr2=0x7fc80419d910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:10:27.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:27.052+0000 7fc80a3c7700 1 --2- 172.21.15.46:0/3638849729 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc804101ae0 0x7fc80419d910 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fc7f800e9a0 tx=0x7fc7f800ecb0 comp rx=0 tx=0).stop 2024-03-31T23:10:27.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:27.052+0000 7fc80a3c7700 1 -- 172.21.15.46:0/3638849729 shutdown_connections 2024-03-31T23:10:27.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:27.052+0000 7fc80a3c7700 1 --2- 172.21.15.46:0/3638849729 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fc7ec0a2640 0x7fc7ec0a4b00 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:27.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:27.052+0000 7fc80a3c7700 1 --2- 172.21.15.46:0/3638849729 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc804107180 0x7fc804197590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:27.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:27.052+0000 7fc80a3c7700 1 --2- 172.21.15.46:0/3638849729 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc804101140 0x7fc80419d3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:27.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:27.052+0000 7fc80a3c7700 1 --2- 172.21.15.46:0/3638849729 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc804101ae0 0x7fc80419d910 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:27.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:27.053+0000 7fc80a3c7700 1 -- 172.21.15.46:0/3638849729 >> 172.21.15.46:0/3638849729 conn(0x7fc804078070 msgr2=0x7fc8040ffea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:10:27.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:27.053+0000 7fc80a3c7700 1 -- 172.21.15.46:0/3638849729 shutdown_connections 2024-03-31T23:10:27.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:27.053+0000 7fc80a3c7700 1 -- 172.21.15.46:0/3638849729 wait complete. 2024-03-31T23:10:27.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:27.250+0000 7efdc61a7700 1 -- 172.21.15.46:0/183418106 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7efdc01a78e0 con 0x7efdc0101de0 2024-03-31T23:10:27.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:27.250+0000 7efdbcff9700 1 -- 172.21.15.46:0/183418106 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7efdb4097030 con 0x7efdc0101de0 2024-03-31T23:10:27.252 INFO:teuthology.orchestra.run.smithi046.stdout:55834574926 2024-03-31T23:10:27.254 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:27.252+0000 7efdc61a7700 1 -- 172.21.15.46:0/183418106 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7efda80a25c0 msgr2=0x7efda80a4a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:10:27.254 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:27.252+0000 7efdc61a7700 1 --2- 172.21.15.46:0/183418106 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7efda80a25c0 0x7efda80a4a80 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7efdb0009990 tx=0x7efdb0005960 comp rx=0 tx=0).stop 2024-03-31T23:10:27.254 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:27.252+0000 7efdc61a7700 1 -- 172.21.15.46:0/183418106 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efdc0101de0 msgr2=0x7efdc0103310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:10:27.254 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:27.252+0000 7efdc61a7700 1 --2- 172.21.15.46:0/183418106 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efdc0101de0 0x7efdc0103310 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7efdb400e9f0 tx=0x7efdb400ed00 comp rx=0 tx=0).stop 2024-03-31T23:10:27.254 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:27.252+0000 7efdc61a7700 1 -- 172.21.15.46:0/183418106 shutdown_connections 2024-03-31T23:10:27.254 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:27.252+0000 7efdc61a7700 1 --2- 172.21.15.46:0/183418106 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7efda80a25c0 0x7efda80a4a80 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:27.254 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:27.252+0000 7efdc61a7700 1 --2- 172.21.15.46:0/183418106 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7efdc0107270 0x7efdc0103850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:27.254 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:27.252+0000 7efdc61a7700 1 --2- 172.21.15.46:0/183418106 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7efdc0101420 0x7efdc0102dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:27.254 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:27.253+0000 7efdc61a7700 1 --2- 172.21.15.46:0/183418106 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efdc0101de0 0x7efdc0103310 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:27.254 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:27.253+0000 7efdc61a7700 1 -- 172.21.15.46:0/183418106 >> 172.21.15.46:0/183418106 conn(0x7efdc0078000 msgr2=0x7efdc006abc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:10:27.255 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:27.253+0000 7efdc61a7700 1 -- 172.21.15.46:0/183418106 shutdown_connections 2024-03-31T23:10:27.255 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:27.253+0000 7efdc61a7700 1 -- 172.21.15.46:0/183418106 wait complete. 2024-03-31T23:10:27.295 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:10:27 smithi162 ceph-mon[36237]: from='client.? 172.21.15.46:0/3143267175' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-31T23:10:27.361 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:10:27 smithi046 ceph-mon[32498]: from='client.? 172.21.15.46:0/3143267175' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-31T23:10:27.443 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:10:27 smithi142 ceph-mon[35271]: from='client.? 172.21.15.46:0/3143267175' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-31T23:10:28.295 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:10:28 smithi162 ceph-mon[36237]: pgmap v332: 129 pgs: 129 active+clean; 642 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-03-31T23:10:28.296 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:10:28 smithi162 ceph-mon[36237]: from='client.? 172.21.15.46:0/3638849729' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-31T23:10:28.296 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:10:28 smithi162 ceph-mon[36237]: from='client.? 172.21.15.46:0/183418106' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-31T23:10:28.353 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:10:28 smithi046 ceph-mon[32498]: pgmap v332: 129 pgs: 129 active+clean; 642 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-03-31T23:10:28.353 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:10:28 smithi046 ceph-mon[32498]: from='client.? 172.21.15.46:0/3638849729' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-31T23:10:28.353 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:10:28 smithi046 ceph-mon[32498]: from='client.? 172.21.15.46:0/183418106' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-31T23:10:28.443 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:10:28 smithi142 ceph-mon[35271]: pgmap v332: 129 pgs: 129 active+clean; 642 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-03-31T23:10:28.443 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:10:28 smithi142 ceph-mon[35271]: from='client.? 172.21.15.46:0/3638849729' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-31T23:10:28.443 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:10:28 smithi142 ceph-mon[35271]: from='client.? 172.21.15.46:0/183418106' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-31T23:10:30.361 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:10:30 smithi046 ceph-mon[32498]: pgmap v333: 129 pgs: 129 active+clean; 642 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-31T23:10:30.443 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:10:30 smithi142 ceph-mon[35271]: pgmap v333: 129 pgs: 129 active+clean; 642 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-31T23:10:30.481 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738447 got 34359738450 for osd.0 2024-03-31T23:10:30.482 DEBUG:teuthology.parallel:result is None 2024-03-31T23:10:30.545 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:10:30 smithi162 ceph-mon[36237]: pgmap v333: 129 pgs: 129 active+clean; 642 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-31T23:10:30.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:30.670+0000 7f805d348700 1 -- 172.21.15.46:0/2197019973 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8058102290 msgr2=0x7f8058102710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:10:30.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:30.670+0000 7f805d348700 1 --2- 172.21.15.46:0/2197019973 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8058102290 0x7f8058102710 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f804c009990 tx=0x7f804c009ca0 comp rx=0 tx=0).stop 2024-03-31T23:10:30.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:30.671+0000 7f805d348700 1 -- 172.21.15.46:0/2197019973 shutdown_connections 2024-03-31T23:10:30.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:30.671+0000 7f805d348700 1 --2- 172.21.15.46:0/2197019973 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8058102c50 0x7f805810f0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:30.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:30.671+0000 7f805d348700 1 --2- 172.21.15.46:0/2197019973 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8058108340 0x7f8058108720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:30.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:30.671+0000 7f805d348700 1 --2- 172.21.15.46:0/2197019973 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8058102290 0x7f8058102710 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:30.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:30.671+0000 7f805d348700 1 -- 172.21.15.46:0/2197019973 >> 172.21.15.46:0/2197019973 conn(0x7f8058079370 msgr2=0x7f8058079780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:10:30.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:30.671+0000 7f805d348700 1 -- 172.21.15.46:0/2197019973 shutdown_connections 2024-03-31T23:10:30.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:30.671+0000 7f805d348700 1 -- 172.21.15.46:0/2197019973 wait complete. 2024-03-31T23:10:30.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:30.671+0000 7f805d348700 1 Processor -- start 2024-03-31T23:10:30.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:30.671+0000 7f805d348700 1 -- start start 2024-03-31T23:10:30.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:30.672+0000 7f805d348700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8058102290 0x7f8058074540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:10:30.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:30.672+0000 7f805d348700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8058102c50 0x7f8058074a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:10:30.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:30.672+0000 7f805d348700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8058108340 0x7f8058074fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:10:30.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:30.672+0000 7f805d348700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f8058112430 con 0x7f8058102290 2024-03-31T23:10:30.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:30.672+0000 7f805d348700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f80581122b0 con 0x7f8058102c50 2024-03-31T23:10:30.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:30.672+0000 7f8056ffd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8058102290 0x7f8058074540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:10:30.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:30.672+0000 7f805d348700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f80581125b0 con 0x7f8058108340 2024-03-31T23:10:30.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:30.672+0000 7f8056ffd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8058102290 0x7f8058074540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59518/0 (socket says 172.21.15.46:59518) 2024-03-31T23:10:30.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:30.672+0000 7f8056ffd700 1 -- 172.21.15.46:0/121069887 learned_addr learned my addr 172.21.15.46:0/121069887 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:10:30.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:30.672+0000 7f80567fc700 1 --2- 172.21.15.46:0/121069887 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8058102c50 0x7f8058074a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:10:30.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:30.672+0000 7f80577fe700 1 --2- 172.21.15.46:0/121069887 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8058108340 0x7f8058074fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:10:30.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:30.672+0000 7f8056ffd700 1 -- 172.21.15.46:0/121069887 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8058108340 msgr2=0x7f8058074fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:10:30.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:30.672+0000 7f8056ffd700 1 --2- 172.21.15.46:0/121069887 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8058108340 0x7f8058074fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:30.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:30.672+0000 7f8056ffd700 1 -- 172.21.15.46:0/121069887 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8058102c50 msgr2=0x7f8058074a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:10:30.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:30.672+0000 7f8056ffd700 1 --2- 172.21.15.46:0/121069887 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8058102c50 0x7f8058074a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:30.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:30.672+0000 7f8056ffd700 1 -- 172.21.15.46:0/121069887 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f80581a6e30 con 0x7f8058102290 2024-03-31T23:10:30.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:30.672+0000 7f8056ffd700 1 --2- 172.21.15.46:0/121069887 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8058102290 0x7f8058074540 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f804800d980 tx=0x7f804800dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:10:30.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:30.672+0000 7f803ffff700 1 -- 172.21.15.46:0/121069887 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8048016070 con 0x7f8058102290 2024-03-31T23:10:30.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:30.672+0000 7f803ffff700 1 -- 172.21.15.46:0/121069887 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8048004830 con 0x7f8058102290 2024-03-31T23:10:30.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:30.672+0000 7f805d348700 1 -- 172.21.15.46:0/121069887 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f80581a7120 con 0x7f8058102290 2024-03-31T23:10:30.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:30.673+0000 7f803ffff700 1 -- 172.21.15.46:0/121069887 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8048005530 con 0x7f8058102290 2024-03-31T23:10:30.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:30.673+0000 7f805d348700 1 -- 172.21.15.46:0/121069887 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f80581a7630 con 0x7f8058102290 2024-03-31T23:10:30.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:30.674+0000 7f805d348700 1 -- 172.21.15.46:0/121069887 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8058103980 con 0x7f8058102290 2024-03-31T23:10:30.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:30.675+0000 7f803ffff700 1 -- 172.21.15.46:0/121069887 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 17) ==== 137371+0+0 (secure 0 0 0) 0x7f80480049e0 con 0x7f8058102290 2024-03-31T23:10:30.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:30.675+0000 7f803ffff700 1 --2- 172.21.15.46:0/121069887 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f80400a25c0 0x7f80400a4a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:10:30.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:30.675+0000 7f80567fc700 1 --2- 172.21.15.46:0/121069887 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f80400a25c0 0x7f80400a4a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:10:30.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:30.675+0000 7f803ffff700 1 -- 172.21.15.46:0/121069887 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7f80480cedc0 con 0x7f8058102290 2024-03-31T23:10:30.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:30.676+0000 7f80567fc700 1 --2- 172.21.15.46:0/121069887 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f80400a25c0 0x7f80400a4a80 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f804c00b2e0 tx=0x7f804c005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:10:30.680 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:30.679+0000 7f803ffff700 1 -- 172.21.15.46:0/121069887 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f804801f020 con 0x7f8058102290 2024-03-31T23:10:30.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:30.886+0000 7f805d348700 1 -- 172.21.15.46:0/121069887 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 8} v 0) -- 0x7f805802db30 con 0x7f8058102290 2024-03-31T23:10:30.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:30.886+0000 7f803ffff700 1 -- 172.21.15.46:0/121069887 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 8}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f804809bac0 con 0x7f8058102290 2024-03-31T23:10:30.887 INFO:teuthology.orchestra.run.smithi046.stdout:257698037804 2024-03-31T23:10:30.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:30.888+0000 7f805d348700 1 -- 172.21.15.46:0/121069887 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f80400a25c0 msgr2=0x7f80400a4a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:10:30.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:30.888+0000 7f805d348700 1 --2- 172.21.15.46:0/121069887 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f80400a25c0 0x7f80400a4a80 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f804c00b2e0 tx=0x7f804c005960 comp rx=0 tx=0).stop 2024-03-31T23:10:30.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:30.888+0000 7f805d348700 1 -- 172.21.15.46:0/121069887 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8058102290 msgr2=0x7f8058074540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:10:30.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:30.888+0000 7f805d348700 1 --2- 172.21.15.46:0/121069887 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8058102290 0x7f8058074540 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f804800d980 tx=0x7f804800dc90 comp rx=0 tx=0).stop 2024-03-31T23:10:30.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:30.889+0000 7f805d348700 1 -- 172.21.15.46:0/121069887 shutdown_connections 2024-03-31T23:10:30.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:30.889+0000 7f805d348700 1 --2- 172.21.15.46:0/121069887 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f80400a25c0 0x7f80400a4a80 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:30.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:30.889+0000 7f805d348700 1 --2- 172.21.15.46:0/121069887 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8058102290 0x7f8058074540 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:30.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:30.889+0000 7f805d348700 1 --2- 172.21.15.46:0/121069887 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8058108340 0x7f8058074fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:30.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:30.889+0000 7f805d348700 1 --2- 172.21.15.46:0/121069887 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8058102c50 0x7f8058074a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:30.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:30.889+0000 7f805d348700 1 -- 172.21.15.46:0/121069887 >> 172.21.15.46:0/121069887 conn(0x7f8058079370 msgr2=0x7f80581011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:10:30.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:30.889+0000 7f805d348700 1 -- 172.21.15.46:0/121069887 shutdown_connections 2024-03-31T23:10:30.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:30.889+0000 7f805d348700 1 -- 172.21.15.46:0/121069887 wait complete. 2024-03-31T23:10:31.211 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:10:31 smithi046 ceph-mon[32498]: from='client.? 172.21.15.46:0/1510664387' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-31T23:10:31.211 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:10:31 smithi046 ceph-mon[32498]: from='client.? 172.21.15.46:0/121069887' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-03-31T23:10:31.243 INFO:tasks.cephadm.ceph_manager.ceph:need seq 141733920828 got 141733920832 for osd.4 2024-03-31T23:10:31.243 DEBUG:teuthology.parallel:result is None 2024-03-31T23:10:31.295 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:10:31 smithi162 ceph-mon[36237]: from='client.? 172.21.15.46:0/1510664387' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-31T23:10:31.295 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:10:31 smithi162 ceph-mon[36237]: from='client.? 172.21.15.46:0/121069887' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-03-31T23:10:31.443 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:10:31 smithi142 ceph-mon[35271]: from='client.? 172.21.15.46:0/1510664387' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-31T23:10:31.443 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:10:31 smithi142 ceph-mon[35271]: from='client.? 172.21.15.46:0/121069887' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-03-31T23:10:31.693 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574920 got 55834574926 for osd.1 2024-03-31T23:10:31.693 DEBUG:teuthology.parallel:result is None 2024-03-31T23:10:31.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:29.869+0000 7f8d596b3700 1 -- 172.21.15.46:0/227108916 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8d54107550 msgr2=0x7f8d5410f120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:10:31.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:29.869+0000 7f8d596b3700 1 --2- 172.21.15.46:0/227108916 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8d54107550 0x7f8d5410f120 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f8d4c00b0a0 tx=0x7f8d4c00b3b0 comp rx=0 tx=0).stop 2024-03-31T23:10:31.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:29.869+0000 7f8d596b3700 1 -- 172.21.15.46:0/227108916 shutdown_connections 2024-03-31T23:10:31.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:29.869+0000 7f8d596b3700 1 --2- 172.21.15.46:0/227108916 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8d54107550 0x7f8d5410f120 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:31.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:29.869+0000 7f8d596b3700 1 --2- 172.21.15.46:0/227108916 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8d54102520 0x7f8d54102900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:31.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:29.869+0000 7f8d596b3700 1 --2- 172.21.15.46:0/227108916 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8d54102ed0 0x7f8d54106e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:31.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:29.869+0000 7f8d596b3700 1 -- 172.21.15.46:0/227108916 >> 172.21.15.46:0/227108916 conn(0x7f8d540792d0 msgr2=0x7f8d540796e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:10:31.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:29.869+0000 7f8d596b3700 1 -- 172.21.15.46:0/227108916 shutdown_connections 2024-03-31T23:10:31.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:29.869+0000 7f8d596b3700 1 -- 172.21.15.46:0/227108916 wait complete. 2024-03-31T23:10:31.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:29.870+0000 7f8d596b3700 1 Processor -- start 2024-03-31T23:10:31.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:29.870+0000 7f8d596b3700 1 -- start start 2024-03-31T23:10:31.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:29.870+0000 7f8d596b3700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8d54102520 0x7f8d5419e590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:10:31.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:29.870+0000 7f8d596b3700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8d54102ed0 0x7f8d5419ead0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:10:31.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:29.870+0000 7f8d596b3700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8d54107550 0x7f8d54198660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:10:31.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:29.870+0000 7f8d596b3700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f8d541123e0 con 0x7f8d54102ed0 2024-03-31T23:10:31.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:29.870+0000 7f8d596b3700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f8d54112260 con 0x7f8d54102520 2024-03-31T23:10:31.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:29.870+0000 7f8d596b3700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f8d54112560 con 0x7f8d54107550 2024-03-31T23:10:31.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:29.870+0000 7f8d527fc700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8d54102ed0 0x7f8d5419ead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:10:31.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:29.870+0000 7f8d52ffd700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8d54102520 0x7f8d5419e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:10:31.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:29.870+0000 7f8d527fc700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8d54102ed0 0x7f8d5419ead0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59500/0 (socket says 172.21.15.46:59500) 2024-03-31T23:10:31.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:29.870+0000 7f8d527fc700 1 -- 172.21.15.46:0/1510664387 learned_addr learned my addr 172.21.15.46:0/1510664387 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:10:31.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:29.870+0000 7f8d537fe700 1 --2- 172.21.15.46:0/1510664387 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8d54107550 0x7f8d54198660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:10:31.792 INFO:teuthology.orchestra.run.smithi046.stdout:77309411403 2024-03-31T23:10:31.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:29.870+0000 7f8d527fc700 1 -- 172.21.15.46:0/1510664387 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8d54107550 msgr2=0x7f8d54198660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:10:31.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:29.870+0000 7f8d527fc700 1 --2- 172.21.15.46:0/1510664387 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8d54107550 0x7f8d54198660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:31.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:29.870+0000 7f8d527fc700 1 -- 172.21.15.46:0/1510664387 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8d54102520 msgr2=0x7f8d5419e590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:10:31.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:29.870+0000 7f8d527fc700 1 --2- 172.21.15.46:0/1510664387 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8d54102520 0x7f8d5419e590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:31.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:29.870+0000 7f8d527fc700 1 -- 172.21.15.46:0/1510664387 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8d54198f40 con 0x7f8d54102ed0 2024-03-31T23:10:31.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:29.870+0000 7f8d537fe700 1 --2- 172.21.15.46:0/1510664387 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8d54107550 0x7f8d54198660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:10:31.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:29.870+0000 7f8d52ffd700 1 --2- 172.21.15.46:0/1510664387 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8d54102520 0x7f8d5419e590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:10:31.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:29.871+0000 7f8d527fc700 1 --2- 172.21.15.46:0/1510664387 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8d54102ed0 0x7f8d5419ead0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f8d4800e970 tx=0x7f8d4800ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:10:31.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:29.871+0000 7f8d3bfff700 1 -- 172.21.15.46:0/1510664387 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d48004a30 con 0x7f8d54102ed0 2024-03-31T23:10:31.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:29.871+0000 7f8d3bfff700 1 -- 172.21.15.46:0/1510664387 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8d4800cc90 con 0x7f8d54102ed0 2024-03-31T23:10:31.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:29.871+0000 7f8d596b3700 1 -- 172.21.15.46:0/1510664387 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8d54199230 con 0x7f8d54102ed0 2024-03-31T23:10:31.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:29.871+0000 7f8d3bfff700 1 -- 172.21.15.46:0/1510664387 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d480055e0 con 0x7f8d54102ed0 2024-03-31T23:10:31.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:29.871+0000 7f8d596b3700 1 -- 172.21.15.46:0/1510664387 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8d541a5510 con 0x7f8d54102ed0 2024-03-31T23:10:31.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:29.872+0000 7f8d596b3700 1 -- 172.21.15.46:0/1510664387 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8d54103950 con 0x7f8d54102ed0 2024-03-31T23:10:31.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:29.873+0000 7f8d3bfff700 1 -- 172.21.15.46:0/1510664387 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 17) ==== 137371+0+0 (secure 0 0 0) 0x7f8d4800c7e0 con 0x7f8d54102ed0 2024-03-31T23:10:31.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:29.873+0000 7f8d3bfff700 1 --2- 172.21.15.46:0/1510664387 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f8d3c0a2640 0x7f8d3c0a4b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:10:31.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:29.874+0000 7f8d52ffd700 1 --2- 172.21.15.46:0/1510664387 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f8d3c0a2640 0x7f8d3c0a4b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:10:31.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:29.874+0000 7f8d3bfff700 1 -- 172.21.15.46:0/1510664387 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7f8d480cecf0 con 0x7f8d54102ed0 2024-03-31T23:10:31.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:29.874+0000 7f8d52ffd700 1 --2- 172.21.15.46:0/1510664387 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f8d3c0a2640 0x7f8d3c0a4b00 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f8d44011490 tx=0x7f8d44009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:10:31.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:29.878+0000 7f8d3bfff700 1 -- 172.21.15.46:0/1510664387 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f8d4809b9f0 con 0x7f8d54102ed0 2024-03-31T23:10:31.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:30.085+0000 7f8d596b3700 1 -- 172.21.15.46:0/1510664387 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7f8d5402db30 con 0x7f8d54102ed0 2024-03-31T23:10:31.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:30.085+0000 7f8d3bfff700 1 -- 172.21.15.46:0/1510664387 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f8d4809b2f0 con 0x7f8d54102ed0 2024-03-31T23:10:31.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:30.087+0000 7f8d596b3700 1 -- 172.21.15.46:0/1510664387 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f8d3c0a2640 msgr2=0x7f8d3c0a4b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:10:31.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:30.087+0000 7f8d596b3700 1 --2- 172.21.15.46:0/1510664387 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f8d3c0a2640 0x7f8d3c0a4b00 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f8d44011490 tx=0x7f8d44009260 comp rx=0 tx=0).stop 2024-03-31T23:10:31.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:30.087+0000 7f8d596b3700 1 -- 172.21.15.46:0/1510664387 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8d54102ed0 msgr2=0x7f8d5419ead0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:10:31.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:30.087+0000 7f8d596b3700 1 --2- 172.21.15.46:0/1510664387 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8d54102ed0 0x7f8d5419ead0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f8d4800e970 tx=0x7f8d4800ec80 comp rx=0 tx=0).stop 2024-03-31T23:10:31.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:30.087+0000 7f8d596b3700 1 -- 172.21.15.46:0/1510664387 shutdown_connections 2024-03-31T23:10:31.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:30.087+0000 7f8d596b3700 1 --2- 172.21.15.46:0/1510664387 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f8d3c0a2640 0x7f8d3c0a4b00 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:31.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:30.087+0000 7f8d596b3700 1 --2- 172.21.15.46:0/1510664387 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8d54107550 0x7f8d54198660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:31.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:30.087+0000 7f8d596b3700 1 --2- 172.21.15.46:0/1510664387 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8d54102520 0x7f8d5419e590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:31.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:30.087+0000 7f8d596b3700 1 --2- 172.21.15.46:0/1510664387 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8d54102ed0 0x7f8d5419ead0 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:31.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:30.087+0000 7f8d596b3700 1 -- 172.21.15.46:0/1510664387 >> 172.21.15.46:0/1510664387 conn(0x7f8d540792d0 msgr2=0x7f8d541005a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:10:31.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:30.087+0000 7f8d596b3700 1 -- 172.21.15.46:0/1510664387 shutdown_connections 2024-03-31T23:10:31.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:30.087+0000 7f8d596b3700 1 -- 172.21.15.46:0/1510664387 wait complete. 2024-03-31T23:10:32.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:32.233+0000 7f705b1b3700 1 -- 172.21.15.46:0/1987308876 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f70541024f0 msgr2=0x7f70541028d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:10:32.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:32.233+0000 7f705b1b3700 1 --2- 172.21.15.46:0/1987308876 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f70541024f0 0x7f70541028d0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f70480099c0 tx=0x7f7048009cd0 comp rx=0 tx=0).stop 2024-03-31T23:10:32.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:32.233+0000 7f705b1b3700 1 -- 172.21.15.46:0/1987308876 shutdown_connections 2024-03-31T23:10:32.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:32.233+0000 7f705b1b3700 1 --2- 172.21.15.46:0/1987308876 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f70541024f0 0x7f70541028d0 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:32.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:32.233+0000 7f705b1b3700 1 --2- 172.21.15.46:0/1987308876 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f70541075c0 0x7f705410f190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:32.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:32.233+0000 7f705b1b3700 1 --2- 172.21.15.46:0/1987308876 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7054102ea0 0x7f7054106ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:32.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:32.233+0000 7f705b1b3700 1 -- 172.21.15.46:0/1987308876 >> 172.21.15.46:0/1987308876 conn(0x7f70540793a0 msgr2=0x7f70540797b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:10:32.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:32.233+0000 7f705b1b3700 1 -- 172.21.15.46:0/1987308876 shutdown_connections 2024-03-31T23:10:32.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:32.233+0000 7f705b1b3700 1 -- 172.21.15.46:0/1987308876 wait complete. 2024-03-31T23:10:32.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:32.234+0000 7f705b1b3700 1 Processor -- start 2024-03-31T23:10:32.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:32.234+0000 7f705b1b3700 1 -- start start 2024-03-31T23:10:32.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:32.234+0000 7f705b1b3700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f70541024f0 0x7f705419e650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:10:32.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:32.234+0000 7f705b1b3700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7054102ea0 0x7f705419eb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:10:32.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:32.234+0000 7f705b1b3700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f70541075c0 0x7f70541987d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:10:32.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:32.234+0000 7f705b1b3700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f7054112450 con 0x7f70541024f0 2024-03-31T23:10:32.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:32.234+0000 7f705b1b3700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f70541122d0 con 0x7f7054102ea0 2024-03-31T23:10:32.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:32.234+0000 7f705b1b3700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f70541125d0 con 0x7f70541075c0 2024-03-31T23:10:32.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:32.234+0000 7f7058f4f700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f70541024f0 0x7f705419e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:10:32.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:32.234+0000 7f7058f4f700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f70541024f0 0x7f705419e650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44984/0 (socket says 172.21.15.46:44984) 2024-03-31T23:10:32.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:32.234+0000 7f7058f4f700 1 -- 172.21.15.46:0/2136694948 learned_addr learned my addr 172.21.15.46:0/2136694948 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:10:32.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:32.234+0000 7f7053fff700 1 --2- 172.21.15.46:0/2136694948 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7054102ea0 0x7f705419eb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:10:32.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:32.234+0000 7f7059750700 1 --2- 172.21.15.46:0/2136694948 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f70541075c0 0x7f70541987d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:10:32.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:32.234+0000 7f7058f4f700 1 -- 172.21.15.46:0/2136694948 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f70541075c0 msgr2=0x7f70541987d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:10:32.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:32.234+0000 7f7058f4f700 1 --2- 172.21.15.46:0/2136694948 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f70541075c0 0x7f70541987d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:32.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:32.234+0000 7f7058f4f700 1 -- 172.21.15.46:0/2136694948 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7054102ea0 msgr2=0x7f705419eb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:10:32.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:32.234+0000 7f7058f4f700 1 --2- 172.21.15.46:0/2136694948 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7054102ea0 0x7f705419eb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:32.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:32.234+0000 7f7058f4f700 1 -- 172.21.15.46:0/2136694948 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7054198fb0 con 0x7f70541024f0 2024-03-31T23:10:32.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:32.235+0000 7f7058f4f700 1 --2- 172.21.15.46:0/2136694948 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f70541024f0 0x7f705419e650 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f7048006010 tx=0x7f7048005280 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:10:32.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:32.235+0000 7f7051ffb700 1 -- 172.21.15.46:0/2136694948 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7048026070 con 0x7f70541024f0 2024-03-31T23:10:32.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:32.235+0000 7f7051ffb700 1 -- 172.21.15.46:0/2136694948 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7048003c70 con 0x7f70541024f0 2024-03-31T23:10:32.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:32.235+0000 7f705b1b3700 1 -- 172.21.15.46:0/2136694948 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7054199240 con 0x7f70541024f0 2024-03-31T23:10:32.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:32.235+0000 7f7051ffb700 1 -- 172.21.15.46:0/2136694948 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f704800fbd0 con 0x7f70541024f0 2024-03-31T23:10:32.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:32.235+0000 7f705b1b3700 1 -- 172.21.15.46:0/2136694948 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f70541a54b0 con 0x7f70541024f0 2024-03-31T23:10:32.240 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:32.236+0000 7f705b1b3700 1 -- 172.21.15.46:0/2136694948 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f70541039c0 con 0x7f70541024f0 2024-03-31T23:10:32.240 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:32.237+0000 7f7051ffb700 1 -- 172.21.15.46:0/2136694948 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 17) ==== 137371+0+0 (secure 0 0 0) 0x7f7048029070 con 0x7f70541024f0 2024-03-31T23:10:32.240 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:32.238+0000 7f7051ffb700 1 --2- 172.21.15.46:0/2136694948 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f70440a2640 0x7f70440a4b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:10:32.240 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:32.238+0000 7f7053fff700 1 --2- 172.21.15.46:0/2136694948 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f70440a2640 0x7f70440a4b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:10:32.240 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:32.238+0000 7f7051ffb700 1 -- 172.21.15.46:0/2136694948 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7f70480cfaa0 con 0x7f70541024f0 2024-03-31T23:10:32.240 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:32.238+0000 7f7053fff700 1 --2- 172.21.15.46:0/2136694948 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f70440a2640 0x7f70440a4b00 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f7040009680 tx=0x7f7040009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:10:32.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:32.242+0000 7f7051ffb700 1 -- 172.21.15.46:0/2136694948 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f7048030020 con 0x7f70541024f0 2024-03-31T23:10:32.344 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:10:32 smithi046 ceph-mon[32498]: pgmap v334: 129 pgs: 129 active+clean; 642 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-03-31T23:10:32.443 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:10:32 smithi142 ceph-mon[35271]: pgmap v334: 129 pgs: 129 active+clean; 642 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-03-31T23:10:32.456 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:32.455+0000 7f705b1b3700 1 -- 172.21.15.46:0/2136694948 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 10} v 0) -- 0x7f7054199d00 con 0x7f70541024f0 2024-03-31T23:10:32.456 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:32.455+0000 7f7051ffb700 1 -- 172.21.15.46:0/2136694948 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 10}]=0 v0) ==== 75+0+13 (secure 0 0 0) 0x7f7048013070 con 0x7f70541024f0 2024-03-31T23:10:32.457 INFO:teuthology.orchestra.run.smithi046.stdout:313532612643 2024-03-31T23:10:32.459 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:32.458+0000 7f705b1b3700 1 -- 172.21.15.46:0/2136694948 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f70440a2640 msgr2=0x7f70440a4b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:10:32.459 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:32.458+0000 7f705b1b3700 1 --2- 172.21.15.46:0/2136694948 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f70440a2640 0x7f70440a4b00 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f7040009680 tx=0x7f7040009260 comp rx=0 tx=0).stop 2024-03-31T23:10:32.459 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:32.458+0000 7f705b1b3700 1 -- 172.21.15.46:0/2136694948 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f70541024f0 msgr2=0x7f705419e650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:10:32.459 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:32.458+0000 7f705b1b3700 1 --2- 172.21.15.46:0/2136694948 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f70541024f0 0x7f705419e650 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f7048006010 tx=0x7f7048005280 comp rx=0 tx=0).stop 2024-03-31T23:10:32.459 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:32.458+0000 7f705b1b3700 1 -- 172.21.15.46:0/2136694948 shutdown_connections 2024-03-31T23:10:32.459 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:32.458+0000 7f705b1b3700 1 --2- 172.21.15.46:0/2136694948 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f70440a2640 0x7f70440a4b00 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:32.459 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:32.458+0000 7f705b1b3700 1 --2- 172.21.15.46:0/2136694948 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f70541024f0 0x7f705419e650 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:32.459 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:32.458+0000 7f705b1b3700 1 --2- 172.21.15.46:0/2136694948 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f70541075c0 0x7f70541987d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:32.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:32.458+0000 7f705b1b3700 1 --2- 172.21.15.46:0/2136694948 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7054102ea0 0x7f705419eb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:32.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:32.458+0000 7f705b1b3700 1 -- 172.21.15.46:0/2136694948 >> 172.21.15.46:0/2136694948 conn(0x7f70540793a0 msgr2=0x7f7054100560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:10:32.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:32.458+0000 7f705b1b3700 1 -- 172.21.15.46:0/2136694948 shutdown_connections 2024-03-31T23:10:32.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:32.458+0000 7f705b1b3700 1 -- 172.21.15.46:0/2136694948 wait complete. 2024-03-31T23:10:32.545 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:10:32 smithi162 ceph-mon[36237]: pgmap v334: 129 pgs: 129 active+clean; 642 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-03-31T23:10:33.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:10:33 smithi046 ceph-mon[32498]: from='client.? 172.21.15.46:0/2136694948' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-03-31T23:10:33.426 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/6a9d8f08-efb2-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-31T23:10:33.443 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:10:33 smithi142 ceph-mon[35271]: from='client.? 172.21.15.46:0/2136694948' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-03-31T23:10:33.545 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:10:33 smithi162 ceph-mon[36237]: from='client.? 172.21.15.46:0/2136694948' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-03-31T23:10:33.861 INFO:tasks.cephadm.ceph_manager.ceph:need seq 167503724598 got 167503724603 for osd.5 2024-03-31T23:10:33.862 DEBUG:teuthology.parallel:result is None 2024-03-31T23:10:34.159 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:32.910+0000 7f56b64b6700 1 -- 172.21.15.46:0/3927095882 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f56b0101ae0 msgr2=0x7f56b010dfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:10:34.159 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:32.910+0000 7f56b64b6700 1 --2- 172.21.15.46:0/3927095882 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f56b0101ae0 0x7f56b010dfc0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f56a800b0a0 tx=0x7f56a800b3b0 comp rx=0 tx=0).stop 2024-03-31T23:10:34.159 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:32.910+0000 7f56b64b6700 1 -- 172.21.15.46:0/3927095882 shutdown_connections 2024-03-31T23:10:34.159 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:32.910+0000 7f56b64b6700 1 --2- 172.21.15.46:0/3927095882 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f56b0101ae0 0x7f56b010dfc0 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:34.159 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:32.910+0000 7f56b64b6700 1 --2- 172.21.15.46:0/3927095882 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f56b0107180 0x7f56b0107560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:34.159 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:32.910+0000 7f56b64b6700 1 --2- 172.21.15.46:0/3927095882 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f56b0101140 0x7f56b01015a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:34.159 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:32.910+0000 7f56b64b6700 1 -- 172.21.15.46:0/3927095882 >> 172.21.15.46:0/3927095882 conn(0x7f56b0078070 msgr2=0x7f56b0078480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:10:34.159 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:32.910+0000 7f56b64b6700 1 -- 172.21.15.46:0/3927095882 shutdown_connections 2024-03-31T23:10:34.159 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:32.911+0000 7f56b64b6700 1 -- 172.21.15.46:0/3927095882 wait complete. 2024-03-31T23:10:34.159 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:32.911+0000 7f56b64b6700 1 Processor -- start 2024-03-31T23:10:34.159 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:32.911+0000 7f56b64b6700 1 -- start start 2024-03-31T23:10:34.159 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:32.911+0000 7f56b64b6700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f56b0101140 0x7f56b0199790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:10:34.159 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:32.911+0000 7f56b64b6700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f56b0101ae0 0x7f56b0199cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:10:34.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:32.911+0000 7f56b64b6700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f56b0107180 0x7f56b01a4810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:10:34.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:32.911+0000 7f56b64b6700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f56b01112b0 con 0x7f56b0101ae0 2024-03-31T23:10:34.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:32.911+0000 7f56b64b6700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f56b0111130 con 0x7f56b0101140 2024-03-31T23:10:34.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:32.911+0000 7f56b64b6700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f56b0111430 con 0x7f56b0107180 2024-03-31T23:10:34.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:32.912+0000 7f56af7fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f56b0101ae0 0x7f56b0199cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:10:34.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:32.912+0000 7f56af7fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f56b0101ae0 0x7f56b0199cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:45004/0 (socket says 172.21.15.46:45004) 2024-03-31T23:10:34.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:32.912+0000 7f56af7fe700 1 -- 172.21.15.46:0/305231135 learned_addr learned my addr 172.21.15.46:0/305231135 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:10:34.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:32.912+0000 7f56affff700 1 --2- 172.21.15.46:0/305231135 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f56b0101140 0x7f56b0199790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:10:34.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:32.912+0000 7f56b4a53700 1 --2- 172.21.15.46:0/305231135 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f56b0107180 0x7f56b01a4810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:10:34.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:32.912+0000 7f56af7fe700 1 -- 172.21.15.46:0/305231135 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f56b0107180 msgr2=0x7f56b01a4810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:10:34.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:32.912+0000 7f56af7fe700 1 --2- 172.21.15.46:0/305231135 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f56b0107180 0x7f56b01a4810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:34.161 INFO:teuthology.orchestra.run.smithi046.stdout:107374182470 2024-03-31T23:10:34.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:32.912+0000 7f56af7fe700 1 -- 172.21.15.46:0/305231135 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f56b0101140 msgr2=0x7f56b0199790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:10:34.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:32.912+0000 7f56af7fe700 1 --2- 172.21.15.46:0/305231135 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f56b0101140 0x7f56b0199790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:34.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:32.912+0000 7f56af7fe700 1 -- 172.21.15.46:0/305231135 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f56b01a4d50 con 0x7f56b0101ae0 2024-03-31T23:10:34.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:32.912+0000 7f56af7fe700 1 --2- 172.21.15.46:0/305231135 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f56b0101ae0 0x7f56b0199cd0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f56a400d980 tx=0x7f56a400dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:10:34.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:32.912+0000 7f56ad7fa700 1 -- 172.21.15.46:0/305231135 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56a4017070 con 0x7f56b0101ae0 2024-03-31T23:10:34.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:32.912+0000 7f56b64b6700 1 -- 172.21.15.46:0/305231135 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f56b01a4f80 con 0x7f56b0101ae0 2024-03-31T23:10:34.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:32.912+0000 7f56ad7fa700 1 -- 172.21.15.46:0/305231135 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f56a4005020 con 0x7f56b0101ae0 2024-03-31T23:10:34.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:32.912+0000 7f56ad7fa700 1 -- 172.21.15.46:0/305231135 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56a4009b50 con 0x7f56b0101ae0 2024-03-31T23:10:34.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:32.912+0000 7f56b64b6700 1 -- 172.21.15.46:0/305231135 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f56b01a5430 con 0x7f56b0101ae0 2024-03-31T23:10:34.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:32.913+0000 7f56b64b6700 1 -- 172.21.15.46:0/305231135 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f56b0068f00 con 0x7f56b0101ae0 2024-03-31T23:10:34.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:32.914+0000 7f56ad7fa700 1 -- 172.21.15.46:0/305231135 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 17) ==== 137371+0+0 (secure 0 0 0) 0x7f56a4009cf0 con 0x7f56b0101ae0 2024-03-31T23:10:34.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:32.915+0000 7f56ad7fa700 1 --2- 172.21.15.46:0/305231135 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f56980a25c0 0x7f56980a4a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:10:34.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:32.915+0000 7f56affff700 1 --2- 172.21.15.46:0/305231135 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f56980a25c0 0x7f56980a4a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:10:34.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:32.915+0000 7f56ad7fa700 1 -- 172.21.15.46:0/305231135 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7f56a40ceb30 con 0x7f56b0101ae0 2024-03-31T23:10:34.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:32.915+0000 7f56affff700 1 --2- 172.21.15.46:0/305231135 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f56980a25c0 0x7f56980a4a80 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f56a0005950 tx=0x7f56a000b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:10:34.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:32.919+0000 7f56ad7fa700 1 -- 172.21.15.46:0/305231135 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f56a4020020 con 0x7f56b0101ae0 2024-03-31T23:10:34.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:33.126+0000 7f56b64b6700 1 -- 172.21.15.46:0/305231135 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7f56b002db30 con 0x7f56b0101ae0 2024-03-31T23:10:34.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:33.126+0000 7f56ad7fa700 1 -- 172.21.15.46:0/305231135 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f56a409b830 con 0x7f56b0101ae0 2024-03-31T23:10:34.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:33.128+0000 7f56b64b6700 1 -- 172.21.15.46:0/305231135 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f56980a25c0 msgr2=0x7f56980a4a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:10:34.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:33.128+0000 7f56b64b6700 1 --2- 172.21.15.46:0/305231135 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f56980a25c0 0x7f56980a4a80 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f56a0005950 tx=0x7f56a000b410 comp rx=0 tx=0).stop 2024-03-31T23:10:34.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:33.128+0000 7f56b64b6700 1 -- 172.21.15.46:0/305231135 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f56b0101ae0 msgr2=0x7f56b0199cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:10:34.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:33.128+0000 7f56b64b6700 1 --2- 172.21.15.46:0/305231135 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f56b0101ae0 0x7f56b0199cd0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f56a400d980 tx=0x7f56a400dc90 comp rx=0 tx=0).stop 2024-03-31T23:10:34.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:33.128+0000 7f56b64b6700 1 -- 172.21.15.46:0/305231135 shutdown_connections 2024-03-31T23:10:34.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:33.128+0000 7f56b64b6700 1 --2- 172.21.15.46:0/305231135 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f56980a25c0 0x7f56980a4a80 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:34.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:33.128+0000 7f56b64b6700 1 --2- 172.21.15.46:0/305231135 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f56b0107180 0x7f56b01a4810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:34.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:33.128+0000 7f56b64b6700 1 --2- 172.21.15.46:0/305231135 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f56b0101140 0x7f56b0199790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:34.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:33.128+0000 7f56b64b6700 1 --2- 172.21.15.46:0/305231135 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f56b0101ae0 0x7f56b0199cd0 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:34.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:33.128+0000 7f56b64b6700 1 -- 172.21.15.46:0/305231135 >> 172.21.15.46:0/305231135 conn(0x7f56b0078070 msgr2=0x7f56b00ffea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:10:34.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:33.128+0000 7f56b64b6700 1 -- 172.21.15.46:0/305231135 shutdown_connections 2024-03-31T23:10:34.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:33.129+0000 7f56b64b6700 1 -- 172.21.15.46:0/305231135 wait complete. 2024-03-31T23:10:34.361 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:10:34 smithi046 ceph-mon[32498]: pgmap v335: 129 pgs: 129 active+clean; 642 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-31T23:10:34.361 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:10:34 smithi046 ceph-mon[32498]: from='client.? 172.21.15.46:0/305231135' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-31T23:10:34.405 INFO:tasks.cephadm.ceph_manager.ceph:need seq 313532612638 got 313532612643 for osd.10 2024-03-31T23:10:34.405 DEBUG:teuthology.parallel:result is None 2024-03-31T23:10:34.443 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:10:34 smithi142 ceph-mon[35271]: pgmap v335: 129 pgs: 129 active+clean; 642 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-31T23:10:34.443 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:10:34 smithi142 ceph-mon[35271]: from='client.? 172.21.15.46:0/305231135' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-31T23:10:34.545 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:10:34 smithi162 ceph-mon[36237]: pgmap v335: 129 pgs: 129 active+clean; 642 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-31T23:10:34.546 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:10:34 smithi162 ceph-mon[36237]: from='client.? 172.21.15.46:0/305231135' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-31T23:10:35.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:35.497+0000 7f1da873e700 1 -- 172.21.15.46:0/1243476642 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1da0100dd0 msgr2=0x7f1da01011b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:10:35.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:35.497+0000 7f1da873e700 1 --2- 172.21.15.46:0/1243476642 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1da0100dd0 0x7f1da01011b0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f1d9c009990 tx=0x7f1d9c009ca0 comp rx=0 tx=0).stop 2024-03-31T23:10:35.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:35.498+0000 7f1da873e700 1 -- 172.21.15.46:0/1243476642 shutdown_connections 2024-03-31T23:10:35.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:35.498+0000 7f1da873e700 1 --2- 172.21.15.46:0/1243476642 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1da0100dd0 0x7f1da01011b0 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:35.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:35.498+0000 7f1da873e700 1 --2- 172.21.15.46:0/1243476642 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1da01075f0 0x7f1da010f170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:35.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:35.498+0000 7f1da873e700 1 --2- 172.21.15.46:0/1243476642 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1da0101780 0x7f1da0106ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:35.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:35.498+0000 7f1da873e700 1 -- 172.21.15.46:0/1243476642 >> 172.21.15.46:0/1243476642 conn(0x7f1da0076470 msgr2=0x7f1da0076880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:10:35.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:35.498+0000 7f1da873e700 1 -- 172.21.15.46:0/1243476642 shutdown_connections 2024-03-31T23:10:35.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:35.498+0000 7f1da873e700 1 -- 172.21.15.46:0/1243476642 wait complete. 2024-03-31T23:10:35.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:35.498+0000 7f1da873e700 1 Processor -- start 2024-03-31T23:10:35.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:35.498+0000 7f1da873e700 1 -- start start 2024-03-31T23:10:35.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:35.498+0000 7f1da873e700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1da0100dd0 0x7f1da0101d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:10:35.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:35.499+0000 7f1da873e700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1da0101780 0x7f1da0102290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:10:35.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:35.499+0000 7f1da873e700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1da01075f0 0x7f1da01027d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:10:35.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:35.499+0000 7f1da873e700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f1da0112410 con 0x7f1da0100dd0 2024-03-31T23:10:35.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:35.499+0000 7f1da873e700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f1da0112290 con 0x7f1da0101780 2024-03-31T23:10:35.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:35.499+0000 7f1da873e700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f1da0112590 con 0x7f1da01075f0 2024-03-31T23:10:35.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:35.499+0000 7f1da64da700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1da0100dd0 0x7f1da0101d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:10:35.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:35.499+0000 7f1da64da700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1da0100dd0 0x7f1da0101d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:45016/0 (socket says 172.21.15.46:45016) 2024-03-31T23:10:35.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:35.499+0000 7f1da64da700 1 -- 172.21.15.46:0/2596731532 learned_addr learned my addr 172.21.15.46:0/2596731532 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:10:35.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:35.499+0000 7f1da5cd9700 1 --2- 172.21.15.46:0/2596731532 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1da0101780 0x7f1da0102290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:10:35.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:35.499+0000 7f1da6cdb700 1 --2- 172.21.15.46:0/2596731532 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1da01075f0 0x7f1da01027d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:10:35.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:35.499+0000 7f1da64da700 1 -- 172.21.15.46:0/2596731532 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1da01075f0 msgr2=0x7f1da01027d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:10:35.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:35.499+0000 7f1da64da700 1 --2- 172.21.15.46:0/2596731532 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1da01075f0 0x7f1da01027d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:35.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:35.499+0000 7f1da64da700 1 -- 172.21.15.46:0/2596731532 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1da0101780 msgr2=0x7f1da0102290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:10:35.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:35.499+0000 7f1da64da700 1 --2- 172.21.15.46:0/2596731532 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1da0101780 0x7f1da0102290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:35.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:35.499+0000 7f1da64da700 1 -- 172.21.15.46:0/2596731532 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1da01a6e70 con 0x7f1da0100dd0 2024-03-31T23:10:35.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:35.499+0000 7f1da64da700 1 --2- 172.21.15.46:0/2596731532 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1da0100dd0 0x7f1da0101d50 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f1d9c003ab0 tx=0x7f1d9c003ae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:10:35.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:35.499+0000 7f1d977fe700 1 -- 172.21.15.46:0/2596731532 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d9c027070 con 0x7f1da0100dd0 2024-03-31T23:10:35.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:35.499+0000 7f1da873e700 1 -- 172.21.15.46:0/2596731532 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1da01a7100 con 0x7f1da0100dd0 2024-03-31T23:10:35.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:35.499+0000 7f1d977fe700 1 -- 172.21.15.46:0/2596731532 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1d9c00bdc0 con 0x7f1da0100dd0 2024-03-31T23:10:35.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:35.499+0000 7f1d977fe700 1 -- 172.21.15.46:0/2596731532 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d9c022bf0 con 0x7f1da0100dd0 2024-03-31T23:10:35.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:35.499+0000 7f1da873e700 1 -- 172.21.15.46:0/2596731532 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1da01a75e0 con 0x7f1da0100dd0 2024-03-31T23:10:35.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:35.500+0000 7f1da873e700 1 -- 172.21.15.46:0/2596731532 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1da010a7a0 con 0x7f1da0100dd0 2024-03-31T23:10:35.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:35.502+0000 7f1d977fe700 1 -- 172.21.15.46:0/2596731532 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 17) ==== 137371+0+0 (secure 0 0 0) 0x7f1d9c02a070 con 0x7f1da0100dd0 2024-03-31T23:10:35.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:35.502+0000 7f1d977fe700 1 --2- 172.21.15.46:0/2596731532 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f1d8c0a25c0 0x7f1d8c0a4a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:10:35.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:35.503+0000 7f1da5cd9700 1 --2- 172.21.15.46:0/2596731532 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f1d8c0a25c0 0x7f1d8c0a4a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:10:35.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:35.503+0000 7f1d977fe700 1 -- 172.21.15.46:0/2596731532 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7f1d9c0cfb20 con 0x7f1da0100dd0 2024-03-31T23:10:35.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:35.503+0000 7f1da5cd9700 1 --2- 172.21.15.46:0/2596731532 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f1d8c0a25c0 0x7f1d8c0a4a80 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f1da01007b0 tx=0x7f1d900073d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:10:35.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:35.506+0000 7f1d977fe700 1 -- 172.21.15.46:0/2596731532 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f1d9c013070 con 0x7f1da0100dd0 2024-03-31T23:10:35.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:35.712+0000 7f1da873e700 1 -- 172.21.15.46:0/2596731532 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 11} v 0) -- 0x7f1da010a180 con 0x7f1da0100dd0 2024-03-31T23:10:35.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:35.712+0000 7f1d977fe700 1 -- 172.21.15.46:0/2596731532 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 11}]=0 v0) ==== 75+0+13 (secure 0 0 0) 0x7f1d9c09c8b0 con 0x7f1da0100dd0 2024-03-31T23:10:35.714 INFO:teuthology.orchestra.run.smithi046.stdout:347892351006 2024-03-31T23:10:35.715 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:35.714+0000 7f1da873e700 1 -- 172.21.15.46:0/2596731532 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f1d8c0a25c0 msgr2=0x7f1d8c0a4a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:10:35.715 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:35.714+0000 7f1da873e700 1 --2- 172.21.15.46:0/2596731532 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f1d8c0a25c0 0x7f1d8c0a4a80 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f1da01007b0 tx=0x7f1d900073d0 comp rx=0 tx=0).stop 2024-03-31T23:10:35.715 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:35.714+0000 7f1da873e700 1 -- 172.21.15.46:0/2596731532 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1da0100dd0 msgr2=0x7f1da0101d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:10:35.715 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:35.714+0000 7f1da873e700 1 --2- 172.21.15.46:0/2596731532 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1da0100dd0 0x7f1da0101d50 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f1d9c003ab0 tx=0x7f1d9c003ae0 comp rx=0 tx=0).stop 2024-03-31T23:10:35.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:35.714+0000 7f1da873e700 1 -- 172.21.15.46:0/2596731532 shutdown_connections 2024-03-31T23:10:35.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:35.714+0000 7f1da873e700 1 --2- 172.21.15.46:0/2596731532 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f1d8c0a25c0 0x7f1d8c0a4a80 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:35.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:35.714+0000 7f1da873e700 1 --2- 172.21.15.46:0/2596731532 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1da0100dd0 0x7f1da0101d50 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:35.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:35.714+0000 7f1da873e700 1 --2- 172.21.15.46:0/2596731532 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1da01075f0 0x7f1da01027d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:35.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:35.714+0000 7f1da873e700 1 --2- 172.21.15.46:0/2596731532 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1da0101780 0x7f1da0102290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:35.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:35.714+0000 7f1da873e700 1 -- 172.21.15.46:0/2596731532 >> 172.21.15.46:0/2596731532 conn(0x7f1da0076470 msgr2=0x7f1da0108720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:10:35.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:35.714+0000 7f1da873e700 1 -- 172.21.15.46:0/2596731532 shutdown_connections 2024-03-31T23:10:35.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:35.714+0000 7f1da873e700 1 -- 172.21.15.46:0/2596731532 wait complete. 2024-03-31T23:10:35.958 INFO:tasks.cephadm.ceph_manager.ceph:need seq 107374182464 got 107374182470 for osd.3 2024-03-31T23:10:35.958 DEBUG:teuthology.parallel:result is None 2024-03-31T23:10:36.361 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:10:36 smithi046 ceph-mon[32498]: pgmap v336: 129 pgs: 129 active+clean; 642 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-31T23:10:36.361 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:10:36 smithi046 ceph-mon[32498]: from='client.? 172.21.15.46:0/2596731532' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-03-31T23:10:36.443 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:10:36 smithi142 ceph-mon[35271]: pgmap v336: 129 pgs: 129 active+clean; 642 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-31T23:10:36.443 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:10:36 smithi142 ceph-mon[35271]: from='client.? 172.21.15.46:0/2596731532' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-03-31T23:10:36.515 INFO:tasks.cephadm.ceph_manager.ceph:need seq 257698037797 got 257698037804 for osd.8 2024-03-31T23:10:36.515 DEBUG:teuthology.parallel:result is None 2024-03-31T23:10:36.545 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:10:36 smithi162 ceph-mon[36237]: pgmap v336: 129 pgs: 129 active+clean; 642 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-31T23:10:36.545 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:10:36 smithi162 ceph-mon[36237]: from='client.? 172.21.15.46:0/2596731532' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-03-31T23:10:37.694 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:37.650+0000 7fe8a831f700 1 -- 172.21.15.46:0/3342339483 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe8a006ac00 msgr2=0x7fe8a006afe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:10:37.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:37.650+0000 7fe8a831f700 1 --2- 172.21.15.46:0/3342339483 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe8a006ac00 0x7fe8a006afe0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fe89c0099c0 tx=0x7fe89c009cd0 comp rx=0 tx=0).stop 2024-03-31T23:10:37.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:37.650+0000 7fe8a831f700 1 -- 172.21.15.46:0/3342339483 shutdown_connections 2024-03-31T23:10:37.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:37.650+0000 7fe8a831f700 1 --2- 172.21.15.46:0/3342339483 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe8a006ac00 0x7fe8a006afe0 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:37.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:37.650+0000 7fe8a831f700 1 --2- 172.21.15.46:0/3342339483 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe8a01075c0 0x7fe8a010f140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:37.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:37.650+0000 7fe8a831f700 1 --2- 172.21.15.46:0/3342339483 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe8a006b5b0 0x7fe8a0106e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:37.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:37.650+0000 7fe8a831f700 1 -- 172.21.15.46:0/3342339483 >> 172.21.15.46:0/3342339483 conn(0x7fe8a0077150 msgr2=0x7fe8a0077560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:10:37.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:37.650+0000 7fe8a831f700 1 -- 172.21.15.46:0/3342339483 shutdown_connections 2024-03-31T23:10:37.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:37.651+0000 7fe8a831f700 1 -- 172.21.15.46:0/3342339483 wait complete. 2024-03-31T23:10:37.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:37.651+0000 7fe8a831f700 1 Processor -- start 2024-03-31T23:10:37.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:37.651+0000 7fe8a831f700 1 -- start start 2024-03-31T23:10:37.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:37.651+0000 7fe8a831f700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe8a006ac00 0x7fe8a019e670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:10:37.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:37.651+0000 7fe8a831f700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe8a006b5b0 0x7fe8a019ebb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:10:37.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:37.651+0000 7fe8a831f700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe8a01075c0 0x7fe8a0198840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:10:37.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:37.652+0000 7fe8a831f700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fe8a0112420 con 0x7fe8a006b5b0 2024-03-31T23:10:37.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:37.652+0000 7fe8a831f700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fe8a01122a0 con 0x7fe8a006ac00 2024-03-31T23:10:37.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:37.652+0000 7fe8a831f700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fe8a01125a0 con 0x7fe8a01075c0 2024-03-31T23:10:37.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:37.652+0000 7fe8a58ba700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe8a006b5b0 0x7fe8a019ebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:10:37.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:37.652+0000 7fe8a58ba700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe8a006b5b0 0x7fe8a019ebb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:45042/0 (socket says 172.21.15.46:45042) 2024-03-31T23:10:37.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:37.652+0000 7fe8a58ba700 1 -- 172.21.15.46:0/1820444721 learned_addr learned my addr 172.21.15.46:0/1820444721 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:10:37.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:37.652+0000 7fe8a60bb700 1 --2- 172.21.15.46:0/1820444721 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe8a006ac00 0x7fe8a019e670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:10:37.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:37.652+0000 7fe8a68bc700 1 --2- 172.21.15.46:0/1820444721 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe8a01075c0 0x7fe8a0198840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:10:37.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:37.652+0000 7fe8a58ba700 1 -- 172.21.15.46:0/1820444721 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe8a01075c0 msgr2=0x7fe8a0198840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:10:37.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:37.652+0000 7fe8a58ba700 1 --2- 172.21.15.46:0/1820444721 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe8a01075c0 0x7fe8a0198840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:37.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:37.652+0000 7fe8a58ba700 1 -- 172.21.15.46:0/1820444721 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe8a006ac00 msgr2=0x7fe8a019e670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:10:37.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:37.652+0000 7fe8a58ba700 1 --2- 172.21.15.46:0/1820444721 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe8a006ac00 0x7fe8a019e670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:37.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:37.652+0000 7fe8a58ba700 1 -- 172.21.15.46:0/1820444721 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe8a0199020 con 0x7fe8a006b5b0 2024-03-31T23:10:37.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:37.652+0000 7fe8a60bb700 1 --2- 172.21.15.46:0/1820444721 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe8a006ac00 0x7fe8a019e670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:10:37.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:37.652+0000 7fe8a58ba700 1 --2- 172.21.15.46:0/1820444721 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe8a006b5b0 0x7fe8a019ebb0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fe89000e9c0 tx=0x7fe89000ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:10:37.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:37.652+0000 7fe8977fe700 1 -- 172.21.15.46:0/1820444721 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe890004a30 con 0x7fe8a006b5b0 2024-03-31T23:10:37.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:37.652+0000 7fe8977fe700 1 -- 172.21.15.46:0/1820444721 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe89000cc70 con 0x7fe8a006b5b0 2024-03-31T23:10:37.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:37.652+0000 7fe8a831f700 1 -- 172.21.15.46:0/1820444721 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe8a0199310 con 0x7fe8a006b5b0 2024-03-31T23:10:37.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:37.652+0000 7fe8977fe700 1 -- 172.21.15.46:0/1820444721 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8900055b0 con 0x7fe8a006b5b0 2024-03-31T23:10:37.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:37.653+0000 7fe8a831f700 1 -- 172.21.15.46:0/1820444721 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe8a0101170 con 0x7fe8a006b5b0 2024-03-31T23:10:37.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:37.654+0000 7fe8a831f700 1 -- 172.21.15.46:0/1820444721 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe8a010a790 con 0x7fe8a006b5b0 2024-03-31T23:10:37.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:37.655+0000 7fe8977fe700 1 -- 172.21.15.46:0/1820444721 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 17) ==== 137371+0+0 (secure 0 0 0) 0x7fe89000c7c0 con 0x7fe8a006b5b0 2024-03-31T23:10:37.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:37.656+0000 7fe8977fe700 1 --2- 172.21.15.46:0/1820444721 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fe88c0a25c0 0x7fe88c0a4a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:10:37.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:37.656+0000 7fe8977fe700 1 -- 172.21.15.46:0/1820444721 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7fe8900cefe0 con 0x7fe8a006b5b0 2024-03-31T23:10:37.698 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:37.656+0000 7fe8a60bb700 1 --2- 172.21.15.46:0/1820444721 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fe88c0a25c0 0x7fe88c0a4a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:10:37.698 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:37.657+0000 7fe8a60bb700 1 --2- 172.21.15.46:0/1820444721 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fe88c0a25c0 0x7fe88c0a4a80 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fe89c003760 tx=0x7fe89c005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:10:37.698 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:37.660+0000 7fe8977fe700 1 -- 172.21.15.46:0/1820444721 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fe890053280 con 0x7fe8a006b5b0 2024-03-31T23:10:37.872 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:37.871+0000 7fe8a831f700 1 -- 172.21.15.46:0/1820444721 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 9} v 0) -- 0x7fe8a0199d20 con 0x7fe8a006b5b0 2024-03-31T23:10:37.872 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:37.871+0000 7fe8977fe700 1 -- 172.21.15.46:0/1820444721 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 9}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7fe89009bcc0 con 0x7fe8a006b5b0 2024-03-31T23:10:37.873 INFO:teuthology.orchestra.run.smithi046.stdout:292057776169 2024-03-31T23:10:37.874 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:37.874+0000 7fe8a831f700 1 -- 172.21.15.46:0/1820444721 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fe88c0a25c0 msgr2=0x7fe88c0a4a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:10:37.874 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:37.874+0000 7fe8a831f700 1 --2- 172.21.15.46:0/1820444721 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fe88c0a25c0 0x7fe88c0a4a80 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fe89c003760 tx=0x7fe89c005fb0 comp rx=0 tx=0).stop 2024-03-31T23:10:37.875 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:37.874+0000 7fe8a831f700 1 -- 172.21.15.46:0/1820444721 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe8a006b5b0 msgr2=0x7fe8a019ebb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:10:37.875 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:37.874+0000 7fe8a831f700 1 --2- 172.21.15.46:0/1820444721 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe8a006b5b0 0x7fe8a019ebb0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fe89000e9c0 tx=0x7fe89000ecd0 comp rx=0 tx=0).stop 2024-03-31T23:10:37.875 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:37.874+0000 7fe8a831f700 1 -- 172.21.15.46:0/1820444721 shutdown_connections 2024-03-31T23:10:37.875 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:37.874+0000 7fe8a831f700 1 --2- 172.21.15.46:0/1820444721 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fe88c0a25c0 0x7fe88c0a4a80 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:37.875 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:37.874+0000 7fe8a831f700 1 --2- 172.21.15.46:0/1820444721 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe8a01075c0 0x7fe8a0198840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:37.875 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:37.874+0000 7fe8a831f700 1 --2- 172.21.15.46:0/1820444721 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe8a006ac00 0x7fe8a019e670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:37.876 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:37.874+0000 7fe8a831f700 1 --2- 172.21.15.46:0/1820444721 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe8a006b5b0 0x7fe8a019ebb0 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:37.876 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:37.874+0000 7fe8a831f700 1 -- 172.21.15.46:0/1820444721 >> 172.21.15.46:0/1820444721 conn(0x7fe8a0077150 msgr2=0x7fe8a00ff6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:10:37.876 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:37.874+0000 7fe8a831f700 1 -- 172.21.15.46:0/1820444721 shutdown_connections 2024-03-31T23:10:37.876 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:37.874+0000 7fe8a831f700 1 -- 172.21.15.46:0/1820444721 wait complete. 2024-03-31T23:10:38.091 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/6a9d8f08-efb2-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-31T23:10:38.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:38.102+0000 7fcbd12bf700 1 -- 172.21.15.46:0/3451819963 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcbcc1022c0 msgr2=0x7fcbcc102740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:10:38.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:38.102+0000 7fcbd12bf700 1 --2- 172.21.15.46:0/3451819963 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcbcc1022c0 0x7fcbcc102740 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fcbc0009920 tx=0x7fcbc0009c30 comp rx=0 tx=0).stop 2024-03-31T23:10:38.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:38.102+0000 7fcbd12bf700 1 -- 172.21.15.46:0/3451819963 shutdown_connections 2024-03-31T23:10:38.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:38.102+0000 7fcbd12bf700 1 --2- 172.21.15.46:0/3451819963 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcbcc108310 0x7fcbcc1086f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:38.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:38.102+0000 7fcbd12bf700 1 --2- 172.21.15.46:0/3451819963 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcbcc102c80 0x7fcbcc10f110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:38.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:38.102+0000 7fcbd12bf700 1 --2- 172.21.15.46:0/3451819963 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcbcc1022c0 0x7fcbcc102740 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:38.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:38.102+0000 7fcbd12bf700 1 -- 172.21.15.46:0/3451819963 >> 172.21.15.46:0/3451819963 conn(0x7fcbcc079390 msgr2=0x7fcbcc0797a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:10:38.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:38.102+0000 7fcbd12bf700 1 -- 172.21.15.46:0/3451819963 shutdown_connections 2024-03-31T23:10:38.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:38.102+0000 7fcbd12bf700 1 -- 172.21.15.46:0/3451819963 wait complete. 2024-03-31T23:10:38.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:38.103+0000 7fcbd12bf700 1 Processor -- start 2024-03-31T23:10:38.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:38.103+0000 7fcbd12bf700 1 -- start start 2024-03-31T23:10:38.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:38.103+0000 7fcbd12bf700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcbcc1022c0 0x7fcbcc0738d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:10:38.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:38.103+0000 7fcbd12bf700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcbcc102c80 0x7fcbcc073e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:10:38.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:38.103+0000 7fcbd12bf700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcbcc108310 0x7fcbcc06caa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:10:38.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:38.104+0000 7fcbd12bf700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fcbcc1123e0 con 0x7fcbcc1022c0 2024-03-31T23:10:38.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:38.104+0000 7fcbd12bf700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fcbcc112260 con 0x7fcbcc108310 2024-03-31T23:10:38.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:38.104+0000 7fcbd12bf700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fcbcc112560 con 0x7fcbcc102c80 2024-03-31T23:10:38.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:38.104+0000 7fcbcaffd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcbcc1022c0 0x7fcbcc0738d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:10:38.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:38.104+0000 7fcbcaffd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcbcc1022c0 0x7fcbcc0738d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:45048/0 (socket says 172.21.15.46:45048) 2024-03-31T23:10:38.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:38.104+0000 7fcbcaffd700 1 -- 172.21.15.46:0/3749097258 learned_addr learned my addr 172.21.15.46:0/3749097258 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:10:38.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:38.104+0000 7fcbca7fc700 1 --2- 172.21.15.46:0/3749097258 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcbcc102c80 0x7fcbcc073e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:10:38.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:38.104+0000 7fcbcb7fe700 1 --2- 172.21.15.46:0/3749097258 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcbcc108310 0x7fcbcc06caa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:10:38.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:38.104+0000 7fcbcb7fe700 1 -- 172.21.15.46:0/3749097258 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcbcc102c80 msgr2=0x7fcbcc073e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:10:38.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:38.104+0000 7fcbcb7fe700 1 --2- 172.21.15.46:0/3749097258 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcbcc102c80 0x7fcbcc073e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:38.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:38.104+0000 7fcbcb7fe700 1 -- 172.21.15.46:0/3749097258 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcbcc1022c0 msgr2=0x7fcbcc0738d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:10:38.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:38.104+0000 7fcbcb7fe700 1 --2- 172.21.15.46:0/3749097258 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcbcc1022c0 0x7fcbcc0738d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:38.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:38.104+0000 7fcbcb7fe700 1 -- 172.21.15.46:0/3749097258 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcbcc06d3a0 con 0x7fcbcc108310 2024-03-31T23:10:38.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:38.104+0000 7fcbcaffd700 1 --2- 172.21.15.46:0/3749097258 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcbcc1022c0 0x7fcbcc0738d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:10:38.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:38.104+0000 7fcbcb7fe700 1 --2- 172.21.15.46:0/3749097258 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcbcc108310 0x7fcbcc06caa0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fcbc400efc0 tx=0x7fcbc400c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:10:38.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:38.105+0000 7fcbb3fff700 1 -- 172.21.15.46:0/3749097258 <== mon.2 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbc4004bb0 con 0x7fcbcc108310 2024-03-31T23:10:38.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:38.105+0000 7fcbd12bf700 1 -- 172.21.15.46:0/3749097258 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcbcc06d690 con 0x7fcbcc108310 2024-03-31T23:10:38.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:38.105+0000 7fcbd12bf700 1 -- 172.21.15.46:0/3749097258 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcbcc071620 con 0x7fcbcc108310 2024-03-31T23:10:38.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:38.105+0000 7fcbb3fff700 1 -- 172.21.15.46:0/3749097258 <== mon.2 v2:172.21.15.142:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fcbc4004d50 con 0x7fcbcc108310 2024-03-31T23:10:38.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:38.105+0000 7fcbb3fff700 1 -- 172.21.15.46:0/3749097258 <== mon.2 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbc4004280 con 0x7fcbcc108310 2024-03-31T23:10:38.109 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:38.106+0000 7fcbd12bf700 1 -- 172.21.15.46:0/3749097258 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcbb8005330 con 0x7fcbcc108310 2024-03-31T23:10:38.109 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:38.107+0000 7fcbb3fff700 1 -- 172.21.15.46:0/3749097258 <== mon.2 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 17) ==== 137371+0+0 (secure 0 0 0) 0x7fcbc4003a00 con 0x7fcbcc108310 2024-03-31T23:10:38.109 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:38.108+0000 7fcbb3fff700 1 --2- 172.21.15.46:0/3749097258 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fcbb40a26c0 0x7fcbb40a4b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:10:38.109 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:38.108+0000 7fcbb3fff700 1 -- 172.21.15.46:0/3749097258 <== mon.2 v2:172.21.15.142:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7fcbc40d23c0 con 0x7fcbcc108310 2024-03-31T23:10:38.109 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:38.108+0000 7fcbcaffd700 1 --2- 172.21.15.46:0/3749097258 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fcbb40a26c0 0x7fcbb40a4b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:10:38.109 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:38.108+0000 7fcbcaffd700 1 --2- 172.21.15.46:0/3749097258 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fcbb40a26c0 0x7fcbb40a4b80 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fcbbc011550 tx=0x7fcbbc009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:10:38.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:38.112+0000 7fcbb3fff700 1 -- 172.21.15.46:0/3749097258 <== mon.2 v2:172.21.15.142:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fcbc409f170 con 0x7fcbcc108310 2024-03-31T23:10:38.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:38.319+0000 7fcbd12bf700 1 -- 172.21.15.46:0/3749097258 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) -- 0x7fcbb80058f0 con 0x7fcbcc108310 2024-03-31T23:10:38.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:38.320+0000 7fcbb3fff700 1 -- 172.21.15.46:0/3749097258 <== mon.2 v2:172.21.15.142:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7fcbc409ea70 con 0x7fcbcc108310 2024-03-31T23:10:38.321 INFO:teuthology.orchestra.run.smithi046.stdout:223338299443 2024-03-31T23:10:38.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:38.322+0000 7fcbd12bf700 1 -- 172.21.15.46:0/3749097258 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fcbb40a26c0 msgr2=0x7fcbb40a4b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:10:38.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:38.322+0000 7fcbd12bf700 1 --2- 172.21.15.46:0/3749097258 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fcbb40a26c0 0x7fcbb40a4b80 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fcbbc011550 tx=0x7fcbbc009260 comp rx=0 tx=0).stop 2024-03-31T23:10:38.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:38.322+0000 7fcbd12bf700 1 -- 172.21.15.46:0/3749097258 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcbcc108310 msgr2=0x7fcbcc06caa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:10:38.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:38.322+0000 7fcbd12bf700 1 --2- 172.21.15.46:0/3749097258 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcbcc108310 0x7fcbcc06caa0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fcbc400efc0 tx=0x7fcbc400c3a0 comp rx=0 tx=0).stop 2024-03-31T23:10:38.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:38.322+0000 7fcbd12bf700 1 -- 172.21.15.46:0/3749097258 shutdown_connections 2024-03-31T23:10:38.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:38.322+0000 7fcbd12bf700 1 --2- 172.21.15.46:0/3749097258 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fcbb40a26c0 0x7fcbb40a4b80 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:38.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:38.322+0000 7fcbd12bf700 1 --2- 172.21.15.46:0/3749097258 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcbcc1022c0 0x7fcbcc0738d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:38.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:38.322+0000 7fcbd12bf700 1 --2- 172.21.15.46:0/3749097258 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcbcc108310 0x7fcbcc06caa0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:38.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:38.322+0000 7fcbd12bf700 1 --2- 172.21.15.46:0/3749097258 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcbcc102c80 0x7fcbcc073e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:38.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:38.322+0000 7fcbd12bf700 1 -- 172.21.15.46:0/3749097258 >> 172.21.15.46:0/3749097258 conn(0x7fcbcc079390 msgr2=0x7fcbcc100fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:10:38.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:38.322+0000 7fcbd12bf700 1 -- 172.21.15.46:0/3749097258 shutdown_connections 2024-03-31T23:10:38.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:38.322+0000 7fcbd12bf700 1 -- 172.21.15.46:0/3749097258 wait complete. 2024-03-31T23:10:38.611 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:10:38 smithi046 ceph-mon[32498]: pgmap v337: 129 pgs: 129 active+clean; 642 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-03-31T23:10:38.611 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:10:38 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:10:38.612 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:10:38 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:10:38.612 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:10:38 smithi046 ceph-mon[32498]: from='client.? 172.21.15.46:0/1820444721' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-03-31T23:10:38.612 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:10:38 smithi046 ceph-mon[32498]: from='client.? 172.21.15.46:0/3749097258' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-31T23:10:38.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:10:38 smithi162 ceph-mon[36237]: pgmap v337: 129 pgs: 129 active+clean; 642 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-03-31T23:10:38.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:10:38 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:10:38.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:10:38 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:10:38.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:10:38 smithi162 ceph-mon[36237]: from='client.? 172.21.15.46:0/1820444721' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-03-31T23:10:38.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:10:38 smithi162 ceph-mon[36237]: from='client.? 172.21.15.46:0/3749097258' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-31T23:10:38.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:10:38 smithi142 ceph-mon[35271]: pgmap v337: 129 pgs: 129 active+clean; 642 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-03-31T23:10:38.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:10:38 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:10:38.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:10:38 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:10:38.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:10:38 smithi142 ceph-mon[35271]: from='client.? 172.21.15.46:0/1820444721' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-03-31T23:10:38.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:10:38 smithi142 ceph-mon[35271]: from='client.? 172.21.15.46:0/3749097258' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-31T23:10:39.088 INFO:tasks.cephadm.ceph_manager.ceph:need seq 292057776164 got 292057776169 for osd.9 2024-03-31T23:10:39.088 DEBUG:teuthology.parallel:result is None 2024-03-31T23:10:39.386 INFO:tasks.cephadm.ceph_manager.ceph:need seq 347892351002 got 347892351006 for osd.11 2024-03-31T23:10:39.386 DEBUG:teuthology.parallel:result is None 2024-03-31T23:10:39.784 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411398 got 77309411403 for osd.2 2024-03-31T23:10:39.784 DEBUG:teuthology.parallel:result is None 2024-03-31T23:10:40.247 INFO:tasks.cephadm.ceph_manager.ceph:need seq 223338299436 got 223338299443 for osd.7 2024-03-31T23:10:40.248 DEBUG:teuthology.parallel:result is None 2024-03-31T23:10:40.795 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:10:40 smithi162 ceph-mon[36237]: pgmap v338: 129 pgs: 129 active+clean; 642 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-31T23:10:40.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:10:40 smithi162 ceph-mon[36237]: mgrmap e18: x(active, since 8m), standbys: y, z 2024-03-31T23:10:40.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:10:40 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:10:40.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:10:40 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-31T23:10:40.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:10:40 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:10:40.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:10:40 smithi046 ceph-mon[32498]: pgmap v338: 129 pgs: 129 active+clean; 642 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-31T23:10:40.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:10:40 smithi046 ceph-mon[32498]: mgrmap e18: x(active, since 8m), standbys: y, z 2024-03-31T23:10:40.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:10:40 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:10:40.862 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:10:40 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-31T23:10:40.862 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:10:40 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:10:40.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:10:40 smithi142 ceph-mon[35271]: pgmap v338: 129 pgs: 129 active+clean; 642 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-31T23:10:40.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:10:40 smithi142 ceph-mon[35271]: mgrmap e18: x(active, since 8m), standbys: y, z 2024-03-31T23:10:40.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:10:40 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:10:40.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:10:40 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-31T23:10:40.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:10:40 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' 2024-03-31T23:10:41.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:41.954+0000 7ff412d50700 1 -- 172.21.15.46:0/646928222 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff40c106960 msgr2=0x7ff40c106d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:10:41.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:41.954+0000 7ff412d50700 1 --2- 172.21.15.46:0/646928222 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff40c106960 0x7ff40c106d40 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7ff40000aaa0 tx=0x7ff40000adb0 comp rx=0 tx=0).stop 2024-03-31T23:10:41.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:41.955+0000 7ff412d50700 1 -- 172.21.15.46:0/646928222 shutdown_connections 2024-03-31T23:10:41.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:41.955+0000 7ff412d50700 1 --2- 172.21.15.46:0/646928222 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff40c106960 0x7ff40c106d40 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:41.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:41.955+0000 7ff412d50700 1 --2- 172.21.15.46:0/646928222 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff40c101540 0x7ff40c10d780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:41.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:41.955+0000 7ff412d50700 1 --2- 172.21.15.46:0/646928222 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff40c100b80 0x7ff40c101000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:41.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:41.955+0000 7ff412d50700 1 -- 172.21.15.46:0/646928222 >> 172.21.15.46:0/646928222 conn(0x7ff40c0764a0 msgr2=0x7ff40c0768b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:10:41.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:41.955+0000 7ff412d50700 1 -- 172.21.15.46:0/646928222 shutdown_connections 2024-03-31T23:10:41.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:41.955+0000 7ff412d50700 1 -- 172.21.15.46:0/646928222 wait complete. 2024-03-31T23:10:41.958 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:41.955+0000 7ff412d50700 1 Processor -- start 2024-03-31T23:10:41.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:41.956+0000 7ff412d50700 1 -- start start 2024-03-31T23:10:41.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:41.956+0000 7ff412d50700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff40c100b80 0x7ff40c110a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:10:41.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:41.956+0000 7ff412d50700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff40c101540 0x7ff40c110fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:10:41.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:41.956+0000 7ff412d50700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff40c106960 0x7ff40c1114f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:10:41.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:41.956+0000 7ff412d50700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7ff40c110220 con 0x7ff40c100b80 2024-03-31T23:10:41.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:41.956+0000 7ff412d50700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7ff40c1100a0 con 0x7ff40c106960 2024-03-31T23:10:41.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:41.956+0000 7ff412d50700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7ff40c1103a0 con 0x7ff40c101540 2024-03-31T23:10:41.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:41.956+0000 7ff410aec700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff40c100b80 0x7ff40c110a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:10:41.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:41.956+0000 7ff40bfff700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff40c101540 0x7ff40c110fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:10:41.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:41.956+0000 7ff40bfff700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff40c101540 0x7ff40c110fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.46:41770/0 (socket says 172.21.15.46:41770) 2024-03-31T23:10:41.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:41.956+0000 7ff40bfff700 1 -- 172.21.15.46:0/2764434557 learned_addr learned my addr 172.21.15.46:0/2764434557 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:10:41.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:41.956+0000 7ff410aec700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff40c100b80 0x7ff40c110a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:51992/0 (socket says 172.21.15.46:51992) 2024-03-31T23:10:41.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:41.956+0000 7ff4112ed700 1 --2- 172.21.15.46:0/2764434557 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff40c106960 0x7ff40c1114f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:10:41.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:41.956+0000 7ff40bfff700 1 -- 172.21.15.46:0/2764434557 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff40c106960 msgr2=0x7ff40c1114f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:10:41.960 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:41.956+0000 7ff40bfff700 1 --2- 172.21.15.46:0/2764434557 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff40c106960 0x7ff40c1114f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:41.960 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:41.956+0000 7ff40bfff700 1 -- 172.21.15.46:0/2764434557 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff40c100b80 msgr2=0x7ff40c110a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:10:41.960 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:41.956+0000 7ff40bfff700 1 --2- 172.21.15.46:0/2764434557 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff40c100b80 0x7ff40c110a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:41.960 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:41.956+0000 7ff40bfff700 1 -- 172.21.15.46:0/2764434557 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff40c1a6d80 con 0x7ff40c101540 2024-03-31T23:10:41.960 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:41.957+0000 7ff40bfff700 1 --2- 172.21.15.46:0/2764434557 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff40c101540 0x7ff40c110fb0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7ff3f800e9c0 tx=0x7ff3f800ecd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:10:41.960 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:41.957+0000 7ff409ffb700 1 -- 172.21.15.46:0/2764434557 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3f800cbd0 con 0x7ff40c101540 2024-03-31T23:10:41.960 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:41.957+0000 7ff409ffb700 1 -- 172.21.15.46:0/2764434557 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff3f800ee60 con 0x7ff40c101540 2024-03-31T23:10:41.960 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:41.957+0000 7ff412d50700 1 -- 172.21.15.46:0/2764434557 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff40c1a7070 con 0x7ff40c101540 2024-03-31T23:10:41.960 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:41.957+0000 7ff409ffb700 1 -- 172.21.15.46:0/2764434557 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3f80055b0 con 0x7ff40c101540 2024-03-31T23:10:41.960 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:41.957+0000 7ff412d50700 1 -- 172.21.15.46:0/2764434557 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff40c1a75b0 con 0x7ff40c101540 2024-03-31T23:10:41.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:41.958+0000 7ff412d50700 1 -- 172.21.15.46:0/2764434557 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff40c108e40 con 0x7ff40c101540 2024-03-31T23:10:41.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:41.959+0000 7ff409ffb700 1 -- 172.21.15.46:0/2764434557 <== mon.1 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 18) ==== 137320+0+0 (secure 0 0 0) 0x7ff3f8044930 con 0x7ff40c101540 2024-03-31T23:10:41.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:41.960+0000 7ff409ffb700 1 --2- 172.21.15.46:0/2764434557 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7ff3fc0a25b0 0x7ff3fc0a4a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:10:41.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:41.960+0000 7ff410aec700 1 --2- 172.21.15.46:0/2764434557 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7ff3fc0a25b0 0x7ff3fc0a4a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:10:41.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:41.960+0000 7ff409ffb700 1 -- 172.21.15.46:0/2764434557 <== mon.1 v2:172.21.15.162:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7ff3f80cec90 con 0x7ff40c101540 2024-03-31T23:10:41.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:41.960+0000 7ff410aec700 1 --2- 172.21.15.46:0/2764434557 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7ff3fc0a25b0 0x7ff3fc0a4a70 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7ff40000a9f0 tx=0x7ff400005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:10:41.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:41.964+0000 7ff409ffb700 1 -- 172.21.15.46:0/2764434557 <== mon.1 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7ff3f801f340 con 0x7ff40c101540 2024-03-31T23:10:42.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:42.164+0000 7ff412d50700 1 -- 172.21.15.46:0/2764434557 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) -- 0x7ff40c111f50 con 0x7ff40c101540 2024-03-31T23:10:42.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:42.165+0000 7ff409ffb700 1 -- 172.21.15.46:0/2764434557 <== mon.1 v2:172.21.15.162:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7ff3f809baf0 con 0x7ff40c101540 2024-03-31T23:10:42.166 INFO:teuthology.orchestra.run.smithi046.stdout:201863462969 2024-03-31T23:10:42.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:42.167+0000 7ff412d50700 1 -- 172.21.15.46:0/2764434557 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7ff3fc0a25b0 msgr2=0x7ff3fc0a4a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:10:42.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:42.167+0000 7ff412d50700 1 --2- 172.21.15.46:0/2764434557 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7ff3fc0a25b0 0x7ff3fc0a4a70 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7ff40000a9f0 tx=0x7ff400005960 comp rx=0 tx=0).stop 2024-03-31T23:10:42.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:42.167+0000 7ff412d50700 1 -- 172.21.15.46:0/2764434557 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff40c101540 msgr2=0x7ff40c110fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:10:42.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:42.167+0000 7ff412d50700 1 --2- 172.21.15.46:0/2764434557 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff40c101540 0x7ff40c110fb0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7ff3f800e9c0 tx=0x7ff3f800ecd0 comp rx=0 tx=0).stop 2024-03-31T23:10:42.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:42.167+0000 7ff412d50700 1 -- 172.21.15.46:0/2764434557 shutdown_connections 2024-03-31T23:10:42.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:42.167+0000 7ff412d50700 1 --2- 172.21.15.46:0/2764434557 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7ff3fc0a25b0 0x7ff3fc0a4a70 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:42.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:42.167+0000 7ff412d50700 1 --2- 172.21.15.46:0/2764434557 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff40c100b80 0x7ff40c110a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:42.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:42.167+0000 7ff412d50700 1 --2- 172.21.15.46:0/2764434557 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff40c106960 0x7ff40c1114f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:42.169 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:42.167+0000 7ff412d50700 1 --2- 172.21.15.46:0/2764434557 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff40c101540 0x7ff40c110fb0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:42.169 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:42.167+0000 7ff412d50700 1 -- 172.21.15.46:0/2764434557 >> 172.21.15.46:0/2764434557 conn(0x7ff40c0764a0 msgr2=0x7ff40c0febd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:10:42.169 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:42.167+0000 7ff412d50700 1 -- 172.21.15.46:0/2764434557 shutdown_connections 2024-03-31T23:10:42.169 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:42.167+0000 7ff412d50700 1 -- 172.21.15.46:0/2764434557 wait complete. 2024-03-31T23:10:42.795 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:10:42 smithi162 ceph-mon[36237]: pgmap v339: 129 pgs: 129 active+clean; 642 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 426 B/s wr, 7 op/s 2024-03-31T23:10:42.795 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:10:42 smithi162 ceph-mon[36237]: from='client.? 172.21.15.46:0/2764434557' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-31T23:10:42.839 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:10:42 smithi046 ceph-mon[32498]: pgmap v339: 129 pgs: 129 active+clean; 642 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 426 B/s wr, 7 op/s 2024-03-31T23:10:42.839 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:10:42 smithi046 ceph-mon[32498]: from='client.? 172.21.15.46:0/2764434557' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-31T23:10:42.870 INFO:tasks.cephadm.ceph_manager.ceph:need seq 201863462963 got 201863462969 for osd.6 2024-03-31T23:10:42.870 DEBUG:teuthology.parallel:result is None 2024-03-31T23:10:42.870 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-03-31T23:10:42.871 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:384f31f3d3aee7f8ba3f60846b8844c049165583 shell --fsid 6a9d8f08-efb2-11ee-b647-cb9ed24678a4 -- ceph pg dump --format=json 2024-03-31T23:10:42.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:10:42 smithi142 ceph-mon[35271]: pgmap v339: 129 pgs: 129 active+clean; 642 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 426 B/s wr, 7 op/s 2024-03-31T23:10:42.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:10:42 smithi142 ceph-mon[35271]: from='client.? 172.21.15.46:0/2764434557' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-31T23:10:43.120 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/6a9d8f08-efb2-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-31T23:10:44.610 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:10:44 smithi046 ceph-mon[32498]: pgmap v340: 129 pgs: 129 active+clean; 642 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 511 B/s wr, 6 op/s 2024-03-31T23:10:44.795 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:10:44 smithi162 ceph-mon[36237]: pgmap v340: 129 pgs: 129 active+clean; 642 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 511 B/s wr, 6 op/s 2024-03-31T23:10:44.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:44.836+0000 7f6214dd8700 1 -- 172.21.15.46:0/329696751 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6210104c70 msgr2=0x7f62101050f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:10:44.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:44.836+0000 7f6214dd8700 1 --2- 172.21.15.46:0/329696751 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6210104c70 0x7f62101050f0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f6204009990 tx=0x7f6204009ca0 comp rx=0 tx=0).stop 2024-03-31T23:10:44.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:44.837+0000 7f6214dd8700 1 -- 172.21.15.46:0/329696751 shutdown_connections 2024-03-31T23:10:44.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:44.837+0000 7f6214dd8700 1 --2- 172.21.15.46:0/329696751 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6210104c70 0x7f62101050f0 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:44.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:44.837+0000 7f6214dd8700 1 --2- 172.21.15.46:0/329696751 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6210107200 0x7f621006a780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:44.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:44.837+0000 7f6214dd8700 1 --2- 172.21.15.46:0/329696751 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f621006acc0 0x7f621006b140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:44.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:44.837+0000 7f6214dd8700 1 -- 172.21.15.46:0/329696751 >> 172.21.15.46:0/329696751 conn(0x7f6210077190 msgr2=0x7f62100775a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:10:44.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:44.837+0000 7f6214dd8700 1 -- 172.21.15.46:0/329696751 shutdown_connections 2024-03-31T23:10:44.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:44.837+0000 7f6214dd8700 1 -- 172.21.15.46:0/329696751 wait complete. 2024-03-31T23:10:44.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:44.837+0000 7f6214dd8700 1 Processor -- start 2024-03-31T23:10:44.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:44.838+0000 7f6214dd8700 1 -- start start 2024-03-31T23:10:44.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:44.838+0000 7f6214dd8700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f621006acc0 0x7f621019ee10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:10:44.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:44.838+0000 7f6214dd8700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6210104c70 0x7f621019f350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:10:44.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:44.838+0000 7f6214dd8700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6210107200 0x7f62101a36e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:10:44.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:44.838+0000 7f6214dd8700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f62101189c0 con 0x7f6210104c70 2024-03-31T23:10:44.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:44.838+0000 7f6214dd8700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f6210118840 con 0x7f6210107200 2024-03-31T23:10:44.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:44.838+0000 7f6214dd8700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f6210118b40 con 0x7f621006acc0 2024-03-31T23:10:44.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:44.838+0000 7f620dd9b700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6210104c70 0x7f621019f350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:10:44.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:44.838+0000 7f620ed9d700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6210107200 0x7f62101a36e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:10:44.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:44.838+0000 7f620dd9b700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6210104c70 0x7f621019f350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:52008/0 (socket says 172.21.15.46:52008) 2024-03-31T23:10:44.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:44.838+0000 7f620dd9b700 1 -- 172.21.15.46:0/547178840 learned_addr learned my addr 172.21.15.46:0/547178840 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:10:44.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:44.838+0000 7f620e59c700 1 --2- 172.21.15.46:0/547178840 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f621006acc0 0x7f621019ee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:10:44.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:44.838+0000 7f620dd9b700 1 -- 172.21.15.46:0/547178840 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f621006acc0 msgr2=0x7f621019ee10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:10:44.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:44.838+0000 7f620dd9b700 1 --2- 172.21.15.46:0/547178840 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f621006acc0 0x7f621019ee10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:44.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:44.838+0000 7f620dd9b700 1 -- 172.21.15.46:0/547178840 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6210107200 msgr2=0x7f62101a36e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:10:44.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:44.838+0000 7f620dd9b700 1 --2- 172.21.15.46:0/547178840 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6210107200 0x7f62101a36e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:44.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:44.838+0000 7f620dd9b700 1 -- 172.21.15.46:0/547178840 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f62101a3e20 con 0x7f6210104c70 2024-03-31T23:10:44.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:44.838+0000 7f620ed9d700 1 --2- 172.21.15.46:0/547178840 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6210107200 0x7f62101a36e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:10:44.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:44.839+0000 7f620dd9b700 1 --2- 172.21.15.46:0/547178840 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6210104c70 0x7f621019f350 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f620800ef30 tx=0x7f620800c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:10:44.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:44.839+0000 7f61ff7fe700 1 -- 172.21.15.46:0/547178840 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62080092e0 con 0x7f6210104c70 2024-03-31T23:10:44.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:44.839+0000 7f6214dd8700 1 -- 172.21.15.46:0/547178840 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f62101a40b0 con 0x7f6210104c70 2024-03-31T23:10:44.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:44.839+0000 7f61ff7fe700 1 -- 172.21.15.46:0/547178840 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6208004bb0 con 0x7f6210104c70 2024-03-31T23:10:44.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:44.839+0000 7f61ff7fe700 1 -- 172.21.15.46:0/547178840 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6208004040 con 0x7f6210104c70 2024-03-31T23:10:44.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:44.839+0000 7f6214dd8700 1 -- 172.21.15.46:0/547178840 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f62101ab990 con 0x7f6210104c70 2024-03-31T23:10:44.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:44.840+0000 7f6214dd8700 1 -- 172.21.15.46:0/547178840 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6210068f00 con 0x7f6210104c70 2024-03-31T23:10:44.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:44.841+0000 7f61ff7fe700 1 -- 172.21.15.46:0/547178840 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 18) ==== 137320+0+0 (secure 0 0 0) 0x7f620801b030 con 0x7f6210104c70 2024-03-31T23:10:44.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:44.841+0000 7f61ff7fe700 1 --2- 172.21.15.46:0/547178840 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f61f80a2530 0x7f61f80a49f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:10:44.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:44.841+0000 7f620e59c700 1 --2- 172.21.15.46:0/547178840 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f61f80a2530 0x7f61f80a49f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:10:44.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:44.841+0000 7f61ff7fe700 1 -- 172.21.15.46:0/547178840 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7f62080ce5a0 con 0x7f6210104c70 2024-03-31T23:10:44.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:44.842+0000 7f620e59c700 1 --2- 172.21.15.46:0/547178840 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f61f80a2530 0x7f61f80a49f0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f6200009be0 tx=0x7f6200009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:10:44.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:44.846+0000 7f61ff7fe700 1 -- 172.21.15.46:0/547178840 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f6208022020 con 0x7f6210104c70 2024-03-31T23:10:44.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:10:44 smithi142 ceph-mon[35271]: pgmap v340: 129 pgs: 129 active+clean; 642 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 511 B/s wr, 6 op/s 2024-03-31T23:10:45.047 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:45.046+0000 7f6214dd8700 1 -- 172.21.15.46:0/547178840 --> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f6210100870 con 0x7f61f80a2530 2024-03-31T23:10:45.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:45.056+0000 7f61ff7fe700 1 -- 172.21.15.46:0/547178840 <== mgr.14152 v2:172.21.15.46:6800/162289569 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340589 (secure 0 0 0) 0x7f6210100870 con 0x7f61f80a2530 2024-03-31T23:10:45.062 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-31T23:10:45.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:45.059+0000 7f6214dd8700 1 -- 172.21.15.46:0/547178840 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f61f80a2530 msgr2=0x7f61f80a49f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:10:45.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:45.059+0000 7f6214dd8700 1 --2- 172.21.15.46:0/547178840 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f61f80a2530 0x7f61f80a49f0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f6200009be0 tx=0x7f6200009260 comp rx=0 tx=0).stop 2024-03-31T23:10:45.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:45.059+0000 7f6214dd8700 1 -- 172.21.15.46:0/547178840 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6210104c70 msgr2=0x7f621019f350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:10:45.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:45.059+0000 7f6214dd8700 1 --2- 172.21.15.46:0/547178840 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6210104c70 0x7f621019f350 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f620800ef30 tx=0x7f620800c3a0 comp rx=0 tx=0).stop 2024-03-31T23:10:45.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:45.059+0000 7f6214dd8700 1 -- 172.21.15.46:0/547178840 shutdown_connections 2024-03-31T23:10:45.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:45.059+0000 7f6214dd8700 1 --2- 172.21.15.46:0/547178840 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f61f80a2530 0x7f61f80a49f0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:45.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:45.059+0000 7f6214dd8700 1 --2- 172.21.15.46:0/547178840 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6210107200 0x7f62101a36e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:45.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:45.059+0000 7f6214dd8700 1 --2- 172.21.15.46:0/547178840 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f621006acc0 0x7f621019ee10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:45.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:45.059+0000 7f6214dd8700 1 --2- 172.21.15.46:0/547178840 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6210104c70 0x7f621019f350 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:45.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:45.059+0000 7f6214dd8700 1 -- 172.21.15.46:0/547178840 >> 172.21.15.46:0/547178840 conn(0x7f6210077190 msgr2=0x7f6210100160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:10:45.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:45.060+0000 7f6214dd8700 1 -- 172.21.15.46:0/547178840 shutdown_connections 2024-03-31T23:10:45.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:45.060+0000 7f6214dd8700 1 -- 172.21.15.46:0/547178840 wait complete. 2024-03-31T23:10:45.065 INFO:teuthology.orchestra.run.smithi046.stderr:dumped all 2024-03-31T23:10:45.898 INFO:teuthology.orchestra.run.smithi046.stdout:{"pg_ready":true,"pg_map":{"version":341,"stamp":"2024-03-31T23:10:44.600775+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":657858,"num_objects":64,"num_object_clones":0,"num_object_copies":194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":590,"num_read_kb":416,"num_write":355,"num_write_kb":3065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":789,"ondisk_log_size":789,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":371904,"kb_used_data":14432,"kb_used_omap":18,"kb_used_meta":357357,"kb_avail":1124569920,"statfs":{"total":1151940427776,"available":1151559598080,"internally_reserved":0,"allocated":14778368,"data_stored":8126288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19140,"internal_metadata":365933884},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":531,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":68,"num_read_kb":33,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003743"},"pg_stats":[{"pgid":"3.3f","version":"93'8","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-31T23:10:38.617486+0000","last_change":"2024-03-31T23:09:11.177502+0000","last_active":"2024-03-31T23:10:38.617486+0000","last_peered":"2024-03-31T23:10:38.617486+0000","last_clean":"2024-03-31T23:10:38.617486+0000","last_became_active":"2024-03-31T23:09:11.177134+0000","last_became_peered":"2024-03-31T23:09:11.177134+0000","last_unstale":"2024-03-31T23:10:38.617486+0000","last_undegraded":"2024-03-31T23:10:38.617486+0000","last_fullsized":"2024-03-31T23:10:38.617486+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T10:29:36.508229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"93'4","reported_seq":31,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-31T23:10:38.608683+0000","last_change":"2024-03-31T23:09:11.172990+0000","last_active":"2024-03-31T23:10:38.608683+0000","last_peered":"2024-03-31T23:10:38.608683+0000","last_clean":"2024-03-31T23:10:38.608683+0000","last_became_active":"2024-03-31T23:09:11.172684+0000","last_became_peered":"2024-03-31T23:09:11.172684+0000","last_unstale":"2024-03-31T23:10:38.608683+0000","last_undegraded":"2024-03-31T23:10:38.608683+0000","last_fullsized":"2024-03-31T23:10:38.608683+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T07:02:59.991757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"93'7","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-31T23:10:38.608415+0000","last_change":"2024-03-31T23:09:11.176805+0000","last_active":"2024-03-31T23:10:38.608415+0000","last_peered":"2024-03-31T23:10:38.608415+0000","last_clean":"2024-03-31T23:10:38.608415+0000","last_became_active":"2024-03-31T23:09:11.176468+0000","last_became_peered":"2024-03-31T23:09:11.176468+0000","last_unstale":"2024-03-31T23:10:38.608415+0000","last_undegraded":"2024-03-31T23:10:38.608415+0000","last_fullsized":"2024-03-31T23:10:38.608415+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T08:23:41.872558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"93'8","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-31T23:10:38.601173+0000","last_change":"2024-03-31T23:09:11.182385+0000","last_active":"2024-03-31T23:10:38.601173+0000","last_peered":"2024-03-31T23:10:38.601173+0000","last_clean":"2024-03-31T23:10:38.601173+0000","last_became_active":"2024-03-31T23:09:11.181908+0000","last_became_peered":"2024-03-31T23:09:11.181908+0000","last_unstale":"2024-03-31T23:10:38.601173+0000","last_undegraded":"2024-03-31T23:10:38.601173+0000","last_fullsized":"2024-03-31T23:10:38.601173+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:52:24.066360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"93'4","reported_seq":31,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-31T23:10:38.614611+0000","last_change":"2024-03-31T23:09:11.181377+0000","last_active":"2024-03-31T23:10:38.614611+0000","last_peered":"2024-03-31T23:10:38.614611+0000","last_clean":"2024-03-31T23:10:38.614611+0000","last_became_active":"2024-03-31T23:09:11.181064+0000","last_became_peered":"2024-03-31T23:09:11.181064+0000","last_unstale":"2024-03-31T23:10:38.614611+0000","last_undegraded":"2024-03-31T23:10:38.614611+0000","last_fullsized":"2024-03-31T23:10:38.614611+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:28:40.766541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"93'6","reported_seq":33,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-31T23:10:38.606476+0000","last_change":"2024-03-31T23:09:11.176087+0000","last_active":"2024-03-31T23:10:38.606476+0000","last_peered":"2024-03-31T23:10:38.606476+0000","last_clean":"2024-03-31T23:10:38.606476+0000","last_became_active":"2024-03-31T23:09:11.175569+0000","last_became_peered":"2024-03-31T23:09:11.175569+0000","last_unstale":"2024-03-31T23:10:38.606476+0000","last_undegraded":"2024-03-31T23:10:38.606476+0000","last_fullsized":"2024-03-31T23:10:38.606476+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T06:33:19.189909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"93'5","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-31T23:10:38.609004+0000","last_change":"2024-03-31T23:09:11.170289+0000","last_active":"2024-03-31T23:10:38.609004+0000","last_peered":"2024-03-31T23:10:38.609004+0000","last_clean":"2024-03-31T23:10:38.609004+0000","last_became_active":"2024-03-31T23:09:11.169959+0000","last_became_peered":"2024-03-31T23:09:11.169959+0000","last_unstale":"2024-03-31T23:10:38.609004+0000","last_undegraded":"2024-03-31T23:10:38.609004+0000","last_fullsized":"2024-03-31T23:10:38.609004+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:10:44.614767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"93'9","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-31T23:10:38.619815+0000","last_change":"2024-03-31T23:09:11.182447+0000","last_active":"2024-03-31T23:10:38.619815+0000","last_peered":"2024-03-31T23:10:38.619815+0000","last_clean":"2024-03-31T23:10:38.619815+0000","last_became_active":"2024-03-31T23:09:11.181629+0000","last_became_peered":"2024-03-31T23:09:11.181629+0000","last_unstale":"2024-03-31T23:10:38.619815+0000","last_undegraded":"2024-03-31T23:10:38.619815+0000","last_fullsized":"2024-03-31T23:10:38.619815+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:18:26.269451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"93'11","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-31T23:10:38.615507+0000","last_change":"2024-03-31T23:09:11.176853+0000","last_active":"2024-03-31T23:10:38.615507+0000","last_peered":"2024-03-31T23:10:38.615507+0000","last_clean":"2024-03-31T23:10:38.615507+0000","last_became_active":"2024-03-31T23:09:11.176413+0000","last_became_peered":"2024-03-31T23:09:11.176413+0000","last_unstale":"2024-03-31T23:10:38.615507+0000","last_undegraded":"2024-03-31T23:10:38.615507+0000","last_fullsized":"2024-03-31T23:10:38.615507+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:23:40.486947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"93'16","reported_seq":49,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-31T23:10:38.604985+0000","last_change":"2024-03-31T23:10:10.543998+0000","last_active":"2024-03-31T23:10:38.604985+0000","last_peered":"2024-03-31T23:10:38.604985+0000","last_clean":"2024-03-31T23:10:38.604985+0000","last_became_active":"2024-03-31T23:10:10.543615+0000","last_became_peered":"2024-03-31T23:10:10.543615+0000","last_unstale":"2024-03-31T23:10:38.604985+0000","last_undegraded":"2024-03-31T23:10:38.604985+0000","last_fullsized":"2024-03-31T23:10:38.604985+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:28:32.883732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,9],"acting":[10,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"93'6","reported_seq":33,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-31T23:10:38.613988+0000","last_change":"2024-03-31T23:09:11.179536+0000","last_active":"2024-03-31T23:10:38.613988+0000","last_peered":"2024-03-31T23:10:38.613988+0000","last_clean":"2024-03-31T23:10:38.613988+0000","last_became_active":"2024-03-31T23:09:11.178521+0000","last_became_peered":"2024-03-31T23:09:11.178521+0000","last_unstale":"2024-03-31T23:10:38.613988+0000","last_undegraded":"2024-03-31T23:10:38.613988+0000","last_fullsized":"2024-03-31T23:10:38.613988+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T09:29:23.128832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"93'7","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-31T23:10:38.609167+0000","last_change":"2024-03-31T23:09:11.182548+0000","last_active":"2024-03-31T23:10:38.609167+0000","last_peered":"2024-03-31T23:10:38.609167+0000","last_clean":"2024-03-31T23:10:38.609167+0000","last_became_active":"2024-03-31T23:09:11.181966+0000","last_became_peered":"2024-03-31T23:09:11.181966+0000","last_unstale":"2024-03-31T23:10:38.609167+0000","last_undegraded":"2024-03-31T23:10:38.609167+0000","last_fullsized":"2024-03-31T23:10:38.609167+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:43:26.061594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"93'10","reported_seq":43,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-31T23:10:38.610143+0000","last_change":"2024-03-31T23:10:10.544976+0000","last_active":"2024-03-31T23:10:38.610143+0000","last_peered":"2024-03-31T23:10:38.610143+0000","last_clean":"2024-03-31T23:10:38.610143+0000","last_became_active":"2024-03-31T23:10:10.544433+0000","last_became_peered":"2024-03-31T23:10:10.544433+0000","last_unstale":"2024-03-31T23:10:38.610143+0000","last_undegraded":"2024-03-31T23:10:38.610143+0000","last_fullsized":"2024-03-31T23:10:38.610143+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:45:54.606274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,9],"acting":[6,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"93'7","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-31T23:10:38.598000+0000","last_change":"2024-03-31T23:09:11.168784+0000","last_active":"2024-03-31T23:10:38.598000+0000","last_peered":"2024-03-31T23:10:38.598000+0000","last_clean":"2024-03-31T23:10:38.598000+0000","last_became_active":"2024-03-31T23:09:11.168442+0000","last_became_peered":"2024-03-31T23:09:11.168442+0000","last_unstale":"2024-03-31T23:10:38.598000+0000","last_undegraded":"2024-03-31T23:10:38.598000+0000","last_fullsized":"2024-03-31T23:10:38.598000+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T09:37:43.568122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"93'7","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-31T23:10:38.595244+0000","last_change":"2024-03-31T23:09:11.176017+0000","last_active":"2024-03-31T23:10:38.595244+0000","last_peered":"2024-03-31T23:10:38.595244+0000","last_clean":"2024-03-31T23:10:38.595244+0000","last_became_active":"2024-03-31T23:09:11.175690+0000","last_became_peered":"2024-03-31T23:09:11.175690+0000","last_unstale":"2024-03-31T23:10:38.595244+0000","last_undegraded":"2024-03-31T23:10:38.595244+0000","last_fullsized":"2024-03-31T23:10:38.595244+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:14:13.259895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"93'9","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-31T23:10:38.613070+0000","last_change":"2024-03-31T23:09:11.171926+0000","last_active":"2024-03-31T23:10:38.613070+0000","last_peered":"2024-03-31T23:10:38.613070+0000","last_clean":"2024-03-31T23:10:38.613070+0000","last_became_active":"2024-03-31T23:09:11.171625+0000","last_became_peered":"2024-03-31T23:09:11.171625+0000","last_unstale":"2024-03-31T23:10:38.613070+0000","last_undegraded":"2024-03-31T23:10:38.613070+0000","last_fullsized":"2024-03-31T23:10:38.613070+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T06:20:20.433802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"93'8","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-31T23:10:38.617506+0000","last_change":"2024-03-31T23:09:11.183805+0000","last_active":"2024-03-31T23:10:38.617506+0000","last_peered":"2024-03-31T23:10:38.617506+0000","last_clean":"2024-03-31T23:10:38.617506+0000","last_became_active":"2024-03-31T23:09:11.183464+0000","last_became_peered":"2024-03-31T23:09:11.183464+0000","last_unstale":"2024-03-31T23:10:38.617506+0000","last_undegraded":"2024-03-31T23:10:38.617506+0000","last_fullsized":"2024-03-31T23:10:38.617506+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T06:02:33.286563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"93'10","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-31T23:10:38.614241+0000","last_change":"2024-03-31T23:09:11.170095+0000","last_active":"2024-03-31T23:10:38.614241+0000","last_peered":"2024-03-31T23:10:38.614241+0000","last_clean":"2024-03-31T23:10:38.614241+0000","last_became_active":"2024-03-31T23:09:11.169683+0000","last_became_peered":"2024-03-31T23:09:11.169683+0000","last_unstale":"2024-03-31T23:10:38.614241+0000","last_undegraded":"2024-03-31T23:10:38.614241+0000","last_fullsized":"2024-03-31T23:10:38.614241+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T08:19:47.101431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"93'7","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-31T23:10:38.578795+0000","last_change":"2024-03-31T23:09:11.181347+0000","last_active":"2024-03-31T23:10:38.578795+0000","last_peered":"2024-03-31T23:10:38.578795+0000","last_clean":"2024-03-31T23:10:38.578795+0000","last_became_active":"2024-03-31T23:09:11.180510+0000","last_became_peered":"2024-03-31T23:09:11.180510+0000","last_unstale":"2024-03-31T23:10:38.578795+0000","last_undegraded":"2024-03-31T23:10:38.578795+0000","last_fullsized":"2024-03-31T23:10:38.578795+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:25:51.083446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":21,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-31T23:10:10.541341+0000","last_change":"2024-03-31T23:09:15.205059+0000","last_active":"2024-03-31T23:10:10.541341+0000","last_peered":"2024-03-31T23:10:10.541341+0000","last_clean":"2024-03-31T23:10:10.541341+0000","last_became_active":"2024-03-31T23:09:15.204612+0000","last_became_peered":"2024-03-31T23:09:15.204612+0000","last_unstale":"2024-03-31T23:10:10.541341+0000","last_undegraded":"2024-03-31T23:10:10.541341+0000","last_fullsized":"2024-03-31T23:10:10.541341+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_clean_scrub_stamp":"2024-03-31T23:09:14.171251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:53:33.998026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"91'9","reported_seq":39,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-31T23:10:10.543485+0000","last_change":"2024-03-31T23:09:10.594147+0000","last_active":"2024-03-31T23:10:10.543485+0000","last_peered":"2024-03-31T23:10:10.543485+0000","last_clean":"2024-03-31T23:10:10.543485+0000","last_became_active":"2024-03-31T23:09:10.592047+0000","last_became_peered":"2024-03-31T23:09:10.592047+0000","last_unstale":"2024-03-31T23:10:10.543485+0000","last_undegraded":"2024-03-31T23:10:10.543485+0000","last_fullsized":"2024-03-31T23:10:10.543485+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_clean_scrub_stamp":"2024-03-31T23:09:09.564263+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:04:34.548282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"93'5","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-31T23:10:38.602438+0000","last_change":"2024-03-31T23:09:11.182292+0000","last_active":"2024-03-31T23:10:38.602438+0000","last_peered":"2024-03-31T23:10:38.602438+0000","last_clean":"2024-03-31T23:10:38.602438+0000","last_became_active":"2024-03-31T23:09:11.181885+0000","last_became_peered":"2024-03-31T23:09:11.181885+0000","last_unstale":"2024-03-31T23:10:38.602438+0000","last_undegraded":"2024-03-31T23:10:38.602438+0000","last_fullsized":"2024-03-31T23:10:38.602438+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T10:38:17.550625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":28,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-31T23:10:10.543802+0000","last_change":"2024-03-31T23:10:10.543802+0000","last_active":"2024-03-31T23:10:10.543802+0000","last_peered":"2024-03-31T23:10:10.543802+0000","last_clean":"2024-03-31T23:10:10.543802+0000","last_became_active":"2024-03-31T23:10:10.543245+0000","last_became_peered":"2024-03-31T23:10:10.543245+0000","last_unstale":"2024-03-31T23:10:10.543802+0000","last_undegraded":"2024-03-31T23:10:10.543802+0000","last_fullsized":"2024-03-31T23:10:10.543802+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_clean_scrub_stamp":"2024-03-31T23:09:14.171251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T06:00:11.707183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,8,3],"acting":[4,6,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"91'2","reported_seq":31,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-31T23:10:10.542591+0000","last_change":"2024-03-31T23:09:10.598375+0000","last_active":"2024-03-31T23:10:10.542591+0000","last_peered":"2024-03-31T23:10:10.542591+0000","last_clean":"2024-03-31T23:10:10.542591+0000","last_became_active":"2024-03-31T23:09:10.598045+0000","last_became_peered":"2024-03-31T23:09:10.598045+0000","last_unstale":"2024-03-31T23:10:10.542591+0000","last_undegraded":"2024-03-31T23:10:10.542591+0000","last_fullsized":"2024-03-31T23:10:10.542591+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_clean_scrub_stamp":"2024-03-31T23:09:09.564263+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:31:27.113317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"93'6","reported_seq":33,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-31T23:10:38.612943+0000","last_change":"2024-03-31T23:09:11.183458+0000","last_active":"2024-03-31T23:10:38.612943+0000","last_peered":"2024-03-31T23:10:38.612943+0000","last_clean":"2024-03-31T23:10:38.612943+0000","last_became_active":"2024-03-31T23:09:11.183100+0000","last_became_peered":"2024-03-31T23:09:11.183100+0000","last_unstale":"2024-03-31T23:10:38.612943+0000","last_undegraded":"2024-03-31T23:10:38.612943+0000","last_fullsized":"2024-03-31T23:10:38.612943+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T10:55:35.512065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":21,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-31T23:10:10.479074+0000","last_change":"2024-03-31T23:09:15.208103+0000","last_active":"2024-03-31T23:10:10.479074+0000","last_peered":"2024-03-31T23:10:10.479074+0000","last_clean":"2024-03-31T23:10:10.479074+0000","last_became_active":"2024-03-31T23:09:15.207723+0000","last_became_peered":"2024-03-31T23:09:15.207723+0000","last_unstale":"2024-03-31T23:10:10.479074+0000","last_undegraded":"2024-03-31T23:10:10.479074+0000","last_fullsized":"2024-03-31T23:10:10.479074+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_clean_scrub_stamp":"2024-03-31T23:09:14.171251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T07:55:01.303693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"91'4","reported_seq":33,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-31T23:10:10.480437+0000","last_change":"2024-03-31T23:09:10.597471+0000","last_active":"2024-03-31T23:10:10.480437+0000","last_peered":"2024-03-31T23:10:10.480437+0000","last_clean":"2024-03-31T23:10:10.480437+0000","last_became_active":"2024-03-31T23:09:10.597165+0000","last_became_peered":"2024-03-31T23:09:10.597165+0000","last_unstale":"2024-03-31T23:10:10.480437+0000","last_undegraded":"2024-03-31T23:10:10.480437+0000","last_fullsized":"2024-03-31T23:10:10.480437+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_clean_scrub_stamp":"2024-03-31T23:09:09.564263+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:10:26.826525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"93'9","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-31T23:10:38.617272+0000","last_change":"2024-03-31T23:09:11.170650+0000","last_active":"2024-03-31T23:10:38.617272+0000","last_peered":"2024-03-31T23:10:38.617272+0000","last_clean":"2024-03-31T23:10:38.617272+0000","last_became_active":"2024-03-31T23:09:11.170291+0000","last_became_peered":"2024-03-31T23:09:11.170291+0000","last_unstale":"2024-03-31T23:10:38.617272+0000","last_undegraded":"2024-03-31T23:10:38.617272+0000","last_fullsized":"2024-03-31T23:10:38.617272+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:29:54.904377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":21,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-31T23:10:10.478810+0000","last_change":"2024-03-31T23:09:15.204857+0000","last_active":"2024-03-31T23:10:10.478810+0000","last_peered":"2024-03-31T23:10:10.478810+0000","last_clean":"2024-03-31T23:10:10.478810+0000","last_became_active":"2024-03-31T23:09:15.204484+0000","last_became_peered":"2024-03-31T23:09:15.204484+0000","last_unstale":"2024-03-31T23:10:10.478810+0000","last_undegraded":"2024-03-31T23:10:10.478810+0000","last_fullsized":"2024-03-31T23:10:10.478810+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_clean_scrub_stamp":"2024-03-31T23:09:14.171251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T09:54:24.646918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"85'2","reported_seq":31,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-31T23:10:10.480174+0000","last_change":"2024-03-31T23:09:10.589922+0000","last_active":"2024-03-31T23:10:10.480174+0000","last_peered":"2024-03-31T23:10:10.480174+0000","last_clean":"2024-03-31T23:10:10.480174+0000","last_became_active":"2024-03-31T23:09:10.589569+0000","last_became_peered":"2024-03-31T23:09:10.589569+0000","last_unstale":"2024-03-31T23:10:10.480174+0000","last_undegraded":"2024-03-31T23:10:10.480174+0000","last_fullsized":"2024-03-31T23:10:10.480174+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_clean_scrub_stamp":"2024-03-31T23:09:09.564263+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:57:05.347344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"93'15","reported_seq":48,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-31T23:10:38.603772+0000","last_change":"2024-03-31T23:10:10.485015+0000","last_active":"2024-03-31T23:10:38.603772+0000","last_peered":"2024-03-31T23:10:38.603772+0000","last_clean":"2024-03-31T23:10:38.603772+0000","last_became_active":"2024-03-31T23:10:10.484663+0000","last_became_peered":"2024-03-31T23:10:10.484663+0000","last_unstale":"2024-03-31T23:10:38.603772+0000","last_undegraded":"2024-03-31T23:10:38.603772+0000","last_fullsized":"2024-03-31T23:10:38.603772+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T06:18:50.834711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,1],"acting":[5,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":21,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-31T23:10:10.542109+0000","last_change":"2024-03-31T23:09:15.193096+0000","last_active":"2024-03-31T23:10:10.542109+0000","last_peered":"2024-03-31T23:10:10.542109+0000","last_clean":"2024-03-31T23:10:10.542109+0000","last_became_active":"2024-03-31T23:09:15.192624+0000","last_became_peered":"2024-03-31T23:09:15.192624+0000","last_unstale":"2024-03-31T23:10:10.542109+0000","last_undegraded":"2024-03-31T23:10:10.542109+0000","last_fullsized":"2024-03-31T23:10:10.542109+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_clean_scrub_stamp":"2024-03-31T23:09:14.171251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:22:24.921346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"93'7","reported_seq":112,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-31T23:10:40.462768+0000","last_change":"2024-03-31T23:09:10.587209+0000","last_active":"2024-03-31T23:10:40.462768+0000","last_peered":"2024-03-31T23:10:40.462768+0000","last_clean":"2024-03-31T23:10:40.462768+0000","last_became_active":"2024-03-31T23:09:10.586486+0000","last_became_peered":"2024-03-31T23:09:10.586486+0000","last_unstale":"2024-03-31T23:10:40.462768+0000","last_undegraded":"2024-03-31T23:10:40.462768+0000","last_fullsized":"2024-03-31T23:10:40.462768+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_clean_scrub_stamp":"2024-03-31T23:09:09.564263+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:27:56.046414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1154,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":4,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"93'7","reported_seq":40,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-31T23:10:38.610724+0000","last_change":"2024-03-31T23:10:10.543713+0000","last_active":"2024-03-31T23:10:38.610724+0000","last_peered":"2024-03-31T23:10:38.610724+0000","last_clean":"2024-03-31T23:10:38.610724+0000","last_became_active":"2024-03-31T23:10:10.543324+0000","last_became_peered":"2024-03-31T23:10:10.543324+0000","last_unstale":"2024-03-31T23:10:38.610724+0000","last_undegraded":"2024-03-31T23:10:38.610724+0000","last_fullsized":"2024-03-31T23:10:38.610724+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:41:21.974325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,6],"acting":[8,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":21,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-31T23:10:10.541840+0000","last_change":"2024-03-31T23:09:15.194717+0000","last_active":"2024-03-31T23:10:10.541840+0000","last_peered":"2024-03-31T23:10:10.541840+0000","last_clean":"2024-03-31T23:10:10.541840+0000","last_became_active":"2024-03-31T23:09:15.194232+0000","last_became_peered":"2024-03-31T23:09:15.194232+0000","last_unstale":"2024-03-31T23:10:10.541840+0000","last_undegraded":"2024-03-31T23:10:10.541840+0000","last_fullsized":"2024-03-31T23:10:10.541840+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_clean_scrub_stamp":"2024-03-31T23:09:14.171251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:21:00.445232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"91'3","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-31T23:10:10.482530+0000","last_change":"2024-03-31T23:09:10.586346+0000","last_active":"2024-03-31T23:10:10.482530+0000","last_peered":"2024-03-31T23:10:10.482530+0000","last_clean":"2024-03-31T23:10:10.482530+0000","last_became_active":"2024-03-31T23:09:10.586043+0000","last_became_peered":"2024-03-31T23:09:10.586043+0000","last_unstale":"2024-03-31T23:10:10.482530+0000","last_undegraded":"2024-03-31T23:10:10.482530+0000","last_fullsized":"2024-03-31T23:10:10.482530+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_clean_scrub_stamp":"2024-03-31T23:09:09.564263+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:17:27.387786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"93'8","reported_seq":41,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-31T23:10:38.591432+0000","last_change":"2024-03-31T23:10:10.543353+0000","last_active":"2024-03-31T23:10:38.591432+0000","last_peered":"2024-03-31T23:10:38.591432+0000","last_clean":"2024-03-31T23:10:38.591432+0000","last_became_active":"2024-03-31T23:10:10.542961+0000","last_became_peered":"2024-03-31T23:10:10.542961+0000","last_unstale":"2024-03-31T23:10:38.591432+0000","last_undegraded":"2024-03-31T23:10:38.591432+0000","last_fullsized":"2024-03-31T23:10:38.591432+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:28:52.950551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3],"acting":[5,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":21,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-31T23:10:10.541405+0000","last_change":"2024-03-31T23:09:15.205855+0000","last_active":"2024-03-31T23:10:10.541405+0000","last_peered":"2024-03-31T23:10:10.541405+0000","last_clean":"2024-03-31T23:10:10.541405+0000","last_became_active":"2024-03-31T23:09:15.205359+0000","last_became_peered":"2024-03-31T23:09:15.205359+0000","last_unstale":"2024-03-31T23:10:10.541405+0000","last_undegraded":"2024-03-31T23:10:10.541405+0000","last_fullsized":"2024-03-31T23:10:10.541405+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_clean_scrub_stamp":"2024-03-31T23:09:14.171251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T09:39:32.461011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"91'2","reported_seq":31,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-31T23:10:10.476727+0000","last_change":"2024-03-31T23:09:10.583274+0000","last_active":"2024-03-31T23:10:10.476727+0000","last_peered":"2024-03-31T23:10:10.476727+0000","last_clean":"2024-03-31T23:10:10.476727+0000","last_became_active":"2024-03-31T23:09:10.582910+0000","last_became_peered":"2024-03-31T23:09:10.582910+0000","last_unstale":"2024-03-31T23:10:10.476727+0000","last_undegraded":"2024-03-31T23:10:10.476727+0000","last_fullsized":"2024-03-31T23:10:10.476727+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_clean_scrub_stamp":"2024-03-31T23:09:09.564263+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:16:32.251169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"93'8","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-31T23:10:38.619376+0000","last_change":"2024-03-31T23:09:11.178619+0000","last_active":"2024-03-31T23:10:38.619376+0000","last_peered":"2024-03-31T23:10:38.619376+0000","last_clean":"2024-03-31T23:10:38.619376+0000","last_became_active":"2024-03-31T23:09:11.178239+0000","last_became_peered":"2024-03-31T23:09:11.178239+0000","last_unstale":"2024-03-31T23:10:38.619376+0000","last_undegraded":"2024-03-31T23:10:38.619376+0000","last_fullsized":"2024-03-31T23:10:38.619376+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:28:58.863199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":21,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-31T23:10:10.480544+0000","last_change":"2024-03-31T23:09:15.205864+0000","last_active":"2024-03-31T23:10:10.480544+0000","last_peered":"2024-03-31T23:10:10.480544+0000","last_clean":"2024-03-31T23:10:10.480544+0000","last_became_active":"2024-03-31T23:09:15.204976+0000","last_became_peered":"2024-03-31T23:09:15.204976+0000","last_unstale":"2024-03-31T23:10:10.480544+0000","last_undegraded":"2024-03-31T23:10:10.480544+0000","last_fullsized":"2024-03-31T23:10:10.480544+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_clean_scrub_stamp":"2024-03-31T23:09:14.171251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:39:08.537827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"91'1","reported_seq":30,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-31T23:10:10.477775+0000","last_change":"2024-03-31T23:09:10.588407+0000","last_active":"2024-03-31T23:10:10.477775+0000","last_peered":"2024-03-31T23:10:10.477775+0000","last_clean":"2024-03-31T23:10:10.477775+0000","last_became_active":"2024-03-31T23:09:10.588109+0000","last_became_peered":"2024-03-31T23:09:10.588109+0000","last_unstale":"2024-03-31T23:10:10.477775+0000","last_undegraded":"2024-03-31T23:10:10.477775+0000","last_fullsized":"2024-03-31T23:10:10.477775+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_clean_scrub_stamp":"2024-03-31T23:09:09.564263+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:01:25.858834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"93'11","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-31T23:10:38.616479+0000","last_change":"2024-03-31T23:09:11.180452+0000","last_active":"2024-03-31T23:10:38.616479+0000","last_peered":"2024-03-31T23:10:38.616479+0000","last_clean":"2024-03-31T23:10:38.616479+0000","last_became_active":"2024-03-31T23:09:11.180117+0000","last_became_peered":"2024-03-31T23:09:11.180117+0000","last_unstale":"2024-03-31T23:10:38.616479+0000","last_undegraded":"2024-03-31T23:10:38.616479+0000","last_fullsized":"2024-03-31T23:10:38.616479+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:04:58.288981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":21,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-31T23:10:10.541506+0000","last_change":"2024-03-31T23:09:15.199669+0000","last_active":"2024-03-31T23:10:10.541506+0000","last_peered":"2024-03-31T23:10:10.541506+0000","last_clean":"2024-03-31T23:10:10.541506+0000","last_became_active":"2024-03-31T23:09:15.199262+0000","last_became_peered":"2024-03-31T23:09:15.199262+0000","last_unstale":"2024-03-31T23:10:10.541506+0000","last_undegraded":"2024-03-31T23:10:10.541506+0000","last_fullsized":"2024-03-31T23:10:10.541506+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_clean_scrub_stamp":"2024-03-31T23:09:14.171251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:44:02.975000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"93'4","reported_seq":103,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-31T23:10:38.560823+0000","last_change":"2024-03-31T23:09:10.583476+0000","last_active":"2024-03-31T23:10:38.560823+0000","last_peered":"2024-03-31T23:10:38.560823+0000","last_clean":"2024-03-31T23:10:38.560823+0000","last_became_active":"2024-03-31T23:09:10.582789+0000","last_became_peered":"2024-03-31T23:09:10.582789+0000","last_unstale":"2024-03-31T23:10:38.560823+0000","last_undegraded":"2024-03-31T23:10:38.560823+0000","last_fullsized":"2024-03-31T23:10:38.560823+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_clean_scrub_stamp":"2024-03-31T23:09:09.564263+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T10:52:35.478325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":70,"num_read_kb":70,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"93'10","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-31T23:10:38.613555+0000","last_change":"2024-03-31T23:09:11.171074+0000","last_active":"2024-03-31T23:10:38.613555+0000","last_peered":"2024-03-31T23:10:38.613555+0000","last_clean":"2024-03-31T23:10:38.613555+0000","last_became_active":"2024-03-31T23:09:11.169572+0000","last_became_peered":"2024-03-31T23:09:11.169572+0000","last_unstale":"2024-03-31T23:10:38.613555+0000","last_undegraded":"2024-03-31T23:10:38.613555+0000","last_fullsized":"2024-03-31T23:10:38.613555+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T07:15:34.833388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":21,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-31T23:10:10.542189+0000","last_change":"2024-03-31T23:09:15.203835+0000","last_active":"2024-03-31T23:10:10.542189+0000","last_peered":"2024-03-31T23:10:10.542189+0000","last_clean":"2024-03-31T23:10:10.542189+0000","last_became_active":"2024-03-31T23:09:15.200861+0000","last_became_peered":"2024-03-31T23:09:15.200861+0000","last_unstale":"2024-03-31T23:10:10.542189+0000","last_undegraded":"2024-03-31T23:10:10.542189+0000","last_fullsized":"2024-03-31T23:10:10.542189+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_clean_scrub_stamp":"2024-03-31T23:09:14.171251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:56:19.664708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"91'2","reported_seq":31,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-31T23:10:10.480844+0000","last_change":"2024-03-31T23:09:10.592601+0000","last_active":"2024-03-31T23:10:10.480844+0000","last_peered":"2024-03-31T23:10:10.480844+0000","last_clean":"2024-03-31T23:10:10.480844+0000","last_became_active":"2024-03-31T23:09:10.592345+0000","last_became_peered":"2024-03-31T23:09:10.592345+0000","last_unstale":"2024-03-31T23:10:10.480844+0000","last_undegraded":"2024-03-31T23:10:10.480844+0000","last_fullsized":"2024-03-31T23:10:10.480844+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_clean_scrub_stamp":"2024-03-31T23:09:09.564263+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:10:21.195544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"93'9","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-31T23:10:38.595752+0000","last_change":"2024-03-31T23:09:11.175305+0000","last_active":"2024-03-31T23:10:38.595752+0000","last_peered":"2024-03-31T23:10:38.595752+0000","last_clean":"2024-03-31T23:10:38.595752+0000","last_became_active":"2024-03-31T23:09:11.174970+0000","last_became_peered":"2024-03-31T23:09:11.174970+0000","last_unstale":"2024-03-31T23:10:38.595752+0000","last_undegraded":"2024-03-31T23:10:38.595752+0000","last_fullsized":"2024-03-31T23:10:38.595752+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:29:39.718127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":21,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-31T23:10:10.479177+0000","last_change":"2024-03-31T23:09:15.200365+0000","last_active":"2024-03-31T23:10:10.479177+0000","last_peered":"2024-03-31T23:10:10.479177+0000","last_clean":"2024-03-31T23:10:10.479177+0000","last_became_active":"2024-03-31T23:09:15.199830+0000","last_became_peered":"2024-03-31T23:09:15.199830+0000","last_unstale":"2024-03-31T23:10:10.479177+0000","last_undegraded":"2024-03-31T23:10:10.479177+0000","last_fullsized":"2024-03-31T23:10:10.479177+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_clean_scrub_stamp":"2024-03-31T23:09:14.171251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:37:30.048859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"91'4","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-31T23:10:10.478220+0000","last_change":"2024-03-31T23:09:10.582654+0000","last_active":"2024-03-31T23:10:10.478220+0000","last_peered":"2024-03-31T23:10:10.478220+0000","last_clean":"2024-03-31T23:10:10.478220+0000","last_became_active":"2024-03-31T23:09:10.582314+0000","last_became_peered":"2024-03-31T23:09:10.582314+0000","last_unstale":"2024-03-31T23:10:10.478220+0000","last_undegraded":"2024-03-31T23:10:10.478220+0000","last_fullsized":"2024-03-31T23:10:10.478220+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_clean_scrub_stamp":"2024-03-31T23:09:09.564263+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:51:34.236575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"93'11","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-31T23:10:38.616002+0000","last_change":"2024-03-31T23:09:11.172811+0000","last_active":"2024-03-31T23:10:38.616002+0000","last_peered":"2024-03-31T23:10:38.616002+0000","last_clean":"2024-03-31T23:10:38.616002+0000","last_became_active":"2024-03-31T23:09:11.172478+0000","last_became_peered":"2024-03-31T23:09:11.172478+0000","last_unstale":"2024-03-31T23:10:38.616002+0000","last_undegraded":"2024-03-31T23:10:38.616002+0000","last_fullsized":"2024-03-31T23:10:38.616002+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T10:00:09.825254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":21,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-31T23:10:10.542583+0000","last_change":"2024-03-31T23:09:15.205494+0000","last_active":"2024-03-31T23:10:10.542583+0000","last_peered":"2024-03-31T23:10:10.542583+0000","last_clean":"2024-03-31T23:10:10.542583+0000","last_became_active":"2024-03-31T23:09:15.205101+0000","last_became_peered":"2024-03-31T23:09:15.205101+0000","last_unstale":"2024-03-31T23:10:10.542583+0000","last_undegraded":"2024-03-31T23:10:10.542583+0000","last_fullsized":"2024-03-31T23:10:10.542583+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_clean_scrub_stamp":"2024-03-31T23:09:14.171251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:52:47.472296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"93'37","reported_seq":215,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-31T23:10:39.460913+0000","last_change":"2024-03-31T23:09:10.585611+0000","last_active":"2024-03-31T23:10:39.460913+0000","last_peered":"2024-03-31T23:10:39.460913+0000","last_clean":"2024-03-31T23:10:39.460913+0000","last_became_active":"2024-03-31T23:09:10.585287+0000","last_became_peered":"2024-03-31T23:09:10.585287+0000","last_unstale":"2024-03-31T23:10:39.460913+0000","last_undegraded":"2024-03-31T23:10:39.460913+0000","last_fullsized":"2024-03-31T23:10:39.460913+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_clean_scrub_stamp":"2024-03-31T23:09:09.564263+0000","objects_scrubbed":0,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:47:44.063831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":62582,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":149,"num_read_kb":136,"num_write":38,"num_write_kb":81,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"93'6","reported_seq":33,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-31T23:10:38.593482+0000","last_change":"2024-03-31T23:09:11.175812+0000","last_active":"2024-03-31T23:10:38.593482+0000","last_peered":"2024-03-31T23:10:38.593482+0000","last_clean":"2024-03-31T23:10:38.593482+0000","last_became_active":"2024-03-31T23:09:11.174972+0000","last_became_peered":"2024-03-31T23:09:11.174972+0000","last_unstale":"2024-03-31T23:10:38.593482+0000","last_undegraded":"2024-03-31T23:10:38.593482+0000","last_fullsized":"2024-03-31T23:10:38.593482+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:17:18.870340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":21,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-31T23:10:10.478914+0000","last_change":"2024-03-31T23:09:15.205018+0000","last_active":"2024-03-31T23:10:10.478914+0000","last_peered":"2024-03-31T23:10:10.478914+0000","last_clean":"2024-03-31T23:10:10.478914+0000","last_became_active":"2024-03-31T23:09:15.204510+0000","last_became_peered":"2024-03-31T23:09:15.204510+0000","last_unstale":"2024-03-31T23:10:10.478914+0000","last_undegraded":"2024-03-31T23:10:10.478914+0000","last_fullsized":"2024-03-31T23:10:10.478914+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_clean_scrub_stamp":"2024-03-31T23:09:14.171251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T07:48:59.819937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"91'6","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-31T23:10:10.541447+0000","last_change":"2024-03-31T23:09:10.590233+0000","last_active":"2024-03-31T23:10:10.541447+0000","last_peered":"2024-03-31T23:10:10.541447+0000","last_clean":"2024-03-31T23:10:10.541447+0000","last_became_active":"2024-03-31T23:09:10.589853+0000","last_became_peered":"2024-03-31T23:09:10.589853+0000","last_unstale":"2024-03-31T23:10:10.541447+0000","last_undegraded":"2024-03-31T23:10:10.541447+0000","last_fullsized":"2024-03-31T23:10:10.541447+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_clean_scrub_stamp":"2024-03-31T23:09:09.564263+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:10:54.980394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"93'9","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-31T23:10:38.620015+0000","last_change":"2024-03-31T23:09:11.177871+0000","last_active":"2024-03-31T23:10:38.620015+0000","last_peered":"2024-03-31T23:10:38.620015+0000","last_clean":"2024-03-31T23:10:38.620015+0000","last_became_active":"2024-03-31T23:09:11.176504+0000","last_became_peered":"2024-03-31T23:09:11.176504+0000","last_unstale":"2024-03-31T23:10:38.620015+0000","last_undegraded":"2024-03-31T23:10:38.620015+0000","last_fullsized":"2024-03-31T23:10:38.620015+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T06:09:08.797695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":21,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-31T23:10:10.541260+0000","last_change":"2024-03-31T23:09:15.207336+0000","last_active":"2024-03-31T23:10:10.541260+0000","last_peered":"2024-03-31T23:10:10.541260+0000","last_clean":"2024-03-31T23:10:10.541260+0000","last_became_active":"2024-03-31T23:09:15.206893+0000","last_became_peered":"2024-03-31T23:09:15.206893+0000","last_unstale":"2024-03-31T23:10:10.541260+0000","last_undegraded":"2024-03-31T23:10:10.541260+0000","last_fullsized":"2024-03-31T23:10:10.541260+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_clean_scrub_stamp":"2024-03-31T23:09:14.171251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:24:59.766421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"91'3","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-31T23:10:10.478146+0000","last_change":"2024-03-31T23:09:10.587099+0000","last_active":"2024-03-31T23:10:10.478146+0000","last_peered":"2024-03-31T23:10:10.478146+0000","last_clean":"2024-03-31T23:10:10.478146+0000","last_became_active":"2024-03-31T23:09:10.586429+0000","last_became_peered":"2024-03-31T23:09:10.586429+0000","last_unstale":"2024-03-31T23:10:10.478146+0000","last_undegraded":"2024-03-31T23:10:10.478146+0000","last_fullsized":"2024-03-31T23:10:10.478146+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_clean_scrub_stamp":"2024-03-31T23:09:09.564263+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:55:09.307378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"93'15","reported_seq":42,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-31T23:10:38.605408+0000","last_change":"2024-03-31T23:09:11.177661+0000","last_active":"2024-03-31T23:10:38.605408+0000","last_peered":"2024-03-31T23:10:38.605408+0000","last_clean":"2024-03-31T23:10:38.605408+0000","last_became_active":"2024-03-31T23:09:11.177314+0000","last_became_peered":"2024-03-31T23:09:11.177314+0000","last_unstale":"2024-03-31T23:10:38.605408+0000","last_undegraded":"2024-03-31T23:10:38.605408+0000","last_fullsized":"2024-03-31T23:10:38.605408+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T09:26:50.556837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-31T23:10:10.544127+0000","last_change":"2024-03-31T23:10:10.543526+0000","last_active":"2024-03-31T23:10:10.544127+0000","last_peered":"2024-03-31T23:10:10.544127+0000","last_clean":"2024-03-31T23:10:10.544127+0000","last_became_active":"2024-03-31T23:10:10.543105+0000","last_became_peered":"2024-03-31T23:10:10.543105+0000","last_unstale":"2024-03-31T23:10:10.544127+0000","last_undegraded":"2024-03-31T23:10:10.544127+0000","last_fullsized":"2024-03-31T23:10:10.544127+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_clean_scrub_stamp":"2024-03-31T23:09:14.171251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T09:16:44.529786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,7,0,3],"acting":[8,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.11","version":"93'5","reported_seq":110,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-31T23:10:41.492735+0000","last_change":"2024-03-31T23:09:10.586570+0000","last_active":"2024-03-31T23:10:41.492735+0000","last_peered":"2024-03-31T23:10:41.492735+0000","last_clean":"2024-03-31T23:10:41.492735+0000","last_became_active":"2024-03-31T23:09:10.586239+0000","last_became_peered":"2024-03-31T23:09:10.586239+0000","last_unstale":"2024-03-31T23:10:41.492735+0000","last_undegraded":"2024-03-31T23:10:41.492735+0000","last_fullsized":"2024-03-31T23:10:41.492735+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_clean_scrub_stamp":"2024-03-31T23:09:09.564263+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:23:37.182954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1120,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":77,"num_read_kb":5,"num_write":4,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"93'6","reported_seq":33,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-31T23:10:38.602491+0000","last_change":"2024-03-31T23:09:11.173294+0000","last_active":"2024-03-31T23:10:38.602491+0000","last_peered":"2024-03-31T23:10:38.602491+0000","last_clean":"2024-03-31T23:10:38.602491+0000","last_became_active":"2024-03-31T23:09:11.172926+0000","last_became_peered":"2024-03-31T23:09:11.172926+0000","last_unstale":"2024-03-31T23:10:38.602491+0000","last_undegraded":"2024-03-31T23:10:38.602491+0000","last_fullsized":"2024-03-31T23:10:38.602491+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T06:21:46.603381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":21,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-31T23:10:10.477198+0000","last_change":"2024-03-31T23:09:15.199893+0000","last_active":"2024-03-31T23:10:10.477198+0000","last_peered":"2024-03-31T23:10:10.477198+0000","last_clean":"2024-03-31T23:10:10.477198+0000","last_became_active":"2024-03-31T23:09:15.199517+0000","last_became_peered":"2024-03-31T23:09:15.199517+0000","last_unstale":"2024-03-31T23:10:10.477198+0000","last_undegraded":"2024-03-31T23:10:10.477198+0000","last_fullsized":"2024-03-31T23:10:10.477198+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_clean_scrub_stamp":"2024-03-31T23:09:14.171251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T07:48:35.261108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"91'15","reported_seq":45,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-31T23:10:10.480140+0000","last_change":"2024-03-31T23:09:10.588213+0000","last_active":"2024-03-31T23:10:10.480140+0000","last_peered":"2024-03-31T23:10:10.480140+0000","last_clean":"2024-03-31T23:10:10.480140+0000","last_became_active":"2024-03-31T23:09:10.587579+0000","last_became_peered":"2024-03-31T23:09:10.587579+0000","last_unstale":"2024-03-31T23:10:10.480140+0000","last_undegraded":"2024-03-31T23:10:10.480140+0000","last_fullsized":"2024-03-31T23:10:10.480140+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_clean_scrub_stamp":"2024-03-31T23:09:09.564263+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:00:17.883435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"93'6","reported_seq":33,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-31T23:10:38.602643+0000","last_change":"2024-03-31T23:09:11.182427+0000","last_active":"2024-03-31T23:10:38.602643+0000","last_peered":"2024-03-31T23:10:38.602643+0000","last_clean":"2024-03-31T23:10:38.602643+0000","last_became_active":"2024-03-31T23:09:11.182120+0000","last_became_peered":"2024-03-31T23:09:11.182120+0000","last_unstale":"2024-03-31T23:10:38.602643+0000","last_undegraded":"2024-03-31T23:10:38.602643+0000","last_fullsized":"2024-03-31T23:10:38.602643+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:17:48.728424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":21,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-31T23:10:10.477809+0000","last_change":"2024-03-31T23:09:15.207831+0000","last_active":"2024-03-31T23:10:10.477809+0000","last_peered":"2024-03-31T23:10:10.477809+0000","last_clean":"2024-03-31T23:10:10.477809+0000","last_became_active":"2024-03-31T23:09:15.207432+0000","last_became_peered":"2024-03-31T23:09:15.207432+0000","last_unstale":"2024-03-31T23:10:10.477809+0000","last_undegraded":"2024-03-31T23:10:10.477809+0000","last_fullsized":"2024-03-31T23:10:10.477809+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_clean_scrub_stamp":"2024-03-31T23:09:14.171251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:05:27.072054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"91'2","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-31T23:10:10.480720+0000","last_change":"2024-03-31T23:09:10.591483+0000","last_active":"2024-03-31T23:10:10.480720+0000","last_peered":"2024-03-31T23:10:10.480720+0000","last_clean":"2024-03-31T23:10:10.480720+0000","last_became_active":"2024-03-31T23:09:10.591173+0000","last_became_peered":"2024-03-31T23:09:10.591173+0000","last_unstale":"2024-03-31T23:10:10.480720+0000","last_undegraded":"2024-03-31T23:10:10.480720+0000","last_fullsized":"2024-03-31T23:10:10.480720+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_clean_scrub_stamp":"2024-03-31T23:09:09.564263+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T10:04:14.275875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"93'5","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-31T23:10:38.602546+0000","last_change":"2024-03-31T23:09:11.181171+0000","last_active":"2024-03-31T23:10:38.602546+0000","last_peered":"2024-03-31T23:10:38.602546+0000","last_clean":"2024-03-31T23:10:38.602546+0000","last_became_active":"2024-03-31T23:09:11.180847+0000","last_became_peered":"2024-03-31T23:09:11.180847+0000","last_unstale":"2024-03-31T23:10:38.602546+0000","last_undegraded":"2024-03-31T23:10:38.602546+0000","last_fullsized":"2024-03-31T23:10:38.602546+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:39:29.429009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":21,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-31T23:10:10.480164+0000","last_change":"2024-03-31T23:09:15.746941+0000","last_active":"2024-03-31T23:10:10.480164+0000","last_peered":"2024-03-31T23:10:10.480164+0000","last_clean":"2024-03-31T23:10:10.480164+0000","last_became_active":"2024-03-31T23:09:15.746352+0000","last_became_peered":"2024-03-31T23:09:15.746352+0000","last_unstale":"2024-03-31T23:10:10.480164+0000","last_undegraded":"2024-03-31T23:10:10.480164+0000","last_fullsized":"2024-03-31T23:10:10.480164+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_clean_scrub_stamp":"2024-03-31T23:09:14.171251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T06:21:43.482575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":29,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-31T23:10:10.542934+0000","last_change":"2024-03-31T23:09:10.593029+0000","last_active":"2024-03-31T23:10:10.542934+0000","last_peered":"2024-03-31T23:10:10.542934+0000","last_clean":"2024-03-31T23:10:10.542934+0000","last_became_active":"2024-03-31T23:09:10.592709+0000","last_became_peered":"2024-03-31T23:09:10.592709+0000","last_unstale":"2024-03-31T23:10:10.542934+0000","last_undegraded":"2024-03-31T23:10:10.542934+0000","last_fullsized":"2024-03-31T23:10:10.542934+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_clean_scrub_stamp":"2024-03-31T23:09:09.564263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:46:45.700207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"93'4","reported_seq":31,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-31T23:10:38.612304+0000","last_change":"2024-03-31T23:09:11.177857+0000","last_active":"2024-03-31T23:10:38.612304+0000","last_peered":"2024-03-31T23:10:38.612304+0000","last_clean":"2024-03-31T23:10:38.612304+0000","last_became_active":"2024-03-31T23:09:11.177513+0000","last_became_peered":"2024-03-31T23:09:11.177513+0000","last_unstale":"2024-03-31T23:10:38.612304+0000","last_undegraded":"2024-03-31T23:10:38.612304+0000","last_fullsized":"2024-03-31T23:10:38.612304+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T06:09:23.579208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":21,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-31T23:10:10.476986+0000","last_change":"2024-03-31T23:09:15.203412+0000","last_active":"2024-03-31T23:10:10.476986+0000","last_peered":"2024-03-31T23:10:10.476986+0000","last_clean":"2024-03-31T23:10:10.476986+0000","last_became_active":"2024-03-31T23:09:15.202983+0000","last_became_peered":"2024-03-31T23:09:15.202983+0000","last_unstale":"2024-03-31T23:10:10.476986+0000","last_undegraded":"2024-03-31T23:10:10.476986+0000","last_fullsized":"2024-03-31T23:10:10.476986+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_clean_scrub_stamp":"2024-03-31T23:09:14.171251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:57:46.357378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"91'1","reported_seq":30,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-31T23:10:10.480628+0000","last_change":"2024-03-31T23:09:10.593299+0000","last_active":"2024-03-31T23:10:10.480628+0000","last_peered":"2024-03-31T23:10:10.480628+0000","last_clean":"2024-03-31T23:10:10.480628+0000","last_became_active":"2024-03-31T23:09:10.593007+0000","last_became_peered":"2024-03-31T23:09:10.593007+0000","last_unstale":"2024-03-31T23:10:10.480628+0000","last_undegraded":"2024-03-31T23:10:10.480628+0000","last_fullsized":"2024-03-31T23:10:10.480628+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_clean_scrub_stamp":"2024-03-31T23:09:09.564263+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:15:07.199645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"93'7","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-31T23:10:38.606243+0000","last_change":"2024-03-31T23:09:11.176846+0000","last_active":"2024-03-31T23:10:38.606243+0000","last_peered":"2024-03-31T23:10:38.606243+0000","last_clean":"2024-03-31T23:10:38.606243+0000","last_became_active":"2024-03-31T23:09:11.176554+0000","last_became_peered":"2024-03-31T23:09:11.176554+0000","last_unstale":"2024-03-31T23:10:38.606243+0000","last_undegraded":"2024-03-31T23:10:38.606243+0000","last_fullsized":"2024-03-31T23:10:38.606243+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:25:23.639093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":21,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-31T23:10:10.477184+0000","last_change":"2024-03-31T23:09:15.746599+0000","last_active":"2024-03-31T23:10:10.477184+0000","last_peered":"2024-03-31T23:10:10.477184+0000","last_clean":"2024-03-31T23:10:10.477184+0000","last_became_active":"2024-03-31T23:09:15.745879+0000","last_became_peered":"2024-03-31T23:09:15.745879+0000","last_unstale":"2024-03-31T23:10:10.477184+0000","last_undegraded":"2024-03-31T23:10:10.477184+0000","last_fullsized":"2024-03-31T23:10:10.477184+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_clean_scrub_stamp":"2024-03-31T23:09:14.171251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T08:02:25.198610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"93'8","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-31T23:10:38.585201+0000","last_change":"2024-03-31T23:09:11.175546+0000","last_active":"2024-03-31T23:10:38.585201+0000","last_peered":"2024-03-31T23:10:38.585201+0000","last_clean":"2024-03-31T23:10:38.585201+0000","last_became_active":"2024-03-31T23:09:11.175253+0000","last_became_peered":"2024-03-31T23:09:11.175253+0000","last_unstale":"2024-03-31T23:10:38.585201+0000","last_undegraded":"2024-03-31T23:10:38.585201+0000","last_fullsized":"2024-03-31T23:10:38.585201+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T10:27:31.753102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"93'5","reported_seq":103,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-31T23:10:38.560800+0000","last_change":"2024-03-31T23:09:10.580888+0000","last_active":"2024-03-31T23:10:38.560800+0000","last_peered":"2024-03-31T23:10:38.560800+0000","last_clean":"2024-03-31T23:10:38.560800+0000","last_became_active":"2024-03-31T23:09:10.580557+0000","last_became_peered":"2024-03-31T23:09:10.580557+0000","last_unstale":"2024-03-31T23:10:38.560800+0000","last_undegraded":"2024-03-31T23:10:38.560800+0000","last_fullsized":"2024-03-31T23:10:38.560800+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_clean_scrub_stamp":"2024-03-31T23:09:09.564263+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:17:38.878570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":69,"num_read_kb":69,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"91'1","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-31T23:10:10.480832+0000","last_change":"2024-03-31T23:09:15.205282+0000","last_active":"2024-03-31T23:10:10.480832+0000","last_peered":"2024-03-31T23:10:10.480832+0000","last_clean":"2024-03-31T23:10:10.480832+0000","last_became_active":"2024-03-31T23:09:15.204873+0000","last_became_peered":"2024-03-31T23:09:15.204873+0000","last_unstale":"2024-03-31T23:10:10.480832+0000","last_undegraded":"2024-03-31T23:10:10.480832+0000","last_fullsized":"2024-03-31T23:10:10.480832+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_clean_scrub_stamp":"2024-03-31T23:09:14.171251+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:56:05.697685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"93'8","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-31T23:10:38.619516+0000","last_change":"2024-03-31T23:09:11.182155+0000","last_active":"2024-03-31T23:10:38.619516+0000","last_peered":"2024-03-31T23:10:38.619516+0000","last_clean":"2024-03-31T23:10:38.619516+0000","last_became_active":"2024-03-31T23:09:11.181829+0000","last_became_peered":"2024-03-31T23:09:11.181829+0000","last_unstale":"2024-03-31T23:10:38.619516+0000","last_undegraded":"2024-03-31T23:10:38.619516+0000","last_fullsized":"2024-03-31T23:10:38.619516+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:19:22.832440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"91'3","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-31T23:10:10.479126+0000","last_change":"2024-03-31T23:09:10.590451+0000","last_active":"2024-03-31T23:10:10.479126+0000","last_peered":"2024-03-31T23:10:10.479126+0000","last_clean":"2024-03-31T23:10:10.479126+0000","last_became_active":"2024-03-31T23:09:10.590123+0000","last_became_peered":"2024-03-31T23:09:10.590123+0000","last_unstale":"2024-03-31T23:10:10.479126+0000","last_undegraded":"2024-03-31T23:10:10.479126+0000","last_fullsized":"2024-03-31T23:10:10.479126+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_clean_scrub_stamp":"2024-03-31T23:09:09.564263+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:27:02.724456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":365,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-31T23:10:10.480271+0000","last_change":"2024-03-31T23:07:57.920942+0000","last_active":"2024-03-31T23:10:10.480271+0000","last_peered":"2024-03-31T23:10:10.480271+0000","last_clean":"2024-03-31T23:10:10.480271+0000","last_became_active":"2024-03-31T23:07:57.920381+0000","last_became_peered":"2024-03-31T23:07:57.920381+0000","last_unstale":"2024-03-31T23:10:10.480271+0000","last_undegraded":"2024-03-31T23:10:10.480271+0000","last_fullsized":"2024-03-31T23:10:10.480271+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:04:33.508180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:04:33.508180+0000","last_clean_scrub_stamp":"2024-03-31T23:04:33.508180+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:20:11.211403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":21,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-31T23:10:10.478319+0000","last_change":"2024-03-31T23:09:15.746822+0000","last_active":"2024-03-31T23:10:10.478319+0000","last_peered":"2024-03-31T23:10:10.478319+0000","last_clean":"2024-03-31T23:10:10.478319+0000","last_became_active":"2024-03-31T23:09:15.746368+0000","last_became_peered":"2024-03-31T23:09:15.746368+0000","last_unstale":"2024-03-31T23:10:10.478319+0000","last_undegraded":"2024-03-31T23:10:10.478319+0000","last_fullsized":"2024-03-31T23:10:10.478319+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_clean_scrub_stamp":"2024-03-31T23:09:14.171251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T06:36:35.658042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"93'5","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-31T23:10:38.608658+0000","last_change":"2024-03-31T23:09:11.175336+0000","last_active":"2024-03-31T23:10:38.608658+0000","last_peered":"2024-03-31T23:10:38.608658+0000","last_clean":"2024-03-31T23:10:38.608658+0000","last_became_active":"2024-03-31T23:09:11.174768+0000","last_became_peered":"2024-03-31T23:09:11.174768+0000","last_unstale":"2024-03-31T23:10:38.608658+0000","last_undegraded":"2024-03-31T23:10:38.608658+0000","last_fullsized":"2024-03-31T23:10:38.608658+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:09:11.678744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"91'3","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-31T23:10:10.480389+0000","last_change":"2024-03-31T23:09:10.591807+0000","last_active":"2024-03-31T23:10:10.480389+0000","last_peered":"2024-03-31T23:10:10.480389+0000","last_clean":"2024-03-31T23:10:10.480389+0000","last_became_active":"2024-03-31T23:09:10.591407+0000","last_became_peered":"2024-03-31T23:09:10.591407+0000","last_unstale":"2024-03-31T23:10:10.480389+0000","last_undegraded":"2024-03-31T23:10:10.480389+0000","last_fullsized":"2024-03-31T23:10:10.480389+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_clean_scrub_stamp":"2024-03-31T23:09:09.564263+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T08:46:33.660717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":21,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-31T23:10:10.541292+0000","last_change":"2024-03-31T23:09:15.203879+0000","last_active":"2024-03-31T23:10:10.541292+0000","last_peered":"2024-03-31T23:10:10.541292+0000","last_clean":"2024-03-31T23:10:10.541292+0000","last_became_active":"2024-03-31T23:09:15.203412+0000","last_became_peered":"2024-03-31T23:09:15.203412+0000","last_unstale":"2024-03-31T23:10:10.541292+0000","last_undegraded":"2024-03-31T23:10:10.541292+0000","last_fullsized":"2024-03-31T23:10:10.541292+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_clean_scrub_stamp":"2024-03-31T23:09:14.171251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:03:54.486038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"91'1","reported_seq":31,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-31T23:10:10.480617+0000","last_change":"2024-03-31T23:09:10.596404+0000","last_active":"2024-03-31T23:10:10.480617+0000","last_peered":"2024-03-31T23:10:10.480617+0000","last_clean":"2024-03-31T23:10:10.480617+0000","last_became_active":"2024-03-31T23:09:10.596031+0000","last_became_peered":"2024-03-31T23:09:10.596031+0000","last_unstale":"2024-03-31T23:10:10.480617+0000","last_undegraded":"2024-03-31T23:10:10.480617+0000","last_fullsized":"2024-03-31T23:10:10.480617+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_clean_scrub_stamp":"2024-03-31T23:09:09.564263+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:30:31.904241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"93'9","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-31T23:10:38.607453+0000","last_change":"2024-03-31T23:09:11.164300+0000","last_active":"2024-03-31T23:10:38.607453+0000","last_peered":"2024-03-31T23:10:38.607453+0000","last_clean":"2024-03-31T23:10:38.607453+0000","last_became_active":"2024-03-31T23:09:11.163960+0000","last_became_peered":"2024-03-31T23:09:11.163960+0000","last_unstale":"2024-03-31T23:10:38.607453+0000","last_undegraded":"2024-03-31T23:10:38.607453+0000","last_fullsized":"2024-03-31T23:10:38.607453+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T10:37:30.512216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":21,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-31T23:10:10.541636+0000","last_change":"2024-03-31T23:09:15.746426+0000","last_active":"2024-03-31T23:10:10.541636+0000","last_peered":"2024-03-31T23:10:10.541636+0000","last_clean":"2024-03-31T23:10:10.541636+0000","last_became_active":"2024-03-31T23:09:15.745887+0000","last_became_peered":"2024-03-31T23:09:15.745887+0000","last_unstale":"2024-03-31T23:10:10.541636+0000","last_undegraded":"2024-03-31T23:10:10.541636+0000","last_fullsized":"2024-03-31T23:10:10.541636+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_clean_scrub_stamp":"2024-03-31T23:09:14.171251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:41:52.005300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"93'9","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-31T23:10:10.543326+0000","last_change":"2024-03-31T23:09:10.590631+0000","last_active":"2024-03-31T23:10:10.543326+0000","last_peered":"2024-03-31T23:10:10.543326+0000","last_clean":"2024-03-31T23:10:10.543326+0000","last_became_active":"2024-03-31T23:09:10.590211+0000","last_became_peered":"2024-03-31T23:09:10.590211+0000","last_unstale":"2024-03-31T23:10:10.543326+0000","last_undegraded":"2024-03-31T23:10:10.543326+0000","last_fullsized":"2024-03-31T23:10:10.543326+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_clean_scrub_stamp":"2024-03-31T23:09:09.564263+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T08:51:24.978892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"93'5","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-31T23:10:38.619166+0000","last_change":"2024-03-31T23:09:11.177353+0000","last_active":"2024-03-31T23:10:38.619166+0000","last_peered":"2024-03-31T23:10:38.619166+0000","last_clean":"2024-03-31T23:10:38.619166+0000","last_became_active":"2024-03-31T23:09:11.176996+0000","last_became_peered":"2024-03-31T23:09:11.176996+0000","last_unstale":"2024-03-31T23:10:38.619166+0000","last_undegraded":"2024-03-31T23:10:38.619166+0000","last_fullsized":"2024-03-31T23:10:38.619166+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:20:52.486281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":21,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-31T23:10:10.541115+0000","last_change":"2024-03-31T23:09:15.198528+0000","last_active":"2024-03-31T23:10:10.541115+0000","last_peered":"2024-03-31T23:10:10.541115+0000","last_clean":"2024-03-31T23:10:10.541115+0000","last_became_active":"2024-03-31T23:09:15.197938+0000","last_became_peered":"2024-03-31T23:09:15.197938+0000","last_unstale":"2024-03-31T23:10:10.541115+0000","last_undegraded":"2024-03-31T23:10:10.541115+0000","last_fullsized":"2024-03-31T23:10:10.541115+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_clean_scrub_stamp":"2024-03-31T23:09:14.171251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:45:46.567268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"91'4","reported_seq":33,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-31T23:10:10.479878+0000","last_change":"2024-03-31T23:09:10.584779+0000","last_active":"2024-03-31T23:10:10.479878+0000","last_peered":"2024-03-31T23:10:10.479878+0000","last_clean":"2024-03-31T23:10:10.479878+0000","last_became_active":"2024-03-31T23:09:10.584400+0000","last_became_peered":"2024-03-31T23:09:10.584400+0000","last_unstale":"2024-03-31T23:10:10.479878+0000","last_undegraded":"2024-03-31T23:10:10.479878+0000","last_fullsized":"2024-03-31T23:10:10.479878+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_clean_scrub_stamp":"2024-03-31T23:09:09.564263+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:10:10.095704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"93'7","reported_seq":40,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-31T23:10:38.618868+0000","last_change":"2024-03-31T23:10:10.543538+0000","last_active":"2024-03-31T23:10:38.618868+0000","last_peered":"2024-03-31T23:10:38.618868+0000","last_clean":"2024-03-31T23:10:38.618868+0000","last_became_active":"2024-03-31T23:10:10.543083+0000","last_became_peered":"2024-03-31T23:10:10.543083+0000","last_unstale":"2024-03-31T23:10:38.618868+0000","last_undegraded":"2024-03-31T23:10:38.618868+0000","last_fullsized":"2024-03-31T23:10:38.618868+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:38:50.656801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":21,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-31T23:10:10.479567+0000","last_change":"2024-03-31T23:09:15.746525+0000","last_active":"2024-03-31T23:10:10.479567+0000","last_peered":"2024-03-31T23:10:10.479567+0000","last_clean":"2024-03-31T23:10:10.479567+0000","last_became_active":"2024-03-31T23:09:15.746126+0000","last_became_peered":"2024-03-31T23:09:15.746126+0000","last_unstale":"2024-03-31T23:10:10.479567+0000","last_undegraded":"2024-03-31T23:10:10.479567+0000","last_fullsized":"2024-03-31T23:10:10.479567+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_clean_scrub_stamp":"2024-03-31T23:09:14.171251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:04:57.313615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"91'2","reported_seq":31,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-31T23:10:10.483343+0000","last_change":"2024-03-31T23:09:10.591469+0000","last_active":"2024-03-31T23:10:10.483343+0000","last_peered":"2024-03-31T23:10:10.483343+0000","last_clean":"2024-03-31T23:10:10.483343+0000","last_became_active":"2024-03-31T23:09:10.591166+0000","last_became_peered":"2024-03-31T23:09:10.591166+0000","last_unstale":"2024-03-31T23:10:10.483343+0000","last_undegraded":"2024-03-31T23:10:10.483343+0000","last_fullsized":"2024-03-31T23:10:10.483343+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_clean_scrub_stamp":"2024-03-31T23:09:09.564263+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T10:30:40.279041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"93'2","reported_seq":29,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-31T23:10:38.562829+0000","last_change":"2024-03-31T23:09:11.174471+0000","last_active":"2024-03-31T23:10:38.562829+0000","last_peered":"2024-03-31T23:10:38.562829+0000","last_clean":"2024-03-31T23:10:38.562829+0000","last_became_active":"2024-03-31T23:09:11.174154+0000","last_became_peered":"2024-03-31T23:09:11.174154+0000","last_unstale":"2024-03-31T23:10:38.562829+0000","last_undegraded":"2024-03-31T23:10:38.562829+0000","last_fullsized":"2024-03-31T23:10:38.562829+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T06:10:14.727009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":21,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-31T23:10:10.543374+0000","last_change":"2024-03-31T23:09:15.201416+0000","last_active":"2024-03-31T23:10:10.543374+0000","last_peered":"2024-03-31T23:10:10.543374+0000","last_clean":"2024-03-31T23:10:10.543374+0000","last_became_active":"2024-03-31T23:09:15.200987+0000","last_became_peered":"2024-03-31T23:09:15.200987+0000","last_unstale":"2024-03-31T23:10:10.543374+0000","last_undegraded":"2024-03-31T23:10:10.543374+0000","last_fullsized":"2024-03-31T23:10:10.543374+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_clean_scrub_stamp":"2024-03-31T23:09:14.171251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T10:56:54.872988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"91'3","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-31T23:10:10.541628+0000","last_change":"2024-03-31T23:09:10.594035+0000","last_active":"2024-03-31T23:10:10.541628+0000","last_peered":"2024-03-31T23:10:10.541628+0000","last_clean":"2024-03-31T23:10:10.541628+0000","last_became_active":"2024-03-31T23:09:10.593710+0000","last_became_peered":"2024-03-31T23:09:10.593710+0000","last_unstale":"2024-03-31T23:10:10.541628+0000","last_undegraded":"2024-03-31T23:10:10.541628+0000","last_fullsized":"2024-03-31T23:10:10.541628+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_clean_scrub_stamp":"2024-03-31T23:09:09.564263+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:46:48.580244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"93'9","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-31T23:10:38.611997+0000","last_change":"2024-03-31T23:09:11.170501+0000","last_active":"2024-03-31T23:10:38.611997+0000","last_peered":"2024-03-31T23:10:38.611997+0000","last_clean":"2024-03-31T23:10:38.611997+0000","last_became_active":"2024-03-31T23:09:11.170068+0000","last_became_peered":"2024-03-31T23:09:11.170068+0000","last_unstale":"2024-03-31T23:10:38.611997+0000","last_undegraded":"2024-03-31T23:10:38.611997+0000","last_fullsized":"2024-03-31T23:10:38.611997+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T06:05:53.488957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"91'1","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-31T23:10:10.479575+0000","last_change":"2024-03-31T23:09:15.208092+0000","last_active":"2024-03-31T23:10:10.479575+0000","last_peered":"2024-03-31T23:10:10.479575+0000","last_clean":"2024-03-31T23:10:10.479575+0000","last_became_active":"2024-03-31T23:09:15.207750+0000","last_became_peered":"2024-03-31T23:09:15.207750+0000","last_unstale":"2024-03-31T23:10:10.479575+0000","last_undegraded":"2024-03-31T23:10:10.479575+0000","last_fullsized":"2024-03-31T23:10:10.479575+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_clean_scrub_stamp":"2024-03-31T23:09:14.171251+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:24:39.649569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"91'1","reported_seq":30,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-31T23:10:10.541384+0000","last_change":"2024-03-31T23:09:10.591275+0000","last_active":"2024-03-31T23:10:10.541384+0000","last_peered":"2024-03-31T23:10:10.541384+0000","last_clean":"2024-03-31T23:10:10.541384+0000","last_became_active":"2024-03-31T23:09:10.590652+0000","last_became_peered":"2024-03-31T23:09:10.590652+0000","last_unstale":"2024-03-31T23:10:10.541384+0000","last_undegraded":"2024-03-31T23:10:10.541384+0000","last_fullsized":"2024-03-31T23:10:10.541384+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_clean_scrub_stamp":"2024-03-31T23:09:09.564263+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:58:39.037196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"93'8","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-31T23:10:38.617415+0000","last_change":"2024-03-31T23:09:11.175169+0000","last_active":"2024-03-31T23:10:38.617415+0000","last_peered":"2024-03-31T23:10:38.617415+0000","last_clean":"2024-03-31T23:10:38.617415+0000","last_became_active":"2024-03-31T23:09:11.174849+0000","last_became_peered":"2024-03-31T23:09:11.174849+0000","last_unstale":"2024-03-31T23:10:38.617415+0000","last_undegraded":"2024-03-31T23:10:38.617415+0000","last_fullsized":"2024-03-31T23:10:38.617415+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:58:06.814034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":21,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-31T23:10:10.479575+0000","last_change":"2024-03-31T23:09:15.746423+0000","last_active":"2024-03-31T23:10:10.479575+0000","last_peered":"2024-03-31T23:10:10.479575+0000","last_clean":"2024-03-31T23:10:10.479575+0000","last_became_active":"2024-03-31T23:09:15.745942+0000","last_became_peered":"2024-03-31T23:09:15.745942+0000","last_unstale":"2024-03-31T23:10:10.479575+0000","last_undegraded":"2024-03-31T23:10:10.479575+0000","last_fullsized":"2024-03-31T23:10:10.479575+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_clean_scrub_stamp":"2024-03-31T23:09:14.171251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:56:02.903212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"91'4","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-31T23:10:10.480085+0000","last_change":"2024-03-31T23:09:10.597524+0000","last_active":"2024-03-31T23:10:10.480085+0000","last_peered":"2024-03-31T23:10:10.480085+0000","last_clean":"2024-03-31T23:10:10.480085+0000","last_became_active":"2024-03-31T23:09:10.597222+0000","last_became_peered":"2024-03-31T23:09:10.597222+0000","last_unstale":"2024-03-31T23:10:10.480085+0000","last_undegraded":"2024-03-31T23:10:10.480085+0000","last_fullsized":"2024-03-31T23:10:10.480085+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_clean_scrub_stamp":"2024-03-31T23:09:09.564263+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T09:24:20.160978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"93'5","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-31T23:10:38.613128+0000","last_change":"2024-03-31T23:09:11.170440+0000","last_active":"2024-03-31T23:10:38.613128+0000","last_peered":"2024-03-31T23:10:38.613128+0000","last_clean":"2024-03-31T23:10:38.613128+0000","last_became_active":"2024-03-31T23:09:11.170081+0000","last_became_peered":"2024-03-31T23:09:11.170081+0000","last_unstale":"2024-03-31T23:10:38.613128+0000","last_undegraded":"2024-03-31T23:10:38.613128+0000","last_fullsized":"2024-03-31T23:10:38.613128+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:53:31.429822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":21,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-31T23:10:10.483461+0000","last_change":"2024-03-31T23:09:15.207146+0000","last_active":"2024-03-31T23:10:10.483461+0000","last_peered":"2024-03-31T23:10:10.483461+0000","last_clean":"2024-03-31T23:10:10.483461+0000","last_became_active":"2024-03-31T23:09:15.206769+0000","last_became_peered":"2024-03-31T23:09:15.206769+0000","last_unstale":"2024-03-31T23:10:10.483461+0000","last_undegraded":"2024-03-31T23:10:10.483461+0000","last_fullsized":"2024-03-31T23:10:10.483461+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_clean_scrub_stamp":"2024-03-31T23:09:14.171251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T07:42:12.961109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"91'3","reported_seq":33,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-31T23:10:10.541960+0000","last_change":"2024-03-31T23:09:10.587291+0000","last_active":"2024-03-31T23:10:10.541960+0000","last_peered":"2024-03-31T23:10:10.541960+0000","last_clean":"2024-03-31T23:10:10.541960+0000","last_became_active":"2024-03-31T23:09:10.585992+0000","last_became_peered":"2024-03-31T23:09:10.585992+0000","last_unstale":"2024-03-31T23:10:10.541960+0000","last_undegraded":"2024-03-31T23:10:10.541960+0000","last_fullsized":"2024-03-31T23:10:10.541960+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_clean_scrub_stamp":"2024-03-31T23:09:09.564263+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:28:00.786268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"93'7","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-31T23:10:38.614229+0000","last_change":"2024-03-31T23:09:11.179369+0000","last_active":"2024-03-31T23:10:38.614229+0000","last_peered":"2024-03-31T23:10:38.614229+0000","last_clean":"2024-03-31T23:10:38.614229+0000","last_became_active":"2024-03-31T23:09:11.178776+0000","last_became_peered":"2024-03-31T23:09:11.178776+0000","last_unstale":"2024-03-31T23:10:38.614229+0000","last_undegraded":"2024-03-31T23:10:38.614229+0000","last_fullsized":"2024-03-31T23:10:38.614229+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T07:56:05.201338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":21,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-31T23:10:10.480088+0000","last_change":"2024-03-31T23:09:15.201462+0000","last_active":"2024-03-31T23:10:10.480088+0000","last_peered":"2024-03-31T23:10:10.480088+0000","last_clean":"2024-03-31T23:10:10.480088+0000","last_became_active":"2024-03-31T23:09:15.201067+0000","last_became_peered":"2024-03-31T23:09:15.201067+0000","last_unstale":"2024-03-31T23:10:10.480088+0000","last_undegraded":"2024-03-31T23:10:10.480088+0000","last_fullsized":"2024-03-31T23:10:10.480088+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_clean_scrub_stamp":"2024-03-31T23:09:14.171251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:54:29.339456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"85'1","reported_seq":31,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-31T23:10:10.478802+0000","last_change":"2024-03-31T23:09:10.584752+0000","last_active":"2024-03-31T23:10:10.478802+0000","last_peered":"2024-03-31T23:10:10.478802+0000","last_clean":"2024-03-31T23:10:10.478802+0000","last_became_active":"2024-03-31T23:09:10.584432+0000","last_became_peered":"2024-03-31T23:09:10.584432+0000","last_unstale":"2024-03-31T23:10:10.478802+0000","last_undegraded":"2024-03-31T23:10:10.478802+0000","last_fullsized":"2024-03-31T23:10:10.478802+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_clean_scrub_stamp":"2024-03-31T23:09:09.564263+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T10:50:53.889719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"93'6","reported_seq":39,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-31T23:10:38.609372+0000","last_change":"2024-03-31T23:10:10.543993+0000","last_active":"2024-03-31T23:10:38.609372+0000","last_peered":"2024-03-31T23:10:38.609372+0000","last_clean":"2024-03-31T23:10:38.609372+0000","last_became_active":"2024-03-31T23:10:10.543627+0000","last_became_peered":"2024-03-31T23:10:10.543627+0000","last_unstale":"2024-03-31T23:10:38.609372+0000","last_undegraded":"2024-03-31T23:10:38.609372+0000","last_fullsized":"2024-03-31T23:10:38.609372+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T08:46:41.221122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":21,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-31T23:10:10.477677+0000","last_change":"2024-03-31T23:09:15.197271+0000","last_active":"2024-03-31T23:10:10.477677+0000","last_peered":"2024-03-31T23:10:10.477677+0000","last_clean":"2024-03-31T23:10:10.477677+0000","last_became_active":"2024-03-31T23:09:15.196339+0000","last_became_peered":"2024-03-31T23:09:15.196339+0000","last_unstale":"2024-03-31T23:10:10.477677+0000","last_undegraded":"2024-03-31T23:10:10.477677+0000","last_fullsized":"2024-03-31T23:10:10.477677+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_clean_scrub_stamp":"2024-03-31T23:09:14.171251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:44:13.044278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"85'2","reported_seq":31,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-31T23:10:10.477160+0000","last_change":"2024-03-31T23:09:10.595131+0000","last_active":"2024-03-31T23:10:10.477160+0000","last_peered":"2024-03-31T23:10:10.477160+0000","last_clean":"2024-03-31T23:10:10.477160+0000","last_became_active":"2024-03-31T23:09:10.594740+0000","last_became_peered":"2024-03-31T23:09:10.594740+0000","last_unstale":"2024-03-31T23:10:10.477160+0000","last_undegraded":"2024-03-31T23:10:10.477160+0000","last_fullsized":"2024-03-31T23:10:10.477160+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_clean_scrub_stamp":"2024-03-31T23:09:09.564263+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:52:41.849621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"93'9","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-31T23:10:38.601765+0000","last_change":"2024-03-31T23:09:11.177530+0000","last_active":"2024-03-31T23:10:38.601765+0000","last_peered":"2024-03-31T23:10:38.601765+0000","last_clean":"2024-03-31T23:10:38.601765+0000","last_became_active":"2024-03-31T23:09:11.176972+0000","last_became_peered":"2024-03-31T23:09:11.176972+0000","last_unstale":"2024-03-31T23:10:38.601765+0000","last_undegraded":"2024-03-31T23:10:38.601765+0000","last_fullsized":"2024-03-31T23:10:38.601765+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T07:57:21.094911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"93'9","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-31T23:10:38.606882+0000","last_change":"2024-03-31T23:09:11.172792+0000","last_active":"2024-03-31T23:10:38.606882+0000","last_peered":"2024-03-31T23:10:38.606882+0000","last_clean":"2024-03-31T23:10:38.606882+0000","last_became_active":"2024-03-31T23:09:11.172463+0000","last_became_peered":"2024-03-31T23:09:11.172463+0000","last_unstale":"2024-03-31T23:10:38.606882+0000","last_undegraded":"2024-03-31T23:10:38.606882+0000","last_fullsized":"2024-03-31T23:10:38.606882+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:07:28.789783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"93'12","reported_seq":39,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-31T23:10:38.582022+0000","last_change":"2024-03-31T23:09:11.174331+0000","last_active":"2024-03-31T23:10:38.582022+0000","last_peered":"2024-03-31T23:10:38.582022+0000","last_clean":"2024-03-31T23:10:38.582022+0000","last_became_active":"2024-03-31T23:09:11.174001+0000","last_became_peered":"2024-03-31T23:09:11.174001+0000","last_unstale":"2024-03-31T23:10:38.582022+0000","last_undegraded":"2024-03-31T23:10:38.582022+0000","last_fullsized":"2024-03-31T23:10:38.582022+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:26:28.005406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"93'8","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-31T23:10:38.579227+0000","last_change":"2024-03-31T23:09:11.180047+0000","last_active":"2024-03-31T23:10:38.579227+0000","last_peered":"2024-03-31T23:10:38.579227+0000","last_clean":"2024-03-31T23:10:38.579227+0000","last_became_active":"2024-03-31T23:09:11.179691+0000","last_became_peered":"2024-03-31T23:09:11.179691+0000","last_unstale":"2024-03-31T23:10:38.579227+0000","last_undegraded":"2024-03-31T23:10:38.579227+0000","last_fullsized":"2024-03-31T23:10:38.579227+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:28:05.083710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"93'7","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-31T23:10:38.608233+0000","last_change":"2024-03-31T23:09:11.177642+0000","last_active":"2024-03-31T23:10:38.608233+0000","last_peered":"2024-03-31T23:10:38.608233+0000","last_clean":"2024-03-31T23:10:38.608233+0000","last_became_active":"2024-03-31T23:09:11.177261+0000","last_became_peered":"2024-03-31T23:09:11.177261+0000","last_unstale":"2024-03-31T23:10:38.608233+0000","last_undegraded":"2024-03-31T23:10:38.608233+0000","last_fullsized":"2024-03-31T23:10:38.608233+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T07:10:43.616863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"93'4","reported_seq":31,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-31T23:10:38.611726+0000","last_change":"2024-03-31T23:09:11.173606+0000","last_active":"2024-03-31T23:10:38.611726+0000","last_peered":"2024-03-31T23:10:38.611726+0000","last_clean":"2024-03-31T23:10:38.611726+0000","last_became_active":"2024-03-31T23:09:11.173304+0000","last_became_peered":"2024-03-31T23:09:11.173304+0000","last_unstale":"2024-03-31T23:10:38.611726+0000","last_undegraded":"2024-03-31T23:10:38.611726+0000","last_fullsized":"2024-03-31T23:10:38.611726+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:53:06.112181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"93'6","reported_seq":33,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-31T23:10:38.602905+0000","last_change":"2024-03-31T23:09:11.176628+0000","last_active":"2024-03-31T23:10:38.602905+0000","last_peered":"2024-03-31T23:10:38.602905+0000","last_clean":"2024-03-31T23:10:38.602905+0000","last_became_active":"2024-03-31T23:09:11.176318+0000","last_became_peered":"2024-03-31T23:09:11.176318+0000","last_unstale":"2024-03-31T23:10:38.602905+0000","last_undegraded":"2024-03-31T23:10:38.602905+0000","last_fullsized":"2024-03-31T23:10:38.602905+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:06:40.723675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"93'15","reported_seq":42,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-31T23:10:38.600813+0000","last_change":"2024-03-31T23:09:11.169318+0000","last_active":"2024-03-31T23:10:38.600813+0000","last_peered":"2024-03-31T23:10:38.600813+0000","last_clean":"2024-03-31T23:10:38.600813+0000","last_became_active":"2024-03-31T23:09:11.169026+0000","last_became_peered":"2024-03-31T23:09:11.169026+0000","last_unstale":"2024-03-31T23:10:38.600813+0000","last_undegraded":"2024-03-31T23:10:38.600813+0000","last_fullsized":"2024-03-31T23:10:38.600813+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:02:18.562854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"93'2","reported_seq":29,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-31T23:10:38.599689+0000","last_change":"2024-03-31T23:09:11.184670+0000","last_active":"2024-03-31T23:10:38.599689+0000","last_peered":"2024-03-31T23:10:38.599689+0000","last_clean":"2024-03-31T23:10:38.599689+0000","last_became_active":"2024-03-31T23:09:11.184354+0000","last_became_peered":"2024-03-31T23:09:11.184354+0000","last_unstale":"2024-03-31T23:10:38.599689+0000","last_undegraded":"2024-03-31T23:10:38.599689+0000","last_fullsized":"2024-03-31T23:10:38.599689+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:16:14.354947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"93'6","reported_seq":33,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-31T23:10:38.591633+0000","last_change":"2024-03-31T23:09:11.180991+0000","last_active":"2024-03-31T23:10:38.591633+0000","last_peered":"2024-03-31T23:10:38.591633+0000","last_clean":"2024-03-31T23:10:38.591633+0000","last_became_active":"2024-03-31T23:09:11.180667+0000","last_became_peered":"2024-03-31T23:09:11.180667+0000","last_unstale":"2024-03-31T23:10:38.591633+0000","last_undegraded":"2024-03-31T23:10:38.591633+0000","last_fullsized":"2024-03-31T23:10:38.591633+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T06:35:33.157176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"93'8","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-31T23:10:38.617978+0000","last_change":"2024-03-31T23:09:11.182025+0000","last_active":"2024-03-31T23:10:38.617978+0000","last_peered":"2024-03-31T23:10:38.617978+0000","last_clean":"2024-03-31T23:10:38.617978+0000","last_became_active":"2024-03-31T23:09:11.179310+0000","last_became_peered":"2024-03-31T23:09:11.179310+0000","last_unstale":"2024-03-31T23:10:38.617978+0000","last_undegraded":"2024-03-31T23:10:38.617978+0000","last_fullsized":"2024-03-31T23:10:38.617978+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T10:21:40.784738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"93'10","reported_seq":43,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-31T23:10:38.615754+0000","last_change":"2024-03-31T23:10:10.485623+0000","last_active":"2024-03-31T23:10:38.615754+0000","last_peered":"2024-03-31T23:10:38.615754+0000","last_clean":"2024-03-31T23:10:38.615754+0000","last_became_active":"2024-03-31T23:10:10.485248+0000","last_became_peered":"2024-03-31T23:10:10.485248+0000","last_unstale":"2024-03-31T23:10:38.615754+0000","last_undegraded":"2024-03-31T23:10:38.615754+0000","last_fullsized":"2024-03-31T23:10:38.615754+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:59:04.668598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,6],"acting":[1,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"93'11","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-31T23:10:38.612432+0000","last_change":"2024-03-31T23:09:11.172585+0000","last_active":"2024-03-31T23:10:38.612432+0000","last_peered":"2024-03-31T23:10:38.612432+0000","last_clean":"2024-03-31T23:10:38.612432+0000","last_became_active":"2024-03-31T23:09:11.172266+0000","last_became_peered":"2024-03-31T23:09:11.172266+0000","last_unstale":"2024-03-31T23:10:38.612432+0000","last_undegraded":"2024-03-31T23:10:38.612432+0000","last_fullsized":"2024-03-31T23:10:38.612432+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:38:26.570175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"93'8","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-31T23:10:38.606543+0000","last_change":"2024-03-31T23:09:11.181434+0000","last_active":"2024-03-31T23:10:38.606543+0000","last_peered":"2024-03-31T23:10:38.606543+0000","last_clean":"2024-03-31T23:10:38.606543+0000","last_became_active":"2024-03-31T23:09:11.181140+0000","last_became_peered":"2024-03-31T23:09:11.181140+0000","last_unstale":"2024-03-31T23:10:38.606543+0000","last_undegraded":"2024-03-31T23:10:38.606543+0000","last_fullsized":"2024-03-31T23:10:38.606543+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:55:06.379696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4157440,"data_stored":4132576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":67256,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":454,"num_read_kb":297,"num_write":172,"num_write_kb":125,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":430080,"data_stored":201768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":81,"seq":347892351008,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36364,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708788,"statfs":{"total":95995035648,"available":95957798912,"internally_reserved":0,"allocated":1052672,"data_stored":514732,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":36174276},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":10,"up_from":73,"seq":313532612645,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28720,"kb_used_data":1704,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716432,"statfs":{"total":95995035648,"available":95965626368,"internally_reserved":0,"allocated":1744896,"data_stored":1165114,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":9,"up_from":68,"seq":292057776171,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28064,"kb_used_data":1048,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717088,"statfs":{"total":95995035648,"available":95966298112,"internally_reserved":0,"allocated":1073152,"data_stored":519064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":60,"seq":257698037807,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28060,"kb_used_data":1044,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717092,"statfs":{"total":95995035648,"available":95966302208,"internally_reserved":0,"allocated":1069056,"data_stored":514888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299444,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28640,"kb_used_data":1624,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716512,"statfs":{"total":95995035648,"available":95965708288,"internally_reserved":0,"allocated":1662976,"data_stored":1111470,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1588,"internal_metadata":27654604},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":47,"seq":201863462969,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28040,"kb_used_data":1024,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717112,"statfs":{"total":95995035648,"available":95966322688,"internally_reserved":0,"allocated":1048576,"data_stored":510144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574929,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28108,"kb_used_data":1092,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717044,"statfs":{"total":95995035648,"available":95966253056,"internally_reserved":0,"allocated":1118208,"data_stored":572636,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738455,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36384,"kb_used_data":1048,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708768,"statfs":{"total":95995035648,"available":95957778432,"internally_reserved":0,"allocated":1073152,"data_stored":511532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411406,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28132,"kb_used_data":1116,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717020,"statfs":{"total":95995035648,"available":95966228480,"internally_reserved":0,"allocated":1142784,"data_stored":574518,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182473,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28052,"kb_used_data":1036,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717100,"statfs":{"total":95995035648,"available":95966310400,"internally_reserved":0,"allocated":1060864,"data_stored":510338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920836,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36372,"kb_used_data":1036,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708780,"statfs":{"total":95995035648,"available":95957790720,"internally_reserved":0,"allocated":1060864,"data_stored":515326,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724607,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36968,"kb_used_data":1632,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708184,"statfs":{"total":95995035648,"available":95957180416,"internally_reserved":0,"allocated":1671168,"data_stored":1106526,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":62582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":64464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":64692,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-31T23:10:45.904 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:384f31f3d3aee7f8ba3f60846b8844c049165583 shell --fsid 6a9d8f08-efb2-11ee-b647-cb9ed24678a4 -- ceph pg dump --format=json 2024-03-31T23:10:46.153 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/6a9d8f08-efb2-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-31T23:10:46.536 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:10:46 smithi046 ceph-mon[32498]: pgmap v341: 129 pgs: 129 active+clean; 642 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 597 B/s wr, 6 op/s 2024-03-31T23:10:46.536 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:10:46 smithi046 ceph-mon[32498]: from='client.15126 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-31T23:10:46.795 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:10:46 smithi162 ceph-mon[36237]: pgmap v341: 129 pgs: 129 active+clean; 642 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 597 B/s wr, 6 op/s 2024-03-31T23:10:46.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:10:46 smithi162 ceph-mon[36237]: from='client.15126 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-31T23:10:46.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:10:46 smithi142 ceph-mon[35271]: pgmap v341: 129 pgs: 129 active+clean; 642 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 597 B/s wr, 6 op/s 2024-03-31T23:10:46.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:10:46 smithi142 ceph-mon[35271]: from='client.15126 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-31T23:10:47.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:47.906+0000 7f469255c700 1 -- 172.21.15.46:0/2582299717 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f468c102c50 msgr2=0x7f468c10f090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:10:47.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:47.906+0000 7f469255c700 1 --2- 172.21.15.46:0/2582299717 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f468c102c50 0x7f468c10f090 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f468400b0d0 tx=0x7f468400b3e0 comp rx=0 tx=0).stop 2024-03-31T23:10:47.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:47.906+0000 7f469255c700 1 -- 172.21.15.46:0/2582299717 shutdown_connections 2024-03-31T23:10:47.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:47.906+0000 7f469255c700 1 --2- 172.21.15.46:0/2582299717 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f468c102c50 0x7f468c10f090 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:47.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:47.906+0000 7f469255c700 1 --2- 172.21.15.46:0/2582299717 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f468c108270 0x7f468c108650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:47.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:47.906+0000 7f469255c700 1 --2- 172.21.15.46:0/2582299717 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f468c102290 0x7f468c102710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:47.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:47.906+0000 7f469255c700 1 -- 172.21.15.46:0/2582299717 >> 172.21.15.46:0/2582299717 conn(0x7f468c079370 msgr2=0x7f468c079780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:10:47.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:47.906+0000 7f469255c700 1 -- 172.21.15.46:0/2582299717 shutdown_connections 2024-03-31T23:10:47.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:47.906+0000 7f469255c700 1 -- 172.21.15.46:0/2582299717 wait complete. 2024-03-31T23:10:47.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:47.907+0000 7f469255c700 1 Processor -- start 2024-03-31T23:10:47.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:47.907+0000 7f469255c700 1 -- start start 2024-03-31T23:10:47.910 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:47.907+0000 7f469255c700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f468c102290 0x7f468c19e600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:10:47.911 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:47.907+0000 7f469255c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f468c102c50 0x7f468c19eb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:10:47.911 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:47.907+0000 7f469255c700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f468c108270 0x7f468c1986d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:10:47.911 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:47.907+0000 7f469255c700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f468c1123d0 con 0x7f468c102c50 2024-03-31T23:10:47.911 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:47.907+0000 7f469255c700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f468c112250 con 0x7f468c108270 2024-03-31T23:10:47.911 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:47.907+0000 7f469255c700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f468c112550 con 0x7f468c102290 2024-03-31T23:10:47.911 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:47.908+0000 7f468b7fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f468c102c50 0x7f468c19eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:10:47.911 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:47.908+0000 7f468b7fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f468c102c50 0x7f468c19eb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:52030/0 (socket says 172.21.15.46:52030) 2024-03-31T23:10:47.911 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:47.908+0000 7f468b7fe700 1 -- 172.21.15.46:0/861381798 learned_addr learned my addr 172.21.15.46:0/861381798 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:10:47.911 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:47.908+0000 7f468bfff700 1 --2- 172.21.15.46:0/861381798 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f468c102290 0x7f468c19e600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:10:47.911 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:47.908+0000 7f4690af9700 1 --2- 172.21.15.46:0/861381798 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f468c108270 0x7f468c1986d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:10:47.911 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:47.908+0000 7f468b7fe700 1 -- 172.21.15.46:0/861381798 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f468c102290 msgr2=0x7f468c19e600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:10:47.912 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:47.908+0000 7f468b7fe700 1 --2- 172.21.15.46:0/861381798 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f468c102290 0x7f468c19e600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:47.912 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:47.908+0000 7f468b7fe700 1 -- 172.21.15.46:0/861381798 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f468c108270 msgr2=0x7f468c1986d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:10:47.912 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:47.908+0000 7f468b7fe700 1 --2- 172.21.15.46:0/861381798 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f468c108270 0x7f468c1986d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:47.912 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:47.908+0000 7f468b7fe700 1 -- 172.21.15.46:0/861381798 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f468c198fb0 con 0x7f468c102c50 2024-03-31T23:10:47.912 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:47.908+0000 7f468bfff700 1 --2- 172.21.15.46:0/861381798 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f468c102290 0x7f468c19e600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:10:47.912 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:47.908+0000 7f468b7fe700 1 --2- 172.21.15.46:0/861381798 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f468c102c50 0x7f468c19eb40 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f467c00e9c0 tx=0x7f467c00ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:10:47.912 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:47.908+0000 7f46897fa700 1 -- 172.21.15.46:0/861381798 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f467c00cbd0 con 0x7f468c102c50 2024-03-31T23:10:47.912 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:47.908+0000 7f46897fa700 1 -- 172.21.15.46:0/861381798 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f467c00ee60 con 0x7f468c102c50 2024-03-31T23:10:47.912 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:47.908+0000 7f469255c700 1 -- 172.21.15.46:0/861381798 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f468c1992a0 con 0x7f468c102c50 2024-03-31T23:10:47.912 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:47.908+0000 7f46897fa700 1 -- 172.21.15.46:0/861381798 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f467c0055b0 con 0x7f468c102c50 2024-03-31T23:10:47.912 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:47.908+0000 7f469255c700 1 -- 172.21.15.46:0/861381798 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f468c1a5580 con 0x7f468c102c50 2024-03-31T23:10:47.912 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:47.909+0000 7f469255c700 1 -- 172.21.15.46:0/861381798 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f468c103920 con 0x7f468c102c50 2024-03-31T23:10:47.913 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:47.910+0000 7f46897fa700 1 -- 172.21.15.46:0/861381798 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 18) ==== 137320+0+0 (secure 0 0 0) 0x7f467c004750 con 0x7f468c102c50 2024-03-31T23:10:47.913 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:47.911+0000 7f46897fa700 1 --2- 172.21.15.46:0/861381798 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f46780a2530 0x7f46780a49f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:10:47.913 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:47.911+0000 7f468bfff700 1 --2- 172.21.15.46:0/861381798 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f46780a2530 0x7f46780a49f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:10:47.913 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:47.911+0000 7f46897fa700 1 -- 172.21.15.46:0/861381798 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7f467c0ced00 con 0x7f468c102c50 2024-03-31T23:10:47.913 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:47.911+0000 7f468bfff700 1 --2- 172.21.15.46:0/861381798 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f46780a2530 0x7f46780a49f0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f4674009fd0 tx=0x7f4674009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:10:47.916 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:47.916+0000 7f46897fa700 1 -- 172.21.15.46:0/861381798 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f467c09bb60 con 0x7f468c102c50 2024-03-31T23:10:48.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:48.119+0000 7f469255c700 1 -- 172.21.15.46:0/861381798 --> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f468c199c00 con 0x7f46780a2530 2024-03-31T23:10:48.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:48.128+0000 7f46897fa700 1 -- 172.21.15.46:0/861381798 <== mgr.14152 v2:172.21.15.46:6800/162289569 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340525 (secure 0 0 0) 0x7f468c199c00 con 0x7f46780a2530 2024-03-31T23:10:48.133 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-31T23:10:48.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:48.131+0000 7f469255c700 1 -- 172.21.15.46:0/861381798 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f46780a2530 msgr2=0x7f46780a49f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:10:48.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:48.131+0000 7f469255c700 1 --2- 172.21.15.46:0/861381798 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f46780a2530 0x7f46780a49f0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f4674009fd0 tx=0x7f4674009260 comp rx=0 tx=0).stop 2024-03-31T23:10:48.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:48.131+0000 7f469255c700 1 -- 172.21.15.46:0/861381798 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f468c102c50 msgr2=0x7f468c19eb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:10:48.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:48.131+0000 7f469255c700 1 --2- 172.21.15.46:0/861381798 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f468c102c50 0x7f468c19eb40 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f467c00e9c0 tx=0x7f467c00ecd0 comp rx=0 tx=0).stop 2024-03-31T23:10:48.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:48.132+0000 7f469255c700 1 -- 172.21.15.46:0/861381798 shutdown_connections 2024-03-31T23:10:48.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:48.132+0000 7f469255c700 1 --2- 172.21.15.46:0/861381798 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f46780a2530 0x7f46780a49f0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:48.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:48.132+0000 7f469255c700 1 --2- 172.21.15.46:0/861381798 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f468c108270 0x7f468c1986d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:48.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:48.132+0000 7f469255c700 1 --2- 172.21.15.46:0/861381798 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f468c102290 0x7f468c19e600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:48.137 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:48.132+0000 7f469255c700 1 --2- 172.21.15.46:0/861381798 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f468c102c50 0x7f468c19eb40 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:48.137 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:48.132+0000 7f469255c700 1 -- 172.21.15.46:0/861381798 >> 172.21.15.46:0/861381798 conn(0x7f468c079370 msgr2=0x7f468c100fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:10:48.137 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:48.132+0000 7f469255c700 1 -- 172.21.15.46:0/861381798 shutdown_connections 2024-03-31T23:10:48.137 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:48.132+0000 7f469255c700 1 -- 172.21.15.46:0/861381798 wait complete. 2024-03-31T23:10:48.137 INFO:teuthology.orchestra.run.smithi046.stderr:dumped all 2024-03-31T23:10:48.745 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:10:48 smithi142 ceph-mon[35271]: pgmap v342: 129 pgs: 129 active+clean; 642 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 597 B/s wr, 7 op/s 2024-03-31T23:10:48.795 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:10:48 smithi162 ceph-mon[36237]: pgmap v342: 129 pgs: 129 active+clean; 642 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 597 B/s wr, 7 op/s 2024-03-31T23:10:48.809 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:10:48 smithi046 ceph-mon[32498]: pgmap v342: 129 pgs: 129 active+clean; 642 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 597 B/s wr, 7 op/s 2024-03-31T23:10:48.840 INFO:teuthology.orchestra.run.smithi046.stdout:{"pg_ready":true,"pg_map":{"version":342,"stamp":"2024-03-31T23:10:46.601242+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":657858,"num_objects":64,"num_object_clones":0,"num_object_copies":194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":610,"num_read_kb":431,"num_write":355,"num_write_kb":3065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":789,"ondisk_log_size":789,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":371904,"kb_used_data":14432,"kb_used_omap":18,"kb_used_meta":357357,"kb_avail":1124569920,"statfs":{"total":1151940427776,"available":1151559598080,"internally_reserved":0,"allocated":14778368,"data_stored":8126288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19140,"internal_metadata":365933884},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":531,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":83,"num_read_kb":48,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003690"},"pg_stats":[{"pgid":"3.3f","version":"93'8","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-31T23:10:38.617486+0000","last_change":"2024-03-31T23:09:11.177502+0000","last_active":"2024-03-31T23:10:38.617486+0000","last_peered":"2024-03-31T23:10:38.617486+0000","last_clean":"2024-03-31T23:10:38.617486+0000","last_became_active":"2024-03-31T23:09:11.177134+0000","last_became_peered":"2024-03-31T23:09:11.177134+0000","last_unstale":"2024-03-31T23:10:38.617486+0000","last_undegraded":"2024-03-31T23:10:38.617486+0000","last_fullsized":"2024-03-31T23:10:38.617486+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T10:29:36.508229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"93'4","reported_seq":31,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-31T23:10:38.608683+0000","last_change":"2024-03-31T23:09:11.172990+0000","last_active":"2024-03-31T23:10:38.608683+0000","last_peered":"2024-03-31T23:10:38.608683+0000","last_clean":"2024-03-31T23:10:38.608683+0000","last_became_active":"2024-03-31T23:09:11.172684+0000","last_became_peered":"2024-03-31T23:09:11.172684+0000","last_unstale":"2024-03-31T23:10:38.608683+0000","last_undegraded":"2024-03-31T23:10:38.608683+0000","last_fullsized":"2024-03-31T23:10:38.608683+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T07:02:59.991757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"93'7","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-31T23:10:38.608415+0000","last_change":"2024-03-31T23:09:11.176805+0000","last_active":"2024-03-31T23:10:38.608415+0000","last_peered":"2024-03-31T23:10:38.608415+0000","last_clean":"2024-03-31T23:10:38.608415+0000","last_became_active":"2024-03-31T23:09:11.176468+0000","last_became_peered":"2024-03-31T23:09:11.176468+0000","last_unstale":"2024-03-31T23:10:38.608415+0000","last_undegraded":"2024-03-31T23:10:38.608415+0000","last_fullsized":"2024-03-31T23:10:38.608415+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T08:23:41.872558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"93'8","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-31T23:10:38.601173+0000","last_change":"2024-03-31T23:09:11.182385+0000","last_active":"2024-03-31T23:10:38.601173+0000","last_peered":"2024-03-31T23:10:38.601173+0000","last_clean":"2024-03-31T23:10:38.601173+0000","last_became_active":"2024-03-31T23:09:11.181908+0000","last_became_peered":"2024-03-31T23:09:11.181908+0000","last_unstale":"2024-03-31T23:10:38.601173+0000","last_undegraded":"2024-03-31T23:10:38.601173+0000","last_fullsized":"2024-03-31T23:10:38.601173+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:52:24.066360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"93'4","reported_seq":31,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-31T23:10:38.614611+0000","last_change":"2024-03-31T23:09:11.181377+0000","last_active":"2024-03-31T23:10:38.614611+0000","last_peered":"2024-03-31T23:10:38.614611+0000","last_clean":"2024-03-31T23:10:38.614611+0000","last_became_active":"2024-03-31T23:09:11.181064+0000","last_became_peered":"2024-03-31T23:09:11.181064+0000","last_unstale":"2024-03-31T23:10:38.614611+0000","last_undegraded":"2024-03-31T23:10:38.614611+0000","last_fullsized":"2024-03-31T23:10:38.614611+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:28:40.766541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"93'6","reported_seq":33,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-31T23:10:38.606476+0000","last_change":"2024-03-31T23:09:11.176087+0000","last_active":"2024-03-31T23:10:38.606476+0000","last_peered":"2024-03-31T23:10:38.606476+0000","last_clean":"2024-03-31T23:10:38.606476+0000","last_became_active":"2024-03-31T23:09:11.175569+0000","last_became_peered":"2024-03-31T23:09:11.175569+0000","last_unstale":"2024-03-31T23:10:38.606476+0000","last_undegraded":"2024-03-31T23:10:38.606476+0000","last_fullsized":"2024-03-31T23:10:38.606476+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T06:33:19.189909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"93'5","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-31T23:10:38.609004+0000","last_change":"2024-03-31T23:09:11.170289+0000","last_active":"2024-03-31T23:10:38.609004+0000","last_peered":"2024-03-31T23:10:38.609004+0000","last_clean":"2024-03-31T23:10:38.609004+0000","last_became_active":"2024-03-31T23:09:11.169959+0000","last_became_peered":"2024-03-31T23:09:11.169959+0000","last_unstale":"2024-03-31T23:10:38.609004+0000","last_undegraded":"2024-03-31T23:10:38.609004+0000","last_fullsized":"2024-03-31T23:10:38.609004+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:10:44.614767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"93'9","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-31T23:10:38.619815+0000","last_change":"2024-03-31T23:09:11.182447+0000","last_active":"2024-03-31T23:10:38.619815+0000","last_peered":"2024-03-31T23:10:38.619815+0000","last_clean":"2024-03-31T23:10:38.619815+0000","last_became_active":"2024-03-31T23:09:11.181629+0000","last_became_peered":"2024-03-31T23:09:11.181629+0000","last_unstale":"2024-03-31T23:10:38.619815+0000","last_undegraded":"2024-03-31T23:10:38.619815+0000","last_fullsized":"2024-03-31T23:10:38.619815+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:18:26.269451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"93'11","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-31T23:10:38.615507+0000","last_change":"2024-03-31T23:09:11.176853+0000","last_active":"2024-03-31T23:10:38.615507+0000","last_peered":"2024-03-31T23:10:38.615507+0000","last_clean":"2024-03-31T23:10:38.615507+0000","last_became_active":"2024-03-31T23:09:11.176413+0000","last_became_peered":"2024-03-31T23:09:11.176413+0000","last_unstale":"2024-03-31T23:10:38.615507+0000","last_undegraded":"2024-03-31T23:10:38.615507+0000","last_fullsized":"2024-03-31T23:10:38.615507+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:23:40.486947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"93'16","reported_seq":49,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-31T23:10:38.604985+0000","last_change":"2024-03-31T23:10:10.543998+0000","last_active":"2024-03-31T23:10:38.604985+0000","last_peered":"2024-03-31T23:10:38.604985+0000","last_clean":"2024-03-31T23:10:38.604985+0000","last_became_active":"2024-03-31T23:10:10.543615+0000","last_became_peered":"2024-03-31T23:10:10.543615+0000","last_unstale":"2024-03-31T23:10:38.604985+0000","last_undegraded":"2024-03-31T23:10:38.604985+0000","last_fullsized":"2024-03-31T23:10:38.604985+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:28:32.883732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,9],"acting":[10,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"93'6","reported_seq":33,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-31T23:10:38.613988+0000","last_change":"2024-03-31T23:09:11.179536+0000","last_active":"2024-03-31T23:10:38.613988+0000","last_peered":"2024-03-31T23:10:38.613988+0000","last_clean":"2024-03-31T23:10:38.613988+0000","last_became_active":"2024-03-31T23:09:11.178521+0000","last_became_peered":"2024-03-31T23:09:11.178521+0000","last_unstale":"2024-03-31T23:10:38.613988+0000","last_undegraded":"2024-03-31T23:10:38.613988+0000","last_fullsized":"2024-03-31T23:10:38.613988+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T09:29:23.128832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"93'7","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-31T23:10:38.609167+0000","last_change":"2024-03-31T23:09:11.182548+0000","last_active":"2024-03-31T23:10:38.609167+0000","last_peered":"2024-03-31T23:10:38.609167+0000","last_clean":"2024-03-31T23:10:38.609167+0000","last_became_active":"2024-03-31T23:09:11.181966+0000","last_became_peered":"2024-03-31T23:09:11.181966+0000","last_unstale":"2024-03-31T23:10:38.609167+0000","last_undegraded":"2024-03-31T23:10:38.609167+0000","last_fullsized":"2024-03-31T23:10:38.609167+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:43:26.061594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"93'10","reported_seq":43,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-31T23:10:38.610143+0000","last_change":"2024-03-31T23:10:10.544976+0000","last_active":"2024-03-31T23:10:38.610143+0000","last_peered":"2024-03-31T23:10:38.610143+0000","last_clean":"2024-03-31T23:10:38.610143+0000","last_became_active":"2024-03-31T23:10:10.544433+0000","last_became_peered":"2024-03-31T23:10:10.544433+0000","last_unstale":"2024-03-31T23:10:38.610143+0000","last_undegraded":"2024-03-31T23:10:38.610143+0000","last_fullsized":"2024-03-31T23:10:38.610143+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:45:54.606274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,9],"acting":[6,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"93'7","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-31T23:10:38.598000+0000","last_change":"2024-03-31T23:09:11.168784+0000","last_active":"2024-03-31T23:10:38.598000+0000","last_peered":"2024-03-31T23:10:38.598000+0000","last_clean":"2024-03-31T23:10:38.598000+0000","last_became_active":"2024-03-31T23:09:11.168442+0000","last_became_peered":"2024-03-31T23:09:11.168442+0000","last_unstale":"2024-03-31T23:10:38.598000+0000","last_undegraded":"2024-03-31T23:10:38.598000+0000","last_fullsized":"2024-03-31T23:10:38.598000+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T09:37:43.568122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"93'7","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-31T23:10:38.595244+0000","last_change":"2024-03-31T23:09:11.176017+0000","last_active":"2024-03-31T23:10:38.595244+0000","last_peered":"2024-03-31T23:10:38.595244+0000","last_clean":"2024-03-31T23:10:38.595244+0000","last_became_active":"2024-03-31T23:09:11.175690+0000","last_became_peered":"2024-03-31T23:09:11.175690+0000","last_unstale":"2024-03-31T23:10:38.595244+0000","last_undegraded":"2024-03-31T23:10:38.595244+0000","last_fullsized":"2024-03-31T23:10:38.595244+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:14:13.259895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"93'9","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-31T23:10:38.613070+0000","last_change":"2024-03-31T23:09:11.171926+0000","last_active":"2024-03-31T23:10:38.613070+0000","last_peered":"2024-03-31T23:10:38.613070+0000","last_clean":"2024-03-31T23:10:38.613070+0000","last_became_active":"2024-03-31T23:09:11.171625+0000","last_became_peered":"2024-03-31T23:09:11.171625+0000","last_unstale":"2024-03-31T23:10:38.613070+0000","last_undegraded":"2024-03-31T23:10:38.613070+0000","last_fullsized":"2024-03-31T23:10:38.613070+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T06:20:20.433802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"93'8","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-31T23:10:38.617506+0000","last_change":"2024-03-31T23:09:11.183805+0000","last_active":"2024-03-31T23:10:38.617506+0000","last_peered":"2024-03-31T23:10:38.617506+0000","last_clean":"2024-03-31T23:10:38.617506+0000","last_became_active":"2024-03-31T23:09:11.183464+0000","last_became_peered":"2024-03-31T23:09:11.183464+0000","last_unstale":"2024-03-31T23:10:38.617506+0000","last_undegraded":"2024-03-31T23:10:38.617506+0000","last_fullsized":"2024-03-31T23:10:38.617506+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T06:02:33.286563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"93'10","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-31T23:10:38.614241+0000","last_change":"2024-03-31T23:09:11.170095+0000","last_active":"2024-03-31T23:10:38.614241+0000","last_peered":"2024-03-31T23:10:38.614241+0000","last_clean":"2024-03-31T23:10:38.614241+0000","last_became_active":"2024-03-31T23:09:11.169683+0000","last_became_peered":"2024-03-31T23:09:11.169683+0000","last_unstale":"2024-03-31T23:10:38.614241+0000","last_undegraded":"2024-03-31T23:10:38.614241+0000","last_fullsized":"2024-03-31T23:10:38.614241+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T08:19:47.101431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"93'7","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-31T23:10:38.578795+0000","last_change":"2024-03-31T23:09:11.181347+0000","last_active":"2024-03-31T23:10:38.578795+0000","last_peered":"2024-03-31T23:10:38.578795+0000","last_clean":"2024-03-31T23:10:38.578795+0000","last_became_active":"2024-03-31T23:09:11.180510+0000","last_became_peered":"2024-03-31T23:09:11.180510+0000","last_unstale":"2024-03-31T23:10:38.578795+0000","last_undegraded":"2024-03-31T23:10:38.578795+0000","last_fullsized":"2024-03-31T23:10:38.578795+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:25:51.083446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":21,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-31T23:10:10.541341+0000","last_change":"2024-03-31T23:09:15.205059+0000","last_active":"2024-03-31T23:10:10.541341+0000","last_peered":"2024-03-31T23:10:10.541341+0000","last_clean":"2024-03-31T23:10:10.541341+0000","last_became_active":"2024-03-31T23:09:15.204612+0000","last_became_peered":"2024-03-31T23:09:15.204612+0000","last_unstale":"2024-03-31T23:10:10.541341+0000","last_undegraded":"2024-03-31T23:10:10.541341+0000","last_fullsized":"2024-03-31T23:10:10.541341+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_clean_scrub_stamp":"2024-03-31T23:09:14.171251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:53:33.998026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"91'9","reported_seq":39,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-31T23:10:10.543485+0000","last_change":"2024-03-31T23:09:10.594147+0000","last_active":"2024-03-31T23:10:10.543485+0000","last_peered":"2024-03-31T23:10:10.543485+0000","last_clean":"2024-03-31T23:10:10.543485+0000","last_became_active":"2024-03-31T23:09:10.592047+0000","last_became_peered":"2024-03-31T23:09:10.592047+0000","last_unstale":"2024-03-31T23:10:10.543485+0000","last_undegraded":"2024-03-31T23:10:10.543485+0000","last_fullsized":"2024-03-31T23:10:10.543485+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_clean_scrub_stamp":"2024-03-31T23:09:09.564263+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:04:34.548282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"93'5","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-31T23:10:38.602438+0000","last_change":"2024-03-31T23:09:11.182292+0000","last_active":"2024-03-31T23:10:38.602438+0000","last_peered":"2024-03-31T23:10:38.602438+0000","last_clean":"2024-03-31T23:10:38.602438+0000","last_became_active":"2024-03-31T23:09:11.181885+0000","last_became_peered":"2024-03-31T23:09:11.181885+0000","last_unstale":"2024-03-31T23:10:38.602438+0000","last_undegraded":"2024-03-31T23:10:38.602438+0000","last_fullsized":"2024-03-31T23:10:38.602438+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T10:38:17.550625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":28,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-31T23:10:10.543802+0000","last_change":"2024-03-31T23:10:10.543802+0000","last_active":"2024-03-31T23:10:10.543802+0000","last_peered":"2024-03-31T23:10:10.543802+0000","last_clean":"2024-03-31T23:10:10.543802+0000","last_became_active":"2024-03-31T23:10:10.543245+0000","last_became_peered":"2024-03-31T23:10:10.543245+0000","last_unstale":"2024-03-31T23:10:10.543802+0000","last_undegraded":"2024-03-31T23:10:10.543802+0000","last_fullsized":"2024-03-31T23:10:10.543802+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_clean_scrub_stamp":"2024-03-31T23:09:14.171251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T06:00:11.707183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,8,3],"acting":[4,6,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"91'2","reported_seq":31,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-31T23:10:10.542591+0000","last_change":"2024-03-31T23:09:10.598375+0000","last_active":"2024-03-31T23:10:10.542591+0000","last_peered":"2024-03-31T23:10:10.542591+0000","last_clean":"2024-03-31T23:10:10.542591+0000","last_became_active":"2024-03-31T23:09:10.598045+0000","last_became_peered":"2024-03-31T23:09:10.598045+0000","last_unstale":"2024-03-31T23:10:10.542591+0000","last_undegraded":"2024-03-31T23:10:10.542591+0000","last_fullsized":"2024-03-31T23:10:10.542591+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_clean_scrub_stamp":"2024-03-31T23:09:09.564263+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:31:27.113317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"93'6","reported_seq":33,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-31T23:10:38.612943+0000","last_change":"2024-03-31T23:09:11.183458+0000","last_active":"2024-03-31T23:10:38.612943+0000","last_peered":"2024-03-31T23:10:38.612943+0000","last_clean":"2024-03-31T23:10:38.612943+0000","last_became_active":"2024-03-31T23:09:11.183100+0000","last_became_peered":"2024-03-31T23:09:11.183100+0000","last_unstale":"2024-03-31T23:10:38.612943+0000","last_undegraded":"2024-03-31T23:10:38.612943+0000","last_fullsized":"2024-03-31T23:10:38.612943+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T10:55:35.512065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":21,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-31T23:10:10.479074+0000","last_change":"2024-03-31T23:09:15.208103+0000","last_active":"2024-03-31T23:10:10.479074+0000","last_peered":"2024-03-31T23:10:10.479074+0000","last_clean":"2024-03-31T23:10:10.479074+0000","last_became_active":"2024-03-31T23:09:15.207723+0000","last_became_peered":"2024-03-31T23:09:15.207723+0000","last_unstale":"2024-03-31T23:10:10.479074+0000","last_undegraded":"2024-03-31T23:10:10.479074+0000","last_fullsized":"2024-03-31T23:10:10.479074+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_clean_scrub_stamp":"2024-03-31T23:09:14.171251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T07:55:01.303693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"91'4","reported_seq":33,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-31T23:10:10.480437+0000","last_change":"2024-03-31T23:09:10.597471+0000","last_active":"2024-03-31T23:10:10.480437+0000","last_peered":"2024-03-31T23:10:10.480437+0000","last_clean":"2024-03-31T23:10:10.480437+0000","last_became_active":"2024-03-31T23:09:10.597165+0000","last_became_peered":"2024-03-31T23:09:10.597165+0000","last_unstale":"2024-03-31T23:10:10.480437+0000","last_undegraded":"2024-03-31T23:10:10.480437+0000","last_fullsized":"2024-03-31T23:10:10.480437+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_clean_scrub_stamp":"2024-03-31T23:09:09.564263+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:10:26.826525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"93'9","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-31T23:10:38.617272+0000","last_change":"2024-03-31T23:09:11.170650+0000","last_active":"2024-03-31T23:10:38.617272+0000","last_peered":"2024-03-31T23:10:38.617272+0000","last_clean":"2024-03-31T23:10:38.617272+0000","last_became_active":"2024-03-31T23:09:11.170291+0000","last_became_peered":"2024-03-31T23:09:11.170291+0000","last_unstale":"2024-03-31T23:10:38.617272+0000","last_undegraded":"2024-03-31T23:10:38.617272+0000","last_fullsized":"2024-03-31T23:10:38.617272+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:29:54.904377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":21,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-31T23:10:10.478810+0000","last_change":"2024-03-31T23:09:15.204857+0000","last_active":"2024-03-31T23:10:10.478810+0000","last_peered":"2024-03-31T23:10:10.478810+0000","last_clean":"2024-03-31T23:10:10.478810+0000","last_became_active":"2024-03-31T23:09:15.204484+0000","last_became_peered":"2024-03-31T23:09:15.204484+0000","last_unstale":"2024-03-31T23:10:10.478810+0000","last_undegraded":"2024-03-31T23:10:10.478810+0000","last_fullsized":"2024-03-31T23:10:10.478810+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_clean_scrub_stamp":"2024-03-31T23:09:14.171251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T09:54:24.646918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"85'2","reported_seq":31,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-31T23:10:10.480174+0000","last_change":"2024-03-31T23:09:10.589922+0000","last_active":"2024-03-31T23:10:10.480174+0000","last_peered":"2024-03-31T23:10:10.480174+0000","last_clean":"2024-03-31T23:10:10.480174+0000","last_became_active":"2024-03-31T23:09:10.589569+0000","last_became_peered":"2024-03-31T23:09:10.589569+0000","last_unstale":"2024-03-31T23:10:10.480174+0000","last_undegraded":"2024-03-31T23:10:10.480174+0000","last_fullsized":"2024-03-31T23:10:10.480174+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_clean_scrub_stamp":"2024-03-31T23:09:09.564263+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:57:05.347344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"93'15","reported_seq":48,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-31T23:10:38.603772+0000","last_change":"2024-03-31T23:10:10.485015+0000","last_active":"2024-03-31T23:10:38.603772+0000","last_peered":"2024-03-31T23:10:38.603772+0000","last_clean":"2024-03-31T23:10:38.603772+0000","last_became_active":"2024-03-31T23:10:10.484663+0000","last_became_peered":"2024-03-31T23:10:10.484663+0000","last_unstale":"2024-03-31T23:10:38.603772+0000","last_undegraded":"2024-03-31T23:10:38.603772+0000","last_fullsized":"2024-03-31T23:10:38.603772+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T06:18:50.834711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,1],"acting":[5,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":21,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-31T23:10:10.542109+0000","last_change":"2024-03-31T23:09:15.193096+0000","last_active":"2024-03-31T23:10:10.542109+0000","last_peered":"2024-03-31T23:10:10.542109+0000","last_clean":"2024-03-31T23:10:10.542109+0000","last_became_active":"2024-03-31T23:09:15.192624+0000","last_became_peered":"2024-03-31T23:09:15.192624+0000","last_unstale":"2024-03-31T23:10:10.542109+0000","last_undegraded":"2024-03-31T23:10:10.542109+0000","last_fullsized":"2024-03-31T23:10:10.542109+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_clean_scrub_stamp":"2024-03-31T23:09:14.171251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:22:24.921346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"93'7","reported_seq":112,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-31T23:10:40.462768+0000","last_change":"2024-03-31T23:09:10.587209+0000","last_active":"2024-03-31T23:10:40.462768+0000","last_peered":"2024-03-31T23:10:40.462768+0000","last_clean":"2024-03-31T23:10:40.462768+0000","last_became_active":"2024-03-31T23:09:10.586486+0000","last_became_peered":"2024-03-31T23:09:10.586486+0000","last_unstale":"2024-03-31T23:10:40.462768+0000","last_undegraded":"2024-03-31T23:10:40.462768+0000","last_fullsized":"2024-03-31T23:10:40.462768+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_clean_scrub_stamp":"2024-03-31T23:09:09.564263+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:27:56.046414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1154,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":4,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"93'7","reported_seq":40,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-31T23:10:38.610724+0000","last_change":"2024-03-31T23:10:10.543713+0000","last_active":"2024-03-31T23:10:38.610724+0000","last_peered":"2024-03-31T23:10:38.610724+0000","last_clean":"2024-03-31T23:10:38.610724+0000","last_became_active":"2024-03-31T23:10:10.543324+0000","last_became_peered":"2024-03-31T23:10:10.543324+0000","last_unstale":"2024-03-31T23:10:38.610724+0000","last_undegraded":"2024-03-31T23:10:38.610724+0000","last_fullsized":"2024-03-31T23:10:38.610724+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:41:21.974325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,6],"acting":[8,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":21,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-31T23:10:10.541840+0000","last_change":"2024-03-31T23:09:15.194717+0000","last_active":"2024-03-31T23:10:10.541840+0000","last_peered":"2024-03-31T23:10:10.541840+0000","last_clean":"2024-03-31T23:10:10.541840+0000","last_became_active":"2024-03-31T23:09:15.194232+0000","last_became_peered":"2024-03-31T23:09:15.194232+0000","last_unstale":"2024-03-31T23:10:10.541840+0000","last_undegraded":"2024-03-31T23:10:10.541840+0000","last_fullsized":"2024-03-31T23:10:10.541840+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_clean_scrub_stamp":"2024-03-31T23:09:14.171251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:21:00.445232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"91'3","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-31T23:10:10.482530+0000","last_change":"2024-03-31T23:09:10.586346+0000","last_active":"2024-03-31T23:10:10.482530+0000","last_peered":"2024-03-31T23:10:10.482530+0000","last_clean":"2024-03-31T23:10:10.482530+0000","last_became_active":"2024-03-31T23:09:10.586043+0000","last_became_peered":"2024-03-31T23:09:10.586043+0000","last_unstale":"2024-03-31T23:10:10.482530+0000","last_undegraded":"2024-03-31T23:10:10.482530+0000","last_fullsized":"2024-03-31T23:10:10.482530+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_clean_scrub_stamp":"2024-03-31T23:09:09.564263+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:17:27.387786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"93'8","reported_seq":41,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-31T23:10:38.591432+0000","last_change":"2024-03-31T23:10:10.543353+0000","last_active":"2024-03-31T23:10:38.591432+0000","last_peered":"2024-03-31T23:10:38.591432+0000","last_clean":"2024-03-31T23:10:38.591432+0000","last_became_active":"2024-03-31T23:10:10.542961+0000","last_became_peered":"2024-03-31T23:10:10.542961+0000","last_unstale":"2024-03-31T23:10:38.591432+0000","last_undegraded":"2024-03-31T23:10:38.591432+0000","last_fullsized":"2024-03-31T23:10:38.591432+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:28:52.950551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3],"acting":[5,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":21,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-31T23:10:10.541405+0000","last_change":"2024-03-31T23:09:15.205855+0000","last_active":"2024-03-31T23:10:10.541405+0000","last_peered":"2024-03-31T23:10:10.541405+0000","last_clean":"2024-03-31T23:10:10.541405+0000","last_became_active":"2024-03-31T23:09:15.205359+0000","last_became_peered":"2024-03-31T23:09:15.205359+0000","last_unstale":"2024-03-31T23:10:10.541405+0000","last_undegraded":"2024-03-31T23:10:10.541405+0000","last_fullsized":"2024-03-31T23:10:10.541405+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_clean_scrub_stamp":"2024-03-31T23:09:14.171251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T09:39:32.461011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"91'2","reported_seq":31,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-31T23:10:10.476727+0000","last_change":"2024-03-31T23:09:10.583274+0000","last_active":"2024-03-31T23:10:10.476727+0000","last_peered":"2024-03-31T23:10:10.476727+0000","last_clean":"2024-03-31T23:10:10.476727+0000","last_became_active":"2024-03-31T23:09:10.582910+0000","last_became_peered":"2024-03-31T23:09:10.582910+0000","last_unstale":"2024-03-31T23:10:10.476727+0000","last_undegraded":"2024-03-31T23:10:10.476727+0000","last_fullsized":"2024-03-31T23:10:10.476727+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_clean_scrub_stamp":"2024-03-31T23:09:09.564263+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:16:32.251169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"93'8","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-31T23:10:38.619376+0000","last_change":"2024-03-31T23:09:11.178619+0000","last_active":"2024-03-31T23:10:38.619376+0000","last_peered":"2024-03-31T23:10:38.619376+0000","last_clean":"2024-03-31T23:10:38.619376+0000","last_became_active":"2024-03-31T23:09:11.178239+0000","last_became_peered":"2024-03-31T23:09:11.178239+0000","last_unstale":"2024-03-31T23:10:38.619376+0000","last_undegraded":"2024-03-31T23:10:38.619376+0000","last_fullsized":"2024-03-31T23:10:38.619376+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:28:58.863199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":21,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-31T23:10:10.480544+0000","last_change":"2024-03-31T23:09:15.205864+0000","last_active":"2024-03-31T23:10:10.480544+0000","last_peered":"2024-03-31T23:10:10.480544+0000","last_clean":"2024-03-31T23:10:10.480544+0000","last_became_active":"2024-03-31T23:09:15.204976+0000","last_became_peered":"2024-03-31T23:09:15.204976+0000","last_unstale":"2024-03-31T23:10:10.480544+0000","last_undegraded":"2024-03-31T23:10:10.480544+0000","last_fullsized":"2024-03-31T23:10:10.480544+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_clean_scrub_stamp":"2024-03-31T23:09:14.171251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:39:08.537827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"91'1","reported_seq":30,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-31T23:10:10.477775+0000","last_change":"2024-03-31T23:09:10.588407+0000","last_active":"2024-03-31T23:10:10.477775+0000","last_peered":"2024-03-31T23:10:10.477775+0000","last_clean":"2024-03-31T23:10:10.477775+0000","last_became_active":"2024-03-31T23:09:10.588109+0000","last_became_peered":"2024-03-31T23:09:10.588109+0000","last_unstale":"2024-03-31T23:10:10.477775+0000","last_undegraded":"2024-03-31T23:10:10.477775+0000","last_fullsized":"2024-03-31T23:10:10.477775+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_clean_scrub_stamp":"2024-03-31T23:09:09.564263+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:01:25.858834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"93'11","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-31T23:10:38.616479+0000","last_change":"2024-03-31T23:09:11.180452+0000","last_active":"2024-03-31T23:10:38.616479+0000","last_peered":"2024-03-31T23:10:38.616479+0000","last_clean":"2024-03-31T23:10:38.616479+0000","last_became_active":"2024-03-31T23:09:11.180117+0000","last_became_peered":"2024-03-31T23:09:11.180117+0000","last_unstale":"2024-03-31T23:10:38.616479+0000","last_undegraded":"2024-03-31T23:10:38.616479+0000","last_fullsized":"2024-03-31T23:10:38.616479+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:04:58.288981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":21,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-31T23:10:10.541506+0000","last_change":"2024-03-31T23:09:15.199669+0000","last_active":"2024-03-31T23:10:10.541506+0000","last_peered":"2024-03-31T23:10:10.541506+0000","last_clean":"2024-03-31T23:10:10.541506+0000","last_became_active":"2024-03-31T23:09:15.199262+0000","last_became_peered":"2024-03-31T23:09:15.199262+0000","last_unstale":"2024-03-31T23:10:10.541506+0000","last_undegraded":"2024-03-31T23:10:10.541506+0000","last_fullsized":"2024-03-31T23:10:10.541506+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_clean_scrub_stamp":"2024-03-31T23:09:14.171251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:44:02.975000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"93'4","reported_seq":108,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-31T23:10:43.465714+0000","last_change":"2024-03-31T23:09:10.583476+0000","last_active":"2024-03-31T23:10:43.465714+0000","last_peered":"2024-03-31T23:10:43.465714+0000","last_clean":"2024-03-31T23:10:43.465714+0000","last_became_active":"2024-03-31T23:09:10.582789+0000","last_became_peered":"2024-03-31T23:09:10.582789+0000","last_unstale":"2024-03-31T23:10:43.465714+0000","last_undegraded":"2024-03-31T23:10:43.465714+0000","last_fullsized":"2024-03-31T23:10:43.465714+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_clean_scrub_stamp":"2024-03-31T23:09:09.564263+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T10:52:35.478325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":75,"num_read_kb":75,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"93'10","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-31T23:10:38.613555+0000","last_change":"2024-03-31T23:09:11.171074+0000","last_active":"2024-03-31T23:10:38.613555+0000","last_peered":"2024-03-31T23:10:38.613555+0000","last_clean":"2024-03-31T23:10:38.613555+0000","last_became_active":"2024-03-31T23:09:11.169572+0000","last_became_peered":"2024-03-31T23:09:11.169572+0000","last_unstale":"2024-03-31T23:10:38.613555+0000","last_undegraded":"2024-03-31T23:10:38.613555+0000","last_fullsized":"2024-03-31T23:10:38.613555+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T07:15:34.833388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":21,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-31T23:10:10.542189+0000","last_change":"2024-03-31T23:09:15.203835+0000","last_active":"2024-03-31T23:10:10.542189+0000","last_peered":"2024-03-31T23:10:10.542189+0000","last_clean":"2024-03-31T23:10:10.542189+0000","last_became_active":"2024-03-31T23:09:15.200861+0000","last_became_peered":"2024-03-31T23:09:15.200861+0000","last_unstale":"2024-03-31T23:10:10.542189+0000","last_undegraded":"2024-03-31T23:10:10.542189+0000","last_fullsized":"2024-03-31T23:10:10.542189+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_clean_scrub_stamp":"2024-03-31T23:09:14.171251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:56:19.664708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"91'2","reported_seq":31,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-31T23:10:10.480844+0000","last_change":"2024-03-31T23:09:10.592601+0000","last_active":"2024-03-31T23:10:10.480844+0000","last_peered":"2024-03-31T23:10:10.480844+0000","last_clean":"2024-03-31T23:10:10.480844+0000","last_became_active":"2024-03-31T23:09:10.592345+0000","last_became_peered":"2024-03-31T23:09:10.592345+0000","last_unstale":"2024-03-31T23:10:10.480844+0000","last_undegraded":"2024-03-31T23:10:10.480844+0000","last_fullsized":"2024-03-31T23:10:10.480844+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_clean_scrub_stamp":"2024-03-31T23:09:09.564263+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:10:21.195544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"93'9","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-31T23:10:38.595752+0000","last_change":"2024-03-31T23:09:11.175305+0000","last_active":"2024-03-31T23:10:38.595752+0000","last_peered":"2024-03-31T23:10:38.595752+0000","last_clean":"2024-03-31T23:10:38.595752+0000","last_became_active":"2024-03-31T23:09:11.174970+0000","last_became_peered":"2024-03-31T23:09:11.174970+0000","last_unstale":"2024-03-31T23:10:38.595752+0000","last_undegraded":"2024-03-31T23:10:38.595752+0000","last_fullsized":"2024-03-31T23:10:38.595752+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:29:39.718127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":21,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-31T23:10:10.479177+0000","last_change":"2024-03-31T23:09:15.200365+0000","last_active":"2024-03-31T23:10:10.479177+0000","last_peered":"2024-03-31T23:10:10.479177+0000","last_clean":"2024-03-31T23:10:10.479177+0000","last_became_active":"2024-03-31T23:09:15.199830+0000","last_became_peered":"2024-03-31T23:09:15.199830+0000","last_unstale":"2024-03-31T23:10:10.479177+0000","last_undegraded":"2024-03-31T23:10:10.479177+0000","last_fullsized":"2024-03-31T23:10:10.479177+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_clean_scrub_stamp":"2024-03-31T23:09:14.171251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:37:30.048859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"91'4","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-31T23:10:10.478220+0000","last_change":"2024-03-31T23:09:10.582654+0000","last_active":"2024-03-31T23:10:10.478220+0000","last_peered":"2024-03-31T23:10:10.478220+0000","last_clean":"2024-03-31T23:10:10.478220+0000","last_became_active":"2024-03-31T23:09:10.582314+0000","last_became_peered":"2024-03-31T23:09:10.582314+0000","last_unstale":"2024-03-31T23:10:10.478220+0000","last_undegraded":"2024-03-31T23:10:10.478220+0000","last_fullsized":"2024-03-31T23:10:10.478220+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_clean_scrub_stamp":"2024-03-31T23:09:09.564263+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:51:34.236575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"93'11","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-31T23:10:38.616002+0000","last_change":"2024-03-31T23:09:11.172811+0000","last_active":"2024-03-31T23:10:38.616002+0000","last_peered":"2024-03-31T23:10:38.616002+0000","last_clean":"2024-03-31T23:10:38.616002+0000","last_became_active":"2024-03-31T23:09:11.172478+0000","last_became_peered":"2024-03-31T23:09:11.172478+0000","last_unstale":"2024-03-31T23:10:38.616002+0000","last_undegraded":"2024-03-31T23:10:38.616002+0000","last_fullsized":"2024-03-31T23:10:38.616002+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T10:00:09.825254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":21,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-31T23:10:10.542583+0000","last_change":"2024-03-31T23:09:15.205494+0000","last_active":"2024-03-31T23:10:10.542583+0000","last_peered":"2024-03-31T23:10:10.542583+0000","last_clean":"2024-03-31T23:10:10.542583+0000","last_became_active":"2024-03-31T23:09:15.205101+0000","last_became_peered":"2024-03-31T23:09:15.205101+0000","last_unstale":"2024-03-31T23:10:10.542583+0000","last_undegraded":"2024-03-31T23:10:10.542583+0000","last_fullsized":"2024-03-31T23:10:10.542583+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_clean_scrub_stamp":"2024-03-31T23:09:14.171251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:52:47.472296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"93'37","reported_seq":225,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-31T23:10:44.466872+0000","last_change":"2024-03-31T23:09:10.585611+0000","last_active":"2024-03-31T23:10:44.466872+0000","last_peered":"2024-03-31T23:10:44.466872+0000","last_clean":"2024-03-31T23:10:44.466872+0000","last_became_active":"2024-03-31T23:09:10.585287+0000","last_became_peered":"2024-03-31T23:09:10.585287+0000","last_unstale":"2024-03-31T23:10:44.466872+0000","last_undegraded":"2024-03-31T23:10:44.466872+0000","last_fullsized":"2024-03-31T23:10:44.466872+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_clean_scrub_stamp":"2024-03-31T23:09:09.564263+0000","objects_scrubbed":0,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:47:44.063831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":62582,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":159,"num_read_kb":141,"num_write":38,"num_write_kb":81,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"93'6","reported_seq":33,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-31T23:10:38.593482+0000","last_change":"2024-03-31T23:09:11.175812+0000","last_active":"2024-03-31T23:10:38.593482+0000","last_peered":"2024-03-31T23:10:38.593482+0000","last_clean":"2024-03-31T23:10:38.593482+0000","last_became_active":"2024-03-31T23:09:11.174972+0000","last_became_peered":"2024-03-31T23:09:11.174972+0000","last_unstale":"2024-03-31T23:10:38.593482+0000","last_undegraded":"2024-03-31T23:10:38.593482+0000","last_fullsized":"2024-03-31T23:10:38.593482+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:17:18.870340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":21,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-31T23:10:10.478914+0000","last_change":"2024-03-31T23:09:15.205018+0000","last_active":"2024-03-31T23:10:10.478914+0000","last_peered":"2024-03-31T23:10:10.478914+0000","last_clean":"2024-03-31T23:10:10.478914+0000","last_became_active":"2024-03-31T23:09:15.204510+0000","last_became_peered":"2024-03-31T23:09:15.204510+0000","last_unstale":"2024-03-31T23:10:10.478914+0000","last_undegraded":"2024-03-31T23:10:10.478914+0000","last_fullsized":"2024-03-31T23:10:10.478914+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_clean_scrub_stamp":"2024-03-31T23:09:14.171251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T07:48:59.819937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"91'6","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-31T23:10:10.541447+0000","last_change":"2024-03-31T23:09:10.590233+0000","last_active":"2024-03-31T23:10:10.541447+0000","last_peered":"2024-03-31T23:10:10.541447+0000","last_clean":"2024-03-31T23:10:10.541447+0000","last_became_active":"2024-03-31T23:09:10.589853+0000","last_became_peered":"2024-03-31T23:09:10.589853+0000","last_unstale":"2024-03-31T23:10:10.541447+0000","last_undegraded":"2024-03-31T23:10:10.541447+0000","last_fullsized":"2024-03-31T23:10:10.541447+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_clean_scrub_stamp":"2024-03-31T23:09:09.564263+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:10:54.980394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"93'9","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-31T23:10:38.620015+0000","last_change":"2024-03-31T23:09:11.177871+0000","last_active":"2024-03-31T23:10:38.620015+0000","last_peered":"2024-03-31T23:10:38.620015+0000","last_clean":"2024-03-31T23:10:38.620015+0000","last_became_active":"2024-03-31T23:09:11.176504+0000","last_became_peered":"2024-03-31T23:09:11.176504+0000","last_unstale":"2024-03-31T23:10:38.620015+0000","last_undegraded":"2024-03-31T23:10:38.620015+0000","last_fullsized":"2024-03-31T23:10:38.620015+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T06:09:08.797695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":21,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-31T23:10:10.541260+0000","last_change":"2024-03-31T23:09:15.207336+0000","last_active":"2024-03-31T23:10:10.541260+0000","last_peered":"2024-03-31T23:10:10.541260+0000","last_clean":"2024-03-31T23:10:10.541260+0000","last_became_active":"2024-03-31T23:09:15.206893+0000","last_became_peered":"2024-03-31T23:09:15.206893+0000","last_unstale":"2024-03-31T23:10:10.541260+0000","last_undegraded":"2024-03-31T23:10:10.541260+0000","last_fullsized":"2024-03-31T23:10:10.541260+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_clean_scrub_stamp":"2024-03-31T23:09:14.171251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:24:59.766421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"91'3","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-31T23:10:10.478146+0000","last_change":"2024-03-31T23:09:10.587099+0000","last_active":"2024-03-31T23:10:10.478146+0000","last_peered":"2024-03-31T23:10:10.478146+0000","last_clean":"2024-03-31T23:10:10.478146+0000","last_became_active":"2024-03-31T23:09:10.586429+0000","last_became_peered":"2024-03-31T23:09:10.586429+0000","last_unstale":"2024-03-31T23:10:10.478146+0000","last_undegraded":"2024-03-31T23:10:10.478146+0000","last_fullsized":"2024-03-31T23:10:10.478146+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_clean_scrub_stamp":"2024-03-31T23:09:09.564263+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:55:09.307378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"93'15","reported_seq":42,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-31T23:10:38.605408+0000","last_change":"2024-03-31T23:09:11.177661+0000","last_active":"2024-03-31T23:10:38.605408+0000","last_peered":"2024-03-31T23:10:38.605408+0000","last_clean":"2024-03-31T23:10:38.605408+0000","last_became_active":"2024-03-31T23:09:11.177314+0000","last_became_peered":"2024-03-31T23:09:11.177314+0000","last_unstale":"2024-03-31T23:10:38.605408+0000","last_undegraded":"2024-03-31T23:10:38.605408+0000","last_fullsized":"2024-03-31T23:10:38.605408+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T09:26:50.556837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-31T23:10:10.544127+0000","last_change":"2024-03-31T23:10:10.543526+0000","last_active":"2024-03-31T23:10:10.544127+0000","last_peered":"2024-03-31T23:10:10.544127+0000","last_clean":"2024-03-31T23:10:10.544127+0000","last_became_active":"2024-03-31T23:10:10.543105+0000","last_became_peered":"2024-03-31T23:10:10.543105+0000","last_unstale":"2024-03-31T23:10:10.544127+0000","last_undegraded":"2024-03-31T23:10:10.544127+0000","last_fullsized":"2024-03-31T23:10:10.544127+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_clean_scrub_stamp":"2024-03-31T23:09:14.171251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T09:16:44.529786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,7,0,3],"acting":[8,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.11","version":"93'5","reported_seq":110,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-31T23:10:41.492735+0000","last_change":"2024-03-31T23:09:10.586570+0000","last_active":"2024-03-31T23:10:41.492735+0000","last_peered":"2024-03-31T23:10:41.492735+0000","last_clean":"2024-03-31T23:10:41.492735+0000","last_became_active":"2024-03-31T23:09:10.586239+0000","last_became_peered":"2024-03-31T23:09:10.586239+0000","last_unstale":"2024-03-31T23:10:41.492735+0000","last_undegraded":"2024-03-31T23:10:41.492735+0000","last_fullsized":"2024-03-31T23:10:41.492735+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_clean_scrub_stamp":"2024-03-31T23:09:09.564263+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:23:37.182954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1120,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":77,"num_read_kb":5,"num_write":4,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"93'6","reported_seq":33,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-31T23:10:38.602491+0000","last_change":"2024-03-31T23:09:11.173294+0000","last_active":"2024-03-31T23:10:38.602491+0000","last_peered":"2024-03-31T23:10:38.602491+0000","last_clean":"2024-03-31T23:10:38.602491+0000","last_became_active":"2024-03-31T23:09:11.172926+0000","last_became_peered":"2024-03-31T23:09:11.172926+0000","last_unstale":"2024-03-31T23:10:38.602491+0000","last_undegraded":"2024-03-31T23:10:38.602491+0000","last_fullsized":"2024-03-31T23:10:38.602491+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T06:21:46.603381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":21,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-31T23:10:10.477198+0000","last_change":"2024-03-31T23:09:15.199893+0000","last_active":"2024-03-31T23:10:10.477198+0000","last_peered":"2024-03-31T23:10:10.477198+0000","last_clean":"2024-03-31T23:10:10.477198+0000","last_became_active":"2024-03-31T23:09:15.199517+0000","last_became_peered":"2024-03-31T23:09:15.199517+0000","last_unstale":"2024-03-31T23:10:10.477198+0000","last_undegraded":"2024-03-31T23:10:10.477198+0000","last_fullsized":"2024-03-31T23:10:10.477198+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_clean_scrub_stamp":"2024-03-31T23:09:14.171251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T07:48:35.261108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"91'15","reported_seq":45,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-31T23:10:10.480140+0000","last_change":"2024-03-31T23:09:10.588213+0000","last_active":"2024-03-31T23:10:10.480140+0000","last_peered":"2024-03-31T23:10:10.480140+0000","last_clean":"2024-03-31T23:10:10.480140+0000","last_became_active":"2024-03-31T23:09:10.587579+0000","last_became_peered":"2024-03-31T23:09:10.587579+0000","last_unstale":"2024-03-31T23:10:10.480140+0000","last_undegraded":"2024-03-31T23:10:10.480140+0000","last_fullsized":"2024-03-31T23:10:10.480140+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_clean_scrub_stamp":"2024-03-31T23:09:09.564263+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:00:17.883435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"93'6","reported_seq":33,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-31T23:10:38.602643+0000","last_change":"2024-03-31T23:09:11.182427+0000","last_active":"2024-03-31T23:10:38.602643+0000","last_peered":"2024-03-31T23:10:38.602643+0000","last_clean":"2024-03-31T23:10:38.602643+0000","last_became_active":"2024-03-31T23:09:11.182120+0000","last_became_peered":"2024-03-31T23:09:11.182120+0000","last_unstale":"2024-03-31T23:10:38.602643+0000","last_undegraded":"2024-03-31T23:10:38.602643+0000","last_fullsized":"2024-03-31T23:10:38.602643+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:17:48.728424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":21,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-31T23:10:10.477809+0000","last_change":"2024-03-31T23:09:15.207831+0000","last_active":"2024-03-31T23:10:10.477809+0000","last_peered":"2024-03-31T23:10:10.477809+0000","last_clean":"2024-03-31T23:10:10.477809+0000","last_became_active":"2024-03-31T23:09:15.207432+0000","last_became_peered":"2024-03-31T23:09:15.207432+0000","last_unstale":"2024-03-31T23:10:10.477809+0000","last_undegraded":"2024-03-31T23:10:10.477809+0000","last_fullsized":"2024-03-31T23:10:10.477809+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_clean_scrub_stamp":"2024-03-31T23:09:14.171251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:05:27.072054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"91'2","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-31T23:10:10.480720+0000","last_change":"2024-03-31T23:09:10.591483+0000","last_active":"2024-03-31T23:10:10.480720+0000","last_peered":"2024-03-31T23:10:10.480720+0000","last_clean":"2024-03-31T23:10:10.480720+0000","last_became_active":"2024-03-31T23:09:10.591173+0000","last_became_peered":"2024-03-31T23:09:10.591173+0000","last_unstale":"2024-03-31T23:10:10.480720+0000","last_undegraded":"2024-03-31T23:10:10.480720+0000","last_fullsized":"2024-03-31T23:10:10.480720+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_clean_scrub_stamp":"2024-03-31T23:09:09.564263+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T10:04:14.275875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"93'5","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-31T23:10:38.602546+0000","last_change":"2024-03-31T23:09:11.181171+0000","last_active":"2024-03-31T23:10:38.602546+0000","last_peered":"2024-03-31T23:10:38.602546+0000","last_clean":"2024-03-31T23:10:38.602546+0000","last_became_active":"2024-03-31T23:09:11.180847+0000","last_became_peered":"2024-03-31T23:09:11.180847+0000","last_unstale":"2024-03-31T23:10:38.602546+0000","last_undegraded":"2024-03-31T23:10:38.602546+0000","last_fullsized":"2024-03-31T23:10:38.602546+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:39:29.429009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":21,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-31T23:10:10.480164+0000","last_change":"2024-03-31T23:09:15.746941+0000","last_active":"2024-03-31T23:10:10.480164+0000","last_peered":"2024-03-31T23:10:10.480164+0000","last_clean":"2024-03-31T23:10:10.480164+0000","last_became_active":"2024-03-31T23:09:15.746352+0000","last_became_peered":"2024-03-31T23:09:15.746352+0000","last_unstale":"2024-03-31T23:10:10.480164+0000","last_undegraded":"2024-03-31T23:10:10.480164+0000","last_fullsized":"2024-03-31T23:10:10.480164+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_clean_scrub_stamp":"2024-03-31T23:09:14.171251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T06:21:43.482575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":29,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-31T23:10:10.542934+0000","last_change":"2024-03-31T23:09:10.593029+0000","last_active":"2024-03-31T23:10:10.542934+0000","last_peered":"2024-03-31T23:10:10.542934+0000","last_clean":"2024-03-31T23:10:10.542934+0000","last_became_active":"2024-03-31T23:09:10.592709+0000","last_became_peered":"2024-03-31T23:09:10.592709+0000","last_unstale":"2024-03-31T23:10:10.542934+0000","last_undegraded":"2024-03-31T23:10:10.542934+0000","last_fullsized":"2024-03-31T23:10:10.542934+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_clean_scrub_stamp":"2024-03-31T23:09:09.564263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:46:45.700207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"93'4","reported_seq":31,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-31T23:10:38.612304+0000","last_change":"2024-03-31T23:09:11.177857+0000","last_active":"2024-03-31T23:10:38.612304+0000","last_peered":"2024-03-31T23:10:38.612304+0000","last_clean":"2024-03-31T23:10:38.612304+0000","last_became_active":"2024-03-31T23:09:11.177513+0000","last_became_peered":"2024-03-31T23:09:11.177513+0000","last_unstale":"2024-03-31T23:10:38.612304+0000","last_undegraded":"2024-03-31T23:10:38.612304+0000","last_fullsized":"2024-03-31T23:10:38.612304+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T06:09:23.579208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":21,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-31T23:10:10.476986+0000","last_change":"2024-03-31T23:09:15.203412+0000","last_active":"2024-03-31T23:10:10.476986+0000","last_peered":"2024-03-31T23:10:10.476986+0000","last_clean":"2024-03-31T23:10:10.476986+0000","last_became_active":"2024-03-31T23:09:15.202983+0000","last_became_peered":"2024-03-31T23:09:15.202983+0000","last_unstale":"2024-03-31T23:10:10.476986+0000","last_undegraded":"2024-03-31T23:10:10.476986+0000","last_fullsized":"2024-03-31T23:10:10.476986+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_clean_scrub_stamp":"2024-03-31T23:09:14.171251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:57:46.357378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"91'1","reported_seq":30,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-31T23:10:10.480628+0000","last_change":"2024-03-31T23:09:10.593299+0000","last_active":"2024-03-31T23:10:10.480628+0000","last_peered":"2024-03-31T23:10:10.480628+0000","last_clean":"2024-03-31T23:10:10.480628+0000","last_became_active":"2024-03-31T23:09:10.593007+0000","last_became_peered":"2024-03-31T23:09:10.593007+0000","last_unstale":"2024-03-31T23:10:10.480628+0000","last_undegraded":"2024-03-31T23:10:10.480628+0000","last_fullsized":"2024-03-31T23:10:10.480628+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_clean_scrub_stamp":"2024-03-31T23:09:09.564263+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:15:07.199645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"93'7","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-31T23:10:38.606243+0000","last_change":"2024-03-31T23:09:11.176846+0000","last_active":"2024-03-31T23:10:38.606243+0000","last_peered":"2024-03-31T23:10:38.606243+0000","last_clean":"2024-03-31T23:10:38.606243+0000","last_became_active":"2024-03-31T23:09:11.176554+0000","last_became_peered":"2024-03-31T23:09:11.176554+0000","last_unstale":"2024-03-31T23:10:38.606243+0000","last_undegraded":"2024-03-31T23:10:38.606243+0000","last_fullsized":"2024-03-31T23:10:38.606243+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:25:23.639093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":21,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-31T23:10:10.477184+0000","last_change":"2024-03-31T23:09:15.746599+0000","last_active":"2024-03-31T23:10:10.477184+0000","last_peered":"2024-03-31T23:10:10.477184+0000","last_clean":"2024-03-31T23:10:10.477184+0000","last_became_active":"2024-03-31T23:09:15.745879+0000","last_became_peered":"2024-03-31T23:09:15.745879+0000","last_unstale":"2024-03-31T23:10:10.477184+0000","last_undegraded":"2024-03-31T23:10:10.477184+0000","last_fullsized":"2024-03-31T23:10:10.477184+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_clean_scrub_stamp":"2024-03-31T23:09:14.171251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T08:02:25.198610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"93'8","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-31T23:10:38.585201+0000","last_change":"2024-03-31T23:09:11.175546+0000","last_active":"2024-03-31T23:10:38.585201+0000","last_peered":"2024-03-31T23:10:38.585201+0000","last_clean":"2024-03-31T23:10:38.585201+0000","last_became_active":"2024-03-31T23:09:11.175253+0000","last_became_peered":"2024-03-31T23:09:11.175253+0000","last_unstale":"2024-03-31T23:10:38.585201+0000","last_undegraded":"2024-03-31T23:10:38.585201+0000","last_fullsized":"2024-03-31T23:10:38.585201+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T10:27:31.753102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"93'5","reported_seq":108,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-31T23:10:43.494417+0000","last_change":"2024-03-31T23:09:10.580888+0000","last_active":"2024-03-31T23:10:43.494417+0000","last_peered":"2024-03-31T23:10:43.494417+0000","last_clean":"2024-03-31T23:10:43.494417+0000","last_became_active":"2024-03-31T23:09:10.580557+0000","last_became_peered":"2024-03-31T23:09:10.580557+0000","last_unstale":"2024-03-31T23:10:43.494417+0000","last_undegraded":"2024-03-31T23:10:43.494417+0000","last_fullsized":"2024-03-31T23:10:43.494417+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_clean_scrub_stamp":"2024-03-31T23:09:09.564263+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:17:38.878570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":74,"num_read_kb":74,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"91'1","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-31T23:10:10.480832+0000","last_change":"2024-03-31T23:09:15.205282+0000","last_active":"2024-03-31T23:10:10.480832+0000","last_peered":"2024-03-31T23:10:10.480832+0000","last_clean":"2024-03-31T23:10:10.480832+0000","last_became_active":"2024-03-31T23:09:15.204873+0000","last_became_peered":"2024-03-31T23:09:15.204873+0000","last_unstale":"2024-03-31T23:10:10.480832+0000","last_undegraded":"2024-03-31T23:10:10.480832+0000","last_fullsized":"2024-03-31T23:10:10.480832+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_clean_scrub_stamp":"2024-03-31T23:09:14.171251+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:56:05.697685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"93'8","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-31T23:10:38.619516+0000","last_change":"2024-03-31T23:09:11.182155+0000","last_active":"2024-03-31T23:10:38.619516+0000","last_peered":"2024-03-31T23:10:38.619516+0000","last_clean":"2024-03-31T23:10:38.619516+0000","last_became_active":"2024-03-31T23:09:11.181829+0000","last_became_peered":"2024-03-31T23:09:11.181829+0000","last_unstale":"2024-03-31T23:10:38.619516+0000","last_undegraded":"2024-03-31T23:10:38.619516+0000","last_fullsized":"2024-03-31T23:10:38.619516+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:19:22.832440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"91'3","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-31T23:10:10.479126+0000","last_change":"2024-03-31T23:09:10.590451+0000","last_active":"2024-03-31T23:10:10.479126+0000","last_peered":"2024-03-31T23:10:10.479126+0000","last_clean":"2024-03-31T23:10:10.479126+0000","last_became_active":"2024-03-31T23:09:10.590123+0000","last_became_peered":"2024-03-31T23:09:10.590123+0000","last_unstale":"2024-03-31T23:10:10.479126+0000","last_undegraded":"2024-03-31T23:10:10.479126+0000","last_fullsized":"2024-03-31T23:10:10.479126+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_clean_scrub_stamp":"2024-03-31T23:09:09.564263+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:27:02.724456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":365,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-31T23:10:10.480271+0000","last_change":"2024-03-31T23:07:57.920942+0000","last_active":"2024-03-31T23:10:10.480271+0000","last_peered":"2024-03-31T23:10:10.480271+0000","last_clean":"2024-03-31T23:10:10.480271+0000","last_became_active":"2024-03-31T23:07:57.920381+0000","last_became_peered":"2024-03-31T23:07:57.920381+0000","last_unstale":"2024-03-31T23:10:10.480271+0000","last_undegraded":"2024-03-31T23:10:10.480271+0000","last_fullsized":"2024-03-31T23:10:10.480271+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:04:33.508180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:04:33.508180+0000","last_clean_scrub_stamp":"2024-03-31T23:04:33.508180+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:20:11.211403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":21,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-31T23:10:10.478319+0000","last_change":"2024-03-31T23:09:15.746822+0000","last_active":"2024-03-31T23:10:10.478319+0000","last_peered":"2024-03-31T23:10:10.478319+0000","last_clean":"2024-03-31T23:10:10.478319+0000","last_became_active":"2024-03-31T23:09:15.746368+0000","last_became_peered":"2024-03-31T23:09:15.746368+0000","last_unstale":"2024-03-31T23:10:10.478319+0000","last_undegraded":"2024-03-31T23:10:10.478319+0000","last_fullsized":"2024-03-31T23:10:10.478319+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_clean_scrub_stamp":"2024-03-31T23:09:14.171251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T06:36:35.658042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"93'5","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-31T23:10:38.608658+0000","last_change":"2024-03-31T23:09:11.175336+0000","last_active":"2024-03-31T23:10:38.608658+0000","last_peered":"2024-03-31T23:10:38.608658+0000","last_clean":"2024-03-31T23:10:38.608658+0000","last_became_active":"2024-03-31T23:09:11.174768+0000","last_became_peered":"2024-03-31T23:09:11.174768+0000","last_unstale":"2024-03-31T23:10:38.608658+0000","last_undegraded":"2024-03-31T23:10:38.608658+0000","last_fullsized":"2024-03-31T23:10:38.608658+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:09:11.678744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"91'3","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-31T23:10:10.480389+0000","last_change":"2024-03-31T23:09:10.591807+0000","last_active":"2024-03-31T23:10:10.480389+0000","last_peered":"2024-03-31T23:10:10.480389+0000","last_clean":"2024-03-31T23:10:10.480389+0000","last_became_active":"2024-03-31T23:09:10.591407+0000","last_became_peered":"2024-03-31T23:09:10.591407+0000","last_unstale":"2024-03-31T23:10:10.480389+0000","last_undegraded":"2024-03-31T23:10:10.480389+0000","last_fullsized":"2024-03-31T23:10:10.480389+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_clean_scrub_stamp":"2024-03-31T23:09:09.564263+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T08:46:33.660717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":21,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-31T23:10:10.541292+0000","last_change":"2024-03-31T23:09:15.203879+0000","last_active":"2024-03-31T23:10:10.541292+0000","last_peered":"2024-03-31T23:10:10.541292+0000","last_clean":"2024-03-31T23:10:10.541292+0000","last_became_active":"2024-03-31T23:09:15.203412+0000","last_became_peered":"2024-03-31T23:09:15.203412+0000","last_unstale":"2024-03-31T23:10:10.541292+0000","last_undegraded":"2024-03-31T23:10:10.541292+0000","last_fullsized":"2024-03-31T23:10:10.541292+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_clean_scrub_stamp":"2024-03-31T23:09:14.171251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:03:54.486038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"91'1","reported_seq":31,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-31T23:10:10.480617+0000","last_change":"2024-03-31T23:09:10.596404+0000","last_active":"2024-03-31T23:10:10.480617+0000","last_peered":"2024-03-31T23:10:10.480617+0000","last_clean":"2024-03-31T23:10:10.480617+0000","last_became_active":"2024-03-31T23:09:10.596031+0000","last_became_peered":"2024-03-31T23:09:10.596031+0000","last_unstale":"2024-03-31T23:10:10.480617+0000","last_undegraded":"2024-03-31T23:10:10.480617+0000","last_fullsized":"2024-03-31T23:10:10.480617+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_clean_scrub_stamp":"2024-03-31T23:09:09.564263+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:30:31.904241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"93'9","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-31T23:10:38.607453+0000","last_change":"2024-03-31T23:09:11.164300+0000","last_active":"2024-03-31T23:10:38.607453+0000","last_peered":"2024-03-31T23:10:38.607453+0000","last_clean":"2024-03-31T23:10:38.607453+0000","last_became_active":"2024-03-31T23:09:11.163960+0000","last_became_peered":"2024-03-31T23:09:11.163960+0000","last_unstale":"2024-03-31T23:10:38.607453+0000","last_undegraded":"2024-03-31T23:10:38.607453+0000","last_fullsized":"2024-03-31T23:10:38.607453+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T10:37:30.512216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":21,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-31T23:10:10.541636+0000","last_change":"2024-03-31T23:09:15.746426+0000","last_active":"2024-03-31T23:10:10.541636+0000","last_peered":"2024-03-31T23:10:10.541636+0000","last_clean":"2024-03-31T23:10:10.541636+0000","last_became_active":"2024-03-31T23:09:15.745887+0000","last_became_peered":"2024-03-31T23:09:15.745887+0000","last_unstale":"2024-03-31T23:10:10.541636+0000","last_undegraded":"2024-03-31T23:10:10.541636+0000","last_fullsized":"2024-03-31T23:10:10.541636+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_clean_scrub_stamp":"2024-03-31T23:09:14.171251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:41:52.005300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"93'9","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-31T23:10:10.543326+0000","last_change":"2024-03-31T23:09:10.590631+0000","last_active":"2024-03-31T23:10:10.543326+0000","last_peered":"2024-03-31T23:10:10.543326+0000","last_clean":"2024-03-31T23:10:10.543326+0000","last_became_active":"2024-03-31T23:09:10.590211+0000","last_became_peered":"2024-03-31T23:09:10.590211+0000","last_unstale":"2024-03-31T23:10:10.543326+0000","last_undegraded":"2024-03-31T23:10:10.543326+0000","last_fullsized":"2024-03-31T23:10:10.543326+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_clean_scrub_stamp":"2024-03-31T23:09:09.564263+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T08:51:24.978892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"93'5","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-31T23:10:38.619166+0000","last_change":"2024-03-31T23:09:11.177353+0000","last_active":"2024-03-31T23:10:38.619166+0000","last_peered":"2024-03-31T23:10:38.619166+0000","last_clean":"2024-03-31T23:10:38.619166+0000","last_became_active":"2024-03-31T23:09:11.176996+0000","last_became_peered":"2024-03-31T23:09:11.176996+0000","last_unstale":"2024-03-31T23:10:38.619166+0000","last_undegraded":"2024-03-31T23:10:38.619166+0000","last_fullsized":"2024-03-31T23:10:38.619166+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:20:52.486281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":21,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-31T23:10:10.541115+0000","last_change":"2024-03-31T23:09:15.198528+0000","last_active":"2024-03-31T23:10:10.541115+0000","last_peered":"2024-03-31T23:10:10.541115+0000","last_clean":"2024-03-31T23:10:10.541115+0000","last_became_active":"2024-03-31T23:09:15.197938+0000","last_became_peered":"2024-03-31T23:09:15.197938+0000","last_unstale":"2024-03-31T23:10:10.541115+0000","last_undegraded":"2024-03-31T23:10:10.541115+0000","last_fullsized":"2024-03-31T23:10:10.541115+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_clean_scrub_stamp":"2024-03-31T23:09:14.171251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:45:46.567268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"91'4","reported_seq":33,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-31T23:10:10.479878+0000","last_change":"2024-03-31T23:09:10.584779+0000","last_active":"2024-03-31T23:10:10.479878+0000","last_peered":"2024-03-31T23:10:10.479878+0000","last_clean":"2024-03-31T23:10:10.479878+0000","last_became_active":"2024-03-31T23:09:10.584400+0000","last_became_peered":"2024-03-31T23:09:10.584400+0000","last_unstale":"2024-03-31T23:10:10.479878+0000","last_undegraded":"2024-03-31T23:10:10.479878+0000","last_fullsized":"2024-03-31T23:10:10.479878+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_clean_scrub_stamp":"2024-03-31T23:09:09.564263+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:10:10.095704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"93'7","reported_seq":40,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-31T23:10:38.618868+0000","last_change":"2024-03-31T23:10:10.543538+0000","last_active":"2024-03-31T23:10:38.618868+0000","last_peered":"2024-03-31T23:10:38.618868+0000","last_clean":"2024-03-31T23:10:38.618868+0000","last_became_active":"2024-03-31T23:10:10.543083+0000","last_became_peered":"2024-03-31T23:10:10.543083+0000","last_unstale":"2024-03-31T23:10:38.618868+0000","last_undegraded":"2024-03-31T23:10:38.618868+0000","last_fullsized":"2024-03-31T23:10:38.618868+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:38:50.656801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":21,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-31T23:10:10.479567+0000","last_change":"2024-03-31T23:09:15.746525+0000","last_active":"2024-03-31T23:10:10.479567+0000","last_peered":"2024-03-31T23:10:10.479567+0000","last_clean":"2024-03-31T23:10:10.479567+0000","last_became_active":"2024-03-31T23:09:15.746126+0000","last_became_peered":"2024-03-31T23:09:15.746126+0000","last_unstale":"2024-03-31T23:10:10.479567+0000","last_undegraded":"2024-03-31T23:10:10.479567+0000","last_fullsized":"2024-03-31T23:10:10.479567+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_clean_scrub_stamp":"2024-03-31T23:09:14.171251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:04:57.313615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"91'2","reported_seq":31,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-31T23:10:10.483343+0000","last_change":"2024-03-31T23:09:10.591469+0000","last_active":"2024-03-31T23:10:10.483343+0000","last_peered":"2024-03-31T23:10:10.483343+0000","last_clean":"2024-03-31T23:10:10.483343+0000","last_became_active":"2024-03-31T23:09:10.591166+0000","last_became_peered":"2024-03-31T23:09:10.591166+0000","last_unstale":"2024-03-31T23:10:10.483343+0000","last_undegraded":"2024-03-31T23:10:10.483343+0000","last_fullsized":"2024-03-31T23:10:10.483343+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_clean_scrub_stamp":"2024-03-31T23:09:09.564263+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T10:30:40.279041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"93'2","reported_seq":29,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-31T23:10:38.562829+0000","last_change":"2024-03-31T23:09:11.174471+0000","last_active":"2024-03-31T23:10:38.562829+0000","last_peered":"2024-03-31T23:10:38.562829+0000","last_clean":"2024-03-31T23:10:38.562829+0000","last_became_active":"2024-03-31T23:09:11.174154+0000","last_became_peered":"2024-03-31T23:09:11.174154+0000","last_unstale":"2024-03-31T23:10:38.562829+0000","last_undegraded":"2024-03-31T23:10:38.562829+0000","last_fullsized":"2024-03-31T23:10:38.562829+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T06:10:14.727009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":21,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-31T23:10:10.543374+0000","last_change":"2024-03-31T23:09:15.201416+0000","last_active":"2024-03-31T23:10:10.543374+0000","last_peered":"2024-03-31T23:10:10.543374+0000","last_clean":"2024-03-31T23:10:10.543374+0000","last_became_active":"2024-03-31T23:09:15.200987+0000","last_became_peered":"2024-03-31T23:09:15.200987+0000","last_unstale":"2024-03-31T23:10:10.543374+0000","last_undegraded":"2024-03-31T23:10:10.543374+0000","last_fullsized":"2024-03-31T23:10:10.543374+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_clean_scrub_stamp":"2024-03-31T23:09:14.171251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T10:56:54.872988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"91'3","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-31T23:10:10.541628+0000","last_change":"2024-03-31T23:09:10.594035+0000","last_active":"2024-03-31T23:10:10.541628+0000","last_peered":"2024-03-31T23:10:10.541628+0000","last_clean":"2024-03-31T23:10:10.541628+0000","last_became_active":"2024-03-31T23:09:10.593710+0000","last_became_peered":"2024-03-31T23:09:10.593710+0000","last_unstale":"2024-03-31T23:10:10.541628+0000","last_undegraded":"2024-03-31T23:10:10.541628+0000","last_fullsized":"2024-03-31T23:10:10.541628+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_clean_scrub_stamp":"2024-03-31T23:09:09.564263+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:46:48.580244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"93'9","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-31T23:10:38.611997+0000","last_change":"2024-03-31T23:09:11.170501+0000","last_active":"2024-03-31T23:10:38.611997+0000","last_peered":"2024-03-31T23:10:38.611997+0000","last_clean":"2024-03-31T23:10:38.611997+0000","last_became_active":"2024-03-31T23:09:11.170068+0000","last_became_peered":"2024-03-31T23:09:11.170068+0000","last_unstale":"2024-03-31T23:10:38.611997+0000","last_undegraded":"2024-03-31T23:10:38.611997+0000","last_fullsized":"2024-03-31T23:10:38.611997+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T06:05:53.488957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"91'1","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-31T23:10:10.479575+0000","last_change":"2024-03-31T23:09:15.208092+0000","last_active":"2024-03-31T23:10:10.479575+0000","last_peered":"2024-03-31T23:10:10.479575+0000","last_clean":"2024-03-31T23:10:10.479575+0000","last_became_active":"2024-03-31T23:09:15.207750+0000","last_became_peered":"2024-03-31T23:09:15.207750+0000","last_unstale":"2024-03-31T23:10:10.479575+0000","last_undegraded":"2024-03-31T23:10:10.479575+0000","last_fullsized":"2024-03-31T23:10:10.479575+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_clean_scrub_stamp":"2024-03-31T23:09:14.171251+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:24:39.649569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"91'1","reported_seq":30,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-31T23:10:10.541384+0000","last_change":"2024-03-31T23:09:10.591275+0000","last_active":"2024-03-31T23:10:10.541384+0000","last_peered":"2024-03-31T23:10:10.541384+0000","last_clean":"2024-03-31T23:10:10.541384+0000","last_became_active":"2024-03-31T23:09:10.590652+0000","last_became_peered":"2024-03-31T23:09:10.590652+0000","last_unstale":"2024-03-31T23:10:10.541384+0000","last_undegraded":"2024-03-31T23:10:10.541384+0000","last_fullsized":"2024-03-31T23:10:10.541384+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_clean_scrub_stamp":"2024-03-31T23:09:09.564263+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:58:39.037196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"93'8","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-31T23:10:38.617415+0000","last_change":"2024-03-31T23:09:11.175169+0000","last_active":"2024-03-31T23:10:38.617415+0000","last_peered":"2024-03-31T23:10:38.617415+0000","last_clean":"2024-03-31T23:10:38.617415+0000","last_became_active":"2024-03-31T23:09:11.174849+0000","last_became_peered":"2024-03-31T23:09:11.174849+0000","last_unstale":"2024-03-31T23:10:38.617415+0000","last_undegraded":"2024-03-31T23:10:38.617415+0000","last_fullsized":"2024-03-31T23:10:38.617415+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:58:06.814034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":21,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-31T23:10:10.479575+0000","last_change":"2024-03-31T23:09:15.746423+0000","last_active":"2024-03-31T23:10:10.479575+0000","last_peered":"2024-03-31T23:10:10.479575+0000","last_clean":"2024-03-31T23:10:10.479575+0000","last_became_active":"2024-03-31T23:09:15.745942+0000","last_became_peered":"2024-03-31T23:09:15.745942+0000","last_unstale":"2024-03-31T23:10:10.479575+0000","last_undegraded":"2024-03-31T23:10:10.479575+0000","last_fullsized":"2024-03-31T23:10:10.479575+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_clean_scrub_stamp":"2024-03-31T23:09:14.171251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:56:02.903212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"91'4","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-31T23:10:10.480085+0000","last_change":"2024-03-31T23:09:10.597524+0000","last_active":"2024-03-31T23:10:10.480085+0000","last_peered":"2024-03-31T23:10:10.480085+0000","last_clean":"2024-03-31T23:10:10.480085+0000","last_became_active":"2024-03-31T23:09:10.597222+0000","last_became_peered":"2024-03-31T23:09:10.597222+0000","last_unstale":"2024-03-31T23:10:10.480085+0000","last_undegraded":"2024-03-31T23:10:10.480085+0000","last_fullsized":"2024-03-31T23:10:10.480085+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_clean_scrub_stamp":"2024-03-31T23:09:09.564263+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T09:24:20.160978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"93'5","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-31T23:10:38.613128+0000","last_change":"2024-03-31T23:09:11.170440+0000","last_active":"2024-03-31T23:10:38.613128+0000","last_peered":"2024-03-31T23:10:38.613128+0000","last_clean":"2024-03-31T23:10:38.613128+0000","last_became_active":"2024-03-31T23:09:11.170081+0000","last_became_peered":"2024-03-31T23:09:11.170081+0000","last_unstale":"2024-03-31T23:10:38.613128+0000","last_undegraded":"2024-03-31T23:10:38.613128+0000","last_fullsized":"2024-03-31T23:10:38.613128+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:53:31.429822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":21,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-31T23:10:10.483461+0000","last_change":"2024-03-31T23:09:15.207146+0000","last_active":"2024-03-31T23:10:10.483461+0000","last_peered":"2024-03-31T23:10:10.483461+0000","last_clean":"2024-03-31T23:10:10.483461+0000","last_became_active":"2024-03-31T23:09:15.206769+0000","last_became_peered":"2024-03-31T23:09:15.206769+0000","last_unstale":"2024-03-31T23:10:10.483461+0000","last_undegraded":"2024-03-31T23:10:10.483461+0000","last_fullsized":"2024-03-31T23:10:10.483461+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_clean_scrub_stamp":"2024-03-31T23:09:14.171251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T07:42:12.961109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"91'3","reported_seq":33,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-31T23:10:10.541960+0000","last_change":"2024-03-31T23:09:10.587291+0000","last_active":"2024-03-31T23:10:10.541960+0000","last_peered":"2024-03-31T23:10:10.541960+0000","last_clean":"2024-03-31T23:10:10.541960+0000","last_became_active":"2024-03-31T23:09:10.585992+0000","last_became_peered":"2024-03-31T23:09:10.585992+0000","last_unstale":"2024-03-31T23:10:10.541960+0000","last_undegraded":"2024-03-31T23:10:10.541960+0000","last_fullsized":"2024-03-31T23:10:10.541960+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_clean_scrub_stamp":"2024-03-31T23:09:09.564263+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:28:00.786268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"93'7","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-31T23:10:38.614229+0000","last_change":"2024-03-31T23:09:11.179369+0000","last_active":"2024-03-31T23:10:38.614229+0000","last_peered":"2024-03-31T23:10:38.614229+0000","last_clean":"2024-03-31T23:10:38.614229+0000","last_became_active":"2024-03-31T23:09:11.178776+0000","last_became_peered":"2024-03-31T23:09:11.178776+0000","last_unstale":"2024-03-31T23:10:38.614229+0000","last_undegraded":"2024-03-31T23:10:38.614229+0000","last_fullsized":"2024-03-31T23:10:38.614229+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T07:56:05.201338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":21,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-31T23:10:10.480088+0000","last_change":"2024-03-31T23:09:15.201462+0000","last_active":"2024-03-31T23:10:10.480088+0000","last_peered":"2024-03-31T23:10:10.480088+0000","last_clean":"2024-03-31T23:10:10.480088+0000","last_became_active":"2024-03-31T23:09:15.201067+0000","last_became_peered":"2024-03-31T23:09:15.201067+0000","last_unstale":"2024-03-31T23:10:10.480088+0000","last_undegraded":"2024-03-31T23:10:10.480088+0000","last_fullsized":"2024-03-31T23:10:10.480088+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_clean_scrub_stamp":"2024-03-31T23:09:14.171251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:54:29.339456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"85'1","reported_seq":31,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-31T23:10:10.478802+0000","last_change":"2024-03-31T23:09:10.584752+0000","last_active":"2024-03-31T23:10:10.478802+0000","last_peered":"2024-03-31T23:10:10.478802+0000","last_clean":"2024-03-31T23:10:10.478802+0000","last_became_active":"2024-03-31T23:09:10.584432+0000","last_became_peered":"2024-03-31T23:09:10.584432+0000","last_unstale":"2024-03-31T23:10:10.478802+0000","last_undegraded":"2024-03-31T23:10:10.478802+0000","last_fullsized":"2024-03-31T23:10:10.478802+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_clean_scrub_stamp":"2024-03-31T23:09:09.564263+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T10:50:53.889719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"93'6","reported_seq":39,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-31T23:10:38.609372+0000","last_change":"2024-03-31T23:10:10.543993+0000","last_active":"2024-03-31T23:10:38.609372+0000","last_peered":"2024-03-31T23:10:38.609372+0000","last_clean":"2024-03-31T23:10:38.609372+0000","last_became_active":"2024-03-31T23:10:10.543627+0000","last_became_peered":"2024-03-31T23:10:10.543627+0000","last_unstale":"2024-03-31T23:10:38.609372+0000","last_undegraded":"2024-03-31T23:10:38.609372+0000","last_fullsized":"2024-03-31T23:10:38.609372+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T08:46:41.221122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":21,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-31T23:10:10.477677+0000","last_change":"2024-03-31T23:09:15.197271+0000","last_active":"2024-03-31T23:10:10.477677+0000","last_peered":"2024-03-31T23:10:10.477677+0000","last_clean":"2024-03-31T23:10:10.477677+0000","last_became_active":"2024-03-31T23:09:15.196339+0000","last_became_peered":"2024-03-31T23:09:15.196339+0000","last_unstale":"2024-03-31T23:10:10.477677+0000","last_undegraded":"2024-03-31T23:10:10.477677+0000","last_fullsized":"2024-03-31T23:10:10.477677+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_clean_scrub_stamp":"2024-03-31T23:09:14.171251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:44:13.044278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"85'2","reported_seq":31,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-31T23:10:10.477160+0000","last_change":"2024-03-31T23:09:10.595131+0000","last_active":"2024-03-31T23:10:10.477160+0000","last_peered":"2024-03-31T23:10:10.477160+0000","last_clean":"2024-03-31T23:10:10.477160+0000","last_became_active":"2024-03-31T23:09:10.594740+0000","last_became_peered":"2024-03-31T23:09:10.594740+0000","last_unstale":"2024-03-31T23:10:10.477160+0000","last_undegraded":"2024-03-31T23:10:10.477160+0000","last_fullsized":"2024-03-31T23:10:10.477160+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_clean_scrub_stamp":"2024-03-31T23:09:09.564263+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:52:41.849621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"93'9","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-31T23:10:38.601765+0000","last_change":"2024-03-31T23:09:11.177530+0000","last_active":"2024-03-31T23:10:38.601765+0000","last_peered":"2024-03-31T23:10:38.601765+0000","last_clean":"2024-03-31T23:10:38.601765+0000","last_became_active":"2024-03-31T23:09:11.176972+0000","last_became_peered":"2024-03-31T23:09:11.176972+0000","last_unstale":"2024-03-31T23:10:38.601765+0000","last_undegraded":"2024-03-31T23:10:38.601765+0000","last_fullsized":"2024-03-31T23:10:38.601765+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T07:57:21.094911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"93'9","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-31T23:10:38.606882+0000","last_change":"2024-03-31T23:09:11.172792+0000","last_active":"2024-03-31T23:10:38.606882+0000","last_peered":"2024-03-31T23:10:38.606882+0000","last_clean":"2024-03-31T23:10:38.606882+0000","last_became_active":"2024-03-31T23:09:11.172463+0000","last_became_peered":"2024-03-31T23:09:11.172463+0000","last_unstale":"2024-03-31T23:10:38.606882+0000","last_undegraded":"2024-03-31T23:10:38.606882+0000","last_fullsized":"2024-03-31T23:10:38.606882+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:07:28.789783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"93'12","reported_seq":39,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-31T23:10:38.582022+0000","last_change":"2024-03-31T23:09:11.174331+0000","last_active":"2024-03-31T23:10:38.582022+0000","last_peered":"2024-03-31T23:10:38.582022+0000","last_clean":"2024-03-31T23:10:38.582022+0000","last_became_active":"2024-03-31T23:09:11.174001+0000","last_became_peered":"2024-03-31T23:09:11.174001+0000","last_unstale":"2024-03-31T23:10:38.582022+0000","last_undegraded":"2024-03-31T23:10:38.582022+0000","last_fullsized":"2024-03-31T23:10:38.582022+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:26:28.005406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"93'8","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-31T23:10:38.579227+0000","last_change":"2024-03-31T23:09:11.180047+0000","last_active":"2024-03-31T23:10:38.579227+0000","last_peered":"2024-03-31T23:10:38.579227+0000","last_clean":"2024-03-31T23:10:38.579227+0000","last_became_active":"2024-03-31T23:09:11.179691+0000","last_became_peered":"2024-03-31T23:09:11.179691+0000","last_unstale":"2024-03-31T23:10:38.579227+0000","last_undegraded":"2024-03-31T23:10:38.579227+0000","last_fullsized":"2024-03-31T23:10:38.579227+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:28:05.083710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"93'7","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-31T23:10:38.608233+0000","last_change":"2024-03-31T23:09:11.177642+0000","last_active":"2024-03-31T23:10:38.608233+0000","last_peered":"2024-03-31T23:10:38.608233+0000","last_clean":"2024-03-31T23:10:38.608233+0000","last_became_active":"2024-03-31T23:09:11.177261+0000","last_became_peered":"2024-03-31T23:09:11.177261+0000","last_unstale":"2024-03-31T23:10:38.608233+0000","last_undegraded":"2024-03-31T23:10:38.608233+0000","last_fullsized":"2024-03-31T23:10:38.608233+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T07:10:43.616863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"93'4","reported_seq":31,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-31T23:10:38.611726+0000","last_change":"2024-03-31T23:09:11.173606+0000","last_active":"2024-03-31T23:10:38.611726+0000","last_peered":"2024-03-31T23:10:38.611726+0000","last_clean":"2024-03-31T23:10:38.611726+0000","last_became_active":"2024-03-31T23:09:11.173304+0000","last_became_peered":"2024-03-31T23:09:11.173304+0000","last_unstale":"2024-03-31T23:10:38.611726+0000","last_undegraded":"2024-03-31T23:10:38.611726+0000","last_fullsized":"2024-03-31T23:10:38.611726+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:53:06.112181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"93'6","reported_seq":33,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-31T23:10:38.602905+0000","last_change":"2024-03-31T23:09:11.176628+0000","last_active":"2024-03-31T23:10:38.602905+0000","last_peered":"2024-03-31T23:10:38.602905+0000","last_clean":"2024-03-31T23:10:38.602905+0000","last_became_active":"2024-03-31T23:09:11.176318+0000","last_became_peered":"2024-03-31T23:09:11.176318+0000","last_unstale":"2024-03-31T23:10:38.602905+0000","last_undegraded":"2024-03-31T23:10:38.602905+0000","last_fullsized":"2024-03-31T23:10:38.602905+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:06:40.723675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"93'15","reported_seq":42,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-31T23:10:38.600813+0000","last_change":"2024-03-31T23:09:11.169318+0000","last_active":"2024-03-31T23:10:38.600813+0000","last_peered":"2024-03-31T23:10:38.600813+0000","last_clean":"2024-03-31T23:10:38.600813+0000","last_became_active":"2024-03-31T23:09:11.169026+0000","last_became_peered":"2024-03-31T23:09:11.169026+0000","last_unstale":"2024-03-31T23:10:38.600813+0000","last_undegraded":"2024-03-31T23:10:38.600813+0000","last_fullsized":"2024-03-31T23:10:38.600813+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:02:18.562854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"93'2","reported_seq":29,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-31T23:10:38.599689+0000","last_change":"2024-03-31T23:09:11.184670+0000","last_active":"2024-03-31T23:10:38.599689+0000","last_peered":"2024-03-31T23:10:38.599689+0000","last_clean":"2024-03-31T23:10:38.599689+0000","last_became_active":"2024-03-31T23:09:11.184354+0000","last_became_peered":"2024-03-31T23:09:11.184354+0000","last_unstale":"2024-03-31T23:10:38.599689+0000","last_undegraded":"2024-03-31T23:10:38.599689+0000","last_fullsized":"2024-03-31T23:10:38.599689+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:16:14.354947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"93'6","reported_seq":33,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-31T23:10:38.591633+0000","last_change":"2024-03-31T23:09:11.180991+0000","last_active":"2024-03-31T23:10:38.591633+0000","last_peered":"2024-03-31T23:10:38.591633+0000","last_clean":"2024-03-31T23:10:38.591633+0000","last_became_active":"2024-03-31T23:09:11.180667+0000","last_became_peered":"2024-03-31T23:09:11.180667+0000","last_unstale":"2024-03-31T23:10:38.591633+0000","last_undegraded":"2024-03-31T23:10:38.591633+0000","last_fullsized":"2024-03-31T23:10:38.591633+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T06:35:33.157176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"93'8","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-31T23:10:38.617978+0000","last_change":"2024-03-31T23:09:11.182025+0000","last_active":"2024-03-31T23:10:38.617978+0000","last_peered":"2024-03-31T23:10:38.617978+0000","last_clean":"2024-03-31T23:10:38.617978+0000","last_became_active":"2024-03-31T23:09:11.179310+0000","last_became_peered":"2024-03-31T23:09:11.179310+0000","last_unstale":"2024-03-31T23:10:38.617978+0000","last_undegraded":"2024-03-31T23:10:38.617978+0000","last_fullsized":"2024-03-31T23:10:38.617978+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T10:21:40.784738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"93'10","reported_seq":43,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-31T23:10:38.615754+0000","last_change":"2024-03-31T23:10:10.485623+0000","last_active":"2024-03-31T23:10:38.615754+0000","last_peered":"2024-03-31T23:10:38.615754+0000","last_clean":"2024-03-31T23:10:38.615754+0000","last_became_active":"2024-03-31T23:10:10.485248+0000","last_became_peered":"2024-03-31T23:10:10.485248+0000","last_unstale":"2024-03-31T23:10:38.615754+0000","last_undegraded":"2024-03-31T23:10:38.615754+0000","last_fullsized":"2024-03-31T23:10:38.615754+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:59:04.668598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,6],"acting":[1,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"93'11","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-31T23:10:38.612432+0000","last_change":"2024-03-31T23:09:11.172585+0000","last_active":"2024-03-31T23:10:38.612432+0000","last_peered":"2024-03-31T23:10:38.612432+0000","last_clean":"2024-03-31T23:10:38.612432+0000","last_became_active":"2024-03-31T23:09:11.172266+0000","last_became_peered":"2024-03-31T23:09:11.172266+0000","last_unstale":"2024-03-31T23:10:38.612432+0000","last_undegraded":"2024-03-31T23:10:38.612432+0000","last_fullsized":"2024-03-31T23:10:38.612432+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:38:26.570175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"93'8","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-31T23:10:38.606543+0000","last_change":"2024-03-31T23:09:11.181434+0000","last_active":"2024-03-31T23:10:38.606543+0000","last_peered":"2024-03-31T23:10:38.606543+0000","last_clean":"2024-03-31T23:10:38.606543+0000","last_became_active":"2024-03-31T23:09:11.181140+0000","last_became_peered":"2024-03-31T23:09:11.181140+0000","last_unstale":"2024-03-31T23:10:38.606543+0000","last_undegraded":"2024-03-31T23:10:38.606543+0000","last_fullsized":"2024-03-31T23:10:38.606543+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:55:06.379696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4157440,"data_stored":4132576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":67256,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":474,"num_read_kb":312,"num_write":172,"num_write_kb":125,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":430080,"data_stored":201768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":81,"seq":347892351009,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36364,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708788,"statfs":{"total":95995035648,"available":95957798912,"internally_reserved":0,"allocated":1052672,"data_stored":514732,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":36174276},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":73,"seq":313532612646,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28720,"kb_used_data":1704,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716432,"statfs":{"total":95995035648,"available":95965626368,"internally_reserved":0,"allocated":1744896,"data_stored":1165114,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":68,"seq":292057776171,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28064,"kb_used_data":1048,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717088,"statfs":{"total":95995035648,"available":95966298112,"internally_reserved":0,"allocated":1073152,"data_stored":519064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":60,"seq":257698037807,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28060,"kb_used_data":1044,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717092,"statfs":{"total":95995035648,"available":95966302208,"internally_reserved":0,"allocated":1069056,"data_stored":514888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299445,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28640,"kb_used_data":1624,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716512,"statfs":{"total":95995035648,"available":95965708288,"internally_reserved":0,"allocated":1662976,"data_stored":1111470,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1588,"internal_metadata":27654604},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863462970,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28040,"kb_used_data":1024,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717112,"statfs":{"total":95995035648,"available":95966322688,"internally_reserved":0,"allocated":1048576,"data_stored":510144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574930,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28108,"kb_used_data":1092,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717044,"statfs":{"total":95995035648,"available":95966253056,"internally_reserved":0,"allocated":1118208,"data_stored":572636,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738455,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36384,"kb_used_data":1048,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708768,"statfs":{"total":95995035648,"available":95957778432,"internally_reserved":0,"allocated":1073152,"data_stored":511532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411406,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28132,"kb_used_data":1116,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717020,"statfs":{"total":95995035648,"available":95966228480,"internally_reserved":0,"allocated":1142784,"data_stored":574518,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182473,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28052,"kb_used_data":1036,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717100,"statfs":{"total":95995035648,"available":95966310400,"internally_reserved":0,"allocated":1060864,"data_stored":510338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920836,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36372,"kb_used_data":1036,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708780,"statfs":{"total":95995035648,"available":95957790720,"internally_reserved":0,"allocated":1060864,"data_stored":515326,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724607,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36968,"kb_used_data":1632,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708184,"statfs":{"total":95995035648,"available":95957180416,"internally_reserved":0,"allocated":1671168,"data_stored":1106526,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":62582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":64464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":64692,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-31T23:10:48.845 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-03-31T23:10:48.846 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-03-31T23:10:48.846 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-03-31T23:10:48.846 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:384f31f3d3aee7f8ba3f60846b8844c049165583 shell --fsid 6a9d8f08-efb2-11ee-b647-cb9ed24678a4 -- ceph health --format=json 2024-03-31T23:10:49.095 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/6a9d8f08-efb2-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-31T23:10:49.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:10:49 smithi162 ceph-mon[36237]: from='client.15132 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-31T23:10:49.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:10:49 smithi046 ceph-mon[32498]: from='client.15132 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-31T23:10:49.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:10:49 smithi142 ceph-mon[35271]: from='client.15132 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-31T23:10:50.710 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:10:50 smithi046 ceph-mon[32498]: pgmap v343: 129 pgs: 129 active+clean; 642 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 597 B/s wr, 6 op/s 2024-03-31T23:10:50.795 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:10:50 smithi162 ceph-mon[36237]: pgmap v343: 129 pgs: 129 active+clean; 642 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 597 B/s wr, 6 op/s 2024-03-31T23:10:50.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:10:50 smithi142 ceph-mon[35271]: pgmap v343: 129 pgs: 129 active+clean; 642 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 597 B/s wr, 6 op/s 2024-03-31T23:10:50.951 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:50.948+0000 7f310c43d700 1 -- 172.21.15.46:0/334590598 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3104102650 msgr2=0x7f310410e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:10:50.951 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:50.948+0000 7f310c43d700 1 --2- 172.21.15.46:0/334590598 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3104102650 0x7f310410e020 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f30fc0098f0 tx=0x7f30fc009c00 comp rx=0 tx=0).stop 2024-03-31T23:10:50.951 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:50.948+0000 7f310c43d700 1 -- 172.21.15.46:0/334590598 shutdown_connections 2024-03-31T23:10:50.951 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:50.948+0000 7f310c43d700 1 --2- 172.21.15.46:0/334590598 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3104106180 0x7f3104106560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:50.951 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:50.948+0000 7f310c43d700 1 --2- 172.21.15.46:0/334590598 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3104102650 0x7f310410e020 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:50.951 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:50.948+0000 7f310c43d700 1 --2- 172.21.15.46:0/334590598 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3104101c90 0x7f3104102110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:50.951 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:50.948+0000 7f310c43d700 1 -- 172.21.15.46:0/334590598 >> 172.21.15.46:0/334590598 conn(0x7f3104076480 msgr2=0x7f3104076890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:10:50.951 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:50.948+0000 7f310c43d700 1 -- 172.21.15.46:0/334590598 shutdown_connections 2024-03-31T23:10:50.951 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:50.948+0000 7f310c43d700 1 -- 172.21.15.46:0/334590598 wait complete. 2024-03-31T23:10:50.951 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:50.949+0000 7f310c43d700 1 Processor -- start 2024-03-31T23:10:50.951 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:50.949+0000 7f310c43d700 1 -- start start 2024-03-31T23:10:50.952 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:50.949+0000 7f310c43d700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3104101c90 0x7f310419e640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:10:50.952 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:50.949+0000 7f310c43d700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3104102650 0x7f310419eb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:10:50.952 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:50.949+0000 7f310c43d700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3104106180 0x7f3104198710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:10:50.952 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:50.949+0000 7f310c43d700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f3104111330 con 0x7f3104106180 2024-03-31T23:10:50.952 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:50.949+0000 7f310c43d700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f31041111b0 con 0x7f3104102650 2024-03-31T23:10:50.952 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:50.949+0000 7f310c43d700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f31041114b0 con 0x7f3104101c90 2024-03-31T23:10:50.952 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:50.949+0000 7f310a9da700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3104106180 0x7f3104198710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:10:50.952 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:50.949+0000 7f310a1d9700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3104101c90 0x7f310419e640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:10:50.952 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:50.949+0000 7f31099d8700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3104102650 0x7f310419eb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:10:50.952 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:50.949+0000 7f310a9da700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3104106180 0x7f3104198710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:52046/0 (socket says 172.21.15.46:52046) 2024-03-31T23:10:50.952 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:50.949+0000 7f310a1d9700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3104101c90 0x7f310419e640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.46:41810/0 (socket says 172.21.15.46:41810) 2024-03-31T23:10:50.952 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:50.949+0000 7f310a9da700 1 -- 172.21.15.46:0/2349364724 learned_addr learned my addr 172.21.15.46:0/2349364724 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:10:50.952 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:50.949+0000 7f310a9da700 1 -- 172.21.15.46:0/2349364724 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3104101c90 msgr2=0x7f310419e640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:10:50.953 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:50.949+0000 7f310a9da700 1 --2- 172.21.15.46:0/2349364724 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3104101c90 0x7f310419e640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:50.953 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:50.949+0000 7f310a9da700 1 -- 172.21.15.46:0/2349364724 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3104102650 msgr2=0x7f310419eb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:10:50.953 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:50.949+0000 7f310a9da700 1 --2- 172.21.15.46:0/2349364724 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3104102650 0x7f310419eb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:50.953 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:50.949+0000 7f310a9da700 1 -- 172.21.15.46:0/2349364724 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3104198ff0 con 0x7f3104106180 2024-03-31T23:10:50.953 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:50.949+0000 7f310a1d9700 1 --2- 172.21.15.46:0/2349364724 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3104101c90 0x7f310419e640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:10:50.953 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:50.950+0000 7f310a9da700 1 --2- 172.21.15.46:0/2349364724 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3104106180 0x7f3104198710 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f30fc009810 tx=0x7f30fc00f9e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:10:50.953 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:50.950+0000 7f31037fe700 1 -- 172.21.15.46:0/2349364724 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30fc0050c0 con 0x7f3104106180 2024-03-31T23:10:50.953 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:50.950+0000 7f31037fe700 1 -- 172.21.15.46:0/2349364724 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f30fc00b560 con 0x7f3104106180 2024-03-31T23:10:50.953 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:50.950+0000 7f310c43d700 1 -- 172.21.15.46:0/2349364724 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3104199280 con 0x7f3104106180 2024-03-31T23:10:50.953 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:50.950+0000 7f31037fe700 1 -- 172.21.15.46:0/2349364724 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30fc003a40 con 0x7f3104106180 2024-03-31T23:10:50.953 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:50.950+0000 7f310c43d700 1 -- 172.21.15.46:0/2349364724 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f31041a53f0 con 0x7f3104106180 2024-03-31T23:10:50.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:50.951+0000 7f310c43d700 1 -- 172.21.15.46:0/2349364724 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f310406a510 con 0x7f3104106180 2024-03-31T23:10:50.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:50.952+0000 7f31037fe700 1 -- 172.21.15.46:0/2349364724 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 18) ==== 137320+0+0 (secure 0 0 0) 0x7f30fc00ba80 con 0x7f3104106180 2024-03-31T23:10:50.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:50.952+0000 7f31037fe700 1 --2- 172.21.15.46:0/2349364724 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f30ec0a2530 0x7f30ec0a49f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:10:50.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:50.953+0000 7f310a1d9700 1 --2- 172.21.15.46:0/2349364724 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f30ec0a2530 0x7f30ec0a49f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:10:50.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:50.953+0000 7f31037fe700 1 -- 172.21.15.46:0/2349364724 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7f30fc0cfb20 con 0x7f3104106180 2024-03-31T23:10:50.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:50.953+0000 7f310a1d9700 1 --2- 172.21.15.46:0/2349364724 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f30ec0a2530 0x7f30ec0a49f0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f30f4009cd0 tx=0x7f30f400b040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:10:50.958 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:50.957+0000 7f31037fe700 1 -- 172.21.15.46:0/2349364724 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f30fc013030 con 0x7f3104106180 2024-03-31T23:10:51.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:51.205+0000 7f310c43d700 1 -- 172.21.15.46:0/2349364724 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7f3104199c70 con 0x7f3104106180 2024-03-31T23:10:51.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:51.206+0000 7f31037fe700 1 -- 172.21.15.46:0/2349364724 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+46 (secure 0 0 0) 0x7f3104199c70 con 0x7f3104106180 2024-03-31T23:10:51.207 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-31T23:10:51.207 INFO:teuthology.orchestra.run.smithi046.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-03-31T23:10:51.209 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:51.208+0000 7f310c43d700 1 -- 172.21.15.46:0/2349364724 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f30ec0a2530 msgr2=0x7f30ec0a49f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:10:51.209 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:51.208+0000 7f310c43d700 1 --2- 172.21.15.46:0/2349364724 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f30ec0a2530 0x7f30ec0a49f0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f30f4009cd0 tx=0x7f30f400b040 comp rx=0 tx=0).stop 2024-03-31T23:10:51.210 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:51.208+0000 7f310c43d700 1 -- 172.21.15.46:0/2349364724 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3104106180 msgr2=0x7f3104198710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:10:51.210 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:51.208+0000 7f310c43d700 1 --2- 172.21.15.46:0/2349364724 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3104106180 0x7f3104198710 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f30fc009810 tx=0x7f30fc00f9e0 comp rx=0 tx=0).stop 2024-03-31T23:10:51.210 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:51.208+0000 7f310c43d700 1 -- 172.21.15.46:0/2349364724 shutdown_connections 2024-03-31T23:10:51.210 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:51.208+0000 7f310c43d700 1 --2- 172.21.15.46:0/2349364724 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f30ec0a2530 0x7f30ec0a49f0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:51.210 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:51.208+0000 7f310c43d700 1 --2- 172.21.15.46:0/2349364724 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3104106180 0x7f3104198710 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:51.210 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:51.208+0000 7f310c43d700 1 --2- 172.21.15.46:0/2349364724 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3104101c90 0x7f310419e640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:51.210 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:51.208+0000 7f310c43d700 1 --2- 172.21.15.46:0/2349364724 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3104102650 0x7f310419eb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:51.210 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:51.208+0000 7f310c43d700 1 -- 172.21.15.46:0/2349364724 >> 172.21.15.46:0/2349364724 conn(0x7f3104076480 msgr2=0x7f3104100d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:10:51.210 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:51.208+0000 7f310c43d700 1 -- 172.21.15.46:0/2349364724 shutdown_connections 2024-03-31T23:10:51.210 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:51.208+0000 7f310c43d700 1 -- 172.21.15.46:0/2349364724 wait complete. 2024-03-31T23:10:51.795 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:10:51 smithi162 ceph-mon[36237]: from='client.? 172.21.15.46:0/2349364724' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-31T23:10:51.844 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:10:51 smithi046 ceph-mon[32498]: from='client.? 172.21.15.46:0/2349364724' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-31T23:10:51.874 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-03-31T23:10:51.875 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-31T23:10:51.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:10:51 smithi142 ceph-mon[35271]: from='client.? 172.21.15.46:0/2349364724' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-31T23:10:51.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:51.982+0000 7f9537765640 1 -- 172.21.15.46:0/4206877661 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9530102960 msgr2=0x7f9530102de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:10:51.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:51.982+0000 7f9537765640 1 --2- 172.21.15.46:0/4206877661 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9530102960 0x7f9530102de0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f952c009900 tx=0x7f952c02bc30 comp rx=0 tx=0).stop 2024-03-31T23:10:51.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:51.982+0000 7f9537765640 1 -- 172.21.15.46:0/4206877661 shutdown_connections 2024-03-31T23:10:51.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:51.982+0000 7f9537765640 1 --2- 172.21.15.46:0/4206877661 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9530103320 0x7f9530109bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:51.986 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:51.982+0000 7f9537765640 1 --2- 172.21.15.46:0/4206877661 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9530102960 0x7f9530102de0 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:51.986 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:51.982+0000 7f9537765640 1 --2- 172.21.15.46:0/4206877661 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9530101770 0x7f9530101b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:51.986 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:51.982+0000 7f9537765640 1 -- 172.21.15.46:0/4206877661 >> 172.21.15.46:0/4206877661 conn(0x7f95300fcd00 msgr2=0x7f95300ff120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:10:51.986 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:51.982+0000 7f9537765640 1 -- 172.21.15.46:0/4206877661 shutdown_connections 2024-03-31T23:10:51.986 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:51.982+0000 7f9537765640 1 -- 172.21.15.46:0/4206877661 wait complete. 2024-03-31T23:10:51.986 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:51.983+0000 7f9537765640 1 Processor -- start 2024-03-31T23:10:51.986 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:51.983+0000 7f9537765640 1 -- start start 2024-03-31T23:10:51.986 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:51.983+0000 7f9537765640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9530101770 0x7f9530075500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:10:51.986 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:51.983+0000 7f9537765640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9530102960 0x7f9530073b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:10:51.986 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:51.983+0000 7f9537765640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9530103320 0x7f9530074080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:10:51.986 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:51.983+0000 7f9537765640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f953010d280 con 0x7f9530103320 2024-03-31T23:10:51.986 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:51.983+0000 7f9537765640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f953010d100 con 0x7f9530102960 2024-03-31T23:10:51.986 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:51.983+0000 7f9537765640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f953010d400 con 0x7f9530101770 2024-03-31T23:10:51.987 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:51.983+0000 7f9535d02640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9530103320 0x7f9530074080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:10:51.987 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:51.983+0000 7f9535d02640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9530103320 0x7f9530074080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:56120/0 (socket says 172.21.15.46:56120) 2024-03-31T23:10:51.987 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:51.983+0000 7f9535d02640 1 -- 172.21.15.46:0/2546390590 learned_addr learned my addr 172.21.15.46:0/2546390590 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:10:51.987 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:51.983+0000 7f9535501640 1 --2- 172.21.15.46:0/2546390590 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9530101770 0x7f9530075500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:10:51.987 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:51.983+0000 7f9534d00640 1 --2- 172.21.15.46:0/2546390590 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9530102960 0x7f9530073b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:10:51.987 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:51.983+0000 7f9535d02640 1 -- 172.21.15.46:0/2546390590 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9530101770 msgr2=0x7f9530075500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:10:51.987 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:51.983+0000 7f9535d02640 1 --2- 172.21.15.46:0/2546390590 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9530101770 0x7f9530075500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:51.987 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:51.984+0000 7f9535d02640 1 -- 172.21.15.46:0/2546390590 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9530102960 msgr2=0x7f9530073b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:10:51.987 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:51.984+0000 7f9535d02640 1 --2- 172.21.15.46:0/2546390590 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9530102960 0x7f9530073b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:51.987 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:51.984+0000 7f9535d02640 1 -- 172.21.15.46:0/2546390590 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9530074910 con 0x7f9530103320 2024-03-31T23:10:51.987 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:51.984+0000 7f9535d02640 1 --2- 172.21.15.46:0/2546390590 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9530103320 0x7f9530074080 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f952800eb10 tx=0x7f952800c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:10:51.987 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:51.984+0000 7f95267fc640 1 -- 172.21.15.46:0/2546390590 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9528004b20 con 0x7f9530103320 2024-03-31T23:10:51.987 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:51.984+0000 7f95267fc640 1 -- 172.21.15.46:0/2546390590 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9528004cc0 con 0x7f9530103320 2024-03-31T23:10:51.987 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:51.984+0000 7f9537765640 1 -- 172.21.15.46:0/2546390590 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f95301a1aa0 con 0x7f9530103320 2024-03-31T23:10:51.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:51.984+0000 7f95267fc640 1 -- 172.21.15.46:0/2546390590 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9528010690 con 0x7f9530103320 2024-03-31T23:10:51.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:51.984+0000 7f9537765640 1 -- 172.21.15.46:0/2546390590 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f95301a1f70 con 0x7f9530103320 2024-03-31T23:10:51.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:51.985+0000 7f9537765640 1 -- 172.21.15.46:0/2546390590 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9530068990 con 0x7f9530103320 2024-03-31T23:10:51.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:51.986+0000 7f95267fc640 1 -- 172.21.15.46:0/2546390590 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 18) ==== 137320+0+0 (secure 0 0 0) 0x7f9528005020 con 0x7f9530103320 2024-03-31T23:10:51.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:51.986+0000 7f95267fc640 1 --2- 172.21.15.46:0/2546390590 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f951c0a24a0 0x7f951c0a4960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:10:51.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:51.986+0000 7f9535501640 1 --2- 172.21.15.46:0/2546390590 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f951c0a24a0 0x7f951c0a4960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:10:51.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:51.986+0000 7f95267fc640 1 -- 172.21.15.46:0/2546390590 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7f95280cf080 con 0x7f9530103320 2024-03-31T23:10:51.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:51.987+0000 7f9535501640 1 --2- 172.21.15.46:0/2546390590 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f951c0a24a0 0x7f951c0a4960 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f95301027c0 tx=0x7f9520005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:10:51.992 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:51.991+0000 7f95267fc640 1 -- 172.21.15.46:0/2546390590 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f952809bee0 con 0x7f9530103320 2024-03-31T23:10:52.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:52.173+0000 7f9537765640 1 -- 172.21.15.46:0/2546390590 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f9530108b60 con 0x7f9530103320 2024-03-31T23:10:52.176 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:52.175+0000 7f95267fc640 1 -- 172.21.15.46:0/2546390590 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v93) ==== 74+0+24575 (secure 0 0 0) 0x7f952809b7e0 con 0x7f9530103320 2024-03-31T23:10:52.176 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-31T23:10:52.176 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":93,"fsid":"6a9d8f08-efb2-11ee-b647-cb9ed24678a4","created":"2024-03-31T23:01:15.463711+0000","modified":"2024-03-31T23:10:10.474156+0000","last_up_change":"2024-03-31T23:08:20.199888+0000","last_in_change":"2024-03-31T23:07:59.438035+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-31T23:04:32.823751+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-31T23:09:09.145711+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-31T23:09:10.148579+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-31T23:09:13.419519+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"91","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"47429095-3f46-4ed7-b7c4-504464719815","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":1824772695},{"type":"v1","addr":"172.21.15.46:6803","nonce":1824772695}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":1824772695},{"type":"v1","addr":"172.21.15.46:6805","nonce":1824772695}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":1824772695},{"type":"v1","addr":"172.21.15.46:6809","nonce":1824772695}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":1824772695},{"type":"v1","addr":"172.21.15.46:6807","nonce":1824772695}]},"public_addr":"172.21.15.46:6803/1824772695","cluster_addr":"172.21.15.46:6805/1824772695","heartbeat_back_addr":"172.21.15.46:6809/1824772695","heartbeat_front_addr":"172.21.15.46:6807/1824772695","state":["exists","up"]},{"osd":1,"uuid":"be56f883-411e-4dbe-b50e-a158134df4a6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6800","nonce":973693922},{"type":"v1","addr":"172.21.15.142:6801","nonce":973693922}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6802","nonce":973693922},{"type":"v1","addr":"172.21.15.142:6803","nonce":973693922}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6806","nonce":973693922},{"type":"v1","addr":"172.21.15.142:6807","nonce":973693922}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6804","nonce":973693922},{"type":"v1","addr":"172.21.15.142:6805","nonce":973693922}]},"public_addr":"172.21.15.142:6801/973693922","cluster_addr":"172.21.15.142:6803/973693922","heartbeat_back_addr":"172.21.15.142:6807/973693922","heartbeat_front_addr":"172.21.15.142:6805/973693922","state":["exists","up"]},{"osd":2,"uuid":"c99a8794-ab19-4260-9a8d-4bc0bd2c8175","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6800","nonce":4084261813},{"type":"v1","addr":"172.21.15.162:6801","nonce":4084261813}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6802","nonce":4084261813},{"type":"v1","addr":"172.21.15.162:6803","nonce":4084261813}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6806","nonce":4084261813},{"type":"v1","addr":"172.21.15.162:6807","nonce":4084261813}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6804","nonce":4084261813},{"type":"v1","addr":"172.21.15.162:6805","nonce":4084261813}]},"public_addr":"172.21.15.162:6801/4084261813","cluster_addr":"172.21.15.162:6803/4084261813","heartbeat_back_addr":"172.21.15.162:6807/4084261813","heartbeat_front_addr":"172.21.15.162:6805/4084261813","state":["exists","up"]},{"osd":3,"uuid":"f877814e-9ef6-4872-a081-5609bb9ebc06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":3496317970},{"type":"v1","addr":"172.21.15.46:6811","nonce":3496317970}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":3496317970},{"type":"v1","addr":"172.21.15.46:6813","nonce":3496317970}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":3496317970},{"type":"v1","addr":"172.21.15.46:6817","nonce":3496317970}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":3496317970},{"type":"v1","addr":"172.21.15.46:6815","nonce":3496317970}]},"public_addr":"172.21.15.46:6811/3496317970","cluster_addr":"172.21.15.46:6813/3496317970","heartbeat_back_addr":"172.21.15.46:6817/3496317970","heartbeat_front_addr":"172.21.15.46:6815/3496317970","state":["exists","up"]},{"osd":4,"uuid":"beecec0a-8280-44fe-8f5f-8498a35eca8f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6808","nonce":2996266929},{"type":"v1","addr":"172.21.15.142:6809","nonce":2996266929}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6810","nonce":2996266929},{"type":"v1","addr":"172.21.15.142:6811","nonce":2996266929}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6814","nonce":2996266929},{"type":"v1","addr":"172.21.15.142:6815","nonce":2996266929}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6812","nonce":2996266929},{"type":"v1","addr":"172.21.15.142:6813","nonce":2996266929}]},"public_addr":"172.21.15.142:6809/2996266929","cluster_addr":"172.21.15.142:6811/2996266929","heartbeat_back_addr":"172.21.15.142:6815/2996266929","heartbeat_front_addr":"172.21.15.142:6813/2996266929","state":["exists","up"]},{"osd":5,"uuid":"9da7ff75-f658-45ea-a02f-1943fdda76b9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6808","nonce":914208293},{"type":"v1","addr":"172.21.15.162:6809","nonce":914208293}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6810","nonce":914208293},{"type":"v1","addr":"172.21.15.162:6811","nonce":914208293}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6814","nonce":914208293},{"type":"v1","addr":"172.21.15.162:6815","nonce":914208293}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6812","nonce":914208293},{"type":"v1","addr":"172.21.15.162:6813","nonce":914208293}]},"public_addr":"172.21.15.162:6809/914208293","cluster_addr":"172.21.15.162:6811/914208293","heartbeat_back_addr":"172.21.15.162:6815/914208293","heartbeat_front_addr":"172.21.15.162:6813/914208293","state":["exists","up"]},{"osd":6,"uuid":"8fa11224-cb26-4d3f-87d1-6a8f31de2a19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":56291966},{"type":"v1","addr":"172.21.15.46:6819","nonce":56291966}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":56291966},{"type":"v1","addr":"172.21.15.46:6821","nonce":56291966}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6824","nonce":56291966},{"type":"v1","addr":"172.21.15.46:6825","nonce":56291966}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":56291966},{"type":"v1","addr":"172.21.15.46:6823","nonce":56291966}]},"public_addr":"172.21.15.46:6819/56291966","cluster_addr":"172.21.15.46:6821/56291966","heartbeat_back_addr":"172.21.15.46:6825/56291966","heartbeat_front_addr":"172.21.15.46:6823/56291966","state":["exists","up"]},{"osd":7,"uuid":"0d0338b7-4d0a-4bb0-b152-8f5224871d9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6816","nonce":2146598446},{"type":"v1","addr":"172.21.15.142:6817","nonce":2146598446}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6818","nonce":2146598446},{"type":"v1","addr":"172.21.15.142:6819","nonce":2146598446}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6822","nonce":2146598446},{"type":"v1","addr":"172.21.15.142:6823","nonce":2146598446}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6820","nonce":2146598446},{"type":"v1","addr":"172.21.15.142:6821","nonce":2146598446}]},"public_addr":"172.21.15.142:6817/2146598446","cluster_addr":"172.21.15.142:6819/2146598446","heartbeat_back_addr":"172.21.15.142:6823/2146598446","heartbeat_front_addr":"172.21.15.142:6821/2146598446","state":["exists","up"]},{"osd":8,"uuid":"8b6f6037-892a-416c-babb-7552648d0b3a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6816","nonce":1173078318},{"type":"v1","addr":"172.21.15.162:6817","nonce":1173078318}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6818","nonce":1173078318},{"type":"v1","addr":"172.21.15.162:6819","nonce":1173078318}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6822","nonce":1173078318},{"type":"v1","addr":"172.21.15.162:6823","nonce":1173078318}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6820","nonce":1173078318},{"type":"v1","addr":"172.21.15.162:6821","nonce":1173078318}]},"public_addr":"172.21.15.162:6817/1173078318","cluster_addr":"172.21.15.162:6819/1173078318","heartbeat_back_addr":"172.21.15.162:6823/1173078318","heartbeat_front_addr":"172.21.15.162:6821/1173078318","state":["exists","up"]},{"osd":9,"uuid":"faccce25-6bf2-4220-8858-cda82b72be68","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":532708881},{"type":"v1","addr":"172.21.15.46:6827","nonce":532708881}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6828","nonce":532708881},{"type":"v1","addr":"172.21.15.46:6829","nonce":532708881}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":532708881},{"type":"v1","addr":"172.21.15.46:6833","nonce":532708881}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":532708881},{"type":"v1","addr":"172.21.15.46:6831","nonce":532708881}]},"public_addr":"172.21.15.46:6827/532708881","cluster_addr":"172.21.15.46:6829/532708881","heartbeat_back_addr":"172.21.15.46:6833/532708881","heartbeat_front_addr":"172.21.15.46:6831/532708881","state":["exists","up"]},{"osd":10,"uuid":"cdd356db-f9eb-4080-95ab-db5d8223e572","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6824","nonce":2845222366},{"type":"v1","addr":"172.21.15.142:6825","nonce":2845222366}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6826","nonce":2845222366},{"type":"v1","addr":"172.21.15.142:6827","nonce":2845222366}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6830","nonce":2845222366},{"type":"v1","addr":"172.21.15.142:6831","nonce":2845222366}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6828","nonce":2845222366},{"type":"v1","addr":"172.21.15.142:6829","nonce":2845222366}]},"public_addr":"172.21.15.142:6825/2845222366","cluster_addr":"172.21.15.142:6827/2845222366","heartbeat_back_addr":"172.21.15.142:6831/2845222366","heartbeat_front_addr":"172.21.15.142:6829/2845222366","state":["exists","up"]},{"osd":11,"uuid":"72104d1c-3ea2-45ad-bc0b-0500f146c06b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6824","nonce":1986323778},{"type":"v1","addr":"172.21.15.162:6825","nonce":1986323778}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6826","nonce":1986323778},{"type":"v1","addr":"172.21.15.162:6827","nonce":1986323778}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6830","nonce":1986323778},{"type":"v1","addr":"172.21.15.162:6831","nonce":1986323778}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6828","nonce":1986323778},{"type":"v1","addr":"172.21.15.162:6829","nonce":1986323778}]},"public_addr":"172.21.15.162:6825/1986323778","cluster_addr":"172.21.15.162:6827/1986323778","heartbeat_back_addr":"172.21.15.162:6831/1986323778","heartbeat_front_addr":"172.21.15.162:6829/1986323778","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:03:44.996331+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:04:07.994738+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:04:30.942255+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:04:55.588884+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:05:19.900600+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:05:43.807442+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:06:09.026976+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:06:34.586784+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:06:59.751605+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:07:26.144495+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:07:52.326909+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:08:17.423042+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.6","mappings":[{"from":4,"to":7}]},{"pgid":"3.c","mappings":[{"from":3,"to":9}]},{"pgid":"3.1a","mappings":[{"from":5,"to":6}]},{"pgid":"3.1b","mappings":[{"from":4,"to":8}]},{"pgid":"3.1d","mappings":[{"from":4,"to":8}]},{"pgid":"3.2a","mappings":[{"from":3,"to":6}]},{"pgid":"3.33","mappings":[{"from":5,"to":9}]},{"pgid":"3.36","mappings":[{"from":4,"to":9}]},{"pgid":"4.17","mappings":[{"from":9,"to":8}]},{"pgid":"4.18","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.46:0/4021039857":"2024-04-01T23:02:08.493619+0000","172.21.15.46:6801/3681182647":"2024-04-01T23:02:08.493619+0000","172.21.15.46:0/4022214806":"2024-04-01T23:02:08.493619+0000","172.21.15.46:6800/1515528215":"2024-04-01T23:01:41.910162+0000","172.21.15.46:6800/3681182647":"2024-04-01T23:02:08.493619+0000","172.21.15.46:0/143885707":"2024-04-01T23:01:41.910162+0000","172.21.15.46:6801/1515528215":"2024-04-01T23:01:41.910162+0000","172.21.15.46:0/1680389778":"2024-04-01T23:01:41.910162+0000","172.21.15.46:0/1782133966":"2024-04-01T23:01:41.910162+0000","172.21.15.46:0/3925209376":"2024-04-01T23:02:08.493619+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-31T23:10:52.178 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:52.177+0000 7f9537765640 1 -- 172.21.15.46:0/2546390590 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f951c0a24a0 msgr2=0x7f951c0a4960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:10:52.179 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:52.177+0000 7f9537765640 1 --2- 172.21.15.46:0/2546390590 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f951c0a24a0 0x7f951c0a4960 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f95301027c0 tx=0x7f9520005f90 comp rx=0 tx=0).stop 2024-03-31T23:10:52.179 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:52.177+0000 7f9537765640 1 -- 172.21.15.46:0/2546390590 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9530103320 msgr2=0x7f9530074080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:10:52.179 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:52.177+0000 7f9537765640 1 --2- 172.21.15.46:0/2546390590 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9530103320 0x7f9530074080 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f952800eb10 tx=0x7f952800c2d0 comp rx=0 tx=0).stop 2024-03-31T23:10:52.179 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:52.177+0000 7f9537765640 1 -- 172.21.15.46:0/2546390590 shutdown_connections 2024-03-31T23:10:52.179 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:52.177+0000 7f9537765640 1 --2- 172.21.15.46:0/2546390590 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f951c0a24a0 0x7f951c0a4960 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:52.179 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:52.177+0000 7f9537765640 1 --2- 172.21.15.46:0/2546390590 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9530103320 0x7f9530074080 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:52.179 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:52.177+0000 7f9537765640 1 --2- 172.21.15.46:0/2546390590 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9530102960 0x7f9530073b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:52.179 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:52.177+0000 7f9537765640 1 --2- 172.21.15.46:0/2546390590 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9530101770 0x7f9530075500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:52.179 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:52.177+0000 7f9537765640 1 -- 172.21.15.46:0/2546390590 >> 172.21.15.46:0/2546390590 conn(0x7f95300fcd00 msgr2=0x7f9530105d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:10:52.179 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:52.177+0000 7f9537765640 1 -- 172.21.15.46:0/2546390590 shutdown_connections 2024-03-31T23:10:52.179 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:52.177+0000 7f9537765640 1 -- 172.21.15.46:0/2546390590 wait complete. 2024-03-31T23:10:52.190 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-31T23:04:32.823751+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-31T23:09:09.145711+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-31T23:09:10.148579+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-31T23:09:13.419519+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '91', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-31T23:10:52.190 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-31T23:10:52.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:52.296+0000 7fe4c230d640 1 -- 172.21.15.46:0/1086836052 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe4bc101840 msgr2=0x7fe4bc101cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:10:52.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:52.296+0000 7fe4c230d640 1 --2- 172.21.15.46:0/1086836052 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe4bc101840 0x7fe4bc101cc0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fe4b0009900 tx=0x7fe4b002bc30 comp rx=0 tx=0).stop 2024-03-31T23:10:52.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:52.296+0000 7fe4c230d640 1 -- 172.21.15.46:0/1086836052 shutdown_connections 2024-03-31T23:10:52.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:52.296+0000 7fe4c230d640 1 --2- 172.21.15.46:0/1086836052 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe4bc102200 0x7fe4bc108a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:52.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:52.296+0000 7fe4c230d640 1 --2- 172.21.15.46:0/1086836052 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe4bc101840 0x7fe4bc101cc0 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:52.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:52.296+0000 7fe4c230d640 1 --2- 172.21.15.46:0/1086836052 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe4bc100650 0x7fe4bc100a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:52.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:52.296+0000 7fe4c230d640 1 -- 172.21.15.46:0/1086836052 >> 172.21.15.46:0/1086836052 conn(0x7fe4bc0fbbe0 msgr2=0x7fe4bc0fe000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:10:52.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:52.296+0000 7fe4c230d640 1 -- 172.21.15.46:0/1086836052 shutdown_connections 2024-03-31T23:10:52.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:52.296+0000 7fe4c230d640 1 -- 172.21.15.46:0/1086836052 wait complete. 2024-03-31T23:10:52.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:52.296+0000 7fe4c230d640 1 Processor -- start 2024-03-31T23:10:52.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:52.297+0000 7fe4c230d640 1 -- start start 2024-03-31T23:10:52.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:52.297+0000 7fe4c230d640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe4bc100650 0x7fe4bc104570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:10:52.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:52.297+0000 7fe4c230d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe4bc101840 0x7fe4bc102c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:10:52.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:52.297+0000 7fe4c230d640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe4bc102200 0x7fe4bc103160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:10:52.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:52.297+0000 7fe4c230d640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fe4bc10c1b0 con 0x7fe4bc101840 2024-03-31T23:10:52.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:52.297+0000 7fe4c230d640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fe4bc10c030 con 0x7fe4bc102200 2024-03-31T23:10:52.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:52.297+0000 7fe4c230d640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fe4bc10c330 con 0x7fe4bc100650 2024-03-31T23:10:52.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:52.297+0000 7fe4bb7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe4bc101840 0x7fe4bc102c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:10:52.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:52.297+0000 7fe4bb7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe4bc101840 0x7fe4bc102c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:56134/0 (socket says 172.21.15.46:56134) 2024-03-31T23:10:52.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:52.297+0000 7fe4bb7fe640 1 -- 172.21.15.46:0/3722506671 learned_addr learned my addr 172.21.15.46:0/3722506671 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:10:52.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:52.297+0000 7fe4bbfff640 1 --2- 172.21.15.46:0/3722506671 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe4bc100650 0x7fe4bc104570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:10:52.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:52.297+0000 7fe4c08aa640 1 --2- 172.21.15.46:0/3722506671 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe4bc102200 0x7fe4bc103160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:10:52.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:52.297+0000 7fe4bb7fe640 1 -- 172.21.15.46:0/3722506671 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe4bc100650 msgr2=0x7fe4bc104570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:10:52.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:52.297+0000 7fe4bb7fe640 1 --2- 172.21.15.46:0/3722506671 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe4bc100650 0x7fe4bc104570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:52.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:52.297+0000 7fe4bb7fe640 1 -- 172.21.15.46:0/3722506671 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe4bc102200 msgr2=0x7fe4bc103160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:10:52.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:52.297+0000 7fe4bb7fe640 1 --2- 172.21.15.46:0/3722506671 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe4bc102200 0x7fe4bc103160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:52.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:52.297+0000 7fe4bb7fe640 1 -- 172.21.15.46:0/3722506671 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe4bc103980 con 0x7fe4bc101840 2024-03-31T23:10:52.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:52.297+0000 7fe4bb7fe640 1 --2- 172.21.15.46:0/3722506671 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe4bc101840 0x7fe4bc102c20 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7fe4b002c840 tx=0x7fe4b0005e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:10:52.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:52.298+0000 7fe4b97fa640 1 -- 172.21.15.46:0/3722506671 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4b0043020 con 0x7fe4bc101840 2024-03-31T23:10:52.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:52.298+0000 7fe4c230d640 1 -- 172.21.15.46:0/3722506671 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe4bc1a1ac0 con 0x7fe4bc101840 2024-03-31T23:10:52.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:52.298+0000 7fe4b97fa640 1 -- 172.21.15.46:0/3722506671 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe4b00043d0 con 0x7fe4bc101840 2024-03-31T23:10:52.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:52.298+0000 7fe4b97fa640 1 -- 172.21.15.46:0/3722506671 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4b0003b10 con 0x7fe4bc101840 2024-03-31T23:10:52.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:52.298+0000 7fe4c230d640 1 -- 172.21.15.46:0/3722506671 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe4bc1a2060 con 0x7fe4bc101840 2024-03-31T23:10:52.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:52.299+0000 7fe4c230d640 1 -- 172.21.15.46:0/3722506671 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe4bc1a22f0 con 0x7fe4bc101840 2024-03-31T23:10:52.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:52.300+0000 7fe4b97fa640 1 -- 172.21.15.46:0/3722506671 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 18) ==== 137320+0+0 (secure 0 0 0) 0x7fe4b0046020 con 0x7fe4bc101840 2024-03-31T23:10:52.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:52.300+0000 7fe4b97fa640 1 --2- 172.21.15.46:0/3722506671 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fe4a40a24a0 0x7fe4a40a4960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:10:52.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:52.300+0000 7fe4bbfff640 1 --2- 172.21.15.46:0/3722506671 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fe4a40a24a0 0x7fe4a40a4960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:10:52.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:52.300+0000 7fe4b97fa640 1 -- 172.21.15.46:0/3722506671 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7fe4b00f0f90 con 0x7fe4bc101840 2024-03-31T23:10:52.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:52.301+0000 7fe4bbfff640 1 --2- 172.21.15.46:0/3722506671 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fe4a40a24a0 0x7fe4a40a4960 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fe4bc1016a0 tx=0x7fe4ac005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:10:52.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:52.304+0000 7fe4b97fa640 1 -- 172.21.15.46:0/3722506671 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fe4b004d020 con 0x7fe4bc101840 2024-03-31T23:10:52.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:52.486+0000 7fe4c230d640 1 -- 172.21.15.46:0/3722506671 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fe4bc1a24c0 con 0x7fe4bc101840 2024-03-31T23:10:52.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:52.486+0000 7fe4b97fa640 1 -- 172.21.15.46:0/3722506671 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v93) ==== 93+0+10 (secure 0 0 0) 0x7fe4b00bddf0 con 0x7fe4bc101840 2024-03-31T23:10:52.487 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 1 2024-03-31T23:10:52.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:52.488+0000 7fe4c230d640 1 -- 172.21.15.46:0/3722506671 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fe4a40a24a0 msgr2=0x7fe4a40a4960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:10:52.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:52.488+0000 7fe4c230d640 1 --2- 172.21.15.46:0/3722506671 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fe4a40a24a0 0x7fe4a40a4960 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fe4bc1016a0 tx=0x7fe4ac005f90 comp rx=0 tx=0).stop 2024-03-31T23:10:52.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:52.488+0000 7fe4c230d640 1 -- 172.21.15.46:0/3722506671 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe4bc101840 msgr2=0x7fe4bc102c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:10:52.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:52.488+0000 7fe4c230d640 1 --2- 172.21.15.46:0/3722506671 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe4bc101840 0x7fe4bc102c20 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7fe4b002c840 tx=0x7fe4b0005e40 comp rx=0 tx=0).stop 2024-03-31T23:10:52.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:52.488+0000 7fe4c230d640 1 -- 172.21.15.46:0/3722506671 shutdown_connections 2024-03-31T23:10:52.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:52.488+0000 7fe4c230d640 1 --2- 172.21.15.46:0/3722506671 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fe4a40a24a0 0x7fe4a40a4960 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:52.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:52.488+0000 7fe4c230d640 1 --2- 172.21.15.46:0/3722506671 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe4bc102200 0x7fe4bc103160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:52.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:52.488+0000 7fe4c230d640 1 --2- 172.21.15.46:0/3722506671 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe4bc101840 0x7fe4bc102c20 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:52.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:52.488+0000 7fe4c230d640 1 --2- 172.21.15.46:0/3722506671 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe4bc100650 0x7fe4bc104570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:52.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:52.488+0000 7fe4c230d640 1 -- 172.21.15.46:0/3722506671 >> 172.21.15.46:0/3722506671 conn(0x7fe4bc0fbbe0 msgr2=0x7fe4bc0fd690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:10:52.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:52.488+0000 7fe4c230d640 1 -- 172.21.15.46:0/3722506671 shutdown_connections 2024-03-31T23:10:52.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:52.488+0000 7fe4c230d640 1 -- 172.21.15.46:0/3722506671 wait complete. 2024-03-31T23:10:52.501 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-31T23:10:52.607 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:52.606+0000 7f8138b6c640 1 -- 172.21.15.46:0/1788203716 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8134102220 msgr2=0x7f8134108ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:10:52.607 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:52.606+0000 7f8138b6c640 1 --2- 172.21.15.46:0/1788203716 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8134102220 0x7f8134108ab0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f812c009ca0 tx=0x7f812c02c0f0 comp rx=0 tx=0).stop 2024-03-31T23:10:52.607 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:52.606+0000 7f8138b6c640 1 -- 172.21.15.46:0/1788203716 shutdown_connections 2024-03-31T23:10:52.607 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:52.606+0000 7f8138b6c640 1 --2- 172.21.15.46:0/1788203716 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8134102220 0x7f8134108ab0 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:52.607 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:52.606+0000 7f8138b6c640 1 --2- 172.21.15.46:0/1788203716 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8134101860 0x7f8134101ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:52.608 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:52.606+0000 7f8138b6c640 1 --2- 172.21.15.46:0/1788203716 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8134100670 0x7f8134100a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:52.608 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:52.606+0000 7f8138b6c640 1 -- 172.21.15.46:0/1788203716 >> 172.21.15.46:0/1788203716 conn(0x7f81340fbc00 msgr2=0x7f81340fe020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:10:52.608 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:52.606+0000 7f8138b6c640 1 -- 172.21.15.46:0/1788203716 shutdown_connections 2024-03-31T23:10:52.608 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:52.606+0000 7f8138b6c640 1 -- 172.21.15.46:0/1788203716 wait complete. 2024-03-31T23:10:52.610 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:52.607+0000 7f8138b6c640 1 Processor -- start 2024-03-31T23:10:52.610 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:52.607+0000 7f8138b6c640 1 -- start start 2024-03-31T23:10:52.610 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:52.607+0000 7f8138b6c640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8134100670 0x7f8134197950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:10:52.610 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:52.607+0000 7f8138b6c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8134101860 0x7f8134197e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:10:52.610 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:52.607+0000 7f8138b6c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8134102220 0x7f813419ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:10:52.610 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:52.607+0000 7f8138b6c640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f813410c180 con 0x7f8134102220 2024-03-31T23:10:52.610 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:52.607+0000 7f8138b6c640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f813410c000 con 0x7f8134100670 2024-03-31T23:10:52.610 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:52.607+0000 7f8138b6c640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f813410c300 con 0x7f8134101860 2024-03-31T23:10:52.610 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:52.607+0000 7f8132d9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8134102220 0x7f813419ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:10:52.610 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:52.607+0000 7f8132d9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8134102220 0x7f813419ef00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:56138/0 (socket says 172.21.15.46:56138) 2024-03-31T23:10:52.611 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:52.607+0000 7f8132d9d640 1 -- 172.21.15.46:0/3151056551 learned_addr learned my addr 172.21.15.46:0/3151056551 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:10:52.611 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:52.607+0000 7f813259c640 1 --2- 172.21.15.46:0/3151056551 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8134100670 0x7f8134197950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:10:52.611 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:52.607+0000 7f8131d9b640 1 --2- 172.21.15.46:0/3151056551 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8134101860 0x7f8134197e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:10:52.611 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:52.607+0000 7f8132d9d640 1 -- 172.21.15.46:0/3151056551 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8134101860 msgr2=0x7f8134197e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:10:52.611 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:52.607+0000 7f8132d9d640 1 --2- 172.21.15.46:0/3151056551 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8134101860 0x7f8134197e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:52.611 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:52.607+0000 7f8132d9d640 1 -- 172.21.15.46:0/3151056551 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8134100670 msgr2=0x7f8134197950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:10:52.611 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:52.607+0000 7f8132d9d640 1 --2- 172.21.15.46:0/3151056551 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8134100670 0x7f8134197950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:52.611 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:52.607+0000 7f8132d9d640 1 -- 172.21.15.46:0/3151056551 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f813419f570 con 0x7f8134102220 2024-03-31T23:10:52.611 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:52.608+0000 7f8132d9d640 1 --2- 172.21.15.46:0/3151056551 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8134102220 0x7f813419ef00 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f812c02c700 tx=0x7f812c0328a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:10:52.611 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:52.608+0000 7f81237fe640 1 -- 172.21.15.46:0/3151056551 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f812c041020 con 0x7f8134102220 2024-03-31T23:10:52.611 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:52.608+0000 7f81237fe640 1 -- 172.21.15.46:0/3151056551 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f812c003e20 con 0x7f8134102220 2024-03-31T23:10:52.611 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:52.608+0000 7f81237fe640 1 -- 172.21.15.46:0/3151056551 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f812c039040 con 0x7f8134102220 2024-03-31T23:10:52.611 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:52.608+0000 7f8138b6c640 1 -- 172.21.15.46:0/3151056551 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f813419f800 con 0x7f8134102220 2024-03-31T23:10:52.611 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:52.608+0000 7f8138b6c640 1 -- 172.21.15.46:0/3151056551 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f813419fd40 con 0x7f8134102220 2024-03-31T23:10:52.611 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:52.609+0000 7f8138b6c640 1 -- 172.21.15.46:0/3151056551 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f813419ffa0 con 0x7f8134102220 2024-03-31T23:10:52.612 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:52.610+0000 7f81237fe640 1 -- 172.21.15.46:0/3151056551 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 18) ==== 137320+0+0 (secure 0 0 0) 0x7f812c044020 con 0x7f8134102220 2024-03-31T23:10:52.612 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:52.610+0000 7f81237fe640 1 --2- 172.21.15.46:0/3151056551 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f811c0a2520 0x7f811c0a49e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:10:52.612 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:52.610+0000 7f813259c640 1 --2- 172.21.15.46:0/3151056551 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f811c0a2520 0x7f811c0a49e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:10:52.612 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:52.610+0000 7f81237fe640 1 -- 172.21.15.46:0/3151056551 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7f812c0f0c00 con 0x7f8134102220 2024-03-31T23:10:52.612 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:52.611+0000 7f813259c640 1 --2- 172.21.15.46:0/3151056551 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f811c0a2520 0x7f811c0a49e0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f81341016c0 tx=0x7f8124005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:10:52.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:52.615+0000 7f81237fe640 1 -- 172.21.15.46:0/3151056551 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f812c04b020 con 0x7f8134102220 2024-03-31T23:10:52.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:10:52 smithi162 ceph-mon[36237]: pgmap v344: 129 pgs: 129 active+clean; 642 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 597 B/s wr, 7 op/s 2024-03-31T23:10:52.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:10:52 smithi162 ceph-mon[36237]: from='client.? 172.21.15.46:0/2546390590' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-31T23:10:52.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:10:52 smithi162 ceph-mon[36237]: from='client.? 172.21.15.46:0/3722506671' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-31T23:10:52.797 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:52.796+0000 7f8138b6c640 1 -- 172.21.15.46:0/3151056551 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f81341a0170 con 0x7f8134102220 2024-03-31T23:10:52.797 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:52.796+0000 7f81237fe640 1 -- 172.21.15.46:0/3151056551 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v93) ==== 104+0+11 (secure 0 0 0) 0x7f812c0bda00 con 0x7f8134102220 2024-03-31T23:10:52.797 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 32 2024-03-31T23:10:52.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:52.798+0000 7f8138b6c640 1 -- 172.21.15.46:0/3151056551 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f811c0a2520 msgr2=0x7f811c0a49e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:10:52.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:52.798+0000 7f8138b6c640 1 --2- 172.21.15.46:0/3151056551 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f811c0a2520 0x7f811c0a49e0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f81341016c0 tx=0x7f8124005f90 comp rx=0 tx=0).stop 2024-03-31T23:10:52.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:52.798+0000 7f8138b6c640 1 -- 172.21.15.46:0/3151056551 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8134102220 msgr2=0x7f813419ef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:10:52.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:52.798+0000 7f8138b6c640 1 --2- 172.21.15.46:0/3151056551 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8134102220 0x7f813419ef00 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f812c02c700 tx=0x7f812c0328a0 comp rx=0 tx=0).stop 2024-03-31T23:10:52.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:52.798+0000 7f8138b6c640 1 -- 172.21.15.46:0/3151056551 shutdown_connections 2024-03-31T23:10:52.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:52.798+0000 7f8138b6c640 1 --2- 172.21.15.46:0/3151056551 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f811c0a2520 0x7f811c0a49e0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:52.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:52.798+0000 7f8138b6c640 1 --2- 172.21.15.46:0/3151056551 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8134102220 0x7f813419ef00 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:52.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:52.798+0000 7f8138b6c640 1 --2- 172.21.15.46:0/3151056551 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8134101860 0x7f8134197e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:52.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:52.798+0000 7f8138b6c640 1 --2- 172.21.15.46:0/3151056551 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8134100670 0x7f8134197950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:52.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:52.798+0000 7f8138b6c640 1 -- 172.21.15.46:0/3151056551 >> 172.21.15.46:0/3151056551 conn(0x7f81340fbc00 msgr2=0x7f81340fd690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:10:52.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:52.798+0000 7f8138b6c640 1 -- 172.21.15.46:0/3151056551 shutdown_connections 2024-03-31T23:10:52.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:52.798+0000 7f8138b6c640 1 -- 172.21.15.46:0/3151056551 wait complete. 2024-03-31T23:10:52.809 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:10:52 smithi046 ceph-mon[32498]: pgmap v344: 129 pgs: 129 active+clean; 642 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 597 B/s wr, 7 op/s 2024-03-31T23:10:52.809 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:10:52 smithi046 ceph-mon[32498]: from='client.? 172.21.15.46:0/2546390590' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-31T23:10:52.809 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:10:52 smithi046 ceph-mon[32498]: from='client.? 172.21.15.46:0/3722506671' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-31T23:10:52.810 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-31T23:10:52.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:52.915+0000 7f0c6cffd640 1 -- 172.21.15.46:0/435390640 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0c68102960 msgr2=0x7f0c68102de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:10:52.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:52.915+0000 7f0c6cffd640 1 --2- 172.21.15.46:0/435390640 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0c68102960 0x7f0c68102de0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f0c50009900 tx=0x7f0c5002bc30 comp rx=0 tx=0).stop 2024-03-31T23:10:52.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:52.915+0000 7f0c6cffd640 1 -- 172.21.15.46:0/435390640 shutdown_connections 2024-03-31T23:10:52.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:52.915+0000 7f0c6cffd640 1 --2- 172.21.15.46:0/435390640 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0c68103320 0x7f0c68109bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:52.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:52.915+0000 7f0c6cffd640 1 --2- 172.21.15.46:0/435390640 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0c68102960 0x7f0c68102de0 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:52.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:52.915+0000 7f0c6cffd640 1 --2- 172.21.15.46:0/435390640 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0c68101770 0x7f0c68101b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:52.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:52.915+0000 7f0c6cffd640 1 -- 172.21.15.46:0/435390640 >> 172.21.15.46:0/435390640 conn(0x7f0c680fcd00 msgr2=0x7f0c680ff120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:10:52.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:52.915+0000 7f0c6cffd640 1 -- 172.21.15.46:0/435390640 shutdown_connections 2024-03-31T23:10:52.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:52.915+0000 7f0c6cffd640 1 -- 172.21.15.46:0/435390640 wait complete. 2024-03-31T23:10:52.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:52.916+0000 7f0c6cffd640 1 Processor -- start 2024-03-31T23:10:52.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:52.916+0000 7f0c6cffd640 1 -- start start 2024-03-31T23:10:52.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:52.916+0000 7f0c6cffd640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0c68101770 0x7f0c68197940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:10:52.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:52.916+0000 7f0c6cffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0c68102960 0x7f0c68197e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:10:52.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:52.916+0000 7f0c6cffd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0c68103320 0x7f0c6819eef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:10:52.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:52.916+0000 7f0c6cffd640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f0c6810d2f0 con 0x7f0c68102960 2024-03-31T23:10:52.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:52.916+0000 7f0c6cffd640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f0c6810d170 con 0x7f0c68101770 2024-03-31T23:10:52.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:52.916+0000 7f0c6cffd640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f0c6810d470 con 0x7f0c68103320 2024-03-31T23:10:52.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:52.916+0000 7f0c5ffff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0c68102960 0x7f0c68197e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:10:52.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:52.916+0000 7f0c5ffff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0c68102960 0x7f0c68197e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:56148/0 (socket says 172.21.15.46:56148) 2024-03-31T23:10:52.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:52.916+0000 7f0c5ffff640 1 -- 172.21.15.46:0/562315376 learned_addr learned my addr 172.21.15.46:0/562315376 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:10:52.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:52.916+0000 7f0c66d9d640 1 --2- 172.21.15.46:0/562315376 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0c68103320 0x7f0c6819eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:10:52.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:52.916+0000 7f0c6659c640 1 --2- 172.21.15.46:0/562315376 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0c68101770 0x7f0c68197940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:10:52.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:52.917+0000 7f0c5ffff640 1 -- 172.21.15.46:0/562315376 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0c68103320 msgr2=0x7f0c6819eef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:10:52.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:52.917+0000 7f0c5ffff640 1 --2- 172.21.15.46:0/562315376 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0c68103320 0x7f0c6819eef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:52.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:52.917+0000 7f0c5ffff640 1 -- 172.21.15.46:0/562315376 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0c68101770 msgr2=0x7f0c68197940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:10:52.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:52.917+0000 7f0c5ffff640 1 --2- 172.21.15.46:0/562315376 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0c68101770 0x7f0c68197940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:52.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:52.917+0000 7f0c5ffff640 1 -- 172.21.15.46:0/562315376 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0c6819f560 con 0x7f0c68102960 2024-03-31T23:10:52.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:52.917+0000 7f0c5ffff640 1 --2- 172.21.15.46:0/562315376 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0c68102960 0x7f0c68197e80 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f0c50000c00 tx=0x7f0c500040c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:10:52.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:52.917+0000 7f0c5f7fe640 1 -- 172.21.15.46:0/562315376 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c50043020 con 0x7f0c68102960 2024-03-31T23:10:52.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:52.917+0000 7f0c5f7fe640 1 -- 172.21.15.46:0/562315376 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0c500044c0 con 0x7f0c68102960 2024-03-31T23:10:52.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:52.917+0000 7f0c6cffd640 1 -- 172.21.15.46:0/562315376 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0c6819f7f0 con 0x7f0c68102960 2024-03-31T23:10:52.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:52.917+0000 7f0c5f7fe640 1 -- 172.21.15.46:0/562315376 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c50003c30 con 0x7f0c68102960 2024-03-31T23:10:52.921 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:52.917+0000 7f0c6cffd640 1 -- 172.21.15.46:0/562315376 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0c6819fd90 con 0x7f0c68102960 2024-03-31T23:10:52.921 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:52.918+0000 7f0c6cffd640 1 -- 172.21.15.46:0/562315376 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0c68108a90 con 0x7f0c68102960 2024-03-31T23:10:52.921 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:52.920+0000 7f0c5f7fe640 1 -- 172.21.15.46:0/562315376 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 18) ==== 137320+0+0 (secure 0 0 0) 0x7f0c50046020 con 0x7f0c68102960 2024-03-31T23:10:52.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:52.920+0000 7f0c5f7fe640 1 --2- 172.21.15.46:0/562315376 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f0c540a2520 0x7f0c540a49e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:10:52.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:52.920+0000 7f0c6659c640 1 --2- 172.21.15.46:0/562315376 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f0c540a2520 0x7f0c540a49e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:10:52.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:52.920+0000 7f0c5f7fe640 1 -- 172.21.15.46:0/562315376 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7f0c500f0fd0 con 0x7f0c68102960 2024-03-31T23:10:52.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:52.921+0000 7f0c6659c640 1 --2- 172.21.15.46:0/562315376 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f0c540a2520 0x7f0c540a49e0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f0c580095f0 tx=0x7f0c58005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:10:52.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:52.924+0000 7f0c5f7fe640 1 -- 172.21.15.46:0/562315376 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f0c5004d020 con 0x7f0c68102960 2024-03-31T23:10:52.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:10:52 smithi142 ceph-mon[35271]: pgmap v344: 129 pgs: 129 active+clean; 642 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 597 B/s wr, 7 op/s 2024-03-31T23:10:52.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:10:52 smithi142 ceph-mon[35271]: from='client.? 172.21.15.46:0/2546390590' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-31T23:10:52.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:10:52 smithi142 ceph-mon[35271]: from='client.? 172.21.15.46:0/3722506671' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-31T23:10:53.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:53.105+0000 7f0c6cffd640 1 -- 172.21.15.46:0/562315376 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f0c6802d300 con 0x7f0c68102960 2024-03-31T23:10:53.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:53.105+0000 7f0c5f7fe640 1 -- 172.21.15.46:0/562315376 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v93) ==== 100+0+11 (secure 0 0 0) 0x7f0c500bdf50 con 0x7f0c68102960 2024-03-31T23:10:53.106 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 64 2024-03-31T23:10:53.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:53.107+0000 7f0c6cffd640 1 -- 172.21.15.46:0/562315376 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f0c540a2520 msgr2=0x7f0c540a49e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:10:53.109 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:53.107+0000 7f0c6cffd640 1 --2- 172.21.15.46:0/562315376 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f0c540a2520 0x7f0c540a49e0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f0c580095f0 tx=0x7f0c58005f90 comp rx=0 tx=0).stop 2024-03-31T23:10:53.109 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:53.107+0000 7f0c6cffd640 1 -- 172.21.15.46:0/562315376 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0c68102960 msgr2=0x7f0c68197e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:10:53.109 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:53.107+0000 7f0c6cffd640 1 --2- 172.21.15.46:0/562315376 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0c68102960 0x7f0c68197e80 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f0c50000c00 tx=0x7f0c500040c0 comp rx=0 tx=0).stop 2024-03-31T23:10:53.109 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:53.107+0000 7f0c6cffd640 1 -- 172.21.15.46:0/562315376 shutdown_connections 2024-03-31T23:10:53.109 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:53.107+0000 7f0c6cffd640 1 --2- 172.21.15.46:0/562315376 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f0c540a2520 0x7f0c540a49e0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:53.109 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:53.107+0000 7f0c6cffd640 1 --2- 172.21.15.46:0/562315376 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0c68103320 0x7f0c6819eef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:53.109 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:53.107+0000 7f0c6cffd640 1 --2- 172.21.15.46:0/562315376 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0c68102960 0x7f0c68197e80 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:53.109 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:53.107+0000 7f0c6cffd640 1 --2- 172.21.15.46:0/562315376 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0c68101770 0x7f0c68197940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:53.109 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:53.107+0000 7f0c6cffd640 1 -- 172.21.15.46:0/562315376 >> 172.21.15.46:0/562315376 conn(0x7f0c680fcd00 msgr2=0x7f0c68105c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:10:53.109 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:53.108+0000 7f0c6cffd640 1 -- 172.21.15.46:0/562315376 shutdown_connections 2024-03-31T23:10:53.109 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:53.108+0000 7f0c6cffd640 1 -- 172.21.15.46:0/562315376 wait complete. 2024-03-31T23:10:53.119 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-31T23:10:53.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:53.224+0000 7fd673d65640 1 -- 172.21.15.46:0/3370372472 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd66c104b00 msgr2=0x7fd66c106ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:10:53.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:53.224+0000 7fd673d65640 1 --2- 172.21.15.46:0/3370372472 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd66c104b00 0x7fd66c106ef0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fd660009900 tx=0x7fd66002bc30 comp rx=0 tx=0).stop 2024-03-31T23:10:53.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:53.224+0000 7fd673d65640 1 -- 172.21.15.46:0/3370372472 shutdown_connections 2024-03-31T23:10:53.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:53.224+0000 7fd673d65640 1 --2- 172.21.15.46:0/3370372472 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd66c107430 0x7fd66c109880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:53.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:53.224+0000 7fd673d65640 1 --2- 172.21.15.46:0/3370372472 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd66c104b00 0x7fd66c106ef0 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:53.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:53.224+0000 7fd673d65640 1 --2- 172.21.15.46:0/3370372472 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd66c1021d0 0x7fd66c1045c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:53.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:53.224+0000 7fd673d65640 1 -- 172.21.15.46:0/3370372472 >> 172.21.15.46:0/3370372472 conn(0x7fd66c0fbbe0 msgr2=0x7fd66c0fe000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:10:53.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:53.224+0000 7fd673d65640 1 -- 172.21.15.46:0/3370372472 shutdown_connections 2024-03-31T23:10:53.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:53.224+0000 7fd673d65640 1 -- 172.21.15.46:0/3370372472 wait complete. 2024-03-31T23:10:53.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:53.225+0000 7fd673d65640 1 Processor -- start 2024-03-31T23:10:53.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:53.225+0000 7fd673d65640 1 -- start start 2024-03-31T23:10:53.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:53.225+0000 7fd673d65640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd66c1021d0 0x7fd66c1979f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:10:53.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:53.225+0000 7fd673d65640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd66c104b00 0x7fd66c197f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:10:53.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:53.225+0000 7fd673d65640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd66c107430 0x7fd66c19efa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:10:53.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:53.225+0000 7fd673d65640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fd66c10d270 con 0x7fd66c104b00 2024-03-31T23:10:53.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:53.225+0000 7fd673d65640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fd66c10d0f0 con 0x7fd66c107430 2024-03-31T23:10:53.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:53.225+0000 7fd673d65640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fd66c10d3f0 con 0x7fd66c1021d0 2024-03-31T23:10:53.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:53.225+0000 7fd671300640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd66c104b00 0x7fd66c197f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:10:53.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:53.225+0000 7fd671300640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd66c104b00 0x7fd66c197f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:56154/0 (socket says 172.21.15.46:56154) 2024-03-31T23:10:53.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:53.225+0000 7fd671300640 1 -- 172.21.15.46:0/2863869923 learned_addr learned my addr 172.21.15.46:0/2863869923 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:10:53.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:53.225+0000 7fd671b01640 1 --2- 172.21.15.46:0/2863869923 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd66c1021d0 0x7fd66c1979f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:10:53.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:53.225+0000 7fd672302640 1 --2- 172.21.15.46:0/2863869923 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd66c107430 0x7fd66c19efa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:10:53.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:53.226+0000 7fd671300640 1 -- 172.21.15.46:0/2863869923 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd66c1021d0 msgr2=0x7fd66c1979f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:10:53.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:53.226+0000 7fd671300640 1 --2- 172.21.15.46:0/2863869923 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd66c1021d0 0x7fd66c1979f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:53.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:53.226+0000 7fd671300640 1 -- 172.21.15.46:0/2863869923 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd66c107430 msgr2=0x7fd66c19efa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:10:53.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:53.226+0000 7fd671300640 1 --2- 172.21.15.46:0/2863869923 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd66c107430 0x7fd66c19efa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:53.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:53.226+0000 7fd671300640 1 -- 172.21.15.46:0/2863869923 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd66c19f610 con 0x7fd66c104b00 2024-03-31T23:10:53.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:53.226+0000 7fd672302640 1 --2- 172.21.15.46:0/2863869923 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd66c107430 0x7fd66c19efa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:10:53.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:53.226+0000 7fd671300640 1 --2- 172.21.15.46:0/2863869923 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd66c104b00 0x7fd66c197f30 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fd660000c00 tx=0x7fd660005dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:10:53.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:53.226+0000 7fd66affd640 1 -- 172.21.15.46:0/2863869923 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd660043020 con 0x7fd66c104b00 2024-03-31T23:10:53.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:53.226+0000 7fd673d65640 1 -- 172.21.15.46:0/2863869923 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd66c19f8a0 con 0x7fd66c104b00 2024-03-31T23:10:53.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:53.226+0000 7fd66affd640 1 -- 172.21.15.46:0/2863869923 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd660004350 con 0x7fd66c104b00 2024-03-31T23:10:53.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:53.226+0000 7fd66affd640 1 -- 172.21.15.46:0/2863869923 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd660003b10 con 0x7fd66c104b00 2024-03-31T23:10:53.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:53.226+0000 7fd673d65640 1 -- 172.21.15.46:0/2863869923 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd66c19fdf0 con 0x7fd66c104b00 2024-03-31T23:10:53.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:53.227+0000 7fd673d65640 1 -- 172.21.15.46:0/2863869923 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd66c0fd6d0 con 0x7fd66c104b00 2024-03-31T23:10:53.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:53.228+0000 7fd66affd640 1 -- 172.21.15.46:0/2863869923 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 18) ==== 137320+0+0 (secure 0 0 0) 0x7fd660046020 con 0x7fd66c104b00 2024-03-31T23:10:53.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:53.229+0000 7fd66affd640 1 --2- 172.21.15.46:0/2863869923 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fd6540a24a0 0x7fd6540a4960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:10:53.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:53.229+0000 7fd671b01640 1 --2- 172.21.15.46:0/2863869923 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fd6540a24a0 0x7fd6540a4960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:10:53.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:53.229+0000 7fd66affd640 1 -- 172.21.15.46:0/2863869923 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7fd6600f0b80 con 0x7fd66c104b00 2024-03-31T23:10:53.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:53.230+0000 7fd671b01640 1 --2- 172.21.15.46:0/2863869923 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fd6540a24a0 0x7fd6540a4960 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fd664009bd0 tx=0x7fd664005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:10:53.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:53.234+0000 7fd66affd640 1 -- 172.21.15.46:0/2863869923 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fd66004d020 con 0x7fd66c104b00 2024-03-31T23:10:53.417 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:53.416+0000 7fd673d65640 1 -- 172.21.15.46:0/2863869923 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7fd66c062350 con 0x7fd66c104b00 2024-03-31T23:10:53.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:53.417+0000 7fd66affd640 1 -- 172.21.15.46:0/2863869923 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v93) ==== 103+0+11 (secure 0 0 0) 0x7fd6600bd9e0 con 0x7fd66c104b00 2024-03-31T23:10:53.418 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 32 2024-03-31T23:10:53.420 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:53.419+0000 7fd673d65640 1 -- 172.21.15.46:0/2863869923 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fd6540a24a0 msgr2=0x7fd6540a4960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:10:53.420 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:53.419+0000 7fd673d65640 1 --2- 172.21.15.46:0/2863869923 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fd6540a24a0 0x7fd6540a4960 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fd664009bd0 tx=0x7fd664005f90 comp rx=0 tx=0).stop 2024-03-31T23:10:53.420 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:53.419+0000 7fd673d65640 1 -- 172.21.15.46:0/2863869923 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd66c104b00 msgr2=0x7fd66c197f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:10:53.420 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:53.419+0000 7fd673d65640 1 --2- 172.21.15.46:0/2863869923 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd66c104b00 0x7fd66c197f30 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fd660000c00 tx=0x7fd660005dc0 comp rx=0 tx=0).stop 2024-03-31T23:10:53.420 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:53.419+0000 7fd673d65640 1 -- 172.21.15.46:0/2863869923 shutdown_connections 2024-03-31T23:10:53.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:53.419+0000 7fd673d65640 1 --2- 172.21.15.46:0/2863869923 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fd6540a24a0 0x7fd6540a4960 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:53.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:53.419+0000 7fd673d65640 1 --2- 172.21.15.46:0/2863869923 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd66c107430 0x7fd66c19efa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:53.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:53.419+0000 7fd673d65640 1 --2- 172.21.15.46:0/2863869923 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd66c104b00 0x7fd66c197f30 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:53.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:53.419+0000 7fd673d65640 1 --2- 172.21.15.46:0/2863869923 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd66c1021d0 0x7fd66c1979f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:53.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:53.419+0000 7fd673d65640 1 -- 172.21.15.46:0/2863869923 >> 172.21.15.46:0/2863869923 conn(0x7fd66c0fbbe0 msgr2=0x7fd66c102bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:10:53.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:53.419+0000 7fd673d65640 1 -- 172.21.15.46:0/2863869923 shutdown_connections 2024-03-31T23:10:53.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:53.419+0000 7fd673d65640 1 -- 172.21.15.46:0/2863869923 wait complete. 2024-03-31T23:10:53.431 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-31T23:10:53.538 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:53.535+0000 7f4bad6ae640 1 -- 172.21.15.46:0/2845435512 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4ba8101770 msgr2=0x7f4ba8101b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:10:53.538 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:53.535+0000 7f4bad6ae640 1 --2- 172.21.15.46:0/2845435512 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4ba8101770 0x7f4ba8101b90 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f4ba0009ca0 tx=0x7f4ba002c0f0 comp rx=0 tx=0).stop 2024-03-31T23:10:53.538 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:53.535+0000 7f4bad6ae640 1 -- 172.21.15.46:0/2845435512 shutdown_connections 2024-03-31T23:10:53.538 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:53.535+0000 7f4bad6ae640 1 --2- 172.21.15.46:0/2845435512 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4ba8103320 0x7f4ba8109bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:53.539 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:53.535+0000 7f4bad6ae640 1 --2- 172.21.15.46:0/2845435512 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4ba8102960 0x7f4ba8102de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:53.539 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:53.535+0000 7f4bad6ae640 1 --2- 172.21.15.46:0/2845435512 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4ba8101770 0x7f4ba8101b90 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:53.539 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:53.535+0000 7f4bad6ae640 1 -- 172.21.15.46:0/2845435512 >> 172.21.15.46:0/2845435512 conn(0x7f4ba80fcd00 msgr2=0x7f4ba80ff120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:10:53.539 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:53.535+0000 7f4bad6ae640 1 -- 172.21.15.46:0/2845435512 shutdown_connections 2024-03-31T23:10:53.539 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:53.535+0000 7f4bad6ae640 1 -- 172.21.15.46:0/2845435512 wait complete. 2024-03-31T23:10:53.539 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:53.536+0000 7f4bad6ae640 1 Processor -- start 2024-03-31T23:10:53.539 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:53.536+0000 7f4bad6ae640 1 -- start start 2024-03-31T23:10:53.539 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:53.536+0000 7f4bad6ae640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4ba8101770 0x7f4ba81979a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:10:53.539 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:53.536+0000 7f4bad6ae640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4ba8102960 0x7f4ba8197ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:10:53.539 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:53.536+0000 7f4bad6ae640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4ba8103320 0x7f4ba819ef50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:10:53.539 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:53.536+0000 7f4bad6ae640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f4ba810d2a0 con 0x7f4ba8101770 2024-03-31T23:10:53.539 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:53.536+0000 7f4bad6ae640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f4ba810d120 con 0x7f4ba8102960 2024-03-31T23:10:53.539 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:53.536+0000 7f4bad6ae640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f4ba810d420 con 0x7f4ba8103320 2024-03-31T23:10:53.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:53.536+0000 7f4ba6ffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4ba8101770 0x7f4ba81979a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:10:53.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:53.536+0000 7f4ba6ffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4ba8101770 0x7f4ba81979a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:56180/0 (socket says 172.21.15.46:56180) 2024-03-31T23:10:53.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:53.536+0000 7f4ba6ffd640 1 -- 172.21.15.46:0/2831969779 learned_addr learned my addr 172.21.15.46:0/2831969779 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:10:53.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:53.536+0000 7f4ba67fc640 1 --2- 172.21.15.46:0/2831969779 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4ba8102960 0x7f4ba8197ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:10:53.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:53.536+0000 7f4ba77fe640 1 --2- 172.21.15.46:0/2831969779 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4ba8103320 0x7f4ba819ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:10:53.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:53.536+0000 7f4ba6ffd640 1 -- 172.21.15.46:0/2831969779 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4ba8103320 msgr2=0x7f4ba819ef50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:10:53.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:53.536+0000 7f4ba6ffd640 1 --2- 172.21.15.46:0/2831969779 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4ba8103320 0x7f4ba819ef50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:53.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:53.536+0000 7f4ba6ffd640 1 -- 172.21.15.46:0/2831969779 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4ba8102960 msgr2=0x7f4ba8197ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:10:53.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:53.536+0000 7f4ba6ffd640 1 --2- 172.21.15.46:0/2831969779 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4ba8102960 0x7f4ba8197ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:53.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:53.536+0000 7f4ba6ffd640 1 -- 172.21.15.46:0/2831969779 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ba819f5c0 con 0x7f4ba8101770 2024-03-31T23:10:53.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:53.537+0000 7f4ba6ffd640 1 --2- 172.21.15.46:0/2831969779 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4ba8101770 0x7f4ba81979a0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f4ba0000f80 tx=0x7f4ba00328a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:10:53.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:53.537+0000 7f4b8ffff640 1 -- 172.21.15.46:0/2831969779 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ba0041020 con 0x7f4ba8101770 2024-03-31T23:10:53.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:53.537+0000 7f4b8ffff640 1 -- 172.21.15.46:0/2831969779 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4ba0034720 con 0x7f4ba8101770 2024-03-31T23:10:53.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:53.537+0000 7f4bad6ae640 1 -- 172.21.15.46:0/2831969779 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4ba819f850 con 0x7f4ba8101770 2024-03-31T23:10:53.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:53.537+0000 7f4b8ffff640 1 -- 172.21.15.46:0/2831969779 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ba0039040 con 0x7f4ba8101770 2024-03-31T23:10:53.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:53.537+0000 7f4bad6ae640 1 -- 172.21.15.46:0/2831969779 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4ba819fd40 con 0x7f4ba8101770 2024-03-31T23:10:53.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:53.538+0000 7f4bad6ae640 1 -- 172.21.15.46:0/2831969779 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4ba8108ae0 con 0x7f4ba8101770 2024-03-31T23:10:53.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:53.539+0000 7f4b8ffff640 1 -- 172.21.15.46:0/2831969779 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 18) ==== 137320+0+0 (secure 0 0 0) 0x7f4ba0044020 con 0x7f4ba8101770 2024-03-31T23:10:53.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:53.539+0000 7f4b8ffff640 1 --2- 172.21.15.46:0/2831969779 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f4b900a24a0 0x7f4b900a4960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:10:53.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:53.539+0000 7f4ba67fc640 1 --2- 172.21.15.46:0/2831969779 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f4b900a24a0 0x7f4b900a4960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:10:53.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:53.540+0000 7f4b8ffff640 1 -- 172.21.15.46:0/2831969779 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7f4ba00f0ac0 con 0x7f4ba8101770 2024-03-31T23:10:53.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:53.540+0000 7f4ba67fc640 1 --2- 172.21.15.46:0/2831969779 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f4b900a24a0 0x7f4b900a4960 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f4ba8198eb0 tx=0x7f4b9c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:10:53.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:53.544+0000 7f4b8ffff640 1 -- 172.21.15.46:0/2831969779 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f4ba004b020 con 0x7f4ba8101770 2024-03-31T23:10:53.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:53.766+0000 7f4bad6ae640 1 -- 172.21.15.46:0/2831969779 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4ba802d170 con 0x7f4ba8101770 2024-03-31T23:10:53.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:53.766+0000 7f4b8ffff640 1 -- 172.21.15.46:0/2831969779 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+11882 (secure 0 0 0) 0x7f4ba00bd920 con 0x7f4ba8101770 2024-03-31T23:10:53.768 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-31T23:10:53.768 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/2169018547","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":2169018547},{"type":"v1","addr":"172.21.15.46:6835","nonce":2169018547}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6837/1069387756","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6836","nonce":1069387756},{"type":"v1","addr":"172.21.15.46:6837","nonce":1069387756}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14634,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6839/3915337844","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6838","nonce":3915337844},{"type":"v1","addr":"172.21.15.46:6839","nonce":3915337844}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14646,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6841/897767145","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6840","nonce":897767145},{"type":"v1","addr":"172.21.15.46:6841","nonce":897767145}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24415,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6833/1746950966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6832","nonce":1746950966},{"type":"v1","addr":"172.21.15.162:6833","nonce":1746950966}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24422,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6833/3394739181","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6832","nonce":3394739181},{"type":"v1","addr":"172.21.15.142:6833","nonce":3394739181}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":24,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-31T23:09:11.009090+0000","modified":"2024-03-31T23:09:31.392282+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24515,"mds_1":24482,"mds_2":24451},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24442":{"gid":24442,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.162:6835/1957208984","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6834","nonce":1957208984},{"type":"v1","addr":"172.21.15.162:6835","nonce":1957208984}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24451":{"gid":24451,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":11,"addr":"172.21.15.162:6837/1371886876","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6836","nonce":1371886876},{"type":"v1","addr":"172.21.15.162:6837","nonce":1371886876}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24455":{"gid":24455,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.142:6835/212637431","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6834","nonce":212637431},{"type":"v1","addr":"172.21.15.142:6835","nonce":212637431}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24478":{"gid":24478,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.162:6839/1976275826","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6838","nonce":1976275826},{"type":"v1","addr":"172.21.15.162:6839","nonce":1976275826}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24482":{"gid":24482,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":10,"addr":"172.21.15.142:6837/2559258399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6836","nonce":2559258399},{"type":"v1","addr":"172.21.15.142:6837","nonce":2559258399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24515":{"gid":24515,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.142:6839/2519143918","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6838","nonce":2519143918},{"type":"v1","addr":"172.21.15.142:6839","nonce":2519143918}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24515,"qdb_cluster":[24451,24482,24515]},"id":1}]} 2024-03-31T23:10:53.768 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 24 2024-03-31T23:10:53.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:53.768+0000 7f4bad6ae640 1 -- 172.21.15.46:0/2831969779 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f4b900a24a0 msgr2=0x7f4b900a4960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:10:53.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:53.768+0000 7f4bad6ae640 1 --2- 172.21.15.46:0/2831969779 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f4b900a24a0 0x7f4b900a4960 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f4ba8198eb0 tx=0x7f4b9c005f90 comp rx=0 tx=0).stop 2024-03-31T23:10:53.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:53.768+0000 7f4bad6ae640 1 -- 172.21.15.46:0/2831969779 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4ba8101770 msgr2=0x7f4ba81979a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:10:53.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:53.768+0000 7f4bad6ae640 1 --2- 172.21.15.46:0/2831969779 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4ba8101770 0x7f4ba81979a0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f4ba0000f80 tx=0x7f4ba00328a0 comp rx=0 tx=0).stop 2024-03-31T23:10:53.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:53.768+0000 7f4bad6ae640 1 -- 172.21.15.46:0/2831969779 shutdown_connections 2024-03-31T23:10:53.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:53.768+0000 7f4bad6ae640 1 --2- 172.21.15.46:0/2831969779 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f4b900a24a0 0x7f4b900a4960 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:53.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:53.768+0000 7f4bad6ae640 1 --2- 172.21.15.46:0/2831969779 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4ba8103320 0x7f4ba819ef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:53.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:53.768+0000 7f4bad6ae640 1 --2- 172.21.15.46:0/2831969779 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4ba8102960 0x7f4ba8197ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:53.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:53.768+0000 7f4bad6ae640 1 --2- 172.21.15.46:0/2831969779 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4ba8101770 0x7f4ba81979a0 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:53.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:53.768+0000 7f4bad6ae640 1 -- 172.21.15.46:0/2831969779 >> 172.21.15.46:0/2831969779 conn(0x7f4ba80fcd00 msgr2=0x7f4ba8105cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:10:53.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:53.769+0000 7f4bad6ae640 1 -- 172.21.15.46:0/2831969779 shutdown_connections 2024-03-31T23:10:53.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:53.769+0000 7f4bad6ae640 1 -- 172.21.15.46:0/2831969779 wait complete. 2024-03-31T23:10:53.780 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:10:53 smithi046 ceph-mon[32498]: from='client.? 172.21.15.46:0/3151056551' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-31T23:10:53.780 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:10:53 smithi046 ceph-mon[32498]: from='client.? 172.21.15.46:0/562315376' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-31T23:10:53.780 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:10:53 smithi046 ceph-mon[32498]: from='client.? 172.21.15.46:0/2863869923' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-31T23:10:53.781 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-31T23:10:53.795 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:10:53 smithi162 ceph-mon[36237]: from='client.? 172.21.15.46:0/3151056551' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-31T23:10:53.795 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:10:53 smithi162 ceph-mon[36237]: from='client.? 172.21.15.46:0/562315376' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-31T23:10:53.795 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:10:53 smithi162 ceph-mon[36237]: from='client.? 172.21.15.46:0/2863869923' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-31T23:10:53.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:53.883+0000 7fb2f006b640 1 -- 172.21.15.46:0/306212596 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb2e81022b0 msgr2=0x7fb2e81046a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:10:53.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:53.883+0000 7fb2f006b640 1 --2- 172.21.15.46:0/306212596 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb2e81022b0 0x7fb2e81046a0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7fb2e0009900 tx=0x7fb2e002bc30 comp rx=0 tx=0).stop 2024-03-31T23:10:53.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:53.883+0000 7fb2f006b640 1 -- 172.21.15.46:0/306212596 shutdown_connections 2024-03-31T23:10:53.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:53.883+0000 7fb2f006b640 1 --2- 172.21.15.46:0/306212596 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb2e8107510 0x7fb2e8109900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:53.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:53.883+0000 7fb2f006b640 1 --2- 172.21.15.46:0/306212596 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb2e8104be0 0x7fb2e8106fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:53.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:53.883+0000 7fb2f006b640 1 --2- 172.21.15.46:0/306212596 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb2e81022b0 0x7fb2e81046a0 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:53.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:53.883+0000 7fb2f006b640 1 -- 172.21.15.46:0/306212596 >> 172.21.15.46:0/306212596 conn(0x7fb2e80fbc00 msgr2=0x7fb2e80fe040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:10:53.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:53.883+0000 7fb2f006b640 1 -- 172.21.15.46:0/306212596 shutdown_connections 2024-03-31T23:10:53.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:53.883+0000 7fb2f006b640 1 -- 172.21.15.46:0/306212596 wait complete. 2024-03-31T23:10:53.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:53.883+0000 7fb2f006b640 1 Processor -- start 2024-03-31T23:10:53.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:53.883+0000 7fb2f006b640 1 -- start start 2024-03-31T23:10:53.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:53.883+0000 7fb2f006b640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb2e81022b0 0x7fb2e81979e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:10:53.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:53.883+0000 7fb2f006b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb2e8104be0 0x7fb2e8197f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:10:53.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:53.883+0000 7fb2f006b640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb2e8107510 0x7fb2e819ef90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:10:53.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:53.884+0000 7fb2f006b640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fb2e810d2b0 con 0x7fb2e8104be0 2024-03-31T23:10:53.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:53.884+0000 7fb2f006b640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fb2e810d130 con 0x7fb2e81022b0 2024-03-31T23:10:53.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:53.884+0000 7fb2f006b640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fb2e810d430 con 0x7fb2e8107510 2024-03-31T23:10:53.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:53.884+0000 7fb2ed606640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb2e8104be0 0x7fb2e8197f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:10:53.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:53.884+0000 7fb2ed606640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb2e8104be0 0x7fb2e8197f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:56194/0 (socket says 172.21.15.46:56194) 2024-03-31T23:10:53.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:53.884+0000 7fb2ede07640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb2e81022b0 0x7fb2e81979e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:10:53.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:53.884+0000 7fb2ed606640 1 -- 172.21.15.46:0/2226561562 learned_addr learned my addr 172.21.15.46:0/2226561562 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:10:53.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:53.884+0000 7fb2ee608640 1 --2- 172.21.15.46:0/2226561562 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb2e8107510 0x7fb2e819ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:10:53.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:53.884+0000 7fb2ed606640 1 -- 172.21.15.46:0/2226561562 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb2e8107510 msgr2=0x7fb2e819ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:10:53.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:53.884+0000 7fb2ed606640 1 --2- 172.21.15.46:0/2226561562 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb2e8107510 0x7fb2e819ef90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:53.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:53.884+0000 7fb2ed606640 1 -- 172.21.15.46:0/2226561562 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb2e81022b0 msgr2=0x7fb2e81979e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:10:53.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:53.884+0000 7fb2ed606640 1 --2- 172.21.15.46:0/2226561562 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb2e81022b0 0x7fb2e81979e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:53.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:53.884+0000 7fb2ed606640 1 -- 172.21.15.46:0/2226561562 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb2e819f600 con 0x7fb2e8104be0 2024-03-31T23:10:53.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:53.884+0000 7fb2ed606640 1 --2- 172.21.15.46:0/2226561562 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb2e8104be0 0x7fb2e8197f20 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7fb2d800d770 tx=0x7fb2d800dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:10:53.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:53.884+0000 7fb2deffd640 1 -- 172.21.15.46:0/2226561562 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2d8017070 con 0x7fb2e8104be0 2024-03-31T23:10:53.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:53.884+0000 7fb2deffd640 1 -- 172.21.15.46:0/2226561562 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb2d8009c30 con 0x7fb2e8104be0 2024-03-31T23:10:53.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:53.884+0000 7fb2deffd640 1 -- 172.21.15.46:0/2226561562 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2d800f620 con 0x7fb2e8104be0 2024-03-31T23:10:53.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:53.884+0000 7fb2f006b640 1 -- 172.21.15.46:0/2226561562 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb2e819f8f0 con 0x7fb2e8104be0 2024-03-31T23:10:53.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:53.884+0000 7fb2f006b640 1 -- 172.21.15.46:0/2226561562 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb2e819fe40 con 0x7fb2e8104be0 2024-03-31T23:10:53.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:53.885+0000 7fb2f006b640 1 -- 172.21.15.46:0/2226561562 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb2e80fd6f0 con 0x7fb2e8104be0 2024-03-31T23:10:53.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:53.886+0000 7fb2deffd640 1 -- 172.21.15.46:0/2226561562 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 18) ==== 137320+0+0 (secure 0 0 0) 0x7fb2d801a070 con 0x7fb2e8104be0 2024-03-31T23:10:53.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:53.887+0000 7fb2deffd640 1 --2- 172.21.15.46:0/2226561562 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fb2d40a24a0 0x7fb2d40a4960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:10:53.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:53.887+0000 7fb2ede07640 1 --2- 172.21.15.46:0/2226561562 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fb2d40a24a0 0x7fb2d40a4960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:10:53.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:53.887+0000 7fb2deffd640 1 -- 172.21.15.46:0/2226561562 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7fb2d80ce580 con 0x7fb2e8104be0 2024-03-31T23:10:53.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:53.887+0000 7fb2ede07640 1 --2- 172.21.15.46:0/2226561562 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fb2d40a24a0 0x7fb2d40a4960 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fb2e0009230 tx=0x7fb2e0005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:10:53.892 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:53.891+0000 7fb2deffd640 1 -- 172.21.15.46:0/2226561562 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fb2d8009de0 con 0x7fb2e8104be0 2024-03-31T23:10:53.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:10:53 smithi142 ceph-mon[35271]: from='client.? 172.21.15.46:0/3151056551' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-31T23:10:53.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:10:53 smithi142 ceph-mon[35271]: from='client.? 172.21.15.46:0/562315376' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-31T23:10:53.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:10:53 smithi142 ceph-mon[35271]: from='client.? 172.21.15.46:0/2863869923' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-31T23:10:54.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:54.074+0000 7fb2f006b640 1 -- 172.21.15.46:0/2226561562 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fb2e8062350 con 0x7fb2e8104be0 2024-03-31T23:10:54.076 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:54.076+0000 7fb2deffd640 1 -- 172.21.15.46:0/2226561562 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v93) ==== 74+0+24575 (secure 0 0 0) 0x7fb2d809ad30 con 0x7fb2e8104be0 2024-03-31T23:10:54.077 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-31T23:10:54.077 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":93,"fsid":"6a9d8f08-efb2-11ee-b647-cb9ed24678a4","created":"2024-03-31T23:01:15.463711+0000","modified":"2024-03-31T23:10:10.474156+0000","last_up_change":"2024-03-31T23:08:20.199888+0000","last_in_change":"2024-03-31T23:07:59.438035+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-31T23:04:32.823751+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-31T23:09:09.145711+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-31T23:09:10.148579+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-31T23:09:13.419519+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"91","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"47429095-3f46-4ed7-b7c4-504464719815","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":1824772695},{"type":"v1","addr":"172.21.15.46:6803","nonce":1824772695}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":1824772695},{"type":"v1","addr":"172.21.15.46:6805","nonce":1824772695}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":1824772695},{"type":"v1","addr":"172.21.15.46:6809","nonce":1824772695}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":1824772695},{"type":"v1","addr":"172.21.15.46:6807","nonce":1824772695}]},"public_addr":"172.21.15.46:6803/1824772695","cluster_addr":"172.21.15.46:6805/1824772695","heartbeat_back_addr":"172.21.15.46:6809/1824772695","heartbeat_front_addr":"172.21.15.46:6807/1824772695","state":["exists","up"]},{"osd":1,"uuid":"be56f883-411e-4dbe-b50e-a158134df4a6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6800","nonce":973693922},{"type":"v1","addr":"172.21.15.142:6801","nonce":973693922}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6802","nonce":973693922},{"type":"v1","addr":"172.21.15.142:6803","nonce":973693922}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6806","nonce":973693922},{"type":"v1","addr":"172.21.15.142:6807","nonce":973693922}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6804","nonce":973693922},{"type":"v1","addr":"172.21.15.142:6805","nonce":973693922}]},"public_addr":"172.21.15.142:6801/973693922","cluster_addr":"172.21.15.142:6803/973693922","heartbeat_back_addr":"172.21.15.142:6807/973693922","heartbeat_front_addr":"172.21.15.142:6805/973693922","state":["exists","up"]},{"osd":2,"uuid":"c99a8794-ab19-4260-9a8d-4bc0bd2c8175","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6800","nonce":4084261813},{"type":"v1","addr":"172.21.15.162:6801","nonce":4084261813}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6802","nonce":4084261813},{"type":"v1","addr":"172.21.15.162:6803","nonce":4084261813}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6806","nonce":4084261813},{"type":"v1","addr":"172.21.15.162:6807","nonce":4084261813}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6804","nonce":4084261813},{"type":"v1","addr":"172.21.15.162:6805","nonce":4084261813}]},"public_addr":"172.21.15.162:6801/4084261813","cluster_addr":"172.21.15.162:6803/4084261813","heartbeat_back_addr":"172.21.15.162:6807/4084261813","heartbeat_front_addr":"172.21.15.162:6805/4084261813","state":["exists","up"]},{"osd":3,"uuid":"f877814e-9ef6-4872-a081-5609bb9ebc06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":3496317970},{"type":"v1","addr":"172.21.15.46:6811","nonce":3496317970}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":3496317970},{"type":"v1","addr":"172.21.15.46:6813","nonce":3496317970}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":3496317970},{"type":"v1","addr":"172.21.15.46:6817","nonce":3496317970}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":3496317970},{"type":"v1","addr":"172.21.15.46:6815","nonce":3496317970}]},"public_addr":"172.21.15.46:6811/3496317970","cluster_addr":"172.21.15.46:6813/3496317970","heartbeat_back_addr":"172.21.15.46:6817/3496317970","heartbeat_front_addr":"172.21.15.46:6815/3496317970","state":["exists","up"]},{"osd":4,"uuid":"beecec0a-8280-44fe-8f5f-8498a35eca8f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6808","nonce":2996266929},{"type":"v1","addr":"172.21.15.142:6809","nonce":2996266929}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6810","nonce":2996266929},{"type":"v1","addr":"172.21.15.142:6811","nonce":2996266929}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6814","nonce":2996266929},{"type":"v1","addr":"172.21.15.142:6815","nonce":2996266929}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6812","nonce":2996266929},{"type":"v1","addr":"172.21.15.142:6813","nonce":2996266929}]},"public_addr":"172.21.15.142:6809/2996266929","cluster_addr":"172.21.15.142:6811/2996266929","heartbeat_back_addr":"172.21.15.142:6815/2996266929","heartbeat_front_addr":"172.21.15.142:6813/2996266929","state":["exists","up"]},{"osd":5,"uuid":"9da7ff75-f658-45ea-a02f-1943fdda76b9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6808","nonce":914208293},{"type":"v1","addr":"172.21.15.162:6809","nonce":914208293}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6810","nonce":914208293},{"type":"v1","addr":"172.21.15.162:6811","nonce":914208293}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6814","nonce":914208293},{"type":"v1","addr":"172.21.15.162:6815","nonce":914208293}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6812","nonce":914208293},{"type":"v1","addr":"172.21.15.162:6813","nonce":914208293}]},"public_addr":"172.21.15.162:6809/914208293","cluster_addr":"172.21.15.162:6811/914208293","heartbeat_back_addr":"172.21.15.162:6815/914208293","heartbeat_front_addr":"172.21.15.162:6813/914208293","state":["exists","up"]},{"osd":6,"uuid":"8fa11224-cb26-4d3f-87d1-6a8f31de2a19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":56291966},{"type":"v1","addr":"172.21.15.46:6819","nonce":56291966}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":56291966},{"type":"v1","addr":"172.21.15.46:6821","nonce":56291966}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6824","nonce":56291966},{"type":"v1","addr":"172.21.15.46:6825","nonce":56291966}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":56291966},{"type":"v1","addr":"172.21.15.46:6823","nonce":56291966}]},"public_addr":"172.21.15.46:6819/56291966","cluster_addr":"172.21.15.46:6821/56291966","heartbeat_back_addr":"172.21.15.46:6825/56291966","heartbeat_front_addr":"172.21.15.46:6823/56291966","state":["exists","up"]},{"osd":7,"uuid":"0d0338b7-4d0a-4bb0-b152-8f5224871d9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6816","nonce":2146598446},{"type":"v1","addr":"172.21.15.142:6817","nonce":2146598446}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6818","nonce":2146598446},{"type":"v1","addr":"172.21.15.142:6819","nonce":2146598446}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6822","nonce":2146598446},{"type":"v1","addr":"172.21.15.142:6823","nonce":2146598446}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6820","nonce":2146598446},{"type":"v1","addr":"172.21.15.142:6821","nonce":2146598446}]},"public_addr":"172.21.15.142:6817/2146598446","cluster_addr":"172.21.15.142:6819/2146598446","heartbeat_back_addr":"172.21.15.142:6823/2146598446","heartbeat_front_addr":"172.21.15.142:6821/2146598446","state":["exists","up"]},{"osd":8,"uuid":"8b6f6037-892a-416c-babb-7552648d0b3a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6816","nonce":1173078318},{"type":"v1","addr":"172.21.15.162:6817","nonce":1173078318}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6818","nonce":1173078318},{"type":"v1","addr":"172.21.15.162:6819","nonce":1173078318}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6822","nonce":1173078318},{"type":"v1","addr":"172.21.15.162:6823","nonce":1173078318}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6820","nonce":1173078318},{"type":"v1","addr":"172.21.15.162:6821","nonce":1173078318}]},"public_addr":"172.21.15.162:6817/1173078318","cluster_addr":"172.21.15.162:6819/1173078318","heartbeat_back_addr":"172.21.15.162:6823/1173078318","heartbeat_front_addr":"172.21.15.162:6821/1173078318","state":["exists","up"]},{"osd":9,"uuid":"faccce25-6bf2-4220-8858-cda82b72be68","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":532708881},{"type":"v1","addr":"172.21.15.46:6827","nonce":532708881}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6828","nonce":532708881},{"type":"v1","addr":"172.21.15.46:6829","nonce":532708881}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":532708881},{"type":"v1","addr":"172.21.15.46:6833","nonce":532708881}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":532708881},{"type":"v1","addr":"172.21.15.46:6831","nonce":532708881}]},"public_addr":"172.21.15.46:6827/532708881","cluster_addr":"172.21.15.46:6829/532708881","heartbeat_back_addr":"172.21.15.46:6833/532708881","heartbeat_front_addr":"172.21.15.46:6831/532708881","state":["exists","up"]},{"osd":10,"uuid":"cdd356db-f9eb-4080-95ab-db5d8223e572","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6824","nonce":2845222366},{"type":"v1","addr":"172.21.15.142:6825","nonce":2845222366}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6826","nonce":2845222366},{"type":"v1","addr":"172.21.15.142:6827","nonce":2845222366}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6830","nonce":2845222366},{"type":"v1","addr":"172.21.15.142:6831","nonce":2845222366}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6828","nonce":2845222366},{"type":"v1","addr":"172.21.15.142:6829","nonce":2845222366}]},"public_addr":"172.21.15.142:6825/2845222366","cluster_addr":"172.21.15.142:6827/2845222366","heartbeat_back_addr":"172.21.15.142:6831/2845222366","heartbeat_front_addr":"172.21.15.142:6829/2845222366","state":["exists","up"]},{"osd":11,"uuid":"72104d1c-3ea2-45ad-bc0b-0500f146c06b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6824","nonce":1986323778},{"type":"v1","addr":"172.21.15.162:6825","nonce":1986323778}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6826","nonce":1986323778},{"type":"v1","addr":"172.21.15.162:6827","nonce":1986323778}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6830","nonce":1986323778},{"type":"v1","addr":"172.21.15.162:6831","nonce":1986323778}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6828","nonce":1986323778},{"type":"v1","addr":"172.21.15.162:6829","nonce":1986323778}]},"public_addr":"172.21.15.162:6825/1986323778","cluster_addr":"172.21.15.162:6827/1986323778","heartbeat_back_addr":"172.21.15.162:6831/1986323778","heartbeat_front_addr":"172.21.15.162:6829/1986323778","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:03:44.996331+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:04:07.994738+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:04:30.942255+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:04:55.588884+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:05:19.900600+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:05:43.807442+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:06:09.026976+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:06:34.586784+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:06:59.751605+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:07:26.144495+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:07:52.326909+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:08:17.423042+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.6","mappings":[{"from":4,"to":7}]},{"pgid":"3.c","mappings":[{"from":3,"to":9}]},{"pgid":"3.1a","mappings":[{"from":5,"to":6}]},{"pgid":"3.1b","mappings":[{"from":4,"to":8}]},{"pgid":"3.1d","mappings":[{"from":4,"to":8}]},{"pgid":"3.2a","mappings":[{"from":3,"to":6}]},{"pgid":"3.33","mappings":[{"from":5,"to":9}]},{"pgid":"3.36","mappings":[{"from":4,"to":9}]},{"pgid":"4.17","mappings":[{"from":9,"to":8}]},{"pgid":"4.18","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.46:0/4021039857":"2024-04-01T23:02:08.493619+0000","172.21.15.46:6801/3681182647":"2024-04-01T23:02:08.493619+0000","172.21.15.46:0/4022214806":"2024-04-01T23:02:08.493619+0000","172.21.15.46:6800/1515528215":"2024-04-01T23:01:41.910162+0000","172.21.15.46:6800/3681182647":"2024-04-01T23:02:08.493619+0000","172.21.15.46:0/143885707":"2024-04-01T23:01:41.910162+0000","172.21.15.46:6801/1515528215":"2024-04-01T23:01:41.910162+0000","172.21.15.46:0/1680389778":"2024-04-01T23:01:41.910162+0000","172.21.15.46:0/1782133966":"2024-04-01T23:01:41.910162+0000","172.21.15.46:0/3925209376":"2024-04-01T23:02:08.493619+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-31T23:10:54.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:54.077+0000 7fb2f006b640 1 -- 172.21.15.46:0/2226561562 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fb2d40a24a0 msgr2=0x7fb2d40a4960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:10:54.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:54.077+0000 7fb2f006b640 1 --2- 172.21.15.46:0/2226561562 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fb2d40a24a0 0x7fb2d40a4960 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fb2e0009230 tx=0x7fb2e0005fd0 comp rx=0 tx=0).stop 2024-03-31T23:10:54.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:54.077+0000 7fb2f006b640 1 -- 172.21.15.46:0/2226561562 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb2e8104be0 msgr2=0x7fb2e8197f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:10:54.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:54.077+0000 7fb2f006b640 1 --2- 172.21.15.46:0/2226561562 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb2e8104be0 0x7fb2e8197f20 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7fb2d800d770 tx=0x7fb2d800dc30 comp rx=0 tx=0).stop 2024-03-31T23:10:54.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:54.078+0000 7fb2f006b640 1 -- 172.21.15.46:0/2226561562 shutdown_connections 2024-03-31T23:10:54.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:54.078+0000 7fb2f006b640 1 --2- 172.21.15.46:0/2226561562 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fb2d40a24a0 0x7fb2d40a4960 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:54.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:54.078+0000 7fb2f006b640 1 --2- 172.21.15.46:0/2226561562 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb2e8107510 0x7fb2e819ef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:54.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:54.078+0000 7fb2f006b640 1 --2- 172.21.15.46:0/2226561562 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb2e8104be0 0x7fb2e8197f20 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:54.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:54.078+0000 7fb2f006b640 1 --2- 172.21.15.46:0/2226561562 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb2e81022b0 0x7fb2e81979e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:54.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:54.078+0000 7fb2f006b640 1 -- 172.21.15.46:0/2226561562 >> 172.21.15.46:0/2226561562 conn(0x7fb2e80fbc00 msgr2=0x7fb2e8102d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:10:54.080 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:54.078+0000 7fb2f006b640 1 -- 172.21.15.46:0/2226561562 shutdown_connections 2024-03-31T23:10:54.080 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:54.078+0000 7fb2f006b640 1 -- 172.21.15.46:0/2226561562 wait complete. 2024-03-31T23:10:54.090 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-31T23:04:32.823751+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-31T23:09:09.145711+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-31T23:09:10.148579+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-31T23:09:13.419519+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '91', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-31T23:10:54.091 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-31T23:10:54.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:54.195+0000 7fd47148a640 1 -- 172.21.15.46:0/678410712 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd46c107430 msgr2=0x7fd46c109880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:10:54.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:54.195+0000 7fd47148a640 1 --2- 172.21.15.46:0/678410712 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd46c107430 0x7fd46c109880 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fd464009ca0 tx=0x7fd46402c0f0 comp rx=0 tx=0).stop 2024-03-31T23:10:54.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:54.195+0000 7fd47148a640 1 -- 172.21.15.46:0/678410712 shutdown_connections 2024-03-31T23:10:54.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:54.195+0000 7fd47148a640 1 --2- 172.21.15.46:0/678410712 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd46c107430 0x7fd46c109880 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:54.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:54.195+0000 7fd47148a640 1 --2- 172.21.15.46:0/678410712 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd46c104b00 0x7fd46c106ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:54.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:54.195+0000 7fd47148a640 1 --2- 172.21.15.46:0/678410712 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd46c1021d0 0x7fd46c1045c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:54.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:54.195+0000 7fd47148a640 1 -- 172.21.15.46:0/678410712 >> 172.21.15.46:0/678410712 conn(0x7fd46c0fbbc0 msgr2=0x7fd46c0fe000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:10:54.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:54.195+0000 7fd47148a640 1 -- 172.21.15.46:0/678410712 shutdown_connections 2024-03-31T23:10:54.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:54.195+0000 7fd47148a640 1 -- 172.21.15.46:0/678410712 wait complete. 2024-03-31T23:10:54.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:54.196+0000 7fd47148a640 1 Processor -- start 2024-03-31T23:10:54.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:54.196+0000 7fd47148a640 1 -- start start 2024-03-31T23:10:54.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:54.196+0000 7fd47148a640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd46c1021d0 0x7fd46c101b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:10:54.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:54.196+0000 7fd47148a640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd46c104b00 0x7fd46c100180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:10:54.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:54.196+0000 7fd47148a640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd46c107430 0x7fd46c1006c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:10:54.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:54.196+0000 7fd47148a640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fd46c10d280 con 0x7fd46c1021d0 2024-03-31T23:10:54.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:54.196+0000 7fd47148a640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fd46c10d100 con 0x7fd46c104b00 2024-03-31T23:10:54.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:54.196+0000 7fd47148a640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fd46c10d400 con 0x7fd46c107430 2024-03-31T23:10:54.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:54.196+0000 7fd46affd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd46c1021d0 0x7fd46c101b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:10:54.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:54.197+0000 7fd46affd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd46c1021d0 0x7fd46c101b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:56202/0 (socket says 172.21.15.46:56202) 2024-03-31T23:10:54.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:54.197+0000 7fd46affd640 1 -- 172.21.15.46:0/3958446485 learned_addr learned my addr 172.21.15.46:0/3958446485 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:10:54.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:54.197+0000 7fd46a7fc640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd46c104b00 0x7fd46c100180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:10:54.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:54.197+0000 7fd46b7fe640 1 --2- 172.21.15.46:0/3958446485 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd46c107430 0x7fd46c1006c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:10:54.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:54.197+0000 7fd46affd640 1 -- 172.21.15.46:0/3958446485 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd46c107430 msgr2=0x7fd46c1006c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:10:54.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:54.197+0000 7fd46affd640 1 --2- 172.21.15.46:0/3958446485 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd46c107430 0x7fd46c1006c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:54.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:54.197+0000 7fd46affd640 1 -- 172.21.15.46:0/3958446485 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd46c104b00 msgr2=0x7fd46c100180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:10:54.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:54.197+0000 7fd46affd640 1 --2- 172.21.15.46:0/3958446485 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd46c104b00 0x7fd46c100180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:54.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:54.197+0000 7fd46affd640 1 -- 172.21.15.46:0/3958446485 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd46c100f50 con 0x7fd46c1021d0 2024-03-31T23:10:54.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:54.197+0000 7fd46a7fc640 1 --2- 172.21.15.46:0/3958446485 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd46c104b00 0x7fd46c100180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:10:54.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:54.197+0000 7fd46affd640 1 --2- 172.21.15.46:0/3958446485 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd46c1021d0 0x7fd46c101b40 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7fd45c00ea60 tx=0x7fd45c00ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:10:54.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:54.197+0000 7fd453fff640 1 -- 172.21.15.46:0/3958446485 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd45c00cde0 con 0x7fd46c1021d0 2024-03-31T23:10:54.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:54.197+0000 7fd453fff640 1 -- 172.21.15.46:0/3958446485 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd45c0035f0 con 0x7fd46c1021d0 2024-03-31T23:10:54.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:54.197+0000 7fd47148a640 1 -- 172.21.15.46:0/3958446485 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd46c1a1a50 con 0x7fd46c1021d0 2024-03-31T23:10:54.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:54.197+0000 7fd453fff640 1 -- 172.21.15.46:0/3958446485 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd45c010690 con 0x7fd46c1021d0 2024-03-31T23:10:54.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:54.197+0000 7fd47148a640 1 -- 172.21.15.46:0/3958446485 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd46c1a1fa0 con 0x7fd46c1021d0 2024-03-31T23:10:54.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:54.198+0000 7fd47148a640 1 -- 172.21.15.46:0/3958446485 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd46c068540 con 0x7fd46c1021d0 2024-03-31T23:10:54.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:54.199+0000 7fd453fff640 1 -- 172.21.15.46:0/3958446485 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 18) ==== 137320+0+0 (secure 0 0 0) 0x7fd45c0037a0 con 0x7fd46c1021d0 2024-03-31T23:10:54.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:54.200+0000 7fd453fff640 1 --2- 172.21.15.46:0/3958446485 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fd4540a24a0 0x7fd4540a4960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:10:54.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:54.200+0000 7fd46a7fc640 1 --2- 172.21.15.46:0/3958446485 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fd4540a24a0 0x7fd4540a4960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:10:54.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:54.200+0000 7fd453fff640 1 -- 172.21.15.46:0/3958446485 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7fd45c0ced60 con 0x7fd46c1021d0 2024-03-31T23:10:54.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:54.200+0000 7fd46a7fc640 1 --2- 172.21.15.46:0/3958446485 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fd4540a24a0 0x7fd4540a4960 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fd4600095f0 tx=0x7fd460005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:10:54.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:54.205+0000 7fd453fff640 1 -- 172.21.15.46:0/3958446485 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fd45c09bbc0 con 0x7fd46c1021d0 2024-03-31T23:10:54.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:54.387+0000 7fd47148a640 1 -- 172.21.15.46:0/3958446485 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fd46c062350 con 0x7fd46c1021d0 2024-03-31T23:10:54.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:54.388+0000 7fd453fff640 1 -- 172.21.15.46:0/3958446485 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v93) ==== 93+0+10 (secure 0 0 0) 0x7fd45c09b4c0 con 0x7fd46c1021d0 2024-03-31T23:10:54.389 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 1 2024-03-31T23:10:54.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:54.390+0000 7fd47148a640 1 -- 172.21.15.46:0/3958446485 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fd4540a24a0 msgr2=0x7fd4540a4960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:10:54.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:54.390+0000 7fd47148a640 1 --2- 172.21.15.46:0/3958446485 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fd4540a24a0 0x7fd4540a4960 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fd4600095f0 tx=0x7fd460005f90 comp rx=0 tx=0).stop 2024-03-31T23:10:54.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:54.390+0000 7fd47148a640 1 -- 172.21.15.46:0/3958446485 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd46c1021d0 msgr2=0x7fd46c101b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:10:54.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:54.390+0000 7fd47148a640 1 --2- 172.21.15.46:0/3958446485 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd46c1021d0 0x7fd46c101b40 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7fd45c00ea60 tx=0x7fd45c00ef20 comp rx=0 tx=0).stop 2024-03-31T23:10:54.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:54.390+0000 7fd47148a640 1 -- 172.21.15.46:0/3958446485 shutdown_connections 2024-03-31T23:10:54.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:54.390+0000 7fd47148a640 1 --2- 172.21.15.46:0/3958446485 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fd4540a24a0 0x7fd4540a4960 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:54.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:54.390+0000 7fd47148a640 1 --2- 172.21.15.46:0/3958446485 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd46c107430 0x7fd46c1006c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:54.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:54.390+0000 7fd47148a640 1 --2- 172.21.15.46:0/3958446485 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd46c104b00 0x7fd46c100180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:54.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:54.390+0000 7fd47148a640 1 --2- 172.21.15.46:0/3958446485 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd46c1021d0 0x7fd46c101b40 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:54.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:54.390+0000 7fd47148a640 1 -- 172.21.15.46:0/3958446485 >> 172.21.15.46:0/3958446485 conn(0x7fd46c0fbbc0 msgr2=0x7fd46c102b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:10:54.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:54.390+0000 7fd47148a640 1 -- 172.21.15.46:0/3958446485 shutdown_connections 2024-03-31T23:10:54.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:54.390+0000 7fd47148a640 1 -- 172.21.15.46:0/3958446485 wait complete. 2024-03-31T23:10:54.402 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-31T23:10:54.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:54.508+0000 7ff9834d6640 1 -- 172.21.15.46:0/4273895858 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff97c102200 msgr2=0x7ff97c108a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:10:54.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:54.508+0000 7ff9834d6640 1 --2- 172.21.15.46:0/4273895858 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff97c102200 0x7ff97c108a90 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7ff974009ca0 tx=0x7ff97402c0f0 comp rx=0 tx=0).stop 2024-03-31T23:10:54.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:54.508+0000 7ff9834d6640 1 -- 172.21.15.46:0/4273895858 shutdown_connections 2024-03-31T23:10:54.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:54.508+0000 7ff9834d6640 1 --2- 172.21.15.46:0/4273895858 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff97c102200 0x7ff97c108a90 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:54.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:54.508+0000 7ff9834d6640 1 --2- 172.21.15.46:0/4273895858 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff97c101840 0x7ff97c101cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:54.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:54.508+0000 7ff9834d6640 1 --2- 172.21.15.46:0/4273895858 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff97c100650 0x7ff97c100a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:54.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:54.508+0000 7ff9834d6640 1 -- 172.21.15.46:0/4273895858 >> 172.21.15.46:0/4273895858 conn(0x7ff97c0fbbe0 msgr2=0x7ff97c0fe000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:10:54.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:54.508+0000 7ff9834d6640 1 -- 172.21.15.46:0/4273895858 shutdown_connections 2024-03-31T23:10:54.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:54.509+0000 7ff9834d6640 1 -- 172.21.15.46:0/4273895858 wait complete. 2024-03-31T23:10:54.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:54.509+0000 7ff9834d6640 1 Processor -- start 2024-03-31T23:10:54.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:54.509+0000 7ff9834d6640 1 -- start start 2024-03-31T23:10:54.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:54.509+0000 7ff9834d6640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff97c100650 0x7ff97c199bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:10:54.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:54.509+0000 7ff9834d6640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff97c101840 0x7ff97c19a100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:10:54.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:54.509+0000 7ff9834d6640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff97c102200 0x7ff97c1a1170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:10:54.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:54.509+0000 7ff9834d6640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7ff97c10c190 con 0x7ff97c102200 2024-03-31T23:10:54.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:54.509+0000 7ff9834d6640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7ff97c10c010 con 0x7ff97c101840 2024-03-31T23:10:54.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:54.509+0000 7ff9834d6640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7ff97c10c310 con 0x7ff97c100650 2024-03-31T23:10:54.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:54.509+0000 7ff981a73640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff97c102200 0x7ff97c1a1170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:10:54.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:54.509+0000 7ff981a73640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff97c102200 0x7ff97c1a1170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:56208/0 (socket says 172.21.15.46:56208) 2024-03-31T23:10:54.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:54.509+0000 7ff981a73640 1 -- 172.21.15.46:0/1171005682 learned_addr learned my addr 172.21.15.46:0/1171005682 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:10:54.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:54.509+0000 7ff980a71640 1 --2- 172.21.15.46:0/1171005682 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff97c101840 0x7ff97c19a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:10:54.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:54.509+0000 7ff981272640 1 --2- 172.21.15.46:0/1171005682 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff97c100650 0x7ff97c199bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:10:54.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:54.510+0000 7ff981a73640 1 -- 172.21.15.46:0/1171005682 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff97c100650 msgr2=0x7ff97c199bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:10:54.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:54.510+0000 7ff981a73640 1 --2- 172.21.15.46:0/1171005682 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff97c100650 0x7ff97c199bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:54.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:54.510+0000 7ff981a73640 1 -- 172.21.15.46:0/1171005682 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff97c101840 msgr2=0x7ff97c19a100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:10:54.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:54.510+0000 7ff981a73640 1 --2- 172.21.15.46:0/1171005682 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff97c101840 0x7ff97c19a100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:54.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:54.510+0000 7ff981a73640 1 -- 172.21.15.46:0/1171005682 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff97c1a17e0 con 0x7ff97c102200 2024-03-31T23:10:54.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:54.510+0000 7ff981a73640 1 --2- 172.21.15.46:0/1171005682 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff97c102200 0x7ff97c1a1170 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7ff974032380 tx=0x7ff974032820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:10:54.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:54.510+0000 7ff9727fc640 1 -- 172.21.15.46:0/1171005682 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff974041020 con 0x7ff97c102200 2024-03-31T23:10:54.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:54.510+0000 7ff9727fc640 1 -- 172.21.15.46:0/1171005682 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff974003e20 con 0x7ff97c102200 2024-03-31T23:10:54.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:54.510+0000 7ff9834d6640 1 -- 172.21.15.46:0/1171005682 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff97c1a1a70 con 0x7ff97c102200 2024-03-31T23:10:54.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:54.510+0000 7ff9727fc640 1 -- 172.21.15.46:0/1171005682 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff974039040 con 0x7ff97c102200 2024-03-31T23:10:54.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:54.510+0000 7ff9834d6640 1 -- 172.21.15.46:0/1171005682 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff97c1a1fb0 con 0x7ff97c102200 2024-03-31T23:10:54.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:54.511+0000 7ff9834d6640 1 -- 172.21.15.46:0/1171005682 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff97c1a21e0 con 0x7ff97c102200 2024-03-31T23:10:54.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:54.512+0000 7ff9727fc640 1 -- 172.21.15.46:0/1171005682 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 18) ==== 137320+0+0 (secure 0 0 0) 0x7ff974044020 con 0x7ff97c102200 2024-03-31T23:10:54.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:54.513+0000 7ff9727fc640 1 --2- 172.21.15.46:0/1171005682 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7ff9680a2520 0x7ff9680a49e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:10:54.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:54.513+0000 7ff981272640 1 --2- 172.21.15.46:0/1171005682 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7ff9680a2520 0x7ff9680a49e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:10:54.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:54.513+0000 7ff9727fc640 1 -- 172.21.15.46:0/1171005682 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7ff9740f0a30 con 0x7ff97c102200 2024-03-31T23:10:54.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:54.513+0000 7ff981272640 1 --2- 172.21.15.46:0/1171005682 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7ff9680a2520 0x7ff9680a49e0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7ff9780095f0 tx=0x7ff978005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:10:54.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:54.517+0000 7ff9727fc640 1 -- 172.21.15.46:0/1171005682 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7ff97404b020 con 0x7ff97c102200 2024-03-31T23:10:54.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:54.701+0000 7ff9834d6640 1 -- 172.21.15.46:0/1171005682 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7ff97c1a23b0 con 0x7ff97c102200 2024-03-31T23:10:54.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:54.701+0000 7ff9727fc640 1 -- 172.21.15.46:0/1171005682 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v93) ==== 104+0+11 (secure 0 0 0) 0x7ff9740bd890 con 0x7ff97c102200 2024-03-31T23:10:54.702 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 32 2024-03-31T23:10:54.704 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:54.703+0000 7ff9834d6640 1 -- 172.21.15.46:0/1171005682 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7ff9680a2520 msgr2=0x7ff9680a49e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:10:54.704 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:54.703+0000 7ff9834d6640 1 --2- 172.21.15.46:0/1171005682 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7ff9680a2520 0x7ff9680a49e0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7ff9780095f0 tx=0x7ff978005f90 comp rx=0 tx=0).stop 2024-03-31T23:10:54.704 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:54.703+0000 7ff9834d6640 1 -- 172.21.15.46:0/1171005682 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff97c102200 msgr2=0x7ff97c1a1170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:10:54.704 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:54.703+0000 7ff9834d6640 1 --2- 172.21.15.46:0/1171005682 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff97c102200 0x7ff97c1a1170 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7ff974032380 tx=0x7ff974032820 comp rx=0 tx=0).stop 2024-03-31T23:10:54.704 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:54.703+0000 7ff9834d6640 1 -- 172.21.15.46:0/1171005682 shutdown_connections 2024-03-31T23:10:54.704 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:54.703+0000 7ff9834d6640 1 --2- 172.21.15.46:0/1171005682 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7ff9680a2520 0x7ff9680a49e0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:54.704 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:54.703+0000 7ff9834d6640 1 --2- 172.21.15.46:0/1171005682 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff97c102200 0x7ff97c1a1170 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:54.704 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:54.703+0000 7ff9834d6640 1 --2- 172.21.15.46:0/1171005682 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff97c101840 0x7ff97c19a100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:54.705 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:54.703+0000 7ff9834d6640 1 --2- 172.21.15.46:0/1171005682 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff97c100650 0x7ff97c199bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:54.705 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:54.703+0000 7ff9834d6640 1 -- 172.21.15.46:0/1171005682 >> 172.21.15.46:0/1171005682 conn(0x7ff97c0fbbe0 msgr2=0x7ff97c0fd6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:10:54.705 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:54.703+0000 7ff9834d6640 1 -- 172.21.15.46:0/1171005682 shutdown_connections 2024-03-31T23:10:54.705 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:54.703+0000 7ff9834d6640 1 -- 172.21.15.46:0/1171005682 wait complete. 2024-03-31T23:10:54.714 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:10:54 smithi046 ceph-mon[32498]: pgmap v345: 129 pgs: 129 active+clean; 642 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 170 B/s wr, 5 op/s 2024-03-31T23:10:54.714 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:10:54 smithi046 ceph-mon[32498]: from='client.? 172.21.15.46:0/2831969779' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-31T23:10:54.714 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:10:54 smithi046 ceph-mon[32498]: from='client.? 172.21.15.46:0/2226561562' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-31T23:10:54.714 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:10:54 smithi046 ceph-mon[32498]: from='client.? 172.21.15.46:0/3958446485' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-31T23:10:54.715 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-31T23:10:54.795 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:10:54 smithi162 ceph-mon[36237]: pgmap v345: 129 pgs: 129 active+clean; 642 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 170 B/s wr, 5 op/s 2024-03-31T23:10:54.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:10:54 smithi162 ceph-mon[36237]: from='client.? 172.21.15.46:0/2831969779' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-31T23:10:54.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:10:54 smithi162 ceph-mon[36237]: from='client.? 172.21.15.46:0/2226561562' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-31T23:10:54.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:10:54 smithi162 ceph-mon[36237]: from='client.? 172.21.15.46:0/3958446485' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-31T23:10:54.823 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:54.820+0000 7f42144aa640 1 -- 172.21.15.46:0/3534148890 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f420c101840 msgr2=0x7f420c101cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:10:54.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:54.820+0000 7f42144aa640 1 --2- 172.21.15.46:0/3534148890 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f420c101840 0x7f420c101cc0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f41fc009900 tx=0x7f41fc02bc30 comp rx=0 tx=0).stop 2024-03-31T23:10:54.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:54.821+0000 7f42144aa640 1 -- 172.21.15.46:0/3534148890 shutdown_connections 2024-03-31T23:10:54.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:54.821+0000 7f42144aa640 1 --2- 172.21.15.46:0/3534148890 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f420c102200 0x7f420c108a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:54.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:54.821+0000 7f42144aa640 1 --2- 172.21.15.46:0/3534148890 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f420c101840 0x7f420c101cc0 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:54.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:54.821+0000 7f42144aa640 1 --2- 172.21.15.46:0/3534148890 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f420c100650 0x7f420c100a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:54.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:54.821+0000 7f42144aa640 1 -- 172.21.15.46:0/3534148890 >> 172.21.15.46:0/3534148890 conn(0x7f420c0fbbc0 msgr2=0x7f420c0fe000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:10:54.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:54.821+0000 7f42144aa640 1 -- 172.21.15.46:0/3534148890 shutdown_connections 2024-03-31T23:10:54.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:54.821+0000 7f42144aa640 1 -- 172.21.15.46:0/3534148890 wait complete. 2024-03-31T23:10:54.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:54.821+0000 7f42144aa640 1 Processor -- start 2024-03-31T23:10:54.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:54.821+0000 7f42144aa640 1 -- start start 2024-03-31T23:10:54.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:54.821+0000 7f42144aa640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f420c100650 0x7f420c1979b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:10:54.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:54.821+0000 7f42144aa640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f420c101840 0x7f420c197ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:10:54.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:54.821+0000 7f42144aa640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f420c102200 0x7f420c19ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:10:54.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:54.821+0000 7f42144aa640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f420c10c190 con 0x7f420c102200 2024-03-31T23:10:54.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:54.821+0000 7f42144aa640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f420c10c010 con 0x7f420c101840 2024-03-31T23:10:54.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:54.821+0000 7f42144aa640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f420c10c310 con 0x7f420c100650 2024-03-31T23:10:54.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:54.821+0000 7f4212a47640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f420c102200 0x7f420c19ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:10:54.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:54.821+0000 7f4212a47640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f420c102200 0x7f420c19ef60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:56230/0 (socket says 172.21.15.46:56230) 2024-03-31T23:10:54.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:54.821+0000 7f4212a47640 1 -- 172.21.15.46:0/338477404 learned_addr learned my addr 172.21.15.46:0/338477404 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:10:54.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:54.821+0000 7f4211a45640 1 --2- 172.21.15.46:0/338477404 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f420c101840 0x7f420c197ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:10:54.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:54.821+0000 7f4212246640 1 --2- 172.21.15.46:0/338477404 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f420c100650 0x7f420c1979b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:10:54.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:54.822+0000 7f4212a47640 1 -- 172.21.15.46:0/338477404 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f420c100650 msgr2=0x7f420c1979b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:10:54.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:54.822+0000 7f4212a47640 1 --2- 172.21.15.46:0/338477404 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f420c100650 0x7f420c1979b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:54.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:54.822+0000 7f4212a47640 1 -- 172.21.15.46:0/338477404 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f420c101840 msgr2=0x7f420c197ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:10:54.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:54.822+0000 7f4212a47640 1 --2- 172.21.15.46:0/338477404 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f420c101840 0x7f420c197ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:54.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:54.822+0000 7f4212a47640 1 -- 172.21.15.46:0/338477404 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f420c19f5d0 con 0x7f420c102200 2024-03-31T23:10:54.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:54.822+0000 7f4211a45640 1 --2- 172.21.15.46:0/338477404 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f420c101840 0x7f420c197ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:10:54.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:54.822+0000 7f4212a47640 1 --2- 172.21.15.46:0/338477404 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f420c102200 0x7f420c19ef60 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f420400eb10 tx=0x7f420400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:10:54.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:54.822+0000 7f42037fe640 1 -- 172.21.15.46:0/338477404 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4204004b20 con 0x7f420c102200 2024-03-31T23:10:54.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:54.822+0000 7f42037fe640 1 -- 172.21.15.46:0/338477404 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4204004cc0 con 0x7f420c102200 2024-03-31T23:10:54.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:54.822+0000 7f42144aa640 1 -- 172.21.15.46:0/338477404 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f420c19f8c0 con 0x7f420c102200 2024-03-31T23:10:54.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:54.822+0000 7f42037fe640 1 -- 172.21.15.46:0/338477404 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4204010690 con 0x7f420c102200 2024-03-31T23:10:54.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:54.822+0000 7f42144aa640 1 -- 172.21.15.46:0/338477404 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f420c19fd90 con 0x7f420c102200 2024-03-31T23:10:54.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:54.823+0000 7f42144aa640 1 -- 172.21.15.46:0/338477404 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f420c1a0050 con 0x7f420c102200 2024-03-31T23:10:54.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:54.824+0000 7f42037fe640 1 -- 172.21.15.46:0/338477404 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 18) ==== 137320+0+0 (secure 0 0 0) 0x7f4204005020 con 0x7f420c102200 2024-03-31T23:10:54.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:54.825+0000 7f42037fe640 1 --2- 172.21.15.46:0/338477404 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f41f80a24a0 0x7f41f80a4960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:10:54.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:54.825+0000 7f4212246640 1 --2- 172.21.15.46:0/338477404 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f41f80a24a0 0x7f41f80a4960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:10:54.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:54.825+0000 7f42037fe640 1 -- 172.21.15.46:0/338477404 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7f42040cf010 con 0x7f420c102200 2024-03-31T23:10:54.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:54.825+0000 7f4212246640 1 --2- 172.21.15.46:0/338477404 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f41f80a24a0 0x7f41f80a4960 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f420c1016a0 tx=0x7f4208005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:10:54.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:54.829+0000 7f42037fe640 1 -- 172.21.15.46:0/338477404 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f420409be70 con 0x7f420c102200 2024-03-31T23:10:54.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:10:54 smithi142 ceph-mon[35271]: pgmap v345: 129 pgs: 129 active+clean; 642 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 170 B/s wr, 5 op/s 2024-03-31T23:10:54.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:10:54 smithi142 ceph-mon[35271]: from='client.? 172.21.15.46:0/2831969779' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-31T23:10:54.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:10:54 smithi142 ceph-mon[35271]: from='client.? 172.21.15.46:0/2226561562' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-31T23:10:54.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:10:54 smithi142 ceph-mon[35271]: from='client.? 172.21.15.46:0/3958446485' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-31T23:10:55.010 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:55.009+0000 7f42144aa640 1 -- 172.21.15.46:0/338477404 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f420c1a02c0 con 0x7f420c102200 2024-03-31T23:10:55.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:55.010+0000 7f42037fe640 1 -- 172.21.15.46:0/338477404 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v93) ==== 100+0+11 (secure 0 0 0) 0x7f420409b770 con 0x7f420c102200 2024-03-31T23:10:55.011 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 64 2024-03-31T23:10:55.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:55.012+0000 7f42144aa640 1 -- 172.21.15.46:0/338477404 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f41f80a24a0 msgr2=0x7f41f80a4960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:10:55.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:55.012+0000 7f42144aa640 1 --2- 172.21.15.46:0/338477404 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f41f80a24a0 0x7f41f80a4960 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f420c1016a0 tx=0x7f4208005f90 comp rx=0 tx=0).stop 2024-03-31T23:10:55.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:55.012+0000 7f42144aa640 1 -- 172.21.15.46:0/338477404 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f420c102200 msgr2=0x7f420c19ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:10:55.014 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:55.012+0000 7f42144aa640 1 --2- 172.21.15.46:0/338477404 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f420c102200 0x7f420c19ef60 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f420400eb10 tx=0x7f420400c2d0 comp rx=0 tx=0).stop 2024-03-31T23:10:55.014 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:55.012+0000 7f42144aa640 1 -- 172.21.15.46:0/338477404 shutdown_connections 2024-03-31T23:10:55.014 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:55.012+0000 7f42144aa640 1 --2- 172.21.15.46:0/338477404 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f41f80a24a0 0x7f41f80a4960 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:55.014 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:55.012+0000 7f42144aa640 1 --2- 172.21.15.46:0/338477404 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f420c102200 0x7f420c19ef60 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:55.014 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:55.012+0000 7f42144aa640 1 --2- 172.21.15.46:0/338477404 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f420c101840 0x7f420c197ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:55.014 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:55.012+0000 7f42144aa640 1 --2- 172.21.15.46:0/338477404 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f420c100650 0x7f420c1979b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:55.014 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:55.012+0000 7f42144aa640 1 -- 172.21.15.46:0/338477404 >> 172.21.15.46:0/338477404 conn(0x7f420c0fbbc0 msgr2=0x7f420c0688b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:10:55.014 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:55.012+0000 7f42144aa640 1 -- 172.21.15.46:0/338477404 shutdown_connections 2024-03-31T23:10:55.014 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:55.012+0000 7f42144aa640 1 -- 172.21.15.46:0/338477404 wait complete. 2024-03-31T23:10:55.024 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-31T23:10:55.132 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:55.129+0000 7f9c93cb5640 1 -- 172.21.15.46:0/2911587316 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9c8c0ffa20 msgr2=0x7f9c8c0ffe40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:10:55.132 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:55.129+0000 7f9c93cb5640 1 --2- 172.21.15.46:0/2911587316 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9c8c0ffa20 0x7f9c8c0ffe40 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f9c88009890 tx=0x7f9c8802bb70 comp rx=0 tx=0).stop 2024-03-31T23:10:55.132 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:55.129+0000 7f9c93cb5640 1 -- 172.21.15.46:0/2911587316 shutdown_connections 2024-03-31T23:10:55.132 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:55.129+0000 7f9c93cb5640 1 --2- 172.21.15.46:0/2911587316 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9c8c1033b0 0x7f9c8c109c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:55.132 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:55.129+0000 7f9c93cb5640 1 --2- 172.21.15.46:0/2911587316 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9c8c1029f0 0x7f9c8c102e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:55.132 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:55.129+0000 7f9c93cb5640 1 --2- 172.21.15.46:0/2911587316 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9c8c0ffa20 0x7f9c8c0ffe40 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:55.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:55.129+0000 7f9c93cb5640 1 -- 172.21.15.46:0/2911587316 >> 172.21.15.46:0/2911587316 conn(0x7f9c8c0fb370 msgr2=0x7f9c8c0fd790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:10:55.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:55.129+0000 7f9c93cb5640 1 -- 172.21.15.46:0/2911587316 shutdown_connections 2024-03-31T23:10:55.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:55.129+0000 7f9c93cb5640 1 -- 172.21.15.46:0/2911587316 wait complete. 2024-03-31T23:10:55.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:55.130+0000 7f9c93cb5640 1 Processor -- start 2024-03-31T23:10:55.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:55.130+0000 7f9c93cb5640 1 -- start start 2024-03-31T23:10:55.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:55.130+0000 7f9c93cb5640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9c8c0ffa20 0x7f9c8c197a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:10:55.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:55.130+0000 7f9c93cb5640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9c8c1029f0 0x7f9c8c197f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:10:55.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:55.130+0000 7f9c93cb5640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9c8c1033b0 0x7f9c8c19efe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:10:55.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:55.130+0000 7f9c93cb5640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f9c8c10d280 con 0x7f9c8c1029f0 2024-03-31T23:10:55.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:55.130+0000 7f9c93cb5640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f9c8c10d100 con 0x7f9c8c0ffa20 2024-03-31T23:10:55.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:55.130+0000 7f9c93cb5640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f9c8c10d400 con 0x7f9c8c1033b0 2024-03-31T23:10:55.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:55.130+0000 7f9c91250640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9c8c1029f0 0x7f9c8c197f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:10:55.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:55.130+0000 7f9c91a51640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9c8c0ffa20 0x7f9c8c197a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:10:55.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:55.130+0000 7f9c91a51640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9c8c0ffa20 0x7f9c8c197a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.46:36546/0 (socket says 172.21.15.46:36546) 2024-03-31T23:10:55.134 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:55.130+0000 7f9c91a51640 1 -- 172.21.15.46:0/4161583140 learned_addr learned my addr 172.21.15.46:0/4161583140 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:10:55.134 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:55.130+0000 7f9c92252640 1 --2- 172.21.15.46:0/4161583140 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9c8c1033b0 0x7f9c8c19efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:10:55.134 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:55.130+0000 7f9c91a51640 1 -- 172.21.15.46:0/4161583140 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9c8c1033b0 msgr2=0x7f9c8c19efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:10:55.134 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:55.130+0000 7f9c91a51640 1 --2- 172.21.15.46:0/4161583140 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9c8c1033b0 0x7f9c8c19efe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:55.134 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:55.130+0000 7f9c91a51640 1 -- 172.21.15.46:0/4161583140 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9c8c1029f0 msgr2=0x7f9c8c197f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:10:55.134 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:55.130+0000 7f9c91a51640 1 --2- 172.21.15.46:0/4161583140 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9c8c1029f0 0x7f9c8c197f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:55.134 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:55.130+0000 7f9c91a51640 1 -- 172.21.15.46:0/4161583140 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9c8c19f650 con 0x7f9c8c0ffa20 2024-03-31T23:10:55.134 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:55.131+0000 7f9c91a51640 1 --2- 172.21.15.46:0/4161583140 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9c8c0ffa20 0x7f9c8c197a30 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f9c880040c0 tx=0x7f9c880040f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:10:55.134 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:55.131+0000 7f9c82ffd640 1 -- 172.21.15.46:0/4161583140 <== mon.2 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c88041020 con 0x7f9c8c0ffa20 2024-03-31T23:10:55.134 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:55.131+0000 7f9c93cb5640 1 -- 172.21.15.46:0/4161583140 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9c8c19f8e0 con 0x7f9c8c0ffa20 2024-03-31T23:10:55.134 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:55.131+0000 7f9c82ffd640 1 -- 172.21.15.46:0/4161583140 <== mon.2 v2:172.21.15.142:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9c8802c9c0 con 0x7f9c8c0ffa20 2024-03-31T23:10:55.134 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:55.131+0000 7f9c82ffd640 1 -- 172.21.15.46:0/4161583140 <== mon.2 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c88039040 con 0x7f9c8c0ffa20 2024-03-31T23:10:55.135 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:55.131+0000 7f9c93cb5640 1 -- 172.21.15.46:0/4161583140 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9c8c19fdd0 con 0x7f9c8c0ffa20 2024-03-31T23:10:55.135 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:55.132+0000 7f9c93cb5640 1 -- 172.21.15.46:0/4161583140 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9c8c108c80 con 0x7f9c8c0ffa20 2024-03-31T23:10:55.135 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:55.133+0000 7f9c82ffd640 1 -- 172.21.15.46:0/4161583140 <== mon.2 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 18) ==== 137320+0+0 (secure 0 0 0) 0x7f9c88044020 con 0x7f9c8c0ffa20 2024-03-31T23:10:55.135 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:55.133+0000 7f9c82ffd640 1 --2- 172.21.15.46:0/4161583140 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f9c780a24a0 0x7f9c780a4960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:10:55.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:55.133+0000 7f9c91250640 1 --2- 172.21.15.46:0/4161583140 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f9c780a24a0 0x7f9c780a4960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:10:55.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:55.134+0000 7f9c82ffd640 1 -- 172.21.15.46:0/4161583140 <== mon.2 v2:172.21.15.142:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7f9c880f0340 con 0x7f9c8c0ffa20 2024-03-31T23:10:55.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:55.134+0000 7f9c91250640 1 --2- 172.21.15.46:0/4161583140 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f9c780a24a0 0x7f9c780a4960 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f9c8c198f40 tx=0x7f9c7c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:10:55.139 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:55.138+0000 7f9c82ffd640 1 -- 172.21.15.46:0/4161583140 <== mon.2 v2:172.21.15.142:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f9c8804b020 con 0x7f9c8c0ffa20 2024-03-31T23:10:55.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:55.322+0000 7f9c93cb5640 1 -- 172.21.15.46:0/4161583140 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f9c8c062350 con 0x7f9c8c0ffa20 2024-03-31T23:10:55.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:55.323+0000 7f9c82ffd640 1 -- 172.21.15.46:0/4161583140 <== mon.2 v2:172.21.15.142:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v93) ==== 103+0+11 (secure 0 0 0) 0x7f9c880bd240 con 0x7f9c8c0ffa20 2024-03-31T23:10:55.324 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 32 2024-03-31T23:10:55.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:55.325+0000 7f9c93cb5640 1 -- 172.21.15.46:0/4161583140 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f9c780a24a0 msgr2=0x7f9c780a4960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:10:55.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:55.325+0000 7f9c93cb5640 1 --2- 172.21.15.46:0/4161583140 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f9c780a24a0 0x7f9c780a4960 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f9c8c198f40 tx=0x7f9c7c009260 comp rx=0 tx=0).stop 2024-03-31T23:10:55.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:55.325+0000 7f9c93cb5640 1 -- 172.21.15.46:0/4161583140 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9c8c0ffa20 msgr2=0x7f9c8c197a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:10:55.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:55.325+0000 7f9c93cb5640 1 --2- 172.21.15.46:0/4161583140 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9c8c0ffa20 0x7f9c8c197a30 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f9c880040c0 tx=0x7f9c880040f0 comp rx=0 tx=0).stop 2024-03-31T23:10:55.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:55.325+0000 7f9c93cb5640 1 -- 172.21.15.46:0/4161583140 shutdown_connections 2024-03-31T23:10:55.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:55.325+0000 7f9c93cb5640 1 --2- 172.21.15.46:0/4161583140 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f9c780a24a0 0x7f9c780a4960 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:55.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:55.325+0000 7f9c93cb5640 1 --2- 172.21.15.46:0/4161583140 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9c8c1033b0 0x7f9c8c19efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:55.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:55.325+0000 7f9c93cb5640 1 --2- 172.21.15.46:0/4161583140 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9c8c1029f0 0x7f9c8c197f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:55.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:55.325+0000 7f9c93cb5640 1 --2- 172.21.15.46:0/4161583140 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9c8c0ffa20 0x7f9c8c197a30 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:55.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:55.325+0000 7f9c93cb5640 1 -- 172.21.15.46:0/4161583140 >> 172.21.15.46:0/4161583140 conn(0x7f9c8c0fb370 msgr2=0x7f9c8c105e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:10:55.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:55.325+0000 7f9c93cb5640 1 -- 172.21.15.46:0/4161583140 shutdown_connections 2024-03-31T23:10:55.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:55.325+0000 7f9c93cb5640 1 -- 172.21.15.46:0/4161583140 wait complete. 2024-03-31T23:10:55.337 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-31T23:10:55.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:55.440+0000 7f6a9b20e640 1 -- 172.21.15.46:0/827993808 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6a94102220 msgr2=0x7f6a94108ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:10:55.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:55.440+0000 7f6a9b20e640 1 --2- 172.21.15.46:0/827993808 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6a94102220 0x7f6a94108ab0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f6a8c009ca0 tx=0x7f6a8c02c0f0 comp rx=0 tx=0).stop 2024-03-31T23:10:55.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:55.440+0000 7f6a9b20e640 1 -- 172.21.15.46:0/827993808 shutdown_connections 2024-03-31T23:10:55.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:55.440+0000 7f6a9b20e640 1 --2- 172.21.15.46:0/827993808 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6a94102220 0x7f6a94108ab0 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:55.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:55.440+0000 7f6a9b20e640 1 --2- 172.21.15.46:0/827993808 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6a94101860 0x7f6a94101ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:55.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:55.440+0000 7f6a9b20e640 1 --2- 172.21.15.46:0/827993808 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6a94100670 0x7f6a94100a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:55.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:55.440+0000 7f6a9b20e640 1 -- 172.21.15.46:0/827993808 >> 172.21.15.46:0/827993808 conn(0x7f6a940fbc00 msgr2=0x7f6a940fe020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:10:55.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:55.440+0000 7f6a9b20e640 1 -- 172.21.15.46:0/827993808 shutdown_connections 2024-03-31T23:10:55.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:55.440+0000 7f6a9b20e640 1 -- 172.21.15.46:0/827993808 wait complete. 2024-03-31T23:10:55.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:55.440+0000 7f6a9b20e640 1 Processor -- start 2024-03-31T23:10:55.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:55.440+0000 7f6a9b20e640 1 -- start start 2024-03-31T23:10:55.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:55.440+0000 7f6a9b20e640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6a94100670 0x7f6a94196850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:10:55.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:55.440+0000 7f6a9b20e640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6a94101860 0x7f6a94196d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:10:55.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:55.440+0000 7f6a9b20e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6a94102220 0x7f6a9419de00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:10:55.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:55.440+0000 7f6a9b20e640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f6a9410c180 con 0x7f6a94102220 2024-03-31T23:10:55.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:55.440+0000 7f6a9b20e640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f6a9410c000 con 0x7f6a94100670 2024-03-31T23:10:55.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:55.440+0000 7f6a9b20e640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f6a9410c300 con 0x7f6a94101860 2024-03-31T23:10:55.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:55.440+0000 7f6a997ab640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6a94102220 0x7f6a9419de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:10:55.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:55.441+0000 7f6a98faa640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6a94100670 0x7f6a94196850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:10:55.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:55.441+0000 7f6a98faa640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6a94100670 0x7f6a94196850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.46:36556/0 (socket says 172.21.15.46:36556) 2024-03-31T23:10:55.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:55.441+0000 7f6a997ab640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6a94102220 0x7f6a9419de00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:56268/0 (socket says 172.21.15.46:56268) 2024-03-31T23:10:55.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:55.441+0000 7f6a997ab640 1 -- 172.21.15.46:0/3346041175 learned_addr learned my addr 172.21.15.46:0/3346041175 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:10:55.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:55.441+0000 7f6a93fff640 1 --2- 172.21.15.46:0/3346041175 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6a94101860 0x7f6a94196d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:10:55.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:55.441+0000 7f6a997ab640 1 -- 172.21.15.46:0/3346041175 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6a94101860 msgr2=0x7f6a94196d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:10:55.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:55.441+0000 7f6a997ab640 1 --2- 172.21.15.46:0/3346041175 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6a94101860 0x7f6a94196d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:55.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:55.441+0000 7f6a997ab640 1 -- 172.21.15.46:0/3346041175 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6a94100670 msgr2=0x7f6a94196850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:10:55.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:55.441+0000 7f6a997ab640 1 --2- 172.21.15.46:0/3346041175 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6a94100670 0x7f6a94196850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:55.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:55.441+0000 7f6a997ab640 1 -- 172.21.15.46:0/3346041175 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6a9419e470 con 0x7f6a94102220 2024-03-31T23:10:55.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:55.441+0000 7f6a997ab640 1 --2- 172.21.15.46:0/3346041175 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6a94102220 0x7f6a9419de00 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f6a8c02c700 tx=0x7f6a8c0328a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:10:55.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:55.441+0000 7f6a91ffb640 1 -- 172.21.15.46:0/3346041175 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a8c041020 con 0x7f6a94102220 2024-03-31T23:10:55.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:55.441+0000 7f6a91ffb640 1 -- 172.21.15.46:0/3346041175 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6a8c003e20 con 0x7f6a94102220 2024-03-31T23:10:55.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:55.441+0000 7f6a9b20e640 1 -- 172.21.15.46:0/3346041175 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6a9419e700 con 0x7f6a94102220 2024-03-31T23:10:55.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:55.441+0000 7f6a91ffb640 1 -- 172.21.15.46:0/3346041175 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a8c039040 con 0x7f6a94102220 2024-03-31T23:10:55.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:55.441+0000 7f6a9b20e640 1 -- 172.21.15.46:0/3346041175 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6a9419ec40 con 0x7f6a94102220 2024-03-31T23:10:55.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:55.442+0000 7f6a9b20e640 1 -- 172.21.15.46:0/3346041175 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6a94107a10 con 0x7f6a94102220 2024-03-31T23:10:55.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:55.443+0000 7f6a91ffb640 1 -- 172.21.15.46:0/3346041175 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 18) ==== 137320+0+0 (secure 0 0 0) 0x7f6a8c044020 con 0x7f6a94102220 2024-03-31T23:10:55.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:55.444+0000 7f6a91ffb640 1 --2- 172.21.15.46:0/3346041175 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f6a840a2520 0x7f6a840a49e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:10:55.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:55.444+0000 7f6a98faa640 1 --2- 172.21.15.46:0/3346041175 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f6a840a2520 0x7f6a840a49e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:10:55.447 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:55.444+0000 7f6a91ffb640 1 -- 172.21.15.46:0/3346041175 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7f6a8c0f0b60 con 0x7f6a94102220 2024-03-31T23:10:55.447 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:55.444+0000 7f6a98faa640 1 --2- 172.21.15.46:0/3346041175 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f6a840a2520 0x7f6a840a49e0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f6a941016c0 tx=0x7f6a88005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:10:55.449 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:55.448+0000 7f6a91ffb640 1 -- 172.21.15.46:0/3346041175 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f6a8c04b020 con 0x7f6a94102220 2024-03-31T23:10:55.611 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:10:55 smithi046 ceph-mon[32498]: from='client.? 172.21.15.46:0/1171005682' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-31T23:10:55.611 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:10:55 smithi046 ceph-mon[32498]: from='client.? 172.21.15.46:0/338477404' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-31T23:10:55.611 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:10:55 smithi046 ceph-mon[32498]: from='client.? 172.21.15.46:0/4161583140' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-31T23:10:55.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:55.670+0000 7f6a9b20e640 1 -- 172.21.15.46:0/3346041175 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6a94062350 con 0x7f6a94102220 2024-03-31T23:10:55.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:55.671+0000 7f6a91ffb640 1 -- 172.21.15.46:0/3346041175 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+11882 (secure 0 0 0) 0x7f6a8c0bd9c0 con 0x7f6a94102220 2024-03-31T23:10:55.672 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-31T23:10:55.672 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/2169018547","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":2169018547},{"type":"v1","addr":"172.21.15.46:6835","nonce":2169018547}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6837/1069387756","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6836","nonce":1069387756},{"type":"v1","addr":"172.21.15.46:6837","nonce":1069387756}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14634,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6839/3915337844","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6838","nonce":3915337844},{"type":"v1","addr":"172.21.15.46:6839","nonce":3915337844}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14646,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6841/897767145","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6840","nonce":897767145},{"type":"v1","addr":"172.21.15.46:6841","nonce":897767145}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24415,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6833/1746950966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6832","nonce":1746950966},{"type":"v1","addr":"172.21.15.162:6833","nonce":1746950966}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24422,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6833/3394739181","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6832","nonce":3394739181},{"type":"v1","addr":"172.21.15.142:6833","nonce":3394739181}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":24,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-31T23:09:11.009090+0000","modified":"2024-03-31T23:09:31.392282+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24515,"mds_1":24482,"mds_2":24451},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24442":{"gid":24442,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.162:6835/1957208984","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6834","nonce":1957208984},{"type":"v1","addr":"172.21.15.162:6835","nonce":1957208984}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24451":{"gid":24451,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":11,"addr":"172.21.15.162:6837/1371886876","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6836","nonce":1371886876},{"type":"v1","addr":"172.21.15.162:6837","nonce":1371886876}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24455":{"gid":24455,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.142:6835/212637431","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6834","nonce":212637431},{"type":"v1","addr":"172.21.15.142:6835","nonce":212637431}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24478":{"gid":24478,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.162:6839/1976275826","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6838","nonce":1976275826},{"type":"v1","addr":"172.21.15.162:6839","nonce":1976275826}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24482":{"gid":24482,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":10,"addr":"172.21.15.142:6837/2559258399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6836","nonce":2559258399},{"type":"v1","addr":"172.21.15.142:6837","nonce":2559258399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24515":{"gid":24515,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.142:6839/2519143918","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6838","nonce":2519143918},{"type":"v1","addr":"172.21.15.142:6839","nonce":2519143918}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24515,"qdb_cluster":[24451,24482,24515]},"id":1}]} 2024-03-31T23:10:55.672 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 24 2024-03-31T23:10:55.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:55.673+0000 7f6a9b20e640 1 -- 172.21.15.46:0/3346041175 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f6a840a2520 msgr2=0x7f6a840a49e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:10:55.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:55.673+0000 7f6a9b20e640 1 --2- 172.21.15.46:0/3346041175 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f6a840a2520 0x7f6a840a49e0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f6a941016c0 tx=0x7f6a88005f90 comp rx=0 tx=0).stop 2024-03-31T23:10:55.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:55.673+0000 7f6a9b20e640 1 -- 172.21.15.46:0/3346041175 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6a94102220 msgr2=0x7f6a9419de00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:10:55.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:55.673+0000 7f6a9b20e640 1 --2- 172.21.15.46:0/3346041175 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6a94102220 0x7f6a9419de00 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f6a8c02c700 tx=0x7f6a8c0328a0 comp rx=0 tx=0).stop 2024-03-31T23:10:55.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:55.673+0000 7f6a9b20e640 1 -- 172.21.15.46:0/3346041175 shutdown_connections 2024-03-31T23:10:55.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:55.673+0000 7f6a9b20e640 1 --2- 172.21.15.46:0/3346041175 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f6a840a2520 0x7f6a840a49e0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:55.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:55.673+0000 7f6a9b20e640 1 --2- 172.21.15.46:0/3346041175 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6a94102220 0x7f6a9419de00 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:55.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:55.673+0000 7f6a9b20e640 1 --2- 172.21.15.46:0/3346041175 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6a94101860 0x7f6a94196d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:55.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:55.673+0000 7f6a9b20e640 1 --2- 172.21.15.46:0/3346041175 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6a94100670 0x7f6a94196850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:55.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:55.673+0000 7f6a9b20e640 1 -- 172.21.15.46:0/3346041175 >> 172.21.15.46:0/3346041175 conn(0x7f6a940fbc00 msgr2=0x7f6a94104be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:10:55.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:55.673+0000 7f6a9b20e640 1 -- 172.21.15.46:0/3346041175 shutdown_connections 2024-03-31T23:10:55.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:55.673+0000 7f6a9b20e640 1 -- 172.21.15.46:0/3346041175 wait complete. 2024-03-31T23:10:55.686 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-31T23:10:55.797 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:10:55 smithi162 ceph-mon[36237]: from='client.? 172.21.15.46:0/1171005682' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-31T23:10:55.797 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:10:55 smithi162 ceph-mon[36237]: from='client.? 172.21.15.46:0/338477404' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-31T23:10:55.797 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:10:55 smithi162 ceph-mon[36237]: from='client.? 172.21.15.46:0/4161583140' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-31T23:10:55.797 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:55.788+0000 7fa315be9640 1 -- 172.21.15.46:0/311511385 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa3100ffa40 msgr2=0x7fa3100ffe60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:10:55.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:55.788+0000 7fa315be9640 1 --2- 172.21.15.46:0/311511385 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa3100ffa40 0x7fa3100ffe60 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7fa300009930 tx=0x7fa30002bc30 comp rx=0 tx=0).stop 2024-03-31T23:10:55.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:55.788+0000 7fa315be9640 1 -- 172.21.15.46:0/311511385 shutdown_connections 2024-03-31T23:10:55.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:55.788+0000 7fa315be9640 1 --2- 172.21.15.46:0/311511385 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa3101033d0 0x7fa310109c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:55.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:55.788+0000 7fa315be9640 1 --2- 172.21.15.46:0/311511385 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa310102a10 0x7fa310102e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:55.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:55.788+0000 7fa315be9640 1 --2- 172.21.15.46:0/311511385 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa3100ffa40 0x7fa3100ffe60 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:55.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:55.788+0000 7fa315be9640 1 -- 172.21.15.46:0/311511385 >> 172.21.15.46:0/311511385 conn(0x7fa3100fb370 msgr2=0x7fa3100fd7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:10:55.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:55.788+0000 7fa315be9640 1 -- 172.21.15.46:0/311511385 shutdown_connections 2024-03-31T23:10:55.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:55.788+0000 7fa315be9640 1 -- 172.21.15.46:0/311511385 wait complete. 2024-03-31T23:10:55.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:55.789+0000 7fa315be9640 1 Processor -- start 2024-03-31T23:10:55.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:55.789+0000 7fa315be9640 1 -- start start 2024-03-31T23:10:55.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:55.789+0000 7fa315be9640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa3100ffa40 0x7fa310197a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:10:55.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:55.789+0000 7fa315be9640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa310102a10 0x7fa310197f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:10:55.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:55.789+0000 7fa315be9640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa3101033d0 0x7fa31019efd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:10:55.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:55.789+0000 7fa315be9640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fa31010d2a0 con 0x7fa310102a10 2024-03-31T23:10:55.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:55.789+0000 7fa315be9640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fa31010d120 con 0x7fa3100ffa40 2024-03-31T23:10:55.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:55.789+0000 7fa315be9640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fa31010d420 con 0x7fa3101033d0 2024-03-31T23:10:55.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:55.789+0000 7fa30effd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa310102a10 0x7fa310197f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:10:55.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:55.789+0000 7fa30effd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa310102a10 0x7fa310197f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:56284/0 (socket says 172.21.15.46:56284) 2024-03-31T23:10:55.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:55.789+0000 7fa30effd640 1 -- 172.21.15.46:0/1877529546 learned_addr learned my addr 172.21.15.46:0/1877529546 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:10:55.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:55.789+0000 7fa30f7fe640 1 --2- 172.21.15.46:0/1877529546 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa3100ffa40 0x7fa310197a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:10:55.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:55.789+0000 7fa30ffff640 1 --2- 172.21.15.46:0/1877529546 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa3101033d0 0x7fa31019efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:10:55.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:55.789+0000 7fa30effd640 1 -- 172.21.15.46:0/1877529546 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa3101033d0 msgr2=0x7fa31019efd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:10:55.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:55.789+0000 7fa30effd640 1 --2- 172.21.15.46:0/1877529546 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa3101033d0 0x7fa31019efd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:55.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:55.789+0000 7fa30effd640 1 -- 172.21.15.46:0/1877529546 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa3100ffa40 msgr2=0x7fa310197a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:10:55.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:55.789+0000 7fa30effd640 1 --2- 172.21.15.46:0/1877529546 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa3100ffa40 0x7fa310197a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:55.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:55.789+0000 7fa30effd640 1 -- 172.21.15.46:0/1877529546 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa31019f640 con 0x7fa310102a10 2024-03-31T23:10:55.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:55.790+0000 7fa30effd640 1 --2- 172.21.15.46:0/1877529546 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa310102a10 0x7fa310197f60 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fa30400e9f0 tx=0x7fa30400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:10:55.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:55.790+0000 7fa30cff9640 1 -- 172.21.15.46:0/1877529546 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa30400cd80 con 0x7fa310102a10 2024-03-31T23:10:55.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:55.790+0000 7fa30cff9640 1 -- 172.21.15.46:0/1877529546 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa3040108e0 con 0x7fa310102a10 2024-03-31T23:10:55.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:55.790+0000 7fa315be9640 1 -- 172.21.15.46:0/1877529546 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa31019f930 con 0x7fa310102a10 2024-03-31T23:10:55.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:55.790+0000 7fa30cff9640 1 -- 172.21.15.46:0/1877529546 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa304018920 con 0x7fa310102a10 2024-03-31T23:10:55.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:55.790+0000 7fa315be9640 1 -- 172.21.15.46:0/1877529546 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa31019fe80 con 0x7fa310102a10 2024-03-31T23:10:55.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:55.791+0000 7fa315be9640 1 -- 172.21.15.46:0/1877529546 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa310108c10 con 0x7fa310102a10 2024-03-31T23:10:55.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:55.792+0000 7fa30cff9640 1 -- 172.21.15.46:0/1877529546 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 18) ==== 137320+0+0 (secure 0 0 0) 0x7fa304010a90 con 0x7fa310102a10 2024-03-31T23:10:55.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:55.792+0000 7fa30cff9640 1 --2- 172.21.15.46:0/1877529546 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fa2f80a24a0 0x7fa2f80a4960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:10:55.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:55.793+0000 7fa30f7fe640 1 --2- 172.21.15.46:0/1877529546 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fa2f80a24a0 0x7fa2f80a4960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:10:55.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:55.793+0000 7fa30cff9640 1 -- 172.21.15.46:0/1877529546 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7fa3040ced50 con 0x7fa310102a10 2024-03-31T23:10:55.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:55.793+0000 7fa30f7fe640 1 --2- 172.21.15.46:0/1877529546 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fa2f80a24a0 0x7fa2f80a4960 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fa300000c00 tx=0x7fa300005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:10:55.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:55.797+0000 7fa30cff9640 1 -- 172.21.15.46:0/1877529546 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fa3040d2030 con 0x7fa310102a10 2024-03-31T23:10:55.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:10:55 smithi142 ceph-mon[35271]: from='client.? 172.21.15.46:0/1171005682' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-31T23:10:55.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:10:55 smithi142 ceph-mon[35271]: from='client.? 172.21.15.46:0/338477404' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-31T23:10:55.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:10:55 smithi142 ceph-mon[35271]: from='client.? 172.21.15.46:0/4161583140' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-31T23:10:55.980 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:55.979+0000 7fa315be9640 1 -- 172.21.15.46:0/1877529546 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fa310062350 con 0x7fa310102a10 2024-03-31T23:10:55.982 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:55.981+0000 7fa30cff9640 1 -- 172.21.15.46:0/1877529546 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v93) ==== 74+0+24575 (secure 0 0 0) 0x7fa30409b560 con 0x7fa310102a10 2024-03-31T23:10:55.983 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-31T23:10:55.983 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":93,"fsid":"6a9d8f08-efb2-11ee-b647-cb9ed24678a4","created":"2024-03-31T23:01:15.463711+0000","modified":"2024-03-31T23:10:10.474156+0000","last_up_change":"2024-03-31T23:08:20.199888+0000","last_in_change":"2024-03-31T23:07:59.438035+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-31T23:04:32.823751+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-31T23:09:09.145711+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-31T23:09:10.148579+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-31T23:09:13.419519+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"91","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"47429095-3f46-4ed7-b7c4-504464719815","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":1824772695},{"type":"v1","addr":"172.21.15.46:6803","nonce":1824772695}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":1824772695},{"type":"v1","addr":"172.21.15.46:6805","nonce":1824772695}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":1824772695},{"type":"v1","addr":"172.21.15.46:6809","nonce":1824772695}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":1824772695},{"type":"v1","addr":"172.21.15.46:6807","nonce":1824772695}]},"public_addr":"172.21.15.46:6803/1824772695","cluster_addr":"172.21.15.46:6805/1824772695","heartbeat_back_addr":"172.21.15.46:6809/1824772695","heartbeat_front_addr":"172.21.15.46:6807/1824772695","state":["exists","up"]},{"osd":1,"uuid":"be56f883-411e-4dbe-b50e-a158134df4a6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6800","nonce":973693922},{"type":"v1","addr":"172.21.15.142:6801","nonce":973693922}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6802","nonce":973693922},{"type":"v1","addr":"172.21.15.142:6803","nonce":973693922}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6806","nonce":973693922},{"type":"v1","addr":"172.21.15.142:6807","nonce":973693922}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6804","nonce":973693922},{"type":"v1","addr":"172.21.15.142:6805","nonce":973693922}]},"public_addr":"172.21.15.142:6801/973693922","cluster_addr":"172.21.15.142:6803/973693922","heartbeat_back_addr":"172.21.15.142:6807/973693922","heartbeat_front_addr":"172.21.15.142:6805/973693922","state":["exists","up"]},{"osd":2,"uuid":"c99a8794-ab19-4260-9a8d-4bc0bd2c8175","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6800","nonce":4084261813},{"type":"v1","addr":"172.21.15.162:6801","nonce":4084261813}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6802","nonce":4084261813},{"type":"v1","addr":"172.21.15.162:6803","nonce":4084261813}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6806","nonce":4084261813},{"type":"v1","addr":"172.21.15.162:6807","nonce":4084261813}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6804","nonce":4084261813},{"type":"v1","addr":"172.21.15.162:6805","nonce":4084261813}]},"public_addr":"172.21.15.162:6801/4084261813","cluster_addr":"172.21.15.162:6803/4084261813","heartbeat_back_addr":"172.21.15.162:6807/4084261813","heartbeat_front_addr":"172.21.15.162:6805/4084261813","state":["exists","up"]},{"osd":3,"uuid":"f877814e-9ef6-4872-a081-5609bb9ebc06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":3496317970},{"type":"v1","addr":"172.21.15.46:6811","nonce":3496317970}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":3496317970},{"type":"v1","addr":"172.21.15.46:6813","nonce":3496317970}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":3496317970},{"type":"v1","addr":"172.21.15.46:6817","nonce":3496317970}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":3496317970},{"type":"v1","addr":"172.21.15.46:6815","nonce":3496317970}]},"public_addr":"172.21.15.46:6811/3496317970","cluster_addr":"172.21.15.46:6813/3496317970","heartbeat_back_addr":"172.21.15.46:6817/3496317970","heartbeat_front_addr":"172.21.15.46:6815/3496317970","state":["exists","up"]},{"osd":4,"uuid":"beecec0a-8280-44fe-8f5f-8498a35eca8f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6808","nonce":2996266929},{"type":"v1","addr":"172.21.15.142:6809","nonce":2996266929}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6810","nonce":2996266929},{"type":"v1","addr":"172.21.15.142:6811","nonce":2996266929}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6814","nonce":2996266929},{"type":"v1","addr":"172.21.15.142:6815","nonce":2996266929}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6812","nonce":2996266929},{"type":"v1","addr":"172.21.15.142:6813","nonce":2996266929}]},"public_addr":"172.21.15.142:6809/2996266929","cluster_addr":"172.21.15.142:6811/2996266929","heartbeat_back_addr":"172.21.15.142:6815/2996266929","heartbeat_front_addr":"172.21.15.142:6813/2996266929","state":["exists","up"]},{"osd":5,"uuid":"9da7ff75-f658-45ea-a02f-1943fdda76b9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6808","nonce":914208293},{"type":"v1","addr":"172.21.15.162:6809","nonce":914208293}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6810","nonce":914208293},{"type":"v1","addr":"172.21.15.162:6811","nonce":914208293}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6814","nonce":914208293},{"type":"v1","addr":"172.21.15.162:6815","nonce":914208293}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6812","nonce":914208293},{"type":"v1","addr":"172.21.15.162:6813","nonce":914208293}]},"public_addr":"172.21.15.162:6809/914208293","cluster_addr":"172.21.15.162:6811/914208293","heartbeat_back_addr":"172.21.15.162:6815/914208293","heartbeat_front_addr":"172.21.15.162:6813/914208293","state":["exists","up"]},{"osd":6,"uuid":"8fa11224-cb26-4d3f-87d1-6a8f31de2a19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":56291966},{"type":"v1","addr":"172.21.15.46:6819","nonce":56291966}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":56291966},{"type":"v1","addr":"172.21.15.46:6821","nonce":56291966}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6824","nonce":56291966},{"type":"v1","addr":"172.21.15.46:6825","nonce":56291966}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":56291966},{"type":"v1","addr":"172.21.15.46:6823","nonce":56291966}]},"public_addr":"172.21.15.46:6819/56291966","cluster_addr":"172.21.15.46:6821/56291966","heartbeat_back_addr":"172.21.15.46:6825/56291966","heartbeat_front_addr":"172.21.15.46:6823/56291966","state":["exists","up"]},{"osd":7,"uuid":"0d0338b7-4d0a-4bb0-b152-8f5224871d9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6816","nonce":2146598446},{"type":"v1","addr":"172.21.15.142:6817","nonce":2146598446}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6818","nonce":2146598446},{"type":"v1","addr":"172.21.15.142:6819","nonce":2146598446}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6822","nonce":2146598446},{"type":"v1","addr":"172.21.15.142:6823","nonce":2146598446}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6820","nonce":2146598446},{"type":"v1","addr":"172.21.15.142:6821","nonce":2146598446}]},"public_addr":"172.21.15.142:6817/2146598446","cluster_addr":"172.21.15.142:6819/2146598446","heartbeat_back_addr":"172.21.15.142:6823/2146598446","heartbeat_front_addr":"172.21.15.142:6821/2146598446","state":["exists","up"]},{"osd":8,"uuid":"8b6f6037-892a-416c-babb-7552648d0b3a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6816","nonce":1173078318},{"type":"v1","addr":"172.21.15.162:6817","nonce":1173078318}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6818","nonce":1173078318},{"type":"v1","addr":"172.21.15.162:6819","nonce":1173078318}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6822","nonce":1173078318},{"type":"v1","addr":"172.21.15.162:6823","nonce":1173078318}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6820","nonce":1173078318},{"type":"v1","addr":"172.21.15.162:6821","nonce":1173078318}]},"public_addr":"172.21.15.162:6817/1173078318","cluster_addr":"172.21.15.162:6819/1173078318","heartbeat_back_addr":"172.21.15.162:6823/1173078318","heartbeat_front_addr":"172.21.15.162:6821/1173078318","state":["exists","up"]},{"osd":9,"uuid":"faccce25-6bf2-4220-8858-cda82b72be68","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":532708881},{"type":"v1","addr":"172.21.15.46:6827","nonce":532708881}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6828","nonce":532708881},{"type":"v1","addr":"172.21.15.46:6829","nonce":532708881}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":532708881},{"type":"v1","addr":"172.21.15.46:6833","nonce":532708881}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":532708881},{"type":"v1","addr":"172.21.15.46:6831","nonce":532708881}]},"public_addr":"172.21.15.46:6827/532708881","cluster_addr":"172.21.15.46:6829/532708881","heartbeat_back_addr":"172.21.15.46:6833/532708881","heartbeat_front_addr":"172.21.15.46:6831/532708881","state":["exists","up"]},{"osd":10,"uuid":"cdd356db-f9eb-4080-95ab-db5d8223e572","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6824","nonce":2845222366},{"type":"v1","addr":"172.21.15.142:6825","nonce":2845222366}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6826","nonce":2845222366},{"type":"v1","addr":"172.21.15.142:6827","nonce":2845222366}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6830","nonce":2845222366},{"type":"v1","addr":"172.21.15.142:6831","nonce":2845222366}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6828","nonce":2845222366},{"type":"v1","addr":"172.21.15.142:6829","nonce":2845222366}]},"public_addr":"172.21.15.142:6825/2845222366","cluster_addr":"172.21.15.142:6827/2845222366","heartbeat_back_addr":"172.21.15.142:6831/2845222366","heartbeat_front_addr":"172.21.15.142:6829/2845222366","state":["exists","up"]},{"osd":11,"uuid":"72104d1c-3ea2-45ad-bc0b-0500f146c06b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6824","nonce":1986323778},{"type":"v1","addr":"172.21.15.162:6825","nonce":1986323778}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6826","nonce":1986323778},{"type":"v1","addr":"172.21.15.162:6827","nonce":1986323778}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6830","nonce":1986323778},{"type":"v1","addr":"172.21.15.162:6831","nonce":1986323778}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6828","nonce":1986323778},{"type":"v1","addr":"172.21.15.162:6829","nonce":1986323778}]},"public_addr":"172.21.15.162:6825/1986323778","cluster_addr":"172.21.15.162:6827/1986323778","heartbeat_back_addr":"172.21.15.162:6831/1986323778","heartbeat_front_addr":"172.21.15.162:6829/1986323778","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:03:44.996331+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:04:07.994738+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:04:30.942255+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:04:55.588884+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:05:19.900600+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:05:43.807442+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:06:09.026976+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:06:34.586784+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:06:59.751605+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:07:26.144495+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:07:52.326909+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:08:17.423042+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.6","mappings":[{"from":4,"to":7}]},{"pgid":"3.c","mappings":[{"from":3,"to":9}]},{"pgid":"3.1a","mappings":[{"from":5,"to":6}]},{"pgid":"3.1b","mappings":[{"from":4,"to":8}]},{"pgid":"3.1d","mappings":[{"from":4,"to":8}]},{"pgid":"3.2a","mappings":[{"from":3,"to":6}]},{"pgid":"3.33","mappings":[{"from":5,"to":9}]},{"pgid":"3.36","mappings":[{"from":4,"to":9}]},{"pgid":"4.17","mappings":[{"from":9,"to":8}]},{"pgid":"4.18","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.46:0/4021039857":"2024-04-01T23:02:08.493619+0000","172.21.15.46:6801/3681182647":"2024-04-01T23:02:08.493619+0000","172.21.15.46:0/4022214806":"2024-04-01T23:02:08.493619+0000","172.21.15.46:6800/1515528215":"2024-04-01T23:01:41.910162+0000","172.21.15.46:6800/3681182647":"2024-04-01T23:02:08.493619+0000","172.21.15.46:0/143885707":"2024-04-01T23:01:41.910162+0000","172.21.15.46:6801/1515528215":"2024-04-01T23:01:41.910162+0000","172.21.15.46:0/1680389778":"2024-04-01T23:01:41.910162+0000","172.21.15.46:0/1782133966":"2024-04-01T23:01:41.910162+0000","172.21.15.46:0/3925209376":"2024-04-01T23:02:08.493619+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-31T23:10:55.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:55.983+0000 7fa315be9640 1 -- 172.21.15.46:0/1877529546 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fa2f80a24a0 msgr2=0x7fa2f80a4960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:10:55.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:55.983+0000 7fa315be9640 1 --2- 172.21.15.46:0/1877529546 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fa2f80a24a0 0x7fa2f80a4960 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fa300000c00 tx=0x7fa300005fd0 comp rx=0 tx=0).stop 2024-03-31T23:10:55.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:55.983+0000 7fa315be9640 1 -- 172.21.15.46:0/1877529546 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa310102a10 msgr2=0x7fa310197f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:10:55.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:55.983+0000 7fa315be9640 1 --2- 172.21.15.46:0/1877529546 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa310102a10 0x7fa310197f60 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fa30400e9f0 tx=0x7fa30400eeb0 comp rx=0 tx=0).stop 2024-03-31T23:10:55.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:55.983+0000 7fa315be9640 1 -- 172.21.15.46:0/1877529546 shutdown_connections 2024-03-31T23:10:55.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:55.983+0000 7fa315be9640 1 --2- 172.21.15.46:0/1877529546 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fa2f80a24a0 0x7fa2f80a4960 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:55.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:55.983+0000 7fa315be9640 1 --2- 172.21.15.46:0/1877529546 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa3101033d0 0x7fa31019efd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:55.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:55.983+0000 7fa315be9640 1 --2- 172.21.15.46:0/1877529546 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa310102a10 0x7fa310197f60 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:55.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:55.983+0000 7fa315be9640 1 --2- 172.21.15.46:0/1877529546 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa3100ffa40 0x7fa310197a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:55.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:55.983+0000 7fa315be9640 1 -- 172.21.15.46:0/1877529546 >> 172.21.15.46:0/1877529546 conn(0x7fa3100fb370 msgr2=0x7fa310105de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:10:55.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:55.984+0000 7fa315be9640 1 -- 172.21.15.46:0/1877529546 shutdown_connections 2024-03-31T23:10:55.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:55.984+0000 7fa315be9640 1 -- 172.21.15.46:0/1877529546 wait complete. 2024-03-31T23:10:55.996 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-31T23:10:56.101 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.098+0000 7f3d49ed4640 1 -- 172.21.15.46:0/1663049761 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3d44103330 msgr2=0x7f3d44109bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:10:56.101 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.098+0000 7f3d49ed4640 1 --2- 172.21.15.46:0/1663049761 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3d44103330 0x7f3d44109bc0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f3d3c009ca0 tx=0x7f3d3c02c0f0 comp rx=0 tx=0).stop 2024-03-31T23:10:56.101 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.098+0000 7f3d49ed4640 1 -- 172.21.15.46:0/1663049761 shutdown_connections 2024-03-31T23:10:56.101 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.098+0000 7f3d49ed4640 1 --2- 172.21.15.46:0/1663049761 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3d44103330 0x7f3d44109bc0 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:56.101 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.098+0000 7f3d49ed4640 1 --2- 172.21.15.46:0/1663049761 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3d44102970 0x7f3d44102df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:56.101 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.098+0000 7f3d49ed4640 1 --2- 172.21.15.46:0/1663049761 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3d44101780 0x7f3d44101ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:56.101 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.098+0000 7f3d49ed4640 1 -- 172.21.15.46:0/1663049761 >> 172.21.15.46:0/1663049761 conn(0x7f3d440fcd30 msgr2=0x7f3d440ff150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:10:56.101 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.098+0000 7f3d49ed4640 1 -- 172.21.15.46:0/1663049761 shutdown_connections 2024-03-31T23:10:56.101 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.098+0000 7f3d49ed4640 1 -- 172.21.15.46:0/1663049761 wait complete. 2024-03-31T23:10:56.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.098+0000 7f3d49ed4640 1 Processor -- start 2024-03-31T23:10:56.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.099+0000 7f3d49ed4640 1 -- start start 2024-03-31T23:10:56.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.099+0000 7f3d49ed4640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3d44101780 0x7f3d44197950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:10:56.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.099+0000 7f3d49ed4640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3d44102970 0x7f3d44197e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:10:56.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.099+0000 7f3d49ed4640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3d44103330 0x7f3d4419ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:10:56.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.099+0000 7f3d49ed4640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f3d4410d2b0 con 0x7f3d44101780 2024-03-31T23:10:56.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.099+0000 7f3d49ed4640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f3d4410d130 con 0x7f3d44103330 2024-03-31T23:10:56.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.099+0000 7f3d49ed4640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f3d4410d430 con 0x7f3d44102970 2024-03-31T23:10:56.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.099+0000 7f3d437fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3d44101780 0x7f3d44197950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:10:56.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.099+0000 7f3d437fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3d44101780 0x7f3d44197950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:56296/0 (socket says 172.21.15.46:56296) 2024-03-31T23:10:56.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.099+0000 7f3d437fe640 1 -- 172.21.15.46:0/4098050552 learned_addr learned my addr 172.21.15.46:0/4098050552 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:10:56.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.099+0000 7f3d42ffd640 1 --2- 172.21.15.46:0/4098050552 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3d44102970 0x7f3d44197e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:10:56.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.099+0000 7f3d43fff640 1 --2- 172.21.15.46:0/4098050552 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3d44103330 0x7f3d4419ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:10:56.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.099+0000 7f3d437fe640 1 -- 172.21.15.46:0/4098050552 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3d44102970 msgr2=0x7f3d44197e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:10:56.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.099+0000 7f3d437fe640 1 --2- 172.21.15.46:0/4098050552 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3d44102970 0x7f3d44197e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:56.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.099+0000 7f3d437fe640 1 -- 172.21.15.46:0/4098050552 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3d44103330 msgr2=0x7f3d4419ef00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:10:56.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.099+0000 7f3d437fe640 1 --2- 172.21.15.46:0/4098050552 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3d44103330 0x7f3d4419ef00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:56.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.099+0000 7f3d437fe640 1 -- 172.21.15.46:0/4098050552 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d4419f570 con 0x7f3d44101780 2024-03-31T23:10:56.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.099+0000 7f3d437fe640 1 --2- 172.21.15.46:0/4098050552 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3d44101780 0x7f3d44197950 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f3d3400e9c0 tx=0x7f3d3400ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:10:56.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.099+0000 7f3d40ff9640 1 -- 172.21.15.46:0/4098050552 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d3400cd80 con 0x7f3d44101780 2024-03-31T23:10:56.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.099+0000 7f3d40ff9640 1 -- 172.21.15.46:0/4098050552 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3d340035f0 con 0x7f3d44101780 2024-03-31T23:10:56.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.099+0000 7f3d49ed4640 1 -- 172.21.15.46:0/4098050552 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3d4419f860 con 0x7f3d44101780 2024-03-31T23:10:56.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.100+0000 7f3d40ff9640 1 -- 172.21.15.46:0/4098050552 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d34010690 con 0x7f3d44101780 2024-03-31T23:10:56.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.100+0000 7f3d49ed4640 1 -- 172.21.15.46:0/4098050552 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3d4419fe00 con 0x7f3d44101780 2024-03-31T23:10:56.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.100+0000 7f3d49ed4640 1 -- 172.21.15.46:0/4098050552 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3d44108b70 con 0x7f3d44101780 2024-03-31T23:10:56.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.101+0000 7f3d40ff9640 1 -- 172.21.15.46:0/4098050552 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 18) ==== 137320+0+0 (secure 0 0 0) 0x7f3d340037a0 con 0x7f3d44101780 2024-03-31T23:10:56.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.102+0000 7f3d40ff9640 1 --2- 172.21.15.46:0/4098050552 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f3d2c0a2520 0x7f3d2c0a49e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:10:56.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.102+0000 7f3d42ffd640 1 --2- 172.21.15.46:0/4098050552 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f3d2c0a2520 0x7f3d2c0a49e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:10:56.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.102+0000 7f3d40ff9640 1 -- 172.21.15.46:0/4098050552 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7f3d340ced70 con 0x7f3d44101780 2024-03-31T23:10:56.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.103+0000 7f3d42ffd640 1 --2- 172.21.15.46:0/4098050552 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f3d2c0a2520 0x7f3d2c0a49e0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f3d44198e60 tx=0x7f3d38005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:10:56.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.107+0000 7f3d40ff9640 1 -- 172.21.15.46:0/4098050552 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f3d3409bbd0 con 0x7f3d44101780 2024-03-31T23:10:56.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.328+0000 7f3d49ed4640 1 -- 172.21.15.46:0/4098050552 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3d4402d170 con 0x7f3d44101780 2024-03-31T23:10:56.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.329+0000 7f3d40ff9640 1 -- 172.21.15.46:0/4098050552 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+11882 (secure 0 0 0) 0x7f3d3409b4d0 con 0x7f3d44101780 2024-03-31T23:10:56.330 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-31T23:10:56.331 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/2169018547","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":2169018547},{"type":"v1","addr":"172.21.15.46:6835","nonce":2169018547}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6837/1069387756","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6836","nonce":1069387756},{"type":"v1","addr":"172.21.15.46:6837","nonce":1069387756}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14634,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6839/3915337844","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6838","nonce":3915337844},{"type":"v1","addr":"172.21.15.46:6839","nonce":3915337844}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14646,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6841/897767145","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6840","nonce":897767145},{"type":"v1","addr":"172.21.15.46:6841","nonce":897767145}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24415,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6833/1746950966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6832","nonce":1746950966},{"type":"v1","addr":"172.21.15.162:6833","nonce":1746950966}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24422,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6833/3394739181","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6832","nonce":3394739181},{"type":"v1","addr":"172.21.15.142:6833","nonce":3394739181}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":24,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-31T23:09:11.009090+0000","modified":"2024-03-31T23:09:31.392282+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24515,"mds_1":24482,"mds_2":24451},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24442":{"gid":24442,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.162:6835/1957208984","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6834","nonce":1957208984},{"type":"v1","addr":"172.21.15.162:6835","nonce":1957208984}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24451":{"gid":24451,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":11,"addr":"172.21.15.162:6837/1371886876","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6836","nonce":1371886876},{"type":"v1","addr":"172.21.15.162:6837","nonce":1371886876}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24455":{"gid":24455,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.142:6835/212637431","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6834","nonce":212637431},{"type":"v1","addr":"172.21.15.142:6835","nonce":212637431}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24478":{"gid":24478,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.162:6839/1976275826","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6838","nonce":1976275826},{"type":"v1","addr":"172.21.15.162:6839","nonce":1976275826}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24482":{"gid":24482,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":10,"addr":"172.21.15.142:6837/2559258399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6836","nonce":2559258399},{"type":"v1","addr":"172.21.15.142:6837","nonce":2559258399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24515":{"gid":24515,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.142:6839/2519143918","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6838","nonce":2519143918},{"type":"v1","addr":"172.21.15.142:6839","nonce":2519143918}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24515,"qdb_cluster":[24451,24482,24515]},"id":1}]} 2024-03-31T23:10:56.331 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 24 2024-03-31T23:10:56.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.331+0000 7f3d49ed4640 1 -- 172.21.15.46:0/4098050552 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f3d2c0a2520 msgr2=0x7f3d2c0a49e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:10:56.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.331+0000 7f3d49ed4640 1 --2- 172.21.15.46:0/4098050552 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f3d2c0a2520 0x7f3d2c0a49e0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f3d44198e60 tx=0x7f3d38005f90 comp rx=0 tx=0).stop 2024-03-31T23:10:56.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.331+0000 7f3d49ed4640 1 -- 172.21.15.46:0/4098050552 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3d44101780 msgr2=0x7f3d44197950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:10:56.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.331+0000 7f3d49ed4640 1 --2- 172.21.15.46:0/4098050552 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3d44101780 0x7f3d44197950 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f3d3400e9c0 tx=0x7f3d3400ee80 comp rx=0 tx=0).stop 2024-03-31T23:10:56.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.331+0000 7f3d49ed4640 1 -- 172.21.15.46:0/4098050552 shutdown_connections 2024-03-31T23:10:56.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.331+0000 7f3d49ed4640 1 --2- 172.21.15.46:0/4098050552 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f3d2c0a2520 0x7f3d2c0a49e0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:56.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.331+0000 7f3d49ed4640 1 --2- 172.21.15.46:0/4098050552 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3d44103330 0x7f3d4419ef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:56.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.331+0000 7f3d49ed4640 1 --2- 172.21.15.46:0/4098050552 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3d44102970 0x7f3d44197e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:56.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.331+0000 7f3d49ed4640 1 --2- 172.21.15.46:0/4098050552 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3d44101780 0x7f3d44197950 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:56.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.331+0000 7f3d49ed4640 1 -- 172.21.15.46:0/4098050552 >> 172.21.15.46:0/4098050552 conn(0x7f3d440fcd30 msgr2=0x7f3d44105d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:10:56.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.332+0000 7f3d49ed4640 1 -- 172.21.15.46:0/4098050552 shutdown_connections 2024-03-31T23:10:56.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.332+0000 7f3d49ed4640 1 -- 172.21.15.46:0/4098050552 wait complete. 2024-03-31T23:10:56.344 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 24, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-03-31T23:09:11.009090+0000', 'modified': '2024-03-31T23:09:31.392282+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 3, 'in': [0, 1, 2], 'up': {'mds_0': 24515, 'mds_1': 24482, 'mds_2': 24451}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24442': {'gid': 24442, 'name': 'f', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.162:6835/1957208984', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6834', 'nonce': 1957208984}, {'type': 'v1', 'addr': '172.21.15.162:6835', 'nonce': 1957208984}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24451': {'gid': 24451, 'name': 'c', 'rank': 2, 'incarnation': 22, 'state': 'up:active', 'state_seq': 11, 'addr': '172.21.15.162:6837/1371886876', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6836', 'nonce': 1371886876}, {'type': 'v1', 'addr': '172.21.15.162:6837', 'nonce': 1371886876}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24455': {'gid': 24455, 'name': 'e', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.142:6835/212637431', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.142:6834', 'nonce': 212637431}, {'type': 'v1', 'addr': '172.21.15.142:6835', 'nonce': 212637431}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24478': {'gid': 24478, 'name': 'i', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.162:6839/1976275826', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6838', 'nonce': 1976275826}, {'type': 'v1', 'addr': '172.21.15.162:6839', 'nonce': 1976275826}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24482': {'gid': 24482, 'name': 'h', 'rank': 1, 'incarnation': 20, 'state': 'up:active', 'state_seq': 10, 'addr': '172.21.15.142:6837/2559258399', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.142:6836', 'nonce': 2559258399}, {'type': 'v1', 'addr': '172.21.15.142:6837', 'nonce': 2559258399}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24515': {'gid': 24515, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.142:6839/2519143918', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.142:6838', 'nonce': 2519143918}, {'type': 'v1', 'addr': '172.21.15.142:6839', 'nonce': 2519143918}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24515, 'qdb_cluster': [24451, 24482, 24515]} 2024-03-31T23:10:56.344 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 3/3 2024-03-31T23:10:56.344 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-03-31T23:10:56.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.447+0000 7f4c2313e640 1 -- 172.21.15.46:0/2841380643 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4c1c100670 msgr2=0x7f4c1c100a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:10:56.452 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.447+0000 7f4c2313e640 1 --2- 172.21.15.46:0/2841380643 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4c1c100670 0x7f4c1c100a90 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f4c10009900 tx=0x7f4c1002bc30 comp rx=0 tx=0).stop 2024-03-31T23:10:56.452 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.447+0000 7f4c2313e640 1 -- 172.21.15.46:0/2841380643 shutdown_connections 2024-03-31T23:10:56.452 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.447+0000 7f4c2313e640 1 --2- 172.21.15.46:0/2841380643 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4c1c102220 0x7f4c1c108ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:56.452 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.447+0000 7f4c2313e640 1 --2- 172.21.15.46:0/2841380643 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4c1c101860 0x7f4c1c101ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:56.452 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.447+0000 7f4c2313e640 1 --2- 172.21.15.46:0/2841380643 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4c1c100670 0x7f4c1c100a90 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:56.452 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.447+0000 7f4c2313e640 1 -- 172.21.15.46:0/2841380643 >> 172.21.15.46:0/2841380643 conn(0x7f4c1c0fbc00 msgr2=0x7f4c1c0fe020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:10:56.452 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.447+0000 7f4c2313e640 1 -- 172.21.15.46:0/2841380643 shutdown_connections 2024-03-31T23:10:56.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.447+0000 7f4c2313e640 1 -- 172.21.15.46:0/2841380643 wait complete. 2024-03-31T23:10:56.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.447+0000 7f4c2313e640 1 Processor -- start 2024-03-31T23:10:56.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.447+0000 7f4c2313e640 1 -- start start 2024-03-31T23:10:56.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.448+0000 7f4c2313e640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4c1c100670 0x7f4c1c06a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:10:56.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.448+0000 7f4c2313e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4c1c101860 0x7f4c1c19c6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:10:56.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.448+0000 7f4c2313e640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4c1c102220 0x7f4c1c19ed30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:10:56.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.448+0000 7f4c2313e640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f4c1c10c1f0 con 0x7f4c1c101860 2024-03-31T23:10:56.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.448+0000 7f4c2313e640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f4c1c10c070 con 0x7f4c1c100670 2024-03-31T23:10:56.454 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.448+0000 7f4c2313e640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f4c1c10c370 con 0x7f4c1c102220 2024-03-31T23:10:56.454 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.448+0000 7f4c1bfff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4c1c101860 0x7f4c1c19c6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:10:56.454 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.448+0000 7f4c1bfff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4c1c101860 0x7f4c1c19c6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:56314/0 (socket says 172.21.15.46:56314) 2024-03-31T23:10:56.454 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.448+0000 7f4c1bfff640 1 -- 172.21.15.46:0/2801756258 learned_addr learned my addr 172.21.15.46:0/2801756258 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:10:56.454 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.448+0000 7f4c20eda640 1 --2- 172.21.15.46:0/2801756258 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4c1c100670 0x7f4c1c06a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:10:56.454 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.448+0000 7f4c216db640 1 --2- 172.21.15.46:0/2801756258 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4c1c102220 0x7f4c1c19ed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:10:56.454 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.448+0000 7f4c1bfff640 1 -- 172.21.15.46:0/2801756258 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4c1c102220 msgr2=0x7f4c1c19ed30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:10:56.454 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.448+0000 7f4c1bfff640 1 --2- 172.21.15.46:0/2801756258 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4c1c102220 0x7f4c1c19ed30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:56.455 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.448+0000 7f4c1bfff640 1 -- 172.21.15.46:0/2801756258 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4c1c100670 msgr2=0x7f4c1c06a8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:10:56.455 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.448+0000 7f4c1bfff640 1 --2- 172.21.15.46:0/2801756258 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4c1c100670 0x7f4c1c06a8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:56.455 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.448+0000 7f4c1bfff640 1 -- 172.21.15.46:0/2801756258 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4c1c19f5f0 con 0x7f4c1c101860 2024-03-31T23:10:56.455 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.448+0000 7f4c1bfff640 1 --2- 172.21.15.46:0/2801756258 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4c1c101860 0x7f4c1c19c6f0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f4c0800d950 tx=0x7f4c0800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:10:56.455 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.448+0000 7f4c19ffb640 1 -- 172.21.15.46:0/2801756258 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c08017070 con 0x7f4c1c101860 2024-03-31T23:10:56.455 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.448+0000 7f4c2313e640 1 -- 172.21.15.46:0/2801756258 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4c1c19f8e0 con 0x7f4c1c101860 2024-03-31T23:10:56.455 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.448+0000 7f4c19ffb640 1 -- 172.21.15.46:0/2801756258 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4c08009c30 con 0x7f4c1c101860 2024-03-31T23:10:56.455 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.448+0000 7f4c19ffb640 1 -- 172.21.15.46:0/2801756258 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c0800f620 con 0x7f4c1c101860 2024-03-31T23:10:56.456 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.449+0000 7f4c2313e640 1 -- 172.21.15.46:0/2801756258 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4c1c19fe30 con 0x7f4c1c101860 2024-03-31T23:10:56.456 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.450+0000 7f4c19ffb640 1 -- 172.21.15.46:0/2801756258 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 18) ==== 137320+0+0 (secure 0 0 0) 0x7f4c080109a0 con 0x7f4c1c101860 2024-03-31T23:10:56.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.451+0000 7f4c19ffb640 1 --2- 172.21.15.46:0/2801756258 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f4c0c0a24a0 0x7f4c0c0a4960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:10:56.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.451+0000 7f4c25df9740 1 Processor -- start 2024-03-31T23:10:56.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.451+0000 7f4c25df9740 1 -- start start 2024-03-31T23:10:56.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.451+0000 7f4c20eda640 1 --2- 172.21.15.46:0/2801756258 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f4c0c0a24a0 0x7f4c0c0a4960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:10:56.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.451+0000 7f4c19ffb640 1 -- 172.21.15.46:0/2801756258 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7f4c080ce6b0 con 0x7f4c1c101860 2024-03-31T23:10:56.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.451+0000 7f4c25df9740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56508d37a610 0x56508d37aa30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:10:56.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.451+0000 7f4c25df9740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56508d37b7e0 0x56508d37bc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:10:56.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.451+0000 7f4c25df9740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56508d37c1a0 0x56508d382a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:10:56.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.451+0000 7f4c25df9740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x56508d2deaf0 con 0x56508d37c1a0 2024-03-31T23:10:56.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.451+0000 7f4c25df9740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x56508d2879d0 con 0x56508d37a610 2024-03-31T23:10:56.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.451+0000 7f4c25df9740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x56508d2dab90 con 0x56508d37b7e0 2024-03-31T23:10:56.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.452+0000 7f4c216db640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56508d37c1a0 0x56508d382a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:10:56.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.452+0000 7f4c1bfff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56508d37b7e0 0x56508d37bc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:10:56.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.452+0000 7f4c216db640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56508d37c1a0 0x56508d382a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:56324/0 (socket says 172.21.15.46:56324) 2024-03-31T23:10:56.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.452+0000 7f4c20eda640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56508d37a610 0x56508d37aa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:10:56.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.452+0000 7f4c216db640 1 -- 172.21.15.46:0/908318518 learned_addr learned my addr 172.21.15.46:0/908318518 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:10:56.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.452+0000 7f4c20eda640 1 --2- 172.21.15.46:0/2801756258 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f4c0c0a24a0 0x7f4c0c0a4960 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f4c1002c240 tx=0x7f4c10005e50 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:10:56.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.452+0000 7f4c20eda640 1 -- 172.21.15.46:0/908318518 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56508d37b7e0 msgr2=0x56508d37bc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:10:56.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.452+0000 7f4c20eda640 1 --2- 172.21.15.46:0/908318518 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56508d37b7e0 0x56508d37bc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:56.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.452+0000 7f4c20eda640 1 -- 172.21.15.46:0/908318518 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56508d37c1a0 msgr2=0x56508d382a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:10:56.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.452+0000 7f4c20eda640 1 --2- 172.21.15.46:0/908318518 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56508d37c1a0 0x56508d382a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:56.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.452+0000 7f4c20eda640 1 -- 172.21.15.46:0/908318518 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56508cff32d0 con 0x56508d37a610 2024-03-31T23:10:56.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.452+0000 7f4c1bfff640 1 --2- 172.21.15.46:0/908318518 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56508d37b7e0 0x56508d37bc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:10:56.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.452+0000 7f4c216db640 1 --2- 172.21.15.46:0/908318518 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56508d37c1a0 0x56508d382a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:10:56.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.452+0000 7f4c20eda640 1 --2- 172.21.15.46:0/908318518 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56508d37a610 0x56508d37aa30 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f4c10036040 tx=0x7f4c10046000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f9a60b5ea1d4b8ad server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:10:56.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.452+0000 7f4c06ffd640 1 -- 172.21.15.46:0/908318518 <== mon.2 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c10047070 con 0x56508d37a610 2024-03-31T23:10:56.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.452+0000 7f4c06ffd640 1 -- 172.21.15.46:0/908318518 <== mon.2 v2:172.21.15.142:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4c10050070 con 0x56508d37a610 2024-03-31T23:10:56.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.452+0000 7f4c06ffd640 1 -- 172.21.15.46:0/908318518 <== mon.2 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c10004e00 con 0x56508d37a610 2024-03-31T23:10:56.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.453+0000 7f4c25df9740 1 -- 172.21.15.46:0/908318518 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56508d37a610 msgr2=0x56508d37aa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:10:56.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.453+0000 7f4c25df9740 1 --2- 172.21.15.46:0/908318518 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56508d37a610 0x56508d37aa30 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f4c10036040 tx=0x7f4c10046000 comp rx=0 tx=0).stop 2024-03-31T23:10:56.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.453+0000 7f4c25df9740 1 -- 172.21.15.46:0/908318518 shutdown_connections 2024-03-31T23:10:56.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.453+0000 7f4c25df9740 1 --2- 172.21.15.46:0/908318518 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56508d37c1a0 0x56508d382a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:56.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.453+0000 7f4c25df9740 1 --2- 172.21.15.46:0/908318518 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56508d37b7e0 0x56508d37bc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:56.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.453+0000 7f4c25df9740 1 --2- 172.21.15.46:0/908318518 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56508d37a610 0x56508d37aa30 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:56.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.453+0000 7f4c25df9740 1 -- 172.21.15.46:0/908318518 >> 172.21.15.46:0/908318518 conn(0x56508d375a00 msgr2=0x56508d375e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:10:56.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.453+0000 7f4c25df9740 1 -- 172.21.15.46:0/908318518 shutdown_connections 2024-03-31T23:10:56.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.453+0000 7f4c25df9740 1 -- 172.21.15.46:0/908318518 wait complete. 2024-03-31T23:10:56.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.453+0000 7f4c25df9740 1 -- start start 2024-03-31T23:10:56.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.453+0000 7f4c25df9740 1 Processor -- start 2024-03-31T23:10:56.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.454+0000 7f4c25df9740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56508d37a610 0x56508d3848c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:10:56.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.454+0000 7f4c25df9740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56508d37b7e0 0x56508d384e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:10:56.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.454+0000 7f4c25df9740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56508d37c1a0 0x56508d38be40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:10:56.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.454+0000 7f4c25df9740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x56508d383150 con 0x56508d37b7e0 2024-03-31T23:10:56.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.454+0000 7f4c25df9740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x56508d29af70 con 0x56508d37c1a0 2024-03-31T23:10:56.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.454+0000 7f4c25df9740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x56508d3832d0 con 0x56508d37a610 2024-03-31T23:10:56.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.454+0000 7f4c25df9740 10 client.0 _ll_register_callbacks cb 0x56508cff2ac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T23:10:56.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.454+0000 7f4c1bfff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56508d37b7e0 0x56508d384e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:10:56.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.454+0000 7f4c1bfff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56508d37b7e0 0x56508d384e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:56330/0 (socket says 172.21.15.46:56330) 2024-03-31T23:10:56.475 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.454+0000 7f4c1bfff640 1 -- 172.21.15.46:0/1917814288 learned_addr learned my addr 172.21.15.46:0/1917814288 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:10:56.475 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.454+0000 7f4c20eda640 1 --2- 172.21.15.46:0/1917814288 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56508d37a610 0x56508d3848c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:10:56.475 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.454+0000 7f4c216db640 1 --2- 172.21.15.46:0/1917814288 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56508d37c1a0 0x56508d38be40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:10:56.475 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.454+0000 7f4c1bfff640 1 -- 172.21.15.46:0/1917814288 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56508d37a610 msgr2=0x56508d3848c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:10:56.475 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.454+0000 7f4c1bfff640 1 --2- 172.21.15.46:0/1917814288 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56508d37a610 0x56508d3848c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:56.475 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.454+0000 7f4c1bfff640 1 -- 172.21.15.46:0/1917814288 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56508d37c1a0 msgr2=0x56508d38be40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:10:56.475 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.454+0000 7f4c1bfff640 1 --2- 172.21.15.46:0/1917814288 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56508d37c1a0 0x56508d38be40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:56.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.454+0000 7f4c1bfff640 1 -- 172.21.15.46:0/1917814288 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56508d0bd6b0 con 0x56508d37b7e0 2024-03-31T23:10:56.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.454+0000 7f4c1bfff640 1 --2- 172.21.15.46:0/1917814288 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56508d37b7e0 0x56508d384e00 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f4c080972c0 tx=0x7f4c0809ef00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aef3d990dad4d790 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:10:56.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.454+0000 7f4bfbfff640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-31T23:10:56.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.454+0000 7f4bfbfff640 1 -- 172.21.15.46:0/1917814288 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c080b2070 con 0x56508d37b7e0 2024-03-31T23:10:56.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.454+0000 7f4bfbfff640 1 -- 172.21.15.46:0/1917814288 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7f4c080974d0 con 0x56508d37b7e0 2024-03-31T23:10:56.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.454+0000 7f4c25df9740 1 -- 172.21.15.46:0/1917814288 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4c10050070 con 0x56508d37b7e0 2024-03-31T23:10:56.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.454+0000 7f4bfbfff640 1 -- 172.21.15.46:0/1917814288 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f4c08020040 con 0x56508d37b7e0 2024-03-31T23:10:56.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.454+0000 7f4c25df9740 10 client.15234 fetch_fsmap learned FSMap version 24 2024-03-31T23:10:56.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.455+0000 7f4c25df9740 1 -- 172.21.15.46:0/1917814288 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x56508d38ee90 con 0x56508d37b7e0 2024-03-31T23:10:56.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.455+0000 7f4bfbfff640 1 -- 172.21.15.46:0/1917814288 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4c10050070 con 0x56508d37b7e0 2024-03-31T23:10:56.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.455+0000 7f4bfbfff640 1 -- 172.21.15.46:0/1917814288 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f4c08010460 con 0x56508d37b7e0 2024-03-31T23:10:56.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.455+0000 7f4c25df9740 10 client.15234 fetch_fsmap finished waiting for FSMap version 24 2024-03-31T23:10:56.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.455+0000 7f4c25df9740 10 client.15234 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-31T23:10:56.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.455+0000 7f4c25df9740 20 client.15234 populate_metadata read hostname 'smithi046' 2024-03-31T23:10:56.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.455+0000 7f4c25df9740 1 --2- 172.21.15.46:0/1917814288 >> [v2:172.21.15.162:6836/1371886876,v1:172.21.15.162:6837/1371886876] conn(0x56508d391490 0x56508d392900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:10:56.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.455+0000 7f4c25df9740 4 client.15234 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-31T23:10:56.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.455+0000 7f4c25df9740 1 -- 172.21.15.46:0/1917814288 --> [v2:172.21.15.162:6836/1371886876,v1:172.21.15.162:6837/1371886876] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56508d2ec030 con 0x56508d391490 2024-03-31T23:10:56.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.455+0000 7f4c20eda640 1 --2- 172.21.15.46:0/1917814288 >> [v2:172.21.15.162:6836/1371886876,v1:172.21.15.162:6837/1371886876] conn(0x56508d391490 0x56508d392900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:10:56.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.455+0000 7f4c20eda640 1 --2- 172.21.15.46:0/1917814288 >> [v2:172.21.15.162:6836/1371886876,v1:172.21.15.162:6837/1371886876] conn(0x56508d391490 0x56508d392900 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=a8712c7a9a7c44e1 server_cookie=dd1b7b40fa075714 in_seq=0 out_seq=0 2024-03-31T23:10:56.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.455+0000 7f4bfbfff640 10 client.15234 ms_handle_connect on v2:172.21.15.162:6836/1371886876 2024-03-31T23:10:56.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.456+0000 7f4bfbfff640 1 -- 172.21.15.46:0/1917814288 <== mds.2 v2:172.21.15.162:6836/1371886876 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x56508d2ec030 con 0x56508d391490 2024-03-31T23:10:56.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.456+0000 7f4bfbfff640 10 client.15234 handle_command_reply: tid=0 2024-03-31T23:10:56.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.456+0000 7f4c25df9740 1 client.15234 shutdown 2024-03-31T23:10:56.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.456+0000 7f4c25df9740 1 -- 172.21.15.46:0/1917814288 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56508d37b7e0 msgr2=0x56508d384e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:10:56.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.456+0000 7f4c25df9740 1 --2- 172.21.15.46:0/1917814288 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56508d37b7e0 0x56508d384e00 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f4c080972c0 tx=0x7f4c0809ef00 comp rx=0 tx=0).stop 2024-03-31T23:10:56.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.456+0000 7f4c25df9740 1 -- 172.21.15.46:0/1917814288 shutdown_connections 2024-03-31T23:10:56.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.456+0000 7f4c25df9740 1 --2- 172.21.15.46:0/1917814288 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56508d37c1a0 0x56508d38be40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:56.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.456+0000 7f4c25df9740 1 --2- 172.21.15.46:0/1917814288 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56508d37b7e0 0x56508d384e00 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:56.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.456+0000 7f4c25df9740 1 --2- 172.21.15.46:0/1917814288 >> [v2:172.21.15.162:6836/1371886876,v1:172.21.15.162:6837/1371886876] conn(0x56508d391490 0x56508d392900 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:56.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.456+0000 7f4bfbfff640 0 client.15234 ms_handle_reset on v2:172.21.15.162:6836/1371886876 2024-03-31T23:10:56.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.456+0000 7f4c25df9740 1 --2- 172.21.15.46:0/1917814288 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56508d37a610 0x56508d3848c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:56.481 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.456+0000 7f4c25df9740 1 -- 172.21.15.46:0/1917814288 >> 172.21.15.46:0/1917814288 conn(0x56508d375a00 msgr2=0x56508d374b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:10:56.481 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.457+0000 7f4c25df9740 1 -- 172.21.15.46:0/1917814288 shutdown_connections 2024-03-31T23:10:56.481 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.457+0000 7f4c25df9740 1 -- 172.21.15.46:0/1917814288 wait complete. 2024-03-31T23:10:56.481 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.457+0000 7f4c25df9740 20 client.15234 trim_cache size 0 max 16384 2024-03-31T23:10:56.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.474+0000 7f4c25df9740 1 Processor -- start 2024-03-31T23:10:56.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.474+0000 7f4c25df9740 1 -- start start 2024-03-31T23:10:56.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.475+0000 7f4c25df9740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56508d37a610 0x56508d375de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:10:56.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.475+0000 7f4c25df9740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56508d37b7e0 0x56508d3742b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:10:56.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.475+0000 7f4c25df9740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56508d37c1a0 0x56508d3747f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:10:56.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.475+0000 7f4c25df9740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x56508d390c40 con 0x56508d37c1a0 2024-03-31T23:10:56.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.475+0000 7f4c25df9740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x56508d391310 con 0x56508d37b7e0 2024-03-31T23:10:56.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.475+0000 7f4c25df9740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x56508d3919e0 con 0x56508d37a610 2024-03-31T23:10:56.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.475+0000 7f4c216db640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56508d37c1a0 0x56508d3747f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:10:56.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.475+0000 7f4c216db640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56508d37c1a0 0x56508d3747f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:56336/0 (socket says 172.21.15.46:56336) 2024-03-31T23:10:56.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.475+0000 7f4c216db640 1 -- 172.21.15.46:0/3088994231 learned_addr learned my addr 172.21.15.46:0/3088994231 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:10:56.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.475+0000 7f4c1bfff640 1 --2- 172.21.15.46:0/3088994231 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56508d37b7e0 0x56508d3742b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:10:56.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.475+0000 7f4c20eda640 1 --2- 172.21.15.46:0/3088994231 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56508d37a610 0x56508d375de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:10:56.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.475+0000 7f4c216db640 1 -- 172.21.15.46:0/3088994231 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56508d37a610 msgr2=0x56508d375de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:10:56.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.475+0000 7f4c216db640 1 --2- 172.21.15.46:0/3088994231 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56508d37a610 0x56508d375de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:56.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.475+0000 7f4c216db640 1 -- 172.21.15.46:0/3088994231 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56508d37b7e0 msgr2=0x56508d3742b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:10:56.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.475+0000 7f4c216db640 1 --2- 172.21.15.46:0/3088994231 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56508d37b7e0 0x56508d3742b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:56.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.475+0000 7f4c216db640 1 -- 172.21.15.46:0/3088994231 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56508d374040 con 0x56508d37c1a0 2024-03-31T23:10:56.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.475+0000 7f4c216db640 1 --2- 172.21.15.46:0/3088994231 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56508d37c1a0 0x56508d3747f0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f4c14013590 tx=0x7f4c14013a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2dea4343a623ae68 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:10:56.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.475+0000 7f4c067fc640 1 -- 172.21.15.46:0/3088994231 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c14004b20 con 0x56508d37c1a0 2024-03-31T23:10:56.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.475+0000 7f4c067fc640 1 -- 172.21.15.46:0/3088994231 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4c14004cc0 con 0x56508d37c1a0 2024-03-31T23:10:56.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.475+0000 7f4c067fc640 1 -- 172.21.15.46:0/3088994231 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c14011560 con 0x56508d37c1a0 2024-03-31T23:10:56.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.476+0000 7f4c25df9740 1 -- 172.21.15.46:0/3088994231 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56508d37c1a0 msgr2=0x56508d3747f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:10:56.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.476+0000 7f4c25df9740 1 --2- 172.21.15.46:0/3088994231 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56508d37c1a0 0x56508d3747f0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f4c14013590 tx=0x7f4c14013a50 comp rx=0 tx=0).stop 2024-03-31T23:10:56.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.476+0000 7f4c25df9740 1 -- 172.21.15.46:0/3088994231 shutdown_connections 2024-03-31T23:10:56.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.476+0000 7f4c25df9740 1 --2- 172.21.15.46:0/3088994231 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56508d37c1a0 0x56508d3747f0 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:56.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.476+0000 7f4c25df9740 1 --2- 172.21.15.46:0/3088994231 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56508d37b7e0 0x56508d3742b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:56.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.476+0000 7f4c25df9740 1 --2- 172.21.15.46:0/3088994231 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56508d37a610 0x56508d375de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:56.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.476+0000 7f4c25df9740 1 -- 172.21.15.46:0/3088994231 >> 172.21.15.46:0/3088994231 conn(0x56508d375a00 msgr2=0x56508d382710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:10:56.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.476+0000 7f4c25df9740 1 -- 172.21.15.46:0/3088994231 shutdown_connections 2024-03-31T23:10:56.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.476+0000 7f4c25df9740 1 -- 172.21.15.46:0/3088994231 wait complete. 2024-03-31T23:10:56.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.476+0000 7f4c25df9740 1 -- start start 2024-03-31T23:10:56.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.476+0000 7f4c25df9740 1 Processor -- start 2024-03-31T23:10:56.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.477+0000 7f4c25df9740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56508d37a610 0x56508d3f3430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:10:56.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.477+0000 7f4c25df9740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56508d37b7e0 0x56508d3f3970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:10:56.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.477+0000 7f4c25df9740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56508d37c1a0 0x56508d3fa2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:10:56.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.477+0000 7f4c25df9740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x56508d392780 con 0x56508d37b7e0 2024-03-31T23:10:56.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.477+0000 7f4c25df9740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x56508d3920b0 con 0x56508d37c1a0 2024-03-31T23:10:56.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.477+0000 7f4c25df9740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x56508d392e50 con 0x56508d37a610 2024-03-31T23:10:56.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.477+0000 7f4c25df9740 10 client.0 _ll_register_callbacks cb 0x7f4c0809a9a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T23:10:56.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.477+0000 7f4c1bfff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56508d37b7e0 0x56508d3f3970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:10:56.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.477+0000 7f4c1bfff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56508d37b7e0 0x56508d3f3970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:56352/0 (socket says 172.21.15.46:56352) 2024-03-31T23:10:56.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.477+0000 7f4c1bfff640 1 -- 172.21.15.46:0/3949810979 learned_addr learned my addr 172.21.15.46:0/3949810979 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:10:56.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.477+0000 7f4c20eda640 1 --2- 172.21.15.46:0/3949810979 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56508d37a610 0x56508d3f3430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:10:56.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.477+0000 7f4c216db640 1 --2- 172.21.15.46:0/3949810979 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56508d37c1a0 0x56508d3fa2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:10:56.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.477+0000 7f4c1bfff640 1 -- 172.21.15.46:0/3949810979 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56508d37a610 msgr2=0x56508d3f3430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:10:56.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.477+0000 7f4c1bfff640 1 --2- 172.21.15.46:0/3949810979 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56508d37a610 0x56508d3f3430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:56.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.477+0000 7f4c1bfff640 1 -- 172.21.15.46:0/3949810979 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56508d37c1a0 msgr2=0x56508d3fa2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:10:56.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.477+0000 7f4c1bfff640 1 --2- 172.21.15.46:0/3949810979 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56508d37c1a0 0x56508d3fa2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:56.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.477+0000 7f4c1bfff640 1 -- 172.21.15.46:0/3949810979 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56508d3fa870 con 0x56508d37b7e0 2024-03-31T23:10:56.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.477+0000 7f4c1bfff640 1 --2- 172.21.15.46:0/3949810979 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56508d37b7e0 0x56508d3f3970 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f4c0809b020 tx=0x7f4c0809ae60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e1299e2b61fb7286 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:10:56.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.477+0000 7f4bfbfff640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-31T23:10:56.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.477+0000 7f4bfbfff640 1 -- 172.21.15.46:0/3949810979 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c08014070 con 0x56508d37b7e0 2024-03-31T23:10:56.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.477+0000 7f4bfbfff640 1 -- 172.21.15.46:0/3949810979 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7f4c08099e60 con 0x56508d37b7e0 2024-03-31T23:10:56.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.477+0000 7f4c25df9740 1 -- 172.21.15.46:0/3949810979 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4c14004cc0 con 0x56508d37b7e0 2024-03-31T23:10:56.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.477+0000 7f4bfbfff640 1 -- 172.21.15.46:0/3949810979 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f4c080b2040 con 0x56508d37b7e0 2024-03-31T23:10:56.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.478+0000 7f4c25df9740 10 client.15240 fetch_fsmap learned FSMap version 24 2024-03-31T23:10:56.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.478+0000 7f4c25df9740 1 -- 172.21.15.46:0/3949810979 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x56508d3fbb70 con 0x56508d37b7e0 2024-03-31T23:10:56.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.478+0000 7f4bfbfff640 1 -- 172.21.15.46:0/3949810979 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4c14004cc0 con 0x56508d37b7e0 2024-03-31T23:10:56.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.478+0000 7f4bfbfff640 1 -- 172.21.15.46:0/3949810979 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f4c08096bb0 con 0x56508d37b7e0 2024-03-31T23:10:56.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.478+0000 7f4c25df9740 10 client.15240 fetch_fsmap finished waiting for FSMap version 24 2024-03-31T23:10:56.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.478+0000 7f4c25df9740 10 client.15240 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-31T23:10:56.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.478+0000 7f4c25df9740 20 client.15240 populate_metadata read hostname 'smithi046' 2024-03-31T23:10:56.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.478+0000 7f4c25df9740 1 --2- 172.21.15.46:0/3949810979 >> [v2:172.21.15.162:6836/1371886876,v1:172.21.15.162:6837/1371886876] conn(0x56508d391490 0x56508d3ff220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:10:56.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.478+0000 7f4c25df9740 4 client.15240 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-31T23:10:56.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.478+0000 7f4c25df9740 1 -- 172.21.15.46:0/3949810979 --> [v2:172.21.15.162:6836/1371886876,v1:172.21.15.162:6837/1371886876] -- command(tid 0: {"prefix": "status"}) -- 0x56508d2e6380 con 0x56508d391490 2024-03-31T23:10:56.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.478+0000 7f4c20eda640 1 --2- 172.21.15.46:0/3949810979 >> [v2:172.21.15.162:6836/1371886876,v1:172.21.15.162:6837/1371886876] conn(0x56508d391490 0x56508d3ff220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:10:56.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.478+0000 7f4bfbfff640 10 client.15240 ms_handle_connect on v2:172.21.15.162:6836/1371886876 2024-03-31T23:10:56.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.478+0000 7f4c20eda640 1 --2- 172.21.15.46:0/3949810979 >> [v2:172.21.15.162:6836/1371886876,v1:172.21.15.162:6837/1371886876] conn(0x56508d391490 0x56508d3ff220 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=fe2742be0e849d67 server_cookie=e93a6aa2cf77bbd7 in_seq=0 out_seq=0 2024-03-31T23:10:56.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.479+0000 7f4bfbfff640 1 -- 172.21.15.46:0/3949810979 <== mds.2 v2:172.21.15.162:6836/1371886876 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x56508d2e6380 con 0x56508d391490 2024-03-31T23:10:56.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.479+0000 7f4bfbfff640 10 client.15240 handle_command_reply: tid=0 2024-03-31T23:10:56.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.479+0000 7f4c25df9740 1 client.15240 shutdown 2024-03-31T23:10:56.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.479+0000 7f4c25df9740 1 -- 172.21.15.46:0/3949810979 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56508d37b7e0 msgr2=0x56508d3f3970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:10:56.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.479+0000 7f4c25df9740 1 --2- 172.21.15.46:0/3949810979 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56508d37b7e0 0x56508d3f3970 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f4c0809b020 tx=0x7f4c0809ae60 comp rx=0 tx=0).stop 2024-03-31T23:10:56.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.479+0000 7f4c25df9740 1 -- 172.21.15.46:0/3949810979 shutdown_connections 2024-03-31T23:10:56.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.479+0000 7f4c25df9740 1 --2- 172.21.15.46:0/3949810979 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56508d37c1a0 0x56508d3fa2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:56.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.479+0000 7f4c25df9740 1 --2- 172.21.15.46:0/3949810979 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56508d37b7e0 0x56508d3f3970 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:56.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.479+0000 7f4c25df9740 1 --2- 172.21.15.46:0/3949810979 >> [v2:172.21.15.162:6836/1371886876,v1:172.21.15.162:6837/1371886876] conn(0x56508d391490 0x56508d3ff220 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:56.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.479+0000 7f4c25df9740 1 --2- 172.21.15.46:0/3949810979 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56508d37a610 0x56508d3f3430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:56.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.479+0000 7f4bfbfff640 0 client.15240 ms_handle_reset on v2:172.21.15.162:6836/1371886876 2024-03-31T23:10:56.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.479+0000 7f4c25df9740 1 -- 172.21.15.46:0/3949810979 >> 172.21.15.46:0/3949810979 conn(0x56508d375a00 msgr2=0x56508d378c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:10:56.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.479+0000 7f4c25df9740 1 -- 172.21.15.46:0/3949810979 shutdown_connections 2024-03-31T23:10:56.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.479+0000 7f4c25df9740 1 -- 172.21.15.46:0/3949810979 wait complete. 2024-03-31T23:10:56.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.479+0000 7f4c25df9740 20 client.15240 trim_cache size 0 max 16384 2024-03-31T23:10:56.493 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-03-31T23:10:56.493 INFO:teuthology.orchestra.run.smithi046.stdout: "cluster_fsid": "6a9d8f08-efb2-11ee-b647-cb9ed24678a4", 2024-03-31T23:10:56.493 INFO:teuthology.orchestra.run.smithi046.stdout: "whoami": 2, 2024-03-31T23:10:56.493 INFO:teuthology.orchestra.run.smithi046.stdout: "id": 24451, 2024-03-31T23:10:56.493 INFO:teuthology.orchestra.run.smithi046.stdout: "want_state": "up:active", 2024-03-31T23:10:56.493 INFO:teuthology.orchestra.run.smithi046.stdout: "state": "up:active", 2024-03-31T23:10:56.493 INFO:teuthology.orchestra.run.smithi046.stdout: "fs_name": "cephfs", 2024-03-31T23:10:56.494 INFO:teuthology.orchestra.run.smithi046.stdout: "rank_uptime": 86.091652288000006, 2024-03-31T23:10:56.494 INFO:teuthology.orchestra.run.smithi046.stdout: "mdsmap_epoch": 24, 2024-03-31T23:10:56.494 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap_epoch": 93, 2024-03-31T23:10:56.494 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap_epoch_barrier": 91, 2024-03-31T23:10:56.494 INFO:teuthology.orchestra.run.smithi046.stdout: "uptime": 123.764803851 2024-03-31T23:10:56.494 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-03-31T23:10:56.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.480+0000 7f4c2313e640 1 -- 172.21.15.46:0/2801756258 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f4c0c0a24a0 msgr2=0x7f4c0c0a4960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:10:56.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.480+0000 7f4c2313e640 1 --2- 172.21.15.46:0/2801756258 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f4c0c0a24a0 0x7f4c0c0a4960 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f4c1002c240 tx=0x7f4c10005e50 comp rx=0 tx=0).stop 2024-03-31T23:10:56.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.480+0000 7f4c2313e640 1 -- 172.21.15.46:0/2801756258 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4c1c101860 msgr2=0x7f4c1c19c6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:10:56.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.480+0000 7f4c2313e640 1 --2- 172.21.15.46:0/2801756258 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4c1c101860 0x7f4c1c19c6f0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f4c0800d950 tx=0x7f4c0800de10 comp rx=0 tx=0).stop 2024-03-31T23:10:56.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.480+0000 7f4c2313e640 1 -- 172.21.15.46:0/2801756258 shutdown_connections 2024-03-31T23:10:56.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.480+0000 7f4c2313e640 1 --2- 172.21.15.46:0/2801756258 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f4c0c0a24a0 0x7f4c0c0a4960 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:56.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.480+0000 7f4c2313e640 1 --2- 172.21.15.46:0/2801756258 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4c1c102220 0x7f4c1c19ed30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:56.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.480+0000 7f4c2313e640 1 --2- 172.21.15.46:0/2801756258 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4c1c101860 0x7f4c1c19c6f0 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:56.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.480+0000 7f4c2313e640 1 --2- 172.21.15.46:0/2801756258 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4c1c100670 0x7f4c1c06a8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:56.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.480+0000 7f4c2313e640 1 -- 172.21.15.46:0/2801756258 >> 172.21.15.46:0/2801756258 conn(0x7f4c1c0fbc00 msgr2=0x7f4c1c104c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:10:56.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.480+0000 7f4c2313e640 1 -- 172.21.15.46:0/2801756258 shutdown_connections 2024-03-31T23:10:56.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.480+0000 7f4c2313e640 1 -- 172.21.15.46:0/2801756258 wait complete. 2024-03-31T23:10:56.496 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.h status 2024-03-31T23:10:56.603 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.600+0000 7f2a5d156640 1 -- 172.21.15.46:0/275221727 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2a58104b00 msgr2=0x7f2a58106ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:10:56.603 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.600+0000 7f2a5d156640 1 --2- 172.21.15.46:0/275221727 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2a58104b00 0x7f2a58106ef0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f2a4c009900 tx=0x7f2a4c02bc30 comp rx=0 tx=0).stop 2024-03-31T23:10:56.603 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.600+0000 7f2a5d156640 1 -- 172.21.15.46:0/275221727 shutdown_connections 2024-03-31T23:10:56.604 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.600+0000 7f2a5d156640 1 --2- 172.21.15.46:0/275221727 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2a58107430 0x7f2a58109880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:56.604 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.600+0000 7f2a5d156640 1 --2- 172.21.15.46:0/275221727 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2a58104b00 0x7f2a58106ef0 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:56.604 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.600+0000 7f2a5d156640 1 --2- 172.21.15.46:0/275221727 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2a581021d0 0x7f2a581045c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:56.604 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.600+0000 7f2a5d156640 1 -- 172.21.15.46:0/275221727 >> 172.21.15.46:0/275221727 conn(0x7f2a580fbbe0 msgr2=0x7f2a580fe000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:10:56.604 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.600+0000 7f2a5d156640 1 -- 172.21.15.46:0/275221727 shutdown_connections 2024-03-31T23:10:56.604 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.600+0000 7f2a5d156640 1 -- 172.21.15.46:0/275221727 wait complete. 2024-03-31T23:10:56.604 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.601+0000 7f2a5d156640 1 Processor -- start 2024-03-31T23:10:56.604 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.601+0000 7f2a5d156640 1 -- start start 2024-03-31T23:10:56.604 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.601+0000 7f2a5d156640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2a581021d0 0x7f2a58197a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:10:56.604 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.601+0000 7f2a5d156640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2a58104b00 0x7f2a58197f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:10:56.604 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.601+0000 7f2a5d156640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2a58107430 0x7f2a5819ef90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:10:56.604 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.601+0000 7f2a5d156640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f2a5810d290 con 0x7f2a58104b00 2024-03-31T23:10:56.604 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.601+0000 7f2a5d156640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f2a5810d110 con 0x7f2a58107430 2024-03-31T23:10:56.605 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.601+0000 7f2a5d156640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f2a5810d410 con 0x7f2a581021d0 2024-03-31T23:10:56.605 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.601+0000 7f2a5659c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2a58104b00 0x7f2a58197f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:10:56.605 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.601+0000 7f2a5659c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2a58104b00 0x7f2a58197f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:56368/0 (socket says 172.21.15.46:56368) 2024-03-31T23:10:56.605 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.601+0000 7f2a5659c640 1 -- 172.21.15.46:0/1526549287 learned_addr learned my addr 172.21.15.46:0/1526549287 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:10:56.605 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.601+0000 7f2a56d9d640 1 --2- 172.21.15.46:0/1526549287 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2a581021d0 0x7f2a58197a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:10:56.605 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.602+0000 7f2a5659c640 1 -- 172.21.15.46:0/1526549287 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2a581021d0 msgr2=0x7f2a58197a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:10:56.605 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.602+0000 7f2a5659c640 1 --2- 172.21.15.46:0/1526549287 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2a581021d0 0x7f2a58197a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:56.605 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.602+0000 7f2a5659c640 1 -- 172.21.15.46:0/1526549287 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2a58107430 msgr2=0x7f2a5819ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:10:56.605 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.602+0000 7f2a5659c640 1 --2- 172.21.15.46:0/1526549287 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2a58107430 0x7f2a5819ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:56.605 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.602+0000 7f2a5659c640 1 -- 172.21.15.46:0/1526549287 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a5819f600 con 0x7f2a58104b00 2024-03-31T23:10:56.605 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.602+0000 7f2a5659c640 1 --2- 172.21.15.46:0/1526549287 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2a58104b00 0x7f2a58197f70 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f2a4c000c00 tx=0x7f2a4c004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:10:56.605 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.602+0000 7f2a3ffff640 1 -- 172.21.15.46:0/1526549287 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a4c043020 con 0x7f2a58104b00 2024-03-31T23:10:56.605 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.602+0000 7f2a5d156640 1 -- 172.21.15.46:0/1526549287 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2a5819f890 con 0x7f2a58104b00 2024-03-31T23:10:56.606 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.602+0000 7f2a3ffff640 1 -- 172.21.15.46:0/1526549287 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f2a4c004440 con 0x7f2a58104b00 2024-03-31T23:10:56.606 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.602+0000 7f2a5d156640 1 -- 172.21.15.46:0/1526549287 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2a5819fe30 con 0x7f2a58104b00 2024-03-31T23:10:56.612 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.602+0000 7f2a3ffff640 1 -- 172.21.15.46:0/1526549287 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a4c003c30 con 0x7f2a58104b00 2024-03-31T23:10:56.612 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.604+0000 7f2a3ffff640 1 -- 172.21.15.46:0/1526549287 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 18) ==== 137320+0+0 (secure 0 0 0) 0x7f2a4c046020 con 0x7f2a58104b00 2024-03-31T23:10:56.612 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.605+0000 7f2a5fe19740 1 Processor -- start 2024-03-31T23:10:56.612 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.605+0000 7f2a3ffff640 1 --2- 172.21.15.46:0/1526549287 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f2a400a24a0 0x7f2a400a4960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:10:56.612 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.605+0000 7f2a5fe19740 1 -- start start 2024-03-31T23:10:56.612 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.605+0000 7f2a5759e640 1 --2- 172.21.15.46:0/1526549287 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f2a400a24a0 0x7f2a400a4960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:10:56.612 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.605+0000 7f2a5fe19740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fbd443a610 0x55fbd443aa30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:10:56.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.605+0000 7f2a5fe19740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fbd443b7e0 0x55fbd443bc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:10:56.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.605+0000 7f2a3ffff640 1 -- 172.21.15.46:0/1526549287 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7f2a4c0f0d40 con 0x7f2a58104b00 2024-03-31T23:10:56.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.605+0000 7f2a5fe19740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55fbd443c1a0 0x55fbd4442a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:10:56.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.605+0000 7f2a5fe19740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55fbd439eaf0 con 0x55fbd443b7e0 2024-03-31T23:10:56.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.605+0000 7f2a5fe19740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55fbd43479d0 con 0x55fbd443c1a0 2024-03-31T23:10:56.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.605+0000 7f2a5fe19740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55fbd439ab90 con 0x55fbd443a610 2024-03-31T23:10:56.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.605+0000 7f2a5659c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fbd443b7e0 0x55fbd443bc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:10:56.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.605+0000 7f2a5659c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fbd443b7e0 0x55fbd443bc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:56376/0 (socket says 172.21.15.46:56376) 2024-03-31T23:10:56.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.605+0000 7f2a5659c640 1 -- 172.21.15.46:0/928276059 learned_addr learned my addr 172.21.15.46:0/928276059 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:10:56.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.605+0000 7f2a56d9d640 1 --2- 172.21.15.46:0/928276059 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fbd443a610 0x55fbd443aa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:10:56.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.605+0000 7f2a5759e640 1 --2- 172.21.15.46:0/928276059 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55fbd443c1a0 0x55fbd4442a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:10:56.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.606+0000 7f2a5659c640 1 -- 172.21.15.46:0/928276059 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fbd443a610 msgr2=0x55fbd443aa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:10:56.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.606+0000 7f2a5659c640 1 --2- 172.21.15.46:0/928276059 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fbd443a610 0x55fbd443aa30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:56.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.606+0000 7f2a5659c640 1 -- 172.21.15.46:0/928276059 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55fbd443c1a0 msgr2=0x55fbd4442a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:10:56.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.606+0000 7f2a5659c640 1 --2- 172.21.15.46:0/928276059 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55fbd443c1a0 0x55fbd4442a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:56.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.606+0000 7f2a5659c640 1 -- 172.21.15.46:0/928276059 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fbd40b32d0 con 0x55fbd443b7e0 2024-03-31T23:10:56.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.606+0000 7f2a5759e640 1 --2- 172.21.15.46:0/1526549287 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f2a400a24a0 0x7f2a400a4960 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f2a48005fd0 tx=0x7f2a4800d6b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:10:56.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.606+0000 7f2a5659c640 1 --2- 172.21.15.46:0/928276059 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fbd443b7e0 0x55fbd443bc60 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f2a4c0bd3d0 tx=0x7f2a4c02bc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b7010c906fd7bf6 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:10:56.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.606+0000 7f2a3d7fa640 1 -- 172.21.15.46:0/928276059 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a4c0bd440 con 0x55fbd443b7e0 2024-03-31T23:10:56.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.606+0000 7f2a3d7fa640 1 -- 172.21.15.46:0/928276059 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f2a4c0c1590 con 0x55fbd443b7e0 2024-03-31T23:10:56.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.606+0000 7f2a3d7fa640 1 -- 172.21.15.46:0/928276059 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a4c0b9ab0 con 0x55fbd443b7e0 2024-03-31T23:10:56.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.607+0000 7f2a5fe19740 1 -- 172.21.15.46:0/928276059 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fbd443b7e0 msgr2=0x55fbd443bc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:10:56.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.607+0000 7f2a5fe19740 1 --2- 172.21.15.46:0/928276059 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fbd443b7e0 0x55fbd443bc60 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f2a4c0bd3d0 tx=0x7f2a4c02bc30 comp rx=0 tx=0).stop 2024-03-31T23:10:56.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.607+0000 7f2a5fe19740 1 -- 172.21.15.46:0/928276059 shutdown_connections 2024-03-31T23:10:56.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.607+0000 7f2a5fe19740 1 --2- 172.21.15.46:0/928276059 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55fbd443c1a0 0x55fbd4442a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:56.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.607+0000 7f2a5fe19740 1 --2- 172.21.15.46:0/928276059 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fbd443b7e0 0x55fbd443bc60 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:56.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.607+0000 7f2a5fe19740 1 --2- 172.21.15.46:0/928276059 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fbd443a610 0x55fbd443aa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:56.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.607+0000 7f2a5fe19740 1 -- 172.21.15.46:0/928276059 >> 172.21.15.46:0/928276059 conn(0x55fbd4435a00 msgr2=0x55fbd4435e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:10:56.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.607+0000 7f2a5fe19740 1 -- 172.21.15.46:0/928276059 shutdown_connections 2024-03-31T23:10:56.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.607+0000 7f2a5fe19740 1 -- 172.21.15.46:0/928276059 wait complete. 2024-03-31T23:10:56.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.607+0000 7f2a5fe19740 1 -- start start 2024-03-31T23:10:56.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.607+0000 7f2a5fe19740 1 Processor -- start 2024-03-31T23:10:56.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.607+0000 7f2a5fe19740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fbd443a610 0x55fbd44445e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:10:56.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.607+0000 7f2a5fe19740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55fbd443b7e0 0x55fbd4444b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:10:56.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.607+0000 7f2a5fe19740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fbd443c1a0 0x55fbd444bb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:10:56.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.607+0000 7f2a5fe19740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55fbd4442f70 con 0x55fbd443a610 2024-03-31T23:10:56.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.607+0000 7f2a5fe19740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55fbd435af70 con 0x55fbd443b7e0 2024-03-31T23:10:56.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.607+0000 7f2a5fe19740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55fbd44430f0 con 0x55fbd443c1a0 2024-03-31T23:10:56.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.607+0000 7f2a5fe19740 10 client.0 _ll_register_callbacks cb 0x55fbd40b2ac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T23:10:56.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.607+0000 7f2a56d9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fbd443a610 0x55fbd44445e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:10:56.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.607+0000 7f2a56d9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fbd443a610 0x55fbd44445e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:56378/0 (socket says 172.21.15.46:56378) 2024-03-31T23:10:56.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.607+0000 7f2a56d9d640 1 -- 172.21.15.46:0/3681604452 learned_addr learned my addr 172.21.15.46:0/3681604452 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:10:56.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.607+0000 7f2a5659c640 1 --2- 172.21.15.46:0/3681604452 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55fbd443b7e0 0x55fbd4444b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:10:56.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.607+0000 7f2a5759e640 1 --2- 172.21.15.46:0/3681604452 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fbd443c1a0 0x55fbd444bb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:10:56.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.608+0000 7f2a56d9d640 1 -- 172.21.15.46:0/3681604452 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fbd443c1a0 msgr2=0x55fbd444bb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:10:56.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.608+0000 7f2a56d9d640 1 --2- 172.21.15.46:0/3681604452 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fbd443c1a0 0x55fbd444bb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:56.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.608+0000 7f2a56d9d640 1 -- 172.21.15.46:0/3681604452 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55fbd443b7e0 msgr2=0x55fbd4444b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:10:56.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.608+0000 7f2a56d9d640 1 --2- 172.21.15.46:0/3681604452 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55fbd443b7e0 0x55fbd4444b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:56.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.608+0000 7f2a56d9d640 1 -- 172.21.15.46:0/3681604452 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fbd417d6b0 con 0x55fbd443a610 2024-03-31T23:10:56.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.608+0000 7f2a56d9d640 1 --2- 172.21.15.46:0/3681604452 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fbd443a610 0x55fbd44445e0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f2a50012700 tx=0x7f2a50012bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b71339fdc26373d server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:10:56.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.608+0000 7f2a3aff5640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-31T23:10:56.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.608+0000 7f2a3aff5640 1 -- 172.21.15.46:0/3681604452 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a50004020 con 0x55fbd443a610 2024-03-31T23:10:56.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.608+0000 7f2a3aff5640 1 -- 172.21.15.46:0/3681604452 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7f2a500041c0 con 0x55fbd443a610 2024-03-31T23:10:56.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.608+0000 7f2a5fe19740 1 -- 172.21.15.46:0/3681604452 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2a4c0c1590 con 0x55fbd443a610 2024-03-31T23:10:56.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.608+0000 7f2a3aff5640 1 -- 172.21.15.46:0/3681604452 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f2a50025050 con 0x55fbd443a610 2024-03-31T23:10:56.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.608+0000 7f2a5fe19740 10 client.15252 fetch_fsmap learned FSMap version 24 2024-03-31T23:10:56.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.608+0000 7f2a5fe19740 1 -- 172.21.15.46:0/3681604452 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55fbd444ec10 con 0x55fbd443a610 2024-03-31T23:10:56.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.608+0000 7f2a3aff5640 1 -- 172.21.15.46:0/3681604452 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f2a4c0c1590 con 0x55fbd443a610 2024-03-31T23:10:56.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.608+0000 7f2a3aff5640 1 -- 172.21.15.46:0/3681604452 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f2a500038b0 con 0x55fbd443a610 2024-03-31T23:10:56.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.608+0000 7f2a5fe19740 10 client.15252 fetch_fsmap finished waiting for FSMap version 24 2024-03-31T23:10:56.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.608+0000 7f2a5fe19740 10 client.15252 resolve_mds: resolved name 'h' to daemon mds.h 2024-03-31T23:10:56.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.608+0000 7f2a5fe19740 20 client.15252 populate_metadata read hostname 'smithi046' 2024-03-31T23:10:56.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.609+0000 7f2a5fe19740 1 --2- 172.21.15.46:0/3681604452 >> [v2:172.21.15.142:6836/2559258399,v1:172.21.15.142:6837/2559258399] conn(0x55fbd4451290 0x55fbd4452700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:10:56.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.609+0000 7f2a5fe19740 4 client.15252 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-31T23:10:56.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.609+0000 7f2a5fe19740 1 -- 172.21.15.46:0/3681604452 --> [v2:172.21.15.142:6836/2559258399,v1:172.21.15.142:6837/2559258399] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55fbd43ac030 con 0x55fbd4451290 2024-03-31T23:10:56.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.609+0000 7f2a5659c640 1 --2- 172.21.15.46:0/3681604452 >> [v2:172.21.15.142:6836/2559258399,v1:172.21.15.142:6837/2559258399] conn(0x55fbd4451290 0x55fbd4452700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:10:56.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.609+0000 7f2a5659c640 1 --2- 172.21.15.46:0/3681604452 >> [v2:172.21.15.142:6836/2559258399,v1:172.21.15.142:6837/2559258399] conn(0x55fbd4451290 0x55fbd4452700 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=5fa689e25eb42baf server_cookie=64822c3caa456e54 in_seq=0 out_seq=0 2024-03-31T23:10:56.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.609+0000 7f2a3aff5640 10 client.15252 ms_handle_connect on v2:172.21.15.142:6836/2559258399 2024-03-31T23:10:56.619 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.610+0000 7f2a3aff5640 1 -- 172.21.15.46:0/3681604452 <== mds.1 v2:172.21.15.142:6836/2559258399 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55fbd43ac030 con 0x55fbd4451290 2024-03-31T23:10:56.619 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.610+0000 7f2a3aff5640 10 client.15252 handle_command_reply: tid=0 2024-03-31T23:10:56.619 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.610+0000 7f2a5fe19740 1 client.15252 shutdown 2024-03-31T23:10:56.619 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.610+0000 7f2a5fe19740 1 -- 172.21.15.46:0/3681604452 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fbd443a610 msgr2=0x55fbd44445e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:10:56.619 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.610+0000 7f2a5fe19740 1 --2- 172.21.15.46:0/3681604452 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fbd443a610 0x55fbd44445e0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f2a50012700 tx=0x7f2a50012bc0 comp rx=0 tx=0).stop 2024-03-31T23:10:56.619 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.610+0000 7f2a5fe19740 1 -- 172.21.15.46:0/3681604452 shutdown_connections 2024-03-31T23:10:56.619 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.610+0000 7f2a5fe19740 1 --2- 172.21.15.46:0/3681604452 >> [v2:172.21.15.142:6836/2559258399,v1:172.21.15.142:6837/2559258399] conn(0x55fbd4451290 0x55fbd4452700 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:56.619 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.610+0000 7f2a5fe19740 1 --2- 172.21.15.46:0/3681604452 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fbd443c1a0 0x55fbd444bb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:56.619 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.610+0000 7f2a5fe19740 1 --2- 172.21.15.46:0/3681604452 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55fbd443b7e0 0x55fbd4444b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:56.619 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.610+0000 7f2a3aff5640 0 client.15252 ms_handle_reset on v2:172.21.15.142:6836/2559258399 2024-03-31T23:10:56.619 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.610+0000 7f2a5fe19740 1 --2- 172.21.15.46:0/3681604452 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fbd443a610 0x55fbd44445e0 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:56.619 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.610+0000 7f2a5fe19740 1 -- 172.21.15.46:0/3681604452 >> 172.21.15.46:0/3681604452 conn(0x55fbd4435a00 msgr2=0x55fbd4434b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:10:56.619 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.610+0000 7f2a5fe19740 1 -- 172.21.15.46:0/3681604452 shutdown_connections 2024-03-31T23:10:56.620 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.610+0000 7f2a5fe19740 1 -- 172.21.15.46:0/3681604452 wait complete. 2024-03-31T23:10:56.620 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.610+0000 7f2a5fe19740 20 client.15252 trim_cache size 0 max 16384 2024-03-31T23:10:56.620 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:10:56 smithi046 ceph-mon[32498]: pgmap v346: 129 pgs: 129 active+clean; 642 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2024-03-31T23:10:56.620 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:10:56 smithi046 ceph-mon[32498]: from='client.? 172.21.15.46:0/3346041175' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-31T23:10:56.620 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:10:56 smithi046 ceph-mon[32498]: from='client.? 172.21.15.46:0/1877529546' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-31T23:10:56.620 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:10:56 smithi046 ceph-mon[32498]: from='client.? 172.21.15.46:0/4098050552' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-31T23:10:56.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.628+0000 7f2a5fe19740 1 Processor -- start 2024-03-31T23:10:56.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.628+0000 7f2a5fe19740 1 -- start start 2024-03-31T23:10:56.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.628+0000 7f2a5fe19740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fbd443a610 0x55fbd4435de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:10:56.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.628+0000 7f2a5fe19740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55fbd443b7e0 0x55fbd444ba80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:10:56.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.628+0000 7f2a5fe19740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fbd443c1a0 0x55fbd44342b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:10:56.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.628+0000 7f2a5fe19740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55fbd4450a40 con 0x55fbd443a610 2024-03-31T23:10:56.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.628+0000 7f2a5fe19740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55fbd4451110 con 0x55fbd443b7e0 2024-03-31T23:10:56.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.628+0000 7f2a5fe19740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55fbd44517e0 con 0x55fbd443c1a0 2024-03-31T23:10:56.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.628+0000 7f2a56d9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fbd443a610 0x55fbd4435de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:10:56.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.628+0000 7f2a56d9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fbd443a610 0x55fbd4435de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:56388/0 (socket says 172.21.15.46:56388) 2024-03-31T23:10:56.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.628+0000 7f2a56d9d640 1 -- 172.21.15.46:0/3991939072 learned_addr learned my addr 172.21.15.46:0/3991939072 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:10:56.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.629+0000 7f2a5659c640 1 --2- 172.21.15.46:0/3991939072 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55fbd443b7e0 0x55fbd444ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:10:56.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.629+0000 7f2a5759e640 1 --2- 172.21.15.46:0/3991939072 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fbd443c1a0 0x55fbd44342b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:10:56.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.629+0000 7f2a56d9d640 1 -- 172.21.15.46:0/3991939072 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fbd443c1a0 msgr2=0x55fbd44342b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:10:56.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.629+0000 7f2a56d9d640 1 --2- 172.21.15.46:0/3991939072 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fbd443c1a0 0x55fbd44342b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:56.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.629+0000 7f2a56d9d640 1 -- 172.21.15.46:0/3991939072 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55fbd443b7e0 msgr2=0x55fbd444ba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:10:56.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.629+0000 7f2a56d9d640 1 --2- 172.21.15.46:0/3991939072 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55fbd443b7e0 0x55fbd444ba80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:56.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.629+0000 7f2a56d9d640 1 -- 172.21.15.46:0/3991939072 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fbd443b260 con 0x55fbd443a610 2024-03-31T23:10:56.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.629+0000 7f2a56d9d640 1 --2- 172.21.15.46:0/3991939072 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fbd443a610 0x55fbd4435de0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f2a50003ec0 tx=0x7f2a5001c680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ce9af10b024164a server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:10:56.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.629+0000 7f2a3cff9640 1 -- 172.21.15.46:0/3991939072 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a50022040 con 0x55fbd443a610 2024-03-31T23:10:56.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.629+0000 7f2a3cff9640 1 -- 172.21.15.46:0/3991939072 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f2a50003aa0 con 0x55fbd443a610 2024-03-31T23:10:56.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.629+0000 7f2a3cff9640 1 -- 172.21.15.46:0/3991939072 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a5000fac0 con 0x55fbd443a610 2024-03-31T23:10:56.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.630+0000 7f2a5fe19740 1 -- 172.21.15.46:0/3991939072 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fbd443a610 msgr2=0x55fbd4435de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:10:56.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.630+0000 7f2a5fe19740 1 --2- 172.21.15.46:0/3991939072 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fbd443a610 0x55fbd4435de0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f2a50003ec0 tx=0x7f2a5001c680 comp rx=0 tx=0).stop 2024-03-31T23:10:56.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.630+0000 7f2a5fe19740 1 -- 172.21.15.46:0/3991939072 shutdown_connections 2024-03-31T23:10:56.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.630+0000 7f2a5fe19740 1 --2- 172.21.15.46:0/3991939072 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fbd443c1a0 0x55fbd44342b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:56.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.630+0000 7f2a5fe19740 1 --2- 172.21.15.46:0/3991939072 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55fbd443b7e0 0x55fbd444ba80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:56.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.630+0000 7f2a5fe19740 1 --2- 172.21.15.46:0/3991939072 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fbd443a610 0x55fbd4435de0 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:56.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.630+0000 7f2a5fe19740 1 -- 172.21.15.46:0/3991939072 >> 172.21.15.46:0/3991939072 conn(0x55fbd4435a00 msgr2=0x55fbd4442780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:10:56.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.630+0000 7f2a5fe19740 1 -- 172.21.15.46:0/3991939072 shutdown_connections 2024-03-31T23:10:56.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.630+0000 7f2a5fe19740 1 -- 172.21.15.46:0/3991939072 wait complete. 2024-03-31T23:10:56.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.630+0000 7f2a5fe19740 1 -- start start 2024-03-31T23:10:56.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.630+0000 7f2a5fe19740 1 Processor -- start 2024-03-31T23:10:56.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.630+0000 7f2a5fe19740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fbd443a610 0x55fbd4447570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:10:56.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.630+0000 7f2a5fe19740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55fbd443b7e0 0x55fbd4447ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:10:56.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.630+0000 7f2a5fe19740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fbd443c1a0 0x55fbd444aad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:10:56.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.630+0000 7f2a5fe19740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55fbd4452580 con 0x55fbd443c1a0 2024-03-31T23:10:56.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.630+0000 7f2a5fe19740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55fbd4451eb0 con 0x55fbd443b7e0 2024-03-31T23:10:56.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.630+0000 7f2a5fe19740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55fbd4452c50 con 0x55fbd443a610 2024-03-31T23:10:56.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.630+0000 7f2a5fe19740 10 client.0 _ll_register_callbacks cb 0x7f2a50009830 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T23:10:56.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.630+0000 7f2a5759e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fbd443c1a0 0x55fbd444aad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:10:56.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.630+0000 7f2a5659c640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55fbd443b7e0 0x55fbd4447ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:10:56.638 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.630+0000 7f2a56d9d640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fbd443a610 0x55fbd4447570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:10:56.638 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.630+0000 7f2a5659c640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55fbd443b7e0 0x55fbd4447ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.46:36696/0 (socket says 172.21.15.46:36696) 2024-03-31T23:10:56.638 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.630+0000 7f2a5759e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fbd443c1a0 0x55fbd444aad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:56394/0 (socket says 172.21.15.46:56394) 2024-03-31T23:10:56.638 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.630+0000 7f2a5659c640 1 -- 172.21.15.46:0/1415364939 learned_addr learned my addr 172.21.15.46:0/1415364939 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:10:56.638 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.631+0000 7f2a56d9d640 1 -- 172.21.15.46:0/1415364939 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55fbd443b7e0 msgr2=0x55fbd4447ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:10:56.638 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.631+0000 7f2a56d9d640 1 --2- 172.21.15.46:0/1415364939 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55fbd443b7e0 0x55fbd4447ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:56.638 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.631+0000 7f2a56d9d640 1 -- 172.21.15.46:0/1415364939 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fbd443c1a0 msgr2=0x55fbd444aad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:10:56.638 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.631+0000 7f2a56d9d640 1 --2- 172.21.15.46:0/1415364939 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fbd443c1a0 0x55fbd444aad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:56.638 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.631+0000 7f2a56d9d640 1 -- 172.21.15.46:0/1415364939 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fbd444b0c0 con 0x55fbd443a610 2024-03-31T23:10:56.638 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.631+0000 7f2a5659c640 1 --2- 172.21.15.46:0/1415364939 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55fbd443b7e0 0x55fbd4447ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:10:56.638 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.631+0000 7f2a5759e640 1 --2- 172.21.15.46:0/1415364939 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fbd443c1a0 0x55fbd444aad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:10:56.638 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.631+0000 7f2a56d9d640 1 --2- 172.21.15.46:0/1415364939 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fbd443a610 0x55fbd4447570 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x55fbd444cc30 tx=0x7f2a50000bc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ed7630f69f3c1077 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:10:56.639 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.631+0000 7f2a3aff5640 10 client.0 ms_handle_connect on v2:172.21.15.162:3300/0 2024-03-31T23:10:56.639 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.631+0000 7f2a3aff5640 1 -- 172.21.15.46:0/1415364939 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a50005280 con 0x55fbd443a610 2024-03-31T23:10:56.639 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.631+0000 7f2a3aff5640 1 -- 172.21.15.46:0/1415364939 <== mon.1 v2:172.21.15.162:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7f2a50005420 con 0x55fbd443a610 2024-03-31T23:10:56.639 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.631+0000 7f2a5fe19740 1 -- 172.21.15.46:0/1415364939 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2a50003aa0 con 0x55fbd443a610 2024-03-31T23:10:56.639 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.631+0000 7f2a3aff5640 1 -- 172.21.15.46:0/1415364939 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f2a50038050 con 0x55fbd443a610 2024-03-31T23:10:56.639 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.631+0000 7f2a5fe19740 10 client.25003 fetch_fsmap learned FSMap version 24 2024-03-31T23:10:56.639 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.631+0000 7f2a5fe19740 1 -- 172.21.15.46:0/1415364939 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55fbd44b73e0 con 0x55fbd443a610 2024-03-31T23:10:56.639 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.631+0000 7f2a3aff5640 1 -- 172.21.15.46:0/1415364939 <== mon.1 v2:172.21.15.162:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f2a50003aa0 con 0x55fbd443a610 2024-03-31T23:10:56.639 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.632+0000 7f2a3aff5640 1 -- 172.21.15.46:0/1415364939 <== mon.1 v2:172.21.15.162:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f2a5001c920 con 0x55fbd443a610 2024-03-31T23:10:56.639 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.632+0000 7f2a5fe19740 10 client.25003 fetch_fsmap finished waiting for FSMap version 24 2024-03-31T23:10:56.639 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.632+0000 7f2a5fe19740 10 client.25003 resolve_mds: resolved name 'h' to daemon mds.h 2024-03-31T23:10:56.639 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.632+0000 7f2a5fe19740 20 client.25003 populate_metadata read hostname 'smithi046' 2024-03-31T23:10:56.639 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.632+0000 7f2a5fe19740 1 --2- 172.21.15.46:0/1415364939 >> [v2:172.21.15.142:6836/2559258399,v1:172.21.15.142:6837/2559258399] conn(0x55fbd44bbba0 0x55fbd44ba8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:10:56.639 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.632+0000 7f2a5fe19740 4 client.25003 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-31T23:10:56.639 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.632+0000 7f2a5fe19740 1 -- 172.21.15.46:0/1415364939 --> [v2:172.21.15.142:6836/2559258399,v1:172.21.15.142:6837/2559258399] -- command(tid 0: {"prefix": "status"}) -- 0x55fbd43a6380 con 0x55fbd44bbba0 2024-03-31T23:10:56.639 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.632+0000 7f2a5659c640 1 --2- 172.21.15.46:0/1415364939 >> [v2:172.21.15.142:6836/2559258399,v1:172.21.15.142:6837/2559258399] conn(0x55fbd44bbba0 0x55fbd44ba8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:10:56.640 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.632+0000 7f2a5659c640 1 --2- 172.21.15.46:0/1415364939 >> [v2:172.21.15.142:6836/2559258399,v1:172.21.15.142:6837/2559258399] conn(0x55fbd44bbba0 0x55fbd44ba8b0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=83c62869fdf83f5a server_cookie=750cb067d3f1f65f in_seq=0 out_seq=0 2024-03-31T23:10:56.640 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.632+0000 7f2a3aff5640 10 client.25003 ms_handle_connect on v2:172.21.15.142:6836/2559258399 2024-03-31T23:10:56.640 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.633+0000 7f2a3aff5640 1 -- 172.21.15.46:0/1415364939 <== mds.1 v2:172.21.15.142:6836/2559258399 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x55fbd43a6380 con 0x55fbd44bbba0 2024-03-31T23:10:56.640 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.633+0000 7f2a3aff5640 10 client.25003 handle_command_reply: tid=0 2024-03-31T23:10:56.640 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.633+0000 7f2a5fe19740 1 client.25003 shutdown 2024-03-31T23:10:56.640 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.633+0000 7f2a5fe19740 1 -- 172.21.15.46:0/1415364939 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fbd443a610 msgr2=0x55fbd4447570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:10:56.640 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.633+0000 7f2a5fe19740 1 --2- 172.21.15.46:0/1415364939 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fbd443a610 0x55fbd4447570 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x55fbd444cc30 tx=0x7f2a50000bc0 comp rx=0 tx=0).stop 2024-03-31T23:10:56.640 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.633+0000 7f2a5fe19740 1 -- 172.21.15.46:0/1415364939 shutdown_connections 2024-03-31T23:10:56.640 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.633+0000 7f2a5fe19740 1 --2- 172.21.15.46:0/1415364939 >> [v2:172.21.15.142:6836/2559258399,v1:172.21.15.142:6837/2559258399] conn(0x55fbd44bbba0 0x55fbd44ba8b0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:56.640 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.633+0000 7f2a5fe19740 1 --2- 172.21.15.46:0/1415364939 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fbd443c1a0 0x55fbd444aad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:56.640 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.633+0000 7f2a3aff5640 0 client.25003 ms_handle_reset on v2:172.21.15.142:6836/2559258399 2024-03-31T23:10:56.640 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.633+0000 7f2a5fe19740 1 --2- 172.21.15.46:0/1415364939 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55fbd443b7e0 0x55fbd4447ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:56.640 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.633+0000 7f2a5fe19740 1 --2- 172.21.15.46:0/1415364939 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fbd443a610 0x55fbd4447570 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:56.640 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.633+0000 7f2a5fe19740 1 -- 172.21.15.46:0/1415364939 >> 172.21.15.46:0/1415364939 conn(0x55fbd4435a00 msgr2=0x55fbd443f110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:10:56.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.633+0000 7f2a5fe19740 1 -- 172.21.15.46:0/1415364939 shutdown_connections 2024-03-31T23:10:56.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.633+0000 7f2a5fe19740 1 -- 172.21.15.46:0/1415364939 wait complete. 2024-03-31T23:10:56.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.633+0000 7f2a5fe19740 20 client.25003 trim_cache size 0 max 16384 2024-03-31T23:10:56.641 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-03-31T23:10:56.641 INFO:teuthology.orchestra.run.smithi046.stdout: "cluster_fsid": "6a9d8f08-efb2-11ee-b647-cb9ed24678a4", 2024-03-31T23:10:56.641 INFO:teuthology.orchestra.run.smithi046.stdout: "whoami": 1, 2024-03-31T23:10:56.641 INFO:teuthology.orchestra.run.smithi046.stdout: "id": 24482, 2024-03-31T23:10:56.641 INFO:teuthology.orchestra.run.smithi046.stdout: "want_state": "up:active", 2024-03-31T23:10:56.641 INFO:teuthology.orchestra.run.smithi046.stdout: "state": "up:active", 2024-03-31T23:10:56.641 INFO:teuthology.orchestra.run.smithi046.stdout: "fs_name": "cephfs", 2024-03-31T23:10:56.641 INFO:teuthology.orchestra.run.smithi046.stdout: "rank_uptime": 87.219465038999999, 2024-03-31T23:10:56.642 INFO:teuthology.orchestra.run.smithi046.stdout: "mdsmap_epoch": 24, 2024-03-31T23:10:56.642 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap_epoch": 93, 2024-03-31T23:10:56.642 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap_epoch_barrier": 91, 2024-03-31T23:10:56.642 INFO:teuthology.orchestra.run.smithi046.stdout: "uptime": 121.025059184 2024-03-31T23:10:56.642 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-03-31T23:10:56.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.633+0000 7f2a5d156640 1 -- 172.21.15.46:0/1526549287 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f2a400a24a0 msgr2=0x7f2a400a4960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:10:56.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.633+0000 7f2a5d156640 1 --2- 172.21.15.46:0/1526549287 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f2a400a24a0 0x7f2a400a4960 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f2a48005fd0 tx=0x7f2a4800d6b0 comp rx=0 tx=0).stop 2024-03-31T23:10:56.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.633+0000 7f2a5d156640 1 -- 172.21.15.46:0/1526549287 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2a58104b00 msgr2=0x7f2a58197f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:10:56.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.633+0000 7f2a5d156640 1 --2- 172.21.15.46:0/1526549287 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2a58104b00 0x7f2a58197f70 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f2a4c000c00 tx=0x7f2a4c004040 comp rx=0 tx=0).stop 2024-03-31T23:10:56.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.633+0000 7f2a5d156640 1 -- 172.21.15.46:0/1526549287 shutdown_connections 2024-03-31T23:10:56.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.633+0000 7f2a5d156640 1 --2- 172.21.15.46:0/1526549287 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f2a400a24a0 0x7f2a400a4960 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:56.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.633+0000 7f2a5d156640 1 --2- 172.21.15.46:0/1526549287 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2a58107430 0x7f2a5819ef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:56.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.633+0000 7f2a5d156640 1 --2- 172.21.15.46:0/1526549287 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2a58104b00 0x7f2a58197f70 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:56.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.633+0000 7f2a5d156640 1 --2- 172.21.15.46:0/1526549287 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2a581021d0 0x7f2a58197a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:56.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.633+0000 7f2a5d156640 1 -- 172.21.15.46:0/1526549287 >> 172.21.15.46:0/1526549287 conn(0x7f2a580fbbe0 msgr2=0x7f2a58102bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:10:56.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.633+0000 7f2a5d156640 1 -- 172.21.15.46:0/1526549287 shutdown_connections 2024-03-31T23:10:56.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.633+0000 7f2a5d156640 1 -- 172.21.15.46:0/1526549287 wait complete. 2024-03-31T23:10:56.645 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-03-31T23:10:56.751 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.748+0000 7fdadff56640 1 -- 172.21.15.46:0/975596757 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdad81074f0 msgr2=0x7fdad81098e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:10:56.751 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.748+0000 7fdadff56640 1 --2- 172.21.15.46:0/975596757 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdad81074f0 0x7fdad81098e0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7fdad0009900 tx=0x7fdad002bc30 comp rx=0 tx=0).stop 2024-03-31T23:10:56.751 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.748+0000 7fdadff56640 1 -- 172.21.15.46:0/975596757 shutdown_connections 2024-03-31T23:10:56.751 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.748+0000 7fdadff56640 1 --2- 172.21.15.46:0/975596757 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdad81074f0 0x7fdad81098e0 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:56.751 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.748+0000 7fdadff56640 1 --2- 172.21.15.46:0/975596757 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdad8104bc0 0x7fdad8106fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:56.751 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.748+0000 7fdadff56640 1 --2- 172.21.15.46:0/975596757 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdad8102290 0x7fdad8104680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:56.752 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.748+0000 7fdadff56640 1 -- 172.21.15.46:0/975596757 >> 172.21.15.46:0/975596757 conn(0x7fdad80fbc00 msgr2=0x7fdad80fe020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:10:56.752 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.748+0000 7fdadff56640 1 -- 172.21.15.46:0/975596757 shutdown_connections 2024-03-31T23:10:56.752 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.748+0000 7fdadff56640 1 -- 172.21.15.46:0/975596757 wait complete. 2024-03-31T23:10:56.752 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.749+0000 7fdadff56640 1 Processor -- start 2024-03-31T23:10:56.752 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.749+0000 7fdadff56640 1 -- start start 2024-03-31T23:10:56.752 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.749+0000 7fdadff56640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdad8102290 0x7fdad8197950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:10:56.752 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.749+0000 7fdadff56640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdad8104bc0 0x7fdad8197e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:10:56.752 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.749+0000 7fdadff56640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdad81074f0 0x7fdad819ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:10:56.752 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.749+0000 7fdadff56640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fdad810d2a0 con 0x7fdad81074f0 2024-03-31T23:10:56.752 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.749+0000 7fdadff56640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fdad810d120 con 0x7fdad8104bc0 2024-03-31T23:10:56.752 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.749+0000 7fdadff56640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fdad810d420 con 0x7fdad8102290 2024-03-31T23:10:56.752 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.749+0000 7fdade4f3640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdad81074f0 0x7fdad819ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:10:56.752 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.749+0000 7fdade4f3640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdad81074f0 0x7fdad819ef00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:56406/0 (socket says 172.21.15.46:56406) 2024-03-31T23:10:56.752 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.749+0000 7fdade4f3640 1 -- 172.21.15.46:0/273563495 learned_addr learned my addr 172.21.15.46:0/273563495 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:10:56.753 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.749+0000 7fdaddcf2640 1 --2- 172.21.15.46:0/273563495 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdad8102290 0x7fdad8197950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:10:56.753 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.749+0000 7fdadd4f1640 1 --2- 172.21.15.46:0/273563495 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdad8104bc0 0x7fdad8197e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:10:56.753 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.749+0000 7fdade4f3640 1 -- 172.21.15.46:0/273563495 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdad8102290 msgr2=0x7fdad8197950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:10:56.753 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.749+0000 7fdade4f3640 1 --2- 172.21.15.46:0/273563495 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdad8102290 0x7fdad8197950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:56.753 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.749+0000 7fdade4f3640 1 -- 172.21.15.46:0/273563495 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdad8104bc0 msgr2=0x7fdad8197e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:10:56.753 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.749+0000 7fdade4f3640 1 --2- 172.21.15.46:0/273563495 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdad8104bc0 0x7fdad8197e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:56.753 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.749+0000 7fdade4f3640 1 -- 172.21.15.46:0/273563495 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdad819f570 con 0x7fdad81074f0 2024-03-31T23:10:56.753 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.749+0000 7fdade4f3640 1 --2- 172.21.15.46:0/273563495 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdad81074f0 0x7fdad819ef00 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7fdad0009230 tx=0x7fdad0005e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:10:56.753 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.749+0000 7fdad6ffd640 1 -- 172.21.15.46:0/273563495 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdad0043020 con 0x7fdad81074f0 2024-03-31T23:10:56.753 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.750+0000 7fdad6ffd640 1 -- 172.21.15.46:0/273563495 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fdad00043d0 con 0x7fdad81074f0 2024-03-31T23:10:56.753 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.750+0000 7fdad6ffd640 1 -- 172.21.15.46:0/273563495 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdad0003b40 con 0x7fdad81074f0 2024-03-31T23:10:56.753 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.750+0000 7fdadff56640 1 -- 172.21.15.46:0/273563495 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdad819f800 con 0x7fdad81074f0 2024-03-31T23:10:56.753 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.750+0000 7fdadff56640 1 -- 172.21.15.46:0/273563495 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdad819fd50 con 0x7fdad81074f0 2024-03-31T23:10:56.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.752+0000 7fdad6ffd640 1 -- 172.21.15.46:0/273563495 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 18) ==== 137320+0+0 (secure 0 0 0) 0x7fdad0046020 con 0x7fdad81074f0 2024-03-31T23:10:56.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.752+0000 7fdae2c12740 1 Processor -- start 2024-03-31T23:10:56.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.752+0000 7fdae2c12740 1 -- start start 2024-03-31T23:10:56.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.752+0000 7fdae2c12740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563011a63610 0x563011a63a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:10:56.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.752+0000 7fdae2c12740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x563011a647e0 0x563011a64c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:10:56.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.752+0000 7fdad6ffd640 1 --2- 172.21.15.46:0/273563495 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fdac00a24a0 0x7fdac00a4960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:10:56.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.752+0000 7fdae2c12740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563011a651a0 0x563011a6ba30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:10:56.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.753+0000 7fdae2c12740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x5630119c7af0 con 0x563011a651a0 2024-03-31T23:10:56.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.753+0000 7fdae2c12740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x5630119709d0 con 0x563011a647e0 2024-03-31T23:10:56.761 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.753+0000 7fdae2c12740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5630119c3b90 con 0x563011a63610 2024-03-31T23:10:56.761 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.753+0000 7fdade4f3640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563011a651a0 0x563011a6ba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:10:56.761 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.753+0000 7fdadd4f1640 1 --2- 172.21.15.46:0/273563495 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fdac00a24a0 0x7fdac00a4960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:10:56.761 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.753+0000 7fdaddcf2640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x563011a647e0 0x563011a64c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:10:56.761 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.753+0000 7fdadd4f1640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563011a63610 0x563011a63a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:10:56.761 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.753+0000 7fdade4f3640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563011a651a0 0x563011a6ba30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:56422/0 (socket says 172.21.15.46:56422) 2024-03-31T23:10:56.761 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.753+0000 7fdaddcf2640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x563011a647e0 0x563011a64c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.46:36704/0 (socket says 172.21.15.46:36704) 2024-03-31T23:10:56.761 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.753+0000 7fdade4f3640 1 -- 172.21.15.46:0/383868100 learned_addr learned my addr 172.21.15.46:0/383868100 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:10:56.761 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.753+0000 7fdad6ffd640 1 -- 172.21.15.46:0/273563495 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7fdad00f0cc0 con 0x7fdad81074f0 2024-03-31T23:10:56.761 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.753+0000 7fdadd4f1640 1 -- 172.21.15.46:0/383868100 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x563011a647e0 msgr2=0x563011a64c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:10:56.761 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.753+0000 7fdadd4f1640 1 --2- 172.21.15.46:0/383868100 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x563011a647e0 0x563011a64c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:56.761 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.753+0000 7fdadd4f1640 1 -- 172.21.15.46:0/383868100 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563011a651a0 msgr2=0x563011a6ba30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:10:56.761 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.753+0000 7fdadd4f1640 1 --2- 172.21.15.46:0/383868100 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563011a651a0 0x563011a6ba30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:56.761 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.753+0000 7fdadd4f1640 1 -- 172.21.15.46:0/383868100 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5630116dc2d0 con 0x563011a63610 2024-03-31T23:10:56.761 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.753+0000 7fdadd4f1640 1 --2- 172.21.15.46:0/383868100 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563011a63610 0x563011a63a30 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fdad806a7e0 tx=0x7fdacc010e20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a24697ab08070c13 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:10:56.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.753+0000 7fdadd4f1640 1 --2- 172.21.15.46:0/273563495 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fdac00a24a0 0x7fdac00a4960 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fdacc00f790 tx=0x7fdacc018000 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:10:56.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.753+0000 7fdabffff640 1 -- 172.21.15.46:0/383868100 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdacc022070 con 0x563011a63610 2024-03-31T23:10:56.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.753+0000 7fdabffff640 1 -- 172.21.15.46:0/383868100 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fdacc010430 con 0x563011a63610 2024-03-31T23:10:56.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.753+0000 7fdabffff640 1 -- 172.21.15.46:0/383868100 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdacc019040 con 0x563011a63610 2024-03-31T23:10:56.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.754+0000 7fdae2c12740 1 -- 172.21.15.46:0/383868100 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563011a63610 msgr2=0x563011a63a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:10:56.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.754+0000 7fdae2c12740 1 --2- 172.21.15.46:0/383868100 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563011a63610 0x563011a63a30 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fdad806a7e0 tx=0x7fdacc010e20 comp rx=0 tx=0).stop 2024-03-31T23:10:56.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.754+0000 7fdae2c12740 1 -- 172.21.15.46:0/383868100 shutdown_connections 2024-03-31T23:10:56.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.754+0000 7fdae2c12740 1 --2- 172.21.15.46:0/383868100 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563011a651a0 0x563011a6ba30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:56.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.754+0000 7fdae2c12740 1 --2- 172.21.15.46:0/383868100 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x563011a647e0 0x563011a64c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:56.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.754+0000 7fdae2c12740 1 --2- 172.21.15.46:0/383868100 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563011a63610 0x563011a63a30 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:56.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.754+0000 7fdae2c12740 1 -- 172.21.15.46:0/383868100 >> 172.21.15.46:0/383868100 conn(0x563011a5ea00 msgr2=0x563011a5ee00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:10:56.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.754+0000 7fdae2c12740 1 -- 172.21.15.46:0/383868100 shutdown_connections 2024-03-31T23:10:56.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.754+0000 7fdae2c12740 1 -- 172.21.15.46:0/383868100 wait complete. 2024-03-31T23:10:56.763 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.755+0000 7fdae2c12740 1 -- start start 2024-03-31T23:10:56.763 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.755+0000 7fdae2c12740 1 Processor -- start 2024-03-31T23:10:56.763 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.755+0000 7fdae2c12740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x563011a63610 0x563011a6d9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:10:56.763 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.755+0000 7fdae2c12740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563011a647e0 0x563011a6df10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:10:56.763 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.755+0000 7fdae2c12740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563011a651a0 0x563011a74ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:10:56.763 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.755+0000 7fdae2c12740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x563011a6c150 con 0x563011a651a0 2024-03-31T23:10:56.763 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.755+0000 7fdae2c12740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x563011983f70 con 0x563011a63610 2024-03-31T23:10:56.763 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.755+0000 7fdae2c12740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x563011a6c2d0 con 0x563011a647e0 2024-03-31T23:10:56.763 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.755+0000 7fdae2c12740 10 client.0 _ll_register_callbacks cb 0x5630116dbac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T23:10:56.763 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.755+0000 7fdade4f3640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563011a651a0 0x563011a74ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:10:56.763 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.755+0000 7fdaddcf2640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x563011a63610 0x563011a6d9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:10:56.763 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.755+0000 7fdade4f3640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563011a651a0 0x563011a74ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:56430/0 (socket says 172.21.15.46:56430) 2024-03-31T23:10:56.763 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.755+0000 7fdade4f3640 1 -- 172.21.15.46:0/1681177665 learned_addr learned my addr 172.21.15.46:0/1681177665 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:10:56.763 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.755+0000 7fdadd4f1640 1 --2- 172.21.15.46:0/1681177665 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563011a647e0 0x563011a6df10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:10:56.763 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.755+0000 7fdade4f3640 1 -- 172.21.15.46:0/1681177665 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563011a647e0 msgr2=0x563011a6df10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:10:56.764 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.755+0000 7fdade4f3640 1 --2- 172.21.15.46:0/1681177665 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563011a647e0 0x563011a6df10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:56.764 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.755+0000 7fdade4f3640 1 -- 172.21.15.46:0/1681177665 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x563011a63610 msgr2=0x563011a6d9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:10:56.764 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.755+0000 7fdade4f3640 1 --2- 172.21.15.46:0/1681177665 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x563011a63610 0x563011a6d9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:56.764 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.755+0000 7fdade4f3640 1 -- 172.21.15.46:0/1681177665 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5630117a66b0 con 0x563011a651a0 2024-03-31T23:10:56.764 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.755+0000 7fdaddcf2640 1 --2- 172.21.15.46:0/1681177665 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x563011a63610 0x563011a6d9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:10:56.764 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.756+0000 7fdade4f3640 1 --2- 172.21.15.46:0/1681177665 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563011a651a0 0x563011a74ee0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fdad0003630 tx=0x7fdad00bd190 comp rx=0 tx=0).ready entity=mon.0 client_cookie=83f838362a863485 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:10:56.764 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.756+0000 7fdabd7fa640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-31T23:10:56.764 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.756+0000 7fdabd7fa640 1 -- 172.21.15.46:0/1681177665 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdad00c2ca0 con 0x563011a651a0 2024-03-31T23:10:56.764 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.756+0000 7fdabd7fa640 1 -- 172.21.15.46:0/1681177665 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7fdad00dd0e0 con 0x563011a651a0 2024-03-31T23:10:56.764 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.756+0000 7fdae2c12740 1 -- 172.21.15.46:0/1681177665 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdacc010430 con 0x563011a651a0 2024-03-31T23:10:56.764 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.756+0000 7fdabd7fa640 1 -- 172.21.15.46:0/1681177665 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fdad00bdac0 con 0x563011a651a0 2024-03-31T23:10:56.764 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.756+0000 7fdae2c12740 10 client.15270 fetch_fsmap learned FSMap version 24 2024-03-31T23:10:56.764 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.756+0000 7fdae2c12740 1 -- 172.21.15.46:0/1681177665 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x563011a77ec0 con 0x563011a651a0 2024-03-31T23:10:56.764 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.756+0000 7fdabd7fa640 1 -- 172.21.15.46:0/1681177665 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fdacc010430 con 0x563011a651a0 2024-03-31T23:10:56.764 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.756+0000 7fdabd7fa640 1 -- 172.21.15.46:0/1681177665 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7fdad0005440 con 0x563011a651a0 2024-03-31T23:10:56.764 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.756+0000 7fdae2c12740 10 client.15270 fetch_fsmap finished waiting for FSMap version 24 2024-03-31T23:10:56.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.756+0000 7fdae2c12740 10 client.15270 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-31T23:10:56.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.756+0000 7fdae2c12740 20 client.15270 populate_metadata read hostname 'smithi046' 2024-03-31T23:10:56.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.756+0000 7fdae2c12740 1 --2- 172.21.15.46:0/1681177665 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x563011a7a600 0x563011a7ba70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:10:56.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.756+0000 7fdae2c12740 4 client.15270 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-31T23:10:56.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.756+0000 7fdae2c12740 1 -- 172.21.15.46:0/1681177665 --> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5630119d5030 con 0x563011a7a600 2024-03-31T23:10:56.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.757+0000 7fdaddcf2640 1 --2- 172.21.15.46:0/1681177665 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x563011a7a600 0x563011a7ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:10:56.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.757+0000 7fdaddcf2640 1 --2- 172.21.15.46:0/1681177665 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x563011a7a600 0x563011a7ba70 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=af72c48c71c37745 server_cookie=8c5aeec31a7928e1 in_seq=0 out_seq=0 2024-03-31T23:10:56.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.757+0000 7fdabd7fa640 10 client.15270 ms_handle_connect on v2:172.21.15.142:6838/2519143918 2024-03-31T23:10:56.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.758+0000 7fdabd7fa640 1 -- 172.21.15.46:0/1681177665 <== mds.0 v2:172.21.15.142:6838/2519143918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5630119d5030 con 0x563011a7a600 2024-03-31T23:10:56.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.758+0000 7fdabd7fa640 10 client.15270 handle_command_reply: tid=0 2024-03-31T23:10:56.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.758+0000 7fdae2c12740 1 client.15270 shutdown 2024-03-31T23:10:56.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.758+0000 7fdae2c12740 1 -- 172.21.15.46:0/1681177665 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563011a651a0 msgr2=0x563011a74ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:10:56.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.758+0000 7fdae2c12740 1 --2- 172.21.15.46:0/1681177665 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563011a651a0 0x563011a74ee0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fdad0003630 tx=0x7fdad00bd190 comp rx=0 tx=0).stop 2024-03-31T23:10:56.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.758+0000 7fdae2c12740 1 -- 172.21.15.46:0/1681177665 shutdown_connections 2024-03-31T23:10:56.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.758+0000 7fdae2c12740 1 --2- 172.21.15.46:0/1681177665 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x563011a7a600 0x563011a7ba70 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:56.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.758+0000 7fdae2c12740 1 --2- 172.21.15.46:0/1681177665 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563011a651a0 0x563011a74ee0 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:56.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.758+0000 7fdabd7fa640 0 client.15270 ms_handle_reset on v2:172.21.15.142:6838/2519143918 2024-03-31T23:10:56.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.758+0000 7fdae2c12740 1 --2- 172.21.15.46:0/1681177665 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563011a647e0 0x563011a6df10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:56.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.758+0000 7fdae2c12740 1 --2- 172.21.15.46:0/1681177665 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x563011a63610 0x563011a6d9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:56.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.758+0000 7fdae2c12740 1 -- 172.21.15.46:0/1681177665 >> 172.21.15.46:0/1681177665 conn(0x563011a5ea00 msgr2=0x563011a5db40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:10:56.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.758+0000 7fdae2c12740 1 -- 172.21.15.46:0/1681177665 shutdown_connections 2024-03-31T23:10:56.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.758+0000 7fdae2c12740 1 -- 172.21.15.46:0/1681177665 wait complete. 2024-03-31T23:10:56.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.759+0000 7fdae2c12740 20 client.15270 trim_cache size 0 max 16384 2024-03-31T23:10:56.783 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.777+0000 7fdae2c12740 1 Processor -- start 2024-03-31T23:10:56.783 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.777+0000 7fdae2c12740 1 -- start start 2024-03-31T23:10:56.783 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.777+0000 7fdae2c12740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563011a63610 0x563011a6d5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:10:56.783 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.777+0000 7fdae2c12740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563011a647e0 0x563011a74ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:10:56.783 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.777+0000 7fdae2c12740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x563011a651a0 0x563011a6db20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:10:56.783 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.777+0000 7fdae2c12740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x563011a79db0 con 0x563011a63610 2024-03-31T23:10:56.783 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.777+0000 7fdae2c12740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x563011a7a480 con 0x563011a651a0 2024-03-31T23:10:56.783 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.777+0000 7fdae2c12740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x563011a7ab50 con 0x563011a647e0 2024-03-31T23:10:56.783 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.777+0000 7fdaddcf2640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563011a63610 0x563011a6d5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:10:56.783 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.777+0000 7fdaddcf2640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563011a63610 0x563011a6d5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:56438/0 (socket says 172.21.15.46:56438) 2024-03-31T23:10:56.784 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.777+0000 7fdaddcf2640 1 -- 172.21.15.46:0/4214657140 learned_addr learned my addr 172.21.15.46:0/4214657140 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:10:56.784 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.777+0000 7fdade4f3640 1 --2- 172.21.15.46:0/4214657140 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x563011a651a0 0x563011a6db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:10:56.784 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.777+0000 7fdadd4f1640 1 --2- 172.21.15.46:0/4214657140 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563011a647e0 0x563011a74ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:10:56.784 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.777+0000 7fdaddcf2640 1 -- 172.21.15.46:0/4214657140 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563011a647e0 msgr2=0x563011a74ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:10:56.784 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.777+0000 7fdaddcf2640 1 --2- 172.21.15.46:0/4214657140 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563011a647e0 0x563011a74ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:56.784 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.777+0000 7fdaddcf2640 1 -- 172.21.15.46:0/4214657140 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x563011a651a0 msgr2=0x563011a6db20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:10:56.784 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.777+0000 7fdaddcf2640 1 --2- 172.21.15.46:0/4214657140 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x563011a651a0 0x563011a6db20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:56.784 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.777+0000 7fdaddcf2640 1 -- 172.21.15.46:0/4214657140 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563011a5d040 con 0x563011a63610 2024-03-31T23:10:56.784 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.777+0000 7fdaddcf2640 1 --2- 172.21.15.46:0/4214657140 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563011a63610 0x563011a6d5e0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7fdac80097f0 tx=0x7fdac8009500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1cda1e25f47a883b server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:10:56.784 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.777+0000 7fdabf7fe640 1 -- 172.21.15.46:0/4214657140 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdac8004d20 con 0x563011a63610 2024-03-31T23:10:56.784 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.777+0000 7fdabf7fe640 1 -- 172.21.15.46:0/4214657140 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fdac80210d0 con 0x563011a63610 2024-03-31T23:10:56.784 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.777+0000 7fdabf7fe640 1 -- 172.21.15.46:0/4214657140 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdac8005240 con 0x563011a63610 2024-03-31T23:10:56.784 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.778+0000 7fdae2c12740 1 -- 172.21.15.46:0/4214657140 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563011a63610 msgr2=0x563011a6d5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:10:56.784 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.778+0000 7fdae2c12740 1 --2- 172.21.15.46:0/4214657140 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563011a63610 0x563011a6d5e0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7fdac80097f0 tx=0x7fdac8009500 comp rx=0 tx=0).stop 2024-03-31T23:10:56.784 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.778+0000 7fdae2c12740 1 -- 172.21.15.46:0/4214657140 shutdown_connections 2024-03-31T23:10:56.785 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.778+0000 7fdae2c12740 1 --2- 172.21.15.46:0/4214657140 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x563011a651a0 0x563011a6db20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:56.785 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.778+0000 7fdae2c12740 1 --2- 172.21.15.46:0/4214657140 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563011a647e0 0x563011a74ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:56.785 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.778+0000 7fdae2c12740 1 --2- 172.21.15.46:0/4214657140 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563011a63610 0x563011a6d5e0 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:56.785 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.778+0000 7fdae2c12740 1 -- 172.21.15.46:0/4214657140 >> 172.21.15.46:0/4214657140 conn(0x563011a5ea00 msgr2=0x563011a5ede0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:10:56.785 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.778+0000 7fdae2c12740 1 -- 172.21.15.46:0/4214657140 shutdown_connections 2024-03-31T23:10:56.785 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.778+0000 7fdae2c12740 1 -- 172.21.15.46:0/4214657140 wait complete. 2024-03-31T23:10:56.785 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.778+0000 7fdae2c12740 1 -- start start 2024-03-31T23:10:56.785 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.778+0000 7fdae2c12740 1 Processor -- start 2024-03-31T23:10:56.785 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.779+0000 7fdae2c12740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563011a63610 0x563011a70930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:10:56.785 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.779+0000 7fdae2c12740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563011a647e0 0x563011a70e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:10:56.785 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.779+0000 7fdae2c12740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x563011a651a0 0x563011a73e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:10:56.785 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.779+0000 7fdae2c12740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x563011a7b8f0 con 0x563011a63610 2024-03-31T23:10:56.785 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.779+0000 7fdae2c12740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x563011a7b220 con 0x563011a651a0 2024-03-31T23:10:56.785 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.779+0000 7fdae2c12740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x563011a7bfc0 con 0x563011a647e0 2024-03-31T23:10:56.785 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.779+0000 7fdae2c12740 10 client.0 _ll_register_callbacks cb 0x7fdad00ee340 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T23:10:56.786 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.779+0000 7fdaddcf2640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563011a63610 0x563011a70930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:10:56.786 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.779+0000 7fdaddcf2640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563011a63610 0x563011a70930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:56446/0 (socket says 172.21.15.46:56446) 2024-03-31T23:10:56.786 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.779+0000 7fdaddcf2640 1 -- 172.21.15.46:0/4175741791 learned_addr learned my addr 172.21.15.46:0/4175741791 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:10:56.786 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.779+0000 7fdade4f3640 1 --2- 172.21.15.46:0/4175741791 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x563011a651a0 0x563011a73e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:10:56.786 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.779+0000 7fdadd4f1640 1 --2- 172.21.15.46:0/4175741791 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563011a647e0 0x563011a70e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:10:56.786 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.779+0000 7fdaddcf2640 1 -- 172.21.15.46:0/4175741791 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563011a647e0 msgr2=0x563011a70e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:10:56.786 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.779+0000 7fdaddcf2640 1 --2- 172.21.15.46:0/4175741791 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563011a647e0 0x563011a70e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:56.786 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.779+0000 7fdaddcf2640 1 -- 172.21.15.46:0/4175741791 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x563011a651a0 msgr2=0x563011a73e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:10:56.786 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.779+0000 7fdaddcf2640 1 --2- 172.21.15.46:0/4175741791 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x563011a651a0 0x563011a73e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:56.786 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.779+0000 7fdaddcf2640 1 -- 172.21.15.46:0/4175741791 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563011a74480 con 0x563011a63610 2024-03-31T23:10:56.786 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.779+0000 7fdadd4f1640 1 --2- 172.21.15.46:0/4175741791 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563011a647e0 0x563011a70e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T23:10:56.786 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.779+0000 7fdaddcf2640 1 --2- 172.21.15.46:0/4175741791 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563011a63610 0x563011a70930 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7fdac8011a00 tx=0x7fdac80044f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=467ff16d5987df6b server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:10:56.786 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.779+0000 7fdabd7fa640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-31T23:10:56.787 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.779+0000 7fdabd7fa640 1 -- 172.21.15.46:0/4175741791 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdac8021280 con 0x563011a63610 2024-03-31T23:10:56.787 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.780+0000 7fdabd7fa640 1 -- 172.21.15.46:0/4175741791 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7fdac8021420 con 0x563011a63610 2024-03-31T23:10:56.787 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.780+0000 7fdae2c12740 1 -- 172.21.15.46:0/4175741791 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdac80210d0 con 0x563011a63610 2024-03-31T23:10:56.787 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.780+0000 7fdabd7fa640 1 -- 172.21.15.46:0/4175741791 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fdac803c050 con 0x563011a63610 2024-03-31T23:10:56.787 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.780+0000 7fdae2c12740 10 client.15276 fetch_fsmap learned FSMap version 24 2024-03-31T23:10:56.787 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.780+0000 7fdae2c12740 1 -- 172.21.15.46:0/4175741791 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x563011ae0730 con 0x563011a63610 2024-03-31T23:10:56.787 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.780+0000 7fdabd7fa640 1 -- 172.21.15.46:0/4175741791 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fdac80210d0 con 0x563011a63610 2024-03-31T23:10:56.787 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.780+0000 7fdabd7fa640 1 -- 172.21.15.46:0/4175741791 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7fdad0003ce0 con 0x563011a63610 2024-03-31T23:10:56.787 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.780+0000 7fdae2c12740 10 client.15276 fetch_fsmap finished waiting for FSMap version 24 2024-03-31T23:10:56.787 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.780+0000 7fdae2c12740 10 client.15276 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-31T23:10:56.787 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.780+0000 7fdae2c12740 20 client.15276 populate_metadata read hostname 'smithi046' 2024-03-31T23:10:56.787 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.780+0000 7fdae2c12740 1 --2- 172.21.15.46:0/4175741791 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x563011ae50a0 0x563011ae14e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:10:56.787 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.780+0000 7fdae2c12740 4 client.15276 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-31T23:10:56.787 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.780+0000 7fdae2c12740 1 -- 172.21.15.46:0/4175741791 --> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] -- command(tid 0: {"prefix": "status"}) -- 0x5630119cf380 con 0x563011ae50a0 2024-03-31T23:10:56.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.780+0000 7fdadd4f1640 1 --2- 172.21.15.46:0/4175741791 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x563011ae50a0 0x563011ae14e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:10:56.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.781+0000 7fdadd4f1640 1 --2- 172.21.15.46:0/4175741791 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x563011ae50a0 0x563011ae14e0 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a3fa38f87180225 server_cookie=5b74dedebdd5eddf in_seq=0 out_seq=0 2024-03-31T23:10:56.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.781+0000 7fdabd7fa640 10 client.15276 ms_handle_connect on v2:172.21.15.142:6838/2519143918 2024-03-31T23:10:56.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.781+0000 7fdabd7fa640 1 -- 172.21.15.46:0/4175741791 <== mds.0 v2:172.21.15.142:6838/2519143918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+321 (crc 0 0 0) 0x5630119cf380 con 0x563011ae50a0 2024-03-31T23:10:56.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.781+0000 7fdabd7fa640 10 client.15276 handle_command_reply: tid=0 2024-03-31T23:10:56.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.781+0000 7fdae2c12740 1 client.15276 shutdown 2024-03-31T23:10:56.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.781+0000 7fdae2c12740 1 -- 172.21.15.46:0/4175741791 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563011a63610 msgr2=0x563011a70930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:10:56.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.781+0000 7fdae2c12740 1 --2- 172.21.15.46:0/4175741791 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563011a63610 0x563011a70930 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7fdac8011a00 tx=0x7fdac80044f0 comp rx=0 tx=0).stop 2024-03-31T23:10:56.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.781+0000 7fdae2c12740 1 -- 172.21.15.46:0/4175741791 shutdown_connections 2024-03-31T23:10:56.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.781+0000 7fdae2c12740 1 --2- 172.21.15.46:0/4175741791 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x563011ae50a0 0x563011ae14e0 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:56.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.781+0000 7fdae2c12740 1 --2- 172.21.15.46:0/4175741791 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x563011a651a0 0x563011a73e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:56.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.781+0000 7fdae2c12740 1 --2- 172.21.15.46:0/4175741791 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563011a647e0 0x563011a70e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:56.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.781+0000 7fdabd7fa640 0 client.15276 ms_handle_reset on v2:172.21.15.142:6838/2519143918 2024-03-31T23:10:56.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.781+0000 7fdae2c12740 1 --2- 172.21.15.46:0/4175741791 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563011a63610 0x563011a70930 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:56.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.781+0000 7fdae2c12740 1 -- 172.21.15.46:0/4175741791 >> 172.21.15.46:0/4175741791 conn(0x563011a5ea00 msgr2=0x563011a5ede0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:10:56.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.781+0000 7fdae2c12740 1 -- 172.21.15.46:0/4175741791 shutdown_connections 2024-03-31T23:10:56.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.781+0000 7fdae2c12740 1 -- 172.21.15.46:0/4175741791 wait complete. 2024-03-31T23:10:56.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.781+0000 7fdae2c12740 20 client.15276 trim_cache size 0 max 16384 2024-03-31T23:10:56.789 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-03-31T23:10:56.789 INFO:teuthology.orchestra.run.smithi046.stdout: "cluster_fsid": "6a9d8f08-efb2-11ee-b647-cb9ed24678a4", 2024-03-31T23:10:56.789 INFO:teuthology.orchestra.run.smithi046.stdout: "whoami": 0, 2024-03-31T23:10:56.789 INFO:teuthology.orchestra.run.smithi046.stdout: "id": 24515, 2024-03-31T23:10:56.789 INFO:teuthology.orchestra.run.smithi046.stdout: "want_state": "up:active", 2024-03-31T23:10:56.789 INFO:teuthology.orchestra.run.smithi046.stdout: "state": "up:active", 2024-03-31T23:10:56.790 INFO:teuthology.orchestra.run.smithi046.stdout: "fs_name": "cephfs", 2024-03-31T23:10:56.790 INFO:teuthology.orchestra.run.smithi046.stdout: "rank_uptime": 105.185403635, 2024-03-31T23:10:56.790 INFO:teuthology.orchestra.run.smithi046.stdout: "mdsmap_epoch": 24, 2024-03-31T23:10:56.790 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap_epoch": 93, 2024-03-31T23:10:56.790 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap_epoch_barrier": 91, 2024-03-31T23:10:56.790 INFO:teuthology.orchestra.run.smithi046.stdout: "uptime": 109.51987625 2024-03-31T23:10:56.790 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-03-31T23:10:56.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.782+0000 7fdadff56640 1 -- 172.21.15.46:0/273563495 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fdac00a24a0 msgr2=0x7fdac00a4960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:10:56.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.782+0000 7fdadff56640 1 --2- 172.21.15.46:0/273563495 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fdac00a24a0 0x7fdac00a4960 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fdacc00f790 tx=0x7fdacc018000 comp rx=0 tx=0).stop 2024-03-31T23:10:56.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.782+0000 7fdadff56640 1 -- 172.21.15.46:0/273563495 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdad81074f0 msgr2=0x7fdad819ef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:10:56.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.782+0000 7fdadff56640 1 --2- 172.21.15.46:0/273563495 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdad81074f0 0x7fdad819ef00 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7fdad0009230 tx=0x7fdad0005e40 comp rx=0 tx=0).stop 2024-03-31T23:10:56.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.782+0000 7fdadff56640 1 -- 172.21.15.46:0/273563495 shutdown_connections 2024-03-31T23:10:56.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.782+0000 7fdadff56640 1 --2- 172.21.15.46:0/273563495 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fdac00a24a0 0x7fdac00a4960 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:56.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.782+0000 7fdadff56640 1 --2- 172.21.15.46:0/273563495 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdad81074f0 0x7fdad819ef00 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:56.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.782+0000 7fdadff56640 1 --2- 172.21.15.46:0/273563495 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdad8104bc0 0x7fdad8197e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:56.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.782+0000 7fdadff56640 1 --2- 172.21.15.46:0/273563495 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdad8102290 0x7fdad8197950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:56.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.782+0000 7fdadff56640 1 -- 172.21.15.46:0/273563495 >> 172.21.15.46:0/273563495 conn(0x7fdad80fbc00 msgr2=0x7fdad8102bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:10:56.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.782+0000 7fdadff56640 1 -- 172.21.15.46:0/273563495 shutdown_connections 2024-03-31T23:10:56.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:56.782+0000 7fdadff56640 1 -- 172.21.15.46:0/273563495 wait complete. 2024-03-31T23:10:56.793 INFO:tasks.cephadm:Setup complete, yielding 2024-03-31T23:10:56.794 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-03-31T23:10:56.806 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi046.front.sepia.ceph.com 2024-03-31T23:10:56.806 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:384f31f3d3aee7f8ba3f60846b8844c049165583 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6a9d8f08-efb2-11ee-b647-cb9ed24678a4 -- bash -c 'ceph orch status' 2024-03-31T23:10:56.807 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:10:56 smithi162 ceph-mon[36237]: pgmap v346: 129 pgs: 129 active+clean; 642 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2024-03-31T23:10:56.807 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:10:56 smithi162 ceph-mon[36237]: from='client.? 172.21.15.46:0/3346041175' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-31T23:10:56.807 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:10:56 smithi162 ceph-mon[36237]: from='client.? 172.21.15.46:0/1877529546' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-31T23:10:56.807 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:10:56 smithi162 ceph-mon[36237]: from='client.? 172.21.15.46:0/4098050552' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-31T23:10:56.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:10:56 smithi142 ceph-mon[35271]: pgmap v346: 129 pgs: 129 active+clean; 642 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2024-03-31T23:10:56.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:10:56 smithi142 ceph-mon[35271]: from='client.? 172.21.15.46:0/3346041175' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-31T23:10:56.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:10:56 smithi142 ceph-mon[35271]: from='client.? 172.21.15.46:0/1877529546' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-31T23:10:56.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:10:56 smithi142 ceph-mon[35271]: from='client.? 172.21.15.46:0/4098050552' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-31T23:10:57.050 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/6a9d8f08-efb2-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-31T23:10:58.716 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:10:58 smithi046 ceph-mon[32498]: pgmap v347: 129 pgs: 129 active+clean; 642 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-03-31T23:10:58.795 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:10:58 smithi162 ceph-mon[36237]: pgmap v347: 129 pgs: 129 active+clean; 642 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-03-31T23:10:58.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:10:58 smithi142 ceph-mon[35271]: pgmap v347: 129 pgs: 129 active+clean; 642 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-03-31T23:10:58.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:58.943+0000 7ff700c7b700 1 -- 172.21.15.46:0/2088988071 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff6fc103470 msgr2=0x7ff6fc1038d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:10:58.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:58.943+0000 7ff700c7b700 1 --2- 172.21.15.46:0/2088988071 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff6fc103470 0x7ff6fc1038d0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7ff6f00099c0 tx=0x7ff6f0009cd0 comp rx=0 tx=0).stop 2024-03-31T23:10:58.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:58.943+0000 7ff700c7b700 1 -- 172.21.15.46:0/2088988071 shutdown_connections 2024-03-31T23:10:58.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:58.943+0000 7ff700c7b700 1 --2- 172.21.15.46:0/2088988071 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff6fc103e10 0x7ff6fc10a6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:58.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:58.943+0000 7ff700c7b700 1 --2- 172.21.15.46:0/2088988071 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff6fc102270 0x7ff6fc102690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:58.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:58.943+0000 7ff700c7b700 1 --2- 172.21.15.46:0/2088988071 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff6fc103470 0x7ff6fc1038d0 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:58.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:58.943+0000 7ff700c7b700 1 -- 172.21.15.46:0/2088988071 >> 172.21.15.46:0/2088988071 conn(0x7ff6fc0fd850 msgr2=0x7ff6fc0ffc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:10:58.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:58.943+0000 7ff700c7b700 1 -- 172.21.15.46:0/2088988071 shutdown_connections 2024-03-31T23:10:58.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:58.943+0000 7ff700c7b700 1 -- 172.21.15.46:0/2088988071 wait complete. 2024-03-31T23:10:58.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:58.944+0000 7ff700c7b700 1 Processor -- start 2024-03-31T23:10:58.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:58.944+0000 7ff700c7b700 1 -- start start 2024-03-31T23:10:58.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:58.944+0000 7ff700c7b700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff6fc102270 0x7ff6fc198380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:10:58.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:58.944+0000 7ff700c7b700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff6fc103470 0x7ff6fc1988c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:10:58.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:58.944+0000 7ff700c7b700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff6fc103e10 0x7ff6fc19f940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:10:58.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:58.944+0000 7ff700c7b700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7ff6fc10dcf0 con 0x7ff6fc102270 2024-03-31T23:10:58.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:58.944+0000 7ff700c7b700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7ff6fc10db70 con 0x7ff6fc103e10 2024-03-31T23:10:58.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:58.944+0000 7ff700c7b700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7ff6fc10de70 con 0x7ff6fc103470 2024-03-31T23:10:58.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:58.944+0000 7ff6fa59c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff6fc102270 0x7ff6fc198380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:10:58.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:58.944+0000 7ff6f9d9b700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff6fc103470 0x7ff6fc1988c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:10:58.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:58.944+0000 7ff6fa59c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff6fc102270 0x7ff6fc198380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:56452/0 (socket says 172.21.15.46:56452) 2024-03-31T23:10:58.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:58.944+0000 7ff6fa59c700 1 -- 172.21.15.46:0/344238829 learned_addr learned my addr 172.21.15.46:0/344238829 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:10:58.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:58.944+0000 7ff6fad9d700 1 --2- 172.21.15.46:0/344238829 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff6fc103e10 0x7ff6fc19f940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:10:58.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:58.944+0000 7ff6f9d9b700 1 -- 172.21.15.46:0/344238829 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff6fc103e10 msgr2=0x7ff6fc19f940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:10:58.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:58.944+0000 7ff6f9d9b700 1 --2- 172.21.15.46:0/344238829 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff6fc103e10 0x7ff6fc19f940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:58.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:58.944+0000 7ff6f9d9b700 1 -- 172.21.15.46:0/344238829 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff6fc102270 msgr2=0x7ff6fc198380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:10:58.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:58.944+0000 7ff6f9d9b700 1 --2- 172.21.15.46:0/344238829 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff6fc102270 0x7ff6fc198380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:58.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:58.944+0000 7ff6f9d9b700 1 -- 172.21.15.46:0/344238829 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6fc19ffe0 con 0x7ff6fc103470 2024-03-31T23:10:58.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:58.944+0000 7ff6fa59c700 1 --2- 172.21.15.46:0/344238829 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff6fc102270 0x7ff6fc198380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:10:58.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:58.945+0000 7ff6f9d9b700 1 --2- 172.21.15.46:0/344238829 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff6fc103470 0x7ff6fc1988c0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7ff6f00098e0 tx=0x7ff6f000f710 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:10:58.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:58.945+0000 7ff6eb7fe700 1 -- 172.21.15.46:0/344238829 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6f0009e60 con 0x7ff6fc103470 2024-03-31T23:10:58.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:58.945+0000 7ff700c7b700 1 -- 172.21.15.46:0/344238829 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff6fc1a0270 con 0x7ff6fc103470 2024-03-31T23:10:58.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:58.945+0000 7ff6eb7fe700 1 -- 172.21.15.46:0/344238829 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff6f0005940 con 0x7ff6fc103470 2024-03-31T23:10:58.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:58.945+0000 7ff6eb7fe700 1 -- 172.21.15.46:0/344238829 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6f0005450 con 0x7ff6fc103470 2024-03-31T23:10:58.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:58.945+0000 7ff700c7b700 1 -- 172.21.15.46:0/344238829 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff6fc1a0750 con 0x7ff6fc103470 2024-03-31T23:10:58.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:58.946+0000 7ff700c7b700 1 -- 172.21.15.46:0/344238829 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff6fc068f00 con 0x7ff6fc103470 2024-03-31T23:10:58.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:58.947+0000 7ff6eb7fe700 1 -- 172.21.15.46:0/344238829 <== mon.1 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 18) ==== 137320+0+0 (secure 0 0 0) 0x7ff6f0003b30 con 0x7ff6fc103470 2024-03-31T23:10:58.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:58.948+0000 7ff6eb7fe700 1 --2- 172.21.15.46:0/344238829 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7ff6e40a2530 0x7ff6e40a49f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:10:58.950 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:58.948+0000 7ff6fa59c700 1 --2- 172.21.15.46:0/344238829 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7ff6e40a2530 0x7ff6e40a49f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:10:58.950 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:58.948+0000 7ff6eb7fe700 1 -- 172.21.15.46:0/344238829 <== mon.1 v2:172.21.15.162:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7ff6f00cf750 con 0x7ff6fc103470 2024-03-31T23:10:58.950 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:58.948+0000 7ff6fa59c700 1 --2- 172.21.15.46:0/344238829 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7ff6e40a2530 0x7ff6e40a49f0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7ff6ec005950 tx=0x7ff6ec009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:10:58.952 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:58.951+0000 7ff6eb7fe700 1 -- 172.21.15.46:0/344238829 <== mon.1 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7ff6f0013070 con 0x7ff6fc103470 2024-03-31T23:10:59.178 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:59.177+0000 7ff700c7b700 1 -- 172.21.15.46:0/344238829 --> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] -- mgr_command(tid 0: {"prefix": "orch status", "target": ["mon-mgr", ""]}) -- 0x7ff6fc062740 con 0x7ff6e40a2530 2024-03-31T23:10:59.179 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:59.178+0000 7ff6eb7fe700 1 -- 172.21.15.46:0/344238829 <== mgr.14152 v2:172.21.15.46:6800/162289569 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+43 (secure 0 0 0) 0x7ff6fc062740 con 0x7ff6e40a2530 2024-03-31T23:10:59.179 INFO:teuthology.orchestra.run.smithi046.stdout:Backend: cephadm 2024-03-31T23:10:59.179 INFO:teuthology.orchestra.run.smithi046.stdout:Available: Yes 2024-03-31T23:10:59.179 INFO:teuthology.orchestra.run.smithi046.stdout:Paused: No 2024-03-31T23:10:59.181 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:59.180+0000 7ff700c7b700 1 -- 172.21.15.46:0/344238829 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7ff6e40a2530 msgr2=0x7ff6e40a49f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:10:59.181 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:59.180+0000 7ff700c7b700 1 --2- 172.21.15.46:0/344238829 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7ff6e40a2530 0x7ff6e40a49f0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7ff6ec005950 tx=0x7ff6ec009260 comp rx=0 tx=0).stop 2024-03-31T23:10:59.181 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:59.180+0000 7ff700c7b700 1 -- 172.21.15.46:0/344238829 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff6fc103470 msgr2=0x7ff6fc1988c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:10:59.181 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:59.180+0000 7ff700c7b700 1 --2- 172.21.15.46:0/344238829 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff6fc103470 0x7ff6fc1988c0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7ff6f00098e0 tx=0x7ff6f000f710 comp rx=0 tx=0).stop 2024-03-31T23:10:59.181 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:59.180+0000 7ff700c7b700 1 -- 172.21.15.46:0/344238829 shutdown_connections 2024-03-31T23:10:59.181 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:59.180+0000 7ff700c7b700 1 --2- 172.21.15.46:0/344238829 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7ff6e40a2530 0x7ff6e40a49f0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:59.181 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:59.180+0000 7ff700c7b700 1 --2- 172.21.15.46:0/344238829 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff6fc102270 0x7ff6fc198380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:59.181 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:59.180+0000 7ff700c7b700 1 --2- 172.21.15.46:0/344238829 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff6fc103e10 0x7ff6fc19f940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:59.182 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:59.180+0000 7ff700c7b700 1 --2- 172.21.15.46:0/344238829 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff6fc103470 0x7ff6fc1988c0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:10:59.182 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:59.180+0000 7ff700c7b700 1 -- 172.21.15.46:0/344238829 >> 172.21.15.46:0/344238829 conn(0x7ff6fc0fd850 msgr2=0x7ff6fc106840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:10:59.182 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:59.180+0000 7ff700c7b700 1 -- 172.21.15.46:0/344238829 shutdown_connections 2024-03-31T23:10:59.182 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:10:59.180+0000 7ff700c7b700 1 -- 172.21.15.46:0/344238829 wait complete. 2024-03-31T23:11:00.032 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:384f31f3d3aee7f8ba3f60846b8844c049165583 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6a9d8f08-efb2-11ee-b647-cb9ed24678a4 -- bash -c 'ceph orch ps' 2024-03-31T23:11:00.283 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/6a9d8f08-efb2-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-31T23:11:00.676 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:11:00 smithi046 ceph-mon[32498]: pgmap v348: 129 pgs: 129 active+clean; 642 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-31T23:11:00.676 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:11:00 smithi046 ceph-mon[32498]: from='client.25027 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-31T23:11:00.795 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:11:00 smithi162 ceph-mon[36237]: pgmap v348: 129 pgs: 129 active+clean; 642 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-31T23:11:00.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:11:00 smithi162 ceph-mon[36237]: from='client.25027 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-31T23:11:00.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:11:00 smithi142 ceph-mon[35271]: pgmap v348: 129 pgs: 129 active+clean; 642 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-31T23:11:00.944 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:11:00 smithi142 ceph-mon[35271]: from='client.25027 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-31T23:11:02.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:02.092+0000 7fefd2997700 1 -- 172.21.15.46:0/1369526860 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fefcc102d10 msgr2=0x7fefcc1095c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:02.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:02.092+0000 7fefd2997700 1 --2- 172.21.15.46:0/1369526860 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fefcc102d10 0x7fefcc1095c0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7fefc400b0d0 tx=0x7fefc400b3e0 comp rx=0 tx=0).stop 2024-03-31T23:11:02.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:02.092+0000 7fefd2997700 1 -- 172.21.15.46:0/1369526860 shutdown_connections 2024-03-31T23:11:02.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:02.092+0000 7fefd2997700 1 --2- 172.21.15.46:0/1369526860 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fefcc102d10 0x7fefcc1095c0 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:02.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:02.092+0000 7fefd2997700 1 --2- 172.21.15.46:0/1369526860 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fefcc101170 0x7fefcc101590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:02.095 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:02.092+0000 7fefd2997700 1 --2- 172.21.15.46:0/1369526860 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fefcc102370 0x7fefcc1027d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:02.095 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:02.092+0000 7fefd2997700 1 -- 172.21.15.46:0/1369526860 >> 172.21.15.46:0/1369526860 conn(0x7fefcc0fc730 msgr2=0x7fefcc0feb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:11:02.095 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:02.092+0000 7fefd2997700 1 -- 172.21.15.46:0/1369526860 shutdown_connections 2024-03-31T23:11:02.095 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:02.092+0000 7fefd2997700 1 -- 172.21.15.46:0/1369526860 wait complete. 2024-03-31T23:11:02.095 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:02.093+0000 7fefd2997700 1 Processor -- start 2024-03-31T23:11:02.095 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:02.093+0000 7fefd2997700 1 -- start start 2024-03-31T23:11:02.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:02.093+0000 7fefd2997700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fefcc101170 0x7fefcc19a4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:02.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:02.093+0000 7fefd2997700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fefcc102370 0x7fefcc19aa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:02.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:02.093+0000 7fefd2997700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fefcc102d10 0x7fefcc1a1aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:02.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:02.093+0000 7fefd2997700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fefcc10cc70 con 0x7fefcc102d10 2024-03-31T23:11:02.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:02.093+0000 7fefd2997700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fefcc10caf0 con 0x7fefcc101170 2024-03-31T23:11:02.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:02.093+0000 7fefd2997700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fefcc10cdf0 con 0x7fefcc102370 2024-03-31T23:11:02.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:02.094+0000 7fefcbfff700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fefcc101170 0x7fefcc19a4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:02.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:02.094+0000 7fefcb7fe700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fefcc102370 0x7fefcc19aa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:02.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:02.094+0000 7fefd0f34700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fefcc102d10 0x7fefcc1a1aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:02.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:02.094+0000 7fefcb7fe700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fefcc102370 0x7fefcc19aa20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.46:39236/0 (socket says 172.21.15.46:39236) 2024-03-31T23:11:02.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:02.094+0000 7fefcb7fe700 1 -- 172.21.15.46:0/33962266 learned_addr learned my addr 172.21.15.46:0/33962266 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:11:02.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:02.094+0000 7fefcbfff700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fefcc101170 0x7fefcc19a4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.46:58562/0 (socket says 172.21.15.46:58562) 2024-03-31T23:11:02.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:02.094+0000 7fefcb7fe700 1 -- 172.21.15.46:0/33962266 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fefcc101170 msgr2=0x7fefcc19a4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:02.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:02.094+0000 7fefcb7fe700 1 --2- 172.21.15.46:0/33962266 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fefcc101170 0x7fefcc19a4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:02.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:02.094+0000 7fefcb7fe700 1 -- 172.21.15.46:0/33962266 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fefcc102d10 msgr2=0x7fefcc1a1aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:02.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:02.094+0000 7fefcb7fe700 1 --2- 172.21.15.46:0/33962266 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fefcc102d10 0x7fefcc1a1aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:02.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:02.094+0000 7fefcb7fe700 1 -- 172.21.15.46:0/33962266 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fefcc1a2140 con 0x7fefcc102370 2024-03-31T23:11:02.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:02.094+0000 7fefcb7fe700 1 --2- 172.21.15.46:0/33962266 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fefcc102370 0x7fefcc19aa20 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fefc000e9f0 tx=0x7fefc000ed00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:11:02.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:02.094+0000 7fefc97fa700 1 -- 172.21.15.46:0/33962266 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefc000cbd0 con 0x7fefcc102370 2024-03-31T23:11:02.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:02.094+0000 7fefc97fa700 1 -- 172.21.15.46:0/33962266 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fefc000ee90 con 0x7fefcc102370 2024-03-31T23:11:02.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:02.094+0000 7fefd2997700 1 -- 172.21.15.46:0/33962266 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fefcc1a2430 con 0x7fefcc102370 2024-03-31T23:11:02.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:02.094+0000 7fefc97fa700 1 -- 172.21.15.46:0/33962266 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefc0005590 con 0x7fefcc102370 2024-03-31T23:11:02.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:02.094+0000 7fefd2997700 1 -- 172.21.15.46:0/33962266 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fefcc1a2920 con 0x7fefcc102370 2024-03-31T23:11:02.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:02.096+0000 7fefd2997700 1 -- 172.21.15.46:0/33962266 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fefcc068f00 con 0x7fefcc102370 2024-03-31T23:11:02.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:02.096+0000 7fefc97fa700 1 -- 172.21.15.46:0/33962266 <== mon.1 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 18) ==== 137320+0+0 (secure 0 0 0) 0x7fefc0004750 con 0x7fefcc102370 2024-03-31T23:11:02.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:02.097+0000 7fefc97fa700 1 --2- 172.21.15.46:0/33962266 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fefb40a2530 0x7fefb40a49f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:02.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:02.097+0000 7fefcbfff700 1 --2- 172.21.15.46:0/33962266 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fefb40a2530 0x7fefb40a49f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:02.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:02.097+0000 7fefc97fa700 1 -- 172.21.15.46:0/33962266 <== mon.1 v2:172.21.15.162:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7fefc00ced20 con 0x7fefcc102370 2024-03-31T23:11:02.100 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:02.098+0000 7fefcbfff700 1 --2- 172.21.15.46:0/33962266 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fefb40a2530 0x7fefb40a49f0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fefcc1021d0 tx=0x7fefbc00a3b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:11:02.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:02.102+0000 7fefc97fa700 1 -- 172.21.15.46:0/33962266 <== mon.1 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fefc009bb80 con 0x7fefcc102370 2024-03-31T23:11:02.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:02.318+0000 7fefd2997700 1 -- 172.21.15.46:0/33962266 --> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) -- 0x7fefcc1a2be0 con 0x7fefb40a2530 2024-03-31T23:11:02.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:02.332+0000 7fefc97fa700 1 -- 172.21.15.46:0/33962266 <== mgr.14152 v2:172.21.15.46:6800/162289569 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+4185 (secure 0 0 0) 0x7fefcc1a2be0 con 0x7fefb40a2530 2024-03-31T23:11:02.334 INFO:teuthology.orchestra.run.smithi046.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-31T23:11:02.334 INFO:teuthology.orchestra.run.smithi046.stdout:mds.a smithi046 running (118s) 24s ago 117s 15.1M - 19.0.0-1977-g384f31f3 ec18d7baa60e c6deca2be3ec 2024-03-31T23:11:02.334 INFO:teuthology.orchestra.run.smithi046.stdout:mds.b smithi142 running (115s) 49s ago 114s 26.1M - 19.0.0-1977-g384f31f3 ec18d7baa60e 162eacae90f3 2024-03-31T23:11:02.334 INFO:teuthology.orchestra.run.smithi046.stdout:mds.c smithi162 running (2m) 49s ago 2m 19.7M - 19.0.0-1977-g384f31f3 ec18d7baa60e 6c86ca0bcc43 2024-03-31T23:11:02.334 INFO:teuthology.orchestra.run.smithi046.stdout:mds.d smithi046 running (2m) 24s ago 2m 16.4M - 19.0.0-1977-g384f31f3 ec18d7baa60e 70ad99574a65 2024-03-31T23:11:02.334 INFO:teuthology.orchestra.run.smithi046.stdout:mds.e smithi142 running (2m) 49s ago 2m 16.2M - 19.0.0-1977-g384f31f3 ec18d7baa60e 42bbbb0a6bd1 2024-03-31T23:11:02.334 INFO:teuthology.orchestra.run.smithi046.stdout:mds.f smithi162 running (2m) 49s ago 2m 15.5M - 19.0.0-1977-g384f31f3 ec18d7baa60e b67d463eb8fe 2024-03-31T23:11:02.334 INFO:teuthology.orchestra.run.smithi046.stdout:mds.g smithi046 running (2m) 24s ago 2m 12.9M - 19.0.0-1977-g384f31f3 ec18d7baa60e b4331229633e 2024-03-31T23:11:02.334 INFO:teuthology.orchestra.run.smithi046.stdout:mds.h smithi142 running (2m) 49s ago 2m 17.6M - 19.0.0-1977-g384f31f3 ec18d7baa60e eeb232fe0e9b 2024-03-31T23:11:02.334 INFO:teuthology.orchestra.run.smithi046.stdout:mds.i smithi162 running (2m) 49s ago 2m 17.2M - 19.0.0-1977-g384f31f3 ec18d7baa60e 608a1c3cee32 2024-03-31T23:11:02.335 INFO:teuthology.orchestra.run.smithi046.stdout:mds.j smithi046 running (2m) 24s ago 2m 13.6M - 19.0.0-1977-g384f31f3 ec18d7baa60e 85b09f805600 2024-03-31T23:11:02.335 INFO:teuthology.orchestra.run.smithi046.stdout:mds.k smithi142 running (2m) 49s ago 2m 15.2M - 19.0.0-1977-g384f31f3 ec18d7baa60e 7f5af54f1631 2024-03-31T23:11:02.335 INFO:teuthology.orchestra.run.smithi046.stdout:mds.l smithi162 running (2m) 49s ago 2m 13.0M - 19.0.0-1977-g384f31f3 ec18d7baa60e dad2ed6cb60b 2024-03-31T23:11:02.335 INFO:teuthology.orchestra.run.smithi046.stdout:mgr.x smithi046 *:9283,8765 running (9m) 24s ago 9m 513M - 19.0.0-1977-g384f31f3 ec18d7baa60e 2adfecce4e56 2024-03-31T23:11:02.335 INFO:teuthology.orchestra.run.smithi046.stdout:mgr.y smithi142 *:8443,8765 running (7m) 49s ago 7m 451M - 19.0.0-1977-g384f31f3 ec18d7baa60e 1db2604d0e85 2024-03-31T23:11:02.335 INFO:teuthology.orchestra.run.smithi046.stdout:mgr.z smithi162 *:8443,8765 running (7m) 49s ago 7m 452M - 19.0.0-1977-g384f31f3 ec18d7baa60e e44587aca31b 2024-03-31T23:11:02.335 INFO:teuthology.orchestra.run.smithi046.stdout:mon.a smithi046 running (9m) 24s ago 9m 52.4M 2048M 19.0.0-1977-g384f31f3 ec18d7baa60e 857af8e9024c 2024-03-31T23:11:02.335 INFO:teuthology.orchestra.run.smithi046.stdout:mon.b smithi142 running (7m) 49s ago 7m 41.4M 2048M 19.0.0-1977-g384f31f3 ec18d7baa60e 5a36e49c19c9 2024-03-31T23:11:02.335 INFO:teuthology.orchestra.run.smithi046.stdout:mon.c smithi162 running (8m) 49s ago 8m 39.6M 2048M 19.0.0-1977-g384f31f3 ec18d7baa60e ce17b19294be 2024-03-31T23:11:02.335 INFO:teuthology.orchestra.run.smithi046.stdout:osd.0 smithi046 running (7m) 24s ago 7m 53.7M 4273M 19.0.0-1977-g384f31f3 ec18d7baa60e 534c29b008e3 2024-03-31T23:11:02.335 INFO:teuthology.orchestra.run.smithi046.stdout:osd.1 smithi142 running (6m) 49s ago 6m 46.7M 4272M 19.0.0-1977-g384f31f3 ec18d7baa60e d72da928432b 2024-03-31T23:11:02.335 INFO:teuthology.orchestra.run.smithi046.stdout:osd.2 smithi162 running (6m) 49s ago 6m 47.4M 4272M 19.0.0-1977-g384f31f3 ec18d7baa60e 65c531cf5c18 2024-03-31T23:11:02.336 INFO:teuthology.orchestra.run.smithi046.stdout:osd.3 smithi046 running (6m) 24s ago 6m 44.7M 4273M 19.0.0-1977-g384f31f3 ec18d7baa60e 39269df5e56d 2024-03-31T23:11:02.336 INFO:teuthology.orchestra.run.smithi046.stdout:osd.4 smithi142 running (5m) 49s ago 5m 50.3M 4272M 19.0.0-1977-g384f31f3 ec18d7baa60e 2f2a509d7641 2024-03-31T23:11:02.336 INFO:teuthology.orchestra.run.smithi046.stdout:osd.5 smithi162 running (5m) 49s ago 5m 53.4M 4272M 19.0.0-1977-g384f31f3 ec18d7baa60e 2d982a602896 2024-03-31T23:11:02.336 INFO:teuthology.orchestra.run.smithi046.stdout:osd.6 smithi046 running (4m) 24s ago 4m 42.2M 4273M 19.0.0-1977-g384f31f3 ec18d7baa60e f2dca9d6110a 2024-03-31T23:11:02.336 INFO:teuthology.orchestra.run.smithi046.stdout:osd.7 smithi142 running (4m) 49s ago 4m 44.0M 4272M 19.0.0-1977-g384f31f3 ec18d7baa60e 3d1126147b49 2024-03-31T23:11:02.336 INFO:teuthology.orchestra.run.smithi046.stdout:osd.8 smithi162 running (4m) 49s ago 4m 46.0M 4272M 19.0.0-1977-g384f31f3 ec18d7baa60e 665072d8f712 2024-03-31T23:11:02.336 INFO:teuthology.orchestra.run.smithi046.stdout:osd.9 smithi046 running (3m) 24s ago 3m 43.0M 4273M 19.0.0-1977-g384f31f3 ec18d7baa60e fad960dfb69f 2024-03-31T23:11:02.336 INFO:teuthology.orchestra.run.smithi046.stdout:osd.10 smithi142 running (3m) 49s ago 3m 43.1M 4272M 19.0.0-1977-g384f31f3 ec18d7baa60e 7bf1dd84a351 2024-03-31T23:11:02.336 INFO:teuthology.orchestra.run.smithi046.stdout:osd.11 smithi162 running (2m) 49s ago 2m 48.8M 4272M 19.0.0-1977-g384f31f3 ec18d7baa60e 642aa2603b73 2024-03-31T23:11:02.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:02.334+0000 7fefd2997700 1 -- 172.21.15.46:0/33962266 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fefb40a2530 msgr2=0x7fefb40a49f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:02.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:02.334+0000 7fefd2997700 1 --2- 172.21.15.46:0/33962266 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fefb40a2530 0x7fefb40a49f0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fefcc1021d0 tx=0x7fefbc00a3b0 comp rx=0 tx=0).stop 2024-03-31T23:11:02.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:02.334+0000 7fefd2997700 1 -- 172.21.15.46:0/33962266 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fefcc102370 msgr2=0x7fefcc19aa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:02.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:02.335+0000 7fefd2997700 1 --2- 172.21.15.46:0/33962266 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fefcc102370 0x7fefcc19aa20 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fefc000e9f0 tx=0x7fefc000ed00 comp rx=0 tx=0).stop 2024-03-31T23:11:02.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:02.335+0000 7fefd2997700 1 -- 172.21.15.46:0/33962266 shutdown_connections 2024-03-31T23:11:02.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:02.335+0000 7fefd2997700 1 --2- 172.21.15.46:0/33962266 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fefb40a2530 0x7fefb40a49f0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:02.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:02.335+0000 7fefd2997700 1 --2- 172.21.15.46:0/33962266 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fefcc102d10 0x7fefcc1a1aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:02.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:02.335+0000 7fefd2997700 1 --2- 172.21.15.46:0/33962266 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fefcc101170 0x7fefcc19a4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:02.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:02.335+0000 7fefd2997700 1 --2- 172.21.15.46:0/33962266 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fefcc102370 0x7fefcc19aa20 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:02.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:02.335+0000 7fefd2997700 1 -- 172.21.15.46:0/33962266 >> 172.21.15.46:0/33962266 conn(0x7fefcc0fc730 msgr2=0x7fefcc0fe0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:11:02.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:02.335+0000 7fefd2997700 1 -- 172.21.15.46:0/33962266 shutdown_connections 2024-03-31T23:11:02.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:02.335+0000 7fefd2997700 1 -- 172.21.15.46:0/33962266 wait complete. 2024-03-31T23:11:02.611 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:11:02 smithi046 ceph-mon[32498]: pgmap v349: 129 pgs: 129 active+clean; 642 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-03-31T23:11:02.795 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:11:02 smithi162 ceph-mon[36237]: pgmap v349: 129 pgs: 129 active+clean; 642 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-03-31T23:11:02.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:11:02 smithi142 ceph-mon[35271]: pgmap v349: 129 pgs: 129 active+clean; 642 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-03-31T23:11:03.140 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:384f31f3d3aee7f8ba3f60846b8844c049165583 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6a9d8f08-efb2-11ee-b647-cb9ed24678a4 -- bash -c 'ceph orch ls' 2024-03-31T23:11:03.391 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/6a9d8f08-efb2-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-31T23:11:03.778 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:11:03 smithi046 ceph-mon[32498]: from='client.25033 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-31T23:11:03.795 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:11:03 smithi162 ceph-mon[36237]: from='client.25033 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-31T23:11:03.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:11:03 smithi142 ceph-mon[35271]: from='client.25033 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-31T23:11:04.795 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:11:04 smithi162 ceph-mon[36237]: pgmap v350: 129 pgs: 129 active+clean; 642 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-31T23:11:04.837 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:11:04 smithi046 ceph-mon[32498]: pgmap v350: 129 pgs: 129 active+clean; 642 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-31T23:11:04.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:11:04 smithi142 ceph-mon[35271]: pgmap v350: 129 pgs: 129 active+clean; 642 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-31T23:11:05.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:05.055+0000 7f1db0ece700 1 -- 172.21.15.46:0/629053467 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1dac105680 msgr2=0x7f1dac107a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:05.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:05.055+0000 7f1db0ece700 1 --2- 172.21.15.46:0/629053467 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1dac105680 0x7f1dac107a70 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f1d94009990 tx=0x7f1d94009ca0 comp rx=0 tx=0).stop 2024-03-31T23:11:05.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:05.055+0000 7f1db0ece700 1 -- 172.21.15.46:0/629053467 shutdown_connections 2024-03-31T23:11:05.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:05.055+0000 7f1db0ece700 1 --2- 172.21.15.46:0/629053467 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1dac107fb0 0x7f1dac10a3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:05.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:05.055+0000 7f1db0ece700 1 --2- 172.21.15.46:0/629053467 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1dac102d50 0x7f1dac105140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:05.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:05.055+0000 7f1db0ece700 1 --2- 172.21.15.46:0/629053467 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1dac105680 0x7f1dac107a70 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:05.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:05.055+0000 7f1db0ece700 1 -- 172.21.15.46:0/629053467 >> 172.21.15.46:0/629053467 conn(0x7f1dac0fc700 msgr2=0x7f1dac0feb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:11:05.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:05.055+0000 7f1db0ece700 1 -- 172.21.15.46:0/629053467 shutdown_connections 2024-03-31T23:11:05.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:05.055+0000 7f1db0ece700 1 -- 172.21.15.46:0/629053467 wait complete. 2024-03-31T23:11:05.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:05.056+0000 7f1db0ece700 1 Processor -- start 2024-03-31T23:11:05.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:05.056+0000 7f1db0ece700 1 -- start start 2024-03-31T23:11:05.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:05.056+0000 7f1db0ece700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1dac102d50 0x7f1dac1983d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:05.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:05.056+0000 7f1db0ece700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1dac105680 0x7f1dac198910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:05.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:05.056+0000 7f1db0ece700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1dac107fb0 0x7f1dac19f990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:05.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:05.056+0000 7f1db0ece700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f1dac10dd20 con 0x7f1dac102d50 2024-03-31T23:11:05.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:05.056+0000 7f1db0ece700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f1dac10dba0 con 0x7f1dac107fb0 2024-03-31T23:11:05.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:05.056+0000 7f1db0ece700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f1dac10dea0 con 0x7f1dac105680 2024-03-31T23:11:05.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:05.056+0000 7f1daa59c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1dac102d50 0x7f1dac1983d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:05.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:05.056+0000 7f1daa59c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1dac102d50 0x7f1dac1983d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39038/0 (socket says 172.21.15.46:39038) 2024-03-31T23:11:05.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:05.056+0000 7f1daa59c700 1 -- 172.21.15.46:0/2154424359 learned_addr learned my addr 172.21.15.46:0/2154424359 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:11:05.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:05.056+0000 7f1da9d9b700 1 --2- 172.21.15.46:0/2154424359 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1dac105680 0x7f1dac198910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:05.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:05.056+0000 7f1daad9d700 1 --2- 172.21.15.46:0/2154424359 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1dac107fb0 0x7f1dac19f990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:05.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:05.056+0000 7f1daa59c700 1 -- 172.21.15.46:0/2154424359 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1dac105680 msgr2=0x7f1dac198910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:05.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:05.056+0000 7f1daa59c700 1 --2- 172.21.15.46:0/2154424359 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1dac105680 0x7f1dac198910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:05.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:05.056+0000 7f1daa59c700 1 -- 172.21.15.46:0/2154424359 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1dac107fb0 msgr2=0x7f1dac19f990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:05.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:05.056+0000 7f1daa59c700 1 --2- 172.21.15.46:0/2154424359 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1dac107fb0 0x7f1dac19f990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:05.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:05.057+0000 7f1daa59c700 1 -- 172.21.15.46:0/2154424359 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1dac1a0030 con 0x7f1dac102d50 2024-03-31T23:11:05.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:05.057+0000 7f1daa59c700 1 --2- 172.21.15.46:0/2154424359 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1dac102d50 0x7f1dac1983d0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f1da400ef60 tx=0x7f1da400ef90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:11:05.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:05.057+0000 7f1da37fe700 1 -- 172.21.15.46:0/2154424359 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1da40092e0 con 0x7f1dac102d50 2024-03-31T23:11:05.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:05.057+0000 7f1db0ece700 1 -- 172.21.15.46:0/2154424359 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1dac1a02c0 con 0x7f1dac102d50 2024-03-31T23:11:05.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:05.057+0000 7f1da37fe700 1 -- 172.21.15.46:0/2154424359 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1da4004bb0 con 0x7f1dac102d50 2024-03-31T23:11:05.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:05.057+0000 7f1da37fe700 1 -- 172.21.15.46:0/2154424359 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1da4004280 con 0x7f1dac102d50 2024-03-31T23:11:05.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:05.057+0000 7f1db0ece700 1 -- 172.21.15.46:0/2154424359 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1dac1a0800 con 0x7f1dac102d50 2024-03-31T23:11:05.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:05.058+0000 7f1db0ece700 1 -- 172.21.15.46:0/2154424359 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1dac068f00 con 0x7f1dac102d50 2024-03-31T23:11:05.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:05.059+0000 7f1da37fe700 1 -- 172.21.15.46:0/2154424359 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 18) ==== 137320+0+0 (secure 0 0 0) 0x7f1da401b030 con 0x7f1dac102d50 2024-03-31T23:11:05.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:05.060+0000 7f1da37fe700 1 --2- 172.21.15.46:0/2154424359 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f1d980a2530 0x7f1d980a49f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:05.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:05.060+0000 7f1da9d9b700 1 --2- 172.21.15.46:0/2154424359 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f1d980a2530 0x7f1d980a49f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:05.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:05.060+0000 7f1da37fe700 1 -- 172.21.15.46:0/2154424359 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7f1da40cef20 con 0x7f1dac102d50 2024-03-31T23:11:05.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:05.060+0000 7f1da9d9b700 1 --2- 172.21.15.46:0/2154424359 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f1d980a2530 0x7f1d980a49f0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f1d9400b710 tx=0x7f1d94005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:11:05.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:05.064+0000 7f1da37fe700 1 -- 172.21.15.46:0/2154424359 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f1da4022020 con 0x7f1dac102d50 2024-03-31T23:11:05.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:05.275+0000 7f1db0ece700 1 -- 172.21.15.46:0/2154424359 --> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] -- mgr_command(tid 0: {"prefix": "orch ls", "target": ["mon-mgr", ""]}) -- 0x7f1dac062740 con 0x7f1d980a2530 2024-03-31T23:11:05.282 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:05.281+0000 7f1da37fe700 1 -- 172.21.15.46:0/2154424359 <== mgr.14152 v2:172.21.15.46:6800/162289569 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+980 (secure 0 0 0) 0x7f1dac062740 con 0x7f1d980a2530 2024-03-31T23:11:05.282 INFO:teuthology.orchestra.run.smithi046.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2024-03-31T23:11:05.282 INFO:teuthology.orchestra.run.smithi046.stdout:mds.all 12/12 52s ago 2m smithi046=a;smithi046=d;smithi046=g;smithi046=j;smithi142=b;smithi142=e;smithi142=h;smithi142=k;smithi162=c;smithi162=f;smithi162=i;smithi162=l;count:12 2024-03-31T23:11:05.282 INFO:teuthology.orchestra.run.smithi046.stdout:mgr 3/3 52s ago 7m smithi046=x;smithi142=y;smithi162=z;count:3 2024-03-31T23:11:05.283 INFO:teuthology.orchestra.run.smithi046.stdout:mon 3/3 52s ago 8m smithi046:172.21.15.46=a;smithi142:172.21.15.142=b;smithi162:172.21.15.162=c;count:3 2024-03-31T23:11:05.283 INFO:teuthology.orchestra.run.smithi046.stdout:osd 12 52s ago - 2024-03-31T23:11:05.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:05.283+0000 7f1db0ece700 1 -- 172.21.15.46:0/2154424359 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f1d980a2530 msgr2=0x7f1d980a49f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:05.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:05.283+0000 7f1db0ece700 1 --2- 172.21.15.46:0/2154424359 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f1d980a2530 0x7f1d980a49f0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f1d9400b710 tx=0x7f1d94005fd0 comp rx=0 tx=0).stop 2024-03-31T23:11:05.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:05.283+0000 7f1db0ece700 1 -- 172.21.15.46:0/2154424359 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1dac102d50 msgr2=0x7f1dac1983d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:05.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:05.283+0000 7f1db0ece700 1 --2- 172.21.15.46:0/2154424359 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1dac102d50 0x7f1dac1983d0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f1da400ef60 tx=0x7f1da400ef90 comp rx=0 tx=0).stop 2024-03-31T23:11:05.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:05.283+0000 7f1db0ece700 1 -- 172.21.15.46:0/2154424359 shutdown_connections 2024-03-31T23:11:05.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:05.283+0000 7f1db0ece700 1 --2- 172.21.15.46:0/2154424359 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f1d980a2530 0x7f1d980a49f0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:05.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:05.283+0000 7f1db0ece700 1 --2- 172.21.15.46:0/2154424359 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1dac102d50 0x7f1dac1983d0 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:05.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:05.283+0000 7f1db0ece700 1 --2- 172.21.15.46:0/2154424359 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1dac107fb0 0x7f1dac19f990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:05.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:05.283+0000 7f1db0ece700 1 --2- 172.21.15.46:0/2154424359 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1dac105680 0x7f1dac198910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:05.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:05.283+0000 7f1db0ece700 1 -- 172.21.15.46:0/2154424359 >> 172.21.15.46:0/2154424359 conn(0x7f1dac0fc700 msgr2=0x7f1dac103870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:11:05.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:05.283+0000 7f1db0ece700 1 -- 172.21.15.46:0/2154424359 shutdown_connections 2024-03-31T23:11:05.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:05.283+0000 7f1db0ece700 1 -- 172.21.15.46:0/2154424359 wait complete. 2024-03-31T23:11:06.003 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:384f31f3d3aee7f8ba3f60846b8844c049165583 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6a9d8f08-efb2-11ee-b647-cb9ed24678a4 -- bash -c 'ceph orch host ls' 2024-03-31T23:11:06.249 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/6a9d8f08-efb2-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-31T23:11:06.645 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:11:06 smithi046 ceph-mon[32498]: pgmap v351: 129 pgs: 129 active+clean; 642 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-31T23:11:06.646 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:11:06 smithi046 ceph-mon[32498]: from='client.15294 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-31T23:11:06.795 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:11:06 smithi162 ceph-mon[36237]: pgmap v351: 129 pgs: 129 active+clean; 642 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-31T23:11:06.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:11:06 smithi162 ceph-mon[36237]: from='client.15294 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-31T23:11:06.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:11:06 smithi142 ceph-mon[35271]: pgmap v351: 129 pgs: 129 active+clean; 642 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-31T23:11:06.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:11:06 smithi142 ceph-mon[35271]: from='client.15294 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-31T23:11:08.154 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:08.153+0000 7ff149a15700 1 -- 172.21.15.46:0/1399649516 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff144102d70 msgr2=0x7ff144109620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:08.155 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:08.153+0000 7ff149a15700 1 --2- 172.21.15.46:0/1399649516 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff144102d70 0x7ff144109620 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7ff13c0099c0 tx=0x7ff13c009cd0 comp rx=0 tx=0).stop 2024-03-31T23:11:08.155 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:08.153+0000 7ff149a15700 1 -- 172.21.15.46:0/1399649516 shutdown_connections 2024-03-31T23:11:08.155 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:08.153+0000 7ff149a15700 1 --2- 172.21.15.46:0/1399649516 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff144102d70 0x7ff144109620 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:08.155 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:08.153+0000 7ff149a15700 1 --2- 172.21.15.46:0/1399649516 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff1441011d0 0x7ff1441015f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:08.155 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:08.153+0000 7ff149a15700 1 --2- 172.21.15.46:0/1399649516 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff1441023d0 0x7ff144102830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:08.155 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:08.153+0000 7ff149a15700 1 -- 172.21.15.46:0/1399649516 >> 172.21.15.46:0/1399649516 conn(0x7ff1440fc790 msgr2=0x7ff1440febb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:11:08.155 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:08.153+0000 7ff149a15700 1 -- 172.21.15.46:0/1399649516 shutdown_connections 2024-03-31T23:11:08.155 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:08.153+0000 7ff149a15700 1 -- 172.21.15.46:0/1399649516 wait complete. 2024-03-31T23:11:08.155 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:08.153+0000 7ff149a15700 1 Processor -- start 2024-03-31T23:11:08.155 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:08.154+0000 7ff149a15700 1 -- start start 2024-03-31T23:11:08.157 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:08.154+0000 7ff149a15700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff1441011d0 0x7ff1441983a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:08.157 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:08.154+0000 7ff149a15700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff1441023d0 0x7ff1441988e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:08.157 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:08.154+0000 7ff149a15700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff144102d70 0x7ff14419f960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:08.157 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:08.154+0000 7ff149a15700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7ff14410cc00 con 0x7ff1441023d0 2024-03-31T23:11:08.157 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:08.154+0000 7ff149a15700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7ff14410ca80 con 0x7ff1441011d0 2024-03-31T23:11:08.157 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:08.154+0000 7ff149a15700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7ff14410cd80 con 0x7ff144102d70 2024-03-31T23:11:08.157 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:08.154+0000 7ff1427fc700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff1441023d0 0x7ff1441988e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:08.157 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:08.154+0000 7ff1427fc700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff1441023d0 0x7ff1441988e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39052/0 (socket says 172.21.15.46:39052) 2024-03-31T23:11:08.157 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:08.154+0000 7ff1427fc700 1 -- 172.21.15.46:0/4265099398 learned_addr learned my addr 172.21.15.46:0/4265099398 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:11:08.158 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:08.154+0000 7ff142ffd700 1 --2- 172.21.15.46:0/4265099398 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff1441011d0 0x7ff1441983a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:08.158 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:08.154+0000 7ff1437fe700 1 --2- 172.21.15.46:0/4265099398 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff144102d70 0x7ff14419f960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:08.158 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:08.154+0000 7ff1427fc700 1 -- 172.21.15.46:0/4265099398 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff144102d70 msgr2=0x7ff14419f960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:08.158 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:08.154+0000 7ff1427fc700 1 --2- 172.21.15.46:0/4265099398 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff144102d70 0x7ff14419f960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:08.158 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:08.154+0000 7ff1427fc700 1 -- 172.21.15.46:0/4265099398 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff1441011d0 msgr2=0x7ff1441983a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:08.158 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:08.154+0000 7ff1427fc700 1 --2- 172.21.15.46:0/4265099398 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff1441011d0 0x7ff1441983a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:08.158 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:08.154+0000 7ff1427fc700 1 -- 172.21.15.46:0/4265099398 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1441a0000 con 0x7ff1441023d0 2024-03-31T23:11:08.158 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:08.155+0000 7ff1427fc700 1 --2- 172.21.15.46:0/4265099398 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff1441023d0 0x7ff1441988e0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7ff12c00d9b0 tx=0x7ff12c00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:11:08.158 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:08.155+0000 7ff148a13700 1 -- 172.21.15.46:0/4265099398 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff12c016070 con 0x7ff1441023d0 2024-03-31T23:11:08.158 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:08.155+0000 7ff148a13700 1 -- 172.21.15.46:0/4265099398 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff12c004830 con 0x7ff1441023d0 2024-03-31T23:11:08.158 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:08.155+0000 7ff149a15700 1 -- 172.21.15.46:0/4265099398 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff1441a02f0 con 0x7ff1441023d0 2024-03-31T23:11:08.158 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:08.155+0000 7ff148a13700 1 -- 172.21.15.46:0/4265099398 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff12c005460 con 0x7ff1441023d0 2024-03-31T23:11:08.158 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:08.155+0000 7ff149a15700 1 -- 172.21.15.46:0/4265099398 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff1441a0830 con 0x7ff1441023d0 2024-03-31T23:11:08.159 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:08.156+0000 7ff149a15700 1 -- 172.21.15.46:0/4265099398 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff144068f00 con 0x7ff1441023d0 2024-03-31T23:11:08.159 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:08.157+0000 7ff148a13700 1 -- 172.21.15.46:0/4265099398 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 18) ==== 137320+0+0 (secure 0 0 0) 0x7ff12c005600 con 0x7ff1441023d0 2024-03-31T23:11:08.159 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:08.157+0000 7ff148a13700 1 --2- 172.21.15.46:0/4265099398 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7ff1300a2530 0x7ff1300a49f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:08.159 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:08.157+0000 7ff142ffd700 1 --2- 172.21.15.46:0/4265099398 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7ff1300a2530 0x7ff1300a49f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:08.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:08.158+0000 7ff148a13700 1 -- 172.21.15.46:0/4265099398 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7ff12c0cecf0 con 0x7ff1441023d0 2024-03-31T23:11:08.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:08.158+0000 7ff142ffd700 1 --2- 172.21.15.46:0/4265099398 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7ff1300a2530 0x7ff1300a49f0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7ff144102230 tx=0x7ff13400b040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:11:08.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:08.161+0000 7ff148a13700 1 -- 172.21.15.46:0/4265099398 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7ff12c01f020 con 0x7ff1441023d0 2024-03-31T23:11:08.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:08.369+0000 7ff149a15700 1 -- 172.21.15.46:0/4265099398 --> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""]}) -- 0x7ff1441a0b50 con 0x7ff1300a2530 2024-03-31T23:11:08.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:08.371+0000 7ff148a13700 1 -- 172.21.15.46:0/4265099398 <== mgr.14152 v2:172.21.15.46:6800/162289569 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+191 (secure 0 0 0) 0x7ff1441a0b50 con 0x7ff1300a2530 2024-03-31T23:11:08.372 INFO:teuthology.orchestra.run.smithi046.stdout:HOST ADDR LABELS STATUS 2024-03-31T23:11:08.372 INFO:teuthology.orchestra.run.smithi046.stdout:smithi046 172.21.15.46 2024-03-31T23:11:08.372 INFO:teuthology.orchestra.run.smithi046.stdout:smithi142 172.21.15.142 2024-03-31T23:11:08.372 INFO:teuthology.orchestra.run.smithi046.stdout:smithi162 172.21.15.162 2024-03-31T23:11:08.372 INFO:teuthology.orchestra.run.smithi046.stdout:3 hosts in cluster 2024-03-31T23:11:08.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:08.373+0000 7ff149a15700 1 -- 172.21.15.46:0/4265099398 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7ff1300a2530 msgr2=0x7ff1300a49f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:08.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:08.373+0000 7ff149a15700 1 --2- 172.21.15.46:0/4265099398 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7ff1300a2530 0x7ff1300a49f0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7ff144102230 tx=0x7ff13400b040 comp rx=0 tx=0).stop 2024-03-31T23:11:08.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:08.373+0000 7ff149a15700 1 -- 172.21.15.46:0/4265099398 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff1441023d0 msgr2=0x7ff1441988e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:08.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:08.373+0000 7ff149a15700 1 --2- 172.21.15.46:0/4265099398 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff1441023d0 0x7ff1441988e0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7ff12c00d9b0 tx=0x7ff12c00dcc0 comp rx=0 tx=0).stop 2024-03-31T23:11:08.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:08.373+0000 7ff149a15700 1 -- 172.21.15.46:0/4265099398 shutdown_connections 2024-03-31T23:11:08.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:08.373+0000 7ff149a15700 1 --2- 172.21.15.46:0/4265099398 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7ff1300a2530 0x7ff1300a49f0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:08.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:08.373+0000 7ff149a15700 1 --2- 172.21.15.46:0/4265099398 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff144102d70 0x7ff14419f960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:08.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:08.373+0000 7ff149a15700 1 --2- 172.21.15.46:0/4265099398 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff1441011d0 0x7ff1441983a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:08.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:08.373+0000 7ff149a15700 1 --2- 172.21.15.46:0/4265099398 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff1441023d0 0x7ff1441988e0 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:08.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:08.373+0000 7ff149a15700 1 -- 172.21.15.46:0/4265099398 >> 172.21.15.46:0/4265099398 conn(0x7ff1440fc790 msgr2=0x7ff1440fe250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:11:08.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:08.373+0000 7ff149a15700 1 -- 172.21.15.46:0/4265099398 shutdown_connections 2024-03-31T23:11:08.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:08.373+0000 7ff149a15700 1 -- 172.21.15.46:0/4265099398 wait complete. 2024-03-31T23:11:08.670 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:11:08 smithi046 ceph-mon[32498]: pgmap v352: 129 pgs: 129 active+clean; 642 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-03-31T23:11:08.795 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:11:08 smithi162 ceph-mon[36237]: pgmap v352: 129 pgs: 129 active+clean; 642 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-03-31T23:11:08.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:11:08 smithi142 ceph-mon[35271]: pgmap v352: 129 pgs: 129 active+clean; 642 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-03-31T23:11:09.180 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:384f31f3d3aee7f8ba3f60846b8844c049165583 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6a9d8f08-efb2-11ee-b647-cb9ed24678a4 -- bash -c 'ceph orch device ls' 2024-03-31T23:11:09.426 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/6a9d8f08-efb2-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-31T23:11:09.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:11:09 smithi162 ceph-mon[36237]: from='client.15300 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-31T23:11:09.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:11:09 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [2, 11]}]: dispatch 2024-03-31T23:11:09.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:11:09 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [8, 11]}]: dispatch 2024-03-31T23:11:09.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:11:09 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [5, 8]}]: dispatch 2024-03-31T23:11:09.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:11:09 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [9, 10]}]: dispatch 2024-03-31T23:11:09.798 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:11:09 smithi046 ceph-mon[32498]: from='client.15300 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-31T23:11:09.798 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:11:09 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [2, 11]}]: dispatch 2024-03-31T23:11:09.798 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:11:09 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [8, 11]}]: dispatch 2024-03-31T23:11:09.798 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:11:09 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [5, 8]}]: dispatch 2024-03-31T23:11:09.798 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:11:09 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [9, 10]}]: dispatch 2024-03-31T23:11:09.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:11:09 smithi142 ceph-mon[35271]: from='client.15300 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-31T23:11:09.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:11:09 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [2, 11]}]: dispatch 2024-03-31T23:11:09.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:11:09 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [8, 11]}]: dispatch 2024-03-31T23:11:09.944 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:11:09 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [5, 8]}]: dispatch 2024-03-31T23:11:09.944 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:11:09 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [9, 10]}]: dispatch 2024-03-31T23:11:10.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:11:10 smithi162 ceph-mon[36237]: pgmap v353: 129 pgs: 129 active+clean; 642 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-31T23:11:10.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:11:10 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [2, 11]}]': finished 2024-03-31T23:11:10.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:11:10 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [8, 11]}]': finished 2024-03-31T23:11:10.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:11:10 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [5, 8]}]': finished 2024-03-31T23:11:10.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:11:10 smithi162 ceph-mon[36237]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [9, 10]}]': finished 2024-03-31T23:11:10.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:11:10 smithi162 ceph-mon[36237]: osdmap e94: 12 total, 12 up, 12 in 2024-03-31T23:11:10.826 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:11:10 smithi046 ceph-mon[32498]: pgmap v353: 129 pgs: 129 active+clean; 642 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-31T23:11:10.826 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:11:10 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [2, 11]}]': finished 2024-03-31T23:11:10.826 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:11:10 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [8, 11]}]': finished 2024-03-31T23:11:10.826 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:11:10 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [5, 8]}]': finished 2024-03-31T23:11:10.826 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:11:10 smithi046 ceph-mon[32498]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [9, 10]}]': finished 2024-03-31T23:11:10.827 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:11:10 smithi046 ceph-mon[32498]: osdmap e94: 12 total, 12 up, 12 in 2024-03-31T23:11:10.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:11:10 smithi142 ceph-mon[35271]: pgmap v353: 129 pgs: 129 active+clean; 642 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-31T23:11:10.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:11:10 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [2, 11]}]': finished 2024-03-31T23:11:10.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:11:10 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [8, 11]}]': finished 2024-03-31T23:11:10.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:11:10 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [5, 8]}]': finished 2024-03-31T23:11:10.944 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:11:10 smithi142 ceph-mon[35271]: from='mgr.14152 172.21.15.46:0/398010483' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [9, 10]}]': finished 2024-03-31T23:11:10.944 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:11:10 smithi142 ceph-mon[35271]: osdmap e94: 12 total, 12 up, 12 in 2024-03-31T23:11:11.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:11.200+0000 7f2f7604d700 1 -- 172.21.15.46:0/1487874907 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2f70102d10 msgr2=0x7f2f701095c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:11.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:11.200+0000 7f2f7604d700 1 --2- 172.21.15.46:0/1487874907 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2f70102d10 0x7f2f701095c0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f2f6800b0d0 tx=0x7f2f6800b3e0 comp rx=0 tx=0).stop 2024-03-31T23:11:11.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:11.200+0000 7f2f7604d700 1 -- 172.21.15.46:0/1487874907 shutdown_connections 2024-03-31T23:11:11.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:11.200+0000 7f2f7604d700 1 --2- 172.21.15.46:0/1487874907 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2f70102d10 0x7f2f701095c0 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:11.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:11.200+0000 7f2f7604d700 1 --2- 172.21.15.46:0/1487874907 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2f70101170 0x7f2f70101590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:11.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:11.200+0000 7f2f7604d700 1 --2- 172.21.15.46:0/1487874907 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2f70102370 0x7f2f701027d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:11.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:11.200+0000 7f2f7604d700 1 -- 172.21.15.46:0/1487874907 >> 172.21.15.46:0/1487874907 conn(0x7f2f700fc710 msgr2=0x7f2f700feb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:11:11.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:11.200+0000 7f2f7604d700 1 -- 172.21.15.46:0/1487874907 shutdown_connections 2024-03-31T23:11:11.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:11.200+0000 7f2f7604d700 1 -- 172.21.15.46:0/1487874907 wait complete. 2024-03-31T23:11:11.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:11.200+0000 7f2f7604d700 1 Processor -- start 2024-03-31T23:11:11.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:11.201+0000 7f2f7604d700 1 -- start start 2024-03-31T23:11:11.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:11.201+0000 7f2f7604d700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2f70101170 0x7f2f70198400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:11.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:11.201+0000 7f2f7604d700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2f70102370 0x7f2f70198940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:11.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:11.201+0000 7f2f7604d700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2f70102d10 0x7f2f7019f950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:11.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:11.201+0000 7f2f7604d700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f2f7010cbf0 con 0x7f2f70102370 2024-03-31T23:11:11.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:11.201+0000 7f2f7604d700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f2f7010ca70 con 0x7f2f70102d10 2024-03-31T23:11:11.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:11.201+0000 7f2f7604d700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f2f7010cd70 con 0x7f2f70101170 2024-03-31T23:11:11.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:11.201+0000 7f2f6effd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2f70102370 0x7f2f70198940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:11.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:11.201+0000 7f2f6effd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2f70102370 0x7f2f70198940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39066/0 (socket says 172.21.15.46:39066) 2024-03-31T23:11:11.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:11.201+0000 7f2f6effd700 1 -- 172.21.15.46:0/3480372583 learned_addr learned my addr 172.21.15.46:0/3480372583 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:11:11.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:11.201+0000 7f2f6ffff700 1 --2- 172.21.15.46:0/3480372583 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2f70102d10 0x7f2f7019f950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:11.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:11.201+0000 7f2f6f7fe700 1 --2- 172.21.15.46:0/3480372583 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2f70101170 0x7f2f70198400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:11.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:11.201+0000 7f2f6effd700 1 -- 172.21.15.46:0/3480372583 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2f70101170 msgr2=0x7f2f70198400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:11.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:11.201+0000 7f2f6effd700 1 --2- 172.21.15.46:0/3480372583 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2f70101170 0x7f2f70198400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:11.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:11.201+0000 7f2f6effd700 1 -- 172.21.15.46:0/3480372583 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2f70102d10 msgr2=0x7f2f7019f950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:11.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:11.201+0000 7f2f6effd700 1 --2- 172.21.15.46:0/3480372583 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2f70102d10 0x7f2f7019f950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:11.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:11.201+0000 7f2f6effd700 1 -- 172.21.15.46:0/3480372583 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2f7019fff0 con 0x7f2f70102370 2024-03-31T23:11:11.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:11.201+0000 7f2f6effd700 1 --2- 172.21.15.46:0/3480372583 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2f70102370 0x7f2f70198940 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f2f6400e9d0 tx=0x7f2f6400ece0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:11:11.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:11.202+0000 7f2f6cff9700 1 -- 172.21.15.46:0/3480372583 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f6400cbd0 con 0x7f2f70102370 2024-03-31T23:11:11.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:11.202+0000 7f2f6cff9700 1 -- 172.21.15.46:0/3480372583 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f2f6400ee70 con 0x7f2f70102370 2024-03-31T23:11:11.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:11.202+0000 7f2f7604d700 1 -- 172.21.15.46:0/3480372583 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2f701a02e0 con 0x7f2f70102370 2024-03-31T23:11:11.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:11.202+0000 7f2f6cff9700 1 -- 172.21.15.46:0/3480372583 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f64018680 con 0x7f2f70102370 2024-03-31T23:11:11.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:11.202+0000 7f2f7604d700 1 -- 172.21.15.46:0/3480372583 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2f701a0820 con 0x7f2f70102370 2024-03-31T23:11:11.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:11.203+0000 7f2f7604d700 1 -- 172.21.15.46:0/3480372583 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2f70068f00 con 0x7f2f70102370 2024-03-31T23:11:11.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:11.204+0000 7f2f6cff9700 1 -- 172.21.15.46:0/3480372583 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 18) ==== 137320+0+0 (secure 0 0 0) 0x7f2f64010430 con 0x7f2f70102370 2024-03-31T23:11:11.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:11.204+0000 7f2f6cff9700 1 --2- 172.21.15.46:0/3480372583 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f2f580a2530 0x7f2f580a49f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:11.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:11.204+0000 7f2f6f7fe700 1 --2- 172.21.15.46:0/3480372583 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f2f580a2530 0x7f2f580a49f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:11.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:11.205+0000 7f2f6cff9700 1 -- 172.21.15.46:0/3480372583 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9208+0+0 (secure 0 0 0) 0x7f2f640ced80 con 0x7f2f70102370 2024-03-31T23:11:11.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:11.205+0000 7f2f6f7fe700 1 --2- 172.21.15.46:0/3480372583 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f2f580a2530 0x7f2f580a49f0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f2f701021d0 tx=0x7f2f60009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:11:11.210 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:11.209+0000 7f2f6cff9700 1 -- 172.21.15.46:0/3480372583 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f2f6400cd70 con 0x7f2f70102370 2024-03-31T23:11:11.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:11.421+0000 7f2f7604d700 1 -- 172.21.15.46:0/3480372583 --> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] -- mgr_command(tid 0: {"prefix": "orch device ls", "target": ["mon-mgr", ""]}) -- 0x7f2f700fe8f0 con 0x7f2f580a2530 2024-03-31T23:11:11.424 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:11.423+0000 7f2f6cff9700 1 -- 172.21.15.46:0/3480372583 <== mgr.14152 v2:172.21.15.46:6800/162289569 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+532 (secure 0 0 0) 0x7f2f700fe8f0 con 0x7f2f580a2530 2024-03-31T23:11:11.424 INFO:teuthology.orchestra.run.smithi046.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2024-03-31T23:11:11.424 INFO:teuthology.orchestra.run.smithi046.stdout:smithi046 /dev/nvme0n1 ssd INTEL_SSDPED1D480GA_PHMB7512009A480DGN 447G No 3m ago Has a FileSystem, LVM detected 2024-03-31T23:11:11.424 INFO:teuthology.orchestra.run.smithi046.stdout:smithi142 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_CVFT6303009Y400BGN 372G No 3m ago Has a FileSystem, LVM detected 2024-03-31T23:11:11.424 INFO:teuthology.orchestra.run.smithi046.stdout:smithi162 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_CVFT623300JU400BGN 372G No 2m ago Has a FileSystem, LVM detected 2024-03-31T23:11:11.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:11.425+0000 7f2f7604d700 1 -- 172.21.15.46:0/3480372583 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f2f580a2530 msgr2=0x7f2f580a49f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:11.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:11.425+0000 7f2f7604d700 1 --2- 172.21.15.46:0/3480372583 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f2f580a2530 0x7f2f580a49f0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f2f701021d0 tx=0x7f2f60009260 comp rx=0 tx=0).stop 2024-03-31T23:11:11.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:11.425+0000 7f2f7604d700 1 -- 172.21.15.46:0/3480372583 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2f70102370 msgr2=0x7f2f70198940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:11.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:11.425+0000 7f2f7604d700 1 --2- 172.21.15.46:0/3480372583 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2f70102370 0x7f2f70198940 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f2f6400e9d0 tx=0x7f2f6400ece0 comp rx=0 tx=0).stop 2024-03-31T23:11:11.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:11.425+0000 7f2f7604d700 1 -- 172.21.15.46:0/3480372583 shutdown_connections 2024-03-31T23:11:11.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:11.425+0000 7f2f7604d700 1 --2- 172.21.15.46:0/3480372583 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f2f580a2530 0x7f2f580a49f0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:11.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:11.425+0000 7f2f7604d700 1 --2- 172.21.15.46:0/3480372583 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2f70102d10 0x7f2f7019f950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:11.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:11.425+0000 7f2f7604d700 1 --2- 172.21.15.46:0/3480372583 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2f70101170 0x7f2f70198400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:11.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:11.425+0000 7f2f7604d700 1 --2- 172.21.15.46:0/3480372583 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2f70102370 0x7f2f70198940 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:11.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:11.425+0000 7f2f7604d700 1 -- 172.21.15.46:0/3480372583 >> 172.21.15.46:0/3480372583 conn(0x7f2f700fc710 msgr2=0x7f2f700fe1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:11:11.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:11.425+0000 7f2f7604d700 1 -- 172.21.15.46:0/3480372583 shutdown_connections 2024-03-31T23:11:11.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:11.425+0000 7f2f7604d700 1 -- 172.21.15.46:0/3480372583 wait complete. 2024-03-31T23:11:11.795 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:11:11 smithi162 ceph-mon[36237]: osdmap e95: 12 total, 12 up, 12 in 2024-03-31T23:11:11.851 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:11:11 smithi046 ceph-mon[32498]: osdmap e95: 12 total, 12 up, 12 in 2024-03-31T23:11:11.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:11:11 smithi142 ceph-mon[35271]: osdmap e95: 12 total, 12 up, 12 in 2024-03-31T23:11:12.238 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-03-31T23:11:12.249 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi046.front.sepia.ceph.com 2024-03-31T23:11:12.249 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:384f31f3d3aee7f8ba3f60846b8844c049165583 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6a9d8f08-efb2-11ee-b647-cb9ed24678a4 -- bash -c 'ceph fs dump' 2024-03-31T23:11:12.502 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/6a9d8f08-efb2-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-31T23:11:12.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:11:12 smithi162 ceph-mon[36237]: pgmap v356: 129 pgs: 2 peering, 127 active+clean; 642 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 6 op/s 2024-03-31T23:11:12.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:11:12 smithi162 ceph-mon[36237]: from='client.15306 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-31T23:11:12.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:11:12 smithi162 ceph-mon[36237]: Health check failed: Reduced data availability: 2 pgs peering (PG_AVAILABILITY) 2024-03-31T23:11:12.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:11:12 smithi046 ceph-mon[32498]: pgmap v356: 129 pgs: 2 peering, 127 active+clean; 642 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 6 op/s 2024-03-31T23:11:12.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:11:12 smithi046 ceph-mon[32498]: from='client.15306 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-31T23:11:12.862 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:11:12 smithi046 ceph-mon[32498]: Health check failed: Reduced data availability: 2 pgs peering (PG_AVAILABILITY) 2024-03-31T23:11:12.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:11:12 smithi142 ceph-mon[35271]: pgmap v356: 129 pgs: 2 peering, 127 active+clean; 642 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 6 op/s 2024-03-31T23:11:12.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:11:12 smithi142 ceph-mon[35271]: from='client.15306 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-31T23:11:12.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:11:12 smithi142 ceph-mon[35271]: Health check failed: Reduced data availability: 2 pgs peering (PG_AVAILABILITY) 2024-03-31T23:11:14.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:14.230+0000 7f176ab77700 1 -- 172.21.15.46:0/3799638040 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1764102370 msgr2=0x7f17641027d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:14.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:14.230+0000 7f176ab77700 1 --2- 172.21.15.46:0/3799638040 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1764102370 0x7f17641027d0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f17500099c0 tx=0x7f1750009cd0 comp rx=0 tx=0).stop 2024-03-31T23:11:14.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:14.230+0000 7f176ab77700 1 -- 172.21.15.46:0/3799638040 shutdown_connections 2024-03-31T23:11:14.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:14.230+0000 7f176ab77700 1 --2- 172.21.15.46:0/3799638040 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1764102d10 0x7f17641095c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:14.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:14.230+0000 7f176ab77700 1 --2- 172.21.15.46:0/3799638040 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1764101170 0x7f1764101590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:14.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:14.230+0000 7f176ab77700 1 --2- 172.21.15.46:0/3799638040 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1764102370 0x7f17641027d0 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:14.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:14.230+0000 7f176ab77700 1 -- 172.21.15.46:0/3799638040 >> 172.21.15.46:0/3799638040 conn(0x7f17640fc730 msgr2=0x7f17640feb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:11:14.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:14.230+0000 7f176ab77700 1 -- 172.21.15.46:0/3799638040 shutdown_connections 2024-03-31T23:11:14.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:14.230+0000 7f176ab77700 1 -- 172.21.15.46:0/3799638040 wait complete. 2024-03-31T23:11:14.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:14.230+0000 7f176ab77700 1 Processor -- start 2024-03-31T23:11:14.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:14.230+0000 7f176ab77700 1 -- start start 2024-03-31T23:11:14.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:14.231+0000 7f176ab77700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1764101170 0x7f176406b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:14.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:14.231+0000 7f176ab77700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1764102370 0x7f176419d140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:14.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:14.231+0000 7f176ab77700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1764102d10 0x7f176419f790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:14.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:14.231+0000 7f176ab77700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f176410cc30 con 0x7f1764102d10 2024-03-31T23:11:14.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:14.231+0000 7f176ab77700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f176410cab0 con 0x7f1764101170 2024-03-31T23:11:14.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:14.231+0000 7f176ab77700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f176410cdb0 con 0x7f1764102370 2024-03-31T23:11:14.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:14.231+0000 7f1769114700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1764102d10 0x7f176419f790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:14.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:14.231+0000 7f1763fff700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1764102370 0x7f176419d140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:14.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:14.231+0000 7f1769114700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1764102d10 0x7f176419f790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:54730/0 (socket says 172.21.15.46:54730) 2024-03-31T23:11:14.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:14.231+0000 7f1769114700 1 -- 172.21.15.46:0/2445313044 learned_addr learned my addr 172.21.15.46:0/2445313044 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:11:14.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:14.231+0000 7f1768913700 1 --2- 172.21.15.46:0/2445313044 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1764101170 0x7f176406b4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:14.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:14.231+0000 7f1763fff700 1 -- 172.21.15.46:0/2445313044 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1764101170 msgr2=0x7f176406b4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:14.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:14.231+0000 7f1763fff700 1 --2- 172.21.15.46:0/2445313044 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1764101170 0x7f176406b4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:14.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:14.231+0000 7f1763fff700 1 -- 172.21.15.46:0/2445313044 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1764102d10 msgr2=0x7f176419f790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:14.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:14.231+0000 7f1763fff700 1 --2- 172.21.15.46:0/2445313044 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1764102d10 0x7f176419f790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:14.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:14.231+0000 7f1763fff700 1 -- 172.21.15.46:0/2445313044 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17641a0060 con 0x7f1764102370 2024-03-31T23:11:14.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:14.231+0000 7f1763fff700 1 --2- 172.21.15.46:0/2445313044 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1764102370 0x7f176419d140 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f1750009990 tx=0x7f175000f690 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:11:14.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:14.232+0000 7f1761ffb700 1 -- 172.21.15.46:0/2445313044 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1750009e60 con 0x7f1764102370 2024-03-31T23:11:14.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:14.232+0000 7f176ab77700 1 -- 172.21.15.46:0/2445313044 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f17641a02f0 con 0x7f1764102370 2024-03-31T23:11:14.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:14.232+0000 7f1761ffb700 1 -- 172.21.15.46:0/2445313044 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f17500038c0 con 0x7f1764102370 2024-03-31T23:11:14.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:14.232+0000 7f1761ffb700 1 -- 172.21.15.46:0/2445313044 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17500055d0 con 0x7f1764102370 2024-03-31T23:11:14.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:14.232+0000 7f176ab77700 1 -- 172.21.15.46:0/2445313044 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f17641a07a0 con 0x7f1764102370 2024-03-31T23:11:14.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:14.233+0000 7f176ab77700 1 -- 172.21.15.46:0/2445313044 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1764068f00 con 0x7f1764102370 2024-03-31T23:11:14.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:14.234+0000 7f1761ffb700 1 -- 172.21.15.46:0/2445313044 <== mon.1 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 18) ==== 137320+0+0 (secure 0 0 0) 0x7f1750003a70 con 0x7f1764102370 2024-03-31T23:11:14.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:14.234+0000 7f1761ffb700 1 --2- 172.21.15.46:0/2445313044 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f17540a2530 0x7f17540a49f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:14.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:14.235+0000 7f1768913700 1 --2- 172.21.15.46:0/2445313044 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f17540a2530 0x7f17540a49f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:14.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:14.235+0000 7f1761ffb700 1 -- 172.21.15.46:0/2445313044 <== mon.1 v2:172.21.15.162:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9208+0+0 (secure 0 0 0) 0x7f17500cf440 con 0x7f1764102370 2024-03-31T23:11:14.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:14.235+0000 7f1768913700 1 --2- 172.21.15.46:0/2445313044 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f17540a2530 0x7f17540a49f0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f17641021d0 tx=0x7f1758009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:11:14.243 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:14.242+0000 7f1761ffb700 1 -- 172.21.15.46:0/2445313044 <== mon.1 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f1750013070 con 0x7f1764102370 2024-03-31T23:11:14.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:14.491+0000 7f176ab77700 1 -- 172.21.15.46:0/2445313044 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "fs dump"} v 0) -- 0x7f1764062e50 con 0x7f1764102370 2024-03-31T23:11:14.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:14.491+0000 7f1761ffb700 1 -- 172.21.15.46:0/2445313044 <== mon.1 v2:172.21.15.162:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 24 v24) ==== 76+0+3058 (secure 0 0 0) 0x7f175009c260 con 0x7f1764102370 2024-03-31T23:11:14.492 INFO:teuthology.orchestra.run.smithi046.stdout:e24 2024-03-31T23:11:14.493 INFO:teuthology.orchestra.run.smithi046.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-31T23:11:14.493 INFO:teuthology.orchestra.run.smithi046.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-03-31T23:11:14.493 INFO:teuthology.orchestra.run.smithi046.stdout:legacy client fscid: 1 2024-03-31T23:11:14.493 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-31T23:11:14.493 INFO:teuthology.orchestra.run.smithi046.stdout:Filesystem 'cephfs' (1) 2024-03-31T23:11:14.493 INFO:teuthology.orchestra.run.smithi046.stdout:fs_name cephfs 2024-03-31T23:11:14.493 INFO:teuthology.orchestra.run.smithi046.stdout:epoch 24 2024-03-31T23:11:14.493 INFO:teuthology.orchestra.run.smithi046.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-03-31T23:11:14.493 INFO:teuthology.orchestra.run.smithi046.stdout:created 2024-03-31T23:09:11.009090+0000 2024-03-31T23:11:14.493 INFO:teuthology.orchestra.run.smithi046.stdout:modified 2024-03-31T23:09:31.392282+0000 2024-03-31T23:11:14.493 INFO:teuthology.orchestra.run.smithi046.stdout:tableserver 0 2024-03-31T23:11:14.493 INFO:teuthology.orchestra.run.smithi046.stdout:root 0 2024-03-31T23:11:14.493 INFO:teuthology.orchestra.run.smithi046.stdout:session_timeout 300 2024-03-31T23:11:14.493 INFO:teuthology.orchestra.run.smithi046.stdout:session_autoclose 300 2024-03-31T23:11:14.493 INFO:teuthology.orchestra.run.smithi046.stdout:max_file_size 1099511627776 2024-03-31T23:11:14.494 INFO:teuthology.orchestra.run.smithi046.stdout:max_xattr_size 65536 2024-03-31T23:11:14.494 INFO:teuthology.orchestra.run.smithi046.stdout:required_client_features {} 2024-03-31T23:11:14.494 INFO:teuthology.orchestra.run.smithi046.stdout:last_failure 0 2024-03-31T23:11:14.494 INFO:teuthology.orchestra.run.smithi046.stdout:last_failure_osd_epoch 0 2024-03-31T23:11:14.494 INFO:teuthology.orchestra.run.smithi046.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-03-31T23:11:14.494 INFO:teuthology.orchestra.run.smithi046.stdout:max_mds 3 2024-03-31T23:11:14.494 INFO:teuthology.orchestra.run.smithi046.stdout:in 0,1,2 2024-03-31T23:11:14.494 INFO:teuthology.orchestra.run.smithi046.stdout:up {0=24515,1=24482,2=24451} 2024-03-31T23:11:14.494 INFO:teuthology.orchestra.run.smithi046.stdout:failed 2024-03-31T23:11:14.494 INFO:teuthology.orchestra.run.smithi046.stdout:damaged 2024-03-31T23:11:14.494 INFO:teuthology.orchestra.run.smithi046.stdout:stopped 2024-03-31T23:11:14.494 INFO:teuthology.orchestra.run.smithi046.stdout:data_pools [3,4] 2024-03-31T23:11:14.495 INFO:teuthology.orchestra.run.smithi046.stdout:metadata_pool 2 2024-03-31T23:11:14.495 INFO:teuthology.orchestra.run.smithi046.stdout:inline_data disabled 2024-03-31T23:11:14.495 INFO:teuthology.orchestra.run.smithi046.stdout:balancer 2024-03-31T23:11:14.495 INFO:teuthology.orchestra.run.smithi046.stdout:bal_rank_mask -1 2024-03-31T23:11:14.495 INFO:teuthology.orchestra.run.smithi046.stdout:standby_count_wanted 0 2024-03-31T23:11:14.495 INFO:teuthology.orchestra.run.smithi046.stdout:qdb_cluster leader: 24515 members: 24451,24482,24515 2024-03-31T23:11:14.495 INFO:teuthology.orchestra.run.smithi046.stdout:[mds.b{0:24515} state up:active seq 3 addr [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] compat {c=[1],r=[1],i=[1fff]}] 2024-03-31T23:11:14.495 INFO:teuthology.orchestra.run.smithi046.stdout:[mds.i{0:24478} state up:standby-replay seq 1 addr [v2:172.21.15.162:6838/1976275826,v1:172.21.15.162:6839/1976275826] compat {c=[1],r=[1],i=[1fff]}] 2024-03-31T23:11:14.495 INFO:teuthology.orchestra.run.smithi046.stdout:[mds.h{1:24482} state up:active seq 10 addr [v2:172.21.15.142:6836/2559258399,v1:172.21.15.142:6837/2559258399] compat {c=[1],r=[1],i=[1fff]}] 2024-03-31T23:11:14.495 INFO:teuthology.orchestra.run.smithi046.stdout:[mds.e{1:24455} state up:standby-replay seq 1 addr [v2:172.21.15.142:6834/212637431,v1:172.21.15.142:6835/212637431] compat {c=[1],r=[1],i=[1fff]}] 2024-03-31T23:11:14.495 INFO:teuthology.orchestra.run.smithi046.stdout:[mds.c{2:24451} state up:active seq 11 addr [v2:172.21.15.162:6836/1371886876,v1:172.21.15.162:6837/1371886876] compat {c=[1],r=[1],i=[1fff]}] 2024-03-31T23:11:14.495 INFO:teuthology.orchestra.run.smithi046.stdout:[mds.f{2:24442} state up:standby-replay seq 1 addr [v2:172.21.15.162:6834/1957208984,v1:172.21.15.162:6835/1957208984] compat {c=[1],r=[1],i=[1fff]}] 2024-03-31T23:11:14.495 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-31T23:11:14.495 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-31T23:11:14.495 INFO:teuthology.orchestra.run.smithi046.stdout:Standby daemons: 2024-03-31T23:11:14.496 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-31T23:11:14.496 INFO:teuthology.orchestra.run.smithi046.stdout:[mds.j{-1:14556} state up:standby seq 1 addr [v2:172.21.15.46:6834/2169018547,v1:172.21.15.46:6835/2169018547] compat {c=[1],r=[1],i=[1fff]}] 2024-03-31T23:11:14.496 INFO:teuthology.orchestra.run.smithi046.stdout:[mds.d{-1:14568} state up:standby seq 1 addr [v2:172.21.15.46:6836/1069387756,v1:172.21.15.46:6837/1069387756] compat {c=[1],r=[1],i=[1fff]}] 2024-03-31T23:11:14.496 INFO:teuthology.orchestra.run.smithi046.stdout:[mds.g{-1:14634} state up:standby seq 1 addr [v2:172.21.15.46:6838/3915337844,v1:172.21.15.46:6839/3915337844] compat {c=[1],r=[1],i=[1fff]}] 2024-03-31T23:11:14.496 INFO:teuthology.orchestra.run.smithi046.stdout:[mds.a{-1:14646} state up:standby seq 1 addr [v2:172.21.15.46:6840/897767145,v1:172.21.15.46:6841/897767145] compat {c=[1],r=[1],i=[1fff]}] 2024-03-31T23:11:14.496 INFO:teuthology.orchestra.run.smithi046.stdout:[mds.l{-1:24415} state up:standby seq 1 addr [v2:172.21.15.162:6832/1746950966,v1:172.21.15.162:6833/1746950966] compat {c=[1],r=[1],i=[1fff]}] 2024-03-31T23:11:14.496 INFO:teuthology.orchestra.run.smithi046.stdout:[mds.k{-1:24422} state up:standby seq 1 addr [v2:172.21.15.142:6832/3394739181,v1:172.21.15.142:6833/3394739181] compat {c=[1],r=[1],i=[1fff]}] 2024-03-31T23:11:14.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:14.493+0000 7f176ab77700 1 -- 172.21.15.46:0/2445313044 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f17540a2530 msgr2=0x7f17540a49f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:14.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:14.493+0000 7f176ab77700 1 --2- 172.21.15.46:0/2445313044 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f17540a2530 0x7f17540a49f0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f17641021d0 tx=0x7f1758009260 comp rx=0 tx=0).stop 2024-03-31T23:11:14.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:14.493+0000 7f176ab77700 1 -- 172.21.15.46:0/2445313044 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1764102370 msgr2=0x7f176419d140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:14.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:14.493+0000 7f176ab77700 1 --2- 172.21.15.46:0/2445313044 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1764102370 0x7f176419d140 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f1750009990 tx=0x7f175000f690 comp rx=0 tx=0).stop 2024-03-31T23:11:14.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:14.493+0000 7f176ab77700 1 -- 172.21.15.46:0/2445313044 shutdown_connections 2024-03-31T23:11:14.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:14.493+0000 7f176ab77700 1 --2- 172.21.15.46:0/2445313044 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f17540a2530 0x7f17540a49f0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:14.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:14.493+0000 7f176ab77700 1 --2- 172.21.15.46:0/2445313044 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1764102d10 0x7f176419f790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:14.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:14.493+0000 7f176ab77700 1 --2- 172.21.15.46:0/2445313044 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1764101170 0x7f176406b4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:14.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:14.493+0000 7f176ab77700 1 --2- 172.21.15.46:0/2445313044 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1764102370 0x7f176419d140 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:14.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:14.493+0000 7f176ab77700 1 -- 172.21.15.46:0/2445313044 >> 172.21.15.46:0/2445313044 conn(0x7f17640fc730 msgr2=0x7f1764105690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:11:14.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:14.494+0000 7f176ab77700 1 -- 172.21.15.46:0/2445313044 shutdown_connections 2024-03-31T23:11:14.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:14.494+0000 7f176ab77700 1 -- 172.21.15.46:0/2445313044 wait complete. 2024-03-31T23:11:14.498 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 24 2024-03-31T23:11:14.855 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:11:14 smithi046 ceph-mon[32498]: pgmap v357: 129 pgs: 2 peering, 127 active+clean; 642 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 6 op/s 2024-03-31T23:11:14.855 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:11:14 smithi046 ceph-mon[32498]: from='client.? 172.21.15.46:0/2445313044' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-31T23:11:14.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:11:14 smithi142 ceph-mon[35271]: pgmap v357: 129 pgs: 2 peering, 127 active+clean; 642 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 6 op/s 2024-03-31T23:11:14.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:11:14 smithi142 ceph-mon[35271]: from='client.? 172.21.15.46:0/2445313044' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-31T23:11:15.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:11:14 smithi162 ceph-mon[36237]: pgmap v357: 129 pgs: 2 peering, 127 active+clean; 642 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 6 op/s 2024-03-31T23:11:15.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:11:14 smithi162 ceph-mon[36237]: from='client.? 172.21.15.46:0/2445313044' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-31T23:11:15.185 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:384f31f3d3aee7f8ba3f60846b8844c049165583 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6a9d8f08-efb2-11ee-b647-cb9ed24678a4 -- bash -c 'ceph osd dump' 2024-03-31T23:11:15.430 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/6a9d8f08-efb2-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-31T23:11:16.808 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:11:16 smithi046 ceph-mon[32498]: pgmap v358: 129 pgs: 2 peering, 127 active+clean; 642 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 4 op/s 2024-03-31T23:11:16.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:11:16 smithi142 ceph-mon[35271]: pgmap v358: 129 pgs: 2 peering, 127 active+clean; 642 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 4 op/s 2024-03-31T23:11:17.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:11:16 smithi162 ceph-mon[36237]: pgmap v358: 129 pgs: 2 peering, 127 active+clean; 642 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 4 op/s 2024-03-31T23:11:17.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:17.065+0000 7f510264f700 1 -- 172.21.15.46:0/3117921520 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f50fc1056b0 msgr2=0x7f50fc107aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:17.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:17.065+0000 7f510264f700 1 --2- 172.21.15.46:0/3117921520 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f50fc1056b0 0x7f50fc107aa0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f50f0009990 tx=0x7f50f0009ca0 comp rx=0 tx=0).stop 2024-03-31T23:11:17.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:17.065+0000 7f510264f700 1 -- 172.21.15.46:0/3117921520 shutdown_connections 2024-03-31T23:11:17.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:17.065+0000 7f510264f700 1 --2- 172.21.15.46:0/3117921520 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f50fc107fe0 0x7f50fc10a3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:17.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:17.065+0000 7f510264f700 1 --2- 172.21.15.46:0/3117921520 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f50fc102d80 0x7f50fc105170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:17.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:17.065+0000 7f510264f700 1 --2- 172.21.15.46:0/3117921520 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f50fc1056b0 0x7f50fc107aa0 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:17.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:17.065+0000 7f510264f700 1 -- 172.21.15.46:0/3117921520 >> 172.21.15.46:0/3117921520 conn(0x7f50fc0fc710 msgr2=0x7f50fc0feb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:11:17.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:17.065+0000 7f510264f700 1 -- 172.21.15.46:0/3117921520 shutdown_connections 2024-03-31T23:11:17.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:17.065+0000 7f510264f700 1 -- 172.21.15.46:0/3117921520 wait complete. 2024-03-31T23:11:17.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:17.066+0000 7f510264f700 1 Processor -- start 2024-03-31T23:11:17.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:17.066+0000 7f510264f700 1 -- start start 2024-03-31T23:11:17.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:17.066+0000 7f510264f700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f50fc102d80 0x7f50fc198410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:17.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:17.066+0000 7f510264f700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f50fc1056b0 0x7f50fc198950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:17.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:17.066+0000 7f510264f700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f50fc107fe0 0x7f50fc19f9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:17.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:17.066+0000 7f510264f700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f50fc10dd30 con 0x7f50fc1056b0 2024-03-31T23:11:17.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:17.066+0000 7f510264f700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f50fc10dbb0 con 0x7f50fc102d80 2024-03-31T23:11:17.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:17.066+0000 7f510264f700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f50fc10deb0 con 0x7f50fc107fe0 2024-03-31T23:11:17.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:17.067+0000 7f50fb7fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f50fc1056b0 0x7f50fc198950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:17.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:17.067+0000 7f50fbfff700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f50fc102d80 0x7f50fc198410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:17.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:17.067+0000 7f50fb7fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f50fc1056b0 0x7f50fc198950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:54742/0 (socket says 172.21.15.46:54742) 2024-03-31T23:11:17.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:17.067+0000 7f50fb7fe700 1 -- 172.21.15.46:0/3454748393 learned_addr learned my addr 172.21.15.46:0/3454748393 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:11:17.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:17.067+0000 7f5100bec700 1 --2- 172.21.15.46:0/3454748393 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f50fc107fe0 0x7f50fc19f9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:17.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:17.067+0000 7f50fbfff700 1 -- 172.21.15.46:0/3454748393 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f50fc107fe0 msgr2=0x7f50fc19f9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:17.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:17.067+0000 7f50fbfff700 1 --2- 172.21.15.46:0/3454748393 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f50fc107fe0 0x7f50fc19f9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:17.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:17.067+0000 7f50fbfff700 1 -- 172.21.15.46:0/3454748393 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f50fc1056b0 msgr2=0x7f50fc198950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:17.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:17.067+0000 7f50fbfff700 1 --2- 172.21.15.46:0/3454748393 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f50fc1056b0 0x7f50fc198950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:17.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:17.067+0000 7f50fbfff700 1 -- 172.21.15.46:0/3454748393 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f50fc1a0070 con 0x7f50fc102d80 2024-03-31T23:11:17.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:17.067+0000 7f5100bec700 1 --2- 172.21.15.46:0/3454748393 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f50fc107fe0 0x7f50fc19f9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:11:17.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:17.067+0000 7f50fb7fe700 1 --2- 172.21.15.46:0/3454748393 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f50fc1056b0 0x7f50fc198950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:11:17.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:17.067+0000 7f50fbfff700 1 --2- 172.21.15.46:0/3454748393 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f50fc102d80 0x7f50fc198410 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f50ec00e9f0 tx=0x7f50ec00ed00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:11:17.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:17.067+0000 7f50f97fa700 1 -- 172.21.15.46:0/3454748393 <== mon.2 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50ec004220 con 0x7f50fc102d80 2024-03-31T23:11:17.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:17.067+0000 7f510264f700 1 -- 172.21.15.46:0/3454748393 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f50fc1a0360 con 0x7f50fc102d80 2024-03-31T23:11:17.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:17.067+0000 7f50f97fa700 1 -- 172.21.15.46:0/3454748393 <== mon.2 v2:172.21.15.142:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f50ec0043c0 con 0x7f50fc102d80 2024-03-31T23:11:17.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:17.067+0000 7f50f97fa700 1 -- 172.21.15.46:0/3454748393 <== mon.2 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50ec005020 con 0x7f50fc102d80 2024-03-31T23:11:17.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:17.067+0000 7f510264f700 1 -- 172.21.15.46:0/3454748393 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f50fc1a08a0 con 0x7f50fc102d80 2024-03-31T23:11:17.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:17.068+0000 7f510264f700 1 -- 172.21.15.46:0/3454748393 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f50fc068f00 con 0x7f50fc102d80 2024-03-31T23:11:17.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:17.070+0000 7f50f97fa700 1 -- 172.21.15.46:0/3454748393 <== mon.2 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 18) ==== 137320+0+0 (secure 0 0 0) 0x7f50ec004750 con 0x7f50fc102d80 2024-03-31T23:11:17.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:17.070+0000 7f50f97fa700 1 --2- 172.21.15.46:0/3454748393 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f50e40a2530 0x7f50e40a49f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:17.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:17.070+0000 7f50fb7fe700 1 --2- 172.21.15.46:0/3454748393 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f50e40a2530 0x7f50e40a49f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:17.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:17.070+0000 7f50f97fa700 1 -- 172.21.15.46:0/3454748393 <== mon.2 v2:172.21.15.142:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9208+0+0 (secure 0 0 0) 0x7f50ec0ceb50 con 0x7f50fc102d80 2024-03-31T23:11:17.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:17.071+0000 7f50fb7fe700 1 --2- 172.21.15.46:0/3454748393 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f50e40a2530 0x7f50e40a49f0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f50f000b2e0 tx=0x7f50f000b800 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:11:17.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:17.076+0000 7f50f97fa700 1 -- 172.21.15.46:0/3454748393 <== mon.2 v2:172.21.15.142:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f50ec09b9d0 con 0x7f50fc102d80 2024-03-31T23:11:17.279 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:17.278+0000 7f510264f700 1 -- 172.21.15.46:0/3454748393 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "osd dump"} v 0) -- 0x7f50fc062e50 con 0x7f50fc102d80 2024-03-31T23:11:17.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:17.279+0000 7f50f97fa700 1 -- 172.21.15.46:0/3454748393 <== mon.2 v2:172.21.15.142:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump"}]=0 v95) ==== 56+0+5698 (secure 0 0 0) 0x7f50ec09b2d0 con 0x7f50fc102d80 2024-03-31T23:11:17.281 INFO:teuthology.orchestra.run.smithi046.stdout:epoch 95 2024-03-31T23:11:17.281 INFO:teuthology.orchestra.run.smithi046.stdout:fsid 6a9d8f08-efb2-11ee-b647-cb9ed24678a4 2024-03-31T23:11:17.281 INFO:teuthology.orchestra.run.smithi046.stdout:created 2024-03-31T23:01:15.463711+0000 2024-03-31T23:11:17.281 INFO:teuthology.orchestra.run.smithi046.stdout:modified 2024-03-31T23:11:10.531495+0000 2024-03-31T23:11:17.281 INFO:teuthology.orchestra.run.smithi046.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-03-31T23:11:17.281 INFO:teuthology.orchestra.run.smithi046.stdout:crush_version 27 2024-03-31T23:11:17.281 INFO:teuthology.orchestra.run.smithi046.stdout:full_ratio 0.95 2024-03-31T23:11:17.281 INFO:teuthology.orchestra.run.smithi046.stdout:backfillfull_ratio 0.9 2024-03-31T23:11:17.281 INFO:teuthology.orchestra.run.smithi046.stdout:nearfull_ratio 0.85 2024-03-31T23:11:17.281 INFO:teuthology.orchestra.run.smithi046.stdout:require_min_compat_client luminous 2024-03-31T23:11:17.281 INFO:teuthology.orchestra.run.smithi046.stdout:min_compat_client luminous 2024-03-31T23:11:17.282 INFO:teuthology.orchestra.run.smithi046.stdout:require_osd_release squid 2024-03-31T23:11:17.282 INFO:teuthology.orchestra.run.smithi046.stdout:stretch_mode_enabled false 2024-03-31T23:11:17.282 INFO:teuthology.orchestra.run.smithi046.stdout:pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 21 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 12.00 2024-03-31T23:11:17.282 INFO:teuthology.orchestra.run.smithi046.stdout:pool 2 'cephfs_metadata' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 85 flags hashpspool stripe_width 0 pg_autoscale_bias 4 pg_num_min 16 recovery_priority 5 application cephfs read_balance_score 1.88 2024-03-31T23:11:17.282 INFO:teuthology.orchestra.run.smithi046.stdout:pool 3 'cephfs_data' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 64 pgp_num 64 autoscale_mode off last_change 86 flags hashpspool stripe_width 0 pg_num_min 64 target_size_ratio 0.05 application cephfs read_balance_score 1.50 2024-03-31T23:11:17.282 INFO:teuthology.orchestra.run.smithi046.stdout:pool 4 'cephfs_data_ec' erasure profile cephfs_data_ec size 4 min_size 3 crush_rule 1 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 91 flags hashpspool,ec_overwrites stripe_width 8192 pg_num_min 64 target_size_ratio 0.9 application cephfs 2024-03-31T23:11:17.282 INFO:teuthology.orchestra.run.smithi046.stdout:max_osd 12 2024-03-31T23:11:17.282 INFO:teuthology.orchestra.run.smithi046.stdout:osd.0 up in weight 1 up_from 8 up_thru 92 down_at 0 last_clean_interval [0,0) [v2:172.21.15.46:6802/1824772695,v1:172.21.15.46:6803/1824772695] [v2:172.21.15.46:6804/1824772695,v1:172.21.15.46:6805/1824772695] exists,up 47429095-3f46-4ed7-b7c4-504464719815 2024-03-31T23:11:17.282 INFO:teuthology.orchestra.run.smithi046.stdout:osd.1 up in weight 1 up_from 13 up_thru 92 down_at 0 last_clean_interval [0,0) [v2:172.21.15.142:6800/973693922,v1:172.21.15.142:6801/973693922] [v2:172.21.15.142:6802/973693922,v1:172.21.15.142:6803/973693922] exists,up be56f883-411e-4dbe-b50e-a158134df4a6 2024-03-31T23:11:17.282 INFO:teuthology.orchestra.run.smithi046.stdout:osd.2 up in weight 1 up_from 18 up_thru 88 down_at 0 last_clean_interval [0,0) [v2:172.21.15.162:6800/4084261813,v1:172.21.15.162:6801/4084261813] [v2:172.21.15.162:6802/4084261813,v1:172.21.15.162:6803/4084261813] exists,up c99a8794-ab19-4260-9a8d-4bc0bd2c8175 2024-03-31T23:11:17.282 INFO:teuthology.orchestra.run.smithi046.stdout:osd.3 up in weight 1 up_from 25 up_thru 88 down_at 0 last_clean_interval [0,0) [v2:172.21.15.46:6810/3496317970,v1:172.21.15.46:6811/3496317970] [v2:172.21.15.46:6812/3496317970,v1:172.21.15.46:6813/3496317970] exists,up f877814e-9ef6-4872-a081-5609bb9ebc06 2024-03-31T23:11:17.282 INFO:teuthology.orchestra.run.smithi046.stdout:osd.4 up in weight 1 up_from 33 up_thru 94 down_at 0 last_clean_interval [0,0) [v2:172.21.15.142:6808/2996266929,v1:172.21.15.142:6809/2996266929] [v2:172.21.15.142:6810/2996266929,v1:172.21.15.142:6811/2996266929] exists,up beecec0a-8280-44fe-8f5f-8498a35eca8f 2024-03-31T23:11:17.282 INFO:teuthology.orchestra.run.smithi046.stdout:osd.5 up in weight 1 up_from 39 up_thru 92 down_at 0 last_clean_interval [0,0) [v2:172.21.15.162:6808/914208293,v1:172.21.15.162:6809/914208293] [v2:172.21.15.162:6810/914208293,v1:172.21.15.162:6811/914208293] exists,up 9da7ff75-f658-45ea-a02f-1943fdda76b9 2024-03-31T23:11:17.283 INFO:teuthology.orchestra.run.smithi046.stdout:osd.6 up in weight 1 up_from 47 up_thru 92 down_at 0 last_clean_interval [0,0) [v2:172.21.15.46:6818/56291966,v1:172.21.15.46:6819/56291966] [v2:172.21.15.46:6820/56291966,v1:172.21.15.46:6821/56291966] exists,up 8fa11224-cb26-4d3f-87d1-6a8f31de2a19 2024-03-31T23:11:17.283 INFO:teuthology.orchestra.run.smithi046.stdout:osd.7 up in weight 1 up_from 52 up_thru 92 down_at 0 last_clean_interval [0,0) [v2:172.21.15.142:6816/2146598446,v1:172.21.15.142:6817/2146598446] [v2:172.21.15.142:6818/2146598446,v1:172.21.15.142:6819/2146598446] exists,up 0d0338b7-4d0a-4bb0-b152-8f5224871d9f 2024-03-31T23:11:17.283 INFO:teuthology.orchestra.run.smithi046.stdout:osd.8 up in weight 1 up_from 60 up_thru 94 down_at 0 last_clean_interval [0,0) [v2:172.21.15.162:6816/1173078318,v1:172.21.15.162:6817/1173078318] [v2:172.21.15.162:6818/1173078318,v1:172.21.15.162:6819/1173078318] exists,up 8b6f6037-892a-416c-babb-7552648d0b3a 2024-03-31T23:11:17.283 INFO:teuthology.orchestra.run.smithi046.stdout:osd.9 up in weight 1 up_from 68 up_thru 88 down_at 0 last_clean_interval [0,0) [v2:172.21.15.46:6826/532708881,v1:172.21.15.46:6827/532708881] [v2:172.21.15.46:6828/532708881,v1:172.21.15.46:6829/532708881] exists,up faccce25-6bf2-4220-8858-cda82b72be68 2024-03-31T23:11:17.283 INFO:teuthology.orchestra.run.smithi046.stdout:osd.10 up in weight 1 up_from 73 up_thru 92 down_at 0 last_clean_interval [0,0) [v2:172.21.15.142:6824/2845222366,v1:172.21.15.142:6825/2845222366] [v2:172.21.15.142:6826/2845222366,v1:172.21.15.142:6827/2845222366] exists,up cdd356db-f9eb-4080-95ab-db5d8223e572 2024-03-31T23:11:17.283 INFO:teuthology.orchestra.run.smithi046.stdout:osd.11 up in weight 1 up_from 81 up_thru 94 down_at 0 last_clean_interval [0,0) [v2:172.21.15.162:6824/1986323778,v1:172.21.15.162:6825/1986323778] [v2:172.21.15.162:6826/1986323778,v1:172.21.15.162:6827/1986323778] exists,up 72104d1c-3ea2-45ad-bc0b-0500f146c06b 2024-03-31T23:11:17.283 INFO:teuthology.orchestra.run.smithi046.stdout:pg_upmap_items 2.a [2,11] 2024-03-31T23:11:17.283 INFO:teuthology.orchestra.run.smithi046.stdout:pg_upmap_items 2.b [8,11] 2024-03-31T23:11:17.283 INFO:teuthology.orchestra.run.smithi046.stdout:pg_upmap_items 3.6 [4,7] 2024-03-31T23:11:17.283 INFO:teuthology.orchestra.run.smithi046.stdout:pg_upmap_items 3.c [3,9] 2024-03-31T23:11:17.283 INFO:teuthology.orchestra.run.smithi046.stdout:pg_upmap_items 3.1a [5,6] 2024-03-31T23:11:17.283 INFO:teuthology.orchestra.run.smithi046.stdout:pg_upmap_items 3.1b [4,8] 2024-03-31T23:11:17.283 INFO:teuthology.orchestra.run.smithi046.stdout:pg_upmap_items 3.1d [4,8] 2024-03-31T23:11:17.284 INFO:teuthology.orchestra.run.smithi046.stdout:pg_upmap_items 3.2a [3,6] 2024-03-31T23:11:17.284 INFO:teuthology.orchestra.run.smithi046.stdout:pg_upmap_items 3.33 [5,9] 2024-03-31T23:11:17.284 INFO:teuthology.orchestra.run.smithi046.stdout:pg_upmap_items 3.36 [4,9] 2024-03-31T23:11:17.284 INFO:teuthology.orchestra.run.smithi046.stdout:pg_upmap_items 4.6 [5,8] 2024-03-31T23:11:17.284 INFO:teuthology.orchestra.run.smithi046.stdout:pg_upmap_items 4.17 [9,8] 2024-03-31T23:11:17.284 INFO:teuthology.orchestra.run.smithi046.stdout:pg_upmap_items 4.18 [9,8] 2024-03-31T23:11:17.284 INFO:teuthology.orchestra.run.smithi046.stdout:pg_upmap_items 4.1b [9,10] 2024-03-31T23:11:17.284 INFO:teuthology.orchestra.run.smithi046.stdout:blocklist 172.21.15.46:0/4021039857 expires 2024-04-01T23:02:08.493619+0000 2024-03-31T23:11:17.284 INFO:teuthology.orchestra.run.smithi046.stdout:blocklist 172.21.15.46:6801/3681182647 expires 2024-04-01T23:02:08.493619+0000 2024-03-31T23:11:17.284 INFO:teuthology.orchestra.run.smithi046.stdout:blocklist 172.21.15.46:0/4022214806 expires 2024-04-01T23:02:08.493619+0000 2024-03-31T23:11:17.284 INFO:teuthology.orchestra.run.smithi046.stdout:blocklist 172.21.15.46:6800/1515528215 expires 2024-04-01T23:01:41.910162+0000 2024-03-31T23:11:17.284 INFO:teuthology.orchestra.run.smithi046.stdout:blocklist 172.21.15.46:6800/3681182647 expires 2024-04-01T23:02:08.493619+0000 2024-03-31T23:11:17.284 INFO:teuthology.orchestra.run.smithi046.stdout:blocklist 172.21.15.46:0/143885707 expires 2024-04-01T23:01:41.910162+0000 2024-03-31T23:11:17.284 INFO:teuthology.orchestra.run.smithi046.stdout:blocklist 172.21.15.46:6801/1515528215 expires 2024-04-01T23:01:41.910162+0000 2024-03-31T23:11:17.284 INFO:teuthology.orchestra.run.smithi046.stdout:blocklist 172.21.15.46:0/1680389778 expires 2024-04-01T23:01:41.910162+0000 2024-03-31T23:11:17.285 INFO:teuthology.orchestra.run.smithi046.stdout:blocklist 172.21.15.46:0/1782133966 expires 2024-04-01T23:01:41.910162+0000 2024-03-31T23:11:17.285 INFO:teuthology.orchestra.run.smithi046.stdout:blocklist 172.21.15.46:0/3925209376 expires 2024-04-01T23:02:08.493619+0000 2024-03-31T23:11:17.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:17.281+0000 7f510264f700 1 -- 172.21.15.46:0/3454748393 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f50e40a2530 msgr2=0x7f50e40a49f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:17.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:17.281+0000 7f510264f700 1 --2- 172.21.15.46:0/3454748393 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f50e40a2530 0x7f50e40a49f0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f50f000b2e0 tx=0x7f50f000b800 comp rx=0 tx=0).stop 2024-03-31T23:11:17.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:17.281+0000 7f510264f700 1 -- 172.21.15.46:0/3454748393 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f50fc102d80 msgr2=0x7f50fc198410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:17.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:17.281+0000 7f510264f700 1 --2- 172.21.15.46:0/3454748393 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f50fc102d80 0x7f50fc198410 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f50ec00e9f0 tx=0x7f50ec00ed00 comp rx=0 tx=0).stop 2024-03-31T23:11:17.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:17.282+0000 7f510264f700 1 -- 172.21.15.46:0/3454748393 shutdown_connections 2024-03-31T23:11:17.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:17.282+0000 7f510264f700 1 --2- 172.21.15.46:0/3454748393 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f50e40a2530 0x7f50e40a49f0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:17.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:17.282+0000 7f510264f700 1 --2- 172.21.15.46:0/3454748393 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f50fc107fe0 0x7f50fc19f9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:17.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:17.282+0000 7f510264f700 1 --2- 172.21.15.46:0/3454748393 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f50fc102d80 0x7f50fc198410 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:17.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:17.282+0000 7f510264f700 1 --2- 172.21.15.46:0/3454748393 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f50fc1056b0 0x7f50fc198950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:17.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:17.282+0000 7f510264f700 1 -- 172.21.15.46:0/3454748393 >> 172.21.15.46:0/3454748393 conn(0x7f50fc0fc710 msgr2=0x7f50fc104c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:11:17.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:17.282+0000 7f510264f700 1 -- 172.21.15.46:0/3454748393 shutdown_connections 2024-03-31T23:11:17.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:17.282+0000 7f510264f700 1 -- 172.21.15.46:0/3454748393 wait complete. 2024-03-31T23:11:17.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:11:17 smithi046 ceph-mon[32498]: from='client.? 172.21.15.46:0/3454748393' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-03-31T23:11:17.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:11:17 smithi142 ceph-mon[35271]: from='client.? 172.21.15.46:0/3454748393' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-03-31T23:11:18.003 INFO:teuthology.run_tasks:Running task fs.ready... 2024-03-31T23:11:18.022 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-31T23:11:18.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:11:17 smithi162 ceph-mon[36237]: from='client.? 172.21.15.46:0/3454748393' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-03-31T23:11:18.132 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:18.129+0000 7fd5f9b56640 1 -- 172.21.15.46:0/2350189378 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd5f4100690 msgr2=0x7fd5f4100ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:18.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:18.129+0000 7fd5f9b56640 1 --2- 172.21.15.46:0/2350189378 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd5f4100690 0x7fd5f4100ab0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fd5e4009900 tx=0x7fd5e402bc30 comp rx=0 tx=0).stop 2024-03-31T23:11:18.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:18.129+0000 7fd5f9b56640 1 -- 172.21.15.46:0/2350189378 shutdown_connections 2024-03-31T23:11:18.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:18.129+0000 7fd5f9b56640 1 --2- 172.21.15.46:0/2350189378 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd5f4102240 0x7fd5f4108ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:18.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:18.129+0000 7fd5f9b56640 1 --2- 172.21.15.46:0/2350189378 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd5f4101880 0x7fd5f4101d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:18.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:18.129+0000 7fd5f9b56640 1 --2- 172.21.15.46:0/2350189378 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd5f4100690 0x7fd5f4100ab0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:18.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:18.129+0000 7fd5f9b56640 1 -- 172.21.15.46:0/2350189378 >> 172.21.15.46:0/2350189378 conn(0x7fd5f40fbc00 msgr2=0x7fd5f40fe040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:11:18.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:18.130+0000 7fd5f9b56640 1 -- 172.21.15.46:0/2350189378 shutdown_connections 2024-03-31T23:11:18.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:18.130+0000 7fd5f9b56640 1 -- 172.21.15.46:0/2350189378 wait complete. 2024-03-31T23:11:18.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:18.130+0000 7fd5f9b56640 1 Processor -- start 2024-03-31T23:11:18.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:18.130+0000 7fd5f9b56640 1 -- start start 2024-03-31T23:11:18.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:18.130+0000 7fd5f9b56640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd5f4100690 0x7fd5f406a950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:18.134 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:18.130+0000 7fd5f9b56640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd5f4101880 0x7fd5f419c700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:18.134 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:18.130+0000 7fd5f9b56640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd5f4102240 0x7fd5f419ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:18.134 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:18.130+0000 7fd5f9b56640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fd5f410c1e0 con 0x7fd5f4102240 2024-03-31T23:11:18.134 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:18.130+0000 7fd5f9b56640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fd5f410c060 con 0x7fd5f4101880 2024-03-31T23:11:18.134 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:18.130+0000 7fd5f9b56640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fd5f410c360 con 0x7fd5f4100690 2024-03-31T23:11:18.134 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:18.131+0000 7fd5f3fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd5f4102240 0x7fd5f419ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:18.134 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:18.131+0000 7fd5f3fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd5f4102240 0x7fd5f419ed70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:54750/0 (socket says 172.21.15.46:54750) 2024-03-31T23:11:18.134 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:18.131+0000 7fd5f3fff640 1 -- 172.21.15.46:0/1967564836 learned_addr learned my addr 172.21.15.46:0/1967564836 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:11:18.134 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:18.131+0000 7fd5f2ffd640 1 --2- 172.21.15.46:0/1967564836 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd5f4101880 0x7fd5f419c700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:18.134 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:18.131+0000 7fd5f37fe640 1 --2- 172.21.15.46:0/1967564836 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd5f4100690 0x7fd5f406a950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:18.134 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:18.131+0000 7fd5f3fff640 1 -- 172.21.15.46:0/1967564836 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd5f4100690 msgr2=0x7fd5f406a950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:18.134 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:18.131+0000 7fd5f3fff640 1 --2- 172.21.15.46:0/1967564836 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd5f4100690 0x7fd5f406a950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:18.134 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:18.131+0000 7fd5f3fff640 1 -- 172.21.15.46:0/1967564836 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd5f4101880 msgr2=0x7fd5f419c700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:18.135 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:18.131+0000 7fd5f3fff640 1 --2- 172.21.15.46:0/1967564836 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd5f4101880 0x7fd5f419c700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:18.135 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:18.131+0000 7fd5f3fff640 1 -- 172.21.15.46:0/1967564836 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5f419f630 con 0x7fd5f4102240 2024-03-31T23:11:18.135 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:18.131+0000 7fd5f3fff640 1 --2- 172.21.15.46:0/1967564836 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd5f4102240 0x7fd5f419ed70 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7fd5ec00efd0 tx=0x7fd5ec00c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:11:18.135 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:18.131+0000 7fd5f0ff9640 1 -- 172.21.15.46:0/1967564836 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5ec003b70 con 0x7fd5f4102240 2024-03-31T23:11:18.135 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:18.131+0000 7fd5f0ff9640 1 -- 172.21.15.46:0/1967564836 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd5ec003d10 con 0x7fd5f4102240 2024-03-31T23:11:18.135 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:18.131+0000 7fd5f9b56640 1 -- 172.21.15.46:0/1967564836 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd5f419f920 con 0x7fd5f4102240 2024-03-31T23:11:18.135 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:18.131+0000 7fd5f0ff9640 1 -- 172.21.15.46:0/1967564836 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5ec010980 con 0x7fd5f4102240 2024-03-31T23:11:18.135 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:18.131+0000 7fd5f9b56640 1 -- 172.21.15.46:0/1967564836 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd5f419fe20 con 0x7fd5f4102240 2024-03-31T23:11:18.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:18.132+0000 7fd5f9b56640 1 -- 172.21.15.46:0/1967564836 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd5f4107a30 con 0x7fd5f4102240 2024-03-31T23:11:18.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:18.133+0000 7fd5f0ff9640 1 -- 172.21.15.46:0/1967564836 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 18) ==== 137320+0+0 (secure 0 0 0) 0x7fd5ec004020 con 0x7fd5f4102240 2024-03-31T23:11:18.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:18.134+0000 7fd5f0ff9640 1 --2- 172.21.15.46:0/1967564836 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fd5dc0a24a0 0x7fd5dc0a4960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:18.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:18.134+0000 7fd5f37fe640 1 --2- 172.21.15.46:0/1967564836 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fd5dc0a24a0 0x7fd5dc0a4960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:18.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:18.134+0000 7fd5f0ff9640 1 -- 172.21.15.46:0/1967564836 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9208+0+0 (secure 0 0 0) 0x7fd5ec0cef90 con 0x7fd5f4102240 2024-03-31T23:11:18.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:18.134+0000 7fd5f37fe640 1 --2- 172.21.15.46:0/1967564836 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fd5dc0a24a0 0x7fd5dc0a4960 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fd5e402c240 tx=0x7fd5e402c7e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:11:18.140 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:18.139+0000 7fd5f0ff9640 1 -- 172.21.15.46:0/1967564836 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fd5ec09bd20 con 0x7fd5f4102240 2024-03-31T23:11:18.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:18.322+0000 7fd5f9b56640 1 -- 172.21.15.46:0/1967564836 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fd5f4062350 con 0x7fd5f4102240 2024-03-31T23:11:18.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:18.325+0000 7fd5f0ff9640 1 -- 172.21.15.46:0/1967564836 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v95) ==== 74+0+24819 (secure 0 0 0) 0x7fd5ec09b620 con 0x7fd5f4102240 2024-03-31T23:11:18.325 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-31T23:11:18.325 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":95,"fsid":"6a9d8f08-efb2-11ee-b647-cb9ed24678a4","created":"2024-03-31T23:01:15.463711+0000","modified":"2024-03-31T23:11:10.531495+0000","last_up_change":"2024-03-31T23:08:20.199888+0000","last_in_change":"2024-03-31T23:07:59.438035+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-31T23:04:32.823751+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-31T23:09:09.145711+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-31T23:09:10.148579+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-31T23:09:13.419519+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"91","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"47429095-3f46-4ed7-b7c4-504464719815","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":1824772695},{"type":"v1","addr":"172.21.15.46:6803","nonce":1824772695}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":1824772695},{"type":"v1","addr":"172.21.15.46:6805","nonce":1824772695}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":1824772695},{"type":"v1","addr":"172.21.15.46:6809","nonce":1824772695}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":1824772695},{"type":"v1","addr":"172.21.15.46:6807","nonce":1824772695}]},"public_addr":"172.21.15.46:6803/1824772695","cluster_addr":"172.21.15.46:6805/1824772695","heartbeat_back_addr":"172.21.15.46:6809/1824772695","heartbeat_front_addr":"172.21.15.46:6807/1824772695","state":["exists","up"]},{"osd":1,"uuid":"be56f883-411e-4dbe-b50e-a158134df4a6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6800","nonce":973693922},{"type":"v1","addr":"172.21.15.142:6801","nonce":973693922}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6802","nonce":973693922},{"type":"v1","addr":"172.21.15.142:6803","nonce":973693922}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6806","nonce":973693922},{"type":"v1","addr":"172.21.15.142:6807","nonce":973693922}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6804","nonce":973693922},{"type":"v1","addr":"172.21.15.142:6805","nonce":973693922}]},"public_addr":"172.21.15.142:6801/973693922","cluster_addr":"172.21.15.142:6803/973693922","heartbeat_back_addr":"172.21.15.142:6807/973693922","heartbeat_front_addr":"172.21.15.142:6805/973693922","state":["exists","up"]},{"osd":2,"uuid":"c99a8794-ab19-4260-9a8d-4bc0bd2c8175","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6800","nonce":4084261813},{"type":"v1","addr":"172.21.15.162:6801","nonce":4084261813}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6802","nonce":4084261813},{"type":"v1","addr":"172.21.15.162:6803","nonce":4084261813}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6806","nonce":4084261813},{"type":"v1","addr":"172.21.15.162:6807","nonce":4084261813}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6804","nonce":4084261813},{"type":"v1","addr":"172.21.15.162:6805","nonce":4084261813}]},"public_addr":"172.21.15.162:6801/4084261813","cluster_addr":"172.21.15.162:6803/4084261813","heartbeat_back_addr":"172.21.15.162:6807/4084261813","heartbeat_front_addr":"172.21.15.162:6805/4084261813","state":["exists","up"]},{"osd":3,"uuid":"f877814e-9ef6-4872-a081-5609bb9ebc06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":3496317970},{"type":"v1","addr":"172.21.15.46:6811","nonce":3496317970}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":3496317970},{"type":"v1","addr":"172.21.15.46:6813","nonce":3496317970}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":3496317970},{"type":"v1","addr":"172.21.15.46:6817","nonce":3496317970}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":3496317970},{"type":"v1","addr":"172.21.15.46:6815","nonce":3496317970}]},"public_addr":"172.21.15.46:6811/3496317970","cluster_addr":"172.21.15.46:6813/3496317970","heartbeat_back_addr":"172.21.15.46:6817/3496317970","heartbeat_front_addr":"172.21.15.46:6815/3496317970","state":["exists","up"]},{"osd":4,"uuid":"beecec0a-8280-44fe-8f5f-8498a35eca8f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6808","nonce":2996266929},{"type":"v1","addr":"172.21.15.142:6809","nonce":2996266929}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6810","nonce":2996266929},{"type":"v1","addr":"172.21.15.142:6811","nonce":2996266929}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6814","nonce":2996266929},{"type":"v1","addr":"172.21.15.142:6815","nonce":2996266929}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6812","nonce":2996266929},{"type":"v1","addr":"172.21.15.142:6813","nonce":2996266929}]},"public_addr":"172.21.15.142:6809/2996266929","cluster_addr":"172.21.15.142:6811/2996266929","heartbeat_back_addr":"172.21.15.142:6815/2996266929","heartbeat_front_addr":"172.21.15.142:6813/2996266929","state":["exists","up"]},{"osd":5,"uuid":"9da7ff75-f658-45ea-a02f-1943fdda76b9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6808","nonce":914208293},{"type":"v1","addr":"172.21.15.162:6809","nonce":914208293}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6810","nonce":914208293},{"type":"v1","addr":"172.21.15.162:6811","nonce":914208293}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6814","nonce":914208293},{"type":"v1","addr":"172.21.15.162:6815","nonce":914208293}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6812","nonce":914208293},{"type":"v1","addr":"172.21.15.162:6813","nonce":914208293}]},"public_addr":"172.21.15.162:6809/914208293","cluster_addr":"172.21.15.162:6811/914208293","heartbeat_back_addr":"172.21.15.162:6815/914208293","heartbeat_front_addr":"172.21.15.162:6813/914208293","state":["exists","up"]},{"osd":6,"uuid":"8fa11224-cb26-4d3f-87d1-6a8f31de2a19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":56291966},{"type":"v1","addr":"172.21.15.46:6819","nonce":56291966}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":56291966},{"type":"v1","addr":"172.21.15.46:6821","nonce":56291966}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6824","nonce":56291966},{"type":"v1","addr":"172.21.15.46:6825","nonce":56291966}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":56291966},{"type":"v1","addr":"172.21.15.46:6823","nonce":56291966}]},"public_addr":"172.21.15.46:6819/56291966","cluster_addr":"172.21.15.46:6821/56291966","heartbeat_back_addr":"172.21.15.46:6825/56291966","heartbeat_front_addr":"172.21.15.46:6823/56291966","state":["exists","up"]},{"osd":7,"uuid":"0d0338b7-4d0a-4bb0-b152-8f5224871d9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6816","nonce":2146598446},{"type":"v1","addr":"172.21.15.142:6817","nonce":2146598446}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6818","nonce":2146598446},{"type":"v1","addr":"172.21.15.142:6819","nonce":2146598446}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6822","nonce":2146598446},{"type":"v1","addr":"172.21.15.142:6823","nonce":2146598446}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6820","nonce":2146598446},{"type":"v1","addr":"172.21.15.142:6821","nonce":2146598446}]},"public_addr":"172.21.15.142:6817/2146598446","cluster_addr":"172.21.15.142:6819/2146598446","heartbeat_back_addr":"172.21.15.142:6823/2146598446","heartbeat_front_addr":"172.21.15.142:6821/2146598446","state":["exists","up"]},{"osd":8,"uuid":"8b6f6037-892a-416c-babb-7552648d0b3a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6816","nonce":1173078318},{"type":"v1","addr":"172.21.15.162:6817","nonce":1173078318}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6818","nonce":1173078318},{"type":"v1","addr":"172.21.15.162:6819","nonce":1173078318}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6822","nonce":1173078318},{"type":"v1","addr":"172.21.15.162:6823","nonce":1173078318}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6820","nonce":1173078318},{"type":"v1","addr":"172.21.15.162:6821","nonce":1173078318}]},"public_addr":"172.21.15.162:6817/1173078318","cluster_addr":"172.21.15.162:6819/1173078318","heartbeat_back_addr":"172.21.15.162:6823/1173078318","heartbeat_front_addr":"172.21.15.162:6821/1173078318","state":["exists","up"]},{"osd":9,"uuid":"faccce25-6bf2-4220-8858-cda82b72be68","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":532708881},{"type":"v1","addr":"172.21.15.46:6827","nonce":532708881}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6828","nonce":532708881},{"type":"v1","addr":"172.21.15.46:6829","nonce":532708881}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":532708881},{"type":"v1","addr":"172.21.15.46:6833","nonce":532708881}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":532708881},{"type":"v1","addr":"172.21.15.46:6831","nonce":532708881}]},"public_addr":"172.21.15.46:6827/532708881","cluster_addr":"172.21.15.46:6829/532708881","heartbeat_back_addr":"172.21.15.46:6833/532708881","heartbeat_front_addr":"172.21.15.46:6831/532708881","state":["exists","up"]},{"osd":10,"uuid":"cdd356db-f9eb-4080-95ab-db5d8223e572","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6824","nonce":2845222366},{"type":"v1","addr":"172.21.15.142:6825","nonce":2845222366}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6826","nonce":2845222366},{"type":"v1","addr":"172.21.15.142:6827","nonce":2845222366}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6830","nonce":2845222366},{"type":"v1","addr":"172.21.15.142:6831","nonce":2845222366}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6828","nonce":2845222366},{"type":"v1","addr":"172.21.15.142:6829","nonce":2845222366}]},"public_addr":"172.21.15.142:6825/2845222366","cluster_addr":"172.21.15.142:6827/2845222366","heartbeat_back_addr":"172.21.15.142:6831/2845222366","heartbeat_front_addr":"172.21.15.142:6829/2845222366","state":["exists","up"]},{"osd":11,"uuid":"72104d1c-3ea2-45ad-bc0b-0500f146c06b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6824","nonce":1986323778},{"type":"v1","addr":"172.21.15.162:6825","nonce":1986323778}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6826","nonce":1986323778},{"type":"v1","addr":"172.21.15.162:6827","nonce":1986323778}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6830","nonce":1986323778},{"type":"v1","addr":"172.21.15.162:6831","nonce":1986323778}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6828","nonce":1986323778},{"type":"v1","addr":"172.21.15.162:6829","nonce":1986323778}]},"public_addr":"172.21.15.162:6825/1986323778","cluster_addr":"172.21.15.162:6827/1986323778","heartbeat_back_addr":"172.21.15.162:6831/1986323778","heartbeat_front_addr":"172.21.15.162:6829/1986323778","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:03:44.996331+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:04:07.994738+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:04:30.942255+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:04:55.588884+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:05:19.900600+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:05:43.807442+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:06:09.026976+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:06:34.586784+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:06:59.751605+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:07:26.144495+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:07:52.326909+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:08:17.423042+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.b","mappings":[{"from":8,"to":11}]},{"pgid":"3.6","mappings":[{"from":4,"to":7}]},{"pgid":"3.c","mappings":[{"from":3,"to":9}]},{"pgid":"3.1a","mappings":[{"from":5,"to":6}]},{"pgid":"3.1b","mappings":[{"from":4,"to":8}]},{"pgid":"3.1d","mappings":[{"from":4,"to":8}]},{"pgid":"3.2a","mappings":[{"from":3,"to":6}]},{"pgid":"3.33","mappings":[{"from":5,"to":9}]},{"pgid":"3.36","mappings":[{"from":4,"to":9}]},{"pgid":"4.6","mappings":[{"from":5,"to":8}]},{"pgid":"4.17","mappings":[{"from":9,"to":8}]},{"pgid":"4.18","mappings":[{"from":9,"to":8}]},{"pgid":"4.1b","mappings":[{"from":9,"to":10}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.46:0/4021039857":"2024-04-01T23:02:08.493619+0000","172.21.15.46:6801/3681182647":"2024-04-01T23:02:08.493619+0000","172.21.15.46:0/4022214806":"2024-04-01T23:02:08.493619+0000","172.21.15.46:6800/1515528215":"2024-04-01T23:01:41.910162+0000","172.21.15.46:6800/3681182647":"2024-04-01T23:02:08.493619+0000","172.21.15.46:0/143885707":"2024-04-01T23:01:41.910162+0000","172.21.15.46:6801/1515528215":"2024-04-01T23:01:41.910162+0000","172.21.15.46:0/1680389778":"2024-04-01T23:01:41.910162+0000","172.21.15.46:0/1782133966":"2024-04-01T23:01:41.910162+0000","172.21.15.46:0/3925209376":"2024-04-01T23:02:08.493619+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-31T23:11:18.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:18.326+0000 7fd5f9b56640 1 -- 172.21.15.46:0/1967564836 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fd5dc0a24a0 msgr2=0x7fd5dc0a4960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:18.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:18.326+0000 7fd5f9b56640 1 --2- 172.21.15.46:0/1967564836 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fd5dc0a24a0 0x7fd5dc0a4960 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fd5e402c240 tx=0x7fd5e402c7e0 comp rx=0 tx=0).stop 2024-03-31T23:11:18.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:18.326+0000 7fd5f9b56640 1 -- 172.21.15.46:0/1967564836 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd5f4102240 msgr2=0x7fd5f419ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:18.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:18.326+0000 7fd5f9b56640 1 --2- 172.21.15.46:0/1967564836 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd5f4102240 0x7fd5f419ed70 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7fd5ec00efd0 tx=0x7fd5ec00c6a0 comp rx=0 tx=0).stop 2024-03-31T23:11:18.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:18.326+0000 7fd5f9b56640 1 -- 172.21.15.46:0/1967564836 shutdown_connections 2024-03-31T23:11:18.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:18.326+0000 7fd5f9b56640 1 --2- 172.21.15.46:0/1967564836 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fd5dc0a24a0 0x7fd5dc0a4960 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:18.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:18.326+0000 7fd5f9b56640 1 --2- 172.21.15.46:0/1967564836 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd5f4102240 0x7fd5f419ed70 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:18.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:18.326+0000 7fd5f9b56640 1 --2- 172.21.15.46:0/1967564836 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd5f4101880 0x7fd5f419c700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:18.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:18.326+0000 7fd5f9b56640 1 --2- 172.21.15.46:0/1967564836 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd5f4100690 0x7fd5f406a950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:18.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:18.326+0000 7fd5f9b56640 1 -- 172.21.15.46:0/1967564836 >> 172.21.15.46:0/1967564836 conn(0x7fd5f40fbc00 msgr2=0x7fd5f4104c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:11:18.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:18.326+0000 7fd5f9b56640 1 -- 172.21.15.46:0/1967564836 shutdown_connections 2024-03-31T23:11:18.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:18.326+0000 7fd5f9b56640 1 -- 172.21.15.46:0/1967564836 wait complete. 2024-03-31T23:11:18.339 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-31T23:04:32.823751+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-31T23:09:09.145711+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-31T23:09:10.148579+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-31T23:09:13.419519+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '91', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-31T23:11:18.339 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-31T23:11:18.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:18.444+0000 7f51f9e84640 1 -- 172.21.15.46:0/1678097781 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f51f4107430 msgr2=0x7f51f4109880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:18.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:18.444+0000 7f51f9e84640 1 --2- 172.21.15.46:0/1678097781 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f51f4107430 0x7f51f4109880 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f51ec009ca0 tx=0x7f51ec02c0f0 comp rx=0 tx=0).stop 2024-03-31T23:11:18.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:18.444+0000 7f51f9e84640 1 -- 172.21.15.46:0/1678097781 shutdown_connections 2024-03-31T23:11:18.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:18.444+0000 7f51f9e84640 1 --2- 172.21.15.46:0/1678097781 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f51f4107430 0x7f51f4109880 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:18.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:18.444+0000 7f51f9e84640 1 --2- 172.21.15.46:0/1678097781 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f51f4104b00 0x7f51f4106ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:18.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:18.444+0000 7f51f9e84640 1 --2- 172.21.15.46:0/1678097781 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f51f41021d0 0x7f51f41045c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:18.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:18.444+0000 7f51f9e84640 1 -- 172.21.15.46:0/1678097781 >> 172.21.15.46:0/1678097781 conn(0x7f51f40fbbc0 msgr2=0x7f51f40fe000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:11:18.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:18.444+0000 7f51f9e84640 1 -- 172.21.15.46:0/1678097781 shutdown_connections 2024-03-31T23:11:18.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:18.444+0000 7f51f9e84640 1 -- 172.21.15.46:0/1678097781 wait complete. 2024-03-31T23:11:18.447 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:18.445+0000 7f51f9e84640 1 Processor -- start 2024-03-31T23:11:18.447 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:18.445+0000 7f51f9e84640 1 -- start start 2024-03-31T23:11:18.449 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:18.445+0000 7f51f9e84640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f51f41021d0 0x7f51f4197940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:18.449 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:18.445+0000 7f51f9e84640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f51f4104b00 0x7f51f4197e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:18.449 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:18.445+0000 7f51f9e84640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f51f4107430 0x7f51f419eea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:18.449 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:18.445+0000 7f51f9e84640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f51f410d2a0 con 0x7f51f41021d0 2024-03-31T23:11:18.449 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:18.445+0000 7f51f9e84640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f51f410d120 con 0x7f51f4107430 2024-03-31T23:11:18.449 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:18.445+0000 7f51f9e84640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f51f410d420 con 0x7f51f4104b00 2024-03-31T23:11:18.449 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:18.445+0000 7f51f37fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f51f41021d0 0x7f51f4197940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:18.449 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:18.445+0000 7f51f37fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f51f41021d0 0x7f51f4197940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:54762/0 (socket says 172.21.15.46:54762) 2024-03-31T23:11:18.449 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:18.445+0000 7f51f37fe640 1 -- 172.21.15.46:0/3718732984 learned_addr learned my addr 172.21.15.46:0/3718732984 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:11:18.449 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:18.445+0000 7f51f2ffd640 1 --2- 172.21.15.46:0/3718732984 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f51f4104b00 0x7f51f4197e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:18.449 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:18.446+0000 7f51f3fff640 1 --2- 172.21.15.46:0/3718732984 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f51f4107430 0x7f51f419eea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:18.449 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:18.446+0000 7f51f37fe640 1 -- 172.21.15.46:0/3718732984 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f51f4104b00 msgr2=0x7f51f4197e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:18.449 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:18.446+0000 7f51f37fe640 1 --2- 172.21.15.46:0/3718732984 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f51f4104b00 0x7f51f4197e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:18.449 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:18.446+0000 7f51f37fe640 1 -- 172.21.15.46:0/3718732984 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f51f4107430 msgr2=0x7f51f419eea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:18.450 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:18.446+0000 7f51f37fe640 1 --2- 172.21.15.46:0/3718732984 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f51f4107430 0x7f51f419eea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:18.450 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:18.446+0000 7f51f37fe640 1 -- 172.21.15.46:0/3718732984 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f51f419f510 con 0x7f51f41021d0 2024-03-31T23:11:18.450 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:18.446+0000 7f51f37fe640 1 --2- 172.21.15.46:0/3718732984 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f51f41021d0 0x7f51f4197940 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f51e400e9f0 tx=0x7f51e400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:11:18.450 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:18.446+0000 7f51f0ff9640 1 -- 172.21.15.46:0/3718732984 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51e400cd80 con 0x7f51f41021d0 2024-03-31T23:11:18.450 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:18.446+0000 7f51f0ff9640 1 -- 172.21.15.46:0/3718732984 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f51e40054d0 con 0x7f51f41021d0 2024-03-31T23:11:18.450 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:18.446+0000 7f51f9e84640 1 -- 172.21.15.46:0/3718732984 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f51f419f800 con 0x7f51f41021d0 2024-03-31T23:11:18.450 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:18.446+0000 7f51f0ff9640 1 -- 172.21.15.46:0/3718732984 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51e4010490 con 0x7f51f41021d0 2024-03-31T23:11:18.450 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:18.446+0000 7f51f9e84640 1 -- 172.21.15.46:0/3718732984 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f51f419fd00 con 0x7f51f41021d0 2024-03-31T23:11:18.450 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:18.447+0000 7f51f9e84640 1 -- 172.21.15.46:0/3718732984 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f51f40fd6b0 con 0x7f51f41021d0 2024-03-31T23:11:18.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:18.448+0000 7f51f0ff9640 1 -- 172.21.15.46:0/3718732984 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 18) ==== 137320+0+0 (secure 0 0 0) 0x7f51e4005020 con 0x7f51f41021d0 2024-03-31T23:11:18.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:18.449+0000 7f51f0ff9640 1 --2- 172.21.15.46:0/3718732984 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f51dc0a24a0 0x7f51dc0a4960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:18.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:18.449+0000 7f51f2ffd640 1 --2- 172.21.15.46:0/3718732984 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f51dc0a24a0 0x7f51dc0a4960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:18.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:18.449+0000 7f51f0ff9640 1 -- 172.21.15.46:0/3718732984 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9208+0+0 (secure 0 0 0) 0x7f51e40cee60 con 0x7f51f41021d0 2024-03-31T23:11:18.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:18.449+0000 7f51f2ffd640 1 --2- 172.21.15.46:0/3718732984 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f51dc0a24a0 0x7f51dc0a4960 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f51f4198e00 tx=0x7f51e8005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:11:18.454 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:18.454+0000 7f51f0ff9640 1 -- 172.21.15.46:0/3718732984 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f51e409bc50 con 0x7f51f41021d0 2024-03-31T23:11:18.611 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:11:18 smithi046 ceph-mon[32498]: pgmap v359: 129 pgs: 129 active+clean; 642 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 5 op/s; 0 B/s, 0 objects/s recovering 2024-03-31T23:11:18.611 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:11:18 smithi046 ceph-mon[32498]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs peering) 2024-03-31T23:11:18.612 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:11:18 smithi046 ceph-mon[32498]: Cluster is now healthy 2024-03-31T23:11:18.612 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:11:18 smithi046 ceph-mon[32498]: from='client.? 172.21.15.46:0/1967564836' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-31T23:11:18.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:18.640+0000 7f51f9e84640 1 -- 172.21.15.46:0/3718732984 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f51f4062350 con 0x7f51f41021d0 2024-03-31T23:11:18.642 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:18.641+0000 7f51f0ff9640 1 -- 172.21.15.46:0/3718732984 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v95) ==== 93+0+10 (secure 0 0 0) 0x7f51e409b550 con 0x7f51f41021d0 2024-03-31T23:11:18.642 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 1 2024-03-31T23:11:18.644 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:18.642+0000 7f51f9e84640 1 -- 172.21.15.46:0/3718732984 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f51dc0a24a0 msgr2=0x7f51dc0a4960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:18.644 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:18.642+0000 7f51f9e84640 1 --2- 172.21.15.46:0/3718732984 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f51dc0a24a0 0x7f51dc0a4960 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f51f4198e00 tx=0x7f51e8005f90 comp rx=0 tx=0).stop 2024-03-31T23:11:18.644 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:18.643+0000 7f51f9e84640 1 -- 172.21.15.46:0/3718732984 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f51f41021d0 msgr2=0x7f51f4197940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:18.644 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:18.643+0000 7f51f9e84640 1 --2- 172.21.15.46:0/3718732984 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f51f41021d0 0x7f51f4197940 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f51e400e9f0 tx=0x7f51e400edf0 comp rx=0 tx=0).stop 2024-03-31T23:11:18.644 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:18.643+0000 7f51f9e84640 1 -- 172.21.15.46:0/3718732984 shutdown_connections 2024-03-31T23:11:18.644 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:18.643+0000 7f51f9e84640 1 --2- 172.21.15.46:0/3718732984 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f51dc0a24a0 0x7f51dc0a4960 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:18.644 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:18.643+0000 7f51f9e84640 1 --2- 172.21.15.46:0/3718732984 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f51f4107430 0x7f51f419eea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:18.644 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:18.643+0000 7f51f9e84640 1 --2- 172.21.15.46:0/3718732984 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f51f4104b00 0x7f51f4197e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:18.644 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:18.643+0000 7f51f9e84640 1 --2- 172.21.15.46:0/3718732984 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f51f41021d0 0x7f51f4197940 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:18.644 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:18.643+0000 7f51f9e84640 1 -- 172.21.15.46:0/3718732984 >> 172.21.15.46:0/3718732984 conn(0x7f51f40fbbc0 msgr2=0x7f51f4102b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:11:18.644 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:18.643+0000 7f51f9e84640 1 -- 172.21.15.46:0/3718732984 shutdown_connections 2024-03-31T23:11:18.645 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:18.643+0000 7f51f9e84640 1 -- 172.21.15.46:0/3718732984 wait complete. 2024-03-31T23:11:18.655 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-31T23:11:18.764 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:18.761+0000 7f4bb676c640 1 -- 172.21.15.46:0/3598111957 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4bb0100650 msgr2=0x7f4bb0100a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:18.764 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:18.761+0000 7f4bb676c640 1 --2- 172.21.15.46:0/3598111957 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4bb0100650 0x7f4bb0100a70 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f4ba0009900 tx=0x7f4ba002bc30 comp rx=0 tx=0).stop 2024-03-31T23:11:18.764 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:18.761+0000 7f4bb676c640 1 -- 172.21.15.46:0/3598111957 shutdown_connections 2024-03-31T23:11:18.764 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:18.761+0000 7f4bb676c640 1 --2- 172.21.15.46:0/3598111957 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4bb0102200 0x7f4bb0108a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:18.764 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:18.761+0000 7f4bb676c640 1 --2- 172.21.15.46:0/3598111957 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4bb0101840 0x7f4bb0101cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:18.764 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:18.761+0000 7f4bb676c640 1 --2- 172.21.15.46:0/3598111957 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4bb0100650 0x7f4bb0100a70 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:18.764 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:18.761+0000 7f4bb676c640 1 -- 172.21.15.46:0/3598111957 >> 172.21.15.46:0/3598111957 conn(0x7f4bb00fbbe0 msgr2=0x7f4bb00fe000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:11:18.764 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:18.761+0000 7f4bb676c640 1 -- 172.21.15.46:0/3598111957 shutdown_connections 2024-03-31T23:11:18.764 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:18.761+0000 7f4bb676c640 1 -- 172.21.15.46:0/3598111957 wait complete. 2024-03-31T23:11:18.764 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:18.761+0000 7f4bb676c640 1 Processor -- start 2024-03-31T23:11:18.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:18.761+0000 7f4bb676c640 1 -- start start 2024-03-31T23:11:18.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:18.761+0000 7f4bb676c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4bb0100650 0x7f4bb0197a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:18.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:18.761+0000 7f4bb676c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4bb0101840 0x7f4bb0197fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:18.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:18.761+0000 7f4bb676c640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4bb0102200 0x7f4bb019eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:18.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:18.761+0000 7f4bb676c640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f4bb010c180 con 0x7f4bb0100650 2024-03-31T23:11:18.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:18.761+0000 7f4bb676c640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f4bb010c000 con 0x7f4bb0102200 2024-03-31T23:11:18.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:18.761+0000 7f4bb676c640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f4bb010c300 con 0x7f4bb0101840 2024-03-31T23:11:18.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:18.761+0000 7f4baffff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4bb0100650 0x7f4bb0197a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:18.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:18.761+0000 7f4baffff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4bb0100650 0x7f4bb0197a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:54784/0 (socket says 172.21.15.46:54784) 2024-03-31T23:11:18.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:18.761+0000 7f4baffff640 1 -- 172.21.15.46:0/3945736985 learned_addr learned my addr 172.21.15.46:0/3945736985 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:11:18.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:18.761+0000 7f4baf7fe640 1 --2- 172.21.15.46:0/3945736985 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4bb0101840 0x7f4bb0197fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:18.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:18.761+0000 7f4bb4d09640 1 --2- 172.21.15.46:0/3945736985 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4bb0102200 0x7f4bb019eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:18.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:18.761+0000 7f4baffff640 1 -- 172.21.15.46:0/3945736985 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4bb0101840 msgr2=0x7f4bb0197fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:18.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:18.761+0000 7f4baffff640 1 --2- 172.21.15.46:0/3945736985 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4bb0101840 0x7f4bb0197fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:18.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:18.761+0000 7f4baffff640 1 -- 172.21.15.46:0/3945736985 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4bb0102200 msgr2=0x7f4bb019eff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:18.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:18.761+0000 7f4baffff640 1 --2- 172.21.15.46:0/3945736985 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4bb0102200 0x7f4bb019eff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:18.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:18.761+0000 7f4baffff640 1 -- 172.21.15.46:0/3945736985 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4bb019f660 con 0x7f4bb0100650 2024-03-31T23:11:18.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:18.761+0000 7f4bb4d09640 1 --2- 172.21.15.46:0/3945736985 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4bb0102200 0x7f4bb019eff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T23:11:18.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:18.761+0000 7f4baffff640 1 --2- 172.21.15.46:0/3945736985 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4bb0100650 0x7f4bb0197a90 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f4ba0000c00 tx=0x7f4ba0005dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:11:18.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:18.761+0000 7f4bad7fa640 1 -- 172.21.15.46:0/3945736985 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ba0043020 con 0x7f4bb0100650 2024-03-31T23:11:18.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:18.761+0000 7f4bb676c640 1 -- 172.21.15.46:0/3945736985 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4bb019f8f0 con 0x7f4bb0100650 2024-03-31T23:11:18.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:18.761+0000 7f4bad7fa640 1 -- 172.21.15.46:0/3945736985 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4ba0004350 con 0x7f4bb0100650 2024-03-31T23:11:18.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:18.761+0000 7f4bad7fa640 1 -- 172.21.15.46:0/3945736985 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ba0003b80 con 0x7f4bb0100650 2024-03-31T23:11:18.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:18.761+0000 7f4bb676c640 1 -- 172.21.15.46:0/3945736985 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4bb019fe90 con 0x7f4bb0100650 2024-03-31T23:11:18.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:18.762+0000 7f4bb676c640 1 -- 172.21.15.46:0/3945736985 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4bb0068540 con 0x7f4bb0100650 2024-03-31T23:11:18.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:18.763+0000 7f4bad7fa640 1 -- 172.21.15.46:0/3945736985 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 18) ==== 137320+0+0 (secure 0 0 0) 0x7f4ba0046020 con 0x7f4bb0100650 2024-03-31T23:11:18.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:18.764+0000 7f4bad7fa640 1 --2- 172.21.15.46:0/3945736985 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f4b980a24a0 0x7f4b980a4960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:18.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:18.764+0000 7f4baf7fe640 1 --2- 172.21.15.46:0/3945736985 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f4b980a24a0 0x7f4b980a4960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:18.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:18.764+0000 7f4bad7fa640 1 -- 172.21.15.46:0/3945736985 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9208+0+0 (secure 0 0 0) 0x7f4ba00f1100 con 0x7f4bb0100650 2024-03-31T23:11:18.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:18.764+0000 7f4baf7fe640 1 --2- 172.21.15.46:0/3945736985 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f4b980a24a0 0x7f4b980a4960 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f4bb0198f50 tx=0x7f4ba4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:11:18.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:18.768+0000 7f4bad7fa640 1 -- 172.21.15.46:0/3945736985 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f4ba004d020 con 0x7f4bb0100650 2024-03-31T23:11:18.844 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:11:18 smithi142 ceph-mon[35271]: pgmap v359: 129 pgs: 129 active+clean; 642 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 5 op/s; 0 B/s, 0 objects/s recovering 2024-03-31T23:11:18.844 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:11:18 smithi142 ceph-mon[35271]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs peering) 2024-03-31T23:11:18.845 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:11:18 smithi142 ceph-mon[35271]: Cluster is now healthy 2024-03-31T23:11:18.845 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:11:18 smithi142 ceph-mon[35271]: from='client.? 172.21.15.46:0/1967564836' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-31T23:11:18.951 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:18.950+0000 7f4bb676c640 1 -- 172.21.15.46:0/3945736985 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f4bb01a01d0 con 0x7f4bb0100650 2024-03-31T23:11:18.951 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:18.950+0000 7f4bad7fa640 1 -- 172.21.15.46:0/3945736985 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v95) ==== 104+0+11 (secure 0 0 0) 0x7f4ba00bde90 con 0x7f4bb0100650 2024-03-31T23:11:18.951 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 32 2024-03-31T23:11:18.953 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:18.952+0000 7f4bb676c640 1 -- 172.21.15.46:0/3945736985 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f4b980a24a0 msgr2=0x7f4b980a4960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:18.953 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:18.952+0000 7f4bb676c640 1 --2- 172.21.15.46:0/3945736985 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f4b980a24a0 0x7f4b980a4960 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f4bb0198f50 tx=0x7f4ba4005f90 comp rx=0 tx=0).stop 2024-03-31T23:11:18.953 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:18.952+0000 7f4bb676c640 1 -- 172.21.15.46:0/3945736985 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4bb0100650 msgr2=0x7f4bb0197a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:18.953 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:18.952+0000 7f4bb676c640 1 --2- 172.21.15.46:0/3945736985 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4bb0100650 0x7f4bb0197a90 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f4ba0000c00 tx=0x7f4ba0005dc0 comp rx=0 tx=0).stop 2024-03-31T23:11:18.953 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:18.952+0000 7f4bb676c640 1 -- 172.21.15.46:0/3945736985 shutdown_connections 2024-03-31T23:11:18.953 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:18.952+0000 7f4bb676c640 1 --2- 172.21.15.46:0/3945736985 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f4b980a24a0 0x7f4b980a4960 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:18.953 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:18.952+0000 7f4bb676c640 1 --2- 172.21.15.46:0/3945736985 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4bb0102200 0x7f4bb019eff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:18.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:18.952+0000 7f4bb676c640 1 --2- 172.21.15.46:0/3945736985 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4bb0101840 0x7f4bb0197fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:18.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:18.952+0000 7f4bb676c640 1 --2- 172.21.15.46:0/3945736985 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4bb0100650 0x7f4bb0197a90 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:18.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:18.952+0000 7f4bb676c640 1 -- 172.21.15.46:0/3945736985 >> 172.21.15.46:0/3945736985 conn(0x7f4bb00fbbe0 msgr2=0x7f4bb00689a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:11:18.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:18.952+0000 7f4bb676c640 1 -- 172.21.15.46:0/3945736985 shutdown_connections 2024-03-31T23:11:18.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:18.952+0000 7f4bb676c640 1 -- 172.21.15.46:0/3945736985 wait complete. 2024-03-31T23:11:18.964 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-31T23:11:19.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:11:18 smithi162 ceph-mon[36237]: pgmap v359: 129 pgs: 129 active+clean; 642 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 5 op/s; 0 B/s, 0 objects/s recovering 2024-03-31T23:11:19.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:11:18 smithi162 ceph-mon[36237]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs peering) 2024-03-31T23:11:19.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:11:18 smithi162 ceph-mon[36237]: Cluster is now healthy 2024-03-31T23:11:19.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:11:18 smithi162 ceph-mon[36237]: from='client.? 172.21.15.46:0/1967564836' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-31T23:11:19.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:19.069+0000 7f6aede2a640 1 -- 172.21.15.46:0/1721487244 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6ae8100650 msgr2=0x7f6ae8100a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:19.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:19.069+0000 7f6aede2a640 1 --2- 172.21.15.46:0/1721487244 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6ae8100650 0x7f6ae8100a70 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f6ad8009900 tx=0x7f6ad802bc30 comp rx=0 tx=0).stop 2024-03-31T23:11:19.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:19.069+0000 7f6aede2a640 1 -- 172.21.15.46:0/1721487244 shutdown_connections 2024-03-31T23:11:19.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:19.069+0000 7f6aede2a640 1 --2- 172.21.15.46:0/1721487244 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6ae8102200 0x7f6ae8108a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:19.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:19.069+0000 7f6aede2a640 1 --2- 172.21.15.46:0/1721487244 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6ae8101840 0x7f6ae8101cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:19.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:19.069+0000 7f6aede2a640 1 --2- 172.21.15.46:0/1721487244 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6ae8100650 0x7f6ae8100a70 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:19.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:19.069+0000 7f6aede2a640 1 -- 172.21.15.46:0/1721487244 >> 172.21.15.46:0/1721487244 conn(0x7f6ae80fbbc0 msgr2=0x7f6ae80fe000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:11:19.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:19.070+0000 7f6aede2a640 1 -- 172.21.15.46:0/1721487244 shutdown_connections 2024-03-31T23:11:19.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:19.070+0000 7f6aede2a640 1 -- 172.21.15.46:0/1721487244 wait complete. 2024-03-31T23:11:19.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:19.070+0000 7f6aede2a640 1 Processor -- start 2024-03-31T23:11:19.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:19.070+0000 7f6aede2a640 1 -- start start 2024-03-31T23:11:19.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:19.070+0000 7f6aede2a640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6ae8100650 0x7f6ae81979f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:19.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:19.070+0000 7f6aede2a640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6ae8101840 0x7f6ae8197f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:19.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:19.070+0000 7f6aede2a640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6ae8102200 0x7f6ae819efa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:19.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:19.070+0000 7f6aede2a640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f6ae810c170 con 0x7f6ae8100650 2024-03-31T23:11:19.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:19.070+0000 7f6aede2a640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f6ae810bff0 con 0x7f6ae8101840 2024-03-31T23:11:19.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:19.070+0000 7f6aede2a640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f6ae810c2f0 con 0x7f6ae8102200 2024-03-31T23:11:19.076 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:19.070+0000 7f6ae77fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6ae8100650 0x7f6ae81979f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:19.076 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:19.071+0000 7f6ae77fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6ae8100650 0x7f6ae81979f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:54800/0 (socket says 172.21.15.46:54800) 2024-03-31T23:11:19.076 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:19.071+0000 7f6ae77fe640 1 -- 172.21.15.46:0/2841507303 learned_addr learned my addr 172.21.15.46:0/2841507303 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:11:19.076 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:19.071+0000 7f6ae6ffd640 1 --2- 172.21.15.46:0/2841507303 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6ae8101840 0x7f6ae8197f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:19.076 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:19.071+0000 7f6ae7fff640 1 --2- 172.21.15.46:0/2841507303 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6ae8102200 0x7f6ae819efa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:19.076 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:19.071+0000 7f6ae77fe640 1 -- 172.21.15.46:0/2841507303 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6ae8102200 msgr2=0x7f6ae819efa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:19.076 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:19.071+0000 7f6ae77fe640 1 --2- 172.21.15.46:0/2841507303 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6ae8102200 0x7f6ae819efa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:19.076 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:19.071+0000 7f6ae77fe640 1 -- 172.21.15.46:0/2841507303 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6ae8101840 msgr2=0x7f6ae8197f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:19.076 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:19.071+0000 7f6ae77fe640 1 --2- 172.21.15.46:0/2841507303 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6ae8101840 0x7f6ae8197f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:19.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:19.071+0000 7f6ae77fe640 1 -- 172.21.15.46:0/2841507303 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ae819f610 con 0x7f6ae8100650 2024-03-31T23:11:19.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:19.071+0000 7f6ae77fe640 1 --2- 172.21.15.46:0/2841507303 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6ae8100650 0x7f6ae81979f0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f6ad8000c00 tx=0x7f6ad8005dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:11:19.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:19.071+0000 7f6ae4ff9640 1 -- 172.21.15.46:0/2841507303 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ad8043020 con 0x7f6ae8100650 2024-03-31T23:11:19.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:19.071+0000 7f6ae4ff9640 1 -- 172.21.15.46:0/2841507303 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6ad8004350 con 0x7f6ae8100650 2024-03-31T23:11:19.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:19.071+0000 7f6aede2a640 1 -- 172.21.15.46:0/2841507303 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6ae819f8a0 con 0x7f6ae8100650 2024-03-31T23:11:19.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:19.071+0000 7f6ae4ff9640 1 -- 172.21.15.46:0/2841507303 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ad8003b80 con 0x7f6ae8100650 2024-03-31T23:11:19.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:19.071+0000 7f6aede2a640 1 -- 172.21.15.46:0/2841507303 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6ae819fdf0 con 0x7f6ae8100650 2024-03-31T23:11:19.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:19.072+0000 7f6aede2a640 1 -- 172.21.15.46:0/2841507303 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6ae819fff0 con 0x7f6ae8100650 2024-03-31T23:11:19.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:19.074+0000 7f6ae4ff9640 1 -- 172.21.15.46:0/2841507303 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 18) ==== 137320+0+0 (secure 0 0 0) 0x7f6ad8046020 con 0x7f6ae8100650 2024-03-31T23:11:19.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:19.076+0000 7f6ae4ff9640 1 --2- 172.21.15.46:0/2841507303 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f6ad00a24a0 0x7f6ad00a4960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:19.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:19.076+0000 7f6ae6ffd640 1 --2- 172.21.15.46:0/2841507303 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f6ad00a24a0 0x7f6ad00a4960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:19.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:19.076+0000 7f6ae4ff9640 1 -- 172.21.15.46:0/2841507303 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9208+0+0 (secure 0 0 0) 0x7f6ad80f1040 con 0x7f6ae8100650 2024-03-31T23:11:19.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:19.076+0000 7f6ae6ffd640 1 --2- 172.21.15.46:0/2841507303 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f6ad00a24a0 0x7f6ad00a4960 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f6ae8198f00 tx=0x7f6adc005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:11:19.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:19.078+0000 7f6ae4ff9640 1 -- 172.21.15.46:0/2841507303 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f6ad804d020 con 0x7f6ae8100650 2024-03-31T23:11:19.264 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:19.263+0000 7f6aede2a640 1 -- 172.21.15.46:0/2841507303 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f6ae81a01c0 con 0x7f6ae8100650 2024-03-31T23:11:19.264 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:19.263+0000 7f6ae4ff9640 1 -- 172.21.15.46:0/2841507303 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v95) ==== 100+0+11 (secure 0 0 0) 0x7f6ad80bde30 con 0x7f6ae8100650 2024-03-31T23:11:19.265 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 64 2024-03-31T23:11:19.267 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:19.265+0000 7f6aede2a640 1 -- 172.21.15.46:0/2841507303 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f6ad00a24a0 msgr2=0x7f6ad00a4960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:19.267 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:19.265+0000 7f6aede2a640 1 --2- 172.21.15.46:0/2841507303 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f6ad00a24a0 0x7f6ad00a4960 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f6ae8198f00 tx=0x7f6adc005f90 comp rx=0 tx=0).stop 2024-03-31T23:11:19.267 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:19.265+0000 7f6aede2a640 1 -- 172.21.15.46:0/2841507303 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6ae8100650 msgr2=0x7f6ae81979f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:19.267 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:19.265+0000 7f6aede2a640 1 --2- 172.21.15.46:0/2841507303 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6ae8100650 0x7f6ae81979f0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f6ad8000c00 tx=0x7f6ad8005dc0 comp rx=0 tx=0).stop 2024-03-31T23:11:19.267 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:19.265+0000 7f6aede2a640 1 -- 172.21.15.46:0/2841507303 shutdown_connections 2024-03-31T23:11:19.267 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:19.265+0000 7f6aede2a640 1 --2- 172.21.15.46:0/2841507303 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f6ad00a24a0 0x7f6ad00a4960 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:19.267 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:19.265+0000 7f6aede2a640 1 --2- 172.21.15.46:0/2841507303 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6ae8102200 0x7f6ae819efa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:19.267 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:19.265+0000 7f6aede2a640 1 --2- 172.21.15.46:0/2841507303 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6ae8101840 0x7f6ae8197f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:19.267 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:19.265+0000 7f6aede2a640 1 --2- 172.21.15.46:0/2841507303 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6ae8100650 0x7f6ae81979f0 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:19.268 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:19.265+0000 7f6aede2a640 1 -- 172.21.15.46:0/2841507303 >> 172.21.15.46:0/2841507303 conn(0x7f6ae80fbbc0 msgr2=0x7f6ae8068920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:11:19.268 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:19.266+0000 7f6aede2a640 1 -- 172.21.15.46:0/2841507303 shutdown_connections 2024-03-31T23:11:19.268 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:19.266+0000 7f6aede2a640 1 -- 172.21.15.46:0/2841507303 wait complete. 2024-03-31T23:11:19.277 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-31T23:11:19.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:19.382+0000 7f6ec109d640 1 -- 172.21.15.46:0/2653969863 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6ebc103320 msgr2=0x7f6ebc109bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:19.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:19.382+0000 7f6ec109d640 1 --2- 172.21.15.46:0/2653969863 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6ebc103320 0x7f6ebc109bb0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f6eb4009ca0 tx=0x7f6eb402c0f0 comp rx=0 tx=0).stop 2024-03-31T23:11:19.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:19.382+0000 7f6ec109d640 1 -- 172.21.15.46:0/2653969863 shutdown_connections 2024-03-31T23:11:19.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:19.382+0000 7f6ec109d640 1 --2- 172.21.15.46:0/2653969863 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6ebc103320 0x7f6ebc109bb0 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:19.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:19.382+0000 7f6ec109d640 1 --2- 172.21.15.46:0/2653969863 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6ebc102960 0x7f6ebc102de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:19.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:19.382+0000 7f6ec109d640 1 --2- 172.21.15.46:0/2653969863 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6ebc101770 0x7f6ebc101b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:19.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:19.382+0000 7f6ec109d640 1 -- 172.21.15.46:0/2653969863 >> 172.21.15.46:0/2653969863 conn(0x7f6ebc0fcce0 msgr2=0x7f6ebc0ff120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:11:19.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:19.382+0000 7f6ec109d640 1 -- 172.21.15.46:0/2653969863 shutdown_connections 2024-03-31T23:11:19.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:19.382+0000 7f6ec109d640 1 -- 172.21.15.46:0/2653969863 wait complete. 2024-03-31T23:11:19.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:19.383+0000 7f6ec109d640 1 Processor -- start 2024-03-31T23:11:19.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:19.383+0000 7f6ec109d640 1 -- start start 2024-03-31T23:11:19.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:19.383+0000 7f6ec109d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6ebc101770 0x7f6ebc075500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:19.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:19.383+0000 7f6ec109d640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6ebc102960 0x7f6ebc073b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:19.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:19.383+0000 7f6ec109d640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6ebc103320 0x7f6ebc074080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:19.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:19.383+0000 7f6ec109d640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f6ebc10d280 con 0x7f6ebc101770 2024-03-31T23:11:19.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:19.383+0000 7f6ec109d640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f6ebc10d100 con 0x7f6ebc102960 2024-03-31T23:11:19.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:19.383+0000 7f6ec109d640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f6ebc10d400 con 0x7f6ebc103320 2024-03-31T23:11:19.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:19.383+0000 7f6ebad9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6ebc101770 0x7f6ebc075500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:19.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:19.384+0000 7f6ebad9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6ebc101770 0x7f6ebc075500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:54814/0 (socket says 172.21.15.46:54814) 2024-03-31T23:11:19.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:19.384+0000 7f6eb3fff640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6ebc102960 0x7f6ebc073b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:19.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:19.384+0000 7f6ebad9d640 1 -- 172.21.15.46:0/1046600875 learned_addr learned my addr 172.21.15.46:0/1046600875 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:11:19.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:19.384+0000 7f6ebb59e640 1 --2- 172.21.15.46:0/1046600875 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6ebc103320 0x7f6ebc074080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:19.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:19.384+0000 7f6ebad9d640 1 -- 172.21.15.46:0/1046600875 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6ebc103320 msgr2=0x7f6ebc074080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:19.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:19.384+0000 7f6ebad9d640 1 --2- 172.21.15.46:0/1046600875 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6ebc103320 0x7f6ebc074080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:19.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:19.384+0000 7f6ebad9d640 1 -- 172.21.15.46:0/1046600875 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6ebc102960 msgr2=0x7f6ebc073b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:19.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:19.384+0000 7f6ebad9d640 1 --2- 172.21.15.46:0/1046600875 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6ebc102960 0x7f6ebc073b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:19.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:19.384+0000 7f6ebad9d640 1 -- 172.21.15.46:0/1046600875 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ebc074910 con 0x7f6ebc101770 2024-03-31T23:11:19.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:19.384+0000 7f6ebad9d640 1 --2- 172.21.15.46:0/1046600875 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6ebc101770 0x7f6ebc075500 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f6eac00e9c0 tx=0x7f6eac00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:11:19.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:19.384+0000 7f6eb8d99640 1 -- 172.21.15.46:0/1046600875 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6eac00cde0 con 0x7f6ebc101770 2024-03-31T23:11:19.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:19.384+0000 7f6eb8d99640 1 -- 172.21.15.46:0/1046600875 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6eac0108e0 con 0x7f6ebc101770 2024-03-31T23:11:19.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:19.384+0000 7f6ec109d640 1 -- 172.21.15.46:0/1046600875 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6ebc1a1a50 con 0x7f6ebc101770 2024-03-31T23:11:19.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:19.384+0000 7f6eb8d99640 1 -- 172.21.15.46:0/1046600875 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6eac018950 con 0x7f6ebc101770 2024-03-31T23:11:19.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:19.384+0000 7f6ec109d640 1 -- 172.21.15.46:0/1046600875 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6ebc1a1fa0 con 0x7f6ebc101770 2024-03-31T23:11:19.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:19.385+0000 7f6ec109d640 1 -- 172.21.15.46:0/1046600875 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6ebc068990 con 0x7f6ebc101770 2024-03-31T23:11:19.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:19.386+0000 7f6eb8d99640 1 -- 172.21.15.46:0/1046600875 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 18) ==== 137320+0+0 (secure 0 0 0) 0x7f6eac010a90 con 0x7f6ebc101770 2024-03-31T23:11:19.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:19.387+0000 7f6eb8d99640 1 --2- 172.21.15.46:0/1046600875 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f6ea80a24a0 0x7f6ea80a4960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:19.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:19.387+0000 7f6eb8d99640 1 -- 172.21.15.46:0/1046600875 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9208+0+0 (secure 0 0 0) 0x7f6eac0cf040 con 0x7f6ebc101770 2024-03-31T23:11:19.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:19.387+0000 7f6eb3fff640 1 --2- 172.21.15.46:0/1046600875 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f6ea80a24a0 0x7f6ea80a4960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:19.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:19.387+0000 7f6eb3fff640 1 --2- 172.21.15.46:0/1046600875 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f6ea80a24a0 0x7f6ea80a4960 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f6ebc075270 tx=0x7f6ea4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:11:19.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:19.391+0000 7f6eb8d99640 1 -- 172.21.15.46:0/1046600875 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f6eac053280 con 0x7f6ebc101770 2024-03-31T23:11:19.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:19.577+0000 7f6ec109d640 1 -- 172.21.15.46:0/1046600875 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f6ebc062350 con 0x7f6ebc101770 2024-03-31T23:11:19.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:19.577+0000 7f6eb8d99640 1 -- 172.21.15.46:0/1046600875 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v95) ==== 103+0+11 (secure 0 0 0) 0x7f6eac09be30 con 0x7f6ebc101770 2024-03-31T23:11:19.577 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 32 2024-03-31T23:11:19.580 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:19.579+0000 7f6ec109d640 1 -- 172.21.15.46:0/1046600875 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f6ea80a24a0 msgr2=0x7f6ea80a4960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:19.580 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:19.579+0000 7f6ec109d640 1 --2- 172.21.15.46:0/1046600875 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f6ea80a24a0 0x7f6ea80a4960 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f6ebc075270 tx=0x7f6ea4005f90 comp rx=0 tx=0).stop 2024-03-31T23:11:19.580 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:19.579+0000 7f6ec109d640 1 -- 172.21.15.46:0/1046600875 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6ebc101770 msgr2=0x7f6ebc075500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:19.580 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:19.579+0000 7f6ec109d640 1 --2- 172.21.15.46:0/1046600875 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6ebc101770 0x7f6ebc075500 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f6eac00e9c0 tx=0x7f6eac00ee80 comp rx=0 tx=0).stop 2024-03-31T23:11:19.580 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:19.579+0000 7f6ec109d640 1 -- 172.21.15.46:0/1046600875 shutdown_connections 2024-03-31T23:11:19.580 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:19.579+0000 7f6ec109d640 1 --2- 172.21.15.46:0/1046600875 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f6ea80a24a0 0x7f6ea80a4960 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:19.580 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:19.579+0000 7f6ec109d640 1 --2- 172.21.15.46:0/1046600875 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6ebc103320 0x7f6ebc074080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:19.580 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:19.579+0000 7f6ec109d640 1 --2- 172.21.15.46:0/1046600875 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6ebc102960 0x7f6ebc073b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:19.580 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:19.579+0000 7f6ec109d640 1 --2- 172.21.15.46:0/1046600875 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6ebc101770 0x7f6ebc075500 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:19.580 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:19.579+0000 7f6ec109d640 1 -- 172.21.15.46:0/1046600875 >> 172.21.15.46:0/1046600875 conn(0x7f6ebc0fcce0 msgr2=0x7f6ebc105c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:11:19.580 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:19.579+0000 7f6ec109d640 1 -- 172.21.15.46:0/1046600875 shutdown_connections 2024-03-31T23:11:19.580 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:19.579+0000 7f6ec109d640 1 -- 172.21.15.46:0/1046600875 wait complete. 2024-03-31T23:11:19.591 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-31T23:11:19.700 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:19.696+0000 7fd14ec07640 1 -- 172.21.15.46:0/3080148439 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd148100670 msgr2=0x7fd148100a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:19.700 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:19.696+0000 7fd14ec07640 1 --2- 172.21.15.46:0/3080148439 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd148100670 0x7fd148100a90 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7fd13c009900 tx=0x7fd13c02bc30 comp rx=0 tx=0).stop 2024-03-31T23:11:19.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:19.696+0000 7fd14ec07640 1 -- 172.21.15.46:0/3080148439 shutdown_connections 2024-03-31T23:11:19.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:19.696+0000 7fd14ec07640 1 --2- 172.21.15.46:0/3080148439 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd148102220 0x7fd148108ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:19.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:19.696+0000 7fd14ec07640 1 --2- 172.21.15.46:0/3080148439 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd148101860 0x7fd148101ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:19.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:19.696+0000 7fd14ec07640 1 --2- 172.21.15.46:0/3080148439 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd148100670 0x7fd148100a90 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:19.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:19.696+0000 7fd14ec07640 1 -- 172.21.15.46:0/3080148439 >> 172.21.15.46:0/3080148439 conn(0x7fd1480fbbe0 msgr2=0x7fd1480fe020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:11:19.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:19.696+0000 7fd14ec07640 1 -- 172.21.15.46:0/3080148439 shutdown_connections 2024-03-31T23:11:19.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:19.696+0000 7fd14ec07640 1 -- 172.21.15.46:0/3080148439 wait complete. 2024-03-31T23:11:19.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:19.697+0000 7fd14ec07640 1 Processor -- start 2024-03-31T23:11:19.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:19.697+0000 7fd14ec07640 1 -- start start 2024-03-31T23:11:19.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:19.697+0000 7fd14ec07640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd148100670 0x7fd148196940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:19.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:19.697+0000 7fd14ec07640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd148101860 0x7fd148196e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:19.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:19.697+0000 7fd14ec07640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd148102220 0x7fd14819def0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:19.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:19.697+0000 7fd14ec07640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fd14810c190 con 0x7fd148100670 2024-03-31T23:11:19.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:19.697+0000 7fd14ec07640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fd14810c010 con 0x7fd148102220 2024-03-31T23:11:19.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:19.697+0000 7fd14ec07640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fd14810c310 con 0x7fd148101860 2024-03-31T23:11:19.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:19.697+0000 7fd14c9a3640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd148100670 0x7fd148196940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:19.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:19.697+0000 7fd14c9a3640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd148100670 0x7fd148196940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:54832/0 (socket says 172.21.15.46:54832) 2024-03-31T23:11:19.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:19.697+0000 7fd14c9a3640 1 -- 172.21.15.46:0/2989957989 learned_addr learned my addr 172.21.15.46:0/2989957989 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:11:19.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:19.697+0000 7fd147fff640 1 --2- 172.21.15.46:0/2989957989 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd148101860 0x7fd148196e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:19.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:19.697+0000 7fd14d1a4640 1 --2- 172.21.15.46:0/2989957989 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd148102220 0x7fd14819def0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:19.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:19.697+0000 7fd14c9a3640 1 -- 172.21.15.46:0/2989957989 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd148101860 msgr2=0x7fd148196e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:19.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:19.697+0000 7fd14c9a3640 1 --2- 172.21.15.46:0/2989957989 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd148101860 0x7fd148196e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:19.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:19.697+0000 7fd14c9a3640 1 -- 172.21.15.46:0/2989957989 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd148102220 msgr2=0x7fd14819def0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:19.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:19.697+0000 7fd14c9a3640 1 --2- 172.21.15.46:0/2989957989 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd148102220 0x7fd14819def0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:19.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:19.697+0000 7fd14c9a3640 1 -- 172.21.15.46:0/2989957989 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd14819e560 con 0x7fd148100670 2024-03-31T23:11:19.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:19.698+0000 7fd14c9a3640 1 --2- 172.21.15.46:0/2989957989 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd148100670 0x7fd148196940 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7fd13c000c00 tx=0x7fd13c005dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:11:19.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:19.698+0000 7fd145ffb640 1 -- 172.21.15.46:0/2989957989 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd13c043020 con 0x7fd148100670 2024-03-31T23:11:19.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:19.698+0000 7fd14ec07640 1 -- 172.21.15.46:0/2989957989 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd14819e7f0 con 0x7fd148100670 2024-03-31T23:11:19.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:19.698+0000 7fd145ffb640 1 -- 172.21.15.46:0/2989957989 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd13c004350 con 0x7fd148100670 2024-03-31T23:11:19.703 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:19.698+0000 7fd145ffb640 1 -- 172.21.15.46:0/2989957989 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd13c003b10 con 0x7fd148100670 2024-03-31T23:11:19.703 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:19.698+0000 7fd14ec07640 1 -- 172.21.15.46:0/2989957989 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd14819ed40 con 0x7fd148100670 2024-03-31T23:11:19.703 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:19.699+0000 7fd14ec07640 1 -- 172.21.15.46:0/2989957989 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd148107a80 con 0x7fd148100670 2024-03-31T23:11:19.703 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:19.700+0000 7fd145ffb640 1 -- 172.21.15.46:0/2989957989 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 18) ==== 137320+0+0 (secure 0 0 0) 0x7fd13c046020 con 0x7fd148100670 2024-03-31T23:11:19.703 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:19.701+0000 7fd145ffb640 1 --2- 172.21.15.46:0/2989957989 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fd1380a24a0 0x7fd1380a4960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:19.703 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:19.701+0000 7fd147fff640 1 --2- 172.21.15.46:0/2989957989 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fd1380a24a0 0x7fd1380a4960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:19.704 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:19.701+0000 7fd145ffb640 1 -- 172.21.15.46:0/2989957989 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9208+0+0 (secure 0 0 0) 0x7fd13c0f0bf0 con 0x7fd148100670 2024-03-31T23:11:19.704 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:19.702+0000 7fd147fff640 1 --2- 172.21.15.46:0/2989957989 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fd1380a24a0 0x7fd1380a4960 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fd148197e50 tx=0x7fd134005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:11:19.706 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:19.706+0000 7fd145ffb640 1 -- 172.21.15.46:0/2989957989 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fd13c04d020 con 0x7fd148100670 2024-03-31T23:11:19.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:11:19 smithi046 ceph-mon[32498]: from='client.? 172.21.15.46:0/3718732984' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-31T23:11:19.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:11:19 smithi046 ceph-mon[32498]: from='client.? 172.21.15.46:0/3945736985' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-31T23:11:19.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:11:19 smithi046 ceph-mon[32498]: from='client.? 172.21.15.46:0/2841507303' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-31T23:11:19.926 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:19.926+0000 7fd14ec07640 1 -- 172.21.15.46:0/2989957989 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd14802d170 con 0x7fd148100670 2024-03-31T23:11:19.926 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:19.926+0000 7fd145ffb640 1 -- 172.21.15.46:0/2989957989 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+11882 (secure 0 0 0) 0x7fd13c0bd9e0 con 0x7fd148100670 2024-03-31T23:11:19.926 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-31T23:11:19.927 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/2169018547","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":2169018547},{"type":"v1","addr":"172.21.15.46:6835","nonce":2169018547}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6837/1069387756","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6836","nonce":1069387756},{"type":"v1","addr":"172.21.15.46:6837","nonce":1069387756}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14634,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6839/3915337844","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6838","nonce":3915337844},{"type":"v1","addr":"172.21.15.46:6839","nonce":3915337844}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14646,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6841/897767145","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6840","nonce":897767145},{"type":"v1","addr":"172.21.15.46:6841","nonce":897767145}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24415,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6833/1746950966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6832","nonce":1746950966},{"type":"v1","addr":"172.21.15.162:6833","nonce":1746950966}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24422,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6833/3394739181","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6832","nonce":3394739181},{"type":"v1","addr":"172.21.15.142:6833","nonce":3394739181}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":24,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-31T23:09:11.009090+0000","modified":"2024-03-31T23:09:31.392282+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24515,"mds_1":24482,"mds_2":24451},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24442":{"gid":24442,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.162:6835/1957208984","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6834","nonce":1957208984},{"type":"v1","addr":"172.21.15.162:6835","nonce":1957208984}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24451":{"gid":24451,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":11,"addr":"172.21.15.162:6837/1371886876","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6836","nonce":1371886876},{"type":"v1","addr":"172.21.15.162:6837","nonce":1371886876}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24455":{"gid":24455,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.142:6835/212637431","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6834","nonce":212637431},{"type":"v1","addr":"172.21.15.142:6835","nonce":212637431}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24478":{"gid":24478,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.162:6839/1976275826","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6838","nonce":1976275826},{"type":"v1","addr":"172.21.15.162:6839","nonce":1976275826}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24482":{"gid":24482,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":10,"addr":"172.21.15.142:6837/2559258399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6836","nonce":2559258399},{"type":"v1","addr":"172.21.15.142:6837","nonce":2559258399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24515":{"gid":24515,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.142:6839/2519143918","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6838","nonce":2519143918},{"type":"v1","addr":"172.21.15.142:6839","nonce":2519143918}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24515,"qdb_cluster":[24451,24482,24515]},"id":1}]} 2024-03-31T23:11:19.927 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 24 2024-03-31T23:11:19.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:19.928+0000 7fd14ec07640 1 -- 172.21.15.46:0/2989957989 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fd1380a24a0 msgr2=0x7fd1380a4960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:19.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:19.928+0000 7fd14ec07640 1 --2- 172.21.15.46:0/2989957989 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fd1380a24a0 0x7fd1380a4960 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fd148197e50 tx=0x7fd134005f90 comp rx=0 tx=0).stop 2024-03-31T23:11:19.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:19.928+0000 7fd14ec07640 1 -- 172.21.15.46:0/2989957989 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd148100670 msgr2=0x7fd148196940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:19.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:19.928+0000 7fd14ec07640 1 --2- 172.21.15.46:0/2989957989 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd148100670 0x7fd148196940 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7fd13c000c00 tx=0x7fd13c005dc0 comp rx=0 tx=0).stop 2024-03-31T23:11:19.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:19.928+0000 7fd14ec07640 1 -- 172.21.15.46:0/2989957989 shutdown_connections 2024-03-31T23:11:19.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:19.928+0000 7fd14ec07640 1 --2- 172.21.15.46:0/2989957989 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fd1380a24a0 0x7fd1380a4960 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:19.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:19.928+0000 7fd14ec07640 1 --2- 172.21.15.46:0/2989957989 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd148102220 0x7fd14819def0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:19.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:19.928+0000 7fd14ec07640 1 --2- 172.21.15.46:0/2989957989 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd148101860 0x7fd148196e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:19.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:19.928+0000 7fd14ec07640 1 --2- 172.21.15.46:0/2989957989 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd148100670 0x7fd148196940 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:19.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:19.928+0000 7fd14ec07640 1 -- 172.21.15.46:0/2989957989 >> 172.21.15.46:0/2989957989 conn(0x7fd1480fbbe0 msgr2=0x7fd148104c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:11:19.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:19.928+0000 7fd14ec07640 1 -- 172.21.15.46:0/2989957989 shutdown_connections 2024-03-31T23:11:19.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:19.929+0000 7fd14ec07640 1 -- 172.21.15.46:0/2989957989 wait complete. 2024-03-31T23:11:19.940 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-31T23:11:19.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:11:19 smithi142 ceph-mon[35271]: from='client.? 172.21.15.46:0/3718732984' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-31T23:11:19.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:11:19 smithi142 ceph-mon[35271]: from='client.? 172.21.15.46:0/3945736985' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-31T23:11:19.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:11:19 smithi142 ceph-mon[35271]: from='client.? 172.21.15.46:0/2841507303' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-31T23:11:20.047 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:20.043+0000 7f2221556640 1 -- 172.21.15.46:0/2808791878 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f221c101860 msgr2=0x7f221c101ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:20.047 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:20.043+0000 7f2221556640 1 --2- 172.21.15.46:0/2808791878 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f221c101860 0x7f221c101ce0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f2204009900 tx=0x7f220402bc30 comp rx=0 tx=0).stop 2024-03-31T23:11:20.047 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:20.043+0000 7f2221556640 1 -- 172.21.15.46:0/2808791878 shutdown_connections 2024-03-31T23:11:20.047 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:20.043+0000 7f2221556640 1 --2- 172.21.15.46:0/2808791878 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f221c102220 0x7f221c108ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:20.047 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:20.043+0000 7f2221556640 1 --2- 172.21.15.46:0/2808791878 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f221c101860 0x7f221c101ce0 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:20.047 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:20.043+0000 7f2221556640 1 --2- 172.21.15.46:0/2808791878 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f221c100670 0x7f221c100a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:20.047 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:20.043+0000 7f2221556640 1 -- 172.21.15.46:0/2808791878 >> 172.21.15.46:0/2808791878 conn(0x7f221c0fbbe0 msgr2=0x7f221c0fe020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:11:20.047 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:20.043+0000 7f2221556640 1 -- 172.21.15.46:0/2808791878 shutdown_connections 2024-03-31T23:11:20.047 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:20.043+0000 7f2221556640 1 -- 172.21.15.46:0/2808791878 wait complete. 2024-03-31T23:11:20.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:20.044+0000 7f2221556640 1 Processor -- start 2024-03-31T23:11:20.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:20.044+0000 7f2221556640 1 -- start start 2024-03-31T23:11:20.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:20.044+0000 7f2221556640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f221c100670 0x7f221c1979d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:20.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:20.044+0000 7f2221556640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f221c101860 0x7f221c197f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:20.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:20.044+0000 7f2221556640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f221c102220 0x7f221c19ef80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:20.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:20.044+0000 7f2221556640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f221c10d2c0 con 0x7f221c102220 2024-03-31T23:11:20.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:20.044+0000 7f2221556640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f221c10d140 con 0x7f221c100670 2024-03-31T23:11:20.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:20.044+0000 7f2221556640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f221c10d440 con 0x7f221c101860 2024-03-31T23:11:20.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:20.044+0000 7f221b7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f221c102220 0x7f221c19ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:20.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:20.044+0000 7f221b7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f221c102220 0x7f221c19ef80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:54836/0 (socket says 172.21.15.46:54836) 2024-03-31T23:11:20.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:20.044+0000 7f221b7fe640 1 -- 172.21.15.46:0/1906361167 learned_addr learned my addr 172.21.15.46:0/1906361167 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:11:20.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:20.044+0000 7f221affd640 1 --2- 172.21.15.46:0/1906361167 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f221c100670 0x7f221c1979d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:20.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:20.044+0000 7f221a7fc640 1 --2- 172.21.15.46:0/1906361167 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f221c101860 0x7f221c197f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:20.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:20.045+0000 7f221b7fe640 1 -- 172.21.15.46:0/1906361167 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f221c101860 msgr2=0x7f221c197f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:20.049 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:20.045+0000 7f221b7fe640 1 --2- 172.21.15.46:0/1906361167 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f221c101860 0x7f221c197f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:20.049 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:20.045+0000 7f221b7fe640 1 -- 172.21.15.46:0/1906361167 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f221c100670 msgr2=0x7f221c1979d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:20.049 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:20.045+0000 7f221b7fe640 1 --2- 172.21.15.46:0/1906361167 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f221c100670 0x7f221c1979d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:20.049 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:20.045+0000 7f221b7fe640 1 -- 172.21.15.46:0/1906361167 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f221c19f5f0 con 0x7f221c102220 2024-03-31T23:11:20.049 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:20.045+0000 7f221a7fc640 1 --2- 172.21.15.46:0/1906361167 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f221c101860 0x7f221c197f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T23:11:20.049 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:20.045+0000 7f221b7fe640 1 --2- 172.21.15.46:0/1906361167 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f221c102220 0x7f221c19ef80 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f220c00eb10 tx=0x7f220c00c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:11:20.049 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:20.045+0000 7f2213fff640 1 -- 172.21.15.46:0/1906361167 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f220c004b20 con 0x7f221c102220 2024-03-31T23:11:20.049 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:20.045+0000 7f2213fff640 1 -- 172.21.15.46:0/1906361167 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f220c004cc0 con 0x7f221c102220 2024-03-31T23:11:20.049 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:20.045+0000 7f2213fff640 1 -- 172.21.15.46:0/1906361167 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f220c010690 con 0x7f221c102220 2024-03-31T23:11:20.049 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:20.045+0000 7f2221556640 1 -- 172.21.15.46:0/1906361167 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f221c19f8e0 con 0x7f221c102220 2024-03-31T23:11:20.049 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:20.045+0000 7f2221556640 1 -- 172.21.15.46:0/1906361167 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f221c19fde0 con 0x7f221c102220 2024-03-31T23:11:20.049 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:20.046+0000 7f2221556640 1 -- 172.21.15.46:0/1906361167 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f221c107ab0 con 0x7f221c102220 2024-03-31T23:11:20.049 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:20.047+0000 7f2213fff640 1 -- 172.21.15.46:0/1906361167 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 18) ==== 137320+0+0 (secure 0 0 0) 0x7f220c005020 con 0x7f221c102220 2024-03-31T23:11:20.050 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:11:19 smithi162 ceph-mon[36237]: from='client.? 172.21.15.46:0/3718732984' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-31T23:11:20.050 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:11:19 smithi162 ceph-mon[36237]: from='client.? 172.21.15.46:0/3945736985' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-31T23:11:20.050 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:11:19 smithi162 ceph-mon[36237]: from='client.? 172.21.15.46:0/2841507303' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-31T23:11:20.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:20.047+0000 7f2213fff640 1 --2- 172.21.15.46:0/1906361167 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f22080a24a0 0x7f22080a4960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:20.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:20.048+0000 7f221affd640 1 --2- 172.21.15.46:0/1906361167 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f22080a24a0 0x7f22080a4960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:20.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:20.048+0000 7f2213fff640 1 -- 172.21.15.46:0/1906361167 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9208+0+0 (secure 0 0 0) 0x7f220c0cf190 con 0x7f221c102220 2024-03-31T23:11:20.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:20.048+0000 7f221affd640 1 --2- 172.21.15.46:0/1906361167 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f22080a24a0 0x7f22080a4960 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f221c1016c0 tx=0x7f2214005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:11:20.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:20.052+0000 7f2213fff640 1 -- 172.21.15.46:0/1906361167 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f220c09bf20 con 0x7f221c102220 2024-03-31T23:11:20.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:20.239+0000 7f2221556640 1 -- 172.21.15.46:0/1906361167 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f221c062350 con 0x7f221c102220 2024-03-31T23:11:20.240 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:20.240+0000 7f2213fff640 1 -- 172.21.15.46:0/1906361167 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v95) ==== 74+0+24819 (secure 0 0 0) 0x7f220c09b820 con 0x7f221c102220 2024-03-31T23:11:20.241 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-31T23:11:20.241 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":95,"fsid":"6a9d8f08-efb2-11ee-b647-cb9ed24678a4","created":"2024-03-31T23:01:15.463711+0000","modified":"2024-03-31T23:11:10.531495+0000","last_up_change":"2024-03-31T23:08:20.199888+0000","last_in_change":"2024-03-31T23:07:59.438035+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-31T23:04:32.823751+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-31T23:09:09.145711+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-31T23:09:10.148579+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-31T23:09:13.419519+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"91","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"47429095-3f46-4ed7-b7c4-504464719815","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":1824772695},{"type":"v1","addr":"172.21.15.46:6803","nonce":1824772695}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":1824772695},{"type":"v1","addr":"172.21.15.46:6805","nonce":1824772695}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":1824772695},{"type":"v1","addr":"172.21.15.46:6809","nonce":1824772695}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":1824772695},{"type":"v1","addr":"172.21.15.46:6807","nonce":1824772695}]},"public_addr":"172.21.15.46:6803/1824772695","cluster_addr":"172.21.15.46:6805/1824772695","heartbeat_back_addr":"172.21.15.46:6809/1824772695","heartbeat_front_addr":"172.21.15.46:6807/1824772695","state":["exists","up"]},{"osd":1,"uuid":"be56f883-411e-4dbe-b50e-a158134df4a6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6800","nonce":973693922},{"type":"v1","addr":"172.21.15.142:6801","nonce":973693922}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6802","nonce":973693922},{"type":"v1","addr":"172.21.15.142:6803","nonce":973693922}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6806","nonce":973693922},{"type":"v1","addr":"172.21.15.142:6807","nonce":973693922}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6804","nonce":973693922},{"type":"v1","addr":"172.21.15.142:6805","nonce":973693922}]},"public_addr":"172.21.15.142:6801/973693922","cluster_addr":"172.21.15.142:6803/973693922","heartbeat_back_addr":"172.21.15.142:6807/973693922","heartbeat_front_addr":"172.21.15.142:6805/973693922","state":["exists","up"]},{"osd":2,"uuid":"c99a8794-ab19-4260-9a8d-4bc0bd2c8175","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6800","nonce":4084261813},{"type":"v1","addr":"172.21.15.162:6801","nonce":4084261813}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6802","nonce":4084261813},{"type":"v1","addr":"172.21.15.162:6803","nonce":4084261813}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6806","nonce":4084261813},{"type":"v1","addr":"172.21.15.162:6807","nonce":4084261813}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6804","nonce":4084261813},{"type":"v1","addr":"172.21.15.162:6805","nonce":4084261813}]},"public_addr":"172.21.15.162:6801/4084261813","cluster_addr":"172.21.15.162:6803/4084261813","heartbeat_back_addr":"172.21.15.162:6807/4084261813","heartbeat_front_addr":"172.21.15.162:6805/4084261813","state":["exists","up"]},{"osd":3,"uuid":"f877814e-9ef6-4872-a081-5609bb9ebc06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":3496317970},{"type":"v1","addr":"172.21.15.46:6811","nonce":3496317970}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":3496317970},{"type":"v1","addr":"172.21.15.46:6813","nonce":3496317970}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":3496317970},{"type":"v1","addr":"172.21.15.46:6817","nonce":3496317970}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":3496317970},{"type":"v1","addr":"172.21.15.46:6815","nonce":3496317970}]},"public_addr":"172.21.15.46:6811/3496317970","cluster_addr":"172.21.15.46:6813/3496317970","heartbeat_back_addr":"172.21.15.46:6817/3496317970","heartbeat_front_addr":"172.21.15.46:6815/3496317970","state":["exists","up"]},{"osd":4,"uuid":"beecec0a-8280-44fe-8f5f-8498a35eca8f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6808","nonce":2996266929},{"type":"v1","addr":"172.21.15.142:6809","nonce":2996266929}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6810","nonce":2996266929},{"type":"v1","addr":"172.21.15.142:6811","nonce":2996266929}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6814","nonce":2996266929},{"type":"v1","addr":"172.21.15.142:6815","nonce":2996266929}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6812","nonce":2996266929},{"type":"v1","addr":"172.21.15.142:6813","nonce":2996266929}]},"public_addr":"172.21.15.142:6809/2996266929","cluster_addr":"172.21.15.142:6811/2996266929","heartbeat_back_addr":"172.21.15.142:6815/2996266929","heartbeat_front_addr":"172.21.15.142:6813/2996266929","state":["exists","up"]},{"osd":5,"uuid":"9da7ff75-f658-45ea-a02f-1943fdda76b9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6808","nonce":914208293},{"type":"v1","addr":"172.21.15.162:6809","nonce":914208293}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6810","nonce":914208293},{"type":"v1","addr":"172.21.15.162:6811","nonce":914208293}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6814","nonce":914208293},{"type":"v1","addr":"172.21.15.162:6815","nonce":914208293}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6812","nonce":914208293},{"type":"v1","addr":"172.21.15.162:6813","nonce":914208293}]},"public_addr":"172.21.15.162:6809/914208293","cluster_addr":"172.21.15.162:6811/914208293","heartbeat_back_addr":"172.21.15.162:6815/914208293","heartbeat_front_addr":"172.21.15.162:6813/914208293","state":["exists","up"]},{"osd":6,"uuid":"8fa11224-cb26-4d3f-87d1-6a8f31de2a19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":56291966},{"type":"v1","addr":"172.21.15.46:6819","nonce":56291966}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":56291966},{"type":"v1","addr":"172.21.15.46:6821","nonce":56291966}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6824","nonce":56291966},{"type":"v1","addr":"172.21.15.46:6825","nonce":56291966}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":56291966},{"type":"v1","addr":"172.21.15.46:6823","nonce":56291966}]},"public_addr":"172.21.15.46:6819/56291966","cluster_addr":"172.21.15.46:6821/56291966","heartbeat_back_addr":"172.21.15.46:6825/56291966","heartbeat_front_addr":"172.21.15.46:6823/56291966","state":["exists","up"]},{"osd":7,"uuid":"0d0338b7-4d0a-4bb0-b152-8f5224871d9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6816","nonce":2146598446},{"type":"v1","addr":"172.21.15.142:6817","nonce":2146598446}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6818","nonce":2146598446},{"type":"v1","addr":"172.21.15.142:6819","nonce":2146598446}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6822","nonce":2146598446},{"type":"v1","addr":"172.21.15.142:6823","nonce":2146598446}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6820","nonce":2146598446},{"type":"v1","addr":"172.21.15.142:6821","nonce":2146598446}]},"public_addr":"172.21.15.142:6817/2146598446","cluster_addr":"172.21.15.142:6819/2146598446","heartbeat_back_addr":"172.21.15.142:6823/2146598446","heartbeat_front_addr":"172.21.15.142:6821/2146598446","state":["exists","up"]},{"osd":8,"uuid":"8b6f6037-892a-416c-babb-7552648d0b3a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6816","nonce":1173078318},{"type":"v1","addr":"172.21.15.162:6817","nonce":1173078318}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6818","nonce":1173078318},{"type":"v1","addr":"172.21.15.162:6819","nonce":1173078318}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6822","nonce":1173078318},{"type":"v1","addr":"172.21.15.162:6823","nonce":1173078318}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6820","nonce":1173078318},{"type":"v1","addr":"172.21.15.162:6821","nonce":1173078318}]},"public_addr":"172.21.15.162:6817/1173078318","cluster_addr":"172.21.15.162:6819/1173078318","heartbeat_back_addr":"172.21.15.162:6823/1173078318","heartbeat_front_addr":"172.21.15.162:6821/1173078318","state":["exists","up"]},{"osd":9,"uuid":"faccce25-6bf2-4220-8858-cda82b72be68","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":532708881},{"type":"v1","addr":"172.21.15.46:6827","nonce":532708881}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6828","nonce":532708881},{"type":"v1","addr":"172.21.15.46:6829","nonce":532708881}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":532708881},{"type":"v1","addr":"172.21.15.46:6833","nonce":532708881}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":532708881},{"type":"v1","addr":"172.21.15.46:6831","nonce":532708881}]},"public_addr":"172.21.15.46:6827/532708881","cluster_addr":"172.21.15.46:6829/532708881","heartbeat_back_addr":"172.21.15.46:6833/532708881","heartbeat_front_addr":"172.21.15.46:6831/532708881","state":["exists","up"]},{"osd":10,"uuid":"cdd356db-f9eb-4080-95ab-db5d8223e572","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6824","nonce":2845222366},{"type":"v1","addr":"172.21.15.142:6825","nonce":2845222366}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6826","nonce":2845222366},{"type":"v1","addr":"172.21.15.142:6827","nonce":2845222366}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6830","nonce":2845222366},{"type":"v1","addr":"172.21.15.142:6831","nonce":2845222366}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6828","nonce":2845222366},{"type":"v1","addr":"172.21.15.142:6829","nonce":2845222366}]},"public_addr":"172.21.15.142:6825/2845222366","cluster_addr":"172.21.15.142:6827/2845222366","heartbeat_back_addr":"172.21.15.142:6831/2845222366","heartbeat_front_addr":"172.21.15.142:6829/2845222366","state":["exists","up"]},{"osd":11,"uuid":"72104d1c-3ea2-45ad-bc0b-0500f146c06b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6824","nonce":1986323778},{"type":"v1","addr":"172.21.15.162:6825","nonce":1986323778}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6826","nonce":1986323778},{"type":"v1","addr":"172.21.15.162:6827","nonce":1986323778}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6830","nonce":1986323778},{"type":"v1","addr":"172.21.15.162:6831","nonce":1986323778}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6828","nonce":1986323778},{"type":"v1","addr":"172.21.15.162:6829","nonce":1986323778}]},"public_addr":"172.21.15.162:6825/1986323778","cluster_addr":"172.21.15.162:6827/1986323778","heartbeat_back_addr":"172.21.15.162:6831/1986323778","heartbeat_front_addr":"172.21.15.162:6829/1986323778","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:03:44.996331+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:04:07.994738+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:04:30.942255+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:04:55.588884+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:05:19.900600+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:05:43.807442+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:06:09.026976+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:06:34.586784+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:06:59.751605+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:07:26.144495+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:07:52.326909+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:08:17.423042+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.b","mappings":[{"from":8,"to":11}]},{"pgid":"3.6","mappings":[{"from":4,"to":7}]},{"pgid":"3.c","mappings":[{"from":3,"to":9}]},{"pgid":"3.1a","mappings":[{"from":5,"to":6}]},{"pgid":"3.1b","mappings":[{"from":4,"to":8}]},{"pgid":"3.1d","mappings":[{"from":4,"to":8}]},{"pgid":"3.2a","mappings":[{"from":3,"to":6}]},{"pgid":"3.33","mappings":[{"from":5,"to":9}]},{"pgid":"3.36","mappings":[{"from":4,"to":9}]},{"pgid":"4.6","mappings":[{"from":5,"to":8}]},{"pgid":"4.17","mappings":[{"from":9,"to":8}]},{"pgid":"4.18","mappings":[{"from":9,"to":8}]},{"pgid":"4.1b","mappings":[{"from":9,"to":10}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.46:0/4021039857":"2024-04-01T23:02:08.493619+0000","172.21.15.46:6801/3681182647":"2024-04-01T23:02:08.493619+0000","172.21.15.46:0/4022214806":"2024-04-01T23:02:08.493619+0000","172.21.15.46:6800/1515528215":"2024-04-01T23:01:41.910162+0000","172.21.15.46:6800/3681182647":"2024-04-01T23:02:08.493619+0000","172.21.15.46:0/143885707":"2024-04-01T23:01:41.910162+0000","172.21.15.46:6801/1515528215":"2024-04-01T23:01:41.910162+0000","172.21.15.46:0/1680389778":"2024-04-01T23:01:41.910162+0000","172.21.15.46:0/1782133966":"2024-04-01T23:01:41.910162+0000","172.21.15.46:0/3925209376":"2024-04-01T23:02:08.493619+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-31T23:11:20.243 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:20.243+0000 7f2221556640 1 -- 172.21.15.46:0/1906361167 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f22080a24a0 msgr2=0x7f22080a4960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:20.243 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:20.243+0000 7f2221556640 1 --2- 172.21.15.46:0/1906361167 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f22080a24a0 0x7f22080a4960 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f221c1016c0 tx=0x7f2214005f90 comp rx=0 tx=0).stop 2024-03-31T23:11:20.243 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:20.243+0000 7f2221556640 1 -- 172.21.15.46:0/1906361167 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f221c102220 msgr2=0x7f221c19ef80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:20.243 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:20.243+0000 7f2221556640 1 --2- 172.21.15.46:0/1906361167 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f221c102220 0x7f221c19ef80 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f220c00eb10 tx=0x7f220c00c2d0 comp rx=0 tx=0).stop 2024-03-31T23:11:20.243 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:20.243+0000 7f2221556640 1 -- 172.21.15.46:0/1906361167 shutdown_connections 2024-03-31T23:11:20.243 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:20.243+0000 7f2221556640 1 --2- 172.21.15.46:0/1906361167 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f22080a24a0 0x7f22080a4960 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:20.243 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:20.243+0000 7f2221556640 1 --2- 172.21.15.46:0/1906361167 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f221c102220 0x7f221c19ef80 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:20.243 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:20.243+0000 7f2221556640 1 --2- 172.21.15.46:0/1906361167 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f221c101860 0x7f221c197f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:20.243 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:20.243+0000 7f2221556640 1 --2- 172.21.15.46:0/1906361167 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f221c100670 0x7f221c1979d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:20.244 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:20.243+0000 7f2221556640 1 -- 172.21.15.46:0/1906361167 >> 172.21.15.46:0/1906361167 conn(0x7f221c0fbbe0 msgr2=0x7f221c104c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:11:20.244 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:20.243+0000 7f2221556640 1 -- 172.21.15.46:0/1906361167 shutdown_connections 2024-03-31T23:11:20.244 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:20.243+0000 7f2221556640 1 -- 172.21.15.46:0/1906361167 wait complete. 2024-03-31T23:11:20.254 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-31T23:04:32.823751+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-31T23:09:09.145711+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-31T23:09:10.148579+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-31T23:09:13.419519+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '91', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-31T23:11:20.255 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-31T23:11:20.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:20.360+0000 7f85c7c8f640 1 -- 172.21.15.46:0/2753573818 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f85c0107450 msgr2=0x7f85c01098a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:20.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:20.360+0000 7f85c7c8f640 1 --2- 172.21.15.46:0/2753573818 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f85c0107450 0x7f85c01098a0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f85bc009be0 tx=0x7f85bc02bef0 comp rx=0 tx=0).stop 2024-03-31T23:11:20.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:20.360+0000 7f85c7c8f640 1 -- 172.21.15.46:0/2753573818 shutdown_connections 2024-03-31T23:11:20.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:20.360+0000 7f85c7c8f640 1 --2- 172.21.15.46:0/2753573818 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f85c0107450 0x7f85c01098a0 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:20.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:20.360+0000 7f85c7c8f640 1 --2- 172.21.15.46:0/2753573818 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f85c0104b20 0x7f85c0106f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:20.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:20.360+0000 7f85c7c8f640 1 --2- 172.21.15.46:0/2753573818 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f85c01021f0 0x7f85c01045e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:20.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:20.360+0000 7f85c7c8f640 1 -- 172.21.15.46:0/2753573818 >> 172.21.15.46:0/2753573818 conn(0x7f85c00fbc20 msgr2=0x7f85c00fe040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:11:20.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:20.360+0000 7f85c7c8f640 1 -- 172.21.15.46:0/2753573818 shutdown_connections 2024-03-31T23:11:20.364 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:20.360+0000 7f85c7c8f640 1 -- 172.21.15.46:0/2753573818 wait complete. 2024-03-31T23:11:20.364 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:20.361+0000 7f85c7c8f640 1 Processor -- start 2024-03-31T23:11:20.364 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:20.361+0000 7f85c7c8f640 1 -- start start 2024-03-31T23:11:20.364 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:20.361+0000 7f85c7c8f640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f85c01021f0 0x7f85c0197910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:20.364 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:20.361+0000 7f85c7c8f640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f85c0104b20 0x7f85c0197e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:20.364 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:20.361+0000 7f85c7c8f640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f85c0107450 0x7f85c019eec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:20.364 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:20.361+0000 7f85c7c8f640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f85c010d2a0 con 0x7f85c01021f0 2024-03-31T23:11:20.364 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:20.361+0000 7f85c7c8f640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f85c010d120 con 0x7f85c0104b20 2024-03-31T23:11:20.364 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:20.361+0000 7f85c7c8f640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f85c010d420 con 0x7f85c0107450 2024-03-31T23:11:20.364 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:20.361+0000 7f85c5a2b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f85c01021f0 0x7f85c0197910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:20.364 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:20.361+0000 7f85c5a2b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f85c01021f0 0x7f85c0197910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:54844/0 (socket says 172.21.15.46:54844) 2024-03-31T23:11:20.364 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:20.361+0000 7f85c5a2b640 1 -- 172.21.15.46:0/819575465 learned_addr learned my addr 172.21.15.46:0/819575465 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:11:20.364 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:20.361+0000 7f85c522a640 1 --2- 172.21.15.46:0/819575465 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f85c0104b20 0x7f85c0197e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:20.364 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:20.361+0000 7f85c622c640 1 --2- 172.21.15.46:0/819575465 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f85c0107450 0x7f85c019eec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:20.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:20.361+0000 7f85c5a2b640 1 -- 172.21.15.46:0/819575465 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f85c0107450 msgr2=0x7f85c019eec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:20.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:20.361+0000 7f85c5a2b640 1 --2- 172.21.15.46:0/819575465 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f85c0107450 0x7f85c019eec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:20.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:20.361+0000 7f85c5a2b640 1 -- 172.21.15.46:0/819575465 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f85c0104b20 msgr2=0x7f85c0197e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:20.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:20.362+0000 7f85c5a2b640 1 --2- 172.21.15.46:0/819575465 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f85c0104b20 0x7f85c0197e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:20.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:20.362+0000 7f85c5a2b640 1 -- 172.21.15.46:0/819575465 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f85c019f530 con 0x7f85c01021f0 2024-03-31T23:11:20.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:20.362+0000 7f85c5a2b640 1 --2- 172.21.15.46:0/819575465 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f85c01021f0 0x7f85c0197910 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f85b800e9c0 tx=0x7f85b800ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:11:20.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:20.362+0000 7f85b6ffd640 1 -- 172.21.15.46:0/819575465 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85b800ccc0 con 0x7f85c01021f0 2024-03-31T23:11:20.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:20.362+0000 7f85b6ffd640 1 -- 172.21.15.46:0/819575465 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f85b80054d0 con 0x7f85c01021f0 2024-03-31T23:11:20.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:20.362+0000 7f85c7c8f640 1 -- 172.21.15.46:0/819575465 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f85c019f820 con 0x7f85c01021f0 2024-03-31T23:11:20.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:20.362+0000 7f85b6ffd640 1 -- 172.21.15.46:0/819575465 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85b8010430 con 0x7f85c01021f0 2024-03-31T23:11:20.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:20.362+0000 7f85c7c8f640 1 -- 172.21.15.46:0/819575465 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f85c019fd70 con 0x7f85c01021f0 2024-03-31T23:11:20.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:20.363+0000 7f85c7c8f640 1 -- 172.21.15.46:0/819575465 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f85c00fd710 con 0x7f85c01021f0 2024-03-31T23:11:20.366 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:20.364+0000 7f85b6ffd640 1 -- 172.21.15.46:0/819575465 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 18) ==== 137320+0+0 (secure 0 0 0) 0x7f85b80046c0 con 0x7f85c01021f0 2024-03-31T23:11:20.366 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:20.364+0000 7f85b6ffd640 1 --2- 172.21.15.46:0/819575465 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f85ac0a24a0 0x7f85ac0a4960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:20.366 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:20.365+0000 7f85c522a640 1 --2- 172.21.15.46:0/819575465 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f85ac0a24a0 0x7f85ac0a4960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:20.366 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:20.365+0000 7f85b6ffd640 1 -- 172.21.15.46:0/819575465 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9208+0+0 (secure 0 0 0) 0x7f85b80ced40 con 0x7f85c01021f0 2024-03-31T23:11:20.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:20.365+0000 7f85c522a640 1 --2- 172.21.15.46:0/819575465 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f85ac0a24a0 0x7f85ac0a4960 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f85b00095f0 tx=0x7f85b0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:11:20.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:20.369+0000 7f85b6ffd640 1 -- 172.21.15.46:0/819575465 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f85b809bb30 con 0x7f85c01021f0 2024-03-31T23:11:20.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:20.552+0000 7f85c7c8f640 1 -- 172.21.15.46:0/819575465 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f85c0062350 con 0x7f85c01021f0 2024-03-31T23:11:20.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:20.552+0000 7f85b6ffd640 1 -- 172.21.15.46:0/819575465 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v95) ==== 93+0+10 (secure 0 0 0) 0x7f85b809b430 con 0x7f85c01021f0 2024-03-31T23:11:20.553 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 1 2024-03-31T23:11:20.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:20.555+0000 7f85c7c8f640 1 -- 172.21.15.46:0/819575465 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f85ac0a24a0 msgr2=0x7f85ac0a4960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:20.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:20.555+0000 7f85c7c8f640 1 --2- 172.21.15.46:0/819575465 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f85ac0a24a0 0x7f85ac0a4960 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f85b00095f0 tx=0x7f85b0005f90 comp rx=0 tx=0).stop 2024-03-31T23:11:20.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:20.555+0000 7f85c7c8f640 1 -- 172.21.15.46:0/819575465 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f85c01021f0 msgr2=0x7f85c0197910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:20.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:20.555+0000 7f85c7c8f640 1 --2- 172.21.15.46:0/819575465 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f85c01021f0 0x7f85c0197910 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f85b800e9c0 tx=0x7f85b800ee80 comp rx=0 tx=0).stop 2024-03-31T23:11:20.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:20.555+0000 7f85c7c8f640 1 -- 172.21.15.46:0/819575465 shutdown_connections 2024-03-31T23:11:20.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:20.555+0000 7f85c7c8f640 1 --2- 172.21.15.46:0/819575465 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f85ac0a24a0 0x7f85ac0a4960 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:20.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:20.555+0000 7f85c7c8f640 1 --2- 172.21.15.46:0/819575465 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f85c0107450 0x7f85c019eec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:20.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:20.555+0000 7f85c7c8f640 1 --2- 172.21.15.46:0/819575465 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f85c0104b20 0x7f85c0197e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:20.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:20.555+0000 7f85c7c8f640 1 --2- 172.21.15.46:0/819575465 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f85c01021f0 0x7f85c0197910 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:20.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:20.555+0000 7f85c7c8f640 1 -- 172.21.15.46:0/819575465 >> 172.21.15.46:0/819575465 conn(0x7f85c00fbc20 msgr2=0x7f85c0102b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:11:20.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:20.555+0000 7f85c7c8f640 1 -- 172.21.15.46:0/819575465 shutdown_connections 2024-03-31T23:11:20.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:20.555+0000 7f85c7c8f640 1 -- 172.21.15.46:0/819575465 wait complete. 2024-03-31T23:11:20.567 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-31T23:11:20.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:20.673+0000 7f07b3199640 1 -- 172.21.15.46:0/3365616114 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f07ac101770 msgr2=0x7f07ac101b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:20.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:20.673+0000 7f07b3199640 1 --2- 172.21.15.46:0/3365616114 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f07ac101770 0x7f07ac101b90 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f07a0009900 tx=0x7f07a002bc30 comp rx=0 tx=0).stop 2024-03-31T23:11:20.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:20.673+0000 7f07b3199640 1 -- 172.21.15.46:0/3365616114 shutdown_connections 2024-03-31T23:11:20.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:20.673+0000 7f07b3199640 1 --2- 172.21.15.46:0/3365616114 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f07ac103320 0x7f07ac109bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:20.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:20.673+0000 7f07b3199640 1 --2- 172.21.15.46:0/3365616114 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f07ac102960 0x7f07ac102de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:20.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:20.673+0000 7f07b3199640 1 --2- 172.21.15.46:0/3365616114 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f07ac101770 0x7f07ac101b90 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:20.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:20.673+0000 7f07b3199640 1 -- 172.21.15.46:0/3365616114 >> 172.21.15.46:0/3365616114 conn(0x7f07ac0fcd00 msgr2=0x7f07ac0ff120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:11:20.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:20.673+0000 7f07b3199640 1 -- 172.21.15.46:0/3365616114 shutdown_connections 2024-03-31T23:11:20.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:20.673+0000 7f07b3199640 1 -- 172.21.15.46:0/3365616114 wait complete. 2024-03-31T23:11:20.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:20.673+0000 7f07b3199640 1 Processor -- start 2024-03-31T23:11:20.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:20.674+0000 7f07b3199640 1 -- start start 2024-03-31T23:11:20.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:20.674+0000 7f07b3199640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f07ac101770 0x7f07ac197a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:20.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:20.674+0000 7f07b3199640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f07ac102960 0x7f07ac197f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:20.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:20.674+0000 7f07b3199640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f07ac103320 0x7f07ac19eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:20.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:20.674+0000 7f07b3199640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f07ac10d290 con 0x7f07ac101770 2024-03-31T23:11:20.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:20.674+0000 7f07b3199640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f07ac10d110 con 0x7f07ac102960 2024-03-31T23:11:20.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:20.674+0000 7f07b3199640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f07ac10d410 con 0x7f07ac103320 2024-03-31T23:11:20.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:20.674+0000 7f07b0f35640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f07ac101770 0x7f07ac197a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:20.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:20.674+0000 7f07b0f35640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f07ac101770 0x7f07ac197a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:54862/0 (socket says 172.21.15.46:54862) 2024-03-31T23:11:20.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:20.674+0000 7f07b0f35640 1 -- 172.21.15.46:0/3538435567 learned_addr learned my addr 172.21.15.46:0/3538435567 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:11:20.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:20.674+0000 7f07abfff640 1 --2- 172.21.15.46:0/3538435567 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f07ac102960 0x7f07ac197f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:20.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:20.674+0000 7f07b1736640 1 --2- 172.21.15.46:0/3538435567 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f07ac103320 0x7f07ac19eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:20.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:20.674+0000 7f07b0f35640 1 -- 172.21.15.46:0/3538435567 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f07ac103320 msgr2=0x7f07ac19eff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:20.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:20.674+0000 7f07b0f35640 1 --2- 172.21.15.46:0/3538435567 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f07ac103320 0x7f07ac19eff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:20.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:20.674+0000 7f07b0f35640 1 -- 172.21.15.46:0/3538435567 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f07ac102960 msgr2=0x7f07ac197f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:20.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:20.674+0000 7f07b0f35640 1 --2- 172.21.15.46:0/3538435567 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f07ac102960 0x7f07ac197f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:20.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:20.674+0000 7f07b0f35640 1 -- 172.21.15.46:0/3538435567 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f07ac19f660 con 0x7f07ac101770 2024-03-31T23:11:20.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:20.674+0000 7f07b1736640 1 --2- 172.21.15.46:0/3538435567 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f07ac103320 0x7f07ac19eff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T23:11:20.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:20.674+0000 7f07b0f35640 1 --2- 172.21.15.46:0/3538435567 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f07ac101770 0x7f07ac197a40 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f07a0000c00 tx=0x7f07a0005dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:11:20.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:20.674+0000 7f07a9ffb640 1 -- 172.21.15.46:0/3538435567 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07a0043020 con 0x7f07ac101770 2024-03-31T23:11:20.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:20.674+0000 7f07a9ffb640 1 -- 172.21.15.46:0/3538435567 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f07a0004350 con 0x7f07ac101770 2024-03-31T23:11:20.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:20.674+0000 7f07a9ffb640 1 -- 172.21.15.46:0/3538435567 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07a0003b80 con 0x7f07ac101770 2024-03-31T23:11:20.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:20.674+0000 7f07b3199640 1 -- 172.21.15.46:0/3538435567 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f07ac19f8f0 con 0x7f07ac101770 2024-03-31T23:11:20.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:20.675+0000 7f07b3199640 1 -- 172.21.15.46:0/3538435567 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f07ac19fe90 con 0x7f07ac101770 2024-03-31T23:11:20.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:20.676+0000 7f07b3199640 1 -- 172.21.15.46:0/3538435567 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f07ac108b80 con 0x7f07ac101770 2024-03-31T23:11:20.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:20.677+0000 7f07a9ffb640 1 -- 172.21.15.46:0/3538435567 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 18) ==== 137320+0+0 (secure 0 0 0) 0x7f07a0046020 con 0x7f07ac101770 2024-03-31T23:11:20.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:20.677+0000 7f07a9ffb640 1 --2- 172.21.15.46:0/3538435567 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f079c0a2520 0x7f079c0a49e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:20.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:20.677+0000 7f07abfff640 1 --2- 172.21.15.46:0/3538435567 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f079c0a2520 0x7f079c0a49e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:20.680 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:20.678+0000 7f07a9ffb640 1 -- 172.21.15.46:0/3538435567 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9208+0+0 (secure 0 0 0) 0x7f07a00f0c40 con 0x7f07ac101770 2024-03-31T23:11:20.680 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:20.678+0000 7f07abfff640 1 --2- 172.21.15.46:0/3538435567 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f079c0a2520 0x7f079c0a49e0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f07ac198f50 tx=0x7f0798005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:11:20.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:20.682+0000 7f07a9ffb640 1 -- 172.21.15.46:0/3538435567 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f07a004d020 con 0x7f07ac101770 2024-03-31T23:11:20.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:11:20 smithi046 ceph-mon[32498]: pgmap v360: 129 pgs: 129 active+clean; 642 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s; 0 B/s, 0 objects/s recovering 2024-03-31T23:11:20.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:11:20 smithi046 ceph-mon[32498]: from='client.? 172.21.15.46:0/1046600875' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-31T23:11:20.862 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:11:20 smithi046 ceph-mon[32498]: from='client.? 172.21.15.46:0/2989957989' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-31T23:11:20.862 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:11:20 smithi046 ceph-mon[32498]: from='client.? 172.21.15.46:0/1906361167' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-31T23:11:20.862 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:11:20 smithi046 ceph-mon[32498]: from='client.? 172.21.15.46:0/819575465' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-31T23:11:20.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:20.862+0000 7f07b3199640 1 -- 172.21.15.46:0/3538435567 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f07ac062350 con 0x7f07ac101770 2024-03-31T23:11:20.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:20.863+0000 7f07a9ffb640 1 -- 172.21.15.46:0/3538435567 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v95) ==== 104+0+11 (secure 0 0 0) 0x7f07a00bd9d0 con 0x7f07ac101770 2024-03-31T23:11:20.863 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 32 2024-03-31T23:11:20.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:20.864+0000 7f07b3199640 1 -- 172.21.15.46:0/3538435567 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f079c0a2520 msgr2=0x7f079c0a49e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:20.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:20.864+0000 7f07b3199640 1 --2- 172.21.15.46:0/3538435567 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f079c0a2520 0x7f079c0a49e0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f07ac198f50 tx=0x7f0798005f90 comp rx=0 tx=0).stop 2024-03-31T23:11:20.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:20.864+0000 7f07b3199640 1 -- 172.21.15.46:0/3538435567 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f07ac101770 msgr2=0x7f07ac197a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:20.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:20.864+0000 7f07b3199640 1 --2- 172.21.15.46:0/3538435567 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f07ac101770 0x7f07ac197a40 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f07a0000c00 tx=0x7f07a0005dc0 comp rx=0 tx=0).stop 2024-03-31T23:11:20.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:20.865+0000 7f07b3199640 1 -- 172.21.15.46:0/3538435567 shutdown_connections 2024-03-31T23:11:20.866 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:20.865+0000 7f07b3199640 1 --2- 172.21.15.46:0/3538435567 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f079c0a2520 0x7f079c0a49e0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:20.866 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:20.865+0000 7f07b3199640 1 --2- 172.21.15.46:0/3538435567 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f07ac103320 0x7f07ac19eff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:20.866 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:20.865+0000 7f07b3199640 1 --2- 172.21.15.46:0/3538435567 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f07ac102960 0x7f07ac197f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:20.866 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:20.865+0000 7f07b3199640 1 --2- 172.21.15.46:0/3538435567 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f07ac101770 0x7f07ac197a40 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:20.866 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:20.865+0000 7f07b3199640 1 -- 172.21.15.46:0/3538435567 >> 172.21.15.46:0/3538435567 conn(0x7f07ac0fcd00 msgr2=0x7f07ac105d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:11:20.866 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:20.865+0000 7f07b3199640 1 -- 172.21.15.46:0/3538435567 shutdown_connections 2024-03-31T23:11:20.866 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:20.865+0000 7f07b3199640 1 -- 172.21.15.46:0/3538435567 wait complete. 2024-03-31T23:11:20.876 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-31T23:11:20.945 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:11:20 smithi142 ceph-mon[35271]: pgmap v360: 129 pgs: 129 active+clean; 642 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s; 0 B/s, 0 objects/s recovering 2024-03-31T23:11:20.946 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:11:20 smithi142 ceph-mon[35271]: from='client.? 172.21.15.46:0/1046600875' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-31T23:11:20.946 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:11:20 smithi142 ceph-mon[35271]: from='client.? 172.21.15.46:0/2989957989' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-31T23:11:20.946 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:11:20 smithi142 ceph-mon[35271]: from='client.? 172.21.15.46:0/1906361167' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-31T23:11:20.946 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:11:20 smithi142 ceph-mon[35271]: from='client.? 172.21.15.46:0/819575465' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-31T23:11:20.947 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:11:20 smithi162 ceph-mon[36237]: pgmap v360: 129 pgs: 129 active+clean; 642 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s; 0 B/s, 0 objects/s recovering 2024-03-31T23:11:20.947 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:11:20 smithi162 ceph-mon[36237]: from='client.? 172.21.15.46:0/1046600875' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-31T23:11:20.947 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:11:20 smithi162 ceph-mon[36237]: from='client.? 172.21.15.46:0/2989957989' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-31T23:11:20.947 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:11:20 smithi162 ceph-mon[36237]: from='client.? 172.21.15.46:0/1906361167' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-31T23:11:20.947 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:11:20 smithi162 ceph-mon[36237]: from='client.? 172.21.15.46:0/819575465' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-31T23:11:20.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:20.982+0000 7fa0cfb8c640 1 -- 172.21.15.46:0/2014173726 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa0c8102a20 msgr2=0x7fa0c8102ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:20.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:20.982+0000 7fa0cfb8c640 1 --2- 172.21.15.46:0/2014173726 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa0c8102a20 0x7fa0c8102ea0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7fa0b8009930 tx=0x7fa0b802bc30 comp rx=0 tx=0).stop 2024-03-31T23:11:20.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:20.982+0000 7fa0cfb8c640 1 -- 172.21.15.46:0/2014173726 shutdown_connections 2024-03-31T23:11:20.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:20.982+0000 7fa0cfb8c640 1 --2- 172.21.15.46:0/2014173726 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa0c81033e0 0x7fa0c8109c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:20.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:20.982+0000 7fa0cfb8c640 1 --2- 172.21.15.46:0/2014173726 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa0c8102a20 0x7fa0c8102ea0 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:20.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:20.982+0000 7fa0cfb8c640 1 --2- 172.21.15.46:0/2014173726 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa0c80ffa50 0x7fa0c80ffe70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:20.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:20.982+0000 7fa0cfb8c640 1 -- 172.21.15.46:0/2014173726 >> 172.21.15.46:0/2014173726 conn(0x7fa0c80fb380 msgr2=0x7fa0c80fd7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:11:20.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:20.982+0000 7fa0cfb8c640 1 -- 172.21.15.46:0/2014173726 shutdown_connections 2024-03-31T23:11:20.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:20.982+0000 7fa0cfb8c640 1 -- 172.21.15.46:0/2014173726 wait complete. 2024-03-31T23:11:20.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:20.982+0000 7fa0cfb8c640 1 Processor -- start 2024-03-31T23:11:20.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:20.983+0000 7fa0cfb8c640 1 -- start start 2024-03-31T23:11:20.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:20.983+0000 7fa0cfb8c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa0c80ffa50 0x7fa0c8197990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:20.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:20.983+0000 7fa0cfb8c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa0c8102a20 0x7fa0c8197ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:20.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:20.983+0000 7fa0cfb8c640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa0c81033e0 0x7fa0c819ef40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:20.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:20.983+0000 7fa0cfb8c640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fa0c810d310 con 0x7fa0c8102a20 2024-03-31T23:11:20.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:20.983+0000 7fa0cfb8c640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fa0c810d190 con 0x7fa0c81033e0 2024-03-31T23:11:20.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:20.983+0000 7fa0cfb8c640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fa0c810d490 con 0x7fa0c80ffa50 2024-03-31T23:11:20.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:20.983+0000 7fa0cd127640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa0c8102a20 0x7fa0c8197ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:20.986 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:20.983+0000 7fa0cd127640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa0c8102a20 0x7fa0c8197ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:54882/0 (socket says 172.21.15.46:54882) 2024-03-31T23:11:20.986 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:20.983+0000 7fa0cd127640 1 -- 172.21.15.46:0/3559172068 learned_addr learned my addr 172.21.15.46:0/3559172068 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:11:20.986 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:20.983+0000 7fa0cd928640 1 --2- 172.21.15.46:0/3559172068 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa0c80ffa50 0x7fa0c8197990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:20.986 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:20.983+0000 7fa0ce129640 1 --2- 172.21.15.46:0/3559172068 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa0c81033e0 0x7fa0c819ef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:20.986 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:20.983+0000 7fa0cd127640 1 -- 172.21.15.46:0/3559172068 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa0c80ffa50 msgr2=0x7fa0c8197990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:20.986 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:20.983+0000 7fa0cd127640 1 --2- 172.21.15.46:0/3559172068 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa0c80ffa50 0x7fa0c8197990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:20.986 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:20.983+0000 7fa0cd127640 1 -- 172.21.15.46:0/3559172068 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa0c81033e0 msgr2=0x7fa0c819ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:20.986 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:20.983+0000 7fa0cd127640 1 --2- 172.21.15.46:0/3559172068 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa0c81033e0 0x7fa0c819ef40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:20.986 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:20.983+0000 7fa0cd127640 1 -- 172.21.15.46:0/3559172068 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa0c819f5b0 con 0x7fa0c8102a20 2024-03-31T23:11:20.986 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:20.983+0000 7fa0cd127640 1 --2- 172.21.15.46:0/3559172068 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa0c8102a20 0x7fa0c8197ed0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7fa0b8000c00 tx=0x7fa0b8005e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:11:20.986 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:20.983+0000 7fa0beffd640 1 -- 172.21.15.46:0/3559172068 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0b8043020 con 0x7fa0c8102a20 2024-03-31T23:11:20.986 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:20.983+0000 7fa0beffd640 1 -- 172.21.15.46:0/3559172068 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa0b80043d0 con 0x7fa0c8102a20 2024-03-31T23:11:20.987 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:20.983+0000 7fa0beffd640 1 -- 172.21.15.46:0/3559172068 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0b8003b40 con 0x7fa0c8102a20 2024-03-31T23:11:20.987 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:20.984+0000 7fa0cfb8c640 1 -- 172.21.15.46:0/3559172068 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa0c819f840 con 0x7fa0c8102a20 2024-03-31T23:11:20.987 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:20.984+0000 7fa0cfb8c640 1 -- 172.21.15.46:0/3559172068 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa0c819fde0 con 0x7fa0c8102a20 2024-03-31T23:11:20.987 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:20.984+0000 7fa0cfb8c640 1 -- 172.21.15.46:0/3559172068 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa0c8108b50 con 0x7fa0c8102a20 2024-03-31T23:11:20.987 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:20.985+0000 7fa0beffd640 1 -- 172.21.15.46:0/3559172068 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 18) ==== 137320+0+0 (secure 0 0 0) 0x7fa0b8046020 con 0x7fa0c8102a20 2024-03-31T23:11:20.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:20.986+0000 7fa0beffd640 1 --2- 172.21.15.46:0/3559172068 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fa0b40a2520 0x7fa0b40a49e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:20.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:20.986+0000 7fa0cd928640 1 --2- 172.21.15.46:0/3559172068 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fa0b40a2520 0x7fa0b40a49e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:20.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:20.986+0000 7fa0beffd640 1 -- 172.21.15.46:0/3559172068 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9208+0+0 (secure 0 0 0) 0x7fa0b80f0c00 con 0x7fa0c8102a20 2024-03-31T23:11:20.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:20.986+0000 7fa0cd928640 1 --2- 172.21.15.46:0/3559172068 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fa0b40a2520 0x7fa0b40a49e0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fa0c8069bc0 tx=0x7fa0c4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:11:20.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:20.990+0000 7fa0beffd640 1 -- 172.21.15.46:0/3559172068 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fa0b804d020 con 0x7fa0c8102a20 2024-03-31T23:11:21.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:21.174+0000 7fa0cfb8c640 1 -- 172.21.15.46:0/3559172068 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fa0c8062350 con 0x7fa0c8102a20 2024-03-31T23:11:21.175 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:21.175+0000 7fa0beffd640 1 -- 172.21.15.46:0/3559172068 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v95) ==== 100+0+11 (secure 0 0 0) 0x7fa0b80bd990 con 0x7fa0c8102a20 2024-03-31T23:11:21.175 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 64 2024-03-31T23:11:21.177 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:21.177+0000 7fa0cfb8c640 1 -- 172.21.15.46:0/3559172068 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fa0b40a2520 msgr2=0x7fa0b40a49e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:21.177 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:21.177+0000 7fa0cfb8c640 1 --2- 172.21.15.46:0/3559172068 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fa0b40a2520 0x7fa0b40a49e0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fa0c8069bc0 tx=0x7fa0c4005f90 comp rx=0 tx=0).stop 2024-03-31T23:11:21.177 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:21.177+0000 7fa0cfb8c640 1 -- 172.21.15.46:0/3559172068 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa0c8102a20 msgr2=0x7fa0c8197ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:21.178 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:21.177+0000 7fa0cfb8c640 1 --2- 172.21.15.46:0/3559172068 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa0c8102a20 0x7fa0c8197ed0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7fa0b8000c00 tx=0x7fa0b8005e60 comp rx=0 tx=0).stop 2024-03-31T23:11:21.178 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:21.177+0000 7fa0cfb8c640 1 -- 172.21.15.46:0/3559172068 shutdown_connections 2024-03-31T23:11:21.178 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:21.177+0000 7fa0cfb8c640 1 --2- 172.21.15.46:0/3559172068 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fa0b40a2520 0x7fa0b40a49e0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:21.178 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:21.177+0000 7fa0cfb8c640 1 --2- 172.21.15.46:0/3559172068 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa0c81033e0 0x7fa0c819ef40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:21.178 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:21.177+0000 7fa0cfb8c640 1 --2- 172.21.15.46:0/3559172068 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa0c8102a20 0x7fa0c8197ed0 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:21.178 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:21.177+0000 7fa0cfb8c640 1 --2- 172.21.15.46:0/3559172068 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa0c80ffa50 0x7fa0c8197990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:21.178 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:21.177+0000 7fa0cfb8c640 1 -- 172.21.15.46:0/3559172068 >> 172.21.15.46:0/3559172068 conn(0x7fa0c80fb380 msgr2=0x7fa0c8105d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:11:21.178 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:21.177+0000 7fa0cfb8c640 1 -- 172.21.15.46:0/3559172068 shutdown_connections 2024-03-31T23:11:21.178 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:21.177+0000 7fa0cfb8c640 1 -- 172.21.15.46:0/3559172068 wait complete. 2024-03-31T23:11:21.188 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-31T23:11:21.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:21.294+0000 7f8d172b5640 1 -- 172.21.15.46:0/4144130255 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8d10104b00 msgr2=0x7f8d10106ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:21.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:21.294+0000 7f8d172b5640 1 --2- 172.21.15.46:0/4144130255 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8d10104b00 0x7f8d10106ef0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f8d04009900 tx=0x7f8d0402bc30 comp rx=0 tx=0).stop 2024-03-31T23:11:21.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:21.294+0000 7f8d172b5640 1 -- 172.21.15.46:0/4144130255 shutdown_connections 2024-03-31T23:11:21.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:21.294+0000 7f8d172b5640 1 --2- 172.21.15.46:0/4144130255 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8d10107430 0x7f8d10109880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:21.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:21.294+0000 7f8d172b5640 1 --2- 172.21.15.46:0/4144130255 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8d10104b00 0x7f8d10106ef0 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:21.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:21.294+0000 7f8d172b5640 1 --2- 172.21.15.46:0/4144130255 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8d101021d0 0x7f8d101045c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:21.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:21.294+0000 7f8d172b5640 1 -- 172.21.15.46:0/4144130255 >> 172.21.15.46:0/4144130255 conn(0x7f8d100fbbe0 msgr2=0x7f8d100fe000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:11:21.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:21.295+0000 7f8d172b5640 1 -- 172.21.15.46:0/4144130255 shutdown_connections 2024-03-31T23:11:21.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:21.295+0000 7f8d172b5640 1 -- 172.21.15.46:0/4144130255 wait complete. 2024-03-31T23:11:21.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:21.295+0000 7f8d172b5640 1 Processor -- start 2024-03-31T23:11:21.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:21.295+0000 7f8d172b5640 1 -- start start 2024-03-31T23:11:21.298 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:21.295+0000 7f8d172b5640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8d101021d0 0x7f8d10197a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:21.298 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:21.295+0000 7f8d172b5640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8d10104b00 0x7f8d10197f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:21.298 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:21.295+0000 7f8d172b5640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8d10107430 0x7f8d1019efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:21.298 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:21.295+0000 7f8d172b5640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f8d1010d280 con 0x7f8d10104b00 2024-03-31T23:11:21.298 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:21.295+0000 7f8d172b5640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f8d1010d100 con 0x7f8d10107430 2024-03-31T23:11:21.298 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:21.295+0000 7f8d172b5640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f8d1010d400 con 0x7f8d101021d0 2024-03-31T23:11:21.298 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:21.295+0000 7f8d14850640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8d10104b00 0x7f8d10197f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:21.298 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:21.295+0000 7f8d14850640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8d10104b00 0x7f8d10197f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:54900/0 (socket says 172.21.15.46:54900) 2024-03-31T23:11:21.298 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:21.295+0000 7f8d14850640 1 -- 172.21.15.46:0/3226846791 learned_addr learned my addr 172.21.15.46:0/3226846791 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:11:21.298 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:21.295+0000 7f8d15852640 1 --2- 172.21.15.46:0/3226846791 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8d10107430 0x7f8d1019efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:21.298 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:21.295+0000 7f8d15051640 1 --2- 172.21.15.46:0/3226846791 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8d101021d0 0x7f8d10197a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:21.298 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:21.296+0000 7f8d14850640 1 -- 172.21.15.46:0/3226846791 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8d101021d0 msgr2=0x7f8d10197a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:21.298 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:21.296+0000 7f8d14850640 1 --2- 172.21.15.46:0/3226846791 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8d101021d0 0x7f8d10197a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:21.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:21.296+0000 7f8d14850640 1 -- 172.21.15.46:0/3226846791 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8d10107430 msgr2=0x7f8d1019efb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:21.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:21.296+0000 7f8d14850640 1 --2- 172.21.15.46:0/3226846791 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8d10107430 0x7f8d1019efb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:21.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:21.296+0000 7f8d14850640 1 -- 172.21.15.46:0/3226846791 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8d1019f620 con 0x7f8d10104b00 2024-03-31T23:11:21.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:21.296+0000 7f8d14850640 1 --2- 172.21.15.46:0/3226846791 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8d10104b00 0x7f8d10197f40 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f8d04000c00 tx=0x7f8d04005dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:11:21.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:21.296+0000 7f8d027fc640 1 -- 172.21.15.46:0/3226846791 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d04043020 con 0x7f8d10104b00 2024-03-31T23:11:21.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:21.296+0000 7f8d027fc640 1 -- 172.21.15.46:0/3226846791 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8d04004350 con 0x7f8d10104b00 2024-03-31T23:11:21.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:21.296+0000 7f8d172b5640 1 -- 172.21.15.46:0/3226846791 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8d1019f8b0 con 0x7f8d10104b00 2024-03-31T23:11:21.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:21.296+0000 7f8d027fc640 1 -- 172.21.15.46:0/3226846791 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d04003b80 con 0x7f8d10104b00 2024-03-31T23:11:21.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:21.296+0000 7f8d172b5640 1 -- 172.21.15.46:0/3226846791 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8d1019fe50 con 0x7f8d10104b00 2024-03-31T23:11:21.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:21.297+0000 7f8d172b5640 1 -- 172.21.15.46:0/3226846791 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8d100fd6d0 con 0x7f8d10104b00 2024-03-31T23:11:21.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:21.298+0000 7f8d027fc640 1 -- 172.21.15.46:0/3226846791 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 18) ==== 137320+0+0 (secure 0 0 0) 0x7f8d04046020 con 0x7f8d10104b00 2024-03-31T23:11:21.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:21.299+0000 7f8d027fc640 1 --2- 172.21.15.46:0/3226846791 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f8cfc0a2520 0x7f8cfc0a49e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:21.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:21.299+0000 7f8d15051640 1 --2- 172.21.15.46:0/3226846791 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f8cfc0a2520 0x7f8cfc0a49e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:21.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:21.299+0000 7f8d027fc640 1 -- 172.21.15.46:0/3226846791 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9208+0+0 (secure 0 0 0) 0x7f8d040f0b90 con 0x7f8d10104b00 2024-03-31T23:11:21.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:21.299+0000 7f8d15051640 1 --2- 172.21.15.46:0/3226846791 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f8cfc0a2520 0x7f8cfc0a49e0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f8d0c009bd0 tx=0x7f8d0c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:11:21.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:21.303+0000 7f8d027fc640 1 -- 172.21.15.46:0/3226846791 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f8d0404d020 con 0x7f8d10104b00 2024-03-31T23:11:21.484 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:21.484+0000 7f8d172b5640 1 -- 172.21.15.46:0/3226846791 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f8d10062350 con 0x7f8d10104b00 2024-03-31T23:11:21.485 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:21.485+0000 7f8d027fc640 1 -- 172.21.15.46:0/3226846791 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v95) ==== 103+0+11 (secure 0 0 0) 0x7f8d040bd920 con 0x7f8d10104b00 2024-03-31T23:11:21.485 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 32 2024-03-31T23:11:21.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:21.487+0000 7f8d172b5640 1 -- 172.21.15.46:0/3226846791 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f8cfc0a2520 msgr2=0x7f8cfc0a49e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:21.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:21.487+0000 7f8d172b5640 1 --2- 172.21.15.46:0/3226846791 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f8cfc0a2520 0x7f8cfc0a49e0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f8d0c009bd0 tx=0x7f8d0c005f90 comp rx=0 tx=0).stop 2024-03-31T23:11:21.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:21.487+0000 7f8d172b5640 1 -- 172.21.15.46:0/3226846791 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8d10104b00 msgr2=0x7f8d10197f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:21.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:21.487+0000 7f8d172b5640 1 --2- 172.21.15.46:0/3226846791 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8d10104b00 0x7f8d10197f40 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f8d04000c00 tx=0x7f8d04005dc0 comp rx=0 tx=0).stop 2024-03-31T23:11:21.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:21.487+0000 7f8d172b5640 1 -- 172.21.15.46:0/3226846791 shutdown_connections 2024-03-31T23:11:21.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:21.487+0000 7f8d172b5640 1 --2- 172.21.15.46:0/3226846791 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f8cfc0a2520 0x7f8cfc0a49e0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:21.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:21.487+0000 7f8d172b5640 1 --2- 172.21.15.46:0/3226846791 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8d10107430 0x7f8d1019efb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:21.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:21.487+0000 7f8d172b5640 1 --2- 172.21.15.46:0/3226846791 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8d10104b00 0x7f8d10197f40 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:21.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:21.487+0000 7f8d172b5640 1 --2- 172.21.15.46:0/3226846791 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8d101021d0 0x7f8d10197a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:21.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:21.487+0000 7f8d172b5640 1 -- 172.21.15.46:0/3226846791 >> 172.21.15.46:0/3226846791 conn(0x7f8d100fbbe0 msgr2=0x7f8d10104060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:11:21.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:21.487+0000 7f8d172b5640 1 -- 172.21.15.46:0/3226846791 shutdown_connections 2024-03-31T23:11:21.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:21.487+0000 7f8d172b5640 1 -- 172.21.15.46:0/3226846791 wait complete. 2024-03-31T23:11:21.498 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-31T23:11:21.606 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:21.601+0000 7ffa5d7e5640 1 -- 172.21.15.46:0/1322610045 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffa58104ba0 msgr2=0x7ffa58106f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:21.606 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:21.601+0000 7ffa5d7e5640 1 --2- 172.21.15.46:0/1322610045 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffa58104ba0 0x7ffa58106f90 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7ffa40009900 tx=0x7ffa4002bc30 comp rx=0 tx=0).stop 2024-03-31T23:11:21.607 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:21.601+0000 7ffa5d7e5640 1 -- 172.21.15.46:0/1322610045 shutdown_connections 2024-03-31T23:11:21.607 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:21.601+0000 7ffa5d7e5640 1 --2- 172.21.15.46:0/1322610045 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ffa581074d0 0x7ffa581098c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:21.607 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:21.601+0000 7ffa5d7e5640 1 --2- 172.21.15.46:0/1322610045 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffa58104ba0 0x7ffa58106f90 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:21.607 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:21.601+0000 7ffa5d7e5640 1 --2- 172.21.15.46:0/1322610045 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ffa58102270 0x7ffa58104660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:21.607 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:21.601+0000 7ffa5d7e5640 1 -- 172.21.15.46:0/1322610045 >> 172.21.15.46:0/1322610045 conn(0x7ffa580fbbe0 msgr2=0x7ffa580fe020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:11:21.607 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:21.601+0000 7ffa5d7e5640 1 -- 172.21.15.46:0/1322610045 shutdown_connections 2024-03-31T23:11:21.607 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:21.601+0000 7ffa5d7e5640 1 -- 172.21.15.46:0/1322610045 wait complete. 2024-03-31T23:11:21.608 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:21.602+0000 7ffa5d7e5640 1 Processor -- start 2024-03-31T23:11:21.608 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:21.602+0000 7ffa5d7e5640 1 -- start start 2024-03-31T23:11:21.608 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:21.602+0000 7ffa5d7e5640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ffa58102270 0x7ffa581979e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:21.608 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:21.602+0000 7ffa5d7e5640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ffa58104ba0 0x7ffa58197f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:21.608 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:21.602+0000 7ffa5d7e5640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffa581074d0 0x7ffa5819ef90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:21.608 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:21.602+0000 7ffa5d7e5640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7ffa5810d2e0 con 0x7ffa581074d0 2024-03-31T23:11:21.608 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:21.602+0000 7ffa5d7e5640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7ffa5810d160 con 0x7ffa58102270 2024-03-31T23:11:21.609 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:21.602+0000 7ffa5d7e5640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7ffa5810d460 con 0x7ffa58104ba0 2024-03-31T23:11:21.609 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:21.602+0000 7ffa577fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffa581074d0 0x7ffa5819ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:21.609 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:21.602+0000 7ffa577fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffa581074d0 0x7ffa5819ef90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:58118/0 (socket says 172.21.15.46:58118) 2024-03-31T23:11:21.609 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:21.602+0000 7ffa577fe640 1 -- 172.21.15.46:0/934170579 learned_addr learned my addr 172.21.15.46:0/934170579 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:11:21.609 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:21.602+0000 7ffa56ffd640 1 --2- 172.21.15.46:0/934170579 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ffa58102270 0x7ffa581979e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:21.609 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:21.602+0000 7ffa567fc640 1 --2- 172.21.15.46:0/934170579 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ffa58104ba0 0x7ffa58197f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:21.610 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:21.602+0000 7ffa577fe640 1 -- 172.21.15.46:0/934170579 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ffa58104ba0 msgr2=0x7ffa58197f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:21.610 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:21.602+0000 7ffa577fe640 1 --2- 172.21.15.46:0/934170579 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ffa58104ba0 0x7ffa58197f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:21.610 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:21.602+0000 7ffa577fe640 1 -- 172.21.15.46:0/934170579 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ffa58102270 msgr2=0x7ffa581979e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:21.610 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:21.602+0000 7ffa577fe640 1 --2- 172.21.15.46:0/934170579 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ffa58102270 0x7ffa581979e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:21.610 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:21.602+0000 7ffa577fe640 1 -- 172.21.15.46:0/934170579 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa5819f600 con 0x7ffa581074d0 2024-03-31T23:11:21.610 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:21.603+0000 7ffa577fe640 1 --2- 172.21.15.46:0/934170579 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffa581074d0 0x7ffa5819ef90 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7ffa4800eb10 tx=0x7ffa4800c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:11:21.611 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:21.603+0000 7ffa4ffff640 1 -- 172.21.15.46:0/934170579 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa48004b20 con 0x7ffa581074d0 2024-03-31T23:11:21.611 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:21.603+0000 7ffa5d7e5640 1 -- 172.21.15.46:0/934170579 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffa5819f8f0 con 0x7ffa581074d0 2024-03-31T23:11:21.611 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:21.603+0000 7ffa4ffff640 1 -- 172.21.15.46:0/934170579 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ffa48004cc0 con 0x7ffa581074d0 2024-03-31T23:11:21.611 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:21.603+0000 7ffa4ffff640 1 -- 172.21.15.46:0/934170579 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa48010690 con 0x7ffa581074d0 2024-03-31T23:11:21.611 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:21.603+0000 7ffa5d7e5640 1 -- 172.21.15.46:0/934170579 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffa5819fdc0 con 0x7ffa581074d0 2024-03-31T23:11:21.611 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:21.604+0000 7ffa5d7e5640 1 -- 172.21.15.46:0/934170579 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffa580fd6d0 con 0x7ffa581074d0 2024-03-31T23:11:21.612 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:21.605+0000 7ffa4ffff640 1 -- 172.21.15.46:0/934170579 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 18) ==== 137320+0+0 (secure 0 0 0) 0x7ffa48005020 con 0x7ffa581074d0 2024-03-31T23:11:21.612 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:21.605+0000 7ffa4ffff640 1 --2- 172.21.15.46:0/934170579 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7ffa440a24a0 0x7ffa440a4960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:21.612 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:21.605+0000 7ffa56ffd640 1 --2- 172.21.15.46:0/934170579 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7ffa440a24a0 0x7ffa440a4960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:21.612 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:21.605+0000 7ffa4ffff640 1 -- 172.21.15.46:0/934170579 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9208+0+0 (secure 0 0 0) 0x7ffa480cf080 con 0x7ffa581074d0 2024-03-31T23:11:21.612 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:21.606+0000 7ffa56ffd640 1 --2- 172.21.15.46:0/934170579 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7ffa440a24a0 0x7ffa440a4960 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7ffa50009bd0 tx=0x7ffa50005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:11:21.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:21.610+0000 7ffa4ffff640 1 -- 172.21.15.46:0/934170579 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7ffa4809be70 con 0x7ffa581074d0 2024-03-31T23:11:21.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:21.830+0000 7ffa5d7e5640 1 -- 172.21.15.46:0/934170579 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ffa5802d170 con 0x7ffa581074d0 2024-03-31T23:11:21.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:21.831+0000 7ffa4ffff640 1 -- 172.21.15.46:0/934170579 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+11882 (secure 0 0 0) 0x7ffa4809b770 con 0x7ffa581074d0 2024-03-31T23:11:21.831 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-31T23:11:21.831 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/2169018547","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":2169018547},{"type":"v1","addr":"172.21.15.46:6835","nonce":2169018547}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6837/1069387756","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6836","nonce":1069387756},{"type":"v1","addr":"172.21.15.46:6837","nonce":1069387756}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14634,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6839/3915337844","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6838","nonce":3915337844},{"type":"v1","addr":"172.21.15.46:6839","nonce":3915337844}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14646,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6841/897767145","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6840","nonce":897767145},{"type":"v1","addr":"172.21.15.46:6841","nonce":897767145}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24415,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6833/1746950966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6832","nonce":1746950966},{"type":"v1","addr":"172.21.15.162:6833","nonce":1746950966}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24422,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6833/3394739181","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6832","nonce":3394739181},{"type":"v1","addr":"172.21.15.142:6833","nonce":3394739181}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":24,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-31T23:09:11.009090+0000","modified":"2024-03-31T23:09:31.392282+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24515,"mds_1":24482,"mds_2":24451},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24442":{"gid":24442,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.162:6835/1957208984","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6834","nonce":1957208984},{"type":"v1","addr":"172.21.15.162:6835","nonce":1957208984}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24451":{"gid":24451,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":11,"addr":"172.21.15.162:6837/1371886876","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6836","nonce":1371886876},{"type":"v1","addr":"172.21.15.162:6837","nonce":1371886876}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24455":{"gid":24455,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.142:6835/212637431","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6834","nonce":212637431},{"type":"v1","addr":"172.21.15.142:6835","nonce":212637431}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24478":{"gid":24478,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.162:6839/1976275826","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6838","nonce":1976275826},{"type":"v1","addr":"172.21.15.162:6839","nonce":1976275826}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24482":{"gid":24482,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":10,"addr":"172.21.15.142:6837/2559258399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6836","nonce":2559258399},{"type":"v1","addr":"172.21.15.142:6837","nonce":2559258399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24515":{"gid":24515,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.142:6839/2519143918","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6838","nonce":2519143918},{"type":"v1","addr":"172.21.15.142:6839","nonce":2519143918}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24515,"qdb_cluster":[24451,24482,24515]},"id":1}]} 2024-03-31T23:11:21.831 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 24 2024-03-31T23:11:21.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:21.832+0000 7ffa5d7e5640 1 -- 172.21.15.46:0/934170579 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7ffa440a24a0 msgr2=0x7ffa440a4960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:21.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:21.832+0000 7ffa5d7e5640 1 --2- 172.21.15.46:0/934170579 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7ffa440a24a0 0x7ffa440a4960 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7ffa50009bd0 tx=0x7ffa50005f90 comp rx=0 tx=0).stop 2024-03-31T23:11:21.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:21.832+0000 7ffa5d7e5640 1 -- 172.21.15.46:0/934170579 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffa581074d0 msgr2=0x7ffa5819ef90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:21.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:21.832+0000 7ffa5d7e5640 1 --2- 172.21.15.46:0/934170579 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffa581074d0 0x7ffa5819ef90 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7ffa4800eb10 tx=0x7ffa4800c2d0 comp rx=0 tx=0).stop 2024-03-31T23:11:21.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:21.833+0000 7ffa5d7e5640 1 -- 172.21.15.46:0/934170579 shutdown_connections 2024-03-31T23:11:21.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:21.833+0000 7ffa5d7e5640 1 --2- 172.21.15.46:0/934170579 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7ffa440a24a0 0x7ffa440a4960 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:21.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:21.833+0000 7ffa5d7e5640 1 --2- 172.21.15.46:0/934170579 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffa581074d0 0x7ffa5819ef90 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:21.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:21.833+0000 7ffa5d7e5640 1 --2- 172.21.15.46:0/934170579 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ffa58104ba0 0x7ffa58197f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:21.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:21.833+0000 7ffa5d7e5640 1 --2- 172.21.15.46:0/934170579 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ffa58102270 0x7ffa581979e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:21.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:21.833+0000 7ffa5d7e5640 1 -- 172.21.15.46:0/934170579 >> 172.21.15.46:0/934170579 conn(0x7ffa580fbbe0 msgr2=0x7ffa58102cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:11:21.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:21.833+0000 7ffa5d7e5640 1 -- 172.21.15.46:0/934170579 shutdown_connections 2024-03-31T23:11:21.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:21.833+0000 7ffa5d7e5640 1 -- 172.21.15.46:0/934170579 wait complete. 2024-03-31T23:11:21.843 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:11:21 smithi046 ceph-mon[32498]: from='client.? 172.21.15.46:0/3538435567' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-31T23:11:21.843 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:11:21 smithi046 ceph-mon[32498]: from='client.? 172.21.15.46:0/3559172068' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-31T23:11:21.843 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:11:21 smithi046 ceph-mon[32498]: from='client.? 172.21.15.46:0/3226846791' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-31T23:11:21.844 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-31T23:11:21.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:11:21 smithi142 ceph-mon[35271]: from='client.? 172.21.15.46:0/3538435567' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-31T23:11:21.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:11:21 smithi142 ceph-mon[35271]: from='client.? 172.21.15.46:0/3559172068' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-31T23:11:21.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:11:21 smithi142 ceph-mon[35271]: from='client.? 172.21.15.46:0/3226846791' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-31T23:11:21.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:21.946+0000 7f2ca3e98640 1 -- 172.21.15.46:0/4041492681 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2c9c1033f0 msgr2=0x7f2c9c109c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:21.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:21.946+0000 7f2ca3e98640 1 --2- 172.21.15.46:0/4041492681 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2c9c1033f0 0x7f2c9c109c80 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f2c94009cd0 tx=0x7f2c9402c0f0 comp rx=0 tx=0).stop 2024-03-31T23:11:21.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:21.946+0000 7f2ca3e98640 1 -- 172.21.15.46:0/4041492681 shutdown_connections 2024-03-31T23:11:21.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:21.946+0000 7f2ca3e98640 1 --2- 172.21.15.46:0/4041492681 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2c9c1033f0 0x7f2c9c109c80 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:21.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:21.946+0000 7f2ca3e98640 1 --2- 172.21.15.46:0/4041492681 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2c9c102a30 0x7f2c9c102eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:21.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:21.946+0000 7f2ca3e98640 1 --2- 172.21.15.46:0/4041492681 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2c9c0ffa60 0x7f2c9c0ffe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:21.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:21.946+0000 7f2ca3e98640 1 -- 172.21.15.46:0/4041492681 >> 172.21.15.46:0/4041492681 conn(0x7f2c9c0fb390 msgr2=0x7f2c9c0fd7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:11:21.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:21.946+0000 7f2ca3e98640 1 -- 172.21.15.46:0/4041492681 shutdown_connections 2024-03-31T23:11:21.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:21.946+0000 7f2ca3e98640 1 -- 172.21.15.46:0/4041492681 wait complete. 2024-03-31T23:11:21.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:21.947+0000 7f2ca3e98640 1 Processor -- start 2024-03-31T23:11:21.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:21.947+0000 7f2ca3e98640 1 -- start start 2024-03-31T23:11:21.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:21.947+0000 7f2ca3e98640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2c9c0ffa60 0x7f2c9c197990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:21.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:21.947+0000 7f2ca3e98640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2c9c102a30 0x7f2c9c197ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:21.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:21.947+0000 7f2ca3e98640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2c9c1033f0 0x7f2c9c19ef40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:21.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:21.947+0000 7f2ca3e98640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f2c9c10d2e0 con 0x7f2c9c102a30 2024-03-31T23:11:21.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:21.947+0000 7f2ca3e98640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f2c9c10d160 con 0x7f2c9c0ffa60 2024-03-31T23:11:21.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:21.947+0000 7f2ca3e98640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f2c9c10d460 con 0x7f2c9c1033f0 2024-03-31T23:11:21.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:21.947+0000 7f2ca1433640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2c9c102a30 0x7f2c9c197ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:21.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:21.947+0000 7f2ca1433640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2c9c102a30 0x7f2c9c197ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:58126/0 (socket says 172.21.15.46:58126) 2024-03-31T23:11:21.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:21.947+0000 7f2ca1433640 1 -- 172.21.15.46:0/2810218097 learned_addr learned my addr 172.21.15.46:0/2810218097 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:11:21.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:21.947+0000 7f2ca2435640 1 --2- 172.21.15.46:0/2810218097 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2c9c1033f0 0x7f2c9c19ef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:21.950 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:21.947+0000 7f2ca1c34640 1 --2- 172.21.15.46:0/2810218097 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2c9c0ffa60 0x7f2c9c197990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:21.950 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:21.947+0000 7f2ca1433640 1 -- 172.21.15.46:0/2810218097 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2c9c1033f0 msgr2=0x7f2c9c19ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:21.950 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:21.947+0000 7f2ca1433640 1 --2- 172.21.15.46:0/2810218097 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2c9c1033f0 0x7f2c9c19ef40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:21.950 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:21.947+0000 7f2ca1433640 1 -- 172.21.15.46:0/2810218097 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2c9c0ffa60 msgr2=0x7f2c9c197990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:21.950 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:21.947+0000 7f2ca1433640 1 --2- 172.21.15.46:0/2810218097 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2c9c0ffa60 0x7f2c9c197990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:21.950 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:21.947+0000 7f2ca1433640 1 -- 172.21.15.46:0/2810218097 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2c9c19f5b0 con 0x7f2c9c102a30 2024-03-31T23:11:21.950 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:21.947+0000 7f2ca1433640 1 --2- 172.21.15.46:0/2810218097 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2c9c102a30 0x7f2c9c197ed0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f2c8c00e9f0 tx=0x7f2c8c00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:11:21.950 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:21.947+0000 7f2c92ffd640 1 -- 172.21.15.46:0/2810218097 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c8c00cd80 con 0x7f2c9c102a30 2024-03-31T23:11:21.951 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:21.948+0000 7f2c92ffd640 1 -- 172.21.15.46:0/2810218097 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f2c8c0108e0 con 0x7f2c9c102a30 2024-03-31T23:11:21.951 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:21.948+0000 7f2ca3e98640 1 -- 172.21.15.46:0/2810218097 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2c9c19f8a0 con 0x7f2c9c102a30 2024-03-31T23:11:21.951 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:21.948+0000 7f2c92ffd640 1 -- 172.21.15.46:0/2810218097 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c8c018920 con 0x7f2c9c102a30 2024-03-31T23:11:21.951 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:21.948+0000 7f2ca3e98640 1 -- 172.21.15.46:0/2810218097 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2c9c19fdf0 con 0x7f2c9c102a30 2024-03-31T23:11:21.952 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:21.948+0000 7f2ca3e98640 1 -- 172.21.15.46:0/2810218097 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2c9c108c20 con 0x7f2c9c102a30 2024-03-31T23:11:21.952 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:21.950+0000 7f2c92ffd640 1 -- 172.21.15.46:0/2810218097 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 18) ==== 137320+0+0 (secure 0 0 0) 0x7f2c8c010a90 con 0x7f2c9c102a30 2024-03-31T23:11:21.952 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:21.951+0000 7f2c92ffd640 1 --2- 172.21.15.46:0/2810218097 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f2c880a24a0 0x7f2c880a4960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:21.952 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:21.951+0000 7f2ca1c34640 1 --2- 172.21.15.46:0/2810218097 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f2c880a24a0 0x7f2c880a4960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:21.952 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:21.951+0000 7f2c92ffd640 1 -- 172.21.15.46:0/2810218097 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9208+0+0 (secure 0 0 0) 0x7f2c8c0cf280 con 0x7f2c9c102a30 2024-03-31T23:11:21.952 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:21.951+0000 7f2ca1c34640 1 --2- 172.21.15.46:0/2810218097 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f2c880a24a0 0x7f2c880a4960 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f2c9c069bc0 tx=0x7f2c98005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:11:21.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:21.955+0000 7f2c92ffd640 1 -- 172.21.15.46:0/2810218097 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f2c8c09ba20 con 0x7f2c9c102a30 2024-03-31T23:11:22.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:11:21 smithi162 ceph-mon[36237]: from='client.? 172.21.15.46:0/3538435567' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-31T23:11:22.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:11:21 smithi162 ceph-mon[36237]: from='client.? 172.21.15.46:0/3559172068' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-31T23:11:22.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:11:21 smithi162 ceph-mon[36237]: from='client.? 172.21.15.46:0/3226846791' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-31T23:11:22.137 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.137+0000 7f2ca3e98640 1 -- 172.21.15.46:0/2810218097 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f2c9c02d3c0 con 0x7f2c9c102a30 2024-03-31T23:11:22.138 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.138+0000 7f2c92ffd640 1 -- 172.21.15.46:0/2810218097 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v95) ==== 74+0+24819 (secure 0 0 0) 0x7f2c8c09fac0 con 0x7f2c9c102a30 2024-03-31T23:11:22.139 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-31T23:11:22.139 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":95,"fsid":"6a9d8f08-efb2-11ee-b647-cb9ed24678a4","created":"2024-03-31T23:01:15.463711+0000","modified":"2024-03-31T23:11:10.531495+0000","last_up_change":"2024-03-31T23:08:20.199888+0000","last_in_change":"2024-03-31T23:07:59.438035+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-31T23:04:32.823751+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-31T23:09:09.145711+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-31T23:09:10.148579+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-31T23:09:13.419519+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"91","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"47429095-3f46-4ed7-b7c4-504464719815","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":1824772695},{"type":"v1","addr":"172.21.15.46:6803","nonce":1824772695}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":1824772695},{"type":"v1","addr":"172.21.15.46:6805","nonce":1824772695}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":1824772695},{"type":"v1","addr":"172.21.15.46:6809","nonce":1824772695}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":1824772695},{"type":"v1","addr":"172.21.15.46:6807","nonce":1824772695}]},"public_addr":"172.21.15.46:6803/1824772695","cluster_addr":"172.21.15.46:6805/1824772695","heartbeat_back_addr":"172.21.15.46:6809/1824772695","heartbeat_front_addr":"172.21.15.46:6807/1824772695","state":["exists","up"]},{"osd":1,"uuid":"be56f883-411e-4dbe-b50e-a158134df4a6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6800","nonce":973693922},{"type":"v1","addr":"172.21.15.142:6801","nonce":973693922}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6802","nonce":973693922},{"type":"v1","addr":"172.21.15.142:6803","nonce":973693922}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6806","nonce":973693922},{"type":"v1","addr":"172.21.15.142:6807","nonce":973693922}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6804","nonce":973693922},{"type":"v1","addr":"172.21.15.142:6805","nonce":973693922}]},"public_addr":"172.21.15.142:6801/973693922","cluster_addr":"172.21.15.142:6803/973693922","heartbeat_back_addr":"172.21.15.142:6807/973693922","heartbeat_front_addr":"172.21.15.142:6805/973693922","state":["exists","up"]},{"osd":2,"uuid":"c99a8794-ab19-4260-9a8d-4bc0bd2c8175","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6800","nonce":4084261813},{"type":"v1","addr":"172.21.15.162:6801","nonce":4084261813}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6802","nonce":4084261813},{"type":"v1","addr":"172.21.15.162:6803","nonce":4084261813}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6806","nonce":4084261813},{"type":"v1","addr":"172.21.15.162:6807","nonce":4084261813}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6804","nonce":4084261813},{"type":"v1","addr":"172.21.15.162:6805","nonce":4084261813}]},"public_addr":"172.21.15.162:6801/4084261813","cluster_addr":"172.21.15.162:6803/4084261813","heartbeat_back_addr":"172.21.15.162:6807/4084261813","heartbeat_front_addr":"172.21.15.162:6805/4084261813","state":["exists","up"]},{"osd":3,"uuid":"f877814e-9ef6-4872-a081-5609bb9ebc06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":3496317970},{"type":"v1","addr":"172.21.15.46:6811","nonce":3496317970}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":3496317970},{"type":"v1","addr":"172.21.15.46:6813","nonce":3496317970}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":3496317970},{"type":"v1","addr":"172.21.15.46:6817","nonce":3496317970}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":3496317970},{"type":"v1","addr":"172.21.15.46:6815","nonce":3496317970}]},"public_addr":"172.21.15.46:6811/3496317970","cluster_addr":"172.21.15.46:6813/3496317970","heartbeat_back_addr":"172.21.15.46:6817/3496317970","heartbeat_front_addr":"172.21.15.46:6815/3496317970","state":["exists","up"]},{"osd":4,"uuid":"beecec0a-8280-44fe-8f5f-8498a35eca8f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6808","nonce":2996266929},{"type":"v1","addr":"172.21.15.142:6809","nonce":2996266929}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6810","nonce":2996266929},{"type":"v1","addr":"172.21.15.142:6811","nonce":2996266929}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6814","nonce":2996266929},{"type":"v1","addr":"172.21.15.142:6815","nonce":2996266929}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6812","nonce":2996266929},{"type":"v1","addr":"172.21.15.142:6813","nonce":2996266929}]},"public_addr":"172.21.15.142:6809/2996266929","cluster_addr":"172.21.15.142:6811/2996266929","heartbeat_back_addr":"172.21.15.142:6815/2996266929","heartbeat_front_addr":"172.21.15.142:6813/2996266929","state":["exists","up"]},{"osd":5,"uuid":"9da7ff75-f658-45ea-a02f-1943fdda76b9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6808","nonce":914208293},{"type":"v1","addr":"172.21.15.162:6809","nonce":914208293}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6810","nonce":914208293},{"type":"v1","addr":"172.21.15.162:6811","nonce":914208293}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6814","nonce":914208293},{"type":"v1","addr":"172.21.15.162:6815","nonce":914208293}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6812","nonce":914208293},{"type":"v1","addr":"172.21.15.162:6813","nonce":914208293}]},"public_addr":"172.21.15.162:6809/914208293","cluster_addr":"172.21.15.162:6811/914208293","heartbeat_back_addr":"172.21.15.162:6815/914208293","heartbeat_front_addr":"172.21.15.162:6813/914208293","state":["exists","up"]},{"osd":6,"uuid":"8fa11224-cb26-4d3f-87d1-6a8f31de2a19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":56291966},{"type":"v1","addr":"172.21.15.46:6819","nonce":56291966}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":56291966},{"type":"v1","addr":"172.21.15.46:6821","nonce":56291966}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6824","nonce":56291966},{"type":"v1","addr":"172.21.15.46:6825","nonce":56291966}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":56291966},{"type":"v1","addr":"172.21.15.46:6823","nonce":56291966}]},"public_addr":"172.21.15.46:6819/56291966","cluster_addr":"172.21.15.46:6821/56291966","heartbeat_back_addr":"172.21.15.46:6825/56291966","heartbeat_front_addr":"172.21.15.46:6823/56291966","state":["exists","up"]},{"osd":7,"uuid":"0d0338b7-4d0a-4bb0-b152-8f5224871d9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6816","nonce":2146598446},{"type":"v1","addr":"172.21.15.142:6817","nonce":2146598446}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6818","nonce":2146598446},{"type":"v1","addr":"172.21.15.142:6819","nonce":2146598446}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6822","nonce":2146598446},{"type":"v1","addr":"172.21.15.142:6823","nonce":2146598446}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6820","nonce":2146598446},{"type":"v1","addr":"172.21.15.142:6821","nonce":2146598446}]},"public_addr":"172.21.15.142:6817/2146598446","cluster_addr":"172.21.15.142:6819/2146598446","heartbeat_back_addr":"172.21.15.142:6823/2146598446","heartbeat_front_addr":"172.21.15.142:6821/2146598446","state":["exists","up"]},{"osd":8,"uuid":"8b6f6037-892a-416c-babb-7552648d0b3a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6816","nonce":1173078318},{"type":"v1","addr":"172.21.15.162:6817","nonce":1173078318}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6818","nonce":1173078318},{"type":"v1","addr":"172.21.15.162:6819","nonce":1173078318}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6822","nonce":1173078318},{"type":"v1","addr":"172.21.15.162:6823","nonce":1173078318}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6820","nonce":1173078318},{"type":"v1","addr":"172.21.15.162:6821","nonce":1173078318}]},"public_addr":"172.21.15.162:6817/1173078318","cluster_addr":"172.21.15.162:6819/1173078318","heartbeat_back_addr":"172.21.15.162:6823/1173078318","heartbeat_front_addr":"172.21.15.162:6821/1173078318","state":["exists","up"]},{"osd":9,"uuid":"faccce25-6bf2-4220-8858-cda82b72be68","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":532708881},{"type":"v1","addr":"172.21.15.46:6827","nonce":532708881}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6828","nonce":532708881},{"type":"v1","addr":"172.21.15.46:6829","nonce":532708881}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":532708881},{"type":"v1","addr":"172.21.15.46:6833","nonce":532708881}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":532708881},{"type":"v1","addr":"172.21.15.46:6831","nonce":532708881}]},"public_addr":"172.21.15.46:6827/532708881","cluster_addr":"172.21.15.46:6829/532708881","heartbeat_back_addr":"172.21.15.46:6833/532708881","heartbeat_front_addr":"172.21.15.46:6831/532708881","state":["exists","up"]},{"osd":10,"uuid":"cdd356db-f9eb-4080-95ab-db5d8223e572","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6824","nonce":2845222366},{"type":"v1","addr":"172.21.15.142:6825","nonce":2845222366}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6826","nonce":2845222366},{"type":"v1","addr":"172.21.15.142:6827","nonce":2845222366}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6830","nonce":2845222366},{"type":"v1","addr":"172.21.15.142:6831","nonce":2845222366}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6828","nonce":2845222366},{"type":"v1","addr":"172.21.15.142:6829","nonce":2845222366}]},"public_addr":"172.21.15.142:6825/2845222366","cluster_addr":"172.21.15.142:6827/2845222366","heartbeat_back_addr":"172.21.15.142:6831/2845222366","heartbeat_front_addr":"172.21.15.142:6829/2845222366","state":["exists","up"]},{"osd":11,"uuid":"72104d1c-3ea2-45ad-bc0b-0500f146c06b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6824","nonce":1986323778},{"type":"v1","addr":"172.21.15.162:6825","nonce":1986323778}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6826","nonce":1986323778},{"type":"v1","addr":"172.21.15.162:6827","nonce":1986323778}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6830","nonce":1986323778},{"type":"v1","addr":"172.21.15.162:6831","nonce":1986323778}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6828","nonce":1986323778},{"type":"v1","addr":"172.21.15.162:6829","nonce":1986323778}]},"public_addr":"172.21.15.162:6825/1986323778","cluster_addr":"172.21.15.162:6827/1986323778","heartbeat_back_addr":"172.21.15.162:6831/1986323778","heartbeat_front_addr":"172.21.15.162:6829/1986323778","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:03:44.996331+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:04:07.994738+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:04:30.942255+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:04:55.588884+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:05:19.900600+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:05:43.807442+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:06:09.026976+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:06:34.586784+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:06:59.751605+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:07:26.144495+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:07:52.326909+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:08:17.423042+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.b","mappings":[{"from":8,"to":11}]},{"pgid":"3.6","mappings":[{"from":4,"to":7}]},{"pgid":"3.c","mappings":[{"from":3,"to":9}]},{"pgid":"3.1a","mappings":[{"from":5,"to":6}]},{"pgid":"3.1b","mappings":[{"from":4,"to":8}]},{"pgid":"3.1d","mappings":[{"from":4,"to":8}]},{"pgid":"3.2a","mappings":[{"from":3,"to":6}]},{"pgid":"3.33","mappings":[{"from":5,"to":9}]},{"pgid":"3.36","mappings":[{"from":4,"to":9}]},{"pgid":"4.6","mappings":[{"from":5,"to":8}]},{"pgid":"4.17","mappings":[{"from":9,"to":8}]},{"pgid":"4.18","mappings":[{"from":9,"to":8}]},{"pgid":"4.1b","mappings":[{"from":9,"to":10}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.46:0/4021039857":"2024-04-01T23:02:08.493619+0000","172.21.15.46:6801/3681182647":"2024-04-01T23:02:08.493619+0000","172.21.15.46:0/4022214806":"2024-04-01T23:02:08.493619+0000","172.21.15.46:6800/1515528215":"2024-04-01T23:01:41.910162+0000","172.21.15.46:6800/3681182647":"2024-04-01T23:02:08.493619+0000","172.21.15.46:0/143885707":"2024-04-01T23:01:41.910162+0000","172.21.15.46:6801/1515528215":"2024-04-01T23:01:41.910162+0000","172.21.15.46:0/1680389778":"2024-04-01T23:01:41.910162+0000","172.21.15.46:0/1782133966":"2024-04-01T23:01:41.910162+0000","172.21.15.46:0/3925209376":"2024-04-01T23:02:08.493619+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-31T23:11:22.141 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.140+0000 7f2ca3e98640 1 -- 172.21.15.46:0/2810218097 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f2c880a24a0 msgr2=0x7f2c880a4960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:22.141 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.140+0000 7f2ca3e98640 1 --2- 172.21.15.46:0/2810218097 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f2c880a24a0 0x7f2c880a4960 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f2c9c069bc0 tx=0x7f2c98005f90 comp rx=0 tx=0).stop 2024-03-31T23:11:22.141 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.140+0000 7f2ca3e98640 1 -- 172.21.15.46:0/2810218097 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2c9c102a30 msgr2=0x7f2c9c197ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:22.141 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.140+0000 7f2ca3e98640 1 --2- 172.21.15.46:0/2810218097 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2c9c102a30 0x7f2c9c197ed0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f2c8c00e9f0 tx=0x7f2c8c00eeb0 comp rx=0 tx=0).stop 2024-03-31T23:11:22.141 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.140+0000 7f2ca3e98640 1 -- 172.21.15.46:0/2810218097 shutdown_connections 2024-03-31T23:11:22.141 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.140+0000 7f2ca3e98640 1 --2- 172.21.15.46:0/2810218097 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f2c880a24a0 0x7f2c880a4960 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:22.141 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.140+0000 7f2ca3e98640 1 --2- 172.21.15.46:0/2810218097 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2c9c1033f0 0x7f2c9c19ef40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:22.141 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.140+0000 7f2ca3e98640 1 --2- 172.21.15.46:0/2810218097 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2c9c102a30 0x7f2c9c197ed0 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:22.141 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.140+0000 7f2ca3e98640 1 --2- 172.21.15.46:0/2810218097 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2c9c0ffa60 0x7f2c9c197990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:22.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.141+0000 7f2ca3e98640 1 -- 172.21.15.46:0/2810218097 >> 172.21.15.46:0/2810218097 conn(0x7f2c9c0fb390 msgr2=0x7f2c9c105df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:11:22.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.141+0000 7f2ca3e98640 1 -- 172.21.15.46:0/2810218097 shutdown_connections 2024-03-31T23:11:22.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.141+0000 7f2ca3e98640 1 -- 172.21.15.46:0/2810218097 wait complete. 2024-03-31T23:11:22.152 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 24, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-03-31T23:09:11.009090+0000', 'modified': '2024-03-31T23:09:31.392282+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 3, 'in': [0, 1, 2], 'up': {'mds_0': 24515, 'mds_1': 24482, 'mds_2': 24451}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24442': {'gid': 24442, 'name': 'f', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.162:6835/1957208984', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6834', 'nonce': 1957208984}, {'type': 'v1', 'addr': '172.21.15.162:6835', 'nonce': 1957208984}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24451': {'gid': 24451, 'name': 'c', 'rank': 2, 'incarnation': 22, 'state': 'up:active', 'state_seq': 11, 'addr': '172.21.15.162:6837/1371886876', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6836', 'nonce': 1371886876}, {'type': 'v1', 'addr': '172.21.15.162:6837', 'nonce': 1371886876}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24455': {'gid': 24455, 'name': 'e', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.142:6835/212637431', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.142:6834', 'nonce': 212637431}, {'type': 'v1', 'addr': '172.21.15.142:6835', 'nonce': 212637431}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24478': {'gid': 24478, 'name': 'i', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.162:6839/1976275826', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6838', 'nonce': 1976275826}, {'type': 'v1', 'addr': '172.21.15.162:6839', 'nonce': 1976275826}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24482': {'gid': 24482, 'name': 'h', 'rank': 1, 'incarnation': 20, 'state': 'up:active', 'state_seq': 10, 'addr': '172.21.15.142:6837/2559258399', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.142:6836', 'nonce': 2559258399}, {'type': 'v1', 'addr': '172.21.15.142:6837', 'nonce': 2559258399}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24515': {'gid': 24515, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.142:6839/2519143918', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.142:6838', 'nonce': 2519143918}, {'type': 'v1', 'addr': '172.21.15.142:6839', 'nonce': 2519143918}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24515, 'qdb_cluster': [24451, 24482, 24515]} 2024-03-31T23:11:22.153 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 3/3 2024-03-31T23:11:22.153 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-03-31T23:11:22.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.256+0000 7f8cd26d6640 1 -- 172.21.15.46:0/229905945 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8ccc0ff7c0 msgr2=0x7f8ccc0ffc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:22.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.256+0000 7f8cd26d6640 1 --2- 172.21.15.46:0/229905945 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8ccc0ff7c0 0x7f8ccc0ffc40 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f8cc0009900 tx=0x7f8cc002bc30 comp rx=0 tx=0).stop 2024-03-31T23:11:22.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.256+0000 7f8cd26d6640 1 -- 172.21.15.46:0/229905945 shutdown_connections 2024-03-31T23:11:22.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.256+0000 7f8cd26d6640 1 --2- 172.21.15.46:0/229905945 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8ccc100180 0x7f8ccc109d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:22.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.256+0000 7f8cd26d6640 1 --2- 172.21.15.46:0/229905945 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8ccc0ff7c0 0x7f8ccc0ffc40 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:22.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.256+0000 7f8cd26d6640 1 --2- 172.21.15.46:0/229905945 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8ccc102d50 0x7f8ccc103170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:22.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.256+0000 7f8cd26d6640 1 -- 172.21.15.46:0/229905945 >> 172.21.15.46:0/229905945 conn(0x7f8ccc0fb340 msgr2=0x7f8ccc0fd7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:11:22.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.256+0000 7f8cd26d6640 1 -- 172.21.15.46:0/229905945 shutdown_connections 2024-03-31T23:11:22.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.256+0000 7f8cd26d6640 1 -- 172.21.15.46:0/229905945 wait complete. 2024-03-31T23:11:22.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.257+0000 7f8cd26d6640 1 Processor -- start 2024-03-31T23:11:22.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.257+0000 7f8cd26d6640 1 -- start start 2024-03-31T23:11:22.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.257+0000 7f8cd26d6640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8ccc0ff7c0 0x7f8ccc197970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:22.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.257+0000 7f8cd26d6640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8ccc100180 0x7f8ccc197eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:22.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.257+0000 7f8cd26d6640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8ccc102d50 0x7f8ccc19ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:22.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.257+0000 7f8cd26d6640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f8ccc10d290 con 0x7f8ccc0ff7c0 2024-03-31T23:11:22.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.257+0000 7f8cd26d6640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f8ccc10d110 con 0x7f8ccc102d50 2024-03-31T23:11:22.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.257+0000 7f8cd26d6640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f8ccc10d410 con 0x7f8ccc100180 2024-03-31T23:11:22.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.257+0000 7f8ccbfff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8ccc0ff7c0 0x7f8ccc197970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:22.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.257+0000 7f8ccbfff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8ccc0ff7c0 0x7f8ccc197970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:58150/0 (socket says 172.21.15.46:58150) 2024-03-31T23:11:22.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.257+0000 7f8ccbfff640 1 -- 172.21.15.46:0/1816407577 learned_addr learned my addr 172.21.15.46:0/1816407577 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:11:22.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.257+0000 7f8ccb7fe640 1 --2- 172.21.15.46:0/1816407577 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8ccc100180 0x7f8ccc197eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:22.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.257+0000 7f8cd0c73640 1 --2- 172.21.15.46:0/1816407577 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8ccc102d50 0x7f8ccc19ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:22.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.257+0000 7f8ccbfff640 1 -- 172.21.15.46:0/1816407577 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8ccc100180 msgr2=0x7f8ccc197eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:22.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.257+0000 7f8ccbfff640 1 --2- 172.21.15.46:0/1816407577 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8ccc100180 0x7f8ccc197eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:22.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.257+0000 7f8ccbfff640 1 -- 172.21.15.46:0/1816407577 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8ccc102d50 msgr2=0x7f8ccc19ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:22.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.257+0000 7f8ccbfff640 1 --2- 172.21.15.46:0/1816407577 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8ccc102d50 0x7f8ccc19ef20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:22.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.257+0000 7f8ccbfff640 1 -- 172.21.15.46:0/1816407577 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ccc19f590 con 0x7f8ccc0ff7c0 2024-03-31T23:11:22.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.258+0000 7f8ccbfff640 1 --2- 172.21.15.46:0/1816407577 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8ccc0ff7c0 0x7f8ccc197970 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f8cbc00e9c0 tx=0x7f8cbc00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:11:22.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.258+0000 7f8cc97fa640 1 -- 172.21.15.46:0/1816407577 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8cbc00cde0 con 0x7f8ccc0ff7c0 2024-03-31T23:11:22.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.258+0000 7f8cc97fa640 1 -- 172.21.15.46:0/1816407577 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8cbc0108e0 con 0x7f8ccc0ff7c0 2024-03-31T23:11:22.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.258+0000 7f8cd26d6640 1 -- 172.21.15.46:0/1816407577 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8ccc19f880 con 0x7f8ccc0ff7c0 2024-03-31T23:11:22.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.258+0000 7f8cc97fa640 1 -- 172.21.15.46:0/1816407577 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8cbc018990 con 0x7f8ccc0ff7c0 2024-03-31T23:11:22.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.258+0000 7f8cd26d6640 1 -- 172.21.15.46:0/1816407577 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8ccc19fda0 con 0x7f8ccc0ff7c0 2024-03-31T23:11:22.268 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.260+0000 7f8cc97fa640 1 -- 172.21.15.46:0/1816407577 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 18) ==== 137320+0+0 (secure 0 0 0) 0x7f8cbc01f020 con 0x7f8ccc0ff7c0 2024-03-31T23:11:22.268 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.261+0000 7f8cd538b740 1 Processor -- start 2024-03-31T23:11:22.268 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.261+0000 7f8cd538b740 1 -- start start 2024-03-31T23:11:22.268 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.261+0000 7f8cd538b740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a20d6db610 0x55a20d6dba30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:22.268 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.261+0000 7f8cd538b740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a20d6dc7e0 0x55a20d6dcc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:22.268 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.261+0000 7f8cd538b740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a20d6dd1a0 0x55a20d6e3a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:22.269 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.261+0000 7f8cd538b740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55a20d63faf0 con 0x55a20d6dc7e0 2024-03-31T23:11:22.269 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.261+0000 7f8cd538b740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55a20d5e89d0 con 0x55a20d6dd1a0 2024-03-31T23:11:22.269 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.261+0000 7f8cd538b740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55a20d63bb90 con 0x55a20d6db610 2024-03-31T23:11:22.269 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.261+0000 7f8cc97fa640 1 --2- 172.21.15.46:0/1816407577 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f8cb40a24a0 0x7f8cb40a4960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:22.269 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.261+0000 7f8ccbfff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a20d6dc7e0 0x55a20d6dcc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:22.269 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.261+0000 7f8cd0c73640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a20d6db610 0x55a20d6dba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:22.270 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.261+0000 7f8cd0c73640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a20d6db610 0x55a20d6dba30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.46:33900/0 (socket says 172.21.15.46:33900) 2024-03-31T23:11:22.270 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.261+0000 7f8ccbfff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a20d6dc7e0 0x55a20d6dcc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:58154/0 (socket says 172.21.15.46:58154) 2024-03-31T23:11:22.270 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.261+0000 7f8cd0c73640 1 -- 172.21.15.46:0/3486414194 learned_addr learned my addr 172.21.15.46:0/3486414194 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:11:22.270 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.261+0000 7f8ccb7fe640 1 --2- 172.21.15.46:0/3486414194 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a20d6dd1a0 0x55a20d6e3a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:22.270 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.261+0000 7f8cd0c73640 1 --2- 172.21.15.46:0/1816407577 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f8cb40a24a0 0x7f8cb40a4960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:22.270 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.261+0000 7f8cc97fa640 1 -- 172.21.15.46:0/1816407577 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9208+0+0 (secure 0 0 0) 0x7f8cbc0cecc0 con 0x7f8ccc0ff7c0 2024-03-31T23:11:22.271 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.261+0000 7f8cd0c73640 1 -- 172.21.15.46:0/3486414194 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a20d6dd1a0 msgr2=0x55a20d6e3a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:11:22.271 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.261+0000 7f8cd0c73640 1 --2- 172.21.15.46:0/3486414194 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a20d6dd1a0 0x55a20d6e3a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:22.271 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.261+0000 7f8cd0c73640 1 -- 172.21.15.46:0/3486414194 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a20d6dc7e0 msgr2=0x55a20d6dcc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:11:22.271 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.261+0000 7f8cd0c73640 1 --2- 172.21.15.46:0/3486414194 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a20d6dc7e0 0x55a20d6dcc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:22.271 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.261+0000 7f8cd0c73640 1 -- 172.21.15.46:0/3486414194 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a20d3542d0 con 0x55a20d6db610 2024-03-31T23:11:22.271 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.262+0000 7f8cd0c73640 1 --2- 172.21.15.46:0/3486414194 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a20d6db610 0x55a20d6dba30 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f8cc4011e20 tx=0x7f8cc400f790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=187d34b3a629b2ce server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:11:22.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.262+0000 7f8cd0c73640 1 --2- 172.21.15.46:0/1816407577 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f8cb40a24a0 0x7f8cb40a4960 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f8cc400fc90 tx=0x7f8cc401e000 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:11:22.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.262+0000 7f8cb27fc640 1 -- 172.21.15.46:0/3486414194 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8cc401dcb0 con 0x55a20d6db610 2024-03-31T23:11:22.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.262+0000 7f8cb27fc640 1 -- 172.21.15.46:0/3486414194 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8cc401de50 con 0x55a20d6db610 2024-03-31T23:11:22.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.262+0000 7f8cb27fc640 1 -- 172.21.15.46:0/3486414194 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8cc4004280 con 0x55a20d6db610 2024-03-31T23:11:22.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.262+0000 7f8cd538b740 1 -- 172.21.15.46:0/3486414194 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a20d6db610 msgr2=0x55a20d6dba30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:22.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.262+0000 7f8cd538b740 1 --2- 172.21.15.46:0/3486414194 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a20d6db610 0x55a20d6dba30 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f8cc4011e20 tx=0x7f8cc400f790 comp rx=0 tx=0).stop 2024-03-31T23:11:22.273 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.262+0000 7f8cd538b740 1 -- 172.21.15.46:0/3486414194 shutdown_connections 2024-03-31T23:11:22.273 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.262+0000 7f8cd538b740 1 --2- 172.21.15.46:0/3486414194 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a20d6dd1a0 0x55a20d6e3a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:22.273 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.262+0000 7f8cd538b740 1 --2- 172.21.15.46:0/3486414194 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a20d6dc7e0 0x55a20d6dcc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:22.273 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.262+0000 7f8cd538b740 1 --2- 172.21.15.46:0/3486414194 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a20d6db610 0x55a20d6dba30 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:22.273 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.262+0000 7f8cd538b740 1 -- 172.21.15.46:0/3486414194 >> 172.21.15.46:0/3486414194 conn(0x55a20d6d6a00 msgr2=0x55a20d6d6e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:11:22.273 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.262+0000 7f8cd538b740 1 -- 172.21.15.46:0/3486414194 shutdown_connections 2024-03-31T23:11:22.273 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.262+0000 7f8cd538b740 1 -- 172.21.15.46:0/3486414194 wait complete. 2024-03-31T23:11:22.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.263+0000 7f8cd538b740 1 -- start start 2024-03-31T23:11:22.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.263+0000 7f8cd538b740 1 Processor -- start 2024-03-31T23:11:22.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.263+0000 7f8cd538b740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a20d6db610 0x55a20d6e58c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:22.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.263+0000 7f8cd538b740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a20d6dc7e0 0x55a20d6e5e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:22.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.263+0000 7f8cd538b740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a20d6dd1a0 0x55a20d6ece40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:22.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.263+0000 7f8cd538b740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55a20d6e40f0 con 0x55a20d6dc7e0 2024-03-31T23:11:22.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.263+0000 7f8cd538b740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55a20d5fbf70 con 0x55a20d6db610 2024-03-31T23:11:22.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.263+0000 7f8cd538b740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55a20d6e4270 con 0x55a20d6dd1a0 2024-03-31T23:11:22.275 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.263+0000 7f8cd538b740 10 client.0 _ll_register_callbacks cb 0x55a20d353ac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T23:11:22.275 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.263+0000 7f8ccb7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a20d6dc7e0 0x55a20d6e5e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:22.275 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.263+0000 7f8ccb7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a20d6dc7e0 0x55a20d6e5e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:58168/0 (socket says 172.21.15.46:58168) 2024-03-31T23:11:22.275 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.263+0000 7f8ccbfff640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a20d6db610 0x55a20d6e58c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:22.275 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.263+0000 7f8ccb7fe640 1 -- 172.21.15.46:0/2208960206 learned_addr learned my addr 172.21.15.46:0/2208960206 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:11:22.275 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.263+0000 7f8cd0c73640 1 --2- 172.21.15.46:0/2208960206 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a20d6dd1a0 0x55a20d6ece40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:22.275 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.264+0000 7f8ccb7fe640 1 -- 172.21.15.46:0/2208960206 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a20d6dd1a0 msgr2=0x55a20d6ece40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:11:22.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.264+0000 7f8ccb7fe640 1 --2- 172.21.15.46:0/2208960206 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a20d6dd1a0 0x55a20d6ece40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:22.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.264+0000 7f8ccb7fe640 1 -- 172.21.15.46:0/2208960206 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a20d6db610 msgr2=0x55a20d6e58c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:11:22.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.264+0000 7f8ccb7fe640 1 --2- 172.21.15.46:0/2208960206 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a20d6db610 0x55a20d6e58c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:22.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.264+0000 7f8ccb7fe640 1 -- 172.21.15.46:0/2208960206 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a20d41e6b0 con 0x55a20d6dc7e0 2024-03-31T23:11:22.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.264+0000 7f8ccb7fe640 1 --2- 172.21.15.46:0/2208960206 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a20d6dc7e0 0x55a20d6e5e00 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f8cc0009900 tx=0x7f8cc0004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aaafe9eadb8f9b34 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:11:22.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.264+0000 7f8cab7fe640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-31T23:11:22.277 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.264+0000 7f8cab7fe640 1 -- 172.21.15.46:0/2208960206 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8cc0005030 con 0x55a20d6dc7e0 2024-03-31T23:11:22.277 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.264+0000 7f8cab7fe640 1 -- 172.21.15.46:0/2208960206 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9208+0+0 (secure 0 0 0) 0x7f8cc00051d0 con 0x55a20d6dc7e0 2024-03-31T23:11:22.277 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.264+0000 7f8cd538b740 1 -- 172.21.15.46:0/2208960206 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8cc401de50 con 0x55a20d6dc7e0 2024-03-31T23:11:22.277 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.264+0000 7f8cab7fe640 1 -- 172.21.15.46:0/2208960206 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f8cc0052050 con 0x55a20d6dc7e0 2024-03-31T23:11:22.277 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.264+0000 7f8cd538b740 10 client.15408 fetch_fsmap learned FSMap version 24 2024-03-31T23:11:22.277 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.264+0000 7f8cd538b740 1 -- 172.21.15.46:0/2208960206 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55a20d6efee0 con 0x55a20d6dc7e0 2024-03-31T23:11:22.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.264+0000 7f8cab7fe640 1 -- 172.21.15.46:0/2208960206 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8cc401de50 con 0x55a20d6dc7e0 2024-03-31T23:11:22.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.264+0000 7f8cab7fe640 1 -- 172.21.15.46:0/2208960206 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f8cc00035f0 con 0x55a20d6dc7e0 2024-03-31T23:11:22.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.265+0000 7f8cd538b740 10 client.15408 fetch_fsmap finished waiting for FSMap version 24 2024-03-31T23:11:22.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.265+0000 7f8cd538b740 10 client.15408 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-31T23:11:22.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.265+0000 7f8cd538b740 20 client.15408 populate_metadata read hostname 'smithi046' 2024-03-31T23:11:22.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.265+0000 7f8cd538b740 1 --2- 172.21.15.46:0/2208960206 >> [v2:172.21.15.162:6836/1371886876,v1:172.21.15.162:6837/1371886876] conn(0x55a20d6f25e0 0x55a20d6f3a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:22.279 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.265+0000 7f8cd538b740 4 client.15408 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-31T23:11:22.279 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.265+0000 7f8cd538b740 1 -- 172.21.15.46:0/2208960206 --> [v2:172.21.15.162:6836/1371886876,v1:172.21.15.162:6837/1371886876] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a20d64d030 con 0x55a20d6f25e0 2024-03-31T23:11:22.279 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.265+0000 7f8ccbfff640 1 --2- 172.21.15.46:0/2208960206 >> [v2:172.21.15.162:6836/1371886876,v1:172.21.15.162:6837/1371886876] conn(0x55a20d6f25e0 0x55a20d6f3a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:22.279 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.265+0000 7f8ccbfff640 1 --2- 172.21.15.46:0/2208960206 >> [v2:172.21.15.162:6836/1371886876,v1:172.21.15.162:6837/1371886876] conn(0x55a20d6f25e0 0x55a20d6f3a50 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=ce00740947858965 server_cookie=607fc4dac6ac61e3 in_seq=0 out_seq=0 2024-03-31T23:11:22.279 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.265+0000 7f8cab7fe640 10 client.15408 ms_handle_connect on v2:172.21.15.162:6836/1371886876 2024-03-31T23:11:22.279 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.266+0000 7f8cab7fe640 1 -- 172.21.15.46:0/2208960206 <== mds.2 v2:172.21.15.162:6836/1371886876 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55a20d64d030 con 0x55a20d6f25e0 2024-03-31T23:11:22.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.266+0000 7f8cab7fe640 10 client.15408 handle_command_reply: tid=0 2024-03-31T23:11:22.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.266+0000 7f8cd538b740 1 client.15408 shutdown 2024-03-31T23:11:22.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.266+0000 7f8cd538b740 1 -- 172.21.15.46:0/2208960206 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a20d6dc7e0 msgr2=0x55a20d6e5e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:22.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.266+0000 7f8cd538b740 1 --2- 172.21.15.46:0/2208960206 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a20d6dc7e0 0x55a20d6e5e00 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f8cc0009900 tx=0x7f8cc0004270 comp rx=0 tx=0).stop 2024-03-31T23:11:22.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.266+0000 7f8cd538b740 1 -- 172.21.15.46:0/2208960206 shutdown_connections 2024-03-31T23:11:22.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.266+0000 7f8cd538b740 1 --2- 172.21.15.46:0/2208960206 >> [v2:172.21.15.162:6836/1371886876,v1:172.21.15.162:6837/1371886876] conn(0x55a20d6f25e0 0x55a20d6f3a50 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:22.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.266+0000 7f8cd538b740 1 --2- 172.21.15.46:0/2208960206 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a20d6dd1a0 0x55a20d6ece40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:22.281 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.266+0000 7f8cd538b740 1 --2- 172.21.15.46:0/2208960206 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a20d6dc7e0 0x55a20d6e5e00 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:22.281 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.266+0000 7f8cab7fe640 0 client.15408 ms_handle_reset on v2:172.21.15.162:6836/1371886876 2024-03-31T23:11:22.281 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.266+0000 7f8cd538b740 1 --2- 172.21.15.46:0/2208960206 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a20d6db610 0x55a20d6e58c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:22.281 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.266+0000 7f8cd538b740 1 -- 172.21.15.46:0/2208960206 >> 172.21.15.46:0/2208960206 conn(0x55a20d6d6a00 msgr2=0x55a20d6d5b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:11:22.281 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.266+0000 7f8cd538b740 1 -- 172.21.15.46:0/2208960206 shutdown_connections 2024-03-31T23:11:22.281 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.266+0000 7f8cd538b740 1 -- 172.21.15.46:0/2208960206 wait complete. 2024-03-31T23:11:22.281 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.266+0000 7f8cd538b740 20 client.15408 trim_cache size 0 max 16384 2024-03-31T23:11:22.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.284+0000 7f8cd538b740 1 Processor -- start 2024-03-31T23:11:22.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.284+0000 7f8cd538b740 1 -- start start 2024-03-31T23:11:22.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.284+0000 7f8cd538b740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a20d6db610 0x55a20d6d1760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:22.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.284+0000 7f8cd538b740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a20d6dc7e0 0x55a20d6ece00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:22.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.284+0000 7f8cd538b740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a20d6dd1a0 0x55a20d6e46a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:22.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.284+0000 7f8cd538b740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55a20d6f1d90 con 0x55a20d6dd1a0 2024-03-31T23:11:22.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.284+0000 7f8cd538b740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55a20d6f2460 con 0x55a20d6db610 2024-03-31T23:11:22.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.284+0000 7f8cd538b740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55a20d6f2b30 con 0x55a20d6dc7e0 2024-03-31T23:11:22.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.284+0000 7f8cd0c73640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a20d6dd1a0 0x55a20d6e46a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:22.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.284+0000 7f8cd0c73640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a20d6dd1a0 0x55a20d6e46a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:58178/0 (socket says 172.21.15.46:58178) 2024-03-31T23:11:22.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.284+0000 7f8cd0c73640 1 -- 172.21.15.46:0/3943976422 learned_addr learned my addr 172.21.15.46:0/3943976422 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:11:22.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.284+0000 7f8ccbfff640 1 --2- 172.21.15.46:0/3943976422 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a20d6db610 0x55a20d6d1760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:22.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.285+0000 7f8ccb7fe640 1 --2- 172.21.15.46:0/3943976422 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a20d6dc7e0 0x55a20d6ece00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:22.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.285+0000 7f8cd0c73640 1 -- 172.21.15.46:0/3943976422 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a20d6dc7e0 msgr2=0x55a20d6ece00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:11:22.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.285+0000 7f8cd0c73640 1 --2- 172.21.15.46:0/3943976422 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a20d6dc7e0 0x55a20d6ece00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:22.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.285+0000 7f8cd0c73640 1 -- 172.21.15.46:0/3943976422 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a20d6db610 msgr2=0x55a20d6d1760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:11:22.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.285+0000 7f8cd0c73640 1 --2- 172.21.15.46:0/3943976422 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a20d6db610 0x55a20d6d1760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:22.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.285+0000 7f8cd0c73640 1 -- 172.21.15.46:0/3943976422 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a20d6e0a70 con 0x55a20d6dd1a0 2024-03-31T23:11:22.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.285+0000 7f8ccbfff640 1 --2- 172.21.15.46:0/3943976422 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a20d6db610 0x55a20d6d1760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:11:22.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.285+0000 7f8cd0c73640 1 --2- 172.21.15.46:0/3943976422 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a20d6dd1a0 0x55a20d6e46a0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f8cc4011e20 tx=0x7f8cc4011d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=206a14e2a75a9585 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:11:22.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.285+0000 7f8cb1ffb640 1 -- 172.21.15.46:0/3943976422 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8cc4034070 con 0x55a20d6dd1a0 2024-03-31T23:11:22.298 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.285+0000 7f8cb1ffb640 1 -- 172.21.15.46:0/3943976422 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8cc401dcb0 con 0x55a20d6dd1a0 2024-03-31T23:11:22.298 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.285+0000 7f8cb1ffb640 1 -- 172.21.15.46:0/3943976422 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8cc402a040 con 0x55a20d6dd1a0 2024-03-31T23:11:22.298 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.286+0000 7f8cd538b740 1 -- 172.21.15.46:0/3943976422 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a20d6dd1a0 msgr2=0x55a20d6e46a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:22.298 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.286+0000 7f8cd538b740 1 --2- 172.21.15.46:0/3943976422 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a20d6dd1a0 0x55a20d6e46a0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f8cc4011e20 tx=0x7f8cc4011d10 comp rx=0 tx=0).stop 2024-03-31T23:11:22.298 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.286+0000 7f8cd538b740 1 -- 172.21.15.46:0/3943976422 shutdown_connections 2024-03-31T23:11:22.298 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.286+0000 7f8cd538b740 1 --2- 172.21.15.46:0/3943976422 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a20d6dd1a0 0x55a20d6e46a0 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:22.298 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.286+0000 7f8cd538b740 1 --2- 172.21.15.46:0/3943976422 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a20d6dc7e0 0x55a20d6ece00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:22.298 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.286+0000 7f8cd538b740 1 --2- 172.21.15.46:0/3943976422 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a20d6db610 0x55a20d6d1760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:22.298 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.286+0000 7f8cd538b740 1 -- 172.21.15.46:0/3943976422 >> 172.21.15.46:0/3943976422 conn(0x55a20d6d6a00 msgr2=0x55a20d6d6de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:11:22.298 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.286+0000 7f8cd538b740 1 -- 172.21.15.46:0/3943976422 shutdown_connections 2024-03-31T23:11:22.298 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.286+0000 7f8cd538b740 1 -- 172.21.15.46:0/3943976422 wait complete. 2024-03-31T23:11:22.298 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.286+0000 7f8cd538b740 1 -- start start 2024-03-31T23:11:22.298 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.286+0000 7f8cd538b740 1 Processor -- start 2024-03-31T23:11:22.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.286+0000 7f8cd538b740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a20d6db610 0x55a20d6e9030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:22.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.286+0000 7f8cd538b740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a20d6dc7e0 0x55a20d6e9570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:22.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.286+0000 7f8cd538b740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a20d6dd1a0 0x55a20d6ebf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:22.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.286+0000 7f8cd538b740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55a20d6f38d0 con 0x55a20d6dd1a0 2024-03-31T23:11:22.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.286+0000 7f8cd538b740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55a20d6f3200 con 0x55a20d6dc7e0 2024-03-31T23:11:22.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.286+0000 7f8cd538b740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55a20d6f3fa0 con 0x55a20d6db610 2024-03-31T23:11:22.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.286+0000 7f8cd538b740 10 client.0 _ll_register_callbacks cb 0x7f8cc0009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T23:11:22.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.286+0000 7f8cd0c73640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a20d6dd1a0 0x55a20d6ebf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:22.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.287+0000 7f8ccb7fe640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a20d6dc7e0 0x55a20d6e9570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:22.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.287+0000 7f8cd0c73640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a20d6dd1a0 0x55a20d6ebf30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:58188/0 (socket says 172.21.15.46:58188) 2024-03-31T23:11:22.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.287+0000 7f8cd0c73640 1 -- 172.21.15.46:0/112749435 learned_addr learned my addr 172.21.15.46:0/112749435 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:11:22.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.287+0000 7f8ccbfff640 1 --2- 172.21.15.46:0/112749435 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a20d6db610 0x55a20d6e9030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:22.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.287+0000 7f8cd0c73640 1 -- 172.21.15.46:0/112749435 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a20d6db610 msgr2=0x55a20d6e9030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:11:22.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.287+0000 7f8cd0c73640 1 --2- 172.21.15.46:0/112749435 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a20d6db610 0x55a20d6e9030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:22.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.287+0000 7f8cd0c73640 1 -- 172.21.15.46:0/112749435 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a20d6dc7e0 msgr2=0x55a20d6e9570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:11:22.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.287+0000 7f8cd0c73640 1 --2- 172.21.15.46:0/112749435 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a20d6dc7e0 0x55a20d6e9570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:22.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.287+0000 7f8cd0c73640 1 -- 172.21.15.46:0/112749435 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a20d6ec520 con 0x55a20d6dd1a0 2024-03-31T23:11:22.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.287+0000 7f8ccb7fe640 1 --2- 172.21.15.46:0/112749435 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a20d6dc7e0 0x55a20d6e9570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:11:22.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.287+0000 7f8cd0c73640 1 --2- 172.21.15.46:0/112749435 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a20d6dd1a0 0x55a20d6ebf30 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f8cc4009b70 tx=0x7f8cc4009500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d67f7f8c3f2330a0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:11:22.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.287+0000 7f8cab7fe640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-31T23:11:22.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.287+0000 7f8cab7fe640 1 -- 172.21.15.46:0/112749435 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8cc402a070 con 0x55a20d6dd1a0 2024-03-31T23:11:22.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.287+0000 7f8cd538b740 1 -- 172.21.15.46:0/112749435 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8cc401dcb0 con 0x55a20d6dd1a0 2024-03-31T23:11:22.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.287+0000 7f8cab7fe640 1 -- 172.21.15.46:0/112749435 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9208+0+0 (secure 0 0 0) 0x7f8cc40044f0 con 0x55a20d6dd1a0 2024-03-31T23:11:22.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.287+0000 7f8cab7fe640 1 -- 172.21.15.46:0/112749435 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f8cc4034050 con 0x55a20d6dd1a0 2024-03-31T23:11:22.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.287+0000 7f8cd538b740 10 client.15414 fetch_fsmap learned FSMap version 24 2024-03-31T23:11:22.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.287+0000 7f8cd538b740 1 -- 172.21.15.46:0/112749435 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55a20d758800 con 0x55a20d6dd1a0 2024-03-31T23:11:22.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.288+0000 7f8cab7fe640 1 -- 172.21.15.46:0/112749435 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8cc401dcb0 con 0x55a20d6dd1a0 2024-03-31T23:11:22.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.288+0000 7f8cab7fe640 1 -- 172.21.15.46:0/112749435 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f8cc4003ca0 con 0x55a20d6dd1a0 2024-03-31T23:11:22.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.288+0000 7f8cd538b740 10 client.15414 fetch_fsmap finished waiting for FSMap version 24 2024-03-31T23:11:22.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.288+0000 7f8cd538b740 10 client.15414 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-31T23:11:22.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.288+0000 7f8cd538b740 20 client.15414 populate_metadata read hostname 'smithi046' 2024-03-31T23:11:22.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.288+0000 7f8cd538b740 1 --2- 172.21.15.46:0/112749435 >> [v2:172.21.15.162:6836/1371886876,v1:172.21.15.162:6837/1371886876] conn(0x55a20d75cda0 0x55a20d75bab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:22.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.288+0000 7f8cd538b740 4 client.15414 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-31T23:11:22.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.288+0000 7f8cd538b740 1 -- 172.21.15.46:0/112749435 --> [v2:172.21.15.162:6836/1371886876,v1:172.21.15.162:6837/1371886876] -- command(tid 0: {"prefix": "status"}) -- 0x55a20d647380 con 0x55a20d75cda0 2024-03-31T23:11:22.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.288+0000 7f8ccbfff640 1 --2- 172.21.15.46:0/112749435 >> [v2:172.21.15.162:6836/1371886876,v1:172.21.15.162:6837/1371886876] conn(0x55a20d75cda0 0x55a20d75bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:22.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.288+0000 7f8ccbfff640 1 --2- 172.21.15.46:0/112749435 >> [v2:172.21.15.162:6836/1371886876,v1:172.21.15.162:6837/1371886876] conn(0x55a20d75cda0 0x55a20d75bab0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=4b61c264d0bcf278 server_cookie=2c524f106e40fd08 in_seq=0 out_seq=0 2024-03-31T23:11:22.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.288+0000 7f8cab7fe640 10 client.15414 ms_handle_connect on v2:172.21.15.162:6836/1371886876 2024-03-31T23:11:22.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.288+0000 7f8cab7fe640 1 -- 172.21.15.46:0/112749435 <== mds.2 v2:172.21.15.162:6836/1371886876 1 ==== command_reply(tid 0: 0 ) ==== 8+0+326 (crc 0 0 0) 0x55a20d647380 con 0x55a20d75cda0 2024-03-31T23:11:22.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.288+0000 7f8cab7fe640 10 client.15414 handle_command_reply: tid=0 2024-03-31T23:11:22.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.289+0000 7f8cd538b740 1 client.15414 shutdown 2024-03-31T23:11:22.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.289+0000 7f8cd538b740 1 -- 172.21.15.46:0/112749435 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a20d6dd1a0 msgr2=0x55a20d6ebf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:22.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.289+0000 7f8cd538b740 1 --2- 172.21.15.46:0/112749435 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a20d6dd1a0 0x55a20d6ebf30 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f8cc4009b70 tx=0x7f8cc4009500 comp rx=0 tx=0).stop 2024-03-31T23:11:22.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.289+0000 7f8cd538b740 1 -- 172.21.15.46:0/112749435 shutdown_connections 2024-03-31T23:11:22.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.289+0000 7f8cd538b740 1 --2- 172.21.15.46:0/112749435 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a20d6dd1a0 0x55a20d6ebf30 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:22.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.289+0000 7f8cd538b740 1 --2- 172.21.15.46:0/112749435 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a20d6dc7e0 0x55a20d6e9570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:22.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.289+0000 7f8cd538b740 1 --2- 172.21.15.46:0/112749435 >> [v2:172.21.15.162:6836/1371886876,v1:172.21.15.162:6837/1371886876] conn(0x55a20d75cda0 0x55a20d75bab0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:22.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.289+0000 7f8cd538b740 1 --2- 172.21.15.46:0/112749435 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a20d6db610 0x55a20d6e9030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:22.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.289+0000 7f8cd538b740 1 -- 172.21.15.46:0/112749435 >> 172.21.15.46:0/112749435 conn(0x55a20d6d6a00 msgr2=0x55a20d6d1760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:11:22.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.289+0000 7f8cab7fe640 0 client.15414 ms_handle_reset on v2:172.21.15.162:6836/1371886876 2024-03-31T23:11:22.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.289+0000 7f8cd538b740 1 -- 172.21.15.46:0/112749435 shutdown_connections 2024-03-31T23:11:22.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.289+0000 7f8cd538b740 1 -- 172.21.15.46:0/112749435 wait complete. 2024-03-31T23:11:22.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.289+0000 7f8cd538b740 20 client.15414 trim_cache size 0 max 16384 2024-03-31T23:11:22.302 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-03-31T23:11:22.303 INFO:teuthology.orchestra.run.smithi046.stdout: "cluster_fsid": "6a9d8f08-efb2-11ee-b647-cb9ed24678a4", 2024-03-31T23:11:22.303 INFO:teuthology.orchestra.run.smithi046.stdout: "whoami": 2, 2024-03-31T23:11:22.303 INFO:teuthology.orchestra.run.smithi046.stdout: "id": 24451, 2024-03-31T23:11:22.303 INFO:teuthology.orchestra.run.smithi046.stdout: "want_state": "up:active", 2024-03-31T23:11:22.303 INFO:teuthology.orchestra.run.smithi046.stdout: "state": "up:active", 2024-03-31T23:11:22.303 INFO:teuthology.orchestra.run.smithi046.stdout: "fs_name": "cephfs", 2024-03-31T23:11:22.303 INFO:teuthology.orchestra.run.smithi046.stdout: "rank_uptime": 111.90049576, 2024-03-31T23:11:22.303 INFO:teuthology.orchestra.run.smithi046.stdout: "mdsmap_epoch": 24, 2024-03-31T23:11:22.303 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap_epoch": 95, 2024-03-31T23:11:22.303 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap_epoch_barrier": 91, 2024-03-31T23:11:22.303 INFO:teuthology.orchestra.run.smithi046.stdout: "uptime": 149.57364399400001 2024-03-31T23:11:22.303 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-03-31T23:11:22.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.289+0000 7f8cd26d6640 1 -- 172.21.15.46:0/1816407577 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f8cb40a24a0 msgr2=0x7f8cb40a4960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:22.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.289+0000 7f8cd26d6640 1 --2- 172.21.15.46:0/1816407577 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f8cb40a24a0 0x7f8cb40a4960 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f8cc400fc90 tx=0x7f8cc401e000 comp rx=0 tx=0).stop 2024-03-31T23:11:22.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.289+0000 7f8cd26d6640 1 -- 172.21.15.46:0/1816407577 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8ccc0ff7c0 msgr2=0x7f8ccc197970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:22.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.289+0000 7f8cd26d6640 1 --2- 172.21.15.46:0/1816407577 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8ccc0ff7c0 0x7f8ccc197970 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f8cbc00e9c0 tx=0x7f8cbc00ee80 comp rx=0 tx=0).stop 2024-03-31T23:11:22.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.289+0000 7f8cd26d6640 1 -- 172.21.15.46:0/1816407577 shutdown_connections 2024-03-31T23:11:22.306 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.290+0000 7f8cd26d6640 1 --2- 172.21.15.46:0/1816407577 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f8cb40a24a0 0x7f8cb40a4960 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:22.306 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.290+0000 7f8cd26d6640 1 --2- 172.21.15.46:0/1816407577 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8ccc102d50 0x7f8ccc19ef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:22.306 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.290+0000 7f8cd26d6640 1 --2- 172.21.15.46:0/1816407577 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8ccc100180 0x7f8ccc197eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:22.306 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.290+0000 7f8cd26d6640 1 --2- 172.21.15.46:0/1816407577 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8ccc0ff7c0 0x7f8ccc197970 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:22.306 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.290+0000 7f8cd26d6640 1 -- 172.21.15.46:0/1816407577 >> 172.21.15.46:0/1816407577 conn(0x7f8ccc0fb340 msgr2=0x7f8ccc105e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:11:22.306 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.290+0000 7f8cd26d6640 1 -- 172.21.15.46:0/1816407577 shutdown_connections 2024-03-31T23:11:22.306 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.290+0000 7f8cd26d6640 1 -- 172.21.15.46:0/1816407577 wait complete. 2024-03-31T23:11:22.306 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.h status 2024-03-31T23:11:22.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.410+0000 7f5e3b556640 1 -- 172.21.15.46:0/3636587264 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5e34101840 msgr2=0x7f5e34101cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:22.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.410+0000 7f5e3b556640 1 --2- 172.21.15.46:0/3636587264 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5e34101840 0x7f5e34101cc0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f5e24009900 tx=0x7f5e2402bc30 comp rx=0 tx=0).stop 2024-03-31T23:11:22.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.410+0000 7f5e3b556640 1 -- 172.21.15.46:0/3636587264 shutdown_connections 2024-03-31T23:11:22.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.410+0000 7f5e3b556640 1 --2- 172.21.15.46:0/3636587264 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5e34102200 0x7f5e34108a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:22.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.410+0000 7f5e3b556640 1 --2- 172.21.15.46:0/3636587264 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5e34101840 0x7f5e34101cc0 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:22.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.410+0000 7f5e3b556640 1 --2- 172.21.15.46:0/3636587264 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5e34100650 0x7f5e34100a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:22.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.410+0000 7f5e3b556640 1 -- 172.21.15.46:0/3636587264 >> 172.21.15.46:0/3636587264 conn(0x7f5e340fbbe0 msgr2=0x7f5e340fe000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:11:22.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.410+0000 7f5e3b556640 1 -- 172.21.15.46:0/3636587264 shutdown_connections 2024-03-31T23:11:22.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.410+0000 7f5e3b556640 1 -- 172.21.15.46:0/3636587264 wait complete. 2024-03-31T23:11:22.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.411+0000 7f5e3b556640 1 Processor -- start 2024-03-31T23:11:22.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.411+0000 7f5e3b556640 1 -- start start 2024-03-31T23:11:22.414 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.411+0000 7f5e3b556640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5e34100650 0x7f5e341979d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:22.414 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.411+0000 7f5e3b556640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5e34101840 0x7f5e34197f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:22.414 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.411+0000 7f5e3b556640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5e34102200 0x7f5e3419ef80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:22.414 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.411+0000 7f5e3b556640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f5e3410c170 con 0x7f5e34102200 2024-03-31T23:11:22.414 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.411+0000 7f5e3b556640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f5e3410bff0 con 0x7f5e34101840 2024-03-31T23:11:22.414 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.411+0000 7f5e3b556640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f5e3410c2f0 con 0x7f5e34100650 2024-03-31T23:11:22.414 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.411+0000 7f5e39af3640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5e34102200 0x7f5e3419ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:22.414 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.411+0000 7f5e39af3640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5e34102200 0x7f5e3419ef80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:58198/0 (socket says 172.21.15.46:58198) 2024-03-31T23:11:22.414 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.411+0000 7f5e392f2640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5e34100650 0x7f5e341979d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:22.414 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.411+0000 7f5e38af1640 1 --2- 172.21.15.46:0/2133344226 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5e34101840 0x7f5e34197f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:22.414 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.411+0000 7f5e39af3640 1 -- 172.21.15.46:0/2133344226 learned_addr learned my addr 172.21.15.46:0/2133344226 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:11:22.414 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.411+0000 7f5e39af3640 1 -- 172.21.15.46:0/2133344226 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5e34100650 msgr2=0x7f5e341979d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:22.414 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.411+0000 7f5e39af3640 1 --2- 172.21.15.46:0/2133344226 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5e34100650 0x7f5e341979d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:22.415 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.411+0000 7f5e39af3640 1 -- 172.21.15.46:0/2133344226 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5e34101840 msgr2=0x7f5e34197f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:22.415 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.411+0000 7f5e39af3640 1 --2- 172.21.15.46:0/2133344226 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5e34101840 0x7f5e34197f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:22.415 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.411+0000 7f5e39af3640 1 -- 172.21.15.46:0/2133344226 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5e3419f5f0 con 0x7f5e34102200 2024-03-31T23:11:22.415 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.411+0000 7f5e392f2640 1 --2- 172.21.15.46:0/2133344226 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5e34100650 0x7f5e341979d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:11:22.415 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.411+0000 7f5e38af1640 1 --2- 172.21.15.46:0/2133344226 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5e34101840 0x7f5e34197f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:11:22.415 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.412+0000 7f5e39af3640 1 --2- 172.21.15.46:0/2133344226 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5e34102200 0x7f5e3419ef80 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f5e2c00eb10 tx=0x7f5e2c00c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:11:22.415 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.412+0000 7f5e2a7fc640 1 -- 172.21.15.46:0/2133344226 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e2c004b20 con 0x7f5e34102200 2024-03-31T23:11:22.415 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.412+0000 7f5e2a7fc640 1 -- 172.21.15.46:0/2133344226 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5e2c004cc0 con 0x7f5e34102200 2024-03-31T23:11:22.415 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.412+0000 7f5e3b556640 1 -- 172.21.15.46:0/2133344226 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5e3419f8e0 con 0x7f5e34102200 2024-03-31T23:11:22.415 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.412+0000 7f5e2a7fc640 1 -- 172.21.15.46:0/2133344226 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e2c010690 con 0x7f5e34102200 2024-03-31T23:11:22.415 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.412+0000 7f5e3b556640 1 -- 172.21.15.46:0/2133344226 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5e3419fe00 con 0x7f5e34102200 2024-03-31T23:11:22.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.415+0000 7f5e2a7fc640 1 -- 172.21.15.46:0/2133344226 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 18) ==== 137320+0+0 (secure 0 0 0) 0x7f5e2c005020 con 0x7f5e34102200 2024-03-31T23:11:22.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.415+0000 7f5e3e212740 1 Processor -- start 2024-03-31T23:11:22.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.415+0000 7f5e3e212740 1 -- start start 2024-03-31T23:11:22.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.415+0000 7f5e3e212740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ada46e6a20 0x55ada46e6e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:22.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.415+0000 7f5e3e212740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ada46e7bf0 0x55ada46e8070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:22.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.415+0000 7f5e2a7fc640 1 --2- 172.21.15.46:0/2133344226 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f5e200a2520 0x7f5e200a49e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:22.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.415+0000 7f5e3e212740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55ada46e85b0 0x55ada46eee40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:22.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.415+0000 7f5e3e212740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55ada4646af0 con 0x55ada46e7bf0 2024-03-31T23:11:22.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.415+0000 7f5e3e212740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55ada45ef9d0 con 0x55ada46e85b0 2024-03-31T23:11:22.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.415+0000 7f5e3e212740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55ada4642b90 con 0x55ada46e6a20 2024-03-31T23:11:22.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.415+0000 7f5e392f2640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ada46e7bf0 0x55ada46e8070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:22.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.415+0000 7f5e38af1640 1 --2- 172.21.15.46:0/2133344226 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f5e200a2520 0x7f5e200a49e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:22.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.415+0000 7f5e392f2640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ada46e7bf0 0x55ada46e8070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:58212/0 (socket says 172.21.15.46:58212) 2024-03-31T23:11:22.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.415+0000 7f5e392f2640 1 -- 172.21.15.46:0/3960493394 learned_addr learned my addr 172.21.15.46:0/3960493394 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:11:22.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.416+0000 7f5e39af3640 1 --2- 172.21.15.46:0/3960493394 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55ada46e85b0 0x55ada46eee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:22.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.416+0000 7f5e38af1640 1 --2- 172.21.15.46:0/3960493394 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ada46e6a20 0x55ada46e6e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:22.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.416+0000 7f5e2a7fc640 1 -- 172.21.15.46:0/2133344226 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9208+0+0 (secure 0 0 0) 0x7f5e2c0cf000 con 0x7f5e34102200 2024-03-31T23:11:22.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.416+0000 7f5e392f2640 1 -- 172.21.15.46:0/3960493394 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ada46e6a20 msgr2=0x55ada46e6e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:11:22.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.416+0000 7f5e392f2640 1 --2- 172.21.15.46:0/3960493394 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ada46e6a20 0x55ada46e6e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:22.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.416+0000 7f5e392f2640 1 -- 172.21.15.46:0/3960493394 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55ada46e85b0 msgr2=0x55ada46eee40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:11:22.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.416+0000 7f5e392f2640 1 --2- 172.21.15.46:0/3960493394 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55ada46e85b0 0x55ada46eee40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:22.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.416+0000 7f5e392f2640 1 -- 172.21.15.46:0/3960493394 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ada435b2d0 con 0x55ada46e7bf0 2024-03-31T23:11:22.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.416+0000 7f5e39af3640 1 --2- 172.21.15.46:0/3960493394 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55ada46e85b0 0x55ada46eee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:11:22.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.416+0000 7f5e392f2640 1 --2- 172.21.15.46:0/3960493394 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ada46e7bf0 0x55ada46e8070 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f5e300114c0 tx=0x7f5e30011980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e82ad5270791b5c server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:11:22.424 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.416+0000 7f5e38af1640 1 --2- 172.21.15.46:0/2133344226 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f5e200a2520 0x7f5e200a49e0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f5e24009230 tx=0x7f5e24005e80 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:11:22.424 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.416+0000 7f5e1f7fe640 1 -- 172.21.15.46:0/3960493394 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e30004020 con 0x55ada46e7bf0 2024-03-31T23:11:22.424 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.416+0000 7f5e1f7fe640 1 -- 172.21.15.46:0/3960493394 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5e300054d0 con 0x55ada46e7bf0 2024-03-31T23:11:22.424 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.416+0000 7f5e1f7fe640 1 -- 172.21.15.46:0/3960493394 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e30004ce0 con 0x55ada46e7bf0 2024-03-31T23:11:22.424 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.417+0000 7f5e3e212740 1 -- 172.21.15.46:0/3960493394 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ada46e7bf0 msgr2=0x55ada46e8070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:22.424 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.417+0000 7f5e3e212740 1 --2- 172.21.15.46:0/3960493394 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ada46e7bf0 0x55ada46e8070 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f5e300114c0 tx=0x7f5e30011980 comp rx=0 tx=0).stop 2024-03-31T23:11:22.424 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.417+0000 7f5e3e212740 1 -- 172.21.15.46:0/3960493394 shutdown_connections 2024-03-31T23:11:22.424 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.417+0000 7f5e3e212740 1 --2- 172.21.15.46:0/3960493394 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55ada46e85b0 0x55ada46eee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:22.424 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.417+0000 7f5e3e212740 1 --2- 172.21.15.46:0/3960493394 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ada46e7bf0 0x55ada46e8070 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:22.424 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.417+0000 7f5e3e212740 1 --2- 172.21.15.46:0/3960493394 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ada46e6a20 0x55ada46e6e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:22.424 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.417+0000 7f5e3e212740 1 -- 172.21.15.46:0/3960493394 >> 172.21.15.46:0/3960493394 conn(0x55ada46dda00 msgr2=0x55ada46dde00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:11:22.424 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.417+0000 7f5e3e212740 1 -- 172.21.15.46:0/3960493394 shutdown_connections 2024-03-31T23:11:22.424 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.417+0000 7f5e3e212740 1 -- 172.21.15.46:0/3960493394 wait complete. 2024-03-31T23:11:22.425 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.417+0000 7f5e3e212740 1 -- start start 2024-03-31T23:11:22.425 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.417+0000 7f5e3e212740 1 Processor -- start 2024-03-31T23:11:22.425 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.418+0000 7f5e3e212740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ada46e6a20 0x55ada46f0cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:22.425 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.418+0000 7f5e3e212740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ada46e7bf0 0x55ada46f1200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:22.425 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.418+0000 7f5e3e212740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55ada46e85b0 0x55ada46f8240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:22.425 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.418+0000 7f5e3e212740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55ada46ef5c0 con 0x55ada46e6a20 2024-03-31T23:11:22.425 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.418+0000 7f5e3e212740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55ada4602f70 con 0x55ada46e85b0 2024-03-31T23:11:22.425 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.418+0000 7f5e3e212740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55ada46ef740 con 0x55ada46e7bf0 2024-03-31T23:11:22.425 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.418+0000 7f5e3e212740 10 client.0 _ll_register_callbacks cb 0x55ada435aac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T23:11:22.425 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.418+0000 7f5e392f2640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ada46e6a20 0x55ada46f0cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:22.425 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.418+0000 7f5e392f2640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ada46e6a20 0x55ada46f0cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:58226/0 (socket says 172.21.15.46:58226) 2024-03-31T23:11:22.425 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.418+0000 7f5e392f2640 1 -- 172.21.15.46:0/2056015474 learned_addr learned my addr 172.21.15.46:0/2056015474 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:11:22.425 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.418+0000 7f5e39af3640 1 --2- 172.21.15.46:0/2056015474 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55ada46e85b0 0x55ada46f8240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:22.425 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.418+0000 7f5e38af1640 1 --2- 172.21.15.46:0/2056015474 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ada46e7bf0 0x55ada46f1200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:22.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.418+0000 7f5e392f2640 1 -- 172.21.15.46:0/2056015474 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ada46e7bf0 msgr2=0x55ada46f1200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:11:22.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.418+0000 7f5e392f2640 1 --2- 172.21.15.46:0/2056015474 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ada46e7bf0 0x55ada46f1200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:22.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.418+0000 7f5e392f2640 1 -- 172.21.15.46:0/2056015474 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55ada46e85b0 msgr2=0x55ada46f8240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:11:22.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.418+0000 7f5e392f2640 1 --2- 172.21.15.46:0/2056015474 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55ada46e85b0 0x55ada46f8240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:22.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.418+0000 7f5e392f2640 1 -- 172.21.15.46:0/2056015474 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ada44256b0 con 0x55ada46e6a20 2024-03-31T23:11:22.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.418+0000 7f5e392f2640 1 --2- 172.21.15.46:0/2056015474 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ada46e6a20 0x55ada46f0cc0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f5e30011e80 tx=0x7f5e3000fca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8eab2371509db132 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:11:22.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.418+0000 7f5e1cff9640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-31T23:11:22.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.418+0000 7f5e1cff9640 1 -- 172.21.15.46:0/2056015474 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e30028020 con 0x55ada46e6a20 2024-03-31T23:11:22.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.418+0000 7f5e1cff9640 1 -- 172.21.15.46:0/2056015474 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9208+0+0 (secure 0 0 0) 0x7f5e300049e0 con 0x55ada46e6a20 2024-03-31T23:11:22.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.418+0000 7f5e3e212740 1 -- 172.21.15.46:0/2056015474 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5e300054d0 con 0x55ada46e6a20 2024-03-31T23:11:22.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.419+0000 7f5e1cff9640 1 -- 172.21.15.46:0/2056015474 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f5e30033050 con 0x55ada46e6a20 2024-03-31T23:11:22.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.419+0000 7f5e3e212740 10 client.15426 fetch_fsmap learned FSMap version 24 2024-03-31T23:11:22.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.419+0000 7f5e3e212740 1 -- 172.21.15.46:0/2056015474 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55ada46fb220 con 0x55ada46e6a20 2024-03-31T23:11:22.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.419+0000 7f5e1cff9640 1 -- 172.21.15.46:0/2056015474 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5e300054d0 con 0x55ada46e6a20 2024-03-31T23:11:22.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.419+0000 7f5e1cff9640 1 -- 172.21.15.46:0/2056015474 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f5e30023470 con 0x55ada46e6a20 2024-03-31T23:11:22.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.419+0000 7f5e3e212740 10 client.15426 fetch_fsmap finished waiting for FSMap version 24 2024-03-31T23:11:22.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.419+0000 7f5e3e212740 10 client.15426 resolve_mds: resolved name 'h' to daemon mds.h 2024-03-31T23:11:22.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.419+0000 7f5e3e212740 20 client.15426 populate_metadata read hostname 'smithi046' 2024-03-31T23:11:22.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.419+0000 7f5e3e212740 1 --2- 172.21.15.46:0/2056015474 >> [v2:172.21.15.142:6836/2559258399,v1:172.21.15.142:6837/2559258399] conn(0x55ada46fd730 0x55ada46feba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:22.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.419+0000 7f5e3e212740 4 client.15426 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-31T23:11:22.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.419+0000 7f5e3e212740 1 -- 172.21.15.46:0/2056015474 --> [v2:172.21.15.142:6836/2559258399,v1:172.21.15.142:6837/2559258399] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ada4654030 con 0x55ada46fd730 2024-03-31T23:11:22.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.419+0000 7f5e38af1640 1 --2- 172.21.15.46:0/2056015474 >> [v2:172.21.15.142:6836/2559258399,v1:172.21.15.142:6837/2559258399] conn(0x55ada46fd730 0x55ada46feba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:22.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.420+0000 7f5e1cff9640 10 client.15426 ms_handle_connect on v2:172.21.15.142:6836/2559258399 2024-03-31T23:11:22.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.420+0000 7f5e38af1640 1 --2- 172.21.15.46:0/2056015474 >> [v2:172.21.15.142:6836/2559258399,v1:172.21.15.142:6837/2559258399] conn(0x55ada46fd730 0x55ada46feba0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=d19d4260106fb740 server_cookie=22dc12c9b13b31cd in_seq=0 out_seq=0 2024-03-31T23:11:22.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.420+0000 7f5e1cff9640 1 -- 172.21.15.46:0/2056015474 <== mds.1 v2:172.21.15.142:6836/2559258399 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55ada4654030 con 0x55ada46fd730 2024-03-31T23:11:22.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.421+0000 7f5e1cff9640 10 client.15426 handle_command_reply: tid=0 2024-03-31T23:11:22.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.421+0000 7f5e3e212740 1 client.15426 shutdown 2024-03-31T23:11:22.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.421+0000 7f5e3e212740 1 -- 172.21.15.46:0/2056015474 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ada46e6a20 msgr2=0x55ada46f0cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:22.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.421+0000 7f5e3e212740 1 --2- 172.21.15.46:0/2056015474 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ada46e6a20 0x55ada46f0cc0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f5e30011e80 tx=0x7f5e3000fca0 comp rx=0 tx=0).stop 2024-03-31T23:11:22.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.421+0000 7f5e3e212740 1 -- 172.21.15.46:0/2056015474 shutdown_connections 2024-03-31T23:11:22.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.421+0000 7f5e3e212740 1 --2- 172.21.15.46:0/2056015474 >> [v2:172.21.15.142:6836/2559258399,v1:172.21.15.142:6837/2559258399] conn(0x55ada46fd730 0x55ada46feba0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:22.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.421+0000 7f5e3e212740 1 --2- 172.21.15.46:0/2056015474 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55ada46e85b0 0x55ada46f8240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:22.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.421+0000 7f5e3e212740 1 --2- 172.21.15.46:0/2056015474 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ada46e7bf0 0x55ada46f1200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:22.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.421+0000 7f5e3e212740 1 --2- 172.21.15.46:0/2056015474 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ada46e6a20 0x55ada46f0cc0 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:22.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.421+0000 7f5e1cff9640 0 client.15426 ms_handle_reset on v2:172.21.15.142:6836/2559258399 2024-03-31T23:11:22.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.421+0000 7f5e3e212740 1 -- 172.21.15.46:0/2056015474 >> 172.21.15.46:0/2056015474 conn(0x55ada46dda00 msgr2=0x55ada46dcb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:11:22.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.421+0000 7f5e3e212740 1 -- 172.21.15.46:0/2056015474 shutdown_connections 2024-03-31T23:11:22.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.421+0000 7f5e3e212740 1 -- 172.21.15.46:0/2056015474 wait complete. 2024-03-31T23:11:22.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.421+0000 7f5e3e212740 20 client.15426 trim_cache size 0 max 16384 2024-03-31T23:11:22.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.439+0000 7f5e3e212740 1 Processor -- start 2024-03-31T23:11:22.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.439+0000 7f5e3e212740 1 -- start start 2024-03-31T23:11:22.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.439+0000 7f5e3e212740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ada46e6a20 0x55ada46ddde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:22.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.439+0000 7f5e3e212740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ada46e7bf0 0x55ada46f8200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:22.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.439+0000 7f5e3e212740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55ada46e85b0 0x55ada46dc2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:22.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.439+0000 7f5e3e212740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55ada46fcee0 con 0x55ada46e7bf0 2024-03-31T23:11:22.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.439+0000 7f5e3e212740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55ada46fd5b0 con 0x55ada46e85b0 2024-03-31T23:11:22.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.439+0000 7f5e3e212740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55ada46fdc80 con 0x55ada46e6a20 2024-03-31T23:11:22.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.439+0000 7f5e38af1640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ada46e7bf0 0x55ada46f8200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:22.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.439+0000 7f5e38af1640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ada46e7bf0 0x55ada46f8200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:58240/0 (socket says 172.21.15.46:58240) 2024-03-31T23:11:22.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.439+0000 7f5e38af1640 1 -- 172.21.15.46:0/3071505000 learned_addr learned my addr 172.21.15.46:0/3071505000 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:11:22.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.440+0000 7f5e39af3640 1 --2- 172.21.15.46:0/3071505000 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55ada46e85b0 0x55ada46dc2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:22.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.440+0000 7f5e392f2640 1 --2- 172.21.15.46:0/3071505000 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ada46e6a20 0x55ada46ddde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:22.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.440+0000 7f5e38af1640 1 -- 172.21.15.46:0/3071505000 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ada46e6a20 msgr2=0x55ada46ddde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:11:22.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.440+0000 7f5e38af1640 1 --2- 172.21.15.46:0/3071505000 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ada46e6a20 0x55ada46ddde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:22.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.440+0000 7f5e38af1640 1 -- 172.21.15.46:0/3071505000 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55ada46e85b0 msgr2=0x55ada46dc2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:11:22.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.440+0000 7f5e38af1640 1 --2- 172.21.15.46:0/3071505000 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55ada46e85b0 0x55ada46dc2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:22.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.440+0000 7f5e38af1640 1 -- 172.21.15.46:0/3071505000 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ada46de770 con 0x55ada46e7bf0 2024-03-31T23:11:22.447 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.440+0000 7f5e39af3640 1 --2- 172.21.15.46:0/3071505000 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55ada46e85b0 0x55ada46dc2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:11:22.447 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.440+0000 7f5e38af1640 1 --2- 172.21.15.46:0/3071505000 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ada46e7bf0 0x55ada46f8200 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f5e24004650 tx=0x7f5e24047000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=75a98111400d047f server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:11:22.447 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.440+0000 7f5e1effd640 1 -- 172.21.15.46:0/3071505000 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e240552a0 con 0x55ada46e7bf0 2024-03-31T23:11:22.447 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.440+0000 7f5e1effd640 1 -- 172.21.15.46:0/3071505000 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5e240046c0 con 0x55ada46e7bf0 2024-03-31T23:11:22.447 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.440+0000 7f5e1effd640 1 -- 172.21.15.46:0/3071505000 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e24003810 con 0x55ada46e7bf0 2024-03-31T23:11:22.447 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.441+0000 7f5e3e212740 1 -- 172.21.15.46:0/3071505000 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ada46e7bf0 msgr2=0x55ada46f8200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:22.447 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.441+0000 7f5e3e212740 1 --2- 172.21.15.46:0/3071505000 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ada46e7bf0 0x55ada46f8200 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f5e24004650 tx=0x7f5e24047000 comp rx=0 tx=0).stop 2024-03-31T23:11:22.447 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.441+0000 7f5e3e212740 1 -- 172.21.15.46:0/3071505000 shutdown_connections 2024-03-31T23:11:22.447 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.441+0000 7f5e3e212740 1 --2- 172.21.15.46:0/3071505000 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55ada46e85b0 0x55ada46dc2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:22.447 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.441+0000 7f5e3e212740 1 --2- 172.21.15.46:0/3071505000 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ada46e7bf0 0x55ada46f8200 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:22.447 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.441+0000 7f5e3e212740 1 --2- 172.21.15.46:0/3071505000 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ada46e6a20 0x55ada46ddde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:22.447 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.441+0000 7f5e3e212740 1 -- 172.21.15.46:0/3071505000 >> 172.21.15.46:0/3071505000 conn(0x55ada46dda00 msgr2=0x55ada46fc680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:11:22.447 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.441+0000 7f5e3e212740 1 -- 172.21.15.46:0/3071505000 shutdown_connections 2024-03-31T23:11:22.447 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.441+0000 7f5e3e212740 1 -- 172.21.15.46:0/3071505000 wait complete. 2024-03-31T23:11:22.448 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.441+0000 7f5e3e212740 1 -- start start 2024-03-31T23:11:22.448 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.441+0000 7f5e3e212740 1 Processor -- start 2024-03-31T23:11:22.448 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.441+0000 7f5e3e212740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ada46e6a20 0x55ada46f3820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:22.448 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.441+0000 7f5e3e212740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ada46e7bf0 0x55ada46f3d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:22.448 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.441+0000 7f5e3e212740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55ada46e85b0 0x55ada46f6d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:22.448 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.441+0000 7f5e3e212740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55ada46fea20 con 0x55ada46e7bf0 2024-03-31T23:11:22.448 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.441+0000 7f5e3e212740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55ada46fe350 con 0x55ada46e85b0 2024-03-31T23:11:22.448 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.441+0000 7f5e3e212740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55ada46ff0f0 con 0x55ada46e6a20 2024-03-31T23:11:22.448 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.441+0000 7f5e3e212740 10 client.0 _ll_register_callbacks cb 0x7f5e2c0057a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T23:11:22.448 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.441+0000 7f5e38af1640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ada46e7bf0 0x55ada46f3d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:22.448 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.441+0000 7f5e38af1640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ada46e7bf0 0x55ada46f3d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:58252/0 (socket says 172.21.15.46:58252) 2024-03-31T23:11:22.448 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.441+0000 7f5e38af1640 1 -- 172.21.15.46:0/1893736058 learned_addr learned my addr 172.21.15.46:0/1893736058 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:11:22.448 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.441+0000 7f5e392f2640 1 --2- 172.21.15.46:0/1893736058 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ada46e6a20 0x55ada46f3820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:22.448 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.442+0000 7f5e39af3640 1 --2- 172.21.15.46:0/1893736058 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55ada46e85b0 0x55ada46f6d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:22.448 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.442+0000 7f5e38af1640 1 -- 172.21.15.46:0/1893736058 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ada46e6a20 msgr2=0x55ada46f3820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:11:22.449 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.442+0000 7f5e38af1640 1 --2- 172.21.15.46:0/1893736058 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ada46e6a20 0x55ada46f3820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:22.449 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.442+0000 7f5e38af1640 1 -- 172.21.15.46:0/1893736058 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55ada46e85b0 msgr2=0x55ada46f6d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:11:22.449 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.442+0000 7f5e38af1640 1 --2- 172.21.15.46:0/1893736058 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55ada46e85b0 0x55ada46f6d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:22.449 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.442+0000 7f5e38af1640 1 -- 172.21.15.46:0/1893736058 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ada46e7410 con 0x55ada46e7bf0 2024-03-31T23:11:22.449 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.442+0000 7f5e38af1640 1 --2- 172.21.15.46:0/1893736058 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ada46e7bf0 0x55ada46f3d60 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f5e2402bc70 tx=0x7f5e24009600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d49340e44cc91a17 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:11:22.449 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.442+0000 7f5e1cff9640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-31T23:11:22.449 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.442+0000 7f5e1cff9640 1 -- 172.21.15.46:0/1893736058 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e24004a80 con 0x55ada46e7bf0 2024-03-31T23:11:22.449 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.442+0000 7f5e1cff9640 1 -- 172.21.15.46:0/1893736058 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9208+0+0 (secure 0 0 0) 0x7f5e240552a0 con 0x55ada46e7bf0 2024-03-31T23:11:22.449 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.442+0000 7f5e3e212740 1 -- 172.21.15.46:0/1893736058 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5e240046c0 con 0x55ada46e7bf0 2024-03-31T23:11:22.449 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.442+0000 7f5e1cff9640 1 -- 172.21.15.46:0/1893736058 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f5e24004c20 con 0x55ada46e7bf0 2024-03-31T23:11:22.449 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.442+0000 7f5e3e212740 10 client.15432 fetch_fsmap learned FSMap version 24 2024-03-31T23:11:22.449 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.442+0000 7f5e3e212740 1 -- 172.21.15.46:0/1893736058 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55ada46f84a0 con 0x55ada46e7bf0 2024-03-31T23:11:22.449 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.442+0000 7f5e1cff9640 1 -- 172.21.15.46:0/1893736058 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5e240046c0 con 0x55ada46e7bf0 2024-03-31T23:11:22.449 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.443+0000 7f5e1cff9640 1 -- 172.21.15.46:0/1893736058 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f5e2404f040 con 0x55ada46e7bf0 2024-03-31T23:11:22.450 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.443+0000 7f5e3e212740 10 client.15432 fetch_fsmap finished waiting for FSMap version 24 2024-03-31T23:11:22.450 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.443+0000 7f5e3e212740 10 client.15432 resolve_mds: resolved name 'h' to daemon mds.h 2024-03-31T23:11:22.450 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.443+0000 7f5e3e212740 20 client.15432 populate_metadata read hostname 'smithi046' 2024-03-31T23:11:22.450 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.443+0000 7f5e3e212740 1 --2- 172.21.15.46:0/1893736058 >> [v2:172.21.15.142:6836/2559258399,v1:172.21.15.142:6837/2559258399] conn(0x55ada4767980 0x55ada4764490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:22.450 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.443+0000 7f5e3e212740 4 client.15432 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-31T23:11:22.450 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.443+0000 7f5e3e212740 1 -- 172.21.15.46:0/1893736058 --> [v2:172.21.15.142:6836/2559258399,v1:172.21.15.142:6837/2559258399] -- command(tid 0: {"prefix": "status"}) -- 0x55ada464e380 con 0x55ada4767980 2024-03-31T23:11:22.450 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.443+0000 7f5e392f2640 1 --2- 172.21.15.46:0/1893736058 >> [v2:172.21.15.142:6836/2559258399,v1:172.21.15.142:6837/2559258399] conn(0x55ada4767980 0x55ada4764490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:22.450 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.443+0000 7f5e392f2640 1 --2- 172.21.15.46:0/1893736058 >> [v2:172.21.15.142:6836/2559258399,v1:172.21.15.142:6837/2559258399] conn(0x55ada4767980 0x55ada4764490 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=98241458f0d9d454 server_cookie=62448f70fd39f6b5 in_seq=0 out_seq=0 2024-03-31T23:11:22.450 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.443+0000 7f5e1cff9640 10 client.15432 ms_handle_connect on v2:172.21.15.142:6836/2559258399 2024-03-31T23:11:22.450 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.443+0000 7f5e1cff9640 1 -- 172.21.15.46:0/1893736058 <== mds.1 v2:172.21.15.142:6836/2559258399 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x55ada464e380 con 0x55ada4767980 2024-03-31T23:11:22.450 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.443+0000 7f5e1cff9640 10 client.15432 handle_command_reply: tid=0 2024-03-31T23:11:22.450 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.443+0000 7f5e3e212740 1 client.15432 shutdown 2024-03-31T23:11:22.450 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.443+0000 7f5e3e212740 1 -- 172.21.15.46:0/1893736058 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ada46e7bf0 msgr2=0x55ada46f3d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:22.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.443+0000 7f5e3e212740 1 --2- 172.21.15.46:0/1893736058 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ada46e7bf0 0x55ada46f3d60 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f5e2402bc70 tx=0x7f5e24009600 comp rx=0 tx=0).stop 2024-03-31T23:11:22.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.443+0000 7f5e3e212740 1 -- 172.21.15.46:0/1893736058 shutdown_connections 2024-03-31T23:11:22.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.443+0000 7f5e3e212740 1 --2- 172.21.15.46:0/1893736058 >> [v2:172.21.15.142:6836/2559258399,v1:172.21.15.142:6837/2559258399] conn(0x55ada4767980 0x55ada4764490 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:22.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.443+0000 7f5e3e212740 1 --2- 172.21.15.46:0/1893736058 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55ada46e85b0 0x55ada46f6d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:22.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.443+0000 7f5e1cff9640 0 client.15432 ms_handle_reset on v2:172.21.15.142:6836/2559258399 2024-03-31T23:11:22.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.443+0000 7f5e3e212740 1 --2- 172.21.15.46:0/1893736058 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ada46e7bf0 0x55ada46f3d60 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:22.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.443+0000 7f5e3e212740 1 --2- 172.21.15.46:0/1893736058 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ada46e6a20 0x55ada46f3820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:22.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.443+0000 7f5e3e212740 1 -- 172.21.15.46:0/1893736058 >> 172.21.15.46:0/1893736058 conn(0x55ada46dda00 msgr2=0x55ada46e0c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:11:22.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.444+0000 7f5e3e212740 1 -- 172.21.15.46:0/1893736058 shutdown_connections 2024-03-31T23:11:22.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.444+0000 7f5e3e212740 1 -- 172.21.15.46:0/1893736058 wait complete. 2024-03-31T23:11:22.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.444+0000 7f5e3e212740 20 client.15432 trim_cache size 0 max 16384 2024-03-31T23:11:22.451 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-03-31T23:11:22.452 INFO:teuthology.orchestra.run.smithi046.stdout: "cluster_fsid": "6a9d8f08-efb2-11ee-b647-cb9ed24678a4", 2024-03-31T23:11:22.452 INFO:teuthology.orchestra.run.smithi046.stdout: "whoami": 1, 2024-03-31T23:11:22.452 INFO:teuthology.orchestra.run.smithi046.stdout: "id": 24482, 2024-03-31T23:11:22.452 INFO:teuthology.orchestra.run.smithi046.stdout: "want_state": "up:active", 2024-03-31T23:11:22.452 INFO:teuthology.orchestra.run.smithi046.stdout: "state": "up:active", 2024-03-31T23:11:22.452 INFO:teuthology.orchestra.run.smithi046.stdout: "fs_name": "cephfs", 2024-03-31T23:11:22.452 INFO:teuthology.orchestra.run.smithi046.stdout: "rank_uptime": 113.029236129, 2024-03-31T23:11:22.452 INFO:teuthology.orchestra.run.smithi046.stdout: "mdsmap_epoch": 24, 2024-03-31T23:11:22.452 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap_epoch": 95, 2024-03-31T23:11:22.452 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap_epoch_barrier": 91, 2024-03-31T23:11:22.452 INFO:teuthology.orchestra.run.smithi046.stdout: "uptime": 146.83483320799999 2024-03-31T23:11:22.452 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-03-31T23:11:22.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.444+0000 7f5e3b556640 1 -- 172.21.15.46:0/2133344226 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f5e200a2520 msgr2=0x7f5e200a49e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:22.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.444+0000 7f5e3b556640 1 --2- 172.21.15.46:0/2133344226 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f5e200a2520 0x7f5e200a49e0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f5e24009230 tx=0x7f5e24005e80 comp rx=0 tx=0).stop 2024-03-31T23:11:22.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.444+0000 7f5e3b556640 1 -- 172.21.15.46:0/2133344226 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5e34102200 msgr2=0x7f5e3419ef80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:22.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.444+0000 7f5e3b556640 1 --2- 172.21.15.46:0/2133344226 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5e34102200 0x7f5e3419ef80 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f5e2c00eb10 tx=0x7f5e2c00c2d0 comp rx=0 tx=0).stop 2024-03-31T23:11:22.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.444+0000 7f5e3b556640 1 -- 172.21.15.46:0/2133344226 shutdown_connections 2024-03-31T23:11:22.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.444+0000 7f5e3b556640 1 --2- 172.21.15.46:0/2133344226 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f5e200a2520 0x7f5e200a49e0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:22.454 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.444+0000 7f5e3b556640 1 --2- 172.21.15.46:0/2133344226 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5e34102200 0x7f5e3419ef80 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:22.454 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.444+0000 7f5e3b556640 1 --2- 172.21.15.46:0/2133344226 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5e34101840 0x7f5e34197f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:22.454 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.444+0000 7f5e3b556640 1 --2- 172.21.15.46:0/2133344226 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5e34100650 0x7f5e341979d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:22.454 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.444+0000 7f5e3b556640 1 -- 172.21.15.46:0/2133344226 >> 172.21.15.46:0/2133344226 conn(0x7f5e340fbbe0 msgr2=0x7f5e340688b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:11:22.454 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.444+0000 7f5e3b556640 1 -- 172.21.15.46:0/2133344226 shutdown_connections 2024-03-31T23:11:22.454 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.444+0000 7f5e3b556640 1 -- 172.21.15.46:0/2133344226 wait complete. 2024-03-31T23:11:22.455 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-03-31T23:11:22.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.559+0000 7f718931e640 1 -- 172.21.15.46:0/2903595391 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7184101780 msgr2=0x7f7184101ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:22.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.559+0000 7f718931e640 1 --2- 172.21.15.46:0/2903595391 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7184101780 0x7f7184101ba0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f717c009900 tx=0x7f717c02bc30 comp rx=0 tx=0).stop 2024-03-31T23:11:22.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.559+0000 7f718931e640 1 -- 172.21.15.46:0/2903595391 shutdown_connections 2024-03-31T23:11:22.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.559+0000 7f718931e640 1 --2- 172.21.15.46:0/2903595391 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7184103330 0x7f7184109bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:22.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.559+0000 7f718931e640 1 --2- 172.21.15.46:0/2903595391 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7184102970 0x7f7184102df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:22.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.559+0000 7f718931e640 1 --2- 172.21.15.46:0/2903595391 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7184101780 0x7f7184101ba0 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:22.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.559+0000 7f718931e640 1 -- 172.21.15.46:0/2903595391 >> 172.21.15.46:0/2903595391 conn(0x7f71840fccf0 msgr2=0x7f71840ff130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:11:22.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.559+0000 7f718931e640 1 -- 172.21.15.46:0/2903595391 shutdown_connections 2024-03-31T23:11:22.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.559+0000 7f718931e640 1 -- 172.21.15.46:0/2903595391 wait complete. 2024-03-31T23:11:22.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.560+0000 7f718931e640 1 Processor -- start 2024-03-31T23:11:22.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.560+0000 7f718931e640 1 -- start start 2024-03-31T23:11:22.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.560+0000 7f718931e640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7184101780 0x7f71841979b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:22.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.560+0000 7f718931e640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7184102970 0x7f7184197ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:22.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.560+0000 7f718931e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7184103330 0x7f718419ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:22.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.560+0000 7f718931e640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f718410d2b0 con 0x7f7184103330 2024-03-31T23:11:22.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.560+0000 7f718931e640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f718410d130 con 0x7f7184102970 2024-03-31T23:11:22.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.560+0000 7f718931e640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f718410d430 con 0x7f7184101780 2024-03-31T23:11:22.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.560+0000 7f71837fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7184103330 0x7f718419ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:22.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.560+0000 7f7182ffd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7184101780 0x7f71841979b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:22.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.561+0000 7f71837fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7184103330 0x7f718419ef60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:58276/0 (socket says 172.21.15.46:58276) 2024-03-31T23:11:22.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.561+0000 7f7182ffd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7184101780 0x7f71841979b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.46:33974/0 (socket says 172.21.15.46:33974) 2024-03-31T23:11:22.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.561+0000 7f71837fe640 1 -- 172.21.15.46:0/3546206815 learned_addr learned my addr 172.21.15.46:0/3546206815 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:11:22.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.561+0000 7f71827fc640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7184102970 0x7f7184197ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:22.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.561+0000 7f7182ffd640 1 -- 172.21.15.46:0/3546206815 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7184102970 msgr2=0x7f7184197ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:22.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.561+0000 7f7182ffd640 1 --2- 172.21.15.46:0/3546206815 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7184102970 0x7f7184197ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:22.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.561+0000 7f7182ffd640 1 -- 172.21.15.46:0/3546206815 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7184103330 msgr2=0x7f718419ef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:22.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.561+0000 7f7182ffd640 1 --2- 172.21.15.46:0/3546206815 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7184103330 0x7f718419ef60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:22.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.561+0000 7f7182ffd640 1 -- 172.21.15.46:0/3546206815 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f718419f5d0 con 0x7f7184101780 2024-03-31T23:11:22.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.561+0000 7f71837fe640 1 --2- 172.21.15.46:0/3546206815 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7184103330 0x7f718419ef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:11:22.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.561+0000 7f7182ffd640 1 --2- 172.21.15.46:0/3546206815 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7184101780 0x7f71841979b0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f717c036040 tx=0x7f717c005dc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:11:22.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.561+0000 7f717bfff640 1 -- 172.21.15.46:0/3546206815 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f717c043020 con 0x7f7184101780 2024-03-31T23:11:22.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.561+0000 7f717bfff640 1 -- 172.21.15.46:0/3546206815 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f717c004350 con 0x7f7184101780 2024-03-31T23:11:22.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.561+0000 7f718931e640 1 -- 172.21.15.46:0/3546206815 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f718419f860 con 0x7f7184101780 2024-03-31T23:11:22.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.561+0000 7f717bfff640 1 -- 172.21.15.46:0/3546206815 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f717c003ab0 con 0x7f7184101780 2024-03-31T23:11:22.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.561+0000 7f718931e640 1 -- 172.21.15.46:0/3546206815 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f718419fd80 con 0x7f7184101780 2024-03-31T23:11:22.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.563+0000 7f717bfff640 1 -- 172.21.15.46:0/3546206815 <== mon.1 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 18) ==== 137320+0+0 (secure 0 0 0) 0x7f717c046020 con 0x7f7184101780 2024-03-31T23:11:22.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.564+0000 7f718bfda740 1 Processor -- start 2024-03-31T23:11:22.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.564+0000 7f718bfda740 1 -- start start 2024-03-31T23:11:22.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.564+0000 7f718bfda740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5556ef826a20 0x5556ef826e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:22.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.564+0000 7f718bfda740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5556ef827bf0 0x5556ef828070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:22.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.564+0000 7f717bfff640 1 --2- 172.21.15.46:0/3546206815 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f71700a24a0 0x7f71700a4960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:22.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.564+0000 7f718bfda740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5556ef8285b0 0x5556ef82ee40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:22.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.564+0000 7f718bfda740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x5556ef786af0 con 0x5556ef827bf0 2024-03-31T23:11:22.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.564+0000 7f718bfda740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x5556ef72f9d0 con 0x5556ef8285b0 2024-03-31T23:11:22.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.564+0000 7f718bfda740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5556ef782b90 con 0x5556ef826a20 2024-03-31T23:11:22.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.564+0000 7f71827fc640 1 --2- 172.21.15.46:0/3546206815 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f71700a24a0 0x7f71700a4960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:22.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.564+0000 7f7182ffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5556ef827bf0 0x5556ef828070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:22.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.564+0000 7f71837fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5556ef826a20 0x5556ef826e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:22.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.564+0000 7f7182ffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5556ef827bf0 0x5556ef828070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:58284/0 (socket says 172.21.15.46:58284) 2024-03-31T23:11:22.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.564+0000 7f7182ffd640 1 -- 172.21.15.46:0/2107787648 learned_addr learned my addr 172.21.15.46:0/2107787648 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:11:22.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.564+0000 7f71837fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5556ef826a20 0x5556ef826e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.46:33986/0 (socket says 172.21.15.46:33986) 2024-03-31T23:11:22.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.564+0000 7f717bfff640 1 -- 172.21.15.46:0/3546206815 <== mon.1 v2:172.21.15.162:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9208+0+0 (secure 0 0 0) 0x7f717c0f0bd0 con 0x7f7184101780 2024-03-31T23:11:22.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.564+0000 7f71837fe640 1 --2- 172.21.15.46:0/2107787648 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5556ef8285b0 0x5556ef82ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:22.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.565+0000 7f7182ffd640 1 -- 172.21.15.46:0/2107787648 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5556ef826a20 msgr2=0x5556ef826e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:11:22.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.565+0000 7f7182ffd640 1 --2- 172.21.15.46:0/2107787648 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5556ef826a20 0x5556ef826e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:22.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.565+0000 7f7182ffd640 1 -- 172.21.15.46:0/2107787648 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5556ef8285b0 msgr2=0x5556ef82ee40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:11:22.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.565+0000 7f7182ffd640 1 --2- 172.21.15.46:0/2107787648 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5556ef8285b0 0x5556ef82ee40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:22.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.565+0000 7f7182ffd640 1 -- 172.21.15.46:0/2107787648 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5556ef49b2d0 con 0x5556ef827bf0 2024-03-31T23:11:22.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.565+0000 7f71827fc640 1 --2- 172.21.15.46:0/3546206815 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f71700a24a0 0x7f71700a4960 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f7184198ec0 tx=0x7f716c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:11:22.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.565+0000 7f7182ffd640 1 --2- 172.21.15.46:0/2107787648 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5556ef827bf0 0x5556ef828070 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f717c0ee300 tx=0x7f717c088640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d2f1372669192f6 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:11:22.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.567+0000 7f71797fa640 1 -- 172.21.15.46:0/2107787648 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f717c0bd090 con 0x5556ef827bf0 2024-03-31T23:11:22.573 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.567+0000 7f71797fa640 1 -- 172.21.15.46:0/2107787648 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f717c0bd9c0 con 0x5556ef827bf0 2024-03-31T23:11:22.573 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.567+0000 7f71797fa640 1 -- 172.21.15.46:0/2107787648 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f717c08f580 con 0x5556ef827bf0 2024-03-31T23:11:22.573 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.567+0000 7f718bfda740 1 -- 172.21.15.46:0/2107787648 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5556ef827bf0 msgr2=0x5556ef828070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:22.573 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.567+0000 7f718bfda740 1 --2- 172.21.15.46:0/2107787648 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5556ef827bf0 0x5556ef828070 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f717c0ee300 tx=0x7f717c088640 comp rx=0 tx=0).stop 2024-03-31T23:11:22.573 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.567+0000 7f718bfda740 1 -- 172.21.15.46:0/2107787648 shutdown_connections 2024-03-31T23:11:22.573 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.567+0000 7f718bfda740 1 --2- 172.21.15.46:0/2107787648 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5556ef8285b0 0x5556ef82ee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:22.573 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.567+0000 7f718bfda740 1 --2- 172.21.15.46:0/2107787648 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5556ef827bf0 0x5556ef828070 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:22.573 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.567+0000 7f718bfda740 1 --2- 172.21.15.46:0/2107787648 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5556ef826a20 0x5556ef826e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:22.573 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.567+0000 7f718bfda740 1 -- 172.21.15.46:0/2107787648 >> 172.21.15.46:0/2107787648 conn(0x5556ef81da00 msgr2=0x5556ef81de00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:11:22.573 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.567+0000 7f718bfda740 1 -- 172.21.15.46:0/2107787648 shutdown_connections 2024-03-31T23:11:22.573 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.567+0000 7f718bfda740 1 -- 172.21.15.46:0/2107787648 wait complete. 2024-03-31T23:11:22.573 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.568+0000 7f718bfda740 1 -- start start 2024-03-31T23:11:22.573 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.568+0000 7f718bfda740 1 Processor -- start 2024-03-31T23:11:22.573 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.568+0000 7f718bfda740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5556ef826a20 0x5556ef8309d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:22.574 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.568+0000 7f718bfda740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5556ef827bf0 0x5556ef830f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:22.574 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.568+0000 7f718bfda740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5556ef8285b0 0x5556ef837f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:22.574 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.568+0000 7f718bfda740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x5556ef82f380 con 0x5556ef827bf0 2024-03-31T23:11:22.574 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.568+0000 7f718bfda740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x5556ef742f70 con 0x5556ef8285b0 2024-03-31T23:11:22.574 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.568+0000 7f718bfda740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5556ef82f500 con 0x5556ef826a20 2024-03-31T23:11:22.574 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.568+0000 7f718bfda740 10 client.0 _ll_register_callbacks cb 0x5556ef49aac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T23:11:22.574 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.568+0000 7f71827fc640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5556ef827bf0 0x5556ef830f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:22.574 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.568+0000 7f71827fc640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5556ef827bf0 0x5556ef830f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:58290/0 (socket says 172.21.15.46:58290) 2024-03-31T23:11:22.574 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.568+0000 7f71827fc640 1 -- 172.21.15.46:0/708021205 learned_addr learned my addr 172.21.15.46:0/708021205 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:11:22.574 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.568+0000 7f7182ffd640 1 --2- 172.21.15.46:0/708021205 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5556ef826a20 0x5556ef8309d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:22.574 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.568+0000 7f71837fe640 1 --2- 172.21.15.46:0/708021205 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5556ef8285b0 0x5556ef837f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:22.574 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.569+0000 7f71827fc640 1 -- 172.21.15.46:0/708021205 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5556ef826a20 msgr2=0x5556ef8309d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:11:22.574 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.569+0000 7f71827fc640 1 --2- 172.21.15.46:0/708021205 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5556ef826a20 0x5556ef8309d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:22.575 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.569+0000 7f71827fc640 1 -- 172.21.15.46:0/708021205 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5556ef8285b0 msgr2=0x5556ef837f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:11:22.575 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.569+0000 7f71827fc640 1 --2- 172.21.15.46:0/708021205 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5556ef8285b0 0x5556ef837f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:22.575 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.569+0000 7f71827fc640 1 -- 172.21.15.46:0/708021205 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5556ef5656b0 con 0x5556ef827bf0 2024-03-31T23:11:22.575 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.569+0000 7f7182ffd640 1 --2- 172.21.15.46:0/708021205 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5556ef826a20 0x5556ef8309d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:11:22.575 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.569+0000 7f71837fe640 1 --2- 172.21.15.46:0/708021205 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5556ef8285b0 0x5556ef837f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:11:22.575 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.569+0000 7f71827fc640 1 --2- 172.21.15.46:0/708021205 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5556ef827bf0 0x5556ef830f10 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f716c00ec30 tx=0x7f716c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=229552f4e89dbef2 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:11:22.575 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.569+0000 7f716a7fc640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-31T23:11:22.575 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.569+0000 7f716a7fc640 1 -- 172.21.15.46:0/708021205 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f716c01caa0 con 0x5556ef827bf0 2024-03-31T23:11:22.575 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.569+0000 7f716a7fc640 1 -- 172.21.15.46:0/708021205 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9208+0+0 (secure 0 0 0) 0x7f716c01cc40 con 0x5556ef827bf0 2024-03-31T23:11:22.575 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.569+0000 7f718bfda740 1 -- 172.21.15.46:0/708021205 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f717c0bd9c0 con 0x5556ef827bf0 2024-03-31T23:11:22.575 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.569+0000 7f716a7fc640 1 -- 172.21.15.46:0/708021205 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f716c01ce40 con 0x5556ef827bf0 2024-03-31T23:11:22.575 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.569+0000 7f718bfda740 10 client.15444 fetch_fsmap learned FSMap version 24 2024-03-31T23:11:22.575 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.569+0000 7f718bfda740 1 -- 172.21.15.46:0/708021205 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x5556ef83b000 con 0x5556ef827bf0 2024-03-31T23:11:22.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.569+0000 7f716a7fc640 1 -- 172.21.15.46:0/708021205 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f717c0bd9c0 con 0x5556ef827bf0 2024-03-31T23:11:22.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.569+0000 7f716a7fc640 1 -- 172.21.15.46:0/708021205 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f716c01d5d0 con 0x5556ef827bf0 2024-03-31T23:11:22.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.569+0000 7f718bfda740 10 client.15444 fetch_fsmap finished waiting for FSMap version 24 2024-03-31T23:11:22.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.569+0000 7f718bfda740 10 client.15444 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-31T23:11:22.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.570+0000 7f718bfda740 20 client.15444 populate_metadata read hostname 'smithi046' 2024-03-31T23:11:22.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.570+0000 7f718bfda740 1 --2- 172.21.15.46:0/708021205 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x5556ef83d5e0 0x5556ef83ea50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:22.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.570+0000 7f718bfda740 4 client.15444 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-31T23:11:22.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.570+0000 7f718bfda740 1 -- 172.21.15.46:0/708021205 --> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5556ef794030 con 0x5556ef83d5e0 2024-03-31T23:11:22.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.570+0000 7f7182ffd640 1 --2- 172.21.15.46:0/708021205 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x5556ef83d5e0 0x5556ef83ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:22.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.570+0000 7f7182ffd640 1 --2- 172.21.15.46:0/708021205 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x5556ef83d5e0 0x5556ef83ea50 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e8cab75f06725f62 server_cookie=a2ef4ee60e8f34ce in_seq=0 out_seq=0 2024-03-31T23:11:22.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.570+0000 7f716a7fc640 10 client.15444 ms_handle_connect on v2:172.21.15.142:6838/2519143918 2024-03-31T23:11:22.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.571+0000 7f716a7fc640 1 -- 172.21.15.46:0/708021205 <== mds.0 v2:172.21.15.142:6838/2519143918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5556ef794030 con 0x5556ef83d5e0 2024-03-31T23:11:22.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.571+0000 7f716a7fc640 10 client.15444 handle_command_reply: tid=0 2024-03-31T23:11:22.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.571+0000 7f718bfda740 1 client.15444 shutdown 2024-03-31T23:11:22.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.571+0000 7f718bfda740 1 -- 172.21.15.46:0/708021205 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5556ef827bf0 msgr2=0x5556ef830f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:22.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.571+0000 7f718bfda740 1 --2- 172.21.15.46:0/708021205 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5556ef827bf0 0x5556ef830f10 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f716c00ec30 tx=0x7f716c004730 comp rx=0 tx=0).stop 2024-03-31T23:11:22.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.571+0000 7f718bfda740 1 -- 172.21.15.46:0/708021205 shutdown_connections 2024-03-31T23:11:22.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.571+0000 7f718bfda740 1 --2- 172.21.15.46:0/708021205 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x5556ef83d5e0 0x5556ef83ea50 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:22.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.571+0000 7f718bfda740 1 --2- 172.21.15.46:0/708021205 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5556ef8285b0 0x5556ef837f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:22.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.571+0000 7f716a7fc640 0 client.15444 ms_handle_reset on v2:172.21.15.142:6838/2519143918 2024-03-31T23:11:22.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.571+0000 7f718bfda740 1 --2- 172.21.15.46:0/708021205 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5556ef827bf0 0x5556ef830f10 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:22.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.571+0000 7f718bfda740 1 --2- 172.21.15.46:0/708021205 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5556ef826a20 0x5556ef8309d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:22.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.571+0000 7f718bfda740 1 -- 172.21.15.46:0/708021205 >> 172.21.15.46:0/708021205 conn(0x5556ef81da00 msgr2=0x5556ef81cb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:11:22.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.571+0000 7f718bfda740 1 -- 172.21.15.46:0/708021205 shutdown_connections 2024-03-31T23:11:22.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.571+0000 7f718bfda740 1 -- 172.21.15.46:0/708021205 wait complete. 2024-03-31T23:11:22.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.571+0000 7f718bfda740 20 client.15444 trim_cache size 0 max 16384 2024-03-31T23:11:22.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.589+0000 7f718bfda740 1 Processor -- start 2024-03-31T23:11:22.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.589+0000 7f718bfda740 1 -- start start 2024-03-31T23:11:22.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.589+0000 7f718bfda740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5556ef826a20 0x5556ef81c2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:22.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.589+0000 7f718bfda740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5556ef827bf0 0x5556ef837f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:22.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.589+0000 7f718bfda740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5556ef8285b0 0x5556ef81c7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:22.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.589+0000 7f718bfda740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x5556ef83cd90 con 0x5556ef826a20 2024-03-31T23:11:22.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.589+0000 7f718bfda740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x5556ef83d460 con 0x5556ef8285b0 2024-03-31T23:11:22.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.589+0000 7f718bfda740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5556ef83db30 con 0x5556ef827bf0 2024-03-31T23:11:22.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.589+0000 7f7182ffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5556ef826a20 0x5556ef81c2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:22.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.590+0000 7f7182ffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5556ef826a20 0x5556ef81c2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:58296/0 (socket says 172.21.15.46:58296) 2024-03-31T23:11:22.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.590+0000 7f7182ffd640 1 -- 172.21.15.46:0/362683035 learned_addr learned my addr 172.21.15.46:0/362683035 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:11:22.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.590+0000 7f71827fc640 1 --2- 172.21.15.46:0/362683035 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5556ef827bf0 0x5556ef837f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:22.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.590+0000 7f71837fe640 1 --2- 172.21.15.46:0/362683035 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5556ef8285b0 0x5556ef81c7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:22.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.590+0000 7f7182ffd640 1 -- 172.21.15.46:0/362683035 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5556ef827bf0 msgr2=0x5556ef837f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:11:22.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.590+0000 7f7182ffd640 1 --2- 172.21.15.46:0/362683035 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5556ef827bf0 0x5556ef837f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:22.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.590+0000 7f7182ffd640 1 -- 172.21.15.46:0/362683035 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5556ef8285b0 msgr2=0x5556ef81c7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:11:22.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.590+0000 7f7182ffd640 1 --2- 172.21.15.46:0/362683035 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5556ef8285b0 0x5556ef81c7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:22.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.590+0000 7f7182ffd640 1 -- 172.21.15.46:0/362683035 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5556ef827670 con 0x5556ef826a20 2024-03-31T23:11:22.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.590+0000 7f7182ffd640 1 --2- 172.21.15.46:0/362683035 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5556ef826a20 0x5556ef81c2b0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f717c0ee670 tx=0x7f717c0dd040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fbdeab003fe28423 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:11:22.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.590+0000 7f7178ff9640 1 -- 172.21.15.46:0/362683035 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f717c0bcc20 con 0x5556ef826a20 2024-03-31T23:11:22.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.590+0000 7f7178ff9640 1 -- 172.21.15.46:0/362683035 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f717c0bd9c0 con 0x5556ef826a20 2024-03-31T23:11:22.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.590+0000 7f7178ff9640 1 -- 172.21.15.46:0/362683035 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f717c0bce00 con 0x5556ef826a20 2024-03-31T23:11:22.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.591+0000 7f718bfda740 1 -- 172.21.15.46:0/362683035 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5556ef826a20 msgr2=0x5556ef81c2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:22.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.591+0000 7f718bfda740 1 --2- 172.21.15.46:0/362683035 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5556ef826a20 0x5556ef81c2b0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f717c0ee670 tx=0x7f717c0dd040 comp rx=0 tx=0).stop 2024-03-31T23:11:22.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.591+0000 7f718bfda740 1 -- 172.21.15.46:0/362683035 shutdown_connections 2024-03-31T23:11:22.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.591+0000 7f718bfda740 1 --2- 172.21.15.46:0/362683035 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5556ef8285b0 0x5556ef81c7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:22.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.591+0000 7f718bfda740 1 --2- 172.21.15.46:0/362683035 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5556ef827bf0 0x5556ef837f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:22.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.591+0000 7f718bfda740 1 --2- 172.21.15.46:0/362683035 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5556ef826a20 0x5556ef81c2b0 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:22.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.591+0000 7f718bfda740 1 -- 172.21.15.46:0/362683035 >> 172.21.15.46:0/362683035 conn(0x5556ef81da00 msgr2=0x5556ef82eb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:11:22.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.591+0000 7f718bfda740 1 -- 172.21.15.46:0/362683035 shutdown_connections 2024-03-31T23:11:22.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.591+0000 7f718bfda740 1 -- 172.21.15.46:0/362683035 wait complete. 2024-03-31T23:11:22.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.591+0000 7f718bfda740 1 -- start start 2024-03-31T23:11:22.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.591+0000 7f718bfda740 1 Processor -- start 2024-03-31T23:11:22.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.591+0000 7f718bfda740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5556ef826a20 0x5556ef82de90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:22.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.591+0000 7f718bfda740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5556ef827bf0 0x5556ef837f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:22.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.591+0000 7f718bfda740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5556ef8285b0 0x5556ef8a1340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:22.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.591+0000 7f718bfda740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x5556ef83e8d0 con 0x5556ef8285b0 2024-03-31T23:11:22.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.591+0000 7f718bfda740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x5556ef83e200 con 0x5556ef827bf0 2024-03-31T23:11:22.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.591+0000 7f718bfda740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5556ef83efa0 con 0x5556ef826a20 2024-03-31T23:11:22.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.591+0000 7f718bfda740 10 client.0 _ll_register_callbacks cb 0x7f716c00e8e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T23:11:22.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.591+0000 7f71837fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5556ef8285b0 0x5556ef8a1340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:22.598 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.592+0000 7f71827fc640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5556ef827bf0 0x5556ef837f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:22.598 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.592+0000 7f7182ffd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5556ef826a20 0x5556ef82de90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:22.598 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.592+0000 7f71837fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5556ef8285b0 0x5556ef8a1340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:58298/0 (socket says 172.21.15.46:58298) 2024-03-31T23:11:22.598 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.592+0000 7f71837fe640 1 -- 172.21.15.46:0/1632607674 learned_addr learned my addr 172.21.15.46:0/1632607674 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:11:22.598 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.592+0000 7f71837fe640 1 -- 172.21.15.46:0/1632607674 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5556ef826a20 msgr2=0x5556ef82de90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:11:22.598 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.592+0000 7f71837fe640 1 --2- 172.21.15.46:0/1632607674 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5556ef826a20 0x5556ef82de90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:22.598 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.592+0000 7f71837fe640 1 -- 172.21.15.46:0/1632607674 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5556ef827bf0 msgr2=0x5556ef837f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:11:22.598 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.592+0000 7f71837fe640 1 --2- 172.21.15.46:0/1632607674 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5556ef827bf0 0x5556ef837f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:22.598 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.592+0000 7f71837fe640 1 -- 172.21.15.46:0/1632607674 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5556ef8a1b70 con 0x5556ef8285b0 2024-03-31T23:11:22.598 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.592+0000 7f71837fe640 1 --2- 172.21.15.46:0/1632607674 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5556ef8285b0 0x5556ef8a1340 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f7174013880 tx=0x7f71740108c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5769273b1379e850 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:11:22.598 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.592+0000 7f716a7fc640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-31T23:11:22.598 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.592+0000 7f716a7fc640 1 -- 172.21.15.46:0/1632607674 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7174020740 con 0x5556ef8285b0 2024-03-31T23:11:22.598 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.592+0000 7f716a7fc640 1 -- 172.21.15.46:0/1632607674 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9208+0+0 (secure 0 0 0) 0x7f7174009040 con 0x5556ef8285b0 2024-03-31T23:11:22.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.592+0000 7f718bfda740 1 -- 172.21.15.46:0/1632607674 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f717c0bd9c0 con 0x5556ef8285b0 2024-03-31T23:11:22.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.592+0000 7f716a7fc640 1 -- 172.21.15.46:0/1632607674 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f717402e050 con 0x5556ef8285b0 2024-03-31T23:11:22.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.592+0000 7f718bfda740 10 client.15450 fetch_fsmap learned FSMap version 24 2024-03-31T23:11:22.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.592+0000 7f718bfda740 1 -- 172.21.15.46:0/1632607674 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x5556ef8a2ee0 con 0x5556ef8285b0 2024-03-31T23:11:22.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.592+0000 7f716a7fc640 1 -- 172.21.15.46:0/1632607674 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f717c0bd9c0 con 0x5556ef8285b0 2024-03-31T23:11:22.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.593+0000 7f716a7fc640 1 -- 172.21.15.46:0/1632607674 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f7174029a50 con 0x5556ef8285b0 2024-03-31T23:11:22.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.593+0000 7f718bfda740 10 client.15450 fetch_fsmap finished waiting for FSMap version 24 2024-03-31T23:11:22.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.593+0000 7f718bfda740 10 client.15450 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-31T23:11:22.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.593+0000 7f718bfda740 20 client.15450 populate_metadata read hostname 'smithi046' 2024-03-31T23:11:22.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.593+0000 7f718bfda740 1 --2- 172.21.15.46:0/1632607674 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x5556ef8a7b90 0x5556ef8a68a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:22.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.593+0000 7f718bfda740 4 client.15450 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-31T23:11:22.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.593+0000 7f718bfda740 1 -- 172.21.15.46:0/1632607674 --> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] -- command(tid 0: {"prefix": "status"}) -- 0x5556ef78e380 con 0x5556ef8a7b90 2024-03-31T23:11:22.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.593+0000 7f7182ffd640 1 --2- 172.21.15.46:0/1632607674 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x5556ef8a7b90 0x5556ef8a68a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:22.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.593+0000 7f716a7fc640 10 client.15450 ms_handle_connect on v2:172.21.15.142:6838/2519143918 2024-03-31T23:11:22.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.593+0000 7f7182ffd640 1 --2- 172.21.15.46:0/1632607674 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x5556ef8a7b90 0x5556ef8a68a0 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c4e2eb0407de559a server_cookie=a4265b2713c0946 in_seq=0 out_seq=0 2024-03-31T23:11:22.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.593+0000 7f716a7fc640 1 -- 172.21.15.46:0/1632607674 <== mds.0 v2:172.21.15.142:6838/2519143918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x5556ef78e380 con 0x5556ef8a7b90 2024-03-31T23:11:22.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.593+0000 7f716a7fc640 10 client.15450 handle_command_reply: tid=0 2024-03-31T23:11:22.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.593+0000 7f718bfda740 1 client.15450 shutdown 2024-03-31T23:11:22.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.594+0000 7f718bfda740 1 -- 172.21.15.46:0/1632607674 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5556ef8285b0 msgr2=0x5556ef8a1340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:22.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.594+0000 7f718bfda740 1 --2- 172.21.15.46:0/1632607674 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5556ef8285b0 0x5556ef8a1340 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f7174013880 tx=0x7f71740108c0 comp rx=0 tx=0).stop 2024-03-31T23:11:22.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.594+0000 7f718bfda740 1 -- 172.21.15.46:0/1632607674 shutdown_connections 2024-03-31T23:11:22.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.594+0000 7f718bfda740 1 --2- 172.21.15.46:0/1632607674 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x5556ef8a7b90 0x5556ef8a68a0 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:22.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.594+0000 7f718bfda740 1 --2- 172.21.15.46:0/1632607674 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5556ef8285b0 0x5556ef8a1340 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:22.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.594+0000 7f718bfda740 1 --2- 172.21.15.46:0/1632607674 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5556ef827bf0 0x5556ef837f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:22.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.594+0000 7f716a7fc640 0 client.15450 ms_handle_reset on v2:172.21.15.142:6838/2519143918 2024-03-31T23:11:22.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.594+0000 7f718bfda740 1 --2- 172.21.15.46:0/1632607674 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5556ef826a20 0x5556ef82de90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:22.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.594+0000 7f718bfda740 1 -- 172.21.15.46:0/1632607674 >> 172.21.15.46:0/1632607674 conn(0x5556ef81da00 msgr2=0x5556ef8214f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:11:22.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.594+0000 7f718bfda740 1 -- 172.21.15.46:0/1632607674 shutdown_connections 2024-03-31T23:11:22.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.594+0000 7f718bfda740 1 -- 172.21.15.46:0/1632607674 wait complete. 2024-03-31T23:11:22.601 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.594+0000 7f718bfda740 20 client.15450 trim_cache size 0 max 16384 2024-03-31T23:11:22.601 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-03-31T23:11:22.601 INFO:teuthology.orchestra.run.smithi046.stdout: "cluster_fsid": "6a9d8f08-efb2-11ee-b647-cb9ed24678a4", 2024-03-31T23:11:22.601 INFO:teuthology.orchestra.run.smithi046.stdout: "whoami": 0, 2024-03-31T23:11:22.601 INFO:teuthology.orchestra.run.smithi046.stdout: "id": 24515, 2024-03-31T23:11:22.601 INFO:teuthology.orchestra.run.smithi046.stdout: "want_state": "up:active", 2024-03-31T23:11:22.601 INFO:teuthology.orchestra.run.smithi046.stdout: "state": "up:active", 2024-03-31T23:11:22.601 INFO:teuthology.orchestra.run.smithi046.stdout: "fs_name": "cephfs", 2024-03-31T23:11:22.601 INFO:teuthology.orchestra.run.smithi046.stdout: "rank_uptime": 130.997016608, 2024-03-31T23:11:22.601 INFO:teuthology.orchestra.run.smithi046.stdout: "mdsmap_epoch": 24, 2024-03-31T23:11:22.601 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap_epoch": 95, 2024-03-31T23:11:22.602 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap_epoch_barrier": 91, 2024-03-31T23:11:22.602 INFO:teuthology.orchestra.run.smithi046.stdout: "uptime": 135.33149463500001 2024-03-31T23:11:22.602 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-03-31T23:11:22.602 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.594+0000 7f718931e640 1 -- 172.21.15.46:0/3546206815 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f71700a24a0 msgr2=0x7f71700a4960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:22.603 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.594+0000 7f718931e640 1 --2- 172.21.15.46:0/3546206815 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f71700a24a0 0x7f71700a4960 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f7184198ec0 tx=0x7f716c005f90 comp rx=0 tx=0).stop 2024-03-31T23:11:22.603 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.594+0000 7f718931e640 1 -- 172.21.15.46:0/3546206815 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7184101780 msgr2=0x7f71841979b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:22.603 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.594+0000 7f718931e640 1 --2- 172.21.15.46:0/3546206815 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7184101780 0x7f71841979b0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f717c036040 tx=0x7f717c005dc0 comp rx=0 tx=0).stop 2024-03-31T23:11:22.603 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.594+0000 7f718931e640 1 -- 172.21.15.46:0/3546206815 shutdown_connections 2024-03-31T23:11:22.603 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.594+0000 7f718931e640 1 --2- 172.21.15.46:0/3546206815 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f71700a24a0 0x7f71700a4960 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:22.603 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.594+0000 7f718931e640 1 --2- 172.21.15.46:0/3546206815 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7184103330 0x7f718419ef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:22.603 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.594+0000 7f718931e640 1 --2- 172.21.15.46:0/3546206815 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7184102970 0x7f7184197ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:22.603 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.594+0000 7f718931e640 1 --2- 172.21.15.46:0/3546206815 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7184101780 0x7f71841979b0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:22.603 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.594+0000 7f718931e640 1 -- 172.21.15.46:0/3546206815 >> 172.21.15.46:0/3546206815 conn(0x7f71840fccf0 msgr2=0x7f7184105d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:11:22.603 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.594+0000 7f718931e640 1 -- 172.21.15.46:0/3546206815 shutdown_connections 2024-03-31T23:11:22.603 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:22.594+0000 7f718931e640 1 -- 172.21.15.46:0/3546206815 wait complete. 2024-03-31T23:11:22.605 INFO:teuthology.run_tasks:Running task ceph-fuse... 2024-03-31T23:11:22.618 INFO:tasks.ceph_fuse:Running ceph_fuse task... 2024-03-31T23:11:22.618 INFO:tasks.ceph_fuse:config is {'client.0': {}} 2024-03-31T23:11:22.618 INFO:tasks.ceph_fuse:client.0 config is {'mount_subvol_num': 0} 2024-03-31T23:11:22.619 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-03-31T23:11:22.619 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-31T23:11:22.619 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T23:11:22.619 DEBUG:teuthology.orchestra.run.smithi046:> ip netns list 2024-03-31T23:11:22.633 INFO:teuthology.orchestra.run.smithi046.stdout:ceph-ns--home-ubuntu-cephtest-mnt.admin (id: 0) 2024-03-31T23:11:22.634 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T23:11:22.634 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-31T23:11:22.701 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T23:11:22.701 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip link delete ceph-brx 2024-03-31T23:11:22.714 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:11:22 smithi046 ceph-mon[32498]: pgmap v361: 129 pgs: 129 active+clean; 642 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s; 0 B/s, 0 objects/s recovering 2024-03-31T23:11:22.714 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:11:22 smithi046 ceph-mon[32498]: from='client.? 172.21.15.46:0/934170579' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-31T23:11:22.714 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:11:22 smithi046 ceph-mon[32498]: from='client.? 172.21.15.46:0/2810218097' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-31T23:11:22.768 INFO:tasks.ceph_fuse:Mounting ceph-fuse clients... 2024-03-31T23:11:22.768 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:384f31f3d3aee7f8ba3f60846b8844c049165583 shell --fsid 6a9d8f08-efb2-11ee-b647-cb9ed24678a4 -- ceph fs ls 2024-03-31T23:11:22.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:11:22 smithi142 ceph-mon[35271]: pgmap v361: 129 pgs: 129 active+clean; 642 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s; 0 B/s, 0 objects/s recovering 2024-03-31T23:11:22.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:11:22 smithi142 ceph-mon[35271]: from='client.? 172.21.15.46:0/934170579' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-31T23:11:22.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:11:22 smithi142 ceph-mon[35271]: from='client.? 172.21.15.46:0/2810218097' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-31T23:11:23.013 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/6a9d8f08-efb2-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-31T23:11:23.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:11:22 smithi162 ceph-mon[36237]: pgmap v361: 129 pgs: 129 active+clean; 642 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s; 0 B/s, 0 objects/s recovering 2024-03-31T23:11:23.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:11:22 smithi162 ceph-mon[36237]: from='client.? 172.21.15.46:0/934170579' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-31T23:11:23.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:11:22 smithi162 ceph-mon[36237]: from='client.? 172.21.15.46:0/2810218097' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-31T23:11:24.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:24.675+0000 7f409e04e700 1 -- 172.21.15.46:0/3234154472 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4098108270 msgr2=0x7f4098108650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:24.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:24.675+0000 7f409e04e700 1 --2- 172.21.15.46:0/3234154472 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4098108270 0x7f4098108650 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f4080009990 tx=0x7f4080009ca0 comp rx=0 tx=0).stop 2024-03-31T23:11:24.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:24.675+0000 7f409e04e700 1 -- 172.21.15.46:0/3234154472 shutdown_connections 2024-03-31T23:11:24.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:24.675+0000 7f409e04e700 1 --2- 172.21.15.46:0/3234154472 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4098102c50 0x7f409810f090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:24.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:24.675+0000 7f409e04e700 1 --2- 172.21.15.46:0/3234154472 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4098108270 0x7f4098108650 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:24.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:24.675+0000 7f409e04e700 1 --2- 172.21.15.46:0/3234154472 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4098102290 0x7f4098102710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:24.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:24.675+0000 7f409e04e700 1 -- 172.21.15.46:0/3234154472 >> 172.21.15.46:0/3234154472 conn(0x7f4098079370 msgr2=0x7f4098079780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:11:24.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:24.675+0000 7f409e04e700 1 -- 172.21.15.46:0/3234154472 shutdown_connections 2024-03-31T23:11:24.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:24.675+0000 7f409e04e700 1 -- 172.21.15.46:0/3234154472 wait complete. 2024-03-31T23:11:24.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:24.676+0000 7f409e04e700 1 Processor -- start 2024-03-31T23:11:24.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:24.676+0000 7f409e04e700 1 -- start start 2024-03-31T23:11:24.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:24.676+0000 7f409e04e700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4098102290 0x7f409819e630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:24.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:24.676+0000 7f409e04e700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4098102c50 0x7f409819eb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:24.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:24.676+0000 7f409e04e700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4098108270 0x7f40981987b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:24.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:24.676+0000 7f409e04e700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f40981123f0 con 0x7f4098108270 2024-03-31T23:11:24.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:24.676+0000 7f409e04e700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f4098112270 con 0x7f4098102290 2024-03-31T23:11:24.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:24.676+0000 7f409e04e700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f4098112570 con 0x7f4098102c50 2024-03-31T23:11:24.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:24.676+0000 7f4097fff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4098108270 0x7f40981987b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:24.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:24.676+0000 7f4097fff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4098108270 0x7f40981987b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:58310/0 (socket says 172.21.15.46:58310) 2024-03-31T23:11:24.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:24.676+0000 7f4097fff700 1 -- 172.21.15.46:0/3338972682 learned_addr learned my addr 172.21.15.46:0/3338972682 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:11:24.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:24.676+0000 7f40977fe700 1 --2- 172.21.15.46:0/3338972682 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4098102290 0x7f409819e630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:24.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:24.676+0000 7f4096ffd700 1 --2- 172.21.15.46:0/3338972682 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4098102c50 0x7f409819eb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:24.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:24.677+0000 7f4097fff700 1 -- 172.21.15.46:0/3338972682 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4098102c50 msgr2=0x7f409819eb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:24.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:24.677+0000 7f4097fff700 1 --2- 172.21.15.46:0/3338972682 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4098102c50 0x7f409819eb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:24.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:24.677+0000 7f4097fff700 1 -- 172.21.15.46:0/3338972682 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4098102290 msgr2=0x7f409819e630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:24.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:24.677+0000 7f4097fff700 1 --2- 172.21.15.46:0/3338972682 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4098102290 0x7f409819e630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:24.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:24.677+0000 7f4097fff700 1 -- 172.21.15.46:0/3338972682 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4098198f90 con 0x7f4098108270 2024-03-31T23:11:24.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:24.677+0000 7f4097fff700 1 --2- 172.21.15.46:0/3338972682 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4098108270 0x7f40981987b0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f409000ef30 tx=0x7f409000c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:11:24.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:24.677+0000 7f4094ff9700 1 -- 172.21.15.46:0/3338972682 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40900092e0 con 0x7f4098108270 2024-03-31T23:11:24.680 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:24.677+0000 7f409e04e700 1 -- 172.21.15.46:0/3338972682 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4098199220 con 0x7f4098108270 2024-03-31T23:11:24.680 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:24.677+0000 7f4094ff9700 1 -- 172.21.15.46:0/3338972682 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4090004bb0 con 0x7f4098108270 2024-03-31T23:11:24.680 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:24.677+0000 7f4094ff9700 1 -- 172.21.15.46:0/3338972682 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4090004020 con 0x7f4098108270 2024-03-31T23:11:24.680 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:24.677+0000 7f409e04e700 1 -- 172.21.15.46:0/3338972682 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f40981a5490 con 0x7f4098108270 2024-03-31T23:11:24.680 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:24.678+0000 7f409e04e700 1 -- 172.21.15.46:0/3338972682 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4098103920 con 0x7f4098108270 2024-03-31T23:11:24.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:24.679+0000 7f4094ff9700 1 -- 172.21.15.46:0/3338972682 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 18) ==== 137320+0+0 (secure 0 0 0) 0x7f409001b030 con 0x7f4098108270 2024-03-31T23:11:24.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:24.680+0000 7f4094ff9700 1 --2- 172.21.15.46:0/3338972682 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f40840a2530 0x7f40840a49f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:24.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:24.680+0000 7f40977fe700 1 --2- 172.21.15.46:0/3338972682 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f40840a2530 0x7f40840a49f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:24.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:24.680+0000 7f4094ff9700 1 -- 172.21.15.46:0/3338972682 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9208+0+0 (secure 0 0 0) 0x7f40900ceec0 con 0x7f4098108270 2024-03-31T23:11:24.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:24.680+0000 7f40977fe700 1 --2- 172.21.15.46:0/3338972682 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f40840a2530 0x7f40840a49f0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f408000b2e0 tx=0x7f4080005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:11:24.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:24.683+0000 7f4094ff9700 1 -- 172.21.15.46:0/3338972682 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f4090022020 con 0x7f4098108270 2024-03-31T23:11:24.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:11:24 smithi046 ceph-mon[32498]: pgmap v362: 129 pgs: 129 active+clean; 642 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s; 0 B/s, 0 objects/s recovering 2024-03-31T23:11:24.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:24.938+0000 7f409e04e700 1 -- 172.21.15.46:0/3338972682 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f4098199d10 con 0x7f4098108270 2024-03-31T23:11:24.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:24.938+0000 7f4094ff9700 1 -- 172.21.15.46:0/3338972682 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v24) ==== 53+0+88 (secure 0 0 0) 0x7f4098199d10 con 0x7f4098108270 2024-03-31T23:11:24.939 INFO:teuthology.orchestra.run.smithi046.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-03-31T23:11:24.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:24.940+0000 7f409e04e700 1 -- 172.21.15.46:0/3338972682 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f40840a2530 msgr2=0x7f40840a49f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:24.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:24.940+0000 7f409e04e700 1 --2- 172.21.15.46:0/3338972682 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f40840a2530 0x7f40840a49f0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f408000b2e0 tx=0x7f4080005fd0 comp rx=0 tx=0).stop 2024-03-31T23:11:24.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:24.940+0000 7f409e04e700 1 -- 172.21.15.46:0/3338972682 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4098108270 msgr2=0x7f40981987b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:24.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:24.940+0000 7f409e04e700 1 --2- 172.21.15.46:0/3338972682 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4098108270 0x7f40981987b0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f409000ef30 tx=0x7f409000c3a0 comp rx=0 tx=0).stop 2024-03-31T23:11:24.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:24.940+0000 7f409e04e700 1 -- 172.21.15.46:0/3338972682 shutdown_connections 2024-03-31T23:11:24.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:24.940+0000 7f409e04e700 1 --2- 172.21.15.46:0/3338972682 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f40840a2530 0x7f40840a49f0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:24.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:24.940+0000 7f409e04e700 1 --2- 172.21.15.46:0/3338972682 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4098108270 0x7f40981987b0 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:24.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:24.940+0000 7f409e04e700 1 --2- 172.21.15.46:0/3338972682 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4098102290 0x7f409819e630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:24.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:24.940+0000 7f409e04e700 1 --2- 172.21.15.46:0/3338972682 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4098102c50 0x7f409819eb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:24.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:24.940+0000 7f409e04e700 1 -- 172.21.15.46:0/3338972682 >> 172.21.15.46:0/3338972682 conn(0x7f4098079370 msgr2=0x7f4098101090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:11:24.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:24.941+0000 7f409e04e700 1 -- 172.21.15.46:0/3338972682 shutdown_connections 2024-03-31T23:11:24.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:24.941+0000 7f409e04e700 1 -- 172.21.15.46:0/3338972682 wait complete. 2024-03-31T23:11:24.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:11:24 smithi142 ceph-mon[35271]: pgmap v362: 129 pgs: 129 active+clean; 642 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s; 0 B/s, 0 objects/s recovering 2024-03-31T23:11:25.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:11:24 smithi162 ceph-mon[36237]: pgmap v362: 129 pgs: 129 active+clean; 642 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s; 0 B/s, 0 objects/s recovering 2024-03-31T23:11:25.661 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-31T23:11:25.661 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-31T23:11:25.661 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi046.front.sepia.ceph.com 2024-03-31T23:11:25.662 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-31T23:11:25.662 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-31T23:11:25.662 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-31T23:11:25.662 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-31T23:11:25.662 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-31T23:11:25.662 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-31T23:11:25.662 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T23:11:25.662 DEBUG:teuthology.orchestra.run.smithi046:> ip addr 2024-03-31T23:11:25.678 INFO:teuthology.orchestra.run.smithi046.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-31T23:11:25.679 INFO:teuthology.orchestra.run.smithi046.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-31T23:11:25.679 INFO:teuthology.orchestra.run.smithi046.stdout: inet 127.0.0.1/8 scope host lo 2024-03-31T23:11:25.679 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-03-31T23:11:25.679 INFO:teuthology.orchestra.run.smithi046.stdout: inet6 ::1/128 scope host 2024-03-31T23:11:25.679 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-03-31T23:11:25.679 INFO:teuthology.orchestra.run.smithi046.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-31T23:11:25.679 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 0c:c4:7a:6b:f6:10 brd ff:ff:ff:ff:ff:ff 2024-03-31T23:11:25.679 INFO:teuthology.orchestra.run.smithi046.stdout: altname enp7s0f0 2024-03-31T23:11:25.679 INFO:teuthology.orchestra.run.smithi046.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-31T23:11:25.679 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 0c:c4:7a:6b:f6:11 brd ff:ff:ff:ff:ff:ff 2024-03-31T23:11:25.679 INFO:teuthology.orchestra.run.smithi046.stdout: altname enp7s0f1 2024-03-31T23:11:25.679 INFO:teuthology.orchestra.run.smithi046.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-31T23:11:25.680 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 0c:c4:7a:bd:15:f8 brd ff:ff:ff:ff:ff:ff 2024-03-31T23:11:25.680 INFO:teuthology.orchestra.run.smithi046.stdout: altname enp1s0f0 2024-03-31T23:11:25.680 INFO:teuthology.orchestra.run.smithi046.stdout: inet 172.21.15.46/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-03-31T23:11:25.680 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft 42126sec preferred_lft 42126sec 2024-03-31T23:11:25.680 INFO:teuthology.orchestra.run.smithi046.stdout: inet6 fe80::ec4:7aff:febd:15f8/64 scope link noprefixroute 2024-03-31T23:11:25.680 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-03-31T23:11:25.680 INFO:teuthology.orchestra.run.smithi046.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-31T23:11:25.680 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 0c:c4:7a:bd:15:f9 brd ff:ff:ff:ff:ff:ff 2024-03-31T23:11:25.680 INFO:teuthology.orchestra.run.smithi046.stdout: altname enp1s0f1 2024-03-31T23:11:25.680 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-03-31T23:11:25.681 DEBUG:teuthology.orchestra.run.smithi046:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-31T23:11:25.681 DEBUG:teuthology.orchestra.run.smithi046:> set -e 2024-03-31T23:11:25.681 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip link add name ceph-brx type bridge 2024-03-31T23:11:25.681 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip addr flush dev ceph-brx 2024-03-31T23:11:25.681 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip link set ceph-brx up 2024-03-31T23:11:25.681 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-03-31T23:11:25.681 DEBUG:teuthology.orchestra.run.smithi046:> ') 2024-03-31T23:11:25.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:25 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-31T23:11:25.822 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:11:25 smithi046 ceph-mon[32498]: from='client.? 172.21.15.46:0/3338972682' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-31T23:11:25.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:25 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-31T23:11:25.832 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T23:11:25.833 DEBUG:teuthology.orchestra.run.smithi046:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-31T23:11:25.859 INFO:teuthology.orchestra.run.smithi046.stdout:1 2024-03-31T23:11:25.861 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T23:11:25.861 DEBUG:teuthology.orchestra.run.smithi046:> route 2024-03-31T23:11:25.916 INFO:teuthology.orchestra.run.smithi046.stdout:Kernel IP routing table 2024-03-31T23:11:25.916 INFO:teuthology.orchestra.run.smithi046.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-31T23:11:25.916 INFO:teuthology.orchestra.run.smithi046.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-03-31T23:11:25.916 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-03-31T23:11:25.916 INFO:teuthology.orchestra.run.smithi046.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-31T23:11:25.917 DEBUG:teuthology.orchestra.run.smithi046:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-31T23:11:25.917 DEBUG:teuthology.orchestra.run.smithi046:> set -e 2024-03-31T23:11:25.917 DEBUG:teuthology.orchestra.run.smithi046:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-03-31T23:11:25.917 DEBUG:teuthology.orchestra.run.smithi046:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-03-31T23:11:25.917 DEBUG:teuthology.orchestra.run.smithi046:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-03-31T23:11:25.917 DEBUG:teuthology.orchestra.run.smithi046:> ') 2024-03-31T23:11:25.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:11:25 smithi142 ceph-mon[35271]: from='client.? 172.21.15.46:0/3338972682' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-31T23:11:26.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:26 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-31T23:11:26.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:11:25 smithi162 ceph-mon[36237]: from='client.? 172.21.15.46:0/3338972682' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-31T23:11:26.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:26 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-31T23:11:26.053 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T23:11:26.053 DEBUG:teuthology.orchestra.run.smithi046:> ip netns list 2024-03-31T23:11:26.068 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T23:11:26.068 DEBUG:teuthology.orchestra.run.smithi046:> ip netns list-id 2024-03-31T23:11:26.122 DEBUG:teuthology.orchestra.run.smithi046:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-31T23:11:26.122 DEBUG:teuthology.orchestra.run.smithi046:> set -e 2024-03-31T23:11:26.123 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-31T23:11:26.123 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-03-31T23:11:26.123 DEBUG:teuthology.orchestra.run.smithi046:> ') 2024-03-31T23:11:26.211 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:26 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-31T23:11:26.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:26 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-31T23:11:26.245 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-03-31T23:11:26.246 DEBUG:teuthology.orchestra.run.smithi046:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-31T23:11:26.246 DEBUG:teuthology.orchestra.run.smithi046:> set -e 2024-03-31T23:11:26.246 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-03-31T23:11:26.246 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-03-31T23:11:26.246 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-03-31T23:11:26.246 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-03-31T23:11:26.246 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-03-31T23:11:26.246 DEBUG:teuthology.orchestra.run.smithi046:> ') 2024-03-31T23:11:26.294 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:26 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-31T23:11:26.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:26 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-31T23:11:26.371 DEBUG:teuthology.orchestra.run.smithi046:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-31T23:11:26.371 DEBUG:teuthology.orchestra.run.smithi046:> set -e 2024-03-31T23:11:26.371 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip link set brx.0 up 2024-03-31T23:11:26.371 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip link set dev brx.0 master ceph-brx 2024-03-31T23:11:26.371 DEBUG:teuthology.orchestra.run.smithi046:> ') 2024-03-31T23:11:26.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:26 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-31T23:11:26.447 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:26 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-31T23:11:26.453 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {'mount_subvol_num': 0} 2024-03-31T23:11:26.453 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-31T23:11:26.453 DEBUG:teuthology.orchestra.run.smithi046:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-31T23:11:26.467 INFO:teuthology.orchestra.run.smithi046.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-31T23:11:26.467 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-31T23:11:26.467 DEBUG:teuthology.orchestra.run.smithi046:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-31T23:11:26.522 DEBUG:teuthology.orchestra.run.smithi046:> sudo modprobe fuse 2024-03-31T23:11:26.590 DEBUG:teuthology.orchestra.run.smithi046:> cat /proc/self/mounts | awk '{print $2}' 2024-03-31T23:11:26.646 INFO:teuthology.orchestra.run.smithi046.stdout:/proc 2024-03-31T23:11:26.646 INFO:teuthology.orchestra.run.smithi046.stdout:/sys 2024-03-31T23:11:26.646 INFO:teuthology.orchestra.run.smithi046.stdout:/dev 2024-03-31T23:11:26.646 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/security 2024-03-31T23:11:26.646 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/shm 2024-03-31T23:11:26.646 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/pts 2024-03-31T23:11:26.646 INFO:teuthology.orchestra.run.smithi046.stdout:/run 2024-03-31T23:11:26.646 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/cgroup 2024-03-31T23:11:26.646 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/pstore 2024-03-31T23:11:26.647 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/bpf 2024-03-31T23:11:26.647 INFO:teuthology.orchestra.run.smithi046.stdout:/ 2024-03-31T23:11:26.647 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/selinux 2024-03-31T23:11:26.647 INFO:teuthology.orchestra.run.smithi046.stdout:/proc/sys/fs/binfmt_misc 2024-03-31T23:11:26.647 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/mqueue 2024-03-31T23:11:26.647 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/hugepages 2024-03-31T23:11:26.647 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/debug 2024-03-31T23:11:26.647 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/tracing 2024-03-31T23:11:26.647 INFO:teuthology.orchestra.run.smithi046.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-31T23:11:26.647 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/fuse/connections 2024-03-31T23:11:26.647 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/config 2024-03-31T23:11:26.647 INFO:teuthology.orchestra.run.smithi046.stdout:/run/credentials/systemd-sysctl.service 2024-03-31T23:11:26.647 INFO:teuthology.orchestra.run.smithi046.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-31T23:11:26.647 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/nfs/rpc_pipefs 2024-03-31T23:11:26.648 INFO:teuthology.orchestra.run.smithi046.stdout:/run/user/1000 2024-03-31T23:11:26.648 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph 2024-03-31T23:11:26.648 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/containers/storage/overlay 2024-03-31T23:11:26.648 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/containers/storage/overlay/af6e3dee319c5cd951091dcc321002d5c94992669c333da4f71a8811f5553b4d/merged 2024-03-31T23:11:26.648 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/containers/storage/overlay/6aceac46eafe077575119569412871a784cac219bbecd4a816eb0dd6e31cb766/merged 2024-03-31T23:11:26.648 INFO:teuthology.orchestra.run.smithi046.stdout:/run/user/0 2024-03-31T23:11:26.648 INFO:teuthology.orchestra.run.smithi046.stdout:/proc/sys/fs/binfmt_misc 2024-03-31T23:11:26.648 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/containers/storage/overlay/9732c8b2bef193a8ef899edc437102cc83b87c3c2b9aeede866239f014160834/merged 2024-03-31T23:11:26.648 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/containers/storage/overlay/164dbd5bc6ff0e68c1ce7493f0d76208f294fcb7e4baf61acb26eb069c007959/merged 2024-03-31T23:11:26.648 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/containers/storage/overlay/3ac5e44512b35cb8a3802d7963e396a049e4c945b1c971c39f38030cdbde1024/merged 2024-03-31T23:11:26.648 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/containers/storage/overlay/d90685ef54c2be4027d89ad1c931156b41c2ccfa453582969c652fb1f066e1cd/merged 2024-03-31T23:11:26.648 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/containers/storage/overlay/65432fa8477e0ccf6f4c64e0305f8fa786782d7091f0c38d3a93b23978a7be8f/merged 2024-03-31T23:11:26.648 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/containers/storage/overlay/d41b0d141c8e4fb859226d0b5cb5ff4152c520b9aa66eb4b49dac96f28e032d0/merged 2024-03-31T23:11:26.649 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/containers/storage/overlay/6691bede7b8bc82e883758a6ef749bc93e4df469e71f598744e9970488ce50e9/merged 2024-03-31T23:11:26.649 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/containers/storage/overlay/8e0bc49805369f19f18f5f8d0ac9abfc75e13ffb949d5193d70953f236d7a403/merged 2024-03-31T23:11:26.649 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns 2024-03-31T23:11:26.649 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-31T23:11:26.649 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-31T23:11:26.649 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T23:11:26.649 DEBUG:teuthology.orchestra.run.smithi046:> ls /sys/fs/fuse/connections 2024-03-31T23:11:26.701 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-03-31T23:11:26.701 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0 --client_mountpoint=/volumes/qa/sv_0/e990713a-13d9-4c54-877d-f0f60b19e88e --client_fs=cephfs) 2024-03-31T23:11:26.744 DEBUG:teuthology.orchestra.run.smithi046:> sudo modprobe fuse 2024-03-31T23:11:26.770 DEBUG:teuthology.orchestra.run.smithi046:> cat /proc/self/mounts | awk '{print $2}' 2024-03-31T23:11:26.827 INFO:teuthology.orchestra.run.smithi046.stdout:/proc 2024-03-31T23:11:26.827 INFO:teuthology.orchestra.run.smithi046.stdout:/sys 2024-03-31T23:11:26.827 INFO:teuthology.orchestra.run.smithi046.stdout:/dev 2024-03-31T23:11:26.827 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/security 2024-03-31T23:11:26.827 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/shm 2024-03-31T23:11:26.828 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/pts 2024-03-31T23:11:26.828 INFO:teuthology.orchestra.run.smithi046.stdout:/run 2024-03-31T23:11:26.828 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/cgroup 2024-03-31T23:11:26.828 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/pstore 2024-03-31T23:11:26.828 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/bpf 2024-03-31T23:11:26.828 INFO:teuthology.orchestra.run.smithi046.stdout:/ 2024-03-31T23:11:26.828 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/selinux 2024-03-31T23:11:26.828 INFO:teuthology.orchestra.run.smithi046.stdout:/proc/sys/fs/binfmt_misc 2024-03-31T23:11:26.828 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/mqueue 2024-03-31T23:11:26.828 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/hugepages 2024-03-31T23:11:26.828 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/debug 2024-03-31T23:11:26.828 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/tracing 2024-03-31T23:11:26.828 INFO:teuthology.orchestra.run.smithi046.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-31T23:11:26.829 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/fuse/connections 2024-03-31T23:11:26.829 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/config 2024-03-31T23:11:26.829 INFO:teuthology.orchestra.run.smithi046.stdout:/run/credentials/systemd-sysctl.service 2024-03-31T23:11:26.829 INFO:teuthology.orchestra.run.smithi046.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-31T23:11:26.829 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/nfs/rpc_pipefs 2024-03-31T23:11:26.829 INFO:teuthology.orchestra.run.smithi046.stdout:/run/user/1000 2024-03-31T23:11:26.829 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph 2024-03-31T23:11:26.829 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/containers/storage/overlay 2024-03-31T23:11:26.829 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/containers/storage/overlay/af6e3dee319c5cd951091dcc321002d5c94992669c333da4f71a8811f5553b4d/merged 2024-03-31T23:11:26.829 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/containers/storage/overlay/6aceac46eafe077575119569412871a784cac219bbecd4a816eb0dd6e31cb766/merged 2024-03-31T23:11:26.829 INFO:teuthology.orchestra.run.smithi046.stdout:/run/user/0 2024-03-31T23:11:26.829 INFO:teuthology.orchestra.run.smithi046.stdout:/proc/sys/fs/binfmt_misc 2024-03-31T23:11:26.829 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/containers/storage/overlay/9732c8b2bef193a8ef899edc437102cc83b87c3c2b9aeede866239f014160834/merged 2024-03-31T23:11:26.829 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/containers/storage/overlay/164dbd5bc6ff0e68c1ce7493f0d76208f294fcb7e4baf61acb26eb069c007959/merged 2024-03-31T23:11:26.830 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/containers/storage/overlay/3ac5e44512b35cb8a3802d7963e396a049e4c945b1c971c39f38030cdbde1024/merged 2024-03-31T23:11:26.830 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/containers/storage/overlay/d90685ef54c2be4027d89ad1c931156b41c2ccfa453582969c652fb1f066e1cd/merged 2024-03-31T23:11:26.830 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/containers/storage/overlay/65432fa8477e0ccf6f4c64e0305f8fa786782d7091f0c38d3a93b23978a7be8f/merged 2024-03-31T23:11:26.830 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/containers/storage/overlay/d41b0d141c8e4fb859226d0b5cb5ff4152c520b9aa66eb4b49dac96f28e032d0/merged 2024-03-31T23:11:26.830 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/containers/storage/overlay/6691bede7b8bc82e883758a6ef749bc93e4df469e71f598744e9970488ce50e9/merged 2024-03-31T23:11:26.830 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/containers/storage/overlay/8e0bc49805369f19f18f5f8d0ac9abfc75e13ffb949d5193d70953f236d7a403/merged 2024-03-31T23:11:26.830 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns 2024-03-31T23:11:26.830 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-31T23:11:26.830 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-31T23:11:26.830 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T23:11:26.830 DEBUG:teuthology.orchestra.run.smithi046:> ls /sys/fs/fuse/connections 2024-03-31T23:11:26.831 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi046.stderr:2024-03-31T23:11:26.830+0000 7f24e5c13480 -1 init, newargv = 0x55a710f36c40 newargc=15 2024-03-31T23:11:26.831 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi046.stderr:ceph-fuse[86040]: starting ceph client 2024-03-31T23:11:26.844 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi046.stderr:ceph-fuse[86040]: starting fuse 2024-03-31T23:11:26.847 INFO:teuthology.orchestra.run.smithi046.stdout:119 2024-03-31T23:11:26.847 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [119] 2024-03-31T23:11:26.847 DEBUG:teuthology.orchestra.run.smithi046:> sudo stdin-killer -- python3 -c ' 2024-03-31T23:11:26.847 DEBUG:teuthology.orchestra.run.smithi046:> import glob 2024-03-31T23:11:26.847 DEBUG:teuthology.orchestra.run.smithi046:> import re 2024-03-31T23:11:26.847 DEBUG:teuthology.orchestra.run.smithi046:> import os 2024-03-31T23:11:26.847 DEBUG:teuthology.orchestra.run.smithi046:> import subprocess 2024-03-31T23:11:26.847 DEBUG:teuthology.orchestra.run.smithi046:> 2024-03-31T23:11:26.847 DEBUG:teuthology.orchestra.run.smithi046:> def _find_admin_socket(client_name): 2024-03-31T23:11:26.848 DEBUG:teuthology.orchestra.run.smithi046:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-03-31T23:11:26.848 DEBUG:teuthology.orchestra.run.smithi046:> files = glob.glob(asok_path) 2024-03-31T23:11:26.848 DEBUG:teuthology.orchestra.run.smithi046:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-03-31T23:11:26.848 DEBUG:teuthology.orchestra.run.smithi046:> 2024-03-31T23:11:26.848 DEBUG:teuthology.orchestra.run.smithi046:> # Given a non-glob path, it better be there 2024-03-31T23:11:26.848 DEBUG:teuthology.orchestra.run.smithi046:> if "*" not in asok_path: 2024-03-31T23:11:26.848 DEBUG:teuthology.orchestra.run.smithi046:> assert(len(files) == 1) 2024-03-31T23:11:26.848 DEBUG:teuthology.orchestra.run.smithi046:> return files[0] 2024-03-31T23:11:26.848 DEBUG:teuthology.orchestra.run.smithi046:> 2024-03-31T23:11:26.848 DEBUG:teuthology.orchestra.run.smithi046:> for f in files: 2024-03-31T23:11:26.848 DEBUG:teuthology.orchestra.run.smithi046:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-31T23:11:26.848 DEBUG:teuthology.orchestra.run.smithi046:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-31T23:11:26.848 DEBUG:teuthology.orchestra.run.smithi046:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-31T23:11:26.848 DEBUG:teuthology.orchestra.run.smithi046:> contents = proc_f.read() 2024-03-31T23:11:26.848 DEBUG:teuthology.orchestra.run.smithi046:> if mountpoint in contents: 2024-03-31T23:11:26.849 DEBUG:teuthology.orchestra.run.smithi046:> return f 2024-03-31T23:11:26.849 DEBUG:teuthology.orchestra.run.smithi046:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-31T23:11:26.849 DEBUG:teuthology.orchestra.run.smithi046:> 2024-03-31T23:11:26.849 DEBUG:teuthology.orchestra.run.smithi046:> print(_find_admin_socket("client.0")) 2024-03-31T23:11:26.849 DEBUG:teuthology.orchestra.run.smithi046:> ' 2024-03-31T23:11:26.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:11:26 smithi046 ceph-mon[32498]: pgmap v363: 129 pgs: 129 active+clean; 642 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s; 0 B/s, 0 objects/s recovering 2024-03-31T23:11:26.941 INFO:teuthology.orchestra.run.smithi046.stdout:/var/run/ceph/ceph-client.0.86040.asok 2024-03-31T23:11:26.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:11:26 smithi142 ceph-mon[35271]: pgmap v363: 129 pgs: 129 active+clean; 642 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s; 0 B/s, 0 objects/s recovering 2024-03-31T23:11:26.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:26 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-31T23:11:26.951 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.86040.asok 2024-03-31T23:11:26.952 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T23:11:26.952 DEBUG:teuthology.orchestra.run.smithi046:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.86040.asok status 2024-03-31T23:11:27.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:11:26 smithi162 ceph-mon[36237]: pgmap v363: 129 pgs: 129 active+clean; 642 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s; 0 B/s, 0 objects/s recovering 2024-03-31T23:11:27.089 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-03-31T23:11:27.089 INFO:teuthology.orchestra.run.smithi046.stdout: "metadata": { 2024-03-31T23:11:27.089 INFO:teuthology.orchestra.run.smithi046.stdout: "ceph_sha1": "384f31f3d3aee7f8ba3f60846b8844c049165583", 2024-03-31T23:11:27.089 INFO:teuthology.orchestra.run.smithi046.stdout: "ceph_version": "ceph version 19.0.0-1977-g384f31f3 (384f31f3d3aee7f8ba3f60846b8844c049165583) squid (dev)", 2024-03-31T23:11:27.090 INFO:teuthology.orchestra.run.smithi046.stdout: "entity_id": "0", 2024-03-31T23:11:27.090 INFO:teuthology.orchestra.run.smithi046.stdout: "hostname": "smithi046", 2024-03-31T23:11:27.090 INFO:teuthology.orchestra.run.smithi046.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-03-31T23:11:27.090 INFO:teuthology.orchestra.run.smithi046.stdout: "pid": "86040", 2024-03-31T23:11:27.090 INFO:teuthology.orchestra.run.smithi046.stdout: "root": "/volumes/qa/sv_0/e990713a-13d9-4c54-877d-f0f60b19e88e" 2024-03-31T23:11:27.090 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:11:27.090 INFO:teuthology.orchestra.run.smithi046.stdout: "dentry_count": 0, 2024-03-31T23:11:27.090 INFO:teuthology.orchestra.run.smithi046.stdout: "dentry_pinned_count": 0, 2024-03-31T23:11:27.090 INFO:teuthology.orchestra.run.smithi046.stdout: "id": 15462, 2024-03-31T23:11:27.090 INFO:teuthology.orchestra.run.smithi046.stdout: "inst": { 2024-03-31T23:11:27.090 INFO:teuthology.orchestra.run.smithi046.stdout: "name": { 2024-03-31T23:11:27.090 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "client", 2024-03-31T23:11:27.090 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 15462 2024-03-31T23:11:27.091 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:11:27.091 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": { 2024-03-31T23:11:27.091 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-03-31T23:11:27.091 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "192.168.0.1:0", 2024-03-31T23:11:27.091 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 2113954864 2024-03-31T23:11:27.091 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-31T23:11:27.091 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:11:27.091 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": { 2024-03-31T23:11:27.091 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-03-31T23:11:27.091 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "192.168.0.1:0", 2024-03-31T23:11:27.091 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 2113954864 2024-03-31T23:11:27.091 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:11:27.092 INFO:teuthology.orchestra.run.smithi046.stdout: "inst_str": "client.15462 192.168.0.1:0/2113954864", 2024-03-31T23:11:27.092 INFO:teuthology.orchestra.run.smithi046.stdout: "addr_str": "192.168.0.1:0/2113954864", 2024-03-31T23:11:27.092 INFO:teuthology.orchestra.run.smithi046.stdout: "inode_count": 5, 2024-03-31T23:11:27.092 INFO:teuthology.orchestra.run.smithi046.stdout: "mds_epoch": 24, 2024-03-31T23:11:27.092 INFO:teuthology.orchestra.run.smithi046.stdout: "osd_epoch": 95, 2024-03-31T23:11:27.092 INFO:teuthology.orchestra.run.smithi046.stdout: "osd_epoch_barrier": 0, 2024-03-31T23:11:27.092 INFO:teuthology.orchestra.run.smithi046.stdout: "blocklisted": false, 2024-03-31T23:11:27.092 INFO:teuthology.orchestra.run.smithi046.stdout: "fs_name": "cephfs" 2024-03-31T23:11:27.092 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-03-31T23:11:27.098 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T23:11:27.098 DEBUG:teuthology.orchestra.run.smithi046:> stat --file-system '--printf=%T 2024-03-31T23:11:27.098 DEBUG:teuthology.orchestra.run.smithi046:> ' -- /home/ubuntu/cephtest/mnt.0 2024-03-31T23:11:27.113 INFO:teuthology.orchestra.run.smithi046.stdout:fuseblk 2024-03-31T23:11:27.113 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-03-31T23:11:27.113 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T23:11:27.114 DEBUG:teuthology.orchestra.run.smithi046:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-31T23:11:27.183 INFO:teuthology.run_tasks:Running task exec... 2024-03-31T23:11:27.193 INFO:teuthology.task.exec:Executing custom commands... 2024-03-31T23:11:27.193 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi046.front.sepia.ceph.com 2024-03-31T23:11:27.193 DEBUG:teuthology.orchestra.run.smithi046:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs set cephfs balance_automate true' 2024-03-31T23:11:27.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:27.326+0000 7efc0970d640 1 -- 172.21.15.46:0/2905792084 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efc04102240 msgr2=0x7efc04108ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:27.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:27.326+0000 7efc0970d640 1 --2- 172.21.15.46:0/2905792084 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efc04102240 0x7efc04108ad0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7efbf4009ca0 tx=0x7efbf402c0f0 comp rx=0 tx=0).stop 2024-03-31T23:11:27.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:27.326+0000 7efc0970d640 1 -- 172.21.15.46:0/2905792084 shutdown_connections 2024-03-31T23:11:27.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:27.326+0000 7efc0970d640 1 --2- 172.21.15.46:0/2905792084 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efc04102240 0x7efc04108ad0 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:27.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:27.326+0000 7efc0970d640 1 --2- 172.21.15.46:0/2905792084 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7efc04101880 0x7efc04101d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:27.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:27.326+0000 7efc0970d640 1 --2- 172.21.15.46:0/2905792084 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7efc04100690 0x7efc04100ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:27.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:27.326+0000 7efc0970d640 1 -- 172.21.15.46:0/2905792084 >> 172.21.15.46:0/2905792084 conn(0x7efc040fbc00 msgr2=0x7efc040fe040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:11:27.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:27.326+0000 7efc0970d640 1 -- 172.21.15.46:0/2905792084 shutdown_connections 2024-03-31T23:11:27.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:27.326+0000 7efc0970d640 1 -- 172.21.15.46:0/2905792084 wait complete. 2024-03-31T23:11:27.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:27.326+0000 7efc0970d640 1 Processor -- start 2024-03-31T23:11:27.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:27.326+0000 7efc0970d640 1 -- start start 2024-03-31T23:11:27.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:27.327+0000 7efc0970d640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7efc04100690 0x7efc041967a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:27.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:27.327+0000 7efc0970d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efc04101880 0x7efc04196ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:27.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:27.327+0000 7efc0970d640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7efc04102240 0x7efc0419dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:27.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:27.327+0000 7efc0970d640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7efc0410c1c0 con 0x7efc04101880 2024-03-31T23:11:27.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:27.327+0000 7efc0970d640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7efc0410c040 con 0x7efc04102240 2024-03-31T23:11:27.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:27.327+0000 7efc0970d640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7efc0410c340 con 0x7efc04100690 2024-03-31T23:11:27.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:27.327+0000 7efc027fc640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efc04101880 0x7efc04196ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:27.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:27.327+0000 7efc027fc640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efc04101880 0x7efc04196ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:58330/0 (socket says 172.21.15.46:58330) 2024-03-31T23:11:27.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:27.327+0000 7efc027fc640 1 -- 172.21.15.46:0/754596586 learned_addr learned my addr 172.21.15.46:0/754596586 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:11:27.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:27.327+0000 7efc02ffd640 1 --2- 172.21.15.46:0/754596586 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7efc04100690 0x7efc041967a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:27.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:27.327+0000 7efc037fe640 1 --2- 172.21.15.46:0/754596586 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7efc04102240 0x7efc0419dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:27.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:27.327+0000 7efc027fc640 1 -- 172.21.15.46:0/754596586 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7efc04100690 msgr2=0x7efc041967a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:27.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:27.327+0000 7efc027fc640 1 --2- 172.21.15.46:0/754596586 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7efc04100690 0x7efc041967a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:27.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:27.327+0000 7efc027fc640 1 -- 172.21.15.46:0/754596586 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7efc04102240 msgr2=0x7efc0419dd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:27.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:27.327+0000 7efc027fc640 1 --2- 172.21.15.46:0/754596586 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7efc04102240 0x7efc0419dd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:27.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:27.327+0000 7efc027fc640 1 -- 172.21.15.46:0/754596586 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efc0419e3c0 con 0x7efc04101880 2024-03-31T23:11:27.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:27.327+0000 7efc027fc640 1 --2- 172.21.15.46:0/754596586 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efc04101880 0x7efc04196ce0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7efbec00eb00 tx=0x7efbec009f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:11:27.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:27.327+0000 7efbfbfff640 1 -- 172.21.15.46:0/754596586 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efbec01c070 con 0x7efc04101880 2024-03-31T23:11:27.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:27.327+0000 7efbfbfff640 1 -- 172.21.15.46:0/754596586 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7efbec00bc30 con 0x7efc04101880 2024-03-31T23:11:27.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:27.327+0000 7efc0970d640 1 -- 172.21.15.46:0/754596586 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efc04069fb0 con 0x7efc04101880 2024-03-31T23:11:27.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:27.327+0000 7efbfbfff640 1 -- 172.21.15.46:0/754596586 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efbec016610 con 0x7efc04101880 2024-03-31T23:11:27.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:27.327+0000 7efc0970d640 1 -- 172.21.15.46:0/754596586 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efc0406a550 con 0x7efc04101880 2024-03-31T23:11:27.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:27.329+0000 7efc0970d640 1 -- 172.21.15.46:0/754596586 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efc04107420 con 0x7efc04101880 2024-03-31T23:11:27.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:27.329+0000 7efbfbfff640 1 -- 172.21.15.46:0/754596586 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 18) ==== 137320+0+0 (secure 0 0 0) 0x7efbec01f070 con 0x7efc04101880 2024-03-31T23:11:27.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:27.330+0000 7efbfbfff640 1 --2- 172.21.15.46:0/754596586 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7efbf00a25c0 0x7efbf00a4a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:27.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:27.330+0000 7efc02ffd640 1 --2- 172.21.15.46:0/754596586 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7efbf00a25c0 0x7efbf00a4a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:27.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:27.330+0000 7efbfbfff640 1 -- 172.21.15.46:0/754596586 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9208+0+0 (secure 0 0 0) 0x7efbec0d00a0 con 0x7efc04101880 2024-03-31T23:11:27.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:27.330+0000 7efc02ffd640 1 --2- 172.21.15.46:0/754596586 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7efbf00a25c0 0x7efbf00a4a80 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7efc041016e0 tx=0x7efbfc005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:11:27.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:27.334+0000 7efbfbfff640 1 -- 172.21.15.46:0/754596586 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7efbec00bde0 con 0x7efc04101880 2024-03-31T23:11:27.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:27.558+0000 7efc0970d640 1 -- 172.21.15.46:0/754596586 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"} v 0) -- 0x7efc0419e590 con 0x7efc04101880 2024-03-31T23:11:27.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:27.576+0000 7efbfbfff640 1 -- 172.21.15.46:0/754596586 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]=0 v25) ==== 117+0+0 (secure 0 0 0) 0x7efbec09c7e0 con 0x7efc04101880 2024-03-31T23:11:27.578 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:27.578+0000 7efc0970d640 1 -- 172.21.15.46:0/754596586 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7efbf00a25c0 msgr2=0x7efbf00a4a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:27.578 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:27.578+0000 7efc0970d640 1 --2- 172.21.15.46:0/754596586 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7efbf00a25c0 0x7efbf00a4a80 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7efc041016e0 tx=0x7efbfc005f90 comp rx=0 tx=0).stop 2024-03-31T23:11:27.578 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:27.578+0000 7efc0970d640 1 -- 172.21.15.46:0/754596586 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efc04101880 msgr2=0x7efc04196ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:27.578 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:27.578+0000 7efc0970d640 1 --2- 172.21.15.46:0/754596586 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efc04101880 0x7efc04196ce0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7efbec00eb00 tx=0x7efbec009f20 comp rx=0 tx=0).stop 2024-03-31T23:11:27.578 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:27.578+0000 7efc0970d640 1 -- 172.21.15.46:0/754596586 shutdown_connections 2024-03-31T23:11:27.578 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:27.578+0000 7efc0970d640 1 --2- 172.21.15.46:0/754596586 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7efbf00a25c0 0x7efbf00a4a80 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:27.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:27.578+0000 7efc0970d640 1 --2- 172.21.15.46:0/754596586 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7efc04102240 0x7efc0419dd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:27.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:27.578+0000 7efc0970d640 1 --2- 172.21.15.46:0/754596586 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efc04101880 0x7efc04196ce0 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:27.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:27.578+0000 7efc0970d640 1 --2- 172.21.15.46:0/754596586 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7efc04100690 0x7efc041967a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:27.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:27.578+0000 7efc0970d640 1 -- 172.21.15.46:0/754596586 >> 172.21.15.46:0/754596586 conn(0x7efc040fbc00 msgr2=0x7efc04104b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:11:27.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:27.578+0000 7efc0970d640 1 -- 172.21.15.46:0/754596586 shutdown_connections 2024-03-31T23:11:27.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:27.578+0000 7efc0970d640 1 -- 172.21.15.46:0/754596586 wait complete. 2024-03-31T23:11:27.589 INFO:teuthology.run_tasks:Running task check-counter... 2024-03-31T23:11:27.599 INFO:teuthology.run_tasks:Running task fwd_scrub... 2024-03-31T23:11:27.614 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-31T23:11:27.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:27.718+0000 7f6500a7c640 1 -- 172.21.15.46:0/953298685 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f64fc100690 msgr2=0x7f64fc100ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:27.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:27.718+0000 7f6500a7c640 1 --2- 172.21.15.46:0/953298685 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f64fc100690 0x7f64fc100ab0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f64ec009900 tx=0x7f64ec02bc30 comp rx=0 tx=0).stop 2024-03-31T23:11:27.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:27.718+0000 7f6500a7c640 1 -- 172.21.15.46:0/953298685 shutdown_connections 2024-03-31T23:11:27.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:27.718+0000 7f6500a7c640 1 --2- 172.21.15.46:0/953298685 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f64fc102240 0x7f64fc108ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:27.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:27.718+0000 7f6500a7c640 1 --2- 172.21.15.46:0/953298685 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f64fc101880 0x7f64fc101d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:27.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:27.718+0000 7f6500a7c640 1 --2- 172.21.15.46:0/953298685 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f64fc100690 0x7f64fc100ab0 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:27.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:27.718+0000 7f6500a7c640 1 -- 172.21.15.46:0/953298685 >> 172.21.15.46:0/953298685 conn(0x7f64fc0fbc00 msgr2=0x7f64fc0fe040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:11:27.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:27.718+0000 7f6500a7c640 1 -- 172.21.15.46:0/953298685 shutdown_connections 2024-03-31T23:11:27.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:27.718+0000 7f6500a7c640 1 -- 172.21.15.46:0/953298685 wait complete. 2024-03-31T23:11:27.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:27.718+0000 7f6500a7c640 1 Processor -- start 2024-03-31T23:11:27.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:27.719+0000 7f6500a7c640 1 -- start start 2024-03-31T23:11:27.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:27.719+0000 7f6500a7c640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f64fc100690 0x7f64fc1968c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:27.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:27.719+0000 7f6500a7c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f64fc101880 0x7f64fc196e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:27.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:27.719+0000 7f6500a7c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f64fc102240 0x7f64fc19de70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:27.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:27.719+0000 7f6500a7c640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f64fc10c1c0 con 0x7f64fc102240 2024-03-31T23:11:27.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:27.719+0000 7f6500a7c640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f64fc10c040 con 0x7f64fc100690 2024-03-31T23:11:27.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:27.719+0000 7f6500a7c640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f64fc10c340 con 0x7f64fc101880 2024-03-31T23:11:27.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:27.719+0000 7f64fad9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f64fc102240 0x7f64fc19de70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:27.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:27.719+0000 7f64fad9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f64fc102240 0x7f64fc19de70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:58350/0 (socket says 172.21.15.46:58350) 2024-03-31T23:11:27.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:27.719+0000 7f64fad9d640 1 -- 172.21.15.46:0/2957863234 learned_addr learned my addr 172.21.15.46:0/2957863234 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:11:27.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:27.719+0000 7f64fa59c640 1 --2- 172.21.15.46:0/2957863234 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f64fc100690 0x7f64fc1968c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:27.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:27.719+0000 7f64f9d9b640 1 --2- 172.21.15.46:0/2957863234 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f64fc101880 0x7f64fc196e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:27.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:27.719+0000 7f64fad9d640 1 -- 172.21.15.46:0/2957863234 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f64fc101880 msgr2=0x7f64fc196e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:27.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:27.719+0000 7f64fad9d640 1 --2- 172.21.15.46:0/2957863234 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f64fc101880 0x7f64fc196e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:27.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:27.719+0000 7f64fad9d640 1 -- 172.21.15.46:0/2957863234 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f64fc100690 msgr2=0x7f64fc1968c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:27.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:27.719+0000 7f64fad9d640 1 --2- 172.21.15.46:0/2957863234 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f64fc100690 0x7f64fc1968c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:27.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:27.719+0000 7f64fad9d640 1 -- 172.21.15.46:0/2957863234 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f64fc19e4e0 con 0x7f64fc102240 2024-03-31T23:11:27.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:27.719+0000 7f64fa59c640 1 --2- 172.21.15.46:0/2957863234 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f64fc100690 0x7f64fc1968c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:11:27.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:27.719+0000 7f64fad9d640 1 --2- 172.21.15.46:0/2957863234 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f64fc102240 0x7f64fc19de70 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f64f400eb10 tx=0x7f64f400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:11:27.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:27.719+0000 7f64eb7fe640 1 -- 172.21.15.46:0/2957863234 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64f4004b20 con 0x7f64fc102240 2024-03-31T23:11:27.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:27.719+0000 7f64eb7fe640 1 -- 172.21.15.46:0/2957863234 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f64f4004cc0 con 0x7f64fc102240 2024-03-31T23:11:27.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:27.720+0000 7f6500a7c640 1 -- 172.21.15.46:0/2957863234 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f64fc19e7d0 con 0x7f64fc102240 2024-03-31T23:11:27.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:27.720+0000 7f64eb7fe640 1 -- 172.21.15.46:0/2957863234 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64f4010690 con 0x7f64fc102240 2024-03-31T23:11:27.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:27.720+0000 7f6500a7c640 1 -- 172.21.15.46:0/2957863234 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f64fc19ed20 con 0x7f64fc102240 2024-03-31T23:11:27.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:27.720+0000 7f6500a7c640 1 -- 172.21.15.46:0/2957863234 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f64fc107a00 con 0x7f64fc102240 2024-03-31T23:11:27.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:27.721+0000 7f64eb7fe640 1 -- 172.21.15.46:0/2957863234 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 18) ==== 137320+0+0 (secure 0 0 0) 0x7f64f4005020 con 0x7f64fc102240 2024-03-31T23:11:27.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:27.722+0000 7f64eb7fe640 1 --2- 172.21.15.46:0/2957863234 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f64e40a24a0 0x7f64e40a4960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:27.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:27.722+0000 7f64fa59c640 1 --2- 172.21.15.46:0/2957863234 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f64e40a24a0 0x7f64e40a4960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:27.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:27.722+0000 7f64eb7fe640 1 -- 172.21.15.46:0/2957863234 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9208+0+0 (secure 0 0 0) 0x7f64f40cf240 con 0x7f64fc102240 2024-03-31T23:11:27.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:27.723+0000 7f64fa59c640 1 --2- 172.21.15.46:0/2957863234 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f64e40a24a0 0x7f64e40a4960 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f64ec005ab0 tx=0x7f64ec005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:11:27.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:27.727+0000 7f64eb7fe640 1 -- 172.21.15.46:0/2957863234 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f64f409bfd0 con 0x7f64fc102240 2024-03-31T23:11:27.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:11:27 smithi046 ceph-mon[32498]: from='client.? 172.21.15.46:0/754596586' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]: dispatch 2024-03-31T23:11:27.911 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:27.911+0000 7f6500a7c640 1 -- 172.21.15.46:0/2957863234 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f64fc062350 con 0x7f64fc102240 2024-03-31T23:11:27.913 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:27.913+0000 7f64eb7fe640 1 -- 172.21.15.46:0/2957863234 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v95) ==== 74+0+24819 (secure 0 0 0) 0x7f64f409b8d0 con 0x7f64fc102240 2024-03-31T23:11:27.913 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-31T23:11:27.913 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":95,"fsid":"6a9d8f08-efb2-11ee-b647-cb9ed24678a4","created":"2024-03-31T23:01:15.463711+0000","modified":"2024-03-31T23:11:10.531495+0000","last_up_change":"2024-03-31T23:08:20.199888+0000","last_in_change":"2024-03-31T23:07:59.438035+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-31T23:04:32.823751+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-31T23:09:09.145711+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-31T23:09:10.148579+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-31T23:09:13.419519+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"91","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"47429095-3f46-4ed7-b7c4-504464719815","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":1824772695},{"type":"v1","addr":"172.21.15.46:6803","nonce":1824772695}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":1824772695},{"type":"v1","addr":"172.21.15.46:6805","nonce":1824772695}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":1824772695},{"type":"v1","addr":"172.21.15.46:6809","nonce":1824772695}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":1824772695},{"type":"v1","addr":"172.21.15.46:6807","nonce":1824772695}]},"public_addr":"172.21.15.46:6803/1824772695","cluster_addr":"172.21.15.46:6805/1824772695","heartbeat_back_addr":"172.21.15.46:6809/1824772695","heartbeat_front_addr":"172.21.15.46:6807/1824772695","state":["exists","up"]},{"osd":1,"uuid":"be56f883-411e-4dbe-b50e-a158134df4a6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6800","nonce":973693922},{"type":"v1","addr":"172.21.15.142:6801","nonce":973693922}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6802","nonce":973693922},{"type":"v1","addr":"172.21.15.142:6803","nonce":973693922}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6806","nonce":973693922},{"type":"v1","addr":"172.21.15.142:6807","nonce":973693922}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6804","nonce":973693922},{"type":"v1","addr":"172.21.15.142:6805","nonce":973693922}]},"public_addr":"172.21.15.142:6801/973693922","cluster_addr":"172.21.15.142:6803/973693922","heartbeat_back_addr":"172.21.15.142:6807/973693922","heartbeat_front_addr":"172.21.15.142:6805/973693922","state":["exists","up"]},{"osd":2,"uuid":"c99a8794-ab19-4260-9a8d-4bc0bd2c8175","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6800","nonce":4084261813},{"type":"v1","addr":"172.21.15.162:6801","nonce":4084261813}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6802","nonce":4084261813},{"type":"v1","addr":"172.21.15.162:6803","nonce":4084261813}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6806","nonce":4084261813},{"type":"v1","addr":"172.21.15.162:6807","nonce":4084261813}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6804","nonce":4084261813},{"type":"v1","addr":"172.21.15.162:6805","nonce":4084261813}]},"public_addr":"172.21.15.162:6801/4084261813","cluster_addr":"172.21.15.162:6803/4084261813","heartbeat_back_addr":"172.21.15.162:6807/4084261813","heartbeat_front_addr":"172.21.15.162:6805/4084261813","state":["exists","up"]},{"osd":3,"uuid":"f877814e-9ef6-4872-a081-5609bb9ebc06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":3496317970},{"type":"v1","addr":"172.21.15.46:6811","nonce":3496317970}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":3496317970},{"type":"v1","addr":"172.21.15.46:6813","nonce":3496317970}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":3496317970},{"type":"v1","addr":"172.21.15.46:6817","nonce":3496317970}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":3496317970},{"type":"v1","addr":"172.21.15.46:6815","nonce":3496317970}]},"public_addr":"172.21.15.46:6811/3496317970","cluster_addr":"172.21.15.46:6813/3496317970","heartbeat_back_addr":"172.21.15.46:6817/3496317970","heartbeat_front_addr":"172.21.15.46:6815/3496317970","state":["exists","up"]},{"osd":4,"uuid":"beecec0a-8280-44fe-8f5f-8498a35eca8f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6808","nonce":2996266929},{"type":"v1","addr":"172.21.15.142:6809","nonce":2996266929}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6810","nonce":2996266929},{"type":"v1","addr":"172.21.15.142:6811","nonce":2996266929}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6814","nonce":2996266929},{"type":"v1","addr":"172.21.15.142:6815","nonce":2996266929}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6812","nonce":2996266929},{"type":"v1","addr":"172.21.15.142:6813","nonce":2996266929}]},"public_addr":"172.21.15.142:6809/2996266929","cluster_addr":"172.21.15.142:6811/2996266929","heartbeat_back_addr":"172.21.15.142:6815/2996266929","heartbeat_front_addr":"172.21.15.142:6813/2996266929","state":["exists","up"]},{"osd":5,"uuid":"9da7ff75-f658-45ea-a02f-1943fdda76b9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6808","nonce":914208293},{"type":"v1","addr":"172.21.15.162:6809","nonce":914208293}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6810","nonce":914208293},{"type":"v1","addr":"172.21.15.162:6811","nonce":914208293}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6814","nonce":914208293},{"type":"v1","addr":"172.21.15.162:6815","nonce":914208293}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6812","nonce":914208293},{"type":"v1","addr":"172.21.15.162:6813","nonce":914208293}]},"public_addr":"172.21.15.162:6809/914208293","cluster_addr":"172.21.15.162:6811/914208293","heartbeat_back_addr":"172.21.15.162:6815/914208293","heartbeat_front_addr":"172.21.15.162:6813/914208293","state":["exists","up"]},{"osd":6,"uuid":"8fa11224-cb26-4d3f-87d1-6a8f31de2a19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":56291966},{"type":"v1","addr":"172.21.15.46:6819","nonce":56291966}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":56291966},{"type":"v1","addr":"172.21.15.46:6821","nonce":56291966}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6824","nonce":56291966},{"type":"v1","addr":"172.21.15.46:6825","nonce":56291966}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":56291966},{"type":"v1","addr":"172.21.15.46:6823","nonce":56291966}]},"public_addr":"172.21.15.46:6819/56291966","cluster_addr":"172.21.15.46:6821/56291966","heartbeat_back_addr":"172.21.15.46:6825/56291966","heartbeat_front_addr":"172.21.15.46:6823/56291966","state":["exists","up"]},{"osd":7,"uuid":"0d0338b7-4d0a-4bb0-b152-8f5224871d9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6816","nonce":2146598446},{"type":"v1","addr":"172.21.15.142:6817","nonce":2146598446}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6818","nonce":2146598446},{"type":"v1","addr":"172.21.15.142:6819","nonce":2146598446}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6822","nonce":2146598446},{"type":"v1","addr":"172.21.15.142:6823","nonce":2146598446}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6820","nonce":2146598446},{"type":"v1","addr":"172.21.15.142:6821","nonce":2146598446}]},"public_addr":"172.21.15.142:6817/2146598446","cluster_addr":"172.21.15.142:6819/2146598446","heartbeat_back_addr":"172.21.15.142:6823/2146598446","heartbeat_front_addr":"172.21.15.142:6821/2146598446","state":["exists","up"]},{"osd":8,"uuid":"8b6f6037-892a-416c-babb-7552648d0b3a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6816","nonce":1173078318},{"type":"v1","addr":"172.21.15.162:6817","nonce":1173078318}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6818","nonce":1173078318},{"type":"v1","addr":"172.21.15.162:6819","nonce":1173078318}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6822","nonce":1173078318},{"type":"v1","addr":"172.21.15.162:6823","nonce":1173078318}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6820","nonce":1173078318},{"type":"v1","addr":"172.21.15.162:6821","nonce":1173078318}]},"public_addr":"172.21.15.162:6817/1173078318","cluster_addr":"172.21.15.162:6819/1173078318","heartbeat_back_addr":"172.21.15.162:6823/1173078318","heartbeat_front_addr":"172.21.15.162:6821/1173078318","state":["exists","up"]},{"osd":9,"uuid":"faccce25-6bf2-4220-8858-cda82b72be68","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":532708881},{"type":"v1","addr":"172.21.15.46:6827","nonce":532708881}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6828","nonce":532708881},{"type":"v1","addr":"172.21.15.46:6829","nonce":532708881}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":532708881},{"type":"v1","addr":"172.21.15.46:6833","nonce":532708881}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":532708881},{"type":"v1","addr":"172.21.15.46:6831","nonce":532708881}]},"public_addr":"172.21.15.46:6827/532708881","cluster_addr":"172.21.15.46:6829/532708881","heartbeat_back_addr":"172.21.15.46:6833/532708881","heartbeat_front_addr":"172.21.15.46:6831/532708881","state":["exists","up"]},{"osd":10,"uuid":"cdd356db-f9eb-4080-95ab-db5d8223e572","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6824","nonce":2845222366},{"type":"v1","addr":"172.21.15.142:6825","nonce":2845222366}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6826","nonce":2845222366},{"type":"v1","addr":"172.21.15.142:6827","nonce":2845222366}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6830","nonce":2845222366},{"type":"v1","addr":"172.21.15.142:6831","nonce":2845222366}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6828","nonce":2845222366},{"type":"v1","addr":"172.21.15.142:6829","nonce":2845222366}]},"public_addr":"172.21.15.142:6825/2845222366","cluster_addr":"172.21.15.142:6827/2845222366","heartbeat_back_addr":"172.21.15.142:6831/2845222366","heartbeat_front_addr":"172.21.15.142:6829/2845222366","state":["exists","up"]},{"osd":11,"uuid":"72104d1c-3ea2-45ad-bc0b-0500f146c06b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6824","nonce":1986323778},{"type":"v1","addr":"172.21.15.162:6825","nonce":1986323778}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6826","nonce":1986323778},{"type":"v1","addr":"172.21.15.162:6827","nonce":1986323778}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6830","nonce":1986323778},{"type":"v1","addr":"172.21.15.162:6831","nonce":1986323778}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6828","nonce":1986323778},{"type":"v1","addr":"172.21.15.162:6829","nonce":1986323778}]},"public_addr":"172.21.15.162:6825/1986323778","cluster_addr":"172.21.15.162:6827/1986323778","heartbeat_back_addr":"172.21.15.162:6831/1986323778","heartbeat_front_addr":"172.21.15.162:6829/1986323778","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:03:44.996331+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:04:07.994738+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:04:30.942255+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:04:55.588884+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:05:19.900600+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:05:43.807442+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:06:09.026976+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:06:34.586784+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:06:59.751605+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:07:26.144495+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:07:52.326909+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:08:17.423042+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.b","mappings":[{"from":8,"to":11}]},{"pgid":"3.6","mappings":[{"from":4,"to":7}]},{"pgid":"3.c","mappings":[{"from":3,"to":9}]},{"pgid":"3.1a","mappings":[{"from":5,"to":6}]},{"pgid":"3.1b","mappings":[{"from":4,"to":8}]},{"pgid":"3.1d","mappings":[{"from":4,"to":8}]},{"pgid":"3.2a","mappings":[{"from":3,"to":6}]},{"pgid":"3.33","mappings":[{"from":5,"to":9}]},{"pgid":"3.36","mappings":[{"from":4,"to":9}]},{"pgid":"4.6","mappings":[{"from":5,"to":8}]},{"pgid":"4.17","mappings":[{"from":9,"to":8}]},{"pgid":"4.18","mappings":[{"from":9,"to":8}]},{"pgid":"4.1b","mappings":[{"from":9,"to":10}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.46:0/4021039857":"2024-04-01T23:02:08.493619+0000","172.21.15.46:6801/3681182647":"2024-04-01T23:02:08.493619+0000","172.21.15.46:0/4022214806":"2024-04-01T23:02:08.493619+0000","172.21.15.46:6800/1515528215":"2024-04-01T23:01:41.910162+0000","172.21.15.46:6800/3681182647":"2024-04-01T23:02:08.493619+0000","172.21.15.46:0/143885707":"2024-04-01T23:01:41.910162+0000","172.21.15.46:6801/1515528215":"2024-04-01T23:01:41.910162+0000","172.21.15.46:0/1680389778":"2024-04-01T23:01:41.910162+0000","172.21.15.46:0/1782133966":"2024-04-01T23:01:41.910162+0000","172.21.15.46:0/3925209376":"2024-04-01T23:02:08.493619+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-31T23:11:27.915 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:27.915+0000 7f6500a7c640 1 -- 172.21.15.46:0/2957863234 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f64e40a24a0 msgr2=0x7f64e40a4960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:27.915 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:27.915+0000 7f6500a7c640 1 --2- 172.21.15.46:0/2957863234 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f64e40a24a0 0x7f64e40a4960 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f64ec005ab0 tx=0x7f64ec005fd0 comp rx=0 tx=0).stop 2024-03-31T23:11:27.915 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:27.915+0000 7f6500a7c640 1 -- 172.21.15.46:0/2957863234 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f64fc102240 msgr2=0x7f64fc19de70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:27.915 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:27.915+0000 7f6500a7c640 1 --2- 172.21.15.46:0/2957863234 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f64fc102240 0x7f64fc19de70 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f64f400eb10 tx=0x7f64f400c2d0 comp rx=0 tx=0).stop 2024-03-31T23:11:27.916 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:27.915+0000 7f6500a7c640 1 -- 172.21.15.46:0/2957863234 shutdown_connections 2024-03-31T23:11:27.916 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:27.915+0000 7f6500a7c640 1 --2- 172.21.15.46:0/2957863234 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f64e40a24a0 0x7f64e40a4960 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:27.916 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:27.915+0000 7f6500a7c640 1 --2- 172.21.15.46:0/2957863234 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f64fc102240 0x7f64fc19de70 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:27.916 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:27.915+0000 7f6500a7c640 1 --2- 172.21.15.46:0/2957863234 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f64fc101880 0x7f64fc196e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:27.916 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:27.915+0000 7f6500a7c640 1 --2- 172.21.15.46:0/2957863234 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f64fc100690 0x7f64fc1968c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:27.916 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:27.915+0000 7f6500a7c640 1 -- 172.21.15.46:0/2957863234 >> 172.21.15.46:0/2957863234 conn(0x7f64fc0fbc00 msgr2=0x7f64fc104bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:11:27.916 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:27.915+0000 7f6500a7c640 1 -- 172.21.15.46:0/2957863234 shutdown_connections 2024-03-31T23:11:27.916 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:27.915+0000 7f6500a7c640 1 -- 172.21.15.46:0/2957863234 wait complete. 2024-03-31T23:11:27.926 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-31T23:04:32.823751+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-31T23:09:09.145711+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-31T23:09:10.148579+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-31T23:09:13.419519+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '91', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-31T23:11:27.927 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-31T23:11:27.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:11:27 smithi142 ceph-mon[35271]: from='client.? 172.21.15.46:0/754596586' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]: dispatch 2024-03-31T23:11:28.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:28.033+0000 7f438fc15640 1 -- 172.21.15.46:0/2891775196 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4388100650 msgr2=0x7f4388100a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:28.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:28.033+0000 7f438fc15640 1 --2- 172.21.15.46:0/2891775196 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4388100650 0x7f4388100a70 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f4384009900 tx=0x7f438402bc30 comp rx=0 tx=0).stop 2024-03-31T23:11:28.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:28.033+0000 7f438fc15640 1 -- 172.21.15.46:0/2891775196 shutdown_connections 2024-03-31T23:11:28.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:28.033+0000 7f438fc15640 1 --2- 172.21.15.46:0/2891775196 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4388102200 0x7f4388108a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:28.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:28.033+0000 7f438fc15640 1 --2- 172.21.15.46:0/2891775196 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4388101840 0x7f4388101cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:28.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:28.033+0000 7f438fc15640 1 --2- 172.21.15.46:0/2891775196 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4388100650 0x7f4388100a70 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:28.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:28.033+0000 7f438fc15640 1 -- 172.21.15.46:0/2891775196 >> 172.21.15.46:0/2891775196 conn(0x7f43880fbbc0 msgr2=0x7f43880fe000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:11:28.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:28.033+0000 7f438fc15640 1 -- 172.21.15.46:0/2891775196 shutdown_connections 2024-03-31T23:11:28.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:28.033+0000 7f438fc15640 1 -- 172.21.15.46:0/2891775196 wait complete. 2024-03-31T23:11:28.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:28.033+0000 7f438fc15640 1 Processor -- start 2024-03-31T23:11:28.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:28.033+0000 7f438fc15640 1 -- start start 2024-03-31T23:11:28.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:28.033+0000 7f438fc15640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4388100650 0x7f438810afd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:28.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:28.033+0000 7f438fc15640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4388101840 0x7f4388109610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:28.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:28.033+0000 7f438fc15640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4388102200 0x7f4388109b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:28.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:28.033+0000 7f438fc15640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f438810d2a0 con 0x7f4388100650 2024-03-31T23:11:28.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:28.033+0000 7f438fc15640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f438810d120 con 0x7f4388102200 2024-03-31T23:11:28.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:28.033+0000 7f438fc15640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f438810d420 con 0x7f4388101840 2024-03-31T23:11:28.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:28.034+0000 7f438d9b1640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4388100650 0x7f438810afd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:28.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:28.034+0000 7f438d9b1640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4388100650 0x7f438810afd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:58366/0 (socket says 172.21.15.46:58366) 2024-03-31T23:11:28.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:28.034+0000 7f438d9b1640 1 -- 172.21.15.46:0/3006365673 learned_addr learned my addr 172.21.15.46:0/3006365673 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:11:28.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:28.034+0000 7f438e1b2640 1 --2- 172.21.15.46:0/3006365673 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4388102200 0x7f4388109b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:28.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:28.034+0000 7f438d1b0640 1 --2- 172.21.15.46:0/3006365673 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4388101840 0x7f4388109610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:28.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:28.034+0000 7f438d9b1640 1 -- 172.21.15.46:0/3006365673 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4388101840 msgr2=0x7f4388109610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:28.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:28.034+0000 7f438d9b1640 1 --2- 172.21.15.46:0/3006365673 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4388101840 0x7f4388109610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:28.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:28.034+0000 7f438d9b1640 1 -- 172.21.15.46:0/3006365673 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4388102200 msgr2=0x7f4388109b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:28.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:28.034+0000 7f438d9b1640 1 --2- 172.21.15.46:0/3006365673 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4388102200 0x7f4388109b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:28.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:28.034+0000 7f438d9b1640 1 -- 172.21.15.46:0/3006365673 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f438810a3e0 con 0x7f4388100650 2024-03-31T23:11:28.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:28.034+0000 7f438d9b1640 1 --2- 172.21.15.46:0/3006365673 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4388100650 0x7f438810afd0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f4384000c00 tx=0x7f4384005dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:11:28.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:28.034+0000 7f437effd640 1 -- 172.21.15.46:0/3006365673 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4384043020 con 0x7f4388100650 2024-03-31T23:11:28.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:28.034+0000 7f438fc15640 1 -- 172.21.15.46:0/3006365673 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f43881a1b10 con 0x7f4388100650 2024-03-31T23:11:28.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:28.034+0000 7f437effd640 1 -- 172.21.15.46:0/3006365673 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4384004350 con 0x7f4388100650 2024-03-31T23:11:28.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:28.034+0000 7f437effd640 1 -- 172.21.15.46:0/3006365673 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4384003b10 con 0x7f4388100650 2024-03-31T23:11:28.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:28.034+0000 7f438fc15640 1 -- 172.21.15.46:0/3006365673 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f43881a2060 con 0x7f4388100650 2024-03-31T23:11:28.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:28.035+0000 7f438fc15640 1 -- 172.21.15.46:0/3006365673 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4388107ae0 con 0x7f4388100650 2024-03-31T23:11:28.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:28.036+0000 7f437effd640 1 -- 172.21.15.46:0/3006365673 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 18) ==== 137320+0+0 (secure 0 0 0) 0x7f4384046020 con 0x7f4388100650 2024-03-31T23:11:28.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:28.037+0000 7f437effd640 1 --2- 172.21.15.46:0/3006365673 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f43740a24a0 0x7f43740a4960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:28.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:28.037+0000 7f438d1b0640 1 --2- 172.21.15.46:0/3006365673 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f43740a24a0 0x7f43740a4960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:28.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:28.037+0000 7f437effd640 1 -- 172.21.15.46:0/3006365673 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9208+0+0 (secure 0 0 0) 0x7f43840f1000 con 0x7f4388100650 2024-03-31T23:11:28.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:28.037+0000 7f438d1b0640 1 --2- 172.21.15.46:0/3006365673 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f43740a24a0 0x7f43740a4960 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f438810ad40 tx=0x7f4378005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:11:28.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:28.040+0000 7f437effd640 1 -- 172.21.15.46:0/3006365673 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f438404d020 con 0x7f4388100650 2024-03-31T23:11:28.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:11:27 smithi162 ceph-mon[36237]: from='client.? 172.21.15.46:0/754596586' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]: dispatch 2024-03-31T23:11:28.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:28.222+0000 7f438fc15640 1 -- 172.21.15.46:0/3006365673 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f438802d300 con 0x7f4388100650 2024-03-31T23:11:28.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:28.223+0000 7f437effd640 1 -- 172.21.15.46:0/3006365673 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v95) ==== 93+0+10 (secure 0 0 0) 0x7f43840bddf0 con 0x7f4388100650 2024-03-31T23:11:28.223 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 1 2024-03-31T23:11:28.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:28.225+0000 7f438fc15640 1 -- 172.21.15.46:0/3006365673 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f43740a24a0 msgr2=0x7f43740a4960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:28.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:28.225+0000 7f438fc15640 1 --2- 172.21.15.46:0/3006365673 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f43740a24a0 0x7f43740a4960 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f438810ad40 tx=0x7f4378005f90 comp rx=0 tx=0).stop 2024-03-31T23:11:28.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:28.225+0000 7f438fc15640 1 -- 172.21.15.46:0/3006365673 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4388100650 msgr2=0x7f438810afd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:28.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:28.225+0000 7f438fc15640 1 --2- 172.21.15.46:0/3006365673 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4388100650 0x7f438810afd0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f4384000c00 tx=0x7f4384005dc0 comp rx=0 tx=0).stop 2024-03-31T23:11:28.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:28.225+0000 7f438fc15640 1 -- 172.21.15.46:0/3006365673 shutdown_connections 2024-03-31T23:11:28.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:28.225+0000 7f438fc15640 1 --2- 172.21.15.46:0/3006365673 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f43740a24a0 0x7f43740a4960 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:28.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:28.225+0000 7f438fc15640 1 --2- 172.21.15.46:0/3006365673 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4388102200 0x7f4388109b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:28.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:28.225+0000 7f438fc15640 1 --2- 172.21.15.46:0/3006365673 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4388101840 0x7f4388109610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:28.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:28.225+0000 7f438fc15640 1 --2- 172.21.15.46:0/3006365673 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4388100650 0x7f438810afd0 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:28.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:28.225+0000 7f438fc15640 1 -- 172.21.15.46:0/3006365673 >> 172.21.15.46:0/3006365673 conn(0x7f43880fbbc0 msgr2=0x7f4388104cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:11:28.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:28.225+0000 7f438fc15640 1 -- 172.21.15.46:0/3006365673 shutdown_connections 2024-03-31T23:11:28.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:28.225+0000 7f438fc15640 1 -- 172.21.15.46:0/3006365673 wait complete. 2024-03-31T23:11:28.236 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-31T23:11:28.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:28.341+0000 7f19f7193640 1 -- 172.21.15.46:0/3251854057 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f19f01021f0 msgr2=0x7f19f01045e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:28.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:28.341+0000 7f19f7193640 1 --2- 172.21.15.46:0/3251854057 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f19f01021f0 0x7f19f01045e0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f19e4009860 tx=0x7f19e402bb70 comp rx=0 tx=0).stop 2024-03-31T23:11:28.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:28.341+0000 7f19f7193640 1 -- 172.21.15.46:0/3251854057 shutdown_connections 2024-03-31T23:11:28.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:28.341+0000 7f19f7193640 1 --2- 172.21.15.46:0/3251854057 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f19f0107450 0x7f19f01098a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:28.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:28.341+0000 7f19f7193640 1 --2- 172.21.15.46:0/3251854057 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f19f0104b20 0x7f19f0106f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:28.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:28.341+0000 7f19f7193640 1 --2- 172.21.15.46:0/3251854057 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f19f01021f0 0x7f19f01045e0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:28.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:28.341+0000 7f19f7193640 1 -- 172.21.15.46:0/3251854057 >> 172.21.15.46:0/3251854057 conn(0x7f19f00fbc00 msgr2=0x7f19f00fe020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:11:28.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:28.341+0000 7f19f7193640 1 -- 172.21.15.46:0/3251854057 shutdown_connections 2024-03-31T23:11:28.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:28.341+0000 7f19f7193640 1 -- 172.21.15.46:0/3251854057 wait complete. 2024-03-31T23:11:28.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:28.342+0000 7f19f7193640 1 Processor -- start 2024-03-31T23:11:28.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:28.342+0000 7f19f7193640 1 -- start start 2024-03-31T23:11:28.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:28.342+0000 7f19f7193640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f19f01021f0 0x7f19f01979c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:28.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:28.342+0000 7f19f7193640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f19f0104b20 0x7f19f0197f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:28.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:28.342+0000 7f19f7193640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f19f0107450 0x7f19f019ef70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:28.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:28.342+0000 7f19f7193640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f19f010d2c0 con 0x7f19f0104b20 2024-03-31T23:11:28.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:28.342+0000 7f19f7193640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f19f010d140 con 0x7f19f01021f0 2024-03-31T23:11:28.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:28.342+0000 7f19f7193640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f19f010d440 con 0x7f19f0107450 2024-03-31T23:11:28.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:28.342+0000 7f19effff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f19f0104b20 0x7f19f0197f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:28.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:28.342+0000 7f19effff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f19f0104b20 0x7f19f0197f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:58380/0 (socket says 172.21.15.46:58380) 2024-03-31T23:11:28.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:28.342+0000 7f19effff640 1 -- 172.21.15.46:0/3062029017 learned_addr learned my addr 172.21.15.46:0/3062029017 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:11:28.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:28.342+0000 7f19f4f2f640 1 --2- 172.21.15.46:0/3062029017 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f19f01021f0 0x7f19f01979c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:28.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:28.342+0000 7f19f5730640 1 --2- 172.21.15.46:0/3062029017 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f19f0107450 0x7f19f019ef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:28.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:28.342+0000 7f19effff640 1 -- 172.21.15.46:0/3062029017 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f19f0107450 msgr2=0x7f19f019ef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:28.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:28.342+0000 7f19effff640 1 --2- 172.21.15.46:0/3062029017 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f19f0107450 0x7f19f019ef70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:28.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:28.342+0000 7f19effff640 1 -- 172.21.15.46:0/3062029017 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f19f01021f0 msgr2=0x7f19f01979c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:28.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:28.342+0000 7f19effff640 1 --2- 172.21.15.46:0/3062029017 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f19f01021f0 0x7f19f01979c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:28.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:28.342+0000 7f19effff640 1 -- 172.21.15.46:0/3062029017 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f19f019f5e0 con 0x7f19f0104b20 2024-03-31T23:11:28.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:28.342+0000 7f19f5730640 1 --2- 172.21.15.46:0/3062029017 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f19f0107450 0x7f19f019ef70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T23:11:28.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:28.343+0000 7f19effff640 1 --2- 172.21.15.46:0/3062029017 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f19f0104b20 0x7f19f0197f00 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f19dc00e9a0 tx=0x7f19dc00ee60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:11:28.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:28.343+0000 7f19edffb640 1 -- 172.21.15.46:0/3062029017 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19dc00cd80 con 0x7f19f0104b20 2024-03-31T23:11:28.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:28.343+0000 7f19f7193640 1 -- 172.21.15.46:0/3062029017 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f19f019f8d0 con 0x7f19f0104b20 2024-03-31T23:11:28.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:28.343+0000 7f19edffb640 1 -- 172.21.15.46:0/3062029017 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f19dc0035f0 con 0x7f19f0104b20 2024-03-31T23:11:28.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:28.343+0000 7f19edffb640 1 -- 172.21.15.46:0/3062029017 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19dc010690 con 0x7f19f0104b20 2024-03-31T23:11:28.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:28.343+0000 7f19f7193640 1 -- 172.21.15.46:0/3062029017 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f19f019fe70 con 0x7f19f0104b20 2024-03-31T23:11:28.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:28.344+0000 7f19f7193640 1 -- 172.21.15.46:0/3062029017 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f19f00fd6f0 con 0x7f19f0104b20 2024-03-31T23:11:28.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:28.345+0000 7f19edffb640 1 -- 172.21.15.46:0/3062029017 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 18) ==== 137320+0+0 (secure 0 0 0) 0x7f19dc0037a0 con 0x7f19f0104b20 2024-03-31T23:11:28.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:28.345+0000 7f19edffb640 1 --2- 172.21.15.46:0/3062029017 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f19e00a24a0 0x7f19e00a4960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:28.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:28.346+0000 7f19f4f2f640 1 --2- 172.21.15.46:0/3062029017 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f19e00a24a0 0x7f19e00a4960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:28.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:28.346+0000 7f19edffb640 1 -- 172.21.15.46:0/3062029017 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9208+0+0 (secure 0 0 0) 0x7f19dc0cee70 con 0x7f19f0104b20 2024-03-31T23:11:28.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:28.346+0000 7f19f4f2f640 1 --2- 172.21.15.46:0/3062029017 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f19e00a24a0 0x7f19e00a4960 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f19e4000c00 tx=0x7f19e40349a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:11:28.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:28.350+0000 7f19edffb640 1 -- 172.21.15.46:0/3062029017 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f19dc09bc00 con 0x7f19f0104b20 2024-03-31T23:11:28.531 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:28.531+0000 7f19f7193640 1 -- 172.21.15.46:0/3062029017 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f19f0062350 con 0x7f19f0104b20 2024-03-31T23:11:28.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:28.531+0000 7f19edffb640 1 -- 172.21.15.46:0/3062029017 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v95) ==== 104+0+11 (secure 0 0 0) 0x7f19dc09b500 con 0x7f19f0104b20 2024-03-31T23:11:28.532 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 32 2024-03-31T23:11:28.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:28.533+0000 7f19f7193640 1 -- 172.21.15.46:0/3062029017 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f19e00a24a0 msgr2=0x7f19e00a4960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:28.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:28.533+0000 7f19f7193640 1 --2- 172.21.15.46:0/3062029017 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f19e00a24a0 0x7f19e00a4960 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f19e4000c00 tx=0x7f19e40349a0 comp rx=0 tx=0).stop 2024-03-31T23:11:28.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:28.533+0000 7f19f7193640 1 -- 172.21.15.46:0/3062029017 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f19f0104b20 msgr2=0x7f19f0197f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:28.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:28.533+0000 7f19f7193640 1 --2- 172.21.15.46:0/3062029017 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f19f0104b20 0x7f19f0197f00 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f19dc00e9a0 tx=0x7f19dc00ee60 comp rx=0 tx=0).stop 2024-03-31T23:11:28.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:28.533+0000 7f19f7193640 1 -- 172.21.15.46:0/3062029017 shutdown_connections 2024-03-31T23:11:28.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:28.533+0000 7f19f7193640 1 --2- 172.21.15.46:0/3062029017 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f19e00a24a0 0x7f19e00a4960 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:28.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:28.533+0000 7f19f7193640 1 --2- 172.21.15.46:0/3062029017 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f19f0107450 0x7f19f019ef70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:28.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:28.533+0000 7f19f7193640 1 --2- 172.21.15.46:0/3062029017 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f19f0104b20 0x7f19f0197f00 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:28.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:28.533+0000 7f19f7193640 1 --2- 172.21.15.46:0/3062029017 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f19f01021f0 0x7f19f01979c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:28.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:28.533+0000 7f19f7193640 1 -- 172.21.15.46:0/3062029017 >> 172.21.15.46:0/3062029017 conn(0x7f19f00fbc00 msgr2=0x7f19f0102cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:11:28.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:28.533+0000 7f19f7193640 1 -- 172.21.15.46:0/3062029017 shutdown_connections 2024-03-31T23:11:28.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:28.533+0000 7f19f7193640 1 -- 172.21.15.46:0/3062029017 wait complete. 2024-03-31T23:11:28.544 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-31T23:11:28.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:28.651+0000 7f40c5d6c640 1 -- 172.21.15.46:0/2432957713 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f40c0102200 msgr2=0x7f40c0108a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:28.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:28.651+0000 7f40c5d6c640 1 --2- 172.21.15.46:0/2432957713 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f40c0102200 0x7f40c0108a90 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f40b8009ca0 tx=0x7f40b802c0f0 comp rx=0 tx=0).stop 2024-03-31T23:11:28.654 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:28.651+0000 7f40c5d6c640 1 -- 172.21.15.46:0/2432957713 shutdown_connections 2024-03-31T23:11:28.654 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:28.651+0000 7f40c5d6c640 1 --2- 172.21.15.46:0/2432957713 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f40c0102200 0x7f40c0108a90 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:28.654 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:28.651+0000 7f40c5d6c640 1 --2- 172.21.15.46:0/2432957713 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f40c0101840 0x7f40c0101cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:28.654 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:28.651+0000 7f40c5d6c640 1 --2- 172.21.15.46:0/2432957713 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f40c0100650 0x7f40c0100a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:28.654 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:28.651+0000 7f40c5d6c640 1 -- 172.21.15.46:0/2432957713 >> 172.21.15.46:0/2432957713 conn(0x7f40c00fbbe0 msgr2=0x7f40c00fe000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:11:28.654 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:28.651+0000 7f40c5d6c640 1 -- 172.21.15.46:0/2432957713 shutdown_connections 2024-03-31T23:11:28.654 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:28.651+0000 7f40c5d6c640 1 -- 172.21.15.46:0/2432957713 wait complete. 2024-03-31T23:11:28.654 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:28.652+0000 7f40c5d6c640 1 Processor -- start 2024-03-31T23:11:28.654 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:28.652+0000 7f40c5d6c640 1 -- start start 2024-03-31T23:11:28.654 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:28.652+0000 7f40c5d6c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f40c0100650 0x7f40c0197960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:28.654 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:28.652+0000 7f40c5d6c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f40c0101840 0x7f40c0197ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:28.654 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:28.652+0000 7f40c5d6c640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f40c0102200 0x7f40c019eec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:28.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:28.652+0000 7f40c5d6c640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f40c010c160 con 0x7f40c0100650 2024-03-31T23:11:28.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:28.652+0000 7f40c5d6c640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f40c010bfe0 con 0x7f40c0102200 2024-03-31T23:11:28.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:28.652+0000 7f40c5d6c640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f40c010c2e0 con 0x7f40c0101840 2024-03-31T23:11:28.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:28.652+0000 7f40bf7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f40c0100650 0x7f40c0197960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:28.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:28.652+0000 7f40bf7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f40c0100650 0x7f40c0197960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:58394/0 (socket says 172.21.15.46:58394) 2024-03-31T23:11:28.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:28.652+0000 7f40bf7fe640 1 -- 172.21.15.46:0/2854046415 learned_addr learned my addr 172.21.15.46:0/2854046415 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:11:28.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:28.652+0000 7f40beffd640 1 --2- 172.21.15.46:0/2854046415 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f40c0101840 0x7f40c0197ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:28.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:28.652+0000 7f40bffff640 1 --2- 172.21.15.46:0/2854046415 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f40c0102200 0x7f40c019eec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:28.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:28.652+0000 7f40bf7fe640 1 -- 172.21.15.46:0/2854046415 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f40c0101840 msgr2=0x7f40c0197ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:28.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:28.652+0000 7f40bf7fe640 1 --2- 172.21.15.46:0/2854046415 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f40c0101840 0x7f40c0197ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:28.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:28.652+0000 7f40bf7fe640 1 -- 172.21.15.46:0/2854046415 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f40c0102200 msgr2=0x7f40c019eec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:28.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:28.652+0000 7f40bf7fe640 1 --2- 172.21.15.46:0/2854046415 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f40c0102200 0x7f40c019eec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:28.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:28.652+0000 7f40bf7fe640 1 -- 172.21.15.46:0/2854046415 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40c019f530 con 0x7f40c0100650 2024-03-31T23:11:28.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:28.653+0000 7f40bf7fe640 1 --2- 172.21.15.46:0/2854046415 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f40c0100650 0x7f40c0197960 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f40b000e9c0 tx=0x7f40b000ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:11:28.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:28.653+0000 7f40bcff9640 1 -- 172.21.15.46:0/2854046415 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40b000cd80 con 0x7f40c0100650 2024-03-31T23:11:28.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:28.653+0000 7f40bcff9640 1 -- 172.21.15.46:0/2854046415 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f40b00035f0 con 0x7f40c0100650 2024-03-31T23:11:28.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:28.653+0000 7f40c5d6c640 1 -- 172.21.15.46:0/2854046415 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f40c019f820 con 0x7f40c0100650 2024-03-31T23:11:28.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:28.653+0000 7f40bcff9640 1 -- 172.21.15.46:0/2854046415 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40b0010690 con 0x7f40c0100650 2024-03-31T23:11:28.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:28.653+0000 7f40c5d6c640 1 -- 172.21.15.46:0/2854046415 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f40c019fdc0 con 0x7f40c0100650 2024-03-31T23:11:28.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:28.654+0000 7f40c5d6c640 1 -- 172.21.15.46:0/2854046415 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f40c01a0020 con 0x7f40c0100650 2024-03-31T23:11:28.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:28.655+0000 7f40bcff9640 1 -- 172.21.15.46:0/2854046415 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 18) ==== 137320+0+0 (secure 0 0 0) 0x7f40b00037a0 con 0x7f40c0100650 2024-03-31T23:11:28.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:28.655+0000 7f40bcff9640 1 --2- 172.21.15.46:0/2854046415 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f40a80a2520 0x7f40a80a49e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:28.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:28.656+0000 7f40beffd640 1 --2- 172.21.15.46:0/2854046415 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f40a80a2520 0x7f40a80a49e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:28.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:28.656+0000 7f40bcff9640 1 -- 172.21.15.46:0/2854046415 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9208+0+0 (secure 0 0 0) 0x7f40b00cee40 con 0x7f40c0100650 2024-03-31T23:11:28.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:28.656+0000 7f40beffd640 1 --2- 172.21.15.46:0/2854046415 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f40a80a2520 0x7f40a80a49e0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f40b40095f0 tx=0x7f40b4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:11:28.660 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:28.660+0000 7f40bcff9640 1 -- 172.21.15.46:0/2854046415 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f40b009bbd0 con 0x7f40c0100650 2024-03-31T23:11:28.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:28.842+0000 7f40c5d6c640 1 -- 172.21.15.46:0/2854046415 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f40c01a0290 con 0x7f40c0100650 2024-03-31T23:11:28.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:28.842+0000 7f40bcff9640 1 -- 172.21.15.46:0/2854046415 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v95) ==== 100+0+11 (secure 0 0 0) 0x7f40b009b4d0 con 0x7f40c0100650 2024-03-31T23:11:28.842 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 64 2024-03-31T23:11:28.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:28.844+0000 7f40c5d6c640 1 -- 172.21.15.46:0/2854046415 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f40a80a2520 msgr2=0x7f40a80a49e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:28.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:28.844+0000 7f40c5d6c640 1 --2- 172.21.15.46:0/2854046415 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f40a80a2520 0x7f40a80a49e0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f40b40095f0 tx=0x7f40b4005f90 comp rx=0 tx=0).stop 2024-03-31T23:11:28.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:28.844+0000 7f40c5d6c640 1 -- 172.21.15.46:0/2854046415 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f40c0100650 msgr2=0x7f40c0197960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:28.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:28.844+0000 7f40c5d6c640 1 --2- 172.21.15.46:0/2854046415 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f40c0100650 0x7f40c0197960 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f40b000e9c0 tx=0x7f40b000ee80 comp rx=0 tx=0).stop 2024-03-31T23:11:28.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:28.844+0000 7f40c5d6c640 1 -- 172.21.15.46:0/2854046415 shutdown_connections 2024-03-31T23:11:28.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:28.844+0000 7f40c5d6c640 1 --2- 172.21.15.46:0/2854046415 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f40a80a2520 0x7f40a80a49e0 secure :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f40b40095f0 tx=0x7f40b4005f90 comp rx=0 tx=0).stop 2024-03-31T23:11:28.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:28.844+0000 7f40c5d6c640 1 --2- 172.21.15.46:0/2854046415 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f40c0102200 0x7f40c019eec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:28.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:28.844+0000 7f40c5d6c640 1 --2- 172.21.15.46:0/2854046415 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f40c0101840 0x7f40c0197ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:28.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:28.844+0000 7f40c5d6c640 1 --2- 172.21.15.46:0/2854046415 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f40c0100650 0x7f40c0197960 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:28.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:28.844+0000 7f40c5d6c640 1 -- 172.21.15.46:0/2854046415 >> 172.21.15.46:0/2854046415 conn(0x7f40c00fbbe0 msgr2=0x7f40c00fd660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:11:28.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:28.844+0000 7f40c5d6c640 1 -- 172.21.15.46:0/2854046415 shutdown_connections 2024-03-31T23:11:28.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:28.844+0000 7f40c5d6c640 1 -- 172.21.15.46:0/2854046415 wait complete. 2024-03-31T23:11:28.854 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:11:28 smithi046 ceph-mon[32498]: pgmap v364: 129 pgs: 129 active+clean; 642 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s; 0 B/s, 0 objects/s recovering 2024-03-31T23:11:28.854 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:11:28 smithi046 ceph-mon[32498]: from='client.? 172.21.15.46:0/754596586' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]': finished 2024-03-31T23:11:28.854 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:11:28 smithi046 ceph-mon[32498]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-03-31T23:11:28.854 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:11:28 smithi046 ceph-mon[32498]: from='client.? 172.21.15.46:0/2957863234' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-31T23:11:28.854 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:11:28 smithi046 ceph-mon[32498]: from='client.? 172.21.15.46:0/3006365673' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-31T23:11:28.854 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:11:28 smithi046 ceph-mon[32498]: from='client.? 172.21.15.46:0/3062029017' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-31T23:11:28.855 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-31T23:11:28.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:11:28 smithi142 ceph-mon[35271]: pgmap v364: 129 pgs: 129 active+clean; 642 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s; 0 B/s, 0 objects/s recovering 2024-03-31T23:11:28.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:11:28 smithi142 ceph-mon[35271]: from='client.? 172.21.15.46:0/754596586' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]': finished 2024-03-31T23:11:28.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:11:28 smithi142 ceph-mon[35271]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-03-31T23:11:28.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:11:28 smithi142 ceph-mon[35271]: from='client.? 172.21.15.46:0/2957863234' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-31T23:11:28.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:11:28 smithi142 ceph-mon[35271]: from='client.? 172.21.15.46:0/3006365673' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-31T23:11:28.944 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:11:28 smithi142 ceph-mon[35271]: from='client.? 172.21.15.46:0/3062029017' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-31T23:11:28.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:28.961+0000 7f4abcd25640 1 -- 172.21.15.46:0/2716704554 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4ab8101850 msgr2=0x7f4ab8101cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:28.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:28.961+0000 7f4abcd25640 1 --2- 172.21.15.46:0/2716704554 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4ab8101850 0x7f4ab8101cd0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f4aa0009900 tx=0x7f4aa002bc30 comp rx=0 tx=0).stop 2024-03-31T23:11:28.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:28.961+0000 7f4abcd25640 1 -- 172.21.15.46:0/2716704554 shutdown_connections 2024-03-31T23:11:28.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:28.961+0000 7f4abcd25640 1 --2- 172.21.15.46:0/2716704554 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4ab8102210 0x7f4ab8108aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:28.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:28.961+0000 7f4abcd25640 1 --2- 172.21.15.46:0/2716704554 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4ab8101850 0x7f4ab8101cd0 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:28.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:28.961+0000 7f4abcd25640 1 --2- 172.21.15.46:0/2716704554 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4ab8100660 0x7f4ab8100a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:28.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:28.961+0000 7f4abcd25640 1 -- 172.21.15.46:0/2716704554 >> 172.21.15.46:0/2716704554 conn(0x7f4ab80fbbf0 msgr2=0x7f4ab80fe010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:11:28.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:28.961+0000 7f4abcd25640 1 -- 172.21.15.46:0/2716704554 shutdown_connections 2024-03-31T23:11:28.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:28.961+0000 7f4abcd25640 1 -- 172.21.15.46:0/2716704554 wait complete. 2024-03-31T23:11:28.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:28.962+0000 7f4abcd25640 1 Processor -- start 2024-03-31T23:11:28.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:28.962+0000 7f4abcd25640 1 -- start start 2024-03-31T23:11:28.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:28.962+0000 7f4abcd25640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4ab8100660 0x7f4ab81045f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:28.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:28.962+0000 7f4abcd25640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4ab8101850 0x7f4ab8102c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:28.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:28.962+0000 7f4abcd25640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4ab8102210 0x7f4ab8103170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:28.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:28.962+0000 7f4abcd25640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f4ab810c180 con 0x7f4ab8101850 2024-03-31T23:11:28.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:28.962+0000 7f4abcd25640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f4ab810c000 con 0x7f4ab8100660 2024-03-31T23:11:28.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:28.962+0000 7f4abcd25640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f4ab810c300 con 0x7f4ab8102210 2024-03-31T23:11:28.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:28.962+0000 7f4aaffff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4ab8101850 0x7f4ab8102c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:28.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:28.962+0000 7f4aaffff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4ab8101850 0x7f4ab8102c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:58410/0 (socket says 172.21.15.46:58410) 2024-03-31T23:11:28.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:28.962+0000 7f4aaffff640 1 -- 172.21.15.46:0/1073321381 learned_addr learned my addr 172.21.15.46:0/1073321381 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:11:28.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:28.962+0000 7f4ab6d9d640 1 --2- 172.21.15.46:0/1073321381 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4ab8102210 0x7f4ab8103170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:28.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:28.962+0000 7f4ab659c640 1 --2- 172.21.15.46:0/1073321381 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4ab8100660 0x7f4ab81045f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:28.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:28.962+0000 7f4aaffff640 1 -- 172.21.15.46:0/1073321381 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4ab8102210 msgr2=0x7f4ab8103170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:28.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:28.962+0000 7f4aaffff640 1 --2- 172.21.15.46:0/1073321381 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4ab8102210 0x7f4ab8103170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:28.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:28.962+0000 7f4aaffff640 1 -- 172.21.15.46:0/1073321381 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4ab8100660 msgr2=0x7f4ab81045f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:28.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:28.963+0000 7f4aaffff640 1 --2- 172.21.15.46:0/1073321381 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4ab8100660 0x7f4ab81045f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:28.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:28.963+0000 7f4aaffff640 1 -- 172.21.15.46:0/1073321381 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ab8103a00 con 0x7f4ab8101850 2024-03-31T23:11:28.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:28.963+0000 7f4aaffff640 1 --2- 172.21.15.46:0/1073321381 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4ab8101850 0x7f4ab8102c30 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f4aa0006010 tx=0x7f4aa0005e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:11:28.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:28.963+0000 7f4aaf7fe640 1 -- 172.21.15.46:0/1073321381 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4aa0043020 con 0x7f4ab8101850 2024-03-31T23:11:28.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:28.963+0000 7f4aaf7fe640 1 -- 172.21.15.46:0/1073321381 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4aa00043d0 con 0x7f4ab8101850 2024-03-31T23:11:28.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:28.963+0000 7f4abcd25640 1 -- 172.21.15.46:0/1073321381 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4ab81a1b60 con 0x7f4ab8101850 2024-03-31T23:11:28.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:28.963+0000 7f4aaf7fe640 1 -- 172.21.15.46:0/1073321381 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4aa0003a80 con 0x7f4ab8101850 2024-03-31T23:11:28.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:28.963+0000 7f4abcd25640 1 -- 172.21.15.46:0/1073321381 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4ab81a20b0 con 0x7f4ab8101850 2024-03-31T23:11:28.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:28.964+0000 7f4abcd25640 1 -- 172.21.15.46:0/1073321381 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4ab81a2280 con 0x7f4ab8101850 2024-03-31T23:11:28.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:28.965+0000 7f4aaf7fe640 1 -- 172.21.15.46:0/1073321381 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 18) ==== 137320+0+0 (secure 0 0 0) 0x7f4aa0046020 con 0x7f4ab8101850 2024-03-31T23:11:28.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:28.965+0000 7f4aaf7fe640 1 --2- 172.21.15.46:0/1073321381 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f4aa40a24a0 0x7f4aa40a4960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:28.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:28.966+0000 7f4ab659c640 1 --2- 172.21.15.46:0/1073321381 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f4aa40a24a0 0x7f4aa40a4960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:28.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:28.966+0000 7f4aaf7fe640 1 -- 172.21.15.46:0/1073321381 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9208+0+0 (secure 0 0 0) 0x7f4aa00f0d30 con 0x7f4ab8101850 2024-03-31T23:11:28.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:28.966+0000 7f4ab659c640 1 --2- 172.21.15.46:0/1073321381 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f4aa40a24a0 0x7f4aa40a4960 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f4ab81016b0 tx=0x7f4aa8005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:11:28.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:28.970+0000 7f4aaf7fe640 1 -- 172.21.15.46:0/1073321381 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f4aa004d020 con 0x7f4ab8101850 2024-03-31T23:11:29.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:11:28 smithi162 ceph-mon[36237]: pgmap v364: 129 pgs: 129 active+clean; 642 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s; 0 B/s, 0 objects/s recovering 2024-03-31T23:11:29.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:11:28 smithi162 ceph-mon[36237]: from='client.? 172.21.15.46:0/754596586' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]': finished 2024-03-31T23:11:29.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:11:28 smithi162 ceph-mon[36237]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-03-31T23:11:29.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:11:28 smithi162 ceph-mon[36237]: from='client.? 172.21.15.46:0/2957863234' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-31T23:11:29.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:11:28 smithi162 ceph-mon[36237]: from='client.? 172.21.15.46:0/3006365673' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-31T23:11:29.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:11:28 smithi162 ceph-mon[36237]: from='client.? 172.21.15.46:0/3062029017' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-31T23:11:29.151 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:29.151+0000 7f4abcd25640 1 -- 172.21.15.46:0/1073321381 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f4ab81a2860 con 0x7f4ab8101850 2024-03-31T23:11:29.152 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:29.152+0000 7f4aaf7fe640 1 -- 172.21.15.46:0/1073321381 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v95) ==== 103+0+11 (secure 0 0 0) 0x7f4aa00bdac0 con 0x7f4ab8101850 2024-03-31T23:11:29.152 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 32 2024-03-31T23:11:29.154 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:29.154+0000 7f4abcd25640 1 -- 172.21.15.46:0/1073321381 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f4aa40a24a0 msgr2=0x7f4aa40a4960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:29.154 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:29.154+0000 7f4abcd25640 1 --2- 172.21.15.46:0/1073321381 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f4aa40a24a0 0x7f4aa40a4960 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f4ab81016b0 tx=0x7f4aa8005f90 comp rx=0 tx=0).stop 2024-03-31T23:11:29.154 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:29.154+0000 7f4abcd25640 1 -- 172.21.15.46:0/1073321381 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4ab8101850 msgr2=0x7f4ab8102c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:29.154 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:29.154+0000 7f4abcd25640 1 --2- 172.21.15.46:0/1073321381 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4ab8101850 0x7f4ab8102c30 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f4aa0006010 tx=0x7f4aa0005e40 comp rx=0 tx=0).stop 2024-03-31T23:11:29.154 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:29.154+0000 7f4abcd25640 1 -- 172.21.15.46:0/1073321381 shutdown_connections 2024-03-31T23:11:29.154 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:29.154+0000 7f4abcd25640 1 --2- 172.21.15.46:0/1073321381 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f4aa40a24a0 0x7f4aa40a4960 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:29.154 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:29.154+0000 7f4abcd25640 1 --2- 172.21.15.46:0/1073321381 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4ab8102210 0x7f4ab8103170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:29.154 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:29.154+0000 7f4abcd25640 1 --2- 172.21.15.46:0/1073321381 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4ab8101850 0x7f4ab8102c30 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:29.154 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:29.154+0000 7f4abcd25640 1 --2- 172.21.15.46:0/1073321381 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4ab8100660 0x7f4ab81045f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:29.154 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:29.154+0000 7f4abcd25640 1 -- 172.21.15.46:0/1073321381 >> 172.21.15.46:0/1073321381 conn(0x7f4ab80fbbf0 msgr2=0x7f4ab8069d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:11:29.154 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:29.154+0000 7f4abcd25640 1 -- 172.21.15.46:0/1073321381 shutdown_connections 2024-03-31T23:11:29.155 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:29.154+0000 7f4abcd25640 1 -- 172.21.15.46:0/1073321381 wait complete. 2024-03-31T23:11:29.165 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-31T23:11:29.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:29.267+0000 7f53c2eae640 1 -- 172.21.15.46:0/3752692329 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f53bc102220 msgr2=0x7f53bc108ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:29.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:29.267+0000 7f53c2eae640 1 --2- 172.21.15.46:0/3752692329 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f53bc102220 0x7f53bc108ab0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f53b4009be0 tx=0x7f53b402bef0 comp rx=0 tx=0).stop 2024-03-31T23:11:29.273 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:29.267+0000 7f53c2eae640 1 -- 172.21.15.46:0/3752692329 shutdown_connections 2024-03-31T23:11:29.273 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:29.267+0000 7f53c2eae640 1 --2- 172.21.15.46:0/3752692329 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f53bc102220 0x7f53bc108ab0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:29.273 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:29.267+0000 7f53c2eae640 1 --2- 172.21.15.46:0/3752692329 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f53bc101860 0x7f53bc101ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:29.273 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:29.267+0000 7f53c2eae640 1 --2- 172.21.15.46:0/3752692329 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f53bc100670 0x7f53bc100a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:29.273 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:29.267+0000 7f53c2eae640 1 -- 172.21.15.46:0/3752692329 >> 172.21.15.46:0/3752692329 conn(0x7f53bc0fbbe0 msgr2=0x7f53bc0fe020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:11:29.273 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:29.267+0000 7f53c2eae640 1 -- 172.21.15.46:0/3752692329 shutdown_connections 2024-03-31T23:11:29.273 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:29.267+0000 7f53c2eae640 1 -- 172.21.15.46:0/3752692329 wait complete. 2024-03-31T23:11:29.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:29.268+0000 7f53c2eae640 1 Processor -- start 2024-03-31T23:11:29.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:29.268+0000 7f53c2eae640 1 -- start start 2024-03-31T23:11:29.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:29.268+0000 7f53c2eae640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f53bc100670 0x7f53bc1967f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:29.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:29.268+0000 7f53c2eae640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f53bc101860 0x7f53bc196d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:29.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:29.268+0000 7f53c2eae640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f53bc102220 0x7f53bc19dda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:29.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:29.268+0000 7f53c2eae640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f53bc10c180 con 0x7f53bc101860 2024-03-31T23:11:29.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:29.268+0000 7f53c2eae640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f53bc10c000 con 0x7f53bc100670 2024-03-31T23:11:29.275 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:29.268+0000 7f53c2eae640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f53bc10c300 con 0x7f53bc102220 2024-03-31T23:11:29.275 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:29.268+0000 7f53bbfff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f53bc101860 0x7f53bc196d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:29.275 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:29.268+0000 7f53c0c4a640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f53bc100670 0x7f53bc1967f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:29.275 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:29.268+0000 7f53bbfff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f53bc101860 0x7f53bc196d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:58434/0 (socket says 172.21.15.46:58434) 2024-03-31T23:11:29.275 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:29.268+0000 7f53bbfff640 1 -- 172.21.15.46:0/136660239 learned_addr learned my addr 172.21.15.46:0/136660239 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:11:29.275 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:29.268+0000 7f53c144b640 1 --2- 172.21.15.46:0/136660239 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f53bc102220 0x7f53bc19dda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:29.275 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:29.268+0000 7f53c0c4a640 1 -- 172.21.15.46:0/136660239 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f53bc102220 msgr2=0x7f53bc19dda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:29.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:29.268+0000 7f53c0c4a640 1 --2- 172.21.15.46:0/136660239 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f53bc102220 0x7f53bc19dda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:29.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:29.268+0000 7f53c0c4a640 1 -- 172.21.15.46:0/136660239 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f53bc101860 msgr2=0x7f53bc196d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:29.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:29.268+0000 7f53c0c4a640 1 --2- 172.21.15.46:0/136660239 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f53bc101860 0x7f53bc196d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:29.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:29.268+0000 7f53c0c4a640 1 -- 172.21.15.46:0/136660239 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53bc19e410 con 0x7f53bc100670 2024-03-31T23:11:29.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:29.268+0000 7f53bbfff640 1 --2- 172.21.15.46:0/136660239 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f53bc101860 0x7f53bc196d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:11:29.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:29.269+0000 7f53c0c4a640 1 --2- 172.21.15.46:0/136660239 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f53bc100670 0x7f53bc1967f0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f53bc1016c0 tx=0x7f53b000ee50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:11:29.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:29.269+0000 7f53b9ffb640 1 -- 172.21.15.46:0/136660239 <== mon.2 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53b0004440 con 0x7f53bc100670 2024-03-31T23:11:29.277 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:29.269+0000 7f53b9ffb640 1 -- 172.21.15.46:0/136660239 <== mon.2 v2:172.21.15.142:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f53b000cd70 con 0x7f53bc100670 2024-03-31T23:11:29.277 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:29.269+0000 7f53c2eae640 1 -- 172.21.15.46:0/136660239 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f53bc19e700 con 0x7f53bc100670 2024-03-31T23:11:29.277 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:29.269+0000 7f53b9ffb640 1 -- 172.21.15.46:0/136660239 <== mon.2 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53b0019480 con 0x7f53bc100670 2024-03-31T23:11:29.277 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:29.269+0000 7f53c2eae640 1 -- 172.21.15.46:0/136660239 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f53bc19ec50 con 0x7f53bc100670 2024-03-31T23:11:29.277 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:29.270+0000 7f53c2eae640 1 -- 172.21.15.46:0/136660239 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f53bc107990 con 0x7f53bc100670 2024-03-31T23:11:29.277 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:29.271+0000 7f53b9ffb640 1 -- 172.21.15.46:0/136660239 <== mon.2 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 18) ==== 137320+0+0 (secure 0 0 0) 0x7f53b0003710 con 0x7f53bc100670 2024-03-31T23:11:29.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:29.271+0000 7f53b9ffb640 1 --2- 172.21.15.46:0/136660239 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f53ac0a24a0 0x7f53ac0a4960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:29.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:29.272+0000 7f53bbfff640 1 --2- 172.21.15.46:0/136660239 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f53ac0a24a0 0x7f53ac0a4960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:29.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:29.272+0000 7f53b9ffb640 1 -- 172.21.15.46:0/136660239 <== mon.2 v2:172.21.15.142:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9208+0+0 (secure 0 0 0) 0x7f53b00ceec0 con 0x7f53bc100670 2024-03-31T23:11:29.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:29.272+0000 7f53bbfff640 1 --2- 172.21.15.46:0/136660239 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f53ac0a24a0 0x7f53ac0a4960 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f53bc197d00 tx=0x7f53a8005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:11:29.279 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:29.278+0000 7f53b9ffb640 1 -- 172.21.15.46:0/136660239 <== mon.2 v2:172.21.15.142:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f53b0009510 con 0x7f53bc100670 2024-03-31T23:11:29.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:29.463+0000 7f53c2eae640 1 -- 172.21.15.46:0/136660239 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f53bc062350 con 0x7f53bc100670 2024-03-31T23:11:29.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:29.465+0000 7f53b9ffb640 1 -- 172.21.15.46:0/136660239 <== mon.2 v2:172.21.15.142:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v95) ==== 74+0+24819 (secure 0 0 0) 0x7f53b009bcb0 con 0x7f53bc100670 2024-03-31T23:11:29.465 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-31T23:11:29.466 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":95,"fsid":"6a9d8f08-efb2-11ee-b647-cb9ed24678a4","created":"2024-03-31T23:01:15.463711+0000","modified":"2024-03-31T23:11:10.531495+0000","last_up_change":"2024-03-31T23:08:20.199888+0000","last_in_change":"2024-03-31T23:07:59.438035+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-31T23:04:32.823751+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-31T23:09:09.145711+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-31T23:09:10.148579+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-31T23:09:13.419519+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"91","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"47429095-3f46-4ed7-b7c4-504464719815","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":1824772695},{"type":"v1","addr":"172.21.15.46:6803","nonce":1824772695}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":1824772695},{"type":"v1","addr":"172.21.15.46:6805","nonce":1824772695}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":1824772695},{"type":"v1","addr":"172.21.15.46:6809","nonce":1824772695}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":1824772695},{"type":"v1","addr":"172.21.15.46:6807","nonce":1824772695}]},"public_addr":"172.21.15.46:6803/1824772695","cluster_addr":"172.21.15.46:6805/1824772695","heartbeat_back_addr":"172.21.15.46:6809/1824772695","heartbeat_front_addr":"172.21.15.46:6807/1824772695","state":["exists","up"]},{"osd":1,"uuid":"be56f883-411e-4dbe-b50e-a158134df4a6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6800","nonce":973693922},{"type":"v1","addr":"172.21.15.142:6801","nonce":973693922}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6802","nonce":973693922},{"type":"v1","addr":"172.21.15.142:6803","nonce":973693922}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6806","nonce":973693922},{"type":"v1","addr":"172.21.15.142:6807","nonce":973693922}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6804","nonce":973693922},{"type":"v1","addr":"172.21.15.142:6805","nonce":973693922}]},"public_addr":"172.21.15.142:6801/973693922","cluster_addr":"172.21.15.142:6803/973693922","heartbeat_back_addr":"172.21.15.142:6807/973693922","heartbeat_front_addr":"172.21.15.142:6805/973693922","state":["exists","up"]},{"osd":2,"uuid":"c99a8794-ab19-4260-9a8d-4bc0bd2c8175","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6800","nonce":4084261813},{"type":"v1","addr":"172.21.15.162:6801","nonce":4084261813}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6802","nonce":4084261813},{"type":"v1","addr":"172.21.15.162:6803","nonce":4084261813}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6806","nonce":4084261813},{"type":"v1","addr":"172.21.15.162:6807","nonce":4084261813}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6804","nonce":4084261813},{"type":"v1","addr":"172.21.15.162:6805","nonce":4084261813}]},"public_addr":"172.21.15.162:6801/4084261813","cluster_addr":"172.21.15.162:6803/4084261813","heartbeat_back_addr":"172.21.15.162:6807/4084261813","heartbeat_front_addr":"172.21.15.162:6805/4084261813","state":["exists","up"]},{"osd":3,"uuid":"f877814e-9ef6-4872-a081-5609bb9ebc06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":3496317970},{"type":"v1","addr":"172.21.15.46:6811","nonce":3496317970}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":3496317970},{"type":"v1","addr":"172.21.15.46:6813","nonce":3496317970}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":3496317970},{"type":"v1","addr":"172.21.15.46:6817","nonce":3496317970}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":3496317970},{"type":"v1","addr":"172.21.15.46:6815","nonce":3496317970}]},"public_addr":"172.21.15.46:6811/3496317970","cluster_addr":"172.21.15.46:6813/3496317970","heartbeat_back_addr":"172.21.15.46:6817/3496317970","heartbeat_front_addr":"172.21.15.46:6815/3496317970","state":["exists","up"]},{"osd":4,"uuid":"beecec0a-8280-44fe-8f5f-8498a35eca8f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6808","nonce":2996266929},{"type":"v1","addr":"172.21.15.142:6809","nonce":2996266929}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6810","nonce":2996266929},{"type":"v1","addr":"172.21.15.142:6811","nonce":2996266929}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6814","nonce":2996266929},{"type":"v1","addr":"172.21.15.142:6815","nonce":2996266929}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6812","nonce":2996266929},{"type":"v1","addr":"172.21.15.142:6813","nonce":2996266929}]},"public_addr":"172.21.15.142:6809/2996266929","cluster_addr":"172.21.15.142:6811/2996266929","heartbeat_back_addr":"172.21.15.142:6815/2996266929","heartbeat_front_addr":"172.21.15.142:6813/2996266929","state":["exists","up"]},{"osd":5,"uuid":"9da7ff75-f658-45ea-a02f-1943fdda76b9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6808","nonce":914208293},{"type":"v1","addr":"172.21.15.162:6809","nonce":914208293}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6810","nonce":914208293},{"type":"v1","addr":"172.21.15.162:6811","nonce":914208293}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6814","nonce":914208293},{"type":"v1","addr":"172.21.15.162:6815","nonce":914208293}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6812","nonce":914208293},{"type":"v1","addr":"172.21.15.162:6813","nonce":914208293}]},"public_addr":"172.21.15.162:6809/914208293","cluster_addr":"172.21.15.162:6811/914208293","heartbeat_back_addr":"172.21.15.162:6815/914208293","heartbeat_front_addr":"172.21.15.162:6813/914208293","state":["exists","up"]},{"osd":6,"uuid":"8fa11224-cb26-4d3f-87d1-6a8f31de2a19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":56291966},{"type":"v1","addr":"172.21.15.46:6819","nonce":56291966}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":56291966},{"type":"v1","addr":"172.21.15.46:6821","nonce":56291966}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6824","nonce":56291966},{"type":"v1","addr":"172.21.15.46:6825","nonce":56291966}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":56291966},{"type":"v1","addr":"172.21.15.46:6823","nonce":56291966}]},"public_addr":"172.21.15.46:6819/56291966","cluster_addr":"172.21.15.46:6821/56291966","heartbeat_back_addr":"172.21.15.46:6825/56291966","heartbeat_front_addr":"172.21.15.46:6823/56291966","state":["exists","up"]},{"osd":7,"uuid":"0d0338b7-4d0a-4bb0-b152-8f5224871d9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6816","nonce":2146598446},{"type":"v1","addr":"172.21.15.142:6817","nonce":2146598446}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6818","nonce":2146598446},{"type":"v1","addr":"172.21.15.142:6819","nonce":2146598446}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6822","nonce":2146598446},{"type":"v1","addr":"172.21.15.142:6823","nonce":2146598446}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6820","nonce":2146598446},{"type":"v1","addr":"172.21.15.142:6821","nonce":2146598446}]},"public_addr":"172.21.15.142:6817/2146598446","cluster_addr":"172.21.15.142:6819/2146598446","heartbeat_back_addr":"172.21.15.142:6823/2146598446","heartbeat_front_addr":"172.21.15.142:6821/2146598446","state":["exists","up"]},{"osd":8,"uuid":"8b6f6037-892a-416c-babb-7552648d0b3a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6816","nonce":1173078318},{"type":"v1","addr":"172.21.15.162:6817","nonce":1173078318}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6818","nonce":1173078318},{"type":"v1","addr":"172.21.15.162:6819","nonce":1173078318}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6822","nonce":1173078318},{"type":"v1","addr":"172.21.15.162:6823","nonce":1173078318}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6820","nonce":1173078318},{"type":"v1","addr":"172.21.15.162:6821","nonce":1173078318}]},"public_addr":"172.21.15.162:6817/1173078318","cluster_addr":"172.21.15.162:6819/1173078318","heartbeat_back_addr":"172.21.15.162:6823/1173078318","heartbeat_front_addr":"172.21.15.162:6821/1173078318","state":["exists","up"]},{"osd":9,"uuid":"faccce25-6bf2-4220-8858-cda82b72be68","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":532708881},{"type":"v1","addr":"172.21.15.46:6827","nonce":532708881}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6828","nonce":532708881},{"type":"v1","addr":"172.21.15.46:6829","nonce":532708881}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":532708881},{"type":"v1","addr":"172.21.15.46:6833","nonce":532708881}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":532708881},{"type":"v1","addr":"172.21.15.46:6831","nonce":532708881}]},"public_addr":"172.21.15.46:6827/532708881","cluster_addr":"172.21.15.46:6829/532708881","heartbeat_back_addr":"172.21.15.46:6833/532708881","heartbeat_front_addr":"172.21.15.46:6831/532708881","state":["exists","up"]},{"osd":10,"uuid":"cdd356db-f9eb-4080-95ab-db5d8223e572","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6824","nonce":2845222366},{"type":"v1","addr":"172.21.15.142:6825","nonce":2845222366}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6826","nonce":2845222366},{"type":"v1","addr":"172.21.15.142:6827","nonce":2845222366}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6830","nonce":2845222366},{"type":"v1","addr":"172.21.15.142:6831","nonce":2845222366}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6828","nonce":2845222366},{"type":"v1","addr":"172.21.15.142:6829","nonce":2845222366}]},"public_addr":"172.21.15.142:6825/2845222366","cluster_addr":"172.21.15.142:6827/2845222366","heartbeat_back_addr":"172.21.15.142:6831/2845222366","heartbeat_front_addr":"172.21.15.142:6829/2845222366","state":["exists","up"]},{"osd":11,"uuid":"72104d1c-3ea2-45ad-bc0b-0500f146c06b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6824","nonce":1986323778},{"type":"v1","addr":"172.21.15.162:6825","nonce":1986323778}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6826","nonce":1986323778},{"type":"v1","addr":"172.21.15.162:6827","nonce":1986323778}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6830","nonce":1986323778},{"type":"v1","addr":"172.21.15.162:6831","nonce":1986323778}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6828","nonce":1986323778},{"type":"v1","addr":"172.21.15.162:6829","nonce":1986323778}]},"public_addr":"172.21.15.162:6825/1986323778","cluster_addr":"172.21.15.162:6827/1986323778","heartbeat_back_addr":"172.21.15.162:6831/1986323778","heartbeat_front_addr":"172.21.15.162:6829/1986323778","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:03:44.996331+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:04:07.994738+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:04:30.942255+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:04:55.588884+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:05:19.900600+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:05:43.807442+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:06:09.026976+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:06:34.586784+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:06:59.751605+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:07:26.144495+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:07:52.326909+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:08:17.423042+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.b","mappings":[{"from":8,"to":11}]},{"pgid":"3.6","mappings":[{"from":4,"to":7}]},{"pgid":"3.c","mappings":[{"from":3,"to":9}]},{"pgid":"3.1a","mappings":[{"from":5,"to":6}]},{"pgid":"3.1b","mappings":[{"from":4,"to":8}]},{"pgid":"3.1d","mappings":[{"from":4,"to":8}]},{"pgid":"3.2a","mappings":[{"from":3,"to":6}]},{"pgid":"3.33","mappings":[{"from":5,"to":9}]},{"pgid":"3.36","mappings":[{"from":4,"to":9}]},{"pgid":"4.6","mappings":[{"from":5,"to":8}]},{"pgid":"4.17","mappings":[{"from":9,"to":8}]},{"pgid":"4.18","mappings":[{"from":9,"to":8}]},{"pgid":"4.1b","mappings":[{"from":9,"to":10}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.46:0/4021039857":"2024-04-01T23:02:08.493619+0000","172.21.15.46:6801/3681182647":"2024-04-01T23:02:08.493619+0000","172.21.15.46:0/4022214806":"2024-04-01T23:02:08.493619+0000","172.21.15.46:6800/1515528215":"2024-04-01T23:01:41.910162+0000","172.21.15.46:6800/3681182647":"2024-04-01T23:02:08.493619+0000","172.21.15.46:0/143885707":"2024-04-01T23:01:41.910162+0000","172.21.15.46:6801/1515528215":"2024-04-01T23:01:41.910162+0000","172.21.15.46:0/1680389778":"2024-04-01T23:01:41.910162+0000","172.21.15.46:0/1782133966":"2024-04-01T23:01:41.910162+0000","172.21.15.46:0/3925209376":"2024-04-01T23:02:08.493619+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-31T23:11:29.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:29.468+0000 7f53c2eae640 1 -- 172.21.15.46:0/136660239 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f53ac0a24a0 msgr2=0x7f53ac0a4960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:29.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:29.468+0000 7f53c2eae640 1 --2- 172.21.15.46:0/136660239 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f53ac0a24a0 0x7f53ac0a4960 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f53bc197d00 tx=0x7f53a8005f90 comp rx=0 tx=0).stop 2024-03-31T23:11:29.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:29.468+0000 7f53c2eae640 1 -- 172.21.15.46:0/136660239 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f53bc100670 msgr2=0x7f53bc1967f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:29.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:29.468+0000 7f53c2eae640 1 --2- 172.21.15.46:0/136660239 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f53bc100670 0x7f53bc1967f0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f53bc1016c0 tx=0x7f53b000ee50 comp rx=0 tx=0).stop 2024-03-31T23:11:29.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:29.468+0000 7f53c2eae640 1 -- 172.21.15.46:0/136660239 shutdown_connections 2024-03-31T23:11:29.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:29.468+0000 7f53c2eae640 1 --2- 172.21.15.46:0/136660239 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f53ac0a24a0 0x7f53ac0a4960 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:29.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:29.468+0000 7f53c2eae640 1 --2- 172.21.15.46:0/136660239 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f53bc102220 0x7f53bc19dda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:29.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:29.468+0000 7f53c2eae640 1 --2- 172.21.15.46:0/136660239 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f53bc101860 0x7f53bc196d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:29.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:29.468+0000 7f53c2eae640 1 --2- 172.21.15.46:0/136660239 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f53bc100670 0x7f53bc1967f0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:29.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:29.468+0000 7f53c2eae640 1 -- 172.21.15.46:0/136660239 >> 172.21.15.46:0/136660239 conn(0x7f53bc0fbbe0 msgr2=0x7f53bc104b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:11:29.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:29.468+0000 7f53c2eae640 1 -- 172.21.15.46:0/136660239 shutdown_connections 2024-03-31T23:11:29.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:29.468+0000 7f53c2eae640 1 -- 172.21.15.46:0/136660239 wait complete. 2024-03-31T23:11:29.479 INFO:tasks.fwd_scrub.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-31T23:04:32.823751+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-31T23:09:09.145711+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-31T23:09:10.148579+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-31T23:09:13.419519+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '91', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-31T23:11:29.480 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-31T23:11:29.588 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:29.586+0000 7fae2f20f640 1 -- 172.21.15.46:0/214548877 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fae28104b00 msgr2=0x7fae28106ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:29.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:29.586+0000 7fae2f20f640 1 --2- 172.21.15.46:0/214548877 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fae28104b00 0x7fae28106ef0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7fae14009900 tx=0x7fae1402bc30 comp rx=0 tx=0).stop 2024-03-31T23:11:29.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:29.586+0000 7fae2f20f640 1 -- 172.21.15.46:0/214548877 shutdown_connections 2024-03-31T23:11:29.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:29.586+0000 7fae2f20f640 1 --2- 172.21.15.46:0/214548877 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fae28107430 0x7fae28109880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:29.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:29.586+0000 7fae2f20f640 1 --2- 172.21.15.46:0/214548877 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fae28104b00 0x7fae28106ef0 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:29.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:29.586+0000 7fae2f20f640 1 --2- 172.21.15.46:0/214548877 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fae281021d0 0x7fae281045c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:29.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:29.586+0000 7fae2f20f640 1 -- 172.21.15.46:0/214548877 >> 172.21.15.46:0/214548877 conn(0x7fae280fbbc0 msgr2=0x7fae280fe000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:11:29.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:29.586+0000 7fae2f20f640 1 -- 172.21.15.46:0/214548877 shutdown_connections 2024-03-31T23:11:29.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:29.586+0000 7fae2f20f640 1 -- 172.21.15.46:0/214548877 wait complete. 2024-03-31T23:11:29.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:29.586+0000 7fae2f20f640 1 Processor -- start 2024-03-31T23:11:29.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:29.586+0000 7fae2f20f640 1 -- start start 2024-03-31T23:11:29.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:29.586+0000 7fae2f20f640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fae281021d0 0x7fae281979a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:29.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:29.586+0000 7fae2f20f640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fae28104b00 0x7fae28197ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:29.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:29.586+0000 7fae2f20f640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fae28107430 0x7fae2819ef50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:29.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:29.586+0000 7fae2f20f640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fae2810d270 con 0x7fae281021d0 2024-03-31T23:11:29.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:29.586+0000 7fae2f20f640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fae2810d0f0 con 0x7fae28104b00 2024-03-31T23:11:29.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:29.586+0000 7fae2f20f640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fae2810d3f0 con 0x7fae28107430 2024-03-31T23:11:29.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:29.586+0000 7fae2cfab640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fae281021d0 0x7fae281979a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:29.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:29.586+0000 7fae2cfab640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fae281021d0 0x7fae281979a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:58448/0 (socket says 172.21.15.46:58448) 2024-03-31T23:11:29.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:29.587+0000 7fae2cfab640 1 -- 172.21.15.46:0/3337492262 learned_addr learned my addr 172.21.15.46:0/3337492262 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:11:29.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:29.587+0000 7fae27fff640 1 --2- 172.21.15.46:0/3337492262 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fae28104b00 0x7fae28197ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:29.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:29.587+0000 7fae2d7ac640 1 --2- 172.21.15.46:0/3337492262 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fae28107430 0x7fae2819ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:29.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:29.587+0000 7fae2cfab640 1 -- 172.21.15.46:0/3337492262 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fae28107430 msgr2=0x7fae2819ef50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:29.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:29.587+0000 7fae2cfab640 1 --2- 172.21.15.46:0/3337492262 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fae28107430 0x7fae2819ef50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:29.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:29.587+0000 7fae2cfab640 1 -- 172.21.15.46:0/3337492262 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fae28104b00 msgr2=0x7fae28197ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:29.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:29.587+0000 7fae2cfab640 1 --2- 172.21.15.46:0/3337492262 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fae28104b00 0x7fae28197ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:29.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:29.587+0000 7fae2cfab640 1 -- 172.21.15.46:0/3337492262 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fae2819f5c0 con 0x7fae281021d0 2024-03-31T23:11:29.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:29.587+0000 7fae2d7ac640 1 --2- 172.21.15.46:0/3337492262 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fae28107430 0x7fae2819ef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T23:11:29.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:29.587+0000 7fae2cfab640 1 --2- 172.21.15.46:0/3337492262 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fae281021d0 0x7fae281979a0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7fae1c00e9f0 tx=0x7fae1c00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:11:29.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:29.587+0000 7fae25ffb640 1 -- 172.21.15.46:0/3337492262 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae1c00cd80 con 0x7fae281021d0 2024-03-31T23:11:29.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:29.587+0000 7fae25ffb640 1 -- 172.21.15.46:0/3337492262 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fae1c0108e0 con 0x7fae281021d0 2024-03-31T23:11:29.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:29.587+0000 7fae2f20f640 1 -- 172.21.15.46:0/3337492262 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fae2819f8b0 con 0x7fae281021d0 2024-03-31T23:11:29.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:29.587+0000 7fae25ffb640 1 -- 172.21.15.46:0/3337492262 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae1c018920 con 0x7fae281021d0 2024-03-31T23:11:29.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:29.587+0000 7fae2f20f640 1 -- 172.21.15.46:0/3337492262 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fae2819fdd0 con 0x7fae281021d0 2024-03-31T23:11:29.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:29.588+0000 7fae2f20f640 1 -- 172.21.15.46:0/3337492262 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fae280fd6b0 con 0x7fae281021d0 2024-03-31T23:11:29.592 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:29.589+0000 7fae25ffb640 1 -- 172.21.15.46:0/3337492262 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 18) ==== 137320+0+0 (secure 0 0 0) 0x7fae1c010a90 con 0x7fae281021d0 2024-03-31T23:11:29.592 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:29.590+0000 7fae25ffb640 1 --2- 172.21.15.46:0/3337492262 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fae180a24a0 0x7fae180a4960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:29.592 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:29.590+0000 7fae25ffb640 1 -- 172.21.15.46:0/3337492262 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9208+0+0 (secure 0 0 0) 0x7fae1c0cee70 con 0x7fae281021d0 2024-03-31T23:11:29.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:29.590+0000 7fae27fff640 1 --2- 172.21.15.46:0/3337492262 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fae180a24a0 0x7fae180a4960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:29.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:29.590+0000 7fae27fff640 1 --2- 172.21.15.46:0/3337492262 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fae180a24a0 0x7fae180a4960 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fae14005ab0 tx=0x7fae14005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:11:29.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:29.594+0000 7fae25ffb640 1 -- 172.21.15.46:0/3337492262 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fae1c09bc60 con 0x7fae281021d0 2024-03-31T23:11:29.776 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:29.775+0000 7fae2f20f640 1 -- 172.21.15.46:0/3337492262 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fae28062380 con 0x7fae281021d0 2024-03-31T23:11:29.776 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:29.776+0000 7fae25ffb640 1 -- 172.21.15.46:0/3337492262 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v95) ==== 93+0+10 (secure 0 0 0) 0x7fae1c09b560 con 0x7fae281021d0 2024-03-31T23:11:29.776 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 1 2024-03-31T23:11:29.778 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:29.778+0000 7fae2f20f640 1 -- 172.21.15.46:0/3337492262 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fae180a24a0 msgr2=0x7fae180a4960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:29.778 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:29.778+0000 7fae2f20f640 1 --2- 172.21.15.46:0/3337492262 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fae180a24a0 0x7fae180a4960 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fae14005ab0 tx=0x7fae14005fd0 comp rx=0 tx=0).stop 2024-03-31T23:11:29.779 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:29.778+0000 7fae2f20f640 1 -- 172.21.15.46:0/3337492262 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fae281021d0 msgr2=0x7fae281979a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:29.779 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:29.778+0000 7fae2f20f640 1 --2- 172.21.15.46:0/3337492262 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fae281021d0 0x7fae281979a0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7fae1c00e9f0 tx=0x7fae1c00eeb0 comp rx=0 tx=0).stop 2024-03-31T23:11:29.779 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:29.778+0000 7fae2f20f640 1 -- 172.21.15.46:0/3337492262 shutdown_connections 2024-03-31T23:11:29.779 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:29.778+0000 7fae2f20f640 1 --2- 172.21.15.46:0/3337492262 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fae180a24a0 0x7fae180a4960 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:29.779 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:29.778+0000 7fae2f20f640 1 --2- 172.21.15.46:0/3337492262 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fae28107430 0x7fae2819ef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:29.779 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:29.778+0000 7fae2f20f640 1 --2- 172.21.15.46:0/3337492262 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fae28104b00 0x7fae28197ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:29.779 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:29.778+0000 7fae2f20f640 1 --2- 172.21.15.46:0/3337492262 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fae281021d0 0x7fae281979a0 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:29.779 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:29.778+0000 7fae2f20f640 1 -- 172.21.15.46:0/3337492262 >> 172.21.15.46:0/3337492262 conn(0x7fae280fbbc0 msgr2=0x7fae28104020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:11:29.779 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:29.778+0000 7fae2f20f640 1 -- 172.21.15.46:0/3337492262 shutdown_connections 2024-03-31T23:11:29.779 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:29.778+0000 7fae2f20f640 1 -- 172.21.15.46:0/3337492262 wait complete. 2024-03-31T23:11:29.788 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:11:29 smithi046 ceph-mon[32498]: from='client.? 172.21.15.46:0/2854046415' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-31T23:11:29.788 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:11:29 smithi046 ceph-mon[32498]: from='client.? 172.21.15.46:0/1073321381' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-31T23:11:29.788 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:11:29 smithi046 ceph-mon[32498]: from='client.? 172.21.15.46:0/136660239' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-31T23:11:29.789 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-31T23:11:29.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:29.895+0000 7f22e7b56640 1 -- 172.21.15.46:0/1756430185 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f22e0107440 msgr2=0x7f22e0109890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:29.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:29.895+0000 7f22e7b56640 1 --2- 172.21.15.46:0/1756430185 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f22e0107440 0x7f22e0109890 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f22d8009ca0 tx=0x7f22d802c0f0 comp rx=0 tx=0).stop 2024-03-31T23:11:29.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:29.895+0000 7f22e7b56640 1 -- 172.21.15.46:0/1756430185 shutdown_connections 2024-03-31T23:11:29.898 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:29.895+0000 7f22e7b56640 1 --2- 172.21.15.46:0/1756430185 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f22e0107440 0x7f22e0109890 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:29.898 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:29.895+0000 7f22e7b56640 1 --2- 172.21.15.46:0/1756430185 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f22e0104b10 0x7f22e0106f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:29.898 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:29.895+0000 7f22e7b56640 1 --2- 172.21.15.46:0/1756430185 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f22e01021e0 0x7f22e01045d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:29.898 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:29.895+0000 7f22e7b56640 1 -- 172.21.15.46:0/1756430185 >> 172.21.15.46:0/1756430185 conn(0x7f22e00fbbf0 msgr2=0x7f22e00fe010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:11:29.898 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:29.895+0000 7f22e7b56640 1 -- 172.21.15.46:0/1756430185 shutdown_connections 2024-03-31T23:11:29.898 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:29.895+0000 7f22e7b56640 1 -- 172.21.15.46:0/1756430185 wait complete. 2024-03-31T23:11:29.898 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:29.896+0000 7f22e7b56640 1 Processor -- start 2024-03-31T23:11:29.898 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:29.896+0000 7f22e7b56640 1 -- start start 2024-03-31T23:11:29.898 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:29.896+0000 7f22e7b56640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f22e01021e0 0x7f22e01979e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:29.898 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:29.896+0000 7f22e7b56640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f22e0104b10 0x7f22e0197f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:29.898 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:29.896+0000 7f22e7b56640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f22e0107440 0x7f22e019ef90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:29.898 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:29.896+0000 7f22e7b56640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f22e010d2b0 con 0x7f22e01021e0 2024-03-31T23:11:29.898 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:29.896+0000 7f22e7b56640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f22e010d130 con 0x7f22e0104b10 2024-03-31T23:11:29.899 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:29.896+0000 7f22e7b56640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f22e010d430 con 0x7f22e0107440 2024-03-31T23:11:29.899 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:29.896+0000 7f22e58f2640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f22e01021e0 0x7f22e01979e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:29.899 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:29.896+0000 7f22e58f2640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f22e01021e0 0x7f22e01979e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:58464/0 (socket says 172.21.15.46:58464) 2024-03-31T23:11:29.899 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:29.896+0000 7f22e58f2640 1 -- 172.21.15.46:0/3255428415 learned_addr learned my addr 172.21.15.46:0/3255428415 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:11:29.899 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:29.896+0000 7f22e60f3640 1 --2- 172.21.15.46:0/3255428415 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f22e0107440 0x7f22e019ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:29.899 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:29.896+0000 7f22e50f1640 1 --2- 172.21.15.46:0/3255428415 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f22e0104b10 0x7f22e0197f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:29.899 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:29.896+0000 7f22e58f2640 1 -- 172.21.15.46:0/3255428415 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f22e0107440 msgr2=0x7f22e019ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:29.899 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:29.896+0000 7f22e58f2640 1 --2- 172.21.15.46:0/3255428415 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f22e0107440 0x7f22e019ef90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:29.899 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:29.896+0000 7f22e58f2640 1 -- 172.21.15.46:0/3255428415 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f22e0104b10 msgr2=0x7f22e0197f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:29.899 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:29.896+0000 7f22e58f2640 1 --2- 172.21.15.46:0/3255428415 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f22e0104b10 0x7f22e0197f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:29.899 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:29.896+0000 7f22e58f2640 1 -- 172.21.15.46:0/3255428415 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22e019f600 con 0x7f22e01021e0 2024-03-31T23:11:29.899 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:29.896+0000 7f22e50f1640 1 --2- 172.21.15.46:0/3255428415 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f22e0104b10 0x7f22e0197f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T23:11:29.899 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:29.896+0000 7f22e58f2640 1 --2- 172.21.15.46:0/3255428415 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f22e01021e0 0x7f22e01979e0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f22dc00e9f0 tx=0x7f22dc00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:11:29.899 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:29.897+0000 7f22d6ffd640 1 -- 172.21.15.46:0/3255428415 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22dc00cd80 con 0x7f22e01021e0 2024-03-31T23:11:29.900 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:29.897+0000 7f22d6ffd640 1 -- 172.21.15.46:0/3255428415 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f22dc0108e0 con 0x7f22e01021e0 2024-03-31T23:11:29.900 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:29.897+0000 7f22e7b56640 1 -- 172.21.15.46:0/3255428415 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f22e019f8f0 con 0x7f22e01021e0 2024-03-31T23:11:29.900 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:29.897+0000 7f22d6ffd640 1 -- 172.21.15.46:0/3255428415 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22dc018920 con 0x7f22e01021e0 2024-03-31T23:11:29.900 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:29.897+0000 7f22e7b56640 1 -- 172.21.15.46:0/3255428415 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f22e019fe90 con 0x7f22e01021e0 2024-03-31T23:11:29.900 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:29.898+0000 7f22e7b56640 1 -- 172.21.15.46:0/3255428415 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f22e00fd6e0 con 0x7f22e01021e0 2024-03-31T23:11:29.901 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:29.899+0000 7f22d6ffd640 1 -- 172.21.15.46:0/3255428415 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 18) ==== 137320+0+0 (secure 0 0 0) 0x7f22dc010a90 con 0x7f22e01021e0 2024-03-31T23:11:29.901 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:29.899+0000 7f22d6ffd640 1 --2- 172.21.15.46:0/3255428415 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f22cc0a2520 0x7f22cc0a49e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:29.901 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:29.899+0000 7f22e50f1640 1 --2- 172.21.15.46:0/3255428415 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f22cc0a2520 0x7f22cc0a49e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:29.901 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:29.900+0000 7f22d6ffd640 1 -- 172.21.15.46:0/3255428415 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9208+0+0 (secure 0 0 0) 0x7f22dc0cee00 con 0x7f22e01021e0 2024-03-31T23:11:29.901 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:29.900+0000 7f22e50f1640 1 --2- 172.21.15.46:0/3255428415 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f22cc0a2520 0x7f22cc0a49e0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f22e0198ef0 tx=0x7f22d0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:11:29.904 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:29.904+0000 7f22d6ffd640 1 -- 172.21.15.46:0/3255428415 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f22dc09bbf0 con 0x7f22e01021e0 2024-03-31T23:11:29.946 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:11:29 smithi142 ceph-mon[35271]: from='client.? 172.21.15.46:0/2854046415' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-31T23:11:29.946 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:11:29 smithi142 ceph-mon[35271]: from='client.? 172.21.15.46:0/1073321381' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-31T23:11:29.946 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:11:29 smithi142 ceph-mon[35271]: from='client.? 172.21.15.46:0/136660239' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-31T23:11:30.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:11:29 smithi162 ceph-mon[36237]: from='client.? 172.21.15.46:0/2854046415' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-31T23:11:30.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:11:29 smithi162 ceph-mon[36237]: from='client.? 172.21.15.46:0/1073321381' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-31T23:11:30.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:11:29 smithi162 ceph-mon[36237]: from='client.? 172.21.15.46:0/136660239' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-31T23:11:30.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:30.085+0000 7f22e7b56640 1 -- 172.21.15.46:0/3255428415 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f22e0062350 con 0x7f22e01021e0 2024-03-31T23:11:30.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:30.085+0000 7f22d6ffd640 1 -- 172.21.15.46:0/3255428415 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v95) ==== 104+0+11 (secure 0 0 0) 0x7f22dc09b4f0 con 0x7f22e01021e0 2024-03-31T23:11:30.086 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 32 2024-03-31T23:11:30.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:30.087+0000 7f22e7b56640 1 -- 172.21.15.46:0/3255428415 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f22cc0a2520 msgr2=0x7f22cc0a49e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:30.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:30.087+0000 7f22e7b56640 1 --2- 172.21.15.46:0/3255428415 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f22cc0a2520 0x7f22cc0a49e0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f22e0198ef0 tx=0x7f22d0005f90 comp rx=0 tx=0).stop 2024-03-31T23:11:30.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:30.087+0000 7f22e7b56640 1 -- 172.21.15.46:0/3255428415 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f22e01021e0 msgr2=0x7f22e01979e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:30.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:30.087+0000 7f22e7b56640 1 --2- 172.21.15.46:0/3255428415 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f22e01021e0 0x7f22e01979e0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f22dc00e9f0 tx=0x7f22dc00eeb0 comp rx=0 tx=0).stop 2024-03-31T23:11:30.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:30.087+0000 7f22e7b56640 1 -- 172.21.15.46:0/3255428415 shutdown_connections 2024-03-31T23:11:30.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:30.087+0000 7f22e7b56640 1 --2- 172.21.15.46:0/3255428415 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f22cc0a2520 0x7f22cc0a49e0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:30.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:30.087+0000 7f22e7b56640 1 --2- 172.21.15.46:0/3255428415 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f22e0107440 0x7f22e019ef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:30.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:30.087+0000 7f22e7b56640 1 --2- 172.21.15.46:0/3255428415 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f22e0104b10 0x7f22e0197f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:30.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:30.087+0000 7f22e7b56640 1 --2- 172.21.15.46:0/3255428415 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f22e01021e0 0x7f22e01979e0 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:30.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:30.087+0000 7f22e7b56640 1 -- 172.21.15.46:0/3255428415 >> 172.21.15.46:0/3255428415 conn(0x7f22e00fbbf0 msgr2=0x7f22e0102c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:11:30.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:30.087+0000 7f22e7b56640 1 -- 172.21.15.46:0/3255428415 shutdown_connections 2024-03-31T23:11:30.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:30.087+0000 7f22e7b56640 1 -- 172.21.15.46:0/3255428415 wait complete. 2024-03-31T23:11:30.098 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-31T23:11:30.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:30.204+0000 7f3a4c0d6640 1 -- 172.21.15.46:0/3306157717 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3a44102200 msgr2=0x7f3a44108a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:30.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:30.204+0000 7f3a4c0d6640 1 --2- 172.21.15.46:0/3306157717 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3a44102200 0x7f3a44108a90 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f3a40009ca0 tx=0x7f3a4002c0f0 comp rx=0 tx=0).stop 2024-03-31T23:11:30.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:30.204+0000 7f3a4c0d6640 1 -- 172.21.15.46:0/3306157717 shutdown_connections 2024-03-31T23:11:30.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:30.204+0000 7f3a4c0d6640 1 --2- 172.21.15.46:0/3306157717 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3a44102200 0x7f3a44108a90 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:30.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:30.204+0000 7f3a4c0d6640 1 --2- 172.21.15.46:0/3306157717 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3a44101840 0x7f3a44101cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:30.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:30.204+0000 7f3a4c0d6640 1 --2- 172.21.15.46:0/3306157717 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3a44100650 0x7f3a44100a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:30.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:30.204+0000 7f3a4c0d6640 1 -- 172.21.15.46:0/3306157717 >> 172.21.15.46:0/3306157717 conn(0x7f3a440fbbe0 msgr2=0x7f3a440fe000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:11:30.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:30.204+0000 7f3a4c0d6640 1 -- 172.21.15.46:0/3306157717 shutdown_connections 2024-03-31T23:11:30.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:30.204+0000 7f3a4c0d6640 1 -- 172.21.15.46:0/3306157717 wait complete. 2024-03-31T23:11:30.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:30.205+0000 7f3a4c0d6640 1 Processor -- start 2024-03-31T23:11:30.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:30.205+0000 7f3a4c0d6640 1 -- start start 2024-03-31T23:11:30.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:30.205+0000 7f3a4c0d6640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3a44100650 0x7f3a441978e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:30.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:30.205+0000 7f3a4c0d6640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3a44101840 0x7f3a44197e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:30.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:30.205+0000 7f3a4c0d6640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3a44102200 0x7f3a4419ee90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:30.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:30.205+0000 7f3a4c0d6640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f3a4410d290 con 0x7f3a44102200 2024-03-31T23:11:30.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:30.205+0000 7f3a4c0d6640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f3a4410d110 con 0x7f3a44100650 2024-03-31T23:11:30.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:30.205+0000 7f3a4c0d6640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f3a4410d410 con 0x7f3a44101840 2024-03-31T23:11:30.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:30.205+0000 7f3a4a673640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3a44102200 0x7f3a4419ee90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:30.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:30.205+0000 7f3a4a673640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3a44102200 0x7f3a4419ee90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:58488/0 (socket says 172.21.15.46:58488) 2024-03-31T23:11:30.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:30.205+0000 7f3a49e72640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3a44100650 0x7f3a441978e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:30.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:30.205+0000 7f3a4a673640 1 -- 172.21.15.46:0/3060866992 learned_addr learned my addr 172.21.15.46:0/3060866992 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:11:30.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:30.205+0000 7f3a49671640 1 --2- 172.21.15.46:0/3060866992 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3a44101840 0x7f3a44197e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:30.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:30.205+0000 7f3a4a673640 1 -- 172.21.15.46:0/3060866992 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3a44101840 msgr2=0x7f3a44197e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:30.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:30.205+0000 7f3a4a673640 1 --2- 172.21.15.46:0/3060866992 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3a44101840 0x7f3a44197e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:30.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:30.205+0000 7f3a4a673640 1 -- 172.21.15.46:0/3060866992 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3a44100650 msgr2=0x7f3a441978e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:30.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:30.205+0000 7f3a4a673640 1 --2- 172.21.15.46:0/3060866992 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3a44100650 0x7f3a441978e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:30.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:30.205+0000 7f3a4a673640 1 -- 172.21.15.46:0/3060866992 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a4419f500 con 0x7f3a44102200 2024-03-31T23:11:30.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:30.205+0000 7f3a49e72640 1 --2- 172.21.15.46:0/3060866992 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3a44100650 0x7f3a441978e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:11:30.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:30.205+0000 7f3a4a673640 1 --2- 172.21.15.46:0/3060866992 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3a44102200 0x7f3a4419ee90 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f3a4002c700 tx=0x7f3a400328a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:11:30.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:30.206+0000 7f3a3affd640 1 -- 172.21.15.46:0/3060866992 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a40041020 con 0x7f3a44102200 2024-03-31T23:11:30.209 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:30.206+0000 7f3a4c0d6640 1 -- 172.21.15.46:0/3060866992 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3a4419f790 con 0x7f3a44102200 2024-03-31T23:11:30.209 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:30.206+0000 7f3a3affd640 1 -- 172.21.15.46:0/3060866992 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3a40003e20 con 0x7f3a44102200 2024-03-31T23:11:30.209 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:30.206+0000 7f3a3affd640 1 -- 172.21.15.46:0/3060866992 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a40039040 con 0x7f3a44102200 2024-03-31T23:11:30.209 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:30.206+0000 7f3a4c0d6640 1 -- 172.21.15.46:0/3060866992 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3a4419fc80 con 0x7f3a44102200 2024-03-31T23:11:30.210 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:30.207+0000 7f3a4c0d6640 1 -- 172.21.15.46:0/3060866992 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3a441079f0 con 0x7f3a44102200 2024-03-31T23:11:30.210 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:30.208+0000 7f3a3affd640 1 -- 172.21.15.46:0/3060866992 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 18) ==== 137320+0+0 (secure 0 0 0) 0x7f3a40044020 con 0x7f3a44102200 2024-03-31T23:11:30.210 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:30.208+0000 7f3a3affd640 1 --2- 172.21.15.46:0/3060866992 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f3a300a24a0 0x7f3a300a4960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:30.210 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:30.208+0000 7f3a49e72640 1 --2- 172.21.15.46:0/3060866992 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f3a300a24a0 0x7f3a300a4960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:30.210 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:30.208+0000 7f3a3affd640 1 -- 172.21.15.46:0/3060866992 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9208+0+0 (secure 0 0 0) 0x7f3a400f1060 con 0x7f3a44102200 2024-03-31T23:11:30.210 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:30.209+0000 7f3a49e72640 1 --2- 172.21.15.46:0/3060866992 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f3a300a24a0 0x7f3a300a4960 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f3a441016a0 tx=0x7f3a3c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:11:30.212 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:30.212+0000 7f3a3affd640 1 -- 172.21.15.46:0/3060866992 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f3a4004b020 con 0x7f3a44102200 2024-03-31T23:11:30.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:30.395+0000 7f3a4c0d6640 1 -- 172.21.15.46:0/3060866992 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f3a4402d170 con 0x7f3a44102200 2024-03-31T23:11:30.396 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:30.395+0000 7f3a3affd640 1 -- 172.21.15.46:0/3060866992 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v95) ==== 100+0+11 (secure 0 0 0) 0x7f3a400bde50 con 0x7f3a44102200 2024-03-31T23:11:30.396 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 64 2024-03-31T23:11:30.398 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:30.397+0000 7f3a4c0d6640 1 -- 172.21.15.46:0/3060866992 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f3a300a24a0 msgr2=0x7f3a300a4960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:30.398 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:30.397+0000 7f3a4c0d6640 1 --2- 172.21.15.46:0/3060866992 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f3a300a24a0 0x7f3a300a4960 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f3a441016a0 tx=0x7f3a3c005f90 comp rx=0 tx=0).stop 2024-03-31T23:11:30.398 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:30.397+0000 7f3a4c0d6640 1 -- 172.21.15.46:0/3060866992 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3a44102200 msgr2=0x7f3a4419ee90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:30.398 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:30.397+0000 7f3a4c0d6640 1 --2- 172.21.15.46:0/3060866992 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3a44102200 0x7f3a4419ee90 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f3a4002c700 tx=0x7f3a400328a0 comp rx=0 tx=0).stop 2024-03-31T23:11:30.398 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:30.398+0000 7f3a4c0d6640 1 -- 172.21.15.46:0/3060866992 shutdown_connections 2024-03-31T23:11:30.398 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:30.398+0000 7f3a4c0d6640 1 --2- 172.21.15.46:0/3060866992 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f3a300a24a0 0x7f3a300a4960 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:30.398 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:30.398+0000 7f3a4c0d6640 1 --2- 172.21.15.46:0/3060866992 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3a44102200 0x7f3a4419ee90 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:30.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:30.398+0000 7f3a4c0d6640 1 --2- 172.21.15.46:0/3060866992 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3a44101840 0x7f3a44197e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:30.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:30.398+0000 7f3a4c0d6640 1 --2- 172.21.15.46:0/3060866992 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3a44100650 0x7f3a441978e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:30.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:30.398+0000 7f3a4c0d6640 1 -- 172.21.15.46:0/3060866992 >> 172.21.15.46:0/3060866992 conn(0x7f3a440fbbe0 msgr2=0x7f3a44104bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:11:30.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:30.398+0000 7f3a4c0d6640 1 -- 172.21.15.46:0/3060866992 shutdown_connections 2024-03-31T23:11:30.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:30.398+0000 7f3a4c0d6640 1 -- 172.21.15.46:0/3060866992 wait complete. 2024-03-31T23:11:30.409 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-31T23:11:30.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:30.514+0000 7f0771325640 1 -- 172.21.15.46:0/597676441 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f076c101c80 msgr2=0x7f076c1020a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:30.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:30.514+0000 7f0771325640 1 --2- 172.21.15.46:0/597676441 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f076c101c80 0x7f076c1020a0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f0764009900 tx=0x7f076402bc30 comp rx=0 tx=0).stop 2024-03-31T23:11:30.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:30.514+0000 7f0771325640 1 -- 172.21.15.46:0/597676441 shutdown_connections 2024-03-31T23:11:30.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:30.514+0000 7f0771325640 1 --2- 172.21.15.46:0/597676441 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f076c101290 0x7f076c108b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:30.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:30.514+0000 7f0771325640 1 --2- 172.21.15.46:0/597676441 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f076c1008d0 0x7f076c100d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:30.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:30.514+0000 7f0771325640 1 --2- 172.21.15.46:0/597676441 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f076c101c80 0x7f076c1020a0 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:30.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:30.514+0000 7f0771325640 1 -- 172.21.15.46:0/597676441 >> 172.21.15.46:0/597676441 conn(0x7f076c0fb380 msgr2=0x7f076c0fd7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:11:30.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:30.515+0000 7f0771325640 1 -- 172.21.15.46:0/597676441 shutdown_connections 2024-03-31T23:11:30.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:30.515+0000 7f0771325640 1 -- 172.21.15.46:0/597676441 wait complete. 2024-03-31T23:11:30.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:30.515+0000 7f0771325640 1 Processor -- start 2024-03-31T23:11:30.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:30.515+0000 7f0771325640 1 -- start start 2024-03-31T23:11:30.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:30.515+0000 7f0771325640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f076c1008d0 0x7f076c197a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:30.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:30.515+0000 7f0771325640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f076c101290 0x7f076c197f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:30.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:30.515+0000 7f0771325640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f076c101c80 0x7f076c19efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:30.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:30.515+0000 7f0771325640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f076c10c1b0 con 0x7f076c101290 2024-03-31T23:11:30.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:30.515+0000 7f0771325640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f076c10c030 con 0x7f076c101c80 2024-03-31T23:11:30.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:30.515+0000 7f0771325640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f076c10c330 con 0x7f076c1008d0 2024-03-31T23:11:30.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:30.515+0000 7f076a7fc640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f076c101290 0x7f076c197f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:30.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:30.516+0000 7f076a7fc640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f076c101290 0x7f076c197f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:58506/0 (socket says 172.21.15.46:58506) 2024-03-31T23:11:30.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:30.516+0000 7f076a7fc640 1 -- 172.21.15.46:0/4229979867 learned_addr learned my addr 172.21.15.46:0/4229979867 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:11:30.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:30.516+0000 7f076b7fe640 1 --2- 172.21.15.46:0/4229979867 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f076c101c80 0x7f076c19efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:30.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:30.516+0000 7f076affd640 1 --2- 172.21.15.46:0/4229979867 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f076c1008d0 0x7f076c197a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:30.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:30.516+0000 7f076a7fc640 1 -- 172.21.15.46:0/4229979867 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f076c1008d0 msgr2=0x7f076c197a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:30.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:30.516+0000 7f076a7fc640 1 --2- 172.21.15.46:0/4229979867 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f076c1008d0 0x7f076c197a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:30.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:30.516+0000 7f076a7fc640 1 -- 172.21.15.46:0/4229979867 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f076c101c80 msgr2=0x7f076c19efb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:30.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:30.516+0000 7f076a7fc640 1 --2- 172.21.15.46:0/4229979867 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f076c101c80 0x7f076c19efb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:30.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:30.516+0000 7f076a7fc640 1 -- 172.21.15.46:0/4229979867 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f076c19f620 con 0x7f076c101290 2024-03-31T23:11:30.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:30.516+0000 7f076a7fc640 1 --2- 172.21.15.46:0/4229979867 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f076c101290 0x7f076c197f40 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f075400e9c0 tx=0x7f075400ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:11:30.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:30.516+0000 7f0763fff640 1 -- 172.21.15.46:0/4229979867 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f075400cd80 con 0x7f076c101290 2024-03-31T23:11:30.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:30.516+0000 7f0763fff640 1 -- 172.21.15.46:0/4229979867 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f07540035f0 con 0x7f076c101290 2024-03-31T23:11:30.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:30.516+0000 7f0771325640 1 -- 172.21.15.46:0/4229979867 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f076c19f910 con 0x7f076c101290 2024-03-31T23:11:30.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:30.516+0000 7f0763fff640 1 -- 172.21.15.46:0/4229979867 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0754010690 con 0x7f076c101290 2024-03-31T23:11:30.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:30.516+0000 7f0771325640 1 -- 172.21.15.46:0/4229979867 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f076c19fe10 con 0x7f076c101290 2024-03-31T23:11:30.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:30.517+0000 7f0771325640 1 -- 172.21.15.46:0/4229979867 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f076c1a00a0 con 0x7f076c101290 2024-03-31T23:11:30.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:30.518+0000 7f0763fff640 1 -- 172.21.15.46:0/4229979867 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 18) ==== 137320+0+0 (secure 0 0 0) 0x7f07540037a0 con 0x7f076c101290 2024-03-31T23:11:30.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:30.519+0000 7f0763fff640 1 --2- 172.21.15.46:0/4229979867 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f07580a24a0 0x7f07580a4960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:30.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:30.519+0000 7f076affd640 1 --2- 172.21.15.46:0/4229979867 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f07580a24a0 0x7f07580a4960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:30.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:30.519+0000 7f0763fff640 1 -- 172.21.15.46:0/4229979867 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9208+0+0 (secure 0 0 0) 0x7f07540cefd0 con 0x7f076c101290 2024-03-31T23:11:30.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:30.519+0000 7f076affd640 1 --2- 172.21.15.46:0/4229979867 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f07580a24a0 0x7f07580a4960 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f0764005ab0 tx=0x7f0764005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:11:30.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:30.523+0000 7f0763fff640 1 -- 172.21.15.46:0/4229979867 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f075409bdc0 con 0x7f076c101290 2024-03-31T23:11:30.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:30.708+0000 7f0771325640 1 -- 172.21.15.46:0/4229979867 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f076c1a0270 con 0x7f076c101290 2024-03-31T23:11:30.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:30.709+0000 7f0763fff640 1 -- 172.21.15.46:0/4229979867 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v95) ==== 103+0+11 (secure 0 0 0) 0x7f075409b6c0 con 0x7f076c101290 2024-03-31T23:11:30.709 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 32 2024-03-31T23:11:30.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:30.711+0000 7f0771325640 1 -- 172.21.15.46:0/4229979867 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f07580a24a0 msgr2=0x7f07580a4960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:30.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:30.711+0000 7f0771325640 1 --2- 172.21.15.46:0/4229979867 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f07580a24a0 0x7f07580a4960 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f0764005ab0 tx=0x7f0764005fd0 comp rx=0 tx=0).stop 2024-03-31T23:11:30.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:30.711+0000 7f0771325640 1 -- 172.21.15.46:0/4229979867 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f076c101290 msgr2=0x7f076c197f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:30.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:30.711+0000 7f0771325640 1 --2- 172.21.15.46:0/4229979867 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f076c101290 0x7f076c197f40 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f075400e9c0 tx=0x7f075400ee80 comp rx=0 tx=0).stop 2024-03-31T23:11:30.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:30.711+0000 7f0771325640 1 -- 172.21.15.46:0/4229979867 shutdown_connections 2024-03-31T23:11:30.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:30.711+0000 7f0771325640 1 --2- 172.21.15.46:0/4229979867 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f07580a24a0 0x7f07580a4960 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:30.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:30.711+0000 7f0771325640 1 --2- 172.21.15.46:0/4229979867 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f076c101c80 0x7f076c19efb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:30.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:30.711+0000 7f0771325640 1 --2- 172.21.15.46:0/4229979867 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f076c101290 0x7f076c197f40 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:30.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:30.711+0000 7f0771325640 1 --2- 172.21.15.46:0/4229979867 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f076c1008d0 0x7f076c197a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:30.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:30.711+0000 7f0771325640 1 -- 172.21.15.46:0/4229979867 >> 172.21.15.46:0/4229979867 conn(0x7f076c0fb380 msgr2=0x7f076c0ff7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:11:30.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:30.711+0000 7f0771325640 1 -- 172.21.15.46:0/4229979867 shutdown_connections 2024-03-31T23:11:30.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:30.711+0000 7f0771325640 1 -- 172.21.15.46:0/4229979867 wait complete. 2024-03-31T23:11:30.721 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:11:30 smithi046 ceph-mon[32498]: pgmap v365: 129 pgs: 129 active+clean; 643 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2024-03-31T23:11:30.721 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:11:30 smithi046 ceph-mon[32498]: from='client.? 172.21.15.46:0/3337492262' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-31T23:11:30.721 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:11:30 smithi046 ceph-mon[32498]: from='client.? 172.21.15.46:0/3255428415' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-31T23:11:30.721 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:11:30 smithi046 ceph-mon[32498]: from='client.? 172.21.15.46:0/3060866992' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-31T23:11:30.722 INFO:tasks.fwd_scrub:Wait for all MDSs to reach steady state... 2024-03-31T23:11:30.723 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-31T23:11:30.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:30.825+0000 7fac97278640 1 -- 172.21.15.46:0/4056979663 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fac90101860 msgr2=0x7fac90101ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:30.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:30.825+0000 7fac97278640 1 --2- 172.21.15.46:0/4056979663 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fac90101860 0x7fac90101ce0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7fac84009900 tx=0x7fac8402bc30 comp rx=0 tx=0).stop 2024-03-31T23:11:30.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:30.825+0000 7fac97278640 1 -- 172.21.15.46:0/4056979663 shutdown_connections 2024-03-31T23:11:30.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:30.825+0000 7fac97278640 1 --2- 172.21.15.46:0/4056979663 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fac90102220 0x7fac90108ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:30.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:30.825+0000 7fac97278640 1 --2- 172.21.15.46:0/4056979663 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fac90101860 0x7fac90101ce0 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:30.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:30.825+0000 7fac97278640 1 --2- 172.21.15.46:0/4056979663 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fac90100670 0x7fac90100a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:30.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:30.825+0000 7fac97278640 1 -- 172.21.15.46:0/4056979663 >> 172.21.15.46:0/4056979663 conn(0x7fac900fbbe0 msgr2=0x7fac900fe020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:11:30.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:30.825+0000 7fac97278640 1 -- 172.21.15.46:0/4056979663 shutdown_connections 2024-03-31T23:11:30.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:30.825+0000 7fac97278640 1 -- 172.21.15.46:0/4056979663 wait complete. 2024-03-31T23:11:30.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:30.826+0000 7fac97278640 1 Processor -- start 2024-03-31T23:11:30.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:30.826+0000 7fac97278640 1 -- start start 2024-03-31T23:11:30.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:30.826+0000 7fac97278640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fac90100670 0x7fac90196890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:30.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:30.826+0000 7fac97278640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fac90101860 0x7fac90196dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:30.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:30.826+0000 7fac97278640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fac90102220 0x7fac9019de40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:30.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:30.826+0000 7fac97278640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fac9010c1b0 con 0x7fac90100670 2024-03-31T23:11:30.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:30.826+0000 7fac97278640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fac9010c030 con 0x7fac90102220 2024-03-31T23:11:30.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:30.826+0000 7fac97278640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fac9010c330 con 0x7fac90101860 2024-03-31T23:11:30.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:30.826+0000 7fac95014640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fac90100670 0x7fac90196890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:30.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:30.826+0000 7fac95014640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fac90100670 0x7fac90196890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:58524/0 (socket says 172.21.15.46:58524) 2024-03-31T23:11:30.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:30.826+0000 7fac95014640 1 -- 172.21.15.46:0/3641190743 learned_addr learned my addr 172.21.15.46:0/3641190743 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:11:30.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:30.826+0000 7fac95815640 1 --2- 172.21.15.46:0/3641190743 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fac90102220 0x7fac9019de40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:30.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:30.826+0000 7fac94813640 1 --2- 172.21.15.46:0/3641190743 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fac90101860 0x7fac90196dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:30.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:30.827+0000 7fac95014640 1 -- 172.21.15.46:0/3641190743 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fac90101860 msgr2=0x7fac90196dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:30.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:30.827+0000 7fac95014640 1 --2- 172.21.15.46:0/3641190743 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fac90101860 0x7fac90196dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:30.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:30.827+0000 7fac95014640 1 -- 172.21.15.46:0/3641190743 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fac90102220 msgr2=0x7fac9019de40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:30.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:30.827+0000 7fac95014640 1 --2- 172.21.15.46:0/3641190743 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fac90102220 0x7fac9019de40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:30.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:30.827+0000 7fac95014640 1 -- 172.21.15.46:0/3641190743 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac9019e4b0 con 0x7fac90100670 2024-03-31T23:11:30.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:30.827+0000 7fac95014640 1 --2- 172.21.15.46:0/3641190743 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fac90100670 0x7fac90196890 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7fac8800e9c0 tx=0x7fac8800ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:11:30.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:30.827+0000 7fac8e7fc640 1 -- 172.21.15.46:0/3641190743 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac8800cd80 con 0x7fac90100670 2024-03-31T23:11:30.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:30.827+0000 7fac97278640 1 -- 172.21.15.46:0/3641190743 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fac9019e7a0 con 0x7fac90100670 2024-03-31T23:11:30.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:30.827+0000 7fac8e7fc640 1 -- 172.21.15.46:0/3641190743 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fac880035f0 con 0x7fac90100670 2024-03-31T23:11:30.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:30.827+0000 7fac8e7fc640 1 -- 172.21.15.46:0/3641190743 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac88010690 con 0x7fac90100670 2024-03-31T23:11:30.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:30.827+0000 7fac97278640 1 -- 172.21.15.46:0/3641190743 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fac9019ecc0 con 0x7fac90100670 2024-03-31T23:11:30.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:30.828+0000 7fac97278640 1 -- 172.21.15.46:0/3641190743 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fac90107a10 con 0x7fac90100670 2024-03-31T23:11:30.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:30.829+0000 7fac8e7fc640 1 -- 172.21.15.46:0/3641190743 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 18) ==== 137320+0+0 (secure 0 0 0) 0x7fac880037a0 con 0x7fac90100670 2024-03-31T23:11:30.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:30.829+0000 7fac8e7fc640 1 --2- 172.21.15.46:0/3641190743 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fac780a24a0 0x7fac780a4960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:30.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:30.830+0000 7fac94813640 1 --2- 172.21.15.46:0/3641190743 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fac780a24a0 0x7fac780a4960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:30.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:30.830+0000 7fac8e7fc640 1 -- 172.21.15.46:0/3641190743 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9208+0+0 (secure 0 0 0) 0x7fac880ceed0 con 0x7fac90100670 2024-03-31T23:11:30.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:30.830+0000 7fac94813640 1 --2- 172.21.15.46:0/3641190743 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fac780a24a0 0x7fac780a4960 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fac84009230 tx=0x7fac84005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:11:30.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:30.834+0000 7fac8e7fc640 1 -- 172.21.15.46:0/3641190743 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fac8809bc60 con 0x7fac90100670 2024-03-31T23:11:30.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:11:30 smithi142 ceph-mon[35271]: pgmap v365: 129 pgs: 129 active+clean; 643 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2024-03-31T23:11:30.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:11:30 smithi142 ceph-mon[35271]: from='client.? 172.21.15.46:0/3337492262' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-31T23:11:30.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:11:30 smithi142 ceph-mon[35271]: from='client.? 172.21.15.46:0/3255428415' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-31T23:11:30.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:11:30 smithi142 ceph-mon[35271]: from='client.? 172.21.15.46:0/3060866992' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-31T23:11:31.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:11:30 smithi162 ceph-mon[36237]: pgmap v365: 129 pgs: 129 active+clean; 643 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2024-03-31T23:11:31.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:11:30 smithi162 ceph-mon[36237]: from='client.? 172.21.15.46:0/3337492262' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-31T23:11:31.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:11:30 smithi162 ceph-mon[36237]: from='client.? 172.21.15.46:0/3255428415' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-31T23:11:31.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:11:30 smithi162 ceph-mon[36237]: from='client.? 172.21.15.46:0/3060866992' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-31T23:11:31.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:31.057+0000 7fac97278640 1 -- 172.21.15.46:0/3641190743 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fac9002d170 con 0x7fac90100670 2024-03-31T23:11:31.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:31.057+0000 7fac8e7fc640 1 -- 172.21.15.46:0/3641190743 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+11882 (secure 0 0 0) 0x7fac8809b560 con 0x7fac90100670 2024-03-31T23:11:31.057 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-31T23:11:31.058 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/2169018547","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":2169018547},{"type":"v1","addr":"172.21.15.46:6835","nonce":2169018547}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6837/1069387756","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6836","nonce":1069387756},{"type":"v1","addr":"172.21.15.46:6837","nonce":1069387756}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14634,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6839/3915337844","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6838","nonce":3915337844},{"type":"v1","addr":"172.21.15.46:6839","nonce":3915337844}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14646,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6841/897767145","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6840","nonce":897767145},{"type":"v1","addr":"172.21.15.46:6841","nonce":897767145}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24415,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6833/1746950966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6832","nonce":1746950966},{"type":"v1","addr":"172.21.15.162:6833","nonce":1746950966}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24422,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6833/3394739181","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6832","nonce":3394739181},{"type":"v1","addr":"172.21.15.142:6833","nonce":3394739181}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":25,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-31T23:09:11.009090+0000","modified":"2024-03-31T23:11:27.558846+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24515,"mds_1":24482,"mds_2":24451},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24442":{"gid":24442,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.162:6835/1957208984","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6834","nonce":1957208984},{"type":"v1","addr":"172.21.15.162:6835","nonce":1957208984}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24451":{"gid":24451,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":11,"addr":"172.21.15.162:6837/1371886876","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6836","nonce":1371886876},{"type":"v1","addr":"172.21.15.162:6837","nonce":1371886876}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24455":{"gid":24455,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.142:6835/212637431","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6834","nonce":212637431},{"type":"v1","addr":"172.21.15.142:6835","nonce":212637431}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24478":{"gid":24478,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.162:6839/1976275826","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6838","nonce":1976275826},{"type":"v1","addr":"172.21.15.162:6839","nonce":1976275826}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24482":{"gid":24482,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":10,"addr":"172.21.15.142:6837/2559258399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6836","nonce":2559258399},{"type":"v1","addr":"172.21.15.142:6837","nonce":2559258399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24515":{"gid":24515,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.142:6839/2519143918","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6838","nonce":2519143918},{"type":"v1","addr":"172.21.15.142:6839","nonce":2519143918}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24515,"qdb_cluster":[24515,24451,24482]},"id":1}]} 2024-03-31T23:11:31.058 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 25 2024-03-31T23:11:31.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:31.059+0000 7fac97278640 1 -- 172.21.15.46:0/3641190743 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fac780a24a0 msgr2=0x7fac780a4960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:31.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:31.059+0000 7fac97278640 1 --2- 172.21.15.46:0/3641190743 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fac780a24a0 0x7fac780a4960 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fac84009230 tx=0x7fac84005fd0 comp rx=0 tx=0).stop 2024-03-31T23:11:31.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:31.059+0000 7fac97278640 1 -- 172.21.15.46:0/3641190743 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fac90100670 msgr2=0x7fac90196890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:31.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:31.059+0000 7fac97278640 1 --2- 172.21.15.46:0/3641190743 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fac90100670 0x7fac90196890 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7fac8800e9c0 tx=0x7fac8800ee80 comp rx=0 tx=0).stop 2024-03-31T23:11:31.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:31.059+0000 7fac97278640 1 -- 172.21.15.46:0/3641190743 shutdown_connections 2024-03-31T23:11:31.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:31.059+0000 7fac97278640 1 --2- 172.21.15.46:0/3641190743 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fac780a24a0 0x7fac780a4960 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:31.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:31.059+0000 7fac97278640 1 --2- 172.21.15.46:0/3641190743 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fac90102220 0x7fac9019de40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:31.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:31.059+0000 7fac97278640 1 --2- 172.21.15.46:0/3641190743 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fac90101860 0x7fac90196dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:31.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:31.059+0000 7fac97278640 1 --2- 172.21.15.46:0/3641190743 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fac90100670 0x7fac90196890 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:31.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:31.059+0000 7fac97278640 1 -- 172.21.15.46:0/3641190743 >> 172.21.15.46:0/3641190743 conn(0x7fac900fbbe0 msgr2=0x7fac90104be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:11:31.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:31.059+0000 7fac97278640 1 -- 172.21.15.46:0/3641190743 shutdown_connections 2024-03-31T23:11:31.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:31.059+0000 7fac97278640 1 -- 172.21.15.46:0/3641190743 wait complete. 2024-03-31T23:11:31.071 INFO:tasks.fwd_scrub:Ready to start scrub thrashing 2024-03-31T23:11:31.071 INFO:tasks.fwd_scrub.ceph_manager:waiting for clean 2024-03-31T23:11:31.071 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-31T23:11:31.179 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:31.174+0000 7f1b65df6640 1 -- 172.21.15.46:0/1492655293 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1b60103330 msgr2=0x7f1b60109bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:31.180 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:31.174+0000 7f1b65df6640 1 --2- 172.21.15.46:0/1492655293 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1b60103330 0x7f1b60109bc0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f1b58009ca0 tx=0x7f1b5802c0f0 comp rx=0 tx=0).stop 2024-03-31T23:11:31.180 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:31.174+0000 7f1b65df6640 1 -- 172.21.15.46:0/1492655293 shutdown_connections 2024-03-31T23:11:31.180 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:31.174+0000 7f1b65df6640 1 --2- 172.21.15.46:0/1492655293 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1b60103330 0x7f1b60109bc0 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:31.180 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:31.174+0000 7f1b65df6640 1 --2- 172.21.15.46:0/1492655293 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1b60102970 0x7f1b60102df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:31.180 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:31.174+0000 7f1b65df6640 1 --2- 172.21.15.46:0/1492655293 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1b60101780 0x7f1b60101ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:31.180 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:31.174+0000 7f1b65df6640 1 -- 172.21.15.46:0/1492655293 >> 172.21.15.46:0/1492655293 conn(0x7f1b600fcd30 msgr2=0x7f1b600ff150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:11:31.180 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:31.174+0000 7f1b65df6640 1 -- 172.21.15.46:0/1492655293 shutdown_connections 2024-03-31T23:11:31.180 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:31.174+0000 7f1b65df6640 1 -- 172.21.15.46:0/1492655293 wait complete. 2024-03-31T23:11:31.180 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:31.175+0000 7f1b65df6640 1 Processor -- start 2024-03-31T23:11:31.180 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:31.175+0000 7f1b65df6640 1 -- start start 2024-03-31T23:11:31.180 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:31.175+0000 7f1b65df6640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1b60101780 0x7f1b60197930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:31.180 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:31.175+0000 7f1b65df6640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1b60102970 0x7f1b60197e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:31.181 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:31.175+0000 7f1b65df6640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1b60103330 0x7f1b6019eee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:31.181 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:31.175+0000 7f1b65df6640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f1b6010d2b0 con 0x7f1b60103330 2024-03-31T23:11:31.181 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:31.175+0000 7f1b65df6640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f1b6010d130 con 0x7f1b60102970 2024-03-31T23:11:31.181 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:31.175+0000 7f1b65df6640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f1b6010d430 con 0x7f1b60101780 2024-03-31T23:11:31.181 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:31.175+0000 7f1b5ffff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1b60103330 0x7f1b6019eee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:31.181 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:31.175+0000 7f1b5ffff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1b60103330 0x7f1b6019eee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:58542/0 (socket says 172.21.15.46:58542) 2024-03-31T23:11:31.181 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:31.175+0000 7f1b5effd640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1b60102970 0x7f1b60197e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:31.181 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:31.175+0000 7f1b5ffff640 1 -- 172.21.15.46:0/2147102252 learned_addr learned my addr 172.21.15.46:0/2147102252 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:11:31.181 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:31.175+0000 7f1b5f7fe640 1 --2- 172.21.15.46:0/2147102252 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1b60101780 0x7f1b60197930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:31.181 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:31.175+0000 7f1b5ffff640 1 -- 172.21.15.46:0/2147102252 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1b60101780 msgr2=0x7f1b60197930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:31.181 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:31.175+0000 7f1b5ffff640 1 --2- 172.21.15.46:0/2147102252 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1b60101780 0x7f1b60197930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:31.181 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:31.175+0000 7f1b5ffff640 1 -- 172.21.15.46:0/2147102252 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1b60102970 msgr2=0x7f1b60197e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:31.181 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:31.175+0000 7f1b5ffff640 1 --2- 172.21.15.46:0/2147102252 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1b60102970 0x7f1b60197e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:31.182 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:31.176+0000 7f1b5ffff640 1 -- 172.21.15.46:0/2147102252 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b6019f550 con 0x7f1b60103330 2024-03-31T23:11:31.182 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:31.176+0000 7f1b5ffff640 1 --2- 172.21.15.46:0/2147102252 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1b60103330 0x7f1b6019eee0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f1b5802c700 tx=0x7f1b580328a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:11:31.182 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:31.176+0000 7f1b5cff9640 1 -- 172.21.15.46:0/2147102252 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b58041020 con 0x7f1b60103330 2024-03-31T23:11:31.182 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:31.176+0000 7f1b5cff9640 1 -- 172.21.15.46:0/2147102252 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1b58003e20 con 0x7f1b60103330 2024-03-31T23:11:31.182 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:31.176+0000 7f1b65df6640 1 -- 172.21.15.46:0/2147102252 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1b6019f7e0 con 0x7f1b60103330 2024-03-31T23:11:31.182 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:31.176+0000 7f1b5cff9640 1 -- 172.21.15.46:0/2147102252 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b58039040 con 0x7f1b60103330 2024-03-31T23:11:31.182 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:31.176+0000 7f1b65df6640 1 -- 172.21.15.46:0/2147102252 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1b6019fcd0 con 0x7f1b60103330 2024-03-31T23:11:31.182 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:31.177+0000 7f1b65df6640 1 -- 172.21.15.46:0/2147102252 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1b60108b70 con 0x7f1b60103330 2024-03-31T23:11:31.182 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:31.178+0000 7f1b5cff9640 1 -- 172.21.15.46:0/2147102252 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 18) ==== 137320+0+0 (secure 0 0 0) 0x7f1b58044020 con 0x7f1b60103330 2024-03-31T23:11:31.183 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:31.179+0000 7f1b5cff9640 1 --2- 172.21.15.46:0/2147102252 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f1b480a24a0 0x7f1b480a4960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:31.183 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:31.179+0000 7f1b5f7fe640 1 --2- 172.21.15.46:0/2147102252 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f1b480a24a0 0x7f1b480a4960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:31.183 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:31.179+0000 7f1b5cff9640 1 -- 172.21.15.46:0/2147102252 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9208+0+0 (secure 0 0 0) 0x7f1b580f1060 con 0x7f1b60103330 2024-03-31T23:11:31.183 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:31.179+0000 7f1b5f7fe640 1 --2- 172.21.15.46:0/2147102252 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f1b480a24a0 0x7f1b480a4960 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f1b601027d0 tx=0x7f1b50005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:11:31.183 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:31.182+0000 7f1b5cff9640 1 -- 172.21.15.46:0/2147102252 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f1b5804b020 con 0x7f1b60103330 2024-03-31T23:11:31.364 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:31.363+0000 7f1b65df6640 1 -- 172.21.15.46:0/2147102252 --> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f1b60061b90 con 0x7f1b480a24a0 2024-03-31T23:11:31.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:31.373+0000 7f1b5cff9640 1 -- 172.21.15.46:0/2147102252 <== mgr.14152 v2:172.21.15.46:6800/162289569 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340476 (secure 0 0 0) 0x7f1b60061b90 con 0x7f1b480a24a0 2024-03-31T23:11:31.377 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-31T23:11:31.380 INFO:teuthology.orchestra.run.smithi046.stderr:dumped all 2024-03-31T23:11:31.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:31.376+0000 7f1b65df6640 1 -- 172.21.15.46:0/2147102252 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f1b480a24a0 msgr2=0x7f1b480a4960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:31.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:31.376+0000 7f1b65df6640 1 --2- 172.21.15.46:0/2147102252 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f1b480a24a0 0x7f1b480a4960 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f1b601027d0 tx=0x7f1b50005f90 comp rx=0 tx=0).stop 2024-03-31T23:11:31.382 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:31.376+0000 7f1b65df6640 1 -- 172.21.15.46:0/2147102252 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1b60103330 msgr2=0x7f1b6019eee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:31.382 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:31.376+0000 7f1b65df6640 1 --2- 172.21.15.46:0/2147102252 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1b60103330 0x7f1b6019eee0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f1b5802c700 tx=0x7f1b580328a0 comp rx=0 tx=0).stop 2024-03-31T23:11:31.382 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:31.376+0000 7f1b65df6640 1 -- 172.21.15.46:0/2147102252 shutdown_connections 2024-03-31T23:11:31.382 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:31.376+0000 7f1b65df6640 1 --2- 172.21.15.46:0/2147102252 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f1b480a24a0 0x7f1b480a4960 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:31.382 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:31.376+0000 7f1b65df6640 1 --2- 172.21.15.46:0/2147102252 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1b60103330 0x7f1b6019eee0 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:31.382 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:31.376+0000 7f1b65df6640 1 --2- 172.21.15.46:0/2147102252 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1b60102970 0x7f1b60197e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:31.382 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:31.376+0000 7f1b65df6640 1 --2- 172.21.15.46:0/2147102252 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1b60101780 0x7f1b60197930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:31.382 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:31.376+0000 7f1b65df6640 1 -- 172.21.15.46:0/2147102252 >> 172.21.15.46:0/2147102252 conn(0x7f1b600fcd30 msgr2=0x7f1b60105d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:11:31.382 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:31.377+0000 7f1b65df6640 1 -- 172.21.15.46:0/2147102252 shutdown_connections 2024-03-31T23:11:31.382 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:31.377+0000 7f1b65df6640 1 -- 172.21.15.46:0/2147102252 wait complete. 2024-03-31T23:11:31.387 INFO:teuthology.orchestra.run.smithi046.stdout:{"pg_ready":true,"pg_map":{"version":366,"stamp":"2024-03-31T23:11:30.612781+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":663158,"num_objects":64,"num_object_clones":0,"num_object_copies":194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":880,"num_read_kb":570,"num_write":365,"num_write_kb":3077,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":797,"ondisk_log_size":797,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":372340,"kb_used_data":14824,"kb_used_omap":18,"kb_used_meta":357357,"kb_avail":1124569484,"statfs":{"total":1151940427776,"available":1151559151616,"internally_reserved":0,"allocated":15179776,"data_stored":8364853,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19140,"internal_metadata":365933884},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":5300,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":82,"num_read_kb":51,"num_write":10,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003267"},"pg_stats":[{"pgid":"3.3f","version":"93'8","reported_seq":40,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:10.543291+0000","last_change":"2024-03-31T23:09:11.177502+0000","last_active":"2024-03-31T23:11:10.543291+0000","last_peered":"2024-03-31T23:11:10.543291+0000","last_clean":"2024-03-31T23:11:10.543291+0000","last_became_active":"2024-03-31T23:09:11.177134+0000","last_became_peered":"2024-03-31T23:09:11.177134+0000","last_unstale":"2024-03-31T23:11:10.543291+0000","last_undegraded":"2024-03-31T23:11:10.543291+0000","last_fullsized":"2024-03-31T23:11:10.543291+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T10:29:36.508229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"93'4","reported_seq":36,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:11.122776+0000","last_change":"2024-03-31T23:09:11.172990+0000","last_active":"2024-03-31T23:11:11.122776+0000","last_peered":"2024-03-31T23:11:11.122776+0000","last_clean":"2024-03-31T23:11:11.122776+0000","last_became_active":"2024-03-31T23:09:11.172684+0000","last_became_peered":"2024-03-31T23:09:11.172684+0000","last_unstale":"2024-03-31T23:11:11.122776+0000","last_undegraded":"2024-03-31T23:11:11.122776+0000","last_fullsized":"2024-03-31T23:11:11.122776+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T07:02:59.991757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"93'7","reported_seq":39,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:11.122648+0000","last_change":"2024-03-31T23:09:11.176805+0000","last_active":"2024-03-31T23:11:11.122648+0000","last_peered":"2024-03-31T23:11:11.122648+0000","last_clean":"2024-03-31T23:11:11.122648+0000","last_became_active":"2024-03-31T23:09:11.176468+0000","last_became_peered":"2024-03-31T23:09:11.176468+0000","last_unstale":"2024-03-31T23:11:11.122648+0000","last_undegraded":"2024-03-31T23:11:11.122648+0000","last_fullsized":"2024-03-31T23:11:11.122648+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T08:23:41.872558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"93'8","reported_seq":40,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:10.545278+0000","last_change":"2024-03-31T23:09:11.182385+0000","last_active":"2024-03-31T23:11:10.545278+0000","last_peered":"2024-03-31T23:11:10.545278+0000","last_clean":"2024-03-31T23:11:10.545278+0000","last_became_active":"2024-03-31T23:09:11.181908+0000","last_became_peered":"2024-03-31T23:09:11.181908+0000","last_unstale":"2024-03-31T23:11:10.545278+0000","last_undegraded":"2024-03-31T23:11:10.545278+0000","last_fullsized":"2024-03-31T23:11:10.545278+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:52:24.066360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"93'4","reported_seq":35,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:10.542115+0000","last_change":"2024-03-31T23:09:11.181377+0000","last_active":"2024-03-31T23:11:10.542115+0000","last_peered":"2024-03-31T23:11:10.542115+0000","last_clean":"2024-03-31T23:11:10.542115+0000","last_became_active":"2024-03-31T23:09:11.181064+0000","last_became_peered":"2024-03-31T23:09:11.181064+0000","last_unstale":"2024-03-31T23:11:10.542115+0000","last_undegraded":"2024-03-31T23:11:10.542115+0000","last_fullsized":"2024-03-31T23:11:10.542115+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:28:40.766541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"93'6","reported_seq":37,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:11.124851+0000","last_change":"2024-03-31T23:09:11.176087+0000","last_active":"2024-03-31T23:11:11.124851+0000","last_peered":"2024-03-31T23:11:11.124851+0000","last_clean":"2024-03-31T23:11:11.124851+0000","last_became_active":"2024-03-31T23:09:11.175569+0000","last_became_peered":"2024-03-31T23:09:11.175569+0000","last_unstale":"2024-03-31T23:11:11.124851+0000","last_undegraded":"2024-03-31T23:11:11.124851+0000","last_fullsized":"2024-03-31T23:11:11.124851+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T06:33:19.189909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"93'5","reported_seq":36,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:10.546125+0000","last_change":"2024-03-31T23:09:11.170289+0000","last_active":"2024-03-31T23:11:10.546125+0000","last_peered":"2024-03-31T23:11:10.546125+0000","last_clean":"2024-03-31T23:11:10.546125+0000","last_became_active":"2024-03-31T23:09:11.169959+0000","last_became_peered":"2024-03-31T23:09:11.169959+0000","last_unstale":"2024-03-31T23:11:10.546125+0000","last_undegraded":"2024-03-31T23:11:10.546125+0000","last_fullsized":"2024-03-31T23:11:10.546125+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:10:44.614767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"93'9","reported_seq":40,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:10.540524+0000","last_change":"2024-03-31T23:09:11.182447+0000","last_active":"2024-03-31T23:11:10.540524+0000","last_peered":"2024-03-31T23:11:10.540524+0000","last_clean":"2024-03-31T23:11:10.540524+0000","last_became_active":"2024-03-31T23:09:11.181629+0000","last_became_peered":"2024-03-31T23:09:11.181629+0000","last_unstale":"2024-03-31T23:11:10.540524+0000","last_undegraded":"2024-03-31T23:11:10.540524+0000","last_fullsized":"2024-03-31T23:11:10.540524+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:18:26.269451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"93'11","reported_seq":42,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:10.934818+0000","last_change":"2024-03-31T23:09:11.176853+0000","last_active":"2024-03-31T23:11:10.934818+0000","last_peered":"2024-03-31T23:11:10.934818+0000","last_clean":"2024-03-31T23:11:10.934818+0000","last_became_active":"2024-03-31T23:09:11.176413+0000","last_became_peered":"2024-03-31T23:09:11.176413+0000","last_unstale":"2024-03-31T23:11:10.934818+0000","last_undegraded":"2024-03-31T23:11:10.934818+0000","last_fullsized":"2024-03-31T23:11:10.934818+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:23:40.486947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"93'16","reported_seq":53,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:11.125686+0000","last_change":"2024-03-31T23:10:10.543998+0000","last_active":"2024-03-31T23:11:11.125686+0000","last_peered":"2024-03-31T23:11:11.125686+0000","last_clean":"2024-03-31T23:11:11.125686+0000","last_became_active":"2024-03-31T23:10:10.543615+0000","last_became_peered":"2024-03-31T23:10:10.543615+0000","last_unstale":"2024-03-31T23:11:11.125686+0000","last_undegraded":"2024-03-31T23:11:11.125686+0000","last_fullsized":"2024-03-31T23:11:11.125686+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:28:32.883732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,9],"acting":[10,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"93'6","reported_seq":37,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:10.545678+0000","last_change":"2024-03-31T23:09:11.179536+0000","last_active":"2024-03-31T23:11:10.545678+0000","last_peered":"2024-03-31T23:11:10.545678+0000","last_clean":"2024-03-31T23:11:10.545678+0000","last_became_active":"2024-03-31T23:09:11.178521+0000","last_became_peered":"2024-03-31T23:09:11.178521+0000","last_unstale":"2024-03-31T23:11:10.545678+0000","last_undegraded":"2024-03-31T23:11:10.545678+0000","last_fullsized":"2024-03-31T23:11:10.545678+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T09:29:23.128832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"93'7","reported_seq":38,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:10.545128+0000","last_change":"2024-03-31T23:09:11.182548+0000","last_active":"2024-03-31T23:11:10.545128+0000","last_peered":"2024-03-31T23:11:10.545128+0000","last_clean":"2024-03-31T23:11:10.545128+0000","last_became_active":"2024-03-31T23:09:11.181966+0000","last_became_peered":"2024-03-31T23:09:11.181966+0000","last_unstale":"2024-03-31T23:11:10.545128+0000","last_undegraded":"2024-03-31T23:11:10.545128+0000","last_fullsized":"2024-03-31T23:11:10.545128+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:43:26.061594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"93'10","reported_seq":47,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:10.716409+0000","last_change":"2024-03-31T23:10:10.544976+0000","last_active":"2024-03-31T23:11:10.716409+0000","last_peered":"2024-03-31T23:11:10.716409+0000","last_clean":"2024-03-31T23:11:10.716409+0000","last_became_active":"2024-03-31T23:10:10.544433+0000","last_became_peered":"2024-03-31T23:10:10.544433+0000","last_unstale":"2024-03-31T23:11:10.716409+0000","last_undegraded":"2024-03-31T23:11:10.716409+0000","last_fullsized":"2024-03-31T23:11:10.716409+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:45:54.606274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,9],"acting":[6,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"93'7","reported_seq":38,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:11.122229+0000","last_change":"2024-03-31T23:09:11.168784+0000","last_active":"2024-03-31T23:11:11.122229+0000","last_peered":"2024-03-31T23:11:11.122229+0000","last_clean":"2024-03-31T23:11:11.122229+0000","last_became_active":"2024-03-31T23:09:11.168442+0000","last_became_peered":"2024-03-31T23:09:11.168442+0000","last_unstale":"2024-03-31T23:11:11.122229+0000","last_undegraded":"2024-03-31T23:11:11.122229+0000","last_fullsized":"2024-03-31T23:11:11.122229+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T09:37:43.568122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"93'7","reported_seq":38,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:11.122185+0000","last_change":"2024-03-31T23:09:11.176017+0000","last_active":"2024-03-31T23:11:11.122185+0000","last_peered":"2024-03-31T23:11:11.122185+0000","last_clean":"2024-03-31T23:11:11.122185+0000","last_became_active":"2024-03-31T23:09:11.175690+0000","last_became_peered":"2024-03-31T23:09:11.175690+0000","last_unstale":"2024-03-31T23:11:11.122185+0000","last_undegraded":"2024-03-31T23:11:11.122185+0000","last_fullsized":"2024-03-31T23:11:11.122185+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:14:13.259895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"93'9","reported_seq":40,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:10.934173+0000","last_change":"2024-03-31T23:09:11.171926+0000","last_active":"2024-03-31T23:11:10.934173+0000","last_peered":"2024-03-31T23:11:10.934173+0000","last_clean":"2024-03-31T23:11:10.934173+0000","last_became_active":"2024-03-31T23:09:11.171625+0000","last_became_peered":"2024-03-31T23:09:11.171625+0000","last_unstale":"2024-03-31T23:11:10.934173+0000","last_undegraded":"2024-03-31T23:11:10.934173+0000","last_fullsized":"2024-03-31T23:11:10.934173+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T06:20:20.433802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"93'8","reported_seq":39,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:10.543011+0000","last_change":"2024-03-31T23:09:11.183805+0000","last_active":"2024-03-31T23:11:10.543011+0000","last_peered":"2024-03-31T23:11:10.543011+0000","last_clean":"2024-03-31T23:11:10.543011+0000","last_became_active":"2024-03-31T23:09:11.183464+0000","last_became_peered":"2024-03-31T23:09:11.183464+0000","last_unstale":"2024-03-31T23:11:10.543011+0000","last_undegraded":"2024-03-31T23:11:10.543011+0000","last_fullsized":"2024-03-31T23:11:10.543011+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T06:02:33.286563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"93'10","reported_seq":41,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:11.117069+0000","last_change":"2024-03-31T23:09:11.170095+0000","last_active":"2024-03-31T23:11:11.117069+0000","last_peered":"2024-03-31T23:11:11.117069+0000","last_clean":"2024-03-31T23:11:11.117069+0000","last_became_active":"2024-03-31T23:09:11.169683+0000","last_became_peered":"2024-03-31T23:09:11.169683+0000","last_unstale":"2024-03-31T23:11:11.117069+0000","last_undegraded":"2024-03-31T23:11:11.117069+0000","last_fullsized":"2024-03-31T23:11:11.117069+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T08:19:47.101431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"93'7","reported_seq":39,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:10.544657+0000","last_change":"2024-03-31T23:09:11.181347+0000","last_active":"2024-03-31T23:11:10.544657+0000","last_peered":"2024-03-31T23:11:10.544657+0000","last_clean":"2024-03-31T23:11:10.544657+0000","last_became_active":"2024-03-31T23:09:11.180510+0000","last_became_peered":"2024-03-31T23:09:11.180510+0000","last_unstale":"2024-03-31T23:11:10.544657+0000","last_undegraded":"2024-03-31T23:11:10.544657+0000","last_fullsized":"2024-03-31T23:11:10.544657+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:25:51.083446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:10.544622+0000","last_change":"2024-03-31T23:09:15.205059+0000","last_active":"2024-03-31T23:11:10.544622+0000","last_peered":"2024-03-31T23:11:10.544622+0000","last_clean":"2024-03-31T23:11:10.544622+0000","last_became_active":"2024-03-31T23:09:15.204612+0000","last_became_peered":"2024-03-31T23:09:15.204612+0000","last_unstale":"2024-03-31T23:11:10.544622+0000","last_undegraded":"2024-03-31T23:11:10.544622+0000","last_fullsized":"2024-03-31T23:11:10.544622+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_clean_scrub_stamp":"2024-03-31T23:09:14.171251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:53:33.998026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"91'9","reported_seq":44,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:11.116248+0000","last_change":"2024-03-31T23:09:10.594147+0000","last_active":"2024-03-31T23:11:11.116248+0000","last_peered":"2024-03-31T23:11:11.116248+0000","last_clean":"2024-03-31T23:11:11.116248+0000","last_became_active":"2024-03-31T23:09:10.592047+0000","last_became_peered":"2024-03-31T23:09:10.592047+0000","last_unstale":"2024-03-31T23:11:11.116248+0000","last_undegraded":"2024-03-31T23:11:11.116248+0000","last_fullsized":"2024-03-31T23:11:11.116248+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_clean_scrub_stamp":"2024-03-31T23:09:09.564263+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:04:34.548282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"93'5","reported_seq":37,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:10.541242+0000","last_change":"2024-03-31T23:09:11.182292+0000","last_active":"2024-03-31T23:11:10.541242+0000","last_peered":"2024-03-31T23:11:10.541242+0000","last_clean":"2024-03-31T23:11:10.541242+0000","last_became_active":"2024-03-31T23:09:11.181885+0000","last_became_peered":"2024-03-31T23:09:11.181885+0000","last_unstale":"2024-03-31T23:11:10.541242+0000","last_undegraded":"2024-03-31T23:11:10.541242+0000","last_fullsized":"2024-03-31T23:11:10.541242+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T10:38:17.550625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":32,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:10.544451+0000","last_change":"2024-03-31T23:10:10.543802+0000","last_active":"2024-03-31T23:11:10.544451+0000","last_peered":"2024-03-31T23:11:10.544451+0000","last_clean":"2024-03-31T23:11:10.544451+0000","last_became_active":"2024-03-31T23:10:10.543245+0000","last_became_peered":"2024-03-31T23:10:10.543245+0000","last_unstale":"2024-03-31T23:11:10.544451+0000","last_undegraded":"2024-03-31T23:11:10.544451+0000","last_fullsized":"2024-03-31T23:11:10.544451+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_clean_scrub_stamp":"2024-03-31T23:09:14.171251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T06:00:11.707183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,8,3],"acting":[4,6,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"91'2","reported_seq":36,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:10.541490+0000","last_change":"2024-03-31T23:09:10.598375+0000","last_active":"2024-03-31T23:11:10.541490+0000","last_peered":"2024-03-31T23:11:10.541490+0000","last_clean":"2024-03-31T23:11:10.541490+0000","last_became_active":"2024-03-31T23:09:10.598045+0000","last_became_peered":"2024-03-31T23:09:10.598045+0000","last_unstale":"2024-03-31T23:11:10.541490+0000","last_undegraded":"2024-03-31T23:11:10.541490+0000","last_fullsized":"2024-03-31T23:11:10.541490+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_clean_scrub_stamp":"2024-03-31T23:09:09.564263+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:31:27.113317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"93'6","reported_seq":38,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:11.122739+0000","last_change":"2024-03-31T23:09:11.183458+0000","last_active":"2024-03-31T23:11:11.122739+0000","last_peered":"2024-03-31T23:11:11.122739+0000","last_clean":"2024-03-31T23:11:11.122739+0000","last_became_active":"2024-03-31T23:09:11.183100+0000","last_became_peered":"2024-03-31T23:09:11.183100+0000","last_unstale":"2024-03-31T23:11:11.122739+0000","last_undegraded":"2024-03-31T23:11:11.122739+0000","last_fullsized":"2024-03-31T23:11:11.122739+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T10:55:35.512065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":32,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:11.271365+0000","last_change":"2024-03-31T23:11:11.271365+0000","last_active":"2024-03-31T23:11:11.271365+0000","last_peered":"2024-03-31T23:11:11.271365+0000","last_clean":"2024-03-31T23:11:11.271365+0000","last_became_active":"2024-03-31T23:11:11.270814+0000","last_became_peered":"2024-03-31T23:11:11.270814+0000","last_unstale":"2024-03-31T23:11:11.271365+0000","last_undegraded":"2024-03-31T23:11:11.271365+0000","last_fullsized":"2024-03-31T23:11:11.271365+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_clean_scrub_stamp":"2024-03-31T23:09:14.171251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:58:27.916348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,10,5],"acting":[4,3,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"91'4","reported_seq":37,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:10.541863+0000","last_change":"2024-03-31T23:09:10.597471+0000","last_active":"2024-03-31T23:11:10.541863+0000","last_peered":"2024-03-31T23:11:10.541863+0000","last_clean":"2024-03-31T23:11:10.541863+0000","last_became_active":"2024-03-31T23:09:10.597165+0000","last_became_peered":"2024-03-31T23:09:10.597165+0000","last_unstale":"2024-03-31T23:11:10.541863+0000","last_undegraded":"2024-03-31T23:11:10.541863+0000","last_fullsized":"2024-03-31T23:11:10.541863+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_clean_scrub_stamp":"2024-03-31T23:09:09.564263+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:10:26.826525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"93'9","reported_seq":40,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:10.934521+0000","last_change":"2024-03-31T23:09:11.170650+0000","last_active":"2024-03-31T23:11:10.934521+0000","last_peered":"2024-03-31T23:11:10.934521+0000","last_clean":"2024-03-31T23:11:10.934521+0000","last_became_active":"2024-03-31T23:09:11.170291+0000","last_became_peered":"2024-03-31T23:09:11.170291+0000","last_unstale":"2024-03-31T23:11:10.934521+0000","last_undegraded":"2024-03-31T23:11:10.934521+0000","last_fullsized":"2024-03-31T23:11:10.934521+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:29:54.904377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:10.544466+0000","last_change":"2024-03-31T23:09:15.204857+0000","last_active":"2024-03-31T23:11:10.544466+0000","last_peered":"2024-03-31T23:11:10.544466+0000","last_clean":"2024-03-31T23:11:10.544466+0000","last_became_active":"2024-03-31T23:09:15.204484+0000","last_became_peered":"2024-03-31T23:09:15.204484+0000","last_unstale":"2024-03-31T23:11:10.544466+0000","last_undegraded":"2024-03-31T23:11:10.544466+0000","last_fullsized":"2024-03-31T23:11:10.544466+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_clean_scrub_stamp":"2024-03-31T23:09:14.171251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T09:54:24.646918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"85'2","reported_seq":35,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:10.541624+0000","last_change":"2024-03-31T23:09:10.589922+0000","last_active":"2024-03-31T23:11:10.541624+0000","last_peered":"2024-03-31T23:11:10.541624+0000","last_clean":"2024-03-31T23:11:10.541624+0000","last_became_active":"2024-03-31T23:09:10.589569+0000","last_became_peered":"2024-03-31T23:09:10.589569+0000","last_unstale":"2024-03-31T23:11:10.541624+0000","last_undegraded":"2024-03-31T23:11:10.541624+0000","last_fullsized":"2024-03-31T23:11:10.541624+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_clean_scrub_stamp":"2024-03-31T23:09:09.564263+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:57:05.347344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"93'15","reported_seq":52,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:11.270163+0000","last_change":"2024-03-31T23:10:10.485015+0000","last_active":"2024-03-31T23:11:11.270163+0000","last_peered":"2024-03-31T23:11:11.270163+0000","last_clean":"2024-03-31T23:11:11.270163+0000","last_became_active":"2024-03-31T23:10:10.484663+0000","last_became_peered":"2024-03-31T23:10:10.484663+0000","last_unstale":"2024-03-31T23:11:11.270163+0000","last_undegraded":"2024-03-31T23:11:11.270163+0000","last_fullsized":"2024-03-31T23:11:11.270163+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T06:18:50.834711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,1],"acting":[5,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:10.544476+0000","last_change":"2024-03-31T23:09:15.193096+0000","last_active":"2024-03-31T23:11:10.544476+0000","last_peered":"2024-03-31T23:11:10.544476+0000","last_clean":"2024-03-31T23:11:10.544476+0000","last_became_active":"2024-03-31T23:09:15.192624+0000","last_became_peered":"2024-03-31T23:09:15.192624+0000","last_unstale":"2024-03-31T23:11:10.544476+0000","last_undegraded":"2024-03-31T23:11:10.544476+0000","last_fullsized":"2024-03-31T23:11:10.544476+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_clean_scrub_stamp":"2024-03-31T23:09:14.171251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:22:24.921346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"93'7","reported_seq":161,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:25.511651+0000","last_change":"2024-03-31T23:09:10.587209+0000","last_active":"2024-03-31T23:11:25.511651+0000","last_peered":"2024-03-31T23:11:25.511651+0000","last_clean":"2024-03-31T23:11:25.511651+0000","last_became_active":"2024-03-31T23:09:10.586486+0000","last_became_peered":"2024-03-31T23:09:10.586486+0000","last_unstale":"2024-03-31T23:11:25.511651+0000","last_undegraded":"2024-03-31T23:11:25.511651+0000","last_fullsized":"2024-03-31T23:11:25.511651+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_clean_scrub_stamp":"2024-03-31T23:09:09.564263+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:27:56.046414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1154,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":121,"num_read_kb":4,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"93'7","reported_seq":44,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:10.541409+0000","last_change":"2024-03-31T23:10:10.543713+0000","last_active":"2024-03-31T23:11:10.541409+0000","last_peered":"2024-03-31T23:11:10.541409+0000","last_clean":"2024-03-31T23:11:10.541409+0000","last_became_active":"2024-03-31T23:10:10.543324+0000","last_became_peered":"2024-03-31T23:10:10.543324+0000","last_unstale":"2024-03-31T23:11:10.541409+0000","last_undegraded":"2024-03-31T23:11:10.541409+0000","last_fullsized":"2024-03-31T23:11:10.541409+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:41:21.974325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,6],"acting":[8,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:10.544389+0000","last_change":"2024-03-31T23:09:15.194717+0000","last_active":"2024-03-31T23:11:10.544389+0000","last_peered":"2024-03-31T23:11:10.544389+0000","last_clean":"2024-03-31T23:11:10.544389+0000","last_became_active":"2024-03-31T23:09:15.194232+0000","last_became_peered":"2024-03-31T23:09:15.194232+0000","last_unstale":"2024-03-31T23:11:10.544389+0000","last_undegraded":"2024-03-31T23:11:10.544389+0000","last_fullsized":"2024-03-31T23:11:10.544389+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_clean_scrub_stamp":"2024-03-31T23:09:14.171251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:21:00.445232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"91'3","reported_seq":36,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:11.124484+0000","last_change":"2024-03-31T23:09:10.586346+0000","last_active":"2024-03-31T23:11:11.124484+0000","last_peered":"2024-03-31T23:11:11.124484+0000","last_clean":"2024-03-31T23:11:11.124484+0000","last_became_active":"2024-03-31T23:09:10.586043+0000","last_became_peered":"2024-03-31T23:09:10.586043+0000","last_unstale":"2024-03-31T23:11:11.124484+0000","last_undegraded":"2024-03-31T23:11:11.124484+0000","last_fullsized":"2024-03-31T23:11:11.124484+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_clean_scrub_stamp":"2024-03-31T23:09:09.564263+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:17:27.387786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"93'8","reported_seq":45,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:11.270118+0000","last_change":"2024-03-31T23:10:10.543353+0000","last_active":"2024-03-31T23:11:11.270118+0000","last_peered":"2024-03-31T23:11:11.270118+0000","last_clean":"2024-03-31T23:11:11.270118+0000","last_became_active":"2024-03-31T23:10:10.542961+0000","last_became_peered":"2024-03-31T23:10:10.542961+0000","last_unstale":"2024-03-31T23:11:11.270118+0000","last_undegraded":"2024-03-31T23:11:11.270118+0000","last_fullsized":"2024-03-31T23:11:11.270118+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:28:52.950551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3],"acting":[5,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":26,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:10.542953+0000","last_change":"2024-03-31T23:09:15.205855+0000","last_active":"2024-03-31T23:11:10.542953+0000","last_peered":"2024-03-31T23:11:10.542953+0000","last_clean":"2024-03-31T23:11:10.542953+0000","last_became_active":"2024-03-31T23:09:15.205359+0000","last_became_peered":"2024-03-31T23:09:15.205359+0000","last_unstale":"2024-03-31T23:11:10.542953+0000","last_undegraded":"2024-03-31T23:11:10.542953+0000","last_fullsized":"2024-03-31T23:11:10.542953+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_clean_scrub_stamp":"2024-03-31T23:09:14.171251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T09:39:32.461011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"91'2","reported_seq":35,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:10.715951+0000","last_change":"2024-03-31T23:09:10.583274+0000","last_active":"2024-03-31T23:11:10.715951+0000","last_peered":"2024-03-31T23:11:10.715951+0000","last_clean":"2024-03-31T23:11:10.715951+0000","last_became_active":"2024-03-31T23:09:10.582910+0000","last_became_peered":"2024-03-31T23:09:10.582910+0000","last_unstale":"2024-03-31T23:11:10.715951+0000","last_undegraded":"2024-03-31T23:11:10.715951+0000","last_fullsized":"2024-03-31T23:11:10.715951+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_clean_scrub_stamp":"2024-03-31T23:09:09.564263+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:16:32.251169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"93'8","reported_seq":39,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:10.540701+0000","last_change":"2024-03-31T23:09:11.178619+0000","last_active":"2024-03-31T23:11:10.540701+0000","last_peered":"2024-03-31T23:11:10.540701+0000","last_clean":"2024-03-31T23:11:10.540701+0000","last_became_active":"2024-03-31T23:09:11.178239+0000","last_became_peered":"2024-03-31T23:09:11.178239+0000","last_unstale":"2024-03-31T23:11:10.540701+0000","last_undegraded":"2024-03-31T23:11:10.540701+0000","last_fullsized":"2024-03-31T23:11:10.540701+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:28:58.863199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":26,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:11.122693+0000","last_change":"2024-03-31T23:09:15.205864+0000","last_active":"2024-03-31T23:11:11.122693+0000","last_peered":"2024-03-31T23:11:11.122693+0000","last_clean":"2024-03-31T23:11:11.122693+0000","last_became_active":"2024-03-31T23:09:15.204976+0000","last_became_peered":"2024-03-31T23:09:15.204976+0000","last_unstale":"2024-03-31T23:11:11.122693+0000","last_undegraded":"2024-03-31T23:11:11.122693+0000","last_fullsized":"2024-03-31T23:11:11.122693+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_clean_scrub_stamp":"2024-03-31T23:09:14.171251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:39:08.537827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"91'1","reported_seq":34,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:11.269487+0000","last_change":"2024-03-31T23:09:10.588407+0000","last_active":"2024-03-31T23:11:11.269487+0000","last_peered":"2024-03-31T23:11:11.269487+0000","last_clean":"2024-03-31T23:11:11.269487+0000","last_became_active":"2024-03-31T23:09:10.588109+0000","last_became_peered":"2024-03-31T23:09:10.588109+0000","last_unstale":"2024-03-31T23:11:11.269487+0000","last_undegraded":"2024-03-31T23:11:11.269487+0000","last_fullsized":"2024-03-31T23:11:11.269487+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_clean_scrub_stamp":"2024-03-31T23:09:09.564263+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:01:25.858834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"93'11","reported_seq":42,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:11.116446+0000","last_change":"2024-03-31T23:09:11.180452+0000","last_active":"2024-03-31T23:11:11.116446+0000","last_peered":"2024-03-31T23:11:11.116446+0000","last_clean":"2024-03-31T23:11:11.116446+0000","last_became_active":"2024-03-31T23:09:11.180117+0000","last_became_peered":"2024-03-31T23:09:11.180117+0000","last_unstale":"2024-03-31T23:11:11.116446+0000","last_undegraded":"2024-03-31T23:11:11.116446+0000","last_fullsized":"2024-03-31T23:11:11.116446+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:04:58.288981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:10.540663+0000","last_change":"2024-03-31T23:09:15.199669+0000","last_active":"2024-03-31T23:11:10.540663+0000","last_peered":"2024-03-31T23:11:10.540663+0000","last_clean":"2024-03-31T23:11:10.540663+0000","last_became_active":"2024-03-31T23:09:15.199262+0000","last_became_peered":"2024-03-31T23:09:15.199262+0000","last_unstale":"2024-03-31T23:11:10.540663+0000","last_undegraded":"2024-03-31T23:11:10.540663+0000","last_fullsized":"2024-03-31T23:11:10.540663+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_clean_scrub_stamp":"2024-03-31T23:09:14.171251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:44:02.975000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"95'5","reported_seq":158,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:28.514876+0000","last_change":"2024-03-31T23:09:10.583476+0000","last_active":"2024-03-31T23:11:28.514876+0000","last_peered":"2024-03-31T23:11:28.514876+0000","last_clean":"2024-03-31T23:11:28.514876+0000","last_became_active":"2024-03-31T23:09:10.582789+0000","last_became_peered":"2024-03-31T23:09:10.582789+0000","last_unstale":"2024-03-31T23:11:28.514876+0000","last_undegraded":"2024-03-31T23:11:28.514876+0000","last_fullsized":"2024-03-31T23:11:28.514876+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_clean_scrub_stamp":"2024-03-31T23:09:09.564263+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T10:52:35.478325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":120,"num_read_kb":120,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"93'10","reported_seq":41,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:10.934738+0000","last_change":"2024-03-31T23:09:11.171074+0000","last_active":"2024-03-31T23:11:10.934738+0000","last_peered":"2024-03-31T23:11:10.934738+0000","last_clean":"2024-03-31T23:11:10.934738+0000","last_became_active":"2024-03-31T23:09:11.169572+0000","last_became_peered":"2024-03-31T23:09:11.169572+0000","last_unstale":"2024-03-31T23:11:10.934738+0000","last_undegraded":"2024-03-31T23:11:10.934738+0000","last_fullsized":"2024-03-31T23:11:10.934738+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T07:15:34.833388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:11.116316+0000","last_change":"2024-03-31T23:09:15.203835+0000","last_active":"2024-03-31T23:11:11.116316+0000","last_peered":"2024-03-31T23:11:11.116316+0000","last_clean":"2024-03-31T23:11:11.116316+0000","last_became_active":"2024-03-31T23:09:15.200861+0000","last_became_peered":"2024-03-31T23:09:15.200861+0000","last_unstale":"2024-03-31T23:11:11.116316+0000","last_undegraded":"2024-03-31T23:11:11.116316+0000","last_fullsized":"2024-03-31T23:11:11.116316+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_clean_scrub_stamp":"2024-03-31T23:09:14.171251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:56:19.664708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"91'2","reported_seq":35,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:10.545911+0000","last_change":"2024-03-31T23:09:10.592601+0000","last_active":"2024-03-31T23:11:10.545911+0000","last_peered":"2024-03-31T23:11:10.545911+0000","last_clean":"2024-03-31T23:11:10.545911+0000","last_became_active":"2024-03-31T23:09:10.592345+0000","last_became_peered":"2024-03-31T23:09:10.592345+0000","last_unstale":"2024-03-31T23:11:10.545911+0000","last_undegraded":"2024-03-31T23:11:10.545911+0000","last_fullsized":"2024-03-31T23:11:10.545911+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_clean_scrub_stamp":"2024-03-31T23:09:09.564263+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:10:21.195544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"93'9","reported_seq":40,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:11.122701+0000","last_change":"2024-03-31T23:09:11.175305+0000","last_active":"2024-03-31T23:11:11.122701+0000","last_peered":"2024-03-31T23:11:11.122701+0000","last_clean":"2024-03-31T23:11:11.122701+0000","last_became_active":"2024-03-31T23:09:11.174970+0000","last_became_peered":"2024-03-31T23:09:11.174970+0000","last_unstale":"2024-03-31T23:11:11.122701+0000","last_undegraded":"2024-03-31T23:11:11.122701+0000","last_fullsized":"2024-03-31T23:11:11.122701+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:29:39.718127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:10.544982+0000","last_change":"2024-03-31T23:09:15.200365+0000","last_active":"2024-03-31T23:11:10.544982+0000","last_peered":"2024-03-31T23:11:10.544982+0000","last_clean":"2024-03-31T23:11:10.544982+0000","last_became_active":"2024-03-31T23:09:15.199830+0000","last_became_peered":"2024-03-31T23:09:15.199830+0000","last_unstale":"2024-03-31T23:11:10.544982+0000","last_undegraded":"2024-03-31T23:11:10.544982+0000","last_fullsized":"2024-03-31T23:11:10.544982+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_clean_scrub_stamp":"2024-03-31T23:09:14.171251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:37:30.048859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"91'4","reported_seq":38,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:11.270163+0000","last_change":"2024-03-31T23:09:10.582654+0000","last_active":"2024-03-31T23:11:11.270163+0000","last_peered":"2024-03-31T23:11:11.270163+0000","last_clean":"2024-03-31T23:11:11.270163+0000","last_became_active":"2024-03-31T23:09:10.582314+0000","last_became_peered":"2024-03-31T23:09:10.582314+0000","last_unstale":"2024-03-31T23:11:11.270163+0000","last_undegraded":"2024-03-31T23:11:11.270163+0000","last_fullsized":"2024-03-31T23:11:11.270163+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_clean_scrub_stamp":"2024-03-31T23:09:09.564263+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:51:34.236575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"93'11","reported_seq":42,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:10.934540+0000","last_change":"2024-03-31T23:09:11.172811+0000","last_active":"2024-03-31T23:11:10.934540+0000","last_peered":"2024-03-31T23:11:10.934540+0000","last_clean":"2024-03-31T23:11:10.934540+0000","last_became_active":"2024-03-31T23:09:11.172478+0000","last_became_peered":"2024-03-31T23:09:11.172478+0000","last_unstale":"2024-03-31T23:11:10.934540+0000","last_undegraded":"2024-03-31T23:11:10.934540+0000","last_fullsized":"2024-03-31T23:11:10.934540+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T10:00:09.825254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:11.116557+0000","last_change":"2024-03-31T23:09:15.205494+0000","last_active":"2024-03-31T23:11:11.116557+0000","last_peered":"2024-03-31T23:11:11.116557+0000","last_clean":"2024-03-31T23:11:11.116557+0000","last_became_active":"2024-03-31T23:09:15.205101+0000","last_became_peered":"2024-03-31T23:09:15.205101+0000","last_unstale":"2024-03-31T23:11:11.116557+0000","last_undegraded":"2024-03-31T23:11:11.116557+0000","last_fullsized":"2024-03-31T23:11:11.116557+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_clean_scrub_stamp":"2024-03-31T23:09:14.171251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:52:47.472296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"95'40","reported_seq":324,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:29.516503+0000","last_change":"2024-03-31T23:09:10.585611+0000","last_active":"2024-03-31T23:11:29.516503+0000","last_peered":"2024-03-31T23:11:29.516503+0000","last_clean":"2024-03-31T23:11:29.516503+0000","last_became_active":"2024-03-31T23:09:10.585287+0000","last_became_peered":"2024-03-31T23:09:10.585287+0000","last_unstale":"2024-03-31T23:11:29.516503+0000","last_undegraded":"2024-03-31T23:11:29.516503+0000","last_fullsized":"2024-03-31T23:11:29.516503+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_clean_scrub_stamp":"2024-03-31T23:09:09.564263+0000","objects_scrubbed":0,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:47:44.063831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":67389,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":250,"num_read_kb":191,"num_write":41,"num_write_kb":88,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"93'6","reported_seq":37,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:10.542061+0000","last_change":"2024-03-31T23:09:11.175812+0000","last_active":"2024-03-31T23:11:10.542061+0000","last_peered":"2024-03-31T23:11:10.542061+0000","last_clean":"2024-03-31T23:11:10.542061+0000","last_became_active":"2024-03-31T23:09:11.174972+0000","last_became_peered":"2024-03-31T23:09:11.174972+0000","last_unstale":"2024-03-31T23:11:10.542061+0000","last_undegraded":"2024-03-31T23:11:10.542061+0000","last_fullsized":"2024-03-31T23:11:10.542061+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:17:18.870340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:11.270348+0000","last_change":"2024-03-31T23:09:15.205018+0000","last_active":"2024-03-31T23:11:11.270348+0000","last_peered":"2024-03-31T23:11:11.270348+0000","last_clean":"2024-03-31T23:11:11.270348+0000","last_became_active":"2024-03-31T23:09:15.204510+0000","last_became_peered":"2024-03-31T23:09:15.204510+0000","last_unstale":"2024-03-31T23:11:11.270348+0000","last_undegraded":"2024-03-31T23:11:11.270348+0000","last_fullsized":"2024-03-31T23:11:11.270348+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_clean_scrub_stamp":"2024-03-31T23:09:14.171251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T07:48:59.819937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"91'6","reported_seq":40,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:10.544535+0000","last_change":"2024-03-31T23:09:10.590233+0000","last_active":"2024-03-31T23:11:10.544535+0000","last_peered":"2024-03-31T23:11:10.544535+0000","last_clean":"2024-03-31T23:11:10.544535+0000","last_became_active":"2024-03-31T23:09:10.589853+0000","last_became_peered":"2024-03-31T23:09:10.589853+0000","last_unstale":"2024-03-31T23:11:10.544535+0000","last_undegraded":"2024-03-31T23:11:10.544535+0000","last_fullsized":"2024-03-31T23:11:10.544535+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_clean_scrub_stamp":"2024-03-31T23:09:09.564263+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:10:54.980394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"93'9","reported_seq":40,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:10.716289+0000","last_change":"2024-03-31T23:09:11.177871+0000","last_active":"2024-03-31T23:11:10.716289+0000","last_peered":"2024-03-31T23:11:10.716289+0000","last_clean":"2024-03-31T23:11:10.716289+0000","last_became_active":"2024-03-31T23:09:11.176504+0000","last_became_peered":"2024-03-31T23:09:11.176504+0000","last_unstale":"2024-03-31T23:11:10.716289+0000","last_undegraded":"2024-03-31T23:11:10.716289+0000","last_fullsized":"2024-03-31T23:11:10.716289+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T06:09:08.797695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:10.542551+0000","last_change":"2024-03-31T23:09:15.207336+0000","last_active":"2024-03-31T23:11:10.542551+0000","last_peered":"2024-03-31T23:11:10.542551+0000","last_clean":"2024-03-31T23:11:10.542551+0000","last_became_active":"2024-03-31T23:09:15.206893+0000","last_became_peered":"2024-03-31T23:09:15.206893+0000","last_unstale":"2024-03-31T23:11:10.542551+0000","last_undegraded":"2024-03-31T23:11:10.542551+0000","last_fullsized":"2024-03-31T23:11:10.542551+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_clean_scrub_stamp":"2024-03-31T23:09:14.171251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:24:59.766421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"91'3","reported_seq":36,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:11.268980+0000","last_change":"2024-03-31T23:09:10.587099+0000","last_active":"2024-03-31T23:11:11.268980+0000","last_peered":"2024-03-31T23:11:11.268980+0000","last_clean":"2024-03-31T23:11:11.268980+0000","last_became_active":"2024-03-31T23:09:10.586429+0000","last_became_peered":"2024-03-31T23:09:10.586429+0000","last_unstale":"2024-03-31T23:11:11.268980+0000","last_undegraded":"2024-03-31T23:11:11.268980+0000","last_fullsized":"2024-03-31T23:11:11.268980+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_clean_scrub_stamp":"2024-03-31T23:09:09.564263+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:55:09.307378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"93'15","reported_seq":46,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:10.546149+0000","last_change":"2024-03-31T23:09:11.177661+0000","last_active":"2024-03-31T23:11:10.546149+0000","last_peered":"2024-03-31T23:11:10.546149+0000","last_clean":"2024-03-31T23:11:10.546149+0000","last_became_active":"2024-03-31T23:09:11.177314+0000","last_became_peered":"2024-03-31T23:09:11.177314+0000","last_unstale":"2024-03-31T23:11:10.546149+0000","last_undegraded":"2024-03-31T23:11:10.546149+0000","last_fullsized":"2024-03-31T23:11:10.546149+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T09:26:50.556837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":16,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:10.541733+0000","last_change":"2024-03-31T23:10:10.543526+0000","last_active":"2024-03-31T23:11:10.541733+0000","last_peered":"2024-03-31T23:11:10.541733+0000","last_clean":"2024-03-31T23:11:10.541733+0000","last_became_active":"2024-03-31T23:10:10.543105+0000","last_became_peered":"2024-03-31T23:10:10.543105+0000","last_unstale":"2024-03-31T23:11:10.541733+0000","last_undegraded":"2024-03-31T23:11:10.541733+0000","last_fullsized":"2024-03-31T23:11:10.541733+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_clean_scrub_stamp":"2024-03-31T23:09:14.171251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T09:16:44.529786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,7,0,3],"acting":[8,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.11","version":"95'6","reported_seq":161,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:27.546382+0000","last_change":"2024-03-31T23:09:10.586570+0000","last_active":"2024-03-31T23:11:27.546382+0000","last_peered":"2024-03-31T23:11:27.546382+0000","last_clean":"2024-03-31T23:11:27.546382+0000","last_became_active":"2024-03-31T23:09:10.586239+0000","last_became_peered":"2024-03-31T23:09:10.586239+0000","last_unstale":"2024-03-31T23:11:27.546382+0000","last_undegraded":"2024-03-31T23:11:27.546382+0000","last_fullsized":"2024-03-31T23:11:27.546382+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_clean_scrub_stamp":"2024-03-31T23:09:09.564263+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:23:37.182954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1613,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":123,"num_read_kb":6,"num_write":5,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"93'6","reported_seq":37,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:10.545691+0000","last_change":"2024-03-31T23:09:11.173294+0000","last_active":"2024-03-31T23:11:10.545691+0000","last_peered":"2024-03-31T23:11:10.545691+0000","last_clean":"2024-03-31T23:11:10.545691+0000","last_became_active":"2024-03-31T23:09:11.172926+0000","last_became_peered":"2024-03-31T23:09:11.172926+0000","last_unstale":"2024-03-31T23:11:10.545691+0000","last_undegraded":"2024-03-31T23:11:10.545691+0000","last_fullsized":"2024-03-31T23:11:10.545691+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T06:21:46.603381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:10.716781+0000","last_change":"2024-03-31T23:09:15.199893+0000","last_active":"2024-03-31T23:11:10.716781+0000","last_peered":"2024-03-31T23:11:10.716781+0000","last_clean":"2024-03-31T23:11:10.716781+0000","last_became_active":"2024-03-31T23:09:15.199517+0000","last_became_peered":"2024-03-31T23:09:15.199517+0000","last_unstale":"2024-03-31T23:11:10.716781+0000","last_undegraded":"2024-03-31T23:11:10.716781+0000","last_fullsized":"2024-03-31T23:11:10.716781+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_clean_scrub_stamp":"2024-03-31T23:09:14.171251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T07:48:35.261108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"91'15","reported_seq":49,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:10.541154+0000","last_change":"2024-03-31T23:09:10.588213+0000","last_active":"2024-03-31T23:11:10.541154+0000","last_peered":"2024-03-31T23:11:10.541154+0000","last_clean":"2024-03-31T23:11:10.541154+0000","last_became_active":"2024-03-31T23:09:10.587579+0000","last_became_peered":"2024-03-31T23:09:10.587579+0000","last_unstale":"2024-03-31T23:11:10.541154+0000","last_undegraded":"2024-03-31T23:11:10.541154+0000","last_fullsized":"2024-03-31T23:11:10.541154+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_clean_scrub_stamp":"2024-03-31T23:09:09.564263+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:00:17.883435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"93'6","reported_seq":37,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:11.268980+0000","last_change":"2024-03-31T23:09:11.182427+0000","last_active":"2024-03-31T23:11:11.268980+0000","last_peered":"2024-03-31T23:11:11.268980+0000","last_clean":"2024-03-31T23:11:11.268980+0000","last_became_active":"2024-03-31T23:09:11.182120+0000","last_became_peered":"2024-03-31T23:09:11.182120+0000","last_unstale":"2024-03-31T23:11:11.268980+0000","last_undegraded":"2024-03-31T23:11:11.268980+0000","last_fullsized":"2024-03-31T23:11:11.268980+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:17:48.728424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":26,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:11.269948+0000","last_change":"2024-03-31T23:09:15.207831+0000","last_active":"2024-03-31T23:11:11.269948+0000","last_peered":"2024-03-31T23:11:11.269948+0000","last_clean":"2024-03-31T23:11:11.269948+0000","last_became_active":"2024-03-31T23:09:15.207432+0000","last_became_peered":"2024-03-31T23:09:15.207432+0000","last_unstale":"2024-03-31T23:11:11.269948+0000","last_undegraded":"2024-03-31T23:11:11.269948+0000","last_fullsized":"2024-03-31T23:11:11.269948+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_clean_scrub_stamp":"2024-03-31T23:09:14.171251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:05:27.072054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"91'2","reported_seq":37,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:10.541954+0000","last_change":"2024-03-31T23:09:10.591483+0000","last_active":"2024-03-31T23:11:10.541954+0000","last_peered":"2024-03-31T23:11:10.541954+0000","last_clean":"2024-03-31T23:11:10.541954+0000","last_became_active":"2024-03-31T23:09:10.591173+0000","last_became_peered":"2024-03-31T23:09:10.591173+0000","last_unstale":"2024-03-31T23:11:10.541954+0000","last_undegraded":"2024-03-31T23:11:10.541954+0000","last_fullsized":"2024-03-31T23:11:10.541954+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_clean_scrub_stamp":"2024-03-31T23:09:09.564263+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T10:04:14.275875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"93'5","reported_seq":37,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:10.543194+0000","last_change":"2024-03-31T23:09:11.181171+0000","last_active":"2024-03-31T23:11:10.543194+0000","last_peered":"2024-03-31T23:11:10.543194+0000","last_clean":"2024-03-31T23:11:10.543194+0000","last_became_active":"2024-03-31T23:09:11.180847+0000","last_became_peered":"2024-03-31T23:09:11.180847+0000","last_unstale":"2024-03-31T23:11:10.543194+0000","last_undegraded":"2024-03-31T23:11:10.543194+0000","last_fullsized":"2024-03-31T23:11:10.543194+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:39:29.429009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":26,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:10.541346+0000","last_change":"2024-03-31T23:09:15.746941+0000","last_active":"2024-03-31T23:11:10.541346+0000","last_peered":"2024-03-31T23:11:10.541346+0000","last_clean":"2024-03-31T23:11:10.541346+0000","last_became_active":"2024-03-31T23:09:15.746352+0000","last_became_peered":"2024-03-31T23:09:15.746352+0000","last_unstale":"2024-03-31T23:11:10.541346+0000","last_undegraded":"2024-03-31T23:11:10.541346+0000","last_fullsized":"2024-03-31T23:11:10.541346+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_clean_scrub_stamp":"2024-03-31T23:09:14.171251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T06:21:43.482575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":34,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:11.116214+0000","last_change":"2024-03-31T23:09:10.593029+0000","last_active":"2024-03-31T23:11:11.116214+0000","last_peered":"2024-03-31T23:11:11.116214+0000","last_clean":"2024-03-31T23:11:11.116214+0000","last_became_active":"2024-03-31T23:09:10.592709+0000","last_became_peered":"2024-03-31T23:09:10.592709+0000","last_unstale":"2024-03-31T23:11:11.116214+0000","last_undegraded":"2024-03-31T23:11:11.116214+0000","last_fullsized":"2024-03-31T23:11:11.116214+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_clean_scrub_stamp":"2024-03-31T23:09:09.564263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:46:45.700207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"93'4","reported_seq":36,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:11.125675+0000","last_change":"2024-03-31T23:09:11.177857+0000","last_active":"2024-03-31T23:11:11.125675+0000","last_peered":"2024-03-31T23:11:11.125675+0000","last_clean":"2024-03-31T23:11:11.125675+0000","last_became_active":"2024-03-31T23:09:11.177513+0000","last_became_peered":"2024-03-31T23:09:11.177513+0000","last_unstale":"2024-03-31T23:11:11.125675+0000","last_undegraded":"2024-03-31T23:11:11.125675+0000","last_fullsized":"2024-03-31T23:11:11.125675+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T06:09:23.579208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":26,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:10.716480+0000","last_change":"2024-03-31T23:09:15.203412+0000","last_active":"2024-03-31T23:11:10.716480+0000","last_peered":"2024-03-31T23:11:10.716480+0000","last_clean":"2024-03-31T23:11:10.716480+0000","last_became_active":"2024-03-31T23:09:15.202983+0000","last_became_peered":"2024-03-31T23:09:15.202983+0000","last_unstale":"2024-03-31T23:11:10.716480+0000","last_undegraded":"2024-03-31T23:11:10.716480+0000","last_fullsized":"2024-03-31T23:11:10.716480+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_clean_scrub_stamp":"2024-03-31T23:09:14.171251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:57:46.357378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"91'1","reported_seq":35,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:10.545484+0000","last_change":"2024-03-31T23:09:10.593299+0000","last_active":"2024-03-31T23:11:10.545484+0000","last_peered":"2024-03-31T23:11:10.545484+0000","last_clean":"2024-03-31T23:11:10.545484+0000","last_became_active":"2024-03-31T23:09:10.593007+0000","last_became_peered":"2024-03-31T23:09:10.593007+0000","last_unstale":"2024-03-31T23:11:10.545484+0000","last_undegraded":"2024-03-31T23:11:10.545484+0000","last_fullsized":"2024-03-31T23:11:10.545484+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_clean_scrub_stamp":"2024-03-31T23:09:09.564263+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:15:07.199645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"93'7","reported_seq":39,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:11.125387+0000","last_change":"2024-03-31T23:09:11.176846+0000","last_active":"2024-03-31T23:11:11.125387+0000","last_peered":"2024-03-31T23:11:11.125387+0000","last_clean":"2024-03-31T23:11:11.125387+0000","last_became_active":"2024-03-31T23:09:11.176554+0000","last_became_peered":"2024-03-31T23:09:11.176554+0000","last_unstale":"2024-03-31T23:11:11.125387+0000","last_undegraded":"2024-03-31T23:11:11.125387+0000","last_fullsized":"2024-03-31T23:11:11.125387+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:25:23.639093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":26,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:10.716905+0000","last_change":"2024-03-31T23:09:15.746599+0000","last_active":"2024-03-31T23:11:10.716905+0000","last_peered":"2024-03-31T23:11:10.716905+0000","last_clean":"2024-03-31T23:11:10.716905+0000","last_became_active":"2024-03-31T23:09:15.745879+0000","last_became_peered":"2024-03-31T23:09:15.745879+0000","last_unstale":"2024-03-31T23:11:10.716905+0000","last_undegraded":"2024-03-31T23:11:10.716905+0000","last_fullsized":"2024-03-31T23:11:10.716905+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_clean_scrub_stamp":"2024-03-31T23:09:14.171251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T08:02:25.198610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"93'8","reported_seq":40,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:11.269210+0000","last_change":"2024-03-31T23:09:11.175546+0000","last_active":"2024-03-31T23:11:11.269210+0000","last_peered":"2024-03-31T23:11:11.269210+0000","last_clean":"2024-03-31T23:11:11.269210+0000","last_became_active":"2024-03-31T23:09:11.175253+0000","last_became_peered":"2024-03-31T23:09:11.175253+0000","last_unstale":"2024-03-31T23:11:11.269210+0000","last_undegraded":"2024-03-31T23:11:11.269210+0000","last_fullsized":"2024-03-31T23:11:11.269210+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T10:27:31.753102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"95'6","reported_seq":159,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:28.547474+0000","last_change":"2024-03-31T23:09:10.580888+0000","last_active":"2024-03-31T23:11:28.547474+0000","last_peered":"2024-03-31T23:11:28.547474+0000","last_clean":"2024-03-31T23:11:28.547474+0000","last_became_active":"2024-03-31T23:09:10.580557+0000","last_became_peered":"2024-03-31T23:09:10.580557+0000","last_unstale":"2024-03-31T23:11:28.547474+0000","last_undegraded":"2024-03-31T23:11:28.547474+0000","last_fullsized":"2024-03-31T23:11:28.547474+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_clean_scrub_stamp":"2024-03-31T23:09:09.564263+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:17:38.878570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":119,"num_read_kb":119,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"91'1","reported_seq":27,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:10.545528+0000","last_change":"2024-03-31T23:09:15.205282+0000","last_active":"2024-03-31T23:11:10.545528+0000","last_peered":"2024-03-31T23:11:10.545528+0000","last_clean":"2024-03-31T23:11:10.545528+0000","last_became_active":"2024-03-31T23:09:15.204873+0000","last_became_peered":"2024-03-31T23:09:15.204873+0000","last_unstale":"2024-03-31T23:11:10.545528+0000","last_undegraded":"2024-03-31T23:11:10.545528+0000","last_fullsized":"2024-03-31T23:11:10.545528+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_clean_scrub_stamp":"2024-03-31T23:09:14.171251+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:56:05.697685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"93'8","reported_seq":40,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:10.540856+0000","last_change":"2024-03-31T23:09:11.182155+0000","last_active":"2024-03-31T23:11:10.540856+0000","last_peered":"2024-03-31T23:11:10.540856+0000","last_clean":"2024-03-31T23:11:10.540856+0000","last_became_active":"2024-03-31T23:09:11.181829+0000","last_became_peered":"2024-03-31T23:09:11.181829+0000","last_unstale":"2024-03-31T23:11:10.540856+0000","last_undegraded":"2024-03-31T23:11:10.540856+0000","last_fullsized":"2024-03-31T23:11:10.540856+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:19:22.832440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"91'3","reported_seq":39,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:10.544924+0000","last_change":"2024-03-31T23:09:10.590451+0000","last_active":"2024-03-31T23:11:10.544924+0000","last_peered":"2024-03-31T23:11:10.544924+0000","last_clean":"2024-03-31T23:11:10.544924+0000","last_became_active":"2024-03-31T23:09:10.590123+0000","last_became_peered":"2024-03-31T23:09:10.590123+0000","last_unstale":"2024-03-31T23:11:10.544924+0000","last_undegraded":"2024-03-31T23:11:10.544924+0000","last_fullsized":"2024-03-31T23:11:10.544924+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_clean_scrub_stamp":"2024-03-31T23:09:09.564263+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:27:02.724456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":369,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:10.545255+0000","last_change":"2024-03-31T23:07:57.920942+0000","last_active":"2024-03-31T23:11:10.545255+0000","last_peered":"2024-03-31T23:11:10.545255+0000","last_clean":"2024-03-31T23:11:10.545255+0000","last_became_active":"2024-03-31T23:07:57.920381+0000","last_became_peered":"2024-03-31T23:07:57.920381+0000","last_unstale":"2024-03-31T23:11:10.545255+0000","last_undegraded":"2024-03-31T23:11:10.545255+0000","last_fullsized":"2024-03-31T23:11:10.545255+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:04:33.508180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:04:33.508180+0000","last_clean_scrub_stamp":"2024-03-31T23:04:33.508180+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:20:11.211403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:11.126466+0000","last_change":"2024-03-31T23:11:11.126137+0000","last_active":"2024-03-31T23:11:11.126466+0000","last_peered":"2024-03-31T23:11:11.126466+0000","last_clean":"2024-03-31T23:11:11.126466+0000","last_became_active":"2024-03-31T23:11:11.125719+0000","last_became_peered":"2024-03-31T23:11:11.125719+0000","last_unstale":"2024-03-31T23:11:11.126466+0000","last_undegraded":"2024-03-31T23:11:11.126466+0000","last_fullsized":"2024-03-31T23:11:11.126466+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_clean_scrub_stamp":"2024-03-31T23:09:14.171251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:34:55.444516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,6,10],"acting":[8,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1","version":"93'5","reported_seq":37,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:10.716368+0000","last_change":"2024-03-31T23:09:11.175336+0000","last_active":"2024-03-31T23:11:10.716368+0000","last_peered":"2024-03-31T23:11:10.716368+0000","last_clean":"2024-03-31T23:11:10.716368+0000","last_became_active":"2024-03-31T23:09:11.174768+0000","last_became_peered":"2024-03-31T23:09:11.174768+0000","last_unstale":"2024-03-31T23:11:10.716368+0000","last_undegraded":"2024-03-31T23:11:10.716368+0000","last_fullsized":"2024-03-31T23:11:10.716368+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:09:11.678744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"91'3","reported_seq":37,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:10.545473+0000","last_change":"2024-03-31T23:09:10.591807+0000","last_active":"2024-03-31T23:11:10.545473+0000","last_peered":"2024-03-31T23:11:10.545473+0000","last_clean":"2024-03-31T23:11:10.545473+0000","last_became_active":"2024-03-31T23:09:10.591407+0000","last_became_peered":"2024-03-31T23:09:10.591407+0000","last_unstale":"2024-03-31T23:11:10.545473+0000","last_undegraded":"2024-03-31T23:11:10.545473+0000","last_fullsized":"2024-03-31T23:11:10.545473+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_clean_scrub_stamp":"2024-03-31T23:09:09.564263+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T08:46:33.660717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":26,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:10.543100+0000","last_change":"2024-03-31T23:09:15.203879+0000","last_active":"2024-03-31T23:11:10.543100+0000","last_peered":"2024-03-31T23:11:10.543100+0000","last_clean":"2024-03-31T23:11:10.543100+0000","last_became_active":"2024-03-31T23:09:15.203412+0000","last_became_peered":"2024-03-31T23:09:15.203412+0000","last_unstale":"2024-03-31T23:11:10.543100+0000","last_undegraded":"2024-03-31T23:11:10.543100+0000","last_fullsized":"2024-03-31T23:11:10.543100+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_clean_scrub_stamp":"2024-03-31T23:09:14.171251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:03:54.486038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"91'1","reported_seq":36,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:10.542010+0000","last_change":"2024-03-31T23:09:10.596404+0000","last_active":"2024-03-31T23:11:10.542010+0000","last_peered":"2024-03-31T23:11:10.542010+0000","last_clean":"2024-03-31T23:11:10.542010+0000","last_became_active":"2024-03-31T23:09:10.596031+0000","last_became_peered":"2024-03-31T23:09:10.596031+0000","last_unstale":"2024-03-31T23:11:10.542010+0000","last_undegraded":"2024-03-31T23:11:10.542010+0000","last_fullsized":"2024-03-31T23:11:10.542010+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_clean_scrub_stamp":"2024-03-31T23:09:09.564263+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:30:31.904241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"93'9","reported_seq":41,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:10.716744+0000","last_change":"2024-03-31T23:09:11.164300+0000","last_active":"2024-03-31T23:11:10.716744+0000","last_peered":"2024-03-31T23:11:10.716744+0000","last_clean":"2024-03-31T23:11:10.716744+0000","last_became_active":"2024-03-31T23:09:11.163960+0000","last_became_peered":"2024-03-31T23:09:11.163960+0000","last_unstale":"2024-03-31T23:11:10.716744+0000","last_undegraded":"2024-03-31T23:11:10.716744+0000","last_fullsized":"2024-03-31T23:11:10.716744+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T10:37:30.512216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":26,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:10.540599+0000","last_change":"2024-03-31T23:09:15.746426+0000","last_active":"2024-03-31T23:11:10.540599+0000","last_peered":"2024-03-31T23:11:10.540599+0000","last_clean":"2024-03-31T23:11:10.540599+0000","last_became_active":"2024-03-31T23:09:15.745887+0000","last_became_peered":"2024-03-31T23:09:15.745887+0000","last_unstale":"2024-03-31T23:11:10.540599+0000","last_undegraded":"2024-03-31T23:11:10.540599+0000","last_fullsized":"2024-03-31T23:11:10.540599+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_clean_scrub_stamp":"2024-03-31T23:09:14.171251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:41:52.005300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"95'10","reported_seq":44,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:23.633720+0000","last_change":"2024-03-31T23:09:10.590631+0000","last_active":"2024-03-31T23:11:23.633720+0000","last_peered":"2024-03-31T23:11:23.633720+0000","last_clean":"2024-03-31T23:11:23.633720+0000","last_became_active":"2024-03-31T23:09:10.590211+0000","last_became_peered":"2024-03-31T23:09:10.590211+0000","last_unstale":"2024-03-31T23:11:23.633720+0000","last_undegraded":"2024-03-31T23:11:23.633720+0000","last_fullsized":"2024-03-31T23:11:23.633720+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_clean_scrub_stamp":"2024-03-31T23:09:09.564263+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T08:51:24.978892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"93'5","reported_seq":37,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:10.541392+0000","last_change":"2024-03-31T23:09:11.177353+0000","last_active":"2024-03-31T23:11:10.541392+0000","last_peered":"2024-03-31T23:11:10.541392+0000","last_clean":"2024-03-31T23:11:10.541392+0000","last_became_active":"2024-03-31T23:09:11.176996+0000","last_became_peered":"2024-03-31T23:09:11.176996+0000","last_unstale":"2024-03-31T23:11:10.541392+0000","last_undegraded":"2024-03-31T23:11:10.541392+0000","last_fullsized":"2024-03-31T23:11:10.541392+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:20:52.486281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":26,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:10.544240+0000","last_change":"2024-03-31T23:09:15.198528+0000","last_active":"2024-03-31T23:11:10.544240+0000","last_peered":"2024-03-31T23:11:10.544240+0000","last_clean":"2024-03-31T23:11:10.544240+0000","last_became_active":"2024-03-31T23:09:15.197938+0000","last_became_peered":"2024-03-31T23:09:15.197938+0000","last_unstale":"2024-03-31T23:11:10.544240+0000","last_undegraded":"2024-03-31T23:11:10.544240+0000","last_fullsized":"2024-03-31T23:11:10.544240+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_clean_scrub_stamp":"2024-03-31T23:09:14.171251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:45:46.567268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"91'4","reported_seq":38,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:10.545572+0000","last_change":"2024-03-31T23:09:10.584779+0000","last_active":"2024-03-31T23:11:10.545572+0000","last_peered":"2024-03-31T23:11:10.545572+0000","last_clean":"2024-03-31T23:11:10.545572+0000","last_became_active":"2024-03-31T23:09:10.584400+0000","last_became_peered":"2024-03-31T23:09:10.584400+0000","last_unstale":"2024-03-31T23:11:10.545572+0000","last_undegraded":"2024-03-31T23:11:10.545572+0000","last_fullsized":"2024-03-31T23:11:10.545572+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_clean_scrub_stamp":"2024-03-31T23:09:09.564263+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:10:10.095704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"93'7","reported_seq":45,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:11.122710+0000","last_change":"2024-03-31T23:10:10.543538+0000","last_active":"2024-03-31T23:11:11.122710+0000","last_peered":"2024-03-31T23:11:11.122710+0000","last_clean":"2024-03-31T23:11:11.122710+0000","last_became_active":"2024-03-31T23:10:10.543083+0000","last_became_peered":"2024-03-31T23:10:10.543083+0000","last_unstale":"2024-03-31T23:11:11.122710+0000","last_undegraded":"2024-03-31T23:11:11.122710+0000","last_fullsized":"2024-03-31T23:11:11.122710+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:38:50.656801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:10.545224+0000","last_change":"2024-03-31T23:09:15.746525+0000","last_active":"2024-03-31T23:11:10.545224+0000","last_peered":"2024-03-31T23:11:10.545224+0000","last_clean":"2024-03-31T23:11:10.545224+0000","last_became_active":"2024-03-31T23:09:15.746126+0000","last_became_peered":"2024-03-31T23:09:15.746126+0000","last_unstale":"2024-03-31T23:11:10.545224+0000","last_undegraded":"2024-03-31T23:11:10.545224+0000","last_fullsized":"2024-03-31T23:11:10.545224+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_clean_scrub_stamp":"2024-03-31T23:09:14.171251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:04:57.313615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"91'2","reported_seq":36,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:10.934097+0000","last_change":"2024-03-31T23:09:10.591469+0000","last_active":"2024-03-31T23:11:10.934097+0000","last_peered":"2024-03-31T23:11:10.934097+0000","last_clean":"2024-03-31T23:11:10.934097+0000","last_became_active":"2024-03-31T23:09:10.591166+0000","last_became_peered":"2024-03-31T23:09:10.591166+0000","last_unstale":"2024-03-31T23:11:10.934097+0000","last_undegraded":"2024-03-31T23:11:10.934097+0000","last_fullsized":"2024-03-31T23:11:10.934097+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_clean_scrub_stamp":"2024-03-31T23:09:09.564263+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T10:30:40.279041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"93'2","reported_seq":34,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:10.541228+0000","last_change":"2024-03-31T23:09:11.174471+0000","last_active":"2024-03-31T23:11:10.541228+0000","last_peered":"2024-03-31T23:11:10.541228+0000","last_clean":"2024-03-31T23:11:10.541228+0000","last_became_active":"2024-03-31T23:09:11.174154+0000","last_became_peered":"2024-03-31T23:09:11.174154+0000","last_unstale":"2024-03-31T23:11:10.541228+0000","last_undegraded":"2024-03-31T23:11:10.541228+0000","last_fullsized":"2024-03-31T23:11:10.541228+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T06:10:14.727009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:11.117002+0000","last_change":"2024-03-31T23:09:15.201416+0000","last_active":"2024-03-31T23:11:11.117002+0000","last_peered":"2024-03-31T23:11:11.117002+0000","last_clean":"2024-03-31T23:11:11.117002+0000","last_became_active":"2024-03-31T23:09:15.200987+0000","last_became_peered":"2024-03-31T23:09:15.200987+0000","last_unstale":"2024-03-31T23:11:11.117002+0000","last_undegraded":"2024-03-31T23:11:11.117002+0000","last_fullsized":"2024-03-31T23:11:11.117002+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_clean_scrub_stamp":"2024-03-31T23:09:14.171251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T10:56:54.872988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"91'3","reported_seq":37,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:10.540834+0000","last_change":"2024-03-31T23:09:10.594035+0000","last_active":"2024-03-31T23:11:10.540834+0000","last_peered":"2024-03-31T23:11:10.540834+0000","last_clean":"2024-03-31T23:11:10.540834+0000","last_became_active":"2024-03-31T23:09:10.593710+0000","last_became_peered":"2024-03-31T23:09:10.593710+0000","last_unstale":"2024-03-31T23:11:10.540834+0000","last_undegraded":"2024-03-31T23:11:10.540834+0000","last_fullsized":"2024-03-31T23:11:10.540834+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_clean_scrub_stamp":"2024-03-31T23:09:09.564263+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:46:48.580244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"93'9","reported_seq":41,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:10.542532+0000","last_change":"2024-03-31T23:09:11.170501+0000","last_active":"2024-03-31T23:11:10.542532+0000","last_peered":"2024-03-31T23:11:10.542532+0000","last_clean":"2024-03-31T23:11:10.542532+0000","last_became_active":"2024-03-31T23:09:11.170068+0000","last_became_peered":"2024-03-31T23:09:11.170068+0000","last_unstale":"2024-03-31T23:11:10.542532+0000","last_undegraded":"2024-03-31T23:11:10.542532+0000","last_fullsized":"2024-03-31T23:11:10.542532+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T06:05:53.488957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"91'1","reported_seq":26,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:10.544928+0000","last_change":"2024-03-31T23:09:15.208092+0000","last_active":"2024-03-31T23:11:10.544928+0000","last_peered":"2024-03-31T23:11:10.544928+0000","last_clean":"2024-03-31T23:11:10.544928+0000","last_became_active":"2024-03-31T23:09:15.207750+0000","last_became_peered":"2024-03-31T23:09:15.207750+0000","last_unstale":"2024-03-31T23:11:10.544928+0000","last_undegraded":"2024-03-31T23:11:10.544928+0000","last_fullsized":"2024-03-31T23:11:10.544928+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_clean_scrub_stamp":"2024-03-31T23:09:14.171251+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:24:39.649569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"95'2","reported_seq":16,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:10.548189+0000","last_change":"2024-03-31T23:11:10.547781+0000","last_active":"2024-03-31T23:11:10.548189+0000","last_peered":"2024-03-31T23:11:10.548189+0000","last_clean":"2024-03-31T23:11:10.548189+0000","last_became_active":"2024-03-31T23:11:10.545156+0000","last_became_peered":"2024-03-31T23:11:10.545156+0000","last_unstale":"2024-03-31T23:11:10.548189+0000","last_undegraded":"2024-03-31T23:11:10.548189+0000","last_fullsized":"2024-03-31T23:11:10.548189+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_clean_scrub_stamp":"2024-03-31T23:09:09.564263+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T08:14:11.270910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,8],"acting":[11,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.b","version":"93'8","reported_seq":40,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:10.541041+0000","last_change":"2024-03-31T23:09:11.175169+0000","last_active":"2024-03-31T23:11:10.541041+0000","last_peered":"2024-03-31T23:11:10.541041+0000","last_clean":"2024-03-31T23:11:10.541041+0000","last_became_active":"2024-03-31T23:09:11.174849+0000","last_became_peered":"2024-03-31T23:09:11.174849+0000","last_unstale":"2024-03-31T23:11:10.541041+0000","last_undegraded":"2024-03-31T23:11:10.541041+0000","last_fullsized":"2024-03-31T23:11:10.541041+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:58:06.814034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:10.545200+0000","last_change":"2024-03-31T23:09:15.746423+0000","last_active":"2024-03-31T23:11:10.545200+0000","last_peered":"2024-03-31T23:11:10.545200+0000","last_clean":"2024-03-31T23:11:10.545200+0000","last_became_active":"2024-03-31T23:09:15.745942+0000","last_became_peered":"2024-03-31T23:09:15.745942+0000","last_unstale":"2024-03-31T23:11:10.545200+0000","last_undegraded":"2024-03-31T23:11:10.545200+0000","last_fullsized":"2024-03-31T23:11:10.545200+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_clean_scrub_stamp":"2024-03-31T23:09:14.171251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:56:02.903212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"91'4","reported_seq":17,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:10.550839+0000","last_change":"2024-03-31T23:11:10.550839+0000","last_active":"2024-03-31T23:11:10.550839+0000","last_peered":"2024-03-31T23:11:10.550839+0000","last_clean":"2024-03-31T23:11:10.550839+0000","last_became_active":"2024-03-31T23:11:10.546702+0000","last_became_peered":"2024-03-31T23:11:10.546702+0000","last_unstale":"2024-03-31T23:11:10.550839+0000","last_undegraded":"2024-03-31T23:11:10.550839+0000","last_fullsized":"2024-03-31T23:11:10.550839+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_clean_scrub_stamp":"2024-03-31T23:09:09.564263+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:39:13.293806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,7],"acting":[11,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.a","version":"93'5","reported_seq":37,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:10.545517+0000","last_change":"2024-03-31T23:09:11.170440+0000","last_active":"2024-03-31T23:11:10.545517+0000","last_peered":"2024-03-31T23:11:10.545517+0000","last_clean":"2024-03-31T23:11:10.545517+0000","last_became_active":"2024-03-31T23:09:11.170081+0000","last_became_peered":"2024-03-31T23:09:11.170081+0000","last_unstale":"2024-03-31T23:11:10.545517+0000","last_undegraded":"2024-03-31T23:11:10.545517+0000","last_fullsized":"2024-03-31T23:11:10.545517+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:53:31.429822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":26,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:10.934387+0000","last_change":"2024-03-31T23:09:15.207146+0000","last_active":"2024-03-31T23:11:10.934387+0000","last_peered":"2024-03-31T23:11:10.934387+0000","last_clean":"2024-03-31T23:11:10.934387+0000","last_became_active":"2024-03-31T23:09:15.206769+0000","last_became_peered":"2024-03-31T23:09:15.206769+0000","last_unstale":"2024-03-31T23:11:10.934387+0000","last_undegraded":"2024-03-31T23:11:10.934387+0000","last_fullsized":"2024-03-31T23:11:10.934387+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_clean_scrub_stamp":"2024-03-31T23:09:14.171251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T07:42:12.961109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"91'3","reported_seq":38,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:10.544591+0000","last_change":"2024-03-31T23:09:10.587291+0000","last_active":"2024-03-31T23:11:10.544591+0000","last_peered":"2024-03-31T23:11:10.544591+0000","last_clean":"2024-03-31T23:11:10.544591+0000","last_became_active":"2024-03-31T23:09:10.585992+0000","last_became_peered":"2024-03-31T23:09:10.585992+0000","last_unstale":"2024-03-31T23:11:10.544591+0000","last_undegraded":"2024-03-31T23:11:10.544591+0000","last_fullsized":"2024-03-31T23:11:10.544591+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_clean_scrub_stamp":"2024-03-31T23:09:09.564263+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:28:00.786268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"93'7","reported_seq":39,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:10.545804+0000","last_change":"2024-03-31T23:09:11.179369+0000","last_active":"2024-03-31T23:11:10.545804+0000","last_peered":"2024-03-31T23:11:10.545804+0000","last_clean":"2024-03-31T23:11:10.545804+0000","last_became_active":"2024-03-31T23:09:11.178776+0000","last_became_peered":"2024-03-31T23:09:11.178776+0000","last_unstale":"2024-03-31T23:11:10.545804+0000","last_undegraded":"2024-03-31T23:11:10.545804+0000","last_fullsized":"2024-03-31T23:11:10.545804+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T07:56:05.201338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:11.122108+0000","last_change":"2024-03-31T23:09:15.201462+0000","last_active":"2024-03-31T23:11:11.122108+0000","last_peered":"2024-03-31T23:11:11.122108+0000","last_clean":"2024-03-31T23:11:11.122108+0000","last_became_active":"2024-03-31T23:09:15.201067+0000","last_became_peered":"2024-03-31T23:09:15.201067+0000","last_unstale":"2024-03-31T23:11:11.122108+0000","last_undegraded":"2024-03-31T23:11:11.122108+0000","last_fullsized":"2024-03-31T23:11:11.122108+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_clean_scrub_stamp":"2024-03-31T23:09:14.171251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:54:29.339456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"85'1","reported_seq":36,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:11.269181+0000","last_change":"2024-03-31T23:09:10.584752+0000","last_active":"2024-03-31T23:11:11.269181+0000","last_peered":"2024-03-31T23:11:11.269181+0000","last_clean":"2024-03-31T23:11:11.269181+0000","last_became_active":"2024-03-31T23:09:10.584432+0000","last_became_peered":"2024-03-31T23:09:10.584432+0000","last_unstale":"2024-03-31T23:11:11.269181+0000","last_undegraded":"2024-03-31T23:11:11.269181+0000","last_fullsized":"2024-03-31T23:11:11.269181+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_clean_scrub_stamp":"2024-03-31T23:09:09.564263+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T10:50:53.889719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"93'6","reported_seq":44,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:10.546403+0000","last_change":"2024-03-31T23:10:10.543993+0000","last_active":"2024-03-31T23:11:10.546403+0000","last_peered":"2024-03-31T23:11:10.546403+0000","last_clean":"2024-03-31T23:11:10.546403+0000","last_became_active":"2024-03-31T23:10:10.543627+0000","last_became_peered":"2024-03-31T23:10:10.543627+0000","last_unstale":"2024-03-31T23:11:10.546403+0000","last_undegraded":"2024-03-31T23:11:10.546403+0000","last_fullsized":"2024-03-31T23:11:10.546403+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T08:46:41.221122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":26,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:11.269712+0000","last_change":"2024-03-31T23:09:15.197271+0000","last_active":"2024-03-31T23:11:11.269712+0000","last_peered":"2024-03-31T23:11:11.269712+0000","last_clean":"2024-03-31T23:11:11.269712+0000","last_became_active":"2024-03-31T23:09:15.196339+0000","last_became_peered":"2024-03-31T23:09:15.196339+0000","last_unstale":"2024-03-31T23:11:11.269712+0000","last_undegraded":"2024-03-31T23:11:11.269712+0000","last_fullsized":"2024-03-31T23:11:11.269712+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_clean_scrub_stamp":"2024-03-31T23:09:14.171251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:44:13.044278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"85'2","reported_seq":36,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:10.716161+0000","last_change":"2024-03-31T23:09:10.595131+0000","last_active":"2024-03-31T23:11:10.716161+0000","last_peered":"2024-03-31T23:11:10.716161+0000","last_clean":"2024-03-31T23:11:10.716161+0000","last_became_active":"2024-03-31T23:09:10.594740+0000","last_became_peered":"2024-03-31T23:09:10.594740+0000","last_unstale":"2024-03-31T23:11:10.716161+0000","last_undegraded":"2024-03-31T23:11:10.716161+0000","last_fullsized":"2024-03-31T23:11:10.716161+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_clean_scrub_stamp":"2024-03-31T23:09:09.564263+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:52:41.849621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"93'9","reported_seq":41,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:10.545854+0000","last_change":"2024-03-31T23:09:11.177530+0000","last_active":"2024-03-31T23:11:10.545854+0000","last_peered":"2024-03-31T23:11:10.545854+0000","last_clean":"2024-03-31T23:11:10.545854+0000","last_became_active":"2024-03-31T23:09:11.176972+0000","last_became_peered":"2024-03-31T23:09:11.176972+0000","last_unstale":"2024-03-31T23:11:10.545854+0000","last_undegraded":"2024-03-31T23:11:10.545854+0000","last_fullsized":"2024-03-31T23:11:10.545854+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T07:57:21.094911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"93'9","reported_seq":41,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:10.542629+0000","last_change":"2024-03-31T23:09:11.172792+0000","last_active":"2024-03-31T23:11:10.542629+0000","last_peered":"2024-03-31T23:11:10.542629+0000","last_clean":"2024-03-31T23:11:10.542629+0000","last_became_active":"2024-03-31T23:09:11.172463+0000","last_became_peered":"2024-03-31T23:09:11.172463+0000","last_unstale":"2024-03-31T23:11:10.542629+0000","last_undegraded":"2024-03-31T23:11:10.542629+0000","last_fullsized":"2024-03-31T23:11:10.542629+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:07:28.789783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"93'12","reported_seq":44,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:10.544865+0000","last_change":"2024-03-31T23:09:11.174331+0000","last_active":"2024-03-31T23:11:10.544865+0000","last_peered":"2024-03-31T23:11:10.544865+0000","last_clean":"2024-03-31T23:11:10.544865+0000","last_became_active":"2024-03-31T23:09:11.174001+0000","last_became_peered":"2024-03-31T23:09:11.174001+0000","last_unstale":"2024-03-31T23:11:10.544865+0000","last_undegraded":"2024-03-31T23:11:10.544865+0000","last_fullsized":"2024-03-31T23:11:10.544865+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:26:28.005406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"93'8","reported_seq":40,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:10.544540+0000","last_change":"2024-03-31T23:09:11.180047+0000","last_active":"2024-03-31T23:11:10.544540+0000","last_peered":"2024-03-31T23:11:10.544540+0000","last_clean":"2024-03-31T23:11:10.544540+0000","last_became_active":"2024-03-31T23:09:11.179691+0000","last_became_peered":"2024-03-31T23:09:11.179691+0000","last_unstale":"2024-03-31T23:11:10.544540+0000","last_undegraded":"2024-03-31T23:11:10.544540+0000","last_fullsized":"2024-03-31T23:11:10.544540+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:28:05.083710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"93'7","reported_seq":39,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:11.122493+0000","last_change":"2024-03-31T23:09:11.177642+0000","last_active":"2024-03-31T23:11:11.122493+0000","last_peered":"2024-03-31T23:11:11.122493+0000","last_clean":"2024-03-31T23:11:11.122493+0000","last_became_active":"2024-03-31T23:09:11.177261+0000","last_became_peered":"2024-03-31T23:09:11.177261+0000","last_unstale":"2024-03-31T23:11:11.122493+0000","last_undegraded":"2024-03-31T23:11:11.122493+0000","last_fullsized":"2024-03-31T23:11:11.122493+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T07:10:43.616863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"93'4","reported_seq":36,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:10.716630+0000","last_change":"2024-03-31T23:09:11.173606+0000","last_active":"2024-03-31T23:11:10.716630+0000","last_peered":"2024-03-31T23:11:10.716630+0000","last_clean":"2024-03-31T23:11:10.716630+0000","last_became_active":"2024-03-31T23:09:11.173304+0000","last_became_peered":"2024-03-31T23:09:11.173304+0000","last_unstale":"2024-03-31T23:11:10.716630+0000","last_undegraded":"2024-03-31T23:11:10.716630+0000","last_fullsized":"2024-03-31T23:11:10.716630+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:53:06.112181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"93'6","reported_seq":38,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:10.543432+0000","last_change":"2024-03-31T23:09:11.176628+0000","last_active":"2024-03-31T23:11:10.543432+0000","last_peered":"2024-03-31T23:11:10.543432+0000","last_clean":"2024-03-31T23:11:10.543432+0000","last_became_active":"2024-03-31T23:09:11.176318+0000","last_became_peered":"2024-03-31T23:09:11.176318+0000","last_unstale":"2024-03-31T23:11:10.543432+0000","last_undegraded":"2024-03-31T23:11:10.543432+0000","last_fullsized":"2024-03-31T23:11:10.543432+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:06:40.723675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"93'15","reported_seq":47,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:10.544863+0000","last_change":"2024-03-31T23:09:11.169318+0000","last_active":"2024-03-31T23:11:10.544863+0000","last_peered":"2024-03-31T23:11:10.544863+0000","last_clean":"2024-03-31T23:11:10.544863+0000","last_became_active":"2024-03-31T23:09:11.169026+0000","last_became_peered":"2024-03-31T23:09:11.169026+0000","last_unstale":"2024-03-31T23:11:10.544863+0000","last_undegraded":"2024-03-31T23:11:10.544863+0000","last_fullsized":"2024-03-31T23:11:10.544863+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:02:18.562854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"93'2","reported_seq":34,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:10.541448+0000","last_change":"2024-03-31T23:09:11.184670+0000","last_active":"2024-03-31T23:11:10.541448+0000","last_peered":"2024-03-31T23:11:10.541448+0000","last_clean":"2024-03-31T23:11:10.541448+0000","last_became_active":"2024-03-31T23:09:11.184354+0000","last_became_peered":"2024-03-31T23:09:11.184354+0000","last_unstale":"2024-03-31T23:11:10.541448+0000","last_undegraded":"2024-03-31T23:11:10.541448+0000","last_fullsized":"2024-03-31T23:11:10.541448+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:16:14.354947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"93'6","reported_seq":38,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:11.269351+0000","last_change":"2024-03-31T23:09:11.180991+0000","last_active":"2024-03-31T23:11:11.269351+0000","last_peered":"2024-03-31T23:11:11.269351+0000","last_clean":"2024-03-31T23:11:11.269351+0000","last_became_active":"2024-03-31T23:09:11.180667+0000","last_became_peered":"2024-03-31T23:09:11.180667+0000","last_unstale":"2024-03-31T23:11:11.269351+0000","last_undegraded":"2024-03-31T23:11:11.269351+0000","last_fullsized":"2024-03-31T23:11:11.269351+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T06:35:33.157176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"93'8","reported_seq":40,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:10.540877+0000","last_change":"2024-03-31T23:09:11.182025+0000","last_active":"2024-03-31T23:11:10.540877+0000","last_peered":"2024-03-31T23:11:10.540877+0000","last_clean":"2024-03-31T23:11:10.540877+0000","last_became_active":"2024-03-31T23:09:11.179310+0000","last_became_peered":"2024-03-31T23:09:11.179310+0000","last_unstale":"2024-03-31T23:11:10.540877+0000","last_undegraded":"2024-03-31T23:11:10.540877+0000","last_fullsized":"2024-03-31T23:11:10.540877+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T10:21:40.784738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"93'10","reported_seq":48,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:10.934477+0000","last_change":"2024-03-31T23:10:10.485623+0000","last_active":"2024-03-31T23:11:10.934477+0000","last_peered":"2024-03-31T23:11:10.934477+0000","last_clean":"2024-03-31T23:11:10.934477+0000","last_became_active":"2024-03-31T23:10:10.485248+0000","last_became_peered":"2024-03-31T23:10:10.485248+0000","last_unstale":"2024-03-31T23:11:10.934477+0000","last_undegraded":"2024-03-31T23:11:10.934477+0000","last_fullsized":"2024-03-31T23:11:10.934477+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:59:04.668598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,6],"acting":[1,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"93'11","reported_seq":43,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:10.716676+0000","last_change":"2024-03-31T23:09:11.172585+0000","last_active":"2024-03-31T23:11:10.716676+0000","last_peered":"2024-03-31T23:11:10.716676+0000","last_clean":"2024-03-31T23:11:10.716676+0000","last_became_active":"2024-03-31T23:09:11.172266+0000","last_became_peered":"2024-03-31T23:09:11.172266+0000","last_unstale":"2024-03-31T23:11:10.716676+0000","last_undegraded":"2024-03-31T23:11:10.716676+0000","last_fullsized":"2024-03-31T23:11:10.716676+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:38:26.570175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"93'8","reported_seq":40,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:10.542801+0000","last_change":"2024-03-31T23:09:11.181434+0000","last_active":"2024-03-31T23:11:10.542801+0000","last_peered":"2024-03-31T23:11:10.542801+0000","last_clean":"2024-03-31T23:11:10.542801+0000","last_became_active":"2024-03-31T23:09:11.181140+0000","last_became_peered":"2024-03-31T23:09:11.181140+0000","last_unstale":"2024-03-31T23:11:10.542801+0000","last_undegraded":"2024-03-31T23:11:10.542801+0000","last_fullsized":"2024-03-31T23:11:10.542801+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:55:06.379696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4157440,"data_stored":4132576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":72556,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":744,"num_read_kb":451,"num_write":182,"num_write_kb":137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":438272,"data_stored":212861,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":158,"ondisk_log_size":158,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":81,"seq":347892351018,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36404,"kb_used_data":1060,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708748,"statfs":{"total":95995035648,"available":95957757952,"internally_reserved":0,"allocated":1085440,"data_stored":533688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":36174276},"hb_peers":[0,1,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":73,"seq":313532612655,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28756,"kb_used_data":1740,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716396,"statfs":{"total":95995035648,"available":95965589504,"internally_reserved":0,"allocated":1781760,"data_stored":1189370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":68,"seq":292057776180,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28096,"kb_used_data":1080,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717056,"statfs":{"total":95995035648,"available":95966265344,"internally_reserved":0,"allocated":1105920,"data_stored":538020,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":60,"seq":257698037816,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28092,"kb_used_data":1076,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717060,"statfs":{"total":95995035648,"available":95966269440,"internally_reserved":0,"allocated":1101824,"data_stored":533844,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299453,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28676,"kb_used_data":1656,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716476,"statfs":{"total":95995035648,"available":95965671424,"internally_reserved":0,"allocated":1695744,"data_stored":1130426,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1588,"internal_metadata":27654604},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863462979,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28080,"kb_used_data":1056,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717072,"statfs":{"total":95995035648,"available":95966281728,"internally_reserved":0,"allocated":1081344,"data_stored":529100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574938,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28148,"kb_used_data":1124,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717004,"statfs":{"total":95995035648,"available":95966212096,"internally_reserved":0,"allocated":1150976,"data_stored":591592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738464,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36424,"kb_used_data":1080,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708728,"statfs":{"total":95995035648,"available":95957737472,"internally_reserved":0,"allocated":1105920,"data_stored":530488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411415,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28168,"kb_used_data":1152,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716984,"statfs":{"total":95995035648,"available":95966191616,"internally_reserved":0,"allocated":1179648,"data_stored":598774,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182482,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28084,"kb_used_data":1068,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717068,"statfs":{"total":95995035648,"available":95966277632,"internally_reserved":0,"allocated":1093632,"data_stored":529294,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920845,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36408,"kb_used_data":1068,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708744,"statfs":{"total":95995035648,"available":95957753856,"internally_reserved":0,"allocated":1093632,"data_stored":534775,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724616,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":37004,"kb_used_data":1664,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708148,"statfs":{"total":95995035648,"available":95957143552,"internally_reserved":0,"allocated":1703936,"data_stored":1125482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":62582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":69764,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1669,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":110592,"data_stored":69992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-31T23:11:31.393 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-31T23:11:31.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:31.532+0000 7f12f2127640 1 -- 172.21.15.46:0/1957771412 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f12ec1022a0 msgr2=0x7f12ec108b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:31.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:31.532+0000 7f12f2127640 1 --2- 172.21.15.46:0/1957771412 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f12ec1022a0 0x7f12ec108b30 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f12e4009ca0 tx=0x7f12e402c0f0 comp rx=0 tx=0).stop 2024-03-31T23:11:31.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:31.532+0000 7f12f2127640 1 -- 172.21.15.46:0/1957771412 shutdown_connections 2024-03-31T23:11:31.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:31.532+0000 7f12f2127640 1 --2- 172.21.15.46:0/1957771412 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f12ec1022a0 0x7f12ec108b30 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:31.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:31.532+0000 7f12f2127640 1 --2- 172.21.15.46:0/1957771412 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f12ec1018e0 0x7f12ec101d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:31.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:31.532+0000 7f12f2127640 1 --2- 172.21.15.46:0/1957771412 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f12ec1006f0 0x7f12ec100b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:31.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:31.532+0000 7f12f2127640 1 -- 172.21.15.46:0/1957771412 >> 172.21.15.46:0/1957771412 conn(0x7f12ec0fbc80 msgr2=0x7f12ec0fe0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:11:31.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:31.532+0000 7f12f2127640 1 -- 172.21.15.46:0/1957771412 shutdown_connections 2024-03-31T23:11:31.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:31.532+0000 7f12f2127640 1 -- 172.21.15.46:0/1957771412 wait complete. 2024-03-31T23:11:31.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:31.533+0000 7f12f2127640 1 Processor -- start 2024-03-31T23:11:31.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:31.533+0000 7f12f2127640 1 -- start start 2024-03-31T23:11:31.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:31.533+0000 7f12f2127640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f12ec1006f0 0x7f12ec197950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:31.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:31.533+0000 7f12f2127640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f12ec1018e0 0x7f12ec197e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:31.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:31.533+0000 7f12f2127640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f12ec1022a0 0x7f12ec19ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:31.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:31.533+0000 7f12f2127640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f12ec10d2d0 con 0x7f12ec1006f0 2024-03-31T23:11:31.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:31.533+0000 7f12f2127640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f12ec10d150 con 0x7f12ec1018e0 2024-03-31T23:11:31.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:31.533+0000 7f12f2127640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f12ec10d450 con 0x7f12ec1022a0 2024-03-31T23:11:31.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:31.533+0000 7f12eb7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f12ec1006f0 0x7f12ec197950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:31.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:31.533+0000 7f12eb7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f12ec1006f0 0x7f12ec197950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:48102/0 (socket says 172.21.15.46:48102) 2024-03-31T23:11:31.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:31.533+0000 7f12eb7fe640 1 -- 172.21.15.46:0/4230565628 learned_addr learned my addr 172.21.15.46:0/4230565628 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:11:31.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:31.533+0000 7f12eaffd640 1 --2- 172.21.15.46:0/4230565628 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f12ec1018e0 0x7f12ec197e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:31.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:31.533+0000 7f12ebfff640 1 --2- 172.21.15.46:0/4230565628 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f12ec1022a0 0x7f12ec19ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:31.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:31.533+0000 7f12eb7fe640 1 -- 172.21.15.46:0/4230565628 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f12ec1022a0 msgr2=0x7f12ec19ef00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:31.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:31.533+0000 7f12eb7fe640 1 --2- 172.21.15.46:0/4230565628 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f12ec1022a0 0x7f12ec19ef00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:31.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:31.533+0000 7f12eb7fe640 1 -- 172.21.15.46:0/4230565628 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f12ec1018e0 msgr2=0x7f12ec197e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:31.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:31.533+0000 7f12eb7fe640 1 --2- 172.21.15.46:0/4230565628 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f12ec1018e0 0x7f12ec197e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:31.537 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:31.533+0000 7f12eb7fe640 1 -- 172.21.15.46:0/4230565628 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f12ec19f570 con 0x7f12ec1006f0 2024-03-31T23:11:31.537 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:31.533+0000 7f12ebfff640 1 --2- 172.21.15.46:0/4230565628 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f12ec1022a0 0x7f12ec19ef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T23:11:31.537 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:31.534+0000 7f12eb7fe640 1 --2- 172.21.15.46:0/4230565628 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f12ec1006f0 0x7f12ec197950 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f12dc00e9c0 tx=0x7f12dc00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:11:31.537 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:31.534+0000 7f12e8ff9640 1 -- 172.21.15.46:0/4230565628 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12dc00cde0 con 0x7f12ec1006f0 2024-03-31T23:11:31.537 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:31.534+0000 7f12f2127640 1 -- 172.21.15.46:0/4230565628 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f12ec19f860 con 0x7f12ec1006f0 2024-03-31T23:11:31.537 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:31.534+0000 7f12e8ff9640 1 -- 172.21.15.46:0/4230565628 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f12dc0108e0 con 0x7f12ec1006f0 2024-03-31T23:11:31.537 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:31.534+0000 7f12e8ff9640 1 -- 172.21.15.46:0/4230565628 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12dc0188e0 con 0x7f12ec1006f0 2024-03-31T23:11:31.537 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:31.534+0000 7f12f2127640 1 -- 172.21.15.46:0/4230565628 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f12ec19fe00 con 0x7f12ec1006f0 2024-03-31T23:11:31.537 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:31.535+0000 7f12f2127640 1 -- 172.21.15.46:0/4230565628 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f12ec107a90 con 0x7f12ec1006f0 2024-03-31T23:11:31.538 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:31.536+0000 7f12e8ff9640 1 -- 172.21.15.46:0/4230565628 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 18) ==== 137320+0+0 (secure 0 0 0) 0x7f12dc01f020 con 0x7f12ec1006f0 2024-03-31T23:11:31.538 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:31.537+0000 7f12e8ff9640 1 --2- 172.21.15.46:0/4230565628 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f12d40a24a0 0x7f12d40a4960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:31.538 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:31.537+0000 7f12eaffd640 1 --2- 172.21.15.46:0/4230565628 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f12d40a24a0 0x7f12d40a4960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:31.538 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:31.537+0000 7f12e8ff9640 1 -- 172.21.15.46:0/4230565628 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9208+0+0 (secure 0 0 0) 0x7f12dc0cedc0 con 0x7f12ec1006f0 2024-03-31T23:11:31.538 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:31.537+0000 7f12eaffd640 1 --2- 172.21.15.46:0/4230565628 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f12d40a24a0 0x7f12d40a4960 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f12ec198e60 tx=0x7f12e0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:11:31.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:31.541+0000 7f12e8ff9640 1 -- 172.21.15.46:0/4230565628 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f12dc09bb50 con 0x7f12ec1006f0 2024-03-31T23:11:31.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:31.720+0000 7f12f2127640 1 -- 172.21.15.46:0/4230565628 --> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f12ec061b90 con 0x7f12d40a24a0 2024-03-31T23:11:31.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:31.730+0000 7f12e8ff9640 1 -- 172.21.15.46:0/4230565628 <== mgr.14152 v2:172.21.15.46:6800/162289569 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340476 (secure 0 0 0) 0x7f12ec061b90 con 0x7f12d40a24a0 2024-03-31T23:11:31.736 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-31T23:11:31.738 INFO:teuthology.orchestra.run.smithi046.stderr:dumped all 2024-03-31T23:11:31.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:31.733+0000 7f12f2127640 1 -- 172.21.15.46:0/4230565628 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f12d40a24a0 msgr2=0x7f12d40a4960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:31.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:31.733+0000 7f12f2127640 1 --2- 172.21.15.46:0/4230565628 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f12d40a24a0 0x7f12d40a4960 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f12ec198e60 tx=0x7f12e0005f90 comp rx=0 tx=0).stop 2024-03-31T23:11:31.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:31.733+0000 7f12f2127640 1 -- 172.21.15.46:0/4230565628 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f12ec1006f0 msgr2=0x7f12ec197950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:31.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:31.733+0000 7f12f2127640 1 --2- 172.21.15.46:0/4230565628 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f12ec1006f0 0x7f12ec197950 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f12dc00e9c0 tx=0x7f12dc00ee80 comp rx=0 tx=0).stop 2024-03-31T23:11:31.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:31.733+0000 7f12f2127640 1 -- 172.21.15.46:0/4230565628 shutdown_connections 2024-03-31T23:11:31.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:31.733+0000 7f12f2127640 1 --2- 172.21.15.46:0/4230565628 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f12d40a24a0 0x7f12d40a4960 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:31.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:31.733+0000 7f12f2127640 1 --2- 172.21.15.46:0/4230565628 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f12ec1022a0 0x7f12ec19ef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:31.741 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:31.733+0000 7f12f2127640 1 --2- 172.21.15.46:0/4230565628 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f12ec1018e0 0x7f12ec197e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:31.741 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:31.733+0000 7f12f2127640 1 --2- 172.21.15.46:0/4230565628 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f12ec1006f0 0x7f12ec197950 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:31.741 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:31.733+0000 7f12f2127640 1 -- 172.21.15.46:0/4230565628 >> 172.21.15.46:0/4230565628 conn(0x7f12ec0fbc80 msgr2=0x7f12ec104c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:11:31.741 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:31.734+0000 7f12f2127640 1 -- 172.21.15.46:0/4230565628 shutdown_connections 2024-03-31T23:11:31.741 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:31.734+0000 7f12f2127640 1 -- 172.21.15.46:0/4230565628 wait complete. 2024-03-31T23:11:31.743 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:11:31 smithi046 ceph-mon[32498]: from='client.? 172.21.15.46:0/4229979867' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-31T23:11:31.743 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:11:31 smithi046 ceph-mon[32498]: from='client.? 172.21.15.46:0/3641190743' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-31T23:11:31.744 INFO:teuthology.orchestra.run.smithi046.stdout:{"pg_ready":true,"pg_map":{"version":366,"stamp":"2024-03-31T23:11:30.612781+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":663158,"num_objects":64,"num_object_clones":0,"num_object_copies":194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":880,"num_read_kb":570,"num_write":365,"num_write_kb":3077,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":797,"ondisk_log_size":797,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":372340,"kb_used_data":14824,"kb_used_omap":18,"kb_used_meta":357357,"kb_avail":1124569484,"statfs":{"total":1151940427776,"available":1151559151616,"internally_reserved":0,"allocated":15179776,"data_stored":8364853,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19140,"internal_metadata":365933884},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":5300,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":82,"num_read_kb":51,"num_write":10,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003267"},"pg_stats":[{"pgid":"3.3f","version":"93'8","reported_seq":40,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:10.543291+0000","last_change":"2024-03-31T23:09:11.177502+0000","last_active":"2024-03-31T23:11:10.543291+0000","last_peered":"2024-03-31T23:11:10.543291+0000","last_clean":"2024-03-31T23:11:10.543291+0000","last_became_active":"2024-03-31T23:09:11.177134+0000","last_became_peered":"2024-03-31T23:09:11.177134+0000","last_unstale":"2024-03-31T23:11:10.543291+0000","last_undegraded":"2024-03-31T23:11:10.543291+0000","last_fullsized":"2024-03-31T23:11:10.543291+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T10:29:36.508229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"93'4","reported_seq":36,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:11.122776+0000","last_change":"2024-03-31T23:09:11.172990+0000","last_active":"2024-03-31T23:11:11.122776+0000","last_peered":"2024-03-31T23:11:11.122776+0000","last_clean":"2024-03-31T23:11:11.122776+0000","last_became_active":"2024-03-31T23:09:11.172684+0000","last_became_peered":"2024-03-31T23:09:11.172684+0000","last_unstale":"2024-03-31T23:11:11.122776+0000","last_undegraded":"2024-03-31T23:11:11.122776+0000","last_fullsized":"2024-03-31T23:11:11.122776+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T07:02:59.991757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"93'7","reported_seq":39,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:11.122648+0000","last_change":"2024-03-31T23:09:11.176805+0000","last_active":"2024-03-31T23:11:11.122648+0000","last_peered":"2024-03-31T23:11:11.122648+0000","last_clean":"2024-03-31T23:11:11.122648+0000","last_became_active":"2024-03-31T23:09:11.176468+0000","last_became_peered":"2024-03-31T23:09:11.176468+0000","last_unstale":"2024-03-31T23:11:11.122648+0000","last_undegraded":"2024-03-31T23:11:11.122648+0000","last_fullsized":"2024-03-31T23:11:11.122648+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T08:23:41.872558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"93'8","reported_seq":40,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:10.545278+0000","last_change":"2024-03-31T23:09:11.182385+0000","last_active":"2024-03-31T23:11:10.545278+0000","last_peered":"2024-03-31T23:11:10.545278+0000","last_clean":"2024-03-31T23:11:10.545278+0000","last_became_active":"2024-03-31T23:09:11.181908+0000","last_became_peered":"2024-03-31T23:09:11.181908+0000","last_unstale":"2024-03-31T23:11:10.545278+0000","last_undegraded":"2024-03-31T23:11:10.545278+0000","last_fullsized":"2024-03-31T23:11:10.545278+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:52:24.066360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"93'4","reported_seq":35,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:10.542115+0000","last_change":"2024-03-31T23:09:11.181377+0000","last_active":"2024-03-31T23:11:10.542115+0000","last_peered":"2024-03-31T23:11:10.542115+0000","last_clean":"2024-03-31T23:11:10.542115+0000","last_became_active":"2024-03-31T23:09:11.181064+0000","last_became_peered":"2024-03-31T23:09:11.181064+0000","last_unstale":"2024-03-31T23:11:10.542115+0000","last_undegraded":"2024-03-31T23:11:10.542115+0000","last_fullsized":"2024-03-31T23:11:10.542115+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:28:40.766541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"93'6","reported_seq":37,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:11.124851+0000","last_change":"2024-03-31T23:09:11.176087+0000","last_active":"2024-03-31T23:11:11.124851+0000","last_peered":"2024-03-31T23:11:11.124851+0000","last_clean":"2024-03-31T23:11:11.124851+0000","last_became_active":"2024-03-31T23:09:11.175569+0000","last_became_peered":"2024-03-31T23:09:11.175569+0000","last_unstale":"2024-03-31T23:11:11.124851+0000","last_undegraded":"2024-03-31T23:11:11.124851+0000","last_fullsized":"2024-03-31T23:11:11.124851+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T06:33:19.189909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"93'5","reported_seq":36,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:10.546125+0000","last_change":"2024-03-31T23:09:11.170289+0000","last_active":"2024-03-31T23:11:10.546125+0000","last_peered":"2024-03-31T23:11:10.546125+0000","last_clean":"2024-03-31T23:11:10.546125+0000","last_became_active":"2024-03-31T23:09:11.169959+0000","last_became_peered":"2024-03-31T23:09:11.169959+0000","last_unstale":"2024-03-31T23:11:10.546125+0000","last_undegraded":"2024-03-31T23:11:10.546125+0000","last_fullsized":"2024-03-31T23:11:10.546125+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:10:44.614767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"93'9","reported_seq":40,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:10.540524+0000","last_change":"2024-03-31T23:09:11.182447+0000","last_active":"2024-03-31T23:11:10.540524+0000","last_peered":"2024-03-31T23:11:10.540524+0000","last_clean":"2024-03-31T23:11:10.540524+0000","last_became_active":"2024-03-31T23:09:11.181629+0000","last_became_peered":"2024-03-31T23:09:11.181629+0000","last_unstale":"2024-03-31T23:11:10.540524+0000","last_undegraded":"2024-03-31T23:11:10.540524+0000","last_fullsized":"2024-03-31T23:11:10.540524+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:18:26.269451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"93'11","reported_seq":42,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:10.934818+0000","last_change":"2024-03-31T23:09:11.176853+0000","last_active":"2024-03-31T23:11:10.934818+0000","last_peered":"2024-03-31T23:11:10.934818+0000","last_clean":"2024-03-31T23:11:10.934818+0000","last_became_active":"2024-03-31T23:09:11.176413+0000","last_became_peered":"2024-03-31T23:09:11.176413+0000","last_unstale":"2024-03-31T23:11:10.934818+0000","last_undegraded":"2024-03-31T23:11:10.934818+0000","last_fullsized":"2024-03-31T23:11:10.934818+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:23:40.486947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"93'16","reported_seq":53,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:11.125686+0000","last_change":"2024-03-31T23:10:10.543998+0000","last_active":"2024-03-31T23:11:11.125686+0000","last_peered":"2024-03-31T23:11:11.125686+0000","last_clean":"2024-03-31T23:11:11.125686+0000","last_became_active":"2024-03-31T23:10:10.543615+0000","last_became_peered":"2024-03-31T23:10:10.543615+0000","last_unstale":"2024-03-31T23:11:11.125686+0000","last_undegraded":"2024-03-31T23:11:11.125686+0000","last_fullsized":"2024-03-31T23:11:11.125686+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:28:32.883732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,9],"acting":[10,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"93'6","reported_seq":37,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:10.545678+0000","last_change":"2024-03-31T23:09:11.179536+0000","last_active":"2024-03-31T23:11:10.545678+0000","last_peered":"2024-03-31T23:11:10.545678+0000","last_clean":"2024-03-31T23:11:10.545678+0000","last_became_active":"2024-03-31T23:09:11.178521+0000","last_became_peered":"2024-03-31T23:09:11.178521+0000","last_unstale":"2024-03-31T23:11:10.545678+0000","last_undegraded":"2024-03-31T23:11:10.545678+0000","last_fullsized":"2024-03-31T23:11:10.545678+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T09:29:23.128832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"93'7","reported_seq":38,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:10.545128+0000","last_change":"2024-03-31T23:09:11.182548+0000","last_active":"2024-03-31T23:11:10.545128+0000","last_peered":"2024-03-31T23:11:10.545128+0000","last_clean":"2024-03-31T23:11:10.545128+0000","last_became_active":"2024-03-31T23:09:11.181966+0000","last_became_peered":"2024-03-31T23:09:11.181966+0000","last_unstale":"2024-03-31T23:11:10.545128+0000","last_undegraded":"2024-03-31T23:11:10.545128+0000","last_fullsized":"2024-03-31T23:11:10.545128+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:43:26.061594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"93'10","reported_seq":47,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:10.716409+0000","last_change":"2024-03-31T23:10:10.544976+0000","last_active":"2024-03-31T23:11:10.716409+0000","last_peered":"2024-03-31T23:11:10.716409+0000","last_clean":"2024-03-31T23:11:10.716409+0000","last_became_active":"2024-03-31T23:10:10.544433+0000","last_became_peered":"2024-03-31T23:10:10.544433+0000","last_unstale":"2024-03-31T23:11:10.716409+0000","last_undegraded":"2024-03-31T23:11:10.716409+0000","last_fullsized":"2024-03-31T23:11:10.716409+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:45:54.606274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,9],"acting":[6,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"93'7","reported_seq":38,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:11.122229+0000","last_change":"2024-03-31T23:09:11.168784+0000","last_active":"2024-03-31T23:11:11.122229+0000","last_peered":"2024-03-31T23:11:11.122229+0000","last_clean":"2024-03-31T23:11:11.122229+0000","last_became_active":"2024-03-31T23:09:11.168442+0000","last_became_peered":"2024-03-31T23:09:11.168442+0000","last_unstale":"2024-03-31T23:11:11.122229+0000","last_undegraded":"2024-03-31T23:11:11.122229+0000","last_fullsized":"2024-03-31T23:11:11.122229+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T09:37:43.568122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"93'7","reported_seq":38,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:11.122185+0000","last_change":"2024-03-31T23:09:11.176017+0000","last_active":"2024-03-31T23:11:11.122185+0000","last_peered":"2024-03-31T23:11:11.122185+0000","last_clean":"2024-03-31T23:11:11.122185+0000","last_became_active":"2024-03-31T23:09:11.175690+0000","last_became_peered":"2024-03-31T23:09:11.175690+0000","last_unstale":"2024-03-31T23:11:11.122185+0000","last_undegraded":"2024-03-31T23:11:11.122185+0000","last_fullsized":"2024-03-31T23:11:11.122185+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:14:13.259895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"93'9","reported_seq":40,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:10.934173+0000","last_change":"2024-03-31T23:09:11.171926+0000","last_active":"2024-03-31T23:11:10.934173+0000","last_peered":"2024-03-31T23:11:10.934173+0000","last_clean":"2024-03-31T23:11:10.934173+0000","last_became_active":"2024-03-31T23:09:11.171625+0000","last_became_peered":"2024-03-31T23:09:11.171625+0000","last_unstale":"2024-03-31T23:11:10.934173+0000","last_undegraded":"2024-03-31T23:11:10.934173+0000","last_fullsized":"2024-03-31T23:11:10.934173+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T06:20:20.433802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"93'8","reported_seq":39,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:10.543011+0000","last_change":"2024-03-31T23:09:11.183805+0000","last_active":"2024-03-31T23:11:10.543011+0000","last_peered":"2024-03-31T23:11:10.543011+0000","last_clean":"2024-03-31T23:11:10.543011+0000","last_became_active":"2024-03-31T23:09:11.183464+0000","last_became_peered":"2024-03-31T23:09:11.183464+0000","last_unstale":"2024-03-31T23:11:10.543011+0000","last_undegraded":"2024-03-31T23:11:10.543011+0000","last_fullsized":"2024-03-31T23:11:10.543011+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T06:02:33.286563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"93'10","reported_seq":41,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:11.117069+0000","last_change":"2024-03-31T23:09:11.170095+0000","last_active":"2024-03-31T23:11:11.117069+0000","last_peered":"2024-03-31T23:11:11.117069+0000","last_clean":"2024-03-31T23:11:11.117069+0000","last_became_active":"2024-03-31T23:09:11.169683+0000","last_became_peered":"2024-03-31T23:09:11.169683+0000","last_unstale":"2024-03-31T23:11:11.117069+0000","last_undegraded":"2024-03-31T23:11:11.117069+0000","last_fullsized":"2024-03-31T23:11:11.117069+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T08:19:47.101431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"93'7","reported_seq":39,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:10.544657+0000","last_change":"2024-03-31T23:09:11.181347+0000","last_active":"2024-03-31T23:11:10.544657+0000","last_peered":"2024-03-31T23:11:10.544657+0000","last_clean":"2024-03-31T23:11:10.544657+0000","last_became_active":"2024-03-31T23:09:11.180510+0000","last_became_peered":"2024-03-31T23:09:11.180510+0000","last_unstale":"2024-03-31T23:11:10.544657+0000","last_undegraded":"2024-03-31T23:11:10.544657+0000","last_fullsized":"2024-03-31T23:11:10.544657+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:25:51.083446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:10.544622+0000","last_change":"2024-03-31T23:09:15.205059+0000","last_active":"2024-03-31T23:11:10.544622+0000","last_peered":"2024-03-31T23:11:10.544622+0000","last_clean":"2024-03-31T23:11:10.544622+0000","last_became_active":"2024-03-31T23:09:15.204612+0000","last_became_peered":"2024-03-31T23:09:15.204612+0000","last_unstale":"2024-03-31T23:11:10.544622+0000","last_undegraded":"2024-03-31T23:11:10.544622+0000","last_fullsized":"2024-03-31T23:11:10.544622+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_clean_scrub_stamp":"2024-03-31T23:09:14.171251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:53:33.998026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"91'9","reported_seq":44,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:11.116248+0000","last_change":"2024-03-31T23:09:10.594147+0000","last_active":"2024-03-31T23:11:11.116248+0000","last_peered":"2024-03-31T23:11:11.116248+0000","last_clean":"2024-03-31T23:11:11.116248+0000","last_became_active":"2024-03-31T23:09:10.592047+0000","last_became_peered":"2024-03-31T23:09:10.592047+0000","last_unstale":"2024-03-31T23:11:11.116248+0000","last_undegraded":"2024-03-31T23:11:11.116248+0000","last_fullsized":"2024-03-31T23:11:11.116248+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_clean_scrub_stamp":"2024-03-31T23:09:09.564263+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:04:34.548282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"93'5","reported_seq":37,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:10.541242+0000","last_change":"2024-03-31T23:09:11.182292+0000","last_active":"2024-03-31T23:11:10.541242+0000","last_peered":"2024-03-31T23:11:10.541242+0000","last_clean":"2024-03-31T23:11:10.541242+0000","last_became_active":"2024-03-31T23:09:11.181885+0000","last_became_peered":"2024-03-31T23:09:11.181885+0000","last_unstale":"2024-03-31T23:11:10.541242+0000","last_undegraded":"2024-03-31T23:11:10.541242+0000","last_fullsized":"2024-03-31T23:11:10.541242+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T10:38:17.550625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":32,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:10.544451+0000","last_change":"2024-03-31T23:10:10.543802+0000","last_active":"2024-03-31T23:11:10.544451+0000","last_peered":"2024-03-31T23:11:10.544451+0000","last_clean":"2024-03-31T23:11:10.544451+0000","last_became_active":"2024-03-31T23:10:10.543245+0000","last_became_peered":"2024-03-31T23:10:10.543245+0000","last_unstale":"2024-03-31T23:11:10.544451+0000","last_undegraded":"2024-03-31T23:11:10.544451+0000","last_fullsized":"2024-03-31T23:11:10.544451+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_clean_scrub_stamp":"2024-03-31T23:09:14.171251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T06:00:11.707183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,8,3],"acting":[4,6,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"91'2","reported_seq":36,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:10.541490+0000","last_change":"2024-03-31T23:09:10.598375+0000","last_active":"2024-03-31T23:11:10.541490+0000","last_peered":"2024-03-31T23:11:10.541490+0000","last_clean":"2024-03-31T23:11:10.541490+0000","last_became_active":"2024-03-31T23:09:10.598045+0000","last_became_peered":"2024-03-31T23:09:10.598045+0000","last_unstale":"2024-03-31T23:11:10.541490+0000","last_undegraded":"2024-03-31T23:11:10.541490+0000","last_fullsized":"2024-03-31T23:11:10.541490+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_clean_scrub_stamp":"2024-03-31T23:09:09.564263+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:31:27.113317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"93'6","reported_seq":38,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:11.122739+0000","last_change":"2024-03-31T23:09:11.183458+0000","last_active":"2024-03-31T23:11:11.122739+0000","last_peered":"2024-03-31T23:11:11.122739+0000","last_clean":"2024-03-31T23:11:11.122739+0000","last_became_active":"2024-03-31T23:09:11.183100+0000","last_became_peered":"2024-03-31T23:09:11.183100+0000","last_unstale":"2024-03-31T23:11:11.122739+0000","last_undegraded":"2024-03-31T23:11:11.122739+0000","last_fullsized":"2024-03-31T23:11:11.122739+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T10:55:35.512065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":32,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:11.271365+0000","last_change":"2024-03-31T23:11:11.271365+0000","last_active":"2024-03-31T23:11:11.271365+0000","last_peered":"2024-03-31T23:11:11.271365+0000","last_clean":"2024-03-31T23:11:11.271365+0000","last_became_active":"2024-03-31T23:11:11.270814+0000","last_became_peered":"2024-03-31T23:11:11.270814+0000","last_unstale":"2024-03-31T23:11:11.271365+0000","last_undegraded":"2024-03-31T23:11:11.271365+0000","last_fullsized":"2024-03-31T23:11:11.271365+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_clean_scrub_stamp":"2024-03-31T23:09:14.171251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:58:27.916348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,10,5],"acting":[4,3,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"91'4","reported_seq":37,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:10.541863+0000","last_change":"2024-03-31T23:09:10.597471+0000","last_active":"2024-03-31T23:11:10.541863+0000","last_peered":"2024-03-31T23:11:10.541863+0000","last_clean":"2024-03-31T23:11:10.541863+0000","last_became_active":"2024-03-31T23:09:10.597165+0000","last_became_peered":"2024-03-31T23:09:10.597165+0000","last_unstale":"2024-03-31T23:11:10.541863+0000","last_undegraded":"2024-03-31T23:11:10.541863+0000","last_fullsized":"2024-03-31T23:11:10.541863+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_clean_scrub_stamp":"2024-03-31T23:09:09.564263+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:10:26.826525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"93'9","reported_seq":40,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:10.934521+0000","last_change":"2024-03-31T23:09:11.170650+0000","last_active":"2024-03-31T23:11:10.934521+0000","last_peered":"2024-03-31T23:11:10.934521+0000","last_clean":"2024-03-31T23:11:10.934521+0000","last_became_active":"2024-03-31T23:09:11.170291+0000","last_became_peered":"2024-03-31T23:09:11.170291+0000","last_unstale":"2024-03-31T23:11:10.934521+0000","last_undegraded":"2024-03-31T23:11:10.934521+0000","last_fullsized":"2024-03-31T23:11:10.934521+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:29:54.904377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:10.544466+0000","last_change":"2024-03-31T23:09:15.204857+0000","last_active":"2024-03-31T23:11:10.544466+0000","last_peered":"2024-03-31T23:11:10.544466+0000","last_clean":"2024-03-31T23:11:10.544466+0000","last_became_active":"2024-03-31T23:09:15.204484+0000","last_became_peered":"2024-03-31T23:09:15.204484+0000","last_unstale":"2024-03-31T23:11:10.544466+0000","last_undegraded":"2024-03-31T23:11:10.544466+0000","last_fullsized":"2024-03-31T23:11:10.544466+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_clean_scrub_stamp":"2024-03-31T23:09:14.171251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T09:54:24.646918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"85'2","reported_seq":35,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:10.541624+0000","last_change":"2024-03-31T23:09:10.589922+0000","last_active":"2024-03-31T23:11:10.541624+0000","last_peered":"2024-03-31T23:11:10.541624+0000","last_clean":"2024-03-31T23:11:10.541624+0000","last_became_active":"2024-03-31T23:09:10.589569+0000","last_became_peered":"2024-03-31T23:09:10.589569+0000","last_unstale":"2024-03-31T23:11:10.541624+0000","last_undegraded":"2024-03-31T23:11:10.541624+0000","last_fullsized":"2024-03-31T23:11:10.541624+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_clean_scrub_stamp":"2024-03-31T23:09:09.564263+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:57:05.347344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"93'15","reported_seq":52,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:11.270163+0000","last_change":"2024-03-31T23:10:10.485015+0000","last_active":"2024-03-31T23:11:11.270163+0000","last_peered":"2024-03-31T23:11:11.270163+0000","last_clean":"2024-03-31T23:11:11.270163+0000","last_became_active":"2024-03-31T23:10:10.484663+0000","last_became_peered":"2024-03-31T23:10:10.484663+0000","last_unstale":"2024-03-31T23:11:11.270163+0000","last_undegraded":"2024-03-31T23:11:11.270163+0000","last_fullsized":"2024-03-31T23:11:11.270163+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T06:18:50.834711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,1],"acting":[5,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:10.544476+0000","last_change":"2024-03-31T23:09:15.193096+0000","last_active":"2024-03-31T23:11:10.544476+0000","last_peered":"2024-03-31T23:11:10.544476+0000","last_clean":"2024-03-31T23:11:10.544476+0000","last_became_active":"2024-03-31T23:09:15.192624+0000","last_became_peered":"2024-03-31T23:09:15.192624+0000","last_unstale":"2024-03-31T23:11:10.544476+0000","last_undegraded":"2024-03-31T23:11:10.544476+0000","last_fullsized":"2024-03-31T23:11:10.544476+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_clean_scrub_stamp":"2024-03-31T23:09:14.171251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:22:24.921346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"93'7","reported_seq":161,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:25.511651+0000","last_change":"2024-03-31T23:09:10.587209+0000","last_active":"2024-03-31T23:11:25.511651+0000","last_peered":"2024-03-31T23:11:25.511651+0000","last_clean":"2024-03-31T23:11:25.511651+0000","last_became_active":"2024-03-31T23:09:10.586486+0000","last_became_peered":"2024-03-31T23:09:10.586486+0000","last_unstale":"2024-03-31T23:11:25.511651+0000","last_undegraded":"2024-03-31T23:11:25.511651+0000","last_fullsized":"2024-03-31T23:11:25.511651+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_clean_scrub_stamp":"2024-03-31T23:09:09.564263+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:27:56.046414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1154,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":121,"num_read_kb":4,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"93'7","reported_seq":44,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:10.541409+0000","last_change":"2024-03-31T23:10:10.543713+0000","last_active":"2024-03-31T23:11:10.541409+0000","last_peered":"2024-03-31T23:11:10.541409+0000","last_clean":"2024-03-31T23:11:10.541409+0000","last_became_active":"2024-03-31T23:10:10.543324+0000","last_became_peered":"2024-03-31T23:10:10.543324+0000","last_unstale":"2024-03-31T23:11:10.541409+0000","last_undegraded":"2024-03-31T23:11:10.541409+0000","last_fullsized":"2024-03-31T23:11:10.541409+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:41:21.974325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,6],"acting":[8,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:10.544389+0000","last_change":"2024-03-31T23:09:15.194717+0000","last_active":"2024-03-31T23:11:10.544389+0000","last_peered":"2024-03-31T23:11:10.544389+0000","last_clean":"2024-03-31T23:11:10.544389+0000","last_became_active":"2024-03-31T23:09:15.194232+0000","last_became_peered":"2024-03-31T23:09:15.194232+0000","last_unstale":"2024-03-31T23:11:10.544389+0000","last_undegraded":"2024-03-31T23:11:10.544389+0000","last_fullsized":"2024-03-31T23:11:10.544389+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_clean_scrub_stamp":"2024-03-31T23:09:14.171251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:21:00.445232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"91'3","reported_seq":36,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:11.124484+0000","last_change":"2024-03-31T23:09:10.586346+0000","last_active":"2024-03-31T23:11:11.124484+0000","last_peered":"2024-03-31T23:11:11.124484+0000","last_clean":"2024-03-31T23:11:11.124484+0000","last_became_active":"2024-03-31T23:09:10.586043+0000","last_became_peered":"2024-03-31T23:09:10.586043+0000","last_unstale":"2024-03-31T23:11:11.124484+0000","last_undegraded":"2024-03-31T23:11:11.124484+0000","last_fullsized":"2024-03-31T23:11:11.124484+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_clean_scrub_stamp":"2024-03-31T23:09:09.564263+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:17:27.387786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"93'8","reported_seq":45,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:11.270118+0000","last_change":"2024-03-31T23:10:10.543353+0000","last_active":"2024-03-31T23:11:11.270118+0000","last_peered":"2024-03-31T23:11:11.270118+0000","last_clean":"2024-03-31T23:11:11.270118+0000","last_became_active":"2024-03-31T23:10:10.542961+0000","last_became_peered":"2024-03-31T23:10:10.542961+0000","last_unstale":"2024-03-31T23:11:11.270118+0000","last_undegraded":"2024-03-31T23:11:11.270118+0000","last_fullsized":"2024-03-31T23:11:11.270118+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:28:52.950551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3],"acting":[5,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":26,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:10.542953+0000","last_change":"2024-03-31T23:09:15.205855+0000","last_active":"2024-03-31T23:11:10.542953+0000","last_peered":"2024-03-31T23:11:10.542953+0000","last_clean":"2024-03-31T23:11:10.542953+0000","last_became_active":"2024-03-31T23:09:15.205359+0000","last_became_peered":"2024-03-31T23:09:15.205359+0000","last_unstale":"2024-03-31T23:11:10.542953+0000","last_undegraded":"2024-03-31T23:11:10.542953+0000","last_fullsized":"2024-03-31T23:11:10.542953+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_clean_scrub_stamp":"2024-03-31T23:09:14.171251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T09:39:32.461011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"91'2","reported_seq":35,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:10.715951+0000","last_change":"2024-03-31T23:09:10.583274+0000","last_active":"2024-03-31T23:11:10.715951+0000","last_peered":"2024-03-31T23:11:10.715951+0000","last_clean":"2024-03-31T23:11:10.715951+0000","last_became_active":"2024-03-31T23:09:10.582910+0000","last_became_peered":"2024-03-31T23:09:10.582910+0000","last_unstale":"2024-03-31T23:11:10.715951+0000","last_undegraded":"2024-03-31T23:11:10.715951+0000","last_fullsized":"2024-03-31T23:11:10.715951+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_clean_scrub_stamp":"2024-03-31T23:09:09.564263+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:16:32.251169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"93'8","reported_seq":39,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:10.540701+0000","last_change":"2024-03-31T23:09:11.178619+0000","last_active":"2024-03-31T23:11:10.540701+0000","last_peered":"2024-03-31T23:11:10.540701+0000","last_clean":"2024-03-31T23:11:10.540701+0000","last_became_active":"2024-03-31T23:09:11.178239+0000","last_became_peered":"2024-03-31T23:09:11.178239+0000","last_unstale":"2024-03-31T23:11:10.540701+0000","last_undegraded":"2024-03-31T23:11:10.540701+0000","last_fullsized":"2024-03-31T23:11:10.540701+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:28:58.863199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":26,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:11.122693+0000","last_change":"2024-03-31T23:09:15.205864+0000","last_active":"2024-03-31T23:11:11.122693+0000","last_peered":"2024-03-31T23:11:11.122693+0000","last_clean":"2024-03-31T23:11:11.122693+0000","last_became_active":"2024-03-31T23:09:15.204976+0000","last_became_peered":"2024-03-31T23:09:15.204976+0000","last_unstale":"2024-03-31T23:11:11.122693+0000","last_undegraded":"2024-03-31T23:11:11.122693+0000","last_fullsized":"2024-03-31T23:11:11.122693+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_clean_scrub_stamp":"2024-03-31T23:09:14.171251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:39:08.537827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"91'1","reported_seq":34,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:11.269487+0000","last_change":"2024-03-31T23:09:10.588407+0000","last_active":"2024-03-31T23:11:11.269487+0000","last_peered":"2024-03-31T23:11:11.269487+0000","last_clean":"2024-03-31T23:11:11.269487+0000","last_became_active":"2024-03-31T23:09:10.588109+0000","last_became_peered":"2024-03-31T23:09:10.588109+0000","last_unstale":"2024-03-31T23:11:11.269487+0000","last_undegraded":"2024-03-31T23:11:11.269487+0000","last_fullsized":"2024-03-31T23:11:11.269487+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_clean_scrub_stamp":"2024-03-31T23:09:09.564263+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:01:25.858834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"93'11","reported_seq":42,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:11.116446+0000","last_change":"2024-03-31T23:09:11.180452+0000","last_active":"2024-03-31T23:11:11.116446+0000","last_peered":"2024-03-31T23:11:11.116446+0000","last_clean":"2024-03-31T23:11:11.116446+0000","last_became_active":"2024-03-31T23:09:11.180117+0000","last_became_peered":"2024-03-31T23:09:11.180117+0000","last_unstale":"2024-03-31T23:11:11.116446+0000","last_undegraded":"2024-03-31T23:11:11.116446+0000","last_fullsized":"2024-03-31T23:11:11.116446+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:04:58.288981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:10.540663+0000","last_change":"2024-03-31T23:09:15.199669+0000","last_active":"2024-03-31T23:11:10.540663+0000","last_peered":"2024-03-31T23:11:10.540663+0000","last_clean":"2024-03-31T23:11:10.540663+0000","last_became_active":"2024-03-31T23:09:15.199262+0000","last_became_peered":"2024-03-31T23:09:15.199262+0000","last_unstale":"2024-03-31T23:11:10.540663+0000","last_undegraded":"2024-03-31T23:11:10.540663+0000","last_fullsized":"2024-03-31T23:11:10.540663+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_clean_scrub_stamp":"2024-03-31T23:09:14.171251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:44:02.975000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"95'5","reported_seq":158,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:28.514876+0000","last_change":"2024-03-31T23:09:10.583476+0000","last_active":"2024-03-31T23:11:28.514876+0000","last_peered":"2024-03-31T23:11:28.514876+0000","last_clean":"2024-03-31T23:11:28.514876+0000","last_became_active":"2024-03-31T23:09:10.582789+0000","last_became_peered":"2024-03-31T23:09:10.582789+0000","last_unstale":"2024-03-31T23:11:28.514876+0000","last_undegraded":"2024-03-31T23:11:28.514876+0000","last_fullsized":"2024-03-31T23:11:28.514876+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_clean_scrub_stamp":"2024-03-31T23:09:09.564263+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T10:52:35.478325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":120,"num_read_kb":120,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"93'10","reported_seq":41,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:10.934738+0000","last_change":"2024-03-31T23:09:11.171074+0000","last_active":"2024-03-31T23:11:10.934738+0000","last_peered":"2024-03-31T23:11:10.934738+0000","last_clean":"2024-03-31T23:11:10.934738+0000","last_became_active":"2024-03-31T23:09:11.169572+0000","last_became_peered":"2024-03-31T23:09:11.169572+0000","last_unstale":"2024-03-31T23:11:10.934738+0000","last_undegraded":"2024-03-31T23:11:10.934738+0000","last_fullsized":"2024-03-31T23:11:10.934738+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T07:15:34.833388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:11.116316+0000","last_change":"2024-03-31T23:09:15.203835+0000","last_active":"2024-03-31T23:11:11.116316+0000","last_peered":"2024-03-31T23:11:11.116316+0000","last_clean":"2024-03-31T23:11:11.116316+0000","last_became_active":"2024-03-31T23:09:15.200861+0000","last_became_peered":"2024-03-31T23:09:15.200861+0000","last_unstale":"2024-03-31T23:11:11.116316+0000","last_undegraded":"2024-03-31T23:11:11.116316+0000","last_fullsized":"2024-03-31T23:11:11.116316+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_clean_scrub_stamp":"2024-03-31T23:09:14.171251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:56:19.664708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"91'2","reported_seq":35,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:10.545911+0000","last_change":"2024-03-31T23:09:10.592601+0000","last_active":"2024-03-31T23:11:10.545911+0000","last_peered":"2024-03-31T23:11:10.545911+0000","last_clean":"2024-03-31T23:11:10.545911+0000","last_became_active":"2024-03-31T23:09:10.592345+0000","last_became_peered":"2024-03-31T23:09:10.592345+0000","last_unstale":"2024-03-31T23:11:10.545911+0000","last_undegraded":"2024-03-31T23:11:10.545911+0000","last_fullsized":"2024-03-31T23:11:10.545911+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_clean_scrub_stamp":"2024-03-31T23:09:09.564263+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:10:21.195544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"93'9","reported_seq":40,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:11.122701+0000","last_change":"2024-03-31T23:09:11.175305+0000","last_active":"2024-03-31T23:11:11.122701+0000","last_peered":"2024-03-31T23:11:11.122701+0000","last_clean":"2024-03-31T23:11:11.122701+0000","last_became_active":"2024-03-31T23:09:11.174970+0000","last_became_peered":"2024-03-31T23:09:11.174970+0000","last_unstale":"2024-03-31T23:11:11.122701+0000","last_undegraded":"2024-03-31T23:11:11.122701+0000","last_fullsized":"2024-03-31T23:11:11.122701+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:29:39.718127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:10.544982+0000","last_change":"2024-03-31T23:09:15.200365+0000","last_active":"2024-03-31T23:11:10.544982+0000","last_peered":"2024-03-31T23:11:10.544982+0000","last_clean":"2024-03-31T23:11:10.544982+0000","last_became_active":"2024-03-31T23:09:15.199830+0000","last_became_peered":"2024-03-31T23:09:15.199830+0000","last_unstale":"2024-03-31T23:11:10.544982+0000","last_undegraded":"2024-03-31T23:11:10.544982+0000","last_fullsized":"2024-03-31T23:11:10.544982+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_clean_scrub_stamp":"2024-03-31T23:09:14.171251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:37:30.048859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"91'4","reported_seq":38,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:11.270163+0000","last_change":"2024-03-31T23:09:10.582654+0000","last_active":"2024-03-31T23:11:11.270163+0000","last_peered":"2024-03-31T23:11:11.270163+0000","last_clean":"2024-03-31T23:11:11.270163+0000","last_became_active":"2024-03-31T23:09:10.582314+0000","last_became_peered":"2024-03-31T23:09:10.582314+0000","last_unstale":"2024-03-31T23:11:11.270163+0000","last_undegraded":"2024-03-31T23:11:11.270163+0000","last_fullsized":"2024-03-31T23:11:11.270163+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_clean_scrub_stamp":"2024-03-31T23:09:09.564263+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:51:34.236575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"93'11","reported_seq":42,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:10.934540+0000","last_change":"2024-03-31T23:09:11.172811+0000","last_active":"2024-03-31T23:11:10.934540+0000","last_peered":"2024-03-31T23:11:10.934540+0000","last_clean":"2024-03-31T23:11:10.934540+0000","last_became_active":"2024-03-31T23:09:11.172478+0000","last_became_peered":"2024-03-31T23:09:11.172478+0000","last_unstale":"2024-03-31T23:11:10.934540+0000","last_undegraded":"2024-03-31T23:11:10.934540+0000","last_fullsized":"2024-03-31T23:11:10.934540+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T10:00:09.825254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:11.116557+0000","last_change":"2024-03-31T23:09:15.205494+0000","last_active":"2024-03-31T23:11:11.116557+0000","last_peered":"2024-03-31T23:11:11.116557+0000","last_clean":"2024-03-31T23:11:11.116557+0000","last_became_active":"2024-03-31T23:09:15.205101+0000","last_became_peered":"2024-03-31T23:09:15.205101+0000","last_unstale":"2024-03-31T23:11:11.116557+0000","last_undegraded":"2024-03-31T23:11:11.116557+0000","last_fullsized":"2024-03-31T23:11:11.116557+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_clean_scrub_stamp":"2024-03-31T23:09:14.171251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:52:47.472296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"95'40","reported_seq":324,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:29.516503+0000","last_change":"2024-03-31T23:09:10.585611+0000","last_active":"2024-03-31T23:11:29.516503+0000","last_peered":"2024-03-31T23:11:29.516503+0000","last_clean":"2024-03-31T23:11:29.516503+0000","last_became_active":"2024-03-31T23:09:10.585287+0000","last_became_peered":"2024-03-31T23:09:10.585287+0000","last_unstale":"2024-03-31T23:11:29.516503+0000","last_undegraded":"2024-03-31T23:11:29.516503+0000","last_fullsized":"2024-03-31T23:11:29.516503+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_clean_scrub_stamp":"2024-03-31T23:09:09.564263+0000","objects_scrubbed":0,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:47:44.063831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":67389,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":250,"num_read_kb":191,"num_write":41,"num_write_kb":88,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"93'6","reported_seq":37,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:10.542061+0000","last_change":"2024-03-31T23:09:11.175812+0000","last_active":"2024-03-31T23:11:10.542061+0000","last_peered":"2024-03-31T23:11:10.542061+0000","last_clean":"2024-03-31T23:11:10.542061+0000","last_became_active":"2024-03-31T23:09:11.174972+0000","last_became_peered":"2024-03-31T23:09:11.174972+0000","last_unstale":"2024-03-31T23:11:10.542061+0000","last_undegraded":"2024-03-31T23:11:10.542061+0000","last_fullsized":"2024-03-31T23:11:10.542061+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:17:18.870340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:11.270348+0000","last_change":"2024-03-31T23:09:15.205018+0000","last_active":"2024-03-31T23:11:11.270348+0000","last_peered":"2024-03-31T23:11:11.270348+0000","last_clean":"2024-03-31T23:11:11.270348+0000","last_became_active":"2024-03-31T23:09:15.204510+0000","last_became_peered":"2024-03-31T23:09:15.204510+0000","last_unstale":"2024-03-31T23:11:11.270348+0000","last_undegraded":"2024-03-31T23:11:11.270348+0000","last_fullsized":"2024-03-31T23:11:11.270348+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_clean_scrub_stamp":"2024-03-31T23:09:14.171251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T07:48:59.819937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"91'6","reported_seq":40,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:10.544535+0000","last_change":"2024-03-31T23:09:10.590233+0000","last_active":"2024-03-31T23:11:10.544535+0000","last_peered":"2024-03-31T23:11:10.544535+0000","last_clean":"2024-03-31T23:11:10.544535+0000","last_became_active":"2024-03-31T23:09:10.589853+0000","last_became_peered":"2024-03-31T23:09:10.589853+0000","last_unstale":"2024-03-31T23:11:10.544535+0000","last_undegraded":"2024-03-31T23:11:10.544535+0000","last_fullsized":"2024-03-31T23:11:10.544535+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_clean_scrub_stamp":"2024-03-31T23:09:09.564263+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:10:54.980394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"93'9","reported_seq":40,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:10.716289+0000","last_change":"2024-03-31T23:09:11.177871+0000","last_active":"2024-03-31T23:11:10.716289+0000","last_peered":"2024-03-31T23:11:10.716289+0000","last_clean":"2024-03-31T23:11:10.716289+0000","last_became_active":"2024-03-31T23:09:11.176504+0000","last_became_peered":"2024-03-31T23:09:11.176504+0000","last_unstale":"2024-03-31T23:11:10.716289+0000","last_undegraded":"2024-03-31T23:11:10.716289+0000","last_fullsized":"2024-03-31T23:11:10.716289+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T06:09:08.797695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:10.542551+0000","last_change":"2024-03-31T23:09:15.207336+0000","last_active":"2024-03-31T23:11:10.542551+0000","last_peered":"2024-03-31T23:11:10.542551+0000","last_clean":"2024-03-31T23:11:10.542551+0000","last_became_active":"2024-03-31T23:09:15.206893+0000","last_became_peered":"2024-03-31T23:09:15.206893+0000","last_unstale":"2024-03-31T23:11:10.542551+0000","last_undegraded":"2024-03-31T23:11:10.542551+0000","last_fullsized":"2024-03-31T23:11:10.542551+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_clean_scrub_stamp":"2024-03-31T23:09:14.171251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:24:59.766421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"91'3","reported_seq":36,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:11.268980+0000","last_change":"2024-03-31T23:09:10.587099+0000","last_active":"2024-03-31T23:11:11.268980+0000","last_peered":"2024-03-31T23:11:11.268980+0000","last_clean":"2024-03-31T23:11:11.268980+0000","last_became_active":"2024-03-31T23:09:10.586429+0000","last_became_peered":"2024-03-31T23:09:10.586429+0000","last_unstale":"2024-03-31T23:11:11.268980+0000","last_undegraded":"2024-03-31T23:11:11.268980+0000","last_fullsized":"2024-03-31T23:11:11.268980+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_clean_scrub_stamp":"2024-03-31T23:09:09.564263+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:55:09.307378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"93'15","reported_seq":46,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:10.546149+0000","last_change":"2024-03-31T23:09:11.177661+0000","last_active":"2024-03-31T23:11:10.546149+0000","last_peered":"2024-03-31T23:11:10.546149+0000","last_clean":"2024-03-31T23:11:10.546149+0000","last_became_active":"2024-03-31T23:09:11.177314+0000","last_became_peered":"2024-03-31T23:09:11.177314+0000","last_unstale":"2024-03-31T23:11:10.546149+0000","last_undegraded":"2024-03-31T23:11:10.546149+0000","last_fullsized":"2024-03-31T23:11:10.546149+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T09:26:50.556837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":16,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:10.541733+0000","last_change":"2024-03-31T23:10:10.543526+0000","last_active":"2024-03-31T23:11:10.541733+0000","last_peered":"2024-03-31T23:11:10.541733+0000","last_clean":"2024-03-31T23:11:10.541733+0000","last_became_active":"2024-03-31T23:10:10.543105+0000","last_became_peered":"2024-03-31T23:10:10.543105+0000","last_unstale":"2024-03-31T23:11:10.541733+0000","last_undegraded":"2024-03-31T23:11:10.541733+0000","last_fullsized":"2024-03-31T23:11:10.541733+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_clean_scrub_stamp":"2024-03-31T23:09:14.171251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T09:16:44.529786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,7,0,3],"acting":[8,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.11","version":"95'6","reported_seq":161,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:27.546382+0000","last_change":"2024-03-31T23:09:10.586570+0000","last_active":"2024-03-31T23:11:27.546382+0000","last_peered":"2024-03-31T23:11:27.546382+0000","last_clean":"2024-03-31T23:11:27.546382+0000","last_became_active":"2024-03-31T23:09:10.586239+0000","last_became_peered":"2024-03-31T23:09:10.586239+0000","last_unstale":"2024-03-31T23:11:27.546382+0000","last_undegraded":"2024-03-31T23:11:27.546382+0000","last_fullsized":"2024-03-31T23:11:27.546382+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_clean_scrub_stamp":"2024-03-31T23:09:09.564263+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:23:37.182954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1613,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":123,"num_read_kb":6,"num_write":5,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"93'6","reported_seq":37,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:10.545691+0000","last_change":"2024-03-31T23:09:11.173294+0000","last_active":"2024-03-31T23:11:10.545691+0000","last_peered":"2024-03-31T23:11:10.545691+0000","last_clean":"2024-03-31T23:11:10.545691+0000","last_became_active":"2024-03-31T23:09:11.172926+0000","last_became_peered":"2024-03-31T23:09:11.172926+0000","last_unstale":"2024-03-31T23:11:10.545691+0000","last_undegraded":"2024-03-31T23:11:10.545691+0000","last_fullsized":"2024-03-31T23:11:10.545691+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T06:21:46.603381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:10.716781+0000","last_change":"2024-03-31T23:09:15.199893+0000","last_active":"2024-03-31T23:11:10.716781+0000","last_peered":"2024-03-31T23:11:10.716781+0000","last_clean":"2024-03-31T23:11:10.716781+0000","last_became_active":"2024-03-31T23:09:15.199517+0000","last_became_peered":"2024-03-31T23:09:15.199517+0000","last_unstale":"2024-03-31T23:11:10.716781+0000","last_undegraded":"2024-03-31T23:11:10.716781+0000","last_fullsized":"2024-03-31T23:11:10.716781+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_clean_scrub_stamp":"2024-03-31T23:09:14.171251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T07:48:35.261108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"91'15","reported_seq":49,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:10.541154+0000","last_change":"2024-03-31T23:09:10.588213+0000","last_active":"2024-03-31T23:11:10.541154+0000","last_peered":"2024-03-31T23:11:10.541154+0000","last_clean":"2024-03-31T23:11:10.541154+0000","last_became_active":"2024-03-31T23:09:10.587579+0000","last_became_peered":"2024-03-31T23:09:10.587579+0000","last_unstale":"2024-03-31T23:11:10.541154+0000","last_undegraded":"2024-03-31T23:11:10.541154+0000","last_fullsized":"2024-03-31T23:11:10.541154+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_clean_scrub_stamp":"2024-03-31T23:09:09.564263+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:00:17.883435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"93'6","reported_seq":37,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:11.268980+0000","last_change":"2024-03-31T23:09:11.182427+0000","last_active":"2024-03-31T23:11:11.268980+0000","last_peered":"2024-03-31T23:11:11.268980+0000","last_clean":"2024-03-31T23:11:11.268980+0000","last_became_active":"2024-03-31T23:09:11.182120+0000","last_became_peered":"2024-03-31T23:09:11.182120+0000","last_unstale":"2024-03-31T23:11:11.268980+0000","last_undegraded":"2024-03-31T23:11:11.268980+0000","last_fullsized":"2024-03-31T23:11:11.268980+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:17:48.728424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":26,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:11.269948+0000","last_change":"2024-03-31T23:09:15.207831+0000","last_active":"2024-03-31T23:11:11.269948+0000","last_peered":"2024-03-31T23:11:11.269948+0000","last_clean":"2024-03-31T23:11:11.269948+0000","last_became_active":"2024-03-31T23:09:15.207432+0000","last_became_peered":"2024-03-31T23:09:15.207432+0000","last_unstale":"2024-03-31T23:11:11.269948+0000","last_undegraded":"2024-03-31T23:11:11.269948+0000","last_fullsized":"2024-03-31T23:11:11.269948+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_clean_scrub_stamp":"2024-03-31T23:09:14.171251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:05:27.072054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"91'2","reported_seq":37,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:10.541954+0000","last_change":"2024-03-31T23:09:10.591483+0000","last_active":"2024-03-31T23:11:10.541954+0000","last_peered":"2024-03-31T23:11:10.541954+0000","last_clean":"2024-03-31T23:11:10.541954+0000","last_became_active":"2024-03-31T23:09:10.591173+0000","last_became_peered":"2024-03-31T23:09:10.591173+0000","last_unstale":"2024-03-31T23:11:10.541954+0000","last_undegraded":"2024-03-31T23:11:10.541954+0000","last_fullsized":"2024-03-31T23:11:10.541954+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_clean_scrub_stamp":"2024-03-31T23:09:09.564263+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T10:04:14.275875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"93'5","reported_seq":37,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:10.543194+0000","last_change":"2024-03-31T23:09:11.181171+0000","last_active":"2024-03-31T23:11:10.543194+0000","last_peered":"2024-03-31T23:11:10.543194+0000","last_clean":"2024-03-31T23:11:10.543194+0000","last_became_active":"2024-03-31T23:09:11.180847+0000","last_became_peered":"2024-03-31T23:09:11.180847+0000","last_unstale":"2024-03-31T23:11:10.543194+0000","last_undegraded":"2024-03-31T23:11:10.543194+0000","last_fullsized":"2024-03-31T23:11:10.543194+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:39:29.429009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":26,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:10.541346+0000","last_change":"2024-03-31T23:09:15.746941+0000","last_active":"2024-03-31T23:11:10.541346+0000","last_peered":"2024-03-31T23:11:10.541346+0000","last_clean":"2024-03-31T23:11:10.541346+0000","last_became_active":"2024-03-31T23:09:15.746352+0000","last_became_peered":"2024-03-31T23:09:15.746352+0000","last_unstale":"2024-03-31T23:11:10.541346+0000","last_undegraded":"2024-03-31T23:11:10.541346+0000","last_fullsized":"2024-03-31T23:11:10.541346+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_clean_scrub_stamp":"2024-03-31T23:09:14.171251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T06:21:43.482575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":34,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:11.116214+0000","last_change":"2024-03-31T23:09:10.593029+0000","last_active":"2024-03-31T23:11:11.116214+0000","last_peered":"2024-03-31T23:11:11.116214+0000","last_clean":"2024-03-31T23:11:11.116214+0000","last_became_active":"2024-03-31T23:09:10.592709+0000","last_became_peered":"2024-03-31T23:09:10.592709+0000","last_unstale":"2024-03-31T23:11:11.116214+0000","last_undegraded":"2024-03-31T23:11:11.116214+0000","last_fullsized":"2024-03-31T23:11:11.116214+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_clean_scrub_stamp":"2024-03-31T23:09:09.564263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:46:45.700207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"93'4","reported_seq":36,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:11.125675+0000","last_change":"2024-03-31T23:09:11.177857+0000","last_active":"2024-03-31T23:11:11.125675+0000","last_peered":"2024-03-31T23:11:11.125675+0000","last_clean":"2024-03-31T23:11:11.125675+0000","last_became_active":"2024-03-31T23:09:11.177513+0000","last_became_peered":"2024-03-31T23:09:11.177513+0000","last_unstale":"2024-03-31T23:11:11.125675+0000","last_undegraded":"2024-03-31T23:11:11.125675+0000","last_fullsized":"2024-03-31T23:11:11.125675+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T06:09:23.579208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":26,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:10.716480+0000","last_change":"2024-03-31T23:09:15.203412+0000","last_active":"2024-03-31T23:11:10.716480+0000","last_peered":"2024-03-31T23:11:10.716480+0000","last_clean":"2024-03-31T23:11:10.716480+0000","last_became_active":"2024-03-31T23:09:15.202983+0000","last_became_peered":"2024-03-31T23:09:15.202983+0000","last_unstale":"2024-03-31T23:11:10.716480+0000","last_undegraded":"2024-03-31T23:11:10.716480+0000","last_fullsized":"2024-03-31T23:11:10.716480+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_clean_scrub_stamp":"2024-03-31T23:09:14.171251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:57:46.357378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"91'1","reported_seq":35,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:10.545484+0000","last_change":"2024-03-31T23:09:10.593299+0000","last_active":"2024-03-31T23:11:10.545484+0000","last_peered":"2024-03-31T23:11:10.545484+0000","last_clean":"2024-03-31T23:11:10.545484+0000","last_became_active":"2024-03-31T23:09:10.593007+0000","last_became_peered":"2024-03-31T23:09:10.593007+0000","last_unstale":"2024-03-31T23:11:10.545484+0000","last_undegraded":"2024-03-31T23:11:10.545484+0000","last_fullsized":"2024-03-31T23:11:10.545484+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_clean_scrub_stamp":"2024-03-31T23:09:09.564263+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:15:07.199645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"93'7","reported_seq":39,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:11.125387+0000","last_change":"2024-03-31T23:09:11.176846+0000","last_active":"2024-03-31T23:11:11.125387+0000","last_peered":"2024-03-31T23:11:11.125387+0000","last_clean":"2024-03-31T23:11:11.125387+0000","last_became_active":"2024-03-31T23:09:11.176554+0000","last_became_peered":"2024-03-31T23:09:11.176554+0000","last_unstale":"2024-03-31T23:11:11.125387+0000","last_undegraded":"2024-03-31T23:11:11.125387+0000","last_fullsized":"2024-03-31T23:11:11.125387+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:25:23.639093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":26,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:10.716905+0000","last_change":"2024-03-31T23:09:15.746599+0000","last_active":"2024-03-31T23:11:10.716905+0000","last_peered":"2024-03-31T23:11:10.716905+0000","last_clean":"2024-03-31T23:11:10.716905+0000","last_became_active":"2024-03-31T23:09:15.745879+0000","last_became_peered":"2024-03-31T23:09:15.745879+0000","last_unstale":"2024-03-31T23:11:10.716905+0000","last_undegraded":"2024-03-31T23:11:10.716905+0000","last_fullsized":"2024-03-31T23:11:10.716905+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_clean_scrub_stamp":"2024-03-31T23:09:14.171251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T08:02:25.198610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"93'8","reported_seq":40,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:11.269210+0000","last_change":"2024-03-31T23:09:11.175546+0000","last_active":"2024-03-31T23:11:11.269210+0000","last_peered":"2024-03-31T23:11:11.269210+0000","last_clean":"2024-03-31T23:11:11.269210+0000","last_became_active":"2024-03-31T23:09:11.175253+0000","last_became_peered":"2024-03-31T23:09:11.175253+0000","last_unstale":"2024-03-31T23:11:11.269210+0000","last_undegraded":"2024-03-31T23:11:11.269210+0000","last_fullsized":"2024-03-31T23:11:11.269210+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T10:27:31.753102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"95'6","reported_seq":159,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:28.547474+0000","last_change":"2024-03-31T23:09:10.580888+0000","last_active":"2024-03-31T23:11:28.547474+0000","last_peered":"2024-03-31T23:11:28.547474+0000","last_clean":"2024-03-31T23:11:28.547474+0000","last_became_active":"2024-03-31T23:09:10.580557+0000","last_became_peered":"2024-03-31T23:09:10.580557+0000","last_unstale":"2024-03-31T23:11:28.547474+0000","last_undegraded":"2024-03-31T23:11:28.547474+0000","last_fullsized":"2024-03-31T23:11:28.547474+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_clean_scrub_stamp":"2024-03-31T23:09:09.564263+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:17:38.878570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":119,"num_read_kb":119,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"91'1","reported_seq":27,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:10.545528+0000","last_change":"2024-03-31T23:09:15.205282+0000","last_active":"2024-03-31T23:11:10.545528+0000","last_peered":"2024-03-31T23:11:10.545528+0000","last_clean":"2024-03-31T23:11:10.545528+0000","last_became_active":"2024-03-31T23:09:15.204873+0000","last_became_peered":"2024-03-31T23:09:15.204873+0000","last_unstale":"2024-03-31T23:11:10.545528+0000","last_undegraded":"2024-03-31T23:11:10.545528+0000","last_fullsized":"2024-03-31T23:11:10.545528+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_clean_scrub_stamp":"2024-03-31T23:09:14.171251+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:56:05.697685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"93'8","reported_seq":40,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:10.540856+0000","last_change":"2024-03-31T23:09:11.182155+0000","last_active":"2024-03-31T23:11:10.540856+0000","last_peered":"2024-03-31T23:11:10.540856+0000","last_clean":"2024-03-31T23:11:10.540856+0000","last_became_active":"2024-03-31T23:09:11.181829+0000","last_became_peered":"2024-03-31T23:09:11.181829+0000","last_unstale":"2024-03-31T23:11:10.540856+0000","last_undegraded":"2024-03-31T23:11:10.540856+0000","last_fullsized":"2024-03-31T23:11:10.540856+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:19:22.832440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"91'3","reported_seq":39,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:10.544924+0000","last_change":"2024-03-31T23:09:10.590451+0000","last_active":"2024-03-31T23:11:10.544924+0000","last_peered":"2024-03-31T23:11:10.544924+0000","last_clean":"2024-03-31T23:11:10.544924+0000","last_became_active":"2024-03-31T23:09:10.590123+0000","last_became_peered":"2024-03-31T23:09:10.590123+0000","last_unstale":"2024-03-31T23:11:10.544924+0000","last_undegraded":"2024-03-31T23:11:10.544924+0000","last_fullsized":"2024-03-31T23:11:10.544924+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_clean_scrub_stamp":"2024-03-31T23:09:09.564263+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:27:02.724456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":369,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:10.545255+0000","last_change":"2024-03-31T23:07:57.920942+0000","last_active":"2024-03-31T23:11:10.545255+0000","last_peered":"2024-03-31T23:11:10.545255+0000","last_clean":"2024-03-31T23:11:10.545255+0000","last_became_active":"2024-03-31T23:07:57.920381+0000","last_became_peered":"2024-03-31T23:07:57.920381+0000","last_unstale":"2024-03-31T23:11:10.545255+0000","last_undegraded":"2024-03-31T23:11:10.545255+0000","last_fullsized":"2024-03-31T23:11:10.545255+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:04:33.508180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:04:33.508180+0000","last_clean_scrub_stamp":"2024-03-31T23:04:33.508180+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:20:11.211403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:11.126466+0000","last_change":"2024-03-31T23:11:11.126137+0000","last_active":"2024-03-31T23:11:11.126466+0000","last_peered":"2024-03-31T23:11:11.126466+0000","last_clean":"2024-03-31T23:11:11.126466+0000","last_became_active":"2024-03-31T23:11:11.125719+0000","last_became_peered":"2024-03-31T23:11:11.125719+0000","last_unstale":"2024-03-31T23:11:11.126466+0000","last_undegraded":"2024-03-31T23:11:11.126466+0000","last_fullsized":"2024-03-31T23:11:11.126466+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_clean_scrub_stamp":"2024-03-31T23:09:14.171251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:34:55.444516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,6,10],"acting":[8,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1","version":"93'5","reported_seq":37,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:10.716368+0000","last_change":"2024-03-31T23:09:11.175336+0000","last_active":"2024-03-31T23:11:10.716368+0000","last_peered":"2024-03-31T23:11:10.716368+0000","last_clean":"2024-03-31T23:11:10.716368+0000","last_became_active":"2024-03-31T23:09:11.174768+0000","last_became_peered":"2024-03-31T23:09:11.174768+0000","last_unstale":"2024-03-31T23:11:10.716368+0000","last_undegraded":"2024-03-31T23:11:10.716368+0000","last_fullsized":"2024-03-31T23:11:10.716368+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:09:11.678744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"91'3","reported_seq":37,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:10.545473+0000","last_change":"2024-03-31T23:09:10.591807+0000","last_active":"2024-03-31T23:11:10.545473+0000","last_peered":"2024-03-31T23:11:10.545473+0000","last_clean":"2024-03-31T23:11:10.545473+0000","last_became_active":"2024-03-31T23:09:10.591407+0000","last_became_peered":"2024-03-31T23:09:10.591407+0000","last_unstale":"2024-03-31T23:11:10.545473+0000","last_undegraded":"2024-03-31T23:11:10.545473+0000","last_fullsized":"2024-03-31T23:11:10.545473+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_clean_scrub_stamp":"2024-03-31T23:09:09.564263+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T08:46:33.660717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":26,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:10.543100+0000","last_change":"2024-03-31T23:09:15.203879+0000","last_active":"2024-03-31T23:11:10.543100+0000","last_peered":"2024-03-31T23:11:10.543100+0000","last_clean":"2024-03-31T23:11:10.543100+0000","last_became_active":"2024-03-31T23:09:15.203412+0000","last_became_peered":"2024-03-31T23:09:15.203412+0000","last_unstale":"2024-03-31T23:11:10.543100+0000","last_undegraded":"2024-03-31T23:11:10.543100+0000","last_fullsized":"2024-03-31T23:11:10.543100+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_clean_scrub_stamp":"2024-03-31T23:09:14.171251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:03:54.486038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"91'1","reported_seq":36,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:10.542010+0000","last_change":"2024-03-31T23:09:10.596404+0000","last_active":"2024-03-31T23:11:10.542010+0000","last_peered":"2024-03-31T23:11:10.542010+0000","last_clean":"2024-03-31T23:11:10.542010+0000","last_became_active":"2024-03-31T23:09:10.596031+0000","last_became_peered":"2024-03-31T23:09:10.596031+0000","last_unstale":"2024-03-31T23:11:10.542010+0000","last_undegraded":"2024-03-31T23:11:10.542010+0000","last_fullsized":"2024-03-31T23:11:10.542010+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_clean_scrub_stamp":"2024-03-31T23:09:09.564263+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:30:31.904241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"93'9","reported_seq":41,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:10.716744+0000","last_change":"2024-03-31T23:09:11.164300+0000","last_active":"2024-03-31T23:11:10.716744+0000","last_peered":"2024-03-31T23:11:10.716744+0000","last_clean":"2024-03-31T23:11:10.716744+0000","last_became_active":"2024-03-31T23:09:11.163960+0000","last_became_peered":"2024-03-31T23:09:11.163960+0000","last_unstale":"2024-03-31T23:11:10.716744+0000","last_undegraded":"2024-03-31T23:11:10.716744+0000","last_fullsized":"2024-03-31T23:11:10.716744+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T10:37:30.512216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":26,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:10.540599+0000","last_change":"2024-03-31T23:09:15.746426+0000","last_active":"2024-03-31T23:11:10.540599+0000","last_peered":"2024-03-31T23:11:10.540599+0000","last_clean":"2024-03-31T23:11:10.540599+0000","last_became_active":"2024-03-31T23:09:15.745887+0000","last_became_peered":"2024-03-31T23:09:15.745887+0000","last_unstale":"2024-03-31T23:11:10.540599+0000","last_undegraded":"2024-03-31T23:11:10.540599+0000","last_fullsized":"2024-03-31T23:11:10.540599+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_clean_scrub_stamp":"2024-03-31T23:09:14.171251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:41:52.005300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"95'10","reported_seq":44,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:23.633720+0000","last_change":"2024-03-31T23:09:10.590631+0000","last_active":"2024-03-31T23:11:23.633720+0000","last_peered":"2024-03-31T23:11:23.633720+0000","last_clean":"2024-03-31T23:11:23.633720+0000","last_became_active":"2024-03-31T23:09:10.590211+0000","last_became_peered":"2024-03-31T23:09:10.590211+0000","last_unstale":"2024-03-31T23:11:23.633720+0000","last_undegraded":"2024-03-31T23:11:23.633720+0000","last_fullsized":"2024-03-31T23:11:23.633720+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_clean_scrub_stamp":"2024-03-31T23:09:09.564263+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T08:51:24.978892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"93'5","reported_seq":37,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:10.541392+0000","last_change":"2024-03-31T23:09:11.177353+0000","last_active":"2024-03-31T23:11:10.541392+0000","last_peered":"2024-03-31T23:11:10.541392+0000","last_clean":"2024-03-31T23:11:10.541392+0000","last_became_active":"2024-03-31T23:09:11.176996+0000","last_became_peered":"2024-03-31T23:09:11.176996+0000","last_unstale":"2024-03-31T23:11:10.541392+0000","last_undegraded":"2024-03-31T23:11:10.541392+0000","last_fullsized":"2024-03-31T23:11:10.541392+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:20:52.486281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":26,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:10.544240+0000","last_change":"2024-03-31T23:09:15.198528+0000","last_active":"2024-03-31T23:11:10.544240+0000","last_peered":"2024-03-31T23:11:10.544240+0000","last_clean":"2024-03-31T23:11:10.544240+0000","last_became_active":"2024-03-31T23:09:15.197938+0000","last_became_peered":"2024-03-31T23:09:15.197938+0000","last_unstale":"2024-03-31T23:11:10.544240+0000","last_undegraded":"2024-03-31T23:11:10.544240+0000","last_fullsized":"2024-03-31T23:11:10.544240+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_clean_scrub_stamp":"2024-03-31T23:09:14.171251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:45:46.567268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"91'4","reported_seq":38,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:10.545572+0000","last_change":"2024-03-31T23:09:10.584779+0000","last_active":"2024-03-31T23:11:10.545572+0000","last_peered":"2024-03-31T23:11:10.545572+0000","last_clean":"2024-03-31T23:11:10.545572+0000","last_became_active":"2024-03-31T23:09:10.584400+0000","last_became_peered":"2024-03-31T23:09:10.584400+0000","last_unstale":"2024-03-31T23:11:10.545572+0000","last_undegraded":"2024-03-31T23:11:10.545572+0000","last_fullsized":"2024-03-31T23:11:10.545572+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_clean_scrub_stamp":"2024-03-31T23:09:09.564263+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:10:10.095704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"93'7","reported_seq":45,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:11.122710+0000","last_change":"2024-03-31T23:10:10.543538+0000","last_active":"2024-03-31T23:11:11.122710+0000","last_peered":"2024-03-31T23:11:11.122710+0000","last_clean":"2024-03-31T23:11:11.122710+0000","last_became_active":"2024-03-31T23:10:10.543083+0000","last_became_peered":"2024-03-31T23:10:10.543083+0000","last_unstale":"2024-03-31T23:11:11.122710+0000","last_undegraded":"2024-03-31T23:11:11.122710+0000","last_fullsized":"2024-03-31T23:11:11.122710+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:38:50.656801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:10.545224+0000","last_change":"2024-03-31T23:09:15.746525+0000","last_active":"2024-03-31T23:11:10.545224+0000","last_peered":"2024-03-31T23:11:10.545224+0000","last_clean":"2024-03-31T23:11:10.545224+0000","last_became_active":"2024-03-31T23:09:15.746126+0000","last_became_peered":"2024-03-31T23:09:15.746126+0000","last_unstale":"2024-03-31T23:11:10.545224+0000","last_undegraded":"2024-03-31T23:11:10.545224+0000","last_fullsized":"2024-03-31T23:11:10.545224+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_clean_scrub_stamp":"2024-03-31T23:09:14.171251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:04:57.313615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"91'2","reported_seq":36,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:10.934097+0000","last_change":"2024-03-31T23:09:10.591469+0000","last_active":"2024-03-31T23:11:10.934097+0000","last_peered":"2024-03-31T23:11:10.934097+0000","last_clean":"2024-03-31T23:11:10.934097+0000","last_became_active":"2024-03-31T23:09:10.591166+0000","last_became_peered":"2024-03-31T23:09:10.591166+0000","last_unstale":"2024-03-31T23:11:10.934097+0000","last_undegraded":"2024-03-31T23:11:10.934097+0000","last_fullsized":"2024-03-31T23:11:10.934097+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_clean_scrub_stamp":"2024-03-31T23:09:09.564263+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T10:30:40.279041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"93'2","reported_seq":34,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:10.541228+0000","last_change":"2024-03-31T23:09:11.174471+0000","last_active":"2024-03-31T23:11:10.541228+0000","last_peered":"2024-03-31T23:11:10.541228+0000","last_clean":"2024-03-31T23:11:10.541228+0000","last_became_active":"2024-03-31T23:09:11.174154+0000","last_became_peered":"2024-03-31T23:09:11.174154+0000","last_unstale":"2024-03-31T23:11:10.541228+0000","last_undegraded":"2024-03-31T23:11:10.541228+0000","last_fullsized":"2024-03-31T23:11:10.541228+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T06:10:14.727009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:11.117002+0000","last_change":"2024-03-31T23:09:15.201416+0000","last_active":"2024-03-31T23:11:11.117002+0000","last_peered":"2024-03-31T23:11:11.117002+0000","last_clean":"2024-03-31T23:11:11.117002+0000","last_became_active":"2024-03-31T23:09:15.200987+0000","last_became_peered":"2024-03-31T23:09:15.200987+0000","last_unstale":"2024-03-31T23:11:11.117002+0000","last_undegraded":"2024-03-31T23:11:11.117002+0000","last_fullsized":"2024-03-31T23:11:11.117002+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_clean_scrub_stamp":"2024-03-31T23:09:14.171251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T10:56:54.872988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"91'3","reported_seq":37,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:10.540834+0000","last_change":"2024-03-31T23:09:10.594035+0000","last_active":"2024-03-31T23:11:10.540834+0000","last_peered":"2024-03-31T23:11:10.540834+0000","last_clean":"2024-03-31T23:11:10.540834+0000","last_became_active":"2024-03-31T23:09:10.593710+0000","last_became_peered":"2024-03-31T23:09:10.593710+0000","last_unstale":"2024-03-31T23:11:10.540834+0000","last_undegraded":"2024-03-31T23:11:10.540834+0000","last_fullsized":"2024-03-31T23:11:10.540834+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_clean_scrub_stamp":"2024-03-31T23:09:09.564263+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:46:48.580244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"93'9","reported_seq":41,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:10.542532+0000","last_change":"2024-03-31T23:09:11.170501+0000","last_active":"2024-03-31T23:11:10.542532+0000","last_peered":"2024-03-31T23:11:10.542532+0000","last_clean":"2024-03-31T23:11:10.542532+0000","last_became_active":"2024-03-31T23:09:11.170068+0000","last_became_peered":"2024-03-31T23:09:11.170068+0000","last_unstale":"2024-03-31T23:11:10.542532+0000","last_undegraded":"2024-03-31T23:11:10.542532+0000","last_fullsized":"2024-03-31T23:11:10.542532+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T06:05:53.488957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"91'1","reported_seq":26,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:10.544928+0000","last_change":"2024-03-31T23:09:15.208092+0000","last_active":"2024-03-31T23:11:10.544928+0000","last_peered":"2024-03-31T23:11:10.544928+0000","last_clean":"2024-03-31T23:11:10.544928+0000","last_became_active":"2024-03-31T23:09:15.207750+0000","last_became_peered":"2024-03-31T23:09:15.207750+0000","last_unstale":"2024-03-31T23:11:10.544928+0000","last_undegraded":"2024-03-31T23:11:10.544928+0000","last_fullsized":"2024-03-31T23:11:10.544928+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_clean_scrub_stamp":"2024-03-31T23:09:14.171251+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:24:39.649569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"95'2","reported_seq":16,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:10.548189+0000","last_change":"2024-03-31T23:11:10.547781+0000","last_active":"2024-03-31T23:11:10.548189+0000","last_peered":"2024-03-31T23:11:10.548189+0000","last_clean":"2024-03-31T23:11:10.548189+0000","last_became_active":"2024-03-31T23:11:10.545156+0000","last_became_peered":"2024-03-31T23:11:10.545156+0000","last_unstale":"2024-03-31T23:11:10.548189+0000","last_undegraded":"2024-03-31T23:11:10.548189+0000","last_fullsized":"2024-03-31T23:11:10.548189+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_clean_scrub_stamp":"2024-03-31T23:09:09.564263+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T08:14:11.270910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,8],"acting":[11,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.b","version":"93'8","reported_seq":40,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:10.541041+0000","last_change":"2024-03-31T23:09:11.175169+0000","last_active":"2024-03-31T23:11:10.541041+0000","last_peered":"2024-03-31T23:11:10.541041+0000","last_clean":"2024-03-31T23:11:10.541041+0000","last_became_active":"2024-03-31T23:09:11.174849+0000","last_became_peered":"2024-03-31T23:09:11.174849+0000","last_unstale":"2024-03-31T23:11:10.541041+0000","last_undegraded":"2024-03-31T23:11:10.541041+0000","last_fullsized":"2024-03-31T23:11:10.541041+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:58:06.814034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:10.545200+0000","last_change":"2024-03-31T23:09:15.746423+0000","last_active":"2024-03-31T23:11:10.545200+0000","last_peered":"2024-03-31T23:11:10.545200+0000","last_clean":"2024-03-31T23:11:10.545200+0000","last_became_active":"2024-03-31T23:09:15.745942+0000","last_became_peered":"2024-03-31T23:09:15.745942+0000","last_unstale":"2024-03-31T23:11:10.545200+0000","last_undegraded":"2024-03-31T23:11:10.545200+0000","last_fullsized":"2024-03-31T23:11:10.545200+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_clean_scrub_stamp":"2024-03-31T23:09:14.171251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:56:02.903212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"91'4","reported_seq":17,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:10.550839+0000","last_change":"2024-03-31T23:11:10.550839+0000","last_active":"2024-03-31T23:11:10.550839+0000","last_peered":"2024-03-31T23:11:10.550839+0000","last_clean":"2024-03-31T23:11:10.550839+0000","last_became_active":"2024-03-31T23:11:10.546702+0000","last_became_peered":"2024-03-31T23:11:10.546702+0000","last_unstale":"2024-03-31T23:11:10.550839+0000","last_undegraded":"2024-03-31T23:11:10.550839+0000","last_fullsized":"2024-03-31T23:11:10.550839+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_clean_scrub_stamp":"2024-03-31T23:09:09.564263+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:39:13.293806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,7],"acting":[11,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.a","version":"93'5","reported_seq":37,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:10.545517+0000","last_change":"2024-03-31T23:09:11.170440+0000","last_active":"2024-03-31T23:11:10.545517+0000","last_peered":"2024-03-31T23:11:10.545517+0000","last_clean":"2024-03-31T23:11:10.545517+0000","last_became_active":"2024-03-31T23:09:11.170081+0000","last_became_peered":"2024-03-31T23:09:11.170081+0000","last_unstale":"2024-03-31T23:11:10.545517+0000","last_undegraded":"2024-03-31T23:11:10.545517+0000","last_fullsized":"2024-03-31T23:11:10.545517+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:53:31.429822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":26,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:10.934387+0000","last_change":"2024-03-31T23:09:15.207146+0000","last_active":"2024-03-31T23:11:10.934387+0000","last_peered":"2024-03-31T23:11:10.934387+0000","last_clean":"2024-03-31T23:11:10.934387+0000","last_became_active":"2024-03-31T23:09:15.206769+0000","last_became_peered":"2024-03-31T23:09:15.206769+0000","last_unstale":"2024-03-31T23:11:10.934387+0000","last_undegraded":"2024-03-31T23:11:10.934387+0000","last_fullsized":"2024-03-31T23:11:10.934387+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_clean_scrub_stamp":"2024-03-31T23:09:14.171251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T07:42:12.961109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"91'3","reported_seq":38,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:10.544591+0000","last_change":"2024-03-31T23:09:10.587291+0000","last_active":"2024-03-31T23:11:10.544591+0000","last_peered":"2024-03-31T23:11:10.544591+0000","last_clean":"2024-03-31T23:11:10.544591+0000","last_became_active":"2024-03-31T23:09:10.585992+0000","last_became_peered":"2024-03-31T23:09:10.585992+0000","last_unstale":"2024-03-31T23:11:10.544591+0000","last_undegraded":"2024-03-31T23:11:10.544591+0000","last_fullsized":"2024-03-31T23:11:10.544591+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_clean_scrub_stamp":"2024-03-31T23:09:09.564263+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:28:00.786268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"93'7","reported_seq":39,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:10.545804+0000","last_change":"2024-03-31T23:09:11.179369+0000","last_active":"2024-03-31T23:11:10.545804+0000","last_peered":"2024-03-31T23:11:10.545804+0000","last_clean":"2024-03-31T23:11:10.545804+0000","last_became_active":"2024-03-31T23:09:11.178776+0000","last_became_peered":"2024-03-31T23:09:11.178776+0000","last_unstale":"2024-03-31T23:11:10.545804+0000","last_undegraded":"2024-03-31T23:11:10.545804+0000","last_fullsized":"2024-03-31T23:11:10.545804+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T07:56:05.201338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:11.122108+0000","last_change":"2024-03-31T23:09:15.201462+0000","last_active":"2024-03-31T23:11:11.122108+0000","last_peered":"2024-03-31T23:11:11.122108+0000","last_clean":"2024-03-31T23:11:11.122108+0000","last_became_active":"2024-03-31T23:09:15.201067+0000","last_became_peered":"2024-03-31T23:09:15.201067+0000","last_unstale":"2024-03-31T23:11:11.122108+0000","last_undegraded":"2024-03-31T23:11:11.122108+0000","last_fullsized":"2024-03-31T23:11:11.122108+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_clean_scrub_stamp":"2024-03-31T23:09:14.171251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:54:29.339456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"85'1","reported_seq":36,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:11.269181+0000","last_change":"2024-03-31T23:09:10.584752+0000","last_active":"2024-03-31T23:11:11.269181+0000","last_peered":"2024-03-31T23:11:11.269181+0000","last_clean":"2024-03-31T23:11:11.269181+0000","last_became_active":"2024-03-31T23:09:10.584432+0000","last_became_peered":"2024-03-31T23:09:10.584432+0000","last_unstale":"2024-03-31T23:11:11.269181+0000","last_undegraded":"2024-03-31T23:11:11.269181+0000","last_fullsized":"2024-03-31T23:11:11.269181+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_clean_scrub_stamp":"2024-03-31T23:09:09.564263+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T10:50:53.889719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"93'6","reported_seq":44,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:10.546403+0000","last_change":"2024-03-31T23:10:10.543993+0000","last_active":"2024-03-31T23:11:10.546403+0000","last_peered":"2024-03-31T23:11:10.546403+0000","last_clean":"2024-03-31T23:11:10.546403+0000","last_became_active":"2024-03-31T23:10:10.543627+0000","last_became_peered":"2024-03-31T23:10:10.543627+0000","last_unstale":"2024-03-31T23:11:10.546403+0000","last_undegraded":"2024-03-31T23:11:10.546403+0000","last_fullsized":"2024-03-31T23:11:10.546403+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T08:46:41.221122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":26,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:11.269712+0000","last_change":"2024-03-31T23:09:15.197271+0000","last_active":"2024-03-31T23:11:11.269712+0000","last_peered":"2024-03-31T23:11:11.269712+0000","last_clean":"2024-03-31T23:11:11.269712+0000","last_became_active":"2024-03-31T23:09:15.196339+0000","last_became_peered":"2024-03-31T23:09:15.196339+0000","last_unstale":"2024-03-31T23:11:11.269712+0000","last_undegraded":"2024-03-31T23:11:11.269712+0000","last_fullsized":"2024-03-31T23:11:11.269712+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_clean_scrub_stamp":"2024-03-31T23:09:14.171251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:44:13.044278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"85'2","reported_seq":36,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:10.716161+0000","last_change":"2024-03-31T23:09:10.595131+0000","last_active":"2024-03-31T23:11:10.716161+0000","last_peered":"2024-03-31T23:11:10.716161+0000","last_clean":"2024-03-31T23:11:10.716161+0000","last_became_active":"2024-03-31T23:09:10.594740+0000","last_became_peered":"2024-03-31T23:09:10.594740+0000","last_unstale":"2024-03-31T23:11:10.716161+0000","last_undegraded":"2024-03-31T23:11:10.716161+0000","last_fullsized":"2024-03-31T23:11:10.716161+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_clean_scrub_stamp":"2024-03-31T23:09:09.564263+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:52:41.849621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"93'9","reported_seq":41,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:10.545854+0000","last_change":"2024-03-31T23:09:11.177530+0000","last_active":"2024-03-31T23:11:10.545854+0000","last_peered":"2024-03-31T23:11:10.545854+0000","last_clean":"2024-03-31T23:11:10.545854+0000","last_became_active":"2024-03-31T23:09:11.176972+0000","last_became_peered":"2024-03-31T23:09:11.176972+0000","last_unstale":"2024-03-31T23:11:10.545854+0000","last_undegraded":"2024-03-31T23:11:10.545854+0000","last_fullsized":"2024-03-31T23:11:10.545854+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T07:57:21.094911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"93'9","reported_seq":41,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:10.542629+0000","last_change":"2024-03-31T23:09:11.172792+0000","last_active":"2024-03-31T23:11:10.542629+0000","last_peered":"2024-03-31T23:11:10.542629+0000","last_clean":"2024-03-31T23:11:10.542629+0000","last_became_active":"2024-03-31T23:09:11.172463+0000","last_became_peered":"2024-03-31T23:09:11.172463+0000","last_unstale":"2024-03-31T23:11:10.542629+0000","last_undegraded":"2024-03-31T23:11:10.542629+0000","last_fullsized":"2024-03-31T23:11:10.542629+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:07:28.789783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"93'12","reported_seq":44,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:10.544865+0000","last_change":"2024-03-31T23:09:11.174331+0000","last_active":"2024-03-31T23:11:10.544865+0000","last_peered":"2024-03-31T23:11:10.544865+0000","last_clean":"2024-03-31T23:11:10.544865+0000","last_became_active":"2024-03-31T23:09:11.174001+0000","last_became_peered":"2024-03-31T23:09:11.174001+0000","last_unstale":"2024-03-31T23:11:10.544865+0000","last_undegraded":"2024-03-31T23:11:10.544865+0000","last_fullsized":"2024-03-31T23:11:10.544865+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:26:28.005406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"93'8","reported_seq":40,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:10.544540+0000","last_change":"2024-03-31T23:09:11.180047+0000","last_active":"2024-03-31T23:11:10.544540+0000","last_peered":"2024-03-31T23:11:10.544540+0000","last_clean":"2024-03-31T23:11:10.544540+0000","last_became_active":"2024-03-31T23:09:11.179691+0000","last_became_peered":"2024-03-31T23:09:11.179691+0000","last_unstale":"2024-03-31T23:11:10.544540+0000","last_undegraded":"2024-03-31T23:11:10.544540+0000","last_fullsized":"2024-03-31T23:11:10.544540+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:28:05.083710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"93'7","reported_seq":39,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:11.122493+0000","last_change":"2024-03-31T23:09:11.177642+0000","last_active":"2024-03-31T23:11:11.122493+0000","last_peered":"2024-03-31T23:11:11.122493+0000","last_clean":"2024-03-31T23:11:11.122493+0000","last_became_active":"2024-03-31T23:09:11.177261+0000","last_became_peered":"2024-03-31T23:09:11.177261+0000","last_unstale":"2024-03-31T23:11:11.122493+0000","last_undegraded":"2024-03-31T23:11:11.122493+0000","last_fullsized":"2024-03-31T23:11:11.122493+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T07:10:43.616863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"93'4","reported_seq":36,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:10.716630+0000","last_change":"2024-03-31T23:09:11.173606+0000","last_active":"2024-03-31T23:11:10.716630+0000","last_peered":"2024-03-31T23:11:10.716630+0000","last_clean":"2024-03-31T23:11:10.716630+0000","last_became_active":"2024-03-31T23:09:11.173304+0000","last_became_peered":"2024-03-31T23:09:11.173304+0000","last_unstale":"2024-03-31T23:11:10.716630+0000","last_undegraded":"2024-03-31T23:11:10.716630+0000","last_fullsized":"2024-03-31T23:11:10.716630+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:53:06.112181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"93'6","reported_seq":38,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:10.543432+0000","last_change":"2024-03-31T23:09:11.176628+0000","last_active":"2024-03-31T23:11:10.543432+0000","last_peered":"2024-03-31T23:11:10.543432+0000","last_clean":"2024-03-31T23:11:10.543432+0000","last_became_active":"2024-03-31T23:09:11.176318+0000","last_became_peered":"2024-03-31T23:09:11.176318+0000","last_unstale":"2024-03-31T23:11:10.543432+0000","last_undegraded":"2024-03-31T23:11:10.543432+0000","last_fullsized":"2024-03-31T23:11:10.543432+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:06:40.723675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"93'15","reported_seq":47,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:10.544863+0000","last_change":"2024-03-31T23:09:11.169318+0000","last_active":"2024-03-31T23:11:10.544863+0000","last_peered":"2024-03-31T23:11:10.544863+0000","last_clean":"2024-03-31T23:11:10.544863+0000","last_became_active":"2024-03-31T23:09:11.169026+0000","last_became_peered":"2024-03-31T23:09:11.169026+0000","last_unstale":"2024-03-31T23:11:10.544863+0000","last_undegraded":"2024-03-31T23:11:10.544863+0000","last_fullsized":"2024-03-31T23:11:10.544863+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:02:18.562854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"93'2","reported_seq":34,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:10.541448+0000","last_change":"2024-03-31T23:09:11.184670+0000","last_active":"2024-03-31T23:11:10.541448+0000","last_peered":"2024-03-31T23:11:10.541448+0000","last_clean":"2024-03-31T23:11:10.541448+0000","last_became_active":"2024-03-31T23:09:11.184354+0000","last_became_peered":"2024-03-31T23:09:11.184354+0000","last_unstale":"2024-03-31T23:11:10.541448+0000","last_undegraded":"2024-03-31T23:11:10.541448+0000","last_fullsized":"2024-03-31T23:11:10.541448+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:16:14.354947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"93'6","reported_seq":38,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:11.269351+0000","last_change":"2024-03-31T23:09:11.180991+0000","last_active":"2024-03-31T23:11:11.269351+0000","last_peered":"2024-03-31T23:11:11.269351+0000","last_clean":"2024-03-31T23:11:11.269351+0000","last_became_active":"2024-03-31T23:09:11.180667+0000","last_became_peered":"2024-03-31T23:09:11.180667+0000","last_unstale":"2024-03-31T23:11:11.269351+0000","last_undegraded":"2024-03-31T23:11:11.269351+0000","last_fullsized":"2024-03-31T23:11:11.269351+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T06:35:33.157176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"93'8","reported_seq":40,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:10.540877+0000","last_change":"2024-03-31T23:09:11.182025+0000","last_active":"2024-03-31T23:11:10.540877+0000","last_peered":"2024-03-31T23:11:10.540877+0000","last_clean":"2024-03-31T23:11:10.540877+0000","last_became_active":"2024-03-31T23:09:11.179310+0000","last_became_peered":"2024-03-31T23:09:11.179310+0000","last_unstale":"2024-03-31T23:11:10.540877+0000","last_undegraded":"2024-03-31T23:11:10.540877+0000","last_fullsized":"2024-03-31T23:11:10.540877+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T10:21:40.784738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"93'10","reported_seq":48,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:10.934477+0000","last_change":"2024-03-31T23:10:10.485623+0000","last_active":"2024-03-31T23:11:10.934477+0000","last_peered":"2024-03-31T23:11:10.934477+0000","last_clean":"2024-03-31T23:11:10.934477+0000","last_became_active":"2024-03-31T23:10:10.485248+0000","last_became_peered":"2024-03-31T23:10:10.485248+0000","last_unstale":"2024-03-31T23:11:10.934477+0000","last_undegraded":"2024-03-31T23:11:10.934477+0000","last_fullsized":"2024-03-31T23:11:10.934477+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:59:04.668598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,6],"acting":[1,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"93'11","reported_seq":43,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:10.716676+0000","last_change":"2024-03-31T23:09:11.172585+0000","last_active":"2024-03-31T23:11:10.716676+0000","last_peered":"2024-03-31T23:11:10.716676+0000","last_clean":"2024-03-31T23:11:10.716676+0000","last_became_active":"2024-03-31T23:09:11.172266+0000","last_became_peered":"2024-03-31T23:09:11.172266+0000","last_unstale":"2024-03-31T23:11:10.716676+0000","last_undegraded":"2024-03-31T23:11:10.716676+0000","last_fullsized":"2024-03-31T23:11:10.716676+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:38:26.570175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"93'8","reported_seq":40,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:10.542801+0000","last_change":"2024-03-31T23:09:11.181434+0000","last_active":"2024-03-31T23:11:10.542801+0000","last_peered":"2024-03-31T23:11:10.542801+0000","last_clean":"2024-03-31T23:11:10.542801+0000","last_became_active":"2024-03-31T23:09:11.181140+0000","last_became_peered":"2024-03-31T23:09:11.181140+0000","last_unstale":"2024-03-31T23:11:10.542801+0000","last_undegraded":"2024-03-31T23:11:10.542801+0000","last_fullsized":"2024-03-31T23:11:10.542801+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:55:06.379696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4157440,"data_stored":4132576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":72556,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":744,"num_read_kb":451,"num_write":182,"num_write_kb":137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":438272,"data_stored":212861,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":158,"ondisk_log_size":158,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":81,"seq":347892351018,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36404,"kb_used_data":1060,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708748,"statfs":{"total":95995035648,"available":95957757952,"internally_reserved":0,"allocated":1085440,"data_stored":533688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":36174276},"hb_peers":[0,1,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":73,"seq":313532612655,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28756,"kb_used_data":1740,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716396,"statfs":{"total":95995035648,"available":95965589504,"internally_reserved":0,"allocated":1781760,"data_stored":1189370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":68,"seq":292057776180,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28096,"kb_used_data":1080,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717056,"statfs":{"total":95995035648,"available":95966265344,"internally_reserved":0,"allocated":1105920,"data_stored":538020,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":60,"seq":257698037816,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28092,"kb_used_data":1076,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717060,"statfs":{"total":95995035648,"available":95966269440,"internally_reserved":0,"allocated":1101824,"data_stored":533844,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299453,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28676,"kb_used_data":1656,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716476,"statfs":{"total":95995035648,"available":95965671424,"internally_reserved":0,"allocated":1695744,"data_stored":1130426,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1588,"internal_metadata":27654604},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863462979,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28080,"kb_used_data":1056,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717072,"statfs":{"total":95995035648,"available":95966281728,"internally_reserved":0,"allocated":1081344,"data_stored":529100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574938,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28148,"kb_used_data":1124,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717004,"statfs":{"total":95995035648,"available":95966212096,"internally_reserved":0,"allocated":1150976,"data_stored":591592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738464,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36424,"kb_used_data":1080,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708728,"statfs":{"total":95995035648,"available":95957737472,"internally_reserved":0,"allocated":1105920,"data_stored":530488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411415,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28168,"kb_used_data":1152,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716984,"statfs":{"total":95995035648,"available":95966191616,"internally_reserved":0,"allocated":1179648,"data_stored":598774,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182482,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28084,"kb_used_data":1068,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717068,"statfs":{"total":95995035648,"available":95966277632,"internally_reserved":0,"allocated":1093632,"data_stored":529294,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920845,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36408,"kb_used_data":1068,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708744,"statfs":{"total":95995035648,"available":95957753856,"internally_reserved":0,"allocated":1093632,"data_stored":534775,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724616,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":37004,"kb_used_data":1664,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708148,"statfs":{"total":95995035648,"available":95957143552,"internally_reserved":0,"allocated":1703936,"data_stored":1125482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":62582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":69764,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1669,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":110592,"data_stored":69992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-31T23:11:31.749 INFO:tasks.fwd_scrub.ceph_manager:clean! 2024-03-31T23:11:31.749 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-31T23:11:31.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:31.888+0000 7f6c875e9640 1 -- 172.21.15.46:0/476444733 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6c800ffa60 msgr2=0x7f6c800ffe80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:31.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:31.888+0000 7f6c875e9640 1 --2- 172.21.15.46:0/476444733 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6c800ffa60 0x7f6c800ffe80 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f6c7c009930 tx=0x7f6c7c02bc30 comp rx=0 tx=0).stop 2024-03-31T23:11:31.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:31.888+0000 7f6c875e9640 1 -- 172.21.15.46:0/476444733 shutdown_connections 2024-03-31T23:11:31.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:31.888+0000 7f6c875e9640 1 --2- 172.21.15.46:0/476444733 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6c801033f0 0x7f6c80109c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:31.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:31.888+0000 7f6c875e9640 1 --2- 172.21.15.46:0/476444733 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6c80102a30 0x7f6c80102eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:31.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:31.888+0000 7f6c875e9640 1 --2- 172.21.15.46:0/476444733 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6c800ffa60 0x7f6c800ffe80 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:31.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:31.888+0000 7f6c875e9640 1 -- 172.21.15.46:0/476444733 >> 172.21.15.46:0/476444733 conn(0x7f6c800fb390 msgr2=0x7f6c800fd7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:11:31.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:31.888+0000 7f6c875e9640 1 -- 172.21.15.46:0/476444733 shutdown_connections 2024-03-31T23:11:31.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:31.888+0000 7f6c875e9640 1 -- 172.21.15.46:0/476444733 wait complete. 2024-03-31T23:11:31.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:31.889+0000 7f6c875e9640 1 Processor -- start 2024-03-31T23:11:31.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:31.889+0000 7f6c875e9640 1 -- start start 2024-03-31T23:11:31.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:31.889+0000 7f6c875e9640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6c800ffa60 0x7f6c801023a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:31.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:31.889+0000 7f6c875e9640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6c80102a30 0x7f6c801009e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:31.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:31.889+0000 7f6c875e9640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6c801033f0 0x7f6c80100f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:31.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:31.889+0000 7f6c875e9640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f6c8010d2c0 con 0x7f6c801033f0 2024-03-31T23:11:31.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:31.889+0000 7f6c875e9640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f6c8010d140 con 0x7f6c800ffa60 2024-03-31T23:11:31.892 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:31.889+0000 7f6c875e9640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f6c8010d440 con 0x7f6c80102a30 2024-03-31T23:11:31.892 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:31.889+0000 7f6c85b86640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6c801033f0 0x7f6c80100f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:31.892 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:31.889+0000 7f6c85b86640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6c801033f0 0x7f6c80100f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:48122/0 (socket says 172.21.15.46:48122) 2024-03-31T23:11:31.892 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:31.889+0000 7f6c85b86640 1 -- 172.21.15.46:0/876424742 learned_addr learned my addr 172.21.15.46:0/876424742 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:11:31.892 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:31.889+0000 7f6c85385640 1 --2- 172.21.15.46:0/876424742 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6c800ffa60 0x7f6c801023a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:31.892 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:31.889+0000 7f6c84b84640 1 --2- 172.21.15.46:0/876424742 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6c80102a30 0x7f6c801009e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:31.892 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:31.889+0000 7f6c85b86640 1 -- 172.21.15.46:0/876424742 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6c80102a30 msgr2=0x7f6c801009e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:31.892 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:31.889+0000 7f6c85b86640 1 --2- 172.21.15.46:0/876424742 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6c80102a30 0x7f6c801009e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:31.892 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:31.889+0000 7f6c85b86640 1 -- 172.21.15.46:0/876424742 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6c800ffa60 msgr2=0x7f6c801023a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:31.892 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:31.889+0000 7f6c85b86640 1 --2- 172.21.15.46:0/876424742 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6c800ffa60 0x7f6c801023a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:31.892 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:31.889+0000 7f6c85b86640 1 -- 172.21.15.46:0/876424742 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6c801017b0 con 0x7f6c801033f0 2024-03-31T23:11:31.892 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:31.890+0000 7f6c85b86640 1 --2- 172.21.15.46:0/876424742 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6c801033f0 0x7f6c80100f20 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f6c7800eb10 tx=0x7f6c7800c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:11:31.892 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:31.890+0000 7f6c767fc640 1 -- 172.21.15.46:0/876424742 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c78004b20 con 0x7f6c801033f0 2024-03-31T23:11:31.892 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:31.890+0000 7f6c767fc640 1 -- 172.21.15.46:0/876424742 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6c78004cc0 con 0x7f6c801033f0 2024-03-31T23:11:31.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:31.890+0000 7f6c767fc640 1 -- 172.21.15.46:0/876424742 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c78010690 con 0x7f6c801033f0 2024-03-31T23:11:31.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:31.890+0000 7f6c875e9640 1 -- 172.21.15.46:0/876424742 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6c801a1b20 con 0x7f6c801033f0 2024-03-31T23:11:31.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:31.890+0000 7f6c875e9640 1 -- 172.21.15.46:0/876424742 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6c801a2070 con 0x7f6c801033f0 2024-03-31T23:11:31.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:31.891+0000 7f6c875e9640 1 -- 172.21.15.46:0/876424742 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6c80108c80 con 0x7f6c801033f0 2024-03-31T23:11:31.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:31.892+0000 7f6c767fc640 1 -- 172.21.15.46:0/876424742 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 18) ==== 137320+0+0 (secure 0 0 0) 0x7f6c78005020 con 0x7f6c801033f0 2024-03-31T23:11:31.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:31.893+0000 7f6c767fc640 1 --2- 172.21.15.46:0/876424742 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f6c6c0a2520 0x7f6c6c0a49e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:31.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:31.893+0000 7f6c85385640 1 --2- 172.21.15.46:0/876424742 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f6c6c0a2520 0x7f6c6c0a49e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:31.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:31.893+0000 7f6c767fc640 1 -- 172.21.15.46:0/876424742 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9208+0+0 (secure 0 0 0) 0x7f6c780cf0d0 con 0x7f6c801033f0 2024-03-31T23:11:31.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:31.893+0000 7f6c85385640 1 --2- 172.21.15.46:0/876424742 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f6c6c0a2520 0x7f6c6c0a49e0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f6c7c000c00 tx=0x7f6c7c005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:11:31.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:31.897+0000 7f6c767fc640 1 -- 172.21.15.46:0/876424742 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f6c7809be60 con 0x7f6c801033f0 2024-03-31T23:11:31.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:11:31 smithi142 ceph-mon[35271]: from='client.? 172.21.15.46:0/4229979867' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-31T23:11:31.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:11:31 smithi142 ceph-mon[35271]: from='client.? 172.21.15.46:0/3641190743' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-31T23:11:32.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:11:31 smithi162 ceph-mon[36237]: from='client.? 172.21.15.46:0/4229979867' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-31T23:11:32.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:11:31 smithi162 ceph-mon[36237]: from='client.? 172.21.15.46:0/3641190743' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-31T23:11:32.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:32.078+0000 7f6c875e9640 1 -- 172.21.15.46:0/876424742 --> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f6c80061b90 con 0x7f6c6c0a2520 2024-03-31T23:11:32.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:32.088+0000 7f6c767fc640 1 -- 172.21.15.46:0/876424742 <== mgr.14152 v2:172.21.15.46:6800/162289569 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340476 (secure 0 0 0) 0x7f6c80061b90 con 0x7f6c6c0a2520 2024-03-31T23:11:32.092 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-31T23:11:32.094 INFO:teuthology.orchestra.run.smithi046.stderr:dumped all 2024-03-31T23:11:32.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:32.091+0000 7f6c875e9640 1 -- 172.21.15.46:0/876424742 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f6c6c0a2520 msgr2=0x7f6c6c0a49e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:32.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:32.091+0000 7f6c875e9640 1 --2- 172.21.15.46:0/876424742 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f6c6c0a2520 0x7f6c6c0a49e0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f6c7c000c00 tx=0x7f6c7c005fd0 comp rx=0 tx=0).stop 2024-03-31T23:11:32.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:32.091+0000 7f6c875e9640 1 -- 172.21.15.46:0/876424742 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6c801033f0 msgr2=0x7f6c80100f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:32.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:32.091+0000 7f6c875e9640 1 --2- 172.21.15.46:0/876424742 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6c801033f0 0x7f6c80100f20 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f6c7800eb10 tx=0x7f6c7800c2d0 comp rx=0 tx=0).stop 2024-03-31T23:11:32.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:32.091+0000 7f6c875e9640 1 -- 172.21.15.46:0/876424742 shutdown_connections 2024-03-31T23:11:32.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:32.091+0000 7f6c875e9640 1 --2- 172.21.15.46:0/876424742 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f6c6c0a2520 0x7f6c6c0a49e0 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:32.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:32.091+0000 7f6c875e9640 1 --2- 172.21.15.46:0/876424742 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6c801033f0 0x7f6c80100f20 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:32.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:32.091+0000 7f6c875e9640 1 --2- 172.21.15.46:0/876424742 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6c80102a30 0x7f6c801009e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:32.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:32.091+0000 7f6c875e9640 1 --2- 172.21.15.46:0/876424742 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6c800ffa60 0x7f6c801023a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:32.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:32.091+0000 7f6c875e9640 1 -- 172.21.15.46:0/876424742 >> 172.21.15.46:0/876424742 conn(0x7f6c800fb390 msgr2=0x7f6c80105e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:11:32.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:32.091+0000 7f6c875e9640 1 -- 172.21.15.46:0/876424742 shutdown_connections 2024-03-31T23:11:32.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:32.091+0000 7f6c875e9640 1 -- 172.21.15.46:0/876424742 wait complete. 2024-03-31T23:11:32.102 INFO:teuthology.orchestra.run.smithi046.stdout:{"pg_ready":true,"pg_map":{"version":366,"stamp":"2024-03-31T23:11:30.612781+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":663158,"num_objects":64,"num_object_clones":0,"num_object_copies":194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":880,"num_read_kb":570,"num_write":365,"num_write_kb":3077,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":797,"ondisk_log_size":797,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":372340,"kb_used_data":14824,"kb_used_omap":18,"kb_used_meta":357357,"kb_avail":1124569484,"statfs":{"total":1151940427776,"available":1151559151616,"internally_reserved":0,"allocated":15179776,"data_stored":8364853,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19140,"internal_metadata":365933884},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":5300,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":82,"num_read_kb":51,"num_write":10,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003267"},"pg_stats":[{"pgid":"3.3f","version":"93'8","reported_seq":40,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:10.543291+0000","last_change":"2024-03-31T23:09:11.177502+0000","last_active":"2024-03-31T23:11:10.543291+0000","last_peered":"2024-03-31T23:11:10.543291+0000","last_clean":"2024-03-31T23:11:10.543291+0000","last_became_active":"2024-03-31T23:09:11.177134+0000","last_became_peered":"2024-03-31T23:09:11.177134+0000","last_unstale":"2024-03-31T23:11:10.543291+0000","last_undegraded":"2024-03-31T23:11:10.543291+0000","last_fullsized":"2024-03-31T23:11:10.543291+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T10:29:36.508229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"93'4","reported_seq":36,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:11.122776+0000","last_change":"2024-03-31T23:09:11.172990+0000","last_active":"2024-03-31T23:11:11.122776+0000","last_peered":"2024-03-31T23:11:11.122776+0000","last_clean":"2024-03-31T23:11:11.122776+0000","last_became_active":"2024-03-31T23:09:11.172684+0000","last_became_peered":"2024-03-31T23:09:11.172684+0000","last_unstale":"2024-03-31T23:11:11.122776+0000","last_undegraded":"2024-03-31T23:11:11.122776+0000","last_fullsized":"2024-03-31T23:11:11.122776+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T07:02:59.991757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"93'7","reported_seq":39,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:11.122648+0000","last_change":"2024-03-31T23:09:11.176805+0000","last_active":"2024-03-31T23:11:11.122648+0000","last_peered":"2024-03-31T23:11:11.122648+0000","last_clean":"2024-03-31T23:11:11.122648+0000","last_became_active":"2024-03-31T23:09:11.176468+0000","last_became_peered":"2024-03-31T23:09:11.176468+0000","last_unstale":"2024-03-31T23:11:11.122648+0000","last_undegraded":"2024-03-31T23:11:11.122648+0000","last_fullsized":"2024-03-31T23:11:11.122648+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T08:23:41.872558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"93'8","reported_seq":40,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:10.545278+0000","last_change":"2024-03-31T23:09:11.182385+0000","last_active":"2024-03-31T23:11:10.545278+0000","last_peered":"2024-03-31T23:11:10.545278+0000","last_clean":"2024-03-31T23:11:10.545278+0000","last_became_active":"2024-03-31T23:09:11.181908+0000","last_became_peered":"2024-03-31T23:09:11.181908+0000","last_unstale":"2024-03-31T23:11:10.545278+0000","last_undegraded":"2024-03-31T23:11:10.545278+0000","last_fullsized":"2024-03-31T23:11:10.545278+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:52:24.066360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"93'4","reported_seq":35,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:10.542115+0000","last_change":"2024-03-31T23:09:11.181377+0000","last_active":"2024-03-31T23:11:10.542115+0000","last_peered":"2024-03-31T23:11:10.542115+0000","last_clean":"2024-03-31T23:11:10.542115+0000","last_became_active":"2024-03-31T23:09:11.181064+0000","last_became_peered":"2024-03-31T23:09:11.181064+0000","last_unstale":"2024-03-31T23:11:10.542115+0000","last_undegraded":"2024-03-31T23:11:10.542115+0000","last_fullsized":"2024-03-31T23:11:10.542115+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:28:40.766541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"93'6","reported_seq":37,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:11.124851+0000","last_change":"2024-03-31T23:09:11.176087+0000","last_active":"2024-03-31T23:11:11.124851+0000","last_peered":"2024-03-31T23:11:11.124851+0000","last_clean":"2024-03-31T23:11:11.124851+0000","last_became_active":"2024-03-31T23:09:11.175569+0000","last_became_peered":"2024-03-31T23:09:11.175569+0000","last_unstale":"2024-03-31T23:11:11.124851+0000","last_undegraded":"2024-03-31T23:11:11.124851+0000","last_fullsized":"2024-03-31T23:11:11.124851+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T06:33:19.189909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"93'5","reported_seq":36,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:10.546125+0000","last_change":"2024-03-31T23:09:11.170289+0000","last_active":"2024-03-31T23:11:10.546125+0000","last_peered":"2024-03-31T23:11:10.546125+0000","last_clean":"2024-03-31T23:11:10.546125+0000","last_became_active":"2024-03-31T23:09:11.169959+0000","last_became_peered":"2024-03-31T23:09:11.169959+0000","last_unstale":"2024-03-31T23:11:10.546125+0000","last_undegraded":"2024-03-31T23:11:10.546125+0000","last_fullsized":"2024-03-31T23:11:10.546125+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:10:44.614767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"93'9","reported_seq":40,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:10.540524+0000","last_change":"2024-03-31T23:09:11.182447+0000","last_active":"2024-03-31T23:11:10.540524+0000","last_peered":"2024-03-31T23:11:10.540524+0000","last_clean":"2024-03-31T23:11:10.540524+0000","last_became_active":"2024-03-31T23:09:11.181629+0000","last_became_peered":"2024-03-31T23:09:11.181629+0000","last_unstale":"2024-03-31T23:11:10.540524+0000","last_undegraded":"2024-03-31T23:11:10.540524+0000","last_fullsized":"2024-03-31T23:11:10.540524+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:18:26.269451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"93'11","reported_seq":42,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:10.934818+0000","last_change":"2024-03-31T23:09:11.176853+0000","last_active":"2024-03-31T23:11:10.934818+0000","last_peered":"2024-03-31T23:11:10.934818+0000","last_clean":"2024-03-31T23:11:10.934818+0000","last_became_active":"2024-03-31T23:09:11.176413+0000","last_became_peered":"2024-03-31T23:09:11.176413+0000","last_unstale":"2024-03-31T23:11:10.934818+0000","last_undegraded":"2024-03-31T23:11:10.934818+0000","last_fullsized":"2024-03-31T23:11:10.934818+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:23:40.486947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"93'16","reported_seq":53,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:11.125686+0000","last_change":"2024-03-31T23:10:10.543998+0000","last_active":"2024-03-31T23:11:11.125686+0000","last_peered":"2024-03-31T23:11:11.125686+0000","last_clean":"2024-03-31T23:11:11.125686+0000","last_became_active":"2024-03-31T23:10:10.543615+0000","last_became_peered":"2024-03-31T23:10:10.543615+0000","last_unstale":"2024-03-31T23:11:11.125686+0000","last_undegraded":"2024-03-31T23:11:11.125686+0000","last_fullsized":"2024-03-31T23:11:11.125686+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:28:32.883732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,9],"acting":[10,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"93'6","reported_seq":37,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:10.545678+0000","last_change":"2024-03-31T23:09:11.179536+0000","last_active":"2024-03-31T23:11:10.545678+0000","last_peered":"2024-03-31T23:11:10.545678+0000","last_clean":"2024-03-31T23:11:10.545678+0000","last_became_active":"2024-03-31T23:09:11.178521+0000","last_became_peered":"2024-03-31T23:09:11.178521+0000","last_unstale":"2024-03-31T23:11:10.545678+0000","last_undegraded":"2024-03-31T23:11:10.545678+0000","last_fullsized":"2024-03-31T23:11:10.545678+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T09:29:23.128832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"93'7","reported_seq":38,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:10.545128+0000","last_change":"2024-03-31T23:09:11.182548+0000","last_active":"2024-03-31T23:11:10.545128+0000","last_peered":"2024-03-31T23:11:10.545128+0000","last_clean":"2024-03-31T23:11:10.545128+0000","last_became_active":"2024-03-31T23:09:11.181966+0000","last_became_peered":"2024-03-31T23:09:11.181966+0000","last_unstale":"2024-03-31T23:11:10.545128+0000","last_undegraded":"2024-03-31T23:11:10.545128+0000","last_fullsized":"2024-03-31T23:11:10.545128+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:43:26.061594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"93'10","reported_seq":47,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:10.716409+0000","last_change":"2024-03-31T23:10:10.544976+0000","last_active":"2024-03-31T23:11:10.716409+0000","last_peered":"2024-03-31T23:11:10.716409+0000","last_clean":"2024-03-31T23:11:10.716409+0000","last_became_active":"2024-03-31T23:10:10.544433+0000","last_became_peered":"2024-03-31T23:10:10.544433+0000","last_unstale":"2024-03-31T23:11:10.716409+0000","last_undegraded":"2024-03-31T23:11:10.716409+0000","last_fullsized":"2024-03-31T23:11:10.716409+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:45:54.606274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,9],"acting":[6,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"93'7","reported_seq":38,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:11.122229+0000","last_change":"2024-03-31T23:09:11.168784+0000","last_active":"2024-03-31T23:11:11.122229+0000","last_peered":"2024-03-31T23:11:11.122229+0000","last_clean":"2024-03-31T23:11:11.122229+0000","last_became_active":"2024-03-31T23:09:11.168442+0000","last_became_peered":"2024-03-31T23:09:11.168442+0000","last_unstale":"2024-03-31T23:11:11.122229+0000","last_undegraded":"2024-03-31T23:11:11.122229+0000","last_fullsized":"2024-03-31T23:11:11.122229+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T09:37:43.568122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"93'7","reported_seq":38,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:11.122185+0000","last_change":"2024-03-31T23:09:11.176017+0000","last_active":"2024-03-31T23:11:11.122185+0000","last_peered":"2024-03-31T23:11:11.122185+0000","last_clean":"2024-03-31T23:11:11.122185+0000","last_became_active":"2024-03-31T23:09:11.175690+0000","last_became_peered":"2024-03-31T23:09:11.175690+0000","last_unstale":"2024-03-31T23:11:11.122185+0000","last_undegraded":"2024-03-31T23:11:11.122185+0000","last_fullsized":"2024-03-31T23:11:11.122185+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:14:13.259895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"93'9","reported_seq":40,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:10.934173+0000","last_change":"2024-03-31T23:09:11.171926+0000","last_active":"2024-03-31T23:11:10.934173+0000","last_peered":"2024-03-31T23:11:10.934173+0000","last_clean":"2024-03-31T23:11:10.934173+0000","last_became_active":"2024-03-31T23:09:11.171625+0000","last_became_peered":"2024-03-31T23:09:11.171625+0000","last_unstale":"2024-03-31T23:11:10.934173+0000","last_undegraded":"2024-03-31T23:11:10.934173+0000","last_fullsized":"2024-03-31T23:11:10.934173+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T06:20:20.433802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"93'8","reported_seq":39,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:10.543011+0000","last_change":"2024-03-31T23:09:11.183805+0000","last_active":"2024-03-31T23:11:10.543011+0000","last_peered":"2024-03-31T23:11:10.543011+0000","last_clean":"2024-03-31T23:11:10.543011+0000","last_became_active":"2024-03-31T23:09:11.183464+0000","last_became_peered":"2024-03-31T23:09:11.183464+0000","last_unstale":"2024-03-31T23:11:10.543011+0000","last_undegraded":"2024-03-31T23:11:10.543011+0000","last_fullsized":"2024-03-31T23:11:10.543011+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T06:02:33.286563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"93'10","reported_seq":41,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:11.117069+0000","last_change":"2024-03-31T23:09:11.170095+0000","last_active":"2024-03-31T23:11:11.117069+0000","last_peered":"2024-03-31T23:11:11.117069+0000","last_clean":"2024-03-31T23:11:11.117069+0000","last_became_active":"2024-03-31T23:09:11.169683+0000","last_became_peered":"2024-03-31T23:09:11.169683+0000","last_unstale":"2024-03-31T23:11:11.117069+0000","last_undegraded":"2024-03-31T23:11:11.117069+0000","last_fullsized":"2024-03-31T23:11:11.117069+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T08:19:47.101431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"93'7","reported_seq":39,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:10.544657+0000","last_change":"2024-03-31T23:09:11.181347+0000","last_active":"2024-03-31T23:11:10.544657+0000","last_peered":"2024-03-31T23:11:10.544657+0000","last_clean":"2024-03-31T23:11:10.544657+0000","last_became_active":"2024-03-31T23:09:11.180510+0000","last_became_peered":"2024-03-31T23:09:11.180510+0000","last_unstale":"2024-03-31T23:11:10.544657+0000","last_undegraded":"2024-03-31T23:11:10.544657+0000","last_fullsized":"2024-03-31T23:11:10.544657+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:25:51.083446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:10.544622+0000","last_change":"2024-03-31T23:09:15.205059+0000","last_active":"2024-03-31T23:11:10.544622+0000","last_peered":"2024-03-31T23:11:10.544622+0000","last_clean":"2024-03-31T23:11:10.544622+0000","last_became_active":"2024-03-31T23:09:15.204612+0000","last_became_peered":"2024-03-31T23:09:15.204612+0000","last_unstale":"2024-03-31T23:11:10.544622+0000","last_undegraded":"2024-03-31T23:11:10.544622+0000","last_fullsized":"2024-03-31T23:11:10.544622+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_clean_scrub_stamp":"2024-03-31T23:09:14.171251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:53:33.998026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"91'9","reported_seq":44,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:11.116248+0000","last_change":"2024-03-31T23:09:10.594147+0000","last_active":"2024-03-31T23:11:11.116248+0000","last_peered":"2024-03-31T23:11:11.116248+0000","last_clean":"2024-03-31T23:11:11.116248+0000","last_became_active":"2024-03-31T23:09:10.592047+0000","last_became_peered":"2024-03-31T23:09:10.592047+0000","last_unstale":"2024-03-31T23:11:11.116248+0000","last_undegraded":"2024-03-31T23:11:11.116248+0000","last_fullsized":"2024-03-31T23:11:11.116248+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_clean_scrub_stamp":"2024-03-31T23:09:09.564263+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:04:34.548282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"93'5","reported_seq":37,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:10.541242+0000","last_change":"2024-03-31T23:09:11.182292+0000","last_active":"2024-03-31T23:11:10.541242+0000","last_peered":"2024-03-31T23:11:10.541242+0000","last_clean":"2024-03-31T23:11:10.541242+0000","last_became_active":"2024-03-31T23:09:11.181885+0000","last_became_peered":"2024-03-31T23:09:11.181885+0000","last_unstale":"2024-03-31T23:11:10.541242+0000","last_undegraded":"2024-03-31T23:11:10.541242+0000","last_fullsized":"2024-03-31T23:11:10.541242+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T10:38:17.550625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":32,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:10.544451+0000","last_change":"2024-03-31T23:10:10.543802+0000","last_active":"2024-03-31T23:11:10.544451+0000","last_peered":"2024-03-31T23:11:10.544451+0000","last_clean":"2024-03-31T23:11:10.544451+0000","last_became_active":"2024-03-31T23:10:10.543245+0000","last_became_peered":"2024-03-31T23:10:10.543245+0000","last_unstale":"2024-03-31T23:11:10.544451+0000","last_undegraded":"2024-03-31T23:11:10.544451+0000","last_fullsized":"2024-03-31T23:11:10.544451+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_clean_scrub_stamp":"2024-03-31T23:09:14.171251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T06:00:11.707183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,8,3],"acting":[4,6,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"91'2","reported_seq":36,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:10.541490+0000","last_change":"2024-03-31T23:09:10.598375+0000","last_active":"2024-03-31T23:11:10.541490+0000","last_peered":"2024-03-31T23:11:10.541490+0000","last_clean":"2024-03-31T23:11:10.541490+0000","last_became_active":"2024-03-31T23:09:10.598045+0000","last_became_peered":"2024-03-31T23:09:10.598045+0000","last_unstale":"2024-03-31T23:11:10.541490+0000","last_undegraded":"2024-03-31T23:11:10.541490+0000","last_fullsized":"2024-03-31T23:11:10.541490+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_clean_scrub_stamp":"2024-03-31T23:09:09.564263+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:31:27.113317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"93'6","reported_seq":38,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:11.122739+0000","last_change":"2024-03-31T23:09:11.183458+0000","last_active":"2024-03-31T23:11:11.122739+0000","last_peered":"2024-03-31T23:11:11.122739+0000","last_clean":"2024-03-31T23:11:11.122739+0000","last_became_active":"2024-03-31T23:09:11.183100+0000","last_became_peered":"2024-03-31T23:09:11.183100+0000","last_unstale":"2024-03-31T23:11:11.122739+0000","last_undegraded":"2024-03-31T23:11:11.122739+0000","last_fullsized":"2024-03-31T23:11:11.122739+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T10:55:35.512065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":32,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:11.271365+0000","last_change":"2024-03-31T23:11:11.271365+0000","last_active":"2024-03-31T23:11:11.271365+0000","last_peered":"2024-03-31T23:11:11.271365+0000","last_clean":"2024-03-31T23:11:11.271365+0000","last_became_active":"2024-03-31T23:11:11.270814+0000","last_became_peered":"2024-03-31T23:11:11.270814+0000","last_unstale":"2024-03-31T23:11:11.271365+0000","last_undegraded":"2024-03-31T23:11:11.271365+0000","last_fullsized":"2024-03-31T23:11:11.271365+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_clean_scrub_stamp":"2024-03-31T23:09:14.171251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:58:27.916348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,10,5],"acting":[4,3,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"91'4","reported_seq":37,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:10.541863+0000","last_change":"2024-03-31T23:09:10.597471+0000","last_active":"2024-03-31T23:11:10.541863+0000","last_peered":"2024-03-31T23:11:10.541863+0000","last_clean":"2024-03-31T23:11:10.541863+0000","last_became_active":"2024-03-31T23:09:10.597165+0000","last_became_peered":"2024-03-31T23:09:10.597165+0000","last_unstale":"2024-03-31T23:11:10.541863+0000","last_undegraded":"2024-03-31T23:11:10.541863+0000","last_fullsized":"2024-03-31T23:11:10.541863+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_clean_scrub_stamp":"2024-03-31T23:09:09.564263+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:10:26.826525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"93'9","reported_seq":40,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:10.934521+0000","last_change":"2024-03-31T23:09:11.170650+0000","last_active":"2024-03-31T23:11:10.934521+0000","last_peered":"2024-03-31T23:11:10.934521+0000","last_clean":"2024-03-31T23:11:10.934521+0000","last_became_active":"2024-03-31T23:09:11.170291+0000","last_became_peered":"2024-03-31T23:09:11.170291+0000","last_unstale":"2024-03-31T23:11:10.934521+0000","last_undegraded":"2024-03-31T23:11:10.934521+0000","last_fullsized":"2024-03-31T23:11:10.934521+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:29:54.904377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:10.544466+0000","last_change":"2024-03-31T23:09:15.204857+0000","last_active":"2024-03-31T23:11:10.544466+0000","last_peered":"2024-03-31T23:11:10.544466+0000","last_clean":"2024-03-31T23:11:10.544466+0000","last_became_active":"2024-03-31T23:09:15.204484+0000","last_became_peered":"2024-03-31T23:09:15.204484+0000","last_unstale":"2024-03-31T23:11:10.544466+0000","last_undegraded":"2024-03-31T23:11:10.544466+0000","last_fullsized":"2024-03-31T23:11:10.544466+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_clean_scrub_stamp":"2024-03-31T23:09:14.171251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T09:54:24.646918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"85'2","reported_seq":35,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:10.541624+0000","last_change":"2024-03-31T23:09:10.589922+0000","last_active":"2024-03-31T23:11:10.541624+0000","last_peered":"2024-03-31T23:11:10.541624+0000","last_clean":"2024-03-31T23:11:10.541624+0000","last_became_active":"2024-03-31T23:09:10.589569+0000","last_became_peered":"2024-03-31T23:09:10.589569+0000","last_unstale":"2024-03-31T23:11:10.541624+0000","last_undegraded":"2024-03-31T23:11:10.541624+0000","last_fullsized":"2024-03-31T23:11:10.541624+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_clean_scrub_stamp":"2024-03-31T23:09:09.564263+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:57:05.347344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"93'15","reported_seq":52,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:11.270163+0000","last_change":"2024-03-31T23:10:10.485015+0000","last_active":"2024-03-31T23:11:11.270163+0000","last_peered":"2024-03-31T23:11:11.270163+0000","last_clean":"2024-03-31T23:11:11.270163+0000","last_became_active":"2024-03-31T23:10:10.484663+0000","last_became_peered":"2024-03-31T23:10:10.484663+0000","last_unstale":"2024-03-31T23:11:11.270163+0000","last_undegraded":"2024-03-31T23:11:11.270163+0000","last_fullsized":"2024-03-31T23:11:11.270163+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T06:18:50.834711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,1],"acting":[5,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:10.544476+0000","last_change":"2024-03-31T23:09:15.193096+0000","last_active":"2024-03-31T23:11:10.544476+0000","last_peered":"2024-03-31T23:11:10.544476+0000","last_clean":"2024-03-31T23:11:10.544476+0000","last_became_active":"2024-03-31T23:09:15.192624+0000","last_became_peered":"2024-03-31T23:09:15.192624+0000","last_unstale":"2024-03-31T23:11:10.544476+0000","last_undegraded":"2024-03-31T23:11:10.544476+0000","last_fullsized":"2024-03-31T23:11:10.544476+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_clean_scrub_stamp":"2024-03-31T23:09:14.171251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:22:24.921346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"93'7","reported_seq":161,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:25.511651+0000","last_change":"2024-03-31T23:09:10.587209+0000","last_active":"2024-03-31T23:11:25.511651+0000","last_peered":"2024-03-31T23:11:25.511651+0000","last_clean":"2024-03-31T23:11:25.511651+0000","last_became_active":"2024-03-31T23:09:10.586486+0000","last_became_peered":"2024-03-31T23:09:10.586486+0000","last_unstale":"2024-03-31T23:11:25.511651+0000","last_undegraded":"2024-03-31T23:11:25.511651+0000","last_fullsized":"2024-03-31T23:11:25.511651+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_clean_scrub_stamp":"2024-03-31T23:09:09.564263+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:27:56.046414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1154,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":121,"num_read_kb":4,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"93'7","reported_seq":44,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:10.541409+0000","last_change":"2024-03-31T23:10:10.543713+0000","last_active":"2024-03-31T23:11:10.541409+0000","last_peered":"2024-03-31T23:11:10.541409+0000","last_clean":"2024-03-31T23:11:10.541409+0000","last_became_active":"2024-03-31T23:10:10.543324+0000","last_became_peered":"2024-03-31T23:10:10.543324+0000","last_unstale":"2024-03-31T23:11:10.541409+0000","last_undegraded":"2024-03-31T23:11:10.541409+0000","last_fullsized":"2024-03-31T23:11:10.541409+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:41:21.974325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,6],"acting":[8,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:10.544389+0000","last_change":"2024-03-31T23:09:15.194717+0000","last_active":"2024-03-31T23:11:10.544389+0000","last_peered":"2024-03-31T23:11:10.544389+0000","last_clean":"2024-03-31T23:11:10.544389+0000","last_became_active":"2024-03-31T23:09:15.194232+0000","last_became_peered":"2024-03-31T23:09:15.194232+0000","last_unstale":"2024-03-31T23:11:10.544389+0000","last_undegraded":"2024-03-31T23:11:10.544389+0000","last_fullsized":"2024-03-31T23:11:10.544389+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_clean_scrub_stamp":"2024-03-31T23:09:14.171251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:21:00.445232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"91'3","reported_seq":36,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:11.124484+0000","last_change":"2024-03-31T23:09:10.586346+0000","last_active":"2024-03-31T23:11:11.124484+0000","last_peered":"2024-03-31T23:11:11.124484+0000","last_clean":"2024-03-31T23:11:11.124484+0000","last_became_active":"2024-03-31T23:09:10.586043+0000","last_became_peered":"2024-03-31T23:09:10.586043+0000","last_unstale":"2024-03-31T23:11:11.124484+0000","last_undegraded":"2024-03-31T23:11:11.124484+0000","last_fullsized":"2024-03-31T23:11:11.124484+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_clean_scrub_stamp":"2024-03-31T23:09:09.564263+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:17:27.387786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"93'8","reported_seq":45,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:11.270118+0000","last_change":"2024-03-31T23:10:10.543353+0000","last_active":"2024-03-31T23:11:11.270118+0000","last_peered":"2024-03-31T23:11:11.270118+0000","last_clean":"2024-03-31T23:11:11.270118+0000","last_became_active":"2024-03-31T23:10:10.542961+0000","last_became_peered":"2024-03-31T23:10:10.542961+0000","last_unstale":"2024-03-31T23:11:11.270118+0000","last_undegraded":"2024-03-31T23:11:11.270118+0000","last_fullsized":"2024-03-31T23:11:11.270118+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:28:52.950551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3],"acting":[5,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":26,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:10.542953+0000","last_change":"2024-03-31T23:09:15.205855+0000","last_active":"2024-03-31T23:11:10.542953+0000","last_peered":"2024-03-31T23:11:10.542953+0000","last_clean":"2024-03-31T23:11:10.542953+0000","last_became_active":"2024-03-31T23:09:15.205359+0000","last_became_peered":"2024-03-31T23:09:15.205359+0000","last_unstale":"2024-03-31T23:11:10.542953+0000","last_undegraded":"2024-03-31T23:11:10.542953+0000","last_fullsized":"2024-03-31T23:11:10.542953+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_clean_scrub_stamp":"2024-03-31T23:09:14.171251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T09:39:32.461011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"91'2","reported_seq":35,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:10.715951+0000","last_change":"2024-03-31T23:09:10.583274+0000","last_active":"2024-03-31T23:11:10.715951+0000","last_peered":"2024-03-31T23:11:10.715951+0000","last_clean":"2024-03-31T23:11:10.715951+0000","last_became_active":"2024-03-31T23:09:10.582910+0000","last_became_peered":"2024-03-31T23:09:10.582910+0000","last_unstale":"2024-03-31T23:11:10.715951+0000","last_undegraded":"2024-03-31T23:11:10.715951+0000","last_fullsized":"2024-03-31T23:11:10.715951+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_clean_scrub_stamp":"2024-03-31T23:09:09.564263+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:16:32.251169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"93'8","reported_seq":39,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:10.540701+0000","last_change":"2024-03-31T23:09:11.178619+0000","last_active":"2024-03-31T23:11:10.540701+0000","last_peered":"2024-03-31T23:11:10.540701+0000","last_clean":"2024-03-31T23:11:10.540701+0000","last_became_active":"2024-03-31T23:09:11.178239+0000","last_became_peered":"2024-03-31T23:09:11.178239+0000","last_unstale":"2024-03-31T23:11:10.540701+0000","last_undegraded":"2024-03-31T23:11:10.540701+0000","last_fullsized":"2024-03-31T23:11:10.540701+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:28:58.863199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":26,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:11.122693+0000","last_change":"2024-03-31T23:09:15.205864+0000","last_active":"2024-03-31T23:11:11.122693+0000","last_peered":"2024-03-31T23:11:11.122693+0000","last_clean":"2024-03-31T23:11:11.122693+0000","last_became_active":"2024-03-31T23:09:15.204976+0000","last_became_peered":"2024-03-31T23:09:15.204976+0000","last_unstale":"2024-03-31T23:11:11.122693+0000","last_undegraded":"2024-03-31T23:11:11.122693+0000","last_fullsized":"2024-03-31T23:11:11.122693+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_clean_scrub_stamp":"2024-03-31T23:09:14.171251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:39:08.537827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"91'1","reported_seq":34,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:11.269487+0000","last_change":"2024-03-31T23:09:10.588407+0000","last_active":"2024-03-31T23:11:11.269487+0000","last_peered":"2024-03-31T23:11:11.269487+0000","last_clean":"2024-03-31T23:11:11.269487+0000","last_became_active":"2024-03-31T23:09:10.588109+0000","last_became_peered":"2024-03-31T23:09:10.588109+0000","last_unstale":"2024-03-31T23:11:11.269487+0000","last_undegraded":"2024-03-31T23:11:11.269487+0000","last_fullsized":"2024-03-31T23:11:11.269487+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_clean_scrub_stamp":"2024-03-31T23:09:09.564263+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:01:25.858834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"93'11","reported_seq":42,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:11.116446+0000","last_change":"2024-03-31T23:09:11.180452+0000","last_active":"2024-03-31T23:11:11.116446+0000","last_peered":"2024-03-31T23:11:11.116446+0000","last_clean":"2024-03-31T23:11:11.116446+0000","last_became_active":"2024-03-31T23:09:11.180117+0000","last_became_peered":"2024-03-31T23:09:11.180117+0000","last_unstale":"2024-03-31T23:11:11.116446+0000","last_undegraded":"2024-03-31T23:11:11.116446+0000","last_fullsized":"2024-03-31T23:11:11.116446+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:04:58.288981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:10.540663+0000","last_change":"2024-03-31T23:09:15.199669+0000","last_active":"2024-03-31T23:11:10.540663+0000","last_peered":"2024-03-31T23:11:10.540663+0000","last_clean":"2024-03-31T23:11:10.540663+0000","last_became_active":"2024-03-31T23:09:15.199262+0000","last_became_peered":"2024-03-31T23:09:15.199262+0000","last_unstale":"2024-03-31T23:11:10.540663+0000","last_undegraded":"2024-03-31T23:11:10.540663+0000","last_fullsized":"2024-03-31T23:11:10.540663+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_clean_scrub_stamp":"2024-03-31T23:09:14.171251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:44:02.975000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"95'5","reported_seq":158,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:28.514876+0000","last_change":"2024-03-31T23:09:10.583476+0000","last_active":"2024-03-31T23:11:28.514876+0000","last_peered":"2024-03-31T23:11:28.514876+0000","last_clean":"2024-03-31T23:11:28.514876+0000","last_became_active":"2024-03-31T23:09:10.582789+0000","last_became_peered":"2024-03-31T23:09:10.582789+0000","last_unstale":"2024-03-31T23:11:28.514876+0000","last_undegraded":"2024-03-31T23:11:28.514876+0000","last_fullsized":"2024-03-31T23:11:28.514876+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_clean_scrub_stamp":"2024-03-31T23:09:09.564263+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T10:52:35.478325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":120,"num_read_kb":120,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"93'10","reported_seq":41,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:10.934738+0000","last_change":"2024-03-31T23:09:11.171074+0000","last_active":"2024-03-31T23:11:10.934738+0000","last_peered":"2024-03-31T23:11:10.934738+0000","last_clean":"2024-03-31T23:11:10.934738+0000","last_became_active":"2024-03-31T23:09:11.169572+0000","last_became_peered":"2024-03-31T23:09:11.169572+0000","last_unstale":"2024-03-31T23:11:10.934738+0000","last_undegraded":"2024-03-31T23:11:10.934738+0000","last_fullsized":"2024-03-31T23:11:10.934738+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T07:15:34.833388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:11.116316+0000","last_change":"2024-03-31T23:09:15.203835+0000","last_active":"2024-03-31T23:11:11.116316+0000","last_peered":"2024-03-31T23:11:11.116316+0000","last_clean":"2024-03-31T23:11:11.116316+0000","last_became_active":"2024-03-31T23:09:15.200861+0000","last_became_peered":"2024-03-31T23:09:15.200861+0000","last_unstale":"2024-03-31T23:11:11.116316+0000","last_undegraded":"2024-03-31T23:11:11.116316+0000","last_fullsized":"2024-03-31T23:11:11.116316+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_clean_scrub_stamp":"2024-03-31T23:09:14.171251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:56:19.664708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"91'2","reported_seq":35,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:10.545911+0000","last_change":"2024-03-31T23:09:10.592601+0000","last_active":"2024-03-31T23:11:10.545911+0000","last_peered":"2024-03-31T23:11:10.545911+0000","last_clean":"2024-03-31T23:11:10.545911+0000","last_became_active":"2024-03-31T23:09:10.592345+0000","last_became_peered":"2024-03-31T23:09:10.592345+0000","last_unstale":"2024-03-31T23:11:10.545911+0000","last_undegraded":"2024-03-31T23:11:10.545911+0000","last_fullsized":"2024-03-31T23:11:10.545911+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_clean_scrub_stamp":"2024-03-31T23:09:09.564263+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:10:21.195544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"93'9","reported_seq":40,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:11.122701+0000","last_change":"2024-03-31T23:09:11.175305+0000","last_active":"2024-03-31T23:11:11.122701+0000","last_peered":"2024-03-31T23:11:11.122701+0000","last_clean":"2024-03-31T23:11:11.122701+0000","last_became_active":"2024-03-31T23:09:11.174970+0000","last_became_peered":"2024-03-31T23:09:11.174970+0000","last_unstale":"2024-03-31T23:11:11.122701+0000","last_undegraded":"2024-03-31T23:11:11.122701+0000","last_fullsized":"2024-03-31T23:11:11.122701+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:29:39.718127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:10.544982+0000","last_change":"2024-03-31T23:09:15.200365+0000","last_active":"2024-03-31T23:11:10.544982+0000","last_peered":"2024-03-31T23:11:10.544982+0000","last_clean":"2024-03-31T23:11:10.544982+0000","last_became_active":"2024-03-31T23:09:15.199830+0000","last_became_peered":"2024-03-31T23:09:15.199830+0000","last_unstale":"2024-03-31T23:11:10.544982+0000","last_undegraded":"2024-03-31T23:11:10.544982+0000","last_fullsized":"2024-03-31T23:11:10.544982+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_clean_scrub_stamp":"2024-03-31T23:09:14.171251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:37:30.048859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"91'4","reported_seq":38,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:11.270163+0000","last_change":"2024-03-31T23:09:10.582654+0000","last_active":"2024-03-31T23:11:11.270163+0000","last_peered":"2024-03-31T23:11:11.270163+0000","last_clean":"2024-03-31T23:11:11.270163+0000","last_became_active":"2024-03-31T23:09:10.582314+0000","last_became_peered":"2024-03-31T23:09:10.582314+0000","last_unstale":"2024-03-31T23:11:11.270163+0000","last_undegraded":"2024-03-31T23:11:11.270163+0000","last_fullsized":"2024-03-31T23:11:11.270163+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_clean_scrub_stamp":"2024-03-31T23:09:09.564263+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:51:34.236575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"93'11","reported_seq":42,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:10.934540+0000","last_change":"2024-03-31T23:09:11.172811+0000","last_active":"2024-03-31T23:11:10.934540+0000","last_peered":"2024-03-31T23:11:10.934540+0000","last_clean":"2024-03-31T23:11:10.934540+0000","last_became_active":"2024-03-31T23:09:11.172478+0000","last_became_peered":"2024-03-31T23:09:11.172478+0000","last_unstale":"2024-03-31T23:11:10.934540+0000","last_undegraded":"2024-03-31T23:11:10.934540+0000","last_fullsized":"2024-03-31T23:11:10.934540+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T10:00:09.825254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:11.116557+0000","last_change":"2024-03-31T23:09:15.205494+0000","last_active":"2024-03-31T23:11:11.116557+0000","last_peered":"2024-03-31T23:11:11.116557+0000","last_clean":"2024-03-31T23:11:11.116557+0000","last_became_active":"2024-03-31T23:09:15.205101+0000","last_became_peered":"2024-03-31T23:09:15.205101+0000","last_unstale":"2024-03-31T23:11:11.116557+0000","last_undegraded":"2024-03-31T23:11:11.116557+0000","last_fullsized":"2024-03-31T23:11:11.116557+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_clean_scrub_stamp":"2024-03-31T23:09:14.171251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:52:47.472296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"95'40","reported_seq":324,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:29.516503+0000","last_change":"2024-03-31T23:09:10.585611+0000","last_active":"2024-03-31T23:11:29.516503+0000","last_peered":"2024-03-31T23:11:29.516503+0000","last_clean":"2024-03-31T23:11:29.516503+0000","last_became_active":"2024-03-31T23:09:10.585287+0000","last_became_peered":"2024-03-31T23:09:10.585287+0000","last_unstale":"2024-03-31T23:11:29.516503+0000","last_undegraded":"2024-03-31T23:11:29.516503+0000","last_fullsized":"2024-03-31T23:11:29.516503+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_clean_scrub_stamp":"2024-03-31T23:09:09.564263+0000","objects_scrubbed":0,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:47:44.063831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":67389,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":250,"num_read_kb":191,"num_write":41,"num_write_kb":88,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"93'6","reported_seq":37,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:10.542061+0000","last_change":"2024-03-31T23:09:11.175812+0000","last_active":"2024-03-31T23:11:10.542061+0000","last_peered":"2024-03-31T23:11:10.542061+0000","last_clean":"2024-03-31T23:11:10.542061+0000","last_became_active":"2024-03-31T23:09:11.174972+0000","last_became_peered":"2024-03-31T23:09:11.174972+0000","last_unstale":"2024-03-31T23:11:10.542061+0000","last_undegraded":"2024-03-31T23:11:10.542061+0000","last_fullsized":"2024-03-31T23:11:10.542061+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:17:18.870340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:11.270348+0000","last_change":"2024-03-31T23:09:15.205018+0000","last_active":"2024-03-31T23:11:11.270348+0000","last_peered":"2024-03-31T23:11:11.270348+0000","last_clean":"2024-03-31T23:11:11.270348+0000","last_became_active":"2024-03-31T23:09:15.204510+0000","last_became_peered":"2024-03-31T23:09:15.204510+0000","last_unstale":"2024-03-31T23:11:11.270348+0000","last_undegraded":"2024-03-31T23:11:11.270348+0000","last_fullsized":"2024-03-31T23:11:11.270348+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_clean_scrub_stamp":"2024-03-31T23:09:14.171251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T07:48:59.819937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"91'6","reported_seq":40,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:10.544535+0000","last_change":"2024-03-31T23:09:10.590233+0000","last_active":"2024-03-31T23:11:10.544535+0000","last_peered":"2024-03-31T23:11:10.544535+0000","last_clean":"2024-03-31T23:11:10.544535+0000","last_became_active":"2024-03-31T23:09:10.589853+0000","last_became_peered":"2024-03-31T23:09:10.589853+0000","last_unstale":"2024-03-31T23:11:10.544535+0000","last_undegraded":"2024-03-31T23:11:10.544535+0000","last_fullsized":"2024-03-31T23:11:10.544535+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_clean_scrub_stamp":"2024-03-31T23:09:09.564263+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:10:54.980394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"93'9","reported_seq":40,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:10.716289+0000","last_change":"2024-03-31T23:09:11.177871+0000","last_active":"2024-03-31T23:11:10.716289+0000","last_peered":"2024-03-31T23:11:10.716289+0000","last_clean":"2024-03-31T23:11:10.716289+0000","last_became_active":"2024-03-31T23:09:11.176504+0000","last_became_peered":"2024-03-31T23:09:11.176504+0000","last_unstale":"2024-03-31T23:11:10.716289+0000","last_undegraded":"2024-03-31T23:11:10.716289+0000","last_fullsized":"2024-03-31T23:11:10.716289+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T06:09:08.797695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:10.542551+0000","last_change":"2024-03-31T23:09:15.207336+0000","last_active":"2024-03-31T23:11:10.542551+0000","last_peered":"2024-03-31T23:11:10.542551+0000","last_clean":"2024-03-31T23:11:10.542551+0000","last_became_active":"2024-03-31T23:09:15.206893+0000","last_became_peered":"2024-03-31T23:09:15.206893+0000","last_unstale":"2024-03-31T23:11:10.542551+0000","last_undegraded":"2024-03-31T23:11:10.542551+0000","last_fullsized":"2024-03-31T23:11:10.542551+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_clean_scrub_stamp":"2024-03-31T23:09:14.171251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:24:59.766421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"91'3","reported_seq":36,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:11.268980+0000","last_change":"2024-03-31T23:09:10.587099+0000","last_active":"2024-03-31T23:11:11.268980+0000","last_peered":"2024-03-31T23:11:11.268980+0000","last_clean":"2024-03-31T23:11:11.268980+0000","last_became_active":"2024-03-31T23:09:10.586429+0000","last_became_peered":"2024-03-31T23:09:10.586429+0000","last_unstale":"2024-03-31T23:11:11.268980+0000","last_undegraded":"2024-03-31T23:11:11.268980+0000","last_fullsized":"2024-03-31T23:11:11.268980+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_clean_scrub_stamp":"2024-03-31T23:09:09.564263+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:55:09.307378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"93'15","reported_seq":46,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:10.546149+0000","last_change":"2024-03-31T23:09:11.177661+0000","last_active":"2024-03-31T23:11:10.546149+0000","last_peered":"2024-03-31T23:11:10.546149+0000","last_clean":"2024-03-31T23:11:10.546149+0000","last_became_active":"2024-03-31T23:09:11.177314+0000","last_became_peered":"2024-03-31T23:09:11.177314+0000","last_unstale":"2024-03-31T23:11:10.546149+0000","last_undegraded":"2024-03-31T23:11:10.546149+0000","last_fullsized":"2024-03-31T23:11:10.546149+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T09:26:50.556837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":16,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:10.541733+0000","last_change":"2024-03-31T23:10:10.543526+0000","last_active":"2024-03-31T23:11:10.541733+0000","last_peered":"2024-03-31T23:11:10.541733+0000","last_clean":"2024-03-31T23:11:10.541733+0000","last_became_active":"2024-03-31T23:10:10.543105+0000","last_became_peered":"2024-03-31T23:10:10.543105+0000","last_unstale":"2024-03-31T23:11:10.541733+0000","last_undegraded":"2024-03-31T23:11:10.541733+0000","last_fullsized":"2024-03-31T23:11:10.541733+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_clean_scrub_stamp":"2024-03-31T23:09:14.171251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T09:16:44.529786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,7,0,3],"acting":[8,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.11","version":"95'6","reported_seq":161,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:27.546382+0000","last_change":"2024-03-31T23:09:10.586570+0000","last_active":"2024-03-31T23:11:27.546382+0000","last_peered":"2024-03-31T23:11:27.546382+0000","last_clean":"2024-03-31T23:11:27.546382+0000","last_became_active":"2024-03-31T23:09:10.586239+0000","last_became_peered":"2024-03-31T23:09:10.586239+0000","last_unstale":"2024-03-31T23:11:27.546382+0000","last_undegraded":"2024-03-31T23:11:27.546382+0000","last_fullsized":"2024-03-31T23:11:27.546382+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_clean_scrub_stamp":"2024-03-31T23:09:09.564263+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:23:37.182954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1613,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":123,"num_read_kb":6,"num_write":5,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"93'6","reported_seq":37,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:10.545691+0000","last_change":"2024-03-31T23:09:11.173294+0000","last_active":"2024-03-31T23:11:10.545691+0000","last_peered":"2024-03-31T23:11:10.545691+0000","last_clean":"2024-03-31T23:11:10.545691+0000","last_became_active":"2024-03-31T23:09:11.172926+0000","last_became_peered":"2024-03-31T23:09:11.172926+0000","last_unstale":"2024-03-31T23:11:10.545691+0000","last_undegraded":"2024-03-31T23:11:10.545691+0000","last_fullsized":"2024-03-31T23:11:10.545691+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T06:21:46.603381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:10.716781+0000","last_change":"2024-03-31T23:09:15.199893+0000","last_active":"2024-03-31T23:11:10.716781+0000","last_peered":"2024-03-31T23:11:10.716781+0000","last_clean":"2024-03-31T23:11:10.716781+0000","last_became_active":"2024-03-31T23:09:15.199517+0000","last_became_peered":"2024-03-31T23:09:15.199517+0000","last_unstale":"2024-03-31T23:11:10.716781+0000","last_undegraded":"2024-03-31T23:11:10.716781+0000","last_fullsized":"2024-03-31T23:11:10.716781+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_clean_scrub_stamp":"2024-03-31T23:09:14.171251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T07:48:35.261108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"91'15","reported_seq":49,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:10.541154+0000","last_change":"2024-03-31T23:09:10.588213+0000","last_active":"2024-03-31T23:11:10.541154+0000","last_peered":"2024-03-31T23:11:10.541154+0000","last_clean":"2024-03-31T23:11:10.541154+0000","last_became_active":"2024-03-31T23:09:10.587579+0000","last_became_peered":"2024-03-31T23:09:10.587579+0000","last_unstale":"2024-03-31T23:11:10.541154+0000","last_undegraded":"2024-03-31T23:11:10.541154+0000","last_fullsized":"2024-03-31T23:11:10.541154+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_clean_scrub_stamp":"2024-03-31T23:09:09.564263+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:00:17.883435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"93'6","reported_seq":37,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:11.268980+0000","last_change":"2024-03-31T23:09:11.182427+0000","last_active":"2024-03-31T23:11:11.268980+0000","last_peered":"2024-03-31T23:11:11.268980+0000","last_clean":"2024-03-31T23:11:11.268980+0000","last_became_active":"2024-03-31T23:09:11.182120+0000","last_became_peered":"2024-03-31T23:09:11.182120+0000","last_unstale":"2024-03-31T23:11:11.268980+0000","last_undegraded":"2024-03-31T23:11:11.268980+0000","last_fullsized":"2024-03-31T23:11:11.268980+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:17:48.728424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":26,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:11.269948+0000","last_change":"2024-03-31T23:09:15.207831+0000","last_active":"2024-03-31T23:11:11.269948+0000","last_peered":"2024-03-31T23:11:11.269948+0000","last_clean":"2024-03-31T23:11:11.269948+0000","last_became_active":"2024-03-31T23:09:15.207432+0000","last_became_peered":"2024-03-31T23:09:15.207432+0000","last_unstale":"2024-03-31T23:11:11.269948+0000","last_undegraded":"2024-03-31T23:11:11.269948+0000","last_fullsized":"2024-03-31T23:11:11.269948+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_clean_scrub_stamp":"2024-03-31T23:09:14.171251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:05:27.072054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"91'2","reported_seq":37,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:10.541954+0000","last_change":"2024-03-31T23:09:10.591483+0000","last_active":"2024-03-31T23:11:10.541954+0000","last_peered":"2024-03-31T23:11:10.541954+0000","last_clean":"2024-03-31T23:11:10.541954+0000","last_became_active":"2024-03-31T23:09:10.591173+0000","last_became_peered":"2024-03-31T23:09:10.591173+0000","last_unstale":"2024-03-31T23:11:10.541954+0000","last_undegraded":"2024-03-31T23:11:10.541954+0000","last_fullsized":"2024-03-31T23:11:10.541954+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_clean_scrub_stamp":"2024-03-31T23:09:09.564263+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T10:04:14.275875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"93'5","reported_seq":37,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:10.543194+0000","last_change":"2024-03-31T23:09:11.181171+0000","last_active":"2024-03-31T23:11:10.543194+0000","last_peered":"2024-03-31T23:11:10.543194+0000","last_clean":"2024-03-31T23:11:10.543194+0000","last_became_active":"2024-03-31T23:09:11.180847+0000","last_became_peered":"2024-03-31T23:09:11.180847+0000","last_unstale":"2024-03-31T23:11:10.543194+0000","last_undegraded":"2024-03-31T23:11:10.543194+0000","last_fullsized":"2024-03-31T23:11:10.543194+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:39:29.429009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":26,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:10.541346+0000","last_change":"2024-03-31T23:09:15.746941+0000","last_active":"2024-03-31T23:11:10.541346+0000","last_peered":"2024-03-31T23:11:10.541346+0000","last_clean":"2024-03-31T23:11:10.541346+0000","last_became_active":"2024-03-31T23:09:15.746352+0000","last_became_peered":"2024-03-31T23:09:15.746352+0000","last_unstale":"2024-03-31T23:11:10.541346+0000","last_undegraded":"2024-03-31T23:11:10.541346+0000","last_fullsized":"2024-03-31T23:11:10.541346+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_clean_scrub_stamp":"2024-03-31T23:09:14.171251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T06:21:43.482575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":34,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:11.116214+0000","last_change":"2024-03-31T23:09:10.593029+0000","last_active":"2024-03-31T23:11:11.116214+0000","last_peered":"2024-03-31T23:11:11.116214+0000","last_clean":"2024-03-31T23:11:11.116214+0000","last_became_active":"2024-03-31T23:09:10.592709+0000","last_became_peered":"2024-03-31T23:09:10.592709+0000","last_unstale":"2024-03-31T23:11:11.116214+0000","last_undegraded":"2024-03-31T23:11:11.116214+0000","last_fullsized":"2024-03-31T23:11:11.116214+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_clean_scrub_stamp":"2024-03-31T23:09:09.564263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:46:45.700207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"93'4","reported_seq":36,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:11.125675+0000","last_change":"2024-03-31T23:09:11.177857+0000","last_active":"2024-03-31T23:11:11.125675+0000","last_peered":"2024-03-31T23:11:11.125675+0000","last_clean":"2024-03-31T23:11:11.125675+0000","last_became_active":"2024-03-31T23:09:11.177513+0000","last_became_peered":"2024-03-31T23:09:11.177513+0000","last_unstale":"2024-03-31T23:11:11.125675+0000","last_undegraded":"2024-03-31T23:11:11.125675+0000","last_fullsized":"2024-03-31T23:11:11.125675+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T06:09:23.579208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":26,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:10.716480+0000","last_change":"2024-03-31T23:09:15.203412+0000","last_active":"2024-03-31T23:11:10.716480+0000","last_peered":"2024-03-31T23:11:10.716480+0000","last_clean":"2024-03-31T23:11:10.716480+0000","last_became_active":"2024-03-31T23:09:15.202983+0000","last_became_peered":"2024-03-31T23:09:15.202983+0000","last_unstale":"2024-03-31T23:11:10.716480+0000","last_undegraded":"2024-03-31T23:11:10.716480+0000","last_fullsized":"2024-03-31T23:11:10.716480+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_clean_scrub_stamp":"2024-03-31T23:09:14.171251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:57:46.357378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"91'1","reported_seq":35,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:10.545484+0000","last_change":"2024-03-31T23:09:10.593299+0000","last_active":"2024-03-31T23:11:10.545484+0000","last_peered":"2024-03-31T23:11:10.545484+0000","last_clean":"2024-03-31T23:11:10.545484+0000","last_became_active":"2024-03-31T23:09:10.593007+0000","last_became_peered":"2024-03-31T23:09:10.593007+0000","last_unstale":"2024-03-31T23:11:10.545484+0000","last_undegraded":"2024-03-31T23:11:10.545484+0000","last_fullsized":"2024-03-31T23:11:10.545484+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_clean_scrub_stamp":"2024-03-31T23:09:09.564263+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:15:07.199645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"93'7","reported_seq":39,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:11.125387+0000","last_change":"2024-03-31T23:09:11.176846+0000","last_active":"2024-03-31T23:11:11.125387+0000","last_peered":"2024-03-31T23:11:11.125387+0000","last_clean":"2024-03-31T23:11:11.125387+0000","last_became_active":"2024-03-31T23:09:11.176554+0000","last_became_peered":"2024-03-31T23:09:11.176554+0000","last_unstale":"2024-03-31T23:11:11.125387+0000","last_undegraded":"2024-03-31T23:11:11.125387+0000","last_fullsized":"2024-03-31T23:11:11.125387+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:25:23.639093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":26,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:10.716905+0000","last_change":"2024-03-31T23:09:15.746599+0000","last_active":"2024-03-31T23:11:10.716905+0000","last_peered":"2024-03-31T23:11:10.716905+0000","last_clean":"2024-03-31T23:11:10.716905+0000","last_became_active":"2024-03-31T23:09:15.745879+0000","last_became_peered":"2024-03-31T23:09:15.745879+0000","last_unstale":"2024-03-31T23:11:10.716905+0000","last_undegraded":"2024-03-31T23:11:10.716905+0000","last_fullsized":"2024-03-31T23:11:10.716905+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_clean_scrub_stamp":"2024-03-31T23:09:14.171251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T08:02:25.198610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"93'8","reported_seq":40,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:11.269210+0000","last_change":"2024-03-31T23:09:11.175546+0000","last_active":"2024-03-31T23:11:11.269210+0000","last_peered":"2024-03-31T23:11:11.269210+0000","last_clean":"2024-03-31T23:11:11.269210+0000","last_became_active":"2024-03-31T23:09:11.175253+0000","last_became_peered":"2024-03-31T23:09:11.175253+0000","last_unstale":"2024-03-31T23:11:11.269210+0000","last_undegraded":"2024-03-31T23:11:11.269210+0000","last_fullsized":"2024-03-31T23:11:11.269210+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T10:27:31.753102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"95'6","reported_seq":159,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:28.547474+0000","last_change":"2024-03-31T23:09:10.580888+0000","last_active":"2024-03-31T23:11:28.547474+0000","last_peered":"2024-03-31T23:11:28.547474+0000","last_clean":"2024-03-31T23:11:28.547474+0000","last_became_active":"2024-03-31T23:09:10.580557+0000","last_became_peered":"2024-03-31T23:09:10.580557+0000","last_unstale":"2024-03-31T23:11:28.547474+0000","last_undegraded":"2024-03-31T23:11:28.547474+0000","last_fullsized":"2024-03-31T23:11:28.547474+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_clean_scrub_stamp":"2024-03-31T23:09:09.564263+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:17:38.878570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":119,"num_read_kb":119,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"91'1","reported_seq":27,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:10.545528+0000","last_change":"2024-03-31T23:09:15.205282+0000","last_active":"2024-03-31T23:11:10.545528+0000","last_peered":"2024-03-31T23:11:10.545528+0000","last_clean":"2024-03-31T23:11:10.545528+0000","last_became_active":"2024-03-31T23:09:15.204873+0000","last_became_peered":"2024-03-31T23:09:15.204873+0000","last_unstale":"2024-03-31T23:11:10.545528+0000","last_undegraded":"2024-03-31T23:11:10.545528+0000","last_fullsized":"2024-03-31T23:11:10.545528+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_clean_scrub_stamp":"2024-03-31T23:09:14.171251+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:56:05.697685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"93'8","reported_seq":40,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:10.540856+0000","last_change":"2024-03-31T23:09:11.182155+0000","last_active":"2024-03-31T23:11:10.540856+0000","last_peered":"2024-03-31T23:11:10.540856+0000","last_clean":"2024-03-31T23:11:10.540856+0000","last_became_active":"2024-03-31T23:09:11.181829+0000","last_became_peered":"2024-03-31T23:09:11.181829+0000","last_unstale":"2024-03-31T23:11:10.540856+0000","last_undegraded":"2024-03-31T23:11:10.540856+0000","last_fullsized":"2024-03-31T23:11:10.540856+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:19:22.832440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"91'3","reported_seq":39,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:10.544924+0000","last_change":"2024-03-31T23:09:10.590451+0000","last_active":"2024-03-31T23:11:10.544924+0000","last_peered":"2024-03-31T23:11:10.544924+0000","last_clean":"2024-03-31T23:11:10.544924+0000","last_became_active":"2024-03-31T23:09:10.590123+0000","last_became_peered":"2024-03-31T23:09:10.590123+0000","last_unstale":"2024-03-31T23:11:10.544924+0000","last_undegraded":"2024-03-31T23:11:10.544924+0000","last_fullsized":"2024-03-31T23:11:10.544924+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_clean_scrub_stamp":"2024-03-31T23:09:09.564263+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:27:02.724456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":369,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:10.545255+0000","last_change":"2024-03-31T23:07:57.920942+0000","last_active":"2024-03-31T23:11:10.545255+0000","last_peered":"2024-03-31T23:11:10.545255+0000","last_clean":"2024-03-31T23:11:10.545255+0000","last_became_active":"2024-03-31T23:07:57.920381+0000","last_became_peered":"2024-03-31T23:07:57.920381+0000","last_unstale":"2024-03-31T23:11:10.545255+0000","last_undegraded":"2024-03-31T23:11:10.545255+0000","last_fullsized":"2024-03-31T23:11:10.545255+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:04:33.508180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:04:33.508180+0000","last_clean_scrub_stamp":"2024-03-31T23:04:33.508180+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:20:11.211403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:11.126466+0000","last_change":"2024-03-31T23:11:11.126137+0000","last_active":"2024-03-31T23:11:11.126466+0000","last_peered":"2024-03-31T23:11:11.126466+0000","last_clean":"2024-03-31T23:11:11.126466+0000","last_became_active":"2024-03-31T23:11:11.125719+0000","last_became_peered":"2024-03-31T23:11:11.125719+0000","last_unstale":"2024-03-31T23:11:11.126466+0000","last_undegraded":"2024-03-31T23:11:11.126466+0000","last_fullsized":"2024-03-31T23:11:11.126466+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_clean_scrub_stamp":"2024-03-31T23:09:14.171251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:34:55.444516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,6,10],"acting":[8,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1","version":"93'5","reported_seq":37,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:10.716368+0000","last_change":"2024-03-31T23:09:11.175336+0000","last_active":"2024-03-31T23:11:10.716368+0000","last_peered":"2024-03-31T23:11:10.716368+0000","last_clean":"2024-03-31T23:11:10.716368+0000","last_became_active":"2024-03-31T23:09:11.174768+0000","last_became_peered":"2024-03-31T23:09:11.174768+0000","last_unstale":"2024-03-31T23:11:10.716368+0000","last_undegraded":"2024-03-31T23:11:10.716368+0000","last_fullsized":"2024-03-31T23:11:10.716368+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:09:11.678744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"91'3","reported_seq":37,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:10.545473+0000","last_change":"2024-03-31T23:09:10.591807+0000","last_active":"2024-03-31T23:11:10.545473+0000","last_peered":"2024-03-31T23:11:10.545473+0000","last_clean":"2024-03-31T23:11:10.545473+0000","last_became_active":"2024-03-31T23:09:10.591407+0000","last_became_peered":"2024-03-31T23:09:10.591407+0000","last_unstale":"2024-03-31T23:11:10.545473+0000","last_undegraded":"2024-03-31T23:11:10.545473+0000","last_fullsized":"2024-03-31T23:11:10.545473+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_clean_scrub_stamp":"2024-03-31T23:09:09.564263+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T08:46:33.660717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":26,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:10.543100+0000","last_change":"2024-03-31T23:09:15.203879+0000","last_active":"2024-03-31T23:11:10.543100+0000","last_peered":"2024-03-31T23:11:10.543100+0000","last_clean":"2024-03-31T23:11:10.543100+0000","last_became_active":"2024-03-31T23:09:15.203412+0000","last_became_peered":"2024-03-31T23:09:15.203412+0000","last_unstale":"2024-03-31T23:11:10.543100+0000","last_undegraded":"2024-03-31T23:11:10.543100+0000","last_fullsized":"2024-03-31T23:11:10.543100+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_clean_scrub_stamp":"2024-03-31T23:09:14.171251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:03:54.486038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"91'1","reported_seq":36,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:10.542010+0000","last_change":"2024-03-31T23:09:10.596404+0000","last_active":"2024-03-31T23:11:10.542010+0000","last_peered":"2024-03-31T23:11:10.542010+0000","last_clean":"2024-03-31T23:11:10.542010+0000","last_became_active":"2024-03-31T23:09:10.596031+0000","last_became_peered":"2024-03-31T23:09:10.596031+0000","last_unstale":"2024-03-31T23:11:10.542010+0000","last_undegraded":"2024-03-31T23:11:10.542010+0000","last_fullsized":"2024-03-31T23:11:10.542010+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_clean_scrub_stamp":"2024-03-31T23:09:09.564263+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:30:31.904241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"93'9","reported_seq":41,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:10.716744+0000","last_change":"2024-03-31T23:09:11.164300+0000","last_active":"2024-03-31T23:11:10.716744+0000","last_peered":"2024-03-31T23:11:10.716744+0000","last_clean":"2024-03-31T23:11:10.716744+0000","last_became_active":"2024-03-31T23:09:11.163960+0000","last_became_peered":"2024-03-31T23:09:11.163960+0000","last_unstale":"2024-03-31T23:11:10.716744+0000","last_undegraded":"2024-03-31T23:11:10.716744+0000","last_fullsized":"2024-03-31T23:11:10.716744+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T10:37:30.512216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":26,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:10.540599+0000","last_change":"2024-03-31T23:09:15.746426+0000","last_active":"2024-03-31T23:11:10.540599+0000","last_peered":"2024-03-31T23:11:10.540599+0000","last_clean":"2024-03-31T23:11:10.540599+0000","last_became_active":"2024-03-31T23:09:15.745887+0000","last_became_peered":"2024-03-31T23:09:15.745887+0000","last_unstale":"2024-03-31T23:11:10.540599+0000","last_undegraded":"2024-03-31T23:11:10.540599+0000","last_fullsized":"2024-03-31T23:11:10.540599+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_clean_scrub_stamp":"2024-03-31T23:09:14.171251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:41:52.005300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"95'10","reported_seq":44,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:23.633720+0000","last_change":"2024-03-31T23:09:10.590631+0000","last_active":"2024-03-31T23:11:23.633720+0000","last_peered":"2024-03-31T23:11:23.633720+0000","last_clean":"2024-03-31T23:11:23.633720+0000","last_became_active":"2024-03-31T23:09:10.590211+0000","last_became_peered":"2024-03-31T23:09:10.590211+0000","last_unstale":"2024-03-31T23:11:23.633720+0000","last_undegraded":"2024-03-31T23:11:23.633720+0000","last_fullsized":"2024-03-31T23:11:23.633720+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_clean_scrub_stamp":"2024-03-31T23:09:09.564263+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T08:51:24.978892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"93'5","reported_seq":37,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:10.541392+0000","last_change":"2024-03-31T23:09:11.177353+0000","last_active":"2024-03-31T23:11:10.541392+0000","last_peered":"2024-03-31T23:11:10.541392+0000","last_clean":"2024-03-31T23:11:10.541392+0000","last_became_active":"2024-03-31T23:09:11.176996+0000","last_became_peered":"2024-03-31T23:09:11.176996+0000","last_unstale":"2024-03-31T23:11:10.541392+0000","last_undegraded":"2024-03-31T23:11:10.541392+0000","last_fullsized":"2024-03-31T23:11:10.541392+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:20:52.486281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":26,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:10.544240+0000","last_change":"2024-03-31T23:09:15.198528+0000","last_active":"2024-03-31T23:11:10.544240+0000","last_peered":"2024-03-31T23:11:10.544240+0000","last_clean":"2024-03-31T23:11:10.544240+0000","last_became_active":"2024-03-31T23:09:15.197938+0000","last_became_peered":"2024-03-31T23:09:15.197938+0000","last_unstale":"2024-03-31T23:11:10.544240+0000","last_undegraded":"2024-03-31T23:11:10.544240+0000","last_fullsized":"2024-03-31T23:11:10.544240+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_clean_scrub_stamp":"2024-03-31T23:09:14.171251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:45:46.567268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"91'4","reported_seq":38,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:10.545572+0000","last_change":"2024-03-31T23:09:10.584779+0000","last_active":"2024-03-31T23:11:10.545572+0000","last_peered":"2024-03-31T23:11:10.545572+0000","last_clean":"2024-03-31T23:11:10.545572+0000","last_became_active":"2024-03-31T23:09:10.584400+0000","last_became_peered":"2024-03-31T23:09:10.584400+0000","last_unstale":"2024-03-31T23:11:10.545572+0000","last_undegraded":"2024-03-31T23:11:10.545572+0000","last_fullsized":"2024-03-31T23:11:10.545572+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_clean_scrub_stamp":"2024-03-31T23:09:09.564263+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:10:10.095704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"93'7","reported_seq":45,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:11.122710+0000","last_change":"2024-03-31T23:10:10.543538+0000","last_active":"2024-03-31T23:11:11.122710+0000","last_peered":"2024-03-31T23:11:11.122710+0000","last_clean":"2024-03-31T23:11:11.122710+0000","last_became_active":"2024-03-31T23:10:10.543083+0000","last_became_peered":"2024-03-31T23:10:10.543083+0000","last_unstale":"2024-03-31T23:11:11.122710+0000","last_undegraded":"2024-03-31T23:11:11.122710+0000","last_fullsized":"2024-03-31T23:11:11.122710+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:38:50.656801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:10.545224+0000","last_change":"2024-03-31T23:09:15.746525+0000","last_active":"2024-03-31T23:11:10.545224+0000","last_peered":"2024-03-31T23:11:10.545224+0000","last_clean":"2024-03-31T23:11:10.545224+0000","last_became_active":"2024-03-31T23:09:15.746126+0000","last_became_peered":"2024-03-31T23:09:15.746126+0000","last_unstale":"2024-03-31T23:11:10.545224+0000","last_undegraded":"2024-03-31T23:11:10.545224+0000","last_fullsized":"2024-03-31T23:11:10.545224+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_clean_scrub_stamp":"2024-03-31T23:09:14.171251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:04:57.313615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"91'2","reported_seq":36,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:10.934097+0000","last_change":"2024-03-31T23:09:10.591469+0000","last_active":"2024-03-31T23:11:10.934097+0000","last_peered":"2024-03-31T23:11:10.934097+0000","last_clean":"2024-03-31T23:11:10.934097+0000","last_became_active":"2024-03-31T23:09:10.591166+0000","last_became_peered":"2024-03-31T23:09:10.591166+0000","last_unstale":"2024-03-31T23:11:10.934097+0000","last_undegraded":"2024-03-31T23:11:10.934097+0000","last_fullsized":"2024-03-31T23:11:10.934097+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_clean_scrub_stamp":"2024-03-31T23:09:09.564263+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T10:30:40.279041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"93'2","reported_seq":34,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:10.541228+0000","last_change":"2024-03-31T23:09:11.174471+0000","last_active":"2024-03-31T23:11:10.541228+0000","last_peered":"2024-03-31T23:11:10.541228+0000","last_clean":"2024-03-31T23:11:10.541228+0000","last_became_active":"2024-03-31T23:09:11.174154+0000","last_became_peered":"2024-03-31T23:09:11.174154+0000","last_unstale":"2024-03-31T23:11:10.541228+0000","last_undegraded":"2024-03-31T23:11:10.541228+0000","last_fullsized":"2024-03-31T23:11:10.541228+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T06:10:14.727009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:11.117002+0000","last_change":"2024-03-31T23:09:15.201416+0000","last_active":"2024-03-31T23:11:11.117002+0000","last_peered":"2024-03-31T23:11:11.117002+0000","last_clean":"2024-03-31T23:11:11.117002+0000","last_became_active":"2024-03-31T23:09:15.200987+0000","last_became_peered":"2024-03-31T23:09:15.200987+0000","last_unstale":"2024-03-31T23:11:11.117002+0000","last_undegraded":"2024-03-31T23:11:11.117002+0000","last_fullsized":"2024-03-31T23:11:11.117002+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_clean_scrub_stamp":"2024-03-31T23:09:14.171251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T10:56:54.872988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"91'3","reported_seq":37,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:10.540834+0000","last_change":"2024-03-31T23:09:10.594035+0000","last_active":"2024-03-31T23:11:10.540834+0000","last_peered":"2024-03-31T23:11:10.540834+0000","last_clean":"2024-03-31T23:11:10.540834+0000","last_became_active":"2024-03-31T23:09:10.593710+0000","last_became_peered":"2024-03-31T23:09:10.593710+0000","last_unstale":"2024-03-31T23:11:10.540834+0000","last_undegraded":"2024-03-31T23:11:10.540834+0000","last_fullsized":"2024-03-31T23:11:10.540834+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_clean_scrub_stamp":"2024-03-31T23:09:09.564263+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:46:48.580244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"93'9","reported_seq":41,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:10.542532+0000","last_change":"2024-03-31T23:09:11.170501+0000","last_active":"2024-03-31T23:11:10.542532+0000","last_peered":"2024-03-31T23:11:10.542532+0000","last_clean":"2024-03-31T23:11:10.542532+0000","last_became_active":"2024-03-31T23:09:11.170068+0000","last_became_peered":"2024-03-31T23:09:11.170068+0000","last_unstale":"2024-03-31T23:11:10.542532+0000","last_undegraded":"2024-03-31T23:11:10.542532+0000","last_fullsized":"2024-03-31T23:11:10.542532+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T06:05:53.488957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"91'1","reported_seq":26,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:10.544928+0000","last_change":"2024-03-31T23:09:15.208092+0000","last_active":"2024-03-31T23:11:10.544928+0000","last_peered":"2024-03-31T23:11:10.544928+0000","last_clean":"2024-03-31T23:11:10.544928+0000","last_became_active":"2024-03-31T23:09:15.207750+0000","last_became_peered":"2024-03-31T23:09:15.207750+0000","last_unstale":"2024-03-31T23:11:10.544928+0000","last_undegraded":"2024-03-31T23:11:10.544928+0000","last_fullsized":"2024-03-31T23:11:10.544928+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_clean_scrub_stamp":"2024-03-31T23:09:14.171251+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:24:39.649569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"95'2","reported_seq":16,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:10.548189+0000","last_change":"2024-03-31T23:11:10.547781+0000","last_active":"2024-03-31T23:11:10.548189+0000","last_peered":"2024-03-31T23:11:10.548189+0000","last_clean":"2024-03-31T23:11:10.548189+0000","last_became_active":"2024-03-31T23:11:10.545156+0000","last_became_peered":"2024-03-31T23:11:10.545156+0000","last_unstale":"2024-03-31T23:11:10.548189+0000","last_undegraded":"2024-03-31T23:11:10.548189+0000","last_fullsized":"2024-03-31T23:11:10.548189+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_clean_scrub_stamp":"2024-03-31T23:09:09.564263+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T08:14:11.270910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,8],"acting":[11,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.b","version":"93'8","reported_seq":40,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:10.541041+0000","last_change":"2024-03-31T23:09:11.175169+0000","last_active":"2024-03-31T23:11:10.541041+0000","last_peered":"2024-03-31T23:11:10.541041+0000","last_clean":"2024-03-31T23:11:10.541041+0000","last_became_active":"2024-03-31T23:09:11.174849+0000","last_became_peered":"2024-03-31T23:09:11.174849+0000","last_unstale":"2024-03-31T23:11:10.541041+0000","last_undegraded":"2024-03-31T23:11:10.541041+0000","last_fullsized":"2024-03-31T23:11:10.541041+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:58:06.814034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:10.545200+0000","last_change":"2024-03-31T23:09:15.746423+0000","last_active":"2024-03-31T23:11:10.545200+0000","last_peered":"2024-03-31T23:11:10.545200+0000","last_clean":"2024-03-31T23:11:10.545200+0000","last_became_active":"2024-03-31T23:09:15.745942+0000","last_became_peered":"2024-03-31T23:09:15.745942+0000","last_unstale":"2024-03-31T23:11:10.545200+0000","last_undegraded":"2024-03-31T23:11:10.545200+0000","last_fullsized":"2024-03-31T23:11:10.545200+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_clean_scrub_stamp":"2024-03-31T23:09:14.171251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:56:02.903212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"91'4","reported_seq":17,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:10.550839+0000","last_change":"2024-03-31T23:11:10.550839+0000","last_active":"2024-03-31T23:11:10.550839+0000","last_peered":"2024-03-31T23:11:10.550839+0000","last_clean":"2024-03-31T23:11:10.550839+0000","last_became_active":"2024-03-31T23:11:10.546702+0000","last_became_peered":"2024-03-31T23:11:10.546702+0000","last_unstale":"2024-03-31T23:11:10.550839+0000","last_undegraded":"2024-03-31T23:11:10.550839+0000","last_fullsized":"2024-03-31T23:11:10.550839+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_clean_scrub_stamp":"2024-03-31T23:09:09.564263+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:39:13.293806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,7],"acting":[11,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.a","version":"93'5","reported_seq":37,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:10.545517+0000","last_change":"2024-03-31T23:09:11.170440+0000","last_active":"2024-03-31T23:11:10.545517+0000","last_peered":"2024-03-31T23:11:10.545517+0000","last_clean":"2024-03-31T23:11:10.545517+0000","last_became_active":"2024-03-31T23:09:11.170081+0000","last_became_peered":"2024-03-31T23:09:11.170081+0000","last_unstale":"2024-03-31T23:11:10.545517+0000","last_undegraded":"2024-03-31T23:11:10.545517+0000","last_fullsized":"2024-03-31T23:11:10.545517+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:53:31.429822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":26,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:10.934387+0000","last_change":"2024-03-31T23:09:15.207146+0000","last_active":"2024-03-31T23:11:10.934387+0000","last_peered":"2024-03-31T23:11:10.934387+0000","last_clean":"2024-03-31T23:11:10.934387+0000","last_became_active":"2024-03-31T23:09:15.206769+0000","last_became_peered":"2024-03-31T23:09:15.206769+0000","last_unstale":"2024-03-31T23:11:10.934387+0000","last_undegraded":"2024-03-31T23:11:10.934387+0000","last_fullsized":"2024-03-31T23:11:10.934387+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_clean_scrub_stamp":"2024-03-31T23:09:14.171251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T07:42:12.961109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"91'3","reported_seq":38,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:10.544591+0000","last_change":"2024-03-31T23:09:10.587291+0000","last_active":"2024-03-31T23:11:10.544591+0000","last_peered":"2024-03-31T23:11:10.544591+0000","last_clean":"2024-03-31T23:11:10.544591+0000","last_became_active":"2024-03-31T23:09:10.585992+0000","last_became_peered":"2024-03-31T23:09:10.585992+0000","last_unstale":"2024-03-31T23:11:10.544591+0000","last_undegraded":"2024-03-31T23:11:10.544591+0000","last_fullsized":"2024-03-31T23:11:10.544591+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_clean_scrub_stamp":"2024-03-31T23:09:09.564263+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:28:00.786268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"93'7","reported_seq":39,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:10.545804+0000","last_change":"2024-03-31T23:09:11.179369+0000","last_active":"2024-03-31T23:11:10.545804+0000","last_peered":"2024-03-31T23:11:10.545804+0000","last_clean":"2024-03-31T23:11:10.545804+0000","last_became_active":"2024-03-31T23:09:11.178776+0000","last_became_peered":"2024-03-31T23:09:11.178776+0000","last_unstale":"2024-03-31T23:11:10.545804+0000","last_undegraded":"2024-03-31T23:11:10.545804+0000","last_fullsized":"2024-03-31T23:11:10.545804+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T07:56:05.201338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:11.122108+0000","last_change":"2024-03-31T23:09:15.201462+0000","last_active":"2024-03-31T23:11:11.122108+0000","last_peered":"2024-03-31T23:11:11.122108+0000","last_clean":"2024-03-31T23:11:11.122108+0000","last_became_active":"2024-03-31T23:09:15.201067+0000","last_became_peered":"2024-03-31T23:09:15.201067+0000","last_unstale":"2024-03-31T23:11:11.122108+0000","last_undegraded":"2024-03-31T23:11:11.122108+0000","last_fullsized":"2024-03-31T23:11:11.122108+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_clean_scrub_stamp":"2024-03-31T23:09:14.171251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:54:29.339456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"85'1","reported_seq":36,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:11.269181+0000","last_change":"2024-03-31T23:09:10.584752+0000","last_active":"2024-03-31T23:11:11.269181+0000","last_peered":"2024-03-31T23:11:11.269181+0000","last_clean":"2024-03-31T23:11:11.269181+0000","last_became_active":"2024-03-31T23:09:10.584432+0000","last_became_peered":"2024-03-31T23:09:10.584432+0000","last_unstale":"2024-03-31T23:11:11.269181+0000","last_undegraded":"2024-03-31T23:11:11.269181+0000","last_fullsized":"2024-03-31T23:11:11.269181+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_clean_scrub_stamp":"2024-03-31T23:09:09.564263+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T10:50:53.889719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"93'6","reported_seq":44,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:10.546403+0000","last_change":"2024-03-31T23:10:10.543993+0000","last_active":"2024-03-31T23:11:10.546403+0000","last_peered":"2024-03-31T23:11:10.546403+0000","last_clean":"2024-03-31T23:11:10.546403+0000","last_became_active":"2024-03-31T23:10:10.543627+0000","last_became_peered":"2024-03-31T23:10:10.543627+0000","last_unstale":"2024-03-31T23:11:10.546403+0000","last_undegraded":"2024-03-31T23:11:10.546403+0000","last_fullsized":"2024-03-31T23:11:10.546403+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T08:46:41.221122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":26,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:11.269712+0000","last_change":"2024-03-31T23:09:15.197271+0000","last_active":"2024-03-31T23:11:11.269712+0000","last_peered":"2024-03-31T23:11:11.269712+0000","last_clean":"2024-03-31T23:11:11.269712+0000","last_became_active":"2024-03-31T23:09:15.196339+0000","last_became_peered":"2024-03-31T23:09:15.196339+0000","last_unstale":"2024-03-31T23:11:11.269712+0000","last_undegraded":"2024-03-31T23:11:11.269712+0000","last_fullsized":"2024-03-31T23:11:11.269712+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:14.171251+0000","last_clean_scrub_stamp":"2024-03-31T23:09:14.171251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:44:13.044278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"85'2","reported_seq":36,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:10.716161+0000","last_change":"2024-03-31T23:09:10.595131+0000","last_active":"2024-03-31T23:11:10.716161+0000","last_peered":"2024-03-31T23:11:10.716161+0000","last_clean":"2024-03-31T23:11:10.716161+0000","last_became_active":"2024-03-31T23:09:10.594740+0000","last_became_peered":"2024-03-31T23:09:10.594740+0000","last_unstale":"2024-03-31T23:11:10.716161+0000","last_undegraded":"2024-03-31T23:11:10.716161+0000","last_fullsized":"2024-03-31T23:11:10.716161+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:09.564263+0000","last_clean_scrub_stamp":"2024-03-31T23:09:09.564263+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:52:41.849621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"93'9","reported_seq":41,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:10.545854+0000","last_change":"2024-03-31T23:09:11.177530+0000","last_active":"2024-03-31T23:11:10.545854+0000","last_peered":"2024-03-31T23:11:10.545854+0000","last_clean":"2024-03-31T23:11:10.545854+0000","last_became_active":"2024-03-31T23:09:11.176972+0000","last_became_peered":"2024-03-31T23:09:11.176972+0000","last_unstale":"2024-03-31T23:11:10.545854+0000","last_undegraded":"2024-03-31T23:11:10.545854+0000","last_fullsized":"2024-03-31T23:11:10.545854+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T07:57:21.094911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"93'9","reported_seq":41,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:10.542629+0000","last_change":"2024-03-31T23:09:11.172792+0000","last_active":"2024-03-31T23:11:10.542629+0000","last_peered":"2024-03-31T23:11:10.542629+0000","last_clean":"2024-03-31T23:11:10.542629+0000","last_became_active":"2024-03-31T23:09:11.172463+0000","last_became_peered":"2024-03-31T23:09:11.172463+0000","last_unstale":"2024-03-31T23:11:10.542629+0000","last_undegraded":"2024-03-31T23:11:10.542629+0000","last_fullsized":"2024-03-31T23:11:10.542629+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:07:28.789783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"93'12","reported_seq":44,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:10.544865+0000","last_change":"2024-03-31T23:09:11.174331+0000","last_active":"2024-03-31T23:11:10.544865+0000","last_peered":"2024-03-31T23:11:10.544865+0000","last_clean":"2024-03-31T23:11:10.544865+0000","last_became_active":"2024-03-31T23:09:11.174001+0000","last_became_peered":"2024-03-31T23:09:11.174001+0000","last_unstale":"2024-03-31T23:11:10.544865+0000","last_undegraded":"2024-03-31T23:11:10.544865+0000","last_fullsized":"2024-03-31T23:11:10.544865+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:26:28.005406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"93'8","reported_seq":40,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:10.544540+0000","last_change":"2024-03-31T23:09:11.180047+0000","last_active":"2024-03-31T23:11:10.544540+0000","last_peered":"2024-03-31T23:11:10.544540+0000","last_clean":"2024-03-31T23:11:10.544540+0000","last_became_active":"2024-03-31T23:09:11.179691+0000","last_became_peered":"2024-03-31T23:09:11.179691+0000","last_unstale":"2024-03-31T23:11:10.544540+0000","last_undegraded":"2024-03-31T23:11:10.544540+0000","last_fullsized":"2024-03-31T23:11:10.544540+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:28:05.083710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"93'7","reported_seq":39,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:11.122493+0000","last_change":"2024-03-31T23:09:11.177642+0000","last_active":"2024-03-31T23:11:11.122493+0000","last_peered":"2024-03-31T23:11:11.122493+0000","last_clean":"2024-03-31T23:11:11.122493+0000","last_became_active":"2024-03-31T23:09:11.177261+0000","last_became_peered":"2024-03-31T23:09:11.177261+0000","last_unstale":"2024-03-31T23:11:11.122493+0000","last_undegraded":"2024-03-31T23:11:11.122493+0000","last_fullsized":"2024-03-31T23:11:11.122493+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T07:10:43.616863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"93'4","reported_seq":36,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:10.716630+0000","last_change":"2024-03-31T23:09:11.173606+0000","last_active":"2024-03-31T23:11:10.716630+0000","last_peered":"2024-03-31T23:11:10.716630+0000","last_clean":"2024-03-31T23:11:10.716630+0000","last_became_active":"2024-03-31T23:09:11.173304+0000","last_became_peered":"2024-03-31T23:09:11.173304+0000","last_unstale":"2024-03-31T23:11:10.716630+0000","last_undegraded":"2024-03-31T23:11:10.716630+0000","last_fullsized":"2024-03-31T23:11:10.716630+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:53:06.112181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"93'6","reported_seq":38,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:10.543432+0000","last_change":"2024-03-31T23:09:11.176628+0000","last_active":"2024-03-31T23:11:10.543432+0000","last_peered":"2024-03-31T23:11:10.543432+0000","last_clean":"2024-03-31T23:11:10.543432+0000","last_became_active":"2024-03-31T23:09:11.176318+0000","last_became_peered":"2024-03-31T23:09:11.176318+0000","last_unstale":"2024-03-31T23:11:10.543432+0000","last_undegraded":"2024-03-31T23:11:10.543432+0000","last_fullsized":"2024-03-31T23:11:10.543432+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:06:40.723675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"93'15","reported_seq":47,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:10.544863+0000","last_change":"2024-03-31T23:09:11.169318+0000","last_active":"2024-03-31T23:11:10.544863+0000","last_peered":"2024-03-31T23:11:10.544863+0000","last_clean":"2024-03-31T23:11:10.544863+0000","last_became_active":"2024-03-31T23:09:11.169026+0000","last_became_peered":"2024-03-31T23:09:11.169026+0000","last_unstale":"2024-03-31T23:11:10.544863+0000","last_undegraded":"2024-03-31T23:11:10.544863+0000","last_fullsized":"2024-03-31T23:11:10.544863+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:02:18.562854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"93'2","reported_seq":34,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:10.541448+0000","last_change":"2024-03-31T23:09:11.184670+0000","last_active":"2024-03-31T23:11:10.541448+0000","last_peered":"2024-03-31T23:11:10.541448+0000","last_clean":"2024-03-31T23:11:10.541448+0000","last_became_active":"2024-03-31T23:09:11.184354+0000","last_became_peered":"2024-03-31T23:09:11.184354+0000","last_unstale":"2024-03-31T23:11:10.541448+0000","last_undegraded":"2024-03-31T23:11:10.541448+0000","last_fullsized":"2024-03-31T23:11:10.541448+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:16:14.354947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"93'6","reported_seq":38,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:11.269351+0000","last_change":"2024-03-31T23:09:11.180991+0000","last_active":"2024-03-31T23:11:11.269351+0000","last_peered":"2024-03-31T23:11:11.269351+0000","last_clean":"2024-03-31T23:11:11.269351+0000","last_became_active":"2024-03-31T23:09:11.180667+0000","last_became_peered":"2024-03-31T23:09:11.180667+0000","last_unstale":"2024-03-31T23:11:11.269351+0000","last_undegraded":"2024-03-31T23:11:11.269351+0000","last_fullsized":"2024-03-31T23:11:11.269351+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T06:35:33.157176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"93'8","reported_seq":40,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:10.540877+0000","last_change":"2024-03-31T23:09:11.182025+0000","last_active":"2024-03-31T23:11:10.540877+0000","last_peered":"2024-03-31T23:11:10.540877+0000","last_clean":"2024-03-31T23:11:10.540877+0000","last_became_active":"2024-03-31T23:09:11.179310+0000","last_became_peered":"2024-03-31T23:09:11.179310+0000","last_unstale":"2024-03-31T23:11:10.540877+0000","last_undegraded":"2024-03-31T23:11:10.540877+0000","last_fullsized":"2024-03-31T23:11:10.540877+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T10:21:40.784738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"93'10","reported_seq":48,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:10.934477+0000","last_change":"2024-03-31T23:10:10.485623+0000","last_active":"2024-03-31T23:11:10.934477+0000","last_peered":"2024-03-31T23:11:10.934477+0000","last_clean":"2024-03-31T23:11:10.934477+0000","last_became_active":"2024-03-31T23:10:10.485248+0000","last_became_peered":"2024-03-31T23:10:10.485248+0000","last_unstale":"2024-03-31T23:11:10.934477+0000","last_undegraded":"2024-03-31T23:11:10.934477+0000","last_fullsized":"2024-03-31T23:11:10.934477+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:59:04.668598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,6],"acting":[1,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"93'11","reported_seq":43,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:10.716676+0000","last_change":"2024-03-31T23:09:11.172585+0000","last_active":"2024-03-31T23:11:10.716676+0000","last_peered":"2024-03-31T23:11:10.716676+0000","last_clean":"2024-03-31T23:11:10.716676+0000","last_became_active":"2024-03-31T23:09:11.172266+0000","last_became_peered":"2024-03-31T23:09:11.172266+0000","last_unstale":"2024-03-31T23:11:10.716676+0000","last_undegraded":"2024-03-31T23:11:10.716676+0000","last_fullsized":"2024-03-31T23:11:10.716676+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:38:26.570175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"93'8","reported_seq":40,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-31T23:11:10.542801+0000","last_change":"2024-03-31T23:09:11.181434+0000","last_active":"2024-03-31T23:11:10.542801+0000","last_peered":"2024-03-31T23:11:10.542801+0000","last_clean":"2024-03-31T23:11:10.542801+0000","last_became_active":"2024-03-31T23:09:11.181140+0000","last_became_peered":"2024-03-31T23:09:11.181140+0000","last_unstale":"2024-03-31T23:11:10.542801+0000","last_undegraded":"2024-03-31T23:11:10.542801+0000","last_fullsized":"2024-03-31T23:11:10.542801+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T23:09:10.570670+0000","last_clean_scrub_stamp":"2024-03-31T23:09:10.570670+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:55:06.379696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4157440,"data_stored":4132576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":72556,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":744,"num_read_kb":451,"num_write":182,"num_write_kb":137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":438272,"data_stored":212861,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":158,"ondisk_log_size":158,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":81,"seq":347892351018,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36404,"kb_used_data":1060,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708748,"statfs":{"total":95995035648,"available":95957757952,"internally_reserved":0,"allocated":1085440,"data_stored":533688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":36174276},"hb_peers":[0,1,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":73,"seq":313532612655,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28756,"kb_used_data":1740,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716396,"statfs":{"total":95995035648,"available":95965589504,"internally_reserved":0,"allocated":1781760,"data_stored":1189370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":68,"seq":292057776180,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28096,"kb_used_data":1080,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717056,"statfs":{"total":95995035648,"available":95966265344,"internally_reserved":0,"allocated":1105920,"data_stored":538020,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":60,"seq":257698037816,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28092,"kb_used_data":1076,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717060,"statfs":{"total":95995035648,"available":95966269440,"internally_reserved":0,"allocated":1101824,"data_stored":533844,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299453,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28676,"kb_used_data":1656,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716476,"statfs":{"total":95995035648,"available":95965671424,"internally_reserved":0,"allocated":1695744,"data_stored":1130426,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1588,"internal_metadata":27654604},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863462979,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28080,"kb_used_data":1056,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717072,"statfs":{"total":95995035648,"available":95966281728,"internally_reserved":0,"allocated":1081344,"data_stored":529100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574938,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28148,"kb_used_data":1124,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717004,"statfs":{"total":95995035648,"available":95966212096,"internally_reserved":0,"allocated":1150976,"data_stored":591592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738464,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36424,"kb_used_data":1080,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708728,"statfs":{"total":95995035648,"available":95957737472,"internally_reserved":0,"allocated":1105920,"data_stored":530488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411415,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28168,"kb_used_data":1152,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716984,"statfs":{"total":95995035648,"available":95966191616,"internally_reserved":0,"allocated":1179648,"data_stored":598774,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182482,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28084,"kb_used_data":1068,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717068,"statfs":{"total":95995035648,"available":95966277632,"internally_reserved":0,"allocated":1093632,"data_stored":529294,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920845,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36408,"kb_used_data":1068,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708744,"statfs":{"total":95995035648,"available":95957753856,"internally_reserved":0,"allocated":1093632,"data_stored":534775,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724616,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":37004,"kb_used_data":1664,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708148,"statfs":{"total":95995035648,"available":95957143552,"internally_reserved":0,"allocated":1703936,"data_stored":1125482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":62582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":69764,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1669,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":110592,"data_stored":69992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-31T23:11:32.108 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-31T23:11:32.248 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:32.246+0000 7fa6c2c88640 1 -- 172.21.15.46:0/2104202206 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa6bc100670 msgr2=0x7fa6bc100a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:32.248 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:32.246+0000 7fa6c2c88640 1 --2- 172.21.15.46:0/2104202206 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa6bc100670 0x7fa6bc100a90 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fa6b4009be0 tx=0x7fa6b402bef0 comp rx=0 tx=0).stop 2024-03-31T23:11:32.248 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:32.246+0000 7fa6c2c88640 1 -- 172.21.15.46:0/2104202206 shutdown_connections 2024-03-31T23:11:32.249 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:32.246+0000 7fa6c2c88640 1 --2- 172.21.15.46:0/2104202206 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa6bc102220 0x7fa6bc108ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:32.249 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:32.246+0000 7fa6c2c88640 1 --2- 172.21.15.46:0/2104202206 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa6bc101860 0x7fa6bc101ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:32.249 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:32.246+0000 7fa6c2c88640 1 --2- 172.21.15.46:0/2104202206 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa6bc100670 0x7fa6bc100a90 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:32.249 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:32.246+0000 7fa6c2c88640 1 -- 172.21.15.46:0/2104202206 >> 172.21.15.46:0/2104202206 conn(0x7fa6bc0fbbe0 msgr2=0x7fa6bc0fe020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:11:32.249 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:32.246+0000 7fa6c2c88640 1 -- 172.21.15.46:0/2104202206 shutdown_connections 2024-03-31T23:11:32.249 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:32.246+0000 7fa6c2c88640 1 -- 172.21.15.46:0/2104202206 wait complete. 2024-03-31T23:11:32.249 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:32.246+0000 7fa6c2c88640 1 Processor -- start 2024-03-31T23:11:32.249 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:32.246+0000 7fa6c2c88640 1 -- start start 2024-03-31T23:11:32.249 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:32.246+0000 7fa6c2c88640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa6bc100670 0x7fa6bc104600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:32.249 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:32.246+0000 7fa6c2c88640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa6bc101860 0x7fa6bc102c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:32.249 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:32.246+0000 7fa6c2c88640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa6bc102220 0x7fa6bc103180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:32.249 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:32.246+0000 7fa6c2c88640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fa6bc10c1a0 con 0x7fa6bc100670 2024-03-31T23:11:32.249 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:32.246+0000 7fa6c2c88640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fa6bc10c020 con 0x7fa6bc101860 2024-03-31T23:11:32.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:32.246+0000 7fa6c2c88640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fa6bc10c320 con 0x7fa6bc102220 2024-03-31T23:11:32.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:32.246+0000 7fa6c0a24640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa6bc100670 0x7fa6bc104600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:32.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:32.247+0000 7fa6c0a24640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa6bc100670 0x7fa6bc104600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:48130/0 (socket says 172.21.15.46:48130) 2024-03-31T23:11:32.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:32.247+0000 7fa6c0a24640 1 -- 172.21.15.46:0/108681534 learned_addr learned my addr 172.21.15.46:0/108681534 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:11:32.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:32.247+0000 7fa6bbfff640 1 --2- 172.21.15.46:0/108681534 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa6bc101860 0x7fa6bc102c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:32.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:32.247+0000 7fa6c1225640 1 --2- 172.21.15.46:0/108681534 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa6bc102220 0x7fa6bc103180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:32.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:32.247+0000 7fa6c0a24640 1 -- 172.21.15.46:0/108681534 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa6bc102220 msgr2=0x7fa6bc103180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:32.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:32.247+0000 7fa6c0a24640 1 --2- 172.21.15.46:0/108681534 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa6bc102220 0x7fa6bc103180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:32.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:32.247+0000 7fa6c0a24640 1 -- 172.21.15.46:0/108681534 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa6bc101860 msgr2=0x7fa6bc102c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:32.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:32.247+0000 7fa6c0a24640 1 --2- 172.21.15.46:0/108681534 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa6bc101860 0x7fa6bc102c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:32.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:32.247+0000 7fa6c0a24640 1 -- 172.21.15.46:0/108681534 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa6bc103a10 con 0x7fa6bc100670 2024-03-31T23:11:32.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:32.247+0000 7fa6bbfff640 1 --2- 172.21.15.46:0/108681534 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa6bc101860 0x7fa6bc102c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:11:32.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:32.247+0000 7fa6c0a24640 1 --2- 172.21.15.46:0/108681534 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa6bc100670 0x7fa6bc104600 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7fa6b4000f80 tx=0x7fa6b4033a60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:11:32.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:32.247+0000 7fa6b9ffb640 1 -- 172.21.15.46:0/108681534 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6b4043020 con 0x7fa6bc100670 2024-03-31T23:11:32.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:32.247+0000 7fa6c2c88640 1 -- 172.21.15.46:0/108681534 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa6bc1a1af0 con 0x7fa6bc100670 2024-03-31T23:11:32.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:32.247+0000 7fa6b9ffb640 1 -- 172.21.15.46:0/108681534 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa6b4033e60 con 0x7fa6bc100670 2024-03-31T23:11:32.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:32.247+0000 7fa6b9ffb640 1 -- 172.21.15.46:0/108681534 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6b4034430 con 0x7fa6bc100670 2024-03-31T23:11:32.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:32.247+0000 7fa6c2c88640 1 -- 172.21.15.46:0/108681534 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa6bc1a2040 con 0x7fa6bc100670 2024-03-31T23:11:32.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:32.248+0000 7fa6c2c88640 1 -- 172.21.15.46:0/108681534 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa6bc1a2210 con 0x7fa6bc100670 2024-03-31T23:11:32.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:32.249+0000 7fa6b9ffb640 1 -- 172.21.15.46:0/108681534 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 18) ==== 137320+0+0 (secure 0 0 0) 0x7fa6b4046020 con 0x7fa6bc100670 2024-03-31T23:11:32.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:32.250+0000 7fa6b9ffb640 1 --2- 172.21.15.46:0/108681534 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fa6a40a24a0 0x7fa6a40a4960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:32.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:32.250+0000 7fa6bbfff640 1 --2- 172.21.15.46:0/108681534 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fa6a40a24a0 0x7fa6a40a4960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:32.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:32.250+0000 7fa6b9ffb640 1 -- 172.21.15.46:0/108681534 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9208+0+0 (secure 0 0 0) 0x7fa6b40f1180 con 0x7fa6bc100670 2024-03-31T23:11:32.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:32.250+0000 7fa6bbfff640 1 --2- 172.21.15.46:0/108681534 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fa6a40a24a0 0x7fa6a40a4960 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fa6b00096c0 tx=0x7fa6b0009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:11:32.254 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:32.254+0000 7fa6b9ffb640 1 -- 172.21.15.46:0/108681534 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fa6b404d020 con 0x7fa6bc100670 2024-03-31T23:11:32.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:32.438+0000 7fa6c2c88640 1 -- 172.21.15.46:0/108681534 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fa6bc1a23e0 con 0x7fa6bc100670 2024-03-31T23:11:32.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:32.439+0000 7fa6b9ffb640 1 -- 172.21.15.46:0/108681534 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v95) ==== 74+0+24819 (secure 0 0 0) 0x7fa6b40bdf70 con 0x7fa6bc100670 2024-03-31T23:11:32.439 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-31T23:11:32.440 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":95,"fsid":"6a9d8f08-efb2-11ee-b647-cb9ed24678a4","created":"2024-03-31T23:01:15.463711+0000","modified":"2024-03-31T23:11:10.531495+0000","last_up_change":"2024-03-31T23:08:20.199888+0000","last_in_change":"2024-03-31T23:07:59.438035+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-31T23:04:32.823751+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-31T23:09:09.145711+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-31T23:09:10.148579+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-31T23:09:13.419519+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"91","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"47429095-3f46-4ed7-b7c4-504464719815","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":1824772695},{"type":"v1","addr":"172.21.15.46:6803","nonce":1824772695}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":1824772695},{"type":"v1","addr":"172.21.15.46:6805","nonce":1824772695}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":1824772695},{"type":"v1","addr":"172.21.15.46:6809","nonce":1824772695}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":1824772695},{"type":"v1","addr":"172.21.15.46:6807","nonce":1824772695}]},"public_addr":"172.21.15.46:6803/1824772695","cluster_addr":"172.21.15.46:6805/1824772695","heartbeat_back_addr":"172.21.15.46:6809/1824772695","heartbeat_front_addr":"172.21.15.46:6807/1824772695","state":["exists","up"]},{"osd":1,"uuid":"be56f883-411e-4dbe-b50e-a158134df4a6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6800","nonce":973693922},{"type":"v1","addr":"172.21.15.142:6801","nonce":973693922}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6802","nonce":973693922},{"type":"v1","addr":"172.21.15.142:6803","nonce":973693922}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6806","nonce":973693922},{"type":"v1","addr":"172.21.15.142:6807","nonce":973693922}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6804","nonce":973693922},{"type":"v1","addr":"172.21.15.142:6805","nonce":973693922}]},"public_addr":"172.21.15.142:6801/973693922","cluster_addr":"172.21.15.142:6803/973693922","heartbeat_back_addr":"172.21.15.142:6807/973693922","heartbeat_front_addr":"172.21.15.142:6805/973693922","state":["exists","up"]},{"osd":2,"uuid":"c99a8794-ab19-4260-9a8d-4bc0bd2c8175","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6800","nonce":4084261813},{"type":"v1","addr":"172.21.15.162:6801","nonce":4084261813}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6802","nonce":4084261813},{"type":"v1","addr":"172.21.15.162:6803","nonce":4084261813}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6806","nonce":4084261813},{"type":"v1","addr":"172.21.15.162:6807","nonce":4084261813}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6804","nonce":4084261813},{"type":"v1","addr":"172.21.15.162:6805","nonce":4084261813}]},"public_addr":"172.21.15.162:6801/4084261813","cluster_addr":"172.21.15.162:6803/4084261813","heartbeat_back_addr":"172.21.15.162:6807/4084261813","heartbeat_front_addr":"172.21.15.162:6805/4084261813","state":["exists","up"]},{"osd":3,"uuid":"f877814e-9ef6-4872-a081-5609bb9ebc06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":3496317970},{"type":"v1","addr":"172.21.15.46:6811","nonce":3496317970}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":3496317970},{"type":"v1","addr":"172.21.15.46:6813","nonce":3496317970}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":3496317970},{"type":"v1","addr":"172.21.15.46:6817","nonce":3496317970}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":3496317970},{"type":"v1","addr":"172.21.15.46:6815","nonce":3496317970}]},"public_addr":"172.21.15.46:6811/3496317970","cluster_addr":"172.21.15.46:6813/3496317970","heartbeat_back_addr":"172.21.15.46:6817/3496317970","heartbeat_front_addr":"172.21.15.46:6815/3496317970","state":["exists","up"]},{"osd":4,"uuid":"beecec0a-8280-44fe-8f5f-8498a35eca8f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6808","nonce":2996266929},{"type":"v1","addr":"172.21.15.142:6809","nonce":2996266929}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6810","nonce":2996266929},{"type":"v1","addr":"172.21.15.142:6811","nonce":2996266929}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6814","nonce":2996266929},{"type":"v1","addr":"172.21.15.142:6815","nonce":2996266929}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6812","nonce":2996266929},{"type":"v1","addr":"172.21.15.142:6813","nonce":2996266929}]},"public_addr":"172.21.15.142:6809/2996266929","cluster_addr":"172.21.15.142:6811/2996266929","heartbeat_back_addr":"172.21.15.142:6815/2996266929","heartbeat_front_addr":"172.21.15.142:6813/2996266929","state":["exists","up"]},{"osd":5,"uuid":"9da7ff75-f658-45ea-a02f-1943fdda76b9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6808","nonce":914208293},{"type":"v1","addr":"172.21.15.162:6809","nonce":914208293}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6810","nonce":914208293},{"type":"v1","addr":"172.21.15.162:6811","nonce":914208293}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6814","nonce":914208293},{"type":"v1","addr":"172.21.15.162:6815","nonce":914208293}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6812","nonce":914208293},{"type":"v1","addr":"172.21.15.162:6813","nonce":914208293}]},"public_addr":"172.21.15.162:6809/914208293","cluster_addr":"172.21.15.162:6811/914208293","heartbeat_back_addr":"172.21.15.162:6815/914208293","heartbeat_front_addr":"172.21.15.162:6813/914208293","state":["exists","up"]},{"osd":6,"uuid":"8fa11224-cb26-4d3f-87d1-6a8f31de2a19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":56291966},{"type":"v1","addr":"172.21.15.46:6819","nonce":56291966}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":56291966},{"type":"v1","addr":"172.21.15.46:6821","nonce":56291966}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6824","nonce":56291966},{"type":"v1","addr":"172.21.15.46:6825","nonce":56291966}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":56291966},{"type":"v1","addr":"172.21.15.46:6823","nonce":56291966}]},"public_addr":"172.21.15.46:6819/56291966","cluster_addr":"172.21.15.46:6821/56291966","heartbeat_back_addr":"172.21.15.46:6825/56291966","heartbeat_front_addr":"172.21.15.46:6823/56291966","state":["exists","up"]},{"osd":7,"uuid":"0d0338b7-4d0a-4bb0-b152-8f5224871d9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6816","nonce":2146598446},{"type":"v1","addr":"172.21.15.142:6817","nonce":2146598446}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6818","nonce":2146598446},{"type":"v1","addr":"172.21.15.142:6819","nonce":2146598446}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6822","nonce":2146598446},{"type":"v1","addr":"172.21.15.142:6823","nonce":2146598446}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6820","nonce":2146598446},{"type":"v1","addr":"172.21.15.142:6821","nonce":2146598446}]},"public_addr":"172.21.15.142:6817/2146598446","cluster_addr":"172.21.15.142:6819/2146598446","heartbeat_back_addr":"172.21.15.142:6823/2146598446","heartbeat_front_addr":"172.21.15.142:6821/2146598446","state":["exists","up"]},{"osd":8,"uuid":"8b6f6037-892a-416c-babb-7552648d0b3a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6816","nonce":1173078318},{"type":"v1","addr":"172.21.15.162:6817","nonce":1173078318}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6818","nonce":1173078318},{"type":"v1","addr":"172.21.15.162:6819","nonce":1173078318}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6822","nonce":1173078318},{"type":"v1","addr":"172.21.15.162:6823","nonce":1173078318}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6820","nonce":1173078318},{"type":"v1","addr":"172.21.15.162:6821","nonce":1173078318}]},"public_addr":"172.21.15.162:6817/1173078318","cluster_addr":"172.21.15.162:6819/1173078318","heartbeat_back_addr":"172.21.15.162:6823/1173078318","heartbeat_front_addr":"172.21.15.162:6821/1173078318","state":["exists","up"]},{"osd":9,"uuid":"faccce25-6bf2-4220-8858-cda82b72be68","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":532708881},{"type":"v1","addr":"172.21.15.46:6827","nonce":532708881}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6828","nonce":532708881},{"type":"v1","addr":"172.21.15.46:6829","nonce":532708881}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":532708881},{"type":"v1","addr":"172.21.15.46:6833","nonce":532708881}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":532708881},{"type":"v1","addr":"172.21.15.46:6831","nonce":532708881}]},"public_addr":"172.21.15.46:6827/532708881","cluster_addr":"172.21.15.46:6829/532708881","heartbeat_back_addr":"172.21.15.46:6833/532708881","heartbeat_front_addr":"172.21.15.46:6831/532708881","state":["exists","up"]},{"osd":10,"uuid":"cdd356db-f9eb-4080-95ab-db5d8223e572","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6824","nonce":2845222366},{"type":"v1","addr":"172.21.15.142:6825","nonce":2845222366}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6826","nonce":2845222366},{"type":"v1","addr":"172.21.15.142:6827","nonce":2845222366}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6830","nonce":2845222366},{"type":"v1","addr":"172.21.15.142:6831","nonce":2845222366}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6828","nonce":2845222366},{"type":"v1","addr":"172.21.15.142:6829","nonce":2845222366}]},"public_addr":"172.21.15.142:6825/2845222366","cluster_addr":"172.21.15.142:6827/2845222366","heartbeat_back_addr":"172.21.15.142:6831/2845222366","heartbeat_front_addr":"172.21.15.142:6829/2845222366","state":["exists","up"]},{"osd":11,"uuid":"72104d1c-3ea2-45ad-bc0b-0500f146c06b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6824","nonce":1986323778},{"type":"v1","addr":"172.21.15.162:6825","nonce":1986323778}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6826","nonce":1986323778},{"type":"v1","addr":"172.21.15.162:6827","nonce":1986323778}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6830","nonce":1986323778},{"type":"v1","addr":"172.21.15.162:6831","nonce":1986323778}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6828","nonce":1986323778},{"type":"v1","addr":"172.21.15.162:6829","nonce":1986323778}]},"public_addr":"172.21.15.162:6825/1986323778","cluster_addr":"172.21.15.162:6827/1986323778","heartbeat_back_addr":"172.21.15.162:6831/1986323778","heartbeat_front_addr":"172.21.15.162:6829/1986323778","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:03:44.996331+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:04:07.994738+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:04:30.942255+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:04:55.588884+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:05:19.900600+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:05:43.807442+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:06:09.026976+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:06:34.586784+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:06:59.751605+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:07:26.144495+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:07:52.326909+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:08:17.423042+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.b","mappings":[{"from":8,"to":11}]},{"pgid":"3.6","mappings":[{"from":4,"to":7}]},{"pgid":"3.c","mappings":[{"from":3,"to":9}]},{"pgid":"3.1a","mappings":[{"from":5,"to":6}]},{"pgid":"3.1b","mappings":[{"from":4,"to":8}]},{"pgid":"3.1d","mappings":[{"from":4,"to":8}]},{"pgid":"3.2a","mappings":[{"from":3,"to":6}]},{"pgid":"3.33","mappings":[{"from":5,"to":9}]},{"pgid":"3.36","mappings":[{"from":4,"to":9}]},{"pgid":"4.6","mappings":[{"from":5,"to":8}]},{"pgid":"4.17","mappings":[{"from":9,"to":8}]},{"pgid":"4.18","mappings":[{"from":9,"to":8}]},{"pgid":"4.1b","mappings":[{"from":9,"to":10}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.46:0/4021039857":"2024-04-01T23:02:08.493619+0000","172.21.15.46:6801/3681182647":"2024-04-01T23:02:08.493619+0000","172.21.15.46:0/4022214806":"2024-04-01T23:02:08.493619+0000","172.21.15.46:6800/1515528215":"2024-04-01T23:01:41.910162+0000","172.21.15.46:6800/3681182647":"2024-04-01T23:02:08.493619+0000","172.21.15.46:0/143885707":"2024-04-01T23:01:41.910162+0000","172.21.15.46:6801/1515528215":"2024-04-01T23:01:41.910162+0000","172.21.15.46:0/1680389778":"2024-04-01T23:01:41.910162+0000","172.21.15.46:0/1782133966":"2024-04-01T23:01:41.910162+0000","172.21.15.46:0/3925209376":"2024-04-01T23:02:08.493619+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-31T23:11:32.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:32.441+0000 7fa6c2c88640 1 -- 172.21.15.46:0/108681534 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fa6a40a24a0 msgr2=0x7fa6a40a4960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:32.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:32.441+0000 7fa6c2c88640 1 --2- 172.21.15.46:0/108681534 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fa6a40a24a0 0x7fa6a40a4960 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fa6b00096c0 tx=0x7fa6b0009260 comp rx=0 tx=0).stop 2024-03-31T23:11:32.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:32.441+0000 7fa6c2c88640 1 -- 172.21.15.46:0/108681534 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa6bc100670 msgr2=0x7fa6bc104600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:32.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:32.441+0000 7fa6c2c88640 1 --2- 172.21.15.46:0/108681534 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa6bc100670 0x7fa6bc104600 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7fa6b4000f80 tx=0x7fa6b4033a60 comp rx=0 tx=0).stop 2024-03-31T23:11:32.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:32.441+0000 7fa6c2c88640 1 -- 172.21.15.46:0/108681534 shutdown_connections 2024-03-31T23:11:32.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:32.441+0000 7fa6c2c88640 1 --2- 172.21.15.46:0/108681534 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fa6a40a24a0 0x7fa6a40a4960 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:32.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:32.441+0000 7fa6c2c88640 1 --2- 172.21.15.46:0/108681534 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa6bc102220 0x7fa6bc103180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:32.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:32.441+0000 7fa6c2c88640 1 --2- 172.21.15.46:0/108681534 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa6bc101860 0x7fa6bc102c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:32.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:32.441+0000 7fa6c2c88640 1 --2- 172.21.15.46:0/108681534 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa6bc100670 0x7fa6bc104600 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:32.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:32.441+0000 7fa6c2c88640 1 -- 172.21.15.46:0/108681534 >> 172.21.15.46:0/108681534 conn(0x7fa6bc0fbbe0 msgr2=0x7fa6bc0688e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:11:32.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:32.441+0000 7fa6c2c88640 1 -- 172.21.15.46:0/108681534 shutdown_connections 2024-03-31T23:11:32.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:32.441+0000 7fa6c2c88640 1 -- 172.21.15.46:0/108681534 wait complete. 2024-03-31T23:11:32.453 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-31T23:04:32.823751+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-31T23:09:09.145711+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-31T23:09:10.148579+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-31T23:09:13.419519+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '91', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-31T23:11:32.453 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-31T23:11:32.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:32.560+0000 7ff65ceeb640 1 -- 172.21.15.46:0/1714982954 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff658101780 msgr2=0x7ff658101ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:32.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:32.560+0000 7ff65ceeb640 1 --2- 172.21.15.46:0/1714982954 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff658101780 0x7ff658101ba0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7ff648009900 tx=0x7ff64802bc30 comp rx=0 tx=0).stop 2024-03-31T23:11:32.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:32.560+0000 7ff65ceeb640 1 -- 172.21.15.46:0/1714982954 shutdown_connections 2024-03-31T23:11:32.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:32.560+0000 7ff65ceeb640 1 --2- 172.21.15.46:0/1714982954 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff658103330 0x7ff658109bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:32.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:32.560+0000 7ff65ceeb640 1 --2- 172.21.15.46:0/1714982954 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff658102970 0x7ff658102df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:32.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:32.560+0000 7ff65ceeb640 1 --2- 172.21.15.46:0/1714982954 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff658101780 0x7ff658101ba0 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:32.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:32.560+0000 7ff65ceeb640 1 -- 172.21.15.46:0/1714982954 >> 172.21.15.46:0/1714982954 conn(0x7ff6580fcd10 msgr2=0x7ff6580ff130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:11:32.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:32.560+0000 7ff65ceeb640 1 -- 172.21.15.46:0/1714982954 shutdown_connections 2024-03-31T23:11:32.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:32.560+0000 7ff65ceeb640 1 -- 172.21.15.46:0/1714982954 wait complete. 2024-03-31T23:11:32.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:32.560+0000 7ff65ceeb640 1 Processor -- start 2024-03-31T23:11:32.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:32.560+0000 7ff65ceeb640 1 -- start start 2024-03-31T23:11:32.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:32.561+0000 7ff65ceeb640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff658101780 0x7ff658197960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:32.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:32.561+0000 7ff65ceeb640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff658102970 0x7ff658197ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:32.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:32.561+0000 7ff65ceeb640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff658103330 0x7ff65819ef10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:32.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:32.561+0000 7ff65ceeb640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7ff65810d280 con 0x7ff658101780 2024-03-31T23:11:32.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:32.561+0000 7ff65ceeb640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7ff65810d100 con 0x7ff658102970 2024-03-31T23:11:32.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:32.561+0000 7ff65ceeb640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7ff65810d400 con 0x7ff658103330 2024-03-31T23:11:32.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:32.561+0000 7ff65659c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff658101780 0x7ff658197960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:32.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:32.561+0000 7ff65659c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff658101780 0x7ff658197960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:48146/0 (socket says 172.21.15.46:48146) 2024-03-31T23:11:32.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:32.561+0000 7ff65659c640 1 -- 172.21.15.46:0/2061813286 learned_addr learned my addr 172.21.15.46:0/2061813286 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:11:32.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:32.561+0000 7ff655d9b640 1 --2- 172.21.15.46:0/2061813286 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff658102970 0x7ff658197ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:32.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:32.561+0000 7ff656d9d640 1 --2- 172.21.15.46:0/2061813286 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff658103330 0x7ff65819ef10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:32.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:32.561+0000 7ff65659c640 1 -- 172.21.15.46:0/2061813286 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff658103330 msgr2=0x7ff65819ef10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:32.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:32.561+0000 7ff65659c640 1 --2- 172.21.15.46:0/2061813286 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff658103330 0x7ff65819ef10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:32.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:32.561+0000 7ff65659c640 1 -- 172.21.15.46:0/2061813286 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff658102970 msgr2=0x7ff658197ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:32.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:32.561+0000 7ff65659c640 1 --2- 172.21.15.46:0/2061813286 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff658102970 0x7ff658197ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:32.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:32.561+0000 7ff65659c640 1 -- 172.21.15.46:0/2061813286 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff65819f580 con 0x7ff658101780 2024-03-31T23:11:32.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:32.561+0000 7ff65659c640 1 --2- 172.21.15.46:0/2061813286 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff658101780 0x7ff658197960 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7ff648000c00 tx=0x7ff648005dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:11:32.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:32.561+0000 7ff6477fe640 1 -- 172.21.15.46:0/2061813286 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff648043020 con 0x7ff658101780 2024-03-31T23:11:32.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:32.561+0000 7ff6477fe640 1 -- 172.21.15.46:0/2061813286 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff648004350 con 0x7ff658101780 2024-03-31T23:11:32.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:32.561+0000 7ff6477fe640 1 -- 172.21.15.46:0/2061813286 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff648003b80 con 0x7ff658101780 2024-03-31T23:11:32.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:32.561+0000 7ff65ceeb640 1 -- 172.21.15.46:0/2061813286 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff65819f810 con 0x7ff658101780 2024-03-31T23:11:32.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:32.561+0000 7ff65ceeb640 1 -- 172.21.15.46:0/2061813286 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff65819fd60 con 0x7ff658101780 2024-03-31T23:11:32.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:32.563+0000 7ff65ceeb640 1 -- 172.21.15.46:0/2061813286 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff658108ac0 con 0x7ff658101780 2024-03-31T23:11:32.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:32.564+0000 7ff6477fe640 1 -- 172.21.15.46:0/2061813286 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 18) ==== 137320+0+0 (secure 0 0 0) 0x7ff648046020 con 0x7ff658101780 2024-03-31T23:11:32.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:32.564+0000 7ff6477fe640 1 --2- 172.21.15.46:0/2061813286 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7ff6400a24a0 0x7ff6400a4960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:32.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:32.564+0000 7ff655d9b640 1 --2- 172.21.15.46:0/2061813286 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7ff6400a24a0 0x7ff6400a4960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:32.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:32.564+0000 7ff6477fe640 1 -- 172.21.15.46:0/2061813286 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9208+0+0 (secure 0 0 0) 0x7ff6480f0c00 con 0x7ff658101780 2024-03-31T23:11:32.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:32.565+0000 7ff655d9b640 1 --2- 172.21.15.46:0/2061813286 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7ff6400a24a0 0x7ff6400a4960 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7ff64c0095f0 tx=0x7ff64c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:11:32.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:32.569+0000 7ff6477fe640 1 -- 172.21.15.46:0/2061813286 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7ff6480f7080 con 0x7ff658101780 2024-03-31T23:11:32.753 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:32.753+0000 7ff65ceeb640 1 -- 172.21.15.46:0/2061813286 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7ff658062350 con 0x7ff658101780 2024-03-31T23:11:32.753 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:32.753+0000 7ff6477fe640 1 -- 172.21.15.46:0/2061813286 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v95) ==== 93+0+10 (secure 0 0 0) 0x7ff64804d020 con 0x7ff658101780 2024-03-31T23:11:32.754 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 1 2024-03-31T23:11:32.756 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:32.756+0000 7ff65ceeb640 1 -- 172.21.15.46:0/2061813286 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7ff6400a24a0 msgr2=0x7ff6400a4960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:32.756 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:32.756+0000 7ff65ceeb640 1 --2- 172.21.15.46:0/2061813286 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7ff6400a24a0 0x7ff6400a4960 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7ff64c0095f0 tx=0x7ff64c005f90 comp rx=0 tx=0).stop 2024-03-31T23:11:32.756 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:32.756+0000 7ff65ceeb640 1 -- 172.21.15.46:0/2061813286 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff658101780 msgr2=0x7ff658197960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:32.756 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:32.756+0000 7ff65ceeb640 1 --2- 172.21.15.46:0/2061813286 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff658101780 0x7ff658197960 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7ff648000c00 tx=0x7ff648005dc0 comp rx=0 tx=0).stop 2024-03-31T23:11:32.756 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:32.756+0000 7ff65ceeb640 1 -- 172.21.15.46:0/2061813286 shutdown_connections 2024-03-31T23:11:32.756 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:32.756+0000 7ff65ceeb640 1 --2- 172.21.15.46:0/2061813286 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7ff6400a24a0 0x7ff6400a4960 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:32.756 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:32.756+0000 7ff65ceeb640 1 --2- 172.21.15.46:0/2061813286 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff658103330 0x7ff65819ef10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:32.756 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:32.756+0000 7ff65ceeb640 1 --2- 172.21.15.46:0/2061813286 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff658102970 0x7ff658197ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:32.757 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:32.756+0000 7ff65ceeb640 1 --2- 172.21.15.46:0/2061813286 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff658101780 0x7ff658197960 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:32.757 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:32.756+0000 7ff65ceeb640 1 -- 172.21.15.46:0/2061813286 >> 172.21.15.46:0/2061813286 conn(0x7ff6580fcd10 msgr2=0x7ff658105c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:11:32.757 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:32.756+0000 7ff65ceeb640 1 -- 172.21.15.46:0/2061813286 shutdown_connections 2024-03-31T23:11:32.757 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:32.756+0000 7ff65ceeb640 1 -- 172.21.15.46:0/2061813286 wait complete. 2024-03-31T23:11:32.766 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:11:32 smithi046 ceph-mon[32498]: pgmap v366: 129 pgs: 129 active+clean; 648 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 1023 B/s wr, 7 op/s 2024-03-31T23:11:32.766 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:11:32 smithi046 ceph-mon[32498]: from='client.15540 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-31T23:11:32.766 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:11:32 smithi046 ceph-mon[32498]: from='client.? 172.21.15.46:0/108681534' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-31T23:11:32.767 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-31T23:11:32.875 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:32.873+0000 7fc9dbffd640 1 -- 172.21.15.46:0/2306191792 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc9d4102220 msgr2=0x7fc9d4108ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:32.875 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:32.873+0000 7fc9dbffd640 1 --2- 172.21.15.46:0/2306191792 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc9d4102220 0x7fc9d4108ab0 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7fc9cc009ca0 tx=0x7fc9cc02c0f0 comp rx=0 tx=0).stop 2024-03-31T23:11:32.875 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:32.873+0000 7fc9dbffd640 1 -- 172.21.15.46:0/2306191792 shutdown_connections 2024-03-31T23:11:32.875 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:32.873+0000 7fc9dbffd640 1 --2- 172.21.15.46:0/2306191792 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc9d4102220 0x7fc9d4108ab0 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:32.875 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:32.873+0000 7fc9dbffd640 1 --2- 172.21.15.46:0/2306191792 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc9d4101860 0x7fc9d4101ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:32.875 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:32.873+0000 7fc9dbffd640 1 --2- 172.21.15.46:0/2306191792 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc9d4100670 0x7fc9d4100a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:32.875 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:32.873+0000 7fc9dbffd640 1 -- 172.21.15.46:0/2306191792 >> 172.21.15.46:0/2306191792 conn(0x7fc9d40fbc00 msgr2=0x7fc9d40fe020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:11:32.875 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:32.873+0000 7fc9dbffd640 1 -- 172.21.15.46:0/2306191792 shutdown_connections 2024-03-31T23:11:32.876 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:32.873+0000 7fc9dbffd640 1 -- 172.21.15.46:0/2306191792 wait complete. 2024-03-31T23:11:32.876 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:32.873+0000 7fc9dbffd640 1 Processor -- start 2024-03-31T23:11:32.876 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:32.873+0000 7fc9dbffd640 1 -- start start 2024-03-31T23:11:32.876 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:32.874+0000 7fc9dbffd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc9d4100670 0x7fc9d4196850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:32.876 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:32.874+0000 7fc9dbffd640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc9d4101860 0x7fc9d4196d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:32.876 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:32.874+0000 7fc9dbffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc9d4102220 0x7fc9d419de00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:32.876 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:32.874+0000 7fc9dbffd640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fc9d410c1b0 con 0x7fc9d4102220 2024-03-31T23:11:32.876 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:32.874+0000 7fc9dbffd640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fc9d410c030 con 0x7fc9d4101860 2024-03-31T23:11:32.876 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:32.874+0000 7fc9dbffd640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fc9d410c330 con 0x7fc9d4100670 2024-03-31T23:11:32.876 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:32.874+0000 7fc9d9d99640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc9d4100670 0x7fc9d4196850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:32.876 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:32.874+0000 7fc9da59a640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc9d4102220 0x7fc9d419de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:32.876 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:32.874+0000 7fc9d9d99640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc9d4100670 0x7fc9d4196850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.46:58984/0 (socket says 172.21.15.46:58984) 2024-03-31T23:11:32.876 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:32.874+0000 7fc9d9d99640 1 -- 172.21.15.46:0/3181814497 learned_addr learned my addr 172.21.15.46:0/3181814497 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:11:32.876 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:32.874+0000 7fc9da59a640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc9d4102220 0x7fc9d419de00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:48164/0 (socket says 172.21.15.46:48164) 2024-03-31T23:11:32.876 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:32.874+0000 7fc9d9598640 1 --2- 172.21.15.46:0/3181814497 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc9d4101860 0x7fc9d4196d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:32.877 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:32.874+0000 7fc9d9d99640 1 -- 172.21.15.46:0/3181814497 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc9d4101860 msgr2=0x7fc9d4196d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:32.877 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:32.874+0000 7fc9d9d99640 1 --2- 172.21.15.46:0/3181814497 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc9d4101860 0x7fc9d4196d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:32.877 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:32.874+0000 7fc9d9d99640 1 -- 172.21.15.46:0/3181814497 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc9d4102220 msgr2=0x7fc9d419de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:32.877 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:32.874+0000 7fc9d9d99640 1 --2- 172.21.15.46:0/3181814497 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc9d4102220 0x7fc9d419de00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:32.877 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:32.874+0000 7fc9d9d99640 1 -- 172.21.15.46:0/3181814497 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9d419e470 con 0x7fc9d4100670 2024-03-31T23:11:32.877 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:32.874+0000 7fc9da59a640 1 --2- 172.21.15.46:0/3181814497 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc9d4102220 0x7fc9d419de00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:11:32.877 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:32.874+0000 7fc9d9d99640 1 --2- 172.21.15.46:0/3181814497 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc9d4100670 0x7fc9d4196850 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fc9d000e9c0 tx=0x7fc9d000ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:11:32.877 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:32.874+0000 7fc9caffd640 1 -- 172.21.15.46:0/3181814497 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9d000cd80 con 0x7fc9d4100670 2024-03-31T23:11:32.877 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:32.875+0000 7fc9dbffd640 1 -- 172.21.15.46:0/3181814497 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc9d419e760 con 0x7fc9d4100670 2024-03-31T23:11:32.877 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:32.875+0000 7fc9caffd640 1 -- 172.21.15.46:0/3181814497 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc9d00035f0 con 0x7fc9d4100670 2024-03-31T23:11:32.877 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:32.875+0000 7fc9caffd640 1 -- 172.21.15.46:0/3181814497 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9d0010690 con 0x7fc9d4100670 2024-03-31T23:11:32.877 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:32.875+0000 7fc9dbffd640 1 -- 172.21.15.46:0/3181814497 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc9d419ed00 con 0x7fc9d4100670 2024-03-31T23:11:32.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:32.875+0000 7fc9dbffd640 1 -- 172.21.15.46:0/3181814497 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc9d4107a10 con 0x7fc9d4100670 2024-03-31T23:11:32.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:32.877+0000 7fc9caffd640 1 -- 172.21.15.46:0/3181814497 <== mon.1 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 18) ==== 137320+0+0 (secure 0 0 0) 0x7fc9d00037a0 con 0x7fc9d4100670 2024-03-31T23:11:32.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:32.877+0000 7fc9caffd640 1 --2- 172.21.15.46:0/3181814497 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fc9c00a24a0 0x7fc9c00a4960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:32.879 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:32.877+0000 7fc9caffd640 1 -- 172.21.15.46:0/3181814497 <== mon.1 v2:172.21.15.162:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9208+0+0 (secure 0 0 0) 0x7fc9d00cee60 con 0x7fc9d4100670 2024-03-31T23:11:32.879 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:32.877+0000 7fc9d9598640 1 --2- 172.21.15.46:0/3181814497 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fc9c00a24a0 0x7fc9c00a4960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:32.879 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:32.878+0000 7fc9d9598640 1 --2- 172.21.15.46:0/3181814497 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fc9c00a24a0 0x7fc9c00a4960 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fc9d4197d60 tx=0x7fc9c4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:11:32.880 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:32.881+0000 7fc9caffd640 1 -- 172.21.15.46:0/3181814497 <== mon.1 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fc9d009bc50 con 0x7fc9d4100670 2024-03-31T23:11:32.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:11:32 smithi142 ceph-mon[35271]: pgmap v366: 129 pgs: 129 active+clean; 648 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 1023 B/s wr, 7 op/s 2024-03-31T23:11:32.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:11:32 smithi142 ceph-mon[35271]: from='client.15540 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-31T23:11:32.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:11:32 smithi142 ceph-mon[35271]: from='client.? 172.21.15.46:0/108681534' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-31T23:11:33.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:11:32 smithi162 ceph-mon[36237]: pgmap v366: 129 pgs: 129 active+clean; 648 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 1023 B/s wr, 7 op/s 2024-03-31T23:11:33.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:11:32 smithi162 ceph-mon[36237]: from='client.15540 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-31T23:11:33.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:11:32 smithi162 ceph-mon[36237]: from='client.? 172.21.15.46:0/108681534' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-31T23:11:33.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:33.063+0000 7fc9dbffd640 1 -- 172.21.15.46:0/3181814497 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fc9d4062350 con 0x7fc9d4100670 2024-03-31T23:11:33.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:33.064+0000 7fc9caffd640 1 -- 172.21.15.46:0/3181814497 <== mon.1 v2:172.21.15.162:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v95) ==== 104+0+11 (secure 0 0 0) 0x7fc9d009b550 con 0x7fc9d4100670 2024-03-31T23:11:33.064 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 32 2024-03-31T23:11:33.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:33.066+0000 7fc9dbffd640 1 -- 172.21.15.46:0/3181814497 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fc9c00a24a0 msgr2=0x7fc9c00a4960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:33.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:33.066+0000 7fc9dbffd640 1 --2- 172.21.15.46:0/3181814497 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fc9c00a24a0 0x7fc9c00a4960 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fc9d4197d60 tx=0x7fc9c4005f90 comp rx=0 tx=0).stop 2024-03-31T23:11:33.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:33.066+0000 7fc9dbffd640 1 -- 172.21.15.46:0/3181814497 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc9d4100670 msgr2=0x7fc9d4196850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:33.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:33.066+0000 7fc9dbffd640 1 --2- 172.21.15.46:0/3181814497 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc9d4100670 0x7fc9d4196850 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fc9d000e9c0 tx=0x7fc9d000ee80 comp rx=0 tx=0).stop 2024-03-31T23:11:33.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:33.066+0000 7fc9dbffd640 1 -- 172.21.15.46:0/3181814497 shutdown_connections 2024-03-31T23:11:33.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:33.066+0000 7fc9dbffd640 1 --2- 172.21.15.46:0/3181814497 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fc9c00a24a0 0x7fc9c00a4960 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:33.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:33.066+0000 7fc9dbffd640 1 --2- 172.21.15.46:0/3181814497 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc9d4102220 0x7fc9d419de00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:33.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:33.066+0000 7fc9dbffd640 1 --2- 172.21.15.46:0/3181814497 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc9d4101860 0x7fc9d4196d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:33.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:33.066+0000 7fc9dbffd640 1 --2- 172.21.15.46:0/3181814497 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc9d4100670 0x7fc9d4196850 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:33.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:33.066+0000 7fc9dbffd640 1 -- 172.21.15.46:0/3181814497 >> 172.21.15.46:0/3181814497 conn(0x7fc9d40fbc00 msgr2=0x7fc9d4104be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:11:33.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:33.066+0000 7fc9dbffd640 1 -- 172.21.15.46:0/3181814497 shutdown_connections 2024-03-31T23:11:33.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:33.066+0000 7fc9dbffd640 1 -- 172.21.15.46:0/3181814497 wait complete. 2024-03-31T23:11:33.077 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-31T23:11:33.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:33.182+0000 7f3891f81640 1 -- 172.21.15.46:0/2690276678 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f388c103480 msgr2=0x7f388c109d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:33.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:33.182+0000 7f3891f81640 1 --2- 172.21.15.46:0/2690276678 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f388c103480 0x7f388c109d10 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f3884009930 tx=0x7f388402bc30 comp rx=0 tx=0).stop 2024-03-31T23:11:33.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:33.182+0000 7f3891f81640 1 -- 172.21.15.46:0/2690276678 shutdown_connections 2024-03-31T23:11:33.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:33.182+0000 7f3891f81640 1 --2- 172.21.15.46:0/2690276678 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f388c103480 0x7f388c109d10 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:33.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:33.182+0000 7f3891f81640 1 --2- 172.21.15.46:0/2690276678 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f388c102ac0 0x7f388c102f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:33.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:33.182+0000 7f3891f81640 1 --2- 172.21.15.46:0/2690276678 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f388c0ffa50 0x7f388c0ffe70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:33.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:33.182+0000 7f3891f81640 1 -- 172.21.15.46:0/2690276678 >> 172.21.15.46:0/2690276678 conn(0x7f388c0fb360 msgr2=0x7f388c0fd7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:11:33.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:33.182+0000 7f3891f81640 1 -- 172.21.15.46:0/2690276678 shutdown_connections 2024-03-31T23:11:33.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:33.182+0000 7f3891f81640 1 -- 172.21.15.46:0/2690276678 wait complete. 2024-03-31T23:11:33.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:33.183+0000 7f3891f81640 1 Processor -- start 2024-03-31T23:11:33.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:33.183+0000 7f3891f81640 1 -- start start 2024-03-31T23:11:33.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:33.183+0000 7f3891f81640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f388c0ffa50 0x7f388c102430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:33.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:33.183+0000 7f3891f81640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f388c102ac0 0x7f388c100a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:33.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:33.183+0000 7f3891f81640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f388c103480 0x7f388c100fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:33.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:33.183+0000 7f3891f81640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f388c10d2b0 con 0x7f388c0ffa50 2024-03-31T23:11:33.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:33.183+0000 7f3891f81640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f388c10d130 con 0x7f388c103480 2024-03-31T23:11:33.186 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:33.183+0000 7f3891f81640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f388c10d430 con 0x7f388c102ac0 2024-03-31T23:11:33.186 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:33.183+0000 7f388b7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f388c0ffa50 0x7f388c102430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:33.186 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:33.183+0000 7f388b7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f388c0ffa50 0x7f388c102430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:48172/0 (socket says 172.21.15.46:48172) 2024-03-31T23:11:33.186 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:33.183+0000 7f388b7fe640 1 -- 172.21.15.46:0/2258191528 learned_addr learned my addr 172.21.15.46:0/2258191528 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:11:33.186 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:33.183+0000 7f388bfff640 1 --2- 172.21.15.46:0/2258191528 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f388c103480 0x7f388c100fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:33.186 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:33.183+0000 7f388affd640 1 --2- 172.21.15.46:0/2258191528 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f388c102ac0 0x7f388c100a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:33.186 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:33.184+0000 7f388b7fe640 1 -- 172.21.15.46:0/2258191528 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f388c102ac0 msgr2=0x7f388c100a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:33.186 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:33.184+0000 7f388b7fe640 1 --2- 172.21.15.46:0/2258191528 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f388c102ac0 0x7f388c100a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:33.186 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:33.184+0000 7f388b7fe640 1 -- 172.21.15.46:0/2258191528 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f388c103480 msgr2=0x7f388c100fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:33.186 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:33.184+0000 7f388b7fe640 1 --2- 172.21.15.46:0/2258191528 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f388c103480 0x7f388c100fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:33.186 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:33.184+0000 7f388b7fe640 1 -- 172.21.15.46:0/2258191528 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f388c101840 con 0x7f388c0ffa50 2024-03-31T23:11:33.186 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:33.184+0000 7f388bfff640 1 --2- 172.21.15.46:0/2258191528 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f388c103480 0x7f388c100fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:11:33.186 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:33.184+0000 7f388b7fe640 1 --2- 172.21.15.46:0/2258191528 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f388c0ffa50 0x7f388c102430 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f387c00ebc0 tx=0x7f387c00c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:11:33.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:33.184+0000 7f3888ff9640 1 -- 172.21.15.46:0/2258191528 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f387c003b70 con 0x7f388c0ffa50 2024-03-31T23:11:33.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:33.184+0000 7f3888ff9640 1 -- 172.21.15.46:0/2258191528 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f387c003d10 con 0x7f388c0ffa50 2024-03-31T23:11:33.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:33.184+0000 7f3891f81640 1 -- 172.21.15.46:0/2258191528 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f388c1a1a80 con 0x7f388c0ffa50 2024-03-31T23:11:33.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:33.184+0000 7f3888ff9640 1 -- 172.21.15.46:0/2258191528 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f387c019840 con 0x7f388c0ffa50 2024-03-31T23:11:33.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:33.184+0000 7f3891f81640 1 -- 172.21.15.46:0/2258191528 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f388c1a1fd0 con 0x7f388c0ffa50 2024-03-31T23:11:33.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:33.185+0000 7f3891f81640 1 -- 172.21.15.46:0/2258191528 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f388c108c60 con 0x7f388c0ffa50 2024-03-31T23:11:33.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:33.186+0000 7f3888ff9640 1 -- 172.21.15.46:0/2258191528 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 18) ==== 137320+0+0 (secure 0 0 0) 0x7f387c0199e0 con 0x7f388c0ffa50 2024-03-31T23:11:33.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:33.186+0000 7f3888ff9640 1 --2- 172.21.15.46:0/2258191528 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f38740a24a0 0x7f38740a4960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:33.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:33.187+0000 7f388affd640 1 --2- 172.21.15.46:0/2258191528 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f38740a24a0 0x7f38740a4960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:33.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:33.187+0000 7f3888ff9640 1 -- 172.21.15.46:0/2258191528 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9208+0+0 (secure 0 0 0) 0x7f387c0cf3d0 con 0x7f388c0ffa50 2024-03-31T23:11:33.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:33.187+0000 7f388affd640 1 --2- 172.21.15.46:0/2258191528 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f38740a24a0 0x7f38740a4960 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f388c1021a0 tx=0x7f3880005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:11:33.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:33.191+0000 7f3888ff9640 1 -- 172.21.15.46:0/2258191528 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f387c09c1c0 con 0x7f388c0ffa50 2024-03-31T23:11:33.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:33.373+0000 7f3891f81640 1 -- 172.21.15.46:0/2258191528 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f388c062350 con 0x7f388c0ffa50 2024-03-31T23:11:33.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:33.373+0000 7f3888ff9640 1 -- 172.21.15.46:0/2258191528 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v95) ==== 100+0+11 (secure 0 0 0) 0x7f387c09bac0 con 0x7f388c0ffa50 2024-03-31T23:11:33.373 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 64 2024-03-31T23:11:33.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:33.375+0000 7f3891f81640 1 -- 172.21.15.46:0/2258191528 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f38740a24a0 msgr2=0x7f38740a4960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:33.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:33.375+0000 7f3891f81640 1 --2- 172.21.15.46:0/2258191528 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f38740a24a0 0x7f38740a4960 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f388c1021a0 tx=0x7f3880005f90 comp rx=0 tx=0).stop 2024-03-31T23:11:33.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:33.375+0000 7f3891f81640 1 -- 172.21.15.46:0/2258191528 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f388c0ffa50 msgr2=0x7f388c102430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:33.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:33.375+0000 7f3891f81640 1 --2- 172.21.15.46:0/2258191528 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f388c0ffa50 0x7f388c102430 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f387c00ebc0 tx=0x7f387c00c6a0 comp rx=0 tx=0).stop 2024-03-31T23:11:33.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:33.375+0000 7f3891f81640 1 -- 172.21.15.46:0/2258191528 shutdown_connections 2024-03-31T23:11:33.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:33.375+0000 7f3891f81640 1 --2- 172.21.15.46:0/2258191528 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f38740a24a0 0x7f38740a4960 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:33.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:33.375+0000 7f3891f81640 1 --2- 172.21.15.46:0/2258191528 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f388c103480 0x7f388c100fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:33.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:33.375+0000 7f3891f81640 1 --2- 172.21.15.46:0/2258191528 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f388c102ac0 0x7f388c100a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:33.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:33.375+0000 7f3891f81640 1 --2- 172.21.15.46:0/2258191528 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f388c0ffa50 0x7f388c102430 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:33.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:33.375+0000 7f3891f81640 1 -- 172.21.15.46:0/2258191528 >> 172.21.15.46:0/2258191528 conn(0x7f388c0fb360 msgr2=0x7f388c105e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:11:33.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:33.375+0000 7f3891f81640 1 -- 172.21.15.46:0/2258191528 shutdown_connections 2024-03-31T23:11:33.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:33.375+0000 7f3891f81640 1 -- 172.21.15.46:0/2258191528 wait complete. 2024-03-31T23:11:33.386 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-31T23:11:33.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:33.491+0000 7f95c508f640 1 -- 172.21.15.46:0/3164903112 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f95c01029f0 msgr2=0x7f95c0102e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:33.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:33.491+0000 7f95c508f640 1 --2- 172.21.15.46:0/3164903112 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f95c01029f0 0x7f95c0102e70 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f95a8009930 tx=0x7f95a802bc30 comp rx=0 tx=0).stop 2024-03-31T23:11:33.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:33.491+0000 7f95c508f640 1 -- 172.21.15.46:0/3164903112 shutdown_connections 2024-03-31T23:11:33.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:33.492+0000 7f95c508f640 1 --2- 172.21.15.46:0/3164903112 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f95c01033b0 0x7f95c0109c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:33.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:33.492+0000 7f95c508f640 1 --2- 172.21.15.46:0/3164903112 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f95c01029f0 0x7f95c0102e70 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:33.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:33.492+0000 7f95c508f640 1 --2- 172.21.15.46:0/3164903112 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f95c00ffa20 0x7f95c00ffe40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:33.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:33.492+0000 7f95c508f640 1 -- 172.21.15.46:0/3164903112 >> 172.21.15.46:0/3164903112 conn(0x7f95c00fb370 msgr2=0x7f95c00fd790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:11:33.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:33.492+0000 7f95c508f640 1 -- 172.21.15.46:0/3164903112 shutdown_connections 2024-03-31T23:11:33.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:33.492+0000 7f95c508f640 1 -- 172.21.15.46:0/3164903112 wait complete. 2024-03-31T23:11:33.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:33.492+0000 7f95c508f640 1 Processor -- start 2024-03-31T23:11:33.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:33.492+0000 7f95c508f640 1 -- start start 2024-03-31T23:11:33.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:33.493+0000 7f95c508f640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f95c00ffa20 0x7f95c0197a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:33.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:33.493+0000 7f95c508f640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f95c01029f0 0x7f95c0197f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:33.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:33.493+0000 7f95c508f640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f95c01033b0 0x7f95c019efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:33.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:33.493+0000 7f95c508f640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f95c010d290 con 0x7f95c00ffa20 2024-03-31T23:11:33.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:33.493+0000 7f95c508f640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f95c010d110 con 0x7f95c01029f0 2024-03-31T23:11:33.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:33.493+0000 7f95c508f640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f95c010d410 con 0x7f95c01033b0 2024-03-31T23:11:33.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:33.493+0000 7f95bed9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f95c00ffa20 0x7f95c0197a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:33.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:33.493+0000 7f95bed9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f95c00ffa20 0x7f95c0197a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:48200/0 (socket says 172.21.15.46:48200) 2024-03-31T23:11:33.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:33.493+0000 7f95bed9d640 1 -- 172.21.15.46:0/268846486 learned_addr learned my addr 172.21.15.46:0/268846486 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:11:33.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:33.493+0000 7f95bf59e640 1 --2- 172.21.15.46:0/268846486 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f95c01033b0 0x7f95c019efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:33.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:33.493+0000 7f95be59c640 1 --2- 172.21.15.46:0/268846486 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f95c01029f0 0x7f95c0197f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:33.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:33.493+0000 7f95bed9d640 1 -- 172.21.15.46:0/268846486 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f95c01033b0 msgr2=0x7f95c019efb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:33.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:33.493+0000 7f95bed9d640 1 --2- 172.21.15.46:0/268846486 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f95c01033b0 0x7f95c019efb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:33.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:33.493+0000 7f95bed9d640 1 -- 172.21.15.46:0/268846486 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f95c01029f0 msgr2=0x7f95c0197f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:33.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:33.493+0000 7f95bed9d640 1 --2- 172.21.15.46:0/268846486 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f95c01029f0 0x7f95c0197f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:33.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:33.493+0000 7f95bed9d640 1 -- 172.21.15.46:0/268846486 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f95c019f620 con 0x7f95c00ffa20 2024-03-31T23:11:33.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:33.493+0000 7f95bed9d640 1 --2- 172.21.15.46:0/268846486 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f95c00ffa20 0x7f95c0197a00 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f95b800e9c0 tx=0x7f95b800ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:11:33.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:33.493+0000 7f95b7fff640 1 -- 172.21.15.46:0/268846486 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95b800cde0 con 0x7f95c00ffa20 2024-03-31T23:11:33.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:33.493+0000 7f95b7fff640 1 -- 172.21.15.46:0/268846486 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f95b80108e0 con 0x7f95c00ffa20 2024-03-31T23:11:33.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:33.493+0000 7f95c508f640 1 -- 172.21.15.46:0/268846486 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f95c019f910 con 0x7f95c00ffa20 2024-03-31T23:11:33.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:33.493+0000 7f95b7fff640 1 -- 172.21.15.46:0/268846486 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95b8018990 con 0x7f95c00ffa20 2024-03-31T23:11:33.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:33.493+0000 7f95c508f640 1 -- 172.21.15.46:0/268846486 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f95c019fe30 con 0x7f95c00ffa20 2024-03-31T23:11:33.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:33.495+0000 7f95c508f640 1 -- 172.21.15.46:0/268846486 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f95c0108c90 con 0x7f95c00ffa20 2024-03-31T23:11:33.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:33.495+0000 7f95b7fff640 1 -- 172.21.15.46:0/268846486 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 18) ==== 137320+0+0 (secure 0 0 0) 0x7f95b801f020 con 0x7f95c00ffa20 2024-03-31T23:11:33.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:33.496+0000 7f95b7fff640 1 --2- 172.21.15.46:0/268846486 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f95ac0a24a0 0x7f95ac0a4960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:33.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:33.496+0000 7f95be59c640 1 --2- 172.21.15.46:0/268846486 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f95ac0a24a0 0x7f95ac0a4960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:33.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:33.496+0000 7f95b7fff640 1 -- 172.21.15.46:0/268846486 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9208+0+0 (secure 0 0 0) 0x7f95b80cecc0 con 0x7f95c00ffa20 2024-03-31T23:11:33.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:33.496+0000 7f95be59c640 1 --2- 172.21.15.46:0/268846486 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f95ac0a24a0 0x7f95ac0a4960 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f95a8009900 tx=0x7f95a8005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:11:33.500 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:33.500+0000 7f95b7fff640 1 -- 172.21.15.46:0/268846486 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f95b809bab0 con 0x7f95c00ffa20 2024-03-31T23:11:33.684 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:33.684+0000 7f95c508f640 1 -- 172.21.15.46:0/268846486 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f95c002d170 con 0x7f95c00ffa20 2024-03-31T23:11:33.684 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:33.684+0000 7f95b7fff640 1 -- 172.21.15.46:0/268846486 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v95) ==== 103+0+11 (secure 0 0 0) 0x7f95c002d170 con 0x7f95c00ffa20 2024-03-31T23:11:33.684 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 32 2024-03-31T23:11:33.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:33.686+0000 7f95c508f640 1 -- 172.21.15.46:0/268846486 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f95ac0a24a0 msgr2=0x7f95ac0a4960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:33.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:33.686+0000 7f95c508f640 1 --2- 172.21.15.46:0/268846486 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f95ac0a24a0 0x7f95ac0a4960 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f95a8009900 tx=0x7f95a8005fd0 comp rx=0 tx=0).stop 2024-03-31T23:11:33.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:33.686+0000 7f95c508f640 1 -- 172.21.15.46:0/268846486 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f95c00ffa20 msgr2=0x7f95c0197a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:33.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:33.686+0000 7f95c508f640 1 --2- 172.21.15.46:0/268846486 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f95c00ffa20 0x7f95c0197a00 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f95b800e9c0 tx=0x7f95b800ee80 comp rx=0 tx=0).stop 2024-03-31T23:11:33.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:33.686+0000 7f95c508f640 1 -- 172.21.15.46:0/268846486 shutdown_connections 2024-03-31T23:11:33.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:33.686+0000 7f95c508f640 1 --2- 172.21.15.46:0/268846486 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f95ac0a24a0 0x7f95ac0a4960 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:33.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:33.686+0000 7f95c508f640 1 --2- 172.21.15.46:0/268846486 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f95c01033b0 0x7f95c019efb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:33.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:33.686+0000 7f95c508f640 1 --2- 172.21.15.46:0/268846486 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f95c01029f0 0x7f95c0197f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:33.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:33.686+0000 7f95c508f640 1 --2- 172.21.15.46:0/268846486 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f95c00ffa20 0x7f95c0197a00 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:33.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:33.686+0000 7f95c508f640 1 -- 172.21.15.46:0/268846486 >> 172.21.15.46:0/268846486 conn(0x7f95c00fb370 msgr2=0x7f95c0105e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:11:33.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:33.687+0000 7f95c508f640 1 -- 172.21.15.46:0/268846486 shutdown_connections 2024-03-31T23:11:33.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:33.687+0000 7f95c508f640 1 -- 172.21.15.46:0/268846486 wait complete. 2024-03-31T23:11:33.696 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:11:33 smithi046 ceph-mon[32498]: from='client.15546 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-31T23:11:33.697 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:11:33 smithi046 ceph-mon[32498]: from='client.15552 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-31T23:11:33.697 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:11:33 smithi046 ceph-mon[32498]: from='client.? 172.21.15.46:0/2061813286' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-31T23:11:33.697 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:11:33 smithi046 ceph-mon[32498]: from='client.? 172.21.15.46:0/3181814497' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-31T23:11:33.697 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:11:33 smithi046 ceph-mon[32498]: from='client.? 172.21.15.46:0/2258191528' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-31T23:11:33.698 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-31T23:11:33.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:33.800+0000 7f046d9e2640 1 -- 172.21.15.46:0/3268599745 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0468102960 msgr2=0x7f0468102de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:33.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:33.800+0000 7f046d9e2640 1 --2- 172.21.15.46:0/3268599745 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0468102960 0x7f0468102de0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f0450009900 tx=0x7f045002bc30 comp rx=0 tx=0).stop 2024-03-31T23:11:33.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:33.800+0000 7f046d9e2640 1 -- 172.21.15.46:0/3268599745 shutdown_connections 2024-03-31T23:11:33.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:33.800+0000 7f046d9e2640 1 --2- 172.21.15.46:0/3268599745 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0468103320 0x7f0468109bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:33.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:33.800+0000 7f046d9e2640 1 --2- 172.21.15.46:0/3268599745 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0468102960 0x7f0468102de0 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:33.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:33.800+0000 7f046d9e2640 1 --2- 172.21.15.46:0/3268599745 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0468101770 0x7f0468101b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:33.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:33.800+0000 7f046d9e2640 1 -- 172.21.15.46:0/3268599745 >> 172.21.15.46:0/3268599745 conn(0x7f04680fcd00 msgr2=0x7f04680ff120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:11:33.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:33.800+0000 7f046d9e2640 1 -- 172.21.15.46:0/3268599745 shutdown_connections 2024-03-31T23:11:33.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:33.800+0000 7f046d9e2640 1 -- 172.21.15.46:0/3268599745 wait complete. 2024-03-31T23:11:33.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:33.801+0000 7f046d9e2640 1 Processor -- start 2024-03-31T23:11:33.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:33.801+0000 7f046d9e2640 1 -- start start 2024-03-31T23:11:33.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:33.801+0000 7f046d9e2640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0468101770 0x7f0468075500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:33.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:33.801+0000 7f046d9e2640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0468102960 0x7f0468073b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:33.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:33.801+0000 7f046d9e2640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0468103320 0x7f0468074080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:33.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:33.801+0000 7f046d9e2640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f046810d280 con 0x7f0468103320 2024-03-31T23:11:33.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:33.801+0000 7f046d9e2640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f046810d100 con 0x7f0468101770 2024-03-31T23:11:33.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:33.801+0000 7f046d9e2640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f046810d400 con 0x7f0468102960 2024-03-31T23:11:33.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:33.801+0000 7f04677fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0468103320 0x7f0468074080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:33.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:33.801+0000 7f04677fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0468103320 0x7f0468074080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:48218/0 (socket says 172.21.15.46:48218) 2024-03-31T23:11:33.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:33.801+0000 7f04677fe640 1 -- 172.21.15.46:0/339498258 learned_addr learned my addr 172.21.15.46:0/339498258 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:11:33.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:33.801+0000 7f0466ffd640 1 --2- 172.21.15.46:0/339498258 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0468101770 0x7f0468075500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:33.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:33.801+0000 7f045ffff640 1 --2- 172.21.15.46:0/339498258 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0468102960 0x7f0468073b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:33.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:33.801+0000 7f04677fe640 1 -- 172.21.15.46:0/339498258 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0468102960 msgr2=0x7f0468073b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:33.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:33.801+0000 7f04677fe640 1 --2- 172.21.15.46:0/339498258 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0468102960 0x7f0468073b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:33.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:33.801+0000 7f04677fe640 1 -- 172.21.15.46:0/339498258 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0468101770 msgr2=0x7f0468075500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:33.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:33.801+0000 7f04677fe640 1 --2- 172.21.15.46:0/339498258 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0468101770 0x7f0468075500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:33.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:33.801+0000 7f04677fe640 1 -- 172.21.15.46:0/339498258 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0468074910 con 0x7f0468103320 2024-03-31T23:11:33.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:33.801+0000 7f0466ffd640 1 --2- 172.21.15.46:0/339498258 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0468101770 0x7f0468075500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:11:33.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:33.802+0000 7f04677fe640 1 --2- 172.21.15.46:0/339498258 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0468103320 0x7f0468074080 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f046000eb10 tx=0x7f046000c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:11:33.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:33.802+0000 7f046c9e0640 1 -- 172.21.15.46:0/339498258 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0460004b20 con 0x7f0468103320 2024-03-31T23:11:33.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:33.802+0000 7f046c9e0640 1 -- 172.21.15.46:0/339498258 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0460004cc0 con 0x7f0468103320 2024-03-31T23:11:33.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:33.802+0000 7f046d9e2640 1 -- 172.21.15.46:0/339498258 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f04681a1aa0 con 0x7f0468103320 2024-03-31T23:11:33.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:33.802+0000 7f046c9e0640 1 -- 172.21.15.46:0/339498258 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0460010690 con 0x7f0468103320 2024-03-31T23:11:33.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:33.802+0000 7f046d9e2640 1 -- 172.21.15.46:0/339498258 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f04681a1fc0 con 0x7f0468103320 2024-03-31T23:11:33.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:33.803+0000 7f046d9e2640 1 -- 172.21.15.46:0/339498258 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0468068990 con 0x7f0468103320 2024-03-31T23:11:33.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:33.804+0000 7f046c9e0640 1 -- 172.21.15.46:0/339498258 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 18) ==== 137320+0+0 (secure 0 0 0) 0x7f0460005020 con 0x7f0468103320 2024-03-31T23:11:33.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:33.804+0000 7f046c9e0640 1 --2- 172.21.15.46:0/339498258 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f04540a2520 0x7f04540a49e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:33.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:33.805+0000 7f0466ffd640 1 --2- 172.21.15.46:0/339498258 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f04540a2520 0x7f04540a49e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:33.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:33.805+0000 7f046c9e0640 1 -- 172.21.15.46:0/339498258 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9208+0+0 (secure 0 0 0) 0x7f04600cf070 con 0x7f0468103320 2024-03-31T23:11:33.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:33.805+0000 7f0466ffd640 1 --2- 172.21.15.46:0/339498258 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f04540a2520 0x7f04540a49e0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f04681027c0 tx=0x7f0458005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:11:33.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:33.809+0000 7f046c9e0640 1 -- 172.21.15.46:0/339498258 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f046009be60 con 0x7f0468103320 2024-03-31T23:11:33.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:11:33 smithi142 ceph-mon[35271]: from='client.15546 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-31T23:11:33.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:11:33 smithi142 ceph-mon[35271]: from='client.15552 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-31T23:11:33.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:11:33 smithi142 ceph-mon[35271]: from='client.? 172.21.15.46:0/2061813286' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-31T23:11:33.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:11:33 smithi142 ceph-mon[35271]: from='client.? 172.21.15.46:0/3181814497' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-31T23:11:33.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:11:33 smithi142 ceph-mon[35271]: from='client.? 172.21.15.46:0/2258191528' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-31T23:11:34.029 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.029+0000 7f046d9e2640 1 -- 172.21.15.46:0/339498258 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f046802d170 con 0x7f0468103320 2024-03-31T23:11:34.029 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.029+0000 7f046c9e0640 1 -- 172.21.15.46:0/339498258 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+11882 (secure 0 0 0) 0x7f046009b760 con 0x7f0468103320 2024-03-31T23:11:34.030 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-31T23:11:34.030 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/2169018547","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":2169018547},{"type":"v1","addr":"172.21.15.46:6835","nonce":2169018547}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6837/1069387756","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6836","nonce":1069387756},{"type":"v1","addr":"172.21.15.46:6837","nonce":1069387756}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14634,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6839/3915337844","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6838","nonce":3915337844},{"type":"v1","addr":"172.21.15.46:6839","nonce":3915337844}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14646,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6841/897767145","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6840","nonce":897767145},{"type":"v1","addr":"172.21.15.46:6841","nonce":897767145}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24415,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6833/1746950966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6832","nonce":1746950966},{"type":"v1","addr":"172.21.15.162:6833","nonce":1746950966}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24422,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6833/3394739181","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6832","nonce":3394739181},{"type":"v1","addr":"172.21.15.142:6833","nonce":3394739181}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":25,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-31T23:09:11.009090+0000","modified":"2024-03-31T23:11:27.558846+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24515,"mds_1":24482,"mds_2":24451},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24442":{"gid":24442,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.162:6835/1957208984","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6834","nonce":1957208984},{"type":"v1","addr":"172.21.15.162:6835","nonce":1957208984}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24451":{"gid":24451,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":11,"addr":"172.21.15.162:6837/1371886876","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6836","nonce":1371886876},{"type":"v1","addr":"172.21.15.162:6837","nonce":1371886876}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24455":{"gid":24455,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.142:6835/212637431","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6834","nonce":212637431},{"type":"v1","addr":"172.21.15.142:6835","nonce":212637431}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24478":{"gid":24478,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.162:6839/1976275826","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6838","nonce":1976275826},{"type":"v1","addr":"172.21.15.162:6839","nonce":1976275826}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24482":{"gid":24482,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":10,"addr":"172.21.15.142:6837/2559258399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6836","nonce":2559258399},{"type":"v1","addr":"172.21.15.142:6837","nonce":2559258399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24515":{"gid":24515,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.142:6839/2519143918","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6838","nonce":2519143918},{"type":"v1","addr":"172.21.15.142:6839","nonce":2519143918}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24515,"qdb_cluster":[24515,24451,24482]},"id":1}]} 2024-03-31T23:11:34.030 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 25 2024-03-31T23:11:34.031 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.031+0000 7f046d9e2640 1 -- 172.21.15.46:0/339498258 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f04540a2520 msgr2=0x7f04540a49e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:34.032 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.031+0000 7f046d9e2640 1 --2- 172.21.15.46:0/339498258 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f04540a2520 0x7f04540a49e0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f04681027c0 tx=0x7f0458005f90 comp rx=0 tx=0).stop 2024-03-31T23:11:34.032 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.031+0000 7f046d9e2640 1 -- 172.21.15.46:0/339498258 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0468103320 msgr2=0x7f0468074080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:34.032 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.031+0000 7f046d9e2640 1 --2- 172.21.15.46:0/339498258 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0468103320 0x7f0468074080 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f046000eb10 tx=0x7f046000c2d0 comp rx=0 tx=0).stop 2024-03-31T23:11:34.032 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.031+0000 7f046d9e2640 1 -- 172.21.15.46:0/339498258 shutdown_connections 2024-03-31T23:11:34.032 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.031+0000 7f046d9e2640 1 --2- 172.21.15.46:0/339498258 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f04540a2520 0x7f04540a49e0 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:34.032 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.031+0000 7f046d9e2640 1 --2- 172.21.15.46:0/339498258 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0468103320 0x7f0468074080 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:34.032 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.031+0000 7f046d9e2640 1 --2- 172.21.15.46:0/339498258 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0468102960 0x7f0468073b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:34.032 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.031+0000 7f046d9e2640 1 --2- 172.21.15.46:0/339498258 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0468101770 0x7f0468075500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:34.032 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.031+0000 7f046d9e2640 1 -- 172.21.15.46:0/339498258 >> 172.21.15.46:0/339498258 conn(0x7f04680fcd00 msgr2=0x7f0468105d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:11:34.032 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.031+0000 7f046d9e2640 1 -- 172.21.15.46:0/339498258 shutdown_connections 2024-03-31T23:11:34.032 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.032+0000 7f046d9e2640 1 -- 172.21.15.46:0/339498258 wait complete. 2024-03-31T23:11:34.043 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-31T23:11:34.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:11:33 smithi162 ceph-mon[36237]: from='client.15546 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-31T23:11:34.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:11:33 smithi162 ceph-mon[36237]: from='client.15552 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-31T23:11:34.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:11:33 smithi162 ceph-mon[36237]: from='client.? 172.21.15.46:0/2061813286' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-31T23:11:34.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:11:33 smithi162 ceph-mon[36237]: from='client.? 172.21.15.46:0/3181814497' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-31T23:11:34.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:11:33 smithi162 ceph-mon[36237]: from='client.? 172.21.15.46:0/2258191528' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-31T23:11:34.148 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.146+0000 7fd2dce4e640 1 -- 172.21.15.46:0/2179723049 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd2d81006f0 msgr2=0x7fd2d8100b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:34.148 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.146+0000 7fd2dce4e640 1 --2- 172.21.15.46:0/2179723049 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd2d81006f0 0x7fd2d8100b10 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7fd2d0009900 tx=0x7fd2d002bc30 comp rx=0 tx=0).stop 2024-03-31T23:11:34.148 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.146+0000 7fd2dce4e640 1 -- 172.21.15.46:0/2179723049 shutdown_connections 2024-03-31T23:11:34.148 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.146+0000 7fd2dce4e640 1 --2- 172.21.15.46:0/2179723049 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd2d81022a0 0x7fd2d8108b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:34.148 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.146+0000 7fd2dce4e640 1 --2- 172.21.15.46:0/2179723049 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd2d81018e0 0x7fd2d8101d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:34.148 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.146+0000 7fd2dce4e640 1 --2- 172.21.15.46:0/2179723049 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd2d81006f0 0x7fd2d8100b10 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:34.148 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.146+0000 7fd2dce4e640 1 -- 172.21.15.46:0/2179723049 >> 172.21.15.46:0/2179723049 conn(0x7fd2d80fbc80 msgr2=0x7fd2d80fe0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:11:34.148 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.146+0000 7fd2dce4e640 1 -- 172.21.15.46:0/2179723049 shutdown_connections 2024-03-31T23:11:34.148 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.146+0000 7fd2dce4e640 1 -- 172.21.15.46:0/2179723049 wait complete. 2024-03-31T23:11:34.149 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.146+0000 7fd2dce4e640 1 Processor -- start 2024-03-31T23:11:34.149 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.146+0000 7fd2dce4e640 1 -- start start 2024-03-31T23:11:34.149 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.147+0000 7fd2dce4e640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd2d81006f0 0x7fd2d8197a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:34.149 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.147+0000 7fd2dce4e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd2d81018e0 0x7fd2d8197f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:34.149 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.147+0000 7fd2dce4e640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd2d81022a0 0x7fd2d819ef50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:34.149 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.147+0000 7fd2dce4e640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fd2d810c1b0 con 0x7fd2d81018e0 2024-03-31T23:11:34.149 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.147+0000 7fd2dce4e640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fd2d810c030 con 0x7fd2d81022a0 2024-03-31T23:11:34.149 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.147+0000 7fd2dce4e640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fd2d810c330 con 0x7fd2d81006f0 2024-03-31T23:11:34.149 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.147+0000 7fd2d5d9b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd2d81018e0 0x7fd2d8197f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:34.149 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.147+0000 7fd2d5d9b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd2d81018e0 0x7fd2d8197f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:48240/0 (socket says 172.21.15.46:48240) 2024-03-31T23:11:34.149 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.147+0000 7fd2d5d9b640 1 -- 172.21.15.46:0/3057859208 learned_addr learned my addr 172.21.15.46:0/3057859208 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:11:34.149 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.147+0000 7fd2d659c640 1 --2- 172.21.15.46:0/3057859208 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd2d81006f0 0x7fd2d8197a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:34.149 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.147+0000 7fd2d6d9d640 1 --2- 172.21.15.46:0/3057859208 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd2d81022a0 0x7fd2d819ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:34.149 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.147+0000 7fd2d5d9b640 1 -- 172.21.15.46:0/3057859208 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd2d81006f0 msgr2=0x7fd2d8197a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:34.149 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.147+0000 7fd2d5d9b640 1 --2- 172.21.15.46:0/3057859208 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd2d81006f0 0x7fd2d8197a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:34.150 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.147+0000 7fd2d5d9b640 1 -- 172.21.15.46:0/3057859208 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd2d81022a0 msgr2=0x7fd2d819ef50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:34.150 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.147+0000 7fd2d5d9b640 1 --2- 172.21.15.46:0/3057859208 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd2d81022a0 0x7fd2d819ef50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:34.150 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.147+0000 7fd2d5d9b640 1 -- 172.21.15.46:0/3057859208 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd2d819f5c0 con 0x7fd2d81018e0 2024-03-31T23:11:34.150 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.147+0000 7fd2d5d9b640 1 --2- 172.21.15.46:0/3057859208 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd2d81018e0 0x7fd2d8197f80 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7fd2c000e9f0 tx=0x7fd2c000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:11:34.150 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.147+0000 7fd2cf7fe640 1 -- 172.21.15.46:0/3057859208 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2c000cd80 con 0x7fd2d81018e0 2024-03-31T23:11:34.150 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.147+0000 7fd2cf7fe640 1 -- 172.21.15.46:0/3057859208 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd2c00108e0 con 0x7fd2d81018e0 2024-03-31T23:11:34.150 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.147+0000 7fd2dce4e640 1 -- 172.21.15.46:0/3057859208 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd2d819f8b0 con 0x7fd2d81018e0 2024-03-31T23:11:34.150 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.147+0000 7fd2cf7fe640 1 -- 172.21.15.46:0/3057859208 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2c0018920 con 0x7fd2d81018e0 2024-03-31T23:11:34.150 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.147+0000 7fd2dce4e640 1 -- 172.21.15.46:0/3057859208 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd2d819fe00 con 0x7fd2d81018e0 2024-03-31T23:11:34.151 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.149+0000 7fd2dce4e640 1 -- 172.21.15.46:0/3057859208 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd2d81a0030 con 0x7fd2d81018e0 2024-03-31T23:11:34.151 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.149+0000 7fd2cf7fe640 1 -- 172.21.15.46:0/3057859208 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 18) ==== 137320+0+0 (secure 0 0 0) 0x7fd2c0010a90 con 0x7fd2d81018e0 2024-03-31T23:11:34.151 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.150+0000 7fd2cf7fe640 1 --2- 172.21.15.46:0/3057859208 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fd2c40a24a0 0x7fd2c40a4960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:34.151 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.150+0000 7fd2d659c640 1 --2- 172.21.15.46:0/3057859208 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fd2c40a24a0 0x7fd2c40a4960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:34.151 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.150+0000 7fd2cf7fe640 1 -- 172.21.15.46:0/3057859208 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9208+0+0 (secure 0 0 0) 0x7fd2c00cee20 con 0x7fd2d81018e0 2024-03-31T23:11:34.151 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.150+0000 7fd2d659c640 1 --2- 172.21.15.46:0/3057859208 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fd2c40a24a0 0x7fd2c40a4960 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fd2d0005ab0 tx=0x7fd2d0005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:11:34.154 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.154+0000 7fd2cf7fe640 1 -- 172.21.15.46:0/3057859208 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fd2c009b560 con 0x7fd2d81018e0 2024-03-31T23:11:34.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.339+0000 7fd2dce4e640 1 -- 172.21.15.46:0/3057859208 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fd2d81a0720 con 0x7fd2d81018e0 2024-03-31T23:11:34.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.341+0000 7fd2cf7fe640 1 -- 172.21.15.46:0/3057859208 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v95) ==== 74+0+24819 (secure 0 0 0) 0x7fd2c009f600 con 0x7fd2d81018e0 2024-03-31T23:11:34.341 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-31T23:11:34.341 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":95,"fsid":"6a9d8f08-efb2-11ee-b647-cb9ed24678a4","created":"2024-03-31T23:01:15.463711+0000","modified":"2024-03-31T23:11:10.531495+0000","last_up_change":"2024-03-31T23:08:20.199888+0000","last_in_change":"2024-03-31T23:07:59.438035+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-31T23:04:32.823751+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-31T23:09:09.145711+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-31T23:09:10.148579+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-31T23:09:13.419519+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"91","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"47429095-3f46-4ed7-b7c4-504464719815","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":1824772695},{"type":"v1","addr":"172.21.15.46:6803","nonce":1824772695}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":1824772695},{"type":"v1","addr":"172.21.15.46:6805","nonce":1824772695}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":1824772695},{"type":"v1","addr":"172.21.15.46:6809","nonce":1824772695}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":1824772695},{"type":"v1","addr":"172.21.15.46:6807","nonce":1824772695}]},"public_addr":"172.21.15.46:6803/1824772695","cluster_addr":"172.21.15.46:6805/1824772695","heartbeat_back_addr":"172.21.15.46:6809/1824772695","heartbeat_front_addr":"172.21.15.46:6807/1824772695","state":["exists","up"]},{"osd":1,"uuid":"be56f883-411e-4dbe-b50e-a158134df4a6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6800","nonce":973693922},{"type":"v1","addr":"172.21.15.142:6801","nonce":973693922}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6802","nonce":973693922},{"type":"v1","addr":"172.21.15.142:6803","nonce":973693922}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6806","nonce":973693922},{"type":"v1","addr":"172.21.15.142:6807","nonce":973693922}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6804","nonce":973693922},{"type":"v1","addr":"172.21.15.142:6805","nonce":973693922}]},"public_addr":"172.21.15.142:6801/973693922","cluster_addr":"172.21.15.142:6803/973693922","heartbeat_back_addr":"172.21.15.142:6807/973693922","heartbeat_front_addr":"172.21.15.142:6805/973693922","state":["exists","up"]},{"osd":2,"uuid":"c99a8794-ab19-4260-9a8d-4bc0bd2c8175","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6800","nonce":4084261813},{"type":"v1","addr":"172.21.15.162:6801","nonce":4084261813}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6802","nonce":4084261813},{"type":"v1","addr":"172.21.15.162:6803","nonce":4084261813}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6806","nonce":4084261813},{"type":"v1","addr":"172.21.15.162:6807","nonce":4084261813}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6804","nonce":4084261813},{"type":"v1","addr":"172.21.15.162:6805","nonce":4084261813}]},"public_addr":"172.21.15.162:6801/4084261813","cluster_addr":"172.21.15.162:6803/4084261813","heartbeat_back_addr":"172.21.15.162:6807/4084261813","heartbeat_front_addr":"172.21.15.162:6805/4084261813","state":["exists","up"]},{"osd":3,"uuid":"f877814e-9ef6-4872-a081-5609bb9ebc06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":3496317970},{"type":"v1","addr":"172.21.15.46:6811","nonce":3496317970}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":3496317970},{"type":"v1","addr":"172.21.15.46:6813","nonce":3496317970}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":3496317970},{"type":"v1","addr":"172.21.15.46:6817","nonce":3496317970}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":3496317970},{"type":"v1","addr":"172.21.15.46:6815","nonce":3496317970}]},"public_addr":"172.21.15.46:6811/3496317970","cluster_addr":"172.21.15.46:6813/3496317970","heartbeat_back_addr":"172.21.15.46:6817/3496317970","heartbeat_front_addr":"172.21.15.46:6815/3496317970","state":["exists","up"]},{"osd":4,"uuid":"beecec0a-8280-44fe-8f5f-8498a35eca8f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6808","nonce":2996266929},{"type":"v1","addr":"172.21.15.142:6809","nonce":2996266929}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6810","nonce":2996266929},{"type":"v1","addr":"172.21.15.142:6811","nonce":2996266929}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6814","nonce":2996266929},{"type":"v1","addr":"172.21.15.142:6815","nonce":2996266929}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6812","nonce":2996266929},{"type":"v1","addr":"172.21.15.142:6813","nonce":2996266929}]},"public_addr":"172.21.15.142:6809/2996266929","cluster_addr":"172.21.15.142:6811/2996266929","heartbeat_back_addr":"172.21.15.142:6815/2996266929","heartbeat_front_addr":"172.21.15.142:6813/2996266929","state":["exists","up"]},{"osd":5,"uuid":"9da7ff75-f658-45ea-a02f-1943fdda76b9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6808","nonce":914208293},{"type":"v1","addr":"172.21.15.162:6809","nonce":914208293}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6810","nonce":914208293},{"type":"v1","addr":"172.21.15.162:6811","nonce":914208293}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6814","nonce":914208293},{"type":"v1","addr":"172.21.15.162:6815","nonce":914208293}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6812","nonce":914208293},{"type":"v1","addr":"172.21.15.162:6813","nonce":914208293}]},"public_addr":"172.21.15.162:6809/914208293","cluster_addr":"172.21.15.162:6811/914208293","heartbeat_back_addr":"172.21.15.162:6815/914208293","heartbeat_front_addr":"172.21.15.162:6813/914208293","state":["exists","up"]},{"osd":6,"uuid":"8fa11224-cb26-4d3f-87d1-6a8f31de2a19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":56291966},{"type":"v1","addr":"172.21.15.46:6819","nonce":56291966}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":56291966},{"type":"v1","addr":"172.21.15.46:6821","nonce":56291966}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6824","nonce":56291966},{"type":"v1","addr":"172.21.15.46:6825","nonce":56291966}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":56291966},{"type":"v1","addr":"172.21.15.46:6823","nonce":56291966}]},"public_addr":"172.21.15.46:6819/56291966","cluster_addr":"172.21.15.46:6821/56291966","heartbeat_back_addr":"172.21.15.46:6825/56291966","heartbeat_front_addr":"172.21.15.46:6823/56291966","state":["exists","up"]},{"osd":7,"uuid":"0d0338b7-4d0a-4bb0-b152-8f5224871d9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6816","nonce":2146598446},{"type":"v1","addr":"172.21.15.142:6817","nonce":2146598446}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6818","nonce":2146598446},{"type":"v1","addr":"172.21.15.142:6819","nonce":2146598446}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6822","nonce":2146598446},{"type":"v1","addr":"172.21.15.142:6823","nonce":2146598446}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6820","nonce":2146598446},{"type":"v1","addr":"172.21.15.142:6821","nonce":2146598446}]},"public_addr":"172.21.15.142:6817/2146598446","cluster_addr":"172.21.15.142:6819/2146598446","heartbeat_back_addr":"172.21.15.142:6823/2146598446","heartbeat_front_addr":"172.21.15.142:6821/2146598446","state":["exists","up"]},{"osd":8,"uuid":"8b6f6037-892a-416c-babb-7552648d0b3a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6816","nonce":1173078318},{"type":"v1","addr":"172.21.15.162:6817","nonce":1173078318}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6818","nonce":1173078318},{"type":"v1","addr":"172.21.15.162:6819","nonce":1173078318}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6822","nonce":1173078318},{"type":"v1","addr":"172.21.15.162:6823","nonce":1173078318}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6820","nonce":1173078318},{"type":"v1","addr":"172.21.15.162:6821","nonce":1173078318}]},"public_addr":"172.21.15.162:6817/1173078318","cluster_addr":"172.21.15.162:6819/1173078318","heartbeat_back_addr":"172.21.15.162:6823/1173078318","heartbeat_front_addr":"172.21.15.162:6821/1173078318","state":["exists","up"]},{"osd":9,"uuid":"faccce25-6bf2-4220-8858-cda82b72be68","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":532708881},{"type":"v1","addr":"172.21.15.46:6827","nonce":532708881}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6828","nonce":532708881},{"type":"v1","addr":"172.21.15.46:6829","nonce":532708881}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":532708881},{"type":"v1","addr":"172.21.15.46:6833","nonce":532708881}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":532708881},{"type":"v1","addr":"172.21.15.46:6831","nonce":532708881}]},"public_addr":"172.21.15.46:6827/532708881","cluster_addr":"172.21.15.46:6829/532708881","heartbeat_back_addr":"172.21.15.46:6833/532708881","heartbeat_front_addr":"172.21.15.46:6831/532708881","state":["exists","up"]},{"osd":10,"uuid":"cdd356db-f9eb-4080-95ab-db5d8223e572","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6824","nonce":2845222366},{"type":"v1","addr":"172.21.15.142:6825","nonce":2845222366}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6826","nonce":2845222366},{"type":"v1","addr":"172.21.15.142:6827","nonce":2845222366}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6830","nonce":2845222366},{"type":"v1","addr":"172.21.15.142:6831","nonce":2845222366}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6828","nonce":2845222366},{"type":"v1","addr":"172.21.15.142:6829","nonce":2845222366}]},"public_addr":"172.21.15.142:6825/2845222366","cluster_addr":"172.21.15.142:6827/2845222366","heartbeat_back_addr":"172.21.15.142:6831/2845222366","heartbeat_front_addr":"172.21.15.142:6829/2845222366","state":["exists","up"]},{"osd":11,"uuid":"72104d1c-3ea2-45ad-bc0b-0500f146c06b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6824","nonce":1986323778},{"type":"v1","addr":"172.21.15.162:6825","nonce":1986323778}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6826","nonce":1986323778},{"type":"v1","addr":"172.21.15.162:6827","nonce":1986323778}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6830","nonce":1986323778},{"type":"v1","addr":"172.21.15.162:6831","nonce":1986323778}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6828","nonce":1986323778},{"type":"v1","addr":"172.21.15.162:6829","nonce":1986323778}]},"public_addr":"172.21.15.162:6825/1986323778","cluster_addr":"172.21.15.162:6827/1986323778","heartbeat_back_addr":"172.21.15.162:6831/1986323778","heartbeat_front_addr":"172.21.15.162:6829/1986323778","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:03:44.996331+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:04:07.994738+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:04:30.942255+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:04:55.588884+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:05:19.900600+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:05:43.807442+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:06:09.026976+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:06:34.586784+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:06:59.751605+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:07:26.144495+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:07:52.326909+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:08:17.423042+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.b","mappings":[{"from":8,"to":11}]},{"pgid":"3.6","mappings":[{"from":4,"to":7}]},{"pgid":"3.c","mappings":[{"from":3,"to":9}]},{"pgid":"3.1a","mappings":[{"from":5,"to":6}]},{"pgid":"3.1b","mappings":[{"from":4,"to":8}]},{"pgid":"3.1d","mappings":[{"from":4,"to":8}]},{"pgid":"3.2a","mappings":[{"from":3,"to":6}]},{"pgid":"3.33","mappings":[{"from":5,"to":9}]},{"pgid":"3.36","mappings":[{"from":4,"to":9}]},{"pgid":"4.6","mappings":[{"from":5,"to":8}]},{"pgid":"4.17","mappings":[{"from":9,"to":8}]},{"pgid":"4.18","mappings":[{"from":9,"to":8}]},{"pgid":"4.1b","mappings":[{"from":9,"to":10}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.46:0/4021039857":"2024-04-01T23:02:08.493619+0000","172.21.15.46:6801/3681182647":"2024-04-01T23:02:08.493619+0000","172.21.15.46:0/4022214806":"2024-04-01T23:02:08.493619+0000","172.21.15.46:6800/1515528215":"2024-04-01T23:01:41.910162+0000","172.21.15.46:6800/3681182647":"2024-04-01T23:02:08.493619+0000","172.21.15.46:0/143885707":"2024-04-01T23:01:41.910162+0000","172.21.15.46:6801/1515528215":"2024-04-01T23:01:41.910162+0000","172.21.15.46:0/1680389778":"2024-04-01T23:01:41.910162+0000","172.21.15.46:0/1782133966":"2024-04-01T23:01:41.910162+0000","172.21.15.46:0/3925209376":"2024-04-01T23:02:08.493619+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-31T23:11:34.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.343+0000 7fd2dce4e640 1 -- 172.21.15.46:0/3057859208 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fd2c40a24a0 msgr2=0x7fd2c40a4960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:34.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.343+0000 7fd2dce4e640 1 --2- 172.21.15.46:0/3057859208 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fd2c40a24a0 0x7fd2c40a4960 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fd2d0005ab0 tx=0x7fd2d0005fd0 comp rx=0 tx=0).stop 2024-03-31T23:11:34.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.343+0000 7fd2dce4e640 1 -- 172.21.15.46:0/3057859208 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd2d81018e0 msgr2=0x7fd2d8197f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:34.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.343+0000 7fd2dce4e640 1 --2- 172.21.15.46:0/3057859208 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd2d81018e0 0x7fd2d8197f80 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7fd2c000e9f0 tx=0x7fd2c000eeb0 comp rx=0 tx=0).stop 2024-03-31T23:11:34.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.343+0000 7fd2dce4e640 1 -- 172.21.15.46:0/3057859208 shutdown_connections 2024-03-31T23:11:34.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.343+0000 7fd2dce4e640 1 --2- 172.21.15.46:0/3057859208 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fd2c40a24a0 0x7fd2c40a4960 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:34.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.343+0000 7fd2dce4e640 1 --2- 172.21.15.46:0/3057859208 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd2d81022a0 0x7fd2d819ef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:34.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.343+0000 7fd2dce4e640 1 --2- 172.21.15.46:0/3057859208 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd2d81018e0 0x7fd2d8197f80 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:34.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.343+0000 7fd2dce4e640 1 --2- 172.21.15.46:0/3057859208 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd2d81006f0 0x7fd2d8197a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:34.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.343+0000 7fd2dce4e640 1 -- 172.21.15.46:0/3057859208 >> 172.21.15.46:0/3057859208 conn(0x7fd2d80fbc80 msgr2=0x7fd2d8069d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:11:34.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.343+0000 7fd2dce4e640 1 -- 172.21.15.46:0/3057859208 shutdown_connections 2024-03-31T23:11:34.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.343+0000 7fd2dce4e640 1 -- 172.21.15.46:0/3057859208 wait complete. 2024-03-31T23:11:34.355 DEBUG:tasks.fwd_scrub:Yielding 2024-03-31T23:11:34.355 INFO:teuthology.run_tasks:Running task exec... 2024-03-31T23:11:34.364 INFO:teuthology.task.exec:Executing custom commands... 2024-03-31T23:11:34.365 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi046.front.sepia.ceph.com 2024-03-31T23:11:34.365 DEBUG:teuthology.orchestra.run.smithi046:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph mgr module enable snap_schedule' 2024-03-31T23:11:34.365 INFO:tasks.fwd_scrub.fs.[cephfs]:start scrubbing fs: cephfs 2024-03-31T23:11:34.365 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-31T23:11:34.365 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T23:11:34.366 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-31T23:11:34.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.469+0000 7f655a3c9640 1 -- 172.21.15.46:0/2700110109 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6554101860 msgr2=0x7f6554101ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:34.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.469+0000 7f655a3c9640 1 --2- 172.21.15.46:0/2700110109 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6554101860 0x7f6554101ce0 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f6548009900 tx=0x7f654802bc30 comp rx=0 tx=0).stop 2024-03-31T23:11:34.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.469+0000 7f655a3c9640 1 -- 172.21.15.46:0/2700110109 shutdown_connections 2024-03-31T23:11:34.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.469+0000 7f655a3c9640 1 --2- 172.21.15.46:0/2700110109 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6554102220 0x7f6554108ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:34.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.469+0000 7f655a3c9640 1 --2- 172.21.15.46:0/2700110109 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6554101860 0x7f6554101ce0 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:34.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.469+0000 7f655a3c9640 1 --2- 172.21.15.46:0/2700110109 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6554100670 0x7f6554100a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:34.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.469+0000 7f655a3c9640 1 -- 172.21.15.46:0/2700110109 >> 172.21.15.46:0/2700110109 conn(0x7f65540fbbe0 msgr2=0x7f65540fe020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:11:34.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.470+0000 7f655a3c9640 1 -- 172.21.15.46:0/2700110109 shutdown_connections 2024-03-31T23:11:34.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.470+0000 7f655a3c9640 1 -- 172.21.15.46:0/2700110109 wait complete. 2024-03-31T23:11:34.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.470+0000 7f655a3c9640 1 Processor -- start 2024-03-31T23:11:34.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.470+0000 7f655a3c9640 1 -- start start 2024-03-31T23:11:34.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.471+0000 7f655a3c9640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6554100670 0x7f6554197930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:34.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.471+0000 7f655a3c9640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6554101860 0x7f6554197e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:34.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.471+0000 7f655a3c9640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6554102220 0x7f655419eee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:34.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.471+0000 7f655a3c9640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f655410c1f0 con 0x7f6554101860 2024-03-31T23:11:34.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.471+0000 7f655a3c9640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f655410c070 con 0x7f6554100670 2024-03-31T23:11:34.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.471+0000 7f655a3c9640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f655410c370 con 0x7f6554102220 2024-03-31T23:11:34.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.471+0000 7f65537fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6554101860 0x7f6554197e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:34.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.471+0000 7f6553fff640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6554100670 0x7f6554197930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:34.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.471+0000 7f6553fff640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6554100670 0x7f6554197930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.46:45196/0 (socket says 172.21.15.46:45196) 2024-03-31T23:11:34.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.471+0000 7f6553fff640 1 -- 172.21.15.46:0/3829663715 learned_addr learned my addr 172.21.15.46:0/3829663715 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:11:34.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.471+0000 7f6558966640 1 --2- 172.21.15.46:0/3829663715 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6554102220 0x7f655419eee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:34.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.471+0000 7f65537fe640 1 -- 172.21.15.46:0/3829663715 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6554102220 msgr2=0x7f655419eee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:34.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.471+0000 7f65537fe640 1 --2- 172.21.15.46:0/3829663715 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6554102220 0x7f655419eee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:34.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.471+0000 7f65537fe640 1 -- 172.21.15.46:0/3829663715 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6554100670 msgr2=0x7f6554197930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:34.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.471+0000 7f65537fe640 1 --2- 172.21.15.46:0/3829663715 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6554100670 0x7f6554197930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:34.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.471+0000 7f65537fe640 1 -- 172.21.15.46:0/3829663715 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f655419f550 con 0x7f6554101860 2024-03-31T23:11:34.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.471+0000 7f6553fff640 1 --2- 172.21.15.46:0/3829663715 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6554100670 0x7f6554197930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:11:34.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.471+0000 7f65537fe640 1 --2- 172.21.15.46:0/3829663715 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6554101860 0x7f6554197e70 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f654802c170 tx=0x7f6548005e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:11:34.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.471+0000 7f65517fa640 1 -- 172.21.15.46:0/3829663715 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6548043020 con 0x7f6554101860 2024-03-31T23:11:34.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.471+0000 7f65517fa640 1 -- 172.21.15.46:0/3829663715 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f65480043d0 con 0x7f6554101860 2024-03-31T23:11:34.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.471+0000 7f655a3c9640 1 -- 172.21.15.46:0/3829663715 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6554102a80 con 0x7f6554101860 2024-03-31T23:11:34.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.471+0000 7f65517fa640 1 -- 172.21.15.46:0/3829663715 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6548003b50 con 0x7f6554101860 2024-03-31T23:11:34.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.471+0000 7f655a3c9640 1 -- 172.21.15.46:0/3829663715 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6554102fa0 con 0x7f6554101860 2024-03-31T23:11:34.475 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.472+0000 7faf9493e640 1 -- 172.21.15.46:0/2450887156 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faf90102240 msgr2=0x7faf90108ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:34.475 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.472+0000 7faf9493e640 1 --2- 172.21.15.46:0/2450887156 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faf90102240 0x7faf90108ad0 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7faf88009ca0 tx=0x7faf8802c0f0 comp rx=0 tx=0).stop 2024-03-31T23:11:34.475 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.472+0000 7faf9493e640 1 -- 172.21.15.46:0/2450887156 shutdown_connections 2024-03-31T23:11:34.475 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.472+0000 7faf9493e640 1 --2- 172.21.15.46:0/2450887156 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faf90102240 0x7faf90108ad0 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:34.475 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.472+0000 7faf9493e640 1 --2- 172.21.15.46:0/2450887156 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7faf90101880 0x7faf90101d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:34.475 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.472+0000 7faf9493e640 1 --2- 172.21.15.46:0/2450887156 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7faf90100690 0x7faf90100ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:34.475 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.472+0000 7faf9493e640 1 -- 172.21.15.46:0/2450887156 >> 172.21.15.46:0/2450887156 conn(0x7faf900fbc00 msgr2=0x7faf900fe040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:11:34.475 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.472+0000 7faf9493e640 1 -- 172.21.15.46:0/2450887156 shutdown_connections 2024-03-31T23:11:34.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.472+0000 7faf9493e640 1 -- 172.21.15.46:0/2450887156 wait complete. 2024-03-31T23:11:34.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.473+0000 7f655a3c9640 1 -- 172.21.15.46:0/3829663715 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f65541031a0 con 0x7f6554101860 2024-03-31T23:11:34.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.473+0000 7faf9493e640 1 Processor -- start 2024-03-31T23:11:34.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.473+0000 7faf9493e640 1 -- start start 2024-03-31T23:11:34.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.473+0000 7faf9493e640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7faf90100690 0x7faf90197a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:34.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.473+0000 7faf9493e640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7faf90101880 0x7faf90197f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:34.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.473+0000 7faf9493e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faf90102240 0x7faf9019efa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:34.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.473+0000 7faf9493e640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7faf9010c1c0 con 0x7faf90102240 2024-03-31T23:11:34.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.473+0000 7faf9493e640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7faf9010c040 con 0x7faf90101880 2024-03-31T23:11:34.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.473+0000 7faf9493e640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7faf9010c340 con 0x7faf90100690 2024-03-31T23:11:34.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.473+0000 7faf8ed9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faf90102240 0x7faf9019efa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:34.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.473+0000 7faf8e59c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7faf90100690 0x7faf90197a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:34.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.473+0000 7faf8ed9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faf90102240 0x7faf9019efa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:48264/0 (socket says 172.21.15.46:48264) 2024-03-31T23:11:34.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.474+0000 7faf8ed9d640 1 -- 172.21.15.46:0/3997304858 learned_addr learned my addr 172.21.15.46:0/3997304858 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:11:34.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.474+0000 7faf8dd9b640 1 --2- 172.21.15.46:0/3997304858 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7faf90101880 0x7faf90197f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:34.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.474+0000 7faf8e59c640 1 -- 172.21.15.46:0/3997304858 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7faf90101880 msgr2=0x7faf90197f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:34.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.474+0000 7faf8e59c640 1 --2- 172.21.15.46:0/3997304858 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7faf90101880 0x7faf90197f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:34.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.474+0000 7faf8e59c640 1 -- 172.21.15.46:0/3997304858 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faf90102240 msgr2=0x7faf9019efa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:34.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.474+0000 7faf8e59c640 1 --2- 172.21.15.46:0/3997304858 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faf90102240 0x7faf9019efa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:34.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.474+0000 7faf8e59c640 1 -- 172.21.15.46:0/3997304858 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faf9019f610 con 0x7faf90100690 2024-03-31T23:11:34.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.474+0000 7faf8ed9d640 1 --2- 172.21.15.46:0/3997304858 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faf90102240 0x7faf9019efa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:11:34.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.474+0000 7faf8e59c640 1 --2- 172.21.15.46:0/3997304858 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7faf90100690 0x7faf90197a40 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7faf8000e9c0 tx=0x7faf8000ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:11:34.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.474+0000 7faf7f7fe640 1 -- 172.21.15.46:0/3997304858 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf8000ccc0 con 0x7faf90100690 2024-03-31T23:11:34.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.474+0000 7faf7f7fe640 1 -- 172.21.15.46:0/3997304858 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7faf800035f0 con 0x7faf90100690 2024-03-31T23:11:34.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.474+0000 7faf9493e640 1 -- 172.21.15.46:0/3997304858 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faf9019f900 con 0x7faf90100690 2024-03-31T23:11:34.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.474+0000 7faf7f7fe640 1 -- 172.21.15.46:0/3997304858 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf8000ce60 con 0x7faf90100690 2024-03-31T23:11:34.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.474+0000 7faf9493e640 1 -- 172.21.15.46:0/3997304858 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faf9019fe00 con 0x7faf90100690 2024-03-31T23:11:34.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.474+0000 7f65517fa640 1 -- 172.21.15.46:0/3829663715 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 18) ==== 137320+0+0 (secure 0 0 0) 0x7f6548046020 con 0x7f6554101860 2024-03-31T23:11:34.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.475+0000 7f65517fa640 1 --2- 172.21.15.46:0/3829663715 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f653c0a25c0 0x7f653c0a4a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:34.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.475+0000 7f6553fff640 1 --2- 172.21.15.46:0/3829663715 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f653c0a25c0 0x7f653c0a4a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:34.481 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.475+0000 7f65517fa640 1 -- 172.21.15.46:0/3829663715 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9208+0+0 (secure 0 0 0) 0x7f65480f12b0 con 0x7f6554101860 2024-03-31T23:11:34.481 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.476+0000 7f6553fff640 1 --2- 172.21.15.46:0/3829663715 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f653c0a25c0 0x7f653c0a4a80 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f65440058c0 tx=0x7f6544012040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:11:34.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.476+0000 7faf7f7fe640 1 -- 172.21.15.46:0/3997304858 <== mon.1 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 18) ==== 137320+0+0 (secure 0 0 0) 0x7faf800037a0 con 0x7faf90100690 2024-03-31T23:11:34.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.477+0000 7faf97601740 1 Processor -- start 2024-03-31T23:11:34.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.477+0000 7faf97601740 1 -- start start 2024-03-31T23:11:34.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.477+0000 7faf7f7fe640 1 --2- 172.21.15.46:0/3997304858 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7faf780a24a0 0x7faf780a4960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:34.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.477+0000 7faf97601740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b3b0f71a10 0x55b3b0f71e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:34.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.477+0000 7faf97601740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b3b0f72400 0x55b3b0f76f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:34.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.477+0000 7faf97601740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55b3b0f77570 0x55b3b0f79a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:34.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.477+0000 7faf97601740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55b3b0e730b0 con 0x55b3b0f72400 2024-03-31T23:11:34.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.477+0000 7faf97601740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55b3b0ed5ea0 con 0x55b3b0f77570 2024-03-31T23:11:34.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.477+0000 7faf97601740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55b3b0e7ec00 con 0x55b3b0f71a10 2024-03-31T23:11:34.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.477+0000 7faf8ed9d640 1 --2- 172.21.15.46:0/3997304858 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7faf780a24a0 0x7faf780a4960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:34.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.477+0000 7faf7f7fe640 1 -- 172.21.15.46:0/3997304858 <== mon.1 v2:172.21.15.162:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9208+0+0 (secure 0 0 0) 0x7faf800cee40 con 0x7faf90100690 2024-03-31T23:11:34.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.477+0000 7faf8dd9b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b3b0f72400 0x55b3b0f76f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:34.484 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.477+0000 7faf8e59c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b3b0f71a10 0x55b3b0f71e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:34.484 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.477+0000 7faf8dd9b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b3b0f72400 0x55b3b0f76f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:48274/0 (socket says 172.21.15.46:48274) 2024-03-31T23:11:34.484 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.477+0000 7faf8dd9b640 1 -- 172.21.15.46:0/3988560296 learned_addr learned my addr 172.21.15.46:0/3988560296 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:11:34.484 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.477+0000 7faf8ed9d640 1 --2- 172.21.15.46:0/3988560296 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55b3b0f77570 0x55b3b0f79a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:34.484 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.478+0000 7faf8dd9b640 1 -- 172.21.15.46:0/3988560296 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b3b0f71a10 msgr2=0x55b3b0f71e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:11:34.484 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.478+0000 7faf8dd9b640 1 --2- 172.21.15.46:0/3988560296 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b3b0f71a10 0x55b3b0f71e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:34.484 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.478+0000 7faf8dd9b640 1 -- 172.21.15.46:0/3988560296 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55b3b0f77570 msgr2=0x55b3b0f79a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:11:34.484 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.478+0000 7faf8dd9b640 1 --2- 172.21.15.46:0/3988560296 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55b3b0f77570 0x55b3b0f79a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:34.484 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.478+0000 7faf8dd9b640 1 -- 172.21.15.46:0/3988560296 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b3b0cb27e0 con 0x55b3b0f72400 2024-03-31T23:11:34.484 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.478+0000 7faf8e59c640 1 --2- 172.21.15.46:0/3988560296 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b3b0f71a10 0x55b3b0f71e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:11:34.484 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.478+0000 7faf8ed9d640 1 --2- 172.21.15.46:0/3997304858 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7faf780a24a0 0x7faf780a4960 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7faf88004af0 tx=0x7faf88009900 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:11:34.484 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.478+0000 7faf8dd9b640 1 --2- 172.21.15.46:0/3988560296 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b3b0f72400 0x55b3b0f76f00 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7faf84009c10 tx=0x7faf84013ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=59d46386a6543cc4 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:11:34.484 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.479+0000 7faf7cff9640 1 -- 172.21.15.46:0/3988560296 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf8400e990 con 0x55b3b0f72400 2024-03-31T23:11:34.484 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.479+0000 7faf7cff9640 1 -- 172.21.15.46:0/3988560296 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7faf8401c400 con 0x55b3b0f72400 2024-03-31T23:11:34.485 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.479+0000 7faf7cff9640 1 -- 172.21.15.46:0/3988560296 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf8401b610 con 0x55b3b0f72400 2024-03-31T23:11:34.485 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.480+0000 7f65517fa640 1 -- 172.21.15.46:0/3829663715 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f654804d020 con 0x7f6554101860 2024-03-31T23:11:34.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.480+0000 7faf97601740 1 -- 172.21.15.46:0/3988560296 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b3b0f72400 msgr2=0x55b3b0f76f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:34.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.480+0000 7faf97601740 1 --2- 172.21.15.46:0/3988560296 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b3b0f72400 0x55b3b0f76f00 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7faf84009c10 tx=0x7faf84013ba0 comp rx=0 tx=0).stop 2024-03-31T23:11:34.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.480+0000 7faf97601740 1 -- 172.21.15.46:0/3988560296 shutdown_connections 2024-03-31T23:11:34.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.480+0000 7faf97601740 1 --2- 172.21.15.46:0/3988560296 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55b3b0f77570 0x55b3b0f79a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:34.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.480+0000 7faf97601740 1 --2- 172.21.15.46:0/3988560296 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b3b0f72400 0x55b3b0f76f00 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:34.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.480+0000 7faf97601740 1 --2- 172.21.15.46:0/3988560296 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b3b0f71a10 0x55b3b0f71e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:34.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.480+0000 7faf97601740 1 -- 172.21.15.46:0/3988560296 >> 172.21.15.46:0/3988560296 conn(0x55b3b0f6cf80 msgr2=0x55b3b0f6d380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:11:34.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.480+0000 7faf97601740 1 -- 172.21.15.46:0/3988560296 shutdown_connections 2024-03-31T23:11:34.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.480+0000 7faf97601740 1 -- 172.21.15.46:0/3988560296 wait complete. 2024-03-31T23:11:34.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.481+0000 7faf97601740 1 -- start start 2024-03-31T23:11:34.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.481+0000 7faf97601740 1 Processor -- start 2024-03-31T23:11:34.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.481+0000 7faf97601740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55b3b0f71a10 0x55b3b0f7bd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:34.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.481+0000 7faf97601740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b3b0f72400 0x55b3b0f7c2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:34.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.481+0000 7faf97601740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b3b0f77570 0x55b3b0f83300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:34.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.481+0000 7faf97601740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55b3b0e743d0 con 0x55b3b0f72400 2024-03-31T23:11:34.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.481+0000 7faf97601740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55b3b0ed1f40 con 0x55b3b0f71a10 2024-03-31T23:11:34.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.481+0000 7faf97601740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55b3b0f7a520 con 0x55b3b0f77570 2024-03-31T23:11:34.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.481+0000 7faf97601740 10 client.0 _ll_register_callbacks cb 0x55b3b0e91b70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T23:11:34.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.481+0000 7faf8dd9b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b3b0f72400 0x55b3b0f7c2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:34.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.481+0000 7faf8dd9b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b3b0f72400 0x55b3b0f7c2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:48278/0 (socket says 172.21.15.46:48278) 2024-03-31T23:11:34.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.481+0000 7faf8dd9b640 1 -- 172.21.15.46:0/2714762600 learned_addr learned my addr 172.21.15.46:0/2714762600 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:11:34.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.481+0000 7faf8e59c640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55b3b0f71a10 0x55b3b0f7bd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:34.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.481+0000 7faf8ed9d640 1 --2- 172.21.15.46:0/2714762600 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b3b0f77570 0x55b3b0f83300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:34.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.481+0000 7faf8dd9b640 1 -- 172.21.15.46:0/2714762600 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b3b0f77570 msgr2=0x55b3b0f83300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:11:34.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.481+0000 7faf8dd9b640 1 --2- 172.21.15.46:0/2714762600 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b3b0f77570 0x55b3b0f83300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:34.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.481+0000 7faf8dd9b640 1 -- 172.21.15.46:0/2714762600 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55b3b0f71a10 msgr2=0x55b3b0f7bd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:11:34.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.481+0000 7faf8dd9b640 1 --2- 172.21.15.46:0/2714762600 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55b3b0f71a10 0x55b3b0f7bd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:34.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.481+0000 7faf8dd9b640 1 -- 172.21.15.46:0/2714762600 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b3b0d12f70 con 0x55b3b0f72400 2024-03-31T23:11:34.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.481+0000 7faf8ed9d640 1 --2- 172.21.15.46:0/2714762600 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b3b0f77570 0x55b3b0f83300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:11:34.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.481+0000 7faf8e59c640 1 --2- 172.21.15.46:0/2714762600 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55b3b0f71a10 0x55b3b0f7bd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:11:34.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.482+0000 7faf8dd9b640 1 --2- 172.21.15.46:0/2714762600 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b3b0f72400 0x55b3b0f7c2c0 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7faf8401d000 tx=0x7faf84005d30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=24c302caeacd170a server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:11:34.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.482+0000 7faf75ffb640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-31T23:11:34.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.482+0000 7faf75ffb640 1 -- 172.21.15.46:0/2714762600 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf84029030 con 0x55b3b0f72400 2024-03-31T23:11:34.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.482+0000 7faf97601740 1 -- 172.21.15.46:0/2714762600 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7faf8401c400 con 0x55b3b0f72400 2024-03-31T23:11:34.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.482+0000 7faf75ffb640 1 -- 172.21.15.46:0/2714762600 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9208+0+0 (secure 0 0 0) 0x7faf84019070 con 0x55b3b0f72400 2024-03-31T23:11:34.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.482+0000 7faf75ffb640 1 -- 172.21.15.46:0/2714762600 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7faf84033050 con 0x55b3b0f72400 2024-03-31T23:11:34.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.482+0000 7faf97601740 10 client.15612 fetch_fsmap learned FSMap version 25 2024-03-31T23:11:34.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.482+0000 7faf97601740 1 -- 172.21.15.46:0/2714762600 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55b3b0baa170 con 0x55b3b0f72400 2024-03-31T23:11:34.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.482+0000 7faf75ffb640 1 -- 172.21.15.46:0/2714762600 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7faf8401c400 con 0x55b3b0f72400 2024-03-31T23:11:34.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.482+0000 7faf75ffb640 1 -- 172.21.15.46:0/2714762600 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7faf840046c0 con 0x55b3b0f72400 2024-03-31T23:11:34.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.482+0000 7faf97601740 10 client.15612 fetch_fsmap finished waiting for FSMap version 25 2024-03-31T23:11:34.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.482+0000 7faf97601740 10 client.15612 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-31T23:11:34.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.482+0000 7faf97601740 20 client.15612 populate_metadata read hostname 'smithi046' 2024-03-31T23:11:34.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.482+0000 7faf97601740 1 --2- 172.21.15.46:0/2714762600 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55b3b0f87530 0x55b3b0f89920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:34.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.482+0000 7faf97601740 4 client.15612 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-31T23:11:34.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.482+0000 7faf97601740 1 -- 172.21.15.46:0/2714762600 --> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b3b0ee33e0 con 0x55b3b0f87530 2024-03-31T23:11:34.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.483+0000 7faf8e59c640 1 --2- 172.21.15.46:0/2714762600 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55b3b0f87530 0x55b3b0f89920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:34.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.483+0000 7faf8e59c640 1 --2- 172.21.15.46:0/2714762600 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55b3b0f87530 0x55b3b0f89920 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=51f6c7a2f97f78ca server_cookie=9b21e7dcb175878e in_seq=0 out_seq=0 2024-03-31T23:11:34.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.483+0000 7faf75ffb640 10 client.15612 ms_handle_connect on v2:172.21.15.142:6838/2519143918 2024-03-31T23:11:34.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.484+0000 7faf75ffb640 1 -- 172.21.15.46:0/2714762600 <== mds.0 v2:172.21.15.142:6838/2519143918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55b3b0ee33e0 con 0x55b3b0f87530 2024-03-31T23:11:34.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.484+0000 7faf75ffb640 10 client.15612 handle_command_reply: tid=0 2024-03-31T23:11:34.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.484+0000 7faf97601740 1 client.15612 shutdown 2024-03-31T23:11:34.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.484+0000 7faf97601740 1 -- 172.21.15.46:0/2714762600 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b3b0f72400 msgr2=0x55b3b0f7c2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:34.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.484+0000 7faf97601740 1 --2- 172.21.15.46:0/2714762600 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b3b0f72400 0x55b3b0f7c2c0 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7faf8401d000 tx=0x7faf84005d30 comp rx=0 tx=0).stop 2024-03-31T23:11:34.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.484+0000 7faf97601740 1 -- 172.21.15.46:0/2714762600 shutdown_connections 2024-03-31T23:11:34.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.484+0000 7faf97601740 1 --2- 172.21.15.46:0/2714762600 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55b3b0f87530 0x55b3b0f89920 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:34.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.484+0000 7faf97601740 1 --2- 172.21.15.46:0/2714762600 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b3b0f77570 0x55b3b0f83300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:34.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.484+0000 7faf97601740 1 --2- 172.21.15.46:0/2714762600 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b3b0f72400 0x55b3b0f7c2c0 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:34.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.484+0000 7faf75ffb640 0 client.15612 ms_handle_reset on v2:172.21.15.142:6838/2519143918 2024-03-31T23:11:34.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.484+0000 7faf97601740 1 --2- 172.21.15.46:0/2714762600 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55b3b0f71a10 0x55b3b0f7bd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:34.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.484+0000 7faf97601740 1 -- 172.21.15.46:0/2714762600 >> 172.21.15.46:0/2714762600 conn(0x55b3b0f6cf80 msgr2=0x55b3b0f6c180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:11:34.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.484+0000 7faf97601740 1 -- 172.21.15.46:0/2714762600 shutdown_connections 2024-03-31T23:11:34.494 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.484+0000 7faf97601740 1 -- 172.21.15.46:0/2714762600 wait complete. 2024-03-31T23:11:34.494 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.484+0000 7faf97601740 20 client.15612 trim_cache size 0 max 16384 2024-03-31T23:11:34.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.502+0000 7faf97601740 1 Processor -- start 2024-03-31T23:11:34.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.502+0000 7faf97601740 1 -- start start 2024-03-31T23:11:34.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.502+0000 7faf97601740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55b3b0f71a10 0x55b3b0f708f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:34.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.502+0000 7faf97601740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b3b0f72400 0x55b3b0f832c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:34.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.502+0000 7faf97601740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b3b0f77570 0x55b3b0f8e1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:34.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.502+0000 7faf97601740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7faf901aab00 con 0x55b3b0f77570 2024-03-31T23:11:34.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.502+0000 7faf97601740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7faf901a83d0 con 0x55b3b0f71a10 2024-03-31T23:11:34.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.502+0000 7faf97601740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7faf901a6220 con 0x55b3b0f72400 2024-03-31T23:11:34.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.503+0000 7faf8ed9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b3b0f77570 0x55b3b0f8e1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:34.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.503+0000 7faf8ed9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b3b0f77570 0x55b3b0f8e1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:48282/0 (socket says 172.21.15.46:48282) 2024-03-31T23:11:34.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.503+0000 7faf8ed9d640 1 -- 172.21.15.46:0/3221167261 learned_addr learned my addr 172.21.15.46:0/3221167261 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:11:34.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.503+0000 7faf8e59c640 1 --2- 172.21.15.46:0/3221167261 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55b3b0f71a10 0x55b3b0f708f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:34.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.503+0000 7faf8dd9b640 1 --2- 172.21.15.46:0/3221167261 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b3b0f72400 0x55b3b0f832c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:34.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.503+0000 7faf8ed9d640 1 -- 172.21.15.46:0/3221167261 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b3b0f72400 msgr2=0x55b3b0f832c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:11:34.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.503+0000 7faf8ed9d640 1 --2- 172.21.15.46:0/3221167261 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b3b0f72400 0x55b3b0f832c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:34.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.503+0000 7faf8ed9d640 1 -- 172.21.15.46:0/3221167261 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55b3b0f71a10 msgr2=0x55b3b0f708f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:11:34.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.503+0000 7faf8ed9d640 1 --2- 172.21.15.46:0/3221167261 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55b3b0f71a10 0x55b3b0f708f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:34.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.503+0000 7faf8ed9d640 1 -- 172.21.15.46:0/3221167261 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b3b0f89f50 con 0x55b3b0f77570 2024-03-31T23:11:34.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.503+0000 7faf8dd9b640 1 --2- 172.21.15.46:0/3221167261 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b3b0f72400 0x55b3b0f832c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:11:34.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.503+0000 7faf8ed9d640 1 --2- 172.21.15.46:0/3221167261 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b3b0f77570 0x55b3b0f8e1b0 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7faf88009dc0 tx=0x7faf8802e050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=88d056c128a02eab server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:11:34.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.503+0000 7faf77fff640 1 -- 172.21.15.46:0/3221167261 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf88039040 con 0x55b3b0f77570 2024-03-31T23:11:34.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.503+0000 7faf77fff640 1 -- 172.21.15.46:0/3221167261 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7faf88053070 con 0x55b3b0f77570 2024-03-31T23:11:34.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.503+0000 7faf77fff640 1 -- 172.21.15.46:0/3221167261 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf8804ed90 con 0x55b3b0f77570 2024-03-31T23:11:34.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.504+0000 7faf97601740 1 -- 172.21.15.46:0/3221167261 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b3b0f77570 msgr2=0x55b3b0f8e1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:34.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.504+0000 7faf97601740 1 --2- 172.21.15.46:0/3221167261 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b3b0f77570 0x55b3b0f8e1b0 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7faf88009dc0 tx=0x7faf8802e050 comp rx=0 tx=0).stop 2024-03-31T23:11:34.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.504+0000 7faf97601740 1 -- 172.21.15.46:0/3221167261 shutdown_connections 2024-03-31T23:11:34.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.504+0000 7faf97601740 1 --2- 172.21.15.46:0/3221167261 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b3b0f77570 0x55b3b0f8e1b0 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:34.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.504+0000 7faf97601740 1 --2- 172.21.15.46:0/3221167261 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b3b0f72400 0x55b3b0f832c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:34.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.504+0000 7faf97601740 1 --2- 172.21.15.46:0/3221167261 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55b3b0f71a10 0x55b3b0f708f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:34.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.504+0000 7faf97601740 1 -- 172.21.15.46:0/3221167261 >> 172.21.15.46:0/3221167261 conn(0x55b3b0f6cf80 msgr2=0x55b3b0f701e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:11:34.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.504+0000 7faf97601740 1 -- 172.21.15.46:0/3221167261 shutdown_connections 2024-03-31T23:11:34.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.504+0000 7faf97601740 1 -- 172.21.15.46:0/3221167261 wait complete. 2024-03-31T23:11:34.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.504+0000 7faf97601740 1 -- start start 2024-03-31T23:11:34.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.504+0000 7faf97601740 1 Processor -- start 2024-03-31T23:11:34.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.505+0000 7faf97601740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55b3b0f71a10 0x55b3b0f81240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:34.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.505+0000 7faf97601740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b3b0f72400 0x55b3b0f81780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:34.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.505+0000 7faf97601740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b3b0f77570 0x55b3b0f82790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:34.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.505+0000 7faf97601740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7faf901a6890 con 0x55b3b0f72400 2024-03-31T23:11:34.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.505+0000 7faf97601740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7faf901a7d00 con 0x55b3b0f71a10 2024-03-31T23:11:34.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.505+0000 7faf97601740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7faf901a6f60 con 0x55b3b0f77570 2024-03-31T23:11:34.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.505+0000 7faf97601740 10 client.0 _ll_register_callbacks cb 0x7faf84009b10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T23:11:34.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.505+0000 7faf8dd9b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b3b0f72400 0x55b3b0f81780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:34.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.505+0000 7faf8dd9b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b3b0f72400 0x55b3b0f81780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:48298/0 (socket says 172.21.15.46:48298) 2024-03-31T23:11:34.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.505+0000 7faf8dd9b640 1 -- 172.21.15.46:0/1675765565 learned_addr learned my addr 172.21.15.46:0/1675765565 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:11:34.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.505+0000 7faf8e59c640 1 --2- 172.21.15.46:0/1675765565 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55b3b0f71a10 0x55b3b0f81240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:34.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.505+0000 7faf8ed9d640 1 --2- 172.21.15.46:0/1675765565 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b3b0f77570 0x55b3b0f82790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:34.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.505+0000 7faf8dd9b640 1 -- 172.21.15.46:0/1675765565 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b3b0f77570 msgr2=0x55b3b0f82790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:11:34.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.505+0000 7faf8dd9b640 1 --2- 172.21.15.46:0/1675765565 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b3b0f77570 0x55b3b0f82790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:34.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.505+0000 7faf8dd9b640 1 -- 172.21.15.46:0/1675765565 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55b3b0f71a10 msgr2=0x55b3b0f81240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:11:34.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.505+0000 7faf8dd9b640 1 --2- 172.21.15.46:0/1675765565 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55b3b0f71a10 0x55b3b0f81240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:34.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.505+0000 7faf8dd9b640 1 -- 172.21.15.46:0/1675765565 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b3b0f82d80 con 0x55b3b0f72400 2024-03-31T23:11:34.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.505+0000 7faf8dd9b640 1 --2- 172.21.15.46:0/1675765565 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b3b0f72400 0x55b3b0f81780 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7faf84001eb0 tx=0x7faf84009e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c88a138aa013196 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:11:34.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.505+0000 7faf75ffb640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-31T23:11:34.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.505+0000 7faf75ffb640 1 -- 172.21.15.46:0/1675765565 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf84029040 con 0x55b3b0f72400 2024-03-31T23:11:34.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.505+0000 7faf75ffb640 1 -- 172.21.15.46:0/1675765565 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9208+0+0 (secure 0 0 0) 0x7faf84022040 con 0x55b3b0f72400 2024-03-31T23:11:34.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.505+0000 7faf97601740 1 -- 172.21.15.46:0/1675765565 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7faf88053070 con 0x55b3b0f72400 2024-03-31T23:11:34.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.506+0000 7faf75ffb640 1 -- 172.21.15.46:0/1675765565 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7faf84045050 con 0x55b3b0f72400 2024-03-31T23:11:34.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.506+0000 7faf97601740 10 client.15618 fetch_fsmap learned FSMap version 25 2024-03-31T23:11:34.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.506+0000 7faf97601740 1 -- 172.21.15.46:0/1675765565 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55b3b0fee010 con 0x55b3b0f72400 2024-03-31T23:11:34.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.506+0000 7faf75ffb640 1 -- 172.21.15.46:0/1675765565 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7faf88053070 con 0x55b3b0f72400 2024-03-31T23:11:34.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.506+0000 7faf75ffb640 1 -- 172.21.15.46:0/1675765565 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7faf84005040 con 0x55b3b0f72400 2024-03-31T23:11:34.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.506+0000 7faf97601740 10 client.15618 fetch_fsmap finished waiting for FSMap version 25 2024-03-31T23:11:34.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.506+0000 7faf97601740 10 client.15618 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-31T23:11:34.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.506+0000 7faf97601740 20 client.15618 populate_metadata read hostname 'smithi046' 2024-03-31T23:11:34.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.506+0000 7faf97601740 1 --2- 172.21.15.46:0/1675765565 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55b3b0feea30 0x55b3b0ff0e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:34.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.506+0000 7faf97601740 4 client.15618 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-31T23:11:34.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.506+0000 7faf97601740 1 -- 172.21.15.46:0/1675765565 --> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55b3b0edd730 con 0x55b3b0feea30 2024-03-31T23:11:34.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.506+0000 7faf8e59c640 1 --2- 172.21.15.46:0/1675765565 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55b3b0feea30 0x55b3b0ff0e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:34.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.506+0000 7faf8e59c640 1 --2- 172.21.15.46:0/1675765565 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55b3b0feea30 0x55b3b0ff0e50 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cdad727fb5bdbad1 server_cookie=ee3bed2487b0008e in_seq=0 out_seq=0 2024-03-31T23:11:34.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.506+0000 7faf75ffb640 10 client.15618 ms_handle_connect on v2:172.21.15.142:6838/2519143918 2024-03-31T23:11:34.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.507+0000 7faf75ffb640 1 -- 172.21.15.46:0/1675765565 <== mds.0 v2:172.21.15.142:6838/2519143918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55b3b0edd730 con 0x55b3b0feea30 2024-03-31T23:11:34.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.507+0000 7faf75ffb640 10 client.15618 handle_command_reply: tid=0 2024-03-31T23:11:34.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.507+0000 7faf97601740 1 client.15618 shutdown 2024-03-31T23:11:34.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.508+0000 7faf97601740 1 -- 172.21.15.46:0/1675765565 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b3b0f72400 msgr2=0x55b3b0f81780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:34.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.508+0000 7faf97601740 1 --2- 172.21.15.46:0/1675765565 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b3b0f72400 0x55b3b0f81780 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7faf84001eb0 tx=0x7faf84009e90 comp rx=0 tx=0).stop 2024-03-31T23:11:34.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.508+0000 7faf97601740 1 -- 172.21.15.46:0/1675765565 shutdown_connections 2024-03-31T23:11:34.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.508+0000 7faf97601740 1 --2- 172.21.15.46:0/1675765565 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55b3b0feea30 0x55b3b0ff0e50 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:34.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.508+0000 7faf97601740 1 --2- 172.21.15.46:0/1675765565 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b3b0f77570 0x55b3b0f82790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:34.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.508+0000 7faf97601740 1 --2- 172.21.15.46:0/1675765565 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b3b0f72400 0x55b3b0f81780 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:34.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.508+0000 7faf75ffb640 0 client.15618 ms_handle_reset on v2:172.21.15.142:6838/2519143918 2024-03-31T23:11:34.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.508+0000 7faf97601740 1 --2- 172.21.15.46:0/1675765565 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55b3b0f71a10 0x55b3b0f81240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:34.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.508+0000 7faf97601740 1 -- 172.21.15.46:0/1675765565 >> 172.21.15.46:0/1675765565 conn(0x55b3b0f6cf80 msgr2=0x55b3b0f70a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:11:34.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.508+0000 7faf97601740 1 -- 172.21.15.46:0/1675765565 shutdown_connections 2024-03-31T23:11:34.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.508+0000 7faf97601740 1 -- 172.21.15.46:0/1675765565 wait complete. 2024-03-31T23:11:34.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.508+0000 7faf97601740 20 client.15618 trim_cache size 0 max 16384 2024-03-31T23:11:34.514 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-03-31T23:11:34.514 INFO:teuthology.orchestra.run.smithi046.stdout: "return_code": 0, 2024-03-31T23:11:34.514 INFO:teuthology.orchestra.run.smithi046.stdout: "scrub_tag": "22d11db0-ed6a-4208-aaa3-8ce32084e80d", 2024-03-31T23:11:34.514 INFO:teuthology.orchestra.run.smithi046.stdout: "mode": "asynchronous" 2024-03-31T23:11:34.514 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-03-31T23:11:34.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.508+0000 7faf9493e640 1 -- 172.21.15.46:0/3997304858 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7faf780a24a0 msgr2=0x7faf780a4960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:34.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.508+0000 7faf9493e640 1 --2- 172.21.15.46:0/3997304858 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7faf780a24a0 0x7faf780a4960 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7faf88004af0 tx=0x7faf88009900 comp rx=0 tx=0).stop 2024-03-31T23:11:34.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.508+0000 7faf9493e640 1 -- 172.21.15.46:0/3997304858 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7faf90100690 msgr2=0x7faf90197a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:34.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.508+0000 7faf9493e640 1 --2- 172.21.15.46:0/3997304858 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7faf90100690 0x7faf90197a40 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7faf8000e9c0 tx=0x7faf8000ee80 comp rx=0 tx=0).stop 2024-03-31T23:11:34.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.508+0000 7faf9493e640 1 -- 172.21.15.46:0/3997304858 shutdown_connections 2024-03-31T23:11:34.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.508+0000 7faf9493e640 1 --2- 172.21.15.46:0/3997304858 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7faf780a24a0 0x7faf780a4960 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:34.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.508+0000 7faf9493e640 1 --2- 172.21.15.46:0/3997304858 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faf90102240 0x7faf9019efa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:34.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.508+0000 7faf9493e640 1 --2- 172.21.15.46:0/3997304858 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7faf90101880 0x7faf90197f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:34.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.508+0000 7faf9493e640 1 --2- 172.21.15.46:0/3997304858 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7faf90100690 0x7faf90197a40 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:34.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.508+0000 7faf9493e640 1 -- 172.21.15.46:0/3997304858 >> 172.21.15.46:0/3997304858 conn(0x7faf900fbc00 msgr2=0x7faf90069a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:11:34.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.509+0000 7faf9493e640 1 -- 172.21.15.46:0/3997304858 shutdown_connections 2024-03-31T23:11:34.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.509+0000 7faf9493e640 1 -- 172.21.15.46:0/3997304858 wait complete. 2024-03-31T23:11:34.520 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-31T23:11:34.520 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-31T23:11:34.611 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:11:34 smithi046 ceph-mon[32498]: pgmap v367: 129 pgs: 129 active+clean; 648 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 1.1 KiB/s wr, 6 op/s 2024-03-31T23:11:34.612 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:11:34 smithi046 ceph-mon[32498]: from='client.? 172.21.15.46:0/268846486' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-31T23:11:34.612 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:11:34 smithi046 ceph-mon[32498]: from='client.? 172.21.15.46:0/339498258' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-31T23:11:34.612 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:11:34 smithi046 ceph-mon[32498]: from='client.? 172.21.15.46:0/3057859208' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-31T23:11:34.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.628+0000 7f4bd2935640 1 -- 172.21.15.46:0/2457598453 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4bcc1021d0 msgr2=0x7f4bcc1045c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:34.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.628+0000 7f4bd2935640 1 --2- 172.21.15.46:0/2457598453 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4bcc1021d0 0x7f4bcc1045c0 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f4bbc009900 tx=0x7f4bbc02bc30 comp rx=0 tx=0).stop 2024-03-31T23:11:34.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.628+0000 7f4bd2935640 1 -- 172.21.15.46:0/2457598453 shutdown_connections 2024-03-31T23:11:34.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.628+0000 7f4bd2935640 1 --2- 172.21.15.46:0/2457598453 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4bcc107430 0x7f4bcc109880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:34.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.628+0000 7f4bd2935640 1 --2- 172.21.15.46:0/2457598453 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4bcc104b00 0x7f4bcc106ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:34.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.628+0000 7f4bd2935640 1 --2- 172.21.15.46:0/2457598453 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4bcc1021d0 0x7f4bcc1045c0 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:34.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.628+0000 7f4bd2935640 1 -- 172.21.15.46:0/2457598453 >> 172.21.15.46:0/2457598453 conn(0x7f4bcc0fbbe0 msgr2=0x7f4bcc0fe000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:11:34.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.628+0000 7f4bd2935640 1 -- 172.21.15.46:0/2457598453 shutdown_connections 2024-03-31T23:11:34.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.628+0000 7f4bd2935640 1 -- 172.21.15.46:0/2457598453 wait complete. 2024-03-31T23:11:34.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.628+0000 7f4bd2935640 1 Processor -- start 2024-03-31T23:11:34.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.628+0000 7f4bd2935640 1 -- start start 2024-03-31T23:11:34.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.629+0000 7f4bd2935640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4bcc1021d0 0x7f4bcc199c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:34.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.629+0000 7f4bd2935640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4bcc104b00 0x7f4bcc19a180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:34.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.629+0000 7f4bd2935640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4bcc107430 0x7f4bcc1a11b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:34.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.629+0000 7f4bd2935640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f4bcc10d2a0 con 0x7f4bcc104b00 2024-03-31T23:11:34.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.629+0000 7f4bd2935640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f4bcc10d120 con 0x7f4bcc1021d0 2024-03-31T23:11:34.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.629+0000 7f4bd2935640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f4bcc10d420 con 0x7f4bcc107430 2024-03-31T23:11:34.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.629+0000 7f4bcb7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4bcc104b00 0x7f4bcc19a180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:34.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.629+0000 7f4bcb7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4bcc104b00 0x7f4bcc19a180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:48316/0 (socket says 172.21.15.46:48316) 2024-03-31T23:11:34.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.629+0000 7f4bcb7fe640 1 -- 172.21.15.46:0/2897013436 learned_addr learned my addr 172.21.15.46:0/2897013436 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:11:34.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.629+0000 7f4bd0ed2640 1 --2- 172.21.15.46:0/2897013436 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4bcc107430 0x7f4bcc1a11b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:34.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.629+0000 7f4bcbfff640 1 --2- 172.21.15.46:0/2897013436 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4bcc1021d0 0x7f4bcc199c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:34.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.629+0000 7f4bcb7fe640 1 -- 172.21.15.46:0/2897013436 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4bcc107430 msgr2=0x7f4bcc1a11b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:34.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.629+0000 7f4bcb7fe640 1 --2- 172.21.15.46:0/2897013436 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4bcc107430 0x7f4bcc1a11b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:34.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.629+0000 7f4bcb7fe640 1 -- 172.21.15.46:0/2897013436 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4bcc1021d0 msgr2=0x7f4bcc199c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:34.632 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.629+0000 7f4bcb7fe640 1 --2- 172.21.15.46:0/2897013436 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4bcc1021d0 0x7f4bcc199c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:34.632 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.629+0000 7f4bcb7fe640 1 -- 172.21.15.46:0/2897013436 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4bcc1a1850 con 0x7f4bcc104b00 2024-03-31T23:11:34.632 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.629+0000 7f4bcbfff640 1 --2- 172.21.15.46:0/2897013436 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4bcc1021d0 0x7f4bcc199c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:11:34.632 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.629+0000 7f4bd0ed2640 1 --2- 172.21.15.46:0/2897013436 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4bcc107430 0x7f4bcc1a11b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:11:34.632 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.629+0000 7f4bcb7fe640 1 --2- 172.21.15.46:0/2897013436 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4bcc104b00 0x7f4bcc19a180 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f4bb400e9c0 tx=0x7f4bb400ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:11:34.632 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.629+0000 7f4bc97fa640 1 -- 172.21.15.46:0/2897013436 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bb400cd80 con 0x7f4bcc104b00 2024-03-31T23:11:34.632 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.630+0000 7f4bd2935640 1 -- 172.21.15.46:0/2897013436 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4bcc1a1b40 con 0x7f4bcc104b00 2024-03-31T23:11:34.632 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.630+0000 7f4bc97fa640 1 -- 172.21.15.46:0/2897013436 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4bb40054d0 con 0x7f4bcc104b00 2024-03-31T23:11:34.632 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.630+0000 7f4bc97fa640 1 -- 172.21.15.46:0/2897013436 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bb4010430 con 0x7f4bcc104b00 2024-03-31T23:11:34.632 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.630+0000 7f4bd2935640 1 -- 172.21.15.46:0/2897013436 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4bcc1a20e0 con 0x7f4bcc104b00 2024-03-31T23:11:34.638 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.632+0000 7f4bc97fa640 1 -- 172.21.15.46:0/2897013436 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 18) ==== 137320+0+0 (secure 0 0 0) 0x7f4bb4004fd0 con 0x7f4bcc104b00 2024-03-31T23:11:34.638 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.632+0000 7f4bd55f0740 1 Processor -- start 2024-03-31T23:11:34.638 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.632+0000 7f4bd55f0740 1 -- start start 2024-03-31T23:11:34.638 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.632+0000 7f4bd55f0740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5616fb5072b0 0x5616fb5076d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:34.638 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.632+0000 7f4bd55f0740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5616fb507ca0 0x5616fb50c7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:34.638 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.632+0000 7f4bd55f0740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5616fb50ce10 0x5616fb50f2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:34.638 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.632+0000 7f4bd55f0740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x5616fb46c180 con 0x5616fb5072b0 2024-03-31T23:11:34.639 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.632+0000 7f4bd55f0740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x5616fb414b70 con 0x5616fb507ca0 2024-03-31T23:11:34.639 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.632+0000 7f4bd55f0740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5616fb468220 con 0x5616fb50ce10 2024-03-31T23:11:34.639 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.632+0000 7f4bd0ed2640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5616fb5072b0 0x5616fb5076d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:34.639 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.632+0000 7f4bd0ed2640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5616fb5072b0 0x5616fb5076d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:48322/0 (socket says 172.21.15.46:48322) 2024-03-31T23:11:34.639 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.632+0000 7f4bd0ed2640 1 -- 172.21.15.46:0/2642959471 learned_addr learned my addr 172.21.15.46:0/2642959471 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:11:34.639 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.632+0000 7f4bcbfff640 1 --2- 172.21.15.46:0/2642959471 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5616fb507ca0 0x5616fb50c7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:34.639 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.632+0000 7f4bcb7fe640 1 --2- 172.21.15.46:0/2642959471 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5616fb50ce10 0x5616fb50f2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:34.639 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.633+0000 7f4bd0ed2640 1 -- 172.21.15.46:0/2642959471 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5616fb50ce10 msgr2=0x5616fb50f2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:11:34.639 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.633+0000 7f4bd0ed2640 1 --2- 172.21.15.46:0/2642959471 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5616fb50ce10 0x5616fb50f2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:34.639 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.633+0000 7f4bd0ed2640 1 -- 172.21.15.46:0/2642959471 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5616fb507ca0 msgr2=0x5616fb50c7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:11:34.639 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.633+0000 7f4bd0ed2640 1 --2- 172.21.15.46:0/2642959471 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5616fb507ca0 0x5616fb50c7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:34.639 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.633+0000 7f4bd0ed2640 1 -- 172.21.15.46:0/2642959471 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5616fb4287c0 con 0x5616fb5072b0 2024-03-31T23:11:34.639 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.633+0000 7f4bc97fa640 1 --2- 172.21.15.46:0/2897013436 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f4bb80a24a0 0x7f4bb80a4960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:34.639 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.633+0000 7f4bd0ed2640 1 --2- 172.21.15.46:0/2642959471 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5616fb5072b0 0x5616fb5076d0 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f4bc40117e0 tx=0x7f4bc4011ca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ddb23fee4c40fa56 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:11:34.640 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.633+0000 7f4bcbfff640 1 --2- 172.21.15.46:0/2897013436 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f4bb80a24a0 0x7f4bb80a4960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:34.640 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.633+0000 7f4bc97fa640 1 -- 172.21.15.46:0/2897013436 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9208+0+0 (secure 0 0 0) 0x7f4bb40cef70 con 0x7f4bcc104b00 2024-03-31T23:11:34.640 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.633+0000 7f4bc27fc640 1 -- 172.21.15.46:0/2642959471 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bc4011310 con 0x5616fb5072b0 2024-03-31T23:11:34.640 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.633+0000 7f4bc27fc640 1 -- 172.21.15.46:0/2642959471 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4bc4003e20 con 0x5616fb5072b0 2024-03-31T23:11:34.640 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.633+0000 7f4bc27fc640 1 -- 172.21.15.46:0/2642959471 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bc4004490 con 0x5616fb5072b0 2024-03-31T23:11:34.640 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.634+0000 7f4bd55f0740 1 -- 172.21.15.46:0/2642959471 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5616fb5072b0 msgr2=0x5616fb5076d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:34.640 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.634+0000 7f4bd55f0740 1 --2- 172.21.15.46:0/2642959471 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5616fb5072b0 0x5616fb5076d0 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f4bc40117e0 tx=0x7f4bc4011ca0 comp rx=0 tx=0).stop 2024-03-31T23:11:34.640 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.634+0000 7f4bcbfff640 1 --2- 172.21.15.46:0/2897013436 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f4bb80a24a0 0x7f4bb80a4960 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f4bcc19b110 tx=0x7f4bbc005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:11:34.640 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.634+0000 7f4bd55f0740 1 -- 172.21.15.46:0/2642959471 shutdown_connections 2024-03-31T23:11:34.640 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.634+0000 7f4bd55f0740 1 --2- 172.21.15.46:0/2642959471 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5616fb50ce10 0x5616fb50f2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:34.640 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.634+0000 7f4bd55f0740 1 --2- 172.21.15.46:0/2642959471 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5616fb507ca0 0x5616fb50c7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:34.640 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.634+0000 7f4bd55f0740 1 --2- 172.21.15.46:0/2642959471 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5616fb5072b0 0x5616fb5076d0 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:34.640 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.634+0000 7f4bd55f0740 1 -- 172.21.15.46:0/2642959471 >> 172.21.15.46:0/2642959471 conn(0x5616fb502820 msgr2=0x5616fb502c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:11:34.640 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.634+0000 7f4bd55f0740 1 -- 172.21.15.46:0/2642959471 shutdown_connections 2024-03-31T23:11:34.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.634+0000 7f4bd55f0740 1 -- 172.21.15.46:0/2642959471 wait complete. 2024-03-31T23:11:34.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.634+0000 7f4bd55f0740 1 -- start start 2024-03-31T23:11:34.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.634+0000 7f4bd55f0740 1 Processor -- start 2024-03-31T23:11:34.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.635+0000 7f4bd55f0740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5616fb5072b0 0x5616fb5117e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:34.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.635+0000 7f4bd55f0740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5616fb507ca0 0x5616fb511d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:34.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.635+0000 7f4bd55f0740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5616fb50ce10 0x5616fb518d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:34.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.635+0000 7f4bd55f0740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x5616fb50fdc0 con 0x5616fb5072b0 2024-03-31T23:11:34.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.635+0000 7f4bd55f0740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x5616fb40a4a0 con 0x5616fb50ce10 2024-03-31T23:11:34.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.635+0000 7f4bd55f0740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5616fb50ff40 con 0x5616fb507ca0 2024-03-31T23:11:34.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.635+0000 7f4bd55f0740 10 client.0 _ll_register_callbacks cb 0x5616fb40dd90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T23:11:34.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.635+0000 7f4bcbfff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5616fb5072b0 0x5616fb5117e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:34.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.635+0000 7f4bcbfff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5616fb5072b0 0x5616fb5117e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:48328/0 (socket says 172.21.15.46:48328) 2024-03-31T23:11:34.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.635+0000 7f4bcbfff640 1 -- 172.21.15.46:0/2886502195 learned_addr learned my addr 172.21.15.46:0/2886502195 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:11:34.642 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.635+0000 7f4bd0ed2640 1 --2- 172.21.15.46:0/2886502195 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5616fb50ce10 0x5616fb518d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:34.642 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.635+0000 7f4bcb7fe640 1 --2- 172.21.15.46:0/2886502195 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5616fb507ca0 0x5616fb511d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:34.642 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.635+0000 7f4bcbfff640 1 -- 172.21.15.46:0/2886502195 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5616fb507ca0 msgr2=0x5616fb511d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:11:34.642 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.635+0000 7f4bcbfff640 1 --2- 172.21.15.46:0/2886502195 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5616fb507ca0 0x5616fb511d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:34.642 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.635+0000 7f4bcbfff640 1 -- 172.21.15.46:0/2886502195 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5616fb50ce10 msgr2=0x5616fb518d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:11:34.642 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.635+0000 7f4bcbfff640 1 --2- 172.21.15.46:0/2886502195 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5616fb50ce10 0x5616fb518d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:34.642 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.635+0000 7f4bcbfff640 1 -- 172.21.15.46:0/2886502195 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5616fb24a7d0 con 0x5616fb5072b0 2024-03-31T23:11:34.642 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.635+0000 7f4bcbfff640 1 --2- 172.21.15.46:0/2886502195 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5616fb5072b0 0x5616fb5117e0 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f4bbc02c840 tx=0x7f4bbc02bc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af7f2bc4ce2b3727 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:11:34.642 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.635+0000 7f4bab7fe640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-31T23:11:34.642 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.635+0000 7f4bab7fe640 1 -- 172.21.15.46:0/2886502195 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bbc050030 con 0x5616fb5072b0 2024-03-31T23:11:34.642 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.635+0000 7f4bab7fe640 1 -- 172.21.15.46:0/2886502195 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9208+0+0 (secure 0 0 0) 0x7f4bbc03a040 con 0x5616fb5072b0 2024-03-31T23:11:34.642 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.635+0000 7f4bd55f0740 1 -- 172.21.15.46:0/2886502195 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4bc4003e20 con 0x5616fb5072b0 2024-03-31T23:11:34.642 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.636+0000 7f4bab7fe640 1 -- 172.21.15.46:0/2886502195 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f4bbc05d050 con 0x5616fb5072b0 2024-03-31T23:11:34.642 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.636+0000 7f4bd55f0740 10 client.15630 fetch_fsmap learned FSMap version 25 2024-03-31T23:11:34.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.636+0000 7f4bd55f0740 1 -- 172.21.15.46:0/2886502195 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x5616fb1803f0 con 0x5616fb5072b0 2024-03-31T23:11:34.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.636+0000 7f4bab7fe640 1 -- 172.21.15.46:0/2886502195 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4bc4003e20 con 0x5616fb5072b0 2024-03-31T23:11:34.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.636+0000 7f4bab7fe640 1 -- 172.21.15.46:0/2886502195 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f4bbc003cc0 con 0x5616fb5072b0 2024-03-31T23:11:34.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.636+0000 7f4bd55f0740 10 client.15630 fetch_fsmap finished waiting for FSMap version 25 2024-03-31T23:11:34.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.636+0000 7f4bd55f0740 10 client.15630 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-31T23:11:34.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.636+0000 7f4bd55f0740 20 client.15630 populate_metadata read hostname 'smithi046' 2024-03-31T23:11:34.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.636+0000 7f4bd55f0740 1 --2- 172.21.15.46:0/2886502195 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x5616fb51d690 0x5616fb51fb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:34.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.636+0000 7f4bd55f0740 4 client.15630 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-31T23:11:34.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.636+0000 7f4bd55f0740 1 -- 172.21.15.46:0/2886502195 --> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5616fb4796c0 con 0x5616fb51d690 2024-03-31T23:11:34.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.636+0000 7f4bcb7fe640 1 --2- 172.21.15.46:0/2886502195 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x5616fb51d690 0x5616fb51fb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:34.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.637+0000 7f4bab7fe640 10 client.15630 ms_handle_connect on v2:172.21.15.142:6838/2519143918 2024-03-31T23:11:34.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.637+0000 7f4bcb7fe640 1 --2- 172.21.15.46:0/2886502195 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x5616fb51d690 0x5616fb51fb30 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=95df84d3339704ea server_cookie=b12d0dd6c51dc814 in_seq=0 out_seq=0 2024-03-31T23:11:34.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.638+0000 7f4bab7fe640 1 -- 172.21.15.46:0/2886502195 <== mds.0 v2:172.21.15.142:6838/2519143918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5616fb4796c0 con 0x5616fb51d690 2024-03-31T23:11:34.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.638+0000 7f4bab7fe640 10 client.15630 handle_command_reply: tid=0 2024-03-31T23:11:34.644 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.638+0000 7f4bd55f0740 1 client.15630 shutdown 2024-03-31T23:11:34.644 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.638+0000 7f4bd55f0740 1 -- 172.21.15.46:0/2886502195 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5616fb5072b0 msgr2=0x5616fb5117e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:34.644 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.638+0000 7f4bd55f0740 1 --2- 172.21.15.46:0/2886502195 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5616fb5072b0 0x5616fb5117e0 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f4bbc02c840 tx=0x7f4bbc02bc30 comp rx=0 tx=0).stop 2024-03-31T23:11:34.644 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.638+0000 7f4bd55f0740 1 -- 172.21.15.46:0/2886502195 shutdown_connections 2024-03-31T23:11:34.644 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.638+0000 7f4bd55f0740 1 --2- 172.21.15.46:0/2886502195 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x5616fb51d690 0x5616fb51fb30 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:34.644 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.638+0000 7f4bd55f0740 1 --2- 172.21.15.46:0/2886502195 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5616fb50ce10 0x5616fb518d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:34.644 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.638+0000 7f4bd55f0740 1 --2- 172.21.15.46:0/2886502195 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5616fb507ca0 0x5616fb511d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:34.644 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.638+0000 7f4bab7fe640 0 client.15630 ms_handle_reset on v2:172.21.15.142:6838/2519143918 2024-03-31T23:11:34.644 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.638+0000 7f4bd55f0740 1 --2- 172.21.15.46:0/2886502195 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5616fb5072b0 0x5616fb5117e0 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:34.644 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.638+0000 7f4bd55f0740 1 -- 172.21.15.46:0/2886502195 >> 172.21.15.46:0/2886502195 conn(0x5616fb502820 msgr2=0x5616fb501c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:11:34.644 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.638+0000 7f4bd55f0740 1 -- 172.21.15.46:0/2886502195 shutdown_connections 2024-03-31T23:11:34.644 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.638+0000 7f4bd55f0740 1 -- 172.21.15.46:0/2886502195 wait complete. 2024-03-31T23:11:34.644 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.638+0000 7f4bd55f0740 20 client.15630 trim_cache size 0 max 16384 2024-03-31T23:11:34.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.655+0000 7f4bd55f0740 1 Processor -- start 2024-03-31T23:11:34.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.655+0000 7f4bd55f0740 1 -- start start 2024-03-31T23:11:34.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.656+0000 7f4bd55f0740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5616fb5072b0 0x5616fb511b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:34.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.656+0000 7f4bd55f0740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5616fb507ca0 0x5616fb518d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:34.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.656+0000 7f4bd55f0740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5616fb50ce10 0x5616fb501390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:34.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.656+0000 7f4bd55f0740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f4bcc1aeeb0 con 0x5616fb5072b0 2024-03-31T23:11:34.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.656+0000 7f4bd55f0740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f4bcc1ac780 con 0x5616fb50ce10 2024-03-31T23:11:34.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.656+0000 7f4bd55f0740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f4bcc1aa5d0 con 0x5616fb507ca0 2024-03-31T23:11:34.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.656+0000 7f4bcbfff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5616fb5072b0 0x5616fb511b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:34.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.656+0000 7f4bcb7fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5616fb507ca0 0x5616fb518d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:34.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.656+0000 7f4bcbfff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5616fb5072b0 0x5616fb511b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:48338/0 (socket says 172.21.15.46:48338) 2024-03-31T23:11:34.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.656+0000 7f4bcbfff640 1 -- 172.21.15.46:0/3727099172 learned_addr learned my addr 172.21.15.46:0/3727099172 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:11:34.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.656+0000 7f4bd0ed2640 1 --2- 172.21.15.46:0/3727099172 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5616fb50ce10 0x5616fb501390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:34.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.656+0000 7f4bcb7fe640 1 -- 172.21.15.46:0/3727099172 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5616fb50ce10 msgr2=0x5616fb501390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:11:34.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.656+0000 7f4bcb7fe640 1 --2- 172.21.15.46:0/3727099172 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5616fb50ce10 0x5616fb501390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:34.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.656+0000 7f4bcb7fe640 1 -- 172.21.15.46:0/3727099172 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5616fb5072b0 msgr2=0x5616fb511b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:11:34.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.656+0000 7f4bcb7fe640 1 --2- 172.21.15.46:0/3727099172 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5616fb5072b0 0x5616fb511b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:34.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.656+0000 7f4bcb7fe640 1 -- 172.21.15.46:0/3727099172 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5616fb1400f0 con 0x5616fb507ca0 2024-03-31T23:11:34.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.656+0000 7f4bcbfff640 1 --2- 172.21.15.46:0/3727099172 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5616fb5072b0 0x5616fb511b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:11:34.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.656+0000 7f4bcb7fe640 1 --2- 172.21.15.46:0/3727099172 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5616fb507ca0 0x5616fb518d20 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f4bb40cc6f0 tx=0x7f4bb4086040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4382b727e78373c3 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:11:34.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.656+0000 7f4bc1ffb640 1 -- 172.21.15.46:0/3727099172 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bb4097870 con 0x5616fb507ca0 2024-03-31T23:11:34.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.656+0000 7f4bc1ffb640 1 -- 172.21.15.46:0/3727099172 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4bb409bd60 con 0x5616fb507ca0 2024-03-31T23:11:34.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.656+0000 7f4bc1ffb640 1 -- 172.21.15.46:0/3727099172 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bb40997a0 con 0x5616fb507ca0 2024-03-31T23:11:34.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.657+0000 7f4bd55f0740 1 -- 172.21.15.46:0/3727099172 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5616fb507ca0 msgr2=0x5616fb518d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:34.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.657+0000 7f4bd55f0740 1 --2- 172.21.15.46:0/3727099172 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5616fb507ca0 0x5616fb518d20 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f4bb40cc6f0 tx=0x7f4bb4086040 comp rx=0 tx=0).stop 2024-03-31T23:11:34.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.657+0000 7f4bd55f0740 1 -- 172.21.15.46:0/3727099172 shutdown_connections 2024-03-31T23:11:34.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.657+0000 7f4bd55f0740 1 --2- 172.21.15.46:0/3727099172 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5616fb50ce10 0x5616fb501390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:34.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.657+0000 7f4bd55f0740 1 --2- 172.21.15.46:0/3727099172 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5616fb507ca0 0x5616fb518d20 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:34.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.657+0000 7f4bd55f0740 1 --2- 172.21.15.46:0/3727099172 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5616fb5072b0 0x5616fb511b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:34.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.657+0000 7f4bd55f0740 1 -- 172.21.15.46:0/3727099172 >> 172.21.15.46:0/3727099172 conn(0x5616fb502820 msgr2=0x5616fb5113f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:11:34.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.657+0000 7f4bd55f0740 1 -- 172.21.15.46:0/3727099172 shutdown_connections 2024-03-31T23:11:34.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.657+0000 7f4bd55f0740 1 -- 172.21.15.46:0/3727099172 wait complete. 2024-03-31T23:11:34.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.657+0000 7f4bd55f0740 1 -- start start 2024-03-31T23:11:34.660 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.658+0000 7f4bd55f0740 1 Processor -- start 2024-03-31T23:11:34.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.658+0000 7f4bd55f0740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5616fb5072b0 0x5616fb516100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:34.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.658+0000 7f4bd55f0740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5616fb507ca0 0x5616fb516640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:34.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.658+0000 7f4bd55f0740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5616fb50ce10 0x5616fb517650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:34.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.658+0000 7f4bd55f0740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f4bcc1aac40 con 0x5616fb507ca0 2024-03-31T23:11:34.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.658+0000 7f4bd55f0740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f4bcc1ac0b0 con 0x5616fb50ce10 2024-03-31T23:11:34.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.658+0000 7f4bd55f0740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f4bcc1ab310 con 0x5616fb5072b0 2024-03-31T23:11:34.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.658+0000 7f4bd55f0740 10 client.0 _ll_register_callbacks cb 0x7f4bbc044050 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T23:11:34.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.658+0000 7f4bcb7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5616fb507ca0 0x5616fb516640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:34.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.658+0000 7f4bcb7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5616fb507ca0 0x5616fb516640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:48354/0 (socket says 172.21.15.46:48354) 2024-03-31T23:11:34.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.658+0000 7f4bcb7fe640 1 -- 172.21.15.46:0/1432498266 learned_addr learned my addr 172.21.15.46:0/1432498266 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:11:34.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.658+0000 7f4bd0ed2640 1 --2- 172.21.15.46:0/1432498266 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5616fb50ce10 0x5616fb517650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:34.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.658+0000 7f4bcbfff640 1 --2- 172.21.15.46:0/1432498266 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5616fb5072b0 0x5616fb516100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:34.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.658+0000 7f4bcb7fe640 1 -- 172.21.15.46:0/1432498266 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5616fb5072b0 msgr2=0x5616fb516100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:11:34.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.658+0000 7f4bcb7fe640 1 --2- 172.21.15.46:0/1432498266 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5616fb5072b0 0x5616fb516100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:34.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.658+0000 7f4bcb7fe640 1 -- 172.21.15.46:0/1432498266 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5616fb50ce10 msgr2=0x5616fb517650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:11:34.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.658+0000 7f4bcb7fe640 1 --2- 172.21.15.46:0/1432498266 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5616fb50ce10 0x5616fb517650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:34.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.658+0000 7f4bcb7fe640 1 -- 172.21.15.46:0/1432498266 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5616fb517c40 con 0x5616fb507ca0 2024-03-31T23:11:34.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.658+0000 7f4bcb7fe640 1 --2- 172.21.15.46:0/1432498266 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5616fb507ca0 0x5616fb516640 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f4bb409b360 tx=0x7f4bb4005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f1d9a4fa4859aa20 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:11:34.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.658+0000 7f4bab7fe640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-31T23:11:34.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.658+0000 7f4bab7fe640 1 -- 172.21.15.46:0/1432498266 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bb409afc0 con 0x5616fb507ca0 2024-03-31T23:11:34.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.658+0000 7f4bab7fe640 1 -- 172.21.15.46:0/1432498266 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9208+0+0 (secure 0 0 0) 0x7f4bb40b4290 con 0x5616fb507ca0 2024-03-31T23:11:34.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.658+0000 7f4bd55f0740 1 -- 172.21.15.46:0/1432498266 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4bb409bd60 con 0x5616fb507ca0 2024-03-31T23:11:34.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.659+0000 7f4bab7fe640 1 -- 172.21.15.46:0/1432498266 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f4bb409b1a0 con 0x5616fb507ca0 2024-03-31T23:11:34.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.659+0000 7f4bd55f0740 10 client.15636 fetch_fsmap learned FSMap version 25 2024-03-31T23:11:34.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.659+0000 7f4bd55f0740 1 -- 172.21.15.46:0/1432498266 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x5616fb518f50 con 0x5616fb507ca0 2024-03-31T23:11:34.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.659+0000 7f4bab7fe640 1 -- 172.21.15.46:0/1432498266 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4bb409bd60 con 0x5616fb507ca0 2024-03-31T23:11:34.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.659+0000 7f4bab7fe640 1 -- 172.21.15.46:0/1432498266 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f4bb40d10b0 con 0x5616fb507ca0 2024-03-31T23:11:34.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.659+0000 7f4bd55f0740 10 client.15636 fetch_fsmap finished waiting for FSMap version 25 2024-03-31T23:11:34.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.659+0000 7f4bd55f0740 10 client.15636 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-31T23:11:34.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.659+0000 7f4bd55f0740 20 client.15636 populate_metadata read hostname 'smithi046' 2024-03-31T23:11:34.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.659+0000 7f4bd55f0740 1 --2- 172.21.15.46:0/1432498266 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x5616fb584020 0x5616fb586440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:34.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.659+0000 7f4bd55f0740 4 client.15636 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-31T23:11:34.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.659+0000 7f4bd55f0740 1 -- 172.21.15.46:0/1432498266 --> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5616fb473a10 con 0x5616fb584020 2024-03-31T23:11:34.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.659+0000 7f4bcbfff640 1 --2- 172.21.15.46:0/1432498266 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x5616fb584020 0x5616fb586440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:34.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.660+0000 7f4bab7fe640 10 client.15636 ms_handle_connect on v2:172.21.15.142:6838/2519143918 2024-03-31T23:11:34.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.660+0000 7f4bcbfff640 1 --2- 172.21.15.46:0/1432498266 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x5616fb584020 0x5616fb586440 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=14543a3ffa27ed01 server_cookie=81eceaaeca9572c1 in_seq=0 out_seq=0 2024-03-31T23:11:34.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.660+0000 7f4bab7fe640 1 -- 172.21.15.46:0/1432498266 <== mds.0 v2:172.21.15.142:6838/2519143918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x5616fb473a10 con 0x5616fb584020 2024-03-31T23:11:34.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.660+0000 7f4bab7fe640 10 client.15636 handle_command_reply: tid=0 2024-03-31T23:11:34.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.660+0000 7f4bd55f0740 1 client.15636 shutdown 2024-03-31T23:11:34.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.660+0000 7f4bd55f0740 1 -- 172.21.15.46:0/1432498266 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5616fb507ca0 msgr2=0x5616fb516640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:34.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.660+0000 7f4bd55f0740 1 --2- 172.21.15.46:0/1432498266 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5616fb507ca0 0x5616fb516640 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f4bb409b360 tx=0x7f4bb4005880 comp rx=0 tx=0).stop 2024-03-31T23:11:34.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.660+0000 7f4bd55f0740 1 -- 172.21.15.46:0/1432498266 shutdown_connections 2024-03-31T23:11:34.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.660+0000 7f4bd55f0740 1 --2- 172.21.15.46:0/1432498266 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x5616fb584020 0x5616fb586440 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:34.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.660+0000 7f4bd55f0740 1 --2- 172.21.15.46:0/1432498266 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5616fb50ce10 0x5616fb517650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:34.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.660+0000 7f4bd55f0740 1 --2- 172.21.15.46:0/1432498266 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5616fb507ca0 0x5616fb516640 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:34.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.660+0000 7f4bab7fe640 0 client.15636 ms_handle_reset on v2:172.21.15.142:6838/2519143918 2024-03-31T23:11:34.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.660+0000 7f4bd55f0740 1 --2- 172.21.15.46:0/1432498266 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5616fb5072b0 0x5616fb516100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:34.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.660+0000 7f4bd55f0740 1 -- 172.21.15.46:0/1432498266 >> 172.21.15.46:0/1432498266 conn(0x5616fb502820 msgr2=0x5616fb501390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:11:34.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.660+0000 7f4bd55f0740 1 -- 172.21.15.46:0/1432498266 shutdown_connections 2024-03-31T23:11:34.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.660+0000 7f4bd55f0740 1 -- 172.21.15.46:0/1432498266 wait complete. 2024-03-31T23:11:34.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.660+0000 7f4bd55f0740 20 client.15636 trim_cache size 0 max 16384 2024-03-31T23:11:34.666 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-03-31T23:11:34.667 INFO:teuthology.orchestra.run.smithi046.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-31T23:11:34.667 INFO:teuthology.orchestra.run.smithi046.stdout: "scrubs": { 2024-03-31T23:11:34.667 INFO:teuthology.orchestra.run.smithi046.stdout: "22d11db0-ed6a-4208-aaa3-8ce32084e80d": { 2024-03-31T23:11:34.667 INFO:teuthology.orchestra.run.smithi046.stdout: "path": "/", 2024-03-31T23:11:34.667 INFO:teuthology.orchestra.run.smithi046.stdout: "tag": "22d11db0-ed6a-4208-aaa3-8ce32084e80d", 2024-03-31T23:11:34.667 INFO:teuthology.orchestra.run.smithi046.stdout: "options": "recursive,force" 2024-03-31T23:11:34.667 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-31T23:11:34.667 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-31T23:11:34.667 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-03-31T23:11:34.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.661+0000 7f4bd2935640 1 -- 172.21.15.46:0/2897013436 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f4bb80a24a0 msgr2=0x7f4bb80a4960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:34.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.661+0000 7f4bd2935640 1 --2- 172.21.15.46:0/2897013436 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f4bb80a24a0 0x7f4bb80a4960 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f4bcc19b110 tx=0x7f4bbc005fd0 comp rx=0 tx=0).stop 2024-03-31T23:11:34.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.661+0000 7f4bd2935640 1 -- 172.21.15.46:0/2897013436 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4bcc104b00 msgr2=0x7f4bcc19a180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:34.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.661+0000 7f4bd2935640 1 --2- 172.21.15.46:0/2897013436 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4bcc104b00 0x7f4bcc19a180 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f4bb400e9c0 tx=0x7f4bb400ee80 comp rx=0 tx=0).stop 2024-03-31T23:11:34.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.661+0000 7f4bd2935640 1 -- 172.21.15.46:0/2897013436 shutdown_connections 2024-03-31T23:11:34.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.661+0000 7f4bd2935640 1 --2- 172.21.15.46:0/2897013436 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f4bb80a24a0 0x7f4bb80a4960 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:34.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.661+0000 7f4bd2935640 1 --2- 172.21.15.46:0/2897013436 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4bcc107430 0x7f4bcc1a11b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:34.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.661+0000 7f4bd2935640 1 --2- 172.21.15.46:0/2897013436 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4bcc104b00 0x7f4bcc19a180 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:34.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.661+0000 7f4bd2935640 1 --2- 172.21.15.46:0/2897013436 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4bcc1021d0 0x7f4bcc199c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:34.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.661+0000 7f4bd2935640 1 -- 172.21.15.46:0/2897013436 >> 172.21.15.46:0/2897013436 conn(0x7f4bcc0fbbe0 msgr2=0x7f4bcc101940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:11:34.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.661+0000 7f4bd2935640 1 -- 172.21.15.46:0/2897013436 shutdown_connections 2024-03-31T23:11:34.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.661+0000 7f4bd2935640 1 -- 172.21.15.46:0/2897013436 wait complete. 2024-03-31T23:11:34.671 INFO:tasks.cephfs.filesystem:scrub status for tag:22d11db0-ed6a-4208-aaa3-8ce32084e80d - {'path': '/', 'tag': '22d11db0-ed6a-4208-aaa3-8ce32084e80d', 'options': 'recursive,force'} 2024-03-31T23:11:34.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:34.725+0000 7f655a3c9640 1 -- 172.21.15.46:0/3829663715 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) -- 0x7f65541035f0 con 0x7f6554101860 2024-03-31T23:11:34.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:11:34 smithi142 ceph-mon[35271]: pgmap v367: 129 pgs: 129 active+clean; 648 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 1.1 KiB/s wr, 6 op/s 2024-03-31T23:11:34.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:11:34 smithi142 ceph-mon[35271]: from='client.? 172.21.15.46:0/268846486' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-31T23:11:34.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:11:34 smithi142 ceph-mon[35271]: from='client.? 172.21.15.46:0/339498258' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-31T23:11:34.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:11:34 smithi142 ceph-mon[35271]: from='client.? 172.21.15.46:0/3057859208' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-31T23:11:35.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:11:34 smithi162 ceph-mon[36237]: pgmap v367: 129 pgs: 129 active+clean; 648 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 1.1 KiB/s wr, 6 op/s 2024-03-31T23:11:35.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:11:34 smithi162 ceph-mon[36237]: from='client.? 172.21.15.46:0/268846486' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-31T23:11:35.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:11:34 smithi162 ceph-mon[36237]: from='client.? 172.21.15.46:0/339498258' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-31T23:11:35.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:11:34 smithi162 ceph-mon[36237]: from='client.? 172.21.15.46:0/3057859208' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-31T23:11:35.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:35.600+0000 7f65517fa640 1 -- 172.21.15.46:0/3829663715 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v19) ==== 92+0+0 (secure 0 0 0) 0x7f65480be040 con 0x7f6554101860 2024-03-31T23:11:35.603 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:35.603+0000 7f655a3c9640 1 -- 172.21.15.46:0/3829663715 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f653c0a25c0 msgr2=0x7f653c0a4a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:35.603 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:35.603+0000 7f655a3c9640 1 --2- 172.21.15.46:0/3829663715 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f653c0a25c0 0x7f653c0a4a80 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f65440058c0 tx=0x7f6544012040 comp rx=0 tx=0).stop 2024-03-31T23:11:35.603 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:35.603+0000 7f655a3c9640 1 -- 172.21.15.46:0/3829663715 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6554101860 msgr2=0x7f6554197e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:35.603 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:35.603+0000 7f655a3c9640 1 --2- 172.21.15.46:0/3829663715 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6554101860 0x7f6554197e70 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f654802c170 tx=0x7f6548005e60 comp rx=0 tx=0).stop 2024-03-31T23:11:35.604 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:35.604+0000 7f655a3c9640 1 -- 172.21.15.46:0/3829663715 shutdown_connections 2024-03-31T23:11:35.604 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:35.604+0000 7f655a3c9640 1 --2- 172.21.15.46:0/3829663715 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f653c0a25c0 0x7f653c0a4a80 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:35.604 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:35.604+0000 7f655a3c9640 1 --2- 172.21.15.46:0/3829663715 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6554102220 0x7f655419eee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:35.604 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:35.604+0000 7f655a3c9640 1 --2- 172.21.15.46:0/3829663715 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6554101860 0x7f6554197e70 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:35.604 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:35.604+0000 7f655a3c9640 1 --2- 172.21.15.46:0/3829663715 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6554100670 0x7f6554197930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:35.604 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:35.604+0000 7f655a3c9640 1 -- 172.21.15.46:0/3829663715 >> 172.21.15.46:0/3829663715 conn(0x7f65540fbbe0 msgr2=0x7f65540fd670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:11:35.604 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:35.604+0000 7f655a3c9640 1 -- 172.21.15.46:0/3829663715 shutdown_connections 2024-03-31T23:11:35.604 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:35.604+0000 7f655a3c9640 1 -- 172.21.15.46:0/3829663715 wait complete. 2024-03-31T23:11:35.622 DEBUG:teuthology.orchestra.run.smithi046:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph config set mgr mgr/snap_schedule/allow_m_granularity true' 2024-03-31T23:11:35.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:35.732+0000 7f6b33d6b640 1 -- 172.21.15.46:0/3063481427 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6b2c1021f0 msgr2=0x7f6b2c1045e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:35.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:35.732+0000 7f6b33d6b640 1 --2- 172.21.15.46:0/3063481427 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6b2c1021f0 0x7f6b2c1045e0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f6b1c009900 tx=0x7f6b1c02bc30 comp rx=0 tx=0).stop 2024-03-31T23:11:35.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:35.732+0000 7f6b33d6b640 1 -- 172.21.15.46:0/3063481427 shutdown_connections 2024-03-31T23:11:35.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:35.732+0000 7f6b33d6b640 1 --2- 172.21.15.46:0/3063481427 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6b2c107450 0x7f6b2c1098a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:35.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:35.732+0000 7f6b33d6b640 1 --2- 172.21.15.46:0/3063481427 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6b2c104b20 0x7f6b2c106f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:35.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:35.732+0000 7f6b33d6b640 1 --2- 172.21.15.46:0/3063481427 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6b2c1021f0 0x7f6b2c1045e0 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:35.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:35.732+0000 7f6b33d6b640 1 -- 172.21.15.46:0/3063481427 >> 172.21.15.46:0/3063481427 conn(0x7f6b2c0fbbe0 msgr2=0x7f6b2c0fe020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:11:35.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:35.732+0000 7f6b33d6b640 1 -- 172.21.15.46:0/3063481427 shutdown_connections 2024-03-31T23:11:35.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:35.732+0000 7f6b33d6b640 1 -- 172.21.15.46:0/3063481427 wait complete. 2024-03-31T23:11:35.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:35.732+0000 7f6b33d6b640 1 Processor -- start 2024-03-31T23:11:35.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:35.733+0000 7f6b33d6b640 1 -- start start 2024-03-31T23:11:35.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:35.733+0000 7f6b33d6b640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6b2c1021f0 0x7f6b2c101b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:35.736 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:35.733+0000 7f6b33d6b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6b2c104b20 0x7f6b2c1001a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:35.736 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:35.733+0000 7f6b33d6b640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6b2c107450 0x7f6b2c1006e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:35.736 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:35.733+0000 7f6b33d6b640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f6b2c10d2a0 con 0x7f6b2c104b20 2024-03-31T23:11:35.736 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:35.733+0000 7f6b33d6b640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f6b2c10d120 con 0x7f6b2c107450 2024-03-31T23:11:35.736 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:35.733+0000 7f6b33d6b640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f6b2c10d420 con 0x7f6b2c1021f0 2024-03-31T23:11:35.736 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:35.733+0000 7f6b31306640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6b2c104b20 0x7f6b2c1001a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:35.736 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:35.733+0000 7f6b31b07640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6b2c1021f0 0x7f6b2c101b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:35.736 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:35.733+0000 7f6b31306640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6b2c104b20 0x7f6b2c1001a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:48374/0 (socket says 172.21.15.46:48374) 2024-03-31T23:11:35.736 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:35.733+0000 7f6b31306640 1 -- 172.21.15.46:0/2953169307 learned_addr learned my addr 172.21.15.46:0/2953169307 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:11:35.736 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:35.733+0000 7f6b32308640 1 --2- 172.21.15.46:0/2953169307 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6b2c107450 0x7f6b2c1006e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:35.736 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:35.733+0000 7f6b31306640 1 -- 172.21.15.46:0/2953169307 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6b2c1021f0 msgr2=0x7f6b2c101b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:35.736 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:35.733+0000 7f6b31306640 1 --2- 172.21.15.46:0/2953169307 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6b2c1021f0 0x7f6b2c101b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:35.736 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:35.733+0000 7f6b31306640 1 -- 172.21.15.46:0/2953169307 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6b2c107450 msgr2=0x7f6b2c1006e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:35.736 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:35.733+0000 7f6b31306640 1 --2- 172.21.15.46:0/2953169307 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6b2c107450 0x7f6b2c1006e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:35.736 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:35.733+0000 7f6b31306640 1 -- 172.21.15.46:0/2953169307 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b2c100f70 con 0x7f6b2c104b20 2024-03-31T23:11:35.736 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:35.733+0000 7f6b31306640 1 --2- 172.21.15.46:0/2953169307 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6b2c104b20 0x7f6b2c1001a0 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f6b2800d770 tx=0x7f6b2800dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:11:35.737 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:35.734+0000 7f6b22ffd640 1 -- 172.21.15.46:0/2953169307 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b28017070 con 0x7f6b2c104b20 2024-03-31T23:11:35.737 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:35.734+0000 7f6b33d6b640 1 -- 172.21.15.46:0/2953169307 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6b2c1a1a70 con 0x7f6b2c104b20 2024-03-31T23:11:35.737 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:35.734+0000 7f6b22ffd640 1 -- 172.21.15.46:0/2953169307 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6b28009c30 con 0x7f6b2c104b20 2024-03-31T23:11:35.737 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:35.734+0000 7f6b22ffd640 1 -- 172.21.15.46:0/2953169307 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b2800f620 con 0x7f6b2c104b20 2024-03-31T23:11:35.737 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:35.734+0000 7f6b33d6b640 1 -- 172.21.15.46:0/2953169307 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6b2c1a1fc0 con 0x7f6b2c104b20 2024-03-31T23:11:35.737 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:35.735+0000 7f6b33d6b640 1 -- 172.21.15.46:0/2953169307 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6b2c0fd6d0 con 0x7f6b2c104b20 2024-03-31T23:11:35.737 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:35.736+0000 7f6b22ffd640 1 -- 172.21.15.46:0/2953169307 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 19) ==== 137337+0+0 (secure 0 0 0) 0x7f6b2801a070 con 0x7f6b2c104b20 2024-03-31T23:11:35.738 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:35.737+0000 7f6b22ffd640 1 --2- 172.21.15.46:0/2953169307 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f6b180a2590 0x7f6b180a4a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:35.738 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:35.737+0000 7f6b31b07640 1 -- 172.21.15.46:0/2953169307 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f6b180a2590 msgr2=0x7f6b180a4a50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.46:6800/162289569 2024-03-31T23:11:35.738 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:35.737+0000 7f6b22ffd640 1 -- 172.21.15.46:0/2953169307 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9208+0+0 (secure 0 0 0) 0x7f6b280ce5d0 con 0x7f6b2c104b20 2024-03-31T23:11:35.738 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:35.737+0000 7f6b31b07640 1 --2- 172.21.15.46:0/2953169307 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f6b180a2590 0x7f6b180a4a50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-31T23:11:35.741 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:35.741+0000 7f6b22ffd640 1 -- 172.21.15.46:0/2953169307 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f6b28009de0 con 0x7f6b2c104b20 2024-03-31T23:11:35.848 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:11:35 smithi046 ceph-mon[32498]: scrub queued for path: / 2024-03-31T23:11:35.848 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:11:35 smithi046 ceph-mon[32498]: scrub summary: idle+waiting paths [/] 2024-03-31T23:11:35.848 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:11:35 smithi046 ceph-mon[32498]: scrub summary: active paths [/] 2024-03-31T23:11:35.848 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:11:35 smithi046 ceph-mon[32498]: scrub summary: idle+waiting paths [/] 2024-03-31T23:11:35.848 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:11:35 smithi046 ceph-mon[32498]: pgmap v368: 129 pgs: 129 active+clean; 648 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 1.1 KiB/s wr, 6 op/s 2024-03-31T23:11:35.848 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:11:35 smithi046 ceph-mon[32498]: from='client.? 172.21.15.46:0/3829663715' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-03-31T23:11:35.848 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 31 23:11:35 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mgr-x[32727]: ignoring --setuser ceph since I am not root 2024-03-31T23:11:35.849 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 31 23:11:35 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mgr-x[32727]: ignoring --setgroup ceph since I am not root 2024-03-31T23:11:35.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:35.926+0000 7f6b33d6b640 1 -- 172.21.15.46:0/2953169307 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) -- 0x7f6b2c02d170 con 0x7f6b2c104b20 2024-03-31T23:11:35.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:35.930+0000 7f6b22ffd640 1 -- 172.21.15.46:0/2953169307 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v27)=0 v27) ==== 138+0+0 (secure 0 0 0) 0x7f6b2809ad10 con 0x7f6b2c104b20 2024-03-31T23:11:35.933 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:35.932+0000 7f6b33d6b640 1 -- 172.21.15.46:0/2953169307 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f6b180a2590 msgr2=0x7f6b180a4a50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-31T23:11:35.933 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:35.932+0000 7f6b33d6b640 1 --2- 172.21.15.46:0/2953169307 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f6b180a2590 0x7f6b180a4a50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:35.934 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:35.932+0000 7f6b33d6b640 1 -- 172.21.15.46:0/2953169307 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6b2c104b20 msgr2=0x7f6b2c1001a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:35.934 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:35.932+0000 7f6b33d6b640 1 --2- 172.21.15.46:0/2953169307 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6b2c104b20 0x7f6b2c1001a0 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f6b2800d770 tx=0x7f6b2800dc30 comp rx=0 tx=0).stop 2024-03-31T23:11:35.934 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:35.932+0000 7f6b33d6b640 1 -- 172.21.15.46:0/2953169307 shutdown_connections 2024-03-31T23:11:35.934 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:35.932+0000 7f6b33d6b640 1 --2- 172.21.15.46:0/2953169307 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7f6b180a2590 0x7f6b180a4a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:35.934 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:35.932+0000 7f6b33d6b640 1 --2- 172.21.15.46:0/2953169307 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6b2c107450 0x7f6b2c1006e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:35.934 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:35.932+0000 7f6b33d6b640 1 --2- 172.21.15.46:0/2953169307 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6b2c104b20 0x7f6b2c1001a0 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:35.934 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:35.932+0000 7f6b33d6b640 1 --2- 172.21.15.46:0/2953169307 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6b2c1021f0 0x7f6b2c101b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:35.934 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:35.932+0000 7f6b33d6b640 1 -- 172.21.15.46:0/2953169307 >> 172.21.15.46:0/2953169307 conn(0x7f6b2c0fbbe0 msgr2=0x7f6b2c102b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:11:35.934 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:35.932+0000 7f6b33d6b640 1 -- 172.21.15.46:0/2953169307 shutdown_connections 2024-03-31T23:11:35.934 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:35.932+0000 7f6b33d6b640 1 -- 172.21.15.46:0/2953169307 wait complete. 2024-03-31T23:11:35.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:11:35 smithi142 ceph-mon[35271]: scrub queued for path: / 2024-03-31T23:11:35.944 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:11:35 smithi142 ceph-mon[35271]: scrub summary: idle+waiting paths [/] 2024-03-31T23:11:35.944 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:11:35 smithi142 ceph-mon[35271]: scrub summary: active paths [/] 2024-03-31T23:11:35.944 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:11:35 smithi142 ceph-mon[35271]: scrub summary: idle+waiting paths [/] 2024-03-31T23:11:35.944 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:11:35 smithi142 ceph-mon[35271]: pgmap v368: 129 pgs: 129 active+clean; 648 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 1.1 KiB/s wr, 6 op/s 2024-03-31T23:11:35.944 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:11:35 smithi142 ceph-mon[35271]: from='client.? 172.21.15.46:0/3829663715' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-03-31T23:11:35.944 INFO:journalctl@ceph.mgr.y.smithi142.stdout:Mar 31 23:11:35 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mgr-y[36663]: ignoring --setuser ceph since I am not root 2024-03-31T23:11:35.944 INFO:journalctl@ceph.mgr.y.smithi142.stdout:Mar 31 23:11:35 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mgr-y[36663]: ignoring --setgroup ceph since I am not root 2024-03-31T23:11:35.944 INFO:journalctl@ceph.mgr.y.smithi142.stdout:Mar 31 23:11:35 smithi142 ceph-mgr[36667]: -- 172.21.15.142:0/2738955029 <== mon.2 v2:172.21.15.142:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x559a7bd045a0 con 0x559a7bcd9000 2024-03-31T23:11:35.945 DEBUG:teuthology.orchestra.run.smithi046:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph config set mgr mgr/snap_schedule/dump_on_update true' 2024-03-31T23:11:36.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:11:35 smithi162 ceph-mon[36237]: scrub queued for path: / 2024-03-31T23:11:36.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:11:35 smithi162 ceph-mon[36237]: scrub summary: idle+waiting paths [/] 2024-03-31T23:11:36.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:11:35 smithi162 ceph-mon[36237]: scrub summary: active paths [/] 2024-03-31T23:11:36.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:11:35 smithi162 ceph-mon[36237]: scrub summary: idle+waiting paths [/] 2024-03-31T23:11:36.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:11:35 smithi162 ceph-mon[36237]: pgmap v368: 129 pgs: 129 active+clean; 648 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 1.1 KiB/s wr, 6 op/s 2024-03-31T23:11:36.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:11:35 smithi162 ceph-mon[36237]: from='client.? 172.21.15.46:0/3829663715' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-03-31T23:11:36.047 INFO:journalctl@ceph.mgr.z.smithi162.stdout:Mar 31 23:11:35 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mgr-z[37924]: ignoring --setuser ceph since I am not root 2024-03-31T23:11:36.047 INFO:journalctl@ceph.mgr.z.smithi162.stdout:Mar 31 23:11:35 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mgr-z[37924]: ignoring --setgroup ceph since I am not root 2024-03-31T23:11:36.047 INFO:journalctl@ceph.mgr.z.smithi162.stdout:Mar 31 23:11:35 smithi162 ceph-mgr[37953]: -- 172.21.15.162:0/548124866 <== mon.1 v2:172.21.15.162:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55774a1b65a0 con 0x55774a18b000 2024-03-31T23:11:36.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:36.049+0000 7fdb6cc40640 1 -- 172.21.15.46:0/3786149774 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdb68109630 msgr2=0x7fdb6810bad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:36.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:36.049+0000 7fdb6cc40640 1 --2- 172.21.15.46:0/3786149774 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdb68109630 0x7fdb6810bad0 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7fdb60009ca0 tx=0x7fdb6002c180 comp rx=0 tx=0).stop 2024-03-31T23:11:36.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:36.049+0000 7fdb6cc40640 1 -- 172.21.15.46:0/3786149774 shutdown_connections 2024-03-31T23:11:36.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:36.049+0000 7fdb6cc40640 1 --2- 172.21.15.46:0/3786149774 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdb68109630 0x7fdb6810bad0 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:36.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:36.049+0000 7fdb6cc40640 1 --2- 172.21.15.46:0/3786149774 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdb68074150 0x7fdb680745b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:36.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:36.049+0000 7fdb6cc40640 1 --2- 172.21.15.46:0/3786149774 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdb68075730 0x7fdb68073b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:36.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:36.049+0000 7fdb6cc40640 1 -- 172.21.15.46:0/3786149774 >> 172.21.15.46:0/3786149774 conn(0x7fdb680fcd10 msgr2=0x7fdb680ff150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:11:36.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:36.049+0000 7fdb6cc40640 1 -- 172.21.15.46:0/3786149774 shutdown_connections 2024-03-31T23:11:36.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:36.049+0000 7fdb6cc40640 1 -- 172.21.15.46:0/3786149774 wait complete. 2024-03-31T23:11:36.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:36.050+0000 7fdb6cc40640 1 Processor -- start 2024-03-31T23:11:36.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:36.050+0000 7fdb6cc40640 1 -- start start 2024-03-31T23:11:36.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:36.050+0000 7fdb6cc40640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdb68074150 0x7fdb68199ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:36.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:36.050+0000 7fdb6cc40640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdb68075730 0x7fdb6819a020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:36.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:36.050+0000 7fdb6cc40640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdb68109630 0x7fdb681a1090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:36.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:36.050+0000 7fdb6cc40640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fdb6810f4d0 con 0x7fdb68109630 2024-03-31T23:11:36.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:36.050+0000 7fdb6cc40640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fdb6810f350 con 0x7fdb68074150 2024-03-31T23:11:36.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:36.050+0000 7fdb6cc40640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fdb6810f650 con 0x7fdb68075730 2024-03-31T23:11:36.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:36.050+0000 7fdb66d9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdb68109630 0x7fdb681a1090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:36.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:36.050+0000 7fdb66d9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdb68109630 0x7fdb681a1090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:48390/0 (socket says 172.21.15.46:48390) 2024-03-31T23:11:36.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:36.050+0000 7fdb66d9d640 1 -- 172.21.15.46:0/1754665220 learned_addr learned my addr 172.21.15.46:0/1754665220 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:11:36.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:36.051+0000 7fdb6659c640 1 --2- 172.21.15.46:0/1754665220 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdb68074150 0x7fdb68199ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:36.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:36.051+0000 7fdb65d9b640 1 --2- 172.21.15.46:0/1754665220 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdb68075730 0x7fdb6819a020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:36.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:36.051+0000 7fdb66d9d640 1 -- 172.21.15.46:0/1754665220 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdb68075730 msgr2=0x7fdb6819a020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:36.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:36.051+0000 7fdb66d9d640 1 --2- 172.21.15.46:0/1754665220 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdb68075730 0x7fdb6819a020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:36.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:36.051+0000 7fdb66d9d640 1 -- 172.21.15.46:0/1754665220 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdb68074150 msgr2=0x7fdb68199ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:36.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:36.051+0000 7fdb66d9d640 1 --2- 172.21.15.46:0/1754665220 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdb68074150 0x7fdb68199ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:36.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:36.051+0000 7fdb66d9d640 1 -- 172.21.15.46:0/1754665220 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb681a1700 con 0x7fdb68109630 2024-03-31T23:11:36.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:36.051+0000 7fdb66d9d640 1 --2- 172.21.15.46:0/1754665220 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdb68109630 0x7fdb681a1090 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7fdb60037040 tx=0x7fdb60009360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:11:36.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:36.051+0000 7fdb577fe640 1 -- 172.21.15.46:0/1754665220 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb60039040 con 0x7fdb68109630 2024-03-31T23:11:36.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:36.051+0000 7fdb577fe640 1 -- 172.21.15.46:0/1754665220 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fdb60044070 con 0x7fdb68109630 2024-03-31T23:11:36.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:36.051+0000 7fdb6cc40640 1 -- 172.21.15.46:0/1754665220 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdb681a1990 con 0x7fdb68109630 2024-03-31T23:11:36.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:36.051+0000 7fdb577fe640 1 -- 172.21.15.46:0/1754665220 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb600042a0 con 0x7fdb68109630 2024-03-31T23:11:36.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:36.051+0000 7fdb6cc40640 1 -- 172.21.15.46:0/1754665220 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdb681a1e80 con 0x7fdb68109630 2024-03-31T23:11:36.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:36.052+0000 7fdb6cc40640 1 -- 172.21.15.46:0/1754665220 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdb681083f0 con 0x7fdb68109630 2024-03-31T23:11:36.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:36.053+0000 7fdb577fe640 1 -- 172.21.15.46:0/1754665220 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 19) ==== 137337+0+0 (secure 0 0 0) 0x7fdb60004500 con 0x7fdb68109630 2024-03-31T23:11:36.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:36.054+0000 7fdb577fe640 1 --2- 172.21.15.46:0/1754665220 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fdb500a2590 0x7fdb500a4a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:36.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:36.054+0000 7fdb6659c640 1 -- 172.21.15.46:0/1754665220 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fdb500a2590 msgr2=0x7fdb500a4a50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.46:6800/162289569 2024-03-31T23:11:36.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:36.054+0000 7fdb577fe640 1 -- 172.21.15.46:0/1754665220 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9208+0+0 (secure 0 0 0) 0x7fdb600f10d0 con 0x7fdb68109630 2024-03-31T23:11:36.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:36.054+0000 7fdb6659c640 1 --2- 172.21.15.46:0/1754665220 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fdb500a2590 0x7fdb500a4a50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-31T23:11:36.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:36.058+0000 7fdb577fe640 1 -- 172.21.15.46:0/1754665220 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fdb600bde60 con 0x7fdb68109630 2024-03-31T23:11:36.243 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:36.242+0000 7fdb6cc40640 1 -- 172.21.15.46:0/1754665220 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) -- 0x7fdb68062460 con 0x7fdb68109630 2024-03-31T23:11:36.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:36.246+0000 7fdb577fe640 1 -- 172.21.15.46:0/1754665220 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v28)=0 v28) ==== 133+0+0 (secure 0 0 0) 0x7fdb600bd760 con 0x7fdb68109630 2024-03-31T23:11:36.249 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:36.248+0000 7fdb6cc40640 1 -- 172.21.15.46:0/1754665220 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fdb500a2590 msgr2=0x7fdb500a4a50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-31T23:11:36.249 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:36.248+0000 7fdb6cc40640 1 --2- 172.21.15.46:0/1754665220 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fdb500a2590 0x7fdb500a4a50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:36.249 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:36.248+0000 7fdb6cc40640 1 -- 172.21.15.46:0/1754665220 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdb68109630 msgr2=0x7fdb681a1090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:36.249 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:36.248+0000 7fdb6cc40640 1 --2- 172.21.15.46:0/1754665220 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdb68109630 0x7fdb681a1090 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7fdb60037040 tx=0x7fdb60009360 comp rx=0 tx=0).stop 2024-03-31T23:11:36.249 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:36.248+0000 7fdb6cc40640 1 -- 172.21.15.46:0/1754665220 shutdown_connections 2024-03-31T23:11:36.249 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:36.248+0000 7fdb6cc40640 1 --2- 172.21.15.46:0/1754665220 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fdb500a2590 0x7fdb500a4a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:36.249 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:36.248+0000 7fdb6cc40640 1 --2- 172.21.15.46:0/1754665220 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdb68109630 0x7fdb681a1090 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:36.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:36.248+0000 7fdb6cc40640 1 --2- 172.21.15.46:0/1754665220 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdb68075730 0x7fdb6819a020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:36.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:36.248+0000 7fdb6cc40640 1 --2- 172.21.15.46:0/1754665220 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdb68074150 0x7fdb68199ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:36.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:36.248+0000 7fdb6cc40640 1 -- 172.21.15.46:0/1754665220 >> 172.21.15.46:0/1754665220 conn(0x7fdb680fcd10 msgr2=0x7fdb681055c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:11:36.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:36.248+0000 7fdb6cc40640 1 -- 172.21.15.46:0/1754665220 shutdown_connections 2024-03-31T23:11:36.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:36.248+0000 7fdb6cc40640 1 -- 172.21.15.46:0/1754665220 wait complete. 2024-03-31T23:11:36.260 DEBUG:teuthology.orchestra.run.smithi046:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule add --fs=cephfs --path=/ --snap_schedule=1m' 2024-03-31T23:11:36.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:36.366+0000 7fe45dba0640 1 -- 172.21.15.46:0/3073272803 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe458107280 msgr2=0x7fe4581096d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:36.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:36.366+0000 7fe45dba0640 1 --2- 172.21.15.46:0/3073272803 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe458107280 0x7fe4581096d0 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7fe450009ca0 tx=0x7fe45002c0f0 comp rx=0 tx=0).stop 2024-03-31T23:11:36.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:36.366+0000 7fe45dba0640 1 -- 172.21.15.46:0/3073272803 shutdown_connections 2024-03-31T23:11:36.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:36.366+0000 7fe45dba0640 1 --2- 172.21.15.46:0/3073272803 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe458107280 0x7fe4581096d0 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:36.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:36.366+0000 7fe45dba0640 1 --2- 172.21.15.46:0/3073272803 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe458104950 0x7fe458106d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:36.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:36.366+0000 7fe45dba0640 1 --2- 172.21.15.46:0/3073272803 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe458102020 0x7fe458104410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:36.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:36.366+0000 7fe45dba0640 1 -- 172.21.15.46:0/3073272803 >> 172.21.15.46:0/3073272803 conn(0x7fe4580fba30 msgr2=0x7fe4580fde50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:11:36.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:36.366+0000 7fe45dba0640 1 -- 172.21.15.46:0/3073272803 shutdown_connections 2024-03-31T23:11:36.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:36.366+0000 7fe45dba0640 1 -- 172.21.15.46:0/3073272803 wait complete. 2024-03-31T23:11:36.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:36.366+0000 7fe45dba0640 1 Processor -- start 2024-03-31T23:11:36.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:36.366+0000 7fe45dba0640 1 -- start start 2024-03-31T23:11:36.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:36.366+0000 7fe45dba0640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe458102020 0x7fe458101990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:36.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:36.366+0000 7fe45dba0640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe458104950 0x7fe4580fffd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:36.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:36.366+0000 7fe45dba0640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe458107280 0x7fe458100510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:36.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:36.366+0000 7fe45dba0640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fe45810d0e0 con 0x7fe458102020 2024-03-31T23:11:36.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:36.366+0000 7fe45dba0640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fe45810cf60 con 0x7fe458107280 2024-03-31T23:11:36.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:36.366+0000 7fe45dba0640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fe45810d260 con 0x7fe458104950 2024-03-31T23:11:36.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:36.367+0000 7fe45cb9e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe458102020 0x7fe458101990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:36.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:36.367+0000 7fe45cb9e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe458102020 0x7fe458101990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:48402/0 (socket says 172.21.15.46:48402) 2024-03-31T23:11:36.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:36.367+0000 7fe45cb9e640 1 -- 172.21.15.46:0/509451667 learned_addr learned my addr 172.21.15.46:0/509451667 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:11:36.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:36.367+0000 7fe45d39f640 1 --2- 172.21.15.46:0/509451667 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe458107280 0x7fe458100510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:36.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:36.367+0000 7fe457fff640 1 --2- 172.21.15.46:0/509451667 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe458104950 0x7fe4580fffd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:36.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:36.367+0000 7fe45cb9e640 1 -- 172.21.15.46:0/509451667 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe458104950 msgr2=0x7fe4580fffd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:36.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:36.367+0000 7fe45cb9e640 1 --2- 172.21.15.46:0/509451667 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe458104950 0x7fe4580fffd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:36.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:36.367+0000 7fe45cb9e640 1 -- 172.21.15.46:0/509451667 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe458107280 msgr2=0x7fe458100510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:36.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:36.367+0000 7fe45cb9e640 1 --2- 172.21.15.46:0/509451667 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe458107280 0x7fe458100510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:36.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:36.367+0000 7fe45cb9e640 1 -- 172.21.15.46:0/509451667 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe458100da0 con 0x7fe458102020 2024-03-31T23:11:36.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:36.367+0000 7fe457fff640 1 --2- 172.21.15.46:0/509451667 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe458104950 0x7fe4580fffd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T23:11:36.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:36.367+0000 7fe45cb9e640 1 --2- 172.21.15.46:0/509451667 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe458102020 0x7fe458101990 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7fe44c00d950 tx=0x7fe44c00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:11:36.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:36.367+0000 7fe455ffb640 1 -- 172.21.15.46:0/509451667 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe44c017070 con 0x7fe458102020 2024-03-31T23:11:36.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:36.367+0000 7fe45dba0640 1 -- 172.21.15.46:0/509451667 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe4581a1880 con 0x7fe458102020 2024-03-31T23:11:36.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:36.367+0000 7fe455ffb640 1 -- 172.21.15.46:0/509451667 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe44c009c30 con 0x7fe458102020 2024-03-31T23:11:36.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:36.367+0000 7fe455ffb640 1 -- 172.21.15.46:0/509451667 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe44c00f620 con 0x7fe458102020 2024-03-31T23:11:36.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:36.367+0000 7fe45dba0640 1 -- 172.21.15.46:0/509451667 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe4581a1dd0 con 0x7fe458102020 2024-03-31T23:11:36.372 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:36.368+0000 7fe45dba0640 1 -- 172.21.15.46:0/509451667 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe4580fd520 con 0x7fe458102020 2024-03-31T23:11:36.372 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:36.369+0000 7fe455ffb640 1 -- 172.21.15.46:0/509451667 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 19) ==== 137337+0+0 (secure 0 0 0) 0x7fe44c0109a0 con 0x7fe458102020 2024-03-31T23:11:36.372 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:36.370+0000 7fe455ffb640 1 --2- 172.21.15.46:0/509451667 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fe4480a24e0 0x7fe4480a49a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:36.372 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:36.370+0000 7fe457fff640 1 -- 172.21.15.46:0/509451667 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fe4480a24e0 msgr2=0x7fe4480a49a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.46:6800/162289569 2024-03-31T23:11:36.372 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:36.370+0000 7fe455ffb640 1 -- 172.21.15.46:0/509451667 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9208+0+0 (secure 0 0 0) 0x7fe44c0ce830 con 0x7fe458102020 2024-03-31T23:11:36.372 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:36.370+0000 7fe457fff640 1 --2- 172.21.15.46:0/509451667 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fe4480a24e0 0x7fe4480a49a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-31T23:11:36.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:36.374+0000 7fe455ffb640 1 -- 172.21.15.46:0/509451667 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fe44c009de0 con 0x7fe458102020 2024-03-31T23:11:36.545 INFO:journalctl@ceph.mgr.z.smithi162.stdout:Mar 31 23:11:36 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mgr-z[37924]: 2024-03-31T23:11:36.229+0000 7f758cd5e200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-31T23:11:36.546 INFO:journalctl@ceph.mgr.z.smithi162.stdout:Mar 31 23:11:36 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mgr-z[37924]: 2024-03-31T23:11:36.347+0000 7f758cd5e200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-31T23:11:36.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:36.571+0000 7fe457fff640 1 -- 172.21.15.46:0/509451667 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fe4480a24e0 msgr2=0x7fe4480a49a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.46:6800/162289569 2024-03-31T23:11:36.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:36.571+0000 7fe457fff640 1 --2- 172.21.15.46:0/509451667 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fe4480a24e0 0x7fe4480a49a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-31T23:11:36.580 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:36.579+0000 7fe45dba0640 1 -- 172.21.15.46:0/509451667 --> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) -- 0x7fe458061b90 con 0x7fe4480a24e0 2024-03-31T23:11:36.600 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 31 23:11:36 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mgr-x[32727]: 2024-03-31T23:11:36.199+0000 7f1d8d69c200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-31T23:11:36.600 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 31 23:11:36 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mgr-x[32727]: 2024-03-31T23:11:36.298+0000 7f1d8d69c200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-31T23:11:36.602 INFO:journalctl@ceph.mgr.y.smithi142.stdout:Mar 31 23:11:36 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mgr-y[36663]: 2024-03-31T23:11:36.256+0000 7fc89ea30200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-31T23:11:36.602 INFO:journalctl@ceph.mgr.y.smithi142.stdout:Mar 31 23:11:36 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mgr-y[36663]: 2024-03-31T23:11:36.398+0000 7fc89ea30200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-31T23:11:36.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:11:36 smithi046 ceph-mon[32498]: from='client.? 172.21.15.46:0/3829663715' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-03-31T23:11:36.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:11:36 smithi046 ceph-mon[32498]: mgrmap e19: x(active, since 9m), standbys: y, z 2024-03-31T23:11:36.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:11:36 smithi046 ceph-mon[32498]: from='client.? 172.21.15.46:0/2953169307' entity='client.admin' 2024-03-31T23:11:36.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:11:36 smithi046 ceph-mon[32498]: from='client.? 172.21.15.46:0/1754665220' entity='client.admin' 2024-03-31T23:11:36.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:11:36 smithi142 ceph-mon[35271]: from='client.? 172.21.15.46:0/3829663715' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-03-31T23:11:36.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:11:36 smithi142 ceph-mon[35271]: mgrmap e19: x(active, since 9m), standbys: y, z 2024-03-31T23:11:36.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:11:36 smithi142 ceph-mon[35271]: from='client.? 172.21.15.46:0/2953169307' entity='client.admin' 2024-03-31T23:11:36.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:11:36 smithi142 ceph-mon[35271]: from='client.? 172.21.15.46:0/1754665220' entity='client.admin' 2024-03-31T23:11:36.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:36.972+0000 7fe457fff640 1 -- 172.21.15.46:0/509451667 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fe4480a24e0 msgr2=0x7fe4480a49a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.46:6800/162289569 2024-03-31T23:11:36.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:36.972+0000 7fe457fff640 1 --2- 172.21.15.46:0/509451667 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fe4480a24e0 0x7fe4480a49a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-31T23:11:37.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:11:36 smithi162 ceph-mon[36237]: from='client.? 172.21.15.46:0/3829663715' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-03-31T23:11:37.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:11:36 smithi162 ceph-mon[36237]: mgrmap e19: x(active, since 9m), standbys: y, z 2024-03-31T23:11:37.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:11:36 smithi162 ceph-mon[36237]: from='client.? 172.21.15.46:0/2953169307' entity='client.admin' 2024-03-31T23:11:37.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:11:36 smithi162 ceph-mon[36237]: from='client.? 172.21.15.46:0/1754665220' entity='client.admin' 2024-03-31T23:11:37.301 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 31 23:11:36 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mgr-x[32727]: 2024-03-31T23:11:36.963+0000 7f1d8d69c200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-31T23:11:37.301 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 31 23:11:37 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mgr-x[32727]: 2024-03-31T23:11:37.058+0000 7f1d8d69c200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-31T23:11:37.301 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 31 23:11:37 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mgr-x[32727]: 2024-03-31T23:11:37.202+0000 7f1d8d69c200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-31T23:11:37.301 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 31 23:11:37 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mgr-x[32727]: 2024-03-31T23:11:37.300+0000 7f1d8d69c200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-31T23:11:37.395 INFO:journalctl@ceph.mgr.y.smithi142.stdout:Mar 31 23:11:37 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mgr-y[36663]: 2024-03-31T23:11:37.131+0000 7fc89ea30200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-31T23:11:37.395 INFO:journalctl@ceph.mgr.y.smithi142.stdout:Mar 31 23:11:37 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mgr-y[36663]: 2024-03-31T23:11:37.225+0000 7fc89ea30200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-31T23:11:37.395 INFO:journalctl@ceph.mgr.y.smithi142.stdout:Mar 31 23:11:37 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mgr-y[36663]: 2024-03-31T23:11:37.394+0000 7fc89ea30200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-31T23:11:37.405 INFO:journalctl@ceph.mgr.z.smithi162.stdout:Mar 31 23:11:37 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mgr-z[37924]: 2024-03-31T23:11:37.059+0000 7f758cd5e200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-31T23:11:37.405 INFO:journalctl@ceph.mgr.z.smithi162.stdout:Mar 31 23:11:37 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mgr-z[37924]: 2024-03-31T23:11:37.153+0000 7f758cd5e200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-31T23:11:37.405 INFO:journalctl@ceph.mgr.z.smithi162.stdout:Mar 31 23:11:37 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mgr-z[37924]: 2024-03-31T23:11:37.299+0000 7f758cd5e200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-31T23:11:37.406 INFO:journalctl@ceph.mgr.z.smithi162.stdout:Mar 31 23:11:37 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mgr-z[37924]: 2024-03-31T23:11:37.405+0000 7f758cd5e200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-31T23:11:37.611 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 31 23:11:37 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mgr-x[32727]: 2024-03-31T23:11:37.385+0000 7f1d8d69c200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-31T23:11:37.693 INFO:journalctl@ceph.mgr.y.smithi142.stdout:Mar 31 23:11:37 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mgr-y[36663]: 2024-03-31T23:11:37.500+0000 7fc89ea30200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-31T23:11:37.693 INFO:journalctl@ceph.mgr.y.smithi142.stdout:Mar 31 23:11:37 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mgr-y[36663]: 2024-03-31T23:11:37.585+0000 7fc89ea30200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-31T23:11:37.774 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:37.773+0000 7fe457fff640 1 -- 172.21.15.46:0/509451667 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fe4480a24e0 msgr2=0x7fe4480a49a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.46:6800/162289569 2024-03-31T23:11:37.774 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:37.773+0000 7fe457fff640 1 --2- 172.21.15.46:0/509451667 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fe4480a24e0 0x7fe4480a49a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-31T23:11:37.795 INFO:journalctl@ceph.mgr.z.smithi162.stdout:Mar 31 23:11:37 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mgr-z[37924]: 2024-03-31T23:11:37.521+0000 7f758cd5e200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-31T23:11:37.960 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 31 23:11:37 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mgr-x[32727]: 2024-03-31T23:11:37.658+0000 7f1d8d69c200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-31T23:11:37.960 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 31 23:11:37 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mgr-x[32727]: 2024-03-31T23:11:37.772+0000 7f1d8d69c200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-31T23:11:37.960 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 31 23:11:37 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mgr-x[32727]: 2024-03-31T23:11:37.861+0000 7f1d8d69c200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-31T23:11:37.960 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 31 23:11:37 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mgr-x[32727]: 2024-03-31T23:11:37.958+0000 7f1d8d69c200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-31T23:11:38.135 INFO:journalctl@ceph.mgr.z.smithi162.stdout:Mar 31 23:11:37 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mgr-z[37924]: 2024-03-31T23:11:37.840+0000 7f758cd5e200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-31T23:11:38.135 INFO:journalctl@ceph.mgr.z.smithi162.stdout:Mar 31 23:11:37 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mgr-z[37924]: 2024-03-31T23:11:37.951+0000 7f758cd5e200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-31T23:11:38.135 INFO:journalctl@ceph.mgr.z.smithi162.stdout:Mar 31 23:11:38 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mgr-z[37924]: 2024-03-31T23:11:38.037+0000 7f758cd5e200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-31T23:11:38.150 INFO:journalctl@ceph.mgr.y.smithi142.stdout:Mar 31 23:11:37 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mgr-y[36663]: 2024-03-31T23:11:37.854+0000 7fc89ea30200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-31T23:11:38.150 INFO:journalctl@ceph.mgr.y.smithi142.stdout:Mar 31 23:11:37 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mgr-y[36663]: 2024-03-31T23:11:37.966+0000 7fc89ea30200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-31T23:11:38.150 INFO:journalctl@ceph.mgr.y.smithi142.stdout:Mar 31 23:11:38 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mgr-y[36663]: 2024-03-31T23:11:38.053+0000 7fc89ea30200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-31T23:11:38.361 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 31 23:11:38 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mgr-x[32727]: 2024-03-31T23:11:38.138+0000 7f1d8d69c200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-31T23:11:38.443 INFO:journalctl@ceph.mgr.y.smithi142.stdout:Mar 31 23:11:38 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mgr-y[36663]: 2024-03-31T23:11:38.149+0000 7fc89ea30200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-31T23:11:38.443 INFO:journalctl@ceph.mgr.y.smithi142.stdout:Mar 31 23:11:38 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mgr-y[36663]: 2024-03-31T23:11:38.330+0000 7fc89ea30200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-31T23:11:38.546 INFO:journalctl@ceph.mgr.z.smithi162.stdout:Mar 31 23:11:38 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mgr-z[37924]: 2024-03-31T23:11:38.134+0000 7f758cd5e200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-31T23:11:38.546 INFO:journalctl@ceph.mgr.z.smithi162.stdout:Mar 31 23:11:38 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mgr-z[37924]: 2024-03-31T23:11:38.314+0000 7f758cd5e200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-31T23:11:39.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:39.374+0000 7fe457fff640 1 -- 172.21.15.46:0/509451667 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fe4480a24e0 msgr2=0x7fe4480a49a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.46:6800/162289569 2024-03-31T23:11:39.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:39.374+0000 7fe457fff640 1 --2- 172.21.15.46:0/509451667 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fe4480a24e0 0x7fe4480a49a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-03-31T23:11:39.611 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 31 23:11:39 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mgr-x[32727]: 2024-03-31T23:11:39.145+0000 7f1d8d69c200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-31T23:11:39.611 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 31 23:11:39 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mgr-x[32727]: 2024-03-31T23:11:39.284+0000 7f1d8d69c200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-31T23:11:39.787 INFO:journalctl@ceph.mgr.z.smithi162.stdout:Mar 31 23:11:39 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mgr-z[37924]: 2024-03-31T23:11:39.305+0000 7f758cd5e200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-31T23:11:39.787 INFO:journalctl@ceph.mgr.z.smithi162.stdout:Mar 31 23:11:39 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mgr-z[37924]: 2024-03-31T23:11:39.438+0000 7f758cd5e200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-31T23:11:39.787 INFO:journalctl@ceph.mgr.z.smithi162.stdout:Mar 31 23:11:39 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mgr-z[37924]: 2024-03-31T23:11:39.786+0000 7f758cd5e200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-31T23:11:39.929 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 31 23:11:39 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mgr-x[32727]: 2024-03-31T23:11:39.628+0000 7f1d8d69c200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-31T23:11:39.930 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 31 23:11:39 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mgr-x[32727]: 2024-03-31T23:11:39.723+0000 7f1d8d69c200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-31T23:11:39.930 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 31 23:11:39 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mgr-x[32727]: 2024-03-31T23:11:39.928+0000 7f1d8d69c200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-31T23:11:39.943 INFO:journalctl@ceph.mgr.y.smithi142.stdout:Mar 31 23:11:39 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mgr-y[36663]: 2024-03-31T23:11:39.484+0000 7fc89ea30200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-31T23:11:39.943 INFO:journalctl@ceph.mgr.y.smithi142.stdout:Mar 31 23:11:39 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mgr-y[36663]: 2024-03-31T23:11:39.665+0000 7fc89ea30200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-31T23:11:40.046 INFO:journalctl@ceph.mgr.z.smithi162.stdout:Mar 31 23:11:39 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mgr-z[37924]: 2024-03-31T23:11:39.887+0000 7f758cd5e200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-31T23:11:40.361 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 31 23:11:40 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mgr-x[32727]: 2024-03-31T23:11:40.120+0000 7f1d8d69c200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-31T23:11:40.377 INFO:journalctl@ceph.mgr.y.smithi142.stdout:Mar 31 23:11:40 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mgr-y[36663]: 2024-03-31T23:11:40.056+0000 7fc89ea30200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-31T23:11:40.377 INFO:journalctl@ceph.mgr.y.smithi142.stdout:Mar 31 23:11:40 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mgr-y[36663]: 2024-03-31T23:11:40.178+0000 7fc89ea30200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-31T23:11:40.546 INFO:journalctl@ceph.mgr.z.smithi162.stdout:Mar 31 23:11:40 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mgr-z[37924]: 2024-03-31T23:11:40.083+0000 7f758cd5e200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-31T23:11:40.546 INFO:journalctl@ceph.mgr.z.smithi162.stdout:Mar 31 23:11:40 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mgr-z[37924]: 2024-03-31T23:11:40.270+0000 7f758cd5e200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-31T23:11:40.679 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 31 23:11:40 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mgr-x[32727]: 2024-03-31T23:11:40.415+0000 7f1d8d69c200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-31T23:11:40.679 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 31 23:11:40 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mgr-x[32727]: 2024-03-31T23:11:40.678+0000 7f1d8d69c200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-31T23:11:40.693 INFO:journalctl@ceph.mgr.y.smithi142.stdout:Mar 31 23:11:40 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mgr-y[36663]: 2024-03-31T23:11:40.376+0000 7fc89ea30200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-31T23:11:40.693 INFO:journalctl@ceph.mgr.y.smithi142.stdout:Mar 31 23:11:40 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mgr-y[36663]: 2024-03-31T23:11:40.566+0000 7fc89ea30200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-31T23:11:40.909 INFO:journalctl@ceph.mgr.z.smithi162.stdout:Mar 31 23:11:40 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mgr-z[37924]: 2024-03-31T23:11:40.639+0000 7f758cd5e200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-31T23:11:40.910 INFO:journalctl@ceph.mgr.z.smithi162.stdout:Mar 31 23:11:40 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mgr-z[37924]: 2024-03-31T23:11:40.909+0000 7f758cd5e200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-31T23:11:41.111 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 31 23:11:40 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mgr-x[32727]: 2024-03-31T23:11:40.924+0000 7f1d8d69c200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-31T23:11:41.122 INFO:journalctl@ceph.mgr.y.smithi142.stdout:Mar 31 23:11:40 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mgr-y[36663]: 2024-03-31T23:11:40.854+0000 7fc89ea30200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-31T23:11:41.295 INFO:journalctl@ceph.mgr.z.smithi162.stdout:Mar 31 23:11:41 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mgr-z[37924]: 2024-03-31T23:11:41.149+0000 7f758cd5e200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-31T23:11:41.443 INFO:journalctl@ceph.mgr.y.smithi142.stdout:Mar 31 23:11:41 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mgr-y[36663]: 2024-03-31T23:11:41.121+0000 7fc89ea30200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-31T23:11:41.943 INFO:journalctl@ceph.mgr.y.smithi142.stdout:Mar 31 23:11:41 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mgr-y[36663]: 2024-03-31T23:11:41.461+0000 7fc89ea30200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-31T23:11:42.111 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 31 23:11:41 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mgr-x[32727]: 2024-03-31T23:11:41.762+0000 7f1d8d69c200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-31T23:11:42.296 INFO:journalctl@ceph.mgr.z.smithi162.stdout:Mar 31 23:11:41 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mgr-z[37924]: 2024-03-31T23:11:41.983+0000 7f758cd5e200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-31T23:11:42.578 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:42.577+0000 7fe457fff640 1 -- 172.21.15.46:0/509451667 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fe4480a24e0 msgr2=0x7fe4480a49a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.46:6800/162289569 2024-03-31T23:11:42.578 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:42.577+0000 7fe457fff640 1 --2- 172.21.15.46:0/509451667 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fe4480a24e0 0x7fe4480a49a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 6.400000 2024-03-31T23:11:42.611 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 31 23:11:42 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mgr-x[32727]: 2024-03-31T23:11:42.345+0000 7f1d8d69c200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-31T23:11:42.611 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 31 23:11:42 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mgr-x[32727]: 2024-03-31T23:11:42.449+0000 7f1d8d69c200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-31T23:11:42.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:42.687+0000 7fe455ffb640 1 -- 172.21.15.46:0/509451667 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mgrmap(e 20) ==== 137005+0+0 (secure 0 0 0) 0x7fe44c097530 con 0x7fe458102020 2024-03-31T23:11:42.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:42.687+0000 7fe455ffb640 1 -- 172.21.15.46:0/509451667 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fe4480a24e0 msgr2=0x7fe4480a49a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-31T23:11:42.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:42.687+0000 7fe455ffb640 1 --2- 172.21.15.46:0/509451667 >> [v2:172.21.15.46:6800/162289569,v1:172.21.15.46:6801/162289569] conn(0x7fe4480a24e0 0x7fe4480a49a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:42.693 INFO:journalctl@ceph.mgr.y.smithi142.stdout:Mar 31 23:11:42 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mgr-y[36663]: 2024-03-31T23:11:42.360+0000 7fc89ea30200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-31T23:11:42.841 INFO:journalctl@ceph.mgr.z.smithi162.stdout:Mar 31 23:11:42 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mgr-z[37924]: 2024-03-31T23:11:42.574+0000 7f758cd5e200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-31T23:11:42.842 INFO:journalctl@ceph.mgr.z.smithi162.stdout:Mar 31 23:11:42 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mgr-z[37924]: 2024-03-31T23:11:42.684+0000 7f758cd5e200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-31T23:11:42.842 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:11:42 smithi162 ceph-mon[36237]: Active manager daemon x restarted 2024-03-31T23:11:42.842 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:11:42 smithi162 ceph-mon[36237]: Activating manager daemon x 2024-03-31T23:11:42.842 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:11:42 smithi162 ceph-mon[36237]: osdmap e96: 12 total, 12 up, 12 in 2024-03-31T23:11:42.843 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:11:42 smithi162 ceph-mon[36237]: mgrmap e20: x(active, starting, since 0.0113048s), standbys: y, z 2024-03-31T23:11:42.843 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:11:42 smithi162 ceph-mon[36237]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-31T23:11:42.843 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:11:42 smithi162 ceph-mon[36237]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-31T23:11:42.843 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:11:42 smithi162 ceph-mon[36237]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-31T23:11:42.843 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:11:42 smithi162 ceph-mon[36237]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-31T23:11:42.843 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:11:42 smithi162 ceph-mon[36237]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-31T23:11:42.843 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:11:42 smithi162 ceph-mon[36237]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-31T23:11:42.843 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:11:42 smithi162 ceph-mon[36237]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-31T23:11:42.843 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:11:42 smithi162 ceph-mon[36237]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-31T23:11:42.843 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:11:42 smithi162 ceph-mon[36237]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-31T23:11:42.843 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:11:42 smithi162 ceph-mon[36237]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-31T23:11:42.843 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:11:42 smithi162 ceph-mon[36237]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-31T23:11:42.843 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:11:42 smithi162 ceph-mon[36237]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-31T23:11:42.843 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:11:42 smithi162 ceph-mon[36237]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-31T23:11:42.843 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:11:42 smithi162 ceph-mon[36237]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-31T23:11:42.844 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:11:42 smithi162 ceph-mon[36237]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-31T23:11:42.844 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:11:42 smithi162 ceph-mon[36237]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-31T23:11:42.844 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:11:42 smithi162 ceph-mon[36237]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-31T23:11:42.844 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:11:42 smithi162 ceph-mon[36237]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-31T23:11:42.844 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:11:42 smithi162 ceph-mon[36237]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-31T23:11:42.844 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:11:42 smithi162 ceph-mon[36237]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-31T23:11:42.844 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:11:42 smithi162 ceph-mon[36237]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-31T23:11:42.844 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:11:42 smithi162 ceph-mon[36237]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-31T23:11:42.844 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:11:42 smithi162 ceph-mon[36237]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-31T23:11:42.844 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:11:42 smithi162 ceph-mon[36237]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-31T23:11:42.844 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:11:42 smithi162 ceph-mon[36237]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-31T23:11:42.844 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:11:42 smithi162 ceph-mon[36237]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-31T23:11:42.844 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:11:42 smithi162 ceph-mon[36237]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-31T23:11:42.844 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:11:42 smithi162 ceph-mon[36237]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-31T23:11:42.845 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:11:42 smithi162 ceph-mon[36237]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-31T23:11:42.845 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:11:42 smithi162 ceph-mon[36237]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-31T23:11:42.845 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:11:42 smithi162 ceph-mon[36237]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-31T23:11:42.845 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:11:42 smithi162 ceph-mon[36237]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-31T23:11:42.845 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:11:42 smithi162 ceph-mon[36237]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-31T23:11:42.845 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:11:42 smithi162 ceph-mon[36237]: Manager daemon x is now available 2024-03-31T23:11:42.986 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 31 23:11:42 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mgr-x[32727]: 2024-03-31T23:11:42.675+0000 7f1d8d69c200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-31T23:11:42.986 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 31 23:11:42 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mgr-x[32727]: 2024-03-31T23:11:42.924+0000 7f1cec872700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-31T23:11:42.986 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 31 23:11:42 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mgr-x[32727]: 2024-03-31T23:11:42.924+0000 7f1cec872700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-31T23:11:42.986 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 31 23:11:42 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mgr-x[32727]: 2024-03-31T23:11:42.924+0000 7f1cec872700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-31T23:11:42.986 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 31 23:11:42 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mgr-x[32727]: 2024-03-31T23:11:42.924+0000 7f1cec872700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-31T23:11:42.987 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 31 23:11:42 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mgr-x[32727]: 2024-03-31T23:11:42.924+0000 7f1cec872700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-31T23:11:42.987 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 31 23:11:42 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mgr-x[32727]: 2024-03-31T23:11:42.924+0000 7f1cf0079700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-31T23:11:42.987 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 31 23:11:42 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mgr-x[32727]: 2024-03-31T23:11:42.924+0000 7f1cf0079700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-31T23:11:42.987 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 31 23:11:42 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mgr-x[32727]: 2024-03-31T23:11:42.924+0000 7f1cf0079700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-31T23:11:42.987 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 31 23:11:42 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mgr-x[32727]: 2024-03-31T23:11:42.924+0000 7f1cf0079700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-31T23:11:42.987 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 31 23:11:42 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mgr-x[32727]: 2024-03-31T23:11:42.924+0000 7f1cf0079700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-31T23:11:42.987 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:11:42 smithi046 ceph-mon[32498]: Active manager daemon x restarted 2024-03-31T23:11:42.987 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:11:42 smithi046 ceph-mon[32498]: Activating manager daemon x 2024-03-31T23:11:42.987 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:11:42 smithi046 ceph-mon[32498]: osdmap e96: 12 total, 12 up, 12 in 2024-03-31T23:11:42.988 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:11:42 smithi046 ceph-mon[32498]: mgrmap e20: x(active, starting, since 0.0113048s), standbys: y, z 2024-03-31T23:11:42.988 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:11:42 smithi046 ceph-mon[32498]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-31T23:11:42.988 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:11:42 smithi046 ceph-mon[32498]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-31T23:11:42.988 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:11:42 smithi046 ceph-mon[32498]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-31T23:11:42.988 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:11:42 smithi046 ceph-mon[32498]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-31T23:11:42.988 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:11:42 smithi046 ceph-mon[32498]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-31T23:11:42.988 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:11:42 smithi046 ceph-mon[32498]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-31T23:11:42.988 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:11:42 smithi046 ceph-mon[32498]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-31T23:11:42.988 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:11:42 smithi046 ceph-mon[32498]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-31T23:11:42.988 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:11:42 smithi046 ceph-mon[32498]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-31T23:11:42.988 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:11:42 smithi046 ceph-mon[32498]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-31T23:11:42.988 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:11:42 smithi046 ceph-mon[32498]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-31T23:11:42.988 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:11:42 smithi046 ceph-mon[32498]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-31T23:11:42.988 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:11:42 smithi046 ceph-mon[32498]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-31T23:11:42.988 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:11:42 smithi046 ceph-mon[32498]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-31T23:11:42.989 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:11:42 smithi046 ceph-mon[32498]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-31T23:11:42.989 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:11:42 smithi046 ceph-mon[32498]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-31T23:11:42.989 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:11:42 smithi046 ceph-mon[32498]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-31T23:11:42.989 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:11:42 smithi046 ceph-mon[32498]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-31T23:11:42.989 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:11:42 smithi046 ceph-mon[32498]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-31T23:11:42.989 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:11:42 smithi046 ceph-mon[32498]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-31T23:11:42.989 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:11:42 smithi046 ceph-mon[32498]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-31T23:11:42.989 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:11:42 smithi046 ceph-mon[32498]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-31T23:11:42.989 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:11:42 smithi046 ceph-mon[32498]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-31T23:11:42.989 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:11:42 smithi046 ceph-mon[32498]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-31T23:11:42.989 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:11:42 smithi046 ceph-mon[32498]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-31T23:11:42.989 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:11:42 smithi046 ceph-mon[32498]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-31T23:11:42.989 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:11:42 smithi046 ceph-mon[32498]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-31T23:11:42.989 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:11:42 smithi046 ceph-mon[32498]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-31T23:11:42.989 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:11:42 smithi046 ceph-mon[32498]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-31T23:11:42.989 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:11:42 smithi046 ceph-mon[32498]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-31T23:11:42.990 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:11:42 smithi046 ceph-mon[32498]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-31T23:11:42.990 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:11:42 smithi046 ceph-mon[32498]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-31T23:11:42.990 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:11:42 smithi046 ceph-mon[32498]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-31T23:11:42.990 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:11:42 smithi046 ceph-mon[32498]: Manager daemon x is now available 2024-03-31T23:11:43.066 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:11:42 smithi142 ceph-mon[35271]: Active manager daemon x restarted 2024-03-31T23:11:43.066 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:11:42 smithi142 ceph-mon[35271]: Activating manager daemon x 2024-03-31T23:11:43.066 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:11:42 smithi142 ceph-mon[35271]: osdmap e96: 12 total, 12 up, 12 in 2024-03-31T23:11:43.066 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:11:42 smithi142 ceph-mon[35271]: mgrmap e20: x(active, starting, since 0.0113048s), standbys: y, z 2024-03-31T23:11:43.066 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:11:42 smithi142 ceph-mon[35271]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-31T23:11:43.067 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:11:42 smithi142 ceph-mon[35271]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-31T23:11:43.067 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:11:42 smithi142 ceph-mon[35271]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-31T23:11:43.067 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:11:42 smithi142 ceph-mon[35271]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-31T23:11:43.067 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:11:42 smithi142 ceph-mon[35271]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-31T23:11:43.067 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:11:42 smithi142 ceph-mon[35271]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-31T23:11:43.067 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:11:42 smithi142 ceph-mon[35271]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-31T23:11:43.067 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:11:42 smithi142 ceph-mon[35271]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-31T23:11:43.067 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:11:42 smithi142 ceph-mon[35271]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-31T23:11:43.067 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:11:42 smithi142 ceph-mon[35271]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-31T23:11:43.068 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:11:42 smithi142 ceph-mon[35271]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-31T23:11:43.068 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:11:42 smithi142 ceph-mon[35271]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-31T23:11:43.068 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:11:42 smithi142 ceph-mon[35271]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-31T23:11:43.068 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:11:42 smithi142 ceph-mon[35271]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-31T23:11:43.068 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:11:42 smithi142 ceph-mon[35271]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-31T23:11:43.068 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:11:42 smithi142 ceph-mon[35271]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-31T23:11:43.068 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:11:42 smithi142 ceph-mon[35271]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-31T23:11:43.068 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:11:42 smithi142 ceph-mon[35271]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-31T23:11:43.068 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:11:42 smithi142 ceph-mon[35271]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-31T23:11:43.068 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:11:42 smithi142 ceph-mon[35271]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-31T23:11:43.068 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:11:42 smithi142 ceph-mon[35271]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-31T23:11:43.068 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:11:42 smithi142 ceph-mon[35271]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-31T23:11:43.068 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:11:42 smithi142 ceph-mon[35271]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-31T23:11:43.068 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:11:42 smithi142 ceph-mon[35271]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-31T23:11:43.069 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:11:42 smithi142 ceph-mon[35271]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-31T23:11:43.069 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:11:42 smithi142 ceph-mon[35271]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-31T23:11:43.069 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:11:42 smithi142 ceph-mon[35271]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-31T23:11:43.069 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:11:42 smithi142 ceph-mon[35271]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-31T23:11:43.069 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:11:42 smithi142 ceph-mon[35271]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-31T23:11:43.069 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:11:42 smithi142 ceph-mon[35271]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-31T23:11:43.069 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:11:42 smithi142 ceph-mon[35271]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-31T23:11:43.069 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:11:42 smithi142 ceph-mon[35271]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-31T23:11:43.069 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:11:42 smithi142 ceph-mon[35271]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-31T23:11:43.069 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:11:42 smithi142 ceph-mon[35271]: Manager daemon x is now available 2024-03-31T23:11:43.296 INFO:journalctl@ceph.mgr.z.smithi162.stdout:Mar 31 23:11:42 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mgr-z[37924]: 2024-03-31T23:11:42.937+0000 7f758cd5e200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-31T23:11:43.443 INFO:journalctl@ceph.mgr.y.smithi142.stdout:Mar 31 23:11:43 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mgr-y[36663]: 2024-03-31T23:11:43.064+0000 7fc89ea30200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-31T23:11:43.443 INFO:journalctl@ceph.mgr.y.smithi142.stdout:Mar 31 23:11:43 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mgr-y[36663]: 2024-03-31T23:11:43.204+0000 7fc89ea30200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-31T23:11:43.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:43.694+0000 7fe455ffb640 1 -- 172.21.15.46:0/509451667 <== mon.0 v2:172.21.15.46:3300/0 8 ==== mgrmap(e 21) ==== 137132+0+0 (secure 0 0 0) 0x7fe44c0cbf80 con 0x7fe458102020 2024-03-31T23:11:43.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:43.696+0000 7fe455ffb640 1 --2- 172.21.15.46:0/509451667 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fe44806a730 0x7fe44806ab10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:43.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:43.696+0000 7fe455ffb640 1 -- 172.21.15.46:0/509451667 --> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) -- 0x7fe44c009090 con 0x7fe44806a730 2024-03-31T23:11:43.698 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:43.697+0000 7fe457fff640 1 --2- 172.21.15.46:0/509451667 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fe44806a730 0x7fe44806ab10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:43.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:43.701+0000 7fe457fff640 1 --2- 172.21.15.46:0/509451667 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fe44806a730 0x7fe44806ab10 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fe458101700 tx=0x7fe444005f90 comp rx=0 tx=0).ready entity=mgr.15654 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:11:43.732 INFO:journalctl@ceph.mgr.y.smithi142.stdout:Mar 31 23:11:43 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mgr-y[36663]: 2024-03-31T23:11:43.476+0000 7fc89ea30200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-31T23:11:43.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:43.764+0000 7fe455ffb640 1 -- 172.21.15.46:0/509451667 <== mgr.15654 v2:172.21.15.46:6800/685378567 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7fe44c009090 con 0x7fe44806a730 2024-03-31T23:11:43.765 INFO:teuthology.orchestra.run.smithi046.stdout:Schedule set for path / 2024-03-31T23:11:43.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:43.766+0000 7fe45dba0640 1 -- 172.21.15.46:0/509451667 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fe44806a730 msgr2=0x7fe44806ab10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:43.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:43.766+0000 7fe45dba0640 1 --2- 172.21.15.46:0/509451667 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fe44806a730 0x7fe44806ab10 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fe458101700 tx=0x7fe444005f90 comp rx=0 tx=0).stop 2024-03-31T23:11:43.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:43.766+0000 7fe45dba0640 1 -- 172.21.15.46:0/509451667 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe458102020 msgr2=0x7fe458101990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:43.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:43.766+0000 7fe45dba0640 1 --2- 172.21.15.46:0/509451667 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe458102020 0x7fe458101990 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7fe44c00d950 tx=0x7fe44c00de10 comp rx=0 tx=0).stop 2024-03-31T23:11:43.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:43.766+0000 7fe45d39f640 1 -- 172.21.15.46:0/509451667 reap_dead start 2024-03-31T23:11:43.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:43.766+0000 7fe45dba0640 1 -- 172.21.15.46:0/509451667 shutdown_connections 2024-03-31T23:11:43.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:43.766+0000 7fe45dba0640 1 -- 172.21.15.46:0/509451667 >> 172.21.15.46:0/509451667 conn(0x7fe4580fba30 msgr2=0x7fe458102960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:11:43.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:43.767+0000 7fe45dba0640 1 -- 172.21.15.46:0/509451667 shutdown_connections 2024-03-31T23:11:43.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:43.767+0000 7fe45dba0640 1 -- 172.21.15.46:0/509451667 wait complete. 2024-03-31T23:11:43.778 DEBUG:teuthology.orchestra.run.smithi046:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule retention add --fs=cephfs --path=/ --retention-spec-or-period=6m3h' 2024-03-31T23:11:43.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:43.885+0000 7fcf4e525640 1 -- 172.21.15.46:0/1118673977 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcf48100900 msgr2=0x7fcf48100d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:43.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:43.885+0000 7fcf4e525640 1 --2- 172.21.15.46:0/1118673977 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcf48100900 0x7fcf48100d80 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7fcf3c009900 tx=0x7fcf3c02bc30 comp rx=0 tx=0).stop 2024-03-31T23:11:43.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:43.885+0000 7fcf4e525640 1 -- 172.21.15.46:0/1118673977 shutdown_connections 2024-03-31T23:11:43.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:43.885+0000 7fcf4e525640 1 --2- 172.21.15.46:0/1118673977 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcf481012c0 0x7fcf48108b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:43.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:43.885+0000 7fcf4e525640 1 --2- 172.21.15.46:0/1118673977 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcf48100900 0x7fcf48100d80 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:43.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:43.885+0000 7fcf4e525640 1 --2- 172.21.15.46:0/1118673977 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcf48101cb0 0x7fcf481020d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:43.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:43.885+0000 7fcf4e525640 1 -- 172.21.15.46:0/1118673977 >> 172.21.15.46:0/1118673977 conn(0x7fcf480fb370 msgr2=0x7fcf480fd7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:11:43.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:43.885+0000 7fcf4e525640 1 -- 172.21.15.46:0/1118673977 shutdown_connections 2024-03-31T23:11:43.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:43.885+0000 7fcf4e525640 1 -- 172.21.15.46:0/1118673977 wait complete. 2024-03-31T23:11:43.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:43.885+0000 7fcf4e525640 1 Processor -- start 2024-03-31T23:11:43.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:43.885+0000 7fcf4e525640 1 -- start start 2024-03-31T23:11:43.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:43.886+0000 7fcf4e525640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcf48100900 0x7fcf481003c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:43.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:43.886+0000 7fcf4e525640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcf481012c0 0x7fcf4819c630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:43.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:43.886+0000 7fcf4e525640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcf48101cb0 0x7fcf4819eca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:43.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:43.886+0000 7fcf4e525640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fcf4810c1c0 con 0x7fcf48101cb0 2024-03-31T23:11:43.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:43.886+0000 7fcf4e525640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fcf4810c040 con 0x7fcf48100900 2024-03-31T23:11:43.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:43.886+0000 7fcf4e525640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fcf4810c340 con 0x7fcf481012c0 2024-03-31T23:11:43.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:43.886+0000 7fcf4cac2640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcf48101cb0 0x7fcf4819eca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:43.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:43.886+0000 7fcf4cac2640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcf48101cb0 0x7fcf4819eca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:48428/0 (socket says 172.21.15.46:48428) 2024-03-31T23:11:43.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:43.886+0000 7fcf4cac2640 1 -- 172.21.15.46:0/668025366 learned_addr learned my addr 172.21.15.46:0/668025366 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:11:43.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:43.886+0000 7fcf47fff640 1 --2- 172.21.15.46:0/668025366 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcf48100900 0x7fcf481003c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:43.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:43.886+0000 7fcf477fe640 1 --2- 172.21.15.46:0/668025366 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcf481012c0 0x7fcf4819c630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:43.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:43.886+0000 7fcf4cac2640 1 -- 172.21.15.46:0/668025366 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcf481012c0 msgr2=0x7fcf4819c630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:43.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:43.886+0000 7fcf4cac2640 1 --2- 172.21.15.46:0/668025366 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcf481012c0 0x7fcf4819c630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:43.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:43.886+0000 7fcf4cac2640 1 -- 172.21.15.46:0/668025366 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcf48100900 msgr2=0x7fcf481003c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:43.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:43.886+0000 7fcf4cac2640 1 --2- 172.21.15.46:0/668025366 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcf48100900 0x7fcf481003c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:43.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:43.886+0000 7fcf4cac2640 1 -- 172.21.15.46:0/668025366 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcf4819f560 con 0x7fcf48101cb0 2024-03-31T23:11:43.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:43.886+0000 7fcf4cac2640 1 --2- 172.21.15.46:0/668025366 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcf48101cb0 0x7fcf4819eca0 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7fcf4000eb10 tx=0x7fcf4000c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:11:43.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:43.886+0000 7fcf457fa640 1 -- 172.21.15.46:0/668025366 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf40004b20 con 0x7fcf48101cb0 2024-03-31T23:11:43.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:43.887+0000 7fcf457fa640 1 -- 172.21.15.46:0/668025366 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fcf40004cc0 con 0x7fcf48101cb0 2024-03-31T23:11:43.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:43.887+0000 7fcf4e525640 1 -- 172.21.15.46:0/668025366 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcf4819f850 con 0x7fcf48101cb0 2024-03-31T23:11:43.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:43.887+0000 7fcf457fa640 1 -- 172.21.15.46:0/668025366 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf40010690 con 0x7fcf48101cb0 2024-03-31T23:11:43.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:43.887+0000 7fcf4e525640 1 -- 172.21.15.46:0/668025366 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcf4819fd20 con 0x7fcf48101cb0 2024-03-31T23:11:43.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:43.888+0000 7fcf4e525640 1 -- 172.21.15.46:0/668025366 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcf48107ab0 con 0x7fcf48101cb0 2024-03-31T23:11:43.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:43.888+0000 7fcf457fa640 1 -- 172.21.15.46:0/668025366 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 21) ==== 137132+0+0 (secure 0 0 0) 0x7fcf40005020 con 0x7fcf48101cb0 2024-03-31T23:11:43.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:43.889+0000 7fcf457fa640 1 --2- 172.21.15.46:0/668025366 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fcf300a2350 0x7fcf300a4810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:43.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:43.889+0000 7fcf47fff640 1 --2- 172.21.15.46:0/668025366 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fcf300a2350 0x7fcf300a4810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:43.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:43.889+0000 7fcf457fa640 1 -- 172.21.15.46:0/668025366 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7fcf400ced80 con 0x7fcf48101cb0 2024-03-31T23:11:43.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:43.890+0000 7fcf47fff640 1 --2- 172.21.15.46:0/668025366 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fcf300a2350 0x7fcf300a4810 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fcf38005fd0 tx=0x7fcf3800b410 comp rx=0 tx=0).ready entity=mgr.15654 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:11:43.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:43.894+0000 7fcf457fa640 1 -- 172.21.15.46:0/668025366 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fcf4009bd60 con 0x7fcf48101cb0 2024-03-31T23:11:43.994 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:11:43 smithi142 ceph-mon[35271]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-31T23:11:43.994 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:11:43 smithi142 ceph-mon[35271]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-31T23:11:43.994 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:11:43 smithi142 ceph-mon[35271]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-31T23:11:43.995 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:11:43 smithi142 ceph-mon[35271]: Standby manager daemon z restarted 2024-03-31T23:11:43.995 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:11:43 smithi142 ceph-mon[35271]: Standby manager daemon z started 2024-03-31T23:11:43.995 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:11:43 smithi142 ceph-mon[35271]: from='mgr.? 172.21.15.162:0/855911626' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-31T23:11:43.995 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:11:43 smithi142 ceph-mon[35271]: from='mgr.? 172.21.15.162:0/855911626' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-31T23:11:43.995 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:11:43 smithi142 ceph-mon[35271]: from='mgr.? 172.21.15.162:0/855911626' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-31T23:11:43.995 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:11:43 smithi142 ceph-mon[35271]: from='mgr.? 172.21.15.162:0/855911626' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-31T23:11:43.995 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:11:43 smithi142 ceph-mon[35271]: [31/Mar/2024:23:11:43] ENGINE Bus STARTING 2024-03-31T23:11:43.995 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:11:43 smithi142 ceph-mon[35271]: [31/Mar/2024:23:11:43] ENGINE Serving on http://172.21.15.46:8765 2024-03-31T23:11:43.995 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:11:43 smithi142 ceph-mon[35271]: [31/Mar/2024:23:11:43] ENGINE Serving on https://172.21.15.46:7150 2024-03-31T23:11:43.995 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:11:43 smithi142 ceph-mon[35271]: [31/Mar/2024:23:11:43] ENGINE Bus STARTED 2024-03-31T23:11:43.995 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:11:43 smithi142 ceph-mon[35271]: Standby manager daemon y restarted 2024-03-31T23:11:43.995 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:11:43 smithi142 ceph-mon[35271]: Standby manager daemon y started 2024-03-31T23:11:43.995 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:11:43 smithi142 ceph-mon[35271]: from='mgr.? 172.21.15.142:0/2357910658' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-31T23:11:43.995 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:11:43 smithi142 ceph-mon[35271]: from='mgr.? 172.21.15.142:0/2357910658' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-31T23:11:43.996 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:11:43 smithi142 ceph-mon[35271]: from='mgr.? 172.21.15.142:0/2357910658' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-31T23:11:43.996 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:11:43 smithi142 ceph-mon[35271]: from='mgr.? 172.21.15.142:0/2357910658' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-31T23:11:43.996 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:11:43 smithi142 ceph-mon[35271]: mgrmap e21: x(active, since 1.02138s), standbys: z, y 2024-03-31T23:11:44.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:11:43 smithi162 ceph-mon[36237]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-31T23:11:44.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:11:43 smithi162 ceph-mon[36237]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-31T23:11:44.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:11:43 smithi162 ceph-mon[36237]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-31T23:11:44.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:11:43 smithi162 ceph-mon[36237]: Standby manager daemon z restarted 2024-03-31T23:11:44.047 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:11:43 smithi162 ceph-mon[36237]: Standby manager daemon z started 2024-03-31T23:11:44.047 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:11:43 smithi162 ceph-mon[36237]: from='mgr.? 172.21.15.162:0/855911626' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-31T23:11:44.047 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:11:43 smithi162 ceph-mon[36237]: from='mgr.? 172.21.15.162:0/855911626' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-31T23:11:44.047 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:11:43 smithi162 ceph-mon[36237]: from='mgr.? 172.21.15.162:0/855911626' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-31T23:11:44.047 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:11:43 smithi162 ceph-mon[36237]: from='mgr.? 172.21.15.162:0/855911626' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-31T23:11:44.047 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:11:43 smithi162 ceph-mon[36237]: [31/Mar/2024:23:11:43] ENGINE Bus STARTING 2024-03-31T23:11:44.047 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:11:43 smithi162 ceph-mon[36237]: [31/Mar/2024:23:11:43] ENGINE Serving on http://172.21.15.46:8765 2024-03-31T23:11:44.047 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:11:43 smithi162 ceph-mon[36237]: [31/Mar/2024:23:11:43] ENGINE Serving on https://172.21.15.46:7150 2024-03-31T23:11:44.047 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:11:43 smithi162 ceph-mon[36237]: [31/Mar/2024:23:11:43] ENGINE Bus STARTED 2024-03-31T23:11:44.047 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:11:43 smithi162 ceph-mon[36237]: Standby manager daemon y restarted 2024-03-31T23:11:44.047 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:11:43 smithi162 ceph-mon[36237]: Standby manager daemon y started 2024-03-31T23:11:44.047 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:11:43 smithi162 ceph-mon[36237]: from='mgr.? 172.21.15.142:0/2357910658' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-31T23:11:44.047 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:11:43 smithi162 ceph-mon[36237]: from='mgr.? 172.21.15.142:0/2357910658' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-31T23:11:44.047 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:11:43 smithi162 ceph-mon[36237]: from='mgr.? 172.21.15.142:0/2357910658' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-31T23:11:44.048 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:11:43 smithi162 ceph-mon[36237]: from='mgr.? 172.21.15.142:0/2357910658' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-31T23:11:44.048 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:11:43 smithi162 ceph-mon[36237]: mgrmap e21: x(active, since 1.02138s), standbys: z, y 2024-03-31T23:11:44.060 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:11:43 smithi046 ceph-mon[32498]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-31T23:11:44.060 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:11:43 smithi046 ceph-mon[32498]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-31T23:11:44.060 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:11:43 smithi046 ceph-mon[32498]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-31T23:11:44.060 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:11:43 smithi046 ceph-mon[32498]: Standby manager daemon z restarted 2024-03-31T23:11:44.060 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:11:43 smithi046 ceph-mon[32498]: Standby manager daemon z started 2024-03-31T23:11:44.060 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:11:43 smithi046 ceph-mon[32498]: from='mgr.? 172.21.15.162:0/855911626' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-31T23:11:44.060 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:11:43 smithi046 ceph-mon[32498]: from='mgr.? 172.21.15.162:0/855911626' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-31T23:11:44.060 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:11:43 smithi046 ceph-mon[32498]: from='mgr.? 172.21.15.162:0/855911626' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-31T23:11:44.060 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:11:43 smithi046 ceph-mon[32498]: from='mgr.? 172.21.15.162:0/855911626' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-31T23:11:44.060 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:11:43 smithi046 ceph-mon[32498]: [31/Mar/2024:23:11:43] ENGINE Bus STARTING 2024-03-31T23:11:44.060 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:11:43 smithi046 ceph-mon[32498]: [31/Mar/2024:23:11:43] ENGINE Serving on http://172.21.15.46:8765 2024-03-31T23:11:44.061 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:11:43 smithi046 ceph-mon[32498]: [31/Mar/2024:23:11:43] ENGINE Serving on https://172.21.15.46:7150 2024-03-31T23:11:44.061 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:11:43 smithi046 ceph-mon[32498]: [31/Mar/2024:23:11:43] ENGINE Bus STARTED 2024-03-31T23:11:44.061 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:11:43 smithi046 ceph-mon[32498]: Standby manager daemon y restarted 2024-03-31T23:11:44.061 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:11:43 smithi046 ceph-mon[32498]: Standby manager daemon y started 2024-03-31T23:11:44.061 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:11:43 smithi046 ceph-mon[32498]: from='mgr.? 172.21.15.142:0/2357910658' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-31T23:11:44.061 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:11:43 smithi046 ceph-mon[32498]: from='mgr.? 172.21.15.142:0/2357910658' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-31T23:11:44.061 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:11:43 smithi046 ceph-mon[32498]: from='mgr.? 172.21.15.142:0/2357910658' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-31T23:11:44.061 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:11:43 smithi046 ceph-mon[32498]: from='mgr.? 172.21.15.142:0/2357910658' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-31T23:11:44.061 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:11:43 smithi046 ceph-mon[32498]: mgrmap e21: x(active, since 1.02138s), standbys: z, y 2024-03-31T23:11:44.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:44.103+0000 7fcf4e525640 1 -- 172.21.15.46:0/668025366 --> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] -- mgr_command(tid 0: {"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}) -- 0x7fcf48061c30 con 0x7fcf300a2350 2024-03-31T23:11:44.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:44.162+0000 7fcf457fa640 1 -- 172.21.15.46:0/668025366 <== mgr.15654 v2:172.21.15.46:6800/685378567 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+26 (secure 0 0 0) 0x7fcf48061c30 con 0x7fcf300a2350 2024-03-31T23:11:44.163 INFO:teuthology.orchestra.run.smithi046.stdout:Retention added to path / 2024-03-31T23:11:44.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:44.164+0000 7fcf4e525640 1 -- 172.21.15.46:0/668025366 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fcf300a2350 msgr2=0x7fcf300a4810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:44.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:44.164+0000 7fcf4e525640 1 --2- 172.21.15.46:0/668025366 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fcf300a2350 0x7fcf300a4810 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fcf38005fd0 tx=0x7fcf3800b410 comp rx=0 tx=0).stop 2024-03-31T23:11:44.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:44.164+0000 7fcf4e525640 1 -- 172.21.15.46:0/668025366 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcf48101cb0 msgr2=0x7fcf4819eca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:44.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:44.164+0000 7fcf4e525640 1 --2- 172.21.15.46:0/668025366 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcf48101cb0 0x7fcf4819eca0 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7fcf4000eb10 tx=0x7fcf4000c2d0 comp rx=0 tx=0).stop 2024-03-31T23:11:44.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:44.164+0000 7fcf4e525640 1 -- 172.21.15.46:0/668025366 shutdown_connections 2024-03-31T23:11:44.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:44.164+0000 7fcf4e525640 1 --2- 172.21.15.46:0/668025366 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fcf300a2350 0x7fcf300a4810 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:44.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:44.164+0000 7fcf4e525640 1 --2- 172.21.15.46:0/668025366 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcf48101cb0 0x7fcf4819eca0 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:44.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:44.164+0000 7fcf4e525640 1 --2- 172.21.15.46:0/668025366 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcf481012c0 0x7fcf4819c630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:44.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:44.164+0000 7fcf4e525640 1 --2- 172.21.15.46:0/668025366 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcf48100900 0x7fcf481003c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:44.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:44.164+0000 7fcf4e525640 1 -- 172.21.15.46:0/668025366 >> 172.21.15.46:0/668025366 conn(0x7fcf480fb370 msgr2=0x7fcf48104c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:11:44.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:44.164+0000 7fcf4e525640 1 -- 172.21.15.46:0/668025366 shutdown_connections 2024-03-31T23:11:44.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:44.164+0000 7fcf4e525640 1 -- 172.21.15.46:0/668025366 wait complete. 2024-03-31T23:11:44.176 DEBUG:teuthology.orchestra.run.smithi046:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule status --fs=cephfs --path=/' 2024-03-31T23:11:44.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:44.281+0000 7f452a348640 1 -- 172.21.15.46:0/3490389034 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4524102980 msgr2=0x7f4524102e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:44.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:44.281+0000 7f452a348640 1 --2- 172.21.15.46:0/3490389034 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4524102980 0x7f4524102e00 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f450c009900 tx=0x7f450c02bc30 comp rx=0 tx=0).stop 2024-03-31T23:11:44.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:44.281+0000 7f452a348640 1 -- 172.21.15.46:0/3490389034 shutdown_connections 2024-03-31T23:11:44.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:44.281+0000 7f452a348640 1 --2- 172.21.15.46:0/3490389034 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4524103340 0x7f4524109bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:44.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:44.281+0000 7f452a348640 1 --2- 172.21.15.46:0/3490389034 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4524102980 0x7f4524102e00 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:44.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:44.281+0000 7f452a348640 1 --2- 172.21.15.46:0/3490389034 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4524101790 0x7f4524101bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:44.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:44.281+0000 7f452a348640 1 -- 172.21.15.46:0/3490389034 >> 172.21.15.46:0/3490389034 conn(0x7f45240fcce0 msgr2=0x7f45240ff140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:11:44.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:44.281+0000 7f452a348640 1 -- 172.21.15.46:0/3490389034 shutdown_connections 2024-03-31T23:11:44.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:44.281+0000 7f452a348640 1 -- 172.21.15.46:0/3490389034 wait complete. 2024-03-31T23:11:44.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:44.282+0000 7f452a348640 1 Processor -- start 2024-03-31T23:11:44.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:44.282+0000 7f452a348640 1 -- start start 2024-03-31T23:11:44.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:44.282+0000 7f452a348640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4524101790 0x7f4524075500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:44.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:44.282+0000 7f452a348640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4524102980 0x7f4524073b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:44.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:44.282+0000 7f452a348640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4524103340 0x7f4524074080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:44.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:44.282+0000 7f452a348640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f452410d2a0 con 0x7f4524102980 2024-03-31T23:11:44.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:44.282+0000 7f452a348640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f452410d120 con 0x7f4524103340 2024-03-31T23:11:44.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:44.282+0000 7f452a348640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f452410d420 con 0x7f4524101790 2024-03-31T23:11:44.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:44.282+0000 7f45237fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4524102980 0x7f4524073b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:44.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:44.282+0000 7f45237fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4524102980 0x7f4524073b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:48458/0 (socket says 172.21.15.46:48458) 2024-03-31T23:11:44.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:44.282+0000 7f45237fe640 1 -- 172.21.15.46:0/2764887249 learned_addr learned my addr 172.21.15.46:0/2764887249 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:11:44.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:44.282+0000 7f4523fff640 1 --2- 172.21.15.46:0/2764887249 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4524101790 0x7f4524075500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:44.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:44.282+0000 7f45288e5640 1 --2- 172.21.15.46:0/2764887249 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4524103340 0x7f4524074080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:44.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:44.283+0000 7f45237fe640 1 -- 172.21.15.46:0/2764887249 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4524101790 msgr2=0x7f4524075500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:44.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:44.283+0000 7f45237fe640 1 --2- 172.21.15.46:0/2764887249 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4524101790 0x7f4524075500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:44.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:44.283+0000 7f45237fe640 1 -- 172.21.15.46:0/2764887249 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4524103340 msgr2=0x7f4524074080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:44.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:44.283+0000 7f45237fe640 1 --2- 172.21.15.46:0/2764887249 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4524103340 0x7f4524074080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:44.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:44.283+0000 7f45237fe640 1 -- 172.21.15.46:0/2764887249 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4524074910 con 0x7f4524102980 2024-03-31T23:11:44.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:44.283+0000 7f45237fe640 1 --2- 172.21.15.46:0/2764887249 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4524102980 0x7f4524073b40 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f450c006010 tx=0x7f450c004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:11:44.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:44.283+0000 7f45217fa640 1 -- 172.21.15.46:0/2764887249 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f450c043020 con 0x7f4524102980 2024-03-31T23:11:44.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:44.283+0000 7f45217fa640 1 -- 172.21.15.46:0/2764887249 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f450c004440 con 0x7f4524102980 2024-03-31T23:11:44.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:44.283+0000 7f45217fa640 1 -- 172.21.15.46:0/2764887249 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f450c003bf0 con 0x7f4524102980 2024-03-31T23:11:44.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:44.283+0000 7f452a348640 1 -- 172.21.15.46:0/2764887249 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f45241a1a30 con 0x7f4524102980 2024-03-31T23:11:44.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:44.283+0000 7f452a348640 1 -- 172.21.15.46:0/2764887249 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f45241a1f80 con 0x7f4524102980 2024-03-31T23:11:44.288 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:44.284+0000 7f452a348640 1 -- 172.21.15.46:0/2764887249 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4524108b30 con 0x7f4524102980 2024-03-31T23:11:44.288 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:44.285+0000 7f45217fa640 1 -- 172.21.15.46:0/2764887249 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 21) ==== 137132+0+0 (secure 0 0 0) 0x7f450c046020 con 0x7f4524102980 2024-03-31T23:11:44.288 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:44.286+0000 7f45217fa640 1 --2- 172.21.15.46:0/2764887249 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f45100a2350 0x7f45100a4810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:44.288 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:44.286+0000 7f4523fff640 1 --2- 172.21.15.46:0/2764887249 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f45100a2350 0x7f45100a4810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:44.288 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:44.286+0000 7f45217fa640 1 -- 172.21.15.46:0/2764887249 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7f450c0f10a0 con 0x7f4524102980 2024-03-31T23:11:44.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:44.286+0000 7f4523fff640 1 --2- 172.21.15.46:0/2764887249 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f45100a2350 0x7f45100a4810 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f45241027e0 tx=0x7f4514005f90 comp rx=0 tx=0).ready entity=mgr.15654 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:11:44.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:44.289+0000 7f45217fa640 1 -- 172.21.15.46:0/2764887249 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f450c04d020 con 0x7f4524102980 2024-03-31T23:11:44.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:44.495+0000 7f452a348640 1 -- 172.21.15.46:0/2764887249 --> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}) -- 0x7f4524061b90 con 0x7f45100a2350 2024-03-31T23:11:44.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:44.502+0000 7f45217fa640 1 -- 172.21.15.46:0/2764887249 <== mgr.15654 v2:172.21.15.46:6800/685378567 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+298 (secure 0 0 0) 0x7f4524061b90 con 0x7f45100a2350 2024-03-31T23:11:44.503 INFO:teuthology.orchestra.run.smithi046.stdout:{"fs": "cephfs", "subvol": null, "group": null, "path": "/", "rel_path": "/", "schedule": "1m", "retention": {"m": 6, "h": 3}, "start": "2024-03-31T00:00:00", "created": "2024-03-31T23:11:43", "first": null, "last": null, "last_pruned": null, "created_count": 0, "pruned_count": 0, "active": true} 2024-03-31T23:11:44.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:44.504+0000 7f452a348640 1 -- 172.21.15.46:0/2764887249 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f45100a2350 msgr2=0x7f45100a4810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:44.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:44.504+0000 7f452a348640 1 --2- 172.21.15.46:0/2764887249 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f45100a2350 0x7f45100a4810 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f45241027e0 tx=0x7f4514005f90 comp rx=0 tx=0).stop 2024-03-31T23:11:44.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:44.504+0000 7f452a348640 1 -- 172.21.15.46:0/2764887249 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4524102980 msgr2=0x7f4524073b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:44.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:44.504+0000 7f452a348640 1 --2- 172.21.15.46:0/2764887249 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4524102980 0x7f4524073b40 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f450c006010 tx=0x7f450c004040 comp rx=0 tx=0).stop 2024-03-31T23:11:44.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:44.504+0000 7f452a348640 1 -- 172.21.15.46:0/2764887249 shutdown_connections 2024-03-31T23:11:44.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:44.504+0000 7f452a348640 1 --2- 172.21.15.46:0/2764887249 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f45100a2350 0x7f45100a4810 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:44.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:44.504+0000 7f452a348640 1 --2- 172.21.15.46:0/2764887249 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4524103340 0x7f4524074080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:44.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:44.504+0000 7f452a348640 1 --2- 172.21.15.46:0/2764887249 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4524102980 0x7f4524073b40 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:44.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:44.504+0000 7f452a348640 1 --2- 172.21.15.46:0/2764887249 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4524101790 0x7f4524075500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:44.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:44.504+0000 7f452a348640 1 -- 172.21.15.46:0/2764887249 >> 172.21.15.46:0/2764887249 conn(0x7f45240fcce0 msgr2=0x7f4524105d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:11:44.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:44.504+0000 7f452a348640 1 -- 172.21.15.46:0/2764887249 shutdown_connections 2024-03-31T23:11:44.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:44.504+0000 7f452a348640 1 -- 172.21.15.46:0/2764887249 wait complete. 2024-03-31T23:11:44.515 DEBUG:teuthology.orchestra.run.smithi046:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true' 2024-03-31T23:11:44.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:44.622+0000 7f4c7b1e2640 1 -- 172.21.15.46:0/189245179 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4c741017a0 msgr2=0x7f4c74101bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:44.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:44.622+0000 7f4c7b1e2640 1 --2- 172.21.15.46:0/189245179 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4c741017a0 0x7f4c74101bc0 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f4c6c009ca0 tx=0x7f4c6c02c0f0 comp rx=0 tx=0).stop 2024-03-31T23:11:44.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:44.622+0000 7f4c7b1e2640 1 -- 172.21.15.46:0/189245179 shutdown_connections 2024-03-31T23:11:44.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:44.622+0000 7f4c7b1e2640 1 --2- 172.21.15.46:0/189245179 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4c74103350 0x7f4c74109be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:44.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:44.622+0000 7f4c7b1e2640 1 --2- 172.21.15.46:0/189245179 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4c74102990 0x7f4c74102e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:44.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:44.622+0000 7f4c7b1e2640 1 --2- 172.21.15.46:0/189245179 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4c741017a0 0x7f4c74101bc0 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:44.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:44.622+0000 7f4c7b1e2640 1 -- 172.21.15.46:0/189245179 >> 172.21.15.46:0/189245179 conn(0x7f4c740fcd10 msgr2=0x7f4c740ff150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:11:44.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:44.622+0000 7f4c7b1e2640 1 -- 172.21.15.46:0/189245179 shutdown_connections 2024-03-31T23:11:44.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:44.622+0000 7f4c7b1e2640 1 -- 172.21.15.46:0/189245179 wait complete. 2024-03-31T23:11:44.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:44.622+0000 7f4c7b1e2640 1 Processor -- start 2024-03-31T23:11:44.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:44.622+0000 7f4c7b1e2640 1 -- start start 2024-03-31T23:11:44.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:44.623+0000 7f4c7b1e2640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4c741017a0 0x7f4c74075500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:44.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:44.623+0000 7f4c7b1e2640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4c74102990 0x7f4c74073b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:44.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:44.623+0000 7f4c7b1e2640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4c74103350 0x7f4c74074080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:44.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:44.623+0000 7f4c7b1e2640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f4c7410d2c0 con 0x7f4c741017a0 2024-03-31T23:11:44.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:44.623+0000 7f4c7b1e2640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f4c7410d140 con 0x7f4c74103350 2024-03-31T23:11:44.632 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:44.623+0000 7f4c7b1e2640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f4c7410d440 con 0x7f4c74102990 2024-03-31T23:11:44.632 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:44.623+0000 7f4c78f7e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4c741017a0 0x7f4c74075500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:44.632 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:44.623+0000 7f4c78f7e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4c741017a0 0x7f4c74075500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:48476/0 (socket says 172.21.15.46:48476) 2024-03-31T23:11:44.632 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:44.623+0000 7f4c78f7e640 1 -- 172.21.15.46:0/1399523341 learned_addr learned my addr 172.21.15.46:0/1399523341 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:11:44.632 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:44.623+0000 7f4c73fff640 1 --2- 172.21.15.46:0/1399523341 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4c74102990 0x7f4c74073b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:44.632 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:44.623+0000 7f4c7977f640 1 --2- 172.21.15.46:0/1399523341 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4c74103350 0x7f4c74074080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:44.632 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:44.623+0000 7f4c78f7e640 1 -- 172.21.15.46:0/1399523341 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4c74102990 msgr2=0x7f4c74073b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:44.632 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:44.623+0000 7f4c78f7e640 1 --2- 172.21.15.46:0/1399523341 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4c74102990 0x7f4c74073b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:44.632 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:44.623+0000 7f4c78f7e640 1 -- 172.21.15.46:0/1399523341 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4c74103350 msgr2=0x7f4c74074080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:44.633 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:44.623+0000 7f4c78f7e640 1 --2- 172.21.15.46:0/1399523341 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4c74103350 0x7f4c74074080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:44.633 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:44.623+0000 7f4c78f7e640 1 -- 172.21.15.46:0/1399523341 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4c74074910 con 0x7f4c741017a0 2024-03-31T23:11:44.633 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:44.623+0000 7f4c78f7e640 1 --2- 172.21.15.46:0/1399523341 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4c741017a0 0x7f4c74075500 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f4c6c032380 tx=0x7f4c6c032820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:11:44.633 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:44.623+0000 7f4c71ffb640 1 -- 172.21.15.46:0/1399523341 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c6c041020 con 0x7f4c741017a0 2024-03-31T23:11:44.633 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:44.624+0000 7f4c7b1e2640 1 -- 172.21.15.46:0/1399523341 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4c741a1a50 con 0x7f4c741017a0 2024-03-31T23:11:44.633 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:44.624+0000 7f4c71ffb640 1 -- 172.21.15.46:0/1399523341 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4c6c003e20 con 0x7f4c741017a0 2024-03-31T23:11:44.633 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:44.624+0000 7f4c71ffb640 1 -- 172.21.15.46:0/1399523341 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c6c039040 con 0x7f4c741017a0 2024-03-31T23:11:44.633 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:44.624+0000 7f4c7b1e2640 1 -- 172.21.15.46:0/1399523341 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4c741a1ff0 con 0x7f4c741017a0 2024-03-31T23:11:44.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:44.625+0000 7f4c7b1e2640 1 -- 172.21.15.46:0/1399523341 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4c74108be0 con 0x7f4c741017a0 2024-03-31T23:11:44.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:44.626+0000 7f4c71ffb640 1 -- 172.21.15.46:0/1399523341 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 21) ==== 137132+0+0 (secure 0 0 0) 0x7f4c6c044020 con 0x7f4c741017a0 2024-03-31T23:11:44.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:44.626+0000 7f4c71ffb640 1 --2- 172.21.15.46:0/1399523341 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f4c600a2350 0x7f4c600a4810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:44.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:44.627+0000 7f4c73fff640 1 --2- 172.21.15.46:0/1399523341 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f4c600a2350 0x7f4c600a4810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:44.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:44.627+0000 7f4c71ffb640 1 -- 172.21.15.46:0/1399523341 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7f4c6c0f0c70 con 0x7f4c741017a0 2024-03-31T23:11:44.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:44.627+0000 7f4c73fff640 1 --2- 172.21.15.46:0/1399523341 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f4c600a2350 0x7f4c600a4810 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f4c74075270 tx=0x7f4c5c005f90 comp rx=0 tx=0).ready entity=mgr.15654 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:11:44.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:44.630+0000 7f4c71ffb640 1 -- 172.21.15.46:0/1399523341 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f4c6c04b020 con 0x7f4c741017a0 2024-03-31T23:11:44.744 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:44.743+0000 7f4c71ffb640 1 -- 172.21.15.46:0/1399523341 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mgrmap(e 22) ==== 137440+0+0 (secure 0 0 0) 0x7f4c6c0347b0 con 0x7f4c741017a0 2024-03-31T23:11:44.746 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:11:44 smithi046 ceph-mon[32498]: from='client.15672 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}]: dispatch 2024-03-31T23:11:44.747 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:11:44 smithi046 ceph-mon[32498]: pgmap v3: 129 pgs: 129 active+clean; 1.1 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-31T23:11:44.747 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:11:44 smithi046 ceph-mon[32498]: from='client.15714 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}]: dispatch 2024-03-31T23:11:44.747 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:11:44 smithi046 ceph-mon[32498]: from='client.15720 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-03-31T23:11:44.915 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:44.914+0000 7f4c7b1e2640 1 -- 172.21.15.46:0/1399523341 --> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}) -- 0x7f4c74061b90 con 0x7f4c600a2350 2024-03-31T23:11:44.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:44.921+0000 7f4c71ffb640 1 -- 172.21.15.46:0/1399523341 <== mgr.15654 v2:172.21.15.46:6800/685378567 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+10 (secure 0 0 0) 0x7f4c74061b90 con 0x7f4c600a2350 2024-03-31T23:11:44.922 INFO:teuthology.orchestra.run.smithi046.stdout:/ 1m 6m3h 2024-03-31T23:11:44.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:44.923+0000 7f4c7b1e2640 1 -- 172.21.15.46:0/1399523341 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f4c600a2350 msgr2=0x7f4c600a4810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:44.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:44.923+0000 7f4c7b1e2640 1 --2- 172.21.15.46:0/1399523341 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f4c600a2350 0x7f4c600a4810 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f4c74075270 tx=0x7f4c5c005f90 comp rx=0 tx=0).stop 2024-03-31T23:11:44.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:44.923+0000 7f4c7b1e2640 1 -- 172.21.15.46:0/1399523341 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4c741017a0 msgr2=0x7f4c74075500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:44.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:44.923+0000 7f4c7b1e2640 1 --2- 172.21.15.46:0/1399523341 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4c741017a0 0x7f4c74075500 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f4c6c032380 tx=0x7f4c6c032820 comp rx=0 tx=0).stop 2024-03-31T23:11:44.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:44.923+0000 7f4c7b1e2640 1 -- 172.21.15.46:0/1399523341 shutdown_connections 2024-03-31T23:11:44.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:44.923+0000 7f4c7b1e2640 1 --2- 172.21.15.46:0/1399523341 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f4c600a2350 0x7f4c600a4810 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:44.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:44.923+0000 7f4c7b1e2640 1 --2- 172.21.15.46:0/1399523341 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4c74103350 0x7f4c74074080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:44.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:44.923+0000 7f4c7b1e2640 1 --2- 172.21.15.46:0/1399523341 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4c74102990 0x7f4c74073b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:44.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:44.923+0000 7f4c7b1e2640 1 --2- 172.21.15.46:0/1399523341 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4c741017a0 0x7f4c74075500 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:44.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:44.923+0000 7f4c7b1e2640 1 -- 172.21.15.46:0/1399523341 >> 172.21.15.46:0/1399523341 conn(0x7f4c740fcd10 msgr2=0x7f4c74105db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:11:44.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:44.923+0000 7f4c7b1e2640 1 -- 172.21.15.46:0/1399523341 shutdown_connections 2024-03-31T23:11:44.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:11:44.923+0000 7f4c7b1e2640 1 -- 172.21.15.46:0/1399523341 wait complete. 2024-03-31T23:11:44.936 DEBUG:teuthology.orchestra.run.smithi046:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'date +%s > START_TIME' 2024-03-31T23:11:44.966 INFO:teuthology.run_tasks:Running task full_sequential_finally... 2024-03-31T23:11:44.977 INFO:teuthology.run_tasks:Running task background_exec... 2024-03-31T23:11:44.989 INFO:teuthology.task.background_exec:Running background command on role mon.a host ubuntu@smithi046.front.sepia.ceph.com 2024-03-31T23:11:44.989 DEBUG:teuthology.orchestra.run.smithi046:> sudo TESTDIR=/home/ubuntu/cephtest daemon-helper kill --kill-group bash -c 'while sleep 13; do ceph tell mds.cephfs:0 flush journal; done' 2024-03-31T23:11:45.009 INFO:teuthology.run_tasks:Running task workunit... 2024-03-31T23:11:45.021 INFO:tasks.workunit:Pulling workunits from ref 384f31f3d3aee7f8ba3f60846b8844c049165583 2024-03-31T23:11:45.022 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-03-31T23:11:45.022 INFO:tasks.workunit:timeout=3h 2024-03-31T23:11:45.022 INFO:tasks.workunit:cleanup=True 2024-03-31T23:11:45.022 DEBUG:teuthology.orchestra.run.smithi046:> stat -- /home/ubuntu/cephtest/mnt.0 2024-03-31T23:11:45.038 INFO:teuthology.orchestra.run.smithi046.stdout: File: /home/ubuntu/cephtest/mnt.0 2024-03-31T23:11:45.038 INFO:teuthology.orchestra.run.smithi046.stdout: Size: 0 Blocks: 1 IO Block: 4194304 directory 2024-03-31T23:11:45.038 INFO:teuthology.orchestra.run.smithi046.stdout:Device: 77h/119d Inode: 1 Links: 2 2024-03-31T23:11:45.038 INFO:teuthology.orchestra.run.smithi046.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-31T23:11:45.038 INFO:teuthology.orchestra.run.smithi046.stdout:Context: system_u:object_r:fusefs_t:s0 2024-03-31T23:11:45.038 INFO:teuthology.orchestra.run.smithi046.stdout:Access: 2024-03-31 23:09:32.106212438 +0000 2024-03-31T23:11:45.038 INFO:teuthology.orchestra.run.smithi046.stdout:Modify: 2024-03-31 23:09:32.106212438 +0000 2024-03-31T23:11:45.038 INFO:teuthology.orchestra.run.smithi046.stdout:Change: 2024-03-31 23:11:27.180262646 +0000 2024-03-31T23:11:45.039 INFO:teuthology.orchestra.run.smithi046.stdout: Birth: - 2024-03-31T23:11:45.039 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.0 2024-03-31T23:11:45.039 DEBUG:teuthology.orchestra.run.smithi046:> cd -- /home/ubuntu/cephtest/mnt.0 && sudo install -d -m 0755 --owner=ubuntu -- client.0 2024-03-31T23:11:45.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:11:44 smithi162 ceph-mon[36237]: from='client.15672 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}]: dispatch 2024-03-31T23:11:45.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:11:44 smithi162 ceph-mon[36237]: pgmap v3: 129 pgs: 129 active+clean; 1.1 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-31T23:11:45.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:11:44 smithi162 ceph-mon[36237]: from='client.15714 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}]: dispatch 2024-03-31T23:11:45.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:11:44 smithi162 ceph-mon[36237]: from='client.15720 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-03-31T23:11:45.112 DEBUG:teuthology.orchestra.run.smithi046:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://git.ceph.com/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 384f31f3d3aee7f8ba3f60846b8844c049165583 2024-03-31T23:11:45.166 INFO:tasks.workunit.client.0.smithi046.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-03-31T23:11:45.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:11:44 smithi142 ceph-mon[35271]: from='client.15672 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}]: dispatch 2024-03-31T23:11:45.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:11:44 smithi142 ceph-mon[35271]: pgmap v3: 129 pgs: 129 active+clean; 1.1 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-31T23:11:45.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:11:44 smithi142 ceph-mon[35271]: from='client.15714 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}]: dispatch 2024-03-31T23:11:45.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:11:44 smithi142 ceph-mon[35271]: from='client.15720 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-03-31T23:11:46.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:11:45 smithi162 ceph-mon[36237]: pgmap v4: 129 pgs: 129 active+clean; 1.1 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-31T23:11:46.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:11:45 smithi162 ceph-mon[36237]: mgrmap e22: x(active, since 2s), standbys: z, y 2024-03-31T23:11:46.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:11:45 smithi162 ceph-mon[36237]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' 2024-03-31T23:11:46.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:11:45 smithi162 ceph-mon[36237]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' 2024-03-31T23:11:46.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:11:45 smithi162 ceph-mon[36237]: from='client.15726 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-31T23:11:46.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:11:45 smithi162 ceph-mon[36237]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' 2024-03-31T23:11:46.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:11:45 smithi162 ceph-mon[36237]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' 2024-03-31T23:11:46.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:11:45 smithi162 ceph-mon[36237]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' 2024-03-31T23:11:46.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:11:45 smithi162 ceph-mon[36237]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' 2024-03-31T23:11:46.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:11:45 smithi046 ceph-mon[32498]: pgmap v4: 129 pgs: 129 active+clean; 1.1 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-31T23:11:46.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:11:45 smithi046 ceph-mon[32498]: mgrmap e22: x(active, since 2s), standbys: z, y 2024-03-31T23:11:46.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:11:45 smithi046 ceph-mon[32498]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' 2024-03-31T23:11:46.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:11:45 smithi046 ceph-mon[32498]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' 2024-03-31T23:11:46.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:11:45 smithi046 ceph-mon[32498]: from='client.15726 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-31T23:11:46.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:11:45 smithi046 ceph-mon[32498]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' 2024-03-31T23:11:46.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:11:45 smithi046 ceph-mon[32498]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' 2024-03-31T23:11:46.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:11:45 smithi046 ceph-mon[32498]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' 2024-03-31T23:11:46.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:11:45 smithi046 ceph-mon[32498]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' 2024-03-31T23:11:46.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:11:45 smithi142 ceph-mon[35271]: pgmap v4: 129 pgs: 129 active+clean; 1.1 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-31T23:11:46.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:11:45 smithi142 ceph-mon[35271]: mgrmap e22: x(active, since 2s), standbys: z, y 2024-03-31T23:11:46.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:11:45 smithi142 ceph-mon[35271]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' 2024-03-31T23:11:46.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:11:45 smithi142 ceph-mon[35271]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' 2024-03-31T23:11:46.194 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:11:45 smithi142 ceph-mon[35271]: from='client.15726 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-31T23:11:46.194 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:11:45 smithi142 ceph-mon[35271]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' 2024-03-31T23:11:46.194 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:11:45 smithi142 ceph-mon[35271]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' 2024-03-31T23:11:46.194 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:11:45 smithi142 ceph-mon[35271]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' 2024-03-31T23:11:46.194 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:11:45 smithi142 ceph-mon[35271]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' 2024-03-31T23:11:47.296 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:11:46 smithi162 ceph-mon[36237]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' 2024-03-31T23:11:47.297 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:11:46 smithi162 ceph-mon[36237]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' 2024-03-31T23:11:47.297 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:11:46 smithi162 ceph-mon[36237]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-31T23:11:47.297 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:11:46 smithi162 ceph-mon[36237]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-31T23:11:47.297 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:11:46 smithi162 ceph-mon[36237]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-31T23:11:47.297 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:11:46 smithi162 ceph-mon[36237]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-31T23:11:47.298 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:11:46 smithi162 ceph-mon[36237]: Adjusting osd_memory_target on smithi162 to 176.2M 2024-03-31T23:11:47.298 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:11:46 smithi162 ceph-mon[36237]: Unable to set osd_memory_target on smithi162 to 184788889: error parsing value: Value '184788889' is below minimum 939524096 2024-03-31T23:11:47.298 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:11:46 smithi162 ceph-mon[36237]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' 2024-03-31T23:11:47.298 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:11:46 smithi162 ceph-mon[36237]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' 2024-03-31T23:11:47.298 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:11:46 smithi162 ceph-mon[36237]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-31T23:11:47.298 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:11:46 smithi162 ceph-mon[36237]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-31T23:11:47.299 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:11:46 smithi162 ceph-mon[36237]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-31T23:11:47.299 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:11:46 smithi162 ceph-mon[36237]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-31T23:11:47.299 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:11:46 smithi162 ceph-mon[36237]: Adjusting osd_memory_target on smithi142 to 176.2M 2024-03-31T23:11:47.299 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:11:46 smithi162 ceph-mon[36237]: Unable to set osd_memory_target on smithi142 to 184770252: error parsing value: Value '184770252' is below minimum 939524096 2024-03-31T23:11:47.299 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:11:46 smithi162 ceph-mon[36237]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' 2024-03-31T23:11:47.299 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:11:46 smithi162 ceph-mon[36237]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' 2024-03-31T23:11:47.299 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:11:46 smithi162 ceph-mon[36237]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-31T23:11:47.300 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:11:46 smithi162 ceph-mon[36237]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-31T23:11:47.300 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:11:46 smithi162 ceph-mon[36237]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-31T23:11:47.300 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:11:46 smithi162 ceph-mon[36237]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-31T23:11:47.300 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:11:46 smithi162 ceph-mon[36237]: Adjusting osd_memory_target on smithi046 to 177.2M 2024-03-31T23:11:47.300 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:11:46 smithi162 ceph-mon[36237]: Unable to set osd_memory_target on smithi046 to 185899212: error parsing value: Value '185899212' is below minimum 939524096 2024-03-31T23:11:47.300 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:11:46 smithi162 ceph-mon[36237]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:11:47.300 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:11:46 smithi162 ceph-mon[36237]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-31T23:11:47.301 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:11:46 smithi162 ceph-mon[36237]: Updating smithi046:/etc/ceph/ceph.conf 2024-03-31T23:11:47.301 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:11:46 smithi162 ceph-mon[36237]: Updating smithi142:/etc/ceph/ceph.conf 2024-03-31T23:11:47.301 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:11:46 smithi162 ceph-mon[36237]: Updating smithi162:/etc/ceph/ceph.conf 2024-03-31T23:11:47.301 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:11:46 smithi162 ceph-mon[36237]: Updating smithi046:/var/lib/ceph/6a9d8f08-efb2-11ee-b647-cb9ed24678a4/config/ceph.conf 2024-03-31T23:11:47.301 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:11:46 smithi162 ceph-mon[36237]: Updating smithi142:/var/lib/ceph/6a9d8f08-efb2-11ee-b647-cb9ed24678a4/config/ceph.conf 2024-03-31T23:11:47.301 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:11:46 smithi162 ceph-mon[36237]: Updating smithi162:/var/lib/ceph/6a9d8f08-efb2-11ee-b647-cb9ed24678a4/config/ceph.conf 2024-03-31T23:11:47.362 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:11:46 smithi046 ceph-mon[32498]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' 2024-03-31T23:11:47.362 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:11:46 smithi046 ceph-mon[32498]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' 2024-03-31T23:11:47.362 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:11:46 smithi046 ceph-mon[32498]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-31T23:11:47.362 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:11:46 smithi046 ceph-mon[32498]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-31T23:11:47.362 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:11:46 smithi046 ceph-mon[32498]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-31T23:11:47.362 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:11:46 smithi046 ceph-mon[32498]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-31T23:11:47.362 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:11:46 smithi046 ceph-mon[32498]: Adjusting osd_memory_target on smithi162 to 176.2M 2024-03-31T23:11:47.362 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:11:46 smithi046 ceph-mon[32498]: Unable to set osd_memory_target on smithi162 to 184788889: error parsing value: Value '184788889' is below minimum 939524096 2024-03-31T23:11:47.362 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:11:46 smithi046 ceph-mon[32498]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' 2024-03-31T23:11:47.363 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:11:46 smithi046 ceph-mon[32498]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' 2024-03-31T23:11:47.363 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:11:46 smithi046 ceph-mon[32498]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-31T23:11:47.363 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:11:46 smithi046 ceph-mon[32498]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-31T23:11:47.363 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:11:46 smithi046 ceph-mon[32498]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-31T23:11:47.363 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:11:46 smithi046 ceph-mon[32498]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-31T23:11:47.363 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:11:46 smithi046 ceph-mon[32498]: Adjusting osd_memory_target on smithi142 to 176.2M 2024-03-31T23:11:47.363 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:11:46 smithi046 ceph-mon[32498]: Unable to set osd_memory_target on smithi142 to 184770252: error parsing value: Value '184770252' is below minimum 939524096 2024-03-31T23:11:47.363 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:11:46 smithi046 ceph-mon[32498]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' 2024-03-31T23:11:47.363 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:11:46 smithi046 ceph-mon[32498]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' 2024-03-31T23:11:47.363 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:11:46 smithi046 ceph-mon[32498]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-31T23:11:47.363 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:11:46 smithi046 ceph-mon[32498]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-31T23:11:47.363 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:11:46 smithi046 ceph-mon[32498]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-31T23:11:47.363 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:11:46 smithi046 ceph-mon[32498]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-31T23:11:47.363 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:11:46 smithi046 ceph-mon[32498]: Adjusting osd_memory_target on smithi046 to 177.2M 2024-03-31T23:11:47.364 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:11:46 smithi046 ceph-mon[32498]: Unable to set osd_memory_target on smithi046 to 185899212: error parsing value: Value '185899212' is below minimum 939524096 2024-03-31T23:11:47.364 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:11:46 smithi046 ceph-mon[32498]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:11:47.364 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:11:46 smithi046 ceph-mon[32498]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-31T23:11:47.364 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:11:46 smithi046 ceph-mon[32498]: Updating smithi046:/etc/ceph/ceph.conf 2024-03-31T23:11:47.364 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:11:46 smithi046 ceph-mon[32498]: Updating smithi142:/etc/ceph/ceph.conf 2024-03-31T23:11:47.364 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:11:46 smithi046 ceph-mon[32498]: Updating smithi162:/etc/ceph/ceph.conf 2024-03-31T23:11:47.364 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:11:46 smithi046 ceph-mon[32498]: Updating smithi046:/var/lib/ceph/6a9d8f08-efb2-11ee-b647-cb9ed24678a4/config/ceph.conf 2024-03-31T23:11:47.364 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:11:46 smithi046 ceph-mon[32498]: Updating smithi142:/var/lib/ceph/6a9d8f08-efb2-11ee-b647-cb9ed24678a4/config/ceph.conf 2024-03-31T23:11:47.364 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:11:46 smithi046 ceph-mon[32498]: Updating smithi162:/var/lib/ceph/6a9d8f08-efb2-11ee-b647-cb9ed24678a4/config/ceph.conf 2024-03-31T23:11:47.444 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:11:46 smithi142 ceph-mon[35271]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' 2024-03-31T23:11:47.444 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:11:46 smithi142 ceph-mon[35271]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' 2024-03-31T23:11:47.444 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:11:46 smithi142 ceph-mon[35271]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-31T23:11:47.444 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:11:46 smithi142 ceph-mon[35271]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-31T23:11:47.444 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:11:46 smithi142 ceph-mon[35271]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-31T23:11:47.444 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:11:46 smithi142 ceph-mon[35271]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-31T23:11:47.444 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:11:46 smithi142 ceph-mon[35271]: Adjusting osd_memory_target on smithi162 to 176.2M 2024-03-31T23:11:47.444 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:11:46 smithi142 ceph-mon[35271]: Unable to set osd_memory_target on smithi162 to 184788889: error parsing value: Value '184788889' is below minimum 939524096 2024-03-31T23:11:47.445 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:11:46 smithi142 ceph-mon[35271]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' 2024-03-31T23:11:47.445 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:11:46 smithi142 ceph-mon[35271]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' 2024-03-31T23:11:47.445 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:11:46 smithi142 ceph-mon[35271]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-31T23:11:47.445 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:11:46 smithi142 ceph-mon[35271]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-31T23:11:47.445 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:11:46 smithi142 ceph-mon[35271]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-31T23:11:47.445 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:11:46 smithi142 ceph-mon[35271]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-31T23:11:47.445 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:11:46 smithi142 ceph-mon[35271]: Adjusting osd_memory_target on smithi142 to 176.2M 2024-03-31T23:11:47.445 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:11:46 smithi142 ceph-mon[35271]: Unable to set osd_memory_target on smithi142 to 184770252: error parsing value: Value '184770252' is below minimum 939524096 2024-03-31T23:11:47.445 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:11:46 smithi142 ceph-mon[35271]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' 2024-03-31T23:11:47.445 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:11:46 smithi142 ceph-mon[35271]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' 2024-03-31T23:11:47.445 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:11:46 smithi142 ceph-mon[35271]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-31T23:11:47.445 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:11:46 smithi142 ceph-mon[35271]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-31T23:11:47.446 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:11:46 smithi142 ceph-mon[35271]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-31T23:11:47.446 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:11:46 smithi142 ceph-mon[35271]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-31T23:11:47.446 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:11:46 smithi142 ceph-mon[35271]: Adjusting osd_memory_target on smithi046 to 177.2M 2024-03-31T23:11:47.446 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:11:46 smithi142 ceph-mon[35271]: Unable to set osd_memory_target on smithi046 to 185899212: error parsing value: Value '185899212' is below minimum 939524096 2024-03-31T23:11:47.446 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:11:46 smithi142 ceph-mon[35271]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:11:47.446 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:11:46 smithi142 ceph-mon[35271]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-31T23:11:47.446 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:11:46 smithi142 ceph-mon[35271]: Updating smithi046:/etc/ceph/ceph.conf 2024-03-31T23:11:47.446 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:11:46 smithi142 ceph-mon[35271]: Updating smithi142:/etc/ceph/ceph.conf 2024-03-31T23:11:47.446 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:11:46 smithi142 ceph-mon[35271]: Updating smithi162:/etc/ceph/ceph.conf 2024-03-31T23:11:47.446 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:11:46 smithi142 ceph-mon[35271]: Updating smithi046:/var/lib/ceph/6a9d8f08-efb2-11ee-b647-cb9ed24678a4/config/ceph.conf 2024-03-31T23:11:47.446 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:11:46 smithi142 ceph-mon[35271]: Updating smithi142:/var/lib/ceph/6a9d8f08-efb2-11ee-b647-cb9ed24678a4/config/ceph.conf 2024-03-31T23:11:47.446 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:11:46 smithi142 ceph-mon[35271]: Updating smithi162:/var/lib/ceph/6a9d8f08-efb2-11ee-b647-cb9ed24678a4/config/ceph.conf 2024-03-31T23:11:48.546 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:11:48 smithi162 ceph-mon[36237]: pgmap v5: 129 pgs: 129 active+clean; 1.1 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-31T23:11:48.546 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:11:48 smithi162 ceph-mon[36237]: Updating smithi046:/etc/ceph/ceph.client.admin.keyring 2024-03-31T23:11:48.546 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:11:48 smithi162 ceph-mon[36237]: Updating smithi142:/etc/ceph/ceph.client.admin.keyring 2024-03-31T23:11:48.546 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:11:48 smithi162 ceph-mon[36237]: Updating smithi162:/etc/ceph/ceph.client.admin.keyring 2024-03-31T23:11:48.546 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:11:48 smithi162 ceph-mon[36237]: Updating smithi046:/var/lib/ceph/6a9d8f08-efb2-11ee-b647-cb9ed24678a4/config/ceph.client.admin.keyring 2024-03-31T23:11:48.546 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:11:48 smithi162 ceph-mon[36237]: Updating smithi142:/var/lib/ceph/6a9d8f08-efb2-11ee-b647-cb9ed24678a4/config/ceph.client.admin.keyring 2024-03-31T23:11:48.546 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:11:48 smithi162 ceph-mon[36237]: Updating smithi162:/var/lib/ceph/6a9d8f08-efb2-11ee-b647-cb9ed24678a4/config/ceph.client.admin.keyring 2024-03-31T23:11:48.547 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:11:48 smithi162 ceph-mon[36237]: scrub summary: idle 2024-03-31T23:11:48.547 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:11:48 smithi162 ceph-mon[36237]: mgrmap e23: x(active, since 4s), standbys: z, y 2024-03-31T23:11:48.547 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:11:48 smithi162 ceph-mon[36237]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' 2024-03-31T23:11:48.547 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:11:48 smithi162 ceph-mon[36237]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' 2024-03-31T23:11:48.547 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:11:48 smithi162 ceph-mon[36237]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' 2024-03-31T23:11:48.547 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:11:48 smithi162 ceph-mon[36237]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' 2024-03-31T23:11:48.547 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:11:48 smithi162 ceph-mon[36237]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' 2024-03-31T23:11:48.547 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:11:48 smithi162 ceph-mon[36237]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' 2024-03-31T23:11:48.547 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:11:48 smithi162 ceph-mon[36237]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' 2024-03-31T23:11:48.547 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:11:48 smithi162 ceph-mon[36237]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-31T23:11:48.547 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:11:48 smithi162 ceph-mon[36237]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:11:48.547 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:11:48 smithi162 ceph-mon[36237]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-31T23:11:48.547 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:11:48 smithi162 ceph-mon[36237]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' 2024-03-31T23:11:48.612 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:11:48 smithi046 ceph-mon[32498]: pgmap v5: 129 pgs: 129 active+clean; 1.1 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-31T23:11:48.612 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:11:48 smithi046 ceph-mon[32498]: Updating smithi046:/etc/ceph/ceph.client.admin.keyring 2024-03-31T23:11:48.612 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:11:48 smithi046 ceph-mon[32498]: Updating smithi142:/etc/ceph/ceph.client.admin.keyring 2024-03-31T23:11:48.612 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:11:48 smithi046 ceph-mon[32498]: Updating smithi162:/etc/ceph/ceph.client.admin.keyring 2024-03-31T23:11:48.612 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:11:48 smithi046 ceph-mon[32498]: Updating smithi046:/var/lib/ceph/6a9d8f08-efb2-11ee-b647-cb9ed24678a4/config/ceph.client.admin.keyring 2024-03-31T23:11:48.612 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:11:48 smithi046 ceph-mon[32498]: Updating smithi142:/var/lib/ceph/6a9d8f08-efb2-11ee-b647-cb9ed24678a4/config/ceph.client.admin.keyring 2024-03-31T23:11:48.612 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:11:48 smithi046 ceph-mon[32498]: Updating smithi162:/var/lib/ceph/6a9d8f08-efb2-11ee-b647-cb9ed24678a4/config/ceph.client.admin.keyring 2024-03-31T23:11:48.612 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:11:48 smithi046 ceph-mon[32498]: scrub summary: idle 2024-03-31T23:11:48.612 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:11:48 smithi046 ceph-mon[32498]: mgrmap e23: x(active, since 4s), standbys: z, y 2024-03-31T23:11:48.613 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:11:48 smithi046 ceph-mon[32498]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' 2024-03-31T23:11:48.613 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:11:48 smithi046 ceph-mon[32498]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' 2024-03-31T23:11:48.613 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:11:48 smithi046 ceph-mon[32498]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' 2024-03-31T23:11:48.613 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:11:48 smithi046 ceph-mon[32498]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' 2024-03-31T23:11:48.613 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:11:48 smithi046 ceph-mon[32498]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' 2024-03-31T23:11:48.613 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:11:48 smithi046 ceph-mon[32498]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' 2024-03-31T23:11:48.613 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:11:48 smithi046 ceph-mon[32498]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' 2024-03-31T23:11:48.613 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:11:48 smithi046 ceph-mon[32498]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-31T23:11:48.613 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:11:48 smithi046 ceph-mon[32498]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:11:48.613 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:11:48 smithi046 ceph-mon[32498]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-31T23:11:48.613 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:11:48 smithi046 ceph-mon[32498]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' 2024-03-31T23:11:48.650 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:11:48 smithi142 ceph-mon[35271]: pgmap v5: 129 pgs: 129 active+clean; 1.1 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-31T23:11:48.650 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:11:48 smithi142 ceph-mon[35271]: Updating smithi046:/etc/ceph/ceph.client.admin.keyring 2024-03-31T23:11:48.650 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:11:48 smithi142 ceph-mon[35271]: Updating smithi142:/etc/ceph/ceph.client.admin.keyring 2024-03-31T23:11:48.650 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:11:48 smithi142 ceph-mon[35271]: Updating smithi162:/etc/ceph/ceph.client.admin.keyring 2024-03-31T23:11:48.650 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:11:48 smithi142 ceph-mon[35271]: Updating smithi046:/var/lib/ceph/6a9d8f08-efb2-11ee-b647-cb9ed24678a4/config/ceph.client.admin.keyring 2024-03-31T23:11:48.650 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:11:48 smithi142 ceph-mon[35271]: Updating smithi142:/var/lib/ceph/6a9d8f08-efb2-11ee-b647-cb9ed24678a4/config/ceph.client.admin.keyring 2024-03-31T23:11:48.650 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:11:48 smithi142 ceph-mon[35271]: Updating smithi162:/var/lib/ceph/6a9d8f08-efb2-11ee-b647-cb9ed24678a4/config/ceph.client.admin.keyring 2024-03-31T23:11:48.650 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:11:48 smithi142 ceph-mon[35271]: scrub summary: idle 2024-03-31T23:11:48.650 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:11:48 smithi142 ceph-mon[35271]: mgrmap e23: x(active, since 4s), standbys: z, y 2024-03-31T23:11:48.650 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:11:48 smithi142 ceph-mon[35271]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' 2024-03-31T23:11:48.650 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:11:48 smithi142 ceph-mon[35271]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' 2024-03-31T23:11:48.651 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:11:48 smithi142 ceph-mon[35271]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' 2024-03-31T23:11:48.651 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:11:48 smithi142 ceph-mon[35271]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' 2024-03-31T23:11:48.651 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:11:48 smithi142 ceph-mon[35271]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' 2024-03-31T23:11:48.651 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:11:48 smithi142 ceph-mon[35271]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' 2024-03-31T23:11:48.651 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:11:48 smithi142 ceph-mon[35271]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' 2024-03-31T23:11:48.651 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:11:48 smithi142 ceph-mon[35271]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-31T23:11:48.651 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:11:48 smithi142 ceph-mon[35271]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:11:48.651 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:11:48 smithi142 ceph-mon[35271]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-31T23:11:48.651 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:11:48 smithi142 ceph-mon[35271]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' 2024-03-31T23:11:50.545 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:11:50 smithi162 ceph-mon[36237]: pgmap v6: 129 pgs: 129 active+clean; 1.1 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-31T23:11:50.612 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:11:50 smithi046 ceph-mon[32498]: pgmap v6: 129 pgs: 129 active+clean; 1.1 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-31T23:11:50.693 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:11:50 smithi142 ceph-mon[35271]: pgmap v6: 129 pgs: 129 active+clean; 1.1 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-31T23:11:52.546 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:11:52 smithi162 ceph-mon[36237]: pgmap v7: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 132 KiB/s wr, 76 op/s 2024-03-31T23:11:52.611 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:11:52 smithi046 ceph-mon[32498]: pgmap v7: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 132 KiB/s wr, 76 op/s 2024-03-31T23:11:52.693 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:11:52 smithi142 ceph-mon[35271]: pgmap v7: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 132 KiB/s wr, 76 op/s 2024-03-31T23:11:54.545 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:11:54 smithi162 ceph-mon[36237]: pgmap v8: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 102 KiB/s wr, 59 op/s 2024-03-31T23:11:54.611 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:11:54 smithi046 ceph-mon[32498]: pgmap v8: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 102 KiB/s wr, 59 op/s 2024-03-31T23:11:54.693 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:11:54 smithi142 ceph-mon[35271]: pgmap v8: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 102 KiB/s wr, 59 op/s 2024-03-31T23:11:56.545 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:11:56 smithi162 ceph-mon[36237]: pgmap v9: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 84 KiB/s wr, 51 op/s 2024-03-31T23:11:56.611 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:11:56 smithi046 ceph-mon[32498]: pgmap v9: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 84 KiB/s wr, 51 op/s 2024-03-31T23:11:56.693 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:11:56 smithi142 ceph-mon[35271]: pgmap v9: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 84 KiB/s wr, 51 op/s 2024-03-31T23:11:58.173 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:11:58.156+0000 7fbe0f88a640 1 -- 172.21.15.46:0/3622833512 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbe08106650 msgr2=0x7fbe08106a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:58.173 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:11:58.156+0000 7fbe0f88a640 1 --2- 172.21.15.46:0/3622833512 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbe08106650 0x7fbe08106a30 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7fbe04009930 tx=0x7fbe0402bc30 comp rx=0 tx=0).stop 2024-03-31T23:11:58.173 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:11:58.156+0000 7fbe0f88a640 1 -- 172.21.15.46:0/3622833512 shutdown_connections 2024-03-31T23:11:58.173 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:11:58.156+0000 7fbe0f88a640 1 --2- 172.21.15.46:0/3622833512 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbe08101000 0x7fbe0810d510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:58.173 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:11:58.156+0000 7fbe0f88a640 1 --2- 172.21.15.46:0/3622833512 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbe08100640 0x7fbe08100ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:58.174 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:11:58.156+0000 7fbe0f88a640 1 --2- 172.21.15.46:0/3622833512 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbe08106650 0x7fbe08106a30 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:58.174 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:11:58.156+0000 7fbe0f88a640 1 -- 172.21.15.46:0/3622833512 >> 172.21.15.46:0/3622833512 conn(0x7fbe08077860 msgr2=0x7fbe08077c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:11:58.174 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:11:58.156+0000 7fbe0f88a640 1 -- 172.21.15.46:0/3622833512 shutdown_connections 2024-03-31T23:11:58.174 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:11:58.156+0000 7fbe0f88a640 1 -- 172.21.15.46:0/3622833512 wait complete. 2024-03-31T23:11:58.174 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:11:58.157+0000 7fbe0f88a640 1 Processor -- start 2024-03-31T23:11:58.174 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:11:58.157+0000 7fbe0f88a640 1 -- start start 2024-03-31T23:11:58.174 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:11:58.157+0000 7fbe0f88a640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbe08100640 0x7fbe08197e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:58.174 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:11:58.157+0000 7fbe0f88a640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbe08101000 0x7fbe081983d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:58.174 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:11:58.157+0000 7fbe0f88a640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbe08106650 0x7fbe0819c750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:58.174 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:11:58.157+0000 7fbe0f88a640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fbe08111a10 con 0x7fbe08106650 2024-03-31T23:11:58.174 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:11:58.157+0000 7fbe0f88a640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fbe08111890 con 0x7fbe08101000 2024-03-31T23:11:58.174 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:11:58.157+0000 7fbe0f88a640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fbe08111b90 con 0x7fbe08100640 2024-03-31T23:11:58.175 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:11:58.157+0000 7fbe0d626640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbe08100640 0x7fbe08197e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:58.175 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:11:58.157+0000 7fbe0de27640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbe08106650 0x7fbe0819c750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:58.175 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:11:58.157+0000 7fbe0ce25640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbe08101000 0x7fbe081983d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:58.175 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:11:58.157+0000 7fbe0d626640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbe08100640 0x7fbe08197e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.46:54080/0 (socket says 172.21.15.46:54080) 2024-03-31T23:11:58.175 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:11:58.157+0000 7fbe0d626640 1 -- 172.21.15.46:0/2156156532 learned_addr learned my addr 172.21.15.46:0/2156156532 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:11:58.175 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:11:58.157+0000 7fbe0ce25640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbe08101000 0x7fbe081983d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.46:36600/0 (socket says 172.21.15.46:36600) 2024-03-31T23:11:58.175 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:11:58.157+0000 7fbe0de27640 1 -- 172.21.15.46:0/2156156532 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbe08100640 msgr2=0x7fbe08197e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:58.175 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:11:58.157+0000 7fbe0de27640 1 --2- 172.21.15.46:0/2156156532 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbe08100640 0x7fbe08197e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:58.175 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:11:58.157+0000 7fbe0de27640 1 -- 172.21.15.46:0/2156156532 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbe08101000 msgr2=0x7fbe081983d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:58.175 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:11:58.157+0000 7fbe0de27640 1 --2- 172.21.15.46:0/2156156532 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbe08101000 0x7fbe081983d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:58.175 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:11:58.158+0000 7fbe0de27640 1 -- 172.21.15.46:0/2156156532 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe0819ce80 con 0x7fbe08106650 2024-03-31T23:11:58.175 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:11:58.158+0000 7fbe0de27640 1 --2- 172.21.15.46:0/2156156532 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbe08106650 0x7fbe0819c750 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7fbe00005fd0 tx=0x7fbe0000ee60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:11:58.175 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:11:58.158+0000 7fbdfe7fc640 1 -- 172.21.15.46:0/2156156532 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe0000cd80 con 0x7fbe08106650 2024-03-31T23:11:58.175 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:11:58.158+0000 7fbe0f88a640 1 -- 172.21.15.46:0/2156156532 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbe0819d170 con 0x7fbe08106650 2024-03-31T23:11:58.176 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:11:58.158+0000 7fbdfe7fc640 1 -- 172.21.15.46:0/2156156532 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fbe00005020 con 0x7fbe08106650 2024-03-31T23:11:58.176 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:11:58.158+0000 7fbdfe7fc640 1 -- 172.21.15.46:0/2156156532 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe00010690 con 0x7fbe08106650 2024-03-31T23:11:58.176 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:11:58.158+0000 7fbe0f88a640 1 -- 172.21.15.46:0/2156156532 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbe081a4a00 con 0x7fbe08106650 2024-03-31T23:11:58.176 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:11:58.160+0000 7fbdfe7fc640 1 -- 172.21.15.46:0/2156156532 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 23) ==== 137496+0+0 (secure 0 0 0) 0x7fbe000051d0 con 0x7fbe08106650 2024-03-31T23:11:58.176 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:11:58.161+0000 7fbe12460740 1 Processor -- start 2024-03-31T23:11:58.176 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:11:58.161+0000 7fbe12460740 1 -- start start 2024-03-31T23:11:58.176 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:11:58.161+0000 7fbe12460740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558b53df46c0 0x558b53df4aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:58.176 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:11:58.161+0000 7fbdfe7fc640 1 --2- 172.21.15.46:0/2156156532 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fbdf40a2750 0x7fbdf40a4c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:58.176 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:11:58.161+0000 7fbe12460740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558b53df4fe0 0x558b53dfd870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:58.176 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:11:58.161+0000 7fbe12460740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558b53dfddb0 0x558b53e001a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:58.176 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:11:58.161+0000 7fbe12460740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x558b53d578f0 con 0x558b53df4fe0 2024-03-31T23:11:58.176 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:11:58.161+0000 7fbe12460740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x558b53d008d0 con 0x558b53dfddb0 2024-03-31T23:11:58.176 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:11:58.161+0000 7fbe12460740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x558b53d53990 con 0x558b53df46c0 2024-03-31T23:11:58.177 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:11:58.161+0000 7fbe0ce25640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558b53df4fe0 0x558b53dfd870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:58.177 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:11:58.161+0000 7fbe0d626640 1 --2- 172.21.15.46:0/2156156532 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fbdf40a2750 0x7fbdf40a4c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:58.177 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:11:58.161+0000 7fbe0de27640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558b53dfddb0 0x558b53e001a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:58.177 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:11:58.161+0000 7fbe0ce25640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558b53df4fe0 0x558b53dfd870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55890/0 (socket says 172.21.15.46:55890) 2024-03-31T23:11:58.177 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:11:58.161+0000 7fbe0ce25640 1 -- 172.21.15.46:0/741827915 learned_addr learned my addr 172.21.15.46:0/741827915 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:11:58.177 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:11:58.161+0000 7fbdfe7fc640 1 -- 172.21.15.46:0/2156156532 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7fbe000cf6a0 con 0x7fbe08106650 2024-03-31T23:11:58.177 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:11:58.161+0000 7fbe0ce25640 1 --2- 172.21.15.46:0/741827915 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558b53df46c0 0x558b53df4aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:58.177 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:11:58.162+0000 7fbe0de27640 1 -- 172.21.15.46:0/741827915 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558b53df46c0 msgr2=0x558b53df4aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:11:58.177 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:11:58.162+0000 7fbe0de27640 1 --2- 172.21.15.46:0/741827915 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558b53df46c0 0x558b53df4aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:58.177 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:11:58.162+0000 7fbe0de27640 1 -- 172.21.15.46:0/741827915 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558b53df4fe0 msgr2=0x558b53dfd870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:11:58.177 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:11:58.162+0000 7fbe0de27640 1 --2- 172.21.15.46:0/741827915 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558b53df4fe0 0x558b53dfd870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:58.177 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:11:58.162+0000 7fbe0de27640 1 -- 172.21.15.46:0/741827915 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558b53a6c210 con 0x558b53dfddb0 2024-03-31T23:11:58.178 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:11:58.162+0000 7fbe0ce25640 1 --2- 172.21.15.46:0/741827915 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558b53df4fe0 0x558b53dfd870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:11:58.178 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:11:58.162+0000 7fbe0d626640 1 --2- 172.21.15.46:0/2156156532 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fbdf40a2750 0x7fbdf40a4c10 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fbe04000c00 tx=0x7fbe040058d0 comp rx=0 tx=0).ready entity=mgr.15654 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:11:58.178 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:11:58.162+0000 7fbe0de27640 1 --2- 172.21.15.46:0/741827915 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558b53dfddb0 0x558b53e001a0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fbe000cc920 tx=0x7fbe000b4ae0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9ecf60758650b28e server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:11:58.178 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:11:58.162+0000 7fbdf37fe640 1 -- 172.21.15.46:0/741827915 <== mon.2 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe0009f9f0 con 0x558b53dfddb0 2024-03-31T23:11:58.178 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:11:58.162+0000 7fbdf37fe640 1 -- 172.21.15.46:0/741827915 <== mon.2 v2:172.21.15.142:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fbe000a42b0 con 0x558b53dfddb0 2024-03-31T23:11:58.178 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:11:58.162+0000 7fbdf37fe640 1 -- 172.21.15.46:0/741827915 <== mon.2 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe000b4c00 con 0x558b53dfddb0 2024-03-31T23:11:58.178 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:11:58.163+0000 7fbe12460740 1 -- 172.21.15.46:0/741827915 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558b53dfddb0 msgr2=0x558b53e001a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:58.178 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:11:58.163+0000 7fbe12460740 1 --2- 172.21.15.46:0/741827915 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558b53dfddb0 0x558b53e001a0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fbe000cc920 tx=0x7fbe000b4ae0 comp rx=0 tx=0).stop 2024-03-31T23:11:58.178 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:11:58.163+0000 7fbe12460740 1 -- 172.21.15.46:0/741827915 shutdown_connections 2024-03-31T23:11:58.178 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:11:58.163+0000 7fbe12460740 1 --2- 172.21.15.46:0/741827915 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558b53dfddb0 0x558b53e001a0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:58.178 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:11:58.163+0000 7fbe12460740 1 --2- 172.21.15.46:0/741827915 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558b53df4fe0 0x558b53dfd870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:58.178 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:11:58.163+0000 7fbe12460740 1 --2- 172.21.15.46:0/741827915 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558b53df46c0 0x558b53df4aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:58.178 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:11:58.163+0000 7fbe12460740 1 -- 172.21.15.46:0/741827915 >> 172.21.15.46:0/741827915 conn(0x558b53deefe0 msgr2=0x558b53def3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:11:58.178 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:11:58.163+0000 7fbe12460740 1 -- 172.21.15.46:0/741827915 shutdown_connections 2024-03-31T23:11:58.179 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:11:58.163+0000 7fbe12460740 1 -- 172.21.15.46:0/741827915 wait complete. 2024-03-31T23:11:58.179 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:11:58.163+0000 7fbe12460740 1 -- start start 2024-03-31T23:11:58.179 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:11:58.163+0000 7fbe12460740 1 Processor -- start 2024-03-31T23:11:58.179 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:11:58.164+0000 7fbe12460740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558b53df46c0 0x558b53dfaec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:58.179 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:11:58.164+0000 7fbe12460740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558b53df4fe0 0x558b53dfb400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:58.179 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:11:58.164+0000 7fbe12460740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558b53dfddb0 0x558b53dfb940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:58.179 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:11:58.164+0000 7fbe12460740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x558b53e009e0 con 0x558b53df46c0 2024-03-31T23:11:58.179 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:11:58.164+0000 7fbe12460740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x558b53cf5f80 con 0x558b53df4fe0 2024-03-31T23:11:58.179 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:11:58.164+0000 7fbe12460740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x558b53e00b60 con 0x558b53dfddb0 2024-03-31T23:11:58.179 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:11:58.164+0000 7fbe12460740 10 client.0 _ll_register_callbacks cb 0x558b53d4e290 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T23:11:58.179 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:11:58.164+0000 7fbe0d626640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558b53df46c0 0x558b53dfaec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:58.179 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:11:58.164+0000 7fbe0ce25640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558b53df4fe0 0x558b53dfb400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:58.179 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:11:58.164+0000 7fbe0de27640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558b53dfddb0 0x558b53dfb940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:58.179 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:11:58.164+0000 7fbe0d626640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558b53df46c0 0x558b53dfaec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55904/0 (socket says 172.21.15.46:55904) 2024-03-31T23:11:58.180 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:11:58.164+0000 7fbe0d626640 1 -- 172.21.15.46:0/3135387884 learned_addr learned my addr 172.21.15.46:0/3135387884 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:11:58.180 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:11:58.164+0000 7fbe0ce25640 1 -- 172.21.15.46:0/3135387884 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558b53dfddb0 msgr2=0x558b53dfb940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:11:58.180 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:11:58.164+0000 7fbe0ce25640 1 --2- 172.21.15.46:0/3135387884 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558b53dfddb0 0x558b53dfb940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:58.180 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:11:58.164+0000 7fbe0ce25640 1 -- 172.21.15.46:0/3135387884 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558b53df46c0 msgr2=0x558b53dfaec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:11:58.180 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:11:58.164+0000 7fbe0ce25640 1 --2- 172.21.15.46:0/3135387884 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558b53df46c0 0x558b53dfaec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:58.180 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:11:58.164+0000 7fbe0ce25640 1 -- 172.21.15.46:0/3135387884 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558b53b38620 con 0x558b53df4fe0 2024-03-31T23:11:58.180 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:11:58.164+0000 7fbe0de27640 1 --2- 172.21.15.46:0/3135387884 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558b53dfddb0 0x558b53dfb940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:11:58.180 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:11:58.165+0000 7fbe0ce25640 1 --2- 172.21.15.46:0/3135387884 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558b53df4fe0 0x558b53dfb400 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fbdf8009fd0 tx=0x7fbdf8009b60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=55c6a5fdf25e5401 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:11:58.180 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:11:58.165+0000 7fbdf0ff9640 10 client.0 ms_handle_connect on v2:172.21.15.142:3300/0 2024-03-31T23:11:58.180 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:11:58.165+0000 7fbdf0ff9640 1 -- 172.21.15.46:0/3135387884 <== mon.2 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbdf800ee10 con 0x558b53df4fe0 2024-03-31T23:11:58.180 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:11:58.165+0000 7fbdf0ff9640 1 -- 172.21.15.46:0/3135387884 <== mon.2 v2:172.21.15.142:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7fbdf8004030 con 0x558b53df4fe0 2024-03-31T23:11:58.180 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:11:58.165+0000 7fbe12460740 1 -- 172.21.15.46:0/3135387884 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbe000a42b0 con 0x558b53df4fe0 2024-03-31T23:11:58.180 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:11:58.165+0000 7fbdf0ff9640 1 -- 172.21.15.46:0/3135387884 <== mon.2 v2:172.21.15.142:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fbdf8027050 con 0x558b53df4fe0 2024-03-31T23:11:58.181 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:11:58.165+0000 7fbe12460740 10 client.25586 fetch_fsmap learned FSMap version 25 2024-03-31T23:11:58.181 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:11:58.165+0000 7fbe12460740 1 -- 172.21.15.46:0/3135387884 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x558b53e0c9f0 con 0x558b53df4fe0 2024-03-31T23:11:58.181 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:11:58.165+0000 7fbdf0ff9640 1 -- 172.21.15.46:0/3135387884 <== mon.2 v2:172.21.15.142:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fbe000a42b0 con 0x558b53df4fe0 2024-03-31T23:11:58.181 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:11:58.165+0000 7fbdf0ff9640 1 -- 172.21.15.46:0/3135387884 <== mon.2 v2:172.21.15.142:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7fbdf801d030 con 0x558b53df4fe0 2024-03-31T23:11:58.181 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:11:58.165+0000 7fbe12460740 10 client.25586 fetch_fsmap finished waiting for FSMap version 25 2024-03-31T23:11:58.181 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:11:58.165+0000 7fbe12460740 10 client.25586 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-31T23:11:58.181 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:11:58.165+0000 7fbe12460740 20 client.25586 populate_metadata read hostname 'smithi046' 2024-03-31T23:11:58.181 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:11:58.165+0000 7fbe12460740 1 --2- 172.21.15.46:0/3135387884 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x558b53e0deb0 0x558b53e102a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:58.181 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:11:58.165+0000 7fbe12460740 4 client.25586 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-31T23:11:58.181 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:11:58.165+0000 7fbe12460740 1 -- 172.21.15.46:0/3135387884 --> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558b53d6b120 con 0x558b53e0deb0 2024-03-31T23:11:58.181 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:11:58.166+0000 7fbe0d626640 1 --2- 172.21.15.46:0/3135387884 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x558b53e0deb0 0x558b53e102a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:58.181 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:11:58.166+0000 7fbe0d626640 1 --2- 172.21.15.46:0/3135387884 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x558b53e0deb0 0x558b53e102a0 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=607c989a261c95bb server_cookie=e58a2a7e0db28002 in_seq=0 out_seq=0 2024-03-31T23:11:58.181 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:11:58.166+0000 7fbdf0ff9640 10 client.25586 ms_handle_connect on v2:172.21.15.142:6838/2519143918 2024-03-31T23:11:58.181 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:11:58.167+0000 7fbdf0ff9640 1 -- 172.21.15.46:0/3135387884 <== mds.0 v2:172.21.15.142:6838/2519143918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x558b53d6b120 con 0x558b53e0deb0 2024-03-31T23:11:58.182 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:11:58.167+0000 7fbdf0ff9640 10 client.25586 handle_command_reply: tid=0 2024-03-31T23:11:58.182 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:11:58.167+0000 7fbe12460740 1 client.25586 shutdown 2024-03-31T23:11:58.182 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:11:58.167+0000 7fbe12460740 1 -- 172.21.15.46:0/3135387884 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558b53df4fe0 msgr2=0x558b53dfb400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:58.182 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:11:58.167+0000 7fbe12460740 1 --2- 172.21.15.46:0/3135387884 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558b53df4fe0 0x558b53dfb400 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fbdf8009fd0 tx=0x7fbdf8009b60 comp rx=0 tx=0).stop 2024-03-31T23:11:58.182 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:11:58.167+0000 7fbe12460740 1 -- 172.21.15.46:0/3135387884 shutdown_connections 2024-03-31T23:11:58.182 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:11:58.167+0000 7fbe12460740 1 --2- 172.21.15.46:0/3135387884 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x558b53e0deb0 0x558b53e102a0 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:58.182 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:11:58.167+0000 7fbe12460740 1 --2- 172.21.15.46:0/3135387884 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558b53dfddb0 0x558b53dfb940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:58.182 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:11:58.167+0000 7fbe12460740 1 --2- 172.21.15.46:0/3135387884 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558b53df4fe0 0x558b53dfb400 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:58.182 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:11:58.167+0000 7fbdf0ff9640 0 client.25586 ms_handle_reset on v2:172.21.15.142:6838/2519143918 2024-03-31T23:11:58.182 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:11:58.167+0000 7fbe12460740 1 --2- 172.21.15.46:0/3135387884 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558b53df46c0 0x558b53dfaec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:58.182 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:11:58.167+0000 7fbe12460740 1 -- 172.21.15.46:0/3135387884 >> 172.21.15.46:0/3135387884 conn(0x558b53deefe0 msgr2=0x558b53df0d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:11:58.182 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:11:58.167+0000 7fbe12460740 1 -- 172.21.15.46:0/3135387884 shutdown_connections 2024-03-31T23:11:58.182 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:11:58.167+0000 7fbe12460740 1 -- 172.21.15.46:0/3135387884 wait complete. 2024-03-31T23:11:58.183 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:11:58.167+0000 7fbe12460740 20 client.25586 trim_cache size 0 max 16384 2024-03-31T23:11:58.187 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:11:58.182+0000 7fbe12460740 1 Processor -- start 2024-03-31T23:11:58.188 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:11:58.182+0000 7fbe12460740 1 -- start start 2024-03-31T23:11:58.188 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:11:58.182+0000 7fbe12460740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558b53df46c0 0x558b53dfd5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:58.188 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:11:58.182+0000 7fbe12460740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558b53df4fe0 0x558b53e09780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:58.188 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:11:58.182+0000 7fbe12460740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558b53dfddb0 0x558b53e15b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:58.188 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:11:58.182+0000 7fbe12460740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fbe081b1430 con 0x558b53df46c0 2024-03-31T23:11:58.188 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:11:58.182+0000 7fbe12460740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fbe081aed00 con 0x558b53df4fe0 2024-03-31T23:11:58.188 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:11:58.182+0000 7fbe12460740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fbe081acb50 con 0x558b53dfddb0 2024-03-31T23:11:58.188 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:11:58.182+0000 7fbe0d626640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558b53df46c0 0x558b53dfd5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:58.188 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:11:58.182+0000 7fbe0d626640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558b53df46c0 0x558b53dfd5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55912/0 (socket says 172.21.15.46:55912) 2024-03-31T23:11:58.188 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:11:58.182+0000 7fbe0d626640 1 -- 172.21.15.46:0/766689097 learned_addr learned my addr 172.21.15.46:0/766689097 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:11:58.188 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:11:58.182+0000 7fbe0ce25640 1 --2- 172.21.15.46:0/766689097 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558b53df4fe0 0x558b53e09780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:58.188 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:11:58.182+0000 7fbe0de27640 1 --2- 172.21.15.46:0/766689097 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558b53dfddb0 0x558b53e15b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:58.188 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:11:58.182+0000 7fbe0d626640 1 -- 172.21.15.46:0/766689097 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558b53dfddb0 msgr2=0x558b53e15b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:11:58.189 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:11:58.182+0000 7fbe0d626640 1 --2- 172.21.15.46:0/766689097 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558b53dfddb0 0x558b53e15b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:58.189 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:11:58.182+0000 7fbe0d626640 1 -- 172.21.15.46:0/766689097 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558b53df4fe0 msgr2=0x558b53e09780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:11:58.189 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:11:58.182+0000 7fbe0d626640 1 --2- 172.21.15.46:0/766689097 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558b53df4fe0 0x558b53e09780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:58.189 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:11:58.182+0000 7fbe0d626640 1 -- 172.21.15.46:0/766689097 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558b53e00510 con 0x558b53df46c0 2024-03-31T23:11:58.189 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:11:58.183+0000 7fbe0d626640 1 --2- 172.21.15.46:0/766689097 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558b53df46c0 0x558b53dfd5f0 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7fbe04009230 tx=0x7fbe0403e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55cdd1d39fdf403f server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:11:58.189 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:11:58.183+0000 7fbdf2ffd640 1 -- 172.21.15.46:0/766689097 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe0405b070 con 0x558b53df46c0 2024-03-31T23:11:58.189 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:11:58.183+0000 7fbdf2ffd640 1 -- 172.21.15.46:0/766689097 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fbe040054f0 con 0x558b53df46c0 2024-03-31T23:11:58.189 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:11:58.183+0000 7fbdf2ffd640 1 -- 172.21.15.46:0/766689097 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe04004c60 con 0x558b53df46c0 2024-03-31T23:11:58.189 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:11:58.183+0000 7fbe12460740 1 -- 172.21.15.46:0/766689097 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558b53df46c0 msgr2=0x558b53dfd5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:58.189 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:11:58.183+0000 7fbe12460740 1 --2- 172.21.15.46:0/766689097 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558b53df46c0 0x558b53dfd5f0 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7fbe04009230 tx=0x7fbe0403e000 comp rx=0 tx=0).stop 2024-03-31T23:11:58.189 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:11:58.184+0000 7fbe12460740 1 -- 172.21.15.46:0/766689097 shutdown_connections 2024-03-31T23:11:58.189 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:11:58.184+0000 7fbe12460740 1 --2- 172.21.15.46:0/766689097 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558b53dfddb0 0x558b53e15b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:58.189 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:11:58.184+0000 7fbe12460740 1 --2- 172.21.15.46:0/766689097 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558b53df4fe0 0x558b53e09780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:58.190 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:11:58.184+0000 7fbe12460740 1 --2- 172.21.15.46:0/766689097 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558b53df46c0 0x558b53dfd5f0 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:58.190 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:11:58.184+0000 7fbe12460740 1 -- 172.21.15.46:0/766689097 >> 172.21.15.46:0/766689097 conn(0x558b53deefe0 msgr2=0x558b53df5670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:11:58.190 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:11:58.184+0000 7fbe12460740 1 -- 172.21.15.46:0/766689097 shutdown_connections 2024-03-31T23:11:58.190 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:11:58.184+0000 7fbe12460740 1 -- 172.21.15.46:0/766689097 wait complete. 2024-03-31T23:11:58.190 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:11:58.184+0000 7fbe12460740 1 -- start start 2024-03-31T23:11:58.190 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:11:58.184+0000 7fbe12460740 1 Processor -- start 2024-03-31T23:11:58.190 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:11:58.184+0000 7fbe12460740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558b53df46c0 0x558b53e060c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:58.190 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:11:58.184+0000 7fbe12460740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558b53df4fe0 0x558b53e01430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:58.190 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:11:58.184+0000 7fbe12460740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558b53dfddb0 0x558b53e01990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:58.190 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:11:58.184+0000 7fbe12460740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fbe081ad1c0 con 0x558b53dfddb0 2024-03-31T23:11:58.190 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:11:58.184+0000 7fbe12460740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fbe081ae630 con 0x558b53df4fe0 2024-03-31T23:11:58.190 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:11:58.184+0000 7fbe12460740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fbe081ad890 con 0x558b53df46c0 2024-03-31T23:11:58.190 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:11:58.184+0000 7fbe12460740 10 client.0 _ll_register_callbacks cb 0x7fbdf8009660 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T23:11:58.191 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:11:58.184+0000 7fbe0de27640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558b53dfddb0 0x558b53e01990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:58.191 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:11:58.184+0000 7fbe0ce25640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558b53df4fe0 0x558b53e01430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:58.191 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:11:58.184+0000 7fbe0d626640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558b53df46c0 0x558b53e060c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:58.191 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:11:58.184+0000 7fbe0de27640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558b53dfddb0 0x558b53e01990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55920/0 (socket says 172.21.15.46:55920) 2024-03-31T23:11:58.191 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:11:58.184+0000 7fbe0de27640 1 -- 172.21.15.46:0/3545266673 learned_addr learned my addr 172.21.15.46:0/3545266673 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:11:58.191 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:11:58.184+0000 7fbe0ce25640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558b53df4fe0 0x558b53e01430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.46:36638/0 (socket says 172.21.15.46:36638) 2024-03-31T23:11:58.191 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:11:58.185+0000 7fbe0de27640 1 -- 172.21.15.46:0/3545266673 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558b53df46c0 msgr2=0x558b53e060c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:11:58.191 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:11:58.185+0000 7fbe0de27640 1 --2- 172.21.15.46:0/3545266673 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558b53df46c0 0x558b53e060c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:58.191 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:11:58.185+0000 7fbe0de27640 1 -- 172.21.15.46:0/3545266673 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558b53df4fe0 msgr2=0x558b53e01430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:11:58.191 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:11:58.185+0000 7fbe0de27640 1 --2- 172.21.15.46:0/3545266673 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558b53df4fe0 0x558b53e01430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:58.191 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:11:58.185+0000 7fbe0de27640 1 -- 172.21.15.46:0/3545266673 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558b53e021c0 con 0x558b53dfddb0 2024-03-31T23:11:58.191 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:11:58.185+0000 7fbe0ce25640 1 --2- 172.21.15.46:0/3545266673 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558b53df4fe0 0x558b53e01430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:11:58.191 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:11:58.185+0000 7fbe0de27640 1 --2- 172.21.15.46:0/3545266673 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558b53dfddb0 0x558b53e01990 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7fbe0009b5a0 tx=0x7fbe00097f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=795aa68bf76f8d31 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:11:58.191 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:11:58.185+0000 7fbdf0ff9640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-31T23:11:58.192 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:11:58.185+0000 7fbdf0ff9640 1 -- 172.21.15.46:0/3545266673 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe0009f9f0 con 0x558b53dfddb0 2024-03-31T23:11:58.192 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:11:58.185+0000 7fbdf0ff9640 1 -- 172.21.15.46:0/3545266673 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7fbe00097f80 con 0x558b53dfddb0 2024-03-31T23:11:58.192 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:11:58.185+0000 7fbe12460740 1 -- 172.21.15.46:0/3545266673 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbe040054f0 con 0x558b53dfddb0 2024-03-31T23:11:58.192 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:11:58.185+0000 7fbdf0ff9640 1 -- 172.21.15.46:0/3545266673 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fbe0009b8a0 con 0x558b53dfddb0 2024-03-31T23:11:58.192 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:11:58.185+0000 7fbe12460740 10 client.15750 fetch_fsmap learned FSMap version 25 2024-03-31T23:11:58.192 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:11:58.185+0000 7fbe12460740 1 -- 172.21.15.46:0/3545266673 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x558b53e74260 con 0x558b53dfddb0 2024-03-31T23:11:58.192 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:11:58.185+0000 7fbdf0ff9640 1 -- 172.21.15.46:0/3545266673 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fbe040054f0 con 0x558b53dfddb0 2024-03-31T23:11:58.192 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:11:58.186+0000 7fbdf0ff9640 1 -- 172.21.15.46:0/3545266673 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7fbe0009cc60 con 0x558b53dfddb0 2024-03-31T23:11:58.192 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:11:58.186+0000 7fbe12460740 10 client.15750 fetch_fsmap finished waiting for FSMap version 25 2024-03-31T23:11:58.192 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:11:58.186+0000 7fbe12460740 10 client.15750 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-31T23:11:58.192 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:11:58.186+0000 7fbe12460740 20 client.15750 populate_metadata read hostname 'smithi046' 2024-03-31T23:11:58.192 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:11:58.186+0000 7fbe12460740 1 --2- 172.21.15.46:0/3545266673 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x558b53e74550 0x558b53e76940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:11:58.192 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:11:58.186+0000 7fbe12460740 4 client.15750 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-31T23:11:58.192 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:11:58.186+0000 7fbe12460740 1 -- 172.21.15.46:0/3545266673 --> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] -- command(tid 0: {"prefix": "flush journal"}) -- 0x558b53d64e30 con 0x558b53e74550 2024-03-31T23:11:58.193 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:11:58.186+0000 7fbe0d626640 1 --2- 172.21.15.46:0/3545266673 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x558b53e74550 0x558b53e76940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:11:58.193 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:11:58.186+0000 7fbe0d626640 1 --2- 172.21.15.46:0/3545266673 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x558b53e74550 0x558b53e76940 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c5d20536a763c0a8 server_cookie=81fe4daab5440a5 in_seq=0 out_seq=0 2024-03-31T23:11:58.193 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:11:58.186+0000 7fbdf0ff9640 10 client.15750 ms_handle_connect on v2:172.21.15.142:6838/2519143918 2024-03-31T23:11:58.200 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:11:58.198+0000 7fbdf0ff9640 1 -- 172.21.15.46:0/3545266673 <== mds.0 v2:172.21.15.142:6838/2519143918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x558b53d64e30 con 0x558b53e74550 2024-03-31T23:11:58.200 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:11:58.198+0000 7fbdf0ff9640 10 client.15750 handle_command_reply: tid=0 2024-03-31T23:11:58.200 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:11:58.198+0000 7fbe12460740 1 client.15750 shutdown 2024-03-31T23:11:58.200 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:11:58.199+0000 7fbe12460740 1 -- 172.21.15.46:0/3545266673 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558b53dfddb0 msgr2=0x558b53e01990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:58.200 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:11:58.199+0000 7fbe12460740 1 --2- 172.21.15.46:0/3545266673 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558b53dfddb0 0x558b53e01990 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7fbe0009b5a0 tx=0x7fbe00097f10 comp rx=0 tx=0).stop 2024-03-31T23:11:58.201 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:11:58.199+0000 7fbe12460740 1 -- 172.21.15.46:0/3545266673 shutdown_connections 2024-03-31T23:11:58.201 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:11:58.199+0000 7fbe12460740 1 --2- 172.21.15.46:0/3545266673 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x558b53e74550 0x558b53e76940 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:58.201 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:11:58.199+0000 7fbe12460740 1 --2- 172.21.15.46:0/3545266673 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558b53dfddb0 0x558b53e01990 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:58.201 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:11:58.199+0000 7fbe12460740 1 --2- 172.21.15.46:0/3545266673 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558b53df4fe0 0x558b53e01430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:58.201 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:11:58.199+0000 7fbdf0ff9640 0 client.15750 ms_handle_reset on v2:172.21.15.142:6838/2519143918 2024-03-31T23:11:58.201 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:11:58.199+0000 7fbe12460740 1 --2- 172.21.15.46:0/3545266673 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558b53df46c0 0x558b53e060c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:58.201 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:11:58.199+0000 7fbe12460740 1 -- 172.21.15.46:0/3545266673 >> 172.21.15.46:0/3545266673 conn(0x558b53deefe0 msgr2=0x558b53ded6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:11:58.201 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:11:58.199+0000 7fbe12460740 1 -- 172.21.15.46:0/3545266673 shutdown_connections 2024-03-31T23:11:58.201 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:11:58.199+0000 7fbe12460740 1 -- 172.21.15.46:0/3545266673 wait complete. 2024-03-31T23:11:58.201 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:11:58.199+0000 7fbe12460740 20 client.15750 trim_cache size 0 max 16384 2024-03-31T23:11:58.202 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout:{ 2024-03-31T23:11:58.202 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout: "message": "", 2024-03-31T23:11:58.202 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout: "return_code": 0 2024-03-31T23:11:58.202 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout:} 2024-03-31T23:11:58.203 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:11:58.199+0000 7fbe0f88a640 1 -- 172.21.15.46:0/2156156532 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fbdf40a2750 msgr2=0x7fbdf40a4c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:58.203 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:11:58.199+0000 7fbe0f88a640 1 --2- 172.21.15.46:0/2156156532 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fbdf40a2750 0x7fbdf40a4c10 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fbe04000c00 tx=0x7fbe040058d0 comp rx=0 tx=0).stop 2024-03-31T23:11:58.203 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:11:58.199+0000 7fbe0f88a640 1 -- 172.21.15.46:0/2156156532 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbe08106650 msgr2=0x7fbe0819c750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:11:58.203 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:11:58.199+0000 7fbe0f88a640 1 --2- 172.21.15.46:0/2156156532 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbe08106650 0x7fbe0819c750 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7fbe00005fd0 tx=0x7fbe0000ee60 comp rx=0 tx=0).stop 2024-03-31T23:11:58.203 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:11:58.200+0000 7fbe0f88a640 1 -- 172.21.15.46:0/2156156532 shutdown_connections 2024-03-31T23:11:58.203 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:11:58.200+0000 7fbe0f88a640 1 --2- 172.21.15.46:0/2156156532 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fbdf40a2750 0x7fbdf40a4c10 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:58.203 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:11:58.200+0000 7fbe0f88a640 1 --2- 172.21.15.46:0/2156156532 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbe08106650 0x7fbe0819c750 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:58.203 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:11:58.200+0000 7fbe0f88a640 1 --2- 172.21.15.46:0/2156156532 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbe08101000 0x7fbe081983d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:58.203 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:11:58.200+0000 7fbe0f88a640 1 --2- 172.21.15.46:0/2156156532 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbe08100640 0x7fbe08197e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:11:58.203 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:11:58.200+0000 7fbe0f88a640 1 -- 172.21.15.46:0/2156156532 >> 172.21.15.46:0/2156156532 conn(0x7fbe08077860 msgr2=0x7fbe08103210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:11:58.203 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:11:58.200+0000 7fbe0f88a640 1 -- 172.21.15.46:0/2156156532 shutdown_connections 2024-03-31T23:11:58.203 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:11:58.200+0000 7fbe0f88a640 1 -- 172.21.15.46:0/2156156532 wait complete. 2024-03-31T23:11:58.545 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:11:58 smithi162 ceph-mon[36237]: pgmap v10: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 77 KiB/s wr, 47 op/s 2024-03-31T23:11:58.611 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:11:58 smithi046 ceph-mon[32498]: pgmap v10: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 77 KiB/s wr, 47 op/s 2024-03-31T23:11:58.693 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:11:58 smithi142 ceph-mon[35271]: pgmap v10: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 77 KiB/s wr, 47 op/s 2024-03-31T23:12:00.545 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:12:00 smithi162 ceph-mon[36237]: pgmap v11: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 77 KiB/s wr, 48 op/s 2024-03-31T23:12:00.611 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:12:00 smithi046 ceph-mon[32498]: pgmap v11: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 77 KiB/s wr, 48 op/s 2024-03-31T23:12:00.612 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 31 23:12:00 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mgr-x[32727]: 2024-03-31T23:12:00.172+0000 7f1ccdc75700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-31T23:12:00.612 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 31 23:12:00 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mgr-x[32727]: 2024-03-31T23:12:00.172+0000 7f1ccdc75700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-31T23:12:00.612 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 31 23:12:00 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mgr-x[32727]: 2024-03-31T23:12:00.172+0000 7f1ccdc75700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-31T23:12:00.612 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 31 23:12:00 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mgr-x[32727]: 2024-03-31T23:12:00.172+0000 7f1ccdc75700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-31T23:12:00.612 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 31 23:12:00 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mgr-x[32727]: 2024-03-31T23:12:00.172+0000 7f1ccdc75700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-31T23:12:00.693 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:12:00 smithi142 ceph-mon[35271]: pgmap v11: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 77 KiB/s wr, 48 op/s 2024-03-31T23:12:02.546 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:12:02 smithi162 ceph-mon[36237]: pgmap v12: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 79 KiB/s wr, 54 op/s 2024-03-31T23:12:02.546 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:12:02 smithi162 ceph-mon[36237]: mgrmap e24: x(active, since 18s), standbys: z, y 2024-03-31T23:12:02.611 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:12:02 smithi046 ceph-mon[32498]: pgmap v12: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 79 KiB/s wr, 54 op/s 2024-03-31T23:12:02.612 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:12:02 smithi046 ceph-mon[32498]: mgrmap e24: x(active, since 18s), standbys: z, y 2024-03-31T23:12:02.693 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:12:02 smithi142 ceph-mon[35271]: pgmap v12: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 79 KiB/s wr, 54 op/s 2024-03-31T23:12:02.694 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:12:02 smithi142 ceph-mon[35271]: mgrmap e24: x(active, since 18s), standbys: z, y 2024-03-31T23:12:04.545 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:12:04 smithi162 ceph-mon[36237]: pgmap v13: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 2.4 KiB/s wr, 10 op/s 2024-03-31T23:12:04.611 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:12:04 smithi046 ceph-mon[32498]: pgmap v13: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 2.4 KiB/s wr, 10 op/s 2024-03-31T23:12:04.672 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-31T23:12:04.672 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-31T23:12:04.693 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:12:04 smithi142 ceph-mon[35271]: pgmap v13: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 2.4 KiB/s wr, 10 op/s 2024-03-31T23:12:04.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:04.790+0000 7fc6f2fd0640 1 -- 172.21.15.46:0/3652132150 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc6ec0692f0 msgr2=0x7fc6ec10e590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:12:04.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:04.790+0000 7fc6f2fd0640 1 --2- 172.21.15.46:0/3652132150 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc6ec0692f0 0x7fc6ec10e590 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7fc6e4009cd0 tx=0x7fc6e402c0f0 comp rx=0 tx=0).stop 2024-03-31T23:12:04.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:04.790+0000 7fc6f2fd0640 1 -- 172.21.15.46:0/3652132150 shutdown_connections 2024-03-31T23:12:04.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:04.790+0000 7fc6f2fd0640 1 --2- 172.21.15.46:0/3652132150 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc6ec0692f0 0x7fc6ec10e590 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:04.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:04.790+0000 7fc6f2fd0640 1 --2- 172.21.15.46:0/3652132150 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc6ec068930 0x7fc6ec068db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:04.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:04.790+0000 7fc6f2fd0640 1 --2- 172.21.15.46:0/3652132150 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc6ec1076d0 0x7fc6ec107ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:04.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:04.790+0000 7fc6f2fd0640 1 -- 172.21.15.46:0/3652132150 >> 172.21.15.46:0/3652132150 conn(0x7fc6ec077750 msgr2=0x7fc6ec077b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:12:04.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:04.790+0000 7fc6f2fd0640 1 -- 172.21.15.46:0/3652132150 shutdown_connections 2024-03-31T23:12:04.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:04.790+0000 7fc6f2fd0640 1 -- 172.21.15.46:0/3652132150 wait complete. 2024-03-31T23:12:04.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:04.791+0000 7fc6f2fd0640 1 Processor -- start 2024-03-31T23:12:04.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:04.791+0000 7fc6f2fd0640 1 -- start start 2024-03-31T23:12:04.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:04.791+0000 7fc6f2fd0640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc6ec068930 0x7fc6ec197ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:04.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:04.791+0000 7fc6f2fd0640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc6ec0692f0 0x7fc6ec198430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:04.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:04.791+0000 7fc6f2fd0640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc6ec1076d0 0x7fc6ec19c7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:04.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:04.791+0000 7fc6f2fd0640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fc6ec111990 con 0x7fc6ec068930 2024-03-31T23:12:04.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:04.791+0000 7fc6f2fd0640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fc6ec111810 con 0x7fc6ec0692f0 2024-03-31T23:12:04.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:04.791+0000 7fc6f2fd0640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fc6ec111b10 con 0x7fc6ec1076d0 2024-03-31T23:12:04.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:04.791+0000 7fc6f0d6c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc6ec068930 0x7fc6ec197ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:04.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:04.791+0000 7fc6f0d6c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc6ec068930 0x7fc6ec197ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60552/0 (socket says 172.21.15.46:60552) 2024-03-31T23:12:04.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:04.791+0000 7fc6f0d6c640 1 -- 172.21.15.46:0/3506354304 learned_addr learned my addr 172.21.15.46:0/3506354304 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:12:04.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:04.791+0000 7fc6f156d640 1 --2- 172.21.15.46:0/3506354304 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc6ec1076d0 0x7fc6ec19c7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:04.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:04.791+0000 7fc6ebfff640 1 --2- 172.21.15.46:0/3506354304 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc6ec0692f0 0x7fc6ec198430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:04.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:04.791+0000 7fc6f0d6c640 1 -- 172.21.15.46:0/3506354304 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc6ec1076d0 msgr2=0x7fc6ec19c7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:12:04.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:04.791+0000 7fc6f0d6c640 1 --2- 172.21.15.46:0/3506354304 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc6ec1076d0 0x7fc6ec19c7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:04.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:04.791+0000 7fc6f0d6c640 1 -- 172.21.15.46:0/3506354304 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc6ec0692f0 msgr2=0x7fc6ec198430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:12:04.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:04.791+0000 7fc6f0d6c640 1 --2- 172.21.15.46:0/3506354304 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc6ec0692f0 0x7fc6ec198430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:04.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:04.791+0000 7fc6f0d6c640 1 -- 172.21.15.46:0/3506354304 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc6ec19cee0 con 0x7fc6ec068930 2024-03-31T23:12:04.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:04.792+0000 7fc6f0d6c640 1 --2- 172.21.15.46:0/3506354304 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc6ec068930 0x7fc6ec197ef0 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7fc6e000e9f0 tx=0x7fc6e000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:12:04.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:04.792+0000 7fc6e9ffb640 1 -- 172.21.15.46:0/3506354304 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6e000cd80 con 0x7fc6ec068930 2024-03-31T23:12:04.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:04.792+0000 7fc6e9ffb640 1 -- 172.21.15.46:0/3506354304 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc6e00108e0 con 0x7fc6ec068930 2024-03-31T23:12:04.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:04.792+0000 7fc6f2fd0640 1 -- 172.21.15.46:0/3506354304 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc6ec19d1d0 con 0x7fc6ec068930 2024-03-31T23:12:04.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:04.792+0000 7fc6e9ffb640 1 -- 172.21.15.46:0/3506354304 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6e0018920 con 0x7fc6ec068930 2024-03-31T23:12:04.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:04.792+0000 7fc6f2fd0640 1 -- 172.21.15.46:0/3506354304 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc6ec1a4a60 con 0x7fc6ec068930 2024-03-31T23:12:04.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:04.794+0000 7fc6e9ffb640 1 -- 172.21.15.46:0/3506354304 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 24) ==== 137553+0+0 (secure 0 0 0) 0x7fc6e0010a90 con 0x7fc6ec068930 2024-03-31T23:12:04.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:04.794+0000 7fc6f5bb5740 1 Processor -- start 2024-03-31T23:12:04.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:04.794+0000 7fc6f5bb5740 1 -- start start 2024-03-31T23:12:04.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:04.795+0000 7fc6f5bb5740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564ccaf4efc0 0x564ccaf4f3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:04.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:04.795+0000 7fc6f5bb5740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564ccaf4f8e0 0x564ccaf5c660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:04.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:04.795+0000 7fc6f5bb5740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x564ccaf5cba0 0x564ccaf5ef90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:04.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:04.795+0000 7fc6f5bb5740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x564ccaeb3180 con 0x564ccaf4efc0 2024-03-31T23:12:04.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:04.795+0000 7fc6f5bb5740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x564ccae5bb70 con 0x564ccaf5cba0 2024-03-31T23:12:04.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:04.795+0000 7fc6f5bb5740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x564ccaeaf220 con 0x564ccaf4f8e0 2024-03-31T23:12:04.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:04.795+0000 7fc6e9ffb640 1 --2- 172.21.15.46:0/3506354304 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fc6dc0a27b0 0x7fc6dc0a4c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:04.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:04.795+0000 7fc6f156d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564ccaf4efc0 0x564ccaf4f3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:04.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:04.795+0000 7fc6f156d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564ccaf4efc0 0x564ccaf4f3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60556/0 (socket says 172.21.15.46:60556) 2024-03-31T23:12:04.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:04.795+0000 7fc6f156d640 1 -- 172.21.15.46:0/2699718771 learned_addr learned my addr 172.21.15.46:0/2699718771 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:12:04.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:04.795+0000 7fc6f0d6c640 1 --2- 172.21.15.46:0/2699718771 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564ccaf4f8e0 0x564ccaf5c660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:04.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:04.795+0000 7fc6ebfff640 1 --2- 172.21.15.46:0/2699718771 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x564ccaf5cba0 0x564ccaf5ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:04.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:04.795+0000 7fc6f156d640 1 --2- 172.21.15.46:0/3506354304 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fc6dc0a27b0 0x7fc6dc0a4c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:04.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:04.795+0000 7fc6e9ffb640 1 -- 172.21.15.46:0/3506354304 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7fc6e00cf410 con 0x7fc6ec068930 2024-03-31T23:12:04.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:04.795+0000 7fc6f156d640 1 -- 172.21.15.46:0/2699718771 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564ccaf4f8e0 msgr2=0x564ccaf5c660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:12:04.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:04.795+0000 7fc6f156d640 1 --2- 172.21.15.46:0/2699718771 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564ccaf4f8e0 0x564ccaf5c660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:04.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:04.795+0000 7fc6f156d640 1 -- 172.21.15.46:0/2699718771 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x564ccaf5cba0 msgr2=0x564ccaf5ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:12:04.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:04.795+0000 7fc6f156d640 1 --2- 172.21.15.46:0/2699718771 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x564ccaf5cba0 0x564ccaf5ef90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:04.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:04.795+0000 7fc6f156d640 1 -- 172.21.15.46:0/2699718771 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564ccae6f7c0 con 0x564ccaf4efc0 2024-03-31T23:12:04.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:04.795+0000 7fc6f0d6c640 1 --2- 172.21.15.46:0/2699718771 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564ccaf4f8e0 0x564ccaf5c660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:12:04.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:04.795+0000 7fc6f156d640 1 --2- 172.21.15.46:0/2699718771 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564ccaf4efc0 0x564ccaf4f3a0 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7fc6e4004af0 tx=0x7fc6e4032740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5bfdc739f44fa9b9 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:12:04.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:04.795+0000 7fc6f156d640 1 --2- 172.21.15.46:0/3506354304 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fc6dc0a27b0 0x7fc6dc0a4c70 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fc6e402c700 tx=0x7fc6e4042000 comp rx=0 tx=0).ready entity=mgr.15654 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:12:04.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:04.796+0000 7fc6d6ffd640 1 -- 172.21.15.46:0/2699718771 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6e4051070 con 0x564ccaf4efc0 2024-03-31T23:12:04.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:04.796+0000 7fc6d6ffd640 1 -- 172.21.15.46:0/2699718771 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc6e4004040 con 0x564ccaf4efc0 2024-03-31T23:12:04.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:04.796+0000 7fc6d6ffd640 1 -- 172.21.15.46:0/2699718771 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6e4039040 con 0x564ccaf4efc0 2024-03-31T23:12:04.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:04.796+0000 7fc6f5bb5740 1 -- 172.21.15.46:0/2699718771 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564ccaf4efc0 msgr2=0x564ccaf4f3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:12:04.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:04.796+0000 7fc6f5bb5740 1 --2- 172.21.15.46:0/2699718771 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564ccaf4efc0 0x564ccaf4f3a0 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7fc6e4004af0 tx=0x7fc6e4032740 comp rx=0 tx=0).stop 2024-03-31T23:12:04.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:04.796+0000 7fc6f5bb5740 1 -- 172.21.15.46:0/2699718771 shutdown_connections 2024-03-31T23:12:04.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:04.796+0000 7fc6f5bb5740 1 --2- 172.21.15.46:0/2699718771 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x564ccaf5cba0 0x564ccaf5ef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:04.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:04.796+0000 7fc6f5bb5740 1 --2- 172.21.15.46:0/2699718771 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564ccaf4f8e0 0x564ccaf5c660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:04.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:04.796+0000 7fc6f5bb5740 1 --2- 172.21.15.46:0/2699718771 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564ccaf4efc0 0x564ccaf4f3a0 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:04.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:04.796+0000 7fc6f5bb5740 1 -- 172.21.15.46:0/2699718771 >> 172.21.15.46:0/2699718771 conn(0x564ccaf499c0 msgr2=0x564ccaf49da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:12:04.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:04.796+0000 7fc6f5bb5740 1 -- 172.21.15.46:0/2699718771 shutdown_connections 2024-03-31T23:12:04.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:04.796+0000 7fc6f5bb5740 1 -- 172.21.15.46:0/2699718771 wait complete. 2024-03-31T23:12:04.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:04.797+0000 7fc6f5bb5740 1 -- start start 2024-03-31T23:12:04.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:04.797+0000 7fc6f5bb5740 1 Processor -- start 2024-03-31T23:12:04.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:04.797+0000 7fc6f5bb5740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564ccaf4efc0 0x564ccaf59c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:04.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:04.797+0000 7fc6f5bb5740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564ccaf4f8e0 0x564ccaf5a190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:04.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:04.797+0000 7fc6f5bb5740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x564ccaf5cba0 0x564ccaf5a6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:04.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:04.797+0000 7fc6f5bb5740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x564ccaf5f900 con 0x564ccaf4efc0 2024-03-31T23:12:04.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:04.797+0000 7fc6f5bb5740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x564ccae514a0 con 0x564ccaf5cba0 2024-03-31T23:12:04.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:04.797+0000 7fc6f5bb5740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x564ccaf5fa80 con 0x564ccaf4f8e0 2024-03-31T23:12:04.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:04.797+0000 7fc6f5bb5740 10 client.0 _ll_register_callbacks cb 0x564ccae54d90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T23:12:04.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:04.797+0000 7fc6f0d6c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564ccaf4efc0 0x564ccaf59c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:04.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:04.797+0000 7fc6f0d6c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564ccaf4efc0 0x564ccaf59c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60570/0 (socket says 172.21.15.46:60570) 2024-03-31T23:12:04.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:04.797+0000 7fc6f0d6c640 1 -- 172.21.15.46:0/1091344481 learned_addr learned my addr 172.21.15.46:0/1091344481 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:12:04.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:04.797+0000 7fc6f156d640 1 --2- 172.21.15.46:0/1091344481 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x564ccaf5cba0 0x564ccaf5a6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:04.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:04.797+0000 7fc6ebfff640 1 --2- 172.21.15.46:0/1091344481 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564ccaf4f8e0 0x564ccaf5a190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:04.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:04.798+0000 7fc6f0d6c640 1 -- 172.21.15.46:0/1091344481 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564ccaf4f8e0 msgr2=0x564ccaf5a190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:12:04.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:04.798+0000 7fc6f0d6c640 1 --2- 172.21.15.46:0/1091344481 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564ccaf4f8e0 0x564ccaf5a190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:04.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:04.798+0000 7fc6f0d6c640 1 -- 172.21.15.46:0/1091344481 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x564ccaf5cba0 msgr2=0x564ccaf5a6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:12:04.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:04.798+0000 7fc6f0d6c640 1 --2- 172.21.15.46:0/1091344481 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x564ccaf5cba0 0x564ccaf5a6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:04.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:04.798+0000 7fc6f0d6c640 1 -- 172.21.15.46:0/1091344481 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564ccac917d0 con 0x564ccaf4efc0 2024-03-31T23:12:04.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:04.798+0000 7fc6f0d6c640 1 --2- 172.21.15.46:0/1091344481 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564ccaf4efc0 0x564ccaf59c50 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7fc6e009b570 tx=0x7fc6e0010de0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b4cc66e6dbc0ba6b server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:12:04.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:04.798+0000 7fc6cbfff640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-31T23:12:04.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:04.798+0000 7fc6cbfff640 1 -- 172.21.15.46:0/1091344481 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6e00a3f40 con 0x564ccaf4efc0 2024-03-31T23:12:04.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:04.798+0000 7fc6cbfff640 1 -- 172.21.15.46:0/1091344481 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7fc6e00bb110 con 0x564ccaf4efc0 2024-03-31T23:12:04.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:04.798+0000 7fc6f5bb5740 1 -- 172.21.15.46:0/1091344481 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc6e4004040 con 0x564ccaf4efc0 2024-03-31T23:12:04.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:04.798+0000 7fc6cbfff640 1 -- 172.21.15.46:0/1091344481 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fc6e00bb330 con 0x564ccaf4efc0 2024-03-31T23:12:04.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:04.798+0000 7fc6f5bb5740 10 client.15768 fetch_fsmap learned FSMap version 25 2024-03-31T23:12:04.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:04.798+0000 7fc6f5bb5740 1 -- 172.21.15.46:0/1091344481 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x564ccabc73f0 con 0x564ccaf4efc0 2024-03-31T23:12:04.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:04.798+0000 7fc6cbfff640 1 -- 172.21.15.46:0/1091344481 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc6e4004040 con 0x564ccaf4efc0 2024-03-31T23:12:04.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:04.798+0000 7fc6cbfff640 1 -- 172.21.15.46:0/1091344481 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7fc6e009aa30 con 0x564ccaf4efc0 2024-03-31T23:12:04.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:04.798+0000 7fc6f5bb5740 10 client.15768 fetch_fsmap finished waiting for FSMap version 25 2024-03-31T23:12:04.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:04.798+0000 7fc6f5bb5740 10 client.15768 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-31T23:12:04.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:04.799+0000 7fc6f5bb5740 20 client.15768 populate_metadata read hostname 'smithi046' 2024-03-31T23:12:04.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:04.799+0000 7fc6f5bb5740 1 --2- 172.21.15.46:0/1091344481 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x564ccaf6cf10 0x564ccaf6f340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:04.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:04.799+0000 7fc6f5bb5740 4 client.15768 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-31T23:12:04.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:04.799+0000 7fc6f5bb5740 1 -- 172.21.15.46:0/1091344481 --> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564ccaec06c0 con 0x564ccaf6cf10 2024-03-31T23:12:04.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:04.799+0000 7fc6ebfff640 1 --2- 172.21.15.46:0/1091344481 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x564ccaf6cf10 0x564ccaf6f340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:04.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:04.799+0000 7fc6cbfff640 10 client.15768 ms_handle_connect on v2:172.21.15.142:6838/2519143918 2024-03-31T23:12:04.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:04.799+0000 7fc6ebfff640 1 --2- 172.21.15.46:0/1091344481 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x564ccaf6cf10 0x564ccaf6f340 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4957b53495159081 server_cookie=5591b791cd102493 in_seq=0 out_seq=0 2024-03-31T23:12:04.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:04.800+0000 7fc6cbfff640 1 -- 172.21.15.46:0/1091344481 <== mds.0 v2:172.21.15.142:6838/2519143918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x564ccaec06c0 con 0x564ccaf6cf10 2024-03-31T23:12:04.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:04.800+0000 7fc6cbfff640 10 client.15768 handle_command_reply: tid=0 2024-03-31T23:12:04.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:04.800+0000 7fc6f5bb5740 1 client.15768 shutdown 2024-03-31T23:12:04.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:04.800+0000 7fc6f5bb5740 1 -- 172.21.15.46:0/1091344481 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564ccaf4efc0 msgr2=0x564ccaf59c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:12:04.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:04.800+0000 7fc6f5bb5740 1 --2- 172.21.15.46:0/1091344481 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564ccaf4efc0 0x564ccaf59c50 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7fc6e009b570 tx=0x7fc6e0010de0 comp rx=0 tx=0).stop 2024-03-31T23:12:04.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:04.800+0000 7fc6f5bb5740 1 -- 172.21.15.46:0/1091344481 shutdown_connections 2024-03-31T23:12:04.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:04.800+0000 7fc6f5bb5740 1 --2- 172.21.15.46:0/1091344481 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x564ccaf6cf10 0x564ccaf6f340 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:04.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:04.800+0000 7fc6f5bb5740 1 --2- 172.21.15.46:0/1091344481 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x564ccaf5cba0 0x564ccaf5a6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:04.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:04.800+0000 7fc6f5bb5740 1 --2- 172.21.15.46:0/1091344481 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564ccaf4f8e0 0x564ccaf5a190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:04.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:04.800+0000 7fc6f5bb5740 1 --2- 172.21.15.46:0/1091344481 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564ccaf4efc0 0x564ccaf59c50 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:04.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:04.800+0000 7fc6cbfff640 0 client.15768 ms_handle_reset on v2:172.21.15.142:6838/2519143918 2024-03-31T23:12:04.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:04.800+0000 7fc6f5bb5740 1 -- 172.21.15.46:0/1091344481 >> 172.21.15.46:0/1091344481 conn(0x564ccaf499c0 msgr2=0x564ccaf4b710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:12:04.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:04.800+0000 7fc6f5bb5740 1 -- 172.21.15.46:0/1091344481 shutdown_connections 2024-03-31T23:12:04.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:04.800+0000 7fc6f5bb5740 1 -- 172.21.15.46:0/1091344481 wait complete. 2024-03-31T23:12:04.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:04.801+0000 7fc6f5bb5740 20 client.15768 trim_cache size 0 max 16384 2024-03-31T23:12:04.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:04.818+0000 7fc6f5bb5740 1 Processor -- start 2024-03-31T23:12:04.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:04.818+0000 7fc6f5bb5740 1 -- start start 2024-03-31T23:12:04.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:04.818+0000 7fc6f5bb5740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564ccaf4efc0 0x564ccaf686e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:04.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:04.818+0000 7fc6f5bb5740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564ccaf4f8e0 0x564ccaf4b4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:04.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:04.818+0000 7fc6f5bb5740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x564ccaf5cba0 0x564ccaf74bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:04.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:04.818+0000 7fc6f5bb5740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fc6ec1af310 con 0x564ccaf4f8e0 2024-03-31T23:12:04.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:04.818+0000 7fc6f5bb5740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fc6ec1acbe0 con 0x564ccaf5cba0 2024-03-31T23:12:04.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:04.818+0000 7fc6f5bb5740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fc6ec1aaa30 con 0x564ccaf4efc0 2024-03-31T23:12:04.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:04.818+0000 7fc6ebfff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564ccaf4f8e0 0x564ccaf4b4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:04.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:04.818+0000 7fc6ebfff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564ccaf4f8e0 0x564ccaf4b4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60580/0 (socket says 172.21.15.46:60580) 2024-03-31T23:12:04.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:04.818+0000 7fc6ebfff640 1 -- 172.21.15.46:0/4037522176 learned_addr learned my addr 172.21.15.46:0/4037522176 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:12:04.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:04.818+0000 7fc6f0d6c640 1 --2- 172.21.15.46:0/4037522176 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564ccaf4efc0 0x564ccaf686e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:04.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:04.818+0000 7fc6f156d640 1 --2- 172.21.15.46:0/4037522176 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x564ccaf5cba0 0x564ccaf74bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:04.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:04.819+0000 7fc6ebfff640 1 -- 172.21.15.46:0/4037522176 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564ccaf4efc0 msgr2=0x564ccaf686e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:12:04.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:04.819+0000 7fc6ebfff640 1 --2- 172.21.15.46:0/4037522176 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564ccaf4efc0 0x564ccaf686e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:04.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:04.819+0000 7fc6ebfff640 1 -- 172.21.15.46:0/4037522176 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x564ccaf5cba0 msgr2=0x564ccaf74bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:12:04.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:04.819+0000 7fc6ebfff640 1 --2- 172.21.15.46:0/4037522176 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x564ccaf5cba0 0x564ccaf74bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:04.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:04.819+0000 7fc6ebfff640 1 -- 172.21.15.46:0/4037522176 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564ccaf4f710 con 0x564ccaf4f8e0 2024-03-31T23:12:04.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:04.819+0000 7fc6ebfff640 1 --2- 172.21.15.46:0/4037522176 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564ccaf4f8e0 0x564ccaf4b4b0 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x564ccaf594b0 tx=0x7fc6d8005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=22c18b0316176c1 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:12:04.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:04.819+0000 7fc6d67fc640 1 -- 172.21.15.46:0/4037522176 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6d8003e40 con 0x564ccaf4f8e0 2024-03-31T23:12:04.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:04.819+0000 7fc6d67fc640 1 -- 172.21.15.46:0/4037522176 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc6d8025070 con 0x564ccaf4f8e0 2024-03-31T23:12:04.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:04.819+0000 7fc6d67fc640 1 -- 172.21.15.46:0/4037522176 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6d8010b80 con 0x564ccaf4f8e0 2024-03-31T23:12:04.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:04.819+0000 7fc6f5bb5740 1 -- 172.21.15.46:0/4037522176 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564ccaf4f8e0 msgr2=0x564ccaf4b4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:12:04.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:04.819+0000 7fc6f5bb5740 1 --2- 172.21.15.46:0/4037522176 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564ccaf4f8e0 0x564ccaf4b4b0 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x564ccaf594b0 tx=0x7fc6d8005f90 comp rx=0 tx=0).stop 2024-03-31T23:12:04.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:04.819+0000 7fc6f5bb5740 1 -- 172.21.15.46:0/4037522176 shutdown_connections 2024-03-31T23:12:04.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:04.819+0000 7fc6f5bb5740 1 --2- 172.21.15.46:0/4037522176 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x564ccaf5cba0 0x564ccaf74bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:04.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:04.819+0000 7fc6f5bb5740 1 --2- 172.21.15.46:0/4037522176 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564ccaf4f8e0 0x564ccaf4b4b0 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:04.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:04.819+0000 7fc6f5bb5740 1 --2- 172.21.15.46:0/4037522176 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564ccaf4efc0 0x564ccaf686e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:04.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:04.819+0000 7fc6f5bb5740 1 -- 172.21.15.46:0/4037522176 >> 172.21.15.46:0/4037522176 conn(0x564ccaf499c0 msgr2=0x564ccaf4ada0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:12:04.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:04.820+0000 7fc6f5bb5740 1 -- 172.21.15.46:0/4037522176 shutdown_connections 2024-03-31T23:12:04.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:04.820+0000 7fc6f5bb5740 1 -- 172.21.15.46:0/4037522176 wait complete. 2024-03-31T23:12:04.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:04.820+0000 7fc6f5bb5740 1 -- start start 2024-03-31T23:12:04.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:04.820+0000 7fc6f5bb5740 1 Processor -- start 2024-03-31T23:12:04.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:04.820+0000 7fc6f5bb5740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x564ccaf4efc0 0x564ccaf56b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:04.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:04.820+0000 7fc6f5bb5740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564ccaf4f8e0 0x564ccaf570b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:04.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:04.820+0000 7fc6f5bb5740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564ccaf5cba0 0x564ccafd21f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:04.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:04.820+0000 7fc6f5bb5740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fc6ec1ab0a0 con 0x564ccaf4f8e0 2024-03-31T23:12:04.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:04.820+0000 7fc6f5bb5740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fc6ec1ac510 con 0x564ccaf4efc0 2024-03-31T23:12:04.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:04.820+0000 7fc6f5bb5740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fc6ec1ab770 con 0x564ccaf5cba0 2024-03-31T23:12:04.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:04.820+0000 7fc6f5bb5740 10 client.0 _ll_register_callbacks cb 0x7fc6e009b3b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T23:12:04.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:04.820+0000 7fc6ebfff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564ccaf4f8e0 0x564ccaf570b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:04.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:04.820+0000 7fc6ebfff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564ccaf4f8e0 0x564ccaf570b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60586/0 (socket says 172.21.15.46:60586) 2024-03-31T23:12:04.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:04.820+0000 7fc6ebfff640 1 -- 172.21.15.46:0/3505456985 learned_addr learned my addr 172.21.15.46:0/3505456985 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:12:04.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:04.820+0000 7fc6f156d640 1 --2- 172.21.15.46:0/3505456985 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564ccaf5cba0 0x564ccafd21f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:04.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:04.820+0000 7fc6f0d6c640 1 --2- 172.21.15.46:0/3505456985 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x564ccaf4efc0 0x564ccaf56b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:04.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:04.821+0000 7fc6ebfff640 1 -- 172.21.15.46:0/3505456985 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564ccaf5cba0 msgr2=0x564ccafd21f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:12:04.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:04.821+0000 7fc6ebfff640 1 --2- 172.21.15.46:0/3505456985 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564ccaf5cba0 0x564ccafd21f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:04.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:04.821+0000 7fc6ebfff640 1 -- 172.21.15.46:0/3505456985 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x564ccaf4efc0 msgr2=0x564ccaf56b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:12:04.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:04.821+0000 7fc6ebfff640 1 --2- 172.21.15.46:0/3505456985 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x564ccaf4efc0 0x564ccaf56b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:04.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:04.821+0000 7fc6ebfff640 1 -- 172.21.15.46:0/3505456985 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564ccab870f0 con 0x564ccaf4f8e0 2024-03-31T23:12:04.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:04.821+0000 7fc6ebfff640 1 --2- 172.21.15.46:0/3505456985 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564ccaf4f8e0 0x564ccaf570b0 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7fc6d8009dd0 tx=0x7fc6d8009b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d6fab73e70416092 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:12:04.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:04.821+0000 7fc6cbfff640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-31T23:12:04.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:04.821+0000 7fc6cbfff640 1 -- 172.21.15.46:0/3505456985 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6d8010b00 con 0x564ccaf4f8e0 2024-03-31T23:12:04.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:04.821+0000 7fc6cbfff640 1 -- 172.21.15.46:0/3505456985 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7fc6d8010ca0 con 0x564ccaf4f8e0 2024-03-31T23:12:04.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:04.821+0000 7fc6f5bb5740 1 -- 172.21.15.46:0/3505456985 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc6d8025070 con 0x564ccaf4f8e0 2024-03-31T23:12:04.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:04.821+0000 7fc6cbfff640 1 -- 172.21.15.46:0/3505456985 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fc6d80200f0 con 0x564ccaf4f8e0 2024-03-31T23:12:04.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:04.821+0000 7fc6f5bb5740 10 client.15774 fetch_fsmap learned FSMap version 25 2024-03-31T23:12:04.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:04.821+0000 7fc6f5bb5740 1 -- 172.21.15.46:0/3505456985 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x564ccafd3680 con 0x564ccaf4f8e0 2024-03-31T23:12:04.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:04.821+0000 7fc6cbfff640 1 -- 172.21.15.46:0/3505456985 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc6d8025070 con 0x564ccaf4f8e0 2024-03-31T23:12:04.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:04.822+0000 7fc6cbfff640 1 -- 172.21.15.46:0/3505456985 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7fc6e0010430 con 0x564ccaf4f8e0 2024-03-31T23:12:04.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:04.822+0000 7fc6f5bb5740 10 client.15774 fetch_fsmap finished waiting for FSMap version 25 2024-03-31T23:12:04.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:04.822+0000 7fc6f5bb5740 10 client.15774 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-31T23:12:04.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:04.822+0000 7fc6f5bb5740 20 client.15774 populate_metadata read hostname 'smithi046' 2024-03-31T23:12:04.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:04.822+0000 7fc6f5bb5740 1 --2- 172.21.15.46:0/3505456985 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x564ccafd3dd0 0x564ccafd61f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:04.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:04.822+0000 7fc6f5bb5740 4 client.15774 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-31T23:12:04.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:04.822+0000 7fc6f5bb5740 1 -- 172.21.15.46:0/3505456985 --> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] -- command(tid 0: {"prefix": "scrub status"}) -- 0x564ccaebaa10 con 0x564ccafd3dd0 2024-03-31T23:12:04.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:04.822+0000 7fc6f0d6c640 1 --2- 172.21.15.46:0/3505456985 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x564ccafd3dd0 0x564ccafd61f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:04.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:04.822+0000 7fc6f0d6c640 1 --2- 172.21.15.46:0/3505456985 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x564ccafd3dd0 0x564ccafd61f0 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7645e871762906df server_cookie=6b502001858bbb41 in_seq=0 out_seq=0 2024-03-31T23:12:04.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:04.822+0000 7fc6cbfff640 10 client.15774 ms_handle_connect on v2:172.21.15.142:6838/2519143918 2024-03-31T23:12:04.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:04.822+0000 7fc6cbfff640 1 -- 172.21.15.46:0/3505456985 <== mds.0 v2:172.21.15.142:6838/2519143918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x564ccaebaa10 con 0x564ccafd3dd0 2024-03-31T23:12:04.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:04.822+0000 7fc6cbfff640 10 client.15774 handle_command_reply: tid=0 2024-03-31T23:12:04.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:04.822+0000 7fc6f5bb5740 1 client.15774 shutdown 2024-03-31T23:12:04.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:04.822+0000 7fc6f5bb5740 1 -- 172.21.15.46:0/3505456985 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564ccaf4f8e0 msgr2=0x564ccaf570b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:12:04.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:04.822+0000 7fc6f5bb5740 1 --2- 172.21.15.46:0/3505456985 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564ccaf4f8e0 0x564ccaf570b0 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7fc6d8009dd0 tx=0x7fc6d8009b90 comp rx=0 tx=0).stop 2024-03-31T23:12:04.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:04.823+0000 7fc6f5bb5740 1 -- 172.21.15.46:0/3505456985 shutdown_connections 2024-03-31T23:12:04.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:04.823+0000 7fc6f5bb5740 1 --2- 172.21.15.46:0/3505456985 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x564ccafd3dd0 0x564ccafd61f0 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:04.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:04.823+0000 7fc6f5bb5740 1 --2- 172.21.15.46:0/3505456985 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564ccaf5cba0 0x564ccafd21f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:04.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:04.823+0000 7fc6f5bb5740 1 --2- 172.21.15.46:0/3505456985 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564ccaf4f8e0 0x564ccaf570b0 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:04.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:04.823+0000 7fc6f5bb5740 1 --2- 172.21.15.46:0/3505456985 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x564ccaf4efc0 0x564ccaf56b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:04.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:04.823+0000 7fc6cbfff640 0 client.15774 ms_handle_reset on v2:172.21.15.142:6838/2519143918 2024-03-31T23:12:04.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:04.823+0000 7fc6f5bb5740 1 -- 172.21.15.46:0/3505456985 >> 172.21.15.46:0/3505456985 conn(0x564ccaf499c0 msgr2=0x564ccaf4b410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:12:04.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:04.823+0000 7fc6f5bb5740 1 -- 172.21.15.46:0/3505456985 shutdown_connections 2024-03-31T23:12:04.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:04.823+0000 7fc6f5bb5740 1 -- 172.21.15.46:0/3505456985 wait complete. 2024-03-31T23:12:04.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:04.823+0000 7fc6f5bb5740 20 client.15774 trim_cache size 0 max 16384 2024-03-31T23:12:04.832 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-03-31T23:12:04.832 INFO:teuthology.orchestra.run.smithi046.stdout: "status": "no active scrubs running", 2024-03-31T23:12:04.832 INFO:teuthology.orchestra.run.smithi046.stdout: "scrubs": {} 2024-03-31T23:12:04.832 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-03-31T23:12:04.832 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:04.823+0000 7fc6f2fd0640 1 -- 172.21.15.46:0/3506354304 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fc6dc0a27b0 msgr2=0x7fc6dc0a4c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:12:04.832 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:04.823+0000 7fc6f2fd0640 1 --2- 172.21.15.46:0/3506354304 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fc6dc0a27b0 0x7fc6dc0a4c70 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fc6e402c700 tx=0x7fc6e4042000 comp rx=0 tx=0).stop 2024-03-31T23:12:04.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:04.823+0000 7fc6f2fd0640 1 -- 172.21.15.46:0/3506354304 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc6ec068930 msgr2=0x7fc6ec197ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:12:04.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:04.823+0000 7fc6f2fd0640 1 --2- 172.21.15.46:0/3506354304 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc6ec068930 0x7fc6ec197ef0 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7fc6e000e9f0 tx=0x7fc6e000eeb0 comp rx=0 tx=0).stop 2024-03-31T23:12:04.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:04.823+0000 7fc6f2fd0640 1 -- 172.21.15.46:0/3506354304 shutdown_connections 2024-03-31T23:12:04.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:04.823+0000 7fc6f2fd0640 1 --2- 172.21.15.46:0/3506354304 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fc6dc0a27b0 0x7fc6dc0a4c70 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:04.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:04.823+0000 7fc6f2fd0640 1 --2- 172.21.15.46:0/3506354304 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc6ec1076d0 0x7fc6ec19c7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:04.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:04.823+0000 7fc6f2fd0640 1 --2- 172.21.15.46:0/3506354304 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc6ec0692f0 0x7fc6ec198430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:04.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:04.823+0000 7fc6f2fd0640 1 --2- 172.21.15.46:0/3506354304 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc6ec068930 0x7fc6ec197ef0 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:04.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:04.823+0000 7fc6f2fd0640 1 -- 172.21.15.46:0/3506354304 >> 172.21.15.46:0/3506354304 conn(0x7fc6ec077750 msgr2=0x7fc6ec0fe320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:12:04.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:04.823+0000 7fc6f2fd0640 1 -- 172.21.15.46:0/3506354304 shutdown_connections 2024-03-31T23:12:04.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:04.823+0000 7fc6f2fd0640 1 -- 172.21.15.46:0/3506354304 wait complete. 2024-03-31T23:12:04.835 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-31T23:12:04.835 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-31T23:12:04.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:04.937+0000 7f5851d13640 1 -- 172.21.15.46:0/4230599202 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f584c1078f0 msgr2=0x7f584c107cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:12:04.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:04.937+0000 7f5851d13640 1 --2- 172.21.15.46:0/4230599202 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f584c1078f0 0x7f584c107cd0 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f583c009ca0 tx=0x7f583c02c180 comp rx=0 tx=0).stop 2024-03-31T23:12:04.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:04.937+0000 7f5851d13640 1 -- 172.21.15.46:0/4230599202 shutdown_connections 2024-03-31T23:12:04.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:04.937+0000 7f5851d13640 1 --2- 172.21.15.46:0/4230599202 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f584c1023a0 0x7f584c10e600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:04.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:04.937+0000 7f5851d13640 1 --2- 172.21.15.46:0/4230599202 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f584c1019e0 0x7f584c101e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:04.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:04.937+0000 7f5851d13640 1 --2- 172.21.15.46:0/4230599202 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f584c1078f0 0x7f584c107cd0 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:04.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:04.937+0000 7f5851d13640 1 -- 172.21.15.46:0/4230599202 >> 172.21.15.46:0/4230599202 conn(0x7f584c079740 msgr2=0x7f584c079b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:12:04.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:04.938+0000 7f5851d13640 1 -- 172.21.15.46:0/4230599202 shutdown_connections 2024-03-31T23:12:04.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:04.938+0000 7f5851d13640 1 -- 172.21.15.46:0/4230599202 wait complete. 2024-03-31T23:12:04.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:04.938+0000 7f5851d13640 1 Processor -- start 2024-03-31T23:12:04.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:04.938+0000 7f5851d13640 1 -- start start 2024-03-31T23:12:04.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:04.938+0000 7f5851d13640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f584c1019e0 0x7f584c197f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:04.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:04.938+0000 7f5851d13640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f584c1023a0 0x7f584c198480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:04.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:04.938+0000 7f5851d13640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f584c1078f0 0x7f584c19c800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:04.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:04.938+0000 7f5851d13640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f584c111a20 con 0x7f584c1078f0 2024-03-31T23:12:04.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:04.938+0000 7f5851d13640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f584c1118a0 con 0x7f584c1023a0 2024-03-31T23:12:04.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:04.938+0000 7f5851d13640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f584c111ba0 con 0x7f584c1019e0 2024-03-31T23:12:04.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:04.939+0000 7f584bfff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f584c1078f0 0x7f584c19c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:04.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:04.939+0000 7f584bfff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f584c1078f0 0x7f584c19c800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60606/0 (socket says 172.21.15.46:60606) 2024-03-31T23:12:04.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:04.939+0000 7f584bfff640 1 -- 172.21.15.46:0/3803581438 learned_addr learned my addr 172.21.15.46:0/3803581438 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:12:04.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:04.939+0000 7f584affd640 1 --2- 172.21.15.46:0/3803581438 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f584c1023a0 0x7f584c198480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:04.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:04.939+0000 7f584b7fe640 1 --2- 172.21.15.46:0/3803581438 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f584c1019e0 0x7f584c197f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:04.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:04.939+0000 7f584bfff640 1 -- 172.21.15.46:0/3803581438 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f584c1019e0 msgr2=0x7f584c197f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:12:04.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:04.939+0000 7f584bfff640 1 --2- 172.21.15.46:0/3803581438 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f584c1019e0 0x7f584c197f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:04.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:04.939+0000 7f584bfff640 1 -- 172.21.15.46:0/3803581438 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f584c1023a0 msgr2=0x7f584c198480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:12:04.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:04.939+0000 7f584bfff640 1 --2- 172.21.15.46:0/3803581438 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f584c1023a0 0x7f584c198480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:04.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:04.939+0000 7f584bfff640 1 -- 172.21.15.46:0/3803581438 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f584c19cf30 con 0x7f584c1078f0 2024-03-31T23:12:04.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:04.939+0000 7f584bfff640 1 --2- 172.21.15.46:0/3803581438 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f584c1078f0 0x7f584c19c800 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f584400e970 tx=0x7f584400ee30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:12:04.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:04.939+0000 7f5848ff9640 1 -- 172.21.15.46:0/3803581438 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f584400ccc0 con 0x7f584c1078f0 2024-03-31T23:12:04.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:04.939+0000 7f5848ff9640 1 -- 172.21.15.46:0/3803581438 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f584400ce60 con 0x7f584c1078f0 2024-03-31T23:12:04.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:04.939+0000 7f5851d13640 1 -- 172.21.15.46:0/3803581438 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f584c19d220 con 0x7f584c1078f0 2024-03-31T23:12:04.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:04.939+0000 7f5848ff9640 1 -- 172.21.15.46:0/3803581438 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5844010430 con 0x7f584c1078f0 2024-03-31T23:12:04.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:04.939+0000 7f5851d13640 1 -- 172.21.15.46:0/3803581438 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f584c1a4ab0 con 0x7f584c1078f0 2024-03-31T23:12:04.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:04.940+0000 7f5851d13640 1 -- 172.21.15.46:0/3803581438 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f584c068540 con 0x7f584c1078f0 2024-03-31T23:12:04.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:04.941+0000 7f5848ff9640 1 -- 172.21.15.46:0/3803581438 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 24) ==== 137553+0+0 (secure 0 0 0) 0x7f5844004f00 con 0x7f584c1078f0 2024-03-31T23:12:04.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:04.942+0000 7f5848ff9640 1 --2- 172.21.15.46:0/3803581438 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f58340a27b0 0x7f58340a4c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:04.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:04.942+0000 7f5848ff9640 1 -- 172.21.15.46:0/3803581438 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7f58440cf3f0 con 0x7f584c1078f0 2024-03-31T23:12:04.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:04.942+0000 7f584b7fe640 1 --2- 172.21.15.46:0/3803581438 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f58340a27b0 0x7f58340a4c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:04.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:04.942+0000 7f584b7fe640 1 --2- 172.21.15.46:0/3803581438 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f58340a27b0 0x7f58340a4c70 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f583c004af0 tx=0x7f583c005c50 comp rx=0 tx=0).ready entity=mgr.15654 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:12:04.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:04.947+0000 7f5848ff9640 1 -- 172.21.15.46:0/3803581438 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f584409bc10 con 0x7f584c1078f0 2024-03-31T23:12:05.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.166+0000 7f5851d13640 1 -- 172.21.15.46:0/3803581438 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f584c109cb0 con 0x7f584c1078f0 2024-03-31T23:12:05.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.167+0000 7f5848ff9640 1 -- 172.21.15.46:0/3803581438 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+11882 (secure 0 0 0) 0x7f584409b510 con 0x7f584c1078f0 2024-03-31T23:12:05.168 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-31T23:12:05.168 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/2169018547","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":2169018547},{"type":"v1","addr":"172.21.15.46:6835","nonce":2169018547}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6837/1069387756","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6836","nonce":1069387756},{"type":"v1","addr":"172.21.15.46:6837","nonce":1069387756}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14634,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6839/3915337844","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6838","nonce":3915337844},{"type":"v1","addr":"172.21.15.46:6839","nonce":3915337844}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14646,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6841/897767145","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6840","nonce":897767145},{"type":"v1","addr":"172.21.15.46:6841","nonce":897767145}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24415,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6833/1746950966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6832","nonce":1746950966},{"type":"v1","addr":"172.21.15.162:6833","nonce":1746950966}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24422,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6833/3394739181","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6832","nonce":3394739181},{"type":"v1","addr":"172.21.15.142:6833","nonce":3394739181}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":25,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-31T23:09:11.009090+0000","modified":"2024-03-31T23:11:27.558846+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24515,"mds_1":24482,"mds_2":24451},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24442":{"gid":24442,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.162:6835/1957208984","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6834","nonce":1957208984},{"type":"v1","addr":"172.21.15.162:6835","nonce":1957208984}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24451":{"gid":24451,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":11,"addr":"172.21.15.162:6837/1371886876","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6836","nonce":1371886876},{"type":"v1","addr":"172.21.15.162:6837","nonce":1371886876}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24455":{"gid":24455,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.142:6835/212637431","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6834","nonce":212637431},{"type":"v1","addr":"172.21.15.142:6835","nonce":212637431}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24478":{"gid":24478,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.162:6839/1976275826","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6838","nonce":1976275826},{"type":"v1","addr":"172.21.15.162:6839","nonce":1976275826}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24482":{"gid":24482,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":10,"addr":"172.21.15.142:6837/2559258399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6836","nonce":2559258399},{"type":"v1","addr":"172.21.15.142:6837","nonce":2559258399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24515":{"gid":24515,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.142:6839/2519143918","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6838","nonce":2519143918},{"type":"v1","addr":"172.21.15.142:6839","nonce":2519143918}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24515,"qdb_cluster":[24515,24451,24482]},"id":1}]} 2024-03-31T23:12:05.168 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 25 2024-03-31T23:12:05.170 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.169+0000 7f5851d13640 1 -- 172.21.15.46:0/3803581438 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f58340a27b0 msgr2=0x7f58340a4c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:12:05.170 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.169+0000 7f5851d13640 1 --2- 172.21.15.46:0/3803581438 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f58340a27b0 0x7f58340a4c70 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f583c004af0 tx=0x7f583c005c50 comp rx=0 tx=0).stop 2024-03-31T23:12:05.170 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.169+0000 7f5851d13640 1 -- 172.21.15.46:0/3803581438 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f584c1078f0 msgr2=0x7f584c19c800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:12:05.170 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.169+0000 7f5851d13640 1 --2- 172.21.15.46:0/3803581438 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f584c1078f0 0x7f584c19c800 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f584400e970 tx=0x7f584400ee30 comp rx=0 tx=0).stop 2024-03-31T23:12:05.170 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.169+0000 7f5851d13640 1 -- 172.21.15.46:0/3803581438 shutdown_connections 2024-03-31T23:12:05.170 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.169+0000 7f5851d13640 1 --2- 172.21.15.46:0/3803581438 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f58340a27b0 0x7f58340a4c70 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:05.170 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.169+0000 7f5851d13640 1 --2- 172.21.15.46:0/3803581438 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f584c1078f0 0x7f584c19c800 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:05.170 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.169+0000 7f5851d13640 1 --2- 172.21.15.46:0/3803581438 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f584c1023a0 0x7f584c198480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:05.171 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.169+0000 7f5851d13640 1 --2- 172.21.15.46:0/3803581438 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f584c1019e0 0x7f584c197f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:05.171 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.169+0000 7f5851d13640 1 -- 172.21.15.46:0/3803581438 >> 172.21.15.46:0/3803581438 conn(0x7f584c079740 msgr2=0x7f584c100710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:12:05.171 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.169+0000 7f5851d13640 1 -- 172.21.15.46:0/3803581438 shutdown_connections 2024-03-31T23:12:05.171 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.169+0000 7f5851d13640 1 -- 172.21.15.46:0/3803581438 wait complete. 2024-03-31T23:12:05.181 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-31T23:12:05.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.284+0000 7f5b3ef8a640 1 -- 172.21.15.46:0/1582428084 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5b38100f80 msgr2=0x7f5b3810d490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:12:05.288 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.284+0000 7f5b3ef8a640 1 --2- 172.21.15.46:0/1582428084 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5b38100f80 0x7f5b3810d490 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f5b30009ca0 tx=0x7f5b3002c0f0 comp rx=0 tx=0).stop 2024-03-31T23:12:05.288 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.284+0000 7f5b3ef8a640 1 -- 172.21.15.46:0/1582428084 shutdown_connections 2024-03-31T23:12:05.288 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.284+0000 7f5b3ef8a640 1 --2- 172.21.15.46:0/1582428084 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5b38100f80 0x7f5b3810d490 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:05.288 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.284+0000 7f5b3ef8a640 1 --2- 172.21.15.46:0/1582428084 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5b381005c0 0x7f5b38100a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:05.288 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.284+0000 7f5b3ef8a640 1 --2- 172.21.15.46:0/1582428084 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5b38106660 0x7f5b38106a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:05.288 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.284+0000 7f5b3ef8a640 1 -- 172.21.15.46:0/1582428084 >> 172.21.15.46:0/1582428084 conn(0x7f5b38077750 msgr2=0x7f5b38077b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:12:05.288 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.285+0000 7f5b3ef8a640 1 -- 172.21.15.46:0/1582428084 shutdown_connections 2024-03-31T23:12:05.288 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.285+0000 7f5b3ef8a640 1 -- 172.21.15.46:0/1582428084 wait complete. 2024-03-31T23:12:05.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.285+0000 7f5b3ef8a640 1 Processor -- start 2024-03-31T23:12:05.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.285+0000 7f5b3ef8a640 1 -- start start 2024-03-31T23:12:05.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.285+0000 7f5b3ef8a640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5b381005c0 0x7f5b38198c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:05.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.285+0000 7f5b3ef8a640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5b38100f80 0x7f5b38199150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:05.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.285+0000 7f5b3ef8a640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5b38106660 0x7f5b381a3c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:05.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.285+0000 7f5b3ef8a640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f5b38110850 con 0x7f5b381005c0 2024-03-31T23:12:05.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.285+0000 7f5b3ef8a640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f5b381106d0 con 0x7f5b38100f80 2024-03-31T23:12:05.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.285+0000 7f5b3ef8a640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f5b381109d0 con 0x7f5b38106660 2024-03-31T23:12:05.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.285+0000 7f5b3cd26640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5b381005c0 0x7f5b38198c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:05.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.285+0000 7f5b3cd26640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5b381005c0 0x7f5b38198c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60618/0 (socket says 172.21.15.46:60618) 2024-03-31T23:12:05.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.285+0000 7f5b3cd26640 1 -- 172.21.15.46:0/2087465464 learned_addr learned my addr 172.21.15.46:0/2087465464 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:12:05.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.285+0000 7f5b37fff640 1 --2- 172.21.15.46:0/2087465464 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5b38100f80 0x7f5b38199150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:05.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.286+0000 7f5b3d527640 1 --2- 172.21.15.46:0/2087465464 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5b38106660 0x7f5b381a3c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:05.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.286+0000 7f5b3cd26640 1 -- 172.21.15.46:0/2087465464 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5b38106660 msgr2=0x7f5b381a3c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:12:05.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.286+0000 7f5b3cd26640 1 --2- 172.21.15.46:0/2087465464 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5b38106660 0x7f5b381a3c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:05.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.286+0000 7f5b3cd26640 1 -- 172.21.15.46:0/2087465464 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5b38100f80 msgr2=0x7f5b38199150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:12:05.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.286+0000 7f5b3cd26640 1 --2- 172.21.15.46:0/2087465464 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5b38100f80 0x7f5b38199150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:05.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.286+0000 7f5b3cd26640 1 -- 172.21.15.46:0/2087465464 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b381a41d0 con 0x7f5b381005c0 2024-03-31T23:12:05.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.286+0000 7f5b3cd26640 1 --2- 172.21.15.46:0/2087465464 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5b381005c0 0x7f5b38198c10 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f5b2c00e9c0 tx=0x7f5b2c00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:12:05.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.286+0000 7f5b35ffb640 1 -- 172.21.15.46:0/2087465464 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b2c00cd80 con 0x7f5b381005c0 2024-03-31T23:12:05.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.286+0000 7f5b3ef8a640 1 -- 172.21.15.46:0/2087465464 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5b381a4400 con 0x7f5b381005c0 2024-03-31T23:12:05.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.286+0000 7f5b35ffb640 1 -- 172.21.15.46:0/2087465464 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5b2c0035f0 con 0x7f5b381005c0 2024-03-31T23:12:05.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.286+0000 7f5b35ffb640 1 -- 172.21.15.46:0/2087465464 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b2c010690 con 0x7f5b381005c0 2024-03-31T23:12:05.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.286+0000 7f5b3ef8a640 1 -- 172.21.15.46:0/2087465464 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5b381a4920 con 0x7f5b381005c0 2024-03-31T23:12:05.294 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.288+0000 7f5b35ffb640 1 -- 172.21.15.46:0/2087465464 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 24) ==== 137553+0+0 (secure 0 0 0) 0x7f5b2c0037a0 con 0x7f5b381005c0 2024-03-31T23:12:05.294 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.289+0000 7f5b41b6f740 1 Processor -- start 2024-03-31T23:12:05.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.289+0000 7f5b41b6f740 1 -- start start 2024-03-31T23:12:05.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.289+0000 7f5b35ffb640 1 --2- 172.21.15.46:0/2087465464 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f5b280a2730 0x7f5b280a4bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:05.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.289+0000 7f5b41b6f740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d8cdad0470 0x55d8cdad0850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:05.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.289+0000 7f5b41b6f740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d8cdad0d90 0x55d8cdad9430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:05.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.289+0000 7f5b41b6f740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d8cdad9970 0x55d8cdadbd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:05.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.289+0000 7f5b41b6f740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55d8cda341b0 con 0x55d8cdad0d90 2024-03-31T23:12:05.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.289+0000 7f5b41b6f740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55d8cd9dcc00 con 0x55d8cdad0470 2024-03-31T23:12:05.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.289+0000 7f5b41b6f740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55d8cda30250 con 0x55d8cdad9970 2024-03-31T23:12:05.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.289+0000 7f5b3d527640 1 --2- 172.21.15.46:0/2087465464 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f5b280a2730 0x7f5b280a4bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:05.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.289+0000 7f5b37fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d8cdad0d90 0x55d8cdad9430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:05.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.289+0000 7f5b37fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d8cdad0d90 0x55d8cdad9430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60630/0 (socket says 172.21.15.46:60630) 2024-03-31T23:12:05.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.289+0000 7f5b37fff640 1 -- 172.21.15.46:0/1017820643 learned_addr learned my addr 172.21.15.46:0/1017820643 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:12:05.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.289+0000 7f5b3cd26640 1 --2- 172.21.15.46:0/1017820643 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d8cdad0470 0x55d8cdad0850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:05.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.289+0000 7f5b35ffb640 1 -- 172.21.15.46:0/2087465464 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7f5b2c0cf490 con 0x7f5b381005c0 2024-03-31T23:12:05.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.289+0000 7f5b3d527640 1 --2- 172.21.15.46:0/1017820643 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d8cdad9970 0x55d8cdadbd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:05.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.290+0000 7f5b37fff640 1 -- 172.21.15.46:0/1017820643 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d8cdad9970 msgr2=0x55d8cdadbd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:12:05.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.290+0000 7f5b37fff640 1 --2- 172.21.15.46:0/1017820643 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d8cdad9970 0x55d8cdadbd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:05.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.290+0000 7f5b37fff640 1 -- 172.21.15.46:0/1017820643 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d8cdad0470 msgr2=0x55d8cdad0850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:12:05.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.290+0000 7f5b37fff640 1 --2- 172.21.15.46:0/1017820643 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d8cdad0470 0x55d8cdad0850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:05.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.290+0000 7f5b37fff640 1 -- 172.21.15.46:0/1017820643 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d8cd9fc3d0 con 0x55d8cdad0d90 2024-03-31T23:12:05.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.290+0000 7f5b3cd26640 1 --2- 172.21.15.46:0/1017820643 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d8cdad0470 0x55d8cdad0850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:12:05.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.290+0000 7f5b3d527640 1 --2- 172.21.15.46:0/2087465464 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f5b280a2730 0x7f5b280a4bf0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f5b30004af0 tx=0x7f5b30009980 comp rx=0 tx=0).ready entity=mgr.15654 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:12:05.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.290+0000 7f5b37fff640 1 --2- 172.21.15.46:0/1017820643 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d8cdad0d90 0x55d8cdad9430 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f5b24009c90 tx=0x7f5b24011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ad6bc86448cc691 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:12:05.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.290+0000 7f5b22ffd640 1 -- 172.21.15.46:0/1017820643 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b2401e070 con 0x55d8cdad0d90 2024-03-31T23:12:05.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.290+0000 7f5b22ffd640 1 -- 172.21.15.46:0/1017820643 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5b24009e40 con 0x55d8cdad0d90 2024-03-31T23:12:05.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.290+0000 7f5b22ffd640 1 -- 172.21.15.46:0/1017820643 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b240186d0 con 0x55d8cdad0d90 2024-03-31T23:12:05.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.291+0000 7f5b41b6f740 1 -- 172.21.15.46:0/1017820643 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d8cdad0d90 msgr2=0x55d8cdad9430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:12:05.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.291+0000 7f5b41b6f740 1 --2- 172.21.15.46:0/1017820643 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d8cdad0d90 0x55d8cdad9430 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f5b24009c90 tx=0x7f5b24011810 comp rx=0 tx=0).stop 2024-03-31T23:12:05.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.291+0000 7f5b41b6f740 1 -- 172.21.15.46:0/1017820643 shutdown_connections 2024-03-31T23:12:05.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.291+0000 7f5b41b6f740 1 --2- 172.21.15.46:0/1017820643 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d8cdad9970 0x55d8cdadbd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:05.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.291+0000 7f5b41b6f740 1 --2- 172.21.15.46:0/1017820643 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d8cdad0d90 0x55d8cdad9430 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:05.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.291+0000 7f5b41b6f740 1 --2- 172.21.15.46:0/1017820643 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d8cdad0470 0x55d8cdad0850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:05.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.291+0000 7f5b41b6f740 1 -- 172.21.15.46:0/1017820643 >> 172.21.15.46:0/1017820643 conn(0x55d8cdacaba0 msgr2=0x55d8cdacaf80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:12:05.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.291+0000 7f5b41b6f740 1 -- 172.21.15.46:0/1017820643 shutdown_connections 2024-03-31T23:12:05.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.291+0000 7f5b41b6f740 1 -- 172.21.15.46:0/1017820643 wait complete. 2024-03-31T23:12:05.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.291+0000 7f5b41b6f740 1 -- start start 2024-03-31T23:12:05.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.291+0000 7f5b41b6f740 1 Processor -- start 2024-03-31T23:12:05.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.292+0000 7f5b41b6f740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d8cdad0470 0x55d8cdad6a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:05.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.292+0000 7f5b41b6f740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d8cdad0d90 0x55d8cdad6f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:05.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.292+0000 7f5b41b6f740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d8cdad9970 0x55d8cdad7490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:05.298 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.292+0000 7f5b41b6f740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55d8cdadc760 con 0x55d8cdad0d90 2024-03-31T23:12:05.298 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.292+0000 7f5b41b6f740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55d8cd9d2470 con 0x55d8cdad0470 2024-03-31T23:12:05.298 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.292+0000 7f5b41b6f740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55d8cdadc8e0 con 0x55d8cdad9970 2024-03-31T23:12:05.298 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.292+0000 7f5b41b6f740 10 client.0 _ll_register_callbacks cb 0x55d8cd9efcb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T23:12:05.298 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.292+0000 7f5b37fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d8cdad0d90 0x55d8cdad6f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:05.298 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.292+0000 7f5b37fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d8cdad0d90 0x55d8cdad6f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60636/0 (socket says 172.21.15.46:60636) 2024-03-31T23:12:05.298 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.292+0000 7f5b37fff640 1 -- 172.21.15.46:0/766950073 learned_addr learned my addr 172.21.15.46:0/766950073 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:12:05.298 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.292+0000 7f5b3cd26640 1 --2- 172.21.15.46:0/766950073 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d8cdad0470 0x55d8cdad6a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:05.298 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.292+0000 7f5b3d527640 1 --2- 172.21.15.46:0/766950073 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d8cdad9970 0x55d8cdad7490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:05.298 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.292+0000 7f5b37fff640 1 -- 172.21.15.46:0/766950073 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d8cdad9970 msgr2=0x55d8cdad7490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:12:05.298 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.292+0000 7f5b37fff640 1 --2- 172.21.15.46:0/766950073 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d8cdad9970 0x55d8cdad7490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:05.298 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.292+0000 7f5b37fff640 1 -- 172.21.15.46:0/766950073 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d8cdad0470 msgr2=0x55d8cdad6a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:12:05.298 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.292+0000 7f5b37fff640 1 --2- 172.21.15.46:0/766950073 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d8cdad0470 0x55d8cdad6a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:05.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.292+0000 7f5b37fff640 1 -- 172.21.15.46:0/766950073 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d8cd7483f0 con 0x55d8cdad0d90 2024-03-31T23:12:05.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.292+0000 7f5b3cd26640 1 --2- 172.21.15.46:0/766950073 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d8cdad0470 0x55d8cdad6a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T23:12:05.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.292+0000 7f5b37fff640 1 --2- 172.21.15.46:0/766950073 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d8cdad0d90 0x55d8cdad6f50 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f5b240058c0 tx=0x7f5b24005720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=44e7398d3d4b1708 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:12:05.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.292+0000 7f5b17fff640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-31T23:12:05.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.294+0000 7f5b17fff640 1 -- 172.21.15.46:0/766950073 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b2401e040 con 0x55d8cdad0d90 2024-03-31T23:12:05.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.294+0000 7f5b17fff640 1 -- 172.21.15.46:0/766950073 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7f5b24019410 con 0x55d8cdad0d90 2024-03-31T23:12:05.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.294+0000 7f5b41b6f740 1 -- 172.21.15.46:0/766950073 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55d8cda416f0 con 0x55d8cdad0d90 2024-03-31T23:12:05.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.294+0000 7f5b17fff640 1 -- 172.21.15.46:0/766950073 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f5b24032050 con 0x55d8cdad0d90 2024-03-31T23:12:05.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.294+0000 7f5b41b6f740 10 client.15792 fetch_fsmap learned FSMap version 25 2024-03-31T23:12:05.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.294+0000 7f5b41b6f740 1 -- 172.21.15.46:0/766950073 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55d8cd8127d0 con 0x55d8cdad0d90 2024-03-31T23:12:05.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.294+0000 7f5b17fff640 1 -- 172.21.15.46:0/766950073 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x55d8cda416f0 con 0x55d8cdad0d90 2024-03-31T23:12:05.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.295+0000 7f5b17fff640 1 -- 172.21.15.46:0/766950073 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f5b24021030 con 0x55d8cdad0d90 2024-03-31T23:12:05.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.295+0000 7f5b41b6f740 10 client.15792 fetch_fsmap finished waiting for FSMap version 25 2024-03-31T23:12:05.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.295+0000 7f5b41b6f740 10 client.15792 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-31T23:12:05.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.295+0000 7f5b41b6f740 20 client.15792 populate_metadata read hostname 'smithi046' 2024-03-31T23:12:05.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.295+0000 7f5b41b6f740 1 --2- 172.21.15.46:0/766950073 >> [v2:172.21.15.162:6836/1371886876,v1:172.21.15.162:6837/1371886876] conn(0x55d8cdaedcc0 0x55d8cdaf00b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:05.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.295+0000 7f5b41b6f740 4 client.15792 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-31T23:12:05.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.295+0000 7f5b41b6f740 1 -- 172.21.15.46:0/766950073 --> [v2:172.21.15.162:6836/1371886876,v1:172.21.15.162:6837/1371886876] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d8cda3ba40 con 0x55d8cdaedcc0 2024-03-31T23:12:05.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.295+0000 7f5b3cd26640 1 --2- 172.21.15.46:0/766950073 >> [v2:172.21.15.162:6836/1371886876,v1:172.21.15.162:6837/1371886876] conn(0x55d8cdaedcc0 0x55d8cdaf00b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:05.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.295+0000 7f5b3cd26640 1 --2- 172.21.15.46:0/766950073 >> [v2:172.21.15.162:6836/1371886876,v1:172.21.15.162:6837/1371886876] conn(0x55d8cdaedcc0 0x55d8cdaf00b0 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=5502b32df479b318 server_cookie=3ec3e624e3663960 in_seq=0 out_seq=0 2024-03-31T23:12:05.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.295+0000 7f5b17fff640 10 client.15792 ms_handle_connect on v2:172.21.15.162:6836/1371886876 2024-03-31T23:12:05.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.296+0000 7f5b17fff640 1 -- 172.21.15.46:0/766950073 <== mds.2 v2:172.21.15.162:6836/1371886876 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55d8cda3ba40 con 0x55d8cdaedcc0 2024-03-31T23:12:05.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.296+0000 7f5b17fff640 10 client.15792 handle_command_reply: tid=0 2024-03-31T23:12:05.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.296+0000 7f5b41b6f740 1 client.15792 shutdown 2024-03-31T23:12:05.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.296+0000 7f5b41b6f740 1 -- 172.21.15.46:0/766950073 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d8cdad0d90 msgr2=0x55d8cdad6f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:12:05.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.296+0000 7f5b41b6f740 1 --2- 172.21.15.46:0/766950073 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d8cdad0d90 0x55d8cdad6f50 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f5b240058c0 tx=0x7f5b24005720 comp rx=0 tx=0).stop 2024-03-31T23:12:05.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.296+0000 7f5b41b6f740 1 -- 172.21.15.46:0/766950073 shutdown_connections 2024-03-31T23:12:05.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.296+0000 7f5b41b6f740 1 --2- 172.21.15.46:0/766950073 >> [v2:172.21.15.162:6836/1371886876,v1:172.21.15.162:6837/1371886876] conn(0x55d8cdaedcc0 0x55d8cdaf00b0 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:05.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.296+0000 7f5b41b6f740 1 --2- 172.21.15.46:0/766950073 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d8cdad9970 0x55d8cdad7490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:05.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.296+0000 7f5b41b6f740 1 --2- 172.21.15.46:0/766950073 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d8cdad0d90 0x55d8cdad6f50 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:05.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.296+0000 7f5b17fff640 0 client.15792 ms_handle_reset on v2:172.21.15.162:6836/1371886876 2024-03-31T23:12:05.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.296+0000 7f5b41b6f740 1 --2- 172.21.15.46:0/766950073 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d8cdad0470 0x55d8cdad6a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:05.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.296+0000 7f5b41b6f740 1 -- 172.21.15.46:0/766950073 >> 172.21.15.46:0/766950073 conn(0x55d8cdacaba0 msgr2=0x55d8cdacc8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:12:05.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.296+0000 7f5b41b6f740 1 -- 172.21.15.46:0/766950073 shutdown_connections 2024-03-31T23:12:05.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.296+0000 7f5b41b6f740 1 -- 172.21.15.46:0/766950073 wait complete. 2024-03-31T23:12:05.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.296+0000 7f5b41b6f740 20 client.15792 trim_cache size 0 max 16384 2024-03-31T23:12:05.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.310+0000 7f5b41b6f740 1 Processor -- start 2024-03-31T23:12:05.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.310+0000 7f5b41b6f740 1 -- start start 2024-03-31T23:12:05.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.310+0000 7f5b41b6f740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d8cdad0470 0x55d8cdae9990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:05.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.310+0000 7f5b41b6f740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d8cdad0d90 0x55d8cdaf58b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:05.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.310+0000 7f5b41b6f740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d8cdad9970 0x55d8cdae5fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:05.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.310+0000 7f5b41b6f740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f5b381af2e0 con 0x55d8cdad0d90 2024-03-31T23:12:05.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.310+0000 7f5b41b6f740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f5b381acbb0 con 0x55d8cdad0470 2024-03-31T23:12:05.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.310+0000 7f5b41b6f740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f5b381aaa00 con 0x55d8cdad9970 2024-03-31T23:12:05.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.310+0000 7f5b37fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d8cdad0d90 0x55d8cdaf58b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:05.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.310+0000 7f5b37fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d8cdad0d90 0x55d8cdaf58b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60648/0 (socket says 172.21.15.46:60648) 2024-03-31T23:12:05.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.310+0000 7f5b37fff640 1 -- 172.21.15.46:0/3232947749 learned_addr learned my addr 172.21.15.46:0/3232947749 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:12:05.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.310+0000 7f5b3cd26640 1 --2- 172.21.15.46:0/3232947749 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d8cdad0470 0x55d8cdae9990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:05.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.310+0000 7f5b3d527640 1 --2- 172.21.15.46:0/3232947749 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d8cdad9970 0x55d8cdae5fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:05.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.311+0000 7f5b37fff640 1 -- 172.21.15.46:0/3232947749 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d8cdad9970 msgr2=0x55d8cdae5fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:12:05.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.311+0000 7f5b37fff640 1 --2- 172.21.15.46:0/3232947749 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d8cdad9970 0x55d8cdae5fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:05.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.311+0000 7f5b37fff640 1 -- 172.21.15.46:0/3232947749 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d8cdad0470 msgr2=0x55d8cdae9990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:12:05.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.311+0000 7f5b37fff640 1 --2- 172.21.15.46:0/3232947749 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d8cdad0470 0x55d8cdae9990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:05.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.311+0000 7f5b37fff640 1 -- 172.21.15.46:0/3232947749 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d8cdad0bc0 con 0x55d8cdad0d90 2024-03-31T23:12:05.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.311+0000 7f5b37fff640 1 --2- 172.21.15.46:0/3232947749 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d8cdad0d90 0x55d8cdaf58b0 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f5b24000c00 tx=0x7f5b24009c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c81902e523b962ef server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:12:05.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.311+0000 7f5b227fc640 1 -- 172.21.15.46:0/3232947749 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b2401e040 con 0x55d8cdad0d90 2024-03-31T23:12:05.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.311+0000 7f5b227fc640 1 -- 172.21.15.46:0/3232947749 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5b24016070 con 0x55d8cdad0d90 2024-03-31T23:12:05.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.311+0000 7f5b227fc640 1 -- 172.21.15.46:0/3232947749 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b24018e80 con 0x55d8cdad0d90 2024-03-31T23:12:05.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.312+0000 7f5b41b6f740 1 -- 172.21.15.46:0/3232947749 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d8cdad0d90 msgr2=0x55d8cdaf58b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:12:05.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.312+0000 7f5b41b6f740 1 --2- 172.21.15.46:0/3232947749 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d8cdad0d90 0x55d8cdaf58b0 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f5b24000c00 tx=0x7f5b24009c40 comp rx=0 tx=0).stop 2024-03-31T23:12:05.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.312+0000 7f5b41b6f740 1 -- 172.21.15.46:0/3232947749 shutdown_connections 2024-03-31T23:12:05.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.312+0000 7f5b41b6f740 1 --2- 172.21.15.46:0/3232947749 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d8cdad9970 0x55d8cdae5fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:05.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.312+0000 7f5b41b6f740 1 --2- 172.21.15.46:0/3232947749 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d8cdad0d90 0x55d8cdaf58b0 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:05.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.312+0000 7f5b41b6f740 1 --2- 172.21.15.46:0/3232947749 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d8cdad0470 0x55d8cdae9990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:05.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.312+0000 7f5b41b6f740 1 -- 172.21.15.46:0/3232947749 >> 172.21.15.46:0/3232947749 conn(0x55d8cdacaba0 msgr2=0x55d8cdad69d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:12:05.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.312+0000 7f5b41b6f740 1 -- 172.21.15.46:0/3232947749 shutdown_connections 2024-03-31T23:12:05.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.312+0000 7f5b41b6f740 1 -- 172.21.15.46:0/3232947749 wait complete. 2024-03-31T23:12:05.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.312+0000 7f5b41b6f740 1 -- start start 2024-03-31T23:12:05.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.312+0000 7f5b41b6f740 1 Processor -- start 2024-03-31T23:12:05.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.312+0000 7f5b41b6f740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d8cdad0470 0x55d8cdae1ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:05.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.312+0000 7f5b41b6f740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d8cdad0d90 0x55d8cdae2010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:05.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.312+0000 7f5b41b6f740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d8cdad9970 0x55d8cdae2550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:05.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.313+0000 7f5b41b6f740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f5b381ab070 con 0x55d8cdad9970 2024-03-31T23:12:05.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.313+0000 7f5b41b6f740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f5b381ac4e0 con 0x55d8cdad0d90 2024-03-31T23:12:05.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.313+0000 7f5b41b6f740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f5b381ab740 con 0x55d8cdad0470 2024-03-31T23:12:05.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.313+0000 7f5b41b6f740 10 client.0 _ll_register_callbacks cb 0x7f5b24005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T23:12:05.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.313+0000 7f5b3d527640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d8cdad9970 0x55d8cdae2550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:05.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.313+0000 7f5b3cd26640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d8cdad0470 0x55d8cdae1ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:05.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.313+0000 7f5b37fff640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d8cdad0d90 0x55d8cdae2010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:05.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.313+0000 7f5b3d527640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d8cdad9970 0x55d8cdae2550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60664/0 (socket says 172.21.15.46:60664) 2024-03-31T23:12:05.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.313+0000 7f5b3d527640 1 -- 172.21.15.46:0/3222151302 learned_addr learned my addr 172.21.15.46:0/3222151302 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:12:05.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.313+0000 7f5b37fff640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d8cdad0d90 0x55d8cdae2010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.46:44226/0 (socket says 172.21.15.46:44226) 2024-03-31T23:12:05.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.313+0000 7f5b3cd26640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d8cdad0470 0x55d8cdae1ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.46:33396/0 (socket says 172.21.15.46:33396) 2024-03-31T23:12:05.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.313+0000 7f5b3d527640 1 -- 172.21.15.46:0/3222151302 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d8cdad0470 msgr2=0x55d8cdae1ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:12:05.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.313+0000 7f5b3d527640 1 --2- 172.21.15.46:0/3222151302 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d8cdad0470 0x55d8cdae1ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:05.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.313+0000 7f5b3d527640 1 -- 172.21.15.46:0/3222151302 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d8cdad0d90 msgr2=0x55d8cdae2010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:12:05.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.313+0000 7f5b3d527640 1 --2- 172.21.15.46:0/3222151302 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d8cdad0d90 0x55d8cdae2010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:05.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.313+0000 7f5b3d527640 1 -- 172.21.15.46:0/3222151302 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d8cdb530e0 con 0x55d8cdad9970 2024-03-31T23:12:05.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.313+0000 7f5b3d527640 1 --2- 172.21.15.46:0/3222151302 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d8cdad9970 0x55d8cdae2550 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f5b30009ca0 tx=0x7f5b3002c0f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=824a93a6531ef12c server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:12:05.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.313+0000 7f5b17fff640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-31T23:12:05.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.313+0000 7f5b17fff640 1 -- 172.21.15.46:0/3222151302 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b30053060 con 0x55d8cdad9970 2024-03-31T23:12:05.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.313+0000 7f5b17fff640 1 -- 172.21.15.46:0/3222151302 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7f5b30004790 con 0x55d8cdad9970 2024-03-31T23:12:05.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.313+0000 7f5b41b6f740 1 -- 172.21.15.46:0/3222151302 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5b24016070 con 0x55d8cdad9970 2024-03-31T23:12:05.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.313+0000 7f5b17fff640 1 -- 172.21.15.46:0/3222151302 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f5b3005a050 con 0x55d8cdad9970 2024-03-31T23:12:05.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.313+0000 7f5b41b6f740 10 client.15798 fetch_fsmap learned FSMap version 25 2024-03-31T23:12:05.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.313+0000 7f5b41b6f740 1 -- 172.21.15.46:0/3222151302 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55d8cdb543a0 con 0x55d8cdad9970 2024-03-31T23:12:05.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.314+0000 7f5b17fff640 1 -- 172.21.15.46:0/3222151302 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5b24016070 con 0x55d8cdad9970 2024-03-31T23:12:05.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.314+0000 7f5b17fff640 1 -- 172.21.15.46:0/3222151302 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f5b3003f040 con 0x55d8cdad9970 2024-03-31T23:12:05.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.314+0000 7f5b41b6f740 10 client.15798 fetch_fsmap finished waiting for FSMap version 25 2024-03-31T23:12:05.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.314+0000 7f5b41b6f740 10 client.15798 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-31T23:12:05.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.314+0000 7f5b41b6f740 20 client.15798 populate_metadata read hostname 'smithi046' 2024-03-31T23:12:05.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.314+0000 7f5b41b6f740 1 --2- 172.21.15.46:0/3222151302 >> [v2:172.21.15.162:6836/1371886876,v1:172.21.15.162:6837/1371886876] conn(0x55d8cdb54b20 0x55d8cdb56f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:05.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.314+0000 7f5b41b6f740 4 client.15798 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-31T23:12:05.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.314+0000 7f5b41b6f740 1 -- 172.21.15.46:0/3222151302 --> [v2:172.21.15.162:6836/1371886876,v1:172.21.15.162:6837/1371886876] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55d8cd9b96a0 con 0x55d8cdb54b20 2024-03-31T23:12:05.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.314+0000 7f5b3cd26640 1 --2- 172.21.15.46:0/3222151302 >> [v2:172.21.15.162:6836/1371886876,v1:172.21.15.162:6837/1371886876] conn(0x55d8cdb54b20 0x55d8cdb56f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:05.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.314+0000 7f5b3cd26640 1 --2- 172.21.15.46:0/3222151302 >> [v2:172.21.15.162:6836/1371886876,v1:172.21.15.162:6837/1371886876] conn(0x55d8cdb54b20 0x55d8cdb56f40 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=ab8eee73bb5906c server_cookie=7469ba73c0fbc63 in_seq=0 out_seq=0 2024-03-31T23:12:05.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.314+0000 7f5b17fff640 10 client.15798 ms_handle_connect on v2:172.21.15.162:6836/1371886876 2024-03-31T23:12:05.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.314+0000 7f5b17fff640 1 -- 172.21.15.46:0/3222151302 <== mds.2 v2:172.21.15.162:6836/1371886876 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55d8cd9b96a0 con 0x55d8cdb54b20 2024-03-31T23:12:05.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.315+0000 7f5b17fff640 10 client.15798 handle_command_reply: tid=0 2024-03-31T23:12:05.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.315+0000 7f5b41b6f740 1 client.15798 shutdown 2024-03-31T23:12:05.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.315+0000 7f5b41b6f740 1 -- 172.21.15.46:0/3222151302 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d8cdad9970 msgr2=0x55d8cdae2550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:12:05.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.315+0000 7f5b41b6f740 1 --2- 172.21.15.46:0/3222151302 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d8cdad9970 0x55d8cdae2550 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f5b30009ca0 tx=0x7f5b3002c0f0 comp rx=0 tx=0).stop 2024-03-31T23:12:05.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.315+0000 7f5b41b6f740 1 -- 172.21.15.46:0/3222151302 shutdown_connections 2024-03-31T23:12:05.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.315+0000 7f5b41b6f740 1 --2- 172.21.15.46:0/3222151302 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d8cdad9970 0x55d8cdae2550 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:05.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.315+0000 7f5b41b6f740 1 --2- 172.21.15.46:0/3222151302 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d8cdad0d90 0x55d8cdae2010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:05.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.315+0000 7f5b41b6f740 1 --2- 172.21.15.46:0/3222151302 >> [v2:172.21.15.162:6836/1371886876,v1:172.21.15.162:6837/1371886876] conn(0x55d8cdb54b20 0x55d8cdb56f40 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:05.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.315+0000 7f5b41b6f740 1 --2- 172.21.15.46:0/3222151302 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d8cdad0470 0x55d8cdae1ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:05.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.315+0000 7f5b41b6f740 1 -- 172.21.15.46:0/3222151302 >> 172.21.15.46:0/3222151302 conn(0x55d8cdacaba0 msgr2=0x55d8cdad71a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:12:05.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.315+0000 7f5b17fff640 0 client.15798 ms_handle_reset on v2:172.21.15.162:6836/1371886876 2024-03-31T23:12:05.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.315+0000 7f5b41b6f740 1 -- 172.21.15.46:0/3222151302 shutdown_connections 2024-03-31T23:12:05.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.315+0000 7f5b41b6f740 1 -- 172.21.15.46:0/3222151302 wait complete. 2024-03-31T23:12:05.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.315+0000 7f5b41b6f740 20 client.15798 trim_cache size 0 max 16384 2024-03-31T23:12:05.323 INFO:teuthology.orchestra.run.smithi046.stdout:[] 2024-03-31T23:12:05.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.315+0000 7f5b3ef8a640 1 -- 172.21.15.46:0/2087465464 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f5b280a2730 msgr2=0x7f5b280a4bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:12:05.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.315+0000 7f5b3ef8a640 1 --2- 172.21.15.46:0/2087465464 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f5b280a2730 0x7f5b280a4bf0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f5b30004af0 tx=0x7f5b30009980 comp rx=0 tx=0).stop 2024-03-31T23:12:05.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.315+0000 7f5b3ef8a640 1 -- 172.21.15.46:0/2087465464 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5b381005c0 msgr2=0x7f5b38198c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:12:05.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.315+0000 7f5b3ef8a640 1 --2- 172.21.15.46:0/2087465464 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5b381005c0 0x7f5b38198c10 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f5b2c00e9c0 tx=0x7f5b2c00ee80 comp rx=0 tx=0).stop 2024-03-31T23:12:05.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.315+0000 7f5b3ef8a640 1 -- 172.21.15.46:0/2087465464 shutdown_connections 2024-03-31T23:12:05.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.315+0000 7f5b3ef8a640 1 --2- 172.21.15.46:0/2087465464 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f5b280a2730 0x7f5b280a4bf0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:05.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.315+0000 7f5b3ef8a640 1 --2- 172.21.15.46:0/2087465464 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5b38106660 0x7f5b381a3c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:05.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.315+0000 7f5b3ef8a640 1 --2- 172.21.15.46:0/2087465464 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5b38100f80 0x7f5b38199150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:05.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.315+0000 7f5b3ef8a640 1 --2- 172.21.15.46:0/2087465464 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5b381005c0 0x7f5b38198c10 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:05.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.315+0000 7f5b3ef8a640 1 -- 172.21.15.46:0/2087465464 >> 172.21.15.46:0/2087465464 conn(0x7f5b38077750 msgr2=0x7f5b380ff150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:12:05.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.316+0000 7f5b3ef8a640 1 -- 172.21.15.46:0/2087465464 shutdown_connections 2024-03-31T23:12:05.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.316+0000 7f5b3ef8a640 1 -- 172.21.15.46:0/2087465464 wait complete. 2024-03-31T23:12:05.327 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-31T23:12:05.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.432+0000 7f1d34f7a640 1 -- 172.21.15.46:0/3310688592 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1d301020a0 msgr2=0x7f1d3010e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:12:05.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.432+0000 7f1d34f7a640 1 --2- 172.21.15.46:0/3310688592 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1d301020a0 0x7f1d3010e5b0 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f1d20009cd0 tx=0x7f1d2002c0f0 comp rx=0 tx=0).stop 2024-03-31T23:12:05.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.432+0000 7f1d34f7a640 1 -- 172.21.15.46:0/3310688592 shutdown_connections 2024-03-31T23:12:05.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.432+0000 7f1d34f7a640 1 --2- 172.21.15.46:0/3310688592 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1d301020a0 0x7f1d3010e5b0 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:05.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.432+0000 7f1d34f7a640 1 --2- 172.21.15.46:0/3310688592 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1d301016e0 0x7f1d30101b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:05.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.432+0000 7f1d34f7a640 1 --2- 172.21.15.46:0/3310688592 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1d301076f0 0x7f1d30107ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:05.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.432+0000 7f1d34f7a640 1 -- 172.21.15.46:0/3310688592 >> 172.21.15.46:0/3310688592 conn(0x7f1d30078b50 msgr2=0x7f1d30078f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:12:05.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.432+0000 7f1d34f7a640 1 -- 172.21.15.46:0/3310688592 shutdown_connections 2024-03-31T23:12:05.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.432+0000 7f1d34f7a640 1 -- 172.21.15.46:0/3310688592 wait complete. 2024-03-31T23:12:05.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.433+0000 7f1d34f7a640 1 Processor -- start 2024-03-31T23:12:05.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.433+0000 7f1d34f7a640 1 -- start start 2024-03-31T23:12:05.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.433+0000 7f1d34f7a640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1d301016e0 0x7f1d30197ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:05.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.433+0000 7f1d34f7a640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1d301020a0 0x7f1d30198410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:05.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.433+0000 7f1d34f7a640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1d301076f0 0x7f1d3019c790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:05.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.433+0000 7f1d34f7a640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f1d30111970 con 0x7f1d301020a0 2024-03-31T23:12:05.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.433+0000 7f1d34f7a640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f1d301117f0 con 0x7f1d301076f0 2024-03-31T23:12:05.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.433+0000 7f1d34f7a640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f1d30111af0 con 0x7f1d301016e0 2024-03-31T23:12:05.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.433+0000 7f1d2dd9b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1d301020a0 0x7f1d30198410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:05.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.433+0000 7f1d2dd9b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1d301020a0 0x7f1d30198410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60682/0 (socket says 172.21.15.46:60682) 2024-03-31T23:12:05.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.433+0000 7f1d2dd9b640 1 -- 172.21.15.46:0/2101881708 learned_addr learned my addr 172.21.15.46:0/2101881708 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:12:05.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.433+0000 7f1d2e59c640 1 --2- 172.21.15.46:0/2101881708 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1d301016e0 0x7f1d30197ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:05.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.433+0000 7f1d2ed9d640 1 --2- 172.21.15.46:0/2101881708 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1d301076f0 0x7f1d3019c790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:05.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.433+0000 7f1d2dd9b640 1 -- 172.21.15.46:0/2101881708 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1d301016e0 msgr2=0x7f1d30197ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:12:05.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.433+0000 7f1d2dd9b640 1 --2- 172.21.15.46:0/2101881708 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1d301016e0 0x7f1d30197ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:05.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.433+0000 7f1d2dd9b640 1 -- 172.21.15.46:0/2101881708 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1d301076f0 msgr2=0x7f1d3019c790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:12:05.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.433+0000 7f1d2dd9b640 1 --2- 172.21.15.46:0/2101881708 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1d301076f0 0x7f1d3019c790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:05.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.433+0000 7f1d2dd9b640 1 -- 172.21.15.46:0/2101881708 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1d3019cec0 con 0x7f1d301020a0 2024-03-31T23:12:05.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.433+0000 7f1d2e59c640 1 --2- 172.21.15.46:0/2101881708 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1d301016e0 0x7f1d30197ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T23:12:05.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.433+0000 7f1d2dd9b640 1 --2- 172.21.15.46:0/2101881708 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1d301020a0 0x7f1d30198410 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f1d1800e9f0 tx=0x7f1d1800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:12:05.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.433+0000 7f1d277fe640 1 -- 172.21.15.46:0/2101881708 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d1800cd80 con 0x7f1d301020a0 2024-03-31T23:12:05.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.434+0000 7f1d34f7a640 1 -- 172.21.15.46:0/2101881708 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1d3019d1b0 con 0x7f1d301020a0 2024-03-31T23:12:05.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.434+0000 7f1d277fe640 1 -- 172.21.15.46:0/2101881708 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1d180108e0 con 0x7f1d301020a0 2024-03-31T23:12:05.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.434+0000 7f1d277fe640 1 -- 172.21.15.46:0/2101881708 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d18018920 con 0x7f1d301020a0 2024-03-31T23:12:05.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.434+0000 7f1d34f7a640 1 -- 172.21.15.46:0/2101881708 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1d301a4a40 con 0x7f1d301020a0 2024-03-31T23:12:05.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.436+0000 7f1d277fe640 1 -- 172.21.15.46:0/2101881708 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 24) ==== 137553+0+0 (secure 0 0 0) 0x7f1d18010a90 con 0x7f1d301020a0 2024-03-31T23:12:05.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.436+0000 7f1d37c39740 1 Processor -- start 2024-03-31T23:12:05.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.436+0000 7f1d277fe640 1 --2- 172.21.15.46:0/2101881708 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f1d1c0a2730 0x7f1d1c0a4bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:05.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.436+0000 7f1d37c39740 1 -- start start 2024-03-31T23:12:05.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.436+0000 7f1d2ed9d640 1 --2- 172.21.15.46:0/2101881708 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f1d1c0a2730 0x7f1d1c0a4bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:05.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.436+0000 7f1d277fe640 1 -- 172.21.15.46:0/2101881708 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7f1d180cf5f0 con 0x7f1d301020a0 2024-03-31T23:12:05.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.437+0000 7f1d37c39740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56053e27d470 0x56053e27d850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:05.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.437+0000 7f1d37c39740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56053e27dd90 0x56053e286430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:05.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.437+0000 7f1d37c39740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56053e286970 0x56053e288d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:05.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.437+0000 7f1d37c39740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x56053e1e11b0 con 0x56053e286970 2024-03-31T23:12:05.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.437+0000 7f1d37c39740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x56053e189c00 con 0x56053e27d470 2024-03-31T23:12:05.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.437+0000 7f1d37c39740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x56053e1dd250 con 0x56053e27dd90 2024-03-31T23:12:05.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.437+0000 7f1d2ed9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56053e286970 0x56053e288d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:05.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.437+0000 7f1d2dd9b640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56053e27dd90 0x56053e286430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:05.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.437+0000 7f1d2ed9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56053e286970 0x56053e288d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60684/0 (socket says 172.21.15.46:60684) 2024-03-31T23:12:05.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.437+0000 7f1d2ed9d640 1 -- 172.21.15.46:0/1120955949 learned_addr learned my addr 172.21.15.46:0/1120955949 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:12:05.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.437+0000 7f1d2e59c640 1 --2- 172.21.15.46:0/1120955949 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56053e27d470 0x56053e27d850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:05.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.437+0000 7f1d2dd9b640 1 -- 172.21.15.46:0/1120955949 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56053e27d470 msgr2=0x56053e27d850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:12:05.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.437+0000 7f1d2dd9b640 1 --2- 172.21.15.46:0/1120955949 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56053e27d470 0x56053e27d850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:05.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.437+0000 7f1d2dd9b640 1 -- 172.21.15.46:0/1120955949 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56053e286970 msgr2=0x56053e288d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:12:05.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.437+0000 7f1d2dd9b640 1 --2- 172.21.15.46:0/1120955949 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56053e286970 0x56053e288d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:05.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.437+0000 7f1d2dd9b640 1 -- 172.21.15.46:0/1120955949 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56053e1a93d0 con 0x56053e27dd90 2024-03-31T23:12:05.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.437+0000 7f1d2e59c640 1 --2- 172.21.15.46:0/1120955949 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56053e27d470 0x56053e27d850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T23:12:05.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.437+0000 7f1d2ed9d640 1 --2- 172.21.15.46:0/2101881708 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f1d1c0a2730 0x7f1d1c0a4bf0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f1d2002c700 tx=0x7f1d20032d70 comp rx=0 tx=0).ready entity=mgr.15654 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:12:05.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.437+0000 7f1d2dd9b640 1 --2- 172.21.15.46:0/1120955949 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56053e27dd90 0x56053e286430 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f1d180cc810 tx=0x7f1d1809b670 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c8e77c37077d6b77 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:12:05.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.437+0000 7f1d24ff9640 1 -- 172.21.15.46:0/1120955949 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d1809f8c0 con 0x56053e27dd90 2024-03-31T23:12:05.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.437+0000 7f1d24ff9640 1 -- 172.21.15.46:0/1120955949 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1d180cf050 con 0x56053e27dd90 2024-03-31T23:12:05.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.437+0000 7f1d24ff9640 1 -- 172.21.15.46:0/1120955949 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d180ccaf0 con 0x56053e27dd90 2024-03-31T23:12:05.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.438+0000 7f1d37c39740 1 -- 172.21.15.46:0/1120955949 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56053e27dd90 msgr2=0x56053e286430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:12:05.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.438+0000 7f1d37c39740 1 --2- 172.21.15.46:0/1120955949 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56053e27dd90 0x56053e286430 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f1d180cc810 tx=0x7f1d1809b670 comp rx=0 tx=0).stop 2024-03-31T23:12:05.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.438+0000 7f1d37c39740 1 -- 172.21.15.46:0/1120955949 shutdown_connections 2024-03-31T23:12:05.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.438+0000 7f1d37c39740 1 --2- 172.21.15.46:0/1120955949 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56053e286970 0x56053e288d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:05.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.438+0000 7f1d37c39740 1 --2- 172.21.15.46:0/1120955949 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56053e27dd90 0x56053e286430 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:05.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.438+0000 7f1d37c39740 1 --2- 172.21.15.46:0/1120955949 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56053e27d470 0x56053e27d850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:05.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.438+0000 7f1d37c39740 1 -- 172.21.15.46:0/1120955949 >> 172.21.15.46:0/1120955949 conn(0x56053e277ba0 msgr2=0x56053e277f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:12:05.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.438+0000 7f1d37c39740 1 -- 172.21.15.46:0/1120955949 shutdown_connections 2024-03-31T23:12:05.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.438+0000 7f1d37c39740 1 -- 172.21.15.46:0/1120955949 wait complete. 2024-03-31T23:12:05.447 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.438+0000 7f1d37c39740 1 -- start start 2024-03-31T23:12:05.447 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.439+0000 7f1d37c39740 1 Processor -- start 2024-03-31T23:12:05.447 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.439+0000 7f1d37c39740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56053e27d470 0x56053e283a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:05.447 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.439+0000 7f1d37c39740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56053e27dd90 0x56053e283f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:05.447 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.439+0000 7f1d37c39740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56053e286970 0x56053e284480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:05.447 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.439+0000 7f1d37c39740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x56053e289700 con 0x56053e27d470 2024-03-31T23:12:05.447 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.439+0000 7f1d37c39740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x56053e17f470 con 0x56053e286970 2024-03-31T23:12:05.447 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.439+0000 7f1d37c39740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x56053e289880 con 0x56053e27dd90 2024-03-31T23:12:05.447 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.439+0000 7f1d37c39740 10 client.0 _ll_register_callbacks cb 0x56053e19ccb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T23:12:05.447 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.439+0000 7f1d2e59c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56053e27d470 0x56053e283a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:05.447 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.439+0000 7f1d2e59c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56053e27d470 0x56053e283a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60692/0 (socket says 172.21.15.46:60692) 2024-03-31T23:12:05.447 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.439+0000 7f1d2e59c640 1 -- 172.21.15.46:0/1072396619 learned_addr learned my addr 172.21.15.46:0/1072396619 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:12:05.447 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.439+0000 7f1d2dd9b640 1 --2- 172.21.15.46:0/1072396619 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56053e27dd90 0x56053e283f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:05.448 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.439+0000 7f1d2ed9d640 1 --2- 172.21.15.46:0/1072396619 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56053e286970 0x56053e284480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:05.448 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.439+0000 7f1d2e59c640 1 -- 172.21.15.46:0/1072396619 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56053e27dd90 msgr2=0x56053e283f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:12:05.448 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.439+0000 7f1d2e59c640 1 --2- 172.21.15.46:0/1072396619 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56053e27dd90 0x56053e283f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:05.448 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.439+0000 7f1d2e59c640 1 -- 172.21.15.46:0/1072396619 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56053e286970 msgr2=0x56053e284480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:12:05.448 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.439+0000 7f1d2e59c640 1 --2- 172.21.15.46:0/1072396619 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56053e286970 0x56053e284480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:05.448 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.439+0000 7f1d2e59c640 1 -- 172.21.15.46:0/1072396619 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56053def53f0 con 0x56053e27d470 2024-03-31T23:12:05.448 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.439+0000 7f1d2e59c640 1 --2- 172.21.15.46:0/1072396619 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56053e27d470 0x56053e283a00 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f1d28005f60 tx=0x7f1d28009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a3c8e0afa30d766c server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:12:05.448 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.439+0000 7f1d15ffb640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-31T23:12:05.448 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.440+0000 7f1d15ffb640 1 -- 172.21.15.46:0/1072396619 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d28011dd0 con 0x56053e27d470 2024-03-31T23:12:05.448 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.440+0000 7f1d15ffb640 1 -- 172.21.15.46:0/1072396619 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7f1d28004020 con 0x56053e27d470 2024-03-31T23:12:05.448 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.440+0000 7f1d37c39740 1 -- 172.21.15.46:0/1072396619 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1d180cf050 con 0x56053e27d470 2024-03-31T23:12:05.448 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.440+0000 7f1d15ffb640 1 -- 172.21.15.46:0/1072396619 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f1d28025050 con 0x56053e27d470 2024-03-31T23:12:05.448 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.440+0000 7f1d37c39740 10 client.15810 fetch_fsmap learned FSMap version 25 2024-03-31T23:12:05.448 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.440+0000 7f1d37c39740 1 -- 172.21.15.46:0/1072396619 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x56053dfbf7d0 con 0x56053e27d470 2024-03-31T23:12:05.449 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.440+0000 7f1d15ffb640 1 -- 172.21.15.46:0/1072396619 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1d180cf050 con 0x56053e27d470 2024-03-31T23:12:05.449 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.440+0000 7f1d15ffb640 1 -- 172.21.15.46:0/1072396619 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f1d2801a030 con 0x56053e27d470 2024-03-31T23:12:05.449 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.440+0000 7f1d37c39740 10 client.15810 fetch_fsmap finished waiting for FSMap version 25 2024-03-31T23:12:05.449 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.440+0000 7f1d37c39740 10 client.15810 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-03-31T23:12:05.449 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.440+0000 7f1d37c39740 20 client.15810 populate_metadata read hostname 'smithi046' 2024-03-31T23:12:05.449 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.440+0000 7f1d37c39740 1 --2- 172.21.15.46:0/1072396619 >> [v2:172.21.15.142:6836/2559258399,v1:172.21.15.142:6837/2559258399] conn(0x56053e296a80 0x56053e298ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:05.449 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.440+0000 7f1d37c39740 4 client.15810 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-31T23:12:05.449 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.440+0000 7f1d37c39740 1 -- 172.21.15.46:0/1072396619 --> [v2:172.21.15.142:6836/2559258399,v1:172.21.15.142:6837/2559258399] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56053e1ee6f0 con 0x56053e296a80 2024-03-31T23:12:05.449 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.440+0000 7f1d2dd9b640 1 --2- 172.21.15.46:0/1072396619 >> [v2:172.21.15.142:6836/2559258399,v1:172.21.15.142:6837/2559258399] conn(0x56053e296a80 0x56053e298ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:05.449 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.441+0000 7f1d2dd9b640 1 --2- 172.21.15.46:0/1072396619 >> [v2:172.21.15.142:6836/2559258399,v1:172.21.15.142:6837/2559258399] conn(0x56053e296a80 0x56053e298ee0 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=ba098c45a98d2a41 server_cookie=af0ff8c7c709af in_seq=0 out_seq=0 2024-03-31T23:12:05.449 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.441+0000 7f1d15ffb640 10 client.15810 ms_handle_connect on v2:172.21.15.142:6836/2559258399 2024-03-31T23:12:05.449 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.441+0000 7f1d15ffb640 1 -- 172.21.15.46:0/1072396619 <== mds.1 v2:172.21.15.142:6836/2559258399 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x56053e1ee6f0 con 0x56053e296a80 2024-03-31T23:12:05.449 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.441+0000 7f1d15ffb640 10 client.15810 handle_command_reply: tid=0 2024-03-31T23:12:05.449 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.441+0000 7f1d37c39740 1 client.15810 shutdown 2024-03-31T23:12:05.450 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.441+0000 7f1d37c39740 1 -- 172.21.15.46:0/1072396619 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56053e27d470 msgr2=0x56053e283a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:12:05.450 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.441+0000 7f1d37c39740 1 --2- 172.21.15.46:0/1072396619 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56053e27d470 0x56053e283a00 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f1d28005f60 tx=0x7f1d28009f90 comp rx=0 tx=0).stop 2024-03-31T23:12:05.450 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.441+0000 7f1d37c39740 1 -- 172.21.15.46:0/1072396619 shutdown_connections 2024-03-31T23:12:05.450 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.441+0000 7f1d37c39740 1 --2- 172.21.15.46:0/1072396619 >> [v2:172.21.15.142:6836/2559258399,v1:172.21.15.142:6837/2559258399] conn(0x56053e296a80 0x56053e298ee0 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:05.450 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.442+0000 7f1d37c39740 1 --2- 172.21.15.46:0/1072396619 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56053e286970 0x56053e284480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:05.450 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.442+0000 7f1d37c39740 1 --2- 172.21.15.46:0/1072396619 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56053e27dd90 0x56053e283f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:05.450 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.442+0000 7f1d37c39740 1 --2- 172.21.15.46:0/1072396619 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56053e27d470 0x56053e283a00 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:05.450 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.442+0000 7f1d37c39740 1 -- 172.21.15.46:0/1072396619 >> 172.21.15.46:0/1072396619 conn(0x56053e277ba0 msgr2=0x56053e2798f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:12:05.450 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.442+0000 7f1d15ffb640 0 client.15810 ms_handle_reset on v2:172.21.15.142:6836/2559258399 2024-03-31T23:12:05.450 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.442+0000 7f1d37c39740 1 -- 172.21.15.46:0/1072396619 shutdown_connections 2024-03-31T23:12:05.450 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.442+0000 7f1d37c39740 1 -- 172.21.15.46:0/1072396619 wait complete. 2024-03-31T23:12:05.450 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.442+0000 7f1d37c39740 20 client.15810 trim_cache size 0 max 16384 2024-03-31T23:12:05.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.455+0000 7f1d37c39740 1 Processor -- start 2024-03-31T23:12:05.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.455+0000 7f1d37c39740 1 -- start start 2024-03-31T23:12:05.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.455+0000 7f1d37c39740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56053e27d470 0x56053e286130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:05.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.455+0000 7f1d37c39740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56053e27dd90 0x56053e2924a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:05.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.455+0000 7f1d37c39740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56053e286970 0x56053e29e7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:05.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.455+0000 7f1d37c39740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f1d301b1520 con 0x56053e286970 2024-03-31T23:12:05.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.455+0000 7f1d37c39740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f1d301aedf0 con 0x56053e27dd90 2024-03-31T23:12:05.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.455+0000 7f1d37c39740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f1d301acc40 con 0x56053e27d470 2024-03-31T23:12:05.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.455+0000 7f1d2ed9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56053e286970 0x56053e29e7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:05.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.455+0000 7f1d2ed9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56053e286970 0x56053e29e7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60698/0 (socket says 172.21.15.46:60698) 2024-03-31T23:12:05.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.455+0000 7f1d2ed9d640 1 -- 172.21.15.46:0/2286541747 learned_addr learned my addr 172.21.15.46:0/2286541747 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:12:05.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.455+0000 7f1d2e59c640 1 --2- 172.21.15.46:0/2286541747 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56053e27d470 0x56053e286130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:05.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.455+0000 7f1d2dd9b640 1 --2- 172.21.15.46:0/2286541747 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56053e27dd90 0x56053e2924a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:05.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.455+0000 7f1d2ed9d640 1 -- 172.21.15.46:0/2286541747 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56053e27d470 msgr2=0x56053e286130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:12:05.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.455+0000 7f1d2ed9d640 1 --2- 172.21.15.46:0/2286541747 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56053e27d470 0x56053e286130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:05.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.455+0000 7f1d2ed9d640 1 -- 172.21.15.46:0/2286541747 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56053e27dd90 msgr2=0x56053e2924a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:12:05.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.455+0000 7f1d2ed9d640 1 --2- 172.21.15.46:0/2286541747 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56053e27dd90 0x56053e2924a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:05.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.455+0000 7f1d2ed9d640 1 -- 172.21.15.46:0/2286541747 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56053e27dbc0 con 0x56053e286970 2024-03-31T23:12:05.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.456+0000 7f1d2ed9d640 1 --2- 172.21.15.46:0/2286541747 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56053e286970 0x56053e29e7b0 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f1d2002c630 tx=0x7f1d20044000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f479f3671324b3 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:12:05.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.456+0000 7f1d17fff640 1 -- 172.21.15.46:0/2286541747 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d20039040 con 0x56053e286970 2024-03-31T23:12:05.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.456+0000 7f1d17fff640 1 -- 172.21.15.46:0/2286541747 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1d20003cf0 con 0x56053e286970 2024-03-31T23:12:05.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.456+0000 7f1d17fff640 1 -- 172.21.15.46:0/2286541747 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d2004d870 con 0x56053e286970 2024-03-31T23:12:05.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.456+0000 7f1d37c39740 1 -- 172.21.15.46:0/2286541747 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56053e286970 msgr2=0x56053e29e7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:12:05.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.456+0000 7f1d37c39740 1 --2- 172.21.15.46:0/2286541747 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56053e286970 0x56053e29e7b0 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f1d2002c630 tx=0x7f1d20044000 comp rx=0 tx=0).stop 2024-03-31T23:12:05.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.456+0000 7f1d37c39740 1 -- 172.21.15.46:0/2286541747 shutdown_connections 2024-03-31T23:12:05.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.456+0000 7f1d37c39740 1 --2- 172.21.15.46:0/2286541747 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56053e286970 0x56053e29e7b0 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:05.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.456+0000 7f1d37c39740 1 --2- 172.21.15.46:0/2286541747 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56053e27dd90 0x56053e2924a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:05.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.456+0000 7f1d37c39740 1 --2- 172.21.15.46:0/2286541747 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56053e27d470 0x56053e286130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:05.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.456+0000 7f1d37c39740 1 -- 172.21.15.46:0/2286541747 >> 172.21.15.46:0/2286541747 conn(0x56053e277ba0 msgr2=0x56053e282a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:12:05.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.456+0000 7f1d37c39740 1 -- 172.21.15.46:0/2286541747 shutdown_connections 2024-03-31T23:12:05.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.456+0000 7f1d37c39740 1 -- 172.21.15.46:0/2286541747 wait complete. 2024-03-31T23:12:05.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.457+0000 7f1d37c39740 1 -- start start 2024-03-31T23:12:05.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.457+0000 7f1d37c39740 1 Processor -- start 2024-03-31T23:12:05.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.457+0000 7f1d37c39740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56053e27d470 0x56053e2807c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:05.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.457+0000 7f1d37c39740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56053e27dd90 0x56053e280d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:05.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.457+0000 7f1d37c39740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56053e286970 0x56053e281240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:05.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.457+0000 7f1d37c39740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f1d301ad2b0 con 0x56053e27d470 2024-03-31T23:12:05.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.457+0000 7f1d37c39740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f1d301ae720 con 0x56053e27dd90 2024-03-31T23:12:05.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.457+0000 7f1d37c39740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f1d301ad980 con 0x56053e286970 2024-03-31T23:12:05.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.457+0000 7f1d37c39740 10 client.0 _ll_register_callbacks cb 0x7f1d28009e50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T23:12:05.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.457+0000 7f1d2e59c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56053e27d470 0x56053e2807c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:05.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.457+0000 7f1d2e59c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56053e27d470 0x56053e2807c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60710/0 (socket says 172.21.15.46:60710) 2024-03-31T23:12:05.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.457+0000 7f1d2e59c640 1 -- 172.21.15.46:0/1929032872 learned_addr learned my addr 172.21.15.46:0/1929032872 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:12:05.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.457+0000 7f1d2ed9d640 1 --2- 172.21.15.46:0/1929032872 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56053e286970 0x56053e281240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:05.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.457+0000 7f1d2dd9b640 1 --2- 172.21.15.46:0/1929032872 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56053e27dd90 0x56053e280d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:05.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.457+0000 7f1d2e59c640 1 -- 172.21.15.46:0/1929032872 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56053e286970 msgr2=0x56053e281240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:12:05.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.457+0000 7f1d2e59c640 1 --2- 172.21.15.46:0/1929032872 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56053e286970 0x56053e281240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:05.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.457+0000 7f1d2e59c640 1 -- 172.21.15.46:0/1929032872 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56053e27dd90 msgr2=0x56053e280d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:12:05.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.457+0000 7f1d2e59c640 1 --2- 172.21.15.46:0/1929032872 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56053e27dd90 0x56053e280d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:05.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.457+0000 7f1d2e59c640 1 -- 172.21.15.46:0/1929032872 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56053e2fbfe0 con 0x56053e27d470 2024-03-31T23:12:05.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.458+0000 7f1d2e59c640 1 --2- 172.21.15.46:0/1929032872 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56053e27d470 0x56053e2807c0 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f1d2800f610 tx=0x7f1d28009dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=45786645b1246e72 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:12:05.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.458+0000 7f1d15ffb640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-31T23:12:05.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.458+0000 7f1d15ffb640 1 -- 172.21.15.46:0/1929032872 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d280117b0 con 0x56053e27d470 2024-03-31T23:12:05.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.458+0000 7f1d15ffb640 1 -- 172.21.15.46:0/1929032872 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7f1d2800fb30 con 0x56053e27d470 2024-03-31T23:12:05.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.458+0000 7f1d37c39740 1 -- 172.21.15.46:0/1929032872 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1d20003cf0 con 0x56053e27d470 2024-03-31T23:12:05.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.458+0000 7f1d15ffb640 1 -- 172.21.15.46:0/1929032872 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f1d28038050 con 0x56053e27d470 2024-03-31T23:12:05.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.458+0000 7f1d37c39740 10 client.15816 fetch_fsmap learned FSMap version 25 2024-03-31T23:12:05.466 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.458+0000 7f1d37c39740 1 -- 172.21.15.46:0/1929032872 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x56053e2fd2c0 con 0x56053e27d470 2024-03-31T23:12:05.466 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.458+0000 7f1d15ffb640 1 -- 172.21.15.46:0/1929032872 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1d20003cf0 con 0x56053e27d470 2024-03-31T23:12:05.466 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.458+0000 7f1d15ffb640 1 -- 172.21.15.46:0/1929032872 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f1d2801a040 con 0x56053e27d470 2024-03-31T23:12:05.466 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.458+0000 7f1d37c39740 10 client.15816 fetch_fsmap finished waiting for FSMap version 25 2024-03-31T23:12:05.466 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.458+0000 7f1d37c39740 10 client.15816 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-03-31T23:12:05.466 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.458+0000 7f1d37c39740 20 client.15816 populate_metadata read hostname 'smithi046' 2024-03-31T23:12:05.466 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.458+0000 7f1d37c39740 1 --2- 172.21.15.46:0/1929032872 >> [v2:172.21.15.142:6836/2559258399,v1:172.21.15.142:6837/2559258399] conn(0x56053e2fda60 0x56053e2ffe80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:05.466 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.458+0000 7f1d37c39740 4 client.15816 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-31T23:12:05.466 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.458+0000 7f1d37c39740 1 -- 172.21.15.46:0/1929032872 --> [v2:172.21.15.142:6836/2559258399,v1:172.21.15.142:6837/2559258399] -- command(tid 0: {"prefix": "damage ls"}) -- 0x56053e1e8a40 con 0x56053e2fda60 2024-03-31T23:12:05.466 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.458+0000 7f1d2dd9b640 1 --2- 172.21.15.46:0/1929032872 >> [v2:172.21.15.142:6836/2559258399,v1:172.21.15.142:6837/2559258399] conn(0x56053e2fda60 0x56053e2ffe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:05.466 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.459+0000 7f1d15ffb640 10 client.15816 ms_handle_connect on v2:172.21.15.142:6836/2559258399 2024-03-31T23:12:05.466 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.459+0000 7f1d2dd9b640 1 --2- 172.21.15.46:0/1929032872 >> [v2:172.21.15.142:6836/2559258399,v1:172.21.15.142:6837/2559258399] conn(0x56053e2fda60 0x56053e2ffe80 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=35df2a70453a9094 server_cookie=30575fec2f58331a in_seq=0 out_seq=0 2024-03-31T23:12:05.466 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.459+0000 7f1d15ffb640 1 -- 172.21.15.46:0/1929032872 <== mds.1 v2:172.21.15.142:6836/2559258399 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x56053e1e8a40 con 0x56053e2fda60 2024-03-31T23:12:05.466 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.459+0000 7f1d15ffb640 10 client.15816 handle_command_reply: tid=0 2024-03-31T23:12:05.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.459+0000 7f1d37c39740 1 client.15816 shutdown 2024-03-31T23:12:05.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.459+0000 7f1d37c39740 1 -- 172.21.15.46:0/1929032872 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56053e27d470 msgr2=0x56053e2807c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:12:05.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.459+0000 7f1d37c39740 1 --2- 172.21.15.46:0/1929032872 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56053e27d470 0x56053e2807c0 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f1d2800f610 tx=0x7f1d28009dc0 comp rx=0 tx=0).stop 2024-03-31T23:12:05.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.459+0000 7f1d37c39740 1 -- 172.21.15.46:0/1929032872 shutdown_connections 2024-03-31T23:12:05.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.459+0000 7f1d37c39740 1 --2- 172.21.15.46:0/1929032872 >> [v2:172.21.15.142:6836/2559258399,v1:172.21.15.142:6837/2559258399] conn(0x56053e2fda60 0x56053e2ffe80 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:05.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.459+0000 7f1d37c39740 1 --2- 172.21.15.46:0/1929032872 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56053e286970 0x56053e281240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:05.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.459+0000 7f1d15ffb640 0 client.15816 ms_handle_reset on v2:172.21.15.142:6836/2559258399 2024-03-31T23:12:05.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.459+0000 7f1d37c39740 1 --2- 172.21.15.46:0/1929032872 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56053e27dd90 0x56053e280d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:05.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.459+0000 7f1d37c39740 1 --2- 172.21.15.46:0/1929032872 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56053e27d470 0x56053e2807c0 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:05.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.459+0000 7f1d37c39740 1 -- 172.21.15.46:0/1929032872 >> 172.21.15.46:0/1929032872 conn(0x56053e277ba0 msgr2=0x56053e2764b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:12:05.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.459+0000 7f1d37c39740 1 -- 172.21.15.46:0/1929032872 shutdown_connections 2024-03-31T23:12:05.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.459+0000 7f1d37c39740 1 -- 172.21.15.46:0/1929032872 wait complete. 2024-03-31T23:12:05.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.459+0000 7f1d37c39740 20 client.15816 trim_cache size 0 max 16384 2024-03-31T23:12:05.468 INFO:teuthology.orchestra.run.smithi046.stdout:[] 2024-03-31T23:12:05.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.460+0000 7f1d34f7a640 1 -- 172.21.15.46:0/2101881708 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f1d1c0a2730 msgr2=0x7f1d1c0a4bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:12:05.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.460+0000 7f1d34f7a640 1 --2- 172.21.15.46:0/2101881708 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f1d1c0a2730 0x7f1d1c0a4bf0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f1d2002c700 tx=0x7f1d20032d70 comp rx=0 tx=0).stop 2024-03-31T23:12:05.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.460+0000 7f1d34f7a640 1 -- 172.21.15.46:0/2101881708 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1d301020a0 msgr2=0x7f1d30198410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:12:05.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.460+0000 7f1d34f7a640 1 --2- 172.21.15.46:0/2101881708 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1d301020a0 0x7f1d30198410 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f1d1800e9f0 tx=0x7f1d1800eeb0 comp rx=0 tx=0).stop 2024-03-31T23:12:05.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.460+0000 7f1d34f7a640 1 -- 172.21.15.46:0/2101881708 shutdown_connections 2024-03-31T23:12:05.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.460+0000 7f1d34f7a640 1 --2- 172.21.15.46:0/2101881708 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f1d1c0a2730 0x7f1d1c0a4bf0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:05.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.460+0000 7f1d34f7a640 1 --2- 172.21.15.46:0/2101881708 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1d301076f0 0x7f1d3019c790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:05.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.460+0000 7f1d34f7a640 1 --2- 172.21.15.46:0/2101881708 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1d301020a0 0x7f1d30198410 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:05.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.460+0000 7f1d34f7a640 1 --2- 172.21.15.46:0/2101881708 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1d301016e0 0x7f1d30197ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:05.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.460+0000 7f1d34f7a640 1 -- 172.21.15.46:0/2101881708 >> 172.21.15.46:0/2101881708 conn(0x7f1d30078b50 msgr2=0x7f1d30104110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:12:05.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.460+0000 7f1d34f7a640 1 -- 172.21.15.46:0/2101881708 shutdown_connections 2024-03-31T23:12:05.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.460+0000 7f1d34f7a640 1 -- 172.21.15.46:0/2101881708 wait complete. 2024-03-31T23:12:05.470 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:12:05 smithi046 ceph-mon[32498]: from='client.? 172.21.15.46:0/3803581438' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-31T23:12:05.471 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-31T23:12:05.578 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.575+0000 7f254dbae640 1 -- 172.21.15.46:0/3415492690 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2548100f80 msgr2=0x7f254810d490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:12:05.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.575+0000 7f254dbae640 1 --2- 172.21.15.46:0/3415492690 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2548100f80 0x7f254810d490 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f2540009c10 tx=0x7f254002bef0 comp rx=0 tx=0).stop 2024-03-31T23:12:05.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.575+0000 7f254dbae640 1 -- 172.21.15.46:0/3415492690 shutdown_connections 2024-03-31T23:12:05.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.575+0000 7f254dbae640 1 --2- 172.21.15.46:0/3415492690 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2548100f80 0x7f254810d490 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:05.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.575+0000 7f254dbae640 1 --2- 172.21.15.46:0/3415492690 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f25481005c0 0x7f2548100a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:05.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.575+0000 7f254dbae640 1 --2- 172.21.15.46:0/3415492690 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2548106660 0x7f2548106a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:05.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.575+0000 7f254dbae640 1 -- 172.21.15.46:0/3415492690 >> 172.21.15.46:0/3415492690 conn(0x7f2548077750 msgr2=0x7f2548077b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:12:05.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.576+0000 7f254dbae640 1 -- 172.21.15.46:0/3415492690 shutdown_connections 2024-03-31T23:12:05.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.576+0000 7f254dbae640 1 -- 172.21.15.46:0/3415492690 wait complete. 2024-03-31T23:12:05.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.576+0000 7f254dbae640 1 Processor -- start 2024-03-31T23:12:05.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.576+0000 7f254dbae640 1 -- start start 2024-03-31T23:12:05.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.576+0000 7f254dbae640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f25481005c0 0x7f2548198c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:05.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.576+0000 7f254dbae640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2548100f80 0x7f25481991c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:05.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.576+0000 7f254dbae640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2548106660 0x7f25481a3d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:05.580 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.576+0000 7f254dbae640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f25481108a0 con 0x7f2548100f80 2024-03-31T23:12:05.580 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.576+0000 7f254dbae640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f2548110720 con 0x7f25481005c0 2024-03-31T23:12:05.580 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.576+0000 7f254dbae640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f2548110a20 con 0x7f2548106660 2024-03-31T23:12:05.580 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.577+0000 7f2546ffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2548100f80 0x7f25481991c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:05.580 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.577+0000 7f2546ffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2548100f80 0x7f25481991c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60728/0 (socket says 172.21.15.46:60728) 2024-03-31T23:12:05.580 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.577+0000 7f2546ffd640 1 -- 172.21.15.46:0/4232733442 learned_addr learned my addr 172.21.15.46:0/4232733442 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:12:05.580 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.577+0000 7f25477fe640 1 --2- 172.21.15.46:0/4232733442 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f25481005c0 0x7f2548198c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:05.580 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.577+0000 7f2547fff640 1 --2- 172.21.15.46:0/4232733442 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2548106660 0x7f25481a3d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:05.580 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.577+0000 7f2546ffd640 1 -- 172.21.15.46:0/4232733442 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2548106660 msgr2=0x7f25481a3d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:12:05.580 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.577+0000 7f2546ffd640 1 --2- 172.21.15.46:0/4232733442 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2548106660 0x7f25481a3d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:05.580 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.577+0000 7f2546ffd640 1 -- 172.21.15.46:0/4232733442 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f25481005c0 msgr2=0x7f2548198c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:12:05.580 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.577+0000 7f2546ffd640 1 --2- 172.21.15.46:0/4232733442 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f25481005c0 0x7f2548198c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:05.580 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.577+0000 7f2546ffd640 1 -- 172.21.15.46:0/4232733442 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f25481a4240 con 0x7f2548100f80 2024-03-31T23:12:05.580 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.577+0000 7f25477fe640 1 --2- 172.21.15.46:0/4232733442 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f25481005c0 0x7f2548198c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:12:05.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.577+0000 7f2546ffd640 1 --2- 172.21.15.46:0/4232733442 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2548100f80 0x7f25481991c0 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f253c00e9c0 tx=0x7f253c00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:12:05.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.577+0000 7f2544ff9640 1 -- 172.21.15.46:0/4232733442 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f253c00cde0 con 0x7f2548100f80 2024-03-31T23:12:05.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.577+0000 7f2544ff9640 1 -- 172.21.15.46:0/4232733442 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f253c0035f0 con 0x7f2548100f80 2024-03-31T23:12:05.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.577+0000 7f254dbae640 1 -- 172.21.15.46:0/4232733442 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f25481a4410 con 0x7f2548100f80 2024-03-31T23:12:05.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.577+0000 7f2544ff9640 1 -- 172.21.15.46:0/4232733442 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f253c010690 con 0x7f2548100f80 2024-03-31T23:12:05.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.577+0000 7f254dbae640 1 -- 172.21.15.46:0/4232733442 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f25481a4930 con 0x7f2548100f80 2024-03-31T23:12:05.587 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.579+0000 7f2544ff9640 1 -- 172.21.15.46:0/4232733442 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 24) ==== 137553+0+0 (secure 0 0 0) 0x7f253c0037a0 con 0x7f2548100f80 2024-03-31T23:12:05.587 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.580+0000 7f255078b740 1 Processor -- start 2024-03-31T23:12:05.587 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.580+0000 7f255078b740 1 -- start start 2024-03-31T23:12:05.587 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.580+0000 7f255078b740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5627e8130470 0x5627e8130850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:05.587 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.580+0000 7f255078b740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5627e8130d90 0x5627e8139430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:05.587 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.580+0000 7f2544ff9640 1 --2- 172.21.15.46:0/4232733442 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f25300a2730 0x7f25300a4bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:05.587 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.580+0000 7f255078b740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5627e8139970 0x5627e813bd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:05.587 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.580+0000 7f255078b740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x5627e80941b0 con 0x5627e8139970 2024-03-31T23:12:05.587 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.580+0000 7f255078b740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x5627e803cc00 con 0x5627e8130d90 2024-03-31T23:12:05.587 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.580+0000 7f255078b740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5627e8090250 con 0x5627e8130470 2024-03-31T23:12:05.588 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.580+0000 7f2546ffd640 1 --2- 172.21.15.46:0/4232733442 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f25300a2730 0x7f25300a4bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:05.588 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.580+0000 7f2547fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5627e8139970 0x5627e813bd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:05.588 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.580+0000 7f2547fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5627e8139970 0x5627e813bd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60736/0 (socket says 172.21.15.46:60736) 2024-03-31T23:12:05.588 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.580+0000 7f2547fff640 1 -- 172.21.15.46:0/1581007280 learned_addr learned my addr 172.21.15.46:0/1581007280 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:12:05.588 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.581+0000 7f2547fff640 1 --2- 172.21.15.46:0/1581007280 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5627e8130470 0x5627e8130850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:05.588 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.581+0000 7f25477fe640 1 --2- 172.21.15.46:0/1581007280 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5627e8130d90 0x5627e8139430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:05.588 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.581+0000 7f2544ff9640 1 -- 172.21.15.46:0/4232733442 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7f253c0cf500 con 0x7f2548100f80 2024-03-31T23:12:05.588 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.581+0000 7f2547fff640 1 -- 172.21.15.46:0/1581007280 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5627e8130470 msgr2=0x5627e8130850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:12:05.588 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.581+0000 7f2547fff640 1 --2- 172.21.15.46:0/1581007280 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5627e8130470 0x5627e8130850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:05.588 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.581+0000 7f2547fff640 1 -- 172.21.15.46:0/1581007280 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5627e8130d90 msgr2=0x5627e8139430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:12:05.588 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.581+0000 7f2547fff640 1 --2- 172.21.15.46:0/1581007280 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5627e8130d90 0x5627e8139430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:05.588 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.581+0000 7f2547fff640 1 -- 172.21.15.46:0/1581007280 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5627e805c3d0 con 0x5627e8139970 2024-03-31T23:12:05.588 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.581+0000 7f2546ffd640 1 --2- 172.21.15.46:0/4232733442 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f25300a2730 0x7f25300a4bf0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f253c0cc720 tx=0x7f253c0979c0 comp rx=0 tx=0).ready entity=mgr.15654 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:12:05.588 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.581+0000 7f2547fff640 1 --2- 172.21.15.46:0/1581007280 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5627e8139970 0x5627e813bd60 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f2540004af0 tx=0x7f254002bef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6cbaa3e56fd3889d server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:12:05.588 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.581+0000 7f252dffb640 1 -- 172.21.15.46:0/1581007280 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2540049070 con 0x5627e8139970 2024-03-31T23:12:05.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.581+0000 7f252dffb640 1 -- 172.21.15.46:0/1581007280 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f2540033a10 con 0x5627e8139970 2024-03-31T23:12:05.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.581+0000 7f252dffb640 1 -- 172.21.15.46:0/1581007280 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2540003dd0 con 0x5627e8139970 2024-03-31T23:12:05.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.582+0000 7f255078b740 1 -- 172.21.15.46:0/1581007280 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5627e8139970 msgr2=0x5627e813bd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:12:05.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.582+0000 7f255078b740 1 --2- 172.21.15.46:0/1581007280 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5627e8139970 0x5627e813bd60 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f2540004af0 tx=0x7f254002bef0 comp rx=0 tx=0).stop 2024-03-31T23:12:05.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.582+0000 7f255078b740 1 -- 172.21.15.46:0/1581007280 shutdown_connections 2024-03-31T23:12:05.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.582+0000 7f255078b740 1 --2- 172.21.15.46:0/1581007280 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5627e8139970 0x5627e813bd60 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:05.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.582+0000 7f255078b740 1 --2- 172.21.15.46:0/1581007280 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5627e8130d90 0x5627e8139430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:05.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.582+0000 7f255078b740 1 --2- 172.21.15.46:0/1581007280 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5627e8130470 0x5627e8130850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:05.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.582+0000 7f255078b740 1 -- 172.21.15.46:0/1581007280 >> 172.21.15.46:0/1581007280 conn(0x5627e812aba0 msgr2=0x5627e812af80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:12:05.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.582+0000 7f255078b740 1 -- 172.21.15.46:0/1581007280 shutdown_connections 2024-03-31T23:12:05.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.582+0000 7f255078b740 1 -- 172.21.15.46:0/1581007280 wait complete. 2024-03-31T23:12:05.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.582+0000 7f255078b740 1 -- start start 2024-03-31T23:12:05.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.582+0000 7f255078b740 1 Processor -- start 2024-03-31T23:12:05.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.583+0000 7f255078b740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5627e8130470 0x5627e8136a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:05.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.583+0000 7f255078b740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5627e8130d90 0x5627e8136f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:05.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.583+0000 7f255078b740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5627e8139970 0x5627e81374a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:05.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.583+0000 7f255078b740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x5627e813c700 con 0x5627e8130470 2024-03-31T23:12:05.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.583+0000 7f255078b740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x5627e8032470 con 0x5627e8130d90 2024-03-31T23:12:05.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.583+0000 7f255078b740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5627e813c880 con 0x5627e8139970 2024-03-31T23:12:05.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.583+0000 7f255078b740 10 client.0 _ll_register_callbacks cb 0x5627e804fcb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T23:12:05.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.583+0000 7f2547fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5627e8130470 0x5627e8136a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:05.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.583+0000 7f2547fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5627e8130470 0x5627e8136a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60740/0 (socket says 172.21.15.46:60740) 2024-03-31T23:12:05.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.583+0000 7f2547fff640 1 -- 172.21.15.46:0/1424772038 learned_addr learned my addr 172.21.15.46:0/1424772038 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:12:05.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.583+0000 7f25477fe640 1 --2- 172.21.15.46:0/1424772038 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5627e8130d90 0x5627e8136f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:05.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.583+0000 7f2547fff640 1 --2- 172.21.15.46:0/1424772038 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5627e8139970 0x5627e81374a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:05.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.583+0000 7f2547fff640 1 -- 172.21.15.46:0/1424772038 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5627e8139970 msgr2=0x5627e81374a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:12:05.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.583+0000 7f2547fff640 1 --2- 172.21.15.46:0/1424772038 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5627e8139970 0x5627e81374a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:05.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.583+0000 7f2547fff640 1 -- 172.21.15.46:0/1424772038 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5627e8130d90 msgr2=0x5627e8136f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:12:05.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.583+0000 7f2547fff640 1 --2- 172.21.15.46:0/1424772038 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5627e8130d90 0x5627e8136f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:05.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.583+0000 7f2547fff640 1 -- 172.21.15.46:0/1424772038 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5627e7da83f0 con 0x5627e8130470 2024-03-31T23:12:05.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.583+0000 7f2547fff640 1 --2- 172.21.15.46:0/1424772038 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5627e8130470 0x5627e8136a20 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f2540002230 tx=0x7f2540009500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1cf3c3f29bbc5c98 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:12:05.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.583+0000 7f2526ffd640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-31T23:12:05.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.583+0000 7f2526ffd640 1 -- 172.21.15.46:0/1424772038 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2540049040 con 0x5627e8130470 2024-03-31T23:12:05.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.583+0000 7f2526ffd640 1 -- 172.21.15.46:0/1424772038 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7f2540004d40 con 0x5627e8130470 2024-03-31T23:12:05.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.583+0000 7f255078b740 1 -- 172.21.15.46:0/1424772038 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2540033a10 con 0x5627e8130470 2024-03-31T23:12:05.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.583+0000 7f2526ffd640 1 -- 172.21.15.46:0/1424772038 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f254005f050 con 0x5627e8130470 2024-03-31T23:12:05.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.583+0000 7f255078b740 10 client.15828 fetch_fsmap learned FSMap version 25 2024-03-31T23:12:05.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.583+0000 7f255078b740 1 -- 172.21.15.46:0/1424772038 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x5627e7e727d0 con 0x5627e8130470 2024-03-31T23:12:05.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.584+0000 7f2526ffd640 1 -- 172.21.15.46:0/1424772038 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f2540033a10 con 0x5627e8130470 2024-03-31T23:12:05.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.584+0000 7f2526ffd640 1 -- 172.21.15.46:0/1424772038 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f2540043440 con 0x5627e8130470 2024-03-31T23:12:05.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.584+0000 7f255078b740 10 client.15828 fetch_fsmap finished waiting for FSMap version 25 2024-03-31T23:12:05.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.584+0000 7f255078b740 10 client.15828 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-31T23:12:05.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.584+0000 7f255078b740 20 client.15828 populate_metadata read hostname 'smithi046' 2024-03-31T23:12:05.592 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.584+0000 7f255078b740 1 --2- 172.21.15.46:0/1424772038 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x5627e8149810 0x5627e814bc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:05.592 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.584+0000 7f255078b740 4 client.15828 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-31T23:12:05.592 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.584+0000 7f255078b740 1 -- 172.21.15.46:0/1424772038 --> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5627e80a16f0 con 0x5627e8149810 2024-03-31T23:12:05.592 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.584+0000 7f25477fe640 1 --2- 172.21.15.46:0/1424772038 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x5627e8149810 0x5627e814bc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:05.592 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.584+0000 7f2526ffd640 10 client.15828 ms_handle_connect on v2:172.21.15.142:6838/2519143918 2024-03-31T23:12:05.592 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.584+0000 7f25477fe640 1 --2- 172.21.15.46:0/1424772038 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x5627e8149810 0x5627e814bc00 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c4f542c77014beb0 server_cookie=6f835d8cf4d9ec63 in_seq=0 out_seq=0 2024-03-31T23:12:05.592 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.585+0000 7f2526ffd640 1 -- 172.21.15.46:0/1424772038 <== mds.0 v2:172.21.15.142:6838/2519143918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5627e80a16f0 con 0x5627e8149810 2024-03-31T23:12:05.592 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.585+0000 7f2526ffd640 10 client.15828 handle_command_reply: tid=0 2024-03-31T23:12:05.592 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.585+0000 7f255078b740 1 client.15828 shutdown 2024-03-31T23:12:05.592 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.586+0000 7f255078b740 1 -- 172.21.15.46:0/1424772038 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5627e8130470 msgr2=0x5627e8136a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:12:05.592 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.586+0000 7f255078b740 1 --2- 172.21.15.46:0/1424772038 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5627e8130470 0x5627e8136a20 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f2540002230 tx=0x7f2540009500 comp rx=0 tx=0).stop 2024-03-31T23:12:05.592 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.586+0000 7f255078b740 1 -- 172.21.15.46:0/1424772038 shutdown_connections 2024-03-31T23:12:05.592 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.586+0000 7f255078b740 1 --2- 172.21.15.46:0/1424772038 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x5627e8149810 0x5627e814bc00 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:05.592 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.586+0000 7f255078b740 1 --2- 172.21.15.46:0/1424772038 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5627e8139970 0x5627e81374a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:05.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.586+0000 7f255078b740 1 --2- 172.21.15.46:0/1424772038 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5627e8130d90 0x5627e8136f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:05.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.586+0000 7f255078b740 1 --2- 172.21.15.46:0/1424772038 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5627e8130470 0x5627e8136a20 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:05.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.586+0000 7f2526ffd640 0 client.15828 ms_handle_reset on v2:172.21.15.142:6838/2519143918 2024-03-31T23:12:05.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.586+0000 7f255078b740 1 -- 172.21.15.46:0/1424772038 >> 172.21.15.46:0/1424772038 conn(0x5627e812aba0 msgr2=0x5627e812c8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:12:05.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.586+0000 7f255078b740 1 -- 172.21.15.46:0/1424772038 shutdown_connections 2024-03-31T23:12:05.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.586+0000 7f255078b740 1 -- 172.21.15.46:0/1424772038 wait complete. 2024-03-31T23:12:05.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.586+0000 7f255078b740 20 client.15828 trim_cache size 0 max 16384 2024-03-31T23:12:05.606 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.599+0000 7f255078b740 1 Processor -- start 2024-03-31T23:12:05.606 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.599+0000 7f255078b740 1 -- start start 2024-03-31T23:12:05.606 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.599+0000 7f255078b740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5627e8130470 0x5627e8129f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:05.606 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.599+0000 7f255078b740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5627e8130d90 0x5627e81454a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:05.606 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.599+0000 7f255078b740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5627e8139970 0x5627e8136210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:05.606 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.599+0000 7f255078b740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f25481b1440 con 0x5627e8130470 2024-03-31T23:12:05.606 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.599+0000 7f255078b740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f25481aed10 con 0x5627e8139970 2024-03-31T23:12:05.606 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.599+0000 7f255078b740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f25481acb60 con 0x5627e8130d90 2024-03-31T23:12:05.606 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.599+0000 7f2547fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5627e8130470 0x5627e8129f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:05.606 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.599+0000 7f2547fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5627e8130470 0x5627e8129f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60742/0 (socket says 172.21.15.46:60742) 2024-03-31T23:12:05.606 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.599+0000 7f2547fff640 1 -- 172.21.15.46:0/3072539645 learned_addr learned my addr 172.21.15.46:0/3072539645 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:12:05.607 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.599+0000 7f25477fe640 1 --2- 172.21.15.46:0/3072539645 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5627e8130d90 0x5627e81454a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:05.607 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.599+0000 7f2547fff640 1 --2- 172.21.15.46:0/3072539645 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5627e8139970 0x5627e8136210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:05.607 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.599+0000 7f2547fff640 1 -- 172.21.15.46:0/3072539645 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5627e8130d90 msgr2=0x5627e81454a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:12:05.607 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.600+0000 7f2547fff640 1 --2- 172.21.15.46:0/3072539645 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5627e8130d90 0x5627e81454a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:05.607 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.600+0000 7f2547fff640 1 -- 172.21.15.46:0/3072539645 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5627e8139970 msgr2=0x5627e8136210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:12:05.607 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.600+0000 7f2547fff640 1 --2- 172.21.15.46:0/3072539645 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5627e8139970 0x5627e8136210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:05.607 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.600+0000 7f2547fff640 1 -- 172.21.15.46:0/3072539645 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5627e8134fb0 con 0x5627e8130470 2024-03-31T23:12:05.607 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.600+0000 7f2547fff640 1 --2- 172.21.15.46:0/3072539645 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5627e8130470 0x5627e8129f80 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x5627e8125c80 tx=0x7f254002bef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f539111896424841 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:12:05.607 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.600+0000 7f252d7fa640 1 -- 172.21.15.46:0/3072539645 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2540043e60 con 0x5627e8130470 2024-03-31T23:12:05.607 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.600+0000 7f252d7fa640 1 -- 172.21.15.46:0/3072539645 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f2540056070 con 0x5627e8130470 2024-03-31T23:12:05.607 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.600+0000 7f252d7fa640 1 -- 172.21.15.46:0/3072539645 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2540049070 con 0x5627e8130470 2024-03-31T23:12:05.607 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.600+0000 7f255078b740 1 -- 172.21.15.46:0/3072539645 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5627e8130470 msgr2=0x5627e8129f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:12:05.607 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.600+0000 7f255078b740 1 --2- 172.21.15.46:0/3072539645 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5627e8130470 0x5627e8129f80 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x5627e8125c80 tx=0x7f254002bef0 comp rx=0 tx=0).stop 2024-03-31T23:12:05.607 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.600+0000 7f255078b740 1 -- 172.21.15.46:0/3072539645 shutdown_connections 2024-03-31T23:12:05.607 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.600+0000 7f255078b740 1 --2- 172.21.15.46:0/3072539645 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5627e8139970 0x5627e8136210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:05.608 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.600+0000 7f255078b740 1 --2- 172.21.15.46:0/3072539645 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5627e8130d90 0x5627e81454a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:05.608 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.600+0000 7f255078b740 1 --2- 172.21.15.46:0/3072539645 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5627e8130470 0x5627e8129f80 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:05.608 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.600+0000 7f255078b740 1 -- 172.21.15.46:0/3072539645 >> 172.21.15.46:0/3072539645 conn(0x5627e812aba0 msgr2=0x5627e8137270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:12:05.608 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.600+0000 7f255078b740 1 -- 172.21.15.46:0/3072539645 shutdown_connections 2024-03-31T23:12:05.608 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.600+0000 7f255078b740 1 -- 172.21.15.46:0/3072539645 wait complete. 2024-03-31T23:12:05.608 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.601+0000 7f255078b740 1 -- start start 2024-03-31T23:12:05.608 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.601+0000 7f255078b740 1 Processor -- start 2024-03-31T23:12:05.608 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.601+0000 7f255078b740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5627e8130470 0x5627e8133930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:05.608 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.601+0000 7f255078b740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5627e8130d90 0x5627e8133e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:05.608 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.601+0000 7f255078b740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5627e8139970 0x5627e81ae7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:05.608 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.601+0000 7f255078b740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f25481ad1d0 con 0x5627e8130d90 2024-03-31T23:12:05.608 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.601+0000 7f255078b740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f25481ae640 con 0x5627e8130470 2024-03-31T23:12:05.608 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.601+0000 7f255078b740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f25481ad8a0 con 0x5627e8139970 2024-03-31T23:12:05.608 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.601+0000 7f255078b740 10 client.0 _ll_register_callbacks cb 0x7f2540033d20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T23:12:05.608 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.601+0000 7f25477fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5627e8130d90 0x5627e8133e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:05.609 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.601+0000 7f25477fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5627e8130d90 0x5627e8133e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60758/0 (socket says 172.21.15.46:60758) 2024-03-31T23:12:05.609 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.601+0000 7f25477fe640 1 -- 172.21.15.46:0/2374213832 learned_addr learned my addr 172.21.15.46:0/2374213832 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:12:05.609 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.601+0000 7f2547fff640 1 --2- 172.21.15.46:0/2374213832 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5627e8139970 0x5627e81ae7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:05.609 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.601+0000 7f2547fff640 1 --2- 172.21.15.46:0/2374213832 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5627e8130470 0x5627e8133930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:05.609 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.602+0000 7f25477fe640 1 -- 172.21.15.46:0/2374213832 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5627e8139970 msgr2=0x5627e81ae7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:12:05.609 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.602+0000 7f25477fe640 1 --2- 172.21.15.46:0/2374213832 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5627e8139970 0x5627e81ae7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:05.609 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.602+0000 7f25477fe640 1 -- 172.21.15.46:0/2374213832 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5627e8130470 msgr2=0x5627e8133930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:12:05.609 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.602+0000 7f25477fe640 1 --2- 172.21.15.46:0/2374213832 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5627e8130470 0x5627e8133930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:05.609 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.602+0000 7f25477fe640 1 -- 172.21.15.46:0/2374213832 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5627e8134700 con 0x5627e8130d90 2024-03-31T23:12:05.609 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.602+0000 7f25477fe640 1 --2- 172.21.15.46:0/2374213832 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5627e8130d90 0x5627e8133e70 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f2538004700 tx=0x7f2538012c50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a8b33bd8fb7211f6 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:12:05.609 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.602+0000 7f2526ffd640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-31T23:12:05.609 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.602+0000 7f2526ffd640 1 -- 172.21.15.46:0/2374213832 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2538004250 con 0x5627e8130d90 2024-03-31T23:12:05.609 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.602+0000 7f2526ffd640 1 -- 172.21.15.46:0/2374213832 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7f25380043f0 con 0x5627e8130d90 2024-03-31T23:12:05.609 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.602+0000 7f255078b740 1 -- 172.21.15.46:0/2374213832 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2540056070 con 0x5627e8130d90 2024-03-31T23:12:05.610 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.602+0000 7f2526ffd640 1 -- 172.21.15.46:0/2374213832 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f253802f050 con 0x5627e8130d90 2024-03-31T23:12:05.610 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.602+0000 7f255078b740 10 client.15834 fetch_fsmap learned FSMap version 25 2024-03-31T23:12:05.610 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.602+0000 7f255078b740 1 -- 172.21.15.46:0/2374213832 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x5627e81afe40 con 0x5627e8130d90 2024-03-31T23:12:05.610 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.602+0000 7f2526ffd640 1 -- 172.21.15.46:0/2374213832 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f2540056070 con 0x5627e8130d90 2024-03-31T23:12:05.610 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.602+0000 7f2526ffd640 1 -- 172.21.15.46:0/2374213832 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f2538020500 con 0x5627e8130d90 2024-03-31T23:12:05.610 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.603+0000 7f255078b740 10 client.15834 fetch_fsmap finished waiting for FSMap version 25 2024-03-31T23:12:05.610 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.603+0000 7f255078b740 10 client.15834 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-31T23:12:05.610 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.603+0000 7f255078b740 20 client.15834 populate_metadata read hostname 'smithi046' 2024-03-31T23:12:05.610 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.603+0000 7f255078b740 1 --2- 172.21.15.46:0/2374213832 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x5627e81b05f0 0x5627e81b2a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:05.610 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.603+0000 7f255078b740 4 client.15834 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-31T23:12:05.610 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.603+0000 7f255078b740 1 -- 172.21.15.46:0/2374213832 --> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5627e809ba40 con 0x5627e81b05f0 2024-03-31T23:12:05.610 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.603+0000 7f2547fff640 1 --2- 172.21.15.46:0/2374213832 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x5627e81b05f0 0x5627e81b2a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:05.610 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.603+0000 7f2547fff640 1 --2- 172.21.15.46:0/2374213832 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x5627e81b05f0 0x5627e81b2a10 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=518e88fd834eac08 server_cookie=4a77b13a464b3bbf in_seq=0 out_seq=0 2024-03-31T23:12:05.610 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.603+0000 7f2526ffd640 10 client.15834 ms_handle_connect on v2:172.21.15.142:6838/2519143918 2024-03-31T23:12:05.611 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.603+0000 7f2526ffd640 1 -- 172.21.15.46:0/2374213832 <== mds.0 v2:172.21.15.142:6838/2519143918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5627e809ba40 con 0x5627e81b05f0 2024-03-31T23:12:05.611 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.603+0000 7f2526ffd640 10 client.15834 handle_command_reply: tid=0 2024-03-31T23:12:05.611 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.603+0000 7f255078b740 1 client.15834 shutdown 2024-03-31T23:12:05.611 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.603+0000 7f255078b740 1 -- 172.21.15.46:0/2374213832 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5627e8130d90 msgr2=0x5627e8133e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:12:05.611 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.603+0000 7f255078b740 1 --2- 172.21.15.46:0/2374213832 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5627e8130d90 0x5627e8133e70 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f2538004700 tx=0x7f2538012c50 comp rx=0 tx=0).stop 2024-03-31T23:12:05.611 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.603+0000 7f255078b740 1 -- 172.21.15.46:0/2374213832 shutdown_connections 2024-03-31T23:12:05.611 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.603+0000 7f255078b740 1 --2- 172.21.15.46:0/2374213832 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x5627e81b05f0 0x5627e81b2a10 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:05.611 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.603+0000 7f255078b740 1 --2- 172.21.15.46:0/2374213832 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5627e8139970 0x5627e81ae7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:05.611 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.603+0000 7f255078b740 1 --2- 172.21.15.46:0/2374213832 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5627e8130d90 0x5627e8133e70 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:05.611 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.603+0000 7f2526ffd640 0 client.15834 ms_handle_reset on v2:172.21.15.142:6838/2519143918 2024-03-31T23:12:05.611 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.603+0000 7f255078b740 1 --2- 172.21.15.46:0/2374213832 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5627e8130470 0x5627e8133930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:05.611 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.603+0000 7f255078b740 1 -- 172.21.15.46:0/2374213832 >> 172.21.15.46:0/2374213832 conn(0x5627e812aba0 msgr2=0x5627e812bc90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:12:05.611 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.603+0000 7f255078b740 1 -- 172.21.15.46:0/2374213832 shutdown_connections 2024-03-31T23:12:05.611 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.603+0000 7f255078b740 1 -- 172.21.15.46:0/2374213832 wait complete. 2024-03-31T23:12:05.612 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.604+0000 7f255078b740 20 client.15834 trim_cache size 0 max 16384 2024-03-31T23:12:05.612 INFO:teuthology.orchestra.run.smithi046.stdout:[] 2024-03-31T23:12:05.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.604+0000 7f254dbae640 1 -- 172.21.15.46:0/4232733442 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f25300a2730 msgr2=0x7f25300a4bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:12:05.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.604+0000 7f254dbae640 1 --2- 172.21.15.46:0/4232733442 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f25300a2730 0x7f25300a4bf0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f253c0cc720 tx=0x7f253c0979c0 comp rx=0 tx=0).stop 2024-03-31T23:12:05.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.604+0000 7f254dbae640 1 -- 172.21.15.46:0/4232733442 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2548100f80 msgr2=0x7f25481991c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:12:05.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.604+0000 7f254dbae640 1 --2- 172.21.15.46:0/4232733442 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2548100f80 0x7f25481991c0 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f253c00e9c0 tx=0x7f253c00ee80 comp rx=0 tx=0).stop 2024-03-31T23:12:05.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.604+0000 7f254dbae640 1 -- 172.21.15.46:0/4232733442 shutdown_connections 2024-03-31T23:12:05.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.604+0000 7f254dbae640 1 --2- 172.21.15.46:0/4232733442 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f25300a2730 0x7f25300a4bf0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:05.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.604+0000 7f254dbae640 1 --2- 172.21.15.46:0/4232733442 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2548106660 0x7f25481a3d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:05.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.604+0000 7f254dbae640 1 --2- 172.21.15.46:0/4232733442 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2548100f80 0x7f25481991c0 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:05.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.604+0000 7f254dbae640 1 --2- 172.21.15.46:0/4232733442 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f25481005c0 0x7f2548198c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:05.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.604+0000 7f254dbae640 1 -- 172.21.15.46:0/4232733442 >> 172.21.15.46:0/4232733442 conn(0x7f2548077750 msgr2=0x7f25480ff140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:12:05.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.604+0000 7f254dbae640 1 -- 172.21.15.46:0/4232733442 shutdown_connections 2024-03-31T23:12:05.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:05.604+0000 7f254dbae640 1 -- 172.21.15.46:0/4232733442 wait complete. 2024-03-31T23:12:05.693 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:12:05 smithi142 ceph-mon[35271]: from='client.? 172.21.15.46:0/3803581438' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-31T23:12:05.795 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:12:05 smithi162 ceph-mon[36237]: from='client.? 172.21.15.46:0/3803581438' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-31T23:12:06.611 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:12:06 smithi046 ceph-mon[32498]: pgmap v14: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 25 KiB/s wr, 28 op/s 2024-03-31T23:12:06.617 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-31T23:12:06.617 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T23:12:06.617 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-31T23:12:06.693 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:12:06 smithi142 ceph-mon[35271]: pgmap v14: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 25 KiB/s wr, 28 op/s 2024-03-31T23:12:06.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.734+0000 7f109a325640 1 -- 172.21.15.46:0/3057573711 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1094100f80 msgr2=0x7f109410d490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:12:06.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.734+0000 7f109a325640 1 --2- 172.21.15.46:0/3057573711 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1094100f80 0x7f109410d490 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f108c009cd0 tx=0x7f108c02c0f0 comp rx=0 tx=0).stop 2024-03-31T23:12:06.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.734+0000 7f109a325640 1 -- 172.21.15.46:0/3057573711 shutdown_connections 2024-03-31T23:12:06.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.734+0000 7f109a325640 1 --2- 172.21.15.46:0/3057573711 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1094100f80 0x7f109410d490 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:06.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.734+0000 7f109a325640 1 --2- 172.21.15.46:0/3057573711 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f10941005c0 0x7f1094100a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:06.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.734+0000 7f109a325640 1 --2- 172.21.15.46:0/3057573711 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1094106660 0x7f1094106a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:06.736 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.734+0000 7f109a325640 1 -- 172.21.15.46:0/3057573711 >> 172.21.15.46:0/3057573711 conn(0x7f1094077750 msgr2=0x7f1094077b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:12:06.736 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.734+0000 7f109a325640 1 -- 172.21.15.46:0/3057573711 shutdown_connections 2024-03-31T23:12:06.736 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.734+0000 7f109a325640 1 -- 172.21.15.46:0/3057573711 wait complete. 2024-03-31T23:12:06.736 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.734+0000 7f109a325640 1 Processor -- start 2024-03-31T23:12:06.736 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.734+0000 7f109a325640 1 -- start start 2024-03-31T23:12:06.738 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.735+0000 7f109a325640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f10941005c0 0x7f1094197e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:06.738 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.735+0000 7f109a325640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1094100f80 0x7f1094198350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:06.738 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.735+0000 7f109a325640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1094106660 0x7f109419c6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:06.738 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.735+0000 7f109a325640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f1094111990 con 0x7f1094100f80 2024-03-31T23:12:06.738 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.735+0000 7f109a325640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f1094111810 con 0x7f10941005c0 2024-03-31T23:12:06.738 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.735+0000 7f109a325640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f1094111b10 con 0x7f1094106660 2024-03-31T23:12:06.738 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.735+0000 7f10937fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1094100f80 0x7f1094198350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:06.738 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.735+0000 7f10937fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1094100f80 0x7f1094198350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60780/0 (socket says 172.21.15.46:60780) 2024-03-31T23:12:06.738 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.735+0000 7f10937fe640 1 -- 172.21.15.46:0/1718711561 learned_addr learned my addr 172.21.15.46:0/1718711561 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:12:06.738 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.735+0000 7f10988c2640 1 --2- 172.21.15.46:0/1718711561 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1094106660 0x7f109419c6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:06.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.735+0000 7f1093fff640 1 --2- 172.21.15.46:0/1718711561 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f10941005c0 0x7f1094197e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:06.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.735+0000 7f10937fe640 1 -- 172.21.15.46:0/1718711561 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1094106660 msgr2=0x7f109419c6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:12:06.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.735+0000 7f10937fe640 1 --2- 172.21.15.46:0/1718711561 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1094106660 0x7f109419c6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:06.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.735+0000 7f10937fe640 1 -- 172.21.15.46:0/1718711561 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f10941005c0 msgr2=0x7f1094197e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:12:06.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.735+0000 7f10937fe640 1 --2- 172.21.15.46:0/1718711561 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f10941005c0 0x7f1094197e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:06.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.735+0000 7f10937fe640 1 -- 172.21.15.46:0/1718711561 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f109419ce00 con 0x7f1094100f80 2024-03-31T23:12:06.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.735+0000 7f1093fff640 1 --2- 172.21.15.46:0/1718711561 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f10941005c0 0x7f1094197e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T23:12:06.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.735+0000 7f10937fe640 1 --2- 172.21.15.46:0/1718711561 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1094100f80 0x7f1094198350 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f108800e9f0 tx=0x7f108800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:12:06.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.735+0000 7f10917fa640 1 -- 172.21.15.46:0/1718711561 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f108800cd80 con 0x7f1094100f80 2024-03-31T23:12:06.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.735+0000 7f10917fa640 1 -- 172.21.15.46:0/1718711561 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f10880108e0 con 0x7f1094100f80 2024-03-31T23:12:06.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.735+0000 7f109a325640 1 -- 172.21.15.46:0/1718711561 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f109419d0f0 con 0x7f1094100f80 2024-03-31T23:12:06.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.735+0000 7f10917fa640 1 -- 172.21.15.46:0/1718711561 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1088018920 con 0x7f1094100f80 2024-03-31T23:12:06.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.736+0000 7f109a325640 1 -- 172.21.15.46:0/1718711561 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f10941a4980 con 0x7f1094100f80 2024-03-31T23:12:06.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.738+0000 7f10917fa640 1 -- 172.21.15.46:0/1718711561 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 24) ==== 137553+0+0 (secure 0 0 0) 0x7f1088010a90 con 0x7f1094100f80 2024-03-31T23:12:06.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.738+0000 7f109cfe5740 1 Processor -- start 2024-03-31T23:12:06.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.739+0000 7f109cfe5740 1 -- start start 2024-03-31T23:12:06.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.739+0000 7f109cfe5740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55abc49949f0 0x55abc4994dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:06.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.739+0000 7f109cfe5740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55abc4995310 0x55abc499d9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:06.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.739+0000 7f109cfe5740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55abc499def0 0x55abc49a02e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:06.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.739+0000 7f109cfe5740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55abc48950b0 con 0x55abc499def0 2024-03-31T23:12:06.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.739+0000 7f109cfe5740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55abc48f7ea0 con 0x55abc49949f0 2024-03-31T23:12:06.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.739+0000 7f109cfe5740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55abc48a0c00 con 0x55abc4995310 2024-03-31T23:12:06.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.739+0000 7f10937fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55abc499def0 0x55abc49a02e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:06.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.739+0000 7f1093fff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55abc4995310 0x55abc499d9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:06.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.739+0000 7f10937fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55abc499def0 0x55abc49a02e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60794/0 (socket says 172.21.15.46:60794) 2024-03-31T23:12:06.748 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.739+0000 7f1093fff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55abc4995310 0x55abc499d9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.46:33534/0 (socket says 172.21.15.46:33534) 2024-03-31T23:12:06.748 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.739+0000 7f10988c2640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55abc49949f0 0x55abc4994dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:06.748 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.739+0000 7f1093fff640 1 -- 172.21.15.46:0/2294867343 learned_addr learned my addr 172.21.15.46:0/2294867343 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:12:06.748 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.739+0000 7f10917fa640 1 --2- 172.21.15.46:0/1718711561 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f107c0a2730 0x7f107c0a4bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:06.748 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.739+0000 7f1093fff640 1 -- 172.21.15.46:0/2294867343 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55abc49949f0 msgr2=0x55abc4994dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:12:06.748 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.739+0000 7f10988c2640 1 --2- 172.21.15.46:0/1718711561 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f107c0a2730 0x7f107c0a4bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:06.748 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.739+0000 7f1093fff640 1 --2- 172.21.15.46:0/2294867343 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55abc49949f0 0x55abc4994dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:06.748 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.739+0000 7f1093fff640 1 -- 172.21.15.46:0/2294867343 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55abc499def0 msgr2=0x55abc49a02e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:12:06.748 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.739+0000 7f1093fff640 1 --2- 172.21.15.46:0/2294867343 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55abc499def0 0x55abc49a02e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:06.748 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.739+0000 7f1093fff640 1 -- 172.21.15.46:0/2294867343 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55abc46d47e0 con 0x55abc4995310 2024-03-31T23:12:06.748 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.739+0000 7f10917fa640 1 -- 172.21.15.46:0/1718711561 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7f10880cf580 con 0x7f1094100f80 2024-03-31T23:12:06.748 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.740+0000 7f1093fff640 1 --2- 172.21.15.46:0/2294867343 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55abc4995310 0x55abc499d9b0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f1084009cf0 tx=0x7f10840117b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=94faa052d07f355a server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:12:06.748 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.740+0000 7f107a7fc640 1 -- 172.21.15.46:0/2294867343 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1084012450 con 0x55abc4995310 2024-03-31T23:12:06.749 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.740+0000 7f107a7fc640 1 -- 172.21.15.46:0/2294867343 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1084012aa0 con 0x55abc4995310 2024-03-31T23:12:06.749 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.740+0000 7f10988c2640 1 --2- 172.21.15.46:0/1718711561 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f107c0a2730 0x7f107c0a4bf0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f108c005cc0 tx=0x7f108c02c0f0 comp rx=0 tx=0).ready entity=mgr.15654 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:12:06.749 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.740+0000 7f107a7fc640 1 -- 172.21.15.46:0/2294867343 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f108400fb20 con 0x55abc4995310 2024-03-31T23:12:06.749 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.740+0000 7f109cfe5740 1 -- 172.21.15.46:0/2294867343 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55abc4995310 msgr2=0x55abc499d9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:12:06.749 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.740+0000 7f109cfe5740 1 --2- 172.21.15.46:0/2294867343 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55abc4995310 0x55abc499d9b0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f1084009cf0 tx=0x7f10840117b0 comp rx=0 tx=0).stop 2024-03-31T23:12:06.749 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.740+0000 7f109cfe5740 1 -- 172.21.15.46:0/2294867343 shutdown_connections 2024-03-31T23:12:06.749 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.740+0000 7f109cfe5740 1 --2- 172.21.15.46:0/2294867343 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55abc499def0 0x55abc49a02e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:06.749 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.740+0000 7f109cfe5740 1 --2- 172.21.15.46:0/2294867343 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55abc4995310 0x55abc499d9b0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:06.749 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.740+0000 7f109cfe5740 1 --2- 172.21.15.46:0/2294867343 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55abc49949f0 0x55abc4994dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:06.749 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.740+0000 7f109cfe5740 1 -- 172.21.15.46:0/2294867343 >> 172.21.15.46:0/2294867343 conn(0x55abc498f120 msgr2=0x55abc498f500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:12:06.749 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.741+0000 7f109cfe5740 1 -- 172.21.15.46:0/2294867343 shutdown_connections 2024-03-31T23:12:06.749 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.741+0000 7f109cfe5740 1 -- 172.21.15.46:0/2294867343 wait complete. 2024-03-31T23:12:06.749 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.741+0000 7f109cfe5740 1 -- start start 2024-03-31T23:12:06.749 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.741+0000 7f109cfe5740 1 Processor -- start 2024-03-31T23:12:06.750 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.741+0000 7f109cfe5740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55abc49949f0 0x55abc499af40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:06.750 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.741+0000 7f109cfe5740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55abc4995310 0x55abc499b480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:06.750 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.741+0000 7f109cfe5740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55abc499def0 0x55abc499b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:06.750 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.741+0000 7f109cfe5740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55abc48963d0 con 0x55abc499def0 2024-03-31T23:12:06.750 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.741+0000 7f109cfe5740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55abc48f3f40 con 0x55abc49949f0 2024-03-31T23:12:06.750 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.741+0000 7f109cfe5740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55abc49a0ce0 con 0x55abc4995310 2024-03-31T23:12:06.750 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.741+0000 7f109cfe5740 10 client.0 _ll_register_callbacks cb 0x55abc48b3b70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T23:12:06.750 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.742+0000 7f10988c2640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55abc499def0 0x55abc499b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:06.750 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.742+0000 7f1093fff640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55abc49949f0 0x55abc499af40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:06.750 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.742+0000 7f10988c2640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55abc499def0 0x55abc499b9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60810/0 (socket says 172.21.15.46:60810) 2024-03-31T23:12:06.750 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.742+0000 7f10988c2640 1 -- 172.21.15.46:0/123549580 learned_addr learned my addr 172.21.15.46:0/123549580 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:12:06.750 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.742+0000 7f10937fe640 1 --2- 172.21.15.46:0/123549580 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55abc4995310 0x55abc499b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:06.751 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.742+0000 7f10988c2640 1 -- 172.21.15.46:0/123549580 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55abc4995310 msgr2=0x55abc499b480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:12:06.751 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.742+0000 7f10988c2640 1 --2- 172.21.15.46:0/123549580 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55abc4995310 0x55abc499b480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:06.751 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.742+0000 7f10988c2640 1 -- 172.21.15.46:0/123549580 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55abc49949f0 msgr2=0x55abc499af40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:12:06.751 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.742+0000 7f10988c2640 1 --2- 172.21.15.46:0/123549580 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55abc49949f0 0x55abc499af40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:06.751 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.742+0000 7f10988c2640 1 -- 172.21.15.46:0/123549580 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55abc4734f70 con 0x55abc499def0 2024-03-31T23:12:06.751 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.742+0000 7f10988c2640 1 --2- 172.21.15.46:0/123549580 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55abc499def0 0x55abc499b9c0 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f108c032940 tx=0x7f108c0095c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=29e8eab34d6e1f07 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:12:06.751 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.742+0000 7f10737fe640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-31T23:12:06.751 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.742+0000 7f10737fe640 1 -- 172.21.15.46:0/123549580 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f108c053060 con 0x55abc499def0 2024-03-31T23:12:06.751 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.742+0000 7f10737fe640 1 -- 172.21.15.46:0/123549580 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7f108c004450 con 0x55abc499def0 2024-03-31T23:12:06.751 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.742+0000 7f109cfe5740 1 -- 172.21.15.46:0/123549580 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1084012aa0 con 0x55abc499def0 2024-03-31T23:12:06.751 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.742+0000 7f10737fe640 1 -- 172.21.15.46:0/123549580 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f108c05a050 con 0x55abc499def0 2024-03-31T23:12:06.751 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.742+0000 7f109cfe5740 10 client.15846 fetch_fsmap learned FSMap version 25 2024-03-31T23:12:06.751 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.742+0000 7f109cfe5740 1 -- 172.21.15.46:0/123549580 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55abc45cc170 con 0x55abc499def0 2024-03-31T23:12:06.751 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.743+0000 7f10737fe640 1 -- 172.21.15.46:0/123549580 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1084012aa0 con 0x55abc499def0 2024-03-31T23:12:06.752 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.743+0000 7f10737fe640 1 -- 172.21.15.46:0/123549580 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f108c03f070 con 0x55abc499def0 2024-03-31T23:12:06.752 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.743+0000 7f109cfe5740 10 client.15846 fetch_fsmap finished waiting for FSMap version 25 2024-03-31T23:12:06.752 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.743+0000 7f109cfe5740 10 client.15846 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-31T23:12:06.752 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.743+0000 7f109cfe5740 20 client.15846 populate_metadata read hostname 'smithi046' 2024-03-31T23:12:06.752 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.743+0000 7f109cfe5740 1 --2- 172.21.15.46:0/123549580 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55abc49add80 0x55abc49b01e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:06.752 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.743+0000 7f109cfe5740 4 client.15846 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-31T23:12:06.752 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.743+0000 7f109cfe5740 1 -- 172.21.15.46:0/123549580 --> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55abc49053e0 con 0x55abc49add80 2024-03-31T23:12:06.752 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.743+0000 7f1093fff640 1 --2- 172.21.15.46:0/123549580 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55abc49add80 0x55abc49b01e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:06.752 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.743+0000 7f1093fff640 1 --2- 172.21.15.46:0/123549580 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55abc49add80 0x55abc49b01e0 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=28fcba092c410ef1 server_cookie=f4b22c7068423727 in_seq=0 out_seq=0 2024-03-31T23:12:06.752 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.743+0000 7f10737fe640 10 client.15846 ms_handle_connect on v2:172.21.15.142:6838/2519143918 2024-03-31T23:12:06.752 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.744+0000 7f10737fe640 1 -- 172.21.15.46:0/123549580 <== mds.0 v2:172.21.15.142:6838/2519143918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55abc49053e0 con 0x55abc49add80 2024-03-31T23:12:06.752 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.744+0000 7f10737fe640 10 client.15846 handle_command_reply: tid=0 2024-03-31T23:12:06.752 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.744+0000 7f109cfe5740 1 client.15846 shutdown 2024-03-31T23:12:06.753 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.744+0000 7f109cfe5740 1 -- 172.21.15.46:0/123549580 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55abc499def0 msgr2=0x55abc499b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:12:06.753 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.744+0000 7f109cfe5740 1 --2- 172.21.15.46:0/123549580 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55abc499def0 0x55abc499b9c0 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f108c032940 tx=0x7f108c0095c0 comp rx=0 tx=0).stop 2024-03-31T23:12:06.753 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.744+0000 7f109cfe5740 1 -- 172.21.15.46:0/123549580 shutdown_connections 2024-03-31T23:12:06.753 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.744+0000 7f109cfe5740 1 --2- 172.21.15.46:0/123549580 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55abc49add80 0x55abc49b01e0 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:06.753 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.744+0000 7f109cfe5740 1 --2- 172.21.15.46:0/123549580 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55abc499def0 0x55abc499b9c0 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:06.753 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.744+0000 7f109cfe5740 1 --2- 172.21.15.46:0/123549580 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55abc4995310 0x55abc499b480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:06.753 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.744+0000 7f10737fe640 0 client.15846 ms_handle_reset on v2:172.21.15.142:6838/2519143918 2024-03-31T23:12:06.753 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.744+0000 7f109cfe5740 1 --2- 172.21.15.46:0/123549580 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55abc49949f0 0x55abc499af40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:06.753 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.744+0000 7f109cfe5740 1 -- 172.21.15.46:0/123549580 >> 172.21.15.46:0/123549580 conn(0x55abc498f120 msgr2=0x55abc4990e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:12:06.753 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.744+0000 7f109cfe5740 1 -- 172.21.15.46:0/123549580 shutdown_connections 2024-03-31T23:12:06.753 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.744+0000 7f109cfe5740 1 -- 172.21.15.46:0/123549580 wait complete. 2024-03-31T23:12:06.753 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.745+0000 7f109cfe5740 20 client.15846 trim_cache size 0 max 16384 2024-03-31T23:12:06.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.762+0000 7f109cfe5740 1 Processor -- start 2024-03-31T23:12:06.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.762+0000 7f109cfe5740 1 -- start start 2024-03-31T23:12:06.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.763+0000 7f109cfe5740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55abc49949f0 0x55abc4999e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:06.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.763+0000 7f109cfe5740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55abc4995310 0x55abc49a9900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:06.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.763+0000 7f109cfe5740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55abc499def0 0x55abc49b5ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:06.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.763+0000 7f109cfe5740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f10941b13b0 con 0x55abc4995310 2024-03-31T23:12:06.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.763+0000 7f109cfe5740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f10941aec80 con 0x55abc49949f0 2024-03-31T23:12:06.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.763+0000 7f109cfe5740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f10941acad0 con 0x55abc499def0 2024-03-31T23:12:06.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.763+0000 7f10937fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55abc4995310 0x55abc49a9900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:06.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.763+0000 7f10937fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55abc4995310 0x55abc49a9900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60812/0 (socket says 172.21.15.46:60812) 2024-03-31T23:12:06.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.763+0000 7f10937fe640 1 -- 172.21.15.46:0/3011874034 learned_addr learned my addr 172.21.15.46:0/3011874034 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:12:06.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.763+0000 7f1093fff640 1 --2- 172.21.15.46:0/3011874034 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55abc49949f0 0x55abc4999e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:06.769 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.763+0000 7f10988c2640 1 --2- 172.21.15.46:0/3011874034 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55abc499def0 0x55abc49b5ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:06.769 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.763+0000 7f10937fe640 1 -- 172.21.15.46:0/3011874034 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55abc499def0 msgr2=0x55abc49b5ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:12:06.769 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.763+0000 7f10937fe640 1 --2- 172.21.15.46:0/3011874034 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55abc499def0 0x55abc49b5ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:06.769 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.763+0000 7f10937fe640 1 -- 172.21.15.46:0/3011874034 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55abc49949f0 msgr2=0x55abc4999e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:12:06.769 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.763+0000 7f10937fe640 1 --2- 172.21.15.46:0/3011874034 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55abc49949f0 0x55abc4999e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:06.769 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.763+0000 7f10937fe640 1 -- 172.21.15.46:0/3011874034 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55abc49a02e0 con 0x55abc4995310 2024-03-31T23:12:06.769 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.763+0000 7f1093fff640 1 --2- 172.21.15.46:0/3011874034 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55abc49949f0 0x55abc4999e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:12:06.769 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.763+0000 7f10937fe640 1 --2- 172.21.15.46:0/3011874034 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55abc4995310 0x55abc49a9900 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f1088098070 tx=0x7f1088097820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=64a6f95689e09d2 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:12:06.769 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.763+0000 7f1079ffb640 1 -- 172.21.15.46:0/3011874034 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f108809b000 con 0x55abc4995310 2024-03-31T23:12:06.769 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.763+0000 7f1079ffb640 1 -- 172.21.15.46:0/3011874034 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1088010de0 con 0x55abc4995310 2024-03-31T23:12:06.769 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.763+0000 7f1079ffb640 1 -- 172.21.15.46:0/3011874034 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10880cb7b0 con 0x55abc4995310 2024-03-31T23:12:06.769 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.764+0000 7f109cfe5740 1 -- 172.21.15.46:0/3011874034 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55abc4995310 msgr2=0x55abc49a9900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:12:06.769 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.764+0000 7f109cfe5740 1 --2- 172.21.15.46:0/3011874034 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55abc4995310 0x55abc49a9900 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f1088098070 tx=0x7f1088097820 comp rx=0 tx=0).stop 2024-03-31T23:12:06.769 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.764+0000 7f109cfe5740 1 -- 172.21.15.46:0/3011874034 shutdown_connections 2024-03-31T23:12:06.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.764+0000 7f109cfe5740 1 --2- 172.21.15.46:0/3011874034 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55abc499def0 0x55abc49b5ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:06.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.764+0000 7f109cfe5740 1 --2- 172.21.15.46:0/3011874034 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55abc4995310 0x55abc49a9900 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:06.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.764+0000 7f109cfe5740 1 --2- 172.21.15.46:0/3011874034 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55abc49949f0 0x55abc4999e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:06.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.764+0000 7f109cfe5740 1 -- 172.21.15.46:0/3011874034 >> 172.21.15.46:0/3011874034 conn(0x55abc498f120 msgr2=0x55abc498e180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:12:06.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.764+0000 7f109cfe5740 1 -- 172.21.15.46:0/3011874034 shutdown_connections 2024-03-31T23:12:06.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.764+0000 7f109cfe5740 1 -- 172.21.15.46:0/3011874034 wait complete. 2024-03-31T23:12:06.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.764+0000 7f109cfe5740 1 -- start start 2024-03-31T23:12:06.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.764+0000 7f109cfe5740 1 Processor -- start 2024-03-31T23:12:06.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.765+0000 7f109cfe5740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55abc49949f0 0x55abc49a6120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:06.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.765+0000 7f109cfe5740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55abc4995310 0x55abc49a2f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:06.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.765+0000 7f109cfe5740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55abc499def0 0x55abc49a3440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:06.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.765+0000 7f109cfe5740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f10941ad140 con 0x55abc49949f0 2024-03-31T23:12:06.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.765+0000 7f109cfe5740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f10941ae5b0 con 0x55abc499def0 2024-03-31T23:12:06.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.765+0000 7f109cfe5740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f10941ad810 con 0x55abc4995310 2024-03-31T23:12:06.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.765+0000 7f109cfe5740 10 client.0 _ll_register_callbacks cb 0x7f108c032800 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T23:12:06.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.765+0000 7f1093fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55abc49949f0 0x55abc49a6120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:06.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.765+0000 7f1093fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55abc49949f0 0x55abc49a6120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60828/0 (socket says 172.21.15.46:60828) 2024-03-31T23:12:06.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.765+0000 7f1093fff640 1 -- 172.21.15.46:0/3301422454 learned_addr learned my addr 172.21.15.46:0/3301422454 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:12:06.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.765+0000 7f10937fe640 1 --2- 172.21.15.46:0/3301422454 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55abc4995310 0x55abc49a2f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:06.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.765+0000 7f10988c2640 1 --2- 172.21.15.46:0/3301422454 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55abc499def0 0x55abc49a3440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:06.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.765+0000 7f1093fff640 1 -- 172.21.15.46:0/3301422454 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55abc4995310 msgr2=0x55abc49a2f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:12:06.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.765+0000 7f1093fff640 1 --2- 172.21.15.46:0/3301422454 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55abc4995310 0x55abc49a2f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:06.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.765+0000 7f1093fff640 1 -- 172.21.15.46:0/3301422454 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55abc499def0 msgr2=0x55abc49a3440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:12:06.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.765+0000 7f1093fff640 1 --2- 172.21.15.46:0/3301422454 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55abc499def0 0x55abc49a3440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:06.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.765+0000 7f1093fff640 1 -- 172.21.15.46:0/3301422454 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55abc49a3a00 con 0x55abc49949f0 2024-03-31T23:12:06.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.765+0000 7f1093fff640 1 --2- 172.21.15.46:0/3301422454 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55abc49949f0 0x55abc49a6120 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f1084011cb0 tx=0x7f1084009620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=efa23280c4be643b server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:12:06.772 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.765+0000 7f10737fe640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-31T23:12:06.772 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.765+0000 7f10737fe640 1 -- 172.21.15.46:0/3301422454 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10840052c0 con 0x55abc49949f0 2024-03-31T23:12:06.772 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.765+0000 7f10737fe640 1 -- 172.21.15.46:0/3301422454 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7f1084020040 con 0x55abc49949f0 2024-03-31T23:12:06.772 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.765+0000 7f109cfe5740 1 -- 172.21.15.46:0/3301422454 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55abc48ff730 con 0x55abc49949f0 2024-03-31T23:12:06.772 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.766+0000 7f10737fe640 1 -- 172.21.15.46:0/3301422454 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f108403c050 con 0x55abc49949f0 2024-03-31T23:12:06.772 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.766+0000 7f109cfe5740 10 client.15852 fetch_fsmap learned FSMap version 25 2024-03-31T23:12:06.772 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.766+0000 7f109cfe5740 1 -- 172.21.15.46:0/3301422454 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55abc4a14f20 con 0x55abc49949f0 2024-03-31T23:12:06.772 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.766+0000 7f10737fe640 1 -- 172.21.15.46:0/3301422454 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x55abc48ff730 con 0x55abc49949f0 2024-03-31T23:12:06.772 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.766+0000 7f10737fe640 1 -- 172.21.15.46:0/3301422454 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f108400f560 con 0x55abc49949f0 2024-03-31T23:12:06.772 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.766+0000 7f109cfe5740 10 client.15852 fetch_fsmap finished waiting for FSMap version 25 2024-03-31T23:12:06.772 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.766+0000 7f109cfe5740 10 client.15852 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-31T23:12:06.772 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.766+0000 7f109cfe5740 20 client.15852 populate_metadata read hostname 'smithi046' 2024-03-31T23:12:06.772 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.766+0000 7f109cfe5740 1 --2- 172.21.15.46:0/3301422454 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55abc4a153c0 0x55abc4a177e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:06.772 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.766+0000 7f109cfe5740 4 client.15852 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-31T23:12:06.773 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.766+0000 7f109cfe5740 1 -- 172.21.15.46:0/3301422454 --> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55abc487dbe0 con 0x55abc4a153c0 2024-03-31T23:12:06.773 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.766+0000 7f10937fe640 1 --2- 172.21.15.46:0/3301422454 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55abc4a153c0 0x55abc4a177e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:06.773 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.767+0000 7f10737fe640 10 client.15852 ms_handle_connect on v2:172.21.15.142:6838/2519143918 2024-03-31T23:12:06.773 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.767+0000 7f10937fe640 1 --2- 172.21.15.46:0/3301422454 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55abc4a153c0 0x55abc4a177e0 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=941187d1c5941c21 server_cookie=a225ba84ad5c6b50 in_seq=0 out_seq=0 2024-03-31T23:12:06.774 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.768+0000 7f10737fe640 1 -- 172.21.15.46:0/3301422454 <== mds.0 v2:172.21.15.142:6838/2519143918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55abc487dbe0 con 0x55abc4a153c0 2024-03-31T23:12:06.774 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.768+0000 7f10737fe640 10 client.15852 handle_command_reply: tid=0 2024-03-31T23:12:06.774 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.768+0000 7f109cfe5740 1 client.15852 shutdown 2024-03-31T23:12:06.774 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.768+0000 7f109cfe5740 1 -- 172.21.15.46:0/3301422454 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55abc49949f0 msgr2=0x55abc49a6120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:12:06.774 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.768+0000 7f109cfe5740 1 --2- 172.21.15.46:0/3301422454 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55abc49949f0 0x55abc49a6120 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f1084011cb0 tx=0x7f1084009620 comp rx=0 tx=0).stop 2024-03-31T23:12:06.774 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.768+0000 7f109cfe5740 1 -- 172.21.15.46:0/3301422454 shutdown_connections 2024-03-31T23:12:06.774 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.768+0000 7f109cfe5740 1 --2- 172.21.15.46:0/3301422454 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55abc4a153c0 0x55abc4a177e0 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:06.774 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.768+0000 7f109cfe5740 1 --2- 172.21.15.46:0/3301422454 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55abc499def0 0x55abc49a3440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:06.774 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.768+0000 7f109cfe5740 1 --2- 172.21.15.46:0/3301422454 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55abc4995310 0x55abc49a2f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:06.775 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.768+0000 7f109cfe5740 1 --2- 172.21.15.46:0/3301422454 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55abc49949f0 0x55abc49a6120 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:06.775 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.768+0000 7f10737fe640 0 client.15852 ms_handle_reset on v2:172.21.15.142:6838/2519143918 2024-03-31T23:12:06.775 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.768+0000 7f109cfe5740 1 -- 172.21.15.46:0/3301422454 >> 172.21.15.46:0/3301422454 conn(0x55abc498f120 msgr2=0x55abc499af00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:12:06.775 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.768+0000 7f109cfe5740 1 -- 172.21.15.46:0/3301422454 shutdown_connections 2024-03-31T23:12:06.775 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.768+0000 7f109cfe5740 1 -- 172.21.15.46:0/3301422454 wait complete. 2024-03-31T23:12:06.775 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.768+0000 7f109cfe5740 20 client.15852 trim_cache size 0 max 16384 2024-03-31T23:12:06.775 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-03-31T23:12:06.775 INFO:teuthology.orchestra.run.smithi046.stdout: "return_code": 0, 2024-03-31T23:12:06.775 INFO:teuthology.orchestra.run.smithi046.stdout: "scrub_tag": "55ee9bbf-24db-4416-ae8b-3a205308a428", 2024-03-31T23:12:06.775 INFO:teuthology.orchestra.run.smithi046.stdout: "mode": "asynchronous" 2024-03-31T23:12:06.776 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-03-31T23:12:06.776 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.769+0000 7f109a325640 1 -- 172.21.15.46:0/1718711561 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f107c0a2730 msgr2=0x7f107c0a4bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:12:06.777 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.769+0000 7f109a325640 1 --2- 172.21.15.46:0/1718711561 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f107c0a2730 0x7f107c0a4bf0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f108c005cc0 tx=0x7f108c02c0f0 comp rx=0 tx=0).stop 2024-03-31T23:12:06.777 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.769+0000 7f109a325640 1 -- 172.21.15.46:0/1718711561 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1094100f80 msgr2=0x7f1094198350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:12:06.777 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.769+0000 7f109a325640 1 --2- 172.21.15.46:0/1718711561 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1094100f80 0x7f1094198350 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f108800e9f0 tx=0x7f108800eeb0 comp rx=0 tx=0).stop 2024-03-31T23:12:06.777 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.769+0000 7f109a325640 1 -- 172.21.15.46:0/1718711561 shutdown_connections 2024-03-31T23:12:06.777 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.769+0000 7f109a325640 1 --2- 172.21.15.46:0/1718711561 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f107c0a2730 0x7f107c0a4bf0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:06.777 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.769+0000 7f109a325640 1 --2- 172.21.15.46:0/1718711561 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1094106660 0x7f109419c6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:06.777 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.769+0000 7f109a325640 1 --2- 172.21.15.46:0/1718711561 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1094100f80 0x7f1094198350 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:06.777 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.769+0000 7f109a325640 1 --2- 172.21.15.46:0/1718711561 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f10941005c0 0x7f1094197e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:06.777 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.769+0000 7f109a325640 1 -- 172.21.15.46:0/1718711561 >> 172.21.15.46:0/1718711561 conn(0x7f1094077750 msgr2=0x7f10940ff1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:12:06.777 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.769+0000 7f109a325640 1 -- 172.21.15.46:0/1718711561 shutdown_connections 2024-03-31T23:12:06.777 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.769+0000 7f109a325640 1 -- 172.21.15.46:0/1718711561 wait complete. 2024-03-31T23:12:06.781 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-31T23:12:06.781 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-31T23:12:06.795 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:12:06 smithi162 ceph-mon[36237]: pgmap v14: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 25 KiB/s wr, 28 op/s 2024-03-31T23:12:06.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.885+0000 7f946bce1640 1 -- 172.21.15.46:0/1703412439 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f94641010d0 msgr2=0x7f9464105590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:12:06.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.885+0000 7f946bce1640 1 --2- 172.21.15.46:0/1703412439 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f94641010d0 0x7f9464105590 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f9454009960 tx=0x7f945402bca0 comp rx=0 tx=0).stop 2024-03-31T23:12:06.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.885+0000 7f946bce1640 1 -- 172.21.15.46:0/1703412439 shutdown_connections 2024-03-31T23:12:06.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.885+0000 7f946bce1640 1 --2- 172.21.15.46:0/1703412439 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9464105bb0 0x7f9464106060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:06.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.885+0000 7f946bce1640 1 --2- 172.21.15.46:0/1703412439 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f94641010d0 0x7f9464105590 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:06.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.885+0000 7f946bce1640 1 --2- 172.21.15.46:0/1703412439 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f94641007b0 0x7f9464100b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:06.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.885+0000 7f946bce1640 1 -- 172.21.15.46:0/1703412439 >> 172.21.15.46:0/1703412439 conn(0x7f94640777b0 msgr2=0x7f9464077bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:12:06.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.885+0000 7f946bce1640 1 -- 172.21.15.46:0/1703412439 shutdown_connections 2024-03-31T23:12:06.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.885+0000 7f946bce1640 1 -- 172.21.15.46:0/1703412439 wait complete. 2024-03-31T23:12:06.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.886+0000 7f946bce1640 1 Processor -- start 2024-03-31T23:12:06.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.886+0000 7f946bce1640 1 -- start start 2024-03-31T23:12:06.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.886+0000 7f946bce1640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f94641007b0 0x7f946419e610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:06.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.886+0000 7f946bce1640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f94641010d0 0x7f946419eb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:06.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.886+0000 7f946bce1640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9464105bb0 0x7f94641a2ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:06.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.886+0000 7f946bce1640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f9464117fe0 con 0x7f94641010d0 2024-03-31T23:12:06.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.886+0000 7f946bce1640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f9464117e60 con 0x7f9464105bb0 2024-03-31T23:12:06.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.886+0000 7f946bce1640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f9464118160 con 0x7f94641007b0 2024-03-31T23:12:06.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.886+0000 7f946927c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f94641010d0 0x7f946419eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:06.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.886+0000 7f946927c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f94641010d0 0x7f946419eb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60850/0 (socket says 172.21.15.46:60850) 2024-03-31T23:12:06.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.886+0000 7f946927c640 1 -- 172.21.15.46:0/2771138114 learned_addr learned my addr 172.21.15.46:0/2771138114 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:12:06.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.886+0000 7f946a27e640 1 --2- 172.21.15.46:0/2771138114 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9464105bb0 0x7f94641a2ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:06.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.886+0000 7f9469a7d640 1 --2- 172.21.15.46:0/2771138114 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f94641007b0 0x7f946419e610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:06.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.887+0000 7f946927c640 1 -- 172.21.15.46:0/2771138114 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f94641007b0 msgr2=0x7f946419e610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:12:06.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.887+0000 7f946927c640 1 --2- 172.21.15.46:0/2771138114 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f94641007b0 0x7f946419e610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:06.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.887+0000 7f946927c640 1 -- 172.21.15.46:0/2771138114 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9464105bb0 msgr2=0x7f94641a2ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:12:06.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.887+0000 7f946927c640 1 --2- 172.21.15.46:0/2771138114 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9464105bb0 0x7f94641a2ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:06.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.887+0000 7f946927c640 1 -- 172.21.15.46:0/2771138114 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f94641a3600 con 0x7f94641010d0 2024-03-31T23:12:06.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.887+0000 7f946927c640 1 --2- 172.21.15.46:0/2771138114 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f94641010d0 0x7f946419eb50 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f9454036040 tx=0x7f94540045e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:12:06.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.887+0000 7f945affd640 1 -- 172.21.15.46:0/2771138114 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9454043020 con 0x7f94641010d0 2024-03-31T23:12:06.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.887+0000 7f946bce1640 1 -- 172.21.15.46:0/2771138114 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f94641a3890 con 0x7f94641010d0 2024-03-31T23:12:06.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.887+0000 7f945affd640 1 -- 172.21.15.46:0/2771138114 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9454004e00 con 0x7f94641010d0 2024-03-31T23:12:06.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.887+0000 7f945affd640 1 -- 172.21.15.46:0/2771138114 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94540038e0 con 0x7f94641010d0 2024-03-31T23:12:06.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.887+0000 7f946bce1640 1 -- 172.21.15.46:0/2771138114 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f94641ab180 con 0x7f94641010d0 2024-03-31T23:12:06.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.889+0000 7f945affd640 1 -- 172.21.15.46:0/2771138114 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 24) ==== 137553+0+0 (secure 0 0 0) 0x7f9454046020 con 0x7f94641010d0 2024-03-31T23:12:06.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.890+0000 7f946e99b740 1 Processor -- start 2024-03-31T23:12:06.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.890+0000 7f946e99b740 1 -- start start 2024-03-31T23:12:06.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.890+0000 7f946e99b740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55599a52b290 0x55599a52b670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:06.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.890+0000 7f946e99b740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55599a52bbb0 0x55599a534250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:06.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.890+0000 7f946e99b740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55599a534790 0x55599a536b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:06.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.890+0000 7f946e99b740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55599a48f180 con 0x55599a52bbb0 2024-03-31T23:12:06.898 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.890+0000 7f946e99b740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55599a437b70 con 0x55599a534790 2024-03-31T23:12:06.898 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.890+0000 7f946e99b740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55599a48b220 con 0x55599a52b290 2024-03-31T23:12:06.898 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.890+0000 7f945affd640 1 --2- 172.21.15.46:0/2771138114 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f94500a2730 0x7f94500a4bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:06.898 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.890+0000 7f9469a7d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55599a52bbb0 0x55599a534250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:06.898 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.890+0000 7f9469a7d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55599a52bbb0 0x55599a534250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60864/0 (socket says 172.21.15.46:60864) 2024-03-31T23:12:06.898 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.890+0000 7f9469a7d640 1 -- 172.21.15.46:0/133660917 learned_addr learned my addr 172.21.15.46:0/133660917 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:12:06.898 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.890+0000 7f946a27e640 1 --2- 172.21.15.46:0/133660917 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55599a52b290 0x55599a52b670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:06.898 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.890+0000 7f946927c640 1 --2- 172.21.15.46:0/133660917 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55599a534790 0x55599a536b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:06.898 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.890+0000 7f946a27e640 1 --2- 172.21.15.46:0/2771138114 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f94500a2730 0x7f94500a4bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:06.898 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.890+0000 7f945affd640 1 -- 172.21.15.46:0/2771138114 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7f94540f0f70 con 0x7f94641010d0 2024-03-31T23:12:06.898 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.890+0000 7f9469a7d640 1 -- 172.21.15.46:0/133660917 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55599a52b290 msgr2=0x55599a52b670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:12:06.898 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.890+0000 7f9469a7d640 1 --2- 172.21.15.46:0/133660917 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55599a52b290 0x55599a52b670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:06.898 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.890+0000 7f9469a7d640 1 -- 172.21.15.46:0/133660917 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55599a534790 msgr2=0x55599a536b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:12:06.899 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.890+0000 7f9469a7d640 1 --2- 172.21.15.46:0/133660917 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55599a534790 0x55599a536b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:06.899 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.890+0000 7f9469a7d640 1 -- 172.21.15.46:0/133660917 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55599a44b7c0 con 0x55599a52bbb0 2024-03-31T23:12:06.899 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.891+0000 7f946a27e640 1 --2- 172.21.15.46:0/2771138114 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f94500a2730 0x7f94500a4bf0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f945c010760 tx=0x7f945c005f90 comp rx=0 tx=0).ready entity=mgr.15654 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:12:06.899 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.891+0000 7f9469a7d640 1 --2- 172.21.15.46:0/133660917 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55599a52bbb0 0x55599a534250 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f94600114f0 tx=0x7f94600118f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=31cb57a0f29f6dea server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:12:06.899 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.891+0000 7f944ffff640 1 -- 172.21.15.46:0/133660917 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f946001e070 con 0x55599a52bbb0 2024-03-31T23:12:06.899 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.891+0000 7f944ffff640 1 -- 172.21.15.46:0/133660917 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9460003c30 con 0x55599a52bbb0 2024-03-31T23:12:06.899 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.891+0000 7f944ffff640 1 -- 172.21.15.46:0/133660917 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f946000f920 con 0x55599a52bbb0 2024-03-31T23:12:06.899 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.892+0000 7f946e99b740 1 -- 172.21.15.46:0/133660917 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55599a52bbb0 msgr2=0x55599a534250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:12:06.899 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.892+0000 7f946e99b740 1 --2- 172.21.15.46:0/133660917 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55599a52bbb0 0x55599a534250 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f94600114f0 tx=0x7f94600118f0 comp rx=0 tx=0).stop 2024-03-31T23:12:06.899 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.892+0000 7f946e99b740 1 -- 172.21.15.46:0/133660917 shutdown_connections 2024-03-31T23:12:06.899 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.892+0000 7f946e99b740 1 --2- 172.21.15.46:0/133660917 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55599a534790 0x55599a536b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:06.899 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.892+0000 7f946e99b740 1 --2- 172.21.15.46:0/133660917 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55599a52bbb0 0x55599a534250 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:06.899 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.892+0000 7f946e99b740 1 --2- 172.21.15.46:0/133660917 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55599a52b290 0x55599a52b670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:06.899 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.892+0000 7f946e99b740 1 -- 172.21.15.46:0/133660917 >> 172.21.15.46:0/133660917 conn(0x55599a5259c0 msgr2=0x55599a525da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:12:06.899 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.892+0000 7f946e99b740 1 -- 172.21.15.46:0/133660917 shutdown_connections 2024-03-31T23:12:06.900 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.892+0000 7f946e99b740 1 -- 172.21.15.46:0/133660917 wait complete. 2024-03-31T23:12:06.900 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.892+0000 7f946e99b740 1 -- start start 2024-03-31T23:12:06.900 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.892+0000 7f946e99b740 1 Processor -- start 2024-03-31T23:12:06.900 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.892+0000 7f946e99b740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55599a52b290 0x55599a531740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:06.900 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.892+0000 7f946e99b740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55599a52bbb0 0x55599a531c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:06.900 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.892+0000 7f946e99b740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55599a534790 0x55599a5321c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:06.900 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.892+0000 7f946e99b740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55599a537580 con 0x55599a52b290 2024-03-31T23:12:06.900 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.893+0000 7f946e99b740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55599a42d4a0 con 0x55599a534790 2024-03-31T23:12:06.900 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.893+0000 7f946e99b740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55599a537700 con 0x55599a52bbb0 2024-03-31T23:12:06.901 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.893+0000 7f946e99b740 10 client.0 _ll_register_callbacks cb 0x55599a430d90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T23:12:06.901 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.893+0000 7f9469a7d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55599a52b290 0x55599a531740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:06.901 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.893+0000 7f9469a7d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55599a52b290 0x55599a531740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60878/0 (socket says 172.21.15.46:60878) 2024-03-31T23:12:06.901 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.893+0000 7f9469a7d640 1 -- 172.21.15.46:0/1500265101 learned_addr learned my addr 172.21.15.46:0/1500265101 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:12:06.901 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.893+0000 7f946927c640 1 --2- 172.21.15.46:0/1500265101 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55599a52bbb0 0x55599a531c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:06.901 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.893+0000 7f946a27e640 1 --2- 172.21.15.46:0/1500265101 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55599a534790 0x55599a5321c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:06.901 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.893+0000 7f9469a7d640 1 -- 172.21.15.46:0/1500265101 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55599a52bbb0 msgr2=0x55599a531c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:12:06.901 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.893+0000 7f9469a7d640 1 --2- 172.21.15.46:0/1500265101 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55599a52bbb0 0x55599a531c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:06.901 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.893+0000 7f9469a7d640 1 -- 172.21.15.46:0/1500265101 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55599a534790 msgr2=0x55599a5321c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:12:06.901 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.893+0000 7f9469a7d640 1 --2- 172.21.15.46:0/1500265101 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55599a534790 0x55599a5321c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:06.901 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.893+0000 7f9469a7d640 1 -- 172.21.15.46:0/1500265101 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55599a26d7d0 con 0x55599a52b290 2024-03-31T23:12:06.901 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.893+0000 7f9469a7d640 1 --2- 172.21.15.46:0/1500265101 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55599a52b290 0x55599a531740 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f94600049d0 tx=0x7f9460004eb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=602f905ca4a0bb25 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:12:06.901 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.893+0000 7f944d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-31T23:12:06.901 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.893+0000 7f944d7fa640 1 -- 172.21.15.46:0/1500265101 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9460028020 con 0x55599a52b290 2024-03-31T23:12:06.902 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.893+0000 7f946e99b740 1 -- 172.21.15.46:0/1500265101 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9460003c30 con 0x55599a52b290 2024-03-31T23:12:06.902 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.893+0000 7f944d7fa640 1 -- 172.21.15.46:0/1500265101 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7f9460004050 con 0x55599a52b290 2024-03-31T23:12:06.902 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.893+0000 7f944d7fa640 1 -- 172.21.15.46:0/1500265101 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f9460033050 con 0x55599a52b290 2024-03-31T23:12:06.902 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.893+0000 7f946e99b740 10 client.15864 fetch_fsmap learned FSMap version 25 2024-03-31T23:12:06.902 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.893+0000 7f946e99b740 1 -- 172.21.15.46:0/1500265101 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55599a1a33f0 con 0x55599a52b290 2024-03-31T23:12:06.902 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.894+0000 7f944d7fa640 1 -- 172.21.15.46:0/1500265101 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9460003c30 con 0x55599a52b290 2024-03-31T23:12:06.902 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.894+0000 7f944d7fa640 1 -- 172.21.15.46:0/1500265101 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f946000fd20 con 0x55599a52b290 2024-03-31T23:12:06.902 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.894+0000 7f946e99b740 10 client.15864 fetch_fsmap finished waiting for FSMap version 25 2024-03-31T23:12:06.902 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.894+0000 7f946e99b740 10 client.15864 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-31T23:12:06.902 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.894+0000 7f946e99b740 20 client.15864 populate_metadata read hostname 'smithi046' 2024-03-31T23:12:06.902 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.894+0000 7f946e99b740 1 --2- 172.21.15.46:0/1500265101 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55599a544ab0 0x55599a546ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:06.902 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.894+0000 7f946e99b740 4 client.15864 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-31T23:12:06.902 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.894+0000 7f946e99b740 1 -- 172.21.15.46:0/1500265101 --> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55599a49c6c0 con 0x55599a544ab0 2024-03-31T23:12:06.902 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.894+0000 7f946927c640 1 --2- 172.21.15.46:0/1500265101 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55599a544ab0 0x55599a546ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:06.903 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.894+0000 7f946927c640 1 --2- 172.21.15.46:0/1500265101 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55599a544ab0 0x55599a546ea0 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fc48cbe9ad67848f server_cookie=b5d5ca0776310460 in_seq=0 out_seq=0 2024-03-31T23:12:06.903 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.894+0000 7f944d7fa640 10 client.15864 ms_handle_connect on v2:172.21.15.142:6838/2519143918 2024-03-31T23:12:06.903 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.895+0000 7f944d7fa640 1 -- 172.21.15.46:0/1500265101 <== mds.0 v2:172.21.15.142:6838/2519143918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55599a49c6c0 con 0x55599a544ab0 2024-03-31T23:12:06.903 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.895+0000 7f944d7fa640 10 client.15864 handle_command_reply: tid=0 2024-03-31T23:12:06.903 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.895+0000 7f946e99b740 1 client.15864 shutdown 2024-03-31T23:12:06.903 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.895+0000 7f946e99b740 1 -- 172.21.15.46:0/1500265101 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55599a52b290 msgr2=0x55599a531740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:12:06.903 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.895+0000 7f946e99b740 1 --2- 172.21.15.46:0/1500265101 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55599a52b290 0x55599a531740 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f94600049d0 tx=0x7f9460004eb0 comp rx=0 tx=0).stop 2024-03-31T23:12:06.903 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.895+0000 7f946e99b740 1 -- 172.21.15.46:0/1500265101 shutdown_connections 2024-03-31T23:12:06.903 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.895+0000 7f946e99b740 1 --2- 172.21.15.46:0/1500265101 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55599a544ab0 0x55599a546ea0 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:06.903 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.895+0000 7f946e99b740 1 --2- 172.21.15.46:0/1500265101 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55599a534790 0x55599a5321c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:06.903 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.895+0000 7f946e99b740 1 --2- 172.21.15.46:0/1500265101 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55599a52bbb0 0x55599a531c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:06.903 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.895+0000 7f944d7fa640 0 client.15864 ms_handle_reset on v2:172.21.15.142:6838/2519143918 2024-03-31T23:12:06.903 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.895+0000 7f946e99b740 1 --2- 172.21.15.46:0/1500265101 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55599a52b290 0x55599a531740 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:06.903 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.895+0000 7f946e99b740 1 -- 172.21.15.46:0/1500265101 >> 172.21.15.46:0/1500265101 conn(0x55599a5259c0 msgr2=0x55599a527710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:12:06.904 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.895+0000 7f946e99b740 1 -- 172.21.15.46:0/1500265101 shutdown_connections 2024-03-31T23:12:06.904 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.895+0000 7f946e99b740 1 -- 172.21.15.46:0/1500265101 wait complete. 2024-03-31T23:12:06.904 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.895+0000 7f946e99b740 20 client.15864 trim_cache size 0 max 16384 2024-03-31T23:12:06.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.913+0000 7f946e99b740 1 Processor -- start 2024-03-31T23:12:06.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.913+0000 7f946e99b740 1 -- start start 2024-03-31T23:12:06.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.913+0000 7f946e99b740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55599a52b290 0x55599a525da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:06.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.913+0000 7f946e99b740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55599a52bbb0 0x55599a540350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:06.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.913+0000 7f946e99b740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55599a534790 0x55599a530ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:06.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.913+0000 7f946e99b740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f94641b7a50 con 0x55599a52b290 2024-03-31T23:12:06.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.913+0000 7f946e99b740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f94641b5320 con 0x55599a534790 2024-03-31T23:12:06.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.913+0000 7f946e99b740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f94641b3170 con 0x55599a52bbb0 2024-03-31T23:12:06.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.913+0000 7f9469a7d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55599a52b290 0x55599a525da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:06.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.913+0000 7f9469a7d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55599a52b290 0x55599a525da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60882/0 (socket says 172.21.15.46:60882) 2024-03-31T23:12:06.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.913+0000 7f9469a7d640 1 -- 172.21.15.46:0/3837376573 learned_addr learned my addr 172.21.15.46:0/3837376573 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:12:06.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.913+0000 7f946927c640 1 --2- 172.21.15.46:0/3837376573 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55599a52bbb0 0x55599a540350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:06.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.913+0000 7f946a27e640 1 --2- 172.21.15.46:0/3837376573 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55599a534790 0x55599a530ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:06.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.913+0000 7f9469a7d640 1 -- 172.21.15.46:0/3837376573 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55599a52bbb0 msgr2=0x55599a540350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:12:06.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.913+0000 7f9469a7d640 1 --2- 172.21.15.46:0/3837376573 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55599a52bbb0 0x55599a540350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:06.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.913+0000 7f9469a7d640 1 -- 172.21.15.46:0/3837376573 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55599a534790 msgr2=0x55599a530ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:12:06.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.913+0000 7f9469a7d640 1 --2- 172.21.15.46:0/3837376573 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55599a534790 0x55599a530ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:06.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.913+0000 7f9469a7d640 1 -- 172.21.15.46:0/3837376573 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55599a1630f0 con 0x55599a52b290 2024-03-31T23:12:06.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.913+0000 7f9469a7d640 1 --2- 172.21.15.46:0/3837376573 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55599a52b290 0x55599a525da0 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f9460000f80 tx=0x7f9460004e20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ab8ba6d78cc13df2 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:12:06.921 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.913+0000 7f944f7fe640 1 -- 172.21.15.46:0/3837376573 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9460028070 con 0x55599a52b290 2024-03-31T23:12:06.921 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.914+0000 7f944f7fe640 1 -- 172.21.15.46:0/3837376573 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9460011df0 con 0x55599a52b290 2024-03-31T23:12:06.921 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.914+0000 7f944f7fe640 1 -- 172.21.15.46:0/3837376573 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f946001e040 con 0x55599a52b290 2024-03-31T23:12:06.921 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.914+0000 7f946e99b740 1 -- 172.21.15.46:0/3837376573 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55599a52b290 msgr2=0x55599a525da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:12:06.921 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.914+0000 7f946e99b740 1 --2- 172.21.15.46:0/3837376573 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55599a52b290 0x55599a525da0 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f9460000f80 tx=0x7f9460004e20 comp rx=0 tx=0).stop 2024-03-31T23:12:06.921 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.914+0000 7f946e99b740 1 -- 172.21.15.46:0/3837376573 shutdown_connections 2024-03-31T23:12:06.921 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.914+0000 7f946e99b740 1 --2- 172.21.15.46:0/3837376573 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55599a534790 0x55599a530ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:06.921 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.914+0000 7f946e99b740 1 --2- 172.21.15.46:0/3837376573 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55599a52bbb0 0x55599a540350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:06.921 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.914+0000 7f946e99b740 1 --2- 172.21.15.46:0/3837376573 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55599a52b290 0x55599a525da0 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:06.921 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.914+0000 7f946e99b740 1 -- 172.21.15.46:0/3837376573 >> 172.21.15.46:0/3837376573 conn(0x55599a5259c0 msgr2=0x55599a531700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:12:06.921 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.914+0000 7f946e99b740 1 -- 172.21.15.46:0/3837376573 shutdown_connections 2024-03-31T23:12:06.921 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.914+0000 7f946e99b740 1 -- 172.21.15.46:0/3837376573 wait complete. 2024-03-31T23:12:06.921 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.914+0000 7f946e99b740 1 -- start start 2024-03-31T23:12:06.921 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.914+0000 7f946e99b740 1 Processor -- start 2024-03-31T23:12:06.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.915+0000 7f946e99b740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55599a52b290 0x55599a537940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:06.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.915+0000 7f946e99b740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55599a52bbb0 0x55599a537e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:06.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.915+0000 7f946e99b740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55599a534790 0x55599a5383c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:06.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.915+0000 7f946e99b740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f94641b37e0 con 0x55599a52b290 2024-03-31T23:12:06.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.915+0000 7f946e99b740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f94641b4c50 con 0x55599a534790 2024-03-31T23:12:06.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.915+0000 7f946e99b740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f94641b3eb0 con 0x55599a52bbb0 2024-03-31T23:12:06.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.915+0000 7f946e99b740 10 client.0 _ll_register_callbacks cb 0x7f946001c070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T23:12:06.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.915+0000 7f9469a7d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55599a52b290 0x55599a537940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:06.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.915+0000 7f9469a7d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55599a52b290 0x55599a537940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60898/0 (socket says 172.21.15.46:60898) 2024-03-31T23:12:06.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.915+0000 7f946927c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55599a52bbb0 0x55599a537e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:06.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.915+0000 7f9469a7d640 1 -- 172.21.15.46:0/1818806352 learned_addr learned my addr 172.21.15.46:0/1818806352 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:12:06.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.915+0000 7f946a27e640 1 --2- 172.21.15.46:0/1818806352 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55599a534790 0x55599a5383c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:06.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.915+0000 7f9469a7d640 1 -- 172.21.15.46:0/1818806352 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55599a52bbb0 msgr2=0x55599a537e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:12:06.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.915+0000 7f9469a7d640 1 --2- 172.21.15.46:0/1818806352 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55599a52bbb0 0x55599a537e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:06.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.915+0000 7f9469a7d640 1 -- 172.21.15.46:0/1818806352 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55599a534790 msgr2=0x55599a5383c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:12:06.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.915+0000 7f9469a7d640 1 --2- 172.21.15.46:0/1818806352 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55599a534790 0x55599a5383c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:06.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.915+0000 7f9469a7d640 1 -- 172.21.15.46:0/1818806352 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55599a5a8f20 con 0x55599a52b290 2024-03-31T23:12:06.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.915+0000 7f946927c640 1 --2- 172.21.15.46:0/1818806352 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55599a52bbb0 0x55599a537e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:12:06.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.915+0000 7f9469a7d640 1 --2- 172.21.15.46:0/1818806352 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55599a52b290 0x55599a537940 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f94600099a0 tx=0x7f9460003c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e602bdbfe406e0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:12:06.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.915+0000 7f944d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-31T23:12:06.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.916+0000 7f944d7fa640 1 -- 172.21.15.46:0/1818806352 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9460028070 con 0x55599a52b290 2024-03-31T23:12:06.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.916+0000 7f944d7fa640 1 -- 172.21.15.46:0/1818806352 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7f9460004000 con 0x55599a52b290 2024-03-31T23:12:06.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.916+0000 7f946e99b740 1 -- 172.21.15.46:0/1818806352 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9460011df0 con 0x55599a52b290 2024-03-31T23:12:06.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.916+0000 7f944d7fa640 1 -- 172.21.15.46:0/1818806352 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f9460047050 con 0x55599a52b290 2024-03-31T23:12:06.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.916+0000 7f946e99b740 10 client.15870 fetch_fsmap learned FSMap version 25 2024-03-31T23:12:06.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.916+0000 7f946e99b740 1 -- 172.21.15.46:0/1818806352 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55599a5aa210 con 0x55599a52b290 2024-03-31T23:12:06.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.916+0000 7f944d7fa640 1 -- 172.21.15.46:0/1818806352 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9460011df0 con 0x55599a52b290 2024-03-31T23:12:06.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.916+0000 7f944d7fa640 1 -- 172.21.15.46:0/1818806352 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f946000f900 con 0x55599a52b290 2024-03-31T23:12:06.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.916+0000 7f946e99b740 10 client.15870 fetch_fsmap finished waiting for FSMap version 25 2024-03-31T23:12:06.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.916+0000 7f946e99b740 10 client.15870 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-31T23:12:06.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.916+0000 7f946e99b740 20 client.15870 populate_metadata read hostname 'smithi046' 2024-03-31T23:12:06.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.916+0000 7f946e99b740 1 --2- 172.21.15.46:0/1818806352 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55599a5aa920 0x55599a5acd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:06.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.916+0000 7f946e99b740 4 client.15870 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-31T23:12:06.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.916+0000 7f946e99b740 1 -- 172.21.15.46:0/1818806352 --> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55599a496a10 con 0x55599a5aa920 2024-03-31T23:12:06.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.916+0000 7f946927c640 1 --2- 172.21.15.46:0/1818806352 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55599a5aa920 0x55599a5acd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:06.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.917+0000 7f946927c640 1 --2- 172.21.15.46:0/1818806352 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55599a5aa920 0x55599a5acd10 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d9351087f0cfce9a server_cookie=4d95c2790fcf7828 in_seq=0 out_seq=0 2024-03-31T23:12:06.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.917+0000 7f944d7fa640 10 client.15870 ms_handle_connect on v2:172.21.15.142:6838/2519143918 2024-03-31T23:12:06.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.917+0000 7f944d7fa640 1 -- 172.21.15.46:0/1818806352 <== mds.0 v2:172.21.15.142:6838/2519143918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55599a496a10 con 0x55599a5aa920 2024-03-31T23:12:06.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.917+0000 7f944d7fa640 10 client.15870 handle_command_reply: tid=0 2024-03-31T23:12:06.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.917+0000 7f946e99b740 1 client.15870 shutdown 2024-03-31T23:12:06.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.917+0000 7f946e99b740 1 -- 172.21.15.46:0/1818806352 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55599a52b290 msgr2=0x55599a537940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:12:06.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.917+0000 7f946e99b740 1 --2- 172.21.15.46:0/1818806352 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55599a52b290 0x55599a537940 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f94600099a0 tx=0x7f9460003c20 comp rx=0 tx=0).stop 2024-03-31T23:12:06.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.917+0000 7f946e99b740 1 -- 172.21.15.46:0/1818806352 shutdown_connections 2024-03-31T23:12:06.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.917+0000 7f946e99b740 1 --2- 172.21.15.46:0/1818806352 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55599a5aa920 0x55599a5acd10 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:06.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.917+0000 7f946e99b740 1 --2- 172.21.15.46:0/1818806352 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55599a534790 0x55599a5383c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:06.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.917+0000 7f946e99b740 1 --2- 172.21.15.46:0/1818806352 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55599a52bbb0 0x55599a537e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:06.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.917+0000 7f944d7fa640 0 client.15870 ms_handle_reset on v2:172.21.15.142:6838/2519143918 2024-03-31T23:12:06.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.917+0000 7f946e99b740 1 --2- 172.21.15.46:0/1818806352 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55599a52b290 0x55599a537940 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:06.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.917+0000 7f946e99b740 1 -- 172.21.15.46:0/1818806352 >> 172.21.15.46:0/1818806352 conn(0x55599a5259c0 msgr2=0x55599a531ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:12:06.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.917+0000 7f946e99b740 1 -- 172.21.15.46:0/1818806352 shutdown_connections 2024-03-31T23:12:06.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.917+0000 7f946e99b740 1 -- 172.21.15.46:0/1818806352 wait complete. 2024-03-31T23:12:06.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.917+0000 7f946e99b740 20 client.15870 trim_cache size 0 max 16384 2024-03-31T23:12:06.926 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-03-31T23:12:06.926 INFO:teuthology.orchestra.run.smithi046.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-31T23:12:06.926 INFO:teuthology.orchestra.run.smithi046.stdout: "scrubs": { 2024-03-31T23:12:06.926 INFO:teuthology.orchestra.run.smithi046.stdout: "55ee9bbf-24db-4416-ae8b-3a205308a428": { 2024-03-31T23:12:06.926 INFO:teuthology.orchestra.run.smithi046.stdout: "path": "/", 2024-03-31T23:12:06.926 INFO:teuthology.orchestra.run.smithi046.stdout: "tag": "55ee9bbf-24db-4416-ae8b-3a205308a428", 2024-03-31T23:12:06.926 INFO:teuthology.orchestra.run.smithi046.stdout: "options": "recursive,force" 2024-03-31T23:12:06.926 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-31T23:12:06.926 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-31T23:12:06.926 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-03-31T23:12:06.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.918+0000 7f946bce1640 1 -- 172.21.15.46:0/2771138114 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f94500a2730 msgr2=0x7f94500a4bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:12:06.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.918+0000 7f946bce1640 1 --2- 172.21.15.46:0/2771138114 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f94500a2730 0x7f94500a4bf0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f945c010760 tx=0x7f945c005f90 comp rx=0 tx=0).stop 2024-03-31T23:12:06.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.918+0000 7f946bce1640 1 -- 172.21.15.46:0/2771138114 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f94641010d0 msgr2=0x7f946419eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:12:06.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.918+0000 7f946bce1640 1 --2- 172.21.15.46:0/2771138114 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f94641010d0 0x7f946419eb50 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f9454036040 tx=0x7f94540045e0 comp rx=0 tx=0).stop 2024-03-31T23:12:06.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.918+0000 7f946bce1640 1 -- 172.21.15.46:0/2771138114 shutdown_connections 2024-03-31T23:12:06.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.918+0000 7f946bce1640 1 --2- 172.21.15.46:0/2771138114 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f94500a2730 0x7f94500a4bf0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:06.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.918+0000 7f946bce1640 1 --2- 172.21.15.46:0/2771138114 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9464105bb0 0x7f94641a2ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:06.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.918+0000 7f946bce1640 1 --2- 172.21.15.46:0/2771138114 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f94641010d0 0x7f946419eb50 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:06.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.918+0000 7f946bce1640 1 --2- 172.21.15.46:0/2771138114 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f94641007b0 0x7f946419e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:06.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.918+0000 7f946bce1640 1 -- 172.21.15.46:0/2771138114 >> 172.21.15.46:0/2771138114 conn(0x7f94640777b0 msgr2=0x7f9464103560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:12:06.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.918+0000 7f946bce1640 1 -- 172.21.15.46:0/2771138114 shutdown_connections 2024-03-31T23:12:06.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:06.918+0000 7f946bce1640 1 -- 172.21.15.46:0/2771138114 wait complete. 2024-03-31T23:12:06.929 INFO:tasks.cephfs.filesystem:scrub status for tag:55ee9bbf-24db-4416-ae8b-3a205308a428 - {'path': '/', 'tag': '55ee9bbf-24db-4416-ae8b-3a205308a428', 'options': 'recursive,force'} 2024-03-31T23:12:07.612 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:12:07 smithi046 ceph-mon[32498]: scrub queued for path: / 2024-03-31T23:12:07.612 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:12:07 smithi046 ceph-mon[32498]: scrub summary: idle+waiting paths [/] 2024-03-31T23:12:07.612 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:12:07 smithi046 ceph-mon[32498]: scrub summary: active paths [/] 2024-03-31T23:12:07.612 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:12:07 smithi046 ceph-mon[32498]: scrub summary: idle+waiting paths [/] 2024-03-31T23:12:07.612 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:12:07 smithi046 ceph-mon[32498]: scrub summary: idle 2024-03-31T23:12:07.693 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:12:07 smithi142 ceph-mon[35271]: scrub queued for path: / 2024-03-31T23:12:07.693 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:12:07 smithi142 ceph-mon[35271]: scrub summary: idle+waiting paths [/] 2024-03-31T23:12:07.693 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:12:07 smithi142 ceph-mon[35271]: scrub summary: active paths [/] 2024-03-31T23:12:07.694 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:12:07 smithi142 ceph-mon[35271]: scrub summary: idle+waiting paths [/] 2024-03-31T23:12:07.694 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:12:07 smithi142 ceph-mon[35271]: scrub summary: idle 2024-03-31T23:12:07.795 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:12:07 smithi162 ceph-mon[36237]: scrub queued for path: / 2024-03-31T23:12:07.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:12:07 smithi162 ceph-mon[36237]: scrub summary: idle+waiting paths [/] 2024-03-31T23:12:07.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:12:07 smithi162 ceph-mon[36237]: scrub summary: active paths [/] 2024-03-31T23:12:07.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:12:07 smithi162 ceph-mon[36237]: scrub summary: idle+waiting paths [/] 2024-03-31T23:12:07.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:12:07 smithi162 ceph-mon[36237]: scrub summary: idle 2024-03-31T23:12:08.612 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:12:08 smithi046 ceph-mon[32498]: pgmap v15: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 24 KiB/s wr, 25 op/s 2024-03-31T23:12:08.693 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:12:08 smithi142 ceph-mon[35271]: pgmap v15: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 24 KiB/s wr, 25 op/s 2024-03-31T23:12:08.795 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:12:08 smithi162 ceph-mon[36237]: pgmap v15: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 24 KiB/s wr, 25 op/s 2024-03-31T23:12:10.611 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:12:10 smithi046 ceph-mon[32498]: pgmap v16: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 24 KiB/s wr, 28 op/s 2024-03-31T23:12:10.693 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:12:10 smithi142 ceph-mon[35271]: pgmap v16: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 24 KiB/s wr, 28 op/s 2024-03-31T23:12:10.795 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:12:10 smithi162 ceph-mon[36237]: pgmap v16: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 24 KiB/s wr, 28 op/s 2024-03-31T23:12:11.304 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:11.289+0000 7faf14be9640 1 -- 172.21.15.46:0/1046309522 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faf101077c0 msgr2=0x7faf10107ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:12:11.304 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:11.289+0000 7faf14be9640 1 --2- 172.21.15.46:0/1046309522 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faf101077c0 0x7faf10107ba0 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7faf00009900 tx=0x7faf0002bc30 comp rx=0 tx=0).stop 2024-03-31T23:12:11.304 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:11.289+0000 7faf14be9640 1 -- 172.21.15.46:0/1046309522 shutdown_connections 2024-03-31T23:12:11.304 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:11.289+0000 7faf14be9640 1 --2- 172.21.15.46:0/1046309522 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7faf10069770 0x7faf1010e5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:11.304 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:11.289+0000 7faf14be9640 1 --2- 172.21.15.46:0/1046309522 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7faf10068dd0 0x7faf10069230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:11.304 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:11.289+0000 7faf14be9640 1 --2- 172.21.15.46:0/1046309522 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faf101077c0 0x7faf10107ba0 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:11.304 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:11.289+0000 7faf14be9640 1 -- 172.21.15.46:0/1046309522 >> 172.21.15.46:0/1046309522 conn(0x7faf100774a0 msgr2=0x7faf100778b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:12:11.304 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:11.289+0000 7faf14be9640 1 -- 172.21.15.46:0/1046309522 shutdown_connections 2024-03-31T23:12:11.304 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:11.289+0000 7faf14be9640 1 -- 172.21.15.46:0/1046309522 wait complete. 2024-03-31T23:12:11.305 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:11.289+0000 7faf14be9640 1 Processor -- start 2024-03-31T23:12:11.305 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:11.290+0000 7faf14be9640 1 -- start start 2024-03-31T23:12:11.305 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:11.290+0000 7faf14be9640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7faf10068dd0 0x7faf10197ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:11.305 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:11.290+0000 7faf14be9640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faf10069770 0x7faf10198400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:11.305 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:11.290+0000 7faf14be9640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7faf101077c0 0x7faf1019c780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:11.305 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:11.290+0000 7faf14be9640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7faf101119a0 con 0x7faf10069770 2024-03-31T23:12:11.305 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:11.290+0000 7faf14be9640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7faf10111820 con 0x7faf101077c0 2024-03-31T23:12:11.305 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:11.290+0000 7faf14be9640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7faf10111b20 con 0x7faf10068dd0 2024-03-31T23:12:11.305 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:11.290+0000 7faf07fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faf10069770 0x7faf10198400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:11.305 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:11.290+0000 7faf07fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faf10069770 0x7faf10198400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60916/0 (socket says 172.21.15.46:60916) 2024-03-31T23:12:11.305 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:11.290+0000 7faf07fff640 1 -- 172.21.15.46:0/2094506435 learned_addr learned my addr 172.21.15.46:0/2094506435 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:12:11.305 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:11.290+0000 7faf0ed9d640 1 --2- 172.21.15.46:0/2094506435 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7faf101077c0 0x7faf1019c780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:11.305 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:11.290+0000 7faf0e59c640 1 --2- 172.21.15.46:0/2094506435 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7faf10068dd0 0x7faf10197ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:11.306 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:11.290+0000 7faf07fff640 1 -- 172.21.15.46:0/2094506435 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7faf10068dd0 msgr2=0x7faf10197ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:12:11.306 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:11.290+0000 7faf07fff640 1 --2- 172.21.15.46:0/2094506435 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7faf10068dd0 0x7faf10197ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:11.306 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:11.290+0000 7faf07fff640 1 -- 172.21.15.46:0/2094506435 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7faf101077c0 msgr2=0x7faf1019c780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:12:11.306 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:11.290+0000 7faf07fff640 1 --2- 172.21.15.46:0/2094506435 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7faf101077c0 0x7faf1019c780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:11.306 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:11.290+0000 7faf07fff640 1 -- 172.21.15.46:0/2094506435 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faf1019ceb0 con 0x7faf10069770 2024-03-31T23:12:11.306 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:11.290+0000 7faf07fff640 1 --2- 172.21.15.46:0/2094506435 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faf10069770 0x7faf10198400 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7faef800e9f0 tx=0x7faef800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:12:11.306 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:11.290+0000 7faf077fe640 1 -- 172.21.15.46:0/2094506435 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faef800cd80 con 0x7faf10069770 2024-03-31T23:12:11.306 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:11.291+0000 7faf077fe640 1 -- 172.21.15.46:0/2094506435 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7faef80108e0 con 0x7faf10069770 2024-03-31T23:12:11.306 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:11.291+0000 7faf14be9640 1 -- 172.21.15.46:0/2094506435 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faf1019d1a0 con 0x7faf10069770 2024-03-31T23:12:11.306 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:11.291+0000 7faf077fe640 1 -- 172.21.15.46:0/2094506435 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faef8018920 con 0x7faf10069770 2024-03-31T23:12:11.306 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:11.291+0000 7faf14be9640 1 -- 172.21.15.46:0/2094506435 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faf101a4a30 con 0x7faf10069770 2024-03-31T23:12:11.306 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:11.293+0000 7faf077fe640 1 -- 172.21.15.46:0/2094506435 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 24) ==== 137553+0+0 (secure 0 0 0) 0x7faef8010a90 con 0x7faf10069770 2024-03-31T23:12:11.306 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:11.293+0000 7faf177d4740 1 Processor -- start 2024-03-31T23:12:11.306 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:11.293+0000 7faf177d4740 1 -- start start 2024-03-31T23:12:11.307 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:11.293+0000 7faf077fe640 1 --2- 172.21.15.46:0/2094506435 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7faefc0a27d0 0x7faefc0a4c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:11.307 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:11.294+0000 7faf177d4740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5600850a96c0 0x5600850a9aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:11.307 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:11.294+0000 7faf177d4740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5600850a9fe0 0x5600850b2870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:11.307 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:11.294+0000 7faf177d4740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5600850b2db0 0x5600850b51a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:11.307 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:11.294+0000 7faf177d4740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x56008500c8f0 con 0x5600850a9fe0 2024-03-31T23:12:11.307 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:11.294+0000 7faf177d4740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x560084fb58d0 con 0x5600850b2db0 2024-03-31T23:12:11.307 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:11.294+0000 7faf177d4740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x560085008990 con 0x5600850a96c0 2024-03-31T23:12:11.307 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:11.294+0000 7faf077fe640 1 -- 172.21.15.46:0/2094506435 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7faef80cf470 con 0x7faf10069770 2024-03-31T23:12:11.307 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:11.294+0000 7faf0ed9d640 1 --2- 172.21.15.46:0/2094506435 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7faefc0a27d0 0x7faefc0a4c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:11.307 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:11.294+0000 7faf0e59c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5600850a96c0 0x5600850a9aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:11.307 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:11.294+0000 7faf07fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5600850a9fe0 0x5600850b2870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:11.307 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:11.294+0000 7faf0e59c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5600850a96c0 0x5600850a9aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.46:33664/0 (socket says 172.21.15.46:33664) 2024-03-31T23:12:11.307 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:11.294+0000 7faf07fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5600850a9fe0 0x5600850b2870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60924/0 (socket says 172.21.15.46:60924) 2024-03-31T23:12:11.307 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:11.294+0000 7faf0e59c640 1 -- 172.21.15.46:0/1597852825 learned_addr learned my addr 172.21.15.46:0/1597852825 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:12:11.308 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:11.294+0000 7faf0ed9d640 1 --2- 172.21.15.46:0/1597852825 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5600850b2db0 0x5600850b51a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:11.308 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:11.294+0000 7faf0e59c640 1 -- 172.21.15.46:0/1597852825 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5600850b2db0 msgr2=0x5600850b51a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:12:11.308 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:11.294+0000 7faf0e59c640 1 --2- 172.21.15.46:0/1597852825 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5600850b2db0 0x5600850b51a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:11.308 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:11.294+0000 7faf0e59c640 1 -- 172.21.15.46:0/1597852825 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5600850a9fe0 msgr2=0x5600850b2870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:12:11.308 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:11.294+0000 7faf0e59c640 1 --2- 172.21.15.46:0/1597852825 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5600850a9fe0 0x5600850b2870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:11.308 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:11.294+0000 7faf0e59c640 1 -- 172.21.15.46:0/1597852825 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560084d21210 con 0x5600850a96c0 2024-03-31T23:12:11.308 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:11.294+0000 7faf0ed9d640 1 --2- 172.21.15.46:0/2094506435 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7faefc0a27d0 0x7faefc0a4c90 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7faf08005fd0 tx=0x7faf08013b40 comp rx=0 tx=0).ready entity=mgr.15654 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:12:11.308 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:11.294+0000 7faf0e59c640 1 --2- 172.21.15.46:0/1597852825 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5600850a96c0 0x5600850a9aa0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7faf00000c00 tx=0x7faf000095e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8c17837c871712bb server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:12:11.308 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:11.294+0000 7faf04ff9640 1 -- 172.21.15.46:0/1597852825 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf00045020 con 0x5600850a96c0 2024-03-31T23:12:11.308 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:11.294+0000 7faf04ff9640 1 -- 172.21.15.46:0/1597852825 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7faf0002c7e0 con 0x5600850a96c0 2024-03-31T23:12:11.308 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:11.294+0000 7faf04ff9640 1 -- 172.21.15.46:0/1597852825 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf000053b0 con 0x5600850a96c0 2024-03-31T23:12:11.308 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:11.295+0000 7faf177d4740 1 -- 172.21.15.46:0/1597852825 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5600850a96c0 msgr2=0x5600850a9aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:12:11.308 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:11.295+0000 7faf177d4740 1 --2- 172.21.15.46:0/1597852825 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5600850a96c0 0x5600850a9aa0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7faf00000c00 tx=0x7faf000095e0 comp rx=0 tx=0).stop 2024-03-31T23:12:11.309 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:11.295+0000 7faf177d4740 1 -- 172.21.15.46:0/1597852825 shutdown_connections 2024-03-31T23:12:11.309 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:11.295+0000 7faf177d4740 1 --2- 172.21.15.46:0/1597852825 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5600850b2db0 0x5600850b51a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:11.309 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:11.295+0000 7faf177d4740 1 --2- 172.21.15.46:0/1597852825 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5600850a9fe0 0x5600850b2870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:11.309 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:11.295+0000 7faf177d4740 1 --2- 172.21.15.46:0/1597852825 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5600850a96c0 0x5600850a9aa0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:11.309 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:11.295+0000 7faf177d4740 1 -- 172.21.15.46:0/1597852825 >> 172.21.15.46:0/1597852825 conn(0x5600850a3fe0 msgr2=0x5600850a43c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:12:11.309 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:11.295+0000 7faf177d4740 1 -- 172.21.15.46:0/1597852825 shutdown_connections 2024-03-31T23:12:11.309 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:11.295+0000 7faf177d4740 1 -- 172.21.15.46:0/1597852825 wait complete. 2024-03-31T23:12:11.309 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:11.296+0000 7faf177d4740 1 -- start start 2024-03-31T23:12:11.309 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:11.296+0000 7faf177d4740 1 Processor -- start 2024-03-31T23:12:11.309 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:11.296+0000 7faf177d4740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5600850a96c0 0x5600850afd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:11.309 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:11.296+0000 7faf177d4740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5600850a9fe0 0x5600850b02b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:11.309 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:11.296+0000 7faf177d4740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5600850b2db0 0x5600850b07f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:11.309 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:11.296+0000 7faf177d4740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x5600850b9d90 con 0x5600850a9fe0 2024-03-31T23:12:11.309 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:11.296+0000 7faf177d4740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x560084faaf80 con 0x5600850a96c0 2024-03-31T23:12:11.309 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:11.296+0000 7faf177d4740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5600850b9f10 con 0x5600850b2db0 2024-03-31T23:12:11.310 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:11.296+0000 7faf177d4740 10 client.0 _ll_register_callbacks cb 0x560085003290 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T23:12:11.310 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:11.296+0000 7faf07fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5600850a9fe0 0x5600850b02b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:11.310 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:11.296+0000 7faf07fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5600850a9fe0 0x5600850b02b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60930/0 (socket says 172.21.15.46:60930) 2024-03-31T23:12:11.310 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:11.296+0000 7faf0ed9d640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5600850b2db0 0x5600850b07f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:11.310 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:11.296+0000 7faf07fff640 1 -- 172.21.15.46:0/2310465920 learned_addr learned my addr 172.21.15.46:0/2310465920 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:12:11.310 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:11.296+0000 7faf0e59c640 1 --2- 172.21.15.46:0/2310465920 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5600850a96c0 0x5600850afd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:11.310 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:11.296+0000 7faf07fff640 1 -- 172.21.15.46:0/2310465920 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5600850b2db0 msgr2=0x5600850b07f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:12:11.310 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:11.296+0000 7faf07fff640 1 --2- 172.21.15.46:0/2310465920 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5600850b2db0 0x5600850b07f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:11.310 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:11.296+0000 7faf07fff640 1 -- 172.21.15.46:0/2310465920 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5600850a96c0 msgr2=0x5600850afd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:12:11.310 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:11.296+0000 7faf07fff640 1 --2- 172.21.15.46:0/2310465920 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5600850a96c0 0x5600850afd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:11.310 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:11.296+0000 7faf07fff640 1 -- 172.21.15.46:0/2310465920 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560084ded620 con 0x5600850a9fe0 2024-03-31T23:12:11.310 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:11.296+0000 7faf0ed9d640 1 --2- 172.21.15.46:0/2310465920 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5600850b2db0 0x5600850b07f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:12:11.310 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:11.296+0000 7faf0e59c640 1 --2- 172.21.15.46:0/2310465920 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5600850a96c0 0x5600850afd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:12:11.311 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:11.297+0000 7faf07fff640 1 --2- 172.21.15.46:0/2310465920 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5600850a9fe0 0x5600850b02b0 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7faef8009380 tx=0x7faef809b0c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca437ee781768561 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:12:11.311 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:11.297+0000 7faef5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-31T23:12:11.311 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:11.297+0000 7faef5ffb640 1 -- 172.21.15.46:0/2310465920 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faef80a3d70 con 0x5600850a9fe0 2024-03-31T23:12:11.311 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:11.297+0000 7faef5ffb640 1 -- 172.21.15.46:0/2310465920 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7faef80cc970 con 0x5600850a9fe0 2024-03-31T23:12:11.311 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:11.297+0000 7faf177d4740 1 -- 172.21.15.46:0/2310465920 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7faf0002c7e0 con 0x5600850a9fe0 2024-03-31T23:12:11.311 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:11.297+0000 7faef5ffb640 1 -- 172.21.15.46:0/2310465920 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7faef80a4000 con 0x5600850a9fe0 2024-03-31T23:12:11.311 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:11.297+0000 7faf177d4740 10 client.15882 fetch_fsmap learned FSMap version 25 2024-03-31T23:12:11.311 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:11.297+0000 7faf177d4740 1 -- 172.21.15.46:0/2310465920 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x5600850c5c50 con 0x5600850a9fe0 2024-03-31T23:12:11.311 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:11.297+0000 7faef5ffb640 1 -- 172.21.15.46:0/2310465920 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7faf0002c7e0 con 0x5600850a9fe0 2024-03-31T23:12:11.311 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:11.297+0000 7faef5ffb640 1 -- 172.21.15.46:0/2310465920 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7faef809aaf0 con 0x5600850a9fe0 2024-03-31T23:12:11.311 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:11.297+0000 7faf177d4740 10 client.15882 fetch_fsmap finished waiting for FSMap version 25 2024-03-31T23:12:11.311 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:11.297+0000 7faf177d4740 10 client.15882 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-31T23:12:11.311 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:11.297+0000 7faf177d4740 20 client.15882 populate_metadata read hostname 'smithi046' 2024-03-31T23:12:11.311 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:11.297+0000 7faf177d4740 1 --2- 172.21.15.46:0/2310465920 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x5600850c7160 0x5600850c9550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:11.312 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:11.297+0000 7faf177d4740 4 client.15882 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-31T23:12:11.312 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:11.297+0000 7faf177d4740 1 -- 172.21.15.46:0/2310465920 --> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560085020120 con 0x5600850c7160 2024-03-31T23:12:11.312 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:11.298+0000 7faf0e59c640 1 --2- 172.21.15.46:0/2310465920 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x5600850c7160 0x5600850c9550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:11.312 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:11.298+0000 7faef5ffb640 10 client.15882 ms_handle_connect on v2:172.21.15.142:6838/2519143918 2024-03-31T23:12:11.312 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:11.298+0000 7faf0e59c640 1 --2- 172.21.15.46:0/2310465920 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x5600850c7160 0x5600850c9550 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=538e72590bb1f508 server_cookie=32924fff7d3f6e5 in_seq=0 out_seq=0 2024-03-31T23:12:11.312 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:11.298+0000 7faef5ffb640 1 -- 172.21.15.46:0/2310465920 <== mds.0 v2:172.21.15.142:6838/2519143918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x560085020120 con 0x5600850c7160 2024-03-31T23:12:11.312 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:11.298+0000 7faef5ffb640 10 client.15882 handle_command_reply: tid=0 2024-03-31T23:12:11.312 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:11.299+0000 7faf177d4740 1 client.15882 shutdown 2024-03-31T23:12:11.312 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:11.299+0000 7faf177d4740 1 -- 172.21.15.46:0/2310465920 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5600850a9fe0 msgr2=0x5600850b02b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:12:11.312 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:11.299+0000 7faf177d4740 1 --2- 172.21.15.46:0/2310465920 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5600850a9fe0 0x5600850b02b0 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7faef8009380 tx=0x7faef809b0c0 comp rx=0 tx=0).stop 2024-03-31T23:12:11.312 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:11.299+0000 7faf177d4740 1 -- 172.21.15.46:0/2310465920 shutdown_connections 2024-03-31T23:12:11.312 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:11.299+0000 7faf177d4740 1 --2- 172.21.15.46:0/2310465920 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x5600850c7160 0x5600850c9550 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:11.312 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:11.299+0000 7faf177d4740 1 --2- 172.21.15.46:0/2310465920 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5600850b2db0 0x5600850b07f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:11.312 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:11.299+0000 7faf177d4740 1 --2- 172.21.15.46:0/2310465920 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5600850a9fe0 0x5600850b02b0 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:11.313 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:11.299+0000 7faef5ffb640 0 client.15882 ms_handle_reset on v2:172.21.15.142:6838/2519143918 2024-03-31T23:12:11.313 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:11.299+0000 7faf177d4740 1 --2- 172.21.15.46:0/2310465920 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5600850a96c0 0x5600850afd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:11.313 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:11.299+0000 7faf177d4740 1 -- 172.21.15.46:0/2310465920 >> 172.21.15.46:0/2310465920 conn(0x5600850a3fe0 msgr2=0x5600850a5d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:12:11.313 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:11.299+0000 7faf177d4740 1 -- 172.21.15.46:0/2310465920 shutdown_connections 2024-03-31T23:12:11.313 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:11.299+0000 7faf177d4740 1 -- 172.21.15.46:0/2310465920 wait complete. 2024-03-31T23:12:11.313 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:11.299+0000 7faf177d4740 20 client.15882 trim_cache size 0 max 16384 2024-03-31T23:12:11.320 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:11.314+0000 7faf177d4740 1 Processor -- start 2024-03-31T23:12:11.320 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:11.314+0000 7faf177d4740 1 -- start start 2024-03-31T23:12:11.320 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:11.314+0000 7faf177d4740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5600850a96c0 0x5600850b24a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:11.320 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:11.314+0000 7faf177d4740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5600850a9fe0 0x5600850c2b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:11.320 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:11.314+0000 7faf177d4740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5600850b2db0 0x5600850a43c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:11.320 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:11.314+0000 7faf177d4740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7faf101af0d0 con 0x5600850a9fe0 2024-03-31T23:12:11.320 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:11.314+0000 7faf177d4740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7faf101ac9a0 con 0x5600850a96c0 2024-03-31T23:12:11.320 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:11.314+0000 7faf177d4740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7faf101aa7f0 con 0x5600850b2db0 2024-03-31T23:12:11.321 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:11.314+0000 7faf07fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5600850a9fe0 0x5600850c2b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:11.321 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:11.314+0000 7faf07fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5600850a9fe0 0x5600850c2b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60938/0 (socket says 172.21.15.46:60938) 2024-03-31T23:12:11.321 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:11.314+0000 7faf07fff640 1 -- 172.21.15.46:0/791588108 learned_addr learned my addr 172.21.15.46:0/791588108 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:12:11.321 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:11.314+0000 7faf0e59c640 1 --2- 172.21.15.46:0/791588108 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5600850a96c0 0x5600850b24a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:11.321 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:11.314+0000 7faf0ed9d640 1 --2- 172.21.15.46:0/791588108 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5600850b2db0 0x5600850a43c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:11.321 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:11.315+0000 7faf0e59c640 1 -- 172.21.15.46:0/791588108 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5600850b2db0 msgr2=0x5600850a43c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:12:11.321 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:11.315+0000 7faf0e59c640 1 --2- 172.21.15.46:0/791588108 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5600850b2db0 0x5600850a43c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:11.321 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:11.315+0000 7faf0e59c640 1 -- 172.21.15.46:0/791588108 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5600850a9fe0 msgr2=0x5600850c2b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:12:11.321 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:11.315+0000 7faf0e59c640 1 --2- 172.21.15.46:0/791588108 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5600850a9fe0 0x5600850c2b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:11.321 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:11.315+0000 7faf0e59c640 1 -- 172.21.15.46:0/791588108 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5600850aed20 con 0x5600850a96c0 2024-03-31T23:12:11.321 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:11.315+0000 7faf07fff640 1 --2- 172.21.15.46:0/791588108 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5600850a9fe0 0x5600850c2b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:12:11.321 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:11.315+0000 7faf0e59c640 1 --2- 172.21.15.46:0/791588108 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5600850a96c0 0x5600850b24a0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7faf0002c170 tx=0x7faf0002bc30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bf7ae1783ae8a7ef server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:12:11.321 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:11.315+0000 7faef7fff640 1 -- 172.21.15.46:0/791588108 <== mon.2 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf00045040 con 0x5600850a96c0 2024-03-31T23:12:11.321 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:11.315+0000 7faef7fff640 1 -- 172.21.15.46:0/791588108 <== mon.2 v2:172.21.15.142:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7faf0002c5d0 con 0x5600850a96c0 2024-03-31T23:12:11.321 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:11.315+0000 7faef7fff640 1 -- 172.21.15.46:0/791588108 <== mon.2 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf0004f740 con 0x5600850a96c0 2024-03-31T23:12:11.322 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:11.316+0000 7faf177d4740 1 -- 172.21.15.46:0/791588108 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5600850a96c0 msgr2=0x5600850b24a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:12:11.322 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:11.316+0000 7faf177d4740 1 --2- 172.21.15.46:0/791588108 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5600850a96c0 0x5600850b24a0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7faf0002c170 tx=0x7faf0002bc30 comp rx=0 tx=0).stop 2024-03-31T23:12:11.322 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:11.316+0000 7faf177d4740 1 -- 172.21.15.46:0/791588108 shutdown_connections 2024-03-31T23:12:11.322 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:11.316+0000 7faf177d4740 1 --2- 172.21.15.46:0/791588108 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5600850b2db0 0x5600850a43c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:11.322 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:11.316+0000 7faf177d4740 1 --2- 172.21.15.46:0/791588108 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5600850a9fe0 0x5600850c2b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:11.322 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:11.316+0000 7faf177d4740 1 --2- 172.21.15.46:0/791588108 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5600850a96c0 0x5600850b24a0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:11.322 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:11.316+0000 7faf177d4740 1 -- 172.21.15.46:0/791588108 >> 172.21.15.46:0/791588108 conn(0x5600850a3fe0 msgr2=0x5600850a2f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:12:11.322 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:11.316+0000 7faf177d4740 1 -- 172.21.15.46:0/791588108 shutdown_connections 2024-03-31T23:12:11.322 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:11.316+0000 7faf177d4740 1 -- 172.21.15.46:0/791588108 wait complete. 2024-03-31T23:12:11.322 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:11.316+0000 7faf177d4740 1 -- start start 2024-03-31T23:12:11.322 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:11.316+0000 7faf177d4740 1 Processor -- start 2024-03-31T23:12:11.322 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:11.316+0000 7faf177d4740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5600850a96c0 0x5600850acee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:11.323 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:11.316+0000 7faf177d4740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5600850a9fe0 0x5600850ad420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:11.323 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:11.316+0000 7faf177d4740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5600850b2db0 0x56008512c470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:11.323 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:11.316+0000 7faf177d4740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7faf101aae60 con 0x5600850a9fe0 2024-03-31T23:12:11.323 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:11.316+0000 7faf177d4740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7faf101ac2d0 con 0x5600850a96c0 2024-03-31T23:12:11.323 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:11.316+0000 7faf177d4740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7faf101ab530 con 0x5600850b2db0 2024-03-31T23:12:11.323 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:11.316+0000 7faf177d4740 10 client.0 _ll_register_callbacks cb 0x7faef80860b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T23:12:11.323 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:11.316+0000 7faf07fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5600850a9fe0 0x5600850ad420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:11.323 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:11.317+0000 7faf0e59c640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5600850a96c0 0x5600850acee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:11.323 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:11.317+0000 7faf07fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5600850a9fe0 0x5600850ad420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60940/0 (socket says 172.21.15.46:60940) 2024-03-31T23:12:11.323 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:11.317+0000 7faf07fff640 1 -- 172.21.15.46:0/3367743078 learned_addr learned my addr 172.21.15.46:0/3367743078 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:12:11.323 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:11.317+0000 7faf0ed9d640 1 --2- 172.21.15.46:0/3367743078 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5600850b2db0 0x56008512c470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:11.323 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:11.317+0000 7faf0e59c640 1 -- 172.21.15.46:0/3367743078 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5600850b2db0 msgr2=0x56008512c470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:12:11.323 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:11.317+0000 7faf0e59c640 1 --2- 172.21.15.46:0/3367743078 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5600850b2db0 0x56008512c470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:11.323 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:11.317+0000 7faf0e59c640 1 -- 172.21.15.46:0/3367743078 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5600850a9fe0 msgr2=0x5600850ad420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:12:11.324 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:11.317+0000 7faf0e59c640 1 --2- 172.21.15.46:0/3367743078 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5600850a9fe0 0x5600850ad420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:11.324 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:11.317+0000 7faf0e59c640 1 -- 172.21.15.46:0/3367743078 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56008512c9b0 con 0x5600850a96c0 2024-03-31T23:12:11.324 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:11.317+0000 7faf0e59c640 1 --2- 172.21.15.46:0/3367743078 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5600850a96c0 0x5600850acee0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7faf0002c170 tx=0x7faf000040c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5734ca4eb8ca47e3 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:12:11.324 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:11.317+0000 7faef5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.142:3300/0 2024-03-31T23:12:11.324 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:11.317+0000 7faef5ffb640 1 -- 172.21.15.46:0/3367743078 <== mon.2 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf00045070 con 0x5600850a96c0 2024-03-31T23:12:11.324 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:11.317+0000 7faf177d4740 1 -- 172.21.15.46:0/3367743078 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7faf0002c5d0 con 0x5600850a96c0 2024-03-31T23:12:11.324 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:11.317+0000 7faef5ffb640 1 -- 172.21.15.46:0/3367743078 <== mon.2 v2:172.21.15.142:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7faf0004f4c0 con 0x5600850a96c0 2024-03-31T23:12:11.324 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:11.317+0000 7faef5ffb640 1 -- 172.21.15.46:0/3367743078 <== mon.2 v2:172.21.15.142:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7faf00061050 con 0x5600850a96c0 2024-03-31T23:12:11.324 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:11.317+0000 7faf177d4740 10 client.25730 fetch_fsmap learned FSMap version 25 2024-03-31T23:12:11.324 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:11.317+0000 7faf177d4740 1 -- 172.21.15.46:0/3367743078 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x56008512db60 con 0x5600850a96c0 2024-03-31T23:12:11.324 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:11.318+0000 7faef5ffb640 1 -- 172.21.15.46:0/3367743078 <== mon.2 v2:172.21.15.142:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7faf0002c5d0 con 0x5600850a96c0 2024-03-31T23:12:11.324 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:11.318+0000 7faef5ffb640 1 -- 172.21.15.46:0/3367743078 <== mon.2 v2:172.21.15.142:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7faf00004730 con 0x5600850a96c0 2024-03-31T23:12:11.324 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:11.318+0000 7faf177d4740 10 client.25730 fetch_fsmap finished waiting for FSMap version 25 2024-03-31T23:12:11.325 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:11.318+0000 7faf177d4740 10 client.25730 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-31T23:12:11.325 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:11.318+0000 7faf177d4740 20 client.25730 populate_metadata read hostname 'smithi046' 2024-03-31T23:12:11.325 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:11.318+0000 7faf177d4740 1 --2- 172.21.15.46:0/3367743078 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x56008512ddf0 0x5600851301e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:11.325 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:11.318+0000 7faf177d4740 4 client.25730 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-31T23:12:11.325 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:11.318+0000 7faf177d4740 1 -- 172.21.15.46:0/3367743078 --> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] -- command(tid 0: {"prefix": "flush journal"}) -- 0x560085019e30 con 0x56008512ddf0 2024-03-31T23:12:11.325 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:11.318+0000 7faf07fff640 1 --2- 172.21.15.46:0/3367743078 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x56008512ddf0 0x5600851301e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:11.325 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:11.318+0000 7faf07fff640 1 --2- 172.21.15.46:0/3367743078 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x56008512ddf0 0x5600851301e0 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a5d453b8163a127a server_cookie=387b984312a5099a in_seq=0 out_seq=0 2024-03-31T23:12:11.325 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:11.318+0000 7faef5ffb640 10 client.25730 ms_handle_connect on v2:172.21.15.142:6838/2519143918 2024-03-31T23:12:11.326 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:11.324+0000 7faef5ffb640 1 -- 172.21.15.46:0/3367743078 <== mds.0 v2:172.21.15.142:6838/2519143918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x560085019e30 con 0x56008512ddf0 2024-03-31T23:12:11.326 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:11.324+0000 7faef5ffb640 10 client.25730 handle_command_reply: tid=0 2024-03-31T23:12:11.327 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:11.324+0000 7faf177d4740 1 client.25730 shutdown 2024-03-31T23:12:11.327 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:11.324+0000 7faf177d4740 1 -- 172.21.15.46:0/3367743078 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5600850a96c0 msgr2=0x5600850acee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:12:11.327 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:11.324+0000 7faf177d4740 1 --2- 172.21.15.46:0/3367743078 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5600850a96c0 0x5600850acee0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7faf0002c170 tx=0x7faf000040c0 comp rx=0 tx=0).stop 2024-03-31T23:12:11.327 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:11.324+0000 7faf177d4740 1 -- 172.21.15.46:0/3367743078 shutdown_connections 2024-03-31T23:12:11.327 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:11.324+0000 7faf177d4740 1 --2- 172.21.15.46:0/3367743078 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x56008512ddf0 0x5600851301e0 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:11.327 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:11.324+0000 7faf177d4740 1 --2- 172.21.15.46:0/3367743078 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5600850b2db0 0x56008512c470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:11.327 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:11.324+0000 7faf177d4740 1 --2- 172.21.15.46:0/3367743078 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5600850a9fe0 0x5600850ad420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:11.327 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:11.324+0000 7faef5ffb640 0 client.25730 ms_handle_reset on v2:172.21.15.142:6838/2519143918 2024-03-31T23:12:11.327 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:11.324+0000 7faf177d4740 1 --2- 172.21.15.46:0/3367743078 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5600850a96c0 0x5600850acee0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:11.327 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:11.324+0000 7faf177d4740 1 -- 172.21.15.46:0/3367743078 >> 172.21.15.46:0/3367743078 conn(0x5600850a3fe0 msgr2=0x5600850a2f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:12:11.327 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:11.324+0000 7faf177d4740 1 -- 172.21.15.46:0/3367743078 shutdown_connections 2024-03-31T23:12:11.327 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:11.324+0000 7faf177d4740 1 -- 172.21.15.46:0/3367743078 wait complete. 2024-03-31T23:12:11.327 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:11.325+0000 7faf177d4740 20 client.25730 trim_cache size 0 max 16384 2024-03-31T23:12:11.328 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout:{ 2024-03-31T23:12:11.328 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout: "message": "", 2024-03-31T23:12:11.328 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout: "return_code": 0 2024-03-31T23:12:11.328 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout:} 2024-03-31T23:12:11.329 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:11.325+0000 7faf14be9640 1 -- 172.21.15.46:0/2094506435 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7faefc0a27d0 msgr2=0x7faefc0a4c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:12:11.329 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:11.325+0000 7faf14be9640 1 --2- 172.21.15.46:0/2094506435 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7faefc0a27d0 0x7faefc0a4c90 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7faf08005fd0 tx=0x7faf08013b40 comp rx=0 tx=0).stop 2024-03-31T23:12:11.329 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:11.325+0000 7faf14be9640 1 -- 172.21.15.46:0/2094506435 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faf10069770 msgr2=0x7faf10198400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:12:11.329 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:11.325+0000 7faf14be9640 1 --2- 172.21.15.46:0/2094506435 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faf10069770 0x7faf10198400 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7faef800e9f0 tx=0x7faef800eeb0 comp rx=0 tx=0).stop 2024-03-31T23:12:11.329 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:11.325+0000 7faf14be9640 1 -- 172.21.15.46:0/2094506435 shutdown_connections 2024-03-31T23:12:11.329 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:11.325+0000 7faf14be9640 1 --2- 172.21.15.46:0/2094506435 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7faefc0a27d0 0x7faefc0a4c90 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:11.329 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:11.325+0000 7faf14be9640 1 --2- 172.21.15.46:0/2094506435 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7faf101077c0 0x7faf1019c780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:11.329 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:11.325+0000 7faf14be9640 1 --2- 172.21.15.46:0/2094506435 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faf10069770 0x7faf10198400 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:11.329 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:11.325+0000 7faf14be9640 1 --2- 172.21.15.46:0/2094506435 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7faf10068dd0 0x7faf10197ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:11.329 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:11.325+0000 7faf14be9640 1 -- 172.21.15.46:0/2094506435 >> 172.21.15.46:0/2094506435 conn(0x7faf100774a0 msgr2=0x7faf100ff410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:12:11.330 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:11.325+0000 7faf14be9640 1 -- 172.21.15.46:0/2094506435 shutdown_connections 2024-03-31T23:12:11.330 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:11.325+0000 7faf14be9640 1 -- 172.21.15.46:0/2094506435 wait complete. 2024-03-31T23:12:12.611 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:12:12 smithi046 ceph-mon[32498]: pgmap v17: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 24 KiB/s wr, 27 op/s 2024-03-31T23:12:12.693 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:12:12 smithi142 ceph-mon[35271]: pgmap v17: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 24 KiB/s wr, 27 op/s 2024-03-31T23:12:12.795 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:12:12 smithi162 ceph-mon[36237]: pgmap v17: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 24 KiB/s wr, 27 op/s 2024-03-31T23:12:14.611 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:12:14 smithi046 ceph-mon[32498]: pgmap v18: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 23 KiB/s wr, 21 op/s 2024-03-31T23:12:14.693 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:12:14 smithi142 ceph-mon[35271]: pgmap v18: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 23 KiB/s wr, 21 op/s 2024-03-31T23:12:14.795 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:12:14 smithi162 ceph-mon[36237]: pgmap v18: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 23 KiB/s wr, 21 op/s 2024-03-31T23:12:16.611 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:12:16 smithi046 ceph-mon[32498]: pgmap v19: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 23 KiB/s wr, 25 op/s 2024-03-31T23:12:16.693 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:12:16 smithi142 ceph-mon[35271]: pgmap v19: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 23 KiB/s wr, 25 op/s 2024-03-31T23:12:16.795 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:12:16 smithi162 ceph-mon[36237]: pgmap v19: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 23 KiB/s wr, 25 op/s 2024-03-31T23:12:18.611 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:12:18 smithi046 ceph-mon[32498]: pgmap v20: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 767 B/s wr, 7 op/s 2024-03-31T23:12:18.693 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:12:18 smithi142 ceph-mon[35271]: pgmap v20: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 767 B/s wr, 7 op/s 2024-03-31T23:12:18.795 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:12:18 smithi162 ceph-mon[36237]: pgmap v20: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 767 B/s wr, 7 op/s 2024-03-31T23:12:20.598 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:12:20 smithi162 ceph-mon[36237]: pgmap v21: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 767 B/s wr, 10 op/s 2024-03-31T23:12:20.611 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:12:20 smithi046 ceph-mon[32498]: pgmap v21: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 767 B/s wr, 10 op/s 2024-03-31T23:12:20.693 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:12:20 smithi142 ceph-mon[35271]: pgmap v21: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 767 B/s wr, 10 op/s 2024-03-31T23:12:22.611 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:12:22 smithi046 ceph-mon[32498]: pgmap v22: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 682 B/s wr, 7 op/s 2024-03-31T23:12:22.693 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:12:22 smithi142 ceph-mon[35271]: pgmap v22: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 682 B/s wr, 7 op/s 2024-03-31T23:12:22.795 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:12:22 smithi162 ceph-mon[36237]: pgmap v22: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 682 B/s wr, 7 op/s 2024-03-31T23:12:24.430 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:24.429+0000 7fb6b246f640 1 -- 172.21.15.46:0/2439515614 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb6ac1055f0 msgr2=0x7fb6ac1059d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:12:24.430 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:24.429+0000 7fb6b246f640 1 --2- 172.21.15.46:0/2439515614 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb6ac1055f0 0x7fb6ac1059d0 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7fb69c009900 tx=0x7fb69c02bc30 comp rx=0 tx=0).stop 2024-03-31T23:12:24.430 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:24.429+0000 7fb6b246f640 1 -- 172.21.15.46:0/2439515614 shutdown_connections 2024-03-31T23:12:24.430 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:24.429+0000 7fb6b246f640 1 --2- 172.21.15.46:0/2439515614 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb6ac101260 0x7fb6ac10d480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:24.430 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:24.429+0000 7fb6b246f640 1 --2- 172.21.15.46:0/2439515614 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb6ac1008a0 0x7fb6ac100d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:24.430 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:24.429+0000 7fb6b246f640 1 --2- 172.21.15.46:0/2439515614 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb6ac1055f0 0x7fb6ac1059d0 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:24.430 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:24.429+0000 7fb6b246f640 1 -- 172.21.15.46:0/2439515614 >> 172.21.15.46:0/2439515614 conn(0x7fb6ac0774a0 msgr2=0x7fb6ac0778b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:12:24.431 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:24.429+0000 7fb6b246f640 1 -- 172.21.15.46:0/2439515614 shutdown_connections 2024-03-31T23:12:24.431 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:24.429+0000 7fb6b246f640 1 -- 172.21.15.46:0/2439515614 wait complete. 2024-03-31T23:12:24.433 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:24.429+0000 7fb6b246f640 1 Processor -- start 2024-03-31T23:12:24.433 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:24.429+0000 7fb6b246f640 1 -- start start 2024-03-31T23:12:24.433 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:24.430+0000 7fb6b246f640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb6ac1008a0 0x7fb6ac198110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:24.433 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:24.430+0000 7fb6b246f640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb6ac101260 0x7fb6ac198650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:24.433 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:24.430+0000 7fb6b246f640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb6ac19c9d0 0x7fb6ac19ce80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:24.433 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:24.430+0000 7fb6b246f640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fb6ac111b10 con 0x7fb6ac19c9d0 2024-03-31T23:12:24.433 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:24.430+0000 7fb6b246f640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fb6ac111990 con 0x7fb6ac1008a0 2024-03-31T23:12:24.433 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:24.430+0000 7fb6b246f640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fb6ac111c90 con 0x7fb6ac101260 2024-03-31T23:12:24.433 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:24.430+0000 7fb6ab7fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb6ac101260 0x7fb6ac198650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:24.433 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:24.430+0000 7fb6ab7fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb6ac101260 0x7fb6ac198650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.46:39184/0 (socket says 172.21.15.46:39184) 2024-03-31T23:12:24.433 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:24.430+0000 7fb6abfff640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb6ac1008a0 0x7fb6ac198110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:24.433 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:24.430+0000 7fb6ab7fe640 1 -- 172.21.15.46:0/2586914138 learned_addr learned my addr 172.21.15.46:0/2586914138 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:12:24.434 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:24.430+0000 7fb6b0a0c640 1 --2- 172.21.15.46:0/2586914138 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb6ac19c9d0 0x7fb6ac19ce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:24.434 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:24.430+0000 7fb6abfff640 1 -- 172.21.15.46:0/2586914138 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb6ac101260 msgr2=0x7fb6ac198650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:12:24.434 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:24.431+0000 7fb6abfff640 1 --2- 172.21.15.46:0/2586914138 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb6ac101260 0x7fb6ac198650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:24.434 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:24.431+0000 7fb6abfff640 1 -- 172.21.15.46:0/2586914138 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb6ac19c9d0 msgr2=0x7fb6ac19ce80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:12:24.434 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:24.431+0000 7fb6abfff640 1 --2- 172.21.15.46:0/2586914138 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb6ac19c9d0 0x7fb6ac19ce80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:24.434 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:24.431+0000 7fb6abfff640 1 -- 172.21.15.46:0/2586914138 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6ac19d4c0 con 0x7fb6ac1008a0 2024-03-31T23:12:24.434 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:24.431+0000 7fb6ab7fe640 1 --2- 172.21.15.46:0/2586914138 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb6ac101260 0x7fb6ac198650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:12:24.434 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:24.431+0000 7fb6abfff640 1 --2- 172.21.15.46:0/2586914138 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb6ac1008a0 0x7fb6ac198110 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fb69c008000 tx=0x7fb69c005e00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:12:24.434 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:24.431+0000 7fb6a97fa640 1 -- 172.21.15.46:0/2586914138 <== mon.2 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb69c043020 con 0x7fb6ac1008a0 2024-03-31T23:12:24.434 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:24.431+0000 7fb6b246f640 1 -- 172.21.15.46:0/2586914138 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb6ac19d750 con 0x7fb6ac1008a0 2024-03-31T23:12:24.434 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:24.431+0000 7fb6a97fa640 1 -- 172.21.15.46:0/2586914138 <== mon.2 v2:172.21.15.142:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb69c02ca90 con 0x7fb6ac1008a0 2024-03-31T23:12:24.434 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:24.431+0000 7fb6a97fa640 1 -- 172.21.15.46:0/2586914138 <== mon.2 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb69c003a50 con 0x7fb6ac1008a0 2024-03-31T23:12:24.434 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:24.431+0000 7fb6b246f640 1 -- 172.21.15.46:0/2586914138 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb6ac1a4ec0 con 0x7fb6ac1008a0 2024-03-31T23:12:24.435 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:24.434+0000 7fb6a97fa640 1 -- 172.21.15.46:0/2586914138 <== mon.2 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 24) ==== 137553+0+0 (secure 0 0 0) 0x7fb69c046020 con 0x7fb6ac1008a0 2024-03-31T23:12:24.435 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:24.434+0000 7fb6b5055740 1 Processor -- start 2024-03-31T23:12:24.436 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:24.435+0000 7fb6b5055740 1 -- start start 2024-03-31T23:12:24.436 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:24.435+0000 7fb6a97fa640 1 --2- 172.21.15.46:0/2586914138 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fb6940a27d0 0x7fb6940a4c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:24.436 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:24.435+0000 7fb6b0a0c640 1 --2- 172.21.15.46:0/2586914138 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fb6940a27d0 0x7fb6940a4c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:24.436 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:24.435+0000 7fb6b5055740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564a071766c0 0x564a07176aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:24.436 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:24.435+0000 7fb6a97fa640 1 -- 172.21.15.46:0/2586914138 <== mon.2 v2:172.21.15.142:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7fb69c0f0d50 con 0x7fb6ac1008a0 2024-03-31T23:12:24.436 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:24.435+0000 7fb6b5055740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564a07176fe0 0x564a0717f870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:24.436 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:24.435+0000 7fb6b5055740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x564a0717fdb0 0x564a071821a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:24.436 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:24.435+0000 7fb6b5055740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x564a070d98f0 con 0x564a07176fe0 2024-03-31T23:12:24.436 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:24.435+0000 7fb6b5055740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x564a070828d0 con 0x564a0717fdb0 2024-03-31T23:12:24.436 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:24.435+0000 7fb6b5055740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x564a070d5990 con 0x564a071766c0 2024-03-31T23:12:24.438 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:24.435+0000 7fb6abfff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564a071766c0 0x564a07176aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:24.438 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:24.435+0000 7fb6b0a0c640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x564a0717fdb0 0x564a071821a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:24.438 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:24.435+0000 7fb6abfff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564a071766c0 0x564a07176aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.46:39198/0 (socket says 172.21.15.46:39198) 2024-03-31T23:12:24.438 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:24.435+0000 7fb6abfff640 1 -- 172.21.15.46:0/2224785955 learned_addr learned my addr 172.21.15.46:0/2224785955 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:12:24.438 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:24.435+0000 7fb6ab7fe640 1 --2- 172.21.15.46:0/2224785955 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564a07176fe0 0x564a0717f870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:24.438 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:24.435+0000 7fb6abfff640 1 -- 172.21.15.46:0/2224785955 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x564a0717fdb0 msgr2=0x564a071821a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:12:24.438 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:24.435+0000 7fb6abfff640 1 --2- 172.21.15.46:0/2224785955 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x564a0717fdb0 0x564a071821a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:24.438 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:24.435+0000 7fb6abfff640 1 -- 172.21.15.46:0/2224785955 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564a07176fe0 msgr2=0x564a0717f870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:12:24.439 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:24.435+0000 7fb6abfff640 1 --2- 172.21.15.46:0/2224785955 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564a07176fe0 0x564a0717f870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:24.439 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:24.435+0000 7fb6abfff640 1 -- 172.21.15.46:0/2224785955 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564a06dee210 con 0x564a071766c0 2024-03-31T23:12:24.439 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:24.435+0000 7fb6b0a0c640 1 --2- 172.21.15.46:0/2586914138 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fb6940a27d0 0x7fb6940a4c90 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fb6a4009fd0 tx=0x7fb6a4011bc0 comp rx=0 tx=0).ready entity=mgr.15654 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:12:24.439 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:24.435+0000 7fb6ab7fe640 1 --2- 172.21.15.46:0/2224785955 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564a07176fe0 0x564a0717f870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:12:24.439 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:24.436+0000 7fb6abfff640 1 --2- 172.21.15.46:0/2224785955 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564a071766c0 0x564a07176aa0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fb69c0bcce0 tx=0x7fb69c0bc940 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5684e03101234082 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:12:24.439 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:24.436+0000 7fb6927fc640 1 -- 172.21.15.46:0/2224785955 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb69c0bcfe0 con 0x564a071766c0 2024-03-31T23:12:24.439 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:24.436+0000 7fb6927fc640 1 -- 172.21.15.46:0/2224785955 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb69c0c1130 con 0x564a071766c0 2024-03-31T23:12:24.439 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:24.436+0000 7fb6927fc640 1 -- 172.21.15.46:0/2224785955 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb69c0bd4d0 con 0x564a071766c0 2024-03-31T23:12:24.439 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:24.437+0000 7fb6b5055740 1 -- 172.21.15.46:0/2224785955 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564a071766c0 msgr2=0x564a07176aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:12:24.439 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:24.437+0000 7fb6b5055740 1 --2- 172.21.15.46:0/2224785955 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564a071766c0 0x564a07176aa0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fb69c0bcce0 tx=0x7fb69c0bc940 comp rx=0 tx=0).stop 2024-03-31T23:12:24.439 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:24.437+0000 7fb6b5055740 1 -- 172.21.15.46:0/2224785955 shutdown_connections 2024-03-31T23:12:24.439 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:24.437+0000 7fb6b5055740 1 --2- 172.21.15.46:0/2224785955 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x564a0717fdb0 0x564a071821a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:24.439 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:24.437+0000 7fb6b5055740 1 --2- 172.21.15.46:0/2224785955 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564a07176fe0 0x564a0717f870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:24.439 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:24.437+0000 7fb6b5055740 1 --2- 172.21.15.46:0/2224785955 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564a071766c0 0x564a07176aa0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:24.440 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:24.437+0000 7fb6b5055740 1 -- 172.21.15.46:0/2224785955 >> 172.21.15.46:0/2224785955 conn(0x564a07170fe0 msgr2=0x564a071713c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:12:24.440 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:24.437+0000 7fb6b5055740 1 -- 172.21.15.46:0/2224785955 shutdown_connections 2024-03-31T23:12:24.440 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:24.437+0000 7fb6b5055740 1 -- 172.21.15.46:0/2224785955 wait complete. 2024-03-31T23:12:24.440 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:24.437+0000 7fb6b5055740 1 -- start start 2024-03-31T23:12:24.443 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:24.437+0000 7fb6b5055740 1 Processor -- start 2024-03-31T23:12:24.443 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:24.438+0000 7fb6b5055740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x564a071766c0 0x564a0717ce80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:24.443 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:24.438+0000 7fb6b5055740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564a07176fe0 0x564a0717d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:24.444 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:24.438+0000 7fb6b5055740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564a0717fdb0 0x564a0717d900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:24.444 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:24.438+0000 7fb6abfff640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x564a071766c0 0x564a0717ce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:24.444 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:24.438+0000 7fb6abfff640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x564a071766c0 0x564a0717ce80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.46:59528/0 (socket says 172.21.15.46:59528) 2024-03-31T23:12:24.444 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:24.438+0000 7fb6ab7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564a07176fe0 0x564a0717d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:24.444 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:24.438+0000 7fb6abfff640 1 -- 172.21.15.46:0/2022731695 learned_addr learned my addr 172.21.15.46:0/2022731695 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:12:24.444 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:24.439+0000 7fb6b5055740 1 -- 172.21.15.46:0/2022731695 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x564a070d5990 con 0x564a07176fe0 2024-03-31T23:12:24.444 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:24.439+0000 7fb6b5055740 1 -- 172.21.15.46:0/2022731695 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x564a07077f80 con 0x564a071766c0 2024-03-31T23:12:24.444 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:24.439+0000 7fb6b5055740 1 -- 172.21.15.46:0/2022731695 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x564a071827a0 con 0x564a0717fdb0 2024-03-31T23:12:24.444 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:24.439+0000 7fb6b0a0c640 1 --2- 172.21.15.46:0/2022731695 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564a0717fdb0 0x564a0717d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:24.444 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:24.439+0000 7fb6b5055740 10 client.0 _ll_register_callbacks cb 0x564a070d0290 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T23:12:24.444 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:24.439+0000 7fb6abfff640 1 -- 172.21.15.46:0/2022731695 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564a0717fdb0 msgr2=0x564a0717d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:12:24.444 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:24.439+0000 7fb6abfff640 1 --2- 172.21.15.46:0/2022731695 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564a0717fdb0 0x564a0717d900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:24.444 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:24.439+0000 7fb6abfff640 1 -- 172.21.15.46:0/2022731695 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564a07176fe0 msgr2=0x564a0717d3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:12:24.445 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:24.439+0000 7fb6abfff640 1 --2- 172.21.15.46:0/2022731695 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564a07176fe0 0x564a0717d3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:24.445 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:24.439+0000 7fb6abfff640 1 -- 172.21.15.46:0/2022731695 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564a06eba620 con 0x564a071766c0 2024-03-31T23:12:24.445 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:24.439+0000 7fb6b0a0c640 1 --2- 172.21.15.46:0/2022731695 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564a0717fdb0 0x564a0717d900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:12:24.445 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:24.439+0000 7fb6ab7fe640 1 --2- 172.21.15.46:0/2022731695 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564a07176fe0 0x564a0717d3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:12:24.445 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:24.439+0000 7fb6b5055740 1 -- 172.21.15.46:0/2022731695 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb69c0c1130 con 0x564a071766c0 2024-03-31T23:12:24.445 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:24.439+0000 7fb6abfff640 1 --2- 172.21.15.46:0/2022731695 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x564a071766c0 0x564a0717ce80 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fb69c009900 tx=0x7fb69c0053b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2e1354b5b5f00f86 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:12:24.445 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:24.439+0000 7fb68b7fe640 10 client.25742 ms_handle_connect on v2:172.21.15.142:3300/0 2024-03-31T23:12:24.445 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:24.440+0000 7fb68b7fe640 1 -- 172.21.15.46:0/2022731695 <== mon.2 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb69c0a3050 con 0x564a071766c0 2024-03-31T23:12:24.445 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:24.440+0000 7fb68b7fe640 1 -- 172.21.15.46:0/2022731695 <== mon.2 v2:172.21.15.142:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7fb69c0bb2e0 con 0x564a071766c0 2024-03-31T23:12:24.445 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:24.440+0000 7fb68b7fe640 1 -- 172.21.15.46:0/2022731695 <== mon.2 v2:172.21.15.142:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fb69c0d9070 con 0x564a071766c0 2024-03-31T23:12:24.445 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:24.440+0000 7fb6b5055740 10 client.25742 fetch_fsmap learned FSMap version 25 2024-03-31T23:12:24.445 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:24.440+0000 7fb6b5055740 1 -- 172.21.15.46:0/2022731695 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({fsmap=25}) -- 0x564a0718e370 con 0x564a071766c0 2024-03-31T23:12:24.446 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:24.440+0000 7fb68b7fe640 1 -- 172.21.15.46:0/2022731695 <== mon.2 v2:172.21.15.142:3300/0 4 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7fb69c0a6060 con 0x564a071766c0 2024-03-31T23:12:24.446 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:24.440+0000 7fb6b5055740 10 client.25742 fetch_fsmap finished waiting for FSMap version 25 2024-03-31T23:12:24.446 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:24.440+0000 7fb6b5055740 10 client.25742 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-31T23:12:24.446 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:24.440+0000 7fb6b5055740 20 client.25742 populate_metadata read hostname 'smithi046' 2024-03-31T23:12:24.446 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:24.441+0000 7fb6b5055740 1 --2- 172.21.15.46:0/2022731695 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x564a0718f7d0 0x564a07191be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:24.446 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:24.441+0000 7fb6b5055740 4 client.25742 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-31T23:12:24.446 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:24.441+0000 7fb6b5055740 1 -- 172.21.15.46:0/2022731695 --> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fb69c0c1130 con 0x564a0718f7d0 2024-03-31T23:12:24.446 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:24.441+0000 7fb6ab7fe640 1 --2- 172.21.15.46:0/2022731695 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x564a0718f7d0 0x564a07191be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:24.446 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:24.441+0000 7fb6ab7fe640 1 --2- 172.21.15.46:0/2022731695 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x564a0718f7d0 0x564a07191be0 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=550328dab0d33e1b server_cookie=caba70590165652a in_seq=0 out_seq=0 2024-03-31T23:12:24.446 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:24.441+0000 7fb68b7fe640 10 client.25742 ms_handle_connect on v2:172.21.15.142:6838/2519143918 2024-03-31T23:12:24.446 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:24.442+0000 7fb68b7fe640 1 -- 172.21.15.46:0/2022731695 <== mds.0 v2:172.21.15.142:6838/2519143918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x7fb69c0c1130 con 0x564a0718f7d0 2024-03-31T23:12:24.446 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:24.442+0000 7fb68b7fe640 10 client.25742 handle_command_reply: tid=0 2024-03-31T23:12:24.447 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:24.442+0000 7fb6b5055740 1 client.25742 shutdown 2024-03-31T23:12:24.447 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:24.442+0000 7fb6b5055740 1 -- 172.21.15.46:0/2022731695 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x564a071766c0 msgr2=0x564a0717ce80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:12:24.447 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:24.442+0000 7fb6b5055740 1 --2- 172.21.15.46:0/2022731695 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x564a071766c0 0x564a0717ce80 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fb69c009900 tx=0x7fb69c0053b0 comp rx=0 tx=0).stop 2024-03-31T23:12:24.447 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:24.442+0000 7fb6b5055740 1 -- 172.21.15.46:0/2022731695 shutdown_connections 2024-03-31T23:12:24.447 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:24.442+0000 7fb6b5055740 1 --2- 172.21.15.46:0/2022731695 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x564a0718f7d0 0x564a07191be0 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:24.447 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:24.442+0000 7fb6b5055740 1 --2- 172.21.15.46:0/2022731695 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564a0717fdb0 0x564a0717d900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:24.447 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:24.442+0000 7fb68b7fe640 0 client.25742 ms_handle_reset on v2:172.21.15.142:6838/2519143918 2024-03-31T23:12:24.447 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:24.442+0000 7fb6b5055740 1 --2- 172.21.15.46:0/2022731695 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564a07176fe0 0x564a0717d3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:24.447 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:24.442+0000 7fb6b5055740 1 --2- 172.21.15.46:0/2022731695 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x564a071766c0 0x564a0717ce80 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:24.447 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:24.442+0000 7fb6b5055740 1 -- 172.21.15.46:0/2022731695 >> 172.21.15.46:0/2022731695 conn(0x564a07170fe0 msgr2=0x564a07172d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:12:24.447 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:24.442+0000 7fb6b5055740 1 -- 172.21.15.46:0/2022731695 shutdown_connections 2024-03-31T23:12:24.447 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:24.442+0000 7fb6b5055740 1 -- 172.21.15.46:0/2022731695 wait complete. 2024-03-31T23:12:24.447 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:24.443+0000 7fb6b5055740 20 client.25742 trim_cache size 0 max 16384 2024-03-31T23:12:24.473 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:24.471+0000 7fb6b5055740 1 Processor -- start 2024-03-31T23:12:24.473 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:24.471+0000 7fb6b5055740 1 -- start start 2024-03-31T23:12:24.473 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:24.471+0000 7fb6b5055740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564a071766c0 0x564a0717f5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:24.473 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:24.471+0000 7fb6b5055740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564a07176fe0 0x564a0718b330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:24.473 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:24.471+0000 7fb6b5055740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x564a0717fdb0 0x564a071974f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:24.473 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:24.471+0000 7fb6b5055740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fb6ac1af5d0 con 0x564a07176fe0 2024-03-31T23:12:24.474 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:24.471+0000 7fb6b5055740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fb6ac1acea0 con 0x564a0717fdb0 2024-03-31T23:12:24.474 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:24.471+0000 7fb6b5055740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fb6ac1aacf0 con 0x564a071766c0 2024-03-31T23:12:24.474 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:24.471+0000 7fb6abfff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564a071766c0 0x564a0717f5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:24.474 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:24.471+0000 7fb6ab7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564a07176fe0 0x564a0718b330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:24.474 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:24.471+0000 7fb6abfff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564a071766c0 0x564a0717f5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.46:39216/0 (socket says 172.21.15.46:39216) 2024-03-31T23:12:24.474 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:24.471+0000 7fb6b0a0c640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x564a0717fdb0 0x564a071974f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:24.474 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:24.471+0000 7fb6abfff640 1 -- 172.21.15.46:0/1676112816 learned_addr learned my addr 172.21.15.46:0/1676112816 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:12:24.474 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:24.472+0000 7fb6abfff640 1 -- 172.21.15.46:0/1676112816 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x564a0717fdb0 msgr2=0x564a071974f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:12:24.474 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:24.472+0000 7fb6abfff640 1 --2- 172.21.15.46:0/1676112816 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x564a0717fdb0 0x564a071974f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:24.474 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:24.472+0000 7fb6abfff640 1 -- 172.21.15.46:0/1676112816 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564a07176fe0 msgr2=0x564a0718b330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:12:24.474 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:24.472+0000 7fb6abfff640 1 --2- 172.21.15.46:0/1676112816 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564a07176fe0 0x564a0718b330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:24.474 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:24.472+0000 7fb6abfff640 1 -- 172.21.15.46:0/1676112816 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564a07178a60 con 0x564a071766c0 2024-03-31T23:12:24.474 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:24.472+0000 7fb6abfff640 1 --2- 172.21.15.46:0/1676112816 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564a071766c0 0x564a0717f5b0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fb69c0ee6f0 tx=0x7fb69c030000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=319b49c34da31332 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:12:24.474 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:24.472+0000 7fb691ffb640 1 -- 172.21.15.46:0/1676112816 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb69c0a3070 con 0x564a071766c0 2024-03-31T23:12:24.475 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:24.472+0000 7fb691ffb640 1 -- 172.21.15.46:0/1676112816 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb69c0c1130 con 0x564a071766c0 2024-03-31T23:12:24.475 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:24.472+0000 7fb691ffb640 1 -- 172.21.15.46:0/1676112816 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb69c0c5560 con 0x564a071766c0 2024-03-31T23:12:24.478 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:24.473+0000 7fb6b5055740 1 -- 172.21.15.46:0/1676112816 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564a071766c0 msgr2=0x564a0717f5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:12:24.478 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:24.473+0000 7fb6b5055740 1 --2- 172.21.15.46:0/1676112816 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564a071766c0 0x564a0717f5b0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fb69c0ee6f0 tx=0x7fb69c030000 comp rx=0 tx=0).stop 2024-03-31T23:12:24.478 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:24.473+0000 7fb6b5055740 1 -- 172.21.15.46:0/1676112816 shutdown_connections 2024-03-31T23:12:24.478 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:24.473+0000 7fb6b5055740 1 --2- 172.21.15.46:0/1676112816 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x564a0717fdb0 0x564a071974f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:24.478 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:24.473+0000 7fb6b5055740 1 --2- 172.21.15.46:0/1676112816 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564a07176fe0 0x564a0718b330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:24.478 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:24.473+0000 7fb6b5055740 1 --2- 172.21.15.46:0/1676112816 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564a071766c0 0x564a0717f5b0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:24.478 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:24.473+0000 7fb6b5055740 1 -- 172.21.15.46:0/1676112816 >> 172.21.15.46:0/1676112816 conn(0x564a07170fe0 msgr2=0x564a071713c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:12:24.479 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:24.473+0000 7fb6b5055740 1 -- 172.21.15.46:0/1676112816 shutdown_connections 2024-03-31T23:12:24.479 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:24.473+0000 7fb6b5055740 1 -- 172.21.15.46:0/1676112816 wait complete. 2024-03-31T23:12:24.479 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:24.473+0000 7fb6b5055740 1 -- start start 2024-03-31T23:12:24.479 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:24.473+0000 7fb6b5055740 1 Processor -- start 2024-03-31T23:12:24.479 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:24.474+0000 7fb6b5055740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x564a071766c0 0x564a07187960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:24.479 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:24.474+0000 7fb6b5055740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564a07176fe0 0x564a07179210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:24.479 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:24.474+0000 7fb6b5055740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564a0717fdb0 0x564a07179750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:24.479 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:24.474+0000 7fb6b5055740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fb6ac1ab360 con 0x564a07176fe0 2024-03-31T23:12:24.479 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:24.474+0000 7fb6b5055740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fb6ac1ac7d0 con 0x564a071766c0 2024-03-31T23:12:24.479 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:24.474+0000 7fb6b5055740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fb6ac1aba30 con 0x564a0717fdb0 2024-03-31T23:12:24.479 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:24.474+0000 7fb6b5055740 10 client.0 _ll_register_callbacks cb 0x7fb6a0009550 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T23:12:24.479 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:24.474+0000 7fb6ab7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564a07176fe0 0x564a07179210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:24.479 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:24.474+0000 7fb6abfff640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x564a071766c0 0x564a07187960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:24.479 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:24.474+0000 7fb6ab7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564a07176fe0 0x564a07179210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:34336/0 (socket says 172.21.15.46:34336) 2024-03-31T23:12:24.480 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:24.474+0000 7fb6ab7fe640 1 -- 172.21.15.46:0/2670995360 learned_addr learned my addr 172.21.15.46:0/2670995360 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:12:24.480 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:24.474+0000 7fb6b0a0c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564a0717fdb0 0x564a07179750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:24.480 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:24.474+0000 7fb6ab7fe640 1 -- 172.21.15.46:0/2670995360 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564a0717fdb0 msgr2=0x564a07179750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:12:24.480 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:24.474+0000 7fb6ab7fe640 1 --2- 172.21.15.46:0/2670995360 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564a0717fdb0 0x564a07179750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:24.480 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:24.474+0000 7fb6ab7fe640 1 -- 172.21.15.46:0/2670995360 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x564a071766c0 msgr2=0x564a07187960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:12:24.480 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:24.474+0000 7fb6ab7fe640 1 --2- 172.21.15.46:0/2670995360 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x564a071766c0 0x564a07187960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:24.480 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:24.474+0000 7fb6ab7fe640 1 -- 172.21.15.46:0/2670995360 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564a07188050 con 0x564a07176fe0 2024-03-31T23:12:24.480 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:24.474+0000 7fb6abfff640 1 --2- 172.21.15.46:0/2670995360 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x564a071766c0 0x564a07187960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:12:24.480 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:24.474+0000 7fb6ab7fe640 1 --2- 172.21.15.46:0/2670995360 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564a07176fe0 0x564a07179210 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7fb6a0009520 tx=0x7fb6a0004550 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4abbb2be26ba382d server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:12:24.480 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:24.474+0000 7fb68b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-31T23:12:24.480 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:24.475+0000 7fb68b7fe640 1 -- 172.21.15.46:0/2670995360 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6a0005220 con 0x564a07176fe0 2024-03-31T23:12:24.480 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:24.475+0000 7fb68b7fe640 1 -- 172.21.15.46:0/2670995360 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7fb6a00053c0 con 0x564a07176fe0 2024-03-31T23:12:24.480 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:24.475+0000 7fb6b5055740 1 -- 172.21.15.46:0/2670995360 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb69c0c1130 con 0x564a07176fe0 2024-03-31T23:12:24.480 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:24.475+0000 7fb68b7fe640 1 -- 172.21.15.46:0/2670995360 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fb6a0036050 con 0x564a07176fe0 2024-03-31T23:12:24.481 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:24.475+0000 7fb6b5055740 10 client.15906 fetch_fsmap learned FSMap version 25 2024-03-31T23:12:24.481 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:24.475+0000 7fb6b5055740 1 -- 172.21.15.46:0/2670995360 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x564a071f5650 con 0x564a07176fe0 2024-03-31T23:12:24.481 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:24.475+0000 7fb68b7fe640 1 -- 172.21.15.46:0/2670995360 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb69c0c1130 con 0x564a07176fe0 2024-03-31T23:12:24.481 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:24.475+0000 7fb68b7fe640 1 -- 172.21.15.46:0/2670995360 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7fb6a0004960 con 0x564a07176fe0 2024-03-31T23:12:24.481 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:24.475+0000 7fb6b5055740 10 client.15906 fetch_fsmap finished waiting for FSMap version 25 2024-03-31T23:12:24.481 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:24.475+0000 7fb6b5055740 10 client.15906 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-31T23:12:24.481 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:24.475+0000 7fb6b5055740 20 client.15906 populate_metadata read hostname 'smithi046' 2024-03-31T23:12:24.481 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:24.475+0000 7fb6b5055740 1 --2- 172.21.15.46:0/2670995360 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x564a0718f7d0 0x564a071f7de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:24.481 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:24.475+0000 7fb6b5055740 4 client.15906 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-31T23:12:24.481 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:24.475+0000 7fb6b5055740 1 -- 172.21.15.46:0/2670995360 --> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] -- command(tid 0: {"prefix": "flush journal"}) -- 0x564a070ed120 con 0x564a0718f7d0 2024-03-31T23:12:24.481 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:24.476+0000 7fb6abfff640 1 --2- 172.21.15.46:0/2670995360 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x564a0718f7d0 0x564a071f7de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:24.481 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:24.476+0000 7fb6abfff640 1 --2- 172.21.15.46:0/2670995360 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x564a0718f7d0 0x564a071f7de0 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=819a96b411cffc84 server_cookie=7e9244416f3916c in_seq=0 out_seq=0 2024-03-31T23:12:24.481 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:24.476+0000 7fb68b7fe640 10 client.15906 ms_handle_connect on v2:172.21.15.142:6838/2519143918 2024-03-31T23:12:24.482 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:24.479+0000 7fb68b7fe640 1 -- 172.21.15.46:0/2670995360 <== mds.0 v2:172.21.15.142:6838/2519143918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x564a070ed120 con 0x564a0718f7d0 2024-03-31T23:12:24.483 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:24.479+0000 7fb68b7fe640 10 client.15906 handle_command_reply: tid=0 2024-03-31T23:12:24.483 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:24.479+0000 7fb6b5055740 1 client.15906 shutdown 2024-03-31T23:12:24.483 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:24.479+0000 7fb6b5055740 1 -- 172.21.15.46:0/2670995360 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564a07176fe0 msgr2=0x564a07179210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:12:24.483 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:24.479+0000 7fb6b5055740 1 --2- 172.21.15.46:0/2670995360 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564a07176fe0 0x564a07179210 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7fb6a0009520 tx=0x7fb6a0004550 comp rx=0 tx=0).stop 2024-03-31T23:12:24.483 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:24.479+0000 7fb6b5055740 1 -- 172.21.15.46:0/2670995360 shutdown_connections 2024-03-31T23:12:24.483 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:24.479+0000 7fb6b5055740 1 --2- 172.21.15.46:0/2670995360 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x564a0718f7d0 0x564a071f7de0 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:24.483 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:24.480+0000 7fb6b5055740 1 --2- 172.21.15.46:0/2670995360 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564a0717fdb0 0x564a07179750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:24.483 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:24.480+0000 7fb6b5055740 1 --2- 172.21.15.46:0/2670995360 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564a07176fe0 0x564a07179210 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:24.483 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:24.480+0000 7fb68b7fe640 0 client.15906 ms_handle_reset on v2:172.21.15.142:6838/2519143918 2024-03-31T23:12:24.483 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:24.480+0000 7fb6b5055740 1 --2- 172.21.15.46:0/2670995360 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x564a071766c0 0x564a07187960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:24.483 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:24.480+0000 7fb6b5055740 1 -- 172.21.15.46:0/2670995360 >> 172.21.15.46:0/2670995360 conn(0x564a07170fe0 msgr2=0x564a0717b730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:12:24.483 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:24.480+0000 7fb6b5055740 1 -- 172.21.15.46:0/2670995360 shutdown_connections 2024-03-31T23:12:24.483 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:24.480+0000 7fb6b5055740 1 -- 172.21.15.46:0/2670995360 wait complete. 2024-03-31T23:12:24.484 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:24.480+0000 7fb6b5055740 20 client.15906 trim_cache size 0 max 16384 2024-03-31T23:12:24.484 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout:{ 2024-03-31T23:12:24.484 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout: "message": "", 2024-03-31T23:12:24.484 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout: "return_code": 0 2024-03-31T23:12:24.484 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout:} 2024-03-31T23:12:24.485 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:24.480+0000 7fb6b246f640 1 -- 172.21.15.46:0/2586914138 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fb6940a27d0 msgr2=0x7fb6940a4c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:12:24.485 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:24.480+0000 7fb6b246f640 1 --2- 172.21.15.46:0/2586914138 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fb6940a27d0 0x7fb6940a4c90 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fb6a4009fd0 tx=0x7fb6a4011bc0 comp rx=0 tx=0).stop 2024-03-31T23:12:24.485 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:24.480+0000 7fb6b246f640 1 -- 172.21.15.46:0/2586914138 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb6ac1008a0 msgr2=0x7fb6ac198110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:12:24.485 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:24.480+0000 7fb6b246f640 1 --2- 172.21.15.46:0/2586914138 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb6ac1008a0 0x7fb6ac198110 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fb69c008000 tx=0x7fb69c005e00 comp rx=0 tx=0).stop 2024-03-31T23:12:24.485 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:24.480+0000 7fb6b246f640 1 -- 172.21.15.46:0/2586914138 shutdown_connections 2024-03-31T23:12:24.485 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:24.480+0000 7fb6b246f640 1 --2- 172.21.15.46:0/2586914138 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fb6940a27d0 0x7fb6940a4c90 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:24.485 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:24.480+0000 7fb6b246f640 1 --2- 172.21.15.46:0/2586914138 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb6ac19c9d0 0x7fb6ac19ce80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:24.485 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:24.480+0000 7fb6b246f640 1 --2- 172.21.15.46:0/2586914138 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb6ac101260 0x7fb6ac198650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:24.485 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:24.480+0000 7fb6b246f640 1 --2- 172.21.15.46:0/2586914138 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb6ac1008a0 0x7fb6ac198110 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:24.485 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:24.480+0000 7fb6b246f640 1 -- 172.21.15.46:0/2586914138 >> 172.21.15.46:0/2586914138 conn(0x7fb6ac0774a0 msgr2=0x7fb6ac0fe910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:12:24.486 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:24.480+0000 7fb6b246f640 1 -- 172.21.15.46:0/2586914138 shutdown_connections 2024-03-31T23:12:24.486 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:24.480+0000 7fb6b246f640 1 -- 172.21.15.46:0/2586914138 wait complete. 2024-03-31T23:12:24.611 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:12:24 smithi046 ceph-mon[32498]: pgmap v23: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 682 B/s wr, 6 op/s 2024-03-31T23:12:24.693 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:12:24 smithi142 ceph-mon[35271]: pgmap v23: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 682 B/s wr, 6 op/s 2024-03-31T23:12:24.795 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:12:24 smithi162 ceph-mon[36237]: pgmap v23: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 682 B/s wr, 6 op/s 2024-03-31T23:12:26.611 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:12:26 smithi046 ceph-mon[32498]: pgmap v24: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 682 B/s wr, 8 op/s 2024-03-31T23:12:26.693 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:12:26 smithi142 ceph-mon[35271]: pgmap v24: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 682 B/s wr, 8 op/s 2024-03-31T23:12:26.795 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:12:26 smithi162 ceph-mon[36237]: pgmap v24: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 682 B/s wr, 8 op/s 2024-03-31T23:12:28.611 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:12:28 smithi046 ceph-mon[32498]: pgmap v25: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-31T23:12:28.693 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:12:28 smithi142 ceph-mon[35271]: pgmap v25: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-31T23:12:28.795 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:12:28 smithi162 ceph-mon[36237]: pgmap v25: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-31T23:12:30.611 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:12:30 smithi046 ceph-mon[32498]: pgmap v26: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 255 B/s wr, 7 op/s 2024-03-31T23:12:30.693 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:12:30 smithi142 ceph-mon[35271]: pgmap v26: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 255 B/s wr, 7 op/s 2024-03-31T23:12:30.795 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:12:30 smithi162 ceph-mon[36237]: pgmap v26: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 255 B/s wr, 7 op/s 2024-03-31T23:12:32.611 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:12:32 smithi046 ceph-mon[32498]: pgmap v27: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-03-31T23:12:32.693 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:12:32 smithi142 ceph-mon[35271]: pgmap v27: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-03-31T23:12:32.795 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:12:32 smithi162 ceph-mon[36237]: pgmap v27: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-03-31T23:12:34.611 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:12:34 smithi046 ceph-mon[32498]: pgmap v28: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-03-31T23:12:34.693 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:12:34 smithi142 ceph-mon[35271]: pgmap v28: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-03-31T23:12:34.795 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:12:34 smithi162 ceph-mon[36237]: pgmap v28: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-03-31T23:12:36.611 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:12:36 smithi046 ceph-mon[32498]: pgmap v29: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 255 B/s wr, 7 op/s 2024-03-31T23:12:36.693 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:12:36 smithi142 ceph-mon[35271]: pgmap v29: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 255 B/s wr, 7 op/s 2024-03-31T23:12:36.795 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:12:36 smithi162 ceph-mon[36237]: pgmap v29: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 255 B/s wr, 7 op/s 2024-03-31T23:12:36.930 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-31T23:12:36.930 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-31T23:12:37.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.043+0000 7fb2c6b81640 1 -- 172.21.15.46:0/4285077842 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb2c01005b0 msgr2=0x7fb2c010e570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:12:37.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.043+0000 7fb2c6b81640 1 --2- 172.21.15.46:0/4285077842 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb2c01005b0 0x7fb2c010e570 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fb2b8009ca0 tx=0x7fb2b802c0f0 comp rx=0 tx=0).stop 2024-03-31T23:12:37.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.043+0000 7fb2c6b81640 1 -- 172.21.15.46:0/4285077842 shutdown_connections 2024-03-31T23:12:37.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.043+0000 7fb2c6b81640 1 --2- 172.21.15.46:0/4285077842 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb2c01005b0 0x7fb2c010e570 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:37.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.043+0000 7fb2c6b81640 1 --2- 172.21.15.46:0/4285077842 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb2c00ffbf0 0x7fb2c0100070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:37.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.043+0000 7fb2c6b81640 1 --2- 172.21.15.46:0/4285077842 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb2c01076f0 0x7fb2c0107ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:37.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.043+0000 7fb2c6b81640 1 -- 172.21.15.46:0/4285077842 >> 172.21.15.46:0/4285077842 conn(0x7fb2c0076b70 msgr2=0x7fb2c00fdf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:12:37.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.043+0000 7fb2c6b81640 1 -- 172.21.15.46:0/4285077842 shutdown_connections 2024-03-31T23:12:37.047 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.043+0000 7fb2c6b81640 1 -- 172.21.15.46:0/4285077842 wait complete. 2024-03-31T23:12:37.047 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.043+0000 7fb2c6b81640 1 Processor -- start 2024-03-31T23:12:37.047 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.043+0000 7fb2c6b81640 1 -- start start 2024-03-31T23:12:37.047 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.044+0000 7fb2c6b81640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb2c00ffbf0 0x7fb2c0197ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:37.047 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.044+0000 7fb2c6b81640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb2c01005b0 0x7fb2c0198430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:37.047 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.044+0000 7fb2c6b81640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb2c01076f0 0x7fb2c019c7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:37.047 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.044+0000 7fb2c6b81640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fb2c0111990 con 0x7fb2c00ffbf0 2024-03-31T23:12:37.047 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.044+0000 7fb2c6b81640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fb2c0111810 con 0x7fb2c01005b0 2024-03-31T23:12:37.047 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.044+0000 7fb2c6b81640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fb2c0111b10 con 0x7fb2c01076f0 2024-03-31T23:12:37.047 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.044+0000 7fb2c491d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb2c00ffbf0 0x7fb2c0197ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:37.047 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.044+0000 7fb2c491d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb2c00ffbf0 0x7fb2c0197ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60848/0 (socket says 172.21.15.46:60848) 2024-03-31T23:12:37.047 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.044+0000 7fb2c491d640 1 -- 172.21.15.46:0/790153342 learned_addr learned my addr 172.21.15.46:0/790153342 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:12:37.047 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.044+0000 7fb2c511e640 1 --2- 172.21.15.46:0/790153342 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb2c01076f0 0x7fb2c019c7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:37.047 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.044+0000 7fb2bffff640 1 --2- 172.21.15.46:0/790153342 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb2c01005b0 0x7fb2c0198430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:37.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.044+0000 7fb2c491d640 1 -- 172.21.15.46:0/790153342 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb2c01076f0 msgr2=0x7fb2c019c7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:12:37.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.044+0000 7fb2c491d640 1 --2- 172.21.15.46:0/790153342 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb2c01076f0 0x7fb2c019c7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:37.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.044+0000 7fb2c491d640 1 -- 172.21.15.46:0/790153342 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb2c01005b0 msgr2=0x7fb2c0198430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:12:37.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.044+0000 7fb2c491d640 1 --2- 172.21.15.46:0/790153342 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb2c01005b0 0x7fb2c0198430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:37.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.044+0000 7fb2c491d640 1 -- 172.21.15.46:0/790153342 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb2c019cee0 con 0x7fb2c00ffbf0 2024-03-31T23:12:37.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.044+0000 7fb2c491d640 1 --2- 172.21.15.46:0/790153342 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb2c00ffbf0 0x7fb2c0197ef0 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7fb2b400e9f0 tx=0x7fb2b400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:12:37.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.044+0000 7fb2bdffb640 1 -- 172.21.15.46:0/790153342 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2b400cd80 con 0x7fb2c00ffbf0 2024-03-31T23:12:37.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.045+0000 7fb2c6b81640 1 -- 172.21.15.46:0/790153342 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb2c019d1d0 con 0x7fb2c00ffbf0 2024-03-31T23:12:37.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.045+0000 7fb2bdffb640 1 -- 172.21.15.46:0/790153342 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb2b40054d0 con 0x7fb2c00ffbf0 2024-03-31T23:12:37.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.045+0000 7fb2bdffb640 1 -- 172.21.15.46:0/790153342 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2b4010490 con 0x7fb2c00ffbf0 2024-03-31T23:12:37.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.045+0000 7fb2c6b81640 1 -- 172.21.15.46:0/790153342 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb2c01a4a60 con 0x7fb2c00ffbf0 2024-03-31T23:12:37.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.047+0000 7fb2bdffb640 1 -- 172.21.15.46:0/790153342 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 24) ==== 137553+0+0 (secure 0 0 0) 0x7fb2b4005020 con 0x7fb2c00ffbf0 2024-03-31T23:12:37.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.048+0000 7fb2c9763740 1 Processor -- start 2024-03-31T23:12:37.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.048+0000 7fb2c9763740 1 -- start start 2024-03-31T23:12:37.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.048+0000 7fb2bdffb640 1 --2- 172.21.15.46:0/790153342 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fb2b00a2730 0x7fb2b00a4bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:37.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.048+0000 7fb2c9763740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562b58097290 0x562b58097670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:37.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.048+0000 7fb2c9763740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x562b58097bb0 0x562b580a0250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:37.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.048+0000 7fb2c9763740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562b580a0790 0x562b580a2b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:37.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.048+0000 7fb2c9763740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x562b57ffb180 con 0x562b580a0790 2024-03-31T23:12:37.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.048+0000 7fb2c9763740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x562b57fa3b70 con 0x562b58097bb0 2024-03-31T23:12:37.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.048+0000 7fb2c9763740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x562b57ff7220 con 0x562b58097290 2024-03-31T23:12:37.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.048+0000 7fb2c511e640 1 --2- 172.21.15.46:0/790153342 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fb2b00a2730 0x7fb2b00a4bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:37.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.048+0000 7fb2bdffb640 1 -- 172.21.15.46:0/790153342 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7fb2b40cf440 con 0x7fb2c00ffbf0 2024-03-31T23:12:37.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.048+0000 7fb2c491d640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562b58097290 0x562b58097670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:37.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.048+0000 7fb2bffff640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x562b58097bb0 0x562b580a0250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:37.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.048+0000 7fb2c491d640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562b58097290 0x562b58097670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.46:45300/0 (socket says 172.21.15.46:45300) 2024-03-31T23:12:37.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.048+0000 7fb2c491d640 1 -- 172.21.15.46:0/3583214901 learned_addr learned my addr 172.21.15.46:0/3583214901 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:12:37.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.048+0000 7fb2c511e640 1 --2- 172.21.15.46:0/3583214901 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562b580a0790 0x562b580a2b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:37.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.048+0000 7fb2c491d640 1 -- 172.21.15.46:0/3583214901 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x562b58097bb0 msgr2=0x562b580a0250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:12:37.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.048+0000 7fb2c491d640 1 --2- 172.21.15.46:0/3583214901 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x562b58097bb0 0x562b580a0250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:37.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.048+0000 7fb2c491d640 1 -- 172.21.15.46:0/3583214901 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562b580a0790 msgr2=0x562b580a2b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:12:37.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.048+0000 7fb2c491d640 1 --2- 172.21.15.46:0/3583214901 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562b580a0790 0x562b580a2b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:37.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.048+0000 7fb2c491d640 1 -- 172.21.15.46:0/3583214901 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562b57fb77c0 con 0x562b58097290 2024-03-31T23:12:37.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.049+0000 7fb2c511e640 1 --2- 172.21.15.46:0/790153342 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fb2b00a2730 0x7fb2b00a4bf0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fb2b8004af0 tx=0x7fb2b8009960 comp rx=0 tx=0).ready entity=mgr.15654 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:12:37.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.049+0000 7fb2c491d640 1 --2- 172.21.15.46:0/3583214901 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562b58097290 0x562b58097670 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fb2b400e9c0 tx=0x7fb2b409b450 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1eab2d3f75a2f3f7 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:12:37.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.049+0000 7fb2aaffd640 1 -- 172.21.15.46:0/3583214901 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2b4005680 con 0x562b58097290 2024-03-31T23:12:37.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.049+0000 7fb2aaffd640 1 -- 172.21.15.46:0/3583214901 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb2b409bcc0 con 0x562b58097290 2024-03-31T23:12:37.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.049+0000 7fb2aaffd640 1 -- 172.21.15.46:0/3583214901 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2b4099990 con 0x562b58097290 2024-03-31T23:12:37.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.050+0000 7fb2c9763740 1 -- 172.21.15.46:0/3583214901 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562b58097290 msgr2=0x562b58097670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:12:37.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.050+0000 7fb2c9763740 1 --2- 172.21.15.46:0/3583214901 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562b58097290 0x562b58097670 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fb2b400e9c0 tx=0x7fb2b409b450 comp rx=0 tx=0).stop 2024-03-31T23:12:37.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.050+0000 7fb2c9763740 1 -- 172.21.15.46:0/3583214901 shutdown_connections 2024-03-31T23:12:37.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.050+0000 7fb2c9763740 1 --2- 172.21.15.46:0/3583214901 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562b580a0790 0x562b580a2b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:37.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.050+0000 7fb2c9763740 1 --2- 172.21.15.46:0/3583214901 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x562b58097bb0 0x562b580a0250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:37.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.050+0000 7fb2c9763740 1 --2- 172.21.15.46:0/3583214901 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562b58097290 0x562b58097670 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:37.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.050+0000 7fb2c9763740 1 -- 172.21.15.46:0/3583214901 >> 172.21.15.46:0/3583214901 conn(0x562b580919c0 msgr2=0x562b58091da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:12:37.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.050+0000 7fb2c9763740 1 -- 172.21.15.46:0/3583214901 shutdown_connections 2024-03-31T23:12:37.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.050+0000 7fb2c9763740 1 -- 172.21.15.46:0/3583214901 wait complete. 2024-03-31T23:12:37.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.050+0000 7fb2c9763740 1 -- start start 2024-03-31T23:12:37.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.050+0000 7fb2c9763740 1 Processor -- start 2024-03-31T23:12:37.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.051+0000 7fb2c9763740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562b58097290 0x562b5809d790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:37.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.051+0000 7fb2c9763740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x562b58097bb0 0x562b5809dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:37.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.051+0000 7fb2c9763740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562b580a0790 0x562b5809e210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:37.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.051+0000 7fb2c9763740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x562b580a34c0 con 0x562b58097290 2024-03-31T23:12:37.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.051+0000 7fb2c9763740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x562b57f994a0 con 0x562b58097bb0 2024-03-31T23:12:37.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.051+0000 7fb2c9763740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x562b580a3640 con 0x562b580a0790 2024-03-31T23:12:37.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.051+0000 7fb2c9763740 10 client.0 _ll_register_callbacks cb 0x562b57f9cd90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T23:12:37.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.051+0000 7fb2c511e640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562b580a0790 0x562b5809e210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:37.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.051+0000 7fb2c491d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562b58097290 0x562b5809d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:37.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.051+0000 7fb2c511e640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562b580a0790 0x562b5809e210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.46:45312/0 (socket says 172.21.15.46:45312) 2024-03-31T23:12:37.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.051+0000 7fb2c511e640 1 -- 172.21.15.46:0/495031825 learned_addr learned my addr 172.21.15.46:0/495031825 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:12:37.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.051+0000 7fb2bffff640 1 --2- 172.21.15.46:0/495031825 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x562b58097bb0 0x562b5809dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:37.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.051+0000 7fb2c511e640 1 -- 172.21.15.46:0/495031825 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x562b58097bb0 msgr2=0x562b5809dcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:12:37.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.051+0000 7fb2c511e640 1 --2- 172.21.15.46:0/495031825 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x562b58097bb0 0x562b5809dcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:37.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.051+0000 7fb2c511e640 1 -- 172.21.15.46:0/495031825 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562b58097290 msgr2=0x562b5809d790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:12:37.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.051+0000 7fb2c511e640 1 --2- 172.21.15.46:0/495031825 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562b58097290 0x562b5809d790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:37.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.051+0000 7fb2c511e640 1 -- 172.21.15.46:0/495031825 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562b57dd97d0 con 0x562b580a0790 2024-03-31T23:12:37.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.051+0000 7fb2c511e640 1 --2- 172.21.15.46:0/495031825 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562b580a0790 0x562b5809e210 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fb2b8036000 tx=0x7fb2b8005c50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=efb856521f78b1bb server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:12:37.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.051+0000 7fb29ffff640 10 client.0 ms_handle_connect on v2:172.21.15.162:3300/0 2024-03-31T23:12:37.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.052+0000 7fb29ffff640 1 -- 172.21.15.46:0/495031825 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2b8051060 con 0x562b580a0790 2024-03-31T23:12:37.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.052+0000 7fb29ffff640 1 -- 172.21.15.46:0/495031825 <== mon.1 v2:172.21.15.162:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7fb2b8004300 con 0x562b580a0790 2024-03-31T23:12:37.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.052+0000 7fb2c9763740 1 -- 172.21.15.46:0/495031825 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb2b409bcc0 con 0x562b580a0790 2024-03-31T23:12:37.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.052+0000 7fb29ffff640 1 -- 172.21.15.46:0/495031825 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fb2b8058050 con 0x562b580a0790 2024-03-31T23:12:37.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.052+0000 7fb2c9763740 10 client.25657 fetch_fsmap learned FSMap version 25 2024-03-31T23:12:37.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.052+0000 7fb2c9763740 1 -- 172.21.15.46:0/495031825 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x562b57d0f3f0 con 0x562b580a0790 2024-03-31T23:12:37.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.052+0000 7fb29ffff640 1 -- 172.21.15.46:0/495031825 <== mon.1 v2:172.21.15.162:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb2b409bcc0 con 0x562b580a0790 2024-03-31T23:12:37.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.053+0000 7fb29ffff640 1 -- 172.21.15.46:0/495031825 <== mon.1 v2:172.21.15.162:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7fb2b803f030 con 0x562b580a0790 2024-03-31T23:12:37.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.053+0000 7fb2c9763740 10 client.25657 fetch_fsmap finished waiting for FSMap version 25 2024-03-31T23:12:37.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.053+0000 7fb2c9763740 10 client.25657 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-31T23:12:37.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.053+0000 7fb2c9763740 20 client.25657 populate_metadata read hostname 'smithi046' 2024-03-31T23:12:37.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.053+0000 7fb2c9763740 1 --2- 172.21.15.46:0/495031825 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x562b580b0a10 0x562b580b2e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:37.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.053+0000 7fb2c9763740 4 client.25657 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-31T23:12:37.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.053+0000 7fb2c9763740 1 -- 172.21.15.46:0/495031825 --> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562b580086c0 con 0x562b580b0a10 2024-03-31T23:12:37.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.053+0000 7fb2c491d640 1 --2- 172.21.15.46:0/495031825 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x562b580b0a10 0x562b580b2e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:37.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.053+0000 7fb29ffff640 10 client.25657 ms_handle_connect on v2:172.21.15.142:6838/2519143918 2024-03-31T23:12:37.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.053+0000 7fb2c491d640 1 --2- 172.21.15.46:0/495031825 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x562b580b0a10 0x562b580b2e50 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=85f0ddd0f92631f server_cookie=bbef793ede47172d in_seq=0 out_seq=0 2024-03-31T23:12:37.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.054+0000 7fb29ffff640 1 -- 172.21.15.46:0/495031825 <== mds.0 v2:172.21.15.142:6838/2519143918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x562b580086c0 con 0x562b580b0a10 2024-03-31T23:12:37.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.054+0000 7fb29ffff640 10 client.25657 handle_command_reply: tid=0 2024-03-31T23:12:37.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.054+0000 7fb2c9763740 1 client.25657 shutdown 2024-03-31T23:12:37.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.055+0000 7fb2c9763740 1 -- 172.21.15.46:0/495031825 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562b580a0790 msgr2=0x562b5809e210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:12:37.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.055+0000 7fb2c9763740 1 --2- 172.21.15.46:0/495031825 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562b580a0790 0x562b5809e210 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fb2b8036000 tx=0x7fb2b8005c50 comp rx=0 tx=0).stop 2024-03-31T23:12:37.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.055+0000 7fb2c9763740 1 -- 172.21.15.46:0/495031825 shutdown_connections 2024-03-31T23:12:37.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.055+0000 7fb2c9763740 1 --2- 172.21.15.46:0/495031825 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x562b580b0a10 0x562b580b2e50 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:37.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.055+0000 7fb2c9763740 1 --2- 172.21.15.46:0/495031825 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562b580a0790 0x562b5809e210 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:37.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.055+0000 7fb2c9763740 1 --2- 172.21.15.46:0/495031825 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x562b58097bb0 0x562b5809dcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:37.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.055+0000 7fb29ffff640 0 client.25657 ms_handle_reset on v2:172.21.15.142:6838/2519143918 2024-03-31T23:12:37.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.055+0000 7fb2c9763740 1 --2- 172.21.15.46:0/495031825 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562b58097290 0x562b5809d790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:37.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.055+0000 7fb2c9763740 1 -- 172.21.15.46:0/495031825 >> 172.21.15.46:0/495031825 conn(0x562b580919c0 msgr2=0x562b58090c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:12:37.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.055+0000 7fb2c9763740 1 -- 172.21.15.46:0/495031825 shutdown_connections 2024-03-31T23:12:37.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.055+0000 7fb2c9763740 1 -- 172.21.15.46:0/495031825 wait complete. 2024-03-31T23:12:37.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.055+0000 7fb2c9763740 20 client.25657 trim_cache size 0 max 16384 2024-03-31T23:12:37.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.090+0000 7fb2c9763740 1 Processor -- start 2024-03-31T23:12:37.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.090+0000 7fb2c9763740 1 -- start start 2024-03-31T23:12:37.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.090+0000 7fb2c9763740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562b58097290 0x562b58090c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:37.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.090+0000 7fb2c9763740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x562b58097bb0 0x562b580ac260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:37.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.091+0000 7fb2c9763740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562b580a0790 0x562b580b8720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:37.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.091+0000 7fb2c9763740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fb2c01af310 con 0x562b580a0790 2024-03-31T23:12:37.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.091+0000 7fb2c9763740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fb2c01acbe0 con 0x562b58097bb0 2024-03-31T23:12:37.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.091+0000 7fb2c9763740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fb2c01aaa30 con 0x562b58097290 2024-03-31T23:12:37.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.091+0000 7fb2c491d640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562b58097290 0x562b58090c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:37.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.091+0000 7fb2c491d640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562b58097290 0x562b58090c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.46:45326/0 (socket says 172.21.15.46:45326) 2024-03-31T23:12:37.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.091+0000 7fb2c491d640 1 -- 172.21.15.46:0/370480068 learned_addr learned my addr 172.21.15.46:0/370480068 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:12:37.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.091+0000 7fb2bffff640 1 --2- 172.21.15.46:0/370480068 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x562b58097bb0 0x562b580ac260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:37.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.091+0000 7fb2c511e640 1 --2- 172.21.15.46:0/370480068 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562b580a0790 0x562b580b8720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:37.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.091+0000 7fb2c491d640 1 -- 172.21.15.46:0/370480068 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x562b58097bb0 msgr2=0x562b580ac260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:12:37.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.091+0000 7fb2c491d640 1 --2- 172.21.15.46:0/370480068 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x562b58097bb0 0x562b580ac260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:37.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.091+0000 7fb2c491d640 1 -- 172.21.15.46:0/370480068 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562b580a0790 msgr2=0x562b580b8720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:12:37.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.091+0000 7fb2c491d640 1 --2- 172.21.15.46:0/370480068 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562b580a0790 0x562b580b8720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:37.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.091+0000 7fb2c491d640 1 -- 172.21.15.46:0/370480068 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562b57ccf0f0 con 0x562b58097290 2024-03-31T23:12:37.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.091+0000 7fb2bffff640 1 --2- 172.21.15.46:0/370480068 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x562b58097bb0 0x562b580ac260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:12:37.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.091+0000 7fb2c491d640 1 --2- 172.21.15.46:0/370480068 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562b58097290 0x562b58090c20 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fb2b4009380 tx=0x7fb2b409b4c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3b9ee34a8a3d937d server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:12:37.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.091+0000 7fb2aa7fc640 1 -- 172.21.15.46:0/370480068 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2b40cc5a0 con 0x562b58097290 2024-03-31T23:12:37.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.091+0000 7fb2aa7fc640 1 -- 172.21.15.46:0/370480068 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb2b409bcc0 con 0x562b58097290 2024-03-31T23:12:37.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.091+0000 7fb2aa7fc640 1 -- 172.21.15.46:0/370480068 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2b40cf120 con 0x562b58097290 2024-03-31T23:12:37.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.092+0000 7fb2c9763740 1 -- 172.21.15.46:0/370480068 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562b58097290 msgr2=0x562b58090c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:12:37.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.092+0000 7fb2c9763740 1 --2- 172.21.15.46:0/370480068 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562b58097290 0x562b58090c20 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fb2b4009380 tx=0x7fb2b409b4c0 comp rx=0 tx=0).stop 2024-03-31T23:12:37.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.092+0000 7fb2c9763740 1 -- 172.21.15.46:0/370480068 shutdown_connections 2024-03-31T23:12:37.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.092+0000 7fb2c9763740 1 --2- 172.21.15.46:0/370480068 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562b580a0790 0x562b580b8720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:37.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.092+0000 7fb2c9763740 1 --2- 172.21.15.46:0/370480068 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x562b58097bb0 0x562b580ac260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:37.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.092+0000 7fb2c9763740 1 --2- 172.21.15.46:0/370480068 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562b58097290 0x562b58090c20 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:37.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.092+0000 7fb2c9763740 1 -- 172.21.15.46:0/370480068 >> 172.21.15.46:0/370480068 conn(0x562b580919c0 msgr2=0x562b5809d750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:12:37.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.092+0000 7fb2c9763740 1 -- 172.21.15.46:0/370480068 shutdown_connections 2024-03-31T23:12:37.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.092+0000 7fb2c9763740 1 -- 172.21.15.46:0/370480068 wait complete. 2024-03-31T23:12:37.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.093+0000 7fb2c9763740 1 -- start start 2024-03-31T23:12:37.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.093+0000 7fb2c9763740 1 Processor -- start 2024-03-31T23:12:37.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.093+0000 7fb2c9763740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562b58097290 0x562b580a87c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:37.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.093+0000 7fb2c9763740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562b58097bb0 0x562b580a8d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:37.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.093+0000 7fb2c9763740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x562b580a0790 0x562b580a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:37.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.093+0000 7fb2c9763740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fb2c01ab0a0 con 0x562b58097290 2024-03-31T23:12:37.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.093+0000 7fb2c9763740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fb2c01ac510 con 0x562b580a0790 2024-03-31T23:12:37.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.093+0000 7fb2c9763740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fb2c01ab770 con 0x562b58097bb0 2024-03-31T23:12:37.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.093+0000 7fb2c9763740 10 client.0 _ll_register_callbacks cb 0x7fb2b4005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T23:12:37.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.093+0000 7fb2bffff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562b58097bb0 0x562b580a8d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:37.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.094+0000 7fb2bffff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562b58097bb0 0x562b580a8d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.46:45334/0 (socket says 172.21.15.46:45334) 2024-03-31T23:12:37.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.094+0000 7fb2c491d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562b58097290 0x562b580a87c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:37.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.094+0000 7fb2bffff640 1 -- 172.21.15.46:0/698165474 learned_addr learned my addr 172.21.15.46:0/698165474 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:12:37.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.094+0000 7fb2c511e640 1 --2- 172.21.15.46:0/698165474 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x562b580a0790 0x562b580a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:37.100 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.094+0000 7fb2bffff640 1 -- 172.21.15.46:0/698165474 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x562b580a0790 msgr2=0x562b580a4d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:12:37.100 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.094+0000 7fb2bffff640 1 --2- 172.21.15.46:0/698165474 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x562b580a0790 0x562b580a4d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:37.100 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.094+0000 7fb2bffff640 1 -- 172.21.15.46:0/698165474 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562b58097290 msgr2=0x562b580a87c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:12:37.100 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.094+0000 7fb2bffff640 1 --2- 172.21.15.46:0/698165474 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562b58097290 0x562b580a87c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:37.100 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.094+0000 7fb2bffff640 1 -- 172.21.15.46:0/698165474 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562b580a5540 con 0x562b58097bb0 2024-03-31T23:12:37.100 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.094+0000 7fb2c491d640 1 --2- 172.21.15.46:0/698165474 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562b58097290 0x562b580a87c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:12:37.100 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.094+0000 7fb2bffff640 1 --2- 172.21.15.46:0/698165474 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562b58097bb0 0x562b580a8d00 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fb2c00fdaf0 tx=0x7fb2ac00f790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9b67e9eb98bfcd2c server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:12:37.100 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.094+0000 7fb29ffff640 10 client.0 ms_handle_connect on v2:172.21.15.162:3300/0 2024-03-31T23:12:37.100 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.094+0000 7fb29ffff640 1 -- 172.21.15.46:0/698165474 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2ac027060 con 0x562b58097bb0 2024-03-31T23:12:37.100 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.094+0000 7fb29ffff640 1 -- 172.21.15.46:0/698165474 <== mon.1 v2:172.21.15.162:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7fb2ac004460 con 0x562b58097bb0 2024-03-31T23:12:37.100 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.094+0000 7fb2c9763740 1 -- 172.21.15.46:0/698165474 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb2b409bcc0 con 0x562b58097bb0 2024-03-31T23:12:37.100 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.095+0000 7fb29ffff640 1 -- 172.21.15.46:0/698165474 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fb2ac02e050 con 0x562b58097bb0 2024-03-31T23:12:37.100 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.095+0000 7fb2c9763740 10 client.25663 fetch_fsmap learned FSMap version 25 2024-03-31T23:12:37.100 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.095+0000 7fb2c9763740 1 -- 172.21.15.46:0/698165474 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x562b58117a90 con 0x562b58097bb0 2024-03-31T23:12:37.100 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.095+0000 7fb29ffff640 1 -- 172.21.15.46:0/698165474 <== mon.1 v2:172.21.15.162:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb2ac0165c0 con 0x562b58097bb0 2024-03-31T23:12:37.101 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.095+0000 7fb29ffff640 1 -- 172.21.15.46:0/698165474 <== mon.1 v2:172.21.15.162:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7fb2ac016ce0 con 0x562b58097bb0 2024-03-31T23:12:37.101 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.095+0000 7fb2c9763740 10 client.25663 fetch_fsmap finished waiting for FSMap version 25 2024-03-31T23:12:37.101 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.095+0000 7fb2c9763740 10 client.25663 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-31T23:12:37.101 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.095+0000 7fb2c9763740 20 client.25663 populate_metadata read hostname 'smithi046' 2024-03-31T23:12:37.101 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.095+0000 7fb2c9763740 1 --2- 172.21.15.46:0/698165474 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x562b581181e0 0x562b5811a600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:37.101 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.095+0000 7fb2c9763740 4 client.25663 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-31T23:12:37.101 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.095+0000 7fb2c9763740 1 -- 172.21.15.46:0/698165474 --> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] -- command(tid 0: {"prefix": "scrub status"}) -- 0x7fb2b409bcc0 con 0x562b581181e0 2024-03-31T23:12:37.101 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.096+0000 7fb2c491d640 1 --2- 172.21.15.46:0/698165474 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x562b581181e0 0x562b5811a600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:37.101 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.096+0000 7fb29ffff640 10 client.25663 ms_handle_connect on v2:172.21.15.142:6838/2519143918 2024-03-31T23:12:37.101 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.096+0000 7fb2c491d640 1 --2- 172.21.15.46:0/698165474 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x562b581181e0 0x562b5811a600 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=482d10ebecb1ff8 server_cookie=505d16e35da0614b in_seq=0 out_seq=0 2024-03-31T23:12:37.101 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.096+0000 7fb29ffff640 1 -- 172.21.15.46:0/698165474 <== mds.0 v2:172.21.15.142:6838/2519143918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x7fb2b409bcc0 con 0x562b581181e0 2024-03-31T23:12:37.101 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.096+0000 7fb29ffff640 10 client.25663 handle_command_reply: tid=0 2024-03-31T23:12:37.101 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.096+0000 7fb2c9763740 1 client.25663 shutdown 2024-03-31T23:12:37.101 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.096+0000 7fb2c9763740 1 -- 172.21.15.46:0/698165474 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562b58097bb0 msgr2=0x562b580a8d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:12:37.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.096+0000 7fb2c9763740 1 --2- 172.21.15.46:0/698165474 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562b58097bb0 0x562b580a8d00 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fb2c00fdaf0 tx=0x7fb2ac00f790 comp rx=0 tx=0).stop 2024-03-31T23:12:37.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.097+0000 7fb2c9763740 1 -- 172.21.15.46:0/698165474 shutdown_connections 2024-03-31T23:12:37.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.097+0000 7fb2c9763740 1 --2- 172.21.15.46:0/698165474 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x562b581181e0 0x562b5811a600 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:37.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.097+0000 7fb29ffff640 0 client.25663 ms_handle_reset on v2:172.21.15.142:6838/2519143918 2024-03-31T23:12:37.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.097+0000 7fb2c9763740 1 --2- 172.21.15.46:0/698165474 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x562b580a0790 0x562b580a4d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:37.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.097+0000 7fb2c9763740 1 --2- 172.21.15.46:0/698165474 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562b58097bb0 0x562b580a8d00 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:37.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.097+0000 7fb2c9763740 1 --2- 172.21.15.46:0/698165474 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562b58097290 0x562b580a87c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:37.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.097+0000 7fb2c9763740 1 -- 172.21.15.46:0/698165474 >> 172.21.15.46:0/698165474 conn(0x562b580919c0 msgr2=0x562b58090aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:12:37.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.097+0000 7fb2c9763740 1 -- 172.21.15.46:0/698165474 shutdown_connections 2024-03-31T23:12:37.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.097+0000 7fb2c9763740 1 -- 172.21.15.46:0/698165474 wait complete. 2024-03-31T23:12:37.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.097+0000 7fb2c9763740 20 client.25663 trim_cache size 0 max 16384 2024-03-31T23:12:37.102 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-03-31T23:12:37.103 INFO:teuthology.orchestra.run.smithi046.stdout: "status": "no active scrubs running", 2024-03-31T23:12:37.103 INFO:teuthology.orchestra.run.smithi046.stdout: "scrubs": {} 2024-03-31T23:12:37.103 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-03-31T23:12:37.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.097+0000 7fb2c6b81640 1 -- 172.21.15.46:0/790153342 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fb2b00a2730 msgr2=0x7fb2b00a4bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:12:37.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.097+0000 7fb2c6b81640 1 --2- 172.21.15.46:0/790153342 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fb2b00a2730 0x7fb2b00a4bf0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fb2b8004af0 tx=0x7fb2b8009960 comp rx=0 tx=0).stop 2024-03-31T23:12:37.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.097+0000 7fb2c6b81640 1 -- 172.21.15.46:0/790153342 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb2c00ffbf0 msgr2=0x7fb2c0197ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:12:37.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.097+0000 7fb2c6b81640 1 --2- 172.21.15.46:0/790153342 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb2c00ffbf0 0x7fb2c0197ef0 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7fb2b400e9f0 tx=0x7fb2b400edf0 comp rx=0 tx=0).stop 2024-03-31T23:12:37.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.097+0000 7fb2c6b81640 1 -- 172.21.15.46:0/790153342 shutdown_connections 2024-03-31T23:12:37.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.097+0000 7fb2c6b81640 1 --2- 172.21.15.46:0/790153342 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fb2b00a2730 0x7fb2b00a4bf0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:37.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.097+0000 7fb2c6b81640 1 --2- 172.21.15.46:0/790153342 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb2c01076f0 0x7fb2c019c7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:37.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.097+0000 7fb2c6b81640 1 --2- 172.21.15.46:0/790153342 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb2c01005b0 0x7fb2c0198430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:37.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.097+0000 7fb2c6b81640 1 --2- 172.21.15.46:0/790153342 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb2c00ffbf0 0x7fb2c0197ef0 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:37.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.098+0000 7fb2c6b81640 1 -- 172.21.15.46:0/790153342 >> 172.21.15.46:0/790153342 conn(0x7fb2c0076b70 msgr2=0x7fb2c00fe2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:12:37.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.098+0000 7fb2c6b81640 1 -- 172.21.15.46:0/790153342 shutdown_connections 2024-03-31T23:12:37.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.098+0000 7fb2c6b81640 1 -- 172.21.15.46:0/790153342 wait complete. 2024-03-31T23:12:37.110 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-31T23:12:37.110 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-31T23:12:37.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.219+0000 7f2ee4655640 1 -- 172.21.15.46:0/1123649231 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2edc106660 msgr2=0x7f2edc106a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:12:37.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.219+0000 7f2ee4655640 1 --2- 172.21.15.46:0/1123649231 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2edc106660 0x7f2edc106a40 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f2ed8009900 tx=0x7f2ed802bc30 comp rx=0 tx=0).stop 2024-03-31T23:12:37.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.219+0000 7f2ee4655640 1 -- 172.21.15.46:0/1123649231 shutdown_connections 2024-03-31T23:12:37.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.219+0000 7f2ee4655640 1 --2- 172.21.15.46:0/1123649231 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2edc101010 0x7f2edc10d520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:37.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.219+0000 7f2ee4655640 1 --2- 172.21.15.46:0/1123649231 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2edc100650 0x7f2edc100ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:37.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.219+0000 7f2ee4655640 1 --2- 172.21.15.46:0/1123649231 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2edc106660 0x7f2edc106a40 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:37.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.219+0000 7f2ee4655640 1 -- 172.21.15.46:0/1123649231 >> 172.21.15.46:0/1123649231 conn(0x7f2edc077750 msgr2=0x7f2edc077b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:12:37.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.219+0000 7f2ee4655640 1 -- 172.21.15.46:0/1123649231 shutdown_connections 2024-03-31T23:12:37.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.219+0000 7f2ee4655640 1 -- 172.21.15.46:0/1123649231 wait complete. 2024-03-31T23:12:37.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.220+0000 7f2ee4655640 1 Processor -- start 2024-03-31T23:12:37.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.220+0000 7f2ee4655640 1 -- start start 2024-03-31T23:12:37.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.220+0000 7f2ee4655640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2edc100650 0x7f2edc197f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:37.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.220+0000 7f2ee4655640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2edc101010 0x7f2edc198480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:37.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.220+0000 7f2ee4655640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2edc106660 0x7f2edc19c800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:37.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.220+0000 7f2ee4655640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f2edc110920 con 0x7f2edc106660 2024-03-31T23:12:37.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.220+0000 7f2ee4655640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f2edc1107a0 con 0x7f2edc101010 2024-03-31T23:12:37.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.220+0000 7f2ee4655640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f2edc110aa0 con 0x7f2edc100650 2024-03-31T23:12:37.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.220+0000 7f2ee2bf2640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2edc106660 0x7f2edc19c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:37.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.220+0000 7f2ee23f1640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2edc100650 0x7f2edc197f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:37.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.220+0000 7f2ee1bf0640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2edc101010 0x7f2edc198480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:37.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.220+0000 7f2ee23f1640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2edc100650 0x7f2edc197f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.46:45360/0 (socket says 172.21.15.46:45360) 2024-03-31T23:12:37.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.220+0000 7f2ee23f1640 1 -- 172.21.15.46:0/3227206547 learned_addr learned my addr 172.21.15.46:0/3227206547 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:12:37.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.221+0000 7f2ee23f1640 1 -- 172.21.15.46:0/3227206547 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2edc101010 msgr2=0x7f2edc198480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:12:37.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.221+0000 7f2ee23f1640 1 --2- 172.21.15.46:0/3227206547 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2edc101010 0x7f2edc198480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:37.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.221+0000 7f2ee23f1640 1 -- 172.21.15.46:0/3227206547 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2edc106660 msgr2=0x7f2edc19c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:12:37.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.221+0000 7f2ee23f1640 1 --2- 172.21.15.46:0/3227206547 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2edc106660 0x7f2edc19c800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:37.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.221+0000 7f2ee23f1640 1 -- 172.21.15.46:0/3227206547 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2edc19cf30 con 0x7f2edc100650 2024-03-31T23:12:37.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.221+0000 7f2ee2bf2640 1 --2- 172.21.15.46:0/3227206547 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2edc106660 0x7f2edc19c800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:12:37.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.221+0000 7f2ee23f1640 1 --2- 172.21.15.46:0/3227206547 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2edc100650 0x7f2edc197f40 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f2ed8006010 tx=0x7f2ed8005d90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:12:37.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.221+0000 7f2ed37fe640 1 -- 172.21.15.46:0/3227206547 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ed8043020 con 0x7f2edc100650 2024-03-31T23:12:37.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.221+0000 7f2ed37fe640 1 -- 172.21.15.46:0/3227206547 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f2ed8004350 con 0x7f2edc100650 2024-03-31T23:12:37.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.221+0000 7f2ed37fe640 1 -- 172.21.15.46:0/3227206547 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ed8003a10 con 0x7f2edc100650 2024-03-31T23:12:37.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.221+0000 7f2ee4655640 1 -- 172.21.15.46:0/3227206547 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2edc19d1c0 con 0x7f2edc100650 2024-03-31T23:12:37.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.221+0000 7f2ee4655640 1 -- 172.21.15.46:0/3227206547 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2edc102c40 con 0x7f2edc100650 2024-03-31T23:12:37.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.223+0000 7f2ee4655640 1 -- 172.21.15.46:0/3227206547 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2edc101d90 con 0x7f2edc100650 2024-03-31T23:12:37.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.224+0000 7f2ed37fe640 1 -- 172.21.15.46:0/3227206547 <== mon.1 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 24) ==== 137553+0+0 (secure 0 0 0) 0x7f2ed8046020 con 0x7f2edc100650 2024-03-31T23:12:37.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.225+0000 7f2ed37fe640 1 --2- 172.21.15.46:0/3227206547 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f2ec80a27b0 0x7f2ec80a4c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:37.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.225+0000 7f2ee1bf0640 1 --2- 172.21.15.46:0/3227206547 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f2ec80a27b0 0x7f2ec80a4c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:37.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.225+0000 7f2ed37fe640 1 -- 172.21.15.46:0/3227206547 <== mon.1 v2:172.21.15.162:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7f2ed80f0b00 con 0x7f2edc100650 2024-03-31T23:12:37.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.226+0000 7f2ee1bf0640 1 --2- 172.21.15.46:0/3227206547 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f2ec80a27b0 0x7f2ec80a4c70 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f2edc100290 tx=0x7f2ecc005f90 comp rx=0 tx=0).ready entity=mgr.15654 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:12:37.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.230+0000 7f2ed37fe640 1 -- 172.21.15.46:0/3227206547 <== mon.1 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f2ed804d070 con 0x7f2edc100650 2024-03-31T23:12:37.614 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:37.612+0000 7f9864654640 1 -- 172.21.15.46:0/477171899 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f985c068dd0 msgr2=0x7f985c069230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:12:37.614 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:37.612+0000 7f9864654640 1 --2- 172.21.15.46:0/477171899 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f985c068dd0 0x7f985c069230 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f984c009900 tx=0x7f984c02bc30 comp rx=0 tx=0).stop 2024-03-31T23:12:37.614 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:37.613+0000 7f9864654640 1 -- 172.21.15.46:0/477171899 shutdown_connections 2024-03-31T23:12:37.614 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:37.613+0000 7f9864654640 1 --2- 172.21.15.46:0/477171899 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f985c069770 0x7f985c10e5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:37.614 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:37.613+0000 7f9864654640 1 --2- 172.21.15.46:0/477171899 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f985c068dd0 0x7f985c069230 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:37.614 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:37.613+0000 7f9864654640 1 --2- 172.21.15.46:0/477171899 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f985c107840 0x7f985c107c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:37.614 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:37.613+0000 7f9864654640 1 -- 172.21.15.46:0/477171899 >> 172.21.15.46:0/477171899 conn(0x7f985c0774a0 msgr2=0x7f985c0778b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:12:37.615 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:37.613+0000 7f9864654640 1 -- 172.21.15.46:0/477171899 shutdown_connections 2024-03-31T23:12:37.615 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:37.613+0000 7f9864654640 1 -- 172.21.15.46:0/477171899 wait complete. 2024-03-31T23:12:37.617 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:37.613+0000 7f9864654640 1 Processor -- start 2024-03-31T23:12:37.617 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:37.614+0000 7f9864654640 1 -- start start 2024-03-31T23:12:37.618 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:37.614+0000 7f9864654640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f985c068dd0 0x7f985c197f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:37.618 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:37.614+0000 7f9864654640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f985c069770 0x7f985c198450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:37.618 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:37.614+0000 7f9864654640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f985c107840 0x7f985c19c780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:37.618 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:37.614+0000 7f98623f0640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f985c068dd0 0x7f985c197f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:37.618 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:37.614+0000 7f98623f0640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f985c068dd0 0x7f985c197f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60900/0 (socket says 172.21.15.46:60900) 2024-03-31T23:12:37.618 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:37.614+0000 7f98623f0640 1 -- 172.21.15.46:0/3572213777 learned_addr learned my addr 172.21.15.46:0/3572213777 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:12:37.618 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:37.614+0000 7f9864654640 1 -- 172.21.15.46:0/3572213777 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f985c111a70 con 0x7f985c068dd0 2024-03-31T23:12:37.618 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:37.614+0000 7f9861bef640 1 --2- 172.21.15.46:0/3572213777 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f985c069770 0x7f985c198450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:37.618 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:37.614+0000 7f9864654640 1 -- 172.21.15.46:0/3572213777 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f985c1118f0 con 0x7f985c069770 2024-03-31T23:12:37.618 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:37.614+0000 7f9864654640 1 -- 172.21.15.46:0/3572213777 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f985c111bf0 con 0x7f985c107840 2024-03-31T23:12:37.618 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:37.614+0000 7f9862bf1640 1 --2- 172.21.15.46:0/3572213777 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f985c107840 0x7f985c19c780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:37.618 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:37.615+0000 7f98623f0640 1 -- 172.21.15.46:0/3572213777 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f985c107840 msgr2=0x7f985c19c780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:12:37.618 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:37.615+0000 7f98623f0640 1 --2- 172.21.15.46:0/3572213777 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f985c107840 0x7f985c19c780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:37.618 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:37.615+0000 7f98623f0640 1 -- 172.21.15.46:0/3572213777 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f985c069770 msgr2=0x7f985c198450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:12:37.619 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:37.615+0000 7f98623f0640 1 --2- 172.21.15.46:0/3572213777 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f985c069770 0x7f985c198450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:37.619 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:37.615+0000 7f98623f0640 1 -- 172.21.15.46:0/3572213777 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f985c19ceb0 con 0x7f985c068dd0 2024-03-31T23:12:37.619 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:37.615+0000 7f9861bef640 1 --2- 172.21.15.46:0/3572213777 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f985c069770 0x7f985c198450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:12:37.619 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:37.615+0000 7f9862bf1640 1 --2- 172.21.15.46:0/3572213777 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f985c107840 0x7f985c19c780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:12:37.619 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:37.615+0000 7f98623f0640 1 --2- 172.21.15.46:0/3572213777 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f985c068dd0 0x7f985c197f10 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f985800e9f0 tx=0x7f985800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:12:37.619 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:37.616+0000 7f98537fe640 1 -- 172.21.15.46:0/3572213777 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f985800cd80 con 0x7f985c068dd0 2024-03-31T23:12:37.619 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:37.616+0000 7f98537fe640 1 -- 172.21.15.46:0/3572213777 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f98580108e0 con 0x7f985c068dd0 2024-03-31T23:12:37.619 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:37.616+0000 7f9864654640 1 -- 172.21.15.46:0/3572213777 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f985c19d1a0 con 0x7f985c068dd0 2024-03-31T23:12:37.619 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:37.616+0000 7f98537fe640 1 -- 172.21.15.46:0/3572213777 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9858018920 con 0x7f985c068dd0 2024-03-31T23:12:37.619 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:37.616+0000 7f9864654640 1 -- 172.21.15.46:0/3572213777 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f985c1a4a30 con 0x7f985c068dd0 2024-03-31T23:12:37.619 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:37.619+0000 7f98537fe640 1 -- 172.21.15.46:0/3572213777 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 24) ==== 137553+0+0 (secure 0 0 0) 0x7f9858010a90 con 0x7f985c068dd0 2024-03-31T23:12:37.620 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:37.619+0000 7f9867231740 1 Processor -- start 2024-03-31T23:12:37.621 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:37.619+0000 7f9867231740 1 -- start start 2024-03-31T23:12:37.621 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:37.620+0000 7f9867231740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5644065b16c0 0x5644065b1aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:37.621 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:37.620+0000 7f9867231740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5644065b1fe0 0x5644065ba870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:37.621 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:37.620+0000 7f9867231740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5644065badb0 0x5644065bd1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:37.621 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:37.620+0000 7f9867231740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x5644065148f0 con 0x5644065b1fe0 2024-03-31T23:12:37.621 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:37.620+0000 7f9867231740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x5644064bd8d0 con 0x5644065b16c0 2024-03-31T23:12:37.621 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:37.620+0000 7f9867231740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x564406510990 con 0x5644065badb0 2024-03-31T23:12:37.622 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:37.620+0000 7f9861bef640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5644065badb0 0x5644065bd1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:37.622 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:37.620+0000 7f9862bf1640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5644065b16c0 0x5644065b1aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:37.622 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:37.620+0000 7f9861bef640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5644065badb0 0x5644065bd1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.46:45394/0 (socket says 172.21.15.46:45394) 2024-03-31T23:12:37.622 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:37.620+0000 7f98537fe640 1 --2- 172.21.15.46:0/3572213777 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f98480a2850 0x7f98480a4d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:37.622 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:37.620+0000 7f9861bef640 1 -- 172.21.15.46:0/3713477667 learned_addr learned my addr 172.21.15.46:0/3713477667 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:12:37.622 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:37.620+0000 7f9861bef640 1 -- 172.21.15.46:0/3713477667 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5644065b16c0 msgr2=0x5644065b1aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:12:37.622 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:37.620+0000 7f9861bef640 1 --2- 172.21.15.46:0/3713477667 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5644065b16c0 0x5644065b1aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:37.622 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:37.620+0000 7f98537fe640 1 -- 172.21.15.46:0/3572213777 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7f98580cf3f0 con 0x7f985c068dd0 2024-03-31T23:12:37.623 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:37.620+0000 7f9861bef640 1 -- 172.21.15.46:0/3713477667 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5644065b1fe0 msgr2=0x5644065ba870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:12:37.623 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:37.620+0000 7f9861bef640 1 --2- 172.21.15.46:0/3713477667 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5644065b1fe0 0x5644065ba870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:37.623 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:37.620+0000 7f9861bef640 1 -- 172.21.15.46:0/3713477667 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564406229210 con 0x5644065badb0 2024-03-31T23:12:37.623 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:37.620+0000 7f9862bf1640 1 --2- 172.21.15.46:0/3572213777 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f98480a2850 0x7f98480a4d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:37.623 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:37.621+0000 7f9861bef640 1 --2- 172.21.15.46:0/3713477667 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5644065badb0 0x5644065bd1a0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f984c009900 tx=0x7f984c0095f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5d627809e1e53747 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:12:37.623 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:37.621+0000 7f9850bfe640 1 -- 172.21.15.46:0/3713477667 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f984c045070 con 0x5644065badb0 2024-03-31T23:12:37.623 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:37.621+0000 7f9850bfe640 1 -- 172.21.15.46:0/3713477667 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f984c02c7e0 con 0x5644065badb0 2024-03-31T23:12:37.623 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:37.621+0000 7f9850bfe640 1 -- 172.21.15.46:0/3713477667 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f984c003a80 con 0x5644065badb0 2024-03-31T23:12:37.628 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:37.622+0000 7f9867231740 1 -- 172.21.15.46:0/3713477667 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5644065badb0 msgr2=0x5644065bd1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:12:37.628 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:37.622+0000 7f9862bf1640 1 --2- 172.21.15.46:0/3572213777 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f98480a2850 0x7f98480a4d10 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f98540049d0 tx=0x7f9854004ab0 comp rx=0 tx=0).ready entity=mgr.15654 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:12:37.628 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:37.622+0000 7f9867231740 1 --2- 172.21.15.46:0/3713477667 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5644065badb0 0x5644065bd1a0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f984c009900 tx=0x7f984c0095f0 comp rx=0 tx=0).stop 2024-03-31T23:12:37.628 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:37.622+0000 7f9867231740 1 -- 172.21.15.46:0/3713477667 shutdown_connections 2024-03-31T23:12:37.628 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:37.622+0000 7f9867231740 1 --2- 172.21.15.46:0/3713477667 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5644065badb0 0x5644065bd1a0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:37.628 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:37.622+0000 7f9867231740 1 --2- 172.21.15.46:0/3713477667 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5644065b1fe0 0x5644065ba870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:37.628 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:37.622+0000 7f9867231740 1 --2- 172.21.15.46:0/3713477667 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5644065b16c0 0x5644065b1aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:37.629 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:37.622+0000 7f9867231740 1 -- 172.21.15.46:0/3713477667 >> 172.21.15.46:0/3713477667 conn(0x5644065abfe0 msgr2=0x5644065ac3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:12:37.629 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:37.622+0000 7f9867231740 1 -- 172.21.15.46:0/3713477667 shutdown_connections 2024-03-31T23:12:37.629 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:37.622+0000 7f9867231740 1 -- 172.21.15.46:0/3713477667 wait complete. 2024-03-31T23:12:37.629 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:37.622+0000 7f9867231740 1 -- start start 2024-03-31T23:12:37.629 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:37.623+0000 7f9867231740 1 Processor -- start 2024-03-31T23:12:37.629 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:37.623+0000 7f9867231740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5644065b16c0 0x5644065b7e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:37.629 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:37.623+0000 7f9867231740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5644065b1fe0 0x5644065b8390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:37.629 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:37.623+0000 7f9861bef640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5644065b1fe0 0x5644065b8390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:37.629 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:37.623+0000 7f9861bef640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5644065b1fe0 0x5644065b8390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60918/0 (socket says 172.21.15.46:60918) 2024-03-31T23:12:37.629 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:37.623+0000 7f9861bef640 1 -- 172.21.15.46:0/3105627351 learned_addr learned my addr 172.21.15.46:0/3105627351 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:12:37.629 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:37.623+0000 7f98623f0640 1 --2- 172.21.15.46:0/3105627351 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5644065b16c0 0x5644065b7e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:37.629 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:37.623+0000 7f9867231740 1 --2- 172.21.15.46:0/3105627351 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5644065badb0 0x5644065b88d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:37.629 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:37.623+0000 7f9867231740 1 -- 172.21.15.46:0/3105627351 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x5644065c1d90 con 0x5644065b1fe0 2024-03-31T23:12:37.629 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:37.623+0000 7f9867231740 1 -- 172.21.15.46:0/3105627351 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x5644064b2f80 con 0x5644065badb0 2024-03-31T23:12:37.630 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:37.623+0000 7f9867231740 1 -- 172.21.15.46:0/3105627351 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5644065c1f10 con 0x5644065b16c0 2024-03-31T23:12:37.630 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:37.623+0000 7f9862bf1640 1 --2- 172.21.15.46:0/3105627351 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5644065badb0 0x5644065b88d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:37.630 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:37.623+0000 7f98623f0640 1 -- 172.21.15.46:0/3105627351 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5644065badb0 msgr2=0x5644065b88d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:12:37.630 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:37.623+0000 7f98623f0640 1 --2- 172.21.15.46:0/3105627351 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5644065badb0 0x5644065b88d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:37.630 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:37.623+0000 7f98623f0640 1 -- 172.21.15.46:0/3105627351 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5644065b1fe0 msgr2=0x5644065b8390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:12:37.630 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:37.623+0000 7f98623f0640 1 --2- 172.21.15.46:0/3105627351 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5644065b1fe0 0x5644065b8390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:37.630 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:37.623+0000 7f98623f0640 1 -- 172.21.15.46:0/3105627351 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5644062f5620 con 0x5644065b16c0 2024-03-31T23:12:37.630 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:37.624+0000 7f9861bef640 1 --2- 172.21.15.46:0/3105627351 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5644065b1fe0 0x5644065b8390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:12:37.630 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:37.624+0000 7f98623f0640 1 --2- 172.21.15.46:0/3105627351 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5644065b16c0 0x5644065b7e50 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f98580cc610 tx=0x7f985809b250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f63eecc2615bc672 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:12:37.630 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:37.624+0000 7f9845ffb640 10 client.0 ms_handle_connect on v2:172.21.15.162:3300/0 2024-03-31T23:12:37.630 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:37.624+0000 7f9867231740 10 client.0 _ll_register_callbacks cb 0x56440650b290 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T23:12:37.630 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:37.624+0000 7f9867231740 1 -- 172.21.15.46:0/3105627351 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f984c02c7e0 con 0x5644065b16c0 2024-03-31T23:12:37.630 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:37.624+0000 7f9845ffb640 1 -- 172.21.15.46:0/3105627351 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9858084d60 con 0x5644065b16c0 2024-03-31T23:12:37.630 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:37.624+0000 7f9845ffb640 1 -- 172.21.15.46:0/3105627351 <== mon.1 v2:172.21.15.162:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7f9858010430 con 0x5644065b16c0 2024-03-31T23:12:37.631 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:37.624+0000 7f9845ffb640 1 -- 172.21.15.46:0/3105627351 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f9858010630 con 0x5644065b16c0 2024-03-31T23:12:37.631 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:37.624+0000 7f9867231740 10 client.25681 fetch_fsmap learned FSMap version 25 2024-03-31T23:12:37.631 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:37.624+0000 7f9867231740 1 -- 172.21.15.46:0/3105627351 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({fsmap=25}) -- 0x5644065cd8c0 con 0x5644065b16c0 2024-03-31T23:12:37.631 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:37.624+0000 7f9845ffb640 1 -- 172.21.15.46:0/3105627351 <== mon.1 v2:172.21.15.162:3300/0 4 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f98580cc8f0 con 0x5644065b16c0 2024-03-31T23:12:37.631 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:37.625+0000 7f9867231740 10 client.25681 fetch_fsmap finished waiting for FSMap version 25 2024-03-31T23:12:37.631 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:37.625+0000 7f9867231740 10 client.25681 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-31T23:12:37.631 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:37.625+0000 7f9867231740 20 client.25681 populate_metadata read hostname 'smithi046' 2024-03-31T23:12:37.631 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:37.625+0000 7f9867231740 1 --2- 172.21.15.46:0/3105627351 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x5644065ced30 0x5644065d11d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:37.631 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:37.625+0000 7f9867231740 4 client.25681 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-31T23:12:37.631 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:37.625+0000 7f9867231740 1 -- 172.21.15.46:0/3105627351 --> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564406528120 con 0x5644065ced30 2024-03-31T23:12:37.631 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:37.625+0000 7f9861bef640 1 --2- 172.21.15.46:0/3105627351 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x5644065ced30 0x5644065d11d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:37.631 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:37.625+0000 7f9861bef640 1 --2- 172.21.15.46:0/3105627351 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x5644065ced30 0x5644065d11d0 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c3e28bd78d0f0c36 server_cookie=15123d516ca6c553 in_seq=0 out_seq=0 2024-03-31T23:12:37.631 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:37.625+0000 7f9845ffb640 10 client.25681 ms_handle_connect on v2:172.21.15.142:6838/2519143918 2024-03-31T23:12:37.631 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:37.626+0000 7f9845ffb640 1 -- 172.21.15.46:0/3105627351 <== mds.0 v2:172.21.15.142:6838/2519143918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x564406528120 con 0x5644065ced30 2024-03-31T23:12:37.631 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:37.626+0000 7f9845ffb640 10 client.25681 handle_command_reply: tid=0 2024-03-31T23:12:37.631 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:37.626+0000 7f9867231740 1 client.25681 shutdown 2024-03-31T23:12:37.632 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:37.626+0000 7f9867231740 1 -- 172.21.15.46:0/3105627351 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5644065b16c0 msgr2=0x5644065b7e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:12:37.632 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:37.626+0000 7f9867231740 1 --2- 172.21.15.46:0/3105627351 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5644065b16c0 0x5644065b7e50 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f98580cc610 tx=0x7f985809b250 comp rx=0 tx=0).stop 2024-03-31T23:12:37.632 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:37.626+0000 7f9867231740 1 -- 172.21.15.46:0/3105627351 shutdown_connections 2024-03-31T23:12:37.632 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:37.626+0000 7f9867231740 1 --2- 172.21.15.46:0/3105627351 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x5644065ced30 0x5644065d11d0 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:37.632 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:37.626+0000 7f9867231740 1 --2- 172.21.15.46:0/3105627351 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5644065badb0 0x5644065b88d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:37.632 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:37.626+0000 7f9867231740 1 --2- 172.21.15.46:0/3105627351 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5644065b1fe0 0x5644065b8390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:37.632 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:37.626+0000 7f9867231740 1 --2- 172.21.15.46:0/3105627351 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5644065b16c0 0x5644065b7e50 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:37.632 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:37.626+0000 7f9867231740 1 -- 172.21.15.46:0/3105627351 >> 172.21.15.46:0/3105627351 conn(0x5644065abfe0 msgr2=0x5644065add30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:12:37.632 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:37.626+0000 7f9845ffb640 0 client.25681 ms_handle_reset on v2:172.21.15.142:6838/2519143918 2024-03-31T23:12:37.632 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:37.626+0000 7f9867231740 1 -- 172.21.15.46:0/3105627351 shutdown_connections 2024-03-31T23:12:37.632 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:37.626+0000 7f9867231740 1 -- 172.21.15.46:0/3105627351 wait complete. 2024-03-31T23:12:37.632 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:37.626+0000 7f9867231740 20 client.25681 trim_cache size 0 max 16384 2024-03-31T23:12:37.657 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:37.656+0000 7f9867231740 1 Processor -- start 2024-03-31T23:12:37.657 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:37.656+0000 7f9867231740 1 -- start start 2024-03-31T23:12:37.657 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:37.656+0000 7f9867231740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5644065b16c0 0x5644065cab30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:37.657 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:37.656+0000 7f9867231740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5644065b1fe0 0x5644065b86f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:37.658 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:37.657+0000 7f9867231740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5644065badb0 0x5644065aae00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:37.658 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:37.657+0000 7f98623f0640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5644065b16c0 0x5644065cab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:37.658 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:37.657+0000 7f98623f0640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5644065b16c0 0x5644065cab30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.46:51788/0 (socket says 172.21.15.46:51788) 2024-03-31T23:12:37.659 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:37.657+0000 7f9861bef640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5644065b1fe0 0x5644065b86f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:37.659 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:37.657+0000 7f9861bef640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5644065b1fe0 0x5644065b86f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.46:45410/0 (socket says 172.21.15.46:45410) 2024-03-31T23:12:37.659 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:37.657+0000 7f9862bf1640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5644065badb0 0x5644065aae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:37.659 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:37.657+0000 7f9862bf1640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5644065badb0 0x5644065aae00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60932/0 (socket says 172.21.15.46:60932) 2024-03-31T23:12:37.660 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:37.657+0000 7f98623f0640 1 -- 172.21.15.46:0/1843408568 learned_addr learned my addr 172.21.15.46:0/1843408568 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:12:37.660 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:37.657+0000 7f9867231740 1 -- 172.21.15.46:0/1843408568 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f985c1af120 con 0x5644065badb0 2024-03-31T23:12:37.660 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:37.657+0000 7f9867231740 1 -- 172.21.15.46:0/1843408568 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f985c1ac9f0 con 0x5644065b16c0 2024-03-31T23:12:37.660 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:37.657+0000 7f9867231740 1 -- 172.21.15.46:0/1843408568 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f985c1aa840 con 0x5644065b1fe0 2024-03-31T23:12:37.660 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:37.657+0000 7f9861bef640 1 -- 172.21.15.46:0/1843408568 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5644065b16c0 msgr2=0x5644065cab30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:12:37.660 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:37.657+0000 7f9861bef640 1 --2- 172.21.15.46:0/1843408568 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5644065b16c0 0x5644065cab30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:37.660 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:37.657+0000 7f9861bef640 1 -- 172.21.15.46:0/1843408568 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5644065badb0 msgr2=0x5644065aae00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:12:37.660 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:37.657+0000 7f9861bef640 1 --2- 172.21.15.46:0/1843408568 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5644065badb0 0x5644065aae00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:37.660 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:37.657+0000 7f9861bef640 1 -- 172.21.15.46:0/1843408568 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5644065cb070 con 0x5644065b1fe0 2024-03-31T23:12:37.660 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:37.657+0000 7f9862bf1640 1 --2- 172.21.15.46:0/1843408568 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5644065badb0 0x5644065aae00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:12:37.660 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:37.658+0000 7f9861bef640 1 --2- 172.21.15.46:0/1843408568 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5644065b1fe0 0x5644065b86f0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x5644065d1a80 tx=0x7f984c004590 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c4261544afe78cfd server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:12:37.660 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:37.658+0000 7f9847fff640 1 -- 172.21.15.46:0/1843408568 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f984c045040 con 0x5644065b1fe0 2024-03-31T23:12:37.660 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:37.658+0000 7f9847fff640 1 -- 172.21.15.46:0/1843408568 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f984c034580 con 0x5644065b1fe0 2024-03-31T23:12:37.660 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:37.658+0000 7f9847fff640 1 -- 172.21.15.46:0/1843408568 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f984c04aa90 con 0x5644065b1fe0 2024-03-31T23:12:37.664 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:37.659+0000 7f9867231740 1 -- 172.21.15.46:0/1843408568 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5644065b1fe0 msgr2=0x5644065b86f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:12:37.664 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:37.659+0000 7f9867231740 1 --2- 172.21.15.46:0/1843408568 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5644065b1fe0 0x5644065b86f0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x5644065d1a80 tx=0x7f984c004590 comp rx=0 tx=0).stop 2024-03-31T23:12:37.664 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:37.659+0000 7f9867231740 1 -- 172.21.15.46:0/1843408568 shutdown_connections 2024-03-31T23:12:37.664 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:37.659+0000 7f9867231740 1 --2- 172.21.15.46:0/1843408568 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5644065badb0 0x5644065aae00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:37.665 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:37.659+0000 7f9867231740 1 --2- 172.21.15.46:0/1843408568 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5644065b1fe0 0x5644065b86f0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:37.665 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:37.659+0000 7f9867231740 1 --2- 172.21.15.46:0/1843408568 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5644065b16c0 0x5644065cab30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:37.665 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:37.659+0000 7f9867231740 1 -- 172.21.15.46:0/1843408568 >> 172.21.15.46:0/1843408568 conn(0x5644065abfe0 msgr2=0x5644065aa6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:12:37.665 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:37.659+0000 7f9867231740 1 -- 172.21.15.46:0/1843408568 shutdown_connections 2024-03-31T23:12:37.665 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:37.659+0000 7f9867231740 1 -- 172.21.15.46:0/1843408568 wait complete. 2024-03-31T23:12:37.665 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:37.660+0000 7f9867231740 1 -- start start 2024-03-31T23:12:37.665 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:37.660+0000 7f9867231740 1 Processor -- start 2024-03-31T23:12:37.665 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:37.660+0000 7f9867231740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5644065b16c0 0x5644065c2310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:37.665 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:37.660+0000 7f9867231740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5644065b1fe0 0x5644065c2850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:37.665 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:37.660+0000 7f98623f0640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5644065b16c0 0x5644065c2310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:37.665 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:37.661+0000 7f9861bef640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5644065b1fe0 0x5644065c2850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:37.665 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:37.661+0000 7f98623f0640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5644065b16c0 0x5644065c2310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.46:45426/0 (socket says 172.21.15.46:45426) 2024-03-31T23:12:37.665 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:37.661+0000 7f9861bef640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5644065b1fe0 0x5644065c2850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60934/0 (socket says 172.21.15.46:60934) 2024-03-31T23:12:37.665 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:37.661+0000 7f98623f0640 1 -- 172.21.15.46:0/869025644 learned_addr learned my addr 172.21.15.46:0/869025644 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:12:37.666 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:37.661+0000 7f9867231740 1 --2- 172.21.15.46:0/869025644 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5644065badb0 0x564406633100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:37.666 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:37.661+0000 7f9867231740 1 -- 172.21.15.46:0/869025644 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f985c1aaeb0 con 0x5644065b1fe0 2024-03-31T23:12:37.666 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:37.661+0000 7f9867231740 1 -- 172.21.15.46:0/869025644 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f985c1ac320 con 0x5644065badb0 2024-03-31T23:12:37.666 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:37.661+0000 7f9862bf1640 1 --2- 172.21.15.46:0/869025644 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5644065badb0 0x564406633100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:37.666 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:37.661+0000 7f9867231740 1 -- 172.21.15.46:0/869025644 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f985c1ab580 con 0x5644065b16c0 2024-03-31T23:12:37.666 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:37.661+0000 7f9867231740 10 client.0 _ll_register_callbacks cb 0x7f98580b9f50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T23:12:37.666 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:37.661+0000 7f98623f0640 1 -- 172.21.15.46:0/869025644 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5644065badb0 msgr2=0x564406633100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:12:37.666 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:37.661+0000 7f98623f0640 1 --2- 172.21.15.46:0/869025644 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5644065badb0 0x564406633100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:37.666 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:37.661+0000 7f98623f0640 1 -- 172.21.15.46:0/869025644 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5644065b1fe0 msgr2=0x5644065c2850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:12:37.666 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:37.662+0000 7f98623f0640 1 --2- 172.21.15.46:0/869025644 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5644065b1fe0 0x5644065c2850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:37.666 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:37.662+0000 7f98623f0640 1 -- 172.21.15.46:0/869025644 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564406633670 con 0x5644065b16c0 2024-03-31T23:12:37.666 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:37.662+0000 7f9861bef640 1 --2- 172.21.15.46:0/869025644 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5644065b1fe0 0x5644065c2850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:12:37.667 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:37.662+0000 7f98623f0640 1 --2- 172.21.15.46:0/869025644 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5644065b16c0 0x5644065c2310 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f985809b5b0 tx=0x7f98580cf6c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9192e5c0400a4c03 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:12:37.667 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:37.662+0000 7f9845ffb640 10 client.0 ms_handle_connect on v2:172.21.15.162:3300/0 2024-03-31T23:12:37.667 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:37.662+0000 7f9845ffb640 1 -- 172.21.15.46:0/869025644 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f985809b160 con 0x5644065b16c0 2024-03-31T23:12:37.667 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:37.662+0000 7f9845ffb640 1 -- 172.21.15.46:0/869025644 <== mon.1 v2:172.21.15.162:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7f985809aa70 con 0x5644065b16c0 2024-03-31T23:12:37.667 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:37.662+0000 7f9867231740 1 -- 172.21.15.46:0/869025644 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x564406521e30 con 0x5644065b16c0 2024-03-31T23:12:37.667 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:37.662+0000 7f9845ffb640 1 -- 172.21.15.46:0/869025644 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f985809b3f0 con 0x5644065b16c0 2024-03-31T23:12:37.667 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:37.662+0000 7f9867231740 10 client.25687 fetch_fsmap learned FSMap version 25 2024-03-31T23:12:37.667 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:37.662+0000 7f9867231740 1 -- 172.21.15.46:0/869025644 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x564406634940 con 0x5644065b16c0 2024-03-31T23:12:37.667 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:37.662+0000 7f9845ffb640 1 -- 172.21.15.46:0/869025644 <== mon.1 v2:172.21.15.162:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x564406521e30 con 0x5644065b16c0 2024-03-31T23:12:37.667 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:37.663+0000 7f9845ffb640 1 -- 172.21.15.46:0/869025644 <== mon.1 v2:172.21.15.162:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f985809f160 con 0x5644065b16c0 2024-03-31T23:12:37.667 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:37.663+0000 7f9867231740 10 client.25687 fetch_fsmap finished waiting for FSMap version 25 2024-03-31T23:12:37.667 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:37.663+0000 7f9867231740 10 client.25687 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-31T23:12:37.667 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:37.663+0000 7f9867231740 20 client.25687 populate_metadata read hostname 'smithi046' 2024-03-31T23:12:37.668 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:37.663+0000 7f9867231740 1 --2- 172.21.15.46:0/869025644 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x564406634ec0 0x5644066372b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:37.668 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:37.663+0000 7f9867231740 4 client.25687 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-31T23:12:37.668 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:37.663+0000 7f9867231740 1 -- 172.21.15.46:0/869025644 --> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] -- command(tid 0: {"prefix": "flush journal"}) -- 0x56440651c180 con 0x564406634ec0 2024-03-31T23:12:37.668 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:37.663+0000 7f9861bef640 1 --2- 172.21.15.46:0/869025644 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x564406634ec0 0x5644066372b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:37.668 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:37.663+0000 7f9845ffb640 10 client.25687 ms_handle_connect on v2:172.21.15.142:6838/2519143918 2024-03-31T23:12:37.668 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:37.663+0000 7f9861bef640 1 --2- 172.21.15.46:0/869025644 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x564406634ec0 0x5644066372b0 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=608d0e1d9700d84f server_cookie=8332fd030437fdfd in_seq=0 out_seq=0 2024-03-31T23:12:37.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.663+0000 7f2ee4655640 1 -- 172.21.15.46:0/3227206547 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2edc199290 con 0x7f2edc100650 2024-03-31T23:12:37.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.664+0000 7f2ed37fe640 1 -- 172.21.15.46:0/3227206547 <== mon.1 v2:172.21.15.162:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+11882 (secure 0 0 0) 0x7f2ed80bd490 con 0x7f2edc100650 2024-03-31T23:12:37.669 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-31T23:12:37.669 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/2169018547","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":2169018547},{"type":"v1","addr":"172.21.15.46:6835","nonce":2169018547}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6837/1069387756","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6836","nonce":1069387756},{"type":"v1","addr":"172.21.15.46:6837","nonce":1069387756}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14634,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6839/3915337844","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6838","nonce":3915337844},{"type":"v1","addr":"172.21.15.46:6839","nonce":3915337844}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14646,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6841/897767145","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6840","nonce":897767145},{"type":"v1","addr":"172.21.15.46:6841","nonce":897767145}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24415,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6833/1746950966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6832","nonce":1746950966},{"type":"v1","addr":"172.21.15.162:6833","nonce":1746950966}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24422,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6833/3394739181","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6832","nonce":3394739181},{"type":"v1","addr":"172.21.15.142:6833","nonce":3394739181}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":25,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-31T23:09:11.009090+0000","modified":"2024-03-31T23:11:27.558846+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24515,"mds_1":24482,"mds_2":24451},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24442":{"gid":24442,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.162:6835/1957208984","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6834","nonce":1957208984},{"type":"v1","addr":"172.21.15.162:6835","nonce":1957208984}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24451":{"gid":24451,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":11,"addr":"172.21.15.162:6837/1371886876","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6836","nonce":1371886876},{"type":"v1","addr":"172.21.15.162:6837","nonce":1371886876}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24455":{"gid":24455,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.142:6835/212637431","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6834","nonce":212637431},{"type":"v1","addr":"172.21.15.142:6835","nonce":212637431}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24478":{"gid":24478,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.162:6839/1976275826","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6838","nonce":1976275826},{"type":"v1","addr":"172.21.15.162:6839","nonce":1976275826}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24482":{"gid":24482,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":10,"addr":"172.21.15.142:6837/2559258399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6836","nonce":2559258399},{"type":"v1","addr":"172.21.15.142:6837","nonce":2559258399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24515":{"gid":24515,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.142:6839/2519143918","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6838","nonce":2519143918},{"type":"v1","addr":"172.21.15.142:6839","nonce":2519143918}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24515,"qdb_cluster":[24515,24451,24482]},"id":1}]} 2024-03-31T23:12:37.669 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 25 2024-03-31T23:12:37.670 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:37.666+0000 7f9845ffb640 1 -- 172.21.15.46:0/869025644 <== mds.0 v2:172.21.15.142:6838/2519143918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x56440651c180 con 0x564406634ec0 2024-03-31T23:12:37.671 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:37.666+0000 7f9845ffb640 10 client.25687 handle_command_reply: tid=0 2024-03-31T23:12:37.671 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:37.667+0000 7f9867231740 1 client.25687 shutdown 2024-03-31T23:12:37.671 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:37.667+0000 7f9867231740 1 -- 172.21.15.46:0/869025644 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5644065b16c0 msgr2=0x5644065c2310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:12:37.671 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:37.667+0000 7f9867231740 1 --2- 172.21.15.46:0/869025644 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5644065b16c0 0x5644065c2310 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f985809b5b0 tx=0x7f98580cf6c0 comp rx=0 tx=0).stop 2024-03-31T23:12:37.671 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:37.667+0000 7f9867231740 1 -- 172.21.15.46:0/869025644 shutdown_connections 2024-03-31T23:12:37.671 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:37.667+0000 7f9867231740 1 --2- 172.21.15.46:0/869025644 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x564406634ec0 0x5644066372b0 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:37.671 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:37.667+0000 7f9867231740 1 --2- 172.21.15.46:0/869025644 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5644065badb0 0x564406633100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:37.671 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:37.667+0000 7f9867231740 1 --2- 172.21.15.46:0/869025644 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5644065b1fe0 0x5644065c2850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:37.671 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:37.667+0000 7f9867231740 1 --2- 172.21.15.46:0/869025644 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5644065b16c0 0x5644065c2310 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:37.671 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:37.667+0000 7f9867231740 1 -- 172.21.15.46:0/869025644 >> 172.21.15.46:0/869025644 conn(0x5644065abfe0 msgr2=0x5644065aaf80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:12:37.671 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:37.667+0000 7f9845ffb640 0 client.25687 ms_handle_reset on v2:172.21.15.142:6838/2519143918 2024-03-31T23:12:37.671 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:37.667+0000 7f9867231740 1 -- 172.21.15.46:0/869025644 shutdown_connections 2024-03-31T23:12:37.671 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:37.667+0000 7f9867231740 1 -- 172.21.15.46:0/869025644 wait complete. 2024-03-31T23:12:37.671 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:37.667+0000 7f9867231740 20 client.25687 trim_cache size 0 max 16384 2024-03-31T23:12:37.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.667+0000 7f2ee4655640 1 -- 172.21.15.46:0/3227206547 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f2ec80a27b0 msgr2=0x7f2ec80a4c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:12:37.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.667+0000 7f2ee4655640 1 --2- 172.21.15.46:0/3227206547 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f2ec80a27b0 0x7f2ec80a4c70 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f2edc100290 tx=0x7f2ecc005f90 comp rx=0 tx=0).stop 2024-03-31T23:12:37.672 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout:{ 2024-03-31T23:12:37.672 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout: "message": "", 2024-03-31T23:12:37.672 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout: "return_code": 0 2024-03-31T23:12:37.672 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout:} 2024-03-31T23:12:37.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.668+0000 7f2ee4655640 1 -- 172.21.15.46:0/3227206547 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2edc100650 msgr2=0x7f2edc197f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:12:37.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.668+0000 7f2ee4655640 1 --2- 172.21.15.46:0/3227206547 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2edc100650 0x7f2edc197f40 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f2ed8006010 tx=0x7f2ed8005d90 comp rx=0 tx=0).stop 2024-03-31T23:12:37.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.668+0000 7f2ee4655640 1 -- 172.21.15.46:0/3227206547 shutdown_connections 2024-03-31T23:12:37.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.668+0000 7f2ee4655640 1 --2- 172.21.15.46:0/3227206547 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f2ec80a27b0 0x7f2ec80a4c70 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:37.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.668+0000 7f2ee4655640 1 --2- 172.21.15.46:0/3227206547 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2edc106660 0x7f2edc19c800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:37.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.668+0000 7f2ee4655640 1 --2- 172.21.15.46:0/3227206547 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2edc101010 0x7f2edc198480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:37.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.668+0000 7f2ee4655640 1 --2- 172.21.15.46:0/3227206547 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2edc100650 0x7f2edc197f40 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:37.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.668+0000 7f2ee4655640 1 -- 172.21.15.46:0/3227206547 >> 172.21.15.46:0/3227206547 conn(0x7f2edc077750 msgr2=0x7f2edc068c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:12:37.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.668+0000 7f2ee4655640 1 -- 172.21.15.46:0/3227206547 shutdown_connections 2024-03-31T23:12:37.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.668+0000 7f2ee4655640 1 -- 172.21.15.46:0/3227206547 wait complete. 2024-03-31T23:12:37.674 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:37.668+0000 7f9864654640 1 -- 172.21.15.46:0/3572213777 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f98480a2850 msgr2=0x7f98480a4d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:12:37.674 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:37.668+0000 7f9864654640 1 --2- 172.21.15.46:0/3572213777 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f98480a2850 0x7f98480a4d10 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f98540049d0 tx=0x7f9854004ab0 comp rx=0 tx=0).stop 2024-03-31T23:12:37.674 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:37.668+0000 7f9864654640 1 -- 172.21.15.46:0/3572213777 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f985c068dd0 msgr2=0x7f985c197f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:12:37.675 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:37.668+0000 7f9864654640 1 --2- 172.21.15.46:0/3572213777 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f985c068dd0 0x7f985c197f10 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f985800e9f0 tx=0x7f985800eeb0 comp rx=0 tx=0).stop 2024-03-31T23:12:37.675 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:37.668+0000 7f9864654640 1 -- 172.21.15.46:0/3572213777 shutdown_connections 2024-03-31T23:12:37.675 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:37.668+0000 7f9864654640 1 --2- 172.21.15.46:0/3572213777 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f98480a2850 0x7f98480a4d10 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:37.675 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:37.668+0000 7f9864654640 1 --2- 172.21.15.46:0/3572213777 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f985c107840 0x7f985c19c780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:37.675 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:37.668+0000 7f9864654640 1 --2- 172.21.15.46:0/3572213777 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f985c069770 0x7f985c198450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:37.675 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:37.668+0000 7f9864654640 1 --2- 172.21.15.46:0/3572213777 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f985c068dd0 0x7f985c197f10 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:37.675 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:37.668+0000 7f9864654640 1 -- 172.21.15.46:0/3572213777 >> 172.21.15.46:0/3572213777 conn(0x7f985c0774a0 msgr2=0x7f985c0fe8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:12:37.675 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:37.668+0000 7f9864654640 1 -- 172.21.15.46:0/3572213777 shutdown_connections 2024-03-31T23:12:37.675 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:37.669+0000 7f9864654640 1 -- 172.21.15.46:0/3572213777 wait complete. 2024-03-31T23:12:37.685 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-31T23:12:37.797 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.796+0000 7fdcfb59e640 1 -- 172.21.15.46:0/3853780556 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdcfc100360 msgr2=0x7fdcfc100740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:12:37.797 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.796+0000 7fdcfb59e640 1 --2- 172.21.15.46:0/3853780556 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdcfc100360 0x7fdcfc100740 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7fdcec009900 tx=0x7fdcec02bc30 comp rx=0 tx=0).stop 2024-03-31T23:12:37.797 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.796+0000 7fdcfb59e640 1 -- 172.21.15.46:0/3853780556 shutdown_connections 2024-03-31T23:12:37.797 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.796+0000 7fdcfb59e640 1 --2- 172.21.15.46:0/3853780556 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdcfc10df10 0x7fdcfc110300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:37.797 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.796+0000 7fdcfb59e640 1 --2- 172.21.15.46:0/3853780556 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdcfc100d10 0x7fdcfc10d9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:37.797 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.796+0000 7fdcfb59e640 1 --2- 172.21.15.46:0/3853780556 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdcfc100360 0x7fdcfc100740 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:37.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.796+0000 7fdcfb59e640 1 -- 172.21.15.46:0/3853780556 >> 172.21.15.46:0/3853780556 conn(0x7fdcfc0fbfa0 msgr2=0x7fdcfc0fe3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:12:37.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.796+0000 7fdcfb59e640 1 -- 172.21.15.46:0/3853780556 shutdown_connections 2024-03-31T23:12:37.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.796+0000 7fdcfb59e640 1 -- 172.21.15.46:0/3853780556 wait complete. 2024-03-31T23:12:37.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.797+0000 7fdcfb59e640 1 Processor -- start 2024-03-31T23:12:37.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.797+0000 7fdcfb59e640 1 -- start start 2024-03-31T23:12:37.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.797+0000 7fdcfb59e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdcfc100360 0x7fdcfc199fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:37.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.797+0000 7fdcfb59e640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdcfc100d10 0x7fdcfc19a4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:37.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.797+0000 7fdcfb59e640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdcfc10df10 0x7fdcfc19e870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:37.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.797+0000 7fdcfb59e640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fdcfc113a70 con 0x7fdcfc100360 2024-03-31T23:12:37.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.797+0000 7fdcfb59e640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fdcfc1138f0 con 0x7fdcfc10df10 2024-03-31T23:12:37.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.797+0000 7fdcfb59e640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fdcfc113bf0 con 0x7fdcfc100d10 2024-03-31T23:12:37.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.797+0000 7fdcfa59c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdcfc100360 0x7fdcfc199fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:37.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.797+0000 7fdcf9d9b640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdcfc100d10 0x7fdcfc19a4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:37.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.797+0000 7fdcf9d9b640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdcfc100d10 0x7fdcfc19a4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.46:45432/0 (socket says 172.21.15.46:45432) 2024-03-31T23:12:37.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.797+0000 7fdcf9d9b640 1 -- 172.21.15.46:0/1938225644 learned_addr learned my addr 172.21.15.46:0/1938225644 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:12:37.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.797+0000 7fdcfad9d640 1 --2- 172.21.15.46:0/1938225644 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdcfc10df10 0x7fdcfc19e870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:37.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.797+0000 7fdcfa59c640 1 -- 172.21.15.46:0/1938225644 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdcfc100d10 msgr2=0x7fdcfc19a4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:12:37.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.797+0000 7fdcfa59c640 1 --2- 172.21.15.46:0/1938225644 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdcfc100d10 0x7fdcfc19a4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:37.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.797+0000 7fdcfa59c640 1 -- 172.21.15.46:0/1938225644 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdcfc10df10 msgr2=0x7fdcfc19e870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:12:37.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.797+0000 7fdcfa59c640 1 --2- 172.21.15.46:0/1938225644 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdcfc10df10 0x7fdcfc19e870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:37.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.797+0000 7fdcfa59c640 1 -- 172.21.15.46:0/1938225644 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdcfc19efa0 con 0x7fdcfc100360 2024-03-31T23:12:37.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.797+0000 7fdcf9d9b640 1 --2- 172.21.15.46:0/1938225644 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdcfc100d10 0x7fdcfc19a4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:12:37.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.798+0000 7fdcfa59c640 1 --2- 172.21.15.46:0/1938225644 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdcfc100360 0x7fdcfc199fb0 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7fdcec036040 tx=0x7fdcec0045e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:12:37.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.798+0000 7fdceb7fe640 1 -- 172.21.15.46:0/1938225644 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdcec043020 con 0x7fdcfc100360 2024-03-31T23:12:37.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.798+0000 7fdceb7fe640 1 -- 172.21.15.46:0/1938225644 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fdcec02ca70 con 0x7fdcfc100360 2024-03-31T23:12:37.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.798+0000 7fdcfb59e640 1 -- 172.21.15.46:0/1938225644 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdcfc19f230 con 0x7fdcfc100360 2024-03-31T23:12:37.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.798+0000 7fdceb7fe640 1 -- 172.21.15.46:0/1938225644 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdcec0039e0 con 0x7fdcfc100360 2024-03-31T23:12:37.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.798+0000 7fdcfb59e640 1 -- 172.21.15.46:0/1938225644 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdcfc1a6b20 con 0x7fdcfc100360 2024-03-31T23:12:37.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.801+0000 7fdd039de740 1 Processor -- start 2024-03-31T23:12:37.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.801+0000 7fdceb7fe640 1 -- 172.21.15.46:0/1938225644 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 24) ==== 137553+0+0 (secure 0 0 0) 0x7fdcec046020 con 0x7fdcfc100360 2024-03-31T23:12:37.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.802+0000 7fdd039de740 1 -- start start 2024-03-31T23:12:37.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.802+0000 7fdd039de740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55f63c87c680 0x55f63c87ca60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:37.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.802+0000 7fdd039de740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f63c87cfa0 0x55f63c885640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:37.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.802+0000 7fdd039de740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f63c885b80 0x55f63c887f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:37.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.802+0000 7fdd039de740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55f63c7e01b0 con 0x55f63c87cfa0 2024-03-31T23:12:37.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.802+0000 7fdd039de740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55f63c788c00 con 0x55f63c87c680 2024-03-31T23:12:37.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.802+0000 7fdd039de740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55f63c7dc250 con 0x55f63c885b80 2024-03-31T23:12:37.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.802+0000 7fdcfa59c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f63c87cfa0 0x55f63c885640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:37.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.802+0000 7fdcfad9d640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55f63c87c680 0x55f63c87ca60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:37.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.802+0000 7fdcfad9d640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55f63c87c680 0x55f63c87ca60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.46:51820/0 (socket says 172.21.15.46:51820) 2024-03-31T23:12:37.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.802+0000 7fdcf9d9b640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f63c885b80 0x55f63c887f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:37.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.802+0000 7fdcfa59c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f63c87cfa0 0x55f63c885640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60972/0 (socket says 172.21.15.46:60972) 2024-03-31T23:12:37.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.802+0000 7fdcfad9d640 1 -- 172.21.15.46:0/3696096501 learned_addr learned my addr 172.21.15.46:0/3696096501 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:12:37.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.803+0000 7fdcfad9d640 1 -- 172.21.15.46:0/3696096501 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f63c885b80 msgr2=0x55f63c887f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:12:37.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.803+0000 7fdcfad9d640 1 --2- 172.21.15.46:0/3696096501 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f63c885b80 0x55f63c887f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:37.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.803+0000 7fdcfad9d640 1 -- 172.21.15.46:0/3696096501 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f63c87cfa0 msgr2=0x55f63c885640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:12:37.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.803+0000 7fdcfad9d640 1 --2- 172.21.15.46:0/3696096501 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f63c87cfa0 0x55f63c885640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:37.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.803+0000 7fdcfad9d640 1 -- 172.21.15.46:0/3696096501 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f63c7a83d0 con 0x55f63c87c680 2024-03-31T23:12:37.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.803+0000 7fdceb7fe640 1 --2- 172.21.15.46:0/1938225644 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fdce40a2730 0x7fdce40a4bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:37.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.803+0000 7fdcfa59c640 1 --2- 172.21.15.46:0/1938225644 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fdce40a2730 0x7fdce40a4bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:37.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.803+0000 7fdcfad9d640 1 --2- 172.21.15.46:0/3696096501 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55f63c87c680 0x55f63c87ca60 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fdcf4009f40 tx=0x7fdcf40118d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8df5f01de2f91fd5 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:12:37.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.803+0000 7fdceb7fe640 1 -- 172.21.15.46:0/1938225644 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7fdcec0f1090 con 0x7fdcfc100360 2024-03-31T23:12:37.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.803+0000 7fdce8ff9640 1 -- 172.21.15.46:0/3696096501 <== mon.2 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdcf4004d20 con 0x55f63c87c680 2024-03-31T23:12:37.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.803+0000 7fdce8ff9640 1 -- 172.21.15.46:0/3696096501 <== mon.2 v2:172.21.15.142:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fdcf4004050 con 0x55f63c87c680 2024-03-31T23:12:37.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.803+0000 7fdce8ff9640 1 -- 172.21.15.46:0/3696096501 <== mon.2 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdcf4005280 con 0x55f63c87c680 2024-03-31T23:12:37.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.804+0000 7fdd039de740 1 -- 172.21.15.46:0/3696096501 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55f63c87c680 msgr2=0x55f63c87ca60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:12:37.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.804+0000 7fdd039de740 1 --2- 172.21.15.46:0/3696096501 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55f63c87c680 0x55f63c87ca60 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fdcf4009f40 tx=0x7fdcf40118d0 comp rx=0 tx=0).stop 2024-03-31T23:12:37.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.804+0000 7fdcfa59c640 1 --2- 172.21.15.46:0/1938225644 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fdce40a2730 0x7fdce40a4bf0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fdcec0ee3c0 tx=0x7fdcec0f4b20 comp rx=0 tx=0).ready entity=mgr.15654 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:12:37.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.804+0000 7fdd039de740 1 -- 172.21.15.46:0/3696096501 shutdown_connections 2024-03-31T23:12:37.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.804+0000 7fdd039de740 1 --2- 172.21.15.46:0/3696096501 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f63c885b80 0x55f63c887f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:37.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.804+0000 7fdd039de740 1 --2- 172.21.15.46:0/3696096501 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f63c87cfa0 0x55f63c885640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:37.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.804+0000 7fdd039de740 1 --2- 172.21.15.46:0/3696096501 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55f63c87c680 0x55f63c87ca60 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:37.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.804+0000 7fdd039de740 1 -- 172.21.15.46:0/3696096501 >> 172.21.15.46:0/3696096501 conn(0x55f63c876db0 msgr2=0x55f63c877190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:12:37.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.804+0000 7fdd039de740 1 -- 172.21.15.46:0/3696096501 shutdown_connections 2024-03-31T23:12:37.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.804+0000 7fdd039de740 1 -- 172.21.15.46:0/3696096501 wait complete. 2024-03-31T23:12:37.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.805+0000 7fdd039de740 1 -- start start 2024-03-31T23:12:37.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.805+0000 7fdd039de740 1 Processor -- start 2024-03-31T23:12:37.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.805+0000 7fdd039de740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f63c87c680 0x55f63c882c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:37.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.805+0000 7fdd039de740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55f63c87cfa0 0x55f63c883150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:37.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.806+0000 7fdd039de740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f63c885b80 0x55f63c883690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:37.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.806+0000 7fdcfad9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f63c87c680 0x55f63c882c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:37.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.806+0000 7fdcfad9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f63c87c680 0x55f63c882c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60988/0 (socket says 172.21.15.46:60988) 2024-03-31T23:12:37.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.806+0000 7fdcfad9d640 1 -- 172.21.15.46:0/456990318 learned_addr learned my addr 172.21.15.46:0/456990318 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:12:37.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.806+0000 7fdcf9d9b640 1 --2- 172.21.15.46:0/456990318 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55f63c87cfa0 0x55f63c883150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:37.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.806+0000 7fdd039de740 1 -- 172.21.15.46:0/456990318 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55f63c888970 con 0x55f63c87c680 2024-03-31T23:12:37.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.806+0000 7fdd039de740 1 -- 172.21.15.46:0/456990318 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55f63c77e470 con 0x55f63c87cfa0 2024-03-31T23:12:37.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.806+0000 7fdd039de740 1 -- 172.21.15.46:0/456990318 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55f63c888af0 con 0x55f63c885b80 2024-03-31T23:12:37.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.806+0000 7fdd039de740 10 client.0 _ll_register_callbacks cb 0x55f63c79bcb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T23:12:37.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.806+0000 7fdcfad9d640 1 --2- 172.21.15.46:0/456990318 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f63c885b80 0x55f63c883690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:37.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.806+0000 7fdcfad9d640 1 -- 172.21.15.46:0/456990318 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f63c885b80 msgr2=0x55f63c883690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:12:37.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.806+0000 7fdcfad9d640 1 --2- 172.21.15.46:0/456990318 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f63c885b80 0x55f63c883690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:37.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.806+0000 7fdcfad9d640 1 -- 172.21.15.46:0/456990318 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55f63c87cfa0 msgr2=0x55f63c883150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:12:37.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.806+0000 7fdcfad9d640 1 --2- 172.21.15.46:0/456990318 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55f63c87cfa0 0x55f63c883150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:37.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.806+0000 7fdcfad9d640 1 -- 172.21.15.46:0/456990318 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f63c4f43f0 con 0x55f63c87c680 2024-03-31T23:12:37.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.807+0000 7fdcfad9d640 1 --2- 172.21.15.46:0/456990318 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f63c87c680 0x55f63c882c10 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7fdcf4000f80 tx=0x7fdcf4005bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=407e87dd641aa5a2 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:12:37.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.807+0000 7fdce1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-31T23:12:37.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.807+0000 7fdce1ffb640 1 -- 172.21.15.46:0/456990318 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdcf4003a30 con 0x55f63c87c680 2024-03-31T23:12:37.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.807+0000 7fdce1ffb640 1 -- 172.21.15.46:0/456990318 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7fdcf4003bd0 con 0x55f63c87c680 2024-03-31T23:12:37.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.807+0000 7fdd039de740 1 -- 172.21.15.46:0/456990318 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdcf4004050 con 0x55f63c87c680 2024-03-31T23:12:37.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.807+0000 7fdce1ffb640 1 -- 172.21.15.46:0/456990318 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fdcf4036050 con 0x55f63c87c680 2024-03-31T23:12:37.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.807+0000 7fdd039de740 10 client.15957 fetch_fsmap learned FSMap version 25 2024-03-31T23:12:37.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.807+0000 7fdd039de740 1 -- 172.21.15.46:0/456990318 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55f63c5be7d0 con 0x55f63c87c680 2024-03-31T23:12:37.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.808+0000 7fdce1ffb640 1 -- 172.21.15.46:0/456990318 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fdcf4004050 con 0x55f63c87c680 2024-03-31T23:12:37.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.808+0000 7fdce1ffb640 1 -- 172.21.15.46:0/456990318 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7fdcf4028480 con 0x55f63c87c680 2024-03-31T23:12:37.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.808+0000 7fdd039de740 10 client.15957 fetch_fsmap finished waiting for FSMap version 25 2024-03-31T23:12:37.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.808+0000 7fdd039de740 10 client.15957 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-31T23:12:37.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.808+0000 7fdd039de740 20 client.15957 populate_metadata read hostname 'smithi046' 2024-03-31T23:12:37.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.808+0000 7fdd039de740 1 --2- 172.21.15.46:0/456990318 >> [v2:172.21.15.162:6836/1371886876,v1:172.21.15.162:6837/1371886876] conn(0x55f63c895b30 0x55f63c897f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:37.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.808+0000 7fdd039de740 4 client.15957 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-31T23:12:37.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.808+0000 7fdd039de740 1 -- 172.21.15.46:0/456990318 --> [v2:172.21.15.162:6836/1371886876,v1:172.21.15.162:6837/1371886876] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f63c7ed6f0 con 0x55f63c895b30 2024-03-31T23:12:37.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.808+0000 7fdcf9d9b640 1 --2- 172.21.15.46:0/456990318 >> [v2:172.21.15.162:6836/1371886876,v1:172.21.15.162:6837/1371886876] conn(0x55f63c895b30 0x55f63c897f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:37.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.809+0000 7fdce1ffb640 10 client.15957 ms_handle_connect on v2:172.21.15.162:6836/1371886876 2024-03-31T23:12:37.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.809+0000 7fdcf9d9b640 1 --2- 172.21.15.46:0/456990318 >> [v2:172.21.15.162:6836/1371886876,v1:172.21.15.162:6837/1371886876] conn(0x55f63c895b30 0x55f63c897f20 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=88d50be912b00440 server_cookie=f5aa607732a0d72c in_seq=0 out_seq=0 2024-03-31T23:12:37.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.809+0000 7fdce1ffb640 1 -- 172.21.15.46:0/456990318 <== mds.2 v2:172.21.15.162:6836/1371886876 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55f63c7ed6f0 con 0x55f63c895b30 2024-03-31T23:12:37.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.809+0000 7fdce1ffb640 10 client.15957 handle_command_reply: tid=0 2024-03-31T23:12:37.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.809+0000 7fdd039de740 1 client.15957 shutdown 2024-03-31T23:12:37.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.809+0000 7fdd039de740 1 -- 172.21.15.46:0/456990318 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f63c87c680 msgr2=0x55f63c882c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:12:37.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.809+0000 7fdd039de740 1 --2- 172.21.15.46:0/456990318 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f63c87c680 0x55f63c882c10 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7fdcf4000f80 tx=0x7fdcf4005bd0 comp rx=0 tx=0).stop 2024-03-31T23:12:37.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.809+0000 7fdd039de740 1 -- 172.21.15.46:0/456990318 shutdown_connections 2024-03-31T23:12:37.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.809+0000 7fdd039de740 1 --2- 172.21.15.46:0/456990318 >> [v2:172.21.15.162:6836/1371886876,v1:172.21.15.162:6837/1371886876] conn(0x55f63c895b30 0x55f63c897f20 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:37.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.809+0000 7fdd039de740 1 --2- 172.21.15.46:0/456990318 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f63c885b80 0x55f63c883690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:37.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.810+0000 7fdd039de740 1 --2- 172.21.15.46:0/456990318 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55f63c87cfa0 0x55f63c883150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:37.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.810+0000 7fdd039de740 1 --2- 172.21.15.46:0/456990318 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f63c87c680 0x55f63c882c10 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:37.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.810+0000 7fdd039de740 1 -- 172.21.15.46:0/456990318 >> 172.21.15.46:0/456990318 conn(0x55f63c876db0 msgr2=0x55f63c878b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:12:37.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.810+0000 7fdce1ffb640 0 client.15957 ms_handle_reset on v2:172.21.15.162:6836/1371886876 2024-03-31T23:12:37.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.810+0000 7fdd039de740 1 -- 172.21.15.46:0/456990318 shutdown_connections 2024-03-31T23:12:37.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.810+0000 7fdd039de740 1 -- 172.21.15.46:0/456990318 wait complete. 2024-03-31T23:12:37.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.810+0000 7fdd039de740 20 client.15957 trim_cache size 0 max 16384 2024-03-31T23:12:37.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.836+0000 7fdd039de740 1 Processor -- start 2024-03-31T23:12:37.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.836+0000 7fdd039de740 1 -- start start 2024-03-31T23:12:37.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.836+0000 7fdd039de740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f63c87c680 0x55f63c885340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:37.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.836+0000 7fdd039de740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55f63c87cfa0 0x55f63c87d380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:37.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.836+0000 7fdd039de740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f63c885b80 0x55f63c891710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:37.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.836+0000 7fdd039de740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fdcfc1b3470 con 0x55f63c87c680 2024-03-31T23:12:37.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.836+0000 7fdd039de740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fdcfc1b0d40 con 0x55f63c87cfa0 2024-03-31T23:12:37.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.836+0000 7fdd039de740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fdcfc1aeb90 con 0x55f63c885b80 2024-03-31T23:12:37.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.836+0000 7fdcf9d9b640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55f63c87cfa0 0x55f63c87d380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:37.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.836+0000 7fdcf9d9b640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55f63c87cfa0 0x55f63c87d380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.46:51840/0 (socket says 172.21.15.46:51840) 2024-03-31T23:12:37.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.836+0000 7fdcfad9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f63c87c680 0x55f63c885340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:37.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.836+0000 7fdcf9d9b640 1 -- 172.21.15.46:0/4115324862 learned_addr learned my addr 172.21.15.46:0/4115324862 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:12:37.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.837+0000 7fdcfad9d640 1 --2- 172.21.15.46:0/4115324862 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f63c885b80 0x55f63c891710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:37.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.837+0000 7fdcf9d9b640 1 -- 172.21.15.46:0/4115324862 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f63c885b80 msgr2=0x55f63c891710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:12:37.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.837+0000 7fdcf9d9b640 1 --2- 172.21.15.46:0/4115324862 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f63c885b80 0x55f63c891710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:37.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.837+0000 7fdcf9d9b640 1 -- 172.21.15.46:0/4115324862 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f63c87c680 msgr2=0x55f63c885340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:12:37.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.837+0000 7fdcf9d9b640 1 --2- 172.21.15.46:0/4115324862 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f63c87c680 0x55f63c885340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:37.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.837+0000 7fdcf9d9b640 1 -- 172.21.15.46:0/4115324862 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f63c881bc0 con 0x55f63c87cfa0 2024-03-31T23:12:37.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.837+0000 7fdcf9d9b640 1 --2- 172.21.15.46:0/4115324862 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55f63c87cfa0 0x55f63c87d380 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x55f63c877f40 tx=0x7fdcf0005f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d619a4368274c75b server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:12:37.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.837+0000 7fdce3fff640 1 -- 172.21.15.46:0/4115324862 <== mon.2 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdcf0004020 con 0x55f63c87cfa0 2024-03-31T23:12:37.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.837+0000 7fdce3fff640 1 -- 172.21.15.46:0/4115324862 <== mon.2 v2:172.21.15.142:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fdcf0010e10 con 0x55f63c87cfa0 2024-03-31T23:12:37.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.837+0000 7fdce3fff640 1 -- 172.21.15.46:0/4115324862 <== mon.2 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdcf0020660 con 0x55f63c87cfa0 2024-03-31T23:12:37.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.838+0000 7fdd039de740 1 -- 172.21.15.46:0/4115324862 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55f63c87cfa0 msgr2=0x55f63c87d380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:12:37.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.838+0000 7fdd039de740 1 --2- 172.21.15.46:0/4115324862 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55f63c87cfa0 0x55f63c87d380 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x55f63c877f40 tx=0x7fdcf0005f90 comp rx=0 tx=0).stop 2024-03-31T23:12:37.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.838+0000 7fdd039de740 1 -- 172.21.15.46:0/4115324862 shutdown_connections 2024-03-31T23:12:37.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.838+0000 7fdd039de740 1 --2- 172.21.15.46:0/4115324862 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f63c885b80 0x55f63c891710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:37.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.838+0000 7fdd039de740 1 --2- 172.21.15.46:0/4115324862 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55f63c87cfa0 0x55f63c87d380 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:37.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.838+0000 7fdd039de740 1 --2- 172.21.15.46:0/4115324862 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f63c87c680 0x55f63c885340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:37.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.838+0000 7fdd039de740 1 -- 172.21.15.46:0/4115324862 >> 172.21.15.46:0/4115324862 conn(0x55f63c876db0 msgr2=0x55f63c875700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:12:37.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.838+0000 7fdd039de740 1 -- 172.21.15.46:0/4115324862 shutdown_connections 2024-03-31T23:12:37.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.838+0000 7fdd039de740 1 -- 172.21.15.46:0/4115324862 wait complete. 2024-03-31T23:12:37.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.839+0000 7fdd039de740 1 -- start start 2024-03-31T23:12:37.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.839+0000 7fdd039de740 1 Processor -- start 2024-03-31T23:12:37.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.839+0000 7fdd039de740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55f63c87c680 0x55f63c88e150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:37.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.839+0000 7fdd039de740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f63c87cfa0 0x55f63c87f5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:37.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.839+0000 7fdd039de740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f63c885b80 0x55f63c87fd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:37.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.839+0000 7fdd039de740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fdcfc1af200 con 0x55f63c885b80 2024-03-31T23:12:37.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.839+0000 7fdd039de740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fdcfc1b0670 con 0x55f63c87c680 2024-03-31T23:12:37.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.839+0000 7fdd039de740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fdcfc1af8d0 con 0x55f63c87cfa0 2024-03-31T23:12:37.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.839+0000 7fdcfad9d640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55f63c87c680 0x55f63c88e150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:37.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.839+0000 7fdd039de740 10 client.0 _ll_register_callbacks cb 0x7fdcf4009e10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T23:12:37.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.839+0000 7fdcf9d9b640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f63c87cfa0 0x55f63c87f5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:37.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.839+0000 7fdcfad9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f63c885b80 0x55f63c87fd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:37.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.839+0000 7fdcf9d9b640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f63c87cfa0 0x55f63c87f5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.46:45476/0 (socket says 172.21.15.46:45476) 2024-03-31T23:12:37.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.839+0000 7fdcf9d9b640 1 -- 172.21.15.46:0/2146417203 learned_addr learned my addr 172.21.15.46:0/2146417203 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:12:37.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.839+0000 7fdcfad9d640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55f63c87c680 0x55f63c88e150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.46:51848/0 (socket says 172.21.15.46:51848) 2024-03-31T23:12:37.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.839+0000 7fdcf9d9b640 1 -- 172.21.15.46:0/2146417203 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55f63c87c680 msgr2=0x55f63c88e150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:12:37.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.839+0000 7fdcf9d9b640 1 --2- 172.21.15.46:0/2146417203 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55f63c87c680 0x55f63c88e150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:37.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.839+0000 7fdcf9d9b640 1 -- 172.21.15.46:0/2146417203 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f63c885b80 msgr2=0x55f63c87fd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:12:37.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.839+0000 7fdcf9d9b640 1 --2- 172.21.15.46:0/2146417203 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f63c885b80 0x55f63c87fd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:37.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.840+0000 7fdcf9d9b640 1 -- 172.21.15.46:0/2146417203 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f63c8fad80 con 0x55f63c87cfa0 2024-03-31T23:12:37.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.840+0000 7fdcf9d9b640 1 --2- 172.21.15.46:0/2146417203 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f63c87cfa0 0x55f63c87f5e0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fdcf0012190 tx=0x7fdcf00124e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bc8df78326b4e6d server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:12:37.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.840+0000 7fdce1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.162:3300/0 2024-03-31T23:12:37.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.840+0000 7fdce1ffb640 1 -- 172.21.15.46:0/2146417203 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdcf0004020 con 0x55f63c87cfa0 2024-03-31T23:12:37.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.840+0000 7fdce1ffb640 1 -- 172.21.15.46:0/2146417203 <== mon.1 v2:172.21.15.162:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7fdcf00041c0 con 0x55f63c87cfa0 2024-03-31T23:12:37.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.840+0000 7fdd039de740 1 -- 172.21.15.46:0/2146417203 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdcf0010e10 con 0x55f63c87cfa0 2024-03-31T23:12:37.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.840+0000 7fdce1ffb640 1 -- 172.21.15.46:0/2146417203 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fdcf003b050 con 0x55f63c87cfa0 2024-03-31T23:12:37.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.840+0000 7fdd039de740 10 client.25705 fetch_fsmap learned FSMap version 25 2024-03-31T23:12:37.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.840+0000 7fdd039de740 1 -- 172.21.15.46:0/2146417203 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55f63c8fbec0 con 0x55f63c87cfa0 2024-03-31T23:12:37.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.840+0000 7fdce1ffb640 1 -- 172.21.15.46:0/2146417203 <== mon.1 v2:172.21.15.162:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fdcf0010e10 con 0x55f63c87cfa0 2024-03-31T23:12:37.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.841+0000 7fdce1ffb640 1 -- 172.21.15.46:0/2146417203 <== mon.1 v2:172.21.15.162:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7fdcf0031030 con 0x55f63c87cfa0 2024-03-31T23:12:37.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.841+0000 7fdd039de740 10 client.25705 fetch_fsmap finished waiting for FSMap version 25 2024-03-31T23:12:37.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.841+0000 7fdd039de740 10 client.25705 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-31T23:12:37.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.841+0000 7fdd039de740 20 client.25705 populate_metadata read hostname 'smithi046' 2024-03-31T23:12:37.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.841+0000 7fdd039de740 1 --2- 172.21.15.46:0/2146417203 >> [v2:172.21.15.162:6836/1371886876,v1:172.21.15.162:6837/1371886876] conn(0x55f63c8fc510 0x55f63c8fe930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:37.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.841+0000 7fdd039de740 4 client.25705 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-31T23:12:37.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.841+0000 7fdd039de740 1 -- 172.21.15.46:0/2146417203 --> [v2:172.21.15.162:6836/1371886876,v1:172.21.15.162:6837/1371886876] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55f63c7e7a40 con 0x55f63c8fc510 2024-03-31T23:12:37.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.841+0000 7fdcfad9d640 1 --2- 172.21.15.46:0/2146417203 >> [v2:172.21.15.162:6836/1371886876,v1:172.21.15.162:6837/1371886876] conn(0x55f63c8fc510 0x55f63c8fe930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:37.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.841+0000 7fdce1ffb640 10 client.25705 ms_handle_connect on v2:172.21.15.162:6836/1371886876 2024-03-31T23:12:37.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.841+0000 7fdcfad9d640 1 --2- 172.21.15.46:0/2146417203 >> [v2:172.21.15.162:6836/1371886876,v1:172.21.15.162:6837/1371886876] conn(0x55f63c8fc510 0x55f63c8fe930 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=815f380f9f38733a server_cookie=3a4d4927e6c75caa in_seq=0 out_seq=0 2024-03-31T23:12:37.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.841+0000 7fdce1ffb640 1 -- 172.21.15.46:0/2146417203 <== mds.2 v2:172.21.15.162:6836/1371886876 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55f63c7e7a40 con 0x55f63c8fc510 2024-03-31T23:12:37.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.841+0000 7fdce1ffb640 10 client.25705 handle_command_reply: tid=0 2024-03-31T23:12:37.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.841+0000 7fdd039de740 1 client.25705 shutdown 2024-03-31T23:12:37.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.842+0000 7fdd039de740 1 -- 172.21.15.46:0/2146417203 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f63c87cfa0 msgr2=0x55f63c87f5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:12:37.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.842+0000 7fdd039de740 1 --2- 172.21.15.46:0/2146417203 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f63c87cfa0 0x55f63c87f5e0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fdcf0012190 tx=0x7fdcf00124e0 comp rx=0 tx=0).stop 2024-03-31T23:12:37.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.842+0000 7fdd039de740 1 -- 172.21.15.46:0/2146417203 shutdown_connections 2024-03-31T23:12:37.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.842+0000 7fdd039de740 1 --2- 172.21.15.46:0/2146417203 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f63c885b80 0x55f63c87fd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:37.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.842+0000 7fdd039de740 1 --2- 172.21.15.46:0/2146417203 >> [v2:172.21.15.162:6836/1371886876,v1:172.21.15.162:6837/1371886876] conn(0x55f63c8fc510 0x55f63c8fe930 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:37.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.842+0000 7fdd039de740 1 --2- 172.21.15.46:0/2146417203 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f63c87cfa0 0x55f63c87f5e0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:37.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.842+0000 7fdce1ffb640 0 client.25705 ms_handle_reset on v2:172.21.15.162:6836/1371886876 2024-03-31T23:12:37.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.842+0000 7fdd039de740 1 --2- 172.21.15.46:0/2146417203 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55f63c87c680 0x55f63c88e150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:37.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.842+0000 7fdd039de740 1 -- 172.21.15.46:0/2146417203 >> 172.21.15.46:0/2146417203 conn(0x55f63c876db0 msgr2=0x55f63c875f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:12:37.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.842+0000 7fdd039de740 1 -- 172.21.15.46:0/2146417203 shutdown_connections 2024-03-31T23:12:37.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.842+0000 7fdd039de740 1 -- 172.21.15.46:0/2146417203 wait complete. 2024-03-31T23:12:37.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.842+0000 7fdd039de740 20 client.25705 trim_cache size 0 max 16384 2024-03-31T23:12:37.849 INFO:teuthology.orchestra.run.smithi046.stdout:[] 2024-03-31T23:12:37.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.842+0000 7fdcfb59e640 1 -- 172.21.15.46:0/1938225644 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fdce40a2730 msgr2=0x7fdce40a4bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:12:37.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.842+0000 7fdcfb59e640 1 --2- 172.21.15.46:0/1938225644 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fdce40a2730 0x7fdce40a4bf0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fdcec0ee3c0 tx=0x7fdcec0f4b20 comp rx=0 tx=0).stop 2024-03-31T23:12:37.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.842+0000 7fdcfb59e640 1 -- 172.21.15.46:0/1938225644 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdcfc100360 msgr2=0x7fdcfc199fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:12:37.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.842+0000 7fdcfb59e640 1 --2- 172.21.15.46:0/1938225644 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdcfc100360 0x7fdcfc199fb0 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7fdcec036040 tx=0x7fdcec0045e0 comp rx=0 tx=0).stop 2024-03-31T23:12:37.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.843+0000 7fdcfb59e640 1 -- 172.21.15.46:0/1938225644 shutdown_connections 2024-03-31T23:12:37.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.843+0000 7fdcfb59e640 1 --2- 172.21.15.46:0/1938225644 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fdce40a2730 0x7fdce40a4bf0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:37.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.843+0000 7fdcfb59e640 1 --2- 172.21.15.46:0/1938225644 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdcfc10df10 0x7fdcfc19e870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:37.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.843+0000 7fdcfb59e640 1 --2- 172.21.15.46:0/1938225644 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdcfc100d10 0x7fdcfc19a4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:37.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.843+0000 7fdcfb59e640 1 --2- 172.21.15.46:0/1938225644 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdcfc100360 0x7fdcfc199fb0 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:37.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.843+0000 7fdcfb59e640 1 -- 172.21.15.46:0/1938225644 >> 172.21.15.46:0/1938225644 conn(0x7fdcfc0fbfa0 msgr2=0x7fdcfc0fe390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:12:37.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.843+0000 7fdcfb59e640 1 -- 172.21.15.46:0/1938225644 shutdown_connections 2024-03-31T23:12:37.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.843+0000 7fdcfb59e640 1 -- 172.21.15.46:0/1938225644 wait complete. 2024-03-31T23:12:37.858 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-31T23:12:37.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.963+0000 7f87db89c640 1 -- 172.21.15.46:0/1357079609 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f87d4069310 msgr2=0x7f87d410e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:12:37.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.963+0000 7f87db89c640 1 --2- 172.21.15.46:0/1357079609 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f87d4069310 0x7f87d410e5b0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f87cc009c10 tx=0x7f87cc02bee0 comp rx=0 tx=0).stop 2024-03-31T23:12:37.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.963+0000 7f87db89c640 1 -- 172.21.15.46:0/1357079609 shutdown_connections 2024-03-31T23:12:37.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.963+0000 7f87db89c640 1 --2- 172.21.15.46:0/1357079609 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f87d4069310 0x7f87d410e5b0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:37.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.963+0000 7f87db89c640 1 --2- 172.21.15.46:0/1357079609 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f87d4068950 0x7f87d4068dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:37.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.963+0000 7f87db89c640 1 --2- 172.21.15.46:0/1357079609 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f87d41077a0 0x7f87d4107b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:37.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.963+0000 7f87db89c640 1 -- 172.21.15.46:0/1357079609 >> 172.21.15.46:0/1357079609 conn(0x7f87d4077750 msgr2=0x7f87d4077b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:12:37.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.964+0000 7f87db89c640 1 -- 172.21.15.46:0/1357079609 shutdown_connections 2024-03-31T23:12:37.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.964+0000 7f87db89c640 1 -- 172.21.15.46:0/1357079609 wait complete. 2024-03-31T23:12:37.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.964+0000 7f87db89c640 1 Processor -- start 2024-03-31T23:12:37.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.964+0000 7f87db89c640 1 -- start start 2024-03-31T23:12:37.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.964+0000 7f87db89c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f87d4068950 0x7f87d4100610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:37.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.964+0000 7f87db89c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f87d4069310 0x7f87d4100b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:37.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.964+0000 7f87db89c640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f87d41077a0 0x7f87d4101090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:37.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.964+0000 7f87db89c640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f87d4111960 con 0x7f87d4069310 2024-03-31T23:12:37.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.964+0000 7f87db89c640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f87d41117e0 con 0x7f87d41077a0 2024-03-31T23:12:37.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.964+0000 7f87db89c640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f87d4111ae0 con 0x7f87d4068950 2024-03-31T23:12:37.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.964+0000 7f87d8e37640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f87d4069310 0x7f87d4100b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:37.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.965+0000 7f87d8e37640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f87d4069310 0x7f87d4100b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:32784/0 (socket says 172.21.15.46:32784) 2024-03-31T23:12:37.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.965+0000 7f87d8e37640 1 -- 172.21.15.46:0/1340631537 learned_addr learned my addr 172.21.15.46:0/1340631537 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:12:37.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.965+0000 7f87d9638640 1 --2- 172.21.15.46:0/1340631537 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f87d4068950 0x7f87d4100610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:37.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.965+0000 7f87d9e39640 1 --2- 172.21.15.46:0/1340631537 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f87d41077a0 0x7f87d4101090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:37.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.965+0000 7f87d8e37640 1 -- 172.21.15.46:0/1340631537 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f87d4068950 msgr2=0x7f87d4100610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:12:37.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.965+0000 7f87d8e37640 1 --2- 172.21.15.46:0/1340631537 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f87d4068950 0x7f87d4100610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:37.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.965+0000 7f87d8e37640 1 -- 172.21.15.46:0/1340631537 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f87d41077a0 msgr2=0x7f87d4101090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:12:37.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.965+0000 7f87d8e37640 1 --2- 172.21.15.46:0/1340631537 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f87d41077a0 0x7f87d4101090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:37.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.965+0000 7f87d8e37640 1 -- 172.21.15.46:0/1340631537 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87d41a6360 con 0x7f87d4069310 2024-03-31T23:12:37.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.965+0000 7f87d9e39640 1 --2- 172.21.15.46:0/1340631537 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f87d41077a0 0x7f87d4101090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T23:12:37.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.965+0000 7f87d9638640 1 --2- 172.21.15.46:0/1340631537 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f87d4068950 0x7f87d4100610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:12:37.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.965+0000 7f87d8e37640 1 --2- 172.21.15.46:0/1340631537 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f87d4069310 0x7f87d4100b50 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f87c400e9f0 tx=0x7f87c400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:12:37.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.965+0000 7f87ca7fc640 1 -- 172.21.15.46:0/1340631537 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87c400cd80 con 0x7f87d4069310 2024-03-31T23:12:37.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.965+0000 7f87db89c640 1 -- 172.21.15.46:0/1340631537 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f87d41a6650 con 0x7f87d4069310 2024-03-31T23:12:37.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.965+0000 7f87ca7fc640 1 -- 172.21.15.46:0/1340631537 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f87c40108e0 con 0x7f87d4069310 2024-03-31T23:12:37.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.965+0000 7f87ca7fc640 1 -- 172.21.15.46:0/1340631537 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87c4018920 con 0x7f87d4069310 2024-03-31T23:12:37.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.965+0000 7f87db89c640 1 -- 172.21.15.46:0/1340631537 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f87d41a6bf0 con 0x7f87d4069310 2024-03-31T23:12:37.975 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.968+0000 7f87ca7fc640 1 -- 172.21.15.46:0/1340631537 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 24) ==== 137553+0+0 (secure 0 0 0) 0x7f87c4010a90 con 0x7f87d4069310 2024-03-31T23:12:37.975 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.969+0000 7f87de47d740 1 Processor -- start 2024-03-31T23:12:37.975 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.969+0000 7f87de47d740 1 -- start start 2024-03-31T23:12:37.975 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.969+0000 7f87ca7fc640 1 --2- 172.21.15.46:0/1340631537 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f87c00a27e0 0x7f87c00a4ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:37.975 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.969+0000 7f87de47d740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a9e2da3470 0x55a9e2da3850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:37.975 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.969+0000 7f87de47d740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a9e2da3d90 0x55a9e2dac430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:37.975 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.969+0000 7f87de47d740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a9e2dac970 0x55a9e2daed60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:37.975 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.969+0000 7f87de47d740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55a9e2d071b0 con 0x55a9e2da3d90 2024-03-31T23:12:37.975 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.969+0000 7f87de47d740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55a9e2cafc00 con 0x55a9e2da3470 2024-03-31T23:12:37.975 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.969+0000 7f87de47d740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55a9e2d03250 con 0x55a9e2dac970 2024-03-31T23:12:37.975 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.969+0000 7f87d9e39640 1 --2- 172.21.15.46:0/1340631537 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f87c00a27e0 0x7f87c00a4ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:37.976 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.969+0000 7f87ca7fc640 1 -- 172.21.15.46:0/1340631537 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7f87c40cf4a0 con 0x7f87d4069310 2024-03-31T23:12:37.976 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.969+0000 7f87d8e37640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a9e2da3d90 0x55a9e2dac430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:37.976 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.969+0000 7f87d9638640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a9e2da3470 0x55a9e2da3850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:37.976 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.969+0000 7f87d9638640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a9e2da3470 0x55a9e2da3850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.46:51858/0 (socket says 172.21.15.46:51858) 2024-03-31T23:12:37.976 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.969+0000 7f87d8e37640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a9e2da3d90 0x55a9e2dac430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:32786/0 (socket says 172.21.15.46:32786) 2024-03-31T23:12:37.976 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.969+0000 7f87d9638640 1 -- 172.21.15.46:0/952591298 learned_addr learned my addr 172.21.15.46:0/952591298 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:12:37.976 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.970+0000 7f87d9638640 1 -- 172.21.15.46:0/952591298 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a9e2dac970 msgr2=0x55a9e2daed60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:12:37.976 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.970+0000 7f87d9638640 1 --2- 172.21.15.46:0/952591298 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a9e2dac970 0x55a9e2daed60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:37.977 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.970+0000 7f87d9638640 1 -- 172.21.15.46:0/952591298 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a9e2da3d90 msgr2=0x55a9e2dac430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:12:37.977 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.970+0000 7f87d9638640 1 --2- 172.21.15.46:0/952591298 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a9e2da3d90 0x55a9e2dac430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:37.977 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.970+0000 7f87d9638640 1 -- 172.21.15.46:0/952591298 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a9e2ccf3d0 con 0x55a9e2da3470 2024-03-31T23:12:37.977 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.970+0000 7f87d8e37640 1 --2- 172.21.15.46:0/952591298 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a9e2da3d90 0x55a9e2dac430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:12:37.977 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.970+0000 7f87d9e39640 1 --2- 172.21.15.46:0/1340631537 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f87c00a27e0 0x7f87c00a4ca0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f87cc0099a0 tx=0x7f87cc009310 comp rx=0 tx=0).ready entity=mgr.15654 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:12:37.977 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.970+0000 7f87d9638640 1 --2- 172.21.15.46:0/952591298 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a9e2da3470 0x55a9e2da3850 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f87d0011470 tx=0x7f87d0011930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fe36859fe2275725 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:12:37.977 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.970+0000 7f87bf7fe640 1 -- 172.21.15.46:0/952591298 <== mon.2 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87d0005020 con 0x55a9e2da3470 2024-03-31T23:12:37.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.970+0000 7f87bf7fe640 1 -- 172.21.15.46:0/952591298 <== mon.2 v2:172.21.15.142:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f87d00047a0 con 0x55a9e2da3470 2024-03-31T23:12:37.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.970+0000 7f87bf7fe640 1 -- 172.21.15.46:0/952591298 <== mon.2 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87d000f980 con 0x55a9e2da3470 2024-03-31T23:12:37.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.971+0000 7f87de47d740 1 -- 172.21.15.46:0/952591298 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a9e2da3470 msgr2=0x55a9e2da3850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:12:37.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.971+0000 7f87de47d740 1 --2- 172.21.15.46:0/952591298 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a9e2da3470 0x55a9e2da3850 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f87d0011470 tx=0x7f87d0011930 comp rx=0 tx=0).stop 2024-03-31T23:12:37.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.971+0000 7f87de47d740 1 -- 172.21.15.46:0/952591298 shutdown_connections 2024-03-31T23:12:37.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.971+0000 7f87de47d740 1 --2- 172.21.15.46:0/952591298 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a9e2dac970 0x55a9e2daed60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:37.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.971+0000 7f87de47d740 1 --2- 172.21.15.46:0/952591298 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a9e2da3d90 0x55a9e2dac430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:37.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.971+0000 7f87de47d740 1 --2- 172.21.15.46:0/952591298 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a9e2da3470 0x55a9e2da3850 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:37.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.971+0000 7f87de47d740 1 -- 172.21.15.46:0/952591298 >> 172.21.15.46:0/952591298 conn(0x55a9e2d9dba0 msgr2=0x55a9e2d9df80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:12:37.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.971+0000 7f87de47d740 1 -- 172.21.15.46:0/952591298 shutdown_connections 2024-03-31T23:12:37.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.971+0000 7f87de47d740 1 -- 172.21.15.46:0/952591298 wait complete. 2024-03-31T23:12:37.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.971+0000 7f87de47d740 1 -- start start 2024-03-31T23:12:37.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.971+0000 7f87de47d740 1 Processor -- start 2024-03-31T23:12:37.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.972+0000 7f87de47d740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a9e2da3470 0x55a9e2da9970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:37.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.972+0000 7f87de47d740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a9e2da3d90 0x55a9e2da9eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:37.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.972+0000 7f87de47d740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a9e2dac970 0x55a9e2daa3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:37.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.972+0000 7f87de47d740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55a9e2daf760 con 0x55a9e2da3d90 2024-03-31T23:12:37.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.972+0000 7f87de47d740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55a9e2ca5470 con 0x55a9e2dac970 2024-03-31T23:12:37.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.972+0000 7f87de47d740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55a9e2daf8e0 con 0x55a9e2da3470 2024-03-31T23:12:37.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.972+0000 7f87de47d740 10 client.0 _ll_register_callbacks cb 0x55a9e2cc2cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T23:12:37.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.972+0000 7f87d8e37640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a9e2da3d90 0x55a9e2da9eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:37.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.972+0000 7f87d9638640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a9e2da3470 0x55a9e2da9970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:37.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.972+0000 7f87d8e37640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a9e2da3d90 0x55a9e2da9eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:32800/0 (socket says 172.21.15.46:32800) 2024-03-31T23:12:37.980 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.972+0000 7f87d8e37640 1 -- 172.21.15.46:0/240180979 learned_addr learned my addr 172.21.15.46:0/240180979 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:12:37.980 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.972+0000 7f87d9e39640 1 --2- 172.21.15.46:0/240180979 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a9e2dac970 0x55a9e2daa3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:37.980 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.972+0000 7f87d8e37640 1 -- 172.21.15.46:0/240180979 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a9e2da3470 msgr2=0x55a9e2da9970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:12:37.980 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.972+0000 7f87d8e37640 1 --2- 172.21.15.46:0/240180979 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a9e2da3470 0x55a9e2da9970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:37.980 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.972+0000 7f87d8e37640 1 -- 172.21.15.46:0/240180979 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a9e2dac970 msgr2=0x55a9e2daa3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:12:37.980 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.972+0000 7f87d8e37640 1 --2- 172.21.15.46:0/240180979 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a9e2dac970 0x55a9e2daa3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:37.980 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.972+0000 7f87d8e37640 1 -- 172.21.15.46:0/240180979 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a9e2a1b3f0 con 0x55a9e2da3d90 2024-03-31T23:12:37.980 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.973+0000 7f87d8e37640 1 --2- 172.21.15.46:0/240180979 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a9e2da3d90 0x55a9e2da9eb0 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f87c409b5b0 tx=0x7f87c400cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b71a1e014b010cb server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:12:37.980 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.973+0000 7f87bcff9640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-31T23:12:37.980 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.973+0000 7f87bcff9640 1 -- 172.21.15.46:0/240180979 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87c40b4060 con 0x55a9e2da3d90 2024-03-31T23:12:37.980 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.973+0000 7f87bcff9640 1 -- 172.21.15.46:0/240180979 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7f87c40cc9a0 con 0x55a9e2da3d90 2024-03-31T23:12:37.980 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.973+0000 7f87de47d740 1 -- 172.21.15.46:0/240180979 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f87d00047a0 con 0x55a9e2da3d90 2024-03-31T23:12:37.980 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.973+0000 7f87bcff9640 1 -- 172.21.15.46:0/240180979 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f87c409f770 con 0x55a9e2da3d90 2024-03-31T23:12:37.980 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.973+0000 7f87de47d740 10 client.15975 fetch_fsmap learned FSMap version 25 2024-03-31T23:12:37.981 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.973+0000 7f87de47d740 1 -- 172.21.15.46:0/240180979 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55a9e2ae57d0 con 0x55a9e2da3d90 2024-03-31T23:12:37.981 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.973+0000 7f87bcff9640 1 -- 172.21.15.46:0/240180979 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f87d00047a0 con 0x55a9e2da3d90 2024-03-31T23:12:37.981 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.974+0000 7f87bcff9640 1 -- 172.21.15.46:0/240180979 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f87c409aa70 con 0x55a9e2da3d90 2024-03-31T23:12:37.981 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.974+0000 7f87de47d740 10 client.15975 fetch_fsmap finished waiting for FSMap version 25 2024-03-31T23:12:37.981 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.974+0000 7f87de47d740 10 client.15975 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-03-31T23:12:37.981 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.974+0000 7f87de47d740 20 client.15975 populate_metadata read hostname 'smithi046' 2024-03-31T23:12:37.981 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.974+0000 7f87de47d740 1 --2- 172.21.15.46:0/240180979 >> [v2:172.21.15.142:6836/2559258399,v1:172.21.15.142:6837/2559258399] conn(0x55a9e2dbc920 0x55a9e2dbed10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:37.981 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.974+0000 7f87de47d740 4 client.15975 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-31T23:12:37.981 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.974+0000 7f87de47d740 1 -- 172.21.15.46:0/240180979 --> [v2:172.21.15.142:6836/2559258399,v1:172.21.15.142:6837/2559258399] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a9e2d146f0 con 0x55a9e2dbc920 2024-03-31T23:12:37.981 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.974+0000 7f87d9638640 1 --2- 172.21.15.46:0/240180979 >> [v2:172.21.15.142:6836/2559258399,v1:172.21.15.142:6837/2559258399] conn(0x55a9e2dbc920 0x55a9e2dbed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:37.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.974+0000 7f87bcff9640 10 client.15975 ms_handle_connect on v2:172.21.15.142:6836/2559258399 2024-03-31T23:12:37.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.974+0000 7f87d9638640 1 --2- 172.21.15.46:0/240180979 >> [v2:172.21.15.142:6836/2559258399,v1:172.21.15.142:6837/2559258399] conn(0x55a9e2dbc920 0x55a9e2dbed10 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=bbf1e2931adfc71d server_cookie=457572072d4dd8de in_seq=0 out_seq=0 2024-03-31T23:12:37.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.975+0000 7f87bcff9640 1 -- 172.21.15.46:0/240180979 <== mds.1 v2:172.21.15.142:6836/2559258399 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55a9e2d146f0 con 0x55a9e2dbc920 2024-03-31T23:12:37.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.975+0000 7f87bcff9640 10 client.15975 handle_command_reply: tid=0 2024-03-31T23:12:37.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.975+0000 7f87de47d740 1 client.15975 shutdown 2024-03-31T23:12:37.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.976+0000 7f87de47d740 1 -- 172.21.15.46:0/240180979 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a9e2da3d90 msgr2=0x55a9e2da9eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:12:37.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.976+0000 7f87de47d740 1 --2- 172.21.15.46:0/240180979 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a9e2da3d90 0x55a9e2da9eb0 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f87c409b5b0 tx=0x7f87c400cf90 comp rx=0 tx=0).stop 2024-03-31T23:12:37.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.976+0000 7f87de47d740 1 -- 172.21.15.46:0/240180979 shutdown_connections 2024-03-31T23:12:37.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.976+0000 7f87de47d740 1 --2- 172.21.15.46:0/240180979 >> [v2:172.21.15.142:6836/2559258399,v1:172.21.15.142:6837/2559258399] conn(0x55a9e2dbc920 0x55a9e2dbed10 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:37.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.976+0000 7f87de47d740 1 --2- 172.21.15.46:0/240180979 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a9e2dac970 0x55a9e2daa3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:37.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.976+0000 7f87bcff9640 0 client.15975 ms_handle_reset on v2:172.21.15.142:6836/2559258399 2024-03-31T23:12:37.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.976+0000 7f87de47d740 1 --2- 172.21.15.46:0/240180979 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a9e2da3d90 0x55a9e2da9eb0 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:37.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.976+0000 7f87de47d740 1 --2- 172.21.15.46:0/240180979 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a9e2da3470 0x55a9e2da9970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:37.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.976+0000 7f87de47d740 1 -- 172.21.15.46:0/240180979 >> 172.21.15.46:0/240180979 conn(0x55a9e2d9dba0 msgr2=0x55a9e2d9f8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:12:37.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.976+0000 7f87de47d740 1 -- 172.21.15.46:0/240180979 shutdown_connections 2024-03-31T23:12:37.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.976+0000 7f87de47d740 1 -- 172.21.15.46:0/240180979 wait complete. 2024-03-31T23:12:37.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:37.976+0000 7f87de47d740 20 client.15975 trim_cache size 0 max 16384 2024-03-31T23:12:38.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.002+0000 7f87de47d740 1 Processor -- start 2024-03-31T23:12:38.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.002+0000 7f87de47d740 1 -- start start 2024-03-31T23:12:38.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.003+0000 7f87de47d740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a9e2da3470 0x55a9e2db8500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:38.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.003+0000 7f87de47d740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a9e2dac970 0x55a9e2dc4570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:38.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.003+0000 7f87de47d740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a9e2dc4ab0 0x55a9e2db4b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:38.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.003+0000 7f87de47d740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f87d41b36d0 con 0x55a9e2da3470 2024-03-31T23:12:38.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.003+0000 7f87de47d740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f87d41b0fa0 con 0x55a9e2dac970 2024-03-31T23:12:38.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.003+0000 7f87de47d740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f87d41aedf0 con 0x55a9e2dc4ab0 2024-03-31T23:12:38.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.003+0000 7f87d9638640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a9e2da3470 0x55a9e2db8500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:38.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.003+0000 7f87d8e37640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a9e2dac970 0x55a9e2dc4570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:38.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.003+0000 7f87d9e39640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a9e2dc4ab0 0x55a9e2db4b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:38.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.003+0000 7f87d9638640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a9e2da3470 0x55a9e2db8500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:32810/0 (socket says 172.21.15.46:32810) 2024-03-31T23:12:38.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.003+0000 7f87d9638640 1 -- 172.21.15.46:0/428518177 learned_addr learned my addr 172.21.15.46:0/428518177 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:12:38.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.003+0000 7f87d9638640 1 -- 172.21.15.46:0/428518177 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a9e2dc4ab0 msgr2=0x55a9e2db4b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:12:38.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.003+0000 7f87d9638640 1 --2- 172.21.15.46:0/428518177 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a9e2dc4ab0 0x55a9e2db4b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:38.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.003+0000 7f87d9638640 1 -- 172.21.15.46:0/428518177 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a9e2dac970 msgr2=0x55a9e2dc4570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:12:38.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.003+0000 7f87d9638640 1 --2- 172.21.15.46:0/428518177 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a9e2dac970 0x55a9e2dc4570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:38.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.003+0000 7f87d9638640 1 -- 172.21.15.46:0/428518177 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a9e2dbf080 con 0x55a9e2da3470 2024-03-31T23:12:38.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.003+0000 7f87d8e37640 1 --2- 172.21.15.46:0/428518177 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a9e2dac970 0x55a9e2dc4570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:12:38.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.003+0000 7f87d9638640 1 --2- 172.21.15.46:0/428518177 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a9e2da3470 0x55a9e2db8500 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f87d0011e30 tx=0x7f87d0000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1112777a627a8c13 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:12:38.009 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.004+0000 7f87beffd640 1 -- 172.21.15.46:0/428518177 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87d0031070 con 0x55a9e2da3470 2024-03-31T23:12:38.009 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.004+0000 7f87beffd640 1 -- 172.21.15.46:0/428518177 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f87d00047e0 con 0x55a9e2da3470 2024-03-31T23:12:38.009 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.004+0000 7f87beffd640 1 -- 172.21.15.46:0/428518177 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87d0025810 con 0x55a9e2da3470 2024-03-31T23:12:38.009 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.004+0000 7f87de47d740 1 -- 172.21.15.46:0/428518177 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a9e2da3470 msgr2=0x55a9e2db8500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:12:38.009 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.004+0000 7f87de47d740 1 --2- 172.21.15.46:0/428518177 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a9e2da3470 0x55a9e2db8500 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f87d0011e30 tx=0x7f87d0000bc0 comp rx=0 tx=0).stop 2024-03-31T23:12:38.010 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.004+0000 7f87de47d740 1 -- 172.21.15.46:0/428518177 shutdown_connections 2024-03-31T23:12:38.010 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.004+0000 7f87de47d740 1 --2- 172.21.15.46:0/428518177 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a9e2dc4ab0 0x55a9e2db4b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:38.010 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.004+0000 7f87de47d740 1 --2- 172.21.15.46:0/428518177 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a9e2dac970 0x55a9e2dc4570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:38.010 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.004+0000 7f87de47d740 1 --2- 172.21.15.46:0/428518177 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a9e2da3470 0x55a9e2db8500 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:38.010 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.004+0000 7f87de47d740 1 -- 172.21.15.46:0/428518177 >> 172.21.15.46:0/428518177 conn(0x55a9e2d9dba0 msgr2=0x55a9e2d9c4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:12:38.010 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.005+0000 7f87de47d740 1 -- 172.21.15.46:0/428518177 shutdown_connections 2024-03-31T23:12:38.010 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.005+0000 7f87de47d740 1 -- 172.21.15.46:0/428518177 wait complete. 2024-03-31T23:12:38.010 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.005+0000 7f87de47d740 1 -- start start 2024-03-31T23:12:38.010 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.005+0000 7f87de47d740 1 Processor -- start 2024-03-31T23:12:38.010 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.005+0000 7f87de47d740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a9e2da3470 0x55a9e2db4f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:38.010 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.005+0000 7f87de47d740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a9e2dac970 0x55a9e2db12c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:38.010 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.006+0000 7f87de47d740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a9e2dc4ab0 0x55a9e2db19e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:38.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.006+0000 7f87d9638640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a9e2da3470 0x55a9e2db4f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:38.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.006+0000 7f87d9638640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a9e2da3470 0x55a9e2db4f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:32820/0 (socket says 172.21.15.46:32820) 2024-03-31T23:12:38.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.006+0000 7f87d8e37640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a9e2dac970 0x55a9e2db12c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:38.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.006+0000 7f87d8e37640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a9e2dac970 0x55a9e2db12c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.46:51876/0 (socket says 172.21.15.46:51876) 2024-03-31T23:12:38.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.006+0000 7f87d9638640 1 -- 172.21.15.46:0/1032401354 learned_addr learned my addr 172.21.15.46:0/1032401354 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:12:38.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.006+0000 7f87de47d740 1 -- 172.21.15.46:0/1032401354 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f87d41af460 con 0x55a9e2da3470 2024-03-31T23:12:38.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.006+0000 7f87de47d740 1 -- 172.21.15.46:0/1032401354 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f87d41b08d0 con 0x55a9e2dac970 2024-03-31T23:12:38.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.006+0000 7f87de47d740 1 -- 172.21.15.46:0/1032401354 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f87d41afb30 con 0x55a9e2dc4ab0 2024-03-31T23:12:38.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.006+0000 7f87d9e39640 1 --2- 172.21.15.46:0/1032401354 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a9e2dc4ab0 0x55a9e2db19e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:38.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.006+0000 7f87de47d740 10 client.0 _ll_register_callbacks cb 0x7f87c409b070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T23:12:38.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.006+0000 7f87d9638640 1 -- 172.21.15.46:0/1032401354 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a9e2dc4ab0 msgr2=0x55a9e2db19e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:12:38.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.006+0000 7f87d9638640 1 --2- 172.21.15.46:0/1032401354 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a9e2dc4ab0 0x55a9e2db19e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:38.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.006+0000 7f87d9638640 1 -- 172.21.15.46:0/1032401354 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a9e2dac970 msgr2=0x55a9e2db12c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:12:38.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.006+0000 7f87d9638640 1 --2- 172.21.15.46:0/1032401354 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a9e2dac970 0x55a9e2db12c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:38.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.006+0000 7f87d9638640 1 -- 172.21.15.46:0/1032401354 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a9e2e22bb0 con 0x55a9e2da3470 2024-03-31T23:12:38.012 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.006+0000 7f87d9e39640 1 --2- 172.21.15.46:0/1032401354 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a9e2dc4ab0 0x55a9e2db19e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:12:38.012 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.006+0000 7f87d9638640 1 --2- 172.21.15.46:0/1032401354 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a9e2da3470 0x55a9e2db4f40 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f87d0010010 tx=0x7f87d0005d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d4cb0a3fe8f32143 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:12:38.012 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.006+0000 7f87bcff9640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-31T23:12:38.012 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.007+0000 7f87bcff9640 1 -- 172.21.15.46:0/1032401354 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87d0031070 con 0x55a9e2da3470 2024-03-31T23:12:38.012 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.007+0000 7f87bcff9640 1 -- 172.21.15.46:0/1032401354 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7f87d000f740 con 0x55a9e2da3470 2024-03-31T23:12:38.012 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.007+0000 7f87de47d740 1 -- 172.21.15.46:0/1032401354 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f87d00047e0 con 0x55a9e2da3470 2024-03-31T23:12:38.012 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.007+0000 7f87bcff9640 1 -- 172.21.15.46:0/1032401354 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f87d0004020 con 0x55a9e2da3470 2024-03-31T23:12:38.012 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.007+0000 7f87de47d740 10 client.15981 fetch_fsmap learned FSMap version 25 2024-03-31T23:12:38.012 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.007+0000 7f87de47d740 1 -- 172.21.15.46:0/1032401354 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55a9e2e23cf0 con 0x55a9e2da3470 2024-03-31T23:12:38.012 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.007+0000 7f87bcff9640 1 -- 172.21.15.46:0/1032401354 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f87d00047e0 con 0x55a9e2da3470 2024-03-31T23:12:38.012 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.008+0000 7f87bcff9640 1 -- 172.21.15.46:0/1032401354 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f87d002c040 con 0x55a9e2da3470 2024-03-31T23:12:38.012 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.008+0000 7f87de47d740 10 client.15981 fetch_fsmap finished waiting for FSMap version 25 2024-03-31T23:12:38.012 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.008+0000 7f87de47d740 10 client.15981 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-03-31T23:12:38.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.008+0000 7f87de47d740 20 client.15981 populate_metadata read hostname 'smithi046' 2024-03-31T23:12:38.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.008+0000 7f87de47d740 1 --2- 172.21.15.46:0/1032401354 >> [v2:172.21.15.142:6836/2559258399,v1:172.21.15.142:6837/2559258399] conn(0x55a9e2e241d0 0x55a9e2e265c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:38.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.008+0000 7f87de47d740 4 client.15981 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-31T23:12:38.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.008+0000 7f87de47d740 1 -- 172.21.15.46:0/1032401354 --> [v2:172.21.15.142:6836/2559258399,v1:172.21.15.142:6837/2559258399] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55a9e2d0ea40 con 0x55a9e2e241d0 2024-03-31T23:12:38.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.008+0000 7f87d8e37640 1 --2- 172.21.15.46:0/1032401354 >> [v2:172.21.15.142:6836/2559258399,v1:172.21.15.142:6837/2559258399] conn(0x55a9e2e241d0 0x55a9e2e265c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:38.014 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.009+0000 7f87d8e37640 1 --2- 172.21.15.46:0/1032401354 >> [v2:172.21.15.142:6836/2559258399,v1:172.21.15.142:6837/2559258399] conn(0x55a9e2e241d0 0x55a9e2e265c0 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=8a9388048d41f87d server_cookie=e21676d5d9515442 in_seq=0 out_seq=0 2024-03-31T23:12:38.014 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.009+0000 7f87bcff9640 10 client.15981 ms_handle_connect on v2:172.21.15.142:6836/2559258399 2024-03-31T23:12:38.014 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.009+0000 7f87bcff9640 1 -- 172.21.15.46:0/1032401354 <== mds.1 v2:172.21.15.142:6836/2559258399 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55a9e2d0ea40 con 0x55a9e2e241d0 2024-03-31T23:12:38.014 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.009+0000 7f87bcff9640 10 client.15981 handle_command_reply: tid=0 2024-03-31T23:12:38.015 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.009+0000 7f87de47d740 1 client.15981 shutdown 2024-03-31T23:12:38.015 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.009+0000 7f87de47d740 1 -- 172.21.15.46:0/1032401354 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a9e2da3470 msgr2=0x55a9e2db4f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:12:38.015 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.009+0000 7f87de47d740 1 --2- 172.21.15.46:0/1032401354 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a9e2da3470 0x55a9e2db4f40 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f87d0010010 tx=0x7f87d0005d00 comp rx=0 tx=0).stop 2024-03-31T23:12:38.015 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.009+0000 7f87de47d740 1 -- 172.21.15.46:0/1032401354 shutdown_connections 2024-03-31T23:12:38.015 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.009+0000 7f87de47d740 1 --2- 172.21.15.46:0/1032401354 >> [v2:172.21.15.142:6836/2559258399,v1:172.21.15.142:6837/2559258399] conn(0x55a9e2e241d0 0x55a9e2e265c0 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:38.015 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.009+0000 7f87de47d740 1 --2- 172.21.15.46:0/1032401354 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a9e2dc4ab0 0x55a9e2db19e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:38.015 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.009+0000 7f87bcff9640 0 client.15981 ms_handle_reset on v2:172.21.15.142:6836/2559258399 2024-03-31T23:12:38.015 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.009+0000 7f87de47d740 1 --2- 172.21.15.46:0/1032401354 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a9e2dac970 0x55a9e2db12c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:38.015 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.009+0000 7f87de47d740 1 --2- 172.21.15.46:0/1032401354 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a9e2da3470 0x55a9e2db4f40 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:38.015 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.009+0000 7f87de47d740 1 -- 172.21.15.46:0/1032401354 >> 172.21.15.46:0/1032401354 conn(0x55a9e2d9dba0 msgr2=0x55a9e2d9cd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:12:38.015 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.009+0000 7f87de47d740 1 -- 172.21.15.46:0/1032401354 shutdown_connections 2024-03-31T23:12:38.015 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.009+0000 7f87de47d740 1 -- 172.21.15.46:0/1032401354 wait complete. 2024-03-31T23:12:38.015 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.009+0000 7f87de47d740 20 client.15981 trim_cache size 0 max 16384 2024-03-31T23:12:38.016 INFO:teuthology.orchestra.run.smithi046.stdout:[] 2024-03-31T23:12:38.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.010+0000 7f87db89c640 1 -- 172.21.15.46:0/1340631537 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f87c00a27e0 msgr2=0x7f87c00a4ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:12:38.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.010+0000 7f87db89c640 1 --2- 172.21.15.46:0/1340631537 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f87c00a27e0 0x7f87c00a4ca0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f87cc0099a0 tx=0x7f87cc009310 comp rx=0 tx=0).stop 2024-03-31T23:12:38.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.010+0000 7f87db89c640 1 -- 172.21.15.46:0/1340631537 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f87d4069310 msgr2=0x7f87d4100b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:12:38.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.010+0000 7f87db89c640 1 --2- 172.21.15.46:0/1340631537 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f87d4069310 0x7f87d4100b50 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f87c400e9f0 tx=0x7f87c400eeb0 comp rx=0 tx=0).stop 2024-03-31T23:12:38.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.010+0000 7f87db89c640 1 -- 172.21.15.46:0/1340631537 shutdown_connections 2024-03-31T23:12:38.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.010+0000 7f87db89c640 1 --2- 172.21.15.46:0/1340631537 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f87c00a27e0 0x7f87c00a4ca0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:38.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.010+0000 7f87db89c640 1 --2- 172.21.15.46:0/1340631537 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f87d41077a0 0x7f87d4101090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:38.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.010+0000 7f87db89c640 1 --2- 172.21.15.46:0/1340631537 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f87d4069310 0x7f87d4100b50 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:38.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.010+0000 7f87db89c640 1 --2- 172.21.15.46:0/1340631537 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f87d4068950 0x7f87d4100610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:38.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.010+0000 7f87db89c640 1 -- 172.21.15.46:0/1340631537 >> 172.21.15.46:0/1340631537 conn(0x7f87d4077750 msgr2=0x7f87d40fe1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:12:38.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.010+0000 7f87db89c640 1 -- 172.21.15.46:0/1340631537 shutdown_connections 2024-03-31T23:12:38.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.010+0000 7f87db89c640 1 -- 172.21.15.46:0/1340631537 wait complete. 2024-03-31T23:12:38.023 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-31T23:12:38.130 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.128+0000 7ff69ebe9640 1 -- 172.21.15.46:0/3132320394 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff6981077a0 msgr2=0x7ff698107b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:12:38.130 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.128+0000 7ff69ebe9640 1 --2- 172.21.15.46:0/3132320394 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff6981077a0 0x7ff698107b80 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7ff68c009960 tx=0x7ff68c02bc20 comp rx=0 tx=0).stop 2024-03-31T23:12:38.130 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.128+0000 7ff69ebe9640 1 -- 172.21.15.46:0/3132320394 shutdown_connections 2024-03-31T23:12:38.130 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.128+0000 7ff69ebe9640 1 --2- 172.21.15.46:0/3132320394 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff698069310 0x7ff69810e5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:38.130 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.128+0000 7ff69ebe9640 1 --2- 172.21.15.46:0/3132320394 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff698068950 0x7ff698068dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:38.130 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.128+0000 7ff69ebe9640 1 --2- 172.21.15.46:0/3132320394 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff6981077a0 0x7ff698107b80 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:38.130 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.128+0000 7ff69ebe9640 1 -- 172.21.15.46:0/3132320394 >> 172.21.15.46:0/3132320394 conn(0x7ff698077750 msgr2=0x7ff698077b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:12:38.131 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.128+0000 7ff69ebe9640 1 -- 172.21.15.46:0/3132320394 shutdown_connections 2024-03-31T23:12:38.131 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.128+0000 7ff69ebe9640 1 -- 172.21.15.46:0/3132320394 wait complete. 2024-03-31T23:12:38.131 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.129+0000 7ff69ebe9640 1 Processor -- start 2024-03-31T23:12:38.131 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.129+0000 7ff69ebe9640 1 -- start start 2024-03-31T23:12:38.131 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.129+0000 7ff69ebe9640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff698068950 0x7ff698197f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:38.131 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.129+0000 7ff69ebe9640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff698069310 0x7ff6981984b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:38.131 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.129+0000 7ff69ebe9640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff6981077a0 0x7ff69819c830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:38.131 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.129+0000 7ff69ebe9640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7ff6981119b0 con 0x7ff6981077a0 2024-03-31T23:12:38.131 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.129+0000 7ff69ebe9640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7ff698111830 con 0x7ff698069310 2024-03-31T23:12:38.131 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.129+0000 7ff69ebe9640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7ff698111b30 con 0x7ff698068950 2024-03-31T23:12:38.131 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.129+0000 7ff69c985640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff698068950 0x7ff698197f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:38.131 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.129+0000 7ff69d186640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff6981077a0 0x7ff69819c830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:38.131 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.129+0000 7ff69d186640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff6981077a0 0x7ff69819c830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:32844/0 (socket says 172.21.15.46:32844) 2024-03-31T23:12:38.132 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.129+0000 7ff69d186640 1 -- 172.21.15.46:0/2083897095 learned_addr learned my addr 172.21.15.46:0/2083897095 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:12:38.132 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.129+0000 7ff697fff640 1 --2- 172.21.15.46:0/2083897095 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff698069310 0x7ff6981984b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:38.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.129+0000 7ff69c985640 1 -- 172.21.15.46:0/2083897095 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff698069310 msgr2=0x7ff6981984b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:12:38.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.129+0000 7ff69c985640 1 --2- 172.21.15.46:0/2083897095 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff698069310 0x7ff6981984b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:38.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.129+0000 7ff69c985640 1 -- 172.21.15.46:0/2083897095 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff6981077a0 msgr2=0x7ff69819c830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:12:38.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.129+0000 7ff69c985640 1 --2- 172.21.15.46:0/2083897095 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff6981077a0 0x7ff69819c830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:38.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.129+0000 7ff69c985640 1 -- 172.21.15.46:0/2083897095 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff69819cf60 con 0x7ff698068950 2024-03-31T23:12:38.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.129+0000 7ff69d186640 1 --2- 172.21.15.46:0/2083897095 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff6981077a0 0x7ff69819c830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:12:38.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.130+0000 7ff69c985640 1 --2- 172.21.15.46:0/2083897095 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff698068950 0x7ff698197f70 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7ff68c02c160 tx=0x7ff68c02c600 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:12:38.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.130+0000 7ff695ffb640 1 -- 172.21.15.46:0/2083897095 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff68c043020 con 0x7ff698068950 2024-03-31T23:12:38.134 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.130+0000 7ff69ebe9640 1 -- 172.21.15.46:0/2083897095 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff69819d1f0 con 0x7ff698068950 2024-03-31T23:12:38.134 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.130+0000 7ff695ffb640 1 -- 172.21.15.46:0/2083897095 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff68c02ca60 con 0x7ff698068950 2024-03-31T23:12:38.134 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.130+0000 7ff695ffb640 1 -- 172.21.15.46:0/2083897095 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff68c003a50 con 0x7ff698068950 2024-03-31T23:12:38.134 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.130+0000 7ff69ebe9640 1 -- 172.21.15.46:0/2083897095 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff6981a4ae0 con 0x7ff698068950 2024-03-31T23:12:38.134 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.132+0000 7ff695ffb640 1 -- 172.21.15.46:0/2083897095 <== mon.1 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 24) ==== 137553+0+0 (secure 0 0 0) 0x7ff68c046020 con 0x7ff698068950 2024-03-31T23:12:38.135 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.133+0000 7ff695ffb640 1 --2- 172.21.15.46:0/2083897095 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7ff6880a2730 0x7ff6880a4bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:38.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.133+0000 7ff6a17cf740 1 Processor -- start 2024-03-31T23:12:38.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.133+0000 7ff6a17cf740 1 -- start start 2024-03-31T23:12:38.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.133+0000 7ff695ffb640 1 -- 172.21.15.46:0/2083897095 <== mon.1 v2:172.21.15.162:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7ff68c0f0fd0 con 0x7ff698068950 2024-03-31T23:12:38.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.133+0000 7ff697fff640 1 --2- 172.21.15.46:0/2083897095 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7ff6880a2730 0x7ff6880a4bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:38.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.133+0000 7ff6a17cf740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5556d7b5c470 0x5556d7b5c850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:38.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.133+0000 7ff6a17cf740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5556d7b5cd90 0x5556d7b65430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:38.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.133+0000 7ff6a17cf740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5556d7b65970 0x5556d7b67d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:38.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.133+0000 7ff6a17cf740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x5556d7ac01b0 con 0x5556d7b5c470 2024-03-31T23:12:38.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.133+0000 7ff6a17cf740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x5556d7a68c00 con 0x5556d7b65970 2024-03-31T23:12:38.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.134+0000 7ff6a17cf740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5556d7abc250 con 0x5556d7b5cd90 2024-03-31T23:12:38.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.134+0000 7ff69c985640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5556d7b5c470 0x5556d7b5c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:38.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.134+0000 7ff69d186640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5556d7b65970 0x5556d7b67d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:38.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.134+0000 7ff69c985640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5556d7b5c470 0x5556d7b5c850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:32848/0 (socket says 172.21.15.46:32848) 2024-03-31T23:12:38.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.134+0000 7ff69c985640 1 -- 172.21.15.46:0/54257414 learned_addr learned my addr 172.21.15.46:0/54257414 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:12:38.137 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.134+0000 7ff697fff640 1 --2- 172.21.15.46:0/54257414 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5556d7b5cd90 0x5556d7b65430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:38.137 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.134+0000 7ff697fff640 1 --2- 172.21.15.46:0/2083897095 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7ff6880a2730 0x7ff6880a4bf0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7ff698100140 tx=0x7ff684005f90 comp rx=0 tx=0).ready entity=mgr.15654 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:12:38.137 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.134+0000 7ff69c985640 1 -- 172.21.15.46:0/54257414 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5556d7b5cd90 msgr2=0x5556d7b65430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:12:38.137 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.134+0000 7ff69c985640 1 --2- 172.21.15.46:0/54257414 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5556d7b5cd90 0x5556d7b65430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:38.137 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.134+0000 7ff69c985640 1 -- 172.21.15.46:0/54257414 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5556d7b65970 msgr2=0x5556d7b67d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:12:38.137 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.134+0000 7ff69c985640 1 --2- 172.21.15.46:0/54257414 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5556d7b65970 0x5556d7b67d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:38.137 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.134+0000 7ff69c985640 1 -- 172.21.15.46:0/54257414 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5556d7a883d0 con 0x5556d7b5c470 2024-03-31T23:12:38.137 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.134+0000 7ff69d186640 1 --2- 172.21.15.46:0/54257414 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5556d7b65970 0x5556d7b67d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:12:38.137 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.134+0000 7ff69c985640 1 --2- 172.21.15.46:0/54257414 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5556d7b5c470 0x5556d7b5c850 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7ff68c0b9bd0 tx=0x7ff68c0bcc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c4db56272493b40 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:12:38.137 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.134+0000 7ff682ffd640 1 -- 172.21.15.46:0/54257414 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff68c0dd070 con 0x5556d7b5c470 2024-03-31T23:12:38.137 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.134+0000 7ff682ffd640 1 -- 172.21.15.46:0/54257414 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff68c0c1350 con 0x5556d7b5c470 2024-03-31T23:12:38.137 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.134+0000 7ff682ffd640 1 -- 172.21.15.46:0/54257414 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff68c0dbbe0 con 0x5556d7b5c470 2024-03-31T23:12:38.141 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.135+0000 7ff6a17cf740 1 -- 172.21.15.46:0/54257414 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5556d7b5c470 msgr2=0x5556d7b5c850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:12:38.141 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.135+0000 7ff6a17cf740 1 --2- 172.21.15.46:0/54257414 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5556d7b5c470 0x5556d7b5c850 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7ff68c0b9bd0 tx=0x7ff68c0bcc20 comp rx=0 tx=0).stop 2024-03-31T23:12:38.141 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.135+0000 7ff6a17cf740 1 -- 172.21.15.46:0/54257414 shutdown_connections 2024-03-31T23:12:38.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.135+0000 7ff6a17cf740 1 --2- 172.21.15.46:0/54257414 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5556d7b65970 0x5556d7b67d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:38.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.135+0000 7ff6a17cf740 1 --2- 172.21.15.46:0/54257414 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5556d7b5cd90 0x5556d7b65430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:38.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.135+0000 7ff6a17cf740 1 --2- 172.21.15.46:0/54257414 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5556d7b5c470 0x5556d7b5c850 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:38.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.135+0000 7ff6a17cf740 1 -- 172.21.15.46:0/54257414 >> 172.21.15.46:0/54257414 conn(0x5556d7b56ba0 msgr2=0x5556d7b56f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:12:38.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.135+0000 7ff6a17cf740 1 -- 172.21.15.46:0/54257414 shutdown_connections 2024-03-31T23:12:38.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.135+0000 7ff6a17cf740 1 -- 172.21.15.46:0/54257414 wait complete. 2024-03-31T23:12:38.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.136+0000 7ff6a17cf740 1 -- start start 2024-03-31T23:12:38.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.136+0000 7ff6a17cf740 1 Processor -- start 2024-03-31T23:12:38.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.136+0000 7ff6a17cf740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5556d7b5c470 0x5556d7b65300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:38.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.136+0000 7ff6a17cf740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5556d7b5cd90 0x5556d7b5e650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:38.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.136+0000 7ff6a17cf740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5556d7b65970 0x5556d7b5ebd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:38.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.136+0000 7ff6a17cf740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x5556d7b6c930 con 0x5556d7b65970 2024-03-31T23:12:38.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.136+0000 7ff6a17cf740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x5556d7a5e470 con 0x5556d7b5cd90 2024-03-31T23:12:38.143 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.136+0000 7ff6a17cf740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5556d7b6cab0 con 0x5556d7b5c470 2024-03-31T23:12:38.143 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.136+0000 7ff6a17cf740 10 client.0 _ll_register_callbacks cb 0x5556d7a7bcb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T23:12:38.143 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.136+0000 7ff69c985640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5556d7b5c470 0x5556d7b65300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:38.143 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.136+0000 7ff69d186640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5556d7b65970 0x5556d7b5ebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:38.143 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.136+0000 7ff69c985640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5556d7b5c470 0x5556d7b65300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.46:45570/0 (socket says 172.21.15.46:45570) 2024-03-31T23:12:38.143 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.136+0000 7ff69c985640 1 -- 172.21.15.46:0/3216290678 learned_addr learned my addr 172.21.15.46:0/3216290678 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:12:38.143 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.136+0000 7ff69d186640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5556d7b65970 0x5556d7b5ebd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:32850/0 (socket says 172.21.15.46:32850) 2024-03-31T23:12:38.143 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.136+0000 7ff697fff640 1 --2- 172.21.15.46:0/3216290678 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5556d7b5cd90 0x5556d7b5e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:38.143 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.137+0000 7ff69c985640 1 -- 172.21.15.46:0/3216290678 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5556d7b5cd90 msgr2=0x5556d7b5e650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:12:38.143 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.137+0000 7ff69c985640 1 --2- 172.21.15.46:0/3216290678 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5556d7b5cd90 0x5556d7b5e650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:38.143 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.137+0000 7ff69c985640 1 -- 172.21.15.46:0/3216290678 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5556d7b65970 msgr2=0x5556d7b5ebd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:12:38.143 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.137+0000 7ff69c985640 1 --2- 172.21.15.46:0/3216290678 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5556d7b65970 0x5556d7b5ebd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:38.143 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.137+0000 7ff69c985640 1 -- 172.21.15.46:0/3216290678 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5556d77d43f0 con 0x5556d7b5c470 2024-03-31T23:12:38.143 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.137+0000 7ff69c985640 1 --2- 172.21.15.46:0/3216290678 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5556d7b5c470 0x5556d7b65300 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7ff68c0bceb0 tx=0x7ff68c009680 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bc5da6835f4a3aeb server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:12:38.144 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.137+0000 7ff677fff640 10 client.0 ms_handle_connect on v2:172.21.15.162:3300/0 2024-03-31T23:12:38.144 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.137+0000 7ff677fff640 1 -- 172.21.15.46:0/3216290678 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff68c0d5070 con 0x5556d7b5c470 2024-03-31T23:12:38.144 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.137+0000 7ff677fff640 1 -- 172.21.15.46:0/3216290678 <== mon.1 v2:172.21.15.162:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7ff68c0bd460 con 0x5556d7b5c470 2024-03-31T23:12:38.144 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.137+0000 7ff6a17cf740 1 -- 172.21.15.46:0/3216290678 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff68c0c1350 con 0x5556d7b5c470 2024-03-31T23:12:38.144 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.137+0000 7ff677fff640 1 -- 172.21.15.46:0/3216290678 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7ff68c0a3050 con 0x5556d7b5c470 2024-03-31T23:12:38.144 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.137+0000 7ff6a17cf740 10 client.25732 fetch_fsmap learned FSMap version 25 2024-03-31T23:12:38.144 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.137+0000 7ff6a17cf740 1 -- 172.21.15.46:0/3216290678 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x5556d789e7d0 con 0x5556d7b5c470 2024-03-31T23:12:38.144 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.138+0000 7ff677fff640 1 -- 172.21.15.46:0/3216290678 <== mon.1 v2:172.21.15.162:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff68c0c5c10 con 0x5556d7b5c470 2024-03-31T23:12:38.144 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.138+0000 7ff677fff640 1 -- 172.21.15.46:0/3216290678 <== mon.1 v2:172.21.15.162:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7ff68c0da020 con 0x5556d7b5c470 2024-03-31T23:12:38.144 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.138+0000 7ff6a17cf740 10 client.25732 fetch_fsmap finished waiting for FSMap version 25 2024-03-31T23:12:38.144 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.138+0000 7ff6a17cf740 10 client.25732 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-31T23:12:38.144 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.138+0000 7ff6a17cf740 20 client.25732 populate_metadata read hostname 'smithi046' 2024-03-31T23:12:38.144 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.138+0000 7ff6a17cf740 1 --2- 172.21.15.46:0/3216290678 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x5556d7b7de00 0x5556d7b801f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:38.144 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.138+0000 7ff6a17cf740 4 client.25732 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-31T23:12:38.145 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.138+0000 7ff6a17cf740 1 -- 172.21.15.46:0/3216290678 --> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7ff68c0c1350 con 0x5556d7b7de00 2024-03-31T23:12:38.145 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.138+0000 7ff697fff640 1 --2- 172.21.15.46:0/3216290678 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x5556d7b7de00 0x5556d7b801f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:38.145 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.138+0000 7ff697fff640 1 --2- 172.21.15.46:0/3216290678 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x5556d7b7de00 0x5556d7b801f0 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2d6111844a064e00 server_cookie=6b00da555dfa9338 in_seq=0 out_seq=0 2024-03-31T23:12:38.146 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.138+0000 7ff677fff640 10 client.25732 ms_handle_connect on v2:172.21.15.142:6838/2519143918 2024-03-31T23:12:38.146 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.139+0000 7ff677fff640 1 -- 172.21.15.46:0/3216290678 <== mds.0 v2:172.21.15.142:6838/2519143918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x7ff68c0c1350 con 0x5556d7b7de00 2024-03-31T23:12:38.146 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.139+0000 7ff677fff640 10 client.25732 handle_command_reply: tid=0 2024-03-31T23:12:38.146 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.139+0000 7ff6a17cf740 1 client.25732 shutdown 2024-03-31T23:12:38.146 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.139+0000 7ff6a17cf740 1 -- 172.21.15.46:0/3216290678 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5556d7b5c470 msgr2=0x5556d7b65300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:12:38.146 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.139+0000 7ff6a17cf740 1 --2- 172.21.15.46:0/3216290678 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5556d7b5c470 0x5556d7b65300 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7ff68c0bceb0 tx=0x7ff68c009680 comp rx=0 tx=0).stop 2024-03-31T23:12:38.146 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.139+0000 7ff6a17cf740 1 -- 172.21.15.46:0/3216290678 shutdown_connections 2024-03-31T23:12:38.146 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.139+0000 7ff6a17cf740 1 --2- 172.21.15.46:0/3216290678 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x5556d7b7de00 0x5556d7b801f0 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:38.146 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.139+0000 7ff677fff640 0 client.25732 ms_handle_reset on v2:172.21.15.142:6838/2519143918 2024-03-31T23:12:38.146 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.139+0000 7ff6a17cf740 1 --2- 172.21.15.46:0/3216290678 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5556d7b65970 0x5556d7b5ebd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:38.146 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.139+0000 7ff6a17cf740 1 --2- 172.21.15.46:0/3216290678 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5556d7b5cd90 0x5556d7b5e650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:38.146 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.139+0000 7ff6a17cf740 1 --2- 172.21.15.46:0/3216290678 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5556d7b5c470 0x5556d7b65300 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:38.146 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.139+0000 7ff6a17cf740 1 -- 172.21.15.46:0/3216290678 >> 172.21.15.46:0/3216290678 conn(0x5556d7b56ba0 msgr2=0x5556d7b588f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:12:38.147 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.139+0000 7ff6a17cf740 1 -- 172.21.15.46:0/3216290678 shutdown_connections 2024-03-31T23:12:38.147 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.139+0000 7ff6a17cf740 1 -- 172.21.15.46:0/3216290678 wait complete. 2024-03-31T23:12:38.147 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.140+0000 7ff6a17cf740 20 client.25732 trim_cache size 0 max 16384 2024-03-31T23:12:38.155 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.153+0000 7ff6a17cf740 1 Processor -- start 2024-03-31T23:12:38.155 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.153+0000 7ff6a17cf740 1 -- start start 2024-03-31T23:12:38.155 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.153+0000 7ff6a17cf740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5556d7b5c470 0x5556d7b64e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:38.155 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.153+0000 7ff6a17cf740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5556d7b5cd90 0x5556d7b85220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:38.155 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.153+0000 7ff6a17cf740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5556d7b65970 0x5556d7b85760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:38.155 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.153+0000 7ff6a17cf740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7ff6981af290 con 0x5556d7b5cd90 2024-03-31T23:12:38.156 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.153+0000 7ff6a17cf740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7ff6981acb60 con 0x5556d7b65970 2024-03-31T23:12:38.156 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.153+0000 7ff6a17cf740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7ff6981aa9b0 con 0x5556d7b5c470 2024-03-31T23:12:38.156 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.153+0000 7ff697fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5556d7b5cd90 0x5556d7b85220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:38.156 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.153+0000 7ff69c985640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5556d7b5c470 0x5556d7b64e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:38.156 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.153+0000 7ff697fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5556d7b5cd90 0x5556d7b85220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:32854/0 (socket says 172.21.15.46:32854) 2024-03-31T23:12:38.156 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.153+0000 7ff697fff640 1 -- 172.21.15.46:0/1479599136 learned_addr learned my addr 172.21.15.46:0/1479599136 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:12:38.156 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.153+0000 7ff69d186640 1 --2- 172.21.15.46:0/1479599136 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5556d7b65970 0x5556d7b85760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:38.156 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.153+0000 7ff697fff640 1 -- 172.21.15.46:0/1479599136 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5556d7b5c470 msgr2=0x5556d7b64e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:12:38.156 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.153+0000 7ff697fff640 1 --2- 172.21.15.46:0/1479599136 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5556d7b5c470 0x5556d7b64e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:38.156 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.153+0000 7ff697fff640 1 -- 172.21.15.46:0/1479599136 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5556d7b65970 msgr2=0x5556d7b85760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:12:38.156 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.153+0000 7ff697fff640 1 --2- 172.21.15.46:0/1479599136 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5556d7b65970 0x5556d7b85760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:38.156 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.154+0000 7ff697fff640 1 -- 172.21.15.46:0/1479599136 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5556d7b5ef40 con 0x5556d7b5cd90 2024-03-31T23:12:38.156 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.154+0000 7ff69c985640 1 --2- 172.21.15.46:0/1479599136 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5556d7b5c470 0x5556d7b64e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:12:38.156 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.154+0000 7ff69d186640 1 --2- 172.21.15.46:0/1479599136 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5556d7b65970 0x5556d7b85760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:12:38.156 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.154+0000 7ff697fff640 1 --2- 172.21.15.46:0/1479599136 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5556d7b5cd90 0x5556d7b85220 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7ff684009a90 tx=0x7ff684019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=68b479ec9e90f782 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:12:38.157 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.154+0000 7ff6827fc640 1 -- 172.21.15.46:0/1479599136 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff684031070 con 0x5556d7b5cd90 2024-03-31T23:12:38.157 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.154+0000 7ff6827fc640 1 -- 172.21.15.46:0/1479599136 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff68402b610 con 0x5556d7b5cd90 2024-03-31T23:12:38.157 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.154+0000 7ff6827fc640 1 -- 172.21.15.46:0/1479599136 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff68401cb60 con 0x5556d7b5cd90 2024-03-31T23:12:38.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.155+0000 7ff6a17cf740 1 -- 172.21.15.46:0/1479599136 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5556d7b5cd90 msgr2=0x5556d7b85220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:12:38.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.155+0000 7ff6a17cf740 1 --2- 172.21.15.46:0/1479599136 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5556d7b5cd90 0x5556d7b85220 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7ff684009a90 tx=0x7ff684019000 comp rx=0 tx=0).stop 2024-03-31T23:12:38.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.155+0000 7ff6a17cf740 1 -- 172.21.15.46:0/1479599136 shutdown_connections 2024-03-31T23:12:38.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.155+0000 7ff6a17cf740 1 --2- 172.21.15.46:0/1479599136 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5556d7b65970 0x5556d7b85760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:38.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.155+0000 7ff6a17cf740 1 --2- 172.21.15.46:0/1479599136 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5556d7b5cd90 0x5556d7b85220 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:38.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.155+0000 7ff6a17cf740 1 --2- 172.21.15.46:0/1479599136 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5556d7b5c470 0x5556d7b64e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:38.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.155+0000 7ff6a17cf740 1 -- 172.21.15.46:0/1479599136 >> 172.21.15.46:0/1479599136 conn(0x5556d7b56ba0 msgr2=0x5556d7b55d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:12:38.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.155+0000 7ff6a17cf740 1 -- 172.21.15.46:0/1479599136 shutdown_connections 2024-03-31T23:12:38.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.155+0000 7ff6a17cf740 1 -- 172.21.15.46:0/1479599136 wait complete. 2024-03-31T23:12:38.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.155+0000 7ff6a17cf740 1 -- start start 2024-03-31T23:12:38.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.155+0000 7ff6a17cf740 1 Processor -- start 2024-03-31T23:12:38.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.156+0000 7ff6a17cf740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5556d7b5c470 0x5556d7b7a500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:38.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.156+0000 7ff6a17cf740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5556d7b5cd90 0x5556d7b7aa40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:38.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.156+0000 7ff697fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5556d7b5cd90 0x5556d7b7aa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:38.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.156+0000 7ff697fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5556d7b5cd90 0x5556d7b7aa40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:32864/0 (socket says 172.21.15.46:32864) 2024-03-31T23:12:38.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.156+0000 7ff697fff640 1 -- 172.21.15.46:0/3361057251 learned_addr learned my addr 172.21.15.46:0/3361057251 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:12:38.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.156+0000 7ff69c985640 1 --2- 172.21.15.46:0/3361057251 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5556d7b5c470 0x5556d7b7a500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:38.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.156+0000 7ff6a17cf740 1 --2- 172.21.15.46:0/3361057251 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5556d7b65970 0x5556d7b774d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:38.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.156+0000 7ff6a17cf740 1 -- 172.21.15.46:0/3361057251 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7ff6981ab020 con 0x5556d7b5cd90 2024-03-31T23:12:38.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.156+0000 7ff6a17cf740 1 -- 172.21.15.46:0/3361057251 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7ff6981ac490 con 0x5556d7b5c470 2024-03-31T23:12:38.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.156+0000 7ff6a17cf740 1 -- 172.21.15.46:0/3361057251 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7ff6981ab6f0 con 0x5556d7b65970 2024-03-31T23:12:38.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.156+0000 7ff6a17cf740 10 client.0 _ll_register_callbacks cb 0x7ff68c0bcd40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T23:12:38.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.156+0000 7ff69d186640 1 --2- 172.21.15.46:0/3361057251 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5556d7b65970 0x5556d7b774d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:38.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.156+0000 7ff69c985640 1 -- 172.21.15.46:0/3361057251 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5556d7b65970 msgr2=0x5556d7b774d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:12:38.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.156+0000 7ff69c985640 1 --2- 172.21.15.46:0/3361057251 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5556d7b65970 0x5556d7b774d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:38.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.156+0000 7ff69c985640 1 -- 172.21.15.46:0/3361057251 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5556d7b5cd90 msgr2=0x5556d7b7aa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:12:38.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.156+0000 7ff69c985640 1 --2- 172.21.15.46:0/3361057251 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5556d7b5cd90 0x5556d7b7aa40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:38.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.156+0000 7ff69c985640 1 -- 172.21.15.46:0/3361057251 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5556d7b7b2d0 con 0x5556d7b5c470 2024-03-31T23:12:38.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.156+0000 7ff697fff640 1 --2- 172.21.15.46:0/3361057251 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5556d7b5cd90 0x5556d7b7aa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:12:38.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.157+0000 7ff69c985640 1 --2- 172.21.15.46:0/3361057251 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5556d7b5c470 0x5556d7b7a500 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7ff68c005d00 tx=0x7ff68c0bb710 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d5a8c12b1da44314 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:12:38.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.157+0000 7ff677fff640 10 client.0 ms_handle_connect on v2:172.21.15.142:3300/0 2024-03-31T23:12:38.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.157+0000 7ff677fff640 1 -- 172.21.15.46:0/3361057251 <== mon.2 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff68c0d5030 con 0x5556d7b5c470 2024-03-31T23:12:38.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.157+0000 7ff677fff640 1 -- 172.21.15.46:0/3361057251 <== mon.2 v2:172.21.15.142:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7ff68c0bc650 con 0x5556d7b5c470 2024-03-31T23:12:38.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.157+0000 7ff6a17cf740 1 -- 172.21.15.46:0/3361057251 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff68402b610 con 0x5556d7b5c470 2024-03-31T23:12:38.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.157+0000 7ff677fff640 1 -- 172.21.15.46:0/3361057251 <== mon.2 v2:172.21.15.142:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7ff68c0c5e60 con 0x5556d7b5c470 2024-03-31T23:12:38.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.157+0000 7ff6a17cf740 10 client.25844 fetch_fsmap learned FSMap version 25 2024-03-31T23:12:38.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.157+0000 7ff6a17cf740 1 -- 172.21.15.46:0/3361057251 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x5556d7b7b2d0 con 0x5556d7b5c470 2024-03-31T23:12:38.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.157+0000 7ff677fff640 1 -- 172.21.15.46:0/3361057251 <== mon.2 v2:172.21.15.142:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff68402b610 con 0x5556d7b5c470 2024-03-31T23:12:38.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.157+0000 7ff677fff640 1 -- 172.21.15.46:0/3361057251 <== mon.2 v2:172.21.15.142:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7ff68c0da070 con 0x5556d7b5c470 2024-03-31T23:12:38.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.157+0000 7ff6a17cf740 10 client.25844 fetch_fsmap finished waiting for FSMap version 25 2024-03-31T23:12:38.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.157+0000 7ff6a17cf740 10 client.25844 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-31T23:12:38.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.157+0000 7ff6a17cf740 20 client.25844 populate_metadata read hostname 'smithi046' 2024-03-31T23:12:38.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.157+0000 7ff6a17cf740 1 --2- 172.21.15.46:0/3361057251 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x5556d7be4740 0x5556d7be6b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:38.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.157+0000 7ff6a17cf740 4 client.25844 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-31T23:12:38.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.157+0000 7ff6a17cf740 1 -- 172.21.15.46:0/3361057251 --> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5556d7acd6f0 con 0x5556d7be4740 2024-03-31T23:12:38.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.158+0000 7ff697fff640 1 --2- 172.21.15.46:0/3361057251 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x5556d7be4740 0x5556d7be6b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:38.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.158+0000 7ff697fff640 1 --2- 172.21.15.46:0/3361057251 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x5556d7be4740 0x5556d7be6b60 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=682a5cfdc9d64cde server_cookie=2c2e6410029dcd3e in_seq=0 out_seq=0 2024-03-31T23:12:38.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.158+0000 7ff677fff640 10 client.25844 ms_handle_connect on v2:172.21.15.142:6838/2519143918 2024-03-31T23:12:38.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.158+0000 7ff677fff640 1 -- 172.21.15.46:0/3361057251 <== mds.0 v2:172.21.15.142:6838/2519143918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5556d7acd6f0 con 0x5556d7be4740 2024-03-31T23:12:38.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.158+0000 7ff677fff640 10 client.25844 handle_command_reply: tid=0 2024-03-31T23:12:38.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.158+0000 7ff6a17cf740 1 client.25844 shutdown 2024-03-31T23:12:38.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.158+0000 7ff6a17cf740 1 -- 172.21.15.46:0/3361057251 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5556d7b5c470 msgr2=0x5556d7b7a500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:12:38.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.158+0000 7ff6a17cf740 1 --2- 172.21.15.46:0/3361057251 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5556d7b5c470 0x5556d7b7a500 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7ff68c005d00 tx=0x7ff68c0bb710 comp rx=0 tx=0).stop 2024-03-31T23:12:38.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.158+0000 7ff6a17cf740 1 -- 172.21.15.46:0/3361057251 shutdown_connections 2024-03-31T23:12:38.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.158+0000 7ff6a17cf740 1 --2- 172.21.15.46:0/3361057251 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x5556d7be4740 0x5556d7be6b60 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:38.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.158+0000 7ff6a17cf740 1 --2- 172.21.15.46:0/3361057251 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5556d7b65970 0x5556d7b774d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:38.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.158+0000 7ff6a17cf740 1 --2- 172.21.15.46:0/3361057251 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5556d7b5cd90 0x5556d7b7aa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:38.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.158+0000 7ff6a17cf740 1 --2- 172.21.15.46:0/3361057251 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5556d7b5c470 0x5556d7b7a500 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:38.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.158+0000 7ff677fff640 0 client.25844 ms_handle_reset on v2:172.21.15.142:6838/2519143918 2024-03-31T23:12:38.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.158+0000 7ff6a17cf740 1 -- 172.21.15.46:0/3361057251 >> 172.21.15.46:0/3361057251 conn(0x5556d7b56ba0 msgr2=0x5556d7b554b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:12:38.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.158+0000 7ff6a17cf740 1 -- 172.21.15.46:0/3361057251 shutdown_connections 2024-03-31T23:12:38.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.158+0000 7ff6a17cf740 1 -- 172.21.15.46:0/3361057251 wait complete. 2024-03-31T23:12:38.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.159+0000 7ff6a17cf740 20 client.25844 trim_cache size 0 max 16384 2024-03-31T23:12:38.165 INFO:teuthology.orchestra.run.smithi046.stdout:[] 2024-03-31T23:12:38.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.159+0000 7ff69ebe9640 1 -- 172.21.15.46:0/2083897095 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7ff6880a2730 msgr2=0x7ff6880a4bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:12:38.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.159+0000 7ff69ebe9640 1 --2- 172.21.15.46:0/2083897095 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7ff6880a2730 0x7ff6880a4bf0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7ff698100140 tx=0x7ff684005f90 comp rx=0 tx=0).stop 2024-03-31T23:12:38.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.159+0000 7ff69ebe9640 1 -- 172.21.15.46:0/2083897095 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff698068950 msgr2=0x7ff698197f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:12:38.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.159+0000 7ff69ebe9640 1 --2- 172.21.15.46:0/2083897095 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff698068950 0x7ff698197f70 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7ff68c02c160 tx=0x7ff68c02c600 comp rx=0 tx=0).stop 2024-03-31T23:12:38.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.159+0000 7ff69ebe9640 1 -- 172.21.15.46:0/2083897095 shutdown_connections 2024-03-31T23:12:38.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.159+0000 7ff69ebe9640 1 --2- 172.21.15.46:0/2083897095 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7ff6880a2730 0x7ff6880a4bf0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:38.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.159+0000 7ff69ebe9640 1 --2- 172.21.15.46:0/2083897095 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff6981077a0 0x7ff69819c830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:38.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.159+0000 7ff69ebe9640 1 --2- 172.21.15.46:0/2083897095 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff698069310 0x7ff6981984b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:38.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.159+0000 7ff69ebe9640 1 --2- 172.21.15.46:0/2083897095 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff698068950 0x7ff698197f70 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:38.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.159+0000 7ff69ebe9640 1 -- 172.21.15.46:0/2083897095 >> 172.21.15.46:0/2083897095 conn(0x7ff698077750 msgr2=0x7ff6980fe3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:12:38.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.159+0000 7ff69ebe9640 1 -- 172.21.15.46:0/2083897095 shutdown_connections 2024-03-31T23:12:38.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:38.159+0000 7ff69ebe9640 1 -- 172.21.15.46:0/2083897095 wait complete. 2024-03-31T23:12:38.611 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:12:38 smithi046 ceph-mon[32498]: pgmap v30: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-03-31T23:12:38.612 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:12:38 smithi046 ceph-mon[32498]: from='client.? 172.21.15.46:0/3227206547' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-31T23:12:38.693 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:12:38 smithi142 ceph-mon[35271]: pgmap v30: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-03-31T23:12:38.693 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:12:38 smithi142 ceph-mon[35271]: from='client.? 172.21.15.46:0/3227206547' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-31T23:12:38.795 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:12:38 smithi162 ceph-mon[36237]: pgmap v30: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-03-31T23:12:38.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:12:38 smithi162 ceph-mon[36237]: from='client.? 172.21.15.46:0/3227206547' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-31T23:12:39.171 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-31T23:12:39.172 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T23:12:39.172 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-31T23:12:39.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.285+0000 7fe86993e640 1 -- 172.21.15.46:0/3021349615 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe8641076f0 msgr2=0x7fe864107ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:12:39.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.285+0000 7fe86993e640 1 --2- 172.21.15.46:0/3021349615 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe8641076f0 0x7fe864107ad0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fe854009900 tx=0x7fe85402bc30 comp rx=0 tx=0).stop 2024-03-31T23:12:39.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.285+0000 7fe86993e640 1 -- 172.21.15.46:0/3021349615 shutdown_connections 2024-03-31T23:12:39.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.285+0000 7fe86993e640 1 --2- 172.21.15.46:0/3021349615 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe8641005b0 0x7fe86410e570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:39.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.285+0000 7fe86993e640 1 --2- 172.21.15.46:0/3021349615 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe8640ffbf0 0x7fe864100070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:39.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.285+0000 7fe86993e640 1 --2- 172.21.15.46:0/3021349615 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe8641076f0 0x7fe864107ad0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:39.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.285+0000 7fe86993e640 1 -- 172.21.15.46:0/3021349615 >> 172.21.15.46:0/3021349615 conn(0x7fe864076b70 msgr2=0x7fe8640fdf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:12:39.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.286+0000 7fe86993e640 1 -- 172.21.15.46:0/3021349615 shutdown_connections 2024-03-31T23:12:39.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.286+0000 7fe86993e640 1 -- 172.21.15.46:0/3021349615 wait complete. 2024-03-31T23:12:39.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.286+0000 7fe86993e640 1 Processor -- start 2024-03-31T23:12:39.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.286+0000 7fe86993e640 1 -- start start 2024-03-31T23:12:39.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.287+0000 7fe86993e640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe8640ffbf0 0x7fe864197f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:39.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.287+0000 7fe86993e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe8641005b0 0x7fe864198470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:39.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.287+0000 7fe8627fc640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe8641005b0 0x7fe864198470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:39.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.287+0000 7fe862ffd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe8640ffbf0 0x7fe864197f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:39.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.287+0000 7fe862ffd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe8640ffbf0 0x7fe864197f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.46:45608/0 (socket says 172.21.15.46:45608) 2024-03-31T23:12:39.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.287+0000 7fe862ffd640 1 -- 172.21.15.46:0/464349197 learned_addr learned my addr 172.21.15.46:0/464349197 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:12:39.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.287+0000 7fe8627fc640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe8641005b0 0x7fe864198470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:32882/0 (socket says 172.21.15.46:32882) 2024-03-31T23:12:39.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.287+0000 7fe86993e640 1 --2- 172.21.15.46:0/464349197 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe8641076f0 0x7fe86419c7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:39.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.287+0000 7fe86993e640 1 -- 172.21.15.46:0/464349197 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fe864111950 con 0x7fe8641005b0 2024-03-31T23:12:39.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.287+0000 7fe86993e640 1 -- 172.21.15.46:0/464349197 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fe8641117d0 con 0x7fe8641076f0 2024-03-31T23:12:39.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.287+0000 7fe86993e640 1 -- 172.21.15.46:0/464349197 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fe864111ad0 con 0x7fe8640ffbf0 2024-03-31T23:12:39.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.287+0000 7fe8637fe640 1 --2- 172.21.15.46:0/464349197 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe8641076f0 0x7fe86419c7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:39.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.287+0000 7fe8627fc640 1 -- 172.21.15.46:0/464349197 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe8640ffbf0 msgr2=0x7fe864197f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:12:39.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.287+0000 7fe8627fc640 1 --2- 172.21.15.46:0/464349197 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe8640ffbf0 0x7fe864197f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:39.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.287+0000 7fe8627fc640 1 -- 172.21.15.46:0/464349197 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe8641076f0 msgr2=0x7fe86419c7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:12:39.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.287+0000 7fe8627fc640 1 --2- 172.21.15.46:0/464349197 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe8641076f0 0x7fe86419c7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:39.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.287+0000 7fe8627fc640 1 -- 172.21.15.46:0/464349197 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe86419cf20 con 0x7fe8641005b0 2024-03-31T23:12:39.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.287+0000 7fe862ffd640 1 --2- 172.21.15.46:0/464349197 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe8640ffbf0 0x7fe864197f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:12:39.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.287+0000 7fe8637fe640 1 --2- 172.21.15.46:0/464349197 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe8641076f0 0x7fe86419c7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T23:12:39.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.288+0000 7fe8627fc640 1 --2- 172.21.15.46:0/464349197 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe8641005b0 0x7fe864198470 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7fe85800e9f0 tx=0x7fe85800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:12:39.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.288+0000 7fe86893c640 1 -- 172.21.15.46:0/464349197 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe85800ccc0 con 0x7fe8641005b0 2024-03-31T23:12:39.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.288+0000 7fe86993e640 1 -- 172.21.15.46:0/464349197 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe86419d210 con 0x7fe8641005b0 2024-03-31T23:12:39.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.288+0000 7fe86893c640 1 -- 172.21.15.46:0/464349197 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe8580108e0 con 0x7fe8641005b0 2024-03-31T23:12:39.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.288+0000 7fe86893c640 1 -- 172.21.15.46:0/464349197 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe858018880 con 0x7fe8641005b0 2024-03-31T23:12:39.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.288+0000 7fe86993e640 1 -- 172.21.15.46:0/464349197 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe8641a4aa0 con 0x7fe8641005b0 2024-03-31T23:12:39.294 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.291+0000 7fe86c5f6740 1 Processor -- start 2024-03-31T23:12:39.294 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.291+0000 7fe86893c640 1 -- 172.21.15.46:0/464349197 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 24) ==== 137553+0+0 (secure 0 0 0) 0x7fe858010a90 con 0x7fe8641005b0 2024-03-31T23:12:39.294 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.291+0000 7fe86c5f6740 1 -- start start 2024-03-31T23:12:39.294 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.292+0000 7fe86c5f6740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x555898ccd3d0 0x555898ccd7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:39.294 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.292+0000 7fe86c5f6740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555898ccdd80 0x555898cdadc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:39.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.292+0000 7fe86c5f6740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555898cdb300 0x555898cdd6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:39.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.292+0000 7fe86c5f6740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x555898bce0b0 con 0x555898cdb300 2024-03-31T23:12:39.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.292+0000 7fe86c5f6740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x555898c30ea0 con 0x555898ccd3d0 2024-03-31T23:12:39.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.292+0000 7fe86c5f6740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x555898bd9c00 con 0x555898ccdd80 2024-03-31T23:12:39.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.292+0000 7fe8627fc640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555898cdb300 0x555898cdd6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:39.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.292+0000 7fe862ffd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555898ccdd80 0x555898cdadc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:39.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.292+0000 7fe86893c640 1 --2- 172.21.15.46:0/464349197 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fe84c0a2730 0x7fe84c0a4bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:39.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.292+0000 7fe8637fe640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x555898ccd3d0 0x555898ccd7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:39.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.292+0000 7fe862ffd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555898ccdd80 0x555898cdadc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.46:45620/0 (socket says 172.21.15.46:45620) 2024-03-31T23:12:39.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.292+0000 7fe862ffd640 1 -- 172.21.15.46:0/371537776 learned_addr learned my addr 172.21.15.46:0/371537776 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:12:39.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.292+0000 7fe8637fe640 1 --2- 172.21.15.46:0/464349197 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fe84c0a2730 0x7fe84c0a4bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:39.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.292+0000 7fe862ffd640 1 -- 172.21.15.46:0/371537776 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x555898ccd3d0 msgr2=0x555898ccd7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:12:39.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.292+0000 7fe862ffd640 1 --2- 172.21.15.46:0/371537776 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x555898ccd3d0 0x555898ccd7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:39.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.292+0000 7fe862ffd640 1 -- 172.21.15.46:0/371537776 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555898cdb300 msgr2=0x555898cdd6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:12:39.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.292+0000 7fe86893c640 1 -- 172.21.15.46:0/464349197 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7fe8580cf570 con 0x7fe8641005b0 2024-03-31T23:12:39.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.293+0000 7fe862ffd640 1 --2- 172.21.15.46:0/371537776 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555898cdb300 0x555898cdd6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:39.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.293+0000 7fe862ffd640 1 -- 172.21.15.46:0/371537776 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555898a0d7e0 con 0x555898ccdd80 2024-03-31T23:12:39.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.293+0000 7fe8627fc640 1 --2- 172.21.15.46:0/371537776 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555898cdb300 0x555898cdd6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:12:39.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.293+0000 7fe862ffd640 1 --2- 172.21.15.46:0/371537776 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555898ccdd80 0x555898cdadc0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fe854036040 tx=0x7fe854009670 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2bb5b8369fcbd363 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:12:39.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.293+0000 7fe8637fe640 1 --2- 172.21.15.46:0/464349197 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fe84c0a2730 0x7fe84c0a4bf0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fe85c005fd0 tx=0x7fe85c004ab0 comp rx=0 tx=0).ready entity=mgr.15654 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:12:39.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.293+0000 7fe849ffb640 1 -- 172.21.15.46:0/371537776 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe854045070 con 0x555898ccdd80 2024-03-31T23:12:39.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.293+0000 7fe849ffb640 1 -- 172.21.15.46:0/371537776 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe85402c7e0 con 0x555898ccdd80 2024-03-31T23:12:39.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.293+0000 7fe849ffb640 1 -- 172.21.15.46:0/371537776 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe854003b20 con 0x555898ccdd80 2024-03-31T23:12:39.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.294+0000 7fe86c5f6740 1 -- 172.21.15.46:0/371537776 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555898ccdd80 msgr2=0x555898cdadc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:12:39.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.294+0000 7fe86c5f6740 1 --2- 172.21.15.46:0/371537776 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555898ccdd80 0x555898cdadc0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fe854036040 tx=0x7fe854009670 comp rx=0 tx=0).stop 2024-03-31T23:12:39.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.294+0000 7fe86c5f6740 1 -- 172.21.15.46:0/371537776 shutdown_connections 2024-03-31T23:12:39.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.294+0000 7fe86c5f6740 1 --2- 172.21.15.46:0/371537776 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555898cdb300 0x555898cdd6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:39.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.294+0000 7fe86c5f6740 1 --2- 172.21.15.46:0/371537776 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555898ccdd80 0x555898cdadc0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:39.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.294+0000 7fe86c5f6740 1 --2- 172.21.15.46:0/371537776 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x555898ccd3d0 0x555898ccd7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:39.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.294+0000 7fe86c5f6740 1 -- 172.21.15.46:0/371537776 >> 172.21.15.46:0/371537776 conn(0x555898cc8120 msgr2=0x555898cc8500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:12:39.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.294+0000 7fe86c5f6740 1 -- 172.21.15.46:0/371537776 shutdown_connections 2024-03-31T23:12:39.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.294+0000 7fe86c5f6740 1 -- 172.21.15.46:0/371537776 wait complete. 2024-03-31T23:12:39.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.294+0000 7fe86c5f6740 1 -- start start 2024-03-31T23:12:39.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.295+0000 7fe86c5f6740 1 Processor -- start 2024-03-31T23:12:39.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.295+0000 7fe86c5f6740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555898ccd3d0 0x555898cdaab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:39.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.295+0000 7fe86c5f6740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555898ccdd80 0x555898cd3e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:39.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.295+0000 7fe86c5f6740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x555898cdb300 0x555898cd43b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:39.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.295+0000 7fe862ffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555898ccd3d0 0x555898cdaab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:39.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.295+0000 7fe862ffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555898ccd3d0 0x555898cdaab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:32900/0 (socket says 172.21.15.46:32900) 2024-03-31T23:12:39.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.295+0000 7fe862ffd640 1 -- 172.21.15.46:0/118197565 learned_addr learned my addr 172.21.15.46:0/118197565 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:12:39.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.295+0000 7fe86c5f6740 1 -- 172.21.15.46:0/118197565 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x555898bcf3d0 con 0x555898ccd3d0 2024-03-31T23:12:39.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.295+0000 7fe86c5f6740 1 -- 172.21.15.46:0/118197565 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x555898c2cf40 con 0x555898cdb300 2024-03-31T23:12:39.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.295+0000 7fe8637fe640 1 --2- 172.21.15.46:0/118197565 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x555898cdb300 0x555898cd43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:39.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.295+0000 7fe8627fc640 1 --2- 172.21.15.46:0/118197565 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555898ccdd80 0x555898cd3e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:39.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.295+0000 7fe86c5f6740 1 -- 172.21.15.46:0/118197565 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x555898cde060 con 0x555898ccdd80 2024-03-31T23:12:39.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.296+0000 7fe86c5f6740 10 client.0 _ll_register_callbacks cb 0x555898becb70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T23:12:39.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.296+0000 7fe862ffd640 1 -- 172.21.15.46:0/118197565 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555898ccdd80 msgr2=0x555898cd3e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:12:39.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.296+0000 7fe862ffd640 1 --2- 172.21.15.46:0/118197565 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555898ccdd80 0x555898cd3e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:39.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.296+0000 7fe862ffd640 1 -- 172.21.15.46:0/118197565 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x555898cdb300 msgr2=0x555898cd43b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:12:39.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.296+0000 7fe862ffd640 1 --2- 172.21.15.46:0/118197565 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x555898cdb300 0x555898cd43b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:39.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.296+0000 7fe862ffd640 1 -- 172.21.15.46:0/118197565 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555898a6df70 con 0x555898ccd3d0 2024-03-31T23:12:39.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.296+0000 7fe8637fe640 1 --2- 172.21.15.46:0/118197565 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x555898cdb300 0x555898cd43b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:12:39.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.296+0000 7fe8627fc640 1 --2- 172.21.15.46:0/118197565 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555898ccdd80 0x555898cd3e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:12:39.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.296+0000 7fe862ffd640 1 --2- 172.21.15.46:0/118197565 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555898ccd3d0 0x555898cdaab0 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7fe85403f670 tx=0x7fe85403f6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d1b645dbe4732d32 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:12:39.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.296+0000 7fe842ffd640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-31T23:12:39.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.296+0000 7fe842ffd640 1 -- 172.21.15.46:0/118197565 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe85404e070 con 0x555898ccd3d0 2024-03-31T23:12:39.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.296+0000 7fe842ffd640 1 -- 172.21.15.46:0/118197565 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7fe85403fcd0 con 0x555898ccd3d0 2024-03-31T23:12:39.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.296+0000 7fe86c5f6740 1 -- 172.21.15.46:0/118197565 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe85402c7e0 con 0x555898ccd3d0 2024-03-31T23:12:39.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.297+0000 7fe842ffd640 1 -- 172.21.15.46:0/118197565 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fe85405a050 con 0x555898ccd3d0 2024-03-31T23:12:39.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.297+0000 7fe86c5f6740 10 client.16011 fetch_fsmap learned FSMap version 25 2024-03-31T23:12:39.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.297+0000 7fe86c5f6740 1 -- 172.21.15.46:0/118197565 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x555898905170 con 0x555898ccd3d0 2024-03-31T23:12:39.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.297+0000 7fe842ffd640 1 -- 172.21.15.46:0/118197565 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe85402c7e0 con 0x555898ccd3d0 2024-03-31T23:12:39.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.297+0000 7fe842ffd640 1 -- 172.21.15.46:0/118197565 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7fe854034440 con 0x555898ccd3d0 2024-03-31T23:12:39.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.297+0000 7fe86c5f6740 10 client.16011 fetch_fsmap finished waiting for FSMap version 25 2024-03-31T23:12:39.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.297+0000 7fe86c5f6740 10 client.16011 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-31T23:12:39.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.297+0000 7fe86c5f6740 20 client.16011 populate_metadata read hostname 'smithi046' 2024-03-31T23:12:39.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.297+0000 7fe86c5f6740 1 --2- 172.21.15.46:0/118197565 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x555898cef370 0x555898cf1760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:39.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.297+0000 7fe86c5f6740 4 client.16011 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-31T23:12:39.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.297+0000 7fe86c5f6740 1 -- 172.21.15.46:0/118197565 --> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555898c3e3e0 con 0x555898cef370 2024-03-31T23:12:39.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.298+0000 7fe8627fc640 1 --2- 172.21.15.46:0/118197565 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x555898cef370 0x555898cf1760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:39.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.298+0000 7fe8627fc640 1 --2- 172.21.15.46:0/118197565 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x555898cef370 0x555898cf1760 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1c267595d14a1de3 server_cookie=68176f6f14cdb61a in_seq=0 out_seq=0 2024-03-31T23:12:39.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.298+0000 7fe842ffd640 10 client.16011 ms_handle_connect on v2:172.21.15.142:6838/2519143918 2024-03-31T23:12:39.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.298+0000 7fe842ffd640 1 -- 172.21.15.46:0/118197565 <== mds.0 v2:172.21.15.142:6838/2519143918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x555898c3e3e0 con 0x555898cef370 2024-03-31T23:12:39.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.298+0000 7fe842ffd640 10 client.16011 handle_command_reply: tid=0 2024-03-31T23:12:39.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.299+0000 7fe86c5f6740 1 client.16011 shutdown 2024-03-31T23:12:39.306 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.299+0000 7fe86c5f6740 1 -- 172.21.15.46:0/118197565 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555898ccd3d0 msgr2=0x555898cdaab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:12:39.306 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.299+0000 7fe86c5f6740 1 --2- 172.21.15.46:0/118197565 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555898ccd3d0 0x555898cdaab0 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7fe85403f670 tx=0x7fe85403f6a0 comp rx=0 tx=0).stop 2024-03-31T23:12:39.306 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.299+0000 7fe86c5f6740 1 -- 172.21.15.46:0/118197565 shutdown_connections 2024-03-31T23:12:39.306 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.299+0000 7fe86c5f6740 1 --2- 172.21.15.46:0/118197565 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x555898cef370 0x555898cf1760 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:39.306 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.299+0000 7fe86c5f6740 1 --2- 172.21.15.46:0/118197565 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x555898cdb300 0x555898cd43b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:39.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.299+0000 7fe86c5f6740 1 --2- 172.21.15.46:0/118197565 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555898ccdd80 0x555898cd3e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:39.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.299+0000 7fe86c5f6740 1 --2- 172.21.15.46:0/118197565 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555898ccd3d0 0x555898cdaab0 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:39.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.299+0000 7fe86c5f6740 1 -- 172.21.15.46:0/118197565 >> 172.21.15.46:0/118197565 conn(0x555898cc8120 msgr2=0x555898cc9e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:12:39.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.299+0000 7fe842ffd640 0 client.16011 ms_handle_reset on v2:172.21.15.142:6838/2519143918 2024-03-31T23:12:39.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.299+0000 7fe86c5f6740 1 -- 172.21.15.46:0/118197565 shutdown_connections 2024-03-31T23:12:39.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.299+0000 7fe86c5f6740 1 -- 172.21.15.46:0/118197565 wait complete. 2024-03-31T23:12:39.308 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.299+0000 7fe86c5f6740 20 client.16011 trim_cache size 0 max 16384 2024-03-31T23:12:39.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.317+0000 7fe86c5f6740 1 Processor -- start 2024-03-31T23:12:39.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.317+0000 7fe86c5f6740 1 -- start start 2024-03-31T23:12:39.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.317+0000 7fe86c5f6740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555898ccd3d0 0x555898cda650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:39.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.317+0000 7fe86c5f6740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555898ccdd80 0x555898cd7cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:39.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.317+0000 7fe86c5f6740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x555898cdb300 0x555898cd81f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:39.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.317+0000 7fe86c5f6740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fe8641af270 con 0x555898ccdd80 2024-03-31T23:12:39.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.317+0000 7fe86c5f6740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fe8641acb40 con 0x555898cdb300 2024-03-31T23:12:39.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.317+0000 7fe86c5f6740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fe8641aa990 con 0x555898ccd3d0 2024-03-31T23:12:39.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.317+0000 7fe8627fc640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555898ccdd80 0x555898cd7cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:39.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.317+0000 7fe8627fc640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555898ccdd80 0x555898cd7cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:32916/0 (socket says 172.21.15.46:32916) 2024-03-31T23:12:39.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.317+0000 7fe8627fc640 1 -- 172.21.15.46:0/1301415631 learned_addr learned my addr 172.21.15.46:0/1301415631 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:12:39.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.317+0000 7fe8637fe640 1 --2- 172.21.15.46:0/1301415631 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x555898cdb300 0x555898cd81f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:39.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.317+0000 7fe862ffd640 1 --2- 172.21.15.46:0/1301415631 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555898ccd3d0 0x555898cda650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:39.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.317+0000 7fe8627fc640 1 -- 172.21.15.46:0/1301415631 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555898ccd3d0 msgr2=0x555898cda650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:12:39.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.317+0000 7fe8627fc640 1 --2- 172.21.15.46:0/1301415631 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555898ccd3d0 0x555898cda650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:39.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.317+0000 7fe8627fc640 1 -- 172.21.15.46:0/1301415631 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x555898cdb300 msgr2=0x555898cd81f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:12:39.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.317+0000 7fe8627fc640 1 --2- 172.21.15.46:0/1301415631 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x555898cdb300 0x555898cd81f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:39.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.317+0000 7fe8627fc640 1 -- 172.21.15.46:0/1301415631 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555898cd2e20 con 0x555898ccdd80 2024-03-31T23:12:39.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.317+0000 7fe8637fe640 1 --2- 172.21.15.46:0/1301415631 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x555898cdb300 0x555898cd81f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:12:39.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.318+0000 7fe8627fc640 1 --2- 172.21.15.46:0/1301415631 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555898ccdd80 0x555898cd7cb0 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7fe85800cf10 tx=0x7fe8580d1a80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7bdef2346464a7fb server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:12:39.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.318+0000 7fe8497fa640 1 -- 172.21.15.46:0/1301415631 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe85809b6f0 con 0x555898ccdd80 2024-03-31T23:12:39.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.318+0000 7fe8497fa640 1 -- 172.21.15.46:0/1301415631 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe8580a4100 con 0x555898ccdd80 2024-03-31T23:12:39.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.318+0000 7fe8497fa640 1 -- 172.21.15.46:0/1301415631 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8580d2bd0 con 0x555898ccdd80 2024-03-31T23:12:39.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.319+0000 7fe86c5f6740 1 -- 172.21.15.46:0/1301415631 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555898ccdd80 msgr2=0x555898cd7cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:12:39.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.319+0000 7fe86c5f6740 1 --2- 172.21.15.46:0/1301415631 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555898ccdd80 0x555898cd7cb0 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7fe85800cf10 tx=0x7fe8580d1a80 comp rx=0 tx=0).stop 2024-03-31T23:12:39.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.319+0000 7fe86c5f6740 1 -- 172.21.15.46:0/1301415631 shutdown_connections 2024-03-31T23:12:39.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.319+0000 7fe86c5f6740 1 --2- 172.21.15.46:0/1301415631 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x555898cdb300 0x555898cd81f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:39.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.319+0000 7fe86c5f6740 1 --2- 172.21.15.46:0/1301415631 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555898ccdd80 0x555898cd7cb0 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:39.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.319+0000 7fe86c5f6740 1 --2- 172.21.15.46:0/1301415631 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555898ccd3d0 0x555898cda650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:39.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.319+0000 7fe86c5f6740 1 -- 172.21.15.46:0/1301415631 >> 172.21.15.46:0/1301415631 conn(0x555898cc8120 msgr2=0x555898cd75a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:12:39.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.319+0000 7fe86c5f6740 1 -- 172.21.15.46:0/1301415631 shutdown_connections 2024-03-31T23:12:39.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.319+0000 7fe86c5f6740 1 -- 172.21.15.46:0/1301415631 wait complete. 2024-03-31T23:12:39.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.319+0000 7fe86c5f6740 1 -- start start 2024-03-31T23:12:39.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.319+0000 7fe86c5f6740 1 Processor -- start 2024-03-31T23:12:39.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.320+0000 7fe86c5f6740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555898ccd3d0 0x555898ce9cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:39.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.320+0000 7fe86c5f6740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x555898ccdd80 0x555898cea210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:39.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.320+0000 7fe86c5f6740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555898cdb300 0x555898cea750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:39.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.320+0000 7fe86c5f6740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fe8641ab000 con 0x555898cdb300 2024-03-31T23:12:39.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.320+0000 7fe86c5f6740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fe8641ac470 con 0x555898ccdd80 2024-03-31T23:12:39.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.320+0000 7fe86c5f6740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fe8641ab6d0 con 0x555898ccd3d0 2024-03-31T23:12:39.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.320+0000 7fe86c5f6740 10 client.0 _ll_register_callbacks cb 0x7fe854009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T23:12:39.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.320+0000 7fe8637fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555898cdb300 0x555898cea750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:39.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.320+0000 7fe8637fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555898cdb300 0x555898cea750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:32926/0 (socket says 172.21.15.46:32926) 2024-03-31T23:12:39.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.320+0000 7fe8637fe640 1 -- 172.21.15.46:0/3420797490 learned_addr learned my addr 172.21.15.46:0/3420797490 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:12:39.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.320+0000 7fe8627fc640 1 --2- 172.21.15.46:0/3420797490 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x555898ccdd80 0x555898cea210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:39.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.320+0000 7fe862ffd640 1 --2- 172.21.15.46:0/3420797490 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555898ccd3d0 0x555898ce9cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:39.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.320+0000 7fe8637fe640 1 -- 172.21.15.46:0/3420797490 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555898ccd3d0 msgr2=0x555898ce9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:12:39.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.320+0000 7fe8637fe640 1 --2- 172.21.15.46:0/3420797490 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555898ccd3d0 0x555898ce9cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:39.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.320+0000 7fe8637fe640 1 -- 172.21.15.46:0/3420797490 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x555898ccdd80 msgr2=0x555898cea210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:12:39.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.320+0000 7fe8637fe640 1 --2- 172.21.15.46:0/3420797490 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x555898ccdd80 0x555898cea210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:39.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.320+0000 7fe8637fe640 1 -- 172.21.15.46:0/3420797490 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555898d54510 con 0x555898cdb300 2024-03-31T23:12:39.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.320+0000 7fe8627fc640 1 --2- 172.21.15.46:0/3420797490 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x555898ccdd80 0x555898cea210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:12:39.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.320+0000 7fe8637fe640 1 --2- 172.21.15.46:0/3420797490 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555898cdb300 0x555898cea750 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7fe85c012fb0 tx=0x7fe85c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d990005613cf8a83 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:12:39.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.320+0000 7fe842ffd640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-31T23:12:39.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.321+0000 7fe842ffd640 1 -- 172.21.15.46:0/3420797490 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe85c004b20 con 0x555898cdb300 2024-03-31T23:12:39.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.321+0000 7fe842ffd640 1 -- 172.21.15.46:0/3420797490 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7fe85c004cc0 con 0x555898cdb300 2024-03-31T23:12:39.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.321+0000 7fe86c5f6740 1 -- 172.21.15.46:0/3420797490 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe8580a4100 con 0x555898cdb300 2024-03-31T23:12:39.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.321+0000 7fe842ffd640 1 -- 172.21.15.46:0/3420797490 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fe85c005020 con 0x555898cdb300 2024-03-31T23:12:39.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.321+0000 7fe86c5f6740 10 client.16017 fetch_fsmap learned FSMap version 25 2024-03-31T23:12:39.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.321+0000 7fe86c5f6740 1 -- 172.21.15.46:0/3420797490 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x555898d55810 con 0x555898cdb300 2024-03-31T23:12:39.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.321+0000 7fe842ffd640 1 -- 172.21.15.46:0/3420797490 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe8580a4100 con 0x555898cdb300 2024-03-31T23:12:39.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.321+0000 7fe842ffd640 1 -- 172.21.15.46:0/3420797490 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7fe85c01e050 con 0x555898cdb300 2024-03-31T23:12:39.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.321+0000 7fe86c5f6740 10 client.16017 fetch_fsmap finished waiting for FSMap version 25 2024-03-31T23:12:39.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.321+0000 7fe86c5f6740 10 client.16017 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-31T23:12:39.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.321+0000 7fe86c5f6740 20 client.16017 populate_metadata read hostname 'smithi046' 2024-03-31T23:12:39.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.321+0000 7fe86c5f6740 1 --2- 172.21.15.46:0/3420797490 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x555898d55f90 0x555898d583b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:39.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.321+0000 7fe86c5f6740 4 client.16017 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-31T23:12:39.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.321+0000 7fe86c5f6740 1 -- 172.21.15.46:0/3420797490 --> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x555898c38730 con 0x555898d55f90 2024-03-31T23:12:39.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.322+0000 7fe862ffd640 1 --2- 172.21.15.46:0/3420797490 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x555898d55f90 0x555898d583b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:39.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.322+0000 7fe842ffd640 10 client.16017 ms_handle_connect on v2:172.21.15.142:6838/2519143918 2024-03-31T23:12:39.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.322+0000 7fe862ffd640 1 --2- 172.21.15.46:0/3420797490 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x555898d55f90 0x555898d583b0 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=92907d49cde37746 server_cookie=83872ad830013ce5 in_seq=0 out_seq=0 2024-03-31T23:12:39.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.323+0000 7fe842ffd640 1 -- 172.21.15.46:0/3420797490 <== mds.0 v2:172.21.15.142:6838/2519143918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x555898c38730 con 0x555898d55f90 2024-03-31T23:12:39.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.323+0000 7fe842ffd640 10 client.16017 handle_command_reply: tid=0 2024-03-31T23:12:39.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.323+0000 7fe86c5f6740 1 client.16017 shutdown 2024-03-31T23:12:39.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.323+0000 7fe86c5f6740 1 -- 172.21.15.46:0/3420797490 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555898cdb300 msgr2=0x555898cea750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:12:39.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.323+0000 7fe86c5f6740 1 --2- 172.21.15.46:0/3420797490 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555898cdb300 0x555898cea750 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7fe85c012fb0 tx=0x7fe85c004a40 comp rx=0 tx=0).stop 2024-03-31T23:12:39.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.324+0000 7fe86c5f6740 1 -- 172.21.15.46:0/3420797490 shutdown_connections 2024-03-31T23:12:39.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.324+0000 7fe86c5f6740 1 --2- 172.21.15.46:0/3420797490 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x555898d55f90 0x555898d583b0 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:39.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.324+0000 7fe86c5f6740 1 --2- 172.21.15.46:0/3420797490 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555898cdb300 0x555898cea750 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:39.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.324+0000 7fe86c5f6740 1 --2- 172.21.15.46:0/3420797490 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x555898ccdd80 0x555898cea210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:39.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.324+0000 7fe86c5f6740 1 --2- 172.21.15.46:0/3420797490 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555898ccd3d0 0x555898ce9cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:39.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.324+0000 7fe86c5f6740 1 -- 172.21.15.46:0/3420797490 >> 172.21.15.46:0/3420797490 conn(0x555898cc8120 msgr2=0x555898cc99b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:12:39.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.324+0000 7fe842ffd640 0 client.16017 ms_handle_reset on v2:172.21.15.142:6838/2519143918 2024-03-31T23:12:39.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.324+0000 7fe86c5f6740 1 -- 172.21.15.46:0/3420797490 shutdown_connections 2024-03-31T23:12:39.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.324+0000 7fe86c5f6740 1 -- 172.21.15.46:0/3420797490 wait complete. 2024-03-31T23:12:39.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.324+0000 7fe86c5f6740 20 client.16017 trim_cache size 0 max 16384 2024-03-31T23:12:39.329 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-03-31T23:12:39.330 INFO:teuthology.orchestra.run.smithi046.stdout: "return_code": 0, 2024-03-31T23:12:39.330 INFO:teuthology.orchestra.run.smithi046.stdout: "scrub_tag": "cbbde91d-d625-44c3-b4c4-d3bec5ed0c77", 2024-03-31T23:12:39.330 INFO:teuthology.orchestra.run.smithi046.stdout: "mode": "asynchronous" 2024-03-31T23:12:39.330 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-03-31T23:12:39.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.324+0000 7fe86993e640 1 -- 172.21.15.46:0/464349197 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fe84c0a2730 msgr2=0x7fe84c0a4bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:12:39.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.324+0000 7fe86993e640 1 --2- 172.21.15.46:0/464349197 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fe84c0a2730 0x7fe84c0a4bf0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fe85c005fd0 tx=0x7fe85c004ab0 comp rx=0 tx=0).stop 2024-03-31T23:12:39.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.324+0000 7fe86993e640 1 -- 172.21.15.46:0/464349197 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe8641005b0 msgr2=0x7fe864198470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:12:39.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.324+0000 7fe86993e640 1 --2- 172.21.15.46:0/464349197 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe8641005b0 0x7fe864198470 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7fe85800e9f0 tx=0x7fe85800eeb0 comp rx=0 tx=0).stop 2024-03-31T23:12:39.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.325+0000 7fe86993e640 1 -- 172.21.15.46:0/464349197 shutdown_connections 2024-03-31T23:12:39.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.325+0000 7fe86993e640 1 --2- 172.21.15.46:0/464349197 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fe84c0a2730 0x7fe84c0a4bf0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:39.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.325+0000 7fe86993e640 1 --2- 172.21.15.46:0/464349197 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe8641076f0 0x7fe86419c7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:39.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.325+0000 7fe86993e640 1 --2- 172.21.15.46:0/464349197 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe8641005b0 0x7fe864198470 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:39.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.325+0000 7fe86993e640 1 --2- 172.21.15.46:0/464349197 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe8640ffbf0 0x7fe864197f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:39.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.325+0000 7fe86993e640 1 -- 172.21.15.46:0/464349197 >> 172.21.15.46:0/464349197 conn(0x7fe864076b70 msgr2=0x7fe8640fe440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:12:39.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.325+0000 7fe86993e640 1 -- 172.21.15.46:0/464349197 shutdown_connections 2024-03-31T23:12:39.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.325+0000 7fe86993e640 1 -- 172.21.15.46:0/464349197 wait complete. 2024-03-31T23:12:39.340 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-31T23:12:39.341 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-31T23:12:39.449 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.448+0000 7f4034440640 1 -- 172.21.15.46:0/3073097485 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f402c1016e0 msgr2=0x7f402c101b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:12:39.449 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.448+0000 7f4034440640 1 --2- 172.21.15.46:0/3073097485 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f402c1016e0 0x7f402c101b60 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f401c009930 tx=0x7f401c02bc50 comp rx=0 tx=0).stop 2024-03-31T23:12:39.450 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.448+0000 7f4034440640 1 -- 172.21.15.46:0/3073097485 shutdown_connections 2024-03-31T23:12:39.450 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.448+0000 7f4034440640 1 --2- 172.21.15.46:0/3073097485 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f402c1020a0 0x7f402c10e5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:39.450 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.448+0000 7f4034440640 1 --2- 172.21.15.46:0/3073097485 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f402c1016e0 0x7f402c101b60 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:39.450 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.448+0000 7f4034440640 1 --2- 172.21.15.46:0/3073097485 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f402c1076f0 0x7f402c107ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:39.450 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.448+0000 7f4034440640 1 -- 172.21.15.46:0/3073097485 >> 172.21.15.46:0/3073097485 conn(0x7f402c0fd1f0 msgr2=0x7f402c0ff610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:12:39.450 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.448+0000 7f4034440640 1 -- 172.21.15.46:0/3073097485 shutdown_connections 2024-03-31T23:12:39.450 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.448+0000 7f4034440640 1 -- 172.21.15.46:0/3073097485 wait complete. 2024-03-31T23:12:39.452 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.449+0000 7f4034440640 1 Processor -- start 2024-03-31T23:12:39.452 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.449+0000 7f4034440640 1 -- start start 2024-03-31T23:12:39.452 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.449+0000 7f4034440640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f402c1016e0 0x7f402c197f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:39.452 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.449+0000 7f4034440640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f402c1020a0 0x7f402c1984c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:39.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.449+0000 7f4034440640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f402c1076f0 0x7f402c19c840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:39.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.449+0000 7f4034440640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f402c1119b0 con 0x7f402c1076f0 2024-03-31T23:12:39.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.449+0000 7f4034440640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f402c111830 con 0x7f402c1016e0 2024-03-31T23:12:39.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.449+0000 7f4034440640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f402c111b30 con 0x7f402c1020a0 2024-03-31T23:12:39.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.449+0000 7f40321dc640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f402c1016e0 0x7f402c197f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:39.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.449+0000 7f40319db640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f402c1020a0 0x7f402c1984c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:39.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.449+0000 7f40321dc640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f402c1016e0 0x7f402c197f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.46:52020/0 (socket says 172.21.15.46:52020) 2024-03-31T23:12:39.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.450+0000 7f40321dc640 1 -- 172.21.15.46:0/4248823771 learned_addr learned my addr 172.21.15.46:0/4248823771 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:12:39.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.450+0000 7f40329dd640 1 --2- 172.21.15.46:0/4248823771 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f402c1076f0 0x7f402c19c840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:39.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.450+0000 7f40321dc640 1 -- 172.21.15.46:0/4248823771 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f402c1020a0 msgr2=0x7f402c1984c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:12:39.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.450+0000 7f40321dc640 1 --2- 172.21.15.46:0/4248823771 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f402c1020a0 0x7f402c1984c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:39.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.450+0000 7f40321dc640 1 -- 172.21.15.46:0/4248823771 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f402c1076f0 msgr2=0x7f402c19c840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:12:39.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.450+0000 7f40321dc640 1 --2- 172.21.15.46:0/4248823771 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f402c1076f0 0x7f402c19c840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:39.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.450+0000 7f40321dc640 1 -- 172.21.15.46:0/4248823771 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f402c19cf70 con 0x7f402c1016e0 2024-03-31T23:12:39.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.450+0000 7f40321dc640 1 --2- 172.21.15.46:0/4248823771 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f402c1016e0 0x7f402c197f80 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f402800e9f0 tx=0x7f402800eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:12:39.454 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.450+0000 7f40237fe640 1 -- 172.21.15.46:0/4248823771 <== mon.2 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f402800cd80 con 0x7f402c1016e0 2024-03-31T23:12:39.454 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.450+0000 7f40237fe640 1 -- 172.21.15.46:0/4248823771 <== mon.2 v2:172.21.15.142:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f40280108e0 con 0x7f402c1016e0 2024-03-31T23:12:39.454 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.450+0000 7f40237fe640 1 -- 172.21.15.46:0/4248823771 <== mon.2 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4028018920 con 0x7f402c1016e0 2024-03-31T23:12:39.454 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.450+0000 7f4034440640 1 -- 172.21.15.46:0/4248823771 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f402c19d260 con 0x7f402c1016e0 2024-03-31T23:12:39.454 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.451+0000 7f4034440640 1 -- 172.21.15.46:0/4248823771 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f402c1a4af0 con 0x7f402c1016e0 2024-03-31T23:12:39.455 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.453+0000 7f403701e740 1 Processor -- start 2024-03-31T23:12:39.455 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.453+0000 7f403701e740 1 -- start start 2024-03-31T23:12:39.455 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.454+0000 7f403701e740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55937a66b290 0x55937a66b670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:39.455 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.454+0000 7f403701e740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55937a66bbb0 0x55937a674250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:39.455 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.454+0000 7f403701e740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55937a674790 0x55937a676b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:39.455 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.454+0000 7f403701e740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55937a5cf180 con 0x55937a674790 2024-03-31T23:12:39.455 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.454+0000 7f40237fe640 1 -- 172.21.15.46:0/4248823771 <== mon.2 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 24) ==== 137553+0+0 (secure 0 0 0) 0x7f4028010a90 con 0x7f402c1016e0 2024-03-31T23:12:39.456 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.454+0000 7f403701e740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55937a577b70 con 0x55937a66bbb0 2024-03-31T23:12:39.456 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.454+0000 7f403701e740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55937a5cb220 con 0x55937a66b290 2024-03-31T23:12:39.456 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.454+0000 7f40329dd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55937a66b290 0x55937a66b670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:39.456 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.454+0000 7f40329dd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55937a66b290 0x55937a66b670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.46:45668/0 (socket says 172.21.15.46:45668) 2024-03-31T23:12:39.456 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.454+0000 7f40329dd640 1 -- 172.21.15.46:0/1428638027 learned_addr learned my addr 172.21.15.46:0/1428638027 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:12:39.456 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.454+0000 7f40319db640 1 --2- 172.21.15.46:0/1428638027 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55937a674790 0x55937a676b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:39.456 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.454+0000 7f40329dd640 1 -- 172.21.15.46:0/1428638027 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55937a66bbb0 msgr2=0x55937a674250 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-31T23:12:39.456 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.454+0000 7f40329dd640 1 --2- 172.21.15.46:0/1428638027 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55937a66bbb0 0x55937a674250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:39.456 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.454+0000 7f40329dd640 1 -- 172.21.15.46:0/1428638027 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55937a674790 msgr2=0x55937a676b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:12:39.456 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.454+0000 7f40329dd640 1 --2- 172.21.15.46:0/1428638027 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55937a674790 0x55937a676b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:39.456 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.454+0000 7f40329dd640 1 -- 172.21.15.46:0/1428638027 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55937a58b7c0 con 0x55937a66b290 2024-03-31T23:12:39.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.455+0000 7f40329dd640 1 --2- 172.21.15.46:0/1428638027 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55937a66b290 0x55937a66b670 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f4024009fd0 tx=0x7f40240118d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4c5c8bca6e4a5cf6 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:12:39.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.455+0000 7f4020bfe640 1 -- 172.21.15.46:0/1428638027 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4024004d20 con 0x55937a66b290 2024-03-31T23:12:39.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.455+0000 7f40237fe640 1 --2- 172.21.15.46:0/4248823771 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f40180a2730 0x7f40180a4bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:39.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.455+0000 7f4020bfe640 1 -- 172.21.15.46:0/1428638027 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4024004050 con 0x55937a66b290 2024-03-31T23:12:39.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.455+0000 7f40321dc640 1 --2- 172.21.15.46:0/4248823771 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f40180a2730 0x7f40180a4bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:39.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.455+0000 7f40237fe640 1 -- 172.21.15.46:0/4248823771 <== mon.2 v2:172.21.15.142:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7f40280cf400 con 0x7f402c1016e0 2024-03-31T23:12:39.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.456+0000 7f403701e740 1 -- 172.21.15.46:0/1428638027 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55937a66b290 msgr2=0x55937a66b670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:12:39.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.456+0000 7f40321dc640 1 --2- 172.21.15.46:0/4248823771 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f40180a2730 0x7f40180a4bf0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f40280cc680 tx=0x7f402806d6b0 comp rx=0 tx=0).ready entity=mgr.15654 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:12:39.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.456+0000 7f403701e740 1 --2- 172.21.15.46:0/1428638027 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55937a66b290 0x55937a66b670 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f4024009fd0 tx=0x7f40240118d0 comp rx=0 tx=0).stop 2024-03-31T23:12:39.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.456+0000 7f403701e740 1 -- 172.21.15.46:0/1428638027 shutdown_connections 2024-03-31T23:12:39.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.456+0000 7f403701e740 1 --2- 172.21.15.46:0/1428638027 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55937a674790 0x55937a676b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:39.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.456+0000 7f403701e740 1 --2- 172.21.15.46:0/1428638027 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55937a66bbb0 0x55937a674250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:39.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.456+0000 7f403701e740 1 --2- 172.21.15.46:0/1428638027 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55937a66b290 0x55937a66b670 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:39.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.456+0000 7f403701e740 1 -- 172.21.15.46:0/1428638027 >> 172.21.15.46:0/1428638027 conn(0x55937a6659c0 msgr2=0x55937a665da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:12:39.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.456+0000 7f403701e740 1 -- 172.21.15.46:0/1428638027 shutdown_connections 2024-03-31T23:12:39.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.456+0000 7f403701e740 1 -- 172.21.15.46:0/1428638027 wait complete. 2024-03-31T23:12:39.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.457+0000 7f403701e740 1 -- start start 2024-03-31T23:12:39.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.457+0000 7f403701e740 1 Processor -- start 2024-03-31T23:12:39.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.457+0000 7f403701e740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55937a66b290 0x55937a6716f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:39.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.457+0000 7f403701e740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55937a66bbb0 0x55937a671c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:39.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.457+0000 7f403701e740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55937a674790 0x55937a672170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:39.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.457+0000 7f403701e740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55937a677620 con 0x55937a674790 2024-03-31T23:12:39.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.457+0000 7f403701e740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55937a56d4a0 con 0x55937a66bbb0 2024-03-31T23:12:39.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.457+0000 7f403701e740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55937a6777a0 con 0x55937a66b290 2024-03-31T23:12:39.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.457+0000 7f40319db640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55937a66bbb0 0x55937a671c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:39.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.457+0000 7f40319db640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55937a66bbb0 0x55937a671c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.46:52022/0 (socket says 172.21.15.46:52022) 2024-03-31T23:12:39.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.457+0000 7f40319db640 1 -- 172.21.15.46:0/3109600771 learned_addr learned my addr 172.21.15.46:0/3109600771 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:12:39.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.457+0000 7f40329dd640 1 --2- 172.21.15.46:0/3109600771 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55937a66b290 0x55937a6716f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:39.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.458+0000 7f40329dd640 1 --2- 172.21.15.46:0/3109600771 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55937a674790 0x55937a672170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:39.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.458+0000 7f40319db640 1 -- 172.21.15.46:0/3109600771 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55937a66b290 msgr2=0x55937a6716f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:12:39.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.458+0000 7f40319db640 1 --2- 172.21.15.46:0/3109600771 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55937a66b290 0x55937a6716f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:39.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.458+0000 7f40319db640 1 -- 172.21.15.46:0/3109600771 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55937a674790 msgr2=0x55937a672170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:12:39.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.458+0000 7f40319db640 1 --2- 172.21.15.46:0/3109600771 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55937a674790 0x55937a672170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:39.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.458+0000 7f40319db640 1 -- 172.21.15.46:0/3109600771 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55937a3ad7d0 con 0x55937a66bbb0 2024-03-31T23:12:39.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.458+0000 7f40329dd640 1 --2- 172.21.15.46:0/3109600771 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55937a674790 0x55937a672170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T23:12:39.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.458+0000 7f403701e740 10 client.0 _ll_register_callbacks cb 0x55937a570d90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T23:12:39.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.458+0000 7f40319db640 1 --2- 172.21.15.46:0/3109600771 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55937a66bbb0 0x55937a671c30 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f401c035000 tx=0x7f401c02cd50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=baf2832223375c6d server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:12:39.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.458+0000 7f4015ffb640 10 client.0 ms_handle_connect on v2:172.21.15.142:3300/0 2024-03-31T23:12:39.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.458+0000 7f403701e740 1 -- 172.21.15.46:0/3109600771 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4024004050 con 0x55937a66bbb0 2024-03-31T23:12:39.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.458+0000 7f4015ffb640 1 -- 172.21.15.46:0/3109600771 <== mon.2 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f401c04d060 con 0x55937a66bbb0 2024-03-31T23:12:39.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.458+0000 7f4015ffb640 1 -- 172.21.15.46:0/3109600771 <== mon.2 v2:172.21.15.142:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7f401c004310 con 0x55937a66bbb0 2024-03-31T23:12:39.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.458+0000 7f4015ffb640 1 -- 172.21.15.46:0/3109600771 <== mon.2 v2:172.21.15.142:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f401c054050 con 0x55937a66bbb0 2024-03-31T23:12:39.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.458+0000 7f403701e740 10 client.25871 fetch_fsmap learned FSMap version 25 2024-03-31T23:12:39.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.459+0000 7f403701e740 1 -- 172.21.15.46:0/3109600771 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({fsmap=25}) -- 0x55937a2e33f0 con 0x55937a66bbb0 2024-03-31T23:12:39.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.459+0000 7f4015ffb640 1 -- 172.21.15.46:0/3109600771 <== mon.2 v2:172.21.15.142:3300/0 4 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f401c005030 con 0x55937a66bbb0 2024-03-31T23:12:39.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.459+0000 7f403701e740 10 client.25871 fetch_fsmap finished waiting for FSMap version 25 2024-03-31T23:12:39.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.459+0000 7f403701e740 10 client.25871 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-31T23:12:39.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.459+0000 7f403701e740 20 client.25871 populate_metadata read hostname 'smithi046' 2024-03-31T23:12:39.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.459+0000 7f403701e740 1 --2- 172.21.15.46:0/3109600771 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55937a684900 0x55937a686d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:39.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.459+0000 7f403701e740 4 client.25871 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-31T23:12:39.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.459+0000 7f403701e740 1 -- 172.21.15.46:0/3109600771 --> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55937a5dc6c0 con 0x55937a684900 2024-03-31T23:12:39.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.459+0000 7f40329dd640 1 --2- 172.21.15.46:0/3109600771 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55937a684900 0x55937a686d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:39.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.460+0000 7f40329dd640 1 --2- 172.21.15.46:0/3109600771 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55937a684900 0x55937a686d10 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=142629fd5a8638c server_cookie=64a96b0dcac289e6 in_seq=0 out_seq=0 2024-03-31T23:12:39.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.460+0000 7f4015ffb640 10 client.25871 ms_handle_connect on v2:172.21.15.142:6838/2519143918 2024-03-31T23:12:39.466 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.460+0000 7f4015ffb640 1 -- 172.21.15.46:0/3109600771 <== mds.0 v2:172.21.15.142:6838/2519143918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55937a5dc6c0 con 0x55937a684900 2024-03-31T23:12:39.466 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.460+0000 7f4015ffb640 10 client.25871 handle_command_reply: tid=0 2024-03-31T23:12:39.466 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.460+0000 7f403701e740 1 client.25871 shutdown 2024-03-31T23:12:39.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.461+0000 7f403701e740 1 -- 172.21.15.46:0/3109600771 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55937a66bbb0 msgr2=0x55937a671c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:12:39.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.461+0000 7f403701e740 1 --2- 172.21.15.46:0/3109600771 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55937a66bbb0 0x55937a671c30 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f401c035000 tx=0x7f401c02cd50 comp rx=0 tx=0).stop 2024-03-31T23:12:39.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.461+0000 7f403701e740 1 -- 172.21.15.46:0/3109600771 shutdown_connections 2024-03-31T23:12:39.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.461+0000 7f403701e740 1 --2- 172.21.15.46:0/3109600771 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55937a684900 0x55937a686d10 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:39.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.461+0000 7f4015ffb640 0 client.25871 ms_handle_reset on v2:172.21.15.142:6838/2519143918 2024-03-31T23:12:39.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.461+0000 7f403701e740 1 --2- 172.21.15.46:0/3109600771 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55937a674790 0x55937a672170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:39.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.461+0000 7f403701e740 1 --2- 172.21.15.46:0/3109600771 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55937a66bbb0 0x55937a671c30 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:39.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.461+0000 7f403701e740 1 --2- 172.21.15.46:0/3109600771 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55937a66b290 0x55937a6716f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:39.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.461+0000 7f403701e740 1 -- 172.21.15.46:0/3109600771 >> 172.21.15.46:0/3109600771 conn(0x55937a6659c0 msgr2=0x55937a664c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:12:39.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.461+0000 7f403701e740 1 -- 172.21.15.46:0/3109600771 shutdown_connections 2024-03-31T23:12:39.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.461+0000 7f403701e740 1 -- 172.21.15.46:0/3109600771 wait complete. 2024-03-31T23:12:39.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.462+0000 7f403701e740 20 client.25871 trim_cache size 0 max 16384 2024-03-31T23:12:39.499 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.497+0000 7f403701e740 1 Processor -- start 2024-03-31T23:12:39.499 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.497+0000 7f403701e740 1 -- start start 2024-03-31T23:12:39.499 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.497+0000 7f403701e740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55937a66b290 0x55937a673e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:39.499 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.497+0000 7f403701e740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55937a66bbb0 0x55937a671f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:39.499 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.497+0000 7f403701e740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55937a674790 0x55937a68c5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:39.499 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.497+0000 7f403701e740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f402c1af290 con 0x55937a66b290 2024-03-31T23:12:39.499 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.497+0000 7f403701e740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f402c1acb60 con 0x55937a674790 2024-03-31T23:12:39.500 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.497+0000 7f403701e740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f402c1aa9b0 con 0x55937a66bbb0 2024-03-31T23:12:39.500 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.497+0000 7f40329dd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55937a66b290 0x55937a673e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:39.500 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.497+0000 7f40319db640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55937a66bbb0 0x55937a671f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:39.500 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.497+0000 7f40329dd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55937a66b290 0x55937a673e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:32952/0 (socket says 172.21.15.46:32952) 2024-03-31T23:12:39.500 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.497+0000 7f40319db640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55937a66bbb0 0x55937a671f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.46:45690/0 (socket says 172.21.15.46:45690) 2024-03-31T23:12:39.500 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.497+0000 7f40329dd640 1 -- 172.21.15.46:0/1335991985 learned_addr learned my addr 172.21.15.46:0/1335991985 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:12:39.500 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.497+0000 7f40329dd640 1 --2- 172.21.15.46:0/1335991985 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55937a674790 0x55937a68c5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:39.500 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.498+0000 7f40319db640 1 -- 172.21.15.46:0/1335991985 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55937a674790 msgr2=0x55937a68c5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:12:39.500 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.498+0000 7f40319db640 1 --2- 172.21.15.46:0/1335991985 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55937a674790 0x55937a68c5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:39.500 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.498+0000 7f40319db640 1 -- 172.21.15.46:0/1335991985 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55937a66b290 msgr2=0x55937a673e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:12:39.500 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.498+0000 7f40319db640 1 --2- 172.21.15.46:0/1335991985 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55937a66b290 0x55937a673e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:39.500 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.498+0000 7f40319db640 1 -- 172.21.15.46:0/1335991985 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55937a6812b0 con 0x55937a66bbb0 2024-03-31T23:12:39.500 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.498+0000 7f40319db640 1 --2- 172.21.15.46:0/1335991985 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55937a66bbb0 0x55937a671f90 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f401c009850 tx=0x7f401c041000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2fb4eab75e6c4c9 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:12:39.500 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.498+0000 7f4017fff640 1 -- 172.21.15.46:0/1335991985 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f401c04d040 con 0x55937a66bbb0 2024-03-31T23:12:39.500 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.498+0000 7f4017fff640 1 -- 172.21.15.46:0/1335991985 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4024004050 con 0x55937a66bbb0 2024-03-31T23:12:39.501 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.498+0000 7f4017fff640 1 -- 172.21.15.46:0/1335991985 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f401c03ed60 con 0x55937a66bbb0 2024-03-31T23:12:39.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.499+0000 7f403701e740 1 -- 172.21.15.46:0/1335991985 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55937a66bbb0 msgr2=0x55937a671f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:12:39.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.499+0000 7f403701e740 1 --2- 172.21.15.46:0/1335991985 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55937a66bbb0 0x55937a671f90 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f401c009850 tx=0x7f401c041000 comp rx=0 tx=0).stop 2024-03-31T23:12:39.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.499+0000 7f403701e740 1 -- 172.21.15.46:0/1335991985 shutdown_connections 2024-03-31T23:12:39.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.499+0000 7f403701e740 1 --2- 172.21.15.46:0/1335991985 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55937a674790 0x55937a68c5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:39.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.499+0000 7f403701e740 1 --2- 172.21.15.46:0/1335991985 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55937a66bbb0 0x55937a671f90 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:39.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.499+0000 7f403701e740 1 --2- 172.21.15.46:0/1335991985 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55937a66b290 0x55937a673e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:39.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.499+0000 7f403701e740 1 -- 172.21.15.46:0/1335991985 >> 172.21.15.46:0/1335991985 conn(0x55937a6659c0 msgr2=0x55937a666cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:12:39.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.499+0000 7f403701e740 1 -- 172.21.15.46:0/1335991985 shutdown_connections 2024-03-31T23:12:39.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.499+0000 7f403701e740 1 -- 172.21.15.46:0/1335991985 wait complete. 2024-03-31T23:12:39.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.499+0000 7f403701e740 1 -- start start 2024-03-31T23:12:39.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.499+0000 7f403701e740 1 Processor -- start 2024-03-31T23:12:39.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.500+0000 7f403701e740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55937a66b290 0x55937a678a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:39.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.500+0000 7f403701e740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55937a66bbb0 0x55937a678fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:39.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.500+0000 7f403701e740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55937a674790 0x55937a6e9c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:39.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.500+0000 7f403701e740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f402c1ab020 con 0x55937a674790 2024-03-31T23:12:39.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.500+0000 7f403701e740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f402c1ac490 con 0x55937a66b290 2024-03-31T23:12:39.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.500+0000 7f403701e740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f402c1ab6f0 con 0x55937a66bbb0 2024-03-31T23:12:39.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.500+0000 7f403701e740 10 client.0 _ll_register_callbacks cb 0x7f401c009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T23:12:39.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.500+0000 7f40329dd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55937a674790 0x55937a6e9c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:39.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.500+0000 7f40329dd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55937a674790 0x55937a6e9c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:32956/0 (socket says 172.21.15.46:32956) 2024-03-31T23:12:39.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.500+0000 7f40329dd640 1 -- 172.21.15.46:0/2022558450 learned_addr learned my addr 172.21.15.46:0/2022558450 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:12:39.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.500+0000 7f40319db640 1 --2- 172.21.15.46:0/2022558450 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55937a66bbb0 0x55937a678fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:39.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.500+0000 7f40329dd640 1 --2- 172.21.15.46:0/2022558450 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55937a66b290 0x55937a678a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:39.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.500+0000 7f40329dd640 1 -- 172.21.15.46:0/2022558450 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55937a66bbb0 msgr2=0x55937a678fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:12:39.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.500+0000 7f40329dd640 1 --2- 172.21.15.46:0/2022558450 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55937a66bbb0 0x55937a678fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:39.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.500+0000 7f40329dd640 1 -- 172.21.15.46:0/2022558450 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55937a66b290 msgr2=0x55937a678a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:12:39.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.500+0000 7f40329dd640 1 --2- 172.21.15.46:0/2022558450 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55937a66b290 0x55937a678a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:39.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.500+0000 7f40329dd640 1 -- 172.21.15.46:0/2022558450 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55937a2a30f0 con 0x55937a674790 2024-03-31T23:12:39.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.500+0000 7f40319db640 1 --2- 172.21.15.46:0/2022558450 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55937a66bbb0 0x55937a678fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:12:39.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.501+0000 7f40329dd640 1 --2- 172.21.15.46:0/2022558450 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55937a674790 0x55937a6e9c10 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f4024009fa0 tx=0x7f40240118d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e60a880de2e7f0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:12:39.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.501+0000 7f4015ffb640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-31T23:12:39.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.501+0000 7f4015ffb640 1 -- 172.21.15.46:0/2022558450 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40240055e0 con 0x55937a674790 2024-03-31T23:12:39.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.501+0000 7f4015ffb640 1 -- 172.21.15.46:0/2022558450 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7f4024005780 con 0x55937a674790 2024-03-31T23:12:39.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.501+0000 7f403701e740 1 -- 172.21.15.46:0/2022558450 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4024004050 con 0x55937a674790 2024-03-31T23:12:39.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.501+0000 7f4015ffb640 1 -- 172.21.15.46:0/2022558450 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f4024005980 con 0x55937a674790 2024-03-31T23:12:39.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.501+0000 7f403701e740 10 client.16035 fetch_fsmap learned FSMap version 25 2024-03-31T23:12:39.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.501+0000 7f403701e740 1 -- 172.21.15.46:0/2022558450 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55937a6eb290 con 0x55937a674790 2024-03-31T23:12:39.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.501+0000 7f4015ffb640 1 -- 172.21.15.46:0/2022558450 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4024004050 con 0x55937a674790 2024-03-31T23:12:39.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.501+0000 7f4015ffb640 1 -- 172.21.15.46:0/2022558450 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f4024025040 con 0x55937a674790 2024-03-31T23:12:39.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.502+0000 7f403701e740 10 client.16035 fetch_fsmap finished waiting for FSMap version 25 2024-03-31T23:12:39.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.502+0000 7f403701e740 10 client.16035 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-31T23:12:39.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.502+0000 7f403701e740 20 client.16035 populate_metadata read hostname 'smithi046' 2024-03-31T23:12:39.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.502+0000 7f403701e740 1 --2- 172.21.15.46:0/2022558450 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55937a684900 0x55937a6ed880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:39.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.502+0000 7f403701e740 4 client.16035 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-31T23:12:39.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.502+0000 7f403701e740 1 -- 172.21.15.46:0/2022558450 --> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55937a5d6a10 con 0x55937a684900 2024-03-31T23:12:39.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.502+0000 7f40319db640 1 --2- 172.21.15.46:0/2022558450 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55937a684900 0x55937a6ed880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:39.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.502+0000 7f4015ffb640 10 client.16035 ms_handle_connect on v2:172.21.15.142:6838/2519143918 2024-03-31T23:12:39.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.502+0000 7f40319db640 1 --2- 172.21.15.46:0/2022558450 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55937a684900 0x55937a6ed880 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fe03106e7e4f2221 server_cookie=fc0a02fa7a6ee093 in_seq=0 out_seq=0 2024-03-31T23:12:39.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.502+0000 7f4015ffb640 1 -- 172.21.15.46:0/2022558450 <== mds.0 v2:172.21.15.142:6838/2519143918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55937a5d6a10 con 0x55937a684900 2024-03-31T23:12:39.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.502+0000 7f4015ffb640 10 client.16035 handle_command_reply: tid=0 2024-03-31T23:12:39.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.502+0000 7f403701e740 1 client.16035 shutdown 2024-03-31T23:12:39.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.503+0000 7f403701e740 1 -- 172.21.15.46:0/2022558450 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55937a674790 msgr2=0x55937a6e9c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:12:39.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.503+0000 7f403701e740 1 --2- 172.21.15.46:0/2022558450 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55937a674790 0x55937a6e9c10 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f4024009fa0 tx=0x7f40240118d0 comp rx=0 tx=0).stop 2024-03-31T23:12:39.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.503+0000 7f403701e740 1 -- 172.21.15.46:0/2022558450 shutdown_connections 2024-03-31T23:12:39.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.503+0000 7f403701e740 1 --2- 172.21.15.46:0/2022558450 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55937a684900 0x55937a6ed880 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:39.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.503+0000 7f403701e740 1 --2- 172.21.15.46:0/2022558450 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55937a674790 0x55937a6e9c10 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:39.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.503+0000 7f4015ffb640 0 client.16035 ms_handle_reset on v2:172.21.15.142:6838/2519143918 2024-03-31T23:12:39.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.503+0000 7f403701e740 1 --2- 172.21.15.46:0/2022558450 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55937a66bbb0 0x55937a678fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:39.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.503+0000 7f403701e740 1 --2- 172.21.15.46:0/2022558450 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55937a66b290 0x55937a678a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:39.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.503+0000 7f403701e740 1 -- 172.21.15.46:0/2022558450 >> 172.21.15.46:0/2022558450 conn(0x55937a6659c0 msgr2=0x55937a667550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:12:39.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.503+0000 7f403701e740 1 -- 172.21.15.46:0/2022558450 shutdown_connections 2024-03-31T23:12:39.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.503+0000 7f403701e740 1 -- 172.21.15.46:0/2022558450 wait complete. 2024-03-31T23:12:39.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.503+0000 7f403701e740 20 client.16035 trim_cache size 0 max 16384 2024-03-31T23:12:39.511 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-03-31T23:12:39.511 INFO:teuthology.orchestra.run.smithi046.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-31T23:12:39.511 INFO:teuthology.orchestra.run.smithi046.stdout: "scrubs": { 2024-03-31T23:12:39.511 INFO:teuthology.orchestra.run.smithi046.stdout: "cbbde91d-d625-44c3-b4c4-d3bec5ed0c77": { 2024-03-31T23:12:39.511 INFO:teuthology.orchestra.run.smithi046.stdout: "path": "/", 2024-03-31T23:12:39.512 INFO:teuthology.orchestra.run.smithi046.stdout: "tag": "cbbde91d-d625-44c3-b4c4-d3bec5ed0c77", 2024-03-31T23:12:39.512 INFO:teuthology.orchestra.run.smithi046.stdout: "options": "recursive,force" 2024-03-31T23:12:39.512 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-31T23:12:39.512 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-31T23:12:39.512 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-03-31T23:12:39.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.504+0000 7f4034440640 1 -- 172.21.15.46:0/4248823771 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f40180a2730 msgr2=0x7f40180a4bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:12:39.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.504+0000 7f4034440640 1 --2- 172.21.15.46:0/4248823771 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f40180a2730 0x7f40180a4bf0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f40280cc680 tx=0x7f402806d6b0 comp rx=0 tx=0).stop 2024-03-31T23:12:39.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.504+0000 7f4034440640 1 -- 172.21.15.46:0/4248823771 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f402c1016e0 msgr2=0x7f402c197f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:12:39.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.504+0000 7f4034440640 1 --2- 172.21.15.46:0/4248823771 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f402c1016e0 0x7f402c197f80 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f402800e9f0 tx=0x7f402800eeb0 comp rx=0 tx=0).stop 2024-03-31T23:12:39.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.504+0000 7f4034440640 1 -- 172.21.15.46:0/4248823771 shutdown_connections 2024-03-31T23:12:39.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.504+0000 7f4034440640 1 --2- 172.21.15.46:0/4248823771 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f40180a2730 0x7f40180a4bf0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:39.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.504+0000 7f4034440640 1 --2- 172.21.15.46:0/4248823771 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f402c1076f0 0x7f402c19c840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:39.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.504+0000 7f4034440640 1 --2- 172.21.15.46:0/4248823771 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f402c1020a0 0x7f402c1984c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:39.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.504+0000 7f4034440640 1 --2- 172.21.15.46:0/4248823771 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f402c1016e0 0x7f402c197f80 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:39.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.504+0000 7f4034440640 1 -- 172.21.15.46:0/4248823771 >> 172.21.15.46:0/4248823771 conn(0x7f402c0fd1f0 msgr2=0x7f402c0fdb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:12:39.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.504+0000 7f4034440640 1 -- 172.21.15.46:0/4248823771 shutdown_connections 2024-03-31T23:12:39.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:12:39.504+0000 7f4034440640 1 -- 172.21.15.46:0/4248823771 wait complete. 2024-03-31T23:12:39.518 INFO:tasks.cephfs.filesystem:scrub status for tag:cbbde91d-d625-44c3-b4c4-d3bec5ed0c77 - {'path': '/', 'tag': 'cbbde91d-d625-44c3-b4c4-d3bec5ed0c77', 'options': 'recursive,force'} 2024-03-31T23:12:40.611 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:12:40 smithi046 ceph-mon[32498]: pgmap v31: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 511 B/s wr, 8 op/s 2024-03-31T23:12:40.611 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:12:40 smithi046 ceph-mon[32498]: scrub queued for path: / 2024-03-31T23:12:40.612 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:12:40 smithi046 ceph-mon[32498]: scrub summary: idle+waiting paths [/] 2024-03-31T23:12:40.612 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:12:40 smithi046 ceph-mon[32498]: scrub summary: active paths [/] 2024-03-31T23:12:40.612 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:12:40 smithi046 ceph-mon[32498]: scrub summary: idle+waiting paths [/] 2024-03-31T23:12:40.693 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:12:40 smithi142 ceph-mon[35271]: pgmap v31: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 511 B/s wr, 8 op/s 2024-03-31T23:12:40.693 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:12:40 smithi142 ceph-mon[35271]: scrub queued for path: / 2024-03-31T23:12:40.693 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:12:40 smithi142 ceph-mon[35271]: scrub summary: idle+waiting paths [/] 2024-03-31T23:12:40.693 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:12:40 smithi142 ceph-mon[35271]: scrub summary: active paths [/] 2024-03-31T23:12:40.693 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:12:40 smithi142 ceph-mon[35271]: scrub summary: idle+waiting paths [/] 2024-03-31T23:12:40.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:12:40 smithi162 ceph-mon[36237]: pgmap v31: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 511 B/s wr, 8 op/s 2024-03-31T23:12:40.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:12:40 smithi162 ceph-mon[36237]: scrub queued for path: / 2024-03-31T23:12:40.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:12:40 smithi162 ceph-mon[36237]: scrub summary: idle+waiting paths [/] 2024-03-31T23:12:40.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:12:40 smithi162 ceph-mon[36237]: scrub summary: active paths [/] 2024-03-31T23:12:40.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:12:40 smithi162 ceph-mon[36237]: scrub summary: idle+waiting paths [/] 2024-03-31T23:12:42.611 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:12:42 smithi046 ceph-mon[32498]: pgmap v32: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-03-31T23:12:42.612 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:12:42 smithi046 ceph-mon[32498]: scrub summary: idle 2024-03-31T23:12:42.693 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:12:42 smithi142 ceph-mon[35271]: pgmap v32: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-03-31T23:12:42.693 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:12:42 smithi142 ceph-mon[35271]: scrub summary: idle 2024-03-31T23:12:42.795 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:12:42 smithi162 ceph-mon[36237]: pgmap v32: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-03-31T23:12:42.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:12:42 smithi162 ceph-mon[36237]: scrub summary: idle 2024-03-31T23:12:44.611 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:12:44 smithi046 ceph-mon[32498]: pgmap v33: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-03-31T23:12:44.693 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:12:44 smithi142 ceph-mon[35271]: pgmap v33: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-03-31T23:12:44.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:12:44 smithi162 ceph-mon[36237]: pgmap v33: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-03-31T23:12:46.611 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:12:46 smithi046 ceph-mon[32498]: pgmap v34: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 255 B/s wr, 8 op/s 2024-03-31T23:12:46.693 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:12:46 smithi142 ceph-mon[35271]: pgmap v34: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 255 B/s wr, 8 op/s 2024-03-31T23:12:46.795 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:12:46 smithi162 ceph-mon[36237]: pgmap v34: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 255 B/s wr, 8 op/s 2024-03-31T23:12:48.612 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:12:48 smithi046 ceph-mon[32498]: pgmap v35: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 255 B/s wr, 6 op/s 2024-03-31T23:12:48.612 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:12:48 smithi046 ceph-mon[32498]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-31T23:12:48.612 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:12:48 smithi046 ceph-mon[32498]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' 2024-03-31T23:12:48.612 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:12:48 smithi046 ceph-mon[32498]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' 2024-03-31T23:12:48.612 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:12:48 smithi046 ceph-mon[32498]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' 2024-03-31T23:12:48.612 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:12:48 smithi046 ceph-mon[32498]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' 2024-03-31T23:12:48.612 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:12:48 smithi046 ceph-mon[32498]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' 2024-03-31T23:12:48.612 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:12:48 smithi046 ceph-mon[32498]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' 2024-03-31T23:12:48.693 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:12:48 smithi142 ceph-mon[35271]: pgmap v35: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 255 B/s wr, 6 op/s 2024-03-31T23:12:48.693 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:12:48 smithi142 ceph-mon[35271]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-31T23:12:48.693 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:12:48 smithi142 ceph-mon[35271]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' 2024-03-31T23:12:48.693 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:12:48 smithi142 ceph-mon[35271]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' 2024-03-31T23:12:48.693 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:12:48 smithi142 ceph-mon[35271]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' 2024-03-31T23:12:48.694 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:12:48 smithi142 ceph-mon[35271]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' 2024-03-31T23:12:48.694 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:12:48 smithi142 ceph-mon[35271]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' 2024-03-31T23:12:48.694 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:12:48 smithi142 ceph-mon[35271]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' 2024-03-31T23:12:48.795 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:12:48 smithi162 ceph-mon[36237]: pgmap v35: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 255 B/s wr, 6 op/s 2024-03-31T23:12:48.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:12:48 smithi162 ceph-mon[36237]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-31T23:12:48.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:12:48 smithi162 ceph-mon[36237]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' 2024-03-31T23:12:48.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:12:48 smithi162 ceph-mon[36237]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' 2024-03-31T23:12:48.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:12:48 smithi162 ceph-mon[36237]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' 2024-03-31T23:12:48.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:12:48 smithi162 ceph-mon[36237]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' 2024-03-31T23:12:48.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:12:48 smithi162 ceph-mon[36237]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' 2024-03-31T23:12:48.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:12:48 smithi162 ceph-mon[36237]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' 2024-03-31T23:12:49.611 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:12:49 smithi046 ceph-mon[32498]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:12:49.611 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:12:49 smithi046 ceph-mon[32498]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-31T23:12:49.611 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:12:49 smithi046 ceph-mon[32498]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' 2024-03-31T23:12:49.693 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:12:49 smithi142 ceph-mon[35271]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:12:49.693 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:12:49 smithi142 ceph-mon[35271]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-31T23:12:49.693 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:12:49 smithi142 ceph-mon[35271]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' 2024-03-31T23:12:49.795 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:12:49 smithi162 ceph-mon[36237]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:12:49.795 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:12:49 smithi162 ceph-mon[36237]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-31T23:12:49.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:12:49 smithi162 ceph-mon[36237]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' 2024-03-31T23:12:50.611 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:12:50 smithi046 ceph-mon[32498]: pgmap v36: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 255 B/s wr, 8 op/s 2024-03-31T23:12:50.693 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:12:50 smithi142 ceph-mon[35271]: pgmap v36: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 255 B/s wr, 8 op/s 2024-03-31T23:12:50.697 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:12:50 smithi162 ceph-mon[36237]: pgmap v36: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 255 B/s wr, 8 op/s 2024-03-31T23:12:50.771 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:50.770+0000 7fcd34ed4640 1 -- 172.21.15.46:0/980147248 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcd300ffe60 msgr2=0x7fcd30100240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:12:50.771 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:50.770+0000 7fcd34ed4640 1 --2- 172.21.15.46:0/980147248 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcd300ffe60 0x7fcd30100240 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7fcd20009900 tx=0x7fcd2002bc30 comp rx=0 tx=0).stop 2024-03-31T23:12:50.771 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:50.770+0000 7fcd34ed4640 1 -- 172.21.15.46:0/980147248 shutdown_connections 2024-03-31T23:12:50.771 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:50.770+0000 7fcd34ed4640 1 --2- 172.21.15.46:0/980147248 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcd30106980 0x7fcd3010e620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:50.771 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:50.770+0000 7fcd34ed4640 1 --2- 172.21.15.46:0/980147248 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcd30100810 0x7fcd301062a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:50.771 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:50.770+0000 7fcd34ed4640 1 --2- 172.21.15.46:0/980147248 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcd300ffe60 0x7fcd30100240 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:50.771 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:50.770+0000 7fcd34ed4640 1 -- 172.21.15.46:0/980147248 >> 172.21.15.46:0/980147248 conn(0x7fcd30076b70 msgr2=0x7fcd300fdf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:12:50.771 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:50.770+0000 7fcd34ed4640 1 -- 172.21.15.46:0/980147248 shutdown_connections 2024-03-31T23:12:50.771 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:50.770+0000 7fcd34ed4640 1 -- 172.21.15.46:0/980147248 wait complete. 2024-03-31T23:12:50.771 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:50.771+0000 7fcd34ed4640 1 Processor -- start 2024-03-31T23:12:50.773 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:50.771+0000 7fcd34ed4640 1 -- start start 2024-03-31T23:12:50.773 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:50.772+0000 7fcd34ed4640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcd300ffe60 0x7fcd30197e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:50.773 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:50.772+0000 7fcd34ed4640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcd30106980 0x7fcd301983b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:50.773 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:50.772+0000 7fcd34ed4640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcd3019c730 0x7fcd3019cbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:50.774 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:50.772+0000 7fcd34ed4640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fcd301119b0 con 0x7fcd300ffe60 2024-03-31T23:12:50.774 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:50.772+0000 7fcd34ed4640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fcd30111830 con 0x7fcd30106980 2024-03-31T23:12:50.774 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:50.772+0000 7fcd34ed4640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fcd30111b30 con 0x7fcd3019c730 2024-03-31T23:12:50.774 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:50.772+0000 7fcd2e59c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcd300ffe60 0x7fcd30197e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:50.774 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:50.772+0000 7fcd2e59c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcd300ffe60 0x7fcd30197e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:36130/0 (socket says 172.21.15.46:36130) 2024-03-31T23:12:50.774 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:50.772+0000 7fcd2e59c640 1 -- 172.21.15.46:0/3462612284 learned_addr learned my addr 172.21.15.46:0/3462612284 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:12:50.774 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:50.772+0000 7fcd25d9b640 1 --2- 172.21.15.46:0/3462612284 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcd30106980 0x7fcd301983b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:50.774 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:50.772+0000 7fcd2ed9d640 1 --2- 172.21.15.46:0/3462612284 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcd3019c730 0x7fcd3019cbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:50.774 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:50.772+0000 7fcd2e59c640 1 -- 172.21.15.46:0/3462612284 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcd3019c730 msgr2=0x7fcd3019cbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:12:50.774 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:50.772+0000 7fcd2e59c640 1 --2- 172.21.15.46:0/3462612284 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcd3019c730 0x7fcd3019cbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:50.774 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:50.772+0000 7fcd2e59c640 1 -- 172.21.15.46:0/3462612284 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcd30106980 msgr2=0x7fcd301983b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:12:50.774 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:50.772+0000 7fcd2e59c640 1 --2- 172.21.15.46:0/3462612284 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcd30106980 0x7fcd301983b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:50.774 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:50.772+0000 7fcd2e59c640 1 -- 172.21.15.46:0/3462612284 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcd3019d220 con 0x7fcd300ffe60 2024-03-31T23:12:50.774 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:50.772+0000 7fcd2ed9d640 1 --2- 172.21.15.46:0/3462612284 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcd3019c730 0x7fcd3019cbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:12:50.774 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:50.772+0000 7fcd2e59c640 1 --2- 172.21.15.46:0/3462612284 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcd300ffe60 0x7fcd30197e70 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7fcd20005940 tx=0x7fcd2002c5f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:12:50.775 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:50.773+0000 7fcd27fff640 1 -- 172.21.15.46:0/3462612284 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd20043020 con 0x7fcd300ffe60 2024-03-31T23:12:50.775 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:50.773+0000 7fcd27fff640 1 -- 172.21.15.46:0/3462612284 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fcd2002ca70 con 0x7fcd300ffe60 2024-03-31T23:12:50.775 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:50.773+0000 7fcd34ed4640 1 -- 172.21.15.46:0/3462612284 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcd3019d4b0 con 0x7fcd300ffe60 2024-03-31T23:12:50.775 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:50.773+0000 7fcd27fff640 1 -- 172.21.15.46:0/3462612284 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd20003a80 con 0x7fcd300ffe60 2024-03-31T23:12:50.775 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:50.773+0000 7fcd34ed4640 1 -- 172.21.15.46:0/3462612284 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcd301a4cd0 con 0x7fcd300ffe60 2024-03-31T23:12:50.775 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:50.775+0000 7fcd27fff640 1 -- 172.21.15.46:0/3462612284 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 24) ==== 137553+0+0 (secure 0 0 0) 0x7fcd20046020 con 0x7fcd300ffe60 2024-03-31T23:12:50.775 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:50.775+0000 7fcd37b98740 1 Processor -- start 2024-03-31T23:12:50.775 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:50.776+0000 7fcd37b98740 1 -- start start 2024-03-31T23:12:50.777 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:50.776+0000 7fcd27fff640 1 --2- 172.21.15.46:0/3462612284 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fcd1c0a27d0 0x7fcd1c0a4c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:50.777 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:50.776+0000 7fcd37b98740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55ec734e06c0 0x55ec734e0aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:50.777 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:50.776+0000 7fcd37b98740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ec734e0fe0 0x55ec734e9870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:50.777 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:50.776+0000 7fcd37b98740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ec734e9db0 0x55ec734ec1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:50.777 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:50.776+0000 7fcd37b98740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55ec734438f0 con 0x55ec734e0fe0 2024-03-31T23:12:50.777 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:50.776+0000 7fcd37b98740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55ec733ec8d0 con 0x55ec734e06c0 2024-03-31T23:12:50.777 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:50.776+0000 7fcd37b98740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55ec7343f990 con 0x55ec734e9db0 2024-03-31T23:12:50.778 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:50.776+0000 7fcd2ed9d640 1 --2- 172.21.15.46:0/3462612284 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fcd1c0a27d0 0x7fcd1c0a4c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:50.778 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:50.776+0000 7fcd25d9b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ec734e0fe0 0x55ec734e9870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:50.778 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:50.776+0000 7fcd25d9b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ec734e0fe0 0x55ec734e9870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:36138/0 (socket says 172.21.15.46:36138) 2024-03-31T23:12:50.778 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:50.776+0000 7fcd25d9b640 1 -- 172.21.15.46:0/1552616610 learned_addr learned my addr 172.21.15.46:0/1552616610 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:12:50.778 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:50.776+0000 7fcd2e59c640 1 --2- 172.21.15.46:0/1552616610 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55ec734e06c0 0x55ec734e0aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:50.778 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:50.776+0000 7fcd27fff640 1 -- 172.21.15.46:0/3462612284 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7fcd200f1100 con 0x7fcd300ffe60 2024-03-31T23:12:50.778 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:50.776+0000 7fcd2ed9d640 1 --2- 172.21.15.46:0/1552616610 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ec734e9db0 0x55ec734ec1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:50.778 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:50.777+0000 7fcd25d9b640 1 -- 172.21.15.46:0/1552616610 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ec734e9db0 msgr2=0x55ec734ec1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:12:50.778 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:50.777+0000 7fcd25d9b640 1 --2- 172.21.15.46:0/1552616610 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ec734e9db0 0x55ec734ec1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:50.778 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:50.777+0000 7fcd25d9b640 1 -- 172.21.15.46:0/1552616610 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55ec734e06c0 msgr2=0x55ec734e0aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:12:50.778 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:50.777+0000 7fcd25d9b640 1 --2- 172.21.15.46:0/1552616610 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55ec734e06c0 0x55ec734e0aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:50.778 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:50.777+0000 7fcd25d9b640 1 -- 172.21.15.46:0/1552616610 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ec73158210 con 0x55ec734e0fe0 2024-03-31T23:12:50.778 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:50.777+0000 7fcd2e59c640 1 --2- 172.21.15.46:0/1552616610 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55ec734e06c0 0x55ec734e0aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:12:50.778 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:50.777+0000 7fcd2ed9d640 1 --2- 172.21.15.46:0/3462612284 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fcd1c0a27d0 0x7fcd1c0a4c90 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fcd28009fd0 tx=0x7fcd28011f10 comp rx=0 tx=0).ready entity=mgr.15654 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:12:50.778 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:50.777+0000 7fcd25d9b640 1 --2- 172.21.15.46:0/1552616610 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ec734e0fe0 0x55ec734e9870 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7fcd18009c60 tx=0x7fcd18011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1954cd5a3799074c server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:12:50.779 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:50.777+0000 7fcd24d99640 1 -- 172.21.15.46:0/1552616610 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd18004230 con 0x55ec734e0fe0 2024-03-31T23:12:50.779 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:50.777+0000 7fcd24d99640 1 -- 172.21.15.46:0/1552616610 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fcd180043d0 con 0x55ec734e0fe0 2024-03-31T23:12:50.779 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:50.777+0000 7fcd24d99640 1 -- 172.21.15.46:0/1552616610 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd18004cf0 con 0x55ec734e0fe0 2024-03-31T23:12:50.783 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:50.778+0000 7fcd37b98740 1 -- 172.21.15.46:0/1552616610 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ec734e0fe0 msgr2=0x55ec734e9870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:12:50.783 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:50.778+0000 7fcd37b98740 1 --2- 172.21.15.46:0/1552616610 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ec734e0fe0 0x55ec734e9870 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7fcd18009c60 tx=0x7fcd18011810 comp rx=0 tx=0).stop 2024-03-31T23:12:50.783 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:50.778+0000 7fcd37b98740 1 -- 172.21.15.46:0/1552616610 shutdown_connections 2024-03-31T23:12:50.783 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:50.778+0000 7fcd37b98740 1 --2- 172.21.15.46:0/1552616610 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ec734e9db0 0x55ec734ec1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:50.783 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:50.778+0000 7fcd37b98740 1 --2- 172.21.15.46:0/1552616610 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ec734e0fe0 0x55ec734e9870 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:50.783 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:50.778+0000 7fcd37b98740 1 --2- 172.21.15.46:0/1552616610 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55ec734e06c0 0x55ec734e0aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:50.783 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:50.778+0000 7fcd37b98740 1 -- 172.21.15.46:0/1552616610 >> 172.21.15.46:0/1552616610 conn(0x55ec734dafe0 msgr2=0x55ec734db3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:12:50.783 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:50.778+0000 7fcd37b98740 1 -- 172.21.15.46:0/1552616610 shutdown_connections 2024-03-31T23:12:50.784 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:50.778+0000 7fcd37b98740 1 -- 172.21.15.46:0/1552616610 wait complete. 2024-03-31T23:12:50.784 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:50.778+0000 7fcd37b98740 1 -- start start 2024-03-31T23:12:50.784 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:50.778+0000 7fcd37b98740 1 Processor -- start 2024-03-31T23:12:50.784 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:50.778+0000 7fcd37b98740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55ec734e06c0 0x55ec734e6e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:50.784 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:50.778+0000 7fcd37b98740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ec734e0fe0 0x55ec734e7340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:50.784 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:50.779+0000 7fcd37b98740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ec734e9db0 0x55ec734e7880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:50.784 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:50.779+0000 7fcd37b98740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55ec734ec9e0 con 0x55ec734e9db0 2024-03-31T23:12:50.784 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:50.779+0000 7fcd37b98740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55ec733e1f80 con 0x55ec734e06c0 2024-03-31T23:12:50.784 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:50.779+0000 7fcd37b98740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55ec734ecb60 con 0x55ec734e0fe0 2024-03-31T23:12:50.784 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:50.779+0000 7fcd37b98740 10 client.0 _ll_register_callbacks cb 0x55ec7343a290 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T23:12:50.784 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:50.779+0000 7fcd2ed9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ec734e9db0 0x55ec734e7880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:50.784 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:50.779+0000 7fcd2e59c640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55ec734e06c0 0x55ec734e6e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:50.784 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:50.779+0000 7fcd2ed9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ec734e9db0 0x55ec734e7880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:36142/0 (socket says 172.21.15.46:36142) 2024-03-31T23:12:50.784 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:50.779+0000 7fcd2ed9d640 1 -- 172.21.15.46:0/2718576298 learned_addr learned my addr 172.21.15.46:0/2718576298 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:12:50.784 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:50.779+0000 7fcd25d9b640 1 --2- 172.21.15.46:0/2718576298 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ec734e0fe0 0x55ec734e7340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:50.784 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:50.779+0000 7fcd2ed9d640 1 -- 172.21.15.46:0/2718576298 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ec734e0fe0 msgr2=0x55ec734e7340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:12:50.785 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:50.779+0000 7fcd2ed9d640 1 --2- 172.21.15.46:0/2718576298 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ec734e0fe0 0x55ec734e7340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:50.785 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:50.779+0000 7fcd2ed9d640 1 -- 172.21.15.46:0/2718576298 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55ec734e06c0 msgr2=0x55ec734e6e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:12:50.785 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:50.779+0000 7fcd2ed9d640 1 --2- 172.21.15.46:0/2718576298 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55ec734e06c0 0x55ec734e6e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:50.785 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:50.779+0000 7fcd2ed9d640 1 -- 172.21.15.46:0/2718576298 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ec73224620 con 0x55ec734e9db0 2024-03-31T23:12:50.785 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:50.779+0000 7fcd2ed9d640 1 --2- 172.21.15.46:0/2718576298 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ec734e9db0 0x55ec734e7880 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7fcd2800e6e0 tx=0x7fcd28004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4cc822a51b95c80d server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:12:50.785 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:50.779+0000 7fcd15ffb640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-31T23:12:50.785 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:50.779+0000 7fcd15ffb640 1 -- 172.21.15.46:0/2718576298 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd28024070 con 0x55ec734e9db0 2024-03-31T23:12:50.785 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:50.779+0000 7fcd15ffb640 1 -- 172.21.15.46:0/2718576298 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7fcd2801da30 con 0x55ec734e9db0 2024-03-31T23:12:50.785 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:50.779+0000 7fcd37b98740 1 -- 172.21.15.46:0/2718576298 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcd180043d0 con 0x55ec734e9db0 2024-03-31T23:12:50.785 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:50.779+0000 7fcd15ffb640 1 -- 172.21.15.46:0/2718576298 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fcd2801dc30 con 0x55ec734e9db0 2024-03-31T23:12:50.785 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:50.779+0000 7fcd37b98740 10 client.16047 fetch_fsmap learned FSMap version 25 2024-03-31T23:12:50.785 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:50.780+0000 7fcd37b98740 1 -- 172.21.15.46:0/2718576298 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55ec734f8960 con 0x55ec734e9db0 2024-03-31T23:12:50.785 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:50.780+0000 7fcd15ffb640 1 -- 172.21.15.46:0/2718576298 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fcd180043d0 con 0x55ec734e9db0 2024-03-31T23:12:50.785 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:50.780+0000 7fcd15ffb640 1 -- 172.21.15.46:0/2718576298 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7fcd2801f070 con 0x55ec734e9db0 2024-03-31T23:12:50.785 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:50.780+0000 7fcd37b98740 10 client.16047 fetch_fsmap finished waiting for FSMap version 25 2024-03-31T23:12:50.786 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:50.780+0000 7fcd37b98740 10 client.16047 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-31T23:12:50.786 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:50.780+0000 7fcd37b98740 20 client.16047 populate_metadata read hostname 'smithi046' 2024-03-31T23:12:50.786 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:50.780+0000 7fcd37b98740 1 --2- 172.21.15.46:0/2718576298 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55ec734f9e20 0x55ec734fc210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:50.786 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:50.780+0000 7fcd37b98740 4 client.16047 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-31T23:12:50.786 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:50.780+0000 7fcd37b98740 1 -- 172.21.15.46:0/2718576298 --> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ec73457120 con 0x55ec734f9e20 2024-03-31T23:12:50.786 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:50.780+0000 7fcd2e59c640 1 --2- 172.21.15.46:0/2718576298 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55ec734f9e20 0x55ec734fc210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:50.786 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:50.781+0000 7fcd2e59c640 1 --2- 172.21.15.46:0/2718576298 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55ec734f9e20 0x55ec734fc210 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ff1ab6abe936a987 server_cookie=c1f1b7e440fc2b55 in_seq=0 out_seq=0 2024-03-31T23:12:50.786 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:50.781+0000 7fcd15ffb640 10 client.16047 ms_handle_connect on v2:172.21.15.142:6838/2519143918 2024-03-31T23:12:50.786 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:50.782+0000 7fcd15ffb640 1 -- 172.21.15.46:0/2718576298 <== mds.0 v2:172.21.15.142:6838/2519143918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55ec73457120 con 0x55ec734f9e20 2024-03-31T23:12:50.786 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:50.782+0000 7fcd15ffb640 10 client.16047 handle_command_reply: tid=0 2024-03-31T23:12:50.786 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:50.782+0000 7fcd37b98740 1 client.16047 shutdown 2024-03-31T23:12:50.786 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:50.782+0000 7fcd37b98740 1 -- 172.21.15.46:0/2718576298 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ec734e9db0 msgr2=0x55ec734e7880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:12:50.786 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:50.782+0000 7fcd37b98740 1 --2- 172.21.15.46:0/2718576298 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ec734e9db0 0x55ec734e7880 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7fcd2800e6e0 tx=0x7fcd28004ab0 comp rx=0 tx=0).stop 2024-03-31T23:12:50.786 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:50.782+0000 7fcd37b98740 1 -- 172.21.15.46:0/2718576298 shutdown_connections 2024-03-31T23:12:50.786 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:50.782+0000 7fcd37b98740 1 --2- 172.21.15.46:0/2718576298 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55ec734f9e20 0x55ec734fc210 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:50.787 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:50.782+0000 7fcd37b98740 1 --2- 172.21.15.46:0/2718576298 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ec734e9db0 0x55ec734e7880 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:50.787 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:50.782+0000 7fcd37b98740 1 --2- 172.21.15.46:0/2718576298 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ec734e0fe0 0x55ec734e7340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:50.787 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:50.782+0000 7fcd15ffb640 0 client.16047 ms_handle_reset on v2:172.21.15.142:6838/2519143918 2024-03-31T23:12:50.787 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:50.782+0000 7fcd37b98740 1 --2- 172.21.15.46:0/2718576298 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55ec734e06c0 0x55ec734e6e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:50.787 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:50.782+0000 7fcd37b98740 1 -- 172.21.15.46:0/2718576298 >> 172.21.15.46:0/2718576298 conn(0x55ec734dafe0 msgr2=0x55ec734dcd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:12:50.787 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:50.782+0000 7fcd37b98740 1 -- 172.21.15.46:0/2718576298 shutdown_connections 2024-03-31T23:12:50.787 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:50.782+0000 7fcd37b98740 1 -- 172.21.15.46:0/2718576298 wait complete. 2024-03-31T23:12:50.787 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:50.782+0000 7fcd37b98740 20 client.16047 trim_cache size 0 max 16384 2024-03-31T23:12:50.801 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:50.797+0000 7fcd37b98740 1 Processor -- start 2024-03-31T23:12:50.801 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:50.797+0000 7fcd37b98740 1 -- start start 2024-03-31T23:12:50.801 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:50.797+0000 7fcd37b98740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ec734e06c0 0x55ec734e9530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:50.801 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:50.797+0000 7fcd37b98740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55ec734e0fe0 0x55ec734f5780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:50.801 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:50.797+0000 7fcd37b98740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ec734e9db0 0x55ec73501b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:50.801 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:50.797+0000 7fcd37b98740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fcd301b12f0 con 0x55ec734e06c0 2024-03-31T23:12:50.801 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:50.797+0000 7fcd37b98740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fcd301aebc0 con 0x55ec734e0fe0 2024-03-31T23:12:50.801 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:50.797+0000 7fcd37b98740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fcd301aca10 con 0x55ec734e9db0 2024-03-31T23:12:50.801 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:50.797+0000 7fcd2e59c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ec734e06c0 0x55ec734e9530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:50.801 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:50.797+0000 7fcd2e59c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ec734e06c0 0x55ec734e9530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:36146/0 (socket says 172.21.15.46:36146) 2024-03-31T23:12:50.801 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:50.797+0000 7fcd2e59c640 1 -- 172.21.15.46:0/3969823572 learned_addr learned my addr 172.21.15.46:0/3969823572 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:12:50.802 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:50.797+0000 7fcd25d9b640 1 --2- 172.21.15.46:0/3969823572 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55ec734e0fe0 0x55ec734f5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:50.802 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:50.797+0000 7fcd2ed9d640 1 --2- 172.21.15.46:0/3969823572 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ec734e9db0 0x55ec73501b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:50.802 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:50.797+0000 7fcd2e59c640 1 -- 172.21.15.46:0/3969823572 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ec734e9db0 msgr2=0x55ec73501b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:12:50.802 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:50.797+0000 7fcd2e59c640 1 --2- 172.21.15.46:0/3969823572 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ec734e9db0 0x55ec73501b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:50.802 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:50.797+0000 7fcd2e59c640 1 -- 172.21.15.46:0/3969823572 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55ec734e0fe0 msgr2=0x55ec734f5780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:12:50.802 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:50.797+0000 7fcd2e59c640 1 --2- 172.21.15.46:0/3969823572 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55ec734e0fe0 0x55ec734f5780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:50.802 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:50.797+0000 7fcd2e59c640 1 -- 172.21.15.46:0/3969823572 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ec734fc580 con 0x55ec734e06c0 2024-03-31T23:12:50.802 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:50.797+0000 7fcd2ed9d640 1 --2- 172.21.15.46:0/3969823572 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ec734e9db0 0x55ec73501b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T23:12:50.802 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:50.798+0000 7fcd2e59c640 1 --2- 172.21.15.46:0/3969823572 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ec734e06c0 0x55ec734e9530 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7fcd200b97f0 tx=0x7fcd200bcd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=367662ea52d75273 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:12:50.802 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:50.798+0000 7fcd17fff640 1 -- 172.21.15.46:0/3969823572 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd2002cc20 con 0x55ec734e06c0 2024-03-31T23:12:50.802 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:50.798+0000 7fcd17fff640 1 -- 172.21.15.46:0/3969823572 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fcd200bda90 con 0x55ec734e06c0 2024-03-31T23:12:50.802 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:50.798+0000 7fcd17fff640 1 -- 172.21.15.46:0/3969823572 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd200c1170 con 0x55ec734e06c0 2024-03-31T23:12:50.802 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:50.798+0000 7fcd37b98740 1 -- 172.21.15.46:0/3969823572 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ec734e06c0 msgr2=0x55ec734e9530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:12:50.802 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:50.798+0000 7fcd37b98740 1 --2- 172.21.15.46:0/3969823572 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ec734e06c0 0x55ec734e9530 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7fcd200b97f0 tx=0x7fcd200bcd50 comp rx=0 tx=0).stop 2024-03-31T23:12:50.802 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:50.798+0000 7fcd37b98740 1 -- 172.21.15.46:0/3969823572 shutdown_connections 2024-03-31T23:12:50.803 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:50.798+0000 7fcd37b98740 1 --2- 172.21.15.46:0/3969823572 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ec734e9db0 0x55ec73501b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:50.803 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:50.798+0000 7fcd37b98740 1 --2- 172.21.15.46:0/3969823572 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55ec734e0fe0 0x55ec734f5780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:50.803 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:50.798+0000 7fcd37b98740 1 --2- 172.21.15.46:0/3969823572 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ec734e06c0 0x55ec734e9530 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:50.803 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:50.798+0000 7fcd37b98740 1 -- 172.21.15.46:0/3969823572 >> 172.21.15.46:0/3969823572 conn(0x55ec734dafe0 msgr2=0x55ec734db3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:12:50.803 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:50.798+0000 7fcd37b98740 1 -- 172.21.15.46:0/3969823572 shutdown_connections 2024-03-31T23:12:50.803 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:50.798+0000 7fcd37b98740 1 -- 172.21.15.46:0/3969823572 wait complete. 2024-03-31T23:12:50.803 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:50.799+0000 7fcd37b98740 1 -- start start 2024-03-31T23:12:50.803 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:50.799+0000 7fcd37b98740 1 Processor -- start 2024-03-31T23:12:50.803 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:50.799+0000 7fcd37b98740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55ec734e06c0 0x55ec734e3c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:50.803 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:50.799+0000 7fcd37b98740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ec734e0fe0 0x55ec734e41c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:50.803 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:50.799+0000 7fcd37b98740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ec734e9db0 0x55ec734e4720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:50.803 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:50.799+0000 7fcd37b98740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fcd301ad080 con 0x55ec734e0fe0 2024-03-31T23:12:50.803 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:50.799+0000 7fcd37b98740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fcd301ae4f0 con 0x55ec734e06c0 2024-03-31T23:12:50.803 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:50.799+0000 7fcd37b98740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fcd301ad750 con 0x55ec734e9db0 2024-03-31T23:12:50.803 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:50.799+0000 7fcd37b98740 10 client.0 _ll_register_callbacks cb 0x7fcd28011580 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T23:12:50.803 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:50.799+0000 7fcd25d9b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ec734e0fe0 0x55ec734e41c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:50.804 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:50.799+0000 7fcd25d9b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ec734e0fe0 0x55ec734e41c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:36160/0 (socket says 172.21.15.46:36160) 2024-03-31T23:12:50.804 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:50.799+0000 7fcd25d9b640 1 -- 172.21.15.46:0/238872812 learned_addr learned my addr 172.21.15.46:0/238872812 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:12:50.804 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:50.799+0000 7fcd2ed9d640 1 --2- 172.21.15.46:0/238872812 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ec734e9db0 0x55ec734e4720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:50.804 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:50.799+0000 7fcd25d9b640 1 -- 172.21.15.46:0/238872812 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ec734e9db0 msgr2=0x55ec734e4720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:12:50.804 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:50.799+0000 7fcd25d9b640 1 --2- 172.21.15.46:0/238872812 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ec734e9db0 0x55ec734e4720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:50.804 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:50.799+0000 7fcd25d9b640 1 -- 172.21.15.46:0/238872812 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55ec734e06c0 msgr2=0x55ec734e3c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:12:50.804 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:50.799+0000 7fcd25d9b640 1 --2- 172.21.15.46:0/238872812 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55ec734e06c0 0x55ec734e3c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:50.804 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:50.799+0000 7fcd25d9b640 1 -- 172.21.15.46:0/238872812 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ec7355f420 con 0x55ec734e0fe0 2024-03-31T23:12:50.804 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:50.799+0000 7fcd25d9b640 1 --2- 172.21.15.46:0/238872812 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ec734e0fe0 0x55ec734e41c0 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7fcd18009c60 tx=0x7fcd180045f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f7ce54ca370bf25c server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:12:50.804 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:50.799+0000 7fcd15ffb640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-31T23:12:50.804 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:50.800+0000 7fcd15ffb640 1 -- 172.21.15.46:0/238872812 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd18028070 con 0x55ec734e0fe0 2024-03-31T23:12:50.804 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:50.800+0000 7fcd15ffb640 1 -- 172.21.15.46:0/238872812 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7fcd180035f0 con 0x55ec734e0fe0 2024-03-31T23:12:50.804 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:50.800+0000 7fcd37b98740 1 -- 172.21.15.46:0/238872812 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcd200bda90 con 0x55ec734e0fe0 2024-03-31T23:12:50.804 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:50.800+0000 7fcd15ffb640 1 -- 172.21.15.46:0/238872812 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fcd18034050 con 0x55ec734e0fe0 2024-03-31T23:12:50.804 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:50.800+0000 7fcd37b98740 10 client.16053 fetch_fsmap learned FSMap version 25 2024-03-31T23:12:50.804 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:50.800+0000 7fcd37b98740 1 -- 172.21.15.46:0/238872812 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55ec73560730 con 0x55ec734e0fe0 2024-03-31T23:12:50.805 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:50.800+0000 7fcd15ffb640 1 -- 172.21.15.46:0/238872812 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fcd200bda90 con 0x55ec734e0fe0 2024-03-31T23:12:50.805 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:50.800+0000 7fcd15ffb640 1 -- 172.21.15.46:0/238872812 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7fcd18018070 con 0x55ec734e0fe0 2024-03-31T23:12:50.805 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:50.800+0000 7fcd37b98740 10 client.16053 fetch_fsmap finished waiting for FSMap version 25 2024-03-31T23:12:50.805 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:50.800+0000 7fcd37b98740 10 client.16053 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-31T23:12:50.805 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:50.800+0000 7fcd37b98740 20 client.16053 populate_metadata read hostname 'smithi046' 2024-03-31T23:12:50.805 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:50.800+0000 7fcd37b98740 1 --2- 172.21.15.46:0/238872812 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55ec73560c00 0x55ec73562ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:12:50.805 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:50.800+0000 7fcd37b98740 4 client.16053 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-31T23:12:50.805 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:50.800+0000 7fcd37b98740 1 -- 172.21.15.46:0/238872812 --> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55ec73450e30 con 0x55ec73560c00 2024-03-31T23:12:50.805 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:50.800+0000 7fcd2e59c640 1 --2- 172.21.15.46:0/238872812 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55ec73560c00 0x55ec73562ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:12:50.805 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:50.801+0000 7fcd2e59c640 1 --2- 172.21.15.46:0/238872812 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55ec73560c00 0x55ec73562ff0 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=244f849c2fd8d1a3 server_cookie=4623b1990a33efc5 in_seq=0 out_seq=0 2024-03-31T23:12:50.805 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:50.801+0000 7fcd15ffb640 10 client.16053 ms_handle_connect on v2:172.21.15.142:6838/2519143918 2024-03-31T23:12:50.806 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:50.804+0000 7fcd15ffb640 1 -- 172.21.15.46:0/238872812 <== mds.0 v2:172.21.15.142:6838/2519143918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55ec73450e30 con 0x55ec73560c00 2024-03-31T23:12:50.806 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:50.804+0000 7fcd15ffb640 10 client.16053 handle_command_reply: tid=0 2024-03-31T23:12:50.807 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:50.804+0000 7fcd37b98740 1 client.16053 shutdown 2024-03-31T23:12:50.807 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:50.804+0000 7fcd37b98740 1 -- 172.21.15.46:0/238872812 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ec734e0fe0 msgr2=0x55ec734e41c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:12:50.807 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:50.804+0000 7fcd37b98740 1 --2- 172.21.15.46:0/238872812 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ec734e0fe0 0x55ec734e41c0 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7fcd18009c60 tx=0x7fcd180045f0 comp rx=0 tx=0).stop 2024-03-31T23:12:50.807 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:50.804+0000 7fcd37b98740 1 -- 172.21.15.46:0/238872812 shutdown_connections 2024-03-31T23:12:50.807 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:50.804+0000 7fcd37b98740 1 --2- 172.21.15.46:0/238872812 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55ec73560c00 0x55ec73562ff0 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:50.807 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:50.804+0000 7fcd37b98740 1 --2- 172.21.15.46:0/238872812 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ec734e9db0 0x55ec734e4720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:50.807 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:50.804+0000 7fcd37b98740 1 --2- 172.21.15.46:0/238872812 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ec734e0fe0 0x55ec734e41c0 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:50.807 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:50.804+0000 7fcd15ffb640 0 client.16053 ms_handle_reset on v2:172.21.15.142:6838/2519143918 2024-03-31T23:12:50.807 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:50.804+0000 7fcd37b98740 1 --2- 172.21.15.46:0/238872812 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55ec734e06c0 0x55ec734e3c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:50.807 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:50.804+0000 7fcd37b98740 1 -- 172.21.15.46:0/238872812 >> 172.21.15.46:0/238872812 conn(0x55ec734dafe0 msgr2=0x55ec734d96f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:12:50.807 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:50.804+0000 7fcd37b98740 1 -- 172.21.15.46:0/238872812 shutdown_connections 2024-03-31T23:12:50.807 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:50.804+0000 7fcd37b98740 1 -- 172.21.15.46:0/238872812 wait complete. 2024-03-31T23:12:50.807 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:50.804+0000 7fcd37b98740 20 client.16053 trim_cache size 0 max 16384 2024-03-31T23:12:50.808 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout:{ 2024-03-31T23:12:50.808 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout: "message": "", 2024-03-31T23:12:50.808 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout: "return_code": 0 2024-03-31T23:12:50.808 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout:} 2024-03-31T23:12:50.808 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:50.804+0000 7fcd34ed4640 1 -- 172.21.15.46:0/3462612284 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fcd1c0a27d0 msgr2=0x7fcd1c0a4c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:12:50.809 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:50.804+0000 7fcd34ed4640 1 --2- 172.21.15.46:0/3462612284 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fcd1c0a27d0 0x7fcd1c0a4c90 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fcd28009fd0 tx=0x7fcd28011f10 comp rx=0 tx=0).stop 2024-03-31T23:12:50.809 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:50.804+0000 7fcd34ed4640 1 -- 172.21.15.46:0/3462612284 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcd300ffe60 msgr2=0x7fcd30197e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:12:50.809 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:50.804+0000 7fcd34ed4640 1 --2- 172.21.15.46:0/3462612284 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcd300ffe60 0x7fcd30197e70 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7fcd20005940 tx=0x7fcd2002c5f0 comp rx=0 tx=0).stop 2024-03-31T23:12:50.809 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:50.804+0000 7fcd34ed4640 1 -- 172.21.15.46:0/3462612284 shutdown_connections 2024-03-31T23:12:50.809 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:50.804+0000 7fcd34ed4640 1 --2- 172.21.15.46:0/3462612284 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fcd1c0a27d0 0x7fcd1c0a4c90 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:50.809 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:50.804+0000 7fcd34ed4640 1 --2- 172.21.15.46:0/3462612284 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcd3019c730 0x7fcd3019cbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:50.809 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:50.804+0000 7fcd34ed4640 1 --2- 172.21.15.46:0/3462612284 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcd30106980 0x7fcd301983b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:50.809 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:50.805+0000 7fcd34ed4640 1 --2- 172.21.15.46:0/3462612284 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcd300ffe60 0x7fcd30197e70 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:12:50.809 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:50.805+0000 7fcd34ed4640 1 -- 172.21.15.46:0/3462612284 >> 172.21.15.46:0/3462612284 conn(0x7fcd30076b70 msgr2=0x7fcd300fdf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:12:50.809 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:50.805+0000 7fcd34ed4640 1 -- 172.21.15.46:0/3462612284 shutdown_connections 2024-03-31T23:12:50.809 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:12:50.805+0000 7fcd34ed4640 1 -- 172.21.15.46:0/3462612284 wait complete. 2024-03-31T23:12:52.407 INFO:tasks.workunit.client.0.smithi046.stderr:Updating files: 95% (11854/12435) Updating files: 96% (11938/12435) Updating files: 97% (12062/12435) Updating files: 98% (12187/12435) Updating files: 99% (12311/12435) Updating files: 100% (12435/12435) Updating files: 100% (12435/12435), done. 2024-03-31T23:12:52.611 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:12:52 smithi046 ceph-mon[32498]: pgmap v37: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 0 B/s wr, 6 op/s 2024-03-31T23:12:52.693 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:12:52 smithi142 ceph-mon[35271]: pgmap v37: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 0 B/s wr, 6 op/s 2024-03-31T23:12:52.795 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:12:52 smithi162 ceph-mon[36237]: pgmap v37: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 0 B/s wr, 6 op/s 2024-03-31T23:12:52.959 INFO:tasks.workunit.client.0.smithi046.stderr:Note: switching to '384f31f3d3aee7f8ba3f60846b8844c049165583'. 2024-03-31T23:12:52.959 INFO:tasks.workunit.client.0.smithi046.stderr: 2024-03-31T23:12:52.959 INFO:tasks.workunit.client.0.smithi046.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-03-31T23:12:52.959 INFO:tasks.workunit.client.0.smithi046.stderr:changes and commit them, and you can discard any commits you make in this 2024-03-31T23:12:52.959 INFO:tasks.workunit.client.0.smithi046.stderr:state without impacting any branches by switching back to a branch. 2024-03-31T23:12:52.959 INFO:tasks.workunit.client.0.smithi046.stderr: 2024-03-31T23:12:52.959 INFO:tasks.workunit.client.0.smithi046.stderr:If you want to create a new branch to retain commits you create, you may 2024-03-31T23:12:52.959 INFO:tasks.workunit.client.0.smithi046.stderr:do so (now or later) by using -c with the switch command. Example: 2024-03-31T23:12:52.959 INFO:tasks.workunit.client.0.smithi046.stderr: 2024-03-31T23:12:52.959 INFO:tasks.workunit.client.0.smithi046.stderr: git switch -c 2024-03-31T23:12:52.959 INFO:tasks.workunit.client.0.smithi046.stderr: 2024-03-31T23:12:52.959 INFO:tasks.workunit.client.0.smithi046.stderr:Or undo this operation with: 2024-03-31T23:12:52.959 INFO:tasks.workunit.client.0.smithi046.stderr: 2024-03-31T23:12:52.959 INFO:tasks.workunit.client.0.smithi046.stderr: git switch - 2024-03-31T23:12:52.960 INFO:tasks.workunit.client.0.smithi046.stderr: 2024-03-31T23:12:52.960 INFO:tasks.workunit.client.0.smithi046.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-03-31T23:12:52.960 INFO:tasks.workunit.client.0.smithi046.stderr: 2024-03-31T23:12:52.960 INFO:tasks.workunit.client.0.smithi046.stderr:HEAD is now at 384f31f3d3a Merge pull request #56593 from zdover23/wip-doc-2024-03-31-backport-56592-to-squid 2024-03-31T23:12:52.962 DEBUG:teuthology.orchestra.run.smithi046:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-03-31T23:12:53.026 INFO:tasks.workunit.client.0.smithi046.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-03-31T23:12:53.028 INFO:tasks.workunit.client.0.smithi046.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-03-31T23:12:53.028 INFO:tasks.workunit.client.0.smithi046.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-03-31T23:12:53.800 INFO:tasks.workunit.client.0.smithi046.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-03-31T23:12:53.870 INFO:tasks.workunit.client.0.smithi046.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-03-31T23:12:53.911 INFO:tasks.workunit.client.0.smithi046.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-03-31T23:12:53.913 INFO:tasks.workunit.client.0.smithi046.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-03-31T23:12:53.913 INFO:tasks.workunit.client.0.smithi046.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-03-31T23:12:53.951 INFO:tasks.workunit.client.0.smithi046.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-03-31T23:12:53.955 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-03-31T23:12:53.955 DEBUG:teuthology.orchestra.run.smithi046:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-03-31T23:12:54.010 INFO:tasks.workunit:Running workunits matching suites/dbench.sh on client.0... 2024-03-31T23:12:54.011 INFO:tasks.workunit:Running workunit suites/dbench.sh... 2024-03-31T23:12:54.011 DEBUG:teuthology.orchestra.run.smithi046:workunit test suites/dbench.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=384f31f3d3aee7f8ba3f60846b8844c049165583 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/dbench.sh 2024-03-31T23:12:54.073 INFO:tasks.workunit.client.0.smithi046.stderr:+ dbench 1 2024-03-31T23:12:54.082 INFO:tasks.workunit.client.0.smithi046.stdout:dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 2024-03-31T23:12:54.082 INFO:tasks.workunit.client.0.smithi046.stdout: 2024-03-31T23:12:54.082 INFO:tasks.workunit.client.0.smithi046.stdout:Running for 600 seconds with load '/usr/share/dbench/client.txt' and minimum warmup 120 secs 2024-03-31T23:12:54.089 INFO:tasks.workunit.client.0.smithi046.stdout:0 of 1 processes prepared for launch 0 sec 2024-03-31T23:12:54.189 INFO:tasks.workunit.client.0.smithi046.stdout:1 of 1 processes prepared for launch 0 sec 2024-03-31T23:12:54.189 INFO:tasks.workunit.client.0.smithi046.stdout:releasing clients 2024-03-31T23:12:54.611 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:12:54 smithi046 ceph-mon[32498]: pgmap v38: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 6 op/s 2024-03-31T23:12:54.693 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:12:54 smithi142 ceph-mon[35271]: pgmap v38: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 6 op/s 2024-03-31T23:12:54.795 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:12:54 smithi162 ceph-mon[36237]: pgmap v38: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 6 op/s 2024-03-31T23:12:55.189 INFO:tasks.workunit.client.0.smithi046.stdout: 1 888 21.68 MB/sec warmup 1 sec latency 15.057 ms 2024-03-31T23:12:56.189 INFO:tasks.workunit.client.0.smithi046.stdout: 1 1555 13.05 MB/sec warmup 2 sec latency 16.680 ms 2024-03-31T23:12:56.611 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:12:56 smithi046 ceph-mon[32498]: pgmap v39: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 255 B/s wr, 8 op/s 2024-03-31T23:12:56.693 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:12:56 smithi142 ceph-mon[35271]: pgmap v39: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 255 B/s wr, 8 op/s 2024-03-31T23:12:56.795 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:12:56 smithi162 ceph-mon[36237]: pgmap v39: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 255 B/s wr, 8 op/s 2024-03-31T23:12:57.189 INFO:tasks.workunit.client.0.smithi046.stdout: 1 2247 8.93 MB/sec warmup 3 sec latency 15.890 ms 2024-03-31T23:12:58.189 INFO:tasks.workunit.client.0.smithi046.stdout: 1 2982 8.38 MB/sec warmup 4 sec latency 19.220 ms 2024-03-31T23:12:58.611 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:12:58 smithi046 ceph-mon[32498]: pgmap v40: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-03-31T23:12:58.693 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:12:58 smithi142 ceph-mon[35271]: pgmap v40: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-03-31T23:12:58.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:12:58 smithi162 ceph-mon[36237]: pgmap v40: 129 pgs: 129 active+clean; 1.3 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-03-31T23:12:59.189 INFO:tasks.workunit.client.0.smithi046.stdout: 1 3557 7.68 MB/sec warmup 5 sec latency 16.981 ms 2024-03-31T23:13:00.189 INFO:tasks.workunit.client.0.smithi046.stdout: 1 4027 6.68 MB/sec warmup 6 sec latency 14.810 ms 2024-03-31T23:13:00.611 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:13:00 smithi046 ceph-mon[32498]: pgmap v41: 129 pgs: 129 active+clean; 25 MiB data, 423 MiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 2.2 MiB/s wr, 42 op/s 2024-03-31T23:13:00.693 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:13:00 smithi142 ceph-mon[35271]: pgmap v41: 129 pgs: 129 active+clean; 25 MiB data, 423 MiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 2.2 MiB/s wr, 42 op/s 2024-03-31T23:13:00.795 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:13:00 smithi162 ceph-mon[36237]: pgmap v41: 129 pgs: 129 active+clean; 25 MiB data, 423 MiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 2.2 MiB/s wr, 42 op/s 2024-03-31T23:13:01.190 INFO:tasks.workunit.client.0.smithi046.stdout: 1 4473 5.83 MB/sec warmup 7 sec latency 23.044 ms 2024-03-31T23:13:02.190 INFO:tasks.workunit.client.0.smithi046.stdout: 1 5091 5.65 MB/sec warmup 8 sec latency 16.516 ms 2024-03-31T23:13:02.693 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:13:02 smithi142 ceph-mon[35271]: pgmap v42: 129 pgs: 129 active+clean; 25 MiB data, 423 MiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 2.2 MiB/s wr, 39 op/s 2024-03-31T23:13:02.795 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:13:02 smithi162 ceph-mon[36237]: pgmap v42: 129 pgs: 129 active+clean; 25 MiB data, 423 MiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 2.2 MiB/s wr, 39 op/s 2024-03-31T23:13:02.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:13:02 smithi046 ceph-mon[32498]: pgmap v42: 129 pgs: 129 active+clean; 25 MiB data, 423 MiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 2.2 MiB/s wr, 39 op/s 2024-03-31T23:13:03.190 INFO:tasks.workunit.client.0.smithi046.stdout: 1 5666 5.09 MB/sec warmup 9 sec latency 26.401 ms 2024-03-31T23:13:03.693 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:13:03 smithi142 ceph-mon[35271]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-03-31T23:13:03.795 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:13:03 smithi162 ceph-mon[36237]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-03-31T23:13:03.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:13:03 smithi046 ceph-mon[32498]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-03-31T23:13:03.903 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:03.902+0000 7f386337a640 1 -- 172.21.15.46:0/380190120 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f385c101260 msgr2=0x7f385c10d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:13:03.903 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:03.902+0000 7f386337a640 1 --2- 172.21.15.46:0/380190120 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f385c101260 0x7f385c10d480 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f3854009ca0 tx=0x7f385402c0f0 comp rx=0 tx=0).stop 2024-03-31T23:13:03.903 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:03.902+0000 7f386337a640 1 -- 172.21.15.46:0/380190120 shutdown_connections 2024-03-31T23:13:03.903 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:03.902+0000 7f386337a640 1 --2- 172.21.15.46:0/380190120 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f385c101260 0x7f385c10d480 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:03.903 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:03.902+0000 7f386337a640 1 --2- 172.21.15.46:0/380190120 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f385c1008a0 0x7f385c100d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:03.903 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:03.902+0000 7f386337a640 1 --2- 172.21.15.46:0/380190120 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f385c1055f0 0x7f385c1059d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:03.903 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:03.902+0000 7f386337a640 1 -- 172.21.15.46:0/380190120 >> 172.21.15.46:0/380190120 conn(0x7f385c0774a0 msgr2=0x7f385c0778b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:13:03.904 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:03.902+0000 7f386337a640 1 -- 172.21.15.46:0/380190120 shutdown_connections 2024-03-31T23:13:03.904 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:03.902+0000 7f386337a640 1 -- 172.21.15.46:0/380190120 wait complete. 2024-03-31T23:13:03.905 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:03.902+0000 7f386337a640 1 Processor -- start 2024-03-31T23:13:03.906 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:03.903+0000 7f386337a640 1 -- start start 2024-03-31T23:13:03.906 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:03.903+0000 7f386337a640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f385c1008a0 0x7f385c196b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:13:03.906 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:03.903+0000 7f386337a640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f385c101260 0x7f385c197050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:13:03.906 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:03.903+0000 7f386337a640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f385c1055f0 0x7f385c197590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:13:03.906 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:03.903+0000 7f386337a640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f385c1109c0 con 0x7f385c101260 2024-03-31T23:13:03.906 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:03.903+0000 7f386337a640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f385c110840 con 0x7f385c1008a0 2024-03-31T23:13:03.906 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:03.903+0000 7f386337a640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f385c110b40 con 0x7f385c1055f0 2024-03-31T23:13:03.906 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:03.903+0000 7f3860915640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f385c101260 0x7f385c197050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:13:03.906 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:03.903+0000 7f3861116640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f385c1008a0 0x7f385c196b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:13:03.906 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:03.903+0000 7f3860915640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f385c101260 0x7f385c197050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:48802/0 (socket says 172.21.15.46:48802) 2024-03-31T23:13:03.906 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:03.903+0000 7f3860915640 1 -- 172.21.15.46:0/3402336549 learned_addr learned my addr 172.21.15.46:0/3402336549 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:13:03.906 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:03.903+0000 7f3861917640 1 --2- 172.21.15.46:0/3402336549 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f385c1055f0 0x7f385c197590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:13:03.906 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:03.903+0000 7f3861116640 1 -- 172.21.15.46:0/3402336549 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f385c1055f0 msgr2=0x7f385c197590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:13:03.907 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:03.903+0000 7f3861116640 1 --2- 172.21.15.46:0/3402336549 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f385c1055f0 0x7f385c197590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:03.907 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:03.903+0000 7f3861116640 1 -- 172.21.15.46:0/3402336549 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f385c101260 msgr2=0x7f385c197050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:13:03.907 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:03.903+0000 7f3861116640 1 --2- 172.21.15.46:0/3402336549 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f385c101260 0x7f385c197050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:03.907 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:03.903+0000 7f3861116640 1 -- 172.21.15.46:0/3402336549 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f385c1a6380 con 0x7f385c1008a0 2024-03-31T23:13:03.907 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:03.903+0000 7f3860915640 1 --2- 172.21.15.46:0/3402336549 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f385c101260 0x7f385c197050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:13:03.907 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:03.903+0000 7f3861116640 1 --2- 172.21.15.46:0/3402336549 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f385c1008a0 0x7f385c196b10 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f385800e9f0 tx=0x7f385800eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:13:03.907 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:03.904+0000 7f38527fc640 1 -- 172.21.15.46:0/3402336549 <== mon.2 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f385800ccc0 con 0x7f385c1008a0 2024-03-31T23:13:03.907 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:03.904+0000 7f38527fc640 1 -- 172.21.15.46:0/3402336549 <== mon.2 v2:172.21.15.142:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f38580108e0 con 0x7f385c1008a0 2024-03-31T23:13:03.907 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:03.904+0000 7f386337a640 1 -- 172.21.15.46:0/3402336549 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f385c1a6670 con 0x7f385c1008a0 2024-03-31T23:13:03.907 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:03.904+0000 7f38527fc640 1 -- 172.21.15.46:0/3402336549 <== mon.2 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3858018880 con 0x7f385c1008a0 2024-03-31T23:13:03.907 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:03.904+0000 7f386337a640 1 -- 172.21.15.46:0/3402336549 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f385c1a6b90 con 0x7f385c1008a0 2024-03-31T23:13:03.914 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:03.906+0000 7f3865bfd740 1 Processor -- start 2024-03-31T23:13:03.914 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:03.906+0000 7f38527fc640 1 -- 172.21.15.46:0/3402336549 <== mon.2 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 24) ==== 137553+0+0 (secure 0 0 0) 0x7f3858010a90 con 0x7f385c1008a0 2024-03-31T23:13:03.914 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:03.906+0000 7f3865bfd740 1 -- start start 2024-03-31T23:13:03.914 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:03.906+0000 7f3865bfd740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562b08c4d6c0 0x562b08c4daa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:13:03.914 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:03.907+0000 7f3865bfd740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x562b08c4dfe0 0x562b08c56870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:13:03.914 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:03.907+0000 7f3865bfd740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562b08c56db0 0x562b08c591a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:13:03.914 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:03.907+0000 7f3865bfd740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x562b08bb08f0 con 0x562b08c4d6c0 2024-03-31T23:13:03.915 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:03.907+0000 7f3865bfd740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x562b08b598d0 con 0x562b08c4dfe0 2024-03-31T23:13:03.915 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:03.907+0000 7f3865bfd740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x562b08bac990 con 0x562b08c56db0 2024-03-31T23:13:03.915 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:03.907+0000 7f3861917640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562b08c4d6c0 0x562b08c4daa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:13:03.915 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:03.907+0000 7f3861917640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562b08c4d6c0 0x562b08c4daa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:48810/0 (socket says 172.21.15.46:48810) 2024-03-31T23:13:03.915 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:03.907+0000 7f3861116640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x562b08c4dfe0 0x562b08c56870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:13:03.915 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:03.907+0000 7f3861917640 1 -- 172.21.15.46:0/2425335595 learned_addr learned my addr 172.21.15.46:0/2425335595 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:13:03.915 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:03.907+0000 7f3860915640 1 --2- 172.21.15.46:0/2425335595 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562b08c56db0 0x562b08c591a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:13:03.915 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:03.907+0000 7f38527fc640 1 --2- 172.21.15.46:0/3402336549 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f38480a27d0 0x7f38480a4c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:13:03.915 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:03.907+0000 7f3860915640 1 -- 172.21.15.46:0/2425335595 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x562b08c4dfe0 msgr2=0x562b08c56870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:13:03.915 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:03.907+0000 7f3860915640 1 --2- 172.21.15.46:0/2425335595 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x562b08c4dfe0 0x562b08c56870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:03.915 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:03.907+0000 7f3860915640 1 -- 172.21.15.46:0/2425335595 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562b08c4d6c0 msgr2=0x562b08c4daa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:13:03.915 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:03.907+0000 7f3860915640 1 --2- 172.21.15.46:0/2425335595 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562b08c4d6c0 0x562b08c4daa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:03.915 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:03.907+0000 7f3860915640 1 -- 172.21.15.46:0/2425335595 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562b088c5210 con 0x562b08c56db0 2024-03-31T23:13:03.916 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:03.907+0000 7f3861917640 1 --2- 172.21.15.46:0/3402336549 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f38480a27d0 0x7f38480a4c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:13:03.916 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:03.907+0000 7f38527fc640 1 -- 172.21.15.46:0/3402336549 <== mon.2 v2:172.21.15.142:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7f38580cf420 con 0x7f385c1008a0 2024-03-31T23:13:03.916 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:03.907+0000 7f3860915640 1 --2- 172.21.15.46:0/2425335595 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562b08c56db0 0x562b08c591a0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f385c198c70 tx=0x7f384c0119a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e5a71fb5678c8705 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:13:03.916 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:03.907+0000 7f38477fe640 1 -- 172.21.15.46:0/2425335595 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f384c01e070 con 0x562b08c56db0 2024-03-31T23:13:03.916 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:03.908+0000 7f38477fe640 1 -- 172.21.15.46:0/2425335595 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f384c00fa70 con 0x562b08c56db0 2024-03-31T23:13:03.916 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:03.908+0000 7f38477fe640 1 -- 172.21.15.46:0/2425335595 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f384c019b60 con 0x562b08c56db0 2024-03-31T23:13:03.916 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:03.908+0000 7f3865bfd740 1 -- 172.21.15.46:0/2425335595 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562b08c56db0 msgr2=0x562b08c591a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:13:03.916 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:03.908+0000 7f3861917640 1 --2- 172.21.15.46:0/3402336549 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f38480a27d0 0x7f38480a4c90 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f3854037040 tx=0x7f385402e040 comp rx=0 tx=0).ready entity=mgr.15654 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:13:03.916 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:03.908+0000 7f3865bfd740 1 --2- 172.21.15.46:0/2425335595 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562b08c56db0 0x562b08c591a0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f385c198c70 tx=0x7f384c0119a0 comp rx=0 tx=0).stop 2024-03-31T23:13:03.916 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:03.908+0000 7f3865bfd740 1 -- 172.21.15.46:0/2425335595 shutdown_connections 2024-03-31T23:13:03.916 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:03.908+0000 7f3865bfd740 1 --2- 172.21.15.46:0/2425335595 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562b08c56db0 0x562b08c591a0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:03.916 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:03.908+0000 7f3865bfd740 1 --2- 172.21.15.46:0/2425335595 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x562b08c4dfe0 0x562b08c56870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:03.916 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:03.908+0000 7f3865bfd740 1 --2- 172.21.15.46:0/2425335595 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562b08c4d6c0 0x562b08c4daa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:03.917 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:03.908+0000 7f3865bfd740 1 -- 172.21.15.46:0/2425335595 >> 172.21.15.46:0/2425335595 conn(0x562b08c47fe0 msgr2=0x562b08c483c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:13:03.917 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:03.908+0000 7f3865bfd740 1 -- 172.21.15.46:0/2425335595 shutdown_connections 2024-03-31T23:13:03.917 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:03.908+0000 7f3865bfd740 1 -- 172.21.15.46:0/2425335595 wait complete. 2024-03-31T23:13:03.917 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:03.909+0000 7f3865bfd740 1 -- start start 2024-03-31T23:13:03.917 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:03.909+0000 7f3865bfd740 1 Processor -- start 2024-03-31T23:13:03.917 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:03.909+0000 7f3865bfd740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562b08c4d6c0 0x562b08c53ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:13:03.917 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:03.909+0000 7f3865bfd740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562b08c4dfe0 0x562b08c543e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:13:03.917 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:03.909+0000 7f3865bfd740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x562b08c56db0 0x562b08c54920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:13:03.917 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:03.909+0000 7f3865bfd740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x562b08c599e0 con 0x562b08c4dfe0 2024-03-31T23:13:03.917 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:03.909+0000 7f3865bfd740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x562b08b4ef80 con 0x562b08c56db0 2024-03-31T23:13:03.917 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:03.909+0000 7f3865bfd740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x562b08c59b60 con 0x562b08c4d6c0 2024-03-31T23:13:03.917 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:03.909+0000 7f3865bfd740 10 client.0 _ll_register_callbacks cb 0x562b08ba7290 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T23:13:03.917 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:03.909+0000 7f3861116640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562b08c4d6c0 0x562b08c53ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:13:03.918 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:03.909+0000 7f3860915640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562b08c4dfe0 0x562b08c543e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:13:03.918 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:03.909+0000 7f3861116640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562b08c4d6c0 0x562b08c53ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.46:40696/0 (socket says 172.21.15.46:40696) 2024-03-31T23:13:03.918 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:03.909+0000 7f3861116640 1 -- 172.21.15.46:0/3010386316 learned_addr learned my addr 172.21.15.46:0/3010386316 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:13:03.918 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:03.910+0000 7f3861917640 1 --2- 172.21.15.46:0/3010386316 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x562b08c56db0 0x562b08c54920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:13:03.918 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:03.910+0000 7f3860915640 1 -- 172.21.15.46:0/3010386316 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562b08c4d6c0 msgr2=0x562b08c53ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:13:03.918 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:03.910+0000 7f3860915640 1 --2- 172.21.15.46:0/3010386316 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562b08c4d6c0 0x562b08c53ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:03.918 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:03.910+0000 7f3860915640 1 -- 172.21.15.46:0/3010386316 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x562b08c56db0 msgr2=0x562b08c54920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:13:03.918 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:03.910+0000 7f3860915640 1 --2- 172.21.15.46:0/3010386316 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x562b08c56db0 0x562b08c54920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:03.918 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:03.910+0000 7f3860915640 1 -- 172.21.15.46:0/3010386316 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562b08991620 con 0x562b08c4dfe0 2024-03-31T23:13:03.918 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:03.910+0000 7f3861116640 1 --2- 172.21.15.46:0/3010386316 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562b08c4d6c0 0x562b08c53ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:13:03.918 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:03.910+0000 7f3860915640 1 --2- 172.21.15.46:0/3010386316 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562b08c4dfe0 0x562b08c543e0 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f384c011ea0 tx=0x7f384c019640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d23d75fb39273558 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:13:03.918 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:03.910+0000 7f3844ff9640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-31T23:13:03.919 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:03.910+0000 7f3844ff9640 1 -- 172.21.15.46:0/3010386316 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f384c01e040 con 0x562b08c4dfe0 2024-03-31T23:13:03.919 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:03.910+0000 7f3844ff9640 1 -- 172.21.15.46:0/3010386316 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7f384c016040 con 0x562b08c4dfe0 2024-03-31T23:13:03.919 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:03.910+0000 7f3865bfd740 1 -- 172.21.15.46:0/3010386316 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f384c00fa70 con 0x562b08c4dfe0 2024-03-31T23:13:03.919 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:03.910+0000 7f3844ff9640 1 -- 172.21.15.46:0/3010386316 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f384c031050 con 0x562b08c4dfe0 2024-03-31T23:13:03.919 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:03.910+0000 7f3865bfd740 10 client.16065 fetch_fsmap learned FSMap version 26 2024-03-31T23:13:03.919 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:03.910+0000 7f3865bfd740 1 -- 172.21.15.46:0/3010386316 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x562b08c65970 con 0x562b08c4dfe0 2024-03-31T23:13:03.919 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:03.911+0000 7f3844ff9640 1 -- 172.21.15.46:0/3010386316 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f384c00fa70 con 0x562b08c4dfe0 2024-03-31T23:13:03.919 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:03.911+0000 7f3844ff9640 1 -- 172.21.15.46:0/3010386316 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 26) ==== 8350+0+0 (secure 0 0 0) 0x7f384c0052d0 con 0x562b08c4dfe0 2024-03-31T23:13:03.919 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:03.911+0000 7f3865bfd740 10 client.16065 fetch_fsmap finished waiting for FSMap version 26 2024-03-31T23:13:03.919 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:03.911+0000 7f3865bfd740 10 client.16065 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-31T23:13:03.919 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:03.911+0000 7f3865bfd740 20 client.16065 populate_metadata read hostname 'smithi046' 2024-03-31T23:13:03.919 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:03.911+0000 7f3865bfd740 1 --2- 172.21.15.46:0/3010386316 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x562b08c66d20 0x562b08c69110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:13:03.919 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:03.911+0000 7f3865bfd740 4 client.16065 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-31T23:13:03.919 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:03.911+0000 7f3865bfd740 1 -- 172.21.15.46:0/3010386316 --> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562b08bc4120 con 0x562b08c66d20 2024-03-31T23:13:03.919 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:03.911+0000 7f3861116640 1 --2- 172.21.15.46:0/3010386316 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x562b08c66d20 0x562b08c69110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:13:03.920 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:03.911+0000 7f3861116640 1 --2- 172.21.15.46:0/3010386316 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x562b08c66d20 0x562b08c69110 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=890193ede929e4e0 server_cookie=7a621b53145924c5 in_seq=0 out_seq=0 2024-03-31T23:13:03.920 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:03.911+0000 7f3844ff9640 10 client.16065 ms_handle_connect on v2:172.21.15.142:6838/2519143918 2024-03-31T23:13:03.920 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:03.912+0000 7f3844ff9640 1 -- 172.21.15.46:0/3010386316 <== mds.0 v2:172.21.15.142:6838/2519143918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x562b08bc4120 con 0x562b08c66d20 2024-03-31T23:13:03.920 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:03.912+0000 7f3844ff9640 10 client.16065 handle_command_reply: tid=0 2024-03-31T23:13:03.920 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:03.912+0000 7f3865bfd740 1 client.16065 shutdown 2024-03-31T23:13:03.920 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:03.912+0000 7f3865bfd740 1 -- 172.21.15.46:0/3010386316 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562b08c4dfe0 msgr2=0x562b08c543e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:13:03.920 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:03.912+0000 7f3865bfd740 1 --2- 172.21.15.46:0/3010386316 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562b08c4dfe0 0x562b08c543e0 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f384c011ea0 tx=0x7f384c019640 comp rx=0 tx=0).stop 2024-03-31T23:13:03.920 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:03.913+0000 7f3865bfd740 1 -- 172.21.15.46:0/3010386316 shutdown_connections 2024-03-31T23:13:03.920 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:03.913+0000 7f3865bfd740 1 --2- 172.21.15.46:0/3010386316 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x562b08c66d20 0x562b08c69110 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:03.920 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:03.913+0000 7f3865bfd740 1 --2- 172.21.15.46:0/3010386316 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x562b08c56db0 0x562b08c54920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:03.920 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:03.913+0000 7f3865bfd740 1 --2- 172.21.15.46:0/3010386316 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562b08c4dfe0 0x562b08c543e0 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:03.920 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:03.913+0000 7f3844ff9640 0 client.16065 ms_handle_reset on v2:172.21.15.142:6838/2519143918 2024-03-31T23:13:03.920 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:03.913+0000 7f3865bfd740 1 --2- 172.21.15.46:0/3010386316 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562b08c4d6c0 0x562b08c53ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:03.920 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:03.913+0000 7f3865bfd740 1 -- 172.21.15.46:0/3010386316 >> 172.21.15.46:0/3010386316 conn(0x562b08c47fe0 msgr2=0x562b08c49d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:13:03.920 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:03.913+0000 7f3865bfd740 1 -- 172.21.15.46:0/3010386316 shutdown_connections 2024-03-31T23:13:03.921 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:03.913+0000 7f3865bfd740 1 -- 172.21.15.46:0/3010386316 wait complete. 2024-03-31T23:13:03.921 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:03.913+0000 7f3865bfd740 20 client.16065 trim_cache size 0 max 16384 2024-03-31T23:13:03.932 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:03.929+0000 7f3865bfd740 1 Processor -- start 2024-03-31T23:13:03.932 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:03.929+0000 7f3865bfd740 1 -- start start 2024-03-31T23:13:03.932 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:03.930+0000 7f3865bfd740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562b08c4d6c0 0x562b08c56560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:13:03.932 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:03.930+0000 7f3865bfd740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562b08c4dfe0 0x562b08c62780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:13:03.932 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:03.930+0000 7f3865bfd740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x562b08c56db0 0x562b08c6e850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:13:03.932 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:03.930+0000 7f3865bfd740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f385c1b35d0 con 0x562b08c4d6c0 2024-03-31T23:13:03.932 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:03.930+0000 7f3865bfd740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f385c1b0ea0 con 0x562b08c56db0 2024-03-31T23:13:03.932 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:03.930+0000 7f3865bfd740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f385c1aecf0 con 0x562b08c4dfe0 2024-03-31T23:13:03.932 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:03.930+0000 7f3861116640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562b08c4d6c0 0x562b08c56560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:13:03.932 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:03.930+0000 7f3861116640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562b08c4d6c0 0x562b08c56560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:48836/0 (socket says 172.21.15.46:48836) 2024-03-31T23:13:03.932 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:03.930+0000 7f3861116640 1 -- 172.21.15.46:0/1977919568 learned_addr learned my addr 172.21.15.46:0/1977919568 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:13:03.933 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:03.930+0000 7f3861917640 1 --2- 172.21.15.46:0/1977919568 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x562b08c56db0 0x562b08c6e850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:13:03.933 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:03.930+0000 7f3860915640 1 --2- 172.21.15.46:0/1977919568 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562b08c4dfe0 0x562b08c62780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:13:03.933 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:03.930+0000 7f3861116640 1 -- 172.21.15.46:0/1977919568 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562b08c4dfe0 msgr2=0x562b08c62780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:13:03.933 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:03.930+0000 7f3861116640 1 --2- 172.21.15.46:0/1977919568 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562b08c4dfe0 0x562b08c62780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:03.933 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:03.930+0000 7f3861116640 1 -- 172.21.15.46:0/1977919568 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x562b08c56db0 msgr2=0x562b08c6e850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:13:03.933 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:03.930+0000 7f3861116640 1 --2- 172.21.15.46:0/1977919568 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x562b08c56db0 0x562b08c6e850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:03.933 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:03.930+0000 7f3861116640 1 -- 172.21.15.46:0/1977919568 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562b08c4e5f0 con 0x562b08c4d6c0 2024-03-31T23:13:03.933 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:03.930+0000 7f3860915640 1 --2- 172.21.15.46:0/1977919568 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562b08c4dfe0 0x562b08c62780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:13:03.933 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:03.930+0000 7f3861116640 1 --2- 172.21.15.46:0/1977919568 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562b08c4d6c0 0x562b08c56560 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f3858097fd0 tx=0x7f38580d1930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=12cdfe07d889db41 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:13:03.933 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:03.930+0000 7f3846ffd640 1 -- 172.21.15.46:0/1977919568 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f385809b190 con 0x562b08c4d6c0 2024-03-31T23:13:03.933 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:03.930+0000 7f3846ffd640 1 -- 172.21.15.46:0/1977919568 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3858010e00 con 0x562b08c4d6c0 2024-03-31T23:13:03.933 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:03.930+0000 7f3846ffd640 1 -- 172.21.15.46:0/1977919568 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f385809afb0 con 0x562b08c4d6c0 2024-03-31T23:13:03.937 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:03.931+0000 7f3865bfd740 1 -- 172.21.15.46:0/1977919568 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562b08c4d6c0 msgr2=0x562b08c56560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:13:03.937 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:03.931+0000 7f3865bfd740 1 --2- 172.21.15.46:0/1977919568 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562b08c4d6c0 0x562b08c56560 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f3858097fd0 tx=0x7f38580d1930 comp rx=0 tx=0).stop 2024-03-31T23:13:03.937 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:03.931+0000 7f3865bfd740 1 -- 172.21.15.46:0/1977919568 shutdown_connections 2024-03-31T23:13:03.937 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:03.931+0000 7f3865bfd740 1 --2- 172.21.15.46:0/1977919568 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x562b08c56db0 0x562b08c6e850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:03.937 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:03.931+0000 7f3865bfd740 1 --2- 172.21.15.46:0/1977919568 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562b08c4dfe0 0x562b08c62780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:03.937 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:03.931+0000 7f3865bfd740 1 --2- 172.21.15.46:0/1977919568 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562b08c4d6c0 0x562b08c56560 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:03.937 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:03.931+0000 7f3865bfd740 1 -- 172.21.15.46:0/1977919568 >> 172.21.15.46:0/1977919568 conn(0x562b08c47fe0 msgr2=0x562b08c483c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:13:03.937 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:03.931+0000 7f3865bfd740 1 -- 172.21.15.46:0/1977919568 shutdown_connections 2024-03-31T23:13:03.938 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:03.931+0000 7f3865bfd740 1 -- 172.21.15.46:0/1977919568 wait complete. 2024-03-31T23:13:03.938 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:03.931+0000 7f3865bfd740 1 -- start start 2024-03-31T23:13:03.938 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:03.932+0000 7f3865bfd740 1 Processor -- start 2024-03-31T23:13:03.938 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:03.932+0000 7f3865bfd740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562b08c4d6c0 0x562b08c5f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:13:03.938 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:03.932+0000 7f3865bfd740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562b08c4dfe0 0x562b08c507f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:13:03.938 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:03.932+0000 7f3865bfd740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x562b08c56db0 0x562b08c50d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:13:03.938 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:03.932+0000 7f3865bfd740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f385c1af360 con 0x562b08c4dfe0 2024-03-31T23:13:03.938 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:03.932+0000 7f3865bfd740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f385c1b07d0 con 0x562b08c56db0 2024-03-31T23:13:03.938 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:03.932+0000 7f3865bfd740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f385c1afa30 con 0x562b08c4d6c0 2024-03-31T23:13:03.938 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:03.932+0000 7f3865bfd740 10 client.0 _ll_register_callbacks cb 0x7f384c01c070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T23:13:03.938 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:03.932+0000 7f3860915640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562b08c4dfe0 0x562b08c507f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:13:03.938 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:03.932+0000 7f3861116640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562b08c4d6c0 0x562b08c5f0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:13:03.938 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:03.932+0000 7f3860915640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562b08c4dfe0 0x562b08c507f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:48838/0 (socket says 172.21.15.46:48838) 2024-03-31T23:13:03.938 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:03.932+0000 7f3860915640 1 -- 172.21.15.46:0/4028736065 learned_addr learned my addr 172.21.15.46:0/4028736065 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:13:03.939 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:03.932+0000 7f3861917640 1 --2- 172.21.15.46:0/4028736065 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x562b08c56db0 0x562b08c50d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:13:03.939 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:03.932+0000 7f3860915640 1 -- 172.21.15.46:0/4028736065 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562b08c4d6c0 msgr2=0x562b08c5f0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:13:03.939 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:03.932+0000 7f3860915640 1 --2- 172.21.15.46:0/4028736065 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562b08c4d6c0 0x562b08c5f0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:03.939 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:03.932+0000 7f3860915640 1 -- 172.21.15.46:0/4028736065 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x562b08c56db0 msgr2=0x562b08c50d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:13:03.939 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:03.932+0000 7f3860915640 1 --2- 172.21.15.46:0/4028736065 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x562b08c56db0 0x562b08c50d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:03.939 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:03.932+0000 7f3860915640 1 -- 172.21.15.46:0/4028736065 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562b08c51560 con 0x562b08c4dfe0 2024-03-31T23:13:03.939 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:03.933+0000 7f3860915640 1 --2- 172.21.15.46:0/4028736065 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562b08c4dfe0 0x562b08c507f0 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x562b08c53700 tx=0x7f384c00f3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b0a3338e9294674 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:13:03.939 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:03.933+0000 7f3844ff9640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-31T23:13:03.939 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:03.933+0000 7f3844ff9640 1 -- 172.21.15.46:0/4028736065 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f384c039020 con 0x562b08c4dfe0 2024-03-31T23:13:03.939 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:03.933+0000 7f3844ff9640 1 -- 172.21.15.46:0/4028736065 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7f384c00f680 con 0x562b08c4dfe0 2024-03-31T23:13:03.939 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:03.933+0000 7f3865bfd740 1 -- 172.21.15.46:0/4028736065 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x562b08bbde30 con 0x562b08c4dfe0 2024-03-31T23:13:03.939 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:03.933+0000 7f3844ff9640 1 -- 172.21.15.46:0/4028736065 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f384c044050 con 0x562b08c4dfe0 2024-03-31T23:13:03.939 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:03.933+0000 7f3865bfd740 10 client.16071 fetch_fsmap learned FSMap version 26 2024-03-31T23:13:03.939 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:03.933+0000 7f3865bfd740 1 -- 172.21.15.46:0/4028736065 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x562b08cccef0 con 0x562b08c4dfe0 2024-03-31T23:13:03.939 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:03.933+0000 7f3844ff9640 1 -- 172.21.15.46:0/4028736065 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x562b08bbde30 con 0x562b08c4dfe0 2024-03-31T23:13:03.939 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:03.933+0000 7f3844ff9640 1 -- 172.21.15.46:0/4028736065 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 26) ==== 8350+0+0 (secure 0 0 0) 0x7f384c004b60 con 0x562b08c4dfe0 2024-03-31T23:13:03.939 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:03.933+0000 7f3865bfd740 10 client.16071 fetch_fsmap finished waiting for FSMap version 26 2024-03-31T23:13:03.940 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:03.933+0000 7f3865bfd740 10 client.16071 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-31T23:13:03.940 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:03.933+0000 7f3865bfd740 20 client.16071 populate_metadata read hostname 'smithi046' 2024-03-31T23:13:03.940 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:03.933+0000 7f3865bfd740 1 --2- 172.21.15.46:0/4028736065 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x562b08ccd630 0x562b08ccfa50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:13:03.940 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:03.933+0000 7f3865bfd740 4 client.16071 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-31T23:13:03.940 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:03.933+0000 7f3865bfd740 1 -- 172.21.15.46:0/4028736065 --> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] -- command(tid 0: {"prefix": "flush journal"}) -- 0x562b08bb8180 con 0x562b08ccd630 2024-03-31T23:13:03.940 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:03.933+0000 7f3861116640 1 --2- 172.21.15.46:0/4028736065 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x562b08ccd630 0x562b08ccfa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:13:03.940 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:03.934+0000 7f3844ff9640 10 client.16071 ms_handle_connect on v2:172.21.15.142:6838/2519143918 2024-03-31T23:13:03.940 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:03.934+0000 7f3861116640 1 --2- 172.21.15.46:0/4028736065 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x562b08ccd630 0x562b08ccfa50 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=baee59c600dc260 server_cookie=de1e4f79b771db98 in_seq=0 out_seq=0 2024-03-31T23:13:04.190 INFO:tasks.workunit.client.0.smithi046.stdout: 1 6275 5.10 MB/sec warmup 10 sec latency 55.581 ms 2024-03-31T23:13:04.693 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:13:04 smithi142 ceph-mon[35271]: pgmap v43: 129 pgs: 129 active+clean; 25 MiB data, 423 MiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 2.2 MiB/s wr, 39 op/s 2024-03-31T23:13:04.795 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:13:04 smithi162 ceph-mon[36237]: pgmap v43: 129 pgs: 129 active+clean; 25 MiB data, 423 MiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 2.2 MiB/s wr, 39 op/s 2024-03-31T23:13:04.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:13:04 smithi046 ceph-mon[32498]: pgmap v43: 129 pgs: 129 active+clean; 25 MiB data, 423 MiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 2.2 MiB/s wr, 39 op/s 2024-03-31T23:13:05.190 INFO:tasks.workunit.client.0.smithi046.stdout: 1 6891 4.95 MB/sec warmup 11 sec latency 17.070 ms 2024-03-31T23:13:05.887 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:05.885+0000 7f3844ff9640 1 -- 172.21.15.46:0/4028736065 <== mds.0 v2:172.21.15.142:6838/2519143918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x562b08bb8180 con 0x562b08ccd630 2024-03-31T23:13:05.887 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:05.885+0000 7f3844ff9640 10 client.16071 handle_command_reply: tid=0 2024-03-31T23:13:05.887 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:05.885+0000 7f3865bfd740 1 client.16071 shutdown 2024-03-31T23:13:05.887 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:05.885+0000 7f3865bfd740 1 -- 172.21.15.46:0/4028736065 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562b08c4dfe0 msgr2=0x562b08c507f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:13:05.887 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:05.885+0000 7f3865bfd740 1 --2- 172.21.15.46:0/4028736065 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562b08c4dfe0 0x562b08c507f0 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x562b08c53700 tx=0x7f384c00f3c0 comp rx=0 tx=0).stop 2024-03-31T23:13:05.888 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:05.885+0000 7f3865bfd740 1 -- 172.21.15.46:0/4028736065 shutdown_connections 2024-03-31T23:13:05.888 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:05.885+0000 7f3865bfd740 1 --2- 172.21.15.46:0/4028736065 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x562b08ccd630 0x562b08ccfa50 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:05.888 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:05.885+0000 7f3865bfd740 1 --2- 172.21.15.46:0/4028736065 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x562b08c56db0 0x562b08c50d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:05.888 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:05.885+0000 7f3865bfd740 1 --2- 172.21.15.46:0/4028736065 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562b08c4dfe0 0x562b08c507f0 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:05.888 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:05.885+0000 7f3844ff9640 0 client.16071 ms_handle_reset on v2:172.21.15.142:6838/2519143918 2024-03-31T23:13:05.888 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:05.885+0000 7f3865bfd740 1 --2- 172.21.15.46:0/4028736065 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562b08c4d6c0 0x562b08c5f0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:05.888 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:05.885+0000 7f3865bfd740 1 -- 172.21.15.46:0/4028736065 >> 172.21.15.46:0/4028736065 conn(0x562b08c47fe0 msgr2=0x562b08c466f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:13:05.888 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:05.885+0000 7f3865bfd740 1 -- 172.21.15.46:0/4028736065 shutdown_connections 2024-03-31T23:13:05.888 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:05.885+0000 7f3865bfd740 1 -- 172.21.15.46:0/4028736065 wait complete. 2024-03-31T23:13:05.888 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:05.885+0000 7f3865bfd740 20 client.16071 trim_cache size 0 max 16384 2024-03-31T23:13:05.889 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout:{ 2024-03-31T23:13:05.889 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout: "message": "", 2024-03-31T23:13:05.889 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout: "return_code": 0 2024-03-31T23:13:05.889 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout:} 2024-03-31T23:13:05.890 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:05.886+0000 7f386337a640 1 -- 172.21.15.46:0/3402336549 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f38480a27d0 msgr2=0x7f38480a4c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:13:05.890 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:05.886+0000 7f386337a640 1 --2- 172.21.15.46:0/3402336549 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f38480a27d0 0x7f38480a4c90 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f3854037040 tx=0x7f385402e040 comp rx=0 tx=0).stop 2024-03-31T23:13:05.890 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:05.886+0000 7f386337a640 1 -- 172.21.15.46:0/3402336549 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f385c1008a0 msgr2=0x7f385c196b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:13:05.890 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:05.886+0000 7f386337a640 1 --2- 172.21.15.46:0/3402336549 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f385c1008a0 0x7f385c196b10 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f385800e9f0 tx=0x7f385800eeb0 comp rx=0 tx=0).stop 2024-03-31T23:13:05.890 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:05.886+0000 7f386337a640 1 -- 172.21.15.46:0/3402336549 shutdown_connections 2024-03-31T23:13:05.890 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:05.886+0000 7f386337a640 1 --2- 172.21.15.46:0/3402336549 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f38480a27d0 0x7f38480a4c90 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:05.890 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:05.886+0000 7f386337a640 1 --2- 172.21.15.46:0/3402336549 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f385c1055f0 0x7f385c197590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:05.890 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:05.886+0000 7f386337a640 1 --2- 172.21.15.46:0/3402336549 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f385c101260 0x7f385c197050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:05.890 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:05.886+0000 7f386337a640 1 --2- 172.21.15.46:0/3402336549 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f385c1008a0 0x7f385c196b10 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:05.890 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:05.886+0000 7f386337a640 1 -- 172.21.15.46:0/3402336549 >> 172.21.15.46:0/3402336549 conn(0x7f385c0774a0 msgr2=0x7f385c0fe7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:13:05.890 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:05.886+0000 7f386337a640 1 -- 172.21.15.46:0/3402336549 shutdown_connections 2024-03-31T23:13:05.890 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:05.886+0000 7f386337a640 1 -- 172.21.15.46:0/3402336549 wait complete. 2024-03-31T23:13:06.190 INFO:tasks.workunit.client.0.smithi046.stdout: 1 7471 4.93 MB/sec warmup 12 sec latency 15.245 ms 2024-03-31T23:13:06.693 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:13:06 smithi142 ceph-mon[35271]: pgmap v44: 129 pgs: 129 active+clean; 28 MiB data, 447 MiB used, 1.0 TiB / 1.0 TiB avail; 235 KiB/s rd, 2.8 MiB/s wr, 94 op/s 2024-03-31T23:13:06.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:13:06 smithi162 ceph-mon[36237]: pgmap v44: 129 pgs: 129 active+clean; 28 MiB data, 447 MiB used, 1.0 TiB / 1.0 TiB avail; 235 KiB/s rd, 2.8 MiB/s wr, 94 op/s 2024-03-31T23:13:06.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:13:06 smithi046 ceph-mon[32498]: pgmap v44: 129 pgs: 129 active+clean; 28 MiB data, 447 MiB used, 1.0 TiB / 1.0 TiB avail; 235 KiB/s rd, 2.8 MiB/s wr, 94 op/s 2024-03-31T23:13:07.190 INFO:tasks.workunit.client.0.smithi046.stdout: 1 7891 4.58 MB/sec warmup 13 sec latency 16.308 ms 2024-03-31T23:13:08.190 INFO:tasks.workunit.client.0.smithi046.stdout: 1 8101 4.30 MB/sec warmup 14 sec latency 456.415 ms 2024-03-31T23:13:08.664 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:13:08 smithi046 ceph-mon[32498]: pgmap v45: 129 pgs: 129 active+clean; 28 MiB data, 447 MiB used, 1.0 TiB / 1.0 TiB avail; 234 KiB/s rd, 2.8 MiB/s wr, 91 op/s 2024-03-31T23:13:08.664 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:13:08 smithi046 ceph-mon[32498]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-03-31T23:13:08.693 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:13:08 smithi142 ceph-mon[35271]: pgmap v45: 129 pgs: 129 active+clean; 28 MiB data, 447 MiB used, 1.0 TiB / 1.0 TiB avail; 234 KiB/s rd, 2.8 MiB/s wr, 91 op/s 2024-03-31T23:13:08.693 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:13:08 smithi142 ceph-mon[35271]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-03-31T23:13:08.795 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:13:08 smithi162 ceph-mon[36237]: pgmap v45: 129 pgs: 129 active+clean; 28 MiB data, 447 MiB used, 1.0 TiB / 1.0 TiB avail; 234 KiB/s rd, 2.8 MiB/s wr, 91 op/s 2024-03-31T23:13:08.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:13:08 smithi162 ceph-mon[36237]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-03-31T23:13:09.190 INFO:tasks.workunit.client.0.smithi046.stdout: 1 8101 4.01 MB/sec warmup 15 sec latency 1456.506 ms 2024-03-31T23:13:09.519 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-31T23:13:09.519 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-31T23:13:09.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:09.634+0000 7f8502640640 1 -- 172.21.15.46:0/1231150929 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f84fc1005c0 msgr2=0x7f84fc100a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:13:09.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:09.634+0000 7f8502640640 1 --2- 172.21.15.46:0/1231150929 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f84fc1005c0 0x7f84fc100a40 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f84f0009930 tx=0x7f84f002bc30 comp rx=0 tx=0).stop 2024-03-31T23:13:09.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:09.634+0000 7f8502640640 1 -- 172.21.15.46:0/1231150929 shutdown_connections 2024-03-31T23:13:09.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:09.634+0000 7f8502640640 1 --2- 172.21.15.46:0/1231150929 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f84fc100f80 0x7f84fc10d490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:09.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:09.634+0000 7f8502640640 1 --2- 172.21.15.46:0/1231150929 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f84fc1005c0 0x7f84fc100a40 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:09.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:09.634+0000 7f8502640640 1 --2- 172.21.15.46:0/1231150929 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f84fc106660 0x7f84fc106a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:09.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:09.634+0000 7f8502640640 1 -- 172.21.15.46:0/1231150929 >> 172.21.15.46:0/1231150929 conn(0x7f84fc077750 msgr2=0x7f84fc077b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:13:09.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:09.634+0000 7f8502640640 1 -- 172.21.15.46:0/1231150929 shutdown_connections 2024-03-31T23:13:09.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:09.634+0000 7f8502640640 1 -- 172.21.15.46:0/1231150929 wait complete. 2024-03-31T23:13:09.638 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:09.635+0000 7f8502640640 1 Processor -- start 2024-03-31T23:13:09.638 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:09.635+0000 7f8502640640 1 -- start start 2024-03-31T23:13:09.639 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:09.635+0000 7f8502640640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f84fc1005c0 0x7f84fc198d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:13:09.639 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:09.635+0000 7f8502640640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f84fc100f80 0x7f84fc1992a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:13:09.639 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:09.635+0000 7f8502640640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f84fc106660 0x7f84fc1a3de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:13:09.639 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:09.635+0000 7f8502640640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f84fc1108e0 con 0x7f84fc1005c0 2024-03-31T23:13:09.639 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:09.635+0000 7f8502640640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f84fc110760 con 0x7f84fc106660 2024-03-31T23:13:09.639 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:09.635+0000 7f8502640640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f84fc110a60 con 0x7f84fc100f80 2024-03-31T23:13:09.639 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:09.635+0000 7f84fbfff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f84fc1005c0 0x7f84fc198d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:13:09.639 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:09.635+0000 7f84fbfff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f84fc1005c0 0x7f84fc198d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:48850/0 (socket says 172.21.15.46:48850) 2024-03-31T23:13:09.639 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:09.635+0000 7f84fbfff640 1 -- 172.21.15.46:0/1764600751 learned_addr learned my addr 172.21.15.46:0/1764600751 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:13:09.639 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:09.635+0000 7f84fb7fe640 1 --2- 172.21.15.46:0/1764600751 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f84fc100f80 0x7f84fc1992a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:13:09.639 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:09.635+0000 7f8500bdd640 1 --2- 172.21.15.46:0/1764600751 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f84fc106660 0x7f84fc1a3de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:13:09.639 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:09.636+0000 7f84fb7fe640 1 -- 172.21.15.46:0/1764600751 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f84fc106660 msgr2=0x7f84fc1a3de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:13:09.640 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:09.636+0000 7f84fb7fe640 1 --2- 172.21.15.46:0/1764600751 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f84fc106660 0x7f84fc1a3de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:09.640 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:09.636+0000 7f84fb7fe640 1 -- 172.21.15.46:0/1764600751 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f84fc1005c0 msgr2=0x7f84fc198d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:13:09.640 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:09.636+0000 7f84fb7fe640 1 --2- 172.21.15.46:0/1764600751 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f84fc1005c0 0x7f84fc198d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:09.640 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:09.636+0000 7f84fb7fe640 1 -- 172.21.15.46:0/1764600751 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f84fc1a4320 con 0x7f84fc100f80 2024-03-31T23:13:09.640 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:09.636+0000 7f8500bdd640 1 --2- 172.21.15.46:0/1764600751 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f84fc106660 0x7f84fc1a3de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:13:09.640 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:09.636+0000 7f84fb7fe640 1 --2- 172.21.15.46:0/1764600751 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f84fc100f80 0x7f84fc1992a0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f84f0036040 tx=0x7f84f002c640 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:13:09.640 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:09.636+0000 7f84f97fa640 1 -- 172.21.15.46:0/1764600751 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84f0043020 con 0x7f84fc100f80 2024-03-31T23:13:09.640 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:09.636+0000 7f8502640640 1 -- 172.21.15.46:0/1764600751 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f84fc1a44f0 con 0x7f84fc100f80 2024-03-31T23:13:09.640 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:09.636+0000 7f84f97fa640 1 -- 172.21.15.46:0/1764600751 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f84f002ca70 con 0x7f84fc100f80 2024-03-31T23:13:09.640 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:09.636+0000 7f84f97fa640 1 -- 172.21.15.46:0/1764600751 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84f0003a10 con 0x7f84fc100f80 2024-03-31T23:13:09.640 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:09.636+0000 7f8502640640 1 -- 172.21.15.46:0/1764600751 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f84fc1a4a10 con 0x7f84fc100f80 2024-03-31T23:13:09.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:09.638+0000 7f84f97fa640 1 -- 172.21.15.46:0/1764600751 <== mon.1 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 24) ==== 137553+0+0 (secure 0 0 0) 0x7f84f0046020 con 0x7f84fc100f80 2024-03-31T23:13:09.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:09.639+0000 7f850521d740 1 Processor -- start 2024-03-31T23:13:09.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:09.639+0000 7f850521d740 1 -- start start 2024-03-31T23:13:09.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:09.639+0000 7f84f97fa640 1 --2- 172.21.15.46:0/1764600751 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f84e40a2730 0x7f84e40a4bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:13:09.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:09.639+0000 7f850521d740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559035812290 0x559035812670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:13:09.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:09.639+0000 7f850521d740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559035812bb0 0x55903581b250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:13:09.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:09.639+0000 7f850521d740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55903581b790 0x55903581db80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:13:09.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:09.639+0000 7f850521d740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x559035776180 con 0x55903581b790 2024-03-31T23:13:09.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:09.639+0000 7f850521d740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55903571eb70 con 0x559035812290 2024-03-31T23:13:09.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:09.639+0000 7f850521d740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x559035772220 con 0x559035812bb0 2024-03-31T23:13:09.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:09.639+0000 7f8500bdd640 1 --2- 172.21.15.46:0/1764600751 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f84e40a2730 0x7f84e40a4bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:13:09.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:09.639+0000 7f8500bdd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55903581b790 0x55903581db80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:13:09.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:09.639+0000 7f84f97fa640 1 -- 172.21.15.46:0/1764600751 <== mon.1 v2:172.21.15.162:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7f84f00f0fe0 con 0x7f84fc100f80 2024-03-31T23:13:09.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:09.639+0000 7f84fb7fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559035812bb0 0x55903581b250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:13:09.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:09.639+0000 7f84fbfff640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559035812290 0x559035812670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:13:09.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:09.639+0000 7f8500bdd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55903581b790 0x55903581db80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:48856/0 (socket says 172.21.15.46:48856) 2024-03-31T23:13:09.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:09.639+0000 7f8500bdd640 1 -- 172.21.15.46:0/204259860 learned_addr learned my addr 172.21.15.46:0/204259860 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:13:09.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:09.639+0000 7f84fb7fe640 1 -- 172.21.15.46:0/204259860 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559035812290 msgr2=0x559035812670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:13:09.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:09.639+0000 7f84fb7fe640 1 --2- 172.21.15.46:0/204259860 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559035812290 0x559035812670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:09.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:09.639+0000 7f84fb7fe640 1 -- 172.21.15.46:0/204259860 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55903581b790 msgr2=0x55903581db80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:13:09.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:09.639+0000 7f84fb7fe640 1 --2- 172.21.15.46:0/204259860 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55903581b790 0x55903581db80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:09.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:09.639+0000 7f84fb7fe640 1 -- 172.21.15.46:0/204259860 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5590357327c0 con 0x559035812bb0 2024-03-31T23:13:09.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:09.640+0000 7f8500bdd640 1 --2- 172.21.15.46:0/1764600751 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f84e40a2730 0x7f84e40a4bf0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f84f400f480 tx=0x7f84f40117c0 comp rx=0 tx=0).ready entity=mgr.15654 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:13:09.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:09.640+0000 7f84fb7fe640 1 --2- 172.21.15.46:0/204259860 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559035812bb0 0x55903581b250 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f84f00ee2d0 tx=0x7f84f00bd130 comp rx=0 tx=0).ready entity=mon.1 client_cookie=14553824fc5d8034 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:13:09.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:09.640+0000 7f84e27fc640 1 -- 172.21.15.46:0/204259860 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84f003e690 con 0x559035812bb0 2024-03-31T23:13:09.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:09.640+0000 7f84e27fc640 1 -- 172.21.15.46:0/204259860 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f84f00473b0 con 0x559035812bb0 2024-03-31T23:13:09.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:09.640+0000 7f84e27fc640 1 -- 172.21.15.46:0/204259860 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84f00dd0d0 con 0x559035812bb0 2024-03-31T23:13:09.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:09.641+0000 7f850521d740 1 -- 172.21.15.46:0/204259860 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559035812bb0 msgr2=0x55903581b250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:13:09.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:09.641+0000 7f850521d740 1 --2- 172.21.15.46:0/204259860 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559035812bb0 0x55903581b250 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f84f00ee2d0 tx=0x7f84f00bd130 comp rx=0 tx=0).stop 2024-03-31T23:13:09.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:09.641+0000 7f850521d740 1 -- 172.21.15.46:0/204259860 shutdown_connections 2024-03-31T23:13:09.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:09.641+0000 7f850521d740 1 --2- 172.21.15.46:0/204259860 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55903581b790 0x55903581db80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:09.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:09.641+0000 7f850521d740 1 --2- 172.21.15.46:0/204259860 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559035812bb0 0x55903581b250 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:09.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:09.641+0000 7f850521d740 1 --2- 172.21.15.46:0/204259860 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559035812290 0x559035812670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:09.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:09.641+0000 7f850521d740 1 -- 172.21.15.46:0/204259860 >> 172.21.15.46:0/204259860 conn(0x55903580c9c0 msgr2=0x55903580cda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:13:09.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:09.641+0000 7f850521d740 1 -- 172.21.15.46:0/204259860 shutdown_connections 2024-03-31T23:13:09.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:09.641+0000 7f850521d740 1 -- 172.21.15.46:0/204259860 wait complete. 2024-03-31T23:13:09.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:09.641+0000 7f850521d740 1 -- start start 2024-03-31T23:13:09.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:09.641+0000 7f850521d740 1 Processor -- start 2024-03-31T23:13:09.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:09.641+0000 7f850521d740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559035812290 0x5590358187c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:13:09.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:09.641+0000 7f850521d740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559035812bb0 0x559035818d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:13:09.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:09.641+0000 7f850521d740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55903581b790 0x559035819240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:13:09.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:09.641+0000 7f850521d740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x559035772220 con 0x559035812bb0 2024-03-31T23:13:09.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:09.641+0000 7f850521d740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x5590357144a0 con 0x55903581b790 2024-03-31T23:13:09.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:09.641+0000 7f850521d740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55903581e340 con 0x559035812290 2024-03-31T23:13:09.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:09.642+0000 7f850521d740 10 client.0 _ll_register_callbacks cb 0x559035717d90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T23:13:09.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:09.642+0000 7f84fb7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559035812bb0 0x559035818d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:13:09.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:09.642+0000 7f8500bdd640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55903581b790 0x559035819240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:13:09.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:09.642+0000 7f84fb7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559035812bb0 0x559035818d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:48862/0 (socket says 172.21.15.46:48862) 2024-03-31T23:13:09.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:09.642+0000 7f84fbfff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559035812290 0x5590358187c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:13:09.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:09.642+0000 7f84fb7fe640 1 -- 172.21.15.46:0/3898975051 learned_addr learned my addr 172.21.15.46:0/3898975051 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:13:09.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:09.642+0000 7f84fb7fe640 1 -- 172.21.15.46:0/3898975051 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559035812290 msgr2=0x5590358187c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:13:09.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:09.642+0000 7f84fb7fe640 1 --2- 172.21.15.46:0/3898975051 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559035812290 0x5590358187c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:09.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:09.642+0000 7f84fb7fe640 1 -- 172.21.15.46:0/3898975051 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55903581b790 msgr2=0x559035819240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:13:09.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:09.642+0000 7f84fb7fe640 1 --2- 172.21.15.46:0/3898975051 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55903581b790 0x559035819240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:09.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:09.642+0000 7f84fb7fe640 1 -- 172.21.15.46:0/3898975051 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5590355547d0 con 0x559035812bb0 2024-03-31T23:13:09.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:09.642+0000 7f8500bdd640 1 --2- 172.21.15.46:0/3898975051 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55903581b790 0x559035819240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:13:09.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:09.642+0000 7f84fbfff640 1 --2- 172.21.15.46:0/3898975051 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559035812290 0x5590358187c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:13:09.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:09.642+0000 7f84fb7fe640 1 --2- 172.21.15.46:0/3898975051 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559035812bb0 0x559035818d00 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f84f00b9c00 tx=0x7f84f0005c90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cbe02a74a48944ca server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:13:09.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:09.642+0000 7f84db7fe640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-31T23:13:09.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:09.642+0000 7f84db7fe640 1 -- 172.21.15.46:0/3898975051 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84f0048400 con 0x559035812bb0 2024-03-31T23:13:09.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:09.642+0000 7f84db7fe640 1 -- 172.21.15.46:0/3898975051 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7f84f003e450 con 0x559035812bb0 2024-03-31T23:13:09.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:09.642+0000 7f850521d740 1 -- 172.21.15.46:0/3898975051 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f84f00473b0 con 0x559035812bb0 2024-03-31T23:13:09.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:09.643+0000 7f84db7fe640 1 -- 172.21.15.46:0/3898975051 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f84f00a4050 con 0x559035812bb0 2024-03-31T23:13:09.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:09.643+0000 7f850521d740 10 client.16083 fetch_fsmap learned FSMap version 27 2024-03-31T23:13:09.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:09.643+0000 7f850521d740 1 -- 172.21.15.46:0/3898975051 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55903548a3f0 con 0x559035812bb0 2024-03-31T23:13:09.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:09.643+0000 7f84db7fe640 1 -- 172.21.15.46:0/3898975051 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f84f009f020 con 0x559035812bb0 2024-03-31T23:13:09.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:09.643+0000 7f84db7fe640 1 -- 172.21.15.46:0/3898975051 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f84f00053b0 con 0x559035812bb0 2024-03-31T23:13:09.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:09.643+0000 7f850521d740 10 client.16083 fetch_fsmap finished waiting for FSMap version 27 2024-03-31T23:13:09.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:09.643+0000 7f850521d740 10 client.16083 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-31T23:13:09.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:09.643+0000 7f850521d740 20 client.16083 populate_metadata read hostname 'smithi046' 2024-03-31T23:13:09.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:09.643+0000 7f850521d740 1 --2- 172.21.15.46:0/3898975051 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55903582b5e0 0x55903582d9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:13:09.652 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:09.643+0000 7f850521d740 4 client.16083 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-31T23:13:09.652 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:09.643+0000 7f850521d740 1 -- 172.21.15.46:0/3898975051 --> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5590357836c0 con 0x55903582b5e0 2024-03-31T23:13:09.652 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:09.643+0000 7f84fbfff640 1 --2- 172.21.15.46:0/3898975051 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55903582b5e0 0x55903582d9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:13:09.652 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:09.644+0000 7f84db7fe640 10 client.16083 ms_handle_connect on v2:172.21.15.142:6838/2519143918 2024-03-31T23:13:09.652 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:09.644+0000 7f84fbfff640 1 --2- 172.21.15.46:0/3898975051 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55903582b5e0 0x55903582d9d0 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3c372a68a2a6be6b server_cookie=b9995f0f21ff837 in_seq=0 out_seq=0 2024-03-31T23:13:09.652 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:09.644+0000 7f84db7fe640 1 -- 172.21.15.46:0/3898975051 <== mds.0 v2:172.21.15.142:6838/2519143918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5590357836c0 con 0x55903582b5e0 2024-03-31T23:13:09.652 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:09.644+0000 7f84db7fe640 10 client.16083 handle_command_reply: tid=0 2024-03-31T23:13:09.652 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:09.644+0000 7f850521d740 1 client.16083 shutdown 2024-03-31T23:13:09.652 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:09.645+0000 7f850521d740 1 -- 172.21.15.46:0/3898975051 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559035812bb0 msgr2=0x559035818d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:13:09.652 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:09.645+0000 7f850521d740 1 --2- 172.21.15.46:0/3898975051 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559035812bb0 0x559035818d00 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f84f00b9c00 tx=0x7f84f0005c90 comp rx=0 tx=0).stop 2024-03-31T23:13:09.652 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:09.645+0000 7f850521d740 1 -- 172.21.15.46:0/3898975051 shutdown_connections 2024-03-31T23:13:09.652 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:09.645+0000 7f850521d740 1 --2- 172.21.15.46:0/3898975051 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55903582b5e0 0x55903582d9d0 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:09.652 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:09.645+0000 7f850521d740 1 --2- 172.21.15.46:0/3898975051 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55903581b790 0x559035819240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:09.652 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:09.645+0000 7f850521d740 1 --2- 172.21.15.46:0/3898975051 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559035812bb0 0x559035818d00 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:09.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:09.645+0000 7f850521d740 1 --2- 172.21.15.46:0/3898975051 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559035812290 0x5590358187c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:09.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:09.645+0000 7f850521d740 1 -- 172.21.15.46:0/3898975051 >> 172.21.15.46:0/3898975051 conn(0x55903580c9c0 msgr2=0x55903580bc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:13:09.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:09.645+0000 7f84db7fe640 0 client.16083 ms_handle_reset on v2:172.21.15.142:6838/2519143918 2024-03-31T23:13:09.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:09.645+0000 7f850521d740 1 -- 172.21.15.46:0/3898975051 shutdown_connections 2024-03-31T23:13:09.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:09.645+0000 7f850521d740 1 -- 172.21.15.46:0/3898975051 wait complete. 2024-03-31T23:13:09.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:09.645+0000 7f850521d740 20 client.16083 trim_cache size 0 max 16384 2024-03-31T23:13:09.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:09.662+0000 7f850521d740 1 Processor -- start 2024-03-31T23:13:09.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:09.662+0000 7f850521d740 1 -- start start 2024-03-31T23:13:09.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:09.662+0000 7f850521d740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55903580c9c0 0x55903581aef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:13:09.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:09.662+0000 7f850521d740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559035812290 0x559035826f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:13:09.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:09.662+0000 7f850521d740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559035812bb0 0x55903580baa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:13:09.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:09.662+0000 7f850521d740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f84fc1b14b0 con 0x559035812bb0 2024-03-31T23:13:09.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:09.662+0000 7f850521d740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f84fc1aed80 con 0x559035812290 2024-03-31T23:13:09.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:09.662+0000 7f850521d740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f84fc1acbd0 con 0x55903580c9c0 2024-03-31T23:13:09.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:09.663+0000 7f8500bdd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559035812bb0 0x55903580baa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:13:09.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:09.663+0000 7f84fbfff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55903580c9c0 0x55903581aef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:13:09.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:09.663+0000 7f8500bdd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559035812bb0 0x55903580baa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:48874/0 (socket says 172.21.15.46:48874) 2024-03-31T23:13:09.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:09.663+0000 7f8500bdd640 1 -- 172.21.15.46:0/659645298 learned_addr learned my addr 172.21.15.46:0/659645298 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:13:09.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:09.663+0000 7f84fb7fe640 1 --2- 172.21.15.46:0/659645298 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559035812290 0x559035826f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:13:09.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:09.663+0000 7f84fbfff640 1 -- 172.21.15.46:0/659645298 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559035812290 msgr2=0x559035826f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:13:09.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:09.663+0000 7f84fbfff640 1 --2- 172.21.15.46:0/659645298 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559035812290 0x559035826f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:09.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:09.663+0000 7f84fbfff640 1 -- 172.21.15.46:0/659645298 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559035812bb0 msgr2=0x55903580baa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:13:09.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:09.663+0000 7f84fbfff640 1 --2- 172.21.15.46:0/659645298 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559035812bb0 0x55903580baa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:09.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:09.663+0000 7f84fbfff640 1 -- 172.21.15.46:0/659645298 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55903544a0f0 con 0x55903580c9c0 2024-03-31T23:13:09.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:09.663+0000 7f8500bdd640 1 --2- 172.21.15.46:0/659645298 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559035812bb0 0x55903580baa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:13:09.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:09.663+0000 7f84fbfff640 1 --2- 172.21.15.46:0/659645298 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55903580c9c0 0x55903581aef0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x55903582e1e0 tx=0x7f84ec005f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e8e2b1925268cb38 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:13:09.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:09.663+0000 7f84e1ffb640 1 -- 172.21.15.46:0/659645298 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84ec004030 con 0x55903580c9c0 2024-03-31T23:13:09.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:09.663+0000 7f84e1ffb640 1 -- 172.21.15.46:0/659645298 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f84ec024070 con 0x55903580c9c0 2024-03-31T23:13:09.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:09.663+0000 7f84e1ffb640 1 -- 172.21.15.46:0/659645298 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84ec004b20 con 0x55903580c9c0 2024-03-31T23:13:09.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:09.664+0000 7f850521d740 1 -- 172.21.15.46:0/659645298 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55903580c9c0 msgr2=0x55903581aef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:13:09.694 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:09.664+0000 7f850521d740 1 --2- 172.21.15.46:0/659645298 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55903580c9c0 0x55903581aef0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x55903582e1e0 tx=0x7f84ec005f90 comp rx=0 tx=0).stop 2024-03-31T23:13:09.694 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:09.664+0000 7f850521d740 1 -- 172.21.15.46:0/659645298 shutdown_connections 2024-03-31T23:13:09.694 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:09.664+0000 7f850521d740 1 --2- 172.21.15.46:0/659645298 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559035812bb0 0x55903580baa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:09.694 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:09.664+0000 7f850521d740 1 --2- 172.21.15.46:0/659645298 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559035812290 0x559035826f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:09.694 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:09.664+0000 7f850521d740 1 --2- 172.21.15.46:0/659645298 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55903580c9c0 0x55903581aef0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:09.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:09.664+0000 7f850521d740 1 -- 172.21.15.46:0/659645298 >> 172.21.15.46:0/659645298 conn(0x559035817400 msgr2=0x55903580b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:13:09.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:09.664+0000 7f850521d740 1 -- 172.21.15.46:0/659645298 shutdown_connections 2024-03-31T23:13:09.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:09.664+0000 7f850521d740 1 -- 172.21.15.46:0/659645298 wait complete. 2024-03-31T23:13:09.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:09.664+0000 7f850521d740 1 -- start start 2024-03-31T23:13:09.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:09.665+0000 7f850521d740 1 Processor -- start 2024-03-31T23:13:09.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:09.665+0000 7f850521d740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55903580c9c0 0x55903581f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:13:09.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:09.665+0000 7f850521d740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559035812290 0x55903581f610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:13:09.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:09.665+0000 7f850521d740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559035812bb0 0x55903581fb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:13:09.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:09.665+0000 7f850521d740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f84fc1ad240 con 0x559035812290 2024-03-31T23:13:09.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:09.665+0000 7f850521d740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f84fc1ae6b0 con 0x55903580c9c0 2024-03-31T23:13:09.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:09.665+0000 7f850521d740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f84fc1ad910 con 0x559035812bb0 2024-03-31T23:13:09.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:09.665+0000 7f850521d740 10 client.0 _ll_register_callbacks cb 0x7f84f4011c10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T23:13:09.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:09.665+0000 7f84fb7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559035812290 0x55903581f610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:13:09.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:09.665+0000 7f8500bdd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559035812bb0 0x55903581fb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:13:09.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:09.665+0000 7f84fbfff640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55903580c9c0 0x55903581f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:13:09.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:09.665+0000 7f84fb7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559035812290 0x55903581f610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:48888/0 (socket says 172.21.15.46:48888) 2024-03-31T23:13:09.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:09.665+0000 7f8500bdd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559035812bb0 0x55903581fb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.46:40774/0 (socket says 172.21.15.46:40774) 2024-03-31T23:13:09.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:09.665+0000 7f84fb7fe640 1 -- 172.21.15.46:0/2998095003 learned_addr learned my addr 172.21.15.46:0/2998095003 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:13:09.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:09.665+0000 7f84fb7fe640 1 -- 172.21.15.46:0/2998095003 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559035812bb0 msgr2=0x55903581fb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:13:09.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:09.665+0000 7f84fb7fe640 1 --2- 172.21.15.46:0/2998095003 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559035812bb0 0x55903581fb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:09.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:09.665+0000 7f84fb7fe640 1 -- 172.21.15.46:0/2998095003 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55903580c9c0 msgr2=0x55903581f0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:13:09.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:09.665+0000 7f84fb7fe640 1 --2- 172.21.15.46:0/2998095003 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55903580c9c0 0x55903581f0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:09.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:09.665+0000 7f84fb7fe640 1 -- 172.21.15.46:0/2998095003 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559035890b20 con 0x559035812290 2024-03-31T23:13:09.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:09.665+0000 7f8500bdd640 1 --2- 172.21.15.46:0/2998095003 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559035812bb0 0x55903581fb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:13:09.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:09.665+0000 7f84fbfff640 1 --2- 172.21.15.46:0/2998095003 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55903580c9c0 0x55903581f0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:13:09.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:09.666+0000 7f84fb7fe640 1 --2- 172.21.15.46:0/2998095003 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559035812290 0x55903581f610 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f84f0006010 tx=0x7f84f00c1230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3049dfad42d1257c server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:13:09.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:09.666+0000 7f84db7fe640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-31T23:13:09.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:09.666+0000 7f84db7fe640 1 -- 172.21.15.46:0/2998095003 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84f00485e0 con 0x559035812290 2024-03-31T23:13:09.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:09.666+0000 7f84db7fe640 1 -- 172.21.15.46:0/2998095003 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7f84f003e450 con 0x559035812290 2024-03-31T23:13:09.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:09.666+0000 7f850521d740 1 -- 172.21.15.46:0/2998095003 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f84ec024070 con 0x559035812290 2024-03-31T23:13:09.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:09.666+0000 7f84db7fe640 1 -- 172.21.15.46:0/2998095003 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f84f00dd0d0 con 0x559035812290 2024-03-31T23:13:09.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:09.666+0000 7f850521d740 10 client.16089 fetch_fsmap learned FSMap version 27 2024-03-31T23:13:09.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:09.666+0000 7f850521d740 1 -- 172.21.15.46:0/2998095003 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x559035890b20 con 0x559035812290 2024-03-31T23:13:09.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:09.666+0000 7f84db7fe640 1 -- 172.21.15.46:0/2998095003 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f84ec024070 con 0x559035812290 2024-03-31T23:13:09.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:09.666+0000 7f84db7fe640 1 -- 172.21.15.46:0/2998095003 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f84f00d1020 con 0x559035812290 2024-03-31T23:13:09.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:09.666+0000 7f850521d740 10 client.16089 fetch_fsmap finished waiting for FSMap version 27 2024-03-31T23:13:09.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:09.666+0000 7f850521d740 10 client.16089 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-31T23:13:09.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:09.666+0000 7f850521d740 20 client.16089 populate_metadata read hostname 'smithi046' 2024-03-31T23:13:09.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:09.666+0000 7f850521d740 1 --2- 172.21.15.46:0/2998095003 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55903581b790 0x559035894290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:13:09.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:09.666+0000 7f850521d740 4 client.16089 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-31T23:13:09.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:09.666+0000 7f850521d740 1 -- 172.21.15.46:0/2998095003 --> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55903577da10 con 0x55903581b790 2024-03-31T23:13:09.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:09.667+0000 7f84fbfff640 1 --2- 172.21.15.46:0/2998095003 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55903581b790 0x559035894290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:13:09.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:09.667+0000 7f84db7fe640 10 client.16089 ms_handle_connect on v2:172.21.15.142:6838/2519143918 2024-03-31T23:13:09.698 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:09.667+0000 7f84fbfff640 1 --2- 172.21.15.46:0/2998095003 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55903581b790 0x559035894290 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2ae0d0e26fe0f841 server_cookie=ce54b1627b44121b in_seq=0 out_seq=0 2024-03-31T23:13:09.698 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:09.667+0000 7f84db7fe640 1 -- 172.21.15.46:0/2998095003 <== mds.0 v2:172.21.15.142:6838/2519143918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55903577da10 con 0x55903581b790 2024-03-31T23:13:09.698 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:09.667+0000 7f84db7fe640 10 client.16089 handle_command_reply: tid=0 2024-03-31T23:13:09.698 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:09.667+0000 7f850521d740 1 client.16089 shutdown 2024-03-31T23:13:09.698 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:09.667+0000 7f850521d740 1 -- 172.21.15.46:0/2998095003 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559035812290 msgr2=0x55903581f610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:13:09.698 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:09.667+0000 7f850521d740 1 --2- 172.21.15.46:0/2998095003 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559035812290 0x55903581f610 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f84f0006010 tx=0x7f84f00c1230 comp rx=0 tx=0).stop 2024-03-31T23:13:09.698 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:09.667+0000 7f850521d740 1 -- 172.21.15.46:0/2998095003 shutdown_connections 2024-03-31T23:13:09.698 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:09.667+0000 7f850521d740 1 --2- 172.21.15.46:0/2998095003 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55903581b790 0x559035894290 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:09.698 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:09.667+0000 7f850521d740 1 --2- 172.21.15.46:0/2998095003 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559035812bb0 0x55903581fb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:09.698 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:09.667+0000 7f850521d740 1 --2- 172.21.15.46:0/2998095003 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559035812290 0x55903581f610 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:09.698 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:09.667+0000 7f84db7fe640 0 client.16089 ms_handle_reset on v2:172.21.15.142:6838/2519143918 2024-03-31T23:13:09.698 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:09.667+0000 7f850521d740 1 --2- 172.21.15.46:0/2998095003 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55903580c9c0 0x55903581f0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:09.698 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:09.667+0000 7f850521d740 1 -- 172.21.15.46:0/2998095003 >> 172.21.15.46:0/2998095003 conn(0x559035817400 msgr2=0x559035818ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:13:09.698 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:09.668+0000 7f850521d740 1 -- 172.21.15.46:0/2998095003 shutdown_connections 2024-03-31T23:13:09.699 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:09.668+0000 7f850521d740 1 -- 172.21.15.46:0/2998095003 wait complete. 2024-03-31T23:13:09.699 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:09.668+0000 7f850521d740 20 client.16089 trim_cache size 0 max 16384 2024-03-31T23:13:09.699 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-03-31T23:13:09.699 INFO:teuthology.orchestra.run.smithi046.stdout: "status": "no active scrubs running", 2024-03-31T23:13:09.699 INFO:teuthology.orchestra.run.smithi046.stdout: "scrubs": {} 2024-03-31T23:13:09.699 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-03-31T23:13:09.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:09.668+0000 7f8502640640 1 -- 172.21.15.46:0/1764600751 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f84e40a2730 msgr2=0x7f84e40a4bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:13:09.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:09.668+0000 7f8502640640 1 --2- 172.21.15.46:0/1764600751 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f84e40a2730 0x7f84e40a4bf0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f84f400f480 tx=0x7f84f40117c0 comp rx=0 tx=0).stop 2024-03-31T23:13:09.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:09.668+0000 7f8502640640 1 -- 172.21.15.46:0/1764600751 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f84fc100f80 msgr2=0x7f84fc1992a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:13:09.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:09.668+0000 7f8502640640 1 --2- 172.21.15.46:0/1764600751 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f84fc100f80 0x7f84fc1992a0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f84f0036040 tx=0x7f84f002c640 comp rx=0 tx=0).stop 2024-03-31T23:13:09.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:09.668+0000 7f8502640640 1 -- 172.21.15.46:0/1764600751 shutdown_connections 2024-03-31T23:13:09.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:09.668+0000 7f8502640640 1 --2- 172.21.15.46:0/1764600751 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f84e40a2730 0x7f84e40a4bf0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:09.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:09.668+0000 7f8502640640 1 --2- 172.21.15.46:0/1764600751 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f84fc106660 0x7f84fc1a3de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:09.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:09.668+0000 7f8502640640 1 --2- 172.21.15.46:0/1764600751 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f84fc100f80 0x7f84fc1992a0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:09.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:09.668+0000 7f8502640640 1 --2- 172.21.15.46:0/1764600751 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f84fc1005c0 0x7f84fc198d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:09.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:09.668+0000 7f8502640640 1 -- 172.21.15.46:0/1764600751 >> 172.21.15.46:0/1764600751 conn(0x7f84fc077750 msgr2=0x7f84fc0ff320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:13:09.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:09.668+0000 7f8502640640 1 -- 172.21.15.46:0/1764600751 shutdown_connections 2024-03-31T23:13:09.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:09.668+0000 7f8502640640 1 -- 172.21.15.46:0/1764600751 wait complete. 2024-03-31T23:13:09.702 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-31T23:13:09.702 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-31T23:13:09.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:09.805+0000 7fc6a089a640 1 -- 172.21.15.46:0/2699791015 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc69c100650 msgr2=0x7fc69c100ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:13:09.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:09.805+0000 7fc6a089a640 1 --2- 172.21.15.46:0/2699791015 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc69c100650 0x7fc69c100ad0 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7fc68c009930 tx=0x7fc68c02bc30 comp rx=0 tx=0).stop 2024-03-31T23:13:09.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:09.805+0000 7fc6a089a640 1 -- 172.21.15.46:0/2699791015 shutdown_connections 2024-03-31T23:13:09.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:09.805+0000 7fc6a089a640 1 --2- 172.21.15.46:0/2699791015 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc69c101010 0x7fc69c10d520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:09.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:09.805+0000 7fc6a089a640 1 --2- 172.21.15.46:0/2699791015 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc69c100650 0x7fc69c100ad0 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:09.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:09.805+0000 7fc6a089a640 1 --2- 172.21.15.46:0/2699791015 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc69c106660 0x7fc69c106a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:09.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:09.805+0000 7fc6a089a640 1 -- 172.21.15.46:0/2699791015 >> 172.21.15.46:0/2699791015 conn(0x7fc69c077750 msgr2=0x7fc69c077b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:13:09.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:09.805+0000 7fc6a089a640 1 -- 172.21.15.46:0/2699791015 shutdown_connections 2024-03-31T23:13:09.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:09.805+0000 7fc6a089a640 1 -- 172.21.15.46:0/2699791015 wait complete. 2024-03-31T23:13:09.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:09.806+0000 7fc6a089a640 1 Processor -- start 2024-03-31T23:13:09.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:09.806+0000 7fc6a089a640 1 -- start start 2024-03-31T23:13:09.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:09.806+0000 7fc6a089a640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc69c100650 0x7fc69c196bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:13:09.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:09.806+0000 7fc6a089a640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc69c101010 0x7fc69c1970f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:13:09.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:09.806+0000 7fc6a089a640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc69c106660 0x7fc69c19bc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:13:09.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:09.806+0000 7fc6a089a640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fc69c110920 con 0x7fc69c106660 2024-03-31T23:13:09.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:09.806+0000 7fc6a089a640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fc69c1107a0 con 0x7fc69c101010 2024-03-31T23:13:09.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:09.806+0000 7fc6a089a640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fc69c110aa0 con 0x7fc69c100650 2024-03-31T23:13:09.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:09.806+0000 7fc69ad9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc69c106660 0x7fc69c19bc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:13:09.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:09.806+0000 7fc69a59c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc69c100650 0x7fc69c196bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:13:09.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:09.806+0000 7fc69ad9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc69c106660 0x7fc69c19bc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:48896/0 (socket says 172.21.15.46:48896) 2024-03-31T23:13:09.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:09.806+0000 7fc69ad9d640 1 -- 172.21.15.46:0/2608545930 learned_addr learned my addr 172.21.15.46:0/2608545930 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:13:09.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:09.806+0000 7fc699d9b640 1 --2- 172.21.15.46:0/2608545930 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc69c101010 0x7fc69c1970f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:13:09.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:09.806+0000 7fc69ad9d640 1 -- 172.21.15.46:0/2608545930 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc69c100650 msgr2=0x7fc69c196bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:13:09.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:09.806+0000 7fc69ad9d640 1 --2- 172.21.15.46:0/2608545930 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc69c100650 0x7fc69c196bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:09.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:09.807+0000 7fc69ad9d640 1 -- 172.21.15.46:0/2608545930 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc69c101010 msgr2=0x7fc69c1970f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:13:09.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:09.807+0000 7fc69ad9d640 1 --2- 172.21.15.46:0/2608545930 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc69c101010 0x7fc69c1970f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:09.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:09.807+0000 7fc69ad9d640 1 -- 172.21.15.46:0/2608545930 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc69c19c140 con 0x7fc69c106660 2024-03-31T23:13:09.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:09.807+0000 7fc69ad9d640 1 --2- 172.21.15.46:0/2608545930 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc69c106660 0x7fc69c19bc00 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7fc69400efd0 tx=0x7fc69400c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:13:09.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:09.807+0000 7fc68b7fe640 1 -- 172.21.15.46:0/2608545930 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc694003b70 con 0x7fc69c106660 2024-03-31T23:13:09.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:09.807+0000 7fc68b7fe640 1 -- 172.21.15.46:0/2608545930 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc694003d10 con 0x7fc69c106660 2024-03-31T23:13:09.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:09.807+0000 7fc6a089a640 1 -- 172.21.15.46:0/2608545930 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc69c19c430 con 0x7fc69c106660 2024-03-31T23:13:09.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:09.807+0000 7fc68b7fe640 1 -- 172.21.15.46:0/2608545930 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc694010980 con 0x7fc69c106660 2024-03-31T23:13:09.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:09.807+0000 7fc6a089a640 1 -- 172.21.15.46:0/2608545930 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc69c19c900 con 0x7fc69c106660 2024-03-31T23:13:09.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:09.808+0000 7fc6a089a640 1 -- 172.21.15.46:0/2608545930 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc69c068540 con 0x7fc69c106660 2024-03-31T23:13:09.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:09.810+0000 7fc68b7fe640 1 -- 172.21.15.46:0/2608545930 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 24) ==== 137553+0+0 (secure 0 0 0) 0x7fc694004b40 con 0x7fc69c106660 2024-03-31T23:13:09.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:09.810+0000 7fc68b7fe640 1 --2- 172.21.15.46:0/2608545930 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fc6840a2730 0x7fc6840a4bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:13:09.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:09.810+0000 7fc68b7fe640 1 -- 172.21.15.46:0/2608545930 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7fc6940cfa20 con 0x7fc69c106660 2024-03-31T23:13:09.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:09.810+0000 7fc69a59c640 1 --2- 172.21.15.46:0/2608545930 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fc6840a2730 0x7fc6840a4bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:13:09.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:09.811+0000 7fc69a59c640 1 --2- 172.21.15.46:0/2608545930 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fc6840a2730 0x7fc6840a4bf0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fc690009b50 tx=0x7fc690005f90 comp rx=0 tx=0).ready entity=mgr.15654 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:13:09.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:09.814+0000 7fc68b7fe640 1 -- 172.21.15.46:0/2608545930 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fc69409c240 con 0x7fc69c106660 2024-03-31T23:13:10.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.040+0000 7fc6a089a640 1 -- 172.21.15.46:0/2608545930 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc69c108bc0 con 0x7fc69c106660 2024-03-31T23:13:10.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.041+0000 7fc68b7fe640 1 -- 172.21.15.46:0/2608545930 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+11885 (secure 0 0 0) 0x7fc69409bb40 con 0x7fc69c106660 2024-03-31T23:13:10.042 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-31T23:13:10.042 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/2169018547","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":2169018547},{"type":"v1","addr":"172.21.15.46:6835","nonce":2169018547}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6837/1069387756","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6836","nonce":1069387756},{"type":"v1","addr":"172.21.15.46:6837","nonce":1069387756}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14634,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6839/3915337844","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6838","nonce":3915337844},{"type":"v1","addr":"172.21.15.46:6839","nonce":3915337844}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14646,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6841/897767145","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6840","nonce":897767145},{"type":"v1","addr":"172.21.15.46:6841","nonce":897767145}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24415,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6833/1746950966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6832","nonce":1746950966},{"type":"v1","addr":"172.21.15.162:6833","nonce":1746950966}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24422,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6833/3394739181","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6832","nonce":3394739181},{"type":"v1","addr":"172.21.15.142:6833","nonce":3394739181}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":27,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-31T23:09:11.009090+0000","modified":"2024-03-31T23:13:07.268129+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24515,"mds_1":24482,"mds_2":24451},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24442":{"gid":24442,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.162:6835/1957208984","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6834","nonce":1957208984},{"type":"v1","addr":"172.21.15.162:6835","nonce":1957208984}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24451":{"gid":24451,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":11,"addr":"172.21.15.162:6837/1371886876","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6836","nonce":1371886876},{"type":"v1","addr":"172.21.15.162:6837","nonce":1371886876}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24455":{"gid":24455,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.142:6835/212637431","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6834","nonce":212637431},{"type":"v1","addr":"172.21.15.142:6835","nonce":212637431}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24478":{"gid":24478,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.162:6839/1976275826","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6838","nonce":1976275826},{"type":"v1","addr":"172.21.15.162:6839","nonce":1976275826}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24482":{"gid":24482,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":10,"addr":"172.21.15.142:6837/2559258399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6836","nonce":2559258399},{"type":"v1","addr":"172.21.15.142:6837","nonce":2559258399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24515":{"gid":24515,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.142:6839/2519143918","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6838","nonce":2519143918},{"type":"v1","addr":"172.21.15.142:6839","nonce":2519143918}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24515,"qdb_cluster":[24451,24482,24515]},"id":1}]} 2024-03-31T23:13:10.042 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 27 2024-03-31T23:13:10.044 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.043+0000 7fc6a089a640 1 -- 172.21.15.46:0/2608545930 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fc6840a2730 msgr2=0x7fc6840a4bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:13:10.044 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.043+0000 7fc6a089a640 1 --2- 172.21.15.46:0/2608545930 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fc6840a2730 0x7fc6840a4bf0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fc690009b50 tx=0x7fc690005f90 comp rx=0 tx=0).stop 2024-03-31T23:13:10.044 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.043+0000 7fc6a089a640 1 -- 172.21.15.46:0/2608545930 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc69c106660 msgr2=0x7fc69c19bc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:13:10.044 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.043+0000 7fc6a089a640 1 --2- 172.21.15.46:0/2608545930 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc69c106660 0x7fc69c19bc00 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7fc69400efd0 tx=0x7fc69400c6a0 comp rx=0 tx=0).stop 2024-03-31T23:13:10.044 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.043+0000 7fc6a089a640 1 -- 172.21.15.46:0/2608545930 shutdown_connections 2024-03-31T23:13:10.044 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.043+0000 7fc6a089a640 1 --2- 172.21.15.46:0/2608545930 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fc6840a2730 0x7fc6840a4bf0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:10.044 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.043+0000 7fc6a089a640 1 --2- 172.21.15.46:0/2608545930 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc69c106660 0x7fc69c19bc00 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:10.044 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.043+0000 7fc6a089a640 1 --2- 172.21.15.46:0/2608545930 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc69c101010 0x7fc69c1970f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:10.044 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.043+0000 7fc6a089a640 1 --2- 172.21.15.46:0/2608545930 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc69c100650 0x7fc69c196bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:10.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.043+0000 7fc6a089a640 1 -- 172.21.15.46:0/2608545930 >> 172.21.15.46:0/2608545930 conn(0x7fc69c077750 msgr2=0x7fc69c103150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:13:10.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.043+0000 7fc6a089a640 1 -- 172.21.15.46:0/2608545930 shutdown_connections 2024-03-31T23:13:10.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.043+0000 7fc6a089a640 1 -- 172.21.15.46:0/2608545930 wait complete. 2024-03-31T23:13:10.055 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-31T23:13:10.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.159+0000 7febd5f25640 1 -- 172.21.15.46:0/1649448605 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7febd0100f80 msgr2=0x7febd010d490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:13:10.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.159+0000 7febd5f25640 1 --2- 172.21.15.46:0/1649448605 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7febd0100f80 0x7febd010d490 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7febc8009ca0 tx=0x7febc802c0f0 comp rx=0 tx=0).stop 2024-03-31T23:13:10.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.159+0000 7febd5f25640 1 -- 172.21.15.46:0/1649448605 shutdown_connections 2024-03-31T23:13:10.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.159+0000 7febd5f25640 1 --2- 172.21.15.46:0/1649448605 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7febd0100f80 0x7febd010d490 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:10.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.160+0000 7febd5f25640 1 --2- 172.21.15.46:0/1649448605 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7febd01005c0 0x7febd0100a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:10.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.160+0000 7febd5f25640 1 --2- 172.21.15.46:0/1649448605 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7febd0106660 0x7febd0106a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:10.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.160+0000 7febd5f25640 1 -- 172.21.15.46:0/1649448605 >> 172.21.15.46:0/1649448605 conn(0x7febd0077750 msgr2=0x7febd0077b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:13:10.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.160+0000 7febd5f25640 1 -- 172.21.15.46:0/1649448605 shutdown_connections 2024-03-31T23:13:10.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.160+0000 7febd5f25640 1 -- 172.21.15.46:0/1649448605 wait complete. 2024-03-31T23:13:10.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.160+0000 7febd5f25640 1 Processor -- start 2024-03-31T23:13:10.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.160+0000 7febd5f25640 1 -- start start 2024-03-31T23:13:10.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.160+0000 7febd5f25640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7febd01005c0 0x7febd0197e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:13:10.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.160+0000 7febd5f25640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7febd0100f80 0x7febd0198350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:13:10.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.160+0000 7febd5f25640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7febd0106660 0x7febd019c6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:13:10.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.160+0000 7febd5f25640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7febd0110890 con 0x7febd0106660 2024-03-31T23:13:10.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.160+0000 7febd5f25640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7febd0110710 con 0x7febd01005c0 2024-03-31T23:13:10.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.160+0000 7febd5f25640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7febd0110a10 con 0x7febd0100f80 2024-03-31T23:13:10.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.161+0000 7febcffff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7febd0106660 0x7febd019c6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:13:10.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.161+0000 7febcf7fe640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7febd01005c0 0x7febd0197e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:13:10.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.161+0000 7febcffff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7febd0106660 0x7febd019c6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:48902/0 (socket says 172.21.15.46:48902) 2024-03-31T23:13:10.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.161+0000 7febcffff640 1 -- 172.21.15.46:0/244732508 learned_addr learned my addr 172.21.15.46:0/244732508 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:13:10.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.161+0000 7febcf7fe640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7febd01005c0 0x7febd0197e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.46:58388/0 (socket says 172.21.15.46:58388) 2024-03-31T23:13:10.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.161+0000 7febceffd640 1 --2- 172.21.15.46:0/244732508 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7febd0100f80 0x7febd0198350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:13:10.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.161+0000 7febcffff640 1 -- 172.21.15.46:0/244732508 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7febd0100f80 msgr2=0x7febd0198350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:13:10.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.161+0000 7febcffff640 1 --2- 172.21.15.46:0/244732508 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7febd0100f80 0x7febd0198350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:10.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.161+0000 7febcffff640 1 -- 172.21.15.46:0/244732508 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7febd01005c0 msgr2=0x7febd0197e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:13:10.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.161+0000 7febcffff640 1 --2- 172.21.15.46:0/244732508 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7febd01005c0 0x7febd0197e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:10.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.161+0000 7febcffff640 1 -- 172.21.15.46:0/244732508 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7febd019ce00 con 0x7febd0106660 2024-03-31T23:13:10.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.161+0000 7febcf7fe640 1 --2- 172.21.15.46:0/244732508 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7febd01005c0 0x7febd0197e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:13:10.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.161+0000 7febcffff640 1 --2- 172.21.15.46:0/244732508 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7febd0106660 0x7febd019c6d0 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7febc802c700 tx=0x7febc8035d60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:13:10.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.161+0000 7febccff9640 1 -- 172.21.15.46:0/244732508 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febc8039040 con 0x7febd0106660 2024-03-31T23:13:10.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.161+0000 7febccff9640 1 -- 172.21.15.46:0/244732508 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7febc8044070 con 0x7febd0106660 2024-03-31T23:13:10.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.161+0000 7febd5f25640 1 -- 172.21.15.46:0/244732508 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7febd019d090 con 0x7febd0106660 2024-03-31T23:13:10.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.161+0000 7febccff9640 1 -- 172.21.15.46:0/244732508 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febc80042a0 con 0x7febd0106660 2024-03-31T23:13:10.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.161+0000 7febd5f25640 1 -- 172.21.15.46:0/244732508 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7febd01a4980 con 0x7febd0106660 2024-03-31T23:13:10.171 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.164+0000 7febccff9640 1 -- 172.21.15.46:0/244732508 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 24) ==== 137553+0+0 (secure 0 0 0) 0x7febc8004440 con 0x7febd0106660 2024-03-31T23:13:10.171 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.164+0000 7febd8be7740 1 Processor -- start 2024-03-31T23:13:10.171 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.164+0000 7febd8be7740 1 -- start start 2024-03-31T23:13:10.171 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.164+0000 7febd8be7740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558767992470 0x558767992850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:13:10.171 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.164+0000 7febccff9640 1 --2- 172.21.15.46:0/244732508 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7febb80a2730 0x7febb80a4bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:13:10.171 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.164+0000 7febd8be7740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558767992d90 0x55876799b430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:13:10.171 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.164+0000 7febd8be7740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55876799b970 0x55876799dd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:13:10.171 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.164+0000 7febd8be7740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x5587678f61b0 con 0x558767992d90 2024-03-31T23:13:10.171 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.164+0000 7febd8be7740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55876789ec00 con 0x558767992470 2024-03-31T23:13:10.171 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.164+0000 7febd8be7740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5587678f2250 con 0x55876799b970 2024-03-31T23:13:10.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.164+0000 7febceffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558767992d90 0x55876799b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:13:10.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.164+0000 7febcf7fe640 1 --2- 172.21.15.46:0/244732508 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7febb80a2730 0x7febb80a4bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:13:10.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.164+0000 7febceffd640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558767992470 0x558767992850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:13:10.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.164+0000 7febceffd640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558767992470 0x558767992850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.46:58390/0 (socket says 172.21.15.46:58390) 2024-03-31T23:13:10.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.164+0000 7febccff9640 1 -- 172.21.15.46:0/244732508 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7febc80f14e0 con 0x7febd0106660 2024-03-31T23:13:10.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.164+0000 7febceffd640 1 -- 172.21.15.46:0/2100554140 learned_addr learned my addr 172.21.15.46:0/2100554140 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:13:10.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.164+0000 7febcffff640 1 --2- 172.21.15.46:0/2100554140 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55876799b970 0x55876799dd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:13:10.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.165+0000 7febceffd640 1 -- 172.21.15.46:0/2100554140 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55876799b970 msgr2=0x55876799dd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:13:10.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.165+0000 7febceffd640 1 --2- 172.21.15.46:0/2100554140 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55876799b970 0x55876799dd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:10.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.165+0000 7febceffd640 1 -- 172.21.15.46:0/2100554140 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558767992d90 msgr2=0x55876799b430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:13:10.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.165+0000 7febceffd640 1 --2- 172.21.15.46:0/2100554140 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558767992d90 0x55876799b430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:10.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.165+0000 7febceffd640 1 -- 172.21.15.46:0/2100554140 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5587678be3d0 con 0x558767992470 2024-03-31T23:13:10.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.165+0000 7febcf7fe640 1 --2- 172.21.15.46:0/244732508 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7febb80a2730 0x7febb80a4bf0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7febd0101700 tx=0x7febc0005f90 comp rx=0 tx=0).ready entity=mgr.15654 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:13:10.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.165+0000 7febceffd640 1 --2- 172.21.15.46:0/2100554140 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558767992470 0x558767992850 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7febc4009730 tx=0x7febc4005f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=924e1515bf5ebbfa server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:13:10.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.165+0000 7febb5ffb640 1 -- 172.21.15.46:0/2100554140 <== mon.2 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febc4016070 con 0x558767992470 2024-03-31T23:13:10.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.165+0000 7febb5ffb640 1 -- 172.21.15.46:0/2100554140 <== mon.2 v2:172.21.15.142:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7febc400f7c0 con 0x558767992470 2024-03-31T23:13:10.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.165+0000 7febb5ffb640 1 -- 172.21.15.46:0/2100554140 <== mon.2 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febc4019040 con 0x558767992470 2024-03-31T23:13:10.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.166+0000 7febd8be7740 1 -- 172.21.15.46:0/2100554140 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558767992470 msgr2=0x558767992850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:13:10.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.166+0000 7febd8be7740 1 --2- 172.21.15.46:0/2100554140 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558767992470 0x558767992850 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7febc4009730 tx=0x7febc4005f90 comp rx=0 tx=0).stop 2024-03-31T23:13:10.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.166+0000 7febd8be7740 1 -- 172.21.15.46:0/2100554140 shutdown_connections 2024-03-31T23:13:10.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.166+0000 7febd8be7740 1 --2- 172.21.15.46:0/2100554140 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55876799b970 0x55876799dd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:10.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.166+0000 7febd8be7740 1 --2- 172.21.15.46:0/2100554140 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558767992d90 0x55876799b430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:10.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.166+0000 7febd8be7740 1 --2- 172.21.15.46:0/2100554140 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558767992470 0x558767992850 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:10.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.166+0000 7febd8be7740 1 -- 172.21.15.46:0/2100554140 >> 172.21.15.46:0/2100554140 conn(0x55876798cba0 msgr2=0x55876798cf80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:13:10.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.166+0000 7febd8be7740 1 -- 172.21.15.46:0/2100554140 shutdown_connections 2024-03-31T23:13:10.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.166+0000 7febd8be7740 1 -- 172.21.15.46:0/2100554140 wait complete. 2024-03-31T23:13:10.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.166+0000 7febd8be7740 1 -- start start 2024-03-31T23:13:10.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.166+0000 7febd8be7740 1 Processor -- start 2024-03-31T23:13:10.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.167+0000 7febd8be7740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558767992470 0x5587679989f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:13:10.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.167+0000 7febd8be7740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558767992d90 0x558767998f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:13:10.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.167+0000 7febd8be7740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55876799b970 0x558767999470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:13:10.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.167+0000 7febd8be7740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55876799e760 con 0x55876799b970 2024-03-31T23:13:10.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.167+0000 7febd8be7740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x558767894470 con 0x558767992470 2024-03-31T23:13:10.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.167+0000 7febd8be7740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55876799e8e0 con 0x558767992d90 2024-03-31T23:13:10.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.167+0000 7febd8be7740 10 client.0 _ll_register_callbacks cb 0x5587678b1cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T23:13:10.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.167+0000 7febcffff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55876799b970 0x558767999470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:13:10.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.167+0000 7febcf7fe640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558767992470 0x5587679989f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:13:10.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.167+0000 7febcf7fe640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558767992470 0x5587679989f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.46:58406/0 (socket says 172.21.15.46:58406) 2024-03-31T23:13:10.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.167+0000 7febcf7fe640 1 -- 172.21.15.46:0/401059139 learned_addr learned my addr 172.21.15.46:0/401059139 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:13:10.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.167+0000 7febcffff640 1 --2- 172.21.15.46:0/401059139 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55876799b970 0x558767999470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:48922/0 (socket says 172.21.15.46:48922) 2024-03-31T23:13:10.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.167+0000 7febceffd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558767992d90 0x558767998f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:13:10.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.167+0000 7febcffff640 1 -- 172.21.15.46:0/401059139 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558767992d90 msgr2=0x558767998f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:13:10.175 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.167+0000 7febcffff640 1 --2- 172.21.15.46:0/401059139 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558767992d90 0x558767998f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:10.175 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.167+0000 7febcffff640 1 -- 172.21.15.46:0/401059139 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558767992470 msgr2=0x5587679989f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:13:10.175 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.167+0000 7febcffff640 1 --2- 172.21.15.46:0/401059139 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558767992470 0x5587679989f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:10.175 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.167+0000 7febcffff640 1 -- 172.21.15.46:0/401059139 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55876760a3f0 con 0x55876799b970 2024-03-31T23:13:10.175 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.167+0000 7febcf7fe640 1 --2- 172.21.15.46:0/401059139 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558767992470 0x5587679989f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:13:10.175 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.167+0000 7febcffff640 1 --2- 172.21.15.46:0/401059139 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55876799b970 0x558767999470 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7febc80ee810 tx=0x7febc80bd330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b43d65cd84f5491 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:13:10.175 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.167+0000 7febaeffd640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-31T23:13:10.175 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.167+0000 7febaeffd640 1 -- 172.21.15.46:0/401059139 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febc8034700 con 0x55876799b970 2024-03-31T23:13:10.175 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.168+0000 7febd8be7740 1 -- 172.21.15.46:0/401059139 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7febc400f7c0 con 0x55876799b970 2024-03-31T23:13:10.175 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.168+0000 7febaeffd640 1 -- 172.21.15.46:0/401059139 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7febc8004730 con 0x55876799b970 2024-03-31T23:13:10.175 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.168+0000 7febaeffd640 1 -- 172.21.15.46:0/401059139 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7febc80bde70 con 0x55876799b970 2024-03-31T23:13:10.175 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.168+0000 7febd8be7740 10 client.16107 fetch_fsmap learned FSMap version 27 2024-03-31T23:13:10.175 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.168+0000 7febd8be7740 1 -- 172.21.15.46:0/401059139 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5587676d47d0 con 0x55876799b970 2024-03-31T23:13:10.175 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.168+0000 7febaeffd640 1 -- 172.21.15.46:0/401059139 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7febc400f7c0 con 0x55876799b970 2024-03-31T23:13:10.175 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.168+0000 7febaeffd640 1 -- 172.21.15.46:0/401059139 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7febc803c030 con 0x55876799b970 2024-03-31T23:13:10.176 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.168+0000 7febd8be7740 10 client.16107 fetch_fsmap finished waiting for FSMap version 27 2024-03-31T23:13:10.176 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.168+0000 7febd8be7740 10 client.16107 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-31T23:13:10.176 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.168+0000 7febd8be7740 20 client.16107 populate_metadata read hostname 'smithi046' 2024-03-31T23:13:10.176 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.168+0000 7febd8be7740 1 --2- 172.21.15.46:0/401059139 >> [v2:172.21.15.162:6836/1371886876,v1:172.21.15.162:6837/1371886876] conn(0x5587679ab9d0 0x5587679addf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:13:10.176 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.168+0000 7febd8be7740 4 client.16107 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-31T23:13:10.176 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.168+0000 7febd8be7740 1 -- 172.21.15.46:0/401059139 --> [v2:172.21.15.162:6836/1371886876,v1:172.21.15.162:6837/1371886876] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5587679036f0 con 0x5587679ab9d0 2024-03-31T23:13:10.176 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.168+0000 7febcf7fe640 1 --2- 172.21.15.46:0/401059139 >> [v2:172.21.15.162:6836/1371886876,v1:172.21.15.162:6837/1371886876] conn(0x5587679ab9d0 0x5587679addf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:13:10.176 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.169+0000 7febcf7fe640 1 --2- 172.21.15.46:0/401059139 >> [v2:172.21.15.162:6836/1371886876,v1:172.21.15.162:6837/1371886876] conn(0x5587679ab9d0 0x5587679addf0 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=69b7ad280cb53a3f server_cookie=b1072fd3d990120f in_seq=0 out_seq=0 2024-03-31T23:13:10.176 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.169+0000 7febaeffd640 10 client.16107 ms_handle_connect on v2:172.21.15.162:6836/1371886876 2024-03-31T23:13:10.176 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.169+0000 7febaeffd640 1 -- 172.21.15.46:0/401059139 <== mds.2 v2:172.21.15.162:6836/1371886876 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5587679036f0 con 0x5587679ab9d0 2024-03-31T23:13:10.176 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.169+0000 7febaeffd640 10 client.16107 handle_command_reply: tid=0 2024-03-31T23:13:10.176 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.169+0000 7febd8be7740 1 client.16107 shutdown 2024-03-31T23:13:10.176 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.170+0000 7febd8be7740 1 -- 172.21.15.46:0/401059139 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55876799b970 msgr2=0x558767999470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:13:10.176 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.170+0000 7febd8be7740 1 --2- 172.21.15.46:0/401059139 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55876799b970 0x558767999470 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7febc80ee810 tx=0x7febc80bd330 comp rx=0 tx=0).stop 2024-03-31T23:13:10.177 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.170+0000 7febd8be7740 1 -- 172.21.15.46:0/401059139 shutdown_connections 2024-03-31T23:13:10.177 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.170+0000 7febd8be7740 1 --2- 172.21.15.46:0/401059139 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55876799b970 0x558767999470 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:10.177 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.170+0000 7febd8be7740 1 --2- 172.21.15.46:0/401059139 >> [v2:172.21.15.162:6836/1371886876,v1:172.21.15.162:6837/1371886876] conn(0x5587679ab9d0 0x5587679addf0 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:10.177 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.170+0000 7febd8be7740 1 --2- 172.21.15.46:0/401059139 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558767992d90 0x558767998f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:10.177 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.170+0000 7febd8be7740 1 --2- 172.21.15.46:0/401059139 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558767992470 0x5587679989f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:10.177 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.170+0000 7febd8be7740 1 -- 172.21.15.46:0/401059139 >> 172.21.15.46:0/401059139 conn(0x55876798cba0 msgr2=0x55876798e8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:13:10.177 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.170+0000 7febaeffd640 0 client.16107 ms_handle_reset on v2:172.21.15.162:6836/1371886876 2024-03-31T23:13:10.177 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.170+0000 7febd8be7740 1 -- 172.21.15.46:0/401059139 shutdown_connections 2024-03-31T23:13:10.177 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.170+0000 7febd8be7740 1 -- 172.21.15.46:0/401059139 wait complete. 2024-03-31T23:13:10.177 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.170+0000 7febd8be7740 20 client.16107 trim_cache size 0 max 16384 2024-03-31T23:13:10.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.183+0000 7febd8be7740 1 Processor -- start 2024-03-31T23:13:10.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.183+0000 7febd8be7740 1 -- start start 2024-03-31T23:13:10.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.183+0000 7febd8be7740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558767992470 0x55876799b120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:13:10.186 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.183+0000 7febd8be7740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558767992d90 0x558767999290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:13:10.186 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.183+0000 7febd8be7740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55876799b970 0x5587679b3700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:13:10.186 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.183+0000 7febd8be7740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7febd01b12e0 con 0x558767992d90 2024-03-31T23:13:10.186 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.183+0000 7febd8be7740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7febd01aebb0 con 0x55876799b970 2024-03-31T23:13:10.186 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.183+0000 7febd8be7740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7febd01aca00 con 0x558767992470 2024-03-31T23:13:10.186 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.183+0000 7febceffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558767992d90 0x558767999290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:13:10.186 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.184+0000 7febcf7fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558767992470 0x55876799b120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:13:10.186 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.184+0000 7febceffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558767992d90 0x558767999290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:48936/0 (socket says 172.21.15.46:48936) 2024-03-31T23:13:10.186 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.184+0000 7febceffd640 1 -- 172.21.15.46:0/2454379937 learned_addr learned my addr 172.21.15.46:0/2454379937 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:13:10.186 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.184+0000 7febcffff640 1 --2- 172.21.15.46:0/2454379937 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55876799b970 0x5587679b3700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:13:10.186 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.184+0000 7febcf7fe640 1 -- 172.21.15.46:0/2454379937 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55876799b970 msgr2=0x5587679b3700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:13:10.186 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.184+0000 7febcf7fe640 1 --2- 172.21.15.46:0/2454379937 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55876799b970 0x5587679b3700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:10.186 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.184+0000 7febcf7fe640 1 -- 172.21.15.46:0/2454379937 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558767992d90 msgr2=0x558767999290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:13:10.186 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.184+0000 7febcf7fe640 1 --2- 172.21.15.46:0/2454379937 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558767992d90 0x558767999290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:10.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.184+0000 7febcf7fe640 1 -- 172.21.15.46:0/2454379937 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5587679ae470 con 0x558767992470 2024-03-31T23:13:10.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.184+0000 7febceffd640 1 --2- 172.21.15.46:0/2454379937 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558767992d90 0x558767999290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:13:10.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.184+0000 7febcffff640 1 --2- 172.21.15.46:0/2454379937 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55876799b970 0x5587679b3700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:13:10.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.184+0000 7febcf7fe640 1 --2- 172.21.15.46:0/2454379937 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558767992470 0x55876799b120 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x5587679ae640 tx=0x7febc0014000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8d1a515f41417ebd server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:13:10.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.184+0000 7febb57fa640 1 -- 172.21.15.46:0/2454379937 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febc0003d40 con 0x558767992470 2024-03-31T23:13:10.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.184+0000 7febb57fa640 1 -- 172.21.15.46:0/2454379937 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7febc0003ee0 con 0x558767992470 2024-03-31T23:13:10.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.184+0000 7febb57fa640 1 -- 172.21.15.46:0/2454379937 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febc000f810 con 0x558767992470 2024-03-31T23:13:10.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.185+0000 7febd8be7740 1 -- 172.21.15.46:0/2454379937 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558767992470 msgr2=0x55876799b120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:13:10.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.185+0000 7febd8be7740 1 --2- 172.21.15.46:0/2454379937 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558767992470 0x55876799b120 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x5587679ae640 tx=0x7febc0014000 comp rx=0 tx=0).stop 2024-03-31T23:13:10.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.185+0000 7febd8be7740 1 -- 172.21.15.46:0/2454379937 shutdown_connections 2024-03-31T23:13:10.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.185+0000 7febd8be7740 1 --2- 172.21.15.46:0/2454379937 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55876799b970 0x5587679b3700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:10.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.185+0000 7febd8be7740 1 --2- 172.21.15.46:0/2454379937 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558767992d90 0x558767999290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:10.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.185+0000 7febd8be7740 1 --2- 172.21.15.46:0/2454379937 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558767992470 0x55876799b120 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:10.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.185+0000 7febd8be7740 1 -- 172.21.15.46:0/2454379937 >> 172.21.15.46:0/2454379937 conn(0x55876798cba0 msgr2=0x55876798b4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:13:10.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.185+0000 7febd8be7740 1 -- 172.21.15.46:0/2454379937 shutdown_connections 2024-03-31T23:13:10.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.185+0000 7febd8be7740 1 -- 172.21.15.46:0/2454379937 wait complete. 2024-03-31T23:13:10.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.185+0000 7febd8be7740 1 -- start start 2024-03-31T23:13:10.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.186+0000 7febd8be7740 1 Processor -- start 2024-03-31T23:13:10.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.186+0000 7febd8be7740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558767992470 0x558767995700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:13:10.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.186+0000 7febd8be7740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558767992d90 0x558767995c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:13:10.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.186+0000 7febd8be7740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55876799b970 0x558767996180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:13:10.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.186+0000 7febd8be7740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7febd01ad070 con 0x558767992470 2024-03-31T23:13:10.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.186+0000 7febd8be7740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7febd01ae4e0 con 0x558767992d90 2024-03-31T23:13:10.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.186+0000 7febd8be7740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7febd01ad740 con 0x55876799b970 2024-03-31T23:13:10.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.186+0000 7febd8be7740 10 client.0 _ll_register_callbacks cb 0x7febc80bd540 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T23:13:10.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.186+0000 7febcf7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558767992470 0x558767995700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:13:10.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.186+0000 7febcf7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558767992470 0x558767995700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:48946/0 (socket says 172.21.15.46:48946) 2024-03-31T23:13:10.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.186+0000 7febcf7fe640 1 -- 172.21.15.46:0/1662693114 learned_addr learned my addr 172.21.15.46:0/1662693114 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:13:10.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.186+0000 7febcffff640 1 --2- 172.21.15.46:0/1662693114 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55876799b970 0x558767996180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:13:10.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.186+0000 7febceffd640 1 --2- 172.21.15.46:0/1662693114 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558767992d90 0x558767995c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:13:10.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.186+0000 7febcf7fe640 1 -- 172.21.15.46:0/1662693114 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55876799b970 msgr2=0x558767996180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:13:10.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.186+0000 7febcf7fe640 1 --2- 172.21.15.46:0/1662693114 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55876799b970 0x558767996180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:10.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.186+0000 7febcf7fe640 1 -- 172.21.15.46:0/1662693114 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558767992d90 msgr2=0x558767995c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:13:10.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.186+0000 7febcf7fe640 1 --2- 172.21.15.46:0/1662693114 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558767992d90 0x558767995c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:10.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.186+0000 7febcf7fe640 1 -- 172.21.15.46:0/1662693114 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558767a10ee0 con 0x558767992470 2024-03-31T23:13:10.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.186+0000 7febcf7fe640 1 --2- 172.21.15.46:0/1662693114 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558767992470 0x558767995700 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7febc0005bc0 tx=0x7febc000f480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3311c6295b8b3c2e server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:13:10.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.186+0000 7febaeffd640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-31T23:13:10.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.187+0000 7febaeffd640 1 -- 172.21.15.46:0/1662693114 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febc0005390 con 0x558767992470 2024-03-31T23:13:10.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.187+0000 7febaeffd640 1 -- 172.21.15.46:0/1662693114 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7febc0005530 con 0x558767992470 2024-03-31T23:13:10.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.187+0000 7febd8be7740 1 -- 172.21.15.46:0/1662693114 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7febc0003ee0 con 0x558767992470 2024-03-31T23:13:10.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.187+0000 7febaeffd640 1 -- 172.21.15.46:0/1662693114 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7febc00046c0 con 0x558767992470 2024-03-31T23:13:10.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.187+0000 7febd8be7740 10 client.16113 fetch_fsmap learned FSMap version 27 2024-03-31T23:13:10.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.187+0000 7febd8be7740 1 -- 172.21.15.46:0/1662693114 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x558767a121a0 con 0x558767992470 2024-03-31T23:13:10.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.187+0000 7febaeffd640 1 -- 172.21.15.46:0/1662693114 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7febc0003ee0 con 0x558767992470 2024-03-31T23:13:10.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.187+0000 7febaeffd640 1 -- 172.21.15.46:0/1662693114 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7febc0029580 con 0x558767992470 2024-03-31T23:13:10.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.187+0000 7febd8be7740 10 client.16113 fetch_fsmap finished waiting for FSMap version 27 2024-03-31T23:13:10.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.187+0000 7febd8be7740 10 client.16113 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-31T23:13:10.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.187+0000 7febd8be7740 20 client.16113 populate_metadata read hostname 'smithi046' 2024-03-31T23:13:10.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.187+0000 7febd8be7740 1 --2- 172.21.15.46:0/1662693114 >> [v2:172.21.15.162:6836/1371886876,v1:172.21.15.162:6837/1371886876] conn(0x558767a12910 0x558767a14d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:13:10.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.187+0000 7febd8be7740 4 client.16113 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-31T23:13:10.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.187+0000 7febd8be7740 1 -- 172.21.15.46:0/1662693114 --> [v2:172.21.15.162:6836/1371886876,v1:172.21.15.162:6837/1371886876] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5587678fda40 con 0x558767a12910 2024-03-31T23:13:10.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.187+0000 7febceffd640 1 --2- 172.21.15.46:0/1662693114 >> [v2:172.21.15.162:6836/1371886876,v1:172.21.15.162:6837/1371886876] conn(0x558767a12910 0x558767a14d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:13:10.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.188+0000 7febaeffd640 10 client.16113 ms_handle_connect on v2:172.21.15.162:6836/1371886876 2024-03-31T23:13:10.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.188+0000 7febceffd640 1 --2- 172.21.15.46:0/1662693114 >> [v2:172.21.15.162:6836/1371886876,v1:172.21.15.162:6837/1371886876] conn(0x558767a12910 0x558767a14d00 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=a4b063095dacf1b8 server_cookie=9c68194a7b0adb03 in_seq=0 out_seq=0 2024-03-31T23:13:10.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.188+0000 7febaeffd640 1 -- 172.21.15.46:0/1662693114 <== mds.2 v2:172.21.15.162:6836/1371886876 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5587678fda40 con 0x558767a12910 2024-03-31T23:13:10.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.188+0000 7febaeffd640 10 client.16113 handle_command_reply: tid=0 2024-03-31T23:13:10.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.188+0000 7febd8be7740 1 client.16113 shutdown 2024-03-31T23:13:10.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.188+0000 7febd8be7740 1 -- 172.21.15.46:0/1662693114 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558767992470 msgr2=0x558767995700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:13:10.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.188+0000 7febd8be7740 1 --2- 172.21.15.46:0/1662693114 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558767992470 0x558767995700 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7febc0005bc0 tx=0x7febc000f480 comp rx=0 tx=0).stop 2024-03-31T23:13:10.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.188+0000 7febd8be7740 1 -- 172.21.15.46:0/1662693114 shutdown_connections 2024-03-31T23:13:10.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.188+0000 7febd8be7740 1 --2- 172.21.15.46:0/1662693114 >> [v2:172.21.15.162:6836/1371886876,v1:172.21.15.162:6837/1371886876] conn(0x558767a12910 0x558767a14d00 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:10.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.188+0000 7febd8be7740 1 --2- 172.21.15.46:0/1662693114 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55876799b970 0x558767996180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:10.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.188+0000 7febd8be7740 1 --2- 172.21.15.46:0/1662693114 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558767992d90 0x558767995c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:10.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.188+0000 7febaeffd640 0 client.16113 ms_handle_reset on v2:172.21.15.162:6836/1371886876 2024-03-31T23:13:10.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.188+0000 7febd8be7740 1 --2- 172.21.15.46:0/1662693114 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558767992470 0x558767995700 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:10.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.188+0000 7febd8be7740 1 -- 172.21.15.46:0/1662693114 >> 172.21.15.46:0/1662693114 conn(0x55876798cba0 msgr2=0x55876798bd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:13:10.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.188+0000 7febd8be7740 1 -- 172.21.15.46:0/1662693114 shutdown_connections 2024-03-31T23:13:10.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.188+0000 7febd8be7740 1 -- 172.21.15.46:0/1662693114 wait complete. 2024-03-31T23:13:10.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.188+0000 7febd8be7740 20 client.16113 trim_cache size 0 max 16384 2024-03-31T23:13:10.196 INFO:teuthology.orchestra.run.smithi046.stdout:[] 2024-03-31T23:13:10.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.189+0000 7febd5f25640 1 -- 172.21.15.46:0/244732508 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7febb80a2730 msgr2=0x7febb80a4bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:13:10.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.189+0000 7febd5f25640 1 --2- 172.21.15.46:0/244732508 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7febb80a2730 0x7febb80a4bf0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7febd0101700 tx=0x7febc0005f90 comp rx=0 tx=0).stop 2024-03-31T23:13:10.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.189+0000 7febd5f25640 1 -- 172.21.15.46:0/244732508 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7febd0106660 msgr2=0x7febd019c6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:13:10.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.189+0000 7febd5f25640 1 --2- 172.21.15.46:0/244732508 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7febd0106660 0x7febd019c6d0 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7febc802c700 tx=0x7febc8035d60 comp rx=0 tx=0).stop 2024-03-31T23:13:10.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.189+0000 7febd5f25640 1 -- 172.21.15.46:0/244732508 shutdown_connections 2024-03-31T23:13:10.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.189+0000 7febd5f25640 1 --2- 172.21.15.46:0/244732508 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7febb80a2730 0x7febb80a4bf0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:10.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.189+0000 7febd5f25640 1 --2- 172.21.15.46:0/244732508 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7febd0106660 0x7febd019c6d0 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:10.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.189+0000 7febd5f25640 1 --2- 172.21.15.46:0/244732508 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7febd0100f80 0x7febd0198350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:10.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.189+0000 7febd5f25640 1 --2- 172.21.15.46:0/244732508 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7febd01005c0 0x7febd0197e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:10.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.189+0000 7febd5f25640 1 -- 172.21.15.46:0/244732508 >> 172.21.15.46:0/244732508 conn(0x7febd0077750 msgr2=0x7febd00ff840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:13:10.197 INFO:tasks.workunit.client.0.smithi046.stdout: 1 8101 3.76 MB/sec warmup 16 sec latency 2456.540 ms 2024-03-31T23:13:10.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.189+0000 7febd5f25640 1 -- 172.21.15.46:0/244732508 shutdown_connections 2024-03-31T23:13:10.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.189+0000 7febd5f25640 1 -- 172.21.15.46:0/244732508 wait complete. 2024-03-31T23:13:10.200 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-31T23:13:10.308 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.305+0000 7f3ebd548640 1 -- 172.21.15.46:0/3653103221 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3eb8100f80 msgr2=0x7f3eb810d490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:13:10.308 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.305+0000 7f3ebd548640 1 --2- 172.21.15.46:0/3653103221 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3eb8100f80 0x7f3eb810d490 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f3ea8009be0 tx=0x7f3ea802bef0 comp rx=0 tx=0).stop 2024-03-31T23:13:10.308 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.305+0000 7f3ebd548640 1 -- 172.21.15.46:0/3653103221 shutdown_connections 2024-03-31T23:13:10.308 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.305+0000 7f3ebd548640 1 --2- 172.21.15.46:0/3653103221 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3eb8100f80 0x7f3eb810d490 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:10.308 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.305+0000 7f3ebd548640 1 --2- 172.21.15.46:0/3653103221 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3eb81005c0 0x7f3eb8100a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:10.308 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.305+0000 7f3ebd548640 1 --2- 172.21.15.46:0/3653103221 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3eb8106660 0x7f3eb8106a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:10.308 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.305+0000 7f3ebd548640 1 -- 172.21.15.46:0/3653103221 >> 172.21.15.46:0/3653103221 conn(0x7f3eb8077750 msgr2=0x7f3eb8077b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:13:10.308 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.305+0000 7f3ebd548640 1 -- 172.21.15.46:0/3653103221 shutdown_connections 2024-03-31T23:13:10.308 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.305+0000 7f3ebd548640 1 -- 172.21.15.46:0/3653103221 wait complete. 2024-03-31T23:13:10.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.305+0000 7f3ebd548640 1 Processor -- start 2024-03-31T23:13:10.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.306+0000 7f3ebd548640 1 -- start start 2024-03-31T23:13:10.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.306+0000 7f3ebd548640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3eb81005c0 0x7f3eb8198c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:13:10.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.306+0000 7f3ebd548640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3eb8100f80 0x7f3eb8199190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:13:10.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.306+0000 7f3ebd548640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3eb8106660 0x7f3eb81a3cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:13:10.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.306+0000 7f3ebd548640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f3eb8110850 con 0x7f3eb81005c0 2024-03-31T23:13:10.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.306+0000 7f3ebd548640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f3eb81106d0 con 0x7f3eb8100f80 2024-03-31T23:13:10.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.306+0000 7f3ebd548640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f3eb81109d0 con 0x7f3eb8106660 2024-03-31T23:13:10.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.306+0000 7f3eb6ffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3eb81005c0 0x7f3eb8198c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:13:10.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.306+0000 7f3eb6ffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3eb81005c0 0x7f3eb8198c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:48966/0 (socket says 172.21.15.46:48966) 2024-03-31T23:13:10.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.306+0000 7f3eb6ffd640 1 -- 172.21.15.46:0/629207103 learned_addr learned my addr 172.21.15.46:0/629207103 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:13:10.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.306+0000 7f3eb67fc640 1 --2- 172.21.15.46:0/629207103 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3eb8100f80 0x7f3eb8199190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:13:10.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.306+0000 7f3eb77fe640 1 --2- 172.21.15.46:0/629207103 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3eb8106660 0x7f3eb81a3cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:13:10.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.306+0000 7f3eb6ffd640 1 -- 172.21.15.46:0/629207103 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3eb8106660 msgr2=0x7f3eb81a3cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:13:10.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.306+0000 7f3eb6ffd640 1 --2- 172.21.15.46:0/629207103 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3eb8106660 0x7f3eb81a3cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:10.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.306+0000 7f3eb6ffd640 1 -- 172.21.15.46:0/629207103 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3eb8100f80 msgr2=0x7f3eb8199190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:13:10.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.306+0000 7f3eb6ffd640 1 --2- 172.21.15.46:0/629207103 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3eb8100f80 0x7f3eb8199190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:10.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.306+0000 7f3eb6ffd640 1 -- 172.21.15.46:0/629207103 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3eb81a4210 con 0x7f3eb81005c0 2024-03-31T23:13:10.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.306+0000 7f3eb6ffd640 1 --2- 172.21.15.46:0/629207103 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3eb81005c0 0x7f3eb8198c50 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f3eb000e9c0 tx=0x7f3eb000ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:13:10.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.306+0000 7f3eaffff640 1 -- 172.21.15.46:0/629207103 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3eb000cd80 con 0x7f3eb81005c0 2024-03-31T23:13:10.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.307+0000 7f3ebd548640 1 -- 172.21.15.46:0/629207103 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3eb81a43e0 con 0x7f3eb81005c0 2024-03-31T23:13:10.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.307+0000 7f3eaffff640 1 -- 172.21.15.46:0/629207103 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3eb00035f0 con 0x7f3eb81005c0 2024-03-31T23:13:10.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.307+0000 7f3eaffff640 1 -- 172.21.15.46:0/629207103 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3eb0010690 con 0x7f3eb81005c0 2024-03-31T23:13:10.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.307+0000 7f3ebd548640 1 -- 172.21.15.46:0/629207103 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3eb81a4980 con 0x7f3eb81005c0 2024-03-31T23:13:10.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.309+0000 7f3eaffff640 1 -- 172.21.15.46:0/629207103 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 24) ==== 137553+0+0 (secure 0 0 0) 0x7f3eb00037a0 con 0x7f3eb81005c0 2024-03-31T23:13:10.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.309+0000 7f3ec0208740 1 Processor -- start 2024-03-31T23:13:10.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.309+0000 7f3ec0208740 1 -- start start 2024-03-31T23:13:10.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.309+0000 7f3ec0208740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d8bc2ea1a0 0x55d8bc2ea580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:13:10.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.309+0000 7f3ec0208740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d8bc2eaac0 0x55d8bc2f7840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:13:10.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.309+0000 7f3ec0208740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d8bc2f7d80 0x55d8bc2fa170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:13:10.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.310+0000 7f3ec0208740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55d8bc24e1b0 con 0x55d8bc2ea1a0 2024-03-31T23:13:10.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.310+0000 7f3ec0208740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55d8bc1f6c00 con 0x55d8bc2eaac0 2024-03-31T23:13:10.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.310+0000 7f3ec0208740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55d8bc24a250 con 0x55d8bc2f7d80 2024-03-31T23:13:10.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.310+0000 7f3eb77fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d8bc2ea1a0 0x55d8bc2ea580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:13:10.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.310+0000 7f3eb6ffd640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d8bc2eaac0 0x55d8bc2f7840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:13:10.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.310+0000 7f3eb77fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d8bc2ea1a0 0x55d8bc2ea580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:48974/0 (socket says 172.21.15.46:48974) 2024-03-31T23:13:10.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.310+0000 7f3eb77fe640 1 -- 172.21.15.46:0/3831211350 learned_addr learned my addr 172.21.15.46:0/3831211350 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:13:10.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.310+0000 7f3eb67fc640 1 --2- 172.21.15.46:0/3831211350 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d8bc2f7d80 0x55d8bc2fa170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:13:10.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.310+0000 7f3eaffff640 1 --2- 172.21.15.46:0/629207103 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f3ea40a2730 0x7f3ea40a4bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:13:10.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.310+0000 7f3eb77fe640 1 --2- 172.21.15.46:0/629207103 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f3ea40a2730 0x7f3ea40a4bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:13:10.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.310+0000 7f3eb6ffd640 1 -- 172.21.15.46:0/3831211350 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d8bc2f7d80 msgr2=0x55d8bc2fa170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:13:10.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.310+0000 7f3eb6ffd640 1 --2- 172.21.15.46:0/3831211350 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d8bc2f7d80 0x55d8bc2fa170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:10.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.310+0000 7f3eb6ffd640 1 -- 172.21.15.46:0/3831211350 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d8bc2ea1a0 msgr2=0x55d8bc2ea580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:13:10.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.310+0000 7f3eb6ffd640 1 --2- 172.21.15.46:0/3831211350 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d8bc2ea1a0 0x55d8bc2ea580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:10.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.310+0000 7f3eb6ffd640 1 -- 172.21.15.46:0/3831211350 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d8bc2163d0 con 0x55d8bc2eaac0 2024-03-31T23:13:10.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.310+0000 7f3eb67fc640 1 --2- 172.21.15.46:0/3831211350 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d8bc2f7d80 0x55d8bc2fa170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:13:10.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.310+0000 7f3eaffff640 1 -- 172.21.15.46:0/629207103 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7f3eb00cf4c0 con 0x7f3eb81005c0 2024-03-31T23:13:10.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.310+0000 7f3eb6ffd640 1 --2- 172.21.15.46:0/3831211350 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d8bc2eaac0 0x55d8bc2f7840 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f3eb00cc6e0 tx=0x7f3eb0021000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=afd286a70dc522d0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:13:10.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.310+0000 7f3eb77fe640 1 --2- 172.21.15.46:0/629207103 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f3ea40a2730 0x7f3ea40a4bf0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f3ea8009be0 tx=0x7f3ea802bef0 comp rx=0 tx=0).ready entity=mgr.15654 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:13:10.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.310+0000 7f3ead7fa640 1 -- 172.21.15.46:0/3831211350 <== mon.2 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3eb0003b30 con 0x55d8bc2eaac0 2024-03-31T23:13:10.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.311+0000 7f3ead7fa640 1 -- 172.21.15.46:0/3831211350 <== mon.2 v2:172.21.15.142:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3eb00d2e20 con 0x55d8bc2eaac0 2024-03-31T23:13:10.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.311+0000 7f3ead7fa640 1 -- 172.21.15.46:0/3831211350 <== mon.2 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3eb000ab20 con 0x55d8bc2eaac0 2024-03-31T23:13:10.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.311+0000 7f3ec0208740 1 -- 172.21.15.46:0/3831211350 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d8bc2eaac0 msgr2=0x55d8bc2f7840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:13:10.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.311+0000 7f3ec0208740 1 --2- 172.21.15.46:0/3831211350 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d8bc2eaac0 0x55d8bc2f7840 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f3eb00cc6e0 tx=0x7f3eb0021000 comp rx=0 tx=0).stop 2024-03-31T23:13:10.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.311+0000 7f3ec0208740 1 -- 172.21.15.46:0/3831211350 shutdown_connections 2024-03-31T23:13:10.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.311+0000 7f3ec0208740 1 --2- 172.21.15.46:0/3831211350 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d8bc2f7d80 0x55d8bc2fa170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:10.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.311+0000 7f3ec0208740 1 --2- 172.21.15.46:0/3831211350 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d8bc2eaac0 0x55d8bc2f7840 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:10.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.311+0000 7f3ec0208740 1 --2- 172.21.15.46:0/3831211350 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d8bc2ea1a0 0x55d8bc2ea580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:10.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.311+0000 7f3ec0208740 1 -- 172.21.15.46:0/3831211350 >> 172.21.15.46:0/3831211350 conn(0x55d8bc2e4ba0 msgr2=0x55d8bc2e4f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:13:10.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.311+0000 7f3ec0208740 1 -- 172.21.15.46:0/3831211350 shutdown_connections 2024-03-31T23:13:10.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.311+0000 7f3ec0208740 1 -- 172.21.15.46:0/3831211350 wait complete. 2024-03-31T23:13:10.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.312+0000 7f3ec0208740 1 -- start start 2024-03-31T23:13:10.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.312+0000 7f3ec0208740 1 Processor -- start 2024-03-31T23:13:10.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.312+0000 7f3ec0208740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d8bc2ea1a0 0x55d8bc2f4df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:13:10.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.312+0000 7f3ec0208740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d8bc2eaac0 0x55d8bc2f5330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:13:10.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.312+0000 7f3ec0208740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d8bc2f7d80 0x55d8bc2f5870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:13:10.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.312+0000 7f3ec0208740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55d8bc2faae0 con 0x55d8bc2f7d80 2024-03-31T23:13:10.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.312+0000 7f3ec0208740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55d8bc1ec470 con 0x55d8bc2ea1a0 2024-03-31T23:13:10.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.312+0000 7f3ec0208740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55d8bc2fac60 con 0x55d8bc2eaac0 2024-03-31T23:13:10.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.312+0000 7f3ec0208740 10 client.0 _ll_register_callbacks cb 0x55d8bc209cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T23:13:10.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.312+0000 7f3eb77fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d8bc2f7d80 0x55d8bc2f5870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:13:10.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.312+0000 7f3eb6ffd640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d8bc2ea1a0 0x55d8bc2f4df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:13:10.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.312+0000 7f3eb67fc640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d8bc2eaac0 0x55d8bc2f5330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:13:10.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.312+0000 7f3eb6ffd640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d8bc2ea1a0 0x55d8bc2f4df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.46:58458/0 (socket says 172.21.15.46:58458) 2024-03-31T23:13:10.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.312+0000 7f3eb6ffd640 1 -- 172.21.15.46:0/482713433 learned_addr learned my addr 172.21.15.46:0/482713433 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:13:10.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.312+0000 7f3eb77fe640 1 -- 172.21.15.46:0/482713433 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d8bc2eaac0 msgr2=0x55d8bc2f5330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:13:10.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.312+0000 7f3eb77fe640 1 --2- 172.21.15.46:0/482713433 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d8bc2eaac0 0x55d8bc2f5330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:10.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.312+0000 7f3eb77fe640 1 -- 172.21.15.46:0/482713433 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d8bc2ea1a0 msgr2=0x55d8bc2f4df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:13:10.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.312+0000 7f3eb77fe640 1 --2- 172.21.15.46:0/482713433 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d8bc2ea1a0 0x55d8bc2f4df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:10.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.312+0000 7f3eb77fe640 1 -- 172.21.15.46:0/482713433 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d8bbf623f0 con 0x55d8bc2f7d80 2024-03-31T23:13:10.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.313+0000 7f3eb77fe640 1 --2- 172.21.15.46:0/482713433 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d8bc2f7d80 0x55d8bc2f5870 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f3ea8006010 tx=0x7f3ea8009500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=33ca6ae40d364a54 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:13:10.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.313+0000 7f3e9e7fc640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-31T23:13:10.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.313+0000 7f3e9e7fc640 1 -- 172.21.15.46:0/482713433 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ea8049040 con 0x55d8bc2f7d80 2024-03-31T23:13:10.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.313+0000 7f3ec0208740 1 -- 172.21.15.46:0/482713433 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3eb00d2e20 con 0x55d8bc2f7d80 2024-03-31T23:13:10.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.313+0000 7f3e9e7fc640 1 -- 172.21.15.46:0/482713433 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7f3ea8003d40 con 0x55d8bc2f7d80 2024-03-31T23:13:10.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.313+0000 7f3e9e7fc640 1 -- 172.21.15.46:0/482713433 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f3ea8004b40 con 0x55d8bc2f7d80 2024-03-31T23:13:10.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.313+0000 7f3ec0208740 10 client.16125 fetch_fsmap learned FSMap version 27 2024-03-31T23:13:10.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.313+0000 7f3ec0208740 1 -- 172.21.15.46:0/482713433 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55d8bc02c7d0 con 0x55d8bc2f7d80 2024-03-31T23:13:10.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.313+0000 7f3e9e7fc640 1 -- 172.21.15.46:0/482713433 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3eb00d2e20 con 0x55d8bc2f7d80 2024-03-31T23:13:10.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.313+0000 7f3e9e7fc640 1 -- 172.21.15.46:0/482713433 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f3ea804c030 con 0x55d8bc2f7d80 2024-03-31T23:13:10.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.313+0000 7f3ec0208740 10 client.16125 fetch_fsmap finished waiting for FSMap version 27 2024-03-31T23:13:10.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.313+0000 7f3ec0208740 10 client.16125 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-03-31T23:13:10.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.313+0000 7f3ec0208740 20 client.16125 populate_metadata read hostname 'smithi046' 2024-03-31T23:13:10.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.313+0000 7f3ec0208740 1 --2- 172.21.15.46:0/482713433 >> [v2:172.21.15.142:6836/2559258399,v1:172.21.15.142:6837/2559258399] conn(0x55d8bc307da0 0x55d8bc30a240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:13:10.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.313+0000 7f3ec0208740 4 client.16125 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-31T23:13:10.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.313+0000 7f3ec0208740 1 -- 172.21.15.46:0/482713433 --> [v2:172.21.15.142:6836/2559258399,v1:172.21.15.142:6837/2559258399] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d8bc25b6f0 con 0x55d8bc307da0 2024-03-31T23:13:10.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.314+0000 7f3eb6ffd640 1 --2- 172.21.15.46:0/482713433 >> [v2:172.21.15.142:6836/2559258399,v1:172.21.15.142:6837/2559258399] conn(0x55d8bc307da0 0x55d8bc30a240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:13:10.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.314+0000 7f3eb6ffd640 1 --2- 172.21.15.46:0/482713433 >> [v2:172.21.15.142:6836/2559258399,v1:172.21.15.142:6837/2559258399] conn(0x55d8bc307da0 0x55d8bc30a240 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=65a84b6e0140c851 server_cookie=a8122a9a611019c0 in_seq=0 out_seq=0 2024-03-31T23:13:10.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.314+0000 7f3e9e7fc640 10 client.16125 ms_handle_connect on v2:172.21.15.142:6836/2559258399 2024-03-31T23:13:10.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.315+0000 7f3e9e7fc640 1 -- 172.21.15.46:0/482713433 <== mds.1 v2:172.21.15.142:6836/2559258399 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55d8bc25b6f0 con 0x55d8bc307da0 2024-03-31T23:13:10.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.315+0000 7f3e9e7fc640 10 client.16125 handle_command_reply: tid=0 2024-03-31T23:13:10.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.315+0000 7f3ec0208740 1 client.16125 shutdown 2024-03-31T23:13:10.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.315+0000 7f3ec0208740 1 -- 172.21.15.46:0/482713433 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d8bc2f7d80 msgr2=0x55d8bc2f5870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:13:10.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.315+0000 7f3ec0208740 1 --2- 172.21.15.46:0/482713433 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d8bc2f7d80 0x55d8bc2f5870 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f3ea8006010 tx=0x7f3ea8009500 comp rx=0 tx=0).stop 2024-03-31T23:13:10.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.315+0000 7f3ec0208740 1 -- 172.21.15.46:0/482713433 shutdown_connections 2024-03-31T23:13:10.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.315+0000 7f3ec0208740 1 --2- 172.21.15.46:0/482713433 >> [v2:172.21.15.142:6836/2559258399,v1:172.21.15.142:6837/2559258399] conn(0x55d8bc307da0 0x55d8bc30a240 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:10.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.315+0000 7f3ec0208740 1 --2- 172.21.15.46:0/482713433 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d8bc2f7d80 0x55d8bc2f5870 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:10.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.315+0000 7f3ec0208740 1 --2- 172.21.15.46:0/482713433 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d8bc2eaac0 0x55d8bc2f5330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:10.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.315+0000 7f3ec0208740 1 --2- 172.21.15.46:0/482713433 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d8bc2ea1a0 0x55d8bc2f4df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:10.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.315+0000 7f3e9e7fc640 0 client.16125 ms_handle_reset on v2:172.21.15.142:6836/2559258399 2024-03-31T23:13:10.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.315+0000 7f3ec0208740 1 -- 172.21.15.46:0/482713433 >> 172.21.15.46:0/482713433 conn(0x55d8bc2e4ba0 msgr2=0x55d8bc2e68f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:13:10.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.315+0000 7f3ec0208740 1 -- 172.21.15.46:0/482713433 shutdown_connections 2024-03-31T23:13:10.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.315+0000 7f3ec0208740 1 -- 172.21.15.46:0/482713433 wait complete. 2024-03-31T23:13:10.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.315+0000 7f3ec0208740 20 client.16125 trim_cache size 0 max 16384 2024-03-31T23:13:10.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.329+0000 7f3ec0208740 1 Processor -- start 2024-03-31T23:13:10.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.329+0000 7f3ec0208740 1 -- start start 2024-03-31T23:13:10.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.329+0000 7f3ec0208740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d8bc2ea1a0 0x55d8bc2e3f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:13:10.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.329+0000 7f3ec0208740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d8bc2eaac0 0x55d8bc3038c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:13:10.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.329+0000 7f3ec0208740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d8bc2f7d80 0x55d8bc2f45a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:13:10.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.329+0000 7f3ec0208740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f3eb81af340 con 0x55d8bc2ea1a0 2024-03-31T23:13:10.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.329+0000 7f3ec0208740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f3eb81acc10 con 0x55d8bc2eaac0 2024-03-31T23:13:10.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.329+0000 7f3ec0208740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f3eb81aaa60 con 0x55d8bc2f7d80 2024-03-31T23:13:10.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.329+0000 7f3eb6ffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d8bc2ea1a0 0x55d8bc2e3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:13:10.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.329+0000 7f3eb6ffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d8bc2ea1a0 0x55d8bc2e3f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:48986/0 (socket says 172.21.15.46:48986) 2024-03-31T23:13:10.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.329+0000 7f3eb6ffd640 1 -- 172.21.15.46:0/674804364 learned_addr learned my addr 172.21.15.46:0/674804364 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:13:10.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.329+0000 7f3eb67fc640 1 --2- 172.21.15.46:0/674804364 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d8bc2eaac0 0x55d8bc3038c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:13:10.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.329+0000 7f3eb77fe640 1 --2- 172.21.15.46:0/674804364 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d8bc2f7d80 0x55d8bc2f45a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:13:10.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.329+0000 7f3eb6ffd640 1 -- 172.21.15.46:0/674804364 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d8bc2f7d80 msgr2=0x55d8bc2f45a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:13:10.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.329+0000 7f3eb6ffd640 1 --2- 172.21.15.46:0/674804364 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d8bc2f7d80 0x55d8bc2f45a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:10.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.329+0000 7f3eb6ffd640 1 -- 172.21.15.46:0/674804364 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d8bc2eaac0 msgr2=0x55d8bc3038c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:13:10.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.329+0000 7f3eb6ffd640 1 --2- 172.21.15.46:0/674804364 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d8bc2eaac0 0x55d8bc3038c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:10.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.329+0000 7f3eb6ffd640 1 -- 172.21.15.46:0/674804364 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d8bc2f09b0 con 0x55d8bc2ea1a0 2024-03-31T23:13:10.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.329+0000 7f3eb67fc640 1 --2- 172.21.15.46:0/674804364 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d8bc2eaac0 0x55d8bc3038c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:13:10.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.330+0000 7f3eb6ffd640 1 --2- 172.21.15.46:0/674804364 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d8bc2ea1a0 0x55d8bc2e3f80 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f3eb0000c00 tx=0x7f3eb0003e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=134747d50c5d17ff server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:13:10.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.330+0000 7f3eacff9640 1 -- 172.21.15.46:0/674804364 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3eb00b3020 con 0x55d8bc2ea1a0 2024-03-31T23:13:10.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.330+0000 7f3eacff9640 1 -- 172.21.15.46:0/674804364 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3eb009f790 con 0x55d8bc2ea1a0 2024-03-31T23:13:10.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.330+0000 7f3eacff9640 1 -- 172.21.15.46:0/674804364 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3eb000ae40 con 0x55d8bc2ea1a0 2024-03-31T23:13:10.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.330+0000 7f3ec0208740 1 -- 172.21.15.46:0/674804364 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d8bc2ea1a0 msgr2=0x55d8bc2e3f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:13:10.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.330+0000 7f3ec0208740 1 --2- 172.21.15.46:0/674804364 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d8bc2ea1a0 0x55d8bc2e3f80 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f3eb0000c00 tx=0x7f3eb0003e30 comp rx=0 tx=0).stop 2024-03-31T23:13:10.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.331+0000 7f3ec0208740 1 -- 172.21.15.46:0/674804364 shutdown_connections 2024-03-31T23:13:10.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.331+0000 7f3ec0208740 1 --2- 172.21.15.46:0/674804364 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d8bc2f7d80 0x55d8bc2f45a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:10.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.331+0000 7f3ec0208740 1 --2- 172.21.15.46:0/674804364 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d8bc2eaac0 0x55d8bc3038c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:10.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.331+0000 7f3ec0208740 1 --2- 172.21.15.46:0/674804364 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d8bc2ea1a0 0x55d8bc2e3f80 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:10.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.331+0000 7f3ec0208740 1 -- 172.21.15.46:0/674804364 >> 172.21.15.46:0/674804364 conn(0x55d8bc2e4ba0 msgr2=0x55d8bc2f6da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:13:10.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.331+0000 7f3ec0208740 1 -- 172.21.15.46:0/674804364 shutdown_connections 2024-03-31T23:13:10.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.331+0000 7f3ec0208740 1 -- 172.21.15.46:0/674804364 wait complete. 2024-03-31T23:13:10.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.331+0000 7f3ec0208740 1 -- start start 2024-03-31T23:13:10.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.331+0000 7f3ec0208740 1 Processor -- start 2024-03-31T23:13:10.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.331+0000 7f3ec0208740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d8bc2ea1a0 0x55d8bc300270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:13:10.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.331+0000 7f3ec0208740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d8bc2eaac0 0x55d8bc2fb590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:13:10.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.331+0000 7f3ec0208740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d8bc2f7d80 0x55d8bc2fbb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:13:10.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.331+0000 7f3ec0208740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f3eb81ab0d0 con 0x55d8bc2eaac0 2024-03-31T23:13:10.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.331+0000 7f3ec0208740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f3eb81ac540 con 0x55d8bc2f7d80 2024-03-31T23:13:10.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.331+0000 7f3ec0208740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f3eb81ab7a0 con 0x55d8bc2ea1a0 2024-03-31T23:13:10.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.331+0000 7f3ec0208740 10 client.0 _ll_register_callbacks cb 0x7f3ea8033e20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T23:13:10.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.331+0000 7f3eb67fc640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d8bc2eaac0 0x55d8bc2fb590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:13:10.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.331+0000 7f3eb67fc640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d8bc2eaac0 0x55d8bc2fb590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:48988/0 (socket says 172.21.15.46:48988) 2024-03-31T23:13:10.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.331+0000 7f3eb67fc640 1 -- 172.21.15.46:0/2792501777 learned_addr learned my addr 172.21.15.46:0/2792501777 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:13:10.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.331+0000 7f3eb6ffd640 1 --2- 172.21.15.46:0/2792501777 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d8bc2ea1a0 0x55d8bc300270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:13:10.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.331+0000 7f3eb77fe640 1 --2- 172.21.15.46:0/2792501777 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d8bc2f7d80 0x55d8bc2fbb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:13:10.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.332+0000 7f3eb67fc640 1 -- 172.21.15.46:0/2792501777 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d8bc2ea1a0 msgr2=0x55d8bc300270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:13:10.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.332+0000 7f3eb67fc640 1 --2- 172.21.15.46:0/2792501777 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d8bc2ea1a0 0x55d8bc300270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:10.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.332+0000 7f3eb67fc640 1 -- 172.21.15.46:0/2792501777 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d8bc2f7d80 msgr2=0x55d8bc2fbb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:13:10.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.332+0000 7f3eb67fc640 1 --2- 172.21.15.46:0/2792501777 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d8bc2f7d80 0x55d8bc2fbb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:10.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.332+0000 7f3eb67fc640 1 -- 172.21.15.46:0/2792501777 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d8bc2fc360 con 0x55d8bc2eaac0 2024-03-31T23:13:10.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.332+0000 7f3eb67fc640 1 --2- 172.21.15.46:0/2792501777 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d8bc2eaac0 0x55d8bc2fb590 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f3ea0009530 tx=0x7f3ea0010e50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ba1f4041caf9745 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:13:10.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.332+0000 7f3e9e7fc640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-31T23:13:10.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.332+0000 7f3e9e7fc640 1 -- 172.21.15.46:0/2792501777 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ea0021030 con 0x55d8bc2eaac0 2024-03-31T23:13:10.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.332+0000 7f3ec0208740 1 -- 172.21.15.46:0/2792501777 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3eb009f790 con 0x55d8bc2eaac0 2024-03-31T23:13:10.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.332+0000 7f3e9e7fc640 1 -- 172.21.15.46:0/2792501777 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7f3ea000fb20 con 0x55d8bc2eaac0 2024-03-31T23:13:10.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.332+0000 7f3e9e7fc640 1 -- 172.21.15.46:0/2792501777 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f3ea002b050 con 0x55d8bc2eaac0 2024-03-31T23:13:10.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.332+0000 7f3ec0208740 10 client.16131 fetch_fsmap learned FSMap version 27 2024-03-31T23:13:10.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.332+0000 7f3ec0208740 1 -- 172.21.15.46:0/2792501777 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55d8bc36e2a0 con 0x55d8bc2eaac0 2024-03-31T23:13:10.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.332+0000 7f3e9e7fc640 1 -- 172.21.15.46:0/2792501777 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3eb009f790 con 0x55d8bc2eaac0 2024-03-31T23:13:10.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.333+0000 7f3e9e7fc640 1 -- 172.21.15.46:0/2792501777 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f3ea0026400 con 0x55d8bc2eaac0 2024-03-31T23:13:10.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.333+0000 7f3ec0208740 10 client.16131 fetch_fsmap finished waiting for FSMap version 27 2024-03-31T23:13:10.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.333+0000 7f3ec0208740 10 client.16131 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-03-31T23:13:10.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.333+0000 7f3ec0208740 20 client.16131 populate_metadata read hostname 'smithi046' 2024-03-31T23:13:10.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.333+0000 7f3ec0208740 1 --2- 172.21.15.46:0/2792501777 >> [v2:172.21.15.142:6836/2559258399,v1:172.21.15.142:6837/2559258399] conn(0x55d8bc36ead0 0x55d8bc370ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:13:10.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.333+0000 7f3ec0208740 4 client.16131 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-31T23:13:10.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.333+0000 7f3ec0208740 1 -- 172.21.15.46:0/2792501777 --> [v2:172.21.15.142:6836/2559258399,v1:172.21.15.142:6837/2559258399] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55d8bc255a40 con 0x55d8bc36ead0 2024-03-31T23:13:10.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.333+0000 7f3eb6ffd640 1 --2- 172.21.15.46:0/2792501777 >> [v2:172.21.15.142:6836/2559258399,v1:172.21.15.142:6837/2559258399] conn(0x55d8bc36ead0 0x55d8bc370ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:13:10.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.333+0000 7f3eb6ffd640 1 --2- 172.21.15.46:0/2792501777 >> [v2:172.21.15.142:6836/2559258399,v1:172.21.15.142:6837/2559258399] conn(0x55d8bc36ead0 0x55d8bc370ef0 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=7edae9f631028f1f server_cookie=319260893bb6fa65 in_seq=0 out_seq=0 2024-03-31T23:13:10.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.333+0000 7f3e9e7fc640 10 client.16131 ms_handle_connect on v2:172.21.15.142:6836/2559258399 2024-03-31T23:13:10.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.333+0000 7f3e9e7fc640 1 -- 172.21.15.46:0/2792501777 <== mds.1 v2:172.21.15.142:6836/2559258399 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55d8bc255a40 con 0x55d8bc36ead0 2024-03-31T23:13:10.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.333+0000 7f3e9e7fc640 10 client.16131 handle_command_reply: tid=0 2024-03-31T23:13:10.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.333+0000 7f3ec0208740 1 client.16131 shutdown 2024-03-31T23:13:10.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.333+0000 7f3ec0208740 1 -- 172.21.15.46:0/2792501777 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d8bc2eaac0 msgr2=0x55d8bc2fb590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:13:10.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.333+0000 7f3ec0208740 1 --2- 172.21.15.46:0/2792501777 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d8bc2eaac0 0x55d8bc2fb590 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f3ea0009530 tx=0x7f3ea0010e50 comp rx=0 tx=0).stop 2024-03-31T23:13:10.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.333+0000 7f3ec0208740 1 -- 172.21.15.46:0/2792501777 shutdown_connections 2024-03-31T23:13:10.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.333+0000 7f3ec0208740 1 --2- 172.21.15.46:0/2792501777 >> [v2:172.21.15.142:6836/2559258399,v1:172.21.15.142:6837/2559258399] conn(0x55d8bc36ead0 0x55d8bc370ef0 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:10.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.334+0000 7f3ec0208740 1 --2- 172.21.15.46:0/2792501777 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d8bc2f7d80 0x55d8bc2fbb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:10.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.334+0000 7f3e9e7fc640 0 client.16131 ms_handle_reset on v2:172.21.15.142:6836/2559258399 2024-03-31T23:13:10.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.334+0000 7f3ec0208740 1 --2- 172.21.15.46:0/2792501777 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d8bc2eaac0 0x55d8bc2fb590 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:10.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.334+0000 7f3ec0208740 1 --2- 172.21.15.46:0/2792501777 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d8bc2ea1a0 0x55d8bc300270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:10.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.334+0000 7f3ec0208740 1 -- 172.21.15.46:0/2792501777 >> 172.21.15.46:0/2792501777 conn(0x55d8bc2e4ba0 msgr2=0x55d8bc2f6430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:13:10.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.334+0000 7f3ec0208740 1 -- 172.21.15.46:0/2792501777 shutdown_connections 2024-03-31T23:13:10.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.334+0000 7f3ec0208740 1 -- 172.21.15.46:0/2792501777 wait complete. 2024-03-31T23:13:10.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.334+0000 7f3ec0208740 20 client.16131 trim_cache size 0 max 16384 2024-03-31T23:13:10.341 INFO:teuthology.orchestra.run.smithi046.stdout:[] 2024-03-31T23:13:10.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.334+0000 7f3ebd548640 1 -- 172.21.15.46:0/629207103 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f3ea40a2730 msgr2=0x7f3ea40a4bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:13:10.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.334+0000 7f3ebd548640 1 --2- 172.21.15.46:0/629207103 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f3ea40a2730 0x7f3ea40a4bf0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f3ea8009be0 tx=0x7f3ea802bef0 comp rx=0 tx=0).stop 2024-03-31T23:13:10.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.334+0000 7f3ebd548640 1 -- 172.21.15.46:0/629207103 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3eb81005c0 msgr2=0x7f3eb8198c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:13:10.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.334+0000 7f3ebd548640 1 --2- 172.21.15.46:0/629207103 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3eb81005c0 0x7f3eb8198c50 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f3eb000e9c0 tx=0x7f3eb000ee80 comp rx=0 tx=0).stop 2024-03-31T23:13:10.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.334+0000 7f3ebd548640 1 -- 172.21.15.46:0/629207103 shutdown_connections 2024-03-31T23:13:10.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.334+0000 7f3ebd548640 1 --2- 172.21.15.46:0/629207103 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f3ea40a2730 0x7f3ea40a4bf0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:10.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.334+0000 7f3ebd548640 1 --2- 172.21.15.46:0/629207103 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3eb8106660 0x7f3eb81a3cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:10.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.334+0000 7f3ebd548640 1 --2- 172.21.15.46:0/629207103 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3eb8100f80 0x7f3eb8199190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:10.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.334+0000 7f3ebd548640 1 --2- 172.21.15.46:0/629207103 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3eb81005c0 0x7f3eb8198c50 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:10.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.334+0000 7f3ebd548640 1 -- 172.21.15.46:0/629207103 >> 172.21.15.46:0/629207103 conn(0x7f3eb8077750 msgr2=0x7f3eb80ff150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:13:10.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.334+0000 7f3ebd548640 1 -- 172.21.15.46:0/629207103 shutdown_connections 2024-03-31T23:13:10.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.334+0000 7f3ebd548640 1 -- 172.21.15.46:0/629207103 wait complete. 2024-03-31T23:13:10.349 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-31T23:13:10.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.455+0000 7fcb97c8f640 1 -- 172.21.15.46:0/1035754403 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcb90100f80 msgr2=0x7fcb9010d490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:13:10.459 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.455+0000 7fcb97c8f640 1 --2- 172.21.15.46:0/1035754403 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcb90100f80 0x7fcb9010d490 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fcb88009c10 tx=0x7fcb8802bef0 comp rx=0 tx=0).stop 2024-03-31T23:13:10.459 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.455+0000 7fcb97c8f640 1 -- 172.21.15.46:0/1035754403 shutdown_connections 2024-03-31T23:13:10.459 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.455+0000 7fcb97c8f640 1 --2- 172.21.15.46:0/1035754403 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcb90100f80 0x7fcb9010d490 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:10.459 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.455+0000 7fcb97c8f640 1 --2- 172.21.15.46:0/1035754403 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcb901005c0 0x7fcb90100a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:10.459 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.455+0000 7fcb97c8f640 1 --2- 172.21.15.46:0/1035754403 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcb90106660 0x7fcb90106a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:10.459 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.455+0000 7fcb97c8f640 1 -- 172.21.15.46:0/1035754403 >> 172.21.15.46:0/1035754403 conn(0x7fcb90077750 msgr2=0x7fcb90077b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:13:10.459 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.455+0000 7fcb97c8f640 1 -- 172.21.15.46:0/1035754403 shutdown_connections 2024-03-31T23:13:10.459 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.455+0000 7fcb97c8f640 1 -- 172.21.15.46:0/1035754403 wait complete. 2024-03-31T23:13:10.459 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.456+0000 7fcb97c8f640 1 Processor -- start 2024-03-31T23:13:10.459 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.456+0000 7fcb97c8f640 1 -- start start 2024-03-31T23:13:10.459 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.456+0000 7fcb97c8f640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcb901005c0 0x7fcb90198d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:13:10.459 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.456+0000 7fcb97c8f640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcb90100f80 0x7fcb90199290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:13:10.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.456+0000 7fcb97c8f640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcb90106660 0x7fcb901a3dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:13:10.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.456+0000 7fcb97c8f640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fcb90110850 con 0x7fcb90100f80 2024-03-31T23:13:10.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.456+0000 7fcb97c8f640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fcb901106d0 con 0x7fcb90106660 2024-03-31T23:13:10.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.456+0000 7fcb97c8f640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fcb901109d0 con 0x7fcb901005c0 2024-03-31T23:13:10.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.456+0000 7fcb9522a640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcb90100f80 0x7fcb90199290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:13:10.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.456+0000 7fcb95a2b640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcb901005c0 0x7fcb90198d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:13:10.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.456+0000 7fcb9522a640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcb90100f80 0x7fcb90199290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:49014/0 (socket says 172.21.15.46:49014) 2024-03-31T23:13:10.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.456+0000 7fcb9522a640 1 -- 172.21.15.46:0/224416416 learned_addr learned my addr 172.21.15.46:0/224416416 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:13:10.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.456+0000 7fcb9622c640 1 --2- 172.21.15.46:0/224416416 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcb90106660 0x7fcb901a3dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:13:10.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.456+0000 7fcb95a2b640 1 -- 172.21.15.46:0/224416416 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcb90106660 msgr2=0x7fcb901a3dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:13:10.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.456+0000 7fcb95a2b640 1 --2- 172.21.15.46:0/224416416 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcb90106660 0x7fcb901a3dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:10.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.456+0000 7fcb95a2b640 1 -- 172.21.15.46:0/224416416 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcb90100f80 msgr2=0x7fcb90199290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:13:10.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.456+0000 7fcb95a2b640 1 --2- 172.21.15.46:0/224416416 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcb90100f80 0x7fcb90199290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:10.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.456+0000 7fcb95a2b640 1 -- 172.21.15.46:0/224416416 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcb901a4310 con 0x7fcb901005c0 2024-03-31T23:13:10.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.456+0000 7fcb9522a640 1 --2- 172.21.15.46:0/224416416 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcb90100f80 0x7fcb90199290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:13:10.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.456+0000 7fcb9622c640 1 --2- 172.21.15.46:0/224416416 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcb90106660 0x7fcb901a3dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:13:10.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.457+0000 7fcb95a2b640 1 --2- 172.21.15.46:0/224416416 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcb901005c0 0x7fcb90198d50 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fcb8000e9c0 tx=0x7fcb8000ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:13:10.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.457+0000 7fcb86ffd640 1 -- 172.21.15.46:0/224416416 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb8000ccc0 con 0x7fcb901005c0 2024-03-31T23:13:10.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.457+0000 7fcb86ffd640 1 -- 172.21.15.46:0/224416416 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fcb800035f0 con 0x7fcb901005c0 2024-03-31T23:13:10.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.457+0000 7fcb97c8f640 1 -- 172.21.15.46:0/224416416 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcb901a44e0 con 0x7fcb901005c0 2024-03-31T23:13:10.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.457+0000 7fcb86ffd640 1 -- 172.21.15.46:0/224416416 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb8000ce60 con 0x7fcb901005c0 2024-03-31T23:13:10.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.457+0000 7fcb97c8f640 1 -- 172.21.15.46:0/224416416 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcb901a49b0 con 0x7fcb901005c0 2024-03-31T23:13:10.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.459+0000 7fcb9a875740 1 Processor -- start 2024-03-31T23:13:10.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.460+0000 7fcb9a875740 1 -- start start 2024-03-31T23:13:10.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.460+0000 7fcb86ffd640 1 -- 172.21.15.46:0/224416416 <== mon.1 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 24) ==== 137553+0+0 (secure 0 0 0) 0x7fcb800037a0 con 0x7fcb901005c0 2024-03-31T23:13:10.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.460+0000 7fcb9a875740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557bbb1e1470 0x557bbb1e1850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:13:10.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.460+0000 7fcb9a875740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557bbb1e1d90 0x557bbb1ea430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:13:10.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.460+0000 7fcb9a875740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557bbb1ea970 0x557bbb1ecd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:13:10.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.460+0000 7fcb9a875740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x557bbb1451b0 con 0x557bbb1e1470 2024-03-31T23:13:10.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.460+0000 7fcb9a875740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x557bbb0edc00 con 0x557bbb1e1d90 2024-03-31T23:13:10.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.460+0000 7fcb9a875740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x557bbb141250 con 0x557bbb1ea970 2024-03-31T23:13:10.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.460+0000 7fcb9622c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557bbb1e1470 0x557bbb1e1850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:13:10.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.460+0000 7fcb9522a640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557bbb1ea970 0x557bbb1ecd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:13:10.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.460+0000 7fcb95a2b640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557bbb1e1d90 0x557bbb1ea430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:13:10.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.460+0000 7fcb9622c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557bbb1e1470 0x557bbb1e1850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:49016/0 (socket says 172.21.15.46:49016) 2024-03-31T23:13:10.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.460+0000 7fcb9622c640 1 -- 172.21.15.46:0/720762482 learned_addr learned my addr 172.21.15.46:0/720762482 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:13:10.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.460+0000 7fcb9522a640 1 -- 172.21.15.46:0/720762482 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557bbb1e1d90 msgr2=0x557bbb1ea430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:13:10.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.460+0000 7fcb9522a640 1 --2- 172.21.15.46:0/720762482 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557bbb1e1d90 0x557bbb1ea430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:10.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.460+0000 7fcb9522a640 1 -- 172.21.15.46:0/720762482 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557bbb1e1470 msgr2=0x557bbb1e1850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:13:10.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.460+0000 7fcb9522a640 1 --2- 172.21.15.46:0/720762482 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557bbb1e1470 0x557bbb1e1850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:10.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.460+0000 7fcb9522a640 1 -- 172.21.15.46:0/720762482 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557bbb10d3d0 con 0x557bbb1ea970 2024-03-31T23:13:10.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.461+0000 7fcb95a2b640 1 --2- 172.21.15.46:0/720762482 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557bbb1e1d90 0x557bbb1ea430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:13:10.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.461+0000 7fcb9522a640 1 --2- 172.21.15.46:0/720762482 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557bbb1ea970 0x557bbb1ecd60 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fcb8c0058c0 tx=0x7fcb8c0119a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c9989920dc7b85d1 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:13:10.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.461+0000 7fcb86ffd640 1 --2- 172.21.15.46:0/224416416 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fcb7c0a2730 0x7fcb7c0a4bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:13:10.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.461+0000 7fcb7bfff640 1 -- 172.21.15.46:0/720762482 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb8c01e070 con 0x557bbb1ea970 2024-03-31T23:13:10.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.461+0000 7fcb7bfff640 1 -- 172.21.15.46:0/720762482 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fcb8c0035f0 con 0x557bbb1ea970 2024-03-31T23:13:10.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.461+0000 7fcb7bfff640 1 -- 172.21.15.46:0/720762482 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb8c00f820 con 0x557bbb1ea970 2024-03-31T23:13:10.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.461+0000 7fcb9622c640 1 --2- 172.21.15.46:0/224416416 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fcb7c0a2730 0x7fcb7c0a4bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:13:10.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.461+0000 7fcb86ffd640 1 -- 172.21.15.46:0/224416416 <== mon.1 v2:172.21.15.162:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7fcb800cf480 con 0x7fcb901005c0 2024-03-31T23:13:10.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.462+0000 7fcb9a875740 1 -- 172.21.15.46:0/720762482 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557bbb1ea970 msgr2=0x557bbb1ecd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:13:10.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.462+0000 7fcb9622c640 1 --2- 172.21.15.46:0/224416416 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fcb7c0a2730 0x7fcb7c0a4bf0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fcb880099a0 tx=0x7fcb88005fd0 comp rx=0 tx=0).ready entity=mgr.15654 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:13:10.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.462+0000 7fcb9a875740 1 --2- 172.21.15.46:0/720762482 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557bbb1ea970 0x557bbb1ecd60 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fcb8c0058c0 tx=0x7fcb8c0119a0 comp rx=0 tx=0).stop 2024-03-31T23:13:10.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.462+0000 7fcb9a875740 1 -- 172.21.15.46:0/720762482 shutdown_connections 2024-03-31T23:13:10.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.462+0000 7fcb9a875740 1 --2- 172.21.15.46:0/720762482 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557bbb1ea970 0x557bbb1ecd60 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:10.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.462+0000 7fcb9a875740 1 --2- 172.21.15.46:0/720762482 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557bbb1e1d90 0x557bbb1ea430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:10.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.462+0000 7fcb9a875740 1 --2- 172.21.15.46:0/720762482 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557bbb1e1470 0x557bbb1e1850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:10.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.462+0000 7fcb9a875740 1 -- 172.21.15.46:0/720762482 >> 172.21.15.46:0/720762482 conn(0x557bbb1dbba0 msgr2=0x557bbb1dbf80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:13:10.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.462+0000 7fcb9a875740 1 -- 172.21.15.46:0/720762482 shutdown_connections 2024-03-31T23:13:10.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.462+0000 7fcb9a875740 1 -- 172.21.15.46:0/720762482 wait complete. 2024-03-31T23:13:10.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.462+0000 7fcb9a875740 1 -- start start 2024-03-31T23:13:10.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.462+0000 7fcb9a875740 1 Processor -- start 2024-03-31T23:13:10.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.463+0000 7fcb9a875740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557bbb1e1470 0x557bbb1e7970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:13:10.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.463+0000 7fcb9a875740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557bbb1e1d90 0x557bbb1e7eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:13:10.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.463+0000 7fcb9a875740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557bbb1ea970 0x557bbb1e83f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:13:10.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.463+0000 7fcb9a875740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x557bbb1ed760 con 0x557bbb1e1470 2024-03-31T23:13:10.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.463+0000 7fcb9a875740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x557bbb0e3470 con 0x557bbb1e1d90 2024-03-31T23:13:10.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.463+0000 7fcb9a875740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x557bbb1ed8e0 con 0x557bbb1ea970 2024-03-31T23:13:10.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.463+0000 7fcb9a875740 10 client.0 _ll_register_callbacks cb 0x557bbb100cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T23:13:10.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.463+0000 7fcb95a2b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557bbb1e1470 0x557bbb1e7970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:13:10.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.463+0000 7fcb95a2b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557bbb1e1470 0x557bbb1e7970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:49030/0 (socket says 172.21.15.46:49030) 2024-03-31T23:13:10.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.463+0000 7fcb95a2b640 1 -- 172.21.15.46:0/1416641774 learned_addr learned my addr 172.21.15.46:0/1416641774 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:13:10.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.463+0000 7fcb9522a640 1 --2- 172.21.15.46:0/1416641774 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557bbb1e1d90 0x557bbb1e7eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:13:10.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.463+0000 7fcb9622c640 1 --2- 172.21.15.46:0/1416641774 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557bbb1ea970 0x557bbb1e83f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:13:10.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.463+0000 7fcb95a2b640 1 -- 172.21.15.46:0/1416641774 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557bbb1ea970 msgr2=0x557bbb1e83f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:13:10.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.463+0000 7fcb95a2b640 1 --2- 172.21.15.46:0/1416641774 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557bbb1ea970 0x557bbb1e83f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:10.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.463+0000 7fcb95a2b640 1 -- 172.21.15.46:0/1416641774 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557bbb1e1d90 msgr2=0x557bbb1e7eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:13:10.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.463+0000 7fcb95a2b640 1 --2- 172.21.15.46:0/1416641774 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557bbb1e1d90 0x557bbb1e7eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:10.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.463+0000 7fcb95a2b640 1 -- 172.21.15.46:0/1416641774 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557bbae593f0 con 0x557bbb1e1470 2024-03-31T23:13:10.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.463+0000 7fcb9522a640 1 --2- 172.21.15.46:0/1416641774 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557bbb1e1d90 0x557bbb1e7eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:13:10.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.463+0000 7fcb95a2b640 1 --2- 172.21.15.46:0/1416641774 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557bbb1e1470 0x557bbb1e7970 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7fcb80097a60 tx=0x7fcb80004540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=33a32140672c5669 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:13:10.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.463+0000 7fcb797fa640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-31T23:13:10.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.463+0000 7fcb797fa640 1 -- 172.21.15.46:0/1416641774 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb8009b3d0 con 0x557bbb1e1470 2024-03-31T23:13:10.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.463+0000 7fcb797fa640 1 -- 172.21.15.46:0/1416641774 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7fcb800cc980 con 0x557bbb1e1470 2024-03-31T23:13:10.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.463+0000 7fcb9a875740 1 -- 172.21.15.46:0/1416641774 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcb8c0035f0 con 0x557bbb1e1470 2024-03-31T23:13:10.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.464+0000 7fcb797fa640 1 -- 172.21.15.46:0/1416641774 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fcb8009af60 con 0x557bbb1e1470 2024-03-31T23:13:10.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.464+0000 7fcb9a875740 10 client.16143 fetch_fsmap learned FSMap version 27 2024-03-31T23:13:10.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.464+0000 7fcb9a875740 1 -- 172.21.15.46:0/1416641774 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x557bbaf237d0 con 0x557bbb1e1470 2024-03-31T23:13:10.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.464+0000 7fcb797fa640 1 -- 172.21.15.46:0/1416641774 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fcb8c0035f0 con 0x557bbb1e1470 2024-03-31T23:13:10.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.464+0000 7fcb797fa640 1 -- 172.21.15.46:0/1416641774 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fcb800cf110 con 0x557bbb1e1470 2024-03-31T23:13:10.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.464+0000 7fcb9a875740 10 client.16143 fetch_fsmap finished waiting for FSMap version 27 2024-03-31T23:13:10.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.464+0000 7fcb9a875740 10 client.16143 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-31T23:13:10.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.464+0000 7fcb9a875740 20 client.16143 populate_metadata read hostname 'smithi046' 2024-03-31T23:13:10.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.464+0000 7fcb9a875740 1 --2- 172.21.15.46:0/1416641774 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x557bbb1fa890 0x557bbb1fcd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:13:10.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.464+0000 7fcb9a875740 4 client.16143 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-31T23:13:10.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.464+0000 7fcb9a875740 1 -- 172.21.15.46:0/1416641774 --> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557bbb1526f0 con 0x557bbb1fa890 2024-03-31T23:13:10.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.464+0000 7fcb9522a640 1 --2- 172.21.15.46:0/1416641774 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x557bbb1fa890 0x557bbb1fcd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:13:10.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.465+0000 7fcb9522a640 1 --2- 172.21.15.46:0/1416641774 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x557bbb1fa890 0x557bbb1fcd20 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1caa2ee9df33acb2 server_cookie=49e0037cfc8b4b82 in_seq=0 out_seq=0 2024-03-31T23:13:10.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.465+0000 7fcb797fa640 10 client.16143 ms_handle_connect on v2:172.21.15.142:6838/2519143918 2024-03-31T23:13:10.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.465+0000 7fcb797fa640 1 -- 172.21.15.46:0/1416641774 <== mds.0 v2:172.21.15.142:6838/2519143918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x557bbb1526f0 con 0x557bbb1fa890 2024-03-31T23:13:10.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.465+0000 7fcb797fa640 10 client.16143 handle_command_reply: tid=0 2024-03-31T23:13:10.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.465+0000 7fcb9a875740 1 client.16143 shutdown 2024-03-31T23:13:10.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.465+0000 7fcb9a875740 1 -- 172.21.15.46:0/1416641774 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557bbb1e1470 msgr2=0x557bbb1e7970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:13:10.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.465+0000 7fcb9a875740 1 --2- 172.21.15.46:0/1416641774 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557bbb1e1470 0x557bbb1e7970 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7fcb80097a60 tx=0x7fcb80004540 comp rx=0 tx=0).stop 2024-03-31T23:13:10.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.465+0000 7fcb9a875740 1 -- 172.21.15.46:0/1416641774 shutdown_connections 2024-03-31T23:13:10.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.465+0000 7fcb9a875740 1 --2- 172.21.15.46:0/1416641774 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x557bbb1fa890 0x557bbb1fcd20 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:10.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.465+0000 7fcb9a875740 1 --2- 172.21.15.46:0/1416641774 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557bbb1ea970 0x557bbb1e83f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:10.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.465+0000 7fcb797fa640 0 client.16143 ms_handle_reset on v2:172.21.15.142:6838/2519143918 2024-03-31T23:13:10.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.465+0000 7fcb9a875740 1 --2- 172.21.15.46:0/1416641774 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557bbb1e1d90 0x557bbb1e7eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:10.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.465+0000 7fcb9a875740 1 --2- 172.21.15.46:0/1416641774 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557bbb1e1470 0x557bbb1e7970 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:10.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.465+0000 7fcb9a875740 1 -- 172.21.15.46:0/1416641774 >> 172.21.15.46:0/1416641774 conn(0x557bbb1dbba0 msgr2=0x557bbb1dd8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:13:10.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.465+0000 7fcb9a875740 1 -- 172.21.15.46:0/1416641774 shutdown_connections 2024-03-31T23:13:10.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.465+0000 7fcb9a875740 1 -- 172.21.15.46:0/1416641774 wait complete. 2024-03-31T23:13:10.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.466+0000 7fcb9a875740 20 client.16143 trim_cache size 0 max 16384 2024-03-31T23:13:10.485 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.479+0000 7fcb9a875740 1 Processor -- start 2024-03-31T23:13:10.485 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.479+0000 7fcb9a875740 1 -- start start 2024-03-31T23:13:10.485 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.479+0000 7fcb9a875740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557bbb1e1470 0x557bbb1e9fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:13:10.485 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.479+0000 7fcb9a875740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557bbb1e1d90 0x557bbb1f6500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:13:10.485 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.479+0000 7fcb9a875740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557bbb1ea970 0x557bbb2025f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:13:10.485 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.479+0000 7fcb9a875740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fcb901af220 con 0x557bbb1e1d90 2024-03-31T23:13:10.485 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.479+0000 7fcb9a875740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fcb901acaf0 con 0x557bbb1ea970 2024-03-31T23:13:10.485 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.479+0000 7fcb9a875740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fcb901aa940 con 0x557bbb1e1470 2024-03-31T23:13:10.485 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.479+0000 7fcb9522a640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557bbb1e1d90 0x557bbb1f6500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:13:10.485 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.479+0000 7fcb9522a640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557bbb1e1d90 0x557bbb1f6500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:49044/0 (socket says 172.21.15.46:49044) 2024-03-31T23:13:10.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.479+0000 7fcb9522a640 1 -- 172.21.15.46:0/1073181202 learned_addr learned my addr 172.21.15.46:0/1073181202 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:13:10.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.479+0000 7fcb95a2b640 1 --2- 172.21.15.46:0/1073181202 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557bbb1e1470 0x557bbb1e9fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:13:10.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.479+0000 7fcb9622c640 1 --2- 172.21.15.46:0/1073181202 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557bbb1ea970 0x557bbb2025f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:13:10.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.479+0000 7fcb9522a640 1 -- 172.21.15.46:0/1073181202 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557bbb1e1470 msgr2=0x557bbb1e9fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:13:10.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.479+0000 7fcb9522a640 1 --2- 172.21.15.46:0/1073181202 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557bbb1e1470 0x557bbb1e9fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:10.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.479+0000 7fcb9522a640 1 -- 172.21.15.46:0/1073181202 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557bbb1ea970 msgr2=0x557bbb2025f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:13:10.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.479+0000 7fcb9522a640 1 --2- 172.21.15.46:0/1073181202 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557bbb1ea970 0x557bbb2025f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:10.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.479+0000 7fcb9522a640 1 -- 172.21.15.46:0/1073181202 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557bbb1e1bc0 con 0x557bbb1e1d90 2024-03-31T23:13:10.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.479+0000 7fcb9622c640 1 --2- 172.21.15.46:0/1073181202 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557bbb1ea970 0x557bbb2025f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T23:13:10.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.479+0000 7fcb9522a640 1 --2- 172.21.15.46:0/1073181202 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557bbb1e1d90 0x557bbb1f6500 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x557bbb1dcd30 tx=0x7fcb8c0092d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=16ae1b6f265e55a6 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:13:10.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.480+0000 7fcb7b7fe640 1 -- 172.21.15.46:0/1073181202 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb8c01e040 con 0x557bbb1e1d90 2024-03-31T23:13:10.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.480+0000 7fcb7b7fe640 1 -- 172.21.15.46:0/1073181202 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fcb8c02f070 con 0x557bbb1e1d90 2024-03-31T23:13:10.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.480+0000 7fcb7b7fe640 1 -- 172.21.15.46:0/1073181202 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb8c005610 con 0x557bbb1e1d90 2024-03-31T23:13:10.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.480+0000 7fcb9a875740 1 -- 172.21.15.46:0/1073181202 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557bbb1e1d90 msgr2=0x557bbb1f6500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:13:10.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.480+0000 7fcb9a875740 1 --2- 172.21.15.46:0/1073181202 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557bbb1e1d90 0x557bbb1f6500 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x557bbb1dcd30 tx=0x7fcb8c0092d0 comp rx=0 tx=0).stop 2024-03-31T23:13:10.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.480+0000 7fcb9a875740 1 -- 172.21.15.46:0/1073181202 shutdown_connections 2024-03-31T23:13:10.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.480+0000 7fcb9a875740 1 --2- 172.21.15.46:0/1073181202 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557bbb1ea970 0x557bbb2025f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:10.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.480+0000 7fcb9a875740 1 --2- 172.21.15.46:0/1073181202 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557bbb1e1d90 0x557bbb1f6500 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:10.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.480+0000 7fcb9a875740 1 --2- 172.21.15.46:0/1073181202 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557bbb1e1470 0x557bbb1e9fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:10.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.480+0000 7fcb9a875740 1 -- 172.21.15.46:0/1073181202 >> 172.21.15.46:0/1073181202 conn(0x557bbb1dbba0 msgr2=0x557bbb1dd860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:13:10.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.480+0000 7fcb9a875740 1 -- 172.21.15.46:0/1073181202 shutdown_connections 2024-03-31T23:13:10.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.480+0000 7fcb9a875740 1 -- 172.21.15.46:0/1073181202 wait complete. 2024-03-31T23:13:10.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.480+0000 7fcb9a875740 1 -- start start 2024-03-31T23:13:10.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.481+0000 7fcb9a875740 1 Processor -- start 2024-03-31T23:13:10.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.481+0000 7fcb9a875740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557bbb1e1470 0x557bbb1f2b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:13:10.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.481+0000 7fcb9a875740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557bbb1e1d90 0x557bbb1edee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:13:10.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.481+0000 7fcb9a875740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557bbb1ea970 0x557bbb1ee420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:13:10.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.481+0000 7fcb9a875740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fcb901aafb0 con 0x557bbb1e1470 2024-03-31T23:13:10.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.481+0000 7fcb9a875740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fcb901ac420 con 0x557bbb1ea970 2024-03-31T23:13:10.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.481+0000 7fcb9a875740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fcb901ab680 con 0x557bbb1e1d90 2024-03-31T23:13:10.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.481+0000 7fcb9a875740 10 client.0 _ll_register_callbacks cb 0x7fcb80003d00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T23:13:10.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.481+0000 7fcb95a2b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557bbb1e1470 0x557bbb1f2b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:13:10.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.481+0000 7fcb95a2b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557bbb1e1470 0x557bbb1f2b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:49048/0 (socket says 172.21.15.46:49048) 2024-03-31T23:13:10.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.481+0000 7fcb95a2b640 1 -- 172.21.15.46:0/188965353 learned_addr learned my addr 172.21.15.46:0/188965353 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:13:10.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.481+0000 7fcb9522a640 1 --2- 172.21.15.46:0/188965353 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557bbb1e1d90 0x557bbb1edee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:13:10.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.481+0000 7fcb9622c640 1 --2- 172.21.15.46:0/188965353 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557bbb1ea970 0x557bbb1ee420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:13:10.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.481+0000 7fcb95a2b640 1 -- 172.21.15.46:0/188965353 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557bbb1e1d90 msgr2=0x557bbb1edee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:13:10.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.481+0000 7fcb95a2b640 1 --2- 172.21.15.46:0/188965353 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557bbb1e1d90 0x557bbb1edee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:10.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.481+0000 7fcb95a2b640 1 -- 172.21.15.46:0/188965353 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557bbb1ea970 msgr2=0x557bbb1ee420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:13:10.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.481+0000 7fcb95a2b640 1 --2- 172.21.15.46:0/188965353 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557bbb1ea970 0x557bbb1ee420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:10.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.481+0000 7fcb95a2b640 1 -- 172.21.15.46:0/188965353 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557bbb1ee9e0 con 0x557bbb1e1470 2024-03-31T23:13:10.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.482+0000 7fcb95a2b640 1 --2- 172.21.15.46:0/188965353 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557bbb1e1470 0x557bbb1f2b70 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7fcb8000c6a0 tx=0x7fcb800ccd60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec5fb624cde318a5 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:13:10.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.482+0000 7fcb797fa640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-31T23:13:10.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.482+0000 7fcb797fa640 1 -- 172.21.15.46:0/188965353 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb8009b140 con 0x557bbb1e1470 2024-03-31T23:13:10.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.482+0000 7fcb797fa640 1 -- 172.21.15.46:0/188965353 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7fcb800b4220 con 0x557bbb1e1470 2024-03-31T23:13:10.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.482+0000 7fcb9a875740 1 -- 172.21.15.46:0/188965353 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcb8c02f070 con 0x557bbb1e1470 2024-03-31T23:13:10.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.482+0000 7fcb797fa640 1 -- 172.21.15.46:0/188965353 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fcb8009b3d0 con 0x557bbb1e1470 2024-03-31T23:13:10.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.482+0000 7fcb9a875740 10 client.16149 fetch_fsmap learned FSMap version 27 2024-03-31T23:13:10.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.482+0000 7fcb9a875740 1 -- 172.21.15.46:0/188965353 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x557bbb2609b0 con 0x557bbb1e1470 2024-03-31T23:13:10.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.482+0000 7fcb797fa640 1 -- 172.21.15.46:0/188965353 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fcb8c02f070 con 0x557bbb1e1470 2024-03-31T23:13:10.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.482+0000 7fcb797fa640 1 -- 172.21.15.46:0/188965353 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fcb80004000 con 0x557bbb1e1470 2024-03-31T23:13:10.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.482+0000 7fcb9a875740 10 client.16149 fetch_fsmap finished waiting for FSMap version 27 2024-03-31T23:13:10.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.482+0000 7fcb9a875740 10 client.16149 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-31T23:13:10.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.482+0000 7fcb9a875740 20 client.16149 populate_metadata read hostname 'smithi046' 2024-03-31T23:13:10.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.482+0000 7fcb9a875740 1 --2- 172.21.15.46:0/188965353 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x557bbb261120 0x557bbb263540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:13:10.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.482+0000 7fcb9a875740 4 client.16149 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-31T23:13:10.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.482+0000 7fcb9a875740 1 -- 172.21.15.46:0/188965353 --> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] -- command(tid 0: {"prefix": "damage ls"}) -- 0x557bbb14ca40 con 0x557bbb261120 2024-03-31T23:13:10.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.483+0000 7fcb9522a640 1 --2- 172.21.15.46:0/188965353 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x557bbb261120 0x557bbb263540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:13:10.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.483+0000 7fcb797fa640 10 client.16149 ms_handle_connect on v2:172.21.15.142:6838/2519143918 2024-03-31T23:13:10.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.483+0000 7fcb9522a640 1 --2- 172.21.15.46:0/188965353 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x557bbb261120 0x557bbb263540 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9a4bb8ccd976bc12 server_cookie=f8d74f469bd1cf43 in_seq=0 out_seq=0 2024-03-31T23:13:10.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.483+0000 7fcb797fa640 1 -- 172.21.15.46:0/188965353 <== mds.0 v2:172.21.15.142:6838/2519143918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x557bbb14ca40 con 0x557bbb261120 2024-03-31T23:13:10.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.483+0000 7fcb797fa640 10 client.16149 handle_command_reply: tid=0 2024-03-31T23:13:10.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.483+0000 7fcb9a875740 1 client.16149 shutdown 2024-03-31T23:13:10.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.483+0000 7fcb9a875740 1 -- 172.21.15.46:0/188965353 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557bbb1e1470 msgr2=0x557bbb1f2b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:13:10.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.483+0000 7fcb9a875740 1 --2- 172.21.15.46:0/188965353 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557bbb1e1470 0x557bbb1f2b70 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7fcb8000c6a0 tx=0x7fcb800ccd60 comp rx=0 tx=0).stop 2024-03-31T23:13:10.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.483+0000 7fcb9a875740 1 -- 172.21.15.46:0/188965353 shutdown_connections 2024-03-31T23:13:10.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.483+0000 7fcb9a875740 1 --2- 172.21.15.46:0/188965353 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x557bbb261120 0x557bbb263540 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:10.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.483+0000 7fcb9a875740 1 --2- 172.21.15.46:0/188965353 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557bbb1ea970 0x557bbb1ee420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:10.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.483+0000 7fcb9a875740 1 --2- 172.21.15.46:0/188965353 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557bbb1e1d90 0x557bbb1edee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:10.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.483+0000 7fcb797fa640 0 client.16149 ms_handle_reset on v2:172.21.15.142:6838/2519143918 2024-03-31T23:13:10.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.483+0000 7fcb9a875740 1 --2- 172.21.15.46:0/188965353 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557bbb1e1470 0x557bbb1f2b70 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:10.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.483+0000 7fcb9a875740 1 -- 172.21.15.46:0/188965353 >> 172.21.15.46:0/188965353 conn(0x557bbb1dbba0 msgr2=0x557bbb1da4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:13:10.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.483+0000 7fcb9a875740 1 -- 172.21.15.46:0/188965353 shutdown_connections 2024-03-31T23:13:10.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.483+0000 7fcb9a875740 1 -- 172.21.15.46:0/188965353 wait complete. 2024-03-31T23:13:10.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.484+0000 7fcb9a875740 20 client.16149 trim_cache size 0 max 16384 2024-03-31T23:13:10.491 INFO:teuthology.orchestra.run.smithi046.stdout:[] 2024-03-31T23:13:10.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.484+0000 7fcb97c8f640 1 -- 172.21.15.46:0/224416416 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fcb7c0a2730 msgr2=0x7fcb7c0a4bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:13:10.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.484+0000 7fcb97c8f640 1 --2- 172.21.15.46:0/224416416 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fcb7c0a2730 0x7fcb7c0a4bf0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fcb880099a0 tx=0x7fcb88005fd0 comp rx=0 tx=0).stop 2024-03-31T23:13:10.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.484+0000 7fcb97c8f640 1 -- 172.21.15.46:0/224416416 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcb901005c0 msgr2=0x7fcb90198d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:13:10.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.484+0000 7fcb97c8f640 1 --2- 172.21.15.46:0/224416416 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcb901005c0 0x7fcb90198d50 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fcb8000e9c0 tx=0x7fcb8000ee80 comp rx=0 tx=0).stop 2024-03-31T23:13:10.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.484+0000 7fcb97c8f640 1 -- 172.21.15.46:0/224416416 shutdown_connections 2024-03-31T23:13:10.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.484+0000 7fcb97c8f640 1 --2- 172.21.15.46:0/224416416 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fcb7c0a2730 0x7fcb7c0a4bf0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:10.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.484+0000 7fcb97c8f640 1 --2- 172.21.15.46:0/224416416 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcb90106660 0x7fcb901a3dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:10.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.484+0000 7fcb97c8f640 1 --2- 172.21.15.46:0/224416416 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcb90100f80 0x7fcb90199290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:10.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.484+0000 7fcb97c8f640 1 --2- 172.21.15.46:0/224416416 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcb901005c0 0x7fcb90198d50 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:10.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.484+0000 7fcb97c8f640 1 -- 172.21.15.46:0/224416416 >> 172.21.15.46:0/224416416 conn(0x7fcb90077750 msgr2=0x7fcb900ff320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:13:10.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.484+0000 7fcb97c8f640 1 -- 172.21.15.46:0/224416416 shutdown_connections 2024-03-31T23:13:10.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:10.484+0000 7fcb97c8f640 1 -- 172.21.15.46:0/224416416 wait complete. 2024-03-31T23:13:10.613 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:13:10 smithi046 ceph-mon[32498]: pgmap v46: 129 pgs: 129 active+clean; 30 MiB data, 457 MiB used, 1.0 TiB / 1.0 TiB avail; 389 KiB/s rd, 3.4 MiB/s wr, 155 op/s 2024-03-31T23:13:10.613 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:13:10 smithi046 ceph-mon[32498]: from='client.? 172.21.15.46:0/2608545930' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-31T23:13:10.693 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:13:10 smithi142 ceph-mon[35271]: pgmap v46: 129 pgs: 129 active+clean; 30 MiB data, 457 MiB used, 1.0 TiB / 1.0 TiB avail; 389 KiB/s rd, 3.4 MiB/s wr, 155 op/s 2024-03-31T23:13:10.693 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:13:10 smithi142 ceph-mon[35271]: from='client.? 172.21.15.46:0/2608545930' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-31T23:13:10.795 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:13:10 smithi162 ceph-mon[36237]: pgmap v46: 129 pgs: 129 active+clean; 30 MiB data, 457 MiB used, 1.0 TiB / 1.0 TiB avail; 389 KiB/s rd, 3.4 MiB/s wr, 155 op/s 2024-03-31T23:13:10.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:13:10 smithi162 ceph-mon[36237]: from='client.? 172.21.15.46:0/2608545930' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-31T23:13:11.190 INFO:tasks.workunit.client.0.smithi046.stdout: 1 8382 3.61 MB/sec warmup 17 sec latency 2885.496 ms 2024-03-31T23:13:11.496 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-31T23:13:11.496 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T23:13:11.496 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-31T23:13:11.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.609+0000 7fd5d9af4640 1 -- 172.21.15.46:0/869533954 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd5d41076e0 msgr2=0x7fd5d4107ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:13:11.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.609+0000 7fd5d9af4640 1 --2- 172.21.15.46:0/869533954 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd5d41076e0 0x7fd5d4107ac0 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7fd5c4009930 tx=0x7fd5c402bc30 comp rx=0 tx=0).stop 2024-03-31T23:13:11.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.609+0000 7fd5d9af4640 1 -- 172.21.15.46:0/869533954 shutdown_connections 2024-03-31T23:13:11.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.609+0000 7fd5d9af4640 1 --2- 172.21.15.46:0/869533954 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd5d4102090 0x7fd5d410e5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:11.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.609+0000 7fd5d9af4640 1 --2- 172.21.15.46:0/869533954 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd5d41016d0 0x7fd5d4101b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:11.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.609+0000 7fd5d9af4640 1 --2- 172.21.15.46:0/869533954 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd5d41076e0 0x7fd5d4107ac0 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:11.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.609+0000 7fd5d9af4640 1 -- 172.21.15.46:0/869533954 >> 172.21.15.46:0/869533954 conn(0x7fd5d40fd1e0 msgr2=0x7fd5d40ff600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:13:11.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.609+0000 7fd5d9af4640 1 -- 172.21.15.46:0/869533954 shutdown_connections 2024-03-31T23:13:11.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.609+0000 7fd5d9af4640 1 -- 172.21.15.46:0/869533954 wait complete. 2024-03-31T23:13:11.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.610+0000 7fd5d9af4640 1 Processor -- start 2024-03-31T23:13:11.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.610+0000 7fd5d9af4640 1 -- start start 2024-03-31T23:13:11.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.610+0000 7fd5d9af4640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd5d41016d0 0x7fd5d4073a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:13:11.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.610+0000 7fd5d9af4640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd5d4102090 0x7fd5d4073f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:13:11.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.610+0000 7fd5d9af4640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd5d41076e0 0x7fd5d40744d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:13:11.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.610+0000 7fd5d9af4640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fd5d41119a0 con 0x7fd5d4102090 2024-03-31T23:13:11.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.610+0000 7fd5d9af4640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fd5d4111820 con 0x7fd5d41076e0 2024-03-31T23:13:11.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.610+0000 7fd5d9af4640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fd5d4111b20 con 0x7fd5d41016d0 2024-03-31T23:13:11.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.610+0000 7fd5d2ffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd5d4102090 0x7fd5d4073f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:13:11.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.610+0000 7fd5d2ffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd5d4102090 0x7fd5d4073f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:33672/0 (socket says 172.21.15.46:33672) 2024-03-31T23:13:11.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.610+0000 7fd5d2ffd640 1 -- 172.21.15.46:0/1966317597 learned_addr learned my addr 172.21.15.46:0/1966317597 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:13:11.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.610+0000 7fd5d37fe640 1 --2- 172.21.15.46:0/1966317597 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd5d41016d0 0x7fd5d4073a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:13:11.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.610+0000 7fd5d3fff640 1 --2- 172.21.15.46:0/1966317597 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd5d41076e0 0x7fd5d40744d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:13:11.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.611+0000 7fd5d2ffd640 1 -- 172.21.15.46:0/1966317597 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd5d41016d0 msgr2=0x7fd5d4073a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:13:11.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.611+0000 7fd5d2ffd640 1 --2- 172.21.15.46:0/1966317597 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd5d41016d0 0x7fd5d4073a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:11.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.611+0000 7fd5d2ffd640 1 -- 172.21.15.46:0/1966317597 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd5d41076e0 msgr2=0x7fd5d40744d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:13:11.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.611+0000 7fd5d2ffd640 1 --2- 172.21.15.46:0/1966317597 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd5d41076e0 0x7fd5d40744d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:11.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.611+0000 7fd5d2ffd640 1 -- 172.21.15.46:0/1966317597 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5d41a6450 con 0x7fd5d4102090 2024-03-31T23:13:11.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.611+0000 7fd5d2ffd640 1 --2- 172.21.15.46:0/1966317597 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd5d4102090 0x7fd5d4073f90 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7fd5c800e9f0 tx=0x7fd5c800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:13:11.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.611+0000 7fd5d0ff9640 1 -- 172.21.15.46:0/1966317597 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5c800cd80 con 0x7fd5d4102090 2024-03-31T23:13:11.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.611+0000 7fd5d0ff9640 1 -- 172.21.15.46:0/1966317597 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd5c80108e0 con 0x7fd5d4102090 2024-03-31T23:13:11.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.611+0000 7fd5d9af4640 1 -- 172.21.15.46:0/1966317597 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd5d41a6740 con 0x7fd5d4102090 2024-03-31T23:13:11.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.611+0000 7fd5d0ff9640 1 -- 172.21.15.46:0/1966317597 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5c8018920 con 0x7fd5d4102090 2024-03-31T23:13:11.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.611+0000 7fd5d9af4640 1 -- 172.21.15.46:0/1966317597 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd5d41a6ce0 con 0x7fd5d4102090 2024-03-31T23:13:11.623 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.613+0000 7fd5d0ff9640 1 -- 172.21.15.46:0/1966317597 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 24) ==== 137553+0+0 (secure 0 0 0) 0x7fd5c8010a90 con 0x7fd5d4102090 2024-03-31T23:13:11.623 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.614+0000 7fd5dc7b2740 1 Processor -- start 2024-03-31T23:13:11.623 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.614+0000 7fd5dc7b2740 1 -- start start 2024-03-31T23:13:11.623 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.614+0000 7fd5dc7b2740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5569f9b169f0 0x5569f9b16dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:13:11.623 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.614+0000 7fd5dc7b2740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5569f9b17310 0x5569f9b1f9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:13:11.623 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.614+0000 7fd5dc7b2740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5569f9b1fef0 0x5569f9b222e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:13:11.623 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.614+0000 7fd5dc7b2740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x5569f9a170b0 con 0x5569f9b1fef0 2024-03-31T23:13:11.623 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.614+0000 7fd5dc7b2740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x5569f9a79ea0 con 0x5569f9b17310 2024-03-31T23:13:11.623 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.614+0000 7fd5dc7b2740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5569f9a22c00 con 0x5569f9b169f0 2024-03-31T23:13:11.623 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.614+0000 7fd5d3fff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5569f9b169f0 0x5569f9b16dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:13:11.624 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.614+0000 7fd5d37fe640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5569f9b17310 0x5569f9b1f9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:13:11.624 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.614+0000 7fd5d3fff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5569f9b169f0 0x5569f9b16dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.46:55048/0 (socket says 172.21.15.46:55048) 2024-03-31T23:13:11.624 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.614+0000 7fd5d2ffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5569f9b1fef0 0x5569f9b222e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:13:11.624 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.614+0000 7fd5d3fff640 1 -- 172.21.15.46:0/4119921069 learned_addr learned my addr 172.21.15.46:0/4119921069 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:13:11.624 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.614+0000 7fd5d0ff9640 1 --2- 172.21.15.46:0/1966317597 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fd5bc0a27b0 0x7fd5bc0a4c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:13:11.624 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.614+0000 7fd5d3fff640 1 --2- 172.21.15.46:0/1966317597 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fd5bc0a27b0 0x7fd5bc0a4c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:13:11.624 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.614+0000 7fd5d37fe640 1 -- 172.21.15.46:0/4119921069 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5569f9b169f0 msgr2=0x5569f9b16dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:13:11.624 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.614+0000 7fd5d37fe640 1 --2- 172.21.15.46:0/4119921069 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5569f9b169f0 0x5569f9b16dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:11.624 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.614+0000 7fd5d37fe640 1 -- 172.21.15.46:0/4119921069 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5569f9b1fef0 msgr2=0x5569f9b222e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:13:11.624 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.614+0000 7fd5d37fe640 1 --2- 172.21.15.46:0/4119921069 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5569f9b1fef0 0x5569f9b222e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:11.624 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.615+0000 7fd5d37fe640 1 -- 172.21.15.46:0/4119921069 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5569f98567e0 con 0x5569f9b17310 2024-03-31T23:13:11.624 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.615+0000 7fd5d0ff9640 1 -- 172.21.15.46:0/1966317597 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7fd5c80cf400 con 0x7fd5d4102090 2024-03-31T23:13:11.625 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.615+0000 7fd5d37fe640 1 --2- 172.21.15.46:0/4119921069 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5569f9b17310 0x5569f9b1f9b0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fd5c4009230 tx=0x7fd5c4009560 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4b2e808f073baa0d server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:13:11.625 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.615+0000 7fd5d3fff640 1 --2- 172.21.15.46:0/1966317597 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fd5bc0a27b0 0x7fd5bc0a4c70 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fd5cc0049d0 tx=0x7fd5cc004ab0 comp rx=0 tx=0).ready entity=mgr.15654 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:13:11.625 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.615+0000 7fd5b9ffb640 1 -- 172.21.15.46:0/4119921069 <== mon.2 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5c4045020 con 0x5569f9b17310 2024-03-31T23:13:11.625 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.615+0000 7fd5b9ffb640 1 -- 172.21.15.46:0/4119921069 <== mon.2 v2:172.21.15.142:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd5c402c7e0 con 0x5569f9b17310 2024-03-31T23:13:11.625 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.615+0000 7fd5b9ffb640 1 -- 172.21.15.46:0/4119921069 <== mon.2 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5c4005400 con 0x5569f9b17310 2024-03-31T23:13:11.625 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.616+0000 7fd5dc7b2740 1 -- 172.21.15.46:0/4119921069 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5569f9b17310 msgr2=0x5569f9b1f9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:13:11.625 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.616+0000 7fd5dc7b2740 1 --2- 172.21.15.46:0/4119921069 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5569f9b17310 0x5569f9b1f9b0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fd5c4009230 tx=0x7fd5c4009560 comp rx=0 tx=0).stop 2024-03-31T23:13:11.625 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.616+0000 7fd5dc7b2740 1 -- 172.21.15.46:0/4119921069 shutdown_connections 2024-03-31T23:13:11.625 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.616+0000 7fd5dc7b2740 1 --2- 172.21.15.46:0/4119921069 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5569f9b1fef0 0x5569f9b222e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:11.625 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.616+0000 7fd5dc7b2740 1 --2- 172.21.15.46:0/4119921069 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5569f9b17310 0x5569f9b1f9b0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:11.625 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.616+0000 7fd5dc7b2740 1 --2- 172.21.15.46:0/4119921069 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5569f9b169f0 0x5569f9b16dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:11.625 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.616+0000 7fd5dc7b2740 1 -- 172.21.15.46:0/4119921069 >> 172.21.15.46:0/4119921069 conn(0x5569f9b11120 msgr2=0x5569f9b11500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:13:11.625 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.616+0000 7fd5dc7b2740 1 -- 172.21.15.46:0/4119921069 shutdown_connections 2024-03-31T23:13:11.626 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.616+0000 7fd5dc7b2740 1 -- 172.21.15.46:0/4119921069 wait complete. 2024-03-31T23:13:11.626 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.616+0000 7fd5dc7b2740 1 -- start start 2024-03-31T23:13:11.626 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.616+0000 7fd5dc7b2740 1 Processor -- start 2024-03-31T23:13:11.626 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.616+0000 7fd5dc7b2740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5569f9b169f0 0x5569f9b1cf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:13:11.626 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.616+0000 7fd5dc7b2740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5569f9b17310 0x5569f9b1d480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:13:11.626 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.617+0000 7fd5dc7b2740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5569f9b1fef0 0x5569f9b1d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:13:11.626 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.617+0000 7fd5dc7b2740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x5569f9a183d0 con 0x5569f9b1fef0 2024-03-31T23:13:11.626 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.617+0000 7fd5dc7b2740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x5569f9a75f40 con 0x5569f9b169f0 2024-03-31T23:13:11.626 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.617+0000 7fd5dc7b2740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5569f9b22c80 con 0x5569f9b17310 2024-03-31T23:13:11.626 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.617+0000 7fd5dc7b2740 10 client.0 _ll_register_callbacks cb 0x5569f9a35b70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T23:13:11.627 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.617+0000 7fd5d37fe640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5569f9b169f0 0x5569f9b1cf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:13:11.627 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.617+0000 7fd5d37fe640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5569f9b169f0 0x5569f9b1cf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.46:37114/0 (socket says 172.21.15.46:37114) 2024-03-31T23:13:11.627 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.617+0000 7fd5d37fe640 1 -- 172.21.15.46:0/914336031 learned_addr learned my addr 172.21.15.46:0/914336031 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:13:11.627 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.617+0000 7fd5d2ffd640 1 --2- 172.21.15.46:0/914336031 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5569f9b17310 0x5569f9b1d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:13:11.627 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.617+0000 7fd5d3fff640 1 --2- 172.21.15.46:0/914336031 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5569f9b1fef0 0x5569f9b1d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:13:11.627 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.617+0000 7fd5d37fe640 1 -- 172.21.15.46:0/914336031 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5569f9b17310 msgr2=0x5569f9b1d480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:13:11.627 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.617+0000 7fd5d37fe640 1 --2- 172.21.15.46:0/914336031 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5569f9b17310 0x5569f9b1d480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:11.627 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.617+0000 7fd5d37fe640 1 -- 172.21.15.46:0/914336031 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5569f9b1fef0 msgr2=0x5569f9b1d9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:13:11.627 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.617+0000 7fd5d37fe640 1 --2- 172.21.15.46:0/914336031 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5569f9b1fef0 0x5569f9b1d9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:11.627 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.617+0000 7fd5d37fe640 1 -- 172.21.15.46:0/914336031 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5569f98b6f70 con 0x5569f9b169f0 2024-03-31T23:13:11.627 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.617+0000 7fd5d37fe640 1 --2- 172.21.15.46:0/914336031 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5569f9b169f0 0x5569f9b1cf40 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fd5c4036040 tx=0x7fd5c40042f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e5676e493d58f98c server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:13:11.627 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.617+0000 7fd5b2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.142:3300/0 2024-03-31T23:13:11.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.617+0000 7fd5b2ffd640 1 -- 172.21.15.46:0/914336031 <== mon.2 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5c4050070 con 0x5569f9b169f0 2024-03-31T23:13:11.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.617+0000 7fd5b2ffd640 1 -- 172.21.15.46:0/914336031 <== mon.2 v2:172.21.15.142:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7fd5c4005550 con 0x5569f9b169f0 2024-03-31T23:13:11.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.617+0000 7fd5dc7b2740 1 -- 172.21.15.46:0/914336031 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd5c402c7e0 con 0x5569f9b169f0 2024-03-31T23:13:11.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.618+0000 7fd5b2ffd640 1 -- 172.21.15.46:0/914336031 <== mon.2 v2:172.21.15.142:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fd5c4059050 con 0x5569f9b169f0 2024-03-31T23:13:11.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.618+0000 7fd5dc7b2740 10 client.26000 fetch_fsmap learned FSMap version 27 2024-03-31T23:13:11.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.618+0000 7fd5dc7b2740 1 -- 172.21.15.46:0/914336031 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5569f974e170 con 0x5569f9b169f0 2024-03-31T23:13:11.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.618+0000 7fd5b2ffd640 1 -- 172.21.15.46:0/914336031 <== mon.2 v2:172.21.15.142:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd5c402c7e0 con 0x5569f9b169f0 2024-03-31T23:13:11.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.618+0000 7fd5b2ffd640 1 -- 172.21.15.46:0/914336031 <== mon.2 v2:172.21.15.142:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fd5c4053020 con 0x5569f9b169f0 2024-03-31T23:13:11.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.618+0000 7fd5dc7b2740 10 client.26000 fetch_fsmap finished waiting for FSMap version 27 2024-03-31T23:13:11.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.618+0000 7fd5dc7b2740 10 client.26000 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-31T23:13:11.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.618+0000 7fd5dc7b2740 20 client.26000 populate_metadata read hostname 'smithi046' 2024-03-31T23:13:11.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.618+0000 7fd5dc7b2740 1 --2- 172.21.15.46:0/914336031 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x5569f9b2fbb0 0x5569f9b31fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:13:11.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.618+0000 7fd5dc7b2740 4 client.26000 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-31T23:13:11.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.618+0000 7fd5dc7b2740 1 -- 172.21.15.46:0/914336031 --> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5569f9a873e0 con 0x5569f9b2fbb0 2024-03-31T23:13:11.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.618+0000 7fd5d2ffd640 1 --2- 172.21.15.46:0/914336031 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x5569f9b2fbb0 0x5569f9b31fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:13:11.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.619+0000 7fd5d2ffd640 1 --2- 172.21.15.46:0/914336031 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x5569f9b2fbb0 0x5569f9b31fa0 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dee2e6c989b38982 server_cookie=a924a1e1236fa8ba in_seq=0 out_seq=0 2024-03-31T23:13:11.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.619+0000 7fd5b2ffd640 10 client.26000 ms_handle_connect on v2:172.21.15.142:6838/2519143918 2024-03-31T23:13:11.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.620+0000 7fd5b2ffd640 1 -- 172.21.15.46:0/914336031 <== mds.0 v2:172.21.15.142:6838/2519143918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5569f9a873e0 con 0x5569f9b2fbb0 2024-03-31T23:13:11.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.620+0000 7fd5b2ffd640 10 client.26000 handle_command_reply: tid=0 2024-03-31T23:13:11.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.620+0000 7fd5dc7b2740 1 client.26000 shutdown 2024-03-31T23:13:11.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.620+0000 7fd5dc7b2740 1 -- 172.21.15.46:0/914336031 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5569f9b169f0 msgr2=0x5569f9b1cf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:13:11.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.620+0000 7fd5dc7b2740 1 --2- 172.21.15.46:0/914336031 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5569f9b169f0 0x5569f9b1cf40 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fd5c4036040 tx=0x7fd5c40042f0 comp rx=0 tx=0).stop 2024-03-31T23:13:11.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.620+0000 7fd5dc7b2740 1 -- 172.21.15.46:0/914336031 shutdown_connections 2024-03-31T23:13:11.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.620+0000 7fd5dc7b2740 1 --2- 172.21.15.46:0/914336031 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x5569f9b2fbb0 0x5569f9b31fa0 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:11.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.620+0000 7fd5dc7b2740 1 --2- 172.21.15.46:0/914336031 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5569f9b1fef0 0x5569f9b1d9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:11.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.620+0000 7fd5dc7b2740 1 --2- 172.21.15.46:0/914336031 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5569f9b17310 0x5569f9b1d480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:11.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.620+0000 7fd5dc7b2740 1 --2- 172.21.15.46:0/914336031 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5569f9b169f0 0x5569f9b1cf40 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:11.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.620+0000 7fd5b2ffd640 0 client.26000 ms_handle_reset on v2:172.21.15.142:6838/2519143918 2024-03-31T23:13:11.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.620+0000 7fd5dc7b2740 1 -- 172.21.15.46:0/914336031 >> 172.21.15.46:0/914336031 conn(0x5569f9b11120 msgr2=0x5569f9b12e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:13:11.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.620+0000 7fd5dc7b2740 1 -- 172.21.15.46:0/914336031 shutdown_connections 2024-03-31T23:13:11.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.620+0000 7fd5dc7b2740 1 -- 172.21.15.46:0/914336031 wait complete. 2024-03-31T23:13:11.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.620+0000 7fd5dc7b2740 20 client.26000 trim_cache size 0 max 16384 2024-03-31T23:13:11.645 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.642+0000 7fd5dc7b2740 1 Processor -- start 2024-03-31T23:13:11.645 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.642+0000 7fd5dc7b2740 1 -- start start 2024-03-31T23:13:11.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.642+0000 7fd5dc7b2740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5569f9b169f0 0x5569f9b1c820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:13:11.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.642+0000 7fd5dc7b2740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5569f9b17310 0x5569f9b2b8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:13:11.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.642+0000 7fd5dc7b2740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5569f9b1fef0 0x5569f9b37690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:13:11.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.642+0000 7fd5dc7b2740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fd5d41b35d0 con 0x5569f9b17310 2024-03-31T23:13:11.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.642+0000 7fd5dc7b2740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fd5d41b0ea0 con 0x5569f9b169f0 2024-03-31T23:13:11.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.642+0000 7fd5dc7b2740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fd5d41aecf0 con 0x5569f9b1fef0 2024-03-31T23:13:11.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.643+0000 7fd5d2ffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5569f9b17310 0x5569f9b2b8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:13:11.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.643+0000 7fd5d2ffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5569f9b17310 0x5569f9b2b8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:33710/0 (socket says 172.21.15.46:33710) 2024-03-31T23:13:11.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.643+0000 7fd5d2ffd640 1 -- 172.21.15.46:0/3693260372 learned_addr learned my addr 172.21.15.46:0/3693260372 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:13:11.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.643+0000 7fd5d37fe640 1 --2- 172.21.15.46:0/3693260372 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5569f9b169f0 0x5569f9b1c820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:13:11.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.643+0000 7fd5d3fff640 1 --2- 172.21.15.46:0/3693260372 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5569f9b1fef0 0x5569f9b37690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:13:11.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.643+0000 7fd5d2ffd640 1 -- 172.21.15.46:0/3693260372 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5569f9b1fef0 msgr2=0x5569f9b37690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:13:11.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.643+0000 7fd5d2ffd640 1 --2- 172.21.15.46:0/3693260372 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5569f9b1fef0 0x5569f9b37690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:11.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.643+0000 7fd5d2ffd640 1 -- 172.21.15.46:0/3693260372 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5569f9b169f0 msgr2=0x5569f9b1c820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:13:11.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.643+0000 7fd5d2ffd640 1 --2- 172.21.15.46:0/3693260372 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5569f9b169f0 0x5569f9b1c820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:11.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.643+0000 7fd5d2ffd640 1 -- 172.21.15.46:0/3693260372 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5569f9b1b490 con 0x5569f9b17310 2024-03-31T23:13:11.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.643+0000 7fd5d2ffd640 1 --2- 172.21.15.46:0/3693260372 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5569f9b17310 0x5569f9b2b8a0 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7fd5c800e9c0 tx=0x7fd5c80bb040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=77b88eca0c6c6770 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:13:11.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.643+0000 7fd5b97fa640 1 -- 172.21.15.46:0/3693260372 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5c809b630 con 0x5569f9b17310 2024-03-31T23:13:11.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.643+0000 7fd5b97fa640 1 -- 172.21.15.46:0/3693260372 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd5c8084070 con 0x5569f9b17310 2024-03-31T23:13:11.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.643+0000 7fd5b97fa640 1 -- 172.21.15.46:0/3693260372 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5c80cc900 con 0x5569f9b17310 2024-03-31T23:13:11.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.644+0000 7fd5dc7b2740 1 -- 172.21.15.46:0/3693260372 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5569f9b17310 msgr2=0x5569f9b2b8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:13:11.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.644+0000 7fd5dc7b2740 1 --2- 172.21.15.46:0/3693260372 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5569f9b17310 0x5569f9b2b8a0 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7fd5c800e9c0 tx=0x7fd5c80bb040 comp rx=0 tx=0).stop 2024-03-31T23:13:11.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.644+0000 7fd5dc7b2740 1 -- 172.21.15.46:0/3693260372 shutdown_connections 2024-03-31T23:13:11.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.644+0000 7fd5dc7b2740 1 --2- 172.21.15.46:0/3693260372 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5569f9b1fef0 0x5569f9b37690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:11.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.644+0000 7fd5dc7b2740 1 --2- 172.21.15.46:0/3693260372 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5569f9b17310 0x5569f9b2b8a0 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:11.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.644+0000 7fd5dc7b2740 1 --2- 172.21.15.46:0/3693260372 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5569f9b169f0 0x5569f9b1c820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:11.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.644+0000 7fd5dc7b2740 1 -- 172.21.15.46:0/3693260372 >> 172.21.15.46:0/3693260372 conn(0x5569f9b11120 msgr2=0x5569f9b12420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:13:11.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.644+0000 7fd5dc7b2740 1 -- 172.21.15.46:0/3693260372 shutdown_connections 2024-03-31T23:13:11.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.644+0000 7fd5dc7b2740 1 -- 172.21.15.46:0/3693260372 wait complete. 2024-03-31T23:13:11.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.644+0000 7fd5dc7b2740 1 -- start start 2024-03-31T23:13:11.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.645+0000 7fd5dc7b2740 1 Processor -- start 2024-03-31T23:13:11.652 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.645+0000 7fd5dc7b2740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5569f9b169f0 0x5569f9b19d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:13:11.652 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.645+0000 7fd5dc7b2740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5569f9b17310 0x5569f9b1a2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:13:11.652 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.645+0000 7fd5dc7b2740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5569f9b1fef0 0x5569f9b1a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:13:11.652 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.645+0000 7fd5dc7b2740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fd5d41af360 con 0x5569f9b169f0 2024-03-31T23:13:11.652 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.645+0000 7fd5dc7b2740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fd5d41b07d0 con 0x5569f9b17310 2024-03-31T23:13:11.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.645+0000 7fd5dc7b2740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fd5d41afa30 con 0x5569f9b1fef0 2024-03-31T23:13:11.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.645+0000 7fd5dc7b2740 10 client.0 _ll_register_callbacks cb 0x7fd5c4009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T23:13:11.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.645+0000 7fd5d37fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5569f9b169f0 0x5569f9b19d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:13:11.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.645+0000 7fd5d37fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5569f9b169f0 0x5569f9b19d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:33714/0 (socket says 172.21.15.46:33714) 2024-03-31T23:13:11.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.645+0000 7fd5d37fe640 1 -- 172.21.15.46:0/3868055351 learned_addr learned my addr 172.21.15.46:0/3868055351 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:13:11.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.645+0000 7fd5d3fff640 1 --2- 172.21.15.46:0/3868055351 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5569f9b1fef0 0x5569f9b1a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:13:11.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.645+0000 7fd5d2ffd640 1 --2- 172.21.15.46:0/3868055351 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5569f9b17310 0x5569f9b1a2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:13:11.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.645+0000 7fd5d37fe640 1 -- 172.21.15.46:0/3868055351 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5569f9b1fef0 msgr2=0x5569f9b1a7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:13:11.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.645+0000 7fd5d37fe640 1 --2- 172.21.15.46:0/3868055351 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5569f9b1fef0 0x5569f9b1a7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:11.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.645+0000 7fd5d37fe640 1 -- 172.21.15.46:0/3868055351 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5569f9b17310 msgr2=0x5569f9b1a2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:13:11.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.645+0000 7fd5d37fe640 1 --2- 172.21.15.46:0/3868055351 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5569f9b17310 0x5569f9b1a2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:11.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.645+0000 7fd5d37fe640 1 -- 172.21.15.46:0/3868055351 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5569f9b94f60 con 0x5569f9b169f0 2024-03-31T23:13:11.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.646+0000 7fd5d37fe640 1 --2- 172.21.15.46:0/3868055351 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5569f9b169f0 0x5569f9b19d60 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7fd5c402bc70 tx=0x7fd5c4004590 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f6ab1d315d6a93f7 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:13:11.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.646+0000 7fd5b2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-31T23:13:11.654 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.646+0000 7fd5b2ffd640 1 -- 172.21.15.46:0/3868055351 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5c4050040 con 0x5569f9b169f0 2024-03-31T23:13:11.654 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.646+0000 7fd5b2ffd640 1 -- 172.21.15.46:0/3868055351 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7fd5c4004a70 con 0x5569f9b169f0 2024-03-31T23:13:11.654 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.646+0000 7fd5dc7b2740 1 -- 172.21.15.46:0/3868055351 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd5c8084070 con 0x5569f9b169f0 2024-03-31T23:13:11.654 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.646+0000 7fd5b2ffd640 1 -- 172.21.15.46:0/3868055351 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fd5c4045070 con 0x5569f9b169f0 2024-03-31T23:13:11.654 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.646+0000 7fd5dc7b2740 10 client.16167 fetch_fsmap learned FSMap version 27 2024-03-31T23:13:11.654 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.646+0000 7fd5dc7b2740 1 -- 172.21.15.46:0/3868055351 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5569f9b96280 con 0x5569f9b169f0 2024-03-31T23:13:11.654 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.646+0000 7fd5b2ffd640 1 -- 172.21.15.46:0/3868055351 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd5c8084070 con 0x5569f9b169f0 2024-03-31T23:13:11.654 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.646+0000 7fd5b2ffd640 1 -- 172.21.15.46:0/3868055351 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fd5c4053040 con 0x5569f9b169f0 2024-03-31T23:13:11.654 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.646+0000 7fd5dc7b2740 10 client.16167 fetch_fsmap finished waiting for FSMap version 27 2024-03-31T23:13:11.654 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.646+0000 7fd5dc7b2740 10 client.16167 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-31T23:13:11.654 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.646+0000 7fd5dc7b2740 20 client.16167 populate_metadata read hostname 'smithi046' 2024-03-31T23:13:11.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.646+0000 7fd5dc7b2740 1 --2- 172.21.15.46:0/3868055351 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x5569f9b96860 0x5569f9b98c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:13:11.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.646+0000 7fd5dc7b2740 4 client.16167 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-31T23:13:11.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.646+0000 7fd5dc7b2740 1 -- 172.21.15.46:0/3868055351 --> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x5569f9a81730 con 0x5569f9b96860 2024-03-31T23:13:11.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.646+0000 7fd5d2ffd640 1 --2- 172.21.15.46:0/3868055351 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x5569f9b96860 0x5569f9b98c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:13:11.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.647+0000 7fd5b2ffd640 10 client.16167 ms_handle_connect on v2:172.21.15.142:6838/2519143918 2024-03-31T23:13:11.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.647+0000 7fd5d2ffd640 1 --2- 172.21.15.46:0/3868055351 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x5569f9b96860 0x5569f9b98c80 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1f2722c10819720b server_cookie=175447505edb285a in_seq=0 out_seq=0 2024-03-31T23:13:11.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.648+0000 7fd5b2ffd640 1 -- 172.21.15.46:0/3868055351 <== mds.0 v2:172.21.15.142:6838/2519143918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x5569f9a81730 con 0x5569f9b96860 2024-03-31T23:13:11.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.648+0000 7fd5b2ffd640 10 client.16167 handle_command_reply: tid=0 2024-03-31T23:13:11.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.648+0000 7fd5dc7b2740 1 client.16167 shutdown 2024-03-31T23:13:11.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.648+0000 7fd5dc7b2740 1 -- 172.21.15.46:0/3868055351 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5569f9b169f0 msgr2=0x5569f9b19d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:13:11.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.648+0000 7fd5dc7b2740 1 --2- 172.21.15.46:0/3868055351 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5569f9b169f0 0x5569f9b19d60 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7fd5c402bc70 tx=0x7fd5c4004590 comp rx=0 tx=0).stop 2024-03-31T23:13:11.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.648+0000 7fd5dc7b2740 1 -- 172.21.15.46:0/3868055351 shutdown_connections 2024-03-31T23:13:11.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.648+0000 7fd5dc7b2740 1 --2- 172.21.15.46:0/3868055351 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x5569f9b96860 0x5569f9b98c80 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:11.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.648+0000 7fd5dc7b2740 1 --2- 172.21.15.46:0/3868055351 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5569f9b1fef0 0x5569f9b1a7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:11.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.648+0000 7fd5dc7b2740 1 --2- 172.21.15.46:0/3868055351 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5569f9b17310 0x5569f9b1a2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:11.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.648+0000 7fd5dc7b2740 1 --2- 172.21.15.46:0/3868055351 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5569f9b169f0 0x5569f9b19d60 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:11.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.648+0000 7fd5dc7b2740 1 -- 172.21.15.46:0/3868055351 >> 172.21.15.46:0/3868055351 conn(0x5569f9b11120 msgr2=0x5569f9b10180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:13:11.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.648+0000 7fd5b2ffd640 0 client.16167 ms_handle_reset on v2:172.21.15.142:6838/2519143918 2024-03-31T23:13:11.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.648+0000 7fd5dc7b2740 1 -- 172.21.15.46:0/3868055351 shutdown_connections 2024-03-31T23:13:11.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.648+0000 7fd5dc7b2740 1 -- 172.21.15.46:0/3868055351 wait complete. 2024-03-31T23:13:11.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.648+0000 7fd5dc7b2740 20 client.16167 trim_cache size 0 max 16384 2024-03-31T23:13:11.657 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-03-31T23:13:11.657 INFO:teuthology.orchestra.run.smithi046.stdout: "return_code": 0, 2024-03-31T23:13:11.657 INFO:teuthology.orchestra.run.smithi046.stdout: "scrub_tag": "d2e919f2-24dd-4d9d-b010-2a9405c0d9e0", 2024-03-31T23:13:11.657 INFO:teuthology.orchestra.run.smithi046.stdout: "mode": "asynchronous" 2024-03-31T23:13:11.657 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-03-31T23:13:11.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.649+0000 7fd5d9af4640 1 -- 172.21.15.46:0/1966317597 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fd5bc0a27b0 msgr2=0x7fd5bc0a4c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:13:11.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.649+0000 7fd5d9af4640 1 --2- 172.21.15.46:0/1966317597 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fd5bc0a27b0 0x7fd5bc0a4c70 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fd5cc0049d0 tx=0x7fd5cc004ab0 comp rx=0 tx=0).stop 2024-03-31T23:13:11.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.649+0000 7fd5d9af4640 1 -- 172.21.15.46:0/1966317597 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd5d4102090 msgr2=0x7fd5d4073f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:13:11.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.649+0000 7fd5d9af4640 1 --2- 172.21.15.46:0/1966317597 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd5d4102090 0x7fd5d4073f90 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7fd5c800e9f0 tx=0x7fd5c800eeb0 comp rx=0 tx=0).stop 2024-03-31T23:13:11.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.649+0000 7fd5d9af4640 1 -- 172.21.15.46:0/1966317597 shutdown_connections 2024-03-31T23:13:11.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.649+0000 7fd5d9af4640 1 --2- 172.21.15.46:0/1966317597 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fd5bc0a27b0 0x7fd5bc0a4c70 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:11.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.649+0000 7fd5d9af4640 1 --2- 172.21.15.46:0/1966317597 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd5d41076e0 0x7fd5d40744d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:11.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.649+0000 7fd5d9af4640 1 --2- 172.21.15.46:0/1966317597 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd5d4102090 0x7fd5d4073f90 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:11.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.649+0000 7fd5d9af4640 1 --2- 172.21.15.46:0/1966317597 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd5d41016d0 0x7fd5d4073a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:11.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.649+0000 7fd5d9af4640 1 -- 172.21.15.46:0/1966317597 >> 172.21.15.46:0/1966317597 conn(0x7fd5d40fd1e0 msgr2=0x7fd5d40fdba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:13:11.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.649+0000 7fd5d9af4640 1 -- 172.21.15.46:0/1966317597 shutdown_connections 2024-03-31T23:13:11.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.649+0000 7fd5d9af4640 1 -- 172.21.15.46:0/1966317597 wait complete. 2024-03-31T23:13:11.660 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-31T23:13:11.661 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-31T23:13:11.777 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.775+0000 7f52adc3b640 1 -- 172.21.15.46:0/338595008 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f52a8105920 msgr2=0x7f52a810d530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:13:11.777 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.775+0000 7f52adc3b640 1 --2- 172.21.15.46:0/338595008 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f52a8105920 0x7f52a810d530 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f52a0009cd0 tx=0x7f52a002c140 comp rx=0 tx=0).stop 2024-03-31T23:13:11.777 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.775+0000 7f52adc3b640 1 -- 172.21.15.46:0/338595008 shutdown_connections 2024-03-31T23:13:11.777 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.775+0000 7f52adc3b640 1 --2- 172.21.15.46:0/338595008 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f52a8105920 0x7f52a810d530 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:11.777 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.775+0000 7f52adc3b640 1 --2- 172.21.15.46:0/338595008 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f52a8101920 0x7f52a81051f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:11.777 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.775+0000 7f52adc3b640 1 --2- 172.21.15.46:0/338595008 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f52a8100f70 0x7f52a8101350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:11.778 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.775+0000 7f52adc3b640 1 -- 172.21.15.46:0/338595008 >> 172.21.15.46:0/338595008 conn(0x7f52a8076b70 msgr2=0x7f52a80fdf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:13:11.778 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.775+0000 7f52adc3b640 1 -- 172.21.15.46:0/338595008 shutdown_connections 2024-03-31T23:13:11.778 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.775+0000 7f52adc3b640 1 -- 172.21.15.46:0/338595008 wait complete. 2024-03-31T23:13:11.778 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.776+0000 7f52adc3b640 1 Processor -- start 2024-03-31T23:13:11.778 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.776+0000 7f52adc3b640 1 -- start start 2024-03-31T23:13:11.780 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.776+0000 7f52adc3b640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f52a8100f70 0x7f52a80fbfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:13:11.780 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.776+0000 7f52adc3b640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f52a8105920 0x7f52a80fc500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:13:11.780 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.776+0000 7f52adc3b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f52a80fca40 0x7f52a81a6090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:13:11.780 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.776+0000 7f52adc3b640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f52a8110890 con 0x7f52a80fca40 2024-03-31T23:13:11.780 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.776+0000 7f52adc3b640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f52a8110710 con 0x7f52a8100f70 2024-03-31T23:13:11.780 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.776+0000 7f52adc3b640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f52a8110a10 con 0x7f52a8105920 2024-03-31T23:13:11.780 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.777+0000 7f52a7fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f52a80fca40 0x7f52a81a6090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:13:11.780 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.777+0000 7f52a77fe640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f52a8100f70 0x7f52a80fbfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:13:11.780 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.777+0000 7f52a77fe640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f52a8100f70 0x7f52a80fbfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.46:37148/0 (socket says 172.21.15.46:37148) 2024-03-31T23:13:11.781 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.777+0000 7f52a7fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f52a80fca40 0x7f52a81a6090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:33730/0 (socket says 172.21.15.46:33730) 2024-03-31T23:13:11.781 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.777+0000 7f52a77fe640 1 -- 172.21.15.46:0/768508242 learned_addr learned my addr 172.21.15.46:0/768508242 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:13:11.781 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.777+0000 7f52a6ffd640 1 --2- 172.21.15.46:0/768508242 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f52a8105920 0x7f52a80fc500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:13:11.781 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.777+0000 7f52a7fff640 1 -- 172.21.15.46:0/768508242 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f52a8105920 msgr2=0x7f52a80fc500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:13:11.781 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.777+0000 7f52a7fff640 1 --2- 172.21.15.46:0/768508242 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f52a8105920 0x7f52a80fc500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:11.781 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.777+0000 7f52a7fff640 1 -- 172.21.15.46:0/768508242 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f52a8100f70 msgr2=0x7f52a80fbfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:13:11.781 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.777+0000 7f52a7fff640 1 --2- 172.21.15.46:0/768508242 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f52a8100f70 0x7f52a80fbfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:11.781 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.777+0000 7f52a7fff640 1 -- 172.21.15.46:0/768508242 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f52a81a6790 con 0x7f52a80fca40 2024-03-31T23:13:11.781 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.777+0000 7f52a77fe640 1 --2- 172.21.15.46:0/768508242 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f52a8100f70 0x7f52a80fbfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:13:11.781 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.777+0000 7f52a7fff640 1 --2- 172.21.15.46:0/768508242 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f52a80fca40 0x7f52a81a6090 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f52a0037040 tx=0x7f52a0035ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:13:11.781 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.777+0000 7f52a4ff9640 1 -- 172.21.15.46:0/768508242 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52a0039040 con 0x7f52a80fca40 2024-03-31T23:13:11.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.777+0000 7f52a4ff9640 1 -- 172.21.15.46:0/768508242 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f52a0044070 con 0x7f52a80fca40 2024-03-31T23:13:11.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.777+0000 7f52adc3b640 1 -- 172.21.15.46:0/768508242 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f52a81a6a20 con 0x7f52a80fca40 2024-03-31T23:13:11.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.777+0000 7f52a4ff9640 1 -- 172.21.15.46:0/768508242 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52a00042a0 con 0x7f52a80fca40 2024-03-31T23:13:11.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.777+0000 7f52adc3b640 1 -- 172.21.15.46:0/768508242 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f52a81a6ee0 con 0x7f52a80fca40 2024-03-31T23:13:11.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.780+0000 7f52a4ff9640 1 -- 172.21.15.46:0/768508242 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 24) ==== 137553+0+0 (secure 0 0 0) 0x7f52a0003970 con 0x7f52a80fca40 2024-03-31T23:13:11.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.780+0000 7f52a4ff9640 1 --2- 172.21.15.46:0/768508242 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f52900a2760 0x7f52900a4c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:13:11.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.780+0000 7f52b081b740 1 Processor -- start 2024-03-31T23:13:11.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.780+0000 7f52b081b740 1 -- start start 2024-03-31T23:13:11.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.780+0000 7f52a77fe640 1 --2- 172.21.15.46:0/768508242 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f52900a2760 0x7f52900a4c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:13:11.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.780+0000 7f52a4ff9640 1 -- 172.21.15.46:0/768508242 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7f52a00f1310 con 0x7f52a80fca40 2024-03-31T23:13:11.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.780+0000 7f52b081b740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55dfa9033290 0x55dfa9033670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:13:11.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.780+0000 7f52b081b740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55dfa9033bb0 0x55dfa903c250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:13:11.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.781+0000 7f52b081b740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dfa903c790 0x55dfa903eb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:13:11.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.781+0000 7f52b081b740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55dfa8f97180 con 0x55dfa9033290 2024-03-31T23:13:11.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.781+0000 7f52b081b740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55dfa8f3fb70 con 0x55dfa9033bb0 2024-03-31T23:13:11.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.781+0000 7f52b081b740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55dfa8f93220 con 0x55dfa903c790 2024-03-31T23:13:11.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.781+0000 7f52a77fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55dfa9033290 0x55dfa9033670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:13:11.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.781+0000 7f52a77fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55dfa9033290 0x55dfa9033670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:33732/0 (socket says 172.21.15.46:33732) 2024-03-31T23:13:11.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.781+0000 7f52a77fe640 1 -- 172.21.15.46:0/1717137948 learned_addr learned my addr 172.21.15.46:0/1717137948 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:13:11.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.781+0000 7f52a6ffd640 1 --2- 172.21.15.46:0/1717137948 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55dfa9033bb0 0x55dfa903c250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:13:11.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.781+0000 7f52a7fff640 1 --2- 172.21.15.46:0/1717137948 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dfa903c790 0x55dfa903eb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:13:11.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.781+0000 7f52a6ffd640 1 -- 172.21.15.46:0/1717137948 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dfa903c790 msgr2=0x55dfa903eb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:13:11.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.781+0000 7f52a6ffd640 1 --2- 172.21.15.46:0/1717137948 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dfa903c790 0x55dfa903eb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:11.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.781+0000 7f52a6ffd640 1 -- 172.21.15.46:0/1717137948 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55dfa9033290 msgr2=0x55dfa9033670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:13:11.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.781+0000 7f52a6ffd640 1 --2- 172.21.15.46:0/1717137948 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55dfa9033290 0x55dfa9033670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:11.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.781+0000 7f52a6ffd640 1 -- 172.21.15.46:0/1717137948 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dfa8f537c0 con 0x55dfa9033bb0 2024-03-31T23:13:11.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.781+0000 7f52a77fe640 1 --2- 172.21.15.46:0/1717137948 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55dfa9033290 0x55dfa9033670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:13:11.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.781+0000 7f52a77fe640 1 --2- 172.21.15.46:0/768508242 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f52900a2760 0x7f52900a4c20 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f5298013ee0 tx=0x7f52980134e0 comp rx=0 tx=0).ready entity=mgr.15654 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:13:11.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.781+0000 7f52a6ffd640 1 --2- 172.21.15.46:0/1717137948 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55dfa9033bb0 0x55dfa903c250 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f52a8068880 tx=0x7f529c005850 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1a02d2d576b48e8c server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:13:11.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.781+0000 7f528dffb640 1 -- 172.21.15.46:0/1717137948 <== mon.2 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f529c011e30 con 0x55dfa9033bb0 2024-03-31T23:13:11.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.781+0000 7f528dffb640 1 -- 172.21.15.46:0/1717137948 <== mon.2 v2:172.21.15.142:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f529c005020 con 0x55dfa9033bb0 2024-03-31T23:13:11.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.781+0000 7f528dffb640 1 -- 172.21.15.46:0/1717137948 <== mon.2 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f529c004a90 con 0x55dfa9033bb0 2024-03-31T23:13:11.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.782+0000 7f52b081b740 1 -- 172.21.15.46:0/1717137948 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55dfa9033bb0 msgr2=0x55dfa903c250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:13:11.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.782+0000 7f52b081b740 1 --2- 172.21.15.46:0/1717137948 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55dfa9033bb0 0x55dfa903c250 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f52a8068880 tx=0x7f529c005850 comp rx=0 tx=0).stop 2024-03-31T23:13:11.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.782+0000 7f52b081b740 1 -- 172.21.15.46:0/1717137948 shutdown_connections 2024-03-31T23:13:11.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.782+0000 7f52b081b740 1 --2- 172.21.15.46:0/1717137948 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dfa903c790 0x55dfa903eb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:11.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.782+0000 7f52b081b740 1 --2- 172.21.15.46:0/1717137948 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55dfa9033bb0 0x55dfa903c250 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:11.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.782+0000 7f52b081b740 1 --2- 172.21.15.46:0/1717137948 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55dfa9033290 0x55dfa9033670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:11.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.782+0000 7f52b081b740 1 -- 172.21.15.46:0/1717137948 >> 172.21.15.46:0/1717137948 conn(0x55dfa902d9c0 msgr2=0x55dfa902dda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:13:11.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.782+0000 7f52b081b740 1 -- 172.21.15.46:0/1717137948 shutdown_connections 2024-03-31T23:13:11.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.782+0000 7f52b081b740 1 -- 172.21.15.46:0/1717137948 wait complete. 2024-03-31T23:13:11.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.783+0000 7f52b081b740 1 -- start start 2024-03-31T23:13:11.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.783+0000 7f52b081b740 1 Processor -- start 2024-03-31T23:13:11.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.783+0000 7f52b081b740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dfa9033290 0x55dfa90397a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:13:11.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.783+0000 7f52b081b740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55dfa9033bb0 0x55dfa9039ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:13:11.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.783+0000 7f52b081b740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55dfa903c790 0x55dfa903a220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:13:11.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.783+0000 7f52b081b740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55dfa903f580 con 0x55dfa9033bb0 2024-03-31T23:13:11.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.783+0000 7f52b081b740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55dfa8f354a0 con 0x55dfa903c790 2024-03-31T23:13:11.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.783+0000 7f52b081b740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55dfa903f700 con 0x55dfa9033290 2024-03-31T23:13:11.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.783+0000 7f52b081b740 10 client.0 _ll_register_callbacks cb 0x55dfa8f38d90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T23:13:11.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.783+0000 7f52a6ffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55dfa9033bb0 0x55dfa9039ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:13:11.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.783+0000 7f52a6ffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55dfa9033bb0 0x55dfa9039ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:33738/0 (socket says 172.21.15.46:33738) 2024-03-31T23:13:11.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.783+0000 7f52a6ffd640 1 -- 172.21.15.46:0/4071210220 learned_addr learned my addr 172.21.15.46:0/4071210220 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:13:11.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.783+0000 7f52a7fff640 1 --2- 172.21.15.46:0/4071210220 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55dfa903c790 0x55dfa903a220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:13:11.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.784+0000 7f52a77fe640 1 --2- 172.21.15.46:0/4071210220 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dfa9033290 0x55dfa90397a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:13:11.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.784+0000 7f52a6ffd640 1 -- 172.21.15.46:0/4071210220 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dfa9033290 msgr2=0x55dfa90397a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:13:11.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.784+0000 7f52a6ffd640 1 --2- 172.21.15.46:0/4071210220 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dfa9033290 0x55dfa90397a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:11.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.784+0000 7f52a6ffd640 1 -- 172.21.15.46:0/4071210220 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55dfa903c790 msgr2=0x55dfa903a220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:13:11.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.784+0000 7f52a6ffd640 1 --2- 172.21.15.46:0/4071210220 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55dfa903c790 0x55dfa903a220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:11.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.784+0000 7f52a6ffd640 1 -- 172.21.15.46:0/4071210220 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dfa8d757d0 con 0x55dfa9033bb0 2024-03-31T23:13:11.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.784+0000 7f52a6ffd640 1 --2- 172.21.15.46:0/4071210220 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55dfa9033bb0 0x55dfa9039ce0 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f529c0096e0 tx=0x7f529c004370 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f74d2709774af50d server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:13:11.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.784+0000 7f5286ffd640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-31T23:13:11.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.784+0000 7f5286ffd640 1 -- 172.21.15.46:0/4071210220 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f529c028020 con 0x55dfa9033bb0 2024-03-31T23:13:11.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.784+0000 7f5286ffd640 1 -- 172.21.15.46:0/4071210220 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7f529c004810 con 0x55dfa9033bb0 2024-03-31T23:13:11.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.784+0000 7f52b081b740 1 -- 172.21.15.46:0/4071210220 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f529c005020 con 0x55dfa9033bb0 2024-03-31T23:13:11.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.784+0000 7f5286ffd640 1 -- 172.21.15.46:0/4071210220 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f529c033050 con 0x55dfa9033bb0 2024-03-31T23:13:11.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.784+0000 7f52b081b740 10 client.16179 fetch_fsmap learned FSMap version 27 2024-03-31T23:13:11.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.784+0000 7f52b081b740 1 -- 172.21.15.46:0/4071210220 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55dfa8cab3f0 con 0x55dfa9033bb0 2024-03-31T23:13:11.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.785+0000 7f5286ffd640 1 -- 172.21.15.46:0/4071210220 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f529c005020 con 0x55dfa9033bb0 2024-03-31T23:13:11.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.785+0000 7f5286ffd640 1 -- 172.21.15.46:0/4071210220 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f529c018070 con 0x55dfa9033bb0 2024-03-31T23:13:11.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.785+0000 7f52b081b740 10 client.16179 fetch_fsmap finished waiting for FSMap version 27 2024-03-31T23:13:11.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.785+0000 7f52b081b740 10 client.16179 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-31T23:13:11.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.785+0000 7f52b081b740 20 client.16179 populate_metadata read hostname 'smithi046' 2024-03-31T23:13:11.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.785+0000 7f52b081b740 1 --2- 172.21.15.46:0/4071210220 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55dfa904ca90 0x55dfa904eee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:13:11.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.785+0000 7f52b081b740 4 client.16179 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-31T23:13:11.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.785+0000 7f52b081b740 1 -- 172.21.15.46:0/4071210220 --> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55dfa8fa46c0 con 0x55dfa904ca90 2024-03-31T23:13:11.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.785+0000 7f52a77fe640 1 --2- 172.21.15.46:0/4071210220 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55dfa904ca90 0x55dfa904eee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:13:11.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.785+0000 7f5286ffd640 10 client.16179 ms_handle_connect on v2:172.21.15.142:6838/2519143918 2024-03-31T23:13:11.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.785+0000 7f52a77fe640 1 --2- 172.21.15.46:0/4071210220 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55dfa904ca90 0x55dfa904eee0 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b871dead15dfaf64 server_cookie=4eb5f3b7dc50f286 in_seq=0 out_seq=0 2024-03-31T23:13:11.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.786+0000 7f5286ffd640 1 -- 172.21.15.46:0/4071210220 <== mds.0 v2:172.21.15.142:6838/2519143918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55dfa8fa46c0 con 0x55dfa904ca90 2024-03-31T23:13:11.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.786+0000 7f5286ffd640 10 client.16179 handle_command_reply: tid=0 2024-03-31T23:13:11.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.786+0000 7f52b081b740 1 client.16179 shutdown 2024-03-31T23:13:11.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.786+0000 7f52b081b740 1 -- 172.21.15.46:0/4071210220 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55dfa9033bb0 msgr2=0x55dfa9039ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:13:11.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.786+0000 7f52b081b740 1 --2- 172.21.15.46:0/4071210220 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55dfa9033bb0 0x55dfa9039ce0 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f529c0096e0 tx=0x7f529c004370 comp rx=0 tx=0).stop 2024-03-31T23:13:11.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.786+0000 7f52b081b740 1 -- 172.21.15.46:0/4071210220 shutdown_connections 2024-03-31T23:13:11.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.786+0000 7f52b081b740 1 --2- 172.21.15.46:0/4071210220 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55dfa904ca90 0x55dfa904eee0 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:11.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.786+0000 7f5286ffd640 0 client.16179 ms_handle_reset on v2:172.21.15.142:6838/2519143918 2024-03-31T23:13:11.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.786+0000 7f52b081b740 1 --2- 172.21.15.46:0/4071210220 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55dfa903c790 0x55dfa903a220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:11.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.786+0000 7f52b081b740 1 --2- 172.21.15.46:0/4071210220 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55dfa9033bb0 0x55dfa9039ce0 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:11.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.786+0000 7f52b081b740 1 --2- 172.21.15.46:0/4071210220 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dfa9033290 0x55dfa90397a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:11.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.786+0000 7f52b081b740 1 -- 172.21.15.46:0/4071210220 >> 172.21.15.46:0/4071210220 conn(0x55dfa902d9c0 msgr2=0x55dfa902cc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:13:11.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.787+0000 7f52b081b740 1 -- 172.21.15.46:0/4071210220 shutdown_connections 2024-03-31T23:13:11.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.787+0000 7f52b081b740 1 -- 172.21.15.46:0/4071210220 wait complete. 2024-03-31T23:13:11.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.787+0000 7f52b081b740 20 client.16179 trim_cache size 0 max 16384 2024-03-31T23:13:11.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.804+0000 7f52b081b740 1 Processor -- start 2024-03-31T23:13:11.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.804+0000 7f52b081b740 1 -- start start 2024-03-31T23:13:11.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.805+0000 7f52b081b740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dfa9033290 0x55dfa903bed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:13:11.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.805+0000 7f52b081b740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55dfa9033bb0 0x55dfa9048320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:13:11.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.805+0000 7f52b081b740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55dfa903c790 0x55dfa90547b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:13:11.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.805+0000 7f52b081b740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f52a81b3580 con 0x55dfa903c790 2024-03-31T23:13:11.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.805+0000 7f52b081b740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f52a81b0e50 con 0x55dfa9033bb0 2024-03-31T23:13:11.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.805+0000 7f52b081b740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f52a81aeca0 con 0x55dfa9033290 2024-03-31T23:13:11.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.805+0000 7f52a7fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55dfa903c790 0x55dfa90547b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:13:11.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.805+0000 7f52a7fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55dfa903c790 0x55dfa90547b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:33742/0 (socket says 172.21.15.46:33742) 2024-03-31T23:13:11.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.805+0000 7f52a77fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dfa9033290 0x55dfa903bed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:13:11.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.805+0000 7f52a7fff640 1 -- 172.21.15.46:0/2371292298 learned_addr learned my addr 172.21.15.46:0/2371292298 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:13:11.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.805+0000 7f52a6ffd640 1 --2- 172.21.15.46:0/2371292298 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55dfa9033bb0 0x55dfa9048320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:13:11.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.805+0000 7f52a7fff640 1 -- 172.21.15.46:0/2371292298 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dfa9033290 msgr2=0x55dfa903bed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:13:11.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.805+0000 7f52a7fff640 1 --2- 172.21.15.46:0/2371292298 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dfa9033290 0x55dfa903bed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:11.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.805+0000 7f52a7fff640 1 -- 172.21.15.46:0/2371292298 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55dfa9033bb0 msgr2=0x55dfa9048320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:13:11.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.805+0000 7f52a7fff640 1 --2- 172.21.15.46:0/2371292298 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55dfa9033bb0 0x55dfa9048320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:11.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.805+0000 7f52a7fff640 1 -- 172.21.15.46:0/2371292298 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dfa8c6b0f0 con 0x55dfa903c790 2024-03-31T23:13:11.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.805+0000 7f52a6ffd640 1 --2- 172.21.15.46:0/2371292298 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55dfa9033bb0 0x55dfa9048320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:13:11.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.805+0000 7f52a7fff640 1 --2- 172.21.15.46:0/2371292298 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55dfa903c790 0x55dfa90547b0 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f52a0004af0 tx=0x7f52a00a8040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b506bff1c82f18d5 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:13:11.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.805+0000 7f528d7fa640 1 -- 172.21.15.46:0/2371292298 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52a00bd0e0 con 0x55dfa903c790 2024-03-31T23:13:11.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.806+0000 7f528d7fa640 1 -- 172.21.15.46:0/2371292298 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f52a0034700 con 0x55dfa903c790 2024-03-31T23:13:11.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.806+0000 7f528d7fa640 1 -- 172.21.15.46:0/2371292298 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52a00c5910 con 0x55dfa903c790 2024-03-31T23:13:11.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.806+0000 7f52b081b740 1 -- 172.21.15.46:0/2371292298 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55dfa903c790 msgr2=0x55dfa90547b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:13:11.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.806+0000 7f52b081b740 1 --2- 172.21.15.46:0/2371292298 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55dfa903c790 0x55dfa90547b0 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f52a0004af0 tx=0x7f52a00a8040 comp rx=0 tx=0).stop 2024-03-31T23:13:11.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.806+0000 7f52b081b740 1 -- 172.21.15.46:0/2371292298 shutdown_connections 2024-03-31T23:13:11.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.806+0000 7f52b081b740 1 --2- 172.21.15.46:0/2371292298 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55dfa903c790 0x55dfa90547b0 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:11.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.806+0000 7f52b081b740 1 --2- 172.21.15.46:0/2371292298 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55dfa9033bb0 0x55dfa9048320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:11.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.806+0000 7f52b081b740 1 --2- 172.21.15.46:0/2371292298 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dfa9033290 0x55dfa903bed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:11.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.806+0000 7f52b081b740 1 -- 172.21.15.46:0/2371292298 >> 172.21.15.46:0/2371292298 conn(0x55dfa902d9c0 msgr2=0x55dfa902cc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:13:11.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.806+0000 7f52b081b740 1 -- 172.21.15.46:0/2371292298 shutdown_connections 2024-03-31T23:13:11.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.806+0000 7f52b081b740 1 -- 172.21.15.46:0/2371292298 wait complete. 2024-03-31T23:13:11.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.806+0000 7f52b081b740 1 -- start start 2024-03-31T23:13:11.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.807+0000 7f52b081b740 1 Processor -- start 2024-03-31T23:13:11.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.807+0000 7f52b081b740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55dfa9033290 0x55dfa90366c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:13:11.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.807+0000 7f52b081b740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55dfa9033bb0 0x55dfa9036c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:13:11.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.807+0000 7f52b081b740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dfa903c790 0x55dfa90b1de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:13:11.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.807+0000 7f52b081b740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f52a81af310 con 0x55dfa9033290 2024-03-31T23:13:11.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.807+0000 7f52b081b740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f52a81b0780 con 0x55dfa9033bb0 2024-03-31T23:13:11.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.807+0000 7f52b081b740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f52a81af9e0 con 0x55dfa903c790 2024-03-31T23:13:11.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.807+0000 7f52b081b740 10 client.0 _ll_register_callbacks cb 0x7f5298013a30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T23:13:11.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.807+0000 7f52a77fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55dfa9033290 0x55dfa90366c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:13:11.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.807+0000 7f52a77fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55dfa9033290 0x55dfa90366c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:33750/0 (socket says 172.21.15.46:33750) 2024-03-31T23:13:11.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.807+0000 7f52a77fe640 1 -- 172.21.15.46:0/3349330377 learned_addr learned my addr 172.21.15.46:0/3349330377 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:13:11.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.807+0000 7f52a6ffd640 1 --2- 172.21.15.46:0/3349330377 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55dfa9033bb0 0x55dfa9036c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:13:11.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.807+0000 7f52a7fff640 1 --2- 172.21.15.46:0/3349330377 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dfa903c790 0x55dfa90b1de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:13:11.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.807+0000 7f52a77fe640 1 -- 172.21.15.46:0/3349330377 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dfa903c790 msgr2=0x55dfa90b1de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:13:11.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.807+0000 7f52a77fe640 1 --2- 172.21.15.46:0/3349330377 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dfa903c790 0x55dfa90b1de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:11.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.807+0000 7f52a77fe640 1 -- 172.21.15.46:0/3349330377 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55dfa9033bb0 msgr2=0x55dfa9036c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:13:11.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.807+0000 7f52a77fe640 1 --2- 172.21.15.46:0/3349330377 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55dfa9033bb0 0x55dfa9036c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:11.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.807+0000 7f52a77fe640 1 -- 172.21.15.46:0/3349330377 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dfa90374e0 con 0x55dfa9033290 2024-03-31T23:13:11.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.808+0000 7f52a77fe640 1 --2- 172.21.15.46:0/3349330377 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55dfa9033290 0x55dfa90366c0 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f5298009250 tx=0x7f529801d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=697fa4aa502a4b99 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:13:11.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.808+0000 7f5286ffd640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-31T23:13:11.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.808+0000 7f5286ffd640 1 -- 172.21.15.46:0/3349330377 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f529802e060 con 0x55dfa9033290 2024-03-31T23:13:11.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.808+0000 7f5286ffd640 1 -- 172.21.15.46:0/3349330377 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7f5298009030 con 0x55dfa9033290 2024-03-31T23:13:11.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.808+0000 7f52b081b740 1 -- 172.21.15.46:0/3349330377 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55dfa8f9ea10 con 0x55dfa9033290 2024-03-31T23:13:11.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.808+0000 7f5286ffd640 1 -- 172.21.15.46:0/3349330377 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f5298035050 con 0x55dfa9033290 2024-03-31T23:13:11.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.808+0000 7f52b081b740 10 client.16185 fetch_fsmap learned FSMap version 27 2024-03-31T23:13:11.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.808+0000 7f52b081b740 1 -- 172.21.15.46:0/3349330377 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55dfa90b3440 con 0x55dfa9033290 2024-03-31T23:13:11.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.808+0000 7f5286ffd640 1 -- 172.21.15.46:0/3349330377 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x55dfa8f9ea10 con 0x55dfa9033290 2024-03-31T23:13:11.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.808+0000 7f5286ffd640 1 -- 172.21.15.46:0/3349330377 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f5298020050 con 0x55dfa9033290 2024-03-31T23:13:11.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.808+0000 7f52b081b740 10 client.16185 fetch_fsmap finished waiting for FSMap version 27 2024-03-31T23:13:11.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.808+0000 7f52b081b740 10 client.16185 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-31T23:13:11.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.808+0000 7f52b081b740 20 client.16185 populate_metadata read hostname 'smithi046' 2024-03-31T23:13:11.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.808+0000 7f52b081b740 1 --2- 172.21.15.46:0/3349330377 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55dfa904ca90 0x55dfa90b5bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:13:11.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.808+0000 7f52b081b740 4 client.16185 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-31T23:13:11.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.808+0000 7f52b081b740 1 -- 172.21.15.46:0/3349330377 --> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55dfa8f1c540 con 0x55dfa904ca90 2024-03-31T23:13:11.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.809+0000 7f52a6ffd640 1 --2- 172.21.15.46:0/3349330377 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55dfa904ca90 0x55dfa90b5bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:13:11.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.809+0000 7f52a6ffd640 1 --2- 172.21.15.46:0/3349330377 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55dfa904ca90 0x55dfa90b5bf0 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=441c57d8b1267a8b server_cookie=228f66e15f1c9aa in_seq=0 out_seq=0 2024-03-31T23:13:11.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.809+0000 7f5286ffd640 10 client.16185 ms_handle_connect on v2:172.21.15.142:6838/2519143918 2024-03-31T23:13:11.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.809+0000 7f5286ffd640 1 -- 172.21.15.46:0/3349330377 <== mds.0 v2:172.21.15.142:6838/2519143918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55dfa8f1c540 con 0x55dfa904ca90 2024-03-31T23:13:11.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.809+0000 7f5286ffd640 10 client.16185 handle_command_reply: tid=0 2024-03-31T23:13:11.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.809+0000 7f52b081b740 1 client.16185 shutdown 2024-03-31T23:13:11.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.810+0000 7f52b081b740 1 -- 172.21.15.46:0/3349330377 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55dfa9033290 msgr2=0x55dfa90366c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:13:11.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.810+0000 7f52b081b740 1 --2- 172.21.15.46:0/3349330377 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55dfa9033290 0x55dfa90366c0 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f5298009250 tx=0x7f529801d000 comp rx=0 tx=0).stop 2024-03-31T23:13:11.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.810+0000 7f52b081b740 1 -- 172.21.15.46:0/3349330377 shutdown_connections 2024-03-31T23:13:11.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.810+0000 7f52b081b740 1 --2- 172.21.15.46:0/3349330377 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55dfa904ca90 0x55dfa90b5bf0 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:11.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.810+0000 7f52b081b740 1 --2- 172.21.15.46:0/3349330377 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dfa903c790 0x55dfa90b1de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:11.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.810+0000 7f52b081b740 1 --2- 172.21.15.46:0/3349330377 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55dfa9033bb0 0x55dfa9036c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:11.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.810+0000 7f5286ffd640 0 client.16185 ms_handle_reset on v2:172.21.15.142:6838/2519143918 2024-03-31T23:13:11.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.810+0000 7f52b081b740 1 --2- 172.21.15.46:0/3349330377 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55dfa9033290 0x55dfa90366c0 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:11.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.810+0000 7f52b081b740 1 -- 172.21.15.46:0/3349330377 >> 172.21.15.46:0/3349330377 conn(0x55dfa902d9c0 msgr2=0x55dfa9039740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:13:11.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.810+0000 7f52b081b740 1 -- 172.21.15.46:0/3349330377 shutdown_connections 2024-03-31T23:13:11.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.810+0000 7f52b081b740 1 -- 172.21.15.46:0/3349330377 wait complete. 2024-03-31T23:13:11.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.810+0000 7f52b081b740 20 client.16185 trim_cache size 0 max 16384 2024-03-31T23:13:11.819 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-03-31T23:13:11.819 INFO:teuthology.orchestra.run.smithi046.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-31T23:13:11.819 INFO:teuthology.orchestra.run.smithi046.stdout: "scrubs": { 2024-03-31T23:13:11.819 INFO:teuthology.orchestra.run.smithi046.stdout: "d2e919f2-24dd-4d9d-b010-2a9405c0d9e0": { 2024-03-31T23:13:11.819 INFO:teuthology.orchestra.run.smithi046.stdout: "path": "/", 2024-03-31T23:13:11.819 INFO:teuthology.orchestra.run.smithi046.stdout: "tag": "d2e919f2-24dd-4d9d-b010-2a9405c0d9e0", 2024-03-31T23:13:11.819 INFO:teuthology.orchestra.run.smithi046.stdout: "options": "recursive,force" 2024-03-31T23:13:11.819 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-31T23:13:11.819 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-31T23:13:11.819 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-03-31T23:13:11.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.810+0000 7f52adc3b640 1 -- 172.21.15.46:0/768508242 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f52900a2760 msgr2=0x7f52900a4c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:13:11.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.810+0000 7f52adc3b640 1 --2- 172.21.15.46:0/768508242 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f52900a2760 0x7f52900a4c20 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f5298013ee0 tx=0x7f52980134e0 comp rx=0 tx=0).stop 2024-03-31T23:13:11.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.810+0000 7f52adc3b640 1 -- 172.21.15.46:0/768508242 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f52a80fca40 msgr2=0x7f52a81a6090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:13:11.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.810+0000 7f52adc3b640 1 --2- 172.21.15.46:0/768508242 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f52a80fca40 0x7f52a81a6090 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f52a0037040 tx=0x7f52a0035ce0 comp rx=0 tx=0).stop 2024-03-31T23:13:11.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.810+0000 7f52adc3b640 1 -- 172.21.15.46:0/768508242 shutdown_connections 2024-03-31T23:13:11.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.810+0000 7f52adc3b640 1 --2- 172.21.15.46:0/768508242 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f52900a2760 0x7f52900a4c20 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:11.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.810+0000 7f52adc3b640 1 --2- 172.21.15.46:0/768508242 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f52a80fca40 0x7f52a81a6090 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:11.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.810+0000 7f52adc3b640 1 --2- 172.21.15.46:0/768508242 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f52a8105920 0x7f52a80fc500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:11.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.810+0000 7f52adc3b640 1 --2- 172.21.15.46:0/768508242 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f52a8100f70 0x7f52a80fbfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:11.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.810+0000 7f52adc3b640 1 -- 172.21.15.46:0/768508242 >> 172.21.15.46:0/768508242 conn(0x7f52a8076b70 msgr2=0x7f52a8100000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:13:11.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.810+0000 7f52adc3b640 1 -- 172.21.15.46:0/768508242 shutdown_connections 2024-03-31T23:13:11.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:11.811+0000 7f52adc3b640 1 -- 172.21.15.46:0/768508242 wait complete. 2024-03-31T23:13:11.822 INFO:tasks.cephfs.filesystem:scrub status for tag:d2e919f2-24dd-4d9d-b010-2a9405c0d9e0 - {'path': '/', 'tag': 'd2e919f2-24dd-4d9d-b010-2a9405c0d9e0', 'options': 'recursive,force'} 2024-03-31T23:13:12.048 INFO:journalctl@ceph.mds.c.smithi162.stdout:Mar 31 23:13:11 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mds-c[67067]: 2024-03-31T23:13:11.661+0000 7f45de802700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000021f [4,head] /volumes/qa/sv_0/e990713a-13d9-4c54-877d-f0f60b19e88e/client.0/tmp/clients/client0/~dmtmp/COREL/ROSE.CPT auth v292 ap=1 s=260096 n(v0 rc2024-03-31T23:12:54.456509+0000 b260096 1=1+0) caps={15462=pAsLsXsFsc/-@1},l=15462(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x55c95b9d6c00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000021f:[<0x10000000215/ROSE.CPT v90>,<0x10000000009/COREL v1586>,<0x10000000008/~dmtmp v93>,<0x10000000007/client0 v41>,<0x10000000006/clients v21>,<0x10000000005/tmp v21>,<0x10000000003/client.0 v22>,<0x10000000002/e990713a-13d9-4c54-877d-f0f60b19e88e v36>,<0x10000000001/sv_0 v58>,<0x10000000000/qa v46>,<0x1/volumes v44>]//[3]","memoryvalue":"(4)0x1000000021f:[<0x10000000215/ROSE.CPT v292>,<0x10000000009/COREL v2308>,<0x10000000008/~dmtmp v96>,<0x10000000007/client0 v44>,<0x10000000006/clients v24>,<0x10000000005/tmp v24>,<0x10000000003/client.0 v24>,<0x10000000002/e990713a-13d9-4c54-877d-f0f60b19e88e v39>,<0x10000000001/sv_0 v60>,<0x10000000000/qa v32>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-31T23:13:12.048 INFO:journalctl@ceph.mds.c.smithi162.stdout:Mar 31 23:13:11 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mds-c[67067]: 2024-03-31T23:13:11.661+0000 7f45de802700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000021b [4,head] /volumes/qa/sv_0/e990713a-13d9-4c54-877d-f0f60b19e88e/client.0/tmp/clients/client0/~dmtmp/COREL/TEAPOT2.CPT auth v284 ap=1 s=260096 n(v0 rc2024-03-31T23:12:54.420804+0000 b260096 1=1+0) caps={15462=pAsLsXsFsc/-@1},l=15462(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x55c95b9d7700]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000021b:[<0x10000000215/TEAPOT2.CPT v54>,<0x10000000009/COREL v1586>,<0x10000000008/~dmtmp v93>,<0x10000000007/client0 v41>,<0x10000000006/clients v21>,<0x10000000005/tmp v21>,<0x10000000003/client.0 v22>,<0x10000000002/e990713a-13d9-4c54-877d-f0f60b19e88e v36>,<0x10000000001/sv_0 v58>,<0x10000000000/qa v46>,<0x1/volumes v44>]//[3]","memoryvalue":"(4)0x1000000021b:[<0x10000000215/TEAPOT2.CPT v284>,<0x10000000009/COREL v2308>,<0x10000000008/~dmtmp v96>,<0x10000000007/client0 v44>,<0x10000000006/clients v24>,<0x10000000005/tmp v24>,<0x10000000003/client.0 v24>,<0x10000000002/e990713a-13d9-4c54-877d-f0f60b19e88e v39>,<0x10000000001/sv_0 v60>,<0x10000000000/qa v32>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-31T23:13:12.048 INFO:journalctl@ceph.mds.c.smithi162.stdout:Mar 31 23:13:11 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mds-c[67067]: 2024-03-31T23:13:11.661+0000 7f45de802700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000220 [4,head] /volumes/qa/sv_0/e990713a-13d9-4c54-877d-f0f60b19e88e/client.0/tmp/clients/client0/~dmtmp/COREL/CORELDRW.CDT auth v290 ap=1 s=260096 n(v0 rc2024-03-31T23:12:54.465133+0000 b260096 1=1+0) caps={15462=pAsLsXsFsc/-@1},l=15462(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x55c95b9ef080]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000220:[<0x10000000215/CORELDRW.CDT v38>,<0x10000000009/COREL v1586>,<0x10000000008/~dmtmp v93>,<0x10000000007/client0 v41>,<0x10000000006/clients v21>,<0x10000000005/tmp v21>,<0x10000000003/client.0 v22>,<0x10000000002/e990713a-13d9-4c54-877d-f0f60b19e88e v36>,<0x10000000001/sv_0 v58>,<0x10000000000/qa v46>,<0x1/volumes v44>]//[3]","memoryvalue":"(4)0x10000000220:[<0x10000000215/CORELDRW.CDT v290>,<0x10000000009/COREL v2308>,<0x10000000008/~dmtmp v96>,<0x10000000007/client0 v44>,<0x10000000006/clients v24>,<0x10000000005/tmp v24>,<0x10000000003/client.0 v24>,<0x10000000002/e990713a-13d9-4c54-877d-f0f60b19e88e v39>,<0x10000000001/sv_0 v60>,<0x10000000000/qa v32>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-31T23:13:12.048 INFO:journalctl@ceph.mds.c.smithi162.stdout:Mar 31 23:13:11 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mds-c[67067]: 2024-03-31T23:13:11.662+0000 7f45de802700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000021a [4,head] /volumes/qa/sv_0/e990713a-13d9-4c54-877d-f0f60b19e88e/client.0/tmp/clients/client0/~dmtmp/COREL/MAILBOX2.CPT auth v282 ap=1 s=260096 n(v0 rc2024-03-31T23:12:54.411834+0000 b260096 1=1+0) caps={15462=pAsLsXsFsc/-@1},l=15462(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x55c95b9ee580]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000021a:[<0x10000000215/MAILBOX2.CPT v52>,<0x10000000009/COREL v1586>,<0x10000000008/~dmtmp v93>,<0x10000000007/client0 v41>,<0x10000000006/clients v21>,<0x10000000005/tmp v21>,<0x10000000003/client.0 v22>,<0x10000000002/e990713a-13d9-4c54-877d-f0f60b19e88e v36>,<0x10000000001/sv_0 v58>,<0x10000000000/qa v46>,<0x1/volumes v44>]//[3]","memoryvalue":"(4)0x1000000021a:[<0x10000000215/MAILBOX2.CPT v282>,<0x10000000009/COREL v2308>,<0x10000000008/~dmtmp v96>,<0x10000000007/client0 v44>,<0x10000000006/clients v24>,<0x10000000005/tmp v24>,<0x10000000003/client.0 v24>,<0x10000000002/e990713a-13d9-4c54-877d-f0f60b19e88e v39>,<0x10000000001/sv_0 v60>,<0x10000000000/qa v32>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-31T23:13:12.048 INFO:journalctl@ceph.mds.c.smithi162.stdout:Mar 31 23:13:11 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mds-c[67067]: 2024-03-31T23:13:11.662+0000 7f45de802700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000021e [4,head] /volumes/qa/sv_0/e990713a-13d9-4c54-877d-f0f60b19e88e/client.0/tmp/clients/client0/~dmtmp/COREL/GRAPHIC2.CDR auth v288 ap=1 s=260096 n(v0 rc2024-03-31T23:12:54.447441+0000 b260096 1=1+0) caps={15462=pAsLsXsFsc/-@1},l=15462(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x55c95b9f0100]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000021e:[<0x10000000215/GRAPHIC2.CDR v78>,<0x10000000009/COREL v1586>,<0x10000000008/~dmtmp v93>,<0x10000000007/client0 v41>,<0x10000000006/clients v21>,<0x10000000005/tmp v21>,<0x10000000003/client.0 v22>,<0x10000000002/e990713a-13d9-4c54-877d-f0f60b19e88e v36>,<0x10000000001/sv_0 v58>,<0x10000000000/qa v46>,<0x1/volumes v44>]//[3]","memoryvalue":"(4)0x1000000021e:[<0x10000000215/GRAPHIC2.CDR v288>,<0x10000000009/COREL v2308>,<0x10000000008/~dmtmp v96>,<0x10000000007/client0 v44>,<0x10000000006/clients v24>,<0x10000000005/tmp v24>,<0x10000000003/client.0 v24>,<0x10000000002/e990713a-13d9-4c54-877d-f0f60b19e88e v39>,<0x10000000001/sv_0 v60>,<0x10000000000/qa v32>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-31T23:13:12.048 INFO:journalctl@ceph.mds.c.smithi162.stdout:Mar 31 23:13:11 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mds-c[67067]: 2024-03-31T23:13:11.662+0000 7f45de802700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000218 [4,head] /volumes/qa/sv_0/e990713a-13d9-4c54-877d-f0f60b19e88e/client.0/tmp/clients/client0/~dmtmp/COREL/ARTISTIC.ACL auth v276 ap=1 s=260096 n(v0 rc2024-03-31T23:12:54.395464+0000 b260096 1=1+0) caps={15462=pAsLsXsFsc/-@1},l=15462(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x55c95b9f1180]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000218:[<0x10000000215/ARTISTIC.ACL v32>,<0x10000000009/COREL v1586>,<0x10000000008/~dmtmp v93>,<0x10000000007/client0 v41>,<0x10000000006/clients v21>,<0x10000000005/tmp v21>,<0x10000000003/client.0 v22>,<0x10000000002/e990713a-13d9-4c54-877d-f0f60b19e88e v36>,<0x10000000001/sv_0 v58>,<0x10000000000/qa v46>,<0x1/volumes v44>]//[3]","memoryvalue":"(4)0x10000000218:[<0x10000000215/ARTISTIC.ACL v276>,<0x10000000009/COREL v2308>,<0x10000000008/~dmtmp v96>,<0x10000000007/client0 v44>,<0x10000000006/clients v24>,<0x10000000005/tmp v24>,<0x10000000003/client.0 v24>,<0x10000000002/e990713a-13d9-4c54-877d-f0f60b19e88e v39>,<0x10000000001/sv_0 v60>,<0x10000000000/qa v32>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-31T23:13:12.048 INFO:journalctl@ceph.mds.c.smithi162.stdout:Mar 31 23:13:11 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mds-c[67067]: 2024-03-31T23:13:11.663+0000 7f45de802700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000021c [4,head] /volumes/qa/sv_0/e990713a-13d9-4c54-877d-f0f60b19e88e/client.0/tmp/clients/client0/~dmtmp/COREL/SIGN2.CPT auth v286 ap=1 s=260096 n(v0 rc2024-03-31T23:12:54.429699+0000 b260096 1=1+0) caps={15462=pAsLsXsFsc/-@1},l=15462(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x55c95b9f8580]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000021c:[<0x10000000215/SIGN2.CPT v56>,<0x10000000009/COREL v1586>,<0x10000000008/~dmtmp v93>,<0x10000000007/client0 v41>,<0x10000000006/clients v21>,<0x10000000005/tmp v21>,<0x10000000003/client.0 v22>,<0x10000000002/e990713a-13d9-4c54-877d-f0f60b19e88e v36>,<0x10000000001/sv_0 v58>,<0x10000000000/qa v46>,<0x1/volumes v44>]//[3]","memoryvalue":"(4)0x1000000021c:[<0x10000000215/SIGN2.CPT v286>,<0x10000000009/COREL v2308>,<0x10000000008/~dmtmp v96>,<0x10000000007/client0 v44>,<0x10000000006/clients v24>,<0x10000000005/tmp v24>,<0x10000000003/client.0 v24>,<0x10000000002/e990713a-13d9-4c54-877d-f0f60b19e88e v39>,<0x10000000001/sv_0 v60>,<0x10000000000/qa v32>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-31T23:13:12.048 INFO:journalctl@ceph.mds.c.smithi162.stdout:Mar 31 23:13:11 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mds-c[67067]: 2024-03-31T23:13:11.663+0000 7f45de802700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000021d [4,head] /volumes/qa/sv_0/e990713a-13d9-4c54-877d-f0f60b19e88e/client.0/tmp/clients/client0/~dmtmp/COREL/TRAIN.CDR auth v280 ap=1 s=260096 n(v0 rc2024-03-31T23:12:54.439024+0000 b260096 1=1+0) caps={15462=pAsLsXsFsc/-@1},l=15462(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x55c95b9fa100]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000021d:[<0x10000000215/TRAIN.CDR v58>,<0x10000000009/COREL v1586>,<0x10000000008/~dmtmp v93>,<0x10000000007/client0 v41>,<0x10000000006/clients v21>,<0x10000000005/tmp v21>,<0x10000000003/client.0 v22>,<0x10000000002/e990713a-13d9-4c54-877d-f0f60b19e88e v36>,<0x10000000001/sv_0 v58>,<0x10000000000/qa v46>,<0x1/volumes v44>]//[3]","memoryvalue":"(4)0x1000000021d:[<0x10000000215/TRAIN.CDR v280>,<0x10000000009/COREL v2308>,<0x10000000008/~dmtmp v96>,<0x10000000007/client0 v44>,<0x10000000006/clients v24>,<0x10000000005/tmp v24>,<0x10000000003/client.0 v24>,<0x10000000002/e990713a-13d9-4c54-877d-f0f60b19e88e v39>,<0x10000000001/sv_0 v60>,<0x10000000000/qa v32>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-31T23:13:12.048 INFO:journalctl@ceph.mds.c.smithi162.stdout:Mar 31 23:13:11 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mds-c[67067]: 2024-03-31T23:13:11.664+0000 7f45de802700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000219 [4,head] /volumes/qa/sv_0/e990713a-13d9-4c54-877d-f0f60b19e88e/client.0/tmp/clients/client0/~dmtmp/COREL/GRAPHIC1.CDR auth v278 ap=1 s=260096 n(v0 rc2024-03-31T23:12:54.404096+0000 b260096 1=1+0) caps={15462=pAsLsXsFsc/-@1},l=15462(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x55c95b9fac00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000219:[<0x10000000215/GRAPHIC1.CDR v40>,<0x10000000009/COREL v1586>,<0x10000000008/~dmtmp v93>,<0x10000000007/client0 v41>,<0x10000000006/clients v21>,<0x10000000005/tmp v21>,<0x10000000003/client.0 v22>,<0x10000000002/e990713a-13d9-4c54-877d-f0f60b19e88e v36>,<0x10000000001/sv_0 v58>,<0x10000000000/qa v46>,<0x1/volumes v44>]//[3]","memoryvalue":"(4)0x10000000219:[<0x10000000215/GRAPHIC1.CDR v278>,<0x10000000009/COREL v2308>,<0x10000000008/~dmtmp v96>,<0x10000000007/client0 v44>,<0x10000000006/clients v24>,<0x10000000005/tmp v24>,<0x10000000003/client.0 v24>,<0x10000000002/e990713a-13d9-4c54-877d-f0f60b19e88e v39>,<0x10000000001/sv_0 v60>,<0x10000000000/qa v32>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-31T23:13:12.049 INFO:journalctl@ceph.mds.c.smithi162.stdout:Mar 31 23:13:11 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mds-c[67067]: 2024-03-31T23:13:11.665+0000 7f45de802700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000217 [4,head] /volumes/qa/sv_0/e990713a-13d9-4c54-877d-f0f60b19e88e/client.0/tmp/clients/client0/~dmtmp/COREL/CDRBARS.CFG auth v294 ap=1 s=260096 n(v0 rc2024-03-31T23:13:05.696921+0000 b260096 1=1+0) caps={15462=pAsLsXsFsc/-@1},l=15462(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x55c95b9f9080]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000217:[<0x10000000215/CDRBARS.CFG v116>,<0x10000000009/COREL v1586>,<0x10000000008/~dmtmp v93>,<0x10000000007/client0 v41>,<0x10000000006/clients v21>,<0x10000000005/tmp v21>,<0x10000000003/client.0 v22>,<0x10000000002/e990713a-13d9-4c54-877d-f0f60b19e88e v36>,<0x10000000001/sv_0 v58>,<0x10000000000/qa v46>,<0x1/volumes v44>]//[3]","memoryvalue":"(4)0x10000000217:[<0x10000000215/CDRBARS.CFG v294>,<0x10000000009/COREL v2308>,<0x10000000008/~dmtmp v96>,<0x10000000007/client0 v44>,<0x10000000006/clients v24>,<0x10000000005/tmp v24>,<0x10000000003/client.0 v24>,<0x10000000002/e990713a-13d9-4c54-877d-f0f60b19e88e v39>,<0x10000000001/sv_0 v60>,<0x10000000000/qa v32>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-31T23:13:12.049 INFO:journalctl@ceph.mds.c.smithi162.stdout:Mar 31 23:13:11 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mds-c[67067]: 2024-03-31T23:13:11.665+0000 7f45de802700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000216 [4,head] /volumes/qa/sv_0/e990713a-13d9-4c54-877d-f0f60b19e88e/client.0/tmp/clients/client0/~dmtmp/COREL/CDRROLS.CFG auth v296 ap=1 s=260096 n(v0 rc2024-03-31T23:13:05.713089+0000 b260096 1=1+0) caps={15462=pAsLsXsFsc/-@1},l=15462(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x55c95b9fb700]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000216:[<0x10000000215/CDRROLS.CFG v114>,<0x10000000009/COREL v1586>,<0x10000000008/~dmtmp v93>,<0x10000000007/client0 v41>,<0x10000000006/clients v21>,<0x10000000005/tmp v21>,<0x10000000003/client.0 v22>,<0x10000000002/e990713a-13d9-4c54-877d-f0f60b19e88e v36>,<0x10000000001/sv_0 v58>,<0x10000000000/qa v46>,<0x1/volumes v44>]//[3]","memoryvalue":"(4)0x10000000216:[<0x10000000215/CDRROLS.CFG v296>,<0x10000000009/COREL v2310>,<0x10000000008/~dmtmp v96>,<0x10000000007/client0 v44>,<0x10000000006/clients v24>,<0x10000000005/tmp v24>,<0x10000000003/client.0 v24>,<0x10000000002/e990713a-13d9-4c54-877d-f0f60b19e88e v39>,<0x10000000001/sv_0 v60>,<0x10000000000/qa v32>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-31T23:13:12.049 INFO:journalctl@ceph.mds.c.smithi162.stdout:Mar 31 23:13:11 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mds-c[67067]: 2024-03-31T23:13:11.665+0000 7f45de802700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000020b [4,head] /volumes/qa/sv_0/e990713a-13d9-4c54-877d-f0f60b19e88e/client.0/tmp/clients/client0/~dmtmp/WORD/FACTS.DOC auth v318 ap=1 s=260096 n(v0 rc2024-03-31T23:12:54.300554+0000 b260096 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15462=0-4194304@3} caps={15462=pAsxLsXsxFsxcrwb/pAsxXsxFsxcrwb@18},l=15462 | ptrwaiter=0 request=0 lock=0 importingcaps=0 caps=1 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x55c95b9d4b00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000020b:[<0x10000000208/FACTS.DOC v182>,<0x10000000009/WORD v1582>,<0x10000000008/~dmtmp v93>,<0x10000000007/client0 v41>,<0x10000000006/clients v21>,<0x10000000005/tmp v21>,<0x10000000003/client.0 v22>,<0x10000000002/e990713a-13d9-4c54-877d-f0f60b19e88e v36>,<0x10000000001/sv_0 v58>,<0x10000000000/qa v46>,<0x1/volumes v44>]//[3]","memoryvalue":"(4)0x1000000020b:[<0x10000000208/FACTS.DOC v318>,<0x10000000009/WORD v2442>,<0x10000000008/~dmtmp v96>,<0x10000000007/client0 v44>,<0x10000000006/clients v24>,<0x10000000005/tmp v24>,<0x10000000003/client.0 v24>,<0x10000000002/e990713a-13d9-4c54-877d-f0f60b19e88e v39>,<0x10000000001/sv_0 v60>,<0x10000000000/qa v32>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-31T23:13:12.049 INFO:journalctl@ceph.mds.c.smithi162.stdout:Mar 31 23:13:11 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mds-c[67067]: 2024-03-31T23:13:11.665+0000 7f45de802700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000020a [4,head] /volumes/qa/sv_0/e990713a-13d9-4c54-877d-f0f60b19e88e/client.0/tmp/clients/client0/~dmtmp/WORD/BASEMACH.DOC auth v291 ap=1 s=260096 n(v0 rc2024-03-31T23:12:54.292207+0000 b260096 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15462=0-4194304@3} caps={15462=pAsxLsXsxFsxcrwb/pAsxXsxFsxcrwb@18},l=15462 | ptrwaiter=0 request=0 lock=0 importingcaps=0 caps=1 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x55c95b9d4000]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000020a:[<0x10000000208/BASEMACH.DOC v158>,<0x10000000009/WORD v1582>,<0x10000000008/~dmtmp v93>,<0x10000000007/client0 v41>,<0x10000000006/clients v21>,<0x10000000005/tmp v21>,<0x10000000003/client.0 v22>,<0x10000000002/e990713a-13d9-4c54-877d-f0f60b19e88e v36>,<0x10000000001/sv_0 v58>,<0x10000000000/qa v46>,<0x1/volumes v44>]//[3]","memoryvalue":"(4)0x1000000020a:[<0x10000000208/BASEMACH.DOC v291>,<0x10000000009/WORD v2442>,<0x10000000008/~dmtmp v96>,<0x10000000007/client0 v44>,<0x10000000006/clients v24>,<0x10000000005/tmp v24>,<0x10000000003/client.0 v24>,<0x10000000002/e990713a-13d9-4c54-877d-f0f60b19e88e v39>,<0x10000000001/sv_0 v60>,<0x10000000000/qa v32>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-31T23:13:12.049 INFO:journalctl@ceph.mds.c.smithi162.stdout:Mar 31 23:13:11 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mds-c[67067]: 2024-03-31T23:13:11.666+0000 7f45de802700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000209 [4,head] /volumes/qa/sv_0/e990713a-13d9-4c54-877d-f0f60b19e88e/client.0/tmp/clients/client0/~dmtmp/WORD/CHAP10.DOC auth v255 ap=1 s=260096 n(v0 rc2024-03-31T23:12:54.283800+0000 b260096 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15462=0-4194304@3} caps={15462=pAsxLsXsxFsxcrwb/pAsxXsxFsxcrwb@18},l=15462 | ptrwaiter=0 request=0 lock=0 importingcaps=0 caps=1 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x55c95b9d5b80]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000209:[<0x10000000208/CHAP10.DOC v130>,<0x10000000009/WORD v1582>,<0x10000000008/~dmtmp v93>,<0x10000000007/client0 v41>,<0x10000000006/clients v21>,<0x10000000005/tmp v21>,<0x10000000003/client.0 v22>,<0x10000000002/e990713a-13d9-4c54-877d-f0f60b19e88e v36>,<0x10000000001/sv_0 v58>,<0x10000000000/qa v46>,<0x1/volumes v44>]//[3]","memoryvalue":"(4)0x10000000209:[<0x10000000208/CHAP10.DOC v255>,<0x10000000009/WORD v2442>,<0x10000000008/~dmtmp v96>,<0x10000000007/client0 v44>,<0x10000000006/clients v24>,<0x10000000005/tmp v24>,<0x10000000003/client.0 v24>,<0x10000000002/e990713a-13d9-4c54-877d-f0f60b19e88e v39>,<0x10000000001/sv_0 v60>,<0x10000000000/qa v32>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-31T23:13:12.049 INFO:journalctl@ceph.mds.c.smithi162.stdout:Mar 31 23:13:11 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mds-c[67067]: 2024-03-31T23:13:11.666+0000 7f45de802700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000241 [4,head] /volumes/qa/sv_0/e990713a-13d9-4c54-877d-f0f60b19e88e/client.0/tmp/clients/client0/~dmtmp/SEED/SMALL.FIL auth v62 pv68 ap=2 s=327680 n(v0 rc2024-03-31T23:13:05.714460+0000 b327680 1=1+0)->n(v0 rc2024-03-31T23:13:10.649104+0000 b327680 1=1+0) (iauth excl) (ifile excl w=1) (ixattr excl) cr={15462=0-4194304@3} caps={15462=pAsxLsXsxFsxcrwb/pAsxXsxFsxcrwb@2},l=15462 | ptrwaiter=1 request=0 lock=1 importingcaps=0 caps=1 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x55c95ba58680]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000241:[<0x10000000240/SMALL.FIL v44>,<0x10000000009/SEED v1592>,<0x10000000008/~dmtmp v93>,<0x10000000007/client0 v41>,<0x10000000006/clients v21>,<0x10000000005/tmp v21>,<0x10000000003/client.0 v22>,<0x10000000002/e990713a-13d9-4c54-877d-f0f60b19e88e v36>,<0x10000000001/sv_0 v58>,<0x10000000000/qa v46>,<0x1/volumes v44>]//[3]","memoryvalue":"(4)0x10000000241:[<0x10000000240/SMALL.FIL v62>,<0x10000000009/SEED v1992>,<0x10000000008/~dmtmp v96>,<0x10000000007/client0 v44>,<0x10000000006/clients v24>,<0x10000000005/tmp v24>,<0x10000000003/client.0 v24>,<0x10000000002/e990713a-13d9-4c54-877d-f0f60b19e88e v39>,<0x10000000001/sv_0 v60>,<0x10000000000/qa v32>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-31T23:13:12.049 INFO:journalctl@ceph.mds.c.smithi162.stdout:Mar 31 23:13:11 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mds-c[67067]: 2024-03-31T23:13:11.667+0000 7f45de802700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000242 [4,head] /volumes/qa/sv_0/e990713a-13d9-4c54-877d-f0f60b19e88e/client.0/tmp/clients/client0/~dmtmp/SEED/MEDIUM.FIL auth v64 pv70 ap=2 s=1048576 n(v0 rc2024-03-31T23:13:05.716005+0000 b1048576 1=1+0)->n(v0 rc2024-03-31T23:13:10.651517+0000 b1048576 1=1+0) (iauth excl) (ifile excl w=1) (ixattr excl) cr={15462=0-4194304@3} caps={15462=pAsxLsXsxFsxcrwb/pAsxXsxFsxcrwb@2},l=15462 | ptrwaiter=1 request=0 lock=1 importingcaps=0 caps=1 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x55c95ba59180]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000242:[<0x10000000240/MEDIUM.FIL v46>,<0x10000000009/SEED v1592>,<0x10000000008/~dmtmp v93>,<0x10000000007/client0 v41>,<0x10000000006/clients v21>,<0x10000000005/tmp v21>,<0x10000000003/client.0 v22>,<0x10000000002/e990713a-13d9-4c54-877d-f0f60b19e88e v36>,<0x10000000001/sv_0 v58>,<0x10000000000/qa v46>,<0x1/volumes v44>]//[3]","memoryvalue":"(4)0x10000000242:[<0x10000000240/MEDIUM.FIL v64>,<0x10000000009/SEED v1992>,<0x10000000008/~dmtmp v96>,<0x10000000007/client0 v44>,<0x10000000006/clients v24>,<0x10000000005/tmp v24>,<0x10000000003/client.0 v24>,<0x10000000002/e990713a-13d9-4c54-877d-f0f60b19e88e v39>,<0x10000000001/sv_0 v60>,<0x10000000000/qa v32>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-31T23:13:12.049 INFO:journalctl@ceph.mds.c.smithi162.stdout:Mar 31 23:13:11 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mds-c[67067]: 2024-03-31T23:13:11.667+0000 7f45de802700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000222 [4,head] /volumes/qa/sv_0/e990713a-13d9-4c54-877d-f0f60b19e88e/client.0/tmp/clients/client0/~dmtmp/WORDPRO/BENCHS.LWP auth v262 ap=1 s=260096 n(v0 rc2024-03-31T23:12:54.475028+0000 b260096 1=1+0) caps={15462=pAsLsXsFsc/-@1},l=15462(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x55c95baaeb00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000222:[<0x10000000221/BENCHS.LWP v138>,<0x10000000009/WORDPRO v1781>,<0x10000000008/~dmtmp v93>,<0x10000000007/client0 v41>,<0x10000000006/clients v21>,<0x10000000005/tmp v21>,<0x10000000003/client.0 v22>,<0x10000000002/e990713a-13d9-4c54-877d-f0f60b19e88e v36>,<0x10000000001/sv_0 v58>,<0x10000000000/qa v46>,<0x1/volumes v44>]//[3]","memoryvalue":"(4)0x10000000222:[<0x10000000221/BENCHS.LWP v262>,<0x10000000009/WORDPRO v2324>,<0x10000000008/~dmtmp v96>,<0x10000000007/client0 v44>,<0x10000000006/clients v24>,<0x10000000005/tmp v24>,<0x10000000003/client.0 v24>,<0x10000000002/e990713a-13d9-4c54-877d-f0f60b19e88e v39>,<0x10000000001/sv_0 v60>,<0x10000000000/qa v32>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-31T23:13:12.049 INFO:journalctl@ceph.mds.c.smithi162.stdout:Mar 31 23:13:11 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mds-c[67067]: 2024-03-31T23:13:11.667+0000 7f45de802700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000243 [4,head] /volumes/qa/sv_0/e990713a-13d9-4c54-877d-f0f60b19e88e/client.0/tmp/clients/client0/~dmtmp/SEED/LARGE.FIL auth v66 pv72 ap=2 s=1867776 n(v0 rc2024-03-31T23:13:05.718049+0000 b1867776 1=1+0)->n(v0 rc2024-03-31T23:13:10.654609+0000 b1867776 1=1+0) (iauth excl) (ifile excl w=1) (ixattr excl) cr={15462=0-4194304@3} caps={15462=pAsxLsXsxFsxcrwb/pAsxXsxFsxcrwb@2},l=15462 | ptrwaiter=1 request=0 lock=1 importingcaps=0 caps=1 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x55c95baae000]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000243:[<0x10000000240/LARGE.FIL v48>,<0x10000000009/SEED v1592>,<0x10000000008/~dmtmp v93>,<0x10000000007/client0 v41>,<0x10000000006/clients v21>,<0x10000000005/tmp v21>,<0x10000000003/client.0 v22>,<0x10000000002/e990713a-13d9-4c54-877d-f0f60b19e88e v36>,<0x10000000001/sv_0 v58>,<0x10000000000/qa v46>,<0x1/volumes v44>]//[3]","memoryvalue":"(4)0x10000000243:[<0x10000000240/LARGE.FIL v66>,<0x10000000009/SEED v1992>,<0x10000000008/~dmtmp v96>,<0x10000000007/client0 v44>,<0x10000000006/clients v24>,<0x10000000005/tmp v24>,<0x10000000003/client.0 v24>,<0x10000000002/e990713a-13d9-4c54-877d-f0f60b19e88e v39>,<0x10000000001/sv_0 v60>,<0x10000000000/qa v32>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-31T23:13:12.049 INFO:journalctl@ceph.mds.c.smithi162.stdout:Mar 31 23:13:11 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mds-c[67067]: 2024-03-31T23:13:11.668+0000 7f45de802700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000223 [4,head] /volumes/qa/sv_0/e990713a-13d9-4c54-877d-f0f60b19e88e/client.0/tmp/clients/client0/~dmtmp/WORDPRO/RESULTS.XLS auth v264 ap=1 s=260096 n(v0 rc2024-03-31T23:12:54.483136+0000 b260096 1=1+0) caps={15462=pAsLsXsFsc/-@1},l=15462(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x55c95baaf600]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000223:[<0x10000000221/RESULTS.XLS v140>,<0x10000000009/WORDPRO v1781>,<0x10000000008/~dmtmp v93>,<0x10000000007/client0 v41>,<0x10000000006/clients v21>,<0x10000000005/tmp v21>,<0x10000000003/client.0 v22>,<0x10000000002/e990713a-13d9-4c54-877d-f0f60b19e88e v36>,<0x10000000001/sv_0 v58>,<0x10000000000/qa v46>,<0x1/volumes v44>]//[3]","memoryvalue":"(4)0x10000000223:[<0x10000000221/RESULTS.XLS v264>,<0x10000000009/WORDPRO v2324>,<0x10000000008/~dmtmp v96>,<0x10000000007/client0 v44>,<0x10000000006/clients v24>,<0x10000000005/tmp v24>,<0x10000000003/client.0 v24>,<0x10000000002/e990713a-13d9-4c54-877d-f0f60b19e88e v39>,<0x10000000001/sv_0 v60>,<0x10000000000/qa v32>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-31T23:13:12.050 INFO:journalctl@ceph.mds.c.smithi162.stdout:Mar 31 23:13:11 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mds-c[67067]: 2024-03-31T23:13:11.668+0000 7f45de802700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000224 [4,head] /volumes/qa/sv_0/e990713a-13d9-4c54-877d-f0f60b19e88e/client.0/tmp/clients/client0/~dmtmp/WORDPRO/NEWS1_1.LWP auth v266 ap=1 s=260096 n(v0 rc2024-03-31T23:12:54.491251+0000 b260096 1=1+0) caps={15462=pAsLsXsFsc/-@1},l=15462(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x55c95bab0100]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000224:[<0x10000000221/NEWS1_1.LWP v142>,<0x10000000009/WORDPRO v1781>,<0x10000000008/~dmtmp v93>,<0x10000000007/client0 v41>,<0x10000000006/clients v21>,<0x10000000005/tmp v21>,<0x10000000003/client.0 v22>,<0x10000000002/e990713a-13d9-4c54-877d-f0f60b19e88e v36>,<0x10000000001/sv_0 v58>,<0x10000000000/qa v46>,<0x1/volumes v44>]//[3]","memoryvalue":"(4)0x10000000224:[<0x10000000221/NEWS1_1.LWP v266>,<0x10000000009/WORDPRO v2324>,<0x10000000008/~dmtmp v96>,<0x10000000007/client0 v44>,<0x10000000006/clients v24>,<0x10000000005/tmp v24>,<0x10000000003/client.0 v24>,<0x10000000002/e990713a-13d9-4c54-877d-f0f60b19e88e v39>,<0x10000000001/sv_0 v60>,<0x10000000000/qa v32>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-31T23:13:12.050 INFO:journalctl@ceph.mds.c.smithi162.stdout:Mar 31 23:13:11 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mds-c[67067]: 2024-03-31T23:13:11.668+0000 7f45de802700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000020e [4,head] /volumes/qa/sv_0/e990713a-13d9-4c54-877d-f0f60b19e88e/client.0/tmp/clients/client0/~dmtmp/EXCEL/SALES.XLS auth v278 ap=1 s=260096 n(v0 rc2024-03-31T23:13:04.002035+0000 b260096 1=1+0) caps={15462=pAsLsXsFsc/-@1},l=15462(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x55c95bab0c00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000020e:[<0x1000000020c/SALES.XLS v222>,<0x10000000009/EXCEL v1787>,<0x10000000008/~dmtmp v93>,<0x10000000007/client0 v41>,<0x10000000006/clients v21>,<0x10000000005/tmp v21>,<0x10000000003/client.0 v22>,<0x10000000002/e990713a-13d9-4c54-877d-f0f60b19e88e v36>,<0x10000000001/sv_0 v58>,<0x10000000000/qa v46>,<0x1/volumes v44>]//[3]","memoryvalue":"(4)0x1000000020e:[<0x1000000020c/SALES.XLS v278>,<0x10000000009/EXCEL v2296>,<0x10000000008/~dmtmp v96>,<0x10000000007/client0 v44>,<0x10000000006/clients v24>,<0x10000000005/tmp v24>,<0x10000000003/client.0 v24>,<0x10000000002/e990713a-13d9-4c54-877d-f0f60b19e88e v39>,<0x10000000001/sv_0 v60>,<0x10000000000/qa v32>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-31T23:13:12.050 INFO:journalctl@ceph.mds.c.smithi162.stdout:Mar 31 23:13:11 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mds-c[67067]: 2024-03-31T23:13:11.668+0000 7f45de802700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000020d [4,head] /volumes/qa/sv_0/e990713a-13d9-4c54-877d-f0f60b19e88e/client.0/tmp/clients/client0/~dmtmp/EXCEL/PCMAGCD.XLS auth v276 ap=1 s=260096 n(v0 rc2024-03-31T23:13:03.525841+0000 b260096 1=1+0) caps={15462=pAsLsXsFsc/-@1},l=15462(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x55c95bab0680]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000020d:[<0x1000000020c/PCMAGCD.XLS v152>,<0x10000000009/EXCEL v1787>,<0x10000000008/~dmtmp v93>,<0x10000000007/client0 v41>,<0x10000000006/clients v21>,<0x10000000005/tmp v21>,<0x10000000003/client.0 v22>,<0x10000000002/e990713a-13d9-4c54-877d-f0f60b19e88e v36>,<0x10000000001/sv_0 v58>,<0x10000000000/qa v46>,<0x1/volumes v44>]//[3]","memoryvalue":"(4)0x1000000020d:[<0x1000000020c/PCMAGCD.XLS v276>,<0x10000000009/EXCEL v2296>,<0x10000000008/~dmtmp v96>,<0x10000000007/client0 v44>,<0x10000000006/clients v24>,<0x10000000005/tmp v24>,<0x10000000003/client.0 v24>,<0x10000000002/e990713a-13d9-4c54-877d-f0f60b19e88e v39>,<0x10000000001/sv_0 v60>,<0x10000000000/qa v32>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-31T23:13:12.050 INFO:journalctl@ceph.mds.c.smithi162.stdout:Mar 31 23:13:11 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mds-c[67067]: 2024-03-31T23:13:11.673+0000 7f45de802700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000228 [4,head] /volumes/qa/sv_0/e990713a-13d9-4c54-877d-f0f60b19e88e/client.0/tmp/clients/client0/~dmtmp/PM/MOVED.DOC auth v350 ap=1 s=260096 n(v0 rc2024-03-31T23:12:54.517740+0000 b260096 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15462=0-4194304@3} caps={15462=pAsxLsXsxFsxcrwb/pAsxXsxFsxcrwb@1},l=15462 | importingcaps=0 caps=1 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x55c95ba57b80]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000228:[<0x10000000225/MOVED.DOC v174>,<0x10000000009/PM v1244>,<0x10000000008/~dmtmp v93>,<0x10000000007/client0 v41>,<0x10000000006/clients v21>,<0x10000000005/tmp v21>,<0x10000000003/client.0 v22>,<0x10000000002/e990713a-13d9-4c54-877d-f0f60b19e88e v36>,<0x10000000001/sv_0 v58>,<0x10000000000/qa v46>,<0x1/volumes v44>]//[3]","memoryvalue":"(4)0x10000000228:[<0x10000000225/MOVED.DOC v350>,<0x10000000009/PM v2328>,<0x10000000008/~dmtmp v96>,<0x10000000007/client0 v44>,<0x10000000006/clients v24>,<0x10000000005/tmp v24>,<0x10000000003/client.0 v24>,<0x10000000002/e990713a-13d9-4c54-877d-f0f60b19e88e v39>,<0x10000000001/sv_0 v60>,<0x10000000000/qa v32>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-31T23:13:12.050 INFO:journalctl@ceph.mds.c.smithi162.stdout:Mar 31 23:13:11 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mds-c[67067]: 2024-03-31T23:13:11.673+0000 7f45de802700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000226 [4,head] /volumes/qa/sv_0/e990713a-13d9-4c54-877d-f0f60b19e88e/client.0/tmp/clients/client0/~dmtmp/PM/T1.XLS auth v266 ap=1 s=260096 n(v0 rc2024-03-31T23:12:54.502162+0000 b260096 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15462=0-4194304@3} caps={15462=pAsxLsXsxFsxcrwb/pAsxXsxFsxcrwb@1},l=15462 | importingcaps=0 caps=1 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x55c95ba56000]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000226:[<0x10000000225/T1.XLS v88>,<0x10000000009/PM v1244>,<0x10000000008/~dmtmp v93>,<0x10000000007/client0 v41>,<0x10000000006/clients v21>,<0x10000000005/tmp v21>,<0x10000000003/client.0 v22>,<0x10000000002/e990713a-13d9-4c54-877d-f0f60b19e88e v36>,<0x10000000001/sv_0 v58>,<0x10000000000/qa v46>,<0x1/volumes v44>]//[3]","memoryvalue":"(4)0x10000000226:[<0x10000000225/T1.XLS v266>,<0x10000000009/PM v2328>,<0x10000000008/~dmtmp v96>,<0x10000000007/client0 v44>,<0x10000000006/clients v24>,<0x10000000005/tmp v24>,<0x10000000003/client.0 v24>,<0x10000000002/e990713a-13d9-4c54-877d-f0f60b19e88e v39>,<0x10000000001/sv_0 v60>,<0x10000000000/qa v32>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-31T23:13:12.050 INFO:journalctl@ceph.mds.c.smithi162.stdout:Mar 31 23:13:11 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mds-c[67067]: 2024-03-31T23:13:11.673+0000 7f45de802700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000227 [4,head] /volumes/qa/sv_0/e990713a-13d9-4c54-877d-f0f60b19e88e/client.0/tmp/clients/client0/~dmtmp/PM/EVENTS.DOC auth v344 ap=1 s=260096 n(v0 rc2024-03-31T23:12:54.510329+0000 b260096 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15462=0-4194304@3} caps={15462=pAsxLsXsxFsxcrwb/pAsxXsxFsxcrwb@1},l=15462 | importingcaps=0 caps=1 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x55c95ba56b00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000227:[<0x10000000225/EVENTS.DOC v168>,<0x10000000009/PM v1244>,<0x10000000008/~dmtmp v93>,<0x10000000007/client0 v41>,<0x10000000006/clients v21>,<0x10000000005/tmp v21>,<0x10000000003/client.0 v22>,<0x10000000002/e990713a-13d9-4c54-877d-f0f60b19e88e v36>,<0x10000000001/sv_0 v58>,<0x10000000000/qa v46>,<0x1/volumes v44>]//[3]","memoryvalue":"(4)0x10000000227:[<0x10000000225/EVENTS.DOC v344>,<0x10000000009/PM v2328>,<0x10000000008/~dmtmp v96>,<0x10000000007/client0 v44>,<0x10000000006/clients v24>,<0x10000000005/tmp v24>,<0x10000000003/client.0 v24>,<0x10000000002/e990713a-13d9-4c54-877d-f0f60b19e88e v39>,<0x10000000001/sv_0 v60>,<0x10000000000/qa v32>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-31T23:13:12.050 INFO:journalctl@ceph.mds.c.smithi162.stdout:Mar 31 23:13:11 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mds-c[67067]: 2024-03-31T23:13:11.682+0000 7f45de802700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000023f [4,head] /volumes/qa/sv_0/e990713a-13d9-4c54-877d-f0f60b19e88e/client.0/tmp/clients/client0/~dmtmp/ACCESS/FASTENER.MDB auth v88 ap=1 s=260096 n(v0 rc2024-03-31T23:13:05.277799+0000 b260096 1=1+0) caps={15462=pAsLsXsFsc/-@1},l=15462(-1) | ptrwaiter=0 lock=0 importingcaps=0 caps=1 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x55c95bab1700]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000023f:[<0x1000000023e/FASTENER.MDB v36>,<0x10000000009/ACCESS v1266>,<0x10000000008/~dmtmp v93>,<0x10000000007/client0 v41>,<0x10000000006/clients v21>,<0x10000000005/tmp v21>,<0x10000000003/client.0 v22>,<0x10000000002/e990713a-13d9-4c54-877d-f0f60b19e88e v36>,<0x10000000001/sv_0 v58>,<0x10000000000/qa v46>,<0x1/volumes v44>]//[3]","memoryvalue":"(4)0x1000000023f:[<0x1000000023e/FASTENER.MDB v88>,<0x10000000009/ACCESS v2342>,<0x10000000008/~dmtmp v96>,<0x10000000007/client0 v44>,<0x10000000006/clients v24>,<0x10000000005/tmp v24>,<0x10000000003/client.0 v24>,<0x10000000002/e990713a-13d9-4c54-877d-f0f60b19e88e v39>,<0x10000000001/sv_0 v60>,<0x10000000000/qa v32>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-31T23:13:12.190 INFO:tasks.workunit.client.0.smithi046.stdout: 1 8971 3.59 MB/sec warmup 18 sec latency 31.081 ms 2024-03-31T23:13:12.694 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:13:12 smithi142 ceph-mon[35271]: pgmap v47: 129 pgs: 129 active+clean; 30 MiB data, 457 MiB used, 1.0 TiB / 1.0 TiB avail; 301 KiB/s rd, 1.2 MiB/s wr, 118 op/s 2024-03-31T23:13:12.694 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:13:12 smithi142 ceph-mon[35271]: scrub queued for path: / 2024-03-31T23:13:12.694 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:13:12 smithi142 ceph-mon[35271]: scrub summary: idle+waiting paths [/] 2024-03-31T23:13:12.694 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:13:12 smithi142 ceph-mon[35271]: scrub summary: active paths [/] 2024-03-31T23:13:12.694 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:13:12 smithi142 ceph-mon[35271]: scrub summary: active paths [/] 2024-03-31T23:13:12.694 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:13:12 smithi142 ceph-mon[35271]: Scrub error on inode 0x1000000021f (/volumes/qa/sv_0/e990713a-13d9-4c54-877d-f0f60b19e88e/client.0/tmp/clients/client0/~dmtmp/COREL/ROSE.CPT) see mds.c log and `damage ls` output for details 2024-03-31T23:13:12.694 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:13:12 smithi142 ceph-mon[35271]: Scrub error on inode 0x1000000021b (/volumes/qa/sv_0/e990713a-13d9-4c54-877d-f0f60b19e88e/client.0/tmp/clients/client0/~dmtmp/COREL/TEAPOT2.CPT) see mds.c log and `damage ls` output for details 2024-03-31T23:13:12.694 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:13:12 smithi142 ceph-mon[35271]: Scrub error on inode 0x10000000220 (/volumes/qa/sv_0/e990713a-13d9-4c54-877d-f0f60b19e88e/client.0/tmp/clients/client0/~dmtmp/COREL/CORELDRW.CDT) see mds.c log and `damage ls` output for details 2024-03-31T23:13:12.695 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:13:12 smithi142 ceph-mon[35271]: Scrub error on inode 0x1000000021a (/volumes/qa/sv_0/e990713a-13d9-4c54-877d-f0f60b19e88e/client.0/tmp/clients/client0/~dmtmp/COREL/MAILBOX2.CPT) see mds.c log and `damage ls` output for details 2024-03-31T23:13:12.695 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:13:12 smithi142 ceph-mon[35271]: Scrub error on inode 0x1000000021e (/volumes/qa/sv_0/e990713a-13d9-4c54-877d-f0f60b19e88e/client.0/tmp/clients/client0/~dmtmp/COREL/GRAPHIC2.CDR) see mds.c log and `damage ls` output for details 2024-03-31T23:13:12.695 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:13:12 smithi142 ceph-mon[35271]: Scrub error on inode 0x10000000218 (/volumes/qa/sv_0/e990713a-13d9-4c54-877d-f0f60b19e88e/client.0/tmp/clients/client0/~dmtmp/COREL/ARTISTIC.ACL) see mds.c log and `damage ls` output for details 2024-03-31T23:13:12.695 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:13:12 smithi142 ceph-mon[35271]: Scrub error on inode 0x1000000021c (/volumes/qa/sv_0/e990713a-13d9-4c54-877d-f0f60b19e88e/client.0/tmp/clients/client0/~dmtmp/COREL/SIGN2.CPT) see mds.c log and `damage ls` output for details 2024-03-31T23:13:12.695 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:13:12 smithi142 ceph-mon[35271]: Scrub error on inode 0x1000000021d (/volumes/qa/sv_0/e990713a-13d9-4c54-877d-f0f60b19e88e/client.0/tmp/clients/client0/~dmtmp/COREL/TRAIN.CDR) see mds.c log and `damage ls` output for details 2024-03-31T23:13:12.695 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:13:12 smithi142 ceph-mon[35271]: Scrub error on inode 0x10000000219 (/volumes/qa/sv_0/e990713a-13d9-4c54-877d-f0f60b19e88e/client.0/tmp/clients/client0/~dmtmp/COREL/GRAPHIC1.CDR) see mds.c log and `damage ls` output for details 2024-03-31T23:13:12.695 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:13:12 smithi142 ceph-mon[35271]: Scrub error on inode 0x10000000217 (/volumes/qa/sv_0/e990713a-13d9-4c54-877d-f0f60b19e88e/client.0/tmp/clients/client0/~dmtmp/COREL/CDRBARS.CFG) see mds.c log and `damage ls` output for details 2024-03-31T23:13:12.695 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:13:12 smithi142 ceph-mon[35271]: Scrub error on inode 0x10000000216 (/volumes/qa/sv_0/e990713a-13d9-4c54-877d-f0f60b19e88e/client.0/tmp/clients/client0/~dmtmp/COREL/CDRROLS.CFG) see mds.c log and `damage ls` output for details 2024-03-31T23:13:12.695 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:13:12 smithi142 ceph-mon[35271]: Scrub error on inode 0x1000000020b (/volumes/qa/sv_0/e990713a-13d9-4c54-877d-f0f60b19e88e/client.0/tmp/clients/client0/~dmtmp/WORD/FACTS.DOC) see mds.c log and `damage ls` output for details 2024-03-31T23:13:12.695 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:13:12 smithi142 ceph-mon[35271]: Scrub error on inode 0x1000000020a (/volumes/qa/sv_0/e990713a-13d9-4c54-877d-f0f60b19e88e/client.0/tmp/clients/client0/~dmtmp/WORD/BASEMACH.DOC) see mds.c log and `damage ls` output for details 2024-03-31T23:13:12.695 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:13:12 smithi142 ceph-mon[35271]: Scrub error on inode 0x10000000209 (/volumes/qa/sv_0/e990713a-13d9-4c54-877d-f0f60b19e88e/client.0/tmp/clients/client0/~dmtmp/WORD/CHAP10.DOC) see mds.c log and `damage ls` output for details 2024-03-31T23:13:12.695 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:13:12 smithi142 ceph-mon[35271]: Scrub error on inode 0x10000000241 (/volumes/qa/sv_0/e990713a-13d9-4c54-877d-f0f60b19e88e/client.0/tmp/clients/client0/~dmtmp/SEED/SMALL.FIL) see mds.c log and `damage ls` output for details 2024-03-31T23:13:12.695 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:13:12 smithi142 ceph-mon[35271]: Scrub error on inode 0x10000000242 (/volumes/qa/sv_0/e990713a-13d9-4c54-877d-f0f60b19e88e/client.0/tmp/clients/client0/~dmtmp/SEED/MEDIUM.FIL) see mds.c log and `damage ls` output for details 2024-03-31T23:13:12.695 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:13:12 smithi142 ceph-mon[35271]: Scrub error on inode 0x10000000222 (/volumes/qa/sv_0/e990713a-13d9-4c54-877d-f0f60b19e88e/client.0/tmp/clients/client0/~dmtmp/WORDPRO/BENCHS.LWP) see mds.c log and `damage ls` output for details 2024-03-31T23:13:12.695 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:13:12 smithi142 ceph-mon[35271]: Scrub error on inode 0x10000000243 (/volumes/qa/sv_0/e990713a-13d9-4c54-877d-f0f60b19e88e/client.0/tmp/clients/client0/~dmtmp/SEED/LARGE.FIL) see mds.c log and `damage ls` output for details 2024-03-31T23:13:12.696 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:13:12 smithi142 ceph-mon[35271]: Scrub error on inode 0x10000000223 (/volumes/qa/sv_0/e990713a-13d9-4c54-877d-f0f60b19e88e/client.0/tmp/clients/client0/~dmtmp/WORDPRO/RESULTS.XLS) see mds.c log and `damage ls` output for details 2024-03-31T23:13:12.696 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:13:12 smithi142 ceph-mon[35271]: Scrub error on inode 0x10000000224 (/volumes/qa/sv_0/e990713a-13d9-4c54-877d-f0f60b19e88e/client.0/tmp/clients/client0/~dmtmp/WORDPRO/NEWS1_1.LWP) see mds.c log and `damage ls` output for details 2024-03-31T23:13:12.696 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:13:12 smithi142 ceph-mon[35271]: Scrub error on inode 0x1000000020e (/volumes/qa/sv_0/e990713a-13d9-4c54-877d-f0f60b19e88e/client.0/tmp/clients/client0/~dmtmp/EXCEL/SALES.XLS) see mds.c log and `damage ls` output for details 2024-03-31T23:13:12.696 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:13:12 smithi142 ceph-mon[35271]: Scrub error on inode 0x1000000020d (/volumes/qa/sv_0/e990713a-13d9-4c54-877d-f0f60b19e88e/client.0/tmp/clients/client0/~dmtmp/EXCEL/PCMAGCD.XLS) see mds.c log and `damage ls` output for details 2024-03-31T23:13:12.696 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:13:12 smithi142 ceph-mon[35271]: scrub summary: idle+waiting paths [/] 2024-03-31T23:13:12.696 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:13:12 smithi142 ceph-mon[35271]: scrub summary: active paths [/] 2024-03-31T23:13:12.696 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:13:12 smithi142 ceph-mon[35271]: Scrub error on inode 0x10000000228 (/volumes/qa/sv_0/e990713a-13d9-4c54-877d-f0f60b19e88e/client.0/tmp/clients/client0/~dmtmp/PM/MOVED.DOC) see mds.c log and `damage ls` output for details 2024-03-31T23:13:12.696 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:13:12 smithi142 ceph-mon[35271]: Scrub error on inode 0x10000000226 (/volumes/qa/sv_0/e990713a-13d9-4c54-877d-f0f60b19e88e/client.0/tmp/clients/client0/~dmtmp/PM/T1.XLS) see mds.c log and `damage ls` output for details 2024-03-31T23:13:12.696 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:13:12 smithi142 ceph-mon[35271]: Scrub error on inode 0x10000000227 (/volumes/qa/sv_0/e990713a-13d9-4c54-877d-f0f60b19e88e/client.0/tmp/clients/client0/~dmtmp/PM/EVENTS.DOC) see mds.c log and `damage ls` output for details 2024-03-31T23:13:12.696 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:13:12 smithi142 ceph-mon[35271]: scrub summary: idle+waiting paths [/] 2024-03-31T23:13:12.696 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:13:12 smithi142 ceph-mon[35271]: scrub summary: active paths [/] 2024-03-31T23:13:12.696 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:13:12 smithi142 ceph-mon[35271]: Scrub error on inode 0x1000000023f (/volumes/qa/sv_0/e990713a-13d9-4c54-877d-f0f60b19e88e/client.0/tmp/clients/client0/~dmtmp/ACCESS/FASTENER.MDB) see mds.c log and `damage ls` output for details 2024-03-31T23:13:12.696 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:13:12 smithi142 ceph-mon[35271]: scrub summary: idle+waiting paths [/] 2024-03-31T23:13:12.697 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:13:12 smithi142 ceph-mon[35271]: scrub summary: idle+waiting paths [/] 2024-03-31T23:13:12.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:13:12 smithi162 ceph-mon[36237]: pgmap v47: 129 pgs: 129 active+clean; 30 MiB data, 457 MiB used, 1.0 TiB / 1.0 TiB avail; 301 KiB/s rd, 1.2 MiB/s wr, 118 op/s 2024-03-31T23:13:12.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:13:12 smithi162 ceph-mon[36237]: scrub queued for path: / 2024-03-31T23:13:12.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:13:12 smithi162 ceph-mon[36237]: scrub summary: idle+waiting paths [/] 2024-03-31T23:13:12.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:13:12 smithi162 ceph-mon[36237]: scrub summary: active paths [/] 2024-03-31T23:13:12.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:13:12 smithi162 ceph-mon[36237]: scrub summary: active paths [/] 2024-03-31T23:13:12.797 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:13:12 smithi162 ceph-mon[36237]: Scrub error on inode 0x1000000021f (/volumes/qa/sv_0/e990713a-13d9-4c54-877d-f0f60b19e88e/client.0/tmp/clients/client0/~dmtmp/COREL/ROSE.CPT) see mds.c log and `damage ls` output for details 2024-03-31T23:13:12.797 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:13:12 smithi162 ceph-mon[36237]: Scrub error on inode 0x1000000021b (/volumes/qa/sv_0/e990713a-13d9-4c54-877d-f0f60b19e88e/client.0/tmp/clients/client0/~dmtmp/COREL/TEAPOT2.CPT) see mds.c log and `damage ls` output for details 2024-03-31T23:13:12.797 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:13:12 smithi162 ceph-mon[36237]: Scrub error on inode 0x10000000220 (/volumes/qa/sv_0/e990713a-13d9-4c54-877d-f0f60b19e88e/client.0/tmp/clients/client0/~dmtmp/COREL/CORELDRW.CDT) see mds.c log and `damage ls` output for details 2024-03-31T23:13:12.797 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:13:12 smithi162 ceph-mon[36237]: Scrub error on inode 0x1000000021a (/volumes/qa/sv_0/e990713a-13d9-4c54-877d-f0f60b19e88e/client.0/tmp/clients/client0/~dmtmp/COREL/MAILBOX2.CPT) see mds.c log and `damage ls` output for details 2024-03-31T23:13:12.797 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:13:12 smithi162 ceph-mon[36237]: Scrub error on inode 0x1000000021e (/volumes/qa/sv_0/e990713a-13d9-4c54-877d-f0f60b19e88e/client.0/tmp/clients/client0/~dmtmp/COREL/GRAPHIC2.CDR) see mds.c log and `damage ls` output for details 2024-03-31T23:13:12.797 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:13:12 smithi162 ceph-mon[36237]: Scrub error on inode 0x10000000218 (/volumes/qa/sv_0/e990713a-13d9-4c54-877d-f0f60b19e88e/client.0/tmp/clients/client0/~dmtmp/COREL/ARTISTIC.ACL) see mds.c log and `damage ls` output for details 2024-03-31T23:13:12.797 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:13:12 smithi162 ceph-mon[36237]: Scrub error on inode 0x1000000021c (/volumes/qa/sv_0/e990713a-13d9-4c54-877d-f0f60b19e88e/client.0/tmp/clients/client0/~dmtmp/COREL/SIGN2.CPT) see mds.c log and `damage ls` output for details 2024-03-31T23:13:12.797 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:13:12 smithi162 ceph-mon[36237]: Scrub error on inode 0x1000000021d (/volumes/qa/sv_0/e990713a-13d9-4c54-877d-f0f60b19e88e/client.0/tmp/clients/client0/~dmtmp/COREL/TRAIN.CDR) see mds.c log and `damage ls` output for details 2024-03-31T23:13:12.797 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:13:12 smithi162 ceph-mon[36237]: Scrub error on inode 0x10000000219 (/volumes/qa/sv_0/e990713a-13d9-4c54-877d-f0f60b19e88e/client.0/tmp/clients/client0/~dmtmp/COREL/GRAPHIC1.CDR) see mds.c log and `damage ls` output for details 2024-03-31T23:13:12.797 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:13:12 smithi162 ceph-mon[36237]: Scrub error on inode 0x10000000217 (/volumes/qa/sv_0/e990713a-13d9-4c54-877d-f0f60b19e88e/client.0/tmp/clients/client0/~dmtmp/COREL/CDRBARS.CFG) see mds.c log and `damage ls` output for details 2024-03-31T23:13:12.797 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:13:12 smithi162 ceph-mon[36237]: Scrub error on inode 0x10000000216 (/volumes/qa/sv_0/e990713a-13d9-4c54-877d-f0f60b19e88e/client.0/tmp/clients/client0/~dmtmp/COREL/CDRROLS.CFG) see mds.c log and `damage ls` output for details 2024-03-31T23:13:12.797 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:13:12 smithi162 ceph-mon[36237]: Scrub error on inode 0x1000000020b (/volumes/qa/sv_0/e990713a-13d9-4c54-877d-f0f60b19e88e/client.0/tmp/clients/client0/~dmtmp/WORD/FACTS.DOC) see mds.c log and `damage ls` output for details 2024-03-31T23:13:12.797 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:13:12 smithi162 ceph-mon[36237]: Scrub error on inode 0x1000000020a (/volumes/qa/sv_0/e990713a-13d9-4c54-877d-f0f60b19e88e/client.0/tmp/clients/client0/~dmtmp/WORD/BASEMACH.DOC) see mds.c log and `damage ls` output for details 2024-03-31T23:13:12.798 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:13:12 smithi162 ceph-mon[36237]: Scrub error on inode 0x10000000209 (/volumes/qa/sv_0/e990713a-13d9-4c54-877d-f0f60b19e88e/client.0/tmp/clients/client0/~dmtmp/WORD/CHAP10.DOC) see mds.c log and `damage ls` output for details 2024-03-31T23:13:12.798 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:13:12 smithi162 ceph-mon[36237]: Scrub error on inode 0x10000000241 (/volumes/qa/sv_0/e990713a-13d9-4c54-877d-f0f60b19e88e/client.0/tmp/clients/client0/~dmtmp/SEED/SMALL.FIL) see mds.c log and `damage ls` output for details 2024-03-31T23:13:12.798 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:13:12 smithi162 ceph-mon[36237]: Scrub error on inode 0x10000000242 (/volumes/qa/sv_0/e990713a-13d9-4c54-877d-f0f60b19e88e/client.0/tmp/clients/client0/~dmtmp/SEED/MEDIUM.FIL) see mds.c log and `damage ls` output for details 2024-03-31T23:13:12.798 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:13:12 smithi162 ceph-mon[36237]: Scrub error on inode 0x10000000222 (/volumes/qa/sv_0/e990713a-13d9-4c54-877d-f0f60b19e88e/client.0/tmp/clients/client0/~dmtmp/WORDPRO/BENCHS.LWP) see mds.c log and `damage ls` output for details 2024-03-31T23:13:12.798 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:13:12 smithi162 ceph-mon[36237]: Scrub error on inode 0x10000000243 (/volumes/qa/sv_0/e990713a-13d9-4c54-877d-f0f60b19e88e/client.0/tmp/clients/client0/~dmtmp/SEED/LARGE.FIL) see mds.c log and `damage ls` output for details 2024-03-31T23:13:12.798 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:13:12 smithi162 ceph-mon[36237]: Scrub error on inode 0x10000000223 (/volumes/qa/sv_0/e990713a-13d9-4c54-877d-f0f60b19e88e/client.0/tmp/clients/client0/~dmtmp/WORDPRO/RESULTS.XLS) see mds.c log and `damage ls` output for details 2024-03-31T23:13:12.798 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:13:12 smithi162 ceph-mon[36237]: Scrub error on inode 0x10000000224 (/volumes/qa/sv_0/e990713a-13d9-4c54-877d-f0f60b19e88e/client.0/tmp/clients/client0/~dmtmp/WORDPRO/NEWS1_1.LWP) see mds.c log and `damage ls` output for details 2024-03-31T23:13:12.798 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:13:12 smithi162 ceph-mon[36237]: Scrub error on inode 0x1000000020e (/volumes/qa/sv_0/e990713a-13d9-4c54-877d-f0f60b19e88e/client.0/tmp/clients/client0/~dmtmp/EXCEL/SALES.XLS) see mds.c log and `damage ls` output for details 2024-03-31T23:13:12.798 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:13:12 smithi162 ceph-mon[36237]: Scrub error on inode 0x1000000020d (/volumes/qa/sv_0/e990713a-13d9-4c54-877d-f0f60b19e88e/client.0/tmp/clients/client0/~dmtmp/EXCEL/PCMAGCD.XLS) see mds.c log and `damage ls` output for details 2024-03-31T23:13:12.798 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:13:12 smithi162 ceph-mon[36237]: scrub summary: idle+waiting paths [/] 2024-03-31T23:13:12.798 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:13:12 smithi162 ceph-mon[36237]: scrub summary: active paths [/] 2024-03-31T23:13:12.798 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:13:12 smithi162 ceph-mon[36237]: Scrub error on inode 0x10000000228 (/volumes/qa/sv_0/e990713a-13d9-4c54-877d-f0f60b19e88e/client.0/tmp/clients/client0/~dmtmp/PM/MOVED.DOC) see mds.c log and `damage ls` output for details 2024-03-31T23:13:12.798 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:13:12 smithi162 ceph-mon[36237]: Scrub error on inode 0x10000000226 (/volumes/qa/sv_0/e990713a-13d9-4c54-877d-f0f60b19e88e/client.0/tmp/clients/client0/~dmtmp/PM/T1.XLS) see mds.c log and `damage ls` output for details 2024-03-31T23:13:12.799 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:13:12 smithi162 ceph-mon[36237]: Scrub error on inode 0x10000000227 (/volumes/qa/sv_0/e990713a-13d9-4c54-877d-f0f60b19e88e/client.0/tmp/clients/client0/~dmtmp/PM/EVENTS.DOC) see mds.c log and `damage ls` output for details 2024-03-31T23:13:12.799 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:13:12 smithi162 ceph-mon[36237]: scrub summary: idle+waiting paths [/] 2024-03-31T23:13:12.799 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:13:12 smithi162 ceph-mon[36237]: scrub summary: active paths [/] 2024-03-31T23:13:12.799 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:13:12 smithi162 ceph-mon[36237]: Scrub error on inode 0x1000000023f (/volumes/qa/sv_0/e990713a-13d9-4c54-877d-f0f60b19e88e/client.0/tmp/clients/client0/~dmtmp/ACCESS/FASTENER.MDB) see mds.c log and `damage ls` output for details 2024-03-31T23:13:12.799 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:13:12 smithi162 ceph-mon[36237]: scrub summary: idle+waiting paths [/] 2024-03-31T23:13:12.799 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:13:12 smithi162 ceph-mon[36237]: scrub summary: idle+waiting paths [/] 2024-03-31T23:13:12.862 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:13:12 smithi046 ceph-mon[32498]: pgmap v47: 129 pgs: 129 active+clean; 30 MiB data, 457 MiB used, 1.0 TiB / 1.0 TiB avail; 301 KiB/s rd, 1.2 MiB/s wr, 118 op/s 2024-03-31T23:13:12.862 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:13:12 smithi046 ceph-mon[32498]: scrub queued for path: / 2024-03-31T23:13:12.862 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:13:12 smithi046 ceph-mon[32498]: scrub summary: idle+waiting paths [/] 2024-03-31T23:13:12.862 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:13:12 smithi046 ceph-mon[32498]: scrub summary: active paths [/] 2024-03-31T23:13:12.862 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:13:12 smithi046 ceph-mon[32498]: scrub summary: active paths [/] 2024-03-31T23:13:12.862 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:13:12 smithi046 ceph-mon[32498]: Scrub error on inode 0x1000000021f (/volumes/qa/sv_0/e990713a-13d9-4c54-877d-f0f60b19e88e/client.0/tmp/clients/client0/~dmtmp/COREL/ROSE.CPT) see mds.c log and `damage ls` output for details 2024-03-31T23:13:12.862 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:13:12 smithi046 ceph-mon[32498]: Scrub error on inode 0x1000000021b (/volumes/qa/sv_0/e990713a-13d9-4c54-877d-f0f60b19e88e/client.0/tmp/clients/client0/~dmtmp/COREL/TEAPOT2.CPT) see mds.c log and `damage ls` output for details 2024-03-31T23:13:12.862 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:13:12 smithi046 ceph-mon[32498]: Scrub error on inode 0x10000000220 (/volumes/qa/sv_0/e990713a-13d9-4c54-877d-f0f60b19e88e/client.0/tmp/clients/client0/~dmtmp/COREL/CORELDRW.CDT) see mds.c log and `damage ls` output for details 2024-03-31T23:13:12.862 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:13:12 smithi046 ceph-mon[32498]: Scrub error on inode 0x1000000021a (/volumes/qa/sv_0/e990713a-13d9-4c54-877d-f0f60b19e88e/client.0/tmp/clients/client0/~dmtmp/COREL/MAILBOX2.CPT) see mds.c log and `damage ls` output for details 2024-03-31T23:13:12.862 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:13:12 smithi046 ceph-mon[32498]: Scrub error on inode 0x1000000021e (/volumes/qa/sv_0/e990713a-13d9-4c54-877d-f0f60b19e88e/client.0/tmp/clients/client0/~dmtmp/COREL/GRAPHIC2.CDR) see mds.c log and `damage ls` output for details 2024-03-31T23:13:12.863 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:13:12 smithi046 ceph-mon[32498]: Scrub error on inode 0x10000000218 (/volumes/qa/sv_0/e990713a-13d9-4c54-877d-f0f60b19e88e/client.0/tmp/clients/client0/~dmtmp/COREL/ARTISTIC.ACL) see mds.c log and `damage ls` output for details 2024-03-31T23:13:12.863 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:13:12 smithi046 ceph-mon[32498]: Scrub error on inode 0x1000000021c (/volumes/qa/sv_0/e990713a-13d9-4c54-877d-f0f60b19e88e/client.0/tmp/clients/client0/~dmtmp/COREL/SIGN2.CPT) see mds.c log and `damage ls` output for details 2024-03-31T23:13:12.863 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:13:12 smithi046 ceph-mon[32498]: Scrub error on inode 0x1000000021d (/volumes/qa/sv_0/e990713a-13d9-4c54-877d-f0f60b19e88e/client.0/tmp/clients/client0/~dmtmp/COREL/TRAIN.CDR) see mds.c log and `damage ls` output for details 2024-03-31T23:13:12.863 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:13:12 smithi046 ceph-mon[32498]: Scrub error on inode 0x10000000219 (/volumes/qa/sv_0/e990713a-13d9-4c54-877d-f0f60b19e88e/client.0/tmp/clients/client0/~dmtmp/COREL/GRAPHIC1.CDR) see mds.c log and `damage ls` output for details 2024-03-31T23:13:12.863 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:13:12 smithi046 ceph-mon[32498]: Scrub error on inode 0x10000000217 (/volumes/qa/sv_0/e990713a-13d9-4c54-877d-f0f60b19e88e/client.0/tmp/clients/client0/~dmtmp/COREL/CDRBARS.CFG) see mds.c log and `damage ls` output for details 2024-03-31T23:13:12.863 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:13:12 smithi046 ceph-mon[32498]: Scrub error on inode 0x10000000216 (/volumes/qa/sv_0/e990713a-13d9-4c54-877d-f0f60b19e88e/client.0/tmp/clients/client0/~dmtmp/COREL/CDRROLS.CFG) see mds.c log and `damage ls` output for details 2024-03-31T23:13:12.863 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:13:12 smithi046 ceph-mon[32498]: Scrub error on inode 0x1000000020b (/volumes/qa/sv_0/e990713a-13d9-4c54-877d-f0f60b19e88e/client.0/tmp/clients/client0/~dmtmp/WORD/FACTS.DOC) see mds.c log and `damage ls` output for details 2024-03-31T23:13:12.863 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:13:12 smithi046 ceph-mon[32498]: Scrub error on inode 0x1000000020a (/volumes/qa/sv_0/e990713a-13d9-4c54-877d-f0f60b19e88e/client.0/tmp/clients/client0/~dmtmp/WORD/BASEMACH.DOC) see mds.c log and `damage ls` output for details 2024-03-31T23:13:12.863 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:13:12 smithi046 ceph-mon[32498]: Scrub error on inode 0x10000000209 (/volumes/qa/sv_0/e990713a-13d9-4c54-877d-f0f60b19e88e/client.0/tmp/clients/client0/~dmtmp/WORD/CHAP10.DOC) see mds.c log and `damage ls` output for details 2024-03-31T23:13:12.863 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:13:12 smithi046 ceph-mon[32498]: Scrub error on inode 0x10000000241 (/volumes/qa/sv_0/e990713a-13d9-4c54-877d-f0f60b19e88e/client.0/tmp/clients/client0/~dmtmp/SEED/SMALL.FIL) see mds.c log and `damage ls` output for details 2024-03-31T23:13:12.863 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:13:12 smithi046 ceph-mon[32498]: Scrub error on inode 0x10000000242 (/volumes/qa/sv_0/e990713a-13d9-4c54-877d-f0f60b19e88e/client.0/tmp/clients/client0/~dmtmp/SEED/MEDIUM.FIL) see mds.c log and `damage ls` output for details 2024-03-31T23:13:12.863 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:13:12 smithi046 ceph-mon[32498]: Scrub error on inode 0x10000000222 (/volumes/qa/sv_0/e990713a-13d9-4c54-877d-f0f60b19e88e/client.0/tmp/clients/client0/~dmtmp/WORDPRO/BENCHS.LWP) see mds.c log and `damage ls` output for details 2024-03-31T23:13:12.864 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:13:12 smithi046 ceph-mon[32498]: Scrub error on inode 0x10000000243 (/volumes/qa/sv_0/e990713a-13d9-4c54-877d-f0f60b19e88e/client.0/tmp/clients/client0/~dmtmp/SEED/LARGE.FIL) see mds.c log and `damage ls` output for details 2024-03-31T23:13:12.864 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:13:12 smithi046 ceph-mon[32498]: Scrub error on inode 0x10000000223 (/volumes/qa/sv_0/e990713a-13d9-4c54-877d-f0f60b19e88e/client.0/tmp/clients/client0/~dmtmp/WORDPRO/RESULTS.XLS) see mds.c log and `damage ls` output for details 2024-03-31T23:13:12.864 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:13:12 smithi046 ceph-mon[32498]: Scrub error on inode 0x10000000224 (/volumes/qa/sv_0/e990713a-13d9-4c54-877d-f0f60b19e88e/client.0/tmp/clients/client0/~dmtmp/WORDPRO/NEWS1_1.LWP) see mds.c log and `damage ls` output for details 2024-03-31T23:13:12.864 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:13:12 smithi046 ceph-mon[32498]: Scrub error on inode 0x1000000020e (/volumes/qa/sv_0/e990713a-13d9-4c54-877d-f0f60b19e88e/client.0/tmp/clients/client0/~dmtmp/EXCEL/SALES.XLS) see mds.c log and `damage ls` output for details 2024-03-31T23:13:12.864 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:13:12 smithi046 ceph-mon[32498]: Scrub error on inode 0x1000000020d (/volumes/qa/sv_0/e990713a-13d9-4c54-877d-f0f60b19e88e/client.0/tmp/clients/client0/~dmtmp/EXCEL/PCMAGCD.XLS) see mds.c log and `damage ls` output for details 2024-03-31T23:13:12.864 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:13:12 smithi046 ceph-mon[32498]: scrub summary: idle+waiting paths [/] 2024-03-31T23:13:12.864 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:13:12 smithi046 ceph-mon[32498]: scrub summary: active paths [/] 2024-03-31T23:13:12.864 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:13:12 smithi046 ceph-mon[32498]: Scrub error on inode 0x10000000228 (/volumes/qa/sv_0/e990713a-13d9-4c54-877d-f0f60b19e88e/client.0/tmp/clients/client0/~dmtmp/PM/MOVED.DOC) see mds.c log and `damage ls` output for details 2024-03-31T23:13:12.864 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:13:12 smithi046 ceph-mon[32498]: Scrub error on inode 0x10000000226 (/volumes/qa/sv_0/e990713a-13d9-4c54-877d-f0f60b19e88e/client.0/tmp/clients/client0/~dmtmp/PM/T1.XLS) see mds.c log and `damage ls` output for details 2024-03-31T23:13:12.864 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:13:12 smithi046 ceph-mon[32498]: Scrub error on inode 0x10000000227 (/volumes/qa/sv_0/e990713a-13d9-4c54-877d-f0f60b19e88e/client.0/tmp/clients/client0/~dmtmp/PM/EVENTS.DOC) see mds.c log and `damage ls` output for details 2024-03-31T23:13:12.864 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:13:12 smithi046 ceph-mon[32498]: scrub summary: idle+waiting paths [/] 2024-03-31T23:13:12.864 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:13:12 smithi046 ceph-mon[32498]: scrub summary: active paths [/] 2024-03-31T23:13:12.865 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:13:12 smithi046 ceph-mon[32498]: Scrub error on inode 0x1000000023f (/volumes/qa/sv_0/e990713a-13d9-4c54-877d-f0f60b19e88e/client.0/tmp/clients/client0/~dmtmp/ACCESS/FASTENER.MDB) see mds.c log and `damage ls` output for details 2024-03-31T23:13:12.865 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:13:12 smithi046 ceph-mon[32498]: scrub summary: idle+waiting paths [/] 2024-03-31T23:13:12.865 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:13:12 smithi046 ceph-mon[32498]: scrub summary: idle+waiting paths [/] 2024-03-31T23:13:13.191 INFO:tasks.workunit.client.0.smithi046.stdout: 1 9540 3.49 MB/sec warmup 19 sec latency 18.288 ms 2024-03-31T23:13:13.693 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:13:13 smithi142 ceph-mon[35271]: scrub summary: active paths [/] 2024-03-31T23:13:13.693 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:13:13 smithi142 ceph-mon[35271]: scrub summary: idle+waiting paths [/] 2024-03-31T23:13:13.795 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:13:13 smithi162 ceph-mon[36237]: scrub summary: active paths [/] 2024-03-31T23:13:13.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:13:13 smithi162 ceph-mon[36237]: scrub summary: idle+waiting paths [/] 2024-03-31T23:13:13.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:13:13 smithi046 ceph-mon[32498]: scrub summary: active paths [/] 2024-03-31T23:13:13.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:13:13 smithi046 ceph-mon[32498]: scrub summary: idle+waiting paths [/] 2024-03-31T23:13:14.190 INFO:tasks.workunit.client.0.smithi046.stdout: 1 10275 3.62 MB/sec warmup 20 sec latency 16.391 ms 2024-03-31T23:13:14.693 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:13:14 smithi142 ceph-mon[35271]: pgmap v48: 129 pgs: 129 active+clean; 30 MiB data, 457 MiB used, 1.0 TiB / 1.0 TiB avail; 301 KiB/s rd, 1.2 MiB/s wr, 118 op/s 2024-03-31T23:13:14.795 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:13:14 smithi162 ceph-mon[36237]: pgmap v48: 129 pgs: 129 active+clean; 30 MiB data, 457 MiB used, 1.0 TiB / 1.0 TiB avail; 301 KiB/s rd, 1.2 MiB/s wr, 118 op/s 2024-03-31T23:13:14.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:13:14 smithi046 ceph-mon[32498]: pgmap v48: 129 pgs: 129 active+clean; 30 MiB data, 457 MiB used, 1.0 TiB / 1.0 TiB avail; 301 KiB/s rd, 1.2 MiB/s wr, 118 op/s 2024-03-31T23:13:15.190 INFO:tasks.workunit.client.0.smithi046.stdout: 1 10653 3.65 MB/sec warmup 21 sec latency 17.113 ms 2024-03-31T23:13:15.693 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:13:15 smithi142 ceph-mon[35271]: Health check failed: 1 MDSs report damaged metadata (MDS_DAMAGE) 2024-03-31T23:13:15.693 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:13:15 smithi142 ceph-mon[35271]: mds.? [v2:172.21.15.162:6836/1371886876,v1:172.21.15.162:6837/1371886876] up:active 2024-03-31T23:13:15.693 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:13:15 smithi142 ceph-mon[35271]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-03-31T23:13:15.693 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:13:15 smithi142 ceph-mon[35271]: mgrmap e25: x(active, since 92s), standbys: z, y 2024-03-31T23:13:15.795 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:13:15 smithi162 ceph-mon[36237]: Health check failed: 1 MDSs report damaged metadata (MDS_DAMAGE) 2024-03-31T23:13:15.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:13:15 smithi162 ceph-mon[36237]: mds.? [v2:172.21.15.162:6836/1371886876,v1:172.21.15.162:6837/1371886876] up:active 2024-03-31T23:13:15.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:13:15 smithi162 ceph-mon[36237]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-03-31T23:13:15.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:13:15 smithi162 ceph-mon[36237]: mgrmap e25: x(active, since 92s), standbys: z, y 2024-03-31T23:13:15.862 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:13:15 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mon-a[32494]: 2024-03-31T23:13:15.387+0000 7fdb0b79b700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 MDSs report damaged metadata (MDS_DAMAGE) 2024-03-31T23:13:15.862 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:13:15 smithi046 ceph-mon[32498]: Health check failed: 1 MDSs report damaged metadata (MDS_DAMAGE) 2024-03-31T23:13:15.862 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:13:15 smithi046 ceph-mon[32498]: mds.? [v2:172.21.15.162:6836/1371886876,v1:172.21.15.162:6837/1371886876] up:active 2024-03-31T23:13:15.862 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:13:15 smithi046 ceph-mon[32498]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-03-31T23:13:15.862 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:13:15 smithi046 ceph-mon[32498]: mgrmap e25: x(active, since 92s), standbys: z, y 2024-03-31T23:13:16.190 INFO:tasks.workunit.client.0.smithi046.stdout: 1 11010 3.56 MB/sec warmup 22 sec latency 18.532 ms 2024-03-31T23:13:16.693 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:13:16 smithi142 ceph-mon[35271]: pgmap v49: 129 pgs: 129 active+clean; 31 MiB data, 472 MiB used, 1.0 TiB / 1.0 TiB avail; 372 KiB/s rd, 1.5 MiB/s wr, 161 op/s 2024-03-31T23:13:16.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:13:16 smithi162 ceph-mon[36237]: pgmap v49: 129 pgs: 129 active+clean; 31 MiB data, 472 MiB used, 1.0 TiB / 1.0 TiB avail; 372 KiB/s rd, 1.5 MiB/s wr, 161 op/s 2024-03-31T23:13:16.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:13:16 smithi046 ceph-mon[32498]: pgmap v49: 129 pgs: 129 active+clean; 31 MiB data, 472 MiB used, 1.0 TiB / 1.0 TiB avail; 372 KiB/s rd, 1.5 MiB/s wr, 161 op/s 2024-03-31T23:13:17.190 INFO:tasks.workunit.client.0.smithi046.stdout: 1 11275 3.41 MB/sec warmup 23 sec latency 14.827 ms 2024-03-31T23:13:17.795 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:13:17 smithi162 ceph-mon[36237]: scrub summary: idle 2024-03-31T23:13:17.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:13:17 smithi046 ceph-mon[32498]: scrub summary: idle 2024-03-31T23:13:17.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:13:17 smithi142 ceph-mon[35271]: scrub summary: idle 2024-03-31T23:13:18.190 INFO:tasks.workunit.client.0.smithi046.stdout: 1 11582 3.30 MB/sec warmup 24 sec latency 434.184 ms 2024-03-31T23:13:18.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:13:18 smithi162 ceph-mon[36237]: pgmap v50: 129 pgs: 129 active+clean; 31 MiB data, 472 MiB used, 1.0 TiB / 1.0 TiB avail; 226 KiB/s rd, 927 KiB/s wr, 107 op/s 2024-03-31T23:13:18.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:13:18 smithi162 ceph-mon[36237]: scrub summary: idle 2024-03-31T23:13:18.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:13:18 smithi162 ceph-mon[36237]: scrub summary: idle 2024-03-31T23:13:18.844 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:13:18 smithi142 ceph-mon[35271]: pgmap v50: 129 pgs: 129 active+clean; 31 MiB data, 472 MiB used, 1.0 TiB / 1.0 TiB avail; 226 KiB/s rd, 927 KiB/s wr, 107 op/s 2024-03-31T23:13:18.845 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:13:18 smithi142 ceph-mon[35271]: scrub summary: idle 2024-03-31T23:13:18.845 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:13:18 smithi142 ceph-mon[35271]: scrub summary: idle 2024-03-31T23:13:18.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:13:18 smithi046 ceph-mon[32498]: pgmap v50: 129 pgs: 129 active+clean; 31 MiB data, 472 MiB used, 1.0 TiB / 1.0 TiB avail; 226 KiB/s rd, 927 KiB/s wr, 107 op/s 2024-03-31T23:13:18.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:13:18 smithi046 ceph-mon[32498]: scrub summary: idle 2024-03-31T23:13:18.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:13:18 smithi046 ceph-mon[32498]: scrub summary: idle 2024-03-31T23:13:18.994 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:18.990+0000 7fe74f63b640 1 -- 172.21.15.46:0/2235065838 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe7481055f0 msgr2=0x7fe7481059d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:13:18.994 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:18.990+0000 7fe74f63b640 1 --2- 172.21.15.46:0/2235065838 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe7481055f0 0x7fe7481059d0 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7fe744009900 tx=0x7fe74402bc30 comp rx=0 tx=0).stop 2024-03-31T23:13:18.994 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:18.991+0000 7fe74f63b640 1 -- 172.21.15.46:0/2235065838 shutdown_connections 2024-03-31T23:13:18.994 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:18.991+0000 7fe74f63b640 1 --2- 172.21.15.46:0/2235065838 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe748101260 0x7fe74810d480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:18.994 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:18.991+0000 7fe74f63b640 1 --2- 172.21.15.46:0/2235065838 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe7481008a0 0x7fe748100d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:18.994 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:18.991+0000 7fe74f63b640 1 --2- 172.21.15.46:0/2235065838 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe7481055f0 0x7fe7481059d0 unknown :-1 s=CLOSED pgs=610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:18.994 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:18.991+0000 7fe74f63b640 1 -- 172.21.15.46:0/2235065838 >> 172.21.15.46:0/2235065838 conn(0x7fe7480774a0 msgr2=0x7fe7480778b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:13:18.995 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:18.991+0000 7fe74f63b640 1 -- 172.21.15.46:0/2235065838 shutdown_connections 2024-03-31T23:13:18.995 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:18.991+0000 7fe74f63b640 1 -- 172.21.15.46:0/2235065838 wait complete. 2024-03-31T23:13:18.995 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:18.991+0000 7fe74f63b640 1 Processor -- start 2024-03-31T23:13:18.995 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:18.991+0000 7fe74f63b640 1 -- start start 2024-03-31T23:13:18.995 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:18.991+0000 7fe74f63b640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe7481008a0 0x7fe748197f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:13:18.995 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:18.992+0000 7fe74f63b640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe748101260 0x7fe748198480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:13:18.995 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:18.992+0000 7fe74f63b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe7481055f0 0x7fe74819c800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:13:18.995 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:18.992+0000 7fe74f63b640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fe748111ae0 con 0x7fe7481055f0 2024-03-31T23:13:18.995 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:18.992+0000 7fe74f63b640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fe748111960 con 0x7fe7481008a0 2024-03-31T23:13:18.995 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:18.992+0000 7fe74f63b640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fe748111c60 con 0x7fe748101260 2024-03-31T23:13:18.995 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:18.992+0000 7fe74dbd8640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe7481055f0 0x7fe74819c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:13:18.995 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:18.992+0000 7fe74d3d7640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe7481008a0 0x7fe748197f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:13:18.996 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:18.992+0000 7fe74dbd8640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe7481055f0 0x7fe74819c800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:33772/0 (socket says 172.21.15.46:33772) 2024-03-31T23:13:18.996 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:18.992+0000 7fe74dbd8640 1 -- 172.21.15.46:0/2347304256 learned_addr learned my addr 172.21.15.46:0/2347304256 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:13:18.996 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:18.992+0000 7fe74cbd6640 1 --2- 172.21.15.46:0/2347304256 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe748101260 0x7fe748198480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:13:18.996 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:18.992+0000 7fe74dbd8640 1 -- 172.21.15.46:0/2347304256 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe748101260 msgr2=0x7fe748198480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:13:18.996 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:18.992+0000 7fe74dbd8640 1 --2- 172.21.15.46:0/2347304256 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe748101260 0x7fe748198480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:18.996 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:18.992+0000 7fe74dbd8640 1 -- 172.21.15.46:0/2347304256 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe7481008a0 msgr2=0x7fe748197f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:13:18.996 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:18.992+0000 7fe74dbd8640 1 --2- 172.21.15.46:0/2347304256 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe7481008a0 0x7fe748197f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:18.996 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:18.992+0000 7fe74dbd8640 1 -- 172.21.15.46:0/2347304256 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe74819cf30 con 0x7fe7481055f0 2024-03-31T23:13:18.996 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:18.992+0000 7fe74d3d7640 1 --2- 172.21.15.46:0/2347304256 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe7481008a0 0x7fe748197f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:13:18.996 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:18.992+0000 7fe74dbd8640 1 --2- 172.21.15.46:0/2347304256 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe7481055f0 0x7fe74819c800 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7fe74000ea60 tx=0x7fe74000ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:13:18.996 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:18.992+0000 7fe73e7fc640 1 -- 172.21.15.46:0/2347304256 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe74000ce40 con 0x7fe7481055f0 2024-03-31T23:13:18.996 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:18.992+0000 7fe73e7fc640 1 -- 172.21.15.46:0/2347304256 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe740005020 con 0x7fe7481055f0 2024-03-31T23:13:18.997 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:18.992+0000 7fe74f63b640 1 -- 172.21.15.46:0/2347304256 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe74819d220 con 0x7fe7481055f0 2024-03-31T23:13:18.997 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:18.992+0000 7fe73e7fc640 1 -- 172.21.15.46:0/2347304256 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe740010670 con 0x7fe7481055f0 2024-03-31T23:13:18.997 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:18.992+0000 7fe74f63b640 1 -- 172.21.15.46:0/2347304256 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe7481a4ab0 con 0x7fe7481055f0 2024-03-31T23:13:19.004 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:18.995+0000 7fe73e7fc640 1 -- 172.21.15.46:0/2347304256 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 25) ==== 137451+0+0 (secure 0 0 0) 0x7fe7400051d0 con 0x7fe7481055f0 2024-03-31T23:13:19.004 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:18.995+0000 7fe752216740 1 Processor -- start 2024-03-31T23:13:19.004 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:18.995+0000 7fe752216740 1 -- start start 2024-03-31T23:13:19.004 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:18.995+0000 7fe752216740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56047efef6c0 0x56047efefaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:13:19.004 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:18.995+0000 7fe752216740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56047efeffe0 0x56047eff8870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:13:19.004 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:18.995+0000 7fe752216740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56047eff8db0 0x56047effb1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:13:19.004 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:18.995+0000 7fe752216740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x56047ef528f0 con 0x56047eff8db0 2024-03-31T23:13:19.004 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:18.995+0000 7fe752216740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x56047eefb8d0 con 0x56047efef6c0 2024-03-31T23:13:19.004 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:18.995+0000 7fe752216740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x56047ef4e990 con 0x56047efeffe0 2024-03-31T23:13:19.004 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:18.996+0000 7fe73e7fc640 1 --2- 172.21.15.46:0/2347304256 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fe7340a26b0 0x7fe7340a4b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:13:19.005 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:18.996+0000 7fe74cbd6640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56047eff8db0 0x56047effb1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:13:19.005 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:18.996+0000 7fe74cbd6640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56047eff8db0 0x56047effb1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:33774/0 (socket says 172.21.15.46:33774) 2024-03-31T23:13:19.005 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:18.996+0000 7fe74cbd6640 1 -- 172.21.15.46:0/1351947696 learned_addr learned my addr 172.21.15.46:0/1351947696 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:13:19.005 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:18.996+0000 7fe74d3d7640 1 --2- 172.21.15.46:0/1351947696 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56047efeffe0 0x56047eff8870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:13:19.005 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:18.996+0000 7fe74cbd6640 1 --2- 172.21.15.46:0/1351947696 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56047efef6c0 0x56047efefaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:13:19.005 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:18.996+0000 7fe74dbd8640 1 --2- 172.21.15.46:0/2347304256 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fe7340a26b0 0x7fe7340a4b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:13:19.005 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:18.996+0000 7fe73e7fc640 1 -- 172.21.15.46:0/2347304256 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7fe7400cf310 con 0x7fe7481055f0 2024-03-31T23:13:19.005 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:18.996+0000 7fe74cbd6640 1 -- 172.21.15.46:0/1351947696 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56047efeffe0 msgr2=0x56047eff8870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:13:19.005 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:18.996+0000 7fe74cbd6640 1 --2- 172.21.15.46:0/1351947696 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56047efeffe0 0x56047eff8870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:19.005 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:18.996+0000 7fe74cbd6640 1 -- 172.21.15.46:0/1351947696 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56047efef6c0 msgr2=0x56047efefaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:13:19.005 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:18.996+0000 7fe74cbd6640 1 --2- 172.21.15.46:0/1351947696 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56047efef6c0 0x56047efefaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:19.005 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:18.996+0000 7fe74cbd6640 1 -- 172.21.15.46:0/1351947696 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56047ec67210 con 0x56047eff8db0 2024-03-31T23:13:19.005 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:18.996+0000 7fe74d3d7640 1 --2- 172.21.15.46:0/1351947696 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56047efeffe0 0x56047eff8870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:13:19.005 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:18.996+0000 7fe74dbd8640 1 --2- 172.21.15.46:0/2347304256 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fe7340a26b0 0x7fe7340a4b70 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fe74009b670 tx=0x7fe74006a610 comp rx=0 tx=0).ready entity=mgr.15654 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:13:19.005 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:18.996+0000 7fe74cbd6640 1 --2- 172.21.15.46:0/1351947696 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56047eff8db0 0x56047effb1a0 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7fe7380058c0 tx=0x7fe738005850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=51c741f89a5f680 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:13:19.006 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:18.996+0000 7fe7337fe640 1 -- 172.21.15.46:0/1351947696 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe738004030 con 0x56047eff8db0 2024-03-31T23:13:19.006 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:18.997+0000 7fe7337fe640 1 -- 172.21.15.46:0/1351947696 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe73800f7c0 con 0x56047eff8db0 2024-03-31T23:13:19.006 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:18.997+0000 7fe7337fe640 1 -- 172.21.15.46:0/1351947696 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe738003970 con 0x56047eff8db0 2024-03-31T23:13:19.006 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:18.997+0000 7fe752216740 1 -- 172.21.15.46:0/1351947696 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56047eff8db0 msgr2=0x56047effb1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:13:19.006 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:18.997+0000 7fe752216740 1 --2- 172.21.15.46:0/1351947696 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56047eff8db0 0x56047effb1a0 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7fe7380058c0 tx=0x7fe738005850 comp rx=0 tx=0).stop 2024-03-31T23:13:19.006 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:18.997+0000 7fe752216740 1 -- 172.21.15.46:0/1351947696 shutdown_connections 2024-03-31T23:13:19.006 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:18.997+0000 7fe752216740 1 --2- 172.21.15.46:0/1351947696 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56047eff8db0 0x56047effb1a0 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:19.006 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:18.997+0000 7fe752216740 1 --2- 172.21.15.46:0/1351947696 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56047efeffe0 0x56047eff8870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:19.006 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:18.997+0000 7fe752216740 1 --2- 172.21.15.46:0/1351947696 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56047efef6c0 0x56047efefaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:19.006 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:18.997+0000 7fe752216740 1 -- 172.21.15.46:0/1351947696 >> 172.21.15.46:0/1351947696 conn(0x56047efe9fe0 msgr2=0x56047efea3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:13:19.006 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:18.997+0000 7fe752216740 1 -- 172.21.15.46:0/1351947696 shutdown_connections 2024-03-31T23:13:19.006 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:18.997+0000 7fe752216740 1 -- 172.21.15.46:0/1351947696 wait complete. 2024-03-31T23:13:19.007 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:18.998+0000 7fe752216740 1 -- start start 2024-03-31T23:13:19.007 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:18.998+0000 7fe752216740 1 Processor -- start 2024-03-31T23:13:19.007 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:18.998+0000 7fe752216740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56047efef6c0 0x56047eff5df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:13:19.007 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:18.998+0000 7fe752216740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56047efeffe0 0x56047eff6330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:13:19.007 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:18.998+0000 7fe752216740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56047eff8db0 0x56047eff6870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:13:19.007 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:18.998+0000 7fe752216740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x56047effb9e0 con 0x56047eff8db0 2024-03-31T23:13:19.007 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:18.998+0000 7fe752216740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x56047eef0f80 con 0x56047efeffe0 2024-03-31T23:13:19.007 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:18.998+0000 7fe752216740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x56047effbb60 con 0x56047efef6c0 2024-03-31T23:13:19.007 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:18.998+0000 7fe752216740 10 client.0 _ll_register_callbacks cb 0x56047ef49290 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T23:13:19.008 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:18.998+0000 7fe74cbd6640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56047eff8db0 0x56047eff6870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:13:19.008 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:18.998+0000 7fe74cbd6640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56047eff8db0 0x56047eff6870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:33784/0 (socket says 172.21.15.46:33784) 2024-03-31T23:13:19.008 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:18.998+0000 7fe74cbd6640 1 -- 172.21.15.46:0/2443139707 learned_addr learned my addr 172.21.15.46:0/2443139707 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:13:19.008 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:18.998+0000 7fe74d3d7640 1 --2- 172.21.15.46:0/2443139707 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56047efeffe0 0x56047eff6330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:13:19.008 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:18.999+0000 7fe74cbd6640 1 --2- 172.21.15.46:0/2443139707 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56047efef6c0 0x56047eff5df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:13:19.008 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:18.999+0000 7fe74cbd6640 1 -- 172.21.15.46:0/2443139707 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56047efef6c0 msgr2=0x56047eff5df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:13:19.008 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:18.999+0000 7fe74cbd6640 1 --2- 172.21.15.46:0/2443139707 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56047efef6c0 0x56047eff5df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:19.008 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:18.999+0000 7fe74cbd6640 1 -- 172.21.15.46:0/2443139707 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56047efeffe0 msgr2=0x56047eff6330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:13:19.008 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:18.999+0000 7fe74cbd6640 1 --2- 172.21.15.46:0/2443139707 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56047efeffe0 0x56047eff6330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:19.008 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:18.999+0000 7fe74cbd6640 1 -- 172.21.15.46:0/2443139707 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56047ed33620 con 0x56047eff8db0 2024-03-31T23:13:19.008 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:18.999+0000 7fe74d3d7640 1 --2- 172.21.15.46:0/2443139707 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56047efeffe0 0x56047eff6330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:13:19.008 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:18.999+0000 7fe74cbd6640 1 --2- 172.21.15.46:0/2443139707 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56047eff8db0 0x56047eff6870 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7fe7380204b0 tx=0x7fe73800ffa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a8dbe6f956c6735 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:13:19.009 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:18.999+0000 7fe730ff9640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-31T23:13:19.009 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:18.999+0000 7fe730ff9640 1 -- 172.21.15.46:0/2443139707 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe738020510 con 0x56047eff8db0 2024-03-31T23:13:19.009 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:18.999+0000 7fe730ff9640 1 -- 172.21.15.46:0/2443139707 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7fe738019040 con 0x56047eff8db0 2024-03-31T23:13:19.009 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:18.999+0000 7fe752216740 1 -- 172.21.15.46:0/2443139707 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe73800f7c0 con 0x56047eff8db0 2024-03-31T23:13:19.009 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:18.999+0000 7fe730ff9640 1 -- 172.21.15.46:0/2443139707 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fe7380206b0 con 0x56047eff8db0 2024-03-31T23:13:19.009 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:18.999+0000 7fe752216740 10 client.16197 fetch_fsmap learned FSMap version 28 2024-03-31T23:13:19.009 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:19.000+0000 7fe752216740 1 -- 172.21.15.46:0/2443139707 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x56047f007910 con 0x56047eff8db0 2024-03-31T23:13:19.009 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:19.000+0000 7fe730ff9640 1 -- 172.21.15.46:0/2443139707 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe73800f7c0 con 0x56047eff8db0 2024-03-31T23:13:19.009 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:19.000+0000 7fe730ff9640 1 -- 172.21.15.46:0/2443139707 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7fe73802c020 con 0x56047eff8db0 2024-03-31T23:13:19.009 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:19.000+0000 7fe752216740 10 client.16197 fetch_fsmap finished waiting for FSMap version 28 2024-03-31T23:13:19.009 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:19.000+0000 7fe752216740 10 client.16197 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-31T23:13:19.009 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:19.000+0000 7fe752216740 20 client.16197 populate_metadata read hostname 'smithi046' 2024-03-31T23:13:19.009 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:19.000+0000 7fe752216740 1 --2- 172.21.15.46:0/2443139707 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x56047f008c60 0x56047f00b050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:13:19.010 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:19.000+0000 7fe752216740 4 client.16197 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-31T23:13:19.010 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:19.000+0000 7fe752216740 1 -- 172.21.15.46:0/2443139707 --> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56047ef66120 con 0x56047f008c60 2024-03-31T23:13:19.010 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:19.000+0000 7fe74d3d7640 1 --2- 172.21.15.46:0/2443139707 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x56047f008c60 0x56047f00b050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:13:19.010 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:19.001+0000 7fe730ff9640 10 client.16197 ms_handle_connect on v2:172.21.15.142:6838/2519143918 2024-03-31T23:13:19.010 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:19.001+0000 7fe74d3d7640 1 --2- 172.21.15.46:0/2443139707 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x56047f008c60 0x56047f00b050 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d9339a35b0d7a4e5 server_cookie=27433b4dc0c87879 in_seq=0 out_seq=0 2024-03-31T23:13:19.010 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:19.001+0000 7fe730ff9640 1 -- 172.21.15.46:0/2443139707 <== mds.0 v2:172.21.15.142:6838/2519143918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x56047ef66120 con 0x56047f008c60 2024-03-31T23:13:19.010 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:19.002+0000 7fe730ff9640 10 client.16197 handle_command_reply: tid=0 2024-03-31T23:13:19.010 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:19.002+0000 7fe752216740 1 client.16197 shutdown 2024-03-31T23:13:19.010 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:19.002+0000 7fe752216740 1 -- 172.21.15.46:0/2443139707 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56047eff8db0 msgr2=0x56047eff6870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:13:19.010 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:19.002+0000 7fe752216740 1 --2- 172.21.15.46:0/2443139707 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56047eff8db0 0x56047eff6870 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7fe7380204b0 tx=0x7fe73800ffa0 comp rx=0 tx=0).stop 2024-03-31T23:13:19.010 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:19.002+0000 7fe752216740 1 -- 172.21.15.46:0/2443139707 shutdown_connections 2024-03-31T23:13:19.010 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:19.002+0000 7fe752216740 1 --2- 172.21.15.46:0/2443139707 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x56047f008c60 0x56047f00b050 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:19.010 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:19.002+0000 7fe752216740 1 --2- 172.21.15.46:0/2443139707 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56047eff8db0 0x56047eff6870 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:19.011 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:19.002+0000 7fe752216740 1 --2- 172.21.15.46:0/2443139707 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56047efeffe0 0x56047eff6330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:19.011 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:19.002+0000 7fe752216740 1 --2- 172.21.15.46:0/2443139707 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56047efef6c0 0x56047eff5df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:19.011 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:19.002+0000 7fe730ff9640 0 client.16197 ms_handle_reset on v2:172.21.15.142:6838/2519143918 2024-03-31T23:13:19.011 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:19.002+0000 7fe752216740 1 -- 172.21.15.46:0/2443139707 >> 172.21.15.46:0/2443139707 conn(0x56047efe9fe0 msgr2=0x56047efebd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:13:19.011 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:19.002+0000 7fe752216740 1 -- 172.21.15.46:0/2443139707 shutdown_connections 2024-03-31T23:13:19.011 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:19.002+0000 7fe752216740 1 -- 172.21.15.46:0/2443139707 wait complete. 2024-03-31T23:13:19.011 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:19.002+0000 7fe752216740 20 client.16197 trim_cache size 0 max 16384 2024-03-31T23:13:19.023 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:19.017+0000 7fe752216740 1 Processor -- start 2024-03-31T23:13:19.023 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:19.017+0000 7fe752216740 1 -- start start 2024-03-31T23:13:19.024 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:19.017+0000 7fe752216740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56047efef6c0 0x56047eff8520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:13:19.024 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:19.017+0000 7fe752216740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56047efeffe0 0x56047f004780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:13:19.024 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:19.017+0000 7fe752216740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56047eff8db0 0x56047eff4f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:13:19.024 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:19.017+0000 7fe752216740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fe7481b1470 con 0x56047efef6c0 2024-03-31T23:13:19.024 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:19.017+0000 7fe752216740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fe7481aed40 con 0x56047eff8db0 2024-03-31T23:13:19.024 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:19.017+0000 7fe752216740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fe7481acb90 con 0x56047efeffe0 2024-03-31T23:13:19.024 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:19.018+0000 7fe74cbd6640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56047efef6c0 0x56047eff8520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:13:19.024 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:19.018+0000 7fe74cbd6640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56047efef6c0 0x56047eff8520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:33792/0 (socket says 172.21.15.46:33792) 2024-03-31T23:13:19.024 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:19.018+0000 7fe74cbd6640 1 -- 172.21.15.46:0/2022032263 learned_addr learned my addr 172.21.15.46:0/2022032263 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:13:19.024 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:19.018+0000 7fe74d3d7640 1 --2- 172.21.15.46:0/2022032263 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56047efeffe0 0x56047f004780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:13:19.024 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:19.018+0000 7fe74cbd6640 1 --2- 172.21.15.46:0/2022032263 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56047eff8db0 0x56047eff4f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:13:19.024 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:19.018+0000 7fe74cbd6640 1 -- 172.21.15.46:0/2022032263 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56047efeffe0 msgr2=0x56047f004780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:13:19.024 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:19.018+0000 7fe74cbd6640 1 --2- 172.21.15.46:0/2022032263 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56047efeffe0 0x56047f004780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:19.025 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:19.018+0000 7fe74cbd6640 1 -- 172.21.15.46:0/2022032263 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56047eff8db0 msgr2=0x56047eff4f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:13:19.025 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:19.018+0000 7fe74cbd6640 1 --2- 172.21.15.46:0/2022032263 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56047eff8db0 0x56047eff4f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:19.025 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:19.018+0000 7fe74cbd6640 1 -- 172.21.15.46:0/2022032263 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56047efefe10 con 0x56047efef6c0 2024-03-31T23:13:19.025 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:19.018+0000 7fe74cbd6640 1 --2- 172.21.15.46:0/2022032263 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56047efef6c0 0x56047eff8520 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7fe738001eb0 tx=0x7fe73801ef50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b5af017792595a98 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:13:19.025 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:19.018+0000 7fe732ffd640 1 -- 172.21.15.46:0/2022032263 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe738009ce0 con 0x56047efef6c0 2024-03-31T23:13:19.025 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:19.018+0000 7fe732ffd640 1 -- 172.21.15.46:0/2022032263 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe7380204e0 con 0x56047efef6c0 2024-03-31T23:13:19.025 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:19.018+0000 7fe732ffd640 1 -- 172.21.15.46:0/2022032263 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe738020850 con 0x56047efef6c0 2024-03-31T23:13:19.025 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:19.019+0000 7fe752216740 1 -- 172.21.15.46:0/2022032263 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56047efef6c0 msgr2=0x56047eff8520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:13:19.025 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:19.019+0000 7fe752216740 1 --2- 172.21.15.46:0/2022032263 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56047efef6c0 0x56047eff8520 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7fe738001eb0 tx=0x7fe73801ef50 comp rx=0 tx=0).stop 2024-03-31T23:13:19.025 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:19.019+0000 7fe752216740 1 -- 172.21.15.46:0/2022032263 shutdown_connections 2024-03-31T23:13:19.025 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:19.019+0000 7fe752216740 1 --2- 172.21.15.46:0/2022032263 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56047eff8db0 0x56047eff4f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:19.025 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:19.019+0000 7fe752216740 1 --2- 172.21.15.46:0/2022032263 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56047efeffe0 0x56047f004780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:19.025 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:19.019+0000 7fe752216740 1 --2- 172.21.15.46:0/2022032263 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56047efef6c0 0x56047eff8520 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:19.025 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:19.019+0000 7fe752216740 1 -- 172.21.15.46:0/2022032263 >> 172.21.15.46:0/2022032263 conn(0x56047efe9fe0 msgr2=0x56047eff0510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:13:19.025 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:19.019+0000 7fe752216740 1 -- 172.21.15.46:0/2022032263 shutdown_connections 2024-03-31T23:13:19.026 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:19.019+0000 7fe752216740 1 -- 172.21.15.46:0/2022032263 wait complete. 2024-03-31T23:13:19.026 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:19.019+0000 7fe752216740 1 -- start start 2024-03-31T23:13:19.026 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:19.019+0000 7fe752216740 1 Processor -- start 2024-03-31T23:13:19.026 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:19.020+0000 7fe752216740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56047efef6c0 0x56047effbd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:13:19.026 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:19.020+0000 7fe752216740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56047efeffe0 0x56047effc2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:13:19.026 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:19.020+0000 7fe752216740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56047eff8db0 0x56047f06cea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:13:19.026 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:19.020+0000 7fe752216740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fe7481ad200 con 0x56047efef6c0 2024-03-31T23:13:19.026 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:19.020+0000 7fe752216740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fe7481ae670 con 0x56047eff8db0 2024-03-31T23:13:19.026 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:19.020+0000 7fe752216740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fe7481ad8d0 con 0x56047efeffe0 2024-03-31T23:13:19.026 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:19.020+0000 7fe752216740 10 client.0 _ll_register_callbacks cb 0x7fe738009ea0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T23:13:19.026 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:19.020+0000 7fe74cbd6640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56047efef6c0 0x56047effbd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:13:19.026 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:19.020+0000 7fe74cbd6640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56047efef6c0 0x56047effbd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:33804/0 (socket says 172.21.15.46:33804) 2024-03-31T23:13:19.026 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:19.020+0000 7fe74cbd6640 1 -- 172.21.15.46:0/2183591322 learned_addr learned my addr 172.21.15.46:0/2183591322 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:13:19.027 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:19.020+0000 7fe74d3d7640 1 --2- 172.21.15.46:0/2183591322 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56047efeffe0 0x56047effc2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:13:19.027 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:19.020+0000 7fe74cbd6640 1 --2- 172.21.15.46:0/2183591322 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56047eff8db0 0x56047f06cea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:13:19.027 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:19.020+0000 7fe74cbd6640 1 -- 172.21.15.46:0/2183591322 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56047efeffe0 msgr2=0x56047effc2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:13:19.027 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:19.020+0000 7fe74cbd6640 1 --2- 172.21.15.46:0/2183591322 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56047efeffe0 0x56047effc2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:19.027 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:19.020+0000 7fe74cbd6640 1 -- 172.21.15.46:0/2183591322 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56047eff8db0 msgr2=0x56047f06cea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:13:19.027 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:19.020+0000 7fe74cbd6640 1 --2- 172.21.15.46:0/2183591322 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56047eff8db0 0x56047f06cea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:19.027 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:19.020+0000 7fe74cbd6640 1 -- 172.21.15.46:0/2183591322 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56047effcba0 con 0x56047efef6c0 2024-03-31T23:13:19.027 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:19.020+0000 7fe74cbd6640 1 --2- 172.21.15.46:0/2183591322 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56047efef6c0 0x56047effbd70 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7fe73801ea60 tx=0x7fe738005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7680fe4cc36cab0f server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:13:19.027 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:19.020+0000 7fe730ff9640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-31T23:13:19.027 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:19.020+0000 7fe730ff9640 1 -- 172.21.15.46:0/2183591322 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe73801e3a0 con 0x56047efef6c0 2024-03-31T23:13:19.027 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:19.021+0000 7fe730ff9640 1 -- 172.21.15.46:0/2183591322 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7fe73801e540 con 0x56047efef6c0 2024-03-31T23:13:19.027 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:19.021+0000 7fe752216740 1 -- 172.21.15.46:0/2183591322 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe7380204e0 con 0x56047efef6c0 2024-03-31T23:13:19.028 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:19.021+0000 7fe730ff9640 1 -- 172.21.15.46:0/2183591322 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fe73801e740 con 0x56047efef6c0 2024-03-31T23:13:19.028 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:19.021+0000 7fe752216740 10 client.16203 fetch_fsmap learned FSMap version 28 2024-03-31T23:13:19.028 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:19.021+0000 7fe752216740 1 -- 172.21.15.46:0/2183591322 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x56047f06e520 con 0x56047efef6c0 2024-03-31T23:13:19.028 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:19.021+0000 7fe730ff9640 1 -- 172.21.15.46:0/2183591322 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe7380204e0 con 0x56047efef6c0 2024-03-31T23:13:19.028 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:19.021+0000 7fe730ff9640 1 -- 172.21.15.46:0/2183591322 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7fe73802c030 con 0x56047efef6c0 2024-03-31T23:13:19.028 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:19.021+0000 7fe752216740 10 client.16203 fetch_fsmap finished waiting for FSMap version 28 2024-03-31T23:13:19.028 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:19.021+0000 7fe752216740 10 client.16203 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-31T23:13:19.028 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:19.021+0000 7fe752216740 20 client.16203 populate_metadata read hostname 'smithi046' 2024-03-31T23:13:19.028 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:19.021+0000 7fe752216740 1 --2- 172.21.15.46:0/2183591322 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x56047f06ec70 0x56047f071090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:13:19.028 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:19.021+0000 7fe752216740 4 client.16203 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-31T23:13:19.028 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:19.021+0000 7fe752216740 1 -- 172.21.15.46:0/2183591322 --> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] -- command(tid 0: {"prefix": "flush journal"}) -- 0x56047ef5fe30 con 0x56047f06ec70 2024-03-31T23:13:19.028 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:19.021+0000 7fe74d3d7640 1 --2- 172.21.15.46:0/2183591322 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x56047f06ec70 0x56047f071090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:13:19.028 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:19.022+0000 7fe730ff9640 10 client.16203 ms_handle_connect on v2:172.21.15.142:6838/2519143918 2024-03-31T23:13:19.029 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:19.022+0000 7fe74d3d7640 1 --2- 172.21.15.46:0/2183591322 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x56047f06ec70 0x56047f071090 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dd5e86345c031c3c server_cookie=e69e4722282ab485 in_seq=0 out_seq=0 2024-03-31T23:13:19.040 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:19.039+0000 7fe730ff9640 1 -- 172.21.15.46:0/2183591322 <== mds.0 v2:172.21.15.142:6838/2519143918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x56047ef5fe30 con 0x56047f06ec70 2024-03-31T23:13:19.040 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:19.039+0000 7fe730ff9640 10 client.16203 handle_command_reply: tid=0 2024-03-31T23:13:19.040 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:19.039+0000 7fe752216740 1 client.16203 shutdown 2024-03-31T23:13:19.040 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:19.039+0000 7fe752216740 1 -- 172.21.15.46:0/2183591322 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56047efef6c0 msgr2=0x56047effbd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:13:19.041 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:19.039+0000 7fe752216740 1 --2- 172.21.15.46:0/2183591322 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56047efef6c0 0x56047effbd70 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7fe73801ea60 tx=0x7fe738005f90 comp rx=0 tx=0).stop 2024-03-31T23:13:19.041 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:19.039+0000 7fe752216740 1 -- 172.21.15.46:0/2183591322 shutdown_connections 2024-03-31T23:13:19.041 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:19.039+0000 7fe752216740 1 --2- 172.21.15.46:0/2183591322 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x56047f06ec70 0x56047f071090 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:19.041 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:19.039+0000 7fe752216740 1 --2- 172.21.15.46:0/2183591322 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56047eff8db0 0x56047f06cea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:19.041 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:19.039+0000 7fe752216740 1 --2- 172.21.15.46:0/2183591322 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56047efeffe0 0x56047effc2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:19.041 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:19.039+0000 7fe752216740 1 --2- 172.21.15.46:0/2183591322 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56047efef6c0 0x56047effbd70 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:19.041 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:19.039+0000 7fe730ff9640 0 client.16203 ms_handle_reset on v2:172.21.15.142:6838/2519143918 2024-03-31T23:13:19.041 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:19.039+0000 7fe752216740 1 -- 172.21.15.46:0/2183591322 >> 172.21.15.46:0/2183591322 conn(0x56047efe9fe0 msgr2=0x56047eff4f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:13:19.041 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:19.039+0000 7fe752216740 1 -- 172.21.15.46:0/2183591322 shutdown_connections 2024-03-31T23:13:19.041 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:19.039+0000 7fe752216740 1 -- 172.21.15.46:0/2183591322 wait complete. 2024-03-31T23:13:19.041 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:19.039+0000 7fe752216740 20 client.16203 trim_cache size 0 max 16384 2024-03-31T23:13:19.041 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout:{ 2024-03-31T23:13:19.042 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout: "message": "", 2024-03-31T23:13:19.042 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout: "return_code": 0 2024-03-31T23:13:19.042 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout:} 2024-03-31T23:13:19.043 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:19.039+0000 7fe74f63b640 1 -- 172.21.15.46:0/2347304256 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fe7340a26b0 msgr2=0x7fe7340a4b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:13:19.043 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:19.039+0000 7fe74f63b640 1 --2- 172.21.15.46:0/2347304256 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fe7340a26b0 0x7fe7340a4b70 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fe74009b670 tx=0x7fe74006a610 comp rx=0 tx=0).stop 2024-03-31T23:13:19.043 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:19.040+0000 7fe74f63b640 1 -- 172.21.15.46:0/2347304256 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe7481055f0 msgr2=0x7fe74819c800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:13:19.043 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:19.040+0000 7fe74f63b640 1 --2- 172.21.15.46:0/2347304256 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe7481055f0 0x7fe74819c800 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7fe74000ea60 tx=0x7fe74000ef20 comp rx=0 tx=0).stop 2024-03-31T23:13:19.043 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:19.040+0000 7fe74f63b640 1 -- 172.21.15.46:0/2347304256 shutdown_connections 2024-03-31T23:13:19.043 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:19.040+0000 7fe74f63b640 1 --2- 172.21.15.46:0/2347304256 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fe7340a26b0 0x7fe7340a4b70 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:19.043 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:19.040+0000 7fe74f63b640 1 --2- 172.21.15.46:0/2347304256 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe7481055f0 0x7fe74819c800 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:19.043 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:19.040+0000 7fe74f63b640 1 --2- 172.21.15.46:0/2347304256 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe748101260 0x7fe748198480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:19.044 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:19.040+0000 7fe74f63b640 1 --2- 172.21.15.46:0/2347304256 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe7481008a0 0x7fe748197f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:19.044 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:19.040+0000 7fe74f63b640 1 -- 172.21.15.46:0/2347304256 >> 172.21.15.46:0/2347304256 conn(0x7fe7480774a0 msgr2=0x7fe7480fe8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:13:19.044 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:19.040+0000 7fe74f63b640 1 -- 172.21.15.46:0/2347304256 shutdown_connections 2024-03-31T23:13:19.044 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:19.040+0000 7fe74f63b640 1 -- 172.21.15.46:0/2347304256 wait complete. 2024-03-31T23:13:19.190 INFO:tasks.workunit.client.0.smithi046.stdout: 1 11582 3.17 MB/sec warmup 25 sec latency 1434.232 ms 2024-03-31T23:13:20.190 INFO:tasks.workunit.client.0.smithi046.stdout: 1 11582 3.04 MB/sec warmup 26 sec latency 2434.286 ms 2024-03-31T23:13:20.611 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:13:20 smithi046 ceph-mon[32498]: pgmap v51: 129 pgs: 129 active+clean; 35 MiB data, 489 MiB used, 1.0 TiB / 1.0 TiB avail; 344 KiB/s rd, 1.4 MiB/s wr, 157 op/s 2024-03-31T23:13:20.795 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:13:20 smithi162 ceph-mon[36237]: pgmap v51: 129 pgs: 129 active+clean; 35 MiB data, 489 MiB used, 1.0 TiB / 1.0 TiB avail; 344 KiB/s rd, 1.4 MiB/s wr, 157 op/s 2024-03-31T23:13:20.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:13:20 smithi142 ceph-mon[35271]: pgmap v51: 129 pgs: 129 active+clean; 35 MiB data, 489 MiB used, 1.0 TiB / 1.0 TiB avail; 344 KiB/s rd, 1.4 MiB/s wr, 157 op/s 2024-03-31T23:13:21.190 INFO:tasks.workunit.client.0.smithi046.stdout: 1 11648 2.94 MB/sec warmup 27 sec latency 2865.434 ms 2024-03-31T23:13:22.191 INFO:tasks.workunit.client.0.smithi046.stdout: 1 11648 2.83 MB/sec warmup 28 sec latency 1399.676 ms 2024-03-31T23:13:22.795 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:13:22 smithi162 ceph-mon[36237]: pgmap v52: 129 pgs: 129 active+clean; 35 MiB data, 489 MiB used, 1.0 TiB / 1.0 TiB avail; 188 KiB/s rd, 819 KiB/s wr, 94 op/s 2024-03-31T23:13:22.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:13:22 smithi046 ceph-mon[32498]: pgmap v52: 129 pgs: 129 active+clean; 35 MiB data, 489 MiB used, 1.0 TiB / 1.0 TiB avail; 188 KiB/s rd, 819 KiB/s wr, 94 op/s 2024-03-31T23:13:22.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:13:22 smithi142 ceph-mon[35271]: pgmap v52: 129 pgs: 129 active+clean; 35 MiB data, 489 MiB used, 1.0 TiB / 1.0 TiB avail; 188 KiB/s rd, 819 KiB/s wr, 94 op/s 2024-03-31T23:13:23.191 INFO:tasks.workunit.client.0.smithi046.stdout: 1 11648 2.73 MB/sec warmup 29 sec latency 2399.790 ms 2024-03-31T23:13:24.191 INFO:tasks.workunit.client.0.smithi046.stdout: 1 11648 2.64 MB/sec warmup 30 sec latency 3399.880 ms 2024-03-31T23:13:24.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:13:24 smithi162 ceph-mon[36237]: pgmap v53: 129 pgs: 129 active+clean; 35 MiB data, 489 MiB used, 1.0 TiB / 1.0 TiB avail; 188 KiB/s rd, 819 KiB/s wr, 94 op/s 2024-03-31T23:13:24.862 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:13:24 smithi046 ceph-mon[32498]: pgmap v53: 129 pgs: 129 active+clean; 35 MiB data, 489 MiB used, 1.0 TiB / 1.0 TiB avail; 188 KiB/s rd, 819 KiB/s wr, 94 op/s 2024-03-31T23:13:24.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:13:24 smithi142 ceph-mon[35271]: pgmap v53: 129 pgs: 129 active+clean; 35 MiB data, 489 MiB used, 1.0 TiB / 1.0 TiB avail; 188 KiB/s rd, 819 KiB/s wr, 94 op/s 2024-03-31T23:13:25.191 INFO:tasks.workunit.client.0.smithi046.stdout: 1 11648 2.56 MB/sec warmup 31 sec latency 4399.969 ms 2024-03-31T23:13:26.191 INFO:tasks.workunit.client.0.smithi046.stdout: 1 11926 2.52 MB/sec warmup 32 sec latency 4827.785 ms 2024-03-31T23:13:26.797 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:13:26 smithi162 ceph-mon[36237]: pgmap v54: 129 pgs: 129 active+clean; 28 MiB data, 471 MiB used, 1.0 TiB / 1.0 TiB avail; 211 KiB/s rd, 852 KiB/s wr, 110 op/s 2024-03-31T23:13:26.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:13:26 smithi046 ceph-mon[32498]: pgmap v54: 129 pgs: 129 active+clean; 28 MiB data, 471 MiB used, 1.0 TiB / 1.0 TiB avail; 211 KiB/s rd, 852 KiB/s wr, 110 op/s 2024-03-31T23:13:26.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:13:26 smithi142 ceph-mon[35271]: pgmap v54: 129 pgs: 129 active+clean; 28 MiB data, 471 MiB used, 1.0 TiB / 1.0 TiB avail; 211 KiB/s rd, 852 KiB/s wr, 110 op/s 2024-03-31T23:13:27.191 INFO:tasks.workunit.client.0.smithi046.stdout: 1 12529 2.54 MB/sec warmup 33 sec latency 16.215 ms 2024-03-31T23:13:28.191 INFO:tasks.workunit.client.0.smithi046.stdout: 1 13101 2.51 MB/sec warmup 34 sec latency 16.578 ms 2024-03-31T23:13:28.795 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:13:28 smithi162 ceph-mon[36237]: pgmap v55: 129 pgs: 129 active+clean; 28 MiB data, 471 MiB used, 1.0 TiB / 1.0 TiB avail; 140 KiB/s rd, 518 KiB/s wr, 66 op/s 2024-03-31T23:13:28.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:13:28 smithi046 ceph-mon[32498]: pgmap v55: 129 pgs: 129 active+clean; 28 MiB data, 471 MiB used, 1.0 TiB / 1.0 TiB avail; 140 KiB/s rd, 518 KiB/s wr, 66 op/s 2024-03-31T23:13:28.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:13:28 smithi142 ceph-mon[35271]: pgmap v55: 129 pgs: 129 active+clean; 28 MiB data, 471 MiB used, 1.0 TiB / 1.0 TiB avail; 140 KiB/s rd, 518 KiB/s wr, 66 op/s 2024-03-31T23:13:29.191 INFO:tasks.workunit.client.0.smithi046.stdout: 1 13856 2.62 MB/sec warmup 35 sec latency 15.833 ms 2024-03-31T23:13:30.191 INFO:tasks.workunit.client.0.smithi046.stdout: 1 14227 2.66 MB/sec warmup 36 sec latency 16.730 ms 2024-03-31T23:13:30.795 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:13:30 smithi162 ceph-mon[36237]: pgmap v56: 129 pgs: 129 active+clean; 30 MiB data, 479 MiB used, 1.0 TiB / 1.0 TiB avail; 182 KiB/s rd, 844 KiB/s wr, 99 op/s 2024-03-31T23:13:30.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:13:30 smithi046 ceph-mon[32498]: pgmap v56: 129 pgs: 129 active+clean; 30 MiB data, 479 MiB used, 1.0 TiB / 1.0 TiB avail; 182 KiB/s rd, 844 KiB/s wr, 99 op/s 2024-03-31T23:13:30.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:13:30 smithi142 ceph-mon[35271]: pgmap v56: 129 pgs: 129 active+clean; 30 MiB data, 479 MiB used, 1.0 TiB / 1.0 TiB avail; 182 KiB/s rd, 844 KiB/s wr, 99 op/s 2024-03-31T23:13:31.191 INFO:tasks.workunit.client.0.smithi046.stdout: 1 14575 2.63 MB/sec warmup 37 sec latency 15.691 ms 2024-03-31T23:13:32.133 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:32.132+0000 7f5b94aae640 1 -- 172.21.15.46:0/2636029102 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5b901055f0 msgr2=0x7f5b901059d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:13:32.133 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:32.132+0000 7f5b94aae640 1 --2- 172.21.15.46:0/2636029102 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5b901055f0 0x7f5b901059d0 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f5b80009900 tx=0x7f5b8002bc30 comp rx=0 tx=0).stop 2024-03-31T23:13:32.133 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:32.132+0000 7f5b94aae640 1 -- 172.21.15.46:0/2636029102 shutdown_connections 2024-03-31T23:13:32.134 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:32.132+0000 7f5b94aae640 1 --2- 172.21.15.46:0/2636029102 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5b90101260 0x7f5b9010d480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:32.134 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:32.132+0000 7f5b94aae640 1 --2- 172.21.15.46:0/2636029102 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5b901008a0 0x7f5b90100d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:32.134 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:32.132+0000 7f5b94aae640 1 --2- 172.21.15.46:0/2636029102 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5b901055f0 0x7f5b901059d0 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:32.134 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:32.132+0000 7f5b94aae640 1 -- 172.21.15.46:0/2636029102 >> 172.21.15.46:0/2636029102 conn(0x7f5b900774a0 msgr2=0x7f5b900778b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:13:32.134 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:32.132+0000 7f5b94aae640 1 -- 172.21.15.46:0/2636029102 shutdown_connections 2024-03-31T23:13:32.134 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:32.132+0000 7f5b94aae640 1 -- 172.21.15.46:0/2636029102 wait complete. 2024-03-31T23:13:32.134 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:32.132+0000 7f5b94aae640 1 Processor -- start 2024-03-31T23:13:32.135 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:32.132+0000 7f5b94aae640 1 -- start start 2024-03-31T23:13:32.137 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:32.133+0000 7f5b94aae640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5b901008a0 0x7f5b901022c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:13:32.137 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:32.133+0000 7f5b94aae640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5b90101260 0x7f5b90102800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:13:32.137 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:32.133+0000 7f5b94aae640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5b901055f0 0x7f5b90102d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:13:32.137 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:32.133+0000 7f5b94aae640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f5b90110980 con 0x7f5b90101260 2024-03-31T23:13:32.137 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:32.133+0000 7f5b94aae640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f5b90110800 con 0x7f5b901008a0 2024-03-31T23:13:32.137 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:32.133+0000 7f5b94aae640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f5b90110b00 con 0x7f5b901055f0 2024-03-31T23:13:32.137 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:32.133+0000 7f5b8dd9b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5b90101260 0x7f5b90102800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:13:32.137 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:32.133+0000 7f5b8e59c640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5b901008a0 0x7f5b901022c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:13:32.137 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:32.133+0000 7f5b8dd9b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5b90101260 0x7f5b90102800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:45856/0 (socket says 172.21.15.46:45856) 2024-03-31T23:13:32.138 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:32.133+0000 7f5b8dd9b640 1 -- 172.21.15.46:0/3431924280 learned_addr learned my addr 172.21.15.46:0/3431924280 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:13:32.138 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:32.133+0000 7f5b8ed9d640 1 --2- 172.21.15.46:0/3431924280 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5b901055f0 0x7f5b90102d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:13:32.138 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:32.133+0000 7f5b8dd9b640 1 -- 172.21.15.46:0/3431924280 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5b901055f0 msgr2=0x7f5b90102d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:13:32.138 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:32.133+0000 7f5b8dd9b640 1 --2- 172.21.15.46:0/3431924280 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5b901055f0 0x7f5b90102d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:32.138 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:32.133+0000 7f5b8dd9b640 1 -- 172.21.15.46:0/3431924280 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5b901008a0 msgr2=0x7f5b901022c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:13:32.138 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:32.133+0000 7f5b8dd9b640 1 --2- 172.21.15.46:0/3431924280 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5b901008a0 0x7f5b901022c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:32.138 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:32.133+0000 7f5b8dd9b640 1 -- 172.21.15.46:0/3431924280 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b901a6420 con 0x7f5b90101260 2024-03-31T23:13:32.138 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:32.133+0000 7f5b8ed9d640 1 --2- 172.21.15.46:0/3431924280 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5b901055f0 0x7f5b90102d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T23:13:32.138 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:32.133+0000 7f5b8dd9b640 1 --2- 172.21.15.46:0/3431924280 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5b90101260 0x7f5b90102800 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f5b8400d950 tx=0x7f5b8400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:13:32.138 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:32.133+0000 7f5b7f7fe640 1 -- 172.21.15.46:0/3431924280 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b84017070 con 0x7f5b90101260 2024-03-31T23:13:32.138 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:32.134+0000 7f5b7f7fe640 1 -- 172.21.15.46:0/3431924280 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5b84009c30 con 0x7f5b90101260 2024-03-31T23:13:32.138 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:32.134+0000 7f5b7f7fe640 1 -- 172.21.15.46:0/3431924280 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b8400f620 con 0x7f5b90101260 2024-03-31T23:13:32.138 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:32.134+0000 7f5b94aae640 1 -- 172.21.15.46:0/3431924280 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5b901a6710 con 0x7f5b90101260 2024-03-31T23:13:32.139 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:32.134+0000 7f5b94aae640 1 -- 172.21.15.46:0/3431924280 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5b901a6c60 con 0x7f5b90101260 2024-03-31T23:13:32.139 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:32.136+0000 7f5b7f7fe640 1 -- 172.21.15.46:0/3431924280 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 25) ==== 137451+0+0 (secure 0 0 0) 0x7f5b840109a0 con 0x7f5b90101260 2024-03-31T23:13:32.139 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:32.136+0000 7f5b9776f740 1 Processor -- start 2024-03-31T23:13:32.145 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:32.136+0000 7f5b9776f740 1 -- start start 2024-03-31T23:13:32.145 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:32.137+0000 7f5b7f7fe640 1 --2- 172.21.15.46:0/3431924280 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f5b780a26b0 0x7f5b780a4b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:13:32.145 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:32.137+0000 7f5b9776f740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x556040d6c6c0 0x556040d6caa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:13:32.145 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:32.137+0000 7f5b9776f740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556040d6cfe0 0x556040d75870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:13:32.145 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:32.137+0000 7f5b8ed9d640 1 --2- 172.21.15.46:0/3431924280 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f5b780a26b0 0x7f5b780a4b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:13:32.145 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:32.137+0000 7f5b7f7fe640 1 -- 172.21.15.46:0/3431924280 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7f5b840ceb60 con 0x7f5b90101260 2024-03-31T23:13:32.145 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:32.137+0000 7f5b9776f740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556040d75db0 0x556040d781a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:13:32.145 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:32.137+0000 7f5b9776f740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x556040ccf8f0 con 0x556040d6cfe0 2024-03-31T23:13:32.145 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:32.137+0000 7f5b9776f740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x556040c788d0 con 0x556040d6c6c0 2024-03-31T23:13:32.145 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:32.137+0000 7f5b9776f740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x556040ccb990 con 0x556040d75db0 2024-03-31T23:13:32.145 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:32.137+0000 7f5b8dd9b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556040d6cfe0 0x556040d75870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:13:32.146 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:32.137+0000 7f5b8dd9b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556040d6cfe0 0x556040d75870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:45860/0 (socket says 172.21.15.46:45860) 2024-03-31T23:13:32.146 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:32.137+0000 7f5b8dd9b640 1 -- 172.21.15.46:0/4124267063 learned_addr learned my addr 172.21.15.46:0/4124267063 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:13:32.146 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:32.137+0000 7f5b8e59c640 1 --2- 172.21.15.46:0/4124267063 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x556040d6c6c0 0x556040d6caa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:13:32.146 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:32.137+0000 7f5b8ed9d640 1 --2- 172.21.15.46:0/4124267063 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556040d75db0 0x556040d781a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:13:32.146 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:32.137+0000 7f5b8dd9b640 1 -- 172.21.15.46:0/4124267063 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556040d75db0 msgr2=0x556040d781a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:13:32.146 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:32.137+0000 7f5b8dd9b640 1 --2- 172.21.15.46:0/4124267063 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556040d75db0 0x556040d781a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:32.146 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:32.137+0000 7f5b8dd9b640 1 -- 172.21.15.46:0/4124267063 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x556040d6c6c0 msgr2=0x556040d6caa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:13:32.146 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:32.137+0000 7f5b8dd9b640 1 --2- 172.21.15.46:0/4124267063 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x556040d6c6c0 0x556040d6caa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:32.146 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:32.137+0000 7f5b8dd9b640 1 -- 172.21.15.46:0/4124267063 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5560409e4210 con 0x556040d6cfe0 2024-03-31T23:13:32.146 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:32.137+0000 7f5b8ed9d640 1 --2- 172.21.15.46:0/3431924280 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f5b780a26b0 0x7f5b780a4b70 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f5b88009fd0 tx=0x7f5b88015ca0 comp rx=0 tx=0).ready entity=mgr.15654 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:13:32.146 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:32.137+0000 7f5b8dd9b640 1 --2- 172.21.15.46:0/4124267063 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556040d6cfe0 0x556040d75870 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f5b840a3410 tx=0x7f5b84009de0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d58f957f3504e0e8 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:13:32.146 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:32.138+0000 7f5b7cff9640 1 -- 172.21.15.46:0/4124267063 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b84020040 con 0x556040d6cfe0 2024-03-31T23:13:32.147 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:32.138+0000 7f5b7cff9640 1 -- 172.21.15.46:0/4124267063 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5b8409af30 con 0x556040d6cfe0 2024-03-31T23:13:32.147 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:32.138+0000 7f5b7cff9640 1 -- 172.21.15.46:0/4124267063 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b84096c80 con 0x556040d6cfe0 2024-03-31T23:13:32.147 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:32.138+0000 7f5b9776f740 1 -- 172.21.15.46:0/4124267063 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556040d6cfe0 msgr2=0x556040d75870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:13:32.147 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:32.138+0000 7f5b9776f740 1 --2- 172.21.15.46:0/4124267063 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556040d6cfe0 0x556040d75870 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f5b840a3410 tx=0x7f5b84009de0 comp rx=0 tx=0).stop 2024-03-31T23:13:32.147 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:32.138+0000 7f5b9776f740 1 -- 172.21.15.46:0/4124267063 shutdown_connections 2024-03-31T23:13:32.147 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:32.138+0000 7f5b9776f740 1 --2- 172.21.15.46:0/4124267063 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556040d75db0 0x556040d781a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:32.147 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:32.138+0000 7f5b9776f740 1 --2- 172.21.15.46:0/4124267063 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556040d6cfe0 0x556040d75870 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:32.147 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:32.138+0000 7f5b9776f740 1 --2- 172.21.15.46:0/4124267063 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x556040d6c6c0 0x556040d6caa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:32.147 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:32.138+0000 7f5b9776f740 1 -- 172.21.15.46:0/4124267063 >> 172.21.15.46:0/4124267063 conn(0x556040d66fe0 msgr2=0x556040d673c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:13:32.147 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:32.139+0000 7f5b9776f740 1 -- 172.21.15.46:0/4124267063 shutdown_connections 2024-03-31T23:13:32.147 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:32.139+0000 7f5b9776f740 1 -- 172.21.15.46:0/4124267063 wait complete. 2024-03-31T23:13:32.147 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:32.139+0000 7f5b9776f740 1 -- start start 2024-03-31T23:13:32.147 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:32.139+0000 7f5b9776f740 1 Processor -- start 2024-03-31T23:13:32.147 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:32.139+0000 7f5b9776f740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556040d6c6c0 0x556040d72e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:13:32.148 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:32.139+0000 7f5b9776f740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x556040d6cfe0 0x556040d73390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:13:32.148 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:32.139+0000 7f5b9776f740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556040d75db0 0x556040d738d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:13:32.148 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:32.139+0000 7f5b9776f740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x556040d789e0 con 0x556040d75db0 2024-03-31T23:13:32.148 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:32.139+0000 7f5b9776f740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x556040c6df80 con 0x556040d6cfe0 2024-03-31T23:13:32.148 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:32.139+0000 7f5b9776f740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x556040d78b60 con 0x556040d6c6c0 2024-03-31T23:13:32.148 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:32.139+0000 7f5b9776f740 10 client.0 _ll_register_callbacks cb 0x556040cc6290 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T23:13:32.148 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:32.139+0000 7f5b8ed9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556040d75db0 0x556040d738d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:13:32.148 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:32.139+0000 7f5b8ed9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556040d75db0 0x556040d738d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:45862/0 (socket says 172.21.15.46:45862) 2024-03-31T23:13:32.148 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:32.139+0000 7f5b8dd9b640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x556040d6cfe0 0x556040d73390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:13:32.148 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:32.139+0000 7f5b8ed9d640 1 -- 172.21.15.46:0/1336590757 learned_addr learned my addr 172.21.15.46:0/1336590757 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:13:32.148 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:32.139+0000 7f5b8e59c640 1 --2- 172.21.15.46:0/1336590757 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556040d6c6c0 0x556040d72e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:13:32.148 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:32.140+0000 7f5b8ed9d640 1 -- 172.21.15.46:0/1336590757 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556040d6c6c0 msgr2=0x556040d72e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:13:32.148 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:32.140+0000 7f5b8ed9d640 1 --2- 172.21.15.46:0/1336590757 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556040d6c6c0 0x556040d72e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:32.149 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:32.140+0000 7f5b8ed9d640 1 -- 172.21.15.46:0/1336590757 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x556040d6cfe0 msgr2=0x556040d73390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:13:32.149 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:32.140+0000 7f5b8ed9d640 1 --2- 172.21.15.46:0/1336590757 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x556040d6cfe0 0x556040d73390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:32.149 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:32.140+0000 7f5b8ed9d640 1 -- 172.21.15.46:0/1336590757 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556040ab0620 con 0x556040d75db0 2024-03-31T23:13:32.149 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:32.140+0000 7f5b8ed9d640 1 --2- 172.21.15.46:0/1336590757 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556040d75db0 0x556040d738d0 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f5b88011b50 tx=0x7f5b88004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c338c78a7c9fb90 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:13:32.149 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:32.140+0000 7f5b75ffb640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-31T23:13:32.149 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:32.140+0000 7f5b75ffb640 1 -- 172.21.15.46:0/1336590757 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b88021440 con 0x556040d75db0 2024-03-31T23:13:32.149 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:32.140+0000 7f5b75ffb640 1 -- 172.21.15.46:0/1336590757 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7f5b880215e0 con 0x556040d75db0 2024-03-31T23:13:32.149 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:32.140+0000 7f5b9776f740 1 -- 172.21.15.46:0/1336590757 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5b8409af30 con 0x556040d75db0 2024-03-31T23:13:32.149 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:32.140+0000 7f5b75ffb640 1 -- 172.21.15.46:0/1336590757 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f5b88030050 con 0x556040d75db0 2024-03-31T23:13:32.149 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:32.140+0000 7f5b9776f740 10 client.16215 fetch_fsmap learned FSMap version 28 2024-03-31T23:13:32.149 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:32.140+0000 7f5b9776f740 1 -- 172.21.15.46:0/1336590757 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x556040d84910 con 0x556040d75db0 2024-03-31T23:13:32.149 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:32.140+0000 7f5b75ffb640 1 -- 172.21.15.46:0/1336590757 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5b8409af30 con 0x556040d75db0 2024-03-31T23:13:32.149 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:32.140+0000 7f5b75ffb640 1 -- 172.21.15.46:0/1336590757 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f5b8800f690 con 0x556040d75db0 2024-03-31T23:13:32.149 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:32.140+0000 7f5b9776f740 10 client.16215 fetch_fsmap finished waiting for FSMap version 28 2024-03-31T23:13:32.150 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:32.141+0000 7f5b9776f740 10 client.16215 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-31T23:13:32.150 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:32.141+0000 7f5b9776f740 20 client.16215 populate_metadata read hostname 'smithi046' 2024-03-31T23:13:32.150 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:32.141+0000 7f5b9776f740 1 --2- 172.21.15.46:0/1336590757 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x556040d85e40 0x556040d88230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:13:32.150 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:32.141+0000 7f5b9776f740 4 client.16215 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-31T23:13:32.150 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:32.141+0000 7f5b9776f740 1 -- 172.21.15.46:0/1336590757 --> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556040ce3120 con 0x556040d85e40 2024-03-31T23:13:32.150 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:32.141+0000 7f5b8e59c640 1 --2- 172.21.15.46:0/1336590757 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x556040d85e40 0x556040d88230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:13:32.150 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:32.141+0000 7f5b75ffb640 10 client.16215 ms_handle_connect on v2:172.21.15.142:6838/2519143918 2024-03-31T23:13:32.150 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:32.141+0000 7f5b8e59c640 1 --2- 172.21.15.46:0/1336590757 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x556040d85e40 0x556040d88230 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=68b82a9d830c9db0 server_cookie=2f76fd8e9c25eaad in_seq=0 out_seq=0 2024-03-31T23:13:32.150 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:32.142+0000 7f5b75ffb640 1 -- 172.21.15.46:0/1336590757 <== mds.0 v2:172.21.15.142:6838/2519143918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x556040ce3120 con 0x556040d85e40 2024-03-31T23:13:32.150 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:32.142+0000 7f5b75ffb640 10 client.16215 handle_command_reply: tid=0 2024-03-31T23:13:32.150 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:32.142+0000 7f5b9776f740 1 client.16215 shutdown 2024-03-31T23:13:32.150 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:32.142+0000 7f5b9776f740 1 -- 172.21.15.46:0/1336590757 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556040d75db0 msgr2=0x556040d738d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:13:32.150 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:32.142+0000 7f5b9776f740 1 --2- 172.21.15.46:0/1336590757 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556040d75db0 0x556040d738d0 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f5b88011b50 tx=0x7f5b88004ab0 comp rx=0 tx=0).stop 2024-03-31T23:13:32.150 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:32.142+0000 7f5b9776f740 1 -- 172.21.15.46:0/1336590757 shutdown_connections 2024-03-31T23:13:32.151 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:32.142+0000 7f5b9776f740 1 --2- 172.21.15.46:0/1336590757 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x556040d85e40 0x556040d88230 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:32.151 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:32.142+0000 7f5b9776f740 1 --2- 172.21.15.46:0/1336590757 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556040d75db0 0x556040d738d0 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:32.151 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:32.142+0000 7f5b9776f740 1 --2- 172.21.15.46:0/1336590757 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x556040d6cfe0 0x556040d73390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:32.151 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:32.142+0000 7f5b9776f740 1 --2- 172.21.15.46:0/1336590757 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556040d6c6c0 0x556040d72e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:32.151 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:32.142+0000 7f5b75ffb640 0 client.16215 ms_handle_reset on v2:172.21.15.142:6838/2519143918 2024-03-31T23:13:32.151 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:32.142+0000 7f5b9776f740 1 -- 172.21.15.46:0/1336590757 >> 172.21.15.46:0/1336590757 conn(0x556040d66fe0 msgr2=0x556040d68d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:13:32.151 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:32.142+0000 7f5b9776f740 1 -- 172.21.15.46:0/1336590757 shutdown_connections 2024-03-31T23:13:32.151 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:32.142+0000 7f5b9776f740 1 -- 172.21.15.46:0/1336590757 wait complete. 2024-03-31T23:13:32.151 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:32.142+0000 7f5b9776f740 20 client.16215 trim_cache size 0 max 16384 2024-03-31T23:13:32.165 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:32.160+0000 7f5b9776f740 1 Processor -- start 2024-03-31T23:13:32.165 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:32.160+0000 7f5b9776f740 1 -- start start 2024-03-31T23:13:32.165 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:32.160+0000 7f5b9776f740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x556040d6c6c0 0x556040d75580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:13:32.165 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:32.160+0000 7f5b9776f740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556040d6cfe0 0x556040d81780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:13:32.165 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:32.160+0000 7f5b9776f740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556040d75db0 0x556040d8db40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:13:32.165 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:32.160+0000 7f5b9776f740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f5b901b3690 con 0x556040d6cfe0 2024-03-31T23:13:32.165 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:32.160+0000 7f5b9776f740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f5b901b0f60 con 0x556040d6c6c0 2024-03-31T23:13:32.165 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:32.160+0000 7f5b9776f740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f5b901aedb0 con 0x556040d75db0 2024-03-31T23:13:32.166 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:32.160+0000 7f5b8dd9b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556040d6cfe0 0x556040d81780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:13:32.166 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:32.160+0000 7f5b8dd9b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556040d6cfe0 0x556040d81780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:45870/0 (socket says 172.21.15.46:45870) 2024-03-31T23:13:32.166 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:32.160+0000 7f5b8dd9b640 1 -- 172.21.15.46:0/64070491 learned_addr learned my addr 172.21.15.46:0/64070491 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:13:32.166 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:32.160+0000 7f5b8ed9d640 1 --2- 172.21.15.46:0/64070491 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556040d75db0 0x556040d8db40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:13:32.166 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:32.160+0000 7f5b8e59c640 1 --2- 172.21.15.46:0/64070491 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x556040d6c6c0 0x556040d75580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:13:32.166 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:32.160+0000 7f5b8dd9b640 1 -- 172.21.15.46:0/64070491 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556040d75db0 msgr2=0x556040d8db40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:13:32.166 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:32.160+0000 7f5b8dd9b640 1 --2- 172.21.15.46:0/64070491 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556040d75db0 0x556040d8db40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:32.166 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:32.160+0000 7f5b8dd9b640 1 -- 172.21.15.46:0/64070491 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x556040d6c6c0 msgr2=0x556040d75580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:13:32.166 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:32.160+0000 7f5b8dd9b640 1 --2- 172.21.15.46:0/64070491 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x556040d6c6c0 0x556040d75580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:32.166 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:32.160+0000 7f5b8dd9b640 1 -- 172.21.15.46:0/64070491 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556040d78510 con 0x556040d6cfe0 2024-03-31T23:13:32.166 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:32.160+0000 7f5b8ed9d640 1 --2- 172.21.15.46:0/64070491 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556040d75db0 0x556040d8db40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:13:32.166 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:32.160+0000 7f5b8dd9b640 1 --2- 172.21.15.46:0/64070491 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556040d6cfe0 0x556040d81780 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f5b84009f90 tx=0x7f5b84005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=95d11d2a50619301 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:13:32.166 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:32.161+0000 7f5b77fff640 1 -- 172.21.15.46:0/64070491 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b8407e070 con 0x556040d6cfe0 2024-03-31T23:13:32.166 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:32.161+0000 7f5b77fff640 1 -- 172.21.15.46:0/64070491 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5b840b9070 con 0x556040d6cfe0 2024-03-31T23:13:32.167 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:32.161+0000 7f5b77fff640 1 -- 172.21.15.46:0/64070491 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b84020040 con 0x556040d6cfe0 2024-03-31T23:13:32.167 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:32.161+0000 7f5b9776f740 1 -- 172.21.15.46:0/64070491 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556040d6cfe0 msgr2=0x556040d81780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:13:32.167 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:32.161+0000 7f5b9776f740 1 --2- 172.21.15.46:0/64070491 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556040d6cfe0 0x556040d81780 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f5b84009f90 tx=0x7f5b84005880 comp rx=0 tx=0).stop 2024-03-31T23:13:32.167 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:32.161+0000 7f5b9776f740 1 -- 172.21.15.46:0/64070491 shutdown_connections 2024-03-31T23:13:32.167 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:32.161+0000 7f5b9776f740 1 --2- 172.21.15.46:0/64070491 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556040d75db0 0x556040d8db40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:32.167 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:32.161+0000 7f5b9776f740 1 --2- 172.21.15.46:0/64070491 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556040d6cfe0 0x556040d81780 unknown :-1 s=CLOSED pgs=620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:32.167 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:32.161+0000 7f5b9776f740 1 --2- 172.21.15.46:0/64070491 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x556040d6c6c0 0x556040d75580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:32.167 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:32.161+0000 7f5b9776f740 1 -- 172.21.15.46:0/64070491 >> 172.21.15.46:0/64070491 conn(0x556040d66fe0 msgr2=0x556040d6d670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:13:32.167 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:32.161+0000 7f5b9776f740 1 -- 172.21.15.46:0/64070491 shutdown_connections 2024-03-31T23:13:32.167 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:32.161+0000 7f5b9776f740 1 -- 172.21.15.46:0/64070491 wait complete. 2024-03-31T23:13:32.167 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:32.162+0000 7f5b9776f740 1 -- start start 2024-03-31T23:13:32.167 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:32.162+0000 7f5b9776f740 1 Processor -- start 2024-03-31T23:13:32.167 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:32.162+0000 7f5b9776f740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556040d6c6c0 0x556040d6fd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:13:32.168 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:32.162+0000 7f5b9776f740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556040d6cfe0 0x556040d702d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:13:32.168 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:32.162+0000 7f5b9776f740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x556040d75db0 0x556040deb170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:13:32.168 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:32.162+0000 7f5b9776f740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f5b901af420 con 0x556040d6cfe0 2024-03-31T23:13:32.168 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:32.162+0000 7f5b9776f740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f5b901b0890 con 0x556040d75db0 2024-03-31T23:13:32.168 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:32.162+0000 7f5b9776f740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f5b901afaf0 con 0x556040d6c6c0 2024-03-31T23:13:32.168 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:32.162+0000 7f5b9776f740 10 client.0 _ll_register_callbacks cb 0x7f5b9006b0a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T23:13:32.168 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:32.162+0000 7f5b8dd9b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556040d6cfe0 0x556040d702d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:13:32.168 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:32.162+0000 7f5b8dd9b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556040d6cfe0 0x556040d702d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:45874/0 (socket says 172.21.15.46:45874) 2024-03-31T23:13:32.168 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:32.162+0000 7f5b8dd9b640 1 -- 172.21.15.46:0/895016360 learned_addr learned my addr 172.21.15.46:0/895016360 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:13:32.168 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:32.162+0000 7f5b8ed9d640 1 --2- 172.21.15.46:0/895016360 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x556040d75db0 0x556040deb170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:13:32.168 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:32.162+0000 7f5b8e59c640 1 --2- 172.21.15.46:0/895016360 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556040d6c6c0 0x556040d6fd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:13:32.168 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:32.162+0000 7f5b8dd9b640 1 -- 172.21.15.46:0/895016360 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556040d6c6c0 msgr2=0x556040d6fd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:13:32.168 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:32.162+0000 7f5b8dd9b640 1 --2- 172.21.15.46:0/895016360 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556040d6c6c0 0x556040d6fd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:32.168 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:32.162+0000 7f5b8dd9b640 1 -- 172.21.15.46:0/895016360 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x556040d75db0 msgr2=0x556040deb170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:13:32.168 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:32.162+0000 7f5b8dd9b640 1 --2- 172.21.15.46:0/895016360 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x556040d75db0 0x556040deb170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:32.169 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:32.162+0000 7f5b8dd9b640 1 -- 172.21.15.46:0/895016360 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556040d70b70 con 0x556040d6cfe0 2024-03-31T23:13:32.169 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:32.162+0000 7f5b8e59c640 1 --2- 172.21.15.46:0/895016360 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556040d6c6c0 0x556040d6fd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T23:13:32.169 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:32.162+0000 7f5b8dd9b640 1 --2- 172.21.15.46:0/895016360 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556040d6cfe0 0x556040d702d0 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f5b84097900 tx=0x7f5b8409aff0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f619bc2a4ebdf25c server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:13:32.169 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:32.162+0000 7f5b75ffb640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-31T23:13:32.169 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:32.163+0000 7f5b75ffb640 1 -- 172.21.15.46:0/895016360 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b8407e030 con 0x556040d6cfe0 2024-03-31T23:13:32.169 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:32.163+0000 7f5b75ffb640 1 -- 172.21.15.46:0/895016360 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7f5b840a3b90 con 0x556040d6cfe0 2024-03-31T23:13:32.169 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:32.163+0000 7f5b9776f740 1 -- 172.21.15.46:0/895016360 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5b840b9070 con 0x556040d6cfe0 2024-03-31T23:13:32.169 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:32.163+0000 7f5b75ffb640 1 -- 172.21.15.46:0/895016360 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f5b84020070 con 0x556040d6cfe0 2024-03-31T23:13:32.169 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:32.163+0000 7f5b9776f740 10 client.16221 fetch_fsmap learned FSMap version 28 2024-03-31T23:13:32.169 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:32.163+0000 7f5b9776f740 1 -- 172.21.15.46:0/895016360 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x556040dec790 con 0x556040d6cfe0 2024-03-31T23:13:32.169 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:32.163+0000 7f5b75ffb640 1 -- 172.21.15.46:0/895016360 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5b840b9070 con 0x556040d6cfe0 2024-03-31T23:13:32.169 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:32.163+0000 7f5b75ffb640 1 -- 172.21.15.46:0/895016360 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f5b84010460 con 0x556040d6cfe0 2024-03-31T23:13:32.169 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:32.163+0000 7f5b9776f740 10 client.16221 fetch_fsmap finished waiting for FSMap version 28 2024-03-31T23:13:32.170 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:32.163+0000 7f5b9776f740 10 client.16221 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-31T23:13:32.170 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:32.163+0000 7f5b9776f740 20 client.16221 populate_metadata read hostname 'smithi046' 2024-03-31T23:13:32.170 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:32.163+0000 7f5b9776f740 1 --2- 172.21.15.46:0/895016360 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x556040dec9c0 0x556040deedb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:13:32.170 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:32.163+0000 7f5b9776f740 4 client.16221 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-31T23:13:32.170 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:32.163+0000 7f5b9776f740 1 -- 172.21.15.46:0/895016360 --> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] -- command(tid 0: {"prefix": "flush journal"}) -- 0x556040cdce30 con 0x556040dec9c0 2024-03-31T23:13:32.170 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:32.163+0000 7f5b8e59c640 1 --2- 172.21.15.46:0/895016360 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x556040dec9c0 0x556040deedb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:13:32.170 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:32.164+0000 7f5b75ffb640 10 client.16221 ms_handle_connect on v2:172.21.15.142:6838/2519143918 2024-03-31T23:13:32.170 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:32.164+0000 7f5b8e59c640 1 --2- 172.21.15.46:0/895016360 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x556040dec9c0 0x556040deedb0 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b46888822414c1d2 server_cookie=63ab33036a98f658 in_seq=0 out_seq=0 2024-03-31T23:13:32.189 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:32.187+0000 7f5b75ffb640 1 -- 172.21.15.46:0/895016360 <== mds.0 v2:172.21.15.142:6838/2519143918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x556040cdce30 con 0x556040dec9c0 2024-03-31T23:13:32.189 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:32.187+0000 7f5b75ffb640 10 client.16221 handle_command_reply: tid=0 2024-03-31T23:13:32.189 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:32.187+0000 7f5b9776f740 1 client.16221 shutdown 2024-03-31T23:13:32.189 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:32.188+0000 7f5b9776f740 1 -- 172.21.15.46:0/895016360 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556040d6cfe0 msgr2=0x556040d702d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:13:32.189 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:32.188+0000 7f5b9776f740 1 --2- 172.21.15.46:0/895016360 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556040d6cfe0 0x556040d702d0 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f5b84097900 tx=0x7f5b8409aff0 comp rx=0 tx=0).stop 2024-03-31T23:13:32.189 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:32.188+0000 7f5b9776f740 1 -- 172.21.15.46:0/895016360 shutdown_connections 2024-03-31T23:13:32.190 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:32.188+0000 7f5b9776f740 1 --2- 172.21.15.46:0/895016360 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x556040dec9c0 0x556040deedb0 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:32.190 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:32.188+0000 7f5b9776f740 1 --2- 172.21.15.46:0/895016360 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x556040d75db0 0x556040deb170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:32.190 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:32.188+0000 7f5b9776f740 1 --2- 172.21.15.46:0/895016360 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556040d6cfe0 0x556040d702d0 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:32.190 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:32.188+0000 7f5b9776f740 1 --2- 172.21.15.46:0/895016360 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556040d6c6c0 0x556040d6fd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:32.190 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:32.188+0000 7f5b75ffb640 0 client.16221 ms_handle_reset on v2:172.21.15.142:6838/2519143918 2024-03-31T23:13:32.190 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:32.188+0000 7f5b9776f740 1 -- 172.21.15.46:0/895016360 >> 172.21.15.46:0/895016360 conn(0x556040d66fe0 msgr2=0x556040d656f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:13:32.190 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:32.188+0000 7f5b9776f740 1 -- 172.21.15.46:0/895016360 shutdown_connections 2024-03-31T23:13:32.190 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:32.188+0000 7f5b9776f740 1 -- 172.21.15.46:0/895016360 wait complete. 2024-03-31T23:13:32.190 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:32.188+0000 7f5b9776f740 20 client.16221 trim_cache size 0 max 16384 2024-03-31T23:13:32.190 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout:{ 2024-03-31T23:13:32.191 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout: "message": "", 2024-03-31T23:13:32.191 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout: "return_code": 0 2024-03-31T23:13:32.191 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout:} 2024-03-31T23:13:32.191 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:32.188+0000 7f5b94aae640 1 -- 172.21.15.46:0/3431924280 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f5b780a26b0 msgr2=0x7f5b780a4b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:13:32.192 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:32.188+0000 7f5b94aae640 1 --2- 172.21.15.46:0/3431924280 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f5b780a26b0 0x7f5b780a4b70 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f5b88009fd0 tx=0x7f5b88015ca0 comp rx=0 tx=0).stop 2024-03-31T23:13:32.192 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:32.188+0000 7f5b94aae640 1 -- 172.21.15.46:0/3431924280 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5b90101260 msgr2=0x7f5b90102800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:13:32.192 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:32.188+0000 7f5b94aae640 1 --2- 172.21.15.46:0/3431924280 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5b90101260 0x7f5b90102800 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f5b8400d950 tx=0x7f5b8400de10 comp rx=0 tx=0).stop 2024-03-31T23:13:32.192 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:32.189+0000 7f5b94aae640 1 -- 172.21.15.46:0/3431924280 shutdown_connections 2024-03-31T23:13:32.192 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:32.189+0000 7f5b94aae640 1 --2- 172.21.15.46:0/3431924280 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f5b780a26b0 0x7f5b780a4b70 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:32.192 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:32.189+0000 7f5b94aae640 1 --2- 172.21.15.46:0/3431924280 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5b901055f0 0x7f5b90102d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:32.192 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:32.189+0000 7f5b94aae640 1 --2- 172.21.15.46:0/3431924280 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5b90101260 0x7f5b90102800 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:32.192 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:32.189+0000 7f5b94aae640 1 --2- 172.21.15.46:0/3431924280 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5b901008a0 0x7f5b901022c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:32.192 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:32.189+0000 7f5b94aae640 1 -- 172.21.15.46:0/3431924280 >> 172.21.15.46:0/3431924280 conn(0x7f5b900774a0 msgr2=0x7f5b900690b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:13:32.192 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:32.189+0000 7f5b94aae640 1 -- 172.21.15.46:0/3431924280 shutdown_connections 2024-03-31T23:13:32.192 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:32.189+0000 7f5b94aae640 1 -- 172.21.15.46:0/3431924280 wait complete. 2024-03-31T23:13:32.192 INFO:tasks.workunit.client.0.smithi046.stdout: 1 14852 2.57 MB/sec warmup 38 sec latency 17.085 ms 2024-03-31T23:13:32.795 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:13:32 smithi162 ceph-mon[36237]: pgmap v57: 129 pgs: 129 active+clean; 30 MiB data, 479 MiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 358 KiB/s wr, 49 op/s 2024-03-31T23:13:32.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:13:32 smithi046 ceph-mon[32498]: pgmap v57: 129 pgs: 129 active+clean; 30 MiB data, 479 MiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 358 KiB/s wr, 49 op/s 2024-03-31T23:13:32.945 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:13:32 smithi142 ceph-mon[35271]: pgmap v57: 129 pgs: 129 active+clean; 30 MiB data, 479 MiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 358 KiB/s wr, 49 op/s 2024-03-31T23:13:33.191 INFO:tasks.workunit.client.0.smithi046.stdout: 1 15338 2.55 MB/sec warmup 39 sec latency 20.037 ms 2024-03-31T23:13:34.191 INFO:tasks.workunit.client.0.smithi046.stdout: 1 15887 2.57 MB/sec warmup 40 sec latency 16.335 ms 2024-03-31T23:13:34.795 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:13:34 smithi162 ceph-mon[36237]: pgmap v58: 129 pgs: 129 active+clean; 30 MiB data, 479 MiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 358 KiB/s wr, 49 op/s 2024-03-31T23:13:34.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:13:34 smithi046 ceph-mon[32498]: pgmap v58: 129 pgs: 129 active+clean; 30 MiB data, 479 MiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 358 KiB/s wr, 49 op/s 2024-03-31T23:13:34.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:13:34 smithi142 ceph-mon[35271]: pgmap v58: 129 pgs: 129 active+clean; 30 MiB data, 479 MiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 358 KiB/s wr, 49 op/s 2024-03-31T23:13:35.192 INFO:tasks.workunit.client.0.smithi046.stdout: 1 16538 2.54 MB/sec warmup 41 sec latency 18.513 ms 2024-03-31T23:13:36.191 INFO:tasks.workunit.client.0.smithi046.stdout: 1 17117 2.62 MB/sec warmup 42 sec latency 15.556 ms 2024-03-31T23:13:36.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:13:36 smithi162 ceph-mon[36237]: pgmap v59: 129 pgs: 129 active+clean; 33 MiB data, 480 MiB used, 1.0 TiB / 1.0 TiB avail; 199 KiB/s rd, 964 KiB/s wr, 111 op/s 2024-03-31T23:13:36.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:13:36 smithi046 ceph-mon[32498]: pgmap v59: 129 pgs: 129 active+clean; 33 MiB data, 480 MiB used, 1.0 TiB / 1.0 TiB avail; 199 KiB/s rd, 964 KiB/s wr, 111 op/s 2024-03-31T23:13:36.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:13:36 smithi142 ceph-mon[35271]: pgmap v59: 129 pgs: 129 active+clean; 33 MiB data, 480 MiB used, 1.0 TiB / 1.0 TiB avail; 199 KiB/s rd, 964 KiB/s wr, 111 op/s 2024-03-31T23:13:37.191 INFO:tasks.workunit.client.0.smithi046.stdout: 1 17566 2.60 MB/sec warmup 43 sec latency 16.919 ms 2024-03-31T23:13:38.191 INFO:tasks.workunit.client.0.smithi046.stdout: 1 18039 2.64 MB/sec warmup 44 sec latency 17.295 ms 2024-03-31T23:13:38.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:13:38 smithi162 ceph-mon[36237]: pgmap v60: 129 pgs: 129 active+clean; 33 MiB data, 480 MiB used, 1.0 TiB / 1.0 TiB avail; 177 KiB/s rd, 932 KiB/s wr, 94 op/s 2024-03-31T23:13:38.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:13:38 smithi046 ceph-mon[32498]: pgmap v60: 129 pgs: 129 active+clean; 33 MiB data, 480 MiB used, 1.0 TiB / 1.0 TiB avail; 177 KiB/s rd, 932 KiB/s wr, 94 op/s 2024-03-31T23:13:38.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:13:38 smithi142 ceph-mon[35271]: pgmap v60: 129 pgs: 129 active+clean; 33 MiB data, 480 MiB used, 1.0 TiB / 1.0 TiB avail; 177 KiB/s rd, 932 KiB/s wr, 94 op/s 2024-03-31T23:13:39.191 INFO:tasks.workunit.client.0.smithi046.stdout: 1 18308 2.59 MB/sec warmup 45 sec latency 15.118 ms 2024-03-31T23:13:40.191 INFO:tasks.workunit.client.0.smithi046.stdout: 1 18762 2.56 MB/sec warmup 46 sec latency 16.252 ms 2024-03-31T23:13:40.795 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:13:40 smithi162 ceph-mon[36237]: pgmap v61: 129 pgs: 129 active+clean; 34 MiB data, 499 MiB used, 1.0 TiB / 1.0 TiB avail; 288 KiB/s rd, 1.5 MiB/s wr, 152 op/s 2024-03-31T23:13:40.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:13:40 smithi046 ceph-mon[32498]: pgmap v61: 129 pgs: 129 active+clean; 34 MiB data, 499 MiB used, 1.0 TiB / 1.0 TiB avail; 288 KiB/s rd, 1.5 MiB/s wr, 152 op/s 2024-03-31T23:13:40.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:13:40 smithi142 ceph-mon[35271]: pgmap v61: 129 pgs: 129 active+clean; 34 MiB data, 499 MiB used, 1.0 TiB / 1.0 TiB avail; 288 KiB/s rd, 1.5 MiB/s wr, 152 op/s 2024-03-31T23:13:41.191 INFO:tasks.workunit.client.0.smithi046.stdout: 1 19281 2.59 MB/sec warmup 47 sec latency 20.049 ms 2024-03-31T23:13:41.823 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-31T23:13:41.823 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-31T23:13:41.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:41.938+0000 7fc82d6c0640 1 -- 172.21.15.46:0/4261297509 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc828069310 msgr2=0x7fc82810e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:13:41.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:41.938+0000 7fc82d6c0640 1 --2- 172.21.15.46:0/4261297509 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc828069310 0x7fc82810e5b0 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7fc820009cd0 tx=0x7fc82002c0f0 comp rx=0 tx=0).stop 2024-03-31T23:13:41.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:41.938+0000 7fc82d6c0640 1 -- 172.21.15.46:0/4261297509 shutdown_connections 2024-03-31T23:13:41.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:41.938+0000 7fc82d6c0640 1 --2- 172.21.15.46:0/4261297509 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc828069310 0x7fc82810e5b0 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:41.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:41.938+0000 7fc82d6c0640 1 --2- 172.21.15.46:0/4261297509 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc828068950 0x7fc828068dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:41.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:41.938+0000 7fc82d6c0640 1 --2- 172.21.15.46:0/4261297509 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc8281077a0 0x7fc828107b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:41.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:41.938+0000 7fc82d6c0640 1 -- 172.21.15.46:0/4261297509 >> 172.21.15.46:0/4261297509 conn(0x7fc828077750 msgr2=0x7fc828077b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:13:41.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:41.938+0000 7fc82d6c0640 1 -- 172.21.15.46:0/4261297509 shutdown_connections 2024-03-31T23:13:41.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:41.938+0000 7fc82d6c0640 1 -- 172.21.15.46:0/4261297509 wait complete. 2024-03-31T23:13:41.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:41.939+0000 7fc82d6c0640 1 Processor -- start 2024-03-31T23:13:41.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:41.939+0000 7fc82d6c0640 1 -- start start 2024-03-31T23:13:41.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:41.939+0000 7fc82d6c0640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc828068950 0x7fc828197e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:13:41.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:41.939+0000 7fc82d6c0640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc828069310 0x7fc8281983c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:13:41.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:41.939+0000 7fc82d6c0640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc8281077a0 0x7fc82819c740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:13:41.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:41.939+0000 7fc82d6c0640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fc8281119a0 con 0x7fc828068950 2024-03-31T23:13:41.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:41.939+0000 7fc82d6c0640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fc828111820 con 0x7fc828069310 2024-03-31T23:13:41.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:41.939+0000 7fc82d6c0640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fc828111b20 con 0x7fc8281077a0 2024-03-31T23:13:41.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:41.939+0000 7fc826ffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc828068950 0x7fc828197e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:13:41.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:41.939+0000 7fc826ffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc828068950 0x7fc828197e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59966/0 (socket says 172.21.15.46:59966) 2024-03-31T23:13:41.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:41.939+0000 7fc826ffd640 1 -- 172.21.15.46:0/503404997 learned_addr learned my addr 172.21.15.46:0/503404997 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:13:41.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:41.939+0000 7fc8277fe640 1 --2- 172.21.15.46:0/503404997 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc8281077a0 0x7fc82819c740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:13:41.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:41.939+0000 7fc81e7fc640 1 --2- 172.21.15.46:0/503404997 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc828069310 0x7fc8281983c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:13:41.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:41.940+0000 7fc826ffd640 1 -- 172.21.15.46:0/503404997 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc8281077a0 msgr2=0x7fc82819c740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:13:41.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:41.940+0000 7fc826ffd640 1 --2- 172.21.15.46:0/503404997 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc8281077a0 0x7fc82819c740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:41.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:41.940+0000 7fc826ffd640 1 -- 172.21.15.46:0/503404997 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc828069310 msgr2=0x7fc8281983c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:13:41.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:41.940+0000 7fc826ffd640 1 --2- 172.21.15.46:0/503404997 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc828069310 0x7fc8281983c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:41.958 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:41.940+0000 7fc826ffd640 1 -- 172.21.15.46:0/503404997 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc82819ce70 con 0x7fc828068950 2024-03-31T23:13:41.958 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:41.940+0000 7fc8277fe640 1 --2- 172.21.15.46:0/503404997 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc8281077a0 0x7fc82819c740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:13:41.958 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:41.940+0000 7fc826ffd640 1 --2- 172.21.15.46:0/503404997 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc828068950 0x7fc828197e80 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7fc81800e9f0 tx=0x7fc81800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:13:41.958 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:41.940+0000 7fc824ff9640 1 -- 172.21.15.46:0/503404997 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc81800cd80 con 0x7fc828068950 2024-03-31T23:13:41.958 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:41.940+0000 7fc82d6c0640 1 -- 172.21.15.46:0/503404997 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc82819d160 con 0x7fc828068950 2024-03-31T23:13:41.958 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:41.940+0000 7fc824ff9640 1 -- 172.21.15.46:0/503404997 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc8180108e0 con 0x7fc828068950 2024-03-31T23:13:41.958 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:41.940+0000 7fc82d6c0640 1 -- 172.21.15.46:0/503404997 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc8281a49f0 con 0x7fc828068950 2024-03-31T23:13:41.958 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:41.940+0000 7fc824ff9640 1 -- 172.21.15.46:0/503404997 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8180188c0 con 0x7fc828068950 2024-03-31T23:13:41.958 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:41.943+0000 7fc830378740 1 Processor -- start 2024-03-31T23:13:41.958 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:41.943+0000 7fc824ff9640 1 -- 172.21.15.46:0/503404997 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 25) ==== 137451+0+0 (secure 0 0 0) 0x7fc818010a90 con 0x7fc828068950 2024-03-31T23:13:41.958 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:41.943+0000 7fc830378740 1 -- start start 2024-03-31T23:13:41.958 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:41.943+0000 7fc830378740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c7e5e13290 0x55c7e5e13670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:13:41.958 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:41.943+0000 7fc830378740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55c7e5e13bb0 0x55c7e5e1c250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:13:41.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:41.943+0000 7fc830378740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c7e5e1c790 0x55c7e5e1eb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:13:41.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:41.943+0000 7fc830378740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55c7e5d77180 con 0x55c7e5e13290 2024-03-31T23:13:41.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:41.943+0000 7fc830378740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55c7e5d1fb70 con 0x55c7e5e13bb0 2024-03-31T23:13:41.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:41.943+0000 7fc830378740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55c7e5d73220 con 0x55c7e5e1c790 2024-03-31T23:13:41.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:41.943+0000 7fc8277fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c7e5e13290 0x55c7e5e13670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:13:41.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:41.943+0000 7fc826ffd640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55c7e5e13bb0 0x55c7e5e1c250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:13:41.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:41.943+0000 7fc8277fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c7e5e13290 0x55c7e5e13670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59976/0 (socket says 172.21.15.46:59976) 2024-03-31T23:13:41.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:41.943+0000 7fc8277fe640 1 -- 172.21.15.46:0/3780139449 learned_addr learned my addr 172.21.15.46:0/3780139449 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:13:41.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:41.943+0000 7fc81e7fc640 1 --2- 172.21.15.46:0/3780139449 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c7e5e1c790 0x55c7e5e1eb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:13:41.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:41.944+0000 7fc824ff9640 1 --2- 172.21.15.46:0/503404997 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fc8140a2610 0x7fc8140a4ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:13:41.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:41.944+0000 7fc8277fe640 1 -- 172.21.15.46:0/3780139449 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c7e5e1c790 msgr2=0x55c7e5e1eb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:13:41.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:41.944+0000 7fc8277fe640 1 --2- 172.21.15.46:0/3780139449 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c7e5e1c790 0x55c7e5e1eb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:41.960 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:41.944+0000 7fc8277fe640 1 -- 172.21.15.46:0/3780139449 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55c7e5e13bb0 msgr2=0x55c7e5e1c250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:13:41.960 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:41.944+0000 7fc8277fe640 1 --2- 172.21.15.46:0/3780139449 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55c7e5e13bb0 0x55c7e5e1c250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:41.960 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:41.944+0000 7fc8277fe640 1 -- 172.21.15.46:0/3780139449 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c7e5d337c0 con 0x55c7e5e13290 2024-03-31T23:13:41.960 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:41.944+0000 7fc826ffd640 1 --2- 172.21.15.46:0/3780139449 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55c7e5e13bb0 0x55c7e5e1c250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:13:41.960 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:41.944+0000 7fc824ff9640 1 -- 172.21.15.46:0/503404997 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7fc8180cf190 con 0x7fc828068950 2024-03-31T23:13:41.960 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:41.944+0000 7fc8277fe640 1 --2- 172.21.15.46:0/3780139449 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c7e5e13290 0x55c7e5e13670 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7fc82002c700 tx=0x7fc820005f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=766a25b40bfbc3bd server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:13:41.960 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:41.944+0000 7fc8277fe640 1 --2- 172.21.15.46:0/503404997 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fc8140a2610 0x7fc8140a4ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:13:41.960 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:41.944+0000 7fc81d7fa640 1 -- 172.21.15.46:0/3780139449 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc820043070 con 0x55c7e5e13290 2024-03-31T23:13:41.960 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:41.944+0000 7fc81d7fa640 1 -- 172.21.15.46:0/3780139449 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc820003e20 con 0x55c7e5e13290 2024-03-31T23:13:41.960 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:41.944+0000 7fc81d7fa640 1 -- 172.21.15.46:0/3780139449 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc820039040 con 0x55c7e5e13290 2024-03-31T23:13:41.960 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:41.945+0000 7fc830378740 1 -- 172.21.15.46:0/3780139449 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c7e5e13290 msgr2=0x55c7e5e13670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:13:41.960 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:41.945+0000 7fc8277fe640 1 --2- 172.21.15.46:0/503404997 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fc8140a2610 0x7fc8140a4ad0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fc820045000 tx=0x7fc820048860 comp rx=0 tx=0).ready entity=mgr.15654 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:13:41.960 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:41.945+0000 7fc830378740 1 --2- 172.21.15.46:0/3780139449 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c7e5e13290 0x55c7e5e13670 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7fc82002c700 tx=0x7fc820005f80 comp rx=0 tx=0).stop 2024-03-31T23:13:41.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:41.945+0000 7fc830378740 1 -- 172.21.15.46:0/3780139449 shutdown_connections 2024-03-31T23:13:41.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:41.945+0000 7fc830378740 1 --2- 172.21.15.46:0/3780139449 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c7e5e1c790 0x55c7e5e1eb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:41.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:41.945+0000 7fc830378740 1 --2- 172.21.15.46:0/3780139449 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55c7e5e13bb0 0x55c7e5e1c250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:41.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:41.945+0000 7fc830378740 1 --2- 172.21.15.46:0/3780139449 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c7e5e13290 0x55c7e5e13670 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:41.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:41.945+0000 7fc830378740 1 -- 172.21.15.46:0/3780139449 >> 172.21.15.46:0/3780139449 conn(0x55c7e5e0d9c0 msgr2=0x55c7e5e0dda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:13:41.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:41.945+0000 7fc830378740 1 -- 172.21.15.46:0/3780139449 shutdown_connections 2024-03-31T23:13:41.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:41.945+0000 7fc830378740 1 -- 172.21.15.46:0/3780139449 wait complete. 2024-03-31T23:13:41.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:41.946+0000 7fc830378740 1 -- start start 2024-03-31T23:13:41.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:41.946+0000 7fc830378740 1 Processor -- start 2024-03-31T23:13:41.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:41.946+0000 7fc830378740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c7e5e13290 0x55c7e5e19830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:13:41.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:41.946+0000 7fc830378740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55c7e5e13bb0 0x55c7e5e19d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:13:41.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:41.946+0000 7fc830378740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c7e5e1c790 0x55c7e5e1a2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:13:41.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:41.946+0000 7fc830378740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55c7e5e1f520 con 0x55c7e5e1c790 2024-03-31T23:13:41.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:41.946+0000 7fc830378740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55c7e5d154a0 con 0x55c7e5e13bb0 2024-03-31T23:13:41.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:41.946+0000 7fc830378740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55c7e5e1f6a0 con 0x55c7e5e13290 2024-03-31T23:13:41.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:41.946+0000 7fc830378740 10 client.0 _ll_register_callbacks cb 0x55c7e5d18d90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T23:13:41.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:41.946+0000 7fc81e7fc640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55c7e5e13bb0 0x55c7e5e19d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:13:41.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:41.946+0000 7fc8277fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c7e5e1c790 0x55c7e5e1a2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:13:41.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:41.946+0000 7fc826ffd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c7e5e13290 0x55c7e5e19830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:13:41.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:41.946+0000 7fc826ffd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c7e5e13290 0x55c7e5e19830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.46:49008/0 (socket says 172.21.15.46:49008) 2024-03-31T23:13:41.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:41.946+0000 7fc826ffd640 1 -- 172.21.15.46:0/1896835152 learned_addr learned my addr 172.21.15.46:0/1896835152 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:13:41.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:41.946+0000 7fc81e7fc640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55c7e5e13bb0 0x55c7e5e19d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.46:48624/0 (socket says 172.21.15.46:48624) 2024-03-31T23:13:41.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:41.946+0000 7fc826ffd640 1 -- 172.21.15.46:0/1896835152 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55c7e5e13bb0 msgr2=0x55c7e5e19d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:13:41.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:41.946+0000 7fc826ffd640 1 --2- 172.21.15.46:0/1896835152 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55c7e5e13bb0 0x55c7e5e19d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:41.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:41.946+0000 7fc826ffd640 1 -- 172.21.15.46:0/1896835152 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c7e5e1c790 msgr2=0x55c7e5e1a2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:13:41.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:41.946+0000 7fc826ffd640 1 --2- 172.21.15.46:0/1896835152 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c7e5e1c790 0x55c7e5e1a2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:41.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:41.946+0000 7fc826ffd640 1 -- 172.21.15.46:0/1896835152 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c7e5b557d0 con 0x55c7e5e13290 2024-03-31T23:13:41.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:41.946+0000 7fc81e7fc640 1 --2- 172.21.15.46:0/1896835152 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55c7e5e13bb0 0x55c7e5e19d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:13:41.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:41.946+0000 7fc8277fe640 1 --2- 172.21.15.46:0/1896835152 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c7e5e1c790 0x55c7e5e1a2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:13:41.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:41.947+0000 7fc826ffd640 1 --2- 172.21.15.46:0/1896835152 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c7e5e13290 0x55c7e5e19830 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fc81809b310 tx=0x7fc8180cf390 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f5ffc8547f76f465 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:13:41.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:41.947+0000 7fc80e7fc640 10 client.0 ms_handle_connect on v2:172.21.15.162:3300/0 2024-03-31T23:13:41.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:41.947+0000 7fc80e7fc640 1 -- 172.21.15.46:0/1896835152 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc81809b3e0 con 0x55c7e5e13290 2024-03-31T23:13:41.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:41.947+0000 7fc80e7fc640 1 -- 172.21.15.46:0/1896835152 <== mon.1 v2:172.21.15.162:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7fc818084d50 con 0x55c7e5e13290 2024-03-31T23:13:41.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:41.947+0000 7fc830378740 1 -- 172.21.15.46:0/1896835152 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc820003e20 con 0x55c7e5e13290 2024-03-31T23:13:41.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:41.947+0000 7fc80e7fc640 1 -- 172.21.15.46:0/1896835152 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fc81809af70 con 0x55c7e5e13290 2024-03-31T23:13:41.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:41.947+0000 7fc830378740 10 client.25972 fetch_fsmap learned FSMap version 28 2024-03-31T23:13:41.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:41.947+0000 7fc830378740 1 -- 172.21.15.46:0/1896835152 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55c7e5a8b3f0 con 0x55c7e5e13290 2024-03-31T23:13:41.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:41.947+0000 7fc80e7fc640 1 -- 172.21.15.46:0/1896835152 <== mon.1 v2:172.21.15.162:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc820003e20 con 0x55c7e5e13290 2024-03-31T23:13:41.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:41.947+0000 7fc80e7fc640 1 -- 172.21.15.46:0/1896835152 <== mon.1 v2:172.21.15.162:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7fc818010430 con 0x55c7e5e13290 2024-03-31T23:13:41.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:41.947+0000 7fc830378740 10 client.25972 fetch_fsmap finished waiting for FSMap version 28 2024-03-31T23:13:41.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:41.947+0000 7fc830378740 10 client.25972 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-31T23:13:41.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:41.947+0000 7fc830378740 20 client.25972 populate_metadata read hostname 'smithi046' 2024-03-31T23:13:41.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:41.947+0000 7fc830378740 1 --2- 172.21.15.46:0/1896835152 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55c7e5e2cb00 0x55c7e5e2eef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:13:41.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:41.947+0000 7fc830378740 4 client.25972 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-31T23:13:41.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:41.947+0000 7fc830378740 1 -- 172.21.15.46:0/1896835152 --> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c7e5d846c0 con 0x55c7e5e2cb00 2024-03-31T23:13:41.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:41.948+0000 7fc81e7fc640 1 --2- 172.21.15.46:0/1896835152 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55c7e5e2cb00 0x55c7e5e2eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:13:41.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:41.948+0000 7fc80e7fc640 10 client.25972 ms_handle_connect on v2:172.21.15.142:6838/2519143918 2024-03-31T23:13:41.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:41.948+0000 7fc81e7fc640 1 --2- 172.21.15.46:0/1896835152 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55c7e5e2cb00 0x55c7e5e2eef0 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5c8f94643a71b818 server_cookie=d6a3dd416599e617 in_seq=0 out_seq=0 2024-03-31T23:13:41.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:41.949+0000 7fc80e7fc640 1 -- 172.21.15.46:0/1896835152 <== mds.0 v2:172.21.15.142:6838/2519143918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55c7e5d846c0 con 0x55c7e5e2cb00 2024-03-31T23:13:41.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:41.949+0000 7fc80e7fc640 10 client.25972 handle_command_reply: tid=0 2024-03-31T23:13:41.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:41.949+0000 7fc830378740 1 client.25972 shutdown 2024-03-31T23:13:41.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:41.949+0000 7fc830378740 1 -- 172.21.15.46:0/1896835152 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c7e5e13290 msgr2=0x55c7e5e19830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:13:41.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:41.949+0000 7fc830378740 1 --2- 172.21.15.46:0/1896835152 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c7e5e13290 0x55c7e5e19830 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fc81809b310 tx=0x7fc8180cf390 comp rx=0 tx=0).stop 2024-03-31T23:13:41.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:41.949+0000 7fc830378740 1 -- 172.21.15.46:0/1896835152 shutdown_connections 2024-03-31T23:13:41.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:41.949+0000 7fc830378740 1 --2- 172.21.15.46:0/1896835152 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55c7e5e2cb00 0x55c7e5e2eef0 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:41.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:41.949+0000 7fc830378740 1 --2- 172.21.15.46:0/1896835152 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c7e5e1c790 0x55c7e5e1a2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:41.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:41.949+0000 7fc80e7fc640 0 client.25972 ms_handle_reset on v2:172.21.15.142:6838/2519143918 2024-03-31T23:13:41.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:41.949+0000 7fc830378740 1 --2- 172.21.15.46:0/1896835152 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55c7e5e13bb0 0x55c7e5e19d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:41.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:41.949+0000 7fc830378740 1 --2- 172.21.15.46:0/1896835152 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c7e5e13290 0x55c7e5e19830 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:41.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:41.949+0000 7fc830378740 1 -- 172.21.15.46:0/1896835152 >> 172.21.15.46:0/1896835152 conn(0x55c7e5e0d9c0 msgr2=0x55c7e5e0cc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:13:41.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:41.949+0000 7fc830378740 1 -- 172.21.15.46:0/1896835152 shutdown_connections 2024-03-31T23:13:41.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:41.949+0000 7fc830378740 1 -- 172.21.15.46:0/1896835152 wait complete. 2024-03-31T23:13:41.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:41.949+0000 7fc830378740 20 client.25972 trim_cache size 0 max 16384 2024-03-31T23:13:41.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:41.967+0000 7fc830378740 1 Processor -- start 2024-03-31T23:13:41.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:41.967+0000 7fc830378740 1 -- start start 2024-03-31T23:13:41.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:41.968+0000 7fc830378740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c7e5e13290 0x55c7e5e1bef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:13:41.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:41.968+0000 7fc830378740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55c7e5e13bb0 0x55c7e5e18770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:13:41.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:41.968+0000 7fc830378740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c7e5e1c790 0x55c7e5e282c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:13:41.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:41.968+0000 7fc830378740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fc8281af2d0 con 0x55c7e5e13290 2024-03-31T23:13:41.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:41.968+0000 7fc830378740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fc8281acba0 con 0x55c7e5e13bb0 2024-03-31T23:13:41.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:41.968+0000 7fc830378740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fc8281aa9f0 con 0x55c7e5e1c790 2024-03-31T23:13:41.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:41.968+0000 7fc826ffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c7e5e13290 0x55c7e5e1bef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:13:41.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:41.968+0000 7fc826ffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c7e5e13290 0x55c7e5e1bef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59988/0 (socket says 172.21.15.46:59988) 2024-03-31T23:13:41.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:41.968+0000 7fc826ffd640 1 -- 172.21.15.46:0/1335244361 learned_addr learned my addr 172.21.15.46:0/1335244361 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:13:41.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:41.968+0000 7fc81e7fc640 1 --2- 172.21.15.46:0/1335244361 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55c7e5e13bb0 0x55c7e5e18770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:13:41.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:41.968+0000 7fc8277fe640 1 --2- 172.21.15.46:0/1335244361 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c7e5e1c790 0x55c7e5e282c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:13:41.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:41.968+0000 7fc826ffd640 1 -- 172.21.15.46:0/1335244361 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c7e5e1c790 msgr2=0x55c7e5e282c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:13:41.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:41.968+0000 7fc826ffd640 1 --2- 172.21.15.46:0/1335244361 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c7e5e1c790 0x55c7e5e282c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:41.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:41.968+0000 7fc826ffd640 1 -- 172.21.15.46:0/1335244361 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55c7e5e13bb0 msgr2=0x55c7e5e18770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:13:41.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:41.968+0000 7fc826ffd640 1 --2- 172.21.15.46:0/1335244361 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55c7e5e13bb0 0x55c7e5e18770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:41.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:41.968+0000 7fc826ffd640 1 -- 172.21.15.46:0/1335244361 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c7e5a4b0f0 con 0x55c7e5e13290 2024-03-31T23:13:41.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:41.968+0000 7fc826ffd640 1 --2- 172.21.15.46:0/1335244361 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c7e5e13290 0x55c7e5e1bef0 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7fc81809b5a0 tx=0x7fc8180cc5e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a4a9902e61d4e server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:13:41.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:41.969+0000 7fc81cff9640 1 -- 172.21.15.46:0/1335244361 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc81809b150 con 0x55c7e5e13290 2024-03-31T23:13:41.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:41.969+0000 7fc81cff9640 1 -- 172.21.15.46:0/1335244361 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc818097040 con 0x55c7e5e13290 2024-03-31T23:13:41.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:41.969+0000 7fc81cff9640 1 -- 172.21.15.46:0/1335244361 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc818010430 con 0x55c7e5e13290 2024-03-31T23:13:41.976 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:41.969+0000 7fc830378740 1 -- 172.21.15.46:0/1335244361 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c7e5e13290 msgr2=0x55c7e5e1bef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:13:41.976 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:41.969+0000 7fc830378740 1 --2- 172.21.15.46:0/1335244361 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c7e5e13290 0x55c7e5e1bef0 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7fc81809b5a0 tx=0x7fc8180cc5e0 comp rx=0 tx=0).stop 2024-03-31T23:13:41.977 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:41.969+0000 7fc830378740 1 -- 172.21.15.46:0/1335244361 shutdown_connections 2024-03-31T23:13:41.977 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:41.969+0000 7fc830378740 1 --2- 172.21.15.46:0/1335244361 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c7e5e1c790 0x55c7e5e282c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:41.977 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:41.969+0000 7fc830378740 1 --2- 172.21.15.46:0/1335244361 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55c7e5e13bb0 0x55c7e5e18770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:41.977 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:41.969+0000 7fc830378740 1 --2- 172.21.15.46:0/1335244361 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c7e5e13290 0x55c7e5e1bef0 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:41.977 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:41.969+0000 7fc830378740 1 -- 172.21.15.46:0/1335244361 >> 172.21.15.46:0/1335244361 conn(0x55c7e5e0d9c0 msgr2=0x55c7e5e0cc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:13:41.977 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:41.969+0000 7fc830378740 1 -- 172.21.15.46:0/1335244361 shutdown_connections 2024-03-31T23:13:41.977 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:41.969+0000 7fc830378740 1 -- 172.21.15.46:0/1335244361 wait complete. 2024-03-31T23:13:41.977 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:41.970+0000 7fc830378740 1 -- start start 2024-03-31T23:13:41.977 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:41.970+0000 7fc830378740 1 Processor -- start 2024-03-31T23:13:41.977 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:41.970+0000 7fc830378740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55c7e5e13290 0x55c7e5e24d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:13:41.977 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:41.970+0000 7fc830378740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c7e5e13bb0 0x55c7e5e20110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:13:41.977 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:41.970+0000 7fc830378740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c7e5e1c790 0x55c7e5e20850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:13:41.977 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:41.970+0000 7fc830378740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fc8281ab060 con 0x55c7e5e13bb0 2024-03-31T23:13:41.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:41.970+0000 7fc830378740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fc8281ac4d0 con 0x55c7e5e13290 2024-03-31T23:13:41.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:41.970+0000 7fc830378740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fc8281ab730 con 0x55c7e5e1c790 2024-03-31T23:13:41.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:41.970+0000 7fc830378740 10 client.0 _ll_register_callbacks cb 0x7fc81800cf20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T23:13:41.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:41.970+0000 7fc81e7fc640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c7e5e13bb0 0x55c7e5e20110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:13:41.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:41.970+0000 7fc81e7fc640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c7e5e13bb0 0x55c7e5e20110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60002/0 (socket says 172.21.15.46:60002) 2024-03-31T23:13:41.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:41.970+0000 7fc81e7fc640 1 -- 172.21.15.46:0/2772074366 learned_addr learned my addr 172.21.15.46:0/2772074366 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:13:41.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:41.970+0000 7fc826ffd640 1 --2- 172.21.15.46:0/2772074366 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55c7e5e13290 0x55c7e5e24d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:13:41.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:41.970+0000 7fc8277fe640 1 --2- 172.21.15.46:0/2772074366 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c7e5e1c790 0x55c7e5e20850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:13:41.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:41.971+0000 7fc81e7fc640 1 -- 172.21.15.46:0/2772074366 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c7e5e1c790 msgr2=0x55c7e5e20850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:13:41.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:41.971+0000 7fc81e7fc640 1 --2- 172.21.15.46:0/2772074366 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c7e5e1c790 0x55c7e5e20850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:41.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:41.971+0000 7fc81e7fc640 1 -- 172.21.15.46:0/2772074366 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55c7e5e13290 msgr2=0x55c7e5e24d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:13:41.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:41.971+0000 7fc81e7fc640 1 --2- 172.21.15.46:0/2772074366 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55c7e5e13290 0x55c7e5e24d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:41.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:41.971+0000 7fc81e7fc640 1 -- 172.21.15.46:0/2772074366 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c7e5e91e30 con 0x55c7e5e13bb0 2024-03-31T23:13:41.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:41.971+0000 7fc826ffd640 1 --2- 172.21.15.46:0/2772074366 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55c7e5e13290 0x55c7e5e24d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:13:41.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:41.971+0000 7fc81e7fc640 1 --2- 172.21.15.46:0/2772074366 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c7e5e13bb0 0x55c7e5e20110 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x55c7e5e19090 tx=0x7fc810012ae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b82b46abe1af3bd2 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:13:41.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:41.971+0000 7fc80e7fc640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-31T23:13:41.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:41.971+0000 7fc80e7fc640 1 -- 172.21.15.46:0/2772074366 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc810020040 con 0x55c7e5e13bb0 2024-03-31T23:13:41.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:41.971+0000 7fc80e7fc640 1 -- 172.21.15.46:0/2772074366 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7fc8100201e0 con 0x55c7e5e13bb0 2024-03-31T23:13:41.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:41.971+0000 7fc830378740 1 -- 172.21.15.46:0/2772074366 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc818097040 con 0x55c7e5e13bb0 2024-03-31T23:13:41.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:41.971+0000 7fc80e7fc640 1 -- 172.21.15.46:0/2772074366 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fc810031050 con 0x55c7e5e13bb0 2024-03-31T23:13:41.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:41.971+0000 7fc830378740 10 client.16239 fetch_fsmap learned FSMap version 28 2024-03-31T23:13:41.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:41.971+0000 7fc830378740 1 -- 172.21.15.46:0/2772074366 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55c7e5e93090 con 0x55c7e5e13bb0 2024-03-31T23:13:41.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:41.971+0000 7fc80e7fc640 1 -- 172.21.15.46:0/2772074366 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc818097040 con 0x55c7e5e13bb0 2024-03-31T23:13:41.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:41.971+0000 7fc80e7fc640 1 -- 172.21.15.46:0/2772074366 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7fc810010430 con 0x55c7e5e13bb0 2024-03-31T23:13:41.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:41.972+0000 7fc830378740 10 client.16239 fetch_fsmap finished waiting for FSMap version 28 2024-03-31T23:13:41.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:41.972+0000 7fc830378740 10 client.16239 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-31T23:13:41.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:41.972+0000 7fc830378740 20 client.16239 populate_metadata read hostname 'smithi046' 2024-03-31T23:13:41.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:41.972+0000 7fc830378740 1 --2- 172.21.15.46:0/2772074366 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55c7e5e938f0 0x55c7e5e95d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:13:41.980 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:41.972+0000 7fc830378740 4 client.16239 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-31T23:13:41.980 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:41.972+0000 7fc830378740 1 -- 172.21.15.46:0/2772074366 --> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55c7e5d7ea10 con 0x55c7e5e938f0 2024-03-31T23:13:41.980 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:41.972+0000 7fc826ffd640 1 --2- 172.21.15.46:0/2772074366 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55c7e5e938f0 0x55c7e5e95d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:13:41.980 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:41.972+0000 7fc80e7fc640 10 client.16239 ms_handle_connect on v2:172.21.15.142:6838/2519143918 2024-03-31T23:13:41.980 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:41.972+0000 7fc826ffd640 1 --2- 172.21.15.46:0/2772074366 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55c7e5e938f0 0x55c7e5e95d10 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b2ad1218364c890d server_cookie=c92654d0c27e9496 in_seq=0 out_seq=0 2024-03-31T23:13:41.980 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:41.972+0000 7fc80e7fc640 1 -- 172.21.15.46:0/2772074366 <== mds.0 v2:172.21.15.142:6838/2519143918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55c7e5d7ea10 con 0x55c7e5e938f0 2024-03-31T23:13:41.980 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:41.972+0000 7fc80e7fc640 10 client.16239 handle_command_reply: tid=0 2024-03-31T23:13:41.980 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:41.972+0000 7fc830378740 1 client.16239 shutdown 2024-03-31T23:13:41.980 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:41.972+0000 7fc830378740 1 -- 172.21.15.46:0/2772074366 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c7e5e13bb0 msgr2=0x55c7e5e20110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:13:41.980 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:41.972+0000 7fc830378740 1 --2- 172.21.15.46:0/2772074366 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c7e5e13bb0 0x55c7e5e20110 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x55c7e5e19090 tx=0x7fc810012ae0 comp rx=0 tx=0).stop 2024-03-31T23:13:41.980 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:41.972+0000 7fc830378740 1 -- 172.21.15.46:0/2772074366 shutdown_connections 2024-03-31T23:13:41.980 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:41.972+0000 7fc830378740 1 --2- 172.21.15.46:0/2772074366 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55c7e5e938f0 0x55c7e5e95d10 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:41.980 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:41.972+0000 7fc830378740 1 --2- 172.21.15.46:0/2772074366 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c7e5e1c790 0x55c7e5e20850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:41.980 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:41.972+0000 7fc80e7fc640 0 client.16239 ms_handle_reset on v2:172.21.15.142:6838/2519143918 2024-03-31T23:13:41.981 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:41.972+0000 7fc830378740 1 --2- 172.21.15.46:0/2772074366 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c7e5e13bb0 0x55c7e5e20110 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:41.981 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:41.972+0000 7fc830378740 1 --2- 172.21.15.46:0/2772074366 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55c7e5e13290 0x55c7e5e24d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:41.981 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:41.972+0000 7fc830378740 1 -- 172.21.15.46:0/2772074366 >> 172.21.15.46:0/2772074366 conn(0x55c7e5e0d9c0 msgr2=0x55c7e5e197f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:13:41.981 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:41.973+0000 7fc830378740 1 -- 172.21.15.46:0/2772074366 shutdown_connections 2024-03-31T23:13:41.981 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:41.973+0000 7fc830378740 1 -- 172.21.15.46:0/2772074366 wait complete. 2024-03-31T23:13:41.981 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:41.973+0000 7fc830378740 20 client.16239 trim_cache size 0 max 16384 2024-03-31T23:13:41.981 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-03-31T23:13:41.982 INFO:teuthology.orchestra.run.smithi046.stdout: "status": "no active scrubs running", 2024-03-31T23:13:41.982 INFO:teuthology.orchestra.run.smithi046.stdout: "scrubs": {} 2024-03-31T23:13:41.982 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-03-31T23:13:41.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:41.973+0000 7fc82d6c0640 1 -- 172.21.15.46:0/503404997 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fc8140a2610 msgr2=0x7fc8140a4ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:13:41.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:41.973+0000 7fc82d6c0640 1 --2- 172.21.15.46:0/503404997 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fc8140a2610 0x7fc8140a4ad0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fc820045000 tx=0x7fc820048860 comp rx=0 tx=0).stop 2024-03-31T23:13:41.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:41.973+0000 7fc82d6c0640 1 -- 172.21.15.46:0/503404997 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc828068950 msgr2=0x7fc828197e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:13:41.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:41.973+0000 7fc82d6c0640 1 --2- 172.21.15.46:0/503404997 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc828068950 0x7fc828197e80 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7fc81800e9f0 tx=0x7fc81800eeb0 comp rx=0 tx=0).stop 2024-03-31T23:13:41.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:41.973+0000 7fc82d6c0640 1 -- 172.21.15.46:0/503404997 shutdown_connections 2024-03-31T23:13:41.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:41.973+0000 7fc82d6c0640 1 --2- 172.21.15.46:0/503404997 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fc8140a2610 0x7fc8140a4ad0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:41.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:41.973+0000 7fc82d6c0640 1 --2- 172.21.15.46:0/503404997 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc8281077a0 0x7fc82819c740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:41.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:41.973+0000 7fc82d6c0640 1 --2- 172.21.15.46:0/503404997 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc828069310 0x7fc8281983c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:41.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:41.973+0000 7fc82d6c0640 1 --2- 172.21.15.46:0/503404997 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc828068950 0x7fc828197e80 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:41.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:41.973+0000 7fc82d6c0640 1 -- 172.21.15.46:0/503404997 >> 172.21.15.46:0/503404997 conn(0x7fc828077750 msgr2=0x7fc8280fe1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:13:41.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:41.973+0000 7fc82d6c0640 1 -- 172.21.15.46:0/503404997 shutdown_connections 2024-03-31T23:13:41.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:41.973+0000 7fc82d6c0640 1 -- 172.21.15.46:0/503404997 wait complete. 2024-03-31T23:13:41.986 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-31T23:13:41.986 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-31T23:13:42.101 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.098+0000 7f2b93247640 1 -- 172.21.15.46:0/3184756190 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2b8c10bd90 msgr2=0x7f2b8c10e230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:13:42.101 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.098+0000 7f2b93247640 1 --2- 172.21.15.46:0/3184756190 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2b8c10bd90 0x7f2b8c10e230 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f2b84009ca0 tx=0x7f2b8402c0f0 comp rx=0 tx=0).stop 2024-03-31T23:13:42.101 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.098+0000 7f2b93247640 1 -- 172.21.15.46:0/3184756190 shutdown_connections 2024-03-31T23:13:42.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.098+0000 7f2b93247640 1 --2- 172.21.15.46:0/3184756190 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2b8c10bd90 0x7f2b8c10e230 unknown :-1 s=CLOSED pgs=627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:42.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.098+0000 7f2b93247640 1 --2- 172.21.15.46:0/3184756190 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2b8c1032e0 0x7f2b8c10b7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:42.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.098+0000 7f2b93247640 1 --2- 172.21.15.46:0/3184756190 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2b8c1029c0 0x7f2b8c102da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:42.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.098+0000 7f2b93247640 1 -- 172.21.15.46:0/3184756190 >> 172.21.15.46:0/3184756190 conn(0x7f2b8c077750 msgr2=0x7f2b8c077b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:13:42.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.098+0000 7f2b93247640 1 -- 172.21.15.46:0/3184756190 shutdown_connections 2024-03-31T23:13:42.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.098+0000 7f2b93247640 1 -- 172.21.15.46:0/3184756190 wait complete. 2024-03-31T23:13:42.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.099+0000 7f2b93247640 1 Processor -- start 2024-03-31T23:13:42.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.099+0000 7f2b93247640 1 -- start start 2024-03-31T23:13:42.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.099+0000 7f2b93247640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2b8c1029c0 0x7f2b8c197ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:13:42.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.099+0000 7f2b93247640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2b8c1032e0 0x7f2b8c198400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:13:42.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.099+0000 7f2b93247640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2b8c10bd90 0x7f2b8c19c780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:13:42.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.099+0000 7f2b93247640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f2b8c111a40 con 0x7f2b8c1032e0 2024-03-31T23:13:42.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.099+0000 7f2b93247640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f2b8c1118c0 con 0x7f2b8c10bd90 2024-03-31T23:13:42.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.099+0000 7f2b93247640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f2b8c111bc0 con 0x7f2b8c1029c0 2024-03-31T23:13:42.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.099+0000 7f2b8bfff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2b8c1032e0 0x7f2b8c198400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:13:42.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.099+0000 7f2b90fe3640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2b8c1029c0 0x7f2b8c197ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:13:42.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.099+0000 7f2b8bfff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2b8c1032e0 0x7f2b8c198400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60012/0 (socket says 172.21.15.46:60012) 2024-03-31T23:13:42.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.099+0000 7f2b8bfff640 1 -- 172.21.15.46:0/2126680105 learned_addr learned my addr 172.21.15.46:0/2126680105 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:13:42.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.099+0000 7f2b917e4640 1 --2- 172.21.15.46:0/2126680105 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2b8c10bd90 0x7f2b8c19c780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:13:42.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.099+0000 7f2b8bfff640 1 -- 172.21.15.46:0/2126680105 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2b8c1029c0 msgr2=0x7f2b8c197ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:13:42.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.099+0000 7f2b8bfff640 1 --2- 172.21.15.46:0/2126680105 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2b8c1029c0 0x7f2b8c197ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:42.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.099+0000 7f2b8bfff640 1 -- 172.21.15.46:0/2126680105 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2b8c10bd90 msgr2=0x7f2b8c19c780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:13:42.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.099+0000 7f2b8bfff640 1 --2- 172.21.15.46:0/2126680105 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2b8c10bd90 0x7f2b8c19c780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:42.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.099+0000 7f2b8bfff640 1 -- 172.21.15.46:0/2126680105 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b8c19ceb0 con 0x7f2b8c1032e0 2024-03-31T23:13:42.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.099+0000 7f2b90fe3640 1 --2- 172.21.15.46:0/2126680105 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2b8c1029c0 0x7f2b8c197ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:13:42.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.100+0000 7f2b8bfff640 1 --2- 172.21.15.46:0/2126680105 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2b8c1032e0 0x7f2b8c198400 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f2b7800e9f0 tx=0x7f2b7800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:13:42.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.100+0000 7f2b89ffb640 1 -- 172.21.15.46:0/2126680105 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b7800cd80 con 0x7f2b8c1032e0 2024-03-31T23:13:42.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.100+0000 7f2b89ffb640 1 -- 172.21.15.46:0/2126680105 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f2b780108e0 con 0x7f2b8c1032e0 2024-03-31T23:13:42.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.100+0000 7f2b93247640 1 -- 172.21.15.46:0/2126680105 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2b8c19d1a0 con 0x7f2b8c1032e0 2024-03-31T23:13:42.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.100+0000 7f2b89ffb640 1 -- 172.21.15.46:0/2126680105 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b78018920 con 0x7f2b8c1032e0 2024-03-31T23:13:42.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.100+0000 7f2b93247640 1 -- 172.21.15.46:0/2126680105 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2b8c1a4a30 con 0x7f2b8c1032e0 2024-03-31T23:13:42.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.101+0000 7f2b93247640 1 -- 172.21.15.46:0/2126680105 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2b8c068540 con 0x7f2b8c1032e0 2024-03-31T23:13:42.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.103+0000 7f2b89ffb640 1 -- 172.21.15.46:0/2126680105 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 25) ==== 137451+0+0 (secure 0 0 0) 0x7f2b78010a90 con 0x7f2b8c1032e0 2024-03-31T23:13:42.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.103+0000 7f2b89ffb640 1 --2- 172.21.15.46:0/2126680105 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f2b7c0a2690 0x7f2b7c0a4b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:13:42.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.103+0000 7f2b90fe3640 1 --2- 172.21.15.46:0/2126680105 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f2b7c0a2690 0x7f2b7c0a4b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:13:42.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.103+0000 7f2b89ffb640 1 -- 172.21.15.46:0/2126680105 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7f2b780cf130 con 0x7f2b8c1032e0 2024-03-31T23:13:42.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.104+0000 7f2b90fe3640 1 --2- 172.21.15.46:0/2126680105 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f2b7c0a2690 0x7f2b7c0a4b50 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f2b80009fc0 tx=0x7f2b80005f90 comp rx=0 tx=0).ready entity=mgr.15654 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:13:42.109 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.108+0000 7f2b89ffb640 1 -- 172.21.15.46:0/2126680105 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f2b7809b5b0 con 0x7f2b8c1032e0 2024-03-31T23:13:42.191 INFO:tasks.workunit.client.0.smithi046.stdout: 1 19888 2.55 MB/sec warmup 48 sec latency 15.699 ms 2024-03-31T23:13:42.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.373+0000 7f2b93247640 1 -- 172.21.15.46:0/2126680105 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2b8c1a4c60 con 0x7f2b8c1032e0 2024-03-31T23:13:42.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.374+0000 7f2b89ffb640 1 -- 172.21.15.46:0/2126680105 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11885 (secure 0 0 0) 0x7f2b7809f650 con 0x7f2b8c1032e0 2024-03-31T23:13:42.375 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-31T23:13:42.375 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/2169018547","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":2169018547},{"type":"v1","addr":"172.21.15.46:6835","nonce":2169018547}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6837/1069387756","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6836","nonce":1069387756},{"type":"v1","addr":"172.21.15.46:6837","nonce":1069387756}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14634,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6839/3915337844","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6838","nonce":3915337844},{"type":"v1","addr":"172.21.15.46:6839","nonce":3915337844}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14646,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6841/897767145","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6840","nonce":897767145},{"type":"v1","addr":"172.21.15.46:6841","nonce":897767145}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24415,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6833/1746950966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6832","nonce":1746950966},{"type":"v1","addr":"172.21.15.162:6833","nonce":1746950966}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24422,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6833/3394739181","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6832","nonce":3394739181},{"type":"v1","addr":"172.21.15.142:6833","nonce":3394739181}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":28,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-31T23:09:11.009090+0000","modified":"2024-03-31T23:13:14.407778+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24515,"mds_1":24482,"mds_2":24451},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24442":{"gid":24442,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.162:6835/1957208984","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6834","nonce":1957208984},{"type":"v1","addr":"172.21.15.162:6835","nonce":1957208984}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24451":{"gid":24451,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":67,"addr":"172.21.15.162:6837/1371886876","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6836","nonce":1371886876},{"type":"v1","addr":"172.21.15.162:6837","nonce":1371886876}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24455":{"gid":24455,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.142:6835/212637431","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6834","nonce":212637431},{"type":"v1","addr":"172.21.15.142:6835","nonce":212637431}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24478":{"gid":24478,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.162:6839/1976275826","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6838","nonce":1976275826},{"type":"v1","addr":"172.21.15.162:6839","nonce":1976275826}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24482":{"gid":24482,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":10,"addr":"172.21.15.142:6837/2559258399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6836","nonce":2559258399},{"type":"v1","addr":"172.21.15.142:6837","nonce":2559258399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24515":{"gid":24515,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.142:6839/2519143918","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6838","nonce":2519143918},{"type":"v1","addr":"172.21.15.142:6839","nonce":2519143918}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24515,"qdb_cluster":[24515,24451,24482]},"id":1}]} 2024-03-31T23:13:42.375 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 28 2024-03-31T23:13:42.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.376+0000 7f2b93247640 1 -- 172.21.15.46:0/2126680105 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f2b7c0a2690 msgr2=0x7f2b7c0a4b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:13:42.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.376+0000 7f2b93247640 1 --2- 172.21.15.46:0/2126680105 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f2b7c0a2690 0x7f2b7c0a4b50 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f2b80009fc0 tx=0x7f2b80005f90 comp rx=0 tx=0).stop 2024-03-31T23:13:42.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.376+0000 7f2b93247640 1 -- 172.21.15.46:0/2126680105 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2b8c1032e0 msgr2=0x7f2b8c198400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:13:42.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.376+0000 7f2b93247640 1 --2- 172.21.15.46:0/2126680105 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2b8c1032e0 0x7f2b8c198400 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f2b7800e9f0 tx=0x7f2b7800eeb0 comp rx=0 tx=0).stop 2024-03-31T23:13:42.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.376+0000 7f2b93247640 1 -- 172.21.15.46:0/2126680105 shutdown_connections 2024-03-31T23:13:42.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.376+0000 7f2b93247640 1 --2- 172.21.15.46:0/2126680105 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f2b7c0a2690 0x7f2b7c0a4b50 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:42.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.376+0000 7f2b93247640 1 --2- 172.21.15.46:0/2126680105 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2b8c10bd90 0x7f2b8c19c780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:42.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.376+0000 7f2b93247640 1 --2- 172.21.15.46:0/2126680105 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2b8c1032e0 0x7f2b8c198400 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:42.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.376+0000 7f2b93247640 1 --2- 172.21.15.46:0/2126680105 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2b8c1029c0 0x7f2b8c197ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:42.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.376+0000 7f2b93247640 1 -- 172.21.15.46:0/2126680105 >> 172.21.15.46:0/2126680105 conn(0x7f2b8c077750 msgr2=0x7f2b8c0fd700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:13:42.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.377+0000 7f2b93247640 1 -- 172.21.15.46:0/2126680105 shutdown_connections 2024-03-31T23:13:42.379 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.377+0000 7f2b93247640 1 -- 172.21.15.46:0/2126680105 wait complete. 2024-03-31T23:13:42.389 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-31T23:13:42.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.505+0000 7fbe96d69640 1 -- 172.21.15.46:0/1698591960 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbe901012f0 msgr2=0x7fbe9010d430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:13:42.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.505+0000 7fbe96d69640 1 --2- 172.21.15.46:0/1698591960 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbe901012f0 0x7fbe9010d430 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7fbe88009ca0 tx=0x7fbe8802c0f0 comp rx=0 tx=0).stop 2024-03-31T23:13:42.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.505+0000 7fbe96d69640 1 -- 172.21.15.46:0/1698591960 shutdown_connections 2024-03-31T23:13:42.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.505+0000 7fbe96d69640 1 --2- 172.21.15.46:0/1698591960 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbe901012f0 0x7fbe9010d430 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:42.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.505+0000 7fbe96d69640 1 --2- 172.21.15.46:0/1698591960 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbe90100930 0x7fbe90100db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:42.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.505+0000 7fbe96d69640 1 --2- 172.21.15.46:0/1698591960 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbe90106680 0x7fbe90106a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:42.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.505+0000 7fbe96d69640 1 -- 172.21.15.46:0/1698591960 >> 172.21.15.46:0/1698591960 conn(0x7fbe90077630 msgr2=0x7fbe90077a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:13:42.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.505+0000 7fbe96d69640 1 -- 172.21.15.46:0/1698591960 shutdown_connections 2024-03-31T23:13:42.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.505+0000 7fbe96d69640 1 -- 172.21.15.46:0/1698591960 wait complete. 2024-03-31T23:13:42.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.506+0000 7fbe96d69640 1 Processor -- start 2024-03-31T23:13:42.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.506+0000 7fbe96d69640 1 -- start start 2024-03-31T23:13:42.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.506+0000 7fbe96d69640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbe90100930 0x7fbe90197e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:13:42.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.506+0000 7fbe96d69640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbe901012f0 0x7fbe90198370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:13:42.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.506+0000 7fbe96d69640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbe90106680 0x7fbe9019c6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:13:42.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.506+0000 7fbe96d69640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fbe90110880 con 0x7fbe90106680 2024-03-31T23:13:42.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.506+0000 7fbe96d69640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fbe90110700 con 0x7fbe90100930 2024-03-31T23:13:42.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.506+0000 7fbe96d69640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fbe90110a00 con 0x7fbe901012f0 2024-03-31T23:13:42.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.506+0000 7fbe95306640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbe90106680 0x7fbe9019c6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:13:42.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.506+0000 7fbe94b05640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbe90100930 0x7fbe90197e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:13:42.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.506+0000 7fbe95306640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbe90106680 0x7fbe9019c6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60038/0 (socket says 172.21.15.46:60038) 2024-03-31T23:13:42.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.506+0000 7fbe8ffff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbe901012f0 0x7fbe90198370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:13:42.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.506+0000 7fbe95306640 1 -- 172.21.15.46:0/1204696500 learned_addr learned my addr 172.21.15.46:0/1204696500 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:13:42.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.507+0000 7fbe95306640 1 -- 172.21.15.46:0/1204696500 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbe901012f0 msgr2=0x7fbe90198370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:13:42.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.507+0000 7fbe95306640 1 --2- 172.21.15.46:0/1204696500 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbe901012f0 0x7fbe90198370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:42.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.507+0000 7fbe95306640 1 -- 172.21.15.46:0/1204696500 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbe90100930 msgr2=0x7fbe90197e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:13:42.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.507+0000 7fbe95306640 1 --2- 172.21.15.46:0/1204696500 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbe90100930 0x7fbe90197e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:42.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.507+0000 7fbe95306640 1 -- 172.21.15.46:0/1204696500 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe9019ce20 con 0x7fbe90106680 2024-03-31T23:13:42.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.507+0000 7fbe94b05640 1 --2- 172.21.15.46:0/1204696500 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbe90100930 0x7fbe90197e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:13:42.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.507+0000 7fbe8ffff640 1 --2- 172.21.15.46:0/1204696500 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbe901012f0 0x7fbe90198370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:13:42.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.507+0000 7fbe95306640 1 --2- 172.21.15.46:0/1204696500 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbe90106680 0x7fbe9019c6f0 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7fbe8802c700 tx=0x7fbe88035d60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:13:42.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.507+0000 7fbe8dffb640 1 -- 172.21.15.46:0/1204696500 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe88039040 con 0x7fbe90106680 2024-03-31T23:13:42.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.507+0000 7fbe8dffb640 1 -- 172.21.15.46:0/1204696500 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fbe88044070 con 0x7fbe90106680 2024-03-31T23:13:42.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.507+0000 7fbe96d69640 1 -- 172.21.15.46:0/1204696500 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbe9019d0b0 con 0x7fbe90106680 2024-03-31T23:13:42.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.507+0000 7fbe8dffb640 1 -- 172.21.15.46:0/1204696500 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe880042a0 con 0x7fbe90106680 2024-03-31T23:13:42.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.508+0000 7fbe96d69640 1 -- 172.21.15.46:0/1204696500 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbe901a49a0 con 0x7fbe90106680 2024-03-31T23:13:42.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.511+0000 7fbe99a22740 1 Processor -- start 2024-03-31T23:13:42.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.511+0000 7fbe8dffb640 1 -- 172.21.15.46:0/1204696500 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 25) ==== 137451+0+0 (secure 0 0 0) 0x7fbe88003970 con 0x7fbe90106680 2024-03-31T23:13:42.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.511+0000 7fbe99a22740 1 -- start start 2024-03-31T23:13:42.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.512+0000 7fbe99a22740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55dfffac2120 0x55dfffac2500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:13:42.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.512+0000 7fbe99a22740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dfffac2a40 0x55dfffacf840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:13:42.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.512+0000 7fbe99a22740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55dfffacfd80 0x55dfffad2170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:13:42.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.512+0000 7fbe99a22740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55dfffa261b0 con 0x55dfffac2120 2024-03-31T23:13:42.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.512+0000 7fbe99a22740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55dfff9cec00 con 0x55dfffacfd80 2024-03-31T23:13:42.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.512+0000 7fbe99a22740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55dfffa22250 con 0x55dfffac2a40 2024-03-31T23:13:42.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.512+0000 7fbe8dffb640 1 --2- 172.21.15.46:0/1204696500 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fbe800a2710 0x7fbe800a4bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:13:42.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.512+0000 7fbe8ffff640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55dfffacfd80 0x55dfffad2170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:13:42.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.512+0000 7fbe8ffff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55dfffac2120 0x55dfffac2500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:13:42.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.512+0000 7fbe8ffff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55dfffac2120 0x55dfffac2500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60050/0 (socket says 172.21.15.46:60050) 2024-03-31T23:13:42.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.512+0000 7fbe8ffff640 1 -- 172.21.15.46:0/598265252 learned_addr learned my addr 172.21.15.46:0/598265252 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:13:42.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.512+0000 7fbe94b05640 1 --2- 172.21.15.46:0/598265252 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dfffac2a40 0x55dfffacf840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:13:42.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.512+0000 7fbe8dffb640 1 -- 172.21.15.46:0/1204696500 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7fbe880f1190 con 0x7fbe90106680 2024-03-31T23:13:42.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.513+0000 7fbe95306640 1 --2- 172.21.15.46:0/1204696500 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fbe800a2710 0x7fbe800a4bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:13:42.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.513+0000 7fbe8ffff640 1 -- 172.21.15.46:0/598265252 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dfffac2a40 msgr2=0x55dfffacf840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:13:42.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.513+0000 7fbe8ffff640 1 --2- 172.21.15.46:0/598265252 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dfffac2a40 0x55dfffacf840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:42.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.513+0000 7fbe8ffff640 1 -- 172.21.15.46:0/598265252 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55dfffacfd80 msgr2=0x55dfffad2170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:13:42.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.513+0000 7fbe8ffff640 1 --2- 172.21.15.46:0/598265252 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55dfffacfd80 0x55dfffad2170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:42.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.513+0000 7fbe8ffff640 1 -- 172.21.15.46:0/598265252 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dfff9ee3d0 con 0x55dfffac2120 2024-03-31T23:13:42.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.513+0000 7fbe94b05640 1 --2- 172.21.15.46:0/598265252 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dfffac2a40 0x55dfffacf840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:13:42.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.513+0000 7fbe8ffff640 1 --2- 172.21.15.46:0/598265252 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55dfffac2120 0x55dfffac2500 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7fbe7c010d40 tx=0x7fbe7c005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7959cdcf08482f8e server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:13:42.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.513+0000 7fbe95306640 1 --2- 172.21.15.46:0/1204696500 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fbe800a2710 0x7fbe800a4bd0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fbe880b9fe0 tx=0x7fbe88035f50 comp rx=0 tx=0).ready entity=mgr.15654 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:13:42.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.513+0000 7fbe7affd640 1 -- 172.21.15.46:0/598265252 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe7c016030 con 0x55dfffac2120 2024-03-31T23:13:42.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.513+0000 7fbe7affd640 1 -- 172.21.15.46:0/598265252 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fbe7c00fe50 con 0x55dfffac2120 2024-03-31T23:13:42.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.513+0000 7fbe7affd640 1 -- 172.21.15.46:0/598265252 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe7c019040 con 0x55dfffac2120 2024-03-31T23:13:42.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.514+0000 7fbe99a22740 1 -- 172.21.15.46:0/598265252 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55dfffac2120 msgr2=0x55dfffac2500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:13:42.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.514+0000 7fbe99a22740 1 --2- 172.21.15.46:0/598265252 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55dfffac2120 0x55dfffac2500 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7fbe7c010d40 tx=0x7fbe7c005f90 comp rx=0 tx=0).stop 2024-03-31T23:13:42.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.514+0000 7fbe99a22740 1 -- 172.21.15.46:0/598265252 shutdown_connections 2024-03-31T23:13:42.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.514+0000 7fbe99a22740 1 --2- 172.21.15.46:0/598265252 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55dfffacfd80 0x55dfffad2170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:42.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.514+0000 7fbe99a22740 1 --2- 172.21.15.46:0/598265252 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dfffac2a40 0x55dfffacf840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:42.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.514+0000 7fbe99a22740 1 --2- 172.21.15.46:0/598265252 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55dfffac2120 0x55dfffac2500 unknown :-1 s=CLOSED pgs=631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:42.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.514+0000 7fbe99a22740 1 -- 172.21.15.46:0/598265252 >> 172.21.15.46:0/598265252 conn(0x55dfffabcba0 msgr2=0x55dfffabcf80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:13:42.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.514+0000 7fbe99a22740 1 -- 172.21.15.46:0/598265252 shutdown_connections 2024-03-31T23:13:42.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.514+0000 7fbe99a22740 1 -- 172.21.15.46:0/598265252 wait complete. 2024-03-31T23:13:42.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.514+0000 7fbe99a22740 1 -- start start 2024-03-31T23:13:42.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.515+0000 7fbe99a22740 1 Processor -- start 2024-03-31T23:13:42.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.515+0000 7fbe99a22740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55dfffac2120 0x55dfffac9000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:13:42.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.515+0000 7fbe99a22740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55dfffac2a40 0x55dfffac9540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:13:42.522 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.515+0000 7fbe99a22740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dfffacfd80 0x55dfffacd7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:13:42.522 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.515+0000 7fbe99a22740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55dfffac35b0 con 0x55dfffac2a40 2024-03-31T23:13:42.522 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.515+0000 7fbe99a22740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55dfff9c4470 con 0x55dfffac2120 2024-03-31T23:13:42.522 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.515+0000 7fbe99a22740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55dfffac3730 con 0x55dfffacfd80 2024-03-31T23:13:42.522 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.515+0000 7fbe99a22740 10 client.0 _ll_register_callbacks cb 0x55dfff9e1cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T23:13:42.522 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.515+0000 7fbe94b05640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55dfffac2a40 0x55dfffac9540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:13:42.522 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.515+0000 7fbe8ffff640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55dfffac2120 0x55dfffac9000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:13:42.522 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.515+0000 7fbe94b05640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55dfffac2a40 0x55dfffac9540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60056/0 (socket says 172.21.15.46:60056) 2024-03-31T23:13:42.522 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.515+0000 7fbe94b05640 1 -- 172.21.15.46:0/4213192880 learned_addr learned my addr 172.21.15.46:0/4213192880 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:13:42.522 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.515+0000 7fbe8ffff640 1 --2- 172.21.15.46:0/4213192880 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dfffacfd80 0x55dfffacd7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:13:42.522 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.515+0000 7fbe94b05640 1 -- 172.21.15.46:0/4213192880 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dfffacfd80 msgr2=0x55dfffacd7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:13:42.522 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.515+0000 7fbe94b05640 1 --2- 172.21.15.46:0/4213192880 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dfffacfd80 0x55dfffacd7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:42.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.515+0000 7fbe94b05640 1 -- 172.21.15.46:0/4213192880 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55dfffac2120 msgr2=0x55dfffac9000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:13:42.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.515+0000 7fbe94b05640 1 --2- 172.21.15.46:0/4213192880 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55dfffac2120 0x55dfffac9000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:42.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.515+0000 7fbe94b05640 1 -- 172.21.15.46:0/4213192880 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dfff73a3f0 con 0x55dfffac2a40 2024-03-31T23:13:42.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.516+0000 7fbe94b05640 1 --2- 172.21.15.46:0/4213192880 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55dfffac2a40 0x55dfffac9540 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7fbe8400e6d0 tx=0x7fbe8400eb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc913e8fd502943d server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:13:42.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.516+0000 7fbe6ffff640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-31T23:13:42.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.516+0000 7fbe6ffff640 1 -- 172.21.15.46:0/4213192880 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe84027060 con 0x55dfffac2a40 2024-03-31T23:13:42.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.516+0000 7fbe6ffff640 1 -- 172.21.15.46:0/4213192880 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7fbe8400da40 con 0x55dfffac2a40 2024-03-31T23:13:42.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.516+0000 7fbe99a22740 1 -- 172.21.15.46:0/4213192880 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbe7c00fe50 con 0x55dfffac2a40 2024-03-31T23:13:42.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.516+0000 7fbe6ffff640 1 -- 172.21.15.46:0/4213192880 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fbe8402e050 con 0x55dfffac2a40 2024-03-31T23:13:42.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.516+0000 7fbe99a22740 10 client.16257 fetch_fsmap learned FSMap version 28 2024-03-31T23:13:42.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.516+0000 7fbe99a22740 1 -- 172.21.15.46:0/4213192880 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55dfff8047d0 con 0x55dfffac2a40 2024-03-31T23:13:42.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.516+0000 7fbe6ffff640 1 -- 172.21.15.46:0/4213192880 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fbe7c00fe50 con 0x55dfffac2a40 2024-03-31T23:13:42.524 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.516+0000 7fbe6ffff640 1 -- 172.21.15.46:0/4213192880 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7fbe840213a0 con 0x55dfffac2a40 2024-03-31T23:13:42.524 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.516+0000 7fbe99a22740 10 client.16257 fetch_fsmap finished waiting for FSMap version 28 2024-03-31T23:13:42.524 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.516+0000 7fbe99a22740 10 client.16257 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-31T23:13:42.524 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.516+0000 7fbe99a22740 20 client.16257 populate_metadata read hostname 'smithi046' 2024-03-31T23:13:42.524 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.516+0000 7fbe99a22740 1 --2- 172.21.15.46:0/4213192880 >> [v2:172.21.15.162:6836/1371886876,v1:172.21.15.162:6837/1371886876] conn(0x55dfffadb9b0 0x55dfffaddda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:13:42.524 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.516+0000 7fbe99a22740 4 client.16257 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-31T23:13:42.524 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.516+0000 7fbe99a22740 1 -- 172.21.15.46:0/4213192880 --> [v2:172.21.15.162:6836/1371886876,v1:172.21.15.162:6837/1371886876] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55dfffa336f0 con 0x55dfffadb9b0 2024-03-31T23:13:42.524 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.517+0000 7fbe8ffff640 1 --2- 172.21.15.46:0/4213192880 >> [v2:172.21.15.162:6836/1371886876,v1:172.21.15.162:6837/1371886876] conn(0x55dfffadb9b0 0x55dfffaddda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:13:42.524 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.517+0000 7fbe6ffff640 10 client.16257 ms_handle_connect on v2:172.21.15.162:6836/1371886876 2024-03-31T23:13:42.524 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.517+0000 7fbe8ffff640 1 --2- 172.21.15.46:0/4213192880 >> [v2:172.21.15.162:6836/1371886876,v1:172.21.15.162:6837/1371886876] conn(0x55dfffadb9b0 0x55dfffaddda0 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=9095b7e468637567 server_cookie=d22e5e1bfb92f02d in_seq=0 out_seq=0 2024-03-31T23:13:42.524 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.517+0000 7fbe6ffff640 1 -- 172.21.15.46:0/4213192880 <== mds.2 v2:172.21.15.162:6836/1371886876 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55dfffa336f0 con 0x55dfffadb9b0 2024-03-31T23:13:42.524 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.518+0000 7fbe6ffff640 10 client.16257 handle_command_reply: tid=0 2024-03-31T23:13:42.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.518+0000 7fbe99a22740 1 client.16257 shutdown 2024-03-31T23:13:42.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.518+0000 7fbe99a22740 1 -- 172.21.15.46:0/4213192880 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55dfffac2a40 msgr2=0x55dfffac9540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:13:42.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.518+0000 7fbe99a22740 1 --2- 172.21.15.46:0/4213192880 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55dfffac2a40 0x55dfffac9540 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7fbe8400e6d0 tx=0x7fbe8400eb90 comp rx=0 tx=0).stop 2024-03-31T23:13:42.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.518+0000 7fbe99a22740 1 -- 172.21.15.46:0/4213192880 shutdown_connections 2024-03-31T23:13:42.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.518+0000 7fbe99a22740 1 --2- 172.21.15.46:0/4213192880 >> [v2:172.21.15.162:6836/1371886876,v1:172.21.15.162:6837/1371886876] conn(0x55dfffadb9b0 0x55dfffaddda0 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:42.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.518+0000 7fbe99a22740 1 --2- 172.21.15.46:0/4213192880 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dfffacfd80 0x55dfffacd7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:42.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.518+0000 7fbe99a22740 1 --2- 172.21.15.46:0/4213192880 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55dfffac2a40 0x55dfffac9540 unknown :-1 s=CLOSED pgs=632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:42.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.518+0000 7fbe6ffff640 0 client.16257 ms_handle_reset on v2:172.21.15.162:6836/1371886876 2024-03-31T23:13:42.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.518+0000 7fbe99a22740 1 --2- 172.21.15.46:0/4213192880 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55dfffac2120 0x55dfffac9000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:42.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.518+0000 7fbe99a22740 1 -- 172.21.15.46:0/4213192880 >> 172.21.15.46:0/4213192880 conn(0x55dfffabcba0 msgr2=0x55dfffabe8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:13:42.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.518+0000 7fbe99a22740 1 -- 172.21.15.46:0/4213192880 shutdown_connections 2024-03-31T23:13:42.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.518+0000 7fbe99a22740 1 -- 172.21.15.46:0/4213192880 wait complete. 2024-03-31T23:13:42.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.518+0000 7fbe99a22740 20 client.16257 trim_cache size 0 max 16384 2024-03-31T23:13:42.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.533+0000 7fbe99a22740 1 Processor -- start 2024-03-31T23:13:42.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.533+0000 7fbe99a22740 1 -- start start 2024-03-31T23:13:42.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.533+0000 7fbe99a22740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55dfffac2120 0x55dfffabe700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:13:42.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.533+0000 7fbe99a22740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dfffac2a40 0x55dfffae3690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:13:42.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.533+0000 7fbe99a22740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55dfffacfd80 0x55dfffacd620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:13:42.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.533+0000 7fbe99a22740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fbe901af150 con 0x55dfffac2120 2024-03-31T23:13:42.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.533+0000 7fbe99a22740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fbe901aca20 con 0x55dfffacfd80 2024-03-31T23:13:42.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.533+0000 7fbe99a22740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fbe901aa870 con 0x55dfffac2a40 2024-03-31T23:13:42.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.533+0000 7fbe8ffff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55dfffac2120 0x55dfffabe700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:13:42.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.533+0000 7fbe8ffff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55dfffac2120 0x55dfffabe700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60072/0 (socket says 172.21.15.46:60072) 2024-03-31T23:13:42.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.533+0000 7fbe8ffff640 1 -- 172.21.15.46:0/2597276062 learned_addr learned my addr 172.21.15.46:0/2597276062 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:13:42.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.533+0000 7fbe94b05640 1 --2- 172.21.15.46:0/2597276062 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dfffac2a40 0x55dfffae3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:13:42.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.533+0000 7fbe8ffff640 1 --2- 172.21.15.46:0/2597276062 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55dfffacfd80 0x55dfffacd620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:13:42.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.533+0000 7fbe8ffff640 1 -- 172.21.15.46:0/2597276062 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dfffac2a40 msgr2=0x55dfffae3690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:13:42.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.533+0000 7fbe8ffff640 1 --2- 172.21.15.46:0/2597276062 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dfffac2a40 0x55dfffae3690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:42.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.533+0000 7fbe8ffff640 1 -- 172.21.15.46:0/2597276062 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55dfffacfd80 msgr2=0x55dfffacd620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:13:42.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.533+0000 7fbe8ffff640 1 --2- 172.21.15.46:0/2597276062 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55dfffacfd80 0x55dfffacd620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:42.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.533+0000 7fbe8ffff640 1 -- 172.21.15.46:0/2597276062 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dfffade110 con 0x55dfffac2120 2024-03-31T23:13:42.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.533+0000 7fbe94b05640 1 --2- 172.21.15.46:0/2597276062 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dfffac2a40 0x55dfffae3690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:13:42.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.534+0000 7fbe8ffff640 1 --2- 172.21.15.46:0/2597276062 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55dfffac2120 0x55dfffabe700 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7fbe7c009380 tx=0x7fbe7c009310 comp rx=0 tx=0).ready entity=mon.0 client_cookie=42abbb9e26a760a8 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:13:42.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.534+0000 7fbe7a7fc640 1 -- 172.21.15.46:0/2597276062 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe7c036070 con 0x55dfffac2120 2024-03-31T23:13:42.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.534+0000 7fbe7a7fc640 1 -- 172.21.15.46:0/2597276062 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fbe7c00fe30 con 0x55dfffac2120 2024-03-31T23:13:42.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.534+0000 7fbe7a7fc640 1 -- 172.21.15.46:0/2597276062 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe7c016040 con 0x55dfffac2120 2024-03-31T23:13:42.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.534+0000 7fbe99a22740 1 -- 172.21.15.46:0/2597276062 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55dfffac2120 msgr2=0x55dfffabe700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:13:42.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.534+0000 7fbe99a22740 1 --2- 172.21.15.46:0/2597276062 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55dfffac2120 0x55dfffabe700 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7fbe7c009380 tx=0x7fbe7c009310 comp rx=0 tx=0).stop 2024-03-31T23:13:42.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.534+0000 7fbe99a22740 1 -- 172.21.15.46:0/2597276062 shutdown_connections 2024-03-31T23:13:42.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.534+0000 7fbe99a22740 1 --2- 172.21.15.46:0/2597276062 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55dfffacfd80 0x55dfffacd620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:42.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.534+0000 7fbe99a22740 1 --2- 172.21.15.46:0/2597276062 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dfffac2a40 0x55dfffae3690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:42.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.534+0000 7fbe99a22740 1 --2- 172.21.15.46:0/2597276062 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55dfffac2120 0x55dfffabe700 unknown :-1 s=CLOSED pgs=633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:42.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.534+0000 7fbe99a22740 1 -- 172.21.15.46:0/2597276062 >> 172.21.15.46:0/2597276062 conn(0x55dfffabcba0 msgr2=0x55dfffabdff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:13:42.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.535+0000 7fbe99a22740 1 -- 172.21.15.46:0/2597276062 shutdown_connections 2024-03-31T23:13:42.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.535+0000 7fbe99a22740 1 -- 172.21.15.46:0/2597276062 wait complete. 2024-03-31T23:13:42.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.535+0000 7fbe99a22740 1 -- start start 2024-03-31T23:13:42.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.535+0000 7fbe99a22740 1 Processor -- start 2024-03-31T23:13:42.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.535+0000 7fbe99a22740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dfffac2120 0x55dfffad8450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:13:42.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.535+0000 7fbe99a22740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55dfffac2a40 0x55dfffad37c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:13:42.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.535+0000 7fbe99a22740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55dfffacfd80 0x55dfffad3f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:13:42.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.535+0000 7fbe99a22740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fbe901aaee0 con 0x55dfffac2a40 2024-03-31T23:13:42.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.535+0000 7fbe99a22740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fbe901ac350 con 0x55dfffacfd80 2024-03-31T23:13:42.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.535+0000 7fbe99a22740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fbe901ab5b0 con 0x55dfffac2120 2024-03-31T23:13:42.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.535+0000 7fbe99a22740 10 client.0 _ll_register_callbacks cb 0x7fbe84009ed0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T23:13:42.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.535+0000 7fbe94b05640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55dfffac2a40 0x55dfffad37c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:13:42.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.535+0000 7fbe94b05640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55dfffac2a40 0x55dfffad37c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60086/0 (socket says 172.21.15.46:60086) 2024-03-31T23:13:42.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.535+0000 7fbe94b05640 1 -- 172.21.15.46:0/3477049364 learned_addr learned my addr 172.21.15.46:0/3477049364 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:13:42.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.536+0000 7fbe8ffff640 1 --2- 172.21.15.46:0/3477049364 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55dfffacfd80 0x55dfffad3f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:13:42.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.536+0000 7fbe8ffff640 1 --2- 172.21.15.46:0/3477049364 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dfffac2120 0x55dfffad8450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:13:42.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.536+0000 7fbe94b05640 1 -- 172.21.15.46:0/3477049364 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dfffac2120 msgr2=0x55dfffad8450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:13:42.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.536+0000 7fbe94b05640 1 --2- 172.21.15.46:0/3477049364 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dfffac2120 0x55dfffad8450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:42.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.536+0000 7fbe94b05640 1 -- 172.21.15.46:0/3477049364 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55dfffacfd80 msgr2=0x55dfffad3f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:13:42.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.536+0000 7fbe94b05640 1 --2- 172.21.15.46:0/3477049364 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55dfffacfd80 0x55dfffad3f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:42.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.536+0000 7fbe94b05640 1 -- 172.21.15.46:0/3477049364 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dfffb40cc0 con 0x55dfffac2a40 2024-03-31T23:13:42.545 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.536+0000 7fbe94b05640 1 --2- 172.21.15.46:0/3477049364 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55dfffac2a40 0x55dfffad37c0 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7fbe8400f010 tx=0x7fbe84021420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a3146420132405b8 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:13:42.545 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.536+0000 7fbe6ffff640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-31T23:13:42.545 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.536+0000 7fbe6ffff640 1 -- 172.21.15.46:0/3477049364 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe84005240 con 0x55dfffac2a40 2024-03-31T23:13:42.545 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.536+0000 7fbe6ffff640 1 -- 172.21.15.46:0/3477049364 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7fbe840053e0 con 0x55dfffac2a40 2024-03-31T23:13:42.545 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.536+0000 7fbe99a22740 1 -- 172.21.15.46:0/3477049364 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbe7c00fe30 con 0x55dfffac2a40 2024-03-31T23:13:42.545 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.536+0000 7fbe6ffff640 1 -- 172.21.15.46:0/3477049364 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fbe84040050 con 0x55dfffac2a40 2024-03-31T23:13:42.545 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.536+0000 7fbe99a22740 10 client.16263 fetch_fsmap learned FSMap version 28 2024-03-31T23:13:42.545 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.536+0000 7fbe99a22740 1 -- 172.21.15.46:0/3477049364 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55dfffb41e60 con 0x55dfffac2a40 2024-03-31T23:13:42.545 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.536+0000 7fbe6ffff640 1 -- 172.21.15.46:0/3477049364 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fbe7c00fe30 con 0x55dfffac2a40 2024-03-31T23:13:42.545 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.537+0000 7fbe6ffff640 1 -- 172.21.15.46:0/3477049364 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7fbe840035f0 con 0x55dfffac2a40 2024-03-31T23:13:42.545 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.537+0000 7fbe99a22740 10 client.16263 fetch_fsmap finished waiting for FSMap version 28 2024-03-31T23:13:42.545 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.537+0000 7fbe99a22740 10 client.16263 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-31T23:13:42.545 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.537+0000 7fbe99a22740 20 client.16263 populate_metadata read hostname 'smithi046' 2024-03-31T23:13:42.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.537+0000 7fbe99a22740 1 --2- 172.21.15.46:0/3477049364 >> [v2:172.21.15.162:6836/1371886876,v1:172.21.15.162:6837/1371886876] conn(0x55dfffb42570 0x55dfffb44990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:13:42.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.537+0000 7fbe99a22740 4 client.16263 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-31T23:13:42.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.537+0000 7fbe99a22740 1 -- 172.21.15.46:0/3477049364 --> [v2:172.21.15.162:6836/1371886876,v1:172.21.15.162:6837/1371886876] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55dfffa2da40 con 0x55dfffb42570 2024-03-31T23:13:42.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.537+0000 7fbe8ffff640 1 --2- 172.21.15.46:0/3477049364 >> [v2:172.21.15.162:6836/1371886876,v1:172.21.15.162:6837/1371886876] conn(0x55dfffb42570 0x55dfffb44990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:13:42.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.537+0000 7fbe6ffff640 10 client.16263 ms_handle_connect on v2:172.21.15.162:6836/1371886876 2024-03-31T23:13:42.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.537+0000 7fbe8ffff640 1 --2- 172.21.15.46:0/3477049364 >> [v2:172.21.15.162:6836/1371886876,v1:172.21.15.162:6837/1371886876] conn(0x55dfffb42570 0x55dfffb44990 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=66afe9cfb380760e server_cookie=5920a8d579d7703c in_seq=0 out_seq=0 2024-03-31T23:13:42.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.538+0000 7fbe6ffff640 1 -- 172.21.15.46:0/3477049364 <== mds.2 v2:172.21.15.162:6836/1371886876 1 ==== command_reply(tid 0: 0 ) ==== 8+0+5977 (crc 0 0 0) 0x55dfffa2da40 con 0x55dfffb42570 2024-03-31T23:13:42.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.538+0000 7fbe6ffff640 10 client.16263 handle_command_reply: tid=0 2024-03-31T23:13:42.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.538+0000 7fbe99a22740 1 client.16263 shutdown 2024-03-31T23:13:42.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.538+0000 7fbe99a22740 1 -- 172.21.15.46:0/3477049364 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55dfffac2a40 msgr2=0x55dfffad37c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:13:42.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.538+0000 7fbe99a22740 1 --2- 172.21.15.46:0/3477049364 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55dfffac2a40 0x55dfffad37c0 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7fbe8400f010 tx=0x7fbe84021420 comp rx=0 tx=0).stop 2024-03-31T23:13:42.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.538+0000 7fbe99a22740 1 -- 172.21.15.46:0/3477049364 shutdown_connections 2024-03-31T23:13:42.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.538+0000 7fbe99a22740 1 --2- 172.21.15.46:0/3477049364 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55dfffacfd80 0x55dfffad3f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:42.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.538+0000 7fbe99a22740 1 --2- 172.21.15.46:0/3477049364 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55dfffac2a40 0x55dfffad37c0 unknown :-1 s=CLOSED pgs=634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:42.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.538+0000 7fbe99a22740 1 --2- 172.21.15.46:0/3477049364 >> [v2:172.21.15.162:6836/1371886876,v1:172.21.15.162:6837/1371886876] conn(0x55dfffb42570 0x55dfffb44990 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:42.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.538+0000 7fbe99a22740 1 --2- 172.21.15.46:0/3477049364 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dfffac2120 0x55dfffad8450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:42.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.538+0000 7fbe6ffff640 0 client.16263 ms_handle_reset on v2:172.21.15.162:6836/1371886876 2024-03-31T23:13:42.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.538+0000 7fbe99a22740 1 -- 172.21.15.46:0/3477049364 >> 172.21.15.46:0/3477049364 conn(0x55dfffabcba0 msgr2=0x55dfffabbd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:13:42.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.538+0000 7fbe99a22740 1 -- 172.21.15.46:0/3477049364 shutdown_connections 2024-03-31T23:13:42.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.538+0000 7fbe99a22740 1 -- 172.21.15.46:0/3477049364 wait complete. 2024-03-31T23:13:42.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.538+0000 7fbe99a22740 20 client.16263 trim_cache size 0 max 16384 2024-03-31T23:13:42.550 INFO:teuthology.orchestra.run.smithi046.stdout:[ 2024-03-31T23:13:42.550 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-03-31T23:13:42.551 INFO:teuthology.orchestra.run.smithi046.stdout: "damage_type": "backtrace", 2024-03-31T23:13:42.551 INFO:teuthology.orchestra.run.smithi046.stdout: "id": 116245510, 2024-03-31T23:13:42.551 INFO:teuthology.orchestra.run.smithi046.stdout: "ino": 1099511628313, 2024-03-31T23:13:42.551 INFO:teuthology.orchestra.run.smithi046.stdout: "path": "/volumes/qa/sv_0/e990713a-13d9-4c54-877d-f0f60b19e88e/client.0/tmp/clients/client0/~dmtmp/COREL/GRAPHIC1.CDR" 2024-03-31T23:13:42.551 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:13:42.551 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-03-31T23:13:42.551 INFO:teuthology.orchestra.run.smithi046.stdout: "damage_type": "backtrace", 2024-03-31T23:13:42.551 INFO:teuthology.orchestra.run.smithi046.stdout: "id": 121823561, 2024-03-31T23:13:42.551 INFO:teuthology.orchestra.run.smithi046.stdout: "ino": 1099511628297, 2024-03-31T23:13:42.551 INFO:teuthology.orchestra.run.smithi046.stdout: "path": "/volumes/qa/sv_0/e990713a-13d9-4c54-877d-f0f60b19e88e/client.0/tmp/clients/client0/~dmtmp/WORD/CHAP10.DOC" 2024-03-31T23:13:42.551 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:13:42.552 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-03-31T23:13:42.552 INFO:teuthology.orchestra.run.smithi046.stdout: "damage_type": "backtrace", 2024-03-31T23:13:42.552 INFO:teuthology.orchestra.run.smithi046.stdout: "id": 197392199, 2024-03-31T23:13:42.552 INFO:teuthology.orchestra.run.smithi046.stdout: "ino": 1099511628312, 2024-03-31T23:13:42.552 INFO:teuthology.orchestra.run.smithi046.stdout: "path": "/volumes/qa/sv_0/e990713a-13d9-4c54-877d-f0f60b19e88e/client.0/tmp/clients/client0/~dmtmp/COREL/ARTISTIC.ACL" 2024-03-31T23:13:42.552 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:13:42.552 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-03-31T23:13:42.552 INFO:teuthology.orchestra.run.smithi046.stdout: "damage_type": "backtrace", 2024-03-31T23:13:42.552 INFO:teuthology.orchestra.run.smithi046.stdout: "id": 309025251, 2024-03-31T23:13:42.552 INFO:teuthology.orchestra.run.smithi046.stdout: "ino": 1099511628351, 2024-03-31T23:13:42.552 INFO:teuthology.orchestra.run.smithi046.stdout: "path": "/volumes/qa/sv_0/e990713a-13d9-4c54-877d-f0f60b19e88e/client.0/tmp/clients/client0/~dmtmp/ACCESS/FASTENER.MDB" 2024-03-31T23:13:42.552 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:13:42.552 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-03-31T23:13:42.553 INFO:teuthology.orchestra.run.smithi046.stdout: "damage_type": "backtrace", 2024-03-31T23:13:42.553 INFO:teuthology.orchestra.run.smithi046.stdout: "id": 973458780, 2024-03-31T23:13:42.553 INFO:teuthology.orchestra.run.smithi046.stdout: "ino": 1099511628320, 2024-03-31T23:13:42.553 INFO:teuthology.orchestra.run.smithi046.stdout: "path": "/volumes/qa/sv_0/e990713a-13d9-4c54-877d-f0f60b19e88e/client.0/tmp/clients/client0/~dmtmp/COREL/CORELDRW.CDT" 2024-03-31T23:13:42.553 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:13:42.553 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-03-31T23:13:42.553 INFO:teuthology.orchestra.run.smithi046.stdout: "damage_type": "backtrace", 2024-03-31T23:13:42.553 INFO:teuthology.orchestra.run.smithi046.stdout: "id": 1001932470, 2024-03-31T23:13:42.553 INFO:teuthology.orchestra.run.smithi046.stdout: "ino": 1099511628324, 2024-03-31T23:13:42.553 INFO:teuthology.orchestra.run.smithi046.stdout: "path": "/volumes/qa/sv_0/e990713a-13d9-4c54-877d-f0f60b19e88e/client.0/tmp/clients/client0/~dmtmp/WORDPRO/NEWS1_1.LWP" 2024-03-31T23:13:42.553 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:13:42.553 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-03-31T23:13:42.554 INFO:teuthology.orchestra.run.smithi046.stdout: "damage_type": "backtrace", 2024-03-31T23:13:42.554 INFO:teuthology.orchestra.run.smithi046.stdout: "id": 1072693154, 2024-03-31T23:13:42.554 INFO:teuthology.orchestra.run.smithi046.stdout: "ino": 1099511628316, 2024-03-31T23:13:42.554 INFO:teuthology.orchestra.run.smithi046.stdout: "path": "/volumes/qa/sv_0/e990713a-13d9-4c54-877d-f0f60b19e88e/client.0/tmp/clients/client0/~dmtmp/COREL/SIGN2.CPT" 2024-03-31T23:13:42.554 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:13:42.554 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-03-31T23:13:42.554 INFO:teuthology.orchestra.run.smithi046.stdout: "damage_type": "backtrace", 2024-03-31T23:13:42.554 INFO:teuthology.orchestra.run.smithi046.stdout: "id": 1266774805, 2024-03-31T23:13:42.554 INFO:teuthology.orchestra.run.smithi046.stdout: "ino": 1099511628298, 2024-03-31T23:13:42.554 INFO:teuthology.orchestra.run.smithi046.stdout: "path": "/volumes/qa/sv_0/e990713a-13d9-4c54-877d-f0f60b19e88e/client.0/tmp/clients/client0/~dmtmp/WORD/BASEMACH.DOC" 2024-03-31T23:13:42.554 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:13:42.554 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-03-31T23:13:42.554 INFO:teuthology.orchestra.run.smithi046.stdout: "damage_type": "backtrace", 2024-03-31T23:13:42.555 INFO:teuthology.orchestra.run.smithi046.stdout: "id": 1302310385, 2024-03-31T23:13:42.555 INFO:teuthology.orchestra.run.smithi046.stdout: "ino": 1099511628322, 2024-03-31T23:13:42.555 INFO:teuthology.orchestra.run.smithi046.stdout: "path": "/volumes/qa/sv_0/e990713a-13d9-4c54-877d-f0f60b19e88e/client.0/tmp/clients/client0/~dmtmp/WORDPRO/BENCHS.LWP" 2024-03-31T23:13:42.555 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:13:42.555 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-03-31T23:13:42.555 INFO:teuthology.orchestra.run.smithi046.stdout: "damage_type": "backtrace", 2024-03-31T23:13:42.555 INFO:teuthology.orchestra.run.smithi046.stdout: "id": 1398107989, 2024-03-31T23:13:42.555 INFO:teuthology.orchestra.run.smithi046.stdout: "ino": 1099511628299, 2024-03-31T23:13:42.555 INFO:teuthology.orchestra.run.smithi046.stdout: "path": "/volumes/qa/sv_0/e990713a-13d9-4c54-877d-f0f60b19e88e/client.0/tmp/clients/client0/~dmtmp/WORD/FACTS.DOC" 2024-03-31T23:13:42.555 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:13:42.555 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-03-31T23:13:42.555 INFO:teuthology.orchestra.run.smithi046.stdout: "damage_type": "backtrace", 2024-03-31T23:13:42.555 INFO:teuthology.orchestra.run.smithi046.stdout: "id": 1405771217, 2024-03-31T23:13:42.556 INFO:teuthology.orchestra.run.smithi046.stdout: "ino": 1099511628310, 2024-03-31T23:13:42.556 INFO:teuthology.orchestra.run.smithi046.stdout: "path": "/volumes/qa/sv_0/e990713a-13d9-4c54-877d-f0f60b19e88e/client.0/tmp/clients/client0/~dmtmp/COREL/CDRROLS.CFG" 2024-03-31T23:13:42.556 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:13:42.556 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-03-31T23:13:42.556 INFO:teuthology.orchestra.run.smithi046.stdout: "damage_type": "backtrace", 2024-03-31T23:13:42.556 INFO:teuthology.orchestra.run.smithi046.stdout: "id": 2072353046, 2024-03-31T23:13:42.556 INFO:teuthology.orchestra.run.smithi046.stdout: "ino": 1099511628317, 2024-03-31T23:13:42.556 INFO:teuthology.orchestra.run.smithi046.stdout: "path": "/volumes/qa/sv_0/e990713a-13d9-4c54-877d-f0f60b19e88e/client.0/tmp/clients/client0/~dmtmp/COREL/TRAIN.CDR" 2024-03-31T23:13:42.556 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:13:42.556 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-03-31T23:13:42.556 INFO:teuthology.orchestra.run.smithi046.stdout: "damage_type": "backtrace", 2024-03-31T23:13:42.556 INFO:teuthology.orchestra.run.smithi046.stdout: "id": 2152932943, 2024-03-31T23:13:42.556 INFO:teuthology.orchestra.run.smithi046.stdout: "ino": 1099511628301, 2024-03-31T23:13:42.556 INFO:teuthology.orchestra.run.smithi046.stdout: "path": "/volumes/qa/sv_0/e990713a-13d9-4c54-877d-f0f60b19e88e/client.0/tmp/clients/client0/~dmtmp/EXCEL/PCMAGCD.XLS" 2024-03-31T23:13:42.557 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:13:42.557 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-03-31T23:13:42.557 INFO:teuthology.orchestra.run.smithi046.stdout: "damage_type": "backtrace", 2024-03-31T23:13:42.557 INFO:teuthology.orchestra.run.smithi046.stdout: "id": 2171341473, 2024-03-31T23:13:42.557 INFO:teuthology.orchestra.run.smithi046.stdout: "ino": 1099511628354, 2024-03-31T23:13:42.557 INFO:teuthology.orchestra.run.smithi046.stdout: "path": "/volumes/qa/sv_0/e990713a-13d9-4c54-877d-f0f60b19e88e/client.0/tmp/clients/client0/~dmtmp/SEED/MEDIUM.FIL" 2024-03-31T23:13:42.557 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:13:42.557 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-03-31T23:13:42.557 INFO:teuthology.orchestra.run.smithi046.stdout: "damage_type": "backtrace", 2024-03-31T23:13:42.557 INFO:teuthology.orchestra.run.smithi046.stdout: "id": 2371159217, 2024-03-31T23:13:42.557 INFO:teuthology.orchestra.run.smithi046.stdout: "ino": 1099511628302, 2024-03-31T23:13:42.557 INFO:teuthology.orchestra.run.smithi046.stdout: "path": "/volumes/qa/sv_0/e990713a-13d9-4c54-877d-f0f60b19e88e/client.0/tmp/clients/client0/~dmtmp/EXCEL/SALES.XLS" 2024-03-31T23:13:42.557 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:13:42.557 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-03-31T23:13:42.558 INFO:teuthology.orchestra.run.smithi046.stdout: "damage_type": "backtrace", 2024-03-31T23:13:42.558 INFO:teuthology.orchestra.run.smithi046.stdout: "id": 2677032042, 2024-03-31T23:13:42.558 INFO:teuthology.orchestra.run.smithi046.stdout: "ino": 1099511628326, 2024-03-31T23:13:42.558 INFO:teuthology.orchestra.run.smithi046.stdout: "path": "/volumes/qa/sv_0/e990713a-13d9-4c54-877d-f0f60b19e88e/client.0/tmp/clients/client0/~dmtmp/PM/T1.XLS" 2024-03-31T23:13:42.558 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:13:42.558 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-03-31T23:13:42.558 INFO:teuthology.orchestra.run.smithi046.stdout: "damage_type": "backtrace", 2024-03-31T23:13:42.558 INFO:teuthology.orchestra.run.smithi046.stdout: "id": 3146452803, 2024-03-31T23:13:42.558 INFO:teuthology.orchestra.run.smithi046.stdout: "ino": 1099511628323, 2024-03-31T23:13:42.558 INFO:teuthology.orchestra.run.smithi046.stdout: "path": "/volumes/qa/sv_0/e990713a-13d9-4c54-877d-f0f60b19e88e/client.0/tmp/clients/client0/~dmtmp/WORDPRO/RESULTS.XLS" 2024-03-31T23:13:42.558 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:13:42.558 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-03-31T23:13:42.559 INFO:teuthology.orchestra.run.smithi046.stdout: "damage_type": "backtrace", 2024-03-31T23:13:42.559 INFO:teuthology.orchestra.run.smithi046.stdout: "id": 3245313740, 2024-03-31T23:13:42.559 INFO:teuthology.orchestra.run.smithi046.stdout: "ino": 1099511628315, 2024-03-31T23:13:42.559 INFO:teuthology.orchestra.run.smithi046.stdout: "path": "/volumes/qa/sv_0/e990713a-13d9-4c54-877d-f0f60b19e88e/client.0/tmp/clients/client0/~dmtmp/COREL/TEAPOT2.CPT" 2024-03-31T23:13:42.559 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:13:42.559 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-03-31T23:13:42.559 INFO:teuthology.orchestra.run.smithi046.stdout: "damage_type": "backtrace", 2024-03-31T23:13:42.559 INFO:teuthology.orchestra.run.smithi046.stdout: "id": 3448543011, 2024-03-31T23:13:42.559 INFO:teuthology.orchestra.run.smithi046.stdout: "ino": 1099511628328, 2024-03-31T23:13:42.559 INFO:teuthology.orchestra.run.smithi046.stdout: "path": "/volumes/qa/sv_0/e990713a-13d9-4c54-877d-f0f60b19e88e/client.0/tmp/clients/client0/~dmtmp/PM/MOVED.DOC" 2024-03-31T23:13:42.559 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:13:42.559 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-03-31T23:13:42.559 INFO:teuthology.orchestra.run.smithi046.stdout: "damage_type": "backtrace", 2024-03-31T23:13:42.559 INFO:teuthology.orchestra.run.smithi046.stdout: "id": 3461088003, 2024-03-31T23:13:42.560 INFO:teuthology.orchestra.run.smithi046.stdout: "ino": 1099511628318, 2024-03-31T23:13:42.560 INFO:teuthology.orchestra.run.smithi046.stdout: "path": "/volumes/qa/sv_0/e990713a-13d9-4c54-877d-f0f60b19e88e/client.0/tmp/clients/client0/~dmtmp/COREL/GRAPHIC2.CDR" 2024-03-31T23:13:42.560 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:13:42.560 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-03-31T23:13:42.560 INFO:teuthology.orchestra.run.smithi046.stdout: "damage_type": "backtrace", 2024-03-31T23:13:42.560 INFO:teuthology.orchestra.run.smithi046.stdout: "id": 3544020290, 2024-03-31T23:13:42.560 INFO:teuthology.orchestra.run.smithi046.stdout: "ino": 1099511628355, 2024-03-31T23:13:42.560 INFO:teuthology.orchestra.run.smithi046.stdout: "path": "/volumes/qa/sv_0/e990713a-13d9-4c54-877d-f0f60b19e88e/client.0/tmp/clients/client0/~dmtmp/SEED/LARGE.FIL" 2024-03-31T23:13:42.560 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:13:42.560 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-03-31T23:13:42.560 INFO:teuthology.orchestra.run.smithi046.stdout: "damage_type": "backtrace", 2024-03-31T23:13:42.560 INFO:teuthology.orchestra.run.smithi046.stdout: "id": 3986028770, 2024-03-31T23:13:42.560 INFO:teuthology.orchestra.run.smithi046.stdout: "ino": 1099511628327, 2024-03-31T23:13:42.560 INFO:teuthology.orchestra.run.smithi046.stdout: "path": "/volumes/qa/sv_0/e990713a-13d9-4c54-877d-f0f60b19e88e/client.0/tmp/clients/client0/~dmtmp/PM/EVENTS.DOC" 2024-03-31T23:13:42.560 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:13:42.561 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-03-31T23:13:42.561 INFO:teuthology.orchestra.run.smithi046.stdout: "damage_type": "backtrace", 2024-03-31T23:13:42.561 INFO:teuthology.orchestra.run.smithi046.stdout: "id": 4057030136, 2024-03-31T23:13:42.561 INFO:teuthology.orchestra.run.smithi046.stdout: "ino": 1099511628353, 2024-03-31T23:13:42.561 INFO:teuthology.orchestra.run.smithi046.stdout: "path": "/volumes/qa/sv_0/e990713a-13d9-4c54-877d-f0f60b19e88e/client.0/tmp/clients/client0/~dmtmp/SEED/SMALL.FIL" 2024-03-31T23:13:42.561 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:13:42.561 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-03-31T23:13:42.561 INFO:teuthology.orchestra.run.smithi046.stdout: "damage_type": "backtrace", 2024-03-31T23:13:42.561 INFO:teuthology.orchestra.run.smithi046.stdout: "id": 4079277184, 2024-03-31T23:13:42.561 INFO:teuthology.orchestra.run.smithi046.stdout: "ino": 1099511628319, 2024-03-31T23:13:42.561 INFO:teuthology.orchestra.run.smithi046.stdout: "path": "/volumes/qa/sv_0/e990713a-13d9-4c54-877d-f0f60b19e88e/client.0/tmp/clients/client0/~dmtmp/COREL/ROSE.CPT" 2024-03-31T23:13:42.561 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:13:42.561 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-03-31T23:13:42.561 INFO:teuthology.orchestra.run.smithi046.stdout: "damage_type": "backtrace", 2024-03-31T23:13:42.562 INFO:teuthology.orchestra.run.smithi046.stdout: "id": 4082973856, 2024-03-31T23:13:42.562 INFO:teuthology.orchestra.run.smithi046.stdout: "ino": 1099511628314, 2024-03-31T23:13:42.562 INFO:teuthology.orchestra.run.smithi046.stdout: "path": "/volumes/qa/sv_0/e990713a-13d9-4c54-877d-f0f60b19e88e/client.0/tmp/clients/client0/~dmtmp/COREL/MAILBOX2.CPT" 2024-03-31T23:13:42.562 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:13:42.562 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-03-31T23:13:42.562 INFO:teuthology.orchestra.run.smithi046.stdout: "damage_type": "backtrace", 2024-03-31T23:13:42.562 INFO:teuthology.orchestra.run.smithi046.stdout: "id": 4106478478, 2024-03-31T23:13:42.562 INFO:teuthology.orchestra.run.smithi046.stdout: "ino": 1099511628311, 2024-03-31T23:13:42.562 INFO:teuthology.orchestra.run.smithi046.stdout: "path": "/volumes/qa/sv_0/e990713a-13d9-4c54-877d-f0f60b19e88e/client.0/tmp/clients/client0/~dmtmp/COREL/CDRBARS.CFG" 2024-03-31T23:13:42.562 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-31T23:13:42.562 INFO:teuthology.orchestra.run.smithi046.stdout:] 2024-03-31T23:13:42.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.538+0000 7fbe96d69640 1 -- 172.21.15.46:0/1204696500 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fbe800a2710 msgr2=0x7fbe800a4bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:13:42.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.538+0000 7fbe96d69640 1 --2- 172.21.15.46:0/1204696500 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fbe800a2710 0x7fbe800a4bd0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fbe880b9fe0 tx=0x7fbe88035f50 comp rx=0 tx=0).stop 2024-03-31T23:13:42.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.539+0000 7fbe96d69640 1 -- 172.21.15.46:0/1204696500 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbe90106680 msgr2=0x7fbe9019c6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:13:42.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.539+0000 7fbe96d69640 1 --2- 172.21.15.46:0/1204696500 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbe90106680 0x7fbe9019c6f0 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7fbe8802c700 tx=0x7fbe88035d60 comp rx=0 tx=0).stop 2024-03-31T23:13:42.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.539+0000 7fbe96d69640 1 -- 172.21.15.46:0/1204696500 shutdown_connections 2024-03-31T23:13:42.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.539+0000 7fbe96d69640 1 --2- 172.21.15.46:0/1204696500 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fbe800a2710 0x7fbe800a4bd0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:42.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.539+0000 7fbe96d69640 1 --2- 172.21.15.46:0/1204696500 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbe90106680 0x7fbe9019c6f0 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:42.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.539+0000 7fbe96d69640 1 --2- 172.21.15.46:0/1204696500 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbe901012f0 0x7fbe90198370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:42.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.539+0000 7fbe96d69640 1 --2- 172.21.15.46:0/1204696500 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbe90100930 0x7fbe90197e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:42.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.539+0000 7fbe96d69640 1 -- 172.21.15.46:0/1204696500 >> 172.21.15.46:0/1204696500 conn(0x7fbe90077630 msgr2=0x7fbe90104740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:13:42.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.539+0000 7fbe96d69640 1 -- 172.21.15.46:0/1204696500 shutdown_connections 2024-03-31T23:13:42.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.539+0000 7fbe96d69640 1 -- 172.21.15.46:0/1204696500 wait complete. 2024-03-31T23:13:42.565 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-31T23:13:42.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.675+0000 7f5fccc14640 1 -- 172.21.15.46:0/1101384712 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5fc8100fc0 msgr2=0x7f5fc810d4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:13:42.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.675+0000 7f5fccc14640 1 --2- 172.21.15.46:0/1101384712 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5fc8100fc0 0x7f5fc810d4d0 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f5fc0009cd0 tx=0x7f5fc002c0f0 comp rx=0 tx=0).stop 2024-03-31T23:13:42.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.676+0000 7f5fccc14640 1 -- 172.21.15.46:0/1101384712 shutdown_connections 2024-03-31T23:13:42.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.676+0000 7f5fccc14640 1 --2- 172.21.15.46:0/1101384712 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5fc8100fc0 0x7f5fc810d4d0 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:42.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.676+0000 7f5fccc14640 1 --2- 172.21.15.46:0/1101384712 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5fc8100600 0x7f5fc8100a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:42.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.676+0000 7f5fccc14640 1 --2- 172.21.15.46:0/1101384712 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5fc81066a0 0x7f5fc8106a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:42.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.676+0000 7f5fccc14640 1 -- 172.21.15.46:0/1101384712 >> 172.21.15.46:0/1101384712 conn(0x7f5fc80769d0 msgr2=0x7f5fc8076de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:13:42.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.676+0000 7f5fccc14640 1 -- 172.21.15.46:0/1101384712 shutdown_connections 2024-03-31T23:13:42.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.676+0000 7f5fccc14640 1 -- 172.21.15.46:0/1101384712 wait complete. 2024-03-31T23:13:42.680 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.676+0000 7f5fccc14640 1 Processor -- start 2024-03-31T23:13:42.680 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.676+0000 7f5fccc14640 1 -- start start 2024-03-31T23:13:42.680 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.677+0000 7f5fccc14640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5fc8100600 0x7f5fc8197e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:13:42.680 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.677+0000 7f5fccc14640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5fc8100fc0 0x7f5fc8198390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:13:42.680 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.677+0000 7f5fccc14640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5fc81066a0 0x7f5fc819c710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:13:42.680 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.677+0000 7f5fccc14640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f5fc81108d0 con 0x7f5fc8100fc0 2024-03-31T23:13:42.680 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.677+0000 7f5fccc14640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f5fc8110750 con 0x7f5fc8100600 2024-03-31T23:13:42.680 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.677+0000 7f5fccc14640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f5fc8110a50 con 0x7f5fc81066a0 2024-03-31T23:13:42.680 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.677+0000 7f5fc5d9b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5fc8100fc0 0x7f5fc8198390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:13:42.680 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.677+0000 7f5fc5d9b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5fc8100fc0 0x7f5fc8198390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60106/0 (socket says 172.21.15.46:60106) 2024-03-31T23:13:42.680 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.677+0000 7f5fc5d9b640 1 -- 172.21.15.46:0/3420875678 learned_addr learned my addr 172.21.15.46:0/3420875678 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:13:42.680 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.677+0000 7f5fc659c640 1 --2- 172.21.15.46:0/3420875678 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5fc8100600 0x7f5fc8197e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:13:42.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.677+0000 7f5fc6d9d640 1 --2- 172.21.15.46:0/3420875678 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5fc81066a0 0x7f5fc819c710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:13:42.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.677+0000 7f5fc5d9b640 1 -- 172.21.15.46:0/3420875678 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5fc81066a0 msgr2=0x7f5fc819c710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:13:42.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.677+0000 7f5fc5d9b640 1 --2- 172.21.15.46:0/3420875678 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5fc81066a0 0x7f5fc819c710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:42.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.677+0000 7f5fc5d9b640 1 -- 172.21.15.46:0/3420875678 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5fc8100600 msgr2=0x7f5fc8197e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:13:42.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.677+0000 7f5fc5d9b640 1 --2- 172.21.15.46:0/3420875678 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5fc8100600 0x7f5fc8197e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:42.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.677+0000 7f5fc5d9b640 1 -- 172.21.15.46:0/3420875678 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5fc819ce40 con 0x7f5fc8100fc0 2024-03-31T23:13:42.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.677+0000 7f5fc6d9d640 1 --2- 172.21.15.46:0/3420875678 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5fc81066a0 0x7f5fc819c710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T23:13:42.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.677+0000 7f5fc5d9b640 1 --2- 172.21.15.46:0/3420875678 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5fc8100fc0 0x7f5fc8198390 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f5fbc00e9f0 tx=0x7f5fbc00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:13:42.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.677+0000 7f5fb77fe640 1 -- 172.21.15.46:0/3420875678 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fbc00cd80 con 0x7f5fc8100fc0 2024-03-31T23:13:42.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.678+0000 7f5fccc14640 1 -- 172.21.15.46:0/3420875678 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5fc819d130 con 0x7f5fc8100fc0 2024-03-31T23:13:42.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.678+0000 7f5fb77fe640 1 -- 172.21.15.46:0/3420875678 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5fbc0054d0 con 0x7f5fc8100fc0 2024-03-31T23:13:42.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.678+0000 7f5fb77fe640 1 -- 172.21.15.46:0/3420875678 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fbc010490 con 0x7f5fc8100fc0 2024-03-31T23:13:42.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.678+0000 7f5fccc14640 1 -- 172.21.15.46:0/3420875678 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5fc81a49c0 con 0x7f5fc8100fc0 2024-03-31T23:13:42.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.680+0000 7f5fb77fe640 1 -- 172.21.15.46:0/3420875678 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 25) ==== 137451+0+0 (secure 0 0 0) 0x7f5fbc005020 con 0x7f5fc8100fc0 2024-03-31T23:13:42.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.680+0000 7f5fcf7f0740 1 Processor -- start 2024-03-31T23:13:42.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.680+0000 7f5fcf7f0740 1 -- start start 2024-03-31T23:13:42.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.680+0000 7f5fcf7f0740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c18cef2470 0x55c18cef2850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:13:42.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.680+0000 7f5fcf7f0740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55c18cef2d90 0x55c18cefb430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:13:42.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.680+0000 7f5fcf7f0740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c18cefb970 0x55c18cefdd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:13:42.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.680+0000 7f5fcf7f0740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55c18ce561b0 con 0x55c18cefb970 2024-03-31T23:13:42.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.681+0000 7f5fcf7f0740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55c18cdfec00 con 0x55c18cef2d90 2024-03-31T23:13:42.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.681+0000 7f5fcf7f0740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55c18ce52250 con 0x55c18cef2470 2024-03-31T23:13:42.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.681+0000 7f5fb77fe640 1 --2- 172.21.15.46:0/3420875678 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f5fb00a2610 0x7f5fb00a4ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:13:42.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.681+0000 7f5fc5d9b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c18cefb970 0x55c18cefdd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:13:42.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.681+0000 7f5fc6d9d640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c18cef2470 0x55c18cef2850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:13:42.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.681+0000 7f5fc6d9d640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c18cef2470 0x55c18cef2850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.46:49092/0 (socket says 172.21.15.46:49092) 2024-03-31T23:13:42.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.681+0000 7f5fc6d9d640 1 -- 172.21.15.46:0/2811150515 learned_addr learned my addr 172.21.15.46:0/2811150515 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:13:42.698 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.681+0000 7f5fc659c640 1 --2- 172.21.15.46:0/2811150515 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55c18cef2d90 0x55c18cefb430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:13:42.698 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.681+0000 7f5fc6d9d640 1 --2- 172.21.15.46:0/3420875678 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f5fb00a2610 0x7f5fb00a4ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:13:42.698 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.681+0000 7f5fb77fe640 1 -- 172.21.15.46:0/3420875678 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7f5fbc0cf300 con 0x7f5fc8100fc0 2024-03-31T23:13:42.698 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.681+0000 7f5fc6d9d640 1 -- 172.21.15.46:0/2811150515 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55c18cef2d90 msgr2=0x55c18cefb430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:13:42.698 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.681+0000 7f5fc6d9d640 1 --2- 172.21.15.46:0/2811150515 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55c18cef2d90 0x55c18cefb430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:42.699 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.681+0000 7f5fc6d9d640 1 -- 172.21.15.46:0/2811150515 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c18cefb970 msgr2=0x55c18cefdd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:13:42.699 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.681+0000 7f5fc6d9d640 1 --2- 172.21.15.46:0/2811150515 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c18cefb970 0x55c18cefdd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:42.699 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.681+0000 7f5fc6d9d640 1 -- 172.21.15.46:0/2811150515 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c18ce1e3d0 con 0x55c18cef2470 2024-03-31T23:13:42.699 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.681+0000 7f5fc659c640 1 --2- 172.21.15.46:0/2811150515 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55c18cef2d90 0x55c18cefb430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:13:42.699 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.681+0000 7f5fc6d9d640 1 --2- 172.21.15.46:0/3420875678 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f5fb00a2610 0x7f5fb00a4ad0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f5fc0009430 tx=0x7f5fc0041000 comp rx=0 tx=0).ready entity=mgr.15654 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:13:42.699 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.681+0000 7f5fc6d9d640 1 --2- 172.21.15.46:0/2811150515 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c18cef2470 0x55c18cef2850 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f5fc0005fe0 tx=0x7f5fc0032780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c506ddb8f7751d11 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:13:42.699 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.682+0000 7f5fb4ff9640 1 -- 172.21.15.46:0/2811150515 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fc0051070 con 0x55c18cef2470 2024-03-31T23:13:42.700 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.682+0000 7f5fb4ff9640 1 -- 172.21.15.46:0/2811150515 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5fc0048070 con 0x55c18cef2470 2024-03-31T23:13:42.700 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.682+0000 7f5fb4ff9640 1 -- 172.21.15.46:0/2811150515 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fc0039040 con 0x55c18cef2470 2024-03-31T23:13:42.700 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.682+0000 7f5fcf7f0740 1 -- 172.21.15.46:0/2811150515 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c18cef2470 msgr2=0x55c18cef2850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:13:42.700 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.682+0000 7f5fcf7f0740 1 --2- 172.21.15.46:0/2811150515 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c18cef2470 0x55c18cef2850 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f5fc0005fe0 tx=0x7f5fc0032780 comp rx=0 tx=0).stop 2024-03-31T23:13:42.700 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.682+0000 7f5fcf7f0740 1 -- 172.21.15.46:0/2811150515 shutdown_connections 2024-03-31T23:13:42.700 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.682+0000 7f5fcf7f0740 1 --2- 172.21.15.46:0/2811150515 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c18cefb970 0x55c18cefdd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:42.700 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.682+0000 7f5fcf7f0740 1 --2- 172.21.15.46:0/2811150515 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55c18cef2d90 0x55c18cefb430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:42.700 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.682+0000 7f5fcf7f0740 1 --2- 172.21.15.46:0/2811150515 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c18cef2470 0x55c18cef2850 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:42.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.682+0000 7f5fcf7f0740 1 -- 172.21.15.46:0/2811150515 >> 172.21.15.46:0/2811150515 conn(0x55c18ceecba0 msgr2=0x55c18ceecf80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:13:42.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.682+0000 7f5fcf7f0740 1 -- 172.21.15.46:0/2811150515 shutdown_connections 2024-03-31T23:13:42.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.682+0000 7f5fcf7f0740 1 -- 172.21.15.46:0/2811150515 wait complete. 2024-03-31T23:13:42.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.683+0000 7f5fcf7f0740 1 -- start start 2024-03-31T23:13:42.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.683+0000 7f5fcf7f0740 1 Processor -- start 2024-03-31T23:13:42.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.683+0000 7f5fcf7f0740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c18cef2470 0x55c18cef8a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:13:42.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.683+0000 7f5fcf7f0740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55c18cef2d90 0x55c18cef8f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:13:42.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.683+0000 7f5fcf7f0740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c18cefb970 0x55c18cef94a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:13:42.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.683+0000 7f5fcf7f0740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55c18cefe700 con 0x55c18cef2470 2024-03-31T23:13:42.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.683+0000 7f5fcf7f0740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55c18cdf4470 con 0x55c18cef2d90 2024-03-31T23:13:42.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.683+0000 7f5fcf7f0740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55c18cefe880 con 0x55c18cefb970 2024-03-31T23:13:42.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.683+0000 7f5fcf7f0740 10 client.0 _ll_register_callbacks cb 0x55c18ce11cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T23:13:42.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.683+0000 7f5fc659c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c18cef2470 0x55c18cef8a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:13:42.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.683+0000 7f5fc659c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c18cef2470 0x55c18cef8a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60120/0 (socket says 172.21.15.46:60120) 2024-03-31T23:13:42.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.683+0000 7f5fc659c640 1 -- 172.21.15.46:0/2936910350 learned_addr learned my addr 172.21.15.46:0/2936910350 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:13:42.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.683+0000 7f5fc5d9b640 1 --2- 172.21.15.46:0/2936910350 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55c18cef2d90 0x55c18cef8f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:13:42.703 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.683+0000 7f5fc6d9d640 1 --2- 172.21.15.46:0/2936910350 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c18cefb970 0x55c18cef94a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:13:42.703 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.683+0000 7f5fc659c640 1 -- 172.21.15.46:0/2936910350 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c18cefb970 msgr2=0x55c18cef94a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:13:42.703 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.683+0000 7f5fc659c640 1 --2- 172.21.15.46:0/2936910350 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c18cefb970 0x55c18cef94a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:42.703 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.683+0000 7f5fc659c640 1 -- 172.21.15.46:0/2936910350 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55c18cef2d90 msgr2=0x55c18cef8f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:13:42.703 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.684+0000 7f5fc659c640 1 --2- 172.21.15.46:0/2936910350 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55c18cef2d90 0x55c18cef8f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:42.704 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.684+0000 7f5fc659c640 1 -- 172.21.15.46:0/2936910350 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c18cb6a3f0 con 0x55c18cef2470 2024-03-31T23:13:42.704 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.684+0000 7f5fc5d9b640 1 --2- 172.21.15.46:0/2936910350 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55c18cef2d90 0x55c18cef8f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:13:42.704 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.684+0000 7f5fc659c640 1 --2- 172.21.15.46:0/2936910350 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c18cef2470 0x55c18cef8a20 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f5fb8012720 tx=0x7f5fb8012be0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf98a1de7cf67789 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:13:42.704 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.684+0000 7f5fadffb640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-31T23:13:42.704 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.684+0000 7f5fadffb640 1 -- 172.21.15.46:0/2936910350 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fb8004030 con 0x55c18cef2470 2024-03-31T23:13:42.704 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.684+0000 7f5fadffb640 1 -- 172.21.15.46:0/2936910350 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7f5fb80041d0 con 0x55c18cef2470 2024-03-31T23:13:42.705 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.684+0000 7f5fcf7f0740 1 -- 172.21.15.46:0/2936910350 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5fc0048070 con 0x55c18cef2470 2024-03-31T23:13:42.705 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.684+0000 7f5fadffb640 1 -- 172.21.15.46:0/2936910350 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f5fb8004410 con 0x55c18cef2470 2024-03-31T23:13:42.705 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.684+0000 7f5fcf7f0740 10 client.16275 fetch_fsmap learned FSMap version 28 2024-03-31T23:13:42.705 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.684+0000 7f5fcf7f0740 1 -- 172.21.15.46:0/2936910350 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55c18cc347d0 con 0x55c18cef2470 2024-03-31T23:13:42.705 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.684+0000 7f5fadffb640 1 -- 172.21.15.46:0/2936910350 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5fc0048070 con 0x55c18cef2470 2024-03-31T23:13:42.706 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.684+0000 7f5fadffb640 1 -- 172.21.15.46:0/2936910350 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f5fb8003b20 con 0x55c18cef2470 2024-03-31T23:13:42.706 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.685+0000 7f5fcf7f0740 10 client.16275 fetch_fsmap finished waiting for FSMap version 28 2024-03-31T23:13:42.706 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.685+0000 7f5fcf7f0740 10 client.16275 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-03-31T23:13:42.706 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.685+0000 7f5fcf7f0740 20 client.16275 populate_metadata read hostname 'smithi046' 2024-03-31T23:13:42.706 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.685+0000 7f5fcf7f0740 1 --2- 172.21.15.46:0/2936910350 >> [v2:172.21.15.142:6836/2559258399,v1:172.21.15.142:6837/2559258399] conn(0x55c18cf0b9f0 0x55c18cf0dde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:13:42.706 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.685+0000 7f5fcf7f0740 4 client.16275 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-31T23:13:42.707 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.685+0000 7f5fcf7f0740 1 -- 172.21.15.46:0/2936910350 --> [v2:172.21.15.142:6836/2559258399,v1:172.21.15.142:6837/2559258399] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c18ce636f0 con 0x55c18cf0b9f0 2024-03-31T23:13:42.707 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.685+0000 7f5fc5d9b640 1 --2- 172.21.15.46:0/2936910350 >> [v2:172.21.15.142:6836/2559258399,v1:172.21.15.142:6837/2559258399] conn(0x55c18cf0b9f0 0x55c18cf0dde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:13:42.707 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.685+0000 7f5fadffb640 10 client.16275 ms_handle_connect on v2:172.21.15.142:6836/2559258399 2024-03-31T23:13:42.707 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.685+0000 7f5fc5d9b640 1 --2- 172.21.15.46:0/2936910350 >> [v2:172.21.15.142:6836/2559258399,v1:172.21.15.142:6837/2559258399] conn(0x55c18cf0b9f0 0x55c18cf0dde0 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=d295deaf904b615 server_cookie=7e482a306e74cd4c in_seq=0 out_seq=0 2024-03-31T23:13:42.707 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.686+0000 7f5fadffb640 1 -- 172.21.15.46:0/2936910350 <== mds.1 v2:172.21.15.142:6836/2559258399 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55c18ce636f0 con 0x55c18cf0b9f0 2024-03-31T23:13:42.707 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.686+0000 7f5fadffb640 10 client.16275 handle_command_reply: tid=0 2024-03-31T23:13:42.707 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.686+0000 7f5fcf7f0740 1 client.16275 shutdown 2024-03-31T23:13:42.708 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.687+0000 7f5fcf7f0740 1 -- 172.21.15.46:0/2936910350 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c18cef2470 msgr2=0x55c18cef8a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:13:42.708 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.687+0000 7f5fcf7f0740 1 --2- 172.21.15.46:0/2936910350 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c18cef2470 0x55c18cef8a20 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f5fb8012720 tx=0x7f5fb8012be0 comp rx=0 tx=0).stop 2024-03-31T23:13:42.708 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.687+0000 7f5fcf7f0740 1 -- 172.21.15.46:0/2936910350 shutdown_connections 2024-03-31T23:13:42.708 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.687+0000 7f5fcf7f0740 1 --2- 172.21.15.46:0/2936910350 >> [v2:172.21.15.142:6836/2559258399,v1:172.21.15.142:6837/2559258399] conn(0x55c18cf0b9f0 0x55c18cf0dde0 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:42.708 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.687+0000 7f5fcf7f0740 1 --2- 172.21.15.46:0/2936910350 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c18cefb970 0x55c18cef94a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:42.708 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.687+0000 7f5fcf7f0740 1 --2- 172.21.15.46:0/2936910350 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55c18cef2d90 0x55c18cef8f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:42.708 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.687+0000 7f5fcf7f0740 1 --2- 172.21.15.46:0/2936910350 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c18cef2470 0x55c18cef8a20 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:42.708 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.687+0000 7f5fcf7f0740 1 -- 172.21.15.46:0/2936910350 >> 172.21.15.46:0/2936910350 conn(0x55c18ceecba0 msgr2=0x55c18ceee8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:13:42.708 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.687+0000 7f5fadffb640 0 client.16275 ms_handle_reset on v2:172.21.15.142:6836/2559258399 2024-03-31T23:13:42.708 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.687+0000 7f5fcf7f0740 1 -- 172.21.15.46:0/2936910350 shutdown_connections 2024-03-31T23:13:42.708 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.687+0000 7f5fcf7f0740 1 -- 172.21.15.46:0/2936910350 wait complete. 2024-03-31T23:13:42.708 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.687+0000 7f5fcf7f0740 20 client.16275 trim_cache size 0 max 16384 2024-03-31T23:13:42.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.702+0000 7f5fcf7f0740 1 Processor -- start 2024-03-31T23:13:42.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.702+0000 7f5fcf7f0740 1 -- start start 2024-03-31T23:13:42.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.702+0000 7f5fcf7f0740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c18cef2470 0x55c18cefb150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:13:42.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.702+0000 7f5fcf7f0740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55c18cef2d90 0x55c18cf074a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:13:42.715 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.702+0000 7f5fcf7f0740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c18cefb970 0x55c18ceee6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:13:42.715 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.702+0000 7f5fcf7f0740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f5fc81b1410 con 0x55c18cefb970 2024-03-31T23:13:42.715 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.702+0000 7f5fcf7f0740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f5fc81aece0 con 0x55c18cef2d90 2024-03-31T23:13:42.715 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.702+0000 7f5fcf7f0740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f5fc81acb30 con 0x55c18cef2470 2024-03-31T23:13:42.715 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.702+0000 7f5fc6d9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c18cefb970 0x55c18ceee6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:13:42.715 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.702+0000 7f5fc6d9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c18cefb970 0x55c18ceee6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60126/0 (socket says 172.21.15.46:60126) 2024-03-31T23:13:42.715 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.702+0000 7f5fc6d9d640 1 -- 172.21.15.46:0/2009181357 learned_addr learned my addr 172.21.15.46:0/2009181357 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:13:42.715 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.702+0000 7f5fc5d9b640 1 --2- 172.21.15.46:0/2009181357 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55c18cef2d90 0x55c18cf074a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:13:42.715 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.702+0000 7f5fc659c640 1 --2- 172.21.15.46:0/2009181357 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c18cef2470 0x55c18cefb150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:13:42.715 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.702+0000 7f5fc6d9d640 1 -- 172.21.15.46:0/2009181357 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c18cef2470 msgr2=0x55c18cefb150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:13:42.715 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.702+0000 7f5fc6d9d640 1 --2- 172.21.15.46:0/2009181357 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c18cef2470 0x55c18cefb150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:42.715 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.702+0000 7f5fc6d9d640 1 -- 172.21.15.46:0/2009181357 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55c18cef2d90 msgr2=0x55c18cf074a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:13:42.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.702+0000 7f5fc6d9d640 1 --2- 172.21.15.46:0/2009181357 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55c18cef2d90 0x55c18cf074a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:42.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.702+0000 7f5fc6d9d640 1 -- 172.21.15.46:0/2009181357 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c18cef6f70 con 0x55c18cefb970 2024-03-31T23:13:42.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.703+0000 7f5fc6d9d640 1 --2- 172.21.15.46:0/2009181357 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c18cefb970 0x55c18ceee6e0 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f5fc002c6d0 tx=0x7f5fc0032400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e8b2db784a059983 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:13:42.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.703+0000 7f5faffff640 1 -- 172.21.15.46:0/2009181357 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fc0059070 con 0x55c18cefb970 2024-03-31T23:13:42.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.703+0000 7f5faffff640 1 -- 172.21.15.46:0/2009181357 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5fc0048050 con 0x55c18cefb970 2024-03-31T23:13:42.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.703+0000 7f5faffff640 1 -- 172.21.15.46:0/2009181357 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fc0039070 con 0x55c18cefb970 2024-03-31T23:13:42.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.703+0000 7f5fcf7f0740 1 -- 172.21.15.46:0/2009181357 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c18cefb970 msgr2=0x55c18ceee6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:13:42.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.703+0000 7f5fcf7f0740 1 --2- 172.21.15.46:0/2009181357 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c18cefb970 0x55c18ceee6e0 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f5fc002c6d0 tx=0x7f5fc0032400 comp rx=0 tx=0).stop 2024-03-31T23:13:42.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.703+0000 7f5fcf7f0740 1 -- 172.21.15.46:0/2009181357 shutdown_connections 2024-03-31T23:13:42.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.703+0000 7f5fcf7f0740 1 --2- 172.21.15.46:0/2009181357 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c18cefb970 0x55c18ceee6e0 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:42.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.703+0000 7f5fcf7f0740 1 --2- 172.21.15.46:0/2009181357 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55c18cef2d90 0x55c18cf074a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:42.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.703+0000 7f5fcf7f0740 1 --2- 172.21.15.46:0/2009181357 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c18cef2470 0x55c18cefb150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:42.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.703+0000 7f5fcf7f0740 1 -- 172.21.15.46:0/2009181357 >> 172.21.15.46:0/2009181357 conn(0x55c18ceecba0 msgr2=0x55c18ceedfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:13:42.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.704+0000 7f5fcf7f0740 1 -- 172.21.15.46:0/2009181357 shutdown_connections 2024-03-31T23:13:42.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.704+0000 7f5fcf7f0740 1 -- 172.21.15.46:0/2009181357 wait complete. 2024-03-31T23:13:42.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.704+0000 7f5fcf7f0740 1 -- start start 2024-03-31T23:13:42.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.704+0000 7f5fcf7f0740 1 Processor -- start 2024-03-31T23:13:42.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.704+0000 7f5fcf7f0740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c18cef2470 0x55c18ceff330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:13:42.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.704+0000 7f5fcf7f0740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55c18cef2d90 0x55c18ceff870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:13:42.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.704+0000 7f5fcf7f0740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c18cefb970 0x55c18ceffdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:13:42.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.704+0000 7f5fcf7f0740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f5fc81ad1a0 con 0x55c18cef2470 2024-03-31T23:13:42.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.704+0000 7f5fcf7f0740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f5fc81ae610 con 0x55c18cef2d90 2024-03-31T23:13:42.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.704+0000 7f5fcf7f0740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f5fc81ad870 con 0x55c18cefb970 2024-03-31T23:13:42.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.704+0000 7f5fcf7f0740 10 client.0 _ll_register_callbacks cb 0x7f5fb8009ee0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T23:13:42.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.704+0000 7f5fc659c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c18cef2470 0x55c18ceff330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:13:42.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.704+0000 7f5fc659c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c18cef2470 0x55c18ceff330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60136/0 (socket says 172.21.15.46:60136) 2024-03-31T23:13:42.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.704+0000 7f5fc659c640 1 -- 172.21.15.46:0/2727646214 learned_addr learned my addr 172.21.15.46:0/2727646214 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:13:42.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.704+0000 7f5fc5d9b640 1 --2- 172.21.15.46:0/2727646214 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55c18cef2d90 0x55c18ceff870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:13:42.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.704+0000 7f5fc6d9d640 1 --2- 172.21.15.46:0/2727646214 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c18cefb970 0x55c18ceffdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:13:42.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.704+0000 7f5fc659c640 1 -- 172.21.15.46:0/2727646214 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c18cefb970 msgr2=0x55c18ceffdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:13:42.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.704+0000 7f5fc659c640 1 --2- 172.21.15.46:0/2727646214 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c18cefb970 0x55c18ceffdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:42.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.704+0000 7f5fc659c640 1 -- 172.21.15.46:0/2727646214 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55c18cef2d90 msgr2=0x55c18ceff870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:13:42.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.704+0000 7f5fc659c640 1 --2- 172.21.15.46:0/2727646214 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55c18cef2d90 0x55c18ceff870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:42.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.704+0000 7f5fc659c640 1 -- 172.21.15.46:0/2727646214 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c18cf70cc0 con 0x55c18cef2470 2024-03-31T23:13:42.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.704+0000 7f5fc5d9b640 1 --2- 172.21.15.46:0/2727646214 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55c18cef2d90 0x55c18ceff870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:13:42.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.705+0000 7f5fc659c640 1 --2- 172.21.15.46:0/2727646214 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c18cef2470 0x55c18ceff330 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f5fb801c550 tx=0x7f5fb80041f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a58ab93bd701a86 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:13:42.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.705+0000 7f5fadffb640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-31T23:13:42.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.705+0000 7f5fadffb640 1 -- 172.21.15.46:0/2727646214 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fb800fab0 con 0x55c18cef2470 2024-03-31T23:13:42.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.705+0000 7f5fadffb640 1 -- 172.21.15.46:0/2727646214 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7f5fb800fc50 con 0x55c18cef2470 2024-03-31T23:13:42.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.705+0000 7f5fcf7f0740 1 -- 172.21.15.46:0/2727646214 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5fc0048050 con 0x55c18cef2470 2024-03-31T23:13:42.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.705+0000 7f5fadffb640 1 -- 172.21.15.46:0/2727646214 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f5fb8037050 con 0x55c18cef2470 2024-03-31T23:13:42.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.705+0000 7f5fcf7f0740 10 client.16281 fetch_fsmap learned FSMap version 28 2024-03-31T23:13:42.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.705+0000 7f5fcf7f0740 1 -- 172.21.15.46:0/2727646214 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55c18cf71f00 con 0x55c18cef2470 2024-03-31T23:13:42.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.705+0000 7f5fadffb640 1 -- 172.21.15.46:0/2727646214 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5fc0048050 con 0x55c18cef2470 2024-03-31T23:13:42.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.705+0000 7f5fadffb640 1 -- 172.21.15.46:0/2727646214 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f5fb801c610 con 0x55c18cef2470 2024-03-31T23:13:42.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.705+0000 7f5fcf7f0740 10 client.16281 fetch_fsmap finished waiting for FSMap version 28 2024-03-31T23:13:42.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.705+0000 7f5fcf7f0740 10 client.16281 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-03-31T23:13:42.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.705+0000 7f5fcf7f0740 20 client.16281 populate_metadata read hostname 'smithi046' 2024-03-31T23:13:42.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.705+0000 7f5fcf7f0740 1 --2- 172.21.15.46:0/2727646214 >> [v2:172.21.15.142:6836/2559258399,v1:172.21.15.142:6837/2559258399] conn(0x55c18cf726d0 0x55c18cf74af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:13:42.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.705+0000 7f5fcf7f0740 4 client.16281 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-31T23:13:42.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.705+0000 7f5fcf7f0740 1 -- 172.21.15.46:0/2727646214 --> [v2:172.21.15.142:6836/2559258399,v1:172.21.15.142:6837/2559258399] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55c18ce5da40 con 0x55c18cf726d0 2024-03-31T23:13:42.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.706+0000 7f5fc5d9b640 1 --2- 172.21.15.46:0/2727646214 >> [v2:172.21.15.142:6836/2559258399,v1:172.21.15.142:6837/2559258399] conn(0x55c18cf726d0 0x55c18cf74af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:13:42.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.706+0000 7f5fc5d9b640 1 --2- 172.21.15.46:0/2727646214 >> [v2:172.21.15.142:6836/2559258399,v1:172.21.15.142:6837/2559258399] conn(0x55c18cf726d0 0x55c18cf74af0 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=b1cc0946cdbf57a server_cookie=31665e9429fdc8c0 in_seq=0 out_seq=0 2024-03-31T23:13:42.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.706+0000 7f5fadffb640 10 client.16281 ms_handle_connect on v2:172.21.15.142:6836/2559258399 2024-03-31T23:13:42.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.706+0000 7f5fadffb640 1 -- 172.21.15.46:0/2727646214 <== mds.1 v2:172.21.15.142:6836/2559258399 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55c18ce5da40 con 0x55c18cf726d0 2024-03-31T23:13:42.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.706+0000 7f5fadffb640 10 client.16281 handle_command_reply: tid=0 2024-03-31T23:13:42.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.706+0000 7f5fcf7f0740 1 client.16281 shutdown 2024-03-31T23:13:42.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.706+0000 7f5fcf7f0740 1 -- 172.21.15.46:0/2727646214 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c18cef2470 msgr2=0x55c18ceff330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:13:42.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.706+0000 7f5fcf7f0740 1 --2- 172.21.15.46:0/2727646214 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c18cef2470 0x55c18ceff330 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f5fb801c550 tx=0x7f5fb80041f0 comp rx=0 tx=0).stop 2024-03-31T23:13:42.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.706+0000 7f5fcf7f0740 1 -- 172.21.15.46:0/2727646214 shutdown_connections 2024-03-31T23:13:42.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.706+0000 7f5fcf7f0740 1 --2- 172.21.15.46:0/2727646214 >> [v2:172.21.15.142:6836/2559258399,v1:172.21.15.142:6837/2559258399] conn(0x55c18cf726d0 0x55c18cf74af0 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:42.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.706+0000 7f5fcf7f0740 1 --2- 172.21.15.46:0/2727646214 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c18cefb970 0x55c18ceffdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:42.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.706+0000 7f5fcf7f0740 1 --2- 172.21.15.46:0/2727646214 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55c18cef2d90 0x55c18ceff870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:42.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.706+0000 7f5fadffb640 0 client.16281 ms_handle_reset on v2:172.21.15.142:6836/2559258399 2024-03-31T23:13:42.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.706+0000 7f5fcf7f0740 1 --2- 172.21.15.46:0/2727646214 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c18cef2470 0x55c18ceff330 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:42.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.706+0000 7f5fcf7f0740 1 -- 172.21.15.46:0/2727646214 >> 172.21.15.46:0/2727646214 conn(0x55c18ceecba0 msgr2=0x55c18ceee860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:13:42.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.706+0000 7f5fcf7f0740 1 -- 172.21.15.46:0/2727646214 shutdown_connections 2024-03-31T23:13:42.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.706+0000 7f5fcf7f0740 1 -- 172.21.15.46:0/2727646214 wait complete. 2024-03-31T23:13:42.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.706+0000 7f5fcf7f0740 20 client.16281 trim_cache size 0 max 16384 2024-03-31T23:13:42.721 INFO:teuthology.orchestra.run.smithi046.stdout:[] 2024-03-31T23:13:42.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.707+0000 7f5fccc14640 1 -- 172.21.15.46:0/3420875678 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f5fb00a2610 msgr2=0x7f5fb00a4ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:13:42.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.707+0000 7f5fccc14640 1 --2- 172.21.15.46:0/3420875678 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f5fb00a2610 0x7f5fb00a4ad0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f5fc0009430 tx=0x7f5fc0041000 comp rx=0 tx=0).stop 2024-03-31T23:13:42.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.707+0000 7f5fccc14640 1 -- 172.21.15.46:0/3420875678 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5fc8100fc0 msgr2=0x7f5fc8198390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:13:42.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.707+0000 7f5fccc14640 1 --2- 172.21.15.46:0/3420875678 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5fc8100fc0 0x7f5fc8198390 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f5fbc00e9f0 tx=0x7f5fbc00edf0 comp rx=0 tx=0).stop 2024-03-31T23:13:42.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.707+0000 7f5fccc14640 1 -- 172.21.15.46:0/3420875678 shutdown_connections 2024-03-31T23:13:42.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.707+0000 7f5fccc14640 1 --2- 172.21.15.46:0/3420875678 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f5fb00a2610 0x7f5fb00a4ad0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:42.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.707+0000 7f5fccc14640 1 --2- 172.21.15.46:0/3420875678 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5fc81066a0 0x7f5fc819c710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:42.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.707+0000 7f5fccc14640 1 --2- 172.21.15.46:0/3420875678 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5fc8100fc0 0x7f5fc8198390 unknown :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:42.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.707+0000 7f5fccc14640 1 --2- 172.21.15.46:0/3420875678 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5fc8100600 0x7f5fc8197e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:42.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.707+0000 7f5fccc14640 1 -- 172.21.15.46:0/3420875678 >> 172.21.15.46:0/3420875678 conn(0x7f5fc80769d0 msgr2=0x7f5fc80ff570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:13:42.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.707+0000 7f5fccc14640 1 -- 172.21.15.46:0/3420875678 shutdown_connections 2024-03-31T23:13:42.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.707+0000 7f5fccc14640 1 -- 172.21.15.46:0/3420875678 wait complete. 2024-03-31T23:13:42.723 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:13:42 smithi046 ceph-mon[32498]: pgmap v62: 129 pgs: 129 active+clean; 34 MiB data, 499 MiB used, 1.0 TiB / 1.0 TiB avail; 246 KiB/s rd, 1.2 MiB/s wr, 120 op/s 2024-03-31T23:13:42.723 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:13:42 smithi046 ceph-mon[32498]: from='client.? 172.21.15.46:0/2126680105' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-31T23:13:42.724 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-31T23:13:42.795 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:13:42 smithi162 ceph-mon[36237]: pgmap v62: 129 pgs: 129 active+clean; 34 MiB data, 499 MiB used, 1.0 TiB / 1.0 TiB avail; 246 KiB/s rd, 1.2 MiB/s wr, 120 op/s 2024-03-31T23:13:42.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:13:42 smithi162 ceph-mon[36237]: from='client.? 172.21.15.46:0/2126680105' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-31T23:13:42.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.830+0000 7f2630d16640 1 -- 172.21.15.46:0/523406505 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f262c069310 msgr2=0x7f262c10e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:13:42.832 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.830+0000 7f2630d16640 1 --2- 172.21.15.46:0/523406505 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f262c069310 0x7f262c10e5b0 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f2624009cd0 tx=0x7f262402c0f0 comp rx=0 tx=0).stop 2024-03-31T23:13:42.832 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.830+0000 7f2630d16640 1 -- 172.21.15.46:0/523406505 shutdown_connections 2024-03-31T23:13:42.832 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.830+0000 7f2630d16640 1 --2- 172.21.15.46:0/523406505 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f262c069310 0x7f262c10e5b0 unknown :-1 s=CLOSED pgs=640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:42.832 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.830+0000 7f2630d16640 1 --2- 172.21.15.46:0/523406505 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f262c068950 0x7f262c068dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:42.832 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.830+0000 7f2630d16640 1 --2- 172.21.15.46:0/523406505 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f262c1077a0 0x7f262c107b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:42.832 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.830+0000 7f2630d16640 1 -- 172.21.15.46:0/523406505 >> 172.21.15.46:0/523406505 conn(0x7f262c077750 msgr2=0x7f262c077b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:13:42.832 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.830+0000 7f2630d16640 1 -- 172.21.15.46:0/523406505 shutdown_connections 2024-03-31T23:13:42.832 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.830+0000 7f2630d16640 1 -- 172.21.15.46:0/523406505 wait complete. 2024-03-31T23:13:42.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.831+0000 7f2630d16640 1 Processor -- start 2024-03-31T23:13:42.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.831+0000 7f2630d16640 1 -- start start 2024-03-31T23:13:42.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.831+0000 7f2630d16640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f262c068950 0x7f262c100610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:13:42.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.831+0000 7f2630d16640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f262c069310 0x7f262c100b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:13:42.836 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.831+0000 7f2630d16640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f262c1077a0 0x7f262c101090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:13:42.836 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.831+0000 7f2630d16640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f262c111960 con 0x7f262c069310 2024-03-31T23:13:42.836 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.831+0000 7f2630d16640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f262c1117e0 con 0x7f262c1077a0 2024-03-31T23:13:42.836 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.831+0000 7f2630d16640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f262c111ae0 con 0x7f262c068950 2024-03-31T23:13:42.836 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.832+0000 7f2629d9b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f262c069310 0x7f262c100b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:13:42.836 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.832+0000 7f262a59c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f262c068950 0x7f262c100610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:13:42.836 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.832+0000 7f262a59c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f262c068950 0x7f262c100610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.46:49134/0 (socket says 172.21.15.46:49134) 2024-03-31T23:13:42.836 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.832+0000 7f2629d9b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f262c069310 0x7f262c100b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60154/0 (socket says 172.21.15.46:60154) 2024-03-31T23:13:42.836 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.832+0000 7f262a59c640 1 -- 172.21.15.46:0/2777658547 learned_addr learned my addr 172.21.15.46:0/2777658547 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:13:42.836 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.832+0000 7f262ad9d640 1 --2- 172.21.15.46:0/2777658547 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f262c1077a0 0x7f262c101090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:13:42.836 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.832+0000 7f262a59c640 1 -- 172.21.15.46:0/2777658547 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f262c1077a0 msgr2=0x7f262c101090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:13:42.836 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.832+0000 7f262a59c640 1 --2- 172.21.15.46:0/2777658547 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f262c1077a0 0x7f262c101090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:42.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.832+0000 7f262a59c640 1 -- 172.21.15.46:0/2777658547 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f262c069310 msgr2=0x7f262c100b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:13:42.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.832+0000 7f262a59c640 1 --2- 172.21.15.46:0/2777658547 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f262c069310 0x7f262c100b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:42.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.832+0000 7f262a59c640 1 -- 172.21.15.46:0/2777658547 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f262c1a6340 con 0x7f262c068950 2024-03-31T23:13:42.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.832+0000 7f2629d9b640 1 --2- 172.21.15.46:0/2777658547 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f262c069310 0x7f262c100b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:13:42.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.832+0000 7f262a59c640 1 --2- 172.21.15.46:0/2777658547 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f262c068950 0x7f262c100610 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f261c00e9f0 tx=0x7f261c00eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:13:42.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.832+0000 7f261b7fe640 1 -- 172.21.15.46:0/2777658547 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f261c00cd80 con 0x7f262c068950 2024-03-31T23:13:42.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.832+0000 7f261b7fe640 1 -- 172.21.15.46:0/2777658547 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f261c0108e0 con 0x7f262c068950 2024-03-31T23:13:42.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.832+0000 7f2630d16640 1 -- 172.21.15.46:0/2777658547 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f262c1a6630 con 0x7f262c068950 2024-03-31T23:13:42.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.832+0000 7f261b7fe640 1 -- 172.21.15.46:0/2777658547 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f261c018920 con 0x7f262c068950 2024-03-31T23:13:42.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.832+0000 7f2630d16640 1 -- 172.21.15.46:0/2777658547 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f262c1a6b80 con 0x7f262c068950 2024-03-31T23:13:42.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.835+0000 7f261b7fe640 1 -- 172.21.15.46:0/2777658547 <== mon.1 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 25) ==== 137451+0+0 (secure 0 0 0) 0x7f261c010a90 con 0x7f262c068950 2024-03-31T23:13:42.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.835+0000 7f26339cb740 1 Processor -- start 2024-03-31T23:13:42.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.835+0000 7f26339cb740 1 -- start start 2024-03-31T23:13:42.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.836+0000 7f26339cb740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5605bf9f7470 0x5605bf9f7850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:13:42.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.836+0000 7f26339cb740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5605bf9f7d90 0x5605bfa00430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:13:42.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.836+0000 7f26339cb740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5605bfa00970 0x5605bfa02d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:13:42.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.836+0000 7f26339cb740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x5605bf95b1b0 con 0x5605bfa00970 2024-03-31T23:13:42.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.836+0000 7f26339cb740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x5605bf903c00 con 0x5605bf9f7d90 2024-03-31T23:13:42.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.836+0000 7f26339cb740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5605bf957250 con 0x5605bf9f7470 2024-03-31T23:13:42.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.836+0000 7f262a59c640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5605bf9f7d90 0x5605bfa00430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:13:42.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.836+0000 7f2629d9b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5605bfa00970 0x5605bfa02d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:13:42.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.836+0000 7f262a59c640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5605bf9f7d90 0x5605bfa00430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.46:48824/0 (socket says 172.21.15.46:48824) 2024-03-31T23:13:42.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.836+0000 7f262a59c640 1 -- 172.21.15.46:0/1066461615 learned_addr learned my addr 172.21.15.46:0/1066461615 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:13:42.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.836+0000 7f262ad9d640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5605bf9f7470 0x5605bf9f7850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:13:42.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.836+0000 7f261b7fe640 1 --2- 172.21.15.46:0/2777658547 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f26140a2640 0x7f26140a4b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:13:42.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.836+0000 7f2629d9b640 1 -- 172.21.15.46:0/1066461615 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5605bf9f7470 msgr2=0x5605bf9f7850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:13:42.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.836+0000 7f2629d9b640 1 --2- 172.21.15.46:0/1066461615 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5605bf9f7470 0x5605bf9f7850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:42.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.836+0000 7f2629d9b640 1 -- 172.21.15.46:0/1066461615 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5605bf9f7d90 msgr2=0x5605bfa00430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:13:42.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.836+0000 7f2629d9b640 1 --2- 172.21.15.46:0/1066461615 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5605bf9f7d90 0x5605bfa00430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:42.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.836+0000 7f2629d9b640 1 -- 172.21.15.46:0/1066461615 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5605bf9233d0 con 0x5605bfa00970 2024-03-31T23:13:42.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.836+0000 7f262ad9d640 1 --2- 172.21.15.46:0/2777658547 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f26140a2640 0x7f26140a4b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:13:42.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.836+0000 7f262a59c640 1 --2- 172.21.15.46:0/1066461615 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5605bf9f7d90 0x5605bfa00430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:13:42.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.836+0000 7f261b7fe640 1 -- 172.21.15.46:0/2777658547 <== mon.1 v2:172.21.15.162:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7f261c0cf180 con 0x7f262c068950 2024-03-31T23:13:42.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.836+0000 7f2629d9b640 1 --2- 172.21.15.46:0/1066461615 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5605bfa00970 0x5605bfa02d60 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f2620009fa0 tx=0x7f26200118d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9968abc7e82b68b7 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:13:42.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.837+0000 7f2618ff9640 1 -- 172.21.15.46:0/1066461615 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2620004030 con 0x5605bfa00970 2024-03-31T23:13:42.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.837+0000 7f262ad9d640 1 --2- 172.21.15.46:0/2777658547 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f26140a2640 0x7f26140a4b00 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f2624005cc0 tx=0x7f262402e040 comp rx=0 tx=0).ready entity=mgr.15654 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:13:42.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.837+0000 7f2618ff9640 1 -- 172.21.15.46:0/1066461615 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f26200041d0 con 0x5605bfa00970 2024-03-31T23:13:42.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.837+0000 7f2618ff9640 1 -- 172.21.15.46:0/1066461615 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f262000f800 con 0x5605bfa00970 2024-03-31T23:13:42.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.837+0000 7f26339cb740 1 -- 172.21.15.46:0/1066461615 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5605bfa00970 msgr2=0x5605bfa02d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:13:42.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.837+0000 7f26339cb740 1 --2- 172.21.15.46:0/1066461615 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5605bfa00970 0x5605bfa02d60 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f2620009fa0 tx=0x7f26200118d0 comp rx=0 tx=0).stop 2024-03-31T23:13:42.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.838+0000 7f26339cb740 1 -- 172.21.15.46:0/1066461615 shutdown_connections 2024-03-31T23:13:42.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.838+0000 7f26339cb740 1 --2- 172.21.15.46:0/1066461615 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5605bfa00970 0x5605bfa02d60 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:42.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.838+0000 7f26339cb740 1 --2- 172.21.15.46:0/1066461615 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5605bf9f7d90 0x5605bfa00430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:42.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.838+0000 7f26339cb740 1 --2- 172.21.15.46:0/1066461615 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5605bf9f7470 0x5605bf9f7850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:42.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.838+0000 7f26339cb740 1 -- 172.21.15.46:0/1066461615 >> 172.21.15.46:0/1066461615 conn(0x5605bf9f1ba0 msgr2=0x5605bf9f1f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:13:42.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.838+0000 7f26339cb740 1 -- 172.21.15.46:0/1066461615 shutdown_connections 2024-03-31T23:13:42.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.838+0000 7f26339cb740 1 -- 172.21.15.46:0/1066461615 wait complete. 2024-03-31T23:13:42.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.838+0000 7f26339cb740 1 -- start start 2024-03-31T23:13:42.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.838+0000 7f26339cb740 1 Processor -- start 2024-03-31T23:13:42.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.838+0000 7f26339cb740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5605bf9f7470 0x5605bf9fd9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:13:42.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.838+0000 7f26339cb740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5605bf9f7d90 0x5605bf9fdf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:13:42.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.838+0000 7f26339cb740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5605bfa00970 0x5605bf9fe470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:13:42.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.838+0000 7f26339cb740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x5605bfa03760 con 0x5605bf9f7470 2024-03-31T23:13:42.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.838+0000 7f26339cb740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x5605bf8f9470 con 0x5605bfa00970 2024-03-31T23:13:42.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.838+0000 7f26339cb740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5605bfa038e0 con 0x5605bf9f7d90 2024-03-31T23:13:42.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.838+0000 7f26339cb740 10 client.0 _ll_register_callbacks cb 0x5605bf916cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T23:13:42.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.838+0000 7f262a59c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5605bf9f7470 0x5605bf9fd9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:13:42.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.838+0000 7f262a59c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5605bf9f7470 0x5605bf9fd9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60168/0 (socket says 172.21.15.46:60168) 2024-03-31T23:13:42.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.838+0000 7f262a59c640 1 -- 172.21.15.46:0/4078182985 learned_addr learned my addr 172.21.15.46:0/4078182985 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:13:42.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.839+0000 7f2629d9b640 1 --2- 172.21.15.46:0/4078182985 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5605bf9f7d90 0x5605bf9fdf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:13:42.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.839+0000 7f262ad9d640 1 --2- 172.21.15.46:0/4078182985 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5605bfa00970 0x5605bf9fe470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:13:42.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.839+0000 7f262a59c640 1 -- 172.21.15.46:0/4078182985 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5605bf9f7d90 msgr2=0x5605bf9fdf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:13:42.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.839+0000 7f262a59c640 1 --2- 172.21.15.46:0/4078182985 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5605bf9f7d90 0x5605bf9fdf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:42.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.839+0000 7f262a59c640 1 -- 172.21.15.46:0/4078182985 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5605bfa00970 msgr2=0x5605bf9fe470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:13:42.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.839+0000 7f262a59c640 1 --2- 172.21.15.46:0/4078182985 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5605bfa00970 0x5605bf9fe470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:42.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.839+0000 7f262a59c640 1 -- 172.21.15.46:0/4078182985 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5605bf66f3f0 con 0x5605bf9f7470 2024-03-31T23:13:42.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.839+0000 7f262a59c640 1 --2- 172.21.15.46:0/4078182985 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5605bf9f7470 0x5605bf9fd9f0 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f261c0a39d0 tx=0x7f261c0cc4e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55c4da8186b8cc82 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:13:42.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.839+0000 7f2611ffb640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-31T23:13:42.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.839+0000 7f2611ffb640 1 -- 172.21.15.46:0/4078182985 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f261c09af00 con 0x5605bf9f7470 2024-03-31T23:13:42.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.839+0000 7f2611ffb640 1 -- 172.21.15.46:0/4078182985 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7f261c0bb0d0 con 0x5605bf9f7470 2024-03-31T23:13:42.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.839+0000 7f26339cb740 1 -- 172.21.15.46:0/4078182985 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f26200041d0 con 0x5605bf9f7470 2024-03-31T23:13:42.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.840+0000 7f2611ffb640 1 -- 172.21.15.46:0/4078182985 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f261c0cc940 con 0x5605bf9f7470 2024-03-31T23:13:42.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.840+0000 7f26339cb740 10 client.16293 fetch_fsmap learned FSMap version 28 2024-03-31T23:13:42.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.840+0000 7f26339cb740 1 -- 172.21.15.46:0/4078182985 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5605bf7397d0 con 0x5605bf9f7470 2024-03-31T23:13:42.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.840+0000 7f2611ffb640 1 -- 172.21.15.46:0/4078182985 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f26200041d0 con 0x5605bf9f7470 2024-03-31T23:13:42.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.840+0000 7f2611ffb640 1 -- 172.21.15.46:0/4078182985 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f261c0d1110 con 0x5605bf9f7470 2024-03-31T23:13:42.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.840+0000 7f26339cb740 10 client.16293 fetch_fsmap finished waiting for FSMap version 28 2024-03-31T23:13:42.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.840+0000 7f26339cb740 10 client.16293 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-31T23:13:42.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.840+0000 7f26339cb740 20 client.16293 populate_metadata read hostname 'smithi046' 2024-03-31T23:13:42.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.840+0000 7f26339cb740 1 --2- 172.21.15.46:0/4078182985 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x5605bfa10950 0x5605bfa12dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:13:42.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.840+0000 7f26339cb740 4 client.16293 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-31T23:13:42.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.840+0000 7f26339cb740 1 -- 172.21.15.46:0/4078182985 --> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5605bf9686f0 con 0x5605bfa10950 2024-03-31T23:13:42.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.841+0000 7f2629d9b640 1 --2- 172.21.15.46:0/4078182985 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x5605bfa10950 0x5605bfa12dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:13:42.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.841+0000 7f2629d9b640 1 --2- 172.21.15.46:0/4078182985 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x5605bfa10950 0x5605bfa12dd0 crc :-1 s=READY pgs=89 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d612f00d125376d5 server_cookie=cf193f01c88da180 in_seq=0 out_seq=0 2024-03-31T23:13:42.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.841+0000 7f2611ffb640 10 client.16293 ms_handle_connect on v2:172.21.15.142:6838/2519143918 2024-03-31T23:13:42.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.842+0000 7f2611ffb640 1 -- 172.21.15.46:0/4078182985 <== mds.0 v2:172.21.15.142:6838/2519143918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5605bf9686f0 con 0x5605bfa10950 2024-03-31T23:13:42.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.842+0000 7f2611ffb640 10 client.16293 handle_command_reply: tid=0 2024-03-31T23:13:42.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.842+0000 7f26339cb740 1 client.16293 shutdown 2024-03-31T23:13:42.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.842+0000 7f26339cb740 1 -- 172.21.15.46:0/4078182985 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5605bf9f7470 msgr2=0x5605bf9fd9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:13:42.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.842+0000 7f26339cb740 1 --2- 172.21.15.46:0/4078182985 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5605bf9f7470 0x5605bf9fd9f0 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f261c0a39d0 tx=0x7f261c0cc4e0 comp rx=0 tx=0).stop 2024-03-31T23:13:42.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.842+0000 7f26339cb740 1 -- 172.21.15.46:0/4078182985 shutdown_connections 2024-03-31T23:13:42.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.842+0000 7f26339cb740 1 --2- 172.21.15.46:0/4078182985 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x5605bfa10950 0x5605bfa12dd0 crc :-1 s=READY pgs=89 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:42.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.842+0000 7f26339cb740 1 --2- 172.21.15.46:0/4078182985 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5605bfa00970 0x5605bf9fe470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:42.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.842+0000 7f2611ffb640 0 client.16293 ms_handle_reset on v2:172.21.15.142:6838/2519143918 2024-03-31T23:13:42.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.842+0000 7f26339cb740 1 --2- 172.21.15.46:0/4078182985 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5605bf9f7d90 0x5605bf9fdf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:42.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.842+0000 7f26339cb740 1 --2- 172.21.15.46:0/4078182985 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5605bf9f7470 0x5605bf9fd9f0 unknown :-1 s=CLOSED pgs=642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:42.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.842+0000 7f26339cb740 1 -- 172.21.15.46:0/4078182985 >> 172.21.15.46:0/4078182985 conn(0x5605bf9f1ba0 msgr2=0x5605bf9f38f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:13:42.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.842+0000 7f26339cb740 1 -- 172.21.15.46:0/4078182985 shutdown_connections 2024-03-31T23:13:42.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.842+0000 7f26339cb740 1 -- 172.21.15.46:0/4078182985 wait complete. 2024-03-31T23:13:42.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.842+0000 7f26339cb740 20 client.16293 trim_cache size 0 max 16384 2024-03-31T23:13:42.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.856+0000 7f26339cb740 1 Processor -- start 2024-03-31T23:13:42.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.857+0000 7f26339cb740 1 -- start start 2024-03-31T23:13:42.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.857+0000 7f26339cb740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5605bf9f7470 0x5605bf9fd2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:13:42.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.857+0000 7f26339cb740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5605bf9f7d90 0x5605bf9f04b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:13:42.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.857+0000 7f26339cb740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5605bfa00970 0x5605bfa0c500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:13:42.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.857+0000 7f26339cb740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f262c1b36e0 con 0x5605bf9f7d90 2024-03-31T23:13:42.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.857+0000 7f26339cb740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f262c1b0fb0 con 0x5605bf9f7470 2024-03-31T23:13:42.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.857+0000 7f26339cb740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f262c1aee00 con 0x5605bfa00970 2024-03-31T23:13:42.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.857+0000 7f2629d9b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5605bf9f7d90 0x5605bf9f04b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:13:42.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.857+0000 7f2629d9b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5605bf9f7d90 0x5605bf9f04b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60174/0 (socket says 172.21.15.46:60174) 2024-03-31T23:13:42.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.857+0000 7f2629d9b640 1 -- 172.21.15.46:0/133122735 learned_addr learned my addr 172.21.15.46:0/133122735 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:13:42.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.857+0000 7f262a59c640 1 --2- 172.21.15.46:0/133122735 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5605bf9f7470 0x5605bf9fd2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:13:42.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.857+0000 7f262ad9d640 1 --2- 172.21.15.46:0/133122735 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5605bfa00970 0x5605bfa0c500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:13:42.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.857+0000 7f2629d9b640 1 -- 172.21.15.46:0/133122735 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5605bfa00970 msgr2=0x5605bfa0c500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:13:42.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.857+0000 7f2629d9b640 1 --2- 172.21.15.46:0/133122735 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5605bfa00970 0x5605bfa0c500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:42.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.857+0000 7f2629d9b640 1 -- 172.21.15.46:0/133122735 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5605bf9f7470 msgr2=0x5605bf9fd2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:13:42.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.857+0000 7f2629d9b640 1 --2- 172.21.15.46:0/133122735 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5605bf9f7470 0x5605bf9fd2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:42.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.857+0000 7f2629d9b640 1 -- 172.21.15.46:0/133122735 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5605bfa02d60 con 0x5605bf9f7d90 2024-03-31T23:13:42.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.857+0000 7f2629d9b640 1 --2- 172.21.15.46:0/133122735 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5605bf9f7d90 0x5605bf9f04b0 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f2620009730 tx=0x7f2620009310 comp rx=0 tx=0).ready entity=mon.0 client_cookie=715fd1d4ce14a4d server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:13:42.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.858+0000 7f2613fff640 1 -- 172.21.15.46:0/133122735 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f262001d040 con 0x5605bf9f7d90 2024-03-31T23:13:42.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.858+0000 7f2613fff640 1 -- 172.21.15.46:0/133122735 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f262002f020 con 0x5605bf9f7d90 2024-03-31T23:13:42.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.858+0000 7f2613fff640 1 -- 172.21.15.46:0/133122735 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f262002b3a0 con 0x5605bf9f7d90 2024-03-31T23:13:42.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.858+0000 7f26339cb740 1 -- 172.21.15.46:0/133122735 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5605bf9f7d90 msgr2=0x5605bf9f04b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:13:42.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.858+0000 7f26339cb740 1 --2- 172.21.15.46:0/133122735 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5605bf9f7d90 0x5605bf9f04b0 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f2620009730 tx=0x7f2620009310 comp rx=0 tx=0).stop 2024-03-31T23:13:42.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.858+0000 7f26339cb740 1 -- 172.21.15.46:0/133122735 shutdown_connections 2024-03-31T23:13:42.866 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.858+0000 7f26339cb740 1 --2- 172.21.15.46:0/133122735 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5605bfa00970 0x5605bfa0c500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:42.866 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.858+0000 7f26339cb740 1 --2- 172.21.15.46:0/133122735 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5605bf9f7d90 0x5605bf9f04b0 unknown :-1 s=CLOSED pgs=643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:42.866 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.858+0000 7f26339cb740 1 --2- 172.21.15.46:0/133122735 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5605bf9f7470 0x5605bf9fd2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:42.866 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.858+0000 7f26339cb740 1 -- 172.21.15.46:0/133122735 >> 172.21.15.46:0/133122735 conn(0x5605bf9f1ba0 msgr2=0x5605bf9fe240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:13:42.866 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.858+0000 7f26339cb740 1 -- 172.21.15.46:0/133122735 shutdown_connections 2024-03-31T23:13:42.866 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.858+0000 7f26339cb740 1 -- 172.21.15.46:0/133122735 wait complete. 2024-03-31T23:13:42.866 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.859+0000 7f26339cb740 1 -- start start 2024-03-31T23:13:42.866 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.859+0000 7f26339cb740 1 Processor -- start 2024-03-31T23:13:42.866 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.859+0000 7f26339cb740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5605bf9f7470 0x5605bf9fa7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:13:42.866 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.859+0000 7f26339cb740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5605bf9f7d90 0x5605bf9face0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:13:42.866 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.859+0000 7f26339cb740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5605bfa00970 0x5605bf9fb220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:13:42.866 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.859+0000 7f26339cb740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f262c1af470 con 0x5605bf9f7d90 2024-03-31T23:13:42.866 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.859+0000 7f26339cb740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f262c1b08e0 con 0x5605bf9f7470 2024-03-31T23:13:42.867 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.859+0000 7f26339cb740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f262c1afb40 con 0x5605bfa00970 2024-03-31T23:13:42.867 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.859+0000 7f26339cb740 10 client.0 _ll_register_callbacks cb 0x7f261c0d2920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T23:13:42.867 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.859+0000 7f2629d9b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5605bf9f7d90 0x5605bf9face0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:13:42.867 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.859+0000 7f2629d9b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5605bf9f7d90 0x5605bf9face0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60180/0 (socket says 172.21.15.46:60180) 2024-03-31T23:13:42.867 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.859+0000 7f2629d9b640 1 -- 172.21.15.46:0/1202480005 learned_addr learned my addr 172.21.15.46:0/1202480005 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:13:42.867 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.859+0000 7f262a59c640 1 --2- 172.21.15.46:0/1202480005 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5605bf9f7470 0x5605bf9fa7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:13:42.867 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.859+0000 7f262ad9d640 1 --2- 172.21.15.46:0/1202480005 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5605bfa00970 0x5605bf9fb220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:13:42.867 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.859+0000 7f2629d9b640 1 -- 172.21.15.46:0/1202480005 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5605bfa00970 msgr2=0x5605bf9fb220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:13:42.867 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.859+0000 7f2629d9b640 1 --2- 172.21.15.46:0/1202480005 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5605bfa00970 0x5605bf9fb220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:42.867 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.859+0000 7f2629d9b640 1 -- 172.21.15.46:0/1202480005 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5605bf9f7470 msgr2=0x5605bf9fa7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:13:42.867 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.859+0000 7f2629d9b640 1 --2- 172.21.15.46:0/1202480005 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5605bf9f7470 0x5605bf9fa7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:42.867 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.859+0000 7f2629d9b640 1 -- 172.21.15.46:0/1202480005 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5605bfa75ed0 con 0x5605bf9f7d90 2024-03-31T23:13:42.867 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.860+0000 7f2629d9b640 1 --2- 172.21.15.46:0/1202480005 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5605bf9f7d90 0x5605bf9face0 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f262002e040 tx=0x7f2620005dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=acf70e896bdd4edb server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:13:42.868 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.860+0000 7f2611ffb640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-31T23:13:42.868 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.860+0000 7f2611ffb640 1 -- 172.21.15.46:0/1202480005 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2620039070 con 0x5605bf9f7d90 2024-03-31T23:13:42.868 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.860+0000 7f2611ffb640 1 -- 172.21.15.46:0/1202480005 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7f262002b4f0 con 0x5605bf9f7d90 2024-03-31T23:13:42.868 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.860+0000 7f26339cb740 1 -- 172.21.15.46:0/1202480005 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f262002f020 con 0x5605bf9f7d90 2024-03-31T23:13:42.868 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.860+0000 7f2611ffb640 1 -- 172.21.15.46:0/1202480005 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f2620042050 con 0x5605bf9f7d90 2024-03-31T23:13:42.868 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.860+0000 7f26339cb740 10 client.16299 fetch_fsmap learned FSMap version 28 2024-03-31T23:13:42.868 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.860+0000 7f26339cb740 1 -- 172.21.15.46:0/1202480005 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5605bfa771b0 con 0x5605bf9f7d90 2024-03-31T23:13:42.868 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.860+0000 7f2611ffb640 1 -- 172.21.15.46:0/1202480005 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f262002f020 con 0x5605bf9f7d90 2024-03-31T23:13:42.868 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.860+0000 7f2611ffb640 1 -- 172.21.15.46:0/1202480005 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f262003c020 con 0x5605bf9f7d90 2024-03-31T23:13:42.868 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.860+0000 7f26339cb740 10 client.16299 fetch_fsmap finished waiting for FSMap version 28 2024-03-31T23:13:42.868 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.860+0000 7f26339cb740 10 client.16299 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-31T23:13:42.868 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.860+0000 7f26339cb740 20 client.16299 populate_metadata read hostname 'smithi046' 2024-03-31T23:13:42.868 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.860+0000 7f26339cb740 1 --2- 172.21.15.46:0/1202480005 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x5605bfa777e0 0x5605bfa79c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:13:42.869 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.860+0000 7f26339cb740 4 client.16299 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-31T23:13:42.869 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.860+0000 7f26339cb740 1 -- 172.21.15.46:0/1202480005 --> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5605bf962a40 con 0x5605bfa777e0 2024-03-31T23:13:42.869 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.860+0000 7f262a59c640 1 --2- 172.21.15.46:0/1202480005 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x5605bfa777e0 0x5605bfa79c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:13:42.869 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.861+0000 7f2611ffb640 10 client.16299 ms_handle_connect on v2:172.21.15.142:6838/2519143918 2024-03-31T23:13:42.869 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.861+0000 7f262a59c640 1 --2- 172.21.15.46:0/1202480005 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x5605bfa777e0 0x5605bfa79c00 crc :-1 s=READY pgs=90 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=225151ce885ef7b1 server_cookie=cd3c3df0271f1a62 in_seq=0 out_seq=0 2024-03-31T23:13:42.869 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.861+0000 7f2611ffb640 1 -- 172.21.15.46:0/1202480005 <== mds.0 v2:172.21.15.142:6838/2519143918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5605bf962a40 con 0x5605bfa777e0 2024-03-31T23:13:42.869 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.861+0000 7f2611ffb640 10 client.16299 handle_command_reply: tid=0 2024-03-31T23:13:42.869 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.861+0000 7f26339cb740 1 client.16299 shutdown 2024-03-31T23:13:42.869 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.861+0000 7f26339cb740 1 -- 172.21.15.46:0/1202480005 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5605bf9f7d90 msgr2=0x5605bf9face0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:13:42.869 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.861+0000 7f26339cb740 1 --2- 172.21.15.46:0/1202480005 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5605bf9f7d90 0x5605bf9face0 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f262002e040 tx=0x7f2620005dc0 comp rx=0 tx=0).stop 2024-03-31T23:13:42.869 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.861+0000 7f26339cb740 1 -- 172.21.15.46:0/1202480005 shutdown_connections 2024-03-31T23:13:42.869 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.861+0000 7f26339cb740 1 --2- 172.21.15.46:0/1202480005 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x5605bfa777e0 0x5605bfa79c00 crc :-1 s=READY pgs=90 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:42.869 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.861+0000 7f26339cb740 1 --2- 172.21.15.46:0/1202480005 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5605bfa00970 0x5605bf9fb220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:42.869 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.861+0000 7f26339cb740 1 --2- 172.21.15.46:0/1202480005 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5605bf9f7d90 0x5605bf9face0 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:42.869 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.861+0000 7f2611ffb640 0 client.16299 ms_handle_reset on v2:172.21.15.142:6838/2519143918 2024-03-31T23:13:42.870 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.861+0000 7f26339cb740 1 --2- 172.21.15.46:0/1202480005 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5605bf9f7470 0x5605bf9fa7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:42.870 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.861+0000 7f26339cb740 1 -- 172.21.15.46:0/1202480005 >> 172.21.15.46:0/1202480005 conn(0x5605bf9f1ba0 msgr2=0x5605bf9f2d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:13:42.870 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.861+0000 7f26339cb740 1 -- 172.21.15.46:0/1202480005 shutdown_connections 2024-03-31T23:13:42.870 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.861+0000 7f26339cb740 1 -- 172.21.15.46:0/1202480005 wait complete. 2024-03-31T23:13:42.870 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.861+0000 7f26339cb740 20 client.16299 trim_cache size 0 max 16384 2024-03-31T23:13:42.870 INFO:teuthology.orchestra.run.smithi046.stdout:[] 2024-03-31T23:13:42.871 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.862+0000 7f2630d16640 1 -- 172.21.15.46:0/2777658547 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f26140a2640 msgr2=0x7f26140a4b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:13:42.871 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.862+0000 7f2630d16640 1 --2- 172.21.15.46:0/2777658547 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f26140a2640 0x7f26140a4b00 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f2624005cc0 tx=0x7f262402e040 comp rx=0 tx=0).stop 2024-03-31T23:13:42.871 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.862+0000 7f2630d16640 1 -- 172.21.15.46:0/2777658547 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f262c068950 msgr2=0x7f262c100610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:13:42.871 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.862+0000 7f2630d16640 1 --2- 172.21.15.46:0/2777658547 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f262c068950 0x7f262c100610 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f261c00e9f0 tx=0x7f261c00eeb0 comp rx=0 tx=0).stop 2024-03-31T23:13:42.871 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.862+0000 7f2630d16640 1 -- 172.21.15.46:0/2777658547 shutdown_connections 2024-03-31T23:13:42.871 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.862+0000 7f2630d16640 1 --2- 172.21.15.46:0/2777658547 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f26140a2640 0x7f26140a4b00 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:42.871 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.862+0000 7f2630d16640 1 --2- 172.21.15.46:0/2777658547 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f262c1077a0 0x7f262c101090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:42.871 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.862+0000 7f2630d16640 1 --2- 172.21.15.46:0/2777658547 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f262c069310 0x7f262c100b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:42.872 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.862+0000 7f2630d16640 1 --2- 172.21.15.46:0/2777658547 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f262c068950 0x7f262c100610 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:42.872 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.862+0000 7f2630d16640 1 -- 172.21.15.46:0/2777658547 >> 172.21.15.46:0/2777658547 conn(0x7f262c077750 msgr2=0x7f262c0fe200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:13:42.872 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.862+0000 7f2630d16640 1 -- 172.21.15.46:0/2777658547 shutdown_connections 2024-03-31T23:13:42.872 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:13:42.862+0000 7f2630d16640 1 -- 172.21.15.46:0/2777658547 wait complete. 2024-03-31T23:13:42.874 ERROR:tasks.fwd_scrub:rank 2 damaged: [{'damage_type': 'backtrace', 'id': 116245510, 'ino': 1099511628313, 'path': '/volumes/qa/sv_0/e990713a-13d9-4c54-877d-f0f60b19e88e/client.0/tmp/clients/client0/~dmtmp/COREL/GRAPHIC1.CDR'}, {'damage_type': 'backtrace', 'id': 121823561, 'ino': 1099511628297, 'path': '/volumes/qa/sv_0/e990713a-13d9-4c54-877d-f0f60b19e88e/client.0/tmp/clients/client0/~dmtmp/WORD/CHAP10.DOC'}, {'damage_type': 'backtrace', 'id': 197392199, 'ino': 1099511628312, 'path': '/volumes/qa/sv_0/e990713a-13d9-4c54-877d-f0f60b19e88e/client.0/tmp/clients/client0/~dmtmp/COREL/ARTISTIC.ACL'}, {'damage_type': 'backtrace', 'id': 309025251, 'ino': 1099511628351, 'path': '/volumes/qa/sv_0/e990713a-13d9-4c54-877d-f0f60b19e88e/client.0/tmp/clients/client0/~dmtmp/ACCESS/FASTENER.MDB'}, {'damage_type': 'backtrace', 'id': 973458780, 'ino': 1099511628320, 'path': '/volumes/qa/sv_0/e990713a-13d9-4c54-877d-f0f60b19e88e/client.0/tmp/clients/client0/~dmtmp/COREL/CORELDRW.CDT'}, {'damage_type': 'backtrace', 'id': 1001932470, 'ino': 1099511628324, 'path': '/volumes/qa/sv_0/e990713a-13d9-4c54-877d-f0f60b19e88e/client.0/tmp/clients/client0/~dmtmp/WORDPRO/NEWS1_1.LWP'}, {'damage_type': 'backtrace', 'id': 1072693154, 'ino': 1099511628316, 'path': '/volumes/qa/sv_0/e990713a-13d9-4c54-877d-f0f60b19e88e/client.0/tmp/clients/client0/~dmtmp/COREL/SIGN2.CPT'}, {'damage_type': 'backtrace', 'id': 1266774805, 'ino': 1099511628298, 'path': '/volumes/qa/sv_0/e990713a-13d9-4c54-877d-f0f60b19e88e/client.0/tmp/clients/client0/~dmtmp/WORD/BASEMACH.DOC'}, {'damage_type': 'backtrace', 'id': 1302310385, 'ino': 1099511628322, 'path': '/volumes/qa/sv_0/e990713a-13d9-4c54-877d-f0f60b19e88e/client.0/tmp/clients/client0/~dmtmp/WORDPRO/BENCHS.LWP'}, {'damage_type': 'backtrace', 'id': 1398107989, 'ino': 1099511628299, 'path': '/volumes/qa/sv_0/e990713a-13d9-4c54-877d-f0f60b19e88e/client.0/tmp/clients/client0/~dmtmp/WORD/FACTS.DOC'}, {'damage_type': 'backtrace', 'id': 1405771217, 'ino': 1099511628310, 'path': '/volumes/qa/sv_0/e990713a-13d9-4c54-877d-f0f60b19e88e/client.0/tmp/clients/client0/~dmtmp/COREL/CDRROLS.CFG'}, {'damage_type': 'backtrace', 'id': 2072353046, 'ino': 1099511628317, 'path': '/volumes/qa/sv_0/e990713a-13d9-4c54-877d-f0f60b19e88e/client.0/tmp/clients/client0/~dmtmp/COREL/TRAIN.CDR'}, {'damage_type': 'backtrace', 'id': 2152932943, 'ino': 1099511628301, 'path': '/volumes/qa/sv_0/e990713a-13d9-4c54-877d-f0f60b19e88e/client.0/tmp/clients/client0/~dmtmp/EXCEL/PCMAGCD.XLS'}, {'damage_type': 'backtrace', 'id': 2171341473, 'ino': 1099511628354, 'path': '/volumes/qa/sv_0/e990713a-13d9-4c54-877d-f0f60b19e88e/client.0/tmp/clients/client0/~dmtmp/SEED/MEDIUM.FIL'}, {'damage_type': 'backtrace', 'id': 2371159217, 'ino': 1099511628302, 'path': '/volumes/qa/sv_0/e990713a-13d9-4c54-877d-f0f60b19e88e/client.0/tmp/clients/client0/~dmtmp/EXCEL/SALES.XLS'}, {'damage_type': 'backtrace', 'id': 2677032042, 'ino': 1099511628326, 'path': '/volumes/qa/sv_0/e990713a-13d9-4c54-877d-f0f60b19e88e/client.0/tmp/clients/client0/~dmtmp/PM/T1.XLS'}, {'damage_type': 'backtrace', 'id': 3146452803, 'ino': 1099511628323, 'path': '/volumes/qa/sv_0/e990713a-13d9-4c54-877d-f0f60b19e88e/client.0/tmp/clients/client0/~dmtmp/WORDPRO/RESULTS.XLS'}, {'damage_type': 'backtrace', 'id': 3245313740, 'ino': 1099511628315, 'path': '/volumes/qa/sv_0/e990713a-13d9-4c54-877d-f0f60b19e88e/client.0/tmp/clients/client0/~dmtmp/COREL/TEAPOT2.CPT'}, {'damage_type': 'backtrace', 'id': 3448543011, 'ino': 1099511628328, 'path': '/volumes/qa/sv_0/e990713a-13d9-4c54-877d-f0f60b19e88e/client.0/tmp/clients/client0/~dmtmp/PM/MOVED.DOC'}, {'damage_type': 'backtrace', 'id': 3461088003, 'ino': 1099511628318, 'path': '/volumes/qa/sv_0/e990713a-13d9-4c54-877d-f0f60b19e88e/client.0/tmp/clients/client0/~dmtmp/COREL/GRAPHIC2.CDR'}, {'damage_type': 'backtrace', 'id': 3544020290, 'ino': 1099511628355, 'path': '/volumes/qa/sv_0/e990713a-13d9-4c54-877d-f0f60b19e88e/client.0/tmp/clients/client0/~dmtmp/SEED/LARGE.FIL'}, {'damage_type': 'backtrace', 'id': 3986028770, 'ino': 1099511628327, 'path': '/volumes/qa/sv_0/e990713a-13d9-4c54-877d-f0f60b19e88e/client.0/tmp/clients/client0/~dmtmp/PM/EVENTS.DOC'}, {'damage_type': 'backtrace', 'id': 4057030136, 'ino': 1099511628353, 'path': '/volumes/qa/sv_0/e990713a-13d9-4c54-877d-f0f60b19e88e/client.0/tmp/clients/client0/~dmtmp/SEED/SMALL.FIL'}, {'damage_type': 'backtrace', 'id': 4079277184, 'ino': 1099511628319, 'path': '/volumes/qa/sv_0/e990713a-13d9-4c54-877d-f0f60b19e88e/client.0/tmp/clients/client0/~dmtmp/COREL/ROSE.CPT'}, {'damage_type': 'backtrace', 'id': 4082973856, 'ino': 1099511628314, 'path': '/volumes/qa/sv_0/e990713a-13d9-4c54-877d-f0f60b19e88e/client.0/tmp/clients/client0/~dmtmp/COREL/MAILBOX2.CPT'}, {'damage_type': 'backtrace', 'id': 4106478478, 'ino': 1099511628311, 'path': '/volumes/qa/sv_0/e990713a-13d9-4c54-877d-f0f60b19e88e/client.0/tmp/clients/client0/~dmtmp/COREL/CDRBARS.CFG'}] 2024-03-31T23:13:42.874 ERROR:tasks.fwd_scrub.fs.[cephfs]:exception: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph_384f31f3d3aee7f8ba3f60846b8844c049165583/qa/tasks/fwd_scrub.py", line 38, in _run self.do_scrub() File "/home/teuthworker/src/git.ceph.com_ceph_384f31f3d3aee7f8ba3f60846b8844c049165583/qa/tasks/fwd_scrub.py", line 55, in do_scrub self._scrub() File "/home/teuthworker/src/git.ceph.com_ceph_384f31f3d3aee7f8ba3f60846b8844c049165583/qa/tasks/fwd_scrub.py", line 79, in _scrub self._check_damage() File "/home/teuthworker/src/git.ceph.com_ceph_384f31f3d3aee7f8ba3f60846b8844c049165583/qa/tasks/fwd_scrub.py", line 90, in _check_damage raise RuntimeError(f"rank damage found: {types}") RuntimeError: rank damage found: {'backtrace'} 2024-03-31T23:13:42.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:13:42 smithi142 ceph-mon[35271]: pgmap v62: 129 pgs: 129 active+clean; 34 MiB data, 499 MiB used, 1.0 TiB / 1.0 TiB avail; 246 KiB/s rd, 1.2 MiB/s wr, 120 op/s 2024-03-31T23:13:42.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:13:42 smithi142 ceph-mon[35271]: from='client.? 172.21.15.46:0/2126680105' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-31T23:13:43.191 INFO:tasks.workunit.client.0.smithi046.stdout: 1 20354 2.54 MB/sec warmup 49 sec latency 70.611 ms 2024-03-31T23:13:44.192 INFO:tasks.workunit.client.0.smithi046.stdout: 1 21070 2.62 MB/sec warmup 50 sec latency 15.875 ms 2024-03-31T23:13:44.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:13:44 smithi162 ceph-mon[36237]: pgmap v63: 129 pgs: 129 active+clean; 34 MiB data, 499 MiB used, 1.0 TiB / 1.0 TiB avail; 246 KiB/s rd, 1.2 MiB/s wr, 120 op/s 2024-03-31T23:13:44.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:13:44 smithi046 ceph-mon[32498]: pgmap v63: 129 pgs: 129 active+clean; 34 MiB data, 499 MiB used, 1.0 TiB / 1.0 TiB avail; 246 KiB/s rd, 1.2 MiB/s wr, 120 op/s 2024-03-31T23:13:44.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:13:44 smithi142 ceph-mon[35271]: pgmap v63: 129 pgs: 129 active+clean; 34 MiB data, 499 MiB used, 1.0 TiB / 1.0 TiB avail; 246 KiB/s rd, 1.2 MiB/s wr, 120 op/s 2024-03-31T23:13:45.191 INFO:tasks.workunit.client.0.smithi046.stdout: 1 21437 2.63 MB/sec warmup 51 sec latency 17.366 ms 2024-03-31T23:13:45.284 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:45.281+0000 7f6306ff4640 1 -- 172.21.15.46:0/1579355542 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f63001055f0 msgr2=0x7f63001059d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:13:45.284 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:45.281+0000 7f6306ff4640 1 --2- 172.21.15.46:0/1579355542 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f63001055f0 0x7f63001059d0 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f62f4009900 tx=0x7f62f402bc30 comp rx=0 tx=0).stop 2024-03-31T23:13:45.285 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:45.281+0000 7f6306ff4640 1 -- 172.21.15.46:0/1579355542 shutdown_connections 2024-03-31T23:13:45.285 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:45.281+0000 7f6306ff4640 1 --2- 172.21.15.46:0/1579355542 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6300101260 0x7f630010d480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:45.285 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:45.281+0000 7f6306ff4640 1 --2- 172.21.15.46:0/1579355542 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f63001008a0 0x7f6300100d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:45.285 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:45.281+0000 7f6306ff4640 1 --2- 172.21.15.46:0/1579355542 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f63001055f0 0x7f63001059d0 unknown :-1 s=CLOSED pgs=645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:45.285 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:45.281+0000 7f6306ff4640 1 -- 172.21.15.46:0/1579355542 >> 172.21.15.46:0/1579355542 conn(0x7f63000774a0 msgr2=0x7f63000778b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:13:45.285 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:45.281+0000 7f6306ff4640 1 -- 172.21.15.46:0/1579355542 shutdown_connections 2024-03-31T23:13:45.285 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:45.281+0000 7f6306ff4640 1 -- 172.21.15.46:0/1579355542 wait complete. 2024-03-31T23:13:45.285 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:45.282+0000 7f6306ff4640 1 Processor -- start 2024-03-31T23:13:45.285 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:45.282+0000 7f6306ff4640 1 -- start start 2024-03-31T23:13:45.285 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:45.282+0000 7f6306ff4640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f63001008a0 0x7f6300198d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:13:45.285 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:45.282+0000 7f6306ff4640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6300101260 0x7f6300199260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:13:45.285 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:45.282+0000 7f6306ff4640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f63001055f0 0x7f63001a3da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:13:45.285 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:45.282+0000 7f6306ff4640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f63001109c0 con 0x7f6300101260 2024-03-31T23:13:45.285 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:45.282+0000 7f6306ff4640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f6300110840 con 0x7f63001008a0 2024-03-31T23:13:45.286 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:45.282+0000 7f6306ff4640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f6300110b40 con 0x7f63001055f0 2024-03-31T23:13:45.286 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:45.282+0000 7f62fffff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6300101260 0x7f6300199260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:13:45.286 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:45.282+0000 7f6304d90640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f63001008a0 0x7f6300198d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:13:45.286 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:45.282+0000 7f6304d90640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f63001008a0 0x7f6300198d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.46:48878/0 (socket says 172.21.15.46:48878) 2024-03-31T23:13:45.286 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:45.282+0000 7f62fffff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6300101260 0x7f6300199260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60196/0 (socket says 172.21.15.46:60196) 2024-03-31T23:13:45.286 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:45.282+0000 7f6304d90640 1 -- 172.21.15.46:0/3425122224 learned_addr learned my addr 172.21.15.46:0/3425122224 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:13:45.286 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:45.282+0000 7f6305591640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f63001055f0 0x7f63001a3da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:13:45.286 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:45.282+0000 7f6304d90640 1 -- 172.21.15.46:0/3425122224 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f63001055f0 msgr2=0x7f63001a3da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:13:45.286 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:45.282+0000 7f6304d90640 1 --2- 172.21.15.46:0/3425122224 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f63001055f0 0x7f63001a3da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:45.286 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:45.282+0000 7f6304d90640 1 -- 172.21.15.46:0/3425122224 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6300101260 msgr2=0x7f6300199260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:13:45.286 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:45.282+0000 7f6304d90640 1 --2- 172.21.15.46:0/3425122224 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6300101260 0x7f6300199260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:45.286 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:45.282+0000 7f6304d90640 1 -- 172.21.15.46:0/3425122224 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f63001a42e0 con 0x7f63001008a0 2024-03-31T23:13:45.286 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:45.282+0000 7f62fffff640 1 --2- 172.21.15.46:0/3425122224 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6300101260 0x7f6300199260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:13:45.287 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:45.282+0000 7f6305591640 1 --2- 172.21.15.46:0/3425122224 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f63001055f0 0x7f63001a3da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:13:45.287 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:45.282+0000 7f6304d90640 1 --2- 172.21.15.46:0/3425122224 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f63001008a0 0x7f6300198d20 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f62f4006010 tx=0x7f62f4005d90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:13:45.287 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:45.283+0000 7f62fdffb640 1 -- 172.21.15.46:0/3425122224 <== mon.2 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62f4043020 con 0x7f63001008a0 2024-03-31T23:13:45.287 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:45.283+0000 7f6306ff4640 1 -- 172.21.15.46:0/3425122224 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f63001a44b0 con 0x7f63001008a0 2024-03-31T23:13:45.287 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:45.283+0000 7f62fdffb640 1 -- 172.21.15.46:0/3425122224 <== mon.2 v2:172.21.15.142:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f62f4004350 con 0x7f63001008a0 2024-03-31T23:13:45.287 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:45.283+0000 7f62fdffb640 1 -- 172.21.15.46:0/3425122224 <== mon.2 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62f4003a10 con 0x7f63001008a0 2024-03-31T23:13:45.287 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:45.283+0000 7f6306ff4640 1 -- 172.21.15.46:0/3425122224 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f63001a49f0 con 0x7f63001008a0 2024-03-31T23:13:45.293 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:45.285+0000 7f62fdffb640 1 -- 172.21.15.46:0/3425122224 <== mon.2 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 25) ==== 137451+0+0 (secure 0 0 0) 0x7f62f4046020 con 0x7f63001008a0 2024-03-31T23:13:45.293 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:45.285+0000 7f6309bd8740 1 Processor -- start 2024-03-31T23:13:45.293 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:45.285+0000 7f6309bd8740 1 -- start start 2024-03-31T23:13:45.293 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:45.285+0000 7f6309bd8740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55cc0ff7c6c0 0x55cc0ff7caa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:13:45.293 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:45.285+0000 7f6309bd8740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55cc0ff7cfe0 0x55cc0ff85870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:13:45.294 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:45.285+0000 7f6309bd8740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cc0ff85db0 0x55cc0ff881a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:13:45.294 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:45.285+0000 7f6309bd8740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55cc0fedf8f0 con 0x55cc0ff7cfe0 2024-03-31T23:13:45.294 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:45.285+0000 7f6309bd8740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55cc0fe888d0 con 0x55cc0ff7c6c0 2024-03-31T23:13:45.294 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:45.285+0000 7f6309bd8740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55cc0fedb990 con 0x55cc0ff85db0 2024-03-31T23:13:45.294 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:45.285+0000 7f62fdffb640 1 --2- 172.21.15.46:0/3425122224 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f62f00a26b0 0x7f62f00a4b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:13:45.294 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:45.286+0000 7f6304d90640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55cc0ff7cfe0 0x55cc0ff85870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:13:45.294 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:45.286+0000 7f6305591640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55cc0ff7c6c0 0x55cc0ff7caa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:13:45.294 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:45.286+0000 7f6305591640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55cc0ff7c6c0 0x55cc0ff7caa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.46:48890/0 (socket says 172.21.15.46:48890) 2024-03-31T23:13:45.294 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:45.286+0000 7f6305591640 1 -- 172.21.15.46:0/1956991989 learned_addr learned my addr 172.21.15.46:0/1956991989 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:13:45.294 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:45.286+0000 7f62fffff640 1 --2- 172.21.15.46:0/1956991989 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cc0ff85db0 0x55cc0ff881a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:13:45.294 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:45.286+0000 7f6305591640 1 --2- 172.21.15.46:0/3425122224 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f62f00a26b0 0x7f62f00a4b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:13:45.294 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:45.286+0000 7f62fdffb640 1 -- 172.21.15.46:0/3425122224 <== mon.2 v2:172.21.15.142:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7f62f40f0890 con 0x7f63001008a0 2024-03-31T23:13:45.294 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:45.286+0000 7f62fffff640 1 -- 172.21.15.46:0/1956991989 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55cc0ff7c6c0 msgr2=0x55cc0ff7caa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:13:45.294 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:45.286+0000 7f62fffff640 1 --2- 172.21.15.46:0/1956991989 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55cc0ff7c6c0 0x55cc0ff7caa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:45.295 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:45.286+0000 7f62fffff640 1 -- 172.21.15.46:0/1956991989 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55cc0ff7cfe0 msgr2=0x55cc0ff85870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:13:45.295 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:45.286+0000 7f62fffff640 1 --2- 172.21.15.46:0/1956991989 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55cc0ff7cfe0 0x55cc0ff85870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:45.295 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:45.286+0000 7f62fffff640 1 -- 172.21.15.46:0/1956991989 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cc0fbf4210 con 0x55cc0ff85db0 2024-03-31T23:13:45.295 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:45.286+0000 7f6305591640 1 --2- 172.21.15.46:0/3425122224 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f62f00a26b0 0x7f62f00a4b70 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f62f8009930 tx=0x7f62f8009500 comp rx=0 tx=0).ready entity=mgr.15654 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:13:45.295 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:45.286+0000 7f62fffff640 1 --2- 172.21.15.46:0/1956991989 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cc0ff85db0 0x55cc0ff881a0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f62ec009fa0 tx=0x7f62ec0118d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6e78820d996439f6 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:13:45.295 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:45.286+0000 7f62eaffd640 1 -- 172.21.15.46:0/1956991989 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62ec004030 con 0x55cc0ff85db0 2024-03-31T23:13:45.295 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:45.286+0000 7f62eaffd640 1 -- 172.21.15.46:0/1956991989 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f62ec0054d0 con 0x55cc0ff85db0 2024-03-31T23:13:45.295 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:45.286+0000 7f62eaffd640 1 -- 172.21.15.46:0/1956991989 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62ec004c40 con 0x55cc0ff85db0 2024-03-31T23:13:45.295 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:45.287+0000 7f6309bd8740 1 -- 172.21.15.46:0/1956991989 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cc0ff85db0 msgr2=0x55cc0ff881a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:13:45.295 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:45.287+0000 7f6309bd8740 1 --2- 172.21.15.46:0/1956991989 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cc0ff85db0 0x55cc0ff881a0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f62ec009fa0 tx=0x7f62ec0118d0 comp rx=0 tx=0).stop 2024-03-31T23:13:45.295 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:45.287+0000 7f6309bd8740 1 -- 172.21.15.46:0/1956991989 shutdown_connections 2024-03-31T23:13:45.295 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:45.287+0000 7f6309bd8740 1 --2- 172.21.15.46:0/1956991989 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cc0ff85db0 0x55cc0ff881a0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:45.295 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:45.287+0000 7f6309bd8740 1 --2- 172.21.15.46:0/1956991989 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55cc0ff7cfe0 0x55cc0ff85870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:45.295 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:45.287+0000 7f6309bd8740 1 --2- 172.21.15.46:0/1956991989 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55cc0ff7c6c0 0x55cc0ff7caa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:45.295 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:45.287+0000 7f6309bd8740 1 -- 172.21.15.46:0/1956991989 >> 172.21.15.46:0/1956991989 conn(0x55cc0ff76fe0 msgr2=0x55cc0ff773c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:13:45.296 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:45.287+0000 7f6309bd8740 1 -- 172.21.15.46:0/1956991989 shutdown_connections 2024-03-31T23:13:45.296 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:45.287+0000 7f6309bd8740 1 -- 172.21.15.46:0/1956991989 wait complete. 2024-03-31T23:13:45.296 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:45.287+0000 7f6309bd8740 1 -- start start 2024-03-31T23:13:45.296 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:45.287+0000 7f6309bd8740 1 Processor -- start 2024-03-31T23:13:45.296 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:45.288+0000 7f6309bd8740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55cc0ff7c6c0 0x55cc0ff82eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:13:45.296 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:45.288+0000 7f6309bd8740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55cc0ff7cfe0 0x55cc0ff833f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:13:45.296 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:45.288+0000 7f6309bd8740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cc0ff85db0 0x55cc0ff83930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:13:45.296 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:45.288+0000 7f6309bd8740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55cc0ff889e0 con 0x55cc0ff7cfe0 2024-03-31T23:13:45.296 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:45.288+0000 7f6309bd8740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55cc0fe7df80 con 0x55cc0ff7c6c0 2024-03-31T23:13:45.296 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:45.288+0000 7f6309bd8740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55cc0ff88b60 con 0x55cc0ff85db0 2024-03-31T23:13:45.296 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:45.288+0000 7f6309bd8740 10 client.0 _ll_register_callbacks cb 0x55cc0fed6290 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T23:13:45.296 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:45.288+0000 7f62fffff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55cc0ff7cfe0 0x55cc0ff833f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:13:45.296 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:45.288+0000 7f62fffff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55cc0ff7cfe0 0x55cc0ff833f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60216/0 (socket says 172.21.15.46:60216) 2024-03-31T23:13:45.297 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:45.288+0000 7f62fffff640 1 -- 172.21.15.46:0/149862965 learned_addr learned my addr 172.21.15.46:0/149862965 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:13:45.297 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:45.288+0000 7f6304d90640 1 --2- 172.21.15.46:0/149862965 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55cc0ff7c6c0 0x55cc0ff82eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:13:45.297 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:45.288+0000 7f6305591640 1 --2- 172.21.15.46:0/149862965 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cc0ff85db0 0x55cc0ff83930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:13:45.297 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:45.288+0000 7f62fffff640 1 -- 172.21.15.46:0/149862965 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cc0ff85db0 msgr2=0x55cc0ff83930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:13:45.297 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:45.288+0000 7f62fffff640 1 --2- 172.21.15.46:0/149862965 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cc0ff85db0 0x55cc0ff83930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:45.297 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:45.288+0000 7f62fffff640 1 -- 172.21.15.46:0/149862965 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55cc0ff7c6c0 msgr2=0x55cc0ff82eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:13:45.297 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:45.288+0000 7f62fffff640 1 --2- 172.21.15.46:0/149862965 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55cc0ff7c6c0 0x55cc0ff82eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:45.297 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:45.288+0000 7f62fffff640 1 -- 172.21.15.46:0/149862965 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cc0fcc0620 con 0x55cc0ff7cfe0 2024-03-31T23:13:45.297 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:45.288+0000 7f6304d90640 1 --2- 172.21.15.46:0/149862965 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55cc0ff7c6c0 0x55cc0ff82eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T23:13:45.297 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:45.288+0000 7f62fffff640 1 --2- 172.21.15.46:0/149862965 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55cc0ff7cfe0 0x55cc0ff833f0 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f62ec000c00 tx=0x7f62ec00fc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9177836734cc1187 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:13:45.297 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:45.288+0000 7f62dffff640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-31T23:13:45.297 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:45.289+0000 7f62dffff640 1 -- 172.21.15.46:0/149862965 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62ec028020 con 0x55cc0ff7cfe0 2024-03-31T23:13:45.297 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:45.289+0000 7f62dffff640 1 -- 172.21.15.46:0/149862965 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7f62ec00fe20 con 0x55cc0ff7cfe0 2024-03-31T23:13:45.297 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:45.289+0000 7f6309bd8740 1 -- 172.21.15.46:0/149862965 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f62ec0054d0 con 0x55cc0ff7cfe0 2024-03-31T23:13:45.297 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:45.289+0000 7f62dffff640 1 -- 172.21.15.46:0/149862965 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f62ec033050 con 0x55cc0ff7cfe0 2024-03-31T23:13:45.297 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:45.289+0000 7f6309bd8740 10 client.16311 fetch_fsmap learned FSMap version 28 2024-03-31T23:13:45.297 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:45.289+0000 7f6309bd8740 1 -- 172.21.15.46:0/149862965 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55cc0ff94990 con 0x55cc0ff7cfe0 2024-03-31T23:13:45.298 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:45.289+0000 7f62dffff640 1 -- 172.21.15.46:0/149862965 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f62ec0054d0 con 0x55cc0ff7cfe0 2024-03-31T23:13:45.298 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:45.289+0000 7f62dffff640 1 -- 172.21.15.46:0/149862965 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f62ec023ca0 con 0x55cc0ff7cfe0 2024-03-31T23:13:45.298 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:45.289+0000 7f6309bd8740 10 client.16311 fetch_fsmap finished waiting for FSMap version 28 2024-03-31T23:13:45.298 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:45.289+0000 7f6309bd8740 10 client.16311 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-31T23:13:45.298 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:45.289+0000 7f6309bd8740 20 client.16311 populate_metadata read hostname 'smithi046' 2024-03-31T23:13:45.298 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:45.289+0000 7f6309bd8740 1 --2- 172.21.15.46:0/149862965 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55cc0ff95df0 0x55cc0ff981e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:13:45.298 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:45.289+0000 7f6309bd8740 4 client.16311 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-31T23:13:45.298 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:45.289+0000 7f6309bd8740 1 -- 172.21.15.46:0/149862965 --> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55cc0fef3120 con 0x55cc0ff95df0 2024-03-31T23:13:45.298 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:45.289+0000 7f6304d90640 1 --2- 172.21.15.46:0/149862965 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55cc0ff95df0 0x55cc0ff981e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:13:45.298 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:45.290+0000 7f6304d90640 1 --2- 172.21.15.46:0/149862965 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55cc0ff95df0 0x55cc0ff981e0 crc :-1 s=READY pgs=91 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=315359c9690ac6a4 server_cookie=76e7d067fea93846 in_seq=0 out_seq=0 2024-03-31T23:13:45.298 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:45.290+0000 7f62dffff640 10 client.16311 ms_handle_connect on v2:172.21.15.142:6838/2519143918 2024-03-31T23:13:45.298 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:45.291+0000 7f62dffff640 1 -- 172.21.15.46:0/149862965 <== mds.0 v2:172.21.15.142:6838/2519143918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55cc0fef3120 con 0x55cc0ff95df0 2024-03-31T23:13:45.298 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:45.291+0000 7f62dffff640 10 client.16311 handle_command_reply: tid=0 2024-03-31T23:13:45.299 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:45.291+0000 7f6309bd8740 1 client.16311 shutdown 2024-03-31T23:13:45.299 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:45.291+0000 7f6309bd8740 1 -- 172.21.15.46:0/149862965 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55cc0ff7cfe0 msgr2=0x55cc0ff833f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:13:45.299 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:45.291+0000 7f6309bd8740 1 --2- 172.21.15.46:0/149862965 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55cc0ff7cfe0 0x55cc0ff833f0 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f62ec000c00 tx=0x7f62ec00fc30 comp rx=0 tx=0).stop 2024-03-31T23:13:45.299 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:45.291+0000 7f6309bd8740 1 -- 172.21.15.46:0/149862965 shutdown_connections 2024-03-31T23:13:45.299 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:45.291+0000 7f6309bd8740 1 --2- 172.21.15.46:0/149862965 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55cc0ff95df0 0x55cc0ff981e0 crc :-1 s=READY pgs=91 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:45.299 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:45.291+0000 7f6309bd8740 1 --2- 172.21.15.46:0/149862965 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cc0ff85db0 0x55cc0ff83930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:45.299 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:45.291+0000 7f6309bd8740 1 --2- 172.21.15.46:0/149862965 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55cc0ff7cfe0 0x55cc0ff833f0 unknown :-1 s=CLOSED pgs=646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:45.299 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:45.291+0000 7f62dffff640 0 client.16311 ms_handle_reset on v2:172.21.15.142:6838/2519143918 2024-03-31T23:13:45.299 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:45.291+0000 7f6309bd8740 1 --2- 172.21.15.46:0/149862965 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55cc0ff7c6c0 0x55cc0ff82eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:45.299 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:45.291+0000 7f6309bd8740 1 -- 172.21.15.46:0/149862965 >> 172.21.15.46:0/149862965 conn(0x55cc0ff76fe0 msgr2=0x55cc0ff78d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:13:45.299 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:45.291+0000 7f6309bd8740 1 -- 172.21.15.46:0/149862965 shutdown_connections 2024-03-31T23:13:45.299 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:45.291+0000 7f6309bd8740 1 -- 172.21.15.46:0/149862965 wait complete. 2024-03-31T23:13:45.299 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:45.291+0000 7f6309bd8740 20 client.16311 trim_cache size 0 max 16384 2024-03-31T23:13:45.309 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:45.307+0000 7f6309bd8740 1 Processor -- start 2024-03-31T23:13:45.309 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:45.307+0000 7f6309bd8740 1 -- start start 2024-03-31T23:13:45.309 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:45.307+0000 7f6309bd8740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55cc0ff7c6c0 0x55cc0ff855e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:13:45.309 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:45.307+0000 7f6309bd8740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cc0ff7cfe0 0x55cc0ff91780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:13:45.309 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:45.307+0000 7f6309bd8740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55cc0ff85db0 0x55cc0ff9da30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:13:45.309 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:45.307+0000 7f6309bd8740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f63001af1b0 con 0x55cc0ff7c6c0 2024-03-31T23:13:45.309 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:45.307+0000 7f6309bd8740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f63001aca80 con 0x55cc0ff85db0 2024-03-31T23:13:45.309 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:45.307+0000 7f6309bd8740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f63001aa8d0 con 0x55cc0ff7cfe0 2024-03-31T23:13:45.310 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:45.307+0000 7f6304d90640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55cc0ff7c6c0 0x55cc0ff855e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:13:45.310 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:45.307+0000 7f6304d90640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55cc0ff7c6c0 0x55cc0ff855e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60230/0 (socket says 172.21.15.46:60230) 2024-03-31T23:13:45.310 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:45.307+0000 7f6304d90640 1 -- 172.21.15.46:0/2477079632 learned_addr learned my addr 172.21.15.46:0/2477079632 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:13:45.310 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:45.307+0000 7f62fffff640 1 --2- 172.21.15.46:0/2477079632 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cc0ff7cfe0 0x55cc0ff91780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:13:45.310 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:45.307+0000 7f6305591640 1 --2- 172.21.15.46:0/2477079632 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55cc0ff85db0 0x55cc0ff9da30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:13:45.310 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:45.307+0000 7f62fffff640 1 -- 172.21.15.46:0/2477079632 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55cc0ff85db0 msgr2=0x55cc0ff9da30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:13:45.310 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:45.307+0000 7f62fffff640 1 --2- 172.21.15.46:0/2477079632 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55cc0ff85db0 0x55cc0ff9da30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:45.310 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:45.307+0000 7f62fffff640 1 -- 172.21.15.46:0/2477079632 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55cc0ff7c6c0 msgr2=0x55cc0ff855e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:13:45.310 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:45.307+0000 7f62fffff640 1 --2- 172.21.15.46:0/2477079632 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55cc0ff7c6c0 0x55cc0ff855e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:45.310 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:45.307+0000 7f62fffff640 1 -- 172.21.15.46:0/2477079632 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cc0ff82030 con 0x55cc0ff7cfe0 2024-03-31T23:13:45.310 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:45.308+0000 7f6304d90640 1 --2- 172.21.15.46:0/2477079632 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55cc0ff7c6c0 0x55cc0ff855e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:13:45.310 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:45.308+0000 7f62fffff640 1 --2- 172.21.15.46:0/2477079632 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cc0ff7cfe0 0x55cc0ff91780 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f62ec0049c0 tx=0x7f62ec005b90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5a04b3026cf934c server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:13:45.310 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:45.308+0000 7f62ea7fc640 1 -- 172.21.15.46:0/2477079632 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62ec028070 con 0x55cc0ff7cfe0 2024-03-31T23:13:45.310 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:45.308+0000 7f62ea7fc640 1 -- 172.21.15.46:0/2477079632 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f62ec009b50 con 0x55cc0ff7cfe0 2024-03-31T23:13:45.310 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:45.308+0000 7f62ea7fc640 1 -- 172.21.15.46:0/2477079632 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62ec01d040 con 0x55cc0ff7cfe0 2024-03-31T23:13:45.314 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:45.309+0000 7f6309bd8740 1 -- 172.21.15.46:0/2477079632 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cc0ff7cfe0 msgr2=0x55cc0ff91780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:13:45.314 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:45.309+0000 7f6309bd8740 1 --2- 172.21.15.46:0/2477079632 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cc0ff7cfe0 0x55cc0ff91780 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f62ec0049c0 tx=0x7f62ec005b90 comp rx=0 tx=0).stop 2024-03-31T23:13:45.314 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:45.309+0000 7f6309bd8740 1 -- 172.21.15.46:0/2477079632 shutdown_connections 2024-03-31T23:13:45.314 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:45.309+0000 7f6309bd8740 1 --2- 172.21.15.46:0/2477079632 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55cc0ff85db0 0x55cc0ff9da30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:45.314 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:45.309+0000 7f6309bd8740 1 --2- 172.21.15.46:0/2477079632 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cc0ff7cfe0 0x55cc0ff91780 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:45.314 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:45.309+0000 7f6309bd8740 1 --2- 172.21.15.46:0/2477079632 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55cc0ff7c6c0 0x55cc0ff855e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:45.314 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:45.309+0000 7f6309bd8740 1 -- 172.21.15.46:0/2477079632 >> 172.21.15.46:0/2477079632 conn(0x55cc0ff76fe0 msgr2=0x55cc0ff773c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:13:45.314 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:45.309+0000 7f6309bd8740 1 -- 172.21.15.46:0/2477079632 shutdown_connections 2024-03-31T23:13:45.314 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:45.309+0000 7f6309bd8740 1 -- 172.21.15.46:0/2477079632 wait complete. 2024-03-31T23:13:45.315 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:45.309+0000 7f6309bd8740 1 -- start start 2024-03-31T23:13:45.315 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:45.309+0000 7f6309bd8740 1 Processor -- start 2024-03-31T23:13:45.315 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:45.309+0000 7f6309bd8740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cc0ff7c6c0 0x55cc0ff7fa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:13:45.315 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:45.309+0000 7f6309bd8740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55cc0ff7cfe0 0x55cc0ff7ffd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:13:45.315 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:45.309+0000 7f6309bd8740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55cc0ff85db0 0x55cc0ff80510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:13:45.315 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:45.309+0000 7f6309bd8740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f63001aaf40 con 0x55cc0ff85db0 2024-03-31T23:13:45.315 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:45.309+0000 7f6309bd8740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f63001ac3b0 con 0x55cc0ff7cfe0 2024-03-31T23:13:45.315 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:45.309+0000 7f6309bd8740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f63001ab610 con 0x55cc0ff7c6c0 2024-03-31T23:13:45.315 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:45.309+0000 7f6309bd8740 10 client.0 _ll_register_callbacks cb 0x7f62f4005b70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T23:13:45.315 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:45.310+0000 7f6305591640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55cc0ff85db0 0x55cc0ff80510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:13:45.315 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:45.310+0000 7f62fffff640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55cc0ff7cfe0 0x55cc0ff7ffd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:13:45.315 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:45.310+0000 7f6305591640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55cc0ff85db0 0x55cc0ff80510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60234/0 (socket says 172.21.15.46:60234) 2024-03-31T23:13:45.315 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:45.310+0000 7f6304d90640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cc0ff7c6c0 0x55cc0ff7fa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:13:45.315 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:45.310+0000 7f6305591640 1 -- 172.21.15.46:0/1946656149 learned_addr learned my addr 172.21.15.46:0/1946656149 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:13:45.316 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:45.310+0000 7f6304d90640 1 -- 172.21.15.46:0/1946656149 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55cc0ff7cfe0 msgr2=0x55cc0ff7ffd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:13:45.316 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:45.310+0000 7f6304d90640 1 --2- 172.21.15.46:0/1946656149 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55cc0ff7cfe0 0x55cc0ff7ffd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:45.316 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:45.310+0000 7f6304d90640 1 -- 172.21.15.46:0/1946656149 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55cc0ff85db0 msgr2=0x55cc0ff80510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:13:45.316 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:45.310+0000 7f6304d90640 1 --2- 172.21.15.46:0/1946656149 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55cc0ff85db0 0x55cc0ff80510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:45.316 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:45.310+0000 7f6304d90640 1 -- 172.21.15.46:0/1946656149 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cc0fffb090 con 0x55cc0ff7c6c0 2024-03-31T23:13:45.316 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:45.310+0000 7f6305591640 1 --2- 172.21.15.46:0/1946656149 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55cc0ff85db0 0x55cc0ff80510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:13:45.316 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:45.310+0000 7f6304d90640 1 --2- 172.21.15.46:0/1946656149 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cc0ff7c6c0 0x55cc0ff7fa90 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f62f402c720 tx=0x7f62f408ed30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ee580ebd750301e5 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:13:45.316 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:45.310+0000 7f62dffff640 10 client.0 ms_handle_connect on v2:172.21.15.162:3300/0 2024-03-31T23:13:45.316 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:45.310+0000 7f62dffff640 1 -- 172.21.15.46:0/1946656149 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62f40db070 con 0x55cc0ff7c6c0 2024-03-31T23:13:45.316 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:45.310+0000 7f6309bd8740 1 -- 172.21.15.46:0/1946656149 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f62ec009b50 con 0x55cc0ff7c6c0 2024-03-31T23:13:45.316 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:45.310+0000 7f62dffff640 1 -- 172.21.15.46:0/1946656149 <== mon.1 v2:172.21.15.162:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7f62f40edeb0 con 0x55cc0ff7c6c0 2024-03-31T23:13:45.316 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:45.310+0000 7f62dffff640 1 -- 172.21.15.46:0/1946656149 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f62f40c5780 con 0x55cc0ff7c6c0 2024-03-31T23:13:45.316 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:45.310+0000 7f6309bd8740 10 client.26056 fetch_fsmap learned FSMap version 28 2024-03-31T23:13:45.316 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:45.310+0000 7f6309bd8740 1 -- 172.21.15.46:0/1946656149 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55cc0fffc3c0 con 0x55cc0ff7c6c0 2024-03-31T23:13:45.317 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:45.311+0000 7f62dffff640 1 -- 172.21.15.46:0/1946656149 <== mon.1 v2:172.21.15.162:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f62ec009b50 con 0x55cc0ff7c6c0 2024-03-31T23:13:45.317 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:45.311+0000 7f62dffff640 1 -- 172.21.15.46:0/1946656149 <== mon.1 v2:172.21.15.162:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f62f4003bb0 con 0x55cc0ff7c6c0 2024-03-31T23:13:45.317 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:45.311+0000 7f6309bd8740 10 client.26056 fetch_fsmap finished waiting for FSMap version 28 2024-03-31T23:13:45.317 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:45.311+0000 7f6309bd8740 10 client.26056 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-31T23:13:45.317 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:45.311+0000 7f6309bd8740 20 client.26056 populate_metadata read hostname 'smithi046' 2024-03-31T23:13:45.317 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:45.311+0000 7f6309bd8740 1 --2- 172.21.15.46:0/1946656149 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55cc0fffcb70 0x55cc0fffef90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:13:45.317 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:45.311+0000 7f6309bd8740 4 client.26056 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-31T23:13:45.317 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:45.311+0000 7f6309bd8740 1 -- 172.21.15.46:0/1946656149 --> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55cc0feece30 con 0x55cc0fffcb70 2024-03-31T23:13:45.317 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:45.311+0000 7f62fffff640 1 --2- 172.21.15.46:0/1946656149 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55cc0fffcb70 0x55cc0fffef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:13:45.317 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:45.311+0000 7f62dffff640 10 client.26056 ms_handle_connect on v2:172.21.15.142:6838/2519143918 2024-03-31T23:13:45.317 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:45.311+0000 7f62fffff640 1 --2- 172.21.15.46:0/1946656149 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55cc0fffcb70 0x55cc0fffef90 crc :-1 s=READY pgs=92 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8e3925a05aca6c9 server_cookie=9081348f6340ffc0 in_seq=0 out_seq=0 2024-03-31T23:13:45.332 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:45.330+0000 7f62dffff640 1 -- 172.21.15.46:0/1946656149 <== mds.0 v2:172.21.15.142:6838/2519143918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55cc0feece30 con 0x55cc0fffcb70 2024-03-31T23:13:45.332 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:45.330+0000 7f62dffff640 10 client.26056 handle_command_reply: tid=0 2024-03-31T23:13:45.332 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:45.330+0000 7f6309bd8740 1 client.26056 shutdown 2024-03-31T23:13:45.332 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:45.330+0000 7f6309bd8740 1 -- 172.21.15.46:0/1946656149 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cc0ff7c6c0 msgr2=0x55cc0ff7fa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:13:45.332 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:45.330+0000 7f6309bd8740 1 --2- 172.21.15.46:0/1946656149 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cc0ff7c6c0 0x55cc0ff7fa90 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f62f402c720 tx=0x7f62f408ed30 comp rx=0 tx=0).stop 2024-03-31T23:13:45.332 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:45.330+0000 7f6309bd8740 1 -- 172.21.15.46:0/1946656149 shutdown_connections 2024-03-31T23:13:45.332 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:45.330+0000 7f6309bd8740 1 --2- 172.21.15.46:0/1946656149 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55cc0fffcb70 0x55cc0fffef90 crc :-1 s=READY pgs=92 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:45.332 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:45.330+0000 7f6309bd8740 1 --2- 172.21.15.46:0/1946656149 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55cc0ff85db0 0x55cc0ff80510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:45.332 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:45.330+0000 7f6309bd8740 1 --2- 172.21.15.46:0/1946656149 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55cc0ff7cfe0 0x55cc0ff7ffd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:45.332 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:45.330+0000 7f6309bd8740 1 --2- 172.21.15.46:0/1946656149 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cc0ff7c6c0 0x55cc0ff7fa90 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:45.333 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:45.330+0000 7f62dffff640 0 client.26056 ms_handle_reset on v2:172.21.15.142:6838/2519143918 2024-03-31T23:13:45.333 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:45.330+0000 7f6309bd8740 1 -- 172.21.15.46:0/1946656149 >> 172.21.15.46:0/1946656149 conn(0x55cc0ff76fe0 msgr2=0x55cc0ff78310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:13:45.333 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:45.330+0000 7f6309bd8740 1 -- 172.21.15.46:0/1946656149 shutdown_connections 2024-03-31T23:13:45.333 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:45.330+0000 7f6309bd8740 1 -- 172.21.15.46:0/1946656149 wait complete. 2024-03-31T23:13:45.333 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:45.331+0000 7f6309bd8740 20 client.26056 trim_cache size 0 max 16384 2024-03-31T23:13:45.333 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout:{ 2024-03-31T23:13:45.333 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout: "message": "", 2024-03-31T23:13:45.333 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout: "return_code": 0 2024-03-31T23:13:45.333 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout:} 2024-03-31T23:13:45.334 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:45.331+0000 7f6306ff4640 1 -- 172.21.15.46:0/3425122224 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f62f00a26b0 msgr2=0x7f62f00a4b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:13:45.334 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:45.331+0000 7f6306ff4640 1 --2- 172.21.15.46:0/3425122224 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f62f00a26b0 0x7f62f00a4b70 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f62f8009930 tx=0x7f62f8009500 comp rx=0 tx=0).stop 2024-03-31T23:13:45.334 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:45.331+0000 7f6306ff4640 1 -- 172.21.15.46:0/3425122224 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f63001008a0 msgr2=0x7f6300198d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:13:45.334 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:45.331+0000 7f6306ff4640 1 --2- 172.21.15.46:0/3425122224 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f63001008a0 0x7f6300198d20 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f62f4006010 tx=0x7f62f4005d90 comp rx=0 tx=0).stop 2024-03-31T23:13:45.335 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:45.331+0000 7f6306ff4640 1 -- 172.21.15.46:0/3425122224 shutdown_connections 2024-03-31T23:13:45.335 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:45.331+0000 7f6306ff4640 1 --2- 172.21.15.46:0/3425122224 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f62f00a26b0 0x7f62f00a4b70 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:45.335 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:45.331+0000 7f6306ff4640 1 --2- 172.21.15.46:0/3425122224 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f63001055f0 0x7f63001a3da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:45.335 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:45.331+0000 7f6306ff4640 1 --2- 172.21.15.46:0/3425122224 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6300101260 0x7f6300199260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:45.335 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:45.331+0000 7f6306ff4640 1 --2- 172.21.15.46:0/3425122224 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f63001008a0 0x7f6300198d20 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:45.335 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:45.331+0000 7f6306ff4640 1 -- 172.21.15.46:0/3425122224 >> 172.21.15.46:0/3425122224 conn(0x7f63000774a0 msgr2=0x7f63000fe8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:13:45.335 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:45.331+0000 7f6306ff4640 1 -- 172.21.15.46:0/3425122224 shutdown_connections 2024-03-31T23:13:45.335 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:45.331+0000 7f6306ff4640 1 -- 172.21.15.46:0/3425122224 wait complete. 2024-03-31T23:13:46.191 INFO:tasks.workunit.client.0.smithi046.stdout: 1 21721 2.61 MB/sec warmup 52 sec latency 16.954 ms 2024-03-31T23:13:46.795 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:13:46 smithi162 ceph-mon[36237]: pgmap v64: 129 pgs: 129 active+clean; 35 MiB data, 508 MiB used, 1.0 TiB / 1.0 TiB avail; 373 KiB/s rd, 1.7 MiB/s wr, 177 op/s 2024-03-31T23:13:46.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:13:46 smithi046 ceph-mon[32498]: pgmap v64: 129 pgs: 129 active+clean; 35 MiB data, 508 MiB used, 1.0 TiB / 1.0 TiB avail; 373 KiB/s rd, 1.7 MiB/s wr, 177 op/s 2024-03-31T23:13:46.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:13:46 smithi142 ceph-mon[35271]: pgmap v64: 129 pgs: 129 active+clean; 35 MiB data, 508 MiB used, 1.0 TiB / 1.0 TiB avail; 373 KiB/s rd, 1.7 MiB/s wr, 177 op/s 2024-03-31T23:13:47.191 INFO:tasks.workunit.client.0.smithi046.stdout: 1 22010 2.57 MB/sec warmup 53 sec latency 16.323 ms 2024-03-31T23:13:48.192 INFO:tasks.workunit.client.0.smithi046.stdout: 1 22499 2.55 MB/sec warmup 54 sec latency 22.772 ms 2024-03-31T23:13:48.745 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:13:48 smithi142 ceph-mon[35271]: pgmap v65: 129 pgs: 129 active+clean; 35 MiB data, 508 MiB used, 1.0 TiB / 1.0 TiB avail; 238 KiB/s rd, 1.1 MiB/s wr, 114 op/s 2024-03-31T23:13:48.795 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:13:48 smithi162 ceph-mon[36237]: pgmap v65: 129 pgs: 129 active+clean; 35 MiB data, 508 MiB used, 1.0 TiB / 1.0 TiB avail; 238 KiB/s rd, 1.1 MiB/s wr, 114 op/s 2024-03-31T23:13:48.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:13:48 smithi046 ceph-mon[32498]: pgmap v65: 129 pgs: 129 active+clean; 35 MiB data, 508 MiB used, 1.0 TiB / 1.0 TiB avail; 238 KiB/s rd, 1.1 MiB/s wr, 114 op/s 2024-03-31T23:13:49.192 INFO:tasks.workunit.client.0.smithi046.stdout: 1 23019 2.56 MB/sec warmup 55 sec latency 23.870 ms 2024-03-31T23:13:49.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:13:49 smithi162 ceph-mon[36237]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-31T23:13:49.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:13:49 smithi046 ceph-mon[32498]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-31T23:13:49.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:13:49 smithi142 ceph-mon[35271]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-31T23:13:50.191 INFO:tasks.workunit.client.0.smithi046.stdout: 1 23622 2.55 MB/sec warmup 56 sec latency 32.425 ms 2024-03-31T23:13:50.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:13:50 smithi162 ceph-mon[36237]: pgmap v66: 129 pgs: 129 active+clean; 36 MiB data, 514 MiB used, 1.0 TiB / 1.0 TiB avail; 371 KiB/s rd, 1.6 MiB/s wr, 176 op/s 2024-03-31T23:13:50.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:13:50 smithi162 ceph-mon[36237]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:13:50.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:13:50 smithi162 ceph-mon[36237]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-31T23:13:50.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:13:50 smithi162 ceph-mon[36237]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' 2024-03-31T23:13:50.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:13:50 smithi046 ceph-mon[32498]: pgmap v66: 129 pgs: 129 active+clean; 36 MiB data, 514 MiB used, 1.0 TiB / 1.0 TiB avail; 371 KiB/s rd, 1.6 MiB/s wr, 176 op/s 2024-03-31T23:13:50.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:13:50 smithi046 ceph-mon[32498]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:13:50.862 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:13:50 smithi046 ceph-mon[32498]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-31T23:13:50.862 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:13:50 smithi046 ceph-mon[32498]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' 2024-03-31T23:13:50.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:13:50 smithi142 ceph-mon[35271]: pgmap v66: 129 pgs: 129 active+clean; 36 MiB data, 514 MiB used, 1.0 TiB / 1.0 TiB avail; 371 KiB/s rd, 1.6 MiB/s wr, 176 op/s 2024-03-31T23:13:50.944 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:13:50 smithi142 ceph-mon[35271]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:13:50.944 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:13:50 smithi142 ceph-mon[35271]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-31T23:13:50.944 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:13:50 smithi142 ceph-mon[35271]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' 2024-03-31T23:13:51.192 INFO:tasks.workunit.client.0.smithi046.stdout: 1 24099 2.57 MB/sec warmup 57 sec latency 15.461 ms 2024-03-31T23:13:52.192 INFO:tasks.workunit.client.0.smithi046.stdout: 1 24648 2.59 MB/sec warmup 58 sec latency 15.837 ms 2024-03-31T23:13:52.795 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:13:52 smithi162 ceph-mon[36237]: pgmap v67: 129 pgs: 129 active+clean; 36 MiB data, 514 MiB used, 1.0 TiB / 1.0 TiB avail; 260 KiB/s rd, 1.0 MiB/s wr, 118 op/s 2024-03-31T23:13:52.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:13:52 smithi046 ceph-mon[32498]: pgmap v67: 129 pgs: 129 active+clean; 36 MiB data, 514 MiB used, 1.0 TiB / 1.0 TiB avail; 260 KiB/s rd, 1.0 MiB/s wr, 118 op/s 2024-03-31T23:13:52.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:13:52 smithi142 ceph-mon[35271]: pgmap v67: 129 pgs: 129 active+clean; 36 MiB data, 514 MiB used, 1.0 TiB / 1.0 TiB avail; 260 KiB/s rd, 1.0 MiB/s wr, 118 op/s 2024-03-31T23:13:53.192 INFO:tasks.workunit.client.0.smithi046.stdout: 1 25118 2.62 MB/sec warmup 59 sec latency 16.331 ms 2024-03-31T23:13:54.192 INFO:tasks.workunit.client.0.smithi046.stdout: 1 25375 2.58 MB/sec warmup 60 sec latency 17.167 ms 2024-03-31T23:13:54.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:13:54 smithi162 ceph-mon[36237]: pgmap v68: 129 pgs: 129 active+clean; 36 MiB data, 514 MiB used, 1.0 TiB / 1.0 TiB avail; 260 KiB/s rd, 1.0 MiB/s wr, 118 op/s 2024-03-31T23:13:54.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:13:54 smithi046 ceph-mon[32498]: pgmap v68: 129 pgs: 129 active+clean; 36 MiB data, 514 MiB used, 1.0 TiB / 1.0 TiB avail; 260 KiB/s rd, 1.0 MiB/s wr, 118 op/s 2024-03-31T23:13:54.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:13:54 smithi142 ceph-mon[35271]: pgmap v68: 129 pgs: 129 active+clean; 36 MiB data, 514 MiB used, 1.0 TiB / 1.0 TiB avail; 260 KiB/s rd, 1.0 MiB/s wr, 118 op/s 2024-03-31T23:13:55.192 INFO:tasks.workunit.client.0.smithi046.stdout: 1 25813 2.55 MB/sec warmup 61 sec latency 15.678 ms 2024-03-31T23:13:56.192 INFO:tasks.workunit.client.0.smithi046.stdout: 1 26367 2.58 MB/sec warmup 62 sec latency 15.792 ms 2024-03-31T23:13:56.795 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:13:56 smithi162 ceph-mon[36237]: pgmap v69: 129 pgs: 129 active+clean; 38 MiB data, 537 MiB used, 1.0 TiB / 1.0 TiB avail; 364 KiB/s rd, 1.6 MiB/s wr, 173 op/s 2024-03-31T23:13:56.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:13:56 smithi046 ceph-mon[32498]: pgmap v69: 129 pgs: 129 active+clean; 38 MiB data, 537 MiB used, 1.0 TiB / 1.0 TiB avail; 364 KiB/s rd, 1.6 MiB/s wr, 173 op/s 2024-03-31T23:13:56.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:13:56 smithi142 ceph-mon[35271]: pgmap v69: 129 pgs: 129 active+clean; 38 MiB data, 537 MiB used, 1.0 TiB / 1.0 TiB avail; 364 KiB/s rd, 1.6 MiB/s wr, 173 op/s 2024-03-31T23:13:57.192 INFO:tasks.workunit.client.0.smithi046.stdout: 1 26941 2.55 MB/sec warmup 63 sec latency 16.422 ms 2024-03-31T23:13:58.192 INFO:tasks.workunit.client.0.smithi046.stdout: 1 27434 2.54 MB/sec warmup 64 sec latency 20.811 ms 2024-03-31T23:13:58.428 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:58.426+0000 7fbc05c8f640 1 -- 172.21.15.46:0/3307601304 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbc001055f0 msgr2=0x7fbc001059d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:13:58.428 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:58.426+0000 7fbc05c8f640 1 --2- 172.21.15.46:0/3307601304 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbc001055f0 0x7fbc001059d0 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7fbbf0009900 tx=0x7fbbf002bc30 comp rx=0 tx=0).stop 2024-03-31T23:13:58.428 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:58.426+0000 7fbc05c8f640 1 -- 172.21.15.46:0/3307601304 shutdown_connections 2024-03-31T23:13:58.428 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:58.426+0000 7fbc05c8f640 1 --2- 172.21.15.46:0/3307601304 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbc00101260 0x7fbc0010d480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:58.428 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:58.426+0000 7fbc05c8f640 1 --2- 172.21.15.46:0/3307601304 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbc001008a0 0x7fbc00100d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:58.428 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:58.426+0000 7fbc05c8f640 1 --2- 172.21.15.46:0/3307601304 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbc001055f0 0x7fbc001059d0 unknown :-1 s=CLOSED pgs=647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:58.428 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:58.426+0000 7fbc05c8f640 1 -- 172.21.15.46:0/3307601304 >> 172.21.15.46:0/3307601304 conn(0x7fbc000774a0 msgr2=0x7fbc000778b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:13:58.428 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:58.426+0000 7fbc05c8f640 1 -- 172.21.15.46:0/3307601304 shutdown_connections 2024-03-31T23:13:58.429 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:58.426+0000 7fbc05c8f640 1 -- 172.21.15.46:0/3307601304 wait complete. 2024-03-31T23:13:58.429 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:58.427+0000 7fbc05c8f640 1 Processor -- start 2024-03-31T23:13:58.429 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:58.427+0000 7fbc05c8f640 1 -- start start 2024-03-31T23:13:58.431 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:58.427+0000 7fbc05c8f640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbc001008a0 0x7fbc00198d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:13:58.431 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:58.427+0000 7fbc05c8f640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbc00101260 0x7fbc001992c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:13:58.431 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:58.427+0000 7fbc05c8f640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbc001055f0 0x7fbc001a3e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:13:58.431 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:58.427+0000 7fbc05c8f640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fbc00110980 con 0x7fbc001008a0 2024-03-31T23:13:58.431 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:58.427+0000 7fbc05c8f640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fbc00110800 con 0x7fbc00101260 2024-03-31T23:13:58.431 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:58.427+0000 7fbc05c8f640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fbc00110b00 con 0x7fbc001055f0 2024-03-31T23:13:58.431 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:58.427+0000 7fbbff7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbc001008a0 0x7fbc00198d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:13:58.431 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:58.427+0000 7fbbff7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbc001008a0 0x7fbc00198d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37614/0 (socket says 172.21.15.46:37614) 2024-03-31T23:13:58.431 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:58.427+0000 7fbbff7fe640 1 -- 172.21.15.46:0/3581041119 learned_addr learned my addr 172.21.15.46:0/3581041119 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:13:58.431 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:58.428+0000 7fbbfeffd640 1 --2- 172.21.15.46:0/3581041119 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbc00101260 0x7fbc001992c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:13:58.432 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:58.428+0000 7fbbfffff640 1 --2- 172.21.15.46:0/3581041119 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbc001055f0 0x7fbc001a3e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:13:58.432 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:58.428+0000 7fbbff7fe640 1 -- 172.21.15.46:0/3581041119 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbc001055f0 msgr2=0x7fbc001a3e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:13:58.432 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:58.428+0000 7fbbff7fe640 1 --2- 172.21.15.46:0/3581041119 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbc001055f0 0x7fbc001a3e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:58.432 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:58.428+0000 7fbbff7fe640 1 -- 172.21.15.46:0/3581041119 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbc00101260 msgr2=0x7fbc001992c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:13:58.432 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:58.428+0000 7fbbff7fe640 1 --2- 172.21.15.46:0/3581041119 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbc00101260 0x7fbc001992c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:58.432 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:58.428+0000 7fbbff7fe640 1 -- 172.21.15.46:0/3581041119 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc001a4340 con 0x7fbc001008a0 2024-03-31T23:13:58.432 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:58.428+0000 7fbbfeffd640 1 --2- 172.21.15.46:0/3581041119 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbc00101260 0x7fbc001992c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:13:58.432 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:58.428+0000 7fbbff7fe640 1 --2- 172.21.15.46:0/3581041119 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbc001008a0 0x7fbc00198d80 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7fbbf0006010 tx=0x7fbbf0005d90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:13:58.432 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:58.428+0000 7fbbfcff9640 1 -- 172.21.15.46:0/3581041119 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbf0043020 con 0x7fbc001008a0 2024-03-31T23:13:58.432 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:58.428+0000 7fbbfcff9640 1 -- 172.21.15.46:0/3581041119 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fbbf0004350 con 0x7fbc001008a0 2024-03-31T23:13:58.432 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:58.428+0000 7fbc05c8f640 1 -- 172.21.15.46:0/3581041119 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbc001a4510 con 0x7fbc001008a0 2024-03-31T23:13:58.432 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:58.428+0000 7fbbfcff9640 1 -- 172.21.15.46:0/3581041119 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbf0003a10 con 0x7fbc001008a0 2024-03-31T23:13:58.432 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:58.428+0000 7fbc05c8f640 1 -- 172.21.15.46:0/3581041119 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbc001a49d0 con 0x7fbc001008a0 2024-03-31T23:13:58.439 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:58.430+0000 7fbbfcff9640 1 -- 172.21.15.46:0/3581041119 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 25) ==== 137451+0+0 (secure 0 0 0) 0x7fbbf0046020 con 0x7fbc001008a0 2024-03-31T23:13:58.439 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:58.431+0000 7fbc08873740 1 Processor -- start 2024-03-31T23:13:58.439 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:58.431+0000 7fbc08873740 1 -- start start 2024-03-31T23:13:58.439 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:58.431+0000 7fbbfcff9640 1 --2- 172.21.15.46:0/3581041119 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fbbe80a2730 0x7fbbe80a4bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:13:58.439 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:58.431+0000 7fbc08873740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5563e648b080 0x5563e648b460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:13:58.439 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:58.431+0000 7fbc08873740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5563e648b9a0 0x5563e6498c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:13:58.439 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:58.431+0000 7fbc08873740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5563e64991c0 0x5563e649b5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:13:58.439 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:58.431+0000 7fbc08873740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x5563e63ee8f0 con 0x5563e648b9a0 2024-03-31T23:13:58.439 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:58.431+0000 7fbc08873740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x5563e63978d0 con 0x5563e648b080 2024-03-31T23:13:58.439 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:58.431+0000 7fbc08873740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5563e63ea990 con 0x5563e64991c0 2024-03-31T23:13:58.440 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:58.431+0000 7fbbfffff640 1 --2- 172.21.15.46:0/3581041119 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fbbe80a2730 0x7fbbe80a4bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:13:58.440 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:58.432+0000 7fbbfcff9640 1 -- 172.21.15.46:0/3581041119 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7fbbf00f07b0 con 0x7fbc001008a0 2024-03-31T23:13:58.440 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:58.432+0000 7fbbfeffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5563e648b9a0 0x5563e6498c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:13:58.440 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:58.432+0000 7fbbfeffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5563e648b9a0 0x5563e6498c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37626/0 (socket says 172.21.15.46:37626) 2024-03-31T23:13:58.440 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:58.432+0000 7fbbfeffd640 1 -- 172.21.15.46:0/376153719 learned_addr learned my addr 172.21.15.46:0/376153719 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:13:58.440 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:58.432+0000 7fbbff7fe640 1 --2- 172.21.15.46:0/376153719 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5563e648b080 0x5563e648b460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:13:58.440 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:58.432+0000 7fbbfffff640 1 --2- 172.21.15.46:0/376153719 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5563e64991c0 0x5563e649b5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:13:58.440 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:58.432+0000 7fbbff7fe640 1 -- 172.21.15.46:0/376153719 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5563e64991c0 msgr2=0x5563e649b5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:13:58.440 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:58.432+0000 7fbbff7fe640 1 --2- 172.21.15.46:0/376153719 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5563e64991c0 0x5563e649b5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:58.440 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:58.432+0000 7fbbff7fe640 1 -- 172.21.15.46:0/376153719 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5563e648b9a0 msgr2=0x5563e6498c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:13:58.440 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:58.432+0000 7fbbff7fe640 1 --2- 172.21.15.46:0/376153719 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5563e648b9a0 0x5563e6498c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:58.440 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:58.432+0000 7fbbff7fe640 1 -- 172.21.15.46:0/376153719 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5563e6103210 con 0x5563e648b080 2024-03-31T23:13:58.440 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:58.432+0000 7fbbfffff640 1 --2- 172.21.15.46:0/376153719 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5563e64991c0 0x5563e649b5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T23:13:58.441 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:58.432+0000 7fbbfffff640 1 --2- 172.21.15.46:0/3581041119 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fbbe80a2730 0x7fbbe80a4bf0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fbbf8005fd0 tx=0x7fbbf8013ae0 comp rx=0 tx=0).ready entity=mgr.15654 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:13:58.441 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:58.432+0000 7fbbff7fe640 1 --2- 172.21.15.46:0/376153719 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5563e648b080 0x5563e648b460 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fbbf00edc60 tx=0x7fbbf00bc6b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1f45b9c1b8392c0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:13:58.441 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:58.432+0000 7fbbe5ffb640 1 -- 172.21.15.46:0/376153719 <== mon.2 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbf003d590 con 0x5563e648b080 2024-03-31T23:13:58.441 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:58.432+0000 7fbbe5ffb640 1 -- 172.21.15.46:0/376153719 <== mon.2 v2:172.21.15.142:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fbbf00bcc90 con 0x5563e648b080 2024-03-31T23:13:58.441 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:58.432+0000 7fbbe5ffb640 1 -- 172.21.15.46:0/376153719 <== mon.2 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbf00eddd0 con 0x5563e648b080 2024-03-31T23:13:58.441 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:58.433+0000 7fbc08873740 1 -- 172.21.15.46:0/376153719 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5563e648b080 msgr2=0x5563e648b460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:13:58.441 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:58.433+0000 7fbc08873740 1 --2- 172.21.15.46:0/376153719 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5563e648b080 0x5563e648b460 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fbbf00edc60 tx=0x7fbbf00bc6b0 comp rx=0 tx=0).stop 2024-03-31T23:13:58.441 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:58.433+0000 7fbc08873740 1 -- 172.21.15.46:0/376153719 shutdown_connections 2024-03-31T23:13:58.441 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:58.433+0000 7fbc08873740 1 --2- 172.21.15.46:0/376153719 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5563e64991c0 0x5563e649b5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:58.441 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:58.433+0000 7fbc08873740 1 --2- 172.21.15.46:0/376153719 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5563e648b9a0 0x5563e6498c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:58.441 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:58.433+0000 7fbc08873740 1 --2- 172.21.15.46:0/376153719 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5563e648b080 0x5563e648b460 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:58.441 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:58.433+0000 7fbc08873740 1 -- 172.21.15.46:0/376153719 >> 172.21.15.46:0/376153719 conn(0x5563e6485fe0 msgr2=0x5563e64863c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:13:58.441 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:58.433+0000 7fbc08873740 1 -- 172.21.15.46:0/376153719 shutdown_connections 2024-03-31T23:13:58.442 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:58.433+0000 7fbc08873740 1 -- 172.21.15.46:0/376153719 wait complete. 2024-03-31T23:13:58.442 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:58.434+0000 7fbc08873740 1 -- start start 2024-03-31T23:13:58.442 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:58.434+0000 7fbc08873740 1 Processor -- start 2024-03-31T23:13:58.442 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:58.434+0000 7fbc08873740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5563e648b080 0x5563e6496250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:13:58.442 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:58.434+0000 7fbc08873740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5563e648b9a0 0x5563e6496790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:13:58.442 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:58.434+0000 7fbc08873740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5563e64991c0 0x5563e6496cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:13:58.442 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:58.434+0000 7fbc08873740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x5563e63978d0 con 0x5563e648b080 2024-03-31T23:13:58.442 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:58.434+0000 7fbc08873740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x5563e638cf80 con 0x5563e648b9a0 2024-03-31T23:13:58.442 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:58.434+0000 7fbc08873740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5563e649bbb0 con 0x5563e64991c0 2024-03-31T23:13:58.442 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:58.434+0000 7fbc08873740 10 client.0 _ll_register_callbacks cb 0x5563e63e5290 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T23:13:58.442 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:58.434+0000 7fbbff7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5563e648b080 0x5563e6496250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:13:58.443 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:58.434+0000 7fbbff7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5563e648b080 0x5563e6496250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37632/0 (socket says 172.21.15.46:37632) 2024-03-31T23:13:58.443 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:58.434+0000 7fbbff7fe640 1 -- 172.21.15.46:0/1038956395 learned_addr learned my addr 172.21.15.46:0/1038956395 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:13:58.443 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:58.434+0000 7fbbfeffd640 1 --2- 172.21.15.46:0/1038956395 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5563e648b9a0 0x5563e6496790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:13:58.443 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:58.434+0000 7fbbfffff640 1 --2- 172.21.15.46:0/1038956395 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5563e64991c0 0x5563e6496cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:13:58.443 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:58.434+0000 7fbbff7fe640 1 -- 172.21.15.46:0/1038956395 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5563e64991c0 msgr2=0x5563e6496cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:13:58.443 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:58.434+0000 7fbbff7fe640 1 --2- 172.21.15.46:0/1038956395 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5563e64991c0 0x5563e6496cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:58.443 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:58.434+0000 7fbbff7fe640 1 -- 172.21.15.46:0/1038956395 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5563e648b9a0 msgr2=0x5563e6496790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:13:58.443 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:58.434+0000 7fbbff7fe640 1 --2- 172.21.15.46:0/1038956395 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5563e648b9a0 0x5563e6496790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:58.443 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:58.435+0000 7fbbff7fe640 1 -- 172.21.15.46:0/1038956395 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5563e61cf620 con 0x5563e648b080 2024-03-31T23:13:58.443 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:58.435+0000 7fbbff7fe640 1 --2- 172.21.15.46:0/1038956395 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5563e648b080 0x5563e6496250 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7fbbf004c000 tx=0x7fbbf00ee000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=21036618c83ab1be server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:13:58.443 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:58.435+0000 7fbbdeffd640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-31T23:13:58.443 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:58.435+0000 7fbbdeffd640 1 -- 172.21.15.46:0/1038956395 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbf00c56a0 con 0x5563e648b080 2024-03-31T23:13:58.443 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:58.435+0000 7fbbdeffd640 1 -- 172.21.15.46:0/1038956395 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7fbbf00f03e0 con 0x5563e648b080 2024-03-31T23:13:58.443 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:58.435+0000 7fbc08873740 1 -- 172.21.15.46:0/1038956395 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbbf00bcc90 con 0x5563e648b080 2024-03-31T23:13:58.444 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:58.435+0000 7fbbdeffd640 1 -- 172.21.15.46:0/1038956395 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fbbf00f05e0 con 0x5563e648b080 2024-03-31T23:13:58.444 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:58.435+0000 7fbc08873740 10 client.16329 fetch_fsmap learned FSMap version 28 2024-03-31T23:13:58.444 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:58.435+0000 7fbc08873740 1 -- 172.21.15.46:0/1038956395 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5563e64a7890 con 0x5563e648b080 2024-03-31T23:13:58.444 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:58.435+0000 7fbbdeffd640 1 -- 172.21.15.46:0/1038956395 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fbbf00bcc90 con 0x5563e648b080 2024-03-31T23:13:58.444 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:58.435+0000 7fbbdeffd640 1 -- 172.21.15.46:0/1038956395 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7fbbf0042250 con 0x5563e648b080 2024-03-31T23:13:58.444 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:58.435+0000 7fbc08873740 10 client.16329 fetch_fsmap finished waiting for FSMap version 28 2024-03-31T23:13:58.444 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:58.435+0000 7fbc08873740 10 client.16329 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-31T23:13:58.444 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:58.435+0000 7fbc08873740 20 client.16329 populate_metadata read hostname 'smithi046' 2024-03-31T23:13:58.444 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:58.435+0000 7fbc08873740 1 --2- 172.21.15.46:0/1038956395 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x5563e64a8ca0 0x5563e64ab0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:13:58.444 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:58.435+0000 7fbc08873740 4 client.16329 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-31T23:13:58.444 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:58.435+0000 7fbc08873740 1 -- 172.21.15.46:0/1038956395 --> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5563e6402120 con 0x5563e64a8ca0 2024-03-31T23:13:58.444 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:58.436+0000 7fbbfeffd640 1 --2- 172.21.15.46:0/1038956395 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x5563e64a8ca0 0x5563e64ab0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:13:58.444 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:58.436+0000 7fbbfeffd640 1 --2- 172.21.15.46:0/1038956395 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x5563e64a8ca0 0x5563e64ab0b0 crc :-1 s=READY pgs=93 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c463ddc05f5ab36f server_cookie=2f1d97e4c348892d in_seq=0 out_seq=0 2024-03-31T23:13:58.444 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:58.436+0000 7fbbdeffd640 10 client.16329 ms_handle_connect on v2:172.21.15.142:6838/2519143918 2024-03-31T23:13:58.445 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:58.437+0000 7fbbdeffd640 1 -- 172.21.15.46:0/1038956395 <== mds.0 v2:172.21.15.142:6838/2519143918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5563e6402120 con 0x5563e64a8ca0 2024-03-31T23:13:58.445 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:58.437+0000 7fbbdeffd640 10 client.16329 handle_command_reply: tid=0 2024-03-31T23:13:58.445 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:58.437+0000 7fbc08873740 1 client.16329 shutdown 2024-03-31T23:13:58.445 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:58.437+0000 7fbc08873740 1 -- 172.21.15.46:0/1038956395 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5563e648b080 msgr2=0x5563e6496250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:13:58.445 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:58.437+0000 7fbc08873740 1 --2- 172.21.15.46:0/1038956395 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5563e648b080 0x5563e6496250 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7fbbf004c000 tx=0x7fbbf00ee000 comp rx=0 tx=0).stop 2024-03-31T23:13:58.445 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:58.437+0000 7fbc08873740 1 -- 172.21.15.46:0/1038956395 shutdown_connections 2024-03-31T23:13:58.445 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:58.437+0000 7fbc08873740 1 --2- 172.21.15.46:0/1038956395 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x5563e64a8ca0 0x5563e64ab0b0 crc :-1 s=READY pgs=93 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:58.445 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:58.437+0000 7fbbdeffd640 0 client.16329 ms_handle_reset on v2:172.21.15.142:6838/2519143918 2024-03-31T23:13:58.445 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:58.437+0000 7fbc08873740 1 --2- 172.21.15.46:0/1038956395 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5563e64991c0 0x5563e6496cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:58.445 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:58.437+0000 7fbc08873740 1 --2- 172.21.15.46:0/1038956395 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5563e648b9a0 0x5563e6496790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:58.445 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:58.437+0000 7fbc08873740 1 --2- 172.21.15.46:0/1038956395 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5563e648b080 0x5563e6496250 unknown :-1 s=CLOSED pgs=649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:58.445 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:58.437+0000 7fbc08873740 1 -- 172.21.15.46:0/1038956395 >> 172.21.15.46:0/1038956395 conn(0x5563e6485fe0 msgr2=0x5563e6487d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:13:58.445 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:58.437+0000 7fbc08873740 1 -- 172.21.15.46:0/1038956395 shutdown_connections 2024-03-31T23:13:58.446 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:58.437+0000 7fbc08873740 1 -- 172.21.15.46:0/1038956395 wait complete. 2024-03-31T23:13:58.446 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:58.437+0000 7fbc08873740 20 client.16329 trim_cache size 0 max 16384 2024-03-31T23:13:58.459 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:58.453+0000 7fbc08873740 1 Processor -- start 2024-03-31T23:13:58.459 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:58.454+0000 7fbc08873740 1 -- start start 2024-03-31T23:13:58.459 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:58.454+0000 7fbc08873740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5563e648b080 0x5563e6498980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:13:58.459 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:58.454+0000 7fbc08873740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5563e648b9a0 0x5563e64a4720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:13:58.459 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:58.454+0000 7fbc08873740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5563e64991c0 0x5563e64b0970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:13:58.459 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:58.454+0000 7fbc08873740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fbc001af1c0 con 0x5563e648b080 2024-03-31T23:13:58.460 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:58.454+0000 7fbc08873740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fbc001aca90 con 0x5563e648b9a0 2024-03-31T23:13:58.460 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:58.454+0000 7fbc08873740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fbc001aa8e0 con 0x5563e64991c0 2024-03-31T23:13:58.460 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:58.454+0000 7fbbff7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5563e648b080 0x5563e6498980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:13:58.460 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:58.454+0000 7fbbff7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5563e648b080 0x5563e6498980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37642/0 (socket says 172.21.15.46:37642) 2024-03-31T23:13:58.460 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:58.454+0000 7fbbfeffd640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5563e648b9a0 0x5563e64a4720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:13:58.460 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:58.454+0000 7fbbff7fe640 1 -- 172.21.15.46:0/122263287 learned_addr learned my addr 172.21.15.46:0/122263287 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:13:58.460 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:58.454+0000 7fbbfffff640 1 --2- 172.21.15.46:0/122263287 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5563e64991c0 0x5563e64b0970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:13:58.460 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:58.454+0000 7fbbff7fe640 1 -- 172.21.15.46:0/122263287 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5563e64991c0 msgr2=0x5563e64b0970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:13:58.460 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:58.454+0000 7fbbff7fe640 1 --2- 172.21.15.46:0/122263287 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5563e64991c0 0x5563e64b0970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:58.460 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:58.454+0000 7fbbff7fe640 1 -- 172.21.15.46:0/122263287 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5563e648b9a0 msgr2=0x5563e64a4720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:13:58.460 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:58.454+0000 7fbbff7fe640 1 --2- 172.21.15.46:0/122263287 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5563e648b9a0 0x5563e64a4720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:58.460 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:58.454+0000 7fbbff7fe640 1 -- 172.21.15.46:0/122263287 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5563e648a9a0 con 0x5563e648b080 2024-03-31T23:13:58.460 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:58.454+0000 7fbbfeffd640 1 --2- 172.21.15.46:0/122263287 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5563e648b9a0 0x5563e64a4720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:13:58.460 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:58.454+0000 7fbbff7fe640 1 --2- 172.21.15.46:0/122263287 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5563e648b080 0x5563e6498980 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7fbbf002c720 tx=0x7fbbf0009290 comp rx=0 tx=0).ready entity=mon.0 client_cookie=656cfdd49766cd35 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:13:58.461 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:58.454+0000 7fbbe57fa640 1 -- 172.21.15.46:0/122263287 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbf00c56a0 con 0x5563e648b080 2024-03-31T23:13:58.461 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:58.455+0000 7fbbe57fa640 1 -- 172.21.15.46:0/122263287 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fbbf00a2070 con 0x5563e648b080 2024-03-31T23:13:58.461 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:58.455+0000 7fbbe57fa640 1 -- 172.21.15.46:0/122263287 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbf00c0de0 con 0x5563e648b080 2024-03-31T23:13:58.461 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:58.455+0000 7fbc08873740 1 -- 172.21.15.46:0/122263287 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5563e648b080 msgr2=0x5563e6498980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:13:58.461 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:58.455+0000 7fbc08873740 1 --2- 172.21.15.46:0/122263287 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5563e648b080 0x5563e6498980 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7fbbf002c720 tx=0x7fbbf0009290 comp rx=0 tx=0).stop 2024-03-31T23:13:58.461 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:58.455+0000 7fbc08873740 1 -- 172.21.15.46:0/122263287 shutdown_connections 2024-03-31T23:13:58.461 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:58.455+0000 7fbc08873740 1 --2- 172.21.15.46:0/122263287 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5563e64991c0 0x5563e64b0970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:58.461 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:58.455+0000 7fbc08873740 1 --2- 172.21.15.46:0/122263287 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5563e648b9a0 0x5563e64a4720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:58.461 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:58.455+0000 7fbc08873740 1 --2- 172.21.15.46:0/122263287 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5563e648b080 0x5563e6498980 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:58.461 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:58.455+0000 7fbc08873740 1 -- 172.21.15.46:0/122263287 >> 172.21.15.46:0/122263287 conn(0x5563e6485fe0 msgr2=0x5563e64863c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:13:58.461 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:58.455+0000 7fbc08873740 1 -- 172.21.15.46:0/122263287 shutdown_connections 2024-03-31T23:13:58.461 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:58.455+0000 7fbc08873740 1 -- 172.21.15.46:0/122263287 wait complete. 2024-03-31T23:13:58.461 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:58.456+0000 7fbc08873740 1 -- start start 2024-03-31T23:13:58.461 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:58.456+0000 7fbc08873740 1 Processor -- start 2024-03-31T23:13:58.462 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:58.456+0000 7fbc08873740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5563e648b080 0x5563e6492d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:13:58.462 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:58.456+0000 7fbc08873740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5563e648b9a0 0x5563e64932d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:13:58.462 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:58.456+0000 7fbc08873740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5563e64991c0 0x5563e6493810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:13:58.462 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:58.456+0000 7fbc08873740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fbc001aaf50 con 0x5563e64991c0 2024-03-31T23:13:58.462 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:58.456+0000 7fbc08873740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fbc001ac3c0 con 0x5563e648b080 2024-03-31T23:13:58.462 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:58.456+0000 7fbc08873740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fbc001ab620 con 0x5563e648b9a0 2024-03-31T23:13:58.462 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:58.456+0000 7fbc08873740 10 client.0 _ll_register_callbacks cb 0x7fbbf4009b50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T23:13:58.462 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:58.456+0000 7fbbfffff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5563e64991c0 0x5563e6493810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:13:58.462 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:58.456+0000 7fbbfffff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5563e64991c0 0x5563e6493810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37658/0 (socket says 172.21.15.46:37658) 2024-03-31T23:13:58.462 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:58.456+0000 7fbbfffff640 1 -- 172.21.15.46:0/1215069700 learned_addr learned my addr 172.21.15.46:0/1215069700 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:13:58.462 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:58.456+0000 7fbbfeffd640 1 --2- 172.21.15.46:0/1215069700 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5563e648b9a0 0x5563e64932d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:13:58.462 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:58.456+0000 7fbbff7fe640 1 --2- 172.21.15.46:0/1215069700 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5563e648b080 0x5563e6492d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:13:58.462 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:58.456+0000 7fbbfffff640 1 -- 172.21.15.46:0/1215069700 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5563e648b9a0 msgr2=0x5563e64932d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:13:58.463 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:58.456+0000 7fbbfffff640 1 --2- 172.21.15.46:0/1215069700 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5563e648b9a0 0x5563e64932d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:58.463 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:58.456+0000 7fbbfffff640 1 -- 172.21.15.46:0/1215069700 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5563e648b080 msgr2=0x5563e6492d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:13:58.463 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:58.456+0000 7fbbfffff640 1 --2- 172.21.15.46:0/1215069700 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5563e648b080 0x5563e6492d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:58.463 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:58.456+0000 7fbbfffff640 1 -- 172.21.15.46:0/1215069700 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5563e650dfa0 con 0x5563e64991c0 2024-03-31T23:13:58.463 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:58.456+0000 7fbbff7fe640 1 --2- 172.21.15.46:0/1215069700 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5563e648b080 0x5563e6492d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:13:58.463 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:58.457+0000 7fbbfffff640 1 --2- 172.21.15.46:0/1215069700 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5563e64991c0 0x5563e6493810 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7fbbf800fa10 tx=0x7fbbf8004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eff728505716a10d server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:13:58.463 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:58.457+0000 7fbbdeffd640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-31T23:13:58.463 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:58.457+0000 7fbbdeffd640 1 -- 172.21.15.46:0/1215069700 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbf8004b20 con 0x5563e64991c0 2024-03-31T23:13:58.463 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:58.457+0000 7fbbdeffd640 1 -- 172.21.15.46:0/1215069700 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7fbbf8004cc0 con 0x5563e64991c0 2024-03-31T23:13:58.463 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:58.457+0000 7fbc08873740 1 -- 172.21.15.46:0/1215069700 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbbf00a2070 con 0x5563e64991c0 2024-03-31T23:13:58.463 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:58.457+0000 7fbbdeffd640 1 -- 172.21.15.46:0/1215069700 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fbbf8004250 con 0x5563e64991c0 2024-03-31T23:13:58.463 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:58.457+0000 7fbc08873740 10 client.16335 fetch_fsmap learned FSMap version 28 2024-03-31T23:13:58.463 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:58.457+0000 7fbc08873740 1 -- 172.21.15.46:0/1215069700 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5563e650f1d0 con 0x5563e64991c0 2024-03-31T23:13:58.463 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:58.457+0000 7fbbdeffd640 1 -- 172.21.15.46:0/1215069700 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fbbf00a2070 con 0x5563e64991c0 2024-03-31T23:13:58.464 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:58.457+0000 7fbbdeffd640 1 -- 172.21.15.46:0/1215069700 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7fbbf8029070 con 0x5563e64991c0 2024-03-31T23:13:58.464 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:58.457+0000 7fbc08873740 10 client.16335 fetch_fsmap finished waiting for FSMap version 28 2024-03-31T23:13:58.464 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:58.457+0000 7fbc08873740 10 client.16335 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-31T23:13:58.464 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:58.457+0000 7fbc08873740 20 client.16335 populate_metadata read hostname 'smithi046' 2024-03-31T23:13:58.464 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:58.457+0000 7fbc08873740 1 --2- 172.21.15.46:0/1215069700 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x5563e64a8ca0 0x5563e6511a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:13:58.464 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:58.457+0000 7fbc08873740 4 client.16335 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-31T23:13:58.464 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:58.457+0000 7fbc08873740 1 -- 172.21.15.46:0/1215069700 --> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5563e63fbe30 con 0x5563e64a8ca0 2024-03-31T23:13:58.464 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:58.458+0000 7fbbff7fe640 1 --2- 172.21.15.46:0/1215069700 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x5563e64a8ca0 0x5563e6511a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:13:58.464 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:58.458+0000 7fbbff7fe640 1 --2- 172.21.15.46:0/1215069700 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x5563e64a8ca0 0x5563e6511a70 crc :-1 s=READY pgs=94 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5a8eabf42f3d35aa server_cookie=dacc415dc4817018 in_seq=0 out_seq=0 2024-03-31T23:13:58.464 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:58.458+0000 7fbbdeffd640 10 client.16335 ms_handle_connect on v2:172.21.15.142:6838/2519143918 2024-03-31T23:13:58.482 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:58.480+0000 7fbbdeffd640 1 -- 172.21.15.46:0/1215069700 <== mds.0 v2:172.21.15.142:6838/2519143918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5563e63fbe30 con 0x5563e64a8ca0 2024-03-31T23:13:58.482 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:58.480+0000 7fbbdeffd640 10 client.16335 handle_command_reply: tid=0 2024-03-31T23:13:58.482 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:58.480+0000 7fbc08873740 1 client.16335 shutdown 2024-03-31T23:13:58.482 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:58.480+0000 7fbc08873740 1 -- 172.21.15.46:0/1215069700 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5563e64991c0 msgr2=0x5563e6493810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:13:58.482 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:58.480+0000 7fbc08873740 1 --2- 172.21.15.46:0/1215069700 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5563e64991c0 0x5563e6493810 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7fbbf800fa10 tx=0x7fbbf8004ab0 comp rx=0 tx=0).stop 2024-03-31T23:13:58.482 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:58.480+0000 7fbc08873740 1 -- 172.21.15.46:0/1215069700 shutdown_connections 2024-03-31T23:13:58.482 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:58.480+0000 7fbc08873740 1 --2- 172.21.15.46:0/1215069700 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x5563e64a8ca0 0x5563e6511a70 crc :-1 s=READY pgs=94 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:58.482 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:58.480+0000 7fbc08873740 1 --2- 172.21.15.46:0/1215069700 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5563e64991c0 0x5563e6493810 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:58.483 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:58.480+0000 7fbc08873740 1 --2- 172.21.15.46:0/1215069700 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5563e648b9a0 0x5563e64932d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:58.483 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:58.480+0000 7fbc08873740 1 --2- 172.21.15.46:0/1215069700 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5563e648b080 0x5563e6492d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:58.483 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:58.480+0000 7fbc08873740 1 -- 172.21.15.46:0/1215069700 >> 172.21.15.46:0/1215069700 conn(0x5563e6485fe0 msgr2=0x5563e64873c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:13:58.483 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:58.480+0000 7fbbdeffd640 0 client.16335 ms_handle_reset on v2:172.21.15.142:6838/2519143918 2024-03-31T23:13:58.483 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:58.480+0000 7fbc08873740 1 -- 172.21.15.46:0/1215069700 shutdown_connections 2024-03-31T23:13:58.483 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:58.480+0000 7fbc08873740 1 -- 172.21.15.46:0/1215069700 wait complete. 2024-03-31T23:13:58.483 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:58.481+0000 7fbc08873740 20 client.16335 trim_cache size 0 max 16384 2024-03-31T23:13:58.483 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout:{ 2024-03-31T23:13:58.483 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout: "message": "", 2024-03-31T23:13:58.483 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout: "return_code": 0 2024-03-31T23:13:58.484 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout:} 2024-03-31T23:13:58.484 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:58.481+0000 7fbc05c8f640 1 -- 172.21.15.46:0/3581041119 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fbbe80a2730 msgr2=0x7fbbe80a4bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:13:58.484 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:58.481+0000 7fbc05c8f640 1 --2- 172.21.15.46:0/3581041119 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fbbe80a2730 0x7fbbe80a4bf0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fbbf8005fd0 tx=0x7fbbf8013ae0 comp rx=0 tx=0).stop 2024-03-31T23:13:58.484 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:58.481+0000 7fbc05c8f640 1 -- 172.21.15.46:0/3581041119 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbc001008a0 msgr2=0x7fbc00198d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:13:58.485 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:58.481+0000 7fbc05c8f640 1 --2- 172.21.15.46:0/3581041119 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbc001008a0 0x7fbc00198d80 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7fbbf0006010 tx=0x7fbbf0005d90 comp rx=0 tx=0).stop 2024-03-31T23:13:58.485 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:58.481+0000 7fbc05c8f640 1 -- 172.21.15.46:0/3581041119 shutdown_connections 2024-03-31T23:13:58.485 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:58.481+0000 7fbc05c8f640 1 --2- 172.21.15.46:0/3581041119 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fbbe80a2730 0x7fbbe80a4bf0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:58.485 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:58.481+0000 7fbc05c8f640 1 --2- 172.21.15.46:0/3581041119 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbc001055f0 0x7fbc001a3e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:58.485 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:58.481+0000 7fbc05c8f640 1 --2- 172.21.15.46:0/3581041119 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbc00101260 0x7fbc001992c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:58.485 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:58.481+0000 7fbc05c8f640 1 --2- 172.21.15.46:0/3581041119 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbc001008a0 0x7fbc00198d80 unknown :-1 s=CLOSED pgs=648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:13:58.485 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:58.481+0000 7fbc05c8f640 1 -- 172.21.15.46:0/3581041119 >> 172.21.15.46:0/3581041119 conn(0x7fbc000774a0 msgr2=0x7fbc000fe8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:13:58.485 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:58.481+0000 7fbc05c8f640 1 -- 172.21.15.46:0/3581041119 shutdown_connections 2024-03-31T23:13:58.485 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:13:58.481+0000 7fbc05c8f640 1 -- 172.21.15.46:0/3581041119 wait complete. 2024-03-31T23:13:58.795 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:13:58 smithi162 ceph-mon[36237]: pgmap v70: 129 pgs: 129 active+clean; 38 MiB data, 537 MiB used, 1.0 TiB / 1.0 TiB avail; 237 KiB/s rd, 1.1 MiB/s wr, 116 op/s 2024-03-31T23:13:58.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:13:58 smithi046 ceph-mon[32498]: pgmap v70: 129 pgs: 129 active+clean; 38 MiB data, 537 MiB used, 1.0 TiB / 1.0 TiB avail; 237 KiB/s rd, 1.1 MiB/s wr, 116 op/s 2024-03-31T23:13:58.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:13:58 smithi142 ceph-mon[35271]: pgmap v70: 129 pgs: 129 active+clean; 38 MiB data, 537 MiB used, 1.0 TiB / 1.0 TiB avail; 237 KiB/s rd, 1.1 MiB/s wr, 116 op/s 2024-03-31T23:13:59.192 INFO:tasks.workunit.client.0.smithi046.stdout: 1 28161 2.60 MB/sec warmup 65 sec latency 24.641 ms 2024-03-31T23:14:00.192 INFO:tasks.workunit.client.0.smithi046.stdout: 1 28551 2.62 MB/sec warmup 66 sec latency 16.018 ms 2024-03-31T23:14:00.795 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:14:00 smithi162 ceph-mon[36237]: pgmap v71: 129 pgs: 129 active+clean; 35 MiB data, 541 MiB used, 1.0 TiB / 1.0 TiB avail; 370 KiB/s rd, 1.6 MiB/s wr, 180 op/s 2024-03-31T23:14:00.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:14:00 smithi046 ceph-mon[32498]: pgmap v71: 129 pgs: 129 active+clean; 35 MiB data, 541 MiB used, 1.0 TiB / 1.0 TiB avail; 370 KiB/s rd, 1.6 MiB/s wr, 180 op/s 2024-03-31T23:14:00.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:14:00 smithi142 ceph-mon[35271]: pgmap v71: 129 pgs: 129 active+clean; 35 MiB data, 541 MiB used, 1.0 TiB / 1.0 TiB avail; 370 KiB/s rd, 1.6 MiB/s wr, 180 op/s 2024-03-31T23:14:01.192 INFO:tasks.workunit.client.0.smithi046.stdout: 1 28836 2.60 MB/sec warmup 67 sec latency 21.682 ms 2024-03-31T23:14:02.192 INFO:tasks.workunit.client.0.smithi046.stdout: 1 29201 2.56 MB/sec warmup 68 sec latency 16.215 ms 2024-03-31T23:14:02.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:14:02 smithi162 ceph-mon[36237]: pgmap v72: 129 pgs: 129 active+clean; 35 MiB data, 541 MiB used, 1.0 TiB / 1.0 TiB avail; 236 KiB/s rd, 1.1 MiB/s wr, 118 op/s 2024-03-31T23:14:02.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:14:02 smithi046 ceph-mon[32498]: pgmap v72: 129 pgs: 129 active+clean; 35 MiB data, 541 MiB used, 1.0 TiB / 1.0 TiB avail; 236 KiB/s rd, 1.1 MiB/s wr, 118 op/s 2024-03-31T23:14:02.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:14:02 smithi142 ceph-mon[35271]: pgmap v72: 129 pgs: 129 active+clean; 35 MiB data, 541 MiB used, 1.0 TiB / 1.0 TiB avail; 236 KiB/s rd, 1.1 MiB/s wr, 118 op/s 2024-03-31T23:14:03.192 INFO:tasks.workunit.client.0.smithi046.stdout: 1 29655 2.55 MB/sec warmup 69 sec latency 19.246 ms 2024-03-31T23:14:04.192 INFO:tasks.workunit.client.0.smithi046.stdout: 1 30264 2.56 MB/sec warmup 70 sec latency 15.856 ms 2024-03-31T23:14:04.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:14:04 smithi162 ceph-mon[36237]: pgmap v73: 129 pgs: 129 active+clean; 35 MiB data, 541 MiB used, 1.0 TiB / 1.0 TiB avail; 236 KiB/s rd, 1.1 MiB/s wr, 118 op/s 2024-03-31T23:14:04.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:14:04 smithi046 ceph-mon[32498]: pgmap v73: 129 pgs: 129 active+clean; 35 MiB data, 541 MiB used, 1.0 TiB / 1.0 TiB avail; 236 KiB/s rd, 1.1 MiB/s wr, 118 op/s 2024-03-31T23:14:04.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:14:04 smithi142 ceph-mon[35271]: pgmap v73: 129 pgs: 129 active+clean; 35 MiB data, 541 MiB used, 1.0 TiB / 1.0 TiB avail; 236 KiB/s rd, 1.1 MiB/s wr, 118 op/s 2024-03-31T23:14:05.192 INFO:tasks.workunit.client.0.smithi046.stdout: 1 30851 2.55 MB/sec warmup 71 sec latency 15.815 ms 2024-03-31T23:14:06.192 INFO:tasks.workunit.client.0.smithi046.stdout: 1 31588 2.60 MB/sec warmup 72 sec latency 16.284 ms 2024-03-31T23:14:06.795 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:14:06 smithi162 ceph-mon[36237]: pgmap v74: 129 pgs: 129 active+clean; 39 MiB data, 570 MiB used, 1.0 TiB / 1.0 TiB avail; 403 KiB/s rd, 1.7 MiB/s wr, 194 op/s 2024-03-31T23:14:06.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:14:06 smithi046 ceph-mon[32498]: pgmap v74: 129 pgs: 129 active+clean; 39 MiB data, 570 MiB used, 1.0 TiB / 1.0 TiB avail; 403 KiB/s rd, 1.7 MiB/s wr, 194 op/s 2024-03-31T23:14:06.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:14:06 smithi142 ceph-mon[35271]: pgmap v74: 129 pgs: 129 active+clean; 39 MiB data, 570 MiB used, 1.0 TiB / 1.0 TiB avail; 403 KiB/s rd, 1.7 MiB/s wr, 194 op/s 2024-03-31T23:14:07.192 INFO:tasks.workunit.client.0.smithi046.stdout: 1 31953 2.62 MB/sec warmup 73 sec latency 16.181 ms 2024-03-31T23:14:08.192 INFO:tasks.workunit.client.0.smithi046.stdout: 1 32329 2.61 MB/sec warmup 74 sec latency 15.328 ms 2024-03-31T23:14:08.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:14:08 smithi162 ceph-mon[36237]: pgmap v75: 129 pgs: 129 active+clean; 39 MiB data, 570 MiB used, 1.0 TiB / 1.0 TiB avail; 300 KiB/s rd, 1.1 MiB/s wr, 139 op/s 2024-03-31T23:14:08.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:14:08 smithi046 ceph-mon[32498]: pgmap v75: 129 pgs: 129 active+clean; 39 MiB data, 570 MiB used, 1.0 TiB / 1.0 TiB avail; 300 KiB/s rd, 1.1 MiB/s wr, 139 op/s 2024-03-31T23:14:08.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:14:08 smithi142 ceph-mon[35271]: pgmap v75: 129 pgs: 129 active+clean; 39 MiB data, 570 MiB used, 1.0 TiB / 1.0 TiB avail; 300 KiB/s rd, 1.1 MiB/s wr, 139 op/s 2024-03-31T23:14:09.192 INFO:tasks.workunit.client.0.smithi046.stdout: 1 32586 2.57 MB/sec warmup 75 sec latency 16.096 ms 2024-03-31T23:14:10.192 INFO:tasks.workunit.client.0.smithi046.stdout: 1 33064 2.57 MB/sec warmup 76 sec latency 16.752 ms 2024-03-31T23:14:10.795 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:14:10 smithi162 ceph-mon[36237]: pgmap v76: 129 pgs: 129 active+clean; 43 MiB data, 579 MiB used, 1.0 TiB / 1.0 TiB avail; 434 KiB/s rd, 1.7 MiB/s wr, 196 op/s 2024-03-31T23:14:10.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:14:10 smithi046 ceph-mon[32498]: pgmap v76: 129 pgs: 129 active+clean; 43 MiB data, 579 MiB used, 1.0 TiB / 1.0 TiB avail; 434 KiB/s rd, 1.7 MiB/s wr, 196 op/s 2024-03-31T23:14:10.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:14:10 smithi142 ceph-mon[35271]: pgmap v76: 129 pgs: 129 active+clean; 43 MiB data, 579 MiB used, 1.0 TiB / 1.0 TiB avail; 434 KiB/s rd, 1.7 MiB/s wr, 196 op/s 2024-03-31T23:14:11.192 INFO:tasks.workunit.client.0.smithi046.stdout: 1 33613 2.58 MB/sec warmup 77 sec latency 16.652 ms 2024-03-31T23:14:11.586 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:11.585+0000 7fba75f80640 1 -- 172.21.15.46:0/4203855193 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fba701055f0 msgr2=0x7fba701059d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:14:11.586 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:11.585+0000 7fba75f80640 1 --2- 172.21.15.46:0/4203855193 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fba701055f0 0x7fba701059d0 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7fba60009900 tx=0x7fba6002bc30 comp rx=0 tx=0).stop 2024-03-31T23:14:11.586 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:11.586+0000 7fba75f80640 1 -- 172.21.15.46:0/4203855193 shutdown_connections 2024-03-31T23:14:11.586 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:11.586+0000 7fba75f80640 1 --2- 172.21.15.46:0/4203855193 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fba70101260 0x7fba7010d480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:14:11.587 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:11.586+0000 7fba75f80640 1 --2- 172.21.15.46:0/4203855193 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fba701008a0 0x7fba70100d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:14:11.587 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:11.586+0000 7fba75f80640 1 --2- 172.21.15.46:0/4203855193 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fba701055f0 0x7fba701059d0 unknown :-1 s=CLOSED pgs=652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:14:11.587 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:11.586+0000 7fba75f80640 1 -- 172.21.15.46:0/4203855193 >> 172.21.15.46:0/4203855193 conn(0x7fba700774a0 msgr2=0x7fba700778b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:14:11.587 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:11.586+0000 7fba75f80640 1 -- 172.21.15.46:0/4203855193 shutdown_connections 2024-03-31T23:14:11.587 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:11.586+0000 7fba75f80640 1 -- 172.21.15.46:0/4203855193 wait complete. 2024-03-31T23:14:11.587 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:11.586+0000 7fba75f80640 1 Processor -- start 2024-03-31T23:14:11.587 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:11.586+0000 7fba75f80640 1 -- start start 2024-03-31T23:14:11.589 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:11.587+0000 7fba75f80640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fba701008a0 0x7fba70197ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:14:11.589 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:11.587+0000 7fba75f80640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fba70101260 0x7fba70198400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:14:11.589 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:11.587+0000 7fba75f80640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fba701055f0 0x7fba7019c730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:14:11.589 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:11.587+0000 7fba75f80640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fba70111ae0 con 0x7fba701008a0 2024-03-31T23:14:11.589 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:11.587+0000 7fba75f80640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fba70111960 con 0x7fba701055f0 2024-03-31T23:14:11.589 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:11.587+0000 7fba75f80640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fba70111c60 con 0x7fba70101260 2024-03-31T23:14:11.589 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:11.587+0000 7fba6f7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fba701008a0 0x7fba70197ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:14:11.589 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:11.587+0000 7fba6f7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fba701008a0 0x7fba70197ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60148/0 (socket says 172.21.15.46:60148) 2024-03-31T23:14:11.589 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:11.587+0000 7fba6ffff640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fba701055f0 0x7fba7019c730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:14:11.589 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:11.587+0000 7fba6f7fe640 1 -- 172.21.15.46:0/550550485 learned_addr learned my addr 172.21.15.46:0/550550485 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:14:11.589 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:11.587+0000 7fba6effd640 1 --2- 172.21.15.46:0/550550485 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fba70101260 0x7fba70198400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:14:11.589 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:11.587+0000 7fba6f7fe640 1 -- 172.21.15.46:0/550550485 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fba70101260 msgr2=0x7fba70198400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:14:11.589 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:11.587+0000 7fba6f7fe640 1 --2- 172.21.15.46:0/550550485 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fba70101260 0x7fba70198400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:14:11.590 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:11.587+0000 7fba6f7fe640 1 -- 172.21.15.46:0/550550485 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fba701055f0 msgr2=0x7fba7019c730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:14:11.590 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:11.587+0000 7fba6f7fe640 1 --2- 172.21.15.46:0/550550485 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fba701055f0 0x7fba7019c730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:14:11.590 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:11.587+0000 7fba6f7fe640 1 -- 172.21.15.46:0/550550485 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fba7019ce60 con 0x7fba701008a0 2024-03-31T23:14:11.590 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:11.587+0000 7fba6f7fe640 1 --2- 172.21.15.46:0/550550485 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fba701008a0 0x7fba70197ec0 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7fba60006010 tx=0x7fba60005d90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:14:11.590 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:11.587+0000 7fba6cff9640 1 -- 172.21.15.46:0/550550485 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba60043020 con 0x7fba701008a0 2024-03-31T23:14:11.590 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:11.588+0000 7fba75f80640 1 -- 172.21.15.46:0/550550485 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fba7019d0f0 con 0x7fba701008a0 2024-03-31T23:14:11.590 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:11.588+0000 7fba6cff9640 1 -- 172.21.15.46:0/550550485 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fba60004350 con 0x7fba701008a0 2024-03-31T23:14:11.590 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:11.588+0000 7fba6cff9640 1 -- 172.21.15.46:0/550550485 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba60003a10 con 0x7fba701008a0 2024-03-31T23:14:11.590 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:11.588+0000 7fba75f80640 1 -- 172.21.15.46:0/550550485 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fba701a49e0 con 0x7fba701008a0 2024-03-31T23:14:11.598 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:11.590+0000 7fba6cff9640 1 -- 172.21.15.46:0/550550485 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 25) ==== 137451+0+0 (secure 0 0 0) 0x7fba60046020 con 0x7fba701008a0 2024-03-31T23:14:11.598 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:11.590+0000 7fba78c33740 1 Processor -- start 2024-03-31T23:14:11.598 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:11.590+0000 7fba78c33740 1 -- start start 2024-03-31T23:14:11.598 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:11.590+0000 7fba78c33740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f7d2be16c0 0x55f7d2be1aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:14:11.598 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:11.590+0000 7fba78c33740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f7d2be1fe0 0x55f7d2bea870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:14:11.598 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:11.590+0000 7fba78c33740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55f7d2beadb0 0x55f7d2bed1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:14:11.598 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:11.590+0000 7fba78c33740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55f7d2b448f0 con 0x55f7d2be16c0 2024-03-31T23:14:11.598 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:11.590+0000 7fba78c33740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55f7d2aed8d0 con 0x55f7d2beadb0 2024-03-31T23:14:11.598 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:11.590+0000 7fba78c33740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55f7d2b40990 con 0x55f7d2be1fe0 2024-03-31T23:14:11.598 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:11.590+0000 7fba6cff9640 1 --2- 172.21.15.46:0/550550485 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fba580a26b0 0x7fba580a4b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:14:11.599 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:11.591+0000 7fba6effd640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55f7d2beadb0 0x55f7d2bed1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:14:11.599 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:11.591+0000 7fba6effd640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55f7d2beadb0 0x55f7d2bed1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.46:58254/0 (socket says 172.21.15.46:58254) 2024-03-31T23:14:11.599 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:11.591+0000 7fba6f7fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f7d2be1fe0 0x55f7d2bea870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:14:11.599 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:11.591+0000 7fba6effd640 1 -- 172.21.15.46:0/437507257 learned_addr learned my addr 172.21.15.46:0/437507257 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:14:11.599 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:11.591+0000 7fba6ffff640 1 --2- 172.21.15.46:0/437507257 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f7d2be16c0 0x55f7d2be1aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:14:11.599 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:11.591+0000 7fba6ffff640 1 --2- 172.21.15.46:0/550550485 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fba580a26b0 0x7fba580a4b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:14:11.599 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:11.591+0000 7fba6cff9640 1 -- 172.21.15.46:0/550550485 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7fba600f0880 con 0x7fba701008a0 2024-03-31T23:14:11.599 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:11.591+0000 7fba6effd640 1 -- 172.21.15.46:0/437507257 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f7d2be1fe0 msgr2=0x55f7d2bea870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:14:11.599 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:11.591+0000 7fba6effd640 1 --2- 172.21.15.46:0/437507257 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f7d2be1fe0 0x55f7d2bea870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:14:11.599 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:11.591+0000 7fba6effd640 1 -- 172.21.15.46:0/437507257 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f7d2be16c0 msgr2=0x55f7d2be1aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:14:11.599 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:11.591+0000 7fba6effd640 1 --2- 172.21.15.46:0/437507257 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f7d2be16c0 0x55f7d2be1aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:14:11.599 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:11.591+0000 7fba6effd640 1 -- 172.21.15.46:0/437507257 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f7d2859210 con 0x55f7d2beadb0 2024-03-31T23:14:11.599 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:11.591+0000 7fba6effd640 1 --2- 172.21.15.46:0/437507257 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55f7d2beadb0 0x55f7d2bed1a0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fba64009c90 tx=0x7fba64011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2ed1c74ae1a1bd4 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:14:11.599 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:11.591+0000 7fba6ffff640 1 --2- 172.21.15.46:0/550550485 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fba580a26b0 0x7fba580a4b70 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fba68005c40 tx=0x7fba68009500 comp rx=0 tx=0).ready entity=mgr.15654 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:14:11.600 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:11.591+0000 7fba55ffb640 1 -- 172.21.15.46:0/437507257 <== mon.2 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba6401e070 con 0x55f7d2beadb0 2024-03-31T23:14:11.600 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:11.591+0000 7fba55ffb640 1 -- 172.21.15.46:0/437507257 <== mon.2 v2:172.21.15.142:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fba64019ae0 con 0x55f7d2beadb0 2024-03-31T23:14:11.600 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:11.591+0000 7fba55ffb640 1 -- 172.21.15.46:0/437507257 <== mon.2 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba6400e940 con 0x55f7d2beadb0 2024-03-31T23:14:11.600 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:11.592+0000 7fba78c33740 1 -- 172.21.15.46:0/437507257 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55f7d2beadb0 msgr2=0x55f7d2bed1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:14:11.600 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:11.592+0000 7fba78c33740 1 --2- 172.21.15.46:0/437507257 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55f7d2beadb0 0x55f7d2bed1a0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fba64009c90 tx=0x7fba64011810 comp rx=0 tx=0).stop 2024-03-31T23:14:11.600 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:11.592+0000 7fba78c33740 1 -- 172.21.15.46:0/437507257 shutdown_connections 2024-03-31T23:14:11.600 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:11.592+0000 7fba78c33740 1 --2- 172.21.15.46:0/437507257 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55f7d2beadb0 0x55f7d2bed1a0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:14:11.600 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:11.592+0000 7fba78c33740 1 --2- 172.21.15.46:0/437507257 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f7d2be1fe0 0x55f7d2bea870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:14:11.600 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:11.592+0000 7fba78c33740 1 --2- 172.21.15.46:0/437507257 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f7d2be16c0 0x55f7d2be1aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:14:11.600 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:11.592+0000 7fba78c33740 1 -- 172.21.15.46:0/437507257 >> 172.21.15.46:0/437507257 conn(0x55f7d2bdbfe0 msgr2=0x55f7d2bdc3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:14:11.600 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:11.592+0000 7fba78c33740 1 -- 172.21.15.46:0/437507257 shutdown_connections 2024-03-31T23:14:11.600 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:11.592+0000 7fba78c33740 1 -- 172.21.15.46:0/437507257 wait complete. 2024-03-31T23:14:11.600 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:11.593+0000 7fba78c33740 1 -- start start 2024-03-31T23:14:11.600 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:11.593+0000 7fba78c33740 1 Processor -- start 2024-03-31T23:14:11.601 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:11.593+0000 7fba78c33740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f7d2be16c0 0x55f7d2be7db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:14:11.601 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:11.593+0000 7fba78c33740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f7d2be1fe0 0x55f7d2be82f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:14:11.601 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:11.593+0000 7fba78c33740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55f7d2beadb0 0x55f7d2be8830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:14:11.601 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:11.593+0000 7fba78c33740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55f7d2bed9e0 con 0x55f7d2be1fe0 2024-03-31T23:14:11.601 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:11.593+0000 7fba78c33740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55f7d2ae2f80 con 0x55f7d2beadb0 2024-03-31T23:14:11.601 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:11.593+0000 7fba78c33740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55f7d2bedb60 con 0x55f7d2be16c0 2024-03-31T23:14:11.601 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:11.593+0000 7fba6f7fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f7d2be16c0 0x55f7d2be7db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:14:11.601 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:11.593+0000 7fba6f7fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f7d2be16c0 0x55f7d2be7db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.46:39568/0 (socket says 172.21.15.46:39568) 2024-03-31T23:14:11.601 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:11.593+0000 7fba6f7fe640 1 -- 172.21.15.46:0/3723074895 learned_addr learned my addr 172.21.15.46:0/3723074895 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:14:11.601 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:11.593+0000 7fba6effd640 1 --2- 172.21.15.46:0/3723074895 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f7d2be1fe0 0x55f7d2be82f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:14:11.601 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:11.593+0000 7fba78c33740 10 client.0 _ll_register_callbacks cb 0x55f7d2b3b290 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T23:14:11.601 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:11.593+0000 7fba6ffff640 1 --2- 172.21.15.46:0/3723074895 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55f7d2beadb0 0x55f7d2be8830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:14:11.601 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:11.593+0000 7fba6f7fe640 1 -- 172.21.15.46:0/3723074895 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55f7d2beadb0 msgr2=0x55f7d2be8830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:14:11.602 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:11.593+0000 7fba6f7fe640 1 --2- 172.21.15.46:0/3723074895 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55f7d2beadb0 0x55f7d2be8830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:14:11.602 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:11.593+0000 7fba6f7fe640 1 -- 172.21.15.46:0/3723074895 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f7d2be1fe0 msgr2=0x55f7d2be82f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:14:11.602 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:11.593+0000 7fba6f7fe640 1 --2- 172.21.15.46:0/3723074895 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f7d2be1fe0 0x55f7d2be82f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:14:11.602 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:11.593+0000 7fba6f7fe640 1 -- 172.21.15.46:0/3723074895 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f7d2925620 con 0x55f7d2be16c0 2024-03-31T23:14:11.602 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:11.594+0000 7fba6effd640 1 --2- 172.21.15.46:0/3723074895 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f7d2be1fe0 0x55f7d2be82f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:14:11.602 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:11.594+0000 7fba6f7fe640 1 --2- 172.21.15.46:0/3723074895 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f7d2be16c0 0x55f7d2be7db0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fba600edde0 tx=0x7fba6008bc90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=de89f7feedcca178 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:14:11.602 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:11.594+0000 7fba4effd640 10 client.0 ms_handle_connect on v2:172.21.15.162:3300/0 2024-03-31T23:14:11.602 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:11.594+0000 7fba4effd640 1 -- 172.21.15.46:0/3723074895 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba600bd460 con 0x55f7d2be16c0 2024-03-31T23:14:11.602 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:11.594+0000 7fba4effd640 1 -- 172.21.15.46:0/3723074895 <== mon.1 v2:172.21.15.162:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7fba600f0470 con 0x55f7d2be16c0 2024-03-31T23:14:11.602 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:11.594+0000 7fba78c33740 1 -- 172.21.15.46:0/3723074895 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fba64019ae0 con 0x55f7d2be16c0 2024-03-31T23:14:11.602 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:11.594+0000 7fba4effd640 1 -- 172.21.15.46:0/3723074895 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fba600da050 con 0x55f7d2be16c0 2024-03-31T23:14:11.602 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:11.594+0000 7fba78c33740 10 client.26086 fetch_fsmap learned FSMap version 28 2024-03-31T23:14:11.603 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:11.594+0000 7fba78c33740 1 -- 172.21.15.46:0/3723074895 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55f7d2bed9e0 con 0x55f7d2be16c0 2024-03-31T23:14:11.603 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:11.594+0000 7fba4effd640 1 -- 172.21.15.46:0/3723074895 <== mon.1 v2:172.21.15.162:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fba600c5770 con 0x55f7d2be16c0 2024-03-31T23:14:11.603 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:11.594+0000 7fba4effd640 1 -- 172.21.15.46:0/3723074895 <== mon.1 v2:172.21.15.162:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7fba600c52e0 con 0x55f7d2be16c0 2024-03-31T23:14:11.603 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:11.594+0000 7fba78c33740 10 client.26086 fetch_fsmap finished waiting for FSMap version 28 2024-03-31T23:14:11.603 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:11.594+0000 7fba78c33740 10 client.26086 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-31T23:14:11.603 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:11.595+0000 7fba78c33740 20 client.26086 populate_metadata read hostname 'smithi046' 2024-03-31T23:14:11.603 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:11.595+0000 7fba78c33740 1 --2- 172.21.15.46:0/3723074895 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55f7d2bfad70 0x55f7d2bfd1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:14:11.603 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:11.595+0000 7fba78c33740 4 client.26086 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-31T23:14:11.603 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:11.595+0000 7fba78c33740 1 -- 172.21.15.46:0/3723074895 --> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fba64019ae0 con 0x55f7d2bfad70 2024-03-31T23:14:11.603 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:11.595+0000 7fba6effd640 1 --2- 172.21.15.46:0/3723074895 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55f7d2bfad70 0x55f7d2bfd1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:14:11.603 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:11.595+0000 7fba6effd640 1 --2- 172.21.15.46:0/3723074895 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55f7d2bfad70 0x55f7d2bfd1d0 crc :-1 s=READY pgs=95 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2adbe3bde91abe2f server_cookie=22fb77e1cb09b2dd in_seq=0 out_seq=0 2024-03-31T23:14:11.603 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:11.595+0000 7fba4effd640 10 client.26086 ms_handle_connect on v2:172.21.15.142:6838/2519143918 2024-03-31T23:14:11.603 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:11.596+0000 7fba4effd640 1 -- 172.21.15.46:0/3723074895 <== mds.0 v2:172.21.15.142:6838/2519143918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x7fba64019ae0 con 0x55f7d2bfad70 2024-03-31T23:14:11.603 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:11.596+0000 7fba4effd640 10 client.26086 handle_command_reply: tid=0 2024-03-31T23:14:11.604 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:11.596+0000 7fba78c33740 1 client.26086 shutdown 2024-03-31T23:14:11.604 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:11.596+0000 7fba78c33740 1 -- 172.21.15.46:0/3723074895 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f7d2be16c0 msgr2=0x55f7d2be7db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:14:11.604 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:11.596+0000 7fba78c33740 1 --2- 172.21.15.46:0/3723074895 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f7d2be16c0 0x55f7d2be7db0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fba600edde0 tx=0x7fba6008bc90 comp rx=0 tx=0).stop 2024-03-31T23:14:11.604 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:11.596+0000 7fba78c33740 1 -- 172.21.15.46:0/3723074895 shutdown_connections 2024-03-31T23:14:11.604 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:11.596+0000 7fba78c33740 1 --2- 172.21.15.46:0/3723074895 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55f7d2bfad70 0x55f7d2bfd1d0 crc :-1 s=READY pgs=95 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:14:11.604 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:11.596+0000 7fba78c33740 1 --2- 172.21.15.46:0/3723074895 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55f7d2beadb0 0x55f7d2be8830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:14:11.604 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:11.596+0000 7fba78c33740 1 --2- 172.21.15.46:0/3723074895 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f7d2be1fe0 0x55f7d2be82f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:14:11.604 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:11.596+0000 7fba78c33740 1 --2- 172.21.15.46:0/3723074895 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f7d2be16c0 0x55f7d2be7db0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:14:11.604 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:11.596+0000 7fba4effd640 0 client.26086 ms_handle_reset on v2:172.21.15.142:6838/2519143918 2024-03-31T23:14:11.604 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:11.596+0000 7fba78c33740 1 -- 172.21.15.46:0/3723074895 >> 172.21.15.46:0/3723074895 conn(0x55f7d2bdbfe0 msgr2=0x55f7d2bddd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:14:11.604 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:11.596+0000 7fba78c33740 1 -- 172.21.15.46:0/3723074895 shutdown_connections 2024-03-31T23:14:11.604 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:11.596+0000 7fba78c33740 1 -- 172.21.15.46:0/3723074895 wait complete. 2024-03-31T23:14:11.604 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:11.596+0000 7fba78c33740 20 client.26086 trim_cache size 0 max 16384 2024-03-31T23:14:11.614 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:11.613+0000 7fba78c33740 1 Processor -- start 2024-03-31T23:14:11.615 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:11.613+0000 7fba78c33740 1 -- start start 2024-03-31T23:14:11.615 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:11.613+0000 7fba78c33740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f7d2be16c0 0x55f7d2bea4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:14:11.615 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:11.613+0000 7fba78c33740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f7d2be1fe0 0x55f7d2bf6780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:14:11.615 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:11.613+0000 7fba78c33740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55f7d2beadb0 0x55f7d2bddb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:14:11.615 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:11.614+0000 7fba78c33740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fba701b12f0 con 0x55f7d2be16c0 2024-03-31T23:14:11.615 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:11.614+0000 7fba78c33740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fba701aebc0 con 0x55f7d2beadb0 2024-03-31T23:14:11.615 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:11.614+0000 7fba78c33740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fba701aca10 con 0x55f7d2be1fe0 2024-03-31T23:14:11.615 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:11.614+0000 7fba6f7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f7d2be16c0 0x55f7d2bea4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:14:11.615 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:11.614+0000 7fba6effd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f7d2be1fe0 0x55f7d2bf6780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:14:11.615 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:11.614+0000 7fba6f7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f7d2be16c0 0x55f7d2bea4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60184/0 (socket says 172.21.15.46:60184) 2024-03-31T23:14:11.615 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:11.614+0000 7fba6ffff640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55f7d2beadb0 0x55f7d2bddb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:14:11.615 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:11.614+0000 7fba6f7fe640 1 -- 172.21.15.46:0/699861790 learned_addr learned my addr 172.21.15.46:0/699861790 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:14:11.615 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:11.614+0000 7fba6f7fe640 1 -- 172.21.15.46:0/699861790 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f7d2be1fe0 msgr2=0x55f7d2bf6780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:14:11.615 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:11.614+0000 7fba6f7fe640 1 --2- 172.21.15.46:0/699861790 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f7d2be1fe0 0x55f7d2bf6780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:14:11.616 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:11.614+0000 7fba6f7fe640 1 -- 172.21.15.46:0/699861790 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55f7d2beadb0 msgr2=0x55f7d2bddb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:14:11.616 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:11.614+0000 7fba6f7fe640 1 --2- 172.21.15.46:0/699861790 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55f7d2beadb0 0x55f7d2bddb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:14:11.616 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:11.614+0000 7fba6f7fe640 1 -- 172.21.15.46:0/699861790 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f7d2bed1a0 con 0x55f7d2be16c0 2024-03-31T23:14:11.616 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:11.614+0000 7fba6f7fe640 1 --2- 172.21.15.46:0/699861790 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f7d2be16c0 0x55f7d2bea4e0 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7fba60000c00 tx=0x7fba600b9000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=89cf4432e5d228a5 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:14:11.616 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:11.614+0000 7fba557fa640 1 -- 172.21.15.46:0/699861790 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba60087070 con 0x55f7d2be16c0 2024-03-31T23:14:11.616 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:11.614+0000 7fba557fa640 1 -- 172.21.15.46:0/699861790 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fba600bd460 con 0x55f7d2be16c0 2024-03-31T23:14:11.616 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:11.614+0000 7fba557fa640 1 -- 172.21.15.46:0/699861790 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba600bcd60 con 0x55f7d2be16c0 2024-03-31T23:14:11.619 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:11.615+0000 7fba78c33740 1 -- 172.21.15.46:0/699861790 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f7d2be16c0 msgr2=0x55f7d2bea4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:14:11.620 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:11.615+0000 7fba78c33740 1 --2- 172.21.15.46:0/699861790 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f7d2be16c0 0x55f7d2bea4e0 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7fba60000c00 tx=0x7fba600b9000 comp rx=0 tx=0).stop 2024-03-31T23:14:11.620 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:11.615+0000 7fba78c33740 1 -- 172.21.15.46:0/699861790 shutdown_connections 2024-03-31T23:14:11.620 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:11.615+0000 7fba78c33740 1 --2- 172.21.15.46:0/699861790 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55f7d2beadb0 0x55f7d2bddb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:14:11.620 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:11.615+0000 7fba78c33740 1 --2- 172.21.15.46:0/699861790 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f7d2be1fe0 0x55f7d2bf6780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:14:11.620 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:11.615+0000 7fba78c33740 1 --2- 172.21.15.46:0/699861790 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f7d2be16c0 0x55f7d2bea4e0 unknown :-1 s=CLOSED pgs=654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:14:11.620 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:11.615+0000 7fba78c33740 1 -- 172.21.15.46:0/699861790 >> 172.21.15.46:0/699861790 conn(0x55f7d2bdbfe0 msgr2=0x55f7d2bdd430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:14:11.620 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:11.615+0000 7fba78c33740 1 -- 172.21.15.46:0/699861790 shutdown_connections 2024-03-31T23:14:11.620 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:11.615+0000 7fba78c33740 1 -- 172.21.15.46:0/699861790 wait complete. 2024-03-31T23:14:11.620 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:11.615+0000 7fba78c33740 1 -- start start 2024-03-31T23:14:11.620 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:11.616+0000 7fba78c33740 1 Processor -- start 2024-03-31T23:14:11.620 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:11.616+0000 7fba78c33740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f7d2be16c0 0x55f7d2bf3210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:14:11.620 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:11.616+0000 7fba78c33740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55f7d2be1fe0 0x55f7d2bee580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:14:11.620 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:11.616+0000 7fba78c33740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f7d2beadb0 0x55f7d2beecc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:14:11.621 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:11.616+0000 7fba78c33740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fba701ad080 con 0x55f7d2beadb0 2024-03-31T23:14:11.621 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:11.616+0000 7fba78c33740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fba701ae4f0 con 0x55f7d2be1fe0 2024-03-31T23:14:11.621 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:11.616+0000 7fba78c33740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fba701ad750 con 0x55f7d2be16c0 2024-03-31T23:14:11.621 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:11.616+0000 7fba78c33740 10 client.0 _ll_register_callbacks cb 0x7fba60009560 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T23:14:11.621 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:11.616+0000 7fba6ffff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f7d2beadb0 0x55f7d2beecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:14:11.621 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:11.616+0000 7fba6f7fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f7d2be16c0 0x55f7d2bf3210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:14:11.621 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:11.616+0000 7fba6ffff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f7d2beadb0 0x55f7d2beecc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60200/0 (socket says 172.21.15.46:60200) 2024-03-31T23:14:11.621 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:11.616+0000 7fba6ffff640 1 -- 172.21.15.46:0/2127861782 learned_addr learned my addr 172.21.15.46:0/2127861782 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:14:11.621 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:11.616+0000 7fba6effd640 1 --2- 172.21.15.46:0/2127861782 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55f7d2be1fe0 0x55f7d2bee580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:14:11.621 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:11.616+0000 7fba6ffff640 1 -- 172.21.15.46:0/2127861782 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f7d2be16c0 msgr2=0x55f7d2bf3210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:14:11.621 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:11.616+0000 7fba6ffff640 1 --2- 172.21.15.46:0/2127861782 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f7d2be16c0 0x55f7d2bf3210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:14:11.621 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:11.616+0000 7fba6ffff640 1 -- 172.21.15.46:0/2127861782 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55f7d2be1fe0 msgr2=0x55f7d2bee580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:14:11.621 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:11.616+0000 7fba6ffff640 1 --2- 172.21.15.46:0/2127861782 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55f7d2be1fe0 0x55f7d2bee580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:14:11.622 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:11.617+0000 7fba6ffff640 1 -- 172.21.15.46:0/2127861782 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f7d2c600d0 con 0x55f7d2beadb0 2024-03-31T23:14:11.622 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:11.617+0000 7fba6f7fe640 1 --2- 172.21.15.46:0/2127861782 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f7d2be16c0 0x55f7d2bf3210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:14:11.622 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:11.617+0000 7fba6ffff640 1 --2- 172.21.15.46:0/2127861782 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f7d2beadb0 0x55f7d2beecc0 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7fba68003ec0 tx=0x7fba68004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5dc8a8904a441211 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:14:11.622 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:11.617+0000 7fba4effd640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-31T23:14:11.622 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:11.617+0000 7fba4effd640 1 -- 172.21.15.46:0/2127861782 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba6802f060 con 0x55f7d2beadb0 2024-03-31T23:14:11.622 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:11.617+0000 7fba4effd640 1 -- 172.21.15.46:0/2127861782 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7fba68020860 con 0x55f7d2beadb0 2024-03-31T23:14:11.622 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:11.617+0000 7fba78c33740 1 -- 172.21.15.46:0/2127861782 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fba600bd460 con 0x55f7d2beadb0 2024-03-31T23:14:11.622 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:11.617+0000 7fba4effd640 1 -- 172.21.15.46:0/2127861782 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fba68036050 con 0x55f7d2beadb0 2024-03-31T23:14:11.622 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:11.617+0000 7fba78c33740 10 client.16353 fetch_fsmap learned FSMap version 28 2024-03-31T23:14:11.622 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:11.617+0000 7fba78c33740 1 -- 172.21.15.46:0/2127861782 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55f7d2c612a0 con 0x55f7d2beadb0 2024-03-31T23:14:11.622 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:11.618+0000 7fba4effd640 1 -- 172.21.15.46:0/2127861782 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fba600bd460 con 0x55f7d2beadb0 2024-03-31T23:14:11.622 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:11.618+0000 7fba4effd640 1 -- 172.21.15.46:0/2127861782 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7fba6801d040 con 0x55f7d2beadb0 2024-03-31T23:14:11.622 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:11.618+0000 7fba78c33740 10 client.16353 fetch_fsmap finished waiting for FSMap version 28 2024-03-31T23:14:11.623 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:11.618+0000 7fba78c33740 10 client.16353 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-31T23:14:11.623 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:11.618+0000 7fba78c33740 20 client.16353 populate_metadata read hostname 'smithi046' 2024-03-31T23:14:11.623 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:11.618+0000 7fba78c33740 1 --2- 172.21.15.46:0/2127861782 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55f7d2c61a60 0x55f7d2c63e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:14:11.623 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:11.618+0000 7fba78c33740 4 client.16353 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-31T23:14:11.623 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:11.618+0000 7fba78c33740 1 -- 172.21.15.46:0/2127861782 --> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55f7d2b58120 con 0x55f7d2c61a60 2024-03-31T23:14:11.623 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:11.618+0000 7fba6f7fe640 1 --2- 172.21.15.46:0/2127861782 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55f7d2c61a60 0x55f7d2c63e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:14:11.623 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:11.618+0000 7fba4effd640 10 client.16353 ms_handle_connect on v2:172.21.15.142:6838/2519143918 2024-03-31T23:14:11.623 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:11.619+0000 7fba6f7fe640 1 --2- 172.21.15.46:0/2127861782 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55f7d2c61a60 0x55f7d2c63e80 crc :-1 s=READY pgs=96 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b659f1f4bcd3180f server_cookie=190d19aa71ec0622 in_seq=0 out_seq=0 2024-03-31T23:14:11.640 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:11.640+0000 7fba4effd640 1 -- 172.21.15.46:0/2127861782 <== mds.0 v2:172.21.15.142:6838/2519143918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55f7d2b58120 con 0x55f7d2c61a60 2024-03-31T23:14:11.640 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:11.640+0000 7fba4effd640 10 client.16353 handle_command_reply: tid=0 2024-03-31T23:14:11.640 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:11.640+0000 7fba78c33740 1 client.16353 shutdown 2024-03-31T23:14:11.641 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:11.640+0000 7fba78c33740 1 -- 172.21.15.46:0/2127861782 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f7d2beadb0 msgr2=0x55f7d2beecc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:14:11.641 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:11.640+0000 7fba78c33740 1 --2- 172.21.15.46:0/2127861782 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f7d2beadb0 0x55f7d2beecc0 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7fba68003ec0 tx=0x7fba68004a40 comp rx=0 tx=0).stop 2024-03-31T23:14:11.641 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:11.640+0000 7fba78c33740 1 -- 172.21.15.46:0/2127861782 shutdown_connections 2024-03-31T23:14:11.641 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:11.640+0000 7fba78c33740 1 --2- 172.21.15.46:0/2127861782 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55f7d2c61a60 0x55f7d2c63e80 crc :-1 s=READY pgs=96 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:14:11.641 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:11.640+0000 7fba78c33740 1 --2- 172.21.15.46:0/2127861782 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f7d2beadb0 0x55f7d2beecc0 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:14:11.641 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:11.640+0000 7fba4effd640 0 client.16353 ms_handle_reset on v2:172.21.15.142:6838/2519143918 2024-03-31T23:14:11.641 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:11.640+0000 7fba78c33740 1 --2- 172.21.15.46:0/2127861782 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55f7d2be1fe0 0x55f7d2bee580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:14:11.641 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:11.640+0000 7fba78c33740 1 --2- 172.21.15.46:0/2127861782 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f7d2be16c0 0x55f7d2bf3210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:14:11.641 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:11.640+0000 7fba78c33740 1 -- 172.21.15.46:0/2127861782 >> 172.21.15.46:0/2127861782 conn(0x55f7d2bdbfe0 msgr2=0x55f7d2bdaf80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:14:11.641 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:11.640+0000 7fba78c33740 1 -- 172.21.15.46:0/2127861782 shutdown_connections 2024-03-31T23:14:11.641 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:11.640+0000 7fba78c33740 1 -- 172.21.15.46:0/2127861782 wait complete. 2024-03-31T23:14:11.641 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:11.640+0000 7fba78c33740 20 client.16353 trim_cache size 0 max 16384 2024-03-31T23:14:11.642 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout:{ 2024-03-31T23:14:11.642 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout: "message": "", 2024-03-31T23:14:11.642 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout: "return_code": 0 2024-03-31T23:14:11.642 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout:} 2024-03-31T23:14:11.643 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:11.641+0000 7fba75f80640 1 -- 172.21.15.46:0/550550485 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fba580a26b0 msgr2=0x7fba580a4b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:14:11.643 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:11.641+0000 7fba75f80640 1 --2- 172.21.15.46:0/550550485 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fba580a26b0 0x7fba580a4b70 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fba68005c40 tx=0x7fba68009500 comp rx=0 tx=0).stop 2024-03-31T23:14:11.643 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:11.641+0000 7fba75f80640 1 -- 172.21.15.46:0/550550485 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fba701008a0 msgr2=0x7fba70197ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:14:11.643 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:11.641+0000 7fba75f80640 1 --2- 172.21.15.46:0/550550485 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fba701008a0 0x7fba70197ec0 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7fba60006010 tx=0x7fba60005d90 comp rx=0 tx=0).stop 2024-03-31T23:14:11.643 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:11.641+0000 7fba75f80640 1 -- 172.21.15.46:0/550550485 shutdown_connections 2024-03-31T23:14:11.643 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:11.641+0000 7fba75f80640 1 --2- 172.21.15.46:0/550550485 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fba580a26b0 0x7fba580a4b70 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:14:11.643 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:11.641+0000 7fba75f80640 1 --2- 172.21.15.46:0/550550485 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fba701055f0 0x7fba7019c730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:14:11.643 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:11.641+0000 7fba75f80640 1 --2- 172.21.15.46:0/550550485 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fba70101260 0x7fba70198400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:14:11.643 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:11.641+0000 7fba75f80640 1 --2- 172.21.15.46:0/550550485 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fba701008a0 0x7fba70197ec0 unknown :-1 s=CLOSED pgs=653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:14:11.644 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:11.641+0000 7fba75f80640 1 -- 172.21.15.46:0/550550485 >> 172.21.15.46:0/550550485 conn(0x7fba700774a0 msgr2=0x7fba700fe850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:14:11.644 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:11.641+0000 7fba75f80640 1 -- 172.21.15.46:0/550550485 shutdown_connections 2024-03-31T23:14:11.644 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:11.641+0000 7fba75f80640 1 -- 172.21.15.46:0/550550485 wait complete. 2024-03-31T23:14:12.192 INFO:tasks.workunit.client.0.smithi046.stdout: 1 34259 2.56 MB/sec warmup 78 sec latency 23.475 ms 2024-03-31T23:14:12.795 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:14:12 smithi162 ceph-mon[36237]: pgmap v77: 129 pgs: 129 active+clean; 43 MiB data, 579 MiB used, 1.0 TiB / 1.0 TiB avail; 301 KiB/s rd, 1.2 MiB/s wr, 132 op/s 2024-03-31T23:14:12.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:14:12 smithi046 ceph-mon[32498]: pgmap v77: 129 pgs: 129 active+clean; 43 MiB data, 579 MiB used, 1.0 TiB / 1.0 TiB avail; 301 KiB/s rd, 1.2 MiB/s wr, 132 op/s 2024-03-31T23:14:12.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:14:12 smithi142 ceph-mon[35271]: pgmap v77: 129 pgs: 129 active+clean; 43 MiB data, 579 MiB used, 1.0 TiB / 1.0 TiB avail; 301 KiB/s rd, 1.2 MiB/s wr, 132 op/s 2024-03-31T23:14:13.192 INFO:tasks.workunit.client.0.smithi046.stdout: 1 34733 2.58 MB/sec warmup 79 sec latency 16.625 ms 2024-03-31T23:14:14.192 INFO:tasks.workunit.client.0.smithi046.stdout: 1 35290 2.59 MB/sec warmup 80 sec latency 16.609 ms 2024-03-31T23:14:14.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:14:14 smithi162 ceph-mon[36237]: pgmap v78: 129 pgs: 129 active+clean; 43 MiB data, 579 MiB used, 1.0 TiB / 1.0 TiB avail; 301 KiB/s rd, 1.2 MiB/s wr, 132 op/s 2024-03-31T23:14:14.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:14:14 smithi046 ceph-mon[32498]: pgmap v78: 129 pgs: 129 active+clean; 43 MiB data, 579 MiB used, 1.0 TiB / 1.0 TiB avail; 301 KiB/s rd, 1.2 MiB/s wr, 132 op/s 2024-03-31T23:14:14.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:14:14 smithi142 ceph-mon[35271]: pgmap v78: 129 pgs: 129 active+clean; 43 MiB data, 579 MiB used, 1.0 TiB / 1.0 TiB avail; 301 KiB/s rd, 1.2 MiB/s wr, 132 op/s 2024-03-31T23:14:15.192 INFO:tasks.workunit.client.0.smithi046.stdout: 1 35745 2.62 MB/sec warmup 81 sec latency 24.110 ms 2024-03-31T23:14:16.192 INFO:tasks.workunit.client.0.smithi046.stdout: 1 35985 2.59 MB/sec warmup 82 sec latency 15.828 ms 2024-03-31T23:14:16.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:14:16 smithi162 ceph-mon[36237]: pgmap v79: 129 pgs: 129 active+clean; 44 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 446 KiB/s rd, 1.7 MiB/s wr, 194 op/s 2024-03-31T23:14:16.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:14:16 smithi046 ceph-mon[32498]: pgmap v79: 129 pgs: 129 active+clean; 44 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 446 KiB/s rd, 1.7 MiB/s wr, 194 op/s 2024-03-31T23:14:16.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:14:16 smithi142 ceph-mon[35271]: pgmap v79: 129 pgs: 129 active+clean; 44 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 446 KiB/s rd, 1.7 MiB/s wr, 194 op/s 2024-03-31T23:14:17.192 INFO:tasks.workunit.client.0.smithi046.stdout: 1 36372 2.56 MB/sec warmup 83 sec latency 15.416 ms 2024-03-31T23:14:18.192 INFO:tasks.workunit.client.0.smithi046.stdout: 1 36897 2.59 MB/sec warmup 84 sec latency 18.704 ms 2024-03-31T23:14:18.795 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:14:18 smithi162 ceph-mon[36237]: pgmap v80: 129 pgs: 129 active+clean; 44 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 279 KiB/s rd, 1.2 MiB/s wr, 118 op/s 2024-03-31T23:14:18.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:14:18 smithi046 ceph-mon[32498]: pgmap v80: 129 pgs: 129 active+clean; 44 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 279 KiB/s rd, 1.2 MiB/s wr, 118 op/s 2024-03-31T23:14:18.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:14:18 smithi142 ceph-mon[35271]: pgmap v80: 129 pgs: 129 active+clean; 44 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 279 KiB/s rd, 1.2 MiB/s wr, 118 op/s 2024-03-31T23:14:19.192 INFO:tasks.workunit.client.0.smithi046.stdout: 1 37476 2.56 MB/sec warmup 85 sec latency 22.052 ms 2024-03-31T23:14:20.193 INFO:tasks.workunit.client.0.smithi046.stdout: 1 37960 2.55 MB/sec warmup 86 sec latency 68.798 ms 2024-03-31T23:14:20.795 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:14:20 smithi162 ceph-mon[36237]: pgmap v81: 129 pgs: 129 active+clean; 44 MiB data, 633 MiB used, 1.0 TiB / 1.0 TiB avail; 431 KiB/s rd, 1.7 MiB/s wr, 178 op/s 2024-03-31T23:14:20.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:14:20 smithi046 ceph-mon[32498]: pgmap v81: 129 pgs: 129 active+clean; 44 MiB data, 633 MiB used, 1.0 TiB / 1.0 TiB avail; 431 KiB/s rd, 1.7 MiB/s wr, 178 op/s 2024-03-31T23:14:20.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:14:20 smithi142 ceph-mon[35271]: pgmap v81: 129 pgs: 129 active+clean; 44 MiB data, 633 MiB used, 1.0 TiB / 1.0 TiB avail; 431 KiB/s rd, 1.7 MiB/s wr, 178 op/s 2024-03-31T23:14:21.193 INFO:tasks.workunit.client.0.smithi046.stdout: 1 38709 2.59 MB/sec warmup 87 sec latency 15.911 ms 2024-03-31T23:14:22.193 INFO:tasks.workunit.client.0.smithi046.stdout: 1 39123 2.61 MB/sec warmup 88 sec latency 15.675 ms 2024-03-31T23:14:22.795 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:14:22 smithi162 ceph-mon[36237]: pgmap v82: 129 pgs: 129 active+clean; 44 MiB data, 633 MiB used, 1.0 TiB / 1.0 TiB avail; 297 KiB/s rd, 1.1 MiB/s wr, 121 op/s 2024-03-31T23:14:22.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:14:22 smithi046 ceph-mon[32498]: pgmap v82: 129 pgs: 129 active+clean; 44 MiB data, 633 MiB used, 1.0 TiB / 1.0 TiB avail; 297 KiB/s rd, 1.1 MiB/s wr, 121 op/s 2024-03-31T23:14:22.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:14:22 smithi142 ceph-mon[35271]: pgmap v82: 129 pgs: 129 active+clean; 44 MiB data, 633 MiB used, 1.0 TiB / 1.0 TiB avail; 297 KiB/s rd, 1.1 MiB/s wr, 121 op/s 2024-03-31T23:14:23.192 INFO:tasks.workunit.client.0.smithi046.stdout: 1 39439 2.60 MB/sec warmup 89 sec latency 15.829 ms 2024-03-31T23:14:24.193 INFO:tasks.workunit.client.0.smithi046.stdout: 1 39744 2.57 MB/sec warmup 90 sec latency 15.427 ms 2024-03-31T23:14:24.734 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:24.731+0000 7ff59b68f640 1 -- 172.21.15.46:0/3039942334 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff594069770 msgr2=0x7ff59410e5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:14:24.735 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:24.731+0000 7ff59b68f640 1 --2- 172.21.15.46:0/3039942334 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff594069770 0x7ff59410e5e0 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7ff58c009ca0 tx=0x7ff58c02c0f0 comp rx=0 tx=0).stop 2024-03-31T23:14:24.735 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:24.731+0000 7ff59b68f640 1 -- 172.21.15.46:0/3039942334 shutdown_connections 2024-03-31T23:14:24.735 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:24.731+0000 7ff59b68f640 1 --2- 172.21.15.46:0/3039942334 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff594069770 0x7ff59410e5e0 unknown :-1 s=CLOSED pgs=656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:14:24.735 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:24.731+0000 7ff59b68f640 1 --2- 172.21.15.46:0/3039942334 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff594068dd0 0x7ff594069230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:14:24.735 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:24.731+0000 7ff59b68f640 1 --2- 172.21.15.46:0/3039942334 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff594107840 0x7ff594107c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:14:24.735 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:24.731+0000 7ff59b68f640 1 -- 172.21.15.46:0/3039942334 >> 172.21.15.46:0/3039942334 conn(0x7ff5940774a0 msgr2=0x7ff5940778b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:14:24.735 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:24.732+0000 7ff59b68f640 1 -- 172.21.15.46:0/3039942334 shutdown_connections 2024-03-31T23:14:24.735 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:24.732+0000 7ff59b68f640 1 -- 172.21.15.46:0/3039942334 wait complete. 2024-03-31T23:14:24.735 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:24.732+0000 7ff59b68f640 1 Processor -- start 2024-03-31T23:14:24.735 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:24.732+0000 7ff59b68f640 1 -- start start 2024-03-31T23:14:24.735 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:24.732+0000 7ff59b68f640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff594068dd0 0x7ff594197e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:14:24.735 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:24.732+0000 7ff59b68f640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff594069770 0x7ff594198390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:14:24.736 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:24.732+0000 7ff59b68f640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff594107840 0x7ff59419c710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:14:24.736 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:24.732+0000 7ff59b68f640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7ff594111a70 con 0x7ff594069770 2024-03-31T23:14:24.736 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:24.732+0000 7ff59b68f640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7ff5941118f0 con 0x7ff594107840 2024-03-31T23:14:24.736 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:24.732+0000 7ff59b68f640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7ff594111bf0 con 0x7ff594068dd0 2024-03-31T23:14:24.736 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:24.733+0000 7ff598c2a640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff594069770 0x7ff594198390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:14:24.736 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:24.733+0000 7ff598c2a640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff594069770 0x7ff594198390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55826/0 (socket says 172.21.15.46:55826) 2024-03-31T23:14:24.736 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:24.733+0000 7ff598c2a640 1 -- 172.21.15.46:0/1402749264 learned_addr learned my addr 172.21.15.46:0/1402749264 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:14:24.736 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:24.733+0000 7ff599c2c640 1 --2- 172.21.15.46:0/1402749264 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff594107840 0x7ff59419c710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:14:24.736 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:24.733+0000 7ff59942b640 1 --2- 172.21.15.46:0/1402749264 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff594068dd0 0x7ff594197e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:14:24.736 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:24.733+0000 7ff598c2a640 1 -- 172.21.15.46:0/1402749264 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff594068dd0 msgr2=0x7ff594197e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:14:24.736 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:24.733+0000 7ff598c2a640 1 --2- 172.21.15.46:0/1402749264 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff594068dd0 0x7ff594197e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:14:24.736 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:24.733+0000 7ff598c2a640 1 -- 172.21.15.46:0/1402749264 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff594107840 msgr2=0x7ff59419c710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:14:24.736 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:24.733+0000 7ff598c2a640 1 --2- 172.21.15.46:0/1402749264 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff594107840 0x7ff59419c710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:14:24.737 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:24.733+0000 7ff598c2a640 1 -- 172.21.15.46:0/1402749264 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff59419ce40 con 0x7ff594069770 2024-03-31T23:14:24.737 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:24.733+0000 7ff598c2a640 1 --2- 172.21.15.46:0/1402749264 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff594069770 0x7ff594198390 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7ff58400e9f0 tx=0x7ff58400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:14:24.737 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:24.733+0000 7ff58a7fc640 1 -- 172.21.15.46:0/1402749264 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff58400cd80 con 0x7ff594069770 2024-03-31T23:14:24.737 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:24.733+0000 7ff59b68f640 1 -- 172.21.15.46:0/1402749264 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff59419d130 con 0x7ff594069770 2024-03-31T23:14:24.737 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:24.733+0000 7ff58a7fc640 1 -- 172.21.15.46:0/1402749264 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff5840108e0 con 0x7ff594069770 2024-03-31T23:14:24.737 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:24.733+0000 7ff58a7fc640 1 -- 172.21.15.46:0/1402749264 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff584018920 con 0x7ff594069770 2024-03-31T23:14:24.737 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:24.733+0000 7ff59b68f640 1 -- 172.21.15.46:0/1402749264 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff5941a49c0 con 0x7ff594069770 2024-03-31T23:14:24.742 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:24.735+0000 7ff58a7fc640 1 -- 172.21.15.46:0/1402749264 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 25) ==== 137451+0+0 (secure 0 0 0) 0x7ff584010a90 con 0x7ff594069770 2024-03-31T23:14:24.742 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:24.736+0000 7ff59e275740 1 Processor -- start 2024-03-31T23:14:24.742 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:24.736+0000 7ff59e275740 1 -- start start 2024-03-31T23:14:24.743 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:24.736+0000 7ff58a7fc640 1 --2- 172.21.15.46:0/1402749264 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7ff5800a26b0 0x7ff5800a4b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:14:24.743 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:24.736+0000 7ff59e275740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5567218196c0 0x556721819aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:14:24.743 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:24.736+0000 7ff59e275740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556721819fe0 0x556721822870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:14:24.743 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:24.736+0000 7ff59e275740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556721822db0 0x5567218251a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:14:24.743 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:24.736+0000 7ff59e275740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55672177c8f0 con 0x556721819fe0 2024-03-31T23:14:24.743 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:24.736+0000 7ff59e275740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x5567217258d0 con 0x5567218196c0 2024-03-31T23:14:24.743 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:24.736+0000 7ff59e275740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x556721778990 con 0x556721822db0 2024-03-31T23:14:24.743 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:24.736+0000 7ff599c2c640 1 --2- 172.21.15.46:0/1402749264 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7ff5800a26b0 0x7ff5800a4b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:14:24.743 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:24.736+0000 7ff58a7fc640 1 -- 172.21.15.46:0/1402749264 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7ff5840cf1d0 con 0x7ff594069770 2024-03-31T23:14:24.743 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:24.736+0000 7ff59942b640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5567218196c0 0x556721819aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:14:24.743 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:24.736+0000 7ff598c2a640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556721819fe0 0x556721822870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:14:24.743 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:24.736+0000 7ff59942b640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5567218196c0 0x556721819aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.46:47388/0 (socket says 172.21.15.46:47388) 2024-03-31T23:14:24.743 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:24.736+0000 7ff59942b640 1 -- 172.21.15.46:0/2642764942 learned_addr learned my addr 172.21.15.46:0/2642764942 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:14:24.743 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:24.737+0000 7ff599c2c640 1 --2- 172.21.15.46:0/2642764942 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556721822db0 0x5567218251a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:14:24.744 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:24.737+0000 7ff598c2a640 1 -- 172.21.15.46:0/2642764942 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556721822db0 msgr2=0x5567218251a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:14:24.744 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:24.737+0000 7ff598c2a640 1 --2- 172.21.15.46:0/2642764942 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556721822db0 0x5567218251a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:14:24.744 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:24.737+0000 7ff598c2a640 1 -- 172.21.15.46:0/2642764942 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5567218196c0 msgr2=0x556721819aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:14:24.744 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:24.737+0000 7ff598c2a640 1 --2- 172.21.15.46:0/2642764942 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5567218196c0 0x556721819aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:14:24.744 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:24.737+0000 7ff598c2a640 1 -- 172.21.15.46:0/2642764942 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556721491210 con 0x556721819fe0 2024-03-31T23:14:24.744 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:24.737+0000 7ff59942b640 1 --2- 172.21.15.46:0/2642764942 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5567218196c0 0x556721819aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:14:24.744 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:24.737+0000 7ff598c2a640 1 --2- 172.21.15.46:0/2642764942 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556721819fe0 0x556721822870 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7ff5840cc570 tx=0x7ff58407f330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=deed6133e95f2a30 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:14:24.744 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:24.737+0000 7ff599c2c640 1 --2- 172.21.15.46:0/1402749264 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7ff5800a26b0 0x7ff5800a4b70 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7ff58c004af0 tx=0x7ff58c009980 comp rx=0 tx=0).ready entity=mgr.15654 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:14:24.744 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:24.737+0000 7ff57f7fe640 1 -- 172.21.15.46:0/2642764942 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff58409b650 con 0x556721819fe0 2024-03-31T23:14:24.744 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:24.737+0000 7ff57f7fe640 1 -- 172.21.15.46:0/2642764942 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff58409f6f0 con 0x556721819fe0 2024-03-31T23:14:24.744 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:24.737+0000 7ff57f7fe640 1 -- 172.21.15.46:0/2642764942 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff58407f510 con 0x556721819fe0 2024-03-31T23:14:24.744 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:24.738+0000 7ff59e275740 1 -- 172.21.15.46:0/2642764942 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556721819fe0 msgr2=0x556721822870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:14:24.744 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:24.738+0000 7ff59e275740 1 --2- 172.21.15.46:0/2642764942 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556721819fe0 0x556721822870 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7ff5840cc570 tx=0x7ff58407f330 comp rx=0 tx=0).stop 2024-03-31T23:14:24.745 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:24.738+0000 7ff59e275740 1 -- 172.21.15.46:0/2642764942 shutdown_connections 2024-03-31T23:14:24.745 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:24.738+0000 7ff59e275740 1 --2- 172.21.15.46:0/2642764942 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556721822db0 0x5567218251a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:14:24.745 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:24.738+0000 7ff59e275740 1 --2- 172.21.15.46:0/2642764942 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556721819fe0 0x556721822870 unknown :-1 s=CLOSED pgs=658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:14:24.745 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:24.738+0000 7ff59e275740 1 --2- 172.21.15.46:0/2642764942 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5567218196c0 0x556721819aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:14:24.745 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:24.738+0000 7ff59e275740 1 -- 172.21.15.46:0/2642764942 >> 172.21.15.46:0/2642764942 conn(0x556721813fe0 msgr2=0x5567218143c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:14:24.745 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:24.738+0000 7ff59e275740 1 -- 172.21.15.46:0/2642764942 shutdown_connections 2024-03-31T23:14:24.745 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:24.738+0000 7ff59e275740 1 -- 172.21.15.46:0/2642764942 wait complete. 2024-03-31T23:14:24.745 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:24.738+0000 7ff59e275740 1 -- start start 2024-03-31T23:14:24.745 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:24.738+0000 7ff59e275740 1 Processor -- start 2024-03-31T23:14:24.745 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:24.739+0000 7ff59e275740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5567218196c0 0x55672181fe90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:14:24.745 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:24.739+0000 7ff59e275740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x556721819fe0 0x5567218203d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:14:24.745 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:24.739+0000 7ff59e275740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556721822db0 0x556721820910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:14:24.745 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:24.739+0000 7ff59e275740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x556721825980 con 0x556721822db0 2024-03-31T23:14:24.745 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:24.739+0000 7ff59e275740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55672171af80 con 0x556721819fe0 2024-03-31T23:14:24.746 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:24.739+0000 7ff59e275740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x556721825b00 con 0x5567218196c0 2024-03-31T23:14:24.746 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:24.739+0000 7ff59e275740 10 client.0 _ll_register_callbacks cb 0x556721773290 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T23:14:24.746 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:24.739+0000 7ff599c2c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556721822db0 0x556721820910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:14:24.746 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:24.739+0000 7ff599c2c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556721822db0 0x556721820910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55840/0 (socket says 172.21.15.46:55840) 2024-03-31T23:14:24.746 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:24.739+0000 7ff599c2c640 1 -- 172.21.15.46:0/373640156 learned_addr learned my addr 172.21.15.46:0/373640156 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:14:24.746 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:24.739+0000 7ff598c2a640 1 --2- 172.21.15.46:0/373640156 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x556721819fe0 0x5567218203d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:14:24.746 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:24.739+0000 7ff59942b640 1 --2- 172.21.15.46:0/373640156 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5567218196c0 0x55672181fe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:14:24.746 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:24.739+0000 7ff599c2c640 1 -- 172.21.15.46:0/373640156 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5567218196c0 msgr2=0x55672181fe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:14:24.746 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:24.739+0000 7ff599c2c640 1 --2- 172.21.15.46:0/373640156 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5567218196c0 0x55672181fe90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:14:24.746 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:24.739+0000 7ff599c2c640 1 -- 172.21.15.46:0/373640156 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x556721819fe0 msgr2=0x5567218203d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:14:24.746 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:24.739+0000 7ff599c2c640 1 --2- 172.21.15.46:0/373640156 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x556721819fe0 0x5567218203d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:14:24.746 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:24.739+0000 7ff599c2c640 1 -- 172.21.15.46:0/373640156 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55672155d620 con 0x556721822db0 2024-03-31T23:14:24.746 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:24.739+0000 7ff598c2a640 1 --2- 172.21.15.46:0/373640156 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x556721819fe0 0x5567218203d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:14:24.747 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:24.739+0000 7ff599c2c640 1 --2- 172.21.15.46:0/373640156 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556721822db0 0x556721820910 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7ff58c032490 tx=0x7ff58c02c0f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2cfe5abba9f4de26 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:14:24.747 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:24.739+0000 7ff57cff9640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-31T23:14:24.747 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:24.740+0000 7ff57cff9640 1 -- 172.21.15.46:0/373640156 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff58c039040 con 0x556721822db0 2024-03-31T23:14:24.747 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:24.740+0000 7ff57cff9640 1 -- 172.21.15.46:0/373640156 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7ff58c004450 con 0x556721822db0 2024-03-31T23:14:24.747 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:24.740+0000 7ff59e275740 1 -- 172.21.15.46:0/373640156 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff58409f6f0 con 0x556721822db0 2024-03-31T23:14:24.747 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:24.740+0000 7ff57cff9640 1 -- 172.21.15.46:0/373640156 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7ff58c059050 con 0x556721822db0 2024-03-31T23:14:24.747 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:24.740+0000 7ff59e275740 10 client.16365 fetch_fsmap learned FSMap version 28 2024-03-31T23:14:24.747 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:24.740+0000 7ff59e275740 1 -- 172.21.15.46:0/373640156 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x556721831910 con 0x556721822db0 2024-03-31T23:14:24.747 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:24.740+0000 7ff57cff9640 1 -- 172.21.15.46:0/373640156 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff58409f6f0 con 0x556721822db0 2024-03-31T23:14:24.747 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:24.740+0000 7ff57cff9640 1 -- 172.21.15.46:0/373640156 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7ff58c03f030 con 0x556721822db0 2024-03-31T23:14:24.747 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:24.740+0000 7ff59e275740 10 client.16365 fetch_fsmap finished waiting for FSMap version 28 2024-03-31T23:14:24.747 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:24.740+0000 7ff59e275740 10 client.16365 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-31T23:14:24.747 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:24.740+0000 7ff59e275740 20 client.16365 populate_metadata read hostname 'smithi046' 2024-03-31T23:14:24.748 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:24.740+0000 7ff59e275740 1 --2- 172.21.15.46:0/373640156 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x556721832e20 0x556721835230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:14:24.748 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:24.740+0000 7ff59e275740 4 client.16365 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-31T23:14:24.748 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:24.740+0000 7ff59e275740 1 -- 172.21.15.46:0/373640156 --> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556721790120 con 0x556721832e20 2024-03-31T23:14:24.748 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:24.740+0000 7ff59942b640 1 --2- 172.21.15.46:0/373640156 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x556721832e20 0x556721835230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:14:24.748 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:24.741+0000 7ff59942b640 1 --2- 172.21.15.46:0/373640156 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x556721832e20 0x556721835230 crc :-1 s=READY pgs=97 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8350b15e4018b169 server_cookie=f913f65f36dbe7bc in_seq=0 out_seq=0 2024-03-31T23:14:24.748 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:24.741+0000 7ff57cff9640 10 client.16365 ms_handle_connect on v2:172.21.15.142:6838/2519143918 2024-03-31T23:14:24.749 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:24.742+0000 7ff57cff9640 1 -- 172.21.15.46:0/373640156 <== mds.0 v2:172.21.15.142:6838/2519143918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x556721790120 con 0x556721832e20 2024-03-31T23:14:24.749 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:24.742+0000 7ff57cff9640 10 client.16365 handle_command_reply: tid=0 2024-03-31T23:14:24.749 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:24.742+0000 7ff59e275740 1 client.16365 shutdown 2024-03-31T23:14:24.749 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:24.742+0000 7ff59e275740 1 -- 172.21.15.46:0/373640156 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556721822db0 msgr2=0x556721820910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:14:24.749 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:24.742+0000 7ff59e275740 1 --2- 172.21.15.46:0/373640156 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556721822db0 0x556721820910 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7ff58c032490 tx=0x7ff58c02c0f0 comp rx=0 tx=0).stop 2024-03-31T23:14:24.749 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:24.742+0000 7ff59e275740 1 -- 172.21.15.46:0/373640156 shutdown_connections 2024-03-31T23:14:24.749 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:24.742+0000 7ff59e275740 1 --2- 172.21.15.46:0/373640156 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x556721832e20 0x556721835230 crc :-1 s=READY pgs=97 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:14:24.749 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:24.742+0000 7ff59e275740 1 --2- 172.21.15.46:0/373640156 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556721822db0 0x556721820910 unknown :-1 s=CLOSED pgs=659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:14:24.750 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:24.742+0000 7ff57cff9640 0 client.16365 ms_handle_reset on v2:172.21.15.142:6838/2519143918 2024-03-31T23:14:24.750 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:24.742+0000 7ff59e275740 1 --2- 172.21.15.46:0/373640156 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x556721819fe0 0x5567218203d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:14:24.750 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:24.742+0000 7ff59e275740 1 --2- 172.21.15.46:0/373640156 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5567218196c0 0x55672181fe90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:14:24.750 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:24.742+0000 7ff59e275740 1 -- 172.21.15.46:0/373640156 >> 172.21.15.46:0/373640156 conn(0x556721813fe0 msgr2=0x556721815d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:14:24.750 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:24.742+0000 7ff59e275740 1 -- 172.21.15.46:0/373640156 shutdown_connections 2024-03-31T23:14:24.750 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:24.742+0000 7ff59e275740 1 -- 172.21.15.46:0/373640156 wait complete. 2024-03-31T23:14:24.750 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:24.743+0000 7ff59e275740 20 client.16365 trim_cache size 0 max 16384 2024-03-31T23:14:24.761 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:24.759+0000 7ff59e275740 1 Processor -- start 2024-03-31T23:14:24.761 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:24.759+0000 7ff59e275740 1 -- start start 2024-03-31T23:14:24.761 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:24.759+0000 7ff59e275740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5567218196c0 0x5567218225c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:14:24.761 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:24.759+0000 7ff59e275740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556721819fe0 0x556721835210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:14:24.761 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:24.759+0000 7ff59e275740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556721822db0 0x55672182e720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:14:24.762 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:24.759+0000 7ff59e275740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7ff5941b1440 con 0x556721822db0 2024-03-31T23:14:24.762 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:24.759+0000 7ff59e275740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7ff5941aed10 con 0x5567218196c0 2024-03-31T23:14:24.762 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:24.759+0000 7ff59e275740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7ff5941acb60 con 0x556721819fe0 2024-03-31T23:14:24.762 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:24.759+0000 7ff599c2c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556721822db0 0x55672182e720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:14:24.762 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:24.759+0000 7ff599c2c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556721822db0 0x55672182e720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55842/0 (socket says 172.21.15.46:55842) 2024-03-31T23:14:24.762 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:24.759+0000 7ff599c2c640 1 -- 172.21.15.46:0/2036183513 learned_addr learned my addr 172.21.15.46:0/2036183513 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:14:24.762 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:24.759+0000 7ff59942b640 1 --2- 172.21.15.46:0/2036183513 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5567218196c0 0x5567218225c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:14:24.762 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:24.760+0000 7ff598c2a640 1 --2- 172.21.15.46:0/2036183513 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556721819fe0 0x556721835210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:14:24.762 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:24.760+0000 7ff599c2c640 1 -- 172.21.15.46:0/2036183513 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556721819fe0 msgr2=0x556721835210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:14:24.762 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:24.760+0000 7ff599c2c640 1 --2- 172.21.15.46:0/2036183513 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556721819fe0 0x556721835210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:14:24.762 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:24.760+0000 7ff599c2c640 1 -- 172.21.15.46:0/2036183513 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5567218196c0 msgr2=0x5567218225c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:14:24.762 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:24.760+0000 7ff599c2c640 1 --2- 172.21.15.46:0/2036183513 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5567218196c0 0x5567218225c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:14:24.762 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:24.760+0000 7ff599c2c640 1 -- 172.21.15.46:0/2036183513 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55672181ee40 con 0x556721822db0 2024-03-31T23:14:24.762 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:24.760+0000 7ff599c2c640 1 --2- 172.21.15.46:0/2036183513 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556721822db0 0x55672182e720 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7ff58c009a60 tx=0x7ff58c0093e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c3c5c742cc2c634 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:14:24.763 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:24.760+0000 7ff57effd640 1 -- 172.21.15.46:0/2036183513 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff58c039070 con 0x556721822db0 2024-03-31T23:14:24.763 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:24.760+0000 7ff57effd640 1 -- 172.21.15.46:0/2036183513 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff58c0348f0 con 0x556721822db0 2024-03-31T23:14:24.763 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:24.760+0000 7ff57effd640 1 -- 172.21.15.46:0/2036183513 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff58c0047e0 con 0x556721822db0 2024-03-31T23:14:24.766 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:24.761+0000 7ff59e275740 1 -- 172.21.15.46:0/2036183513 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556721822db0 msgr2=0x55672182e720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:14:24.766 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:24.761+0000 7ff59e275740 1 --2- 172.21.15.46:0/2036183513 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556721822db0 0x55672182e720 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7ff58c009a60 tx=0x7ff58c0093e0 comp rx=0 tx=0).stop 2024-03-31T23:14:24.766 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:24.761+0000 7ff59e275740 1 -- 172.21.15.46:0/2036183513 shutdown_connections 2024-03-31T23:14:24.766 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:24.761+0000 7ff59e275740 1 --2- 172.21.15.46:0/2036183513 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556721822db0 0x55672182e720 unknown :-1 s=CLOSED pgs=660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:14:24.766 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:24.761+0000 7ff59e275740 1 --2- 172.21.15.46:0/2036183513 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556721819fe0 0x556721835210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:14:24.766 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:24.761+0000 7ff59e275740 1 --2- 172.21.15.46:0/2036183513 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5567218196c0 0x5567218225c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:14:24.766 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:24.761+0000 7ff59e275740 1 -- 172.21.15.46:0/2036183513 >> 172.21.15.46:0/2036183513 conn(0x556721813fe0 msgr2=0x556721812f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:14:24.767 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:24.761+0000 7ff59e275740 1 -- 172.21.15.46:0/2036183513 shutdown_connections 2024-03-31T23:14:24.767 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:24.761+0000 7ff59e275740 1 -- 172.21.15.46:0/2036183513 wait complete. 2024-03-31T23:14:24.767 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:24.761+0000 7ff59e275740 1 -- start start 2024-03-31T23:14:24.767 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:24.761+0000 7ff59e275740 1 Processor -- start 2024-03-31T23:14:24.767 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:24.762+0000 7ff59e275740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5567218196c0 0x556721826a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:14:24.767 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:24.762+0000 7ff59e275740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556721819fe0 0x556721826f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:14:24.767 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:24.762+0000 7ff59e275740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556721822db0 0x55672189c580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:14:24.767 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:24.762+0000 7ff59e275740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7ff5941ad1d0 con 0x556721822db0 2024-03-31T23:14:24.767 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:24.762+0000 7ff59e275740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7ff5941ae640 con 0x5567218196c0 2024-03-31T23:14:24.767 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:24.762+0000 7ff59e275740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7ff5941ad8a0 con 0x556721819fe0 2024-03-31T23:14:24.767 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:24.762+0000 7ff59e275740 10 client.0 _ll_register_callbacks cb 0x7ff584005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T23:14:24.767 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:24.762+0000 7ff599c2c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556721822db0 0x55672189c580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:14:24.767 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:24.762+0000 7ff59942b640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5567218196c0 0x556721826a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:14:24.767 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:24.762+0000 7ff599c2c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556721822db0 0x55672189c580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55854/0 (socket says 172.21.15.46:55854) 2024-03-31T23:14:24.768 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:24.762+0000 7ff599c2c640 1 -- 172.21.15.46:0/2360819771 learned_addr learned my addr 172.21.15.46:0/2360819771 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:14:24.768 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:24.762+0000 7ff598c2a640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556721819fe0 0x556721826f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:14:24.768 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:24.762+0000 7ff599c2c640 1 -- 172.21.15.46:0/2360819771 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556721819fe0 msgr2=0x556721826f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:14:24.768 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:24.762+0000 7ff599c2c640 1 --2- 172.21.15.46:0/2360819771 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556721819fe0 0x556721826f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:14:24.768 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:24.762+0000 7ff599c2c640 1 -- 172.21.15.46:0/2360819771 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5567218196c0 msgr2=0x556721826a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:14:24.768 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:24.762+0000 7ff599c2c640 1 --2- 172.21.15.46:0/2360819771 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5567218196c0 0x556721826a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:14:24.768 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:24.762+0000 7ff599c2c640 1 -- 172.21.15.46:0/2360819771 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55672189cac0 con 0x556721822db0 2024-03-31T23:14:24.768 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:24.762+0000 7ff59942b640 1 --2- 172.21.15.46:0/2360819771 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5567218196c0 0x556721826a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:14:24.768 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:24.762+0000 7ff599c2c640 1 --2- 172.21.15.46:0/2360819771 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556721822db0 0x55672189c580 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7ff58c032490 tx=0x7ff58c02c770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a3d48ea1207052c7 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:14:24.768 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:24.762+0000 7ff57cff9640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-31T23:14:24.768 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:24.763+0000 7ff57cff9640 1 -- 172.21.15.46:0/2360819771 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff58c06b020 con 0x556721822db0 2024-03-31T23:14:24.768 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:24.763+0000 7ff57cff9640 1 -- 172.21.15.46:0/2360819771 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7ff58c034650 con 0x556721822db0 2024-03-31T23:14:24.768 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:24.763+0000 7ff59e275740 1 -- 172.21.15.46:0/2360819771 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff58c0348f0 con 0x556721822db0 2024-03-31T23:14:24.768 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:24.763+0000 7ff57cff9640 1 -- 172.21.15.46:0/2360819771 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7ff58c073050 con 0x556721822db0 2024-03-31T23:14:24.769 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:24.763+0000 7ff59e275740 10 client.16371 fetch_fsmap learned FSMap version 28 2024-03-31T23:14:24.769 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:24.763+0000 7ff59e275740 1 -- 172.21.15.46:0/2360819771 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55672189dc30 con 0x556721822db0 2024-03-31T23:14:24.769 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:24.763+0000 7ff57cff9640 1 -- 172.21.15.46:0/2360819771 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff58c0348f0 con 0x556721822db0 2024-03-31T23:14:24.769 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:24.763+0000 7ff57cff9640 1 -- 172.21.15.46:0/2360819771 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7ff58c066070 con 0x556721822db0 2024-03-31T23:14:24.769 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:24.763+0000 7ff59e275740 10 client.16371 fetch_fsmap finished waiting for FSMap version 28 2024-03-31T23:14:24.769 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:24.763+0000 7ff59e275740 10 client.16371 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-31T23:14:24.769 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:24.763+0000 7ff59e275740 20 client.16371 populate_metadata read hostname 'smithi046' 2024-03-31T23:14:24.769 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:24.763+0000 7ff59e275740 1 --2- 172.21.15.46:0/2360819771 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55672189e1e0 0x5567218a0600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:14:24.769 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:24.763+0000 7ff59e275740 4 client.16371 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-31T23:14:24.769 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:24.763+0000 7ff59e275740 1 -- 172.21.15.46:0/2360819771 --> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] -- command(tid 0: {"prefix": "flush journal"}) -- 0x556721789e30 con 0x55672189e1e0 2024-03-31T23:14:24.769 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:24.763+0000 7ff59942b640 1 --2- 172.21.15.46:0/2360819771 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55672189e1e0 0x5567218a0600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:14:24.769 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:24.764+0000 7ff59942b640 1 --2- 172.21.15.46:0/2360819771 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55672189e1e0 0x5567218a0600 crc :-1 s=READY pgs=98 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fb78482e3db87d06 server_cookie=c9df1b4705769715 in_seq=0 out_seq=0 2024-03-31T23:14:24.769 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:24.764+0000 7ff57cff9640 10 client.16371 ms_handle_connect on v2:172.21.15.142:6838/2519143918 2024-03-31T23:14:24.789 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:24.787+0000 7ff57cff9640 1 -- 172.21.15.46:0/2360819771 <== mds.0 v2:172.21.15.142:6838/2519143918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x556721789e30 con 0x55672189e1e0 2024-03-31T23:14:24.789 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:24.787+0000 7ff57cff9640 10 client.16371 handle_command_reply: tid=0 2024-03-31T23:14:24.789 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:24.787+0000 7ff59e275740 1 client.16371 shutdown 2024-03-31T23:14:24.789 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:24.787+0000 7ff59e275740 1 -- 172.21.15.46:0/2360819771 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556721822db0 msgr2=0x55672189c580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:14:24.789 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:24.787+0000 7ff59e275740 1 --2- 172.21.15.46:0/2360819771 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556721822db0 0x55672189c580 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7ff58c032490 tx=0x7ff58c02c770 comp rx=0 tx=0).stop 2024-03-31T23:14:24.789 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:24.787+0000 7ff59e275740 1 -- 172.21.15.46:0/2360819771 shutdown_connections 2024-03-31T23:14:24.789 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:24.787+0000 7ff59e275740 1 --2- 172.21.15.46:0/2360819771 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55672189e1e0 0x5567218a0600 crc :-1 s=READY pgs=98 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:14:24.789 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:24.787+0000 7ff59e275740 1 --2- 172.21.15.46:0/2360819771 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556721822db0 0x55672189c580 unknown :-1 s=CLOSED pgs=661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:14:24.789 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:24.787+0000 7ff59e275740 1 --2- 172.21.15.46:0/2360819771 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556721819fe0 0x556721826f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:14:24.789 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:24.787+0000 7ff59e275740 1 --2- 172.21.15.46:0/2360819771 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5567218196c0 0x556721826a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:14:24.789 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:24.787+0000 7ff57cff9640 0 client.16371 ms_handle_reset on v2:172.21.15.142:6838/2519143918 2024-03-31T23:14:24.789 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:24.787+0000 7ff59e275740 1 -- 172.21.15.46:0/2360819771 >> 172.21.15.46:0/2360819771 conn(0x556721813fe0 msgr2=0x556721812f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:14:24.790 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:24.787+0000 7ff59e275740 1 -- 172.21.15.46:0/2360819771 shutdown_connections 2024-03-31T23:14:24.790 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:24.787+0000 7ff59e275740 1 -- 172.21.15.46:0/2360819771 wait complete. 2024-03-31T23:14:24.790 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:24.788+0000 7ff59e275740 20 client.16371 trim_cache size 0 max 16384 2024-03-31T23:14:24.790 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout:{ 2024-03-31T23:14:24.790 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout: "message": "", 2024-03-31T23:14:24.790 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout: "return_code": 0 2024-03-31T23:14:24.790 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout:} 2024-03-31T23:14:24.791 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:24.788+0000 7ff59b68f640 1 -- 172.21.15.46:0/1402749264 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7ff5800a26b0 msgr2=0x7ff5800a4b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:14:24.791 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:24.788+0000 7ff59b68f640 1 --2- 172.21.15.46:0/1402749264 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7ff5800a26b0 0x7ff5800a4b70 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7ff58c004af0 tx=0x7ff58c009980 comp rx=0 tx=0).stop 2024-03-31T23:14:24.791 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:24.788+0000 7ff59b68f640 1 -- 172.21.15.46:0/1402749264 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff594069770 msgr2=0x7ff594198390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:14:24.791 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:24.788+0000 7ff59b68f640 1 --2- 172.21.15.46:0/1402749264 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff594069770 0x7ff594198390 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7ff58400e9f0 tx=0x7ff58400eeb0 comp rx=0 tx=0).stop 2024-03-31T23:14:24.791 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:24.788+0000 7ff59b68f640 1 -- 172.21.15.46:0/1402749264 shutdown_connections 2024-03-31T23:14:24.791 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:24.788+0000 7ff59b68f640 1 --2- 172.21.15.46:0/1402749264 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7ff5800a26b0 0x7ff5800a4b70 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:14:24.791 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:24.788+0000 7ff59b68f640 1 --2- 172.21.15.46:0/1402749264 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff594107840 0x7ff59419c710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:14:24.792 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:24.788+0000 7ff59b68f640 1 --2- 172.21.15.46:0/1402749264 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff594069770 0x7ff594198390 unknown :-1 s=CLOSED pgs=657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:14:24.792 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:24.788+0000 7ff59b68f640 1 --2- 172.21.15.46:0/1402749264 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff594068dd0 0x7ff594197e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:14:24.792 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:24.788+0000 7ff59b68f640 1 -- 172.21.15.46:0/1402749264 >> 172.21.15.46:0/1402749264 conn(0x7ff5940774a0 msgr2=0x7ff5940fe950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:14:24.792 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:24.788+0000 7ff59b68f640 1 -- 172.21.15.46:0/1402749264 shutdown_connections 2024-03-31T23:14:24.792 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:24.788+0000 7ff59b68f640 1 -- 172.21.15.46:0/1402749264 wait complete. 2024-03-31T23:14:24.795 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:14:24 smithi162 ceph-mon[36237]: pgmap v83: 129 pgs: 129 active+clean; 44 MiB data, 633 MiB used, 1.0 TiB / 1.0 TiB avail; 297 KiB/s rd, 1.1 MiB/s wr, 121 op/s 2024-03-31T23:14:24.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:14:24 smithi046 ceph-mon[32498]: pgmap v83: 129 pgs: 129 active+clean; 44 MiB data, 633 MiB used, 1.0 TiB / 1.0 TiB avail; 297 KiB/s rd, 1.1 MiB/s wr, 121 op/s 2024-03-31T23:14:24.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:14:24 smithi142 ceph-mon[35271]: pgmap v83: 129 pgs: 129 active+clean; 44 MiB data, 633 MiB used, 1.0 TiB / 1.0 TiB avail; 297 KiB/s rd, 1.1 MiB/s wr, 121 op/s 2024-03-31T23:14:25.193 INFO:tasks.workunit.client.0.smithi046.stdout: 1 40231 2.56 MB/sec warmup 91 sec latency 23.995 ms 2024-03-31T23:14:26.193 INFO:tasks.workunit.client.0.smithi046.stdout: 1 40779 2.57 MB/sec warmup 92 sec latency 19.576 ms 2024-03-31T23:14:26.795 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:14:26 smithi162 ceph-mon[36237]: pgmap v84: 129 pgs: 129 active+clean; 48 MiB data, 624 MiB used, 1.0 TiB / 1.0 TiB avail; 428 KiB/s rd, 1.7 MiB/s wr, 176 op/s 2024-03-31T23:14:26.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:14:26 smithi046 ceph-mon[32498]: pgmap v84: 129 pgs: 129 active+clean; 48 MiB data, 624 MiB used, 1.0 TiB / 1.0 TiB avail; 428 KiB/s rd, 1.7 MiB/s wr, 176 op/s 2024-03-31T23:14:26.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:14:26 smithi142 ceph-mon[35271]: pgmap v84: 129 pgs: 129 active+clean; 48 MiB data, 624 MiB used, 1.0 TiB / 1.0 TiB avail; 428 KiB/s rd, 1.7 MiB/s wr, 176 op/s 2024-03-31T23:14:27.193 INFO:tasks.workunit.client.0.smithi046.stdout: 1 41412 2.56 MB/sec warmup 93 sec latency 16.082 ms 2024-03-31T23:14:28.193 INFO:tasks.workunit.client.0.smithi046.stdout: 1 41983 2.59 MB/sec warmup 94 sec latency 51.851 ms 2024-03-31T23:14:28.795 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:14:28 smithi162 ceph-mon[36237]: pgmap v85: 129 pgs: 129 active+clean; 48 MiB data, 624 MiB used, 1.0 TiB / 1.0 TiB avail; 282 KiB/s rd, 1.1 MiB/s wr, 115 op/s 2024-03-31T23:14:28.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:14:28 smithi046 ceph-mon[32498]: pgmap v85: 129 pgs: 129 active+clean; 48 MiB data, 624 MiB used, 1.0 TiB / 1.0 TiB avail; 282 KiB/s rd, 1.1 MiB/s wr, 115 op/s 2024-03-31T23:14:28.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:14:28 smithi142 ceph-mon[35271]: pgmap v85: 129 pgs: 129 active+clean; 48 MiB data, 624 MiB used, 1.0 TiB / 1.0 TiB avail; 282 KiB/s rd, 1.1 MiB/s wr, 115 op/s 2024-03-31T23:14:29.193 INFO:tasks.workunit.client.0.smithi046.stdout: 1 42444 2.59 MB/sec warmup 95 sec latency 15.817 ms 2024-03-31T23:14:30.193 INFO:tasks.workunit.client.0.smithi046.stdout: 1 42909 2.61 MB/sec warmup 96 sec latency 14.705 ms 2024-03-31T23:14:30.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:14:30 smithi162 ceph-mon[36237]: pgmap v86: 129 pgs: 129 active+clean; 45 MiB data, 644 MiB used, 1.0 TiB / 1.0 TiB avail; 428 KiB/s rd, 1.7 MiB/s wr, 177 op/s 2024-03-31T23:14:30.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:14:30 smithi046 ceph-mon[32498]: pgmap v86: 129 pgs: 129 active+clean; 45 MiB data, 644 MiB used, 1.0 TiB / 1.0 TiB avail; 428 KiB/s rd, 1.7 MiB/s wr, 177 op/s 2024-03-31T23:14:30.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:14:30 smithi142 ceph-mon[35271]: pgmap v86: 129 pgs: 129 active+clean; 45 MiB data, 644 MiB used, 1.0 TiB / 1.0 TiB avail; 428 KiB/s rd, 1.7 MiB/s wr, 177 op/s 2024-03-31T23:14:31.193 INFO:tasks.workunit.client.0.smithi046.stdout: 1 43183 2.58 MB/sec warmup 97 sec latency 16.605 ms 2024-03-31T23:14:32.193 INFO:tasks.workunit.client.0.smithi046.stdout: 1 43647 2.58 MB/sec warmup 98 sec latency 15.481 ms 2024-03-31T23:14:32.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:14:32 smithi162 ceph-mon[36237]: pgmap v87: 129 pgs: 129 active+clean; 45 MiB data, 644 MiB used, 1.0 TiB / 1.0 TiB avail; 276 KiB/s rd, 1.2 MiB/s wr, 117 op/s 2024-03-31T23:14:32.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:14:32 smithi046 ceph-mon[32498]: pgmap v87: 129 pgs: 129 active+clean; 45 MiB data, 644 MiB used, 1.0 TiB / 1.0 TiB avail; 276 KiB/s rd, 1.2 MiB/s wr, 117 op/s 2024-03-31T23:14:32.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:14:32 smithi142 ceph-mon[35271]: pgmap v87: 129 pgs: 129 active+clean; 45 MiB data, 644 MiB used, 1.0 TiB / 1.0 TiB avail; 276 KiB/s rd, 1.2 MiB/s wr, 117 op/s 2024-03-31T23:14:33.193 INFO:tasks.workunit.client.0.smithi046.stdout: 1 44175 2.58 MB/sec warmup 99 sec latency 16.460 ms 2024-03-31T23:14:34.193 INFO:tasks.workunit.client.0.smithi046.stdout: 1 44780 2.56 MB/sec warmup 100 sec latency 16.206 ms 2024-03-31T23:14:34.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:14:34 smithi162 ceph-mon[36237]: pgmap v88: 129 pgs: 129 active+clean; 45 MiB data, 644 MiB used, 1.0 TiB / 1.0 TiB avail; 276 KiB/s rd, 1.2 MiB/s wr, 117 op/s 2024-03-31T23:14:34.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:14:34 smithi046 ceph-mon[32498]: pgmap v88: 129 pgs: 129 active+clean; 45 MiB data, 644 MiB used, 1.0 TiB / 1.0 TiB avail; 276 KiB/s rd, 1.2 MiB/s wr, 117 op/s 2024-03-31T23:14:34.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:14:34 smithi142 ceph-mon[35271]: pgmap v88: 129 pgs: 129 active+clean; 45 MiB data, 644 MiB used, 1.0 TiB / 1.0 TiB avail; 276 KiB/s rd, 1.2 MiB/s wr, 117 op/s 2024-03-31T23:14:35.193 INFO:tasks.workunit.client.0.smithi046.stdout: 1 45218 2.56 MB/sec warmup 101 sec latency 23.318 ms 2024-03-31T23:14:36.193 INFO:tasks.workunit.client.0.smithi046.stdout: 1 45912 2.60 MB/sec warmup 102 sec latency 16.834 ms 2024-03-31T23:14:36.795 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:14:36 smithi162 ceph-mon[36237]: pgmap v89: 129 pgs: 129 active+clean; 45 MiB data, 629 MiB used, 1.0 TiB / 1.0 TiB avail; 429 KiB/s rd, 1.7 MiB/s wr, 179 op/s 2024-03-31T23:14:36.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:14:36 smithi046 ceph-mon[32498]: pgmap v89: 129 pgs: 129 active+clean; 45 MiB data, 629 MiB used, 1.0 TiB / 1.0 TiB avail; 429 KiB/s rd, 1.7 MiB/s wr, 179 op/s 2024-03-31T23:14:36.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:14:36 smithi142 ceph-mon[35271]: pgmap v89: 129 pgs: 129 active+clean; 45 MiB data, 629 MiB used, 1.0 TiB / 1.0 TiB avail; 429 KiB/s rd, 1.7 MiB/s wr, 179 op/s 2024-03-31T23:14:37.194 INFO:tasks.workunit.client.0.smithi046.stdout: 1 46373 2.61 MB/sec warmup 103 sec latency 16.225 ms 2024-03-31T23:14:37.880 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:37.877+0000 7f746bb6c640 1 -- 172.21.15.46:0/2304891915 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7464068dd0 msgr2=0x7f7464069230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:14:37.880 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:37.877+0000 7f746bb6c640 1 --2- 172.21.15.46:0/2304891915 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7464068dd0 0x7f7464069230 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f7454009930 tx=0x7f745402bc30 comp rx=0 tx=0).stop 2024-03-31T23:14:37.881 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:37.877+0000 7f746bb6c640 1 -- 172.21.15.46:0/2304891915 shutdown_connections 2024-03-31T23:14:37.881 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:37.877+0000 7f746bb6c640 1 --2- 172.21.15.46:0/2304891915 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7464069770 0x7f746410e5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:14:37.881 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:37.877+0000 7f746bb6c640 1 --2- 172.21.15.46:0/2304891915 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7464068dd0 0x7f7464069230 unknown :-1 s=CLOSED pgs=662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:14:37.881 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:37.877+0000 7f746bb6c640 1 --2- 172.21.15.46:0/2304891915 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7464107820 0x7f7464107c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:14:37.881 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:37.877+0000 7f746bb6c640 1 -- 172.21.15.46:0/2304891915 >> 172.21.15.46:0/2304891915 conn(0x7f74640774a0 msgr2=0x7f74640778b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:14:37.881 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:37.877+0000 7f746bb6c640 1 -- 172.21.15.46:0/2304891915 shutdown_connections 2024-03-31T23:14:37.881 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:37.877+0000 7f746bb6c640 1 -- 172.21.15.46:0/2304891915 wait complete. 2024-03-31T23:14:37.881 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:37.878+0000 7f746bb6c640 1 Processor -- start 2024-03-31T23:14:37.881 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:37.878+0000 7f746bb6c640 1 -- start start 2024-03-31T23:14:37.881 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:37.878+0000 7f746bb6c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7464068dd0 0x7f7464197ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:14:37.881 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:37.878+0000 7f746bb6c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7464069770 0x7f74641983e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:14:37.881 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:37.878+0000 7f746bb6c640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7464107820 0x7f746419c710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:14:37.881 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:37.878+0000 7f746bb6c640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f74641119c0 con 0x7f7464069770 2024-03-31T23:14:37.881 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:37.878+0000 7f746bb6c640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f7464111840 con 0x7f7464107820 2024-03-31T23:14:37.882 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:37.878+0000 7f746bb6c640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f7464111b40 con 0x7f7464068dd0 2024-03-31T23:14:37.882 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:37.878+0000 7f7469107640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7464069770 0x7f74641983e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:14:37.882 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:37.878+0000 7f7469908640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7464068dd0 0x7f7464197ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:14:37.882 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:37.878+0000 7f7469908640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7464068dd0 0x7f7464197ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.46:47208/0 (socket says 172.21.15.46:47208) 2024-03-31T23:14:37.882 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:37.878+0000 7f7469908640 1 -- 172.21.15.46:0/3274204344 learned_addr learned my addr 172.21.15.46:0/3274204344 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:14:37.882 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:37.878+0000 7f7469107640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7464069770 0x7f74641983e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:38842/0 (socket says 172.21.15.46:38842) 2024-03-31T23:14:37.882 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:37.878+0000 7f746a109640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7464107820 0x7f746419c710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:14:37.882 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:37.879+0000 7f7469908640 1 -- 172.21.15.46:0/3274204344 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7464107820 msgr2=0x7f746419c710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:14:37.882 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:37.879+0000 7f7469908640 1 --2- 172.21.15.46:0/3274204344 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7464107820 0x7f746419c710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:14:37.882 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:37.879+0000 7f7469908640 1 -- 172.21.15.46:0/3274204344 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7464069770 msgr2=0x7f74641983e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:14:37.882 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:37.879+0000 7f7469908640 1 --2- 172.21.15.46:0/3274204344 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7464069770 0x7f74641983e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:14:37.882 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:37.879+0000 7f7469908640 1 -- 172.21.15.46:0/3274204344 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f746419ce40 con 0x7f7464068dd0 2024-03-31T23:14:37.882 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:37.879+0000 7f7469908640 1 --2- 172.21.15.46:0/3274204344 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7464068dd0 0x7f7464197ea0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f746000e9f0 tx=0x7f746000edf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:14:37.883 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:37.879+0000 7f745affd640 1 -- 172.21.15.46:0/3274204344 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f746000cd80 con 0x7f7464068dd0 2024-03-31T23:14:37.883 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:37.879+0000 7f745affd640 1 -- 172.21.15.46:0/3274204344 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f74600054d0 con 0x7f7464068dd0 2024-03-31T23:14:37.883 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:37.879+0000 7f746bb6c640 1 -- 172.21.15.46:0/3274204344 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f746419d130 con 0x7f7464068dd0 2024-03-31T23:14:37.883 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:37.879+0000 7f745affd640 1 -- 172.21.15.46:0/3274204344 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7460010490 con 0x7f7464068dd0 2024-03-31T23:14:37.883 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:37.879+0000 7f746bb6c640 1 -- 172.21.15.46:0/3274204344 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f74641a49c0 con 0x7f7464068dd0 2024-03-31T23:14:37.888 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:37.881+0000 7f745affd640 1 -- 172.21.15.46:0/3274204344 <== mon.1 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 25) ==== 137451+0+0 (secure 0 0 0) 0x7f7460005020 con 0x7f7464068dd0 2024-03-31T23:14:37.888 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:37.882+0000 7f746e82f740 1 Processor -- start 2024-03-31T23:14:37.888 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:37.882+0000 7f746e82f740 1 -- start start 2024-03-31T23:14:37.888 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:37.882+0000 7f745affd640 1 --2- 172.21.15.46:0/3274204344 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f74500a2730 0x7f74500a4bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:14:37.888 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:37.882+0000 7f746e82f740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x564b958691a0 0x564b95869580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:14:37.889 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:37.882+0000 7f746e82f740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564b95869ac0 0x564b95876c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:14:37.889 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:37.882+0000 7f746e82f740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564b958771c0 0x564b958795b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:14:37.889 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:37.882+0000 7f746e82f740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x564b957cc8f0 con 0x564b95869ac0 2024-03-31T23:14:37.889 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:37.882+0000 7f746e82f740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x564b957758d0 con 0x564b958691a0 2024-03-31T23:14:37.889 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:37.882+0000 7f746e82f740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x564b957c8990 con 0x564b958771c0 2024-03-31T23:14:37.889 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:37.882+0000 7f745affd640 1 -- 172.21.15.46:0/3274204344 <== mon.1 v2:172.21.15.162:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7f74600cf1b0 con 0x7f7464068dd0 2024-03-31T23:14:37.889 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:37.882+0000 7f746a109640 1 --2- 172.21.15.46:0/3274204344 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f74500a2730 0x7f74500a4bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:14:37.889 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:37.883+0000 7f7469107640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564b95869ac0 0x564b95876c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:14:37.889 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:37.883+0000 7f7469908640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x564b958691a0 0x564b95869580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:14:37.889 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:37.883+0000 7f7469107640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564b95869ac0 0x564b95876c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:38858/0 (socket says 172.21.15.46:38858) 2024-03-31T23:14:37.889 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:37.883+0000 7f7469107640 1 -- 172.21.15.46:0/67864534 learned_addr learned my addr 172.21.15.46:0/67864534 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:14:37.889 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:37.883+0000 7f7469908640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x564b958691a0 0x564b95869580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.46:37808/0 (socket says 172.21.15.46:37808) 2024-03-31T23:14:37.890 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:37.883+0000 7f7469908640 1 -- 172.21.15.46:0/67864534 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564b958771c0 msgr2=0x564b958795b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:14:37.890 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:37.883+0000 7f7469908640 1 --2- 172.21.15.46:0/67864534 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564b958771c0 0x564b958795b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:14:37.890 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:37.883+0000 7f7469908640 1 -- 172.21.15.46:0/67864534 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564b95869ac0 msgr2=0x564b95876c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:14:37.890 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:37.883+0000 7f7469908640 1 --2- 172.21.15.46:0/67864534 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564b95869ac0 0x564b95876c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:14:37.890 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:37.883+0000 7f7469908640 1 -- 172.21.15.46:0/67864534 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564b954e1210 con 0x564b958691a0 2024-03-31T23:14:37.890 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:37.883+0000 7f7469107640 1 --2- 172.21.15.46:0/67864534 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564b95869ac0 0x564b95876c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:14:37.890 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:37.883+0000 7f746a109640 1 --2- 172.21.15.46:0/3274204344 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f74500a2730 0x7f74500a4bf0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f745c005fd0 tx=0x7f745c011810 comp rx=0 tx=0).ready entity=mgr.15654 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:14:37.890 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:37.883+0000 7f7469908640 1 --2- 172.21.15.46:0/67864534 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x564b958691a0 0x564b95869580 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f74600cc550 tx=0x7f746009b0c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=354d5935fd03a8f server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:14:37.890 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:37.883+0000 7f744ffff640 1 -- 172.21.15.46:0/67864534 <== mon.2 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74600203a0 con 0x564b958691a0 2024-03-31T23:14:37.890 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:37.883+0000 7f744ffff640 1 -- 172.21.15.46:0/67864534 <== mon.2 v2:172.21.15.142:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7460005310 con 0x564b958691a0 2024-03-31T23:14:37.890 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:37.883+0000 7f744ffff640 1 -- 172.21.15.46:0/67864534 <== mon.2 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f746000f3c0 con 0x564b958691a0 2024-03-31T23:14:37.890 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:37.884+0000 7f746e82f740 1 -- 172.21.15.46:0/67864534 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x564b958691a0 msgr2=0x564b95869580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:14:37.890 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:37.884+0000 7f746e82f740 1 --2- 172.21.15.46:0/67864534 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x564b958691a0 0x564b95869580 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f74600cc550 tx=0x7f746009b0c0 comp rx=0 tx=0).stop 2024-03-31T23:14:37.890 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:37.884+0000 7f746e82f740 1 -- 172.21.15.46:0/67864534 shutdown_connections 2024-03-31T23:14:37.890 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:37.884+0000 7f746e82f740 1 --2- 172.21.15.46:0/67864534 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564b958771c0 0x564b958795b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:14:37.890 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:37.884+0000 7f746e82f740 1 --2- 172.21.15.46:0/67864534 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564b95869ac0 0x564b95876c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:14:37.891 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:37.884+0000 7f746e82f740 1 --2- 172.21.15.46:0/67864534 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x564b958691a0 0x564b95869580 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:14:37.891 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:37.884+0000 7f746e82f740 1 -- 172.21.15.46:0/67864534 >> 172.21.15.46:0/67864534 conn(0x564b95863fe0 msgr2=0x564b958643c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:14:37.891 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:37.884+0000 7f746e82f740 1 -- 172.21.15.46:0/67864534 shutdown_connections 2024-03-31T23:14:37.891 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:37.884+0000 7f746e82f740 1 -- 172.21.15.46:0/67864534 wait complete. 2024-03-31T23:14:37.891 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:37.884+0000 7f746e82f740 1 -- start start 2024-03-31T23:14:37.891 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:37.885+0000 7f746e82f740 1 Processor -- start 2024-03-31T23:14:37.891 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:37.885+0000 7f746e82f740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564b958691a0 0x564b958741c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:14:37.891 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:37.885+0000 7f746e82f740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x564b95869ac0 0x564b95874700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:14:37.891 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:37.885+0000 7f746e82f740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564b958771c0 0x564b95874c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:14:37.891 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:37.885+0000 7f746e82f740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x564b95879d90 con 0x564b958691a0 2024-03-31T23:14:37.891 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:37.885+0000 7f746e82f740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x564b9576af80 con 0x564b95869ac0 2024-03-31T23:14:37.891 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:37.885+0000 7f746e82f740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x564b95879f10 con 0x564b958771c0 2024-03-31T23:14:37.891 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:37.885+0000 7f7469908640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564b958691a0 0x564b958741c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:14:37.892 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:37.885+0000 7f746e82f740 10 client.0 _ll_register_callbacks cb 0x564b957c3290 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T23:14:37.892 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:37.885+0000 7f7469908640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564b958691a0 0x564b958741c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:38872/0 (socket says 172.21.15.46:38872) 2024-03-31T23:14:37.892 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:37.885+0000 7f7469908640 1 -- 172.21.15.46:0/3221679190 learned_addr learned my addr 172.21.15.46:0/3221679190 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:14:37.892 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:37.885+0000 7f7469107640 1 --2- 172.21.15.46:0/3221679190 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x564b95869ac0 0x564b95874700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:14:37.892 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:37.885+0000 7f746a109640 1 --2- 172.21.15.46:0/3221679190 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564b958771c0 0x564b95874c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:14:37.892 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:37.885+0000 7f7469908640 1 -- 172.21.15.46:0/3221679190 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564b958771c0 msgr2=0x564b95874c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:14:37.892 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:37.885+0000 7f7469908640 1 --2- 172.21.15.46:0/3221679190 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564b958771c0 0x564b95874c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:14:37.892 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:37.886+0000 7f7469908640 1 -- 172.21.15.46:0/3221679190 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x564b95869ac0 msgr2=0x564b95874700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:14:37.892 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:37.886+0000 7f7469908640 1 --2- 172.21.15.46:0/3221679190 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x564b95869ac0 0x564b95874700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:14:37.892 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:37.886+0000 7f7469908640 1 -- 172.21.15.46:0/3221679190 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564b955ad620 con 0x564b958691a0 2024-03-31T23:14:37.892 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:37.886+0000 7f7469908640 1 --2- 172.21.15.46:0/3221679190 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564b958691a0 0x564b958741c0 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7f74600097f0 tx=0x7f746000cf40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc5632988cd27e8d server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:14:37.892 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:37.886+0000 7f744d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-31T23:14:37.892 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:37.886+0000 7f744d7fa640 1 -- 172.21.15.46:0/3221679190 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74600203a0 con 0x564b958691a0 2024-03-31T23:14:37.892 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:37.886+0000 7f744d7fa640 1 -- 172.21.15.46:0/3221679190 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7f7460097bf0 con 0x564b958691a0 2024-03-31T23:14:37.892 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:37.886+0000 7f746e82f740 1 -- 172.21.15.46:0/3221679190 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x564b957e0120 con 0x564b958691a0 2024-03-31T23:14:37.893 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:37.886+0000 7f744d7fa640 1 -- 172.21.15.46:0/3221679190 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f746009bc80 con 0x564b958691a0 2024-03-31T23:14:37.893 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:37.886+0000 7f746e82f740 10 client.16383 fetch_fsmap learned FSMap version 28 2024-03-31T23:14:37.893 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:37.886+0000 7f746e82f740 1 -- 172.21.15.46:0/3221679190 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x564b95885ca0 con 0x564b958691a0 2024-03-31T23:14:37.893 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:37.886+0000 7f744d7fa640 1 -- 172.21.15.46:0/3221679190 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x564b957e0120 con 0x564b958691a0 2024-03-31T23:14:37.893 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:37.886+0000 7f744d7fa640 1 -- 172.21.15.46:0/3221679190 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f74600a3b00 con 0x564b958691a0 2024-03-31T23:14:37.893 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:37.887+0000 7f746e82f740 10 client.16383 fetch_fsmap finished waiting for FSMap version 28 2024-03-31T23:14:37.893 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:37.887+0000 7f746e82f740 10 client.16383 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-31T23:14:37.893 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:37.887+0000 7f746e82f740 20 client.16383 populate_metadata read hostname 'smithi046' 2024-03-31T23:14:37.893 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:37.887+0000 7f746e82f740 1 --2- 172.21.15.46:0/3221679190 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x564b95886ff0 0x564b958893e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:14:37.893 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:37.887+0000 7f746e82f740 4 client.16383 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-31T23:14:37.893 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:37.887+0000 7f746e82f740 1 -- 172.21.15.46:0/3221679190 --> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564b957d9e30 con 0x564b95886ff0 2024-03-31T23:14:37.893 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:37.887+0000 7f7469107640 1 --2- 172.21.15.46:0/3221679190 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x564b95886ff0 0x564b958893e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:14:37.893 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:37.887+0000 7f7469107640 1 --2- 172.21.15.46:0/3221679190 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x564b95886ff0 0x564b958893e0 crc :-1 s=READY pgs=99 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5672cf5cbd4de728 server_cookie=2052357869208123 in_seq=0 out_seq=0 2024-03-31T23:14:37.894 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:37.887+0000 7f744d7fa640 10 client.16383 ms_handle_connect on v2:172.21.15.142:6838/2519143918 2024-03-31T23:14:37.895 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:37.888+0000 7f744d7fa640 1 -- 172.21.15.46:0/3221679190 <== mds.0 v2:172.21.15.142:6838/2519143918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x564b957d9e30 con 0x564b95886ff0 2024-03-31T23:14:37.895 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:37.888+0000 7f744d7fa640 10 client.16383 handle_command_reply: tid=0 2024-03-31T23:14:37.895 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:37.888+0000 7f746e82f740 1 client.16383 shutdown 2024-03-31T23:14:37.895 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:37.888+0000 7f746e82f740 1 -- 172.21.15.46:0/3221679190 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564b958691a0 msgr2=0x564b958741c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:14:37.895 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:37.888+0000 7f746e82f740 1 --2- 172.21.15.46:0/3221679190 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564b958691a0 0x564b958741c0 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7f74600097f0 tx=0x7f746000cf40 comp rx=0 tx=0).stop 2024-03-31T23:14:37.895 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:37.888+0000 7f746e82f740 1 -- 172.21.15.46:0/3221679190 shutdown_connections 2024-03-31T23:14:37.895 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:37.888+0000 7f746e82f740 1 --2- 172.21.15.46:0/3221679190 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x564b95886ff0 0x564b958893e0 crc :-1 s=READY pgs=99 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:14:37.895 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:37.888+0000 7f746e82f740 1 --2- 172.21.15.46:0/3221679190 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564b958771c0 0x564b95874c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:14:37.895 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:37.888+0000 7f746e82f740 1 --2- 172.21.15.46:0/3221679190 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x564b95869ac0 0x564b95874700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:14:37.895 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:37.889+0000 7f744d7fa640 0 client.16383 ms_handle_reset on v2:172.21.15.142:6838/2519143918 2024-03-31T23:14:37.895 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:37.889+0000 7f746e82f740 1 --2- 172.21.15.46:0/3221679190 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564b958691a0 0x564b958741c0 unknown :-1 s=CLOSED pgs=663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:14:37.896 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:37.889+0000 7f746e82f740 1 -- 172.21.15.46:0/3221679190 >> 172.21.15.46:0/3221679190 conn(0x564b95863fe0 msgr2=0x564b95865d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:14:37.896 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:37.889+0000 7f746e82f740 1 -- 172.21.15.46:0/3221679190 shutdown_connections 2024-03-31T23:14:37.896 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:37.889+0000 7f746e82f740 1 -- 172.21.15.46:0/3221679190 wait complete. 2024-03-31T23:14:37.896 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:37.889+0000 7f746e82f740 20 client.16383 trim_cache size 0 max 16384 2024-03-31T23:14:37.907 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:37.905+0000 7f746e82f740 1 Processor -- start 2024-03-31T23:14:37.907 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:37.905+0000 7f746e82f740 1 -- start start 2024-03-31T23:14:37.908 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:37.906+0000 7f746e82f740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564b958691a0 0x564b95872a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:14:37.908 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:37.906+0000 7f746e82f740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x564b95869ac0 0x564b95872fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:14:37.908 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:37.906+0000 7f746e82f740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564b958771c0 0x564b95871420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:14:37.908 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:37.906+0000 7f746e82f740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f74641af1e0 con 0x564b958691a0 2024-03-31T23:14:37.908 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:37.906+0000 7f746e82f740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f74641acab0 con 0x564b95869ac0 2024-03-31T23:14:37.908 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:37.906+0000 7f746e82f740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f74641aa900 con 0x564b958771c0 2024-03-31T23:14:37.908 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:37.906+0000 7f7469908640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564b958691a0 0x564b95872a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:14:37.908 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:37.906+0000 7f7469908640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564b958691a0 0x564b95872a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:38880/0 (socket says 172.21.15.46:38880) 2024-03-31T23:14:37.908 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:37.906+0000 7f7469908640 1 -- 172.21.15.46:0/171254131 learned_addr learned my addr 172.21.15.46:0/171254131 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:14:37.908 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:37.906+0000 7f746a109640 1 --2- 172.21.15.46:0/171254131 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564b958771c0 0x564b95871420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:14:37.908 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:37.906+0000 7f7469107640 1 --2- 172.21.15.46:0/171254131 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x564b95869ac0 0x564b95872fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:14:37.908 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:37.906+0000 7f7469908640 1 -- 172.21.15.46:0/171254131 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564b958771c0 msgr2=0x564b95871420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:14:37.908 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:37.906+0000 7f7469908640 1 --2- 172.21.15.46:0/171254131 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564b958771c0 0x564b95871420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:14:37.908 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:37.906+0000 7f7469908640 1 -- 172.21.15.46:0/171254131 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x564b95869ac0 msgr2=0x564b95872fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:14:37.909 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:37.906+0000 7f7469908640 1 --2- 172.21.15.46:0/171254131 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x564b95869ac0 0x564b95872fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:14:37.909 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:37.906+0000 7f7469908640 1 -- 172.21.15.46:0/171254131 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564b95868f20 con 0x564b958691a0 2024-03-31T23:14:37.909 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:37.906+0000 7f7469908640 1 --2- 172.21.15.46:0/171254131 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564b958691a0 0x564b95872a90 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7f74600ccb50 tx=0x7f74600860b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f96675d305ecd9f3 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:14:37.909 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:37.906+0000 7f744f7fe640 1 -- 172.21.15.46:0/171254131 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74600ae070 con 0x564b958691a0 2024-03-31T23:14:37.909 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:37.906+0000 7f744f7fe640 1 -- 172.21.15.46:0/171254131 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f74600a3f90 con 0x564b958691a0 2024-03-31T23:14:37.909 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:37.906+0000 7f744f7fe640 1 -- 172.21.15.46:0/171254131 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74600a3b00 con 0x564b958691a0 2024-03-31T23:14:37.912 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:37.907+0000 7f746e82f740 1 -- 172.21.15.46:0/171254131 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564b958691a0 msgr2=0x564b95872a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:14:37.912 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:37.907+0000 7f746e82f740 1 --2- 172.21.15.46:0/171254131 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564b958691a0 0x564b95872a90 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7f74600ccb50 tx=0x7f74600860b0 comp rx=0 tx=0).stop 2024-03-31T23:14:37.912 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:37.907+0000 7f746e82f740 1 -- 172.21.15.46:0/171254131 shutdown_connections 2024-03-31T23:14:37.912 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:37.907+0000 7f746e82f740 1 --2- 172.21.15.46:0/171254131 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564b958771c0 0x564b95871420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:14:37.912 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:37.907+0000 7f746e82f740 1 --2- 172.21.15.46:0/171254131 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x564b95869ac0 0x564b95872fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:14:37.912 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:37.907+0000 7f746e82f740 1 --2- 172.21.15.46:0/171254131 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564b958691a0 0x564b95872a90 unknown :-1 s=CLOSED pgs=664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:14:37.913 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:37.907+0000 7f746e82f740 1 -- 172.21.15.46:0/171254131 >> 172.21.15.46:0/171254131 conn(0x564b95863fe0 msgr2=0x564b958643c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:14:37.913 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:37.907+0000 7f746e82f740 1 -- 172.21.15.46:0/171254131 shutdown_connections 2024-03-31T23:14:37.913 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:37.907+0000 7f746e82f740 1 -- 172.21.15.46:0/171254131 wait complete. 2024-03-31T23:14:37.913 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:37.907+0000 7f746e82f740 1 -- start start 2024-03-31T23:14:37.913 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:37.908+0000 7f746e82f740 1 Processor -- start 2024-03-31T23:14:37.913 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:37.908+0000 7f746e82f740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x564b958691a0 0x564b9588c380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:14:37.913 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:37.908+0000 7f746e82f740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564b95869ac0 0x564b95889710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:14:37.913 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:37.908+0000 7f746e82f740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564b958771c0 0x564b95889c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:14:37.913 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:37.908+0000 7f746e82f740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f74641aaf70 con 0x564b958771c0 2024-03-31T23:14:37.913 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:37.908+0000 7f746e82f740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f74641ac3e0 con 0x564b958691a0 2024-03-31T23:14:37.913 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:37.908+0000 7f746e82f740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f74641ab640 con 0x564b95869ac0 2024-03-31T23:14:37.913 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:37.908+0000 7f746e82f740 10 client.0 _ll_register_callbacks cb 0x7f7460005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T23:14:37.913 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:37.908+0000 7f746a109640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564b958771c0 0x564b95889c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:14:37.913 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:37.908+0000 7f746a109640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564b958771c0 0x564b95889c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:38886/0 (socket says 172.21.15.46:38886) 2024-03-31T23:14:37.914 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:37.908+0000 7f7469107640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564b95869ac0 0x564b95889710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:14:37.914 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:37.908+0000 7f746a109640 1 -- 172.21.15.46:0/3263803387 learned_addr learned my addr 172.21.15.46:0/3263803387 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:14:37.914 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:37.908+0000 7f7469908640 1 --2- 172.21.15.46:0/3263803387 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x564b958691a0 0x564b9588c380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:14:37.914 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:37.908+0000 7f746a109640 1 -- 172.21.15.46:0/3263803387 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564b95869ac0 msgr2=0x564b95889710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:14:37.914 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:37.908+0000 7f746a109640 1 --2- 172.21.15.46:0/3263803387 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564b95869ac0 0x564b95889710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:14:37.914 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:37.908+0000 7f746a109640 1 -- 172.21.15.46:0/3263803387 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x564b958691a0 msgr2=0x564b9588c380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:14:37.914 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:37.908+0000 7f746a109640 1 --2- 172.21.15.46:0/3263803387 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x564b958691a0 0x564b9588c380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:14:37.914 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:37.908+0000 7f746a109640 1 -- 172.21.15.46:0/3263803387 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564b95873f90 con 0x564b958771c0 2024-03-31T23:14:37.914 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:37.909+0000 7f746a109640 1 --2- 172.21.15.46:0/3263803387 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564b958771c0 0x564b95889c50 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f745c00fe50 tx=0x7f745c004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e85901e335b57515 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:14:37.914 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:37.909+0000 7f744d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-31T23:14:37.914 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:37.909+0000 7f744d7fa640 1 -- 172.21.15.46:0/3263803387 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f745c02d060 con 0x564b958771c0 2024-03-31T23:14:37.914 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:37.909+0000 7f744d7fa640 1 -- 172.21.15.46:0/3263803387 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7f745c009040 con 0x564b958771c0 2024-03-31T23:14:37.914 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:37.909+0000 7f746e82f740 1 -- 172.21.15.46:0/3263803387 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f74600a3f90 con 0x564b958771c0 2024-03-31T23:14:37.914 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:37.909+0000 7f744d7fa640 1 -- 172.21.15.46:0/3263803387 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f745c034050 con 0x564b958771c0 2024-03-31T23:14:37.915 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:37.909+0000 7f746e82f740 10 client.16389 fetch_fsmap learned FSMap version 28 2024-03-31T23:14:37.915 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:37.909+0000 7f746e82f740 1 -- 172.21.15.46:0/3263803387 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x564b958ecb00 con 0x564b958771c0 2024-03-31T23:14:37.915 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:37.909+0000 7f744d7fa640 1 -- 172.21.15.46:0/3263803387 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f74600a3f90 con 0x564b958771c0 2024-03-31T23:14:37.915 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:37.909+0000 7f744d7fa640 1 -- 172.21.15.46:0/3263803387 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f745c01e400 con 0x564b958771c0 2024-03-31T23:14:37.915 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:37.909+0000 7f746e82f740 10 client.16389 fetch_fsmap finished waiting for FSMap version 28 2024-03-31T23:14:37.915 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:37.909+0000 7f746e82f740 10 client.16389 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-31T23:14:37.915 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:37.909+0000 7f746e82f740 20 client.16389 populate_metadata read hostname 'smithi046' 2024-03-31T23:14:37.915 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:37.909+0000 7f746e82f740 1 --2- 172.21.15.46:0/3263803387 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x564b958ed310 0x564b958ef730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:14:37.915 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:37.910+0000 7f746e82f740 4 client.16389 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-31T23:14:37.915 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:37.910+0000 7f746e82f740 1 -- 172.21.15.46:0/3263803387 --> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] -- command(tid 0: {"prefix": "flush journal"}) -- 0x564b957d4180 con 0x564b958ed310 2024-03-31T23:14:37.915 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:37.910+0000 7f7469908640 1 --2- 172.21.15.46:0/3263803387 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x564b958ed310 0x564b958ef730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:14:37.915 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:37.910+0000 7f7469908640 1 --2- 172.21.15.46:0/3263803387 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x564b958ed310 0x564b958ef730 crc :-1 s=READY pgs=100 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dcc7253db9a2f0e0 server_cookie=5387a8d7109e2f6d in_seq=0 out_seq=0 2024-03-31T23:14:37.915 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:37.910+0000 7f744d7fa640 10 client.16389 ms_handle_connect on v2:172.21.15.142:6838/2519143918 2024-03-31T23:14:37.932 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:37.930+0000 7f744d7fa640 1 -- 172.21.15.46:0/3263803387 <== mds.0 v2:172.21.15.142:6838/2519143918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x564b957d4180 con 0x564b958ed310 2024-03-31T23:14:37.932 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:37.930+0000 7f744d7fa640 10 client.16389 handle_command_reply: tid=0 2024-03-31T23:14:37.932 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:37.930+0000 7f746e82f740 1 client.16389 shutdown 2024-03-31T23:14:37.932 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:37.930+0000 7f746e82f740 1 -- 172.21.15.46:0/3263803387 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564b958771c0 msgr2=0x564b95889c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:14:37.932 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:37.930+0000 7f746e82f740 1 --2- 172.21.15.46:0/3263803387 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564b958771c0 0x564b95889c50 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f745c00fe50 tx=0x7f745c004ab0 comp rx=0 tx=0).stop 2024-03-31T23:14:37.932 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:37.930+0000 7f746e82f740 1 -- 172.21.15.46:0/3263803387 shutdown_connections 2024-03-31T23:14:37.932 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:37.930+0000 7f746e82f740 1 --2- 172.21.15.46:0/3263803387 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x564b958ed310 0x564b958ef730 crc :-1 s=READY pgs=100 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:14:37.932 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:37.930+0000 7f746e82f740 1 --2- 172.21.15.46:0/3263803387 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564b958771c0 0x564b95889c50 unknown :-1 s=CLOSED pgs=665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:14:37.932 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:37.930+0000 7f744d7fa640 0 client.16389 ms_handle_reset on v2:172.21.15.142:6838/2519143918 2024-03-31T23:14:37.932 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:37.930+0000 7f746e82f740 1 --2- 172.21.15.46:0/3263803387 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564b95869ac0 0x564b95889710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:14:37.932 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:37.930+0000 7f746e82f740 1 --2- 172.21.15.46:0/3263803387 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x564b958691a0 0x564b9588c380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:14:37.933 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:37.930+0000 7f746e82f740 1 -- 172.21.15.46:0/3263803387 >> 172.21.15.46:0/3263803387 conn(0x564b95863fe0 msgr2=0x564b95872920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:14:37.933 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:37.931+0000 7f746e82f740 1 -- 172.21.15.46:0/3263803387 shutdown_connections 2024-03-31T23:14:37.933 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:37.931+0000 7f746e82f740 1 -- 172.21.15.46:0/3263803387 wait complete. 2024-03-31T23:14:37.933 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:37.931+0000 7f746e82f740 20 client.16389 trim_cache size 0 max 16384 2024-03-31T23:14:37.933 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout:{ 2024-03-31T23:14:37.933 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout: "message": "", 2024-03-31T23:14:37.933 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout: "return_code": 0 2024-03-31T23:14:37.933 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout:} 2024-03-31T23:14:37.934 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:37.931+0000 7f746bb6c640 1 -- 172.21.15.46:0/3274204344 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f74500a2730 msgr2=0x7f74500a4bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:14:37.934 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:37.931+0000 7f746bb6c640 1 --2- 172.21.15.46:0/3274204344 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f74500a2730 0x7f74500a4bf0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f745c005fd0 tx=0x7f745c011810 comp rx=0 tx=0).stop 2024-03-31T23:14:37.934 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:37.931+0000 7f746bb6c640 1 -- 172.21.15.46:0/3274204344 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7464068dd0 msgr2=0x7f7464197ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:14:37.934 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:37.931+0000 7f746bb6c640 1 --2- 172.21.15.46:0/3274204344 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7464068dd0 0x7f7464197ea0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f746000e9f0 tx=0x7f746000edf0 comp rx=0 tx=0).stop 2024-03-31T23:14:37.934 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:37.931+0000 7f746bb6c640 1 -- 172.21.15.46:0/3274204344 shutdown_connections 2024-03-31T23:14:37.934 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:37.931+0000 7f746bb6c640 1 --2- 172.21.15.46:0/3274204344 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f74500a2730 0x7f74500a4bf0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:14:37.934 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:37.931+0000 7f746bb6c640 1 --2- 172.21.15.46:0/3274204344 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7464107820 0x7f746419c710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:14:37.935 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:37.931+0000 7f746bb6c640 1 --2- 172.21.15.46:0/3274204344 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7464069770 0x7f74641983e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:14:37.935 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:37.931+0000 7f746bb6c640 1 --2- 172.21.15.46:0/3274204344 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7464068dd0 0x7f7464197ea0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:14:37.935 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:37.931+0000 7f746bb6c640 1 -- 172.21.15.46:0/3274204344 >> 172.21.15.46:0/3274204344 conn(0x7f74640774a0 msgr2=0x7f74640fe990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:14:37.935 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:37.931+0000 7f746bb6c640 1 -- 172.21.15.46:0/3274204344 shutdown_connections 2024-03-31T23:14:37.935 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:37.931+0000 7f746bb6c640 1 -- 172.21.15.46:0/3274204344 wait complete. 2024-03-31T23:14:38.193 INFO:tasks.workunit.client.0.smithi046.stdout: 1 46615 2.59 MB/sec warmup 104 sec latency 14.754 ms 2024-03-31T23:14:38.795 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:14:38 smithi162 ceph-mon[36237]: pgmap v90: 129 pgs: 129 active+clean; 45 MiB data, 629 MiB used, 1.0 TiB / 1.0 TiB avail; 298 KiB/s rd, 1.1 MiB/s wr, 124 op/s 2024-03-31T23:14:38.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:14:38 smithi046 ceph-mon[32498]: pgmap v90: 129 pgs: 129 active+clean; 45 MiB data, 629 MiB used, 1.0 TiB / 1.0 TiB avail; 298 KiB/s rd, 1.1 MiB/s wr, 124 op/s 2024-03-31T23:14:38.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:14:38 smithi142 ceph-mon[35271]: pgmap v90: 129 pgs: 129 active+clean; 45 MiB data, 629 MiB used, 1.0 TiB / 1.0 TiB avail; 298 KiB/s rd, 1.1 MiB/s wr, 124 op/s 2024-03-31T23:14:39.193 INFO:tasks.workunit.client.0.smithi046.stdout: 1 47011 2.57 MB/sec warmup 105 sec latency 15.911 ms 2024-03-31T23:14:40.193 INFO:tasks.workunit.client.0.smithi046.stdout: 1 47539 2.59 MB/sec warmup 106 sec latency 19.763 ms 2024-03-31T23:14:40.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:14:40 smithi162 ceph-mon[36237]: pgmap v91: 129 pgs: 129 active+clean; 49 MiB data, 632 MiB used, 1.0 TiB / 1.0 TiB avail; 442 KiB/s rd, 1.7 MiB/s wr, 184 op/s 2024-03-31T23:14:40.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:14:40 smithi046 ceph-mon[32498]: pgmap v91: 129 pgs: 129 active+clean; 49 MiB data, 632 MiB used, 1.0 TiB / 1.0 TiB avail; 442 KiB/s rd, 1.7 MiB/s wr, 184 op/s 2024-03-31T23:14:40.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:14:40 smithi142 ceph-mon[35271]: pgmap v91: 129 pgs: 129 active+clean; 49 MiB data, 632 MiB used, 1.0 TiB / 1.0 TiB avail; 442 KiB/s rd, 1.7 MiB/s wr, 184 op/s 2024-03-31T23:14:41.193 INFO:tasks.workunit.client.0.smithi046.stdout: 1 48073 2.57 MB/sec warmup 107 sec latency 23.830 ms 2024-03-31T23:14:42.193 INFO:tasks.workunit.client.0.smithi046.stdout: 1 48602 2.56 MB/sec warmup 108 sec latency 16.216 ms 2024-03-31T23:14:42.795 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:14:42 smithi162 ceph-mon[36237]: pgmap v92: 129 pgs: 129 active+clean; 49 MiB data, 632 MiB used, 1.0 TiB / 1.0 TiB avail; 296 KiB/s rd, 1.1 MiB/s wr, 121 op/s 2024-03-31T23:14:42.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:14:42 smithi046 ceph-mon[32498]: pgmap v92: 129 pgs: 129 active+clean; 49 MiB data, 632 MiB used, 1.0 TiB / 1.0 TiB avail; 296 KiB/s rd, 1.1 MiB/s wr, 121 op/s 2024-03-31T23:14:42.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:14:42 smithi142 ceph-mon[35271]: pgmap v92: 129 pgs: 129 active+clean; 49 MiB data, 632 MiB used, 1.0 TiB / 1.0 TiB avail; 296 KiB/s rd, 1.1 MiB/s wr, 121 op/s 2024-03-31T23:14:43.193 INFO:tasks.workunit.client.0.smithi046.stdout: 1 49342 2.60 MB/sec warmup 109 sec latency 16.295 ms 2024-03-31T23:14:44.193 INFO:tasks.workunit.client.0.smithi046.stdout: 1 49758 2.61 MB/sec warmup 110 sec latency 15.631 ms 2024-03-31T23:14:44.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:14:44 smithi162 ceph-mon[36237]: pgmap v93: 129 pgs: 129 active+clean; 49 MiB data, 632 MiB used, 1.0 TiB / 1.0 TiB avail; 296 KiB/s rd, 1.1 MiB/s wr, 121 op/s 2024-03-31T23:14:44.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:14:44 smithi046 ceph-mon[32498]: pgmap v93: 129 pgs: 129 active+clean; 49 MiB data, 632 MiB used, 1.0 TiB / 1.0 TiB avail; 296 KiB/s rd, 1.1 MiB/s wr, 121 op/s 2024-03-31T23:14:44.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:14:44 smithi142 ceph-mon[35271]: pgmap v93: 129 pgs: 129 active+clean; 49 MiB data, 632 MiB used, 1.0 TiB / 1.0 TiB avail; 296 KiB/s rd, 1.1 MiB/s wr, 121 op/s 2024-03-31T23:14:45.193 INFO:tasks.workunit.client.0.smithi046.stdout: 1 50077 2.60 MB/sec warmup 111 sec latency 15.160 ms 2024-03-31T23:14:46.193 INFO:tasks.workunit.client.0.smithi046.stdout: 1 50383 2.58 MB/sec warmup 112 sec latency 16.882 ms 2024-03-31T23:14:46.795 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:14:46 smithi162 ceph-mon[36237]: pgmap v94: 129 pgs: 129 active+clean; 50 MiB data, 632 MiB used, 1.0 TiB / 1.0 TiB avail; 429 KiB/s rd, 1.7 MiB/s wr, 177 op/s 2024-03-31T23:14:46.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:14:46 smithi046 ceph-mon[32498]: pgmap v94: 129 pgs: 129 active+clean; 50 MiB data, 632 MiB used, 1.0 TiB / 1.0 TiB avail; 429 KiB/s rd, 1.7 MiB/s wr, 177 op/s 2024-03-31T23:14:46.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:14:46 smithi142 ceph-mon[35271]: pgmap v94: 129 pgs: 129 active+clean; 50 MiB data, 632 MiB used, 1.0 TiB / 1.0 TiB avail; 429 KiB/s rd, 1.7 MiB/s wr, 177 op/s 2024-03-31T23:14:47.193 INFO:tasks.workunit.client.0.smithi046.stdout: 1 50866 2.57 MB/sec warmup 113 sec latency 18.084 ms 2024-03-31T23:14:48.193 INFO:tasks.workunit.client.0.smithi046.stdout: 1 51460 2.58 MB/sec warmup 114 sec latency 19.944 ms 2024-03-31T23:14:48.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:14:48 smithi162 ceph-mon[36237]: pgmap v95: 129 pgs: 129 active+clean; 50 MiB data, 632 MiB used, 1.0 TiB / 1.0 TiB avail; 276 KiB/s rd, 1.2 MiB/s wr, 115 op/s 2024-03-31T23:14:48.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:14:48 smithi046 ceph-mon[32498]: pgmap v95: 129 pgs: 129 active+clean; 50 MiB data, 632 MiB used, 1.0 TiB / 1.0 TiB avail; 276 KiB/s rd, 1.2 MiB/s wr, 115 op/s 2024-03-31T23:14:48.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:14:48 smithi142 ceph-mon[35271]: pgmap v95: 129 pgs: 129 active+clean; 50 MiB data, 632 MiB used, 1.0 TiB / 1.0 TiB avail; 276 KiB/s rd, 1.2 MiB/s wr, 115 op/s 2024-03-31T23:14:49.193 INFO:tasks.workunit.client.0.smithi046.stdout: 1 52059 2.57 MB/sec warmup 115 sec latency 19.330 ms 2024-03-31T23:14:50.193 INFO:tasks.workunit.client.0.smithi046.stdout: 1 52614 2.60 MB/sec warmup 116 sec latency 44.164 ms 2024-03-31T23:14:50.795 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:14:50 smithi162 ceph-mon[36237]: pgmap v96: 129 pgs: 129 active+clean; 49 MiB data, 632 MiB used, 1.0 TiB / 1.0 TiB avail; 428 KiB/s rd, 1.7 MiB/s wr, 179 op/s 2024-03-31T23:14:50.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:14:50 smithi162 ceph-mon[36237]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-31T23:14:50.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:14:50 smithi046 ceph-mon[32498]: pgmap v96: 129 pgs: 129 active+clean; 49 MiB data, 632 MiB used, 1.0 TiB / 1.0 TiB avail; 428 KiB/s rd, 1.7 MiB/s wr, 179 op/s 2024-03-31T23:14:50.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:14:50 smithi046 ceph-mon[32498]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-31T23:14:50.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:14:50 smithi142 ceph-mon[35271]: pgmap v96: 129 pgs: 129 active+clean; 49 MiB data, 632 MiB used, 1.0 TiB / 1.0 TiB avail; 428 KiB/s rd, 1.7 MiB/s wr, 179 op/s 2024-03-31T23:14:50.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:14:50 smithi142 ceph-mon[35271]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-31T23:14:51.030 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:51.027+0000 7f2b0641c640 1 -- 172.21.15.46:0/1004545225 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2b001055f0 msgr2=0x7f2b001059d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:14:51.031 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:51.027+0000 7f2b0641c640 1 --2- 172.21.15.46:0/1004545225 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2b001055f0 0x7f2b001059d0 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f2af0009900 tx=0x7f2af002bc30 comp rx=0 tx=0).stop 2024-03-31T23:14:51.031 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:51.027+0000 7f2b0641c640 1 -- 172.21.15.46:0/1004545225 shutdown_connections 2024-03-31T23:14:51.031 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:51.027+0000 7f2b0641c640 1 --2- 172.21.15.46:0/1004545225 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2b00101260 0x7f2b0010d480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:14:51.031 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:51.027+0000 7f2b0641c640 1 --2- 172.21.15.46:0/1004545225 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2b001008a0 0x7f2b00100d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:14:51.031 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:51.027+0000 7f2b0641c640 1 --2- 172.21.15.46:0/1004545225 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2b001055f0 0x7f2b001059d0 unknown :-1 s=CLOSED pgs=666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:14:51.031 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:51.027+0000 7f2b0641c640 1 -- 172.21.15.46:0/1004545225 >> 172.21.15.46:0/1004545225 conn(0x7f2b000774a0 msgr2=0x7f2b000778b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:14:51.031 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:51.027+0000 7f2b0641c640 1 -- 172.21.15.46:0/1004545225 shutdown_connections 2024-03-31T23:14:51.031 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:51.027+0000 7f2b0641c640 1 -- 172.21.15.46:0/1004545225 wait complete. 2024-03-31T23:14:51.031 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:51.028+0000 7f2b0641c640 1 Processor -- start 2024-03-31T23:14:51.031 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:51.028+0000 7f2b0641c640 1 -- start start 2024-03-31T23:14:51.031 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:51.028+0000 7f2b0641c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2b001008a0 0x7f2b00197f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:14:51.032 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:51.028+0000 7f2b0641c640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2b00101260 0x7f2b00198460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:14:51.032 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:51.028+0000 7f2b0641c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2b001055f0 0x7f2b0019c790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:14:51.032 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:51.028+0000 7f2b0641c640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f2b00069b80 con 0x7f2b001008a0 2024-03-31T23:14:51.032 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:51.028+0000 7f2b0641c640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f2b00069a00 con 0x7f2b00101260 2024-03-31T23:14:51.032 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:51.028+0000 7f2b0641c640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f2b00069d00 con 0x7f2b001055f0 2024-03-31T23:14:51.032 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:51.028+0000 7f2afffff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2b001008a0 0x7f2b00197f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:14:51.032 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:51.029+0000 7f2afffff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2b001008a0 0x7f2b00197f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:36720/0 (socket says 172.21.15.46:36720) 2024-03-31T23:14:51.032 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:51.029+0000 7f2aff7fe640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2b00101260 0x7f2b00198460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:14:51.032 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:51.029+0000 7f2afffff640 1 -- 172.21.15.46:0/1754778976 learned_addr learned my addr 172.21.15.46:0/1754778976 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:14:51.032 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:51.029+0000 7f2b049b9640 1 --2- 172.21.15.46:0/1754778976 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2b001055f0 0x7f2b0019c790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:14:51.032 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:51.029+0000 7f2aff7fe640 1 -- 172.21.15.46:0/1754778976 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2b001055f0 msgr2=0x7f2b0019c790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:14:51.032 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:51.029+0000 7f2aff7fe640 1 --2- 172.21.15.46:0/1754778976 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2b001055f0 0x7f2b0019c790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:14:51.032 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:51.029+0000 7f2aff7fe640 1 -- 172.21.15.46:0/1754778976 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2b001008a0 msgr2=0x7f2b00197f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:14:51.032 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:51.029+0000 7f2aff7fe640 1 --2- 172.21.15.46:0/1754778976 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2b001008a0 0x7f2b00197f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:14:51.032 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:51.029+0000 7f2aff7fe640 1 -- 172.21.15.46:0/1754778976 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b0019cec0 con 0x7f2b00101260 2024-03-31T23:14:51.033 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:51.029+0000 7f2afffff640 1 --2- 172.21.15.46:0/1754778976 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2b001008a0 0x7f2b00197f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:14:51.033 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:51.029+0000 7f2b049b9640 1 --2- 172.21.15.46:0/1754778976 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2b001055f0 0x7f2b0019c790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:14:51.033 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:51.029+0000 7f2aff7fe640 1 --2- 172.21.15.46:0/1754778976 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2b00101260 0x7f2b00198460 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f2af400e9f0 tx=0x7f2af400eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:14:51.033 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:51.029+0000 7f2afd7fa640 1 -- 172.21.15.46:0/1754778976 <== mon.2 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2af400ccc0 con 0x7f2b00101260 2024-03-31T23:14:51.033 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:51.029+0000 7f2b0641c640 1 -- 172.21.15.46:0/1754778976 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2b0019d1b0 con 0x7f2b00101260 2024-03-31T23:14:51.033 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:51.029+0000 7f2afd7fa640 1 -- 172.21.15.46:0/1754778976 <== mon.2 v2:172.21.15.142:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f2af4005020 con 0x7f2b00101260 2024-03-31T23:14:51.033 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:51.029+0000 7f2afd7fa640 1 -- 172.21.15.46:0/1754778976 <== mon.2 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2af4004a00 con 0x7f2b00101260 2024-03-31T23:14:51.033 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:51.029+0000 7f2b0641c640 1 -- 172.21.15.46:0/1754778976 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2b001a4a40 con 0x7f2b00101260 2024-03-31T23:14:51.038 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:51.032+0000 7f2afd7fa640 1 -- 172.21.15.46:0/1754778976 <== mon.2 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 25) ==== 137451+0+0 (secure 0 0 0) 0x7f2af4004ba0 con 0x7f2b00101260 2024-03-31T23:14:51.038 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:51.032+0000 7f2b08fff740 1 Processor -- start 2024-03-31T23:14:51.038 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:51.032+0000 7f2b08fff740 1 -- start start 2024-03-31T23:14:51.038 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:51.032+0000 7f2b08fff740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55dc9843c6c0 0x55dc9843caa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:14:51.038 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:51.032+0000 7f2b08fff740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55dc9843cfe0 0x55dc98445870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:14:51.039 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:51.032+0000 7f2b08fff740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dc98445db0 0x55dc984481a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:14:51.039 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:51.032+0000 7f2afd7fa640 1 --2- 172.21.15.46:0/1754778976 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f2ae80a26b0 0x7f2ae80a4b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:14:51.039 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:51.032+0000 7f2b08fff740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55dc9839f8f0 con 0x55dc9843cfe0 2024-03-31T23:14:51.039 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:51.032+0000 7f2b08fff740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55dc983488d0 con 0x55dc9843c6c0 2024-03-31T23:14:51.039 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:51.032+0000 7f2b08fff740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55dc9839b990 con 0x55dc98445db0 2024-03-31T23:14:51.039 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:51.032+0000 7f2b049b9640 1 --2- 172.21.15.46:0/1754778976 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f2ae80a26b0 0x7f2ae80a4b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:14:51.039 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:51.032+0000 7f2afffff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55dc9843cfe0 0x55dc98445870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:14:51.039 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:51.032+0000 7f2b049b9640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55dc9843c6c0 0x55dc9843caa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:14:51.039 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:51.032+0000 7f2afffff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55dc9843cfe0 0x55dc98445870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:36734/0 (socket says 172.21.15.46:36734) 2024-03-31T23:14:51.039 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:51.032+0000 7f2afffff640 1 -- 172.21.15.46:0/1970588805 learned_addr learned my addr 172.21.15.46:0/1970588805 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:14:51.039 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:51.032+0000 7f2aff7fe640 1 --2- 172.21.15.46:0/1970588805 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dc98445db0 0x55dc984481a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:14:51.039 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:51.032+0000 7f2afd7fa640 1 -- 172.21.15.46:0/1754778976 <== mon.2 v2:172.21.15.142:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7f2af40cee50 con 0x7f2b00101260 2024-03-31T23:14:51.040 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:51.033+0000 7f2b049b9640 1 -- 172.21.15.46:0/1970588805 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dc98445db0 msgr2=0x55dc984481a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:14:51.040 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:51.033+0000 7f2b049b9640 1 --2- 172.21.15.46:0/1970588805 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dc98445db0 0x55dc984481a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:14:51.040 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:51.033+0000 7f2b049b9640 1 -- 172.21.15.46:0/1970588805 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55dc9843cfe0 msgr2=0x55dc98445870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:14:51.040 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:51.033+0000 7f2b049b9640 1 --2- 172.21.15.46:0/1970588805 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55dc9843cfe0 0x55dc98445870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:14:51.040 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:51.033+0000 7f2b049b9640 1 -- 172.21.15.46:0/1970588805 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dc980b4210 con 0x55dc9843c6c0 2024-03-31T23:14:51.040 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:51.033+0000 7f2afffff640 1 --2- 172.21.15.46:0/1970588805 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55dc9843cfe0 0x55dc98445870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:14:51.040 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:51.033+0000 7f2b049b9640 1 --2- 172.21.15.46:0/1970588805 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55dc9843c6c0 0x55dc9843caa0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f2af8005fd0 tx=0x7f2af800fa50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=879d63583d571616 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:14:51.040 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:51.033+0000 7f2b049b9640 1 --2- 172.21.15.46:0/1754778976 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f2ae80a26b0 0x7f2ae80a4b70 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f2af800ff50 tx=0x7f2af801e000 comp rx=0 tx=0).ready entity=mgr.15654 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:14:51.040 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:51.033+0000 7f2ae67fc640 1 -- 172.21.15.46:0/1970588805 <== mon.2 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2af802b070 con 0x55dc9843c6c0 2024-03-31T23:14:51.040 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:51.033+0000 7f2ae67fc640 1 -- 172.21.15.46:0/1970588805 <== mon.2 v2:172.21.15.142:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f2af8010570 con 0x55dc9843c6c0 2024-03-31T23:14:51.040 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:51.033+0000 7f2ae67fc640 1 -- 172.21.15.46:0/1970588805 <== mon.2 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2af8019040 con 0x55dc9843c6c0 2024-03-31T23:14:51.040 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:51.034+0000 7f2b08fff740 1 -- 172.21.15.46:0/1970588805 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55dc9843c6c0 msgr2=0x55dc9843caa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:14:51.040 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:51.034+0000 7f2b08fff740 1 --2- 172.21.15.46:0/1970588805 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55dc9843c6c0 0x55dc9843caa0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f2af8005fd0 tx=0x7f2af800fa50 comp rx=0 tx=0).stop 2024-03-31T23:14:51.040 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:51.034+0000 7f2b08fff740 1 -- 172.21.15.46:0/1970588805 shutdown_connections 2024-03-31T23:14:51.041 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:51.034+0000 7f2b08fff740 1 --2- 172.21.15.46:0/1970588805 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dc98445db0 0x55dc984481a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:14:51.041 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:51.034+0000 7f2b08fff740 1 --2- 172.21.15.46:0/1970588805 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55dc9843cfe0 0x55dc98445870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:14:51.041 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:51.034+0000 7f2b08fff740 1 --2- 172.21.15.46:0/1970588805 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55dc9843c6c0 0x55dc9843caa0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:14:51.041 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:51.034+0000 7f2b08fff740 1 -- 172.21.15.46:0/1970588805 >> 172.21.15.46:0/1970588805 conn(0x55dc98436fe0 msgr2=0x55dc984373c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:14:51.041 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:51.034+0000 7f2b08fff740 1 -- 172.21.15.46:0/1970588805 shutdown_connections 2024-03-31T23:14:51.041 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:51.034+0000 7f2b08fff740 1 -- 172.21.15.46:0/1970588805 wait complete. 2024-03-31T23:14:51.041 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:51.034+0000 7f2b08fff740 1 -- start start 2024-03-31T23:14:51.041 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:51.034+0000 7f2b08fff740 1 Processor -- start 2024-03-31T23:14:51.041 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:51.035+0000 7f2b08fff740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55dc9843c6c0 0x55dc98442e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:14:51.041 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:51.035+0000 7f2b08fff740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dc9843cfe0 0x55dc98443370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:14:51.041 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:51.035+0000 7f2b08fff740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55dc98445db0 0x55dc984438b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:14:51.041 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:51.035+0000 7f2b08fff740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55dc98448980 con 0x55dc98445db0 2024-03-31T23:14:51.041 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:51.035+0000 7f2b08fff740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55dc9833df80 con 0x55dc9843c6c0 2024-03-31T23:14:51.041 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:51.035+0000 7f2b08fff740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55dc98448b00 con 0x55dc9843cfe0 2024-03-31T23:14:51.042 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:51.035+0000 7f2b08fff740 10 client.0 _ll_register_callbacks cb 0x55dc98396290 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T23:14:51.042 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:51.035+0000 7f2b049b9640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55dc98445db0 0x55dc984438b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:14:51.042 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:51.035+0000 7f2afffff640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55dc9843c6c0 0x55dc98442e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:14:51.042 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:51.035+0000 7f2aff7fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dc9843cfe0 0x55dc98443370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:14:51.042 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:51.035+0000 7f2b049b9640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55dc98445db0 0x55dc984438b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:36744/0 (socket says 172.21.15.46:36744) 2024-03-31T23:14:51.042 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:51.035+0000 7f2b049b9640 1 -- 172.21.15.46:0/724724183 learned_addr learned my addr 172.21.15.46:0/724724183 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:14:51.042 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:51.035+0000 7f2b049b9640 1 -- 172.21.15.46:0/724724183 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dc9843cfe0 msgr2=0x55dc98443370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:14:51.042 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:51.035+0000 7f2b049b9640 1 --2- 172.21.15.46:0/724724183 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dc9843cfe0 0x55dc98443370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:14:51.042 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:51.035+0000 7f2b049b9640 1 -- 172.21.15.46:0/724724183 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55dc9843c6c0 msgr2=0x55dc98442e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:14:51.042 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:51.035+0000 7f2b049b9640 1 --2- 172.21.15.46:0/724724183 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55dc9843c6c0 0x55dc98442e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:14:51.042 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:51.035+0000 7f2b049b9640 1 -- 172.21.15.46:0/724724183 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dc98180620 con 0x55dc98445db0 2024-03-31T23:14:51.042 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:51.035+0000 7f2b049b9640 1 --2- 172.21.15.46:0/724724183 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55dc98445db0 0x55dc984438b0 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f2af8029040 tx=0x7f2af8010e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5090b532568e9156 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:14:51.042 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:51.035+0000 7f2adf7fe640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-31T23:14:51.043 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:51.035+0000 7f2adf7fe640 1 -- 172.21.15.46:0/724724183 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2af8034060 con 0x55dc98445db0 2024-03-31T23:14:51.043 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:51.035+0000 7f2adf7fe640 1 -- 172.21.15.46:0/724724183 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7f2af8026450 con 0x55dc98445db0 2024-03-31T23:14:51.043 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:51.035+0000 7f2b08fff740 1 -- 172.21.15.46:0/724724183 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2af8010570 con 0x55dc98445db0 2024-03-31T23:14:51.043 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:51.036+0000 7f2adf7fe640 1 -- 172.21.15.46:0/724724183 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f2af803b050 con 0x55dc98445db0 2024-03-31T23:14:51.043 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:51.036+0000 7f2b08fff740 10 client.16401 fetch_fsmap learned FSMap version 28 2024-03-31T23:14:51.043 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:51.036+0000 7f2b08fff740 1 -- 172.21.15.46:0/724724183 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55dc98458cb0 con 0x55dc98445db0 2024-03-31T23:14:51.045 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:51.036+0000 7f2adf7fe640 1 -- 172.21.15.46:0/724724183 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f2af8010570 con 0x55dc98445db0 2024-03-31T23:14:51.045 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:51.036+0000 7f2adf7fe640 1 -- 172.21.15.46:0/724724183 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f2af8005750 con 0x55dc98445db0 2024-03-31T23:14:51.045 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:51.036+0000 7f2b08fff740 10 client.16401 fetch_fsmap finished waiting for FSMap version 28 2024-03-31T23:14:51.045 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:51.036+0000 7f2b08fff740 10 client.16401 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-31T23:14:51.046 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:51.036+0000 7f2b08fff740 20 client.16401 populate_metadata read hostname 'smithi046' 2024-03-31T23:14:51.046 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:51.036+0000 7f2b08fff740 1 --2- 172.21.15.46:0/724724183 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55dc9845a170 0x55dc9845c560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:14:51.046 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:51.036+0000 7f2b08fff740 4 client.16401 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-31T23:14:51.046 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:51.036+0000 7f2b08fff740 1 -- 172.21.15.46:0/724724183 --> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55dc983b3120 con 0x55dc9845a170 2024-03-31T23:14:51.046 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:51.036+0000 7f2afffff640 1 --2- 172.21.15.46:0/724724183 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55dc9845a170 0x55dc9845c560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:14:51.046 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:51.037+0000 7f2adf7fe640 10 client.16401 ms_handle_connect on v2:172.21.15.142:6838/2519143918 2024-03-31T23:14:51.046 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:51.037+0000 7f2afffff640 1 --2- 172.21.15.46:0/724724183 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55dc9845a170 0x55dc9845c560 crc :-1 s=READY pgs=101 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6f568c0b78b01846 server_cookie=85ac874eed1ede7f in_seq=0 out_seq=0 2024-03-31T23:14:51.047 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:51.038+0000 7f2adf7fe640 1 -- 172.21.15.46:0/724724183 <== mds.0 v2:172.21.15.142:6838/2519143918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55dc983b3120 con 0x55dc9845a170 2024-03-31T23:14:51.047 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:51.038+0000 7f2adf7fe640 10 client.16401 handle_command_reply: tid=0 2024-03-31T23:14:51.047 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:51.038+0000 7f2b08fff740 1 client.16401 shutdown 2024-03-31T23:14:51.048 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:51.038+0000 7f2b08fff740 1 -- 172.21.15.46:0/724724183 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55dc98445db0 msgr2=0x55dc984438b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:14:51.048 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:51.038+0000 7f2b08fff740 1 --2- 172.21.15.46:0/724724183 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55dc98445db0 0x55dc984438b0 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f2af8029040 tx=0x7f2af8010e70 comp rx=0 tx=0).stop 2024-03-31T23:14:51.048 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:51.038+0000 7f2b08fff740 1 -- 172.21.15.46:0/724724183 shutdown_connections 2024-03-31T23:14:51.048 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:51.038+0000 7f2b08fff740 1 --2- 172.21.15.46:0/724724183 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55dc9845a170 0x55dc9845c560 crc :-1 s=READY pgs=101 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:14:51.048 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:51.038+0000 7f2b08fff740 1 --2- 172.21.15.46:0/724724183 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55dc98445db0 0x55dc984438b0 unknown :-1 s=CLOSED pgs=667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:14:51.048 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:51.038+0000 7f2b08fff740 1 --2- 172.21.15.46:0/724724183 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dc9843cfe0 0x55dc98443370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:14:51.048 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:51.038+0000 7f2b08fff740 1 --2- 172.21.15.46:0/724724183 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55dc9843c6c0 0x55dc98442e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:14:51.048 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:51.038+0000 7f2adf7fe640 0 client.16401 ms_handle_reset on v2:172.21.15.142:6838/2519143918 2024-03-31T23:14:51.048 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:51.038+0000 7f2b08fff740 1 -- 172.21.15.46:0/724724183 >> 172.21.15.46:0/724724183 conn(0x55dc98436fe0 msgr2=0x55dc98438d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:14:51.048 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:51.038+0000 7f2b08fff740 1 -- 172.21.15.46:0/724724183 shutdown_connections 2024-03-31T23:14:51.048 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:51.038+0000 7f2b08fff740 1 -- 172.21.15.46:0/724724183 wait complete. 2024-03-31T23:14:51.048 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:51.038+0000 7f2b08fff740 20 client.16401 trim_cache size 0 max 16384 2024-03-31T23:14:51.058 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:51.056+0000 7f2b08fff740 1 Processor -- start 2024-03-31T23:14:51.058 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:51.056+0000 7f2b08fff740 1 -- start start 2024-03-31T23:14:51.058 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:51.056+0000 7f2b08fff740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55dc9843c6c0 0x55dc98441740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:14:51.058 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:51.056+0000 7f2b08fff740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55dc9843cfe0 0x55dc98451720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:14:51.059 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:51.056+0000 7f2b08fff740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dc98445db0 0x55dc984436d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:14:51.059 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:51.056+0000 7f2b08fff740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f2b001af160 con 0x55dc9843cfe0 2024-03-31T23:14:51.059 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:51.056+0000 7f2b08fff740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f2b001aca30 con 0x55dc9843c6c0 2024-03-31T23:14:51.059 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:51.056+0000 7f2b08fff740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f2b001aa880 con 0x55dc98445db0 2024-03-31T23:14:51.059 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:51.056+0000 7f2aff7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55dc9843cfe0 0x55dc98451720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:14:51.059 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:51.056+0000 7f2aff7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55dc9843cfe0 0x55dc98451720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:36746/0 (socket says 172.21.15.46:36746) 2024-03-31T23:14:51.059 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:51.056+0000 7f2aff7fe640 1 -- 172.21.15.46:0/568257008 learned_addr learned my addr 172.21.15.46:0/568257008 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:14:51.059 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:51.056+0000 7f2b049b9640 1 --2- 172.21.15.46:0/568257008 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dc98445db0 0x55dc984436d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:14:51.059 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:51.056+0000 7f2afffff640 1 --2- 172.21.15.46:0/568257008 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55dc9843c6c0 0x55dc98441740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:14:51.059 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:51.057+0000 7f2aff7fe640 1 -- 172.21.15.46:0/568257008 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dc98445db0 msgr2=0x55dc984436d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:14:51.059 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:51.057+0000 7f2aff7fe640 1 --2- 172.21.15.46:0/568257008 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dc98445db0 0x55dc984436d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:14:51.060 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:51.057+0000 7f2aff7fe640 1 -- 172.21.15.46:0/568257008 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55dc9843c6c0 msgr2=0x55dc98441740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:14:51.060 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:51.057+0000 7f2aff7fe640 1 --2- 172.21.15.46:0/568257008 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55dc9843c6c0 0x55dc98441740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:14:51.060 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:51.057+0000 7f2aff7fe640 1 -- 172.21.15.46:0/568257008 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dc98441e20 con 0x55dc9843cfe0 2024-03-31T23:14:51.060 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:51.057+0000 7f2b049b9640 1 --2- 172.21.15.46:0/568257008 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dc98445db0 0x55dc984436d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:14:51.060 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:51.057+0000 7f2aff7fe640 1 --2- 172.21.15.46:0/568257008 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55dc9843cfe0 0x55dc98451720 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f2af40cc250 tx=0x7f2af40054d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a5916cf346e29c1 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:14:51.060 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:51.057+0000 7f2ae5ffb640 1 -- 172.21.15.46:0/568257008 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2af40b41c0 con 0x55dc9843cfe0 2024-03-31T23:14:51.060 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:51.057+0000 7f2ae5ffb640 1 -- 172.21.15.46:0/568257008 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f2af409f160 con 0x55dc9843cfe0 2024-03-31T23:14:51.060 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:51.057+0000 7f2ae5ffb640 1 -- 172.21.15.46:0/568257008 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2af40051d0 con 0x55dc9843cfe0 2024-03-31T23:14:51.064 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:51.058+0000 7f2b08fff740 1 -- 172.21.15.46:0/568257008 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55dc9843cfe0 msgr2=0x55dc98451720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:14:51.064 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:51.058+0000 7f2b08fff740 1 --2- 172.21.15.46:0/568257008 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55dc9843cfe0 0x55dc98451720 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f2af40cc250 tx=0x7f2af40054d0 comp rx=0 tx=0).stop 2024-03-31T23:14:51.064 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:51.058+0000 7f2b08fff740 1 -- 172.21.15.46:0/568257008 shutdown_connections 2024-03-31T23:14:51.064 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:51.058+0000 7f2b08fff740 1 --2- 172.21.15.46:0/568257008 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dc98445db0 0x55dc984436d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:14:51.064 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:51.058+0000 7f2b08fff740 1 --2- 172.21.15.46:0/568257008 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55dc9843cfe0 0x55dc98451720 unknown :-1 s=CLOSED pgs=668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:14:51.064 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:51.058+0000 7f2b08fff740 1 --2- 172.21.15.46:0/568257008 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55dc9843c6c0 0x55dc98441740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:14:51.064 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:51.058+0000 7f2b08fff740 1 -- 172.21.15.46:0/568257008 >> 172.21.15.46:0/568257008 conn(0x55dc98436fe0 msgr2=0x55dc98435f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:14:51.064 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:51.058+0000 7f2b08fff740 1 -- 172.21.15.46:0/568257008 shutdown_connections 2024-03-31T23:14:51.064 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:51.058+0000 7f2b08fff740 1 -- 172.21.15.46:0/568257008 wait complete. 2024-03-31T23:14:51.065 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:51.058+0000 7f2b08fff740 1 -- start start 2024-03-31T23:14:51.065 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:51.058+0000 7f2b08fff740 1 Processor -- start 2024-03-31T23:14:51.065 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:51.058+0000 7f2b08fff740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dc9843c6c0 0x55dc9843fb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:14:51.065 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:51.058+0000 7f2b08fff740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55dc9843cfe0 0x55dc98440060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:14:51.065 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:51.059+0000 7f2b08fff740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55dc98445db0 0x55dc984405a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:14:51.065 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:51.059+0000 7f2b08fff740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f2b001aaef0 con 0x55dc9843cfe0 2024-03-31T23:14:51.065 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:51.059+0000 7f2b08fff740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f2b001ac360 con 0x55dc98445db0 2024-03-31T23:14:51.065 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:51.059+0000 7f2b08fff740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f2b001ab5c0 con 0x55dc9843c6c0 2024-03-31T23:14:51.065 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:51.059+0000 7f2b08fff740 10 client.0 _ll_register_callbacks cb 0x7f2af8029070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T23:14:51.065 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:51.059+0000 7f2aff7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55dc9843cfe0 0x55dc98440060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:14:51.065 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:51.059+0000 7f2afffff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dc9843c6c0 0x55dc9843fb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:14:51.065 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:51.059+0000 7f2aff7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55dc9843cfe0 0x55dc98440060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:36748/0 (socket says 172.21.15.46:36748) 2024-03-31T23:14:51.065 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:51.059+0000 7f2aff7fe640 1 -- 172.21.15.46:0/232389805 learned_addr learned my addr 172.21.15.46:0/232389805 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:14:51.065 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:51.059+0000 7f2b049b9640 1 --2- 172.21.15.46:0/232389805 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55dc98445db0 0x55dc984405a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:14:51.066 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:51.059+0000 7f2afffff640 1 -- 172.21.15.46:0/232389805 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55dc98445db0 msgr2=0x55dc984405a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:14:51.066 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:51.059+0000 7f2afffff640 1 --2- 172.21.15.46:0/232389805 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55dc98445db0 0x55dc984405a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:14:51.066 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:51.059+0000 7f2afffff640 1 -- 172.21.15.46:0/232389805 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55dc9843cfe0 msgr2=0x55dc98440060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:14:51.066 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:51.059+0000 7f2afffff640 1 --2- 172.21.15.46:0/232389805 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55dc9843cfe0 0x55dc98440060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:14:51.066 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:51.059+0000 7f2afffff640 1 -- 172.21.15.46:0/232389805 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dc984bf320 con 0x55dc9843c6c0 2024-03-31T23:14:51.066 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:51.059+0000 7f2aff7fe640 1 --2- 172.21.15.46:0/232389805 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55dc9843cfe0 0x55dc98440060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:14:51.066 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:51.059+0000 7f2afffff640 1 --2- 172.21.15.46:0/232389805 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dc9843c6c0 0x55dc9843fb20 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f2af0009230 tx=0x7f2af002cb10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3848e72c935f5565 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:14:51.066 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:51.059+0000 7f2adf7fe640 10 client.0 ms_handle_connect on v2:172.21.15.162:3300/0 2024-03-31T23:14:51.066 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:51.059+0000 7f2adf7fe640 1 -- 172.21.15.46:0/232389805 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2af0050070 con 0x55dc9843c6c0 2024-03-31T23:14:51.066 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:51.059+0000 7f2b08fff740 1 -- 172.21.15.46:0/232389805 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2af409f160 con 0x55dc9843c6c0 2024-03-31T23:14:51.066 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:51.059+0000 7f2adf7fe640 1 -- 172.21.15.46:0/232389805 <== mon.1 v2:172.21.15.162:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7f2af0049d20 con 0x55dc9843c6c0 2024-03-31T23:14:51.066 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:51.060+0000 7f2adf7fe640 1 -- 172.21.15.46:0/232389805 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f2af005c050 con 0x55dc9843c6c0 2024-03-31T23:14:51.066 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:51.060+0000 7f2b08fff740 10 client.26143 fetch_fsmap learned FSMap version 28 2024-03-31T23:14:51.067 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:51.060+0000 7f2b08fff740 1 -- 172.21.15.46:0/232389805 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55dc984c05b0 con 0x55dc9843c6c0 2024-03-31T23:14:51.067 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:51.060+0000 7f2adf7fe640 1 -- 172.21.15.46:0/232389805 <== mon.1 v2:172.21.15.162:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f2af409f160 con 0x55dc9843c6c0 2024-03-31T23:14:51.067 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:51.060+0000 7f2adf7fe640 1 -- 172.21.15.46:0/232389805 <== mon.1 v2:172.21.15.162:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f2af00035f0 con 0x55dc9843c6c0 2024-03-31T23:14:51.067 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:51.060+0000 7f2b08fff740 10 client.26143 fetch_fsmap finished waiting for FSMap version 28 2024-03-31T23:14:51.067 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:51.060+0000 7f2b08fff740 10 client.26143 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-31T23:14:51.067 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:51.060+0000 7f2b08fff740 20 client.26143 populate_metadata read hostname 'smithi046' 2024-03-31T23:14:51.067 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:51.060+0000 7f2b08fff740 1 --2- 172.21.15.46:0/232389805 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55dc984c0d10 0x55dc984c3130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:14:51.067 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:51.060+0000 7f2b08fff740 4 client.26143 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-31T23:14:51.067 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:51.060+0000 7f2b08fff740 1 -- 172.21.15.46:0/232389805 --> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55dc983ace30 con 0x55dc984c0d10 2024-03-31T23:14:51.067 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:51.060+0000 7f2aff7fe640 1 --2- 172.21.15.46:0/232389805 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55dc984c0d10 0x55dc984c3130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:14:51.067 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:51.060+0000 7f2adf7fe640 10 client.26143 ms_handle_connect on v2:172.21.15.142:6838/2519143918 2024-03-31T23:14:51.067 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:51.060+0000 7f2aff7fe640 1 --2- 172.21.15.46:0/232389805 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55dc984c0d10 0x55dc984c3130 crc :-1 s=READY pgs=102 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=def0721d7bec5c6 server_cookie=b24d99894003caa in_seq=0 out_seq=0 2024-03-31T23:14:51.088 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:51.086+0000 7f2adf7fe640 1 -- 172.21.15.46:0/232389805 <== mds.0 v2:172.21.15.142:6838/2519143918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55dc983ace30 con 0x55dc984c0d10 2024-03-31T23:14:51.088 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:51.086+0000 7f2adf7fe640 10 client.26143 handle_command_reply: tid=0 2024-03-31T23:14:51.088 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:51.086+0000 7f2b08fff740 1 client.26143 shutdown 2024-03-31T23:14:51.088 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:51.086+0000 7f2b08fff740 1 -- 172.21.15.46:0/232389805 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dc9843c6c0 msgr2=0x55dc9843fb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:14:51.088 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:51.086+0000 7f2b08fff740 1 --2- 172.21.15.46:0/232389805 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dc9843c6c0 0x55dc9843fb20 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f2af0009230 tx=0x7f2af002cb10 comp rx=0 tx=0).stop 2024-03-31T23:14:51.088 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:51.086+0000 7f2b08fff740 1 -- 172.21.15.46:0/232389805 shutdown_connections 2024-03-31T23:14:51.088 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:51.086+0000 7f2b08fff740 1 --2- 172.21.15.46:0/232389805 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55dc984c0d10 0x55dc984c3130 crc :-1 s=READY pgs=102 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:14:51.088 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:51.086+0000 7f2b08fff740 1 --2- 172.21.15.46:0/232389805 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55dc98445db0 0x55dc984405a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:14:51.088 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:51.086+0000 7f2adf7fe640 0 client.26143 ms_handle_reset on v2:172.21.15.142:6838/2519143918 2024-03-31T23:14:51.088 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:51.086+0000 7f2b08fff740 1 --2- 172.21.15.46:0/232389805 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55dc9843cfe0 0x55dc98440060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:14:51.088 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:51.086+0000 7f2b08fff740 1 --2- 172.21.15.46:0/232389805 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dc9843c6c0 0x55dc9843fb20 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:14:51.089 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:51.086+0000 7f2b08fff740 1 -- 172.21.15.46:0/232389805 >> 172.21.15.46:0/232389805 conn(0x55dc98436fe0 msgr2=0x55dc98435f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:14:51.089 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:51.086+0000 7f2b08fff740 1 -- 172.21.15.46:0/232389805 shutdown_connections 2024-03-31T23:14:51.089 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:51.086+0000 7f2b08fff740 1 -- 172.21.15.46:0/232389805 wait complete. 2024-03-31T23:14:51.089 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:51.086+0000 7f2b08fff740 20 client.26143 trim_cache size 0 max 16384 2024-03-31T23:14:51.089 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout:{ 2024-03-31T23:14:51.089 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout: "message": "", 2024-03-31T23:14:51.089 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout: "return_code": 0 2024-03-31T23:14:51.089 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout:} 2024-03-31T23:14:51.090 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:51.087+0000 7f2b0641c640 1 -- 172.21.15.46:0/1754778976 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f2ae80a26b0 msgr2=0x7f2ae80a4b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:14:51.090 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:51.087+0000 7f2b0641c640 1 --2- 172.21.15.46:0/1754778976 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f2ae80a26b0 0x7f2ae80a4b70 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f2af800ff50 tx=0x7f2af801e000 comp rx=0 tx=0).stop 2024-03-31T23:14:51.090 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:51.087+0000 7f2b0641c640 1 -- 172.21.15.46:0/1754778976 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2b00101260 msgr2=0x7f2b00198460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:14:51.090 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:51.087+0000 7f2b0641c640 1 --2- 172.21.15.46:0/1754778976 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2b00101260 0x7f2b00198460 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f2af400e9f0 tx=0x7f2af400eeb0 comp rx=0 tx=0).stop 2024-03-31T23:14:51.091 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:51.087+0000 7f2b0641c640 1 -- 172.21.15.46:0/1754778976 shutdown_connections 2024-03-31T23:14:51.091 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:51.087+0000 7f2b0641c640 1 --2- 172.21.15.46:0/1754778976 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f2ae80a26b0 0x7f2ae80a4b70 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:14:51.091 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:51.087+0000 7f2b0641c640 1 --2- 172.21.15.46:0/1754778976 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2b001055f0 0x7f2b0019c790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:14:51.091 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:51.087+0000 7f2b0641c640 1 --2- 172.21.15.46:0/1754778976 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2b00101260 0x7f2b00198460 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:14:51.091 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:51.087+0000 7f2b0641c640 1 --2- 172.21.15.46:0/1754778976 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2b001008a0 0x7f2b00197f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:14:51.091 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:51.087+0000 7f2b0641c640 1 -- 172.21.15.46:0/1754778976 >> 172.21.15.46:0/1754778976 conn(0x7f2b000774a0 msgr2=0x7f2b000fe8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:14:51.091 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:51.087+0000 7f2b0641c640 1 -- 172.21.15.46:0/1754778976 shutdown_connections 2024-03-31T23:14:51.091 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:14:51.087+0000 7f2b0641c640 1 -- 172.21.15.46:0/1754778976 wait complete. 2024-03-31T23:14:51.193 INFO:tasks.workunit.client.0.smithi046.stdout: 1 53020 2.59 MB/sec warmup 117 sec latency 24.283 ms 2024-03-31T23:14:51.795 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:14:51 smithi162 ceph-mon[36237]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:14:51.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:14:51 smithi162 ceph-mon[36237]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-31T23:14:51.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:14:51 smithi162 ceph-mon[36237]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' 2024-03-31T23:14:51.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:14:51 smithi046 ceph-mon[32498]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:14:51.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:14:51 smithi046 ceph-mon[32498]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-31T23:14:51.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:14:51 smithi046 ceph-mon[32498]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' 2024-03-31T23:14:51.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:14:51 smithi142 ceph-mon[35271]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:14:51.944 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:14:51 smithi142 ceph-mon[35271]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-31T23:14:51.944 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:14:51 smithi142 ceph-mon[35271]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' 2024-03-31T23:14:52.193 INFO:tasks.workunit.client.0.smithi046.stdout: 1 53498 2.61 MB/sec warmup 118 sec latency 15.478 ms 2024-03-31T23:14:52.795 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:14:52 smithi162 ceph-mon[36237]: pgmap v97: 129 pgs: 129 active+clean; 49 MiB data, 632 MiB used, 1.0 TiB / 1.0 TiB avail; 285 KiB/s rd, 1.0 MiB/s wr, 119 op/s 2024-03-31T23:14:52.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:14:52 smithi046 ceph-mon[32498]: pgmap v97: 129 pgs: 129 active+clean; 49 MiB data, 632 MiB used, 1.0 TiB / 1.0 TiB avail; 285 KiB/s rd, 1.0 MiB/s wr, 119 op/s 2024-03-31T23:14:52.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:14:52 smithi142 ceph-mon[35271]: pgmap v97: 129 pgs: 129 active+clean; 49 MiB data, 632 MiB used, 1.0 TiB / 1.0 TiB avail; 285 KiB/s rd, 1.0 MiB/s wr, 119 op/s 2024-03-31T23:14:53.194 INFO:tasks.workunit.client.0.smithi046.stdout: 1 53778 2.59 MB/sec warmup 119 sec latency 15.348 ms 2024-03-31T23:14:54.795 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:14:54 smithi162 ceph-mon[36237]: pgmap v98: 129 pgs: 129 active+clean; 49 MiB data, 632 MiB used, 1.0 TiB / 1.0 TiB avail; 285 KiB/s rd, 1.0 MiB/s wr, 119 op/s 2024-03-31T23:14:54.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:14:54 smithi046 ceph-mon[32498]: pgmap v98: 129 pgs: 129 active+clean; 49 MiB data, 632 MiB used, 1.0 TiB / 1.0 TiB avail; 285 KiB/s rd, 1.0 MiB/s wr, 119 op/s 2024-03-31T23:14:54.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:14:54 smithi142 ceph-mon[35271]: pgmap v98: 129 pgs: 129 active+clean; 49 MiB data, 632 MiB used, 1.0 TiB / 1.0 TiB avail; 285 KiB/s rd, 1.0 MiB/s wr, 119 op/s 2024-03-31T23:14:55.194 INFO:tasks.workunit.client.0.smithi046.stdout: 1 54732 4.27 MB/sec execute 1 sec latency 15.449 ms 2024-03-31T23:14:56.194 INFO:tasks.workunit.client.0.smithi046.stdout: 1 55327 2.44 MB/sec execute 2 sec latency 15.693 ms 2024-03-31T23:14:56.795 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:14:56 smithi162 ceph-mon[36237]: pgmap v99: 129 pgs: 129 active+clean; 50 MiB data, 638 MiB used, 1.0 TiB / 1.0 TiB avail; 433 KiB/s rd, 1.7 MiB/s wr, 178 op/s 2024-03-31T23:14:56.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:14:56 smithi046 ceph-mon[32498]: pgmap v99: 129 pgs: 129 active+clean; 50 MiB data, 638 MiB used, 1.0 TiB / 1.0 TiB avail; 433 KiB/s rd, 1.7 MiB/s wr, 178 op/s 2024-03-31T23:14:56.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:14:56 smithi142 ceph-mon[35271]: pgmap v99: 129 pgs: 129 active+clean; 50 MiB data, 638 MiB used, 1.0 TiB / 1.0 TiB avail; 433 KiB/s rd, 1.7 MiB/s wr, 178 op/s 2024-03-31T23:14:57.194 INFO:tasks.workunit.client.0.smithi046.stdout: 1 55822 2.39 MB/sec execute 3 sec latency 16.451 ms 2024-03-31T23:14:58.194 INFO:tasks.workunit.client.0.smithi046.stdout: 1 56542 3.40 MB/sec execute 4 sec latency 17.147 ms 2024-03-31T23:14:58.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:14:58 smithi162 ceph-mon[36237]: pgmap v100: 129 pgs: 129 active+clean; 50 MiB data, 638 MiB used, 1.0 TiB / 1.0 TiB avail; 301 KiB/s rd, 1.1 MiB/s wr, 122 op/s 2024-03-31T23:14:58.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:14:58 smithi046 ceph-mon[32498]: pgmap v100: 129 pgs: 129 active+clean; 50 MiB data, 638 MiB used, 1.0 TiB / 1.0 TiB avail; 301 KiB/s rd, 1.1 MiB/s wr, 122 op/s 2024-03-31T23:14:58.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:14:58 smithi142 ceph-mon[35271]: pgmap v100: 129 pgs: 129 active+clean; 50 MiB data, 638 MiB used, 1.0 TiB / 1.0 TiB avail; 301 KiB/s rd, 1.1 MiB/s wr, 122 op/s 2024-03-31T23:14:59.194 INFO:tasks.workunit.client.0.smithi046.stdout: 1 56992 3.60 MB/sec execute 5 sec latency 15.563 ms 2024-03-31T23:15:00.194 INFO:tasks.workunit.client.0.smithi046.stdout: 1 57244 3.05 MB/sec execute 6 sec latency 13.117 ms 2024-03-31T23:15:00.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:15:00 smithi162 ceph-mon[36237]: pgmap v101: 129 pgs: 129 active+clean; 51 MiB data, 655 MiB used, 1.0 TiB / 1.0 TiB avail; 425 KiB/s rd, 1.8 MiB/s wr, 179 op/s 2024-03-31T23:15:00.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:15:00 smithi046 ceph-mon[32498]: pgmap v101: 129 pgs: 129 active+clean; 51 MiB data, 655 MiB used, 1.0 TiB / 1.0 TiB avail; 425 KiB/s rd, 1.8 MiB/s wr, 179 op/s 2024-03-31T23:15:00.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:15:00 smithi142 ceph-mon[35271]: pgmap v101: 129 pgs: 129 active+clean; 51 MiB data, 655 MiB used, 1.0 TiB / 1.0 TiB avail; 425 KiB/s rd, 1.8 MiB/s wr, 179 op/s 2024-03-31T23:15:01.194 INFO:tasks.workunit.client.0.smithi046.stdout: 1 57605 2.69 MB/sec execute 7 sec latency 22.598 ms 2024-03-31T23:15:02.194 INFO:tasks.workunit.client.0.smithi046.stdout: 1 58069 2.56 MB/sec execute 8 sec latency 21.010 ms 2024-03-31T23:15:02.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:15:02 smithi046 ceph-mon[32498]: pgmap v102: 129 pgs: 129 active+clean; 51 MiB data, 655 MiB used, 1.0 TiB / 1.0 TiB avail; 272 KiB/s rd, 1.3 MiB/s wr, 115 op/s 2024-03-31T23:15:02.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:15:02 smithi142 ceph-mon[35271]: pgmap v102: 129 pgs: 129 active+clean; 51 MiB data, 655 MiB used, 1.0 TiB / 1.0 TiB avail; 272 KiB/s rd, 1.3 MiB/s wr, 115 op/s 2024-03-31T23:15:03.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:15:02 smithi162 ceph-mon[36237]: pgmap v102: 129 pgs: 129 active+clean; 51 MiB data, 655 MiB used, 1.0 TiB / 1.0 TiB avail; 272 KiB/s rd, 1.3 MiB/s wr, 115 op/s 2024-03-31T23:15:03.194 INFO:tasks.workunit.client.0.smithi046.stdout: 1 58684 2.64 MB/sec execute 9 sec latency 26.461 ms 2024-03-31T23:15:04.183 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:04.181+0000 7ff81a302640 1 -- 172.21.15.46:0/3885574351 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff814101260 msgr2=0x7ff81410d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:15:04.183 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:04.181+0000 7ff81a302640 1 --2- 172.21.15.46:0/3885574351 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff814101260 0x7ff81410d480 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7ff80c009ca0 tx=0x7ff80c02c0f0 comp rx=0 tx=0).stop 2024-03-31T23:15:04.183 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:04.182+0000 7ff81a302640 1 -- 172.21.15.46:0/3885574351 shutdown_connections 2024-03-31T23:15:04.183 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:04.182+0000 7ff81a302640 1 --2- 172.21.15.46:0/3885574351 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff814101260 0x7ff81410d480 unknown :-1 s=CLOSED pgs=669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:15:04.183 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:04.182+0000 7ff81a302640 1 --2- 172.21.15.46:0/3885574351 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff8141008a0 0x7ff814100d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:15:04.183 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:04.182+0000 7ff81a302640 1 --2- 172.21.15.46:0/3885574351 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff8141055f0 0x7ff8141059d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:15:04.184 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:04.182+0000 7ff81a302640 1 -- 172.21.15.46:0/3885574351 >> 172.21.15.46:0/3885574351 conn(0x7ff8140774a0 msgr2=0x7ff8140778b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:15:04.184 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:04.182+0000 7ff81a302640 1 -- 172.21.15.46:0/3885574351 shutdown_connections 2024-03-31T23:15:04.184 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:04.182+0000 7ff81a302640 1 -- 172.21.15.46:0/3885574351 wait complete. 2024-03-31T23:15:04.184 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:04.182+0000 7ff81a302640 1 Processor -- start 2024-03-31T23:15:04.186 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:04.183+0000 7ff81a302640 1 -- start start 2024-03-31T23:15:04.186 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:04.183+0000 7ff81a302640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff8141008a0 0x7ff814197eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:15:04.186 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:04.183+0000 7ff81a302640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff814101260 0x7ff8141983f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:15:04.186 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:04.183+0000 7ff81a302640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff8141055f0 0x7ff81419c770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:15:04.186 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:04.183+0000 7ff81a302640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7ff8141109c0 con 0x7ff8141008a0 2024-03-31T23:15:04.187 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:04.183+0000 7ff81a302640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7ff814110840 con 0x7ff814101260 2024-03-31T23:15:04.187 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:04.183+0000 7ff81a302640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7ff814110b40 con 0x7ff8141055f0 2024-03-31T23:15:04.187 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:04.183+0000 7ff813fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff8141008a0 0x7ff814197eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:15:04.187 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:04.183+0000 7ff8137fe640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff814101260 0x7ff8141983f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:15:04.187 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:04.183+0000 7ff8137fe640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff814101260 0x7ff8141983f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.46:52786/0 (socket says 172.21.15.46:52786) 2024-03-31T23:15:04.187 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:04.183+0000 7ff8137fe640 1 -- 172.21.15.46:0/2814120497 learned_addr learned my addr 172.21.15.46:0/2814120497 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:15:04.187 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:04.183+0000 7ff81889f640 1 --2- 172.21.15.46:0/2814120497 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff8141055f0 0x7ff81419c770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:15:04.187 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:04.183+0000 7ff8137fe640 1 -- 172.21.15.46:0/2814120497 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff8141055f0 msgr2=0x7ff81419c770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:15:04.187 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:04.183+0000 7ff8137fe640 1 --2- 172.21.15.46:0/2814120497 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff8141055f0 0x7ff81419c770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:15:04.188 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:04.183+0000 7ff8137fe640 1 -- 172.21.15.46:0/2814120497 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff8141008a0 msgr2=0x7ff814197eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:15:04.188 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:04.183+0000 7ff8137fe640 1 --2- 172.21.15.46:0/2814120497 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff8141008a0 0x7ff814197eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:15:04.188 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:04.183+0000 7ff8137fe640 1 -- 172.21.15.46:0/2814120497 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff81419cea0 con 0x7ff814101260 2024-03-31T23:15:04.188 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:04.183+0000 7ff81889f640 1 --2- 172.21.15.46:0/2814120497 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff8141055f0 0x7ff81419c770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T23:15:04.188 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:04.184+0000 7ff8137fe640 1 --2- 172.21.15.46:0/2814120497 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff814101260 0x7ff8141983f0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7ff7fc009f20 tx=0x7ff7fc00ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:15:04.188 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:04.184+0000 7ff8117fa640 1 -- 172.21.15.46:0/2814120497 <== mon.2 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7fc020070 con 0x7ff814101260 2024-03-31T23:15:04.188 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:04.184+0000 7ff81a302640 1 -- 172.21.15.46:0/2814120497 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff81419d190 con 0x7ff814101260 2024-03-31T23:15:04.188 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:04.184+0000 7ff8117fa640 1 -- 172.21.15.46:0/2814120497 <== mon.2 v2:172.21.15.142:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff7fc00cd80 con 0x7ff814101260 2024-03-31T23:15:04.188 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:04.184+0000 7ff8117fa640 1 -- 172.21.15.46:0/2814120497 <== mon.2 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7fc01bae0 con 0x7ff814101260 2024-03-31T23:15:04.188 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:04.184+0000 7ff81a302640 1 -- 172.21.15.46:0/2814120497 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff8141021f0 con 0x7ff814101260 2024-03-31T23:15:04.195 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:04.186+0000 7ff8117fa640 1 -- 172.21.15.46:0/2814120497 <== mon.2 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 25) ==== 137451+0+0 (secure 0 0 0) 0x7ff7fc023070 con 0x7ff814101260 2024-03-31T23:15:04.195 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:04.187+0000 7ff81cfbf740 1 Processor -- start 2024-03-31T23:15:04.195 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:04.187+0000 7ff81cfbf740 1 -- start start 2024-03-31T23:15:04.195 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:04.187+0000 7ff81cfbf740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a5575a56c0 0x55a5575a5aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:15:04.195 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:04.187+0000 7ff81cfbf740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a5575a5fe0 0x55a5575ae870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:15:04.196 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:04.187+0000 7ff81cfbf740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a5575aedb0 0x55a5575b11a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:15:04.196 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:04.187+0000 7ff81cfbf740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55a5575088f0 con 0x55a5575aedb0 2024-03-31T23:15:04.196 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:04.187+0000 7ff81cfbf740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55a5574b18d0 con 0x55a5575a5fe0 2024-03-31T23:15:04.196 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:04.187+0000 7ff81cfbf740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55a557504990 con 0x55a5575a56c0 2024-03-31T23:15:04.196 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:04.187+0000 7ff8137fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a5575aedb0 0x55a5575b11a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:15:04.196 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:04.187+0000 7ff813fff640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a5575a5fe0 0x55a5575ae870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:15:04.196 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:04.187+0000 7ff81889f640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a5575a56c0 0x55a5575a5aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:15:04.196 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:04.187+0000 7ff8137fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a5575aedb0 0x55a5575b11a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:47232/0 (socket says 172.21.15.46:47232) 2024-03-31T23:15:04.196 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:04.187+0000 7ff8137fe640 1 -- 172.21.15.46:0/3003217267 learned_addr learned my addr 172.21.15.46:0/3003217267 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:15:04.196 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:04.187+0000 7ff8117fa640 1 --2- 172.21.15.46:0/2814120497 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7ff8000a26b0 0x7ff8000a4b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:15:04.196 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:04.187+0000 7ff81889f640 1 --2- 172.21.15.46:0/2814120497 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7ff8000a26b0 0x7ff8000a4b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:15:04.196 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:04.187+0000 7ff813fff640 1 -- 172.21.15.46:0/3003217267 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a5575a56c0 msgr2=0x55a5575a5aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:15:04.196 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:04.187+0000 7ff813fff640 1 --2- 172.21.15.46:0/3003217267 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a5575a56c0 0x55a5575a5aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:15:04.196 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:04.187+0000 7ff813fff640 1 -- 172.21.15.46:0/3003217267 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a5575aedb0 msgr2=0x55a5575b11a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:15:04.197 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:04.187+0000 7ff813fff640 1 --2- 172.21.15.46:0/3003217267 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a5575aedb0 0x55a5575b11a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:15:04.197 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:04.187+0000 7ff813fff640 1 -- 172.21.15.46:0/3003217267 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a55721d210 con 0x55a5575a5fe0 2024-03-31T23:15:04.197 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:04.188+0000 7ff8117fa640 1 -- 172.21.15.46:0/2814120497 <== mon.2 v2:172.21.15.142:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7ff7fc0d2fc0 con 0x7ff814101260 2024-03-31T23:15:04.197 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:04.188+0000 7ff813fff640 1 --2- 172.21.15.46:0/3003217267 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a5575a5fe0 0x55a5575ae870 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7ff804009c00 tx=0x7ff804011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8084e0cba5736cb2 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:15:04.197 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:04.188+0000 7ff81889f640 1 --2- 172.21.15.46:0/2814120497 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7ff8000a26b0 0x7ff8000a4b70 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7ff80c009430 tx=0x7ff80c02c0f0 comp rx=0 tx=0).ready entity=mgr.15654 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:15:04.197 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:04.188+0000 7ff80a7fc640 1 -- 172.21.15.46:0/3003217267 <== mon.2 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff804004230 con 0x55a5575a5fe0 2024-03-31T23:15:04.197 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:04.188+0000 7ff80a7fc640 1 -- 172.21.15.46:0/3003217267 <== mon.2 v2:172.21.15.142:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff8040043d0 con 0x55a5575a5fe0 2024-03-31T23:15:04.197 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:04.188+0000 7ff80a7fc640 1 -- 172.21.15.46:0/3003217267 <== mon.2 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff804004c60 con 0x55a5575a5fe0 2024-03-31T23:15:04.197 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:04.189+0000 7ff81cfbf740 1 -- 172.21.15.46:0/3003217267 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a5575a5fe0 msgr2=0x55a5575ae870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:15:04.197 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:04.189+0000 7ff81cfbf740 1 --2- 172.21.15.46:0/3003217267 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a5575a5fe0 0x55a5575ae870 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7ff804009c00 tx=0x7ff804011810 comp rx=0 tx=0).stop 2024-03-31T23:15:04.197 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:04.189+0000 7ff81cfbf740 1 -- 172.21.15.46:0/3003217267 shutdown_connections 2024-03-31T23:15:04.197 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:04.189+0000 7ff81cfbf740 1 --2- 172.21.15.46:0/3003217267 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a5575aedb0 0x55a5575b11a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:15:04.197 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:04.189+0000 7ff81cfbf740 1 --2- 172.21.15.46:0/3003217267 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a5575a5fe0 0x55a5575ae870 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:15:04.198 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:04.189+0000 7ff81cfbf740 1 --2- 172.21.15.46:0/3003217267 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a5575a56c0 0x55a5575a5aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:15:04.198 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:04.189+0000 7ff81cfbf740 1 -- 172.21.15.46:0/3003217267 >> 172.21.15.46:0/3003217267 conn(0x55a55759ffe0 msgr2=0x55a5575a03c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:15:04.198 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:04.189+0000 7ff81cfbf740 1 -- 172.21.15.46:0/3003217267 shutdown_connections 2024-03-31T23:15:04.198 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:04.189+0000 7ff81cfbf740 1 -- 172.21.15.46:0/3003217267 wait complete. 2024-03-31T23:15:04.198 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:04.189+0000 7ff81cfbf740 1 -- start start 2024-03-31T23:15:04.198 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:04.189+0000 7ff81cfbf740 1 Processor -- start 2024-03-31T23:15:04.198 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:04.189+0000 7ff81cfbf740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a5575a56c0 0x55a5575abe40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:15:04.198 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:04.189+0000 7ff81cfbf740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a5575a5fe0 0x55a5575ac380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:15:04.198 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:04.189+0000 7ff81cfbf740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a5575aedb0 0x55a5575ac8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:15:04.198 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:04.190+0000 7ff81cfbf740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55a5575b19e0 con 0x55a5575aedb0 2024-03-31T23:15:04.198 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:04.190+0000 7ff81cfbf740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55a5574a6f80 con 0x55a5575a5fe0 2024-03-31T23:15:04.198 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:04.190+0000 7ff81cfbf740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55a5575b1b60 con 0x55a5575a56c0 2024-03-31T23:15:04.198 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:04.190+0000 7ff81cfbf740 10 client.0 _ll_register_callbacks cb 0x55a5574ff290 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T23:15:04.199 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:04.190+0000 7ff81889f640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a5575aedb0 0x55a5575ac8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:15:04.199 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:04.190+0000 7ff81889f640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a5575aedb0 0x55a5575ac8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:47238/0 (socket says 172.21.15.46:47238) 2024-03-31T23:15:04.199 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:04.190+0000 7ff81889f640 1 -- 172.21.15.46:0/1306485154 learned_addr learned my addr 172.21.15.46:0/1306485154 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:15:04.199 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:04.190+0000 7ff8137fe640 1 --2- 172.21.15.46:0/1306485154 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a5575a5fe0 0x55a5575ac380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:15:04.199 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:04.190+0000 7ff813fff640 1 --2- 172.21.15.46:0/1306485154 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a5575a56c0 0x55a5575abe40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:15:04.199 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:04.190+0000 7ff81889f640 1 -- 172.21.15.46:0/1306485154 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a5575a56c0 msgr2=0x55a5575abe40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:15:04.199 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:04.190+0000 7ff81889f640 1 --2- 172.21.15.46:0/1306485154 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a5575a56c0 0x55a5575abe40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:15:04.199 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:04.190+0000 7ff81889f640 1 -- 172.21.15.46:0/1306485154 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a5575a5fe0 msgr2=0x55a5575ac380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:15:04.199 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:04.190+0000 7ff81889f640 1 --2- 172.21.15.46:0/1306485154 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a5575a5fe0 0x55a5575ac380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:15:04.199 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:04.190+0000 7ff81889f640 1 -- 172.21.15.46:0/1306485154 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a5572e9620 con 0x55a5575aedb0 2024-03-31T23:15:04.199 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:04.190+0000 7ff8137fe640 1 --2- 172.21.15.46:0/1306485154 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a5575a5fe0 0x55a5575ac380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:15:04.199 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:04.190+0000 7ff81889f640 1 --2- 172.21.15.46:0/1306485154 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a5575aedb0 0x55a5575ac8c0 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7ff80c036000 tx=0x7ff80c0095c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a5e3679e51c48505 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:15:04.199 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:04.190+0000 7ff7f37fe640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-31T23:15:04.199 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:04.190+0000 7ff7f37fe640 1 -- 172.21.15.46:0/1306485154 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff80c039040 con 0x55a5575aedb0 2024-03-31T23:15:04.200 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:04.190+0000 7ff7f37fe640 1 -- 172.21.15.46:0/1306485154 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7ff80c004250 con 0x55a5575aedb0 2024-03-31T23:15:04.200 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:04.190+0000 7ff81cfbf740 1 -- 172.21.15.46:0/1306485154 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff8040043d0 con 0x55a5575aedb0 2024-03-31T23:15:04.200 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:04.191+0000 7ff7f37fe640 1 -- 172.21.15.46:0/1306485154 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7ff80c059050 con 0x55a5575aedb0 2024-03-31T23:15:04.200 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:04.191+0000 7ff81cfbf740 10 client.16419 fetch_fsmap learned FSMap version 28 2024-03-31T23:15:04.200 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:04.191+0000 7ff81cfbf740 1 -- 172.21.15.46:0/1306485154 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55a5575bd910 con 0x55a5575aedb0 2024-03-31T23:15:04.200 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:04.191+0000 7ff7f37fe640 1 -- 172.21.15.46:0/1306485154 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff8040043d0 con 0x55a5575aedb0 2024-03-31T23:15:04.200 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:04.191+0000 7ff7f37fe640 1 -- 172.21.15.46:0/1306485154 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7ff80c034890 con 0x55a5575aedb0 2024-03-31T23:15:04.200 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:04.191+0000 7ff81cfbf740 10 client.16419 fetch_fsmap finished waiting for FSMap version 28 2024-03-31T23:15:04.200 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:04.191+0000 7ff81cfbf740 10 client.16419 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-31T23:15:04.200 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:04.191+0000 7ff81cfbf740 20 client.16419 populate_metadata read hostname 'smithi046' 2024-03-31T23:15:04.200 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:04.191+0000 7ff81cfbf740 1 --2- 172.21.15.46:0/1306485154 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55a5575bee90 0x55a5575c12a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:15:04.200 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:04.191+0000 7ff81cfbf740 4 client.16419 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-31T23:15:04.200 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:04.191+0000 7ff81cfbf740 1 -- 172.21.15.46:0/1306485154 --> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a55751c120 con 0x55a5575bee90 2024-03-31T23:15:04.200 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:04.191+0000 7ff813fff640 1 --2- 172.21.15.46:0/1306485154 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55a5575bee90 0x55a5575c12a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:15:04.201 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:04.192+0000 7ff7f37fe640 10 client.16419 ms_handle_connect on v2:172.21.15.142:6838/2519143918 2024-03-31T23:15:04.201 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:04.192+0000 7ff813fff640 1 --2- 172.21.15.46:0/1306485154 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55a5575bee90 0x55a5575c12a0 crc :-1 s=READY pgs=103 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=143ddaca80d73473 server_cookie=6939df9c08292b46 in_seq=0 out_seq=0 2024-03-31T23:15:04.201 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:04.192+0000 7ff7f37fe640 1 -- 172.21.15.46:0/1306485154 <== mds.0 v2:172.21.15.142:6838/2519143918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55a55751c120 con 0x55a5575bee90 2024-03-31T23:15:04.201 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:04.192+0000 7ff7f37fe640 10 client.16419 handle_command_reply: tid=0 2024-03-31T23:15:04.201 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:04.192+0000 7ff81cfbf740 1 client.16419 shutdown 2024-03-31T23:15:04.201 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:04.192+0000 7ff81cfbf740 1 -- 172.21.15.46:0/1306485154 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a5575aedb0 msgr2=0x55a5575ac8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:15:04.201 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:04.192+0000 7ff81cfbf740 1 --2- 172.21.15.46:0/1306485154 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a5575aedb0 0x55a5575ac8c0 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7ff80c036000 tx=0x7ff80c0095c0 comp rx=0 tx=0).stop 2024-03-31T23:15:04.201 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:04.192+0000 7ff81cfbf740 1 -- 172.21.15.46:0/1306485154 shutdown_connections 2024-03-31T23:15:04.201 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:04.192+0000 7ff81cfbf740 1 --2- 172.21.15.46:0/1306485154 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55a5575bee90 0x55a5575c12a0 crc :-1 s=READY pgs=103 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:15:04.201 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:04.192+0000 7ff81cfbf740 1 --2- 172.21.15.46:0/1306485154 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a5575aedb0 0x55a5575ac8c0 unknown :-1 s=CLOSED pgs=670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:15:04.201 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:04.192+0000 7ff81cfbf740 1 --2- 172.21.15.46:0/1306485154 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a5575a5fe0 0x55a5575ac380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:15:04.201 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:04.192+0000 7ff81cfbf740 1 --2- 172.21.15.46:0/1306485154 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a5575a56c0 0x55a5575abe40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:15:04.201 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:04.192+0000 7ff81cfbf740 1 -- 172.21.15.46:0/1306485154 >> 172.21.15.46:0/1306485154 conn(0x55a55759ffe0 msgr2=0x55a5575a1d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:15:04.201 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:04.192+0000 7ff7f37fe640 0 client.16419 ms_handle_reset on v2:172.21.15.142:6838/2519143918 2024-03-31T23:15:04.202 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:04.193+0000 7ff81cfbf740 1 -- 172.21.15.46:0/1306485154 shutdown_connections 2024-03-31T23:15:04.202 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:04.193+0000 7ff81cfbf740 1 -- 172.21.15.46:0/1306485154 wait complete. 2024-03-31T23:15:04.202 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:04.193+0000 7ff81cfbf740 20 client.16419 trim_cache size 0 max 16384 2024-03-31T23:15:04.202 INFO:tasks.workunit.client.0.smithi046.stdout: 1 59229 2.55 MB/sec execute 10 sec latency 15.741 ms 2024-03-31T23:15:04.211 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:04.208+0000 7ff81cfbf740 1 Processor -- start 2024-03-31T23:15:04.211 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:04.208+0000 7ff81cfbf740 1 -- start start 2024-03-31T23:15:04.211 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:04.208+0000 7ff81cfbf740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a5575a56c0 0x55a5575ae570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:15:04.211 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:04.208+0000 7ff81cfbf740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a5575a5fe0 0x55a5575ba780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:15:04.211 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:04.208+0000 7ff81cfbf740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a5575aedb0 0x55a55759ee00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:15:04.211 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:04.208+0000 7ff81cfbf740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7ff8141af110 con 0x55a5575aedb0 2024-03-31T23:15:04.212 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:04.208+0000 7ff81cfbf740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7ff8141ac9e0 con 0x55a5575a5fe0 2024-03-31T23:15:04.212 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:04.208+0000 7ff81cfbf740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7ff8141aa830 con 0x55a5575a56c0 2024-03-31T23:15:04.212 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:04.208+0000 7ff81889f640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a5575aedb0 0x55a55759ee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:15:04.212 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:04.208+0000 7ff8137fe640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a5575a5fe0 0x55a5575ba780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:15:04.212 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:04.208+0000 7ff813fff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a5575a56c0 0x55a5575ae570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:15:04.212 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:04.208+0000 7ff81889f640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a5575aedb0 0x55a55759ee00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:47242/0 (socket says 172.21.15.46:47242) 2024-03-31T23:15:04.212 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:04.208+0000 7ff8137fe640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a5575a5fe0 0x55a5575ba780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.46:52820/0 (socket says 172.21.15.46:52820) 2024-03-31T23:15:04.212 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:04.208+0000 7ff813fff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a5575a56c0 0x55a5575ae570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.46:42618/0 (socket says 172.21.15.46:42618) 2024-03-31T23:15:04.212 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:04.208+0000 7ff81889f640 1 -- 172.21.15.46:0/852803921 learned_addr learned my addr 172.21.15.46:0/852803921 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:15:04.212 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:04.209+0000 7ff8137fe640 1 -- 172.21.15.46:0/852803921 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a5575a56c0 msgr2=0x55a5575ae570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:15:04.212 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:04.209+0000 7ff8137fe640 1 --2- 172.21.15.46:0/852803921 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a5575a56c0 0x55a5575ae570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:15:04.212 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:04.209+0000 7ff8137fe640 1 -- 172.21.15.46:0/852803921 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a5575aedb0 msgr2=0x55a55759ee00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:15:04.212 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:04.209+0000 7ff8137fe640 1 --2- 172.21.15.46:0/852803921 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a5575aedb0 0x55a55759ee00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:15:04.213 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:04.209+0000 7ff8137fe640 1 -- 172.21.15.46:0/852803921 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a5575aa3b0 con 0x55a5575a5fe0 2024-03-31T23:15:04.213 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:04.209+0000 7ff8137fe640 1 --2- 172.21.15.46:0/852803921 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a5575a5fe0 0x55a5575ba780 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7ff7fc0a7e20 tx=0x7ff7fc08a040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5e465fdd436775d4 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:15:04.213 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:04.209+0000 7ff809ffb640 1 -- 172.21.15.46:0/852803921 <== mon.2 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7fc09eeb0 con 0x55a5575a5fe0 2024-03-31T23:15:04.213 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:04.209+0000 7ff809ffb640 1 -- 172.21.15.46:0/852803921 <== mon.2 v2:172.21.15.142:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff7fc013da0 con 0x55a5575a5fe0 2024-03-31T23:15:04.213 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:04.209+0000 7ff809ffb640 1 -- 172.21.15.46:0/852803921 <== mon.2 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7fc09f090 con 0x55a5575a5fe0 2024-03-31T23:15:04.213 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:04.210+0000 7ff81cfbf740 1 -- 172.21.15.46:0/852803921 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a5575a5fe0 msgr2=0x55a5575ba780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:15:04.213 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:04.210+0000 7ff81cfbf740 1 --2- 172.21.15.46:0/852803921 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a5575a5fe0 0x55a5575ba780 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7ff7fc0a7e20 tx=0x7ff7fc08a040 comp rx=0 tx=0).stop 2024-03-31T23:15:04.213 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:04.210+0000 7ff81cfbf740 1 -- 172.21.15.46:0/852803921 shutdown_connections 2024-03-31T23:15:04.213 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:04.210+0000 7ff81cfbf740 1 --2- 172.21.15.46:0/852803921 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a5575aedb0 0x55a55759ee00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:15:04.213 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:04.210+0000 7ff81cfbf740 1 --2- 172.21.15.46:0/852803921 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a5575a5fe0 0x55a5575ba780 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:15:04.213 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:04.210+0000 7ff81cfbf740 1 --2- 172.21.15.46:0/852803921 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a5575a56c0 0x55a5575ae570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:15:04.213 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:04.210+0000 7ff81cfbf740 1 -- 172.21.15.46:0/852803921 >> 172.21.15.46:0/852803921 conn(0x55a55759ffe0 msgr2=0x55a55759e6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:15:04.213 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:04.210+0000 7ff81cfbf740 1 -- 172.21.15.46:0/852803921 shutdown_connections 2024-03-31T23:15:04.213 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:04.210+0000 7ff81cfbf740 1 -- 172.21.15.46:0/852803921 wait complete. 2024-03-31T23:15:04.213 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:04.210+0000 7ff81cfbf740 1 -- start start 2024-03-31T23:15:04.214 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:04.210+0000 7ff81cfbf740 1 Processor -- start 2024-03-31T23:15:04.217 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:04.211+0000 7ff81cfbf740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a5575a56c0 0x55a5575a8da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:15:04.217 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:04.211+0000 7ff81cfbf740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a5575a5fe0 0x55a5575a92e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:15:04.217 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:04.211+0000 7ff81cfbf740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a5575aedb0 0x55a5576241e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:15:04.217 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:04.211+0000 7ff81cfbf740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7ff8141aaea0 con 0x55a5575a56c0 2024-03-31T23:15:04.217 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:04.211+0000 7ff81cfbf740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7ff8141ac310 con 0x55a5575aedb0 2024-03-31T23:15:04.217 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:04.211+0000 7ff81cfbf740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7ff8141ab570 con 0x55a5575a5fe0 2024-03-31T23:15:04.217 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:04.211+0000 7ff81cfbf740 10 client.0 _ll_register_callbacks cb 0x7ff80c037070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T23:15:04.217 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:04.211+0000 7ff813fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a5575a56c0 0x55a5575a8da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:15:04.218 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:04.211+0000 7ff813fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a5575a56c0 0x55a5575a8da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:47252/0 (socket says 172.21.15.46:47252) 2024-03-31T23:15:04.218 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:04.211+0000 7ff813fff640 1 -- 172.21.15.46:0/4263145932 learned_addr learned my addr 172.21.15.46:0/4263145932 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:15:04.218 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:04.211+0000 7ff81889f640 1 --2- 172.21.15.46:0/4263145932 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a5575aedb0 0x55a5576241e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:15:04.218 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:04.211+0000 7ff8137fe640 1 --2- 172.21.15.46:0/4263145932 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a5575a5fe0 0x55a5575a92e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:15:04.218 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:04.211+0000 7ff813fff640 1 -- 172.21.15.46:0/4263145932 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a5575a5fe0 msgr2=0x55a5575a92e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:15:04.218 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:04.211+0000 7ff813fff640 1 --2- 172.21.15.46:0/4263145932 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a5575a5fe0 0x55a5575a92e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:15:04.218 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:04.211+0000 7ff813fff640 1 -- 172.21.15.46:0/4263145932 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a5575aedb0 msgr2=0x55a5576241e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:15:04.218 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:04.211+0000 7ff813fff640 1 --2- 172.21.15.46:0/4263145932 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a5575aedb0 0x55a5576241e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:15:04.218 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:04.211+0000 7ff813fff640 1 -- 172.21.15.46:0/4263145932 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a557624720 con 0x55a5575a56c0 2024-03-31T23:15:04.219 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:04.211+0000 7ff81889f640 1 --2- 172.21.15.46:0/4263145932 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a5575aedb0 0x55a5576241e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:15:04.219 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:04.211+0000 7ff813fff640 1 --2- 172.21.15.46:0/4263145932 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a5575a56c0 0x55a5575a8da0 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7ff8040058c0 tx=0x7ff804005d30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eed7b10569f2c338 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:15:04.219 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:04.211+0000 7ff7f37fe640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-31T23:15:04.219 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:04.211+0000 7ff7f37fe640 1 -- 172.21.15.46:0/4263145932 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff80401e040 con 0x55a5575a56c0 2024-03-31T23:15:04.219 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:04.212+0000 7ff7f37fe640 1 -- 172.21.15.46:0/4263145932 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7ff8040035f0 con 0x55a5575a56c0 2024-03-31T23:15:04.219 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:04.212+0000 7ff81cfbf740 1 -- 172.21.15.46:0/4263145932 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55a557515e30 con 0x55a5575a56c0 2024-03-31T23:15:04.219 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:04.212+0000 7ff7f37fe640 1 -- 172.21.15.46:0/4263145932 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7ff804005590 con 0x55a5575a56c0 2024-03-31T23:15:04.219 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:04.212+0000 7ff81cfbf740 10 client.16425 fetch_fsmap learned FSMap version 28 2024-03-31T23:15:04.219 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:04.212+0000 7ff81cfbf740 1 -- 172.21.15.46:0/4263145932 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55a557625890 con 0x55a5575a56c0 2024-03-31T23:15:04.219 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:04.212+0000 7ff7f37fe640 1 -- 172.21.15.46:0/4263145932 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x55a557515e30 con 0x55a5575a56c0 2024-03-31T23:15:04.219 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:04.212+0000 7ff7f37fe640 1 -- 172.21.15.46:0/4263145932 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7ff804024040 con 0x55a5575a56c0 2024-03-31T23:15:04.219 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:04.212+0000 7ff81cfbf740 10 client.16425 fetch_fsmap finished waiting for FSMap version 28 2024-03-31T23:15:04.219 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:04.212+0000 7ff81cfbf740 10 client.16425 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-31T23:15:04.220 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:04.212+0000 7ff81cfbf740 20 client.16425 populate_metadata read hostname 'smithi046' 2024-03-31T23:15:04.220 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:04.212+0000 7ff81cfbf740 1 --2- 172.21.15.46:0/4263145932 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55a557625e20 0x55a557628240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:15:04.220 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:04.212+0000 7ff81cfbf740 4 client.16425 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-31T23:15:04.220 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:04.212+0000 7ff81cfbf740 1 -- 172.21.15.46:0/4263145932 --> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55a557510180 con 0x55a557625e20 2024-03-31T23:15:04.220 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:04.212+0000 7ff8137fe640 1 --2- 172.21.15.46:0/4263145932 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55a557625e20 0x55a557628240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:15:04.220 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:04.213+0000 7ff7f37fe640 10 client.16425 ms_handle_connect on v2:172.21.15.142:6838/2519143918 2024-03-31T23:15:04.220 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:04.213+0000 7ff8137fe640 1 --2- 172.21.15.46:0/4263145932 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55a557625e20 0x55a557628240 crc :-1 s=READY pgs=104 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5a80ab0200b4d802 server_cookie=6e1cf1e4a42792be in_seq=0 out_seq=0 2024-03-31T23:15:04.233 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:04.231+0000 7ff7f37fe640 1 -- 172.21.15.46:0/4263145932 <== mds.0 v2:172.21.15.142:6838/2519143918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55a557510180 con 0x55a557625e20 2024-03-31T23:15:04.233 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:04.231+0000 7ff7f37fe640 10 client.16425 handle_command_reply: tid=0 2024-03-31T23:15:04.233 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:04.231+0000 7ff81cfbf740 1 client.16425 shutdown 2024-03-31T23:15:04.234 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:04.231+0000 7ff81cfbf740 1 -- 172.21.15.46:0/4263145932 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a5575a56c0 msgr2=0x55a5575a8da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:15:04.234 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:04.231+0000 7ff81cfbf740 1 --2- 172.21.15.46:0/4263145932 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a5575a56c0 0x55a5575a8da0 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7ff8040058c0 tx=0x7ff804005d30 comp rx=0 tx=0).stop 2024-03-31T23:15:04.234 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:04.231+0000 7ff81cfbf740 1 -- 172.21.15.46:0/4263145932 shutdown_connections 2024-03-31T23:15:04.234 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:04.231+0000 7ff81cfbf740 1 --2- 172.21.15.46:0/4263145932 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55a557625e20 0x55a557628240 crc :-1 s=READY pgs=104 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:15:04.234 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:04.231+0000 7ff81cfbf740 1 --2- 172.21.15.46:0/4263145932 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a5575aedb0 0x55a5576241e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:15:04.234 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:04.231+0000 7ff81cfbf740 1 --2- 172.21.15.46:0/4263145932 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a5575a5fe0 0x55a5575a92e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:15:04.234 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:04.231+0000 7ff81cfbf740 1 --2- 172.21.15.46:0/4263145932 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a5575a56c0 0x55a5575a8da0 unknown :-1 s=CLOSED pgs=671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:15:04.234 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:04.232+0000 7ff81cfbf740 1 -- 172.21.15.46:0/4263145932 >> 172.21.15.46:0/4263145932 conn(0x55a55759ffe0 msgr2=0x55a55759ef80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:15:04.234 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:04.232+0000 7ff7f37fe640 0 client.16425 ms_handle_reset on v2:172.21.15.142:6838/2519143918 2024-03-31T23:15:04.234 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:04.232+0000 7ff81cfbf740 1 -- 172.21.15.46:0/4263145932 shutdown_connections 2024-03-31T23:15:04.234 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:04.232+0000 7ff81cfbf740 1 -- 172.21.15.46:0/4263145932 wait complete. 2024-03-31T23:15:04.234 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:04.232+0000 7ff81cfbf740 20 client.16425 trim_cache size 0 max 16384 2024-03-31T23:15:04.235 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout:{ 2024-03-31T23:15:04.235 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout: "message": "", 2024-03-31T23:15:04.235 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout: "return_code": 0 2024-03-31T23:15:04.235 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout:} 2024-03-31T23:15:04.236 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:04.232+0000 7ff81a302640 1 -- 172.21.15.46:0/2814120497 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7ff8000a26b0 msgr2=0x7ff8000a4b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:15:04.236 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:04.232+0000 7ff81a302640 1 --2- 172.21.15.46:0/2814120497 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7ff8000a26b0 0x7ff8000a4b70 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7ff80c009430 tx=0x7ff80c02c0f0 comp rx=0 tx=0).stop 2024-03-31T23:15:04.236 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:04.232+0000 7ff81a302640 1 -- 172.21.15.46:0/2814120497 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff814101260 msgr2=0x7ff8141983f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:15:04.236 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:04.232+0000 7ff81a302640 1 --2- 172.21.15.46:0/2814120497 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff814101260 0x7ff8141983f0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7ff7fc009f20 tx=0x7ff7fc00ee80 comp rx=0 tx=0).stop 2024-03-31T23:15:04.236 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:04.232+0000 7ff81a302640 1 -- 172.21.15.46:0/2814120497 shutdown_connections 2024-03-31T23:15:04.236 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:04.232+0000 7ff81a302640 1 --2- 172.21.15.46:0/2814120497 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7ff8000a26b0 0x7ff8000a4b70 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:15:04.236 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:04.232+0000 7ff81a302640 1 --2- 172.21.15.46:0/2814120497 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff8141055f0 0x7ff81419c770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:15:04.236 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:04.232+0000 7ff81a302640 1 --2- 172.21.15.46:0/2814120497 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff814101260 0x7ff8141983f0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:15:04.236 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:04.232+0000 7ff81a302640 1 --2- 172.21.15.46:0/2814120497 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff8141008a0 0x7ff814197eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:15:04.236 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:04.232+0000 7ff81a302640 1 -- 172.21.15.46:0/2814120497 >> 172.21.15.46:0/2814120497 conn(0x7ff8140774a0 msgr2=0x7ff8140690b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:15:04.236 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:04.233+0000 7ff81a302640 1 -- 172.21.15.46:0/2814120497 shutdown_connections 2024-03-31T23:15:04.236 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:04.233+0000 7ff81a302640 1 -- 172.21.15.46:0/2814120497 wait complete. 2024-03-31T23:15:04.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:15:04 smithi046 ceph-mon[32498]: pgmap v103: 129 pgs: 129 active+clean; 51 MiB data, 655 MiB used, 1.0 TiB / 1.0 TiB avail; 272 KiB/s rd, 1.3 MiB/s wr, 115 op/s 2024-03-31T23:15:04.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:15:04 smithi142 ceph-mon[35271]: pgmap v103: 129 pgs: 129 active+clean; 51 MiB data, 655 MiB used, 1.0 TiB / 1.0 TiB avail; 272 KiB/s rd, 1.3 MiB/s wr, 115 op/s 2024-03-31T23:15:05.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:15:04 smithi162 ceph-mon[36237]: pgmap v103: 129 pgs: 129 active+clean; 51 MiB data, 655 MiB used, 1.0 TiB / 1.0 TiB avail; 272 KiB/s rd, 1.3 MiB/s wr, 115 op/s 2024-03-31T23:15:05.194 INFO:tasks.workunit.client.0.smithi046.stdout: 1 59975 2.88 MB/sec execute 11 sec latency 18.163 ms 2024-03-31T23:15:06.194 INFO:tasks.workunit.client.0.smithi046.stdout: 1 60356 2.98 MB/sec execute 12 sec latency 16.884 ms 2024-03-31T23:15:06.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:15:06 smithi046 ceph-mon[32498]: pgmap v104: 129 pgs: 129 active+clean; 54 MiB data, 698 MiB used, 1.0 TiB / 1.0 TiB avail; 454 KiB/s rd, 1.7 MiB/s wr, 192 op/s 2024-03-31T23:15:06.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:15:06 smithi142 ceph-mon[35271]: pgmap v104: 129 pgs: 129 active+clean; 54 MiB data, 698 MiB used, 1.0 TiB / 1.0 TiB avail; 454 KiB/s rd, 1.7 MiB/s wr, 192 op/s 2024-03-31T23:15:07.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:15:06 smithi162 ceph-mon[36237]: pgmap v104: 129 pgs: 129 active+clean; 54 MiB data, 698 MiB used, 1.0 TiB / 1.0 TiB avail; 454 KiB/s rd, 1.7 MiB/s wr, 192 op/s 2024-03-31T23:15:07.194 INFO:tasks.workunit.client.0.smithi046.stdout: 1 60702 2.88 MB/sec execute 13 sec latency 15.595 ms 2024-03-31T23:15:08.194 INFO:tasks.workunit.client.0.smithi046.stdout: 1 60996 2.68 MB/sec execute 14 sec latency 16.448 ms 2024-03-31T23:15:08.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:15:08 smithi046 ceph-mon[32498]: pgmap v105: 129 pgs: 129 active+clean; 54 MiB data, 698 MiB used, 1.0 TiB / 1.0 TiB avail; 306 KiB/s rd, 1.1 MiB/s wr, 133 op/s 2024-03-31T23:15:08.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:15:08 smithi142 ceph-mon[35271]: pgmap v105: 129 pgs: 129 active+clean; 54 MiB data, 698 MiB used, 1.0 TiB / 1.0 TiB avail; 306 KiB/s rd, 1.1 MiB/s wr, 133 op/s 2024-03-31T23:15:09.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:15:08 smithi162 ceph-mon[36237]: pgmap v105: 129 pgs: 129 active+clean; 54 MiB data, 698 MiB used, 1.0 TiB / 1.0 TiB avail; 306 KiB/s rd, 1.1 MiB/s wr, 133 op/s 2024-03-31T23:15:09.194 INFO:tasks.workunit.client.0.smithi046.stdout: 1 61492 2.63 MB/sec execute 15 sec latency 18.451 ms 2024-03-31T23:15:10.194 INFO:tasks.workunit.client.0.smithi046.stdout: 1 62034 2.67 MB/sec execute 16 sec latency 15.317 ms 2024-03-31T23:15:10.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:15:10 smithi046 ceph-mon[32498]: pgmap v106: 129 pgs: 129 active+clean; 59 MiB data, 721 MiB used, 1.0 TiB / 1.0 TiB avail; 484 KiB/s rd, 1.8 MiB/s wr, 197 op/s 2024-03-31T23:15:10.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:15:10 smithi142 ceph-mon[35271]: pgmap v106: 129 pgs: 129 active+clean; 59 MiB data, 721 MiB used, 1.0 TiB / 1.0 TiB avail; 484 KiB/s rd, 1.8 MiB/s wr, 197 op/s 2024-03-31T23:15:11.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:15:10 smithi162 ceph-mon[36237]: pgmap v106: 129 pgs: 129 active+clean; 59 MiB data, 721 MiB used, 1.0 TiB / 1.0 TiB avail; 484 KiB/s rd, 1.8 MiB/s wr, 197 op/s 2024-03-31T23:15:11.194 INFO:tasks.workunit.client.0.smithi046.stdout: 1 62638 2.60 MB/sec execute 17 sec latency 18.866 ms 2024-03-31T23:15:12.194 INFO:tasks.workunit.client.0.smithi046.stdout: 1 63202 2.70 MB/sec execute 18 sec latency 16.910 ms 2024-03-31T23:15:12.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:15:12 smithi046 ceph-mon[32498]: pgmap v107: 129 pgs: 129 active+clean; 59 MiB data, 721 MiB used, 1.0 TiB / 1.0 TiB avail; 359 KiB/s rd, 1.2 MiB/s wr, 140 op/s 2024-03-31T23:15:12.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:15:12 smithi142 ceph-mon[35271]: pgmap v107: 129 pgs: 129 active+clean; 59 MiB data, 721 MiB used, 1.0 TiB / 1.0 TiB avail; 359 KiB/s rd, 1.2 MiB/s wr, 140 op/s 2024-03-31T23:15:13.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:15:12 smithi162 ceph-mon[36237]: pgmap v107: 129 pgs: 129 active+clean; 59 MiB data, 721 MiB used, 1.0 TiB / 1.0 TiB avail; 359 KiB/s rd, 1.2 MiB/s wr, 140 op/s 2024-03-31T23:15:13.194 INFO:tasks.workunit.client.0.smithi046.stdout: 1 63658 2.73 MB/sec execute 19 sec latency 17.738 ms 2024-03-31T23:15:14.194 INFO:tasks.workunit.client.0.smithi046.stdout: 1 64138 2.81 MB/sec execute 20 sec latency 15.302 ms 2024-03-31T23:15:14.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:15:14 smithi046 ceph-mon[32498]: pgmap v108: 129 pgs: 129 active+clean; 59 MiB data, 721 MiB used, 1.0 TiB / 1.0 TiB avail; 359 KiB/s rd, 1.2 MiB/s wr, 140 op/s 2024-03-31T23:15:14.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:15:14 smithi142 ceph-mon[35271]: pgmap v108: 129 pgs: 129 active+clean; 59 MiB data, 721 MiB used, 1.0 TiB / 1.0 TiB avail; 359 KiB/s rd, 1.2 MiB/s wr, 140 op/s 2024-03-31T23:15:15.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:15:14 smithi162 ceph-mon[36237]: pgmap v108: 129 pgs: 129 active+clean; 59 MiB data, 721 MiB used, 1.0 TiB / 1.0 TiB avail; 359 KiB/s rd, 1.2 MiB/s wr, 140 op/s 2024-03-31T23:15:15.194 INFO:tasks.workunit.client.0.smithi046.stdout: 1 64415 2.69 MB/sec execute 21 sec latency 15.558 ms 2024-03-31T23:15:16.194 INFO:tasks.workunit.client.0.smithi046.stdout: 1 64816 2.61 MB/sec execute 22 sec latency 15.865 ms 2024-03-31T23:15:16.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:15:16 smithi046 ceph-mon[32498]: pgmap v109: 129 pgs: 129 active+clean; 60 MiB data, 770 MiB used, 1.0 TiB / 1.0 TiB avail; 501 KiB/s rd, 1.8 MiB/s wr, 195 op/s 2024-03-31T23:15:16.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:15:16 smithi142 ceph-mon[35271]: pgmap v109: 129 pgs: 129 active+clean; 60 MiB data, 770 MiB used, 1.0 TiB / 1.0 TiB avail; 501 KiB/s rd, 1.8 MiB/s wr, 195 op/s 2024-03-31T23:15:17.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:15:16 smithi162 ceph-mon[36237]: pgmap v109: 129 pgs: 129 active+clean; 60 MiB data, 770 MiB used, 1.0 TiB / 1.0 TiB avail; 501 KiB/s rd, 1.8 MiB/s wr, 195 op/s 2024-03-31T23:15:17.194 INFO:tasks.workunit.client.0.smithi046.stdout: 1 65380 2.68 MB/sec execute 23 sec latency 18.737 ms 2024-03-31T23:15:17.338 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:17.334+0000 7f254ddeb640 1 -- 172.21.15.46:0/1701011299 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f25481055f0 msgr2=0x7f25481059d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:15:17.339 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:17.334+0000 7f254ddeb640 1 --2- 172.21.15.46:0/1701011299 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f25481055f0 0x7f25481059d0 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f2538009900 tx=0x7f253802bc30 comp rx=0 tx=0).stop 2024-03-31T23:15:17.339 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:17.334+0000 7f254ddeb640 1 -- 172.21.15.46:0/1701011299 shutdown_connections 2024-03-31T23:15:17.339 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:17.334+0000 7f254ddeb640 1 --2- 172.21.15.46:0/1701011299 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2548101260 0x7f254810d480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:15:17.339 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:17.334+0000 7f254ddeb640 1 --2- 172.21.15.46:0/1701011299 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f25481008a0 0x7f2548100d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:15:17.339 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:17.334+0000 7f254ddeb640 1 --2- 172.21.15.46:0/1701011299 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f25481055f0 0x7f25481059d0 unknown :-1 s=CLOSED pgs=672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:15:17.339 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:17.334+0000 7f254ddeb640 1 -- 172.21.15.46:0/1701011299 >> 172.21.15.46:0/1701011299 conn(0x7f25480774a0 msgr2=0x7f25480778b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:15:17.339 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:17.334+0000 7f254ddeb640 1 -- 172.21.15.46:0/1701011299 shutdown_connections 2024-03-31T23:15:17.339 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:17.335+0000 7f254ddeb640 1 -- 172.21.15.46:0/1701011299 wait complete. 2024-03-31T23:15:17.339 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:17.335+0000 7f254ddeb640 1 Processor -- start 2024-03-31T23:15:17.340 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:17.335+0000 7f254ddeb640 1 -- start start 2024-03-31T23:15:17.340 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:17.335+0000 7f254ddeb640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f25481008a0 0x7f2548198d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:15:17.340 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:17.335+0000 7f254ddeb640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2548101260 0x7f2548199280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:15:17.340 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:17.335+0000 7f254ddeb640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f25481055f0 0x7f25481a3dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:15:17.340 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:17.335+0000 7f254ddeb640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f25481109c0 con 0x7f25481008a0 2024-03-31T23:15:17.340 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:17.335+0000 7f254ddeb640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f2548110840 con 0x7f2548101260 2024-03-31T23:15:17.340 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:17.335+0000 7f254ddeb640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f2548110b40 con 0x7f25481055f0 2024-03-31T23:15:17.340 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:17.336+0000 7f25477fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f25481008a0 0x7f2548198d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:15:17.340 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:17.336+0000 7f2546ffd640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2548101260 0x7f2548199280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:15:17.340 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:17.336+0000 7f25477fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f25481008a0 0x7f2548198d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39130/0 (socket says 172.21.15.46:39130) 2024-03-31T23:15:17.340 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:17.336+0000 7f2546ffd640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2548101260 0x7f2548199280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.46:45686/0 (socket says 172.21.15.46:45686) 2024-03-31T23:15:17.340 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:17.336+0000 7f25477fe640 1 -- 172.21.15.46:0/128446346 learned_addr learned my addr 172.21.15.46:0/128446346 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:15:17.340 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:17.336+0000 7f2547fff640 1 --2- 172.21.15.46:0/128446346 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f25481055f0 0x7f25481a3dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:15:17.341 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:17.336+0000 7f2547fff640 1 -- 172.21.15.46:0/128446346 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2548101260 msgr2=0x7f2548199280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:15:17.341 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:17.336+0000 7f2547fff640 1 --2- 172.21.15.46:0/128446346 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2548101260 0x7f2548199280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:15:17.341 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:17.336+0000 7f2547fff640 1 -- 172.21.15.46:0/128446346 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f25481008a0 msgr2=0x7f2548198d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:15:17.341 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:17.336+0000 7f2547fff640 1 --2- 172.21.15.46:0/128446346 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f25481008a0 0x7f2548198d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:15:17.341 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:17.336+0000 7f2547fff640 1 -- 172.21.15.46:0/128446346 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f25481a4300 con 0x7f25481055f0 2024-03-31T23:15:17.341 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:17.336+0000 7f25477fe640 1 --2- 172.21.15.46:0/128446346 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f25481008a0 0x7f2548198d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:15:17.341 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:17.336+0000 7f2546ffd640 1 --2- 172.21.15.46:0/128446346 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2548101260 0x7f2548199280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:15:17.341 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:17.336+0000 7f2547fff640 1 --2- 172.21.15.46:0/128446346 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f25481055f0 0x7f25481a3dc0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f2540005fd0 tx=0x7f254000ee60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:15:17.341 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:17.336+0000 7f2544ff9640 1 -- 172.21.15.46:0/128446346 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f254000cd80 con 0x7f25481055f0 2024-03-31T23:15:17.341 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:17.336+0000 7f2544ff9640 1 -- 172.21.15.46:0/128446346 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f2540005020 con 0x7f25481055f0 2024-03-31T23:15:17.341 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:17.336+0000 7f254ddeb640 1 -- 172.21.15.46:0/128446346 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f25481a4530 con 0x7f25481055f0 2024-03-31T23:15:17.341 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:17.336+0000 7f2544ff9640 1 -- 172.21.15.46:0/128446346 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2540010690 con 0x7f25481055f0 2024-03-31T23:15:17.341 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:17.336+0000 7f254ddeb640 1 -- 172.21.15.46:0/128446346 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f25481a4a50 con 0x7f25481055f0 2024-03-31T23:15:17.348 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:17.338+0000 7f2544ff9640 1 -- 172.21.15.46:0/128446346 <== mon.1 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 25) ==== 137451+0+0 (secure 0 0 0) 0x7f25400051d0 con 0x7f25481055f0 2024-03-31T23:15:17.348 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:17.339+0000 7f25509c5740 1 Processor -- start 2024-03-31T23:15:17.348 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:17.339+0000 7f25509c5740 1 -- start start 2024-03-31T23:15:17.348 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:17.339+0000 7f25509c5740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a3df6676c0 0x55a3df667aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:15:17.349 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:17.339+0000 7f25509c5740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a3df667fe0 0x55a3df670870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:15:17.349 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:17.339+0000 7f25509c5740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a3df670db0 0x55a3df6731a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:15:17.349 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:17.339+0000 7f25509c5740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55a3df5ca8f0 con 0x55a3df670db0 2024-03-31T23:15:17.349 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:17.339+0000 7f2544ff9640 1 --2- 172.21.15.46:0/128446346 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f25300a26b0 0x7f25300a4b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:15:17.349 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:17.339+0000 7f25509c5740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55a3df5738d0 con 0x55a3df6676c0 2024-03-31T23:15:17.349 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:17.339+0000 7f25509c5740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55a3df5c6990 con 0x55a3df667fe0 2024-03-31T23:15:17.349 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:17.339+0000 7f2546ffd640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a3df6676c0 0x55a3df667aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:15:17.349 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:17.339+0000 7f2546ffd640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a3df6676c0 0x55a3df667aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.46:45696/0 (socket says 172.21.15.46:45696) 2024-03-31T23:15:17.349 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:17.339+0000 7f2546ffd640 1 -- 172.21.15.46:0/2167004344 learned_addr learned my addr 172.21.15.46:0/2167004344 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:15:17.349 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:17.339+0000 7f25477fe640 1 --2- 172.21.15.46:0/2167004344 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a3df667fe0 0x55a3df670870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:15:17.349 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:17.339+0000 7f2547fff640 1 --2- 172.21.15.46:0/128446346 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f25300a26b0 0x7f25300a4b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:15:17.349 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:17.339+0000 7f2546ffd640 1 --2- 172.21.15.46:0/2167004344 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a3df670db0 0x55a3df6731a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:15:17.349 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:17.339+0000 7f2544ff9640 1 -- 172.21.15.46:0/128446346 <== mon.1 v2:172.21.15.162:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7f25400cf4a0 con 0x7f25481055f0 2024-03-31T23:15:17.350 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:17.340+0000 7f2546ffd640 1 -- 172.21.15.46:0/2167004344 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a3df667fe0 msgr2=0x55a3df670870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:15:17.350 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:17.340+0000 7f2546ffd640 1 --2- 172.21.15.46:0/2167004344 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a3df667fe0 0x55a3df670870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:15:17.350 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:17.340+0000 7f2546ffd640 1 -- 172.21.15.46:0/2167004344 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a3df670db0 msgr2=0x55a3df6731a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:15:17.350 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:17.340+0000 7f2546ffd640 1 --2- 172.21.15.46:0/2167004344 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a3df670db0 0x55a3df6731a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:15:17.350 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:17.340+0000 7f2546ffd640 1 -- 172.21.15.46:0/2167004344 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a3df2df210 con 0x55a3df6676c0 2024-03-31T23:15:17.350 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:17.340+0000 7f25477fe640 1 --2- 172.21.15.46:0/2167004344 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a3df667fe0 0x55a3df670870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:15:17.350 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:17.340+0000 7f2547fff640 1 --2- 172.21.15.46:0/128446346 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f25300a26b0 0x7f25300a4b70 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f25400cc8f0 tx=0x7f2540090d00 comp rx=0 tx=0).ready entity=mgr.15654 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:15:17.350 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:17.340+0000 7f2546ffd640 1 --2- 172.21.15.46:0/2167004344 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a3df6676c0 0x55a3df667aa0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f253c009fd0 tx=0x7f253c009ba0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=91861018aaff9165 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:15:17.350 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:17.340+0000 7f252dffb640 1 -- 172.21.15.46:0/2167004344 <== mon.2 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f253c00fc50 con 0x55a3df6676c0 2024-03-31T23:15:17.350 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:17.340+0000 7f252dffb640 1 -- 172.21.15.46:0/2167004344 <== mon.2 v2:172.21.15.142:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f253c00fdf0 con 0x55a3df6676c0 2024-03-31T23:15:17.350 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:17.340+0000 7f252dffb640 1 -- 172.21.15.46:0/2167004344 <== mon.2 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f253c00e640 con 0x55a3df6676c0 2024-03-31T23:15:17.350 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:17.341+0000 7f25509c5740 1 -- 172.21.15.46:0/2167004344 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a3df6676c0 msgr2=0x55a3df667aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:15:17.351 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:17.341+0000 7f25509c5740 1 --2- 172.21.15.46:0/2167004344 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a3df6676c0 0x55a3df667aa0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f253c009fd0 tx=0x7f253c009ba0 comp rx=0 tx=0).stop 2024-03-31T23:15:17.351 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:17.341+0000 7f25509c5740 1 -- 172.21.15.46:0/2167004344 shutdown_connections 2024-03-31T23:15:17.351 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:17.341+0000 7f25509c5740 1 --2- 172.21.15.46:0/2167004344 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a3df670db0 0x55a3df6731a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:15:17.351 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:17.341+0000 7f25509c5740 1 --2- 172.21.15.46:0/2167004344 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a3df667fe0 0x55a3df670870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:15:17.351 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:17.341+0000 7f25509c5740 1 --2- 172.21.15.46:0/2167004344 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a3df6676c0 0x55a3df667aa0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:15:17.351 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:17.341+0000 7f25509c5740 1 -- 172.21.15.46:0/2167004344 >> 172.21.15.46:0/2167004344 conn(0x55a3df661fe0 msgr2=0x55a3df6623c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:15:17.351 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:17.341+0000 7f25509c5740 1 -- 172.21.15.46:0/2167004344 shutdown_connections 2024-03-31T23:15:17.351 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:17.341+0000 7f25509c5740 1 -- 172.21.15.46:0/2167004344 wait complete. 2024-03-31T23:15:17.351 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:17.341+0000 7f25509c5740 1 -- start start 2024-03-31T23:15:17.351 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:17.341+0000 7f25509c5740 1 Processor -- start 2024-03-31T23:15:17.351 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:17.341+0000 7f25509c5740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a3df6676c0 0x55a3df66de50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:15:17.351 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:17.341+0000 7f25509c5740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a3df667fe0 0x55a3df66e390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:15:17.352 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:17.341+0000 7f25509c5740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a3df670db0 0x55a3df66e8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:15:17.352 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:17.341+0000 7f25509c5740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55a3df6739e0 con 0x55a3df6676c0 2024-03-31T23:15:17.352 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:17.341+0000 7f25509c5740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55a3df568f80 con 0x55a3df667fe0 2024-03-31T23:15:17.352 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:17.341+0000 7f25509c5740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55a3df673b60 con 0x55a3df670db0 2024-03-31T23:15:17.352 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:17.342+0000 7f25509c5740 10 client.0 _ll_register_callbacks cb 0x55a3df5c1290 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T23:15:17.352 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:17.342+0000 7f2546ffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a3df6676c0 0x55a3df66de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:15:17.352 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:17.342+0000 7f2546ffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a3df6676c0 0x55a3df66de50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39144/0 (socket says 172.21.15.46:39144) 2024-03-31T23:15:17.352 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:17.342+0000 7f2546ffd640 1 -- 172.21.15.46:0/2602915079 learned_addr learned my addr 172.21.15.46:0/2602915079 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:15:17.352 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:17.342+0000 7f25477fe640 1 --2- 172.21.15.46:0/2602915079 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a3df667fe0 0x55a3df66e390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:15:17.352 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:17.342+0000 7f2546ffd640 1 --2- 172.21.15.46:0/2602915079 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a3df670db0 0x55a3df66e8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:15:17.352 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:17.342+0000 7f2546ffd640 1 -- 172.21.15.46:0/2602915079 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a3df670db0 msgr2=0x55a3df66e8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:15:17.352 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:17.342+0000 7f2546ffd640 1 --2- 172.21.15.46:0/2602915079 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a3df670db0 0x55a3df66e8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:15:17.352 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:17.342+0000 7f2546ffd640 1 -- 172.21.15.46:0/2602915079 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a3df667fe0 msgr2=0x55a3df66e390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:15:17.352 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:17.342+0000 7f2546ffd640 1 --2- 172.21.15.46:0/2602915079 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a3df667fe0 0x55a3df66e390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:15:17.353 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:17.342+0000 7f2546ffd640 1 -- 172.21.15.46:0/2602915079 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a3df3ab620 con 0x55a3df6676c0 2024-03-31T23:15:17.353 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:17.342+0000 7f2546ffd640 1 --2- 172.21.15.46:0/2602915079 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a3df6676c0 0x55a3df66de50 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f253c006010 tx=0x7f253c0119b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6af98c679efd4264 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:15:17.353 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:17.342+0000 7f2526ffd640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-31T23:15:17.353 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:17.342+0000 7f2526ffd640 1 -- 172.21.15.46:0/2602915079 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f253c005460 con 0x55a3df6676c0 2024-03-31T23:15:17.353 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:17.342+0000 7f25509c5740 1 -- 172.21.15.46:0/2602915079 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f253c00fdf0 con 0x55a3df6676c0 2024-03-31T23:15:17.353 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:17.342+0000 7f2526ffd640 1 -- 172.21.15.46:0/2602915079 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7f253c0046c0 con 0x55a3df6676c0 2024-03-31T23:15:17.353 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:17.342+0000 7f2526ffd640 1 -- 172.21.15.46:0/2602915079 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f253c034050 con 0x55a3df6676c0 2024-03-31T23:15:17.353 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:17.342+0000 7f25509c5740 10 client.16437 fetch_fsmap learned FSMap version 28 2024-03-31T23:15:17.353 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:17.342+0000 7f25509c5740 1 -- 172.21.15.46:0/2602915079 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55a3df67f910 con 0x55a3df6676c0 2024-03-31T23:15:17.353 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:17.343+0000 7f2526ffd640 1 -- 172.21.15.46:0/2602915079 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f253c00fdf0 con 0x55a3df6676c0 2024-03-31T23:15:17.353 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:17.343+0000 7f2526ffd640 1 -- 172.21.15.46:0/2602915079 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f253c01d020 con 0x55a3df6676c0 2024-03-31T23:15:17.353 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:17.343+0000 7f25509c5740 10 client.16437 fetch_fsmap finished waiting for FSMap version 28 2024-03-31T23:15:17.353 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:17.343+0000 7f25509c5740 10 client.16437 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-31T23:15:17.354 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:17.343+0000 7f25509c5740 20 client.16437 populate_metadata read hostname 'smithi046' 2024-03-31T23:15:17.354 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:17.343+0000 7f25509c5740 1 --2- 172.21.15.46:0/2602915079 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55a3df680c60 0x55a3df683050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:15:17.354 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:17.343+0000 7f25509c5740 4 client.16437 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-31T23:15:17.354 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:17.343+0000 7f25509c5740 1 -- 172.21.15.46:0/2602915079 --> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a3df5de120 con 0x55a3df680c60 2024-03-31T23:15:17.354 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:17.343+0000 7f25477fe640 1 --2- 172.21.15.46:0/2602915079 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55a3df680c60 0x55a3df683050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:15:17.354 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:17.344+0000 7f25477fe640 1 --2- 172.21.15.46:0/2602915079 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55a3df680c60 0x55a3df683050 crc :-1 s=READY pgs=105 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bbd78ff09afabc89 server_cookie=e864f8495840e09d in_seq=0 out_seq=0 2024-03-31T23:15:17.354 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:17.344+0000 7f2526ffd640 10 client.16437 ms_handle_connect on v2:172.21.15.142:6838/2519143918 2024-03-31T23:15:17.354 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:17.344+0000 7f2526ffd640 1 -- 172.21.15.46:0/2602915079 <== mds.0 v2:172.21.15.142:6838/2519143918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55a3df5de120 con 0x55a3df680c60 2024-03-31T23:15:17.354 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:17.344+0000 7f2526ffd640 10 client.16437 handle_command_reply: tid=0 2024-03-31T23:15:17.354 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:17.344+0000 7f25509c5740 1 client.16437 shutdown 2024-03-31T23:15:17.354 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:17.345+0000 7f25509c5740 1 -- 172.21.15.46:0/2602915079 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a3df6676c0 msgr2=0x55a3df66de50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:15:17.354 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:17.345+0000 7f25509c5740 1 --2- 172.21.15.46:0/2602915079 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a3df6676c0 0x55a3df66de50 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f253c006010 tx=0x7f253c0119b0 comp rx=0 tx=0).stop 2024-03-31T23:15:17.355 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:17.345+0000 7f25509c5740 1 -- 172.21.15.46:0/2602915079 shutdown_connections 2024-03-31T23:15:17.355 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:17.345+0000 7f25509c5740 1 --2- 172.21.15.46:0/2602915079 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55a3df680c60 0x55a3df683050 crc :-1 s=READY pgs=105 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:15:17.355 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:17.345+0000 7f25509c5740 1 --2- 172.21.15.46:0/2602915079 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a3df670db0 0x55a3df66e8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:15:17.355 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:17.345+0000 7f25509c5740 1 --2- 172.21.15.46:0/2602915079 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a3df667fe0 0x55a3df66e390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:15:17.355 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:17.345+0000 7f2526ffd640 0 client.16437 ms_handle_reset on v2:172.21.15.142:6838/2519143918 2024-03-31T23:15:17.355 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:17.345+0000 7f25509c5740 1 --2- 172.21.15.46:0/2602915079 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a3df6676c0 0x55a3df66de50 unknown :-1 s=CLOSED pgs=673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:15:17.355 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:17.345+0000 7f25509c5740 1 -- 172.21.15.46:0/2602915079 >> 172.21.15.46:0/2602915079 conn(0x55a3df661fe0 msgr2=0x55a3df663d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:15:17.355 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:17.345+0000 7f25509c5740 1 -- 172.21.15.46:0/2602915079 shutdown_connections 2024-03-31T23:15:17.355 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:17.345+0000 7f25509c5740 1 -- 172.21.15.46:0/2602915079 wait complete. 2024-03-31T23:15:17.355 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:17.345+0000 7f25509c5740 20 client.16437 trim_cache size 0 max 16384 2024-03-31T23:15:17.363 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:17.361+0000 7f25509c5740 1 Processor -- start 2024-03-31T23:15:17.363 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:17.361+0000 7f25509c5740 1 -- start start 2024-03-31T23:15:17.363 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:17.361+0000 7f25509c5740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a3df6676c0 0x55a3df670580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:15:17.363 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:17.361+0000 7f25509c5740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a3df667fe0 0x55a3df67c780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:15:17.363 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:17.361+0000 7f25509c5740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a3df670db0 0x55a3df660e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:15:17.363 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:17.361+0000 7f25509c5740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f25481b1490 con 0x55a3df670db0 2024-03-31T23:15:17.363 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:17.361+0000 7f25509c5740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f25481aed60 con 0x55a3df667fe0 2024-03-31T23:15:17.363 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:17.361+0000 7f25509c5740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f25481acbb0 con 0x55a3df6676c0 2024-03-31T23:15:17.363 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:17.361+0000 7f2546ffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a3df670db0 0x55a3df660e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:15:17.363 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:17.361+0000 7f2546ffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a3df670db0 0x55a3df660e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39160/0 (socket says 172.21.15.46:39160) 2024-03-31T23:15:17.363 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:17.361+0000 7f2546ffd640 1 -- 172.21.15.46:0/1390043402 learned_addr learned my addr 172.21.15.46:0/1390043402 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:15:17.364 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:17.361+0000 7f25477fe640 1 --2- 172.21.15.46:0/1390043402 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a3df667fe0 0x55a3df67c780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:15:17.364 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:17.361+0000 7f2546ffd640 1 --2- 172.21.15.46:0/1390043402 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a3df6676c0 0x55a3df670580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:15:17.364 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:17.361+0000 7f2546ffd640 1 -- 172.21.15.46:0/1390043402 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a3df6676c0 msgr2=0x55a3df670580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:15:17.364 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:17.361+0000 7f2546ffd640 1 --2- 172.21.15.46:0/1390043402 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a3df6676c0 0x55a3df670580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:15:17.364 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:17.361+0000 7f2546ffd640 1 -- 172.21.15.46:0/1390043402 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a3df667fe0 msgr2=0x55a3df67c780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:15:17.364 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:17.361+0000 7f2546ffd640 1 --2- 172.21.15.46:0/1390043402 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a3df667fe0 0x55a3df67c780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:15:17.364 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:17.361+0000 7f2546ffd640 1 -- 172.21.15.46:0/1390043402 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a3df667e10 con 0x55a3df670db0 2024-03-31T23:15:17.364 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:17.362+0000 7f2546ffd640 1 --2- 172.21.15.46:0/1390043402 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a3df670db0 0x55a3df660e00 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f253c0059f0 tx=0x7f253c004350 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f3acb914d2a70b28 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:15:17.364 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:17.362+0000 7f252d7fa640 1 -- 172.21.15.46:0/1390043402 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f253c00fdf0 con 0x55a3df670db0 2024-03-31T23:15:17.364 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:17.362+0000 7f252d7fa640 1 -- 172.21.15.46:0/1390043402 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f253c0035f0 con 0x55a3df670db0 2024-03-31T23:15:17.364 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:17.362+0000 7f252d7fa640 1 -- 172.21.15.46:0/1390043402 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f253c01f440 con 0x55a3df670db0 2024-03-31T23:15:17.368 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:17.362+0000 7f25509c5740 1 -- 172.21.15.46:0/1390043402 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a3df670db0 msgr2=0x55a3df660e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:15:17.368 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:17.362+0000 7f25509c5740 1 --2- 172.21.15.46:0/1390043402 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a3df670db0 0x55a3df660e00 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f253c0059f0 tx=0x7f253c004350 comp rx=0 tx=0).stop 2024-03-31T23:15:17.368 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:17.363+0000 7f25509c5740 1 -- 172.21.15.46:0/1390043402 shutdown_connections 2024-03-31T23:15:17.368 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:17.363+0000 7f25509c5740 1 --2- 172.21.15.46:0/1390043402 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a3df670db0 0x55a3df660e00 unknown :-1 s=CLOSED pgs=674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:15:17.368 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:17.363+0000 7f25509c5740 1 --2- 172.21.15.46:0/1390043402 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a3df667fe0 0x55a3df67c780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:15:17.368 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:17.363+0000 7f25509c5740 1 --2- 172.21.15.46:0/1390043402 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a3df6676c0 0x55a3df670580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:15:17.368 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:17.363+0000 7f25509c5740 1 -- 172.21.15.46:0/1390043402 >> 172.21.15.46:0/1390043402 conn(0x55a3df661fe0 msgr2=0x55a3df6606f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:15:17.368 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:17.363+0000 7f25509c5740 1 -- 172.21.15.46:0/1390043402 shutdown_connections 2024-03-31T23:15:17.368 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:17.363+0000 7f25509c5740 1 -- 172.21.15.46:0/1390043402 wait complete. 2024-03-31T23:15:17.368 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:17.363+0000 7f25509c5740 1 -- start start 2024-03-31T23:15:17.368 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:17.363+0000 7f25509c5740 1 Processor -- start 2024-03-31T23:15:17.368 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:17.363+0000 7f25509c5740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a3df6676c0 0x55a3df679100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:15:17.369 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:17.363+0000 7f25509c5740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a3df667fe0 0x55a3df674470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:15:17.369 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:17.363+0000 7f25509c5740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a3df670db0 0x55a3df6749f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:15:17.369 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:17.363+0000 7f25509c5740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f25481ad220 con 0x55a3df670db0 2024-03-31T23:15:17.369 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:17.363+0000 7f25509c5740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f25481ae690 con 0x55a3df667fe0 2024-03-31T23:15:17.369 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:17.363+0000 7f25509c5740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f25481ad8f0 con 0x55a3df6676c0 2024-03-31T23:15:17.369 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:17.363+0000 7f25509c5740 10 client.0 _ll_register_callbacks cb 0x7f253c0116b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T23:15:17.369 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:17.364+0000 7f2546ffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a3df670db0 0x55a3df6749f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:15:17.369 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:17.364+0000 7f25477fe640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a3df667fe0 0x55a3df674470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:15:17.369 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:17.364+0000 7f25477fe640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a3df667fe0 0x55a3df674470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.46:45724/0 (socket says 172.21.15.46:45724) 2024-03-31T23:15:17.369 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:17.364+0000 7f2546ffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a3df670db0 0x55a3df6749f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39164/0 (socket says 172.21.15.46:39164) 2024-03-31T23:15:17.369 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:17.364+0000 7f25477fe640 1 -- 172.21.15.46:0/2409647272 learned_addr learned my addr 172.21.15.46:0/2409647272 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:15:17.369 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:17.364+0000 7f2546ffd640 1 --2- 172.21.15.46:0/2409647272 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a3df6676c0 0x55a3df679100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:15:17.369 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:17.364+0000 7f25477fe640 1 -- 172.21.15.46:0/2409647272 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a3df6676c0 msgr2=0x55a3df679100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:15:17.370 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:17.364+0000 7f25477fe640 1 --2- 172.21.15.46:0/2409647272 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a3df6676c0 0x55a3df679100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:15:17.370 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:17.364+0000 7f25477fe640 1 -- 172.21.15.46:0/2409647272 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a3df670db0 msgr2=0x55a3df6749f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:15:17.370 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:17.364+0000 7f25477fe640 1 --2- 172.21.15.46:0/2409647272 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a3df670db0 0x55a3df6749f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:15:17.370 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:17.364+0000 7f25477fe640 1 -- 172.21.15.46:0/2409647272 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a3df675270 con 0x55a3df667fe0 2024-03-31T23:15:17.370 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:17.364+0000 7f25477fe640 1 --2- 172.21.15.46:0/2409647272 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a3df667fe0 0x55a3df674470 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f2538006010 tx=0x7f253802c920 comp rx=0 tx=0).ready entity=mon.2 client_cookie=30015d6e13350056 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:15:17.370 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:17.364+0000 7f2526ffd640 10 client.0 ms_handle_connect on v2:172.21.15.142:3300/0 2024-03-31T23:15:17.370 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:17.364+0000 7f2526ffd640 1 -- 172.21.15.46:0/2409647272 <== mon.2 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2538049b10 con 0x55a3df667fe0 2024-03-31T23:15:17.370 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:17.364+0000 7f2526ffd640 1 -- 172.21.15.46:0/2409647272 <== mon.2 v2:172.21.15.142:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7f2538049cb0 con 0x55a3df667fe0 2024-03-31T23:15:17.370 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:17.364+0000 7f25509c5740 1 -- 172.21.15.46:0/2409647272 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f253c0035f0 con 0x55a3df667fe0 2024-03-31T23:15:17.370 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:17.365+0000 7f2526ffd640 1 -- 172.21.15.46:0/2409647272 <== mon.2 v2:172.21.15.142:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f253805a050 con 0x55a3df667fe0 2024-03-31T23:15:17.370 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:17.365+0000 7f25509c5740 10 client.26282 fetch_fsmap learned FSMap version 28 2024-03-31T23:15:17.370 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:17.365+0000 7f25509c5740 1 -- 172.21.15.46:0/2409647272 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55a3df6e6ef0 con 0x55a3df667fe0 2024-03-31T23:15:17.370 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:17.365+0000 7f2526ffd640 1 -- 172.21.15.46:0/2409647272 <== mon.2 v2:172.21.15.142:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f253c0035f0 con 0x55a3df667fe0 2024-03-31T23:15:17.370 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:17.365+0000 7f2526ffd640 1 -- 172.21.15.46:0/2409647272 <== mon.2 v2:172.21.15.142:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f25380035f0 con 0x55a3df667fe0 2024-03-31T23:15:17.371 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:17.365+0000 7f25509c5740 10 client.26282 fetch_fsmap finished waiting for FSMap version 28 2024-03-31T23:15:17.371 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:17.365+0000 7f25509c5740 10 client.26282 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-31T23:15:17.371 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:17.365+0000 7f25509c5740 20 client.26282 populate_metadata read hostname 'smithi046' 2024-03-31T23:15:17.371 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:17.365+0000 7f25509c5740 1 --2- 172.21.15.46:0/2409647272 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55a3df6e76c0 0x55a3df6e9ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:15:17.371 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:17.365+0000 7f25509c5740 4 client.26282 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-31T23:15:17.371 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:17.365+0000 7f25509c5740 1 -- 172.21.15.46:0/2409647272 --> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55a3df5d7e30 con 0x55a3df6e76c0 2024-03-31T23:15:17.371 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:17.365+0000 7f2546ffd640 1 --2- 172.21.15.46:0/2409647272 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55a3df6e76c0 0x55a3df6e9ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:15:17.371 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:17.366+0000 7f2546ffd640 1 --2- 172.21.15.46:0/2409647272 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55a3df6e76c0 0x55a3df6e9ae0 crc :-1 s=READY pgs=106 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=14afc6c66d875fc0 server_cookie=8d9c9d1693aa48ce in_seq=0 out_seq=0 2024-03-31T23:15:17.371 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:17.366+0000 7f2526ffd640 10 client.26282 ms_handle_connect on v2:172.21.15.142:6838/2519143918 2024-03-31T23:15:17.389 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:17.387+0000 7f2526ffd640 1 -- 172.21.15.46:0/2409647272 <== mds.0 v2:172.21.15.142:6838/2519143918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55a3df5d7e30 con 0x55a3df6e76c0 2024-03-31T23:15:17.389 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:17.387+0000 7f2526ffd640 10 client.26282 handle_command_reply: tid=0 2024-03-31T23:15:17.390 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:17.387+0000 7f25509c5740 1 client.26282 shutdown 2024-03-31T23:15:17.390 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:17.388+0000 7f25509c5740 1 -- 172.21.15.46:0/2409647272 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a3df667fe0 msgr2=0x55a3df674470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:15:17.390 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:17.388+0000 7f25509c5740 1 --2- 172.21.15.46:0/2409647272 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a3df667fe0 0x55a3df674470 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f2538006010 tx=0x7f253802c920 comp rx=0 tx=0).stop 2024-03-31T23:15:17.390 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:17.388+0000 7f25509c5740 1 -- 172.21.15.46:0/2409647272 shutdown_connections 2024-03-31T23:15:17.390 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:17.388+0000 7f25509c5740 1 --2- 172.21.15.46:0/2409647272 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55a3df6e76c0 0x55a3df6e9ae0 crc :-1 s=READY pgs=106 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:15:17.390 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:17.388+0000 7f25509c5740 1 --2- 172.21.15.46:0/2409647272 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a3df670db0 0x55a3df6749f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:15:17.390 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:17.388+0000 7f25509c5740 1 --2- 172.21.15.46:0/2409647272 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a3df667fe0 0x55a3df674470 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:15:17.390 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:17.388+0000 7f2526ffd640 0 client.26282 ms_handle_reset on v2:172.21.15.142:6838/2519143918 2024-03-31T23:15:17.390 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:17.388+0000 7f25509c5740 1 --2- 172.21.15.46:0/2409647272 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a3df6676c0 0x55a3df679100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:15:17.390 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:17.388+0000 7f25509c5740 1 -- 172.21.15.46:0/2409647272 >> 172.21.15.46:0/2409647272 conn(0x55a3df661fe0 msgr2=0x55a3df660f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:15:17.390 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:17.388+0000 7f25509c5740 1 -- 172.21.15.46:0/2409647272 shutdown_connections 2024-03-31T23:15:17.390 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:17.388+0000 7f25509c5740 1 -- 172.21.15.46:0/2409647272 wait complete. 2024-03-31T23:15:17.391 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:17.388+0000 7f25509c5740 20 client.26282 trim_cache size 0 max 16384 2024-03-31T23:15:17.391 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout:{ 2024-03-31T23:15:17.391 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout: "message": "", 2024-03-31T23:15:17.391 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout: "return_code": 0 2024-03-31T23:15:17.391 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout:} 2024-03-31T23:15:17.392 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:17.389+0000 7f254ddeb640 1 -- 172.21.15.46:0/128446346 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f25300a26b0 msgr2=0x7f25300a4b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:15:17.392 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:17.389+0000 7f254ddeb640 1 --2- 172.21.15.46:0/128446346 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f25300a26b0 0x7f25300a4b70 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f25400cc8f0 tx=0x7f2540090d00 comp rx=0 tx=0).stop 2024-03-31T23:15:17.392 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:17.389+0000 7f254ddeb640 1 -- 172.21.15.46:0/128446346 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f25481055f0 msgr2=0x7f25481a3dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:15:17.392 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:17.389+0000 7f254ddeb640 1 --2- 172.21.15.46:0/128446346 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f25481055f0 0x7f25481a3dc0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f2540005fd0 tx=0x7f254000ee60 comp rx=0 tx=0).stop 2024-03-31T23:15:17.392 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:17.389+0000 7f254ddeb640 1 -- 172.21.15.46:0/128446346 shutdown_connections 2024-03-31T23:15:17.392 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:17.389+0000 7f254ddeb640 1 --2- 172.21.15.46:0/128446346 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f25300a26b0 0x7f25300a4b70 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:15:17.392 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:17.389+0000 7f254ddeb640 1 --2- 172.21.15.46:0/128446346 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f25481055f0 0x7f25481a3dc0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:15:17.392 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:17.389+0000 7f254ddeb640 1 --2- 172.21.15.46:0/128446346 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2548101260 0x7f2548199280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:15:17.392 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:17.389+0000 7f254ddeb640 1 --2- 172.21.15.46:0/128446346 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f25481008a0 0x7f2548198d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:15:17.393 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:17.389+0000 7f254ddeb640 1 -- 172.21.15.46:0/128446346 >> 172.21.15.46:0/128446346 conn(0x7f25480774a0 msgr2=0x7f25480fe950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:15:17.393 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:17.389+0000 7f254ddeb640 1 -- 172.21.15.46:0/128446346 shutdown_connections 2024-03-31T23:15:17.393 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:17.389+0000 7f254ddeb640 1 -- 172.21.15.46:0/128446346 wait complete. 2024-03-31T23:15:18.194 INFO:tasks.workunit.client.0.smithi046.stdout: 1 65969 2.59 MB/sec execute 24 sec latency 26.560 ms 2024-03-31T23:15:18.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:15:18 smithi046 ceph-mon[32498]: pgmap v110: 129 pgs: 129 active+clean; 60 MiB data, 770 MiB used, 1.0 TiB / 1.0 TiB avail; 319 KiB/s rd, 1.3 MiB/s wr, 118 op/s 2024-03-31T23:15:18.944 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:15:18 smithi142 ceph-mon[35271]: pgmap v110: 129 pgs: 129 active+clean; 60 MiB data, 770 MiB used, 1.0 TiB / 1.0 TiB avail; 319 KiB/s rd, 1.3 MiB/s wr, 118 op/s 2024-03-31T23:15:19.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:15:18 smithi162 ceph-mon[36237]: pgmap v110: 129 pgs: 129 active+clean; 60 MiB data, 770 MiB used, 1.0 TiB / 1.0 TiB avail; 319 KiB/s rd, 1.3 MiB/s wr, 118 op/s 2024-03-31T23:15:19.195 INFO:tasks.workunit.client.0.smithi046.stdout: 1 66464 2.58 MB/sec execute 25 sec latency 16.248 ms 2024-03-31T23:15:20.194 INFO:tasks.workunit.client.0.smithi046.stdout: 1 67179 2.73 MB/sec execute 26 sec latency 16.564 ms 2024-03-31T23:15:20.847 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:15:20 smithi162 ceph-mon[36237]: pgmap v111: 129 pgs: 129 active+clean; 56 MiB data, 780 MiB used, 1.0 TiB / 1.0 TiB avail; 495 KiB/s rd, 1.8 MiB/s wr, 183 op/s 2024-03-31T23:15:20.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:15:20 smithi046 ceph-mon[32498]: pgmap v111: 129 pgs: 129 active+clean; 56 MiB data, 780 MiB used, 1.0 TiB / 1.0 TiB avail; 495 KiB/s rd, 1.8 MiB/s wr, 183 op/s 2024-03-31T23:15:20.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:15:20 smithi142 ceph-mon[35271]: pgmap v111: 129 pgs: 129 active+clean; 56 MiB data, 780 MiB used, 1.0 TiB / 1.0 TiB avail; 495 KiB/s rd, 1.8 MiB/s wr, 183 op/s 2024-03-31T23:15:21.194 INFO:tasks.workunit.client.0.smithi046.stdout: 1 67641 2.79 MB/sec execute 27 sec latency 17.719 ms 2024-03-31T23:15:22.194 INFO:tasks.workunit.client.0.smithi046.stdout: 1 67887 2.70 MB/sec execute 28 sec latency 16.352 ms 2024-03-31T23:15:22.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:15:22 smithi046 ceph-mon[32498]: pgmap v112: 129 pgs: 129 active+clean; 56 MiB data, 780 MiB used, 1.0 TiB / 1.0 TiB avail; 317 KiB/s rd, 1.1 MiB/s wr, 119 op/s 2024-03-31T23:15:22.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:15:22 smithi142 ceph-mon[35271]: pgmap v112: 129 pgs: 129 active+clean; 56 MiB data, 780 MiB used, 1.0 TiB / 1.0 TiB avail; 317 KiB/s rd, 1.1 MiB/s wr, 119 op/s 2024-03-31T23:15:23.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:15:22 smithi162 ceph-mon[36237]: pgmap v112: 129 pgs: 129 active+clean; 56 MiB data, 780 MiB used, 1.0 TiB / 1.0 TiB avail; 317 KiB/s rd, 1.1 MiB/s wr, 119 op/s 2024-03-31T23:15:23.194 INFO:tasks.workunit.client.0.smithi046.stdout: 1 68290 2.63 MB/sec execute 29 sec latency 15.927 ms 2024-03-31T23:15:24.194 INFO:tasks.workunit.client.0.smithi046.stdout: 1 68805 2.69 MB/sec execute 30 sec latency 16.125 ms 2024-03-31T23:15:24.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:15:24 smithi046 ceph-mon[32498]: pgmap v113: 129 pgs: 129 active+clean; 56 MiB data, 780 MiB used, 1.0 TiB / 1.0 TiB avail; 317 KiB/s rd, 1.1 MiB/s wr, 119 op/s 2024-03-31T23:15:24.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:15:24 smithi142 ceph-mon[35271]: pgmap v113: 129 pgs: 129 active+clean; 56 MiB data, 780 MiB used, 1.0 TiB / 1.0 TiB avail; 317 KiB/s rd, 1.1 MiB/s wr, 119 op/s 2024-03-31T23:15:25.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:15:24 smithi162 ceph-mon[36237]: pgmap v113: 129 pgs: 129 active+clean; 56 MiB data, 780 MiB used, 1.0 TiB / 1.0 TiB avail; 317 KiB/s rd, 1.1 MiB/s wr, 119 op/s 2024-03-31T23:15:25.194 INFO:tasks.workunit.client.0.smithi046.stdout: 1 69335 2.62 MB/sec execute 31 sec latency 20.380 ms 2024-03-31T23:15:26.195 INFO:tasks.workunit.client.0.smithi046.stdout: 1 69862 2.59 MB/sec execute 32 sec latency 16.630 ms 2024-03-31T23:15:26.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:15:26 smithi046 ceph-mon[32498]: pgmap v114: 129 pgs: 129 active+clean; 61 MiB data, 784 MiB used, 1.0 TiB / 1.0 TiB avail; 498 KiB/s rd, 1.8 MiB/s wr, 179 op/s 2024-03-31T23:15:26.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:15:26 smithi142 ceph-mon[35271]: pgmap v114: 129 pgs: 129 active+clean; 61 MiB data, 784 MiB used, 1.0 TiB / 1.0 TiB avail; 498 KiB/s rd, 1.8 MiB/s wr, 179 op/s 2024-03-31T23:15:27.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:15:26 smithi162 ceph-mon[36237]: pgmap v114: 129 pgs: 129 active+clean; 61 MiB data, 784 MiB used, 1.0 TiB / 1.0 TiB avail; 498 KiB/s rd, 1.8 MiB/s wr, 179 op/s 2024-03-31T23:15:27.194 INFO:tasks.workunit.client.0.smithi046.stdout: 1 70605 2.70 MB/sec execute 33 sec latency 18.942 ms 2024-03-31T23:15:28.195 INFO:tasks.workunit.client.0.smithi046.stdout: 1 70970 2.74 MB/sec execute 34 sec latency 18.978 ms 2024-03-31T23:15:28.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:15:28 smithi046 ceph-mon[32498]: pgmap v115: 129 pgs: 129 active+clean; 61 MiB data, 784 MiB used, 1.0 TiB / 1.0 TiB avail; 356 KiB/s rd, 1.2 MiB/s wr, 124 op/s 2024-03-31T23:15:28.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:15:28 smithi142 ceph-mon[35271]: pgmap v115: 129 pgs: 129 active+clean; 61 MiB data, 784 MiB used, 1.0 TiB / 1.0 TiB avail; 356 KiB/s rd, 1.2 MiB/s wr, 124 op/s 2024-03-31T23:15:29.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:15:28 smithi162 ceph-mon[36237]: pgmap v115: 129 pgs: 129 active+clean; 61 MiB data, 784 MiB used, 1.0 TiB / 1.0 TiB avail; 356 KiB/s rd, 1.2 MiB/s wr, 124 op/s 2024-03-31T23:15:29.195 INFO:tasks.workunit.client.0.smithi046.stdout: 1 71339 2.71 MB/sec execute 35 sec latency 15.307 ms 2024-03-31T23:15:30.195 INFO:tasks.workunit.client.0.smithi046.stdout: 1 71627 2.64 MB/sec execute 36 sec latency 13.438 ms 2024-03-31T23:15:30.484 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:30.483+0000 7f48de3a7640 1 -- 172.21.15.46:0/4124444139 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f48d8106980 msgr2=0x7f48d810e620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:15:30.484 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:30.483+0000 7f48de3a7640 1 --2- 172.21.15.46:0/4124444139 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f48d8106980 0x7f48d810e620 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f48d0009ca0 tx=0x7f48d002c0f0 comp rx=0 tx=0).stop 2024-03-31T23:15:30.484 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:30.483+0000 7f48de3a7640 1 -- 172.21.15.46:0/4124444139 shutdown_connections 2024-03-31T23:15:30.484 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:30.483+0000 7f48de3a7640 1 --2- 172.21.15.46:0/4124444139 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f48d8106980 0x7f48d810e620 unknown :-1 s=CLOSED pgs=675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:15:30.484 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:30.483+0000 7f48de3a7640 1 --2- 172.21.15.46:0/4124444139 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f48d8100810 0x7f48d81062a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:15:30.485 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:30.483+0000 7f48de3a7640 1 --2- 172.21.15.46:0/4124444139 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f48d80ffe60 0x7f48d8100240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:15:30.485 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:30.483+0000 7f48de3a7640 1 -- 172.21.15.46:0/4124444139 >> 172.21.15.46:0/4124444139 conn(0x7f48d8076b70 msgr2=0x7f48d80fdf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:15:30.485 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:30.483+0000 7f48de3a7640 1 -- 172.21.15.46:0/4124444139 shutdown_connections 2024-03-31T23:15:30.485 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:30.484+0000 7f48de3a7640 1 -- 172.21.15.46:0/4124444139 wait complete. 2024-03-31T23:15:30.485 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:30.484+0000 7f48de3a7640 1 Processor -- start 2024-03-31T23:15:30.485 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:30.484+0000 7f48de3a7640 1 -- start start 2024-03-31T23:15:30.487 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:30.484+0000 7f48de3a7640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f48d80ffe60 0x7f48d819a0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:15:30.487 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:30.484+0000 7f48de3a7640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f48d8106980 0x7f48d819a5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:15:30.487 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:30.484+0000 7f48de3a7640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f48d819e970 0x7f48d819ee20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:15:30.487 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:30.485+0000 7f48de3a7640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f48d81119b0 con 0x7f48d80ffe60 2024-03-31T23:15:30.487 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:30.485+0000 7f48de3a7640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f48d8111830 con 0x7f48d8106980 2024-03-31T23:15:30.487 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:30.485+0000 7f48de3a7640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f48d8111b30 con 0x7f48d819e970 2024-03-31T23:15:30.487 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:30.485+0000 7f48d7fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f48d80ffe60 0x7f48d819a0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:15:30.487 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:30.485+0000 7f48d7fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f48d80ffe60 0x7f48d819a0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40284/0 (socket says 172.21.15.46:40284) 2024-03-31T23:15:30.487 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:30.485+0000 7f48d7fff640 1 -- 172.21.15.46:0/142747992 learned_addr learned my addr 172.21.15.46:0/142747992 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:15:30.488 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:30.485+0000 7f48d77fe640 1 --2- 172.21.15.46:0/142747992 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f48d8106980 0x7f48d819a5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:15:30.488 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:30.485+0000 7f48dc944640 1 --2- 172.21.15.46:0/142747992 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f48d819e970 0x7f48d819ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:15:30.488 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:30.485+0000 7f48d7fff640 1 -- 172.21.15.46:0/142747992 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f48d819e970 msgr2=0x7f48d819ee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:15:30.488 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:30.485+0000 7f48d7fff640 1 --2- 172.21.15.46:0/142747992 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f48d819e970 0x7f48d819ee20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:15:30.488 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:30.485+0000 7f48d7fff640 1 -- 172.21.15.46:0/142747992 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f48d8106980 msgr2=0x7f48d819a5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:15:30.488 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:30.485+0000 7f48d7fff640 1 --2- 172.21.15.46:0/142747992 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f48d8106980 0x7f48d819a5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:15:30.488 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:30.485+0000 7f48d7fff640 1 -- 172.21.15.46:0/142747992 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f48d819f460 con 0x7f48d80ffe60 2024-03-31T23:15:30.488 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:30.485+0000 7f48d7fff640 1 --2- 172.21.15.46:0/142747992 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f48d80ffe60 0x7f48d819a0b0 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f48c800e9f0 tx=0x7f48c800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:15:30.488 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:30.485+0000 7f48d57fa640 1 -- 172.21.15.46:0/142747992 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48c800cd80 con 0x7f48d80ffe60 2024-03-31T23:15:30.488 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:30.485+0000 7f48de3a7640 1 -- 172.21.15.46:0/142747992 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f48d819f750 con 0x7f48d80ffe60 2024-03-31T23:15:30.488 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:30.485+0000 7f48d57fa640 1 -- 172.21.15.46:0/142747992 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f48c80108e0 con 0x7f48d80ffe60 2024-03-31T23:15:30.488 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:30.485+0000 7f48d57fa640 1 -- 172.21.15.46:0/142747992 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48c8018920 con 0x7f48d80ffe60 2024-03-31T23:15:30.488 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:30.485+0000 7f48de3a7640 1 -- 172.21.15.46:0/142747992 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f48d81a6f40 con 0x7f48d80ffe60 2024-03-31T23:15:30.493 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:30.488+0000 7f48d57fa640 1 -- 172.21.15.46:0/142747992 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 25) ==== 137451+0+0 (secure 0 0 0) 0x7f48c8018ac0 con 0x7f48d80ffe60 2024-03-31T23:15:30.493 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:30.488+0000 7f48d57fa640 1 --2- 172.21.15.46:0/142747992 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f48c00a26b0 0x7f48c00a4b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:15:30.494 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:30.488+0000 7f48e0f82740 1 Processor -- start 2024-03-31T23:15:30.494 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:30.488+0000 7f48e0f82740 1 -- start start 2024-03-31T23:15:30.494 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:30.488+0000 7f48d57fa640 1 -- 172.21.15.46:0/142747992 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7f48c80cf200 con 0x7f48d80ffe60 2024-03-31T23:15:30.494 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:30.488+0000 7f48d77fe640 1 --2- 172.21.15.46:0/142747992 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f48c00a26b0 0x7f48c00a4b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:15:30.494 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:30.488+0000 7f48e0f82740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55dde1bf36c0 0x55dde1bf3aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:15:30.494 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:30.488+0000 7f48e0f82740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55dde1bf3fe0 0x55dde1bfc870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:15:30.494 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:30.488+0000 7f48e0f82740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dde1bfcdb0 0x55dde1bff1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:15:30.494 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:30.488+0000 7f48e0f82740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55dde1b568f0 con 0x55dde1bf3fe0 2024-03-31T23:15:30.494 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:30.488+0000 7f48e0f82740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55dde1aff8d0 con 0x55dde1bf36c0 2024-03-31T23:15:30.494 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:30.488+0000 7f48e0f82740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55dde1b52990 con 0x55dde1bfcdb0 2024-03-31T23:15:30.494 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:30.488+0000 7f48d77fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55dde1bf3fe0 0x55dde1bfc870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:15:30.494 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:30.489+0000 7f48d77fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55dde1bf3fe0 0x55dde1bfc870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40294/0 (socket says 172.21.15.46:40294) 2024-03-31T23:15:30.495 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:30.489+0000 7f48d77fe640 1 -- 172.21.15.46:0/2604260378 learned_addr learned my addr 172.21.15.46:0/2604260378 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:15:30.495 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:30.489+0000 7f48d7fff640 1 --2- 172.21.15.46:0/2604260378 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55dde1bf36c0 0x55dde1bf3aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:15:30.495 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:30.489+0000 7f48dc944640 1 --2- 172.21.15.46:0/2604260378 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dde1bfcdb0 0x55dde1bff1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:15:30.495 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:30.489+0000 7f48d7fff640 1 -- 172.21.15.46:0/2604260378 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dde1bfcdb0 msgr2=0x55dde1bff1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:15:30.495 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:30.489+0000 7f48d7fff640 1 --2- 172.21.15.46:0/2604260378 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dde1bfcdb0 0x55dde1bff1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:15:30.495 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:30.489+0000 7f48d7fff640 1 -- 172.21.15.46:0/2604260378 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55dde1bf3fe0 msgr2=0x55dde1bfc870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:15:30.495 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:30.489+0000 7f48d7fff640 1 --2- 172.21.15.46:0/2604260378 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55dde1bf3fe0 0x55dde1bfc870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:15:30.495 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:30.489+0000 7f48d7fff640 1 -- 172.21.15.46:0/2604260378 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dde186b210 con 0x55dde1bf36c0 2024-03-31T23:15:30.495 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:30.489+0000 7f48d77fe640 1 --2- 172.21.15.46:0/142747992 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f48c00a26b0 0x7f48c00a4b70 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f48cc013460 tx=0x7f48cc013860 comp rx=0 tx=0).ready entity=mgr.15654 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:15:30.495 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:30.489+0000 7f48d7fff640 1 --2- 172.21.15.46:0/2604260378 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55dde1bf36c0 0x55dde1bf3aa0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f48c80cc5a0 tx=0x7f48c8010de0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f604eeb3a396a8e0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:15:30.495 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:30.489+0000 7f48be7fc640 1 -- 172.21.15.46:0/2604260378 <== mon.2 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48c80cc820 con 0x55dde1bf36c0 2024-03-31T23:15:30.495 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:30.489+0000 7f48be7fc640 1 -- 172.21.15.46:0/2604260378 <== mon.2 v2:172.21.15.142:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f48c8010a90 con 0x55dde1bf36c0 2024-03-31T23:15:30.495 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:30.489+0000 7f48be7fc640 1 -- 172.21.15.46:0/2604260378 <== mon.2 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48c8097520 con 0x55dde1bf36c0 2024-03-31T23:15:30.495 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:30.490+0000 7f48e0f82740 1 -- 172.21.15.46:0/2604260378 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55dde1bf36c0 msgr2=0x55dde1bf3aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:15:30.496 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:30.490+0000 7f48e0f82740 1 --2- 172.21.15.46:0/2604260378 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55dde1bf36c0 0x55dde1bf3aa0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f48c80cc5a0 tx=0x7f48c8010de0 comp rx=0 tx=0).stop 2024-03-31T23:15:30.496 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:30.490+0000 7f48e0f82740 1 -- 172.21.15.46:0/2604260378 shutdown_connections 2024-03-31T23:15:30.496 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:30.490+0000 7f48e0f82740 1 --2- 172.21.15.46:0/2604260378 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dde1bfcdb0 0x55dde1bff1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:15:30.496 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:30.490+0000 7f48e0f82740 1 --2- 172.21.15.46:0/2604260378 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55dde1bf3fe0 0x55dde1bfc870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:15:30.496 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:30.490+0000 7f48e0f82740 1 --2- 172.21.15.46:0/2604260378 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55dde1bf36c0 0x55dde1bf3aa0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:15:30.496 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:30.490+0000 7f48e0f82740 1 -- 172.21.15.46:0/2604260378 >> 172.21.15.46:0/2604260378 conn(0x55dde1bedfe0 msgr2=0x55dde1bee3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:15:30.496 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:30.490+0000 7f48e0f82740 1 -- 172.21.15.46:0/2604260378 shutdown_connections 2024-03-31T23:15:30.496 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:30.490+0000 7f48e0f82740 1 -- 172.21.15.46:0/2604260378 wait complete. 2024-03-31T23:15:30.496 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:30.491+0000 7f48e0f82740 1 -- start start 2024-03-31T23:15:30.496 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:30.491+0000 7f48e0f82740 1 Processor -- start 2024-03-31T23:15:30.496 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:30.491+0000 7f48e0f82740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55dde1bf36c0 0x55dde1bf9e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:15:30.496 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:30.491+0000 7f48e0f82740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dde1bf3fe0 0x55dde1bfa370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:15:30.496 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:30.491+0000 7f48e0f82740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55dde1bfcdb0 0x55dde1bfa8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:15:30.496 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:30.491+0000 7f48e0f82740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55dde1bff980 con 0x55dde1bfcdb0 2024-03-31T23:15:30.497 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:30.491+0000 7f48e0f82740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55dde1af4f80 con 0x55dde1bf36c0 2024-03-31T23:15:30.497 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:30.491+0000 7f48e0f82740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55dde1bffb00 con 0x55dde1bf3fe0 2024-03-31T23:15:30.497 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:30.491+0000 7f48e0f82740 10 client.0 _ll_register_callbacks cb 0x55dde1b4d290 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T23:15:30.497 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:30.491+0000 7f48dc944640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55dde1bfcdb0 0x55dde1bfa8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:15:30.497 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:30.492+0000 7f48d7fff640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55dde1bf36c0 0x55dde1bf9e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:15:30.497 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:30.492+0000 7f48dc944640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55dde1bfcdb0 0x55dde1bfa8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40304/0 (socket says 172.21.15.46:40304) 2024-03-31T23:15:30.497 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:30.492+0000 7f48dc944640 1 -- 172.21.15.46:0/2638381121 learned_addr learned my addr 172.21.15.46:0/2638381121 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:15:30.497 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:30.492+0000 7f48d77fe640 1 --2- 172.21.15.46:0/2638381121 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dde1bf3fe0 0x55dde1bfa370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:15:30.497 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:30.492+0000 7f48dc944640 1 -- 172.21.15.46:0/2638381121 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dde1bf3fe0 msgr2=0x55dde1bfa370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:15:30.497 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:30.492+0000 7f48dc944640 1 --2- 172.21.15.46:0/2638381121 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dde1bf3fe0 0x55dde1bfa370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:15:30.497 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:30.492+0000 7f48dc944640 1 -- 172.21.15.46:0/2638381121 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55dde1bf36c0 msgr2=0x55dde1bf9e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:15:30.497 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:30.492+0000 7f48dc944640 1 --2- 172.21.15.46:0/2638381121 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55dde1bf36c0 0x55dde1bf9e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:15:30.497 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:30.492+0000 7f48dc944640 1 -- 172.21.15.46:0/2638381121 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dde1937620 con 0x55dde1bfcdb0 2024-03-31T23:15:30.497 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:30.492+0000 7f48d7fff640 1 --2- 172.21.15.46:0/2638381121 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55dde1bf36c0 0x55dde1bf9e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:15:30.498 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:30.492+0000 7f48dc944640 1 --2- 172.21.15.46:0/2638381121 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55dde1bfcdb0 0x55dde1bfa8b0 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7f48d0009ca0 tx=0x7f48d00323d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a82c433f433766e7 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:15:30.498 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:30.492+0000 7f48b77fe640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-31T23:15:30.498 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:30.492+0000 7f48b77fe640 1 -- 172.21.15.46:0/2638381121 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48d0044070 con 0x55dde1bfcdb0 2024-03-31T23:15:30.498 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:30.492+0000 7f48b77fe640 1 -- 172.21.15.46:0/2638381121 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7f48d0003970 con 0x55dde1bfcdb0 2024-03-31T23:15:30.498 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:30.492+0000 7f48e0f82740 1 -- 172.21.15.46:0/2638381121 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f48c8010a90 con 0x55dde1bfcdb0 2024-03-31T23:15:30.498 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:30.492+0000 7f48b77fe640 1 -- 172.21.15.46:0/2638381121 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f48d0039040 con 0x55dde1bfcdb0 2024-03-31T23:15:30.498 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:30.492+0000 7f48e0f82740 10 client.16455 fetch_fsmap learned FSMap version 28 2024-03-31T23:15:30.498 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:30.492+0000 7f48e0f82740 1 -- 172.21.15.46:0/2638381121 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55dde1c0b8a0 con 0x55dde1bfcdb0 2024-03-31T23:15:30.498 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:30.493+0000 7f48b77fe640 1 -- 172.21.15.46:0/2638381121 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f48c8010a90 con 0x55dde1bfcdb0 2024-03-31T23:15:30.498 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:30.493+0000 7f48b77fe640 1 -- 172.21.15.46:0/2638381121 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f48d0034440 con 0x55dde1bfcdb0 2024-03-31T23:15:30.498 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:30.493+0000 7f48e0f82740 10 client.16455 fetch_fsmap finished waiting for FSMap version 28 2024-03-31T23:15:30.498 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:30.493+0000 7f48e0f82740 10 client.16455 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-31T23:15:30.498 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:30.493+0000 7f48e0f82740 20 client.16455 populate_metadata read hostname 'smithi046' 2024-03-31T23:15:30.499 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:30.493+0000 7f48e0f82740 1 --2- 172.21.15.46:0/2638381121 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55dde1c0cdd0 0x55dde1c0f250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:15:30.499 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:30.493+0000 7f48e0f82740 4 client.16455 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-31T23:15:30.499 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:30.493+0000 7f48e0f82740 1 -- 172.21.15.46:0/2638381121 --> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55dde1b6a120 con 0x55dde1c0cdd0 2024-03-31T23:15:30.499 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:30.494+0000 7f48d7fff640 1 --2- 172.21.15.46:0/2638381121 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55dde1c0cdd0 0x55dde1c0f250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:15:30.500 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:30.494+0000 7f48d7fff640 1 --2- 172.21.15.46:0/2638381121 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55dde1c0cdd0 0x55dde1c0f250 crc :-1 s=READY pgs=107 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e6ae86257d1bb619 server_cookie=7be54dfda7334cc1 in_seq=0 out_seq=0 2024-03-31T23:15:30.500 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:30.494+0000 7f48b77fe640 10 client.16455 ms_handle_connect on v2:172.21.15.142:6838/2519143918 2024-03-31T23:15:30.500 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:30.495+0000 7f48b77fe640 1 -- 172.21.15.46:0/2638381121 <== mds.0 v2:172.21.15.142:6838/2519143918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55dde1b6a120 con 0x55dde1c0cdd0 2024-03-31T23:15:30.500 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:30.495+0000 7f48b77fe640 10 client.16455 handle_command_reply: tid=0 2024-03-31T23:15:30.500 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:30.495+0000 7f48e0f82740 1 client.16455 shutdown 2024-03-31T23:15:30.500 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:30.495+0000 7f48e0f82740 1 -- 172.21.15.46:0/2638381121 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55dde1bfcdb0 msgr2=0x55dde1bfa8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:15:30.500 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:30.495+0000 7f48e0f82740 1 --2- 172.21.15.46:0/2638381121 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55dde1bfcdb0 0x55dde1bfa8b0 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7f48d0009ca0 tx=0x7f48d00323d0 comp rx=0 tx=0).stop 2024-03-31T23:15:30.500 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:30.495+0000 7f48e0f82740 1 -- 172.21.15.46:0/2638381121 shutdown_connections 2024-03-31T23:15:30.500 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:30.495+0000 7f48e0f82740 1 --2- 172.21.15.46:0/2638381121 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55dde1c0cdd0 0x55dde1c0f250 crc :-1 s=READY pgs=107 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:15:30.500 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:30.495+0000 7f48e0f82740 1 --2- 172.21.15.46:0/2638381121 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55dde1bfcdb0 0x55dde1bfa8b0 unknown :-1 s=CLOSED pgs=677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:15:30.500 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:30.495+0000 7f48e0f82740 1 --2- 172.21.15.46:0/2638381121 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dde1bf3fe0 0x55dde1bfa370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:15:30.501 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:30.495+0000 7f48e0f82740 1 --2- 172.21.15.46:0/2638381121 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55dde1bf36c0 0x55dde1bf9e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:15:30.501 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:30.495+0000 7f48e0f82740 1 -- 172.21.15.46:0/2638381121 >> 172.21.15.46:0/2638381121 conn(0x55dde1bedfe0 msgr2=0x55dde1befd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:15:30.501 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:30.495+0000 7f48b77fe640 0 client.16455 ms_handle_reset on v2:172.21.15.142:6838/2519143918 2024-03-31T23:15:30.501 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:30.495+0000 7f48e0f82740 1 -- 172.21.15.46:0/2638381121 shutdown_connections 2024-03-31T23:15:30.501 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:30.495+0000 7f48e0f82740 1 -- 172.21.15.46:0/2638381121 wait complete. 2024-03-31T23:15:30.501 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:30.495+0000 7f48e0f82740 20 client.16455 trim_cache size 0 max 16384 2024-03-31T23:15:30.516 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:30.514+0000 7f48e0f82740 1 Processor -- start 2024-03-31T23:15:30.516 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:30.514+0000 7f48e0f82740 1 -- start start 2024-03-31T23:15:30.516 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:30.514+0000 7f48e0f82740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dde1bf36c0 0x55dde1bfc560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:15:30.516 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:30.514+0000 7f48e0f82740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55dde1bf3fe0 0x55dde1c08720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:15:30.516 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:30.514+0000 7f48e0f82740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55dde1bfcdb0 0x55dde1c14b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:15:30.516 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:30.514+0000 7f48e0f82740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f48d81b3560 con 0x55dde1bf3fe0 2024-03-31T23:15:30.517 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:30.514+0000 7f48e0f82740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f48d81b0e30 con 0x55dde1bfcdb0 2024-03-31T23:15:30.517 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:30.514+0000 7f48e0f82740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f48d81aec80 con 0x55dde1bf36c0 2024-03-31T23:15:30.517 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:30.514+0000 7f48d77fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55dde1bf3fe0 0x55dde1c08720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:15:30.517 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:30.514+0000 7f48d77fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55dde1bf3fe0 0x55dde1c08720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40308/0 (socket says 172.21.15.46:40308) 2024-03-31T23:15:30.517 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:30.514+0000 7f48d77fe640 1 -- 172.21.15.46:0/3752727468 learned_addr learned my addr 172.21.15.46:0/3752727468 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:15:30.517 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:30.515+0000 7f48d7fff640 1 --2- 172.21.15.46:0/3752727468 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dde1bf36c0 0x55dde1bfc560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:15:30.517 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:30.515+0000 7f48dc944640 1 --2- 172.21.15.46:0/3752727468 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55dde1bfcdb0 0x55dde1c14b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:15:30.517 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:30.515+0000 7f48d77fe640 1 -- 172.21.15.46:0/3752727468 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dde1bf36c0 msgr2=0x55dde1bfc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:15:30.517 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:30.515+0000 7f48d77fe640 1 --2- 172.21.15.46:0/3752727468 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dde1bf36c0 0x55dde1bfc560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:15:30.517 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:30.515+0000 7f48d77fe640 1 -- 172.21.15.46:0/3752727468 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55dde1bfcdb0 msgr2=0x55dde1c14b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:15:30.517 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:30.515+0000 7f48d77fe640 1 --2- 172.21.15.46:0/3752727468 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55dde1bfcdb0 0x55dde1c14b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:15:30.517 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:30.515+0000 7f48d77fe640 1 -- 172.21.15.46:0/3752727468 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dde1beebb0 con 0x55dde1bf3fe0 2024-03-31T23:15:30.517 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:30.515+0000 7f48d77fe640 1 --2- 172.21.15.46:0/3752727468 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55dde1bf3fe0 0x55dde1c08720 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7f48cc013d60 tx=0x7f48cc012040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=48ac49fa2ba9a168 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:15:30.518 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:30.515+0000 7f48bdffb640 1 -- 172.21.15.46:0/3752727468 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48cc02e070 con 0x55dde1bf3fe0 2024-03-31T23:15:30.518 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:30.515+0000 7f48bdffb640 1 -- 172.21.15.46:0/3752727468 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f48cc010ba0 con 0x55dde1bf3fe0 2024-03-31T23:15:30.518 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:30.515+0000 7f48bdffb640 1 -- 172.21.15.46:0/3752727468 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48cc020a90 con 0x55dde1bf3fe0 2024-03-31T23:15:30.518 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:30.516+0000 7f48e0f82740 1 -- 172.21.15.46:0/3752727468 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55dde1bf3fe0 msgr2=0x55dde1c08720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:15:30.518 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:30.516+0000 7f48e0f82740 1 --2- 172.21.15.46:0/3752727468 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55dde1bf3fe0 0x55dde1c08720 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7f48cc013d60 tx=0x7f48cc012040 comp rx=0 tx=0).stop 2024-03-31T23:15:30.518 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:30.516+0000 7f48e0f82740 1 -- 172.21.15.46:0/3752727468 shutdown_connections 2024-03-31T23:15:30.518 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:30.516+0000 7f48e0f82740 1 --2- 172.21.15.46:0/3752727468 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55dde1bfcdb0 0x55dde1c14b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:15:30.518 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:30.516+0000 7f48e0f82740 1 --2- 172.21.15.46:0/3752727468 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55dde1bf3fe0 0x55dde1c08720 unknown :-1 s=CLOSED pgs=678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:15:30.518 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:30.516+0000 7f48e0f82740 1 --2- 172.21.15.46:0/3752727468 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dde1bf36c0 0x55dde1bfc560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:15:30.518 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:30.516+0000 7f48e0f82740 1 -- 172.21.15.46:0/3752727468 >> 172.21.15.46:0/3752727468 conn(0x55dde1bedfe0 msgr2=0x55dde1bf8740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:15:30.518 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:30.516+0000 7f48e0f82740 1 -- 172.21.15.46:0/3752727468 shutdown_connections 2024-03-31T23:15:30.518 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:30.516+0000 7f48e0f82740 1 -- 172.21.15.46:0/3752727468 wait complete. 2024-03-31T23:15:30.518 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:30.516+0000 7f48e0f82740 1 -- start start 2024-03-31T23:15:30.521 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:30.517+0000 7f48e0f82740 1 Processor -- start 2024-03-31T23:15:30.521 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:30.517+0000 7f48e0f82740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dde1bf36c0 0x55dde1c05040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:15:30.521 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:30.517+0000 7f48e0f82740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55dde1bf3fe0 0x55dde1bf67c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:15:30.521 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:30.517+0000 7f48e0f82740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55dde1bfcdb0 0x55dde1bf6d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:15:30.521 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:30.517+0000 7f48e0f82740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f48d81af2f0 con 0x55dde1bf3fe0 2024-03-31T23:15:30.521 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:30.517+0000 7f48e0f82740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f48d81b0760 con 0x55dde1bfcdb0 2024-03-31T23:15:30.522 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:30.517+0000 7f48e0f82740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f48d81af9c0 con 0x55dde1bf36c0 2024-03-31T23:15:30.522 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:30.517+0000 7f48e0f82740 10 client.0 _ll_register_callbacks cb 0x7f48c8005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T23:15:30.522 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:30.517+0000 7f48d77fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55dde1bf3fe0 0x55dde1bf67c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:15:30.522 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:30.517+0000 7f48d77fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55dde1bf3fe0 0x55dde1bf67c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40324/0 (socket says 172.21.15.46:40324) 2024-03-31T23:15:30.522 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:30.517+0000 7f48d77fe640 1 -- 172.21.15.46:0/647413240 learned_addr learned my addr 172.21.15.46:0/647413240 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:15:30.522 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:30.517+0000 7f48d7fff640 1 --2- 172.21.15.46:0/647413240 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dde1bf36c0 0x55dde1c05040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:15:30.522 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:30.517+0000 7f48dc944640 1 --2- 172.21.15.46:0/647413240 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55dde1bfcdb0 0x55dde1bf6d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:15:30.522 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:30.517+0000 7f48d77fe640 1 -- 172.21.15.46:0/647413240 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dde1bf36c0 msgr2=0x55dde1c05040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:15:30.522 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:30.517+0000 7f48d77fe640 1 --2- 172.21.15.46:0/647413240 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dde1bf36c0 0x55dde1c05040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:15:30.522 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:30.517+0000 7f48d77fe640 1 -- 172.21.15.46:0/647413240 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55dde1bfcdb0 msgr2=0x55dde1bf6d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:15:30.522 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:30.517+0000 7f48d77fe640 1 --2- 172.21.15.46:0/647413240 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55dde1bfcdb0 0x55dde1bf6d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:15:30.522 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:30.517+0000 7f48d77fe640 1 -- 172.21.15.46:0/647413240 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dde1bf7530 con 0x55dde1bf3fe0 2024-03-31T23:15:30.522 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:30.517+0000 7f48d77fe640 1 --2- 172.21.15.46:0/647413240 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55dde1bf3fe0 0x55dde1bf67c0 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f48cc0097b0 tx=0x7f48cc005f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=596bf95719cefa78 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:15:30.523 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:30.517+0000 7f48b77fe640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-31T23:15:30.523 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:30.518+0000 7f48b77fe640 1 -- 172.21.15.46:0/647413240 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48cc02e070 con 0x55dde1bf3fe0 2024-03-31T23:15:30.523 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:30.518+0000 7f48b77fe640 1 -- 172.21.15.46:0/647413240 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7f48cc010700 con 0x55dde1bf3fe0 2024-03-31T23:15:30.523 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:30.518+0000 7f48e0f82740 1 -- 172.21.15.46:0/647413240 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f48cc010ba0 con 0x55dde1bf3fe0 2024-03-31T23:15:30.523 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:30.518+0000 7f48b77fe640 1 -- 172.21.15.46:0/647413240 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f48cc03f050 con 0x55dde1bf3fe0 2024-03-31T23:15:30.523 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:30.518+0000 7f48e0f82740 10 client.16461 fetch_fsmap learned FSMap version 28 2024-03-31T23:15:30.523 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:30.518+0000 7f48e0f82740 1 -- 172.21.15.46:0/647413240 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55dde1c73210 con 0x55dde1bf3fe0 2024-03-31T23:15:30.523 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:30.518+0000 7f48b77fe640 1 -- 172.21.15.46:0/647413240 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f48cc010ba0 con 0x55dde1bf3fe0 2024-03-31T23:15:30.523 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:30.518+0000 7f48b77fe640 1 -- 172.21.15.46:0/647413240 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f48cc03a030 con 0x55dde1bf3fe0 2024-03-31T23:15:30.523 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:30.518+0000 7f48e0f82740 10 client.16461 fetch_fsmap finished waiting for FSMap version 28 2024-03-31T23:15:30.523 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:30.518+0000 7f48e0f82740 10 client.16461 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-31T23:15:30.523 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:30.518+0000 7f48e0f82740 20 client.16461 populate_metadata read hostname 'smithi046' 2024-03-31T23:15:30.523 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:30.518+0000 7f48e0f82740 1 --2- 172.21.15.46:0/647413240 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55dde1c73a10 0x55dde1c75e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:15:30.523 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:30.518+0000 7f48e0f82740 4 client.16461 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-31T23:15:30.524 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:30.518+0000 7f48e0f82740 1 -- 172.21.15.46:0/647413240 --> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55dde1b63e30 con 0x55dde1c73a10 2024-03-31T23:15:30.524 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:30.518+0000 7f48d7fff640 1 --2- 172.21.15.46:0/647413240 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55dde1c73a10 0x55dde1c75e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:15:30.524 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:30.519+0000 7f48d7fff640 1 --2- 172.21.15.46:0/647413240 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55dde1c73a10 0x55dde1c75e30 crc :-1 s=READY pgs=108 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b272b9dfca4a1efc server_cookie=a5c0b80c07e60eae in_seq=0 out_seq=0 2024-03-31T23:15:30.524 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:30.519+0000 7f48b77fe640 10 client.16461 ms_handle_connect on v2:172.21.15.142:6838/2519143918 2024-03-31T23:15:30.540 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:30.539+0000 7f48b77fe640 1 -- 172.21.15.46:0/647413240 <== mds.0 v2:172.21.15.142:6838/2519143918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55dde1b63e30 con 0x55dde1c73a10 2024-03-31T23:15:30.540 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:30.539+0000 7f48b77fe640 10 client.16461 handle_command_reply: tid=0 2024-03-31T23:15:30.540 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:30.539+0000 7f48e0f82740 1 client.16461 shutdown 2024-03-31T23:15:30.540 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:30.540+0000 7f48e0f82740 1 -- 172.21.15.46:0/647413240 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55dde1bf3fe0 msgr2=0x55dde1bf67c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:15:30.540 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:30.540+0000 7f48e0f82740 1 --2- 172.21.15.46:0/647413240 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55dde1bf3fe0 0x55dde1bf67c0 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f48cc0097b0 tx=0x7f48cc005f10 comp rx=0 tx=0).stop 2024-03-31T23:15:30.540 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:30.540+0000 7f48e0f82740 1 -- 172.21.15.46:0/647413240 shutdown_connections 2024-03-31T23:15:30.540 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:30.540+0000 7f48e0f82740 1 --2- 172.21.15.46:0/647413240 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55dde1c73a10 0x55dde1c75e30 crc :-1 s=READY pgs=108 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:15:30.540 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:30.540+0000 7f48e0f82740 1 --2- 172.21.15.46:0/647413240 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55dde1bfcdb0 0x55dde1bf6d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:15:30.541 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:30.540+0000 7f48e0f82740 1 --2- 172.21.15.46:0/647413240 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55dde1bf3fe0 0x55dde1bf67c0 unknown :-1 s=CLOSED pgs=679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:15:30.541 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:30.540+0000 7f48e0f82740 1 --2- 172.21.15.46:0/647413240 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dde1bf36c0 0x55dde1c05040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:15:30.541 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:30.540+0000 7f48e0f82740 1 -- 172.21.15.46:0/647413240 >> 172.21.15.46:0/647413240 conn(0x55dde1bedfe0 msgr2=0x55dde1bec6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:15:30.541 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:30.540+0000 7f48b77fe640 0 client.16461 ms_handle_reset on v2:172.21.15.142:6838/2519143918 2024-03-31T23:15:30.541 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:30.540+0000 7f48e0f82740 1 -- 172.21.15.46:0/647413240 shutdown_connections 2024-03-31T23:15:30.541 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:30.540+0000 7f48e0f82740 1 -- 172.21.15.46:0/647413240 wait complete. 2024-03-31T23:15:30.541 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:30.540+0000 7f48e0f82740 20 client.16461 trim_cache size 0 max 16384 2024-03-31T23:15:30.541 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout:{ 2024-03-31T23:15:30.541 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout: "message": "", 2024-03-31T23:15:30.541 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout: "return_code": 0 2024-03-31T23:15:30.541 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout:} 2024-03-31T23:15:30.542 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:30.541+0000 7f48de3a7640 1 -- 172.21.15.46:0/142747992 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f48c00a26b0 msgr2=0x7f48c00a4b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:15:30.542 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:30.541+0000 7f48de3a7640 1 --2- 172.21.15.46:0/142747992 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f48c00a26b0 0x7f48c00a4b70 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f48cc013460 tx=0x7f48cc013860 comp rx=0 tx=0).stop 2024-03-31T23:15:30.542 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:30.541+0000 7f48de3a7640 1 -- 172.21.15.46:0/142747992 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f48d80ffe60 msgr2=0x7f48d819a0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:15:30.542 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:30.541+0000 7f48de3a7640 1 --2- 172.21.15.46:0/142747992 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f48d80ffe60 0x7f48d819a0b0 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f48c800e9f0 tx=0x7f48c800eeb0 comp rx=0 tx=0).stop 2024-03-31T23:15:30.542 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:30.541+0000 7f48de3a7640 1 -- 172.21.15.46:0/142747992 shutdown_connections 2024-03-31T23:15:30.543 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:30.541+0000 7f48de3a7640 1 --2- 172.21.15.46:0/142747992 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f48c00a26b0 0x7f48c00a4b70 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:15:30.543 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:30.541+0000 7f48de3a7640 1 --2- 172.21.15.46:0/142747992 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f48d819e970 0x7f48d819ee20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:15:30.543 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:30.541+0000 7f48de3a7640 1 --2- 172.21.15.46:0/142747992 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f48d8106980 0x7f48d819a5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:15:30.543 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:30.541+0000 7f48de3a7640 1 --2- 172.21.15.46:0/142747992 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f48d80ffe60 0x7f48d819a0b0 unknown :-1 s=CLOSED pgs=676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:15:30.543 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:30.541+0000 7f48de3a7640 1 -- 172.21.15.46:0/142747992 >> 172.21.15.46:0/142747992 conn(0x7f48d8076b70 msgr2=0x7f48d8105060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:15:30.543 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:30.541+0000 7f48de3a7640 1 -- 172.21.15.46:0/142747992 shutdown_connections 2024-03-31T23:15:30.543 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:30.541+0000 7f48de3a7640 1 -- 172.21.15.46:0/142747992 wait complete. 2024-03-31T23:15:30.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:15:30 smithi046 ceph-mon[32498]: pgmap v116: 129 pgs: 129 active+clean; 62 MiB data, 797 MiB used, 1.0 TiB / 1.0 TiB avail; 482 KiB/s rd, 1.8 MiB/s wr, 181 op/s 2024-03-31T23:15:30.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:15:30 smithi142 ceph-mon[35271]: pgmap v116: 129 pgs: 129 active+clean; 62 MiB data, 797 MiB used, 1.0 TiB / 1.0 TiB avail; 482 KiB/s rd, 1.8 MiB/s wr, 181 op/s 2024-03-31T23:15:31.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:15:30 smithi162 ceph-mon[36237]: pgmap v116: 129 pgs: 129 active+clean; 62 MiB data, 797 MiB used, 1.0 TiB / 1.0 TiB avail; 482 KiB/s rd, 1.8 MiB/s wr, 181 op/s 2024-03-31T23:15:31.195 INFO:tasks.workunit.client.0.smithi046.stdout: 1 72093 2.62 MB/sec execute 37 sec latency 23.498 ms 2024-03-31T23:15:32.195 INFO:tasks.workunit.client.0.smithi046.stdout: 1 72629 2.63 MB/sec execute 38 sec latency 23.702 ms 2024-03-31T23:15:32.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:15:32 smithi046 ceph-mon[32498]: pgmap v117: 129 pgs: 129 active+clean; 62 MiB data, 797 MiB used, 1.0 TiB / 1.0 TiB avail; 307 KiB/s rd, 1.3 MiB/s wr, 116 op/s 2024-03-31T23:15:32.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:15:32 smithi142 ceph-mon[35271]: pgmap v117: 129 pgs: 129 active+clean; 62 MiB data, 797 MiB used, 1.0 TiB / 1.0 TiB avail; 307 KiB/s rd, 1.3 MiB/s wr, 116 op/s 2024-03-31T23:15:33.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:15:32 smithi162 ceph-mon[36237]: pgmap v117: 129 pgs: 129 active+clean; 62 MiB data, 797 MiB used, 1.0 TiB / 1.0 TiB avail; 307 KiB/s rd, 1.3 MiB/s wr, 116 op/s 2024-03-31T23:15:33.195 INFO:tasks.workunit.client.0.smithi046.stdout: 1 73322 2.61 MB/sec execute 39 sec latency 16.902 ms 2024-03-31T23:15:34.195 INFO:tasks.workunit.client.0.smithi046.stdout: 1 73874 2.65 MB/sec execute 40 sec latency 16.709 ms 2024-03-31T23:15:34.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:15:34 smithi046 ceph-mon[32498]: pgmap v118: 129 pgs: 129 active+clean; 62 MiB data, 797 MiB used, 1.0 TiB / 1.0 TiB avail; 307 KiB/s rd, 1.3 MiB/s wr, 116 op/s 2024-03-31T23:15:34.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:15:34 smithi142 ceph-mon[35271]: pgmap v118: 129 pgs: 129 active+clean; 62 MiB data, 797 MiB used, 1.0 TiB / 1.0 TiB avail; 307 KiB/s rd, 1.3 MiB/s wr, 116 op/s 2024-03-31T23:15:35.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:15:34 smithi162 ceph-mon[36237]: pgmap v118: 129 pgs: 129 active+clean; 62 MiB data, 797 MiB used, 1.0 TiB / 1.0 TiB avail; 307 KiB/s rd, 1.3 MiB/s wr, 116 op/s 2024-03-31T23:15:35.195 INFO:tasks.workunit.client.0.smithi046.stdout: 1 74330 2.67 MB/sec execute 41 sec latency 16.244 ms 2024-03-31T23:15:36.195 INFO:tasks.workunit.client.0.smithi046.stdout: 1 74814 2.71 MB/sec execute 42 sec latency 15.544 ms 2024-03-31T23:15:36.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:15:36 smithi046 ceph-mon[32498]: pgmap v119: 129 pgs: 129 active+clean; 58 MiB data, 800 MiB used, 1.0 TiB / 1.0 TiB avail; 485 KiB/s rd, 1.8 MiB/s wr, 180 op/s 2024-03-31T23:15:36.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:15:36 smithi142 ceph-mon[35271]: pgmap v119: 129 pgs: 129 active+clean; 58 MiB data, 800 MiB used, 1.0 TiB / 1.0 TiB avail; 485 KiB/s rd, 1.8 MiB/s wr, 180 op/s 2024-03-31T23:15:37.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:15:36 smithi162 ceph-mon[36237]: pgmap v119: 129 pgs: 129 active+clean; 58 MiB data, 800 MiB used, 1.0 TiB / 1.0 TiB avail; 485 KiB/s rd, 1.8 MiB/s wr, 180 op/s 2024-03-31T23:15:37.195 INFO:tasks.workunit.client.0.smithi046.stdout: 1 75084 2.65 MB/sec execute 43 sec latency 16.298 ms 2024-03-31T23:15:38.195 INFO:tasks.workunit.client.0.smithi046.stdout: 1 75487 2.61 MB/sec execute 44 sec latency 16.249 ms 2024-03-31T23:15:38.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:15:38 smithi046 ceph-mon[32498]: pgmap v120: 129 pgs: 129 active+clean; 58 MiB data, 800 MiB used, 1.0 TiB / 1.0 TiB avail; 305 KiB/s rd, 1.1 MiB/s wr, 120 op/s 2024-03-31T23:15:38.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:15:38 smithi046 ceph-mon[32498]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-03-31T23:15:38.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:15:38 smithi142 ceph-mon[35271]: pgmap v120: 129 pgs: 129 active+clean; 58 MiB data, 800 MiB used, 1.0 TiB / 1.0 TiB avail; 305 KiB/s rd, 1.1 MiB/s wr, 120 op/s 2024-03-31T23:15:38.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:15:38 smithi142 ceph-mon[35271]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-03-31T23:15:39.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:15:38 smithi162 ceph-mon[36237]: pgmap v120: 129 pgs: 129 active+clean; 58 MiB data, 800 MiB used, 1.0 TiB / 1.0 TiB avail; 305 KiB/s rd, 1.1 MiB/s wr, 120 op/s 2024-03-31T23:15:39.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:15:38 smithi162 ceph-mon[36237]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-03-31T23:15:39.195 INFO:tasks.workunit.client.0.smithi046.stdout: 1 76037 2.65 MB/sec execute 45 sec latency 15.626 ms 2024-03-31T23:15:40.195 INFO:tasks.workunit.client.0.smithi046.stdout: 1 76625 2.60 MB/sec execute 46 sec latency 17.241 ms 2024-03-31T23:15:40.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:15:40 smithi046 ceph-mon[32498]: pgmap v121: 129 pgs: 129 active+clean; 63 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 484 KiB/s rd, 1.7 MiB/s wr, 179 op/s 2024-03-31T23:15:40.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:15:40 smithi142 ceph-mon[35271]: pgmap v121: 129 pgs: 129 active+clean; 63 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 484 KiB/s rd, 1.7 MiB/s wr, 179 op/s 2024-03-31T23:15:41.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:15:40 smithi162 ceph-mon[36237]: pgmap v121: 129 pgs: 129 active+clean; 63 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 484 KiB/s rd, 1.7 MiB/s wr, 179 op/s 2024-03-31T23:15:41.195 INFO:tasks.workunit.client.0.smithi046.stdout: 1 77119 2.59 MB/sec execute 47 sec latency 17.002 ms 2024-03-31T23:15:42.195 INFO:tasks.workunit.client.0.smithi046.stdout: 1 77833 2.67 MB/sec execute 48 sec latency 17.154 ms 2024-03-31T23:15:42.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:15:42 smithi046 ceph-mon[32498]: pgmap v122: 129 pgs: 129 active+clean; 63 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 358 KiB/s rd, 1.1 MiB/s wr, 123 op/s 2024-03-31T23:15:42.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:15:42 smithi142 ceph-mon[35271]: pgmap v122: 129 pgs: 129 active+clean; 63 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 358 KiB/s rd, 1.1 MiB/s wr, 123 op/s 2024-03-31T23:15:43.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:15:42 smithi162 ceph-mon[36237]: pgmap v122: 129 pgs: 129 active+clean; 63 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 358 KiB/s rd, 1.1 MiB/s wr, 123 op/s 2024-03-31T23:15:43.195 INFO:tasks.workunit.client.0.smithi046.stdout: 1 78224 2.69 MB/sec execute 49 sec latency 17.914 ms 2024-03-31T23:15:43.641 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:43.639+0000 7f5dbf23b640 1 -- 172.21.15.46:0/3171040772 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5db8068dd0 msgr2=0x7f5db8069230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:15:43.641 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:43.639+0000 7f5dbf23b640 1 --2- 172.21.15.46:0/3171040772 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5db8068dd0 0x7f5db8069230 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7f5da4009930 tx=0x7f5da402bc30 comp rx=0 tx=0).stop 2024-03-31T23:15:43.642 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:43.639+0000 7f5dbf23b640 1 -- 172.21.15.46:0/3171040772 shutdown_connections 2024-03-31T23:15:43.642 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:43.639+0000 7f5dbf23b640 1 --2- 172.21.15.46:0/3171040772 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5db8069770 0x7f5db810e5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:15:43.642 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:43.639+0000 7f5dbf23b640 1 --2- 172.21.15.46:0/3171040772 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5db8068dd0 0x7f5db8069230 unknown :-1 s=CLOSED pgs=680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:15:43.642 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:43.639+0000 7f5dbf23b640 1 --2- 172.21.15.46:0/3171040772 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5db8107820 0x7f5db8107c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:15:43.642 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:43.639+0000 7f5dbf23b640 1 -- 172.21.15.46:0/3171040772 >> 172.21.15.46:0/3171040772 conn(0x7f5db80774a0 msgr2=0x7f5db80778b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:15:43.642 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:43.639+0000 7f5dbf23b640 1 -- 172.21.15.46:0/3171040772 shutdown_connections 2024-03-31T23:15:43.642 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:43.639+0000 7f5dbf23b640 1 -- 172.21.15.46:0/3171040772 wait complete. 2024-03-31T23:15:43.642 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:43.639+0000 7f5dbf23b640 1 Processor -- start 2024-03-31T23:15:43.642 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:43.639+0000 7f5dbf23b640 1 -- start start 2024-03-31T23:15:43.642 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:43.640+0000 7f5dbf23b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5db8068dd0 0x7f5db8197f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:15:43.642 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:43.640+0000 7f5dbf23b640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5db8069770 0x7f5db8198460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:15:43.642 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:43.640+0000 7f5dbf23b640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5db8107820 0x7f5db819c7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:15:43.643 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:43.640+0000 7f5dbf23b640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f5db81119c0 con 0x7f5db8068dd0 2024-03-31T23:15:43.643 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:43.640+0000 7f5dbf23b640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f5db8111840 con 0x7f5db8069770 2024-03-31T23:15:43.643 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:43.640+0000 7f5dbf23b640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f5db8111b40 con 0x7f5db8107820 2024-03-31T23:15:43.643 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:43.640+0000 7f5dbcfd7640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5db8068dd0 0x7f5db8197f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:15:43.643 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:43.640+0000 7f5dbcfd7640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5db8068dd0 0x7f5db8197f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:33846/0 (socket says 172.21.15.46:33846) 2024-03-31T23:15:43.643 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:43.640+0000 7f5dbcfd7640 1 -- 172.21.15.46:0/4051371935 learned_addr learned my addr 172.21.15.46:0/4051371935 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:15:43.643 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:43.640+0000 7f5db7fff640 1 --2- 172.21.15.46:0/4051371935 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5db8069770 0x7f5db8198460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:15:43.643 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:43.640+0000 7f5dbd7d8640 1 --2- 172.21.15.46:0/4051371935 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5db8107820 0x7f5db819c7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:15:43.643 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:43.640+0000 7f5dbcfd7640 1 -- 172.21.15.46:0/4051371935 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5db8107820 msgr2=0x7f5db819c7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:15:43.643 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:43.640+0000 7f5dbcfd7640 1 --2- 172.21.15.46:0/4051371935 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5db8107820 0x7f5db819c7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:15:43.643 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:43.640+0000 7f5dbcfd7640 1 -- 172.21.15.46:0/4051371935 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5db8069770 msgr2=0x7f5db8198460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:15:43.643 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:43.640+0000 7f5dbcfd7640 1 --2- 172.21.15.46:0/4051371935 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5db8069770 0x7f5db8198460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:15:43.643 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:43.640+0000 7f5dbcfd7640 1 -- 172.21.15.46:0/4051371935 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5db819cf10 con 0x7f5db8068dd0 2024-03-31T23:15:43.643 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:43.640+0000 7f5db7fff640 1 --2- 172.21.15.46:0/4051371935 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5db8069770 0x7f5db8198460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:15:43.644 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:43.640+0000 7f5dbcfd7640 1 --2- 172.21.15.46:0/4051371935 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5db8068dd0 0x7f5db8197f20 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7f5dac00e9f0 tx=0x7f5dac00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:15:43.644 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:43.640+0000 7f5db5ffb640 1 -- 172.21.15.46:0/4051371935 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5dac00cd80 con 0x7f5db8068dd0 2024-03-31T23:15:43.644 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:43.640+0000 7f5db5ffb640 1 -- 172.21.15.46:0/4051371935 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5dac0108e0 con 0x7f5db8068dd0 2024-03-31T23:15:43.644 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:43.640+0000 7f5dbf23b640 1 -- 172.21.15.46:0/4051371935 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5db819d200 con 0x7f5db8068dd0 2024-03-31T23:15:43.644 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:43.640+0000 7f5db5ffb640 1 -- 172.21.15.46:0/4051371935 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5dac018920 con 0x7f5db8068dd0 2024-03-31T23:15:43.644 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:43.641+0000 7f5dbf23b640 1 -- 172.21.15.46:0/4051371935 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5db81a4a90 con 0x7f5db8068dd0 2024-03-31T23:15:43.650 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:43.643+0000 7f5db5ffb640 1 -- 172.21.15.46:0/4051371935 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 25) ==== 137451+0+0 (secure 0 0 0) 0x7f5dac010a90 con 0x7f5db8068dd0 2024-03-31T23:15:43.651 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:43.643+0000 7f5dc1e16740 1 Processor -- start 2024-03-31T23:15:43.651 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:43.643+0000 7f5dc1e16740 1 -- start start 2024-03-31T23:15:43.651 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:43.644+0000 7f5db5ffb640 1 --2- 172.21.15.46:0/4051371935 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f5da80a2730 0x7f5da80a4bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:15:43.651 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:43.644+0000 7f5dc1e16740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55b9df9676c0 0x55b9df967aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:15:43.651 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:43.644+0000 7f5dc1e16740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b9df967fe0 0x55b9df970870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:15:43.651 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:43.644+0000 7f5dc1e16740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b9df970db0 0x55b9df9731a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:15:43.651 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:43.644+0000 7f5dc1e16740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55b9df8ca8f0 con 0x55b9df970db0 2024-03-31T23:15:43.651 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:43.644+0000 7f5dc1e16740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55b9df8738d0 con 0x55b9df9676c0 2024-03-31T23:15:43.651 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:43.644+0000 7f5dc1e16740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55b9df8c6990 con 0x55b9df967fe0 2024-03-31T23:15:43.651 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:43.644+0000 7f5dbd7d8640 1 --2- 172.21.15.46:0/4051371935 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f5da80a2730 0x7f5da80a4bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:15:43.652 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:43.644+0000 7f5db5ffb640 1 -- 172.21.15.46:0/4051371935 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7f5dac0cf0f0 con 0x7f5db8068dd0 2024-03-31T23:15:43.652 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:43.644+0000 7f5db7fff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b9df967fe0 0x55b9df970870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:15:43.652 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:43.644+0000 7f5dbd7d8640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b9df970db0 0x55b9df9731a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:15:43.652 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:43.644+0000 7f5dbcfd7640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55b9df9676c0 0x55b9df967aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:15:43.652 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:43.644+0000 7f5db7fff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b9df967fe0 0x55b9df970870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.46:47796/0 (socket says 172.21.15.46:47796) 2024-03-31T23:15:43.652 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:43.644+0000 7f5db7fff640 1 -- 172.21.15.46:0/3172360514 learned_addr learned my addr 172.21.15.46:0/3172360514 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:15:43.652 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:43.644+0000 7f5db7fff640 1 -- 172.21.15.46:0/3172360514 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55b9df9676c0 msgr2=0x55b9df967aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:15:43.652 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:43.644+0000 7f5db7fff640 1 --2- 172.21.15.46:0/3172360514 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55b9df9676c0 0x55b9df967aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:15:43.652 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:43.644+0000 7f5db7fff640 1 -- 172.21.15.46:0/3172360514 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b9df970db0 msgr2=0x55b9df9731a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:15:43.652 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:43.644+0000 7f5db7fff640 1 --2- 172.21.15.46:0/3172360514 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b9df970db0 0x55b9df9731a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:15:43.652 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:43.644+0000 7f5db7fff640 1 -- 172.21.15.46:0/3172360514 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b9df5df210 con 0x55b9df967fe0 2024-03-31T23:15:43.652 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:43.644+0000 7f5dbcfd7640 1 --2- 172.21.15.46:0/3172360514 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55b9df9676c0 0x55b9df967aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:15:43.652 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:43.644+0000 7f5dbd7d8640 1 --2- 172.21.15.46:0/4051371935 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f5da80a2730 0x7f5da80a4bf0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f5db0005fd0 tx=0x7f5db0009e50 comp rx=0 tx=0).ready entity=mgr.15654 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:15:43.652 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:43.644+0000 7f5db7fff640 1 --2- 172.21.15.46:0/3172360514 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b9df967fe0 0x55b9df970870 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f5da4009930 tx=0x7f5da4009660 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cddde2220cde277c server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:15:43.653 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:43.645+0000 7f5da2ffd640 1 -- 172.21.15.46:0/3172360514 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5da402c780 con 0x55b9df967fe0 2024-03-31T23:15:43.653 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:43.645+0000 7f5da2ffd640 1 -- 172.21.15.46:0/3172360514 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5da402c920 con 0x55b9df967fe0 2024-03-31T23:15:43.653 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:43.645+0000 7f5da2ffd640 1 -- 172.21.15.46:0/3172360514 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5da4003ba0 con 0x55b9df967fe0 2024-03-31T23:15:43.653 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:43.645+0000 7f5dc1e16740 1 -- 172.21.15.46:0/3172360514 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b9df967fe0 msgr2=0x55b9df970870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:15:43.653 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:43.645+0000 7f5dc1e16740 1 --2- 172.21.15.46:0/3172360514 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b9df967fe0 0x55b9df970870 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f5da4009930 tx=0x7f5da4009660 comp rx=0 tx=0).stop 2024-03-31T23:15:43.653 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:43.645+0000 7f5dc1e16740 1 -- 172.21.15.46:0/3172360514 shutdown_connections 2024-03-31T23:15:43.653 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:43.645+0000 7f5dc1e16740 1 --2- 172.21.15.46:0/3172360514 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b9df970db0 0x55b9df9731a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:15:43.653 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:43.645+0000 7f5dc1e16740 1 --2- 172.21.15.46:0/3172360514 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b9df967fe0 0x55b9df970870 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:15:43.653 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:43.645+0000 7f5dc1e16740 1 --2- 172.21.15.46:0/3172360514 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55b9df9676c0 0x55b9df967aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:15:43.653 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:43.645+0000 7f5dc1e16740 1 -- 172.21.15.46:0/3172360514 >> 172.21.15.46:0/3172360514 conn(0x55b9df961fe0 msgr2=0x55b9df9623c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:15:43.653 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:43.645+0000 7f5dc1e16740 1 -- 172.21.15.46:0/3172360514 shutdown_connections 2024-03-31T23:15:43.653 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:43.645+0000 7f5dc1e16740 1 -- 172.21.15.46:0/3172360514 wait complete. 2024-03-31T23:15:43.653 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:43.646+0000 7f5dc1e16740 1 -- start start 2024-03-31T23:15:43.653 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:43.646+0000 7f5dc1e16740 1 Processor -- start 2024-03-31T23:15:43.653 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:43.646+0000 7f5dc1e16740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b9df9676c0 0x55b9df96dd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:15:43.654 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:43.646+0000 7f5dc1e16740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b9df967fe0 0x55b9df96e2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:15:43.654 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:43.646+0000 7f5dc1e16740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55b9df970db0 0x55b9df96e810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:15:43.654 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:43.646+0000 7f5dc1e16740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55b9df973980 con 0x55b9df967fe0 2024-03-31T23:15:43.654 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:43.646+0000 7f5dc1e16740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55b9df868f80 con 0x55b9df970db0 2024-03-31T23:15:43.654 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:43.646+0000 7f5dc1e16740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55b9df973b00 con 0x55b9df9676c0 2024-03-31T23:15:43.654 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:43.646+0000 7f5dc1e16740 10 client.0 _ll_register_callbacks cb 0x55b9df8c1290 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T23:15:43.654 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:43.646+0000 7f5db7fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b9df967fe0 0x55b9df96e2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:15:43.654 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:43.646+0000 7f5dbcfd7640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b9df9676c0 0x55b9df96dd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:15:43.654 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:43.646+0000 7f5db7fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b9df967fe0 0x55b9df96e2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:33856/0 (socket says 172.21.15.46:33856) 2024-03-31T23:15:43.654 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:43.646+0000 7f5db7fff640 1 -- 172.21.15.46:0/1981323729 learned_addr learned my addr 172.21.15.46:0/1981323729 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:15:43.654 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:43.646+0000 7f5dbd7d8640 1 --2- 172.21.15.46:0/1981323729 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55b9df970db0 0x55b9df96e810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:15:43.654 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:43.647+0000 7f5dbcfd7640 1 -- 172.21.15.46:0/1981323729 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55b9df970db0 msgr2=0x55b9df96e810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:15:43.654 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:43.647+0000 7f5dbcfd7640 1 --2- 172.21.15.46:0/1981323729 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55b9df970db0 0x55b9df96e810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:15:43.655 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:43.647+0000 7f5dbcfd7640 1 -- 172.21.15.46:0/1981323729 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b9df967fe0 msgr2=0x55b9df96e2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:15:43.655 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:43.647+0000 7f5dbcfd7640 1 --2- 172.21.15.46:0/1981323729 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b9df967fe0 0x55b9df96e2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:15:43.655 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:43.647+0000 7f5dbcfd7640 1 -- 172.21.15.46:0/1981323729 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b9df6ab620 con 0x55b9df9676c0 2024-03-31T23:15:43.655 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:43.647+0000 7f5db7fff640 1 --2- 172.21.15.46:0/1981323729 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b9df967fe0 0x55b9df96e2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:15:43.655 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:43.647+0000 7f5dbd7d8640 1 --2- 172.21.15.46:0/1981323729 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55b9df970db0 0x55b9df96e810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:15:43.655 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:43.647+0000 7f5dbcfd7640 1 --2- 172.21.15.46:0/1981323729 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b9df9676c0 0x55b9df96dd90 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f5dac0cc490 tx=0x7f5dac010ea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=11c836f017fd0544 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:15:43.655 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:43.647+0000 7f5d97fff640 10 client.0 ms_handle_connect on v2:172.21.15.162:3300/0 2024-03-31T23:15:43.655 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:43.647+0000 7f5d97fff640 1 -- 172.21.15.46:0/1981323729 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5dac0a3ed0 con 0x55b9df9676c0 2024-03-31T23:15:43.655 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:43.647+0000 7f5d97fff640 1 -- 172.21.15.46:0/1981323729 <== mon.1 v2:172.21.15.162:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7f5dac0bb0d0 con 0x55b9df9676c0 2024-03-31T23:15:43.655 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:43.647+0000 7f5dc1e16740 1 -- 172.21.15.46:0/1981323729 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5da402c920 con 0x55b9df9676c0 2024-03-31T23:15:43.655 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:43.647+0000 7f5d97fff640 1 -- 172.21.15.46:0/1981323729 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f5dac0bb2d0 con 0x55b9df9676c0 2024-03-31T23:15:43.655 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:43.647+0000 7f5dc1e16740 10 client.26209 fetch_fsmap learned FSMap version 30 2024-03-31T23:15:43.655 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:43.647+0000 7f5dc1e16740 1 -- 172.21.15.46:0/1981323729 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55b9df97f840 con 0x55b9df9676c0 2024-03-31T23:15:43.655 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:43.647+0000 7f5d97fff640 1 -- 172.21.15.46:0/1981323729 <== mon.1 v2:172.21.15.162:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5da402c920 con 0x55b9df9676c0 2024-03-31T23:15:43.655 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:43.648+0000 7f5d97fff640 1 -- 172.21.15.46:0/1981323729 <== mon.1 v2:172.21.15.162:3300/0 5 ==== fsmap(e 30) ==== 8346+0+0 (secure 0 0 0) 0x7f5dac09f1e0 con 0x55b9df9676c0 2024-03-31T23:15:43.656 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:43.648+0000 7f5dc1e16740 10 client.26209 fetch_fsmap finished waiting for FSMap version 30 2024-03-31T23:15:43.656 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:43.648+0000 7f5dc1e16740 10 client.26209 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-31T23:15:43.656 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:43.648+0000 7f5dc1e16740 20 client.26209 populate_metadata read hostname 'smithi046' 2024-03-31T23:15:43.656 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:43.648+0000 7f5dc1e16740 1 --2- 172.21.15.46:0/1981323729 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55b9df980d00 0x55b9df9830f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:15:43.656 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:43.648+0000 7f5dc1e16740 4 client.26209 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-31T23:15:43.656 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:43.648+0000 7f5dc1e16740 1 -- 172.21.15.46:0/1981323729 --> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b9df8de120 con 0x55b9df980d00 2024-03-31T23:15:43.656 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:43.648+0000 7f5db7fff640 1 --2- 172.21.15.46:0/1981323729 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55b9df980d00 0x55b9df9830f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:15:43.656 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:43.648+0000 7f5db7fff640 1 --2- 172.21.15.46:0/1981323729 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55b9df980d00 0x55b9df9830f0 crc :-1 s=READY pgs=109 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=81be34a9d5f0461f server_cookie=ccb528f16054ccc7 in_seq=0 out_seq=0 2024-03-31T23:15:43.656 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:43.648+0000 7f5d97fff640 10 client.26209 ms_handle_connect on v2:172.21.15.142:6838/2519143918 2024-03-31T23:15:43.656 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:43.649+0000 7f5d97fff640 1 -- 172.21.15.46:0/1981323729 <== mds.0 v2:172.21.15.142:6838/2519143918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55b9df8de120 con 0x55b9df980d00 2024-03-31T23:15:43.656 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:43.649+0000 7f5d97fff640 10 client.26209 handle_command_reply: tid=0 2024-03-31T23:15:43.656 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:43.649+0000 7f5dc1e16740 1 client.26209 shutdown 2024-03-31T23:15:43.656 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:43.649+0000 7f5dc1e16740 1 -- 172.21.15.46:0/1981323729 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b9df9676c0 msgr2=0x55b9df96dd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:15:43.656 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:43.649+0000 7f5dc1e16740 1 --2- 172.21.15.46:0/1981323729 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b9df9676c0 0x55b9df96dd90 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f5dac0cc490 tx=0x7f5dac010ea0 comp rx=0 tx=0).stop 2024-03-31T23:15:43.657 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:43.649+0000 7f5dc1e16740 1 -- 172.21.15.46:0/1981323729 shutdown_connections 2024-03-31T23:15:43.657 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:43.649+0000 7f5dc1e16740 1 --2- 172.21.15.46:0/1981323729 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55b9df980d00 0x55b9df9830f0 crc :-1 s=READY pgs=109 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:15:43.657 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:43.649+0000 7f5dc1e16740 1 --2- 172.21.15.46:0/1981323729 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55b9df970db0 0x55b9df96e810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:15:43.657 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:43.649+0000 7f5dc1e16740 1 --2- 172.21.15.46:0/1981323729 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b9df967fe0 0x55b9df96e2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:15:43.657 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:43.649+0000 7f5dc1e16740 1 --2- 172.21.15.46:0/1981323729 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b9df9676c0 0x55b9df96dd90 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:15:43.657 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:43.649+0000 7f5d97fff640 0 client.26209 ms_handle_reset on v2:172.21.15.142:6838/2519143918 2024-03-31T23:15:43.657 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:43.649+0000 7f5dc1e16740 1 -- 172.21.15.46:0/1981323729 >> 172.21.15.46:0/1981323729 conn(0x55b9df961fe0 msgr2=0x55b9df963d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:15:43.657 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:43.649+0000 7f5dc1e16740 1 -- 172.21.15.46:0/1981323729 shutdown_connections 2024-03-31T23:15:43.657 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:43.649+0000 7f5dc1e16740 1 -- 172.21.15.46:0/1981323729 wait complete. 2024-03-31T23:15:43.657 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:43.649+0000 7f5dc1e16740 20 client.26209 trim_cache size 0 max 16384 2024-03-31T23:15:43.669 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:43.664+0000 7f5dc1e16740 1 Processor -- start 2024-03-31T23:15:43.669 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:43.664+0000 7f5dc1e16740 1 -- start start 2024-03-31T23:15:43.669 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:43.664+0000 7f5dc1e16740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b9df9676c0 0x55b9df9704c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:15:43.669 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:43.664+0000 7f5dc1e16740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55b9df967fe0 0x55b9df97c720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:15:43.669 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:43.665+0000 7f5dc1e16740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b9df970db0 0x55b9df9623c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:15:43.670 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:43.665+0000 7f5dc1e16740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f5db81b13b0 con 0x55b9df9676c0 2024-03-31T23:15:43.670 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:43.665+0000 7f5dc1e16740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f5db81aec80 con 0x55b9df967fe0 2024-03-31T23:15:43.670 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:43.665+0000 7f5dc1e16740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f5db81acad0 con 0x55b9df970db0 2024-03-31T23:15:43.670 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:43.665+0000 7f5dbcfd7640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b9df9676c0 0x55b9df9704c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:15:43.670 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:43.665+0000 7f5dbcfd7640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b9df9676c0 0x55b9df9704c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:33872/0 (socket says 172.21.15.46:33872) 2024-03-31T23:15:43.670 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:43.665+0000 7f5dbcfd7640 1 -- 172.21.15.46:0/2557298819 learned_addr learned my addr 172.21.15.46:0/2557298819 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:15:43.670 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:43.665+0000 7f5db7fff640 1 --2- 172.21.15.46:0/2557298819 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55b9df967fe0 0x55b9df97c720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:15:43.670 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:43.665+0000 7f5dbd7d8640 1 --2- 172.21.15.46:0/2557298819 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b9df970db0 0x55b9df9623c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:15:43.670 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:43.665+0000 7f5dbcfd7640 1 -- 172.21.15.46:0/2557298819 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b9df970db0 msgr2=0x55b9df9623c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:15:43.670 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:43.665+0000 7f5dbcfd7640 1 --2- 172.21.15.46:0/2557298819 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b9df970db0 0x55b9df9623c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:15:43.670 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:43.665+0000 7f5dbcfd7640 1 -- 172.21.15.46:0/2557298819 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55b9df967fe0 msgr2=0x55b9df97c720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:15:43.670 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:43.665+0000 7f5dbcfd7640 1 --2- 172.21.15.46:0/2557298819 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55b9df967fe0 0x55b9df97c720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:15:43.670 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:43.665+0000 7f5dbcfd7640 1 -- 172.21.15.46:0/2557298819 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b9df96cd40 con 0x55b9df9676c0 2024-03-31T23:15:43.671 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:43.665+0000 7f5dbd7d8640 1 --2- 172.21.15.46:0/2557298819 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b9df970db0 0x55b9df9623c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T23:15:43.671 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:43.665+0000 7f5dbcfd7640 1 --2- 172.21.15.46:0/2557298819 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b9df9676c0 0x55b9df9704c0 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7f5dac09b500 tx=0x7f5dac09f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ebb78ac10f2f19b server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:15:43.671 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:43.665+0000 7f5da27fc640 1 -- 172.21.15.46:0/2557298819 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5dac0b4230 con 0x55b9df9676c0 2024-03-31T23:15:43.671 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:43.665+0000 7f5da27fc640 1 -- 172.21.15.46:0/2557298819 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5dac0bb0d0 con 0x55b9df9676c0 2024-03-31T23:15:43.671 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:43.665+0000 7f5da27fc640 1 -- 172.21.15.46:0/2557298819 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5dac0a3a40 con 0x55b9df9676c0 2024-03-31T23:15:43.672 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:43.666+0000 7f5dc1e16740 1 -- 172.21.15.46:0/2557298819 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b9df9676c0 msgr2=0x55b9df9704c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:15:43.672 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:43.666+0000 7f5dc1e16740 1 --2- 172.21.15.46:0/2557298819 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b9df9676c0 0x55b9df9704c0 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7f5dac09b500 tx=0x7f5dac09f790 comp rx=0 tx=0).stop 2024-03-31T23:15:43.672 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:43.666+0000 7f5dc1e16740 1 -- 172.21.15.46:0/2557298819 shutdown_connections 2024-03-31T23:15:43.672 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:43.666+0000 7f5dc1e16740 1 --2- 172.21.15.46:0/2557298819 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b9df970db0 0x55b9df9623c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:15:43.672 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:43.666+0000 7f5dc1e16740 1 --2- 172.21.15.46:0/2557298819 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55b9df967fe0 0x55b9df97c720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:15:43.672 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:43.666+0000 7f5dc1e16740 1 --2- 172.21.15.46:0/2557298819 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b9df9676c0 0x55b9df9704c0 unknown :-1 s=CLOSED pgs=682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:15:43.672 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:43.666+0000 7f5dc1e16740 1 -- 172.21.15.46:0/2557298819 >> 172.21.15.46:0/2557298819 conn(0x55b9df961fe0 msgr2=0x55b9df960f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:15:43.672 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:43.666+0000 7f5dc1e16740 1 -- 172.21.15.46:0/2557298819 shutdown_connections 2024-03-31T23:15:43.672 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:43.666+0000 7f5dc1e16740 1 -- 172.21.15.46:0/2557298819 wait complete. 2024-03-31T23:15:43.672 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:43.666+0000 7f5dc1e16740 1 -- start start 2024-03-31T23:15:43.672 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:43.666+0000 7f5dc1e16740 1 Processor -- start 2024-03-31T23:15:43.672 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:43.667+0000 7f5dc1e16740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b9df9676c0 0x55b9df96af10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:15:43.672 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:43.667+0000 7f5dc1e16740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55b9df967fe0 0x55b9df96b450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:15:43.673 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:43.667+0000 7f5dc1e16740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b9df970db0 0x55b9df9e6030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:15:43.673 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:43.667+0000 7f5dc1e16740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f5db81ad140 con 0x55b9df970db0 2024-03-31T23:15:43.673 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:43.667+0000 7f5dc1e16740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f5db81ae5b0 con 0x55b9df967fe0 2024-03-31T23:15:43.673 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:43.667+0000 7f5dc1e16740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f5db81ad810 con 0x55b9df9676c0 2024-03-31T23:15:43.673 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:43.667+0000 7f5dc1e16740 10 client.0 _ll_register_callbacks cb 0x7f5dac00e940 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T23:15:43.673 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:43.667+0000 7f5dbd7d8640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b9df970db0 0x55b9df9e6030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:15:43.673 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:43.667+0000 7f5dbcfd7640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b9df9676c0 0x55b9df96af10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:15:43.673 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:43.667+0000 7f5db7fff640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55b9df967fe0 0x55b9df96b450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:15:43.673 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:43.667+0000 7f5dbd7d8640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b9df970db0 0x55b9df9e6030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:33888/0 (socket says 172.21.15.46:33888) 2024-03-31T23:15:43.673 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:43.667+0000 7f5dbd7d8640 1 -- 172.21.15.46:0/1521192785 learned_addr learned my addr 172.21.15.46:0/1521192785 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:15:43.673 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:43.667+0000 7f5dbcfd7640 1 -- 172.21.15.46:0/1521192785 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55b9df967fe0 msgr2=0x55b9df96b450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:15:43.673 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:43.667+0000 7f5dbcfd7640 1 --2- 172.21.15.46:0/1521192785 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55b9df967fe0 0x55b9df96b450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:15:43.673 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:43.667+0000 7f5dbcfd7640 1 -- 172.21.15.46:0/1521192785 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b9df970db0 msgr2=0x55b9df9e6030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:15:43.674 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:43.667+0000 7f5dbcfd7640 1 --2- 172.21.15.46:0/1521192785 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b9df970db0 0x55b9df9e6030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:15:43.674 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:43.667+0000 7f5dbcfd7640 1 -- 172.21.15.46:0/1521192785 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b9df967e10 con 0x55b9df9676c0 2024-03-31T23:15:43.674 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:43.667+0000 7f5dbd7d8640 1 --2- 172.21.15.46:0/1521192785 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b9df970db0 0x55b9df9e6030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:15:43.674 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:43.667+0000 7f5dbcfd7640 1 --2- 172.21.15.46:0/1521192785 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b9df9676c0 0x55b9df96af10 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f5dac0bb2a0 tx=0x7f5dac0bb470 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e9d39efb618a692f server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:15:43.674 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:43.667+0000 7f5d97fff640 10 client.0 ms_handle_connect on v2:172.21.15.162:3300/0 2024-03-31T23:15:43.674 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:43.668+0000 7f5d97fff640 1 -- 172.21.15.46:0/1521192785 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5dac0b4230 con 0x55b9df9676c0 2024-03-31T23:15:43.674 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:43.668+0000 7f5d97fff640 1 -- 172.21.15.46:0/1521192785 <== mon.1 v2:172.21.15.162:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7f5dac0a3a40 con 0x55b9df9676c0 2024-03-31T23:15:43.674 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:43.668+0000 7f5dc1e16740 1 -- 172.21.15.46:0/1521192785 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5dac0bb0d0 con 0x55b9df9676c0 2024-03-31T23:15:43.674 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:43.668+0000 7f5d97fff640 1 -- 172.21.15.46:0/1521192785 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f5dac04ce30 con 0x55b9df9676c0 2024-03-31T23:15:43.674 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:43.668+0000 7f5dc1e16740 10 client.26215 fetch_fsmap learned FSMap version 30 2024-03-31T23:15:43.674 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:43.668+0000 7f5dc1e16740 1 -- 172.21.15.46:0/1521192785 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55b9df9e7570 con 0x55b9df9676c0 2024-03-31T23:15:43.674 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:43.668+0000 7f5d97fff640 1 -- 172.21.15.46:0/1521192785 <== mon.1 v2:172.21.15.162:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5dac0bb0d0 con 0x55b9df9676c0 2024-03-31T23:15:43.674 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:43.668+0000 7f5d97fff640 1 -- 172.21.15.46:0/1521192785 <== mon.1 v2:172.21.15.162:3300/0 5 ==== fsmap(e 30) ==== 8346+0+0 (secure 0 0 0) 0x7f5dac0972c0 con 0x55b9df9676c0 2024-03-31T23:15:43.674 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:43.668+0000 7f5dc1e16740 10 client.26215 fetch_fsmap finished waiting for FSMap version 30 2024-03-31T23:15:43.674 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:43.668+0000 7f5dc1e16740 10 client.26215 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-31T23:15:43.675 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:43.668+0000 7f5dc1e16740 20 client.26215 populate_metadata read hostname 'smithi046' 2024-03-31T23:15:43.675 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:43.668+0000 7f5dc1e16740 1 --2- 172.21.15.46:0/1521192785 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55b9df9e7860 0x55b9df9e9c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:15:43.675 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:43.668+0000 7f5dc1e16740 4 client.26215 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-31T23:15:43.675 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:43.668+0000 7f5dc1e16740 1 -- 172.21.15.46:0/1521192785 --> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55b9df8d7e30 con 0x55b9df9e7860 2024-03-31T23:15:43.675 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:43.668+0000 7f5db7fff640 1 --2- 172.21.15.46:0/1521192785 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55b9df9e7860 0x55b9df9e9c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:15:43.675 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:43.669+0000 7f5d97fff640 10 client.26215 ms_handle_connect on v2:172.21.15.142:6838/2519143918 2024-03-31T23:15:43.675 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:43.669+0000 7f5db7fff640 1 --2- 172.21.15.46:0/1521192785 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55b9df9e7860 0x55b9df9e9c50 crc :-1 s=READY pgs=110 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8dd5b63adc093b16 server_cookie=ff8751ac1befcd7 in_seq=0 out_seq=0 2024-03-31T23:15:43.686 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:43.685+0000 7f5d97fff640 1 -- 172.21.15.46:0/1521192785 <== mds.0 v2:172.21.15.142:6838/2519143918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55b9df8d7e30 con 0x55b9df9e7860 2024-03-31T23:15:43.686 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:43.685+0000 7f5d97fff640 10 client.26215 handle_command_reply: tid=0 2024-03-31T23:15:43.686 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:43.685+0000 7f5dc1e16740 1 client.26215 shutdown 2024-03-31T23:15:43.686 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:43.686+0000 7f5dc1e16740 1 -- 172.21.15.46:0/1521192785 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b9df9676c0 msgr2=0x55b9df96af10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:15:43.686 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:43.686+0000 7f5dc1e16740 1 --2- 172.21.15.46:0/1521192785 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b9df9676c0 0x55b9df96af10 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f5dac0bb2a0 tx=0x7f5dac0bb470 comp rx=0 tx=0).stop 2024-03-31T23:15:43.686 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:43.686+0000 7f5dc1e16740 1 -- 172.21.15.46:0/1521192785 shutdown_connections 2024-03-31T23:15:43.686 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:43.686+0000 7f5dc1e16740 1 --2- 172.21.15.46:0/1521192785 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55b9df9e7860 0x55b9df9e9c50 crc :-1 s=READY pgs=110 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:15:43.687 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:43.686+0000 7f5dc1e16740 1 --2- 172.21.15.46:0/1521192785 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b9df970db0 0x55b9df9e6030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:15:43.687 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:43.686+0000 7f5dc1e16740 1 --2- 172.21.15.46:0/1521192785 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55b9df967fe0 0x55b9df96b450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:15:43.687 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:43.686+0000 7f5dc1e16740 1 --2- 172.21.15.46:0/1521192785 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b9df9676c0 0x55b9df96af10 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:15:43.687 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:43.686+0000 7f5d97fff640 0 client.26215 ms_handle_reset on v2:172.21.15.142:6838/2519143918 2024-03-31T23:15:43.687 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:43.686+0000 7f5dc1e16740 1 -- 172.21.15.46:0/1521192785 >> 172.21.15.46:0/1521192785 conn(0x55b9df961fe0 msgr2=0x55b9df960f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:15:43.687 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:43.686+0000 7f5dc1e16740 1 -- 172.21.15.46:0/1521192785 shutdown_connections 2024-03-31T23:15:43.687 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:43.686+0000 7f5dc1e16740 1 -- 172.21.15.46:0/1521192785 wait complete. 2024-03-31T23:15:43.687 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:43.686+0000 7f5dc1e16740 20 client.26215 trim_cache size 0 max 16384 2024-03-31T23:15:43.687 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout:{ 2024-03-31T23:15:43.687 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout: "message": "", 2024-03-31T23:15:43.687 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout: "return_code": 0 2024-03-31T23:15:43.688 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout:} 2024-03-31T23:15:43.689 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:43.686+0000 7f5dbf23b640 1 -- 172.21.15.46:0/4051371935 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f5da80a2730 msgr2=0x7f5da80a4bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:15:43.689 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:43.686+0000 7f5dbf23b640 1 --2- 172.21.15.46:0/4051371935 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f5da80a2730 0x7f5da80a4bf0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f5db0005fd0 tx=0x7f5db0009e50 comp rx=0 tx=0).stop 2024-03-31T23:15:43.689 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:43.686+0000 7f5dbf23b640 1 -- 172.21.15.46:0/4051371935 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5db8068dd0 msgr2=0x7f5db8197f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:15:43.689 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:43.686+0000 7f5dbf23b640 1 --2- 172.21.15.46:0/4051371935 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5db8068dd0 0x7f5db8197f20 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7f5dac00e9f0 tx=0x7f5dac00eeb0 comp rx=0 tx=0).stop 2024-03-31T23:15:43.689 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:43.686+0000 7f5dbf23b640 1 -- 172.21.15.46:0/4051371935 shutdown_connections 2024-03-31T23:15:43.689 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:43.686+0000 7f5dbf23b640 1 --2- 172.21.15.46:0/4051371935 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f5da80a2730 0x7f5da80a4bf0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:15:43.689 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:43.686+0000 7f5dbf23b640 1 --2- 172.21.15.46:0/4051371935 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5db8107820 0x7f5db819c7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:15:43.689 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:43.686+0000 7f5dbf23b640 1 --2- 172.21.15.46:0/4051371935 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5db8069770 0x7f5db8198460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:15:43.689 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:43.686+0000 7f5dbf23b640 1 --2- 172.21.15.46:0/4051371935 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5db8068dd0 0x7f5db8197f20 unknown :-1 s=CLOSED pgs=681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:15:43.689 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:43.686+0000 7f5dbf23b640 1 -- 172.21.15.46:0/4051371935 >> 172.21.15.46:0/4051371935 conn(0x7f5db80774a0 msgr2=0x7f5db80fe9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:15:43.689 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:43.687+0000 7f5dbf23b640 1 -- 172.21.15.46:0/4051371935 shutdown_connections 2024-03-31T23:15:43.690 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:43.687+0000 7f5dbf23b640 1 -- 172.21.15.46:0/4051371935 wait complete. 2024-03-31T23:15:43.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:15:43 smithi046 ceph-mon[32498]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-03-31T23:15:43.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:15:43 smithi142 ceph-mon[35271]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-03-31T23:15:44.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:15:43 smithi162 ceph-mon[36237]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-03-31T23:15:44.195 INFO:tasks.workunit.client.0.smithi046.stdout: 1 78504 2.66 MB/sec execute 50 sec latency 26.446 ms 2024-03-31T23:15:44.862 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:15:44 smithi046 ceph-mon[32498]: pgmap v123: 129 pgs: 129 active+clean; 63 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 358 KiB/s rd, 1.1 MiB/s wr, 123 op/s 2024-03-31T23:15:44.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:15:44 smithi142 ceph-mon[35271]: pgmap v123: 129 pgs: 129 active+clean; 63 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 358 KiB/s rd, 1.1 MiB/s wr, 123 op/s 2024-03-31T23:15:45.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:15:44 smithi162 ceph-mon[36237]: pgmap v123: 129 pgs: 129 active+clean; 63 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 358 KiB/s rd, 1.1 MiB/s wr, 123 op/s 2024-03-31T23:15:45.195 INFO:tasks.workunit.client.0.smithi046.stdout: 1 78828 2.62 MB/sec execute 51 sec latency 16.094 ms 2024-03-31T23:15:46.195 INFO:tasks.workunit.client.0.smithi046.stdout: 1 79293 2.60 MB/sec execute 52 sec latency 20.257 ms 2024-03-31T23:15:46.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:15:46 smithi046 ceph-mon[32498]: pgmap v124: 129 pgs: 129 active+clean; 64 MiB data, 853 MiB used, 1.0 TiB / 1.0 TiB avail; 489 KiB/s rd, 1.8 MiB/s wr, 182 op/s 2024-03-31T23:15:46.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:15:46 smithi142 ceph-mon[35271]: pgmap v124: 129 pgs: 129 active+clean; 64 MiB data, 853 MiB used, 1.0 TiB / 1.0 TiB avail; 489 KiB/s rd, 1.8 MiB/s wr, 182 op/s 2024-03-31T23:15:47.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:15:46 smithi162 ceph-mon[36237]: pgmap v124: 129 pgs: 129 active+clean; 64 MiB data, 853 MiB used, 1.0 TiB / 1.0 TiB avail; 489 KiB/s rd, 1.8 MiB/s wr, 182 op/s 2024-03-31T23:15:47.195 INFO:tasks.workunit.client.0.smithi046.stdout: 1 79862 2.61 MB/sec execute 53 sec latency 22.138 ms 2024-03-31T23:15:48.195 INFO:tasks.workunit.client.0.smithi046.stdout: 1 80450 2.59 MB/sec execute 54 sec latency 21.169 ms 2024-03-31T23:15:48.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:15:48 smithi046 ceph-mon[32498]: pgmap v125: 129 pgs: 129 active+clean; 64 MiB data, 853 MiB used, 1.0 TiB / 1.0 TiB avail; 310 KiB/s rd, 1.3 MiB/s wr, 118 op/s 2024-03-31T23:15:48.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:15:48 smithi142 ceph-mon[35271]: pgmap v125: 129 pgs: 129 active+clean; 64 MiB data, 853 MiB used, 1.0 TiB / 1.0 TiB avail; 310 KiB/s rd, 1.3 MiB/s wr, 118 op/s 2024-03-31T23:15:49.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:15:48 smithi162 ceph-mon[36237]: pgmap v125: 129 pgs: 129 active+clean; 64 MiB data, 853 MiB used, 1.0 TiB / 1.0 TiB avail; 310 KiB/s rd, 1.3 MiB/s wr, 118 op/s 2024-03-31T23:15:49.195 INFO:tasks.workunit.client.0.smithi046.stdout: 1 81049 2.65 MB/sec execute 55 sec latency 16.119 ms 2024-03-31T23:15:50.195 INFO:tasks.workunit.client.0.smithi046.stdout: 1 81413 2.63 MB/sec execute 56 sec latency 239.346 ms 2024-03-31T23:15:50.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:15:50 smithi046 ceph-mon[32498]: pgmap v126: 129 pgs: 129 active+clean; 64 MiB data, 886 MiB used, 1.0 TiB / 1.0 TiB avail; 478 KiB/s rd, 1.7 MiB/s wr, 177 op/s 2024-03-31T23:15:50.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:15:50 smithi142 ceph-mon[35271]: pgmap v126: 129 pgs: 129 active+clean; 64 MiB data, 886 MiB used, 1.0 TiB / 1.0 TiB avail; 478 KiB/s rd, 1.7 MiB/s wr, 177 op/s 2024-03-31T23:15:51.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:15:50 smithi162 ceph-mon[36237]: pgmap v126: 129 pgs: 129 active+clean; 64 MiB data, 886 MiB used, 1.0 TiB / 1.0 TiB avail; 478 KiB/s rd, 1.7 MiB/s wr, 177 op/s 2024-03-31T23:15:51.195 INFO:tasks.workunit.client.0.smithi046.stdout: 1 81873 2.66 MB/sec execute 57 sec latency 18.645 ms 2024-03-31T23:15:51.862 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:15:51 smithi046 ceph-mon[32498]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-31T23:15:51.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:15:51 smithi142 ceph-mon[35271]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-31T23:15:52.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:15:51 smithi162 ceph-mon[36237]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-31T23:15:52.195 INFO:tasks.workunit.client.0.smithi046.stdout: 1 82102 2.62 MB/sec execute 58 sec latency 15.560 ms 2024-03-31T23:15:52.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:15:52 smithi046 ceph-mon[32498]: pgmap v127: 129 pgs: 129 active+clean; 64 MiB data, 886 MiB used, 1.0 TiB / 1.0 TiB avail; 299 KiB/s rd, 1.1 MiB/s wr, 117 op/s 2024-03-31T23:15:52.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:15:52 smithi046 ceph-mon[32498]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:15:52.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:15:52 smithi046 ceph-mon[32498]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-31T23:15:52.862 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:15:52 smithi046 ceph-mon[32498]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' 2024-03-31T23:15:52.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:15:52 smithi142 ceph-mon[35271]: pgmap v127: 129 pgs: 129 active+clean; 64 MiB data, 886 MiB used, 1.0 TiB / 1.0 TiB avail; 299 KiB/s rd, 1.1 MiB/s wr, 117 op/s 2024-03-31T23:15:52.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:15:52 smithi142 ceph-mon[35271]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:15:52.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:15:52 smithi142 ceph-mon[35271]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-31T23:15:52.944 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:15:52 smithi142 ceph-mon[35271]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' 2024-03-31T23:15:53.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:15:52 smithi162 ceph-mon[36237]: pgmap v127: 129 pgs: 129 active+clean; 64 MiB data, 886 MiB used, 1.0 TiB / 1.0 TiB avail; 299 KiB/s rd, 1.1 MiB/s wr, 117 op/s 2024-03-31T23:15:53.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:15:52 smithi162 ceph-mon[36237]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:15:53.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:15:52 smithi162 ceph-mon[36237]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-31T23:15:53.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:15:52 smithi162 ceph-mon[36237]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' 2024-03-31T23:15:53.195 INFO:tasks.workunit.client.0.smithi046.stdout: 1 82508 2.59 MB/sec execute 59 sec latency 17.855 ms 2024-03-31T23:15:54.195 INFO:tasks.workunit.client.0.smithi046.stdout: 1 83039 2.62 MB/sec execute 60 sec latency 16.133 ms 2024-03-31T23:15:54.866 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:15:54 smithi046 ceph-mon[32498]: pgmap v128: 129 pgs: 129 active+clean; 64 MiB data, 886 MiB used, 1.0 TiB / 1.0 TiB avail; 299 KiB/s rd, 1.1 MiB/s wr, 117 op/s 2024-03-31T23:15:54.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:15:54 smithi142 ceph-mon[35271]: pgmap v128: 129 pgs: 129 active+clean; 64 MiB data, 886 MiB used, 1.0 TiB / 1.0 TiB avail; 299 KiB/s rd, 1.1 MiB/s wr, 117 op/s 2024-03-31T23:15:55.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:15:54 smithi162 ceph-mon[36237]: pgmap v128: 129 pgs: 129 active+clean; 64 MiB data, 886 MiB used, 1.0 TiB / 1.0 TiB avail; 299 KiB/s rd, 1.1 MiB/s wr, 117 op/s 2024-03-31T23:15:55.196 INFO:tasks.workunit.client.0.smithi046.stdout: 1 83626 2.59 MB/sec execute 61 sec latency 15.980 ms 2024-03-31T23:15:56.198 INFO:tasks.workunit.client.0.smithi046.stdout: 1 84120 2.58 MB/sec execute 62 sec latency 16.550 ms 2024-03-31T23:15:56.784 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:56.782+0000 7f523e015640 1 -- 172.21.15.46:0/582445488 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5238100fe0 msgr2=0x7f523810d4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:15:56.784 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:56.782+0000 7f523e015640 1 --2- 172.21.15.46:0/582445488 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5238100fe0 0x7f523810d4f0 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f5230009cd0 tx=0x7f523002c0f0 comp rx=0 tx=0).stop 2024-03-31T23:15:56.784 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:56.782+0000 7f523e015640 1 -- 172.21.15.46:0/582445488 shutdown_connections 2024-03-31T23:15:56.785 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:56.782+0000 7f523e015640 1 --2- 172.21.15.46:0/582445488 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5238100fe0 0x7f523810d4f0 unknown :-1 s=CLOSED pgs=683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:15:56.785 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:56.782+0000 7f523e015640 1 --2- 172.21.15.46:0/582445488 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5238100620 0x7f5238100aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:15:56.785 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:56.782+0000 7f523e015640 1 --2- 172.21.15.46:0/582445488 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5238106630 0x7f5238106a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:15:56.785 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:56.782+0000 7f523e015640 1 -- 172.21.15.46:0/582445488 >> 172.21.15.46:0/582445488 conn(0x7f5238077780 msgr2=0x7f5238077b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:15:56.785 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:56.783+0000 7f523e015640 1 -- 172.21.15.46:0/582445488 shutdown_connections 2024-03-31T23:15:56.785 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:56.783+0000 7f523e015640 1 -- 172.21.15.46:0/582445488 wait complete. 2024-03-31T23:15:56.785 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:56.783+0000 7f523e015640 1 Processor -- start 2024-03-31T23:15:56.787 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:56.784+0000 7f523e015640 1 -- start start 2024-03-31T23:15:56.787 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:56.784+0000 7f523e015640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5238100620 0x7f5238197e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:15:56.787 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:56.784+0000 7f523e015640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5238100fe0 0x7f52381983c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:15:56.787 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:56.784+0000 7f523e015640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5238106630 0x7f523819c740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:15:56.787 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:56.784+0000 7f523e015640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f52381108f0 con 0x7f5238100620 2024-03-31T23:15:56.787 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:56.784+0000 7f523e015640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f5238110770 con 0x7f5238100fe0 2024-03-31T23:15:56.788 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:56.784+0000 7f523e015640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f5238110a70 con 0x7f5238106630 2024-03-31T23:15:56.788 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:56.784+0000 7f52377fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5238100620 0x7f5238197e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:15:56.788 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:56.784+0000 7f52377fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5238100620 0x7f5238197e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:49584/0 (socket says 172.21.15.46:49584) 2024-03-31T23:15:56.788 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:56.784+0000 7f52377fe640 1 -- 172.21.15.46:0/140080666 learned_addr learned my addr 172.21.15.46:0/140080666 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:15:56.788 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:56.784+0000 7f5237fff640 1 --2- 172.21.15.46:0/140080666 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5238106630 0x7f523819c740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:15:56.788 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:56.784+0000 7f5236ffd640 1 --2- 172.21.15.46:0/140080666 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5238100fe0 0x7f52381983c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:15:56.788 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:56.784+0000 7f52377fe640 1 -- 172.21.15.46:0/140080666 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5238106630 msgr2=0x7f523819c740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:15:56.788 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:56.784+0000 7f52377fe640 1 --2- 172.21.15.46:0/140080666 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5238106630 0x7f523819c740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:15:56.788 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:56.784+0000 7f52377fe640 1 -- 172.21.15.46:0/140080666 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5238100fe0 msgr2=0x7f52381983c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:15:56.788 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:56.784+0000 7f52377fe640 1 --2- 172.21.15.46:0/140080666 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5238100fe0 0x7f52381983c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:15:56.788 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:56.784+0000 7f52377fe640 1 -- 172.21.15.46:0/140080666 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f523819ce70 con 0x7f5238100620 2024-03-31T23:15:56.788 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:56.784+0000 7f5236ffd640 1 --2- 172.21.15.46:0/140080666 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5238100fe0 0x7f52381983c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:15:56.788 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:56.784+0000 7f5237fff640 1 --2- 172.21.15.46:0/140080666 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5238106630 0x7f523819c740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:15:56.788 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:56.785+0000 7f52377fe640 1 --2- 172.21.15.46:0/140080666 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5238100620 0x7f5238197e80 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f522800e9f0 tx=0x7f522800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:15:56.789 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:56.785+0000 7f5234ff9640 1 -- 172.21.15.46:0/140080666 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f522800cd80 con 0x7f5238100620 2024-03-31T23:15:56.789 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:56.785+0000 7f5234ff9640 1 -- 172.21.15.46:0/140080666 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f52280108e0 con 0x7f5238100620 2024-03-31T23:15:56.789 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:56.785+0000 7f523e015640 1 -- 172.21.15.46:0/140080666 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f523819d160 con 0x7f5238100620 2024-03-31T23:15:56.789 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:56.785+0000 7f5234ff9640 1 -- 172.21.15.46:0/140080666 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5228018920 con 0x7f5238100620 2024-03-31T23:15:56.789 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:56.785+0000 7f523e015640 1 -- 172.21.15.46:0/140080666 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f52381a49f0 con 0x7f5238100620 2024-03-31T23:15:56.796 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:56.787+0000 7f5234ff9640 1 -- 172.21.15.46:0/140080666 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 25) ==== 137451+0+0 (secure 0 0 0) 0x7f5228010a90 con 0x7f5238100620 2024-03-31T23:15:56.796 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:56.787+0000 7f5240bf6740 1 Processor -- start 2024-03-31T23:15:56.796 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:56.787+0000 7f5240bf6740 1 -- start start 2024-03-31T23:15:56.796 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:56.788+0000 7f5234ff9640 1 --2- 172.21.15.46:0/140080666 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f52200a26b0 0x7f52200a4b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:15:56.796 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:56.788+0000 7f5240bf6740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f08b7e86c0 0x55f08b7e8aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:15:56.796 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:56.788+0000 7f5240bf6740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f08b7e8fe0 0x55f08b7f1870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:15:56.796 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:56.788+0000 7f5240bf6740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55f08b7f1db0 0x55f08b7f41a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:15:56.796 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:56.788+0000 7f5240bf6740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55f08b74b8f0 con 0x55f08b7e86c0 2024-03-31T23:15:56.796 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:56.788+0000 7f5240bf6740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55f08b6f48d0 con 0x55f08b7f1db0 2024-03-31T23:15:56.796 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:56.788+0000 7f5240bf6740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55f08b747990 con 0x55f08b7e8fe0 2024-03-31T23:15:56.797 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:56.788+0000 7f52377fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f08b7e86c0 0x55f08b7e8aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:15:56.797 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:56.788+0000 7f5237fff640 1 --2- 172.21.15.46:0/140080666 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f52200a26b0 0x7f52200a4b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:15:56.797 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:56.788+0000 7f52377fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f08b7e86c0 0x55f08b7e8aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:49594/0 (socket says 172.21.15.46:49594) 2024-03-31T23:15:56.797 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:56.788+0000 7f5236ffd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f08b7e8fe0 0x55f08b7f1870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:15:56.797 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:56.788+0000 7f52377fe640 1 -- 172.21.15.46:0/2871095858 learned_addr learned my addr 172.21.15.46:0/2871095858 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:15:56.797 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:56.788+0000 7f5234ff9640 1 -- 172.21.15.46:0/140080666 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7f52280cf140 con 0x7f5238100620 2024-03-31T23:15:56.797 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:56.788+0000 7f5237fff640 1 --2- 172.21.15.46:0/2871095858 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55f08b7f1db0 0x55f08b7f41a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:15:56.797 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:56.788+0000 7f52377fe640 1 -- 172.21.15.46:0/2871095858 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f08b7e8fe0 msgr2=0x55f08b7f1870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:15:56.797 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:56.788+0000 7f52377fe640 1 --2- 172.21.15.46:0/2871095858 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f08b7e8fe0 0x55f08b7f1870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:15:56.797 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:56.788+0000 7f52377fe640 1 -- 172.21.15.46:0/2871095858 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55f08b7f1db0 msgr2=0x55f08b7f41a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:15:56.797 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:56.788+0000 7f52377fe640 1 --2- 172.21.15.46:0/2871095858 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55f08b7f1db0 0x55f08b7f41a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:15:56.797 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:56.788+0000 7f52377fe640 1 -- 172.21.15.46:0/2871095858 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f08b460210 con 0x55f08b7e86c0 2024-03-31T23:15:56.797 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:56.788+0000 7f5236ffd640 1 --2- 172.21.15.46:0/2871095858 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f08b7e8fe0 0x55f08b7f1870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:15:56.797 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:56.788+0000 7f5237fff640 1 --2- 172.21.15.46:0/2871095858 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55f08b7f1db0 0x55f08b7f41a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T23:15:56.797 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:56.788+0000 7f52377fe640 1 --2- 172.21.15.46:0/2871095858 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f08b7e86c0 0x55f08b7e8aa0 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f522809b550 tx=0x7f522809af20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a23b6b594d3767e server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:15:56.798 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:56.788+0000 7f5237fff640 1 --2- 172.21.15.46:0/140080666 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f52200a26b0 0x7f52200a4b70 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f5230005cc0 tx=0x7f523002e040 comp rx=0 tx=0).ready entity=mgr.15654 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:15:56.798 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:56.788+0000 7f521dffb640 1 -- 172.21.15.46:0/2871095858 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5228026040 con 0x55f08b7e86c0 2024-03-31T23:15:56.798 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:56.788+0000 7f521dffb640 1 -- 172.21.15.46:0/2871095858 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f522809f660 con 0x55f08b7e86c0 2024-03-31T23:15:56.798 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:56.788+0000 7f521dffb640 1 -- 172.21.15.46:0/2871095858 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5228097c30 con 0x55f08b7e86c0 2024-03-31T23:15:56.798 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:56.789+0000 7f5240bf6740 1 -- 172.21.15.46:0/2871095858 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f08b7e86c0 msgr2=0x55f08b7e8aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:15:56.798 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:56.789+0000 7f5240bf6740 1 --2- 172.21.15.46:0/2871095858 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f08b7e86c0 0x55f08b7e8aa0 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f522809b550 tx=0x7f522809af20 comp rx=0 tx=0).stop 2024-03-31T23:15:56.798 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:56.789+0000 7f5240bf6740 1 -- 172.21.15.46:0/2871095858 shutdown_connections 2024-03-31T23:15:56.798 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:56.789+0000 7f5240bf6740 1 --2- 172.21.15.46:0/2871095858 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55f08b7f1db0 0x55f08b7f41a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:15:56.798 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:56.789+0000 7f5240bf6740 1 --2- 172.21.15.46:0/2871095858 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f08b7e8fe0 0x55f08b7f1870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:15:56.798 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:56.789+0000 7f5240bf6740 1 --2- 172.21.15.46:0/2871095858 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f08b7e86c0 0x55f08b7e8aa0 unknown :-1 s=CLOSED pgs=685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:15:56.798 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:56.789+0000 7f5240bf6740 1 -- 172.21.15.46:0/2871095858 >> 172.21.15.46:0/2871095858 conn(0x55f08b7e2fe0 msgr2=0x55f08b7e33c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:15:56.798 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:56.789+0000 7f5240bf6740 1 -- 172.21.15.46:0/2871095858 shutdown_connections 2024-03-31T23:15:56.798 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:56.789+0000 7f5240bf6740 1 -- 172.21.15.46:0/2871095858 wait complete. 2024-03-31T23:15:56.798 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:56.790+0000 7f5240bf6740 1 -- start start 2024-03-31T23:15:56.799 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:56.790+0000 7f5240bf6740 1 Processor -- start 2024-03-31T23:15:56.799 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:56.790+0000 7f5240bf6740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55f08b7e86c0 0x55f08b7eee60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:15:56.799 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:56.790+0000 7f5240bf6740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f08b7e8fe0 0x55f08b7ef3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:15:56.799 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:56.790+0000 7f5240bf6740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f08b7f1db0 0x55f08b7ef8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:15:56.799 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:56.790+0000 7f5240bf6740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55f08b7f4920 con 0x55f08b7e8fe0 2024-03-31T23:15:56.799 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:56.790+0000 7f5240bf6740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55f08b6e9f80 con 0x55f08b7e86c0 2024-03-31T23:15:56.799 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:56.790+0000 7f5240bf6740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55f08b7f4aa0 con 0x55f08b7f1db0 2024-03-31T23:15:56.799 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:56.790+0000 7f5240bf6740 10 client.0 _ll_register_callbacks cb 0x55f08b742290 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T23:15:56.799 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:56.791+0000 7f5236ffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f08b7e8fe0 0x55f08b7ef3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:15:56.799 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:56.791+0000 7f52377fe640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55f08b7e86c0 0x55f08b7eee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:15:56.799 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:56.791+0000 7f5236ffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f08b7e8fe0 0x55f08b7ef3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:49606/0 (socket says 172.21.15.46:49606) 2024-03-31T23:15:56.799 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:56.791+0000 7f5236ffd640 1 -- 172.21.15.46:0/3371762789 learned_addr learned my addr 172.21.15.46:0/3371762789 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:15:56.799 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:56.791+0000 7f5237fff640 1 --2- 172.21.15.46:0/3371762789 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f08b7f1db0 0x55f08b7ef8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:15:56.800 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:56.791+0000 7f52377fe640 1 -- 172.21.15.46:0/3371762789 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f08b7f1db0 msgr2=0x55f08b7ef8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:15:56.800 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:56.791+0000 7f52377fe640 1 --2- 172.21.15.46:0/3371762789 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f08b7f1db0 0x55f08b7ef8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:15:56.800 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:56.791+0000 7f52377fe640 1 -- 172.21.15.46:0/3371762789 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f08b7e8fe0 msgr2=0x55f08b7ef3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:15:56.800 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:56.791+0000 7f52377fe640 1 --2- 172.21.15.46:0/3371762789 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f08b7e8fe0 0x55f08b7ef3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:15:56.800 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:56.791+0000 7f52377fe640 1 -- 172.21.15.46:0/3371762789 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f08b52c620 con 0x55f08b7e86c0 2024-03-31T23:15:56.800 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:56.791+0000 7f5236ffd640 1 --2- 172.21.15.46:0/3371762789 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f08b7e8fe0 0x55f08b7ef3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:15:56.800 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:56.791+0000 7f52377fe640 1 --2- 172.21.15.46:0/3371762789 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55f08b7e86c0 0x55f08b7eee60 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f522809b550 tx=0x7f5228009310 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d88dca21a76f4273 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:15:56.800 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:56.791+0000 7f5216ffd640 10 client.0 ms_handle_connect on v2:172.21.15.142:3300/0 2024-03-31T23:15:56.800 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:56.791+0000 7f5216ffd640 1 -- 172.21.15.46:0/3371762789 <== mon.2 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52280a3c90 con 0x55f08b7e86c0 2024-03-31T23:15:56.800 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:56.791+0000 7f5216ffd640 1 -- 172.21.15.46:0/3371762789 <== mon.2 v2:172.21.15.142:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7f52280ca610 con 0x55f08b7e86c0 2024-03-31T23:15:56.800 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:56.791+0000 7f5240bf6740 1 -- 172.21.15.46:0/3371762789 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f522809f660 con 0x55f08b7e86c0 2024-03-31T23:15:56.800 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:56.791+0000 7f5216ffd640 1 -- 172.21.15.46:0/3371762789 <== mon.2 v2:172.21.15.142:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f52280b9050 con 0x55f08b7e86c0 2024-03-31T23:15:56.800 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:56.791+0000 7f5240bf6740 10 client.26330 fetch_fsmap learned FSMap version 30 2024-03-31T23:15:56.800 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:56.791+0000 7f5240bf6740 1 -- 172.21.15.46:0/3371762789 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55f08b800880 con 0x55f08b7e86c0 2024-03-31T23:15:56.801 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:56.792+0000 7f5216ffd640 1 -- 172.21.15.46:0/3371762789 <== mon.2 v2:172.21.15.142:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f522809f660 con 0x55f08b7e86c0 2024-03-31T23:15:56.801 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:56.792+0000 7f5216ffd640 1 -- 172.21.15.46:0/3371762789 <== mon.2 v2:172.21.15.142:3300/0 5 ==== fsmap(e 30) ==== 8346+0+0 (secure 0 0 0) 0x7f522809ab30 con 0x55f08b7e86c0 2024-03-31T23:15:56.801 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:56.792+0000 7f5240bf6740 10 client.26330 fetch_fsmap finished waiting for FSMap version 30 2024-03-31T23:15:56.801 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:56.792+0000 7f5240bf6740 10 client.26330 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-31T23:15:56.801 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:56.792+0000 7f5240bf6740 20 client.26330 populate_metadata read hostname 'smithi046' 2024-03-31T23:15:56.801 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:56.792+0000 7f5240bf6740 1 --2- 172.21.15.46:0/3371762789 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55f08b801bd0 0x55f08b803fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:15:56.801 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:56.792+0000 7f5240bf6740 4 client.26330 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-31T23:15:56.801 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:56.792+0000 7f5240bf6740 1 -- 172.21.15.46:0/3371762789 --> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f08b75f120 con 0x55f08b801bd0 2024-03-31T23:15:56.801 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:56.792+0000 7f5236ffd640 1 --2- 172.21.15.46:0/3371762789 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55f08b801bd0 0x55f08b803fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:15:56.801 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:56.792+0000 7f5236ffd640 1 --2- 172.21.15.46:0/3371762789 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55f08b801bd0 0x55f08b803fc0 crc :-1 s=READY pgs=111 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f2af39ac512dbc6f server_cookie=cb0380e5f9fc5537 in_seq=0 out_seq=0 2024-03-31T23:15:56.801 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:56.792+0000 7f5216ffd640 10 client.26330 ms_handle_connect on v2:172.21.15.142:6838/2519143918 2024-03-31T23:15:56.801 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:56.793+0000 7f5216ffd640 1 -- 172.21.15.46:0/3371762789 <== mds.0 v2:172.21.15.142:6838/2519143918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55f08b75f120 con 0x55f08b801bd0 2024-03-31T23:15:56.801 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:56.793+0000 7f5216ffd640 10 client.26330 handle_command_reply: tid=0 2024-03-31T23:15:56.801 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:56.794+0000 7f5240bf6740 1 client.26330 shutdown 2024-03-31T23:15:56.801 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:56.794+0000 7f5240bf6740 1 -- 172.21.15.46:0/3371762789 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55f08b7e86c0 msgr2=0x55f08b7eee60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:15:56.802 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:56.794+0000 7f5240bf6740 1 --2- 172.21.15.46:0/3371762789 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55f08b7e86c0 0x55f08b7eee60 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f522809b550 tx=0x7f5228009310 comp rx=0 tx=0).stop 2024-03-31T23:15:56.802 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:56.794+0000 7f5240bf6740 1 -- 172.21.15.46:0/3371762789 shutdown_connections 2024-03-31T23:15:56.802 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:56.794+0000 7f5240bf6740 1 --2- 172.21.15.46:0/3371762789 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55f08b801bd0 0x55f08b803fc0 crc :-1 s=READY pgs=111 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:15:56.802 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:56.794+0000 7f5240bf6740 1 --2- 172.21.15.46:0/3371762789 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f08b7f1db0 0x55f08b7ef8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:15:56.802 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:56.794+0000 7f5240bf6740 1 --2- 172.21.15.46:0/3371762789 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f08b7e8fe0 0x55f08b7ef3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:15:56.802 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:56.794+0000 7f5240bf6740 1 --2- 172.21.15.46:0/3371762789 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55f08b7e86c0 0x55f08b7eee60 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:15:56.802 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:56.794+0000 7f5216ffd640 0 client.26330 ms_handle_reset on v2:172.21.15.142:6838/2519143918 2024-03-31T23:15:56.802 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:56.794+0000 7f5240bf6740 1 -- 172.21.15.46:0/3371762789 >> 172.21.15.46:0/3371762789 conn(0x55f08b7e2fe0 msgr2=0x55f08b7e4d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:15:56.802 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:56.794+0000 7f5240bf6740 1 -- 172.21.15.46:0/3371762789 shutdown_connections 2024-03-31T23:15:56.802 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:56.794+0000 7f5240bf6740 1 -- 172.21.15.46:0/3371762789 wait complete. 2024-03-31T23:15:56.802 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:56.794+0000 7f5240bf6740 20 client.26330 trim_cache size 0 max 16384 2024-03-31T23:15:56.811 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:56.809+0000 7f5240bf6740 1 Processor -- start 2024-03-31T23:15:56.812 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:56.809+0000 7f5240bf6740 1 -- start start 2024-03-31T23:15:56.812 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:56.809+0000 7f5240bf6740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f08b7e86c0 0x55f08b7f1590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:15:56.812 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:56.809+0000 7f5240bf6740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f08b7e8fe0 0x55f08b7e1e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:15:56.812 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:56.809+0000 7f5240bf6740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55f08b7f1db0 0x55f08b806fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:15:56.812 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:56.809+0000 7f5240bf6740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f52381b1440 con 0x55f08b7e86c0 2024-03-31T23:15:56.812 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:56.809+0000 7f5240bf6740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f52381aed10 con 0x55f08b7f1db0 2024-03-31T23:15:56.812 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:56.809+0000 7f5240bf6740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f52381acb60 con 0x55f08b7e8fe0 2024-03-31T23:15:56.812 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:56.810+0000 7f52377fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f08b7e86c0 0x55f08b7f1590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:15:56.812 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:56.810+0000 7f52377fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f08b7e86c0 0x55f08b7f1590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:49622/0 (socket says 172.21.15.46:49622) 2024-03-31T23:15:56.812 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:56.810+0000 7f52377fe640 1 -- 172.21.15.46:0/602912210 learned_addr learned my addr 172.21.15.46:0/602912210 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:15:56.812 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:56.810+0000 7f5237fff640 1 --2- 172.21.15.46:0/602912210 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55f08b7f1db0 0x55f08b806fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:15:56.812 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:56.810+0000 7f5236ffd640 1 --2- 172.21.15.46:0/602912210 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f08b7e8fe0 0x55f08b7e1e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:15:56.812 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:56.810+0000 7f52377fe640 1 -- 172.21.15.46:0/602912210 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f08b7e8fe0 msgr2=0x55f08b7e1e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:15:56.812 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:56.810+0000 7f52377fe640 1 --2- 172.21.15.46:0/602912210 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f08b7e8fe0 0x55f08b7e1e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:15:56.813 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:56.810+0000 7f52377fe640 1 -- 172.21.15.46:0/602912210 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55f08b7f1db0 msgr2=0x55f08b806fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:15:56.813 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:56.810+0000 7f52377fe640 1 --2- 172.21.15.46:0/602912210 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55f08b7f1db0 0x55f08b806fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:15:56.813 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:56.810+0000 7f52377fe640 1 -- 172.21.15.46:0/602912210 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f08b7e8e10 con 0x55f08b7e86c0 2024-03-31T23:15:56.813 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:56.810+0000 7f5237fff640 1 --2- 172.21.15.46:0/602912210 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55f08b7f1db0 0x55f08b806fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:15:56.813 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:56.810+0000 7f52377fe640 1 --2- 172.21.15.46:0/602912210 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f08b7e86c0 0x55f08b7f1590 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f522800e9c0 tx=0x7f52280cc590 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d3fc177146f794b server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:15:56.813 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:56.810+0000 7f521d7fa640 1 -- 172.21.15.46:0/602912210 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5228010de0 con 0x55f08b7e86c0 2024-03-31T23:15:56.813 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:56.810+0000 7f521d7fa640 1 -- 172.21.15.46:0/602912210 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f522809f660 con 0x55f08b7e86c0 2024-03-31T23:15:56.813 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:56.810+0000 7f521d7fa640 1 -- 172.21.15.46:0/602912210 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52280cae50 con 0x55f08b7e86c0 2024-03-31T23:15:56.816 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:56.811+0000 7f5240bf6740 1 -- 172.21.15.46:0/602912210 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f08b7e86c0 msgr2=0x55f08b7f1590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:15:56.816 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:56.811+0000 7f5240bf6740 1 --2- 172.21.15.46:0/602912210 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f08b7e86c0 0x55f08b7f1590 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f522800e9c0 tx=0x7f52280cc590 comp rx=0 tx=0).stop 2024-03-31T23:15:56.816 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:56.811+0000 7f5240bf6740 1 -- 172.21.15.46:0/602912210 shutdown_connections 2024-03-31T23:15:56.817 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:56.811+0000 7f5240bf6740 1 --2- 172.21.15.46:0/602912210 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55f08b7f1db0 0x55f08b806fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:15:56.817 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:56.811+0000 7f5240bf6740 1 --2- 172.21.15.46:0/602912210 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f08b7e8fe0 0x55f08b7e1e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:15:56.817 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:56.811+0000 7f5240bf6740 1 --2- 172.21.15.46:0/602912210 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f08b7e86c0 0x55f08b7f1590 unknown :-1 s=CLOSED pgs=686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:15:56.817 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:56.811+0000 7f5240bf6740 1 -- 172.21.15.46:0/602912210 >> 172.21.15.46:0/602912210 conn(0x55f08b7e2fe0 msgr2=0x55f08b7e16f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:15:56.817 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:56.811+0000 7f5240bf6740 1 -- 172.21.15.46:0/602912210 shutdown_connections 2024-03-31T23:15:56.817 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:56.811+0000 7f5240bf6740 1 -- 172.21.15.46:0/602912210 wait complete. 2024-03-31T23:15:56.817 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:56.811+0000 7f5240bf6740 1 -- start start 2024-03-31T23:15:56.817 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:56.812+0000 7f5240bf6740 1 Processor -- start 2024-03-31T23:15:56.817 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:56.812+0000 7f5240bf6740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55f08b7e86c0 0x55f08b8070c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:15:56.817 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:56.812+0000 7f5240bf6740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f08b7e8fe0 0x55f08b804450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:15:56.817 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:56.812+0000 7f5240bf6740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f08b7f1db0 0x55f08b804990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:15:56.817 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:56.812+0000 7f5240bf6740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f52381ad1d0 con 0x55f08b7e8fe0 2024-03-31T23:15:56.817 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:56.812+0000 7f5240bf6740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f52381ae640 con 0x55f08b7e86c0 2024-03-31T23:15:56.817 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:56.812+0000 7f5240bf6740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f52381ad8a0 con 0x55f08b7f1db0 2024-03-31T23:15:56.818 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:56.812+0000 7f5240bf6740 10 client.0 _ll_register_callbacks cb 0x7f522c009b30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T23:15:56.818 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:56.812+0000 7f5236ffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f08b7e8fe0 0x55f08b804450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:15:56.818 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:56.812+0000 7f5236ffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f08b7e8fe0 0x55f08b804450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:49630/0 (socket says 172.21.15.46:49630) 2024-03-31T23:15:56.818 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:56.812+0000 7f5236ffd640 1 -- 172.21.15.46:0/2392071361 learned_addr learned my addr 172.21.15.46:0/2392071361 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:15:56.818 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:56.812+0000 7f5237fff640 1 --2- 172.21.15.46:0/2392071361 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f08b7f1db0 0x55f08b804990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:15:56.818 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:56.812+0000 7f52377fe640 1 --2- 172.21.15.46:0/2392071361 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55f08b7e86c0 0x55f08b8070c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:15:56.818 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:56.812+0000 7f5236ffd640 1 -- 172.21.15.46:0/2392071361 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f08b7f1db0 msgr2=0x55f08b804990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:15:56.818 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:56.812+0000 7f5236ffd640 1 --2- 172.21.15.46:0/2392071361 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f08b7f1db0 0x55f08b804990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:15:56.818 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:56.812+0000 7f5236ffd640 1 -- 172.21.15.46:0/2392071361 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55f08b7e86c0 msgr2=0x55f08b8070c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:15:56.818 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:56.812+0000 7f5236ffd640 1 --2- 172.21.15.46:0/2392071361 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55f08b7e86c0 0x55f08b8070c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:15:56.818 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:56.812+0000 7f5236ffd640 1 -- 172.21.15.46:0/2392071361 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f08b804fd0 con 0x55f08b7e8fe0 2024-03-31T23:15:56.818 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:56.812+0000 7f5236ffd640 1 --2- 172.21.15.46:0/2392071361 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f08b7e8fe0 0x55f08b804450 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x55f08b7f4b20 tx=0x7f522c01f7c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=46a9dd32d94c172d server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:15:56.818 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:56.812+0000 7f5216ffd640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-31T23:15:56.819 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:56.813+0000 7f5216ffd640 1 -- 172.21.15.46:0/2392071361 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f522c012e60 con 0x55f08b7e8fe0 2024-03-31T23:15:56.819 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:56.813+0000 7f5216ffd640 1 -- 172.21.15.46:0/2392071361 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7f522c004050 con 0x55f08b7e8fe0 2024-03-31T23:15:56.819 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:56.813+0000 7f5240bf6740 1 -- 172.21.15.46:0/2392071361 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f522809f660 con 0x55f08b7e8fe0 2024-03-31T23:15:56.819 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:56.813+0000 7f5216ffd640 1 -- 172.21.15.46:0/2392071361 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f522c030050 con 0x55f08b7e8fe0 2024-03-31T23:15:56.819 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:56.813+0000 7f5240bf6740 10 client.16497 fetch_fsmap learned FSMap version 30 2024-03-31T23:15:56.819 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:56.813+0000 7f5240bf6740 1 -- 172.21.15.46:0/2392071361 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55f08b867a40 con 0x55f08b7e8fe0 2024-03-31T23:15:56.819 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:56.813+0000 7f5216ffd640 1 -- 172.21.15.46:0/2392071361 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f522809f660 con 0x55f08b7e8fe0 2024-03-31T23:15:56.819 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:56.813+0000 7f5216ffd640 1 -- 172.21.15.46:0/2392071361 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 30) ==== 8346+0+0 (secure 0 0 0) 0x7f522c004810 con 0x55f08b7e8fe0 2024-03-31T23:15:56.819 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:56.813+0000 7f5240bf6740 10 client.16497 fetch_fsmap finished waiting for FSMap version 30 2024-03-31T23:15:56.819 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:56.813+0000 7f5240bf6740 10 client.16497 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-31T23:15:56.819 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:56.813+0000 7f5240bf6740 20 client.16497 populate_metadata read hostname 'smithi046' 2024-03-31T23:15:56.819 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:56.813+0000 7f5240bf6740 1 --2- 172.21.15.46:0/2392071361 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55f08b868240 0x55f08b86a660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:15:56.819 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:56.813+0000 7f5240bf6740 4 client.16497 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-31T23:15:56.819 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:56.813+0000 7f5240bf6740 1 -- 172.21.15.46:0/2392071361 --> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55f08b758e30 con 0x55f08b868240 2024-03-31T23:15:56.819 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:56.813+0000 7f52377fe640 1 --2- 172.21.15.46:0/2392071361 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55f08b868240 0x55f08b86a660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:15:56.820 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:56.814+0000 7f52377fe640 1 --2- 172.21.15.46:0/2392071361 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55f08b868240 0x55f08b86a660 crc :-1 s=READY pgs=112 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cbd48e97b06c1fdc server_cookie=a18ef0b75881ecce in_seq=0 out_seq=0 2024-03-31T23:15:56.820 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:56.814+0000 7f5216ffd640 10 client.16497 ms_handle_connect on v2:172.21.15.142:6838/2519143918 2024-03-31T23:15:56.836 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:56.834+0000 7f5216ffd640 1 -- 172.21.15.46:0/2392071361 <== mds.0 v2:172.21.15.142:6838/2519143918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55f08b758e30 con 0x55f08b868240 2024-03-31T23:15:56.836 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:56.834+0000 7f5216ffd640 10 client.16497 handle_command_reply: tid=0 2024-03-31T23:15:56.836 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:56.834+0000 7f5240bf6740 1 client.16497 shutdown 2024-03-31T23:15:56.836 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:56.834+0000 7f5240bf6740 1 -- 172.21.15.46:0/2392071361 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f08b7e8fe0 msgr2=0x55f08b804450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:15:56.836 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:56.834+0000 7f5240bf6740 1 --2- 172.21.15.46:0/2392071361 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f08b7e8fe0 0x55f08b804450 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x55f08b7f4b20 tx=0x7f522c01f7c0 comp rx=0 tx=0).stop 2024-03-31T23:15:56.836 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:56.835+0000 7f5240bf6740 1 -- 172.21.15.46:0/2392071361 shutdown_connections 2024-03-31T23:15:56.836 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:56.835+0000 7f5240bf6740 1 --2- 172.21.15.46:0/2392071361 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55f08b868240 0x55f08b86a660 crc :-1 s=READY pgs=112 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:15:56.836 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:56.835+0000 7f5240bf6740 1 --2- 172.21.15.46:0/2392071361 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f08b7f1db0 0x55f08b804990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:15:56.837 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:56.835+0000 7f5240bf6740 1 --2- 172.21.15.46:0/2392071361 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f08b7e8fe0 0x55f08b804450 unknown :-1 s=CLOSED pgs=687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:15:56.837 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:56.835+0000 7f5216ffd640 0 client.16497 ms_handle_reset on v2:172.21.15.142:6838/2519143918 2024-03-31T23:15:56.837 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:56.835+0000 7f5240bf6740 1 --2- 172.21.15.46:0/2392071361 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55f08b7e86c0 0x55f08b8070c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:15:56.837 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:56.835+0000 7f5240bf6740 1 -- 172.21.15.46:0/2392071361 >> 172.21.15.46:0/2392071361 conn(0x55f08b7e2fe0 msgr2=0x55f08b7e1f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:15:56.837 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:56.835+0000 7f5240bf6740 1 -- 172.21.15.46:0/2392071361 shutdown_connections 2024-03-31T23:15:56.837 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:56.835+0000 7f5240bf6740 1 -- 172.21.15.46:0/2392071361 wait complete. 2024-03-31T23:15:56.837 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:56.835+0000 7f5240bf6740 20 client.16497 trim_cache size 0 max 16384 2024-03-31T23:15:56.837 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout:{ 2024-03-31T23:15:56.837 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout: "message": "", 2024-03-31T23:15:56.837 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout: "return_code": 0 2024-03-31T23:15:56.837 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout:} 2024-03-31T23:15:56.838 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:56.835+0000 7f523e015640 1 -- 172.21.15.46:0/140080666 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f52200a26b0 msgr2=0x7f52200a4b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:15:56.838 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:56.835+0000 7f523e015640 1 --2- 172.21.15.46:0/140080666 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f52200a26b0 0x7f52200a4b70 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f5230005cc0 tx=0x7f523002e040 comp rx=0 tx=0).stop 2024-03-31T23:15:56.839 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:56.835+0000 7f523e015640 1 -- 172.21.15.46:0/140080666 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5238100620 msgr2=0x7f5238197e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:15:56.839 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:56.835+0000 7f523e015640 1 --2- 172.21.15.46:0/140080666 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5238100620 0x7f5238197e80 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f522800e9f0 tx=0x7f522800eeb0 comp rx=0 tx=0).stop 2024-03-31T23:15:56.839 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:56.835+0000 7f523e015640 1 -- 172.21.15.46:0/140080666 shutdown_connections 2024-03-31T23:15:56.839 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:56.835+0000 7f523e015640 1 --2- 172.21.15.46:0/140080666 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f52200a26b0 0x7f52200a4b70 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:15:56.839 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:56.835+0000 7f523e015640 1 --2- 172.21.15.46:0/140080666 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5238106630 0x7f523819c740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:15:56.839 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:56.835+0000 7f523e015640 1 --2- 172.21.15.46:0/140080666 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5238100fe0 0x7f52381983c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:15:56.839 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:56.835+0000 7f523e015640 1 --2- 172.21.15.46:0/140080666 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5238100620 0x7f5238197e80 unknown :-1 s=CLOSED pgs=684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:15:56.839 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:56.835+0000 7f523e015640 1 -- 172.21.15.46:0/140080666 >> 172.21.15.46:0/140080666 conn(0x7f5238077780 msgr2=0x7f523806a230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:15:56.839 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:56.835+0000 7f523e015640 1 -- 172.21.15.46:0/140080666 shutdown_connections 2024-03-31T23:15:56.839 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:15:56.836+0000 7f523e015640 1 -- 172.21.15.46:0/140080666 wait complete. 2024-03-31T23:15:56.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:15:56 smithi142 ceph-mon[35271]: pgmap v129: 129 pgs: 129 active+clean; 68 MiB data, 905 MiB used, 1.0 TiB / 1.0 TiB avail; 473 KiB/s rd, 1.7 MiB/s wr, 173 op/s 2024-03-31T23:15:57.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:15:56 smithi162 ceph-mon[36237]: pgmap v129: 129 pgs: 129 active+clean; 68 MiB data, 905 MiB used, 1.0 TiB / 1.0 TiB avail; 473 KiB/s rd, 1.7 MiB/s wr, 173 op/s 2024-03-31T23:15:57.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:15:56 smithi046 ceph-mon[32498]: pgmap v129: 129 pgs: 129 active+clean; 68 MiB data, 905 MiB used, 1.0 TiB / 1.0 TiB avail; 473 KiB/s rd, 1.7 MiB/s wr, 173 op/s 2024-03-31T23:15:57.195 INFO:tasks.workunit.client.0.smithi046.stdout: 1 84898 2.64 MB/sec execute 63 sec latency 15.567 ms 2024-03-31T23:15:58.195 INFO:tasks.workunit.client.0.smithi046.stdout: 1 85212 2.65 MB/sec execute 64 sec latency 23.709 ms 2024-03-31T23:15:58.945 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:15:58 smithi142 ceph-mon[35271]: pgmap v130: 129 pgs: 129 active+clean; 68 MiB data, 905 MiB used, 1.0 TiB / 1.0 TiB avail; 342 KiB/s rd, 1.1 MiB/s wr, 115 op/s 2024-03-31T23:15:59.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:15:58 smithi162 ceph-mon[36237]: pgmap v130: 129 pgs: 129 active+clean; 68 MiB data, 905 MiB used, 1.0 TiB / 1.0 TiB avail; 342 KiB/s rd, 1.1 MiB/s wr, 115 op/s 2024-03-31T23:15:59.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:15:58 smithi046 ceph-mon[32498]: pgmap v130: 129 pgs: 129 active+clean; 68 MiB data, 905 MiB used, 1.0 TiB / 1.0 TiB avail; 342 KiB/s rd, 1.1 MiB/s wr, 115 op/s 2024-03-31T23:15:59.195 INFO:tasks.workunit.client.0.smithi046.stdout: 1 85557 2.64 MB/sec execute 65 sec latency 15.265 ms 2024-03-31T23:16:00.196 INFO:tasks.workunit.client.0.smithi046.stdout: 1 85841 2.60 MB/sec execute 66 sec latency 16.750 ms 2024-03-31T23:16:00.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:16:00 smithi142 ceph-mon[35271]: pgmap v131: 129 pgs: 129 active+clean; 65 MiB data, 923 MiB used, 1.0 TiB / 1.0 TiB avail; 455 KiB/s rd, 1.7 MiB/s wr, 174 op/s 2024-03-31T23:16:01.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:16:00 smithi162 ceph-mon[36237]: pgmap v131: 129 pgs: 129 active+clean; 65 MiB data, 923 MiB used, 1.0 TiB / 1.0 TiB avail; 455 KiB/s rd, 1.7 MiB/s wr, 174 op/s 2024-03-31T23:16:01.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:16:00 smithi046 ceph-mon[32498]: pgmap v131: 129 pgs: 129 active+clean; 65 MiB data, 923 MiB used, 1.0 TiB / 1.0 TiB avail; 455 KiB/s rd, 1.7 MiB/s wr, 174 op/s 2024-03-31T23:16:01.196 INFO:tasks.workunit.client.0.smithi046.stdout: 1 86307 2.59 MB/sec execute 67 sec latency 18.102 ms 2024-03-31T23:16:02.196 INFO:tasks.workunit.client.0.smithi046.stdout: 1 86826 2.60 MB/sec execute 68 sec latency 42.756 ms 2024-03-31T23:16:02.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:16:02 smithi142 ceph-mon[35271]: pgmap v132: 129 pgs: 129 active+clean; 65 MiB data, 923 MiB used, 1.0 TiB / 1.0 TiB avail; 288 KiB/s rd, 1.2 MiB/s wr, 115 op/s 2024-03-31T23:16:03.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:16:02 smithi162 ceph-mon[36237]: pgmap v132: 129 pgs: 129 active+clean; 65 MiB data, 923 MiB used, 1.0 TiB / 1.0 TiB avail; 288 KiB/s rd, 1.2 MiB/s wr, 115 op/s 2024-03-31T23:16:03.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:16:02 smithi046 ceph-mon[32498]: pgmap v132: 129 pgs: 129 active+clean; 65 MiB data, 923 MiB used, 1.0 TiB / 1.0 TiB avail; 288 KiB/s rd, 1.2 MiB/s wr, 115 op/s 2024-03-31T23:16:03.196 INFO:tasks.workunit.client.0.smithi046.stdout: 1 87481 2.58 MB/sec execute 69 sec latency 20.420 ms 2024-03-31T23:16:04.196 INFO:tasks.workunit.client.0.smithi046.stdout: 1 88051 2.61 MB/sec execute 70 sec latency 15.721 ms 2024-03-31T23:16:04.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:16:04 smithi142 ceph-mon[35271]: pgmap v133: 129 pgs: 129 active+clean; 65 MiB data, 923 MiB used, 1.0 TiB / 1.0 TiB avail; 288 KiB/s rd, 1.2 MiB/s wr, 115 op/s 2024-03-31T23:16:05.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:16:04 smithi162 ceph-mon[36237]: pgmap v133: 129 pgs: 129 active+clean; 65 MiB data, 923 MiB used, 1.0 TiB / 1.0 TiB avail; 288 KiB/s rd, 1.2 MiB/s wr, 115 op/s 2024-03-31T23:16:05.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:16:04 smithi046 ceph-mon[32498]: pgmap v133: 129 pgs: 129 active+clean; 65 MiB data, 923 MiB used, 1.0 TiB / 1.0 TiB avail; 288 KiB/s rd, 1.2 MiB/s wr, 115 op/s 2024-03-31T23:16:05.196 INFO:tasks.workunit.client.0.smithi046.stdout: 1 88501 2.62 MB/sec execute 71 sec latency 20.035 ms 2024-03-31T23:16:06.196 INFO:tasks.workunit.client.0.smithi046.stdout: 1 88962 2.64 MB/sec execute 72 sec latency 16.277 ms 2024-03-31T23:16:06.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:16:06 smithi142 ceph-mon[35271]: pgmap v134: 129 pgs: 129 active+clean; 66 MiB data, 924 MiB used, 1.0 TiB / 1.0 TiB avail; 485 KiB/s rd, 1.8 MiB/s wr, 190 op/s 2024-03-31T23:16:07.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:16:06 smithi162 ceph-mon[36237]: pgmap v134: 129 pgs: 129 active+clean; 66 MiB data, 924 MiB used, 1.0 TiB / 1.0 TiB avail; 485 KiB/s rd, 1.8 MiB/s wr, 190 op/s 2024-03-31T23:16:07.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:16:06 smithi046 ceph-mon[32498]: pgmap v134: 129 pgs: 129 active+clean; 66 MiB data, 924 MiB used, 1.0 TiB / 1.0 TiB avail; 485 KiB/s rd, 1.8 MiB/s wr, 190 op/s 2024-03-31T23:16:07.196 INFO:tasks.workunit.client.0.smithi046.stdout: 1 89200 2.61 MB/sec execute 73 sec latency 15.077 ms 2024-03-31T23:16:08.196 INFO:tasks.workunit.client.0.smithi046.stdout: 1 89319 2.57 MB/sec execute 74 sec latency 628.381 ms 2024-03-31T23:16:08.914 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:16:08 smithi046 ceph-mon[32498]: pgmap v135: 129 pgs: 129 active+clean; 66 MiB data, 924 MiB used, 1.0 TiB / 1.0 TiB avail; 311 KiB/s rd, 1.1 MiB/s wr, 133 op/s 2024-03-31T23:16:08.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:16:08 smithi142 ceph-mon[35271]: pgmap v135: 129 pgs: 129 active+clean; 66 MiB data, 924 MiB used, 1.0 TiB / 1.0 TiB avail; 311 KiB/s rd, 1.1 MiB/s wr, 133 op/s 2024-03-31T23:16:09.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:16:08 smithi162 ceph-mon[36237]: pgmap v135: 129 pgs: 129 active+clean; 66 MiB data, 924 MiB used, 1.0 TiB / 1.0 TiB avail; 311 KiB/s rd, 1.1 MiB/s wr, 133 op/s 2024-03-31T23:16:09.196 INFO:tasks.workunit.client.0.smithi046.stdout: 1 89319 2.54 MB/sec execute 75 sec latency 1628.467 ms 2024-03-31T23:16:09.937 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:09.933+0000 7f1152840640 1 -- 172.21.15.46:0/3936780692 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f114c101240 msgr2=0x7f114c10d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:16:09.937 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:09.933+0000 7f1152840640 1 --2- 172.21.15.46:0/3936780692 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f114c101240 0x7f114c10d480 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f1144009ca0 tx=0x7f114402c0f0 comp rx=0 tx=0).stop 2024-03-31T23:16:09.937 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:09.933+0000 7f1152840640 1 -- 172.21.15.46:0/3936780692 shutdown_connections 2024-03-31T23:16:09.937 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:09.933+0000 7f1152840640 1 --2- 172.21.15.46:0/3936780692 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f114c101240 0x7f114c10d480 unknown :-1 s=CLOSED pgs=688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:16:09.937 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:09.933+0000 7f1152840640 1 --2- 172.21.15.46:0/3936780692 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f114c1008a0 0x7f114c100d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:16:09.937 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:09.933+0000 7f1152840640 1 --2- 172.21.15.46:0/3936780692 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f114c1055d0 0x7f114c1059b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:16:09.937 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:09.934+0000 7f1152840640 1 -- 172.21.15.46:0/3936780692 >> 172.21.15.46:0/3936780692 conn(0x7f114c077420 msgr2=0x7f114c077830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:16:09.937 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:09.934+0000 7f1152840640 1 -- 172.21.15.46:0/3936780692 shutdown_connections 2024-03-31T23:16:09.937 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:09.934+0000 7f1152840640 1 -- 172.21.15.46:0/3936780692 wait complete. 2024-03-31T23:16:09.938 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:09.934+0000 7f1152840640 1 Processor -- start 2024-03-31T23:16:09.938 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:09.934+0000 7f1152840640 1 -- start start 2024-03-31T23:16:09.938 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:09.935+0000 7f1152840640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f114c1008a0 0x7f114c197e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:16:09.938 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:09.935+0000 7f1152840640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f114c101240 0x7f114c1983d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:16:09.938 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:09.935+0000 7f1152840640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f114c1055d0 0x7f114c19c700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:16:09.938 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:09.935+0000 7f1152840640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f114c111ab0 con 0x7f114c1055d0 2024-03-31T23:16:09.938 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:09.935+0000 7f1152840640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f114c111930 con 0x7f114c101240 2024-03-31T23:16:09.938 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:09.935+0000 7f1152840640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f114c111c30 con 0x7f114c1008a0 2024-03-31T23:16:09.938 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:09.935+0000 7f1150ddd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f114c1055d0 0x7f114c19c700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:16:09.938 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:09.935+0000 7f114b7fe640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f114c101240 0x7f114c1983d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:16:09.938 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:09.935+0000 7f1150ddd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f114c1055d0 0x7f114c19c700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:48658/0 (socket says 172.21.15.46:48658) 2024-03-31T23:16:09.938 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:09.935+0000 7f1150ddd640 1 -- 172.21.15.46:0/4081525920 learned_addr learned my addr 172.21.15.46:0/4081525920 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:16:09.938 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:09.935+0000 7f114bfff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f114c1008a0 0x7f114c197e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:16:09.939 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:09.935+0000 7f1150ddd640 1 -- 172.21.15.46:0/4081525920 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f114c1008a0 msgr2=0x7f114c197e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:16:09.939 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:09.935+0000 7f1150ddd640 1 --2- 172.21.15.46:0/4081525920 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f114c1008a0 0x7f114c197e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:16:09.939 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:09.935+0000 7f1150ddd640 1 -- 172.21.15.46:0/4081525920 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f114c101240 msgr2=0x7f114c1983d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:16:09.939 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:09.935+0000 7f1150ddd640 1 --2- 172.21.15.46:0/4081525920 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f114c101240 0x7f114c1983d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:16:09.939 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:09.935+0000 7f1150ddd640 1 -- 172.21.15.46:0/4081525920 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f114c19ce30 con 0x7f114c1055d0 2024-03-31T23:16:09.939 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:09.935+0000 7f1150ddd640 1 --2- 172.21.15.46:0/4081525920 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f114c1055d0 0x7f114c19c700 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f114402c700 tx=0x7f1144035d60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:16:09.939 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:09.935+0000 7f11497fa640 1 -- 172.21.15.46:0/4081525920 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1144039040 con 0x7f114c1055d0 2024-03-31T23:16:09.939 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:09.936+0000 7f11497fa640 1 -- 172.21.15.46:0/4081525920 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1144044070 con 0x7f114c1055d0 2024-03-31T23:16:09.939 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:09.936+0000 7f1152840640 1 -- 172.21.15.46:0/4081525920 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f114c19d0c0 con 0x7f114c1055d0 2024-03-31T23:16:09.939 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:09.936+0000 7f11497fa640 1 -- 172.21.15.46:0/4081525920 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11440042a0 con 0x7f114c1055d0 2024-03-31T23:16:09.939 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:09.936+0000 7f1152840640 1 -- 172.21.15.46:0/4081525920 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f114c1a49b0 con 0x7f114c1055d0 2024-03-31T23:16:09.945 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:09.938+0000 7f11497fa640 1 -- 172.21.15.46:0/4081525920 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 25) ==== 137451+0+0 (secure 0 0 0) 0x7f1144003970 con 0x7f114c1055d0 2024-03-31T23:16:09.945 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:09.938+0000 7f1155426740 1 Processor -- start 2024-03-31T23:16:09.946 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:09.938+0000 7f11497fa640 1 --2- 172.21.15.46:0/4081525920 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f11340a26b0 0x7f11340a4b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:16:09.946 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:09.938+0000 7f1155426740 1 -- start start 2024-03-31T23:16:09.946 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:09.939+0000 7f114b7fe640 1 --2- 172.21.15.46:0/4081525920 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f11340a26b0 0x7f11340a4b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:16:09.946 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:09.939+0000 7f1155426740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55f1947526c0 0x55f194752aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:16:09.946 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:09.939+0000 7f1155426740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f194752fe0 0x55f19475b870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:16:09.946 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:09.939+0000 7f11497fa640 1 -- 172.21.15.46:0/4081525920 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7f11440f11c0 con 0x7f114c1055d0 2024-03-31T23:16:09.946 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:09.939+0000 7f1155426740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f19475bdb0 0x55f19475e1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:16:09.946 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:09.939+0000 7f1155426740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55f1946b58f0 con 0x55f194752fe0 2024-03-31T23:16:09.946 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:09.939+0000 7f1155426740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55f19465e8d0 con 0x55f1947526c0 2024-03-31T23:16:09.946 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:09.939+0000 7f1155426740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55f1946b1990 con 0x55f19475bdb0 2024-03-31T23:16:09.946 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:09.939+0000 7f114bfff640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55f1947526c0 0x55f194752aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:16:09.946 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:09.939+0000 7f114b7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f194752fe0 0x55f19475b870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:16:09.947 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:09.939+0000 7f114bfff640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55f1947526c0 0x55f194752aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.46:47732/0 (socket says 172.21.15.46:47732) 2024-03-31T23:16:09.947 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:09.939+0000 7f114bfff640 1 -- 172.21.15.46:0/605160989 learned_addr learned my addr 172.21.15.46:0/605160989 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:16:09.947 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:09.939+0000 7f1150ddd640 1 --2- 172.21.15.46:0/605160989 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f19475bdb0 0x55f19475e1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:16:09.947 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:09.939+0000 7f114bfff640 1 -- 172.21.15.46:0/605160989 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f19475bdb0 msgr2=0x55f19475e1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:16:09.947 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:09.939+0000 7f114bfff640 1 --2- 172.21.15.46:0/605160989 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f19475bdb0 0x55f19475e1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:16:09.947 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:09.939+0000 7f114bfff640 1 -- 172.21.15.46:0/605160989 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f194752fe0 msgr2=0x55f19475b870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:16:09.947 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:09.939+0000 7f114bfff640 1 --2- 172.21.15.46:0/605160989 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f194752fe0 0x55f19475b870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:16:09.947 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:09.939+0000 7f114bfff640 1 -- 172.21.15.46:0/605160989 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f1943ca210 con 0x55f1947526c0 2024-03-31T23:16:09.947 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:09.939+0000 7f1150ddd640 1 --2- 172.21.15.46:0/605160989 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f19475bdb0 0x55f19475e1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T23:16:09.947 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:09.939+0000 7f114b7fe640 1 --2- 172.21.15.46:0/4081525920 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f11340a26b0 0x7f11340a4b70 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f114c100310 tx=0x7f1140013830 comp rx=0 tx=0).ready entity=mgr.15654 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:16:09.947 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:09.939+0000 7f114bfff640 1 --2- 172.21.15.46:0/605160989 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55f1947526c0 0x55f194752aa0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f113c009cf0 tx=0x7f113c011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ee4ce2853a275004 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:16:09.948 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:09.940+0000 7f11327fc640 1 -- 172.21.15.46:0/605160989 <== mon.2 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f113c009de0 con 0x55f1947526c0 2024-03-31T23:16:09.948 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:09.940+0000 7f11327fc640 1 -- 172.21.15.46:0/605160989 <== mon.2 v2:172.21.15.142:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f113c005020 con 0x55f1947526c0 2024-03-31T23:16:09.948 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:09.940+0000 7f11327fc640 1 -- 172.21.15.46:0/605160989 <== mon.2 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f113c004ba0 con 0x55f1947526c0 2024-03-31T23:16:09.948 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:09.940+0000 7f1155426740 1 -- 172.21.15.46:0/605160989 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55f1947526c0 msgr2=0x55f194752aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:16:09.948 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:09.940+0000 7f1155426740 1 --2- 172.21.15.46:0/605160989 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55f1947526c0 0x55f194752aa0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f113c009cf0 tx=0x7f113c011810 comp rx=0 tx=0).stop 2024-03-31T23:16:09.948 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:09.940+0000 7f1155426740 1 -- 172.21.15.46:0/605160989 shutdown_connections 2024-03-31T23:16:09.948 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:09.940+0000 7f1155426740 1 --2- 172.21.15.46:0/605160989 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f19475bdb0 0x55f19475e1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:16:09.948 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:09.940+0000 7f1155426740 1 --2- 172.21.15.46:0/605160989 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f194752fe0 0x55f19475b870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:16:09.948 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:09.940+0000 7f1155426740 1 --2- 172.21.15.46:0/605160989 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55f1947526c0 0x55f194752aa0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:16:09.948 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:09.940+0000 7f1155426740 1 -- 172.21.15.46:0/605160989 >> 172.21.15.46:0/605160989 conn(0x55f19474cfe0 msgr2=0x55f19474d3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:16:09.948 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:09.940+0000 7f1155426740 1 -- 172.21.15.46:0/605160989 shutdown_connections 2024-03-31T23:16:09.948 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:09.940+0000 7f1155426740 1 -- 172.21.15.46:0/605160989 wait complete. 2024-03-31T23:16:09.948 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:09.941+0000 7f1155426740 1 -- start start 2024-03-31T23:16:09.949 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:09.941+0000 7f1155426740 1 Processor -- start 2024-03-31T23:16:09.949 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:09.941+0000 7f1155426740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f1947526c0 0x55f194758da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:16:09.949 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:09.941+0000 7f1155426740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55f194752fe0 0x55f1947592e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:16:09.949 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:09.941+0000 7f1155426740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f19475bdb0 0x55f194759820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:16:09.949 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:09.941+0000 7f1155426740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55f19475e9e0 con 0x55f1947526c0 2024-03-31T23:16:09.949 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:09.941+0000 7f1155426740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55f194653f80 con 0x55f194752fe0 2024-03-31T23:16:09.949 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:09.941+0000 7f1155426740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55f19475eb60 con 0x55f19475bdb0 2024-03-31T23:16:09.949 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:09.941+0000 7f1155426740 10 client.0 _ll_register_callbacks cb 0x55f1946ac290 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T23:16:09.949 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:09.941+0000 7f114bfff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f1947526c0 0x55f194758da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:16:09.949 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:09.941+0000 7f114b7fe640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55f194752fe0 0x55f1947592e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:16:09.949 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:09.941+0000 7f114bfff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f1947526c0 0x55f194758da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:48680/0 (socket says 172.21.15.46:48680) 2024-03-31T23:16:09.949 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:09.941+0000 7f114b7fe640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55f194752fe0 0x55f1947592e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.46:47748/0 (socket says 172.21.15.46:47748) 2024-03-31T23:16:09.949 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:09.941+0000 7f114bfff640 1 -- 172.21.15.46:0/3258069294 learned_addr learned my addr 172.21.15.46:0/3258069294 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:16:09.949 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:09.941+0000 7f1150ddd640 1 --2- 172.21.15.46:0/3258069294 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f19475bdb0 0x55f194759820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:16:09.950 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:09.942+0000 7f114bfff640 1 -- 172.21.15.46:0/3258069294 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f19475bdb0 msgr2=0x55f194759820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:16:09.950 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:09.942+0000 7f114bfff640 1 --2- 172.21.15.46:0/3258069294 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f19475bdb0 0x55f194759820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:16:09.950 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:09.942+0000 7f114bfff640 1 -- 172.21.15.46:0/3258069294 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55f194752fe0 msgr2=0x55f1947592e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:16:09.950 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:09.942+0000 7f114bfff640 1 --2- 172.21.15.46:0/3258069294 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55f194752fe0 0x55f1947592e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:16:09.950 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:09.942+0000 7f114bfff640 1 -- 172.21.15.46:0/3258069294 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f194496620 con 0x55f1947526c0 2024-03-31T23:16:09.950 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:09.942+0000 7f114b7fe640 1 --2- 172.21.15.46:0/3258069294 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55f194752fe0 0x55f1947592e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:16:09.950 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:09.942+0000 7f1150ddd640 1 --2- 172.21.15.46:0/3258069294 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f19475bdb0 0x55f194759820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:16:09.950 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:09.942+0000 7f114bfff640 1 --2- 172.21.15.46:0/3258069294 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f1947526c0 0x55f194758da0 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f113c01b040 tx=0x7f113c0056b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e6781a82146d02d server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:16:09.950 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:09.942+0000 7f112b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-31T23:16:09.950 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:09.942+0000 7f112b7fe640 1 -- 172.21.15.46:0/3258069294 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f113c027020 con 0x55f1947526c0 2024-03-31T23:16:09.950 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:09.942+0000 7f112b7fe640 1 -- 172.21.15.46:0/3258069294 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7f113c023460 con 0x55f1947526c0 2024-03-31T23:16:09.950 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:09.942+0000 7f1155426740 1 -- 172.21.15.46:0/3258069294 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f113c005020 con 0x55f1947526c0 2024-03-31T23:16:09.950 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:09.942+0000 7f112b7fe640 1 -- 172.21.15.46:0/3258069294 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f113c032050 con 0x55f1947526c0 2024-03-31T23:16:09.950 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:09.942+0000 7f1155426740 10 client.16509 fetch_fsmap learned FSMap version 30 2024-03-31T23:16:09.951 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:09.942+0000 7f1155426740 1 -- 172.21.15.46:0/3258069294 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55f19476a7d0 con 0x55f1947526c0 2024-03-31T23:16:09.951 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:09.942+0000 7f112b7fe640 1 -- 172.21.15.46:0/3258069294 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f113c005020 con 0x55f1947526c0 2024-03-31T23:16:09.951 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:09.943+0000 7f112b7fe640 1 -- 172.21.15.46:0/3258069294 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 30) ==== 8346+0+0 (secure 0 0 0) 0x7f113c019450 con 0x55f1947526c0 2024-03-31T23:16:09.951 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:09.943+0000 7f1155426740 10 client.16509 fetch_fsmap finished waiting for FSMap version 30 2024-03-31T23:16:09.951 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:09.943+0000 7f1155426740 10 client.16509 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-31T23:16:09.951 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:09.943+0000 7f1155426740 20 client.16509 populate_metadata read hostname 'smithi046' 2024-03-31T23:16:09.951 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:09.943+0000 7f1155426740 1 --2- 172.21.15.46:0/3258069294 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55f19476bbd0 0x55f19476dfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:16:09.951 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:09.943+0000 7f1155426740 4 client.16509 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-31T23:16:09.951 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:09.943+0000 7f1155426740 1 -- 172.21.15.46:0/3258069294 --> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f1946c9120 con 0x55f19476bbd0 2024-03-31T23:16:09.951 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:09.943+0000 7f114b7fe640 1 --2- 172.21.15.46:0/3258069294 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55f19476bbd0 0x55f19476dfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:16:09.951 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:09.943+0000 7f112b7fe640 10 client.16509 ms_handle_connect on v2:172.21.15.142:6838/2519143918 2024-03-31T23:16:09.951 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:09.943+0000 7f114b7fe640 1 --2- 172.21.15.46:0/3258069294 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55f19476bbd0 0x55f19476dfc0 crc :-1 s=READY pgs=113 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5d22fc93d35e24ea server_cookie=4e899c2e1eb40674 in_seq=0 out_seq=0 2024-03-31T23:16:09.951 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:09.944+0000 7f112b7fe640 1 -- 172.21.15.46:0/3258069294 <== mds.0 v2:172.21.15.142:6838/2519143918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55f1946c9120 con 0x55f19476bbd0 2024-03-31T23:16:09.952 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:09.944+0000 7f112b7fe640 10 client.16509 handle_command_reply: tid=0 2024-03-31T23:16:09.952 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:09.944+0000 7f1155426740 1 client.16509 shutdown 2024-03-31T23:16:09.952 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:09.944+0000 7f1155426740 1 -- 172.21.15.46:0/3258069294 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f1947526c0 msgr2=0x55f194758da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:16:09.952 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:09.944+0000 7f1155426740 1 --2- 172.21.15.46:0/3258069294 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f1947526c0 0x55f194758da0 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f113c01b040 tx=0x7f113c0056b0 comp rx=0 tx=0).stop 2024-03-31T23:16:09.952 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:09.944+0000 7f1155426740 1 -- 172.21.15.46:0/3258069294 shutdown_connections 2024-03-31T23:16:09.952 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:09.944+0000 7f1155426740 1 --2- 172.21.15.46:0/3258069294 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55f19476bbd0 0x55f19476dfc0 crc :-1 s=READY pgs=113 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:16:09.952 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:09.944+0000 7f1155426740 1 --2- 172.21.15.46:0/3258069294 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f19475bdb0 0x55f194759820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:16:09.952 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:09.944+0000 7f1155426740 1 --2- 172.21.15.46:0/3258069294 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55f194752fe0 0x55f1947592e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:16:09.952 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:09.944+0000 7f1155426740 1 --2- 172.21.15.46:0/3258069294 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f1947526c0 0x55f194758da0 unknown :-1 s=CLOSED pgs=690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:16:09.952 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:09.944+0000 7f112b7fe640 0 client.16509 ms_handle_reset on v2:172.21.15.142:6838/2519143918 2024-03-31T23:16:09.952 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:09.944+0000 7f1155426740 1 -- 172.21.15.46:0/3258069294 >> 172.21.15.46:0/3258069294 conn(0x55f19474cfe0 msgr2=0x55f19474ed30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:16:09.952 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:09.944+0000 7f1155426740 1 -- 172.21.15.46:0/3258069294 shutdown_connections 2024-03-31T23:16:09.952 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:09.944+0000 7f1155426740 1 -- 172.21.15.46:0/3258069294 wait complete. 2024-03-31T23:16:09.952 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:09.944+0000 7f1155426740 20 client.16509 trim_cache size 0 max 16384 2024-03-31T23:16:09.965 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:09.959+0000 7f1155426740 1 Processor -- start 2024-03-31T23:16:09.965 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:09.959+0000 7f1155426740 1 -- start start 2024-03-31T23:16:09.965 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:09.959+0000 7f1155426740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f1947526c0 0x55f19475b4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:16:09.965 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:09.959+0000 7f1155426740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55f194752fe0 0x55f194767780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:16:09.965 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:09.959+0000 7f1155426740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f19475bdb0 0x55f1947737c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:16:09.965 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:09.959+0000 7f1155426740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f114c1b12d0 con 0x55f19475bdb0 2024-03-31T23:16:09.965 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:09.959+0000 7f1155426740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f114c1aeba0 con 0x55f194752fe0 2024-03-31T23:16:09.965 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:09.959+0000 7f1155426740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f114c1ac9f0 con 0x55f1947526c0 2024-03-31T23:16:09.965 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:09.959+0000 7f1150ddd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f19475bdb0 0x55f1947737c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:16:09.965 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:09.960+0000 7f1150ddd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f19475bdb0 0x55f1947737c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:48686/0 (socket says 172.21.15.46:48686) 2024-03-31T23:16:09.965 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:09.960+0000 7f1150ddd640 1 -- 172.21.15.46:0/3348679857 learned_addr learned my addr 172.21.15.46:0/3348679857 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:16:09.966 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:09.960+0000 7f114b7fe640 1 --2- 172.21.15.46:0/3348679857 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55f194752fe0 0x55f194767780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:16:09.966 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:09.960+0000 7f114bfff640 1 --2- 172.21.15.46:0/3348679857 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f1947526c0 0x55f19475b4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:16:09.966 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:09.960+0000 7f1150ddd640 1 -- 172.21.15.46:0/3348679857 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f1947526c0 msgr2=0x55f19475b4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:16:09.966 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:09.960+0000 7f1150ddd640 1 --2- 172.21.15.46:0/3348679857 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f1947526c0 0x55f19475b4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:16:09.966 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:09.960+0000 7f1150ddd640 1 -- 172.21.15.46:0/3348679857 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55f194752fe0 msgr2=0x55f194767780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:16:09.966 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:09.960+0000 7f1150ddd640 1 --2- 172.21.15.46:0/3348679857 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55f194752fe0 0x55f194767780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:16:09.966 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:09.960+0000 7f1150ddd640 1 -- 172.21.15.46:0/3348679857 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f194752e10 con 0x55f19475bdb0 2024-03-31T23:16:09.966 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:09.960+0000 7f1150ddd640 1 --2- 172.21.15.46:0/3348679857 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f19475bdb0 0x55f1947737c0 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f11440ee720 tx=0x7f11440a8040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8fd96ffb592a704c server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:16:09.966 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:09.960+0000 7f1131ffb640 1 -- 172.21.15.46:0/3348679857 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11440098e0 con 0x55f19475bdb0 2024-03-31T23:16:09.966 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:09.960+0000 7f1131ffb640 1 -- 172.21.15.46:0/3348679857 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f11440326f0 con 0x55f19475bdb0 2024-03-31T23:16:09.966 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:09.960+0000 7f1131ffb640 1 -- 172.21.15.46:0/3348679857 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11440b9d10 con 0x55f19475bdb0 2024-03-31T23:16:09.966 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:09.961+0000 7f1155426740 1 -- 172.21.15.46:0/3348679857 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f19475bdb0 msgr2=0x55f1947737c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:16:09.966 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:09.961+0000 7f1155426740 1 --2- 172.21.15.46:0/3348679857 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f19475bdb0 0x55f1947737c0 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f11440ee720 tx=0x7f11440a8040 comp rx=0 tx=0).stop 2024-03-31T23:16:09.967 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:09.961+0000 7f1155426740 1 -- 172.21.15.46:0/3348679857 shutdown_connections 2024-03-31T23:16:09.967 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:09.961+0000 7f1155426740 1 --2- 172.21.15.46:0/3348679857 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f19475bdb0 0x55f1947737c0 unknown :-1 s=CLOSED pgs=691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:16:09.967 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:09.961+0000 7f1155426740 1 --2- 172.21.15.46:0/3348679857 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55f194752fe0 0x55f194767780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:16:09.967 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:09.961+0000 7f1155426740 1 --2- 172.21.15.46:0/3348679857 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f1947526c0 0x55f19475b4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:16:09.967 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:09.961+0000 7f1155426740 1 -- 172.21.15.46:0/3348679857 >> 172.21.15.46:0/3348679857 conn(0x55f19474cfe0 msgr2=0x55f194753540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:16:09.967 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:09.961+0000 7f1155426740 1 -- 172.21.15.46:0/3348679857 shutdown_connections 2024-03-31T23:16:09.967 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:09.961+0000 7f1155426740 1 -- 172.21.15.46:0/3348679857 wait complete. 2024-03-31T23:16:09.967 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:09.961+0000 7f1155426740 1 -- start start 2024-03-31T23:16:09.967 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:09.961+0000 7f1155426740 1 Processor -- start 2024-03-31T23:16:09.967 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:09.962+0000 7f1155426740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55f1947526c0 0x55f1947ca0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:16:09.967 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:09.962+0000 7f1155426740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f194752fe0 0x55f1947ca630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:16:09.967 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:09.962+0000 7f1155426740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f19475bdb0 0x55f19475ee70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:16:09.967 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:09.962+0000 7f1155426740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f114c1ad060 con 0x55f194752fe0 2024-03-31T23:16:09.967 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:09.962+0000 7f1155426740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f114c1ae4d0 con 0x55f1947526c0 2024-03-31T23:16:09.967 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:09.962+0000 7f1155426740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f114c1ad730 con 0x55f19475bdb0 2024-03-31T23:16:09.968 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:09.962+0000 7f1155426740 10 client.0 _ll_register_callbacks cb 0x7f1140009aa0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T23:16:09.968 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:09.962+0000 7f114b7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f194752fe0 0x55f1947ca630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:16:09.968 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:09.962+0000 7f114bfff640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55f1947526c0 0x55f1947ca0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:16:09.968 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:09.962+0000 7f114b7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f194752fe0 0x55f1947ca630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:48690/0 (socket says 172.21.15.46:48690) 2024-03-31T23:16:09.968 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:09.962+0000 7f114b7fe640 1 -- 172.21.15.46:0/2120136612 learned_addr learned my addr 172.21.15.46:0/2120136612 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:16:09.968 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:09.962+0000 7f1150ddd640 1 --2- 172.21.15.46:0/2120136612 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f19475bdb0 0x55f19475ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:16:09.968 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:09.962+0000 7f114b7fe640 1 -- 172.21.15.46:0/2120136612 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f19475bdb0 msgr2=0x55f19475ee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:16:09.968 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:09.962+0000 7f114b7fe640 1 --2- 172.21.15.46:0/2120136612 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f19475bdb0 0x55f19475ee70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:16:09.968 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:09.962+0000 7f114b7fe640 1 -- 172.21.15.46:0/2120136612 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55f1947526c0 msgr2=0x55f1947ca0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:16:09.968 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:09.962+0000 7f114b7fe640 1 --2- 172.21.15.46:0/2120136612 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55f1947526c0 0x55f1947ca0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:16:09.968 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:09.962+0000 7f114b7fe640 1 -- 172.21.15.46:0/2120136612 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f19475f560 con 0x55f194752fe0 2024-03-31T23:16:09.968 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:09.962+0000 7f1150ddd640 1 --2- 172.21.15.46:0/2120136612 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f19475bdb0 0x55f19475ee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T23:16:09.968 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:09.962+0000 7f114bfff640 1 --2- 172.21.15.46:0/2120136612 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55f1947526c0 0x55f1947ca0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:16:09.969 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:09.962+0000 7f114b7fe640 1 --2- 172.21.15.46:0/2120136612 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f194752fe0 0x55f1947ca630 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7f1140000c00 tx=0x7f1140004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e15adec624477087 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:16:09.969 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:09.962+0000 7f112b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-31T23:16:09.969 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:09.962+0000 7f112b7fe640 1 -- 172.21.15.46:0/2120136612 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1140027040 con 0x55f194752fe0 2024-03-31T23:16:09.969 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:09.962+0000 7f112b7fe640 1 -- 172.21.15.46:0/2120136612 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7f1140026040 con 0x55f194752fe0 2024-03-31T23:16:09.969 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:09.962+0000 7f1155426740 1 -- 172.21.15.46:0/2120136612 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55f1946c2e30 con 0x55f194752fe0 2024-03-31T23:16:09.969 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:09.963+0000 7f112b7fe640 1 -- 172.21.15.46:0/2120136612 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f114003d050 con 0x55f194752fe0 2024-03-31T23:16:09.969 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:09.963+0000 7f1155426740 10 client.16515 fetch_fsmap learned FSMap version 30 2024-03-31T23:16:09.969 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:09.963+0000 7f1155426740 1 -- 172.21.15.46:0/2120136612 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55f1947d1930 con 0x55f194752fe0 2024-03-31T23:16:09.969 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:09.963+0000 7f112b7fe640 1 -- 172.21.15.46:0/2120136612 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x55f1946c2e30 con 0x55f194752fe0 2024-03-31T23:16:09.969 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:09.963+0000 7f112b7fe640 1 -- 172.21.15.46:0/2120136612 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 30) ==== 8346+0+0 (secure 0 0 0) 0x7f114001f070 con 0x55f194752fe0 2024-03-31T23:16:09.969 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:09.963+0000 7f1155426740 10 client.16515 fetch_fsmap finished waiting for FSMap version 30 2024-03-31T23:16:09.969 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:09.963+0000 7f1155426740 10 client.16515 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-31T23:16:09.969 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:09.963+0000 7f1155426740 20 client.16515 populate_metadata read hostname 'smithi046' 2024-03-31T23:16:09.970 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:09.963+0000 7f1155426740 1 --2- 172.21.15.46:0/2120136612 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55f1947d2090 0x55f1947d44b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:16:09.970 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:09.963+0000 7f1155426740 4 client.16515 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-31T23:16:09.970 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:09.963+0000 7f1155426740 1 -- 172.21.15.46:0/2120136612 --> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55f1946bd180 con 0x55f1947d2090 2024-03-31T23:16:09.970 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:09.963+0000 7f114bfff640 1 --2- 172.21.15.46:0/2120136612 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55f1947d2090 0x55f1947d44b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:16:09.970 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:09.963+0000 7f114bfff640 1 --2- 172.21.15.46:0/2120136612 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55f1947d2090 0x55f1947d44b0 crc :-1 s=READY pgs=114 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=314c9504f0655850 server_cookie=f8f3c9f4b6d7638 in_seq=0 out_seq=0 2024-03-31T23:16:09.970 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:09.963+0000 7f112b7fe640 10 client.16515 ms_handle_connect on v2:172.21.15.142:6838/2519143918 2024-03-31T23:16:09.983 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:09.981+0000 7f112b7fe640 1 -- 172.21.15.46:0/2120136612 <== mds.0 v2:172.21.15.142:6838/2519143918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55f1946bd180 con 0x55f1947d2090 2024-03-31T23:16:09.983 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:09.981+0000 7f112b7fe640 10 client.16515 handle_command_reply: tid=0 2024-03-31T23:16:09.983 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:09.981+0000 7f1155426740 1 client.16515 shutdown 2024-03-31T23:16:09.983 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:09.981+0000 7f1155426740 1 -- 172.21.15.46:0/2120136612 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f194752fe0 msgr2=0x55f1947ca630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:16:09.983 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:09.981+0000 7f1155426740 1 --2- 172.21.15.46:0/2120136612 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f194752fe0 0x55f1947ca630 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7f1140000c00 tx=0x7f1140004730 comp rx=0 tx=0).stop 2024-03-31T23:16:09.983 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:09.981+0000 7f1155426740 1 -- 172.21.15.46:0/2120136612 shutdown_connections 2024-03-31T23:16:09.983 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:09.981+0000 7f1155426740 1 --2- 172.21.15.46:0/2120136612 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55f1947d2090 0x55f1947d44b0 crc :-1 s=READY pgs=114 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:16:09.983 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:09.981+0000 7f1155426740 1 --2- 172.21.15.46:0/2120136612 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f19475bdb0 0x55f19475ee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:16:09.983 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:09.981+0000 7f1155426740 1 --2- 172.21.15.46:0/2120136612 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f194752fe0 0x55f1947ca630 unknown :-1 s=CLOSED pgs=692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:16:09.983 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:09.981+0000 7f112b7fe640 0 client.16515 ms_handle_reset on v2:172.21.15.142:6838/2519143918 2024-03-31T23:16:09.984 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:09.981+0000 7f1155426740 1 --2- 172.21.15.46:0/2120136612 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55f1947526c0 0x55f1947ca0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:16:09.984 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:09.981+0000 7f1155426740 1 -- 172.21.15.46:0/2120136612 >> 172.21.15.46:0/2120136612 conn(0x55f19474cfe0 msgr2=0x55f194758d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:16:09.984 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:09.981+0000 7f1155426740 1 -- 172.21.15.46:0/2120136612 shutdown_connections 2024-03-31T23:16:09.984 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:09.981+0000 7f1155426740 1 -- 172.21.15.46:0/2120136612 wait complete. 2024-03-31T23:16:09.984 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:09.981+0000 7f1155426740 20 client.16515 trim_cache size 0 max 16384 2024-03-31T23:16:09.984 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout:{ 2024-03-31T23:16:09.984 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout: "message": "", 2024-03-31T23:16:09.984 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout: "return_code": 0 2024-03-31T23:16:09.984 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout:} 2024-03-31T23:16:09.985 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:09.982+0000 7f1152840640 1 -- 172.21.15.46:0/4081525920 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f11340a26b0 msgr2=0x7f11340a4b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:16:09.985 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:09.982+0000 7f1152840640 1 --2- 172.21.15.46:0/4081525920 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f11340a26b0 0x7f11340a4b70 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f114c100310 tx=0x7f1140013830 comp rx=0 tx=0).stop 2024-03-31T23:16:09.985 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:09.982+0000 7f1152840640 1 -- 172.21.15.46:0/4081525920 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f114c1055d0 msgr2=0x7f114c19c700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:16:09.985 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:09.982+0000 7f1152840640 1 --2- 172.21.15.46:0/4081525920 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f114c1055d0 0x7f114c19c700 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f114402c700 tx=0x7f1144035d60 comp rx=0 tx=0).stop 2024-03-31T23:16:09.985 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:09.982+0000 7f1152840640 1 -- 172.21.15.46:0/4081525920 shutdown_connections 2024-03-31T23:16:09.986 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:09.982+0000 7f1152840640 1 --2- 172.21.15.46:0/4081525920 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f11340a26b0 0x7f11340a4b70 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:16:09.986 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:09.982+0000 7f1152840640 1 --2- 172.21.15.46:0/4081525920 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f114c1055d0 0x7f114c19c700 unknown :-1 s=CLOSED pgs=689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:16:09.986 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:09.982+0000 7f1152840640 1 --2- 172.21.15.46:0/4081525920 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f114c101240 0x7f114c1983d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:16:09.986 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:09.982+0000 7f1152840640 1 --2- 172.21.15.46:0/4081525920 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f114c1008a0 0x7f114c197e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:16:09.986 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:09.982+0000 7f1152840640 1 -- 172.21.15.46:0/4081525920 >> 172.21.15.46:0/4081525920 conn(0x7f114c077420 msgr2=0x7f114c0fe3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:16:09.986 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:09.982+0000 7f1152840640 1 -- 172.21.15.46:0/4081525920 shutdown_connections 2024-03-31T23:16:09.986 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:09.982+0000 7f1152840640 1 -- 172.21.15.46:0/4081525920 wait complete. 2024-03-31T23:16:10.199 INFO:tasks.workunit.client.0.smithi046.stdout: 1 89319 2.51 MB/sec execute 76 sec latency 2628.510 ms 2024-03-31T23:16:10.944 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:16:10 smithi142 ceph-mon[35271]: pgmap v136: 129 pgs: 129 active+clean; 70 MiB data, 934 MiB used, 1.0 TiB / 1.0 TiB avail; 485 KiB/s rd, 1.6 MiB/s wr, 178 op/s 2024-03-31T23:16:11.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:16:10 smithi162 ceph-mon[36237]: pgmap v136: 129 pgs: 129 active+clean; 70 MiB data, 934 MiB used, 1.0 TiB / 1.0 TiB avail; 485 KiB/s rd, 1.6 MiB/s wr, 178 op/s 2024-03-31T23:16:11.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:16:10 smithi046 ceph-mon[32498]: pgmap v136: 129 pgs: 129 active+clean; 70 MiB data, 934 MiB used, 1.0 TiB / 1.0 TiB avail; 485 KiB/s rd, 1.6 MiB/s wr, 178 op/s 2024-03-31T23:16:11.196 INFO:tasks.workunit.client.0.smithi046.stdout: 1 89552 2.48 MB/sec execute 77 sec latency 3102.173 ms 2024-03-31T23:16:12.196 INFO:tasks.workunit.client.0.smithi046.stdout: 1 90111 2.51 MB/sec execute 78 sec latency 14.921 ms 2024-03-31T23:16:12.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:16:12 smithi142 ceph-mon[35271]: pgmap v137: 129 pgs: 129 active+clean; 70 MiB data, 934 MiB used, 1.0 TiB / 1.0 TiB avail; 372 KiB/s rd, 1.0 MiB/s wr, 119 op/s 2024-03-31T23:16:13.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:16:12 smithi162 ceph-mon[36237]: pgmap v137: 129 pgs: 129 active+clean; 70 MiB data, 934 MiB used, 1.0 TiB / 1.0 TiB avail; 372 KiB/s rd, 1.0 MiB/s wr, 119 op/s 2024-03-31T23:16:13.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:16:12 smithi046 ceph-mon[32498]: pgmap v137: 129 pgs: 129 active+clean; 70 MiB data, 934 MiB used, 1.0 TiB / 1.0 TiB avail; 372 KiB/s rd, 1.0 MiB/s wr, 119 op/s 2024-03-31T23:16:13.196 INFO:tasks.workunit.client.0.smithi046.stdout: 1 90566 2.48 MB/sec execute 79 sec latency 21.326 ms 2024-03-31T23:16:14.196 INFO:tasks.workunit.client.0.smithi046.stdout: 1 91110 2.47 MB/sec execute 80 sec latency 18.869 ms 2024-03-31T23:16:14.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:16:14 smithi142 ceph-mon[35271]: pgmap v138: 129 pgs: 129 active+clean; 70 MiB data, 934 MiB used, 1.0 TiB / 1.0 TiB avail; 372 KiB/s rd, 1.0 MiB/s wr, 119 op/s 2024-03-31T23:16:15.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:16:14 smithi162 ceph-mon[36237]: pgmap v138: 129 pgs: 129 active+clean; 70 MiB data, 934 MiB used, 1.0 TiB / 1.0 TiB avail; 372 KiB/s rd, 1.0 MiB/s wr, 119 op/s 2024-03-31T23:16:15.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:16:14 smithi046 ceph-mon[32498]: pgmap v138: 129 pgs: 129 active+clean; 70 MiB data, 934 MiB used, 1.0 TiB / 1.0 TiB avail; 372 KiB/s rd, 1.0 MiB/s wr, 119 op/s 2024-03-31T23:16:15.196 INFO:tasks.workunit.client.0.smithi046.stdout: 1 91713 2.51 MB/sec execute 81 sec latency 16.422 ms 2024-03-31T23:16:16.196 INFO:tasks.workunit.client.0.smithi046.stdout: 1 92116 2.50 MB/sec execute 82 sec latency 16.440 ms 2024-03-31T23:16:16.946 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:16:16 smithi142 ceph-mon[35271]: pgmap v139: 129 pgs: 129 active+clean; 65 MiB data, 957 MiB used, 1.0 TiB / 1.0 TiB avail; 479 KiB/s rd, 1.3 MiB/s wr, 163 op/s 2024-03-31T23:16:17.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:16:16 smithi162 ceph-mon[36237]: pgmap v139: 129 pgs: 129 active+clean; 65 MiB data, 957 MiB used, 1.0 TiB / 1.0 TiB avail; 479 KiB/s rd, 1.3 MiB/s wr, 163 op/s 2024-03-31T23:16:17.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:16:16 smithi046 ceph-mon[32498]: pgmap v139: 129 pgs: 129 active+clean; 65 MiB data, 957 MiB used, 1.0 TiB / 1.0 TiB avail; 479 KiB/s rd, 1.3 MiB/s wr, 163 op/s 2024-03-31T23:16:17.197 INFO:tasks.workunit.client.0.smithi046.stdout: 1 92572 2.52 MB/sec execute 83 sec latency 15.991 ms 2024-03-31T23:16:18.196 INFO:tasks.workunit.client.0.smithi046.stdout: 1 92830 2.50 MB/sec execute 84 sec latency 16.600 ms 2024-03-31T23:16:18.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:16:18 smithi142 ceph-mon[35271]: pgmap v140: 129 pgs: 129 active+clean; 65 MiB data, 957 MiB used, 1.0 TiB / 1.0 TiB avail; 281 KiB/s rd, 786 KiB/s wr, 89 op/s 2024-03-31T23:16:19.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:16:18 smithi162 ceph-mon[36237]: pgmap v140: 129 pgs: 129 active+clean; 65 MiB data, 957 MiB used, 1.0 TiB / 1.0 TiB avail; 281 KiB/s rd, 786 KiB/s wr, 89 op/s 2024-03-31T23:16:19.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:16:18 smithi046 ceph-mon[32498]: pgmap v140: 129 pgs: 129 active+clean; 65 MiB data, 957 MiB used, 1.0 TiB / 1.0 TiB avail; 281 KiB/s rd, 786 KiB/s wr, 89 op/s 2024-03-31T23:16:19.196 INFO:tasks.workunit.client.0.smithi046.stdout: 1 93250 2.48 MB/sec execute 85 sec latency 19.996 ms 2024-03-31T23:16:20.196 INFO:tasks.workunit.client.0.smithi046.stdout: 1 93799 2.50 MB/sec execute 86 sec latency 19.493 ms 2024-03-31T23:16:20.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:16:20 smithi142 ceph-mon[35271]: pgmap v141: 129 pgs: 129 active+clean; 70 MiB data, 985 MiB used, 1.0 TiB / 1.0 TiB avail; 455 KiB/s rd, 1.5 MiB/s wr, 145 op/s 2024-03-31T23:16:21.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:16:20 smithi162 ceph-mon[36237]: pgmap v141: 129 pgs: 129 active+clean; 70 MiB data, 985 MiB used, 1.0 TiB / 1.0 TiB avail; 455 KiB/s rd, 1.5 MiB/s wr, 145 op/s 2024-03-31T23:16:21.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:16:20 smithi046 ceph-mon[32498]: pgmap v141: 129 pgs: 129 active+clean; 70 MiB data, 985 MiB used, 1.0 TiB / 1.0 TiB avail; 455 KiB/s rd, 1.5 MiB/s wr, 145 op/s 2024-03-31T23:16:21.196 INFO:tasks.workunit.client.0.smithi046.stdout: 1 94237 2.47 MB/sec execute 87 sec latency 154.099 ms 2024-03-31T23:16:22.197 INFO:tasks.workunit.client.0.smithi046.stdout: 1 94711 2.46 MB/sec execute 88 sec latency 193.650 ms 2024-03-31T23:16:22.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:16:22 smithi142 ceph-mon[35271]: pgmap v142: 129 pgs: 129 active+clean; 70 MiB data, 985 MiB used, 1.0 TiB / 1.0 TiB avail; 281 KiB/s rd, 989 KiB/s wr, 100 op/s 2024-03-31T23:16:23.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:16:22 smithi162 ceph-mon[36237]: pgmap v142: 129 pgs: 129 active+clean; 70 MiB data, 985 MiB used, 1.0 TiB / 1.0 TiB avail; 281 KiB/s rd, 989 KiB/s wr, 100 op/s 2024-03-31T23:16:23.079 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:23.076+0000 7fa7ffac0640 1 -- 172.21.15.46:0/1951637941 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa7f8101260 msgr2=0x7fa7f810d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:16:23.079 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:23.076+0000 7fa7ffac0640 1 --2- 172.21.15.46:0/1951637941 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa7f8101260 0x7fa7f810d480 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7fa7f0009ca0 tx=0x7fa7f002c0f0 comp rx=0 tx=0).stop 2024-03-31T23:16:23.079 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:23.076+0000 7fa7ffac0640 1 -- 172.21.15.46:0/1951637941 shutdown_connections 2024-03-31T23:16:23.079 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:23.076+0000 7fa7ffac0640 1 --2- 172.21.15.46:0/1951637941 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa7f8101260 0x7fa7f810d480 unknown :-1 s=CLOSED pgs=693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:16:23.079 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:23.076+0000 7fa7ffac0640 1 --2- 172.21.15.46:0/1951637941 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa7f81008a0 0x7fa7f8100d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:16:23.080 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:23.076+0000 7fa7ffac0640 1 --2- 172.21.15.46:0/1951637941 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa7f81055f0 0x7fa7f81059d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:16:23.080 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:23.076+0000 7fa7ffac0640 1 -- 172.21.15.46:0/1951637941 >> 172.21.15.46:0/1951637941 conn(0x7fa7f80774a0 msgr2=0x7fa7f80778b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:16:23.080 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:23.076+0000 7fa7ffac0640 1 -- 172.21.15.46:0/1951637941 shutdown_connections 2024-03-31T23:16:23.080 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:23.076+0000 7fa7ffac0640 1 -- 172.21.15.46:0/1951637941 wait complete. 2024-03-31T23:16:23.080 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:23.077+0000 7fa7ffac0640 1 Processor -- start 2024-03-31T23:16:23.080 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:23.077+0000 7fa7ffac0640 1 -- start start 2024-03-31T23:16:23.080 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:23.077+0000 7fa7ffac0640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa7f81008a0 0x7fa7f8196dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:16:23.080 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:23.077+0000 7fa7ffac0640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa7f8101260 0x7fa7f8197300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:16:23.080 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:23.077+0000 7fa7ffac0640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa7f81055f0 0x7fa7f819b680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:16:23.080 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:23.077+0000 7fa7ffac0640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fa7f81109c0 con 0x7fa7f81008a0 2024-03-31T23:16:23.080 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:23.077+0000 7fa7ffac0640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fa7f8110840 con 0x7fa7f81055f0 2024-03-31T23:16:23.080 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:23.077+0000 7fa7ffac0640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fa7f8110b40 con 0x7fa7f8101260 2024-03-31T23:16:23.080 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:23.077+0000 7fa7fd85c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa7f81008a0 0x7fa7f8196dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:16:23.080 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:23.077+0000 7fa7fd85c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa7f81008a0 0x7fa7f8196dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37696/0 (socket says 172.21.15.46:37696) 2024-03-31T23:16:23.081 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:23.077+0000 7fa7fd85c640 1 -- 172.21.15.46:0/3663919787 learned_addr learned my addr 172.21.15.46:0/3663919787 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:16:23.081 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:23.077+0000 7fa7fd05b640 1 --2- 172.21.15.46:0/3663919787 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa7f8101260 0x7fa7f8197300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:16:23.081 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:23.077+0000 7fa7fe05d640 1 --2- 172.21.15.46:0/3663919787 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa7f81055f0 0x7fa7f819b680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:16:23.081 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:23.077+0000 7fa7fd85c640 1 -- 172.21.15.46:0/3663919787 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa7f8101260 msgr2=0x7fa7f8197300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:16:23.081 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:23.077+0000 7fa7fd85c640 1 --2- 172.21.15.46:0/3663919787 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa7f8101260 0x7fa7f8197300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:16:23.081 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:23.077+0000 7fa7fd85c640 1 -- 172.21.15.46:0/3663919787 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa7f81055f0 msgr2=0x7fa7f819b680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:16:23.081 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:23.077+0000 7fa7fd85c640 1 --2- 172.21.15.46:0/3663919787 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa7f81055f0 0x7fa7f819b680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:16:23.081 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:23.077+0000 7fa7fd85c640 1 -- 172.21.15.46:0/3663919787 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa7f819bdb0 con 0x7fa7f81008a0 2024-03-31T23:16:23.081 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:23.078+0000 7fa7fd85c640 1 --2- 172.21.15.46:0/3663919787 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa7f81008a0 0x7fa7f8196dc0 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7fa7f400e9f0 tx=0x7fa7f400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:16:23.081 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:23.078+0000 7fa7eeffd640 1 -- 172.21.15.46:0/3663919787 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7f400cd80 con 0x7fa7f81008a0 2024-03-31T23:16:23.081 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:23.078+0000 7fa7eeffd640 1 -- 172.21.15.46:0/3663919787 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa7f40108e0 con 0x7fa7f81008a0 2024-03-31T23:16:23.081 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:23.078+0000 7fa7ffac0640 1 -- 172.21.15.46:0/3663919787 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa7f819c0a0 con 0x7fa7f81008a0 2024-03-31T23:16:23.081 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:23.078+0000 7fa7eeffd640 1 -- 172.21.15.46:0/3663919787 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7f4018920 con 0x7fa7f81008a0 2024-03-31T23:16:23.081 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:23.078+0000 7fa7ffac0640 1 -- 172.21.15.46:0/3663919787 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa7f81a3930 con 0x7fa7f81008a0 2024-03-31T23:16:23.084 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:23.080+0000 7fa7eeffd640 1 -- 172.21.15.46:0/3663919787 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 25) ==== 137451+0+0 (secure 0 0 0) 0x7fa7f4010a90 con 0x7fa7f81008a0 2024-03-31T23:16:23.084 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:23.081+0000 7fa802780740 1 Processor -- start 2024-03-31T23:16:23.085 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:23.081+0000 7fa802780740 1 -- start start 2024-03-31T23:16:23.085 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:23.081+0000 7fa802780740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55905a9b4160 0x55905a9b4540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:16:23.085 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:23.081+0000 7fa802780740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55905a9b4a80 0x55905a9bfa80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:16:23.085 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:23.081+0000 7fa802780740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55905a9bffc0 0x55905a9c23b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:16:23.085 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:23.081+0000 7fa802780740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55905a9178f0 con 0x55905a9b4160 2024-03-31T23:16:23.085 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:23.081+0000 7fa802780740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55905a8c08d0 con 0x55905a9bffc0 2024-03-31T23:16:23.085 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:23.081+0000 7fa802780740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55905a913990 con 0x55905a9b4a80 2024-03-31T23:16:23.085 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:23.081+0000 7fa7eeffd640 1 --2- 172.21.15.46:0/3663919787 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fa7e40a2730 0x7fa7e40a4bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:16:23.085 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:23.081+0000 7fa7fe05d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55905a9b4160 0x55905a9b4540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:16:23.085 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:23.081+0000 7fa7fd85c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55905a9b4a80 0x55905a9bfa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:16:23.085 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:23.081+0000 7fa7fe05d640 1 --2- 172.21.15.46:0/3663919787 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fa7e40a2730 0x7fa7e40a4bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:16:23.085 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:23.081+0000 7fa7fd05b640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55905a9bffc0 0x55905a9c23b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:16:23.085 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:23.081+0000 7fa7fd05b640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55905a9bffc0 0x55905a9c23b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.46:47202/0 (socket says 172.21.15.46:47202) 2024-03-31T23:16:23.085 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:23.081+0000 7fa7fd05b640 1 -- 172.21.15.46:0/3825522956 learned_addr learned my addr 172.21.15.46:0/3825522956 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:16:23.086 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:23.082+0000 7fa7eeffd640 1 -- 172.21.15.46:0/3663919787 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7fa7f40cf0e0 con 0x7fa7f81008a0 2024-03-31T23:16:23.086 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:23.082+0000 7fa7fd05b640 1 -- 172.21.15.46:0/3825522956 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55905a9b4a80 msgr2=0x55905a9bfa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:16:23.086 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:23.082+0000 7fa7fd05b640 1 --2- 172.21.15.46:0/3825522956 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55905a9b4a80 0x55905a9bfa80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:16:23.086 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:23.082+0000 7fa7fd05b640 1 -- 172.21.15.46:0/3825522956 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55905a9b4160 msgr2=0x55905a9b4540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:16:23.086 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:23.082+0000 7fa7fd05b640 1 --2- 172.21.15.46:0/3825522956 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55905a9b4160 0x55905a9b4540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:16:23.086 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:23.082+0000 7fa7fd05b640 1 -- 172.21.15.46:0/3825522956 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55905a62c210 con 0x55905a9bffc0 2024-03-31T23:16:23.086 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:23.082+0000 7fa7fd85c640 1 --2- 172.21.15.46:0/3825522956 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55905a9b4a80 0x55905a9bfa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:16:23.086 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:23.082+0000 7fa7fe05d640 1 --2- 172.21.15.46:0/3663919787 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fa7e40a2730 0x7fa7e40a4bf0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fa7f0004af0 tx=0x7fa7f0009360 comp rx=0 tx=0).ready entity=mgr.15654 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:16:23.086 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:23.082+0000 7fa7fd05b640 1 --2- 172.21.15.46:0/3825522956 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55905a9bffc0 0x55905a9c23b0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fa7e8009cd0 tx=0x7fa7e8011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=62301e4b016c17f server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:16:23.086 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:23.082+0000 7fa7e3fff640 1 -- 172.21.15.46:0/3825522956 <== mon.2 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7e801e070 con 0x55905a9bffc0 2024-03-31T23:16:23.086 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:23.082+0000 7fa7e3fff640 1 -- 172.21.15.46:0/3825522956 <== mon.2 v2:172.21.15.142:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa7e8019ae0 con 0x55905a9bffc0 2024-03-31T23:16:23.086 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:23.082+0000 7fa7e3fff640 1 -- 172.21.15.46:0/3825522956 <== mon.2 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7e800e950 con 0x55905a9bffc0 2024-03-31T23:16:23.086 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:23.083+0000 7fa802780740 1 -- 172.21.15.46:0/3825522956 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55905a9bffc0 msgr2=0x55905a9c23b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:16:23.087 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:23.083+0000 7fa802780740 1 --2- 172.21.15.46:0/3825522956 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55905a9bffc0 0x55905a9c23b0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fa7e8009cd0 tx=0x7fa7e8011810 comp rx=0 tx=0).stop 2024-03-31T23:16:23.087 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:23.083+0000 7fa802780740 1 -- 172.21.15.46:0/3825522956 shutdown_connections 2024-03-31T23:16:23.087 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:23.083+0000 7fa802780740 1 --2- 172.21.15.46:0/3825522956 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55905a9bffc0 0x55905a9c23b0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:16:23.087 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:23.083+0000 7fa802780740 1 --2- 172.21.15.46:0/3825522956 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55905a9b4a80 0x55905a9bfa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:16:23.087 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:23.083+0000 7fa802780740 1 --2- 172.21.15.46:0/3825522956 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55905a9b4160 0x55905a9b4540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:16:23.087 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:23.083+0000 7fa802780740 1 -- 172.21.15.46:0/3825522956 >> 172.21.15.46:0/3825522956 conn(0x55905a9aefe0 msgr2=0x55905a9af3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:16:23.087 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:23.083+0000 7fa802780740 1 -- 172.21.15.46:0/3825522956 shutdown_connections 2024-03-31T23:16:23.087 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:23.083+0000 7fa802780740 1 -- 172.21.15.46:0/3825522956 wait complete. 2024-03-31T23:16:23.087 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:23.083+0000 7fa802780740 1 -- start start 2024-03-31T23:16:23.087 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:23.084+0000 7fa802780740 1 Processor -- start 2024-03-31T23:16:23.091 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:23.084+0000 7fa802780740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55905a9b4160 0x55905a9bcf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:16:23.091 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:23.084+0000 7fa802780740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55905a9b4a80 0x55905a9bd490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:16:23.091 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:23.084+0000 7fa802780740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55905a9bffc0 0x55905a9bd9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:16:23.091 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:23.084+0000 7fa802780740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55905a9c2bf0 con 0x55905a9b4a80 2024-03-31T23:16:23.091 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:23.084+0000 7fa802780740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55905a8b5f80 con 0x55905a9bffc0 2024-03-31T23:16:23.091 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:23.084+0000 7fa802780740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55905a9c2d70 con 0x55905a9b4160 2024-03-31T23:16:23.091 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:23.084+0000 7fa802780740 10 client.0 _ll_register_callbacks cb 0x55905a90e290 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T23:16:23.091 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:23.084+0000 7fa7fd05b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55905a9b4a80 0x55905a9bd490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:16:23.092 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:23.084+0000 7fa7fd05b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55905a9b4a80 0x55905a9bd490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37722/0 (socket says 172.21.15.46:37722) 2024-03-31T23:16:23.092 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:23.084+0000 7fa7fd05b640 1 -- 172.21.15.46:0/1525689708 learned_addr learned my addr 172.21.15.46:0/1525689708 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:16:23.092 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:23.084+0000 7fa7fd85c640 1 --2- 172.21.15.46:0/1525689708 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55905a9b4160 0x55905a9bcf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:16:23.092 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:23.084+0000 7fa7fe05d640 1 --2- 172.21.15.46:0/1525689708 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55905a9bffc0 0x55905a9bd9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:16:23.092 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:23.084+0000 7fa7fd05b640 1 -- 172.21.15.46:0/1525689708 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55905a9b4160 msgr2=0x55905a9bcf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:16:23.092 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:23.084+0000 7fa7fd05b640 1 --2- 172.21.15.46:0/1525689708 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55905a9b4160 0x55905a9bcf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:16:23.092 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:23.084+0000 7fa7fd05b640 1 -- 172.21.15.46:0/1525689708 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55905a9bffc0 msgr2=0x55905a9bd9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:16:23.092 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:23.084+0000 7fa7fd05b640 1 --2- 172.21.15.46:0/1525689708 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55905a9bffc0 0x55905a9bd9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:16:23.092 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:23.084+0000 7fa7fd05b640 1 -- 172.21.15.46:0/1525689708 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55905a6f8620 con 0x55905a9b4a80 2024-03-31T23:16:23.092 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:23.085+0000 7fa7fd05b640 1 --2- 172.21.15.46:0/1525689708 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55905a9b4a80 0x55905a9bd490 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7fa7e8005890 tx=0x7fa7e8019660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=73ac796fb2012946 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:16:23.092 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:23.085+0000 7fa7e17fa640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-31T23:16:23.092 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:23.085+0000 7fa7e17fa640 1 -- 172.21.15.46:0/1525689708 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7e8027030 con 0x55905a9b4a80 2024-03-31T23:16:23.092 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:23.085+0000 7fa802780740 1 -- 172.21.15.46:0/1525689708 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa7e8019ae0 con 0x55905a9b4a80 2024-03-31T23:16:23.092 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:23.085+0000 7fa7e17fa640 1 -- 172.21.15.46:0/1525689708 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7fa7e8004020 con 0x55905a9b4a80 2024-03-31T23:16:23.093 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:23.085+0000 7fa7e17fa640 1 -- 172.21.15.46:0/1525689708 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fa7e8031050 con 0x55905a9b4a80 2024-03-31T23:16:23.093 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:23.085+0000 7fa802780740 10 client.16527 fetch_fsmap learned FSMap version 30 2024-03-31T23:16:23.093 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:23.085+0000 7fa802780740 1 -- 172.21.15.46:0/1525689708 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55905a9ceb00 con 0x55905a9b4a80 2024-03-31T23:16:23.093 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:23.085+0000 7fa7e17fa640 1 -- 172.21.15.46:0/1525689708 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa7e8019ae0 con 0x55905a9b4a80 2024-03-31T23:16:23.093 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:23.085+0000 7fa7e17fa640 1 -- 172.21.15.46:0/1525689708 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 30) ==== 8346+0+0 (secure 0 0 0) 0x7fa7e80046c0 con 0x55905a9b4a80 2024-03-31T23:16:23.093 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:23.085+0000 7fa802780740 10 client.16527 fetch_fsmap finished waiting for FSMap version 30 2024-03-31T23:16:23.093 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:23.085+0000 7fa802780740 10 client.16527 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-31T23:16:23.093 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:23.085+0000 7fa802780740 20 client.16527 populate_metadata read hostname 'smithi046' 2024-03-31T23:16:23.093 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:23.085+0000 7fa802780740 1 --2- 172.21.15.46:0/1525689708 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55905a9cff10 0x55905a9d2300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:16:23.093 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:23.086+0000 7fa802780740 4 client.16527 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-31T23:16:23.093 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:23.086+0000 7fa802780740 1 -- 172.21.15.46:0/1525689708 --> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55905a92b120 con 0x55905a9cff10 2024-03-31T23:16:23.093 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:23.086+0000 7fa7fd85c640 1 --2- 172.21.15.46:0/1525689708 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55905a9cff10 0x55905a9d2300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:16:23.093 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:23.086+0000 7fa7e17fa640 10 client.16527 ms_handle_connect on v2:172.21.15.142:6838/2519143918 2024-03-31T23:16:23.093 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:23.086+0000 7fa7fd85c640 1 --2- 172.21.15.46:0/1525689708 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55905a9cff10 0x55905a9d2300 crc :-1 s=READY pgs=115 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a93b8c6a6414961 server_cookie=f47a4e3141623f1 in_seq=0 out_seq=0 2024-03-31T23:16:23.094 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:23.087+0000 7fa7e17fa640 1 -- 172.21.15.46:0/1525689708 <== mds.0 v2:172.21.15.142:6838/2519143918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55905a92b120 con 0x55905a9cff10 2024-03-31T23:16:23.094 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:23.087+0000 7fa7e17fa640 10 client.16527 handle_command_reply: tid=0 2024-03-31T23:16:23.094 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:23.087+0000 7fa802780740 1 client.16527 shutdown 2024-03-31T23:16:23.094 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:23.087+0000 7fa802780740 1 -- 172.21.15.46:0/1525689708 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55905a9b4a80 msgr2=0x55905a9bd490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:16:23.094 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:23.087+0000 7fa802780740 1 --2- 172.21.15.46:0/1525689708 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55905a9b4a80 0x55905a9bd490 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7fa7e8005890 tx=0x7fa7e8019660 comp rx=0 tx=0).stop 2024-03-31T23:16:23.094 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:23.087+0000 7fa802780740 1 -- 172.21.15.46:0/1525689708 shutdown_connections 2024-03-31T23:16:23.094 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:23.087+0000 7fa802780740 1 --2- 172.21.15.46:0/1525689708 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55905a9cff10 0x55905a9d2300 crc :-1 s=READY pgs=115 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:16:23.094 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:23.087+0000 7fa802780740 1 --2- 172.21.15.46:0/1525689708 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55905a9bffc0 0x55905a9bd9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:16:23.094 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:23.087+0000 7fa802780740 1 --2- 172.21.15.46:0/1525689708 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55905a9b4a80 0x55905a9bd490 unknown :-1 s=CLOSED pgs=695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:16:23.094 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:23.087+0000 7fa7e17fa640 0 client.16527 ms_handle_reset on v2:172.21.15.142:6838/2519143918 2024-03-31T23:16:23.094 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:23.087+0000 7fa802780740 1 --2- 172.21.15.46:0/1525689708 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55905a9b4160 0x55905a9bcf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:16:23.094 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:23.087+0000 7fa802780740 1 -- 172.21.15.46:0/1525689708 >> 172.21.15.46:0/1525689708 conn(0x55905a9aefe0 msgr2=0x55905a9b0d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:16:23.094 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:23.087+0000 7fa802780740 1 -- 172.21.15.46:0/1525689708 shutdown_connections 2024-03-31T23:16:23.094 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:23.087+0000 7fa802780740 1 -- 172.21.15.46:0/1525689708 wait complete. 2024-03-31T23:16:23.095 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:23.087+0000 7fa802780740 20 client.16527 trim_cache size 0 max 16384 2024-03-31T23:16:23.107 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:23.104+0000 7fa802780740 1 Processor -- start 2024-03-31T23:16:23.107 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:23.104+0000 7fa802780740 1 -- start start 2024-03-31T23:16:23.107 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:23.104+0000 7fa802780740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55905a9b4160 0x55905a9bf680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:16:23.107 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:23.104+0000 7fa802780740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55905a9b4a80 0x55905a9cba30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:16:23.107 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:23.104+0000 7fa802780740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55905a9bffc0 0x55905a9a9bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:16:23.107 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:23.104+0000 7fa802780740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fa7f81af1d0 con 0x55905a9bffc0 2024-03-31T23:16:23.107 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:23.104+0000 7fa802780740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fa7f81acaa0 con 0x55905a9b4a80 2024-03-31T23:16:23.107 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:23.104+0000 7fa802780740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fa7f81aa8f0 con 0x55905a9b4160 2024-03-31T23:16:23.107 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:23.104+0000 7fa7fe05d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55905a9bffc0 0x55905a9a9bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:16:23.107 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:23.104+0000 7fa7fd85c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55905a9b4160 0x55905a9bf680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:16:23.107 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:23.104+0000 7fa7fd05b640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55905a9b4a80 0x55905a9cba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:16:23.107 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:23.104+0000 7fa7fd85c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55905a9b4160 0x55905a9bf680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.46:55960/0 (socket says 172.21.15.46:55960) 2024-03-31T23:16:23.108 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:23.104+0000 7fa7fe05d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55905a9bffc0 0x55905a9a9bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37726/0 (socket says 172.21.15.46:37726) 2024-03-31T23:16:23.108 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:23.104+0000 7fa7fd85c640 1 -- 172.21.15.46:0/1823974334 learned_addr learned my addr 172.21.15.46:0/1823974334 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:16:23.108 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:23.104+0000 7fa7fd85c640 1 -- 172.21.15.46:0/1823974334 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55905a9b4a80 msgr2=0x55905a9cba30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:16:23.108 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:23.104+0000 7fa7fd85c640 1 --2- 172.21.15.46:0/1823974334 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55905a9b4a80 0x55905a9cba30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:16:23.108 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:23.104+0000 7fa7fd85c640 1 -- 172.21.15.46:0/1823974334 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55905a9bffc0 msgr2=0x55905a9a9bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:16:23.108 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:23.104+0000 7fa7fd85c640 1 --2- 172.21.15.46:0/1823974334 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55905a9bffc0 0x55905a9a9bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:16:23.108 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:23.104+0000 7fa7fd85c640 1 -- 172.21.15.46:0/1823974334 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55905a9bbf00 con 0x55905a9b4160 2024-03-31T23:16:23.108 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:23.104+0000 7fa7fd05b640 1 --2- 172.21.15.46:0/1823974334 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55905a9b4a80 0x55905a9cba30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:16:23.108 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:23.104+0000 7fa7fe05d640 1 --2- 172.21.15.46:0/1823974334 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55905a9bffc0 0x55905a9a9bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:16:23.108 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:23.105+0000 7fa7fd85c640 1 --2- 172.21.15.46:0/1823974334 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55905a9b4160 0x55905a9bf680 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fa7f40979c0 tx=0x7fa7f40664b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9b4181aee64e7462 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:16:23.108 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:23.105+0000 7fa7e37fe640 1 -- 172.21.15.46:0/1823974334 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7f4026040 con 0x55905a9b4160 2024-03-31T23:16:23.108 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:23.105+0000 7fa7e37fe640 1 -- 172.21.15.46:0/1823974334 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa7f40a3ec0 con 0x55905a9b4160 2024-03-31T23:16:23.108 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:23.105+0000 7fa7e37fe640 1 -- 172.21.15.46:0/1823974334 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7f40a3a30 con 0x55905a9b4160 2024-03-31T23:16:23.108 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:23.105+0000 7fa802780740 1 -- 172.21.15.46:0/1823974334 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55905a9b4160 msgr2=0x55905a9bf680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:16:23.109 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:23.105+0000 7fa802780740 1 --2- 172.21.15.46:0/1823974334 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55905a9b4160 0x55905a9bf680 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fa7f40979c0 tx=0x7fa7f40664b0 comp rx=0 tx=0).stop 2024-03-31T23:16:23.109 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:23.105+0000 7fa802780740 1 -- 172.21.15.46:0/1823974334 shutdown_connections 2024-03-31T23:16:23.109 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:23.105+0000 7fa802780740 1 --2- 172.21.15.46:0/1823974334 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55905a9bffc0 0x55905a9a9bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:16:23.109 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:23.105+0000 7fa802780740 1 --2- 172.21.15.46:0/1823974334 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55905a9b4a80 0x55905a9cba30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:16:23.109 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:23.105+0000 7fa802780740 1 --2- 172.21.15.46:0/1823974334 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55905a9b4160 0x55905a9bf680 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:16:23.109 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:23.105+0000 7fa802780740 1 -- 172.21.15.46:0/1823974334 >> 172.21.15.46:0/1823974334 conn(0x55905a9aefe0 msgr2=0x55905a9adf80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:16:23.109 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:23.105+0000 7fa802780740 1 -- 172.21.15.46:0/1823974334 shutdown_connections 2024-03-31T23:16:23.109 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:23.105+0000 7fa802780740 1 -- 172.21.15.46:0/1823974334 wait complete. 2024-03-31T23:16:23.109 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:23.106+0000 7fa802780740 1 -- start start 2024-03-31T23:16:23.109 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:23.106+0000 7fa802780740 1 Processor -- start 2024-03-31T23:16:23.112 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:23.106+0000 7fa802780740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55905a9b4160 0x55905a9c82d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:16:23.112 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:23.106+0000 7fa802780740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55905a9b4a80 0x55905a9c3640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:16:23.112 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:23.106+0000 7fa802780740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55905a9bffc0 0x55905a9c3b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:16:23.112 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:23.106+0000 7fa802780740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fa7f81aaf60 con 0x55905a9bffc0 2024-03-31T23:16:23.112 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:23.106+0000 7fa802780740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fa7f81ac3d0 con 0x55905a9b4160 2024-03-31T23:16:23.112 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:23.106+0000 7fa802780740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fa7f81ab630 con 0x55905a9b4a80 2024-03-31T23:16:23.112 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:23.106+0000 7fa802780740 10 client.0 _ll_register_callbacks cb 0x7fa7e801c070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T23:16:23.112 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:23.106+0000 7fa7fe05d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55905a9bffc0 0x55905a9c3b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:16:23.112 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:23.106+0000 7fa7fe05d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55905a9bffc0 0x55905a9c3b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37730/0 (socket says 172.21.15.46:37730) 2024-03-31T23:16:23.112 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:23.106+0000 7fa7fe05d640 1 -- 172.21.15.46:0/1239398542 learned_addr learned my addr 172.21.15.46:0/1239398542 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:16:23.112 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:23.106+0000 7fa7fd05b640 1 --2- 172.21.15.46:0/1239398542 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55905a9b4a80 0x55905a9c3640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:16:23.113 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:23.106+0000 7fa7fd85c640 1 --2- 172.21.15.46:0/1239398542 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55905a9b4160 0x55905a9c82d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:16:23.113 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:23.107+0000 7fa7fe05d640 1 -- 172.21.15.46:0/1239398542 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55905a9b4a80 msgr2=0x55905a9c3640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:16:23.113 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:23.107+0000 7fa7fe05d640 1 --2- 172.21.15.46:0/1239398542 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55905a9b4a80 0x55905a9c3640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:16:23.113 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:23.107+0000 7fa7fe05d640 1 -- 172.21.15.46:0/1239398542 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55905a9b4160 msgr2=0x55905a9c82d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:16:23.113 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:23.107+0000 7fa7fe05d640 1 --2- 172.21.15.46:0/1239398542 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55905a9b4160 0x55905a9c82d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:16:23.113 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:23.107+0000 7fa7fe05d640 1 -- 172.21.15.46:0/1239398542 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55905a9d2670 con 0x55905a9bffc0 2024-03-31T23:16:23.113 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:23.107+0000 7fa7fe05d640 1 --2- 172.21.15.46:0/1239398542 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55905a9bffc0 0x55905a9c3b80 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7fa7f0037040 tx=0x7fa7f002e040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a37941a14da51069 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:16:23.113 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:23.107+0000 7fa7e17fa640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-31T23:16:23.113 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:23.107+0000 7fa7e17fa640 1 -- 172.21.15.46:0/1239398542 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7f0052020 con 0x55905a9bffc0 2024-03-31T23:16:23.113 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:23.107+0000 7fa7e17fa640 1 -- 172.21.15.46:0/1239398542 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7fa7f0032880 con 0x55905a9bffc0 2024-03-31T23:16:23.113 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:23.107+0000 7fa802780740 1 -- 172.21.15.46:0/1239398542 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa7f40a3ec0 con 0x55905a9bffc0 2024-03-31T23:16:23.113 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:23.107+0000 7fa7e17fa640 1 -- 172.21.15.46:0/1239398542 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fa7f005d050 con 0x55905a9bffc0 2024-03-31T23:16:23.113 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:23.107+0000 7fa802780740 10 client.16533 fetch_fsmap learned FSMap version 30 2024-03-31T23:16:23.114 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:23.107+0000 7fa802780740 1 -- 172.21.15.46:0/1239398542 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55905aa35f30 con 0x55905a9bffc0 2024-03-31T23:16:23.114 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:23.107+0000 7fa7e17fa640 1 -- 172.21.15.46:0/1239398542 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa7f40a3ec0 con 0x55905a9bffc0 2024-03-31T23:16:23.114 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:23.107+0000 7fa7e17fa640 1 -- 172.21.15.46:0/1239398542 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 30) ==== 8346+0+0 (secure 0 0 0) 0x7fa7f003f040 con 0x55905a9bffc0 2024-03-31T23:16:23.114 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:23.107+0000 7fa802780740 10 client.16533 fetch_fsmap finished waiting for FSMap version 30 2024-03-31T23:16:23.114 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:23.107+0000 7fa802780740 10 client.16533 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-31T23:16:23.114 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:23.107+0000 7fa802780740 20 client.16533 populate_metadata read hostname 'smithi046' 2024-03-31T23:16:23.114 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:23.107+0000 7fa802780740 1 --2- 172.21.15.46:0/1239398542 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55905aa36700 0x55905aa38b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:16:23.114 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:23.107+0000 7fa802780740 4 client.16533 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-31T23:16:23.114 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:23.108+0000 7fa802780740 1 -- 172.21.15.46:0/1239398542 --> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55905a924e30 con 0x55905aa36700 2024-03-31T23:16:23.114 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:23.108+0000 7fa7fd85c640 1 --2- 172.21.15.46:0/1239398542 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55905aa36700 0x55905aa38b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:16:23.114 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:23.108+0000 7fa7fd85c640 1 --2- 172.21.15.46:0/1239398542 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55905aa36700 0x55905aa38b20 crc :-1 s=READY pgs=116 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7da0480cd53543dd server_cookie=d0afc78b45ef7760 in_seq=0 out_seq=0 2024-03-31T23:16:23.114 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:23.108+0000 7fa7e17fa640 10 client.16533 ms_handle_connect on v2:172.21.15.142:6838/2519143918 2024-03-31T23:16:23.115 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:16:22 smithi046 ceph-mon[32498]: pgmap v142: 129 pgs: 129 active+clean; 70 MiB data, 985 MiB used, 1.0 TiB / 1.0 TiB avail; 281 KiB/s rd, 989 KiB/s wr, 100 op/s 2024-03-31T23:16:23.128 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:23.126+0000 7fa7e17fa640 1 -- 172.21.15.46:0/1239398542 <== mds.0 v2:172.21.15.142:6838/2519143918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55905a924e30 con 0x55905aa36700 2024-03-31T23:16:23.128 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:23.126+0000 7fa7e17fa640 10 client.16533 handle_command_reply: tid=0 2024-03-31T23:16:23.129 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:23.126+0000 7fa802780740 1 client.16533 shutdown 2024-03-31T23:16:23.129 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:23.126+0000 7fa802780740 1 -- 172.21.15.46:0/1239398542 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55905a9bffc0 msgr2=0x55905a9c3b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:16:23.129 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:23.126+0000 7fa802780740 1 --2- 172.21.15.46:0/1239398542 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55905a9bffc0 0x55905a9c3b80 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7fa7f0037040 tx=0x7fa7f002e040 comp rx=0 tx=0).stop 2024-03-31T23:16:23.129 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:23.126+0000 7fa802780740 1 -- 172.21.15.46:0/1239398542 shutdown_connections 2024-03-31T23:16:23.129 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:23.127+0000 7fa802780740 1 --2- 172.21.15.46:0/1239398542 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55905aa36700 0x55905aa38b20 crc :-1 s=READY pgs=116 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:16:23.129 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:23.127+0000 7fa7e17fa640 0 client.16533 ms_handle_reset on v2:172.21.15.142:6838/2519143918 2024-03-31T23:16:23.129 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:23.127+0000 7fa802780740 1 --2- 172.21.15.46:0/1239398542 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55905a9bffc0 0x55905a9c3b80 unknown :-1 s=CLOSED pgs=696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:16:23.129 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:23.127+0000 7fa802780740 1 --2- 172.21.15.46:0/1239398542 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55905a9b4a80 0x55905a9c3640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:16:23.129 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:23.127+0000 7fa802780740 1 --2- 172.21.15.46:0/1239398542 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55905a9b4160 0x55905a9c82d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:16:23.129 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:23.127+0000 7fa802780740 1 -- 172.21.15.46:0/1239398542 >> 172.21.15.46:0/1239398542 conn(0x55905a9aefe0 msgr2=0x55905a9adf80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:16:23.129 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:23.127+0000 7fa802780740 1 -- 172.21.15.46:0/1239398542 shutdown_connections 2024-03-31T23:16:23.130 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:23.127+0000 7fa802780740 1 -- 172.21.15.46:0/1239398542 wait complete. 2024-03-31T23:16:23.130 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:23.127+0000 7fa802780740 20 client.16533 trim_cache size 0 max 16384 2024-03-31T23:16:23.130 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout:{ 2024-03-31T23:16:23.130 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout: "message": "", 2024-03-31T23:16:23.130 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout: "return_code": 0 2024-03-31T23:16:23.131 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout:} 2024-03-31T23:16:23.131 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:23.127+0000 7fa7ffac0640 1 -- 172.21.15.46:0/3663919787 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fa7e40a2730 msgr2=0x7fa7e40a4bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:16:23.131 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:23.127+0000 7fa7ffac0640 1 --2- 172.21.15.46:0/3663919787 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fa7e40a2730 0x7fa7e40a4bf0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fa7f0004af0 tx=0x7fa7f0009360 comp rx=0 tx=0).stop 2024-03-31T23:16:23.132 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:23.127+0000 7fa7ffac0640 1 -- 172.21.15.46:0/3663919787 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa7f81008a0 msgr2=0x7fa7f8196dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:16:23.132 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:23.127+0000 7fa7ffac0640 1 --2- 172.21.15.46:0/3663919787 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa7f81008a0 0x7fa7f8196dc0 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7fa7f400e9f0 tx=0x7fa7f400eeb0 comp rx=0 tx=0).stop 2024-03-31T23:16:23.132 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:23.128+0000 7fa7ffac0640 1 -- 172.21.15.46:0/3663919787 shutdown_connections 2024-03-31T23:16:23.132 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:23.128+0000 7fa7ffac0640 1 --2- 172.21.15.46:0/3663919787 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fa7e40a2730 0x7fa7e40a4bf0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:16:23.132 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:23.128+0000 7fa7ffac0640 1 --2- 172.21.15.46:0/3663919787 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa7f81055f0 0x7fa7f819b680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:16:23.132 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:23.128+0000 7fa7ffac0640 1 --2- 172.21.15.46:0/3663919787 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa7f8101260 0x7fa7f8197300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:16:23.132 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:23.128+0000 7fa7ffac0640 1 --2- 172.21.15.46:0/3663919787 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa7f81008a0 0x7fa7f8196dc0 unknown :-1 s=CLOSED pgs=694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:16:23.132 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:23.128+0000 7fa7ffac0640 1 -- 172.21.15.46:0/3663919787 >> 172.21.15.46:0/3663919787 conn(0x7fa7f80774a0 msgr2=0x7fa7f80fe7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:16:23.132 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:23.128+0000 7fa7ffac0640 1 -- 172.21.15.46:0/3663919787 shutdown_connections 2024-03-31T23:16:23.132 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:23.128+0000 7fa7ffac0640 1 -- 172.21.15.46:0/3663919787 wait complete. 2024-03-31T23:16:23.196 INFO:tasks.workunit.client.0.smithi046.stdout: 1 95451 2.50 MB/sec execute 89 sec latency 14.829 ms 2024-03-31T23:16:24.196 INFO:tasks.workunit.client.0.smithi046.stdout: 1 95848 2.52 MB/sec execute 90 sec latency 17.155 ms 2024-03-31T23:16:24.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:16:24 smithi142 ceph-mon[35271]: pgmap v143: 129 pgs: 129 active+clean; 70 MiB data, 985 MiB used, 1.0 TiB / 1.0 TiB avail; 281 KiB/s rd, 989 KiB/s wr, 100 op/s 2024-03-31T23:16:25.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:16:24 smithi162 ceph-mon[36237]: pgmap v143: 129 pgs: 129 active+clean; 70 MiB data, 985 MiB used, 1.0 TiB / 1.0 TiB avail; 281 KiB/s rd, 989 KiB/s wr, 100 op/s 2024-03-31T23:16:25.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:16:24 smithi046 ceph-mon[32498]: pgmap v143: 129 pgs: 129 active+clean; 70 MiB data, 985 MiB used, 1.0 TiB / 1.0 TiB avail; 281 KiB/s rd, 989 KiB/s wr, 100 op/s 2024-03-31T23:16:25.196 INFO:tasks.workunit.client.0.smithi046.stdout: 1 96193 2.51 MB/sec execute 91 sec latency 17.219 ms 2024-03-31T23:16:26.197 INFO:tasks.workunit.client.0.smithi046.stdout: 1 96474 2.49 MB/sec execute 92 sec latency 19.911 ms 2024-03-31T23:16:26.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:16:26 smithi142 ceph-mon[35271]: pgmap v144: 129 pgs: 129 active+clean; 71 MiB data, 990 MiB used, 1.0 TiB / 1.0 TiB avail; 441 KiB/s rd, 1.5 MiB/s wr, 154 op/s 2024-03-31T23:16:27.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:16:26 smithi162 ceph-mon[36237]: pgmap v144: 129 pgs: 129 active+clean; 71 MiB data, 990 MiB used, 1.0 TiB / 1.0 TiB avail; 441 KiB/s rd, 1.5 MiB/s wr, 154 op/s 2024-03-31T23:16:27.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:16:26 smithi046 ceph-mon[32498]: pgmap v144: 129 pgs: 129 active+clean; 71 MiB data, 990 MiB used, 1.0 TiB / 1.0 TiB avail; 441 KiB/s rd, 1.5 MiB/s wr, 154 op/s 2024-03-31T23:16:27.196 INFO:tasks.workunit.client.0.smithi046.stdout: 1 96972 2.48 MB/sec execute 93 sec latency 16.366 ms 2024-03-31T23:16:28.197 INFO:tasks.workunit.client.0.smithi046.stdout: 1 97466 2.49 MB/sec execute 94 sec latency 25.082 ms 2024-03-31T23:16:28.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:16:28 smithi142 ceph-mon[35271]: pgmap v145: 129 pgs: 129 active+clean; 71 MiB data, 990 MiB used, 1.0 TiB / 1.0 TiB avail; 334 KiB/s rd, 1.3 MiB/s wr, 110 op/s 2024-03-31T23:16:29.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:16:28 smithi162 ceph-mon[36237]: pgmap v145: 129 pgs: 129 active+clean; 71 MiB data, 990 MiB used, 1.0 TiB / 1.0 TiB avail; 334 KiB/s rd, 1.3 MiB/s wr, 110 op/s 2024-03-31T23:16:29.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:16:28 smithi046 ceph-mon[32498]: pgmap v145: 129 pgs: 129 active+clean; 71 MiB data, 990 MiB used, 1.0 TiB / 1.0 TiB avail; 334 KiB/s rd, 1.3 MiB/s wr, 110 op/s 2024-03-31T23:16:29.197 INFO:tasks.workunit.client.0.smithi046.stdout: 1 98039 2.47 MB/sec execute 95 sec latency 20.499 ms 2024-03-31T23:16:30.197 INFO:tasks.workunit.client.0.smithi046.stdout: 1 98535 2.50 MB/sec execute 96 sec latency 15.648 ms 2024-03-31T23:16:30.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:16:30 smithi142 ceph-mon[35271]: pgmap v146: 129 pgs: 129 active+clean; 71 MiB data, 999 MiB used, 1.0 TiB / 1.0 TiB avail; 553 KiB/s rd, 1.8 MiB/s wr, 172 op/s 2024-03-31T23:16:31.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:16:30 smithi162 ceph-mon[36237]: pgmap v146: 129 pgs: 129 active+clean; 71 MiB data, 999 MiB used, 1.0 TiB / 1.0 TiB avail; 553 KiB/s rd, 1.8 MiB/s wr, 172 op/s 2024-03-31T23:16:31.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:16:30 smithi046 ceph-mon[32498]: pgmap v146: 129 pgs: 129 active+clean; 71 MiB data, 999 MiB used, 1.0 TiB / 1.0 TiB avail; 553 KiB/s rd, 1.8 MiB/s wr, 172 op/s 2024-03-31T23:16:31.198 INFO:tasks.workunit.client.0.smithi046.stdout: 1 99138 2.51 MB/sec execute 97 sec latency 15.692 ms 2024-03-31T23:16:32.197 INFO:tasks.workunit.client.0.smithi046.stdout: 1 99596 2.53 MB/sec execute 98 sec latency 15.613 ms 2024-03-31T23:16:32.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:16:32 smithi142 ceph-mon[35271]: pgmap v147: 129 pgs: 129 active+clean; 71 MiB data, 999 MiB used, 1.0 TiB / 1.0 TiB avail; 379 KiB/s rd, 1.1 MiB/s wr, 116 op/s 2024-03-31T23:16:33.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:16:32 smithi162 ceph-mon[36237]: pgmap v147: 129 pgs: 129 active+clean; 71 MiB data, 999 MiB used, 1.0 TiB / 1.0 TiB avail; 379 KiB/s rd, 1.1 MiB/s wr, 116 op/s 2024-03-31T23:16:33.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:16:32 smithi046 ceph-mon[32498]: pgmap v147: 129 pgs: 129 active+clean; 71 MiB data, 999 MiB used, 1.0 TiB / 1.0 TiB avail; 379 KiB/s rd, 1.1 MiB/s wr, 116 op/s 2024-03-31T23:16:33.197 INFO:tasks.workunit.client.0.smithi046.stdout: 1 99832 2.50 MB/sec execute 99 sec latency 15.735 ms 2024-03-31T23:16:34.197 INFO:tasks.workunit.client.0.smithi046.stdout: 1 100228 2.48 MB/sec execute 100 sec latency 16.024 ms 2024-03-31T23:16:34.944 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:16:34 smithi142 ceph-mon[35271]: pgmap v148: 129 pgs: 129 active+clean; 71 MiB data, 999 MiB used, 1.0 TiB / 1.0 TiB avail; 379 KiB/s rd, 1.1 MiB/s wr, 116 op/s 2024-03-31T23:16:35.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:16:34 smithi162 ceph-mon[36237]: pgmap v148: 129 pgs: 129 active+clean; 71 MiB data, 999 MiB used, 1.0 TiB / 1.0 TiB avail; 379 KiB/s rd, 1.1 MiB/s wr, 116 op/s 2024-03-31T23:16:35.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:16:34 smithi046 ceph-mon[32498]: pgmap v148: 129 pgs: 129 active+clean; 71 MiB data, 999 MiB used, 1.0 TiB / 1.0 TiB avail; 379 KiB/s rd, 1.1 MiB/s wr, 116 op/s 2024-03-31T23:16:35.197 INFO:tasks.workunit.client.0.smithi046.stdout: 1 100744 2.50 MB/sec execute 101 sec latency 16.099 ms 2024-03-31T23:16:36.197 INFO:tasks.workunit.client.0.smithi046.stdout: 1 101193 2.48 MB/sec execute 102 sec latency 30.777 ms 2024-03-31T23:16:36.220 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:36.218+0000 7fb2de46f640 1 -- 172.21.15.46:0/157152175 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb2d8068dd0 msgr2=0x7fb2d8069230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:16:36.220 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:36.218+0000 7fb2de46f640 1 --2- 172.21.15.46:0/157152175 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb2d8068dd0 0x7fb2d8069230 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7fb2cc009930 tx=0x7fb2cc02bc30 comp rx=0 tx=0).stop 2024-03-31T23:16:36.220 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:36.219+0000 7fb2de46f640 1 -- 172.21.15.46:0/157152175 shutdown_connections 2024-03-31T23:16:36.221 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:36.219+0000 7fb2de46f640 1 --2- 172.21.15.46:0/157152175 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb2d8069770 0x7fb2d810e5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:16:36.221 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:36.219+0000 7fb2de46f640 1 --2- 172.21.15.46:0/157152175 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb2d8068dd0 0x7fb2d8069230 unknown :-1 s=CLOSED pgs=697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:16:36.221 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:36.219+0000 7fb2de46f640 1 --2- 172.21.15.46:0/157152175 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb2d8107820 0x7fb2d8107c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:16:36.221 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:36.219+0000 7fb2de46f640 1 -- 172.21.15.46:0/157152175 >> 172.21.15.46:0/157152175 conn(0x7fb2d80774a0 msgr2=0x7fb2d80778b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:16:36.221 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:36.219+0000 7fb2de46f640 1 -- 172.21.15.46:0/157152175 shutdown_connections 2024-03-31T23:16:36.221 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:36.219+0000 7fb2de46f640 1 -- 172.21.15.46:0/157152175 wait complete. 2024-03-31T23:16:36.221 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:36.219+0000 7fb2de46f640 1 Processor -- start 2024-03-31T23:16:36.221 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:36.219+0000 7fb2de46f640 1 -- start start 2024-03-31T23:16:36.223 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:36.220+0000 7fb2de46f640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb2d8068dd0 0x7fb2d8197ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:16:36.223 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:36.220+0000 7fb2de46f640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb2d8069770 0x7fb2d81983e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:16:36.223 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:36.220+0000 7fb2de46f640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb2d8107820 0x7fb2d819c710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:16:36.223 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:36.220+0000 7fb2de46f640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fb2d81119c0 con 0x7fb2d8068dd0 2024-03-31T23:16:36.224 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:36.220+0000 7fb2de46f640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fb2d8111840 con 0x7fb2d8107820 2024-03-31T23:16:36.224 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:36.220+0000 7fb2de46f640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fb2d8111b40 con 0x7fb2d8069770 2024-03-31T23:16:36.224 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:36.220+0000 7fb2d7fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb2d8068dd0 0x7fb2d8197ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:16:36.224 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:36.220+0000 7fb2d7fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb2d8068dd0 0x7fb2d8197ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:57074/0 (socket says 172.21.15.46:57074) 2024-03-31T23:16:36.224 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:36.220+0000 7fb2d7fff640 1 -- 172.21.15.46:0/433984285 learned_addr learned my addr 172.21.15.46:0/433984285 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:16:36.224 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:36.220+0000 7fb2dca0c640 1 --2- 172.21.15.46:0/433984285 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb2d8107820 0x7fb2d819c710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:16:36.224 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:36.220+0000 7fb2d77fe640 1 --2- 172.21.15.46:0/433984285 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb2d8069770 0x7fb2d81983e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:16:36.224 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:36.220+0000 7fb2d7fff640 1 -- 172.21.15.46:0/433984285 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb2d8069770 msgr2=0x7fb2d81983e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:16:36.224 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:36.220+0000 7fb2d7fff640 1 --2- 172.21.15.46:0/433984285 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb2d8069770 0x7fb2d81983e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:16:36.224 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:36.220+0000 7fb2d7fff640 1 -- 172.21.15.46:0/433984285 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb2d8107820 msgr2=0x7fb2d819c710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:16:36.224 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:36.220+0000 7fb2d7fff640 1 --2- 172.21.15.46:0/433984285 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb2d8107820 0x7fb2d819c710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:16:36.224 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:36.220+0000 7fb2d7fff640 1 -- 172.21.15.46:0/433984285 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb2d819ce40 con 0x7fb2d8068dd0 2024-03-31T23:16:36.225 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:36.220+0000 7fb2d7fff640 1 --2- 172.21.15.46:0/433984285 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb2d8068dd0 0x7fb2d8197ea0 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7fb2c800e9f0 tx=0x7fb2c800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:16:36.225 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:36.221+0000 7fb2d57fa640 1 -- 172.21.15.46:0/433984285 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2c800cd80 con 0x7fb2d8068dd0 2024-03-31T23:16:36.225 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:36.221+0000 7fb2de46f640 1 -- 172.21.15.46:0/433984285 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb2d819d130 con 0x7fb2d8068dd0 2024-03-31T23:16:36.225 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:36.221+0000 7fb2d57fa640 1 -- 172.21.15.46:0/433984285 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb2c80054d0 con 0x7fb2d8068dd0 2024-03-31T23:16:36.225 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:36.221+0000 7fb2d57fa640 1 -- 172.21.15.46:0/433984285 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2c8010490 con 0x7fb2d8068dd0 2024-03-31T23:16:36.225 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:36.221+0000 7fb2de46f640 1 -- 172.21.15.46:0/433984285 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb2d81a49c0 con 0x7fb2d8068dd0 2024-03-31T23:16:36.231 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:36.223+0000 7fb2e1054740 1 Processor -- start 2024-03-31T23:16:36.232 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:36.223+0000 7fb2e1054740 1 -- start start 2024-03-31T23:16:36.232 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:36.223+0000 7fb2d57fa640 1 -- 172.21.15.46:0/433984285 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 25) ==== 137451+0+0 (secure 0 0 0) 0x7fb2c8005020 con 0x7fb2d8068dd0 2024-03-31T23:16:36.232 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:36.223+0000 7fb2e1054740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x563a3094e080 0x563a3094e460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:16:36.232 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:36.223+0000 7fb2e1054740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563a3094e9a0 0x563a3095bc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:16:36.232 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:36.223+0000 7fb2e1054740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563a3095c1c0 0x563a3095e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:16:36.232 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:36.223+0000 7fb2e1054740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x563a308b18f0 con 0x563a3094e9a0 2024-03-31T23:16:36.232 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:36.223+0000 7fb2e1054740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x563a3085a8d0 con 0x563a3094e080 2024-03-31T23:16:36.232 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:36.223+0000 7fb2e1054740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x563a308ad990 con 0x563a3095c1c0 2024-03-31T23:16:36.232 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:36.223+0000 7fb2d7fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563a3094e9a0 0x563a3095bc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:16:36.232 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:36.224+0000 7fb2d77fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563a3095c1c0 0x563a3095e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:16:36.232 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:36.224+0000 7fb2d7fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563a3094e9a0 0x563a3095bc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:57084/0 (socket says 172.21.15.46:57084) 2024-03-31T23:16:36.232 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:36.224+0000 7fb2d7fff640 1 -- 172.21.15.46:0/572342764 learned_addr learned my addr 172.21.15.46:0/572342764 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:16:36.233 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:36.224+0000 7fb2dca0c640 1 --2- 172.21.15.46:0/572342764 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x563a3094e080 0x563a3094e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:16:36.233 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:36.224+0000 7fb2d77fe640 1 -- 172.21.15.46:0/572342764 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x563a3094e080 msgr2=0x563a3094e460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:16:36.233 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:36.224+0000 7fb2d77fe640 1 --2- 172.21.15.46:0/572342764 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x563a3094e080 0x563a3094e460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:16:36.233 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:36.224+0000 7fb2d77fe640 1 -- 172.21.15.46:0/572342764 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563a3094e9a0 msgr2=0x563a3095bc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:16:36.233 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:36.224+0000 7fb2d77fe640 1 --2- 172.21.15.46:0/572342764 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563a3094e9a0 0x563a3095bc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:16:36.233 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:36.224+0000 7fb2d77fe640 1 -- 172.21.15.46:0/572342764 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563a305c6210 con 0x563a3095c1c0 2024-03-31T23:16:36.233 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:36.224+0000 7fb2d57fa640 1 --2- 172.21.15.46:0/433984285 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fb2c00a26b0 0x7fb2c00a4b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:16:36.233 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:36.224+0000 7fb2d77fe640 1 --2- 172.21.15.46:0/572342764 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563a3095c1c0 0x563a3095e5b0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fb2cc009230 tx=0x7fb2cc009560 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cc0ca574f456149d server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:16:36.233 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:36.224+0000 7fb2dca0c640 1 --2- 172.21.15.46:0/433984285 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fb2c00a26b0 0x7fb2c00a4b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:16:36.233 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:36.224+0000 7fb2d57fa640 1 -- 172.21.15.46:0/433984285 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7fb2c80cf240 con 0x7fb2d8068dd0 2024-03-31T23:16:36.233 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:36.224+0000 7fb2be7fc640 1 -- 172.21.15.46:0/572342764 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2cc02c780 con 0x563a3095c1c0 2024-03-31T23:16:36.234 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:36.224+0000 7fb2be7fc640 1 -- 172.21.15.46:0/572342764 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb2cc02c920 con 0x563a3095c1c0 2024-03-31T23:16:36.234 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:36.224+0000 7fb2be7fc640 1 -- 172.21.15.46:0/572342764 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2cc0046c0 con 0x563a3095c1c0 2024-03-31T23:16:36.234 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:36.225+0000 7fb2e1054740 1 -- 172.21.15.46:0/572342764 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563a3095c1c0 msgr2=0x563a3095e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:16:36.234 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:36.225+0000 7fb2e1054740 1 --2- 172.21.15.46:0/572342764 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563a3095c1c0 0x563a3095e5b0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fb2cc009230 tx=0x7fb2cc009560 comp rx=0 tx=0).stop 2024-03-31T23:16:36.234 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:36.225+0000 7fb2dca0c640 1 --2- 172.21.15.46:0/433984285 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fb2c00a26b0 0x7fb2c00a4b70 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fb2d0005fd0 tx=0x7fb2d0009500 comp rx=0 tx=0).ready entity=mgr.15654 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:16:36.234 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:36.225+0000 7fb2e1054740 1 -- 172.21.15.46:0/572342764 shutdown_connections 2024-03-31T23:16:36.234 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:36.225+0000 7fb2e1054740 1 --2- 172.21.15.46:0/572342764 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563a3095c1c0 0x563a3095e5b0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:16:36.234 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:36.225+0000 7fb2e1054740 1 --2- 172.21.15.46:0/572342764 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563a3094e9a0 0x563a3095bc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:16:36.234 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:36.225+0000 7fb2e1054740 1 --2- 172.21.15.46:0/572342764 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x563a3094e080 0x563a3094e460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:16:36.234 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:36.225+0000 7fb2e1054740 1 -- 172.21.15.46:0/572342764 >> 172.21.15.46:0/572342764 conn(0x563a30948fe0 msgr2=0x563a309493c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:16:36.234 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:36.225+0000 7fb2e1054740 1 -- 172.21.15.46:0/572342764 shutdown_connections 2024-03-31T23:16:36.234 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:36.225+0000 7fb2e1054740 1 -- 172.21.15.46:0/572342764 wait complete. 2024-03-31T23:16:36.234 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:36.225+0000 7fb2e1054740 1 -- start start 2024-03-31T23:16:36.234 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:36.226+0000 7fb2e1054740 1 Processor -- start 2024-03-31T23:16:36.235 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:36.226+0000 7fb2e1054740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563a3094e080 0x563a30959220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:16:36.235 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:36.226+0000 7fb2e1054740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563a3094e9a0 0x563a30959760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:16:36.235 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:36.226+0000 7fb2e1054740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x563a3095c1c0 0x563a30959ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:16:36.235 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:36.226+0000 7fb2e1054740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x563a3095ed90 con 0x563a3094e080 2024-03-31T23:16:36.235 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:36.226+0000 7fb2e1054740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x563a3084ff80 con 0x563a3095c1c0 2024-03-31T23:16:36.235 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:36.226+0000 7fb2e1054740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x563a3095ef10 con 0x563a3094e9a0 2024-03-31T23:16:36.235 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:36.226+0000 7fb2e1054740 10 client.0 _ll_register_callbacks cb 0x563a308a8290 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T23:16:36.235 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:36.226+0000 7fb2d7fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563a3094e080 0x563a30959220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:16:36.235 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:36.226+0000 7fb2d7fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563a3094e080 0x563a30959220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:57088/0 (socket says 172.21.15.46:57088) 2024-03-31T23:16:36.235 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:36.226+0000 7fb2d7fff640 1 -- 172.21.15.46:0/1495417209 learned_addr learned my addr 172.21.15.46:0/1495417209 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:16:36.235 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:36.226+0000 7fb2d77fe640 1 --2- 172.21.15.46:0/1495417209 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563a3094e9a0 0x563a30959760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:16:36.235 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:36.226+0000 7fb2dca0c640 1 --2- 172.21.15.46:0/1495417209 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x563a3095c1c0 0x563a30959ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:16:36.235 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:36.226+0000 7fb2d7fff640 1 -- 172.21.15.46:0/1495417209 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563a3094e9a0 msgr2=0x563a30959760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:16:36.235 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:36.226+0000 7fb2d7fff640 1 --2- 172.21.15.46:0/1495417209 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563a3094e9a0 0x563a30959760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:16:36.236 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:36.226+0000 7fb2d7fff640 1 -- 172.21.15.46:0/1495417209 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x563a3095c1c0 msgr2=0x563a30959ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:16:36.236 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:36.226+0000 7fb2d7fff640 1 --2- 172.21.15.46:0/1495417209 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x563a3095c1c0 0x563a30959ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:16:36.236 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:36.226+0000 7fb2d7fff640 1 -- 172.21.15.46:0/1495417209 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563a30692620 con 0x563a3094e080 2024-03-31T23:16:36.236 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:36.226+0000 7fb2d7fff640 1 --2- 172.21.15.46:0/1495417209 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563a3094e080 0x563a30959220 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7fb2c800e9c0 tx=0x7fb2c8004aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=609fe8ccc25b9bf8 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:16:36.236 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:36.226+0000 7fb2b77fe640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-31T23:16:36.236 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:36.227+0000 7fb2b77fe640 1 -- 172.21.15.46:0/1495417209 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2c809b3e0 con 0x563a3094e080 2024-03-31T23:16:36.236 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:36.227+0000 7fb2b77fe640 1 -- 172.21.15.46:0/1495417209 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7fb2c80a3b90 con 0x563a3094e080 2024-03-31T23:16:36.236 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:36.227+0000 7fb2e1054740 1 -- 172.21.15.46:0/1495417209 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb2cc02c920 con 0x563a3094e080 2024-03-31T23:16:36.236 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:36.227+0000 7fb2b77fe640 1 -- 172.21.15.46:0/1495417209 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fb2c809af70 con 0x563a3094e080 2024-03-31T23:16:36.236 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:36.227+0000 7fb2e1054740 10 client.16545 fetch_fsmap learned FSMap version 30 2024-03-31T23:16:36.236 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:36.227+0000 7fb2e1054740 1 -- 172.21.15.46:0/1495417209 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x563a3096ac90 con 0x563a3094e080 2024-03-31T23:16:36.236 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:36.227+0000 7fb2b77fe640 1 -- 172.21.15.46:0/1495417209 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb2cc02c920 con 0x563a3094e080 2024-03-31T23:16:36.236 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:36.227+0000 7fb2b77fe640 1 -- 172.21.15.46:0/1495417209 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 30) ==== 8346+0+0 (secure 0 0 0) 0x7fb2c80b4230 con 0x563a3094e080 2024-03-31T23:16:36.236 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:36.227+0000 7fb2e1054740 10 client.16545 fetch_fsmap finished waiting for FSMap version 30 2024-03-31T23:16:36.237 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:36.227+0000 7fb2e1054740 10 client.16545 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-31T23:16:36.237 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:36.227+0000 7fb2e1054740 20 client.16545 populate_metadata read hostname 'smithi046' 2024-03-31T23:16:36.237 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:36.227+0000 7fb2e1054740 1 --2- 172.21.15.46:0/1495417209 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x563a3096c150 0x563a3096e540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:16:36.237 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:36.227+0000 7fb2e1054740 4 client.16545 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-31T23:16:36.237 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:36.227+0000 7fb2e1054740 1 -- 172.21.15.46:0/1495417209 --> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563a308c5120 con 0x563a3096c150 2024-03-31T23:16:36.237 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:36.227+0000 7fb2d77fe640 1 --2- 172.21.15.46:0/1495417209 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x563a3096c150 0x563a3096e540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:16:36.237 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:36.228+0000 7fb2d77fe640 1 --2- 172.21.15.46:0/1495417209 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x563a3096c150 0x563a3096e540 crc :-1 s=READY pgs=117 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d454dcd84d8e0a5f server_cookie=e4c2935931a8df83 in_seq=0 out_seq=0 2024-03-31T23:16:36.237 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:36.228+0000 7fb2b77fe640 10 client.16545 ms_handle_connect on v2:172.21.15.142:6838/2519143918 2024-03-31T23:16:36.237 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:36.229+0000 7fb2b77fe640 1 -- 172.21.15.46:0/1495417209 <== mds.0 v2:172.21.15.142:6838/2519143918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x563a308c5120 con 0x563a3096c150 2024-03-31T23:16:36.237 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:36.229+0000 7fb2b77fe640 10 client.16545 handle_command_reply: tid=0 2024-03-31T23:16:36.237 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:36.229+0000 7fb2e1054740 1 client.16545 shutdown 2024-03-31T23:16:36.237 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:36.229+0000 7fb2e1054740 1 -- 172.21.15.46:0/1495417209 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563a3094e080 msgr2=0x563a30959220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:16:36.237 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:36.229+0000 7fb2e1054740 1 --2- 172.21.15.46:0/1495417209 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563a3094e080 0x563a30959220 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7fb2c800e9c0 tx=0x7fb2c8004aa0 comp rx=0 tx=0).stop 2024-03-31T23:16:36.237 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:36.229+0000 7fb2e1054740 1 -- 172.21.15.46:0/1495417209 shutdown_connections 2024-03-31T23:16:36.238 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:36.229+0000 7fb2e1054740 1 --2- 172.21.15.46:0/1495417209 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x563a3096c150 0x563a3096e540 crc :-1 s=READY pgs=117 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:16:36.238 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:36.229+0000 7fb2e1054740 1 --2- 172.21.15.46:0/1495417209 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x563a3095c1c0 0x563a30959ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:16:36.238 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:36.229+0000 7fb2e1054740 1 --2- 172.21.15.46:0/1495417209 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563a3094e9a0 0x563a30959760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:16:36.238 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:36.229+0000 7fb2e1054740 1 --2- 172.21.15.46:0/1495417209 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563a3094e080 0x563a30959220 unknown :-1 s=CLOSED pgs=699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:16:36.238 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:36.229+0000 7fb2e1054740 1 -- 172.21.15.46:0/1495417209 >> 172.21.15.46:0/1495417209 conn(0x563a30948fe0 msgr2=0x563a3094ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:16:36.238 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:36.229+0000 7fb2b77fe640 0 client.16545 ms_handle_reset on v2:172.21.15.142:6838/2519143918 2024-03-31T23:16:36.238 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:36.229+0000 7fb2e1054740 1 -- 172.21.15.46:0/1495417209 shutdown_connections 2024-03-31T23:16:36.238 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:36.229+0000 7fb2e1054740 1 -- 172.21.15.46:0/1495417209 wait complete. 2024-03-31T23:16:36.238 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:36.230+0000 7fb2e1054740 20 client.16545 trim_cache size 0 max 16384 2024-03-31T23:16:36.252 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:36.246+0000 7fb2e1054740 1 Processor -- start 2024-03-31T23:16:36.252 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:36.246+0000 7fb2e1054740 1 -- start start 2024-03-31T23:16:36.253 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:36.246+0000 7fb2e1054740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563a3094e080 0x563a3095b950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:16:36.253 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:36.246+0000 7fb2e1054740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563a3094e9a0 0x563a30967b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:16:36.253 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:36.246+0000 7fb2e1054740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x563a3095c1c0 0x563a30973ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:16:36.253 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:36.246+0000 7fb2e1054740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fb2d81af110 con 0x563a3094e080 2024-03-31T23:16:36.253 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:36.246+0000 7fb2e1054740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fb2d81ac9e0 con 0x563a3095c1c0 2024-03-31T23:16:36.253 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:36.246+0000 7fb2e1054740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fb2d81aa830 con 0x563a3094e9a0 2024-03-31T23:16:36.253 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:36.247+0000 7fb2d7fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563a3094e080 0x563a3095b950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:16:36.253 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:36.247+0000 7fb2d7fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563a3094e080 0x563a3095b950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:57100/0 (socket says 172.21.15.46:57100) 2024-03-31T23:16:36.253 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:36.247+0000 7fb2d7fff640 1 -- 172.21.15.46:0/1622695288 learned_addr learned my addr 172.21.15.46:0/1622695288 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:16:36.253 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:36.247+0000 7fb2d77fe640 1 --2- 172.21.15.46:0/1622695288 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563a3094e9a0 0x563a30967b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:16:36.253 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:36.247+0000 7fb2dca0c640 1 --2- 172.21.15.46:0/1622695288 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x563a3095c1c0 0x563a30973ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:16:36.253 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:36.247+0000 7fb2d7fff640 1 -- 172.21.15.46:0/1622695288 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563a3094e9a0 msgr2=0x563a30967b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:16:36.253 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:36.247+0000 7fb2d7fff640 1 --2- 172.21.15.46:0/1622695288 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563a3094e9a0 0x563a30967b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:16:36.254 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:36.247+0000 7fb2d7fff640 1 -- 172.21.15.46:0/1622695288 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x563a3095c1c0 msgr2=0x563a30973ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:16:36.254 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:36.247+0000 7fb2d7fff640 1 --2- 172.21.15.46:0/1622695288 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x563a3095c1c0 0x563a30973ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:16:36.254 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:36.247+0000 7fb2d7fff640 1 -- 172.21.15.46:0/1622695288 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563a30968070 con 0x563a3094e080 2024-03-31T23:16:36.254 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:36.247+0000 7fb2d77fe640 1 --2- 172.21.15.46:0/1622695288 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563a3094e9a0 0x563a30967b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:16:36.254 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:36.247+0000 7fb2d7fff640 1 --2- 172.21.15.46:0/1622695288 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563a3094e080 0x563a3095b950 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7fb2c8097fe0 tx=0x7fb2c80cf170 comp rx=0 tx=0).ready entity=mon.0 client_cookie=97304f46ec8aa626 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:16:36.254 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:36.247+0000 7fb2bdffb640 1 -- 172.21.15.46:0/1622695288 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2c80b9070 con 0x563a3094e080 2024-03-31T23:16:36.254 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:36.247+0000 7fb2bdffb640 1 -- 172.21.15.46:0/1622695288 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb2c8097040 con 0x563a3094e080 2024-03-31T23:16:36.254 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:36.247+0000 7fb2bdffb640 1 -- 172.21.15.46:0/1622695288 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2c809b150 con 0x563a3094e080 2024-03-31T23:16:36.254 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:36.248+0000 7fb2e1054740 1 -- 172.21.15.46:0/1622695288 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563a3094e080 msgr2=0x563a3095b950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:16:36.254 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:36.248+0000 7fb2e1054740 1 --2- 172.21.15.46:0/1622695288 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563a3094e080 0x563a3095b950 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7fb2c8097fe0 tx=0x7fb2c80cf170 comp rx=0 tx=0).stop 2024-03-31T23:16:36.254 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:36.248+0000 7fb2e1054740 1 -- 172.21.15.46:0/1622695288 shutdown_connections 2024-03-31T23:16:36.254 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:36.248+0000 7fb2e1054740 1 --2- 172.21.15.46:0/1622695288 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x563a3095c1c0 0x563a30973ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:16:36.255 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:36.248+0000 7fb2e1054740 1 --2- 172.21.15.46:0/1622695288 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563a3094e9a0 0x563a30967b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:16:36.255 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:36.248+0000 7fb2e1054740 1 --2- 172.21.15.46:0/1622695288 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563a3094e080 0x563a3095b950 unknown :-1 s=CLOSED pgs=700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:16:36.255 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:36.248+0000 7fb2e1054740 1 -- 172.21.15.46:0/1622695288 >> 172.21.15.46:0/1622695288 conn(0x563a30948fe0 msgr2=0x563a30954540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:16:36.255 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:36.248+0000 7fb2e1054740 1 -- 172.21.15.46:0/1622695288 shutdown_connections 2024-03-31T23:16:36.255 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:36.248+0000 7fb2e1054740 1 -- 172.21.15.46:0/1622695288 wait complete. 2024-03-31T23:16:36.255 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:36.248+0000 7fb2e1054740 1 -- start start 2024-03-31T23:16:36.255 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:36.249+0000 7fb2e1054740 1 Processor -- start 2024-03-31T23:16:36.255 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:36.249+0000 7fb2e1054740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563a3094e080 0x563a30955e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:16:36.255 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:36.249+0000 7fb2e1054740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x563a3094e9a0 0x563a30956380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:16:36.255 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:36.249+0000 7fb2e1054740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563a3095c1c0 0x563a309568c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:16:36.255 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:36.249+0000 7fb2e1054740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fb2d81aaea0 con 0x563a3094e080 2024-03-31T23:16:36.256 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:36.249+0000 7fb2e1054740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fb2d81ac310 con 0x563a3094e9a0 2024-03-31T23:16:36.256 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:36.249+0000 7fb2e1054740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fb2d81ab570 con 0x563a3095c1c0 2024-03-31T23:16:36.256 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:36.249+0000 7fb2e1054740 10 client.0 _ll_register_callbacks cb 0x7fb2c80d2a80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T23:16:36.256 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:36.249+0000 7fb2d7fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563a3094e080 0x563a30955e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:16:36.256 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:36.249+0000 7fb2d7fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563a3094e080 0x563a30955e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:57110/0 (socket says 172.21.15.46:57110) 2024-03-31T23:16:36.256 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:36.249+0000 7fb2d7fff640 1 -- 172.21.15.46:0/3536446584 learned_addr learned my addr 172.21.15.46:0/3536446584 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:16:36.256 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:36.249+0000 7fb2dca0c640 1 --2- 172.21.15.46:0/3536446584 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563a3095c1c0 0x563a309568c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:16:36.256 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:36.249+0000 7fb2d77fe640 1 --2- 172.21.15.46:0/3536446584 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x563a3094e9a0 0x563a30956380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:16:36.256 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:36.249+0000 7fb2d7fff640 1 -- 172.21.15.46:0/3536446584 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563a3095c1c0 msgr2=0x563a309568c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:16:36.256 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:36.249+0000 7fb2d7fff640 1 --2- 172.21.15.46:0/3536446584 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563a3095c1c0 0x563a309568c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:16:36.256 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:36.249+0000 7fb2d7fff640 1 -- 172.21.15.46:0/3536446584 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x563a3094e9a0 msgr2=0x563a30956380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:16:36.257 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:36.249+0000 7fb2d7fff640 1 --2- 172.21.15.46:0/3536446584 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x563a3094e9a0 0x563a30956380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:16:36.257 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:36.249+0000 7fb2d7fff640 1 -- 172.21.15.46:0/3536446584 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563a309d14d0 con 0x563a3094e080 2024-03-31T23:16:36.257 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:36.249+0000 7fb2dca0c640 1 --2- 172.21.15.46:0/3536446584 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563a3095c1c0 0x563a309568c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:16:36.257 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:36.249+0000 7fb2d77fe640 1 --2- 172.21.15.46:0/3536446584 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x563a3094e9a0 0x563a30956380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:16:36.257 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:36.250+0000 7fb2d7fff640 1 --2- 172.21.15.46:0/3536446584 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563a3094e080 0x563a30955e40 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7fb2c80ccbe0 tx=0x7fb2c80d2d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=641d0d37ce2ae77a server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:16:36.257 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:36.250+0000 7fb2b77fe640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-31T23:16:36.257 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:36.250+0000 7fb2b77fe640 1 -- 172.21.15.46:0/3536446584 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2c80b9030 con 0x563a3094e080 2024-03-31T23:16:36.257 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:36.250+0000 7fb2b77fe640 1 -- 172.21.15.46:0/3536446584 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7fb2c80a3d90 con 0x563a3094e080 2024-03-31T23:16:36.257 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:36.250+0000 7fb2e1054740 1 -- 172.21.15.46:0/3536446584 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb2c8097040 con 0x563a3094e080 2024-03-31T23:16:36.257 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:36.250+0000 7fb2b77fe640 1 -- 172.21.15.46:0/3536446584 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fb2c809b150 con 0x563a3094e080 2024-03-31T23:16:36.257 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:36.250+0000 7fb2e1054740 10 client.16551 fetch_fsmap learned FSMap version 30 2024-03-31T23:16:36.257 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:36.250+0000 7fb2e1054740 1 -- 172.21.15.46:0/3536446584 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x563a309d27c0 con 0x563a3094e080 2024-03-31T23:16:36.258 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:36.250+0000 7fb2b77fe640 1 -- 172.21.15.46:0/3536446584 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb2c8097040 con 0x563a3094e080 2024-03-31T23:16:36.258 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:36.250+0000 7fb2b77fe640 1 -- 172.21.15.46:0/3536446584 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 30) ==== 8346+0+0 (secure 0 0 0) 0x7fb2c809aa60 con 0x563a3094e080 2024-03-31T23:16:36.258 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:36.250+0000 7fb2e1054740 10 client.16551 fetch_fsmap finished waiting for FSMap version 30 2024-03-31T23:16:36.258 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:36.250+0000 7fb2e1054740 10 client.16551 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-31T23:16:36.258 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:36.250+0000 7fb2e1054740 20 client.16551 populate_metadata read hostname 'smithi046' 2024-03-31T23:16:36.258 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:36.250+0000 7fb2e1054740 1 --2- 172.21.15.46:0/3536446584 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x563a309d2d80 0x563a309d51a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:16:36.258 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:36.250+0000 7fb2e1054740 4 client.16551 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-31T23:16:36.258 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:36.250+0000 7fb2e1054740 1 -- 172.21.15.46:0/3536446584 --> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] -- command(tid 0: {"prefix": "flush journal"}) -- 0x563a308bee30 con 0x563a309d2d80 2024-03-31T23:16:36.258 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:36.251+0000 7fb2d77fe640 1 --2- 172.21.15.46:0/3536446584 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x563a309d2d80 0x563a309d51a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:16:36.258 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:36.251+0000 7fb2b77fe640 10 client.16551 ms_handle_connect on v2:172.21.15.142:6838/2519143918 2024-03-31T23:16:36.258 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:36.251+0000 7fb2d77fe640 1 --2- 172.21.15.46:0/3536446584 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x563a309d2d80 0x563a309d51a0 crc :-1 s=READY pgs=118 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e77d9fc13313b060 server_cookie=d95a1ab3efb94378 in_seq=0 out_seq=0 2024-03-31T23:16:36.273 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:36.271+0000 7fb2b77fe640 1 -- 172.21.15.46:0/3536446584 <== mds.0 v2:172.21.15.142:6838/2519143918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x563a308bee30 con 0x563a309d2d80 2024-03-31T23:16:36.273 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:36.271+0000 7fb2b77fe640 10 client.16551 handle_command_reply: tid=0 2024-03-31T23:16:36.273 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:36.271+0000 7fb2e1054740 1 client.16551 shutdown 2024-03-31T23:16:36.273 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:36.271+0000 7fb2e1054740 1 -- 172.21.15.46:0/3536446584 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563a3094e080 msgr2=0x563a30955e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:16:36.273 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:36.271+0000 7fb2e1054740 1 --2- 172.21.15.46:0/3536446584 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563a3094e080 0x563a30955e40 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7fb2c80ccbe0 tx=0x7fb2c80d2d10 comp rx=0 tx=0).stop 2024-03-31T23:16:36.273 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:36.271+0000 7fb2e1054740 1 -- 172.21.15.46:0/3536446584 shutdown_connections 2024-03-31T23:16:36.273 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:36.271+0000 7fb2e1054740 1 --2- 172.21.15.46:0/3536446584 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x563a309d2d80 0x563a309d51a0 crc :-1 s=READY pgs=118 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:16:36.274 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:36.271+0000 7fb2e1054740 1 --2- 172.21.15.46:0/3536446584 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563a3095c1c0 0x563a309568c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:16:36.274 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:36.271+0000 7fb2b77fe640 0 client.16551 ms_handle_reset on v2:172.21.15.142:6838/2519143918 2024-03-31T23:16:36.274 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:36.271+0000 7fb2e1054740 1 --2- 172.21.15.46:0/3536446584 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x563a3094e9a0 0x563a30956380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:16:36.274 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:36.271+0000 7fb2e1054740 1 --2- 172.21.15.46:0/3536446584 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563a3094e080 0x563a30955e40 unknown :-1 s=CLOSED pgs=701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:16:36.274 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:36.271+0000 7fb2e1054740 1 -- 172.21.15.46:0/3536446584 >> 172.21.15.46:0/3536446584 conn(0x563a30948fe0 msgr2=0x563a309591e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:16:36.274 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:36.271+0000 7fb2e1054740 1 -- 172.21.15.46:0/3536446584 shutdown_connections 2024-03-31T23:16:36.274 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:36.271+0000 7fb2e1054740 1 -- 172.21.15.46:0/3536446584 wait complete. 2024-03-31T23:16:36.274 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:36.271+0000 7fb2e1054740 20 client.16551 trim_cache size 0 max 16384 2024-03-31T23:16:36.275 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout:{ 2024-03-31T23:16:36.275 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout: "message": "", 2024-03-31T23:16:36.275 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout: "return_code": 0 2024-03-31T23:16:36.275 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout:} 2024-03-31T23:16:36.276 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:36.272+0000 7fb2de46f640 1 -- 172.21.15.46:0/433984285 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fb2c00a26b0 msgr2=0x7fb2c00a4b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:16:36.276 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:36.272+0000 7fb2de46f640 1 --2- 172.21.15.46:0/433984285 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fb2c00a26b0 0x7fb2c00a4b70 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fb2d0005fd0 tx=0x7fb2d0009500 comp rx=0 tx=0).stop 2024-03-31T23:16:36.277 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:36.272+0000 7fb2de46f640 1 -- 172.21.15.46:0/433984285 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb2d8068dd0 msgr2=0x7fb2d8197ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:16:36.277 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:36.272+0000 7fb2de46f640 1 --2- 172.21.15.46:0/433984285 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb2d8068dd0 0x7fb2d8197ea0 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7fb2c800e9f0 tx=0x7fb2c800edf0 comp rx=0 tx=0).stop 2024-03-31T23:16:36.277 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:36.272+0000 7fb2de46f640 1 -- 172.21.15.46:0/433984285 shutdown_connections 2024-03-31T23:16:36.277 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:36.272+0000 7fb2de46f640 1 --2- 172.21.15.46:0/433984285 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fb2c00a26b0 0x7fb2c00a4b70 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:16:36.277 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:36.272+0000 7fb2de46f640 1 --2- 172.21.15.46:0/433984285 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb2d8107820 0x7fb2d819c710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:16:36.277 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:36.272+0000 7fb2de46f640 1 --2- 172.21.15.46:0/433984285 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb2d8069770 0x7fb2d81983e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:16:36.277 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:36.272+0000 7fb2de46f640 1 --2- 172.21.15.46:0/433984285 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb2d8068dd0 0x7fb2d8197ea0 unknown :-1 s=CLOSED pgs=698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:16:36.277 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:36.272+0000 7fb2de46f640 1 -- 172.21.15.46:0/433984285 >> 172.21.15.46:0/433984285 conn(0x7fb2d80774a0 msgr2=0x7fb2d80fe990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:16:36.278 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:36.272+0000 7fb2de46f640 1 -- 172.21.15.46:0/433984285 shutdown_connections 2024-03-31T23:16:36.278 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:36.272+0000 7fb2de46f640 1 -- 172.21.15.46:0/433984285 wait complete. 2024-03-31T23:16:36.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:16:36 smithi142 ceph-mon[35271]: pgmap v149: 129 pgs: 129 active+clean; 76 MiB data, 1001 MiB used, 1.0 TiB / 1.0 TiB avail; 548 KiB/s rd, 1.8 MiB/s wr, 171 op/s 2024-03-31T23:16:37.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:16:36 smithi162 ceph-mon[36237]: pgmap v149: 129 pgs: 129 active+clean; 76 MiB data, 1001 MiB used, 1.0 TiB / 1.0 TiB avail; 548 KiB/s rd, 1.8 MiB/s wr, 171 op/s 2024-03-31T23:16:37.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:16:36 smithi046 ceph-mon[32498]: pgmap v149: 129 pgs: 129 active+clean; 76 MiB data, 1001 MiB used, 1.0 TiB / 1.0 TiB avail; 548 KiB/s rd, 1.8 MiB/s wr, 171 op/s 2024-03-31T23:16:37.197 INFO:tasks.workunit.client.0.smithi046.stdout: 1 101735 2.47 MB/sec execute 103 sec latency 21.370 ms 2024-03-31T23:16:38.197 INFO:tasks.workunit.client.0.smithi046.stdout: 1 102332 2.50 MB/sec execute 104 sec latency 15.355 ms 2024-03-31T23:16:38.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:16:38 smithi142 ceph-mon[35271]: pgmap v150: 129 pgs: 129 active+clean; 76 MiB data, 1001 MiB used, 1.0 TiB / 1.0 TiB avail; 388 KiB/s rd, 1.2 MiB/s wr, 117 op/s 2024-03-31T23:16:39.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:16:38 smithi162 ceph-mon[36237]: pgmap v150: 129 pgs: 129 active+clean; 76 MiB data, 1001 MiB used, 1.0 TiB / 1.0 TiB avail; 388 KiB/s rd, 1.2 MiB/s wr, 117 op/s 2024-03-31T23:16:39.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:16:38 smithi046 ceph-mon[32498]: pgmap v150: 129 pgs: 129 active+clean; 76 MiB data, 1001 MiB used, 1.0 TiB / 1.0 TiB avail; 388 KiB/s rd, 1.2 MiB/s wr, 117 op/s 2024-03-31T23:16:39.197 INFO:tasks.workunit.client.0.smithi046.stdout: 1 102748 2.50 MB/sec execute 105 sec latency 17.588 ms 2024-03-31T23:16:40.197 INFO:tasks.workunit.client.0.smithi046.stdout: 1 103203 2.52 MB/sec execute 106 sec latency 16.513 ms 2024-03-31T23:16:40.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:16:40 smithi142 ceph-mon[35271]: pgmap v151: 129 pgs: 129 active+clean; 74 MiB data, 1021 MiB used, 1.0 TiB / 1.0 TiB avail; 569 KiB/s rd, 1.8 MiB/s wr, 176 op/s 2024-03-31T23:16:41.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:16:40 smithi162 ceph-mon[36237]: pgmap v151: 129 pgs: 129 active+clean; 74 MiB data, 1021 MiB used, 1.0 TiB / 1.0 TiB avail; 569 KiB/s rd, 1.8 MiB/s wr, 176 op/s 2024-03-31T23:16:41.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:16:40 smithi046 ceph-mon[32498]: pgmap v151: 129 pgs: 129 active+clean; 74 MiB data, 1021 MiB used, 1.0 TiB / 1.0 TiB avail; 569 KiB/s rd, 1.8 MiB/s wr, 176 op/s 2024-03-31T23:16:41.197 INFO:tasks.workunit.client.0.smithi046.stdout: 1 103470 2.50 MB/sec execute 107 sec latency 15.883 ms 2024-03-31T23:16:42.197 INFO:tasks.workunit.client.0.smithi046.stdout: 1 103884 2.48 MB/sec execute 108 sec latency 18.485 ms 2024-03-31T23:16:42.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:16:42 smithi142 ceph-mon[35271]: pgmap v152: 129 pgs: 129 active+clean; 74 MiB data, 1021 MiB used, 1.0 TiB / 1.0 TiB avail; 350 KiB/s rd, 1.3 MiB/s wr, 114 op/s 2024-03-31T23:16:43.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:16:42 smithi162 ceph-mon[36237]: pgmap v152: 129 pgs: 129 active+clean; 74 MiB data, 1021 MiB used, 1.0 TiB / 1.0 TiB avail; 350 KiB/s rd, 1.3 MiB/s wr, 114 op/s 2024-03-31T23:16:43.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:16:42 smithi046 ceph-mon[32498]: pgmap v152: 129 pgs: 129 active+clean; 74 MiB data, 1021 MiB used, 1.0 TiB / 1.0 TiB avail; 350 KiB/s rd, 1.3 MiB/s wr, 114 op/s 2024-03-31T23:16:43.197 INFO:tasks.workunit.client.0.smithi046.stdout: 1 104409 2.50 MB/sec execute 109 sec latency 18.361 ms 2024-03-31T23:16:44.197 INFO:tasks.workunit.client.0.smithi046.stdout: 1 104941 2.48 MB/sec execute 110 sec latency 21.508 ms 2024-03-31T23:16:44.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:16:44 smithi142 ceph-mon[35271]: pgmap v153: 129 pgs: 129 active+clean; 74 MiB data, 1021 MiB used, 1.0 TiB / 1.0 TiB avail; 350 KiB/s rd, 1.3 MiB/s wr, 114 op/s 2024-03-31T23:16:45.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:16:44 smithi162 ceph-mon[36237]: pgmap v153: 129 pgs: 129 active+clean; 74 MiB data, 1021 MiB used, 1.0 TiB / 1.0 TiB avail; 350 KiB/s rd, 1.3 MiB/s wr, 114 op/s 2024-03-31T23:16:45.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:16:44 smithi046 ceph-mon[32498]: pgmap v153: 129 pgs: 129 active+clean; 74 MiB data, 1021 MiB used, 1.0 TiB / 1.0 TiB avail; 350 KiB/s rd, 1.3 MiB/s wr, 114 op/s 2024-03-31T23:16:45.197 INFO:tasks.workunit.client.0.smithi046.stdout: 1 105410 2.48 MB/sec execute 111 sec latency 18.541 ms 2024-03-31T23:16:46.197 INFO:tasks.workunit.client.0.smithi046.stdout: 1 106188 2.51 MB/sec execute 112 sec latency 16.680 ms 2024-03-31T23:16:46.950 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:16:46 smithi142 ceph-mon[35271]: pgmap v154: 129 pgs: 129 active+clean; 74 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 541 KiB/s rd, 1.8 MiB/s wr, 175 op/s 2024-03-31T23:16:47.128 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:16:46 smithi046 ceph-mon[32498]: pgmap v154: 129 pgs: 129 active+clean; 74 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 541 KiB/s rd, 1.8 MiB/s wr, 175 op/s 2024-03-31T23:16:47.129 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:16:46 smithi162 ceph-mon[36237]: pgmap v154: 129 pgs: 129 active+clean; 74 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 541 KiB/s rd, 1.8 MiB/s wr, 175 op/s 2024-03-31T23:16:47.198 INFO:tasks.workunit.client.0.smithi046.stdout: 1 106525 2.52 MB/sec execute 113 sec latency 19.032 ms 2024-03-31T23:16:48.197 INFO:tasks.workunit.client.0.smithi046.stdout: 1 106833 2.51 MB/sec execute 114 sec latency 16.342 ms 2024-03-31T23:16:48.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:16:48 smithi142 ceph-mon[35271]: pgmap v155: 129 pgs: 129 active+clean; 74 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 371 KiB/s rd, 1.1 MiB/s wr, 119 op/s 2024-03-31T23:16:49.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:16:48 smithi162 ceph-mon[36237]: pgmap v155: 129 pgs: 129 active+clean; 74 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 371 KiB/s rd, 1.1 MiB/s wr, 119 op/s 2024-03-31T23:16:49.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:16:48 smithi046 ceph-mon[32498]: pgmap v155: 129 pgs: 129 active+clean; 74 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 371 KiB/s rd, 1.1 MiB/s wr, 119 op/s 2024-03-31T23:16:49.197 INFO:tasks.workunit.client.0.smithi046.stdout: 1 107107 2.49 MB/sec execute 115 sec latency 19.676 ms 2024-03-31T23:16:49.370 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:49.366+0000 7f7efbf7a640 1 -- 172.21.15.46:0/2253995252 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7ef41008a0 msgr2=0x7f7ef4100d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:16:49.370 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:49.366+0000 7f7efbf7a640 1 --2- 172.21.15.46:0/2253995252 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7ef41008a0 0x7f7ef4100d20 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f7ee4009900 tx=0x7f7ee402bc30 comp rx=0 tx=0).stop 2024-03-31T23:16:49.370 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:49.366+0000 7f7efbf7a640 1 -- 172.21.15.46:0/2253995252 shutdown_connections 2024-03-31T23:16:49.371 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:49.366+0000 7f7efbf7a640 1 --2- 172.21.15.46:0/2253995252 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7ef4101260 0x7f7ef410d480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:16:49.371 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:49.366+0000 7f7efbf7a640 1 --2- 172.21.15.46:0/2253995252 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7ef41008a0 0x7f7ef4100d20 unknown :-1 s=CLOSED pgs=702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:16:49.371 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:49.366+0000 7f7efbf7a640 1 --2- 172.21.15.46:0/2253995252 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7ef41055f0 0x7f7ef41059d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:16:49.371 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:49.366+0000 7f7efbf7a640 1 -- 172.21.15.46:0/2253995252 >> 172.21.15.46:0/2253995252 conn(0x7f7ef40774a0 msgr2=0x7f7ef40778b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:16:49.371 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:49.366+0000 7f7efbf7a640 1 -- 172.21.15.46:0/2253995252 shutdown_connections 2024-03-31T23:16:49.371 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:49.366+0000 7f7efbf7a640 1 -- 172.21.15.46:0/2253995252 wait complete. 2024-03-31T23:16:49.371 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:49.366+0000 7f7efbf7a640 1 Processor -- start 2024-03-31T23:16:49.372 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:49.366+0000 7f7efbf7a640 1 -- start start 2024-03-31T23:16:49.372 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:49.366+0000 7f7efbf7a640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7ef41008a0 0x7f7ef4197f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:16:49.372 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:49.367+0000 7f7efbf7a640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7ef4101260 0x7f7ef41984d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:16:49.372 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:49.367+0000 7f7efbf7a640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7ef41055f0 0x7f7ef419c850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:16:49.372 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:49.367+0000 7f7efbf7a640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f7ef41109c0 con 0x7f7ef4101260 2024-03-31T23:16:49.372 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:49.367+0000 7f7efbf7a640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f7ef4110840 con 0x7f7ef41008a0 2024-03-31T23:16:49.372 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:49.367+0000 7f7efbf7a640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f7ef4110b40 con 0x7f7ef41055f0 2024-03-31T23:16:49.372 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:49.367+0000 7f7ef9515640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7ef4101260 0x7f7ef41984d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:16:49.372 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:49.367+0000 7f7ef9515640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7ef4101260 0x7f7ef41984d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:49922/0 (socket says 172.21.15.46:49922) 2024-03-31T23:16:49.372 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:49.367+0000 7f7ef9515640 1 -- 172.21.15.46:0/648275003 learned_addr learned my addr 172.21.15.46:0/648275003 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:16:49.373 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:49.367+0000 7f7ef9d16640 1 --2- 172.21.15.46:0/648275003 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7ef41008a0 0x7f7ef4197f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:16:49.373 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:49.367+0000 7f7efa517640 1 --2- 172.21.15.46:0/648275003 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7ef41055f0 0x7f7ef419c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:16:49.373 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:49.367+0000 7f7ef9515640 1 -- 172.21.15.46:0/648275003 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7ef41055f0 msgr2=0x7f7ef419c850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:16:49.373 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:49.367+0000 7f7ef9515640 1 --2- 172.21.15.46:0/648275003 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7ef41055f0 0x7f7ef419c850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:16:49.373 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:49.367+0000 7f7ef9515640 1 -- 172.21.15.46:0/648275003 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7ef41008a0 msgr2=0x7f7ef4197f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:16:49.374 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:49.367+0000 7f7ef9515640 1 --2- 172.21.15.46:0/648275003 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7ef41008a0 0x7f7ef4197f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:16:49.374 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:49.367+0000 7f7ef9515640 1 -- 172.21.15.46:0/648275003 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ef419cf80 con 0x7f7ef4101260 2024-03-31T23:16:49.374 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:49.367+0000 7f7ef9515640 1 --2- 172.21.15.46:0/648275003 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7ef4101260 0x7f7ef41984d0 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f7ee4006010 tx=0x7f7ee4005d90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:16:49.374 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:49.367+0000 7f7eeaffd640 1 -- 172.21.15.46:0/648275003 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ee4043020 con 0x7f7ef4101260 2024-03-31T23:16:49.375 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:49.367+0000 7f7efbf7a640 1 -- 172.21.15.46:0/648275003 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7ef419d210 con 0x7f7ef4101260 2024-03-31T23:16:49.375 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:49.367+0000 7f7eeaffd640 1 -- 172.21.15.46:0/648275003 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7ee4004350 con 0x7f7ef4101260 2024-03-31T23:16:49.375 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:49.367+0000 7f7eeaffd640 1 -- 172.21.15.46:0/648275003 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ee4003a10 con 0x7f7ef4101260 2024-03-31T23:16:49.375 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:49.367+0000 7f7efbf7a640 1 -- 172.21.15.46:0/648275003 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7ef41a4b00 con 0x7f7ef4101260 2024-03-31T23:16:49.388 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:49.370+0000 7f7eeaffd640 1 -- 172.21.15.46:0/648275003 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 25) ==== 137451+0+0 (secure 0 0 0) 0x7f7ee4046020 con 0x7f7ef4101260 2024-03-31T23:16:49.389 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:49.370+0000 7f7eeaffd640 1 --2- 172.21.15.46:0/648275003 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f7ee00a2730 0x7f7ee00a4bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:16:49.389 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:49.370+0000 7f7efec38740 1 Processor -- start 2024-03-31T23:16:49.389 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:49.370+0000 7f7ef9d16640 1 --2- 172.21.15.46:0/648275003 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f7ee00a2730 0x7f7ee00a4bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:16:49.389 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:49.370+0000 7f7eeaffd640 1 -- 172.21.15.46:0/648275003 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7f7ee40f07b0 con 0x7f7ef4101260 2024-03-31T23:16:49.389 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:49.370+0000 7f7efec38740 1 -- start start 2024-03-31T23:16:49.389 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:49.371+0000 7f7efec38740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e98b07d6c0 0x55e98b07daa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:16:49.390 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:49.371+0000 7f7efec38740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55e98b07dfe0 0x55e98b086870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:16:49.390 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:49.371+0000 7f7efec38740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e98b086db0 0x55e98b0891a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:16:49.390 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:49.371+0000 7f7efec38740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55e98afe08f0 con 0x55e98b07d6c0 2024-03-31T23:16:49.390 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:49.371+0000 7f7efec38740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55e98af898d0 con 0x55e98b07dfe0 2024-03-31T23:16:49.390 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:49.371+0000 7f7efec38740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55e98afdc990 con 0x55e98b086db0 2024-03-31T23:16:49.390 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:49.371+0000 7f7ef9d16640 1 --2- 172.21.15.46:0/648275003 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f7ee00a2730 0x7f7ee00a4bf0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f7ef0009b50 tx=0x7f7ef0005f90 comp rx=0 tx=0).ready entity=mgr.15654 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:16:49.391 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:49.371+0000 7f7ef9d16640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e98b07d6c0 0x55e98b07daa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:16:49.391 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:49.371+0000 7f7ef9d16640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e98b07d6c0 0x55e98b07daa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:49924/0 (socket says 172.21.15.46:49924) 2024-03-31T23:16:49.391 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:49.371+0000 7f7ef9d16640 1 -- 172.21.15.46:0/1708759064 learned_addr learned my addr 172.21.15.46:0/1708759064 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:16:49.391 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:49.371+0000 7f7ef9515640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55e98b07dfe0 0x55e98b086870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:16:49.391 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:49.371+0000 7f7efa517640 1 --2- 172.21.15.46:0/1708759064 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e98b086db0 0x55e98b0891a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:16:49.391 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:49.371+0000 7f7ef9515640 1 -- 172.21.15.46:0/1708759064 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e98b086db0 msgr2=0x55e98b0891a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:16:49.391 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:49.371+0000 7f7ef9515640 1 --2- 172.21.15.46:0/1708759064 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e98b086db0 0x55e98b0891a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:16:49.392 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:49.371+0000 7f7ef9515640 1 -- 172.21.15.46:0/1708759064 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e98b07d6c0 msgr2=0x55e98b07daa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:16:49.392 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:49.371+0000 7f7ef9515640 1 --2- 172.21.15.46:0/1708759064 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e98b07d6c0 0x55e98b07daa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:16:49.392 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:49.371+0000 7f7ef9515640 1 -- 172.21.15.46:0/1708759064 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e98acf5210 con 0x55e98b07dfe0 2024-03-31T23:16:49.392 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:49.371+0000 7f7ef9515640 1 --2- 172.21.15.46:0/1708759064 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55e98b07dfe0 0x55e98b086870 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f7ee40edc60 tx=0x7f7ee40bc5f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=899548f4297e4b33 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:16:49.392 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:49.372+0000 7f7edffff640 1 -- 172.21.15.46:0/1708759064 <== mon.2 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ee40d4020 con 0x55e98b07dfe0 2024-03-31T23:16:49.392 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:49.372+0000 7f7edffff640 1 -- 172.21.15.46:0/1708759064 <== mon.2 v2:172.21.15.142:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7ee40c0de0 con 0x55e98b07dfe0 2024-03-31T23:16:49.392 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:49.372+0000 7f7edffff640 1 -- 172.21.15.46:0/1708759064 <== mon.2 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ee40b9300 con 0x55e98b07dfe0 2024-03-31T23:16:49.393 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:49.372+0000 7f7efec38740 1 -- 172.21.15.46:0/1708759064 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55e98b07dfe0 msgr2=0x55e98b086870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:16:49.393 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:49.372+0000 7f7efec38740 1 --2- 172.21.15.46:0/1708759064 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55e98b07dfe0 0x55e98b086870 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f7ee40edc60 tx=0x7f7ee40bc5f0 comp rx=0 tx=0).stop 2024-03-31T23:16:49.393 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:49.372+0000 7f7efec38740 1 -- 172.21.15.46:0/1708759064 shutdown_connections 2024-03-31T23:16:49.393 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:49.372+0000 7f7efec38740 1 --2- 172.21.15.46:0/1708759064 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e98b086db0 0x55e98b0891a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:16:49.393 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:49.372+0000 7f7efec38740 1 --2- 172.21.15.46:0/1708759064 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55e98b07dfe0 0x55e98b086870 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:16:49.393 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:49.372+0000 7f7efec38740 1 --2- 172.21.15.46:0/1708759064 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e98b07d6c0 0x55e98b07daa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:16:49.393 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:49.372+0000 7f7efec38740 1 -- 172.21.15.46:0/1708759064 >> 172.21.15.46:0/1708759064 conn(0x55e98b077fe0 msgr2=0x55e98b0783c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:16:49.394 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:49.372+0000 7f7efec38740 1 -- 172.21.15.46:0/1708759064 shutdown_connections 2024-03-31T23:16:49.394 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:49.373+0000 7f7efec38740 1 -- 172.21.15.46:0/1708759064 wait complete. 2024-03-31T23:16:49.394 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:49.373+0000 7f7efec38740 1 -- start start 2024-03-31T23:16:49.394 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:49.373+0000 7f7efec38740 1 Processor -- start 2024-03-31T23:16:49.394 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:49.373+0000 7f7efec38740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55e98b07d6c0 0x55e98b083e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:16:49.394 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:49.373+0000 7f7efec38740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e98b07dfe0 0x55e98b084390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:16:49.395 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:49.373+0000 7f7efec38740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e98b086db0 0x55e98b0848d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:16:49.395 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:49.373+0000 7f7efec38740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55e98af898d0 con 0x55e98b086db0 2024-03-31T23:16:49.395 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:49.373+0000 7f7efec38740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55e98af7ef80 con 0x55e98b07d6c0 2024-03-31T23:16:49.395 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:49.373+0000 7f7efec38740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55e98b0897a0 con 0x55e98b07dfe0 2024-03-31T23:16:49.395 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:49.373+0000 7f7efec38740 10 client.0 _ll_register_callbacks cb 0x55e98afd7290 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T23:16:49.395 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:49.373+0000 7f7efa517640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e98b086db0 0x55e98b0848d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:16:49.396 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:49.373+0000 7f7efa517640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e98b086db0 0x55e98b0848d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:49934/0 (socket says 172.21.15.46:49934) 2024-03-31T23:16:49.396 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:49.373+0000 7f7efa517640 1 -- 172.21.15.46:0/3904482081 learned_addr learned my addr 172.21.15.46:0/3904482081 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:16:49.396 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:49.373+0000 7f7ef9d16640 1 --2- 172.21.15.46:0/3904482081 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55e98b07d6c0 0x55e98b083e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:16:49.396 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:49.373+0000 7f7ef9515640 1 --2- 172.21.15.46:0/3904482081 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e98b07dfe0 0x55e98b084390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:16:49.396 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:49.374+0000 7f7efa517640 1 -- 172.21.15.46:0/3904482081 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e98b07dfe0 msgr2=0x55e98b084390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:16:49.397 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:49.374+0000 7f7efa517640 1 --2- 172.21.15.46:0/3904482081 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e98b07dfe0 0x55e98b084390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:16:49.397 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:49.374+0000 7f7efa517640 1 -- 172.21.15.46:0/3904482081 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55e98b07d6c0 msgr2=0x55e98b083e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:16:49.397 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:49.374+0000 7f7efa517640 1 --2- 172.21.15.46:0/3904482081 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55e98b07d6c0 0x55e98b083e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:16:49.397 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:49.374+0000 7f7efa517640 1 -- 172.21.15.46:0/3904482081 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e98adc1620 con 0x55e98b086db0 2024-03-31T23:16:49.397 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:49.374+0000 7f7efa517640 1 --2- 172.21.15.46:0/3904482081 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e98b086db0 0x55e98b0848d0 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f7eec011d10 tx=0x7f7eec00f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=beeda7904d74324e server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:16:49.398 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:49.374+0000 7f7edd7fa640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-31T23:16:49.398 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:49.374+0000 7f7edd7fa640 1 -- 172.21.15.46:0/3904482081 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7eec00fe60 con 0x55e98b086db0 2024-03-31T23:16:49.398 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:49.374+0000 7f7edd7fa640 1 -- 172.21.15.46:0/3904482081 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7f7eec003970 con 0x55e98b086db0 2024-03-31T23:16:49.398 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:49.374+0000 7f7efec38740 1 -- 172.21.15.46:0/3904482081 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7ee40c0de0 con 0x55e98b086db0 2024-03-31T23:16:49.398 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:49.374+0000 7f7edd7fa640 1 -- 172.21.15.46:0/3904482081 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f7eec003b70 con 0x55e98b086db0 2024-03-31T23:16:49.399 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:49.374+0000 7f7efec38740 10 client.16563 fetch_fsmap learned FSMap version 30 2024-03-31T23:16:49.399 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:49.374+0000 7f7efec38740 1 -- 172.21.15.46:0/3904482081 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55e98b095520 con 0x55e98b086db0 2024-03-31T23:16:49.399 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:49.374+0000 7f7edd7fa640 1 -- 172.21.15.46:0/3904482081 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7ee40c0de0 con 0x55e98b086db0 2024-03-31T23:16:49.399 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:49.375+0000 7f7edd7fa640 1 -- 172.21.15.46:0/3904482081 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 30) ==== 8346+0+0 (secure 0 0 0) 0x7f7eec019070 con 0x55e98b086db0 2024-03-31T23:16:49.399 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:49.375+0000 7f7efec38740 10 client.16563 fetch_fsmap finished waiting for FSMap version 30 2024-03-31T23:16:49.400 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:49.375+0000 7f7efec38740 10 client.16563 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-31T23:16:49.400 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:49.375+0000 7f7efec38740 20 client.16563 populate_metadata read hostname 'smithi046' 2024-03-31T23:16:49.400 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:49.375+0000 7f7efec38740 1 --2- 172.21.15.46:0/3904482081 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55e98b096a50 0x55e98b098e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:16:49.400 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:49.375+0000 7f7efec38740 4 client.16563 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-31T23:16:49.400 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:49.375+0000 7f7efec38740 1 -- 172.21.15.46:0/3904482081 --> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e98aff4120 con 0x55e98b096a50 2024-03-31T23:16:49.401 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:49.375+0000 7f7ef9d16640 1 --2- 172.21.15.46:0/3904482081 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55e98b096a50 0x55e98b098e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:16:49.401 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:49.375+0000 7f7ef9d16640 1 --2- 172.21.15.46:0/3904482081 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55e98b096a50 0x55e98b098e40 crc :-1 s=READY pgs=119 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=102183b150143802 server_cookie=25cf6b2f91cef450 in_seq=0 out_seq=0 2024-03-31T23:16:49.401 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:49.375+0000 7f7edd7fa640 10 client.16563 ms_handle_connect on v2:172.21.15.142:6838/2519143918 2024-03-31T23:16:49.401 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:49.376+0000 7f7edd7fa640 1 -- 172.21.15.46:0/3904482081 <== mds.0 v2:172.21.15.142:6838/2519143918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55e98aff4120 con 0x55e98b096a50 2024-03-31T23:16:49.402 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:49.376+0000 7f7edd7fa640 10 client.16563 handle_command_reply: tid=0 2024-03-31T23:16:49.402 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:49.376+0000 7f7efec38740 1 client.16563 shutdown 2024-03-31T23:16:49.402 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:49.376+0000 7f7efec38740 1 -- 172.21.15.46:0/3904482081 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e98b086db0 msgr2=0x55e98b0848d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:16:49.402 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:49.376+0000 7f7efec38740 1 --2- 172.21.15.46:0/3904482081 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e98b086db0 0x55e98b0848d0 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f7eec011d10 tx=0x7f7eec00f790 comp rx=0 tx=0).stop 2024-03-31T23:16:49.403 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:49.376+0000 7f7efec38740 1 -- 172.21.15.46:0/3904482081 shutdown_connections 2024-03-31T23:16:49.403 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:49.376+0000 7f7efec38740 1 --2- 172.21.15.46:0/3904482081 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55e98b096a50 0x55e98b098e40 crc :-1 s=READY pgs=119 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:16:49.403 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:49.376+0000 7f7efec38740 1 --2- 172.21.15.46:0/3904482081 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e98b086db0 0x55e98b0848d0 unknown :-1 s=CLOSED pgs=704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:16:49.403 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:49.376+0000 7f7efec38740 1 --2- 172.21.15.46:0/3904482081 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e98b07dfe0 0x55e98b084390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:16:49.403 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:49.376+0000 7f7efec38740 1 --2- 172.21.15.46:0/3904482081 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55e98b07d6c0 0x55e98b083e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:16:49.404 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:49.376+0000 7f7edd7fa640 0 client.16563 ms_handle_reset on v2:172.21.15.142:6838/2519143918 2024-03-31T23:16:49.404 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:49.376+0000 7f7efec38740 1 -- 172.21.15.46:0/3904482081 >> 172.21.15.46:0/3904482081 conn(0x55e98b077fe0 msgr2=0x55e98b079d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:16:49.404 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:49.376+0000 7f7efec38740 1 -- 172.21.15.46:0/3904482081 shutdown_connections 2024-03-31T23:16:49.404 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:49.377+0000 7f7efec38740 1 -- 172.21.15.46:0/3904482081 wait complete. 2024-03-31T23:16:49.404 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:49.377+0000 7f7efec38740 20 client.16563 trim_cache size 0 max 16384 2024-03-31T23:16:49.412 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:49.397+0000 7f7efec38740 1 Processor -- start 2024-03-31T23:16:49.412 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:49.397+0000 7f7efec38740 1 -- start start 2024-03-31T23:16:49.412 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:49.397+0000 7f7efec38740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55e98b07d6c0 0x55e98b086580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:16:49.412 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:49.397+0000 7f7efec38740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e98b07dfe0 0x55e98b0783c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:16:49.412 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:49.397+0000 7f7efec38740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e98b086db0 0x55e98b0922f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:16:49.413 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:49.397+0000 7f7efec38740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f7ef41b1390 con 0x55e98b07dfe0 2024-03-31T23:16:49.413 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:49.397+0000 7f7efec38740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f7ef41aec60 con 0x55e98b07d6c0 2024-03-31T23:16:49.413 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:49.397+0000 7f7efec38740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f7ef41acab0 con 0x55e98b086db0 2024-03-31T23:16:49.413 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:49.397+0000 7f7ef9515640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e98b07dfe0 0x55e98b0783c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:16:49.413 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:49.397+0000 7f7ef9515640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e98b07dfe0 0x55e98b0783c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:49944/0 (socket says 172.21.15.46:49944) 2024-03-31T23:16:49.413 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:49.397+0000 7f7ef9d16640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55e98b07d6c0 0x55e98b086580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:16:49.413 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:49.397+0000 7f7ef9515640 1 -- 172.21.15.46:0/3761263835 learned_addr learned my addr 172.21.15.46:0/3761263835 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:16:49.413 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:49.397+0000 7f7efa517640 1 --2- 172.21.15.46:0/3761263835 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e98b086db0 0x55e98b0922f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:16:49.413 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:49.397+0000 7f7ef9515640 1 -- 172.21.15.46:0/3761263835 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e98b086db0 msgr2=0x55e98b0922f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:16:49.413 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:49.397+0000 7f7ef9515640 1 --2- 172.21.15.46:0/3761263835 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e98b086db0 0x55e98b0922f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:16:49.414 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:49.397+0000 7f7ef9515640 1 -- 172.21.15.46:0/3761263835 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55e98b07d6c0 msgr2=0x55e98b086580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:16:49.414 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:49.397+0000 7f7ef9515640 1 --2- 172.21.15.46:0/3761263835 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55e98b07d6c0 0x55e98b086580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:16:49.414 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:49.397+0000 7f7ef9515640 1 -- 172.21.15.46:0/3761263835 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e98b082e00 con 0x55e98b07dfe0 2024-03-31T23:16:49.414 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:49.398+0000 7f7ef9515640 1 --2- 172.21.15.46:0/3761263835 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e98b07dfe0 0x55e98b0783c0 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f7ee40edd10 tx=0x7f7ee40bce10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2fdbba8145f4e253 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:16:49.414 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:49.398+0000 7f7edf7fe640 1 -- 172.21.15.46:0/3761263835 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ee40d4040 con 0x55e98b07dfe0 2024-03-31T23:16:49.414 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:49.398+0000 7f7edf7fe640 1 -- 172.21.15.46:0/3761263835 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7ee40db070 con 0x55e98b07dfe0 2024-03-31T23:16:49.414 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:49.398+0000 7f7edf7fe640 1 -- 172.21.15.46:0/3761263835 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ee40edd40 con 0x55e98b07dfe0 2024-03-31T23:16:49.414 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:49.399+0000 7f7efec38740 1 -- 172.21.15.46:0/3761263835 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e98b07dfe0 msgr2=0x55e98b0783c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:16:49.414 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:49.399+0000 7f7efec38740 1 --2- 172.21.15.46:0/3761263835 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e98b07dfe0 0x55e98b0783c0 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f7ee40edd10 tx=0x7f7ee40bce10 comp rx=0 tx=0).stop 2024-03-31T23:16:49.414 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:49.399+0000 7f7efec38740 1 -- 172.21.15.46:0/3761263835 shutdown_connections 2024-03-31T23:16:49.414 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:49.399+0000 7f7efec38740 1 --2- 172.21.15.46:0/3761263835 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e98b086db0 0x55e98b0922f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:16:49.415 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:49.399+0000 7f7efec38740 1 --2- 172.21.15.46:0/3761263835 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e98b07dfe0 0x55e98b0783c0 unknown :-1 s=CLOSED pgs=705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:16:49.415 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:49.399+0000 7f7efec38740 1 --2- 172.21.15.46:0/3761263835 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55e98b07d6c0 0x55e98b086580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:16:49.415 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:49.399+0000 7f7efec38740 1 -- 172.21.15.46:0/3761263835 >> 172.21.15.46:0/3761263835 conn(0x55e98b077fe0 msgr2=0x55e98b076f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:16:49.415 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:49.399+0000 7f7efec38740 1 -- 172.21.15.46:0/3761263835 shutdown_connections 2024-03-31T23:16:49.415 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:49.399+0000 7f7efec38740 1 -- 172.21.15.46:0/3761263835 wait complete. 2024-03-31T23:16:49.415 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:49.399+0000 7f7efec38740 1 -- start start 2024-03-31T23:16:49.415 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:49.400+0000 7f7efec38740 1 Processor -- start 2024-03-31T23:16:49.415 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:49.400+0000 7f7efec38740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e98b07d6c0 0x55e98b08e510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:16:49.415 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:49.400+0000 7f7efec38740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e98b07dfe0 0x55e98b08ea50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:16:49.415 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:49.400+0000 7f7efec38740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55e98b086db0 0x55e98b08a990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:16:49.415 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:49.400+0000 7f7efec38740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f7ef41ad120 con 0x55e98b07d6c0 2024-03-31T23:16:49.415 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:49.400+0000 7f7efec38740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f7ef41ae590 con 0x55e98b086db0 2024-03-31T23:16:49.416 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:49.400+0000 7f7efec38740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f7ef41ad7f0 con 0x55e98b07dfe0 2024-03-31T23:16:49.416 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:49.400+0000 7f7efec38740 10 client.0 _ll_register_callbacks cb 0x7f7eec009cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T23:16:49.416 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:49.400+0000 7f7ef9d16640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e98b07d6c0 0x55e98b08e510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:16:49.416 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:49.400+0000 7f7ef9d16640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e98b07d6c0 0x55e98b08e510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:49954/0 (socket says 172.21.15.46:49954) 2024-03-31T23:16:49.416 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:49.400+0000 7f7ef9d16640 1 -- 172.21.15.46:0/1848680026 learned_addr learned my addr 172.21.15.46:0/1848680026 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:16:49.416 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:49.400+0000 7f7efa517640 1 --2- 172.21.15.46:0/1848680026 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55e98b086db0 0x55e98b08a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:16:49.416 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:49.400+0000 7f7ef9515640 1 --2- 172.21.15.46:0/1848680026 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e98b07dfe0 0x55e98b08ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:16:49.416 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:49.400+0000 7f7ef9d16640 1 -- 172.21.15.46:0/1848680026 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e98b07dfe0 msgr2=0x55e98b08ea50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:16:49.416 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:49.400+0000 7f7ef9d16640 1 --2- 172.21.15.46:0/1848680026 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e98b07dfe0 0x55e98b08ea50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:16:49.416 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:49.400+0000 7f7ef9d16640 1 -- 172.21.15.46:0/1848680026 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55e98b086db0 msgr2=0x55e98b08a990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:16:49.416 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:49.400+0000 7f7ef9d16640 1 --2- 172.21.15.46:0/1848680026 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55e98b086db0 0x55e98b08a990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:16:49.416 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:49.400+0000 7f7ef9d16640 1 -- 172.21.15.46:0/1848680026 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e98b0991b0 con 0x55e98b07d6c0 2024-03-31T23:16:49.417 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:49.401+0000 7f7ef9d16640 1 --2- 172.21.15.46:0/1848680026 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e98b07d6c0 0x55e98b08e510 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f7ef0004650 tx=0x7f7ef0004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=78de4fa2b3053258 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:16:49.417 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:49.401+0000 7f7edd7fa640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-31T23:16:49.417 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:49.401+0000 7f7edd7fa640 1 -- 172.21.15.46:0/1848680026 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ef002a620 con 0x55e98b07d6c0 2024-03-31T23:16:49.417 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:49.401+0000 7f7edd7fa640 1 -- 172.21.15.46:0/1848680026 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7f7ef002a7c0 con 0x55e98b07d6c0 2024-03-31T23:16:49.417 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:49.401+0000 7f7efec38740 1 -- 172.21.15.46:0/1848680026 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7ee40db070 con 0x55e98b07d6c0 2024-03-31T23:16:49.417 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:49.401+0000 7f7edd7fa640 1 -- 172.21.15.46:0/1848680026 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f7ef0038050 con 0x55e98b07d6c0 2024-03-31T23:16:49.417 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:49.401+0000 7f7efec38740 10 client.16569 fetch_fsmap learned FSMap version 30 2024-03-31T23:16:49.417 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:49.401+0000 7f7efec38740 1 -- 172.21.15.46:0/1848680026 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55e98b0fd6b0 con 0x55e98b07d6c0 2024-03-31T23:16:49.417 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:49.402+0000 7f7edd7fa640 1 -- 172.21.15.46:0/1848680026 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7ee40db070 con 0x55e98b07d6c0 2024-03-31T23:16:49.417 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:49.402+0000 7f7edd7fa640 1 -- 172.21.15.46:0/1848680026 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 30) ==== 8346+0+0 (secure 0 0 0) 0x7f7ef0004c50 con 0x55e98b07d6c0 2024-03-31T23:16:49.417 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:49.402+0000 7f7efec38740 10 client.16569 fetch_fsmap finished waiting for FSMap version 30 2024-03-31T23:16:49.418 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:49.402+0000 7f7efec38740 10 client.16569 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-31T23:16:49.418 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:49.402+0000 7f7efec38740 20 client.16569 populate_metadata read hostname 'smithi046' 2024-03-31T23:16:49.418 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:49.402+0000 7f7efec38740 1 --2- 172.21.15.46:0/1848680026 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55e98b0fde00 0x55e98b100220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:16:49.418 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:49.402+0000 7f7efec38740 4 client.16569 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-31T23:16:49.418 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:49.402+0000 7f7efec38740 1 -- 172.21.15.46:0/1848680026 --> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55e98afede30 con 0x55e98b0fde00 2024-03-31T23:16:49.418 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:49.402+0000 7f7ef9515640 1 --2- 172.21.15.46:0/1848680026 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55e98b0fde00 0x55e98b100220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:16:49.418 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:49.403+0000 7f7ef9515640 1 --2- 172.21.15.46:0/1848680026 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55e98b0fde00 0x55e98b100220 crc :-1 s=READY pgs=120 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=700677d73133644d server_cookie=33617aad3bf4f6a8 in_seq=0 out_seq=0 2024-03-31T23:16:49.418 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:49.403+0000 7f7edd7fa640 10 client.16569 ms_handle_connect on v2:172.21.15.142:6838/2519143918 2024-03-31T23:16:49.425 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:49.423+0000 7f7edd7fa640 1 -- 172.21.15.46:0/1848680026 <== mds.0 v2:172.21.15.142:6838/2519143918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55e98afede30 con 0x55e98b0fde00 2024-03-31T23:16:49.425 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:49.423+0000 7f7edd7fa640 10 client.16569 handle_command_reply: tid=0 2024-03-31T23:16:49.426 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:49.423+0000 7f7efec38740 1 client.16569 shutdown 2024-03-31T23:16:49.426 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:49.424+0000 7f7efec38740 1 -- 172.21.15.46:0/1848680026 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e98b07d6c0 msgr2=0x55e98b08e510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:16:49.426 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:49.424+0000 7f7efec38740 1 --2- 172.21.15.46:0/1848680026 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e98b07d6c0 0x55e98b08e510 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f7ef0004650 tx=0x7f7ef0004730 comp rx=0 tx=0).stop 2024-03-31T23:16:49.426 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:49.424+0000 7f7efec38740 1 -- 172.21.15.46:0/1848680026 shutdown_connections 2024-03-31T23:16:49.426 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:49.424+0000 7f7efec38740 1 --2- 172.21.15.46:0/1848680026 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55e98b0fde00 0x55e98b100220 crc :-1 s=READY pgs=120 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:16:49.426 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:49.424+0000 7f7efec38740 1 --2- 172.21.15.46:0/1848680026 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55e98b086db0 0x55e98b08a990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:16:49.426 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:49.424+0000 7f7efec38740 1 --2- 172.21.15.46:0/1848680026 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e98b07dfe0 0x55e98b08ea50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:16:49.426 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:49.424+0000 7f7edd7fa640 0 client.16569 ms_handle_reset on v2:172.21.15.142:6838/2519143918 2024-03-31T23:16:49.426 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:49.424+0000 7f7efec38740 1 --2- 172.21.15.46:0/1848680026 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e98b07d6c0 0x55e98b08e510 unknown :-1 s=CLOSED pgs=706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:16:49.426 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:49.424+0000 7f7efec38740 1 -- 172.21.15.46:0/1848680026 >> 172.21.15.46:0/1848680026 conn(0x55e98b077fe0 msgr2=0x55e98b076f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:16:49.426 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:49.424+0000 7f7efec38740 1 -- 172.21.15.46:0/1848680026 shutdown_connections 2024-03-31T23:16:49.427 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:49.424+0000 7f7efec38740 1 -- 172.21.15.46:0/1848680026 wait complete. 2024-03-31T23:16:49.427 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:49.424+0000 7f7efec38740 20 client.16569 trim_cache size 0 max 16384 2024-03-31T23:16:49.427 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout:{ 2024-03-31T23:16:49.427 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout: "message": "", 2024-03-31T23:16:49.427 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout: "return_code": 0 2024-03-31T23:16:49.427 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout:} 2024-03-31T23:16:49.428 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:49.425+0000 7f7efbf7a640 1 -- 172.21.15.46:0/648275003 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f7ee00a2730 msgr2=0x7f7ee00a4bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:16:49.428 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:49.425+0000 7f7efbf7a640 1 --2- 172.21.15.46:0/648275003 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f7ee00a2730 0x7f7ee00a4bf0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f7ef0009b50 tx=0x7f7ef0005f90 comp rx=0 tx=0).stop 2024-03-31T23:16:49.429 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:49.425+0000 7f7efbf7a640 1 -- 172.21.15.46:0/648275003 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7ef4101260 msgr2=0x7f7ef41984d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:16:49.429 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:49.425+0000 7f7efbf7a640 1 --2- 172.21.15.46:0/648275003 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7ef4101260 0x7f7ef41984d0 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f7ee4006010 tx=0x7f7ee4005d90 comp rx=0 tx=0).stop 2024-03-31T23:16:49.429 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:49.425+0000 7f7efbf7a640 1 -- 172.21.15.46:0/648275003 shutdown_connections 2024-03-31T23:16:49.429 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:49.425+0000 7f7efbf7a640 1 --2- 172.21.15.46:0/648275003 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f7ee00a2730 0x7f7ee00a4bf0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:16:49.429 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:49.425+0000 7f7efbf7a640 1 --2- 172.21.15.46:0/648275003 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7ef41055f0 0x7f7ef419c850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:16:49.429 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:49.425+0000 7f7efbf7a640 1 --2- 172.21.15.46:0/648275003 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7ef4101260 0x7f7ef41984d0 unknown :-1 s=CLOSED pgs=703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:16:49.429 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:49.425+0000 7f7efbf7a640 1 --2- 172.21.15.46:0/648275003 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7ef41008a0 0x7f7ef4197f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:16:49.429 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:49.425+0000 7f7efbf7a640 1 -- 172.21.15.46:0/648275003 >> 172.21.15.46:0/648275003 conn(0x7f7ef40774a0 msgr2=0x7f7ef4069170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:16:49.430 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:49.425+0000 7f7efbf7a640 1 -- 172.21.15.46:0/648275003 shutdown_connections 2024-03-31T23:16:49.430 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:16:49.425+0000 7f7efbf7a640 1 -- 172.21.15.46:0/648275003 wait complete. 2024-03-31T23:16:50.197 INFO:tasks.workunit.client.0.smithi046.stdout: 1 107574 2.48 MB/sec execute 116 sec latency 15.628 ms 2024-03-31T23:16:50.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:16:50 smithi142 ceph-mon[35271]: pgmap v156: 129 pgs: 129 active+clean; 78 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 546 KiB/s rd, 1.8 MiB/s wr, 172 op/s 2024-03-31T23:16:51.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:16:50 smithi162 ceph-mon[36237]: pgmap v156: 129 pgs: 129 active+clean; 78 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 546 KiB/s rd, 1.8 MiB/s wr, 172 op/s 2024-03-31T23:16:51.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:16:50 smithi046 ceph-mon[32498]: pgmap v156: 129 pgs: 129 active+clean; 78 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 546 KiB/s rd, 1.8 MiB/s wr, 172 op/s 2024-03-31T23:16:51.197 INFO:tasks.workunit.client.0.smithi046.stdout: 1 108070 2.49 MB/sec execute 117 sec latency 18.975 ms 2024-03-31T23:16:52.198 INFO:tasks.workunit.client.0.smithi046.stdout: 1 108591 2.47 MB/sec execute 118 sec latency 22.359 ms 2024-03-31T23:16:52.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:16:52 smithi142 ceph-mon[35271]: pgmap v157: 129 pgs: 129 active+clean; 78 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 365 KiB/s rd, 1.2 MiB/s wr, 113 op/s 2024-03-31T23:16:52.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:16:52 smithi142 ceph-mon[35271]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-31T23:16:52.944 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:16:52 smithi142 ceph-mon[35271]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:16:52.944 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:16:52 smithi142 ceph-mon[35271]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-31T23:16:52.944 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:16:52 smithi142 ceph-mon[35271]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' 2024-03-31T23:16:53.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:16:52 smithi162 ceph-mon[36237]: pgmap v157: 129 pgs: 129 active+clean; 78 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 365 KiB/s rd, 1.2 MiB/s wr, 113 op/s 2024-03-31T23:16:53.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:16:52 smithi162 ceph-mon[36237]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-31T23:16:53.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:16:52 smithi162 ceph-mon[36237]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:16:53.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:16:52 smithi162 ceph-mon[36237]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-31T23:16:53.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:16:52 smithi162 ceph-mon[36237]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' 2024-03-31T23:16:53.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:16:52 smithi046 ceph-mon[32498]: pgmap v157: 129 pgs: 129 active+clean; 78 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 365 KiB/s rd, 1.2 MiB/s wr, 113 op/s 2024-03-31T23:16:53.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:16:52 smithi046 ceph-mon[32498]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-31T23:16:53.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:16:52 smithi046 ceph-mon[32498]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:16:53.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:16:52 smithi046 ceph-mon[32498]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-31T23:16:53.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:16:52 smithi046 ceph-mon[32498]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' 2024-03-31T23:16:53.197 INFO:tasks.workunit.client.0.smithi046.stdout: 1 109031 2.47 MB/sec execute 119 sec latency 21.910 ms 2024-03-31T23:16:54.197 INFO:tasks.workunit.client.0.smithi046.stdout: 1 109736 2.50 MB/sec execute 120 sec latency 16.948 ms 2024-03-31T23:16:54.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:16:54 smithi142 ceph-mon[35271]: pgmap v158: 129 pgs: 129 active+clean; 78 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 365 KiB/s rd, 1.2 MiB/s wr, 113 op/s 2024-03-31T23:16:55.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:16:54 smithi162 ceph-mon[36237]: pgmap v158: 129 pgs: 129 active+clean; 78 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 365 KiB/s rd, 1.2 MiB/s wr, 113 op/s 2024-03-31T23:16:55.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:16:54 smithi046 ceph-mon[32498]: pgmap v158: 129 pgs: 129 active+clean; 78 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 365 KiB/s rd, 1.2 MiB/s wr, 113 op/s 2024-03-31T23:16:55.197 INFO:tasks.workunit.client.0.smithi046.stdout: 1 110083 2.51 MB/sec execute 121 sec latency 17.122 ms 2024-03-31T23:16:56.197 INFO:tasks.workunit.client.0.smithi046.stdout: 1 110385 2.50 MB/sec execute 122 sec latency 15.118 ms 2024-03-31T23:16:56.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:16:56 smithi142 ceph-mon[35271]: pgmap v159: 129 pgs: 129 active+clean; 76 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 552 KiB/s rd, 1.8 MiB/s wr, 172 op/s 2024-03-31T23:16:57.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:16:56 smithi162 ceph-mon[36237]: pgmap v159: 129 pgs: 129 active+clean; 76 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 552 KiB/s rd, 1.8 MiB/s wr, 172 op/s 2024-03-31T23:16:57.127 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:16:56 smithi046 ceph-mon[32498]: pgmap v159: 129 pgs: 129 active+clean; 76 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 552 KiB/s rd, 1.8 MiB/s wr, 172 op/s 2024-03-31T23:16:57.198 INFO:tasks.workunit.client.0.smithi046.stdout: 1 110658 2.48 MB/sec execute 123 sec latency 16.782 ms 2024-03-31T23:16:58.197 INFO:tasks.workunit.client.0.smithi046.stdout: 1 111126 2.48 MB/sec execute 124 sec latency 17.728 ms 2024-03-31T23:16:58.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:16:58 smithi142 ceph-mon[35271]: pgmap v160: 129 pgs: 129 active+clean; 76 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 361 KiB/s rd, 1.3 MiB/s wr, 112 op/s 2024-03-31T23:16:59.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:16:58 smithi162 ceph-mon[36237]: pgmap v160: 129 pgs: 129 active+clean; 76 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 361 KiB/s rd, 1.3 MiB/s wr, 112 op/s 2024-03-31T23:16:59.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:16:58 smithi046 ceph-mon[32498]: pgmap v160: 129 pgs: 129 active+clean; 76 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 361 KiB/s rd, 1.3 MiB/s wr, 112 op/s 2024-03-31T23:16:59.197 INFO:tasks.workunit.client.0.smithi046.stdout: 1 111612 2.48 MB/sec execute 125 sec latency 23.638 ms 2024-03-31T23:17:00.198 INFO:tasks.workunit.client.0.smithi046.stdout: 1 112184 2.47 MB/sec execute 126 sec latency 21.736 ms 2024-03-31T23:17:00.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:17:00 smithi142 ceph-mon[35271]: pgmap v161: 129 pgs: 129 active+clean; 77 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 552 KiB/s rd, 1.8 MiB/s wr, 170 op/s 2024-03-31T23:17:01.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:17:00 smithi162 ceph-mon[36237]: pgmap v161: 129 pgs: 129 active+clean; 77 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 552 KiB/s rd, 1.8 MiB/s wr, 170 op/s 2024-03-31T23:17:01.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:17:00 smithi046 ceph-mon[32498]: pgmap v161: 129 pgs: 129 active+clean; 77 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 552 KiB/s rd, 1.8 MiB/s wr, 170 op/s 2024-03-31T23:17:01.198 INFO:tasks.workunit.client.0.smithi046.stdout: 1 112605 2.47 MB/sec execute 127 sec latency 23.597 ms 2024-03-31T23:17:02.198 INFO:tasks.workunit.client.0.smithi046.stdout: 1 113304 2.50 MB/sec execute 128 sec latency 16.176 ms 2024-03-31T23:17:02.521 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:02.519+0000 7fd6dbf27640 1 -- 172.21.15.46:0/925648932 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd6d4107820 msgr2=0x7fd6d4107c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:17:02.521 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:02.519+0000 7fd6dbf27640 1 --2- 172.21.15.46:0/925648932 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd6d4107820 0x7fd6d4107c00 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7fd6d0009930 tx=0x7fd6d002bc30 comp rx=0 tx=0).stop 2024-03-31T23:17:02.521 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:02.519+0000 7fd6dbf27640 1 -- 172.21.15.46:0/925648932 shutdown_connections 2024-03-31T23:17:02.521 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:02.519+0000 7fd6dbf27640 1 --2- 172.21.15.46:0/925648932 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd6d4069770 0x7fd6d410e5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:17:02.521 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:02.519+0000 7fd6dbf27640 1 --2- 172.21.15.46:0/925648932 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd6d4068dd0 0x7fd6d4069230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:17:02.522 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:02.519+0000 7fd6dbf27640 1 --2- 172.21.15.46:0/925648932 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd6d4107820 0x7fd6d4107c00 unknown :-1 s=CLOSED pgs=707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:17:02.522 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:02.519+0000 7fd6dbf27640 1 -- 172.21.15.46:0/925648932 >> 172.21.15.46:0/925648932 conn(0x7fd6d40774a0 msgr2=0x7fd6d40778b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:17:02.522 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:02.519+0000 7fd6dbf27640 1 -- 172.21.15.46:0/925648932 shutdown_connections 2024-03-31T23:17:02.522 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:02.519+0000 7fd6dbf27640 1 -- 172.21.15.46:0/925648932 wait complete. 2024-03-31T23:17:02.522 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:02.519+0000 7fd6dbf27640 1 Processor -- start 2024-03-31T23:17:02.522 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:02.519+0000 7fd6dbf27640 1 -- start start 2024-03-31T23:17:02.522 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:02.520+0000 7fd6dbf27640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd6d4068dd0 0x7fd6d4100970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:17:02.522 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:02.520+0000 7fd6dbf27640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd6d4069770 0x7fd6d4100eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:17:02.522 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:02.520+0000 7fd6dbf27640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd6d4107820 0x7fd6d41013f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:17:02.522 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:02.520+0000 7fd6dbf27640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fd6d41119c0 con 0x7fd6d4068dd0 2024-03-31T23:17:02.522 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:02.520+0000 7fd6dbf27640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fd6d4111840 con 0x7fd6d4107820 2024-03-31T23:17:02.522 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:02.520+0000 7fd6dbf27640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fd6d4111b40 con 0x7fd6d4069770 2024-03-31T23:17:02.523 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:02.520+0000 7fd6d9cc3640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd6d4068dd0 0x7fd6d4100970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:17:02.523 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:02.520+0000 7fd6d9cc3640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd6d4068dd0 0x7fd6d4100970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:51100/0 (socket says 172.21.15.46:51100) 2024-03-31T23:17:02.523 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:02.520+0000 7fd6d9cc3640 1 -- 172.21.15.46:0/363501491 learned_addr learned my addr 172.21.15.46:0/363501491 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:17:02.523 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:02.520+0000 7fd6d94c2640 1 --2- 172.21.15.46:0/363501491 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd6d4069770 0x7fd6d4100eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:17:02.523 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:02.520+0000 7fd6da4c4640 1 --2- 172.21.15.46:0/363501491 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd6d4107820 0x7fd6d41013f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:17:02.523 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:02.520+0000 7fd6d9cc3640 1 -- 172.21.15.46:0/363501491 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd6d4069770 msgr2=0x7fd6d4100eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:17:02.523 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:02.520+0000 7fd6d9cc3640 1 --2- 172.21.15.46:0/363501491 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd6d4069770 0x7fd6d4100eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:17:02.523 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:02.520+0000 7fd6d9cc3640 1 -- 172.21.15.46:0/363501491 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd6d4107820 msgr2=0x7fd6d41013f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:17:02.523 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:02.520+0000 7fd6d9cc3640 1 --2- 172.21.15.46:0/363501491 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd6d4107820 0x7fd6d41013f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:17:02.523 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:02.520+0000 7fd6d9cc3640 1 -- 172.21.15.46:0/363501491 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6d41a63c0 con 0x7fd6d4068dd0 2024-03-31T23:17:02.523 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:02.520+0000 7fd6d9cc3640 1 --2- 172.21.15.46:0/363501491 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd6d4068dd0 0x7fd6d4100970 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7fd6d0005940 tx=0x7fd6d0005dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:17:02.523 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:02.520+0000 7fd6caffd640 1 -- 172.21.15.46:0/363501491 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6d0043020 con 0x7fd6d4068dd0 2024-03-31T23:17:02.523 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:02.520+0000 7fd6caffd640 1 -- 172.21.15.46:0/363501491 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd6d0004350 con 0x7fd6d4068dd0 2024-03-31T23:17:02.523 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:02.520+0000 7fd6caffd640 1 -- 172.21.15.46:0/363501491 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6d0003a80 con 0x7fd6d4068dd0 2024-03-31T23:17:02.524 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:02.520+0000 7fd6dbf27640 1 -- 172.21.15.46:0/363501491 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd6d41a6650 con 0x7fd6d4068dd0 2024-03-31T23:17:02.524 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:02.521+0000 7fd6dbf27640 1 -- 172.21.15.46:0/363501491 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd6d41a6bf0 con 0x7fd6d4068dd0 2024-03-31T23:17:02.529 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:02.523+0000 7fd6caffd640 1 -- 172.21.15.46:0/363501491 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 25) ==== 137451+0+0 (secure 0 0 0) 0x7fd6d0046020 con 0x7fd6d4068dd0 2024-03-31T23:17:02.529 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:02.523+0000 7fd6debdf740 1 Processor -- start 2024-03-31T23:17:02.529 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:02.524+0000 7fd6debdf740 1 -- start start 2024-03-31T23:17:02.529 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:02.524+0000 7fd6debdf740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d0f19626c0 0x55d0f1962aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:17:02.529 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:02.524+0000 7fd6debdf740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d0f1962fe0 0x55d0f196b870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:17:02.529 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:02.524+0000 7fd6debdf740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d0f196bdb0 0x55d0f196e1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:17:02.529 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:02.524+0000 7fd6debdf740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55d0f18c58f0 con 0x55d0f1962fe0 2024-03-31T23:17:02.529 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:02.524+0000 7fd6debdf740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55d0f186e8d0 con 0x55d0f196bdb0 2024-03-31T23:17:02.529 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:02.524+0000 7fd6debdf740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55d0f18c1990 con 0x55d0f19626c0 2024-03-31T23:17:02.529 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:02.524+0000 7fd6d9cc3640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d0f1962fe0 0x55d0f196b870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:17:02.529 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:02.524+0000 7fd6da4c4640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d0f19626c0 0x55d0f1962aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:17:02.530 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:02.524+0000 7fd6d9cc3640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d0f1962fe0 0x55d0f196b870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:51114/0 (socket says 172.21.15.46:51114) 2024-03-31T23:17:02.530 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:02.524+0000 7fd6d9cc3640 1 -- 172.21.15.46:0/2235444399 learned_addr learned my addr 172.21.15.46:0/2235444399 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:17:02.530 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:02.524+0000 7fd6d94c2640 1 --2- 172.21.15.46:0/2235444399 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d0f196bdb0 0x55d0f196e1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:17:02.530 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:02.524+0000 7fd6caffd640 1 --2- 172.21.15.46:0/363501491 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fd6c00a2730 0x7fd6c00a4bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:17:02.530 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:02.524+0000 7fd6da4c4640 1 --2- 172.21.15.46:0/363501491 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fd6c00a2730 0x7fd6c00a4bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:17:02.530 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:02.524+0000 7fd6caffd640 1 -- 172.21.15.46:0/363501491 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7fd6d00f06a0 con 0x7fd6d4068dd0 2024-03-31T23:17:02.530 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:02.524+0000 7fd6d94c2640 1 -- 172.21.15.46:0/2235444399 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d0f19626c0 msgr2=0x55d0f1962aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:17:02.530 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:02.524+0000 7fd6d94c2640 1 --2- 172.21.15.46:0/2235444399 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d0f19626c0 0x55d0f1962aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:17:02.530 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:02.524+0000 7fd6d94c2640 1 -- 172.21.15.46:0/2235444399 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d0f1962fe0 msgr2=0x55d0f196b870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:17:02.530 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:02.524+0000 7fd6d94c2640 1 --2- 172.21.15.46:0/2235444399 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d0f1962fe0 0x55d0f196b870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:17:02.530 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:02.524+0000 7fd6d94c2640 1 -- 172.21.15.46:0/2235444399 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d0f15da210 con 0x55d0f196bdb0 2024-03-31T23:17:02.530 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:02.525+0000 7fd6d94c2640 1 --2- 172.21.15.46:0/2235444399 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d0f196bdb0 0x55d0f196e1a0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fd6c4009bd0 tx=0x7fd6c4011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=99710a6c3304f1cd server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:17:02.530 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:02.525+0000 7fd6da4c4640 1 --2- 172.21.15.46:0/363501491 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fd6c00a2730 0x7fd6c00a4bf0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fd6cc005fd0 tx=0x7fd6cc004ab0 comp rx=0 tx=0).ready entity=mgr.15654 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:17:02.531 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:02.525+0000 7fd6bffff640 1 -- 172.21.15.46:0/2235444399 <== mon.2 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6c4004230 con 0x55d0f196bdb0 2024-03-31T23:17:02.531 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:02.525+0000 7fd6bffff640 1 -- 172.21.15.46:0/2235444399 <== mon.2 v2:172.21.15.142:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd6c4009e40 con 0x55d0f196bdb0 2024-03-31T23:17:02.531 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:02.525+0000 7fd6bffff640 1 -- 172.21.15.46:0/2235444399 <== mon.2 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6c4004c60 con 0x55d0f196bdb0 2024-03-31T23:17:02.531 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:02.526+0000 7fd6debdf740 1 -- 172.21.15.46:0/2235444399 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d0f196bdb0 msgr2=0x55d0f196e1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:17:02.531 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:02.526+0000 7fd6debdf740 1 --2- 172.21.15.46:0/2235444399 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d0f196bdb0 0x55d0f196e1a0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fd6c4009bd0 tx=0x7fd6c4011810 comp rx=0 tx=0).stop 2024-03-31T23:17:02.531 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:02.526+0000 7fd6debdf740 1 -- 172.21.15.46:0/2235444399 shutdown_connections 2024-03-31T23:17:02.531 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:02.526+0000 7fd6debdf740 1 --2- 172.21.15.46:0/2235444399 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d0f196bdb0 0x55d0f196e1a0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:17:02.531 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:02.526+0000 7fd6debdf740 1 --2- 172.21.15.46:0/2235444399 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d0f1962fe0 0x55d0f196b870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:17:02.531 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:02.526+0000 7fd6debdf740 1 --2- 172.21.15.46:0/2235444399 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d0f19626c0 0x55d0f1962aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:17:02.531 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:02.526+0000 7fd6debdf740 1 -- 172.21.15.46:0/2235444399 >> 172.21.15.46:0/2235444399 conn(0x55d0f195cfe0 msgr2=0x55d0f195d3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:17:02.531 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:02.526+0000 7fd6debdf740 1 -- 172.21.15.46:0/2235444399 shutdown_connections 2024-03-31T23:17:02.531 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:02.526+0000 7fd6debdf740 1 -- 172.21.15.46:0/2235444399 wait complete. 2024-03-31T23:17:02.531 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:02.526+0000 7fd6debdf740 1 -- start start 2024-03-31T23:17:02.531 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:02.526+0000 7fd6debdf740 1 Processor -- start 2024-03-31T23:17:02.531 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:02.527+0000 7fd6debdf740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d0f19626c0 0x55d0f1968dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:17:02.532 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:02.527+0000 7fd6debdf740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d0f1962fe0 0x55d0f1969310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:17:02.532 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:02.527+0000 7fd6debdf740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d0f196bdb0 0x55d0f1969850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:17:02.532 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:02.527+0000 7fd6debdf740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55d0f196e9e0 con 0x55d0f1962fe0 2024-03-31T23:17:02.532 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:02.527+0000 7fd6debdf740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55d0f1863f80 con 0x55d0f19626c0 2024-03-31T23:17:02.532 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:02.527+0000 7fd6debdf740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55d0f196eb60 con 0x55d0f196bdb0 2024-03-31T23:17:02.532 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:02.527+0000 7fd6debdf740 10 client.0 _ll_register_callbacks cb 0x55d0f18bc290 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T23:17:02.532 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:02.527+0000 7fd6d94c2640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d0f1962fe0 0x55d0f1969310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:17:02.532 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:02.527+0000 7fd6d94c2640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d0f1962fe0 0x55d0f1969310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:51116/0 (socket says 172.21.15.46:51116) 2024-03-31T23:17:02.532 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:02.527+0000 7fd6d94c2640 1 -- 172.21.15.46:0/1813049095 learned_addr learned my addr 172.21.15.46:0/1813049095 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:17:02.532 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:02.527+0000 7fd6d9cc3640 1 --2- 172.21.15.46:0/1813049095 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d0f19626c0 0x55d0f1968dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:17:02.532 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:02.527+0000 7fd6da4c4640 1 --2- 172.21.15.46:0/1813049095 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d0f196bdb0 0x55d0f1969850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:17:02.532 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:02.527+0000 7fd6d94c2640 1 -- 172.21.15.46:0/1813049095 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d0f196bdb0 msgr2=0x55d0f1969850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:17:02.532 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:02.527+0000 7fd6d94c2640 1 --2- 172.21.15.46:0/1813049095 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d0f196bdb0 0x55d0f1969850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:17:02.532 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:02.527+0000 7fd6d94c2640 1 -- 172.21.15.46:0/1813049095 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d0f19626c0 msgr2=0x55d0f1968dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:17:02.533 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:02.527+0000 7fd6d94c2640 1 --2- 172.21.15.46:0/1813049095 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d0f19626c0 0x55d0f1968dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:17:02.533 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:02.527+0000 7fd6d94c2640 1 -- 172.21.15.46:0/1813049095 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d0f16a6620 con 0x55d0f1962fe0 2024-03-31T23:17:02.533 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:02.527+0000 7fd6d9cc3640 1 --2- 172.21.15.46:0/1813049095 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d0f19626c0 0x55d0f1968dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:17:02.533 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:02.527+0000 7fd6d94c2640 1 --2- 172.21.15.46:0/1813049095 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d0f1962fe0 0x55d0f1969310 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7fd6c40047c0 tx=0x7fd6c4009830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=36c7958ff73e3e4 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:17:02.533 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:02.527+0000 7fd6bd7fa640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-31T23:17:02.533 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:02.528+0000 7fd6bd7fa640 1 -- 172.21.15.46:0/1813049095 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6c4027020 con 0x55d0f1962fe0 2024-03-31T23:17:02.533 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:02.528+0000 7fd6bd7fa640 1 -- 172.21.15.46:0/1813049095 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7fd6c40038f0 con 0x55d0f1962fe0 2024-03-31T23:17:02.533 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:02.528+0000 7fd6debdf740 1 -- 172.21.15.46:0/1813049095 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55d0f18d9120 con 0x55d0f1962fe0 2024-03-31T23:17:02.533 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:02.528+0000 7fd6bd7fa640 1 -- 172.21.15.46:0/1813049095 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fd6c4032050 con 0x55d0f1962fe0 2024-03-31T23:17:02.533 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:02.528+0000 7fd6debdf740 10 client.16581 fetch_fsmap learned FSMap version 30 2024-03-31T23:17:02.533 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:02.528+0000 7fd6debdf740 1 -- 172.21.15.46:0/1813049095 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55d0f197a890 con 0x55d0f1962fe0 2024-03-31T23:17:02.533 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:02.528+0000 7fd6bd7fa640 1 -- 172.21.15.46:0/1813049095 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x55d0f18d9120 con 0x55d0f1962fe0 2024-03-31T23:17:02.533 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:02.528+0000 7fd6bd7fa640 1 -- 172.21.15.46:0/1813049095 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 30) ==== 8346+0+0 (secure 0 0 0) 0x7fd6c4018070 con 0x55d0f1962fe0 2024-03-31T23:17:02.533 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:02.528+0000 7fd6debdf740 10 client.16581 fetch_fsmap finished waiting for FSMap version 30 2024-03-31T23:17:02.534 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:02.528+0000 7fd6debdf740 10 client.16581 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-31T23:17:02.534 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:02.528+0000 7fd6debdf740 20 client.16581 populate_metadata read hostname 'smithi046' 2024-03-31T23:17:02.534 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:02.528+0000 7fd6debdf740 1 --2- 172.21.15.46:0/1813049095 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55d0f197bcf0 0x55d0f197e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:17:02.534 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:02.528+0000 7fd6debdf740 4 client.16581 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-31T23:17:02.534 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:02.528+0000 7fd6debdf740 1 -- 172.21.15.46:0/1813049095 --> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d0f18d2e30 con 0x55d0f197bcf0 2024-03-31T23:17:02.534 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:02.528+0000 7fd6d9cc3640 1 --2- 172.21.15.46:0/1813049095 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55d0f197bcf0 0x55d0f197e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:17:02.534 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:02.529+0000 7fd6bd7fa640 10 client.16581 ms_handle_connect on v2:172.21.15.142:6838/2519143918 2024-03-31T23:17:02.534 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:02.529+0000 7fd6d9cc3640 1 --2- 172.21.15.46:0/1813049095 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55d0f197bcf0 0x55d0f197e0e0 crc :-1 s=READY pgs=121 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5ff41334a2631558 server_cookie=d59e92a664e5033d in_seq=0 out_seq=0 2024-03-31T23:17:02.535 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:02.530+0000 7fd6bd7fa640 1 -- 172.21.15.46:0/1813049095 <== mds.0 v2:172.21.15.142:6838/2519143918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55d0f18d2e30 con 0x55d0f197bcf0 2024-03-31T23:17:02.535 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:02.530+0000 7fd6bd7fa640 10 client.16581 handle_command_reply: tid=0 2024-03-31T23:17:02.535 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:02.530+0000 7fd6debdf740 1 client.16581 shutdown 2024-03-31T23:17:02.535 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:02.530+0000 7fd6debdf740 1 -- 172.21.15.46:0/1813049095 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d0f1962fe0 msgr2=0x55d0f1969310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:17:02.535 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:02.530+0000 7fd6debdf740 1 --2- 172.21.15.46:0/1813049095 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d0f1962fe0 0x55d0f1969310 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7fd6c40047c0 tx=0x7fd6c4009830 comp rx=0 tx=0).stop 2024-03-31T23:17:02.536 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:02.530+0000 7fd6debdf740 1 -- 172.21.15.46:0/1813049095 shutdown_connections 2024-03-31T23:17:02.536 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:02.530+0000 7fd6debdf740 1 --2- 172.21.15.46:0/1813049095 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55d0f197bcf0 0x55d0f197e0e0 crc :-1 s=READY pgs=121 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:17:02.536 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:02.530+0000 7fd6debdf740 1 --2- 172.21.15.46:0/1813049095 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d0f196bdb0 0x55d0f1969850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:17:02.536 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:02.530+0000 7fd6debdf740 1 --2- 172.21.15.46:0/1813049095 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d0f1962fe0 0x55d0f1969310 unknown :-1 s=CLOSED pgs=709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:17:02.536 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:02.530+0000 7fd6debdf740 1 --2- 172.21.15.46:0/1813049095 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d0f19626c0 0x55d0f1968dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:17:02.536 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:02.530+0000 7fd6debdf740 1 -- 172.21.15.46:0/1813049095 >> 172.21.15.46:0/1813049095 conn(0x55d0f195cfe0 msgr2=0x55d0f195ed30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:17:02.536 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:02.530+0000 7fd6bd7fa640 0 client.16581 ms_handle_reset on v2:172.21.15.142:6838/2519143918 2024-03-31T23:17:02.536 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:02.530+0000 7fd6debdf740 1 -- 172.21.15.46:0/1813049095 shutdown_connections 2024-03-31T23:17:02.536 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:02.530+0000 7fd6debdf740 1 -- 172.21.15.46:0/1813049095 wait complete. 2024-03-31T23:17:02.536 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:02.530+0000 7fd6debdf740 20 client.16581 trim_cache size 0 max 16384 2024-03-31T23:17:02.547 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:02.546+0000 7fd6debdf740 1 Processor -- start 2024-03-31T23:17:02.547 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:02.546+0000 7fd6debdf740 1 -- start start 2024-03-31T23:17:02.548 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:02.546+0000 7fd6debdf740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d0f19626c0 0x55d0f196b500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:17:02.548 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:02.546+0000 7fd6debdf740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d0f1962fe0 0x55d0f1977780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:17:02.548 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:02.546+0000 7fd6debdf740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d0f196bdb0 0x55d0f19838a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:17:02.548 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:02.546+0000 7fd6debdf740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fd6d41b3520 con 0x55d0f196bdb0 2024-03-31T23:17:02.548 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:02.546+0000 7fd6debdf740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fd6d41b0df0 con 0x55d0f1962fe0 2024-03-31T23:17:02.548 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:02.546+0000 7fd6debdf740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fd6d41aec40 con 0x55d0f19626c0 2024-03-31T23:17:02.548 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:02.547+0000 7fd6da4c4640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d0f196bdb0 0x55d0f19838a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:17:02.548 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:02.547+0000 7fd6d94c2640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d0f1962fe0 0x55d0f1977780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:17:02.548 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:02.547+0000 7fd6d9cc3640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d0f19626c0 0x55d0f196b500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:17:02.548 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:02.547+0000 7fd6da4c4640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d0f196bdb0 0x55d0f19838a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:51122/0 (socket says 172.21.15.46:51122) 2024-03-31T23:17:02.548 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:02.547+0000 7fd6da4c4640 1 -- 172.21.15.46:0/534312345 learned_addr learned my addr 172.21.15.46:0/534312345 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:17:02.548 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:02.547+0000 7fd6da4c4640 1 -- 172.21.15.46:0/534312345 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d0f19626c0 msgr2=0x55d0f196b500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:17:02.548 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:02.547+0000 7fd6da4c4640 1 --2- 172.21.15.46:0/534312345 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d0f19626c0 0x55d0f196b500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:17:02.548 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:02.547+0000 7fd6da4c4640 1 -- 172.21.15.46:0/534312345 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d0f1962fe0 msgr2=0x55d0f1977780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:17:02.549 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:02.547+0000 7fd6da4c4640 1 --2- 172.21.15.46:0/534312345 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d0f1962fe0 0x55d0f1977780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:17:02.549 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:02.547+0000 7fd6da4c4640 1 -- 172.21.15.46:0/534312345 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d0f1962e10 con 0x55d0f196bdb0 2024-03-31T23:17:02.549 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:02.547+0000 7fd6da4c4640 1 --2- 172.21.15.46:0/534312345 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d0f196bdb0 0x55d0f19838a0 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7fd6cc0131c0 tx=0x7fd6cc009c70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=528366a19781aa6b server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:17:02.549 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:02.547+0000 7fd6bf7fe640 1 -- 172.21.15.46:0/534312345 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6cc02f070 con 0x55d0f196bdb0 2024-03-31T23:17:02.549 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:02.547+0000 7fd6bf7fe640 1 -- 172.21.15.46:0/534312345 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd6cc0132b0 con 0x55d0f196bdb0 2024-03-31T23:17:02.549 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:02.547+0000 7fd6bf7fe640 1 -- 172.21.15.46:0/534312345 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6cc022e60 con 0x55d0f196bdb0 2024-03-31T23:17:02.552 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:02.548+0000 7fd6debdf740 1 -- 172.21.15.46:0/534312345 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d0f196bdb0 msgr2=0x55d0f19838a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:17:02.552 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:02.548+0000 7fd6debdf740 1 --2- 172.21.15.46:0/534312345 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d0f196bdb0 0x55d0f19838a0 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7fd6cc0131c0 tx=0x7fd6cc009c70 comp rx=0 tx=0).stop 2024-03-31T23:17:02.552 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:02.548+0000 7fd6debdf740 1 -- 172.21.15.46:0/534312345 shutdown_connections 2024-03-31T23:17:02.553 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:02.548+0000 7fd6debdf740 1 --2- 172.21.15.46:0/534312345 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d0f196bdb0 0x55d0f19838a0 unknown :-1 s=CLOSED pgs=710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:17:02.553 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:02.548+0000 7fd6debdf740 1 --2- 172.21.15.46:0/534312345 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d0f1962fe0 0x55d0f1977780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:17:02.553 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:02.548+0000 7fd6debdf740 1 --2- 172.21.15.46:0/534312345 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d0f19626c0 0x55d0f196b500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:17:02.553 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:02.548+0000 7fd6debdf740 1 -- 172.21.15.46:0/534312345 >> 172.21.15.46:0/534312345 conn(0x55d0f195cfe0 msgr2=0x55d0f195d3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:17:02.553 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:02.548+0000 7fd6debdf740 1 -- 172.21.15.46:0/534312345 shutdown_connections 2024-03-31T23:17:02.553 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:02.548+0000 7fd6debdf740 1 -- 172.21.15.46:0/534312345 wait complete. 2024-03-31T23:17:02.553 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:02.548+0000 7fd6debdf740 1 -- start start 2024-03-31T23:17:02.553 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:02.548+0000 7fd6debdf740 1 Processor -- start 2024-03-31T23:17:02.553 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:02.549+0000 7fd6debdf740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d0f19626c0 0x55d0f1966480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:17:02.553 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:02.549+0000 7fd6debdf740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d0f1962fe0 0x55d0f19669c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:17:02.553 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:02.549+0000 7fd6debdf740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d0f196bdb0 0x55d0f1965040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:17:02.553 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:02.549+0000 7fd6debdf740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fd6d41af2b0 con 0x55d0f19626c0 2024-03-31T23:17:02.553 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:02.549+0000 7fd6debdf740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fd6d41b0720 con 0x55d0f1962fe0 2024-03-31T23:17:02.553 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:02.549+0000 7fd6debdf740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fd6d41af980 con 0x55d0f196bdb0 2024-03-31T23:17:02.554 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:02.549+0000 7fd6debdf740 10 client.0 _ll_register_callbacks cb 0x7fd6d00b8ea0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T23:17:02.554 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:02.549+0000 7fd6d9cc3640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d0f19626c0 0x55d0f1966480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:17:02.554 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:02.549+0000 7fd6d9cc3640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d0f19626c0 0x55d0f1966480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:51124/0 (socket says 172.21.15.46:51124) 2024-03-31T23:17:02.554 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:02.549+0000 7fd6d9cc3640 1 -- 172.21.15.46:0/708009866 learned_addr learned my addr 172.21.15.46:0/708009866 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:17:02.554 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:02.549+0000 7fd6d94c2640 1 --2- 172.21.15.46:0/708009866 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d0f1962fe0 0x55d0f19669c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:17:02.554 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:02.549+0000 7fd6da4c4640 1 --2- 172.21.15.46:0/708009866 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d0f196bdb0 0x55d0f1965040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:17:02.554 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:02.549+0000 7fd6d9cc3640 1 -- 172.21.15.46:0/708009866 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d0f196bdb0 msgr2=0x55d0f1965040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:17:02.554 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:02.549+0000 7fd6d9cc3640 1 --2- 172.21.15.46:0/708009866 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d0f196bdb0 0x55d0f1965040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:17:02.554 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:02.549+0000 7fd6d9cc3640 1 -- 172.21.15.46:0/708009866 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d0f1962fe0 msgr2=0x55d0f19669c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:17:02.554 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:02.549+0000 7fd6d9cc3640 1 --2- 172.21.15.46:0/708009866 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d0f1962fe0 0x55d0f19669c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:17:02.554 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:02.549+0000 7fd6d9cc3640 1 -- 172.21.15.46:0/708009866 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d0f1965580 con 0x55d0f19626c0 2024-03-31T23:17:02.554 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:02.549+0000 7fd6d94c2640 1 --2- 172.21.15.46:0/708009866 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d0f1962fe0 0x55d0f19669c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:17:02.554 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:02.549+0000 7fd6d9cc3640 1 --2- 172.21.15.46:0/708009866 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d0f19626c0 0x55d0f1966480 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7fd6d0000c00 tx=0x7fd6d0009620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f6da705322a2c056 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:17:02.554 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:02.549+0000 7fd6bd7fa640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-31T23:17:02.555 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:02.550+0000 7fd6bd7fa640 1 -- 172.21.15.46:0/708009866 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6d00bd280 con 0x55d0f19626c0 2024-03-31T23:17:02.555 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:02.550+0000 7fd6bd7fa640 1 -- 172.21.15.46:0/708009866 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7fd6d0005440 con 0x55d0f19626c0 2024-03-31T23:17:02.555 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:02.550+0000 7fd6debdf740 1 -- 172.21.15.46:0/708009866 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd6cc0132b0 con 0x55d0f19626c0 2024-03-31T23:17:02.555 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:02.550+0000 7fd6bd7fa640 1 -- 172.21.15.46:0/708009866 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fd6d00bcb80 con 0x55d0f19626c0 2024-03-31T23:17:02.555 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:02.550+0000 7fd6debdf740 10 client.16587 fetch_fsmap learned FSMap version 30 2024-03-31T23:17:02.555 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:02.550+0000 7fd6debdf740 1 -- 172.21.15.46:0/708009866 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55d0f19e1620 con 0x55d0f19626c0 2024-03-31T23:17:02.555 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:02.550+0000 7fd6bd7fa640 1 -- 172.21.15.46:0/708009866 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd6cc0132b0 con 0x55d0f19626c0 2024-03-31T23:17:02.555 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:02.550+0000 7fd6bd7fa640 1 -- 172.21.15.46:0/708009866 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 30) ==== 8346+0+0 (secure 0 0 0) 0x7fd6d00a7320 con 0x55d0f19626c0 2024-03-31T23:17:02.555 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:02.550+0000 7fd6debdf740 10 client.16587 fetch_fsmap finished waiting for FSMap version 30 2024-03-31T23:17:02.555 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:02.550+0000 7fd6debdf740 10 client.16587 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-31T23:17:02.555 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:02.550+0000 7fd6debdf740 20 client.16587 populate_metadata read hostname 'smithi046' 2024-03-31T23:17:02.555 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:02.550+0000 7fd6debdf740 1 --2- 172.21.15.46:0/708009866 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55d0f19e1cf0 0x55d0f19e40e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:17:02.555 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:02.550+0000 7fd6debdf740 4 client.16587 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-31T23:17:02.555 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:02.550+0000 7fd6debdf740 1 -- 172.21.15.46:0/708009866 --> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55d0f18cd180 con 0x55d0f19e1cf0 2024-03-31T23:17:02.556 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:02.550+0000 7fd6d94c2640 1 --2- 172.21.15.46:0/708009866 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55d0f19e1cf0 0x55d0f19e40e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:17:02.556 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:02.551+0000 7fd6d94c2640 1 --2- 172.21.15.46:0/708009866 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55d0f19e1cf0 0x55d0f19e40e0 crc :-1 s=READY pgs=122 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=52f313a3dd51f517 server_cookie=e801ca717b20596d in_seq=0 out_seq=0 2024-03-31T23:17:02.556 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:02.551+0000 7fd6bd7fa640 10 client.16587 ms_handle_connect on v2:172.21.15.142:6838/2519143918 2024-03-31T23:17:02.573 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:02.572+0000 7fd6bd7fa640 1 -- 172.21.15.46:0/708009866 <== mds.0 v2:172.21.15.142:6838/2519143918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55d0f18cd180 con 0x55d0f19e1cf0 2024-03-31T23:17:02.573 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:02.572+0000 7fd6bd7fa640 10 client.16587 handle_command_reply: tid=0 2024-03-31T23:17:02.573 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:02.572+0000 7fd6debdf740 1 client.16587 shutdown 2024-03-31T23:17:02.573 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:02.572+0000 7fd6debdf740 1 -- 172.21.15.46:0/708009866 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d0f19626c0 msgr2=0x55d0f1966480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:17:02.573 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:02.572+0000 7fd6debdf740 1 --2- 172.21.15.46:0/708009866 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d0f19626c0 0x55d0f1966480 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7fd6d0000c00 tx=0x7fd6d0009620 comp rx=0 tx=0).stop 2024-03-31T23:17:02.573 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:02.572+0000 7fd6debdf740 1 -- 172.21.15.46:0/708009866 shutdown_connections 2024-03-31T23:17:02.573 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:02.572+0000 7fd6debdf740 1 --2- 172.21.15.46:0/708009866 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55d0f19e1cf0 0x55d0f19e40e0 crc :-1 s=READY pgs=122 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:17:02.573 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:02.572+0000 7fd6debdf740 1 --2- 172.21.15.46:0/708009866 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d0f196bdb0 0x55d0f1965040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:17:02.573 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:02.572+0000 7fd6bd7fa640 0 client.16587 ms_handle_reset on v2:172.21.15.142:6838/2519143918 2024-03-31T23:17:02.573 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:02.572+0000 7fd6debdf740 1 --2- 172.21.15.46:0/708009866 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d0f1962fe0 0x55d0f19669c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:17:02.573 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:02.572+0000 7fd6debdf740 1 --2- 172.21.15.46:0/708009866 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d0f19626c0 0x55d0f1966480 unknown :-1 s=CLOSED pgs=711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:17:02.573 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:02.572+0000 7fd6debdf740 1 -- 172.21.15.46:0/708009866 >> 172.21.15.46:0/708009866 conn(0x55d0f195cfe0 msgr2=0x55d0f19675d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:17:02.573 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:02.572+0000 7fd6debdf740 1 -- 172.21.15.46:0/708009866 shutdown_connections 2024-03-31T23:17:02.574 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:02.572+0000 7fd6debdf740 1 -- 172.21.15.46:0/708009866 wait complete. 2024-03-31T23:17:02.574 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:02.573+0000 7fd6debdf740 20 client.16587 trim_cache size 0 max 16384 2024-03-31T23:17:02.574 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout:{ 2024-03-31T23:17:02.574 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout: "message": "", 2024-03-31T23:17:02.574 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout: "return_code": 0 2024-03-31T23:17:02.574 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout:} 2024-03-31T23:17:02.575 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:02.573+0000 7fd6dbf27640 1 -- 172.21.15.46:0/363501491 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fd6c00a2730 msgr2=0x7fd6c00a4bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:17:02.575 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:02.573+0000 7fd6dbf27640 1 --2- 172.21.15.46:0/363501491 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fd6c00a2730 0x7fd6c00a4bf0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fd6cc005fd0 tx=0x7fd6cc004ab0 comp rx=0 tx=0).stop 2024-03-31T23:17:02.575 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:02.573+0000 7fd6dbf27640 1 -- 172.21.15.46:0/363501491 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd6d4068dd0 msgr2=0x7fd6d4100970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:17:02.575 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:02.573+0000 7fd6dbf27640 1 --2- 172.21.15.46:0/363501491 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd6d4068dd0 0x7fd6d4100970 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7fd6d0005940 tx=0x7fd6d0005dc0 comp rx=0 tx=0).stop 2024-03-31T23:17:02.575 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:02.573+0000 7fd6dbf27640 1 -- 172.21.15.46:0/363501491 shutdown_connections 2024-03-31T23:17:02.575 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:02.573+0000 7fd6dbf27640 1 --2- 172.21.15.46:0/363501491 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fd6c00a2730 0x7fd6c00a4bf0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:17:02.575 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:02.573+0000 7fd6dbf27640 1 --2- 172.21.15.46:0/363501491 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd6d4107820 0x7fd6d41013f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:17:02.575 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:02.573+0000 7fd6dbf27640 1 --2- 172.21.15.46:0/363501491 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd6d4069770 0x7fd6d4100eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:17:02.576 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:02.573+0000 7fd6dbf27640 1 --2- 172.21.15.46:0/363501491 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd6d4068dd0 0x7fd6d4100970 unknown :-1 s=CLOSED pgs=708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:17:02.576 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:02.573+0000 7fd6dbf27640 1 -- 172.21.15.46:0/363501491 >> 172.21.15.46:0/363501491 conn(0x7fd6d40774a0 msgr2=0x7fd6d40fe9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:17:02.576 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:02.573+0000 7fd6dbf27640 1 -- 172.21.15.46:0/363501491 shutdown_connections 2024-03-31T23:17:02.576 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:02.573+0000 7fd6dbf27640 1 -- 172.21.15.46:0/363501491 wait complete. 2024-03-31T23:17:02.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:17:02 smithi142 ceph-mon[35271]: pgmap v162: 129 pgs: 129 active+clean; 77 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 377 KiB/s rd, 1.1 MiB/s wr, 117 op/s 2024-03-31T23:17:03.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:17:02 smithi162 ceph-mon[36237]: pgmap v162: 129 pgs: 129 active+clean; 77 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 377 KiB/s rd, 1.1 MiB/s wr, 117 op/s 2024-03-31T23:17:03.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:17:02 smithi046 ceph-mon[32498]: pgmap v162: 129 pgs: 129 active+clean; 77 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 377 KiB/s rd, 1.1 MiB/s wr, 117 op/s 2024-03-31T23:17:03.197 INFO:tasks.workunit.client.0.smithi046.stdout: 1 113701 2.50 MB/sec execute 129 sec latency 20.435 ms 2024-03-31T23:17:04.198 INFO:tasks.workunit.client.0.smithi046.stdout: 1 113975 2.49 MB/sec execute 130 sec latency 12.740 ms 2024-03-31T23:17:04.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:17:04 smithi142 ceph-mon[35271]: pgmap v163: 129 pgs: 129 active+clean; 77 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 377 KiB/s rd, 1.1 MiB/s wr, 117 op/s 2024-03-31T23:17:05.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:17:04 smithi162 ceph-mon[36237]: pgmap v163: 129 pgs: 129 active+clean; 77 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 377 KiB/s rd, 1.1 MiB/s wr, 117 op/s 2024-03-31T23:17:05.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:17:04 smithi046 ceph-mon[32498]: pgmap v163: 129 pgs: 129 active+clean; 77 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 377 KiB/s rd, 1.1 MiB/s wr, 117 op/s 2024-03-31T23:17:05.198 INFO:tasks.workunit.client.0.smithi046.stdout: 1 114299 2.48 MB/sec execute 131 sec latency 19.229 ms 2024-03-31T23:17:06.198 INFO:tasks.workunit.client.0.smithi046.stdout: 1 114753 2.47 MB/sec execute 132 sec latency 16.797 ms 2024-03-31T23:17:06.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:17:06 smithi142 ceph-mon[35271]: pgmap v164: 129 pgs: 129 active+clean; 77 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 562 KiB/s rd, 1.8 MiB/s wr, 188 op/s 2024-03-31T23:17:07.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:17:06 smithi162 ceph-mon[36237]: pgmap v164: 129 pgs: 129 active+clean; 77 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 562 KiB/s rd, 1.8 MiB/s wr, 188 op/s 2024-03-31T23:17:07.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:17:06 smithi046 ceph-mon[32498]: pgmap v164: 129 pgs: 129 active+clean; 77 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 562 KiB/s rd, 1.8 MiB/s wr, 188 op/s 2024-03-31T23:17:07.198 INFO:tasks.workunit.client.0.smithi046.stdout: 1 115240 2.48 MB/sec execute 133 sec latency 25.497 ms 2024-03-31T23:17:08.198 INFO:tasks.workunit.client.0.smithi046.stdout: 1 115848 2.47 MB/sec execute 134 sec latency 21.507 ms 2024-03-31T23:17:09.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:17:08 smithi162 ceph-mon[36237]: pgmap v165: 129 pgs: 129 active+clean; 77 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 376 KiB/s rd, 1.2 MiB/s wr, 129 op/s 2024-03-31T23:17:09.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:17:08 smithi046 ceph-mon[32498]: pgmap v165: 129 pgs: 129 active+clean; 77 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 376 KiB/s rd, 1.2 MiB/s wr, 129 op/s 2024-03-31T23:17:09.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:17:08 smithi142 ceph-mon[35271]: pgmap v165: 129 pgs: 129 active+clean; 77 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 376 KiB/s rd, 1.2 MiB/s wr, 129 op/s 2024-03-31T23:17:09.198 INFO:tasks.workunit.client.0.smithi046.stdout: 1 116301 2.48 MB/sec execute 135 sec latency 16.030 ms 2024-03-31T23:17:10.198 INFO:tasks.workunit.client.0.smithi046.stdout: 1 116859 2.49 MB/sec execute 136 sec latency 17.048 ms 2024-03-31T23:17:11.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:17:10 smithi162 ceph-mon[36237]: pgmap v166: 129 pgs: 129 active+clean; 78 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 583 KiB/s rd, 1.7 MiB/s wr, 185 op/s 2024-03-31T23:17:11.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:17:10 smithi046 ceph-mon[32498]: pgmap v166: 129 pgs: 129 active+clean; 78 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 583 KiB/s rd, 1.7 MiB/s wr, 185 op/s 2024-03-31T23:17:11.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:17:10 smithi142 ceph-mon[35271]: pgmap v166: 129 pgs: 129 active+clean; 78 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 583 KiB/s rd, 1.7 MiB/s wr, 185 op/s 2024-03-31T23:17:11.198 INFO:tasks.workunit.client.0.smithi046.stdout: 1 117314 2.50 MB/sec execute 137 sec latency 15.456 ms 2024-03-31T23:17:12.198 INFO:tasks.workunit.client.0.smithi046.stdout: 1 117530 2.49 MB/sec execute 138 sec latency 69.620 ms 2024-03-31T23:17:13.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:17:12 smithi162 ceph-mon[36237]: pgmap v167: 129 pgs: 129 active+clean; 78 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 392 KiB/s rd, 1.2 MiB/s wr, 127 op/s 2024-03-31T23:17:13.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:17:12 smithi046 ceph-mon[32498]: pgmap v167: 129 pgs: 129 active+clean; 78 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 392 KiB/s rd, 1.2 MiB/s wr, 127 op/s 2024-03-31T23:17:13.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:17:12 smithi142 ceph-mon[35271]: pgmap v167: 129 pgs: 129 active+clean; 78 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 392 KiB/s rd, 1.2 MiB/s wr, 127 op/s 2024-03-31T23:17:13.198 INFO:tasks.workunit.client.0.smithi046.stdout: 1 117879 2.47 MB/sec execute 139 sec latency 17.256 ms 2024-03-31T23:17:14.198 INFO:tasks.workunit.client.0.smithi046.stdout: 1 118340 2.47 MB/sec execute 140 sec latency 16.654 ms 2024-03-31T23:17:15.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:17:14 smithi162 ceph-mon[36237]: pgmap v168: 129 pgs: 129 active+clean; 78 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 392 KiB/s rd, 1.2 MiB/s wr, 127 op/s 2024-03-31T23:17:15.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:17:14 smithi046 ceph-mon[32498]: pgmap v168: 129 pgs: 129 active+clean; 78 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 392 KiB/s rd, 1.2 MiB/s wr, 127 op/s 2024-03-31T23:17:15.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:17:14 smithi142 ceph-mon[35271]: pgmap v168: 129 pgs: 129 active+clean; 78 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 392 KiB/s rd, 1.2 MiB/s wr, 127 op/s 2024-03-31T23:17:15.198 INFO:tasks.workunit.client.0.smithi046.stdout: 1 118868 2.47 MB/sec execute 141 sec latency 21.676 ms 2024-03-31T23:17:15.668 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:15.665+0000 7f80ead25640 1 -- 172.21.15.46:0/1632147682 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f80e4104160 msgr2=0x7f80e41045c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:17:15.668 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:15.665+0000 7f80ead25640 1 --2- 172.21.15.46:0/1632147682 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f80e4104160 0x7f80e41045c0 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f80dc009ca0 tx=0x7f80dc02c180 comp rx=0 tx=0).stop 2024-03-31T23:17:15.668 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:15.665+0000 7f80ead25640 1 -- 172.21.15.46:0/1632147682 shutdown_connections 2024-03-31T23:17:15.668 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:15.665+0000 7f80ead25640 1 --2- 172.21.15.46:0/1632147682 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f80e4104160 0x7f80e41045c0 unknown :-1 s=CLOSED pgs=712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:17:15.668 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:15.665+0000 7f80ead25640 1 --2- 172.21.15.46:0/1632147682 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f80e4069310 0x7f80e4069750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:17:15.668 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:15.665+0000 7f80ead25640 1 --2- 172.21.15.46:0/1632147682 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f80e4106710 0x7f80e4068dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:17:15.668 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:15.665+0000 7f80ead25640 1 -- 172.21.15.46:0/1632147682 >> 172.21.15.46:0/1632147682 conn(0x7f80e40774a0 msgr2=0x7f80e40778b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:17:15.668 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:15.665+0000 7f80ead25640 1 -- 172.21.15.46:0/1632147682 shutdown_connections 2024-03-31T23:17:15.669 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:15.665+0000 7f80ead25640 1 -- 172.21.15.46:0/1632147682 wait complete. 2024-03-31T23:17:15.669 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:15.665+0000 7f80ead25640 1 Processor -- start 2024-03-31T23:17:15.669 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:15.665+0000 7f80ead25640 1 -- start start 2024-03-31T23:17:15.669 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:15.666+0000 7f80ead25640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f80e4069310 0x7f80e419e2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:17:15.669 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:15.666+0000 7f80ead25640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f80e4104160 0x7f80e419e820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:17:15.669 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:15.666+0000 7f80ead25640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f80e4106710 0x7f80e41a2ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:17:15.669 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:15.666+0000 7f80ead25640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f80e4117f00 con 0x7f80e4069310 2024-03-31T23:17:15.669 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:15.666+0000 7f80ead25640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f80e4117d80 con 0x7f80e4106710 2024-03-31T23:17:15.669 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:15.666+0000 7f80ead25640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f80e4118080 con 0x7f80e4104160 2024-03-31T23:17:15.669 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:15.666+0000 7f80e8ac1640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f80e4069310 0x7f80e419e2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:17:15.669 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:15.666+0000 7f80e8ac1640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f80e4069310 0x7f80e419e2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:35080/0 (socket says 172.21.15.46:35080) 2024-03-31T23:17:15.669 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:15.666+0000 7f80e8ac1640 1 -- 172.21.15.46:0/1178994562 learned_addr learned my addr 172.21.15.46:0/1178994562 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:17:15.669 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:15.666+0000 7f80e3fff640 1 --2- 172.21.15.46:0/1178994562 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f80e4104160 0x7f80e419e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:17:15.669 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:15.666+0000 7f80e92c2640 1 --2- 172.21.15.46:0/1178994562 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f80e4106710 0x7f80e41a2ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:17:15.669 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:15.666+0000 7f80e8ac1640 1 -- 172.21.15.46:0/1178994562 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f80e4104160 msgr2=0x7f80e419e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:17:15.670 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:15.666+0000 7f80e8ac1640 1 --2- 172.21.15.46:0/1178994562 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f80e4104160 0x7f80e419e820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:17:15.670 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:15.666+0000 7f80e8ac1640 1 -- 172.21.15.46:0/1178994562 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f80e4106710 msgr2=0x7f80e41a2ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:17:15.670 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:15.666+0000 7f80e8ac1640 1 --2- 172.21.15.46:0/1178994562 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f80e4106710 0x7f80e41a2ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:17:15.670 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:15.666+0000 7f80e8ac1640 1 -- 172.21.15.46:0/1178994562 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f80e41a32d0 con 0x7f80e4069310 2024-03-31T23:17:15.670 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:15.666+0000 7f80e8ac1640 1 --2- 172.21.15.46:0/1178994562 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f80e4069310 0x7f80e419e2e0 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7f80d800e9f0 tx=0x7f80d800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:17:15.670 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:15.666+0000 7f80e1ffb640 1 -- 172.21.15.46:0/1178994562 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80d800cd80 con 0x7f80e4069310 2024-03-31T23:17:15.670 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:15.666+0000 7f80e1ffb640 1 -- 172.21.15.46:0/1178994562 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f80d80108e0 con 0x7f80e4069310 2024-03-31T23:17:15.670 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:15.666+0000 7f80ead25640 1 -- 172.21.15.46:0/1178994562 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f80e41a35c0 con 0x7f80e4069310 2024-03-31T23:17:15.670 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:15.666+0000 7f80e1ffb640 1 -- 172.21.15.46:0/1178994562 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80d8018920 con 0x7f80e4069310 2024-03-31T23:17:15.670 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:15.667+0000 7f80ead25640 1 -- 172.21.15.46:0/1178994562 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f80e41aae50 con 0x7f80e4069310 2024-03-31T23:17:15.672 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:15.669+0000 7f80e1ffb640 1 -- 172.21.15.46:0/1178994562 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 25) ==== 137451+0+0 (secure 0 0 0) 0x7f80d8010a90 con 0x7f80e4069310 2024-03-31T23:17:15.672 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:15.669+0000 7f80ed9e9740 1 Processor -- start 2024-03-31T23:17:15.672 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:15.669+0000 7f80ed9e9740 1 -- start start 2024-03-31T23:17:15.672 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:15.670+0000 7f80ed9e9740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559ce37e46c0 0x559ce37e4aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:17:15.672 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:15.670+0000 7f80ed9e9740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559ce37e4fe0 0x559ce37ed870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:17:15.672 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:15.670+0000 7f80ed9e9740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559ce37eddb0 0x559ce37f01a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:17:15.672 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:15.670+0000 7f80ed9e9740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x559ce37478f0 con 0x559ce37e46c0 2024-03-31T23:17:15.673 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:15.670+0000 7f80ed9e9740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x559ce36f08d0 con 0x559ce37eddb0 2024-03-31T23:17:15.673 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:15.670+0000 7f80ed9e9740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x559ce3743990 con 0x559ce37e4fe0 2024-03-31T23:17:15.673 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:15.670+0000 7f80e1ffb640 1 --2- 172.21.15.46:0/1178994562 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f80d40a26b0 0x7f80d40a4b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:17:15.673 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:15.670+0000 7f80e8ac1640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559ce37e4fe0 0x559ce37ed870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:17:15.673 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:15.670+0000 7f80e3fff640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559ce37eddb0 0x559ce37f01a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:17:15.673 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:15.670+0000 7f80e92c2640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559ce37e46c0 0x559ce37e4aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:17:15.673 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:15.670+0000 7f80e3fff640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559ce37eddb0 0x559ce37f01a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.46:57708/0 (socket says 172.21.15.46:57708) 2024-03-31T23:17:15.673 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:15.670+0000 7f80e3fff640 1 -- 172.21.15.46:0/2980127946 learned_addr learned my addr 172.21.15.46:0/2980127946 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:17:15.673 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:15.670+0000 7f80e92c2640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559ce37e46c0 0x559ce37e4aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:35090/0 (socket says 172.21.15.46:35090) 2024-03-31T23:17:15.673 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:15.670+0000 7f80e1ffb640 1 -- 172.21.15.46:0/1178994562 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7f80d80cf1b0 con 0x7f80e4069310 2024-03-31T23:17:15.673 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:15.670+0000 7f80e92c2640 1 --2- 172.21.15.46:0/1178994562 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f80d40a26b0 0x7f80d40a4b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:17:15.673 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:15.670+0000 7f80e3fff640 1 -- 172.21.15.46:0/2980127946 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559ce37e4fe0 msgr2=0x559ce37ed870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:17:15.673 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:15.670+0000 7f80e3fff640 1 --2- 172.21.15.46:0/2980127946 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559ce37e4fe0 0x559ce37ed870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:17:15.673 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:15.670+0000 7f80e3fff640 1 -- 172.21.15.46:0/2980127946 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559ce37e46c0 msgr2=0x559ce37e4aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:17:15.674 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:15.670+0000 7f80e3fff640 1 --2- 172.21.15.46:0/2980127946 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559ce37e46c0 0x559ce37e4aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:17:15.674 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:15.670+0000 7f80e3fff640 1 -- 172.21.15.46:0/2980127946 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559ce345c210 con 0x559ce37eddb0 2024-03-31T23:17:15.674 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:15.670+0000 7f80e3fff640 1 --2- 172.21.15.46:0/2980127946 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559ce37eddb0 0x559ce37f01a0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f80d0009c90 tx=0x7f80d0011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2577738074a2ce60 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:17:15.674 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:15.671+0000 7f80e92c2640 1 --2- 172.21.15.46:0/1178994562 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f80d40a26b0 0x7f80d40a4b70 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f80dc004af0 tx=0x7f80dc005ef0 comp rx=0 tx=0).ready entity=mgr.15654 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:17:15.674 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:15.671+0000 7f80ceffd640 1 -- 172.21.15.46:0/2980127946 <== mon.2 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80d001e070 con 0x559ce37eddb0 2024-03-31T23:17:15.674 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:15.671+0000 7f80ceffd640 1 -- 172.21.15.46:0/2980127946 <== mon.2 v2:172.21.15.142:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f80d0019ae0 con 0x559ce37eddb0 2024-03-31T23:17:15.674 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:15.671+0000 7f80ceffd640 1 -- 172.21.15.46:0/2980127946 <== mon.2 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80d000e8a0 con 0x559ce37eddb0 2024-03-31T23:17:15.677 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:15.671+0000 7f80ed9e9740 1 -- 172.21.15.46:0/2980127946 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559ce37eddb0 msgr2=0x559ce37f01a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:17:15.677 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:15.671+0000 7f80ed9e9740 1 --2- 172.21.15.46:0/2980127946 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559ce37eddb0 0x559ce37f01a0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f80d0009c90 tx=0x7f80d0011810 comp rx=0 tx=0).stop 2024-03-31T23:17:15.678 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:15.672+0000 7f80ed9e9740 1 -- 172.21.15.46:0/2980127946 shutdown_connections 2024-03-31T23:17:15.678 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:15.672+0000 7f80ed9e9740 1 --2- 172.21.15.46:0/2980127946 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559ce37eddb0 0x559ce37f01a0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:17:15.678 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:15.672+0000 7f80ed9e9740 1 --2- 172.21.15.46:0/2980127946 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559ce37e4fe0 0x559ce37ed870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:17:15.678 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:15.672+0000 7f80ed9e9740 1 --2- 172.21.15.46:0/2980127946 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559ce37e46c0 0x559ce37e4aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:17:15.678 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:15.672+0000 7f80ed9e9740 1 -- 172.21.15.46:0/2980127946 >> 172.21.15.46:0/2980127946 conn(0x559ce37defe0 msgr2=0x559ce37df3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:17:15.678 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:15.672+0000 7f80ed9e9740 1 -- 172.21.15.46:0/2980127946 shutdown_connections 2024-03-31T23:17:15.678 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:15.672+0000 7f80ed9e9740 1 -- 172.21.15.46:0/2980127946 wait complete. 2024-03-31T23:17:15.678 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:15.672+0000 7f80ed9e9740 1 -- start start 2024-03-31T23:17:15.678 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:15.672+0000 7f80ed9e9740 1 Processor -- start 2024-03-31T23:17:15.678 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:15.672+0000 7f80ed9e9740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559ce37e46c0 0x559ce37ead80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:17:15.678 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:15.672+0000 7f80ed9e9740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559ce37e4fe0 0x559ce37eb2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:17:15.678 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:15.673+0000 7f80ed9e9740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559ce37eddb0 0x559ce37eb800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:17:15.678 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:15.673+0000 7f80ed9e9740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x559ce37f09e0 con 0x559ce37e4fe0 2024-03-31T23:17:15.679 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:15.673+0000 7f80ed9e9740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x559ce36e5f80 con 0x559ce37e46c0 2024-03-31T23:17:15.679 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:15.673+0000 7f80ed9e9740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x559ce37f0b60 con 0x559ce37eddb0 2024-03-31T23:17:15.679 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:15.673+0000 7f80ed9e9740 10 client.0 _ll_register_callbacks cb 0x559ce373e290 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T23:17:15.679 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:15.673+0000 7f80e3fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559ce37e4fe0 0x559ce37eb2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:17:15.679 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:15.673+0000 7f80e92c2640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559ce37eddb0 0x559ce37eb800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:17:15.679 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:15.673+0000 7f80e3fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559ce37e4fe0 0x559ce37eb2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:35098/0 (socket says 172.21.15.46:35098) 2024-03-31T23:17:15.679 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:15.673+0000 7f80e3fff640 1 -- 172.21.15.46:0/2491773652 learned_addr learned my addr 172.21.15.46:0/2491773652 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:17:15.679 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:15.673+0000 7f80e8ac1640 1 --2- 172.21.15.46:0/2491773652 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559ce37e46c0 0x559ce37ead80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:17:15.679 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:15.673+0000 7f80e3fff640 1 -- 172.21.15.46:0/2491773652 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559ce37eddb0 msgr2=0x559ce37eb800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:17:15.679 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:15.673+0000 7f80e3fff640 1 --2- 172.21.15.46:0/2491773652 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559ce37eddb0 0x559ce37eb800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:17:15.679 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:15.673+0000 7f80e3fff640 1 -- 172.21.15.46:0/2491773652 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559ce37e46c0 msgr2=0x559ce37ead80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:17:15.679 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:15.673+0000 7f80e3fff640 1 --2- 172.21.15.46:0/2491773652 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559ce37e46c0 0x559ce37ead80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:17:15.679 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:15.673+0000 7f80e3fff640 1 -- 172.21.15.46:0/2491773652 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559ce3528620 con 0x559ce37e4fe0 2024-03-31T23:17:15.679 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:15.673+0000 7f80e8ac1640 1 --2- 172.21.15.46:0/2491773652 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559ce37e46c0 0x559ce37ead80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:17:15.680 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:15.673+0000 7f80e92c2640 1 --2- 172.21.15.46:0/2491773652 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559ce37eddb0 0x559ce37eb800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:17:15.680 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:15.673+0000 7f80e3fff640 1 --2- 172.21.15.46:0/2491773652 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559ce37e4fe0 0x559ce37eb2c0 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f80d0019ab0 tx=0x7f80d000e3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ccdaf4334244125c server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:17:15.680 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:15.673+0000 7f80c3fff640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-31T23:17:15.680 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:15.673+0000 7f80c3fff640 1 -- 172.21.15.46:0/2491773652 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80d001e070 con 0x559ce37e4fe0 2024-03-31T23:17:15.680 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:15.674+0000 7f80ed9e9740 1 -- 172.21.15.46:0/2491773652 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f80d0019ae0 con 0x559ce37e4fe0 2024-03-31T23:17:15.680 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:15.674+0000 7f80c3fff640 1 -- 172.21.15.46:0/2491773652 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7f80d0005460 con 0x559ce37e4fe0 2024-03-31T23:17:15.680 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:15.674+0000 7f80c3fff640 1 -- 172.21.15.46:0/2491773652 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f80d0031050 con 0x559ce37e4fe0 2024-03-31T23:17:15.680 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:15.674+0000 7f80ed9e9740 10 client.16599 fetch_fsmap learned FSMap version 30 2024-03-31T23:17:15.680 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:15.674+0000 7f80ed9e9740 1 -- 172.21.15.46:0/2491773652 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x559ce37fc850 con 0x559ce37e4fe0 2024-03-31T23:17:15.680 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:15.674+0000 7f80c3fff640 1 -- 172.21.15.46:0/2491773652 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f80d0019ae0 con 0x559ce37e4fe0 2024-03-31T23:17:15.680 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:15.674+0000 7f80c3fff640 1 -- 172.21.15.46:0/2491773652 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 30) ==== 8346+0+0 (secure 0 0 0) 0x7f80d00035f0 con 0x559ce37e4fe0 2024-03-31T23:17:15.680 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:15.674+0000 7f80ed9e9740 10 client.16599 fetch_fsmap finished waiting for FSMap version 30 2024-03-31T23:17:15.680 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:15.674+0000 7f80ed9e9740 10 client.16599 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-31T23:17:15.680 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:15.674+0000 7f80ed9e9740 20 client.16599 populate_metadata read hostname 'smithi046' 2024-03-31T23:17:15.680 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:15.674+0000 7f80ed9e9740 1 --2- 172.21.15.46:0/2491773652 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x559ce37fdc60 0x559ce3800050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:17:15.681 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:15.674+0000 7f80ed9e9740 4 client.16599 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-31T23:17:15.681 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:15.674+0000 7f80ed9e9740 1 -- 172.21.15.46:0/2491773652 --> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559ce375b120 con 0x559ce37fdc60 2024-03-31T23:17:15.681 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:15.674+0000 7f80e8ac1640 1 --2- 172.21.15.46:0/2491773652 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x559ce37fdc60 0x559ce3800050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:17:15.681 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:15.675+0000 7f80c3fff640 10 client.16599 ms_handle_connect on v2:172.21.15.142:6838/2519143918 2024-03-31T23:17:15.681 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:15.675+0000 7f80e8ac1640 1 --2- 172.21.15.46:0/2491773652 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x559ce37fdc60 0x559ce3800050 crc :-1 s=READY pgs=123 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=731bb5f9f4234695 server_cookie=a7ebd68fb8fa2d6a in_seq=0 out_seq=0 2024-03-31T23:17:15.681 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:15.676+0000 7f80c3fff640 1 -- 172.21.15.46:0/2491773652 <== mds.0 v2:172.21.15.142:6838/2519143918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x559ce375b120 con 0x559ce37fdc60 2024-03-31T23:17:15.681 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:15.676+0000 7f80c3fff640 10 client.16599 handle_command_reply: tid=0 2024-03-31T23:17:15.681 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:15.677+0000 7f80ed9e9740 1 client.16599 shutdown 2024-03-31T23:17:15.682 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:15.677+0000 7f80ed9e9740 1 -- 172.21.15.46:0/2491773652 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559ce37e4fe0 msgr2=0x559ce37eb2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:17:15.682 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:15.677+0000 7f80ed9e9740 1 --2- 172.21.15.46:0/2491773652 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559ce37e4fe0 0x559ce37eb2c0 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f80d0019ab0 tx=0x7f80d000e3b0 comp rx=0 tx=0).stop 2024-03-31T23:17:15.682 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:15.677+0000 7f80ed9e9740 1 -- 172.21.15.46:0/2491773652 shutdown_connections 2024-03-31T23:17:15.682 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:15.677+0000 7f80ed9e9740 1 --2- 172.21.15.46:0/2491773652 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x559ce37fdc60 0x559ce3800050 crc :-1 s=READY pgs=123 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:17:15.682 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:15.677+0000 7f80ed9e9740 1 --2- 172.21.15.46:0/2491773652 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559ce37eddb0 0x559ce37eb800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:17:15.682 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:15.677+0000 7f80c3fff640 0 client.16599 ms_handle_reset on v2:172.21.15.142:6838/2519143918 2024-03-31T23:17:15.682 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:15.677+0000 7f80ed9e9740 1 --2- 172.21.15.46:0/2491773652 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559ce37e4fe0 0x559ce37eb2c0 unknown :-1 s=CLOSED pgs=714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:17:15.682 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:15.677+0000 7f80ed9e9740 1 --2- 172.21.15.46:0/2491773652 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559ce37e46c0 0x559ce37ead80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:17:15.682 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:15.677+0000 7f80ed9e9740 1 -- 172.21.15.46:0/2491773652 >> 172.21.15.46:0/2491773652 conn(0x559ce37defe0 msgr2=0x559ce37e0d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:17:15.682 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:15.677+0000 7f80ed9e9740 1 -- 172.21.15.46:0/2491773652 shutdown_connections 2024-03-31T23:17:15.682 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:15.677+0000 7f80ed9e9740 1 -- 172.21.15.46:0/2491773652 wait complete. 2024-03-31T23:17:15.683 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:15.678+0000 7f80ed9e9740 20 client.16599 trim_cache size 0 max 16384 2024-03-31T23:17:15.700 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:15.698+0000 7f80ed9e9740 1 Processor -- start 2024-03-31T23:17:15.701 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:15.698+0000 7f80ed9e9740 1 -- start start 2024-03-31T23:17:15.701 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:15.699+0000 7f80ed9e9740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559ce37e46c0 0x559ce37ed4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:17:15.701 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:15.699+0000 7f80ed9e9740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559ce37e4fe0 0x559ce37f9780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:17:15.701 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:15.699+0000 7f80ed9e9740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559ce37eddb0 0x559ce3805810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:17:15.701 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:15.699+0000 7f80ed9e9740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f80e41b78e0 con 0x559ce37e4fe0 2024-03-31T23:17:15.701 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:15.699+0000 7f80ed9e9740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f80e41b51b0 con 0x559ce37e46c0 2024-03-31T23:17:15.701 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:15.699+0000 7f80ed9e9740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f80e41b3000 con 0x559ce37eddb0 2024-03-31T23:17:15.701 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:15.699+0000 7f80e3fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559ce37e4fe0 0x559ce37f9780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:17:15.701 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:15.699+0000 7f80e8ac1640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559ce37e46c0 0x559ce37ed4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:17:15.701 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:15.699+0000 7f80e3fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559ce37e4fe0 0x559ce37f9780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:35102/0 (socket says 172.21.15.46:35102) 2024-03-31T23:17:15.701 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:15.699+0000 7f80e3fff640 1 -- 172.21.15.46:0/3388901194 learned_addr learned my addr 172.21.15.46:0/3388901194 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:17:15.701 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:15.699+0000 7f80e92c2640 1 --2- 172.21.15.46:0/3388901194 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559ce37eddb0 0x559ce3805810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:17:15.701 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:15.699+0000 7f80e3fff640 1 -- 172.21.15.46:0/3388901194 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559ce37eddb0 msgr2=0x559ce3805810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:17:15.702 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:15.699+0000 7f80e3fff640 1 --2- 172.21.15.46:0/3388901194 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559ce37eddb0 0x559ce3805810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:17:15.702 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:15.699+0000 7f80e3fff640 1 -- 172.21.15.46:0/3388901194 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559ce37e46c0 msgr2=0x559ce37ed4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:17:15.702 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:15.699+0000 7f80e3fff640 1 --2- 172.21.15.46:0/3388901194 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559ce37e46c0 0x559ce37ed4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:17:15.702 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:15.699+0000 7f80e3fff640 1 -- 172.21.15.46:0/3388901194 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559ce37e5670 con 0x559ce37e4fe0 2024-03-31T23:17:15.702 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:15.699+0000 7f80e3fff640 1 --2- 172.21.15.46:0/3388901194 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559ce37e4fe0 0x559ce37f9780 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7f80d001c040 tx=0x7f80d00197f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c4fdf35764f136b3 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:17:15.702 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:15.701+0000 7f80ce7fc640 1 -- 172.21.15.46:0/3388901194 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80d001e040 con 0x559ce37e4fe0 2024-03-31T23:17:15.702 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:15.701+0000 7f80ce7fc640 1 -- 172.21.15.46:0/3388901194 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f80d001f040 con 0x559ce37e4fe0 2024-03-31T23:17:15.703 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:15.702+0000 7f80ce7fc640 1 -- 172.21.15.46:0/3388901194 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80d000e720 con 0x559ce37e4fe0 2024-03-31T23:17:15.704 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:15.702+0000 7f80ed9e9740 1 -- 172.21.15.46:0/3388901194 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559ce37e4fe0 msgr2=0x559ce37f9780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:17:15.704 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:15.702+0000 7f80ed9e9740 1 --2- 172.21.15.46:0/3388901194 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559ce37e4fe0 0x559ce37f9780 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7f80d001c040 tx=0x7f80d00197f0 comp rx=0 tx=0).stop 2024-03-31T23:17:15.704 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:15.702+0000 7f80ed9e9740 1 -- 172.21.15.46:0/3388901194 shutdown_connections 2024-03-31T23:17:15.704 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:15.702+0000 7f80ed9e9740 1 --2- 172.21.15.46:0/3388901194 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559ce37eddb0 0x559ce3805810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:17:15.704 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:15.702+0000 7f80ed9e9740 1 --2- 172.21.15.46:0/3388901194 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559ce37e4fe0 0x559ce37f9780 unknown :-1 s=CLOSED pgs=715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:17:15.704 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:15.702+0000 7f80ed9e9740 1 --2- 172.21.15.46:0/3388901194 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559ce37e46c0 0x559ce37ed4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:17:15.704 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:15.702+0000 7f80ed9e9740 1 -- 172.21.15.46:0/3388901194 >> 172.21.15.46:0/3388901194 conn(0x559ce37defe0 msgr2=0x559ce37df3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:17:15.704 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:15.702+0000 7f80ed9e9740 1 -- 172.21.15.46:0/3388901194 shutdown_connections 2024-03-31T23:17:15.704 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:15.702+0000 7f80ed9e9740 1 -- 172.21.15.46:0/3388901194 wait complete. 2024-03-31T23:17:15.704 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:15.703+0000 7f80ed9e9740 1 -- start start 2024-03-31T23:17:15.704 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:15.703+0000 7f80ed9e9740 1 Processor -- start 2024-03-31T23:17:15.707 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:15.703+0000 7f80ed9e9740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559ce37e46c0 0x559ce37f0f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:17:15.707 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:15.703+0000 7f80ed9e9740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559ce37e4fe0 0x559ce37f14a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:17:15.707 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:15.703+0000 7f80ed9e9740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559ce37eddb0 0x559ce3861e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:17:15.707 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:15.703+0000 7f80ed9e9740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f80e41b3670 con 0x559ce37e4fe0 2024-03-31T23:17:15.707 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:15.703+0000 7f80ed9e9740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f80e41b4ae0 con 0x559ce37e46c0 2024-03-31T23:17:15.707 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:15.703+0000 7f80ed9e9740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f80e41b3d40 con 0x559ce37eddb0 2024-03-31T23:17:15.707 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:15.703+0000 7f80ed9e9740 10 client.0 _ll_register_callbacks cb 0x7f80d001c070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T23:17:15.707 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:15.703+0000 7f80e3fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559ce37e4fe0 0x559ce37f14a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:17:15.707 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:15.703+0000 7f80e3fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559ce37e4fe0 0x559ce37f14a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:35118/0 (socket says 172.21.15.46:35118) 2024-03-31T23:17:15.708 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:15.703+0000 7f80e3fff640 1 -- 172.21.15.46:0/623340534 learned_addr learned my addr 172.21.15.46:0/623340534 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:17:15.708 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:15.703+0000 7f80e8ac1640 1 --2- 172.21.15.46:0/623340534 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559ce37e46c0 0x559ce37f0f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:17:15.708 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:15.703+0000 7f80e92c2640 1 --2- 172.21.15.46:0/623340534 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559ce37eddb0 0x559ce3861e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:17:15.708 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:15.704+0000 7f80e3fff640 1 -- 172.21.15.46:0/623340534 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559ce37eddb0 msgr2=0x559ce3861e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:17:15.708 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:15.704+0000 7f80e3fff640 1 --2- 172.21.15.46:0/623340534 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559ce37eddb0 0x559ce3861e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:17:15.708 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:15.704+0000 7f80e3fff640 1 -- 172.21.15.46:0/623340534 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559ce37e46c0 msgr2=0x559ce37f0f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:17:15.708 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:15.704+0000 7f80e3fff640 1 --2- 172.21.15.46:0/623340534 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559ce37e46c0 0x559ce37f0f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:17:15.708 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:15.704+0000 7f80e3fff640 1 -- 172.21.15.46:0/623340534 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559ce38623f0 con 0x559ce37e4fe0 2024-03-31T23:17:15.708 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:15.704+0000 7f80e3fff640 1 --2- 172.21.15.46:0/623340534 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559ce37e4fe0 0x559ce37f14a0 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7f80d0003b80 tx=0x7f80d0005d30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8cce80b65fb4dc server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:17:15.708 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:15.704+0000 7f80c3fff640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-31T23:17:15.708 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:15.704+0000 7f80c3fff640 1 -- 172.21.15.46:0/623340534 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80d001e070 con 0x559ce37e4fe0 2024-03-31T23:17:15.708 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:15.704+0000 7f80c3fff640 1 -- 172.21.15.46:0/623340534 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7f80d0019a90 con 0x559ce37e4fe0 2024-03-31T23:17:15.708 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:15.704+0000 7f80ed9e9740 1 -- 172.21.15.46:0/623340534 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f80d001f040 con 0x559ce37e4fe0 2024-03-31T23:17:15.708 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:15.704+0000 7f80c3fff640 1 -- 172.21.15.46:0/623340534 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f80d0045050 con 0x559ce37e4fe0 2024-03-31T23:17:15.708 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:15.704+0000 7f80ed9e9740 10 client.16605 fetch_fsmap learned FSMap version 30 2024-03-31T23:17:15.709 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:15.704+0000 7f80ed9e9740 1 -- 172.21.15.46:0/623340534 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x559ce38636d0 con 0x559ce37e4fe0 2024-03-31T23:17:15.709 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:15.704+0000 7f80c3fff640 1 -- 172.21.15.46:0/623340534 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f80d001f040 con 0x559ce37e4fe0 2024-03-31T23:17:15.709 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:15.704+0000 7f80c3fff640 1 -- 172.21.15.46:0/623340534 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 30) ==== 8346+0+0 (secure 0 0 0) 0x7f80d000ebb0 con 0x559ce37e4fe0 2024-03-31T23:17:15.709 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:15.705+0000 7f80ed9e9740 10 client.16605 fetch_fsmap finished waiting for FSMap version 30 2024-03-31T23:17:15.709 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:15.705+0000 7f80ed9e9740 10 client.16605 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-31T23:17:15.709 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:15.705+0000 7f80ed9e9740 20 client.16605 populate_metadata read hostname 'smithi046' 2024-03-31T23:17:15.709 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:15.705+0000 7f80ed9e9740 1 --2- 172.21.15.46:0/623340534 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x559ce37fdc60 0x559ce3865e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:17:15.709 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:15.705+0000 7f80ed9e9740 4 client.16605 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-31T23:17:15.709 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:15.705+0000 7f80ed9e9740 1 -- 172.21.15.46:0/623340534 --> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] -- command(tid 0: {"prefix": "flush journal"}) -- 0x559ce3754e30 con 0x559ce37fdc60 2024-03-31T23:17:15.709 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:15.705+0000 7f80e8ac1640 1 --2- 172.21.15.46:0/623340534 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x559ce37fdc60 0x559ce3865e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:17:15.709 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:15.705+0000 7f80e8ac1640 1 --2- 172.21.15.46:0/623340534 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x559ce37fdc60 0x559ce3865e10 crc :-1 s=READY pgs=124 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9688cbf303eab332 server_cookie=fd6653dd9c82ae3c in_seq=0 out_seq=0 2024-03-31T23:17:15.709 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:15.705+0000 7f80c3fff640 10 client.16605 ms_handle_connect on v2:172.21.15.142:6838/2519143918 2024-03-31T23:17:15.733 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:15.730+0000 7f80c3fff640 1 -- 172.21.15.46:0/623340534 <== mds.0 v2:172.21.15.142:6838/2519143918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x559ce3754e30 con 0x559ce37fdc60 2024-03-31T23:17:15.733 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:15.730+0000 7f80c3fff640 10 client.16605 handle_command_reply: tid=0 2024-03-31T23:17:15.733 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:15.730+0000 7f80ed9e9740 1 client.16605 shutdown 2024-03-31T23:17:15.733 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:15.730+0000 7f80ed9e9740 1 -- 172.21.15.46:0/623340534 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559ce37e4fe0 msgr2=0x559ce37f14a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:17:15.734 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:15.730+0000 7f80ed9e9740 1 --2- 172.21.15.46:0/623340534 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559ce37e4fe0 0x559ce37f14a0 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7f80d0003b80 tx=0x7f80d0005d30 comp rx=0 tx=0).stop 2024-03-31T23:17:15.734 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:15.730+0000 7f80ed9e9740 1 -- 172.21.15.46:0/623340534 shutdown_connections 2024-03-31T23:17:15.734 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:15.730+0000 7f80ed9e9740 1 --2- 172.21.15.46:0/623340534 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x559ce37fdc60 0x559ce3865e10 crc :-1 s=READY pgs=124 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:17:15.734 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:15.730+0000 7f80ed9e9740 1 --2- 172.21.15.46:0/623340534 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559ce37eddb0 0x559ce3861e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:17:15.734 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:15.730+0000 7f80ed9e9740 1 --2- 172.21.15.46:0/623340534 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559ce37e4fe0 0x559ce37f14a0 unknown :-1 s=CLOSED pgs=716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:17:15.734 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:15.730+0000 7f80ed9e9740 1 --2- 172.21.15.46:0/623340534 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559ce37e46c0 0x559ce37f0f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:17:15.734 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:15.730+0000 7f80c3fff640 0 client.16605 ms_handle_reset on v2:172.21.15.142:6838/2519143918 2024-03-31T23:17:15.734 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:15.730+0000 7f80ed9e9740 1 -- 172.21.15.46:0/623340534 >> 172.21.15.46:0/623340534 conn(0x559ce37defe0 msgr2=0x559ce37e9590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:17:15.734 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:15.730+0000 7f80ed9e9740 1 -- 172.21.15.46:0/623340534 shutdown_connections 2024-03-31T23:17:15.734 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:15.730+0000 7f80ed9e9740 1 -- 172.21.15.46:0/623340534 wait complete. 2024-03-31T23:17:15.734 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:15.731+0000 7f80ed9e9740 20 client.16605 trim_cache size 0 max 16384 2024-03-31T23:17:15.735 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout:{ 2024-03-31T23:17:15.735 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout: "message": "", 2024-03-31T23:17:15.735 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout: "return_code": 0 2024-03-31T23:17:15.735 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout:} 2024-03-31T23:17:15.735 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:15.731+0000 7f80ead25640 1 -- 172.21.15.46:0/1178994562 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f80d40a26b0 msgr2=0x7f80d40a4b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:17:15.736 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:15.731+0000 7f80ead25640 1 --2- 172.21.15.46:0/1178994562 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f80d40a26b0 0x7f80d40a4b70 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f80dc004af0 tx=0x7f80dc005ef0 comp rx=0 tx=0).stop 2024-03-31T23:17:15.736 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:15.731+0000 7f80ead25640 1 -- 172.21.15.46:0/1178994562 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f80e4069310 msgr2=0x7f80e419e2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:17:15.736 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:15.731+0000 7f80ead25640 1 --2- 172.21.15.46:0/1178994562 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f80e4069310 0x7f80e419e2e0 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7f80d800e9f0 tx=0x7f80d800eeb0 comp rx=0 tx=0).stop 2024-03-31T23:17:15.736 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:15.731+0000 7f80ead25640 1 -- 172.21.15.46:0/1178994562 shutdown_connections 2024-03-31T23:17:15.736 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:15.731+0000 7f80ead25640 1 --2- 172.21.15.46:0/1178994562 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f80d40a26b0 0x7f80d40a4b70 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:17:15.736 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:15.731+0000 7f80ead25640 1 --2- 172.21.15.46:0/1178994562 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f80e4106710 0x7f80e41a2ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:17:15.736 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:15.731+0000 7f80ead25640 1 --2- 172.21.15.46:0/1178994562 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f80e4104160 0x7f80e419e820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:17:15.736 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:15.731+0000 7f80ead25640 1 --2- 172.21.15.46:0/1178994562 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f80e4069310 0x7f80e419e2e0 unknown :-1 s=CLOSED pgs=713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:17:15.736 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:15.731+0000 7f80ead25640 1 -- 172.21.15.46:0/1178994562 >> 172.21.15.46:0/1178994562 conn(0x7f80e40774a0 msgr2=0x7f80e40ff890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:17:15.736 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:15.731+0000 7f80ead25640 1 -- 172.21.15.46:0/1178994562 shutdown_connections 2024-03-31T23:17:15.736 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:15.731+0000 7f80ead25640 1 -- 172.21.15.46:0/1178994562 wait complete. 2024-03-31T23:17:16.198 INFO:tasks.workunit.client.0.smithi046.stdout: 1 119421 2.47 MB/sec execute 142 sec latency 23.683 ms 2024-03-31T23:17:17.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:17:16 smithi162 ceph-mon[36237]: pgmap v169: 129 pgs: 129 active+clean; 83 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 595 KiB/s rd, 1.8 MiB/s wr, 183 op/s 2024-03-31T23:17:17.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:17:16 smithi046 ceph-mon[32498]: pgmap v169: 129 pgs: 129 active+clean; 83 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 595 KiB/s rd, 1.8 MiB/s wr, 183 op/s 2024-03-31T23:17:17.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:17:16 smithi142 ceph-mon[35271]: pgmap v169: 129 pgs: 129 active+clean; 83 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 595 KiB/s rd, 1.8 MiB/s wr, 183 op/s 2024-03-31T23:17:17.198 INFO:tasks.workunit.client.0.smithi046.stdout: 1 119985 2.48 MB/sec execute 143 sec latency 17.337 ms 2024-03-31T23:17:18.198 INFO:tasks.workunit.client.0.smithi046.stdout: 1 120421 2.49 MB/sec execute 144 sec latency 21.076 ms 2024-03-31T23:17:18.995 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:17:18 smithi142 ceph-mon[35271]: pgmap v170: 129 pgs: 129 active+clean; 83 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 410 KiB/s rd, 1.1 MiB/s wr, 112 op/s 2024-03-31T23:17:19.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:17:18 smithi162 ceph-mon[36237]: pgmap v170: 129 pgs: 129 active+clean; 83 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 410 KiB/s rd, 1.1 MiB/s wr, 112 op/s 2024-03-31T23:17:19.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:17:18 smithi046 ceph-mon[32498]: pgmap v170: 129 pgs: 129 active+clean; 83 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 410 KiB/s rd, 1.1 MiB/s wr, 112 op/s 2024-03-31T23:17:19.198 INFO:tasks.workunit.client.0.smithi046.stdout: 1 120892 2.50 MB/sec execute 145 sec latency 16.505 ms 2024-03-31T23:17:20.198 INFO:tasks.workunit.client.0.smithi046.stdout: 1 121166 2.48 MB/sec execute 146 sec latency 13.694 ms 2024-03-31T23:17:21.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:17:20 smithi162 ceph-mon[36237]: pgmap v171: 129 pgs: 129 active+clean; 84 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 584 KiB/s rd, 1.8 MiB/s wr, 167 op/s 2024-03-31T23:17:21.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:17:20 smithi046 ceph-mon[32498]: pgmap v171: 129 pgs: 129 active+clean; 84 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 584 KiB/s rd, 1.8 MiB/s wr, 167 op/s 2024-03-31T23:17:21.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:17:20 smithi142 ceph-mon[35271]: pgmap v171: 129 pgs: 129 active+clean; 84 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 584 KiB/s rd, 1.8 MiB/s wr, 167 op/s 2024-03-31T23:17:21.198 INFO:tasks.workunit.client.0.smithi046.stdout: 1 121547 2.47 MB/sec execute 147 sec latency 16.082 ms 2024-03-31T23:17:22.198 INFO:tasks.workunit.client.0.smithi046.stdout: 1 122061 2.48 MB/sec execute 148 sec latency 16.384 ms 2024-03-31T23:17:23.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:17:22 smithi162 ceph-mon[36237]: pgmap v172: 129 pgs: 129 active+clean; 84 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 377 KiB/s rd, 1.3 MiB/s wr, 111 op/s 2024-03-31T23:17:23.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:17:22 smithi046 ceph-mon[32498]: pgmap v172: 129 pgs: 129 active+clean; 84 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 377 KiB/s rd, 1.3 MiB/s wr, 111 op/s 2024-03-31T23:17:23.194 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:17:22 smithi142 ceph-mon[35271]: pgmap v172: 129 pgs: 129 active+clean; 84 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 377 KiB/s rd, 1.3 MiB/s wr, 111 op/s 2024-03-31T23:17:23.198 INFO:tasks.workunit.client.0.smithi046.stdout: 1 122518 2.47 MB/sec execute 149 sec latency 25.200 ms 2024-03-31T23:17:24.198 INFO:tasks.workunit.client.0.smithi046.stdout: 1 123058 2.46 MB/sec execute 150 sec latency 23.888 ms 2024-03-31T23:17:25.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:17:24 smithi162 ceph-mon[36237]: pgmap v173: 129 pgs: 129 active+clean; 84 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 377 KiB/s rd, 1.3 MiB/s wr, 111 op/s 2024-03-31T23:17:25.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:17:24 smithi046 ceph-mon[32498]: pgmap v173: 129 pgs: 129 active+clean; 84 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 377 KiB/s rd, 1.3 MiB/s wr, 111 op/s 2024-03-31T23:17:25.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:17:24 smithi142 ceph-mon[35271]: pgmap v173: 129 pgs: 129 active+clean; 84 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 377 KiB/s rd, 1.3 MiB/s wr, 111 op/s 2024-03-31T23:17:25.198 INFO:tasks.workunit.client.0.smithi046.stdout: 1 123654 2.48 MB/sec execute 151 sec latency 15.829 ms 2024-03-31T23:17:26.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:17:25 smithi162 ceph-mon[36237]: pgmap v174: 129 pgs: 129 active+clean; 85 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 604 KiB/s rd, 1.7 MiB/s wr, 170 op/s 2024-03-31T23:17:26.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:17:25 smithi046 ceph-mon[32498]: pgmap v174: 129 pgs: 129 active+clean; 85 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 604 KiB/s rd, 1.7 MiB/s wr, 170 op/s 2024-03-31T23:17:26.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:17:25 smithi142 ceph-mon[35271]: pgmap v174: 129 pgs: 129 active+clean; 85 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 604 KiB/s rd, 1.7 MiB/s wr, 170 op/s 2024-03-31T23:17:26.198 INFO:tasks.workunit.client.0.smithi046.stdout: 1 124158 2.50 MB/sec execute 152 sec latency 16.527 ms 2024-03-31T23:17:27.198 INFO:tasks.workunit.client.0.smithi046.stdout: 1 124518 2.49 MB/sec execute 153 sec latency 13.669 ms 2024-03-31T23:17:28.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:17:27 smithi046 ceph-mon[32498]: pgmap v175: 129 pgs: 129 active+clean; 85 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 401 KiB/s rd, 1.1 MiB/s wr, 114 op/s 2024-03-31T23:17:28.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:17:27 smithi142 ceph-mon[35271]: pgmap v175: 129 pgs: 129 active+clean; 85 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 401 KiB/s rd, 1.1 MiB/s wr, 114 op/s 2024-03-31T23:17:28.198 INFO:tasks.workunit.client.0.smithi046.stdout: 1 124775 2.48 MB/sec execute 154 sec latency 15.853 ms 2024-03-31T23:17:28.295 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:17:27 smithi162 ceph-mon[36237]: pgmap v175: 129 pgs: 129 active+clean; 85 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 401 KiB/s rd, 1.1 MiB/s wr, 114 op/s 2024-03-31T23:17:28.836 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:28.833+0000 7f5054ca4640 1 -- 172.21.15.46:0/1590352244 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5050107730 msgr2=0x7f5050107b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:17:28.836 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:28.833+0000 7f5054ca4640 1 --2- 172.21.15.46:0/1590352244 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5050107730 0x7f5050107b10 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7f5040009930 tx=0x7f504002bc30 comp rx=0 tx=0).stop 2024-03-31T23:17:28.836 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:28.833+0000 7f5054ca4640 1 -- 172.21.15.46:0/1590352244 shutdown_connections 2024-03-31T23:17:28.836 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:28.833+0000 7f5054ca4640 1 --2- 172.21.15.46:0/1590352244 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5050100610 0x7f505010e5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:17:28.837 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:28.833+0000 7f5054ca4640 1 --2- 172.21.15.46:0/1590352244 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f50500ffc50 0x7f50501000d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:17:28.837 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:28.834+0000 7f5054ca4640 1 --2- 172.21.15.46:0/1590352244 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5050107730 0x7f5050107b10 unknown :-1 s=CLOSED pgs=717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:17:28.837 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:28.834+0000 7f5054ca4640 1 -- 172.21.15.46:0/1590352244 >> 172.21.15.46:0/1590352244 conn(0x7f5050076b90 msgr2=0x7f50500fdf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:17:28.837 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:28.834+0000 7f5054ca4640 1 -- 172.21.15.46:0/1590352244 shutdown_connections 2024-03-31T23:17:28.837 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:28.834+0000 7f5054ca4640 1 -- 172.21.15.46:0/1590352244 wait complete. 2024-03-31T23:17:28.837 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:28.834+0000 7f5054ca4640 1 Processor -- start 2024-03-31T23:17:28.837 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:28.834+0000 7f5054ca4640 1 -- start start 2024-03-31T23:17:28.837 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:28.835+0000 7f5054ca4640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f50500ffc50 0x7f505019a130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:17:28.837 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:28.835+0000 7f5054ca4640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5050100610 0x7f505019a670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:17:28.837 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:28.835+0000 7f5054ca4640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5050107730 0x7f505019e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:17:28.837 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:28.835+0000 7f5054ca4640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f50501119f0 con 0x7f5050100610 2024-03-31T23:17:28.837 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:28.835+0000 7f5054ca4640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f5050111870 con 0x7f5050107730 2024-03-31T23:17:28.837 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:28.835+0000 7f5054ca4640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f5050111b70 con 0x7f50500ffc50 2024-03-31T23:17:28.838 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:28.835+0000 7f504e59c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f50500ffc50 0x7f505019a130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:17:28.838 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:28.835+0000 7f504e59c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f50500ffc50 0x7f505019a130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.46:46086/0 (socket says 172.21.15.46:46086) 2024-03-31T23:17:28.838 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:28.835+0000 7f504e59c640 1 -- 172.21.15.46:0/1950655846 learned_addr learned my addr 172.21.15.46:0/1950655846 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:17:28.838 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:28.835+0000 7f5047fff640 1 --2- 172.21.15.46:0/1950655846 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5050100610 0x7f505019a670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:17:28.838 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:28.835+0000 7f504ed9d640 1 --2- 172.21.15.46:0/1950655846 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5050107730 0x7f505019e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:17:28.843 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:28.835+0000 7f504e59c640 1 -- 172.21.15.46:0/1950655846 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5050107730 msgr2=0x7f505019e9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:17:28.843 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:28.835+0000 7f504e59c640 1 --2- 172.21.15.46:0/1950655846 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5050107730 0x7f505019e9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:17:28.844 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:28.835+0000 7f504e59c640 1 -- 172.21.15.46:0/1950655846 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5050100610 msgr2=0x7f505019a670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:17:28.844 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:28.836+0000 7f504e59c640 1 --2- 172.21.15.46:0/1950655846 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5050100610 0x7f505019a670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:17:28.844 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:28.836+0000 7f504e59c640 1 -- 172.21.15.46:0/1950655846 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f505019f120 con 0x7f50500ffc50 2024-03-31T23:17:28.844 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:28.836+0000 7f5047fff640 1 --2- 172.21.15.46:0/1950655846 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5050100610 0x7f505019a670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:17:28.844 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:28.836+0000 7f504e59c640 1 --2- 172.21.15.46:0/1950655846 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f50500ffc50 0x7f505019a130 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f5040009230 tx=0x7f504002c5d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:17:28.844 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:28.836+0000 7f50477fe640 1 -- 172.21.15.46:0/1950655846 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5040043020 con 0x7f50500ffc50 2024-03-31T23:17:28.844 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:28.836+0000 7f50477fe640 1 -- 172.21.15.46:0/1950655846 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f504002ca70 con 0x7f50500ffc50 2024-03-31T23:17:28.844 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:28.836+0000 7f50477fe640 1 -- 172.21.15.46:0/1950655846 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5040003a50 con 0x7f50500ffc50 2024-03-31T23:17:28.844 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:28.836+0000 7f5054ca4640 1 -- 172.21.15.46:0/1950655846 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f505019f3b0 con 0x7f50500ffc50 2024-03-31T23:17:28.844 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:28.837+0000 7f5054ca4640 1 -- 172.21.15.46:0/1950655846 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f50501a6ca0 con 0x7f50500ffc50 2024-03-31T23:17:28.844 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:28.839+0000 7f50477fe640 1 -- 172.21.15.46:0/1950655846 <== mon.1 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 25) ==== 137451+0+0 (secure 0 0 0) 0x7f5040046020 con 0x7f50500ffc50 2024-03-31T23:17:28.844 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:28.840+0000 7f5057964740 1 Processor -- start 2024-03-31T23:17:28.844 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:28.840+0000 7f50477fe640 1 --2- 172.21.15.46:0/1950655846 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f503c0a2730 0x7f503c0a4bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:17:28.845 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:28.840+0000 7f50477fe640 1 -- 172.21.15.46:0/1950655846 <== mon.1 v2:172.21.15.162:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7f50400f0d50 con 0x7f50500ffc50 2024-03-31T23:17:28.845 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:28.840+0000 7f5057964740 1 -- start start 2024-03-31T23:17:28.845 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:28.841+0000 7f5057964740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556627a88080 0x556627a88460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:17:28.845 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:28.841+0000 7f5057964740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556627a889a0 0x556627a9a080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:17:28.845 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:28.841+0000 7f5057964740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x556627a9a5c0 0x556627a9c9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:17:28.845 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:28.841+0000 7f5057964740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x5566279eb8f0 con 0x556627a889a0 2024-03-31T23:17:28.845 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:28.841+0000 7f5057964740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x5566279948d0 con 0x556627a9a5c0 2024-03-31T23:17:28.845 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:28.841+0000 7f5057964740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5566279e7990 con 0x556627a88080 2024-03-31T23:17:28.845 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:28.840+0000 7f504ed9d640 1 --2- 172.21.15.46:0/1950655846 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f503c0a2730 0x7f503c0a4bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:17:28.845 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:28.841+0000 7f5047fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556627a889a0 0x556627a9a080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:17:28.845 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:28.841+0000 7f504e59c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556627a88080 0x556627a88460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:17:28.845 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:28.841+0000 7f5047fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556627a889a0 0x556627a9a080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:32910/0 (socket says 172.21.15.46:32910) 2024-03-31T23:17:28.845 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:28.841+0000 7f5047fff640 1 -- 172.21.15.46:0/493277334 learned_addr learned my addr 172.21.15.46:0/493277334 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:17:28.845 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:28.841+0000 7f504ed9d640 1 --2- 172.21.15.46:0/493277334 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x556627a9a5c0 0x556627a9c9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:17:28.846 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:28.841+0000 7f5047fff640 1 -- 172.21.15.46:0/493277334 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556627a88080 msgr2=0x556627a88460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:17:28.846 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:28.841+0000 7f5047fff640 1 --2- 172.21.15.46:0/493277334 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556627a88080 0x556627a88460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:17:28.846 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:28.841+0000 7f504ed9d640 1 --2- 172.21.15.46:0/1950655846 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f503c0a2730 0x7f503c0a4bf0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f5048005fd0 tx=0x7f5048011b40 comp rx=0 tx=0).ready entity=mgr.15654 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:17:28.846 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:28.841+0000 7f5047fff640 1 -- 172.21.15.46:0/493277334 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x556627a9a5c0 msgr2=0x556627a9c9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:17:28.846 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:28.841+0000 7f5047fff640 1 --2- 172.21.15.46:0/493277334 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x556627a9a5c0 0x556627a9c9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:17:28.846 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:28.841+0000 7f5047fff640 1 -- 172.21.15.46:0/493277334 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556627700210 con 0x556627a889a0 2024-03-31T23:17:28.846 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:28.842+0000 7f5047fff640 1 --2- 172.21.15.46:0/493277334 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556627a889a0 0x556627a9a080 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7f50380114f0 tx=0x7f50380119b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=89ac32fa69df44ac server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:17:28.846 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:28.842+0000 7f5044ff9640 1 -- 172.21.15.46:0/493277334 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5038004020 con 0x556627a889a0 2024-03-31T23:17:28.846 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:28.842+0000 7f5044ff9640 1 -- 172.21.15.46:0/493277334 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f50380041c0 con 0x556627a889a0 2024-03-31T23:17:28.846 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:28.842+0000 7f5044ff9640 1 -- 172.21.15.46:0/493277334 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f503800f820 con 0x556627a889a0 2024-03-31T23:17:28.851 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:28.843+0000 7f5057964740 1 -- 172.21.15.46:0/493277334 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556627a889a0 msgr2=0x556627a9a080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:17:28.851 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:28.843+0000 7f5057964740 1 --2- 172.21.15.46:0/493277334 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556627a889a0 0x556627a9a080 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7f50380114f0 tx=0x7f50380119b0 comp rx=0 tx=0).stop 2024-03-31T23:17:28.851 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:28.843+0000 7f5057964740 1 -- 172.21.15.46:0/493277334 shutdown_connections 2024-03-31T23:17:28.851 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:28.843+0000 7f5057964740 1 --2- 172.21.15.46:0/493277334 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x556627a9a5c0 0x556627a9c9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:17:28.851 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:28.843+0000 7f5057964740 1 --2- 172.21.15.46:0/493277334 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556627a889a0 0x556627a9a080 unknown :-1 s=CLOSED pgs=718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:17:28.851 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:28.843+0000 7f5057964740 1 --2- 172.21.15.46:0/493277334 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556627a88080 0x556627a88460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:17:28.851 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:28.843+0000 7f5057964740 1 -- 172.21.15.46:0/493277334 >> 172.21.15.46:0/493277334 conn(0x556627a82fe0 msgr2=0x556627a833c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:17:28.851 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:28.843+0000 7f5057964740 1 -- 172.21.15.46:0/493277334 shutdown_connections 2024-03-31T23:17:28.851 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:28.843+0000 7f5057964740 1 -- 172.21.15.46:0/493277334 wait complete. 2024-03-31T23:17:28.851 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:28.843+0000 7f5057964740 1 -- start start 2024-03-31T23:17:28.851 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:28.843+0000 7f5057964740 1 Processor -- start 2024-03-31T23:17:28.851 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:28.844+0000 7f5057964740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x556627a88080 0x556627a97690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:17:28.852 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:28.844+0000 7f5057964740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556627a889a0 0x556627a97bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:17:28.852 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:28.844+0000 7f5057964740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556627a9a5c0 0x556627a98110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:17:28.852 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:28.844+0000 7f5057964740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x556627a9d1f0 con 0x556627a9a5c0 2024-03-31T23:17:28.852 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:28.844+0000 7f5057964740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x556627989f80 con 0x556627a88080 2024-03-31T23:17:28.852 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:28.844+0000 7f5057964740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x556627a9d370 con 0x556627a889a0 2024-03-31T23:17:28.852 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:28.844+0000 7f5047fff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556627a889a0 0x556627a97bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:17:28.852 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:28.844+0000 7f504e59c640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x556627a88080 0x556627a97690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:17:28.852 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:28.844+0000 7f504ed9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556627a9a5c0 0x556627a98110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:17:28.852 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:28.844+0000 7f504e59c640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x556627a88080 0x556627a97690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.46:41122/0 (socket says 172.21.15.46:41122) 2024-03-31T23:17:28.852 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:28.844+0000 7f504e59c640 1 -- 172.21.15.46:0/810388131 learned_addr learned my addr 172.21.15.46:0/810388131 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:17:28.852 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:28.844+0000 7f5057964740 10 client.0 _ll_register_callbacks cb 0x5566279e2290 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T23:17:28.852 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:28.844+0000 7f504e59c640 1 -- 172.21.15.46:0/810388131 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556627a889a0 msgr2=0x556627a97bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:17:28.852 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:28.844+0000 7f504e59c640 1 --2- 172.21.15.46:0/810388131 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556627a889a0 0x556627a97bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:17:28.852 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:28.844+0000 7f504e59c640 1 -- 172.21.15.46:0/810388131 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556627a9a5c0 msgr2=0x556627a98110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:17:28.853 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:28.844+0000 7f504e59c640 1 --2- 172.21.15.46:0/810388131 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556627a9a5c0 0x556627a98110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:17:28.853 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:28.844+0000 7f504e59c640 1 -- 172.21.15.46:0/810388131 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5566277cc620 con 0x556627a88080 2024-03-31T23:17:28.853 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:28.844+0000 7f504ed9d640 1 --2- 172.21.15.46:0/810388131 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556627a9a5c0 0x556627a98110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:17:28.853 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:28.844+0000 7f5057964740 1 -- 172.21.15.46:0/810388131 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f50380041c0 con 0x556627a88080 2024-03-31T23:17:28.853 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:28.845+0000 7f504e59c640 1 --2- 172.21.15.46:0/810388131 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x556627a88080 0x556627a97690 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f50400ee260 tx=0x7f50400dd170 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1867b71620abcab1 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:17:28.853 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:28.845+0000 7f5035ffb640 10 client.26456 ms_handle_connect on v2:172.21.15.142:3300/0 2024-03-31T23:17:28.853 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:28.845+0000 7f5035ffb640 1 -- 172.21.15.46:0/810388131 <== mon.2 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50400c1170 con 0x556627a88080 2024-03-31T23:17:28.853 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:28.845+0000 7f5035ffb640 1 -- 172.21.15.46:0/810388131 <== mon.2 v2:172.21.15.142:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7f50400b98a0 con 0x556627a88080 2024-03-31T23:17:28.853 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:28.845+0000 7f5035ffb640 1 -- 172.21.15.46:0/810388131 <== mon.2 v2:172.21.15.142:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f50400c1380 con 0x556627a88080 2024-03-31T23:17:28.853 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:28.845+0000 7f5057964740 10 client.26456 fetch_fsmap learned FSMap version 30 2024-03-31T23:17:28.853 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:28.845+0000 7f5057964740 1 -- 172.21.15.46:0/810388131 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({fsmap=30}) -- 0x556627aa9040 con 0x556627a88080 2024-03-31T23:17:28.853 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:28.845+0000 7f5035ffb640 1 -- 172.21.15.46:0/810388131 <== mon.2 v2:172.21.15.142:3300/0 4 ==== fsmap(e 30) ==== 8346+0+0 (secure 0 0 0) 0x7f50400053d0 con 0x556627a88080 2024-03-31T23:17:28.853 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:28.845+0000 7f5057964740 10 client.26456 fetch_fsmap finished waiting for FSMap version 30 2024-03-31T23:17:28.853 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:28.845+0000 7f5057964740 10 client.26456 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-31T23:17:28.854 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:28.845+0000 7f5057964740 20 client.26456 populate_metadata read hostname 'smithi046' 2024-03-31T23:17:28.854 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:28.845+0000 7f5057964740 1 --2- 172.21.15.46:0/810388131 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x556627aaa400 0x556627aac890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:17:28.854 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:28.845+0000 7f5057964740 4 client.26456 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-31T23:17:28.854 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:28.845+0000 7f5057964740 1 -- 172.21.15.46:0/810388131 --> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5566279ff120 con 0x556627aaa400 2024-03-31T23:17:28.854 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:28.845+0000 7f5047fff640 1 --2- 172.21.15.46:0/810388131 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x556627aaa400 0x556627aac890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:17:28.854 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:28.846+0000 7f5047fff640 1 --2- 172.21.15.46:0/810388131 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x556627aaa400 0x556627aac890 crc :-1 s=READY pgs=125 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c0d295a07f73b82c server_cookie=d77a8307fa7a65d4 in_seq=0 out_seq=0 2024-03-31T23:17:28.854 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:28.846+0000 7f5035ffb640 10 client.26456 ms_handle_connect on v2:172.21.15.142:6838/2519143918 2024-03-31T23:17:28.854 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:28.847+0000 7f5035ffb640 1 -- 172.21.15.46:0/810388131 <== mds.0 v2:172.21.15.142:6838/2519143918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5566279ff120 con 0x556627aaa400 2024-03-31T23:17:28.854 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:28.847+0000 7f5035ffb640 10 client.26456 handle_command_reply: tid=0 2024-03-31T23:17:28.854 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:28.847+0000 7f5057964740 1 client.26456 shutdown 2024-03-31T23:17:28.854 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:28.847+0000 7f5057964740 1 -- 172.21.15.46:0/810388131 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x556627a88080 msgr2=0x556627a97690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:17:28.854 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:28.847+0000 7f5057964740 1 --2- 172.21.15.46:0/810388131 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x556627a88080 0x556627a97690 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f50400ee260 tx=0x7f50400dd170 comp rx=0 tx=0).stop 2024-03-31T23:17:28.855 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:28.847+0000 7f5057964740 1 -- 172.21.15.46:0/810388131 shutdown_connections 2024-03-31T23:17:28.855 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:28.847+0000 7f5057964740 1 --2- 172.21.15.46:0/810388131 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x556627aaa400 0x556627aac890 crc :-1 s=READY pgs=125 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:17:28.855 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:28.847+0000 7f5057964740 1 --2- 172.21.15.46:0/810388131 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556627a9a5c0 0x556627a98110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:17:28.855 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:28.847+0000 7f5057964740 1 --2- 172.21.15.46:0/810388131 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556627a889a0 0x556627a97bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:17:28.855 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:28.847+0000 7f5057964740 1 --2- 172.21.15.46:0/810388131 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x556627a88080 0x556627a97690 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:17:28.855 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:28.847+0000 7f5035ffb640 0 client.26456 ms_handle_reset on v2:172.21.15.142:6838/2519143918 2024-03-31T23:17:28.855 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:28.847+0000 7f5057964740 1 -- 172.21.15.46:0/810388131 >> 172.21.15.46:0/810388131 conn(0x556627a82fe0 msgr2=0x556627a84d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:17:28.855 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:28.847+0000 7f5057964740 1 -- 172.21.15.46:0/810388131 shutdown_connections 2024-03-31T23:17:28.855 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:28.847+0000 7f5057964740 1 -- 172.21.15.46:0/810388131 wait complete. 2024-03-31T23:17:28.855 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:28.847+0000 7f5057964740 20 client.26456 trim_cache size 0 max 16384 2024-03-31T23:17:28.865 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:28.863+0000 7f5057964740 1 Processor -- start 2024-03-31T23:17:28.865 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:28.863+0000 7f5057964740 1 -- start start 2024-03-31T23:17:28.865 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:28.863+0000 7f5057964740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556627a88080 0x556627a99dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:17:28.865 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:28.863+0000 7f5057964740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x556627a889a0 0x556627aa5f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:17:28.865 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:28.863+0000 7f5057964740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556627a9a5c0 0x556627a833c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:17:28.865 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:28.863+0000 7f5057964740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f50501af0f0 con 0x556627a9a5c0 2024-03-31T23:17:28.865 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:28.863+0000 7f5057964740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f50501ac9c0 con 0x556627a889a0 2024-03-31T23:17:28.865 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:28.863+0000 7f5057964740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f50501aa850 con 0x556627a88080 2024-03-31T23:17:28.866 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:28.863+0000 7f504ed9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556627a9a5c0 0x556627a833c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:17:28.866 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:28.863+0000 7f504ed9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556627a9a5c0 0x556627a833c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:32938/0 (socket says 172.21.15.46:32938) 2024-03-31T23:17:28.866 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:28.863+0000 7f504ed9d640 1 -- 172.21.15.46:0/2038058803 learned_addr learned my addr 172.21.15.46:0/2038058803 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:17:28.866 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:28.863+0000 7f504e59c640 1 --2- 172.21.15.46:0/2038058803 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556627a88080 0x556627a99dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:17:28.866 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:28.863+0000 7f5047fff640 1 --2- 172.21.15.46:0/2038058803 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x556627a889a0 0x556627aa5f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:17:28.866 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:28.863+0000 7f504ed9d640 1 -- 172.21.15.46:0/2038058803 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556627a88080 msgr2=0x556627a99dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:17:28.866 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:28.863+0000 7f504ed9d640 1 --2- 172.21.15.46:0/2038058803 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556627a88080 0x556627a99dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:17:28.866 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:28.863+0000 7f504ed9d640 1 -- 172.21.15.46:0/2038058803 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x556627a889a0 msgr2=0x556627aa5f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:17:28.866 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:28.863+0000 7f504ed9d640 1 --2- 172.21.15.46:0/2038058803 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x556627a889a0 0x556627aa5f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:17:28.866 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:28.863+0000 7f504ed9d640 1 -- 172.21.15.46:0/2038058803 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556627a96640 con 0x556627a9a5c0 2024-03-31T23:17:28.866 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:28.864+0000 7f504ed9d640 1 --2- 172.21.15.46:0/2038058803 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556627a9a5c0 0x556627a833c0 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7f504800ffc0 tx=0x7f504801f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d10a1bbf1922291f server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:17:28.866 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:28.864+0000 7f5037fff640 1 -- 172.21.15.46:0/2038058803 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f504801a040 con 0x556627a9a5c0 2024-03-31T23:17:28.866 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:28.864+0000 7f5037fff640 1 -- 172.21.15.46:0/2038058803 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5048004b20 con 0x556627a9a5c0 2024-03-31T23:17:28.867 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:28.864+0000 7f5037fff640 1 -- 172.21.15.46:0/2038058803 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5048005790 con 0x556627a9a5c0 2024-03-31T23:17:28.870 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:28.864+0000 7f5057964740 1 -- 172.21.15.46:0/2038058803 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556627a9a5c0 msgr2=0x556627a833c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:17:28.870 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:28.864+0000 7f5057964740 1 --2- 172.21.15.46:0/2038058803 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556627a9a5c0 0x556627a833c0 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7f504800ffc0 tx=0x7f504801f000 comp rx=0 tx=0).stop 2024-03-31T23:17:28.870 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:28.864+0000 7f5057964740 1 -- 172.21.15.46:0/2038058803 shutdown_connections 2024-03-31T23:17:28.870 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:28.864+0000 7f5057964740 1 --2- 172.21.15.46:0/2038058803 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556627a9a5c0 0x556627a833c0 unknown :-1 s=CLOSED pgs=719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:17:28.870 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:28.864+0000 7f5057964740 1 --2- 172.21.15.46:0/2038058803 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x556627a889a0 0x556627aa5f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:17:28.870 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:28.865+0000 7f5057964740 1 --2- 172.21.15.46:0/2038058803 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556627a88080 0x556627a99dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:17:28.870 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:28.865+0000 7f5057964740 1 -- 172.21.15.46:0/2038058803 >> 172.21.15.46:0/2038058803 conn(0x556627a82fe0 msgr2=0x556627a81f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:17:28.870 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:28.865+0000 7f5057964740 1 -- 172.21.15.46:0/2038058803 shutdown_connections 2024-03-31T23:17:28.871 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:28.865+0000 7f5057964740 1 -- 172.21.15.46:0/2038058803 wait complete. 2024-03-31T23:17:28.871 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:28.865+0000 7f5057964740 1 -- start start 2024-03-31T23:17:28.871 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:28.865+0000 7f5057964740 1 Processor -- start 2024-03-31T23:17:28.871 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:28.865+0000 7f5057964740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x556627a88080 0x556627a9e170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:17:28.871 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:28.865+0000 7f5057964740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556627a889a0 0x556627a9e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:17:28.871 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:28.865+0000 7f5057964740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556627a9a5c0 0x556627a9ec10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:17:28.871 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:28.865+0000 7f5057964740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f50501aaea0 con 0x556627a9a5c0 2024-03-31T23:17:28.871 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:28.865+0000 7f5057964740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f50501ac2f0 con 0x556627a88080 2024-03-31T23:17:28.871 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:28.865+0000 7f5057964740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f50501ab550 con 0x556627a889a0 2024-03-31T23:17:28.871 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:28.865+0000 7f5057964740 10 client.0 _ll_register_callbacks cb 0x7f50400bd040 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T23:17:28.871 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:28.865+0000 7f504ed9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556627a9a5c0 0x556627a9ec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:17:28.871 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:28.865+0000 7f504e59c640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x556627a88080 0x556627a9e170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:17:28.871 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:28.865+0000 7f504ed9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556627a9a5c0 0x556627a9ec10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:32942/0 (socket says 172.21.15.46:32942) 2024-03-31T23:17:28.871 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:28.865+0000 7f504ed9d640 1 -- 172.21.15.46:0/1501351135 learned_addr learned my addr 172.21.15.46:0/1501351135 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:17:28.872 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:28.865+0000 7f5047fff640 1 --2- 172.21.15.46:0/1501351135 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556627a889a0 0x556627a9e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:17:28.872 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:28.866+0000 7f504ed9d640 1 -- 172.21.15.46:0/1501351135 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556627a889a0 msgr2=0x556627a9e6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:17:28.872 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:28.866+0000 7f504ed9d640 1 --2- 172.21.15.46:0/1501351135 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556627a889a0 0x556627a9e6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:17:28.872 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:28.866+0000 7f504ed9d640 1 -- 172.21.15.46:0/1501351135 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x556627a88080 msgr2=0x556627a9e170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:17:28.872 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:28.866+0000 7f504ed9d640 1 --2- 172.21.15.46:0/1501351135 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x556627a88080 0x556627a9e170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:17:28.872 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:28.866+0000 7f504ed9d640 1 -- 172.21.15.46:0/1501351135 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556627b0fa60 con 0x556627a9a5c0 2024-03-31T23:17:28.872 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:28.866+0000 7f504ed9d640 1 --2- 172.21.15.46:0/1501351135 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556627a9a5c0 0x556627a9ec10 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7f5048005050 tx=0x7f5048005390 comp rx=0 tx=0).ready entity=mon.0 client_cookie=efe800f1118f60e2 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:17:28.872 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:28.866+0000 7f5035ffb640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-31T23:17:28.872 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:28.866+0000 7f5035ffb640 1 -- 172.21.15.46:0/1501351135 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5048032070 con 0x556627a9a5c0 2024-03-31T23:17:28.872 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:28.866+0000 7f5035ffb640 1 -- 172.21.15.46:0/1501351135 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7f5048004760 con 0x556627a9a5c0 2024-03-31T23:17:28.872 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:28.866+0000 7f5057964740 1 -- 172.21.15.46:0/1501351135 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5048004b20 con 0x556627a9a5c0 2024-03-31T23:17:28.872 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:28.866+0000 7f5035ffb640 1 -- 172.21.15.46:0/1501351135 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f504803e050 con 0x556627a9a5c0 2024-03-31T23:17:28.872 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:28.866+0000 7f5057964740 10 client.16623 fetch_fsmap learned FSMap version 30 2024-03-31T23:17:28.873 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:28.866+0000 7f5057964740 1 -- 172.21.15.46:0/1501351135 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x556627b10d10 con 0x556627a9a5c0 2024-03-31T23:17:28.873 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:28.866+0000 7f5035ffb640 1 -- 172.21.15.46:0/1501351135 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5048004b20 con 0x556627a9a5c0 2024-03-31T23:17:28.873 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:28.866+0000 7f5035ffb640 1 -- 172.21.15.46:0/1501351135 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 30) ==== 8346+0+0 (secure 0 0 0) 0x7f5048028070 con 0x556627a9a5c0 2024-03-31T23:17:28.873 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:28.866+0000 7f5057964740 10 client.16623 fetch_fsmap finished waiting for FSMap version 30 2024-03-31T23:17:28.873 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:28.866+0000 7f5057964740 10 client.16623 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-31T23:17:28.873 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:28.866+0000 7f5057964740 20 client.16623 populate_metadata read hostname 'smithi046' 2024-03-31T23:17:28.873 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:28.866+0000 7f5057964740 1 --2- 172.21.15.46:0/1501351135 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x556627b11380 0x556627b13770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:17:28.873 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:28.866+0000 7f5057964740 4 client.16623 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-31T23:17:28.873 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:28.867+0000 7f5057964740 1 -- 172.21.15.46:0/1501351135 --> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5566279f8e30 con 0x556627b11380 2024-03-31T23:17:28.873 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:28.867+0000 7f504e59c640 1 --2- 172.21.15.46:0/1501351135 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x556627b11380 0x556627b13770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:17:28.873 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:28.867+0000 7f504e59c640 1 --2- 172.21.15.46:0/1501351135 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x556627b11380 0x556627b13770 crc :-1 s=READY pgs=126 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=413a61153a8f1893 server_cookie=6801e86e0a7f2cba in_seq=0 out_seq=0 2024-03-31T23:17:28.873 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:28.867+0000 7f5035ffb640 10 client.16623 ms_handle_connect on v2:172.21.15.142:6838/2519143918 2024-03-31T23:17:28.889 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:28.887+0000 7f5035ffb640 1 -- 172.21.15.46:0/1501351135 <== mds.0 v2:172.21.15.142:6838/2519143918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5566279f8e30 con 0x556627b11380 2024-03-31T23:17:28.889 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:28.887+0000 7f5035ffb640 10 client.16623 handle_command_reply: tid=0 2024-03-31T23:17:28.889 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:28.887+0000 7f5057964740 1 client.16623 shutdown 2024-03-31T23:17:28.889 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:28.887+0000 7f5057964740 1 -- 172.21.15.46:0/1501351135 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556627a9a5c0 msgr2=0x556627a9ec10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:17:28.889 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:28.887+0000 7f5057964740 1 --2- 172.21.15.46:0/1501351135 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556627a9a5c0 0x556627a9ec10 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7f5048005050 tx=0x7f5048005390 comp rx=0 tx=0).stop 2024-03-31T23:17:28.889 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:28.887+0000 7f5057964740 1 -- 172.21.15.46:0/1501351135 shutdown_connections 2024-03-31T23:17:28.889 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:28.887+0000 7f5057964740 1 --2- 172.21.15.46:0/1501351135 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x556627b11380 0x556627b13770 crc :-1 s=READY pgs=126 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:17:28.890 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:28.887+0000 7f5057964740 1 --2- 172.21.15.46:0/1501351135 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556627a9a5c0 0x556627a9ec10 unknown :-1 s=CLOSED pgs=720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:17:28.890 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:28.887+0000 7f5035ffb640 0 client.16623 ms_handle_reset on v2:172.21.15.142:6838/2519143918 2024-03-31T23:17:28.890 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:28.887+0000 7f5057964740 1 --2- 172.21.15.46:0/1501351135 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556627a889a0 0x556627a9e6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:17:28.890 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:28.887+0000 7f5057964740 1 --2- 172.21.15.46:0/1501351135 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x556627a88080 0x556627a9e170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:17:28.890 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:28.887+0000 7f5057964740 1 -- 172.21.15.46:0/1501351135 >> 172.21.15.46:0/1501351135 conn(0x556627a82fe0 msgr2=0x556627a81f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:17:28.890 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:28.888+0000 7f5057964740 1 -- 172.21.15.46:0/1501351135 shutdown_connections 2024-03-31T23:17:28.890 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:28.888+0000 7f5057964740 1 -- 172.21.15.46:0/1501351135 wait complete. 2024-03-31T23:17:28.890 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:28.888+0000 7f5057964740 20 client.16623 trim_cache size 0 max 16384 2024-03-31T23:17:28.890 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout:{ 2024-03-31T23:17:28.890 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout: "message": "", 2024-03-31T23:17:28.891 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout: "return_code": 0 2024-03-31T23:17:28.891 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout:} 2024-03-31T23:17:28.891 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:28.888+0000 7f5054ca4640 1 -- 172.21.15.46:0/1950655846 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f503c0a2730 msgr2=0x7f503c0a4bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:17:28.892 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:28.888+0000 7f5054ca4640 1 --2- 172.21.15.46:0/1950655846 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f503c0a2730 0x7f503c0a4bf0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f5048005fd0 tx=0x7f5048011b40 comp rx=0 tx=0).stop 2024-03-31T23:17:28.892 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:28.888+0000 7f5054ca4640 1 -- 172.21.15.46:0/1950655846 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f50500ffc50 msgr2=0x7f505019a130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:17:28.892 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:28.888+0000 7f5054ca4640 1 --2- 172.21.15.46:0/1950655846 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f50500ffc50 0x7f505019a130 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f5040009230 tx=0x7f504002c5d0 comp rx=0 tx=0).stop 2024-03-31T23:17:28.892 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:28.888+0000 7f5054ca4640 1 -- 172.21.15.46:0/1950655846 shutdown_connections 2024-03-31T23:17:28.892 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:28.888+0000 7f5054ca4640 1 --2- 172.21.15.46:0/1950655846 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f503c0a2730 0x7f503c0a4bf0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:17:28.892 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:28.888+0000 7f5054ca4640 1 --2- 172.21.15.46:0/1950655846 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5050107730 0x7f505019e9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:17:28.892 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:28.888+0000 7f5054ca4640 1 --2- 172.21.15.46:0/1950655846 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5050100610 0x7f505019a670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:17:28.892 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:28.888+0000 7f5054ca4640 1 --2- 172.21.15.46:0/1950655846 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f50500ffc50 0x7f505019a130 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:17:28.892 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:28.888+0000 7f5054ca4640 1 -- 172.21.15.46:0/1950655846 >> 172.21.15.46:0/1950655846 conn(0x7f5050076b90 msgr2=0x7f50500fe040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:17:28.892 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:28.888+0000 7f5054ca4640 1 -- 172.21.15.46:0/1950655846 shutdown_connections 2024-03-31T23:17:28.892 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:28.888+0000 7f5054ca4640 1 -- 172.21.15.46:0/1950655846 wait complete. 2024-03-31T23:17:29.198 INFO:tasks.workunit.client.0.smithi046.stdout: 1 125232 2.47 MB/sec execute 155 sec latency 18.708 ms 2024-03-31T23:17:30.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:17:29 smithi046 ceph-mon[32498]: pgmap v176: 129 pgs: 129 active+clean; 90 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 600 KiB/s rd, 1.8 MiB/s wr, 172 op/s 2024-03-31T23:17:30.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:17:29 smithi142 ceph-mon[35271]: pgmap v176: 129 pgs: 129 active+clean; 90 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 600 KiB/s rd, 1.8 MiB/s wr, 172 op/s 2024-03-31T23:17:30.198 INFO:tasks.workunit.client.0.smithi046.stdout: 1 125716 2.48 MB/sec execute 156 sec latency 17.226 ms 2024-03-31T23:17:30.295 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:17:29 smithi162 ceph-mon[36237]: pgmap v176: 129 pgs: 129 active+clean; 90 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 600 KiB/s rd, 1.8 MiB/s wr, 172 op/s 2024-03-31T23:17:31.199 INFO:tasks.workunit.client.0.smithi046.stdout: 1 126250 2.47 MB/sec execute 157 sec latency 32.529 ms 2024-03-31T23:17:32.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:17:31 smithi046 ceph-mon[32498]: pgmap v177: 129 pgs: 129 active+clean; 90 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 426 KiB/s rd, 1.2 MiB/s wr, 116 op/s 2024-03-31T23:17:32.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:17:31 smithi142 ceph-mon[35271]: pgmap v177: 129 pgs: 129 active+clean; 90 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 426 KiB/s rd, 1.2 MiB/s wr, 116 op/s 2024-03-31T23:17:32.198 INFO:tasks.workunit.client.0.smithi046.stdout: 1 126763 2.47 MB/sec execute 158 sec latency 23.289 ms 2024-03-31T23:17:32.295 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:17:31 smithi162 ceph-mon[36237]: pgmap v177: 129 pgs: 129 active+clean; 90 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 426 KiB/s rd, 1.2 MiB/s wr, 116 op/s 2024-03-31T23:17:33.199 INFO:tasks.workunit.client.0.smithi046.stdout: 1 127485 2.49 MB/sec execute 159 sec latency 15.594 ms 2024-03-31T23:17:34.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:17:33 smithi046 ceph-mon[32498]: pgmap v178: 129 pgs: 129 active+clean; 90 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 426 KiB/s rd, 1.2 MiB/s wr, 116 op/s 2024-03-31T23:17:34.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:17:33 smithi142 ceph-mon[35271]: pgmap v178: 129 pgs: 129 active+clean; 90 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 426 KiB/s rd, 1.2 MiB/s wr, 116 op/s 2024-03-31T23:17:34.199 INFO:tasks.workunit.client.0.smithi046.stdout: 1 127869 2.50 MB/sec execute 160 sec latency 16.294 ms 2024-03-31T23:17:34.296 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:17:33 smithi162 ceph-mon[36237]: pgmap v178: 129 pgs: 129 active+clean; 90 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 426 KiB/s rd, 1.2 MiB/s wr, 116 op/s 2024-03-31T23:17:35.199 INFO:tasks.workunit.client.0.smithi046.stdout: 1 128154 2.49 MB/sec execute 161 sec latency 15.534 ms 2024-03-31T23:17:36.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:17:35 smithi046 ceph-mon[32498]: pgmap v179: 129 pgs: 129 active+clean; 87 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 621 KiB/s rd, 1.9 MiB/s wr, 172 op/s 2024-03-31T23:17:36.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:17:35 smithi142 ceph-mon[35271]: pgmap v179: 129 pgs: 129 active+clean; 87 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 621 KiB/s rd, 1.9 MiB/s wr, 172 op/s 2024-03-31T23:17:36.199 INFO:tasks.workunit.client.0.smithi046.stdout: 1 128485 2.48 MB/sec execute 162 sec latency 17.224 ms 2024-03-31T23:17:36.295 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:17:35 smithi162 ceph-mon[36237]: pgmap v179: 129 pgs: 129 active+clean; 87 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 621 KiB/s rd, 1.9 MiB/s wr, 172 op/s 2024-03-31T23:17:37.199 INFO:tasks.workunit.client.0.smithi046.stdout: 1 128954 2.47 MB/sec execute 163 sec latency 51.181 ms 2024-03-31T23:17:38.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:17:37 smithi046 ceph-mon[32498]: pgmap v180: 129 pgs: 129 active+clean; 87 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 394 KiB/s rd, 1.4 MiB/s wr, 114 op/s 2024-03-31T23:17:38.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:17:37 smithi142 ceph-mon[35271]: pgmap v180: 129 pgs: 129 active+clean; 87 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 394 KiB/s rd, 1.4 MiB/s wr, 114 op/s 2024-03-31T23:17:38.199 INFO:tasks.workunit.client.0.smithi046.stdout: 1 129431 2.48 MB/sec execute 164 sec latency 20.862 ms 2024-03-31T23:17:38.295 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:17:37 smithi162 ceph-mon[36237]: pgmap v180: 129 pgs: 129 active+clean; 87 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 394 KiB/s rd, 1.4 MiB/s wr, 114 op/s 2024-03-31T23:17:39.199 INFO:tasks.workunit.client.0.smithi046.stdout: 1 130035 2.47 MB/sec execute 165 sec latency 21.843 ms 2024-03-31T23:17:40.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:17:39 smithi046 ceph-mon[32498]: pgmap v181: 129 pgs: 129 active+clean; 88 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 620 KiB/s rd, 1.9 MiB/s wr, 175 op/s 2024-03-31T23:17:40.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:17:39 smithi142 ceph-mon[35271]: pgmap v181: 129 pgs: 129 active+clean; 88 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 620 KiB/s rd, 1.9 MiB/s wr, 175 op/s 2024-03-31T23:17:40.199 INFO:tasks.workunit.client.0.smithi046.stdout: 1 130598 2.48 MB/sec execute 166 sec latency 18.130 ms 2024-03-31T23:17:40.296 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:17:39 smithi162 ceph-mon[36237]: pgmap v181: 129 pgs: 129 active+clean; 88 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 620 KiB/s rd, 1.9 MiB/s wr, 175 op/s 2024-03-31T23:17:41.199 INFO:tasks.workunit.client.0.smithi046.stdout: 1 131069 2.49 MB/sec execute 167 sec latency 16.619 ms 2024-03-31T23:17:41.986 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:41.983+0000 7ffb70f65640 1 -- 172.21.15.46:0/1454988625 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffb6c101240 msgr2=0x7ffb6c10d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:17:41.986 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:41.983+0000 7ffb70f65640 1 --2- 172.21.15.46:0/1454988625 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffb6c101240 0x7ffb6c10d480 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7ffb64009ca0 tx=0x7ffb6402c0f0 comp rx=0 tx=0).stop 2024-03-31T23:17:41.987 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:41.983+0000 7ffb70f65640 1 -- 172.21.15.46:0/1454988625 shutdown_connections 2024-03-31T23:17:41.987 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:41.983+0000 7ffb70f65640 1 --2- 172.21.15.46:0/1454988625 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffb6c101240 0x7ffb6c10d480 unknown :-1 s=CLOSED pgs=721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:17:41.987 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:41.983+0000 7ffb70f65640 1 --2- 172.21.15.46:0/1454988625 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ffb6c1008a0 0x7ffb6c100d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:17:41.987 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:41.983+0000 7ffb70f65640 1 --2- 172.21.15.46:0/1454988625 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ffb6c1055d0 0x7ffb6c1059b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:17:41.987 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:41.983+0000 7ffb70f65640 1 -- 172.21.15.46:0/1454988625 >> 172.21.15.46:0/1454988625 conn(0x7ffb6c077420 msgr2=0x7ffb6c077830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:17:41.987 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:41.983+0000 7ffb70f65640 1 -- 172.21.15.46:0/1454988625 shutdown_connections 2024-03-31T23:17:41.987 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:41.983+0000 7ffb70f65640 1 -- 172.21.15.46:0/1454988625 wait complete. 2024-03-31T23:17:41.987 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:41.984+0000 7ffb70f65640 1 Processor -- start 2024-03-31T23:17:41.987 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:41.984+0000 7ffb70f65640 1 -- start start 2024-03-31T23:17:41.987 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:41.984+0000 7ffb70f65640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ffb6c1008a0 0x7ffb6c0fe2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:17:41.987 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:41.984+0000 7ffb70f65640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffb6c101240 0x7ffb6c0fe830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:17:41.987 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:41.984+0000 7ffb70f65640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ffb6c1055d0 0x7ffb6c0fed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:17:41.988 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:41.984+0000 7ffb70f65640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7ffb6c110990 con 0x7ffb6c101240 2024-03-31T23:17:41.988 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:41.984+0000 7ffb70f65640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7ffb6c110810 con 0x7ffb6c1008a0 2024-03-31T23:17:41.988 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:41.984+0000 7ffb70f65640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7ffb6c110b10 con 0x7ffb6c1055d0 2024-03-31T23:17:41.988 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:41.984+0000 7ffb63fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffb6c101240 0x7ffb6c0fe830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:17:41.988 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:41.984+0000 7ffb63fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffb6c101240 0x7ffb6c0fe830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:50888/0 (socket says 172.21.15.46:50888) 2024-03-31T23:17:41.988 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:41.984+0000 7ffb6a59c640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ffb6c1008a0 0x7ffb6c0fe2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:17:41.988 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:41.984+0000 7ffb63fff640 1 -- 172.21.15.46:0/2731648025 learned_addr learned my addr 172.21.15.46:0/2731648025 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:17:41.988 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:41.984+0000 7ffb6ad9d640 1 --2- 172.21.15.46:0/2731648025 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ffb6c1055d0 0x7ffb6c0fed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:17:41.988 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:41.984+0000 7ffb63fff640 1 -- 172.21.15.46:0/2731648025 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ffb6c1055d0 msgr2=0x7ffb6c0fed70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:17:41.988 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:41.984+0000 7ffb63fff640 1 --2- 172.21.15.46:0/2731648025 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ffb6c1055d0 0x7ffb6c0fed70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:17:41.988 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:41.984+0000 7ffb63fff640 1 -- 172.21.15.46:0/2731648025 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ffb6c1008a0 msgr2=0x7ffb6c0fe2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:17:41.988 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:41.984+0000 7ffb63fff640 1 --2- 172.21.15.46:0/2731648025 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ffb6c1008a0 0x7ffb6c0fe2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:17:41.988 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:41.984+0000 7ffb63fff640 1 -- 172.21.15.46:0/2731648025 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffb6c1a6400 con 0x7ffb6c101240 2024-03-31T23:17:41.988 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:41.984+0000 7ffb6ad9d640 1 --2- 172.21.15.46:0/2731648025 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ffb6c1055d0 0x7ffb6c0fed70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:17:41.989 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:41.984+0000 7ffb6a59c640 1 --2- 172.21.15.46:0/2731648025 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ffb6c1008a0 0x7ffb6c0fe2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:17:41.989 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:41.985+0000 7ffb63fff640 1 --2- 172.21.15.46:0/2731648025 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffb6c101240 0x7ffb6c0fe830 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7ffb5400e9f0 tx=0x7ffb5400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:17:41.989 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:41.985+0000 7ffb637fe640 1 -- 172.21.15.46:0/2731648025 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb5400cd80 con 0x7ffb6c101240 2024-03-31T23:17:41.989 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:41.985+0000 7ffb70f65640 1 -- 172.21.15.46:0/2731648025 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffb6c1a66f0 con 0x7ffb6c101240 2024-03-31T23:17:41.989 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:41.985+0000 7ffb637fe640 1 -- 172.21.15.46:0/2731648025 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ffb540108e0 con 0x7ffb6c101240 2024-03-31T23:17:41.989 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:41.985+0000 7ffb637fe640 1 -- 172.21.15.46:0/2731648025 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb54018920 con 0x7ffb6c101240 2024-03-31T23:17:41.989 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:41.985+0000 7ffb70f65640 1 -- 172.21.15.46:0/2731648025 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffb6c1a6c90 con 0x7ffb6c101240 2024-03-31T23:17:41.995 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:41.987+0000 7ffb637fe640 1 -- 172.21.15.46:0/2731648025 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 25) ==== 137451+0+0 (secure 0 0 0) 0x7ffb54010a90 con 0x7ffb6c101240 2024-03-31T23:17:41.995 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:41.987+0000 7ffb73c25740 1 Processor -- start 2024-03-31T23:17:41.995 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:41.988+0000 7ffb73c25740 1 -- start start 2024-03-31T23:17:41.995 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:41.988+0000 7ffb73c25740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557e802cb6c0 0x557e802cbaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:17:41.995 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:41.988+0000 7ffb73c25740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557e802cbfe0 0x557e802d4870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:17:41.996 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:41.988+0000 7ffb73c25740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557e802d4db0 0x557e802d71a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:17:41.996 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:41.988+0000 7ffb73c25740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x557e8022e8f0 con 0x557e802cb6c0 2024-03-31T23:17:41.996 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:41.988+0000 7ffb73c25740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x557e801d78d0 con 0x557e802d4db0 2024-03-31T23:17:41.996 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:41.988+0000 7ffb73c25740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x557e8022a990 con 0x557e802cbfe0 2024-03-31T23:17:41.996 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:41.988+0000 7ffb637fe640 1 --2- 172.21.15.46:0/2731648025 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7ffb580a26b0 0x7ffb580a4b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:17:41.996 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:41.988+0000 7ffb6a59c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557e802cbfe0 0x557e802d4870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:17:41.996 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:41.988+0000 7ffb6ad9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557e802cb6c0 0x557e802cbaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:17:41.996 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:41.988+0000 7ffb6a59c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557e802cbfe0 0x557e802d4870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.46:40878/0 (socket says 172.21.15.46:40878) 2024-03-31T23:17:41.996 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:41.988+0000 7ffb6a59c640 1 -- 172.21.15.46:0/2128309946 learned_addr learned my addr 172.21.15.46:0/2128309946 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:17:41.996 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:41.988+0000 7ffb63fff640 1 --2- 172.21.15.46:0/2128309946 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557e802d4db0 0x557e802d71a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:17:41.996 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:41.988+0000 7ffb6ad9d640 1 --2- 172.21.15.46:0/2731648025 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7ffb580a26b0 0x7ffb580a4b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:17:41.996 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:41.988+0000 7ffb637fe640 1 -- 172.21.15.46:0/2731648025 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7ffb540cf1d0 con 0x7ffb6c101240 2024-03-31T23:17:41.996 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:41.988+0000 7ffb6a59c640 1 -- 172.21.15.46:0/2128309946 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557e802d4db0 msgr2=0x557e802d71a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:17:41.997 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:41.988+0000 7ffb6a59c640 1 --2- 172.21.15.46:0/2128309946 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557e802d4db0 0x557e802d71a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:17:41.997 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:41.988+0000 7ffb6a59c640 1 -- 172.21.15.46:0/2128309946 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557e802cb6c0 msgr2=0x557e802cbaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:17:41.997 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:41.988+0000 7ffb6a59c640 1 --2- 172.21.15.46:0/2128309946 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557e802cb6c0 0x557e802cbaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:17:41.997 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:41.988+0000 7ffb6a59c640 1 -- 172.21.15.46:0/2128309946 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557e7ff43210 con 0x557e802cbfe0 2024-03-31T23:17:41.997 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:41.989+0000 7ffb63fff640 1 --2- 172.21.15.46:0/2128309946 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557e802d4db0 0x557e802d71a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T23:17:41.997 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:41.989+0000 7ffb6a59c640 1 --2- 172.21.15.46:0/2128309946 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557e802cbfe0 0x557e802d4870 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7ffb5c0114c0 tx=0x7ffb5c011980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=78f6a7402ee41e78 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:17:41.997 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:41.989+0000 7ffb6ad9d640 1 --2- 172.21.15.46:0/2731648025 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7ffb580a26b0 0x7ffb580a4b70 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7ffb64004af0 tx=0x7ffb64005c50 comp rx=0 tx=0).ready entity=mgr.15654 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:17:41.997 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:41.989+0000 7ffb60ff9640 1 -- 172.21.15.46:0/2128309946 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb5c004020 con 0x557e802cbfe0 2024-03-31T23:17:41.997 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:41.989+0000 7ffb60ff9640 1 -- 172.21.15.46:0/2128309946 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ffb5c0041c0 con 0x557e802cbfe0 2024-03-31T23:17:41.997 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:41.989+0000 7ffb60ff9640 1 -- 172.21.15.46:0/2128309946 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb5c00f800 con 0x557e802cbfe0 2024-03-31T23:17:41.997 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:41.990+0000 7ffb73c25740 1 -- 172.21.15.46:0/2128309946 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557e802cbfe0 msgr2=0x557e802d4870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:17:41.997 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:41.990+0000 7ffb73c25740 1 --2- 172.21.15.46:0/2128309946 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557e802cbfe0 0x557e802d4870 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7ffb5c0114c0 tx=0x7ffb5c011980 comp rx=0 tx=0).stop 2024-03-31T23:17:41.997 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:41.990+0000 7ffb73c25740 1 -- 172.21.15.46:0/2128309946 shutdown_connections 2024-03-31T23:17:41.997 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:41.990+0000 7ffb73c25740 1 --2- 172.21.15.46:0/2128309946 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557e802d4db0 0x557e802d71a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:17:41.998 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:41.990+0000 7ffb73c25740 1 --2- 172.21.15.46:0/2128309946 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557e802cbfe0 0x557e802d4870 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:17:41.998 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:41.990+0000 7ffb73c25740 1 --2- 172.21.15.46:0/2128309946 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557e802cb6c0 0x557e802cbaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:17:41.998 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:41.990+0000 7ffb73c25740 1 -- 172.21.15.46:0/2128309946 >> 172.21.15.46:0/2128309946 conn(0x557e802c5fe0 msgr2=0x557e802c63c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:17:41.998 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:41.990+0000 7ffb73c25740 1 -- 172.21.15.46:0/2128309946 shutdown_connections 2024-03-31T23:17:41.998 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:41.990+0000 7ffb73c25740 1 -- 172.21.15.46:0/2128309946 wait complete. 2024-03-31T23:17:41.998 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:41.990+0000 7ffb73c25740 1 -- start start 2024-03-31T23:17:41.998 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:41.990+0000 7ffb73c25740 1 Processor -- start 2024-03-31T23:17:41.998 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:41.990+0000 7ffb73c25740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557e802cb6c0 0x557e802d1e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:17:41.998 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:41.990+0000 7ffb73c25740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557e802cbfe0 0x557e802d2390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:17:41.998 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:41.990+0000 7ffb73c25740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557e802d4db0 0x557e802d28d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:17:41.998 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:41.990+0000 7ffb73c25740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x557e802d79e0 con 0x557e802cbfe0 2024-03-31T23:17:41.998 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:41.990+0000 7ffb73c25740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x557e801ccf80 con 0x557e802cb6c0 2024-03-31T23:17:41.998 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:41.990+0000 7ffb73c25740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x557e802d7b60 con 0x557e802d4db0 2024-03-31T23:17:41.999 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:41.990+0000 7ffb73c25740 10 client.0 _ll_register_callbacks cb 0x557e80225290 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T23:17:41.999 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:41.990+0000 7ffb63fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557e802cbfe0 0x557e802d2390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:17:41.999 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:41.991+0000 7ffb63fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557e802cbfe0 0x557e802d2390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:50910/0 (socket says 172.21.15.46:50910) 2024-03-31T23:17:41.999 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:41.991+0000 7ffb63fff640 1 -- 172.21.15.46:0/4127829829 learned_addr learned my addr 172.21.15.46:0/4127829829 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:17:41.999 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:41.991+0000 7ffb6ad9d640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557e802d4db0 0x557e802d28d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:17:41.999 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:41.991+0000 7ffb6a59c640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557e802cb6c0 0x557e802d1e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:17:41.999 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:41.991+0000 7ffb63fff640 1 -- 172.21.15.46:0/4127829829 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557e802d4db0 msgr2=0x557e802d28d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:17:41.999 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:41.991+0000 7ffb63fff640 1 --2- 172.21.15.46:0/4127829829 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557e802d4db0 0x557e802d28d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:17:41.999 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:41.991+0000 7ffb63fff640 1 -- 172.21.15.46:0/4127829829 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557e802cb6c0 msgr2=0x557e802d1e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:17:41.999 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:41.991+0000 7ffb63fff640 1 --2- 172.21.15.46:0/4127829829 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557e802cb6c0 0x557e802d1e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:17:41.999 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:41.991+0000 7ffb63fff640 1 -- 172.21.15.46:0/4127829829 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557e8000f620 con 0x557e802cbfe0 2024-03-31T23:17:41.999 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:41.991+0000 7ffb6ad9d640 1 --2- 172.21.15.46:0/4127829829 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557e802d4db0 0x557e802d28d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:17:41.999 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:41.991+0000 7ffb6a59c640 1 --2- 172.21.15.46:0/4127829829 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557e802cb6c0 0x557e802d1e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:17:42.000 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:41.991+0000 7ffb63fff640 1 --2- 172.21.15.46:0/4127829829 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557e802cbfe0 0x557e802d2390 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7ffb5409b5e0 tx=0x7ffb54014420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e30cf61c43b33320 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:17:42.000 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:41.991+0000 7ffb51ffb640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-31T23:17:42.000 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:41.991+0000 7ffb51ffb640 1 -- 172.21.15.46:0/4127829829 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb540978f0 con 0x557e802cbfe0 2024-03-31T23:17:42.000 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:41.991+0000 7ffb51ffb640 1 -- 172.21.15.46:0/4127829829 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7ffb54084d90 con 0x557e802cbfe0 2024-03-31T23:17:42.000 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:41.991+0000 7ffb73c25740 1 -- 172.21.15.46:0/4127829829 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ffb5c0041c0 con 0x557e802cbfe0 2024-03-31T23:17:42.000 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:41.991+0000 7ffb51ffb640 1 -- 172.21.15.46:0/4127829829 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7ffb54010de0 con 0x557e802cbfe0 2024-03-31T23:17:42.000 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:41.991+0000 7ffb73c25740 10 client.16635 fetch_fsmap learned FSMap version 30 2024-03-31T23:17:42.000 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:41.991+0000 7ffb73c25740 1 -- 172.21.15.46:0/4127829829 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x557e802e3880 con 0x557e802cbfe0 2024-03-31T23:17:42.000 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:41.992+0000 7ffb51ffb640 1 -- 172.21.15.46:0/4127829829 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ffb5c0041c0 con 0x557e802cbfe0 2024-03-31T23:17:42.000 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:41.992+0000 7ffb51ffb640 1 -- 172.21.15.46:0/4127829829 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 30) ==== 8346+0+0 (secure 0 0 0) 0x7ffb5409aaa0 con 0x557e802cbfe0 2024-03-31T23:17:42.000 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:41.992+0000 7ffb73c25740 10 client.16635 fetch_fsmap finished waiting for FSMap version 30 2024-03-31T23:17:42.000 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:41.992+0000 7ffb73c25740 10 client.16635 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-31T23:17:42.000 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:41.992+0000 7ffb73c25740 20 client.16635 populate_metadata read hostname 'smithi046' 2024-03-31T23:17:42.000 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:41.992+0000 7ffb73c25740 1 --2- 172.21.15.46:0/4127829829 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x557e802e4d40 0x557e802e7170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:17:42.001 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:41.992+0000 7ffb73c25740 4 client.16635 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-31T23:17:42.001 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:41.992+0000 7ffb73c25740 1 -- 172.21.15.46:0/4127829829 --> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557e80242120 con 0x557e802e4d40 2024-03-31T23:17:42.001 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:41.992+0000 7ffb6a59c640 1 --2- 172.21.15.46:0/4127829829 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x557e802e4d40 0x557e802e7170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:17:42.001 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:41.992+0000 7ffb6a59c640 1 --2- 172.21.15.46:0/4127829829 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x557e802e4d40 0x557e802e7170 crc :-1 s=READY pgs=127 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6530293ce5079106 server_cookie=c5125dee5d634b6e in_seq=0 out_seq=0 2024-03-31T23:17:42.001 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:41.992+0000 7ffb51ffb640 10 client.16635 ms_handle_connect on v2:172.21.15.142:6838/2519143918 2024-03-31T23:17:42.001 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:41.993+0000 7ffb51ffb640 1 -- 172.21.15.46:0/4127829829 <== mds.0 v2:172.21.15.142:6838/2519143918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x557e80242120 con 0x557e802e4d40 2024-03-31T23:17:42.001 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:41.993+0000 7ffb51ffb640 10 client.16635 handle_command_reply: tid=0 2024-03-31T23:17:42.001 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:41.993+0000 7ffb73c25740 1 client.16635 shutdown 2024-03-31T23:17:42.001 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:41.993+0000 7ffb73c25740 1 -- 172.21.15.46:0/4127829829 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557e802cbfe0 msgr2=0x557e802d2390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:17:42.001 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:41.993+0000 7ffb73c25740 1 --2- 172.21.15.46:0/4127829829 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557e802cbfe0 0x557e802d2390 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7ffb5409b5e0 tx=0x7ffb54014420 comp rx=0 tx=0).stop 2024-03-31T23:17:42.001 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:41.994+0000 7ffb73c25740 1 -- 172.21.15.46:0/4127829829 shutdown_connections 2024-03-31T23:17:42.001 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:41.994+0000 7ffb73c25740 1 --2- 172.21.15.46:0/4127829829 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x557e802e4d40 0x557e802e7170 crc :-1 s=READY pgs=127 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:17:42.001 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:41.994+0000 7ffb73c25740 1 --2- 172.21.15.46:0/4127829829 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557e802d4db0 0x557e802d28d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:17:42.002 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:41.994+0000 7ffb73c25740 1 --2- 172.21.15.46:0/4127829829 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557e802cbfe0 0x557e802d2390 unknown :-1 s=CLOSED pgs=723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:17:42.002 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:41.994+0000 7ffb73c25740 1 --2- 172.21.15.46:0/4127829829 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557e802cb6c0 0x557e802d1e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:17:42.002 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:41.994+0000 7ffb73c25740 1 -- 172.21.15.46:0/4127829829 >> 172.21.15.46:0/4127829829 conn(0x557e802c5fe0 msgr2=0x557e802c7d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:17:42.002 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:41.994+0000 7ffb51ffb640 0 client.16635 ms_handle_reset on v2:172.21.15.142:6838/2519143918 2024-03-31T23:17:42.002 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:41.994+0000 7ffb73c25740 1 -- 172.21.15.46:0/4127829829 shutdown_connections 2024-03-31T23:17:42.002 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:41.994+0000 7ffb73c25740 1 -- 172.21.15.46:0/4127829829 wait complete. 2024-03-31T23:17:42.002 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:41.994+0000 7ffb73c25740 20 client.16635 trim_cache size 0 max 16384 2024-03-31T23:17:42.012 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:42.010+0000 7ffb73c25740 1 Processor -- start 2024-03-31T23:17:42.012 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:42.010+0000 7ffb73c25740 1 -- start start 2024-03-31T23:17:42.012 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:42.010+0000 7ffb73c25740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557e802cb6c0 0x557e802d4580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:17:42.012 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:42.010+0000 7ffb73c25740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557e802cbfe0 0x557e802e0780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:17:42.012 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:42.010+0000 7ffb73c25740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557e802d4db0 0x557e802eca50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:17:42.012 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:42.010+0000 7ffb73c25740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7ffb6c1b35f0 con 0x557e802d4db0 2024-03-31T23:17:42.012 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:42.010+0000 7ffb73c25740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7ffb6c1b0ec0 con 0x557e802cbfe0 2024-03-31T23:17:42.012 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:42.010+0000 7ffb73c25740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7ffb6c1aed10 con 0x557e802cb6c0 2024-03-31T23:17:42.013 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:42.010+0000 7ffb6ad9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557e802d4db0 0x557e802eca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:17:42.013 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:42.010+0000 7ffb6a59c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557e802cb6c0 0x557e802d4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:17:42.013 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:42.010+0000 7ffb6a59c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557e802cb6c0 0x557e802d4580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.46:40892/0 (socket says 172.21.15.46:40892) 2024-03-31T23:17:42.013 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:42.010+0000 7ffb6a59c640 1 -- 172.21.15.46:0/237662087 learned_addr learned my addr 172.21.15.46:0/237662087 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:17:42.013 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:42.010+0000 7ffb63fff640 1 --2- 172.21.15.46:0/237662087 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557e802cbfe0 0x557e802e0780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:17:42.013 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:42.010+0000 7ffb6a59c640 1 -- 172.21.15.46:0/237662087 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557e802cbfe0 msgr2=0x557e802e0780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:17:42.013 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:42.010+0000 7ffb6a59c640 1 --2- 172.21.15.46:0/237662087 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557e802cbfe0 0x557e802e0780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:17:42.013 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:42.010+0000 7ffb6a59c640 1 -- 172.21.15.46:0/237662087 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557e802d4db0 msgr2=0x557e802eca50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:17:42.013 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:42.010+0000 7ffb6a59c640 1 --2- 172.21.15.46:0/237662087 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557e802d4db0 0x557e802eca50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:17:42.013 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:42.010+0000 7ffb6a59c640 1 -- 172.21.15.46:0/237662087 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557e802cbe10 con 0x557e802cb6c0 2024-03-31T23:17:42.013 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:42.011+0000 7ffb6a59c640 1 --2- 172.21.15.46:0/237662087 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557e802cb6c0 0x557e802d4580 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7ffb5c009380 tx=0x7ffb5c009d10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c261d36dbf7b7e07 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:17:42.013 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:42.011+0000 7ffb53fff640 1 -- 172.21.15.46:0/237662087 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb5c02e070 con 0x557e802cb6c0 2024-03-31T23:17:42.013 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:42.011+0000 7ffb53fff640 1 -- 172.21.15.46:0/237662087 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ffb5c004060 con 0x557e802cb6c0 2024-03-31T23:17:42.013 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:42.011+0000 7ffb53fff640 1 -- 172.21.15.46:0/237662087 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb5c01d040 con 0x557e802cb6c0 2024-03-31T23:17:42.017 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:42.012+0000 7ffb73c25740 1 -- 172.21.15.46:0/237662087 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557e802cb6c0 msgr2=0x557e802d4580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:17:42.017 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:42.012+0000 7ffb73c25740 1 --2- 172.21.15.46:0/237662087 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557e802cb6c0 0x557e802d4580 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7ffb5c009380 tx=0x7ffb5c009d10 comp rx=0 tx=0).stop 2024-03-31T23:17:42.017 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:42.012+0000 7ffb73c25740 1 -- 172.21.15.46:0/237662087 shutdown_connections 2024-03-31T23:17:42.017 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:42.012+0000 7ffb73c25740 1 --2- 172.21.15.46:0/237662087 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557e802d4db0 0x557e802eca50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:17:42.017 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:42.012+0000 7ffb73c25740 1 --2- 172.21.15.46:0/237662087 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557e802cbfe0 0x557e802e0780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:17:42.017 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:42.012+0000 7ffb73c25740 1 --2- 172.21.15.46:0/237662087 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557e802cb6c0 0x557e802d4580 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:17:42.017 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:42.012+0000 7ffb73c25740 1 -- 172.21.15.46:0/237662087 >> 172.21.15.46:0/237662087 conn(0x557e802c5fe0 msgr2=0x557e802cd130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:17:42.018 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:42.012+0000 7ffb73c25740 1 -- 172.21.15.46:0/237662087 shutdown_connections 2024-03-31T23:17:42.018 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:42.012+0000 7ffb73c25740 1 -- 172.21.15.46:0/237662087 wait complete. 2024-03-31T23:17:42.018 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:42.012+0000 7ffb73c25740 1 -- start start 2024-03-31T23:17:42.018 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:42.012+0000 7ffb73c25740 1 Processor -- start 2024-03-31T23:17:42.018 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:42.012+0000 7ffb73c25740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557e802cb6c0 0x557e802dd190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:17:42.018 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:42.012+0000 7ffb73c25740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557e802cbfe0 0x557e802d8500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:17:42.018 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:42.012+0000 7ffb73c25740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557e802d4db0 0x557e802d8c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:17:42.018 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:42.012+0000 7ffb73c25740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7ffb6c1af380 con 0x557e802cbfe0 2024-03-31T23:17:42.018 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:42.012+0000 7ffb73c25740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7ffb6c1b07f0 con 0x557e802d4db0 2024-03-31T23:17:42.018 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:42.012+0000 7ffb73c25740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7ffb6c1afa50 con 0x557e802cb6c0 2024-03-31T23:17:42.018 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:42.012+0000 7ffb73c25740 10 client.0 _ll_register_callbacks cb 0x7ffb5400e940 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T23:17:42.018 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:42.012+0000 7ffb63fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557e802cbfe0 0x557e802d8500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:17:42.018 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:42.012+0000 7ffb63fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557e802cbfe0 0x557e802d8500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:50938/0 (socket says 172.21.15.46:50938) 2024-03-31T23:17:42.019 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:42.012+0000 7ffb63fff640 1 -- 172.21.15.46:0/1294130628 learned_addr learned my addr 172.21.15.46:0/1294130628 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:17:42.019 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:42.012+0000 7ffb6ad9d640 1 --2- 172.21.15.46:0/1294130628 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557e802d4db0 0x557e802d8c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:17:42.019 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:42.013+0000 7ffb6a59c640 1 --2- 172.21.15.46:0/1294130628 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557e802cb6c0 0x557e802dd190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:17:42.019 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:42.013+0000 7ffb63fff640 1 -- 172.21.15.46:0/1294130628 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557e802cb6c0 msgr2=0x557e802dd190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:17:42.019 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:42.013+0000 7ffb63fff640 1 --2- 172.21.15.46:0/1294130628 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557e802cb6c0 0x557e802dd190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:17:42.019 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:42.013+0000 7ffb63fff640 1 -- 172.21.15.46:0/1294130628 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557e802d4db0 msgr2=0x557e802d8c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:17:42.019 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:42.013+0000 7ffb63fff640 1 --2- 172.21.15.46:0/1294130628 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557e802d4db0 0x557e802d8c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:17:42.019 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:42.013+0000 7ffb63fff640 1 -- 172.21.15.46:0/1294130628 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557e8034a080 con 0x557e802cbfe0 2024-03-31T23:17:42.019 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:42.013+0000 7ffb63fff640 1 --2- 172.21.15.46:0/1294130628 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557e802cbfe0 0x557e802d8500 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7ffb540ccb70 tx=0x7ffb54009700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a9d25147b6681f7 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:17:42.019 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:42.013+0000 7ffb51ffb640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-31T23:17:42.019 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:42.013+0000 7ffb51ffb640 1 -- 172.21.15.46:0/1294130628 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb540978f0 con 0x557e802cbfe0 2024-03-31T23:17:42.019 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:42.013+0000 7ffb51ffb640 1 -- 172.21.15.46:0/1294130628 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7ffb54084b90 con 0x557e802cbfe0 2024-03-31T23:17:42.019 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:42.013+0000 7ffb73c25740 1 -- 172.21.15.46:0/1294130628 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ffb5c004060 con 0x557e802cbfe0 2024-03-31T23:17:42.020 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:42.013+0000 7ffb51ffb640 1 -- 172.21.15.46:0/1294130628 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7ffb54014230 con 0x557e802cbfe0 2024-03-31T23:17:42.020 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:42.013+0000 7ffb73c25740 10 client.16641 fetch_fsmap learned FSMap version 30 2024-03-31T23:17:42.020 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:42.013+0000 7ffb73c25740 1 -- 172.21.15.46:0/1294130628 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x557e8034b1c0 con 0x557e802cbfe0 2024-03-31T23:17:42.020 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:42.013+0000 7ffb51ffb640 1 -- 172.21.15.46:0/1294130628 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ffb5c004060 con 0x557e802cbfe0 2024-03-31T23:17:42.020 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:42.014+0000 7ffb51ffb640 1 -- 172.21.15.46:0/1294130628 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 30) ==== 8346+0+0 (secure 0 0 0) 0x7ffb540bb0b0 con 0x557e802cbfe0 2024-03-31T23:17:42.020 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:42.014+0000 7ffb73c25740 10 client.16641 fetch_fsmap finished waiting for FSMap version 30 2024-03-31T23:17:42.020 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:42.014+0000 7ffb73c25740 10 client.16641 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-31T23:17:42.020 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:42.014+0000 7ffb73c25740 20 client.16641 populate_metadata read hostname 'smithi046' 2024-03-31T23:17:42.020 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:42.014+0000 7ffb73c25740 1 --2- 172.21.15.46:0/1294130628 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x557e8034b760 0x557e8034db80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:17:42.020 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:42.014+0000 7ffb73c25740 4 client.16641 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-31T23:17:42.020 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:42.014+0000 7ffb73c25740 1 -- 172.21.15.46:0/1294130628 --> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] -- command(tid 0: {"prefix": "flush journal"}) -- 0x557e8023be30 con 0x557e8034b760 2024-03-31T23:17:42.020 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:42.014+0000 7ffb6a59c640 1 --2- 172.21.15.46:0/1294130628 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x557e8034b760 0x557e8034db80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:17:42.020 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:42.014+0000 7ffb6a59c640 1 --2- 172.21.15.46:0/1294130628 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x557e8034b760 0x557e8034db80 crc :-1 s=READY pgs=128 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cfbb8182ff7cde70 server_cookie=14e77d307ffdefc in_seq=0 out_seq=0 2024-03-31T23:17:42.020 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:42.014+0000 7ffb51ffb640 10 client.16641 ms_handle_connect on v2:172.21.15.142:6838/2519143918 2024-03-31T23:17:42.036 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:42.034+0000 7ffb51ffb640 1 -- 172.21.15.46:0/1294130628 <== mds.0 v2:172.21.15.142:6838/2519143918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x557e8023be30 con 0x557e8034b760 2024-03-31T23:17:42.036 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:42.034+0000 7ffb51ffb640 10 client.16641 handle_command_reply: tid=0 2024-03-31T23:17:42.036 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:42.034+0000 7ffb73c25740 1 client.16641 shutdown 2024-03-31T23:17:42.036 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:42.034+0000 7ffb73c25740 1 -- 172.21.15.46:0/1294130628 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557e802cbfe0 msgr2=0x557e802d8500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:17:42.036 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:42.034+0000 7ffb73c25740 1 --2- 172.21.15.46:0/1294130628 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557e802cbfe0 0x557e802d8500 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7ffb540ccb70 tx=0x7ffb54009700 comp rx=0 tx=0).stop 2024-03-31T23:17:42.036 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:42.034+0000 7ffb73c25740 1 -- 172.21.15.46:0/1294130628 shutdown_connections 2024-03-31T23:17:42.036 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:42.034+0000 7ffb73c25740 1 --2- 172.21.15.46:0/1294130628 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x557e8034b760 0x557e8034db80 crc :-1 s=READY pgs=128 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:17:42.037 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:42.034+0000 7ffb73c25740 1 --2- 172.21.15.46:0/1294130628 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557e802d4db0 0x557e802d8c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:17:42.037 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:42.034+0000 7ffb73c25740 1 --2- 172.21.15.46:0/1294130628 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557e802cbfe0 0x557e802d8500 unknown :-1 s=CLOSED pgs=724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:17:42.037 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:42.034+0000 7ffb51ffb640 0 client.16641 ms_handle_reset on v2:172.21.15.142:6838/2519143918 2024-03-31T23:17:42.037 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:42.034+0000 7ffb73c25740 1 --2- 172.21.15.46:0/1294130628 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557e802cb6c0 0x557e802dd190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:17:42.037 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:42.034+0000 7ffb73c25740 1 -- 172.21.15.46:0/1294130628 >> 172.21.15.46:0/1294130628 conn(0x557e802c5fe0 msgr2=0x557e802c7350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:17:42.037 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:42.034+0000 7ffb73c25740 1 -- 172.21.15.46:0/1294130628 shutdown_connections 2024-03-31T23:17:42.037 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:42.034+0000 7ffb73c25740 1 -- 172.21.15.46:0/1294130628 wait complete. 2024-03-31T23:17:42.037 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:42.035+0000 7ffb73c25740 20 client.16641 trim_cache size 0 max 16384 2024-03-31T23:17:42.037 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout:{ 2024-03-31T23:17:42.038 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout: "message": "", 2024-03-31T23:17:42.038 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout: "return_code": 0 2024-03-31T23:17:42.038 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout:} 2024-03-31T23:17:42.038 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:42.035+0000 7ffb70f65640 1 -- 172.21.15.46:0/2731648025 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7ffb580a26b0 msgr2=0x7ffb580a4b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:17:42.038 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:42.035+0000 7ffb70f65640 1 --2- 172.21.15.46:0/2731648025 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7ffb580a26b0 0x7ffb580a4b70 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7ffb64004af0 tx=0x7ffb64005c50 comp rx=0 tx=0).stop 2024-03-31T23:17:42.039 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:42.035+0000 7ffb70f65640 1 -- 172.21.15.46:0/2731648025 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffb6c101240 msgr2=0x7ffb6c0fe830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:17:42.039 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:42.035+0000 7ffb70f65640 1 --2- 172.21.15.46:0/2731648025 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffb6c101240 0x7ffb6c0fe830 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7ffb5400e9f0 tx=0x7ffb5400eeb0 comp rx=0 tx=0).stop 2024-03-31T23:17:42.039 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:42.035+0000 7ffb70f65640 1 -- 172.21.15.46:0/2731648025 shutdown_connections 2024-03-31T23:17:42.039 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:42.035+0000 7ffb70f65640 1 --2- 172.21.15.46:0/2731648025 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7ffb580a26b0 0x7ffb580a4b70 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:17:42.039 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:42.035+0000 7ffb70f65640 1 --2- 172.21.15.46:0/2731648025 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ffb6c1055d0 0x7ffb6c0fed70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:17:42.039 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:42.035+0000 7ffb70f65640 1 --2- 172.21.15.46:0/2731648025 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffb6c101240 0x7ffb6c0fe830 unknown :-1 s=CLOSED pgs=722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:17:42.039 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:42.035+0000 7ffb70f65640 1 --2- 172.21.15.46:0/2731648025 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ffb6c1008a0 0x7ffb6c0fe2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:17:42.039 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:42.035+0000 7ffb70f65640 1 -- 172.21.15.46:0/2731648025 >> 172.21.15.46:0/2731648025 conn(0x7ffb6c077420 msgr2=0x7ffb6c06a230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:17:42.039 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:42.035+0000 7ffb70f65640 1 -- 172.21.15.46:0/2731648025 shutdown_connections 2024-03-31T23:17:42.039 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:42.035+0000 7ffb70f65640 1 -- 172.21.15.46:0/2731648025 wait complete. 2024-03-31T23:17:42.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:17:41 smithi046 ceph-mon[32498]: pgmap v182: 129 pgs: 129 active+clean; 88 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 421 KiB/s rd, 1.2 MiB/s wr, 117 op/s 2024-03-31T23:17:42.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:17:41 smithi142 ceph-mon[35271]: pgmap v182: 129 pgs: 129 active+clean; 88 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 421 KiB/s rd, 1.2 MiB/s wr, 117 op/s 2024-03-31T23:17:42.199 INFO:tasks.workunit.client.0.smithi046.stdout: 1 131538 2.50 MB/sec execute 168 sec latency 15.756 ms 2024-03-31T23:17:42.295 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:17:41 smithi162 ceph-mon[36237]: pgmap v182: 129 pgs: 129 active+clean; 88 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 421 KiB/s rd, 1.2 MiB/s wr, 117 op/s 2024-03-31T23:17:43.199 INFO:tasks.workunit.client.0.smithi046.stdout: 1 131788 2.48 MB/sec execute 169 sec latency 15.465 ms 2024-03-31T23:17:44.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:17:43 smithi046 ceph-mon[32498]: pgmap v183: 129 pgs: 129 active+clean; 88 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 421 KiB/s rd, 1.2 MiB/s wr, 117 op/s 2024-03-31T23:17:44.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:17:43 smithi142 ceph-mon[35271]: pgmap v183: 129 pgs: 129 active+clean; 88 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 421 KiB/s rd, 1.2 MiB/s wr, 117 op/s 2024-03-31T23:17:44.199 INFO:tasks.workunit.client.0.smithi046.stdout: 1 132203 2.47 MB/sec execute 170 sec latency 15.905 ms 2024-03-31T23:17:44.296 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:17:43 smithi162 ceph-mon[36237]: pgmap v183: 129 pgs: 129 active+clean; 88 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 421 KiB/s rd, 1.2 MiB/s wr, 117 op/s 2024-03-31T23:17:45.199 INFO:tasks.workunit.client.0.smithi046.stdout: 1 132724 2.49 MB/sec execute 171 sec latency 17.298 ms 2024-03-31T23:17:46.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:17:45 smithi046 ceph-mon[32498]: pgmap v184: 129 pgs: 129 active+clean; 89 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 626 KiB/s rd, 1.9 MiB/s wr, 176 op/s 2024-03-31T23:17:46.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:17:45 smithi142 ceph-mon[35271]: pgmap v184: 129 pgs: 129 active+clean; 89 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 626 KiB/s rd, 1.9 MiB/s wr, 176 op/s 2024-03-31T23:17:46.199 INFO:tasks.workunit.client.0.smithi046.stdout: 1 133185 2.47 MB/sec execute 172 sec latency 29.713 ms 2024-03-31T23:17:46.295 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:17:45 smithi162 ceph-mon[36237]: pgmap v184: 129 pgs: 129 active+clean; 89 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 626 KiB/s rd, 1.9 MiB/s wr, 176 op/s 2024-03-31T23:17:47.199 INFO:tasks.workunit.client.0.smithi046.stdout: 1 133708 2.47 MB/sec execute 173 sec latency 23.533 ms 2024-03-31T23:17:48.114 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:17:47 smithi046 ceph-mon[32498]: pgmap v185: 129 pgs: 129 active+clean; 89 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 431 KiB/s rd, 1.2 MiB/s wr, 120 op/s 2024-03-31T23:17:48.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:17:47 smithi142 ceph-mon[35271]: pgmap v185: 129 pgs: 129 active+clean; 89 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 431 KiB/s rd, 1.2 MiB/s wr, 120 op/s 2024-03-31T23:17:48.199 INFO:tasks.workunit.client.0.smithi046.stdout: 1 134445 2.49 MB/sec execute 174 sec latency 15.166 ms 2024-03-31T23:17:48.296 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:17:47 smithi162 ceph-mon[36237]: pgmap v185: 129 pgs: 129 active+clean; 89 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 431 KiB/s rd, 1.2 MiB/s wr, 120 op/s 2024-03-31T23:17:49.199 INFO:tasks.workunit.client.0.smithi046.stdout: 1 134809 2.50 MB/sec execute 175 sec latency 17.525 ms 2024-03-31T23:17:50.097 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:17:49 smithi162 ceph-mon[36237]: pgmap v186: 129 pgs: 129 active+clean; 90 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 626 KiB/s rd, 1.8 MiB/s wr, 175 op/s 2024-03-31T23:17:50.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:17:49 smithi046 ceph-mon[32498]: pgmap v186: 129 pgs: 129 active+clean; 90 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 626 KiB/s rd, 1.8 MiB/s wr, 175 op/s 2024-03-31T23:17:50.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:17:49 smithi142 ceph-mon[35271]: pgmap v186: 129 pgs: 129 active+clean; 90 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 626 KiB/s rd, 1.8 MiB/s wr, 175 op/s 2024-03-31T23:17:50.199 INFO:tasks.workunit.client.0.smithi046.stdout: 1 135182 2.49 MB/sec execute 176 sec latency 15.290 ms 2024-03-31T23:17:51.199 INFO:tasks.workunit.client.0.smithi046.stdout: 1 135440 2.48 MB/sec execute 177 sec latency 18.283 ms 2024-03-31T23:17:52.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:17:51 smithi046 ceph-mon[32498]: pgmap v187: 129 pgs: 129 active+clean; 90 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 400 KiB/s rd, 1.3 MiB/s wr, 114 op/s 2024-03-31T23:17:52.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:17:51 smithi142 ceph-mon[35271]: pgmap v187: 129 pgs: 129 active+clean; 90 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 400 KiB/s rd, 1.3 MiB/s wr, 114 op/s 2024-03-31T23:17:52.199 INFO:tasks.workunit.client.0.smithi046.stdout: 1 135941 2.48 MB/sec execute 178 sec latency 16.419 ms 2024-03-31T23:17:52.295 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:17:51 smithi162 ceph-mon[36237]: pgmap v187: 129 pgs: 129 active+clean; 90 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 400 KiB/s rd, 1.3 MiB/s wr, 114 op/s 2024-03-31T23:17:53.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:17:52 smithi046 ceph-mon[32498]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-31T23:17:53.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:17:52 smithi142 ceph-mon[35271]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-31T23:17:53.199 INFO:tasks.workunit.client.0.smithi046.stdout: 1 136415 2.48 MB/sec execute 179 sec latency 21.136 ms 2024-03-31T23:17:53.295 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:17:52 smithi162 ceph-mon[36237]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-31T23:17:54.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:17:53 smithi046 ceph-mon[32498]: pgmap v188: 129 pgs: 129 active+clean; 90 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 400 KiB/s rd, 1.3 MiB/s wr, 114 op/s 2024-03-31T23:17:54.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:17:53 smithi046 ceph-mon[32498]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:17:54.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:17:53 smithi046 ceph-mon[32498]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-31T23:17:54.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:17:53 smithi046 ceph-mon[32498]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' 2024-03-31T23:17:54.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:17:53 smithi142 ceph-mon[35271]: pgmap v188: 129 pgs: 129 active+clean; 90 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 400 KiB/s rd, 1.3 MiB/s wr, 114 op/s 2024-03-31T23:17:54.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:17:53 smithi142 ceph-mon[35271]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:17:54.194 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:17:53 smithi142 ceph-mon[35271]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-31T23:17:54.194 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:17:53 smithi142 ceph-mon[35271]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' 2024-03-31T23:17:54.199 INFO:tasks.workunit.client.0.smithi046.stdout: 1 136917 2.47 MB/sec execute 180 sec latency 25.685 ms 2024-03-31T23:17:54.296 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:17:53 smithi162 ceph-mon[36237]: pgmap v188: 129 pgs: 129 active+clean; 90 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 400 KiB/s rd, 1.3 MiB/s wr, 114 op/s 2024-03-31T23:17:54.296 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:17:53 smithi162 ceph-mon[36237]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:17:54.296 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:17:53 smithi162 ceph-mon[36237]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-31T23:17:54.296 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:17:53 smithi162 ceph-mon[36237]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' 2024-03-31T23:17:55.128 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:55.127+0000 7f7ced979640 1 -- 172.21.15.46:0/3642183609 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7ce81055f0 msgr2=0x7f7ce81059d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:17:55.129 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:55.127+0000 7f7ced979640 1 --2- 172.21.15.46:0/3642183609 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7ce81055f0 0x7f7ce81059d0 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7f7ce0009ca0 tx=0x7f7ce002c0f0 comp rx=0 tx=0).stop 2024-03-31T23:17:55.129 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:55.127+0000 7f7ced979640 1 -- 172.21.15.46:0/3642183609 shutdown_connections 2024-03-31T23:17:55.129 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:55.127+0000 7f7ced979640 1 --2- 172.21.15.46:0/3642183609 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7ce8101260 0x7f7ce810d480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:17:55.129 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:55.127+0000 7f7ced979640 1 --2- 172.21.15.46:0/3642183609 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7ce81008a0 0x7f7ce8100d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:17:55.129 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:55.127+0000 7f7ced979640 1 --2- 172.21.15.46:0/3642183609 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7ce81055f0 0x7f7ce81059d0 unknown :-1 s=CLOSED pgs=725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:17:55.129 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:55.127+0000 7f7ced979640 1 -- 172.21.15.46:0/3642183609 >> 172.21.15.46:0/3642183609 conn(0x7f7ce80774a0 msgr2=0x7f7ce80778b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:17:55.129 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:55.127+0000 7f7ced979640 1 -- 172.21.15.46:0/3642183609 shutdown_connections 2024-03-31T23:17:55.129 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:55.127+0000 7f7ced979640 1 -- 172.21.15.46:0/3642183609 wait complete. 2024-03-31T23:17:55.129 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:55.127+0000 7f7ced979640 1 Processor -- start 2024-03-31T23:17:55.129 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:55.128+0000 7f7ced979640 1 -- start start 2024-03-31T23:17:55.131 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:55.128+0000 7f7ced979640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7ce81008a0 0x7f7ce8198ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:17:55.131 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:55.128+0000 7f7ced979640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7ce8101260 0x7f7ce8199220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:17:55.131 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:55.128+0000 7f7ced979640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7ce81055f0 0x7f7ce81a3d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:17:55.131 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:55.128+0000 7f7ced979640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f7ce81109c0 con 0x7f7ce81055f0 2024-03-31T23:17:55.131 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:55.128+0000 7f7ced979640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f7ce8110840 con 0x7f7ce8101260 2024-03-31T23:17:55.131 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:55.128+0000 7f7ced979640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f7ce8110b40 con 0x7f7ce81008a0 2024-03-31T23:17:55.131 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:55.128+0000 7f7ce77fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7ce81055f0 0x7f7ce81a3d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:17:55.131 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:55.128+0000 7f7ce77fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7ce81055f0 0x7f7ce81a3d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:45310/0 (socket says 172.21.15.46:45310) 2024-03-31T23:17:55.132 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:55.128+0000 7f7ce77fe640 1 -- 172.21.15.46:0/2276881099 learned_addr learned my addr 172.21.15.46:0/2276881099 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:17:55.132 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:55.128+0000 7f7ce6ffd640 1 --2- 172.21.15.46:0/2276881099 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7ce81008a0 0x7f7ce8198ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:17:55.132 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:55.128+0000 7f7ce67fc640 1 --2- 172.21.15.46:0/2276881099 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7ce8101260 0x7f7ce8199220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:17:55.132 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:55.128+0000 7f7ce77fe640 1 -- 172.21.15.46:0/2276881099 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7ce81008a0 msgr2=0x7f7ce8198ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:17:55.132 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:55.128+0000 7f7ce77fe640 1 --2- 172.21.15.46:0/2276881099 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7ce81008a0 0x7f7ce8198ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:17:55.132 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:55.128+0000 7f7ce77fe640 1 -- 172.21.15.46:0/2276881099 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7ce8101260 msgr2=0x7f7ce8199220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:17:55.132 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:55.128+0000 7f7ce77fe640 1 --2- 172.21.15.46:0/2276881099 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7ce8101260 0x7f7ce8199220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:17:55.132 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:55.128+0000 7f7ce77fe640 1 -- 172.21.15.46:0/2276881099 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ce81a42a0 con 0x7f7ce81055f0 2024-03-31T23:17:55.132 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:55.129+0000 7f7ce77fe640 1 --2- 172.21.15.46:0/2276881099 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7ce81055f0 0x7f7ce81a3d60 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7f7cd800d680 tx=0x7f7cd800db40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:17:55.132 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:55.129+0000 7f7cec977640 1 -- 172.21.15.46:0/2276881099 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cd8017070 con 0x7f7ce81055f0 2024-03-31T23:17:55.132 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:55.129+0000 7f7ced979640 1 -- 172.21.15.46:0/2276881099 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7ce81a44d0 con 0x7f7ce81055f0 2024-03-31T23:17:55.132 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:55.129+0000 7f7cec977640 1 -- 172.21.15.46:0/2276881099 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7cd8009820 con 0x7f7ce81055f0 2024-03-31T23:17:55.132 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:55.129+0000 7f7cec977640 1 -- 172.21.15.46:0/2276881099 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cd8004c40 con 0x7f7ce81055f0 2024-03-31T23:17:55.133 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:55.129+0000 7f7ced979640 1 -- 172.21.15.46:0/2276881099 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7ce81a49f0 con 0x7f7ce81055f0 2024-03-31T23:17:55.142 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:55.131+0000 7f7cec977640 1 -- 172.21.15.46:0/2276881099 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 25) ==== 137451+0+0 (secure 0 0 0) 0x7f7cd80099d0 con 0x7f7ce81055f0 2024-03-31T23:17:55.142 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:55.132+0000 7f7cf0630740 1 Processor -- start 2024-03-31T23:17:55.142 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:55.132+0000 7f7cf0630740 1 -- start start 2024-03-31T23:17:55.142 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:55.132+0000 7f7cf0630740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5566014f86c0 0x5566014f8aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:17:55.142 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:55.132+0000 7f7cf0630740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5566014f8fe0 0x556601501870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:17:55.142 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:55.132+0000 7f7cec977640 1 --2- 172.21.15.46:0/2276881099 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f7cd40a26b0 0x7f7cd40a4b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:17:55.142 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:55.132+0000 7f7cf0630740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556601501db0 0x5566015041a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:17:55.143 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:55.132+0000 7f7cf0630740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55660145b8f0 con 0x5566014f86c0 2024-03-31T23:17:55.143 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:55.132+0000 7f7ce67fc640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5566014f86c0 0x5566014f8aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:17:55.143 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:55.132+0000 7f7cf0630740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x5566014048d0 con 0x5566014f8fe0 2024-03-31T23:17:55.143 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:55.132+0000 7f7ce6ffd640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5566014f8fe0 0x556601501870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:17:55.143 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:55.132+0000 7f7cf0630740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x556601457990 con 0x556601501db0 2024-03-31T23:17:55.143 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:55.132+0000 7f7ce67fc640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5566014f86c0 0x5566014f8aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:45320/0 (socket says 172.21.15.46:45320) 2024-03-31T23:17:55.143 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:55.132+0000 7f7ce67fc640 1 -- 172.21.15.46:0/3842549760 learned_addr learned my addr 172.21.15.46:0/3842549760 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:17:55.143 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:55.132+0000 7f7ce6ffd640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5566014f8fe0 0x556601501870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.46:56604/0 (socket says 172.21.15.46:56604) 2024-03-31T23:17:55.143 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:55.132+0000 7f7ce77fe640 1 --2- 172.21.15.46:0/3842549760 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556601501db0 0x5566015041a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:17:55.143 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:55.132+0000 7f7ce67fc640 1 --2- 172.21.15.46:0/2276881099 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f7cd40a26b0 0x7f7cd40a4b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:17:55.143 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:55.132+0000 7f7cec977640 1 -- 172.21.15.46:0/2276881099 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7f7cd80ce6b0 con 0x7f7ce81055f0 2024-03-31T23:17:55.143 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:55.132+0000 7f7ce67fc640 1 -- 172.21.15.46:0/3842549760 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556601501db0 msgr2=0x5566015041a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:17:55.144 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:55.132+0000 7f7ce67fc640 1 --2- 172.21.15.46:0/3842549760 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556601501db0 0x5566015041a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:17:55.144 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:55.133+0000 7f7ce67fc640 1 -- 172.21.15.46:0/3842549760 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5566014f8fe0 msgr2=0x556601501870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:17:55.144 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:55.133+0000 7f7ce67fc640 1 --2- 172.21.15.46:0/3842549760 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5566014f8fe0 0x556601501870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:17:55.144 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:55.133+0000 7f7ce67fc640 1 -- 172.21.15.46:0/3842549760 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556601170210 con 0x5566014f86c0 2024-03-31T23:17:55.144 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:55.133+0000 7f7ce67fc640 1 --2- 172.21.15.46:0/3842549760 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5566014f86c0 0x5566014f8aa0 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f7cd0012ab0 tx=0x7f7cd0012eb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f4ef6a9350c21684 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:17:55.144 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:55.133+0000 7f7ce67fc640 1 --2- 172.21.15.46:0/2276881099 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f7cd40a26b0 0x7f7cd40a4b70 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f7cd00058a0 tx=0x7f7cd0017000 comp rx=0 tx=0).ready entity=mgr.15654 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:17:55.144 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:55.133+0000 7f7cddffb640 1 -- 172.21.15.46:0/3842549760 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cd0023070 con 0x5566014f86c0 2024-03-31T23:17:55.144 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:55.133+0000 7f7cddffb640 1 -- 172.21.15.46:0/3842549760 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7cd0012590 con 0x5566014f86c0 2024-03-31T23:17:55.144 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:55.133+0000 7f7cddffb640 1 -- 172.21.15.46:0/3842549760 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cd001f480 con 0x5566014f86c0 2024-03-31T23:17:55.144 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:55.134+0000 7f7cf0630740 1 -- 172.21.15.46:0/3842549760 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5566014f86c0 msgr2=0x5566014f8aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:17:55.144 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:55.134+0000 7f7cf0630740 1 --2- 172.21.15.46:0/3842549760 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5566014f86c0 0x5566014f8aa0 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f7cd0012ab0 tx=0x7f7cd0012eb0 comp rx=0 tx=0).stop 2024-03-31T23:17:55.144 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:55.134+0000 7f7cf0630740 1 -- 172.21.15.46:0/3842549760 shutdown_connections 2024-03-31T23:17:55.144 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:55.134+0000 7f7cf0630740 1 --2- 172.21.15.46:0/3842549760 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556601501db0 0x5566015041a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:17:55.145 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:55.134+0000 7f7cf0630740 1 --2- 172.21.15.46:0/3842549760 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5566014f8fe0 0x556601501870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:17:55.145 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:55.134+0000 7f7cf0630740 1 --2- 172.21.15.46:0/3842549760 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5566014f86c0 0x5566014f8aa0 unknown :-1 s=CLOSED pgs=727 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:17:55.145 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:55.134+0000 7f7cf0630740 1 -- 172.21.15.46:0/3842549760 >> 172.21.15.46:0/3842549760 conn(0x5566014f2fe0 msgr2=0x5566014f33c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:17:55.145 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:55.134+0000 7f7cf0630740 1 -- 172.21.15.46:0/3842549760 shutdown_connections 2024-03-31T23:17:55.145 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:55.134+0000 7f7cf0630740 1 -- 172.21.15.46:0/3842549760 wait complete. 2024-03-31T23:17:55.145 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:55.134+0000 7f7cf0630740 1 -- start start 2024-03-31T23:17:55.145 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:55.134+0000 7f7cf0630740 1 Processor -- start 2024-03-31T23:17:55.145 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:55.135+0000 7f7cf0630740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5566014f86c0 0x5566014feef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:17:55.145 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:55.135+0000 7f7cf0630740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5566014f8fe0 0x5566014ff430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:17:55.145 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:55.135+0000 7f7cf0630740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x556601501db0 0x5566014ff970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:17:55.145 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:55.135+0000 7f7cf0630740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x556601504980 con 0x5566014f8fe0 2024-03-31T23:17:55.145 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:55.135+0000 7f7cf0630740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x5566013f9f80 con 0x556601501db0 2024-03-31T23:17:55.145 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:55.135+0000 7f7cf0630740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x556601504b00 con 0x5566014f86c0 2024-03-31T23:17:55.146 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:55.135+0000 7f7cf0630740 10 client.0 _ll_register_callbacks cb 0x556601452290 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T23:17:55.146 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:55.135+0000 7f7ce67fc640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5566014f8fe0 0x5566014ff430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:17:55.146 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:55.135+0000 7f7ce77fe640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x556601501db0 0x5566014ff970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:17:55.146 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:55.135+0000 7f7ce67fc640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5566014f8fe0 0x5566014ff430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:45328/0 (socket says 172.21.15.46:45328) 2024-03-31T23:17:55.146 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:55.135+0000 7f7ce67fc640 1 -- 172.21.15.46:0/2552137503 learned_addr learned my addr 172.21.15.46:0/2552137503 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:17:55.146 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:55.135+0000 7f7ce6ffd640 1 --2- 172.21.15.46:0/2552137503 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5566014f86c0 0x5566014feef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:17:55.146 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:55.135+0000 7f7ce67fc640 1 -- 172.21.15.46:0/2552137503 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5566014f86c0 msgr2=0x5566014feef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:17:55.146 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:55.135+0000 7f7ce67fc640 1 --2- 172.21.15.46:0/2552137503 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5566014f86c0 0x5566014feef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:17:55.146 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:55.135+0000 7f7ce67fc640 1 -- 172.21.15.46:0/2552137503 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x556601501db0 msgr2=0x5566014ff970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:17:55.146 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:55.135+0000 7f7ce67fc640 1 --2- 172.21.15.46:0/2552137503 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x556601501db0 0x5566014ff970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:17:55.146 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:55.135+0000 7f7ce67fc640 1 -- 172.21.15.46:0/2552137503 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55660123c620 con 0x5566014f8fe0 2024-03-31T23:17:55.146 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:55.135+0000 7f7ce67fc640 1 --2- 172.21.15.46:0/2552137503 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5566014f8fe0 0x5566014ff430 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f7cd0005bc0 tx=0x7f7cd000dfa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6d090eb627271ab2 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:17:55.147 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:55.135+0000 7f7cc6ffd640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-31T23:17:55.147 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:55.135+0000 7f7cc6ffd640 1 -- 172.21.15.46:0/2552137503 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cd002c070 con 0x5566014f8fe0 2024-03-31T23:17:55.147 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:55.136+0000 7f7cc6ffd640 1 -- 172.21.15.46:0/2552137503 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7f7cd000d720 con 0x5566014f8fe0 2024-03-31T23:17:55.147 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:55.136+0000 7f7cf0630740 1 -- 172.21.15.46:0/2552137503 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7cd0012590 con 0x5566014f8fe0 2024-03-31T23:17:55.147 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:55.136+0000 7f7cc6ffd640 1 -- 172.21.15.46:0/2552137503 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f7cd0035050 con 0x5566014f8fe0 2024-03-31T23:17:55.147 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:55.136+0000 7f7cf0630740 10 client.16653 fetch_fsmap learned FSMap version 30 2024-03-31T23:17:55.147 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:55.136+0000 7f7cf0630740 1 -- 172.21.15.46:0/2552137503 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x556601510960 con 0x5566014f8fe0 2024-03-31T23:17:55.147 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:55.136+0000 7f7cc6ffd640 1 -- 172.21.15.46:0/2552137503 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7cd0012590 con 0x5566014f8fe0 2024-03-31T23:17:55.147 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:55.136+0000 7f7cc6ffd640 1 -- 172.21.15.46:0/2552137503 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 30) ==== 8346+0+0 (secure 0 0 0) 0x7f7cd002f020 con 0x5566014f8fe0 2024-03-31T23:17:55.147 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:55.136+0000 7f7cf0630740 10 client.16653 fetch_fsmap finished waiting for FSMap version 30 2024-03-31T23:17:55.147 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:55.136+0000 7f7cf0630740 10 client.16653 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-31T23:17:55.147 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:55.136+0000 7f7cf0630740 20 client.16653 populate_metadata read hostname 'smithi046' 2024-03-31T23:17:55.147 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:55.136+0000 7f7cf0630740 1 --2- 172.21.15.46:0/2552137503 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x556601511e20 0x556601514210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:17:55.147 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:55.136+0000 7f7cf0630740 4 client.16653 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-31T23:17:55.148 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:55.136+0000 7f7cf0630740 1 -- 172.21.15.46:0/2552137503 --> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55660146f120 con 0x556601511e20 2024-03-31T23:17:55.148 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:55.136+0000 7f7ce6ffd640 1 --2- 172.21.15.46:0/2552137503 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x556601511e20 0x556601514210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:17:55.148 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:55.137+0000 7f7ce6ffd640 1 --2- 172.21.15.46:0/2552137503 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x556601511e20 0x556601514210 crc :-1 s=READY pgs=129 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=243fff25ba1099d2 server_cookie=8d32e6145c755bce in_seq=0 out_seq=0 2024-03-31T23:17:55.148 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:55.137+0000 7f7cc6ffd640 10 client.16653 ms_handle_connect on v2:172.21.15.142:6838/2519143918 2024-03-31T23:17:55.148 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:55.137+0000 7f7cc6ffd640 1 -- 172.21.15.46:0/2552137503 <== mds.0 v2:172.21.15.142:6838/2519143918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55660146f120 con 0x556601511e20 2024-03-31T23:17:55.148 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:55.137+0000 7f7cc6ffd640 10 client.16653 handle_command_reply: tid=0 2024-03-31T23:17:55.148 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:55.138+0000 7f7cf0630740 1 client.16653 shutdown 2024-03-31T23:17:55.148 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:55.138+0000 7f7cf0630740 1 -- 172.21.15.46:0/2552137503 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5566014f8fe0 msgr2=0x5566014ff430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:17:55.148 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:55.138+0000 7f7cf0630740 1 --2- 172.21.15.46:0/2552137503 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5566014f8fe0 0x5566014ff430 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f7cd0005bc0 tx=0x7f7cd000dfa0 comp rx=0 tx=0).stop 2024-03-31T23:17:55.148 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:55.138+0000 7f7cf0630740 1 -- 172.21.15.46:0/2552137503 shutdown_connections 2024-03-31T23:17:55.148 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:55.138+0000 7f7cf0630740 1 --2- 172.21.15.46:0/2552137503 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x556601511e20 0x556601514210 crc :-1 s=READY pgs=129 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:17:55.148 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:55.138+0000 7f7cf0630740 1 --2- 172.21.15.46:0/2552137503 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x556601501db0 0x5566014ff970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:17:55.148 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:55.138+0000 7f7cf0630740 1 --2- 172.21.15.46:0/2552137503 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5566014f8fe0 0x5566014ff430 unknown :-1 s=CLOSED pgs=728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:17:55.148 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:55.138+0000 7f7cc6ffd640 0 client.16653 ms_handle_reset on v2:172.21.15.142:6838/2519143918 2024-03-31T23:17:55.149 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:55.138+0000 7f7cf0630740 1 --2- 172.21.15.46:0/2552137503 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5566014f86c0 0x5566014feef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:17:55.149 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:55.138+0000 7f7cf0630740 1 -- 172.21.15.46:0/2552137503 >> 172.21.15.46:0/2552137503 conn(0x5566014f2fe0 msgr2=0x5566014f4d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:17:55.149 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:55.138+0000 7f7cf0630740 1 -- 172.21.15.46:0/2552137503 shutdown_connections 2024-03-31T23:17:55.149 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:55.138+0000 7f7cf0630740 1 -- 172.21.15.46:0/2552137503 wait complete. 2024-03-31T23:17:55.149 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:55.138+0000 7f7cf0630740 20 client.16653 trim_cache size 0 max 16384 2024-03-31T23:17:55.156 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:55.154+0000 7f7cf0630740 1 Processor -- start 2024-03-31T23:17:55.156 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:55.154+0000 7f7cf0630740 1 -- start start 2024-03-31T23:17:55.157 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:55.154+0000 7f7cf0630740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5566014f86c0 0x556601501620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:17:55.157 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:55.154+0000 7f7cf0630740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5566014f8fe0 0x55660150d720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:17:55.157 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:55.154+0000 7f7cf0630740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x556601501db0 0x5566015197e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:17:55.157 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:55.154+0000 7f7cf0630740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f7ce81af130 con 0x5566014f8fe0 2024-03-31T23:17:55.157 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:55.154+0000 7f7cf0630740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f7ce81aca00 con 0x556601501db0 2024-03-31T23:17:55.157 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:55.154+0000 7f7cf0630740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f7ce81aa850 con 0x5566014f86c0 2024-03-31T23:17:55.157 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:55.154+0000 7f7ce6ffd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5566014f86c0 0x556601501620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:17:55.157 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:55.154+0000 7f7ce67fc640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5566014f8fe0 0x55660150d720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:17:55.157 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:55.154+0000 7f7ce6ffd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5566014f86c0 0x556601501620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.46:47262/0 (socket says 172.21.15.46:47262) 2024-03-31T23:17:55.157 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:55.154+0000 7f7ce6ffd640 1 -- 172.21.15.46:0/2615466059 learned_addr learned my addr 172.21.15.46:0/2615466059 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:17:55.157 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:55.154+0000 7f7ce77fe640 1 --2- 172.21.15.46:0/2615466059 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x556601501db0 0x5566015197e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:17:55.157 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:55.155+0000 7f7ce67fc640 1 -- 172.21.15.46:0/2615466059 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5566014f86c0 msgr2=0x556601501620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:17:55.157 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:55.155+0000 7f7ce67fc640 1 --2- 172.21.15.46:0/2615466059 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5566014f86c0 0x556601501620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:17:55.157 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:55.155+0000 7f7ce67fc640 1 -- 172.21.15.46:0/2615466059 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x556601501db0 msgr2=0x5566015197e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:17:55.158 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:55.155+0000 7f7ce67fc640 1 --2- 172.21.15.46:0/2615466059 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x556601501db0 0x5566015197e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:17:55.158 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:55.155+0000 7f7ce67fc640 1 -- 172.21.15.46:0/2615466059 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556601504aa0 con 0x5566014f8fe0 2024-03-31T23:17:55.158 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:55.155+0000 7f7ce67fc640 1 --2- 172.21.15.46:0/2615466059 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5566014f8fe0 0x55660150d720 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f7cd0003a70 tx=0x7f7cd0012eb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=83cf6f22bfd71a38 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:17:55.158 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:55.155+0000 7f7cdd7fa640 1 -- 172.21.15.46:0/2615466059 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cd002c040 con 0x5566014f8fe0 2024-03-31T23:17:55.158 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:55.155+0000 7f7cdd7fa640 1 -- 172.21.15.46:0/2615466059 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7cd000b480 con 0x5566014f8fe0 2024-03-31T23:17:55.158 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:55.155+0000 7f7cdd7fa640 1 -- 172.21.15.46:0/2615466059 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cd0023050 con 0x5566014f8fe0 2024-03-31T23:17:55.161 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:55.156+0000 7f7cf0630740 1 -- 172.21.15.46:0/2615466059 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5566014f8fe0 msgr2=0x55660150d720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:17:55.161 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:55.156+0000 7f7cf0630740 1 --2- 172.21.15.46:0/2615466059 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5566014f8fe0 0x55660150d720 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f7cd0003a70 tx=0x7f7cd0012eb0 comp rx=0 tx=0).stop 2024-03-31T23:17:55.162 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:55.156+0000 7f7cf0630740 1 -- 172.21.15.46:0/2615466059 shutdown_connections 2024-03-31T23:17:55.162 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:55.156+0000 7f7cf0630740 1 --2- 172.21.15.46:0/2615466059 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x556601501db0 0x5566015197e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:17:55.162 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:55.156+0000 7f7cf0630740 1 --2- 172.21.15.46:0/2615466059 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5566014f8fe0 0x55660150d720 unknown :-1 s=CLOSED pgs=729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:17:55.162 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:55.156+0000 7f7cf0630740 1 --2- 172.21.15.46:0/2615466059 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5566014f86c0 0x556601501620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:17:55.162 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:55.156+0000 7f7cf0630740 1 -- 172.21.15.46:0/2615466059 >> 172.21.15.46:0/2615466059 conn(0x5566014f2fe0 msgr2=0x5566014fd7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:17:55.162 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:55.156+0000 7f7cf0630740 1 -- 172.21.15.46:0/2615466059 shutdown_connections 2024-03-31T23:17:55.162 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:55.156+0000 7f7cf0630740 1 -- 172.21.15.46:0/2615466059 wait complete. 2024-03-31T23:17:55.162 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:55.156+0000 7f7cf0630740 1 -- start start 2024-03-31T23:17:55.162 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:55.157+0000 7f7cf0630740 1 Processor -- start 2024-03-31T23:17:55.162 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:55.157+0000 7f7cf0630740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5566014f86c0 0x556601509f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:17:55.162 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:55.157+0000 7f7cf0630740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5566014f8fe0 0x5566015052f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:17:55.162 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:55.157+0000 7f7cf0630740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556601501db0 0x556601505830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:17:55.162 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:55.157+0000 7f7cf0630740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f7ce81aaec0 con 0x556601501db0 2024-03-31T23:17:55.163 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:55.157+0000 7f7cf0630740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f7ce81ac330 con 0x5566014f8fe0 2024-03-31T23:17:55.163 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:55.157+0000 7f7cf0630740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f7ce81ab590 con 0x5566014f86c0 2024-03-31T23:17:55.163 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:55.157+0000 7f7cf0630740 10 client.0 _ll_register_callbacks cb 0x7f7ce0009b20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T23:17:55.163 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:55.157+0000 7f7ce77fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556601501db0 0x556601505830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:17:55.163 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:55.157+0000 7f7ce6ffd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5566014f86c0 0x556601509f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:17:55.163 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:55.157+0000 7f7ce67fc640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5566014f8fe0 0x5566015052f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:17:55.163 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:55.157+0000 7f7ce6ffd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5566014f86c0 0x556601509f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.46:47276/0 (socket says 172.21.15.46:47276) 2024-03-31T23:17:55.163 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:55.157+0000 7f7ce6ffd640 1 -- 172.21.15.46:0/3581261079 learned_addr learned my addr 172.21.15.46:0/3581261079 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:17:55.163 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:55.157+0000 7f7ce67fc640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5566014f8fe0 0x5566015052f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.46:56628/0 (socket says 172.21.15.46:56628) 2024-03-31T23:17:55.163 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:55.157+0000 7f7ce77fe640 1 --2- 172.21.15.46:0/3581261079 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556601501db0 0x556601505830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:45346/0 (socket says 172.21.15.46:45346) 2024-03-31T23:17:55.163 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:55.157+0000 7f7ce77fe640 1 -- 172.21.15.46:0/3581261079 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5566014f86c0 msgr2=0x556601509f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:17:55.163 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:55.157+0000 7f7ce77fe640 1 --2- 172.21.15.46:0/3581261079 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5566014f86c0 0x556601509f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:17:55.163 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:55.157+0000 7f7ce77fe640 1 -- 172.21.15.46:0/3581261079 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5566014f8fe0 msgr2=0x5566015052f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:17:55.164 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:55.157+0000 7f7ce77fe640 1 --2- 172.21.15.46:0/3581261079 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5566014f8fe0 0x5566015052f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:17:55.164 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:55.157+0000 7f7ce77fe640 1 -- 172.21.15.46:0/3581261079 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556601506030 con 0x556601501db0 2024-03-31T23:17:55.164 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:55.157+0000 7f7ce6ffd640 1 --2- 172.21.15.46:0/3581261079 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5566014f86c0 0x556601509f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:17:55.164 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:55.157+0000 7f7ce67fc640 1 --2- 172.21.15.46:0/3581261079 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5566014f8fe0 0x5566015052f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:17:55.164 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:55.158+0000 7f7ce77fe640 1 --2- 172.21.15.46:0/3581261079 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556601501db0 0x556601505830 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7f7cd80cba50 tx=0x7f7cd809bc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b0932f305bc7738 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:17:55.164 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:55.158+0000 7f7cc6ffd640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-31T23:17:55.164 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:55.158+0000 7f7cc6ffd640 1 -- 172.21.15.46:0/3581261079 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cd8020040 con 0x556601501db0 2024-03-31T23:17:55.164 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:55.158+0000 7f7cc6ffd640 1 -- 172.21.15.46:0/3581261079 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9466+0+0 (secure 0 0 0) 0x7f7cd80c9c20 con 0x556601501db0 2024-03-31T23:17:55.164 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:55.158+0000 7f7cf0630740 1 -- 172.21.15.46:0/3581261079 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7cd000b480 con 0x556601501db0 2024-03-31T23:17:55.164 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:55.158+0000 7f7cc6ffd640 1 -- 172.21.15.46:0/3581261079 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f7cd809ab30 con 0x556601501db0 2024-03-31T23:17:55.164 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:55.158+0000 7f7cf0630740 10 client.16659 fetch_fsmap learned FSMap version 30 2024-03-31T23:17:55.164 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:55.158+0000 7f7cf0630740 1 -- 172.21.15.46:0/3581261079 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x556601577db0 con 0x556601501db0 2024-03-31T23:17:55.164 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:55.158+0000 7f7cc6ffd640 1 -- 172.21.15.46:0/3581261079 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7cd000b480 con 0x556601501db0 2024-03-31T23:17:55.164 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:55.158+0000 7f7cc6ffd640 1 -- 172.21.15.46:0/3581261079 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 30) ==== 8346+0+0 (secure 0 0 0) 0x7f7cd80b9080 con 0x556601501db0 2024-03-31T23:17:55.165 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:55.158+0000 7f7cf0630740 10 client.16659 fetch_fsmap finished waiting for FSMap version 30 2024-03-31T23:17:55.165 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:55.158+0000 7f7cf0630740 10 client.16659 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-31T23:17:55.165 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:55.158+0000 7f7cf0630740 20 client.16659 populate_metadata read hostname 'smithi046' 2024-03-31T23:17:55.165 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:55.158+0000 7f7cf0630740 1 --2- 172.21.15.46:0/3581261079 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x5566015784d0 0x55660157a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:17:55.165 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:55.158+0000 7f7cf0630740 4 client.16659 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-31T23:17:55.165 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:55.158+0000 7f7cf0630740 1 -- 172.21.15.46:0/3581261079 --> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] -- command(tid 0: {"prefix": "flush journal"}) -- 0x556601468e30 con 0x5566015784d0 2024-03-31T23:17:55.165 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:55.159+0000 7f7ce6ffd640 1 --2- 172.21.15.46:0/3581261079 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x5566015784d0 0x55660157a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:17:55.165 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:55.159+0000 7f7ce6ffd640 1 --2- 172.21.15.46:0/3581261079 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x5566015784d0 0x55660157a8f0 crc :-1 s=READY pgs=130 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9f3440bac23cd5e7 server_cookie=8895f45737637dcb in_seq=0 out_seq=0 2024-03-31T23:17:55.165 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:55.159+0000 7f7cc6ffd640 10 client.16659 ms_handle_connect on v2:172.21.15.142:6838/2519143918 2024-03-31T23:17:55.182 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:55.180+0000 7f7cc6ffd640 1 -- 172.21.15.46:0/3581261079 <== mds.0 v2:172.21.15.142:6838/2519143918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x556601468e30 con 0x5566015784d0 2024-03-31T23:17:55.182 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:55.180+0000 7f7cc6ffd640 10 client.16659 handle_command_reply: tid=0 2024-03-31T23:17:55.182 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:55.180+0000 7f7cf0630740 1 client.16659 shutdown 2024-03-31T23:17:55.182 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:55.180+0000 7f7cf0630740 1 -- 172.21.15.46:0/3581261079 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556601501db0 msgr2=0x556601505830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:17:55.182 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:55.180+0000 7f7cf0630740 1 --2- 172.21.15.46:0/3581261079 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556601501db0 0x556601505830 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7f7cd80cba50 tx=0x7f7cd809bc70 comp rx=0 tx=0).stop 2024-03-31T23:17:55.182 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:55.180+0000 7f7cf0630740 1 -- 172.21.15.46:0/3581261079 shutdown_connections 2024-03-31T23:17:55.182 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:55.180+0000 7f7cf0630740 1 --2- 172.21.15.46:0/3581261079 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x5566015784d0 0x55660157a8f0 crc :-1 s=READY pgs=130 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:17:55.182 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:55.180+0000 7f7cf0630740 1 --2- 172.21.15.46:0/3581261079 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556601501db0 0x556601505830 unknown :-1 s=CLOSED pgs=730 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:17:55.182 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:55.180+0000 7f7cf0630740 1 --2- 172.21.15.46:0/3581261079 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5566014f8fe0 0x5566015052f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:17:55.183 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:55.180+0000 7f7cc6ffd640 0 client.16659 ms_handle_reset on v2:172.21.15.142:6838/2519143918 2024-03-31T23:17:55.183 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:55.180+0000 7f7cf0630740 1 --2- 172.21.15.46:0/3581261079 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5566014f86c0 0x556601509f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:17:55.183 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:55.180+0000 7f7cf0630740 1 -- 172.21.15.46:0/3581261079 >> 172.21.15.46:0/3581261079 conn(0x5566014f2fe0 msgr2=0x5566014f16f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:17:55.183 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:55.180+0000 7f7cf0630740 1 -- 172.21.15.46:0/3581261079 shutdown_connections 2024-03-31T23:17:55.183 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:55.180+0000 7f7cf0630740 1 -- 172.21.15.46:0/3581261079 wait complete. 2024-03-31T23:17:55.183 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:55.181+0000 7f7cf0630740 20 client.16659 trim_cache size 0 max 16384 2024-03-31T23:17:55.183 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout:{ 2024-03-31T23:17:55.183 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout: "message": "", 2024-03-31T23:17:55.183 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout: "return_code": 0 2024-03-31T23:17:55.183 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout:} 2024-03-31T23:17:55.184 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:55.181+0000 7f7ced979640 1 -- 172.21.15.46:0/2276881099 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f7cd40a26b0 msgr2=0x7f7cd40a4b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:17:55.184 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:55.181+0000 7f7ced979640 1 --2- 172.21.15.46:0/2276881099 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f7cd40a26b0 0x7f7cd40a4b70 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f7cd00058a0 tx=0x7f7cd0017000 comp rx=0 tx=0).stop 2024-03-31T23:17:55.184 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:55.181+0000 7f7ced979640 1 -- 172.21.15.46:0/2276881099 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7ce81055f0 msgr2=0x7f7ce81a3d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:17:55.184 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:55.181+0000 7f7ced979640 1 --2- 172.21.15.46:0/2276881099 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7ce81055f0 0x7f7ce81a3d60 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7f7cd800d680 tx=0x7f7cd800db40 comp rx=0 tx=0).stop 2024-03-31T23:17:55.184 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:55.181+0000 7f7ced979640 1 -- 172.21.15.46:0/2276881099 shutdown_connections 2024-03-31T23:17:55.185 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:55.181+0000 7f7ced979640 1 --2- 172.21.15.46:0/2276881099 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f7cd40a26b0 0x7f7cd40a4b70 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:17:55.185 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:55.181+0000 7f7ced979640 1 --2- 172.21.15.46:0/2276881099 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7ce81055f0 0x7f7ce81a3d60 unknown :-1 s=CLOSED pgs=726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:17:55.185 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:55.181+0000 7f7ced979640 1 --2- 172.21.15.46:0/2276881099 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7ce8101260 0x7f7ce8199220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:17:55.185 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:55.181+0000 7f7ced979640 1 --2- 172.21.15.46:0/2276881099 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7ce81008a0 0x7f7ce8198ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:17:55.185 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:55.181+0000 7f7ced979640 1 -- 172.21.15.46:0/2276881099 >> 172.21.15.46:0/2276881099 conn(0x7f7ce80774a0 msgr2=0x7f7ce80fe8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:17:55.185 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:55.181+0000 7f7ced979640 1 -- 172.21.15.46:0/2276881099 shutdown_connections 2024-03-31T23:17:55.185 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:17:55.181+0000 7f7ced979640 1 -- 172.21.15.46:0/2276881099 wait complete. 2024-03-31T23:17:55.199 INFO:tasks.workunit.client.0.smithi046.stdout: 1 137400 2.47 MB/sec execute 181 sec latency 24.033 ms 2024-03-31T23:17:56.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:17:55 smithi046 ceph-mon[32498]: pgmap v189: 129 pgs: 129 active+clean; 91 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 619 KiB/s rd, 1.9 MiB/s wr, 175 op/s 2024-03-31T23:17:56.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:17:55 smithi142 ceph-mon[35271]: pgmap v189: 129 pgs: 129 active+clean; 91 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 619 KiB/s rd, 1.9 MiB/s wr, 175 op/s 2024-03-31T23:17:56.199 INFO:tasks.workunit.client.0.smithi046.stdout: 1 138116 2.49 MB/sec execute 182 sec latency 16.779 ms 2024-03-31T23:17:56.296 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:17:55 smithi162 ceph-mon[36237]: pgmap v189: 129 pgs: 129 active+clean; 91 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 619 KiB/s rd, 1.9 MiB/s wr, 175 op/s 2024-03-31T23:17:57.199 INFO:tasks.workunit.client.0.smithi046.stdout: 1 138567 2.50 MB/sec execute 183 sec latency 16.604 ms 2024-03-31T23:17:58.114 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:17:57 smithi046 ceph-mon[32498]: pgmap v190: 129 pgs: 129 active+clean; 91 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 414 KiB/s rd, 1.2 MiB/s wr, 116 op/s 2024-03-31T23:17:58.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:17:57 smithi142 ceph-mon[35271]: pgmap v190: 129 pgs: 129 active+clean; 91 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 414 KiB/s rd, 1.2 MiB/s wr, 116 op/s 2024-03-31T23:17:58.199 INFO:tasks.workunit.client.0.smithi046.stdout: 1 138819 2.49 MB/sec execute 184 sec latency 16.240 ms 2024-03-31T23:17:58.296 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:17:57 smithi162 ceph-mon[36237]: pgmap v190: 129 pgs: 129 active+clean; 91 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 414 KiB/s rd, 1.2 MiB/s wr, 116 op/s 2024-03-31T23:17:59.200 INFO:tasks.workunit.client.0.smithi046.stdout: 1 139146 2.48 MB/sec execute 185 sec latency 71.498 ms 2024-03-31T23:18:00.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:17:59 smithi046 ceph-mon[32498]: pgmap v191: 129 pgs: 129 active+clean; 92 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 613 KiB/s rd, 1.9 MiB/s wr, 175 op/s 2024-03-31T23:18:00.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:17:59 smithi142 ceph-mon[35271]: pgmap v191: 129 pgs: 129 active+clean; 92 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 613 KiB/s rd, 1.9 MiB/s wr, 175 op/s 2024-03-31T23:18:00.199 INFO:tasks.workunit.client.0.smithi046.stdout: 1 139606 2.47 MB/sec execute 186 sec latency 17.485 ms 2024-03-31T23:18:00.296 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:17:59 smithi162 ceph-mon[36237]: pgmap v191: 129 pgs: 129 active+clean; 92 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 613 KiB/s rd, 1.9 MiB/s wr, 175 op/s 2024-03-31T23:18:01.200 INFO:tasks.workunit.client.0.smithi046.stdout: 1 140132 2.48 MB/sec execute 187 sec latency 24.241 ms 2024-03-31T23:18:02.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:18:01 smithi046 ceph-mon[32498]: pgmap v192: 129 pgs: 129 active+clean; 92 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 418 KiB/s rd, 1.3 MiB/s wr, 120 op/s 2024-03-31T23:18:02.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:18:01 smithi142 ceph-mon[35271]: pgmap v192: 129 pgs: 129 active+clean; 92 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 418 KiB/s rd, 1.3 MiB/s wr, 120 op/s 2024-03-31T23:18:02.199 INFO:tasks.workunit.client.0.smithi046.stdout: 1 140728 2.47 MB/sec execute 188 sec latency 19.214 ms 2024-03-31T23:18:02.295 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:18:01 smithi162 ceph-mon[36237]: pgmap v192: 129 pgs: 129 active+clean; 92 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 418 KiB/s rd, 1.3 MiB/s wr, 120 op/s 2024-03-31T23:18:03.200 INFO:tasks.workunit.client.0.smithi046.stdout: 1 141283 2.49 MB/sec execute 189 sec latency 22.765 ms 2024-03-31T23:18:04.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:18:03 smithi142 ceph-mon[35271]: pgmap v193: 129 pgs: 129 active+clean; 92 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 418 KiB/s rd, 1.3 MiB/s wr, 120 op/s 2024-03-31T23:18:04.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:18:03 smithi142 ceph-mon[35271]: osdmap e97: 12 total, 12 up, 12 in 2024-03-31T23:18:04.200 INFO:tasks.workunit.client.0.smithi046.stdout: 1 141729 2.49 MB/sec execute 190 sec latency 18.950 ms 2024-03-31T23:18:04.296 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:18:03 smithi162 ceph-mon[36237]: pgmap v193: 129 pgs: 129 active+clean; 92 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 418 KiB/s rd, 1.3 MiB/s wr, 120 op/s 2024-03-31T23:18:04.296 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:18:03 smithi162 ceph-mon[36237]: osdmap e97: 12 total, 12 up, 12 in 2024-03-31T23:18:04.362 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:18:03 smithi046 ceph-mon[32498]: pgmap v193: 129 pgs: 129 active+clean; 92 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 418 KiB/s rd, 1.3 MiB/s wr, 120 op/s 2024-03-31T23:18:04.362 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:18:03 smithi046 ceph-mon[32498]: osdmap e97: 12 total, 12 up, 12 in 2024-03-31T23:18:05.200 INFO:tasks.workunit.client.0.smithi046.stdout: 1 142206 2.50 MB/sec execute 191 sec latency 14.848 ms 2024-03-31T23:18:06.200 INFO:tasks.workunit.client.0.smithi046.stdout: 1 142463 2.49 MB/sec execute 192 sec latency 15.844 ms 2024-03-31T23:18:06.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:18:06 smithi162 ceph-mon[36237]: pgmap v195: 129 pgs: 129 active+clean; 95 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 504 KiB/s rd, 1.7 MiB/s wr, 159 op/s 2024-03-31T23:18:06.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:18:06 smithi162 ceph-mon[36237]: osdmap e98: 12 total, 12 up, 12 in 2024-03-31T23:18:06.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:18:06 smithi046 ceph-mon[32498]: pgmap v195: 129 pgs: 129 active+clean; 95 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 504 KiB/s rd, 1.7 MiB/s wr, 159 op/s 2024-03-31T23:18:06.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:18:06 smithi046 ceph-mon[32498]: osdmap e98: 12 total, 12 up, 12 in 2024-03-31T23:18:06.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:18:06 smithi142 ceph-mon[35271]: pgmap v195: 129 pgs: 129 active+clean; 95 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 504 KiB/s rd, 1.7 MiB/s wr, 159 op/s 2024-03-31T23:18:06.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:18:06 smithi142 ceph-mon[35271]: osdmap e98: 12 total, 12 up, 12 in 2024-03-31T23:18:07.200 INFO:tasks.workunit.client.0.smithi046.stdout: 1 142885 2.48 MB/sec execute 193 sec latency 17.574 ms 2024-03-31T23:18:08.200 INFO:tasks.workunit.client.0.smithi046.stdout: 1 143408 2.49 MB/sec execute 194 sec latency 23.250 ms 2024-03-31T23:18:08.278 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:08.275+0000 7f84a2f25640 1 -- 172.21.15.46:0/3370348713 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f849c101260 msgr2=0x7f849c10d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:18:08.279 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:08.275+0000 7f84a2f25640 1 --2- 172.21.15.46:0/3370348713 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f849c101260 0x7f849c10d480 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f8494009ca0 tx=0x7f849402c0f0 comp rx=0 tx=0).stop 2024-03-31T23:18:08.279 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:08.275+0000 7f84a2f25640 1 -- 172.21.15.46:0/3370348713 shutdown_connections 2024-03-31T23:18:08.279 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:08.275+0000 7f84a2f25640 1 --2- 172.21.15.46:0/3370348713 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f849c101260 0x7f849c10d480 unknown :-1 s=CLOSED pgs=731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:18:08.279 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:08.275+0000 7f84a2f25640 1 --2- 172.21.15.46:0/3370348713 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f849c1008a0 0x7f849c100d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:18:08.279 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:08.275+0000 7f84a2f25640 1 --2- 172.21.15.46:0/3370348713 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f849c1055f0 0x7f849c1059d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:18:08.279 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:08.275+0000 7f84a2f25640 1 -- 172.21.15.46:0/3370348713 >> 172.21.15.46:0/3370348713 conn(0x7f849c0774b0 msgr2=0x7f849c0778c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:18:08.279 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:08.275+0000 7f84a2f25640 1 -- 172.21.15.46:0/3370348713 shutdown_connections 2024-03-31T23:18:08.279 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:08.275+0000 7f84a2f25640 1 -- 172.21.15.46:0/3370348713 wait complete. 2024-03-31T23:18:08.279 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:08.276+0000 7f84a2f25640 1 Processor -- start 2024-03-31T23:18:08.279 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:08.276+0000 7f84a2f25640 1 -- start start 2024-03-31T23:18:08.279 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:08.276+0000 7f84a2f25640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f849c1008a0 0x7f849c198b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:18:08.280 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:08.276+0000 7f84a2f25640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f849c101260 0x7f849c1990c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:18:08.280 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:08.276+0000 7f84a2f25640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f849c1055f0 0x7f849c1a3c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:18:08.280 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:08.276+0000 7f84a2f25640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f849c1108e0 con 0x7f849c101260 2024-03-31T23:18:08.280 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:08.276+0000 7f84a2f25640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f849c110760 con 0x7f849c1055f0 2024-03-31T23:18:08.280 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:08.276+0000 7f84a2f25640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f849c110a60 con 0x7f849c1008a0 2024-03-31T23:18:08.280 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:08.276+0000 7f849bfff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f849c101260 0x7f849c1990c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:18:08.280 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:08.276+0000 7f849bfff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f849c101260 0x7f849c1990c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:52318/0 (socket says 172.21.15.46:52318) 2024-03-31T23:18:08.280 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:08.276+0000 7f849bfff640 1 -- 172.21.15.46:0/4094700107 learned_addr learned my addr 172.21.15.46:0/4094700107 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:18:08.280 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:08.276+0000 7f84a14c2640 1 --2- 172.21.15.46:0/4094700107 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f849c1055f0 0x7f849c1a3c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:18:08.280 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:08.276+0000 7f84a0cc1640 1 --2- 172.21.15.46:0/4094700107 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f849c1008a0 0x7f849c198b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:18:08.280 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:08.277+0000 7f849bfff640 1 -- 172.21.15.46:0/4094700107 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f849c1008a0 msgr2=0x7f849c198b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:18:08.281 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:08.277+0000 7f849bfff640 1 --2- 172.21.15.46:0/4094700107 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f849c1008a0 0x7f849c198b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:18:08.281 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:08.277+0000 7f849bfff640 1 -- 172.21.15.46:0/4094700107 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f849c1055f0 msgr2=0x7f849c1a3c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:18:08.281 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:08.277+0000 7f849bfff640 1 --2- 172.21.15.46:0/4094700107 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f849c1055f0 0x7f849c1a3c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:18:08.281 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:08.277+0000 7f849bfff640 1 -- 172.21.15.46:0/4094700107 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f849c1a4140 con 0x7f849c101260 2024-03-31T23:18:08.281 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:08.277+0000 7f849bfff640 1 --2- 172.21.15.46:0/4094700107 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f849c101260 0x7f849c1990c0 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f848800e9f0 tx=0x7f848800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:18:08.281 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:08.277+0000 7f8499ffb640 1 -- 172.21.15.46:0/4094700107 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f848800cd80 con 0x7f849c101260 2024-03-31T23:18:08.281 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:08.277+0000 7f8499ffb640 1 -- 172.21.15.46:0/4094700107 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f84880108e0 con 0x7f849c101260 2024-03-31T23:18:08.281 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:08.277+0000 7f84a2f25640 1 -- 172.21.15.46:0/4094700107 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f849c1a4310 con 0x7f849c101260 2024-03-31T23:18:08.281 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:08.277+0000 7f8499ffb640 1 -- 172.21.15.46:0/4094700107 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8488018920 con 0x7f849c101260 2024-03-31T23:18:08.281 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:08.277+0000 7f84a2f25640 1 -- 172.21.15.46:0/4094700107 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f849c1a48b0 con 0x7f849c101260 2024-03-31T23:18:08.288 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:08.279+0000 7f8499ffb640 1 -- 172.21.15.46:0/4094700107 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 25) ==== 137451+0+0 (secure 0 0 0) 0x7f8488010a90 con 0x7f849c101260 2024-03-31T23:18:08.288 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:08.280+0000 7f84a5be8740 1 Processor -- start 2024-03-31T23:18:08.288 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:08.280+0000 7f84a5be8740 1 -- start start 2024-03-31T23:18:08.288 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:08.280+0000 7f8499ffb640 1 --2- 172.21.15.46:0/4094700107 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f848c0a2730 0x7f848c0a4bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:18:08.288 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:08.280+0000 7f84a5be8740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56385259d6c0 0x56385259daa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:18:08.288 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:08.280+0000 7f84a5be8740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56385259dfe0 0x5638525a6870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:18:08.288 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:08.280+0000 7f84a5be8740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5638525a6db0 0x5638525a91a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:18:08.289 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:08.280+0000 7f84a5be8740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x5638525008f0 con 0x56385259dfe0 2024-03-31T23:18:08.289 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:08.280+0000 7f84a5be8740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x5638524a98d0 con 0x56385259d6c0 2024-03-31T23:18:08.289 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:08.280+0000 7f84a5be8740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5638524fc990 con 0x5638525a6db0 2024-03-31T23:18:08.289 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:08.280+0000 7f8499ffb640 1 -- 172.21.15.46:0/4094700107 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9554+0+0 (secure 0 0 0) 0x7f84880cf1a0 con 0x7f849c101260 2024-03-31T23:18:08.289 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:08.280+0000 7f84a14c2640 1 --2- 172.21.15.46:0/4094700107 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f848c0a2730 0x7f848c0a4bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:18:08.289 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:08.280+0000 7f84a0cc1640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56385259d6c0 0x56385259daa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:18:08.289 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:08.280+0000 7f84a0cc1640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56385259d6c0 0x56385259daa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.46:42048/0 (socket says 172.21.15.46:42048) 2024-03-31T23:18:08.289 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:08.280+0000 7f84a0cc1640 1 -- 172.21.15.46:0/1931414935 learned_addr learned my addr 172.21.15.46:0/1931414935 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:18:08.289 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:08.280+0000 7f849bfff640 1 --2- 172.21.15.46:0/1931414935 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56385259dfe0 0x5638525a6870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:18:08.289 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:08.281+0000 7f84a14c2640 1 --2- 172.21.15.46:0/1931414935 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5638525a6db0 0x5638525a91a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:18:08.289 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:08.281+0000 7f84a0cc1640 1 -- 172.21.15.46:0/1931414935 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5638525a6db0 msgr2=0x5638525a91a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:18:08.289 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:08.281+0000 7f84a0cc1640 1 --2- 172.21.15.46:0/1931414935 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5638525a6db0 0x5638525a91a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:18:08.289 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:08.281+0000 7f84a0cc1640 1 -- 172.21.15.46:0/1931414935 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56385259dfe0 msgr2=0x5638525a6870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:18:08.290 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:08.281+0000 7f84a0cc1640 1 --2- 172.21.15.46:0/1931414935 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56385259dfe0 0x5638525a6870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:18:08.290 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:08.281+0000 7f84a0cc1640 1 -- 172.21.15.46:0/1931414935 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563852215210 con 0x56385259d6c0 2024-03-31T23:18:08.290 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:08.281+0000 7f84a14c2640 1 --2- 172.21.15.46:0/4094700107 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f848c0a2730 0x7f848c0a4bf0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f8494004af0 tx=0x7f8494032e60 comp rx=0 tx=0).ready entity=mgr.15654 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:18:08.290 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:08.281+0000 7f84a0cc1640 1 --2- 172.21.15.46:0/1931414935 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56385259d6c0 0x56385259daa0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f8490009ca0 tx=0x7f8490011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6580106bdf6d71ee server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:18:08.290 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:08.281+0000 7f8486ffd640 1 -- 172.21.15.46:0/1931414935 <== mon.2 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8490019620 con 0x56385259d6c0 2024-03-31T23:18:08.290 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:08.281+0000 7f8486ffd640 1 -- 172.21.15.46:0/1931414935 <== mon.2 v2:172.21.15.142:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8490019c70 con 0x56385259d6c0 2024-03-31T23:18:08.290 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:08.281+0000 7f8486ffd640 1 -- 172.21.15.46:0/1931414935 <== mon.2 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8490018b00 con 0x56385259d6c0 2024-03-31T23:18:08.290 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:08.282+0000 7f84a5be8740 1 -- 172.21.15.46:0/1931414935 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56385259d6c0 msgr2=0x56385259daa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:18:08.290 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:08.282+0000 7f84a5be8740 1 --2- 172.21.15.46:0/1931414935 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56385259d6c0 0x56385259daa0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f8490009ca0 tx=0x7f8490011810 comp rx=0 tx=0).stop 2024-03-31T23:18:08.290 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:08.282+0000 7f84a5be8740 1 -- 172.21.15.46:0/1931414935 shutdown_connections 2024-03-31T23:18:08.290 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:08.282+0000 7f84a5be8740 1 --2- 172.21.15.46:0/1931414935 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5638525a6db0 0x5638525a91a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:18:08.290 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:08.282+0000 7f84a5be8740 1 --2- 172.21.15.46:0/1931414935 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56385259dfe0 0x5638525a6870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:18:08.290 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:08.282+0000 7f84a5be8740 1 --2- 172.21.15.46:0/1931414935 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56385259d6c0 0x56385259daa0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:18:08.291 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:08.282+0000 7f84a5be8740 1 -- 172.21.15.46:0/1931414935 >> 172.21.15.46:0/1931414935 conn(0x563852597fe0 msgr2=0x5638525983c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:18:08.291 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:08.282+0000 7f84a5be8740 1 -- 172.21.15.46:0/1931414935 shutdown_connections 2024-03-31T23:18:08.291 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:08.282+0000 7f84a5be8740 1 -- 172.21.15.46:0/1931414935 wait complete. 2024-03-31T23:18:08.291 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:08.282+0000 7f84a5be8740 1 -- start start 2024-03-31T23:18:08.291 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:08.282+0000 7f84a5be8740 1 Processor -- start 2024-03-31T23:18:08.291 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:08.283+0000 7f84a5be8740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56385259d6c0 0x5638525a3d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:18:08.291 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:08.283+0000 7f84a5be8740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56385259dfe0 0x5638525a42c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:18:08.291 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:08.283+0000 7f84a5be8740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5638525a6db0 0x5638525a4800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:18:08.291 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:08.283+0000 7f84a5be8740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x5638525a99e0 con 0x5638525a6db0 2024-03-31T23:18:08.291 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:08.283+0000 7f84a5be8740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x56385249ef80 con 0x56385259d6c0 2024-03-31T23:18:08.291 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:08.283+0000 7f84a5be8740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5638525a9b60 con 0x56385259dfe0 2024-03-31T23:18:08.291 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:08.283+0000 7f84a5be8740 10 client.0 _ll_register_callbacks cb 0x5638524f7290 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T23:18:08.291 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:08.283+0000 7f84a14c2640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5638525a6db0 0x5638525a4800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:18:08.292 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:08.283+0000 7f84a0cc1640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56385259d6c0 0x5638525a3d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:18:08.292 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:08.283+0000 7f84a14c2640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5638525a6db0 0x5638525a4800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:52332/0 (socket says 172.21.15.46:52332) 2024-03-31T23:18:08.292 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:08.283+0000 7f84a14c2640 1 -- 172.21.15.46:0/4265242244 learned_addr learned my addr 172.21.15.46:0/4265242244 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:18:08.292 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:08.283+0000 7f849bfff640 1 --2- 172.21.15.46:0/4265242244 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56385259dfe0 0x5638525a42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:18:08.292 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:08.283+0000 7f84a14c2640 1 -- 172.21.15.46:0/4265242244 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56385259dfe0 msgr2=0x5638525a42c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:18:08.292 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:08.283+0000 7f84a14c2640 1 --2- 172.21.15.46:0/4265242244 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56385259dfe0 0x5638525a42c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:18:08.292 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:08.283+0000 7f84a14c2640 1 -- 172.21.15.46:0/4265242244 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56385259d6c0 msgr2=0x5638525a3d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:18:08.292 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:08.283+0000 7f84a14c2640 1 --2- 172.21.15.46:0/4265242244 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56385259d6c0 0x5638525a3d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:18:08.292 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:08.283+0000 7f84a14c2640 1 -- 172.21.15.46:0/4265242244 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5638522e1620 con 0x5638525a6db0 2024-03-31T23:18:08.292 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:08.283+0000 7f84a0cc1640 1 --2- 172.21.15.46:0/4265242244 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56385259d6c0 0x5638525a3d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:18:08.292 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:08.283+0000 7f849bfff640 1 --2- 172.21.15.46:0/4265242244 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56385259dfe0 0x5638525a42c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:18:08.292 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:08.283+0000 7f84a14c2640 1 --2- 172.21.15.46:0/4265242244 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5638525a6db0 0x5638525a4800 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7f84940098e0 tx=0x7f849402c0f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d07ef1a0ba5a603f server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:18:08.292 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:08.283+0000 7f847bfff640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-31T23:18:08.292 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:08.284+0000 7f847bfff640 1 -- 172.21.15.46:0/4265242244 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8494051060 con 0x5638525a6db0 2024-03-31T23:18:08.292 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:08.284+0000 7f847bfff640 1 -- 172.21.15.46:0/4265242244 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9554+0+0 (secure 0 0 0) 0x7f8494004040 con 0x5638525a6db0 2024-03-31T23:18:08.293 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:08.284+0000 7f84a5be8740 1 -- 172.21.15.46:0/4265242244 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8490019c70 con 0x5638525a6db0 2024-03-31T23:18:08.293 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:08.284+0000 7f847bfff640 1 -- 172.21.15.46:0/4265242244 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f8494058050 con 0x5638525a6db0 2024-03-31T23:18:08.293 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:08.284+0000 7f84a5be8740 10 client.16671 fetch_fsmap learned FSMap version 30 2024-03-31T23:18:08.293 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:08.284+0000 7f84a5be8740 1 -- 172.21.15.46:0/4265242244 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x5638525b5850 con 0x5638525a6db0 2024-03-31T23:18:08.293 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:08.284+0000 7f847bfff640 1 -- 172.21.15.46:0/4265242244 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8490019c70 con 0x5638525a6db0 2024-03-31T23:18:08.293 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:08.284+0000 7f847bfff640 1 -- 172.21.15.46:0/4265242244 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 30) ==== 8346+0+0 (secure 0 0 0) 0x7f8494034850 con 0x5638525a6db0 2024-03-31T23:18:08.293 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:08.284+0000 7f84a5be8740 10 client.16671 fetch_fsmap finished waiting for FSMap version 30 2024-03-31T23:18:08.293 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:08.284+0000 7f84a5be8740 10 client.16671 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-31T23:18:08.293 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:08.284+0000 7f84a5be8740 20 client.16671 populate_metadata read hostname 'smithi046' 2024-03-31T23:18:08.293 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:08.284+0000 7f84a5be8740 1 --2- 172.21.15.46:0/4265242244 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x5638525b6d10 0x5638525b9120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:18:08.293 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:08.284+0000 7f84a5be8740 4 client.16671 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-31T23:18:08.293 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:08.284+0000 7f84a5be8740 1 -- 172.21.15.46:0/4265242244 --> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563852514120 con 0x5638525b6d10 2024-03-31T23:18:08.293 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:08.285+0000 7f84a0cc1640 1 --2- 172.21.15.46:0/4265242244 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x5638525b6d10 0x5638525b9120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:18:08.293 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:08.285+0000 7f84a0cc1640 1 --2- 172.21.15.46:0/4265242244 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x5638525b6d10 0x5638525b9120 crc :-1 s=READY pgs=131 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=60f1d53546036c01 server_cookie=b2ee4cf3aa45df5 in_seq=0 out_seq=0 2024-03-31T23:18:08.294 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:08.285+0000 7f847bfff640 10 client.16671 ms_handle_connect on v2:172.21.15.142:6838/2519143918 2024-03-31T23:18:08.294 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:08.286+0000 7f847bfff640 1 -- 172.21.15.46:0/4265242244 <== mds.0 v2:172.21.15.142:6838/2519143918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x563852514120 con 0x5638525b6d10 2024-03-31T23:18:08.294 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:08.286+0000 7f847bfff640 10 client.16671 handle_command_reply: tid=0 2024-03-31T23:18:08.294 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:08.286+0000 7f84a5be8740 1 client.16671 shutdown 2024-03-31T23:18:08.294 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:08.286+0000 7f84a5be8740 1 -- 172.21.15.46:0/4265242244 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5638525a6db0 msgr2=0x5638525a4800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:18:08.294 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:08.286+0000 7f84a5be8740 1 --2- 172.21.15.46:0/4265242244 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5638525a6db0 0x5638525a4800 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7f84940098e0 tx=0x7f849402c0f0 comp rx=0 tx=0).stop 2024-03-31T23:18:08.294 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:08.286+0000 7f84a5be8740 1 -- 172.21.15.46:0/4265242244 shutdown_connections 2024-03-31T23:18:08.294 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:08.286+0000 7f84a5be8740 1 --2- 172.21.15.46:0/4265242244 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x5638525b6d10 0x5638525b9120 crc :-1 s=READY pgs=131 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:18:08.294 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:08.286+0000 7f84a5be8740 1 --2- 172.21.15.46:0/4265242244 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5638525a6db0 0x5638525a4800 unknown :-1 s=CLOSED pgs=733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:18:08.294 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:08.286+0000 7f84a5be8740 1 --2- 172.21.15.46:0/4265242244 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56385259dfe0 0x5638525a42c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:18:08.294 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:08.286+0000 7f847bfff640 0 client.16671 ms_handle_reset on v2:172.21.15.142:6838/2519143918 2024-03-31T23:18:08.295 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:08.286+0000 7f84a5be8740 1 --2- 172.21.15.46:0/4265242244 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56385259d6c0 0x5638525a3d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:18:08.295 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:08.286+0000 7f84a5be8740 1 -- 172.21.15.46:0/4265242244 >> 172.21.15.46:0/4265242244 conn(0x563852597fe0 msgr2=0x563852599d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:18:08.295 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:08.286+0000 7f84a5be8740 1 -- 172.21.15.46:0/4265242244 shutdown_connections 2024-03-31T23:18:08.295 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:08.286+0000 7f84a5be8740 1 -- 172.21.15.46:0/4265242244 wait complete. 2024-03-31T23:18:08.295 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:08.286+0000 7f84a5be8740 20 client.16671 trim_cache size 0 max 16384 2024-03-31T23:18:08.306 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:08.301+0000 7f84a5be8740 1 Processor -- start 2024-03-31T23:18:08.307 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:08.301+0000 7f84a5be8740 1 -- start start 2024-03-31T23:18:08.307 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:08.301+0000 7f84a5be8740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56385259d6c0 0x5638525a64b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:18:08.307 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:08.301+0000 7f84a5be8740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56385259dfe0 0x5638525b9100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:18:08.307 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:08.301+0000 7f84a5be8740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5638525a6db0 0x5638525b2780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:18:08.307 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:08.301+0000 7f84a5be8740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f849c1b1450 con 0x56385259dfe0 2024-03-31T23:18:08.307 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:08.301+0000 7f84a5be8740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f849c1aed20 con 0x56385259d6c0 2024-03-31T23:18:08.307 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:08.301+0000 7f84a5be8740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f849c1acb90 con 0x5638525a6db0 2024-03-31T23:18:08.307 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:08.301+0000 7f849bfff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56385259dfe0 0x5638525b9100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:18:08.307 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:08.301+0000 7f849bfff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56385259dfe0 0x5638525b9100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:52346/0 (socket says 172.21.15.46:52346) 2024-03-31T23:18:08.307 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:08.301+0000 7f849bfff640 1 -- 172.21.15.46:0/4236220639 learned_addr learned my addr 172.21.15.46:0/4236220639 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:18:08.307 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:08.301+0000 7f84a0cc1640 1 --2- 172.21.15.46:0/4236220639 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56385259d6c0 0x5638525a64b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:18:08.307 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:08.301+0000 7f84a14c2640 1 --2- 172.21.15.46:0/4236220639 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5638525a6db0 0x5638525b2780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:18:08.307 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:08.301+0000 7f849bfff640 1 -- 172.21.15.46:0/4236220639 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5638525a6db0 msgr2=0x5638525b2780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:18:08.308 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:08.301+0000 7f849bfff640 1 --2- 172.21.15.46:0/4236220639 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5638525a6db0 0x5638525b2780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:18:08.308 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:08.301+0000 7f849bfff640 1 -- 172.21.15.46:0/4236220639 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56385259d6c0 msgr2=0x5638525a64b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:18:08.308 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:08.301+0000 7f849bfff640 1 --2- 172.21.15.46:0/4236220639 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56385259d6c0 0x5638525a64b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:18:08.308 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:08.302+0000 7f849bfff640 1 -- 172.21.15.46:0/4236220639 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563852592170 con 0x56385259dfe0 2024-03-31T23:18:08.308 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:08.302+0000 7f849bfff640 1 --2- 172.21.15.46:0/4236220639 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56385259dfe0 0x5638525b9100 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7f848800c6a0 tx=0x7f848805e040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f62b8cbc719ef2a2 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:18:08.308 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:08.302+0000 7f84867fc640 1 -- 172.21.15.46:0/4236220639 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84880a3f20 con 0x56385259dfe0 2024-03-31T23:18:08.308 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:08.302+0000 7f84867fc640 1 -- 172.21.15.46:0/4236220639 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f84880cc760 con 0x56385259dfe0 2024-03-31T23:18:08.308 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:08.302+0000 7f84867fc640 1 -- 172.21.15.46:0/4236220639 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f848809f3c0 con 0x56385259dfe0 2024-03-31T23:18:08.308 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:08.302+0000 7f84a5be8740 1 -- 172.21.15.46:0/4236220639 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56385259dfe0 msgr2=0x5638525b9100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:18:08.308 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:08.302+0000 7f84a5be8740 1 --2- 172.21.15.46:0/4236220639 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56385259dfe0 0x5638525b9100 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7f848800c6a0 tx=0x7f848805e040 comp rx=0 tx=0).stop 2024-03-31T23:18:08.308 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:08.303+0000 7f84a5be8740 1 -- 172.21.15.46:0/4236220639 shutdown_connections 2024-03-31T23:18:08.308 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:08.303+0000 7f84a5be8740 1 --2- 172.21.15.46:0/4236220639 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5638525a6db0 0x5638525b2780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:18:08.308 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:08.303+0000 7f84a5be8740 1 --2- 172.21.15.46:0/4236220639 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56385259dfe0 0x5638525b9100 unknown :-1 s=CLOSED pgs=734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:18:08.308 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:08.303+0000 7f84a5be8740 1 --2- 172.21.15.46:0/4236220639 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56385259d6c0 0x5638525a64b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:18:08.309 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:08.303+0000 7f84a5be8740 1 -- 172.21.15.46:0/4236220639 >> 172.21.15.46:0/4236220639 conn(0x563852597fe0 msgr2=0x5638525966f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:18:08.309 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:08.303+0000 7f84a5be8740 1 -- 172.21.15.46:0/4236220639 shutdown_connections 2024-03-31T23:18:08.309 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:08.303+0000 7f84a5be8740 1 -- 172.21.15.46:0/4236220639 wait complete. 2024-03-31T23:18:08.309 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:08.303+0000 7f84a5be8740 1 -- start start 2024-03-31T23:18:08.309 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:08.303+0000 7f84a5be8740 1 Processor -- start 2024-03-31T23:18:08.309 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:08.303+0000 7f84a5be8740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56385259d6c0 0x5638525aa6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:18:08.309 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:08.303+0000 7f84a5be8740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56385259dfe0 0x5638525aac10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:18:08.309 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:08.303+0000 7f84a5be8740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5638525a6db0 0x5638525ab150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:18:08.309 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:08.303+0000 7f84a5be8740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f849c1ad1e0 con 0x56385259d6c0 2024-03-31T23:18:08.309 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:08.303+0000 7f84a5be8740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f849c1ae650 con 0x5638525a6db0 2024-03-31T23:18:08.309 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:08.303+0000 7f84a5be8740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f849c1ad8b0 con 0x56385259dfe0 2024-03-31T23:18:08.309 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:08.303+0000 7f84a5be8740 10 client.0 _ll_register_callbacks cb 0x7f8494037070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T23:18:08.309 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:08.303+0000 7f84a0cc1640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56385259d6c0 0x5638525aa6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:18:08.310 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:08.303+0000 7f84a0cc1640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56385259d6c0 0x5638525aa6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:52358/0 (socket says 172.21.15.46:52358) 2024-03-31T23:18:08.310 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:08.304+0000 7f84a0cc1640 1 -- 172.21.15.46:0/1045319260 learned_addr learned my addr 172.21.15.46:0/1045319260 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:18:08.310 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:08.304+0000 7f84a14c2640 1 --2- 172.21.15.46:0/1045319260 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5638525a6db0 0x5638525ab150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:18:08.310 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:08.304+0000 7f849bfff640 1 --2- 172.21.15.46:0/1045319260 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56385259dfe0 0x5638525aac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:18:08.310 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:08.304+0000 7f84a0cc1640 1 -- 172.21.15.46:0/1045319260 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56385259dfe0 msgr2=0x5638525aac10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:18:08.310 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:08.304+0000 7f84a0cc1640 1 --2- 172.21.15.46:0/1045319260 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56385259dfe0 0x5638525aac10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:18:08.310 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:08.304+0000 7f84a0cc1640 1 -- 172.21.15.46:0/1045319260 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5638525a6db0 msgr2=0x5638525ab150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:18:08.310 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:08.304+0000 7f84a0cc1640 1 --2- 172.21.15.46:0/1045319260 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5638525a6db0 0x5638525ab150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:18:08.310 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:08.304+0000 7f84a0cc1640 1 -- 172.21.15.46:0/1045319260 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56385261c090 con 0x56385259d6c0 2024-03-31T23:18:08.310 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:08.304+0000 7f84a0cc1640 1 --2- 172.21.15.46:0/1045319260 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56385259d6c0 0x5638525aa6d0 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7f849001a000 tx=0x7f8490018760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ff9282c1d957959 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:18:08.310 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:08.304+0000 7f847bfff640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-31T23:18:08.310 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:08.304+0000 7f847bfff640 1 -- 172.21.15.46:0/1045319260 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8490005550 con 0x56385259d6c0 2024-03-31T23:18:08.310 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:08.304+0000 7f847bfff640 1 -- 172.21.15.46:0/1045319260 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9554+0+0 (secure 0 0 0) 0x7f849002ba30 con 0x56385259d6c0 2024-03-31T23:18:08.311 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:08.304+0000 7f84a5be8740 1 -- 172.21.15.46:0/1045319260 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f84880cc760 con 0x56385259d6c0 2024-03-31T23:18:08.311 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:08.304+0000 7f847bfff640 1 -- 172.21.15.46:0/1045319260 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f849003c050 con 0x56385259d6c0 2024-03-31T23:18:08.311 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:08.304+0000 7f84a5be8740 10 client.16677 fetch_fsmap learned FSMap version 30 2024-03-31T23:18:08.311 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:08.304+0000 7f84a5be8740 1 -- 172.21.15.46:0/1045319260 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x56385261d380 con 0x56385259d6c0 2024-03-31T23:18:08.311 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:08.305+0000 7f847bfff640 1 -- 172.21.15.46:0/1045319260 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f84880cc760 con 0x56385259d6c0 2024-03-31T23:18:08.311 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:08.305+0000 7f847bfff640 1 -- 172.21.15.46:0/1045319260 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 30) ==== 8346+0+0 (secure 0 0 0) 0x7f84900187d0 con 0x56385259d6c0 2024-03-31T23:18:08.311 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:08.305+0000 7f84a5be8740 10 client.16677 fetch_fsmap finished waiting for FSMap version 30 2024-03-31T23:18:08.311 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:08.305+0000 7f84a5be8740 10 client.16677 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-31T23:18:08.311 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:08.305+0000 7f84a5be8740 20 client.16677 populate_metadata read hostname 'smithi046' 2024-03-31T23:18:08.311 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:08.305+0000 7f84a5be8740 1 --2- 172.21.15.46:0/1045319260 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x56385261d970 0x56385261fd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:18:08.311 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:08.305+0000 7f84a5be8740 4 client.16677 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-31T23:18:08.311 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:08.305+0000 7f84a5be8740 1 -- 172.21.15.46:0/1045319260 --> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] -- command(tid 0: {"prefix": "flush journal"}) -- 0x56385250de30 con 0x56385261d970 2024-03-31T23:18:08.312 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:08.305+0000 7f849bfff640 1 --2- 172.21.15.46:0/1045319260 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x56385261d970 0x56385261fd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:18:08.312 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:08.305+0000 7f849bfff640 1 --2- 172.21.15.46:0/1045319260 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x56385261d970 0x56385261fd90 crc :-1 s=READY pgs=132 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a255e1235f0ab0d2 server_cookie=a543159f197753ab in_seq=0 out_seq=0 2024-03-31T23:18:08.312 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:08.305+0000 7f847bfff640 10 client.16677 ms_handle_connect on v2:172.21.15.142:6838/2519143918 2024-03-31T23:18:08.332 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:08.330+0000 7f847bfff640 1 -- 172.21.15.46:0/1045319260 <== mds.0 v2:172.21.15.142:6838/2519143918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x56385250de30 con 0x56385261d970 2024-03-31T23:18:08.332 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:08.330+0000 7f847bfff640 10 client.16677 handle_command_reply: tid=0 2024-03-31T23:18:08.332 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:08.330+0000 7f84a5be8740 1 client.16677 shutdown 2024-03-31T23:18:08.332 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:08.330+0000 7f84a5be8740 1 -- 172.21.15.46:0/1045319260 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56385259d6c0 msgr2=0x5638525aa6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:18:08.332 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:08.330+0000 7f84a5be8740 1 --2- 172.21.15.46:0/1045319260 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56385259d6c0 0x5638525aa6d0 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7f849001a000 tx=0x7f8490018760 comp rx=0 tx=0).stop 2024-03-31T23:18:08.332 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:08.330+0000 7f84a5be8740 1 -- 172.21.15.46:0/1045319260 shutdown_connections 2024-03-31T23:18:08.332 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:08.330+0000 7f84a5be8740 1 --2- 172.21.15.46:0/1045319260 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x56385261d970 0x56385261fd90 crc :-1 s=READY pgs=132 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:18:08.332 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:08.330+0000 7f84a5be8740 1 --2- 172.21.15.46:0/1045319260 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5638525a6db0 0x5638525ab150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:18:08.332 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:08.330+0000 7f84a5be8740 1 --2- 172.21.15.46:0/1045319260 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56385259dfe0 0x5638525aac10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:18:08.332 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:08.330+0000 7f847bfff640 0 client.16677 ms_handle_reset on v2:172.21.15.142:6838/2519143918 2024-03-31T23:18:08.332 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:08.330+0000 7f84a5be8740 1 --2- 172.21.15.46:0/1045319260 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56385259d6c0 0x5638525aa6d0 unknown :-1 s=CLOSED pgs=735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:18:08.333 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:08.330+0000 7f84a5be8740 1 -- 172.21.15.46:0/1045319260 >> 172.21.15.46:0/1045319260 conn(0x563852597fe0 msgr2=0x563852596f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:18:08.333 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:08.330+0000 7f84a5be8740 1 -- 172.21.15.46:0/1045319260 shutdown_connections 2024-03-31T23:18:08.333 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:08.330+0000 7f84a5be8740 1 -- 172.21.15.46:0/1045319260 wait complete. 2024-03-31T23:18:08.333 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:08.331+0000 7f84a5be8740 20 client.16677 trim_cache size 0 max 16384 2024-03-31T23:18:08.333 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout:{ 2024-03-31T23:18:08.333 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout: "message": "", 2024-03-31T23:18:08.333 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout: "return_code": 0 2024-03-31T23:18:08.333 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout:} 2024-03-31T23:18:08.334 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:08.331+0000 7f84a2f25640 1 -- 172.21.15.46:0/4094700107 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f848c0a2730 msgr2=0x7f848c0a4bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:18:08.334 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:08.331+0000 7f84a2f25640 1 --2- 172.21.15.46:0/4094700107 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f848c0a2730 0x7f848c0a4bf0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f8494004af0 tx=0x7f8494032e60 comp rx=0 tx=0).stop 2024-03-31T23:18:08.334 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:08.331+0000 7f84a2f25640 1 -- 172.21.15.46:0/4094700107 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f849c101260 msgr2=0x7f849c1990c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:18:08.334 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:08.331+0000 7f84a2f25640 1 --2- 172.21.15.46:0/4094700107 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f849c101260 0x7f849c1990c0 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f848800e9f0 tx=0x7f848800eeb0 comp rx=0 tx=0).stop 2024-03-31T23:18:08.334 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:08.331+0000 7f84a2f25640 1 -- 172.21.15.46:0/4094700107 shutdown_connections 2024-03-31T23:18:08.334 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:08.331+0000 7f84a2f25640 1 --2- 172.21.15.46:0/4094700107 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f848c0a2730 0x7f848c0a4bf0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:18:08.334 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:08.331+0000 7f84a2f25640 1 --2- 172.21.15.46:0/4094700107 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f849c1055f0 0x7f849c1a3c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:18:08.335 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:08.331+0000 7f84a2f25640 1 --2- 172.21.15.46:0/4094700107 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f849c101260 0x7f849c1990c0 unknown :-1 s=CLOSED pgs=732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:18:08.335 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:08.331+0000 7f84a2f25640 1 --2- 172.21.15.46:0/4094700107 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f849c1008a0 0x7f849c198b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:18:08.335 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:08.331+0000 7f84a2f25640 1 -- 172.21.15.46:0/4094700107 >> 172.21.15.46:0/4094700107 conn(0x7f849c0774b0 msgr2=0x7f849c0fe320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:18:08.335 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:08.331+0000 7f84a2f25640 1 -- 172.21.15.46:0/4094700107 shutdown_connections 2024-03-31T23:18:08.335 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:08.331+0000 7f84a2f25640 1 -- 172.21.15.46:0/4094700107 wait complete. 2024-03-31T23:18:08.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:18:08 smithi162 ceph-mon[36237]: pgmap v197: 129 pgs: 129 active+clean; 95 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 332 KiB/s rd, 1.0 MiB/s wr, 110 op/s 2024-03-31T23:18:08.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:18:08 smithi046 ceph-mon[32498]: pgmap v197: 129 pgs: 129 active+clean; 95 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 332 KiB/s rd, 1.0 MiB/s wr, 110 op/s 2024-03-31T23:18:08.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:18:08 smithi142 ceph-mon[35271]: pgmap v197: 129 pgs: 129 active+clean; 95 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 332 KiB/s rd, 1.0 MiB/s wr, 110 op/s 2024-03-31T23:18:09.200 INFO:tasks.workunit.client.0.smithi046.stdout: 1 143890 2.48 MB/sec execute 195 sec latency 25.403 ms 2024-03-31T23:18:10.200 INFO:tasks.workunit.client.0.smithi046.stdout: 1 144363 2.47 MB/sec execute 196 sec latency 21.752 ms 2024-03-31T23:18:10.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:18:10 smithi162 ceph-mon[36237]: pgmap v198: 129 pgs: 129 active+clean; 94 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 743 KiB/s rd, 1.9 MiB/s wr, 204 op/s 2024-03-31T23:18:10.862 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:18:10 smithi046 ceph-mon[32498]: pgmap v198: 129 pgs: 129 active+clean; 94 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 743 KiB/s rd, 1.9 MiB/s wr, 204 op/s 2024-03-31T23:18:10.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:18:10 smithi142 ceph-mon[35271]: pgmap v198: 129 pgs: 129 active+clean; 94 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 743 KiB/s rd, 1.9 MiB/s wr, 204 op/s 2024-03-31T23:18:11.200 INFO:tasks.workunit.client.0.smithi046.stdout: 1 145117 2.49 MB/sec execute 197 sec latency 15.403 ms 2024-03-31T23:18:12.200 INFO:tasks.workunit.client.0.smithi046.stdout: 1 145476 2.50 MB/sec execute 198 sec latency 16.969 ms 2024-03-31T23:18:12.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:18:12 smithi162 ceph-mon[36237]: pgmap v199: 129 pgs: 129 active+clean; 94 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 743 KiB/s rd, 1.9 MiB/s wr, 204 op/s 2024-03-31T23:18:12.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:18:12 smithi046 ceph-mon[32498]: pgmap v199: 129 pgs: 129 active+clean; 94 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 743 KiB/s rd, 1.9 MiB/s wr, 204 op/s 2024-03-31T23:18:12.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:18:12 smithi142 ceph-mon[35271]: pgmap v199: 129 pgs: 129 active+clean; 94 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 743 KiB/s rd, 1.9 MiB/s wr, 204 op/s 2024-03-31T23:18:13.200 INFO:tasks.workunit.client.0.smithi046.stdout: 1 145828 2.49 MB/sec execute 199 sec latency 15.794 ms 2024-03-31T23:18:14.200 INFO:tasks.workunit.client.0.smithi046.stdout: 1 146081 2.48 MB/sec execute 200 sec latency 17.171 ms 2024-03-31T23:18:14.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:18:14 smithi162 ceph-mon[36237]: pgmap v200: 129 pgs: 129 active+clean; 94 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 599 KiB/s rd, 1.5 MiB/s wr, 164 op/s 2024-03-31T23:18:14.862 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:18:14 smithi046 ceph-mon[32498]: pgmap v200: 129 pgs: 129 active+clean; 94 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 599 KiB/s rd, 1.5 MiB/s wr, 164 op/s 2024-03-31T23:18:14.945 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:18:14 smithi142 ceph-mon[35271]: pgmap v200: 129 pgs: 129 active+clean; 94 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 599 KiB/s rd, 1.5 MiB/s wr, 164 op/s 2024-03-31T23:18:15.200 INFO:tasks.workunit.client.0.smithi046.stdout: 1 146581 2.48 MB/sec execute 201 sec latency 18.338 ms 2024-03-31T23:18:16.200 INFO:tasks.workunit.client.0.smithi046.stdout: 1 147068 2.48 MB/sec execute 202 sec latency 22.122 ms 2024-03-31T23:18:16.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:18:16 smithi162 ceph-mon[36237]: pgmap v201: 129 pgs: 129 active+clean; 97 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 571 KiB/s rd, 1.6 MiB/s wr, 140 op/s 2024-03-31T23:18:16.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:18:16 smithi046 ceph-mon[32498]: pgmap v201: 129 pgs: 129 active+clean; 97 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 571 KiB/s rd, 1.6 MiB/s wr, 140 op/s 2024-03-31T23:18:16.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:18:16 smithi142 ceph-mon[35271]: pgmap v201: 129 pgs: 129 active+clean; 97 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 571 KiB/s rd, 1.6 MiB/s wr, 140 op/s 2024-03-31T23:18:17.200 INFO:tasks.workunit.client.0.smithi046.stdout: 1 147637 2.47 MB/sec execute 203 sec latency 21.567 ms 2024-03-31T23:18:18.200 INFO:tasks.workunit.client.0.smithi046.stdout: 1 148069 2.47 MB/sec execute 204 sec latency 16.112 ms 2024-03-31T23:18:18.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:18:18 smithi162 ceph-mon[36237]: pgmap v202: 129 pgs: 129 active+clean; 97 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 505 KiB/s rd, 1.4 MiB/s wr, 124 op/s 2024-03-31T23:18:18.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:18:18 smithi046 ceph-mon[32498]: pgmap v202: 129 pgs: 129 active+clean; 97 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 505 KiB/s rd, 1.4 MiB/s wr, 124 op/s 2024-03-31T23:18:18.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:18:18 smithi142 ceph-mon[35271]: pgmap v202: 129 pgs: 129 active+clean; 97 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 505 KiB/s rd, 1.4 MiB/s wr, 124 op/s 2024-03-31T23:18:19.200 INFO:tasks.workunit.client.0.smithi046.stdout: 1 148763 2.49 MB/sec execute 205 sec latency 16.592 ms 2024-03-31T23:18:20.200 INFO:tasks.workunit.client.0.smithi046.stdout: 1 149203 2.50 MB/sec execute 206 sec latency 16.285 ms 2024-03-31T23:18:20.744 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:18:20 smithi142 ceph-mon[35271]: pgmap v203: 129 pgs: 129 active+clean; 99 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 689 KiB/s rd, 1.9 MiB/s wr, 174 op/s 2024-03-31T23:18:20.795 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:18:20 smithi162 ceph-mon[36237]: pgmap v203: 129 pgs: 129 active+clean; 99 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 689 KiB/s rd, 1.9 MiB/s wr, 174 op/s 2024-03-31T23:18:20.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:18:20 smithi046 ceph-mon[32498]: pgmap v203: 129 pgs: 129 active+clean; 99 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 689 KiB/s rd, 1.9 MiB/s wr, 174 op/s 2024-03-31T23:18:21.200 INFO:tasks.workunit.client.0.smithi046.stdout: 1 149444 2.49 MB/sec execute 207 sec latency 15.648 ms 2024-03-31T23:18:21.431 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:21.430+0000 7ff13b2c9640 1 -- 172.21.15.46:0/1575192800 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff134107700 msgr2=0x7ff134107ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:18:21.431 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:21.430+0000 7ff13b2c9640 1 --2- 172.21.15.46:0/1575192800 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff134107700 0x7ff134107ae0 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7ff130009930 tx=0x7ff13002bc30 comp rx=0 tx=0).stop 2024-03-31T23:18:21.431 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:21.430+0000 7ff13b2c9640 1 -- 172.21.15.46:0/1575192800 shutdown_connections 2024-03-31T23:18:21.431 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:21.430+0000 7ff13b2c9640 1 --2- 172.21.15.46:0/1575192800 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff1341020b0 0x7ff13410e5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:18:21.431 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:21.430+0000 7ff13b2c9640 1 --2- 172.21.15.46:0/1575192800 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff1341016f0 0x7ff134101b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:18:21.431 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:21.430+0000 7ff13b2c9640 1 --2- 172.21.15.46:0/1575192800 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff134107700 0x7ff134107ae0 unknown :-1 s=CLOSED pgs=736 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:18:21.431 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:21.430+0000 7ff13b2c9640 1 -- 172.21.15.46:0/1575192800 >> 172.21.15.46:0/1575192800 conn(0x7ff1340fd1e0 msgr2=0x7ff1340ff600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:18:21.431 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:21.430+0000 7ff13b2c9640 1 -- 172.21.15.46:0/1575192800 shutdown_connections 2024-03-31T23:18:21.431 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:21.431+0000 7ff13b2c9640 1 -- 172.21.15.46:0/1575192800 wait complete. 2024-03-31T23:18:21.434 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:21.431+0000 7ff13b2c9640 1 Processor -- start 2024-03-31T23:18:21.434 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:21.431+0000 7ff13b2c9640 1 -- start start 2024-03-31T23:18:21.434 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:21.431+0000 7ff13b2c9640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff1341016f0 0x7ff134197ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:18:21.434 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:21.431+0000 7ff13b2c9640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff1341020b0 0x7ff134198400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:18:21.434 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:21.431+0000 7ff13b2c9640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff134107700 0x7ff13419c780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:18:21.434 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:21.431+0000 7ff13b2c9640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7ff1341119c0 con 0x7ff1341016f0 2024-03-31T23:18:21.434 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:21.431+0000 7ff13b2c9640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7ff134111840 con 0x7ff134107700 2024-03-31T23:18:21.434 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:21.432+0000 7ff13b2c9640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7ff134111b40 con 0x7ff1341020b0 2024-03-31T23:18:21.434 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:21.432+0000 7ff139065640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff1341016f0 0x7ff134197ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:18:21.434 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:21.432+0000 7ff139065640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff1341016f0 0x7ff134197ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55598/0 (socket says 172.21.15.46:55598) 2024-03-31T23:18:21.434 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:21.432+0000 7ff139065640 1 -- 172.21.15.46:0/1828878674 learned_addr learned my addr 172.21.15.46:0/1828878674 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:18:21.434 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:21.432+0000 7ff139866640 1 --2- 172.21.15.46:0/1828878674 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff134107700 0x7ff13419c780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:18:21.435 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:21.432+0000 7ff138864640 1 --2- 172.21.15.46:0/1828878674 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff1341020b0 0x7ff134198400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:18:21.435 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:21.432+0000 7ff139065640 1 -- 172.21.15.46:0/1828878674 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff1341020b0 msgr2=0x7ff134198400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:18:21.435 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:21.432+0000 7ff139065640 1 --2- 172.21.15.46:0/1828878674 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff1341020b0 0x7ff134198400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:18:21.435 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:21.432+0000 7ff139065640 1 -- 172.21.15.46:0/1828878674 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff134107700 msgr2=0x7ff13419c780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:18:21.435 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:21.432+0000 7ff139065640 1 --2- 172.21.15.46:0/1828878674 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff134107700 0x7ff13419c780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:18:21.435 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:21.432+0000 7ff139065640 1 -- 172.21.15.46:0/1828878674 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff13419ceb0 con 0x7ff1341016f0 2024-03-31T23:18:21.435 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:21.432+0000 7ff139065640 1 --2- 172.21.15.46:0/1828878674 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff1341016f0 0x7ff134197ec0 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7ff130009230 tx=0x7ff13002c710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:18:21.435 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:21.432+0000 7ff12a7fc640 1 -- 172.21.15.46:0/1828878674 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff130043020 con 0x7ff1341016f0 2024-03-31T23:18:21.435 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:21.432+0000 7ff12a7fc640 1 -- 172.21.15.46:0/1828878674 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff13002caf0 con 0x7ff1341016f0 2024-03-31T23:18:21.435 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:21.432+0000 7ff13b2c9640 1 -- 172.21.15.46:0/1828878674 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff13419d140 con 0x7ff1341016f0 2024-03-31T23:18:21.435 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:21.432+0000 7ff12a7fc640 1 -- 172.21.15.46:0/1828878674 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff130003ae0 con 0x7ff1341016f0 2024-03-31T23:18:21.435 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:21.433+0000 7ff13b2c9640 1 -- 172.21.15.46:0/1828878674 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff1341a4a30 con 0x7ff1341016f0 2024-03-31T23:18:21.440 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:21.435+0000 7ff12a7fc640 1 -- 172.21.15.46:0/1828878674 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 25) ==== 137451+0+0 (secure 0 0 0) 0x7ff130046020 con 0x7ff1341016f0 2024-03-31T23:18:21.440 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:21.435+0000 7ff13df84740 1 Processor -- start 2024-03-31T23:18:21.440 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:21.435+0000 7ff13df84740 1 -- start start 2024-03-31T23:18:21.440 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:21.436+0000 7ff13df84740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5577359646c0 0x557735964aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:18:21.440 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:21.436+0000 7ff12a7fc640 1 --2- 172.21.15.46:0/1828878674 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7ff1200a26b0 0x7ff1200a4b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:18:21.441 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:21.436+0000 7ff13df84740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557735964fe0 0x55773596d870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:18:21.441 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:21.436+0000 7ff13df84740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55773596ddb0 0x5577359701a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:18:21.441 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:21.436+0000 7ff139065640 1 --2- 172.21.15.46:0/1828878674 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7ff1200a26b0 0x7ff1200a4b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:18:21.441 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:21.436+0000 7ff138864640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557735964fe0 0x55773596d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:18:21.441 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:21.436+0000 7ff13df84740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x5577358c78f0 con 0x557735964fe0 2024-03-31T23:18:21.441 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:21.436+0000 7ff13df84740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x5577358708d0 con 0x55773596ddb0 2024-03-31T23:18:21.441 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:21.436+0000 7ff13df84740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5577358c3990 con 0x5577359646c0 2024-03-31T23:18:21.441 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:21.436+0000 7ff138864640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557735964fe0 0x55773596d870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55602/0 (socket says 172.21.15.46:55602) 2024-03-31T23:18:21.441 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:21.436+0000 7ff139866640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5577359646c0 0x557735964aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:18:21.441 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:21.436+0000 7ff138864640 1 -- 172.21.15.46:0/1533326954 learned_addr learned my addr 172.21.15.46:0/1533326954 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:18:21.441 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:21.436+0000 7ff12a7fc640 1 -- 172.21.15.46:0/1828878674 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9554+0+0 (secure 0 0 0) 0x7ff1300f0e50 con 0x7ff1341016f0 2024-03-31T23:18:21.441 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:21.436+0000 7ff139866640 1 --2- 172.21.15.46:0/1533326954 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55773596ddb0 0x5577359701a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:18:21.441 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:21.436+0000 7ff138864640 1 -- 172.21.15.46:0/1533326954 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5577359646c0 msgr2=0x557735964aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:18:21.441 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:21.436+0000 7ff138864640 1 --2- 172.21.15.46:0/1533326954 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5577359646c0 0x557735964aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:18:21.442 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:21.436+0000 7ff138864640 1 -- 172.21.15.46:0/1533326954 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55773596ddb0 msgr2=0x5577359701a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:18:21.442 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:21.436+0000 7ff138864640 1 --2- 172.21.15.46:0/1533326954 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55773596ddb0 0x5577359701a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:18:21.442 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:21.436+0000 7ff138864640 1 -- 172.21.15.46:0/1533326954 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5577355dc210 con 0x557735964fe0 2024-03-31T23:18:21.442 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:21.436+0000 7ff139866640 1 --2- 172.21.15.46:0/1533326954 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5577359646c0 0x557735964aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:18:21.442 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:21.436+0000 7ff139065640 1 --2- 172.21.15.46:0/1828878674 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7ff1200a26b0 0x7ff1200a4b70 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7ff1300ee350 tx=0x7ff1300c1040 comp rx=0 tx=0).ready entity=mgr.15654 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:18:21.442 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:21.436+0000 7ff138864640 1 --2- 172.21.15.46:0/1533326954 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557735964fe0 0x55773596d870 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7ff124009800 tx=0x7ff124009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c08b2378db85623 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:18:21.442 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:21.437+0000 7ff11f7fe640 1 -- 172.21.15.46:0/1533326954 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff12400e990 con 0x557735964fe0 2024-03-31T23:18:21.442 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:21.437+0000 7ff11f7fe640 1 -- 172.21.15.46:0/1533326954 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff124005020 con 0x557735964fe0 2024-03-31T23:18:21.442 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:21.437+0000 7ff11f7fe640 1 -- 172.21.15.46:0/1533326954 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff124004a70 con 0x557735964fe0 2024-03-31T23:18:21.442 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:21.437+0000 7ff13df84740 1 -- 172.21.15.46:0/1533326954 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557735964fe0 msgr2=0x55773596d870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:18:21.442 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:21.437+0000 7ff13df84740 1 --2- 172.21.15.46:0/1533326954 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557735964fe0 0x55773596d870 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7ff124009800 tx=0x7ff124009c00 comp rx=0 tx=0).stop 2024-03-31T23:18:21.442 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:21.438+0000 7ff13df84740 1 -- 172.21.15.46:0/1533326954 shutdown_connections 2024-03-31T23:18:21.442 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:21.438+0000 7ff13df84740 1 --2- 172.21.15.46:0/1533326954 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55773596ddb0 0x5577359701a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:18:21.442 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:21.438+0000 7ff13df84740 1 --2- 172.21.15.46:0/1533326954 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557735964fe0 0x55773596d870 unknown :-1 s=CLOSED pgs=738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:18:21.443 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:21.438+0000 7ff13df84740 1 --2- 172.21.15.46:0/1533326954 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5577359646c0 0x557735964aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:18:21.443 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:21.438+0000 7ff13df84740 1 -- 172.21.15.46:0/1533326954 >> 172.21.15.46:0/1533326954 conn(0x55773595efe0 msgr2=0x55773595f3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:18:21.443 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:21.438+0000 7ff13df84740 1 -- 172.21.15.46:0/1533326954 shutdown_connections 2024-03-31T23:18:21.443 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:21.438+0000 7ff13df84740 1 -- 172.21.15.46:0/1533326954 wait complete. 2024-03-31T23:18:21.443 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:21.438+0000 7ff13df84740 1 -- start start 2024-03-31T23:18:21.443 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:21.438+0000 7ff13df84740 1 Processor -- start 2024-03-31T23:18:21.443 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:21.438+0000 7ff13df84740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5577359646c0 0x557735967070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:18:21.443 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:21.438+0000 7ff13df84740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557735964fe0 0x5577359675b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:18:21.443 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:21.438+0000 7ff13df84740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55773596ddb0 0x55773596b7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:18:21.443 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:21.438+0000 7ff13df84740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x557735970920 con 0x5577359646c0 2024-03-31T23:18:21.443 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:21.438+0000 7ff13df84740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x557735865f80 con 0x557735964fe0 2024-03-31T23:18:21.443 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:21.438+0000 7ff13df84740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x557735970aa0 con 0x55773596ddb0 2024-03-31T23:18:21.443 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:21.438+0000 7ff13df84740 10 client.0 _ll_register_callbacks cb 0x5577358be290 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T23:18:21.444 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:21.438+0000 7ff139866640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5577359646c0 0x557735967070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:18:21.444 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:21.439+0000 7ff139866640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5577359646c0 0x557735967070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55610/0 (socket says 172.21.15.46:55610) 2024-03-31T23:18:21.444 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:21.439+0000 7ff139866640 1 -- 172.21.15.46:0/1805352846 learned_addr learned my addr 172.21.15.46:0/1805352846 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:18:21.444 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:21.439+0000 7ff138864640 1 --2- 172.21.15.46:0/1805352846 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557735964fe0 0x5577359675b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:18:21.444 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:21.439+0000 7ff139866640 1 --2- 172.21.15.46:0/1805352846 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55773596ddb0 0x55773596b7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:18:21.444 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:21.439+0000 7ff139866640 1 -- 172.21.15.46:0/1805352846 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55773596ddb0 msgr2=0x55773596b7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:18:21.444 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:21.439+0000 7ff139866640 1 --2- 172.21.15.46:0/1805352846 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55773596ddb0 0x55773596b7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:18:21.444 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:21.439+0000 7ff139866640 1 -- 172.21.15.46:0/1805352846 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557735964fe0 msgr2=0x5577359675b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:18:21.444 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:21.439+0000 7ff139866640 1 --2- 172.21.15.46:0/1805352846 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557735964fe0 0x5577359675b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:18:21.444 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:21.439+0000 7ff139866640 1 -- 172.21.15.46:0/1805352846 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5577356a8620 con 0x5577359646c0 2024-03-31T23:18:21.444 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:21.439+0000 7ff139866640 1 --2- 172.21.15.46:0/1805352846 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5577359646c0 0x557735967070 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7ff12c014ec0 tx=0x7ff12c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e781a1ac4164e4 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:18:21.444 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:21.439+0000 7ff11cff9640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-31T23:18:21.444 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:21.439+0000 7ff11cff9640 1 -- 172.21.15.46:0/1805352846 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff12c004b20 con 0x5577359646c0 2024-03-31T23:18:21.444 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:21.439+0000 7ff11cff9640 1 -- 172.21.15.46:0/1805352846 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9554+0+0 (secure 0 0 0) 0x7ff12c004cc0 con 0x5577359646c0 2024-03-31T23:18:21.445 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:21.439+0000 7ff13df84740 1 -- 172.21.15.46:0/1805352846 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff124005020 con 0x5577359646c0 2024-03-31T23:18:21.445 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:21.439+0000 7ff11cff9640 1 -- 172.21.15.46:0/1805352846 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7ff12c02a050 con 0x5577359646c0 2024-03-31T23:18:21.445 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:21.439+0000 7ff13df84740 10 client.16689 fetch_fsmap learned FSMap version 30 2024-03-31T23:18:21.445 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:21.439+0000 7ff13df84740 1 -- 172.21.15.46:0/1805352846 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55773597c980 con 0x5577359646c0 2024-03-31T23:18:21.445 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:21.440+0000 7ff11cff9640 1 -- 172.21.15.46:0/1805352846 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff124005020 con 0x5577359646c0 2024-03-31T23:18:21.445 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:21.440+0000 7ff11cff9640 1 -- 172.21.15.46:0/1805352846 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 30) ==== 8346+0+0 (secure 0 0 0) 0x7ff12c020030 con 0x5577359646c0 2024-03-31T23:18:21.445 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:21.440+0000 7ff13df84740 10 client.16689 fetch_fsmap finished waiting for FSMap version 30 2024-03-31T23:18:21.445 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:21.440+0000 7ff13df84740 10 client.16689 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-31T23:18:21.445 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:21.440+0000 7ff13df84740 20 client.16689 populate_metadata read hostname 'smithi046' 2024-03-31T23:18:21.445 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:21.440+0000 7ff13df84740 1 --2- 172.21.15.46:0/1805352846 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55773597de40 0x557735980230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:18:21.445 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:21.440+0000 7ff13df84740 4 client.16689 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-31T23:18:21.445 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:21.440+0000 7ff13df84740 1 -- 172.21.15.46:0/1805352846 --> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5577358db120 con 0x55773597de40 2024-03-31T23:18:21.445 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:21.440+0000 7ff138864640 1 --2- 172.21.15.46:0/1805352846 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55773597de40 0x557735980230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:18:21.447 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:21.441+0000 7ff138864640 1 --2- 172.21.15.46:0/1805352846 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55773597de40 0x557735980230 crc :-1 s=READY pgs=133 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=521dbe2d74eb9c7b server_cookie=bccb50739cf36c61 in_seq=0 out_seq=0 2024-03-31T23:18:21.447 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:21.441+0000 7ff11cff9640 10 client.16689 ms_handle_connect on v2:172.21.15.142:6838/2519143918 2024-03-31T23:18:21.447 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:21.441+0000 7ff11cff9640 1 -- 172.21.15.46:0/1805352846 <== mds.0 v2:172.21.15.142:6838/2519143918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5577358db120 con 0x55773597de40 2024-03-31T23:18:21.447 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:21.441+0000 7ff11cff9640 10 client.16689 handle_command_reply: tid=0 2024-03-31T23:18:21.447 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:21.441+0000 7ff13df84740 1 client.16689 shutdown 2024-03-31T23:18:21.447 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:21.441+0000 7ff13df84740 1 -- 172.21.15.46:0/1805352846 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5577359646c0 msgr2=0x557735967070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:18:21.448 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:21.441+0000 7ff13df84740 1 --2- 172.21.15.46:0/1805352846 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5577359646c0 0x557735967070 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7ff12c014ec0 tx=0x7ff12c004a40 comp rx=0 tx=0).stop 2024-03-31T23:18:21.448 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:21.441+0000 7ff13df84740 1 -- 172.21.15.46:0/1805352846 shutdown_connections 2024-03-31T23:18:21.448 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:21.442+0000 7ff13df84740 1 --2- 172.21.15.46:0/1805352846 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55773597de40 0x557735980230 crc :-1 s=READY pgs=133 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:18:21.448 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:21.442+0000 7ff13df84740 1 --2- 172.21.15.46:0/1805352846 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55773596ddb0 0x55773596b7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:18:21.448 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:21.442+0000 7ff11cff9640 0 client.16689 ms_handle_reset on v2:172.21.15.142:6838/2519143918 2024-03-31T23:18:21.448 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:21.442+0000 7ff13df84740 1 --2- 172.21.15.46:0/1805352846 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557735964fe0 0x5577359675b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:18:21.448 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:21.442+0000 7ff13df84740 1 --2- 172.21.15.46:0/1805352846 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5577359646c0 0x557735967070 unknown :-1 s=CLOSED pgs=739 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:18:21.448 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:21.442+0000 7ff13df84740 1 -- 172.21.15.46:0/1805352846 >> 172.21.15.46:0/1805352846 conn(0x55773595efe0 msgr2=0x557735960d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:18:21.448 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:21.442+0000 7ff13df84740 1 -- 172.21.15.46:0/1805352846 shutdown_connections 2024-03-31T23:18:21.448 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:21.442+0000 7ff13df84740 1 -- 172.21.15.46:0/1805352846 wait complete. 2024-03-31T23:18:21.448 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:21.442+0000 7ff13df84740 20 client.16689 trim_cache size 0 max 16384 2024-03-31T23:18:21.459 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:21.458+0000 7ff13df84740 1 Processor -- start 2024-03-31T23:18:21.459 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:21.458+0000 7ff13df84740 1 -- start start 2024-03-31T23:18:21.459 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:21.458+0000 7ff13df84740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5577359646c0 0x55773596d740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:18:21.460 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:21.458+0000 7ff13df84740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557735964fe0 0x5577359661d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:18:21.460 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:21.458+0000 7ff13df84740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55773596ddb0 0x55773595de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:18:21.460 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:21.458+0000 7ff13df84740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7ff1341af2f0 con 0x5577359646c0 2024-03-31T23:18:21.460 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:21.458+0000 7ff13df84740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7ff1341acbc0 con 0x55773596ddb0 2024-03-31T23:18:21.460 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:21.458+0000 7ff13df84740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7ff1341aaa30 con 0x557735964fe0 2024-03-31T23:18:21.460 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:21.458+0000 7ff139866640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5577359646c0 0x55773596d740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:18:21.460 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:21.458+0000 7ff139866640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5577359646c0 0x55773596d740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:58618/0 (socket says 172.21.15.46:58618) 2024-03-31T23:18:21.460 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:21.458+0000 7ff139866640 1 -- 172.21.15.46:0/1215243091 learned_addr learned my addr 172.21.15.46:0/1215243091 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:18:21.460 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:21.458+0000 7ff138864640 1 --2- 172.21.15.46:0/1215243091 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557735964fe0 0x5577359661d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:18:21.460 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:21.458+0000 7ff139866640 1 --2- 172.21.15.46:0/1215243091 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55773596ddb0 0x55773595de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:18:21.460 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:21.459+0000 7ff138864640 1 -- 172.21.15.46:0/1215243091 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55773596ddb0 msgr2=0x55773595de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:18:21.460 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:21.459+0000 7ff138864640 1 --2- 172.21.15.46:0/1215243091 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55773596ddb0 0x55773595de00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:18:21.460 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:21.459+0000 7ff138864640 1 -- 172.21.15.46:0/1215243091 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5577359646c0 msgr2=0x55773596d740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:18:21.461 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:21.459+0000 7ff138864640 1 --2- 172.21.15.46:0/1215243091 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5577359646c0 0x55773596d740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:18:21.461 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:21.459+0000 7ff138864640 1 -- 172.21.15.46:0/1215243091 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557735964e10 con 0x557735964fe0 2024-03-31T23:18:21.461 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:21.459+0000 7ff139866640 1 --2- 172.21.15.46:0/1215243091 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5577359646c0 0x55773596d740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:18:21.461 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:21.459+0000 7ff138864640 1 --2- 172.21.15.46:0/1215243091 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557735964fe0 0x5577359661d0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7ff12400e780 tx=0x7ff124009c00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f448ef6eef9ecfcc server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:18:21.461 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:21.459+0000 7ff11effd640 1 -- 172.21.15.46:0/1215243091 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff124032a30 con 0x557735964fe0 2024-03-31T23:18:21.461 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:21.459+0000 7ff11effd640 1 -- 172.21.15.46:0/1215243091 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff124032bd0 con 0x557735964fe0 2024-03-31T23:18:21.461 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:21.459+0000 7ff11effd640 1 -- 172.21.15.46:0/1215243091 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff124032790 con 0x557735964fe0 2024-03-31T23:18:21.464 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:21.460+0000 7ff13df84740 1 -- 172.21.15.46:0/1215243091 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557735964fe0 msgr2=0x5577359661d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:18:21.464 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:21.460+0000 7ff13df84740 1 --2- 172.21.15.46:0/1215243091 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557735964fe0 0x5577359661d0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7ff12400e780 tx=0x7ff124009c00 comp rx=0 tx=0).stop 2024-03-31T23:18:21.464 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:21.460+0000 7ff13df84740 1 -- 172.21.15.46:0/1215243091 shutdown_connections 2024-03-31T23:18:21.465 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:21.460+0000 7ff13df84740 1 --2- 172.21.15.46:0/1215243091 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55773596ddb0 0x55773595de00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:18:21.465 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:21.460+0000 7ff13df84740 1 --2- 172.21.15.46:0/1215243091 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557735964fe0 0x5577359661d0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:18:21.465 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:21.460+0000 7ff13df84740 1 --2- 172.21.15.46:0/1215243091 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5577359646c0 0x55773596d740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:18:21.465 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:21.460+0000 7ff13df84740 1 -- 172.21.15.46:0/1215243091 >> 172.21.15.46:0/1215243091 conn(0x55773595efe0 msgr2=0x55773595d6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:18:21.465 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:21.460+0000 7ff13df84740 1 -- 172.21.15.46:0/1215243091 shutdown_connections 2024-03-31T23:18:21.465 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:21.460+0000 7ff13df84740 1 -- 172.21.15.46:0/1215243091 wait complete. 2024-03-31T23:18:21.465 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:21.460+0000 7ff13df84740 1 -- start start 2024-03-31T23:18:21.465 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:21.460+0000 7ff13df84740 1 Processor -- start 2024-03-31T23:18:21.465 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:21.461+0000 7ff13df84740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5577359646c0 0x557735974f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:18:21.465 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:21.461+0000 7ff13df84740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557735964fe0 0x5577359754a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:18:21.465 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:21.461+0000 7ff13df84740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55773596ddb0 0x5577359e20d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:18:21.465 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:21.461+0000 7ff13df84740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7ff1341ab080 con 0x5577359646c0 2024-03-31T23:18:21.465 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:21.461+0000 7ff13df84740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7ff1341ac4f0 con 0x55773596ddb0 2024-03-31T23:18:21.466 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:21.461+0000 7ff13df84740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7ff1341ab750 con 0x557735964fe0 2024-03-31T23:18:21.466 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:21.461+0000 7ff13df84740 10 client.0 _ll_register_callbacks cb 0x7ff12c009ed0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T23:18:21.466 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:21.461+0000 7ff139866640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5577359646c0 0x557735974f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:18:21.466 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:21.461+0000 7ff139866640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5577359646c0 0x557735974f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:58626/0 (socket says 172.21.15.46:58626) 2024-03-31T23:18:21.466 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:21.461+0000 7ff139866640 1 -- 172.21.15.46:0/1689449333 learned_addr learned my addr 172.21.15.46:0/1689449333 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:18:21.466 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:21.461+0000 7ff139866640 1 --2- 172.21.15.46:0/1689449333 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55773596ddb0 0x5577359e20d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:18:21.466 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:21.461+0000 7ff138864640 1 --2- 172.21.15.46:0/1689449333 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557735964fe0 0x5577359754a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:18:21.466 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:21.461+0000 7ff139866640 1 -- 172.21.15.46:0/1689449333 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557735964fe0 msgr2=0x5577359754a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:18:21.466 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:21.461+0000 7ff139866640 1 --2- 172.21.15.46:0/1689449333 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557735964fe0 0x5577359754a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:18:21.466 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:21.461+0000 7ff139866640 1 -- 172.21.15.46:0/1689449333 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55773596ddb0 msgr2=0x5577359e20d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:18:21.466 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:21.461+0000 7ff139866640 1 --2- 172.21.15.46:0/1689449333 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55773596ddb0 0x5577359e20d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:18:21.466 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:21.461+0000 7ff139866640 1 -- 172.21.15.46:0/1689449333 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5577359e2610 con 0x5577359646c0 2024-03-31T23:18:21.466 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:21.461+0000 7ff139866640 1 --2- 172.21.15.46:0/1689449333 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5577359646c0 0x557735974f60 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7ff12c009fe0 tx=0x7ff12c009600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c7029dd9dda1997 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:18:21.466 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:21.461+0000 7ff11cff9640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-31T23:18:21.467 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:21.461+0000 7ff11cff9640 1 -- 172.21.15.46:0/1689449333 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff12c021ad0 con 0x5577359646c0 2024-03-31T23:18:21.467 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:21.462+0000 7ff11cff9640 1 -- 172.21.15.46:0/1689449333 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9554+0+0 (secure 0 0 0) 0x7ff12c01d040 con 0x5577359646c0 2024-03-31T23:18:21.467 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:21.462+0000 7ff13df84740 1 -- 172.21.15.46:0/1689449333 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff124032bd0 con 0x5577359646c0 2024-03-31T23:18:21.467 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:21.462+0000 7ff11cff9640 1 -- 172.21.15.46:0/1689449333 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7ff12c012450 con 0x5577359646c0 2024-03-31T23:18:21.467 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:21.462+0000 7ff13df84740 10 client.16695 fetch_fsmap learned FSMap version 30 2024-03-31T23:18:21.467 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:21.462+0000 7ff13df84740 1 -- 172.21.15.46:0/1689449333 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x5577359e37c0 con 0x5577359646c0 2024-03-31T23:18:21.467 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:21.462+0000 7ff11cff9640 1 -- 172.21.15.46:0/1689449333 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff124032bd0 con 0x5577359646c0 2024-03-31T23:18:21.467 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:21.462+0000 7ff11cff9640 1 -- 172.21.15.46:0/1689449333 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 30) ==== 8346+0+0 (secure 0 0 0) 0x7ff12c020070 con 0x5577359646c0 2024-03-31T23:18:21.467 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:21.462+0000 7ff13df84740 10 client.16695 fetch_fsmap finished waiting for FSMap version 30 2024-03-31T23:18:21.467 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:21.462+0000 7ff13df84740 10 client.16695 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-31T23:18:21.467 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:21.462+0000 7ff13df84740 20 client.16695 populate_metadata read hostname 'smithi046' 2024-03-31T23:18:21.467 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:21.462+0000 7ff13df84740 1 --2- 172.21.15.46:0/1689449333 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55773597de40 0x5577359e5eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:18:21.468 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:21.462+0000 7ff13df84740 4 client.16695 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-31T23:18:21.468 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:21.462+0000 7ff13df84740 1 -- 172.21.15.46:0/1689449333 --> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5577358d4e30 con 0x55773597de40 2024-03-31T23:18:21.468 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:21.463+0000 7ff138864640 1 --2- 172.21.15.46:0/1689449333 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55773597de40 0x5577359e5eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:18:21.468 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:21.463+0000 7ff138864640 1 --2- 172.21.15.46:0/1689449333 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55773597de40 0x5577359e5eb0 crc :-1 s=READY pgs=134 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6791fd7ca4ebadf8 server_cookie=f6d73ba9fdc30c79 in_seq=0 out_seq=0 2024-03-31T23:18:21.468 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:21.463+0000 7ff11cff9640 10 client.16695 ms_handle_connect on v2:172.21.15.142:6838/2519143918 2024-03-31T23:18:21.485 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:21.484+0000 7ff11cff9640 1 -- 172.21.15.46:0/1689449333 <== mds.0 v2:172.21.15.142:6838/2519143918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5577358d4e30 con 0x55773597de40 2024-03-31T23:18:21.485 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:21.484+0000 7ff11cff9640 10 client.16695 handle_command_reply: tid=0 2024-03-31T23:18:21.485 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:21.484+0000 7ff13df84740 1 client.16695 shutdown 2024-03-31T23:18:21.485 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:21.484+0000 7ff13df84740 1 -- 172.21.15.46:0/1689449333 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5577359646c0 msgr2=0x557735974f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:18:21.485 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:21.484+0000 7ff13df84740 1 --2- 172.21.15.46:0/1689449333 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5577359646c0 0x557735974f60 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7ff12c009fe0 tx=0x7ff12c009600 comp rx=0 tx=0).stop 2024-03-31T23:18:21.485 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:21.484+0000 7ff13df84740 1 -- 172.21.15.46:0/1689449333 shutdown_connections 2024-03-31T23:18:21.485 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:21.484+0000 7ff13df84740 1 --2- 172.21.15.46:0/1689449333 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55773597de40 0x5577359e5eb0 crc :-1 s=READY pgs=134 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:18:21.485 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:21.484+0000 7ff13df84740 1 --2- 172.21.15.46:0/1689449333 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55773596ddb0 0x5577359e20d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:18:21.485 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:21.484+0000 7ff13df84740 1 --2- 172.21.15.46:0/1689449333 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557735964fe0 0x5577359754a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:18:21.485 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:21.484+0000 7ff11cff9640 0 client.16695 ms_handle_reset on v2:172.21.15.142:6838/2519143918 2024-03-31T23:18:21.485 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:21.484+0000 7ff13df84740 1 --2- 172.21.15.46:0/1689449333 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5577359646c0 0x557735974f60 unknown :-1 s=CLOSED pgs=740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:18:21.486 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:21.484+0000 7ff13df84740 1 -- 172.21.15.46:0/1689449333 >> 172.21.15.46:0/1689449333 conn(0x55773595efe0 msgr2=0x55773595ffe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:18:21.486 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:21.485+0000 7ff13df84740 1 -- 172.21.15.46:0/1689449333 shutdown_connections 2024-03-31T23:18:21.486 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:21.485+0000 7ff13df84740 1 -- 172.21.15.46:0/1689449333 wait complete. 2024-03-31T23:18:21.486 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:21.485+0000 7ff13df84740 20 client.16695 trim_cache size 0 max 16384 2024-03-31T23:18:21.486 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout:{ 2024-03-31T23:18:21.486 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout: "message": "", 2024-03-31T23:18:21.486 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout: "return_code": 0 2024-03-31T23:18:21.486 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout:} 2024-03-31T23:18:21.487 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:21.485+0000 7ff13b2c9640 1 -- 172.21.15.46:0/1828878674 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7ff1200a26b0 msgr2=0x7ff1200a4b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:18:21.487 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:21.485+0000 7ff13b2c9640 1 --2- 172.21.15.46:0/1828878674 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7ff1200a26b0 0x7ff1200a4b70 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7ff1300ee350 tx=0x7ff1300c1040 comp rx=0 tx=0).stop 2024-03-31T23:18:21.487 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:21.485+0000 7ff13b2c9640 1 -- 172.21.15.46:0/1828878674 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff1341016f0 msgr2=0x7ff134197ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:18:21.487 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:21.485+0000 7ff13b2c9640 1 --2- 172.21.15.46:0/1828878674 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff1341016f0 0x7ff134197ec0 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7ff130009230 tx=0x7ff13002c710 comp rx=0 tx=0).stop 2024-03-31T23:18:21.487 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:21.485+0000 7ff13b2c9640 1 -- 172.21.15.46:0/1828878674 shutdown_connections 2024-03-31T23:18:21.487 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:21.485+0000 7ff13b2c9640 1 --2- 172.21.15.46:0/1828878674 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7ff1200a26b0 0x7ff1200a4b70 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:18:21.487 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:21.485+0000 7ff13b2c9640 1 --2- 172.21.15.46:0/1828878674 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff134107700 0x7ff13419c780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:18:21.488 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:21.485+0000 7ff13b2c9640 1 --2- 172.21.15.46:0/1828878674 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff1341020b0 0x7ff134198400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:18:21.488 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:21.485+0000 7ff13b2c9640 1 --2- 172.21.15.46:0/1828878674 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff1341016f0 0x7ff134197ec0 unknown :-1 s=CLOSED pgs=737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:18:21.488 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:21.485+0000 7ff13b2c9640 1 -- 172.21.15.46:0/1828878674 >> 172.21.15.46:0/1828878674 conn(0x7ff1340fd1e0 msgr2=0x7ff1340fda00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:18:21.488 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:21.486+0000 7ff13b2c9640 1 -- 172.21.15.46:0/1828878674 shutdown_connections 2024-03-31T23:18:21.488 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:21.486+0000 7ff13b2c9640 1 -- 172.21.15.46:0/1828878674 wait complete. 2024-03-31T23:18:22.200 INFO:tasks.workunit.client.0.smithi046.stdout: 1 149783 2.48 MB/sec execute 208 sec latency 24.736 ms 2024-03-31T23:18:22.795 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:18:22 smithi162 ceph-mon[36237]: pgmap v204: 129 pgs: 129 active+clean; 99 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 415 KiB/s rd, 1.3 MiB/s wr, 112 op/s 2024-03-31T23:18:22.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:18:22 smithi046 ceph-mon[32498]: pgmap v204: 129 pgs: 129 active+clean; 99 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 415 KiB/s rd, 1.3 MiB/s wr, 112 op/s 2024-03-31T23:18:22.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:18:22 smithi142 ceph-mon[35271]: pgmap v204: 129 pgs: 129 active+clean; 99 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 415 KiB/s rd, 1.3 MiB/s wr, 112 op/s 2024-03-31T23:18:23.200 INFO:tasks.workunit.client.0.smithi046.stdout: 1 150238 2.48 MB/sec execute 209 sec latency 17.941 ms 2024-03-31T23:18:24.200 INFO:tasks.workunit.client.0.smithi046.stdout: 1 150707 2.48 MB/sec execute 210 sec latency 20.867 ms 2024-03-31T23:18:24.795 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:18:24 smithi162 ceph-mon[36237]: pgmap v205: 129 pgs: 129 active+clean; 99 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 415 KiB/s rd, 1.3 MiB/s wr, 112 op/s 2024-03-31T23:18:24.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:18:24 smithi046 ceph-mon[32498]: pgmap v205: 129 pgs: 129 active+clean; 99 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 415 KiB/s rd, 1.3 MiB/s wr, 112 op/s 2024-03-31T23:18:24.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:18:24 smithi142 ceph-mon[35271]: pgmap v205: 129 pgs: 129 active+clean; 99 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 415 KiB/s rd, 1.3 MiB/s wr, 112 op/s 2024-03-31T23:18:25.201 INFO:tasks.workunit.client.0.smithi046.stdout: 1 151309 2.48 MB/sec execute 211 sec latency 21.919 ms 2024-03-31T23:18:26.200 INFO:tasks.workunit.client.0.smithi046.stdout: 1 151781 2.48 MB/sec execute 212 sec latency 18.396 ms 2024-03-31T23:18:26.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:18:26 smithi162 ceph-mon[36237]: pgmap v206: 129 pgs: 129 active+clean; 98 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 654 KiB/s rd, 1.9 MiB/s wr, 172 op/s 2024-03-31T23:18:26.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:18:26 smithi046 ceph-mon[32498]: pgmap v206: 129 pgs: 129 active+clean; 98 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 654 KiB/s rd, 1.9 MiB/s wr, 172 op/s 2024-03-31T23:18:26.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:18:26 smithi142 ceph-mon[35271]: pgmap v206: 129 pgs: 129 active+clean; 98 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 654 KiB/s rd, 1.9 MiB/s wr, 172 op/s 2024-03-31T23:18:27.200 INFO:tasks.workunit.client.0.smithi046.stdout: 1 152329 2.49 MB/sec execute 213 sec latency 16.091 ms 2024-03-31T23:18:28.200 INFO:tasks.workunit.client.0.smithi046.stdout: 1 152798 2.50 MB/sec execute 214 sec latency 22.379 ms 2024-03-31T23:18:28.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:18:28 smithi162 ceph-mon[36237]: pgmap v207: 129 pgs: 129 active+clean; 98 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 452 KiB/s rd, 1.2 MiB/s wr, 117 op/s 2024-03-31T23:18:28.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:18:28 smithi046 ceph-mon[32498]: pgmap v207: 129 pgs: 129 active+clean; 98 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 452 KiB/s rd, 1.2 MiB/s wr, 117 op/s 2024-03-31T23:18:28.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:18:28 smithi142 ceph-mon[35271]: pgmap v207: 129 pgs: 129 active+clean; 98 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 452 KiB/s rd, 1.2 MiB/s wr, 117 op/s 2024-03-31T23:18:29.200 INFO:tasks.workunit.client.0.smithi046.stdout: 1 153058 2.49 MB/sec execute 215 sec latency 17.089 ms 2024-03-31T23:18:30.201 INFO:tasks.workunit.client.0.smithi046.stdout: 1 153459 2.48 MB/sec execute 216 sec latency 16.110 ms 2024-03-31T23:18:30.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:18:30 smithi162 ceph-mon[36237]: pgmap v208: 129 pgs: 129 active+clean; 100 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 650 KiB/s rd, 1.9 MiB/s wr, 173 op/s 2024-03-31T23:18:30.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:18:30 smithi046 ceph-mon[32498]: pgmap v208: 129 pgs: 129 active+clean; 100 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 650 KiB/s rd, 1.9 MiB/s wr, 173 op/s 2024-03-31T23:18:30.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:18:30 smithi142 ceph-mon[35271]: pgmap v208: 129 pgs: 129 active+clean; 100 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 650 KiB/s rd, 1.9 MiB/s wr, 173 op/s 2024-03-31T23:18:31.200 INFO:tasks.workunit.client.0.smithi046.stdout: 1 153975 2.49 MB/sec execute 217 sec latency 18.075 ms 2024-03-31T23:18:32.201 INFO:tasks.workunit.client.0.smithi046.stdout: 1 154431 2.48 MB/sec execute 218 sec latency 28.645 ms 2024-03-31T23:18:32.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:18:32 smithi162 ceph-mon[36237]: pgmap v209: 129 pgs: 129 active+clean; 100 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 437 KiB/s rd, 1.3 MiB/s wr, 116 op/s 2024-03-31T23:18:32.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:18:32 smithi046 ceph-mon[32498]: pgmap v209: 129 pgs: 129 active+clean; 100 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 437 KiB/s rd, 1.3 MiB/s wr, 116 op/s 2024-03-31T23:18:32.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:18:32 smithi142 ceph-mon[35271]: pgmap v209: 129 pgs: 129 active+clean; 100 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 437 KiB/s rd, 1.3 MiB/s wr, 116 op/s 2024-03-31T23:18:33.200 INFO:tasks.workunit.client.0.smithi046.stdout: 1 154975 2.47 MB/sec execute 219 sec latency 20.780 ms 2024-03-31T23:18:34.201 INFO:tasks.workunit.client.0.smithi046.stdout: 1 155565 2.49 MB/sec execute 220 sec latency 20.393 ms 2024-03-31T23:18:34.584 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:34.580+0000 7fbac3855640 1 -- 172.21.15.46:0/874602041 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbabc1008a0 msgr2=0x7fbabc100d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:18:34.584 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:34.580+0000 7fbac3855640 1 --2- 172.21.15.46:0/874602041 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbabc1008a0 0x7fbabc100d20 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7fbaac009900 tx=0x7fbaac02bc30 comp rx=0 tx=0).stop 2024-03-31T23:18:34.584 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:34.580+0000 7fbac3855640 1 -- 172.21.15.46:0/874602041 shutdown_connections 2024-03-31T23:18:34.584 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:34.580+0000 7fbac3855640 1 --2- 172.21.15.46:0/874602041 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbabc101260 0x7fbabc10d480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:18:34.584 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:34.580+0000 7fbac3855640 1 --2- 172.21.15.46:0/874602041 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbabc1008a0 0x7fbabc100d20 unknown :-1 s=CLOSED pgs=741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:18:34.584 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:34.580+0000 7fbac3855640 1 --2- 172.21.15.46:0/874602041 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbabc1055f0 0x7fbabc1059d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:18:34.584 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:34.580+0000 7fbac3855640 1 -- 172.21.15.46:0/874602041 >> 172.21.15.46:0/874602041 conn(0x7fbabc0774a0 msgr2=0x7fbabc0778b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:18:34.584 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:34.580+0000 7fbac3855640 1 -- 172.21.15.46:0/874602041 shutdown_connections 2024-03-31T23:18:34.584 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:34.580+0000 7fbac3855640 1 -- 172.21.15.46:0/874602041 wait complete. 2024-03-31T23:18:34.585 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:34.581+0000 7fbac3855640 1 Processor -- start 2024-03-31T23:18:34.585 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:34.581+0000 7fbac3855640 1 -- start start 2024-03-31T23:18:34.585 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:34.581+0000 7fbac3855640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbabc1008a0 0x7fbabc1022c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:18:34.585 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:34.581+0000 7fbac3855640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbabc101260 0x7fbabc102800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:18:34.585 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:34.581+0000 7fbac3855640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbabc1055f0 0x7fbabc102d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:18:34.585 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:34.581+0000 7fbac3855640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fbabc110a10 con 0x7fbabc1055f0 2024-03-31T23:18:34.585 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:34.581+0000 7fbac3855640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fbabc110890 con 0x7fbabc101260 2024-03-31T23:18:34.585 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:34.581+0000 7fbac3855640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fbabc110b90 con 0x7fbabc1008a0 2024-03-31T23:18:34.585 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:34.581+0000 7fbac1df2640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbabc1055f0 0x7fbabc102d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:18:34.585 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:34.581+0000 7fbac15f1640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbabc1008a0 0x7fbabc1022c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:18:34.585 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:34.581+0000 7fbac1df2640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbabc1055f0 0x7fbabc102d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:53366/0 (socket says 172.21.15.46:53366) 2024-03-31T23:18:34.586 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:34.581+0000 7fbac1df2640 1 -- 172.21.15.46:0/1391910489 learned_addr learned my addr 172.21.15.46:0/1391910489 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:18:34.586 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:34.581+0000 7fbac0df0640 1 --2- 172.21.15.46:0/1391910489 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbabc101260 0x7fbabc102800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:18:34.586 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:34.581+0000 7fbac15f1640 1 -- 172.21.15.46:0/1391910489 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbabc101260 msgr2=0x7fbabc102800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:18:34.586 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:34.581+0000 7fbac15f1640 1 --2- 172.21.15.46:0/1391910489 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbabc101260 0x7fbabc102800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:18:34.586 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:34.581+0000 7fbac15f1640 1 -- 172.21.15.46:0/1391910489 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbabc1055f0 msgr2=0x7fbabc102d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:18:34.586 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:34.581+0000 7fbac15f1640 1 --2- 172.21.15.46:0/1391910489 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbabc1055f0 0x7fbabc102d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:18:34.586 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:34.581+0000 7fbac15f1640 1 -- 172.21.15.46:0/1391910489 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbabc1a6420 con 0x7fbabc1008a0 2024-03-31T23:18:34.586 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:34.581+0000 7fbac1df2640 1 --2- 172.21.15.46:0/1391910489 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbabc1055f0 0x7fbabc102d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:18:34.586 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:34.581+0000 7fbac0df0640 1 --2- 172.21.15.46:0/1391910489 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbabc101260 0x7fbabc102800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:18:34.586 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:34.582+0000 7fbac15f1640 1 --2- 172.21.15.46:0/1391910489 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbabc1008a0 0x7fbabc1022c0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fbab800e9f0 tx=0x7fbab800eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:18:34.586 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:34.582+0000 7fbab27fc640 1 -- 172.21.15.46:0/1391910489 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbab800ccc0 con 0x7fbabc1008a0 2024-03-31T23:18:34.587 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:34.582+0000 7fbab27fc640 1 -- 172.21.15.46:0/1391910489 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fbab8005020 con 0x7fbabc1008a0 2024-03-31T23:18:34.587 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:34.582+0000 7fbac3855640 1 -- 172.21.15.46:0/1391910489 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbabc1a6710 con 0x7fbabc1008a0 2024-03-31T23:18:34.587 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:34.582+0000 7fbab27fc640 1 -- 172.21.15.46:0/1391910489 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbab8004a00 con 0x7fbabc1008a0 2024-03-31T23:18:34.587 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:34.582+0000 7fbac3855640 1 -- 172.21.15.46:0/1391910489 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbabc1a6be0 con 0x7fbabc1008a0 2024-03-31T23:18:34.594 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:34.584+0000 7fbab27fc640 1 -- 172.21.15.46:0/1391910489 <== mon.1 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 25) ==== 137451+0+0 (secure 0 0 0) 0x7fbab8004ba0 con 0x7fbabc1008a0 2024-03-31T23:18:34.594 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:34.585+0000 7fbac643a740 1 Processor -- start 2024-03-31T23:18:34.594 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:34.585+0000 7fbac643a740 1 -- start start 2024-03-31T23:18:34.594 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:34.585+0000 7fbac643a740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559db207b6c0 0x559db207baa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:18:34.594 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:34.585+0000 7fbac643a740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559db207bfe0 0x559db2084870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:18:34.595 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:34.585+0000 7fbac643a740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559db2084db0 0x559db20871a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:18:34.595 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:34.585+0000 7fbac643a740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x559db1fde8f0 con 0x559db207bfe0 2024-03-31T23:18:34.595 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:34.585+0000 7fbac643a740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x559db1f878d0 con 0x559db2084db0 2024-03-31T23:18:34.595 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:34.585+0000 7fbac643a740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x559db1fda990 con 0x559db207b6c0 2024-03-31T23:18:34.595 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:34.585+0000 7fbab27fc640 1 --2- 172.21.15.46:0/1391910489 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fbaa80a26b0 0x7fbaa80a4b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:18:34.595 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:34.585+0000 7fbac15f1640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559db207bfe0 0x559db2084870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:18:34.595 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:34.585+0000 7fbac0df0640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559db2084db0 0x559db20871a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:18:34.595 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:34.585+0000 7fbac1df2640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559db207b6c0 0x559db207baa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:18:34.595 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:34.585+0000 7fbac0df0640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559db2084db0 0x559db20871a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.46:45266/0 (socket says 172.21.15.46:45266) 2024-03-31T23:18:34.595 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:34.585+0000 7fbac0df0640 1 -- 172.21.15.46:0/2178044228 learned_addr learned my addr 172.21.15.46:0/2178044228 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:18:34.595 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:34.585+0000 7fbac1df2640 1 --2- 172.21.15.46:0/1391910489 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fbaa80a26b0 0x7fbaa80a4b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:18:34.596 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:34.585+0000 7fbab27fc640 1 -- 172.21.15.46:0/1391910489 <== mon.1 v2:172.21.15.162:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9554+0+0 (secure 0 0 0) 0x7fbab80cee90 con 0x7fbabc1008a0 2024-03-31T23:18:34.596 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:34.585+0000 7fbac0df0640 1 -- 172.21.15.46:0/2178044228 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559db207b6c0 msgr2=0x559db207baa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:18:34.596 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:34.585+0000 7fbac0df0640 1 --2- 172.21.15.46:0/2178044228 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559db207b6c0 0x559db207baa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:18:34.596 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:34.585+0000 7fbac0df0640 1 -- 172.21.15.46:0/2178044228 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559db207bfe0 msgr2=0x559db2084870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:18:34.596 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:34.585+0000 7fbac0df0640 1 --2- 172.21.15.46:0/2178044228 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559db207bfe0 0x559db2084870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:18:34.596 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:34.586+0000 7fbac0df0640 1 -- 172.21.15.46:0/2178044228 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559db1cf3210 con 0x559db2084db0 2024-03-31T23:18:34.596 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:34.586+0000 7fbac1df2640 1 --2- 172.21.15.46:0/1391910489 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fbaa80a26b0 0x7fbaa80a4b70 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fbab4009fd0 tx=0x7fbab4009500 comp rx=0 tx=0).ready entity=mgr.15654 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:18:34.596 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:34.586+0000 7fbac0df0640 1 --2- 172.21.15.46:0/2178044228 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559db2084db0 0x559db20871a0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fbaac000c00 tx=0x7fbaac009560 comp rx=0 tx=0).ready entity=mon.2 client_cookie=532639ca98a0afec server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:18:34.596 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:34.586+0000 7fbaa77fe640 1 -- 172.21.15.46:0/2178044228 <== mon.2 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbaac045070 con 0x559db2084db0 2024-03-31T23:18:34.596 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:34.586+0000 7fbaa77fe640 1 -- 172.21.15.46:0/2178044228 <== mon.2 v2:172.21.15.142:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fbaac02c780 con 0x559db2084db0 2024-03-31T23:18:34.596 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:34.586+0000 7fbaa77fe640 1 -- 172.21.15.46:0/2178044228 <== mon.2 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbaac003bd0 con 0x559db2084db0 2024-03-31T23:18:34.596 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:34.587+0000 7fbac643a740 1 -- 172.21.15.46:0/2178044228 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559db2084db0 msgr2=0x559db20871a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:18:34.597 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:34.587+0000 7fbac643a740 1 --2- 172.21.15.46:0/2178044228 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559db2084db0 0x559db20871a0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fbaac000c00 tx=0x7fbaac009560 comp rx=0 tx=0).stop 2024-03-31T23:18:34.597 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:34.587+0000 7fbac643a740 1 -- 172.21.15.46:0/2178044228 shutdown_connections 2024-03-31T23:18:34.597 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:34.587+0000 7fbac643a740 1 --2- 172.21.15.46:0/2178044228 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559db2084db0 0x559db20871a0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:18:34.597 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:34.587+0000 7fbac643a740 1 --2- 172.21.15.46:0/2178044228 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559db207bfe0 0x559db2084870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:18:34.597 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:34.587+0000 7fbac643a740 1 --2- 172.21.15.46:0/2178044228 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559db207b6c0 0x559db207baa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:18:34.597 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:34.587+0000 7fbac643a740 1 -- 172.21.15.46:0/2178044228 >> 172.21.15.46:0/2178044228 conn(0x559db2075fe0 msgr2=0x559db20763c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:18:34.597 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:34.587+0000 7fbac643a740 1 -- 172.21.15.46:0/2178044228 shutdown_connections 2024-03-31T23:18:34.597 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:34.587+0000 7fbac643a740 1 -- 172.21.15.46:0/2178044228 wait complete. 2024-03-31T23:18:34.598 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:34.587+0000 7fbac643a740 1 -- start start 2024-03-31T23:18:34.598 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:34.587+0000 7fbac643a740 1 Processor -- start 2024-03-31T23:18:34.598 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:34.588+0000 7fbac643a740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559db207b6c0 0x559db2081dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:18:34.598 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:34.588+0000 7fbac643a740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559db207bfe0 0x559db2082300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:18:34.598 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:34.588+0000 7fbac643a740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559db2084db0 0x559db2082840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:18:34.598 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:34.588+0000 7fbac643a740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x559db2087980 con 0x559db2084db0 2024-03-31T23:18:34.598 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:34.588+0000 7fbac643a740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x559db1f7cf80 con 0x559db207bfe0 2024-03-31T23:18:34.598 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:34.588+0000 7fbac643a740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x559db2087b00 con 0x559db207b6c0 2024-03-31T23:18:34.599 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:34.588+0000 7fbac643a740 10 client.0 _ll_register_callbacks cb 0x559db1fd5290 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T23:18:34.599 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:34.588+0000 7fbac1df2640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559db2084db0 0x559db2082840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:18:34.599 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:34.588+0000 7fbac0df0640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559db207bfe0 0x559db2082300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:18:34.599 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:34.588+0000 7fbac15f1640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559db207b6c0 0x559db2081dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:18:34.599 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:34.588+0000 7fbac1df2640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559db2084db0 0x559db2082840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:53376/0 (socket says 172.21.15.46:53376) 2024-03-31T23:18:34.599 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:34.588+0000 7fbac1df2640 1 -- 172.21.15.46:0/2355616034 learned_addr learned my addr 172.21.15.46:0/2355616034 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:18:34.599 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:34.588+0000 7fbac15f1640 1 -- 172.21.15.46:0/2355616034 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559db207bfe0 msgr2=0x559db2082300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:18:34.599 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:34.588+0000 7fbac15f1640 1 --2- 172.21.15.46:0/2355616034 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559db207bfe0 0x559db2082300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:18:34.599 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:34.588+0000 7fbac15f1640 1 -- 172.21.15.46:0/2355616034 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559db2084db0 msgr2=0x559db2082840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:18:34.599 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:34.588+0000 7fbac15f1640 1 --2- 172.21.15.46:0/2355616034 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559db2084db0 0x559db2082840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:18:34.599 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:34.588+0000 7fbac15f1640 1 -- 172.21.15.46:0/2355616034 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559db1dbf620 con 0x559db207b6c0 2024-03-31T23:18:34.599 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:34.588+0000 7fbac1df2640 1 --2- 172.21.15.46:0/2355616034 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559db2084db0 0x559db2082840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:18:34.600 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:34.588+0000 7fbac0df0640 1 --2- 172.21.15.46:0/2355616034 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559db207bfe0 0x559db2082300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:18:34.600 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:34.588+0000 7fbac15f1640 1 --2- 172.21.15.46:0/2355616034 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559db207b6c0 0x559db2081dc0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fbab80cc140 tx=0x7fbab806d160 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8c9c47931dcf4544 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:18:34.600 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:34.588+0000 7fbaa4ff9640 10 client.0 ms_handle_connect on v2:172.21.15.162:3300/0 2024-03-31T23:18:34.600 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:34.588+0000 7fbaa4ff9640 1 -- 172.21.15.46:0/2355616034 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbab801c040 con 0x559db207b6c0 2024-03-31T23:18:34.600 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:34.589+0000 7fbac643a740 1 -- 172.21.15.46:0/2355616034 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbaac02c780 con 0x559db207b6c0 2024-03-31T23:18:34.600 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:34.589+0000 7fbaa4ff9640 1 -- 172.21.15.46:0/2355616034 <== mon.1 v2:172.21.15.162:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9554+0+0 (secure 0 0 0) 0x7fbab80cc3f0 con 0x559db207b6c0 2024-03-31T23:18:34.600 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:34.589+0000 7fbaa4ff9640 1 -- 172.21.15.46:0/2355616034 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fbab80cc5f0 con 0x559db207b6c0 2024-03-31T23:18:34.600 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:34.589+0000 7fbac643a740 10 client.26440 fetch_fsmap learned FSMap version 30 2024-03-31T23:18:34.600 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:34.589+0000 7fbac643a740 1 -- 172.21.15.46:0/2355616034 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x559db2093880 con 0x559db207b6c0 2024-03-31T23:18:34.600 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:34.589+0000 7fbaa4ff9640 1 -- 172.21.15.46:0/2355616034 <== mon.1 v2:172.21.15.162:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fbaac02c780 con 0x559db207b6c0 2024-03-31T23:18:34.600 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:34.589+0000 7fbaa4ff9640 1 -- 172.21.15.46:0/2355616034 <== mon.1 v2:172.21.15.162:3300/0 5 ==== fsmap(e 30) ==== 8346+0+0 (secure 0 0 0) 0x7fbab80a3720 con 0x559db207b6c0 2024-03-31T23:18:34.601 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:34.589+0000 7fbac643a740 10 client.26440 fetch_fsmap finished waiting for FSMap version 30 2024-03-31T23:18:34.601 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:34.589+0000 7fbac643a740 10 client.26440 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-31T23:18:34.601 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:34.589+0000 7fbac643a740 20 client.26440 populate_metadata read hostname 'smithi046' 2024-03-31T23:18:34.601 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:34.589+0000 7fbac643a740 1 --2- 172.21.15.46:0/2355616034 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x559db2094d40 0x559db2097130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:18:34.601 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:34.589+0000 7fbac643a740 4 client.26440 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-31T23:18:34.601 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:34.589+0000 7fbac643a740 1 -- 172.21.15.46:0/2355616034 --> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559db1ff2120 con 0x559db2094d40 2024-03-31T23:18:34.601 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:34.589+0000 7fbac0df0640 1 --2- 172.21.15.46:0/2355616034 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x559db2094d40 0x559db2097130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:18:34.601 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:34.590+0000 7fbac0df0640 1 --2- 172.21.15.46:0/2355616034 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x559db2094d40 0x559db2097130 crc :-1 s=READY pgs=135 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8e151d4de01bbfd5 server_cookie=8d83a38f44c1d109 in_seq=0 out_seq=0 2024-03-31T23:18:34.601 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:34.590+0000 7fbaa4ff9640 10 client.26440 ms_handle_connect on v2:172.21.15.142:6838/2519143918 2024-03-31T23:18:34.601 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:34.591+0000 7fbaa4ff9640 1 -- 172.21.15.46:0/2355616034 <== mds.0 v2:172.21.15.142:6838/2519143918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x559db1ff2120 con 0x559db2094d40 2024-03-31T23:18:34.601 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:34.591+0000 7fbaa4ff9640 10 client.26440 handle_command_reply: tid=0 2024-03-31T23:18:34.601 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:34.591+0000 7fbac643a740 1 client.26440 shutdown 2024-03-31T23:18:34.602 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:34.591+0000 7fbac643a740 1 -- 172.21.15.46:0/2355616034 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559db207b6c0 msgr2=0x559db2081dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:18:34.602 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:34.591+0000 7fbac643a740 1 --2- 172.21.15.46:0/2355616034 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559db207b6c0 0x559db2081dc0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fbab80cc140 tx=0x7fbab806d160 comp rx=0 tx=0).stop 2024-03-31T23:18:34.602 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:34.591+0000 7fbac643a740 1 -- 172.21.15.46:0/2355616034 shutdown_connections 2024-03-31T23:18:34.602 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:34.591+0000 7fbac643a740 1 --2- 172.21.15.46:0/2355616034 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x559db2094d40 0x559db2097130 crc :-1 s=READY pgs=135 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:18:34.602 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:34.591+0000 7fbac643a740 1 --2- 172.21.15.46:0/2355616034 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559db2084db0 0x559db2082840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:18:34.602 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:34.591+0000 7fbaa4ff9640 0 client.26440 ms_handle_reset on v2:172.21.15.142:6838/2519143918 2024-03-31T23:18:34.602 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:34.591+0000 7fbac643a740 1 --2- 172.21.15.46:0/2355616034 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559db207bfe0 0x559db2082300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:18:34.602 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:34.591+0000 7fbac643a740 1 --2- 172.21.15.46:0/2355616034 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559db207b6c0 0x559db2081dc0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:18:34.602 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:34.591+0000 7fbac643a740 1 -- 172.21.15.46:0/2355616034 >> 172.21.15.46:0/2355616034 conn(0x559db2075fe0 msgr2=0x559db2077d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:18:34.602 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:34.591+0000 7fbac643a740 1 -- 172.21.15.46:0/2355616034 shutdown_connections 2024-03-31T23:18:34.602 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:34.591+0000 7fbac643a740 1 -- 172.21.15.46:0/2355616034 wait complete. 2024-03-31T23:18:34.602 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:34.591+0000 7fbac643a740 20 client.26440 trim_cache size 0 max 16384 2024-03-31T23:18:34.615 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:34.609+0000 7fbac643a740 1 Processor -- start 2024-03-31T23:18:34.615 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:34.609+0000 7fbac643a740 1 -- start start 2024-03-31T23:18:34.615 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:34.609+0000 7fbac643a740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559db207b6c0 0x559db20844f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:18:34.615 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:34.609+0000 7fbac643a740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559db207bfe0 0x559db2090720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:18:34.615 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:34.609+0000 7fbac643a740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559db2084db0 0x559db209c980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:18:34.615 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:34.609+0000 7fbac643a740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fbabc1b3590 con 0x559db2084db0 2024-03-31T23:18:34.615 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:34.609+0000 7fbac643a740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fbabc1b0e60 con 0x559db207b6c0 2024-03-31T23:18:34.615 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:34.609+0000 7fbac643a740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fbabc1aecd0 con 0x559db207bfe0 2024-03-31T23:18:34.615 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:34.609+0000 7fbac1df2640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559db2084db0 0x559db209c980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:18:34.615 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:34.609+0000 7fbac1df2640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559db2084db0 0x559db209c980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:53388/0 (socket says 172.21.15.46:53388) 2024-03-31T23:18:34.615 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:34.609+0000 7fbac1df2640 1 -- 172.21.15.46:0/3779317463 learned_addr learned my addr 172.21.15.46:0/3779317463 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:18:34.616 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:34.609+0000 7fbac0df0640 1 --2- 172.21.15.46:0/3779317463 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559db207bfe0 0x559db2090720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:18:34.616 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:34.609+0000 7fbac15f1640 1 --2- 172.21.15.46:0/3779317463 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559db207b6c0 0x559db20844f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:18:34.616 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:34.610+0000 7fbac1df2640 1 -- 172.21.15.46:0/3779317463 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559db207bfe0 msgr2=0x559db2090720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:18:34.616 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:34.610+0000 7fbac1df2640 1 --2- 172.21.15.46:0/3779317463 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559db207bfe0 0x559db2090720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:18:34.616 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:34.610+0000 7fbac1df2640 1 -- 172.21.15.46:0/3779317463 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559db207b6c0 msgr2=0x559db20844f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:18:34.616 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:34.610+0000 7fbac1df2640 1 --2- 172.21.15.46:0/3779317463 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559db207b6c0 0x559db20844f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:18:34.616 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:34.610+0000 7fbac1df2640 1 -- 172.21.15.46:0/3779317463 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559db2087510 con 0x559db2084db0 2024-03-31T23:18:34.616 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:34.610+0000 7fbac1df2640 1 --2- 172.21.15.46:0/3779317463 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559db2084db0 0x559db209c980 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7fbab40094b0 tx=0x7fbab4016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df9957da9a4bec27 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:18:34.616 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:34.610+0000 7fbaa6ffd640 1 -- 172.21.15.46:0/3779317463 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbab4004b20 con 0x559db2084db0 2024-03-31T23:18:34.616 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:34.610+0000 7fbaa6ffd640 1 -- 172.21.15.46:0/3779317463 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fbab4004cc0 con 0x559db2084db0 2024-03-31T23:18:34.616 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:34.610+0000 7fbaa6ffd640 1 -- 172.21.15.46:0/3779317463 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbab40051f0 con 0x559db2084db0 2024-03-31T23:18:34.617 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:34.611+0000 7fbac643a740 1 -- 172.21.15.46:0/3779317463 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559db2084db0 msgr2=0x559db209c980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:18:34.617 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:34.611+0000 7fbac643a740 1 --2- 172.21.15.46:0/3779317463 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559db2084db0 0x559db209c980 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7fbab40094b0 tx=0x7fbab4016000 comp rx=0 tx=0).stop 2024-03-31T23:18:34.617 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:34.611+0000 7fbac643a740 1 -- 172.21.15.46:0/3779317463 shutdown_connections 2024-03-31T23:18:34.617 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:34.611+0000 7fbac643a740 1 --2- 172.21.15.46:0/3779317463 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559db2084db0 0x559db209c980 unknown :-1 s=CLOSED pgs=742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:18:34.617 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:34.611+0000 7fbac643a740 1 --2- 172.21.15.46:0/3779317463 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559db207bfe0 0x559db2090720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:18:34.617 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:34.611+0000 7fbac643a740 1 --2- 172.21.15.46:0/3779317463 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559db207b6c0 0x559db20844f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:18:34.617 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:34.611+0000 7fbac643a740 1 -- 172.21.15.46:0/3779317463 >> 172.21.15.46:0/3779317463 conn(0x559db2075fe0 msgr2=0x559db20763c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:18:34.617 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:34.611+0000 7fbac643a740 1 -- 172.21.15.46:0/3779317463 shutdown_connections 2024-03-31T23:18:34.617 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:34.611+0000 7fbac643a740 1 -- 172.21.15.46:0/3779317463 wait complete. 2024-03-31T23:18:34.617 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:34.611+0000 7fbac643a740 1 -- start start 2024-03-31T23:18:34.617 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:34.611+0000 7fbac643a740 1 Processor -- start 2024-03-31T23:18:34.617 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:34.612+0000 7fbac643a740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559db207b6c0 0x559db208d090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:18:34.618 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:34.612+0000 7fbac643a740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559db207bfe0 0x559db207e6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:18:34.618 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:34.612+0000 7fbac643a740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559db2084db0 0x559db207ec50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:18:34.618 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:34.612+0000 7fbac643a740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fbabc1af320 con 0x559db207b6c0 2024-03-31T23:18:34.618 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:34.612+0000 7fbac643a740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fbabc1b0790 con 0x559db207bfe0 2024-03-31T23:18:34.618 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:34.612+0000 7fbac643a740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fbabc1af9f0 con 0x559db2084db0 2024-03-31T23:18:34.618 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:34.612+0000 7fbac643a740 10 client.0 _ll_register_callbacks cb 0x7fbab80bb0b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T23:18:34.618 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:34.612+0000 7fbac15f1640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559db207b6c0 0x559db208d090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:18:34.618 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:34.612+0000 7fbac0df0640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559db207bfe0 0x559db207e6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:18:34.618 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:34.612+0000 7fbac15f1640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559db207b6c0 0x559db208d090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:53404/0 (socket says 172.21.15.46:53404) 2024-03-31T23:18:34.618 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:34.612+0000 7fbac15f1640 1 -- 172.21.15.46:0/2234702844 learned_addr learned my addr 172.21.15.46:0/2234702844 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:18:34.619 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:34.612+0000 7fbac1df2640 1 --2- 172.21.15.46:0/2234702844 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559db2084db0 0x559db207ec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:18:34.619 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:34.612+0000 7fbac15f1640 1 -- 172.21.15.46:0/2234702844 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559db2084db0 msgr2=0x559db207ec50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:18:34.619 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:34.612+0000 7fbac15f1640 1 --2- 172.21.15.46:0/2234702844 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559db2084db0 0x559db207ec50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:18:34.619 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:34.612+0000 7fbac15f1640 1 -- 172.21.15.46:0/2234702844 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559db207bfe0 msgr2=0x559db207e6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:18:34.619 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:34.612+0000 7fbac15f1640 1 --2- 172.21.15.46:0/2234702844 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559db207bfe0 0x559db207e6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:18:34.619 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:34.612+0000 7fbac15f1640 1 -- 172.21.15.46:0/2234702844 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559db207f480 con 0x559db207b6c0 2024-03-31T23:18:34.619 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:34.612+0000 7fbac15f1640 1 --2- 172.21.15.46:0/2234702844 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559db207b6c0 0x559db208d090 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7fbab809b130 tx=0x7fbab8005400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=72a7833d5c962621 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:18:34.619 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:34.612+0000 7fbaa4ff9640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-31T23:18:34.619 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:34.613+0000 7fbaa4ff9640 1 -- 172.21.15.46:0/2234702844 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbab809ace0 con 0x559db207b6c0 2024-03-31T23:18:34.619 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:34.613+0000 7fbaa4ff9640 1 -- 172.21.15.46:0/2234702844 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9554+0+0 (secure 0 0 0) 0x7fbab80c5d40 con 0x559db207b6c0 2024-03-31T23:18:34.620 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:34.613+0000 7fbac643a740 1 -- 172.21.15.46:0/2234702844 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbab4004cc0 con 0x559db207b6c0 2024-03-31T23:18:34.620 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:34.613+0000 7fbaa4ff9640 1 -- 172.21.15.46:0/2234702844 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fbab80cc600 con 0x559db207b6c0 2024-03-31T23:18:34.620 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:34.613+0000 7fbac643a740 10 client.16713 fetch_fsmap learned FSMap version 30 2024-03-31T23:18:34.620 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:34.613+0000 7fbac643a740 1 -- 172.21.15.46:0/2234702844 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x559db20fb060 con 0x559db207b6c0 2024-03-31T23:18:34.620 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:34.613+0000 7fbaa4ff9640 1 -- 172.21.15.46:0/2234702844 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fbab4004cc0 con 0x559db207b6c0 2024-03-31T23:18:34.620 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:34.613+0000 7fbaa4ff9640 1 -- 172.21.15.46:0/2234702844 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 30) ==== 8346+0+0 (secure 0 0 0) 0x7fbab8096ef0 con 0x559db207b6c0 2024-03-31T23:18:34.620 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:34.613+0000 7fbac643a740 10 client.16713 fetch_fsmap finished waiting for FSMap version 30 2024-03-31T23:18:34.620 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:34.613+0000 7fbac643a740 10 client.16713 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-31T23:18:34.620 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:34.613+0000 7fbac643a740 20 client.16713 populate_metadata read hostname 'smithi046' 2024-03-31T23:18:34.620 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:34.613+0000 7fbac643a740 1 --2- 172.21.15.46:0/2234702844 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x559db20fb5f0 0x559db20fd9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:18:34.621 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:34.613+0000 7fbac643a740 4 client.16713 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-31T23:18:34.621 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:34.613+0000 7fbac643a740 1 -- 172.21.15.46:0/2234702844 --> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] -- command(tid 0: {"prefix": "flush journal"}) -- 0x559db1febe30 con 0x559db20fb5f0 2024-03-31T23:18:34.621 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:34.613+0000 7fbac0df0640 1 --2- 172.21.15.46:0/2234702844 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x559db20fb5f0 0x559db20fd9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:18:34.621 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:34.614+0000 7fbaa4ff9640 10 client.16713 ms_handle_connect on v2:172.21.15.142:6838/2519143918 2024-03-31T23:18:34.621 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:34.614+0000 7fbac0df0640 1 --2- 172.21.15.46:0/2234702844 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x559db20fb5f0 0x559db20fd9e0 crc :-1 s=READY pgs=136 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=382492edf1c2ffaf server_cookie=2f236128baf282db in_seq=0 out_seq=0 2024-03-31T23:18:34.637 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:34.635+0000 7fbaa4ff9640 1 -- 172.21.15.46:0/2234702844 <== mds.0 v2:172.21.15.142:6838/2519143918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x559db1febe30 con 0x559db20fb5f0 2024-03-31T23:18:34.637 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:34.635+0000 7fbaa4ff9640 10 client.16713 handle_command_reply: tid=0 2024-03-31T23:18:34.637 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:34.635+0000 7fbac643a740 1 client.16713 shutdown 2024-03-31T23:18:34.637 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:34.635+0000 7fbac643a740 1 -- 172.21.15.46:0/2234702844 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559db207b6c0 msgr2=0x559db208d090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:18:34.637 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:34.635+0000 7fbac643a740 1 --2- 172.21.15.46:0/2234702844 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559db207b6c0 0x559db208d090 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7fbab809b130 tx=0x7fbab8005400 comp rx=0 tx=0).stop 2024-03-31T23:18:34.637 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:34.635+0000 7fbac643a740 1 -- 172.21.15.46:0/2234702844 shutdown_connections 2024-03-31T23:18:34.637 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:34.635+0000 7fbac643a740 1 --2- 172.21.15.46:0/2234702844 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x559db20fb5f0 0x559db20fd9e0 crc :-1 s=READY pgs=136 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:18:34.637 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:34.635+0000 7fbac643a740 1 --2- 172.21.15.46:0/2234702844 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559db2084db0 0x559db207ec50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:18:34.637 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:34.635+0000 7fbac643a740 1 --2- 172.21.15.46:0/2234702844 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559db207bfe0 0x559db207e6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:18:34.638 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:34.635+0000 7fbaa4ff9640 0 client.16713 ms_handle_reset on v2:172.21.15.142:6838/2519143918 2024-03-31T23:18:34.638 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:34.635+0000 7fbac643a740 1 --2- 172.21.15.46:0/2234702844 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559db207b6c0 0x559db208d090 unknown :-1 s=CLOSED pgs=743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:18:34.638 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:34.635+0000 7fbac643a740 1 -- 172.21.15.46:0/2234702844 >> 172.21.15.46:0/2234702844 conn(0x559db2075fe0 msgr2=0x559db20746f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:18:34.638 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:34.635+0000 7fbac643a740 1 -- 172.21.15.46:0/2234702844 shutdown_connections 2024-03-31T23:18:34.638 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:34.635+0000 7fbac643a740 1 -- 172.21.15.46:0/2234702844 wait complete. 2024-03-31T23:18:34.638 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:34.635+0000 7fbac643a740 20 client.16713 trim_cache size 0 max 16384 2024-03-31T23:18:34.639 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout:{ 2024-03-31T23:18:34.639 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout: "message": "", 2024-03-31T23:18:34.639 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout: "return_code": 0 2024-03-31T23:18:34.639 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout:} 2024-03-31T23:18:34.640 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:34.636+0000 7fbac3855640 1 -- 172.21.15.46:0/1391910489 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fbaa80a26b0 msgr2=0x7fbaa80a4b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:18:34.640 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:34.636+0000 7fbac3855640 1 --2- 172.21.15.46:0/1391910489 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fbaa80a26b0 0x7fbaa80a4b70 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fbab4009fd0 tx=0x7fbab4009500 comp rx=0 tx=0).stop 2024-03-31T23:18:34.640 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:34.636+0000 7fbac3855640 1 -- 172.21.15.46:0/1391910489 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbabc1008a0 msgr2=0x7fbabc1022c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:18:34.640 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:34.636+0000 7fbac3855640 1 --2- 172.21.15.46:0/1391910489 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbabc1008a0 0x7fbabc1022c0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fbab800e9f0 tx=0x7fbab800eeb0 comp rx=0 tx=0).stop 2024-03-31T23:18:34.640 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:34.636+0000 7fbac3855640 1 -- 172.21.15.46:0/1391910489 shutdown_connections 2024-03-31T23:18:34.641 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:34.636+0000 7fbac3855640 1 --2- 172.21.15.46:0/1391910489 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fbaa80a26b0 0x7fbaa80a4b70 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:18:34.641 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:34.636+0000 7fbac3855640 1 --2- 172.21.15.46:0/1391910489 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbabc1055f0 0x7fbabc102d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:18:34.641 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:34.636+0000 7fbac3855640 1 --2- 172.21.15.46:0/1391910489 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbabc101260 0x7fbabc102800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:18:34.641 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:34.636+0000 7fbac3855640 1 --2- 172.21.15.46:0/1391910489 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbabc1008a0 0x7fbabc1022c0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:18:34.641 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:34.636+0000 7fbac3855640 1 -- 172.21.15.46:0/1391910489 >> 172.21.15.46:0/1391910489 conn(0x7fbabc0774a0 msgr2=0x7fbabc069170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:18:34.641 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:34.636+0000 7fbac3855640 1 -- 172.21.15.46:0/1391910489 shutdown_connections 2024-03-31T23:18:34.641 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:34.636+0000 7fbac3855640 1 -- 172.21.15.46:0/1391910489 wait complete. 2024-03-31T23:18:34.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:18:34 smithi162 ceph-mon[36237]: pgmap v210: 129 pgs: 129 active+clean; 100 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 437 KiB/s rd, 1.3 MiB/s wr, 116 op/s 2024-03-31T23:18:34.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:18:34 smithi046 ceph-mon[32498]: pgmap v210: 129 pgs: 129 active+clean; 100 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 437 KiB/s rd, 1.3 MiB/s wr, 116 op/s 2024-03-31T23:18:34.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:18:34 smithi142 ceph-mon[35271]: pgmap v210: 129 pgs: 129 active+clean; 100 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 437 KiB/s rd, 1.3 MiB/s wr, 116 op/s 2024-03-31T23:18:35.201 INFO:tasks.workunit.client.0.smithi046.stdout: 1 156039 2.50 MB/sec execute 221 sec latency 22.026 ms 2024-03-31T23:18:36.201 INFO:tasks.workunit.client.0.smithi046.stdout: 1 156402 2.49 MB/sec execute 222 sec latency 15.654 ms 2024-03-31T23:18:36.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:18:36 smithi162 ceph-mon[36237]: pgmap v211: 129 pgs: 129 active+clean; 101 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 653 KiB/s rd, 1.8 MiB/s wr, 172 op/s 2024-03-31T23:18:36.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:18:36 smithi046 ceph-mon[32498]: pgmap v211: 129 pgs: 129 active+clean; 101 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 653 KiB/s rd, 1.8 MiB/s wr, 172 op/s 2024-03-31T23:18:36.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:18:36 smithi142 ceph-mon[35271]: pgmap v211: 129 pgs: 129 active+clean; 101 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 653 KiB/s rd, 1.8 MiB/s wr, 172 op/s 2024-03-31T23:18:37.201 INFO:tasks.workunit.client.0.smithi046.stdout: 1 156669 2.48 MB/sec execute 223 sec latency 15.450 ms 2024-03-31T23:18:38.201 INFO:tasks.workunit.client.0.smithi046.stdout: 1 157129 2.48 MB/sec execute 224 sec latency 20.114 ms 2024-03-31T23:18:38.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:18:38 smithi162 ceph-mon[36237]: pgmap v212: 129 pgs: 129 active+clean; 101 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 414 KiB/s rd, 1.2 MiB/s wr, 112 op/s 2024-03-31T23:18:38.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:18:38 smithi046 ceph-mon[32498]: pgmap v212: 129 pgs: 129 active+clean; 101 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 414 KiB/s rd, 1.2 MiB/s wr, 112 op/s 2024-03-31T23:18:38.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:18:38 smithi142 ceph-mon[35271]: pgmap v212: 129 pgs: 129 active+clean; 101 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 414 KiB/s rd, 1.2 MiB/s wr, 112 op/s 2024-03-31T23:18:39.201 INFO:tasks.workunit.client.0.smithi046.stdout: 1 157601 2.48 MB/sec execute 225 sec latency 21.565 ms 2024-03-31T23:18:40.201 INFO:tasks.workunit.client.0.smithi046.stdout: 1 158129 2.48 MB/sec execute 226 sec latency 23.022 ms 2024-03-31T23:18:40.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:18:40 smithi162 ceph-mon[36237]: pgmap v213: 129 pgs: 129 active+clean; 102 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 661 KiB/s rd, 1.9 MiB/s wr, 174 op/s 2024-03-31T23:18:40.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:18:40 smithi046 ceph-mon[32498]: pgmap v213: 129 pgs: 129 active+clean; 102 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 661 KiB/s rd, 1.9 MiB/s wr, 174 op/s 2024-03-31T23:18:40.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:18:40 smithi142 ceph-mon[35271]: pgmap v213: 129 pgs: 129 active+clean; 102 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 661 KiB/s rd, 1.9 MiB/s wr, 174 op/s 2024-03-31T23:18:41.201 INFO:tasks.workunit.client.0.smithi046.stdout: 1 158589 2.47 MB/sec execute 227 sec latency 20.310 ms 2024-03-31T23:18:42.201 INFO:tasks.workunit.client.0.smithi046.stdout: 1 159371 2.49 MB/sec execute 228 sec latency 16.777 ms 2024-03-31T23:18:42.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:18:42 smithi162 ceph-mon[36237]: pgmap v214: 129 pgs: 129 active+clean; 102 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 463 KiB/s rd, 1.2 MiB/s wr, 117 op/s 2024-03-31T23:18:42.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:18:42 smithi046 ceph-mon[32498]: pgmap v214: 129 pgs: 129 active+clean; 102 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 463 KiB/s rd, 1.2 MiB/s wr, 117 op/s 2024-03-31T23:18:42.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:18:42 smithi142 ceph-mon[35271]: pgmap v214: 129 pgs: 129 active+clean; 102 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 463 KiB/s rd, 1.2 MiB/s wr, 117 op/s 2024-03-31T23:18:43.201 INFO:tasks.workunit.client.0.smithi046.stdout: 1 159671 2.49 MB/sec execute 229 sec latency 18.538 ms 2024-03-31T23:18:44.201 INFO:tasks.workunit.client.0.smithi046.stdout: 1 160018 2.49 MB/sec execute 230 sec latency 16.428 ms 2024-03-31T23:18:44.795 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:18:44 smithi162 ceph-mon[36237]: pgmap v215: 129 pgs: 129 active+clean; 102 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 463 KiB/s rd, 1.2 MiB/s wr, 117 op/s 2024-03-31T23:18:44.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:18:44 smithi046 ceph-mon[32498]: pgmap v215: 129 pgs: 129 active+clean; 102 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 463 KiB/s rd, 1.2 MiB/s wr, 117 op/s 2024-03-31T23:18:44.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:18:44 smithi142 ceph-mon[35271]: pgmap v215: 129 pgs: 129 active+clean; 102 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 463 KiB/s rd, 1.2 MiB/s wr, 117 op/s 2024-03-31T23:18:45.201 INFO:tasks.workunit.client.0.smithi046.stdout: 1 160323 2.48 MB/sec execute 231 sec latency 18.703 ms 2024-03-31T23:18:46.201 INFO:tasks.workunit.client.0.smithi046.stdout: 1 160796 2.48 MB/sec execute 232 sec latency 20.686 ms 2024-03-31T23:18:46.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:18:46 smithi162 ceph-mon[36237]: pgmap v216: 129 pgs: 129 active+clean; 104 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 648 KiB/s rd, 1.9 MiB/s wr, 170 op/s 2024-03-31T23:18:46.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:18:46 smithi046 ceph-mon[32498]: pgmap v216: 129 pgs: 129 active+clean; 104 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 648 KiB/s rd, 1.9 MiB/s wr, 170 op/s 2024-03-31T23:18:46.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:18:46 smithi142 ceph-mon[35271]: pgmap v216: 129 pgs: 129 active+clean; 104 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 648 KiB/s rd, 1.9 MiB/s wr, 170 op/s 2024-03-31T23:18:47.201 INFO:tasks.workunit.client.0.smithi046.stdout: 1 161281 2.48 MB/sec execute 233 sec latency 31.541 ms 2024-03-31T23:18:47.733 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:47.732+0000 7f59f6516640 1 -- 172.21.15.46:0/1129736944 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f59f01016f0 msgr2=0x7f59f0101b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:18:47.733 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:47.732+0000 7f59f6516640 1 --2- 172.21.15.46:0/1129736944 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f59f01016f0 0x7f59f0101b70 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7f59e4009930 tx=0x7f59e402bc30 comp rx=0 tx=0).stop 2024-03-31T23:18:47.733 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:47.732+0000 7f59f6516640 1 -- 172.21.15.46:0/1129736944 shutdown_connections 2024-03-31T23:18:47.733 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:47.732+0000 7f59f6516640 1 --2- 172.21.15.46:0/1129736944 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f59f01020b0 0x7f59f010e5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:18:47.734 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:47.732+0000 7f59f6516640 1 --2- 172.21.15.46:0/1129736944 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f59f01016f0 0x7f59f0101b70 unknown :-1 s=CLOSED pgs=744 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:18:47.734 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:47.732+0000 7f59f6516640 1 --2- 172.21.15.46:0/1129736944 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f59f0107700 0x7f59f0107ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:18:47.734 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:47.732+0000 7f59f6516640 1 -- 172.21.15.46:0/1129736944 >> 172.21.15.46:0/1129736944 conn(0x7f59f00fd1e0 msgr2=0x7f59f00ff600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:18:47.734 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:47.732+0000 7f59f6516640 1 -- 172.21.15.46:0/1129736944 shutdown_connections 2024-03-31T23:18:47.734 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:47.732+0000 7f59f6516640 1 -- 172.21.15.46:0/1129736944 wait complete. 2024-03-31T23:18:47.736 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:47.733+0000 7f59f6516640 1 Processor -- start 2024-03-31T23:18:47.736 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:47.733+0000 7f59f6516640 1 -- start start 2024-03-31T23:18:47.736 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:47.733+0000 7f59f6516640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f59f01016f0 0x7f59f010dac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:18:47.736 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:47.733+0000 7f59f6516640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f59f01020b0 0x7f59f010e000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:18:47.736 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:47.733+0000 7f59f6516640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f59f0107700 0x7f59f01a2d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:18:47.737 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:47.733+0000 7f59f6516640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f59f01119c0 con 0x7f59f01020b0 2024-03-31T23:18:47.737 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:47.733+0000 7f59f6516640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f59f0111840 con 0x7f59f01016f0 2024-03-31T23:18:47.737 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:47.733+0000 7f59f6516640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f59f0111b40 con 0x7f59f0107700 2024-03-31T23:18:47.737 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:47.734+0000 7f59ef7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f59f01020b0 0x7f59f010e000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:18:47.737 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:47.734+0000 7f59ef7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f59f01020b0 0x7f59f010e000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59634/0 (socket says 172.21.15.46:59634) 2024-03-31T23:18:47.737 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:47.734+0000 7f59ef7fe640 1 -- 172.21.15.46:0/4140296651 learned_addr learned my addr 172.21.15.46:0/4140296651 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:18:47.737 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:47.734+0000 7f59effff640 1 --2- 172.21.15.46:0/4140296651 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f59f01016f0 0x7f59f010dac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:18:47.737 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:47.734+0000 7f59f4ab3640 1 --2- 172.21.15.46:0/4140296651 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f59f0107700 0x7f59f01a2d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:18:47.737 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:47.734+0000 7f59ef7fe640 1 -- 172.21.15.46:0/4140296651 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f59f0107700 msgr2=0x7f59f01a2d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:18:47.737 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:47.734+0000 7f59ef7fe640 1 --2- 172.21.15.46:0/4140296651 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f59f0107700 0x7f59f01a2d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:18:47.737 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:47.734+0000 7f59ef7fe640 1 -- 172.21.15.46:0/4140296651 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f59f01016f0 msgr2=0x7f59f010dac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:18:47.737 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:47.734+0000 7f59ef7fe640 1 --2- 172.21.15.46:0/4140296651 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f59f01016f0 0x7f59f010dac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:18:47.737 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:47.734+0000 7f59ef7fe640 1 -- 172.21.15.46:0/4140296651 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59f01a34b0 con 0x7f59f01020b0 2024-03-31T23:18:47.737 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:47.734+0000 7f59effff640 1 --2- 172.21.15.46:0/4140296651 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f59f01016f0 0x7f59f010dac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:18:47.738 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:47.734+0000 7f59ef7fe640 1 --2- 172.21.15.46:0/4140296651 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f59f01020b0 0x7f59f010e000 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7f59e4009230 tx=0x7f59e402c5d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:18:47.738 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:47.734+0000 7f59ed7fa640 1 -- 172.21.15.46:0/4140296651 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59e4043020 con 0x7f59f01020b0 2024-03-31T23:18:47.738 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:47.735+0000 7f59ed7fa640 1 -- 172.21.15.46:0/4140296651 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f59e402ca70 con 0x7f59f01020b0 2024-03-31T23:18:47.738 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:47.735+0000 7f59f6516640 1 -- 172.21.15.46:0/4140296651 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f59f01a3740 con 0x7f59f01020b0 2024-03-31T23:18:47.738 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:47.735+0000 7f59ed7fa640 1 -- 172.21.15.46:0/4140296651 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59e4003a50 con 0x7f59f01020b0 2024-03-31T23:18:47.738 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:47.735+0000 7f59f6516640 1 -- 172.21.15.46:0/4140296651 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f59f01ab030 con 0x7f59f01020b0 2024-03-31T23:18:47.740 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:47.737+0000 7f59ed7fa640 1 -- 172.21.15.46:0/4140296651 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 25) ==== 137451+0+0 (secure 0 0 0) 0x7f59e4046020 con 0x7f59f01020b0 2024-03-31T23:18:47.741 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:47.738+0000 7f59ed7fa640 1 --2- 172.21.15.46:0/4140296651 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f59d80a26b0 0x7f59d80a4b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:18:47.741 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:47.738+0000 7f59ed7fa640 1 -- 172.21.15.46:0/4140296651 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9554+0+0 (secure 0 0 0) 0x7f59e40f0e00 con 0x7f59f01020b0 2024-03-31T23:18:47.741 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:47.738+0000 7f59effff640 1 --2- 172.21.15.46:0/4140296651 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f59d80a26b0 0x7f59d80a4b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:18:47.741 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:47.738+0000 7f59f91cc740 1 Processor -- start 2024-03-31T23:18:47.741 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:47.738+0000 7f59f91cc740 1 -- start start 2024-03-31T23:18:47.741 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:47.738+0000 7f59f91cc740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x556feebc86c0 0x556feebc8aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:18:47.741 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:47.738+0000 7f59f91cc740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556feebc8fe0 0x556feebd1870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:18:47.741 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:47.738+0000 7f59f91cc740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556feebd1db0 0x556feebd41a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:18:47.741 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:47.738+0000 7f59f91cc740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x556feeb2b8f0 con 0x556feebd1db0 2024-03-31T23:18:47.741 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:47.738+0000 7f59f91cc740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x556feead48d0 con 0x556feebc86c0 2024-03-31T23:18:47.742 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:47.738+0000 7f59f91cc740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x556feeb27990 con 0x556feebc8fe0 2024-03-31T23:18:47.742 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:47.738+0000 7f59effff640 1 --2- 172.21.15.46:0/4140296651 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f59d80a26b0 0x7f59d80a4b70 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f59e0009f40 tx=0x7f59e0005f90 comp rx=0 tx=0).ready entity=mgr.15654 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:18:47.742 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:47.738+0000 7f59effff640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x556feebc86c0 0x556feebc8aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:18:47.742 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:47.738+0000 7f59f4ab3640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556feebd1db0 0x556feebd41a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:18:47.742 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:47.738+0000 7f59effff640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x556feebc86c0 0x556feebc8aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.46:42736/0 (socket says 172.21.15.46:42736) 2024-03-31T23:18:47.742 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:47.739+0000 7f59effff640 1 -- 172.21.15.46:0/737619029 learned_addr learned my addr 172.21.15.46:0/737619029 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:18:47.742 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:47.739+0000 7f59ef7fe640 1 --2- 172.21.15.46:0/737619029 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556feebc8fe0 0x556feebd1870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:18:47.742 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:47.739+0000 7f59effff640 1 -- 172.21.15.46:0/737619029 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556feebc8fe0 msgr2=0x556feebd1870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:18:47.742 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:47.739+0000 7f59effff640 1 --2- 172.21.15.46:0/737619029 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556feebc8fe0 0x556feebd1870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:18:47.742 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:47.739+0000 7f59effff640 1 -- 172.21.15.46:0/737619029 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556feebd1db0 msgr2=0x556feebd41a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:18:47.742 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:47.739+0000 7f59effff640 1 --2- 172.21.15.46:0/737619029 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556feebd1db0 0x556feebd41a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:18:47.742 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:47.739+0000 7f59effff640 1 -- 172.21.15.46:0/737619029 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556fee840210 con 0x556feebc86c0 2024-03-31T23:18:47.742 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:47.739+0000 7f59effff640 1 --2- 172.21.15.46:0/737619029 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x556feebc86c0 0x556feebc8aa0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f59e000f640 tx=0x7f59e0014000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=67a043c27701d043 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:18:47.742 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:47.739+0000 7f59d67fc640 1 -- 172.21.15.46:0/737619029 <== mon.2 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59e0026430 con 0x556feebc86c0 2024-03-31T23:18:47.743 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:47.739+0000 7f59d67fc640 1 -- 172.21.15.46:0/737619029 <== mon.2 v2:172.21.15.142:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f59e001d8c0 con 0x556feebc86c0 2024-03-31T23:18:47.743 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:47.739+0000 7f59d67fc640 1 -- 172.21.15.46:0/737619029 <== mon.2 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59e0025830 con 0x556feebc86c0 2024-03-31T23:18:47.747 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:47.740+0000 7f59f91cc740 1 -- 172.21.15.46:0/737619029 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x556feebc86c0 msgr2=0x556feebc8aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:18:47.747 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:47.740+0000 7f59f91cc740 1 --2- 172.21.15.46:0/737619029 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x556feebc86c0 0x556feebc8aa0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f59e000f640 tx=0x7f59e0014000 comp rx=0 tx=0).stop 2024-03-31T23:18:47.747 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:47.740+0000 7f59f91cc740 1 -- 172.21.15.46:0/737619029 shutdown_connections 2024-03-31T23:18:47.747 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:47.740+0000 7f59f91cc740 1 --2- 172.21.15.46:0/737619029 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556feebd1db0 0x556feebd41a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:18:47.747 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:47.740+0000 7f59f91cc740 1 --2- 172.21.15.46:0/737619029 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556feebc8fe0 0x556feebd1870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:18:47.747 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:47.740+0000 7f59f91cc740 1 --2- 172.21.15.46:0/737619029 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x556feebc86c0 0x556feebc8aa0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:18:47.747 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:47.740+0000 7f59f91cc740 1 -- 172.21.15.46:0/737619029 >> 172.21.15.46:0/737619029 conn(0x556feebc2fe0 msgr2=0x556feebc33c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:18:47.747 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:47.740+0000 7f59f91cc740 1 -- 172.21.15.46:0/737619029 shutdown_connections 2024-03-31T23:18:47.747 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:47.740+0000 7f59f91cc740 1 -- 172.21.15.46:0/737619029 wait complete. 2024-03-31T23:18:47.747 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:47.740+0000 7f59f91cc740 1 -- start start 2024-03-31T23:18:47.747 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:47.741+0000 7f59f91cc740 1 Processor -- start 2024-03-31T23:18:47.748 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:47.741+0000 7f59f91cc740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x556feebc86c0 0x556feebcee10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:18:47.748 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:47.741+0000 7f59f91cc740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556feebc8fe0 0x556feebcf350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:18:47.748 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:47.741+0000 7f59f91cc740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556feebd1db0 0x556feebcf890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:18:47.748 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:47.741+0000 7f59f91cc740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x556feebd49e0 con 0x556feebd1db0 2024-03-31T23:18:47.748 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:47.741+0000 7f59f91cc740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x556feeac9f80 con 0x556feebc86c0 2024-03-31T23:18:47.748 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:47.741+0000 7f59f91cc740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x556feebd4b60 con 0x556feebc8fe0 2024-03-31T23:18:47.748 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:47.741+0000 7f59f91cc740 10 client.0 _ll_register_callbacks cb 0x556feeb22290 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T23:18:47.748 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:47.741+0000 7f59f4ab3640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556feebd1db0 0x556feebcf890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:18:47.748 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:47.741+0000 7f59f4ab3640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556feebd1db0 0x556feebcf890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59650/0 (socket says 172.21.15.46:59650) 2024-03-31T23:18:47.748 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:47.741+0000 7f59ef7fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556feebc8fe0 0x556feebcf350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:18:47.748 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:47.741+0000 7f59f4ab3640 1 -- 172.21.15.46:0/3288013565 learned_addr learned my addr 172.21.15.46:0/3288013565 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:18:47.748 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:47.741+0000 7f59effff640 1 --2- 172.21.15.46:0/3288013565 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x556feebc86c0 0x556feebcee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:18:47.748 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:47.741+0000 7f59ef7fe640 1 -- 172.21.15.46:0/3288013565 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x556feebc86c0 msgr2=0x556feebcee10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:18:47.748 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:47.741+0000 7f59ef7fe640 1 --2- 172.21.15.46:0/3288013565 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x556feebc86c0 0x556feebcee10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:18:47.749 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:47.741+0000 7f59ef7fe640 1 -- 172.21.15.46:0/3288013565 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556feebd1db0 msgr2=0x556feebcf890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:18:47.749 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:47.741+0000 7f59ef7fe640 1 --2- 172.21.15.46:0/3288013565 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556feebd1db0 0x556feebcf890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:18:47.749 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:47.741+0000 7f59ef7fe640 1 -- 172.21.15.46:0/3288013565 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556fee90c620 con 0x556feebc8fe0 2024-03-31T23:18:47.749 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:47.741+0000 7f59f4ab3640 1 --2- 172.21.15.46:0/3288013565 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556feebd1db0 0x556feebcf890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:18:47.749 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:47.742+0000 7f59ef7fe640 1 --2- 172.21.15.46:0/3288013565 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556feebc8fe0 0x556feebcf350 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f59e40ee220 tx=0x7f59e4009260 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ae2324b551f43cc7 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:18:47.749 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:47.742+0000 7f59cf7fe640 10 client.0 ms_handle_connect on v2:172.21.15.162:3300/0 2024-03-31T23:18:47.749 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:47.742+0000 7f59cf7fe640 1 -- 172.21.15.46:0/3288013565 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59e40c5c90 con 0x556feebc8fe0 2024-03-31T23:18:47.749 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:47.742+0000 7f59f91cc740 1 -- 172.21.15.46:0/3288013565 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f59e001d8c0 con 0x556feebc8fe0 2024-03-31T23:18:47.749 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:47.742+0000 7f59cf7fe640 1 -- 172.21.15.46:0/3288013565 <== mon.1 v2:172.21.15.162:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9554+0+0 (secure 0 0 0) 0x7f59e40bb320 con 0x556feebc8fe0 2024-03-31T23:18:47.749 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:47.742+0000 7f59cf7fe640 1 -- 172.21.15.46:0/3288013565 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f59e40d6020 con 0x556feebc8fe0 2024-03-31T23:18:47.749 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:47.742+0000 7f59f91cc740 10 client.26458 fetch_fsmap learned FSMap version 30 2024-03-31T23:18:47.749 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:47.742+0000 7f59f91cc740 1 -- 172.21.15.46:0/3288013565 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x556feebd4a90 con 0x556feebc8fe0 2024-03-31T23:18:47.749 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:47.742+0000 7f59cf7fe640 1 -- 172.21.15.46:0/3288013565 <== mon.1 v2:172.21.15.162:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f59e001d8c0 con 0x556feebc8fe0 2024-03-31T23:18:47.749 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:47.742+0000 7f59cf7fe640 1 -- 172.21.15.46:0/3288013565 <== mon.1 v2:172.21.15.162:3300/0 5 ==== fsmap(e 30) ==== 8346+0+0 (secure 0 0 0) 0x7f59e40bd570 con 0x556feebc8fe0 2024-03-31T23:18:47.750 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:47.742+0000 7f59f91cc740 10 client.26458 fetch_fsmap finished waiting for FSMap version 30 2024-03-31T23:18:47.750 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:47.742+0000 7f59f91cc740 10 client.26458 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-31T23:18:47.750 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:47.742+0000 7f59f91cc740 20 client.26458 populate_metadata read hostname 'smithi046' 2024-03-31T23:18:47.750 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:47.742+0000 7f59f91cc740 1 --2- 172.21.15.46:0/3288013565 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x556feebe1dc0 0x556feebe41d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:18:47.750 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:47.742+0000 7f59f91cc740 4 client.26458 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-31T23:18:47.750 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:47.742+0000 7f59f91cc740 1 -- 172.21.15.46:0/3288013565 --> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556feeb3f120 con 0x556feebe1dc0 2024-03-31T23:18:47.750 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:47.743+0000 7f59effff640 1 --2- 172.21.15.46:0/3288013565 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x556feebe1dc0 0x556feebe41d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:18:47.750 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:47.743+0000 7f59effff640 1 --2- 172.21.15.46:0/3288013565 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x556feebe1dc0 0x556feebe41d0 crc :-1 s=READY pgs=137 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6f744ce8bafd7fd server_cookie=727a9c4d556e9493 in_seq=0 out_seq=0 2024-03-31T23:18:47.750 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:47.743+0000 7f59cf7fe640 10 client.26458 ms_handle_connect on v2:172.21.15.142:6838/2519143918 2024-03-31T23:18:47.750 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:47.744+0000 7f59cf7fe640 1 -- 172.21.15.46:0/3288013565 <== mds.0 v2:172.21.15.142:6838/2519143918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x556feeb3f120 con 0x556feebe1dc0 2024-03-31T23:18:47.750 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:47.744+0000 7f59cf7fe640 10 client.26458 handle_command_reply: tid=0 2024-03-31T23:18:47.750 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:47.744+0000 7f59f91cc740 1 client.26458 shutdown 2024-03-31T23:18:47.751 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:47.744+0000 7f59f91cc740 1 -- 172.21.15.46:0/3288013565 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556feebc8fe0 msgr2=0x556feebcf350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:18:47.751 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:47.744+0000 7f59f91cc740 1 --2- 172.21.15.46:0/3288013565 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556feebc8fe0 0x556feebcf350 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f59e40ee220 tx=0x7f59e4009260 comp rx=0 tx=0).stop 2024-03-31T23:18:47.751 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:47.744+0000 7f59f91cc740 1 -- 172.21.15.46:0/3288013565 shutdown_connections 2024-03-31T23:18:47.751 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:47.744+0000 7f59f91cc740 1 --2- 172.21.15.46:0/3288013565 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x556feebe1dc0 0x556feebe41d0 crc :-1 s=READY pgs=137 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:18:47.751 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:47.744+0000 7f59f91cc740 1 --2- 172.21.15.46:0/3288013565 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556feebd1db0 0x556feebcf890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:18:47.751 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:47.744+0000 7f59f91cc740 1 --2- 172.21.15.46:0/3288013565 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556feebc8fe0 0x556feebcf350 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:18:47.751 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:47.744+0000 7f59cf7fe640 0 client.26458 ms_handle_reset on v2:172.21.15.142:6838/2519143918 2024-03-31T23:18:47.751 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:47.744+0000 7f59f91cc740 1 --2- 172.21.15.46:0/3288013565 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x556feebc86c0 0x556feebcee10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:18:47.751 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:47.744+0000 7f59f91cc740 1 -- 172.21.15.46:0/3288013565 >> 172.21.15.46:0/3288013565 conn(0x556feebc2fe0 msgr2=0x556feebc4d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:18:47.751 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:47.744+0000 7f59f91cc740 1 -- 172.21.15.46:0/3288013565 shutdown_connections 2024-03-31T23:18:47.751 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:47.744+0000 7f59f91cc740 1 -- 172.21.15.46:0/3288013565 wait complete. 2024-03-31T23:18:47.751 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:47.744+0000 7f59f91cc740 20 client.26458 trim_cache size 0 max 16384 2024-03-31T23:18:47.765 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:47.764+0000 7f59f91cc740 1 Processor -- start 2024-03-31T23:18:47.766 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:47.764+0000 7f59f91cc740 1 -- start start 2024-03-31T23:18:47.766 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:47.764+0000 7f59f91cc740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556feebc86c0 0x556feebd1540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:18:47.766 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:47.764+0000 7f59f91cc740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556feebc8fe0 0x556feebdd780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:18:47.766 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:47.764+0000 7f59f91cc740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x556feebd1db0 0x556feebe9ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:18:47.766 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:47.765+0000 7f59ef7fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556feebc8fe0 0x556feebdd780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:18:47.767 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:47.765+0000 7f59f91cc740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f59f01b7b90 con 0x556feebc86c0 2024-03-31T23:18:47.767 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:47.765+0000 7f59f91cc740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f59f01b5460 con 0x556feebd1db0 2024-03-31T23:18:47.767 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:47.765+0000 7f59f91cc740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f59f01b32d0 con 0x556feebc8fe0 2024-03-31T23:18:47.767 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:47.765+0000 7f59ef7fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556feebc8fe0 0x556feebdd780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.46:40960/0 (socket says 172.21.15.46:40960) 2024-03-31T23:18:47.767 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:47.765+0000 7f59ef7fe640 1 -- 172.21.15.46:0/2354073646 learned_addr learned my addr 172.21.15.46:0/2354073646 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:18:47.767 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:47.765+0000 7f59f4ab3640 1 --2- 172.21.15.46:0/2354073646 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x556feebd1db0 0x556feebe9ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:18:47.767 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:47.766+0000 7f59ef7fe640 1 -- 172.21.15.46:0/2354073646 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x556feebd1db0 msgr2=0x556feebe9ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:18:47.767 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:47.766+0000 7f59ef7fe640 1 --2- 172.21.15.46:0/2354073646 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x556feebd1db0 0x556feebe9ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:18:47.767 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:47.766+0000 7f59ef7fe640 1 -- 172.21.15.46:0/2354073646 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556feebc86c0 msgr2=0x556feebd1540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:18:47.767 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:47.766+0000 7f59ef7fe640 1 --2- 172.21.15.46:0/2354073646 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556feebc86c0 0x556feebd1540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:18:47.767 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:47.766+0000 7f59ef7fe640 1 -- 172.21.15.46:0/2354073646 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556feebde410 con 0x556feebc8fe0 2024-03-31T23:18:47.768 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:47.766+0000 7f59ef7fe640 1 --2- 172.21.15.46:0/2354073646 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556feebc8fe0 0x556feebdd780 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f59e4005ec0 tx=0x7f59e40c1550 comp rx=0 tx=0).ready entity=mon.1 client_cookie=244faab1436092b5 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:18:47.768 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:47.767+0000 7f59d5ffb640 1 -- 172.21.15.46:0/2354073646 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59e409f020 con 0x556feebc8fe0 2024-03-31T23:18:47.768 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:47.767+0000 7f59d5ffb640 1 -- 172.21.15.46:0/2354073646 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f59e40bd280 con 0x556feebc8fe0 2024-03-31T23:18:47.768 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:47.767+0000 7f59d5ffb640 1 -- 172.21.15.46:0/2354073646 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59e40053d0 con 0x556feebc8fe0 2024-03-31T23:18:47.769 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:47.768+0000 7f59f91cc740 1 -- 172.21.15.46:0/2354073646 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556feebc8fe0 msgr2=0x556feebdd780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:18:47.769 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:47.768+0000 7f59f91cc740 1 --2- 172.21.15.46:0/2354073646 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556feebc8fe0 0x556feebdd780 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f59e4005ec0 tx=0x7f59e40c1550 comp rx=0 tx=0).stop 2024-03-31T23:18:47.769 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:47.768+0000 7f59f91cc740 1 -- 172.21.15.46:0/2354073646 shutdown_connections 2024-03-31T23:18:47.769 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:47.768+0000 7f59f91cc740 1 --2- 172.21.15.46:0/2354073646 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x556feebd1db0 0x556feebe9ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:18:47.769 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:47.768+0000 7f59f91cc740 1 --2- 172.21.15.46:0/2354073646 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556feebc8fe0 0x556feebdd780 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:18:47.769 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:47.768+0000 7f59f91cc740 1 --2- 172.21.15.46:0/2354073646 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556feebc86c0 0x556feebd1540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:18:47.770 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:47.768+0000 7f59f91cc740 1 -- 172.21.15.46:0/2354073646 >> 172.21.15.46:0/2354073646 conn(0x556feebc2fe0 msgr2=0x556feebc33c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:18:47.770 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:47.768+0000 7f59f91cc740 1 -- 172.21.15.46:0/2354073646 shutdown_connections 2024-03-31T23:18:47.770 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:47.768+0000 7f59f91cc740 1 -- 172.21.15.46:0/2354073646 wait complete. 2024-03-31T23:18:47.770 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:47.768+0000 7f59f91cc740 1 -- start start 2024-03-31T23:18:47.772 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:47.769+0000 7f59f91cc740 1 Processor -- start 2024-03-31T23:18:47.773 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:47.769+0000 7f59f91cc740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556feebc86c0 0x556feebcbca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:18:47.773 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:47.769+0000 7f59f91cc740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556feebc8fe0 0x556feebcc1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:18:47.773 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:47.769+0000 7f59f91cc740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x556feebd1db0 0x556feec47110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:18:47.773 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:47.769+0000 7f59f91cc740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f59f01b3920 con 0x556feebc8fe0 2024-03-31T23:18:47.773 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:47.769+0000 7f59f91cc740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f59f01b4d90 con 0x556feebd1db0 2024-03-31T23:18:47.773 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:47.769+0000 7f59f91cc740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f59f01b3ff0 con 0x556feebc86c0 2024-03-31T23:18:47.773 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:47.769+0000 7f59f91cc740 10 client.0 _ll_register_callbacks cb 0x7f59e40bd0d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T23:18:47.773 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:47.769+0000 7f59ef7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556feebc8fe0 0x556feebcc1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:18:47.773 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:47.769+0000 7f59ef7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556feebc8fe0 0x556feebcc1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59654/0 (socket says 172.21.15.46:59654) 2024-03-31T23:18:47.773 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:47.769+0000 7f59effff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556feebc86c0 0x556feebcbca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:18:47.773 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:47.769+0000 7f59ef7fe640 1 -- 172.21.15.46:0/2578297589 learned_addr learned my addr 172.21.15.46:0/2578297589 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:18:47.773 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:47.769+0000 7f59f4ab3640 1 --2- 172.21.15.46:0/2578297589 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x556feebd1db0 0x556feec47110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:18:47.773 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:47.769+0000 7f59ef7fe640 1 -- 172.21.15.46:0/2578297589 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556feebc86c0 msgr2=0x556feebcbca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:18:47.774 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:47.769+0000 7f59ef7fe640 1 --2- 172.21.15.46:0/2578297589 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556feebc86c0 0x556feebcbca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:18:47.774 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:47.769+0000 7f59ef7fe640 1 -- 172.21.15.46:0/2578297589 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x556feebd1db0 msgr2=0x556feec47110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:18:47.774 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:47.769+0000 7f59ef7fe640 1 --2- 172.21.15.46:0/2578297589 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x556feebd1db0 0x556feec47110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:18:47.774 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:47.769+0000 7f59ef7fe640 1 -- 172.21.15.46:0/2578297589 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556feec47650 con 0x556feebc8fe0 2024-03-31T23:18:47.774 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:47.769+0000 7f59effff640 1 --2- 172.21.15.46:0/2578297589 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556feebc86c0 0x556feebcbca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:18:47.774 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:47.769+0000 7f59f4ab3640 1 --2- 172.21.15.46:0/2578297589 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x556feebd1db0 0x556feec47110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:18:47.774 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:47.770+0000 7f59ef7fe640 1 --2- 172.21.15.46:0/2578297589 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556feebc8fe0 0x556feebcc1e0 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7f59e4006010 tx=0x7f59e40bcbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f5b84e2bd74aeae7 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:18:47.774 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:47.770+0000 7f59cf7fe640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-31T23:18:47.774 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:47.770+0000 7f59cf7fe640 1 -- 172.21.15.46:0/2578297589 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59e409f050 con 0x556feebc8fe0 2024-03-31T23:18:47.774 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:47.770+0000 7f59cf7fe640 1 -- 172.21.15.46:0/2578297589 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9554+0+0 (secure 0 0 0) 0x7f59e40bd4e0 con 0x556feebc8fe0 2024-03-31T23:18:47.774 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:47.770+0000 7f59f91cc740 1 -- 172.21.15.46:0/2578297589 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f59e40bd280 con 0x556feebc8fe0 2024-03-31T23:18:47.774 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:47.770+0000 7f59cf7fe640 1 -- 172.21.15.46:0/2578297589 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f59e40cf050 con 0x556feebc8fe0 2024-03-31T23:18:47.774 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:47.770+0000 7f59f91cc740 10 client.16728 fetch_fsmap learned FSMap version 30 2024-03-31T23:18:47.775 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:47.770+0000 7f59f91cc740 1 -- 172.21.15.46:0/2578297589 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x556feec47650 con 0x556feebc8fe0 2024-03-31T23:18:47.775 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:47.770+0000 7f59cf7fe640 1 -- 172.21.15.46:0/2578297589 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f59e40bd280 con 0x556feebc8fe0 2024-03-31T23:18:47.775 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:47.770+0000 7f59cf7fe640 1 -- 172.21.15.46:0/2578297589 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 30) ==== 8346+0+0 (secure 0 0 0) 0x7f59e40b90c0 con 0x556feebc8fe0 2024-03-31T23:18:47.775 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:47.770+0000 7f59f91cc740 10 client.16728 fetch_fsmap finished waiting for FSMap version 30 2024-03-31T23:18:47.775 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:47.770+0000 7f59f91cc740 10 client.16728 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-31T23:18:47.775 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:47.770+0000 7f59f91cc740 20 client.16728 populate_metadata read hostname 'smithi046' 2024-03-31T23:18:47.775 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:47.770+0000 7f59f91cc740 1 --2- 172.21.15.46:0/2578297589 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x556feec489e0 0x556feec4add0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:18:47.775 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:47.771+0000 7f59f91cc740 4 client.16728 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-31T23:18:47.775 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:47.771+0000 7f59f91cc740 1 -- 172.21.15.46:0/2578297589 --> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] -- command(tid 0: {"prefix": "flush journal"}) -- 0x556feeb38e30 con 0x556feec489e0 2024-03-31T23:18:47.775 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:47.771+0000 7f59effff640 1 --2- 172.21.15.46:0/2578297589 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x556feec489e0 0x556feec4add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:18:47.775 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:47.771+0000 7f59cf7fe640 10 client.16728 ms_handle_connect on v2:172.21.15.142:6838/2519143918 2024-03-31T23:18:47.775 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:47.771+0000 7f59effff640 1 --2- 172.21.15.46:0/2578297589 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x556feec489e0 0x556feec4add0 crc :-1 s=READY pgs=138 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4991bd4557fc5701 server_cookie=3ea47aed4673d4b5 in_seq=0 out_seq=0 2024-03-31T23:18:47.793 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:47.792+0000 7f59cf7fe640 1 -- 172.21.15.46:0/2578297589 <== mds.0 v2:172.21.15.142:6838/2519143918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x556feeb38e30 con 0x556feec489e0 2024-03-31T23:18:47.794 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:47.792+0000 7f59cf7fe640 10 client.16728 handle_command_reply: tid=0 2024-03-31T23:18:47.794 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:47.792+0000 7f59f91cc740 1 client.16728 shutdown 2024-03-31T23:18:47.794 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:47.792+0000 7f59f91cc740 1 -- 172.21.15.46:0/2578297589 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556feebc8fe0 msgr2=0x556feebcc1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:18:47.794 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:47.792+0000 7f59f91cc740 1 --2- 172.21.15.46:0/2578297589 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556feebc8fe0 0x556feebcc1e0 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7f59e4006010 tx=0x7f59e40bcbe0 comp rx=0 tx=0).stop 2024-03-31T23:18:47.794 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:47.792+0000 7f59f91cc740 1 -- 172.21.15.46:0/2578297589 shutdown_connections 2024-03-31T23:18:47.794 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:47.792+0000 7f59f91cc740 1 --2- 172.21.15.46:0/2578297589 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x556feec489e0 0x556feec4add0 crc :-1 s=READY pgs=138 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:18:47.794 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:47.792+0000 7f59f91cc740 1 --2- 172.21.15.46:0/2578297589 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x556feebd1db0 0x556feec47110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:18:47.794 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:47.792+0000 7f59f91cc740 1 --2- 172.21.15.46:0/2578297589 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556feebc8fe0 0x556feebcc1e0 unknown :-1 s=CLOSED pgs=746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:18:47.794 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:47.792+0000 7f59cf7fe640 0 client.16728 ms_handle_reset on v2:172.21.15.142:6838/2519143918 2024-03-31T23:18:47.794 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:47.792+0000 7f59f91cc740 1 --2- 172.21.15.46:0/2578297589 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556feebc86c0 0x556feebcbca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:18:47.794 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:47.792+0000 7f59f91cc740 1 -- 172.21.15.46:0/2578297589 >> 172.21.15.46:0/2578297589 conn(0x556feebc2fe0 msgr2=0x556feebcedd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:18:47.794 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:47.792+0000 7f59f91cc740 1 -- 172.21.15.46:0/2578297589 shutdown_connections 2024-03-31T23:18:47.795 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:47.792+0000 7f59f91cc740 1 -- 172.21.15.46:0/2578297589 wait complete. 2024-03-31T23:18:47.795 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:47.792+0000 7f59f91cc740 20 client.16728 trim_cache size 0 max 16384 2024-03-31T23:18:47.795 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout:{ 2024-03-31T23:18:47.795 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout: "message": "", 2024-03-31T23:18:47.795 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout: "return_code": 0 2024-03-31T23:18:47.795 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout:} 2024-03-31T23:18:47.796 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:47.793+0000 7f59f6516640 1 -- 172.21.15.46:0/4140296651 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f59d80a26b0 msgr2=0x7f59d80a4b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:18:47.796 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:47.793+0000 7f59f6516640 1 --2- 172.21.15.46:0/4140296651 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f59d80a26b0 0x7f59d80a4b70 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f59e0009f40 tx=0x7f59e0005f90 comp rx=0 tx=0).stop 2024-03-31T23:18:47.796 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:47.793+0000 7f59f6516640 1 -- 172.21.15.46:0/4140296651 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f59f01020b0 msgr2=0x7f59f010e000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:18:47.796 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:47.793+0000 7f59f6516640 1 --2- 172.21.15.46:0/4140296651 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f59f01020b0 0x7f59f010e000 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7f59e4009230 tx=0x7f59e402c5d0 comp rx=0 tx=0).stop 2024-03-31T23:18:47.796 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:47.793+0000 7f59f6516640 1 -- 172.21.15.46:0/4140296651 shutdown_connections 2024-03-31T23:18:47.796 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:47.793+0000 7f59f6516640 1 --2- 172.21.15.46:0/4140296651 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f59d80a26b0 0x7f59d80a4b70 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:18:47.796 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:47.793+0000 7f59f6516640 1 --2- 172.21.15.46:0/4140296651 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f59f0107700 0x7f59f01a2d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:18:47.796 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:47.793+0000 7f59f6516640 1 --2- 172.21.15.46:0/4140296651 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f59f01020b0 0x7f59f010e000 unknown :-1 s=CLOSED pgs=745 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:18:47.796 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:47.793+0000 7f59f6516640 1 --2- 172.21.15.46:0/4140296651 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f59f01016f0 0x7f59f010dac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:18:47.796 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:47.793+0000 7f59f6516640 1 -- 172.21.15.46:0/4140296651 >> 172.21.15.46:0/4140296651 conn(0x7f59f00fd1e0 msgr2=0x7f59f00fd860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:18:47.797 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:47.793+0000 7f59f6516640 1 -- 172.21.15.46:0/4140296651 shutdown_connections 2024-03-31T23:18:47.797 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:18:47.793+0000 7f59f6516640 1 -- 172.21.15.46:0/4140296651 wait complete. 2024-03-31T23:18:48.201 INFO:tasks.workunit.client.0.smithi046.stdout: 1 161817 2.47 MB/sec execute 234 sec latency 24.482 ms 2024-03-31T23:18:48.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:18:48 smithi162 ceph-mon[36237]: pgmap v217: 129 pgs: 129 active+clean; 104 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 431 KiB/s rd, 1.4 MiB/s wr, 114 op/s 2024-03-31T23:18:48.862 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:18:48 smithi046 ceph-mon[32498]: pgmap v217: 129 pgs: 129 active+clean; 104 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 431 KiB/s rd, 1.4 MiB/s wr, 114 op/s 2024-03-31T23:18:48.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:18:48 smithi142 ceph-mon[35271]: pgmap v217: 129 pgs: 129 active+clean; 104 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 431 KiB/s rd, 1.4 MiB/s wr, 114 op/s 2024-03-31T23:18:49.201 INFO:tasks.workunit.client.0.smithi046.stdout: 1 162259 2.47 MB/sec execute 235 sec latency 35.643 ms 2024-03-31T23:18:50.201 INFO:tasks.workunit.client.0.smithi046.stdout: 1 162948 2.49 MB/sec execute 236 sec latency 17.438 ms 2024-03-31T23:18:50.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:18:50 smithi162 ceph-mon[36237]: pgmap v218: 129 pgs: 129 active+clean; 100 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 644 KiB/s rd, 1.9 MiB/s wr, 176 op/s 2024-03-31T23:18:50.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:18:50 smithi046 ceph-mon[32498]: pgmap v218: 129 pgs: 129 active+clean; 100 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 644 KiB/s rd, 1.9 MiB/s wr, 176 op/s 2024-03-31T23:18:50.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:18:50 smithi142 ceph-mon[35271]: pgmap v218: 129 pgs: 129 active+clean; 100 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 644 KiB/s rd, 1.9 MiB/s wr, 176 op/s 2024-03-31T23:18:51.202 INFO:tasks.workunit.client.0.smithi046.stdout: 1 163410 2.50 MB/sec execute 237 sec latency 15.723 ms 2024-03-31T23:18:52.201 INFO:tasks.workunit.client.0.smithi046.stdout: 1 163650 2.49 MB/sec execute 238 sec latency 15.333 ms 2024-03-31T23:18:52.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:18:52 smithi162 ceph-mon[36237]: pgmap v219: 129 pgs: 129 active+clean; 100 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 397 KiB/s rd, 1.2 MiB/s wr, 114 op/s 2024-03-31T23:18:52.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:18:52 smithi046 ceph-mon[32498]: pgmap v219: 129 pgs: 129 active+clean; 100 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 397 KiB/s rd, 1.2 MiB/s wr, 114 op/s 2024-03-31T23:18:52.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:18:52 smithi142 ceph-mon[35271]: pgmap v219: 129 pgs: 129 active+clean; 100 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 397 KiB/s rd, 1.2 MiB/s wr, 114 op/s 2024-03-31T23:18:53.201 INFO:tasks.workunit.client.0.smithi046.stdout: 1 164050 2.48 MB/sec execute 239 sec latency 15.911 ms 2024-03-31T23:18:54.201 INFO:tasks.workunit.client.0.smithi046.stdout: 1 164567 2.49 MB/sec execute 240 sec latency 16.825 ms 2024-03-31T23:18:54.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:18:54 smithi162 ceph-mon[36237]: pgmap v220: 129 pgs: 129 active+clean; 100 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 397 KiB/s rd, 1.2 MiB/s wr, 114 op/s 2024-03-31T23:18:54.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:18:54 smithi162 ceph-mon[36237]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-31T23:18:54.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:18:54 smithi162 ceph-mon[36237]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:18:54.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:18:54 smithi162 ceph-mon[36237]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-31T23:18:54.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:18:54 smithi162 ceph-mon[36237]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' 2024-03-31T23:18:54.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:18:54 smithi046 ceph-mon[32498]: pgmap v220: 129 pgs: 129 active+clean; 100 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 397 KiB/s rd, 1.2 MiB/s wr, 114 op/s 2024-03-31T23:18:54.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:18:54 smithi046 ceph-mon[32498]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-31T23:18:54.862 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:18:54 smithi046 ceph-mon[32498]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:18:54.862 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:18:54 smithi046 ceph-mon[32498]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-31T23:18:54.862 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:18:54 smithi046 ceph-mon[32498]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' 2024-03-31T23:18:54.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:18:54 smithi142 ceph-mon[35271]: pgmap v220: 129 pgs: 129 active+clean; 100 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 397 KiB/s rd, 1.2 MiB/s wr, 114 op/s 2024-03-31T23:18:54.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:18:54 smithi142 ceph-mon[35271]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-31T23:18:54.944 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:18:54 smithi142 ceph-mon[35271]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:18:54.944 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:18:54 smithi142 ceph-mon[35271]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-31T23:18:54.944 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:18:54 smithi142 ceph-mon[35271]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' 2024-03-31T23:18:55.202 INFO:tasks.workunit.client.0.smithi046.stdout: 1 165015 2.48 MB/sec execute 241 sec latency 26.318 ms 2024-03-31T23:18:56.201 INFO:tasks.workunit.client.0.smithi046.stdout: 1 165572 2.48 MB/sec execute 242 sec latency 19.758 ms 2024-03-31T23:18:56.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:18:56 smithi162 ceph-mon[36237]: pgmap v221: 129 pgs: 129 active+clean; 106 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 636 KiB/s rd, 1.9 MiB/s wr, 174 op/s 2024-03-31T23:18:56.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:18:56 smithi046 ceph-mon[32498]: pgmap v221: 129 pgs: 129 active+clean; 106 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 636 KiB/s rd, 1.9 MiB/s wr, 174 op/s 2024-03-31T23:18:56.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:18:56 smithi142 ceph-mon[35271]: pgmap v221: 129 pgs: 129 active+clean; 106 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 636 KiB/s rd, 1.9 MiB/s wr, 174 op/s 2024-03-31T23:18:57.201 INFO:tasks.workunit.client.0.smithi046.stdout: 1 166169 2.49 MB/sec execute 243 sec latency 16.678 ms 2024-03-31T23:18:58.201 INFO:tasks.workunit.client.0.smithi046.stdout: 1 166675 2.50 MB/sec execute 244 sec latency 21.932 ms 2024-03-31T23:18:58.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:18:58 smithi162 ceph-mon[36237]: pgmap v222: 129 pgs: 129 active+clean; 106 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 452 KiB/s rd, 1.3 MiB/s wr, 121 op/s 2024-03-31T23:18:58.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:18:58 smithi046 ceph-mon[32498]: pgmap v222: 129 pgs: 129 active+clean; 106 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 452 KiB/s rd, 1.3 MiB/s wr, 121 op/s 2024-03-31T23:18:58.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:18:58 smithi142 ceph-mon[35271]: pgmap v222: 129 pgs: 129 active+clean; 106 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 452 KiB/s rd, 1.3 MiB/s wr, 121 op/s 2024-03-31T23:18:59.201 INFO:tasks.workunit.client.0.smithi046.stdout: 1 167044 2.49 MB/sec execute 245 sec latency 15.678 ms 2024-03-31T23:19:00.201 INFO:tasks.workunit.client.0.smithi046.stdout: 1 167309 2.48 MB/sec execute 246 sec latency 16.143 ms 2024-03-31T23:19:00.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:19:00 smithi162 ceph-mon[36237]: pgmap v223: 129 pgs: 129 active+clean; 107 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 638 KiB/s rd, 1.9 MiB/s wr, 175 op/s 2024-03-31T23:19:00.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:19:00 smithi046 ceph-mon[32498]: pgmap v223: 129 pgs: 129 active+clean; 107 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 638 KiB/s rd, 1.9 MiB/s wr, 175 op/s 2024-03-31T23:19:00.891 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:00.887+0000 7f1b0186b640 1 -- 172.21.15.46:0/993386700 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1afc102100 msgr2=0x7f1afc10e610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:19:00.891 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:00.887+0000 7f1b0186b640 1 --2- 172.21.15.46:0/993386700 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1afc102100 0x7f1afc10e610 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7f1af4009cd0 tx=0x7f1af402c0f0 comp rx=0 tx=0).stop 2024-03-31T23:19:00.891 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:00.887+0000 7f1b0186b640 1 -- 172.21.15.46:0/993386700 shutdown_connections 2024-03-31T23:19:00.891 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:00.887+0000 7f1b0186b640 1 --2- 172.21.15.46:0/993386700 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1afc102100 0x7f1afc10e610 unknown :-1 s=CLOSED pgs=747 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:19:00.891 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:00.887+0000 7f1b0186b640 1 --2- 172.21.15.46:0/993386700 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1afc101740 0x7f1afc101bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:19:00.891 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:00.887+0000 7f1b0186b640 1 --2- 172.21.15.46:0/993386700 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1afc107750 0x7f1afc107b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:19:00.891 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:00.888+0000 7f1b0186b640 1 -- 172.21.15.46:0/993386700 >> 172.21.15.46:0/993386700 conn(0x7f1afc0fd210 msgr2=0x7f1afc0ff630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:19:00.891 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:00.888+0000 7f1b0186b640 1 -- 172.21.15.46:0/993386700 shutdown_connections 2024-03-31T23:19:00.891 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:00.888+0000 7f1b0186b640 1 -- 172.21.15.46:0/993386700 wait complete. 2024-03-31T23:19:00.891 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:00.888+0000 7f1b0186b640 1 Processor -- start 2024-03-31T23:19:00.892 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:00.888+0000 7f1b0186b640 1 -- start start 2024-03-31T23:19:00.892 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:00.888+0000 7f1b0186b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1afc101740 0x7f1afc197f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:19:00.892 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:00.888+0000 7f1b0186b640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1afc102100 0x7f1afc198490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:19:00.892 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:00.888+0000 7f1b0186b640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1afc107750 0x7f1afc19c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:19:00.892 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:00.889+0000 7f1b0186b640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f1afc111a10 con 0x7f1afc101740 2024-03-31T23:19:00.892 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:00.889+0000 7f1b0186b640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f1afc111890 con 0x7f1afc107750 2024-03-31T23:19:00.892 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:00.889+0000 7f1b0186b640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f1afc111b90 con 0x7f1afc102100 2024-03-31T23:19:00.892 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:00.889+0000 7f1afaffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1afc101740 0x7f1afc197f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:19:00.892 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:00.889+0000 7f1afaffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1afc101740 0x7f1afc197f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:47552/0 (socket says 172.21.15.46:47552) 2024-03-31T23:19:00.892 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:00.889+0000 7f1afaffd640 1 -- 172.21.15.46:0/2161550040 learned_addr learned my addr 172.21.15.46:0/2161550040 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:19:00.892 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:00.889+0000 7f1afa7fc640 1 --2- 172.21.15.46:0/2161550040 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1afc102100 0x7f1afc198490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:19:00.892 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:00.889+0000 7f1afb7fe640 1 --2- 172.21.15.46:0/2161550040 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1afc107750 0x7f1afc19c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:19:00.892 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:00.889+0000 7f1afaffd640 1 -- 172.21.15.46:0/2161550040 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1afc102100 msgr2=0x7f1afc198490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:19:00.893 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:00.889+0000 7f1afaffd640 1 --2- 172.21.15.46:0/2161550040 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1afc102100 0x7f1afc198490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:19:00.893 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:00.889+0000 7f1afaffd640 1 -- 172.21.15.46:0/2161550040 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1afc107750 msgr2=0x7f1afc19c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:19:00.893 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:00.889+0000 7f1afaffd640 1 --2- 172.21.15.46:0/2161550040 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1afc107750 0x7f1afc19c7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:19:00.893 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:00.889+0000 7f1afaffd640 1 -- 172.21.15.46:0/2161550040 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1afc19cef0 con 0x7f1afc101740 2024-03-31T23:19:00.893 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:00.889+0000 7f1afaffd640 1 --2- 172.21.15.46:0/2161550040 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1afc101740 0x7f1afc197f50 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7f1aec00e9f0 tx=0x7f1aec00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:19:00.893 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:00.889+0000 7f1b00869640 1 -- 172.21.15.46:0/2161550040 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1aec00cd80 con 0x7f1afc101740 2024-03-31T23:19:00.893 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:00.889+0000 7f1b00869640 1 -- 172.21.15.46:0/2161550040 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1aec0108e0 con 0x7f1afc101740 2024-03-31T23:19:00.893 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:00.889+0000 7f1b0186b640 1 -- 172.21.15.46:0/2161550040 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1afc19d1e0 con 0x7f1afc101740 2024-03-31T23:19:00.893 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:00.889+0000 7f1b00869640 1 -- 172.21.15.46:0/2161550040 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1aec018920 con 0x7f1afc101740 2024-03-31T23:19:00.893 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:00.889+0000 7f1b0186b640 1 -- 172.21.15.46:0/2161550040 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1afc1a4a70 con 0x7f1afc101740 2024-03-31T23:19:00.898 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:00.892+0000 7f1b00869640 1 -- 172.21.15.46:0/2161550040 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 25) ==== 137451+0+0 (secure 0 0 0) 0x7f1aec010a90 con 0x7f1afc101740 2024-03-31T23:19:00.898 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:00.892+0000 7f1b04446740 1 Processor -- start 2024-03-31T23:19:00.898 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:00.892+0000 7f1b04446740 1 -- start start 2024-03-31T23:19:00.898 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:00.892+0000 7f1b00869640 1 --2- 172.21.15.46:0/2161550040 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f1ae40a26b0 0x7f1ae40a4b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:19:00.898 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:00.892+0000 7f1b04446740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56500ccab6c0 0x56500ccabaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:19:00.899 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:00.892+0000 7f1b04446740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56500ccabfe0 0x56500ccb4870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:19:00.899 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:00.892+0000 7f1afb7fe640 1 --2- 172.21.15.46:0/2161550040 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f1ae40a26b0 0x7f1ae40a4b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:19:00.899 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:00.892+0000 7f1b04446740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56500ccb4db0 0x56500ccb71a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:19:00.899 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:00.892+0000 7f1b04446740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x56500cc0e8f0 con 0x56500ccab6c0 2024-03-31T23:19:00.899 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:00.892+0000 7f1b00869640 1 -- 172.21.15.46:0/2161550040 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9554+0+0 (secure 0 0 0) 0x7f1aec0cf1a0 con 0x7f1afc101740 2024-03-31T23:19:00.899 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:00.892+0000 7f1b04446740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x56500cbb78d0 con 0x56500ccabfe0 2024-03-31T23:19:00.899 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:00.892+0000 7f1b04446740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x56500cc0a990 con 0x56500ccb4db0 2024-03-31T23:19:00.899 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:00.893+0000 7f1afa7fc640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56500ccabfe0 0x56500ccb4870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:19:00.899 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:00.893+0000 7f1afa7fc640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56500ccabfe0 0x56500ccb4870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.46:60150/0 (socket says 172.21.15.46:60150) 2024-03-31T23:19:00.899 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:00.893+0000 7f1afa7fc640 1 -- 172.21.15.46:0/1907282335 learned_addr learned my addr 172.21.15.46:0/1907282335 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:19:00.899 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:00.893+0000 7f1afaffd640 1 --2- 172.21.15.46:0/1907282335 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56500ccab6c0 0x56500ccabaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:19:00.900 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:00.893+0000 7f1afb7fe640 1 --2- 172.21.15.46:0/1907282335 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56500ccb4db0 0x56500ccb71a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:19:00.900 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:00.893+0000 7f1afa7fc640 1 -- 172.21.15.46:0/1907282335 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56500ccb4db0 msgr2=0x56500ccb71a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:19:00.900 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:00.893+0000 7f1afa7fc640 1 --2- 172.21.15.46:0/1907282335 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56500ccb4db0 0x56500ccb71a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:19:00.900 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:00.893+0000 7f1afa7fc640 1 -- 172.21.15.46:0/1907282335 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56500ccab6c0 msgr2=0x56500ccabaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:19:00.900 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:00.893+0000 7f1afa7fc640 1 --2- 172.21.15.46:0/1907282335 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56500ccab6c0 0x56500ccabaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:19:00.900 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:00.893+0000 7f1afa7fc640 1 -- 172.21.15.46:0/1907282335 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56500c923210 con 0x56500ccabfe0 2024-03-31T23:19:00.900 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:00.893+0000 7f1afb7fe640 1 --2- 172.21.15.46:0/2161550040 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f1ae40a26b0 0x7f1ae40a4b70 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f1af402c630 tx=0x7f1af4032d80 comp rx=0 tx=0).ready entity=mgr.15654 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:19:00.900 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:00.893+0000 7f1afa7fc640 1 --2- 172.21.15.46:0/1907282335 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56500ccabfe0 0x56500ccb4870 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f1af000bc90 tx=0x7f1af00107a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9440bc42fb0f17fb server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:19:00.900 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:00.893+0000 7f1ae1ffb640 1 -- 172.21.15.46:0/1907282335 <== mon.2 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1af0005020 con 0x56500ccabfe0 2024-03-31T23:19:00.900 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:00.893+0000 7f1ae1ffb640 1 -- 172.21.15.46:0/1907282335 <== mon.2 v2:172.21.15.142:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1af00047a0 con 0x56500ccabfe0 2024-03-31T23:19:00.900 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:00.893+0000 7f1ae1ffb640 1 -- 172.21.15.46:0/1907282335 <== mon.2 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1af00116a0 con 0x56500ccabfe0 2024-03-31T23:19:00.900 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:00.894+0000 7f1b04446740 1 -- 172.21.15.46:0/1907282335 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56500ccabfe0 msgr2=0x56500ccb4870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:19:00.900 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:00.894+0000 7f1b04446740 1 --2- 172.21.15.46:0/1907282335 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56500ccabfe0 0x56500ccb4870 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f1af000bc90 tx=0x7f1af00107a0 comp rx=0 tx=0).stop 2024-03-31T23:19:00.900 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:00.894+0000 7f1b04446740 1 -- 172.21.15.46:0/1907282335 shutdown_connections 2024-03-31T23:19:00.901 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:00.894+0000 7f1b04446740 1 --2- 172.21.15.46:0/1907282335 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56500ccb4db0 0x56500ccb71a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:19:00.901 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:00.894+0000 7f1b04446740 1 --2- 172.21.15.46:0/1907282335 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56500ccabfe0 0x56500ccb4870 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:19:00.901 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:00.894+0000 7f1b04446740 1 --2- 172.21.15.46:0/1907282335 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56500ccab6c0 0x56500ccabaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:19:00.901 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:00.894+0000 7f1b04446740 1 -- 172.21.15.46:0/1907282335 >> 172.21.15.46:0/1907282335 conn(0x56500cca5fe0 msgr2=0x56500cca63c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:19:00.901 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:00.894+0000 7f1b04446740 1 -- 172.21.15.46:0/1907282335 shutdown_connections 2024-03-31T23:19:00.901 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:00.894+0000 7f1b04446740 1 -- 172.21.15.46:0/1907282335 wait complete. 2024-03-31T23:19:00.901 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:00.894+0000 7f1b04446740 1 -- start start 2024-03-31T23:19:00.901 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:00.895+0000 7f1b04446740 1 Processor -- start 2024-03-31T23:19:00.901 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:00.895+0000 7f1b04446740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56500ccab6c0 0x56500ccb1dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:19:00.901 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:00.895+0000 7f1b04446740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56500ccabfe0 0x56500ccb2300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:19:00.901 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:00.895+0000 7f1b04446740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56500ccb4db0 0x56500ccb2840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:19:00.901 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:00.895+0000 7f1b04446740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x56500ccb79e0 con 0x56500ccabfe0 2024-03-31T23:19:00.901 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:00.895+0000 7f1b04446740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x56500cbacf80 con 0x56500ccab6c0 2024-03-31T23:19:00.901 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:00.895+0000 7f1b04446740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x56500ccb7b60 con 0x56500ccb4db0 2024-03-31T23:19:00.902 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:00.895+0000 7f1b04446740 10 client.0 _ll_register_callbacks cb 0x56500cc05290 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T23:19:00.902 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:00.895+0000 7f1afa7fc640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56500ccabfe0 0x56500ccb2300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:19:00.902 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:00.895+0000 7f1afa7fc640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56500ccabfe0 0x56500ccb2300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:47570/0 (socket says 172.21.15.46:47570) 2024-03-31T23:19:00.902 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:00.895+0000 7f1afa7fc640 1 -- 172.21.15.46:0/3695687010 learned_addr learned my addr 172.21.15.46:0/3695687010 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:19:00.902 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:00.895+0000 7f1afaffd640 1 --2- 172.21.15.46:0/3695687010 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56500ccab6c0 0x56500ccb1dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:19:00.902 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:00.895+0000 7f1afb7fe640 1 --2- 172.21.15.46:0/3695687010 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56500ccb4db0 0x56500ccb2840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:19:00.902 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:00.895+0000 7f1afa7fc640 1 -- 172.21.15.46:0/3695687010 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56500ccb4db0 msgr2=0x56500ccb2840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:19:00.902 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:00.895+0000 7f1afa7fc640 1 --2- 172.21.15.46:0/3695687010 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56500ccb4db0 0x56500ccb2840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:19:00.902 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:00.895+0000 7f1afa7fc640 1 -- 172.21.15.46:0/3695687010 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56500ccab6c0 msgr2=0x56500ccb1dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:19:00.902 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:00.895+0000 7f1afa7fc640 1 --2- 172.21.15.46:0/3695687010 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56500ccab6c0 0x56500ccb1dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:19:00.902 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:00.895+0000 7f1afa7fc640 1 -- 172.21.15.46:0/3695687010 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56500c9ef620 con 0x56500ccabfe0 2024-03-31T23:19:00.902 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:00.895+0000 7f1afaffd640 1 --2- 172.21.15.46:0/3695687010 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56500ccab6c0 0x56500ccb1dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:19:00.902 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:00.896+0000 7f1afa7fc640 1 --2- 172.21.15.46:0/3695687010 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56500ccabfe0 0x56500ccb2300 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7f1af0010ca0 tx=0x7f1af0004b80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b30439ace8bef7d3 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:19:00.903 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:00.896+0000 7f1adaffd640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-31T23:19:00.903 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:00.896+0000 7f1adaffd640 1 -- 172.21.15.46:0/3695687010 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1af0004020 con 0x56500ccabfe0 2024-03-31T23:19:00.903 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:00.896+0000 7f1adaffd640 1 -- 172.21.15.46:0/3695687010 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9554+0+0 (secure 0 0 0) 0x7f1af00041c0 con 0x56500ccabfe0 2024-03-31T23:19:00.903 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:00.896+0000 7f1b04446740 1 -- 172.21.15.46:0/3695687010 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1af00047a0 con 0x56500ccabfe0 2024-03-31T23:19:00.903 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:00.896+0000 7f1adaffd640 1 -- 172.21.15.46:0/3695687010 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f1af0030050 con 0x56500ccabfe0 2024-03-31T23:19:00.903 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:00.896+0000 7f1b04446740 10 client.16740 fetch_fsmap learned FSMap version 30 2024-03-31T23:19:00.903 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:00.896+0000 7f1b04446740 1 -- 172.21.15.46:0/3695687010 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x56500ccc3890 con 0x56500ccabfe0 2024-03-31T23:19:00.903 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:00.896+0000 7f1adaffd640 1 -- 172.21.15.46:0/3695687010 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1af00047a0 con 0x56500ccabfe0 2024-03-31T23:19:00.903 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:00.896+0000 7f1adaffd640 1 -- 172.21.15.46:0/3695687010 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 30) ==== 8346+0+0 (secure 0 0 0) 0x7f1af0003880 con 0x56500ccabfe0 2024-03-31T23:19:00.903 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:00.896+0000 7f1b04446740 10 client.16740 fetch_fsmap finished waiting for FSMap version 30 2024-03-31T23:19:00.903 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:00.896+0000 7f1b04446740 10 client.16740 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-31T23:19:00.903 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:00.896+0000 7f1b04446740 20 client.16740 populate_metadata read hostname 'smithi046' 2024-03-31T23:19:00.903 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:00.896+0000 7f1b04446740 1 --2- 172.21.15.46:0/3695687010 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x56500ccc4c90 0x56500ccc7080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:19:00.903 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:00.896+0000 7f1b04446740 4 client.16740 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-31T23:19:00.903 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:00.896+0000 7f1b04446740 1 -- 172.21.15.46:0/3695687010 --> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56500cc22120 con 0x56500ccc4c90 2024-03-31T23:19:00.904 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:00.897+0000 7f1afaffd640 1 --2- 172.21.15.46:0/3695687010 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x56500ccc4c90 0x56500ccc7080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:19:00.904 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:00.897+0000 7f1adaffd640 10 client.16740 ms_handle_connect on v2:172.21.15.142:6838/2519143918 2024-03-31T23:19:00.904 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:00.897+0000 7f1afaffd640 1 --2- 172.21.15.46:0/3695687010 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x56500ccc4c90 0x56500ccc7080 crc :-1 s=READY pgs=139 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2d67adfb2e006d6d server_cookie=dbd73215aad92e5a in_seq=0 out_seq=0 2024-03-31T23:19:00.905 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:00.898+0000 7f1adaffd640 1 -- 172.21.15.46:0/3695687010 <== mds.0 v2:172.21.15.142:6838/2519143918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x56500cc22120 con 0x56500ccc4c90 2024-03-31T23:19:00.905 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:00.898+0000 7f1adaffd640 10 client.16740 handle_command_reply: tid=0 2024-03-31T23:19:00.905 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:00.898+0000 7f1b04446740 1 client.16740 shutdown 2024-03-31T23:19:00.905 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:00.898+0000 7f1b04446740 1 -- 172.21.15.46:0/3695687010 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56500ccabfe0 msgr2=0x56500ccb2300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:19:00.905 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:00.898+0000 7f1b04446740 1 --2- 172.21.15.46:0/3695687010 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56500ccabfe0 0x56500ccb2300 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7f1af0010ca0 tx=0x7f1af0004b80 comp rx=0 tx=0).stop 2024-03-31T23:19:00.905 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:00.899+0000 7f1b04446740 1 -- 172.21.15.46:0/3695687010 shutdown_connections 2024-03-31T23:19:00.905 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:00.899+0000 7f1b04446740 1 --2- 172.21.15.46:0/3695687010 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x56500ccc4c90 0x56500ccc7080 crc :-1 s=READY pgs=139 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:19:00.905 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:00.899+0000 7f1b04446740 1 --2- 172.21.15.46:0/3695687010 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56500ccb4db0 0x56500ccb2840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:19:00.906 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:00.899+0000 7f1b04446740 1 --2- 172.21.15.46:0/3695687010 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56500ccabfe0 0x56500ccb2300 unknown :-1 s=CLOSED pgs=749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:19:00.906 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:00.899+0000 7f1adaffd640 0 client.16740 ms_handle_reset on v2:172.21.15.142:6838/2519143918 2024-03-31T23:19:00.906 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:00.899+0000 7f1b04446740 1 --2- 172.21.15.46:0/3695687010 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56500ccab6c0 0x56500ccb1dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:19:00.906 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:00.899+0000 7f1b04446740 1 -- 172.21.15.46:0/3695687010 >> 172.21.15.46:0/3695687010 conn(0x56500cca5fe0 msgr2=0x56500cca7d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:19:00.906 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:00.899+0000 7f1b04446740 1 -- 172.21.15.46:0/3695687010 shutdown_connections 2024-03-31T23:19:00.906 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:00.899+0000 7f1b04446740 1 -- 172.21.15.46:0/3695687010 wait complete. 2024-03-31T23:19:00.906 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:00.899+0000 7f1b04446740 20 client.16740 trim_cache size 0 max 16384 2024-03-31T23:19:00.916 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:00.914+0000 7f1b04446740 1 Processor -- start 2024-03-31T23:19:00.916 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:00.914+0000 7f1b04446740 1 -- start start 2024-03-31T23:19:00.916 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:00.914+0000 7f1b04446740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56500ccab6c0 0x56500ccb44f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:19:00.916 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:00.914+0000 7f1b04446740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56500ccabfe0 0x56500ccc0780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:19:00.916 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:00.914+0000 7f1b04446740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56500ccb4db0 0x56500cccc710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:19:00.916 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:00.914+0000 7f1b04446740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f1afc1b15a0 con 0x56500ccabfe0 2024-03-31T23:19:00.916 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:00.914+0000 7f1b04446740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f1afc1aee70 con 0x56500ccab6c0 2024-03-31T23:19:00.916 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:00.914+0000 7f1b04446740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f1afc1acce0 con 0x56500ccb4db0 2024-03-31T23:19:00.917 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:00.914+0000 7f1afa7fc640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56500ccabfe0 0x56500ccc0780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:19:00.917 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:00.914+0000 7f1afa7fc640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56500ccabfe0 0x56500ccc0780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:47572/0 (socket says 172.21.15.46:47572) 2024-03-31T23:19:00.917 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:00.914+0000 7f1afa7fc640 1 -- 172.21.15.46:0/1837784553 learned_addr learned my addr 172.21.15.46:0/1837784553 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:19:00.917 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:00.914+0000 7f1afaffd640 1 --2- 172.21.15.46:0/1837784553 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56500ccab6c0 0x56500ccb44f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:19:00.917 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:00.914+0000 7f1afb7fe640 1 --2- 172.21.15.46:0/1837784553 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56500ccb4db0 0x56500cccc710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:19:00.917 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:00.914+0000 7f1afa7fc640 1 -- 172.21.15.46:0/1837784553 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56500ccb4db0 msgr2=0x56500cccc710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:19:00.917 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:00.914+0000 7f1afa7fc640 1 --2- 172.21.15.46:0/1837784553 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56500ccb4db0 0x56500cccc710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:19:00.917 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:00.914+0000 7f1afa7fc640 1 -- 172.21.15.46:0/1837784553 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56500ccab6c0 msgr2=0x56500ccb44f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:19:00.917 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:00.914+0000 7f1afa7fc640 1 --2- 172.21.15.46:0/1837784553 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56500ccab6c0 0x56500ccb44f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:19:00.917 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:00.914+0000 7f1afa7fc640 1 -- 172.21.15.46:0/1837784553 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56500ccb0310 con 0x56500ccabfe0 2024-03-31T23:19:00.917 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:00.915+0000 7f1afa7fc640 1 --2- 172.21.15.46:0/1837784553 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56500ccabfe0 0x56500ccc0780 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7f1af0003e40 tx=0x7f1af000bae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=91c64f101dca6bb2 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:19:00.917 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:00.915+0000 7f1ae17fa640 1 -- 172.21.15.46:0/1837784553 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1af003a070 con 0x56500ccabfe0 2024-03-31T23:19:00.917 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:00.915+0000 7f1ae17fa640 1 -- 172.21.15.46:0/1837784553 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1af000b410 con 0x56500ccabfe0 2024-03-31T23:19:00.917 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:00.915+0000 7f1ae17fa640 1 -- 172.21.15.46:0/1837784553 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1af0004120 con 0x56500ccabfe0 2024-03-31T23:19:00.921 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:00.915+0000 7f1b04446740 1 -- 172.21.15.46:0/1837784553 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56500ccabfe0 msgr2=0x56500ccc0780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:19:00.921 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:00.915+0000 7f1b04446740 1 --2- 172.21.15.46:0/1837784553 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56500ccabfe0 0x56500ccc0780 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7f1af0003e40 tx=0x7f1af000bae0 comp rx=0 tx=0).stop 2024-03-31T23:19:00.921 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:00.915+0000 7f1b04446740 1 -- 172.21.15.46:0/1837784553 shutdown_connections 2024-03-31T23:19:00.921 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:00.915+0000 7f1b04446740 1 --2- 172.21.15.46:0/1837784553 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56500ccb4db0 0x56500cccc710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:19:00.921 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:00.915+0000 7f1b04446740 1 --2- 172.21.15.46:0/1837784553 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56500ccabfe0 0x56500ccc0780 unknown :-1 s=CLOSED pgs=750 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:19:00.921 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:00.916+0000 7f1b04446740 1 --2- 172.21.15.46:0/1837784553 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56500ccab6c0 0x56500ccb44f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:19:00.921 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:00.916+0000 7f1b04446740 1 -- 172.21.15.46:0/1837784553 >> 172.21.15.46:0/1837784553 conn(0x56500cca5fe0 msgr2=0x56500ccad130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:19:00.921 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:00.916+0000 7f1b04446740 1 -- 172.21.15.46:0/1837784553 shutdown_connections 2024-03-31T23:19:00.921 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:00.916+0000 7f1b04446740 1 -- 172.21.15.46:0/1837784553 wait complete. 2024-03-31T23:19:00.921 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:00.916+0000 7f1b04446740 1 -- start start 2024-03-31T23:19:00.921 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:00.916+0000 7f1b04446740 1 Processor -- start 2024-03-31T23:19:00.921 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:00.916+0000 7f1b04446740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56500ccab6c0 0x56500ccae940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:19:00.921 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:00.916+0000 7f1b04446740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56500ccabfe0 0x56500ccaee80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:19:00.922 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:00.916+0000 7f1b04446740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56500ccb4db0 0x56500ccaf3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:19:00.922 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:00.916+0000 7f1b04446740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f1afc1ad330 con 0x56500ccabfe0 2024-03-31T23:19:00.922 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:00.916+0000 7f1b04446740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f1afc1ae7a0 con 0x56500ccb4db0 2024-03-31T23:19:00.922 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:00.916+0000 7f1b04446740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f1afc1ada00 con 0x56500ccab6c0 2024-03-31T23:19:00.922 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:00.916+0000 7f1b04446740 10 client.0 _ll_register_callbacks cb 0x7f1aec097140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T23:19:00.922 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:00.916+0000 7f1afa7fc640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56500ccabfe0 0x56500ccaee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:19:00.922 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:00.917+0000 7f1afa7fc640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56500ccabfe0 0x56500ccaee80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:47588/0 (socket says 172.21.15.46:47588) 2024-03-31T23:19:00.922 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:00.917+0000 7f1afa7fc640 1 -- 172.21.15.46:0/11559615 learned_addr learned my addr 172.21.15.46:0/11559615 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:19:00.922 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:00.917+0000 7f1afb7fe640 1 --2- 172.21.15.46:0/11559615 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56500ccb4db0 0x56500ccaf3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:19:00.922 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:00.917+0000 7f1afaffd640 1 --2- 172.21.15.46:0/11559615 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56500ccab6c0 0x56500ccae940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:19:00.922 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:00.917+0000 7f1afa7fc640 1 -- 172.21.15.46:0/11559615 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56500ccab6c0 msgr2=0x56500ccae940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:19:00.922 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:00.917+0000 7f1afa7fc640 1 --2- 172.21.15.46:0/11559615 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56500ccab6c0 0x56500ccae940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:19:00.922 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:00.917+0000 7f1afa7fc640 1 -- 172.21.15.46:0/11559615 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56500ccb4db0 msgr2=0x56500ccaf3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:19:00.923 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:00.917+0000 7f1afa7fc640 1 --2- 172.21.15.46:0/11559615 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56500ccb4db0 0x56500ccaf3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:19:00.923 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:00.917+0000 7f1afa7fc640 1 -- 172.21.15.46:0/11559615 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56500cd29dc0 con 0x56500ccabfe0 2024-03-31T23:19:00.923 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:00.917+0000 7f1afa7fc640 1 --2- 172.21.15.46:0/11559615 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56500ccabfe0 0x56500ccaee80 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7f1af000be80 tx=0x7f1af000b810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dea491f692470b44 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:19:00.923 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:00.917+0000 7f1adaffd640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-31T23:19:00.923 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:00.917+0000 7f1adaffd640 1 -- 172.21.15.46:0/11559615 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1af003a040 con 0x56500ccabfe0 2024-03-31T23:19:00.923 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:00.917+0000 7f1adaffd640 1 -- 172.21.15.46:0/11559615 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9554+0+0 (secure 0 0 0) 0x7f1af00035f0 con 0x56500ccabfe0 2024-03-31T23:19:00.923 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:00.917+0000 7f1b04446740 1 -- 172.21.15.46:0/11559615 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x56500cc1be30 con 0x56500ccabfe0 2024-03-31T23:19:00.923 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:00.917+0000 7f1adaffd640 1 -- 172.21.15.46:0/11559615 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f1af0048050 con 0x56500ccabfe0 2024-03-31T23:19:00.923 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:00.917+0000 7f1b04446740 10 client.16746 fetch_fsmap learned FSMap version 30 2024-03-31T23:19:00.923 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:00.917+0000 7f1b04446740 1 -- 172.21.15.46:0/11559615 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x56500cd2b090 con 0x56500ccabfe0 2024-03-31T23:19:00.923 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:00.917+0000 7f1adaffd640 1 -- 172.21.15.46:0/11559615 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x56500cc1be30 con 0x56500ccabfe0 2024-03-31T23:19:00.923 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:00.918+0000 7f1adaffd640 1 -- 172.21.15.46:0/11559615 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 30) ==== 8346+0+0 (secure 0 0 0) 0x7f1af003d020 con 0x56500ccabfe0 2024-03-31T23:19:00.923 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:00.918+0000 7f1b04446740 10 client.16746 fetch_fsmap finished waiting for FSMap version 30 2024-03-31T23:19:00.923 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:00.918+0000 7f1b04446740 10 client.16746 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-31T23:19:00.924 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:00.918+0000 7f1b04446740 20 client.16746 populate_metadata read hostname 'smithi046' 2024-03-31T23:19:00.924 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:00.918+0000 7f1b04446740 1 --2- 172.21.15.46:0/11559615 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x56500cd2b7a0 0x56500cd2dbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:19:00.924 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:00.918+0000 7f1b04446740 4 client.16746 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-31T23:19:00.924 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:00.918+0000 7f1b04446740 1 -- 172.21.15.46:0/11559615 --> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] -- command(tid 0: {"prefix": "flush journal"}) -- 0x56500cc16180 con 0x56500cd2b7a0 2024-03-31T23:19:00.924 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:00.918+0000 7f1afaffd640 1 --2- 172.21.15.46:0/11559615 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x56500cd2b7a0 0x56500cd2dbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:19:00.924 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:00.918+0000 7f1adaffd640 10 client.16746 ms_handle_connect on v2:172.21.15.142:6838/2519143918 2024-03-31T23:19:00.924 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:00.918+0000 7f1afaffd640 1 --2- 172.21.15.46:0/11559615 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x56500cd2b7a0 0x56500cd2dbc0 crc :-1 s=READY pgs=140 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=981fb96cb510c818 server_cookie=c67d78f9953000b6 in_seq=0 out_seq=0 2024-03-31T23:19:00.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:19:00 smithi142 ceph-mon[35271]: pgmap v223: 129 pgs: 129 active+clean; 107 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 638 KiB/s rd, 1.9 MiB/s wr, 175 op/s 2024-03-31T23:19:00.944 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:00.943+0000 7f1adaffd640 1 -- 172.21.15.46:0/11559615 <== mds.0 v2:172.21.15.142:6838/2519143918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x56500cc16180 con 0x56500cd2b7a0 2024-03-31T23:19:00.944 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:00.943+0000 7f1adaffd640 10 client.16746 handle_command_reply: tid=0 2024-03-31T23:19:00.944 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:00.943+0000 7f1b04446740 1 client.16746 shutdown 2024-03-31T23:19:00.945 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:00.943+0000 7f1b04446740 1 -- 172.21.15.46:0/11559615 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56500ccabfe0 msgr2=0x56500ccaee80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:19:00.945 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:00.943+0000 7f1b04446740 1 --2- 172.21.15.46:0/11559615 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56500ccabfe0 0x56500ccaee80 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7f1af000be80 tx=0x7f1af000b810 comp rx=0 tx=0).stop 2024-03-31T23:19:00.945 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:00.943+0000 7f1b04446740 1 -- 172.21.15.46:0/11559615 shutdown_connections 2024-03-31T23:19:00.945 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:00.943+0000 7f1b04446740 1 --2- 172.21.15.46:0/11559615 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x56500cd2b7a0 0x56500cd2dbc0 crc :-1 s=READY pgs=140 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:19:00.945 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:00.943+0000 7f1b04446740 1 --2- 172.21.15.46:0/11559615 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56500ccb4db0 0x56500ccaf3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:19:00.945 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:00.943+0000 7f1b04446740 1 --2- 172.21.15.46:0/11559615 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56500ccabfe0 0x56500ccaee80 unknown :-1 s=CLOSED pgs=751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:19:00.946 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:00.943+0000 7f1b04446740 1 --2- 172.21.15.46:0/11559615 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56500ccab6c0 0x56500ccae940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:19:00.946 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:00.943+0000 7f1b04446740 1 -- 172.21.15.46:0/11559615 >> 172.21.15.46:0/11559615 conn(0x56500cca5fe0 msgr2=0x56500ccb1d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:19:00.946 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:00.943+0000 7f1adaffd640 0 client.16746 ms_handle_reset on v2:172.21.15.142:6838/2519143918 2024-03-31T23:19:00.946 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:00.944+0000 7f1b04446740 1 -- 172.21.15.46:0/11559615 shutdown_connections 2024-03-31T23:19:00.946 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:00.944+0000 7f1b04446740 1 -- 172.21.15.46:0/11559615 wait complete. 2024-03-31T23:19:00.946 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:00.944+0000 7f1b04446740 20 client.16746 trim_cache size 0 max 16384 2024-03-31T23:19:00.946 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout:{ 2024-03-31T23:19:00.946 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout: "message": "", 2024-03-31T23:19:00.946 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout: "return_code": 0 2024-03-31T23:19:00.946 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout:} 2024-03-31T23:19:00.947 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:00.944+0000 7f1b0186b640 1 -- 172.21.15.46:0/2161550040 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f1ae40a26b0 msgr2=0x7f1ae40a4b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:19:00.947 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:00.944+0000 7f1b0186b640 1 --2- 172.21.15.46:0/2161550040 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f1ae40a26b0 0x7f1ae40a4b70 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f1af402c630 tx=0x7f1af4032d80 comp rx=0 tx=0).stop 2024-03-31T23:19:00.947 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:00.944+0000 7f1b0186b640 1 -- 172.21.15.46:0/2161550040 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1afc101740 msgr2=0x7f1afc197f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:19:00.947 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:00.944+0000 7f1b0186b640 1 --2- 172.21.15.46:0/2161550040 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1afc101740 0x7f1afc197f50 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7f1aec00e9f0 tx=0x7f1aec00eeb0 comp rx=0 tx=0).stop 2024-03-31T23:19:00.947 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:00.945+0000 7f1b0186b640 1 -- 172.21.15.46:0/2161550040 shutdown_connections 2024-03-31T23:19:00.948 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:00.945+0000 7f1b0186b640 1 --2- 172.21.15.46:0/2161550040 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f1ae40a26b0 0x7f1ae40a4b70 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:19:00.948 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:00.945+0000 7f1b0186b640 1 --2- 172.21.15.46:0/2161550040 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1afc107750 0x7f1afc19c7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:19:00.948 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:00.945+0000 7f1b0186b640 1 --2- 172.21.15.46:0/2161550040 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1afc102100 0x7f1afc198490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:19:00.948 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:00.945+0000 7f1b0186b640 1 --2- 172.21.15.46:0/2161550040 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1afc101740 0x7f1afc197f50 unknown :-1 s=CLOSED pgs=748 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:19:00.948 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:00.945+0000 7f1b0186b640 1 -- 172.21.15.46:0/2161550040 >> 172.21.15.46:0/2161550040 conn(0x7f1afc0fd210 msgr2=0x7f1afc0fd9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:19:00.948 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:00.945+0000 7f1b0186b640 1 -- 172.21.15.46:0/2161550040 shutdown_connections 2024-03-31T23:19:00.948 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:00.945+0000 7f1b0186b640 1 -- 172.21.15.46:0/2161550040 wait complete. 2024-03-31T23:19:01.202 INFO:tasks.workunit.client.0.smithi046.stdout: 1 167711 2.48 MB/sec execute 247 sec latency 25.171 ms 2024-03-31T23:19:02.202 INFO:tasks.workunit.client.0.smithi046.stdout: 1 168224 2.48 MB/sec execute 248 sec latency 20.453 ms 2024-03-31T23:19:02.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:19:02 smithi162 ceph-mon[36237]: pgmap v224: 129 pgs: 129 active+clean; 107 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 425 KiB/s rd, 1.4 MiB/s wr, 114 op/s 2024-03-31T23:19:02.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:19:02 smithi046 ceph-mon[32498]: pgmap v224: 129 pgs: 129 active+clean; 107 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 425 KiB/s rd, 1.4 MiB/s wr, 114 op/s 2024-03-31T23:19:02.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:19:02 smithi142 ceph-mon[35271]: pgmap v224: 129 pgs: 129 active+clean; 107 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 425 KiB/s rd, 1.4 MiB/s wr, 114 op/s 2024-03-31T23:19:03.202 INFO:tasks.workunit.client.0.smithi046.stdout: 1 168712 2.48 MB/sec execute 249 sec latency 24.129 ms 2024-03-31T23:19:03.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:19:03 smithi162 ceph-mon[36237]: osdmap e99: 12 total, 12 up, 12 in 2024-03-31T23:19:03.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:19:03 smithi046 ceph-mon[32498]: osdmap e99: 12 total, 12 up, 12 in 2024-03-31T23:19:03.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:19:03 smithi142 ceph-mon[35271]: osdmap e99: 12 total, 12 up, 12 in 2024-03-31T23:19:04.202 INFO:tasks.workunit.client.0.smithi046.stdout: 1 169204 2.48 MB/sec execute 250 sec latency 22.431 ms 2024-03-31T23:19:04.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:19:04 smithi162 ceph-mon[36237]: pgmap v226: 129 pgs: 129 active+clean; 107 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 510 KiB/s rd, 1.7 MiB/s wr, 136 op/s 2024-03-31T23:19:04.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:19:04 smithi046 ceph-mon[32498]: pgmap v226: 129 pgs: 129 active+clean; 107 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 510 KiB/s rd, 1.7 MiB/s wr, 136 op/s 2024-03-31T23:19:04.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:19:04 smithi142 ceph-mon[35271]: pgmap v226: 129 pgs: 129 active+clean; 107 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 510 KiB/s rd, 1.7 MiB/s wr, 136 op/s 2024-03-31T23:19:05.202 INFO:tasks.workunit.client.0.smithi046.stdout: 1 169918 2.49 MB/sec execute 251 sec latency 16.431 ms 2024-03-31T23:19:06.202 INFO:tasks.workunit.client.0.smithi046.stdout: 1 170296 2.49 MB/sec execute 252 sec latency 19.101 ms 2024-03-31T23:19:06.798 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:19:06 smithi162 ceph-mon[36237]: pgmap v227: 129 pgs: 129 active+clean; 101 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 543 KiB/s rd, 1.4 MiB/s wr, 156 op/s 2024-03-31T23:19:06.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:19:06 smithi046 ceph-mon[32498]: pgmap v227: 129 pgs: 129 active+clean; 101 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 543 KiB/s rd, 1.4 MiB/s wr, 156 op/s 2024-03-31T23:19:06.946 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:19:06 smithi142 ceph-mon[35271]: pgmap v227: 129 pgs: 129 active+clean; 101 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 543 KiB/s rd, 1.4 MiB/s wr, 156 op/s 2024-03-31T23:19:07.202 INFO:tasks.workunit.client.0.smithi046.stdout: 1 170637 2.49 MB/sec execute 253 sec latency 15.817 ms 2024-03-31T23:19:08.202 INFO:tasks.workunit.client.0.smithi046.stdout: 1 170879 2.48 MB/sec execute 254 sec latency 19.877 ms 2024-03-31T23:19:08.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:19:08 smithi162 ceph-mon[36237]: pgmap v228: 129 pgs: 129 active+clean; 101 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 543 KiB/s rd, 1.4 MiB/s wr, 156 op/s 2024-03-31T23:19:08.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:19:08 smithi046 ceph-mon[32498]: pgmap v228: 129 pgs: 129 active+clean; 101 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 543 KiB/s rd, 1.4 MiB/s wr, 156 op/s 2024-03-31T23:19:08.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:19:08 smithi142 ceph-mon[35271]: pgmap v228: 129 pgs: 129 active+clean; 101 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 543 KiB/s rd, 1.4 MiB/s wr, 156 op/s 2024-03-31T23:19:09.202 INFO:tasks.workunit.client.0.smithi046.stdout: 1 171329 2.48 MB/sec execute 255 sec latency 26.742 ms 2024-03-31T23:19:10.202 INFO:tasks.workunit.client.0.smithi046.stdout: 1 171796 2.48 MB/sec execute 256 sec latency 25.640 ms 2024-03-31T23:19:10.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:19:10 smithi162 ceph-mon[36237]: pgmap v229: 129 pgs: 129 active+clean; 106 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 594 KiB/s rd, 1.5 MiB/s wr, 163 op/s 2024-03-31T23:19:10.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:19:10 smithi162 ceph-mon[36237]: osdmap e100: 12 total, 12 up, 12 in 2024-03-31T23:19:10.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:19:10 smithi046 ceph-mon[32498]: pgmap v229: 129 pgs: 129 active+clean; 106 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 594 KiB/s rd, 1.5 MiB/s wr, 163 op/s 2024-03-31T23:19:10.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:19:10 smithi046 ceph-mon[32498]: osdmap e100: 12 total, 12 up, 12 in 2024-03-31T23:19:10.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:19:10 smithi142 ceph-mon[35271]: pgmap v229: 129 pgs: 129 active+clean; 106 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 594 KiB/s rd, 1.5 MiB/s wr, 163 op/s 2024-03-31T23:19:10.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:19:10 smithi142 ceph-mon[35271]: osdmap e100: 12 total, 12 up, 12 in 2024-03-31T23:19:11.202 INFO:tasks.workunit.client.0.smithi046.stdout: 1 172318 2.47 MB/sec execute 257 sec latency 25.264 ms 2024-03-31T23:19:12.202 INFO:tasks.workunit.client.0.smithi046.stdout: 1 172769 2.47 MB/sec execute 258 sec latency 22.383 ms 2024-03-31T23:19:12.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:19:12 smithi162 ceph-mon[36237]: pgmap v231: 129 pgs: 129 active+clean; 106 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 719 KiB/s rd, 1.8 MiB/s wr, 198 op/s 2024-03-31T23:19:12.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:19:12 smithi046 ceph-mon[32498]: pgmap v231: 129 pgs: 129 active+clean; 106 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 719 KiB/s rd, 1.8 MiB/s wr, 198 op/s 2024-03-31T23:19:12.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:19:12 smithi142 ceph-mon[35271]: pgmap v231: 129 pgs: 129 active+clean; 106 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 719 KiB/s rd, 1.8 MiB/s wr, 198 op/s 2024-03-31T23:19:13.202 INFO:tasks.workunit.client.0.smithi046.stdout: 1 173527 2.49 MB/sec execute 259 sec latency 16.302 ms 2024-03-31T23:19:14.041 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:14.038+0000 7fb12f82a640 1 -- 172.21.15.46:0/3703105336 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb128107820 msgr2=0x7fb128107c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:19:14.042 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:14.038+0000 7fb12f82a640 1 --2- 172.21.15.46:0/3703105336 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb128107820 0x7fb128107c00 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7fb124009930 tx=0x7fb12402bc30 comp rx=0 tx=0).stop 2024-03-31T23:19:14.042 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:14.038+0000 7fb12f82a640 1 -- 172.21.15.46:0/3703105336 shutdown_connections 2024-03-31T23:19:14.042 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:14.038+0000 7fb12f82a640 1 --2- 172.21.15.46:0/3703105336 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb128069770 0x7fb12810e5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:19:14.042 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:14.038+0000 7fb12f82a640 1 --2- 172.21.15.46:0/3703105336 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb128068dd0 0x7fb128069230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:19:14.042 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:14.038+0000 7fb12f82a640 1 --2- 172.21.15.46:0/3703105336 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb128107820 0x7fb128107c00 unknown :-1 s=CLOSED pgs=752 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:19:14.042 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:14.038+0000 7fb12f82a640 1 -- 172.21.15.46:0/3703105336 >> 172.21.15.46:0/3703105336 conn(0x7fb1280774a0 msgr2=0x7fb1280778b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:19:14.042 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:14.038+0000 7fb12f82a640 1 -- 172.21.15.46:0/3703105336 shutdown_connections 2024-03-31T23:19:14.042 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:14.038+0000 7fb12f82a640 1 -- 172.21.15.46:0/3703105336 wait complete. 2024-03-31T23:19:14.042 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:14.039+0000 7fb12f82a640 1 Processor -- start 2024-03-31T23:19:14.042 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:14.039+0000 7fb12f82a640 1 -- start start 2024-03-31T23:19:14.043 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:14.039+0000 7fb12f82a640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb128068dd0 0x7fb128197e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:19:14.043 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:14.039+0000 7fb12f82a640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb128069770 0x7fb128198340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:19:14.043 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:14.039+0000 7fb12f82a640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb128107820 0x7fb12819c6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:19:14.043 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:14.039+0000 7fb12f82a640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fb1281119c0 con 0x7fb128107820 2024-03-31T23:19:14.043 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:14.039+0000 7fb12f82a640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fb128111840 con 0x7fb128068dd0 2024-03-31T23:19:14.043 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:14.039+0000 7fb12f82a640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fb128111b40 con 0x7fb128069770 2024-03-31T23:19:14.043 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:14.039+0000 7fb12ddc7640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb128107820 0x7fb12819c6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:19:14.043 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:14.039+0000 7fb12ddc7640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb128107820 0x7fb12819c6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:43498/0 (socket says 172.21.15.46:43498) 2024-03-31T23:19:14.043 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:14.039+0000 7fb12d5c6640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb128068dd0 0x7fb128197e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:19:14.043 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:14.039+0000 7fb12ddc7640 1 -- 172.21.15.46:0/566099916 learned_addr learned my addr 172.21.15.46:0/566099916 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:19:14.043 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:14.039+0000 7fb12cdc5640 1 --2- 172.21.15.46:0/566099916 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb128069770 0x7fb128198340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:19:14.043 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:14.040+0000 7fb12ddc7640 1 -- 172.21.15.46:0/566099916 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb128069770 msgr2=0x7fb128198340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:19:14.044 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:14.040+0000 7fb12ddc7640 1 --2- 172.21.15.46:0/566099916 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb128069770 0x7fb128198340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:19:14.044 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:14.040+0000 7fb12ddc7640 1 -- 172.21.15.46:0/566099916 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb128068dd0 msgr2=0x7fb128197e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:19:14.044 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:14.040+0000 7fb12ddc7640 1 --2- 172.21.15.46:0/566099916 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb128068dd0 0x7fb128197e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:19:14.044 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:14.040+0000 7fb12ddc7640 1 -- 172.21.15.46:0/566099916 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb12819cdf0 con 0x7fb128107820 2024-03-31T23:19:14.044 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:14.040+0000 7fb12ddc7640 1 --2- 172.21.15.46:0/566099916 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb128107820 0x7fb12819c6c0 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7fb12000ea60 tx=0x7fb12000ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:19:14.044 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:14.040+0000 7fb11e7fc640 1 -- 172.21.15.46:0/566099916 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb12000ce40 con 0x7fb128107820 2024-03-31T23:19:14.044 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:14.040+0000 7fb11e7fc640 1 -- 172.21.15.46:0/566099916 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb120005020 con 0x7fb128107820 2024-03-31T23:19:14.044 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:14.040+0000 7fb12f82a640 1 -- 172.21.15.46:0/566099916 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb12819d0e0 con 0x7fb128107820 2024-03-31T23:19:14.044 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:14.040+0000 7fb11e7fc640 1 -- 172.21.15.46:0/566099916 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb120010670 con 0x7fb128107820 2024-03-31T23:19:14.044 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:14.040+0000 7fb12f82a640 1 -- 172.21.15.46:0/566099916 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb1281008a0 con 0x7fb128107820 2024-03-31T23:19:14.050 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:14.042+0000 7fb11e7fc640 1 -- 172.21.15.46:0/566099916 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 25) ==== 137451+0+0 (secure 0 0 0) 0x7fb1200051d0 con 0x7fb128107820 2024-03-31T23:19:14.051 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:14.043+0000 7fb13240d740 1 Processor -- start 2024-03-31T23:19:14.051 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:14.043+0000 7fb13240d740 1 -- start start 2024-03-31T23:19:14.051 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:14.043+0000 7fb11e7fc640 1 --2- 172.21.15.46:0/566099916 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fb1140a26b0 0x7fb1140a4b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:19:14.051 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:14.043+0000 7fb13240d740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5619828c06c0 0x5619828c0aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:19:14.051 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:14.043+0000 7fb13240d740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5619828c0fe0 0x5619828c9870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:19:14.051 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:14.043+0000 7fb13240d740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5619828c9db0 0x5619828cc1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:19:14.051 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:14.043+0000 7fb13240d740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x5619828238f0 con 0x5619828c0fe0 2024-03-31T23:19:14.051 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:14.043+0000 7fb13240d740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x5619827cc8d0 con 0x5619828c9db0 2024-03-31T23:19:14.051 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:14.043+0000 7fb13240d740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x56198281f990 con 0x5619828c06c0 2024-03-31T23:19:14.051 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:14.043+0000 7fb12cdc5640 1 --2- 172.21.15.46:0/566099916 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fb1140a26b0 0x7fb1140a4b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:19:14.051 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:14.043+0000 7fb11e7fc640 1 -- 172.21.15.46:0/566099916 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9554+0+0 (secure 0 0 0) 0x7fb1200cf410 con 0x7fb128107820 2024-03-31T23:19:14.051 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:14.043+0000 7fb12d5c6640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5619828c06c0 0x5619828c0aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:19:14.052 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:14.043+0000 7fb12cdc5640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5619828c0fe0 0x5619828c9870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:19:14.052 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:14.043+0000 7fb12d5c6640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5619828c06c0 0x5619828c0aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.46:41166/0 (socket says 172.21.15.46:41166) 2024-03-31T23:19:14.052 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:14.043+0000 7fb12d5c6640 1 -- 172.21.15.46:0/561912128 learned_addr learned my addr 172.21.15.46:0/561912128 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:19:14.052 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:14.043+0000 7fb12ddc7640 1 --2- 172.21.15.46:0/561912128 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5619828c9db0 0x5619828cc1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:19:14.052 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:14.044+0000 7fb12d5c6640 1 -- 172.21.15.46:0/561912128 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5619828c9db0 msgr2=0x5619828cc1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:19:14.052 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:14.044+0000 7fb12d5c6640 1 --2- 172.21.15.46:0/561912128 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5619828c9db0 0x5619828cc1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:19:14.052 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:14.044+0000 7fb12d5c6640 1 -- 172.21.15.46:0/561912128 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5619828c0fe0 msgr2=0x5619828c9870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:19:14.052 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:14.044+0000 7fb12d5c6640 1 --2- 172.21.15.46:0/561912128 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5619828c0fe0 0x5619828c9870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:19:14.052 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:14.044+0000 7fb12d5c6640 1 -- 172.21.15.46:0/561912128 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561982538210 con 0x5619828c06c0 2024-03-31T23:19:14.052 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:14.044+0000 7fb12cdc5640 1 --2- 172.21.15.46:0/566099916 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fb1140a26b0 0x7fb1140a4b70 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fb128077b70 tx=0x7fb118010d10 comp rx=0 tx=0).ready entity=mgr.15654 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:19:14.052 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:14.044+0000 7fb12d5c6640 1 --2- 172.21.15.46:0/561912128 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5619828c06c0 0x5619828c0aa0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fb124036040 tx=0x7fb124009560 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5ed6e851630243f0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:19:14.052 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:14.044+0000 7fb1137fe640 1 -- 172.21.15.46:0/561912128 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb12402c780 con 0x5619828c06c0 2024-03-31T23:19:14.052 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:14.044+0000 7fb1137fe640 1 -- 172.21.15.46:0/561912128 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb12402c920 con 0x5619828c06c0 2024-03-31T23:19:14.053 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:14.044+0000 7fb1137fe640 1 -- 172.21.15.46:0/561912128 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb124005500 con 0x5619828c06c0 2024-03-31T23:19:14.053 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:14.045+0000 7fb13240d740 1 -- 172.21.15.46:0/561912128 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5619828c06c0 msgr2=0x5619828c0aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:19:14.053 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:14.045+0000 7fb13240d740 1 --2- 172.21.15.46:0/561912128 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5619828c06c0 0x5619828c0aa0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fb124036040 tx=0x7fb124009560 comp rx=0 tx=0).stop 2024-03-31T23:19:14.053 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:14.045+0000 7fb13240d740 1 -- 172.21.15.46:0/561912128 shutdown_connections 2024-03-31T23:19:14.053 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:14.045+0000 7fb13240d740 1 --2- 172.21.15.46:0/561912128 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5619828c9db0 0x5619828cc1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:19:14.053 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:14.045+0000 7fb13240d740 1 --2- 172.21.15.46:0/561912128 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5619828c0fe0 0x5619828c9870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:19:14.053 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:14.045+0000 7fb13240d740 1 --2- 172.21.15.46:0/561912128 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5619828c06c0 0x5619828c0aa0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:19:14.053 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:14.045+0000 7fb13240d740 1 -- 172.21.15.46:0/561912128 >> 172.21.15.46:0/561912128 conn(0x5619828bafe0 msgr2=0x5619828bb3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:19:14.053 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:14.045+0000 7fb13240d740 1 -- 172.21.15.46:0/561912128 shutdown_connections 2024-03-31T23:19:14.053 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:14.045+0000 7fb13240d740 1 -- 172.21.15.46:0/561912128 wait complete. 2024-03-31T23:19:14.053 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:14.045+0000 7fb13240d740 1 -- start start 2024-03-31T23:19:14.053 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:14.045+0000 7fb13240d740 1 Processor -- start 2024-03-31T23:19:14.053 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:14.046+0000 7fb13240d740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5619828c06c0 0x5619828c6e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:19:14.053 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:14.046+0000 7fb13240d740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5619828c0fe0 0x5619828c7350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:19:14.054 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:14.046+0000 7fb13240d740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5619828c9db0 0x5619828c7890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:19:14.054 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:14.046+0000 7fb13240d740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x5619828cc980 con 0x5619828c0fe0 2024-03-31T23:19:14.054 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:14.046+0000 7fb13240d740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x5619827c1f80 con 0x5619828c06c0 2024-03-31T23:19:14.054 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:14.046+0000 7fb13240d740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5619828ccb00 con 0x5619828c9db0 2024-03-31T23:19:14.054 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:14.046+0000 7fb13240d740 10 client.0 _ll_register_callbacks cb 0x56198281a290 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T23:19:14.054 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:14.046+0000 7fb12cdc5640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5619828c0fe0 0x5619828c7350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:19:14.054 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:14.046+0000 7fb12cdc5640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5619828c0fe0 0x5619828c7350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:43516/0 (socket says 172.21.15.46:43516) 2024-03-31T23:19:14.054 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:14.046+0000 7fb12cdc5640 1 -- 172.21.15.46:0/4175271372 learned_addr learned my addr 172.21.15.46:0/4175271372 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:19:14.054 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:14.046+0000 7fb12d5c6640 1 --2- 172.21.15.46:0/4175271372 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5619828c06c0 0x5619828c6e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:19:14.054 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:14.046+0000 7fb12ddc7640 1 --2- 172.21.15.46:0/4175271372 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5619828c9db0 0x5619828c7890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:19:14.054 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:14.046+0000 7fb12cdc5640 1 -- 172.21.15.46:0/4175271372 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5619828c9db0 msgr2=0x5619828c7890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:19:14.054 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:14.046+0000 7fb12cdc5640 1 --2- 172.21.15.46:0/4175271372 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5619828c9db0 0x5619828c7890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:19:14.054 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:14.046+0000 7fb12cdc5640 1 -- 172.21.15.46:0/4175271372 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5619828c06c0 msgr2=0x5619828c6e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:19:14.055 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:14.046+0000 7fb12cdc5640 1 --2- 172.21.15.46:0/4175271372 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5619828c06c0 0x5619828c6e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:19:14.055 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:14.046+0000 7fb12cdc5640 1 -- 172.21.15.46:0/4175271372 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561982604620 con 0x5619828c0fe0 2024-03-31T23:19:14.055 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:14.047+0000 7fb12cdc5640 1 --2- 172.21.15.46:0/4175271372 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5619828c0fe0 0x5619828c7350 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7fb1180042e0 tx=0x7fb118005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c292d913638ff19 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:19:14.055 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:14.047+0000 7fb110ff9640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-31T23:19:14.055 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:14.047+0000 7fb110ff9640 1 -- 172.21.15.46:0/4175271372 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb118005220 con 0x5619828c0fe0 2024-03-31T23:19:14.055 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:14.047+0000 7fb110ff9640 1 -- 172.21.15.46:0/4175271372 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9554+0+0 (secure 0 0 0) 0x7fb1180053c0 con 0x5619828c0fe0 2024-03-31T23:19:14.055 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:14.047+0000 7fb13240d740 1 -- 172.21.15.46:0/4175271372 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb12402c920 con 0x5619828c0fe0 2024-03-31T23:19:14.055 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:14.047+0000 7fb110ff9640 1 -- 172.21.15.46:0/4175271372 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fb118035050 con 0x5619828c0fe0 2024-03-31T23:19:14.055 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:14.047+0000 7fb13240d740 10 client.16758 fetch_fsmap learned FSMap version 31 2024-03-31T23:19:14.055 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:14.047+0000 7fb13240d740 1 -- 172.21.15.46:0/4175271372 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5619828d8880 con 0x5619828c0fe0 2024-03-31T23:19:14.055 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:14.047+0000 7fb110ff9640 1 -- 172.21.15.46:0/4175271372 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb12402c920 con 0x5619828c0fe0 2024-03-31T23:19:14.055 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:14.047+0000 7fb110ff9640 1 -- 172.21.15.46:0/4175271372 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 31) ==== 8350+0+0 (secure 0 0 0) 0x7fb11802a460 con 0x5619828c0fe0 2024-03-31T23:19:14.055 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:14.047+0000 7fb13240d740 10 client.16758 fetch_fsmap finished waiting for FSMap version 31 2024-03-31T23:19:14.055 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:14.047+0000 7fb13240d740 10 client.16758 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-31T23:19:14.055 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:14.047+0000 7fb13240d740 20 client.16758 populate_metadata read hostname 'smithi046' 2024-03-31T23:19:14.056 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:14.047+0000 7fb13240d740 1 --2- 172.21.15.46:0/4175271372 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x5619828d9d40 0x5619828dc1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:19:14.056 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:14.047+0000 7fb13240d740 4 client.16758 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-31T23:19:14.056 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:14.047+0000 7fb13240d740 1 -- 172.21.15.46:0/4175271372 --> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561982837120 con 0x5619828d9d40 2024-03-31T23:19:14.056 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:14.048+0000 7fb12d5c6640 1 --2- 172.21.15.46:0/4175271372 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x5619828d9d40 0x5619828dc1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:19:14.056 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:14.048+0000 7fb12d5c6640 1 --2- 172.21.15.46:0/4175271372 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x5619828d9d40 0x5619828dc1d0 crc :-1 s=READY pgs=141 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a0e8819ffd8d0a7e server_cookie=d18efbdfbe3bd632 in_seq=0 out_seq=0 2024-03-31T23:19:14.056 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:14.048+0000 7fb110ff9640 10 client.16758 ms_handle_connect on v2:172.21.15.142:6838/2519143918 2024-03-31T23:19:14.056 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:14.049+0000 7fb110ff9640 1 -- 172.21.15.46:0/4175271372 <== mds.0 v2:172.21.15.142:6838/2519143918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x561982837120 con 0x5619828d9d40 2024-03-31T23:19:14.056 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:14.049+0000 7fb110ff9640 10 client.16758 handle_command_reply: tid=0 2024-03-31T23:19:14.056 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:14.049+0000 7fb13240d740 1 client.16758 shutdown 2024-03-31T23:19:14.056 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:14.049+0000 7fb13240d740 1 -- 172.21.15.46:0/4175271372 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5619828c0fe0 msgr2=0x5619828c7350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:19:14.056 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:14.049+0000 7fb13240d740 1 --2- 172.21.15.46:0/4175271372 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5619828c0fe0 0x5619828c7350 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7fb1180042e0 tx=0x7fb118005f90 comp rx=0 tx=0).stop 2024-03-31T23:19:14.056 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:14.049+0000 7fb13240d740 1 -- 172.21.15.46:0/4175271372 shutdown_connections 2024-03-31T23:19:14.056 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:14.049+0000 7fb13240d740 1 --2- 172.21.15.46:0/4175271372 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x5619828d9d40 0x5619828dc1d0 crc :-1 s=READY pgs=141 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:19:14.056 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:14.049+0000 7fb13240d740 1 --2- 172.21.15.46:0/4175271372 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5619828c9db0 0x5619828c7890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:19:14.056 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:14.049+0000 7fb110ff9640 0 client.16758 ms_handle_reset on v2:172.21.15.142:6838/2519143918 2024-03-31T23:19:14.057 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:14.049+0000 7fb13240d740 1 --2- 172.21.15.46:0/4175271372 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5619828c0fe0 0x5619828c7350 unknown :-1 s=CLOSED pgs=754 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:19:14.057 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:14.049+0000 7fb13240d740 1 --2- 172.21.15.46:0/4175271372 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5619828c06c0 0x5619828c6e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:19:14.057 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:14.049+0000 7fb13240d740 1 -- 172.21.15.46:0/4175271372 >> 172.21.15.46:0/4175271372 conn(0x5619828bafe0 msgr2=0x5619828bcd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:19:14.057 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:14.049+0000 7fb13240d740 1 -- 172.21.15.46:0/4175271372 shutdown_connections 2024-03-31T23:19:14.057 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:14.049+0000 7fb13240d740 1 -- 172.21.15.46:0/4175271372 wait complete. 2024-03-31T23:19:14.057 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:14.049+0000 7fb13240d740 20 client.16758 trim_cache size 0 max 16384 2024-03-31T23:19:14.066 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:14.064+0000 7fb13240d740 1 Processor -- start 2024-03-31T23:19:14.066 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:14.064+0000 7fb13240d740 1 -- start start 2024-03-31T23:19:14.067 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:14.064+0000 7fb13240d740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5619828c06c0 0x5619828c9540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:19:14.067 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:14.064+0000 7fb13240d740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5619828c0fe0 0x5619828d5720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:19:14.067 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:14.064+0000 7fb13240d740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5619828c9db0 0x5619828b9e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:19:14.067 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:14.064+0000 7fb13240d740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fb1281af290 con 0x5619828c06c0 2024-03-31T23:19:14.067 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:14.064+0000 7fb13240d740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fb1281acb30 con 0x5619828c0fe0 2024-03-31T23:19:14.067 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:14.064+0000 7fb13240d740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fb1281aa9a0 con 0x5619828c9db0 2024-03-31T23:19:14.067 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:14.065+0000 7fb12d5c6640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5619828c06c0 0x5619828c9540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:19:14.067 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:14.065+0000 7fb12d5c6640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5619828c06c0 0x5619828c9540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:43522/0 (socket says 172.21.15.46:43522) 2024-03-31T23:19:14.067 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:14.065+0000 7fb12d5c6640 1 -- 172.21.15.46:0/2524342469 learned_addr learned my addr 172.21.15.46:0/2524342469 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:19:14.067 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:14.065+0000 7fb12cdc5640 1 --2- 172.21.15.46:0/2524342469 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5619828c0fe0 0x5619828d5720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:19:14.067 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:14.065+0000 7fb12ddc7640 1 --2- 172.21.15.46:0/2524342469 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5619828c9db0 0x5619828b9e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:19:14.067 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:14.065+0000 7fb12d5c6640 1 -- 172.21.15.46:0/2524342469 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5619828c9db0 msgr2=0x5619828b9e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:19:14.067 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:14.065+0000 7fb12d5c6640 1 --2- 172.21.15.46:0/2524342469 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5619828c9db0 0x5619828b9e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:19:14.068 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:14.065+0000 7fb12d5c6640 1 -- 172.21.15.46:0/2524342469 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5619828c0fe0 msgr2=0x5619828d5720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:19:14.068 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:14.065+0000 7fb12d5c6640 1 --2- 172.21.15.46:0/2524342469 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5619828c0fe0 0x5619828d5720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:19:14.068 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:14.065+0000 7fb12d5c6640 1 -- 172.21.15.46:0/2524342469 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5619828dc800 con 0x5619828c06c0 2024-03-31T23:19:14.068 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:14.065+0000 7fb12d5c6640 1 --2- 172.21.15.46:0/2524342469 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5619828c06c0 0x5619828c9540 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7fb124036040 tx=0x7fb1240050e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=911e6a65033ed098 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:19:14.068 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:14.065+0000 7fb112ffd640 1 -- 172.21.15.46:0/2524342469 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb124005d50 con 0x5619828c06c0 2024-03-31T23:19:14.068 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:14.065+0000 7fb112ffd640 1 -- 172.21.15.46:0/2524342469 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb12402ca00 con 0x5619828c06c0 2024-03-31T23:19:14.068 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:14.065+0000 7fb112ffd640 1 -- 172.21.15.46:0/2524342469 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb124005650 con 0x5619828c06c0 2024-03-31T23:19:14.071 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:14.066+0000 7fb13240d740 1 -- 172.21.15.46:0/2524342469 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5619828c06c0 msgr2=0x5619828c9540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:19:14.071 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:14.066+0000 7fb13240d740 1 --2- 172.21.15.46:0/2524342469 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5619828c06c0 0x5619828c9540 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7fb124036040 tx=0x7fb1240050e0 comp rx=0 tx=0).stop 2024-03-31T23:19:14.071 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:14.066+0000 7fb13240d740 1 -- 172.21.15.46:0/2524342469 shutdown_connections 2024-03-31T23:19:14.071 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:14.066+0000 7fb13240d740 1 --2- 172.21.15.46:0/2524342469 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5619828c9db0 0x5619828b9e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:19:14.071 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:14.066+0000 7fb13240d740 1 --2- 172.21.15.46:0/2524342469 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5619828c0fe0 0x5619828d5720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:19:14.072 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:14.066+0000 7fb13240d740 1 --2- 172.21.15.46:0/2524342469 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5619828c06c0 0x5619828c9540 unknown :-1 s=CLOSED pgs=755 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:19:14.072 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:14.066+0000 7fb13240d740 1 -- 172.21.15.46:0/2524342469 >> 172.21.15.46:0/2524342469 conn(0x5619828bafe0 msgr2=0x5619828b96f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:19:14.072 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:14.066+0000 7fb13240d740 1 -- 172.21.15.46:0/2524342469 shutdown_connections 2024-03-31T23:19:14.072 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:14.066+0000 7fb13240d740 1 -- 172.21.15.46:0/2524342469 wait complete. 2024-03-31T23:19:14.072 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:14.066+0000 7fb13240d740 1 -- start start 2024-03-31T23:19:14.072 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:14.066+0000 7fb13240d740 1 Processor -- start 2024-03-31T23:19:14.072 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:14.067+0000 7fb13240d740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5619828c06c0 0x5619828c41f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:19:14.072 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:14.067+0000 7fb13240d740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5619828c0fe0 0x5619828c4730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:19:14.072 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:14.067+0000 7fb13240d740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5619828c9db0 0x561982939090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:19:14.072 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:14.067+0000 7fb13240d740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fb1281aaff0 con 0x5619828c06c0 2024-03-31T23:19:14.072 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:14.067+0000 7fb13240d740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fb1281ac460 con 0x5619828c9db0 2024-03-31T23:19:14.072 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:14.067+0000 7fb13240d740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fb1281ab6c0 con 0x5619828c0fe0 2024-03-31T23:19:14.072 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:14.067+0000 7fb13240d740 10 client.0 _ll_register_callbacks cb 0x7fb118024070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T23:19:14.072 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:14.067+0000 7fb12d5c6640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5619828c06c0 0x5619828c41f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:19:14.073 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:14.067+0000 7fb12ddc7640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5619828c9db0 0x561982939090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:19:14.073 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:14.067+0000 7fb12d5c6640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5619828c06c0 0x5619828c41f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:43524/0 (socket says 172.21.15.46:43524) 2024-03-31T23:19:14.073 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:14.067+0000 7fb12d5c6640 1 -- 172.21.15.46:0/2331563257 learned_addr learned my addr 172.21.15.46:0/2331563257 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:19:14.073 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:14.067+0000 7fb12cdc5640 1 --2- 172.21.15.46:0/2331563257 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5619828c0fe0 0x5619828c4730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:19:14.073 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:14.067+0000 7fb12ddc7640 1 -- 172.21.15.46:0/2331563257 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5619828c0fe0 msgr2=0x5619828c4730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:19:14.073 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:14.067+0000 7fb12ddc7640 1 --2- 172.21.15.46:0/2331563257 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5619828c0fe0 0x5619828c4730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:19:14.073 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:14.067+0000 7fb12ddc7640 1 -- 172.21.15.46:0/2331563257 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5619828c06c0 msgr2=0x5619828c41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:19:14.073 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:14.067+0000 7fb12ddc7640 1 --2- 172.21.15.46:0/2331563257 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5619828c06c0 0x5619828c41f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:19:14.073 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:14.067+0000 7fb12ddc7640 1 -- 172.21.15.46:0/2331563257 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561982939680 con 0x5619828c9db0 2024-03-31T23:19:14.073 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:14.067+0000 7fb12d5c6640 1 --2- 172.21.15.46:0/2331563257 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5619828c06c0 0x5619828c41f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:19:14.073 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:14.067+0000 7fb12cdc5640 1 --2- 172.21.15.46:0/2331563257 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5619828c0fe0 0x5619828c4730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:19:14.073 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:14.067+0000 7fb12ddc7640 1 --2- 172.21.15.46:0/2331563257 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5619828c9db0 0x561982939090 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fb1200cc750 tx=0x7fb12009bf50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8e96810e3b8dd5d9 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:19:14.073 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:14.067+0000 7fb110ff9640 10 client.0 ms_handle_connect on v2:172.21.15.142:3300/0 2024-03-31T23:19:14.073 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:14.068+0000 7fb110ff9640 1 -- 172.21.15.46:0/2331563257 <== mon.2 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1200b3020 con 0x5619828c9db0 2024-03-31T23:19:14.074 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:14.068+0000 7fb13240d740 1 -- 172.21.15.46:0/2331563257 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb12402ca00 con 0x5619828c9db0 2024-03-31T23:19:14.074 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:14.068+0000 7fb110ff9640 1 -- 172.21.15.46:0/2331563257 <== mon.2 v2:172.21.15.142:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9554+0+0 (secure 0 0 0) 0x7fb1200cf130 con 0x5619828c9db0 2024-03-31T23:19:14.074 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:14.068+0000 7fb110ff9640 1 -- 172.21.15.46:0/2331563257 <== mon.2 v2:172.21.15.142:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fb12009b780 con 0x5619828c9db0 2024-03-31T23:19:14.074 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:14.068+0000 7fb13240d740 10 client.26606 fetch_fsmap learned FSMap version 31 2024-03-31T23:19:14.074 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:14.068+0000 7fb13240d740 1 -- 172.21.15.46:0/2331563257 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x561982940a70 con 0x5619828c9db0 2024-03-31T23:19:14.074 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:14.068+0000 7fb110ff9640 1 -- 172.21.15.46:0/2331563257 <== mon.2 v2:172.21.15.142:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb12402ca00 con 0x5619828c9db0 2024-03-31T23:19:14.074 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:14.068+0000 7fb110ff9640 1 -- 172.21.15.46:0/2331563257 <== mon.2 v2:172.21.15.142:3300/0 5 ==== fsmap(e 31) ==== 8350+0+0 (secure 0 0 0) 0x7fb120097040 con 0x5619828c9db0 2024-03-31T23:19:14.074 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:14.068+0000 7fb13240d740 10 client.26606 fetch_fsmap finished waiting for FSMap version 31 2024-03-31T23:19:14.074 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:14.068+0000 7fb13240d740 10 client.26606 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-31T23:19:14.074 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:14.068+0000 7fb13240d740 20 client.26606 populate_metadata read hostname 'smithi046' 2024-03-31T23:19:14.074 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:14.068+0000 7fb13240d740 1 --2- 172.21.15.46:0/2331563257 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x561982941190 0x561982943580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:19:14.074 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:14.068+0000 7fb13240d740 4 client.26606 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-31T23:19:14.074 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:14.068+0000 7fb13240d740 1 -- 172.21.15.46:0/2331563257 --> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] -- command(tid 0: {"prefix": "flush journal"}) -- 0x561982830e30 con 0x561982941190 2024-03-31T23:19:14.074 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:14.068+0000 7fb12d5c6640 1 --2- 172.21.15.46:0/2331563257 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x561982941190 0x561982943580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:19:14.075 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:14.069+0000 7fb12d5c6640 1 --2- 172.21.15.46:0/2331563257 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x561982941190 0x561982943580 crc :-1 s=READY pgs=142 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=92998b5df2b5c517 server_cookie=c27c536b56a335e9 in_seq=0 out_seq=0 2024-03-31T23:19:14.075 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:14.069+0000 7fb110ff9640 10 client.26606 ms_handle_connect on v2:172.21.15.142:6838/2519143918 2024-03-31T23:19:14.099 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:14.098+0000 7fb110ff9640 1 -- 172.21.15.46:0/2331563257 <== mds.0 v2:172.21.15.142:6838/2519143918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x561982830e30 con 0x561982941190 2024-03-31T23:19:14.100 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:14.098+0000 7fb110ff9640 10 client.26606 handle_command_reply: tid=0 2024-03-31T23:19:14.100 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:14.098+0000 7fb13240d740 1 client.26606 shutdown 2024-03-31T23:19:14.100 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:14.098+0000 7fb13240d740 1 -- 172.21.15.46:0/2331563257 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5619828c9db0 msgr2=0x561982939090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:19:14.100 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:14.098+0000 7fb13240d740 1 --2- 172.21.15.46:0/2331563257 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5619828c9db0 0x561982939090 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fb1200cc750 tx=0x7fb12009bf50 comp rx=0 tx=0).stop 2024-03-31T23:19:14.100 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:14.098+0000 7fb13240d740 1 -- 172.21.15.46:0/2331563257 shutdown_connections 2024-03-31T23:19:14.100 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:14.098+0000 7fb13240d740 1 --2- 172.21.15.46:0/2331563257 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x561982941190 0x561982943580 crc :-1 s=READY pgs=142 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:19:14.100 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:14.098+0000 7fb13240d740 1 --2- 172.21.15.46:0/2331563257 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5619828c9db0 0x561982939090 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:19:14.100 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:14.098+0000 7fb13240d740 1 --2- 172.21.15.46:0/2331563257 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5619828c0fe0 0x5619828c4730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:19:14.100 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:14.098+0000 7fb110ff9640 0 client.26606 ms_handle_reset on v2:172.21.15.142:6838/2519143918 2024-03-31T23:19:14.100 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:14.098+0000 7fb13240d740 1 --2- 172.21.15.46:0/2331563257 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5619828c06c0 0x5619828c41f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:19:14.100 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:14.098+0000 7fb13240d740 1 -- 172.21.15.46:0/2331563257 >> 172.21.15.46:0/2331563257 conn(0x5619828bafe0 msgr2=0x5619828b9f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:19:14.100 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:14.098+0000 7fb13240d740 1 -- 172.21.15.46:0/2331563257 shutdown_connections 2024-03-31T23:19:14.100 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:14.098+0000 7fb13240d740 1 -- 172.21.15.46:0/2331563257 wait complete. 2024-03-31T23:19:14.101 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:14.098+0000 7fb13240d740 20 client.26606 trim_cache size 0 max 16384 2024-03-31T23:19:14.101 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout:{ 2024-03-31T23:19:14.101 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout: "message": "", 2024-03-31T23:19:14.101 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout: "return_code": 0 2024-03-31T23:19:14.101 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout:} 2024-03-31T23:19:14.102 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:14.099+0000 7fb12f82a640 1 -- 172.21.15.46:0/566099916 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fb1140a26b0 msgr2=0x7fb1140a4b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:19:14.102 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:14.099+0000 7fb12f82a640 1 --2- 172.21.15.46:0/566099916 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fb1140a26b0 0x7fb1140a4b70 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fb128077b70 tx=0x7fb118010d10 comp rx=0 tx=0).stop 2024-03-31T23:19:14.102 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:14.099+0000 7fb12f82a640 1 -- 172.21.15.46:0/566099916 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb128107820 msgr2=0x7fb12819c6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:19:14.102 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:14.099+0000 7fb12f82a640 1 --2- 172.21.15.46:0/566099916 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb128107820 0x7fb12819c6c0 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7fb12000ea60 tx=0x7fb12000ef20 comp rx=0 tx=0).stop 2024-03-31T23:19:14.102 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:14.099+0000 7fb12f82a640 1 -- 172.21.15.46:0/566099916 shutdown_connections 2024-03-31T23:19:14.102 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:14.099+0000 7fb12f82a640 1 --2- 172.21.15.46:0/566099916 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fb1140a26b0 0x7fb1140a4b70 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:19:14.102 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:14.099+0000 7fb12f82a640 1 --2- 172.21.15.46:0/566099916 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb128107820 0x7fb12819c6c0 unknown :-1 s=CLOSED pgs=753 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:19:14.102 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:14.099+0000 7fb12f82a640 1 --2- 172.21.15.46:0/566099916 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb128069770 0x7fb128198340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:19:14.102 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:14.099+0000 7fb12f82a640 1 --2- 172.21.15.46:0/566099916 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb128068dd0 0x7fb128197e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:19:14.103 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:14.099+0000 7fb12f82a640 1 -- 172.21.15.46:0/566099916 >> 172.21.15.46:0/566099916 conn(0x7fb1280774a0 msgr2=0x7fb1280fe8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:19:14.103 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:14.099+0000 7fb12f82a640 1 -- 172.21.15.46:0/566099916 shutdown_connections 2024-03-31T23:19:14.103 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:14.099+0000 7fb12f82a640 1 -- 172.21.15.46:0/566099916 wait complete. 2024-03-31T23:19:14.202 INFO:tasks.workunit.client.0.smithi046.stdout: 1 173817 2.49 MB/sec execute 260 sec latency 16.010 ms 2024-03-31T23:19:14.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:19:14 smithi162 ceph-mon[36237]: pgmap v232: 129 pgs: 129 active+clean; 106 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 594 KiB/s rd, 1.5 MiB/s wr, 163 op/s 2024-03-31T23:19:14.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:19:14 smithi162 ceph-mon[36237]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-03-31T23:19:14.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:19:14 smithi046 ceph-mon[32498]: pgmap v232: 129 pgs: 129 active+clean; 106 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 594 KiB/s rd, 1.5 MiB/s wr, 163 op/s 2024-03-31T23:19:14.862 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:19:14 smithi046 ceph-mon[32498]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-03-31T23:19:14.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:19:14 smithi142 ceph-mon[35271]: pgmap v232: 129 pgs: 129 active+clean; 106 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 594 KiB/s rd, 1.5 MiB/s wr, 163 op/s 2024-03-31T23:19:14.944 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:19:14 smithi142 ceph-mon[35271]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-03-31T23:19:15.202 INFO:tasks.workunit.client.0.smithi046.stdout: 1 173891 2.48 MB/sec execute 261 sec latency 856.974 ms 2024-03-31T23:19:16.202 INFO:tasks.workunit.client.0.smithi046.stdout: 1 173891 2.47 MB/sec execute 262 sec latency 1857.050 ms 2024-03-31T23:19:16.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:19:16 smithi046 ceph-mon[32498]: pgmap v233: 129 pgs: 129 active+clean; 108 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 495 KiB/s rd, 1.6 MiB/s wr, 133 op/s 2024-03-31T23:19:16.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:19:16 smithi142 ceph-mon[35271]: pgmap v233: 129 pgs: 129 active+clean; 108 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 495 KiB/s rd, 1.6 MiB/s wr, 133 op/s 2024-03-31T23:19:17.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:19:16 smithi162 ceph-mon[36237]: pgmap v233: 129 pgs: 129 active+clean; 108 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 495 KiB/s rd, 1.6 MiB/s wr, 133 op/s 2024-03-31T23:19:17.202 INFO:tasks.workunit.client.0.smithi046.stdout: 1 173891 2.46 MB/sec execute 263 sec latency 2857.138 ms 2024-03-31T23:19:18.202 INFO:tasks.workunit.client.0.smithi046.stdout: 1 173958 2.45 MB/sec execute 264 sec latency 3401.094 ms 2024-03-31T23:19:18.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:19:18 smithi046 ceph-mon[32498]: pgmap v234: 129 pgs: 129 active+clean; 108 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 495 KiB/s rd, 1.6 MiB/s wr, 133 op/s 2024-03-31T23:19:18.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:19:18 smithi142 ceph-mon[35271]: pgmap v234: 129 pgs: 129 active+clean; 108 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 495 KiB/s rd, 1.6 MiB/s wr, 133 op/s 2024-03-31T23:19:19.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:19:18 smithi162 ceph-mon[36237]: pgmap v234: 129 pgs: 129 active+clean; 108 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 495 KiB/s rd, 1.6 MiB/s wr, 133 op/s 2024-03-31T23:19:19.202 INFO:tasks.workunit.client.0.smithi046.stdout: 1 173958 2.45 MB/sec execute 265 sec latency 1205.962 ms 2024-03-31T23:19:20.203 INFO:tasks.workunit.client.0.smithi046.stdout: 1 173958 2.44 MB/sec execute 266 sec latency 2206.063 ms 2024-03-31T23:19:20.848 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:19:20 smithi162 ceph-mon[36237]: pgmap v235: 129 pgs: 129 active+clean; 105 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 317 KiB/s rd, 918 KiB/s wr, 87 op/s 2024-03-31T23:19:20.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:19:20 smithi046 ceph-mon[32498]: pgmap v235: 129 pgs: 129 active+clean; 105 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 317 KiB/s rd, 918 KiB/s wr, 87 op/s 2024-03-31T23:19:20.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:19:20 smithi142 ceph-mon[35271]: pgmap v235: 129 pgs: 129 active+clean; 105 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 317 KiB/s rd, 918 KiB/s wr, 87 op/s 2024-03-31T23:19:21.203 INFO:tasks.workunit.client.0.smithi046.stdout: 1 173958 2.43 MB/sec execute 267 sec latency 3206.177 ms 2024-03-31T23:19:22.203 INFO:tasks.workunit.client.0.smithi046.stdout: 1 173958 2.42 MB/sec execute 268 sec latency 4206.309 ms 2024-03-31T23:19:22.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:19:22 smithi046 ceph-mon[32498]: pgmap v236: 129 pgs: 129 active+clean; 105 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 308 KiB/s rd, 890 KiB/s wr, 84 op/s 2024-03-31T23:19:22.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:19:22 smithi142 ceph-mon[35271]: pgmap v236: 129 pgs: 129 active+clean; 105 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 308 KiB/s rd, 890 KiB/s wr, 84 op/s 2024-03-31T23:19:23.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:19:22 smithi162 ceph-mon[36237]: pgmap v236: 129 pgs: 129 active+clean; 105 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 308 KiB/s rd, 890 KiB/s wr, 84 op/s 2024-03-31T23:19:23.203 INFO:tasks.workunit.client.0.smithi046.stdout: 1 174087 2.41 MB/sec execute 269 sec latency 4744.375 ms 2024-03-31T23:19:24.203 INFO:tasks.workunit.client.0.smithi046.stdout: 1 174334 2.41 MB/sec execute 270 sec latency 15.596 ms 2024-03-31T23:19:24.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:19:24 smithi046 ceph-mon[32498]: pgmap v237: 129 pgs: 129 active+clean; 105 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 264 KiB/s rd, 765 KiB/s wr, 72 op/s 2024-03-31T23:19:24.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:19:24 smithi142 ceph-mon[35271]: pgmap v237: 129 pgs: 129 active+clean; 105 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 264 KiB/s rd, 765 KiB/s wr, 72 op/s 2024-03-31T23:19:25.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:19:24 smithi162 ceph-mon[36237]: pgmap v237: 129 pgs: 129 active+clean; 105 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 264 KiB/s rd, 765 KiB/s wr, 72 op/s 2024-03-31T23:19:25.203 INFO:tasks.workunit.client.0.smithi046.stdout: 1 174736 2.40 MB/sec execute 271 sec latency 15.998 ms 2024-03-31T23:19:26.203 INFO:tasks.workunit.client.0.smithi046.stdout: 1 175250 2.41 MB/sec execute 272 sec latency 17.660 ms 2024-03-31T23:19:26.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:19:26 smithi046 ceph-mon[32498]: pgmap v238: 129 pgs: 129 active+clean; 106 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 312 KiB/s rd, 946 KiB/s wr, 89 op/s 2024-03-31T23:19:26.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:19:26 smithi142 ceph-mon[35271]: pgmap v238: 129 pgs: 129 active+clean; 106 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 312 KiB/s rd, 946 KiB/s wr, 89 op/s 2024-03-31T23:19:27.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:19:26 smithi162 ceph-mon[36237]: pgmap v238: 129 pgs: 129 active+clean; 106 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 312 KiB/s rd, 946 KiB/s wr, 89 op/s 2024-03-31T23:19:27.198 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:27.194+0000 7f936307c640 1 -- 172.21.15.46:0/394584808 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f935c1055f0 msgr2=0x7f935c1059d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:19:27.198 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:27.194+0000 7f936307c640 1 --2- 172.21.15.46:0/394584808 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f935c1055f0 0x7f935c1059d0 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7f9350009900 tx=0x7f935002bc30 comp rx=0 tx=0).stop 2024-03-31T23:19:27.198 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:27.194+0000 7f936307c640 1 -- 172.21.15.46:0/394584808 shutdown_connections 2024-03-31T23:19:27.198 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:27.194+0000 7f936307c640 1 --2- 172.21.15.46:0/394584808 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f935c101260 0x7f935c10d480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:19:27.198 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:27.194+0000 7f936307c640 1 --2- 172.21.15.46:0/394584808 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f935c1008a0 0x7f935c100d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:19:27.198 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:27.194+0000 7f936307c640 1 --2- 172.21.15.46:0/394584808 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f935c1055f0 0x7f935c1059d0 unknown :-1 s=CLOSED pgs=756 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:19:27.198 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:27.194+0000 7f936307c640 1 -- 172.21.15.46:0/394584808 >> 172.21.15.46:0/394584808 conn(0x7f935c0774a0 msgr2=0x7f935c0778b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:19:27.198 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:27.194+0000 7f936307c640 1 -- 172.21.15.46:0/394584808 shutdown_connections 2024-03-31T23:19:27.199 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:27.194+0000 7f936307c640 1 -- 172.21.15.46:0/394584808 wait complete. 2024-03-31T23:19:27.199 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:27.195+0000 7f936307c640 1 Processor -- start 2024-03-31T23:19:27.199 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:27.195+0000 7f936307c640 1 -- start start 2024-03-31T23:19:27.199 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:27.195+0000 7f936307c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f935c1008a0 0x7f935c1022c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:19:27.199 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:27.195+0000 7f936307c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f935c101260 0x7f935c102800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:19:27.199 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:27.195+0000 7f936307c640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f935c1055f0 0x7f935c102d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:19:27.199 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:27.195+0000 7f936307c640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f935c1109c0 con 0x7f935c101260 2024-03-31T23:19:27.199 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:27.195+0000 7f936307c640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f935c110840 con 0x7f935c1055f0 2024-03-31T23:19:27.199 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:27.195+0000 7f936307c640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f935c110b40 con 0x7f935c1008a0 2024-03-31T23:19:27.199 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:27.195+0000 7f935bfff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f935c101260 0x7f935c102800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:19:27.199 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:27.195+0000 7f935bfff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f935c101260 0x7f935c102800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:54280/0 (socket says 172.21.15.46:54280) 2024-03-31T23:19:27.200 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:27.195+0000 7f935bfff640 1 -- 172.21.15.46:0/3158722276 learned_addr learned my addr 172.21.15.46:0/3158722276 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:19:27.200 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:27.195+0000 7f9360e18640 1 --2- 172.21.15.46:0/3158722276 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f935c1008a0 0x7f935c1022c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:19:27.200 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:27.195+0000 7f9361619640 1 --2- 172.21.15.46:0/3158722276 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f935c1055f0 0x7f935c102d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:19:27.200 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:27.195+0000 7f935bfff640 1 -- 172.21.15.46:0/3158722276 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f935c1008a0 msgr2=0x7f935c1022c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:19:27.200 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:27.195+0000 7f935bfff640 1 --2- 172.21.15.46:0/3158722276 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f935c1008a0 0x7f935c1022c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:19:27.200 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:27.195+0000 7f935bfff640 1 -- 172.21.15.46:0/3158722276 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f935c1055f0 msgr2=0x7f935c102d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:19:27.200 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:27.195+0000 7f935bfff640 1 --2- 172.21.15.46:0/3158722276 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f935c1055f0 0x7f935c102d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:19:27.200 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:27.195+0000 7f935bfff640 1 -- 172.21.15.46:0/3158722276 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f935c1a63e0 con 0x7f935c101260 2024-03-31T23:19:27.200 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:27.196+0000 7f935bfff640 1 --2- 172.21.15.46:0/3158722276 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f935c101260 0x7f935c102800 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7f934800d950 tx=0x7f934800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:19:27.200 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:27.196+0000 7f9359ffb640 1 -- 172.21.15.46:0/3158722276 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9348017070 con 0x7f935c101260 2024-03-31T23:19:27.201 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:27.196+0000 7f9359ffb640 1 -- 172.21.15.46:0/3158722276 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9348009c30 con 0x7f935c101260 2024-03-31T23:19:27.201 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:27.196+0000 7f9359ffb640 1 -- 172.21.15.46:0/3158722276 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f934800f620 con 0x7f935c101260 2024-03-31T23:19:27.201 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:27.196+0000 7f936307c640 1 -- 172.21.15.46:0/3158722276 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f935c1a66d0 con 0x7f935c101260 2024-03-31T23:19:27.201 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:27.196+0000 7f936307c640 1 -- 172.21.15.46:0/3158722276 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f935c1a6bd0 con 0x7f935c101260 2024-03-31T23:19:27.205 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:27.199+0000 7f9359ffb640 1 -- 172.21.15.46:0/3158722276 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 25) ==== 137451+0+0 (secure 0 0 0) 0x7f93480109a0 con 0x7f935c101260 2024-03-31T23:19:27.205 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:27.199+0000 7f9365c5c740 1 Processor -- start 2024-03-31T23:19:27.205 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:27.199+0000 7f9365c5c740 1 -- start start 2024-03-31T23:19:27.205 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:27.200+0000 7f9359ffb640 1 --2- 172.21.15.46:0/3158722276 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f934c0a2730 0x7f934c0a4bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:19:27.205 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:27.200+0000 7f9365c5c740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555e6eca96c0 0x555e6eca9aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:19:27.205 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:27.200+0000 7f9365c5c740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555e6eca9fe0 0x555e6ecb2870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:19:27.205 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:27.200+0000 7f9365c5c740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x555e6ecb2db0 0x555e6ecb51a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:19:27.206 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:27.200+0000 7f9365c5c740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x555e6ec0c8f0 con 0x555e6eca96c0 2024-03-31T23:19:27.206 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:27.200+0000 7f9365c5c740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x555e6ebb58d0 con 0x555e6ecb2db0 2024-03-31T23:19:27.206 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:27.200+0000 7f9365c5c740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x555e6ec08990 con 0x555e6eca9fe0 2024-03-31T23:19:27.206 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:27.200+0000 7f9360e18640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555e6eca96c0 0x555e6eca9aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:19:27.206 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:27.200+0000 7f9361619640 1 --2- 172.21.15.46:0/3158722276 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f934c0a2730 0x7f934c0a4bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:19:27.206 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:27.200+0000 7f9360e18640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555e6eca96c0 0x555e6eca9aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:54288/0 (socket says 172.21.15.46:54288) 2024-03-31T23:19:27.206 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:27.200+0000 7f9360e18640 1 -- 172.21.15.46:0/1218296741 learned_addr learned my addr 172.21.15.46:0/1218296741 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:19:27.207 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:27.200+0000 7f935bfff640 1 --2- 172.21.15.46:0/1218296741 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555e6eca9fe0 0x555e6ecb2870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:19:27.207 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:27.200+0000 7f9359ffb640 1 -- 172.21.15.46:0/3158722276 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9554+0+0 (secure 0 0 0) 0x7f93480cea90 con 0x7f935c101260 2024-03-31T23:19:27.207 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:27.200+0000 7f9361619640 1 --2- 172.21.15.46:0/1218296741 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x555e6ecb2db0 0x555e6ecb51a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:19:27.207 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:27.200+0000 7f935bfff640 1 -- 172.21.15.46:0/1218296741 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x555e6ecb2db0 msgr2=0x555e6ecb51a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:19:27.207 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:27.200+0000 7f935bfff640 1 --2- 172.21.15.46:0/1218296741 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x555e6ecb2db0 0x555e6ecb51a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:19:27.207 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:27.200+0000 7f935bfff640 1 -- 172.21.15.46:0/1218296741 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555e6eca96c0 msgr2=0x555e6eca9aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:19:27.207 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:27.200+0000 7f935bfff640 1 --2- 172.21.15.46:0/1218296741 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555e6eca96c0 0x555e6eca9aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:19:27.207 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:27.200+0000 7f935bfff640 1 -- 172.21.15.46:0/1218296741 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555e6e921210 con 0x555e6eca9fe0 2024-03-31T23:19:27.207 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:27.200+0000 7f9360e18640 1 --2- 172.21.15.46:0/1218296741 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555e6eca96c0 0x555e6eca9aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:19:27.208 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:27.200+0000 7f9361619640 1 --2- 172.21.15.46:0/3158722276 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f934c0a2730 0x7f934c0a4bf0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f9354005fd0 tx=0x7f9354014b50 comp rx=0 tx=0).ready entity=mgr.15654 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:19:27.208 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:27.200+0000 7f935bfff640 1 --2- 172.21.15.46:0/1218296741 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555e6eca9fe0 0x555e6ecb2870 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f93480cbdd0 tx=0x7f934809ab20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d4340700f6a2f372 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:19:27.208 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:27.201+0000 7f9346ffd640 1 -- 172.21.15.46:0/1218296741 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9348020070 con 0x555e6eca9fe0 2024-03-31T23:19:27.208 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:27.201+0000 7f9346ffd640 1 -- 172.21.15.46:0/1218296741 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f934809aeb0 con 0x555e6eca9fe0 2024-03-31T23:19:27.208 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:27.201+0000 7f9346ffd640 1 -- 172.21.15.46:0/1218296741 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9348097520 con 0x555e6eca9fe0 2024-03-31T23:19:27.208 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:27.201+0000 7f9365c5c740 1 -- 172.21.15.46:0/1218296741 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555e6eca9fe0 msgr2=0x555e6ecb2870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:19:27.208 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:27.201+0000 7f9365c5c740 1 --2- 172.21.15.46:0/1218296741 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555e6eca9fe0 0x555e6ecb2870 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f93480cbdd0 tx=0x7f934809ab20 comp rx=0 tx=0).stop 2024-03-31T23:19:27.208 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:27.201+0000 7f9365c5c740 1 -- 172.21.15.46:0/1218296741 shutdown_connections 2024-03-31T23:19:27.208 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:27.201+0000 7f9365c5c740 1 --2- 172.21.15.46:0/1218296741 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x555e6ecb2db0 0x555e6ecb51a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:19:27.208 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:27.201+0000 7f9365c5c740 1 --2- 172.21.15.46:0/1218296741 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555e6eca9fe0 0x555e6ecb2870 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:19:27.208 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:27.201+0000 7f9365c5c740 1 --2- 172.21.15.46:0/1218296741 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555e6eca96c0 0x555e6eca9aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:19:27.208 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:27.201+0000 7f9365c5c740 1 -- 172.21.15.46:0/1218296741 >> 172.21.15.46:0/1218296741 conn(0x555e6eca3fe0 msgr2=0x555e6eca43c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:19:27.209 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:27.201+0000 7f9365c5c740 1 -- 172.21.15.46:0/1218296741 shutdown_connections 2024-03-31T23:19:27.209 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:27.201+0000 7f9365c5c740 1 -- 172.21.15.46:0/1218296741 wait complete. 2024-03-31T23:19:27.209 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:27.202+0000 7f9365c5c740 1 -- start start 2024-03-31T23:19:27.209 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:27.202+0000 7f9365c5c740 1 Processor -- start 2024-03-31T23:19:27.209 INFO:tasks.workunit.client.0.smithi046.stdout: 1 175696 2.40 MB/sec execute 273 sec latency 21.863 ms 2024-03-31T23:19:27.213 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:27.202+0000 7f9365c5c740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x555e6eca96c0 0x555e6ecafed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:19:27.213 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:27.202+0000 7f9365c5c740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555e6eca9fe0 0x555e6ecb0410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:19:27.213 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:27.202+0000 7f9365c5c740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555e6ecb2db0 0x555e6ecb0950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:19:27.213 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:27.202+0000 7f9365c5c740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x555e6ecb59e0 con 0x555e6eca9fe0 2024-03-31T23:19:27.213 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:27.202+0000 7f9365c5c740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x555e6ebaaf80 con 0x555e6eca96c0 2024-03-31T23:19:27.213 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:27.202+0000 7f9365c5c740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x555e6ecb5b60 con 0x555e6ecb2db0 2024-03-31T23:19:27.213 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:27.202+0000 7f9365c5c740 10 client.0 _ll_register_callbacks cb 0x555e6ec03290 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T23:19:27.213 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:27.202+0000 7f935bfff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555e6eca9fe0 0x555e6ecb0410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:19:27.213 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:27.202+0000 7f935bfff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555e6eca9fe0 0x555e6ecb0410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:54290/0 (socket says 172.21.15.46:54290) 2024-03-31T23:19:27.214 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:27.202+0000 7f935bfff640 1 -- 172.21.15.46:0/1625341522 learned_addr learned my addr 172.21.15.46:0/1625341522 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:19:27.214 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:27.202+0000 7f9360e18640 1 --2- 172.21.15.46:0/1625341522 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x555e6eca96c0 0x555e6ecafed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:19:27.214 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:27.202+0000 7f9361619640 1 --2- 172.21.15.46:0/1625341522 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555e6ecb2db0 0x555e6ecb0950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:19:27.214 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:27.203+0000 7f935bfff640 1 -- 172.21.15.46:0/1625341522 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555e6ecb2db0 msgr2=0x555e6ecb0950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:19:27.214 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:27.203+0000 7f935bfff640 1 --2- 172.21.15.46:0/1625341522 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555e6ecb2db0 0x555e6ecb0950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:19:27.214 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:27.203+0000 7f935bfff640 1 -- 172.21.15.46:0/1625341522 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x555e6eca96c0 msgr2=0x555e6ecafed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:19:27.214 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:27.203+0000 7f935bfff640 1 --2- 172.21.15.46:0/1625341522 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x555e6eca96c0 0x555e6ecafed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:19:27.214 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:27.203+0000 7f935bfff640 1 -- 172.21.15.46:0/1625341522 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555e6e9ed620 con 0x555e6eca9fe0 2024-03-31T23:19:27.214 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:27.203+0000 7f935bfff640 1 --2- 172.21.15.46:0/1625341522 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555e6eca9fe0 0x555e6ecb0410 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7f93480cc3d0 tx=0x7f93480cc280 comp rx=0 tx=0).ready entity=mon.0 client_cookie=573b575b94dc7306 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:19:27.214 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:27.203+0000 7f933bfff640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-31T23:19:27.215 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:27.203+0000 7f933bfff640 1 -- 172.21.15.46:0/1625341522 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9348082030 con 0x555e6eca9fe0 2024-03-31T23:19:27.215 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:27.203+0000 7f933bfff640 1 -- 172.21.15.46:0/1625341522 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9554+0+0 (secure 0 0 0) 0x7f934809e020 con 0x555e6eca9fe0 2024-03-31T23:19:27.215 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:27.203+0000 7f9365c5c740 1 -- 172.21.15.46:0/1625341522 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f934809aeb0 con 0x555e6eca9fe0 2024-03-31T23:19:27.215 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:27.203+0000 7f933bfff640 1 -- 172.21.15.46:0/1625341522 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f9348020070 con 0x555e6eca9fe0 2024-03-31T23:19:27.215 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:27.203+0000 7f9365c5c740 10 client.16776 fetch_fsmap learned FSMap version 31 2024-03-31T23:19:27.215 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:27.203+0000 7f9365c5c740 1 -- 172.21.15.46:0/1625341522 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x555e6ecc19a0 con 0x555e6eca9fe0 2024-03-31T23:19:27.215 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:27.203+0000 7f933bfff640 1 -- 172.21.15.46:0/1625341522 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f934809aeb0 con 0x555e6eca9fe0 2024-03-31T23:19:27.215 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:27.203+0000 7f933bfff640 1 -- 172.21.15.46:0/1625341522 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 31) ==== 8350+0+0 (secure 0 0 0) 0x7f9348096c00 con 0x555e6eca9fe0 2024-03-31T23:19:27.215 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:27.203+0000 7f9365c5c740 10 client.16776 fetch_fsmap finished waiting for FSMap version 31 2024-03-31T23:19:27.215 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:27.203+0000 7f9365c5c740 10 client.16776 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-31T23:19:27.215 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:27.203+0000 7f9365c5c740 20 client.16776 populate_metadata read hostname 'smithi046' 2024-03-31T23:19:27.215 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:27.203+0000 7f9365c5c740 1 --2- 172.21.15.46:0/1625341522 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x555e6ecc2cf0 0x555e6ecc50e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:19:27.216 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:27.203+0000 7f9365c5c740 4 client.16776 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-31T23:19:27.216 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:27.203+0000 7f9365c5c740 1 -- 172.21.15.46:0/1625341522 --> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555e6ec20120 con 0x555e6ecc2cf0 2024-03-31T23:19:27.216 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:27.204+0000 7f9360e18640 1 --2- 172.21.15.46:0/1625341522 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x555e6ecc2cf0 0x555e6ecc50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:19:27.216 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:27.204+0000 7f933bfff640 10 client.16776 ms_handle_connect on v2:172.21.15.142:6838/2519143918 2024-03-31T23:19:27.216 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:27.204+0000 7f9360e18640 1 --2- 172.21.15.46:0/1625341522 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x555e6ecc2cf0 0x555e6ecc50e0 crc :-1 s=READY pgs=143 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3f9b83eef77834b server_cookie=9b764ce0798b8567 in_seq=0 out_seq=0 2024-03-31T23:19:27.216 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:27.205+0000 7f933bfff640 1 -- 172.21.15.46:0/1625341522 <== mds.0 v2:172.21.15.142:6838/2519143918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x555e6ec20120 con 0x555e6ecc2cf0 2024-03-31T23:19:27.216 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:27.205+0000 7f933bfff640 10 client.16776 handle_command_reply: tid=0 2024-03-31T23:19:27.216 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:27.205+0000 7f9365c5c740 1 client.16776 shutdown 2024-03-31T23:19:27.216 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:27.205+0000 7f9365c5c740 1 -- 172.21.15.46:0/1625341522 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555e6eca9fe0 msgr2=0x555e6ecb0410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:19:27.216 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:27.205+0000 7f9365c5c740 1 --2- 172.21.15.46:0/1625341522 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555e6eca9fe0 0x555e6ecb0410 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7f93480cc3d0 tx=0x7f93480cc280 comp rx=0 tx=0).stop 2024-03-31T23:19:27.216 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:27.205+0000 7f9365c5c740 1 -- 172.21.15.46:0/1625341522 shutdown_connections 2024-03-31T23:19:27.216 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:27.205+0000 7f9365c5c740 1 --2- 172.21.15.46:0/1625341522 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x555e6ecc2cf0 0x555e6ecc50e0 crc :-1 s=READY pgs=143 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:19:27.217 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:27.205+0000 7f9365c5c740 1 --2- 172.21.15.46:0/1625341522 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555e6ecb2db0 0x555e6ecb0950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:19:27.217 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:27.205+0000 7f9365c5c740 1 --2- 172.21.15.46:0/1625341522 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555e6eca9fe0 0x555e6ecb0410 unknown :-1 s=CLOSED pgs=758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:19:27.217 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:27.205+0000 7f933bfff640 0 client.16776 ms_handle_reset on v2:172.21.15.142:6838/2519143918 2024-03-31T23:19:27.217 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:27.205+0000 7f9365c5c740 1 --2- 172.21.15.46:0/1625341522 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x555e6eca96c0 0x555e6ecafed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:19:27.217 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:27.205+0000 7f9365c5c740 1 -- 172.21.15.46:0/1625341522 >> 172.21.15.46:0/1625341522 conn(0x555e6eca3fe0 msgr2=0x555e6eca5d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:19:27.217 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:27.205+0000 7f9365c5c740 1 -- 172.21.15.46:0/1625341522 shutdown_connections 2024-03-31T23:19:27.217 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:27.205+0000 7f9365c5c740 1 -- 172.21.15.46:0/1625341522 wait complete. 2024-03-31T23:19:27.217 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:27.205+0000 7f9365c5c740 20 client.16776 trim_cache size 0 max 16384 2024-03-31T23:19:27.226 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:27.220+0000 7f9365c5c740 1 Processor -- start 2024-03-31T23:19:27.226 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:27.220+0000 7f9365c5c740 1 -- start start 2024-03-31T23:19:27.227 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:27.221+0000 7f9365c5c740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555e6eca96c0 0x555e6ecb2600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:19:27.227 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:27.221+0000 7f9365c5c740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x555e6eca9fe0 0x555e6eca2e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:19:27.227 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:27.221+0000 7f9365c5c740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555e6ecb2db0 0x555e6ecc8100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:19:27.227 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:27.221+0000 7f9365c5c740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f935c1b37a0 con 0x555e6eca96c0 2024-03-31T23:19:27.227 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:27.221+0000 7f9365c5c740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f935c1b1040 con 0x555e6eca9fe0 2024-03-31T23:19:27.227 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:27.221+0000 7f9365c5c740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f935c1aeeb0 con 0x555e6ecb2db0 2024-03-31T23:19:27.227 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:27.221+0000 7f9360e18640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555e6eca96c0 0x555e6ecb2600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:19:27.227 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:27.221+0000 7f9360e18640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555e6eca96c0 0x555e6ecb2600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:54300/0 (socket says 172.21.15.46:54300) 2024-03-31T23:19:27.227 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:27.221+0000 7f9360e18640 1 -- 172.21.15.46:0/1925358768 learned_addr learned my addr 172.21.15.46:0/1925358768 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:19:27.227 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:27.221+0000 7f9361619640 1 --2- 172.21.15.46:0/1925358768 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555e6ecb2db0 0x555e6ecc8100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:19:27.227 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:27.221+0000 7f935bfff640 1 --2- 172.21.15.46:0/1925358768 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x555e6eca9fe0 0x555e6eca2e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:19:27.227 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:27.221+0000 7f9360e18640 1 -- 172.21.15.46:0/1925358768 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555e6ecb2db0 msgr2=0x555e6ecc8100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:19:27.227 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:27.221+0000 7f9360e18640 1 --2- 172.21.15.46:0/1925358768 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555e6ecb2db0 0x555e6ecc8100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:19:27.228 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:27.221+0000 7f9360e18640 1 -- 172.21.15.46:0/1925358768 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x555e6eca9fe0 msgr2=0x555e6eca2e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:19:27.228 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:27.221+0000 7f9360e18640 1 --2- 172.21.15.46:0/1925358768 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x555e6eca9fe0 0x555e6eca2e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:19:27.228 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:27.221+0000 7f9360e18640 1 -- 172.21.15.46:0/1925358768 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555e6ec9e170 con 0x555e6eca96c0 2024-03-31T23:19:27.228 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:27.221+0000 7f9360e18640 1 --2- 172.21.15.46:0/1925358768 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555e6eca96c0 0x555e6ecb2600 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7f935002c170 tx=0x7f9350009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=38566e048348d7e3 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:19:27.228 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:27.221+0000 7f93467fc640 1 -- 172.21.15.46:0/1925358768 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f935002cb00 con 0x555e6eca96c0 2024-03-31T23:19:27.228 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:27.221+0000 7f93467fc640 1 -- 172.21.15.46:0/1925358768 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9350005040 con 0x555e6eca96c0 2024-03-31T23:19:27.228 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:27.221+0000 7f93467fc640 1 -- 172.21.15.46:0/1925358768 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9350004920 con 0x555e6eca96c0 2024-03-31T23:19:27.228 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:27.222+0000 7f9365c5c740 1 -- 172.21.15.46:0/1925358768 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555e6eca96c0 msgr2=0x555e6ecb2600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:19:27.228 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:27.222+0000 7f9365c5c740 1 --2- 172.21.15.46:0/1925358768 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555e6eca96c0 0x555e6ecb2600 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7f935002c170 tx=0x7f9350009260 comp rx=0 tx=0).stop 2024-03-31T23:19:27.228 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:27.222+0000 7f9365c5c740 1 -- 172.21.15.46:0/1925358768 shutdown_connections 2024-03-31T23:19:27.228 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:27.222+0000 7f9365c5c740 1 --2- 172.21.15.46:0/1925358768 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555e6ecb2db0 0x555e6ecc8100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:19:27.228 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:27.222+0000 7f9365c5c740 1 --2- 172.21.15.46:0/1925358768 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x555e6eca9fe0 0x555e6eca2e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:19:27.229 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:27.222+0000 7f9365c5c740 1 --2- 172.21.15.46:0/1925358768 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555e6eca96c0 0x555e6ecb2600 unknown :-1 s=CLOSED pgs=759 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:19:27.229 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:27.222+0000 7f9365c5c740 1 -- 172.21.15.46:0/1925358768 >> 172.21.15.46:0/1925358768 conn(0x555e6eca3fe0 msgr2=0x555e6eca26f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:19:27.229 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:27.222+0000 7f9365c5c740 1 -- 172.21.15.46:0/1925358768 shutdown_connections 2024-03-31T23:19:27.229 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:27.222+0000 7f9365c5c740 1 -- 172.21.15.46:0/1925358768 wait complete. 2024-03-31T23:19:27.229 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:27.222+0000 7f9365c5c740 1 -- start start 2024-03-31T23:19:27.229 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:27.222+0000 7f9365c5c740 1 Processor -- start 2024-03-31T23:19:27.229 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:27.223+0000 7f9365c5c740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555e6eca96c0 0x555e6ecc80e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:19:27.229 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:27.223+0000 7f9365c5c740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x555e6eca9fe0 0x555e6ecc64d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:19:27.229 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:27.223+0000 7f9365c5c740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555e6ecb2db0 0x555e6ecc6a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:19:27.229 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:27.223+0000 7f9365c5c740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f935c1af500 con 0x555e6eca96c0 2024-03-31T23:19:27.229 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:27.223+0000 7f9365c5c740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f935c1b0970 con 0x555e6eca9fe0 2024-03-31T23:19:27.229 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:27.223+0000 7f9365c5c740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f935c1afbd0 con 0x555e6ecb2db0 2024-03-31T23:19:27.229 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:27.223+0000 7f9365c5c740 10 client.0 _ll_register_callbacks cb 0x7f9350005b50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T23:19:27.230 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:27.223+0000 7f9360e18640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555e6eca96c0 0x555e6ecc80e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:19:27.230 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:27.223+0000 7f9360e18640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555e6eca96c0 0x555e6ecc80e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:54306/0 (socket says 172.21.15.46:54306) 2024-03-31T23:19:27.230 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:27.223+0000 7f9360e18640 1 -- 172.21.15.46:0/1555769418 learned_addr learned my addr 172.21.15.46:0/1555769418 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:19:27.230 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:27.223+0000 7f9361619640 1 --2- 172.21.15.46:0/1555769418 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555e6ecb2db0 0x555e6ecc6a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:19:27.230 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:27.223+0000 7f935bfff640 1 --2- 172.21.15.46:0/1555769418 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x555e6eca9fe0 0x555e6ecc64d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:19:27.230 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:27.223+0000 7f9360e18640 1 -- 172.21.15.46:0/1555769418 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555e6ecb2db0 msgr2=0x555e6ecc6a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:19:27.230 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:27.223+0000 7f9360e18640 1 --2- 172.21.15.46:0/1555769418 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555e6ecb2db0 0x555e6ecc6a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:19:27.230 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:27.223+0000 7f9360e18640 1 -- 172.21.15.46:0/1555769418 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x555e6eca9fe0 msgr2=0x555e6ecc64d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:19:27.230 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:27.223+0000 7f9360e18640 1 --2- 172.21.15.46:0/1555769418 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x555e6eca9fe0 0x555e6ecc64d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:19:27.230 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:27.223+0000 7f9360e18640 1 -- 172.21.15.46:0/1555769418 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555e6ecc6f50 con 0x555e6eca96c0 2024-03-31T23:19:27.230 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:27.223+0000 7f9360e18640 1 --2- 172.21.15.46:0/1555769418 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555e6eca96c0 0x555e6ecc80e0 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7f9350036040 tx=0x7f935002c3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=19c9bbe6af77e1e1 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:19:27.230 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:27.223+0000 7f933bfff640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-31T23:19:27.230 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:27.223+0000 7f933bfff640 1 -- 172.21.15.46:0/1555769418 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9350004040 con 0x555e6eca96c0 2024-03-31T23:19:27.231 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:27.223+0000 7f9365c5c740 1 -- 172.21.15.46:0/1555769418 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9350005040 con 0x555e6eca96c0 2024-03-31T23:19:27.231 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:27.223+0000 7f933bfff640 1 -- 172.21.15.46:0/1555769418 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9554+0+0 (secure 0 0 0) 0x7f93500041e0 con 0x555e6eca96c0 2024-03-31T23:19:27.231 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:27.224+0000 7f933bfff640 1 -- 172.21.15.46:0/1555769418 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f93500043e0 con 0x555e6eca96c0 2024-03-31T23:19:27.231 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:27.224+0000 7f9365c5c740 10 client.16782 fetch_fsmap learned FSMap version 31 2024-03-31T23:19:27.231 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:27.224+0000 7f9365c5c740 1 -- 172.21.15.46:0/1555769418 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x555e6ed299d0 con 0x555e6eca96c0 2024-03-31T23:19:27.231 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:27.224+0000 7f933bfff640 1 -- 172.21.15.46:0/1555769418 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9350005040 con 0x555e6eca96c0 2024-03-31T23:19:27.231 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:27.224+0000 7f933bfff640 1 -- 172.21.15.46:0/1555769418 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 31) ==== 8350+0+0 (secure 0 0 0) 0x7f935005c030 con 0x555e6eca96c0 2024-03-31T23:19:27.231 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:27.224+0000 7f9365c5c740 10 client.16782 fetch_fsmap finished waiting for FSMap version 31 2024-03-31T23:19:27.231 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:27.224+0000 7f9365c5c740 10 client.16782 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-31T23:19:27.231 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:27.224+0000 7f9365c5c740 20 client.16782 populate_metadata read hostname 'smithi046' 2024-03-31T23:19:27.231 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:27.224+0000 7f9365c5c740 1 --2- 172.21.15.46:0/1555769418 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x555e6ed2a100 0x555e6ed2c520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:19:27.231 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:27.224+0000 7f9365c5c740 4 client.16782 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-31T23:19:27.231 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:27.224+0000 7f9365c5c740 1 -- 172.21.15.46:0/1555769418 --> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] -- command(tid 0: {"prefix": "flush journal"}) -- 0x555e6ec19e30 con 0x555e6ed2a100 2024-03-31T23:19:27.231 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:27.224+0000 7f935bfff640 1 --2- 172.21.15.46:0/1555769418 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x555e6ed2a100 0x555e6ed2c520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:19:27.232 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:27.225+0000 7f933bfff640 10 client.16782 ms_handle_connect on v2:172.21.15.142:6838/2519143918 2024-03-31T23:19:27.232 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:27.225+0000 7f935bfff640 1 --2- 172.21.15.46:0/1555769418 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x555e6ed2a100 0x555e6ed2c520 crc :-1 s=READY pgs=144 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b807e3143b7deb63 server_cookie=2f642bfef578e89f in_seq=0 out_seq=0 2024-03-31T23:19:27.251 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:27.248+0000 7f933bfff640 1 -- 172.21.15.46:0/1555769418 <== mds.0 v2:172.21.15.142:6838/2519143918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x555e6ec19e30 con 0x555e6ed2a100 2024-03-31T23:19:27.251 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:27.248+0000 7f933bfff640 10 client.16782 handle_command_reply: tid=0 2024-03-31T23:19:27.251 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:27.248+0000 7f9365c5c740 1 client.16782 shutdown 2024-03-31T23:19:27.251 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:27.249+0000 7f9365c5c740 1 -- 172.21.15.46:0/1555769418 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555e6eca96c0 msgr2=0x555e6ecc80e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:19:27.251 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:27.249+0000 7f9365c5c740 1 --2- 172.21.15.46:0/1555769418 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555e6eca96c0 0x555e6ecc80e0 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7f9350036040 tx=0x7f935002c3e0 comp rx=0 tx=0).stop 2024-03-31T23:19:27.252 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:27.249+0000 7f9365c5c740 1 -- 172.21.15.46:0/1555769418 shutdown_connections 2024-03-31T23:19:27.252 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:27.249+0000 7f9365c5c740 1 --2- 172.21.15.46:0/1555769418 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x555e6ed2a100 0x555e6ed2c520 crc :-1 s=READY pgs=144 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:19:27.252 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:27.249+0000 7f9365c5c740 1 --2- 172.21.15.46:0/1555769418 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555e6ecb2db0 0x555e6ecc6a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:19:27.252 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:27.249+0000 7f9365c5c740 1 --2- 172.21.15.46:0/1555769418 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x555e6eca9fe0 0x555e6ecc64d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:19:27.252 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:27.249+0000 7f933bfff640 0 client.16782 ms_handle_reset on v2:172.21.15.142:6838/2519143918 2024-03-31T23:19:27.252 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:27.249+0000 7f9365c5c740 1 --2- 172.21.15.46:0/1555769418 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555e6eca96c0 0x555e6ecc80e0 unknown :-1 s=CLOSED pgs=760 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:19:27.252 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:27.249+0000 7f9365c5c740 1 -- 172.21.15.46:0/1555769418 >> 172.21.15.46:0/1555769418 conn(0x555e6eca3fe0 msgr2=0x555e6ecbe640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:19:27.252 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:27.249+0000 7f9365c5c740 1 -- 172.21.15.46:0/1555769418 shutdown_connections 2024-03-31T23:19:27.252 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:27.249+0000 7f9365c5c740 1 -- 172.21.15.46:0/1555769418 wait complete. 2024-03-31T23:19:27.252 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:27.249+0000 7f9365c5c740 20 client.16782 trim_cache size 0 max 16384 2024-03-31T23:19:27.252 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout:{ 2024-03-31T23:19:27.253 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout: "message": "", 2024-03-31T23:19:27.253 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout: "return_code": 0 2024-03-31T23:19:27.253 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout:} 2024-03-31T23:19:27.254 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:27.250+0000 7f936307c640 1 -- 172.21.15.46:0/3158722276 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f934c0a2730 msgr2=0x7f934c0a4bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:19:27.254 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:27.250+0000 7f936307c640 1 --2- 172.21.15.46:0/3158722276 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f934c0a2730 0x7f934c0a4bf0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f9354005fd0 tx=0x7f9354014b50 comp rx=0 tx=0).stop 2024-03-31T23:19:27.254 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:27.250+0000 7f936307c640 1 -- 172.21.15.46:0/3158722276 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f935c101260 msgr2=0x7f935c102800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:19:27.254 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:27.250+0000 7f936307c640 1 --2- 172.21.15.46:0/3158722276 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f935c101260 0x7f935c102800 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7f934800d950 tx=0x7f934800de10 comp rx=0 tx=0).stop 2024-03-31T23:19:27.254 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:27.250+0000 7f936307c640 1 -- 172.21.15.46:0/3158722276 shutdown_connections 2024-03-31T23:19:27.254 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:27.250+0000 7f936307c640 1 --2- 172.21.15.46:0/3158722276 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f934c0a2730 0x7f934c0a4bf0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:19:27.254 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:27.250+0000 7f936307c640 1 --2- 172.21.15.46:0/3158722276 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f935c1055f0 0x7f935c102d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:19:27.254 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:27.250+0000 7f936307c640 1 --2- 172.21.15.46:0/3158722276 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f935c101260 0x7f935c102800 unknown :-1 s=CLOSED pgs=757 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:19:27.254 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:27.250+0000 7f936307c640 1 --2- 172.21.15.46:0/3158722276 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f935c1008a0 0x7f935c1022c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:19:27.254 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:27.250+0000 7f936307c640 1 -- 172.21.15.46:0/3158722276 >> 172.21.15.46:0/3158722276 conn(0x7f935c0774a0 msgr2=0x7f935c0690b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:19:27.254 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:27.250+0000 7f936307c640 1 -- 172.21.15.46:0/3158722276 shutdown_connections 2024-03-31T23:19:27.254 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:27.250+0000 7f936307c640 1 -- 172.21.15.46:0/3158722276 wait complete. 2024-03-31T23:19:28.203 INFO:tasks.workunit.client.0.smithi046.stdout: 1 176215 2.40 MB/sec execute 274 sec latency 22.355 ms 2024-03-31T23:19:28.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:19:28 smithi046 ceph-mon[32498]: pgmap v239: 129 pgs: 129 active+clean; 106 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 127 KiB/s rd, 349 KiB/s wr, 37 op/s 2024-03-31T23:19:28.862 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:19:28 smithi046 ceph-mon[32498]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-03-31T23:19:28.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:19:28 smithi142 ceph-mon[35271]: pgmap v239: 129 pgs: 129 active+clean; 106 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 127 KiB/s rd, 349 KiB/s wr, 37 op/s 2024-03-31T23:19:28.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:19:28 smithi142 ceph-mon[35271]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-03-31T23:19:29.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:19:28 smithi162 ceph-mon[36237]: pgmap v239: 129 pgs: 129 active+clean; 106 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 127 KiB/s rd, 349 KiB/s wr, 37 op/s 2024-03-31T23:19:29.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:19:28 smithi162 ceph-mon[36237]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-03-31T23:19:29.203 INFO:tasks.workunit.client.0.smithi046.stdout: 1 176812 2.41 MB/sec execute 275 sec latency 16.956 ms 2024-03-31T23:19:30.203 INFO:tasks.workunit.client.0.smithi046.stdout: 1 177216 2.41 MB/sec execute 276 sec latency 17.202 ms 2024-03-31T23:19:30.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:19:30 smithi046 ceph-mon[32498]: pgmap v240: 129 pgs: 129 active+clean; 103 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 343 KiB/s rd, 891 KiB/s wr, 96 op/s 2024-03-31T23:19:30.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:19:30 smithi142 ceph-mon[35271]: pgmap v240: 129 pgs: 129 active+clean; 103 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 343 KiB/s rd, 891 KiB/s wr, 96 op/s 2024-03-31T23:19:31.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:19:30 smithi162 ceph-mon[36237]: pgmap v240: 129 pgs: 129 active+clean; 103 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 343 KiB/s rd, 891 KiB/s wr, 96 op/s 2024-03-31T23:19:31.203 INFO:tasks.workunit.client.0.smithi046.stdout: 1 177436 2.41 MB/sec execute 277 sec latency 709.701 ms 2024-03-31T23:19:32.203 INFO:tasks.workunit.client.0.smithi046.stdout: 1 177436 2.40 MB/sec execute 278 sec latency 1709.815 ms 2024-03-31T23:19:32.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:19:32 smithi046 ceph-mon[32498]: pgmap v241: 129 pgs: 129 active+clean; 103 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 263 KiB/s rd, 724 KiB/s wr, 76 op/s 2024-03-31T23:19:32.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:19:32 smithi046 ceph-mon[32498]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-03-31T23:19:32.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:19:32 smithi142 ceph-mon[35271]: pgmap v241: 129 pgs: 129 active+clean; 103 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 263 KiB/s rd, 724 KiB/s wr, 76 op/s 2024-03-31T23:19:32.944 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:19:32 smithi142 ceph-mon[35271]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-03-31T23:19:33.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:19:32 smithi162 ceph-mon[36237]: pgmap v241: 129 pgs: 129 active+clean; 103 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 263 KiB/s rd, 724 KiB/s wr, 76 op/s 2024-03-31T23:19:33.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:19:32 smithi162 ceph-mon[36237]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-03-31T23:19:33.203 INFO:tasks.workunit.client.0.smithi046.stdout: 1 177503 2.39 MB/sec execute 279 sec latency 2246.856 ms 2024-03-31T23:19:33.862 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:19:33 smithi046 ceph-mon[32498]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-03-31T23:19:33.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:19:33 smithi142 ceph-mon[35271]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-03-31T23:19:34.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:19:33 smithi162 ceph-mon[36237]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-03-31T23:19:34.203 INFO:tasks.workunit.client.0.smithi046.stdout: 1 177503 2.38 MB/sec execute 280 sec latency 1229.355 ms 2024-03-31T23:19:34.862 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:19:34 smithi046 ceph-mon[32498]: pgmap v242: 129 pgs: 129 active+clean; 103 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 263 KiB/s rd, 724 KiB/s wr, 76 op/s 2024-03-31T23:19:34.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:19:34 smithi142 ceph-mon[35271]: pgmap v242: 129 pgs: 129 active+clean; 103 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 263 KiB/s rd, 724 KiB/s wr, 76 op/s 2024-03-31T23:19:35.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:19:34 smithi162 ceph-mon[36237]: pgmap v242: 129 pgs: 129 active+clean; 103 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 263 KiB/s rd, 724 KiB/s wr, 76 op/s 2024-03-31T23:19:35.203 INFO:tasks.workunit.client.0.smithi046.stdout: 1 177503 2.37 MB/sec execute 281 sec latency 2229.433 ms 2024-03-31T23:19:36.204 INFO:tasks.workunit.client.0.smithi046.stdout: 1 177503 2.37 MB/sec execute 282 sec latency 3229.550 ms 2024-03-31T23:19:36.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:19:36 smithi046 ceph-mon[32498]: pgmap v243: 129 pgs: 129 active+clean; 105 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 380 KiB/s rd, 1.0 MiB/s wr, 102 op/s 2024-03-31T23:19:36.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:19:36 smithi142 ceph-mon[35271]: pgmap v243: 129 pgs: 129 active+clean; 105 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 380 KiB/s rd, 1.0 MiB/s wr, 102 op/s 2024-03-31T23:19:37.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:19:36 smithi162 ceph-mon[36237]: pgmap v243: 129 pgs: 129 active+clean; 105 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 380 KiB/s rd, 1.0 MiB/s wr, 102 op/s 2024-03-31T23:19:37.204 INFO:tasks.workunit.client.0.smithi046.stdout: 1 177503 2.36 MB/sec execute 283 sec latency 4229.660 ms 2024-03-31T23:19:37.862 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:19:37 smithi046 ceph-mon[32498]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-03-31T23:19:37.944 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:19:37 smithi142 ceph-mon[35271]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-03-31T23:19:38.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:19:37 smithi162 ceph-mon[36237]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-03-31T23:19:38.204 INFO:tasks.workunit.client.0.smithi046.stdout: 1 177623 2.35 MB/sec execute 284 sec latency 4766.220 ms 2024-03-31T23:19:38.862 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:19:38 smithi046 ceph-mon[32498]: pgmap v244: 129 pgs: 129 active+clean; 105 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 333 KiB/s rd, 862 KiB/s wr, 85 op/s 2024-03-31T23:19:38.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:19:38 smithi142 ceph-mon[35271]: pgmap v244: 129 pgs: 129 active+clean; 105 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 333 KiB/s rd, 862 KiB/s wr, 85 op/s 2024-03-31T23:19:39.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:19:38 smithi162 ceph-mon[36237]: pgmap v244: 129 pgs: 129 active+clean; 105 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 333 KiB/s rd, 862 KiB/s wr, 85 op/s 2024-03-31T23:19:39.204 INFO:tasks.workunit.client.0.smithi046.stdout: 1 177890 2.35 MB/sec execute 285 sec latency 15.468 ms 2024-03-31T23:19:40.204 INFO:tasks.workunit.client.0.smithi046.stdout: 1 178288 2.34 MB/sec execute 286 sec latency 17.884 ms 2024-03-31T23:19:40.346 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:40.342+0000 7f71d6030640 1 -- 172.21.15.46:0/4264778189 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f71d01020c0 msgr2=0x7f71d010e5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:19:40.346 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:40.342+0000 7f71d6030640 1 --2- 172.21.15.46:0/4264778189 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f71d01020c0 0x7f71d010e5d0 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7f71c0009930 tx=0x7f71c002bc30 comp rx=0 tx=0).stop 2024-03-31T23:19:40.346 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:40.343+0000 7f71d6030640 1 -- 172.21.15.46:0/4264778189 shutdown_connections 2024-03-31T23:19:40.346 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:40.343+0000 7f71d6030640 1 --2- 172.21.15.46:0/4264778189 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f71d01020c0 0x7f71d010e5d0 unknown :-1 s=CLOSED pgs=761 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:19:40.346 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:40.343+0000 7f71d6030640 1 --2- 172.21.15.46:0/4264778189 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f71d0101700 0x7f71d0101b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:19:40.346 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:40.343+0000 7f71d6030640 1 --2- 172.21.15.46:0/4264778189 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f71d0107710 0x7f71d0107af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:19:40.346 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:40.343+0000 7f71d6030640 1 -- 172.21.15.46:0/4264778189 >> 172.21.15.46:0/4264778189 conn(0x7f71d00fd210 msgr2=0x7f71d00ff630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:19:40.346 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:40.343+0000 7f71d6030640 1 -- 172.21.15.46:0/4264778189 shutdown_connections 2024-03-31T23:19:40.347 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:40.343+0000 7f71d6030640 1 -- 172.21.15.46:0/4264778189 wait complete. 2024-03-31T23:19:40.347 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:40.343+0000 7f71d6030640 1 Processor -- start 2024-03-31T23:19:40.347 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:40.343+0000 7f71d6030640 1 -- start start 2024-03-31T23:19:40.347 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:40.343+0000 7f71d6030640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f71d0101700 0x7f71d0073a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:19:40.347 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:40.343+0000 7f71d6030640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f71d01020c0 0x7f71d0073f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:19:40.347 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:40.343+0000 7f71d6030640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f71d0107710 0x7f71d00744d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:19:40.347 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:40.343+0000 7f71d6030640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f71d0111990 con 0x7f71d0107710 2024-03-31T23:19:40.347 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:40.343+0000 7f71d6030640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f71d0111810 con 0x7f71d0101700 2024-03-31T23:19:40.347 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:40.343+0000 7f71d6030640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f71d0111b10 con 0x7f71d01020c0 2024-03-31T23:19:40.347 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:40.344+0000 7f71cffff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f71d0107710 0x7f71d00744d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:19:40.347 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:40.344+0000 7f71ceffd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f71d01020c0 0x7f71d0073f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:19:40.347 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:40.344+0000 7f71cffff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f71d0107710 0x7f71d00744d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:33900/0 (socket says 172.21.15.46:33900) 2024-03-31T23:19:40.347 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:40.344+0000 7f71cffff640 1 -- 172.21.15.46:0/645228776 learned_addr learned my addr 172.21.15.46:0/645228776 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:19:40.347 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:40.344+0000 7f71cf7fe640 1 --2- 172.21.15.46:0/645228776 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f71d0101700 0x7f71d0073a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:19:40.348 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:40.344+0000 7f71cffff640 1 -- 172.21.15.46:0/645228776 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f71d01020c0 msgr2=0x7f71d0073f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:19:40.348 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:40.344+0000 7f71cffff640 1 --2- 172.21.15.46:0/645228776 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f71d01020c0 0x7f71d0073f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:19:40.348 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:40.344+0000 7f71cffff640 1 -- 172.21.15.46:0/645228776 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f71d0101700 msgr2=0x7f71d0073a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:19:40.348 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:40.344+0000 7f71cffff640 1 --2- 172.21.15.46:0/645228776 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f71d0101700 0x7f71d0073a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:19:40.348 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:40.344+0000 7f71cffff640 1 -- 172.21.15.46:0/645228776 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f71d01a6350 con 0x7f71d0107710 2024-03-31T23:19:40.348 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:40.344+0000 7f71ceffd640 1 --2- 172.21.15.46:0/645228776 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f71d01020c0 0x7f71d0073f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:19:40.348 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:40.344+0000 7f71cffff640 1 --2- 172.21.15.46:0/645228776 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f71d0107710 0x7f71d00744d0 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7f71c0005940 tx=0x7f71c00045e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:19:40.348 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:40.344+0000 7f71ccff9640 1 -- 172.21.15.46:0/645228776 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71c0043020 con 0x7f71d0107710 2024-03-31T23:19:40.348 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:40.344+0000 7f71d6030640 1 -- 172.21.15.46:0/645228776 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f71d01a65e0 con 0x7f71d0107710 2024-03-31T23:19:40.348 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:40.344+0000 7f71ccff9640 1 -- 172.21.15.46:0/645228776 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f71c002c9f0 con 0x7f71d0107710 2024-03-31T23:19:40.348 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:40.344+0000 7f71ccff9640 1 -- 172.21.15.46:0/645228776 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71c0003a10 con 0x7f71d0107710 2024-03-31T23:19:40.348 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:40.344+0000 7f71d6030640 1 -- 172.21.15.46:0/645228776 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f71d01a6b30 con 0x7f71d0107710 2024-03-31T23:19:40.355 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:40.347+0000 7f71ccff9640 1 -- 172.21.15.46:0/645228776 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 25) ==== 137451+0+0 (secure 0 0 0) 0x7f71c0046020 con 0x7f71d0107710 2024-03-31T23:19:40.355 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:40.347+0000 7f71d8c0c740 1 Processor -- start 2024-03-31T23:19:40.355 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:40.347+0000 7f71d8c0c740 1 -- start start 2024-03-31T23:19:40.355 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:40.347+0000 7f71d8c0c740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55db0fed26c0 0x55db0fed2aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:19:40.355 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:40.347+0000 7f71d8c0c740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55db0fed2fe0 0x55db0fedb870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:19:40.355 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:40.347+0000 7f71d8c0c740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55db0fedbdb0 0x55db0fede1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:19:40.355 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:40.347+0000 7f71d8c0c740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55db0fe358f0 con 0x55db0fedbdb0 2024-03-31T23:19:40.356 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:40.347+0000 7f71d8c0c740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55db0fdde8d0 con 0x55db0fed2fe0 2024-03-31T23:19:40.356 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:40.347+0000 7f71d8c0c740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55db0fe31990 con 0x55db0fed26c0 2024-03-31T23:19:40.356 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:40.347+0000 7f71ccff9640 1 --2- 172.21.15.46:0/645228776 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f71bc0a26b0 0x7f71bc0a4b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:19:40.356 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:40.347+0000 7f71ceffd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55db0fed26c0 0x55db0fed2aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:19:40.356 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:40.347+0000 7f71ceffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55db0fedbdb0 0x55db0fede1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:19:40.356 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:40.347+0000 7f71cf7fe640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55db0fed2fe0 0x55db0fedb870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:19:40.356 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:40.347+0000 7f71cf7fe640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55db0fed2fe0 0x55db0fedb870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.46:58918/0 (socket says 172.21.15.46:58918) 2024-03-31T23:19:40.356 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:40.347+0000 7f71ceffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55db0fedbdb0 0x55db0fede1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:33914/0 (socket says 172.21.15.46:33914) 2024-03-31T23:19:40.356 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:40.347+0000 7f71cf7fe640 1 -- 172.21.15.46:0/168362196 learned_addr learned my addr 172.21.15.46:0/168362196 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:19:40.356 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:40.348+0000 7f71ccff9640 1 -- 172.21.15.46:0/645228776 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9554+0+0 (secure 0 0 0) 0x7f71c00f0e20 con 0x7f71d0107710 2024-03-31T23:19:40.356 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:40.348+0000 7f71cf7fe640 1 -- 172.21.15.46:0/168362196 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55db0fed26c0 msgr2=0x55db0fed2aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:19:40.356 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:40.348+0000 7f71cf7fe640 1 --2- 172.21.15.46:0/168362196 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55db0fed26c0 0x55db0fed2aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:19:40.356 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:40.348+0000 7f71cf7fe640 1 -- 172.21.15.46:0/168362196 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55db0fedbdb0 msgr2=0x55db0fede1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:19:40.356 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:40.348+0000 7f71cf7fe640 1 --2- 172.21.15.46:0/168362196 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55db0fedbdb0 0x55db0fede1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:19:40.357 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:40.348+0000 7f71cf7fe640 1 -- 172.21.15.46:0/168362196 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55db0fb4a210 con 0x55db0fed2fe0 2024-03-31T23:19:40.357 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:40.348+0000 7f71ceffd640 1 --2- 172.21.15.46:0/168362196 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55db0fedbdb0 0x55db0fede1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:19:40.357 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:40.348+0000 7f71cffff640 1 --2- 172.21.15.46:0/645228776 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f71bc0a26b0 0x7f71bc0a4b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:19:40.357 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:40.348+0000 7f71cf7fe640 1 --2- 172.21.15.46:0/168362196 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55db0fed2fe0 0x55db0fedb870 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f71c8009fd0 tx=0x7f71c80118d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dab55d0a070c73b8 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:19:40.357 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:40.348+0000 7f71c5ffb640 1 -- 172.21.15.46:0/168362196 <== mon.2 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71c801e070 con 0x55db0fed2fe0 2024-03-31T23:19:40.357 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:40.348+0000 7f71cffff640 1 --2- 172.21.15.46:0/645228776 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f71bc0a26b0 0x7f71bc0a4b70 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f71c00b9c70 tx=0x7f71c00dd040 comp rx=0 tx=0).ready entity=mgr.15654 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:19:40.357 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:40.348+0000 7f71c5ffb640 1 -- 172.21.15.46:0/168362196 <== mon.2 v2:172.21.15.142:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f71c8003c30 con 0x55db0fed2fe0 2024-03-31T23:19:40.357 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:40.348+0000 7f71c5ffb640 1 -- 172.21.15.46:0/168362196 <== mon.2 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71c800f980 con 0x55db0fed2fe0 2024-03-31T23:19:40.357 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:40.349+0000 7f71d8c0c740 1 -- 172.21.15.46:0/168362196 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55db0fed2fe0 msgr2=0x55db0fedb870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:19:40.357 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:40.349+0000 7f71d8c0c740 1 --2- 172.21.15.46:0/168362196 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55db0fed2fe0 0x55db0fedb870 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f71c8009fd0 tx=0x7f71c80118d0 comp rx=0 tx=0).stop 2024-03-31T23:19:40.357 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:40.349+0000 7f71d8c0c740 1 -- 172.21.15.46:0/168362196 shutdown_connections 2024-03-31T23:19:40.357 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:40.349+0000 7f71d8c0c740 1 --2- 172.21.15.46:0/168362196 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55db0fedbdb0 0x55db0fede1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:19:40.357 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:40.349+0000 7f71d8c0c740 1 --2- 172.21.15.46:0/168362196 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55db0fed2fe0 0x55db0fedb870 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:19:40.357 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:40.349+0000 7f71d8c0c740 1 --2- 172.21.15.46:0/168362196 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55db0fed26c0 0x55db0fed2aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:19:40.357 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:40.349+0000 7f71d8c0c740 1 -- 172.21.15.46:0/168362196 >> 172.21.15.46:0/168362196 conn(0x55db0feccfe0 msgr2=0x55db0fecd3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:19:40.358 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:40.349+0000 7f71d8c0c740 1 -- 172.21.15.46:0/168362196 shutdown_connections 2024-03-31T23:19:40.358 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:40.349+0000 7f71d8c0c740 1 -- 172.21.15.46:0/168362196 wait complete. 2024-03-31T23:19:40.358 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:40.349+0000 7f71d8c0c740 1 -- start start 2024-03-31T23:19:40.358 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:40.350+0000 7f71d8c0c740 1 Processor -- start 2024-03-31T23:19:40.358 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:40.350+0000 7f71d8c0c740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55db0fed26c0 0x55db0fed8ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:19:40.358 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:40.350+0000 7f71d8c0c740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55db0fed2fe0 0x55db0fed9430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:19:40.358 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:40.350+0000 7f71d8c0c740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55db0fedbdb0 0x55db0fed9970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:19:40.358 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:40.350+0000 7f71d8c0c740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55db0fede9e0 con 0x55db0fed26c0 2024-03-31T23:19:40.358 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:40.350+0000 7f71d8c0c740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55db0fdd3f80 con 0x55db0fed2fe0 2024-03-31T23:19:40.358 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:40.350+0000 7f71d8c0c740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55db0fedeb60 con 0x55db0fedbdb0 2024-03-31T23:19:40.358 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:40.350+0000 7f71d8c0c740 10 client.0 _ll_register_callbacks cb 0x55db0fe2c290 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T23:19:40.358 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:40.350+0000 7f71ceffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55db0fed26c0 0x55db0fed8ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:19:40.358 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:40.350+0000 7f71cf7fe640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55db0fed2fe0 0x55db0fed9430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:19:40.358 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:40.350+0000 7f71ceffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55db0fed26c0 0x55db0fed8ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:33930/0 (socket says 172.21.15.46:33930) 2024-03-31T23:19:40.359 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:40.350+0000 7f71ceffd640 1 -- 172.21.15.46:0/3813005020 learned_addr learned my addr 172.21.15.46:0/3813005020 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:19:40.359 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:40.350+0000 7f71ceffd640 1 --2- 172.21.15.46:0/3813005020 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55db0fedbdb0 0x55db0fed9970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:19:40.359 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:40.350+0000 7f71cf7fe640 1 -- 172.21.15.46:0/3813005020 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55db0fedbdb0 msgr2=0x55db0fed9970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:19:40.359 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:40.350+0000 7f71cf7fe640 1 --2- 172.21.15.46:0/3813005020 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55db0fedbdb0 0x55db0fed9970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:19:40.359 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:40.350+0000 7f71cf7fe640 1 -- 172.21.15.46:0/3813005020 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55db0fed26c0 msgr2=0x55db0fed8ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:19:40.359 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:40.350+0000 7f71cf7fe640 1 --2- 172.21.15.46:0/3813005020 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55db0fed26c0 0x55db0fed8ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:19:40.359 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:40.350+0000 7f71cf7fe640 1 -- 172.21.15.46:0/3813005020 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55db0fc16620 con 0x55db0fed2fe0 2024-03-31T23:19:40.359 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:40.350+0000 7f71ceffd640 1 --2- 172.21.15.46:0/3813005020 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55db0fed26c0 0x55db0fed8ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:19:40.359 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:40.351+0000 7f71cf7fe640 1 --2- 172.21.15.46:0/3813005020 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55db0fed2fe0 0x55db0fed9430 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f71c80049d0 tx=0x7f71c800f580 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7d1c0ff10646b73d server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:19:40.359 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:40.351+0000 7f71aeffd640 10 client.0 ms_handle_connect on v2:172.21.15.142:3300/0 2024-03-31T23:19:40.359 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:40.351+0000 7f71aeffd640 1 -- 172.21.15.46:0/3813005020 <== mon.2 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71c801e040 con 0x55db0fed2fe0 2024-03-31T23:19:40.359 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:40.351+0000 7f71d8c0c740 1 -- 172.21.15.46:0/3813005020 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f71c8003c30 con 0x55db0fed2fe0 2024-03-31T23:19:40.359 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:40.351+0000 7f71aeffd640 1 -- 172.21.15.46:0/3813005020 <== mon.2 v2:172.21.15.142:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9554+0+0 (secure 0 0 0) 0x7f71c8004050 con 0x55db0fed2fe0 2024-03-31T23:19:40.360 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:40.351+0000 7f71aeffd640 1 -- 172.21.15.46:0/3813005020 <== mon.2 v2:172.21.15.142:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f71c8032050 con 0x55db0fed2fe0 2024-03-31T23:19:40.360 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:40.351+0000 7f71d8c0c740 10 client.26636 fetch_fsmap learned FSMap version 35 2024-03-31T23:19:40.360 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:40.351+0000 7f71d8c0c740 1 -- 172.21.15.46:0/3813005020 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x55db0feea9c0 con 0x55db0fed2fe0 2024-03-31T23:19:40.360 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:40.351+0000 7f71aeffd640 1 -- 172.21.15.46:0/3813005020 <== mon.2 v2:172.21.15.142:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f71c8003c30 con 0x55db0fed2fe0 2024-03-31T23:19:40.360 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:40.351+0000 7f71aeffd640 1 -- 172.21.15.46:0/3813005020 <== mon.2 v2:172.21.15.142:3300/0 5 ==== fsmap(e 35) ==== 8366+0+0 (secure 0 0 0) 0x7f71c8021030 con 0x55db0fed2fe0 2024-03-31T23:19:40.360 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:40.351+0000 7f71d8c0c740 10 client.26636 fetch_fsmap finished waiting for FSMap version 35 2024-03-31T23:19:40.360 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:40.351+0000 7f71d8c0c740 10 client.26636 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-31T23:19:40.360 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:40.351+0000 7f71d8c0c740 20 client.26636 populate_metadata read hostname 'smithi046' 2024-03-31T23:19:40.360 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:40.351+0000 7f71d8c0c740 1 --2- 172.21.15.46:0/3813005020 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55db0feebdc0 0x55db0feee1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:19:40.360 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:40.351+0000 7f71d8c0c740 4 client.26636 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-31T23:19:40.360 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:40.351+0000 7f71d8c0c740 1 -- 172.21.15.46:0/3813005020 --> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55db0fe49120 con 0x55db0feebdc0 2024-03-31T23:19:40.360 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:40.352+0000 7f71ceffd640 1 --2- 172.21.15.46:0/3813005020 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55db0feebdc0 0x55db0feee1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:19:40.360 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:40.352+0000 7f71ceffd640 1 --2- 172.21.15.46:0/3813005020 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55db0feebdc0 0x55db0feee1b0 crc :-1 s=READY pgs=145 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=69952605d79382b server_cookie=91486e318f558f60 in_seq=0 out_seq=0 2024-03-31T23:19:40.361 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:40.352+0000 7f71aeffd640 10 client.26636 ms_handle_connect on v2:172.21.15.142:6838/2519143918 2024-03-31T23:19:40.361 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:40.353+0000 7f71aeffd640 1 -- 172.21.15.46:0/3813005020 <== mds.0 v2:172.21.15.142:6838/2519143918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55db0fe49120 con 0x55db0feebdc0 2024-03-31T23:19:40.361 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:40.353+0000 7f71aeffd640 10 client.26636 handle_command_reply: tid=0 2024-03-31T23:19:40.361 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:40.353+0000 7f71d8c0c740 1 client.26636 shutdown 2024-03-31T23:19:40.361 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:40.353+0000 7f71d8c0c740 1 -- 172.21.15.46:0/3813005020 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55db0fed2fe0 msgr2=0x55db0fed9430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:19:40.361 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:40.353+0000 7f71d8c0c740 1 --2- 172.21.15.46:0/3813005020 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55db0fed2fe0 0x55db0fed9430 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f71c80049d0 tx=0x7f71c800f580 comp rx=0 tx=0).stop 2024-03-31T23:19:40.361 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:40.353+0000 7f71d8c0c740 1 -- 172.21.15.46:0/3813005020 shutdown_connections 2024-03-31T23:19:40.361 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:40.353+0000 7f71d8c0c740 1 --2- 172.21.15.46:0/3813005020 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55db0feebdc0 0x55db0feee1b0 crc :-1 s=READY pgs=145 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:19:40.361 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:40.353+0000 7f71d8c0c740 1 --2- 172.21.15.46:0/3813005020 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55db0fedbdb0 0x55db0fed9970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:19:40.361 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:40.353+0000 7f71aeffd640 0 client.26636 ms_handle_reset on v2:172.21.15.142:6838/2519143918 2024-03-31T23:19:40.361 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:40.353+0000 7f71d8c0c740 1 --2- 172.21.15.46:0/3813005020 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55db0fed2fe0 0x55db0fed9430 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:19:40.361 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:40.353+0000 7f71d8c0c740 1 --2- 172.21.15.46:0/3813005020 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55db0fed26c0 0x55db0fed8ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:19:40.361 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:40.353+0000 7f71d8c0c740 1 -- 172.21.15.46:0/3813005020 >> 172.21.15.46:0/3813005020 conn(0x55db0feccfe0 msgr2=0x55db0feced30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:19:40.361 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:40.353+0000 7f71d8c0c740 1 -- 172.21.15.46:0/3813005020 shutdown_connections 2024-03-31T23:19:40.362 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:40.353+0000 7f71d8c0c740 1 -- 172.21.15.46:0/3813005020 wait complete. 2024-03-31T23:19:40.362 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:40.354+0000 7f71d8c0c740 20 client.26636 trim_cache size 0 max 16384 2024-03-31T23:19:40.371 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:40.369+0000 7f71d8c0c740 1 Processor -- start 2024-03-31T23:19:40.371 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:40.369+0000 7f71d8c0c740 1 -- start start 2024-03-31T23:19:40.372 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:40.369+0000 7f71d8c0c740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55db0fed26c0 0x55db0fedb620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:19:40.372 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:40.369+0000 7f71d8c0c740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55db0fed2fe0 0x55db0fee7780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:19:40.372 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:40.369+0000 7f71d8c0c740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55db0fedbdb0 0x55db0fecbe00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:19:40.372 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:40.369+0000 7f71d8c0c740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f71d01b3830 con 0x55db0fedbdb0 2024-03-31T23:19:40.372 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:40.369+0000 7f71d8c0c740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f71d01b1070 con 0x55db0fed2fe0 2024-03-31T23:19:40.372 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:40.369+0000 7f71d8c0c740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f71d01aeee0 con 0x55db0fed26c0 2024-03-31T23:19:40.372 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:40.370+0000 7f71ceffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55db0fedbdb0 0x55db0fecbe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:19:40.372 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:40.370+0000 7f71ceffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55db0fedbdb0 0x55db0fecbe00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:33932/0 (socket says 172.21.15.46:33932) 2024-03-31T23:19:40.372 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:40.370+0000 7f71ceffd640 1 -- 172.21.15.46:0/988909094 learned_addr learned my addr 172.21.15.46:0/988909094 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:19:40.373 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:40.370+0000 7f71cf7fe640 1 --2- 172.21.15.46:0/988909094 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55db0fed2fe0 0x55db0fee7780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:19:40.373 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:40.370+0000 7f71ceffd640 1 --2- 172.21.15.46:0/988909094 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55db0fed26c0 0x55db0fedb620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:19:40.373 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:40.370+0000 7f71ceffd640 1 -- 172.21.15.46:0/988909094 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55db0fed26c0 msgr2=0x55db0fedb620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:19:40.373 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:40.370+0000 7f71ceffd640 1 --2- 172.21.15.46:0/988909094 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55db0fed26c0 0x55db0fedb620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:19:40.373 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:40.370+0000 7f71ceffd640 1 -- 172.21.15.46:0/988909094 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55db0fed2fe0 msgr2=0x55db0fee7780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:19:40.373 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:40.370+0000 7f71ceffd640 1 --2- 172.21.15.46:0/988909094 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55db0fed2fe0 0x55db0fee7780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:19:40.373 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:40.370+0000 7f71ceffd640 1 -- 172.21.15.46:0/988909094 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55db0fede8c0 con 0x55db0fedbdb0 2024-03-31T23:19:40.373 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:40.370+0000 7f71ceffd640 1 --2- 172.21.15.46:0/988909094 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55db0fedbdb0 0x55db0fecbe00 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7f71b8005e00 tx=0x7f71b8004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4155539a0c734221 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:19:40.373 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:40.370+0000 7f71c57fa640 1 -- 172.21.15.46:0/988909094 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71b8019040 con 0x55db0fedbdb0 2024-03-31T23:19:40.373 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:40.370+0000 7f71c57fa640 1 -- 172.21.15.46:0/988909094 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f71b8004220 con 0x55db0fedbdb0 2024-03-31T23:19:40.373 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:40.370+0000 7f71c57fa640 1 -- 172.21.15.46:0/988909094 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71b8020630 con 0x55db0fedbdb0 2024-03-31T23:19:40.377 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:40.371+0000 7f71d8c0c740 1 -- 172.21.15.46:0/988909094 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55db0fedbdb0 msgr2=0x55db0fecbe00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:19:40.377 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:40.371+0000 7f71d8c0c740 1 --2- 172.21.15.46:0/988909094 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55db0fedbdb0 0x55db0fecbe00 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7f71b8005e00 tx=0x7f71b8004730 comp rx=0 tx=0).stop 2024-03-31T23:19:40.377 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:40.371+0000 7f71d8c0c740 1 -- 172.21.15.46:0/988909094 shutdown_connections 2024-03-31T23:19:40.377 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:40.371+0000 7f71d8c0c740 1 --2- 172.21.15.46:0/988909094 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55db0fedbdb0 0x55db0fecbe00 unknown :-1 s=CLOSED pgs=763 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:19:40.377 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:40.371+0000 7f71d8c0c740 1 --2- 172.21.15.46:0/988909094 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55db0fed2fe0 0x55db0fee7780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:19:40.378 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:40.371+0000 7f71d8c0c740 1 --2- 172.21.15.46:0/988909094 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55db0fed26c0 0x55db0fedb620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:19:40.378 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:40.371+0000 7f71d8c0c740 1 -- 172.21.15.46:0/988909094 >> 172.21.15.46:0/988909094 conn(0x55db0feccfe0 msgr2=0x55db0fecb6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:19:40.378 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:40.371+0000 7f71d8c0c740 1 -- 172.21.15.46:0/988909094 shutdown_connections 2024-03-31T23:19:40.378 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:40.371+0000 7f71d8c0c740 1 -- 172.21.15.46:0/988909094 wait complete. 2024-03-31T23:19:40.378 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:40.371+0000 7f71d8c0c740 1 -- start start 2024-03-31T23:19:40.378 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:40.372+0000 7f71d8c0c740 1 Processor -- start 2024-03-31T23:19:40.378 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:40.372+0000 7f71d8c0c740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55db0fed26c0 0x55db0fed5df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:19:40.378 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:40.372+0000 7f71d8c0c740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55db0fed2fe0 0x55db0fed6330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:19:40.378 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:40.372+0000 7f71d8c0c740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55db0fedbdb0 0x55db0ff50fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:19:40.378 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:40.372+0000 7f71d8c0c740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f71d01af530 con 0x55db0fedbdb0 2024-03-31T23:19:40.378 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:40.372+0000 7f71d8c0c740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f71d01b09a0 con 0x55db0fed2fe0 2024-03-31T23:19:40.378 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:40.372+0000 7f71d8c0c740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f71d01afc00 con 0x55db0fed26c0 2024-03-31T23:19:40.378 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:40.372+0000 7f71d8c0c740 10 client.0 _ll_register_callbacks cb 0x7f71c801c070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T23:19:40.379 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:40.372+0000 7f71ceffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55db0fedbdb0 0x55db0ff50fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:19:40.379 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:40.372+0000 7f71cf7fe640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55db0fed2fe0 0x55db0fed6330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:19:40.379 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:40.372+0000 7f71ceffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55db0fedbdb0 0x55db0ff50fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:33942/0 (socket says 172.21.15.46:33942) 2024-03-31T23:19:40.379 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:40.372+0000 7f71ceffd640 1 -- 172.21.15.46:0/3946780917 learned_addr learned my addr 172.21.15.46:0/3946780917 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:19:40.379 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:40.372+0000 7f71ceffd640 1 --2- 172.21.15.46:0/3946780917 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55db0fed26c0 0x55db0fed5df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:19:40.379 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:40.372+0000 7f71cf7fe640 1 -- 172.21.15.46:0/3946780917 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55db0fed26c0 msgr2=0x55db0fed5df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:19:40.379 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:40.372+0000 7f71cf7fe640 1 --2- 172.21.15.46:0/3946780917 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55db0fed26c0 0x55db0fed5df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:19:40.379 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:40.372+0000 7f71cf7fe640 1 -- 172.21.15.46:0/3946780917 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55db0fedbdb0 msgr2=0x55db0ff50fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:19:40.379 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:40.372+0000 7f71cf7fe640 1 --2- 172.21.15.46:0/3946780917 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55db0fedbdb0 0x55db0ff50fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:19:40.379 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:40.372+0000 7f71cf7fe640 1 -- 172.21.15.46:0/3946780917 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55db0fed6c30 con 0x55db0fed2fe0 2024-03-31T23:19:40.379 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:40.372+0000 7f71ceffd640 1 --2- 172.21.15.46:0/3946780917 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55db0fedbdb0 0x55db0ff50fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:19:40.379 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:40.373+0000 7f71cf7fe640 1 --2- 172.21.15.46:0/3946780917 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55db0fed2fe0 0x55db0fed6330 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f71c800fcb0 tx=0x7f71c8009300 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2b64ed69b0aa4cce server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:19:40.380 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:40.373+0000 7f71aeffd640 10 client.0 ms_handle_connect on v2:172.21.15.142:3300/0 2024-03-31T23:19:40.380 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:40.373+0000 7f71aeffd640 1 -- 172.21.15.46:0/3946780917 <== mon.2 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71c803a070 con 0x55db0fed2fe0 2024-03-31T23:19:40.380 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:40.373+0000 7f71d8c0c740 1 -- 172.21.15.46:0/3946780917 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f71b8004220 con 0x55db0fed2fe0 2024-03-31T23:19:40.380 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:40.373+0000 7f71aeffd640 1 -- 172.21.15.46:0/3946780917 <== mon.2 v2:172.21.15.142:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9554+0+0 (secure 0 0 0) 0x7f71c8003970 con 0x55db0fed2fe0 2024-03-31T23:19:40.380 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:40.373+0000 7f71aeffd640 1 -- 172.21.15.46:0/3946780917 <== mon.2 v2:172.21.15.142:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f71c8042050 con 0x55db0fed2fe0 2024-03-31T23:19:40.380 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:40.373+0000 7f71d8c0c740 10 client.26642 fetch_fsmap learned FSMap version 35 2024-03-31T23:19:40.380 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:40.373+0000 7f71d8c0c740 1 -- 172.21.15.46:0/3946780917 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x55db0ff52600 con 0x55db0fed2fe0 2024-03-31T23:19:40.380 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:40.373+0000 7f71aeffd640 1 -- 172.21.15.46:0/3946780917 <== mon.2 v2:172.21.15.142:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f71b8004220 con 0x55db0fed2fe0 2024-03-31T23:19:40.380 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:40.373+0000 7f71aeffd640 1 -- 172.21.15.46:0/3946780917 <== mon.2 v2:172.21.15.142:3300/0 5 ==== fsmap(e 35) ==== 8366+0+0 (secure 0 0 0) 0x7f71c803d030 con 0x55db0fed2fe0 2024-03-31T23:19:40.380 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:40.373+0000 7f71d8c0c740 10 client.26642 fetch_fsmap finished waiting for FSMap version 35 2024-03-31T23:19:40.380 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:40.373+0000 7f71d8c0c740 10 client.26642 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-31T23:19:40.380 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:40.373+0000 7f71d8c0c740 20 client.26642 populate_metadata read hostname 'smithi046' 2024-03-31T23:19:40.381 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:40.373+0000 7f71d8c0c740 1 --2- 172.21.15.46:0/3946780917 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55db0ff52d50 0x55db0ff55170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:19:40.381 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:40.373+0000 7f71d8c0c740 4 client.26642 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-31T23:19:40.381 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:40.373+0000 7f71d8c0c740 1 -- 172.21.15.46:0/3946780917 --> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55db0fe42e30 con 0x55db0ff52d50 2024-03-31T23:19:40.381 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:40.374+0000 7f71ceffd640 1 --2- 172.21.15.46:0/3946780917 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55db0ff52d50 0x55db0ff55170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:19:40.381 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:40.374+0000 7f71ceffd640 1 --2- 172.21.15.46:0/3946780917 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55db0ff52d50 0x55db0ff55170 crc :-1 s=READY pgs=146 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=58c4772c31c79a04 server_cookie=e61ad34bbde4747d in_seq=0 out_seq=0 2024-03-31T23:19:40.381 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:40.374+0000 7f71aeffd640 10 client.26642 ms_handle_connect on v2:172.21.15.142:6838/2519143918 2024-03-31T23:19:40.400 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:40.398+0000 7f71aeffd640 1 -- 172.21.15.46:0/3946780917 <== mds.0 v2:172.21.15.142:6838/2519143918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55db0fe42e30 con 0x55db0ff52d50 2024-03-31T23:19:40.400 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:40.398+0000 7f71aeffd640 10 client.26642 handle_command_reply: tid=0 2024-03-31T23:19:40.400 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:40.398+0000 7f71d8c0c740 1 client.26642 shutdown 2024-03-31T23:19:40.400 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:40.398+0000 7f71d8c0c740 1 -- 172.21.15.46:0/3946780917 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55db0fed2fe0 msgr2=0x55db0fed6330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:19:40.400 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:40.398+0000 7f71d8c0c740 1 --2- 172.21.15.46:0/3946780917 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55db0fed2fe0 0x55db0fed6330 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f71c800fcb0 tx=0x7f71c8009300 comp rx=0 tx=0).stop 2024-03-31T23:19:40.400 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:40.398+0000 7f71d8c0c740 1 -- 172.21.15.46:0/3946780917 shutdown_connections 2024-03-31T23:19:40.400 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:40.398+0000 7f71d8c0c740 1 --2- 172.21.15.46:0/3946780917 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55db0ff52d50 0x55db0ff55170 crc :-1 s=READY pgs=146 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:19:40.400 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:40.398+0000 7f71d8c0c740 1 --2- 172.21.15.46:0/3946780917 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55db0fedbdb0 0x55db0ff50fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:19:40.400 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:40.398+0000 7f71d8c0c740 1 --2- 172.21.15.46:0/3946780917 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55db0fed2fe0 0x55db0fed6330 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:19:40.400 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:40.398+0000 7f71d8c0c740 1 --2- 172.21.15.46:0/3946780917 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55db0fed26c0 0x55db0fed5df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:19:40.400 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:40.398+0000 7f71aeffd640 0 client.26642 ms_handle_reset on v2:172.21.15.142:6838/2519143918 2024-03-31T23:19:40.401 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:40.398+0000 7f71d8c0c740 1 -- 172.21.15.46:0/3946780917 >> 172.21.15.46:0/3946780917 conn(0x55db0feccfe0 msgr2=0x55db0fecbf80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:19:40.401 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:40.398+0000 7f71d8c0c740 1 -- 172.21.15.46:0/3946780917 shutdown_connections 2024-03-31T23:19:40.401 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:40.398+0000 7f71d8c0c740 1 -- 172.21.15.46:0/3946780917 wait complete. 2024-03-31T23:19:40.401 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:40.398+0000 7f71d8c0c740 20 client.26642 trim_cache size 0 max 16384 2024-03-31T23:19:40.401 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout:{ 2024-03-31T23:19:40.401 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout: "message": "", 2024-03-31T23:19:40.401 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout: "return_code": 0 2024-03-31T23:19:40.401 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout:} 2024-03-31T23:19:40.402 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:40.399+0000 7f71d6030640 1 -- 172.21.15.46:0/645228776 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f71bc0a26b0 msgr2=0x7f71bc0a4b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:19:40.402 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:40.399+0000 7f71d6030640 1 --2- 172.21.15.46:0/645228776 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f71bc0a26b0 0x7f71bc0a4b70 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f71c00b9c70 tx=0x7f71c00dd040 comp rx=0 tx=0).stop 2024-03-31T23:19:40.402 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:40.399+0000 7f71d6030640 1 -- 172.21.15.46:0/645228776 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f71d0107710 msgr2=0x7f71d00744d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:19:40.402 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:40.399+0000 7f71d6030640 1 --2- 172.21.15.46:0/645228776 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f71d0107710 0x7f71d00744d0 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7f71c0005940 tx=0x7f71c00045e0 comp rx=0 tx=0).stop 2024-03-31T23:19:40.402 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:40.399+0000 7f71d6030640 1 -- 172.21.15.46:0/645228776 shutdown_connections 2024-03-31T23:19:40.402 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:40.399+0000 7f71d6030640 1 --2- 172.21.15.46:0/645228776 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f71bc0a26b0 0x7f71bc0a4b70 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:19:40.403 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:40.399+0000 7f71d6030640 1 --2- 172.21.15.46:0/645228776 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f71d0107710 0x7f71d00744d0 unknown :-1 s=CLOSED pgs=762 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:19:40.403 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:40.399+0000 7f71d6030640 1 --2- 172.21.15.46:0/645228776 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f71d01020c0 0x7f71d0073f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:19:40.403 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:40.399+0000 7f71d6030640 1 --2- 172.21.15.46:0/645228776 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f71d0101700 0x7f71d0073a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:19:40.403 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:40.399+0000 7f71d6030640 1 -- 172.21.15.46:0/645228776 >> 172.21.15.46:0/645228776 conn(0x7f71d00fd210 msgr2=0x7f71d00fd770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:19:40.403 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:40.399+0000 7f71d6030640 1 -- 172.21.15.46:0/645228776 shutdown_connections 2024-03-31T23:19:40.403 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:40.399+0000 7f71d6030640 1 -- 172.21.15.46:0/645228776 wait complete. 2024-03-31T23:19:40.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:19:40 smithi046 ceph-mon[32498]: pgmap v245: 129 pgs: 129 active+clean; 106 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 365 KiB/s rd, 1.0 MiB/s wr, 102 op/s 2024-03-31T23:19:40.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:19:40 smithi142 ceph-mon[35271]: pgmap v245: 129 pgs: 129 active+clean; 106 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 365 KiB/s rd, 1.0 MiB/s wr, 102 op/s 2024-03-31T23:19:41.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:19:40 smithi162 ceph-mon[36237]: pgmap v245: 129 pgs: 129 active+clean; 106 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 365 KiB/s rd, 1.0 MiB/s wr, 102 op/s 2024-03-31T23:19:41.204 INFO:tasks.workunit.client.0.smithi046.stdout: 1 178775 2.35 MB/sec execute 287 sec latency 19.677 ms 2024-03-31T23:19:42.204 INFO:tasks.workunit.client.0.smithi046.stdout: 1 179241 2.34 MB/sec execute 288 sec latency 22.619 ms 2024-03-31T23:19:42.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:19:42 smithi046 ceph-mon[32498]: pgmap v246: 129 pgs: 129 active+clean; 106 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 150 KiB/s rd, 502 KiB/s wr, 42 op/s 2024-03-31T23:19:42.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:19:42 smithi142 ceph-mon[35271]: pgmap v246: 129 pgs: 129 active+clean; 106 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 150 KiB/s rd, 502 KiB/s wr, 42 op/s 2024-03-31T23:19:43.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:19:42 smithi162 ceph-mon[36237]: pgmap v246: 129 pgs: 129 active+clean; 106 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 150 KiB/s rd, 502 KiB/s wr, 42 op/s 2024-03-31T23:19:43.204 INFO:tasks.workunit.client.0.smithi046.stdout: 1 179765 2.34 MB/sec execute 289 sec latency 21.215 ms 2024-03-31T23:19:44.204 INFO:tasks.workunit.client.0.smithi046.stdout: 1 180357 2.35 MB/sec execute 290 sec latency 16.872 ms 2024-03-31T23:19:44.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:19:44 smithi046 ceph-mon[32498]: pgmap v247: 129 pgs: 129 active+clean; 106 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 150 KiB/s rd, 502 KiB/s wr, 42 op/s 2024-03-31T23:19:44.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:19:44 smithi142 ceph-mon[35271]: pgmap v247: 129 pgs: 129 active+clean; 106 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 150 KiB/s rd, 502 KiB/s wr, 42 op/s 2024-03-31T23:19:45.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:19:44 smithi162 ceph-mon[36237]: pgmap v247: 129 pgs: 129 active+clean; 106 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 150 KiB/s rd, 502 KiB/s wr, 42 op/s 2024-03-31T23:19:45.204 INFO:tasks.workunit.client.0.smithi046.stdout: 1 180874 2.36 MB/sec execute 291 sec latency 16.125 ms 2024-03-31T23:19:46.204 INFO:tasks.workunit.client.0.smithi046.stdout: 1 180982 2.35 MB/sec execute 292 sec latency 778.164 ms 2024-03-31T23:19:46.862 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:19:46 smithi046 ceph-mon[32498]: pgmap v248: 129 pgs: 129 active+clean; 107 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 378 KiB/s rd, 1.0 MiB/s wr, 101 op/s 2024-03-31T23:19:46.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:19:46 smithi142 ceph-mon[35271]: pgmap v248: 129 pgs: 129 active+clean; 107 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 378 KiB/s rd, 1.0 MiB/s wr, 101 op/s 2024-03-31T23:19:47.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:19:46 smithi162 ceph-mon[36237]: pgmap v248: 129 pgs: 129 active+clean; 107 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 378 KiB/s rd, 1.0 MiB/s wr, 101 op/s 2024-03-31T23:19:47.204 INFO:tasks.workunit.client.0.smithi046.stdout: 1 180982 2.34 MB/sec execute 293 sec latency 1778.293 ms 2024-03-31T23:19:48.204 INFO:tasks.workunit.client.0.smithi046.stdout: 1 181049 2.33 MB/sec execute 294 sec latency 2313.672 ms 2024-03-31T23:19:48.863 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:19:48 smithi046 ceph-mon[32498]: pgmap v249: 129 pgs: 129 active+clean; 107 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 261 KiB/s rd, 724 KiB/s wr, 74 op/s 2024-03-31T23:19:48.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:19:48 smithi142 ceph-mon[35271]: pgmap v249: 129 pgs: 129 active+clean; 107 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 261 KiB/s rd, 724 KiB/s wr, 74 op/s 2024-03-31T23:19:49.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:19:48 smithi162 ceph-mon[36237]: pgmap v249: 129 pgs: 129 active+clean; 107 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 261 KiB/s rd, 724 KiB/s wr, 74 op/s 2024-03-31T23:19:49.204 INFO:tasks.workunit.client.0.smithi046.stdout: 1 181049 2.33 MB/sec execute 295 sec latency 1246.038 ms 2024-03-31T23:19:50.204 INFO:tasks.workunit.client.0.smithi046.stdout: 1 181049 2.32 MB/sec execute 296 sec latency 2246.128 ms 2024-03-31T23:19:50.862 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:19:50 smithi046 ceph-mon[32498]: pgmap v250: 129 pgs: 129 active+clean; 109 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 375 KiB/s rd, 1.0 MiB/s wr, 101 op/s 2024-03-31T23:19:50.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:19:50 smithi142 ceph-mon[35271]: pgmap v250: 129 pgs: 129 active+clean; 109 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 375 KiB/s rd, 1.0 MiB/s wr, 101 op/s 2024-03-31T23:19:51.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:19:50 smithi162 ceph-mon[36237]: pgmap v250: 129 pgs: 129 active+clean; 109 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 375 KiB/s rd, 1.0 MiB/s wr, 101 op/s 2024-03-31T23:19:51.205 INFO:tasks.workunit.client.0.smithi046.stdout: 1 181049 2.31 MB/sec execute 297 sec latency 3246.213 ms 2024-03-31T23:19:52.205 INFO:tasks.workunit.client.0.smithi046.stdout: 1 181049 2.30 MB/sec execute 298 sec latency 4246.307 ms 2024-03-31T23:19:52.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:19:52 smithi046 ceph-mon[32498]: pgmap v251: 129 pgs: 129 active+clean; 109 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 342 KiB/s rd, 853 KiB/s wr, 85 op/s 2024-03-31T23:19:52.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:19:52 smithi142 ceph-mon[35271]: pgmap v251: 129 pgs: 129 active+clean; 109 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 342 KiB/s rd, 853 KiB/s wr, 85 op/s 2024-03-31T23:19:53.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:19:52 smithi162 ceph-mon[36237]: pgmap v251: 129 pgs: 129 active+clean; 109 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 342 KiB/s rd, 853 KiB/s wr, 85 op/s 2024-03-31T23:19:53.205 INFO:tasks.workunit.client.0.smithi046.stdout: 1 181189 2.30 MB/sec execute 299 sec latency 4783.457 ms 2024-03-31T23:19:53.497 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:53.496+0000 7f3864593640 1 -- 172.21.15.46:0/3569337366 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f385c107750 msgr2=0x7f385c107b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:19:53.497 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:53.496+0000 7f3864593640 1 --2- 172.21.15.46:0/3569337366 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f385c107750 0x7f385c107b30 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7f3858009930 tx=0x7f385802bc30 comp rx=0 tx=0).stop 2024-03-31T23:19:53.497 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:53.497+0000 7f3864593640 1 -- 172.21.15.46:0/3569337366 shutdown_connections 2024-03-31T23:19:53.497 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:53.497+0000 7f3864593640 1 --2- 172.21.15.46:0/3569337366 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f385c100630 0x7f385c10e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:19:53.498 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:53.497+0000 7f3864593640 1 --2- 172.21.15.46:0/3569337366 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f385c0ffc90 0x7f385c1000f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:19:53.498 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:53.497+0000 7f3864593640 1 --2- 172.21.15.46:0/3569337366 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f385c107750 0x7f385c107b30 unknown :-1 s=CLOSED pgs=764 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:19:53.498 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:53.497+0000 7f3864593640 1 -- 172.21.15.46:0/3569337366 >> 172.21.15.46:0/3569337366 conn(0x7f385c076ba0 msgr2=0x7f385c0fdf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:19:53.498 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:53.497+0000 7f3864593640 1 -- 172.21.15.46:0/3569337366 shutdown_connections 2024-03-31T23:19:53.498 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:53.497+0000 7f3864593640 1 -- 172.21.15.46:0/3569337366 wait complete. 2024-03-31T23:19:53.498 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:53.497+0000 7f3864593640 1 Processor -- start 2024-03-31T23:19:53.498 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:53.497+0000 7f3864593640 1 -- start start 2024-03-31T23:19:53.500 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:53.497+0000 7f3864593640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f385c0ffc90 0x7f385c197ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:19:53.500 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:53.498+0000 7f3864593640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f385c100630 0x7f385c198400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:19:53.500 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:53.498+0000 7f3864593640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f385c107750 0x7f385c19c780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:19:53.500 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:53.498+0000 7f3864593640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f385c111a40 con 0x7f385c107750 2024-03-31T23:19:53.500 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:53.498+0000 7f3864593640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f385c1118c0 con 0x7f385c0ffc90 2024-03-31T23:19:53.500 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:53.498+0000 7f3864593640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f385c111bc0 con 0x7f385c100630 2024-03-31T23:19:53.500 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:53.498+0000 7f3862b30640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f385c107750 0x7f385c19c780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:19:53.500 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:53.498+0000 7f386232f640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f385c0ffc90 0x7f385c197ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:19:53.500 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:53.498+0000 7f3862b30640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f385c107750 0x7f385c19c780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:38602/0 (socket says 172.21.15.46:38602) 2024-03-31T23:19:53.500 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:53.498+0000 7f3862b30640 1 -- 172.21.15.46:0/1891346254 learned_addr learned my addr 172.21.15.46:0/1891346254 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:19:53.501 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:53.498+0000 7f3861b2e640 1 --2- 172.21.15.46:0/1891346254 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f385c100630 0x7f385c198400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:19:53.501 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:53.498+0000 7f3862b30640 1 -- 172.21.15.46:0/1891346254 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f385c100630 msgr2=0x7f385c198400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:19:53.501 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:53.498+0000 7f3862b30640 1 --2- 172.21.15.46:0/1891346254 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f385c100630 0x7f385c198400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:19:53.501 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:53.498+0000 7f3862b30640 1 -- 172.21.15.46:0/1891346254 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f385c0ffc90 msgr2=0x7f385c197ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:19:53.501 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:53.498+0000 7f3862b30640 1 --2- 172.21.15.46:0/1891346254 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f385c0ffc90 0x7f385c197ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:19:53.501 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:53.498+0000 7f3862b30640 1 -- 172.21.15.46:0/1891346254 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f385c19ceb0 con 0x7f385c107750 2024-03-31T23:19:53.501 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:53.498+0000 7f386232f640 1 --2- 172.21.15.46:0/1891346254 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f385c0ffc90 0x7f385c197ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:19:53.501 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:53.498+0000 7f3862b30640 1 --2- 172.21.15.46:0/1891346254 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f385c107750 0x7f385c19c780 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7f385400eb10 tx=0x7f385400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:19:53.501 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:53.498+0000 7f38537fe640 1 -- 172.21.15.46:0/1891346254 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3854004b20 con 0x7f385c107750 2024-03-31T23:19:53.501 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:53.498+0000 7f38537fe640 1 -- 172.21.15.46:0/1891346254 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3854004cc0 con 0x7f385c107750 2024-03-31T23:19:53.501 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:53.498+0000 7f3864593640 1 -- 172.21.15.46:0/1891346254 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f385c19d1a0 con 0x7f385c107750 2024-03-31T23:19:53.501 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:53.498+0000 7f38537fe640 1 -- 172.21.15.46:0/1891346254 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3854010690 con 0x7f385c107750 2024-03-31T23:19:53.501 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:53.498+0000 7f3864593640 1 -- 172.21.15.46:0/1891346254 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f385c1a4a30 con 0x7f385c107750 2024-03-31T23:19:53.507 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:53.501+0000 7f38537fe640 1 -- 172.21.15.46:0/1891346254 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 25) ==== 137451+0+0 (secure 0 0 0) 0x7f3854005020 con 0x7f385c107750 2024-03-31T23:19:53.507 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:53.501+0000 7f386717a740 1 Processor -- start 2024-03-31T23:19:53.507 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:53.501+0000 7f386717a740 1 -- start start 2024-03-31T23:19:53.507 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:53.502+0000 7f386717a740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5593a49966c0 0x5593a4996aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:19:53.507 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:53.502+0000 7f386717a740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5593a4996fe0 0x5593a499f870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:19:53.507 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:53.502+0000 7f386717a740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5593a499fdb0 0x5593a49a21a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:19:53.508 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:53.502+0000 7f386717a740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x5593a48f98f0 con 0x5593a49966c0 2024-03-31T23:19:53.508 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:53.502+0000 7f386717a740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x5593a48a28d0 con 0x5593a4996fe0 2024-03-31T23:19:53.508 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:53.502+0000 7f386717a740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5593a48f5990 con 0x5593a499fdb0 2024-03-31T23:19:53.508 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:53.502+0000 7f38537fe640 1 --2- 172.21.15.46:0/1891346254 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f38480a26b0 0x7f38480a4b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:19:53.508 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:53.502+0000 7f386232f640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5593a4996fe0 0x5593a499f870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:19:53.508 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:53.502+0000 7f3861b2e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5593a49966c0 0x5593a4996aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:19:53.508 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:53.502+0000 7f386232f640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5593a4996fe0 0x5593a499f870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.46:49822/0 (socket says 172.21.15.46:49822) 2024-03-31T23:19:53.508 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:53.502+0000 7f3862b30640 1 --2- 172.21.15.46:0/1891346254 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f38480a26b0 0x7f38480a4b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:19:53.508 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:53.502+0000 7f386232f640 1 -- 172.21.15.46:0/1131090218 learned_addr learned my addr 172.21.15.46:0/1131090218 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:19:53.508 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:53.502+0000 7f3861b2e640 1 --2- 172.21.15.46:0/1131090218 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5593a499fdb0 0x5593a49a21a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:19:53.509 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:53.502+0000 7f38537fe640 1 -- 172.21.15.46:0/1891346254 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9554+0+0 (secure 0 0 0) 0x7f38540cf510 con 0x7f385c107750 2024-03-31T23:19:53.509 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:53.502+0000 7f386232f640 1 -- 172.21.15.46:0/1131090218 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5593a499fdb0 msgr2=0x5593a49a21a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:19:53.509 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:53.502+0000 7f386232f640 1 --2- 172.21.15.46:0/1131090218 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5593a499fdb0 0x5593a49a21a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:19:53.509 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:53.502+0000 7f386232f640 1 -- 172.21.15.46:0/1131090218 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5593a49966c0 msgr2=0x5593a4996aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:19:53.509 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:53.502+0000 7f386232f640 1 --2- 172.21.15.46:0/1131090218 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5593a49966c0 0x5593a4996aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:19:53.509 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:53.502+0000 7f386232f640 1 -- 172.21.15.46:0/1131090218 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5593a460e210 con 0x5593a4996fe0 2024-03-31T23:19:53.509 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:53.503+0000 7f386232f640 1 --2- 172.21.15.46:0/1131090218 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5593a4996fe0 0x5593a499f870 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f3858036040 tx=0x7f38580095f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=aa802a175e98ed60 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:19:53.509 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:53.503+0000 7f3862b30640 1 --2- 172.21.15.46:0/1891346254 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f38480a26b0 0x7f38480a4b70 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f38540cc9a0 tx=0x7f385406e990 comp rx=0 tx=0).ready entity=mgr.15654 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:19:53.509 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:53.503+0000 7f3850bfe640 1 -- 172.21.15.46:0/1131090218 <== mon.2 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3858045070 con 0x5593a4996fe0 2024-03-31T23:19:53.509 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:53.503+0000 7f3850bfe640 1 -- 172.21.15.46:0/1131090218 <== mon.2 v2:172.21.15.142:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f385802c7e0 con 0x5593a4996fe0 2024-03-31T23:19:53.509 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:53.503+0000 7f3850bfe640 1 -- 172.21.15.46:0/1131090218 <== mon.2 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38580039b0 con 0x5593a4996fe0 2024-03-31T23:19:53.509 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:53.504+0000 7f386717a740 1 -- 172.21.15.46:0/1131090218 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5593a4996fe0 msgr2=0x5593a499f870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:19:53.509 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:53.504+0000 7f386717a740 1 --2- 172.21.15.46:0/1131090218 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5593a4996fe0 0x5593a499f870 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f3858036040 tx=0x7f38580095f0 comp rx=0 tx=0).stop 2024-03-31T23:19:53.510 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:53.504+0000 7f386717a740 1 -- 172.21.15.46:0/1131090218 shutdown_connections 2024-03-31T23:19:53.510 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:53.504+0000 7f386717a740 1 --2- 172.21.15.46:0/1131090218 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5593a499fdb0 0x5593a49a21a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:19:53.510 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:53.504+0000 7f386717a740 1 --2- 172.21.15.46:0/1131090218 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5593a4996fe0 0x5593a499f870 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:19:53.510 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:53.504+0000 7f386717a740 1 --2- 172.21.15.46:0/1131090218 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5593a49966c0 0x5593a4996aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:19:53.510 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:53.504+0000 7f386717a740 1 -- 172.21.15.46:0/1131090218 >> 172.21.15.46:0/1131090218 conn(0x5593a4990fe0 msgr2=0x5593a49913c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:19:53.510 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:53.504+0000 7f386717a740 1 -- 172.21.15.46:0/1131090218 shutdown_connections 2024-03-31T23:19:53.510 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:53.504+0000 7f386717a740 1 -- 172.21.15.46:0/1131090218 wait complete. 2024-03-31T23:19:53.510 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:53.504+0000 7f386717a740 1 -- start start 2024-03-31T23:19:53.510 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:53.504+0000 7f386717a740 1 Processor -- start 2024-03-31T23:19:53.510 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:53.504+0000 7f386717a740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5593a49966c0 0x5593a499cd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:19:53.510 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:53.504+0000 7f386717a740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5593a4996fe0 0x5593a499d270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:19:53.510 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:53.504+0000 7f386717a740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5593a499fdb0 0x5593a499d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:19:53.510 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:53.504+0000 7f386717a740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x5593a49a6d90 con 0x5593a49966c0 2024-03-31T23:19:53.510 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:53.504+0000 7f386717a740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x5593a4897f80 con 0x5593a4996fe0 2024-03-31T23:19:53.511 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:53.504+0000 7f386717a740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5593a49a6f10 con 0x5593a499fdb0 2024-03-31T23:19:53.511 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:53.504+0000 7f386717a740 10 client.0 _ll_register_callbacks cb 0x5593a48f0290 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T23:19:53.511 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:53.505+0000 7f3861b2e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5593a49966c0 0x5593a499cd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:19:53.511 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:53.505+0000 7f3861b2e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5593a49966c0 0x5593a499cd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:38616/0 (socket says 172.21.15.46:38616) 2024-03-31T23:19:53.511 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:53.505+0000 7f3861b2e640 1 -- 172.21.15.46:0/198276593 learned_addr learned my addr 172.21.15.46:0/198276593 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:19:53.511 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:53.505+0000 7f386232f640 1 --2- 172.21.15.46:0/198276593 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5593a4996fe0 0x5593a499d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:19:53.511 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:53.505+0000 7f3861b2e640 1 --2- 172.21.15.46:0/198276593 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5593a499fdb0 0x5593a499d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:19:53.511 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:53.505+0000 7f3861b2e640 1 -- 172.21.15.46:0/198276593 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5593a499fdb0 msgr2=0x5593a499d7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:19:53.511 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:53.505+0000 7f3861b2e640 1 --2- 172.21.15.46:0/198276593 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5593a499fdb0 0x5593a499d7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:19:53.511 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:53.505+0000 7f3861b2e640 1 -- 172.21.15.46:0/198276593 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5593a4996fe0 msgr2=0x5593a499d270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:19:53.511 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:53.505+0000 7f3861b2e640 1 --2- 172.21.15.46:0/198276593 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5593a4996fe0 0x5593a499d270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:19:53.511 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:53.505+0000 7f3861b2e640 1 -- 172.21.15.46:0/198276593 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5593a46da620 con 0x5593a49966c0 2024-03-31T23:19:53.511 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:53.505+0000 7f3861b2e640 1 --2- 172.21.15.46:0/198276593 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5593a49966c0 0x5593a499cd30 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7f385c0fdac0 tx=0x7f384c015040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f9c56098337fc7ec server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:19:53.512 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:53.505+0000 7f3845ffb640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-31T23:19:53.512 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:53.505+0000 7f3845ffb640 1 -- 172.21.15.46:0/198276593 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f384c005020 con 0x5593a49966c0 2024-03-31T23:19:53.512 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:53.505+0000 7f3845ffb640 1 -- 172.21.15.46:0/198276593 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9554+0+0 (secure 0 0 0) 0x7f384c0051c0 con 0x5593a49966c0 2024-03-31T23:19:53.512 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:53.505+0000 7f386717a740 1 -- 172.21.15.46:0/198276593 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f385802c7e0 con 0x5593a49966c0 2024-03-31T23:19:53.512 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:53.505+0000 7f3845ffb640 1 -- 172.21.15.46:0/198276593 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f384c02b050 con 0x5593a49966c0 2024-03-31T23:19:53.512 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:53.505+0000 7f386717a740 10 client.16812 fetch_fsmap learned FSMap version 35 2024-03-31T23:19:53.512 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:53.505+0000 7f386717a740 1 -- 172.21.15.46:0/198276593 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x5593a49b2be0 con 0x5593a49966c0 2024-03-31T23:19:53.512 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:53.506+0000 7f3845ffb640 1 -- 172.21.15.46:0/198276593 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f385802c7e0 con 0x5593a49966c0 2024-03-31T23:19:53.512 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:53.506+0000 7f3845ffb640 1 -- 172.21.15.46:0/198276593 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 35) ==== 8366+0+0 (secure 0 0 0) 0x7f384c025510 con 0x5593a49966c0 2024-03-31T23:19:53.512 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:53.506+0000 7f386717a740 10 client.16812 fetch_fsmap finished waiting for FSMap version 35 2024-03-31T23:19:53.512 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:53.506+0000 7f386717a740 10 client.16812 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-31T23:19:53.512 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:53.506+0000 7f386717a740 20 client.16812 populate_metadata read hostname 'smithi046' 2024-03-31T23:19:53.512 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:53.506+0000 7f386717a740 1 --2- 172.21.15.46:0/198276593 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x5593a49b40c0 0x5593a49b6560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:19:53.512 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:53.506+0000 7f386717a740 4 client.16812 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-31T23:19:53.512 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:53.506+0000 7f386717a740 1 -- 172.21.15.46:0/198276593 --> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5593a490d120 con 0x5593a49b40c0 2024-03-31T23:19:53.513 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:53.506+0000 7f386232f640 1 --2- 172.21.15.46:0/198276593 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x5593a49b40c0 0x5593a49b6560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:19:53.513 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:53.506+0000 7f3845ffb640 10 client.16812 ms_handle_connect on v2:172.21.15.142:6838/2519143918 2024-03-31T23:19:53.513 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:53.506+0000 7f386232f640 1 --2- 172.21.15.46:0/198276593 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x5593a49b40c0 0x5593a49b6560 crc :-1 s=READY pgs=147 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fbf67b1976417675 server_cookie=2f7cdb3a1e46c0c6 in_seq=0 out_seq=0 2024-03-31T23:19:53.514 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:53.508+0000 7f3845ffb640 1 -- 172.21.15.46:0/198276593 <== mds.0 v2:172.21.15.142:6838/2519143918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5593a490d120 con 0x5593a49b40c0 2024-03-31T23:19:53.514 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:53.508+0000 7f3845ffb640 10 client.16812 handle_command_reply: tid=0 2024-03-31T23:19:53.514 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:53.508+0000 7f386717a740 1 client.16812 shutdown 2024-03-31T23:19:53.514 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:53.508+0000 7f386717a740 1 -- 172.21.15.46:0/198276593 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5593a49966c0 msgr2=0x5593a499cd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:19:53.514 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:53.508+0000 7f386717a740 1 --2- 172.21.15.46:0/198276593 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5593a49966c0 0x5593a499cd30 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7f385c0fdac0 tx=0x7f384c015040 comp rx=0 tx=0).stop 2024-03-31T23:19:53.514 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:53.508+0000 7f386717a740 1 -- 172.21.15.46:0/198276593 shutdown_connections 2024-03-31T23:19:53.514 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:53.508+0000 7f386717a740 1 --2- 172.21.15.46:0/198276593 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x5593a49b40c0 0x5593a49b6560 crc :-1 s=READY pgs=147 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:19:53.515 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:53.508+0000 7f3845ffb640 0 client.16812 ms_handle_reset on v2:172.21.15.142:6838/2519143918 2024-03-31T23:19:53.515 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:53.508+0000 7f386717a740 1 --2- 172.21.15.46:0/198276593 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5593a499fdb0 0x5593a499d7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:19:53.515 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:53.508+0000 7f386717a740 1 --2- 172.21.15.46:0/198276593 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5593a4996fe0 0x5593a499d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:19:53.515 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:53.508+0000 7f386717a740 1 --2- 172.21.15.46:0/198276593 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5593a49966c0 0x5593a499cd30 unknown :-1 s=CLOSED pgs=766 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:19:53.515 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:53.508+0000 7f386717a740 1 -- 172.21.15.46:0/198276593 >> 172.21.15.46:0/198276593 conn(0x5593a4990fe0 msgr2=0x5593a4992d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:19:53.515 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:53.508+0000 7f386717a740 1 -- 172.21.15.46:0/198276593 shutdown_connections 2024-03-31T23:19:53.515 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:53.508+0000 7f386717a740 1 -- 172.21.15.46:0/198276593 wait complete. 2024-03-31T23:19:53.515 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:53.508+0000 7f386717a740 20 client.16812 trim_cache size 0 max 16384 2024-03-31T23:19:53.526 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:53.525+0000 7f386717a740 1 Processor -- start 2024-03-31T23:19:53.526 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:53.525+0000 7f386717a740 1 -- start start 2024-03-31T23:19:53.526 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:53.525+0000 7f386717a740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5593a49966c0 0x5593a499c610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:19:53.526 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:53.525+0000 7f386717a740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5593a4996fe0 0x5593a49afb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:19:53.526 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:53.525+0000 7f386717a740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5593a499fdb0 0x5593a49913c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:19:53.526 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:53.525+0000 7f386717a740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f385c1af400 con 0x5593a49966c0 2024-03-31T23:19:53.527 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:53.525+0000 7f386717a740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f385c1acc40 con 0x5593a4996fe0 2024-03-31T23:19:53.527 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:53.525+0000 7f386717a740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f385c1aaab0 con 0x5593a499fdb0 2024-03-31T23:19:53.527 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:53.525+0000 7f3861b2e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5593a49966c0 0x5593a499c610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:19:53.527 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:53.525+0000 7f3861b2e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5593a49966c0 0x5593a499c610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:38620/0 (socket says 172.21.15.46:38620) 2024-03-31T23:19:53.527 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:53.525+0000 7f3861b2e640 1 -- 172.21.15.46:0/770998091 learned_addr learned my addr 172.21.15.46:0/770998091 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:19:53.527 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:53.525+0000 7f386232f640 1 --2- 172.21.15.46:0/770998091 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5593a4996fe0 0x5593a49afb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:19:53.527 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:53.525+0000 7f3861b2e640 1 --2- 172.21.15.46:0/770998091 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5593a499fdb0 0x5593a49913c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:19:53.527 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:53.525+0000 7f3861b2e640 1 -- 172.21.15.46:0/770998091 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5593a499fdb0 msgr2=0x5593a49913c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:19:53.527 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:53.525+0000 7f3861b2e640 1 --2- 172.21.15.46:0/770998091 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5593a499fdb0 0x5593a49913c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:19:53.527 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:53.525+0000 7f3861b2e640 1 -- 172.21.15.46:0/770998091 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5593a4996fe0 msgr2=0x5593a49afb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:19:53.527 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:53.525+0000 7f3861b2e640 1 --2- 172.21.15.46:0/770998091 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5593a4996fe0 0x5593a49afb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:19:53.527 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:53.525+0000 7f3861b2e640 1 -- 172.21.15.46:0/770998091 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5593a499b280 con 0x5593a49966c0 2024-03-31T23:19:53.527 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:53.525+0000 7f386232f640 1 --2- 172.21.15.46:0/770998091 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5593a4996fe0 0x5593a49afb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:19:53.527 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:53.526+0000 7f3861b2e640 1 --2- 172.21.15.46:0/770998091 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5593a49966c0 0x5593a499c610 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7f384c004440 tx=0x7f384c0043b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f8de94b6567431ab server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:19:53.528 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:53.526+0000 7f3847fff640 1 -- 172.21.15.46:0/770998091 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f384c025b20 con 0x5593a49966c0 2024-03-31T23:19:53.528 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:53.526+0000 7f3847fff640 1 -- 172.21.15.46:0/770998091 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f384c025cc0 con 0x5593a49966c0 2024-03-31T23:19:53.528 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:53.526+0000 7f3847fff640 1 -- 172.21.15.46:0/770998091 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f384c005590 con 0x5593a49966c0 2024-03-31T23:19:53.531 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:53.526+0000 7f386717a740 1 -- 172.21.15.46:0/770998091 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5593a49966c0 msgr2=0x5593a499c610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:19:53.531 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:53.526+0000 7f386717a740 1 --2- 172.21.15.46:0/770998091 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5593a49966c0 0x5593a499c610 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7f384c004440 tx=0x7f384c0043b0 comp rx=0 tx=0).stop 2024-03-31T23:19:53.531 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:53.526+0000 7f386717a740 1 -- 172.21.15.46:0/770998091 shutdown_connections 2024-03-31T23:19:53.531 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:53.526+0000 7f386717a740 1 --2- 172.21.15.46:0/770998091 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5593a499fdb0 0x5593a49913c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:19:53.531 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:53.526+0000 7f386717a740 1 --2- 172.21.15.46:0/770998091 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5593a4996fe0 0x5593a49afb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:19:53.532 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:53.526+0000 7f386717a740 1 --2- 172.21.15.46:0/770998091 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5593a49966c0 0x5593a499c610 unknown :-1 s=CLOSED pgs=767 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:19:53.532 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:53.526+0000 7f386717a740 1 -- 172.21.15.46:0/770998091 >> 172.21.15.46:0/770998091 conn(0x5593a4990fe0 msgr2=0x5593a498ff80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:19:53.532 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:53.527+0000 7f386717a740 1 -- 172.21.15.46:0/770998091 shutdown_connections 2024-03-31T23:19:53.532 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:53.527+0000 7f386717a740 1 -- 172.21.15.46:0/770998091 wait complete. 2024-03-31T23:19:53.532 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:53.527+0000 7f386717a740 1 -- start start 2024-03-31T23:19:53.532 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:53.527+0000 7f386717a740 1 Processor -- start 2024-03-31T23:19:53.532 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:53.527+0000 7f386717a740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5593a49966c0 0x5593a49ac160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:19:53.532 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:53.527+0000 7f386717a740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5593a4996fe0 0x5593a49991c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:19:53.532 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:53.527+0000 7f386717a740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5593a499fdb0 0x5593a4999700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:19:53.532 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:53.527+0000 7f386717a740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f385c1ab100 con 0x5593a49966c0 2024-03-31T23:19:53.532 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:53.527+0000 7f386717a740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f385c1ac570 con 0x5593a499fdb0 2024-03-31T23:19:53.532 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:53.527+0000 7f386717a740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f385c1ab7d0 con 0x5593a4996fe0 2024-03-31T23:19:53.532 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:53.527+0000 7f386717a740 10 client.0 _ll_register_callbacks cb 0x7f384c009680 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T23:19:53.532 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:53.527+0000 7f3861b2e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5593a49966c0 0x5593a49ac160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:19:53.533 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:53.527+0000 7f3861b2e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5593a49966c0 0x5593a49ac160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:38634/0 (socket says 172.21.15.46:38634) 2024-03-31T23:19:53.533 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:53.527+0000 7f3861b2e640 1 -- 172.21.15.46:0/3018840876 learned_addr learned my addr 172.21.15.46:0/3018840876 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:19:53.533 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:53.527+0000 7f386232f640 1 --2- 172.21.15.46:0/3018840876 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5593a4996fe0 0x5593a49991c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:19:53.533 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:53.527+0000 7f3861b2e640 1 --2- 172.21.15.46:0/3018840876 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5593a499fdb0 0x5593a4999700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:19:53.533 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:53.528+0000 7f386232f640 1 -- 172.21.15.46:0/3018840876 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5593a499fdb0 msgr2=0x5593a4999700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:19:53.533 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:53.528+0000 7f386232f640 1 --2- 172.21.15.46:0/3018840876 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5593a499fdb0 0x5593a4999700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:19:53.533 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:53.528+0000 7f386232f640 1 -- 172.21.15.46:0/3018840876 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5593a49966c0 msgr2=0x5593a49ac160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:19:53.533 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:53.528+0000 7f386232f640 1 --2- 172.21.15.46:0/3018840876 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5593a49966c0 0x5593a49ac160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:19:53.533 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:53.528+0000 7f386232f640 1 -- 172.21.15.46:0/3018840876 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5593a4999c40 con 0x5593a4996fe0 2024-03-31T23:19:53.533 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:53.528+0000 7f3861b2e640 1 --2- 172.21.15.46:0/3018840876 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5593a49966c0 0x5593a49ac160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:19:53.533 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:53.528+0000 7f386232f640 1 --2- 172.21.15.46:0/3018840876 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5593a4996fe0 0x5593a49991c0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f3858005ab0 tx=0x7f38580035f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4115d6807d150c01 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:19:53.533 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:53.528+0000 7f3845ffb640 10 client.0 ms_handle_connect on v2:172.21.15.162:3300/0 2024-03-31T23:19:53.533 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:53.528+0000 7f3845ffb640 1 -- 172.21.15.46:0/3018840876 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3858045040 con 0x5593a4996fe0 2024-03-31T23:19:53.533 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:53.528+0000 7f3845ffb640 1 -- 172.21.15.46:0/3018840876 <== mon.1 v2:172.21.15.162:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9554+0+0 (secure 0 0 0) 0x7f385804a660 con 0x5593a4996fe0 2024-03-31T23:19:53.533 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:53.528+0000 7f386717a740 1 -- 172.21.15.46:0/3018840876 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f384c025cc0 con 0x5593a4996fe0 2024-03-31T23:19:53.534 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:53.528+0000 7f3845ffb640 1 -- 172.21.15.46:0/3018840876 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f385805f050 con 0x5593a4996fe0 2024-03-31T23:19:53.534 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:53.528+0000 7f386717a740 10 client.26554 fetch_fsmap learned FSMap version 35 2024-03-31T23:19:53.534 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:53.528+0000 7f386717a740 1 -- 172.21.15.46:0/3018840876 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x5593a4a1a120 con 0x5593a4996fe0 2024-03-31T23:19:53.534 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:53.528+0000 7f3845ffb640 1 -- 172.21.15.46:0/3018840876 <== mon.1 v2:172.21.15.162:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f384c025cc0 con 0x5593a4996fe0 2024-03-31T23:19:53.534 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:53.529+0000 7f3845ffb640 1 -- 172.21.15.46:0/3018840876 <== mon.1 v2:172.21.15.162:3300/0 5 ==== fsmap(e 35) ==== 8366+0+0 (secure 0 0 0) 0x7f3858034440 con 0x5593a4996fe0 2024-03-31T23:19:53.534 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:53.529+0000 7f386717a740 10 client.26554 fetch_fsmap finished waiting for FSMap version 35 2024-03-31T23:19:53.534 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:53.529+0000 7f386717a740 10 client.26554 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-31T23:19:53.534 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:53.529+0000 7f386717a740 20 client.26554 populate_metadata read hostname 'smithi046' 2024-03-31T23:19:53.534 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:53.529+0000 7f386717a740 1 --2- 172.21.15.46:0/3018840876 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x5593a4a1a4b0 0x5593a4a1c8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:19:53.534 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:53.529+0000 7f386717a740 4 client.26554 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-31T23:19:53.534 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:53.529+0000 7f386717a740 1 -- 172.21.15.46:0/3018840876 --> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5593a4906e30 con 0x5593a4a1a4b0 2024-03-31T23:19:53.534 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:53.529+0000 7f3861b2e640 1 --2- 172.21.15.46:0/3018840876 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x5593a4a1a4b0 0x5593a4a1c8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:19:53.534 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:53.529+0000 7f3861b2e640 1 --2- 172.21.15.46:0/3018840876 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x5593a4a1a4b0 0x5593a4a1c8d0 crc :-1 s=READY pgs=148 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bf11cf90585fc273 server_cookie=59b6474b352da0f2 in_seq=0 out_seq=0 2024-03-31T23:19:53.534 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:53.529+0000 7f3845ffb640 10 client.26554 ms_handle_connect on v2:172.21.15.142:6838/2519143918 2024-03-31T23:19:53.553 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:53.551+0000 7f3845ffb640 1 -- 172.21.15.46:0/3018840876 <== mds.0 v2:172.21.15.142:6838/2519143918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5593a4906e30 con 0x5593a4a1a4b0 2024-03-31T23:19:53.553 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:53.551+0000 7f3845ffb640 10 client.26554 handle_command_reply: tid=0 2024-03-31T23:19:53.553 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:53.551+0000 7f386717a740 1 client.26554 shutdown 2024-03-31T23:19:53.553 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:53.552+0000 7f386717a740 1 -- 172.21.15.46:0/3018840876 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5593a4996fe0 msgr2=0x5593a49991c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:19:53.553 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:53.552+0000 7f386717a740 1 --2- 172.21.15.46:0/3018840876 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5593a4996fe0 0x5593a49991c0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f3858005ab0 tx=0x7f38580035f0 comp rx=0 tx=0).stop 2024-03-31T23:19:53.553 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:53.552+0000 7f386717a740 1 -- 172.21.15.46:0/3018840876 shutdown_connections 2024-03-31T23:19:53.553 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:53.552+0000 7f386717a740 1 --2- 172.21.15.46:0/3018840876 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x5593a4a1a4b0 0x5593a4a1c8d0 crc :-1 s=READY pgs=148 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:19:53.553 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:53.552+0000 7f386717a740 1 --2- 172.21.15.46:0/3018840876 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5593a499fdb0 0x5593a4999700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:19:53.553 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:53.552+0000 7f386717a740 1 --2- 172.21.15.46:0/3018840876 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5593a4996fe0 0x5593a49991c0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:19:53.554 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:53.552+0000 7f3845ffb640 0 client.26554 ms_handle_reset on v2:172.21.15.142:6838/2519143918 2024-03-31T23:19:53.554 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:53.552+0000 7f386717a740 1 --2- 172.21.15.46:0/3018840876 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5593a49966c0 0x5593a49ac160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:19:53.554 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:53.552+0000 7f386717a740 1 -- 172.21.15.46:0/3018840876 >> 172.21.15.46:0/3018840876 conn(0x5593a4990fe0 msgr2=0x5593a498ff80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:19:53.554 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:53.552+0000 7f386717a740 1 -- 172.21.15.46:0/3018840876 shutdown_connections 2024-03-31T23:19:53.554 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:53.552+0000 7f386717a740 1 -- 172.21.15.46:0/3018840876 wait complete. 2024-03-31T23:19:53.554 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:53.552+0000 7f386717a740 20 client.26554 trim_cache size 0 max 16384 2024-03-31T23:19:53.554 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout:{ 2024-03-31T23:19:53.554 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout: "message": "", 2024-03-31T23:19:53.554 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout: "return_code": 0 2024-03-31T23:19:53.555 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout:} 2024-03-31T23:19:53.555 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:53.552+0000 7f3864593640 1 -- 172.21.15.46:0/1891346254 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f38480a26b0 msgr2=0x7f38480a4b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:19:53.555 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:53.552+0000 7f3864593640 1 --2- 172.21.15.46:0/1891346254 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f38480a26b0 0x7f38480a4b70 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f38540cc9a0 tx=0x7f385406e990 comp rx=0 tx=0).stop 2024-03-31T23:19:53.555 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:53.552+0000 7f3864593640 1 -- 172.21.15.46:0/1891346254 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f385c107750 msgr2=0x7f385c19c780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:19:53.555 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:53.552+0000 7f3864593640 1 --2- 172.21.15.46:0/1891346254 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f385c107750 0x7f385c19c780 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7f385400eb10 tx=0x7f385400c2d0 comp rx=0 tx=0).stop 2024-03-31T23:19:53.555 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:53.552+0000 7f3864593640 1 -- 172.21.15.46:0/1891346254 shutdown_connections 2024-03-31T23:19:53.556 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:53.552+0000 7f3864593640 1 --2- 172.21.15.46:0/1891346254 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f38480a26b0 0x7f38480a4b70 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:19:53.556 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:53.552+0000 7f3864593640 1 --2- 172.21.15.46:0/1891346254 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f385c107750 0x7f385c19c780 unknown :-1 s=CLOSED pgs=765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:19:53.556 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:53.552+0000 7f3864593640 1 --2- 172.21.15.46:0/1891346254 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f385c100630 0x7f385c198400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:19:53.556 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:53.552+0000 7f3864593640 1 --2- 172.21.15.46:0/1891346254 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f385c0ffc90 0x7f385c197ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:19:53.556 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:53.552+0000 7f3864593640 1 -- 172.21.15.46:0/1891346254 >> 172.21.15.46:0/1891346254 conn(0x7f385c076ba0 msgr2=0x7f385c0fe270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:19:53.556 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:53.552+0000 7f3864593640 1 -- 172.21.15.46:0/1891346254 shutdown_connections 2024-03-31T23:19:53.556 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:19:53.553+0000 7f3864593640 1 -- 172.21.15.46:0/1891346254 wait complete. 2024-03-31T23:19:54.205 INFO:tasks.workunit.client.0.smithi046.stdout: 1 181451 2.29 MB/sec execute 300 sec latency 15.552 ms 2024-03-31T23:19:54.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:19:54 smithi142 ceph-mon[35271]: pgmap v252: 129 pgs: 129 active+clean; 109 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 342 KiB/s rd, 853 KiB/s wr, 85 op/s 2024-03-31T23:19:55.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:19:54 smithi162 ceph-mon[36237]: pgmap v252: 129 pgs: 129 active+clean; 109 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 342 KiB/s rd, 853 KiB/s wr, 85 op/s 2024-03-31T23:19:55.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:19:54 smithi046 ceph-mon[32498]: pgmap v252: 129 pgs: 129 active+clean; 109 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 342 KiB/s rd, 853 KiB/s wr, 85 op/s 2024-03-31T23:19:55.205 INFO:tasks.workunit.client.0.smithi046.stdout: 1 181853 2.29 MB/sec execute 301 sec latency 15.841 ms 2024-03-31T23:19:55.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:19:55 smithi142 ceph-mon[35271]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-31T23:19:55.944 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:19:55 smithi142 ceph-mon[35271]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:19:55.944 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:19:55 smithi142 ceph-mon[35271]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-31T23:19:55.944 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:19:55 smithi142 ceph-mon[35271]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' 2024-03-31T23:19:56.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:19:55 smithi162 ceph-mon[36237]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-31T23:19:56.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:19:55 smithi162 ceph-mon[36237]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:19:56.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:19:55 smithi162 ceph-mon[36237]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-31T23:19:56.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:19:55 smithi162 ceph-mon[36237]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' 2024-03-31T23:19:56.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:19:55 smithi046 ceph-mon[32498]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-31T23:19:56.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:19:55 smithi046 ceph-mon[32498]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:19:56.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:19:55 smithi046 ceph-mon[32498]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-31T23:19:56.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:19:55 smithi046 ceph-mon[32498]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' 2024-03-31T23:19:56.205 INFO:tasks.workunit.client.0.smithi046.stdout: 1 182338 2.29 MB/sec execute 302 sec latency 16.989 ms 2024-03-31T23:19:56.944 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:19:56 smithi142 ceph-mon[35271]: pgmap v253: 129 pgs: 129 active+clean; 107 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 376 KiB/s rd, 1.0 MiB/s wr, 107 op/s 2024-03-31T23:19:57.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:19:56 smithi162 ceph-mon[36237]: pgmap v253: 129 pgs: 129 active+clean; 107 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 376 KiB/s rd, 1.0 MiB/s wr, 107 op/s 2024-03-31T23:19:57.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:19:56 smithi046 ceph-mon[32498]: pgmap v253: 129 pgs: 129 active+clean; 107 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 376 KiB/s rd, 1.0 MiB/s wr, 107 op/s 2024-03-31T23:19:57.205 INFO:tasks.workunit.client.0.smithi046.stdout: 1 182803 2.29 MB/sec execute 303 sec latency 23.176 ms 2024-03-31T23:19:58.205 INFO:tasks.workunit.client.0.smithi046.stdout: 1 183346 2.29 MB/sec execute 304 sec latency 23.236 ms 2024-03-31T23:19:58.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:19:58 smithi142 ceph-mon[35271]: pgmap v254: 129 pgs: 129 active+clean; 107 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 148 KiB/s rd, 514 KiB/s wr, 49 op/s 2024-03-31T23:19:59.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:19:58 smithi162 ceph-mon[36237]: pgmap v254: 129 pgs: 129 active+clean; 107 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 148 KiB/s rd, 514 KiB/s wr, 49 op/s 2024-03-31T23:19:59.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:19:58 smithi046 ceph-mon[32498]: pgmap v254: 129 pgs: 129 active+clean; 107 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 148 KiB/s rd, 514 KiB/s wr, 49 op/s 2024-03-31T23:19:59.205 INFO:tasks.workunit.client.0.smithi046.stdout: 1 184099 2.30 MB/sec execute 305 sec latency 15.409 ms 2024-03-31T23:20:00.205 INFO:tasks.workunit.client.0.smithi046.stdout: 1 184454 2.30 MB/sec execute 306 sec latency 16.094 ms 2024-03-31T23:20:00.361 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:20:00 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mon-a[32494]: 2024-03-31T23:19:59.998+0000 7fdb0b79b700 -1 log_channel(cluster) log [ERR] : Health detail: HEALTH_ERR 1 MDSs report damaged metadata 2024-03-31T23:20:00.362 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:20:00 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mon-a[32494]: 2024-03-31T23:19:59.998+0000 7fdb0b79b700 -1 log_channel(cluster) log [ERR] : [ERR] MDS_DAMAGE: 1 MDSs report damaged metadata 2024-03-31T23:20:00.362 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:20:00 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mon-a[32494]: 2024-03-31T23:19:59.998+0000 7fdb0b79b700 -1 log_channel(cluster) log [ERR] : mds.c(mds.2): Metadata damage detected 2024-03-31T23:20:00.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:20:00 smithi142 ceph-mon[35271]: pgmap v255: 129 pgs: 129 active+clean; 107 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 379 KiB/s rd, 1.0 MiB/s wr, 103 op/s 2024-03-31T23:20:00.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:20:00 smithi142 ceph-mon[35271]: Health detail: HEALTH_ERR 1 MDSs report damaged metadata 2024-03-31T23:20:00.944 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:20:00 smithi142 ceph-mon[35271]: [ERR] MDS_DAMAGE: 1 MDSs report damaged metadata 2024-03-31T23:20:00.944 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:20:00 smithi142 ceph-mon[35271]: mds.c(mds.2): Metadata damage detected 2024-03-31T23:20:01.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:20:00 smithi162 ceph-mon[36237]: pgmap v255: 129 pgs: 129 active+clean; 107 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 379 KiB/s rd, 1.0 MiB/s wr, 103 op/s 2024-03-31T23:20:01.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:20:00 smithi162 ceph-mon[36237]: Health detail: HEALTH_ERR 1 MDSs report damaged metadata 2024-03-31T23:20:01.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:20:00 smithi162 ceph-mon[36237]: [ERR] MDS_DAMAGE: 1 MDSs report damaged metadata 2024-03-31T23:20:01.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:20:00 smithi162 ceph-mon[36237]: mds.c(mds.2): Metadata damage detected 2024-03-31T23:20:01.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:20:00 smithi046 ceph-mon[32498]: pgmap v255: 129 pgs: 129 active+clean; 107 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 379 KiB/s rd, 1.0 MiB/s wr, 103 op/s 2024-03-31T23:20:01.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:20:00 smithi046 ceph-mon[32498]: Health detail: HEALTH_ERR 1 MDSs report damaged metadata 2024-03-31T23:20:01.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:20:00 smithi046 ceph-mon[32498]: [ERR] MDS_DAMAGE: 1 MDSs report damaged metadata 2024-03-31T23:20:01.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:20:00 smithi046 ceph-mon[32498]: mds.c(mds.2): Metadata damage detected 2024-03-31T23:20:01.205 INFO:tasks.workunit.client.0.smithi046.stdout: 1 184528 2.30 MB/sec execute 307 sec latency 871.071 ms 2024-03-31T23:20:02.205 INFO:tasks.workunit.client.0.smithi046.stdout: 1 184528 2.29 MB/sec execute 308 sec latency 1871.130 ms 2024-03-31T23:20:02.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:20:02 smithi142 ceph-mon[35271]: pgmap v256: 129 pgs: 129 active+clean; 107 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 266 KiB/s rd, 729 KiB/s wr, 76 op/s 2024-03-31T23:20:03.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:20:02 smithi162 ceph-mon[36237]: pgmap v256: 129 pgs: 129 active+clean; 107 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 266 KiB/s rd, 729 KiB/s wr, 76 op/s 2024-03-31T23:20:03.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:20:02 smithi046 ceph-mon[32498]: pgmap v256: 129 pgs: 129 active+clean; 107 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 266 KiB/s rd, 729 KiB/s wr, 76 op/s 2024-03-31T23:20:03.205 INFO:tasks.workunit.client.0.smithi046.stdout: 1 184595 2.28 MB/sec execute 309 sec latency 2406.893 ms 2024-03-31T23:20:04.205 INFO:tasks.workunit.client.0.smithi046.stdout: 1 184595 2.28 MB/sec execute 310 sec latency 1212.753 ms 2024-03-31T23:20:04.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:20:04 smithi142 ceph-mon[35271]: pgmap v257: 129 pgs: 129 active+clean; 107 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 266 KiB/s rd, 729 KiB/s wr, 76 op/s 2024-03-31T23:20:05.053 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:20:04 smithi162 ceph-mon[36237]: pgmap v257: 129 pgs: 129 active+clean; 107 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 266 KiB/s rd, 729 KiB/s wr, 76 op/s 2024-03-31T23:20:05.114 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:20:04 smithi046 ceph-mon[32498]: pgmap v257: 129 pgs: 129 active+clean; 107 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 266 KiB/s rd, 729 KiB/s wr, 76 op/s 2024-03-31T23:20:05.205 INFO:tasks.workunit.client.0.smithi046.stdout: 1 184595 2.27 MB/sec execute 311 sec latency 2212.833 ms 2024-03-31T23:20:06.205 INFO:tasks.workunit.client.0.smithi046.stdout: 1 184595 2.26 MB/sec execute 312 sec latency 3212.867 ms 2024-03-31T23:20:06.656 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:06.651+0000 7f764fe30640 1 -- 172.21.15.46:0/1621867357 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f76481009a0 msgr2=0x7f7648100e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:20:06.656 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:06.651+0000 7f764fe30640 1 --2- 172.21.15.46:0/1621867357 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f76481009a0 0x7f7648100e20 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7f7638009930 tx=0x7f763802bc30 comp rx=0 tx=0).stop 2024-03-31T23:20:06.656 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:06.651+0000 7f764fe30640 1 -- 172.21.15.46:0/1621867357 shutdown_connections 2024-03-31T23:20:06.656 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:06.651+0000 7f764fe30640 1 --2- 172.21.15.46:0/1621867357 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7648101360 0x7f764810d430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:20:06.656 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:06.651+0000 7f764fe30640 1 --2- 172.21.15.46:0/1621867357 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f76481009a0 0x7f7648100e20 unknown :-1 s=CLOSED pgs=768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:20:06.657 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:06.651+0000 7f764fe30640 1 --2- 172.21.15.46:0/1621867357 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7648105630 0x7f7648105a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:20:06.657 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:06.651+0000 7f764fe30640 1 -- 172.21.15.46:0/1621867357 >> 172.21.15.46:0/1621867357 conn(0x7f7648076450 msgr2=0x7f7648076860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:20:06.657 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:06.651+0000 7f764fe30640 1 -- 172.21.15.46:0/1621867357 shutdown_connections 2024-03-31T23:20:06.657 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:06.651+0000 7f764fe30640 1 -- 172.21.15.46:0/1621867357 wait complete. 2024-03-31T23:20:06.657 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:06.652+0000 7f764fe30640 1 Processor -- start 2024-03-31T23:20:06.657 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:06.652+0000 7f764fe30640 1 -- start start 2024-03-31T23:20:06.657 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:06.652+0000 7f764fe30640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f76481009a0 0x7f7648198d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:20:06.657 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:06.652+0000 7f764fe30640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7648101360 0x7f7648199270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:20:06.657 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:06.652+0000 7f764fe30640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7648105630 0x7f76481a3db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:20:06.658 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:06.652+0000 7f764fe30640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f76481108d0 con 0x7f7648101360 2024-03-31T23:20:06.658 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:06.652+0000 7f764fe30640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f7648110750 con 0x7f7648105630 2024-03-31T23:20:06.658 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:06.652+0000 7f764fe30640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f7648110a50 con 0x7f76481009a0 2024-03-31T23:20:06.658 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:06.652+0000 7f764d3cb640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7648101360 0x7f7648199270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:20:06.658 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:06.652+0000 7f764d3cb640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7648101360 0x7f7648199270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40366/0 (socket says 172.21.15.46:40366) 2024-03-31T23:20:06.658 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:06.652+0000 7f764e3cd640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7648105630 0x7f76481a3db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:20:06.658 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:06.652+0000 7f764d3cb640 1 -- 172.21.15.46:0/3241652662 learned_addr learned my addr 172.21.15.46:0/3241652662 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:20:06.658 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:06.652+0000 7f764dbcc640 1 --2- 172.21.15.46:0/3241652662 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f76481009a0 0x7f7648198d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:20:06.659 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:06.652+0000 7f764d3cb640 1 -- 172.21.15.46:0/3241652662 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f76481009a0 msgr2=0x7f7648198d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:20:06.659 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:06.652+0000 7f764d3cb640 1 --2- 172.21.15.46:0/3241652662 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f76481009a0 0x7f7648198d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:20:06.659 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:06.652+0000 7f764d3cb640 1 -- 172.21.15.46:0/3241652662 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7648105630 msgr2=0x7f76481a3db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:20:06.659 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:06.652+0000 7f764d3cb640 1 --2- 172.21.15.46:0/3241652662 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7648105630 0x7f76481a3db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:20:06.659 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:06.652+0000 7f764d3cb640 1 -- 172.21.15.46:0/3241652662 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f76481a42f0 con 0x7f7648101360 2024-03-31T23:20:06.659 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:06.652+0000 7f764e3cd640 1 --2- 172.21.15.46:0/3241652662 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7648105630 0x7f76481a3db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:20:06.659 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:06.653+0000 7f764d3cb640 1 --2- 172.21.15.46:0/3241652662 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7648101360 0x7f7648199270 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7f7638005940 tx=0x7f7638005dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:20:06.659 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:06.653+0000 7f763effd640 1 -- 172.21.15.46:0/3241652662 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7638043020 con 0x7f7648101360 2024-03-31T23:20:06.659 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:06.653+0000 7f764fe30640 1 -- 172.21.15.46:0/3241652662 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f76481a44c0 con 0x7f7648101360 2024-03-31T23:20:06.660 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:06.653+0000 7f763effd640 1 -- 172.21.15.46:0/3241652662 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7638004350 con 0x7f7648101360 2024-03-31T23:20:06.660 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:06.653+0000 7f763effd640 1 -- 172.21.15.46:0/3241652662 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7638003950 con 0x7f7648101360 2024-03-31T23:20:06.660 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:06.653+0000 7f764fe30640 1 -- 172.21.15.46:0/3241652662 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f76481a4980 con 0x7f7648101360 2024-03-31T23:20:06.669 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:06.655+0000 7f763effd640 1 -- 172.21.15.46:0/3241652662 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 25) ==== 137451+0+0 (secure 0 0 0) 0x7f7638046020 con 0x7f7648101360 2024-03-31T23:20:06.669 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:06.656+0000 7f7652a0b740 1 Processor -- start 2024-03-31T23:20:06.669 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:06.656+0000 7f7652a0b740 1 -- start start 2024-03-31T23:20:06.669 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:06.656+0000 7f7652a0b740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55adc0856160 0x55adc0856540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:20:06.670 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:06.656+0000 7f7652a0b740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55adc0856a80 0x55adc0863c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:20:06.670 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:06.656+0000 7f7652a0b740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55adc08641c0 0x55adc08665b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:20:06.670 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:06.656+0000 7f7652a0b740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55adc07b98f0 con 0x55adc08641c0 2024-03-31T23:20:06.670 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:06.656+0000 7f7652a0b740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55adc07628d0 con 0x55adc0856a80 2024-03-31T23:20:06.670 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:06.656+0000 7f7652a0b740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55adc07b5990 con 0x55adc0856160 2024-03-31T23:20:06.670 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:06.656+0000 7f763effd640 1 --2- 172.21.15.46:0/3241652662 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f76340a26b0 0x7f76340a4b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:20:06.670 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:06.656+0000 7f764d3cb640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55adc08641c0 0x55adc08665b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:20:06.670 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:06.656+0000 7f764dbcc640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55adc0856a80 0x55adc0863c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:20:06.670 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:06.656+0000 7f764dbcc640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55adc0856a80 0x55adc0863c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.46:54800/0 (socket says 172.21.15.46:54800) 2024-03-31T23:20:06.670 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:06.656+0000 7f764e3cd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55adc0856160 0x55adc0856540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:20:06.671 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:06.656+0000 7f764dbcc640 1 -- 172.21.15.46:0/597810742 learned_addr learned my addr 172.21.15.46:0/597810742 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:20:06.671 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:06.656+0000 7f764e3cd640 1 --2- 172.21.15.46:0/3241652662 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f76340a26b0 0x7f76340a4b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:20:06.671 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:06.657+0000 7f763effd640 1 -- 172.21.15.46:0/3241652662 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9554+0+0 (secure 0 0 0) 0x7f76380f0800 con 0x7f7648101360 2024-03-31T23:20:06.671 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:06.657+0000 7f764dbcc640 1 -- 172.21.15.46:0/597810742 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55adc0856160 msgr2=0x55adc0856540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:20:06.671 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:06.657+0000 7f764dbcc640 1 --2- 172.21.15.46:0/597810742 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55adc0856160 0x55adc0856540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:20:06.671 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:06.657+0000 7f764dbcc640 1 -- 172.21.15.46:0/597810742 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55adc08641c0 msgr2=0x55adc08665b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:20:06.671 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:06.657+0000 7f764dbcc640 1 --2- 172.21.15.46:0/597810742 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55adc08641c0 0x55adc08665b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:20:06.671 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:06.657+0000 7f764dbcc640 1 -- 172.21.15.46:0/597810742 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55adc04ce210 con 0x55adc0856a80 2024-03-31T23:20:06.671 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:06.657+0000 7f764e3cd640 1 --2- 172.21.15.46:0/597810742 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55adc0856160 0x55adc0856540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T23:20:06.672 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:06.657+0000 7f764dbcc640 1 --2- 172.21.15.46:0/597810742 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55adc0856a80 0x55adc0863c80 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f7644009ca0 tx=0x7f7644011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=94849ae4c99a5915 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:20:06.672 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:06.657+0000 7f764e3cd640 1 --2- 172.21.15.46:0/3241652662 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f76340a26b0 0x7f76340a4b70 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f7640009b80 tx=0x7f7640009500 comp rx=0 tx=0).ready entity=mgr.15654 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:20:06.672 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:06.657+0000 7f7633fff640 1 -- 172.21.15.46:0/597810742 <== mon.2 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7644004250 con 0x55adc0856a80 2024-03-31T23:20:06.672 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:06.657+0000 7f7633fff640 1 -- 172.21.15.46:0/597810742 <== mon.2 v2:172.21.15.142:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f76440043f0 con 0x55adc0856a80 2024-03-31T23:20:06.672 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:06.657+0000 7f7633fff640 1 -- 172.21.15.46:0/597810742 <== mon.2 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76440197c0 con 0x55adc0856a80 2024-03-31T23:20:06.672 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:06.658+0000 7f7652a0b740 1 -- 172.21.15.46:0/597810742 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55adc0856a80 msgr2=0x55adc0863c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:20:06.672 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:06.658+0000 7f7652a0b740 1 --2- 172.21.15.46:0/597810742 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55adc0856a80 0x55adc0863c80 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f7644009ca0 tx=0x7f7644011810 comp rx=0 tx=0).stop 2024-03-31T23:20:06.672 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:06.658+0000 7f7652a0b740 1 -- 172.21.15.46:0/597810742 shutdown_connections 2024-03-31T23:20:06.672 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:06.658+0000 7f7652a0b740 1 --2- 172.21.15.46:0/597810742 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55adc08641c0 0x55adc08665b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:20:06.672 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:06.658+0000 7f7652a0b740 1 --2- 172.21.15.46:0/597810742 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55adc0856a80 0x55adc0863c80 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:20:06.673 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:06.658+0000 7f7652a0b740 1 --2- 172.21.15.46:0/597810742 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55adc0856160 0x55adc0856540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:20:06.673 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:06.658+0000 7f7652a0b740 1 -- 172.21.15.46:0/597810742 >> 172.21.15.46:0/597810742 conn(0x55adc0850fe0 msgr2=0x55adc08513c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:20:06.673 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:06.658+0000 7f7652a0b740 1 -- 172.21.15.46:0/597810742 shutdown_connections 2024-03-31T23:20:06.673 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:06.658+0000 7f7652a0b740 1 -- 172.21.15.46:0/597810742 wait complete. 2024-03-31T23:20:06.673 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:06.658+0000 7f7652a0b740 1 -- start start 2024-03-31T23:20:06.673 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:06.658+0000 7f7652a0b740 1 Processor -- start 2024-03-31T23:20:06.673 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:06.659+0000 7f7652a0b740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55adc0856160 0x55adc08611b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:20:06.673 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:06.659+0000 7f7652a0b740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55adc0856a80 0x55adc08616f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:20:06.674 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:06.659+0000 7f7652a0b740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55adc08641c0 0x55adc0861c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:20:06.674 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:06.659+0000 7f7652a0b740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55adc0866df0 con 0x55adc0856160 2024-03-31T23:20:06.674 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:06.659+0000 7f7652a0b740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55adc0757f80 con 0x55adc08641c0 2024-03-31T23:20:06.674 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:06.659+0000 7f7652a0b740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55adc0866f70 con 0x55adc0856a80 2024-03-31T23:20:06.674 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:06.659+0000 7f7652a0b740 10 client.0 _ll_register_callbacks cb 0x55adc07b0290 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T23:20:06.674 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:06.659+0000 7f764dbcc640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55adc0856160 0x55adc08611b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:20:06.675 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:06.659+0000 7f764dbcc640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55adc0856160 0x55adc08611b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40390/0 (socket says 172.21.15.46:40390) 2024-03-31T23:20:06.675 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:06.659+0000 7f764dbcc640 1 -- 172.21.15.46:0/3741104149 learned_addr learned my addr 172.21.15.46:0/3741104149 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:20:06.675 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:06.659+0000 7f764d3cb640 1 --2- 172.21.15.46:0/3741104149 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55adc0856a80 0x55adc08616f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:20:06.675 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:06.659+0000 7f764e3cd640 1 --2- 172.21.15.46:0/3741104149 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55adc08641c0 0x55adc0861c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:20:06.675 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:06.659+0000 7f764dbcc640 1 -- 172.21.15.46:0/3741104149 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55adc0856a80 msgr2=0x55adc08616f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:20:06.675 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:06.659+0000 7f764dbcc640 1 --2- 172.21.15.46:0/3741104149 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55adc0856a80 0x55adc08616f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:20:06.675 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:06.659+0000 7f764dbcc640 1 -- 172.21.15.46:0/3741104149 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55adc08641c0 msgr2=0x55adc0861c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:20:06.675 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:06.659+0000 7f764dbcc640 1 --2- 172.21.15.46:0/3741104149 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55adc08641c0 0x55adc0861c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:20:06.675 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:06.659+0000 7f764dbcc640 1 -- 172.21.15.46:0/3741104149 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55adc059a620 con 0x55adc0856160 2024-03-31T23:20:06.676 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:06.660+0000 7f764dbcc640 1 --2- 172.21.15.46:0/3741104149 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55adc0856160 0x55adc08611b0 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7f7644003f80 tx=0x7f7644005d30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=96d13814282f19ac server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:20:06.676 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:06.660+0000 7f76317fa640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-31T23:20:06.676 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:06.660+0000 7f76317fa640 1 -- 172.21.15.46:0/3741104149 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7644027020 con 0x55adc0856160 2024-03-31T23:20:06.676 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:06.660+0000 7f76317fa640 1 -- 172.21.15.46:0/3741104149 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9554+0+0 (secure 0 0 0) 0x7f7644003800 con 0x55adc0856160 2024-03-31T23:20:06.676 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:06.660+0000 7f7652a0b740 1 -- 172.21.15.46:0/3741104149 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f76440043f0 con 0x55adc0856160 2024-03-31T23:20:06.676 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:06.660+0000 7f76317fa640 1 -- 172.21.15.46:0/3741104149 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f7644032050 con 0x55adc0856160 2024-03-31T23:20:06.678 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:06.660+0000 7f7652a0b740 10 client.16830 fetch_fsmap learned FSMap version 35 2024-03-31T23:20:06.678 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:06.660+0000 7f7652a0b740 1 -- 172.21.15.46:0/3741104149 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x55adc0872cf0 con 0x55adc0856160 2024-03-31T23:20:06.678 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:06.660+0000 7f76317fa640 1 -- 172.21.15.46:0/3741104149 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f76440043f0 con 0x55adc0856160 2024-03-31T23:20:06.678 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:06.660+0000 7f76317fa640 1 -- 172.21.15.46:0/3741104149 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 35) ==== 8366+0+0 (secure 0 0 0) 0x7f7644018070 con 0x55adc0856160 2024-03-31T23:20:06.678 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:06.660+0000 7f7652a0b740 10 client.16830 fetch_fsmap finished waiting for FSMap version 35 2024-03-31T23:20:06.678 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:06.660+0000 7f7652a0b740 10 client.16830 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-31T23:20:06.678 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:06.660+0000 7f7652a0b740 20 client.16830 populate_metadata read hostname 'smithi046' 2024-03-31T23:20:06.679 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:06.660+0000 7f7652a0b740 1 --2- 172.21.15.46:0/3741104149 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55adc08740f0 0x55adc08764e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:20:06.679 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:06.660+0000 7f7652a0b740 4 client.16830 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-31T23:20:06.679 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:06.660+0000 7f7652a0b740 1 -- 172.21.15.46:0/3741104149 --> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55adc07cd120 con 0x55adc08740f0 2024-03-31T23:20:06.679 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:06.661+0000 7f764d3cb640 1 --2- 172.21.15.46:0/3741104149 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55adc08740f0 0x55adc08764e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:20:06.679 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:06.661+0000 7f76317fa640 10 client.16830 ms_handle_connect on v2:172.21.15.142:6838/2519143918 2024-03-31T23:20:06.679 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:06.661+0000 7f764d3cb640 1 --2- 172.21.15.46:0/3741104149 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55adc08740f0 0x55adc08764e0 crc :-1 s=READY pgs=149 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=69aed2c18263a515 server_cookie=8729ca1cf4839992 in_seq=0 out_seq=0 2024-03-31T23:20:06.679 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:06.662+0000 7f76317fa640 1 -- 172.21.15.46:0/3741104149 <== mds.0 v2:172.21.15.142:6838/2519143918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55adc07cd120 con 0x55adc08740f0 2024-03-31T23:20:06.679 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:06.662+0000 7f76317fa640 10 client.16830 handle_command_reply: tid=0 2024-03-31T23:20:06.679 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:06.662+0000 7f7652a0b740 1 client.16830 shutdown 2024-03-31T23:20:06.680 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:06.662+0000 7f7652a0b740 1 -- 172.21.15.46:0/3741104149 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55adc0856160 msgr2=0x55adc08611b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:20:06.680 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:06.662+0000 7f7652a0b740 1 --2- 172.21.15.46:0/3741104149 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55adc0856160 0x55adc08611b0 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7f7644003f80 tx=0x7f7644005d30 comp rx=0 tx=0).stop 2024-03-31T23:20:06.680 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:06.662+0000 7f7652a0b740 1 -- 172.21.15.46:0/3741104149 shutdown_connections 2024-03-31T23:20:06.680 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:06.662+0000 7f7652a0b740 1 --2- 172.21.15.46:0/3741104149 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55adc08740f0 0x55adc08764e0 crc :-1 s=READY pgs=149 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:20:06.680 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:06.662+0000 7f7652a0b740 1 --2- 172.21.15.46:0/3741104149 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55adc08641c0 0x55adc0861c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:20:06.680 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:06.662+0000 7f7652a0b740 1 --2- 172.21.15.46:0/3741104149 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55adc0856a80 0x55adc08616f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:20:06.680 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:06.662+0000 7f7652a0b740 1 --2- 172.21.15.46:0/3741104149 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55adc0856160 0x55adc08611b0 unknown :-1 s=CLOSED pgs=770 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:20:06.680 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:06.662+0000 7f76317fa640 0 client.16830 ms_handle_reset on v2:172.21.15.142:6838/2519143918 2024-03-31T23:20:06.680 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:06.662+0000 7f7652a0b740 1 -- 172.21.15.46:0/3741104149 >> 172.21.15.46:0/3741104149 conn(0x55adc0850fe0 msgr2=0x55adc0852d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:20:06.680 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:06.662+0000 7f7652a0b740 1 -- 172.21.15.46:0/3741104149 shutdown_connections 2024-03-31T23:20:06.680 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:06.662+0000 7f7652a0b740 1 -- 172.21.15.46:0/3741104149 wait complete. 2024-03-31T23:20:06.680 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:06.662+0000 7f7652a0b740 20 client.16830 trim_cache size 0 max 16384 2024-03-31T23:20:06.687 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:06.677+0000 7f7652a0b740 1 Processor -- start 2024-03-31T23:20:06.687 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:06.677+0000 7f7652a0b740 1 -- start start 2024-03-31T23:20:06.687 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:06.677+0000 7f7652a0b740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55adc0856160 0x55adc08638e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:20:06.687 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:06.677+0000 7f7652a0b740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55adc0856a80 0x55adc086fbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:20:06.687 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:06.677+0000 7f7652a0b740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55adc08641c0 0x55adc08513c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:20:06.687 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:06.677+0000 7f7652a0b740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f76481af350 con 0x55adc0856a80 2024-03-31T23:20:06.687 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:06.677+0000 7f7652a0b740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f76481acb90 con 0x55adc08641c0 2024-03-31T23:20:06.687 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:06.677+0000 7f7652a0b740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f76481aaa00 con 0x55adc0856160 2024-03-31T23:20:06.687 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:06.677+0000 7f764d3cb640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55adc0856a80 0x55adc086fbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:20:06.687 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:06.678+0000 7f764d3cb640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55adc0856a80 0x55adc086fbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40394/0 (socket says 172.21.15.46:40394) 2024-03-31T23:20:06.688 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:06.678+0000 7f764d3cb640 1 -- 172.21.15.46:0/3395350624 learned_addr learned my addr 172.21.15.46:0/3395350624 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:20:06.688 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:06.678+0000 7f764e3cd640 1 --2- 172.21.15.46:0/3395350624 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55adc08641c0 0x55adc08513c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:20:06.688 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:06.678+0000 7f764dbcc640 1 --2- 172.21.15.46:0/3395350624 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55adc0856160 0x55adc08638e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:20:06.688 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:06.678+0000 7f764d3cb640 1 -- 172.21.15.46:0/3395350624 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55adc0856160 msgr2=0x55adc08638e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:20:06.688 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:06.678+0000 7f764d3cb640 1 --2- 172.21.15.46:0/3395350624 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55adc0856160 0x55adc08638e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:20:06.688 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:06.678+0000 7f764d3cb640 1 -- 172.21.15.46:0/3395350624 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55adc08641c0 msgr2=0x55adc08513c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:20:06.688 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:06.678+0000 7f764d3cb640 1 --2- 172.21.15.46:0/3395350624 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55adc08641c0 0x55adc08513c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:20:06.688 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:06.678+0000 7f764d3cb640 1 -- 172.21.15.46:0/3395350624 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55adc0860160 con 0x55adc0856a80 2024-03-31T23:20:06.688 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:06.678+0000 7f764d3cb640 1 --2- 172.21.15.46:0/3395350624 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55adc0856a80 0x55adc086fbd0 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7f76380ee140 tx=0x7f76380f2040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=27cb780e9f40cbcf server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:20:06.688 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:06.678+0000 7f76337fe640 1 -- 172.21.15.46:0/3395350624 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76380c0dd0 con 0x55adc0856a80 2024-03-31T23:20:06.688 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:06.678+0000 7f76337fe640 1 -- 172.21.15.46:0/3395350624 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f76380c5690 con 0x55adc0856a80 2024-03-31T23:20:06.689 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:06.678+0000 7f76337fe640 1 -- 172.21.15.46:0/3395350624 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76380c5200 con 0x55adc0856a80 2024-03-31T23:20:06.689 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:06.679+0000 7f7652a0b740 1 -- 172.21.15.46:0/3395350624 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55adc0856a80 msgr2=0x55adc086fbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:20:06.689 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:06.679+0000 7f7652a0b740 1 --2- 172.21.15.46:0/3395350624 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55adc0856a80 0x55adc086fbd0 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7f76380ee140 tx=0x7f76380f2040 comp rx=0 tx=0).stop 2024-03-31T23:20:06.689 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:06.679+0000 7f7652a0b740 1 -- 172.21.15.46:0/3395350624 shutdown_connections 2024-03-31T23:20:06.689 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:06.679+0000 7f7652a0b740 1 --2- 172.21.15.46:0/3395350624 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55adc08641c0 0x55adc08513c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:20:06.689 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:06.679+0000 7f7652a0b740 1 --2- 172.21.15.46:0/3395350624 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55adc0856a80 0x55adc086fbd0 unknown :-1 s=CLOSED pgs=771 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:20:06.689 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:06.679+0000 7f7652a0b740 1 --2- 172.21.15.46:0/3395350624 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55adc0856160 0x55adc08638e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:20:06.689 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:06.679+0000 7f7652a0b740 1 -- 172.21.15.46:0/3395350624 >> 172.21.15.46:0/3395350624 conn(0x55adc0850fe0 msgr2=0x55adc084ff80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:20:06.689 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:06.679+0000 7f7652a0b740 1 -- 172.21.15.46:0/3395350624 shutdown_connections 2024-03-31T23:20:06.689 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:06.679+0000 7f7652a0b740 1 -- 172.21.15.46:0/3395350624 wait complete. 2024-03-31T23:20:06.690 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:06.679+0000 7f7652a0b740 1 -- start start 2024-03-31T23:20:06.690 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:06.679+0000 7f7652a0b740 1 Processor -- start 2024-03-31T23:20:06.690 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:06.680+0000 7f7652a0b740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55adc0856160 0x55adc085e000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:20:06.690 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:06.680+0000 7f7652a0b740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55adc0856a80 0x55adc085e540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:20:06.690 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:06.680+0000 7f7652a0b740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55adc08641c0 0x55adc085eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:20:06.690 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:06.680+0000 7f7652a0b740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f76481ab050 con 0x55adc08641c0 2024-03-31T23:20:06.690 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:06.680+0000 7f7652a0b740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f76481ac4c0 con 0x55adc0856a80 2024-03-31T23:20:06.690 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:06.680+0000 7f7652a0b740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f76481ab720 con 0x55adc0856160 2024-03-31T23:20:06.690 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:06.680+0000 7f7652a0b740 10 client.0 _ll_register_callbacks cb 0x7f7638005b70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T23:20:06.691 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:06.680+0000 7f764e3cd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55adc08641c0 0x55adc085eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:20:06.691 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:06.680+0000 7f764e3cd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55adc08641c0 0x55adc085eaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40410/0 (socket says 172.21.15.46:40410) 2024-03-31T23:20:06.691 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:06.680+0000 7f764e3cd640 1 -- 172.21.15.46:0/4082298932 learned_addr learned my addr 172.21.15.46:0/4082298932 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:20:06.691 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:06.680+0000 7f764d3cb640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55adc0856a80 0x55adc085e540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:20:06.691 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:06.680+0000 7f764dbcc640 1 --2- 172.21.15.46:0/4082298932 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55adc0856160 0x55adc085e000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:20:06.691 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:06.680+0000 7f764e3cd640 1 -- 172.21.15.46:0/4082298932 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55adc0856160 msgr2=0x55adc085e000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:20:06.691 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:06.680+0000 7f764e3cd640 1 --2- 172.21.15.46:0/4082298932 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55adc0856160 0x55adc085e000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:20:06.691 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:06.680+0000 7f764e3cd640 1 -- 172.21.15.46:0/4082298932 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55adc0856a80 msgr2=0x55adc085e540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:20:06.691 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:06.680+0000 7f764e3cd640 1 --2- 172.21.15.46:0/4082298932 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55adc0856a80 0x55adc085e540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:20:06.691 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:06.680+0000 7f764e3cd640 1 -- 172.21.15.46:0/4082298932 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55adc08568b0 con 0x55adc08641c0 2024-03-31T23:20:06.692 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:06.680+0000 7f764d3cb640 1 --2- 172.21.15.46:0/4082298932 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55adc0856a80 0x55adc085e540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:20:06.692 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:06.680+0000 7f764e3cd640 1 --2- 172.21.15.46:0/4082298932 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55adc08641c0 0x55adc085eaa0 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7f7640010000 tx=0x7f7640004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d41d74626315760a server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:20:06.692 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:06.680+0000 7f76317fa640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-31T23:20:06.692 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:06.680+0000 7f76317fa640 1 -- 172.21.15.46:0/4082298932 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f764002d060 con 0x55adc08641c0 2024-03-31T23:20:06.692 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:06.680+0000 7f76317fa640 1 -- 172.21.15.46:0/4082298932 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9554+0+0 (secure 0 0 0) 0x7f764001d930 con 0x55adc08641c0 2024-03-31T23:20:06.692 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:06.680+0000 7f7652a0b740 1 -- 172.21.15.46:0/4082298932 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f76380c5690 con 0x55adc08641c0 2024-03-31T23:20:06.692 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:06.681+0000 7f76317fa640 1 -- 172.21.15.46:0/4082298932 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f7640034050 con 0x55adc08641c0 2024-03-31T23:20:06.692 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:06.681+0000 7f7652a0b740 10 client.16836 fetch_fsmap learned FSMap version 35 2024-03-31T23:20:06.692 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:06.681+0000 7f7652a0b740 1 -- 172.21.15.46:0/4082298932 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x55adc08da6e0 con 0x55adc08641c0 2024-03-31T23:20:06.692 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:06.681+0000 7f76317fa640 1 -- 172.21.15.46:0/4082298932 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f76380c5690 con 0x55adc08641c0 2024-03-31T23:20:06.692 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:06.681+0000 7f76317fa640 1 -- 172.21.15.46:0/4082298932 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 35) ==== 8366+0+0 (secure 0 0 0) 0x7f764001eb20 con 0x55adc08641c0 2024-03-31T23:20:06.693 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:06.681+0000 7f7652a0b740 10 client.16836 fetch_fsmap finished waiting for FSMap version 35 2024-03-31T23:20:06.693 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:06.681+0000 7f7652a0b740 10 client.16836 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-31T23:20:06.693 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:06.681+0000 7f7652a0b740 20 client.16836 populate_metadata read hostname 'smithi046' 2024-03-31T23:20:06.693 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:06.681+0000 7f7652a0b740 1 --2- 172.21.15.46:0/4082298932 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55adc08daef0 0x55adc08dd310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:20:06.693 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:06.681+0000 7f7652a0b740 4 client.16836 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-31T23:20:06.693 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:06.681+0000 7f7652a0b740 1 -- 172.21.15.46:0/4082298932 --> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55adc07c6e30 con 0x55adc08daef0 2024-03-31T23:20:06.693 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:06.681+0000 7f764dbcc640 1 --2- 172.21.15.46:0/4082298932 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55adc08daef0 0x55adc08dd310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:20:06.693 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:06.681+0000 7f76317fa640 10 client.16836 ms_handle_connect on v2:172.21.15.142:6838/2519143918 2024-03-31T23:20:06.693 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:06.681+0000 7f764dbcc640 1 --2- 172.21.15.46:0/4082298932 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55adc08daef0 0x55adc08dd310 crc :-1 s=READY pgs=150 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=974930d42f2741a7 server_cookie=3996f768663ebc14 in_seq=0 out_seq=0 2024-03-31T23:20:06.706 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:06.704+0000 7f76317fa640 1 -- 172.21.15.46:0/4082298932 <== mds.0 v2:172.21.15.142:6838/2519143918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55adc07c6e30 con 0x55adc08daef0 2024-03-31T23:20:06.706 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:06.704+0000 7f76317fa640 10 client.16836 handle_command_reply: tid=0 2024-03-31T23:20:06.707 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:06.704+0000 7f7652a0b740 1 client.16836 shutdown 2024-03-31T23:20:06.707 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:06.705+0000 7f7652a0b740 1 -- 172.21.15.46:0/4082298932 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55adc08641c0 msgr2=0x55adc085eaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:20:06.707 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:06.705+0000 7f7652a0b740 1 --2- 172.21.15.46:0/4082298932 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55adc08641c0 0x55adc085eaa0 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7f7640010000 tx=0x7f7640004ab0 comp rx=0 tx=0).stop 2024-03-31T23:20:06.707 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:06.705+0000 7f7652a0b740 1 -- 172.21.15.46:0/4082298932 shutdown_connections 2024-03-31T23:20:06.707 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:06.705+0000 7f7652a0b740 1 --2- 172.21.15.46:0/4082298932 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55adc08daef0 0x55adc08dd310 crc :-1 s=READY pgs=150 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:20:06.707 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:06.705+0000 7f7652a0b740 1 --2- 172.21.15.46:0/4082298932 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55adc08641c0 0x55adc085eaa0 unknown :-1 s=CLOSED pgs=772 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:20:06.707 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:06.705+0000 7f7652a0b740 1 --2- 172.21.15.46:0/4082298932 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55adc0856a80 0x55adc085e540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:20:06.707 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:06.705+0000 7f7652a0b740 1 --2- 172.21.15.46:0/4082298932 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55adc0856160 0x55adc085e000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:20:06.707 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:06.705+0000 7f76317fa640 0 client.16836 ms_handle_reset on v2:172.21.15.142:6838/2519143918 2024-03-31T23:20:06.707 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:06.705+0000 7f7652a0b740 1 -- 172.21.15.46:0/4082298932 >> 172.21.15.46:0/4082298932 conn(0x55adc0850fe0 msgr2=0x55adc084ff80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:20:06.707 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:06.705+0000 7f7652a0b740 1 -- 172.21.15.46:0/4082298932 shutdown_connections 2024-03-31T23:20:06.707 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:06.705+0000 7f7652a0b740 1 -- 172.21.15.46:0/4082298932 wait complete. 2024-03-31T23:20:06.707 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:06.705+0000 7f7652a0b740 20 client.16836 trim_cache size 0 max 16384 2024-03-31T23:20:06.708 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout:{ 2024-03-31T23:20:06.708 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout: "message": "", 2024-03-31T23:20:06.708 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout: "return_code": 0 2024-03-31T23:20:06.708 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout:} 2024-03-31T23:20:06.709 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:06.705+0000 7f764fe30640 1 -- 172.21.15.46:0/3241652662 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f76340a26b0 msgr2=0x7f76340a4b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:20:06.709 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:06.705+0000 7f764fe30640 1 --2- 172.21.15.46:0/3241652662 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f76340a26b0 0x7f76340a4b70 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f7640009b80 tx=0x7f7640009500 comp rx=0 tx=0).stop 2024-03-31T23:20:06.709 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:06.705+0000 7f764fe30640 1 -- 172.21.15.46:0/3241652662 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7648101360 msgr2=0x7f7648199270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:20:06.709 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:06.705+0000 7f764fe30640 1 --2- 172.21.15.46:0/3241652662 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7648101360 0x7f7648199270 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7f7638005940 tx=0x7f7638005dc0 comp rx=0 tx=0).stop 2024-03-31T23:20:06.709 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:06.705+0000 7f764fe30640 1 -- 172.21.15.46:0/3241652662 shutdown_connections 2024-03-31T23:20:06.709 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:06.705+0000 7f764fe30640 1 --2- 172.21.15.46:0/3241652662 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f76340a26b0 0x7f76340a4b70 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:20:06.709 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:06.705+0000 7f764fe30640 1 --2- 172.21.15.46:0/3241652662 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7648105630 0x7f76481a3db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:20:06.710 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:06.705+0000 7f764fe30640 1 --2- 172.21.15.46:0/3241652662 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7648101360 0x7f7648199270 unknown :-1 s=CLOSED pgs=769 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:20:06.710 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:06.705+0000 7f764fe30640 1 --2- 172.21.15.46:0/3241652662 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f76481009a0 0x7f7648198d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:20:06.710 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:06.705+0000 7f764fe30640 1 -- 172.21.15.46:0/3241652662 >> 172.21.15.46:0/3241652662 conn(0x7f7648076450 msgr2=0x7f76480fe510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:20:06.710 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:06.706+0000 7f764fe30640 1 -- 172.21.15.46:0/3241652662 shutdown_connections 2024-03-31T23:20:06.710 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:06.706+0000 7f764fe30640 1 -- 172.21.15.46:0/3241652662 wait complete. 2024-03-31T23:20:06.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:20:06 smithi142 ceph-mon[35271]: pgmap v258: 129 pgs: 129 active+clean; 110 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 388 KiB/s rd, 1.1 MiB/s wr, 118 op/s 2024-03-31T23:20:07.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:20:06 smithi162 ceph-mon[36237]: pgmap v258: 129 pgs: 129 active+clean; 110 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 388 KiB/s rd, 1.1 MiB/s wr, 118 op/s 2024-03-31T23:20:07.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:20:06 smithi046 ceph-mon[32498]: pgmap v258: 129 pgs: 129 active+clean; 110 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 388 KiB/s rd, 1.1 MiB/s wr, 118 op/s 2024-03-31T23:20:07.205 INFO:tasks.workunit.client.0.smithi046.stdout: 1 184595 2.25 MB/sec execute 313 sec latency 4212.910 ms 2024-03-31T23:20:08.205 INFO:tasks.workunit.client.0.smithi046.stdout: 1 184735 2.25 MB/sec execute 314 sec latency 4749.479 ms 2024-03-31T23:20:08.944 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:20:08 smithi142 ceph-mon[35271]: pgmap v259: 129 pgs: 129 active+clean; 110 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 353 KiB/s rd, 881 KiB/s wr, 96 op/s 2024-03-31T23:20:08.944 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:20:08 smithi142 ceph-mon[35271]: osdmap e101: 12 total, 12 up, 12 in 2024-03-31T23:20:09.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:20:08 smithi162 ceph-mon[36237]: pgmap v259: 129 pgs: 129 active+clean; 110 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 353 KiB/s rd, 881 KiB/s wr, 96 op/s 2024-03-31T23:20:09.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:20:08 smithi162 ceph-mon[36237]: osdmap e101: 12 total, 12 up, 12 in 2024-03-31T23:20:09.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:20:08 smithi046 ceph-mon[32498]: pgmap v259: 129 pgs: 129 active+clean; 110 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 353 KiB/s rd, 881 KiB/s wr, 96 op/s 2024-03-31T23:20:09.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:20:08 smithi046 ceph-mon[32498]: osdmap e101: 12 total, 12 up, 12 in 2024-03-31T23:20:09.205 INFO:tasks.workunit.client.0.smithi046.stdout: 1 185007 2.24 MB/sec execute 315 sec latency 14.740 ms 2024-03-31T23:20:10.205 INFO:tasks.workunit.client.0.smithi046.stdout: 1 185426 2.24 MB/sec execute 316 sec latency 15.488 ms 2024-03-31T23:20:10.913 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:20:10 smithi046 ceph-mon[32498]: pgmap v261: 129 pgs: 129 active+clean; 109 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 217 KiB/s rd, 683 KiB/s wr, 78 op/s 2024-03-31T23:20:10.913 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:20:10 smithi046 ceph-mon[32498]: osdmap e102: 12 total, 12 up, 12 in 2024-03-31T23:20:10.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:20:10 smithi142 ceph-mon[35271]: pgmap v261: 129 pgs: 129 active+clean; 109 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 217 KiB/s rd, 683 KiB/s wr, 78 op/s 2024-03-31T23:20:10.944 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:20:10 smithi142 ceph-mon[35271]: osdmap e102: 12 total, 12 up, 12 in 2024-03-31T23:20:11.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:20:10 smithi162 ceph-mon[36237]: pgmap v261: 129 pgs: 129 active+clean; 109 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 217 KiB/s rd, 683 KiB/s wr, 78 op/s 2024-03-31T23:20:11.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:20:10 smithi162 ceph-mon[36237]: osdmap e102: 12 total, 12 up, 12 in 2024-03-31T23:20:11.205 INFO:tasks.workunit.client.0.smithi046.stdout: 1 185891 2.25 MB/sec execute 317 sec latency 78.902 ms 2024-03-31T23:20:12.205 INFO:tasks.workunit.client.0.smithi046.stdout: 1 186360 2.24 MB/sec execute 318 sec latency 22.425 ms 2024-03-31T23:20:12.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:20:12 smithi142 ceph-mon[35271]: pgmap v263: 129 pgs: 129 active+clean; 109 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 271 KiB/s rd, 853 KiB/s wr, 97 op/s 2024-03-31T23:20:13.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:20:12 smithi162 ceph-mon[36237]: pgmap v263: 129 pgs: 129 active+clean; 109 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 271 KiB/s rd, 853 KiB/s wr, 97 op/s 2024-03-31T23:20:13.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:20:12 smithi046 ceph-mon[32498]: pgmap v263: 129 pgs: 129 active+clean; 109 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 271 KiB/s rd, 853 KiB/s wr, 97 op/s 2024-03-31T23:20:13.205 INFO:tasks.workunit.client.0.smithi046.stdout: 1 186863 2.24 MB/sec execute 319 sec latency 23.143 ms 2024-03-31T23:20:14.205 INFO:tasks.workunit.client.0.smithi046.stdout: 1 187460 2.25 MB/sec execute 320 sec latency 15.616 ms 2024-03-31T23:20:14.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:20:14 smithi142 ceph-mon[35271]: pgmap v264: 129 pgs: 129 active+clean; 109 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 88 KiB/s rd, 321 KiB/s wr, 33 op/s 2024-03-31T23:20:15.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:20:14 smithi162 ceph-mon[36237]: pgmap v264: 129 pgs: 129 active+clean; 109 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 88 KiB/s rd, 321 KiB/s wr, 33 op/s 2024-03-31T23:20:15.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:20:14 smithi046 ceph-mon[32498]: pgmap v264: 129 pgs: 129 active+clean; 109 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 88 KiB/s rd, 321 KiB/s wr, 33 op/s 2024-03-31T23:20:15.205 INFO:tasks.workunit.client.0.smithi046.stdout: 1 187863 2.25 MB/sec execute 321 sec latency 17.429 ms 2024-03-31T23:20:16.206 INFO:tasks.workunit.client.0.smithi046.stdout: 1 188073 2.25 MB/sec execute 322 sec latency 726.229 ms 2024-03-31T23:20:16.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:20:16 smithi142 ceph-mon[35271]: pgmap v265: 129 pgs: 129 active+clean; 112 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 425 KiB/s rd, 1.1 MiB/s wr, 116 op/s 2024-03-31T23:20:17.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:20:16 smithi162 ceph-mon[36237]: pgmap v265: 129 pgs: 129 active+clean; 112 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 425 KiB/s rd, 1.1 MiB/s wr, 116 op/s 2024-03-31T23:20:17.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:20:16 smithi046 ceph-mon[32498]: pgmap v265: 129 pgs: 129 active+clean; 112 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 425 KiB/s rd, 1.1 MiB/s wr, 116 op/s 2024-03-31T23:20:17.206 INFO:tasks.workunit.client.0.smithi046.stdout: 1 188073 2.24 MB/sec execute 323 sec latency 1726.333 ms 2024-03-31T23:20:18.206 INFO:tasks.workunit.client.0.smithi046.stdout: 1 188140 2.24 MB/sec execute 324 sec latency 2263.017 ms 2024-03-31T23:20:18.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:20:18 smithi142 ceph-mon[35271]: pgmap v266: 129 pgs: 129 active+clean; 112 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 370 KiB/s rd, 972 KiB/s wr, 101 op/s 2024-03-31T23:20:19.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:20:18 smithi162 ceph-mon[36237]: pgmap v266: 129 pgs: 129 active+clean; 112 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 370 KiB/s rd, 972 KiB/s wr, 101 op/s 2024-03-31T23:20:19.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:20:18 smithi046 ceph-mon[32498]: pgmap v266: 129 pgs: 129 active+clean; 112 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 370 KiB/s rd, 972 KiB/s wr, 101 op/s 2024-03-31T23:20:19.206 INFO:tasks.workunit.client.0.smithi046.stdout: 1 188140 2.23 MB/sec execute 325 sec latency 1229.643 ms 2024-03-31T23:20:19.804 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:19.803+0000 7f9ae1049640 1 -- 172.21.15.46:0/1134007222 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9adc1055d0 msgr2=0x7f9adc1059b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:20:19.805 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:19.803+0000 7f9ae1049640 1 --2- 172.21.15.46:0/1134007222 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9adc1055d0 0x7f9adc1059b0 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7f9acc009900 tx=0x7f9acc02bc30 comp rx=0 tx=0).stop 2024-03-31T23:20:19.805 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:19.803+0000 7f9ae1049640 1 -- 172.21.15.46:0/1134007222 shutdown_connections 2024-03-31T23:20:19.805 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:19.803+0000 7f9ae1049640 1 --2- 172.21.15.46:0/1134007222 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9adc101240 0x7f9adc10d480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:20:19.805 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:19.803+0000 7f9ae1049640 1 --2- 172.21.15.46:0/1134007222 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9adc1008a0 0x7f9adc100d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:20:19.805 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:19.803+0000 7f9ae1049640 1 --2- 172.21.15.46:0/1134007222 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9adc1055d0 0x7f9adc1059b0 unknown :-1 s=CLOSED pgs=773 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:20:19.805 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:19.803+0000 7f9ae1049640 1 -- 172.21.15.46:0/1134007222 >> 172.21.15.46:0/1134007222 conn(0x7f9adc077420 msgr2=0x7f9adc077830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:20:19.806 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:19.803+0000 7f9ae1049640 1 -- 172.21.15.46:0/1134007222 shutdown_connections 2024-03-31T23:20:19.806 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:19.803+0000 7f9ae1049640 1 -- 172.21.15.46:0/1134007222 wait complete. 2024-03-31T23:20:19.808 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:19.804+0000 7f9ae1049640 1 Processor -- start 2024-03-31T23:20:19.808 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:19.804+0000 7f9ae1049640 1 -- start start 2024-03-31T23:20:19.808 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:19.804+0000 7f9ae1049640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9adc1008a0 0x7f9adc197f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:20:19.809 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:19.804+0000 7f9ae1049640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9adc101240 0x7f9adc198450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:20:19.809 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:19.804+0000 7f9ae1049640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9adc1055d0 0x7f9adc19c7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:20:19.809 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:19.804+0000 7f9ae1049640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f9adc110950 con 0x7f9adc1008a0 2024-03-31T23:20:19.809 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:19.804+0000 7f9ae1049640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f9adc1107d0 con 0x7f9adc1055d0 2024-03-31T23:20:19.809 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:19.804+0000 7f9ae1049640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f9adc110ad0 con 0x7f9adc101240 2024-03-31T23:20:19.809 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:19.804+0000 7f9adad9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9adc1008a0 0x7f9adc197f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:20:19.809 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:19.804+0000 7f9adb59e640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9adc1055d0 0x7f9adc19c7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:20:19.809 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:19.804+0000 7f9adad9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9adc1008a0 0x7f9adc197f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:34980/0 (socket says 172.21.15.46:34980) 2024-03-31T23:20:19.809 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:19.804+0000 7f9adad9d640 1 -- 172.21.15.46:0/2231549109 learned_addr learned my addr 172.21.15.46:0/2231549109 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:20:19.809 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:19.804+0000 7f9ada59c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9adc101240 0x7f9adc198450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:20:19.809 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:19.805+0000 7f9adb59e640 1 -- 172.21.15.46:0/2231549109 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9adc101240 msgr2=0x7f9adc198450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:20:19.809 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:19.805+0000 7f9adb59e640 1 --2- 172.21.15.46:0/2231549109 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9adc101240 0x7f9adc198450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:20:19.810 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:19.805+0000 7f9adb59e640 1 -- 172.21.15.46:0/2231549109 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9adc1008a0 msgr2=0x7f9adc197f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:20:19.810 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:19.805+0000 7f9adb59e640 1 --2- 172.21.15.46:0/2231549109 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9adc1008a0 0x7f9adc197f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:20:19.810 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:19.805+0000 7f9adb59e640 1 -- 172.21.15.46:0/2231549109 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9adc19cf00 con 0x7f9adc1055d0 2024-03-31T23:20:19.810 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:19.805+0000 7f9adb59e640 1 --2- 172.21.15.46:0/2231549109 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9adc1055d0 0x7f9adc19c7d0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f9ad4009fd0 tx=0x7f9ad400ef10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:20:19.810 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:19.805+0000 7f9ac3fff640 1 -- 172.21.15.46:0/2231549109 <== mon.2 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ad400ce20 con 0x7f9adc1055d0 2024-03-31T23:20:19.810 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:19.805+0000 7f9ac3fff640 1 -- 172.21.15.46:0/2231549109 <== mon.2 v2:172.21.15.142:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9ad4005020 con 0x7f9adc1055d0 2024-03-31T23:20:19.810 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:19.805+0000 7f9ae1049640 1 -- 172.21.15.46:0/2231549109 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9adc19d1f0 con 0x7f9adc1055d0 2024-03-31T23:20:19.810 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:19.805+0000 7f9ac3fff640 1 -- 172.21.15.46:0/2231549109 <== mon.2 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ad4010690 con 0x7f9adc1055d0 2024-03-31T23:20:19.810 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:19.805+0000 7f9ae1049640 1 -- 172.21.15.46:0/2231549109 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9adc1a4a80 con 0x7f9adc1055d0 2024-03-31T23:20:19.811 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:19.807+0000 7f9ac3fff640 1 -- 172.21.15.46:0/2231549109 <== mon.2 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 25) ==== 137451+0+0 (secure 0 0 0) 0x7f9ad40051d0 con 0x7f9adc1055d0 2024-03-31T23:20:19.818 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:19.808+0000 7f9ae3c1f740 1 Processor -- start 2024-03-31T23:20:19.818 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:19.808+0000 7f9ae3c1f740 1 -- start start 2024-03-31T23:20:19.819 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:19.808+0000 7f9ac3fff640 1 --2- 172.21.15.46:0/2231549109 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f9ac40a26b0 0x7f9ac40a4b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:20:19.819 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:19.808+0000 7f9ae3c1f740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5628ecb916c0 0x5628ecb91aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:20:19.819 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:19.808+0000 7f9ada59c640 1 --2- 172.21.15.46:0/2231549109 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f9ac40a26b0 0x7f9ac40a4b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:20:19.819 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:19.808+0000 7f9ae3c1f740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5628ecb91fe0 0x5628ecb9a870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:20:19.819 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:19.808+0000 7f9ae3c1f740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5628ecb9adb0 0x5628ecb9d1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:20:19.819 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:19.808+0000 7f9ae3c1f740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x5628ecaf48f0 con 0x5628ecb916c0 2024-03-31T23:20:19.819 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:19.808+0000 7f9ae3c1f740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x5628eca9d8d0 con 0x5628ecb91fe0 2024-03-31T23:20:19.819 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:19.808+0000 7f9ac3fff640 1 -- 172.21.15.46:0/2231549109 <== mon.2 v2:172.21.15.142:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 9554+0+0 (secure 0 0 0) 0x7f9ad40cf680 con 0x7f9adc1055d0 2024-03-31T23:20:19.819 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:19.808+0000 7f9ae3c1f740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5628ecaf0990 con 0x5628ecb9adb0 2024-03-31T23:20:19.819 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:19.808+0000 7f9adad9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5628ecb916c0 0x5628ecb91aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:20:19.819 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:19.808+0000 7f9adad9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5628ecb916c0 0x5628ecb91aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:34994/0 (socket says 172.21.15.46:34994) 2024-03-31T23:20:19.820 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:19.808+0000 7f9adad9d640 1 -- 172.21.15.46:0/3778845986 learned_addr learned my addr 172.21.15.46:0/3778845986 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:20:19.820 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:19.808+0000 7f9adb59e640 1 --2- 172.21.15.46:0/3778845986 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5628ecb9adb0 0x5628ecb9d1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:20:19.820 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:19.808+0000 7f9ada59c640 1 --2- 172.21.15.46:0/3778845986 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5628ecb91fe0 0x5628ecb9a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:20:19.820 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:19.808+0000 7f9adad9d640 1 -- 172.21.15.46:0/3778845986 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5628ecb9adb0 msgr2=0x5628ecb9d1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:20:19.820 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:19.808+0000 7f9adad9d640 1 --2- 172.21.15.46:0/3778845986 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5628ecb9adb0 0x5628ecb9d1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:20:19.820 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:19.808+0000 7f9adad9d640 1 -- 172.21.15.46:0/3778845986 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5628ecb91fe0 msgr2=0x5628ecb9a870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:20:19.820 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:19.808+0000 7f9adad9d640 1 --2- 172.21.15.46:0/3778845986 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5628ecb91fe0 0x5628ecb9a870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:20:19.820 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:19.809+0000 7f9adad9d640 1 -- 172.21.15.46:0/3778845986 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5628ec809210 con 0x5628ecb916c0 2024-03-31T23:20:19.820 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:19.809+0000 7f9ada59c640 1 --2- 172.21.15.46:0/2231549109 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f9ac40a26b0 0x7f9ac40a4b70 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f9adc104f90 tx=0x7f9ad0011440 comp rx=0 tx=0).ready entity=mgr.15654 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:20:19.820 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:19.809+0000 7f9adad9d640 1 --2- 172.21.15.46:0/3778845986 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5628ecb916c0 0x5628ecb91aa0 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7f9acc036040 tx=0x7f9acc0095e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=812b3582f2e18f42 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:20:19.820 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:19.809+0000 7f9ac17fa640 1 -- 172.21.15.46:0/3778845986 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9acc045020 con 0x5628ecb916c0 2024-03-31T23:20:19.820 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:19.809+0000 7f9ac17fa640 1 -- 172.21.15.46:0/3778845986 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9acc02c7e0 con 0x5628ecb916c0 2024-03-31T23:20:19.820 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:19.809+0000 7f9ac17fa640 1 -- 172.21.15.46:0/3778845986 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9acc005500 con 0x5628ecb916c0 2024-03-31T23:20:19.820 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:19.809+0000 7f9ae3c1f740 1 -- 172.21.15.46:0/3778845986 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5628ecb916c0 msgr2=0x5628ecb91aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:20:19.821 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:19.809+0000 7f9ae3c1f740 1 --2- 172.21.15.46:0/3778845986 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5628ecb916c0 0x5628ecb91aa0 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7f9acc036040 tx=0x7f9acc0095e0 comp rx=0 tx=0).stop 2024-03-31T23:20:19.821 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:19.810+0000 7f9ae3c1f740 1 -- 172.21.15.46:0/3778845986 shutdown_connections 2024-03-31T23:20:19.821 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:19.810+0000 7f9ae3c1f740 1 --2- 172.21.15.46:0/3778845986 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5628ecb9adb0 0x5628ecb9d1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:20:19.821 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:19.810+0000 7f9ae3c1f740 1 --2- 172.21.15.46:0/3778845986 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5628ecb91fe0 0x5628ecb9a870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:20:19.821 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:19.810+0000 7f9ae3c1f740 1 --2- 172.21.15.46:0/3778845986 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5628ecb916c0 0x5628ecb91aa0 unknown :-1 s=CLOSED pgs=774 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:20:19.821 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:19.810+0000 7f9ae3c1f740 1 -- 172.21.15.46:0/3778845986 >> 172.21.15.46:0/3778845986 conn(0x5628ecb8bfe0 msgr2=0x5628ecb8c3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:20:19.821 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:19.810+0000 7f9ae3c1f740 1 -- 172.21.15.46:0/3778845986 shutdown_connections 2024-03-31T23:20:19.821 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:19.810+0000 7f9ae3c1f740 1 -- 172.21.15.46:0/3778845986 wait complete. 2024-03-31T23:20:19.821 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:19.810+0000 7f9ae3c1f740 1 -- start start 2024-03-31T23:20:19.821 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:19.810+0000 7f9ae3c1f740 1 Processor -- start 2024-03-31T23:20:19.821 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:19.810+0000 7f9ae3c1f740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5628ecb916c0 0x5628ecb97d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:20:19.821 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:19.810+0000 7f9ae3c1f740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5628ecb91fe0 0x5628ecb98290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:20:19.821 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:19.810+0000 7f9ae3c1f740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5628ecb9adb0 0x5628ecb987d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:20:19.822 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:19.810+0000 7f9ae3c1f740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x5628ecb9d980 con 0x5628ecb9adb0 2024-03-31T23:20:19.822 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:19.810+0000 7f9ae3c1f740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x5628eca92f80 con 0x5628ecb916c0 2024-03-31T23:20:19.822 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:19.810+0000 7f9ae3c1f740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5628ecb9db00 con 0x5628ecb91fe0 2024-03-31T23:20:19.822 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:19.810+0000 7f9ae3c1f740 10 client.0 _ll_register_callbacks cb 0x5628ecaeb290 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T23:20:19.822 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:19.810+0000 7f9adb59e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5628ecb9adb0 0x5628ecb987d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:20:19.822 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:19.810+0000 7f9adb59e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5628ecb9adb0 0x5628ecb987d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:35010/0 (socket says 172.21.15.46:35010) 2024-03-31T23:20:19.822 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:19.810+0000 7f9adb59e640 1 -- 172.21.15.46:0/655297738 learned_addr learned my addr 172.21.15.46:0/655297738 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:20:19.822 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:19.811+0000 7f9adad9d640 1 --2- 172.21.15.46:0/655297738 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5628ecb916c0 0x5628ecb97d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:20:19.822 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:19.811+0000 7f9ada59c640 1 --2- 172.21.15.46:0/655297738 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5628ecb91fe0 0x5628ecb98290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:20:19.822 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:19.811+0000 7f9adb59e640 1 -- 172.21.15.46:0/655297738 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5628ecb91fe0 msgr2=0x5628ecb98290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:20:19.822 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:19.811+0000 7f9adb59e640 1 --2- 172.21.15.46:0/655297738 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5628ecb91fe0 0x5628ecb98290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:20:19.822 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:19.811+0000 7f9adb59e640 1 -- 172.21.15.46:0/655297738 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5628ecb916c0 msgr2=0x5628ecb97d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:20:19.822 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:19.811+0000 7f9adb59e640 1 --2- 172.21.15.46:0/655297738 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5628ecb916c0 0x5628ecb97d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:20:19.823 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:19.811+0000 7f9adb59e640 1 -- 172.21.15.46:0/655297738 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5628ec8d5620 con 0x5628ecb9adb0 2024-03-31T23:20:19.823 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:19.811+0000 7f9adad9d640 1 --2- 172.21.15.46:0/655297738 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5628ecb916c0 0x5628ecb97d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:20:19.823 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:19.811+0000 7f9adb59e640 1 --2- 172.21.15.46:0/655297738 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5628ecb9adb0 0x5628ecb987d0 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7f9ad4098360 tx=0x7f9ad409ba50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c5d5454f5220d140 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:20:19.823 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:19.811+0000 7f9abeff5640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-31T23:20:19.823 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:19.811+0000 7f9abeff5640 1 -- 172.21.15.46:0/655297738 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ad40b3070 con 0x5628ecb9adb0 2024-03-31T23:20:19.823 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:19.811+0000 7f9abeff5640 1 -- 172.21.15.46:0/655297738 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 9554+0+0 (secure 0 0 0) 0x7f9ad40a3f10 con 0x5628ecb9adb0 2024-03-31T23:20:19.823 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:19.811+0000 7f9ae3c1f740 1 -- 172.21.15.46:0/655297738 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9acc02c7e0 con 0x5628ecb9adb0 2024-03-31T23:20:19.823 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:19.811+0000 7f9abeff5640 1 -- 172.21.15.46:0/655297738 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f9ad40a4110 con 0x5628ecb9adb0 2024-03-31T23:20:19.823 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:19.811+0000 7f9ae3c1f740 10 client.16848 fetch_fsmap learned FSMap version 35 2024-03-31T23:20:19.823 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:19.811+0000 7f9ae3c1f740 1 -- 172.21.15.46:0/655297738 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x5628ecba9810 con 0x5628ecb9adb0 2024-03-31T23:20:19.823 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:19.811+0000 7f9abeff5640 1 -- 172.21.15.46:0/655297738 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9acc02c7e0 con 0x5628ecb9adb0 2024-03-31T23:20:19.823 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:19.812+0000 7f9abeff5640 1 -- 172.21.15.46:0/655297738 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 35) ==== 8366+0+0 (secure 0 0 0) 0x7f9ad40cf220 con 0x5628ecb9adb0 2024-03-31T23:20:19.823 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:19.812+0000 7f9ae3c1f740 10 client.16848 fetch_fsmap finished waiting for FSMap version 35 2024-03-31T23:20:19.823 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:19.812+0000 7f9ae3c1f740 10 client.16848 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-31T23:20:19.823 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:19.812+0000 7f9ae3c1f740 20 client.16848 populate_metadata read hostname 'smithi046' 2024-03-31T23:20:19.824 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:19.812+0000 7f9ae3c1f740 1 --2- 172.21.15.46:0/655297738 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x5628ecbaacf0 0x5628ecbad120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:20:19.824 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:19.812+0000 7f9ae3c1f740 4 client.16848 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-31T23:20:19.824 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:19.812+0000 7f9ae3c1f740 1 -- 172.21.15.46:0/655297738 --> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5628ecb08120 con 0x5628ecbaacf0 2024-03-31T23:20:19.824 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:19.812+0000 7f9adad9d640 1 --2- 172.21.15.46:0/655297738 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x5628ecbaacf0 0x5628ecbad120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:20:19.824 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:19.812+0000 7f9abeff5640 10 client.16848 ms_handle_connect on v2:172.21.15.142:6838/2519143918 2024-03-31T23:20:19.824 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:19.812+0000 7f9adad9d640 1 --2- 172.21.15.46:0/655297738 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x5628ecbaacf0 0x5628ecbad120 crc :-1 s=READY pgs=151 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7ac24b2fe8a5120a server_cookie=9720f7928e39ee69 in_seq=0 out_seq=0 2024-03-31T23:20:19.824 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:19.813+0000 7f9abeff5640 1 -- 172.21.15.46:0/655297738 <== mds.0 v2:172.21.15.142:6838/2519143918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5628ecb08120 con 0x5628ecbaacf0 2024-03-31T23:20:19.824 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:19.813+0000 7f9abeff5640 10 client.16848 handle_command_reply: tid=0 2024-03-31T23:20:19.824 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:19.813+0000 7f9ae3c1f740 1 client.16848 shutdown 2024-03-31T23:20:19.824 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:19.813+0000 7f9ae3c1f740 1 -- 172.21.15.46:0/655297738 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5628ecb9adb0 msgr2=0x5628ecb987d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:20:19.824 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:19.813+0000 7f9ae3c1f740 1 --2- 172.21.15.46:0/655297738 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5628ecb9adb0 0x5628ecb987d0 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7f9ad4098360 tx=0x7f9ad409ba50 comp rx=0 tx=0).stop 2024-03-31T23:20:19.824 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:19.813+0000 7f9ae3c1f740 1 -- 172.21.15.46:0/655297738 shutdown_connections 2024-03-31T23:20:19.824 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:19.813+0000 7f9ae3c1f740 1 --2- 172.21.15.46:0/655297738 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x5628ecbaacf0 0x5628ecbad120 crc :-1 s=READY pgs=151 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:20:19.824 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:19.813+0000 7f9ae3c1f740 1 --2- 172.21.15.46:0/655297738 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5628ecb9adb0 0x5628ecb987d0 unknown :-1 s=CLOSED pgs=775 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:20:19.824 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:19.813+0000 7f9ae3c1f740 1 --2- 172.21.15.46:0/655297738 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5628ecb91fe0 0x5628ecb98290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:20:19.825 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:19.813+0000 7f9abeff5640 0 client.16848 ms_handle_reset on v2:172.21.15.142:6838/2519143918 2024-03-31T23:20:19.825 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:19.813+0000 7f9ae3c1f740 1 --2- 172.21.15.46:0/655297738 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5628ecb916c0 0x5628ecb97d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:20:19.825 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:19.813+0000 7f9ae3c1f740 1 -- 172.21.15.46:0/655297738 >> 172.21.15.46:0/655297738 conn(0x5628ecb8bfe0 msgr2=0x5628ecb8dd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:20:19.825 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:19.813+0000 7f9ae3c1f740 1 -- 172.21.15.46:0/655297738 shutdown_connections 2024-03-31T23:20:19.825 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:19.813+0000 7f9ae3c1f740 1 -- 172.21.15.46:0/655297738 wait complete. 2024-03-31T23:20:19.825 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:19.813+0000 7f9ae3c1f740 20 client.16848 trim_cache size 0 max 16384 2024-03-31T23:20:19.830 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:19.829+0000 7f9ae3c1f740 1 Processor -- start 2024-03-31T23:20:19.830 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:19.829+0000 7f9ae3c1f740 1 -- start start 2024-03-31T23:20:19.830 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:19.829+0000 7f9ae3c1f740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5628ecb916c0 0x5628ecb9a480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:20:19.831 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:19.829+0000 7f9ae3c1f740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5628ecb91fe0 0x5628ecba6720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:20:19.831 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:19.829+0000 7f9ae3c1f740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5628ecb9adb0 0x5628ecbb2a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:20:19.831 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:19.829+0000 7f9ae3c1f740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f9adc1b14a0 con 0x5628ecb91fe0 2024-03-31T23:20:19.831 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:19.829+0000 7f9ae3c1f740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f9adc1aece0 con 0x5628ecb9adb0 2024-03-31T23:20:19.831 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:19.829+0000 7f9ae3c1f740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f9adc1acb50 con 0x5628ecb916c0 2024-03-31T23:20:19.831 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:19.829+0000 7f9ada59c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5628ecb91fe0 0x5628ecba6720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:20:19.831 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:19.829+0000 7f9ada59c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5628ecb91fe0 0x5628ecba6720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:35014/0 (socket says 172.21.15.46:35014) 2024-03-31T23:20:19.831 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:19.829+0000 7f9ada59c640 1 -- 172.21.15.46:0/1054777386 learned_addr learned my addr 172.21.15.46:0/1054777386 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:20:19.831 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:19.829+0000 7f9adb59e640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5628ecb9adb0 0x5628ecbb2a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:20:19.831 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:19.829+0000 7f9adad9d640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5628ecb916c0 0x5628ecb9a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:20:19.835 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:19.830+0000 7f9ada59c640 1 -- 172.21.15.46:0/1054777386 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5628ecb916c0 msgr2=0x5628ecb9a480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:20:19.836 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:19.830+0000 7f9ada59c640 1 --2- 172.21.15.46:0/1054777386 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5628ecb916c0 0x5628ecb9a480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:20:19.836 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:19.830+0000 7f9ada59c640 1 -- 172.21.15.46:0/1054777386 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5628ecb9adb0 msgr2=0x5628ecbb2a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:20:19.836 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:19.830+0000 7f9ada59c640 1 --2- 172.21.15.46:0/1054777386 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5628ecb9adb0 0x5628ecbb2a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:20:19.836 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:19.830+0000 7f9ada59c640 1 -- 172.21.15.46:0/1054777386 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5628ecb8e2a0 con 0x5628ecb91fe0 2024-03-31T23:20:19.836 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:19.830+0000 7f9ada59c640 1 --2- 172.21.15.46:0/1054777386 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5628ecb91fe0 0x5628ecba6720 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7f9ad000f8d0 tx=0x7f9ad001d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=61510adc99fda54e server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:20:19.836 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:19.830+0000 7f9ac0ff9640 1 -- 172.21.15.46:0/1054777386 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ad002d070 con 0x5628ecb91fe0 2024-03-31T23:20:19.836 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:19.830+0000 7f9ac0ff9640 1 -- 172.21.15.46:0/1054777386 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9ad0019050 con 0x5628ecb91fe0 2024-03-31T23:20:19.836 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:19.830+0000 7f9ac0ff9640 1 -- 172.21.15.46:0/1054777386 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ad0017040 con 0x5628ecb91fe0 2024-03-31T23:20:19.836 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:19.831+0000 7f9ae3c1f740 1 -- 172.21.15.46:0/1054777386 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5628ecb91fe0 msgr2=0x5628ecba6720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:20:19.836 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:19.831+0000 7f9ae3c1f740 1 --2- 172.21.15.46:0/1054777386 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5628ecb91fe0 0x5628ecba6720 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7f9ad000f8d0 tx=0x7f9ad001d000 comp rx=0 tx=0).stop 2024-03-31T23:20:19.836 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:19.831+0000 7f9ae3c1f740 1 -- 172.21.15.46:0/1054777386 shutdown_connections 2024-03-31T23:20:19.836 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:19.831+0000 7f9ae3c1f740 1 --2- 172.21.15.46:0/1054777386 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5628ecb9adb0 0x5628ecbb2a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:20:19.836 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:19.831+0000 7f9ae3c1f740 1 --2- 172.21.15.46:0/1054777386 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5628ecb91fe0 0x5628ecba6720 unknown :-1 s=CLOSED pgs=776 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:20:19.837 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:19.831+0000 7f9ae3c1f740 1 --2- 172.21.15.46:0/1054777386 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5628ecb916c0 0x5628ecb9a480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:20:19.837 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:19.831+0000 7f9ae3c1f740 1 -- 172.21.15.46:0/1054777386 >> 172.21.15.46:0/1054777386 conn(0x5628ecb8bfe0 msgr2=0x5628ecb8c3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:20:19.837 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:19.831+0000 7f9ae3c1f740 1 -- 172.21.15.46:0/1054777386 shutdown_connections 2024-03-31T23:20:19.837 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:19.831+0000 7f9ae3c1f740 1 -- 172.21.15.46:0/1054777386 wait complete. 2024-03-31T23:20:19.838 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:19.831+0000 7f9ae3c1f740 1 -- start start 2024-03-31T23:20:19.838 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:19.831+0000 7f9ae3c1f740 1 Processor -- start 2024-03-31T23:20:19.838 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:19.832+0000 7f9ae3c1f740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5628ecb916c0 0x5628ecb9e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:20:19.838 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:19.832+0000 7f9ae3c1f740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5628ecb91fe0 0x5628ecb9ebe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:20:19.838 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:19.832+0000 7f9ae3c1f740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5628ecb9adb0 0x5628ecb9f120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:20:19.838 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:19.832+0000 7f9ae3c1f740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f9adc1ad1a0 con 0x5628ecb916c0 2024-03-31T23:20:19.838 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:19.832+0000 7f9ae3c1f740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f9adc1ae610 con 0x5628ecb9adb0 2024-03-31T23:20:19.838 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:19.832+0000 7f9ae3c1f740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f9adc1ad870 con 0x5628ecb91fe0 2024-03-31T23:20:19.838 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:19.832+0000 7f9ae3c1f740 10 client.0 _ll_register_callbacks cb 0x7f9ad409b550 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T23:20:19.838 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:19.832+0000 7f9adad9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5628ecb916c0 0x5628ecb9e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:20:19.838 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:19.832+0000 7f9ada59c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5628ecb91fe0 0x5628ecb9ebe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:20:19.838 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:19.832+0000 7f9adad9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5628ecb916c0 0x5628ecb9e6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:35028/0 (socket says 172.21.15.46:35028) 2024-03-31T23:20:19.839 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:19.832+0000 7f9adad9d640 1 -- 172.21.15.46:0/1213989549 learned_addr learned my addr 172.21.15.46:0/1213989549 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:20:19.839 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:19.832+0000 7f9adb59e640 1 --2- 172.21.15.46:0/1213989549 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5628ecb9adb0 0x5628ecb9f120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:20:19.839 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:19.832+0000 7f9adad9d640 1 -- 172.21.15.46:0/1213989549 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5628ecb91fe0 msgr2=0x5628ecb9ebe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:20:19.839 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:19.832+0000 7f9adad9d640 1 --2- 172.21.15.46:0/1213989549 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5628ecb91fe0 0x5628ecb9ebe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:20:19.839 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:19.832+0000 7f9adad9d640 1 -- 172.21.15.46:0/1213989549 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5628ecb9adb0 msgr2=0x5628ecb9f120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:20:19.839 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:19.832+0000 7f9adad9d640 1 --2- 172.21.15.46:0/1213989549 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5628ecb9adb0 0x5628ecb9f120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:20:19.839 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:19.832+0000 7f9adad9d640 1 -- 172.21.15.46:0/1213989549 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5628ecc100b0 con 0x5628ecb916c0 2024-03-31T23:20:19.839 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:19.832+0000 7f9adad9d640 1 --2- 172.21.15.46:0/1213989549 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5628ecb916c0 0x5628ecb9e6a0 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7f9acc046000 tx=0x7f9acc0050e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f7fb3864bc7dccb3 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:20:19.839 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:19.832+0000 7f9abeff5640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-31T23:20:19.839 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:19.833+0000 7f9abeff5640 1 -- 172.21.15.46:0/1213989549 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9acc056070 con 0x5628ecb916c0 2024-03-31T23:20:19.840 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:19.833+0000 7f9abeff5640 1 -- 172.21.15.46:0/1213989549 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 9554+0+0 (secure 0 0 0) 0x7f9acc052ca0 con 0x5628ecb916c0 2024-03-31T23:20:19.840 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:19.833+0000 7f9ae3c1f740 1 -- 172.21.15.46:0/1213989549 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9ad0019050 con 0x5628ecb916c0 2024-03-31T23:20:19.840 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:19.833+0000 7f9abeff5640 1 -- 172.21.15.46:0/1213989549 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f9acc061050 con 0x5628ecb916c0 2024-03-31T23:20:19.840 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:19.833+0000 7f9ae3c1f740 10 client.16854 fetch_fsmap learned FSMap version 35 2024-03-31T23:20:19.840 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:19.833+0000 7f9ae3c1f740 1 -- 172.21.15.46:0/1213989549 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x5628ecc113c0 con 0x5628ecb916c0 2024-03-31T23:20:19.840 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:19.833+0000 7f9abeff5640 1 -- 172.21.15.46:0/1213989549 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9ad0019050 con 0x5628ecb916c0 2024-03-31T23:20:19.840 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:19.833+0000 7f9abeff5640 1 -- 172.21.15.46:0/1213989549 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 35) ==== 8366+0+0 (secure 0 0 0) 0x7f9acc004730 con 0x5628ecb916c0 2024-03-31T23:20:19.840 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:19.833+0000 7f9ae3c1f740 10 client.16854 fetch_fsmap finished waiting for FSMap version 35 2024-03-31T23:20:19.840 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:19.833+0000 7f9ae3c1f740 10 client.16854 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-31T23:20:19.840 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:19.833+0000 7f9ae3c1f740 20 client.16854 populate_metadata read hostname 'smithi046' 2024-03-31T23:20:19.840 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:19.833+0000 7f9ae3c1f740 1 --2- 172.21.15.46:0/1213989549 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x5628ecc11810 0x5628ecc13c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:20:19.840 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:19.833+0000 7f9ae3c1f740 4 client.16854 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-31T23:20:19.841 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:19.833+0000 7f9ae3c1f740 1 -- 172.21.15.46:0/1213989549 --> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5628ecb01e30 con 0x5628ecc11810 2024-03-31T23:20:19.841 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:19.833+0000 7f9ada59c640 1 --2- 172.21.15.46:0/1213989549 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x5628ecc11810 0x5628ecc13c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:20:19.841 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:19.834+0000 7f9abeff5640 10 client.16854 ms_handle_connect on v2:172.21.15.142:6838/2519143918 2024-03-31T23:20:19.841 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:19.834+0000 7f9ada59c640 1 --2- 172.21.15.46:0/1213989549 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x5628ecc11810 0x5628ecc13c00 crc :-1 s=READY pgs=152 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=851b54b6f64275fa server_cookie=efb63d068bebbf66 in_seq=0 out_seq=0 2024-03-31T23:20:19.862 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:19.860+0000 7f9abeff5640 1 -- 172.21.15.46:0/1213989549 <== mds.0 v2:172.21.15.142:6838/2519143918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5628ecb01e30 con 0x5628ecc11810 2024-03-31T23:20:19.862 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:19.860+0000 7f9abeff5640 10 client.16854 handle_command_reply: tid=0 2024-03-31T23:20:19.862 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:19.860+0000 7f9ae3c1f740 1 client.16854 shutdown 2024-03-31T23:20:19.863 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:19.860+0000 7f9ae3c1f740 1 -- 172.21.15.46:0/1213989549 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5628ecb916c0 msgr2=0x5628ecb9e6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:20:19.863 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:19.860+0000 7f9ae3c1f740 1 --2- 172.21.15.46:0/1213989549 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5628ecb916c0 0x5628ecb9e6a0 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7f9acc046000 tx=0x7f9acc0050e0 comp rx=0 tx=0).stop 2024-03-31T23:20:19.863 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:19.860+0000 7f9ae3c1f740 1 -- 172.21.15.46:0/1213989549 shutdown_connections 2024-03-31T23:20:19.863 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:19.860+0000 7f9ae3c1f740 1 --2- 172.21.15.46:0/1213989549 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x5628ecc11810 0x5628ecc13c00 crc :-1 s=READY pgs=152 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:20:19.863 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:19.860+0000 7f9ae3c1f740 1 --2- 172.21.15.46:0/1213989549 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5628ecb9adb0 0x5628ecb9f120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:20:19.863 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:19.860+0000 7f9ae3c1f740 1 --2- 172.21.15.46:0/1213989549 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5628ecb91fe0 0x5628ecb9ebe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:20:19.863 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:19.860+0000 7f9ae3c1f740 1 --2- 172.21.15.46:0/1213989549 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5628ecb916c0 0x5628ecb9e6a0 unknown :-1 s=CLOSED pgs=777 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:20:19.863 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:19.860+0000 7f9abeff5640 0 client.16854 ms_handle_reset on v2:172.21.15.142:6838/2519143918 2024-03-31T23:20:19.863 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:19.861+0000 7f9ae3c1f740 1 -- 172.21.15.46:0/1213989549 >> 172.21.15.46:0/1213989549 conn(0x5628ecb8bfe0 msgr2=0x5628ecb8d200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:20:19.863 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:19.861+0000 7f9ae3c1f740 1 -- 172.21.15.46:0/1213989549 shutdown_connections 2024-03-31T23:20:19.863 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:19.861+0000 7f9ae3c1f740 1 -- 172.21.15.46:0/1213989549 wait complete. 2024-03-31T23:20:19.863 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:19.861+0000 7f9ae3c1f740 20 client.16854 trim_cache size 0 max 16384 2024-03-31T23:20:19.864 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout:{ 2024-03-31T23:20:19.864 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout: "message": "", 2024-03-31T23:20:19.864 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout: "return_code": 0 2024-03-31T23:20:19.864 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout:} 2024-03-31T23:20:19.865 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:19.861+0000 7f9ae1049640 1 -- 172.21.15.46:0/2231549109 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f9ac40a26b0 msgr2=0x7f9ac40a4b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:20:19.865 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:19.861+0000 7f9ae1049640 1 --2- 172.21.15.46:0/2231549109 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f9ac40a26b0 0x7f9ac40a4b70 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f9adc104f90 tx=0x7f9ad0011440 comp rx=0 tx=0).stop 2024-03-31T23:20:19.865 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:19.861+0000 7f9ae1049640 1 -- 172.21.15.46:0/2231549109 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9adc1055d0 msgr2=0x7f9adc19c7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:20:19.865 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:19.861+0000 7f9ae1049640 1 --2- 172.21.15.46:0/2231549109 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9adc1055d0 0x7f9adc19c7d0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f9ad4009fd0 tx=0x7f9ad400ef10 comp rx=0 tx=0).stop 2024-03-31T23:20:19.865 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:19.861+0000 7f9ae1049640 1 -- 172.21.15.46:0/2231549109 shutdown_connections 2024-03-31T23:20:19.865 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:19.861+0000 7f9ae1049640 1 --2- 172.21.15.46:0/2231549109 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f9ac40a26b0 0x7f9ac40a4b70 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:20:19.865 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:19.861+0000 7f9ae1049640 1 --2- 172.21.15.46:0/2231549109 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9adc1055d0 0x7f9adc19c7d0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:20:19.865 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:19.861+0000 7f9ae1049640 1 --2- 172.21.15.46:0/2231549109 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9adc101240 0x7f9adc198450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:20:19.865 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:19.861+0000 7f9ae1049640 1 --2- 172.21.15.46:0/2231549109 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9adc1008a0 0x7f9adc197f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:20:19.865 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:19.861+0000 7f9ae1049640 1 -- 172.21.15.46:0/2231549109 >> 172.21.15.46:0/2231549109 conn(0x7f9adc077420 msgr2=0x7f9adc06a250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:20:19.865 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:19.862+0000 7f9ae1049640 1 -- 172.21.15.46:0/2231549109 shutdown_connections 2024-03-31T23:20:19.865 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:19.862+0000 7f9ae1049640 1 -- 172.21.15.46:0/2231549109 wait complete. 2024-03-31T23:20:20.206 INFO:tasks.workunit.client.0.smithi046.stdout: 1 188140 2.22 MB/sec execute 326 sec latency 2229.711 ms 2024-03-31T23:20:20.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:20:20 smithi142 ceph-mon[35271]: pgmap v267: 129 pgs: 129 active+clean; 114 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 402 KiB/s rd, 1007 KiB/s wr, 98 op/s 2024-03-31T23:20:21.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:20:20 smithi162 ceph-mon[36237]: pgmap v267: 129 pgs: 129 active+clean; 114 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 402 KiB/s rd, 1007 KiB/s wr, 98 op/s 2024-03-31T23:20:21.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:20:20 smithi046 ceph-mon[32498]: pgmap v267: 129 pgs: 129 active+clean; 114 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 402 KiB/s rd, 1007 KiB/s wr, 98 op/s 2024-03-31T23:20:21.206 INFO:tasks.workunit.client.0.smithi046.stdout: 1 188140 2.22 MB/sec execute 327 sec latency 3229.774 ms 2024-03-31T23:20:22.206 INFO:tasks.workunit.client.0.smithi046.stdout: 1 188140 2.21 MB/sec execute 328 sec latency 4229.874 ms 2024-03-31T23:20:22.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:20:22 smithi142 ceph-mon[35271]: pgmap v268: 129 pgs: 129 active+clean; 114 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 390 KiB/s rd, 976 KiB/s wr, 95 op/s 2024-03-31T23:20:23.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:20:22 smithi162 ceph-mon[36237]: pgmap v268: 129 pgs: 129 active+clean; 114 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 390 KiB/s rd, 976 KiB/s wr, 95 op/s 2024-03-31T23:20:23.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:20:22 smithi046 ceph-mon[32498]: pgmap v268: 129 pgs: 129 active+clean; 114 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 390 KiB/s rd, 976 KiB/s wr, 95 op/s 2024-03-31T23:20:23.206 INFO:tasks.workunit.client.0.smithi046.stdout: 1 188275 2.21 MB/sec execute 329 sec latency 4767.482 ms 2024-03-31T23:20:24.206 INFO:tasks.workunit.client.0.smithi046.stdout: 1 188543 2.20 MB/sec execute 330 sec latency 15.608 ms 2024-03-31T23:20:24.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:20:24 smithi142 ceph-mon[35271]: pgmap v269: 129 pgs: 129 active+clean; 114 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 335 KiB/s rd, 839 KiB/s wr, 81 op/s 2024-03-31T23:20:25.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:20:24 smithi162 ceph-mon[36237]: pgmap v269: 129 pgs: 129 active+clean; 114 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 335 KiB/s rd, 839 KiB/s wr, 81 op/s 2024-03-31T23:20:25.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:20:24 smithi046 ceph-mon[32498]: pgmap v269: 129 pgs: 129 active+clean; 114 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 335 KiB/s rd, 839 KiB/s wr, 81 op/s 2024-03-31T23:20:25.206 INFO:tasks.workunit.client.0.smithi046.stdout: 1 188945 2.20 MB/sec execute 331 sec latency 15.613 ms 2024-03-31T23:20:26.206 INFO:tasks.workunit.client.0.smithi046.stdout: 1 189429 2.20 MB/sec execute 332 sec latency 40.463 ms 2024-03-31T23:20:26.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:20:26 smithi142 ceph-mon[35271]: pgmap v270: 129 pgs: 129 active+clean; 111 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 390 KiB/s rd, 1.0 MiB/s wr, 104 op/s 2024-03-31T23:20:27.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:20:26 smithi162 ceph-mon[36237]: pgmap v270: 129 pgs: 129 active+clean; 111 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 390 KiB/s rd, 1.0 MiB/s wr, 104 op/s 2024-03-31T23:20:27.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:20:26 smithi046 ceph-mon[32498]: pgmap v270: 129 pgs: 129 active+clean; 111 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 390 KiB/s rd, 1.0 MiB/s wr, 104 op/s 2024-03-31T23:20:27.206 INFO:tasks.workunit.client.0.smithi046.stdout: 1 189874 2.20 MB/sec execute 333 sec latency 23.336 ms 2024-03-31T23:20:28.209 INFO:tasks.workunit.client.0.smithi046.stdout: 1 190409 2.19 MB/sec execute 334 sec latency 20.674 ms 2024-03-31T23:20:28.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:20:28 smithi142 ceph-mon[35271]: pgmap v271: 129 pgs: 129 active+clean; 111 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 165 KiB/s rd, 513 KiB/s wr, 50 op/s 2024-03-31T23:20:29.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:20:28 smithi162 ceph-mon[36237]: pgmap v271: 129 pgs: 129 active+clean; 111 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 165 KiB/s rd, 513 KiB/s wr, 50 op/s 2024-03-31T23:20:29.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:20:28 smithi046 ceph-mon[32498]: pgmap v271: 129 pgs: 129 active+clean; 111 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 165 KiB/s rd, 513 KiB/s wr, 50 op/s 2024-03-31T23:20:29.206 INFO:tasks.workunit.client.0.smithi046.stdout: 1 190999 2.20 MB/sec execute 335 sec latency 15.606 ms 2024-03-31T23:20:30.206 INFO:tasks.workunit.client.0.smithi046.stdout: 1 191493 2.21 MB/sec execute 336 sec latency 16.771 ms 2024-03-31T23:20:30.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:20:30 smithi142 ceph-mon[35271]: pgmap v272: 129 pgs: 129 active+clean; 112 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 384 KiB/s rd, 1.0 MiB/s wr, 105 op/s 2024-03-31T23:20:31.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:20:30 smithi162 ceph-mon[36237]: pgmap v272: 129 pgs: 129 active+clean; 112 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 384 KiB/s rd, 1.0 MiB/s wr, 105 op/s 2024-03-31T23:20:31.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:20:30 smithi046 ceph-mon[32498]: pgmap v272: 129 pgs: 129 active+clean; 112 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 384 KiB/s rd, 1.0 MiB/s wr, 105 op/s 2024-03-31T23:20:31.206 INFO:tasks.workunit.client.0.smithi046.stdout: 1 191620 2.21 MB/sec execute 337 sec latency 753.575 ms 2024-03-31T23:20:32.206 INFO:tasks.workunit.client.0.smithi046.stdout: 1 191620 2.20 MB/sec execute 338 sec latency 1753.644 ms 2024-03-31T23:20:32.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:20:32 smithi142 ceph-mon[35271]: pgmap v273: 129 pgs: 129 active+clean; 112 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 274 KiB/s rd, 733 KiB/s wr, 78 op/s 2024-03-31T23:20:32.958 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:32.957+0000 7f7f59f3e640 1 -- 172.21.15.46:0/3780870805 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7f541055f0 msgr2=0x7f7f541059d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:20:32.959 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:32.957+0000 7f7f59f3e640 1 --2- 172.21.15.46:0/3780870805 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7f541055f0 0x7f7f541059d0 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7f7f44009900 tx=0x7f7f4402bc30 comp rx=0 tx=0).stop 2024-03-31T23:20:32.959 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:32.957+0000 7f7f59f3e640 1 -- 172.21.15.46:0/3780870805 shutdown_connections 2024-03-31T23:20:32.959 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:32.957+0000 7f7f59f3e640 1 --2- 172.21.15.46:0/3780870805 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7f54101260 0x7f7f5410d480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:20:32.959 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:32.957+0000 7f7f59f3e640 1 --2- 172.21.15.46:0/3780870805 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7f541008a0 0x7f7f54100d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:20:32.959 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:32.957+0000 7f7f59f3e640 1 --2- 172.21.15.46:0/3780870805 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7f541055f0 0x7f7f541059d0 unknown :-1 s=CLOSED pgs=778 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:20:32.959 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:32.957+0000 7f7f59f3e640 1 -- 172.21.15.46:0/3780870805 >> 172.21.15.46:0/3780870805 conn(0x7f7f540774a0 msgr2=0x7f7f540778b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:20:32.959 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:32.957+0000 7f7f59f3e640 1 -- 172.21.15.46:0/3780870805 shutdown_connections 2024-03-31T23:20:32.959 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:32.957+0000 7f7f59f3e640 1 -- 172.21.15.46:0/3780870805 wait complete. 2024-03-31T23:20:32.959 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:32.957+0000 7f7f59f3e640 1 Processor -- start 2024-03-31T23:20:32.961 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:32.958+0000 7f7f59f3e640 1 -- start start 2024-03-31T23:20:32.961 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:32.958+0000 7f7f59f3e640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7f541008a0 0x7f7f54197ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:20:32.961 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:32.958+0000 7f7f59f3e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7f54101260 0x7f7f54198420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:20:32.961 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:32.958+0000 7f7f59f3e640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7f541055f0 0x7f7f5419c7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:20:32.961 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:32.958+0000 7f7f59f3e640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f7f541109c0 con 0x7f7f54101260 2024-03-31T23:20:32.962 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:32.958+0000 7f7f59f3e640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f7f54110840 con 0x7f7f541008a0 2024-03-31T23:20:32.962 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:32.958+0000 7f7f59f3e640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f7f54110b40 con 0x7f7f541055f0 2024-03-31T23:20:32.962 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:32.958+0000 7f7f52ffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7f54101260 0x7f7f54198420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:20:32.962 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:32.958+0000 7f7f537fe640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7f541008a0 0x7f7f54197ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:20:32.962 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:32.958+0000 7f7f52ffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7f54101260 0x7f7f54198420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37344/0 (socket says 172.21.15.46:37344) 2024-03-31T23:20:32.962 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:32.958+0000 7f7f53fff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7f541055f0 0x7f7f5419c7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:20:32.962 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:32.958+0000 7f7f52ffd640 1 -- 172.21.15.46:0/3651011016 learned_addr learned my addr 172.21.15.46:0/3651011016 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:20:32.962 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:32.958+0000 7f7f52ffd640 1 -- 172.21.15.46:0/3651011016 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7f541055f0 msgr2=0x7f7f5419c7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:20:32.962 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:32.958+0000 7f7f52ffd640 1 --2- 172.21.15.46:0/3651011016 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7f541055f0 0x7f7f5419c7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:20:32.962 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:32.958+0000 7f7f52ffd640 1 -- 172.21.15.46:0/3651011016 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7f541008a0 msgr2=0x7f7f54197ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:20:32.962 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:32.958+0000 7f7f52ffd640 1 --2- 172.21.15.46:0/3651011016 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7f541008a0 0x7f7f54197ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:20:32.962 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:32.958+0000 7f7f52ffd640 1 -- 172.21.15.46:0/3651011016 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f5419ced0 con 0x7f7f54101260 2024-03-31T23:20:32.962 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:32.958+0000 7f7f537fe640 1 --2- 172.21.15.46:0/3651011016 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7f541008a0 0x7f7f54197ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:20:32.963 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:32.959+0000 7f7f52ffd640 1 --2- 172.21.15.46:0/3651011016 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7f54101260 0x7f7f54198420 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7f7f4800e9f0 tx=0x7f7f4800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:20:32.963 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:32.959+0000 7f7f50ff9640 1 -- 172.21.15.46:0/3651011016 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f4800cd80 con 0x7f7f54101260 2024-03-31T23:20:32.963 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:32.959+0000 7f7f50ff9640 1 -- 172.21.15.46:0/3651011016 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7f480108e0 con 0x7f7f54101260 2024-03-31T23:20:32.963 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:32.959+0000 7f7f50ff9640 1 -- 172.21.15.46:0/3651011016 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f48018920 con 0x7f7f54101260 2024-03-31T23:20:32.963 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:32.959+0000 7f7f59f3e640 1 -- 172.21.15.46:0/3651011016 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7f5419d1c0 con 0x7f7f54101260 2024-03-31T23:20:32.963 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:32.959+0000 7f7f59f3e640 1 -- 172.21.15.46:0/3651011016 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7f541a4a50 con 0x7f7f54101260 2024-03-31T23:20:32.969 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:32.961+0000 7f7f50ff9640 1 -- 172.21.15.46:0/3651011016 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 25) ==== 137451+0+0 (secure 0 0 0) 0x7f7f48010a90 con 0x7f7f54101260 2024-03-31T23:20:32.969 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:32.961+0000 7f7f5cc00740 1 Processor -- start 2024-03-31T23:20:32.969 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:32.961+0000 7f7f5cc00740 1 -- start start 2024-03-31T23:20:32.969 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:32.962+0000 7f7f50ff9640 1 --2- 172.21.15.46:0/3651011016 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f7f3c0a26b0 0x7f7f3c0a4b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:20:32.969 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:32.962+0000 7f7f5cc00740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556a2298e6c0 0x556a2298eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:20:32.969 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:32.962+0000 7f7f5cc00740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x556a2298efe0 0x556a22997870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:20:32.969 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:32.962+0000 7f7f5cc00740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556a22997db0 0x556a2299a1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:20:32.969 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:32.962+0000 7f7f5cc00740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x556a228f18f0 con 0x556a2298e6c0 2024-03-31T23:20:32.969 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:32.962+0000 7f7f5cc00740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x556a2289a8d0 con 0x556a2298efe0 2024-03-31T23:20:32.969 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:32.962+0000 7f7f5cc00740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x556a228ed990 con 0x556a22997db0 2024-03-31T23:20:32.969 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:32.962+0000 7f7f50ff9640 1 -- 172.21.15.46:0/3651011016 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 9554+0+0 (secure 0 0 0) 0x7f7f480cf210 con 0x7f7f54101260 2024-03-31T23:20:32.969 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:32.962+0000 7f7f53fff640 1 --2- 172.21.15.46:0/3651011016 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f7f3c0a26b0 0x7f7f3c0a4b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:20:32.970 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:32.962+0000 7f7f537fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556a2298e6c0 0x556a2298eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:20:32.970 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:32.962+0000 7f7f537fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556a2298e6c0 0x556a2298eaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37350/0 (socket says 172.21.15.46:37350) 2024-03-31T23:20:32.970 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:32.962+0000 7f7f537fe640 1 -- 172.21.15.46:0/4201439180 learned_addr learned my addr 172.21.15.46:0/4201439180 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:20:32.970 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:32.962+0000 7f7f53fff640 1 --2- 172.21.15.46:0/4201439180 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556a22997db0 0x556a2299a1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:20:32.970 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:32.962+0000 7f7f53fff640 1 -- 172.21.15.46:0/4201439180 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x556a2298efe0 msgr2=0x556a22997870 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-31T23:20:32.970 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:32.962+0000 7f7f53fff640 1 --2- 172.21.15.46:0/4201439180 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x556a2298efe0 0x556a22997870 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:20:32.970 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:32.962+0000 7f7f53fff640 1 -- 172.21.15.46:0/4201439180 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556a2298e6c0 msgr2=0x556a2298eaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:20:32.970 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:32.962+0000 7f7f53fff640 1 --2- 172.21.15.46:0/4201439180 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556a2298e6c0 0x556a2298eaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:20:32.970 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:32.962+0000 7f7f53fff640 1 -- 172.21.15.46:0/4201439180 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556a22606210 con 0x556a22997db0 2024-03-31T23:20:32.970 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:32.962+0000 7f7f537fe640 1 --2- 172.21.15.46:0/4201439180 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556a2298e6c0 0x556a2298eaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:20:32.970 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:32.963+0000 7f7f53fff640 1 --2- 172.21.15.46:0/3651011016 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f7f3c0a26b0 0x7f7f3c0a4b70 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f7f4c005fd0 tx=0x7f7f4c009f00 comp rx=0 tx=0).ready entity=mgr.15654 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:20:32.970 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:32.963+0000 7f7f53fff640 1 --2- 172.21.15.46:0/4201439180 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556a22997db0 0x556a2299a1a0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f7f4c00fc20 tx=0x7f7f4c01d000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6d6131d3da999bb9 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:20:32.970 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:32.963+0000 7f7f39ffb640 1 -- 172.21.15.46:0/4201439180 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f4c024070 con 0x556a22997db0 2024-03-31T23:20:32.971 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:32.963+0000 7f7f39ffb640 1 -- 172.21.15.46:0/4201439180 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7f4c003e20 con 0x556a22997db0 2024-03-31T23:20:32.971 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:32.963+0000 7f7f39ffb640 1 -- 172.21.15.46:0/4201439180 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f4c004280 con 0x556a22997db0 2024-03-31T23:20:32.971 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:32.964+0000 7f7f5cc00740 1 -- 172.21.15.46:0/4201439180 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556a22997db0 msgr2=0x556a2299a1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:20:32.971 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:32.964+0000 7f7f5cc00740 1 --2- 172.21.15.46:0/4201439180 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556a22997db0 0x556a2299a1a0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f7f4c00fc20 tx=0x7f7f4c01d000 comp rx=0 tx=0).stop 2024-03-31T23:20:32.971 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:32.964+0000 7f7f5cc00740 1 -- 172.21.15.46:0/4201439180 shutdown_connections 2024-03-31T23:20:32.971 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:32.964+0000 7f7f5cc00740 1 --2- 172.21.15.46:0/4201439180 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556a22997db0 0x556a2299a1a0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:20:32.971 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:32.964+0000 7f7f5cc00740 1 --2- 172.21.15.46:0/4201439180 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x556a2298efe0 0x556a22997870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:20:32.971 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:32.964+0000 7f7f5cc00740 1 --2- 172.21.15.46:0/4201439180 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556a2298e6c0 0x556a2298eaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:20:32.971 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:32.964+0000 7f7f5cc00740 1 -- 172.21.15.46:0/4201439180 >> 172.21.15.46:0/4201439180 conn(0x556a22988fe0 msgr2=0x556a229893c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:20:32.971 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:32.964+0000 7f7f5cc00740 1 -- 172.21.15.46:0/4201439180 shutdown_connections 2024-03-31T23:20:32.971 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:32.964+0000 7f7f5cc00740 1 -- 172.21.15.46:0/4201439180 wait complete. 2024-03-31T23:20:32.971 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:32.964+0000 7f7f5cc00740 1 -- start start 2024-03-31T23:20:32.971 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:32.965+0000 7f7f5cc00740 1 Processor -- start 2024-03-31T23:20:32.972 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:32.965+0000 7f7f5cc00740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556a2298e6c0 0x556a22994e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:20:32.972 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:32.965+0000 7f7f5cc00740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556a2298efe0 0x556a229953c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:20:32.972 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:32.965+0000 7f7f5cc00740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x556a22997db0 0x556a22995900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:20:32.972 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:32.965+0000 7f7f537fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556a2298e6c0 0x556a22994e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:20:32.972 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:32.965+0000 7f7f5cc00740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x556a2299a980 con 0x556a2298e6c0 2024-03-31T23:20:32.972 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:32.965+0000 7f7f5cc00740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x556a2288ff80 con 0x556a22997db0 2024-03-31T23:20:32.972 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:32.965+0000 7f7f5cc00740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x556a2299ab00 con 0x556a2298efe0 2024-03-31T23:20:32.972 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:32.965+0000 7f7f537fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556a2298e6c0 0x556a22994e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37356/0 (socket says 172.21.15.46:37356) 2024-03-31T23:20:32.972 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:32.965+0000 7f7f5cc00740 10 client.0 _ll_register_callbacks cb 0x556a228e8290 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T23:20:32.972 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:32.965+0000 7f7f537fe640 1 -- 172.21.15.46:0/3190189743 learned_addr learned my addr 172.21.15.46:0/3190189743 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:20:32.972 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:32.965+0000 7f7f52ffd640 1 --2- 172.21.15.46:0/3190189743 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556a2298efe0 0x556a229953c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:20:32.972 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:32.966+0000 7f7f53fff640 1 --2- 172.21.15.46:0/3190189743 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x556a22997db0 0x556a22995900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:20:32.972 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:32.966+0000 7f7f537fe640 1 -- 172.21.15.46:0/3190189743 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556a2298efe0 msgr2=0x556a229953c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:20:32.973 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:32.966+0000 7f7f537fe640 1 --2- 172.21.15.46:0/3190189743 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556a2298efe0 0x556a229953c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:20:32.973 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:32.966+0000 7f7f537fe640 1 -- 172.21.15.46:0/3190189743 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x556a22997db0 msgr2=0x556a22995900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:20:32.973 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:32.966+0000 7f7f537fe640 1 --2- 172.21.15.46:0/3190189743 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x556a22997db0 0x556a22995900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:20:32.973 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:32.966+0000 7f7f537fe640 1 -- 172.21.15.46:0/3190189743 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556a226d2620 con 0x556a2298e6c0 2024-03-31T23:20:32.973 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:32.966+0000 7f7f52ffd640 1 --2- 172.21.15.46:0/3190189743 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556a2298efe0 0x556a229953c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:20:32.973 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:32.966+0000 7f7f537fe640 1 --2- 172.21.15.46:0/3190189743 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556a2298e6c0 0x556a22994e80 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7f7f44000c00 tx=0x7f7f44004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=14b02be421647e6 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:20:32.973 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:32.966+0000 7f7f32ffd640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-31T23:20:32.973 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:32.966+0000 7f7f32ffd640 1 -- 172.21.15.46:0/3190189743 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f440035f0 con 0x556a2298e6c0 2024-03-31T23:20:32.973 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:32.966+0000 7f7f32ffd640 1 -- 172.21.15.46:0/3190189743 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 9554+0+0 (secure 0 0 0) 0x7f7f44003790 con 0x556a2298e6c0 2024-03-31T23:20:32.973 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:32.966+0000 7f7f5cc00740 1 -- 172.21.15.46:0/3190189743 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7f4c003e20 con 0x556a2298e6c0 2024-03-31T23:20:32.973 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:32.966+0000 7f7f32ffd640 1 -- 172.21.15.46:0/3190189743 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f7f44003990 con 0x556a2298e6c0 2024-03-31T23:20:32.973 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:32.966+0000 7f7f5cc00740 10 client.16866 fetch_fsmap learned FSMap version 35 2024-03-31T23:20:32.974 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:32.966+0000 7f7f5cc00740 1 -- 172.21.15.46:0/3190189743 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x556a229a6920 con 0x556a2298e6c0 2024-03-31T23:20:32.974 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:32.967+0000 7f7f32ffd640 1 -- 172.21.15.46:0/3190189743 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7f4c003e20 con 0x556a2298e6c0 2024-03-31T23:20:32.974 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:32.967+0000 7f7f32ffd640 1 -- 172.21.15.46:0/3190189743 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 35) ==== 8366+0+0 (secure 0 0 0) 0x7f7f44034bb0 con 0x556a2298e6c0 2024-03-31T23:20:32.974 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:32.967+0000 7f7f5cc00740 10 client.16866 fetch_fsmap finished waiting for FSMap version 35 2024-03-31T23:20:32.974 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:32.967+0000 7f7f5cc00740 10 client.16866 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-31T23:20:32.974 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:32.967+0000 7f7f5cc00740 20 client.16866 populate_metadata read hostname 'smithi046' 2024-03-31T23:20:32.974 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:32.967+0000 7f7f5cc00740 1 --2- 172.21.15.46:0/3190189743 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x556a229a7de0 0x556a229aa1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:20:32.974 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:32.967+0000 7f7f5cc00740 4 client.16866 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-31T23:20:32.974 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:32.967+0000 7f7f5cc00740 1 -- 172.21.15.46:0/3190189743 --> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556a22905120 con 0x556a229a7de0 2024-03-31T23:20:32.974 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:32.967+0000 7f7f52ffd640 1 --2- 172.21.15.46:0/3190189743 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x556a229a7de0 0x556a229aa1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:20:32.974 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:32.968+0000 7f7f52ffd640 1 --2- 172.21.15.46:0/3190189743 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x556a229a7de0 0x556a229aa1d0 crc :-1 s=READY pgs=153 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5aa8f0f128229de7 server_cookie=bb32772f6f968f70 in_seq=0 out_seq=0 2024-03-31T23:20:32.974 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:32.968+0000 7f7f32ffd640 10 client.16866 ms_handle_connect on v2:172.21.15.142:6838/2519143918 2024-03-31T23:20:32.975 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:32.969+0000 7f7f32ffd640 1 -- 172.21.15.46:0/3190189743 <== mds.0 v2:172.21.15.142:6838/2519143918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x556a22905120 con 0x556a229a7de0 2024-03-31T23:20:32.976 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:32.969+0000 7f7f32ffd640 10 client.16866 handle_command_reply: tid=0 2024-03-31T23:20:32.976 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:32.969+0000 7f7f5cc00740 1 client.16866 shutdown 2024-03-31T23:20:32.976 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:32.969+0000 7f7f5cc00740 1 -- 172.21.15.46:0/3190189743 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556a2298e6c0 msgr2=0x556a22994e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:20:32.976 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:32.969+0000 7f7f5cc00740 1 --2- 172.21.15.46:0/3190189743 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556a2298e6c0 0x556a22994e80 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7f7f44000c00 tx=0x7f7f44004270 comp rx=0 tx=0).stop 2024-03-31T23:20:32.976 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:32.969+0000 7f7f5cc00740 1 -- 172.21.15.46:0/3190189743 shutdown_connections 2024-03-31T23:20:32.976 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:32.969+0000 7f7f5cc00740 1 --2- 172.21.15.46:0/3190189743 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x556a229a7de0 0x556a229aa1d0 crc :-1 s=READY pgs=153 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:20:32.976 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:32.969+0000 7f7f5cc00740 1 --2- 172.21.15.46:0/3190189743 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x556a22997db0 0x556a22995900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:20:32.976 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:32.969+0000 7f7f5cc00740 1 --2- 172.21.15.46:0/3190189743 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556a2298efe0 0x556a229953c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:20:32.976 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:32.969+0000 7f7f5cc00740 1 --2- 172.21.15.46:0/3190189743 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556a2298e6c0 0x556a22994e80 unknown :-1 s=CLOSED pgs=780 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:20:32.976 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:32.969+0000 7f7f5cc00740 1 -- 172.21.15.46:0/3190189743 >> 172.21.15.46:0/3190189743 conn(0x556a22988fe0 msgr2=0x556a2298ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:20:32.976 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:32.969+0000 7f7f32ffd640 0 client.16866 ms_handle_reset on v2:172.21.15.142:6838/2519143918 2024-03-31T23:20:32.976 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:32.969+0000 7f7f5cc00740 1 -- 172.21.15.46:0/3190189743 shutdown_connections 2024-03-31T23:20:32.976 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:32.969+0000 7f7f5cc00740 1 -- 172.21.15.46:0/3190189743 wait complete. 2024-03-31T23:20:32.977 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:32.969+0000 7f7f5cc00740 20 client.16866 trim_cache size 0 max 16384 2024-03-31T23:20:32.988 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:32.985+0000 7f7f5cc00740 1 Processor -- start 2024-03-31T23:20:32.989 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:32.986+0000 7f7f5cc00740 1 -- start start 2024-03-31T23:20:32.989 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:32.986+0000 7f7f5cc00740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556a2298e6c0 0x556a229a3720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:20:32.989 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:32.986+0000 7f7f5cc00740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556a2298efe0 0x556a229afaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:20:32.989 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:32.986+0000 7f7f5cc00740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x556a22997db0 0x556a22993770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:20:32.989 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:32.986+0000 7f7f5cc00740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f7f541b1570 con 0x556a2298efe0 2024-03-31T23:20:32.989 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:32.986+0000 7f7f5cc00740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f7f541aedb0 con 0x556a22997db0 2024-03-31T23:20:32.989 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:32.986+0000 7f7f5cc00740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f7f541acc20 con 0x556a2298e6c0 2024-03-31T23:20:32.989 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:32.986+0000 7f7f52ffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556a2298efe0 0x556a229afaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:20:32.989 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:32.986+0000 7f7f52ffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556a2298efe0 0x556a229afaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37362/0 (socket says 172.21.15.46:37362) 2024-03-31T23:20:32.989 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:32.986+0000 7f7f52ffd640 1 -- 172.21.15.46:0/2863815470 learned_addr learned my addr 172.21.15.46:0/2863815470 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:20:32.989 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:32.986+0000 7f7f537fe640 1 --2- 172.21.15.46:0/2863815470 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556a2298e6c0 0x556a229a3720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:20:32.989 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:32.986+0000 7f7f53fff640 1 --2- 172.21.15.46:0/2863815470 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x556a22997db0 0x556a22993770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:20:32.989 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:32.986+0000 7f7f52ffd640 1 -- 172.21.15.46:0/2863815470 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556a2298e6c0 msgr2=0x556a229a3720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:20:32.990 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:32.986+0000 7f7f52ffd640 1 --2- 172.21.15.46:0/2863815470 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556a2298e6c0 0x556a229a3720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:20:32.990 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:32.986+0000 7f7f52ffd640 1 -- 172.21.15.46:0/2863815470 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x556a22997db0 msgr2=0x556a22993770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:20:32.990 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:32.986+0000 7f7f52ffd640 1 --2- 172.21.15.46:0/2863815470 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x556a22997db0 0x556a22993770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:20:32.990 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:32.986+0000 7f7f52ffd640 1 -- 172.21.15.46:0/2863815470 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556a22990a60 con 0x556a2298efe0 2024-03-31T23:20:32.990 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:32.986+0000 7f7f52ffd640 1 --2- 172.21.15.46:0/2863815470 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556a2298efe0 0x556a229afaa0 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7f7f480cc600 tx=0x7f7f48086040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cba68d19185cfc4 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:20:32.990 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:32.986+0000 7f7f397fa640 1 -- 172.21.15.46:0/2863815470 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f480a3f90 con 0x556a2298efe0 2024-03-31T23:20:32.990 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:32.987+0000 7f7f397fa640 1 -- 172.21.15.46:0/2863815470 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7f48099930 con 0x556a2298efe0 2024-03-31T23:20:32.990 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:32.987+0000 7f7f397fa640 1 -- 172.21.15.46:0/2863815470 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f4809f410 con 0x556a2298efe0 2024-03-31T23:20:32.990 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:32.987+0000 7f7f5cc00740 1 -- 172.21.15.46:0/2863815470 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556a2298efe0 msgr2=0x556a229afaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:20:32.990 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:32.987+0000 7f7f5cc00740 1 --2- 172.21.15.46:0/2863815470 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556a2298efe0 0x556a229afaa0 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7f7f480cc600 tx=0x7f7f48086040 comp rx=0 tx=0).stop 2024-03-31T23:20:32.990 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:32.987+0000 7f7f5cc00740 1 -- 172.21.15.46:0/2863815470 shutdown_connections 2024-03-31T23:20:32.990 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:32.987+0000 7f7f5cc00740 1 --2- 172.21.15.46:0/2863815470 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x556a22997db0 0x556a22993770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:20:32.990 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:32.987+0000 7f7f5cc00740 1 --2- 172.21.15.46:0/2863815470 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556a2298efe0 0x556a229afaa0 unknown :-1 s=CLOSED pgs=781 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:20:32.991 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:32.987+0000 7f7f5cc00740 1 --2- 172.21.15.46:0/2863815470 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556a2298e6c0 0x556a229a3720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:20:32.991 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:32.987+0000 7f7f5cc00740 1 -- 172.21.15.46:0/2863815470 >> 172.21.15.46:0/2863815470 conn(0x556a22988fe0 msgr2=0x556a229893c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:20:32.991 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:32.987+0000 7f7f5cc00740 1 -- 172.21.15.46:0/2863815470 shutdown_connections 2024-03-31T23:20:32.991 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:32.987+0000 7f7f5cc00740 1 -- 172.21.15.46:0/2863815470 wait complete. 2024-03-31T23:20:32.991 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:32.988+0000 7f7f5cc00740 1 -- start start 2024-03-31T23:20:32.991 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:32.988+0000 7f7f5cc00740 1 Processor -- start 2024-03-31T23:20:32.994 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:32.988+0000 7f7f5cc00740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x556a2298e6c0 0x556a2299ff40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:20:32.994 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:32.988+0000 7f7f5cc00740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556a2298efe0 0x556a22991730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:20:32.994 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:32.988+0000 7f7f5cc00740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556a22997db0 0x556a22991c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:20:32.994 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:32.988+0000 7f7f5cc00740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f7f541ad270 con 0x556a2298efe0 2024-03-31T23:20:32.994 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:32.988+0000 7f7f5cc00740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f7f541ae6e0 con 0x556a2298e6c0 2024-03-31T23:20:32.994 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:32.988+0000 7f7f5cc00740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f7f541ad940 con 0x556a22997db0 2024-03-31T23:20:32.994 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:32.988+0000 7f7f5cc00740 10 client.0 _ll_register_callbacks cb 0x7f7f44009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T23:20:32.994 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:32.988+0000 7f7f52ffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556a2298efe0 0x556a22991730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:20:32.994 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:32.988+0000 7f7f52ffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556a2298efe0 0x556a22991730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37374/0 (socket says 172.21.15.46:37374) 2024-03-31T23:20:32.994 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:32.988+0000 7f7f52ffd640 1 -- 172.21.15.46:0/1775508051 learned_addr learned my addr 172.21.15.46:0/1775508051 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:20:32.994 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:32.988+0000 7f7f537fe640 1 --2- 172.21.15.46:0/1775508051 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x556a2298e6c0 0x556a2299ff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:20:32.995 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:32.988+0000 7f7f53fff640 1 --2- 172.21.15.46:0/1775508051 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556a22997db0 0x556a22991c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:20:32.995 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:32.988+0000 7f7f52ffd640 1 -- 172.21.15.46:0/1775508051 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556a22997db0 msgr2=0x556a22991c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:20:32.995 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:32.988+0000 7f7f52ffd640 1 --2- 172.21.15.46:0/1775508051 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556a22997db0 0x556a22991c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:20:32.995 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:32.988+0000 7f7f52ffd640 1 -- 172.21.15.46:0/1775508051 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x556a2298e6c0 msgr2=0x556a2299ff40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:20:32.995 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:32.988+0000 7f7f52ffd640 1 --2- 172.21.15.46:0/1775508051 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x556a2298e6c0 0x556a2299ff40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:20:32.995 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:32.988+0000 7f7f52ffd640 1 -- 172.21.15.46:0/1775508051 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556a229924a0 con 0x556a2298efe0 2024-03-31T23:20:32.995 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:32.988+0000 7f7f537fe640 1 --2- 172.21.15.46:0/1775508051 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x556a2298e6c0 0x556a2299ff40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:20:32.995 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:32.989+0000 7f7f52ffd640 1 --2- 172.21.15.46:0/1775508051 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556a2298efe0 0x556a22991730 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7f7f48097a40 tx=0x7f7f4809f820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=87216742a78136e0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:20:32.995 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:32.989+0000 7f7f32ffd640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-31T23:20:32.995 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:32.989+0000 7f7f32ffd640 1 -- 172.21.15.46:0/1775508051 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f480b4060 con 0x556a2298efe0 2024-03-31T23:20:32.995 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:32.989+0000 7f7f32ffd640 1 -- 172.21.15.46:0/1775508051 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 9554+0+0 (secure 0 0 0) 0x7f7f48097ca0 con 0x556a2298efe0 2024-03-31T23:20:32.995 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:32.989+0000 7f7f5cc00740 1 -- 172.21.15.46:0/1775508051 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7f48099930 con 0x556a2298efe0 2024-03-31T23:20:32.995 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:32.989+0000 7f7f32ffd640 1 -- 172.21.15.46:0/1775508051 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f7f480ae030 con 0x556a2298efe0 2024-03-31T23:20:32.995 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:32.989+0000 7f7f5cc00740 10 client.16872 fetch_fsmap learned FSMap version 35 2024-03-31T23:20:32.996 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:32.989+0000 7f7f5cc00740 1 -- 172.21.15.46:0/1775508051 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x556a22a0dfc0 con 0x556a2298efe0 2024-03-31T23:20:32.996 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:32.989+0000 7f7f32ffd640 1 -- 172.21.15.46:0/1775508051 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7f48099930 con 0x556a2298efe0 2024-03-31T23:20:32.996 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:32.989+0000 7f7f32ffd640 1 -- 172.21.15.46:0/1775508051 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 35) ==== 8366+0+0 (secure 0 0 0) 0x7f7f48084c00 con 0x556a2298efe0 2024-03-31T23:20:32.996 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:32.989+0000 7f7f5cc00740 10 client.16872 fetch_fsmap finished waiting for FSMap version 35 2024-03-31T23:20:32.996 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:32.989+0000 7f7f5cc00740 10 client.16872 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-31T23:20:32.996 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:32.989+0000 7f7f5cc00740 20 client.16872 populate_metadata read hostname 'smithi046' 2024-03-31T23:20:32.996 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:32.989+0000 7f7f5cc00740 1 --2- 172.21.15.46:0/1775508051 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x556a22a0e640 0x556a22a10a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:20:32.996 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:32.989+0000 7f7f5cc00740 4 client.16872 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-31T23:20:32.996 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:32.989+0000 7f7f5cc00740 1 -- 172.21.15.46:0/1775508051 --> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] -- command(tid 0: {"prefix": "flush journal"}) -- 0x556a228fee30 con 0x556a22a0e640 2024-03-31T23:20:32.996 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:32.990+0000 7f7f537fe640 1 --2- 172.21.15.46:0/1775508051 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x556a22a0e640 0x556a22a10a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:20:32.996 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:32.990+0000 7f7f32ffd640 10 client.16872 ms_handle_connect on v2:172.21.15.142:6838/2519143918 2024-03-31T23:20:32.996 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:32.990+0000 7f7f537fe640 1 --2- 172.21.15.46:0/1775508051 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x556a22a0e640 0x556a22a10a60 crc :-1 s=READY pgs=154 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b6d70c0f655833b8 server_cookie=a9209cf993c3c74b in_seq=0 out_seq=0 2024-03-31T23:20:33.015 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:33.013+0000 7f7f32ffd640 1 -- 172.21.15.46:0/1775508051 <== mds.0 v2:172.21.15.142:6838/2519143918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x556a228fee30 con 0x556a22a0e640 2024-03-31T23:20:33.015 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:33.013+0000 7f7f32ffd640 10 client.16872 handle_command_reply: tid=0 2024-03-31T23:20:33.015 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:33.013+0000 7f7f5cc00740 1 client.16872 shutdown 2024-03-31T23:20:33.015 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:33.013+0000 7f7f5cc00740 1 -- 172.21.15.46:0/1775508051 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556a2298efe0 msgr2=0x556a22991730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:20:33.015 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:33.013+0000 7f7f5cc00740 1 --2- 172.21.15.46:0/1775508051 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556a2298efe0 0x556a22991730 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7f7f48097a40 tx=0x7f7f4809f820 comp rx=0 tx=0).stop 2024-03-31T23:20:33.016 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:33.013+0000 7f7f5cc00740 1 -- 172.21.15.46:0/1775508051 shutdown_connections 2024-03-31T23:20:33.016 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:33.013+0000 7f7f5cc00740 1 --2- 172.21.15.46:0/1775508051 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x556a22a0e640 0x556a22a10a60 crc :-1 s=READY pgs=154 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:20:33.016 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:33.013+0000 7f7f5cc00740 1 --2- 172.21.15.46:0/1775508051 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556a22997db0 0x556a22991c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:20:33.016 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:33.013+0000 7f7f32ffd640 0 client.16872 ms_handle_reset on v2:172.21.15.142:6838/2519143918 2024-03-31T23:20:33.016 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:33.013+0000 7f7f5cc00740 1 --2- 172.21.15.46:0/1775508051 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556a2298efe0 0x556a22991730 unknown :-1 s=CLOSED pgs=782 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:20:33.016 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:33.013+0000 7f7f5cc00740 1 --2- 172.21.15.46:0/1775508051 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x556a2298e6c0 0x556a2299ff40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:20:33.016 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:33.013+0000 7f7f5cc00740 1 -- 172.21.15.46:0/1775508051 >> 172.21.15.46:0/1775508051 conn(0x556a22988fe0 msgr2=0x556a22993600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:20:33.016 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:33.014+0000 7f7f5cc00740 1 -- 172.21.15.46:0/1775508051 shutdown_connections 2024-03-31T23:20:33.016 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:33.014+0000 7f7f5cc00740 1 -- 172.21.15.46:0/1775508051 wait complete. 2024-03-31T23:20:33.016 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:33.014+0000 7f7f5cc00740 20 client.16872 trim_cache size 0 max 16384 2024-03-31T23:20:33.017 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout:{ 2024-03-31T23:20:33.017 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout: "message": "", 2024-03-31T23:20:33.017 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout: "return_code": 0 2024-03-31T23:20:33.017 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout:} 2024-03-31T23:20:33.018 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:33.014+0000 7f7f59f3e640 1 -- 172.21.15.46:0/3651011016 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f7f3c0a26b0 msgr2=0x7f7f3c0a4b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:20:33.018 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:33.014+0000 7f7f59f3e640 1 --2- 172.21.15.46:0/3651011016 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f7f3c0a26b0 0x7f7f3c0a4b70 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f7f4c005fd0 tx=0x7f7f4c009f00 comp rx=0 tx=0).stop 2024-03-31T23:20:33.018 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:33.014+0000 7f7f59f3e640 1 -- 172.21.15.46:0/3651011016 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7f54101260 msgr2=0x7f7f54198420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:20:33.018 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:33.014+0000 7f7f59f3e640 1 --2- 172.21.15.46:0/3651011016 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7f54101260 0x7f7f54198420 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7f7f4800e9f0 tx=0x7f7f4800eeb0 comp rx=0 tx=0).stop 2024-03-31T23:20:33.018 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:33.014+0000 7f7f59f3e640 1 -- 172.21.15.46:0/3651011016 shutdown_connections 2024-03-31T23:20:33.018 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:33.014+0000 7f7f59f3e640 1 --2- 172.21.15.46:0/3651011016 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f7f3c0a26b0 0x7f7f3c0a4b70 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:20:33.018 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:33.014+0000 7f7f59f3e640 1 --2- 172.21.15.46:0/3651011016 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7f541055f0 0x7f7f5419c7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:20:33.018 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:33.014+0000 7f7f59f3e640 1 --2- 172.21.15.46:0/3651011016 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7f54101260 0x7f7f54198420 unknown :-1 s=CLOSED pgs=779 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:20:33.018 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:33.014+0000 7f7f59f3e640 1 --2- 172.21.15.46:0/3651011016 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7f541008a0 0x7f7f54197ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:20:33.018 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:33.014+0000 7f7f59f3e640 1 -- 172.21.15.46:0/3651011016 >> 172.21.15.46:0/3651011016 conn(0x7f7f540774a0 msgr2=0x7f7f540690b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:20:33.019 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:33.014+0000 7f7f59f3e640 1 -- 172.21.15.46:0/3651011016 shutdown_connections 2024-03-31T23:20:33.019 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:33.014+0000 7f7f59f3e640 1 -- 172.21.15.46:0/3651011016 wait complete. 2024-03-31T23:20:33.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:20:32 smithi162 ceph-mon[36237]: pgmap v273: 129 pgs: 129 active+clean; 112 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 274 KiB/s rd, 733 KiB/s wr, 78 op/s 2024-03-31T23:20:33.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:20:32 smithi046 ceph-mon[32498]: pgmap v273: 129 pgs: 129 active+clean; 112 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 274 KiB/s rd, 733 KiB/s wr, 78 op/s 2024-03-31T23:20:33.207 INFO:tasks.workunit.client.0.smithi046.stdout: 1 191687 2.19 MB/sec execute 339 sec latency 2288.119 ms 2024-03-31T23:20:34.207 INFO:tasks.workunit.client.0.smithi046.stdout: 1 191687 2.19 MB/sec execute 340 sec latency 1230.276 ms 2024-03-31T23:20:34.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:20:34 smithi142 ceph-mon[35271]: pgmap v274: 129 pgs: 129 active+clean; 112 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 274 KiB/s rd, 733 KiB/s wr, 78 op/s 2024-03-31T23:20:35.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:20:34 smithi162 ceph-mon[36237]: pgmap v274: 129 pgs: 129 active+clean; 112 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 274 KiB/s rd, 733 KiB/s wr, 78 op/s 2024-03-31T23:20:35.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:20:34 smithi046 ceph-mon[32498]: pgmap v274: 129 pgs: 129 active+clean; 112 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 274 KiB/s rd, 733 KiB/s wr, 78 op/s 2024-03-31T23:20:35.207 INFO:tasks.workunit.client.0.smithi046.stdout: 1 191687 2.18 MB/sec execute 341 sec latency 2230.374 ms 2024-03-31T23:20:36.207 INFO:tasks.workunit.client.0.smithi046.stdout: 1 191687 2.17 MB/sec execute 342 sec latency 3230.481 ms 2024-03-31T23:20:36.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:20:36 smithi142 ceph-mon[35271]: pgmap v275: 129 pgs: 129 active+clean; 111 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 392 KiB/s rd, 1.0 MiB/s wr, 110 op/s 2024-03-31T23:20:37.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:20:36 smithi162 ceph-mon[36237]: pgmap v275: 129 pgs: 129 active+clean; 111 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 392 KiB/s rd, 1.0 MiB/s wr, 110 op/s 2024-03-31T23:20:37.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:20:36 smithi046 ceph-mon[32498]: pgmap v275: 129 pgs: 129 active+clean; 111 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 392 KiB/s rd, 1.0 MiB/s wr, 110 op/s 2024-03-31T23:20:37.207 INFO:tasks.workunit.client.0.smithi046.stdout: 1 191687 2.17 MB/sec execute 343 sec latency 4230.609 ms 2024-03-31T23:20:38.207 INFO:tasks.workunit.client.0.smithi046.stdout: 1 191825 2.16 MB/sec execute 344 sec latency 4767.080 ms 2024-03-31T23:20:38.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:20:38 smithi142 ceph-mon[35271]: pgmap v276: 129 pgs: 129 active+clean; 111 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 337 KiB/s rd, 859 KiB/s wr, 87 op/s 2024-03-31T23:20:39.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:20:38 smithi162 ceph-mon[36237]: pgmap v276: 129 pgs: 129 active+clean; 111 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 337 KiB/s rd, 859 KiB/s wr, 87 op/s 2024-03-31T23:20:39.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:20:38 smithi046 ceph-mon[32498]: pgmap v276: 129 pgs: 129 active+clean; 111 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 337 KiB/s rd, 859 KiB/s wr, 87 op/s 2024-03-31T23:20:39.207 INFO:tasks.workunit.client.0.smithi046.stdout: 1 192089 2.16 MB/sec execute 345 sec latency 16.096 ms 2024-03-31T23:20:40.207 INFO:tasks.workunit.client.0.smithi046.stdout: 1 192488 2.16 MB/sec execute 346 sec latency 15.767 ms 2024-03-31T23:20:40.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:20:40 smithi142 ceph-mon[35271]: pgmap v277: 129 pgs: 129 active+clean; 112 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 385 KiB/s rd, 1.0 MiB/s wr, 104 op/s 2024-03-31T23:20:41.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:20:40 smithi162 ceph-mon[36237]: pgmap v277: 129 pgs: 129 active+clean; 112 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 385 KiB/s rd, 1.0 MiB/s wr, 104 op/s 2024-03-31T23:20:41.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:20:40 smithi046 ceph-mon[32498]: pgmap v277: 129 pgs: 129 active+clean; 112 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 385 KiB/s rd, 1.0 MiB/s wr, 104 op/s 2024-03-31T23:20:41.207 INFO:tasks.workunit.client.0.smithi046.stdout: 1 193002 2.16 MB/sec execute 347 sec latency 21.160 ms 2024-03-31T23:20:42.207 INFO:tasks.workunit.client.0.smithi046.stdout: 1 193462 2.16 MB/sec execute 348 sec latency 20.021 ms 2024-03-31T23:20:42.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:20:42 smithi142 ceph-mon[35271]: pgmap v278: 129 pgs: 129 active+clean; 112 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 166 KiB/s rd, 515 KiB/s wr, 49 op/s 2024-03-31T23:20:43.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:20:42 smithi162 ceph-mon[36237]: pgmap v278: 129 pgs: 129 active+clean; 112 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 166 KiB/s rd, 515 KiB/s wr, 49 op/s 2024-03-31T23:20:43.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:20:42 smithi046 ceph-mon[32498]: pgmap v278: 129 pgs: 129 active+clean; 112 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 166 KiB/s rd, 515 KiB/s wr, 49 op/s 2024-03-31T23:20:43.207 INFO:tasks.workunit.client.0.smithi046.stdout: 1 193981 2.16 MB/sec execute 349 sec latency 22.415 ms 2024-03-31T23:20:44.207 INFO:tasks.workunit.client.0.smithi046.stdout: 1 194566 2.16 MB/sec execute 350 sec latency 16.558 ms 2024-03-31T23:20:44.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:20:44 smithi142 ceph-mon[35271]: pgmap v279: 129 pgs: 129 active+clean; 112 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 166 KiB/s rd, 515 KiB/s wr, 49 op/s 2024-03-31T23:20:45.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:20:44 smithi162 ceph-mon[36237]: pgmap v279: 129 pgs: 129 active+clean; 112 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 166 KiB/s rd, 515 KiB/s wr, 49 op/s 2024-03-31T23:20:45.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:20:44 smithi046 ceph-mon[32498]: pgmap v279: 129 pgs: 129 active+clean; 112 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 166 KiB/s rd, 515 KiB/s wr, 49 op/s 2024-03-31T23:20:45.207 INFO:tasks.workunit.client.0.smithi046.stdout: 1 195039 2.17 MB/sec execute 351 sec latency 16.759 ms 2024-03-31T23:20:46.130 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:46.107+0000 7fd288efb640 1 -- 172.21.15.46:0/3263291077 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd284101720 msgr2=0x7fd284101ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:20:46.130 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:46.107+0000 7fd288efb640 1 --2- 172.21.15.46:0/3263291077 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd284101720 0x7fd284101ba0 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7fd278009930 tx=0x7fd27802bc30 comp rx=0 tx=0).stop 2024-03-31T23:20:46.130 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:46.107+0000 7fd288efb640 1 -- 172.21.15.46:0/3263291077 shutdown_connections 2024-03-31T23:20:46.130 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:46.107+0000 7fd288efb640 1 --2- 172.21.15.46:0/3263291077 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd2841020e0 0x7fd28410e5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:20:46.131 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:46.107+0000 7fd288efb640 1 --2- 172.21.15.46:0/3263291077 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd284101720 0x7fd284101ba0 unknown :-1 s=CLOSED pgs=783 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:20:46.131 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:46.107+0000 7fd288efb640 1 --2- 172.21.15.46:0/3263291077 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd284107730 0x7fd284107b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:20:46.131 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:46.107+0000 7fd288efb640 1 -- 172.21.15.46:0/3263291077 >> 172.21.15.46:0/3263291077 conn(0x7fd2840fd230 msgr2=0x7fd2840ff650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:20:46.131 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:46.107+0000 7fd288efb640 1 -- 172.21.15.46:0/3263291077 shutdown_connections 2024-03-31T23:20:46.132 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:46.107+0000 7fd288efb640 1 -- 172.21.15.46:0/3263291077 wait complete. 2024-03-31T23:20:46.132 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:46.108+0000 7fd288efb640 1 Processor -- start 2024-03-31T23:20:46.132 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:46.108+0000 7fd288efb640 1 -- start start 2024-03-31T23:20:46.132 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:46.108+0000 7fd288efb640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd284101720 0x7fd284197eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:20:46.132 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:46.108+0000 7fd288efb640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd2841020e0 0x7fd2841983f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:20:46.133 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:46.108+0000 7fd288efb640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd284107730 0x7fd28419c770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:20:46.133 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:46.108+0000 7fd288efb640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fd2841119f0 con 0x7fd2841020e0 2024-03-31T23:20:46.133 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:46.108+0000 7fd288efb640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fd284111870 con 0x7fd284107730 2024-03-31T23:20:46.133 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:46.108+0000 7fd288efb640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fd284111b70 con 0x7fd284101720 2024-03-31T23:20:46.134 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:46.108+0000 7fd281d9b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd2841020e0 0x7fd2841983f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:20:46.134 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:46.108+0000 7fd281d9b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd2841020e0 0x7fd2841983f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:36168/0 (socket says 172.21.15.46:36168) 2024-03-31T23:20:46.134 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:46.108+0000 7fd281d9b640 1 -- 172.21.15.46:0/684364717 learned_addr learned my addr 172.21.15.46:0/684364717 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:20:46.134 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:46.108+0000 7fd28259c640 1 --2- 172.21.15.46:0/684364717 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd284101720 0x7fd284197eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:20:46.134 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:46.108+0000 7fd282d9d640 1 --2- 172.21.15.46:0/684364717 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd284107730 0x7fd28419c770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:20:46.134 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:46.109+0000 7fd281d9b640 1 -- 172.21.15.46:0/684364717 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd284101720 msgr2=0x7fd284197eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:20:46.134 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:46.109+0000 7fd281d9b640 1 --2- 172.21.15.46:0/684364717 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd284101720 0x7fd284197eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:20:46.134 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:46.109+0000 7fd281d9b640 1 -- 172.21.15.46:0/684364717 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd284107730 msgr2=0x7fd28419c770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:20:46.134 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:46.109+0000 7fd281d9b640 1 --2- 172.21.15.46:0/684364717 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd284107730 0x7fd28419c770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:20:46.134 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:46.109+0000 7fd281d9b640 1 -- 172.21.15.46:0/684364717 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd28419cea0 con 0x7fd2841020e0 2024-03-31T23:20:46.134 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:46.109+0000 7fd281d9b640 1 --2- 172.21.15.46:0/684364717 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd2841020e0 0x7fd2841983f0 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7fd278009230 tx=0x7fd27802c5d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:20:46.134 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:46.109+0000 7fd2737fe640 1 -- 172.21.15.46:0/684364717 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd278043020 con 0x7fd2841020e0 2024-03-31T23:20:46.135 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:46.109+0000 7fd288efb640 1 -- 172.21.15.46:0/684364717 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd28419d130 con 0x7fd2841020e0 2024-03-31T23:20:46.135 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:46.109+0000 7fd2737fe640 1 -- 172.21.15.46:0/684364717 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd27802ca70 con 0x7fd2841020e0 2024-03-31T23:20:46.135 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:46.109+0000 7fd2737fe640 1 -- 172.21.15.46:0/684364717 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd278003a50 con 0x7fd2841020e0 2024-03-31T23:20:46.135 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:46.109+0000 7fd288efb640 1 -- 172.21.15.46:0/684364717 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd2841a4a20 con 0x7fd2841020e0 2024-03-31T23:20:46.135 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:46.112+0000 7fd2737fe640 1 -- 172.21.15.46:0/684364717 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 25) ==== 137451+0+0 (secure 0 0 0) 0x7fd278046020 con 0x7fd2841020e0 2024-03-31T23:20:46.135 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:46.112+0000 7fd28bbba740 1 Processor -- start 2024-03-31T23:20:46.135 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:46.112+0000 7fd28bbba740 1 -- start start 2024-03-31T23:20:46.135 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:46.113+0000 7fd28bbba740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5626cffdb1e0 0x5626cffdb5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:20:46.135 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:46.113+0000 7fd28bbba740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5626cffdbb00 0x5626cffe8c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:20:46.135 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:46.113+0000 7fd28bbba740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5626cffe91c0 0x5626cffeb5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:20:46.135 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:46.113+0000 7fd28bbba740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x5626cff3e8f0 con 0x5626cffe91c0 2024-03-31T23:20:46.135 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:46.113+0000 7fd28bbba740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x5626cfee78d0 con 0x5626cffdb1e0 2024-03-31T23:20:46.135 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:46.113+0000 7fd28bbba740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5626cff3a990 con 0x5626cffdbb00 2024-03-31T23:20:46.136 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:46.113+0000 7fd2737fe640 1 --2- 172.21.15.46:0/684364717 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fd26c0a26b0 0x7fd26c0a4b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:20:46.136 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:46.113+0000 7fd281d9b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5626cffe91c0 0x5626cffeb5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:20:46.136 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:46.113+0000 7fd28259c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5626cffdbb00 0x5626cffe8c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:20:46.136 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:46.113+0000 7fd28259c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5626cffdbb00 0x5626cffe8c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.46:42184/0 (socket says 172.21.15.46:42184) 2024-03-31T23:20:46.136 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:46.113+0000 7fd28259c640 1 -- 172.21.15.46:0/3036603923 learned_addr learned my addr 172.21.15.46:0/3036603923 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:20:46.136 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:46.113+0000 7fd282d9d640 1 --2- 172.21.15.46:0/3036603923 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5626cffdb1e0 0x5626cffdb5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:20:46.136 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:46.113+0000 7fd282d9d640 1 --2- 172.21.15.46:0/684364717 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fd26c0a26b0 0x7fd26c0a4b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:20:46.136 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:46.113+0000 7fd2737fe640 1 -- 172.21.15.46:0/684364717 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 9554+0+0 (secure 0 0 0) 0x7fd2780f0e40 con 0x7fd2841020e0 2024-03-31T23:20:46.136 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:46.113+0000 7fd28259c640 1 -- 172.21.15.46:0/3036603923 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5626cffdb1e0 msgr2=0x5626cffdb5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:20:46.136 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:46.113+0000 7fd28259c640 1 --2- 172.21.15.46:0/3036603923 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5626cffdb1e0 0x5626cffdb5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:20:46.136 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:46.113+0000 7fd28259c640 1 -- 172.21.15.46:0/3036603923 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5626cffe91c0 msgr2=0x5626cffeb5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:20:46.136 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:46.113+0000 7fd28259c640 1 --2- 172.21.15.46:0/3036603923 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5626cffe91c0 0x5626cffeb5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:20:46.136 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:46.113+0000 7fd28259c640 1 -- 172.21.15.46:0/3036603923 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5626cfc53210 con 0x5626cffdbb00 2024-03-31T23:20:46.137 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:46.113+0000 7fd282d9d640 1 --2- 172.21.15.46:0/3036603923 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5626cffdb1e0 0x5626cffdb5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:20:46.137 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:46.113+0000 7fd28259c640 1 --2- 172.21.15.46:0/3036603923 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5626cffdbb00 0x5626cffe8c80 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fd274009c70 tx=0x7fd274011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=781a18ca923c8d79 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:20:46.137 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:46.114+0000 7fd282d9d640 1 --2- 172.21.15.46:0/684364717 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fd26c0a26b0 0x7fd26c0a4b70 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fd27c005c40 tx=0x7fd27c009500 comp rx=0 tx=0).ready entity=mgr.15654 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:20:46.137 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:46.114+0000 7fd270ff9640 1 -- 172.21.15.46:0/3036603923 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd274004230 con 0x5626cffdbb00 2024-03-31T23:20:46.137 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:46.114+0000 7fd270ff9640 1 -- 172.21.15.46:0/3036603923 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd2740043d0 con 0x5626cffdbb00 2024-03-31T23:20:46.137 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:46.114+0000 7fd270ff9640 1 -- 172.21.15.46:0/3036603923 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2740196d0 con 0x5626cffdbb00 2024-03-31T23:20:46.137 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:46.114+0000 7fd28bbba740 1 -- 172.21.15.46:0/3036603923 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5626cffdbb00 msgr2=0x5626cffe8c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:20:46.137 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:46.114+0000 7fd28bbba740 1 --2- 172.21.15.46:0/3036603923 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5626cffdbb00 0x5626cffe8c80 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fd274009c70 tx=0x7fd274011810 comp rx=0 tx=0).stop 2024-03-31T23:20:46.137 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:46.114+0000 7fd28bbba740 1 -- 172.21.15.46:0/3036603923 shutdown_connections 2024-03-31T23:20:46.137 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:46.114+0000 7fd28bbba740 1 --2- 172.21.15.46:0/3036603923 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5626cffe91c0 0x5626cffeb5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:20:46.137 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:46.114+0000 7fd28bbba740 1 --2- 172.21.15.46:0/3036603923 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5626cffdbb00 0x5626cffe8c80 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:20:46.137 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:46.114+0000 7fd28bbba740 1 --2- 172.21.15.46:0/3036603923 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5626cffdb1e0 0x5626cffdb5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:20:46.137 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:46.114+0000 7fd28bbba740 1 -- 172.21.15.46:0/3036603923 >> 172.21.15.46:0/3036603923 conn(0x5626cffd5fe0 msgr2=0x5626cffd63c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:20:46.137 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:46.114+0000 7fd28bbba740 1 -- 172.21.15.46:0/3036603923 shutdown_connections 2024-03-31T23:20:46.138 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:46.114+0000 7fd28bbba740 1 -- 172.21.15.46:0/3036603923 wait complete. 2024-03-31T23:20:46.138 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:46.115+0000 7fd28bbba740 1 -- start start 2024-03-31T23:20:46.138 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:46.115+0000 7fd28bbba740 1 Processor -- start 2024-03-31T23:20:46.138 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:46.115+0000 7fd28bbba740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5626cffdb1e0 0x5626cffe6180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:20:46.138 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:46.115+0000 7fd28bbba740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5626cffdbb00 0x5626cffe66c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:20:46.138 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:46.115+0000 7fd28bbba740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5626cffe91c0 0x5626cffe6c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:20:46.138 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:46.115+0000 7fd28bbba740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x5626cffebd80 con 0x5626cffdbb00 2024-03-31T23:20:46.138 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:46.115+0000 7fd28bbba740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x5626cfedcf80 con 0x5626cffe91c0 2024-03-31T23:20:46.138 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:46.115+0000 7fd28bbba740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5626cffebf00 con 0x5626cffdb1e0 2024-03-31T23:20:46.138 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:46.115+0000 7fd28bbba740 10 client.0 _ll_register_callbacks cb 0x5626cff35290 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T23:20:46.139 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:46.115+0000 7fd281d9b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5626cffdbb00 0x5626cffe66c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:20:46.139 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:46.115+0000 7fd281d9b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5626cffdbb00 0x5626cffe66c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:36190/0 (socket says 172.21.15.46:36190) 2024-03-31T23:20:46.139 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:46.115+0000 7fd281d9b640 1 -- 172.21.15.46:0/179282547 learned_addr learned my addr 172.21.15.46:0/179282547 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:20:46.139 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:46.115+0000 7fd28259c640 1 --2- 172.21.15.46:0/179282547 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5626cffdb1e0 0x5626cffe6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:20:46.139 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:46.115+0000 7fd282d9d640 1 --2- 172.21.15.46:0/179282547 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5626cffe91c0 0x5626cffe6c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:20:46.139 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:46.115+0000 7fd281d9b640 1 -- 172.21.15.46:0/179282547 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5626cffdb1e0 msgr2=0x5626cffe6180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:20:46.139 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:46.115+0000 7fd281d9b640 1 --2- 172.21.15.46:0/179282547 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5626cffdb1e0 0x5626cffe6180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:20:46.139 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:46.116+0000 7fd281d9b640 1 -- 172.21.15.46:0/179282547 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5626cffe91c0 msgr2=0x5626cffe6c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:20:46.139 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:46.116+0000 7fd281d9b640 1 --2- 172.21.15.46:0/179282547 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5626cffe91c0 0x5626cffe6c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:20:46.139 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:46.116+0000 7fd281d9b640 1 -- 172.21.15.46:0/179282547 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5626cfd1f620 con 0x5626cffdbb00 2024-03-31T23:20:46.139 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:46.116+0000 7fd28259c640 1 --2- 172.21.15.46:0/179282547 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5626cffdb1e0 0x5626cffe6180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:20:46.140 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:46.116+0000 7fd281d9b640 1 --2- 172.21.15.46:0/179282547 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5626cffdbb00 0x5626cffe66c0 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7fd2780b9c30 tx=0x7fd2780bcc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aff2b858908154c0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:20:46.140 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:46.116+0000 7fd269ffb640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-31T23:20:46.140 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:46.116+0000 7fd269ffb640 1 -- 172.21.15.46:0/179282547 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2780c13b0 con 0x5626cffdbb00 2024-03-31T23:20:46.140 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:46.116+0000 7fd269ffb640 1 -- 172.21.15.46:0/179282547 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 9554+0+0 (secure 0 0 0) 0x7fd2780053d0 con 0x5626cffdbb00 2024-03-31T23:20:46.140 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:46.116+0000 7fd28bbba740 1 -- 172.21.15.46:0/179282547 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd2740043d0 con 0x5626cffdbb00 2024-03-31T23:20:46.140 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:46.116+0000 7fd269ffb640 1 -- 172.21.15.46:0/179282547 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7fd27809f050 con 0x5626cffdbb00 2024-03-31T23:20:46.141 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:46.116+0000 7fd28bbba740 10 client.16884 fetch_fsmap learned FSMap version 35 2024-03-31T23:20:46.141 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:46.116+0000 7fd28bbba740 1 -- 172.21.15.46:0/179282547 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x5626cfff7c70 con 0x5626cffdbb00 2024-03-31T23:20:46.141 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:46.116+0000 7fd269ffb640 1 -- 172.21.15.46:0/179282547 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd2740043d0 con 0x5626cffdbb00 2024-03-31T23:20:46.141 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:46.116+0000 7fd269ffb640 1 -- 172.21.15.46:0/179282547 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 35) ==== 8366+0+0 (secure 0 0 0) 0x7fd2780ee430 con 0x5626cffdbb00 2024-03-31T23:20:46.141 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:46.116+0000 7fd28bbba740 10 client.16884 fetch_fsmap finished waiting for FSMap version 35 2024-03-31T23:20:46.141 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:46.116+0000 7fd28bbba740 10 client.16884 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-31T23:20:46.141 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:46.116+0000 7fd28bbba740 20 client.16884 populate_metadata read hostname 'smithi046' 2024-03-31T23:20:46.141 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:46.116+0000 7fd28bbba740 1 --2- 172.21.15.46:0/179282547 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x5626cfff9130 0x5626cfffb5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:20:46.141 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:46.116+0000 7fd28bbba740 4 client.16884 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-31T23:20:46.142 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:46.116+0000 7fd28bbba740 1 -- 172.21.15.46:0/179282547 --> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5626cff52120 con 0x5626cfff9130 2024-03-31T23:20:46.142 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:46.117+0000 7fd28259c640 1 --2- 172.21.15.46:0/179282547 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x5626cfff9130 0x5626cfffb5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:20:46.142 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:46.117+0000 7fd269ffb640 10 client.16884 ms_handle_connect on v2:172.21.15.142:6838/2519143918 2024-03-31T23:20:46.142 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:46.117+0000 7fd28259c640 1 --2- 172.21.15.46:0/179282547 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x5626cfff9130 0x5626cfffb5f0 crc :-1 s=READY pgs=155 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=689c16131c6bce0a server_cookie=f210ba2c882174a in_seq=0 out_seq=0 2024-03-31T23:20:46.142 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:46.118+0000 7fd269ffb640 1 -- 172.21.15.46:0/179282547 <== mds.0 v2:172.21.15.142:6838/2519143918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5626cff52120 con 0x5626cfff9130 2024-03-31T23:20:46.142 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:46.118+0000 7fd269ffb640 10 client.16884 handle_command_reply: tid=0 2024-03-31T23:20:46.142 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:46.118+0000 7fd28bbba740 1 client.16884 shutdown 2024-03-31T23:20:46.142 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:46.118+0000 7fd28bbba740 1 -- 172.21.15.46:0/179282547 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5626cffdbb00 msgr2=0x5626cffe66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:20:46.142 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:46.118+0000 7fd28bbba740 1 --2- 172.21.15.46:0/179282547 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5626cffdbb00 0x5626cffe66c0 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7fd2780b9c30 tx=0x7fd2780bcc80 comp rx=0 tx=0).stop 2024-03-31T23:20:46.143 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:46.118+0000 7fd28bbba740 1 -- 172.21.15.46:0/179282547 shutdown_connections 2024-03-31T23:20:46.143 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:46.118+0000 7fd28bbba740 1 --2- 172.21.15.46:0/179282547 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x5626cfff9130 0x5626cfffb5f0 crc :-1 s=READY pgs=155 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:20:46.143 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:46.118+0000 7fd28bbba740 1 --2- 172.21.15.46:0/179282547 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5626cffe91c0 0x5626cffe6c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:20:46.143 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:46.118+0000 7fd28bbba740 1 --2- 172.21.15.46:0/179282547 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5626cffdbb00 0x5626cffe66c0 unknown :-1 s=CLOSED pgs=785 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:20:46.143 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:46.118+0000 7fd269ffb640 0 client.16884 ms_handle_reset on v2:172.21.15.142:6838/2519143918 2024-03-31T23:20:46.143 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:46.118+0000 7fd28bbba740 1 --2- 172.21.15.46:0/179282547 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5626cffdb1e0 0x5626cffe6180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:20:46.143 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:46.118+0000 7fd28bbba740 1 -- 172.21.15.46:0/179282547 >> 172.21.15.46:0/179282547 conn(0x5626cffd5fe0 msgr2=0x5626cffd7d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:20:46.143 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:46.118+0000 7fd28bbba740 1 -- 172.21.15.46:0/179282547 shutdown_connections 2024-03-31T23:20:46.143 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:46.118+0000 7fd28bbba740 1 -- 172.21.15.46:0/179282547 wait complete. 2024-03-31T23:20:46.143 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:46.118+0000 7fd28bbba740 20 client.16884 trim_cache size 0 max 16384 2024-03-31T23:20:46.151 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:46.133+0000 7fd28bbba740 1 Processor -- start 2024-03-31T23:20:46.151 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:46.133+0000 7fd28bbba740 1 -- start start 2024-03-31T23:20:46.151 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:46.133+0000 7fd28bbba740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5626cffdb1e0 0x5626cffe88b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:20:46.152 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:46.133+0000 7fd28bbba740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5626cffdbb00 0x5626cffd0e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:20:46.152 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:46.133+0000 7fd28bbba740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5626cffe91c0 0x5626cfff4b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:20:46.152 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:46.133+0000 7fd28bbba740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fd2841af3c0 con 0x5626cffdbb00 2024-03-31T23:20:46.152 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:46.133+0000 7fd28bbba740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fd2841acc00 con 0x5626cffe91c0 2024-03-31T23:20:46.152 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:46.133+0000 7fd28bbba740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fd2841aaa70 con 0x5626cffdb1e0 2024-03-31T23:20:46.152 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:46.133+0000 7fd281d9b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5626cffdbb00 0x5626cffd0e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:20:46.152 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:46.133+0000 7fd281d9b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5626cffdbb00 0x5626cffd0e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:36204/0 (socket says 172.21.15.46:36204) 2024-03-31T23:20:46.152 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:46.133+0000 7fd281d9b640 1 -- 172.21.15.46:0/1861894092 learned_addr learned my addr 172.21.15.46:0/1861894092 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:20:46.152 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:46.133+0000 7fd282d9d640 1 --2- 172.21.15.46:0/1861894092 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5626cffe91c0 0x5626cfff4b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:20:46.152 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:46.133+0000 7fd28259c640 1 --2- 172.21.15.46:0/1861894092 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5626cffdb1e0 0x5626cffe88b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:20:46.152 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:46.133+0000 7fd281d9b640 1 -- 172.21.15.46:0/1861894092 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5626cffdb1e0 msgr2=0x5626cffe88b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:20:46.152 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:46.133+0000 7fd281d9b640 1 --2- 172.21.15.46:0/1861894092 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5626cffdb1e0 0x5626cffe88b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:20:46.152 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:46.133+0000 7fd281d9b640 1 -- 172.21.15.46:0/1861894092 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5626cffe91c0 msgr2=0x5626cfff4b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:20:46.152 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:46.133+0000 7fd281d9b640 1 --2- 172.21.15.46:0/1861894092 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5626cffe91c0 0x5626cfff4b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:20:46.153 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:46.133+0000 7fd281d9b640 1 -- 172.21.15.46:0/1861894092 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5626cffe5130 con 0x5626cffdbb00 2024-03-31T23:20:46.153 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:46.133+0000 7fd28259c640 1 --2- 172.21.15.46:0/1861894092 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5626cffdb1e0 0x5626cffe88b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T23:20:46.153 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:46.134+0000 7fd281d9b640 1 --2- 172.21.15.46:0/1861894092 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5626cffdbb00 0x5626cffd0e20 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7fd278005ec0 tx=0x7fd2780b9630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d5051b1d3077c3d server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:20:46.153 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:46.134+0000 7fd26bfff640 1 -- 172.21.15.46:0/1861894092 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2780c13b0 con 0x5626cffdbb00 2024-03-31T23:20:46.153 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:46.134+0000 7fd26bfff640 1 -- 172.21.15.46:0/1861894092 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd2780c5c70 con 0x5626cffdbb00 2024-03-31T23:20:46.153 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:46.134+0000 7fd26bfff640 1 -- 172.21.15.46:0/1861894092 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2780ee310 con 0x5626cffdbb00 2024-03-31T23:20:46.153 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:46.134+0000 7fd28bbba740 1 -- 172.21.15.46:0/1861894092 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5626cffdbb00 msgr2=0x5626cffd0e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:20:46.153 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:46.134+0000 7fd28bbba740 1 --2- 172.21.15.46:0/1861894092 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5626cffdbb00 0x5626cffd0e20 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7fd278005ec0 tx=0x7fd2780b9630 comp rx=0 tx=0).stop 2024-03-31T23:20:46.153 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:46.134+0000 7fd28bbba740 1 -- 172.21.15.46:0/1861894092 shutdown_connections 2024-03-31T23:20:46.153 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:46.134+0000 7fd28bbba740 1 --2- 172.21.15.46:0/1861894092 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5626cffe91c0 0x5626cfff4b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:20:46.153 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:46.134+0000 7fd28bbba740 1 --2- 172.21.15.46:0/1861894092 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5626cffdbb00 0x5626cffd0e20 unknown :-1 s=CLOSED pgs=786 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:20:46.153 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:46.134+0000 7fd28bbba740 1 --2- 172.21.15.46:0/1861894092 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5626cffdb1e0 0x5626cffe88b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:20:46.153 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:46.134+0000 7fd28bbba740 1 -- 172.21.15.46:0/1861894092 >> 172.21.15.46:0/1861894092 conn(0x5626cffd5fe0 msgr2=0x5626cffd4f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:20:46.154 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:46.134+0000 7fd28bbba740 1 -- 172.21.15.46:0/1861894092 shutdown_connections 2024-03-31T23:20:46.154 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:46.134+0000 7fd28bbba740 1 -- 172.21.15.46:0/1861894092 wait complete. 2024-03-31T23:20:46.154 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:46.135+0000 7fd28bbba740 1 -- start start 2024-03-31T23:20:46.154 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:46.135+0000 7fd28bbba740 1 Processor -- start 2024-03-31T23:20:46.154 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:46.135+0000 7fd28bbba740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5626cffdb1e0 0x5626cffe3000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:20:46.154 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:46.135+0000 7fd28bbba740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5626cffdbb00 0x5626cffe3540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:20:46.154 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:46.135+0000 7fd28bbba740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5626cffe91c0 0x5626cffe3a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:20:46.154 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:46.135+0000 7fd28bbba740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fd2841ab0c0 con 0x5626cffdb1e0 2024-03-31T23:20:46.154 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:46.135+0000 7fd28bbba740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fd2841ac530 con 0x5626cffdbb00 2024-03-31T23:20:46.154 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:46.135+0000 7fd28bbba740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fd2841ab790 con 0x5626cffe91c0 2024-03-31T23:20:46.154 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:46.135+0000 7fd28bbba740 10 client.0 _ll_register_callbacks cb 0x7fd278009880 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T23:20:46.154 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:46.135+0000 7fd28259c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5626cffdb1e0 0x5626cffe3000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:20:46.154 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:46.135+0000 7fd28259c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5626cffdb1e0 0x5626cffe3000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:36206/0 (socket says 172.21.15.46:36206) 2024-03-31T23:20:46.155 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:46.135+0000 7fd28259c640 1 -- 172.21.15.46:0/1467471716 learned_addr learned my addr 172.21.15.46:0/1467471716 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:20:46.155 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:46.135+0000 7fd281d9b640 1 --2- 172.21.15.46:0/1467471716 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5626cffdbb00 0x5626cffe3540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:20:46.155 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:46.135+0000 7fd282d9d640 1 --2- 172.21.15.46:0/1467471716 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5626cffe91c0 0x5626cffe3a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:20:46.155 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:46.135+0000 7fd28259c640 1 -- 172.21.15.46:0/1467471716 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5626cffe91c0 msgr2=0x5626cffe3a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:20:46.155 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:46.135+0000 7fd28259c640 1 --2- 172.21.15.46:0/1467471716 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5626cffe91c0 0x5626cffe3a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:20:46.155 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:46.135+0000 7fd28259c640 1 -- 172.21.15.46:0/1467471716 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5626cffdbb00 msgr2=0x5626cffe3540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:20:46.155 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:46.135+0000 7fd28259c640 1 --2- 172.21.15.46:0/1467471716 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5626cffdbb00 0x5626cffe3540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:20:46.155 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:46.135+0000 7fd28259c640 1 -- 172.21.15.46:0/1467471716 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5626d005e7d0 con 0x5626cffdb1e0 2024-03-31T23:20:46.155 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:46.135+0000 7fd28259c640 1 --2- 172.21.15.46:0/1467471716 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5626cffdb1e0 0x5626cffe3000 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7fd274009c70 tx=0x7fd274005f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e43915db2ecb7c server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:20:46.155 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:46.135+0000 7fd269ffb640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-31T23:20:46.155 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:46.136+0000 7fd269ffb640 1 -- 172.21.15.46:0/1467471716 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2740053d0 con 0x5626cffdb1e0 2024-03-31T23:20:46.155 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:46.136+0000 7fd269ffb640 1 -- 172.21.15.46:0/1467471716 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 9554+0+0 (secure 0 0 0) 0x7fd27401f040 con 0x5626cffdb1e0 2024-03-31T23:20:46.155 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:46.136+0000 7fd28bbba740 1 -- 172.21.15.46:0/1467471716 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd2780c5c70 con 0x5626cffdb1e0 2024-03-31T23:20:46.156 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:46.136+0000 7fd269ffb640 1 -- 172.21.15.46:0/1467471716 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7fd27403a050 con 0x5626cffdb1e0 2024-03-31T23:20:46.156 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:46.136+0000 7fd28bbba740 10 client.16890 fetch_fsmap learned FSMap version 35 2024-03-31T23:20:46.156 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:46.136+0000 7fd28bbba740 1 -- 172.21.15.46:0/1467471716 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x5626d005fac0 con 0x5626cffdb1e0 2024-03-31T23:20:46.156 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:46.136+0000 7fd269ffb640 1 -- 172.21.15.46:0/1467471716 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd2780c5c70 con 0x5626cffdb1e0 2024-03-31T23:20:46.156 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:46.136+0000 7fd269ffb640 1 -- 172.21.15.46:0/1467471716 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 35) ==== 8366+0+0 (secure 0 0 0) 0x7fd274018070 con 0x5626cffdb1e0 2024-03-31T23:20:46.156 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:46.136+0000 7fd28bbba740 10 client.16890 fetch_fsmap finished waiting for FSMap version 35 2024-03-31T23:20:46.156 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:46.136+0000 7fd28bbba740 10 client.16890 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-31T23:20:46.156 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:46.136+0000 7fd28bbba740 20 client.16890 populate_metadata read hostname 'smithi046' 2024-03-31T23:20:46.156 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:46.136+0000 7fd28bbba740 1 --2- 172.21.15.46:0/1467471716 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x5626d0060120 0x5626d0062540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:20:46.156 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:46.136+0000 7fd28bbba740 4 client.16890 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-31T23:20:46.156 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:46.136+0000 7fd28bbba740 1 -- 172.21.15.46:0/1467471716 --> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5626cff4be30 con 0x5626d0060120 2024-03-31T23:20:46.156 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:46.136+0000 7fd281d9b640 1 --2- 172.21.15.46:0/1467471716 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x5626d0060120 0x5626d0062540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:20:46.156 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:46.137+0000 7fd269ffb640 10 client.16890 ms_handle_connect on v2:172.21.15.142:6838/2519143918 2024-03-31T23:20:46.156 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:46.137+0000 7fd281d9b640 1 --2- 172.21.15.46:0/1467471716 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x5626d0060120 0x5626d0062540 crc :-1 s=READY pgs=156 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8ffed1d3850a40e2 server_cookie=f423b5f758740c9b in_seq=0 out_seq=0 2024-03-31T23:20:46.165 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:46.162+0000 7fd269ffb640 1 -- 172.21.15.46:0/1467471716 <== mds.0 v2:172.21.15.142:6838/2519143918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5626cff4be30 con 0x5626d0060120 2024-03-31T23:20:46.165 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:46.162+0000 7fd269ffb640 10 client.16890 handle_command_reply: tid=0 2024-03-31T23:20:46.165 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:46.162+0000 7fd28bbba740 1 client.16890 shutdown 2024-03-31T23:20:46.165 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:46.162+0000 7fd28bbba740 1 -- 172.21.15.46:0/1467471716 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5626cffdb1e0 msgr2=0x5626cffe3000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:20:46.165 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:46.162+0000 7fd28bbba740 1 --2- 172.21.15.46:0/1467471716 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5626cffdb1e0 0x5626cffe3000 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7fd274009c70 tx=0x7fd274005f10 comp rx=0 tx=0).stop 2024-03-31T23:20:46.165 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:46.163+0000 7fd28bbba740 1 -- 172.21.15.46:0/1467471716 shutdown_connections 2024-03-31T23:20:46.166 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:46.163+0000 7fd28bbba740 1 --2- 172.21.15.46:0/1467471716 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x5626d0060120 0x5626d0062540 crc :-1 s=READY pgs=156 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:20:46.166 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:46.163+0000 7fd28bbba740 1 --2- 172.21.15.46:0/1467471716 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5626cffe91c0 0x5626cffe3a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:20:46.166 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:46.163+0000 7fd28bbba740 1 --2- 172.21.15.46:0/1467471716 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5626cffdbb00 0x5626cffe3540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:20:46.166 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:46.163+0000 7fd269ffb640 0 client.16890 ms_handle_reset on v2:172.21.15.142:6838/2519143918 2024-03-31T23:20:46.166 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:46.163+0000 7fd28bbba740 1 --2- 172.21.15.46:0/1467471716 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5626cffdb1e0 0x5626cffe3000 unknown :-1 s=CLOSED pgs=787 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:20:46.167 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:46.163+0000 7fd28bbba740 1 -- 172.21.15.46:0/1467471716 >> 172.21.15.46:0/1467471716 conn(0x5626cffd5fe0 msgr2=0x5626cffd7c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:20:46.167 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:46.163+0000 7fd28bbba740 1 -- 172.21.15.46:0/1467471716 shutdown_connections 2024-03-31T23:20:46.167 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:46.163+0000 7fd28bbba740 1 -- 172.21.15.46:0/1467471716 wait complete. 2024-03-31T23:20:46.167 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:46.163+0000 7fd28bbba740 20 client.16890 trim_cache size 0 max 16384 2024-03-31T23:20:46.168 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout:{ 2024-03-31T23:20:46.168 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout: "message": "", 2024-03-31T23:20:46.168 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout: "return_code": 0 2024-03-31T23:20:46.168 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout:} 2024-03-31T23:20:46.169 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:46.163+0000 7fd288efb640 1 -- 172.21.15.46:0/684364717 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fd26c0a26b0 msgr2=0x7fd26c0a4b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:20:46.169 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:46.163+0000 7fd288efb640 1 --2- 172.21.15.46:0/684364717 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fd26c0a26b0 0x7fd26c0a4b70 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fd27c005c40 tx=0x7fd27c009500 comp rx=0 tx=0).stop 2024-03-31T23:20:46.169 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:46.163+0000 7fd288efb640 1 -- 172.21.15.46:0/684364717 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd2841020e0 msgr2=0x7fd2841983f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:20:46.169 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:46.163+0000 7fd288efb640 1 --2- 172.21.15.46:0/684364717 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd2841020e0 0x7fd2841983f0 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7fd278009230 tx=0x7fd27802c5d0 comp rx=0 tx=0).stop 2024-03-31T23:20:46.169 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:46.163+0000 7fd288efb640 1 -- 172.21.15.46:0/684364717 shutdown_connections 2024-03-31T23:20:46.169 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:46.163+0000 7fd288efb640 1 --2- 172.21.15.46:0/684364717 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fd26c0a26b0 0x7fd26c0a4b70 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:20:46.169 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:46.163+0000 7fd288efb640 1 --2- 172.21.15.46:0/684364717 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd284107730 0x7fd28419c770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:20:46.170 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:46.163+0000 7fd288efb640 1 --2- 172.21.15.46:0/684364717 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd2841020e0 0x7fd2841983f0 unknown :-1 s=CLOSED pgs=784 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:20:46.170 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:46.163+0000 7fd288efb640 1 --2- 172.21.15.46:0/684364717 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd284101720 0x7fd284197eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:20:46.170 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:46.163+0000 7fd288efb640 1 -- 172.21.15.46:0/684364717 >> 172.21.15.46:0/684364717 conn(0x7fd2840fd230 msgr2=0x7fd2840fd870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:20:46.170 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:46.163+0000 7fd288efb640 1 -- 172.21.15.46:0/684364717 shutdown_connections 2024-03-31T23:20:46.170 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:46.163+0000 7fd288efb640 1 -- 172.21.15.46:0/684364717 wait complete. 2024-03-31T23:20:46.211 INFO:tasks.workunit.client.0.smithi046.stdout: 1 195165 2.17 MB/sec execute 352 sec latency 733.979 ms 2024-03-31T23:20:46.944 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:20:46 smithi142 ceph-mon[35271]: pgmap v280: 129 pgs: 129 active+clean; 112 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 379 KiB/s rd, 1.0 MiB/s wr, 103 op/s 2024-03-31T23:20:47.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:20:46 smithi162 ceph-mon[36237]: pgmap v280: 129 pgs: 129 active+clean; 112 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 379 KiB/s rd, 1.0 MiB/s wr, 103 op/s 2024-03-31T23:20:47.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:20:46 smithi046 ceph-mon[32498]: pgmap v280: 129 pgs: 129 active+clean; 112 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 379 KiB/s rd, 1.0 MiB/s wr, 103 op/s 2024-03-31T23:20:47.207 INFO:tasks.workunit.client.0.smithi046.stdout: 1 195165 2.16 MB/sec execute 353 sec latency 1734.058 ms 2024-03-31T23:20:48.207 INFO:tasks.workunit.client.0.smithi046.stdout: 1 195232 2.15 MB/sec execute 354 sec latency 2271.722 ms 2024-03-31T23:20:48.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:20:48 smithi142 ceph-mon[35271]: pgmap v281: 129 pgs: 129 active+clean; 112 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 261 KiB/s rd, 706 KiB/s wr, 71 op/s 2024-03-31T23:20:49.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:20:48 smithi162 ceph-mon[36237]: pgmap v281: 129 pgs: 129 active+clean; 112 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 261 KiB/s rd, 706 KiB/s wr, 71 op/s 2024-03-31T23:20:49.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:20:48 smithi046 ceph-mon[32498]: pgmap v281: 129 pgs: 129 active+clean; 112 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 261 KiB/s rd, 706 KiB/s wr, 71 op/s 2024-03-31T23:20:49.208 INFO:tasks.workunit.client.0.smithi046.stdout: 1 195232 2.15 MB/sec execute 355 sec latency 1227.632 ms 2024-03-31T23:20:50.208 INFO:tasks.workunit.client.0.smithi046.stdout: 1 195232 2.14 MB/sec execute 356 sec latency 2227.737 ms 2024-03-31T23:20:50.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:20:50 smithi142 ceph-mon[35271]: pgmap v282: 129 pgs: 129 active+clean; 115 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 373 KiB/s rd, 1.0 MiB/s wr, 102 op/s 2024-03-31T23:20:51.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:20:50 smithi162 ceph-mon[36237]: pgmap v282: 129 pgs: 129 active+clean; 115 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 373 KiB/s rd, 1.0 MiB/s wr, 102 op/s 2024-03-31T23:20:51.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:20:50 smithi046 ceph-mon[32498]: pgmap v282: 129 pgs: 129 active+clean; 115 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 373 KiB/s rd, 1.0 MiB/s wr, 102 op/s 2024-03-31T23:20:51.208 INFO:tasks.workunit.client.0.smithi046.stdout: 1 195232 2.14 MB/sec execute 357 sec latency 3227.840 ms 2024-03-31T23:20:52.208 INFO:tasks.workunit.client.0.smithi046.stdout: 1 195232 2.13 MB/sec execute 358 sec latency 4227.959 ms 2024-03-31T23:20:52.945 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:20:52 smithi142 ceph-mon[35271]: pgmap v283: 129 pgs: 129 active+clean; 115 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 326 KiB/s rd, 849 KiB/s wr, 85 op/s 2024-03-31T23:20:53.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:20:52 smithi162 ceph-mon[36237]: pgmap v283: 129 pgs: 129 active+clean; 115 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 326 KiB/s rd, 849 KiB/s wr, 85 op/s 2024-03-31T23:20:53.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:20:52 smithi046 ceph-mon[32498]: pgmap v283: 129 pgs: 129 active+clean; 115 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 326 KiB/s rd, 849 KiB/s wr, 85 op/s 2024-03-31T23:20:53.208 INFO:tasks.workunit.client.0.smithi046.stdout: 1 195252 2.12 MB/sec execute 359 sec latency 4761.513 ms 2024-03-31T23:20:54.208 INFO:tasks.workunit.client.0.smithi046.stdout: 1 195252 2.12 MB/sec execute 360 sec latency 1383.133 ms 2024-03-31T23:20:54.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:20:54 smithi142 ceph-mon[35271]: pgmap v284: 129 pgs: 129 active+clean; 115 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 326 KiB/s rd, 849 KiB/s wr, 85 op/s 2024-03-31T23:20:55.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:20:54 smithi162 ceph-mon[36237]: pgmap v284: 129 pgs: 129 active+clean; 115 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 326 KiB/s rd, 849 KiB/s wr, 85 op/s 2024-03-31T23:20:55.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:20:54 smithi046 ceph-mon[32498]: pgmap v284: 129 pgs: 129 active+clean; 115 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 326 KiB/s rd, 849 KiB/s wr, 85 op/s 2024-03-31T23:20:55.209 INFO:tasks.workunit.client.0.smithi046.stdout: 1 195252 2.11 MB/sec execute 361 sec latency 2383.237 ms 2024-03-31T23:20:56.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:20:55 smithi162 ceph-mon[36237]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-31T23:20:56.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:20:55 smithi046 ceph-mon[32498]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-31T23:20:56.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:20:55 smithi142 ceph-mon[35271]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-31T23:20:56.208 INFO:tasks.workunit.client.0.smithi046.stdout: 1 195397 2.11 MB/sec execute 362 sec latency 2812.301 ms 2024-03-31T23:20:57.047 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:20:56 smithi162 ceph-mon[36237]: pgmap v285: 129 pgs: 129 active+clean; 115 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 352 KiB/s rd, 878 KiB/s wr, 92 op/s 2024-03-31T23:20:57.047 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:20:56 smithi162 ceph-mon[36237]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:20:57.047 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:20:56 smithi162 ceph-mon[36237]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-31T23:20:57.047 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:20:56 smithi162 ceph-mon[36237]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' 2024-03-31T23:20:57.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:20:56 smithi046 ceph-mon[32498]: pgmap v285: 129 pgs: 129 active+clean; 115 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 352 KiB/s rd, 878 KiB/s wr, 92 op/s 2024-03-31T23:20:57.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:20:56 smithi046 ceph-mon[32498]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:20:57.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:20:56 smithi046 ceph-mon[32498]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-31T23:20:57.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:20:56 smithi046 ceph-mon[32498]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' 2024-03-31T23:20:57.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:20:56 smithi142 ceph-mon[35271]: pgmap v285: 129 pgs: 129 active+clean; 115 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 352 KiB/s rd, 878 KiB/s wr, 92 op/s 2024-03-31T23:20:57.194 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:20:56 smithi142 ceph-mon[35271]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:20:57.194 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:20:56 smithi142 ceph-mon[35271]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-31T23:20:57.194 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:20:56 smithi142 ceph-mon[35271]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' 2024-03-31T23:20:57.209 INFO:tasks.workunit.client.0.smithi046.stdout: 1 195658 2.10 MB/sec execute 363 sec latency 15.457 ms 2024-03-31T23:20:58.208 INFO:tasks.workunit.client.0.smithi046.stdout: 1 196067 2.10 MB/sec execute 364 sec latency 18.870 ms 2024-03-31T23:20:59.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:20:58 smithi162 ceph-mon[36237]: pgmap v286: 129 pgs: 129 active+clean; 115 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 139 KiB/s rd, 358 KiB/s wr, 38 op/s 2024-03-31T23:20:59.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:20:58 smithi046 ceph-mon[32498]: pgmap v286: 129 pgs: 129 active+clean; 115 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 139 KiB/s rd, 358 KiB/s wr, 38 op/s 2024-03-31T23:20:59.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:20:58 smithi142 ceph-mon[35271]: pgmap v286: 129 pgs: 129 active+clean; 115 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 139 KiB/s rd, 358 KiB/s wr, 38 op/s 2024-03-31T23:20:59.208 INFO:tasks.workunit.client.0.smithi046.stdout: 1 196577 2.11 MB/sec execute 365 sec latency 19.431 ms 2024-03-31T23:20:59.263 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:59.260+0000 7ff84cb69640 1 -- 172.21.15.46:0/1717966176 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff848068dd0 msgr2=0x7ff848069230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:20:59.264 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:59.260+0000 7ff84cb69640 1 --2- 172.21.15.46:0/1717966176 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff848068dd0 0x7ff848069230 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7ff8300098c0 tx=0x7ff83002bb50 comp rx=0 tx=0).stop 2024-03-31T23:20:59.264 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:59.260+0000 7ff84cb69640 1 -- 172.21.15.46:0/1717966176 shutdown_connections 2024-03-31T23:20:59.264 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:59.260+0000 7ff84cb69640 1 --2- 172.21.15.46:0/1717966176 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff848069770 0x7ff84810e5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:20:59.264 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:59.260+0000 7ff84cb69640 1 --2- 172.21.15.46:0/1717966176 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff848068dd0 0x7ff848069230 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:20:59.264 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:59.260+0000 7ff84cb69640 1 --2- 172.21.15.46:0/1717966176 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff848107820 0x7ff848107c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:20:59.264 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:59.260+0000 7ff84cb69640 1 -- 172.21.15.46:0/1717966176 >> 172.21.15.46:0/1717966176 conn(0x7ff8480774a0 msgr2=0x7ff8480778b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:20:59.264 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:59.260+0000 7ff84cb69640 1 -- 172.21.15.46:0/1717966176 shutdown_connections 2024-03-31T23:20:59.264 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:59.260+0000 7ff84cb69640 1 -- 172.21.15.46:0/1717966176 wait complete. 2024-03-31T23:20:59.264 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:59.261+0000 7ff84cb69640 1 Processor -- start 2024-03-31T23:20:59.264 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:59.261+0000 7ff84cb69640 1 -- start start 2024-03-31T23:20:59.264 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:59.261+0000 7ff84cb69640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff848068dd0 0x7ff84819a060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:20:59.265 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:59.261+0000 7ff84cb69640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff848069770 0x7ff84819a5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:20:59.265 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:59.261+0000 7ff84cb69640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff848107820 0x7ff84819e920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:20:59.265 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:59.261+0000 7ff84cb69640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7ff848111980 con 0x7ff848068dd0 2024-03-31T23:20:59.265 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:59.261+0000 7ff84cb69640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7ff848111800 con 0x7ff848107820 2024-03-31T23:20:59.265 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:59.261+0000 7ff84cb69640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7ff848111b00 con 0x7ff848069770 2024-03-31T23:20:59.265 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:59.261+0000 7ff84659c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff848068dd0 0x7ff84819a060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:20:59.265 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:59.261+0000 7ff84659c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff848068dd0 0x7ff84819a060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37898/0 (socket says 172.21.15.46:37898) 2024-03-31T23:20:59.265 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:59.261+0000 7ff84659c640 1 -- 172.21.15.46:0/3920656246 learned_addr learned my addr 172.21.15.46:0/3920656246 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:20:59.265 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:59.261+0000 7ff83ffff640 1 --2- 172.21.15.46:0/3920656246 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff848069770 0x7ff84819a5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:20:59.265 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:59.262+0000 7ff846d9d640 1 --2- 172.21.15.46:0/3920656246 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff848107820 0x7ff84819e920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:20:59.265 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:59.262+0000 7ff84659c640 1 -- 172.21.15.46:0/3920656246 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff848069770 msgr2=0x7ff84819a5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:20:59.265 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:59.262+0000 7ff84659c640 1 --2- 172.21.15.46:0/3920656246 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff848069770 0x7ff84819a5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:20:59.265 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:59.262+0000 7ff84659c640 1 -- 172.21.15.46:0/3920656246 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff848107820 msgr2=0x7ff84819e920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:20:59.265 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:59.262+0000 7ff84659c640 1 --2- 172.21.15.46:0/3920656246 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff848107820 0x7ff84819e920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:20:59.266 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:59.262+0000 7ff84659c640 1 -- 172.21.15.46:0/3920656246 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff84819f050 con 0x7ff848068dd0 2024-03-31T23:20:59.266 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:59.262+0000 7ff83ffff640 1 --2- 172.21.15.46:0/3920656246 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff848069770 0x7ff84819a5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:20:59.266 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:59.262+0000 7ff84659c640 1 --2- 172.21.15.46:0/3920656246 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff848068dd0 0x7ff84819a060 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7ff83800e9c0 tx=0x7ff83800ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:20:59.266 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:59.262+0000 7ff83f7fe640 1 -- 172.21.15.46:0/3920656246 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff83800cde0 con 0x7ff848068dd0 2024-03-31T23:20:59.266 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:59.262+0000 7ff83f7fe640 1 -- 172.21.15.46:0/3920656246 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff8380035f0 con 0x7ff848068dd0 2024-03-31T23:20:59.266 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:59.262+0000 7ff84cb69640 1 -- 172.21.15.46:0/3920656246 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff84819f340 con 0x7ff848068dd0 2024-03-31T23:20:59.266 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:59.262+0000 7ff83f7fe640 1 -- 172.21.15.46:0/3920656246 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff838010690 con 0x7ff848068dd0 2024-03-31T23:20:59.266 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:59.262+0000 7ff84cb69640 1 -- 172.21.15.46:0/3920656246 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff8481a6bd0 con 0x7ff848068dd0 2024-03-31T23:20:59.271 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:59.264+0000 7ff83f7fe640 1 -- 172.21.15.46:0/3920656246 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 25) ==== 137451+0+0 (secure 0 0 0) 0x7ff8380037a0 con 0x7ff848068dd0 2024-03-31T23:20:59.271 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:59.265+0000 7ff84f822740 1 Processor -- start 2024-03-31T23:20:59.271 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:59.265+0000 7ff84f822740 1 -- start start 2024-03-31T23:20:59.271 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:59.265+0000 7ff84f822740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ceb45106c0 0x55ceb4510aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:20:59.272 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:59.265+0000 7ff84f822740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ceb4510fe0 0x55ceb4519870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:20:59.272 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:59.265+0000 7ff84f822740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55ceb4519db0 0x55ceb451c1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:20:59.272 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:59.265+0000 7ff84f822740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55ceb44738f0 con 0x55ceb45106c0 2024-03-31T23:20:59.272 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:59.265+0000 7ff84f822740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55ceb441c8d0 con 0x55ceb4519db0 2024-03-31T23:20:59.272 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:59.265+0000 7ff84f822740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55ceb446f990 con 0x55ceb4510fe0 2024-03-31T23:20:59.272 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:59.265+0000 7ff83f7fe640 1 --2- 172.21.15.46:0/3920656246 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7ff8340a26b0 0x7ff8340a4b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:20:59.272 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:59.265+0000 7ff846d9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ceb45106c0 0x55ceb4510aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:20:59.272 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:59.265+0000 7ff84659c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ceb4510fe0 0x55ceb4519870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:20:59.272 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:59.265+0000 7ff846d9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ceb45106c0 0x55ceb4510aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37910/0 (socket says 172.21.15.46:37910) 2024-03-31T23:20:59.272 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:59.265+0000 7ff846d9d640 1 -- 172.21.15.46:0/1776835979 learned_addr learned my addr 172.21.15.46:0/1776835979 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:20:59.272 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:59.266+0000 7ff83ffff640 1 --2- 172.21.15.46:0/1776835979 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55ceb4519db0 0x55ceb451c1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:20:59.272 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:59.266+0000 7ff846d9d640 1 --2- 172.21.15.46:0/3920656246 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7ff8340a26b0 0x7ff8340a4b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:20:59.272 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:59.266+0000 7ff83f7fe640 1 -- 172.21.15.46:0/3920656246 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 9554+0+0 (secure 0 0 0) 0x7ff8380cf240 con 0x7ff848068dd0 2024-03-31T23:20:59.272 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:59.266+0000 7ff846d9d640 1 -- 172.21.15.46:0/1776835979 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ceb4510fe0 msgr2=0x55ceb4519870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:20:59.273 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:59.266+0000 7ff846d9d640 1 --2- 172.21.15.46:0/1776835979 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ceb4510fe0 0x55ceb4519870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:20:59.273 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:59.266+0000 7ff846d9d640 1 -- 172.21.15.46:0/1776835979 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55ceb4519db0 msgr2=0x55ceb451c1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:20:59.273 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:59.266+0000 7ff846d9d640 1 --2- 172.21.15.46:0/1776835979 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55ceb4519db0 0x55ceb451c1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:20:59.273 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:59.266+0000 7ff846d9d640 1 -- 172.21.15.46:0/1776835979 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ceb4188210 con 0x55ceb45106c0 2024-03-31T23:20:59.273 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:59.266+0000 7ff846d9d640 1 --2- 172.21.15.46:0/1776835979 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ceb45106c0 0x55ceb4510aa0 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7ff840011e00 tx=0x7ff84000f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c22344052a039012 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:20:59.273 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:59.266+0000 7ff846d9d640 1 --2- 172.21.15.46:0/3920656246 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7ff8340a26b0 0x7ff8340a4b70 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7ff84000fc90 tx=0x7ff84001e000 comp rx=0 tx=0).ready entity=mgr.15654 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:20:59.273 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:59.266+0000 7ff83cff9640 1 -- 172.21.15.46:0/1776835979 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff84001dcb0 con 0x55ceb45106c0 2024-03-31T23:20:59.273 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:59.266+0000 7ff83cff9640 1 -- 172.21.15.46:0/1776835979 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff84001de50 con 0x55ceb45106c0 2024-03-31T23:20:59.273 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:59.266+0000 7ff83cff9640 1 -- 172.21.15.46:0/1776835979 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff840004020 con 0x55ceb45106c0 2024-03-31T23:20:59.273 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:59.267+0000 7ff84f822740 1 -- 172.21.15.46:0/1776835979 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ceb45106c0 msgr2=0x55ceb4510aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:20:59.273 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:59.267+0000 7ff84f822740 1 --2- 172.21.15.46:0/1776835979 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ceb45106c0 0x55ceb4510aa0 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7ff840011e00 tx=0x7ff84000f790 comp rx=0 tx=0).stop 2024-03-31T23:20:59.273 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:59.267+0000 7ff84f822740 1 -- 172.21.15.46:0/1776835979 shutdown_connections 2024-03-31T23:20:59.273 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:59.267+0000 7ff84f822740 1 --2- 172.21.15.46:0/1776835979 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55ceb4519db0 0x55ceb451c1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:20:59.273 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:59.267+0000 7ff84f822740 1 --2- 172.21.15.46:0/1776835979 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ceb4510fe0 0x55ceb4519870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:20:59.274 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:59.267+0000 7ff84f822740 1 --2- 172.21.15.46:0/1776835979 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ceb45106c0 0x55ceb4510aa0 unknown :-1 s=CLOSED pgs=789 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:20:59.274 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:59.267+0000 7ff84f822740 1 -- 172.21.15.46:0/1776835979 >> 172.21.15.46:0/1776835979 conn(0x55ceb450afe0 msgr2=0x55ceb450b3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:20:59.274 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:59.267+0000 7ff84f822740 1 -- 172.21.15.46:0/1776835979 shutdown_connections 2024-03-31T23:20:59.274 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:59.267+0000 7ff84f822740 1 -- 172.21.15.46:0/1776835979 wait complete. 2024-03-31T23:20:59.274 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:59.267+0000 7ff84f822740 1 -- start start 2024-03-31T23:20:59.274 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:59.268+0000 7ff84f822740 1 Processor -- start 2024-03-31T23:20:59.274 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:59.268+0000 7ff84f822740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ceb45106c0 0x55ceb4516ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:20:59.274 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:59.268+0000 7ff84f822740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ceb4510fe0 0x55ceb45173e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:20:59.274 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:59.268+0000 7ff84f822740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55ceb4519db0 0x55ceb4517920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:20:59.274 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:59.268+0000 7ff84f822740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55ceb451c920 con 0x55ceb45106c0 2024-03-31T23:20:59.274 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:59.268+0000 7ff84f822740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55ceb4411f80 con 0x55ceb4519db0 2024-03-31T23:20:59.274 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:59.268+0000 7ff84f822740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55ceb451caa0 con 0x55ceb4510fe0 2024-03-31T23:20:59.274 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:59.268+0000 7ff84f822740 10 client.0 _ll_register_callbacks cb 0x55ceb446a290 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T23:20:59.275 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:59.268+0000 7ff84659c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ceb45106c0 0x55ceb4516ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:20:59.275 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:59.268+0000 7ff83ffff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ceb4510fe0 0x55ceb45173e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:20:59.275 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:59.268+0000 7ff84659c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ceb45106c0 0x55ceb4516ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37922/0 (socket says 172.21.15.46:37922) 2024-03-31T23:20:59.275 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:59.268+0000 7ff84659c640 1 -- 172.21.15.46:0/141610360 learned_addr learned my addr 172.21.15.46:0/141610360 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:20:59.275 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:59.268+0000 7ff846d9d640 1 --2- 172.21.15.46:0/141610360 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55ceb4519db0 0x55ceb4517920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:20:59.275 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:59.268+0000 7ff84659c640 1 -- 172.21.15.46:0/141610360 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ceb4510fe0 msgr2=0x55ceb45173e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:20:59.275 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:59.268+0000 7ff84659c640 1 --2- 172.21.15.46:0/141610360 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ceb4510fe0 0x55ceb45173e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:20:59.275 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:59.268+0000 7ff84659c640 1 -- 172.21.15.46:0/141610360 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55ceb4519db0 msgr2=0x55ceb4517920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:20:59.275 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:59.268+0000 7ff84659c640 1 --2- 172.21.15.46:0/141610360 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55ceb4519db0 0x55ceb4517920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:20:59.275 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:59.268+0000 7ff84659c640 1 -- 172.21.15.46:0/141610360 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ceb4254620 con 0x55ceb45106c0 2024-03-31T23:20:59.275 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:59.269+0000 7ff84659c640 1 --2- 172.21.15.46:0/141610360 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ceb45106c0 0x55ceb4516ea0 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7ff838097a10 tx=0x7ff838015aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d45dbc995aafa9b server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:20:59.275 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:59.269+0000 7ff82dffb640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-31T23:20:59.275 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:59.269+0000 7ff82dffb640 1 -- 172.21.15.46:0/141610360 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff83809bcb0 con 0x55ceb45106c0 2024-03-31T23:20:59.275 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:59.269+0000 7ff82dffb640 1 -- 172.21.15.46:0/141610360 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 9554+0+0 (secure 0 0 0) 0x7ff8380094f0 con 0x55ceb45106c0 2024-03-31T23:20:59.275 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:59.269+0000 7ff84f822740 1 -- 172.21.15.46:0/141610360 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff84001de50 con 0x55ceb45106c0 2024-03-31T23:20:59.276 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:59.269+0000 7ff82dffb640 1 -- 172.21.15.46:0/141610360 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7ff83809b5b0 con 0x55ceb45106c0 2024-03-31T23:20:59.276 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:59.269+0000 7ff84f822740 10 client.16902 fetch_fsmap learned FSMap version 35 2024-03-31T23:20:59.276 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:59.269+0000 7ff84f822740 1 -- 172.21.15.46:0/141610360 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x55ceb45288b0 con 0x55ceb45106c0 2024-03-31T23:20:59.276 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:59.269+0000 7ff82dffb640 1 -- 172.21.15.46:0/141610360 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff84001de50 con 0x55ceb45106c0 2024-03-31T23:20:59.276 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:59.270+0000 7ff82dffb640 1 -- 172.21.15.46:0/141610360 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 35) ==== 8366+0+0 (secure 0 0 0) 0x7ff8380cdce0 con 0x55ceb45106c0 2024-03-31T23:20:59.276 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:59.270+0000 7ff84f822740 10 client.16902 fetch_fsmap finished waiting for FSMap version 35 2024-03-31T23:20:59.276 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:59.270+0000 7ff84f822740 10 client.16902 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-31T23:20:59.276 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:59.270+0000 7ff84f822740 20 client.16902 populate_metadata read hostname 'smithi046' 2024-03-31T23:20:59.276 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:59.270+0000 7ff84f822740 1 --2- 172.21.15.46:0/141610360 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55ceb4529d20 0x55ceb452c110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:20:59.276 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:59.270+0000 7ff84f822740 4 client.16902 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-31T23:20:59.276 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:59.270+0000 7ff84f822740 1 -- 172.21.15.46:0/141610360 --> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ceb4487120 con 0x55ceb4529d20 2024-03-31T23:20:59.276 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:59.270+0000 7ff83ffff640 1 --2- 172.21.15.46:0/141610360 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55ceb4529d20 0x55ceb452c110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:20:59.278 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:59.271+0000 7ff82dffb640 10 client.16902 ms_handle_connect on v2:172.21.15.142:6838/2519143918 2024-03-31T23:20:59.278 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:59.271+0000 7ff83ffff640 1 --2- 172.21.15.46:0/141610360 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55ceb4529d20 0x55ceb452c110 crc :-1 s=READY pgs=157 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=16b7d7aca481f651 server_cookie=ccb911d161a0bd8b in_seq=0 out_seq=0 2024-03-31T23:20:59.278 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:59.271+0000 7ff82dffb640 1 -- 172.21.15.46:0/141610360 <== mds.0 v2:172.21.15.142:6838/2519143918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55ceb4487120 con 0x55ceb4529d20 2024-03-31T23:20:59.278 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:59.272+0000 7ff82dffb640 10 client.16902 handle_command_reply: tid=0 2024-03-31T23:20:59.278 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:59.272+0000 7ff84f822740 1 client.16902 shutdown 2024-03-31T23:20:59.278 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:59.272+0000 7ff84f822740 1 -- 172.21.15.46:0/141610360 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ceb45106c0 msgr2=0x55ceb4516ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:20:59.278 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:59.272+0000 7ff84f822740 1 --2- 172.21.15.46:0/141610360 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ceb45106c0 0x55ceb4516ea0 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7ff838097a10 tx=0x7ff838015aa0 comp rx=0 tx=0).stop 2024-03-31T23:20:59.278 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:59.272+0000 7ff84f822740 1 -- 172.21.15.46:0/141610360 shutdown_connections 2024-03-31T23:20:59.278 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:59.272+0000 7ff84f822740 1 --2- 172.21.15.46:0/141610360 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55ceb4529d20 0x55ceb452c110 crc :-1 s=READY pgs=157 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:20:59.278 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:59.272+0000 7ff84f822740 1 --2- 172.21.15.46:0/141610360 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55ceb4519db0 0x55ceb4517920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:20:59.278 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:59.272+0000 7ff82dffb640 0 client.16902 ms_handle_reset on v2:172.21.15.142:6838/2519143918 2024-03-31T23:20:59.278 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:59.272+0000 7ff84f822740 1 --2- 172.21.15.46:0/141610360 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ceb4510fe0 0x55ceb45173e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:20:59.278 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:59.272+0000 7ff84f822740 1 --2- 172.21.15.46:0/141610360 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ceb45106c0 0x55ceb4516ea0 unknown :-1 s=CLOSED pgs=790 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:20:59.279 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:59.272+0000 7ff84f822740 1 -- 172.21.15.46:0/141610360 >> 172.21.15.46:0/141610360 conn(0x55ceb450afe0 msgr2=0x55ceb450cd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:20:59.279 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:59.272+0000 7ff84f822740 1 -- 172.21.15.46:0/141610360 shutdown_connections 2024-03-31T23:20:59.279 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:59.272+0000 7ff84f822740 1 -- 172.21.15.46:0/141610360 wait complete. 2024-03-31T23:20:59.279 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:59.272+0000 7ff84f822740 20 client.16902 trim_cache size 0 max 16384 2024-03-31T23:20:59.289 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:59.287+0000 7ff84f822740 1 Processor -- start 2024-03-31T23:20:59.289 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:59.287+0000 7ff84f822740 1 -- start start 2024-03-31T23:20:59.289 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:59.287+0000 7ff84f822740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ceb4510fe0 0x55ceb45195d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:20:59.289 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:59.287+0000 7ff84f822740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ceb4519db0 0x55ceb45256c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:20:59.289 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:59.287+0000 7ff84f822740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55ceb4529d20 0x55ceb4515770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:20:59.289 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:59.287+0000 7ff84f822740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7ff8481b3820 con 0x55ceb4519db0 2024-03-31T23:20:59.289 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:59.287+0000 7ff84f822740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7ff8481b1060 con 0x55ceb4529d20 2024-03-31T23:20:59.289 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:59.287+0000 7ff84f822740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7ff8481aeed0 con 0x55ceb4510fe0 2024-03-31T23:20:59.290 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:59.287+0000 7ff83ffff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ceb4519db0 0x55ceb45256c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:20:59.290 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:59.287+0000 7ff83ffff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ceb4519db0 0x55ceb45256c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37932/0 (socket says 172.21.15.46:37932) 2024-03-31T23:20:59.290 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:59.287+0000 7ff83ffff640 1 -- 172.21.15.46:0/4114424135 learned_addr learned my addr 172.21.15.46:0/4114424135 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:20:59.290 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:59.287+0000 7ff84659c640 1 --2- 172.21.15.46:0/4114424135 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ceb4510fe0 0x55ceb45195d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:20:59.290 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:59.287+0000 7ff846d9d640 1 --2- 172.21.15.46:0/4114424135 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55ceb4529d20 0x55ceb4515770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:20:59.290 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:59.287+0000 7ff83ffff640 1 -- 172.21.15.46:0/4114424135 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ceb4510fe0 msgr2=0x55ceb45195d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:20:59.290 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:59.287+0000 7ff83ffff640 1 --2- 172.21.15.46:0/4114424135 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ceb4510fe0 0x55ceb45195d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:20:59.290 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:59.287+0000 7ff83ffff640 1 -- 172.21.15.46:0/4114424135 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55ceb4529d20 msgr2=0x55ceb4515770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:20:59.290 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:59.287+0000 7ff83ffff640 1 --2- 172.21.15.46:0/4114424135 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55ceb4529d20 0x55ceb4515770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:20:59.290 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:59.287+0000 7ff83ffff640 1 -- 172.21.15.46:0/4114424135 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ceb4510e10 con 0x55ceb4519db0 2024-03-31T23:20:59.290 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:59.288+0000 7ff83ffff640 1 --2- 172.21.15.46:0/4114424135 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ceb4519db0 0x55ceb45256c0 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7ff830005b60 tx=0x7ff830005a90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc2a3e35420ece4a server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:20:59.290 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:59.288+0000 7ff82ffff640 1 -- 172.21.15.46:0/4114424135 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff83004d020 con 0x55ceb4519db0 2024-03-31T23:20:59.290 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:59.288+0000 7ff82ffff640 1 -- 172.21.15.46:0/4114424135 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff830049d60 con 0x55ceb4519db0 2024-03-31T23:20:59.290 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:59.288+0000 7ff82ffff640 1 -- 172.21.15.46:0/4114424135 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff830049640 con 0x55ceb4519db0 2024-03-31T23:20:59.294 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:59.289+0000 7ff84f822740 1 -- 172.21.15.46:0/4114424135 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ceb4519db0 msgr2=0x55ceb45256c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:20:59.294 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:59.289+0000 7ff84f822740 1 --2- 172.21.15.46:0/4114424135 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ceb4519db0 0x55ceb45256c0 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7ff830005b60 tx=0x7ff830005a90 comp rx=0 tx=0).stop 2024-03-31T23:20:59.294 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:59.289+0000 7ff84f822740 1 -- 172.21.15.46:0/4114424135 shutdown_connections 2024-03-31T23:20:59.294 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:59.289+0000 7ff84f822740 1 --2- 172.21.15.46:0/4114424135 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55ceb4529d20 0x55ceb4515770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:20:59.294 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:59.289+0000 7ff84f822740 1 --2- 172.21.15.46:0/4114424135 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ceb4519db0 0x55ceb45256c0 unknown :-1 s=CLOSED pgs=791 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:20:59.294 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:59.289+0000 7ff84f822740 1 --2- 172.21.15.46:0/4114424135 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ceb4510fe0 0x55ceb45195d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:20:59.294 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:59.289+0000 7ff84f822740 1 -- 172.21.15.46:0/4114424135 >> 172.21.15.46:0/4114424135 conn(0x55ceb450afe0 msgr2=0x55ceb450b3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:20:59.294 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:59.289+0000 7ff84f822740 1 -- 172.21.15.46:0/4114424135 shutdown_connections 2024-03-31T23:20:59.294 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:59.289+0000 7ff84f822740 1 -- 172.21.15.46:0/4114424135 wait complete. 2024-03-31T23:20:59.294 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:59.289+0000 7ff84f822740 1 -- start start 2024-03-31T23:20:59.295 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:59.289+0000 7ff84f822740 1 Processor -- start 2024-03-31T23:20:59.295 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:59.289+0000 7ff84f822740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ceb4510fe0 0x55ceb4521ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:20:59.295 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:59.289+0000 7ff84f822740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ceb4519db0 0x55ceb45136f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:20:59.295 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:59.289+0000 7ff84f822740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55ceb4529d20 0x55ceb4513c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:20:59.295 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:59.289+0000 7ff84f822740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7ff8481af520 con 0x55ceb4519db0 2024-03-31T23:20:59.295 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:59.289+0000 7ff84f822740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7ff8481b0990 con 0x55ceb4529d20 2024-03-31T23:20:59.295 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:59.289+0000 7ff84f822740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7ff8481afbf0 con 0x55ceb4510fe0 2024-03-31T23:20:59.295 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:59.290+0000 7ff84f822740 10 client.0 _ll_register_callbacks cb 0x7ff838097190 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T23:20:59.295 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:59.290+0000 7ff83ffff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ceb4519db0 0x55ceb45136f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:20:59.295 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:59.290+0000 7ff846d9d640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55ceb4529d20 0x55ceb4513c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:20:59.295 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:59.290+0000 7ff84659c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ceb4510fe0 0x55ceb4521ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:20:59.295 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:59.290+0000 7ff83ffff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ceb4519db0 0x55ceb45136f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37948/0 (socket says 172.21.15.46:37948) 2024-03-31T23:20:59.295 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:59.290+0000 7ff83ffff640 1 -- 172.21.15.46:0/228561027 learned_addr learned my addr 172.21.15.46:0/228561027 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:20:59.295 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:59.290+0000 7ff846d9d640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55ceb4529d20 0x55ceb4513c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.46:43926/0 (socket says 172.21.15.46:43926) 2024-03-31T23:20:59.296 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:59.290+0000 7ff83ffff640 1 -- 172.21.15.46:0/228561027 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ceb4510fe0 msgr2=0x55ceb4521ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:20:59.296 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:59.290+0000 7ff83ffff640 1 --2- 172.21.15.46:0/228561027 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ceb4510fe0 0x55ceb4521ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:20:59.296 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:59.290+0000 7ff83ffff640 1 -- 172.21.15.46:0/228561027 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55ceb4529d20 msgr2=0x55ceb4513c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:20:59.296 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:59.290+0000 7ff83ffff640 1 --2- 172.21.15.46:0/228561027 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55ceb4529d20 0x55ceb4513c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:20:59.296 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:59.290+0000 7ff83ffff640 1 -- 172.21.15.46:0/228561027 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ceb4514320 con 0x55ceb4519db0 2024-03-31T23:20:59.296 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:59.290+0000 7ff83ffff640 1 --2- 172.21.15.46:0/228561027 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ceb4519db0 0x55ceb45136f0 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7ff830037040 tx=0x7ff830049f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c59276958f0822dd server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:20:59.296 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:59.290+0000 7ff82dffb640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-31T23:20:59.296 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:59.290+0000 7ff82dffb640 1 -- 172.21.15.46:0/228561027 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff830058070 con 0x55ceb4519db0 2024-03-31T23:20:59.296 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:59.290+0000 7ff82dffb640 1 -- 172.21.15.46:0/228561027 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 9554+0+0 (secure 0 0 0) 0x7ff83002c430 con 0x55ceb4519db0 2024-03-31T23:20:59.296 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:59.290+0000 7ff84f822740 1 -- 172.21.15.46:0/228561027 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff830049d60 con 0x55ceb4519db0 2024-03-31T23:20:59.297 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:59.290+0000 7ff82dffb640 1 -- 172.21.15.46:0/228561027 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7ff830064050 con 0x55ceb4519db0 2024-03-31T23:20:59.297 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:59.290+0000 7ff84f822740 10 client.16908 fetch_fsmap learned FSMap version 35 2024-03-31T23:20:59.297 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:59.291+0000 7ff84f822740 1 -- 172.21.15.46:0/228561027 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x55ceb458ff00 con 0x55ceb4519db0 2024-03-31T23:20:59.297 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:59.291+0000 7ff82dffb640 1 -- 172.21.15.46:0/228561027 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff830049d60 con 0x55ceb4519db0 2024-03-31T23:20:59.297 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:59.291+0000 7ff82dffb640 1 -- 172.21.15.46:0/228561027 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 35) ==== 8366+0+0 (secure 0 0 0) 0x7ff830003680 con 0x55ceb4519db0 2024-03-31T23:20:59.297 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:59.291+0000 7ff84f822740 10 client.16908 fetch_fsmap finished waiting for FSMap version 35 2024-03-31T23:20:59.297 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:59.291+0000 7ff84f822740 10 client.16908 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-31T23:20:59.297 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:59.291+0000 7ff84f822740 20 client.16908 populate_metadata read hostname 'smithi046' 2024-03-31T23:20:59.297 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:59.291+0000 7ff84f822740 1 --2- 172.21.15.46:0/228561027 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55ceb4590650 0x55ceb4592a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:20:59.297 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:59.291+0000 7ff84f822740 4 client.16908 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-31T23:20:59.297 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:59.291+0000 7ff84f822740 1 -- 172.21.15.46:0/228561027 --> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55ceb4480e30 con 0x55ceb4590650 2024-03-31T23:20:59.297 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:59.291+0000 7ff84659c640 1 --2- 172.21.15.46:0/228561027 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55ceb4590650 0x55ceb4592a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:20:59.297 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:59.291+0000 7ff84659c640 1 --2- 172.21.15.46:0/228561027 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55ceb4590650 0x55ceb4592a70 crc :-1 s=READY pgs=158 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=109ab717f86a879f server_cookie=6be24f3c7df893e5 in_seq=0 out_seq=0 2024-03-31T23:20:59.298 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:59.291+0000 7ff82dffb640 10 client.16908 ms_handle_connect on v2:172.21.15.142:6838/2519143918 2024-03-31T23:20:59.316 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:59.314+0000 7ff82dffb640 1 -- 172.21.15.46:0/228561027 <== mds.0 v2:172.21.15.142:6838/2519143918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55ceb4480e30 con 0x55ceb4590650 2024-03-31T23:20:59.316 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:59.314+0000 7ff82dffb640 10 client.16908 handle_command_reply: tid=0 2024-03-31T23:20:59.317 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:59.314+0000 7ff84f822740 1 client.16908 shutdown 2024-03-31T23:20:59.317 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:59.315+0000 7ff84f822740 1 -- 172.21.15.46:0/228561027 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ceb4519db0 msgr2=0x55ceb45136f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:20:59.317 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:59.315+0000 7ff84f822740 1 --2- 172.21.15.46:0/228561027 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ceb4519db0 0x55ceb45136f0 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7ff830037040 tx=0x7ff830049f10 comp rx=0 tx=0).stop 2024-03-31T23:20:59.317 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:59.315+0000 7ff84f822740 1 -- 172.21.15.46:0/228561027 shutdown_connections 2024-03-31T23:20:59.317 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:59.315+0000 7ff84f822740 1 --2- 172.21.15.46:0/228561027 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55ceb4590650 0x55ceb4592a70 crc :-1 s=READY pgs=158 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:20:59.317 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:59.315+0000 7ff84f822740 1 --2- 172.21.15.46:0/228561027 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55ceb4529d20 0x55ceb4513c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:20:59.317 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:59.315+0000 7ff84f822740 1 --2- 172.21.15.46:0/228561027 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ceb4519db0 0x55ceb45136f0 unknown :-1 s=CLOSED pgs=792 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:20:59.317 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:59.315+0000 7ff84f822740 1 --2- 172.21.15.46:0/228561027 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ceb4510fe0 0x55ceb4521ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:20:59.317 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:59.315+0000 7ff84f822740 1 -- 172.21.15.46:0/228561027 >> 172.21.15.46:0/228561027 conn(0x55ceb450afe0 msgr2=0x55ceb4515730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:20:59.317 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:59.315+0000 7ff82dffb640 0 client.16908 ms_handle_reset on v2:172.21.15.142:6838/2519143918 2024-03-31T23:20:59.317 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:59.315+0000 7ff84f822740 1 -- 172.21.15.46:0/228561027 shutdown_connections 2024-03-31T23:20:59.318 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:59.315+0000 7ff84f822740 1 -- 172.21.15.46:0/228561027 wait complete. 2024-03-31T23:20:59.318 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:59.315+0000 7ff84f822740 20 client.16908 trim_cache size 0 max 16384 2024-03-31T23:20:59.318 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout:{ 2024-03-31T23:20:59.318 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout: "message": "", 2024-03-31T23:20:59.318 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout: "return_code": 0 2024-03-31T23:20:59.318 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout:} 2024-03-31T23:20:59.319 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:59.316+0000 7ff84cb69640 1 -- 172.21.15.46:0/3920656246 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7ff8340a26b0 msgr2=0x7ff8340a4b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:20:59.319 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:59.316+0000 7ff84cb69640 1 --2- 172.21.15.46:0/3920656246 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7ff8340a26b0 0x7ff8340a4b70 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7ff84000fc90 tx=0x7ff84001e000 comp rx=0 tx=0).stop 2024-03-31T23:20:59.319 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:59.316+0000 7ff84cb69640 1 -- 172.21.15.46:0/3920656246 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff848068dd0 msgr2=0x7ff84819a060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:20:59.319 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:59.316+0000 7ff84cb69640 1 --2- 172.21.15.46:0/3920656246 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff848068dd0 0x7ff84819a060 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7ff83800e9c0 tx=0x7ff83800ee80 comp rx=0 tx=0).stop 2024-03-31T23:20:59.320 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:59.316+0000 7ff84cb69640 1 -- 172.21.15.46:0/3920656246 shutdown_connections 2024-03-31T23:20:59.320 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:59.316+0000 7ff84cb69640 1 --2- 172.21.15.46:0/3920656246 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7ff8340a26b0 0x7ff8340a4b70 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:20:59.320 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:59.316+0000 7ff84cb69640 1 --2- 172.21.15.46:0/3920656246 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff848107820 0x7ff84819e920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:20:59.320 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:59.316+0000 7ff84cb69640 1 --2- 172.21.15.46:0/3920656246 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff848069770 0x7ff84819a5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:20:59.320 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:59.316+0000 7ff84cb69640 1 --2- 172.21.15.46:0/3920656246 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff848068dd0 0x7ff84819a060 unknown :-1 s=CLOSED pgs=788 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:20:59.320 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:59.316+0000 7ff84cb69640 1 -- 172.21.15.46:0/3920656246 >> 172.21.15.46:0/3920656246 conn(0x7ff8480774a0 msgr2=0x7ff84810cb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:20:59.320 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:59.316+0000 7ff84cb69640 1 -- 172.21.15.46:0/3920656246 shutdown_connections 2024-03-31T23:20:59.320 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:20:59.316+0000 7ff84cb69640 1 -- 172.21.15.46:0/3920656246 wait complete. 2024-03-31T23:21:00.208 INFO:tasks.workunit.client.0.smithi046.stdout: 1 197063 2.10 MB/sec execute 366 sec latency 27.070 ms 2024-03-31T23:21:01.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:21:00 smithi162 ceph-mon[36237]: pgmap v287: 129 pgs: 129 active+clean; 118 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 281 KiB/s rd, 764 KiB/s wr, 76 op/s 2024-03-31T23:21:01.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:21:00 smithi046 ceph-mon[32498]: pgmap v287: 129 pgs: 129 active+clean; 118 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 281 KiB/s rd, 764 KiB/s wr, 76 op/s 2024-03-31T23:21:01.194 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:21:00 smithi142 ceph-mon[35271]: pgmap v287: 129 pgs: 129 active+clean; 118 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 281 KiB/s rd, 764 KiB/s wr, 76 op/s 2024-03-31T23:21:01.209 INFO:tasks.workunit.client.0.smithi046.stdout: 1 197544 2.10 MB/sec execute 367 sec latency 22.905 ms 2024-03-31T23:21:02.209 INFO:tasks.workunit.client.0.smithi046.stdout: 1 198268 2.11 MB/sec execute 368 sec latency 15.911 ms 2024-03-31T23:21:03.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:21:02 smithi162 ceph-mon[36237]: pgmap v288: 129 pgs: 129 active+clean; 118 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 168 KiB/s rd, 435 KiB/s wr, 44 op/s 2024-03-31T23:21:03.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:21:02 smithi046 ceph-mon[32498]: pgmap v288: 129 pgs: 129 active+clean; 118 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 168 KiB/s rd, 435 KiB/s wr, 44 op/s 2024-03-31T23:21:03.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:21:02 smithi142 ceph-mon[35271]: pgmap v288: 129 pgs: 129 active+clean; 118 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 168 KiB/s rd, 435 KiB/s wr, 44 op/s 2024-03-31T23:21:03.208 INFO:tasks.workunit.client.0.smithi046.stdout: 1 198619 2.12 MB/sec execute 369 sec latency 40.063 ms 2024-03-31T23:21:04.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:21:03 smithi162 ceph-mon[36237]: osdmap e103: 12 total, 12 up, 12 in 2024-03-31T23:21:04.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:21:03 smithi046 ceph-mon[32498]: osdmap e103: 12 total, 12 up, 12 in 2024-03-31T23:21:04.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:21:03 smithi142 ceph-mon[35271]: osdmap e103: 12 total, 12 up, 12 in 2024-03-31T23:21:04.209 INFO:tasks.workunit.client.0.smithi046.stdout: 1 198710 2.11 MB/sec execute 370 sec latency 820.422 ms 2024-03-31T23:21:05.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:21:04 smithi162 ceph-mon[36237]: pgmap v290: 129 pgs: 129 active+clean; 118 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 201 KiB/s rd, 522 KiB/s wr, 53 op/s 2024-03-31T23:21:05.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:21:04 smithi046 ceph-mon[32498]: pgmap v290: 129 pgs: 129 active+clean; 118 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 201 KiB/s rd, 522 KiB/s wr, 53 op/s 2024-03-31T23:21:05.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:21:04 smithi142 ceph-mon[35271]: pgmap v290: 129 pgs: 129 active+clean; 118 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 201 KiB/s rd, 522 KiB/s wr, 53 op/s 2024-03-31T23:21:05.209 INFO:tasks.workunit.client.0.smithi046.stdout: 1 198710 2.11 MB/sec execute 371 sec latency 1820.478 ms 2024-03-31T23:21:06.209 INFO:tasks.workunit.client.0.smithi046.stdout: 1 198710 2.10 MB/sec execute 372 sec latency 2820.596 ms 2024-03-31T23:21:07.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:21:06 smithi162 ceph-mon[36237]: pgmap v291: 129 pgs: 129 active+clean; 112 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 427 KiB/s rd, 1.2 MiB/s wr, 127 op/s 2024-03-31T23:21:07.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:21:06 smithi046 ceph-mon[32498]: pgmap v291: 129 pgs: 129 active+clean; 112 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 427 KiB/s rd, 1.2 MiB/s wr, 127 op/s 2024-03-31T23:21:07.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:21:06 smithi142 ceph-mon[35271]: pgmap v291: 129 pgs: 129 active+clean; 112 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 427 KiB/s rd, 1.2 MiB/s wr, 127 op/s 2024-03-31T23:21:07.209 INFO:tasks.workunit.client.0.smithi046.stdout: 1 198710 2.10 MB/sec execute 373 sec latency 3820.708 ms 2024-03-31T23:21:08.209 INFO:tasks.workunit.client.0.smithi046.stdout: 1 198777 2.09 MB/sec execute 374 sec latency 4356.294 ms 2024-03-31T23:21:09.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:21:08 smithi162 ceph-mon[36237]: pgmap v292: 129 pgs: 129 active+clean; 112 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 427 KiB/s rd, 1.2 MiB/s wr, 127 op/s 2024-03-31T23:21:09.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:21:08 smithi046 ceph-mon[32498]: pgmap v292: 129 pgs: 129 active+clean; 112 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 427 KiB/s rd, 1.2 MiB/s wr, 127 op/s 2024-03-31T23:21:09.194 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:21:08 smithi142 ceph-mon[35271]: pgmap v292: 129 pgs: 129 active+clean; 112 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 427 KiB/s rd, 1.2 MiB/s wr, 127 op/s 2024-03-31T23:21:09.209 INFO:tasks.workunit.client.0.smithi046.stdout: 1 198777 2.08 MB/sec execute 375 sec latency 1230.767 ms 2024-03-31T23:21:10.209 INFO:tasks.workunit.client.0.smithi046.stdout: 1 198777 2.08 MB/sec execute 376 sec latency 2230.881 ms 2024-03-31T23:21:11.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:21:10 smithi162 ceph-mon[36237]: pgmap v293: 129 pgs: 129 active+clean; 113 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 303 KiB/s rd, 816 KiB/s wr, 95 op/s 2024-03-31T23:21:11.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:21:10 smithi162 ceph-mon[36237]: osdmap e104: 12 total, 12 up, 12 in 2024-03-31T23:21:11.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:21:10 smithi046 ceph-mon[32498]: pgmap v293: 129 pgs: 129 active+clean; 113 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 303 KiB/s rd, 816 KiB/s wr, 95 op/s 2024-03-31T23:21:11.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:21:10 smithi046 ceph-mon[32498]: osdmap e104: 12 total, 12 up, 12 in 2024-03-31T23:21:11.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:21:10 smithi142 ceph-mon[35271]: pgmap v293: 129 pgs: 129 active+clean; 113 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 303 KiB/s rd, 816 KiB/s wr, 95 op/s 2024-03-31T23:21:11.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:21:10 smithi142 ceph-mon[35271]: osdmap e104: 12 total, 12 up, 12 in 2024-03-31T23:21:11.209 INFO:tasks.workunit.client.0.smithi046.stdout: 1 198777 2.07 MB/sec execute 377 sec latency 3230.990 ms 2024-03-31T23:21:12.210 INFO:tasks.workunit.client.0.smithi046.stdout: 1 198777 2.07 MB/sec execute 378 sec latency 4231.094 ms 2024-03-31T23:21:12.415 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:12.414+0000 7fa068b99640 1 -- 172.21.15.46:0/2295215315 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa064107730 msgr2=0x7fa064107b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:21:12.415 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:12.414+0000 7fa068b99640 1 --2- 172.21.15.46:0/2295215315 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa064107730 0x7fa064107b10 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7fa054009930 tx=0x7fa05402bc30 comp rx=0 tx=0).stop 2024-03-31T23:21:12.415 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:12.414+0000 7fa068b99640 1 -- 172.21.15.46:0/2295215315 shutdown_connections 2024-03-31T23:21:12.415 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:12.414+0000 7fa068b99640 1 --2- 172.21.15.46:0/2295215315 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa0641020e0 0x7fa06410e5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:21:12.415 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:12.414+0000 7fa068b99640 1 --2- 172.21.15.46:0/2295215315 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa064101720 0x7fa064101ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:21:12.415 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:12.414+0000 7fa068b99640 1 --2- 172.21.15.46:0/2295215315 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa064107730 0x7fa064107b10 unknown :-1 s=CLOSED pgs=793 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:21:12.415 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:12.414+0000 7fa068b99640 1 -- 172.21.15.46:0/2295215315 >> 172.21.15.46:0/2295215315 conn(0x7fa0640fd230 msgr2=0x7fa0640ff650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:21:12.415 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:12.414+0000 7fa068b99640 1 -- 172.21.15.46:0/2295215315 shutdown_connections 2024-03-31T23:21:12.415 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:12.414+0000 7fa068b99640 1 -- 172.21.15.46:0/2295215315 wait complete. 2024-03-31T23:21:12.417 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:12.415+0000 7fa068b99640 1 Processor -- start 2024-03-31T23:21:12.417 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:12.415+0000 7fa068b99640 1 -- start start 2024-03-31T23:21:12.418 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:12.416+0000 7fa068b99640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa064101720 0x7fa064197ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:21:12.418 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:12.416+0000 7fa068b99640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa0641020e0 0x7fa064198430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:21:12.418 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:12.416+0000 7fa068b99640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa064107730 0x7fa06419c760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:21:12.418 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:12.416+0000 7fa068b99640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fa0641119f0 con 0x7fa064101720 2024-03-31T23:21:12.418 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:12.416+0000 7fa068b99640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fa064111870 con 0x7fa0641020e0 2024-03-31T23:21:12.418 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:12.416+0000 7fa068b99640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fa064111b70 con 0x7fa064107730 2024-03-31T23:21:12.418 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:12.416+0000 7fa06259c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa064101720 0x7fa064197ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:21:12.418 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:12.416+0000 7fa06259c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa064101720 0x7fa064197ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:42010/0 (socket says 172.21.15.46:42010) 2024-03-31T23:21:12.418 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:12.416+0000 7fa06259c640 1 -- 172.21.15.46:0/3382562470 learned_addr learned my addr 172.21.15.46:0/3382562470 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:21:12.418 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:12.416+0000 7fa061d9b640 1 --2- 172.21.15.46:0/3382562470 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa0641020e0 0x7fa064198430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:21:12.418 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:12.416+0000 7fa062d9d640 1 --2- 172.21.15.46:0/3382562470 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa064107730 0x7fa06419c760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:21:12.418 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:12.416+0000 7fa06259c640 1 -- 172.21.15.46:0/3382562470 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa064107730 msgr2=0x7fa06419c760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:21:12.418 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:12.416+0000 7fa06259c640 1 --2- 172.21.15.46:0/3382562470 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa064107730 0x7fa06419c760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:21:12.418 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:12.416+0000 7fa06259c640 1 -- 172.21.15.46:0/3382562470 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa0641020e0 msgr2=0x7fa064198430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:21:12.419 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:12.416+0000 7fa06259c640 1 --2- 172.21.15.46:0/3382562470 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa0641020e0 0x7fa064198430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:21:12.419 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:12.416+0000 7fa06259c640 1 -- 172.21.15.46:0/3382562470 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa06419ce90 con 0x7fa064101720 2024-03-31T23:21:12.419 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:12.416+0000 7fa06259c640 1 --2- 172.21.15.46:0/3382562470 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa064101720 0x7fa064197ef0 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7fa054009230 tx=0x7fa05402c5d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:21:12.419 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:12.416+0000 7fa0537fe640 1 -- 172.21.15.46:0/3382562470 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa054043020 con 0x7fa064101720 2024-03-31T23:21:12.419 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:12.416+0000 7fa0537fe640 1 -- 172.21.15.46:0/3382562470 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa05402ca70 con 0x7fa064101720 2024-03-31T23:21:12.419 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:12.416+0000 7fa068b99640 1 -- 172.21.15.46:0/3382562470 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa06419d120 con 0x7fa064101720 2024-03-31T23:21:12.419 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:12.417+0000 7fa0537fe640 1 -- 172.21.15.46:0/3382562470 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa054003a50 con 0x7fa064101720 2024-03-31T23:21:12.419 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:12.417+0000 7fa068b99640 1 -- 172.21.15.46:0/3382562470 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa0641a4a10 con 0x7fa064101720 2024-03-31T23:21:12.430 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:12.419+0000 7fa0537fe640 1 -- 172.21.15.46:0/3382562470 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 25) ==== 137451+0+0 (secure 0 0 0) 0x7fa054046020 con 0x7fa064101720 2024-03-31T23:21:12.430 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:12.419+0000 7fa06b776740 1 Processor -- start 2024-03-31T23:21:12.430 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:12.420+0000 7fa06b776740 1 -- start start 2024-03-31T23:21:12.430 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:12.420+0000 7fa06b776740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e2158046c0 0x55e215804aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:21:12.430 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:12.420+0000 7fa06b776740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55e215804fe0 0x55e21580d870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:21:12.431 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:12.420+0000 7fa06b776740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e21580ddb0 0x55e2158101a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:21:12.431 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:12.420+0000 7fa06b776740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55e2157678f0 con 0x55e2158046c0 2024-03-31T23:21:12.431 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:12.420+0000 7fa06b776740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55e2157108d0 con 0x55e215804fe0 2024-03-31T23:21:12.431 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:12.420+0000 7fa06b776740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55e215763990 con 0x55e21580ddb0 2024-03-31T23:21:12.431 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:12.420+0000 7fa0537fe640 1 --2- 172.21.15.46:0/3382562470 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fa04c0a26b0 0x7fa04c0a4b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:21:12.432 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:12.420+0000 7fa062d9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e2158046c0 0x55e215804aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:21:12.432 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:12.420+0000 7fa06259c640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55e215804fe0 0x55e21580d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:21:12.432 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:12.420+0000 7fa062d9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e2158046c0 0x55e215804aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:42026/0 (socket says 172.21.15.46:42026) 2024-03-31T23:21:12.432 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:12.420+0000 7fa062d9d640 1 -- 172.21.15.46:0/3388834851 learned_addr learned my addr 172.21.15.46:0/3388834851 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:21:12.433 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:12.420+0000 7fa062d9d640 1 --2- 172.21.15.46:0/3382562470 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fa04c0a26b0 0x7fa04c0a4b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:21:12.433 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:12.420+0000 7fa061d9b640 1 --2- 172.21.15.46:0/3388834851 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e21580ddb0 0x55e2158101a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:21:12.433 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:12.420+0000 7fa0537fe640 1 -- 172.21.15.46:0/3382562470 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 9554+0+0 (secure 0 0 0) 0x7fa0540f0e10 con 0x7fa064101720 2024-03-31T23:21:12.434 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:12.420+0000 7fa062d9d640 1 -- 172.21.15.46:0/3388834851 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e21580ddb0 msgr2=0x55e2158101a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:21:12.434 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:12.420+0000 7fa062d9d640 1 --2- 172.21.15.46:0/3388834851 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e21580ddb0 0x55e2158101a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:21:12.434 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:12.420+0000 7fa062d9d640 1 -- 172.21.15.46:0/3388834851 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55e215804fe0 msgr2=0x55e21580d870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:21:12.434 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:12.420+0000 7fa062d9d640 1 --2- 172.21.15.46:0/3388834851 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55e215804fe0 0x55e21580d870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:21:12.434 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:12.420+0000 7fa062d9d640 1 -- 172.21.15.46:0/3388834851 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e21547c210 con 0x55e2158046c0 2024-03-31T23:21:12.435 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:12.420+0000 7fa06259c640 1 --2- 172.21.15.46:0/3388834851 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55e215804fe0 0x55e21580d870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:21:12.435 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:12.421+0000 7fa062d9d640 1 --2- 172.21.15.46:0/3388834851 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e2158046c0 0x55e215804aa0 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7fa05c011da0 tx=0x7fa05c00f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b8a693901dac63de server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:21:12.435 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:12.421+0000 7fa062d9d640 1 --2- 172.21.15.46:0/3382562470 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fa04c0a26b0 0x7fa04c0a4b70 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fa05c00fc90 tx=0x7fa05c01e000 comp rx=0 tx=0).ready entity=mgr.15654 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:21:12.435 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:12.421+0000 7fa050ff9640 1 -- 172.21.15.46:0/3388834851 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa05c022070 con 0x55e2158046c0 2024-03-31T23:21:12.435 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:12.421+0000 7fa050ff9640 1 -- 172.21.15.46:0/3388834851 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa05c009dc0 con 0x55e2158046c0 2024-03-31T23:21:12.435 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:12.421+0000 7fa050ff9640 1 -- 172.21.15.46:0/3388834851 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa05c004050 con 0x55e2158046c0 2024-03-31T23:21:12.435 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:12.422+0000 7fa06b776740 1 -- 172.21.15.46:0/3388834851 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e2158046c0 msgr2=0x55e215804aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:21:12.435 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:12.422+0000 7fa06b776740 1 --2- 172.21.15.46:0/3388834851 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e2158046c0 0x55e215804aa0 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7fa05c011da0 tx=0x7fa05c00f790 comp rx=0 tx=0).stop 2024-03-31T23:21:12.435 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:12.422+0000 7fa06b776740 1 -- 172.21.15.46:0/3388834851 shutdown_connections 2024-03-31T23:21:12.435 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:12.422+0000 7fa06b776740 1 --2- 172.21.15.46:0/3388834851 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e21580ddb0 0x55e2158101a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:21:12.435 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:12.422+0000 7fa06b776740 1 --2- 172.21.15.46:0/3388834851 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55e215804fe0 0x55e21580d870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:21:12.435 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:12.422+0000 7fa06b776740 1 --2- 172.21.15.46:0/3388834851 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e2158046c0 0x55e215804aa0 unknown :-1 s=CLOSED pgs=795 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:21:12.436 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:12.422+0000 7fa06b776740 1 -- 172.21.15.46:0/3388834851 >> 172.21.15.46:0/3388834851 conn(0x55e2157fefe0 msgr2=0x55e2157ff3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:21:12.436 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:12.422+0000 7fa06b776740 1 -- 172.21.15.46:0/3388834851 shutdown_connections 2024-03-31T23:21:12.436 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:12.422+0000 7fa06b776740 1 -- 172.21.15.46:0/3388834851 wait complete. 2024-03-31T23:21:12.436 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:12.422+0000 7fa06b776740 1 -- start start 2024-03-31T23:21:12.436 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:12.422+0000 7fa06b776740 1 Processor -- start 2024-03-31T23:21:12.436 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:12.422+0000 7fa06b776740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e2158046c0 0x55e21580adf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:21:12.436 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:12.422+0000 7fa06b776740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e215804fe0 0x55e21580b330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:21:12.436 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:12.422+0000 7fa06b776740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55e21580ddb0 0x55e21580b870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:21:12.436 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:12.422+0000 7fa06b776740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55e215810980 con 0x55e2158046c0 2024-03-31T23:21:12.436 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:12.422+0000 7fa06b776740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55e215705f80 con 0x55e21580ddb0 2024-03-31T23:21:12.436 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:12.422+0000 7fa06b776740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55e215810b00 con 0x55e215804fe0 2024-03-31T23:21:12.436 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:12.422+0000 7fa06259c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e2158046c0 0x55e21580adf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:21:12.436 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:12.422+0000 7fa06b776740 10 client.0 _ll_register_callbacks cb 0x55e21575e290 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T23:21:12.436 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:12.423+0000 7fa06259c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e2158046c0 0x55e21580adf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:42028/0 (socket says 172.21.15.46:42028) 2024-03-31T23:21:12.437 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:12.423+0000 7fa062d9d640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55e21580ddb0 0x55e21580b870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:21:12.437 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:12.423+0000 7fa06259c640 1 -- 172.21.15.46:0/3029605179 learned_addr learned my addr 172.21.15.46:0/3029605179 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:21:12.437 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:12.423+0000 7fa061d9b640 1 --2- 172.21.15.46:0/3029605179 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e215804fe0 0x55e21580b330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:21:12.437 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:12.423+0000 7fa06259c640 1 -- 172.21.15.46:0/3029605179 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e215804fe0 msgr2=0x55e21580b330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:21:12.437 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:12.423+0000 7fa06259c640 1 --2- 172.21.15.46:0/3029605179 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e215804fe0 0x55e21580b330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:21:12.437 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:12.423+0000 7fa06259c640 1 -- 172.21.15.46:0/3029605179 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55e21580ddb0 msgr2=0x55e21580b870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:21:12.437 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:12.423+0000 7fa06259c640 1 --2- 172.21.15.46:0/3029605179 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55e21580ddb0 0x55e21580b870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:21:12.437 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:12.423+0000 7fa06259c640 1 -- 172.21.15.46:0/3029605179 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e215548620 con 0x55e2158046c0 2024-03-31T23:21:12.437 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:12.423+0000 7fa062d9d640 1 --2- 172.21.15.46:0/3029605179 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55e21580ddb0 0x55e21580b870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:21:12.437 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:12.423+0000 7fa06259c640 1 --2- 172.21.15.46:0/3029605179 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e2158046c0 0x55e21580adf0 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7fa054009900 tx=0x7fa054050f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=38562aace0bf9a18 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:21:12.437 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:12.423+0000 7fa049ffb640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-31T23:21:12.437 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:12.423+0000 7fa049ffb640 1 -- 172.21.15.46:0/3029605179 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0540bd990 con 0x55e2158046c0 2024-03-31T23:21:12.437 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:12.423+0000 7fa049ffb640 1 -- 172.21.15.46:0/3029605179 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 9554+0+0 (secure 0 0 0) 0x7fa0540bb590 con 0x55e2158046c0 2024-03-31T23:21:12.438 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:12.423+0000 7fa06b776740 1 -- 172.21.15.46:0/3029605179 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa05c009dc0 con 0x55e2158046c0 2024-03-31T23:21:12.438 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:12.423+0000 7fa049ffb640 1 -- 172.21.15.46:0/3029605179 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7fa0540bd290 con 0x55e2158046c0 2024-03-31T23:21:12.438 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:12.423+0000 7fa06b776740 10 client.16920 fetch_fsmap learned FSMap version 35 2024-03-31T23:21:12.438 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:12.423+0000 7fa06b776740 1 -- 172.21.15.46:0/3029605179 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x55e21581c8b0 con 0x55e2158046c0 2024-03-31T23:21:12.438 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:12.424+0000 7fa049ffb640 1 -- 172.21.15.46:0/3029605179 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa05c009dc0 con 0x55e2158046c0 2024-03-31T23:21:12.438 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:12.424+0000 7fa049ffb640 1 -- 172.21.15.46:0/3029605179 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 35) ==== 8366+0+0 (secure 0 0 0) 0x7fa0540053d0 con 0x55e2158046c0 2024-03-31T23:21:12.438 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:12.424+0000 7fa06b776740 10 client.16920 fetch_fsmap finished waiting for FSMap version 35 2024-03-31T23:21:12.438 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:12.424+0000 7fa06b776740 10 client.16920 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-31T23:21:12.438 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:12.424+0000 7fa06b776740 20 client.16920 populate_metadata read hostname 'smithi046' 2024-03-31T23:21:12.438 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:12.424+0000 7fa06b776740 1 --2- 172.21.15.46:0/3029605179 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55e21581ddc0 0x55e215820240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:21:12.438 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:12.424+0000 7fa06b776740 4 client.16920 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-31T23:21:12.438 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:12.424+0000 7fa06b776740 1 -- 172.21.15.46:0/3029605179 --> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e21577b120 con 0x55e21581ddc0 2024-03-31T23:21:12.438 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:12.424+0000 7fa061d9b640 1 --2- 172.21.15.46:0/3029605179 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55e21581ddc0 0x55e215820240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:21:12.439 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:12.424+0000 7fa049ffb640 10 client.16920 ms_handle_connect on v2:172.21.15.142:6838/2519143918 2024-03-31T23:21:12.439 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:12.424+0000 7fa061d9b640 1 --2- 172.21.15.46:0/3029605179 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55e21581ddc0 0x55e215820240 crc :-1 s=READY pgs=159 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f20f42494758902d server_cookie=57a10b103d3804c1 in_seq=0 out_seq=0 2024-03-31T23:21:12.439 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:12.425+0000 7fa049ffb640 1 -- 172.21.15.46:0/3029605179 <== mds.0 v2:172.21.15.142:6838/2519143918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55e21577b120 con 0x55e21581ddc0 2024-03-31T23:21:12.439 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:12.425+0000 7fa049ffb640 10 client.16920 handle_command_reply: tid=0 2024-03-31T23:21:12.439 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:12.425+0000 7fa06b776740 1 client.16920 shutdown 2024-03-31T23:21:12.439 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:12.425+0000 7fa06b776740 1 -- 172.21.15.46:0/3029605179 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e2158046c0 msgr2=0x55e21580adf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:21:12.439 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:12.425+0000 7fa06b776740 1 --2- 172.21.15.46:0/3029605179 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e2158046c0 0x55e21580adf0 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7fa054009900 tx=0x7fa054050f90 comp rx=0 tx=0).stop 2024-03-31T23:21:12.439 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:12.426+0000 7fa06b776740 1 -- 172.21.15.46:0/3029605179 shutdown_connections 2024-03-31T23:21:12.439 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:12.426+0000 7fa06b776740 1 --2- 172.21.15.46:0/3029605179 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55e21581ddc0 0x55e215820240 crc :-1 s=READY pgs=159 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:21:12.439 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:12.426+0000 7fa06b776740 1 --2- 172.21.15.46:0/3029605179 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55e21580ddb0 0x55e21580b870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:21:12.439 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:12.426+0000 7fa06b776740 1 --2- 172.21.15.46:0/3029605179 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e215804fe0 0x55e21580b330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:21:12.439 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:12.426+0000 7fa06b776740 1 --2- 172.21.15.46:0/3029605179 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e2158046c0 0x55e21580adf0 unknown :-1 s=CLOSED pgs=796 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:21:12.439 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:12.426+0000 7fa049ffb640 0 client.16920 ms_handle_reset on v2:172.21.15.142:6838/2519143918 2024-03-31T23:21:12.440 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:12.426+0000 7fa06b776740 1 -- 172.21.15.46:0/3029605179 >> 172.21.15.46:0/3029605179 conn(0x55e2157fefe0 msgr2=0x55e215800d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:21:12.440 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:12.426+0000 7fa06b776740 1 -- 172.21.15.46:0/3029605179 shutdown_connections 2024-03-31T23:21:12.440 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:12.426+0000 7fa06b776740 1 -- 172.21.15.46:0/3029605179 wait complete. 2024-03-31T23:21:12.440 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:12.426+0000 7fa06b776740 20 client.16920 trim_cache size 0 max 16384 2024-03-31T23:21:12.445 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:12.440+0000 7fa06b776740 1 Processor -- start 2024-03-31T23:21:12.445 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:12.441+0000 7fa06b776740 1 -- start start 2024-03-31T23:21:12.445 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:12.441+0000 7fa06b776740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e2158046c0 0x55e2157f9e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:21:12.445 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:12.441+0000 7fa06b776740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55e215804fe0 0x55e215816130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:21:12.446 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:12.441+0000 7fa06b776740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e21580ddb0 0x55e215819720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:21:12.446 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:12.441+0000 7fa06b776740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fa0641b1650 con 0x55e2158046c0 2024-03-31T23:21:12.446 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:12.441+0000 7fa06b776740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fa0641aee90 con 0x55e215804fe0 2024-03-31T23:21:12.446 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:12.441+0000 7fa06b776740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fa0641acd00 con 0x55e21580ddb0 2024-03-31T23:21:12.446 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:12.441+0000 7fa06259c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e2158046c0 0x55e2157f9e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:21:12.446 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:12.441+0000 7fa06259c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e2158046c0 0x55e2157f9e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:42034/0 (socket says 172.21.15.46:42034) 2024-03-31T23:21:12.446 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:12.441+0000 7fa06259c640 1 -- 172.21.15.46:0/3916069156 learned_addr learned my addr 172.21.15.46:0/3916069156 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:21:12.446 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:12.441+0000 7fa062d9d640 1 --2- 172.21.15.46:0/3916069156 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e21580ddb0 0x55e215819720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:21:12.446 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:12.441+0000 7fa061d9b640 1 --2- 172.21.15.46:0/3916069156 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55e215804fe0 0x55e215816130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:21:12.446 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:12.441+0000 7fa06259c640 1 -- 172.21.15.46:0/3916069156 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e21580ddb0 msgr2=0x55e215819720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:21:12.446 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:12.441+0000 7fa06259c640 1 --2- 172.21.15.46:0/3916069156 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e21580ddb0 0x55e215819720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:21:12.446 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:12.441+0000 7fa06259c640 1 -- 172.21.15.46:0/3916069156 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55e215804fe0 msgr2=0x55e215816130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:21:12.446 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:12.441+0000 7fa06259c640 1 --2- 172.21.15.46:0/3916069156 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55e215804fe0 0x55e215816130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:21:12.446 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:12.441+0000 7fa06259c640 1 -- 172.21.15.46:0/3916069156 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e2158040a0 con 0x55e2158046c0 2024-03-31T23:21:12.447 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:12.441+0000 7fa06259c640 1 --2- 172.21.15.46:0/3916069156 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e2158046c0 0x55e2157f9e20 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7fa0540bb540 tx=0x7fa0540b9530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=34e329de08176db8 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:21:12.447 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:12.441+0000 7fa04bfff640 1 -- 172.21.15.46:0/3916069156 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0540bcbf0 con 0x55e2158046c0 2024-03-31T23:21:12.447 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:12.442+0000 7fa04bfff640 1 -- 172.21.15.46:0/3916069156 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa0540bd990 con 0x55e2158046c0 2024-03-31T23:21:12.447 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:12.442+0000 7fa04bfff640 1 -- 172.21.15.46:0/3916069156 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0540b9900 con 0x55e2158046c0 2024-03-31T23:21:12.447 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:12.442+0000 7fa06b776740 1 -- 172.21.15.46:0/3916069156 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e2158046c0 msgr2=0x55e2157f9e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:21:12.447 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:12.442+0000 7fa06b776740 1 --2- 172.21.15.46:0/3916069156 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e2158046c0 0x55e2157f9e20 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7fa0540bb540 tx=0x7fa0540b9530 comp rx=0 tx=0).stop 2024-03-31T23:21:12.447 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:12.442+0000 7fa06b776740 1 -- 172.21.15.46:0/3916069156 shutdown_connections 2024-03-31T23:21:12.447 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:12.442+0000 7fa06b776740 1 --2- 172.21.15.46:0/3916069156 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e21580ddb0 0x55e215819720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:21:12.447 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:12.442+0000 7fa06b776740 1 --2- 172.21.15.46:0/3916069156 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55e215804fe0 0x55e215816130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:21:12.447 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:12.442+0000 7fa06b776740 1 --2- 172.21.15.46:0/3916069156 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e2158046c0 0x55e2157f9e20 unknown :-1 s=CLOSED pgs=797 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:21:12.447 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:12.442+0000 7fa06b776740 1 -- 172.21.15.46:0/3916069156 >> 172.21.15.46:0/3916069156 conn(0x55e2157fefe0 msgr2=0x55e2158055f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:21:12.447 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:12.442+0000 7fa06b776740 1 -- 172.21.15.46:0/3916069156 shutdown_connections 2024-03-31T23:21:12.447 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:12.442+0000 7fa06b776740 1 -- 172.21.15.46:0/3916069156 wait complete. 2024-03-31T23:21:12.447 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:12.442+0000 7fa06b776740 1 -- start start 2024-03-31T23:21:12.448 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:12.442+0000 7fa06b776740 1 Processor -- start 2024-03-31T23:21:12.448 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:12.443+0000 7fa06b776740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55e2158046c0 0x55e215808380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:21:12.448 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:12.443+0000 7fa06b776740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e215804fe0 0x55e2158088c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:21:12.448 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:12.443+0000 7fa06b776740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e21580ddb0 0x55e2158070a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:21:12.448 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:12.443+0000 7fa06b776740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fa0641ad350 con 0x55e215804fe0 2024-03-31T23:21:12.448 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:12.443+0000 7fa06b776740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fa0641ae7c0 con 0x55e2158046c0 2024-03-31T23:21:12.448 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:12.443+0000 7fa06b776740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fa0641ada20 con 0x55e21580ddb0 2024-03-31T23:21:12.448 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:12.443+0000 7fa06b776740 10 client.0 _ll_register_callbacks cb 0x7fa054005b50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T23:21:12.448 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:12.443+0000 7fa061d9b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e215804fe0 0x55e2158088c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:21:12.448 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:12.443+0000 7fa061d9b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e215804fe0 0x55e2158088c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:42044/0 (socket says 172.21.15.46:42044) 2024-03-31T23:21:12.449 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:12.443+0000 7fa061d9b640 1 -- 172.21.15.46:0/3916537528 learned_addr learned my addr 172.21.15.46:0/3916537528 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:21:12.449 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:12.443+0000 7fa062d9d640 1 --2- 172.21.15.46:0/3916537528 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e21580ddb0 0x55e2158070a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:21:12.449 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:12.443+0000 7fa06259c640 1 --2- 172.21.15.46:0/3916537528 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55e2158046c0 0x55e215808380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:21:12.449 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:12.443+0000 7fa061d9b640 1 -- 172.21.15.46:0/3916537528 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e21580ddb0 msgr2=0x55e2158070a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:21:12.449 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:12.443+0000 7fa061d9b640 1 --2- 172.21.15.46:0/3916537528 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e21580ddb0 0x55e2158070a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:21:12.449 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:12.443+0000 7fa061d9b640 1 -- 172.21.15.46:0/3916537528 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55e2158046c0 msgr2=0x55e215808380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:21:12.449 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:12.443+0000 7fa061d9b640 1 --2- 172.21.15.46:0/3916537528 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55e2158046c0 0x55e215808380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:21:12.449 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:12.443+0000 7fa061d9b640 1 -- 172.21.15.46:0/3916537528 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e215807790 con 0x55e215804fe0 2024-03-31T23:21:12.449 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:12.443+0000 7fa06259c640 1 --2- 172.21.15.46:0/3916537528 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55e2158046c0 0x55e215808380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T23:21:12.449 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:12.443+0000 7fa061d9b640 1 --2- 172.21.15.46:0/3916537528 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e215804fe0 0x55e2158088c0 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7fa058000c00 tx=0x7fa058010b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b70cd9767d0148b5 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:21:12.449 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:12.443+0000 7fa049ffb640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-31T23:21:12.450 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:12.443+0000 7fa049ffb640 1 -- 172.21.15.46:0/3916537528 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa05801f810 con 0x55e215804fe0 2024-03-31T23:21:12.450 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:12.443+0000 7fa049ffb640 1 -- 172.21.15.46:0/3916537528 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 9554+0+0 (secure 0 0 0) 0x7fa058005020 con 0x55e215804fe0 2024-03-31T23:21:12.450 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:12.443+0000 7fa06b776740 1 -- 172.21.15.46:0/3916537528 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa0540bd990 con 0x55e215804fe0 2024-03-31T23:21:12.450 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:12.444+0000 7fa049ffb640 1 -- 172.21.15.46:0/3916537528 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7fa05802f050 con 0x55e215804fe0 2024-03-31T23:21:12.450 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:12.444+0000 7fa06b776740 10 client.16926 fetch_fsmap learned FSMap version 35 2024-03-31T23:21:12.450 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:12.444+0000 7fa06b776740 1 -- 172.21.15.46:0/3916537528 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x55e215888bb0 con 0x55e215804fe0 2024-03-31T23:21:12.450 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:12.444+0000 7fa049ffb640 1 -- 172.21.15.46:0/3916537528 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa0540bd990 con 0x55e215804fe0 2024-03-31T23:21:12.450 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:12.444+0000 7fa049ffb640 1 -- 172.21.15.46:0/3916537528 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 35) ==== 8366+0+0 (secure 0 0 0) 0x7fa0580047a0 con 0x55e215804fe0 2024-03-31T23:21:12.450 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:12.444+0000 7fa06b776740 10 client.16926 fetch_fsmap finished waiting for FSMap version 35 2024-03-31T23:21:12.450 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:12.444+0000 7fa06b776740 10 client.16926 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-31T23:21:12.451 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:12.444+0000 7fa06b776740 20 client.16926 populate_metadata read hostname 'smithi046' 2024-03-31T23:21:12.451 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:12.444+0000 7fa06b776740 1 --2- 172.21.15.46:0/3916537528 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55e215889390 0x55e21588b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:21:12.451 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:12.444+0000 7fa06b776740 4 client.16926 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-31T23:21:12.451 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:12.444+0000 7fa06b776740 1 -- 172.21.15.46:0/3916537528 --> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55e215774e30 con 0x55e215889390 2024-03-31T23:21:12.451 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:12.444+0000 7fa06259c640 1 --2- 172.21.15.46:0/3916537528 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55e215889390 0x55e21588b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:21:12.451 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:12.445+0000 7fa06259c640 1 --2- 172.21.15.46:0/3916537528 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55e215889390 0x55e21588b7b0 crc :-1 s=READY pgs=160 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1a25e5447346afc6 server_cookie=d762d788d6e0d606 in_seq=0 out_seq=0 2024-03-31T23:21:12.451 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:12.445+0000 7fa049ffb640 10 client.16926 ms_handle_connect on v2:172.21.15.142:6838/2519143918 2024-03-31T23:21:12.466 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:12.465+0000 7fa049ffb640 1 -- 172.21.15.46:0/3916537528 <== mds.0 v2:172.21.15.142:6838/2519143918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55e215774e30 con 0x55e215889390 2024-03-31T23:21:12.467 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:12.465+0000 7fa049ffb640 10 client.16926 handle_command_reply: tid=0 2024-03-31T23:21:12.467 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:12.465+0000 7fa06b776740 1 client.16926 shutdown 2024-03-31T23:21:12.467 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:12.466+0000 7fa06b776740 1 -- 172.21.15.46:0/3916537528 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e215804fe0 msgr2=0x55e2158088c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:21:12.467 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:12.466+0000 7fa06b776740 1 --2- 172.21.15.46:0/3916537528 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e215804fe0 0x55e2158088c0 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7fa058000c00 tx=0x7fa058010b90 comp rx=0 tx=0).stop 2024-03-31T23:21:12.467 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:12.466+0000 7fa06b776740 1 -- 172.21.15.46:0/3916537528 shutdown_connections 2024-03-31T23:21:12.467 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:12.466+0000 7fa06b776740 1 --2- 172.21.15.46:0/3916537528 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55e215889390 0x55e21588b7b0 crc :-1 s=READY pgs=160 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:21:12.467 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:12.466+0000 7fa06b776740 1 --2- 172.21.15.46:0/3916537528 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e21580ddb0 0x55e2158070a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:21:12.467 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:12.466+0000 7fa06b776740 1 --2- 172.21.15.46:0/3916537528 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e215804fe0 0x55e2158088c0 unknown :-1 s=CLOSED pgs=798 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:21:12.467 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:12.466+0000 7fa049ffb640 0 client.16926 ms_handle_reset on v2:172.21.15.142:6838/2519143918 2024-03-31T23:21:12.467 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:12.466+0000 7fa06b776740 1 --2- 172.21.15.46:0/3916537528 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55e2158046c0 0x55e215808380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:21:12.467 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:12.466+0000 7fa06b776740 1 -- 172.21.15.46:0/3916537528 >> 172.21.15.46:0/3916537528 conn(0x55e2157fefe0 msgr2=0x55e21580c4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:21:12.467 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:12.466+0000 7fa06b776740 1 -- 172.21.15.46:0/3916537528 shutdown_connections 2024-03-31T23:21:12.468 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:12.466+0000 7fa06b776740 1 -- 172.21.15.46:0/3916537528 wait complete. 2024-03-31T23:21:12.468 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:12.466+0000 7fa06b776740 20 client.16926 trim_cache size 0 max 16384 2024-03-31T23:21:12.468 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout:{ 2024-03-31T23:21:12.468 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout: "message": "", 2024-03-31T23:21:12.468 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout: "return_code": 0 2024-03-31T23:21:12.468 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout:} 2024-03-31T23:21:12.469 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:12.466+0000 7fa068b99640 1 -- 172.21.15.46:0/3382562470 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fa04c0a26b0 msgr2=0x7fa04c0a4b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:21:12.469 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:12.466+0000 7fa068b99640 1 --2- 172.21.15.46:0/3382562470 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fa04c0a26b0 0x7fa04c0a4b70 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fa05c00fc90 tx=0x7fa05c01e000 comp rx=0 tx=0).stop 2024-03-31T23:21:12.469 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:12.466+0000 7fa068b99640 1 -- 172.21.15.46:0/3382562470 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa064101720 msgr2=0x7fa064197ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:21:12.469 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:12.466+0000 7fa068b99640 1 --2- 172.21.15.46:0/3382562470 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa064101720 0x7fa064197ef0 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7fa054009230 tx=0x7fa05402c5d0 comp rx=0 tx=0).stop 2024-03-31T23:21:12.469 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:12.466+0000 7fa068b99640 1 -- 172.21.15.46:0/3382562470 shutdown_connections 2024-03-31T23:21:12.470 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:12.466+0000 7fa068b99640 1 --2- 172.21.15.46:0/3382562470 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fa04c0a26b0 0x7fa04c0a4b70 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:21:12.470 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:12.466+0000 7fa068b99640 1 --2- 172.21.15.46:0/3382562470 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa064107730 0x7fa06419c760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:21:12.470 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:12.466+0000 7fa068b99640 1 --2- 172.21.15.46:0/3382562470 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa0641020e0 0x7fa064198430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:21:12.470 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:12.466+0000 7fa068b99640 1 --2- 172.21.15.46:0/3382562470 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa064101720 0x7fa064197ef0 unknown :-1 s=CLOSED pgs=794 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:21:12.470 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:12.466+0000 7fa068b99640 1 -- 172.21.15.46:0/3382562470 >> 172.21.15.46:0/3382562470 conn(0x7fa0640fd230 msgr2=0x7fa0640fd9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:21:12.470 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:12.467+0000 7fa068b99640 1 -- 172.21.15.46:0/3382562470 shutdown_connections 2024-03-31T23:21:12.470 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:12.467+0000 7fa068b99640 1 -- 172.21.15.46:0/3382562470 wait complete. 2024-03-31T23:21:13.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:21:12 smithi162 ceph-mon[36237]: pgmap v295: 129 pgs: 129 active+clean; 113 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 373 KiB/s rd, 1005 KiB/s wr, 117 op/s 2024-03-31T23:21:13.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:21:12 smithi046 ceph-mon[32498]: pgmap v295: 129 pgs: 129 active+clean; 113 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 373 KiB/s rd, 1005 KiB/s wr, 117 op/s 2024-03-31T23:21:13.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:21:12 smithi142 ceph-mon[35271]: pgmap v295: 129 pgs: 129 active+clean; 113 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 373 KiB/s rd, 1005 KiB/s wr, 117 op/s 2024-03-31T23:21:13.209 INFO:tasks.workunit.client.0.smithi046.stdout: 1 198916 2.07 MB/sec execute 379 sec latency 4764.567 ms 2024-03-31T23:21:14.210 INFO:tasks.workunit.client.0.smithi046.stdout: 1 199179 2.06 MB/sec execute 380 sec latency 14.865 ms 2024-03-31T23:21:15.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:21:14 smithi162 ceph-mon[36237]: pgmap v296: 129 pgs: 129 active+clean; 113 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 303 KiB/s rd, 816 KiB/s wr, 95 op/s 2024-03-31T23:21:15.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:21:14 smithi046 ceph-mon[32498]: pgmap v296: 129 pgs: 129 active+clean; 113 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 303 KiB/s rd, 816 KiB/s wr, 95 op/s 2024-03-31T23:21:15.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:21:14 smithi142 ceph-mon[35271]: pgmap v296: 129 pgs: 129 active+clean; 113 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 303 KiB/s rd, 816 KiB/s wr, 95 op/s 2024-03-31T23:21:15.210 INFO:tasks.workunit.client.0.smithi046.stdout: 1 199576 2.06 MB/sec execute 381 sec latency 17.502 ms 2024-03-31T23:21:16.210 INFO:tasks.workunit.client.0.smithi046.stdout: 1 200049 2.06 MB/sec execute 382 sec latency 20.091 ms 2024-03-31T23:21:17.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:21:16 smithi162 ceph-mon[36237]: pgmap v297: 129 pgs: 129 active+clean; 114 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 133 KiB/s rd, 299 KiB/s wr, 38 op/s 2024-03-31T23:21:17.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:21:16 smithi046 ceph-mon[32498]: pgmap v297: 129 pgs: 129 active+clean; 114 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 133 KiB/s rd, 299 KiB/s wr, 38 op/s 2024-03-31T23:21:17.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:21:16 smithi142 ceph-mon[35271]: pgmap v297: 129 pgs: 129 active+clean; 114 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 133 KiB/s rd, 299 KiB/s wr, 38 op/s 2024-03-31T23:21:17.210 INFO:tasks.workunit.client.0.smithi046.stdout: 1 200508 2.06 MB/sec execute 383 sec latency 22.693 ms 2024-03-31T23:21:18.210 INFO:tasks.workunit.client.0.smithi046.stdout: 1 201037 2.06 MB/sec execute 384 sec latency 21.955 ms 2024-03-31T23:21:19.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:21:18 smithi162 ceph-mon[36237]: pgmap v298: 129 pgs: 129 active+clean; 114 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 133 KiB/s rd, 299 KiB/s wr, 38 op/s 2024-03-31T23:21:19.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:21:18 smithi046 ceph-mon[32498]: pgmap v298: 129 pgs: 129 active+clean; 114 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 133 KiB/s rd, 299 KiB/s wr, 38 op/s 2024-03-31T23:21:19.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:21:18 smithi142 ceph-mon[35271]: pgmap v298: 129 pgs: 129 active+clean; 114 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 133 KiB/s rd, 299 KiB/s wr, 38 op/s 2024-03-31T23:21:19.210 INFO:tasks.workunit.client.0.smithi046.stdout: 1 201614 2.07 MB/sec execute 385 sec latency 17.828 ms 2024-03-31T23:21:20.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:21:19 smithi162 ceph-mon[36237]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-03-31T23:21:20.094 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:21:19 smithi142 ceph-mon[35271]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-03-31T23:21:20.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:21:19 smithi046 ceph-mon[32498]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-03-31T23:21:20.210 INFO:tasks.workunit.client.0.smithi046.stdout: 1 202001 2.07 MB/sec execute 386 sec latency 16.292 ms 2024-03-31T23:21:21.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:21:20 smithi162 ceph-mon[36237]: pgmap v299: 129 pgs: 129 active+clean; 115 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 342 KiB/s rd, 876 KiB/s wr, 90 op/s 2024-03-31T23:21:21.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:21:20 smithi046 ceph-mon[32498]: pgmap v299: 129 pgs: 129 active+clean; 115 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 342 KiB/s rd, 876 KiB/s wr, 90 op/s 2024-03-31T23:21:21.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:21:20 smithi142 ceph-mon[35271]: pgmap v299: 129 pgs: 129 active+clean; 115 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 342 KiB/s rd, 876 KiB/s wr, 90 op/s 2024-03-31T23:21:21.210 INFO:tasks.workunit.client.0.smithi046.stdout: 1 202255 2.07 MB/sec execute 387 sec latency 530.321 ms 2024-03-31T23:21:22.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:21:21 smithi162 ceph-mon[36237]: pgmap v300: 129 pgs: 129 active+clean; 115 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 331 KiB/s rd, 849 KiB/s wr, 87 op/s 2024-03-31T23:21:22.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:21:21 smithi046 ceph-mon[32498]: pgmap v300: 129 pgs: 129 active+clean; 115 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 331 KiB/s rd, 849 KiB/s wr, 87 op/s 2024-03-31T23:21:22.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:21:21 smithi142 ceph-mon[35271]: pgmap v300: 129 pgs: 129 active+clean; 115 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 331 KiB/s rd, 849 KiB/s wr, 87 op/s 2024-03-31T23:21:22.210 INFO:tasks.workunit.client.0.smithi046.stdout: 1 202255 2.06 MB/sec execute 388 sec latency 1530.406 ms 2024-03-31T23:21:23.210 INFO:tasks.workunit.client.0.smithi046.stdout: 1 202267 2.06 MB/sec execute 389 sec latency 2063.465 ms 2024-03-31T23:21:24.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:21:23 smithi142 ceph-mon[35271]: pgmap v301: 129 pgs: 129 active+clean; 115 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 285 KiB/s rd, 730 KiB/s wr, 75 op/s 2024-03-31T23:21:24.210 INFO:tasks.workunit.client.0.smithi046.stdout: 1 202267 2.05 MB/sec execute 390 sec latency 1407.994 ms 2024-03-31T23:21:24.296 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:21:23 smithi162 ceph-mon[36237]: pgmap v301: 129 pgs: 129 active+clean; 115 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 285 KiB/s rd, 730 KiB/s wr, 75 op/s 2024-03-31T23:21:24.362 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:21:23 smithi046 ceph-mon[32498]: pgmap v301: 129 pgs: 129 active+clean; 115 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 285 KiB/s rd, 730 KiB/s wr, 75 op/s 2024-03-31T23:21:25.210 INFO:tasks.workunit.client.0.smithi046.stdout: 1 202267 2.05 MB/sec execute 391 sec latency 2408.098 ms 2024-03-31T23:21:25.565 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:25.563+0000 7f16cd27c640 1 -- 172.21.15.46:0/2939993047 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f16c8101700 msgr2=0x7f16c8101b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:21:25.565 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:25.563+0000 7f16cd27c640 1 --2- 172.21.15.46:0/2939993047 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f16c8101700 0x7f16c8101b80 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7f16bc009930 tx=0x7f16bc02bc30 comp rx=0 tx=0).stop 2024-03-31T23:21:25.565 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:25.563+0000 7f16cd27c640 1 -- 172.21.15.46:0/2939993047 shutdown_connections 2024-03-31T23:21:25.565 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:25.563+0000 7f16cd27c640 1 --2- 172.21.15.46:0/2939993047 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f16c81020c0 0x7f16c810e5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:21:25.565 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:25.563+0000 7f16cd27c640 1 --2- 172.21.15.46:0/2939993047 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f16c8101700 0x7f16c8101b80 unknown :-1 s=CLOSED pgs=799 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:21:25.566 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:25.563+0000 7f16cd27c640 1 --2- 172.21.15.46:0/2939993047 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f16c8107710 0x7f16c8107af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:21:25.566 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:25.563+0000 7f16cd27c640 1 -- 172.21.15.46:0/2939993047 >> 172.21.15.46:0/2939993047 conn(0x7f16c80fd210 msgr2=0x7f16c80ff630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:21:25.566 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:25.563+0000 7f16cd27c640 1 -- 172.21.15.46:0/2939993047 shutdown_connections 2024-03-31T23:21:25.566 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:25.563+0000 7f16cd27c640 1 -- 172.21.15.46:0/2939993047 wait complete. 2024-03-31T23:21:25.566 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:25.564+0000 7f16cd27c640 1 Processor -- start 2024-03-31T23:21:25.566 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:25.564+0000 7f16cd27c640 1 -- start start 2024-03-31T23:21:25.568 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:25.564+0000 7f16cd27c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f16c8101700 0x7f16c8197f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:21:25.568 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:25.564+0000 7f16cd27c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f16c81020c0 0x7f16c8198470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:21:25.568 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:25.564+0000 7f16cd27c640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f16c8107710 0x7f16c819c7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:21:25.568 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:25.564+0000 7f16cd27c640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f16c81119d0 con 0x7f16c81020c0 2024-03-31T23:21:25.568 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:25.564+0000 7f16cd27c640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f16c8111850 con 0x7f16c8107710 2024-03-31T23:21:25.568 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:25.564+0000 7f16cd27c640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f16c8111b50 con 0x7f16c8101700 2024-03-31T23:21:25.568 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:25.565+0000 7f16c67fc640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f16c81020c0 0x7f16c8198470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:21:25.568 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:25.565+0000 7f16c67fc640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f16c81020c0 0x7f16c8198470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:56122/0 (socket says 172.21.15.46:56122) 2024-03-31T23:21:25.568 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:25.565+0000 7f16c67fc640 1 -- 172.21.15.46:0/419860184 learned_addr learned my addr 172.21.15.46:0/419860184 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:21:25.568 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:25.565+0000 7f16c77fe640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f16c8107710 0x7f16c819c7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:21:25.569 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:25.565+0000 7f16c6ffd640 1 --2- 172.21.15.46:0/419860184 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f16c8101700 0x7f16c8197f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:21:25.569 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:25.565+0000 7f16c67fc640 1 -- 172.21.15.46:0/419860184 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f16c8101700 msgr2=0x7f16c8197f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:21:25.569 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:25.565+0000 7f16c67fc640 1 --2- 172.21.15.46:0/419860184 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f16c8101700 0x7f16c8197f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:21:25.569 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:25.565+0000 7f16c67fc640 1 -- 172.21.15.46:0/419860184 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f16c8107710 msgr2=0x7f16c819c7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:21:25.569 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:25.565+0000 7f16c67fc640 1 --2- 172.21.15.46:0/419860184 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f16c8107710 0x7f16c819c7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:21:25.569 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:25.565+0000 7f16c67fc640 1 -- 172.21.15.46:0/419860184 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f16c819cf20 con 0x7f16c81020c0 2024-03-31T23:21:25.569 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:25.565+0000 7f16c77fe640 1 --2- 172.21.15.46:0/419860184 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f16c8107710 0x7f16c819c7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:21:25.569 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:25.565+0000 7f16c67fc640 1 --2- 172.21.15.46:0/419860184 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f16c81020c0 0x7f16c8198470 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7f16bc009230 tx=0x7f16bc02c5d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:21:25.569 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:25.565+0000 7f16affff640 1 -- 172.21.15.46:0/419860184 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16bc043020 con 0x7f16c81020c0 2024-03-31T23:21:25.569 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:25.565+0000 7f16affff640 1 -- 172.21.15.46:0/419860184 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f16bc02ca70 con 0x7f16c81020c0 2024-03-31T23:21:25.570 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:25.565+0000 7f16cd27c640 1 -- 172.21.15.46:0/419860184 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f16c819d1b0 con 0x7f16c81020c0 2024-03-31T23:21:25.570 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:25.565+0000 7f16affff640 1 -- 172.21.15.46:0/419860184 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16bc003a50 con 0x7f16c81020c0 2024-03-31T23:21:25.570 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:25.565+0000 7f16cd27c640 1 -- 172.21.15.46:0/419860184 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f16c81a4aa0 con 0x7f16c81020c0 2024-03-31T23:21:25.576 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:25.568+0000 7f16affff640 1 -- 172.21.15.46:0/419860184 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 25) ==== 137451+0+0 (secure 0 0 0) 0x7f16bc046020 con 0x7f16c81020c0 2024-03-31T23:21:25.576 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:25.568+0000 7f16cfe59740 1 Processor -- start 2024-03-31T23:21:25.576 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:25.568+0000 7f16cfe59740 1 -- start start 2024-03-31T23:21:25.576 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:25.568+0000 7f16affff640 1 --2- 172.21.15.46:0/419860184 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f16b00a2730 0x7f16b00a4bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:21:25.576 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:25.568+0000 7f16cfe59740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556b62ef61e0 0x556b62ef65c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:21:25.576 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:25.568+0000 7f16cfe59740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x556b62ef6b00 0x556b62f03c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:21:25.576 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:25.568+0000 7f16cfe59740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556b62f041c0 0x556b62f065b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:21:25.576 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:25.568+0000 7f16cfe59740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x556b62e598f0 con 0x556b62f041c0 2024-03-31T23:21:25.576 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:25.568+0000 7f16cfe59740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x556b62e028d0 con 0x556b62ef6b00 2024-03-31T23:21:25.576 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:25.568+0000 7f16cfe59740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x556b62e55990 con 0x556b62ef61e0 2024-03-31T23:21:25.576 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:25.568+0000 7f16c77fe640 1 --2- 172.21.15.46:0/419860184 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f16b00a2730 0x7f16b00a4bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:21:25.576 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:25.568+0000 7f16affff640 1 -- 172.21.15.46:0/419860184 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 9554+0+0 (secure 0 0 0) 0x7f16bc0f0db0 con 0x7f16c81020c0 2024-03-31T23:21:25.577 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:25.568+0000 7f16c77fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556b62f041c0 0x556b62f065b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:21:25.577 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:25.568+0000 7f16c77fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556b62f041c0 0x556b62f065b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:56126/0 (socket says 172.21.15.46:56126) 2024-03-31T23:21:25.577 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:25.569+0000 7f16c67fc640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x556b62ef6b00 0x556b62f03c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:21:25.577 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:25.569+0000 7f16c77fe640 1 -- 172.21.15.46:0/3913076498 learned_addr learned my addr 172.21.15.46:0/3913076498 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:21:25.577 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:25.569+0000 7f16c6ffd640 1 --2- 172.21.15.46:0/3913076498 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556b62ef61e0 0x556b62ef65c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:21:25.577 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:25.569+0000 7f16c67fc640 1 -- 172.21.15.46:0/3913076498 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556b62ef61e0 msgr2=0x556b62ef65c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:21:25.577 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:25.569+0000 7f16c67fc640 1 --2- 172.21.15.46:0/3913076498 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556b62ef61e0 0x556b62ef65c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:21:25.577 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:25.569+0000 7f16c67fc640 1 -- 172.21.15.46:0/3913076498 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556b62f041c0 msgr2=0x556b62f065b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:21:25.577 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:25.569+0000 7f16c67fc640 1 --2- 172.21.15.46:0/3913076498 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556b62f041c0 0x556b62f065b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:21:25.577 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:25.569+0000 7f16c67fc640 1 -- 172.21.15.46:0/3913076498 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556b62b6e210 con 0x556b62ef6b00 2024-03-31T23:21:25.577 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:25.569+0000 7f16c77fe640 1 --2- 172.21.15.46:0/419860184 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f16b00a2730 0x7f16b00a4bf0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f16c000f480 tx=0x7f16c00117c0 comp rx=0 tx=0).ready entity=mgr.15654 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:21:25.577 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:25.569+0000 7f16c67fc640 1 --2- 172.21.15.46:0/3913076498 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x556b62ef6b00 0x556b62f03c80 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f16bc0ee200 tx=0x7f16bc0b9600 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5dd67d22e20a7bf8 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:21:25.577 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:25.569+0000 7f16ad7fa640 1 -- 172.21.15.46:0/3913076498 <== mon.2 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16bc0dd0b0 con 0x556b62ef6b00 2024-03-31T23:21:25.577 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:25.569+0000 7f16ad7fa640 1 -- 172.21.15.46:0/3913076498 <== mon.2 v2:172.21.15.142:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f16bc0bd930 con 0x556b62ef6b00 2024-03-31T23:21:25.578 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:25.569+0000 7f16ad7fa640 1 -- 172.21.15.46:0/3913076498 <== mon.2 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16bc0ee370 con 0x556b62ef6b00 2024-03-31T23:21:25.578 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:25.570+0000 7f16cfe59740 1 -- 172.21.15.46:0/3913076498 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x556b62ef6b00 msgr2=0x556b62f03c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:21:25.578 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:25.570+0000 7f16cfe59740 1 --2- 172.21.15.46:0/3913076498 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x556b62ef6b00 0x556b62f03c80 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f16bc0ee200 tx=0x7f16bc0b9600 comp rx=0 tx=0).stop 2024-03-31T23:21:25.578 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:25.570+0000 7f16cfe59740 1 -- 172.21.15.46:0/3913076498 shutdown_connections 2024-03-31T23:21:25.578 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:25.570+0000 7f16cfe59740 1 --2- 172.21.15.46:0/3913076498 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556b62f041c0 0x556b62f065b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:21:25.578 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:25.570+0000 7f16cfe59740 1 --2- 172.21.15.46:0/3913076498 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x556b62ef6b00 0x556b62f03c80 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:21:25.578 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:25.570+0000 7f16cfe59740 1 --2- 172.21.15.46:0/3913076498 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556b62ef61e0 0x556b62ef65c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:21:25.578 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:25.570+0000 7f16cfe59740 1 -- 172.21.15.46:0/3913076498 >> 172.21.15.46:0/3913076498 conn(0x556b62ef0fe0 msgr2=0x556b62ef13c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:21:25.578 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:25.570+0000 7f16cfe59740 1 -- 172.21.15.46:0/3913076498 shutdown_connections 2024-03-31T23:21:25.578 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:25.570+0000 7f16cfe59740 1 -- 172.21.15.46:0/3913076498 wait complete. 2024-03-31T23:21:25.578 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:25.570+0000 7f16cfe59740 1 -- start start 2024-03-31T23:21:25.578 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:25.570+0000 7f16cfe59740 1 Processor -- start 2024-03-31T23:21:25.578 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:25.571+0000 7f16cfe59740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556b62ef61e0 0x556b62f01240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:21:25.579 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:25.571+0000 7f16cfe59740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x556b62ef6b00 0x556b62f01780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:21:25.579 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:25.571+0000 7f16cfe59740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556b62f041c0 0x556b62f01cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:21:25.579 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:25.571+0000 7f16cfe59740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x556b62f06b40 con 0x556b62f041c0 2024-03-31T23:21:25.579 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:25.571+0000 7f16cfe59740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x556b62df7f80 con 0x556b62ef6b00 2024-03-31T23:21:25.579 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:25.571+0000 7f16cfe59740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x556b62f06cc0 con 0x556b62ef61e0 2024-03-31T23:21:25.579 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:25.571+0000 7f16cfe59740 10 client.0 _ll_register_callbacks cb 0x556b62e50290 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T23:21:25.579 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:25.571+0000 7f16c77fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556b62f041c0 0x556b62f01cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:21:25.579 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:25.571+0000 7f16c77fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556b62f041c0 0x556b62f01cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:56130/0 (socket says 172.21.15.46:56130) 2024-03-31T23:21:25.579 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:25.571+0000 7f16c77fe640 1 -- 172.21.15.46:0/4199226344 learned_addr learned my addr 172.21.15.46:0/4199226344 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:21:25.579 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:25.571+0000 7f16c77fe640 1 -- 172.21.15.46:0/4199226344 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556b62ef61e0 msgr2=0x556b62f01240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:21:25.579 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:25.571+0000 7f16c77fe640 1 --2- 172.21.15.46:0/4199226344 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556b62ef61e0 0x556b62f01240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:21:25.579 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:25.571+0000 7f16c77fe640 1 -- 172.21.15.46:0/4199226344 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x556b62ef6b00 msgr2=0x556b62f01780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:21:25.579 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:25.571+0000 7f16c77fe640 1 --2- 172.21.15.46:0/4199226344 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x556b62ef6b00 0x556b62f01780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:21:25.580 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:25.571+0000 7f16c77fe640 1 -- 172.21.15.46:0/4199226344 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556b62c3a620 con 0x556b62f041c0 2024-03-31T23:21:25.580 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:25.571+0000 7f16c77fe640 1 --2- 172.21.15.46:0/4199226344 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556b62f041c0 0x556b62f01cc0 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7f16c0003e40 tx=0x7f16c001d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2bf906b1a4697436 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:21:25.580 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:25.571+0000 7f16aaff5640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-31T23:21:25.580 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:25.571+0000 7f16aaff5640 1 -- 172.21.15.46:0/4199226344 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16c0024070 con 0x556b62f041c0 2024-03-31T23:21:25.580 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:25.571+0000 7f16aaff5640 1 -- 172.21.15.46:0/4199226344 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 9554+0+0 (secure 0 0 0) 0x7f16c0004b20 con 0x556b62f041c0 2024-03-31T23:21:25.580 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:25.571+0000 7f16cfe59740 1 -- 172.21.15.46:0/4199226344 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f16bc0bd930 con 0x556b62f041c0 2024-03-31T23:21:25.580 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:25.572+0000 7f16aaff5640 1 -- 172.21.15.46:0/4199226344 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f16c0033050 con 0x556b62f041c0 2024-03-31T23:21:25.580 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:25.572+0000 7f16cfe59740 10 client.16938 fetch_fsmap learned FSMap version 36 2024-03-31T23:21:25.580 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:25.572+0000 7f16cfe59740 1 -- 172.21.15.46:0/4199226344 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x556b62f12a70 con 0x556b62f041c0 2024-03-31T23:21:25.580 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:25.572+0000 7f16aaff5640 1 -- 172.21.15.46:0/4199226344 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f16bc0bd930 con 0x556b62f041c0 2024-03-31T23:21:25.580 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:25.572+0000 7f16aaff5640 1 -- 172.21.15.46:0/4199226344 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 36) ==== 8362+0+0 (secure 0 0 0) 0x7f16c0005020 con 0x556b62f041c0 2024-03-31T23:21:25.580 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:25.572+0000 7f16cfe59740 10 client.16938 fetch_fsmap finished waiting for FSMap version 36 2024-03-31T23:21:25.580 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:25.572+0000 7f16cfe59740 10 client.16938 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-31T23:21:25.580 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:25.572+0000 7f16cfe59740 20 client.16938 populate_metadata read hostname 'smithi046' 2024-03-31T23:21:25.581 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:25.572+0000 7f16cfe59740 1 --2- 172.21.15.46:0/4199226344 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x556b62f13fa0 0x556b62f16430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:21:25.581 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:25.572+0000 7f16cfe59740 4 client.16938 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-31T23:21:25.581 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:25.572+0000 7f16cfe59740 1 -- 172.21.15.46:0/4199226344 --> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556b62e6d120 con 0x556b62f13fa0 2024-03-31T23:21:25.581 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:25.572+0000 7f16c6ffd640 1 --2- 172.21.15.46:0/4199226344 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x556b62f13fa0 0x556b62f16430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:21:25.581 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:25.573+0000 7f16c6ffd640 1 --2- 172.21.15.46:0/4199226344 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x556b62f13fa0 0x556b62f16430 crc :-1 s=READY pgs=161 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1e2d86938506c1be server_cookie=74dd5330898d9da3 in_seq=0 out_seq=0 2024-03-31T23:21:25.581 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:25.573+0000 7f16aaff5640 10 client.16938 ms_handle_connect on v2:172.21.15.142:6838/2519143918 2024-03-31T23:21:25.581 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:25.574+0000 7f16aaff5640 1 -- 172.21.15.46:0/4199226344 <== mds.0 v2:172.21.15.142:6838/2519143918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x556b62e6d120 con 0x556b62f13fa0 2024-03-31T23:21:25.581 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:25.574+0000 7f16aaff5640 10 client.16938 handle_command_reply: tid=0 2024-03-31T23:21:25.581 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:25.574+0000 7f16cfe59740 1 client.16938 shutdown 2024-03-31T23:21:25.581 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:25.574+0000 7f16cfe59740 1 -- 172.21.15.46:0/4199226344 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556b62f041c0 msgr2=0x556b62f01cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:21:25.581 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:25.574+0000 7f16cfe59740 1 --2- 172.21.15.46:0/4199226344 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556b62f041c0 0x556b62f01cc0 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7f16c0003e40 tx=0x7f16c001d000 comp rx=0 tx=0).stop 2024-03-31T23:21:25.581 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:25.574+0000 7f16cfe59740 1 -- 172.21.15.46:0/4199226344 shutdown_connections 2024-03-31T23:21:25.581 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:25.574+0000 7f16cfe59740 1 --2- 172.21.15.46:0/4199226344 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x556b62f13fa0 0x556b62f16430 crc :-1 s=READY pgs=161 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:21:25.581 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:25.574+0000 7f16cfe59740 1 --2- 172.21.15.46:0/4199226344 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556b62f041c0 0x556b62f01cc0 unknown :-1 s=CLOSED pgs=801 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:21:25.582 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:25.574+0000 7f16cfe59740 1 --2- 172.21.15.46:0/4199226344 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x556b62ef6b00 0x556b62f01780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:21:25.582 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:25.574+0000 7f16cfe59740 1 --2- 172.21.15.46:0/4199226344 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556b62ef61e0 0x556b62f01240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:21:25.582 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:25.574+0000 7f16aaff5640 0 client.16938 ms_handle_reset on v2:172.21.15.142:6838/2519143918 2024-03-31T23:21:25.582 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:25.574+0000 7f16cfe59740 1 -- 172.21.15.46:0/4199226344 >> 172.21.15.46:0/4199226344 conn(0x556b62ef0fe0 msgr2=0x556b62ef2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:21:25.582 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:25.574+0000 7f16cfe59740 1 -- 172.21.15.46:0/4199226344 shutdown_connections 2024-03-31T23:21:25.582 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:25.574+0000 7f16cfe59740 1 -- 172.21.15.46:0/4199226344 wait complete. 2024-03-31T23:21:25.582 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:25.574+0000 7f16cfe59740 20 client.16938 trim_cache size 0 max 16384 2024-03-31T23:21:25.594 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:25.589+0000 7f16cfe59740 1 Processor -- start 2024-03-31T23:21:25.595 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:25.589+0000 7f16cfe59740 1 -- start start 2024-03-31T23:21:25.595 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:25.589+0000 7f16cfe59740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x556b62ef61e0 0x556b62f03970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:21:25.595 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:25.589+0000 7f16cfe59740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556b62ef6b00 0x556b62eebe20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:21:25.595 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:25.589+0000 7f16cfe59740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556b62f041c0 0x556b62f0f8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:21:25.595 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:25.589+0000 7f16cfe59740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f16c81af310 con 0x556b62ef6b00 2024-03-31T23:21:25.595 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:25.589+0000 7f16cfe59740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f16c81acb80 con 0x556b62ef61e0 2024-03-31T23:21:25.595 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:25.589+0000 7f16cfe59740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f16c81aa9f0 con 0x556b62f041c0 2024-03-31T23:21:25.595 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:25.589+0000 7f16c67fc640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556b62ef6b00 0x556b62eebe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:21:25.595 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:25.589+0000 7f16c67fc640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556b62ef6b00 0x556b62eebe20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:56134/0 (socket says 172.21.15.46:56134) 2024-03-31T23:21:25.595 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:25.589+0000 7f16c67fc640 1 -- 172.21.15.46:0/1156003931 learned_addr learned my addr 172.21.15.46:0/1156003931 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:21:25.595 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:25.589+0000 7f16c6ffd640 1 --2- 172.21.15.46:0/1156003931 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x556b62ef61e0 0x556b62f03970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:21:25.595 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:25.589+0000 7f16c77fe640 1 --2- 172.21.15.46:0/1156003931 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556b62f041c0 0x556b62f0f8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:21:25.596 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:25.589+0000 7f16c67fc640 1 -- 172.21.15.46:0/1156003931 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556b62f041c0 msgr2=0x556b62f0f8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:21:25.596 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:25.589+0000 7f16c67fc640 1 --2- 172.21.15.46:0/1156003931 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556b62f041c0 0x556b62f0f8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:21:25.596 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:25.589+0000 7f16c67fc640 1 -- 172.21.15.46:0/1156003931 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x556b62ef61e0 msgr2=0x556b62f03970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:21:25.596 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:25.589+0000 7f16c67fc640 1 --2- 172.21.15.46:0/1156003931 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x556b62ef61e0 0x556b62f03970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:21:25.596 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:25.589+0000 7f16c67fc640 1 -- 172.21.15.46:0/1156003931 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556b62f001f0 con 0x556b62ef6b00 2024-03-31T23:21:25.596 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:25.589+0000 7f16c6ffd640 1 --2- 172.21.15.46:0/1156003931 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x556b62ef61e0 0x556b62f03970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:21:25.596 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:25.590+0000 7f16c67fc640 1 --2- 172.21.15.46:0/1156003931 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556b62ef6b00 0x556b62eebe20 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7f16bc0ee2b0 tx=0x7f16bc02bc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c9e1de6af01e6b80 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:21:25.596 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:25.590+0000 7f16acff9640 1 -- 172.21.15.46:0/1156003931 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16bc0c1380 con 0x556b62ef6b00 2024-03-31T23:21:25.596 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:25.590+0000 7f16acff9640 1 -- 172.21.15.46:0/1156003931 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f16bc0dd0b0 con 0x556b62ef6b00 2024-03-31T23:21:25.596 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:25.590+0000 7f16acff9640 1 -- 172.21.15.46:0/1156003931 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16bc0053d0 con 0x556b62ef6b00 2024-03-31T23:21:25.596 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:25.590+0000 7f16cfe59740 1 -- 172.21.15.46:0/1156003931 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556b62ef6b00 msgr2=0x556b62eebe20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:21:25.596 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:25.590+0000 7f16cfe59740 1 --2- 172.21.15.46:0/1156003931 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556b62ef6b00 0x556b62eebe20 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7f16bc0ee2b0 tx=0x7f16bc02bc30 comp rx=0 tx=0).stop 2024-03-31T23:21:25.596 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:25.590+0000 7f16cfe59740 1 -- 172.21.15.46:0/1156003931 shutdown_connections 2024-03-31T23:21:25.597 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:25.590+0000 7f16cfe59740 1 --2- 172.21.15.46:0/1156003931 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556b62f041c0 0x556b62f0f8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:21:25.597 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:25.590+0000 7f16cfe59740 1 --2- 172.21.15.46:0/1156003931 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556b62ef6b00 0x556b62eebe20 unknown :-1 s=CLOSED pgs=802 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:21:25.597 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:25.590+0000 7f16cfe59740 1 --2- 172.21.15.46:0/1156003931 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x556b62ef61e0 0x556b62f03970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:21:25.597 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:25.590+0000 7f16cfe59740 1 -- 172.21.15.46:0/1156003931 >> 172.21.15.46:0/1156003931 conn(0x556b62ef0fe0 msgr2=0x556b62eeff80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:21:25.597 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:25.590+0000 7f16cfe59740 1 -- 172.21.15.46:0/1156003931 shutdown_connections 2024-03-31T23:21:25.597 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:25.590+0000 7f16cfe59740 1 -- 172.21.15.46:0/1156003931 wait complete. 2024-03-31T23:21:25.597 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:25.591+0000 7f16cfe59740 1 -- start start 2024-03-31T23:21:25.597 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:25.591+0000 7f16cfe59740 1 Processor -- start 2024-03-31T23:21:25.597 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:25.591+0000 7f16cfe59740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556b62ef61e0 0x556b62efe420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:21:25.597 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:25.591+0000 7f16cfe59740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556b62ef6b00 0x556b62efe960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:21:25.597 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:25.591+0000 7f16cfe59740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x556b62f041c0 0x556b62f79370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:21:25.597 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:25.591+0000 7f16cfe59740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f16c81ab040 con 0x556b62ef6b00 2024-03-31T23:21:25.597 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:25.591+0000 7f16cfe59740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f16c81ac4b0 con 0x556b62f041c0 2024-03-31T23:21:25.597 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:25.591+0000 7f16cfe59740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f16c81ab710 con 0x556b62ef61e0 2024-03-31T23:21:25.598 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:25.591+0000 7f16cfe59740 10 client.0 _ll_register_callbacks cb 0x7f16c0003d50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T23:21:25.598 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:25.591+0000 7f16c67fc640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556b62ef6b00 0x556b62efe960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:21:25.598 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:25.591+0000 7f16c6ffd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556b62ef61e0 0x556b62efe420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:21:25.598 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:25.591+0000 7f16c67fc640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556b62ef6b00 0x556b62efe960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:56142/0 (socket says 172.21.15.46:56142) 2024-03-31T23:21:25.598 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:25.591+0000 7f16c77fe640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x556b62f041c0 0x556b62f79370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:21:25.598 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:25.591+0000 7f16c67fc640 1 -- 172.21.15.46:0/4265707058 learned_addr learned my addr 172.21.15.46:0/4265707058 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:21:25.598 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:25.591+0000 7f16c6ffd640 1 -- 172.21.15.46:0/4265707058 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x556b62f041c0 msgr2=0x556b62f79370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:21:25.598 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:25.591+0000 7f16c6ffd640 1 --2- 172.21.15.46:0/4265707058 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x556b62f041c0 0x556b62f79370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:21:25.598 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:25.592+0000 7f16c6ffd640 1 -- 172.21.15.46:0/4265707058 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556b62ef6b00 msgr2=0x556b62efe960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:21:25.598 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:25.592+0000 7f16c6ffd640 1 --2- 172.21.15.46:0/4265707058 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556b62ef6b00 0x556b62efe960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:21:25.598 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:25.592+0000 7f16c6ffd640 1 -- 172.21.15.46:0/4265707058 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556b62f798b0 con 0x556b62ef61e0 2024-03-31T23:21:25.598 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:25.592+0000 7f16c67fc640 1 --2- 172.21.15.46:0/4265707058 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556b62ef6b00 0x556b62efe960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:21:25.599 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:25.592+0000 7f16c77fe640 1 --2- 172.21.15.46:0/4265707058 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x556b62f041c0 0x556b62f79370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:21:25.599 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:25.592+0000 7f16c6ffd640 1 --2- 172.21.15.46:0/4265707058 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556b62ef61e0 0x556b62efe420 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f16b80058c0 tx=0x7f16b8010800 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e33a892aa3202100 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:21:25.599 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:25.592+0000 7f16aaff5640 10 client.0 ms_handle_connect on v2:172.21.15.162:3300/0 2024-03-31T23:21:25.599 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:25.592+0000 7f16aaff5640 1 -- 172.21.15.46:0/4265707058 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16b801de60 con 0x556b62ef61e0 2024-03-31T23:21:25.599 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:25.592+0000 7f16cfe59740 1 -- 172.21.15.46:0/4265707058 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x556b62e66e30 con 0x556b62ef61e0 2024-03-31T23:21:25.599 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:25.592+0000 7f16aaff5640 1 -- 172.21.15.46:0/4265707058 <== mon.1 v2:172.21.15.162:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 9554+0+0 (secure 0 0 0) 0x7f16b8004250 con 0x556b62ef61e0 2024-03-31T23:21:25.599 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:25.592+0000 7f16aaff5640 1 -- 172.21.15.46:0/4265707058 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f16b802e050 con 0x556b62ef61e0 2024-03-31T23:21:25.599 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:25.592+0000 7f16cfe59740 10 client.26677 fetch_fsmap learned FSMap version 36 2024-03-31T23:21:25.599 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:25.592+0000 7f16cfe59740 1 -- 172.21.15.46:0/4265707058 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x556b62f7ab20 con 0x556b62ef61e0 2024-03-31T23:21:25.599 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:25.592+0000 7f16aaff5640 1 -- 172.21.15.46:0/4265707058 <== mon.1 v2:172.21.15.162:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x556b62e66e30 con 0x556b62ef61e0 2024-03-31T23:21:25.599 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:25.592+0000 7f16aaff5640 1 -- 172.21.15.46:0/4265707058 <== mon.1 v2:172.21.15.162:3300/0 5 ==== fsmap(e 36) ==== 8362+0+0 (secure 0 0 0) 0x7f16b8003b60 con 0x556b62ef61e0 2024-03-31T23:21:25.599 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:25.593+0000 7f16cfe59740 10 client.26677 fetch_fsmap finished waiting for FSMap version 36 2024-03-31T23:21:25.599 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:25.593+0000 7f16cfe59740 10 client.26677 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-31T23:21:25.599 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:25.593+0000 7f16cfe59740 20 client.26677 populate_metadata read hostname 'smithi046' 2024-03-31T23:21:25.600 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:25.593+0000 7f16cfe59740 1 --2- 172.21.15.46:0/4265707058 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x556b62f7b2a0 0x556b62f7d6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:21:25.600 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:25.593+0000 7f16cfe59740 4 client.26677 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-31T23:21:25.600 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:25.593+0000 7f16cfe59740 1 -- 172.21.15.46:0/4265707058 --> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] -- command(tid 0: {"prefix": "flush journal"}) -- 0x556b62e61180 con 0x556b62f7b2a0 2024-03-31T23:21:25.600 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:25.593+0000 7f16c67fc640 1 --2- 172.21.15.46:0/4265707058 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x556b62f7b2a0 0x556b62f7d6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:21:25.600 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:25.593+0000 7f16c67fc640 1 --2- 172.21.15.46:0/4265707058 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x556b62f7b2a0 0x556b62f7d6c0 crc :-1 s=READY pgs=162 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a229a269ae27a286 server_cookie=85e3ad5a35877ee5 in_seq=0 out_seq=0 2024-03-31T23:21:25.600 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:25.593+0000 7f16aaff5640 10 client.26677 ms_handle_connect on v2:172.21.15.142:6838/2519143918 2024-03-31T23:21:25.619 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:25.617+0000 7f16aaff5640 1 -- 172.21.15.46:0/4265707058 <== mds.0 v2:172.21.15.142:6838/2519143918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x556b62e61180 con 0x556b62f7b2a0 2024-03-31T23:21:25.619 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:25.617+0000 7f16aaff5640 10 client.26677 handle_command_reply: tid=0 2024-03-31T23:21:25.619 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:25.617+0000 7f16cfe59740 1 client.26677 shutdown 2024-03-31T23:21:25.619 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:25.617+0000 7f16cfe59740 1 -- 172.21.15.46:0/4265707058 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556b62ef61e0 msgr2=0x556b62efe420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:21:25.619 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:25.617+0000 7f16cfe59740 1 --2- 172.21.15.46:0/4265707058 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556b62ef61e0 0x556b62efe420 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f16b80058c0 tx=0x7f16b8010800 comp rx=0 tx=0).stop 2024-03-31T23:21:25.619 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:25.617+0000 7f16cfe59740 1 -- 172.21.15.46:0/4265707058 shutdown_connections 2024-03-31T23:21:25.619 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:25.617+0000 7f16cfe59740 1 --2- 172.21.15.46:0/4265707058 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x556b62f7b2a0 0x556b62f7d6c0 crc :-1 s=READY pgs=162 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:21:25.619 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:25.617+0000 7f16cfe59740 1 --2- 172.21.15.46:0/4265707058 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x556b62f041c0 0x556b62f79370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:21:25.619 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:25.617+0000 7f16aaff5640 0 client.26677 ms_handle_reset on v2:172.21.15.142:6838/2519143918 2024-03-31T23:21:25.620 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:25.617+0000 7f16cfe59740 1 --2- 172.21.15.46:0/4265707058 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556b62ef6b00 0x556b62efe960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:21:25.620 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:25.617+0000 7f16cfe59740 1 --2- 172.21.15.46:0/4265707058 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556b62ef61e0 0x556b62efe420 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:21:25.620 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:25.617+0000 7f16cfe59740 1 -- 172.21.15.46:0/4265707058 >> 172.21.15.46:0/4265707058 conn(0x556b62ef0fe0 msgr2=0x556b62eeff80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:21:25.620 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:25.617+0000 7f16cfe59740 1 -- 172.21.15.46:0/4265707058 shutdown_connections 2024-03-31T23:21:25.620 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:25.617+0000 7f16cfe59740 1 -- 172.21.15.46:0/4265707058 wait complete. 2024-03-31T23:21:25.620 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:25.617+0000 7f16cfe59740 20 client.26677 trim_cache size 0 max 16384 2024-03-31T23:21:25.620 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout:{ 2024-03-31T23:21:25.620 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout: "message": "", 2024-03-31T23:21:25.620 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout: "return_code": 0 2024-03-31T23:21:25.620 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout:} 2024-03-31T23:21:25.621 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:25.618+0000 7f16cd27c640 1 -- 172.21.15.46:0/419860184 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f16b00a2730 msgr2=0x7f16b00a4bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:21:25.621 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:25.618+0000 7f16cd27c640 1 --2- 172.21.15.46:0/419860184 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f16b00a2730 0x7f16b00a4bf0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f16c000f480 tx=0x7f16c00117c0 comp rx=0 tx=0).stop 2024-03-31T23:21:25.621 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:25.618+0000 7f16cd27c640 1 -- 172.21.15.46:0/419860184 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f16c81020c0 msgr2=0x7f16c8198470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:21:25.621 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:25.618+0000 7f16cd27c640 1 --2- 172.21.15.46:0/419860184 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f16c81020c0 0x7f16c8198470 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7f16bc009230 tx=0x7f16bc02c5d0 comp rx=0 tx=0).stop 2024-03-31T23:21:25.622 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:25.618+0000 7f16cd27c640 1 -- 172.21.15.46:0/419860184 shutdown_connections 2024-03-31T23:21:25.622 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:25.618+0000 7f16cd27c640 1 --2- 172.21.15.46:0/419860184 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f16b00a2730 0x7f16b00a4bf0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:21:25.622 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:25.618+0000 7f16cd27c640 1 --2- 172.21.15.46:0/419860184 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f16c8107710 0x7f16c819c7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:21:25.622 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:25.618+0000 7f16cd27c640 1 --2- 172.21.15.46:0/419860184 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f16c81020c0 0x7f16c8198470 unknown :-1 s=CLOSED pgs=800 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:21:25.622 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:25.618+0000 7f16cd27c640 1 --2- 172.21.15.46:0/419860184 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f16c8101700 0x7f16c8197f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:21:25.622 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:25.618+0000 7f16cd27c640 1 -- 172.21.15.46:0/419860184 >> 172.21.15.46:0/419860184 conn(0x7f16c80fd210 msgr2=0x7f16c80fd850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:21:25.622 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:25.618+0000 7f16cd27c640 1 -- 172.21.15.46:0/419860184 shutdown_connections 2024-03-31T23:21:25.622 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:25.618+0000 7f16cd27c640 1 -- 172.21.15.46:0/419860184 wait complete. 2024-03-31T23:21:26.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:21:25 smithi142 ceph-mon[35271]: pgmap v302: 129 pgs: 129 active+clean; 117 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 381 KiB/s rd, 1.0 MiB/s wr, 101 op/s 2024-03-31T23:21:26.210 INFO:tasks.workunit.client.0.smithi046.stdout: 1 202322 2.04 MB/sec execute 392 sec latency 2835.316 ms 2024-03-31T23:21:26.295 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:21:25 smithi162 ceph-mon[36237]: pgmap v302: 129 pgs: 129 active+clean; 117 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 381 KiB/s rd, 1.0 MiB/s wr, 101 op/s 2024-03-31T23:21:26.362 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:21:25 smithi046 ceph-mon[32498]: pgmap v302: 129 pgs: 129 active+clean; 117 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 381 KiB/s rd, 1.0 MiB/s wr, 101 op/s 2024-03-31T23:21:27.210 INFO:tasks.workunit.client.0.smithi046.stdout: 1 202322 2.04 MB/sec execute 393 sec latency 1382.898 ms 2024-03-31T23:21:28.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:21:27 smithi142 ceph-mon[35271]: pgmap v303: 129 pgs: 129 active+clean; 117 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 308 KiB/s rd, 833 KiB/s wr, 80 op/s 2024-03-31T23:21:28.210 INFO:tasks.workunit.client.0.smithi046.stdout: 1 202334 2.03 MB/sec execute 394 sec latency 1916.175 ms 2024-03-31T23:21:28.296 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:21:27 smithi162 ceph-mon[36237]: pgmap v303: 129 pgs: 129 active+clean; 117 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 308 KiB/s rd, 833 KiB/s wr, 80 op/s 2024-03-31T23:21:28.362 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:21:27 smithi046 ceph-mon[32498]: pgmap v303: 129 pgs: 129 active+clean; 117 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 308 KiB/s rd, 833 KiB/s wr, 80 op/s 2024-03-31T23:21:29.211 INFO:tasks.workunit.client.0.smithi046.stdout: 1 202334 2.03 MB/sec execute 395 sec latency 1414.342 ms 2024-03-31T23:21:30.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:21:29 smithi142 ceph-mon[35271]: pgmap v304: 129 pgs: 129 active+clean; 114 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 352 KiB/s rd, 889 KiB/s wr, 93 op/s 2024-03-31T23:21:30.211 INFO:tasks.workunit.client.0.smithi046.stdout: 1 202334 2.02 MB/sec execute 396 sec latency 2414.465 ms 2024-03-31T23:21:30.296 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:21:29 smithi162 ceph-mon[36237]: pgmap v304: 129 pgs: 129 active+clean; 114 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 352 KiB/s rd, 889 KiB/s wr, 93 op/s 2024-03-31T23:21:30.362 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:21:29 smithi046 ceph-mon[32498]: pgmap v304: 129 pgs: 129 active+clean; 114 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 352 KiB/s rd, 889 KiB/s wr, 93 op/s 2024-03-31T23:21:31.214 INFO:tasks.workunit.client.0.smithi046.stdout: 1 202495 2.02 MB/sec execute 397 sec latency 2834.785 ms 2024-03-31T23:21:32.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:21:31 smithi142 ceph-mon[35271]: pgmap v305: 129 pgs: 129 active+clean; 114 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 139 KiB/s rd, 367 KiB/s wr, 39 op/s 2024-03-31T23:21:32.267 INFO:tasks.workunit.client.0.smithi046.stdout: 1 202754 2.02 MB/sec execute 398 sec latency 16.692 ms 2024-03-31T23:21:32.296 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:21:31 smithi162 ceph-mon[36237]: pgmap v305: 129 pgs: 129 active+clean; 114 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 139 KiB/s rd, 367 KiB/s wr, 39 op/s 2024-03-31T23:21:32.361 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:21:31 smithi046 ceph-mon[32498]: pgmap v305: 129 pgs: 129 active+clean; 114 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 139 KiB/s rd, 367 KiB/s wr, 39 op/s 2024-03-31T23:21:33.211 INFO:tasks.workunit.client.0.smithi046.stdout: 1 203160 2.01 MB/sec execute 399 sec latency 17.926 ms 2024-03-31T23:21:34.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:21:33 smithi142 ceph-mon[35271]: pgmap v306: 129 pgs: 129 active+clean; 114 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 139 KiB/s rd, 367 KiB/s wr, 39 op/s 2024-03-31T23:21:34.211 INFO:tasks.workunit.client.0.smithi046.stdout: 1 203679 2.02 MB/sec execute 400 sec latency 24.833 ms 2024-03-31T23:21:34.296 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:21:33 smithi162 ceph-mon[36237]: pgmap v306: 129 pgs: 129 active+clean; 114 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 139 KiB/s rd, 367 KiB/s wr, 39 op/s 2024-03-31T23:21:34.362 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:21:33 smithi046 ceph-mon[32498]: pgmap v306: 129 pgs: 129 active+clean; 114 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 139 KiB/s rd, 367 KiB/s wr, 39 op/s 2024-03-31T23:21:35.211 INFO:tasks.workunit.client.0.smithi046.stdout: 1 204167 2.01 MB/sec execute 401 sec latency 24.548 ms 2024-03-31T23:21:36.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:21:35 smithi142 ceph-mon[35271]: pgmap v307: 129 pgs: 129 active+clean; 117 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 296 KiB/s rd, 785 KiB/s wr, 78 op/s 2024-03-31T23:21:36.211 INFO:tasks.workunit.client.0.smithi046.stdout: 1 204674 2.02 MB/sec execute 402 sec latency 21.089 ms 2024-03-31T23:21:36.296 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:21:35 smithi162 ceph-mon[36237]: pgmap v307: 129 pgs: 129 active+clean; 117 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 296 KiB/s rd, 785 KiB/s wr, 78 op/s 2024-03-31T23:21:36.361 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:21:35 smithi046 ceph-mon[32498]: pgmap v307: 129 pgs: 129 active+clean; 117 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 296 KiB/s rd, 785 KiB/s wr, 78 op/s 2024-03-31T23:21:37.211 INFO:tasks.workunit.client.0.smithi046.stdout: 1 205390 2.02 MB/sec execute 403 sec latency 16.534 ms 2024-03-31T23:21:38.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:21:37 smithi142 ceph-mon[35271]: pgmap v308: 129 pgs: 129 active+clean; 117 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 200 KiB/s rd, 474 KiB/s wr, 52 op/s 2024-03-31T23:21:38.211 INFO:tasks.workunit.client.0.smithi046.stdout: 1 205752 2.03 MB/sec execute 404 sec latency 21.807 ms 2024-03-31T23:21:38.295 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:21:37 smithi162 ceph-mon[36237]: pgmap v308: 129 pgs: 129 active+clean; 117 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 200 KiB/s rd, 474 KiB/s wr, 52 op/s 2024-03-31T23:21:38.365 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:21:37 smithi046 ceph-mon[32498]: pgmap v308: 129 pgs: 129 active+clean; 117 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 200 KiB/s rd, 474 KiB/s wr, 52 op/s 2024-03-31T23:21:38.713 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:38.710+0000 7fd87481c640 1 -- 172.21.15.46:0/523438679 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd870101260 msgr2=0x7fd87010d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:21:38.713 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:38.710+0000 7fd87481c640 1 --2- 172.21.15.46:0/523438679 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd870101260 0x7fd87010d480 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7fd868009ca0 tx=0x7fd86802c120 comp rx=0 tx=0).stop 2024-03-31T23:21:38.713 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:38.710+0000 7fd87481c640 1 -- 172.21.15.46:0/523438679 shutdown_connections 2024-03-31T23:21:38.713 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:38.710+0000 7fd87481c640 1 --2- 172.21.15.46:0/523438679 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd870101260 0x7fd87010d480 unknown :-1 s=CLOSED pgs=803 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:21:38.713 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:38.710+0000 7fd87481c640 1 --2- 172.21.15.46:0/523438679 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd8701008a0 0x7fd870100d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:21:38.713 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:38.710+0000 7fd87481c640 1 --2- 172.21.15.46:0/523438679 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd8701055f0 0x7fd8701059d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:21:38.713 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:38.710+0000 7fd87481c640 1 -- 172.21.15.46:0/523438679 >> 172.21.15.46:0/523438679 conn(0x7fd8700774a0 msgr2=0x7fd8700778b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:21:38.713 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:38.710+0000 7fd87481c640 1 -- 172.21.15.46:0/523438679 shutdown_connections 2024-03-31T23:21:38.713 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:38.710+0000 7fd87481c640 1 -- 172.21.15.46:0/523438679 wait complete. 2024-03-31T23:21:38.713 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:38.710+0000 7fd87481c640 1 Processor -- start 2024-03-31T23:21:38.714 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:38.710+0000 7fd87481c640 1 -- start start 2024-03-31T23:21:38.714 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:38.711+0000 7fd87481c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd8701008a0 0x7fd870197f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:21:38.714 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:38.711+0000 7fd87481c640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd870101260 0x7fd870198480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:21:38.714 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:38.711+0000 7fd87481c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd8701055f0 0x7fd87019c800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:21:38.714 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:38.711+0000 7fd87481c640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fd8701109e0 con 0x7fd8701055f0 2024-03-31T23:21:38.714 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:38.711+0000 7fd87481c640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fd870110860 con 0x7fd870101260 2024-03-31T23:21:38.714 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:38.711+0000 7fd87481c640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fd870110b60 con 0x7fd8701008a0 2024-03-31T23:21:38.714 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:38.711+0000 7fd86ed9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd8701055f0 0x7fd87019c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:21:38.714 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:38.711+0000 7fd86ed9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd8701055f0 0x7fd87019c800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55516/0 (socket says 172.21.15.46:55516) 2024-03-31T23:21:38.714 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:38.711+0000 7fd867fff640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd870101260 0x7fd870198480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:21:38.714 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:38.711+0000 7fd86ed9d640 1 -- 172.21.15.46:0/250532032 learned_addr learned my addr 172.21.15.46:0/250532032 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:21:38.714 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:38.711+0000 7fd86e59c640 1 --2- 172.21.15.46:0/250532032 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd8701008a0 0x7fd870197f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:21:38.714 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:38.711+0000 7fd86ed9d640 1 -- 172.21.15.46:0/250532032 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd8701008a0 msgr2=0x7fd870197f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:21:38.714 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:38.711+0000 7fd86ed9d640 1 --2- 172.21.15.46:0/250532032 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd8701008a0 0x7fd870197f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:21:38.715 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:38.711+0000 7fd86ed9d640 1 -- 172.21.15.46:0/250532032 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd870101260 msgr2=0x7fd870198480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:21:38.715 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:38.711+0000 7fd86ed9d640 1 --2- 172.21.15.46:0/250532032 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd870101260 0x7fd870198480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:21:38.715 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:38.711+0000 7fd86ed9d640 1 -- 172.21.15.46:0/250532032 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd87019cf30 con 0x7fd8701055f0 2024-03-31T23:21:38.715 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:38.711+0000 7fd86ed9d640 1 --2- 172.21.15.46:0/250532032 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd8701055f0 0x7fd87019c800 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7fd86802c730 tx=0x7fd868035d60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:21:38.715 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:38.711+0000 7fd8677fe640 1 -- 172.21.15.46:0/250532032 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd868039040 con 0x7fd8701055f0 2024-03-31T23:21:38.715 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:38.711+0000 7fd8677fe640 1 -- 172.21.15.46:0/250532032 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd868044070 con 0x7fd8701055f0 2024-03-31T23:21:38.715 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:38.711+0000 7fd87481c640 1 -- 172.21.15.46:0/250532032 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd87019d1c0 con 0x7fd8701055f0 2024-03-31T23:21:38.715 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:38.711+0000 7fd8677fe640 1 -- 172.21.15.46:0/250532032 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd868004280 con 0x7fd8701055f0 2024-03-31T23:21:38.715 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:38.711+0000 7fd87481c640 1 -- 172.21.15.46:0/250532032 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd8701a4ab0 con 0x7fd8701055f0 2024-03-31T23:21:38.723 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:38.714+0000 7fd8677fe640 1 -- 172.21.15.46:0/250532032 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 25) ==== 137451+0+0 (secure 0 0 0) 0x7fd868004420 con 0x7fd8701055f0 2024-03-31T23:21:38.723 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:38.714+0000 7fd877403740 1 Processor -- start 2024-03-31T23:21:38.723 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:38.714+0000 7fd877403740 1 -- start start 2024-03-31T23:21:38.723 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:38.714+0000 7fd877403740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ab49c376c0 0x55ab49c37aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:21:38.723 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:38.714+0000 7fd877403740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ab49c37fe0 0x55ab49c40870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:21:38.723 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:38.714+0000 7fd877403740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55ab49c40db0 0x55ab49c431a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:21:38.723 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:38.714+0000 7fd877403740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55ab49b9a8f0 con 0x55ab49c376c0 2024-03-31T23:21:38.723 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:38.714+0000 7fd877403740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55ab49b438d0 con 0x55ab49c40db0 2024-03-31T23:21:38.723 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:38.714+0000 7fd877403740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55ab49b96990 con 0x55ab49c37fe0 2024-03-31T23:21:38.723 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:38.715+0000 7fd8677fe640 1 --2- 172.21.15.46:0/250532032 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fd85c0a26b0 0x7fd85c0a4b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:21:38.723 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:38.715+0000 7fd867fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ab49c376c0 0x55ab49c37aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:21:38.723 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:38.715+0000 7fd86e59c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ab49c37fe0 0x55ab49c40870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:21:38.723 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:38.715+0000 7fd867fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ab49c376c0 0x55ab49c37aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55522/0 (socket says 172.21.15.46:55522) 2024-03-31T23:21:38.724 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:38.715+0000 7fd867fff640 1 -- 172.21.15.46:0/3649657490 learned_addr learned my addr 172.21.15.46:0/3649657490 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:21:38.724 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:38.715+0000 7fd86ed9d640 1 --2- 172.21.15.46:0/250532032 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fd85c0a26b0 0x7fd85c0a4b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:21:38.724 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:38.715+0000 7fd867fff640 1 --2- 172.21.15.46:0/3649657490 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55ab49c40db0 0x55ab49c431a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:21:38.724 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:38.715+0000 7fd8677fe640 1 -- 172.21.15.46:0/250532032 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 9554+0+0 (secure 0 0 0) 0x7fd8680f1350 con 0x7fd8701055f0 2024-03-31T23:21:38.724 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:38.715+0000 7fd86e59c640 1 -- 172.21.15.46:0/3649657490 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55ab49c40db0 msgr2=0x55ab49c431a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:21:38.724 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:38.715+0000 7fd86e59c640 1 --2- 172.21.15.46:0/3649657490 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55ab49c40db0 0x55ab49c431a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:21:38.724 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:38.715+0000 7fd86e59c640 1 -- 172.21.15.46:0/3649657490 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ab49c376c0 msgr2=0x55ab49c37aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:21:38.724 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:38.715+0000 7fd86e59c640 1 --2- 172.21.15.46:0/3649657490 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ab49c376c0 0x55ab49c37aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:21:38.724 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:38.715+0000 7fd86e59c640 1 -- 172.21.15.46:0/3649657490 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ab498af210 con 0x55ab49c37fe0 2024-03-31T23:21:38.724 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:38.715+0000 7fd86ed9d640 1 --2- 172.21.15.46:0/250532032 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fd85c0a26b0 0x7fd85c0a4b70 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fd8680bd6c0 tx=0x7fd8680b99d0 comp rx=0 tx=0).ready entity=mgr.15654 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:21:38.724 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:38.715+0000 7fd86e59c640 1 --2- 172.21.15.46:0/3649657490 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ab49c37fe0 0x55ab49c40870 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fd860005fd0 tx=0x7fd8600107a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3ba3728de0df0ae server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:21:38.724 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:38.716+0000 7fd864ff9640 1 -- 172.21.15.46:0/3649657490 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd86001c480 con 0x55ab49c37fe0 2024-03-31T23:21:38.724 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:38.716+0000 7fd864ff9640 1 -- 172.21.15.46:0/3649657490 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd86001cad0 con 0x55ab49c37fe0 2024-03-31T23:21:38.725 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:38.716+0000 7fd864ff9640 1 -- 172.21.15.46:0/3649657490 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd860011b90 con 0x55ab49c37fe0 2024-03-31T23:21:38.725 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:38.716+0000 7fd877403740 1 -- 172.21.15.46:0/3649657490 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ab49c37fe0 msgr2=0x55ab49c40870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:21:38.725 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:38.716+0000 7fd877403740 1 --2- 172.21.15.46:0/3649657490 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ab49c37fe0 0x55ab49c40870 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fd860005fd0 tx=0x7fd8600107a0 comp rx=0 tx=0).stop 2024-03-31T23:21:38.725 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:38.716+0000 7fd877403740 1 -- 172.21.15.46:0/3649657490 shutdown_connections 2024-03-31T23:21:38.725 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:38.716+0000 7fd877403740 1 --2- 172.21.15.46:0/3649657490 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55ab49c40db0 0x55ab49c431a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:21:38.725 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:38.716+0000 7fd877403740 1 --2- 172.21.15.46:0/3649657490 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ab49c37fe0 0x55ab49c40870 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:21:38.725 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:38.716+0000 7fd877403740 1 --2- 172.21.15.46:0/3649657490 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ab49c376c0 0x55ab49c37aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:21:38.725 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:38.716+0000 7fd877403740 1 -- 172.21.15.46:0/3649657490 >> 172.21.15.46:0/3649657490 conn(0x55ab49c31fe0 msgr2=0x55ab49c323c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:21:38.725 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:38.716+0000 7fd877403740 1 -- 172.21.15.46:0/3649657490 shutdown_connections 2024-03-31T23:21:38.725 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:38.716+0000 7fd877403740 1 -- 172.21.15.46:0/3649657490 wait complete. 2024-03-31T23:21:38.725 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:38.717+0000 7fd877403740 1 -- start start 2024-03-31T23:21:38.725 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:38.717+0000 7fd877403740 1 Processor -- start 2024-03-31T23:21:38.725 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:38.717+0000 7fd877403740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55ab49c376c0 0x55ab49c3ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:21:38.725 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:38.717+0000 7fd877403740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ab49c37fe0 0x55ab49c3e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:21:38.726 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:38.717+0000 7fd877403740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ab49c40db0 0x55ab49c3e830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:21:38.726 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:38.717+0000 7fd877403740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55ab49c439e0 con 0x55ab49c40db0 2024-03-31T23:21:38.726 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:38.717+0000 7fd877403740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55ab49b38f80 con 0x55ab49c376c0 2024-03-31T23:21:38.726 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:38.717+0000 7fd877403740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55ab49c43b60 con 0x55ab49c37fe0 2024-03-31T23:21:38.726 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:38.717+0000 7fd877403740 10 client.0 _ll_register_callbacks cb 0x55ab49b91290 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T23:21:38.726 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:38.717+0000 7fd867fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ab49c40db0 0x55ab49c3e830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:21:38.726 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:38.717+0000 7fd867fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ab49c40db0 0x55ab49c3e830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55528/0 (socket says 172.21.15.46:55528) 2024-03-31T23:21:38.726 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:38.717+0000 7fd867fff640 1 -- 172.21.15.46:0/980161042 learned_addr learned my addr 172.21.15.46:0/980161042 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:21:38.726 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:38.717+0000 7fd867fff640 1 --2- 172.21.15.46:0/980161042 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55ab49c376c0 0x55ab49c3ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:21:38.726 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:38.717+0000 7fd86e59c640 1 --2- 172.21.15.46:0/980161042 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ab49c37fe0 0x55ab49c3e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:21:38.726 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:38.717+0000 7fd867fff640 1 -- 172.21.15.46:0/980161042 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ab49c37fe0 msgr2=0x55ab49c3e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:21:38.726 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:38.717+0000 7fd867fff640 1 --2- 172.21.15.46:0/980161042 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ab49c37fe0 0x55ab49c3e2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:21:38.726 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:38.717+0000 7fd867fff640 1 -- 172.21.15.46:0/980161042 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55ab49c376c0 msgr2=0x55ab49c3ddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:21:38.726 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:38.717+0000 7fd867fff640 1 --2- 172.21.15.46:0/980161042 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55ab49c376c0 0x55ab49c3ddb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:21:38.727 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:38.717+0000 7fd867fff640 1 -- 172.21.15.46:0/980161042 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ab4997b620 con 0x55ab49c40db0 2024-03-31T23:21:38.727 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:38.718+0000 7fd867fff640 1 --2- 172.21.15.46:0/980161042 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ab49c40db0 0x55ab49c3e830 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7fd85801ae60 tx=0x7fd858008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f0f904f7159cd96 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:21:38.727 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:38.718+0000 7fd855ffb640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-31T23:21:38.727 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:38.718+0000 7fd855ffb640 1 -- 172.21.15.46:0/980161042 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd85801a270 con 0x55ab49c40db0 2024-03-31T23:21:38.727 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:38.718+0000 7fd855ffb640 1 -- 172.21.15.46:0/980161042 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 9554+0+0 (secure 0 0 0) 0x7fd858015580 con 0x55ab49c40db0 2024-03-31T23:21:38.727 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:38.718+0000 7fd877403740 1 -- 172.21.15.46:0/980161042 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd86001cad0 con 0x55ab49c40db0 2024-03-31T23:21:38.727 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:38.718+0000 7fd855ffb640 1 -- 172.21.15.46:0/980161042 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7fd85802c050 con 0x55ab49c40db0 2024-03-31T23:21:38.727 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:38.718+0000 7fd877403740 10 client.16956 fetch_fsmap learned FSMap version 36 2024-03-31T23:21:38.727 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:38.718+0000 7fd877403740 1 -- 172.21.15.46:0/980161042 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x55ab49c4f8b0 con 0x55ab49c40db0 2024-03-31T23:21:38.727 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:38.718+0000 7fd855ffb640 1 -- 172.21.15.46:0/980161042 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd86001cad0 con 0x55ab49c40db0 2024-03-31T23:21:38.727 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:38.718+0000 7fd855ffb640 1 -- 172.21.15.46:0/980161042 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 36) ==== 8362+0+0 (secure 0 0 0) 0x7fd858015050 con 0x55ab49c40db0 2024-03-31T23:21:38.727 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:38.718+0000 7fd877403740 10 client.16956 fetch_fsmap finished waiting for FSMap version 36 2024-03-31T23:21:38.727 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:38.718+0000 7fd877403740 10 client.16956 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-31T23:21:38.727 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:38.718+0000 7fd877403740 20 client.16956 populate_metadata read hostname 'smithi046' 2024-03-31T23:21:38.728 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:38.718+0000 7fd877403740 1 --2- 172.21.15.46:0/980161042 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55ab49c50d70 0x55ab49c53160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:21:38.728 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:38.718+0000 7fd877403740 4 client.16956 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-31T23:21:38.728 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:38.718+0000 7fd877403740 1 -- 172.21.15.46:0/980161042 --> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ab49bae120 con 0x55ab49c50d70 2024-03-31T23:21:38.728 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:38.719+0000 7fd86e59c640 1 --2- 172.21.15.46:0/980161042 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55ab49c50d70 0x55ab49c53160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:21:38.728 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:38.719+0000 7fd855ffb640 10 client.16956 ms_handle_connect on v2:172.21.15.142:6838/2519143918 2024-03-31T23:21:38.728 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:38.719+0000 7fd86e59c640 1 --2- 172.21.15.46:0/980161042 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55ab49c50d70 0x55ab49c53160 crc :-1 s=READY pgs=163 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=709f21507c0d8e89 server_cookie=8ac6552c9aecbe41 in_seq=0 out_seq=0 2024-03-31T23:21:38.728 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:38.720+0000 7fd855ffb640 1 -- 172.21.15.46:0/980161042 <== mds.0 v2:172.21.15.142:6838/2519143918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55ab49bae120 con 0x55ab49c50d70 2024-03-31T23:21:38.728 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:38.720+0000 7fd855ffb640 10 client.16956 handle_command_reply: tid=0 2024-03-31T23:21:38.728 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:38.720+0000 7fd877403740 1 client.16956 shutdown 2024-03-31T23:21:38.728 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:38.720+0000 7fd877403740 1 -- 172.21.15.46:0/980161042 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ab49c40db0 msgr2=0x55ab49c3e830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:21:38.728 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:38.720+0000 7fd877403740 1 --2- 172.21.15.46:0/980161042 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ab49c40db0 0x55ab49c3e830 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7fd85801ae60 tx=0x7fd858008040 comp rx=0 tx=0).stop 2024-03-31T23:21:38.728 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:38.720+0000 7fd877403740 1 -- 172.21.15.46:0/980161042 shutdown_connections 2024-03-31T23:21:38.728 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:38.720+0000 7fd877403740 1 --2- 172.21.15.46:0/980161042 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55ab49c50d70 0x55ab49c53160 crc :-1 s=READY pgs=163 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:21:38.728 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:38.720+0000 7fd877403740 1 --2- 172.21.15.46:0/980161042 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ab49c40db0 0x55ab49c3e830 unknown :-1 s=CLOSED pgs=805 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:21:38.729 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:38.720+0000 7fd877403740 1 --2- 172.21.15.46:0/980161042 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ab49c37fe0 0x55ab49c3e2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:21:38.729 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:38.720+0000 7fd855ffb640 0 client.16956 ms_handle_reset on v2:172.21.15.142:6838/2519143918 2024-03-31T23:21:38.729 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:38.720+0000 7fd877403740 1 --2- 172.21.15.46:0/980161042 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55ab49c376c0 0x55ab49c3ddb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:21:38.729 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:38.720+0000 7fd877403740 1 -- 172.21.15.46:0/980161042 >> 172.21.15.46:0/980161042 conn(0x55ab49c31fe0 msgr2=0x55ab49c33d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:21:38.729 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:38.720+0000 7fd877403740 1 -- 172.21.15.46:0/980161042 shutdown_connections 2024-03-31T23:21:38.729 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:38.720+0000 7fd877403740 1 -- 172.21.15.46:0/980161042 wait complete. 2024-03-31T23:21:38.729 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:38.720+0000 7fd877403740 20 client.16956 trim_cache size 0 max 16384 2024-03-31T23:21:38.741 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:38.735+0000 7fd877403740 1 Processor -- start 2024-03-31T23:21:38.741 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:38.735+0000 7fd877403740 1 -- start start 2024-03-31T23:21:38.741 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:38.735+0000 7fd877403740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ab49c376c0 0x55ab49c2ce20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:21:38.741 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:38.735+0000 7fd877403740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ab49c37fe0 0x55ab49c4c780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:21:38.742 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:38.735+0000 7fd877403740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55ab49c40db0 0x55ab49c58a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:21:38.742 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:38.735+0000 7fd877403740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fd8701b14b0 con 0x55ab49c376c0 2024-03-31T23:21:38.742 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:38.735+0000 7fd877403740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fd8701aed20 con 0x55ab49c40db0 2024-03-31T23:21:38.742 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:38.735+0000 7fd877403740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fd8701acb90 con 0x55ab49c37fe0 2024-03-31T23:21:38.742 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:38.735+0000 7fd867fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ab49c376c0 0x55ab49c2ce20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:21:38.742 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:38.736+0000 7fd867fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ab49c376c0 0x55ab49c2ce20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55544/0 (socket says 172.21.15.46:55544) 2024-03-31T23:21:38.742 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:38.736+0000 7fd867fff640 1 -- 172.21.15.46:0/3220879075 learned_addr learned my addr 172.21.15.46:0/3220879075 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:21:38.742 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:38.736+0000 7fd86e59c640 1 --2- 172.21.15.46:0/3220879075 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ab49c37fe0 0x55ab49c4c780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:21:38.742 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:38.736+0000 7fd867fff640 1 --2- 172.21.15.46:0/3220879075 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55ab49c40db0 0x55ab49c58a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:21:38.742 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:38.736+0000 7fd867fff640 1 -- 172.21.15.46:0/3220879075 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ab49c37fe0 msgr2=0x55ab49c4c780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:21:38.742 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:38.736+0000 7fd867fff640 1 --2- 172.21.15.46:0/3220879075 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ab49c37fe0 0x55ab49c4c780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:21:38.742 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:38.736+0000 7fd867fff640 1 -- 172.21.15.46:0/3220879075 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55ab49c40db0 msgr2=0x55ab49c58a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:21:38.742 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:38.736+0000 7fd867fff640 1 --2- 172.21.15.46:0/3220879075 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55ab49c40db0 0x55ab49c58a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:21:38.743 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:38.736+0000 7fd867fff640 1 -- 172.21.15.46:0/3220879075 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ab49c37dd0 con 0x55ab49c376c0 2024-03-31T23:21:38.743 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:38.736+0000 7fd86e59c640 1 --2- 172.21.15.46:0/3220879075 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ab49c37fe0 0x55ab49c4c780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:21:38.743 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:38.736+0000 7fd867fff640 1 --2- 172.21.15.46:0/3220879075 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ab49c376c0 0x55ab49c2ce20 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7fd85800a010 tx=0x7fd85801aa40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b814b6e1cfabde7 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:21:38.743 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:38.736+0000 7fd857fff640 1 -- 172.21.15.46:0/3220879075 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd85801a2b0 con 0x55ab49c376c0 2024-03-31T23:21:38.743 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:38.736+0000 7fd857fff640 1 -- 172.21.15.46:0/3220879075 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd8580044e0 con 0x55ab49c376c0 2024-03-31T23:21:38.743 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:38.736+0000 7fd857fff640 1 -- 172.21.15.46:0/3220879075 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd858003690 con 0x55ab49c376c0 2024-03-31T23:21:38.743 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:38.737+0000 7fd877403740 1 -- 172.21.15.46:0/3220879075 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ab49c376c0 msgr2=0x55ab49c2ce20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:21:38.743 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:38.737+0000 7fd877403740 1 --2- 172.21.15.46:0/3220879075 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ab49c376c0 0x55ab49c2ce20 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7fd85800a010 tx=0x7fd85801aa40 comp rx=0 tx=0).stop 2024-03-31T23:21:38.743 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:38.737+0000 7fd877403740 1 -- 172.21.15.46:0/3220879075 shutdown_connections 2024-03-31T23:21:38.743 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:38.737+0000 7fd877403740 1 --2- 172.21.15.46:0/3220879075 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55ab49c40db0 0x55ab49c58a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:21:38.743 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:38.737+0000 7fd877403740 1 --2- 172.21.15.46:0/3220879075 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ab49c37fe0 0x55ab49c4c780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:21:38.743 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:38.737+0000 7fd877403740 1 --2- 172.21.15.46:0/3220879075 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ab49c376c0 0x55ab49c2ce20 unknown :-1 s=CLOSED pgs=806 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:21:38.743 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:38.737+0000 7fd877403740 1 -- 172.21.15.46:0/3220879075 >> 172.21.15.46:0/3220879075 conn(0x55ab49c31fe0 msgr2=0x55ab49c38540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:21:38.743 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:38.737+0000 7fd877403740 1 -- 172.21.15.46:0/3220879075 shutdown_connections 2024-03-31T23:21:38.744 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:38.737+0000 7fd877403740 1 -- 172.21.15.46:0/3220879075 wait complete. 2024-03-31T23:21:38.744 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:38.737+0000 7fd877403740 1 -- start start 2024-03-31T23:21:38.744 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:38.737+0000 7fd877403740 1 Processor -- start 2024-03-31T23:21:38.744 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:38.737+0000 7fd877403740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55ab49c376c0 0x55ab49c44cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:21:38.744 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:38.737+0000 7fd877403740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ab49c37fe0 0x55ab49c45210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:21:38.744 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:38.737+0000 7fd877403740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ab49c40db0 0x55ab49cb60a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:21:38.744 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:38.737+0000 7fd877403740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fd8701ad1e0 con 0x55ab49c40db0 2024-03-31T23:21:38.744 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:38.737+0000 7fd877403740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fd8701ae650 con 0x55ab49c376c0 2024-03-31T23:21:38.744 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:38.737+0000 7fd877403740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fd8701ad8b0 con 0x55ab49c37fe0 2024-03-31T23:21:38.744 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:38.737+0000 7fd877403740 10 client.0 _ll_register_callbacks cb 0x7fd858009370 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T23:21:38.744 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:38.738+0000 7fd867fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ab49c40db0 0x55ab49cb60a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:21:38.744 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:38.738+0000 7fd867fff640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55ab49c376c0 0x55ab49c44cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:21:38.744 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:38.738+0000 7fd867fff640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55ab49c376c0 0x55ab49c44cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.46:39546/0 (socket says 172.21.15.46:39546) 2024-03-31T23:21:38.745 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:38.738+0000 7fd867fff640 1 -- 172.21.15.46:0/4090864117 learned_addr learned my addr 172.21.15.46:0/4090864117 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:21:38.745 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:38.738+0000 7fd86e59c640 1 --2- 172.21.15.46:0/4090864117 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ab49c37fe0 0x55ab49c45210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:21:38.745 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:38.738+0000 7fd867fff640 1 -- 172.21.15.46:0/4090864117 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ab49c37fe0 msgr2=0x55ab49c45210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:21:38.745 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:38.738+0000 7fd867fff640 1 --2- 172.21.15.46:0/4090864117 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ab49c37fe0 0x55ab49c45210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:21:38.745 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:38.738+0000 7fd867fff640 1 -- 172.21.15.46:0/4090864117 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ab49c40db0 msgr2=0x55ab49cb60a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:21:38.745 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:38.738+0000 7fd867fff640 1 --2- 172.21.15.46:0/4090864117 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ab49c40db0 0x55ab49cb60a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:21:38.745 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:38.738+0000 7fd867fff640 1 -- 172.21.15.46:0/4090864117 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ab49cb65e0 con 0x55ab49c376c0 2024-03-31T23:21:38.745 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:38.738+0000 7fd867fff640 1 --2- 172.21.15.46:0/4090864117 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55ab49c376c0 0x55ab49c44cd0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fd858000ea0 tx=0x7fd8580090e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=94a29ddb2dcb57f5 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:21:38.745 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:38.738+0000 7fd855ffb640 10 client.0 ms_handle_connect on v2:172.21.15.142:3300/0 2024-03-31T23:21:38.745 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:38.738+0000 7fd855ffb640 1 -- 172.21.15.46:0/4090864117 <== mon.2 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8580184d0 con 0x55ab49c376c0 2024-03-31T23:21:38.745 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:38.738+0000 7fd855ffb640 1 -- 172.21.15.46:0/4090864117 <== mon.2 v2:172.21.15.142:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 9554+0+0 (secure 0 0 0) 0x7fd85801a040 con 0x55ab49c376c0 2024-03-31T23:21:38.745 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:38.738+0000 7fd877403740 1 -- 172.21.15.46:0/4090864117 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd8580044e0 con 0x55ab49c376c0 2024-03-31T23:21:38.745 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:38.738+0000 7fd855ffb640 1 -- 172.21.15.46:0/4090864117 <== mon.2 v2:172.21.15.142:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7fd858018670 con 0x55ab49c376c0 2024-03-31T23:21:38.745 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:38.738+0000 7fd877403740 10 client.26798 fetch_fsmap learned FSMap version 36 2024-03-31T23:21:38.746 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:38.738+0000 7fd877403740 1 -- 172.21.15.46:0/4090864117 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x55ab49cb77f0 con 0x55ab49c376c0 2024-03-31T23:21:38.746 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:38.739+0000 7fd855ffb640 1 -- 172.21.15.46:0/4090864117 <== mon.2 v2:172.21.15.142:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd8580044e0 con 0x55ab49c376c0 2024-03-31T23:21:38.746 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:38.739+0000 7fd855ffb640 1 -- 172.21.15.46:0/4090864117 <== mon.2 v2:172.21.15.142:3300/0 5 ==== fsmap(e 36) ==== 8362+0+0 (secure 0 0 0) 0x7fd858015c60 con 0x55ab49c376c0 2024-03-31T23:21:38.746 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:38.739+0000 7fd877403740 10 client.26798 fetch_fsmap finished waiting for FSMap version 36 2024-03-31T23:21:38.746 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:38.739+0000 7fd877403740 10 client.26798 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-31T23:21:38.746 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:38.739+0000 7fd877403740 20 client.26798 populate_metadata read hostname 'smithi046' 2024-03-31T23:21:38.746 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:38.739+0000 7fd877403740 1 --2- 172.21.15.46:0/4090864117 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55ab49c50d70 0x55ab49cb9f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:21:38.746 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:38.739+0000 7fd877403740 4 client.26798 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-31T23:21:38.746 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:38.739+0000 7fd877403740 1 -- 172.21.15.46:0/4090864117 --> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55ab49ba7e30 con 0x55ab49c50d70 2024-03-31T23:21:38.746 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:38.739+0000 7fd86e59c640 1 --2- 172.21.15.46:0/4090864117 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55ab49c50d70 0x55ab49cb9f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:21:38.746 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:38.739+0000 7fd855ffb640 10 client.26798 ms_handle_connect on v2:172.21.15.142:6838/2519143918 2024-03-31T23:21:38.746 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:38.739+0000 7fd86e59c640 1 --2- 172.21.15.46:0/4090864117 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55ab49c50d70 0x55ab49cb9f80 crc :-1 s=READY pgs=164 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=410dd7c9e36dc612 server_cookie=b186588310236227 in_seq=0 out_seq=0 2024-03-31T23:21:38.766 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:38.763+0000 7fd855ffb640 1 -- 172.21.15.46:0/4090864117 <== mds.0 v2:172.21.15.142:6838/2519143918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55ab49ba7e30 con 0x55ab49c50d70 2024-03-31T23:21:38.766 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:38.763+0000 7fd855ffb640 10 client.26798 handle_command_reply: tid=0 2024-03-31T23:21:38.766 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:38.764+0000 7fd877403740 1 client.26798 shutdown 2024-03-31T23:21:38.766 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:38.764+0000 7fd877403740 1 -- 172.21.15.46:0/4090864117 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55ab49c376c0 msgr2=0x55ab49c44cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:21:38.766 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:38.764+0000 7fd877403740 1 --2- 172.21.15.46:0/4090864117 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55ab49c376c0 0x55ab49c44cd0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fd858000ea0 tx=0x7fd8580090e0 comp rx=0 tx=0).stop 2024-03-31T23:21:38.766 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:38.764+0000 7fd877403740 1 -- 172.21.15.46:0/4090864117 shutdown_connections 2024-03-31T23:21:38.766 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:38.764+0000 7fd877403740 1 --2- 172.21.15.46:0/4090864117 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55ab49c50d70 0x55ab49cb9f80 crc :-1 s=READY pgs=164 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:21:38.766 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:38.764+0000 7fd877403740 1 --2- 172.21.15.46:0/4090864117 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ab49c40db0 0x55ab49cb60a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:21:38.766 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:38.764+0000 7fd855ffb640 0 client.26798 ms_handle_reset on v2:172.21.15.142:6838/2519143918 2024-03-31T23:21:38.766 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:38.764+0000 7fd877403740 1 --2- 172.21.15.46:0/4090864117 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ab49c37fe0 0x55ab49c45210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:21:38.766 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:38.764+0000 7fd877403740 1 --2- 172.21.15.46:0/4090864117 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55ab49c376c0 0x55ab49c44cd0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:21:38.767 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:38.764+0000 7fd877403740 1 -- 172.21.15.46:0/4090864117 >> 172.21.15.46:0/4090864117 conn(0x55ab49c31fe0 msgr2=0x55ab49c306f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:21:38.767 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:38.764+0000 7fd877403740 1 -- 172.21.15.46:0/4090864117 shutdown_connections 2024-03-31T23:21:38.767 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:38.764+0000 7fd877403740 1 -- 172.21.15.46:0/4090864117 wait complete. 2024-03-31T23:21:38.767 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:38.764+0000 7fd877403740 20 client.26798 trim_cache size 0 max 16384 2024-03-31T23:21:38.767 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout:{ 2024-03-31T23:21:38.767 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout: "message": "", 2024-03-31T23:21:38.767 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout: "return_code": 0 2024-03-31T23:21:38.767 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout:} 2024-03-31T23:21:38.768 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:38.765+0000 7fd87481c640 1 -- 172.21.15.46:0/250532032 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fd85c0a26b0 msgr2=0x7fd85c0a4b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:21:38.768 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:38.765+0000 7fd87481c640 1 --2- 172.21.15.46:0/250532032 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fd85c0a26b0 0x7fd85c0a4b70 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fd8680bd6c0 tx=0x7fd8680b99d0 comp rx=0 tx=0).stop 2024-03-31T23:21:38.768 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:38.765+0000 7fd87481c640 1 -- 172.21.15.46:0/250532032 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd8701055f0 msgr2=0x7fd87019c800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:21:38.768 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:38.765+0000 7fd87481c640 1 --2- 172.21.15.46:0/250532032 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd8701055f0 0x7fd87019c800 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7fd86802c730 tx=0x7fd868035d60 comp rx=0 tx=0).stop 2024-03-31T23:21:38.768 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:38.765+0000 7fd87481c640 1 -- 172.21.15.46:0/250532032 shutdown_connections 2024-03-31T23:21:38.769 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:38.765+0000 7fd87481c640 1 --2- 172.21.15.46:0/250532032 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fd85c0a26b0 0x7fd85c0a4b70 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:21:38.769 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:38.765+0000 7fd87481c640 1 --2- 172.21.15.46:0/250532032 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd8701055f0 0x7fd87019c800 unknown :-1 s=CLOSED pgs=804 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:21:38.769 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:38.765+0000 7fd87481c640 1 --2- 172.21.15.46:0/250532032 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd870101260 0x7fd870198480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:21:38.769 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:38.765+0000 7fd87481c640 1 --2- 172.21.15.46:0/250532032 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd8701008a0 0x7fd870197f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:21:38.769 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:38.765+0000 7fd87481c640 1 -- 172.21.15.46:0/250532032 >> 172.21.15.46:0/250532032 conn(0x7fd8700774a0 msgr2=0x7fd870069010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:21:38.769 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:38.765+0000 7fd87481c640 1 -- 172.21.15.46:0/250532032 shutdown_connections 2024-03-31T23:21:38.769 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:38.765+0000 7fd87481c640 1 -- 172.21.15.46:0/250532032 wait complete. 2024-03-31T23:21:39.211 INFO:tasks.workunit.client.0.smithi046.stdout: 1 205802 2.02 MB/sec execute 405 sec latency 946.656 ms 2024-03-31T23:21:40.163 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:21:39 smithi046 ceph-mon[32498]: pgmap v309: 129 pgs: 129 active+clean; 118 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 381 KiB/s rd, 1.0 MiB/s wr, 99 op/s 2024-03-31T23:21:40.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:21:39 smithi142 ceph-mon[35271]: pgmap v309: 129 pgs: 129 active+clean; 118 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 381 KiB/s rd, 1.0 MiB/s wr, 99 op/s 2024-03-31T23:21:40.211 INFO:tasks.workunit.client.0.smithi046.stdout: 1 205802 2.02 MB/sec execute 406 sec latency 1946.759 ms 2024-03-31T23:21:40.296 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:21:39 smithi162 ceph-mon[36237]: pgmap v309: 129 pgs: 129 active+clean; 118 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 381 KiB/s rd, 1.0 MiB/s wr, 99 op/s 2024-03-31T23:21:41.211 INFO:tasks.workunit.client.0.smithi046.stdout: 1 205802 2.01 MB/sec execute 407 sec latency 2946.845 ms 2024-03-31T23:21:42.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:21:41 smithi142 ceph-mon[35271]: pgmap v310: 129 pgs: 129 active+clean; 118 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 338 KiB/s rd, 985 KiB/s wr, 85 op/s 2024-03-31T23:21:42.211 INFO:tasks.workunit.client.0.smithi046.stdout: 1 205802 2.01 MB/sec execute 408 sec latency 3946.943 ms 2024-03-31T23:21:42.295 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:21:41 smithi162 ceph-mon[36237]: pgmap v310: 129 pgs: 129 active+clean; 118 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 338 KiB/s rd, 985 KiB/s wr, 85 op/s 2024-03-31T23:21:42.362 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:21:41 smithi046 ceph-mon[32498]: pgmap v310: 129 pgs: 129 active+clean; 118 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 338 KiB/s rd, 985 KiB/s wr, 85 op/s 2024-03-31T23:21:43.212 INFO:tasks.workunit.client.0.smithi046.stdout: 1 205869 2.01 MB/sec execute 409 sec latency 4480.461 ms 2024-03-31T23:21:44.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:21:43 smithi142 ceph-mon[35271]: pgmap v311: 129 pgs: 129 active+clean; 118 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 338 KiB/s rd, 985 KiB/s wr, 85 op/s 2024-03-31T23:21:44.212 INFO:tasks.workunit.client.0.smithi046.stdout: 1 205869 2.00 MB/sec execute 410 sec latency 1226.119 ms 2024-03-31T23:21:44.295 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:21:43 smithi162 ceph-mon[36237]: pgmap v311: 129 pgs: 129 active+clean; 118 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 338 KiB/s rd, 985 KiB/s wr, 85 op/s 2024-03-31T23:21:44.361 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:21:43 smithi046 ceph-mon[32498]: pgmap v311: 129 pgs: 129 active+clean; 118 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 338 KiB/s rd, 985 KiB/s wr, 85 op/s 2024-03-31T23:21:45.212 INFO:tasks.workunit.client.0.smithi046.stdout: 1 205869 2.00 MB/sec execute 411 sec latency 2226.198 ms 2024-03-31T23:21:46.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:21:45 smithi142 ceph-mon[35271]: pgmap v312: 129 pgs: 129 active+clean; 118 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 379 KiB/s rd, 1.0 MiB/s wr, 101 op/s 2024-03-31T23:21:46.212 INFO:tasks.workunit.client.0.smithi046.stdout: 1 205869 1.99 MB/sec execute 412 sec latency 3226.305 ms 2024-03-31T23:21:46.296 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:21:45 smithi162 ceph-mon[36237]: pgmap v312: 129 pgs: 129 active+clean; 118 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 379 KiB/s rd, 1.0 MiB/s wr, 101 op/s 2024-03-31T23:21:46.362 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:21:45 smithi046 ceph-mon[32498]: pgmap v312: 129 pgs: 129 active+clean; 118 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 379 KiB/s rd, 1.0 MiB/s wr, 101 op/s 2024-03-31T23:21:47.212 INFO:tasks.workunit.client.0.smithi046.stdout: 1 205869 1.99 MB/sec execute 413 sec latency 4226.394 ms 2024-03-31T23:21:48.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:21:47 smithi142 ceph-mon[35271]: pgmap v313: 129 pgs: 129 active+clean; 118 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 222 KiB/s rd, 620 KiB/s wr, 62 op/s 2024-03-31T23:21:48.212 INFO:tasks.workunit.client.0.smithi046.stdout: 1 206005 1.98 MB/sec execute 414 sec latency 4756.271 ms 2024-03-31T23:21:48.295 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:21:47 smithi162 ceph-mon[36237]: pgmap v313: 129 pgs: 129 active+clean; 118 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 222 KiB/s rd, 620 KiB/s wr, 62 op/s 2024-03-31T23:21:48.361 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:21:47 smithi046 ceph-mon[32498]: pgmap v313: 129 pgs: 129 active+clean; 118 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 222 KiB/s rd, 620 KiB/s wr, 62 op/s 2024-03-31T23:21:49.212 INFO:tasks.workunit.client.0.smithi046.stdout: 1 206255 1.98 MB/sec execute 415 sec latency 15.300 ms 2024-03-31T23:21:50.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:21:49 smithi142 ceph-mon[35271]: pgmap v314: 129 pgs: 129 active+clean; 119 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 285 KiB/s rd, 808 KiB/s wr, 79 op/s 2024-03-31T23:21:50.212 INFO:tasks.workunit.client.0.smithi046.stdout: 1 206643 1.98 MB/sec execute 416 sec latency 16.345 ms 2024-03-31T23:21:50.296 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:21:49 smithi162 ceph-mon[36237]: pgmap v314: 129 pgs: 129 active+clean; 119 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 285 KiB/s rd, 808 KiB/s wr, 79 op/s 2024-03-31T23:21:50.361 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:21:49 smithi046 ceph-mon[32498]: pgmap v314: 129 pgs: 129 active+clean; 119 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 285 KiB/s rd, 808 KiB/s wr, 79 op/s 2024-03-31T23:21:51.212 INFO:tasks.workunit.client.0.smithi046.stdout: 1 207153 1.98 MB/sec execute 417 sec latency 21.185 ms 2024-03-31T23:21:51.864 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:51.862+0000 7f737beed640 1 -- 172.21.15.46:0/2968292263 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f737410aa10 msgr2=0x7f7374113160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:21:51.864 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:51.862+0000 7f737beed640 1 --2- 172.21.15.46:0/2968292263 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f737410aa10 0x7f7374113160 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7f737000dbf0 tx=0x7f737002fc10 comp rx=0 tx=0).stop 2024-03-31T23:21:51.864 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:51.862+0000 7f737beed640 1 -- 172.21.15.46:0/2968292263 shutdown_connections 2024-03-31T23:21:51.864 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:51.862+0000 7f737beed640 1 --2- 172.21.15.46:0/2968292263 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f73741136a0 0x7f7374115a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:21:51.864 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:51.862+0000 7f737beed640 1 --2- 172.21.15.46:0/2968292263 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f737410aa10 0x7f7374113160 unknown :-1 s=CLOSED pgs=807 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:21:51.864 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:51.862+0000 7f737beed640 1 --2- 172.21.15.46:0/2968292263 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f737410a0f0 0x7f737410a4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:21:51.864 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:51.862+0000 7f737beed640 1 -- 172.21.15.46:0/2968292263 >> 172.21.15.46:0/2968292263 conn(0x7f737406c7f0 msgr2=0x7f737406cc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:21:51.864 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:51.862+0000 7f737beed640 1 -- 172.21.15.46:0/2968292263 shutdown_connections 2024-03-31T23:21:51.865 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:51.862+0000 7f737beed640 1 -- 172.21.15.46:0/2968292263 wait complete. 2024-03-31T23:21:51.865 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:51.863+0000 7f737beed640 1 Processor -- start 2024-03-31T23:21:51.865 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:51.863+0000 7f737beed640 1 -- start start 2024-03-31T23:21:51.866 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:51.863+0000 7f737beed640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f737410a0f0 0x7f737419e440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:21:51.866 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:51.863+0000 7f737beed640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f737410aa10 0x7f737419e980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:21:51.866 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:51.863+0000 7f737beed640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f73741136a0 0x7f73741a2cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:21:51.866 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:51.863+0000 7f737beed640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f7374117f20 con 0x7f737410aa10 2024-03-31T23:21:51.866 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:51.863+0000 7f737beed640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f7374117da0 con 0x7f73741136a0 2024-03-31T23:21:51.866 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:51.863+0000 7f737beed640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f73741180a0 con 0x7f737410a0f0 2024-03-31T23:21:51.866 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:51.863+0000 7f7379488640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f737410aa10 0x7f737419e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:21:51.867 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:51.863+0000 7f7379c89640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f737410a0f0 0x7f737419e440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:21:51.867 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:51.863+0000 7f7379488640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f737410aa10 0x7f737419e980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:34644/0 (socket says 172.21.15.46:34644) 2024-03-31T23:21:51.867 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:51.863+0000 7f7379488640 1 -- 172.21.15.46:0/412583853 learned_addr learned my addr 172.21.15.46:0/412583853 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:21:51.867 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:51.863+0000 7f737a48a640 1 --2- 172.21.15.46:0/412583853 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f73741136a0 0x7f73741a2cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:21:51.867 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:51.864+0000 7f7379488640 1 -- 172.21.15.46:0/412583853 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f737410a0f0 msgr2=0x7f737419e440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:21:51.867 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:51.864+0000 7f7379488640 1 --2- 172.21.15.46:0/412583853 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f737410a0f0 0x7f737419e440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:21:51.867 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:51.864+0000 7f7379488640 1 -- 172.21.15.46:0/412583853 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f73741136a0 msgr2=0x7f73741a2cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:21:51.867 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:51.864+0000 7f7379488640 1 --2- 172.21.15.46:0/412583853 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f73741136a0 0x7f73741a2cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:21:51.867 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:51.864+0000 7f7379488640 1 -- 172.21.15.46:0/412583853 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f73741a33e0 con 0x7f737410aa10 2024-03-31T23:21:51.867 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:51.864+0000 7f7379c89640 1 --2- 172.21.15.46:0/412583853 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f737410a0f0 0x7f737419e440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:21:51.867 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:51.864+0000 7f7379488640 1 --2- 172.21.15.46:0/412583853 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f737410aa10 0x7f737419e980 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7f737000dd10 tx=0x7f7370030d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:21:51.867 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:51.864+0000 7f736affd640 1 -- 172.21.15.46:0/412583853 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7370046030 con 0x7f737410aa10 2024-03-31T23:21:51.867 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:51.864+0000 7f736affd640 1 -- 172.21.15.46:0/412583853 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7370004250 con 0x7f737410aa10 2024-03-31T23:21:51.868 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:51.864+0000 7f736affd640 1 -- 172.21.15.46:0/412583853 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7370003c50 con 0x7f737410aa10 2024-03-31T23:21:51.868 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:51.864+0000 7f737beed640 1 -- 172.21.15.46:0/412583853 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f73741a3670 con 0x7f737410aa10 2024-03-31T23:21:51.868 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:51.865+0000 7f737beed640 1 -- 172.21.15.46:0/412583853 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f73741aaf60 con 0x7f737410aa10 2024-03-31T23:21:51.868 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:51.867+0000 7f736affd640 1 -- 172.21.15.46:0/412583853 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 25) ==== 137451+0+0 (secure 0 0 0) 0x7f7370049070 con 0x7f737410aa10 2024-03-31T23:21:51.870 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:51.867+0000 7f736affd640 1 --2- 172.21.15.46:0/412583853 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f73600a2730 0x7f73600a4bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:21:51.870 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:51.868+0000 7f737ebad740 1 Processor -- start 2024-03-31T23:21:51.870 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:51.868+0000 7f736affd640 1 -- 172.21.15.46:0/412583853 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 9554+0+0 (secure 0 0 0) 0x7f73700f4930 con 0x7f737410aa10 2024-03-31T23:21:51.870 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:51.868+0000 7f7379c89640 1 --2- 172.21.15.46:0/412583853 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f73600a2730 0x7f73600a4bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:21:51.870 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:51.868+0000 7f737ebad740 1 -- start start 2024-03-31T23:21:51.871 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:51.868+0000 7f737ebad740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5601d56866c0 0x5601d5686aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:21:51.871 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:51.868+0000 7f737ebad740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5601d5686fe0 0x5601d568f870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:21:51.871 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:51.868+0000 7f737ebad740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5601d568fdb0 0x5601d56921a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:21:51.871 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:51.868+0000 7f737ebad740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x5601d55e98f0 con 0x5601d56866c0 2024-03-31T23:21:51.871 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:51.868+0000 7f737ebad740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x5601d55928d0 con 0x5601d568fdb0 2024-03-31T23:21:51.871 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:51.868+0000 7f737ebad740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5601d55e5990 con 0x5601d5686fe0 2024-03-31T23:21:51.871 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:51.868+0000 7f7379c89640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5601d56866c0 0x5601d5686aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:21:51.871 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:51.868+0000 7f737a48a640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5601d568fdb0 0x5601d56921a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:21:51.871 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:51.868+0000 7f7379488640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5601d5686fe0 0x5601d568f870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:21:51.871 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:51.868+0000 7f7379c89640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5601d56866c0 0x5601d5686aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:34658/0 (socket says 172.21.15.46:34658) 2024-03-31T23:21:51.871 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:51.868+0000 7f7379c89640 1 -- 172.21.15.46:0/2547173017 learned_addr learned my addr 172.21.15.46:0/2547173017 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:21:51.871 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:51.868+0000 7f737a48a640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5601d568fdb0 0x5601d56921a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.46:50322/0 (socket says 172.21.15.46:50322) 2024-03-31T23:21:51.871 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:51.868+0000 7f7379c89640 1 --2- 172.21.15.46:0/412583853 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f73600a2730 0x7f73600a4bf0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f7364005fd0 tx=0x7f736400b390 comp rx=0 tx=0).ready entity=mgr.15654 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:21:51.871 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:51.869+0000 7f7379488640 1 -- 172.21.15.46:0/2547173017 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5601d568fdb0 msgr2=0x5601d56921a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:21:51.872 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:51.869+0000 7f7379488640 1 --2- 172.21.15.46:0/2547173017 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5601d568fdb0 0x5601d56921a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:21:51.872 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:51.869+0000 7f7379488640 1 -- 172.21.15.46:0/2547173017 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5601d56866c0 msgr2=0x5601d5686aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:21:51.872 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:51.869+0000 7f7379488640 1 --2- 172.21.15.46:0/2547173017 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5601d56866c0 0x5601d5686aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:21:51.872 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:51.869+0000 7f7379488640 1 -- 172.21.15.46:0/2547173017 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5601d52fe210 con 0x5601d5686fe0 2024-03-31T23:21:51.872 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:51.869+0000 7f7379c89640 1 --2- 172.21.15.46:0/2547173017 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5601d56866c0 0x5601d5686aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:21:51.872 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:51.869+0000 7f7379488640 1 --2- 172.21.15.46:0/2547173017 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5601d5686fe0 0x5601d568f870 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f73700f1d80 tx=0x7f737002fc10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cf0947b81865480b server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:21:51.872 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:51.869+0000 7f735ffff640 1 -- 172.21.15.46:0/2547173017 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73700dd020 con 0x5601d5686fe0 2024-03-31T23:21:51.872 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:51.869+0000 7f735ffff640 1 -- 172.21.15.46:0/2547173017 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f73700c0e70 con 0x5601d5686fe0 2024-03-31T23:21:51.872 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:51.869+0000 7f735ffff640 1 -- 172.21.15.46:0/2547173017 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73700bd4e0 con 0x5601d5686fe0 2024-03-31T23:21:51.876 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:51.870+0000 7f737ebad740 1 -- 172.21.15.46:0/2547173017 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5601d5686fe0 msgr2=0x5601d568f870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:21:51.876 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:51.870+0000 7f737ebad740 1 --2- 172.21.15.46:0/2547173017 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5601d5686fe0 0x5601d568f870 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f73700f1d80 tx=0x7f737002fc10 comp rx=0 tx=0).stop 2024-03-31T23:21:51.876 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:51.870+0000 7f737ebad740 1 -- 172.21.15.46:0/2547173017 shutdown_connections 2024-03-31T23:21:51.877 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:51.870+0000 7f737ebad740 1 --2- 172.21.15.46:0/2547173017 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5601d568fdb0 0x5601d56921a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:21:51.877 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:51.870+0000 7f737ebad740 1 --2- 172.21.15.46:0/2547173017 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5601d5686fe0 0x5601d568f870 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:21:51.877 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:51.870+0000 7f737ebad740 1 --2- 172.21.15.46:0/2547173017 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5601d56866c0 0x5601d5686aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:21:51.877 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:51.870+0000 7f737ebad740 1 -- 172.21.15.46:0/2547173017 >> 172.21.15.46:0/2547173017 conn(0x5601d5680fe0 msgr2=0x5601d56813c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:21:51.877 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:51.870+0000 7f737ebad740 1 -- 172.21.15.46:0/2547173017 shutdown_connections 2024-03-31T23:21:51.877 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:51.870+0000 7f737ebad740 1 -- 172.21.15.46:0/2547173017 wait complete. 2024-03-31T23:21:51.877 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:51.870+0000 7f737ebad740 1 -- start start 2024-03-31T23:21:51.877 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:51.870+0000 7f737ebad740 1 Processor -- start 2024-03-31T23:21:51.877 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:51.871+0000 7f737ebad740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5601d56866c0 0x5601d568ce10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:21:51.877 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:51.871+0000 7f737ebad740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5601d5686fe0 0x5601d568d350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:21:51.877 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:51.871+0000 7f737ebad740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5601d568fdb0 0x5601d568d890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:21:51.877 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:51.871+0000 7f737ebad740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x5601d55e5990 con 0x5601d5686fe0 2024-03-31T23:21:51.877 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:51.871+0000 7f737ebad740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x5601d5587f80 con 0x5601d56866c0 2024-03-31T23:21:51.877 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:51.871+0000 7f737ebad740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5601d56927a0 con 0x5601d568fdb0 2024-03-31T23:21:51.878 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:51.871+0000 7f737ebad740 10 client.0 _ll_register_callbacks cb 0x5601d55e0290 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T23:21:51.878 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:51.871+0000 7f7379c89640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5601d56866c0 0x5601d568ce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:21:51.878 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:51.871+0000 7f7379488640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5601d5686fe0 0x5601d568d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:21:51.878 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:51.871+0000 7f7379c89640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5601d56866c0 0x5601d568ce10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.46:50338/0 (socket says 172.21.15.46:50338) 2024-03-31T23:21:51.878 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:51.871+0000 7f7379c89640 1 -- 172.21.15.46:0/4027661202 learned_addr learned my addr 172.21.15.46:0/4027661202 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:21:51.878 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:51.871+0000 7f737a48a640 1 --2- 172.21.15.46:0/4027661202 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5601d568fdb0 0x5601d568d890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:21:51.878 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:51.871+0000 7f7379488640 1 -- 172.21.15.46:0/4027661202 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5601d568fdb0 msgr2=0x5601d568d890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:21:51.878 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:51.871+0000 7f7379488640 1 --2- 172.21.15.46:0/4027661202 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5601d568fdb0 0x5601d568d890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:21:51.878 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:51.871+0000 7f7379488640 1 -- 172.21.15.46:0/4027661202 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5601d56866c0 msgr2=0x5601d568ce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:21:51.878 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:51.871+0000 7f7379488640 1 --2- 172.21.15.46:0/4027661202 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5601d56866c0 0x5601d568ce10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:21:51.878 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:51.871+0000 7f7379488640 1 -- 172.21.15.46:0/4027661202 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5601d53ca620 con 0x5601d5686fe0 2024-03-31T23:21:51.878 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:51.871+0000 7f7379c89640 1 --2- 172.21.15.46:0/4027661202 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5601d56866c0 0x5601d568ce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:21:51.878 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:51.871+0000 7f7379488640 1 --2- 172.21.15.46:0/4027661202 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5601d5686fe0 0x5601d568d350 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7f7370003780 tx=0x7f7370005da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae1a1ba32fb325e4 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:21:51.879 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:51.871+0000 7f735d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-31T23:21:51.879 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:51.872+0000 7f735d7fa640 1 -- 172.21.15.46:0/4027661202 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73700af070 con 0x5601d5686fe0 2024-03-31T23:21:51.879 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:51.872+0000 7f735d7fa640 1 -- 172.21.15.46:0/4027661202 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 9554+0+0 (secure 0 0 0) 0x7f73700c12e0 con 0x5601d5686fe0 2024-03-31T23:21:51.879 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:51.872+0000 7f737ebad740 1 -- 172.21.15.46:0/4027661202 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f73700c0e70 con 0x5601d5686fe0 2024-03-31T23:21:51.879 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:51.872+0000 7f735d7fa640 1 -- 172.21.15.46:0/4027661202 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f73700dd030 con 0x5601d5686fe0 2024-03-31T23:21:51.879 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:51.872+0000 7f737ebad740 10 client.16974 fetch_fsmap learned FSMap version 36 2024-03-31T23:21:51.879 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:51.872+0000 7f737ebad740 1 -- 172.21.15.46:0/4027661202 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x5601d569e490 con 0x5601d5686fe0 2024-03-31T23:21:51.879 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:51.872+0000 7f735d7fa640 1 -- 172.21.15.46:0/4027661202 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f73700c0e70 con 0x5601d5686fe0 2024-03-31T23:21:51.879 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:51.872+0000 7f735d7fa640 1 -- 172.21.15.46:0/4027661202 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 36) ==== 8362+0+0 (secure 0 0 0) 0x7f73700df2d0 con 0x5601d5686fe0 2024-03-31T23:21:51.879 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:51.872+0000 7f737ebad740 10 client.16974 fetch_fsmap finished waiting for FSMap version 36 2024-03-31T23:21:51.879 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:51.872+0000 7f737ebad740 10 client.16974 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-31T23:21:51.879 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:51.872+0000 7f737ebad740 20 client.16974 populate_metadata read hostname 'smithi046' 2024-03-31T23:21:51.879 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:51.872+0000 7f737ebad740 1 --2- 172.21.15.46:0/4027661202 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x5601d569f6b0 0x5601d56a1ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:21:51.879 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:51.872+0000 7f737ebad740 4 client.16974 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-31T23:21:51.880 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:51.872+0000 7f737ebad740 1 -- 172.21.15.46:0/4027661202 --> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5601d55fd120 con 0x5601d569f6b0 2024-03-31T23:21:51.880 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:51.872+0000 7f7379c89640 1 --2- 172.21.15.46:0/4027661202 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x5601d569f6b0 0x5601d56a1ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:21:51.880 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:51.873+0000 7f7379c89640 1 --2- 172.21.15.46:0/4027661202 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x5601d569f6b0 0x5601d56a1ac0 crc :-1 s=READY pgs=165 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d024b66bf3f96536 server_cookie=6d003e9e221eabb9 in_seq=0 out_seq=0 2024-03-31T23:21:51.880 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:51.873+0000 7f735d7fa640 10 client.16974 ms_handle_connect on v2:172.21.15.142:6838/2519143918 2024-03-31T23:21:51.880 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:51.874+0000 7f735d7fa640 1 -- 172.21.15.46:0/4027661202 <== mds.0 v2:172.21.15.142:6838/2519143918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5601d55fd120 con 0x5601d569f6b0 2024-03-31T23:21:51.880 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:51.874+0000 7f735d7fa640 10 client.16974 handle_command_reply: tid=0 2024-03-31T23:21:51.880 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:51.874+0000 7f737ebad740 1 client.16974 shutdown 2024-03-31T23:21:51.880 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:51.874+0000 7f737ebad740 1 -- 172.21.15.46:0/4027661202 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5601d5686fe0 msgr2=0x5601d568d350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:21:51.880 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:51.874+0000 7f737ebad740 1 --2- 172.21.15.46:0/4027661202 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5601d5686fe0 0x5601d568d350 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7f7370003780 tx=0x7f7370005da0 comp rx=0 tx=0).stop 2024-03-31T23:21:51.880 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:51.874+0000 7f737ebad740 1 -- 172.21.15.46:0/4027661202 shutdown_connections 2024-03-31T23:21:51.880 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:51.874+0000 7f737ebad740 1 --2- 172.21.15.46:0/4027661202 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x5601d569f6b0 0x5601d56a1ac0 crc :-1 s=READY pgs=165 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:21:51.880 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:51.874+0000 7f737ebad740 1 --2- 172.21.15.46:0/4027661202 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5601d568fdb0 0x5601d568d890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:21:51.880 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:51.874+0000 7f737ebad740 1 --2- 172.21.15.46:0/4027661202 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5601d5686fe0 0x5601d568d350 unknown :-1 s=CLOSED pgs=809 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:21:51.880 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:51.874+0000 7f735d7fa640 0 client.16974 ms_handle_reset on v2:172.21.15.142:6838/2519143918 2024-03-31T23:21:51.881 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:51.874+0000 7f737ebad740 1 --2- 172.21.15.46:0/4027661202 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5601d56866c0 0x5601d568ce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:21:51.881 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:51.874+0000 7f737ebad740 1 -- 172.21.15.46:0/4027661202 >> 172.21.15.46:0/4027661202 conn(0x5601d5680fe0 msgr2=0x5601d5682d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:21:51.881 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:51.874+0000 7f737ebad740 1 -- 172.21.15.46:0/4027661202 shutdown_connections 2024-03-31T23:21:51.881 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:51.874+0000 7f737ebad740 1 -- 172.21.15.46:0/4027661202 wait complete. 2024-03-31T23:21:51.881 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:51.875+0000 7f737ebad740 20 client.16974 trim_cache size 0 max 16384 2024-03-31T23:21:51.897 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:51.896+0000 7f737ebad740 1 Processor -- start 2024-03-31T23:21:51.898 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:51.897+0000 7f737ebad740 1 -- start start 2024-03-31T23:21:51.899 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:51.897+0000 7f737ebad740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5601d56866c0 0x5601d568f540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:21:51.899 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:51.897+0000 7f737ebad740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5601d5686fe0 0x5601d569b330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:21:51.899 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:51.897+0000 7f737ebad740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5601d568fdb0 0x5601d56a73d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:21:51.899 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:51.897+0000 7f737ebad740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f73741b7ae0 con 0x5601d568fdb0 2024-03-31T23:21:51.899 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:51.897+0000 7f737ebad740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f73741b5350 con 0x5601d56866c0 2024-03-31T23:21:51.899 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:51.897+0000 7f737ebad740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f73741b31c0 con 0x5601d5686fe0 2024-03-31T23:21:51.899 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:51.897+0000 7f7379488640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5601d5686fe0 0x5601d569b330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:21:51.899 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:51.897+0000 7f7379488640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5601d5686fe0 0x5601d569b330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.46:52260/0 (socket says 172.21.15.46:52260) 2024-03-31T23:21:51.899 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:51.897+0000 7f7379488640 1 -- 172.21.15.46:0/2064327166 learned_addr learned my addr 172.21.15.46:0/2064327166 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:21:51.900 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:51.898+0000 7f737a48a640 1 --2- 172.21.15.46:0/2064327166 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5601d568fdb0 0x5601d56a73d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:21:51.900 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:51.898+0000 7f7379488640 1 -- 172.21.15.46:0/2064327166 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5601d56866c0 msgr2=0x5601d568f540 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-31T23:21:51.900 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:51.898+0000 7f7379488640 1 --2- 172.21.15.46:0/2064327166 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5601d56866c0 0x5601d568f540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:21:51.900 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:51.898+0000 7f7379488640 1 -- 172.21.15.46:0/2064327166 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5601d568fdb0 msgr2=0x5601d56a73d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:21:51.900 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:51.898+0000 7f7379488640 1 --2- 172.21.15.46:0/2064327166 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5601d568fdb0 0x5601d56a73d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:21:51.900 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:51.898+0000 7f7379488640 1 -- 172.21.15.46:0/2064327166 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5601d56a2140 con 0x5601d5686fe0 2024-03-31T23:21:51.901 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:51.899+0000 7f7379488640 1 --2- 172.21.15.46:0/2064327166 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5601d5686fe0 0x5601d569b330 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f73700c0e00 tx=0x7f7370000bc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c1004a38701372a5 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:21:51.901 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:51.899+0000 7f735f7fe640 1 -- 172.21.15.46:0/2064327166 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73700af070 con 0x5601d5686fe0 2024-03-31T23:21:51.901 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:51.899+0000 7f735f7fe640 1 -- 172.21.15.46:0/2064327166 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f73700c1570 con 0x5601d5686fe0 2024-03-31T23:21:51.901 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:51.899+0000 7f735f7fe640 1 -- 172.21.15.46:0/2064327166 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73700c12e0 con 0x5601d5686fe0 2024-03-31T23:21:51.901 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:51.900+0000 7f737ebad740 1 -- 172.21.15.46:0/2064327166 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5601d5686fe0 msgr2=0x5601d569b330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:21:51.901 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:51.900+0000 7f737ebad740 1 --2- 172.21.15.46:0/2064327166 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5601d5686fe0 0x5601d569b330 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f73700c0e00 tx=0x7f7370000bc0 comp rx=0 tx=0).stop 2024-03-31T23:21:51.902 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:51.900+0000 7f737ebad740 1 -- 172.21.15.46:0/2064327166 shutdown_connections 2024-03-31T23:21:51.902 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:51.900+0000 7f737ebad740 1 --2- 172.21.15.46:0/2064327166 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5601d568fdb0 0x5601d56a73d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:21:51.902 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:51.900+0000 7f737ebad740 1 --2- 172.21.15.46:0/2064327166 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5601d5686fe0 0x5601d569b330 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:21:51.902 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:51.900+0000 7f737ebad740 1 --2- 172.21.15.46:0/2064327166 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5601d56866c0 0x5601d568f540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:21:51.902 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:51.900+0000 7f737ebad740 1 -- 172.21.15.46:0/2064327166 >> 172.21.15.46:0/2064327166 conn(0x5601d5680fe0 msgr2=0x5601d56813c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:21:51.902 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:51.901+0000 7f737ebad740 1 -- 172.21.15.46:0/2064327166 shutdown_connections 2024-03-31T23:21:51.902 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:51.901+0000 7f737ebad740 1 -- 172.21.15.46:0/2064327166 wait complete. 2024-03-31T23:21:51.905 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:51.901+0000 7f737ebad740 1 -- start start 2024-03-31T23:21:51.905 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:51.901+0000 7f737ebad740 1 Processor -- start 2024-03-31T23:21:51.905 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:51.902+0000 7f737ebad740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5601d56866c0 0x5601d5693630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:21:51.905 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:51.902+0000 7f737ebad740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5601d5686fe0 0x5601d5693b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:21:51.905 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:51.902+0000 7f737ebad740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5601d568fdb0 0x5601d5704a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:21:51.905 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:51.902+0000 7f737ebad740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f73741b3810 con 0x5601d56866c0 2024-03-31T23:21:51.905 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:51.902+0000 7f737ebad740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f73741b4c80 con 0x5601d568fdb0 2024-03-31T23:21:51.905 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:51.902+0000 7f737ebad740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f73741b3ee0 con 0x5601d5686fe0 2024-03-31T23:21:51.905 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:51.902+0000 7f737ebad740 10 client.0 _ll_register_callbacks cb 0x7f736400bc60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T23:21:51.905 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:51.902+0000 7f737a48a640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5601d568fdb0 0x5601d5704a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:21:51.905 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:51.902+0000 7f737a48a640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5601d568fdb0 0x5601d5704a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.46:50354/0 (socket says 172.21.15.46:50354) 2024-03-31T23:21:51.905 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:51.902+0000 7f7379c89640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5601d56866c0 0x5601d5693630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:21:51.905 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:51.902+0000 7f737a48a640 1 -- 172.21.15.46:0/3754052709 learned_addr learned my addr 172.21.15.46:0/3754052709 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:21:51.906 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:51.902+0000 7f7379488640 1 --2- 172.21.15.46:0/3754052709 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5601d5686fe0 0x5601d5693b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:21:51.906 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:51.902+0000 7f7379c89640 1 -- 172.21.15.46:0/3754052709 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5601d5686fe0 msgr2=0x5601d5693b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:21:51.906 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:51.902+0000 7f7379c89640 1 --2- 172.21.15.46:0/3754052709 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5601d5686fe0 0x5601d5693b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:21:51.906 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:51.902+0000 7f7379c89640 1 -- 172.21.15.46:0/3754052709 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5601d568fdb0 msgr2=0x5601d5704a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:21:51.906 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:51.902+0000 7f7379c89640 1 --2- 172.21.15.46:0/3754052709 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5601d568fdb0 0x5601d5704a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:21:51.906 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:51.902+0000 7f7379c89640 1 -- 172.21.15.46:0/3754052709 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5601d5704f40 con 0x5601d56866c0 2024-03-31T23:21:51.906 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:51.902+0000 7f737a48a640 1 --2- 172.21.15.46:0/3754052709 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5601d568fdb0 0x5601d5704a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:21:51.906 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:51.902+0000 7f7379c89640 1 --2- 172.21.15.46:0/3754052709 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5601d56866c0 0x5601d5693630 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x5601d56a20c0 tx=0x7f7364004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7043efaa3749008a server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:21:51.906 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:51.902+0000 7f735d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-31T23:21:51.906 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:51.902+0000 7f735d7fa640 1 -- 172.21.15.46:0/3754052709 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f736402a620 con 0x5601d56866c0 2024-03-31T23:21:51.906 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:51.903+0000 7f735d7fa640 1 -- 172.21.15.46:0/3754052709 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 9554+0+0 (secure 0 0 0) 0x7f736402a7c0 con 0x5601d56866c0 2024-03-31T23:21:51.906 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:51.903+0000 7f737ebad740 1 -- 172.21.15.46:0/3754052709 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f73700c1570 con 0x5601d56866c0 2024-03-31T23:21:51.906 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:51.903+0000 7f735d7fa640 1 -- 172.21.15.46:0/3754052709 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f7364004020 con 0x5601d56866c0 2024-03-31T23:21:51.907 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:51.903+0000 7f737ebad740 10 client.16980 fetch_fsmap learned FSMap version 36 2024-03-31T23:21:51.907 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:51.903+0000 7f737ebad740 1 -- 172.21.15.46:0/3754052709 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x5601d5706210 con 0x5601d56866c0 2024-03-31T23:21:51.907 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:51.903+0000 7f735d7fa640 1 -- 172.21.15.46:0/3754052709 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f73700c1570 con 0x5601d56866c0 2024-03-31T23:21:51.907 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:51.903+0000 7f735d7fa640 1 -- 172.21.15.46:0/3754052709 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 36) ==== 8362+0+0 (secure 0 0 0) 0x7f73640035f0 con 0x5601d56866c0 2024-03-31T23:21:51.907 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:51.903+0000 7f737ebad740 10 client.16980 fetch_fsmap finished waiting for FSMap version 36 2024-03-31T23:21:51.907 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:51.903+0000 7f737ebad740 10 client.16980 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-31T23:21:51.907 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:51.903+0000 7f737ebad740 20 client.16980 populate_metadata read hostname 'smithi046' 2024-03-31T23:21:51.907 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:51.903+0000 7f737ebad740 1 --2- 172.21.15.46:0/3754052709 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x5601d57068d0 0x5601d5708cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:21:51.907 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:51.903+0000 7f737ebad740 4 client.16980 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-31T23:21:51.907 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:51.903+0000 7f737ebad740 1 -- 172.21.15.46:0/3754052709 --> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5601d55f6e30 con 0x5601d57068d0 2024-03-31T23:21:51.907 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:51.903+0000 7f7379488640 1 --2- 172.21.15.46:0/3754052709 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x5601d57068d0 0x5601d5708cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:21:51.907 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:51.904+0000 7f735d7fa640 10 client.16980 ms_handle_connect on v2:172.21.15.142:6838/2519143918 2024-03-31T23:21:51.907 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:51.904+0000 7f7379488640 1 --2- 172.21.15.46:0/3754052709 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x5601d57068d0 0x5601d5708cf0 crc :-1 s=READY pgs=166 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ea7c58f6697b9d6a server_cookie=9661048b9d61189e in_seq=0 out_seq=0 2024-03-31T23:21:51.934 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:51.933+0000 7f735d7fa640 1 -- 172.21.15.46:0/3754052709 <== mds.0 v2:172.21.15.142:6838/2519143918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5601d55f6e30 con 0x5601d57068d0 2024-03-31T23:21:51.935 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:51.933+0000 7f735d7fa640 10 client.16980 handle_command_reply: tid=0 2024-03-31T23:21:51.935 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:51.933+0000 7f737ebad740 1 client.16980 shutdown 2024-03-31T23:21:51.935 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:51.933+0000 7f737ebad740 1 -- 172.21.15.46:0/3754052709 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5601d56866c0 msgr2=0x5601d5693630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:21:51.935 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:51.933+0000 7f737ebad740 1 --2- 172.21.15.46:0/3754052709 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5601d56866c0 0x5601d5693630 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x5601d56a20c0 tx=0x7f7364004730 comp rx=0 tx=0).stop 2024-03-31T23:21:51.935 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:51.933+0000 7f737ebad740 1 -- 172.21.15.46:0/3754052709 shutdown_connections 2024-03-31T23:21:51.935 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:51.933+0000 7f737ebad740 1 --2- 172.21.15.46:0/3754052709 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x5601d57068d0 0x5601d5708cf0 crc :-1 s=READY pgs=166 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:21:51.935 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:51.933+0000 7f737ebad740 1 --2- 172.21.15.46:0/3754052709 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5601d568fdb0 0x5601d5704a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:21:51.935 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:51.933+0000 7f737ebad740 1 --2- 172.21.15.46:0/3754052709 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5601d5686fe0 0x5601d5693b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:21:51.935 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:51.933+0000 7f737ebad740 1 --2- 172.21.15.46:0/3754052709 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5601d56866c0 0x5601d5693630 unknown :-1 s=CLOSED pgs=810 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:21:51.935 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:51.933+0000 7f737ebad740 1 -- 172.21.15.46:0/3754052709 >> 172.21.15.46:0/3754052709 conn(0x5601d5680fe0 msgr2=0x5601d5682430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:21:51.935 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:51.933+0000 7f735d7fa640 0 client.16980 ms_handle_reset on v2:172.21.15.142:6838/2519143918 2024-03-31T23:21:51.935 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:51.933+0000 7f737ebad740 1 -- 172.21.15.46:0/3754052709 shutdown_connections 2024-03-31T23:21:51.936 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:51.933+0000 7f737ebad740 1 -- 172.21.15.46:0/3754052709 wait complete. 2024-03-31T23:21:51.936 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:51.933+0000 7f737ebad740 20 client.16980 trim_cache size 0 max 16384 2024-03-31T23:21:51.936 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout:{ 2024-03-31T23:21:51.936 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout: "message": "", 2024-03-31T23:21:51.936 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout: "return_code": 0 2024-03-31T23:21:51.936 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout:} 2024-03-31T23:21:51.937 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:51.934+0000 7f737beed640 1 -- 172.21.15.46:0/412583853 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f73600a2730 msgr2=0x7f73600a4bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:21:51.937 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:51.934+0000 7f737beed640 1 --2- 172.21.15.46:0/412583853 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f73600a2730 0x7f73600a4bf0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f7364005fd0 tx=0x7f736400b390 comp rx=0 tx=0).stop 2024-03-31T23:21:51.937 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:51.934+0000 7f737beed640 1 -- 172.21.15.46:0/412583853 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f737410aa10 msgr2=0x7f737419e980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:21:51.937 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:51.934+0000 7f737beed640 1 --2- 172.21.15.46:0/412583853 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f737410aa10 0x7f737419e980 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7f737000dd10 tx=0x7f7370030d10 comp rx=0 tx=0).stop 2024-03-31T23:21:51.937 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:51.934+0000 7f737beed640 1 -- 172.21.15.46:0/412583853 shutdown_connections 2024-03-31T23:21:51.937 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:51.934+0000 7f737beed640 1 --2- 172.21.15.46:0/412583853 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f73600a2730 0x7f73600a4bf0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:21:51.937 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:51.934+0000 7f737beed640 1 --2- 172.21.15.46:0/412583853 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f73741136a0 0x7f73741a2cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:21:51.938 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:51.934+0000 7f737beed640 1 --2- 172.21.15.46:0/412583853 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f737410aa10 0x7f737419e980 unknown :-1 s=CLOSED pgs=808 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:21:51.938 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:51.934+0000 7f737beed640 1 --2- 172.21.15.46:0/412583853 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f737410a0f0 0x7f737419e440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:21:51.938 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:51.934+0000 7f737beed640 1 -- 172.21.15.46:0/412583853 >> 172.21.15.46:0/412583853 conn(0x7f737406c7f0 msgr2=0x7f73740703e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:21:51.938 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:51.934+0000 7f737beed640 1 -- 172.21.15.46:0/412583853 shutdown_connections 2024-03-31T23:21:51.938 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:21:51.934+0000 7f737beed640 1 -- 172.21.15.46:0/412583853 wait complete. 2024-03-31T23:21:52.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:21:51 smithi142 ceph-mon[35271]: pgmap v315: 129 pgs: 129 active+clean; 119 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 241 KiB/s wr, 32 op/s 2024-03-31T23:21:52.212 INFO:tasks.workunit.client.0.smithi046.stdout: 1 207597 1.98 MB/sec execute 418 sec latency 26.775 ms 2024-03-31T23:21:52.296 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:21:51 smithi162 ceph-mon[36237]: pgmap v315: 129 pgs: 129 active+clean; 119 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 241 KiB/s wr, 32 op/s 2024-03-31T23:21:52.361 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:21:51 smithi046 ceph-mon[32498]: pgmap v315: 129 pgs: 129 active+clean; 119 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 241 KiB/s wr, 32 op/s 2024-03-31T23:21:53.212 INFO:tasks.workunit.client.0.smithi046.stdout: 1 208119 1.98 MB/sec execute 419 sec latency 21.672 ms 2024-03-31T23:21:54.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:21:53 smithi142 ceph-mon[35271]: pgmap v316: 129 pgs: 129 active+clean; 119 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 241 KiB/s wr, 32 op/s 2024-03-31T23:21:54.212 INFO:tasks.workunit.client.0.smithi046.stdout: 1 208703 1.99 MB/sec execute 420 sec latency 16.319 ms 2024-03-31T23:21:54.296 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:21:53 smithi162 ceph-mon[36237]: pgmap v316: 129 pgs: 129 active+clean; 119 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 241 KiB/s wr, 32 op/s 2024-03-31T23:21:54.361 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:21:53 smithi046 ceph-mon[32498]: pgmap v316: 129 pgs: 129 active+clean; 119 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 241 KiB/s wr, 32 op/s 2024-03-31T23:21:55.212 INFO:tasks.workunit.client.0.smithi046.stdout: 1 209077 1.99 MB/sec execute 421 sec latency 16.468 ms 2024-03-31T23:21:56.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:21:55 smithi142 ceph-mon[35271]: pgmap v317: 129 pgs: 129 active+clean; 116 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 326 KiB/s rd, 777 KiB/s wr, 90 op/s 2024-03-31T23:21:56.212 INFO:tasks.workunit.client.0.smithi046.stdout: 1 209346 1.99 MB/sec execute 422 sec latency 575.824 ms 2024-03-31T23:21:56.295 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:21:55 smithi162 ceph-mon[36237]: pgmap v317: 129 pgs: 129 active+clean; 116 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 326 KiB/s rd, 777 KiB/s wr, 90 op/s 2024-03-31T23:21:56.361 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:21:55 smithi046 ceph-mon[32498]: pgmap v317: 129 pgs: 129 active+clean; 116 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 326 KiB/s rd, 777 KiB/s wr, 90 op/s 2024-03-31T23:21:57.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:21:56 smithi142 ceph-mon[35271]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-31T23:21:57.212 INFO:tasks.workunit.client.0.smithi046.stdout: 1 209346 1.98 MB/sec execute 423 sec latency 1575.931 ms 2024-03-31T23:21:57.295 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:21:56 smithi162 ceph-mon[36237]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-31T23:21:57.361 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:21:56 smithi046 ceph-mon[32498]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-31T23:21:58.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:21:57 smithi046 ceph-mon[32498]: pgmap v318: 129 pgs: 129 active+clean; 116 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 285 KiB/s rd, 724 KiB/s wr, 75 op/s 2024-03-31T23:21:58.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:21:57 smithi142 ceph-mon[35271]: pgmap v318: 129 pgs: 129 active+clean; 116 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 285 KiB/s rd, 724 KiB/s wr, 75 op/s 2024-03-31T23:21:58.212 INFO:tasks.workunit.client.0.smithi046.stdout: 1 209413 1.98 MB/sec execute 424 sec latency 2108.133 ms 2024-03-31T23:21:58.295 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:21:57 smithi162 ceph-mon[36237]: pgmap v318: 129 pgs: 129 active+clean; 116 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 285 KiB/s rd, 724 KiB/s wr, 75 op/s 2024-03-31T23:21:59.212 INFO:tasks.workunit.client.0.smithi046.stdout: 1 209413 1.97 MB/sec execute 425 sec latency 1204.787 ms 2024-03-31T23:21:59.612 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:21:59 smithi046 ceph-mon[32498]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' 2024-03-31T23:21:59.612 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:21:59 smithi046 ceph-mon[32498]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' 2024-03-31T23:21:59.612 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:21:59 smithi046 ceph-mon[32498]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' 2024-03-31T23:21:59.612 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:21:59 smithi046 ceph-mon[32498]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' 2024-03-31T23:21:59.612 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:21:59 smithi046 ceph-mon[32498]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' 2024-03-31T23:21:59.612 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:21:59 smithi046 ceph-mon[32498]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' 2024-03-31T23:21:59.612 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:21:59 smithi046 ceph-mon[32498]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-31T23:21:59.612 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:21:59 smithi046 ceph-mon[32498]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-31T23:21:59.612 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:21:59 smithi046 ceph-mon[32498]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-31T23:21:59.612 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:21:59 smithi046 ceph-mon[32498]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-31T23:21:59.613 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:21:59 smithi046 ceph-mon[32498]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-31T23:21:59.613 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:21:59 smithi046 ceph-mon[32498]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-31T23:21:59.613 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:21:59 smithi046 ceph-mon[32498]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-31T23:21:59.613 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:21:59 smithi046 ceph-mon[32498]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-31T23:21:59.613 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:21:59 smithi046 ceph-mon[32498]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-31T23:21:59.613 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:21:59 smithi046 ceph-mon[32498]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-31T23:21:59.613 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:21:59 smithi046 ceph-mon[32498]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-31T23:21:59.613 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:21:59 smithi046 ceph-mon[32498]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-31T23:21:59.613 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:21:59 smithi046 ceph-mon[32498]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:21:59.613 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:21:59 smithi046 ceph-mon[32498]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-31T23:21:59.613 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:21:59 smithi046 ceph-mon[32498]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' 2024-03-31T23:21:59.694 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:21:59 smithi142 ceph-mon[35271]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' 2024-03-31T23:21:59.694 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:21:59 smithi142 ceph-mon[35271]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' 2024-03-31T23:21:59.694 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:21:59 smithi142 ceph-mon[35271]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' 2024-03-31T23:21:59.694 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:21:59 smithi142 ceph-mon[35271]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' 2024-03-31T23:21:59.694 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:21:59 smithi142 ceph-mon[35271]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' 2024-03-31T23:21:59.694 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:21:59 smithi142 ceph-mon[35271]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' 2024-03-31T23:21:59.694 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:21:59 smithi142 ceph-mon[35271]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-31T23:21:59.694 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:21:59 smithi142 ceph-mon[35271]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-31T23:21:59.694 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:21:59 smithi142 ceph-mon[35271]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-31T23:21:59.694 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:21:59 smithi142 ceph-mon[35271]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-31T23:21:59.695 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:21:59 smithi142 ceph-mon[35271]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-31T23:21:59.695 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:21:59 smithi142 ceph-mon[35271]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-31T23:21:59.695 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:21:59 smithi142 ceph-mon[35271]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-31T23:21:59.695 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:21:59 smithi142 ceph-mon[35271]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-31T23:21:59.695 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:21:59 smithi142 ceph-mon[35271]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-31T23:21:59.695 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:21:59 smithi142 ceph-mon[35271]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-31T23:21:59.695 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:21:59 smithi142 ceph-mon[35271]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-31T23:21:59.695 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:21:59 smithi142 ceph-mon[35271]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-31T23:21:59.695 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:21:59 smithi142 ceph-mon[35271]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:21:59.695 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:21:59 smithi142 ceph-mon[35271]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-31T23:21:59.695 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:21:59 smithi142 ceph-mon[35271]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' 2024-03-31T23:21:59.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:21:59 smithi162 ceph-mon[36237]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' 2024-03-31T23:21:59.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:21:59 smithi162 ceph-mon[36237]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' 2024-03-31T23:21:59.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:21:59 smithi162 ceph-mon[36237]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' 2024-03-31T23:21:59.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:21:59 smithi162 ceph-mon[36237]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' 2024-03-31T23:21:59.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:21:59 smithi162 ceph-mon[36237]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' 2024-03-31T23:21:59.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:21:59 smithi162 ceph-mon[36237]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' 2024-03-31T23:21:59.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:21:59 smithi162 ceph-mon[36237]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-31T23:21:59.797 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:21:59 smithi162 ceph-mon[36237]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-31T23:21:59.797 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:21:59 smithi162 ceph-mon[36237]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-31T23:21:59.797 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:21:59 smithi162 ceph-mon[36237]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-31T23:21:59.797 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:21:59 smithi162 ceph-mon[36237]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-31T23:21:59.797 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:21:59 smithi162 ceph-mon[36237]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-31T23:21:59.797 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:21:59 smithi162 ceph-mon[36237]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-31T23:21:59.797 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:21:59 smithi162 ceph-mon[36237]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-31T23:21:59.797 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:21:59 smithi162 ceph-mon[36237]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-31T23:21:59.797 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:21:59 smithi162 ceph-mon[36237]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-31T23:21:59.797 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:21:59 smithi162 ceph-mon[36237]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-31T23:21:59.797 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:21:59 smithi162 ceph-mon[36237]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-31T23:21:59.797 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:21:59 smithi162 ceph-mon[36237]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:21:59.797 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:21:59 smithi162 ceph-mon[36237]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-31T23:21:59.797 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:21:59 smithi162 ceph-mon[36237]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' 2024-03-31T23:22:00.213 INFO:tasks.workunit.client.0.smithi046.stdout: 1 209413 1.97 MB/sec execute 426 sec latency 2204.920 ms 2024-03-31T23:22:00.612 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:22:00 smithi046 ceph-mon[32498]: pgmap v319: 129 pgs: 129 active+clean; 119 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 382 KiB/s rd, 1.0 MiB/s wr, 102 op/s 2024-03-31T23:22:00.612 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:22:00 smithi046 ceph-mon[32498]: Adjusting osd_memory_target on smithi162 to 176.2M 2024-03-31T23:22:00.612 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:22:00 smithi046 ceph-mon[32498]: Unable to set osd_memory_target on smithi162 to 184788889: error parsing value: Value '184788889' is below minimum 939524096 2024-03-31T23:22:00.612 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:22:00 smithi046 ceph-mon[32498]: Adjusting osd_memory_target on smithi046 to 177.2M 2024-03-31T23:22:00.612 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:22:00 smithi046 ceph-mon[32498]: Unable to set osd_memory_target on smithi046 to 185899212: error parsing value: Value '185899212' is below minimum 939524096 2024-03-31T23:22:00.612 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:22:00 smithi046 ceph-mon[32498]: Adjusting osd_memory_target on smithi142 to 176.2M 2024-03-31T23:22:00.613 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:22:00 smithi046 ceph-mon[32498]: Unable to set osd_memory_target on smithi142 to 184770252: error parsing value: Value '184770252' is below minimum 939524096 2024-03-31T23:22:00.694 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:22:00 smithi142 ceph-mon[35271]: pgmap v319: 129 pgs: 129 active+clean; 119 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 382 KiB/s rd, 1.0 MiB/s wr, 102 op/s 2024-03-31T23:22:00.694 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:22:00 smithi142 ceph-mon[35271]: Adjusting osd_memory_target on smithi162 to 176.2M 2024-03-31T23:22:00.694 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:22:00 smithi142 ceph-mon[35271]: Unable to set osd_memory_target on smithi162 to 184788889: error parsing value: Value '184788889' is below minimum 939524096 2024-03-31T23:22:00.694 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:22:00 smithi142 ceph-mon[35271]: Adjusting osd_memory_target on smithi046 to 177.2M 2024-03-31T23:22:00.694 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:22:00 smithi142 ceph-mon[35271]: Unable to set osd_memory_target on smithi046 to 185899212: error parsing value: Value '185899212' is below minimum 939524096 2024-03-31T23:22:00.694 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:22:00 smithi142 ceph-mon[35271]: Adjusting osd_memory_target on smithi142 to 176.2M 2024-03-31T23:22:00.694 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:22:00 smithi142 ceph-mon[35271]: Unable to set osd_memory_target on smithi142 to 184770252: error parsing value: Value '184770252' is below minimum 939524096 2024-03-31T23:22:00.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:22:00 smithi162 ceph-mon[36237]: pgmap v319: 129 pgs: 129 active+clean; 119 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 382 KiB/s rd, 1.0 MiB/s wr, 102 op/s 2024-03-31T23:22:00.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:22:00 smithi162 ceph-mon[36237]: Adjusting osd_memory_target on smithi162 to 176.2M 2024-03-31T23:22:00.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:22:00 smithi162 ceph-mon[36237]: Unable to set osd_memory_target on smithi162 to 184788889: error parsing value: Value '184788889' is below minimum 939524096 2024-03-31T23:22:00.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:22:00 smithi162 ceph-mon[36237]: Adjusting osd_memory_target on smithi046 to 177.2M 2024-03-31T23:22:00.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:22:00 smithi162 ceph-mon[36237]: Unable to set osd_memory_target on smithi046 to 185899212: error parsing value: Value '185899212' is below minimum 939524096 2024-03-31T23:22:00.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:22:00 smithi162 ceph-mon[36237]: Adjusting osd_memory_target on smithi142 to 176.2M 2024-03-31T23:22:00.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:22:00 smithi162 ceph-mon[36237]: Unable to set osd_memory_target on smithi142 to 184770252: error parsing value: Value '184770252' is below minimum 939524096 2024-03-31T23:22:01.213 INFO:tasks.workunit.client.0.smithi046.stdout: 1 209413 1.97 MB/sec execute 427 sec latency 3205.010 ms 2024-03-31T23:22:02.213 INFO:tasks.workunit.client.0.smithi046.stdout: 1 209413 1.96 MB/sec execute 428 sec latency 4205.112 ms 2024-03-31T23:22:02.611 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:22:02 smithi046 ceph-mon[32498]: pgmap v320: 129 pgs: 129 active+clean; 119 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 319 KiB/s rd, 856 KiB/s wr, 85 op/s 2024-03-31T23:22:02.693 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:22:02 smithi142 ceph-mon[35271]: pgmap v320: 129 pgs: 129 active+clean; 119 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 319 KiB/s rd, 856 KiB/s wr, 85 op/s 2024-03-31T23:22:02.795 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:22:02 smithi162 ceph-mon[36237]: pgmap v320: 129 pgs: 129 active+clean; 119 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 319 KiB/s rd, 856 KiB/s wr, 85 op/s 2024-03-31T23:22:03.213 INFO:tasks.workunit.client.0.smithi046.stdout: 1 209538 1.96 MB/sec execute 429 sec latency 4734.468 ms 2024-03-31T23:22:04.213 INFO:tasks.workunit.client.0.smithi046.stdout: 1 209800 1.95 MB/sec execute 430 sec latency 15.061 ms 2024-03-31T23:22:04.611 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:22:04 smithi046 ceph-mon[32498]: pgmap v321: 129 pgs: 129 active+clean; 119 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 319 KiB/s rd, 856 KiB/s wr, 85 op/s 2024-03-31T23:22:04.693 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:22:04 smithi142 ceph-mon[35271]: pgmap v321: 129 pgs: 129 active+clean; 119 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 319 KiB/s rd, 856 KiB/s wr, 85 op/s 2024-03-31T23:22:04.795 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:22:04 smithi162 ceph-mon[36237]: pgmap v321: 129 pgs: 129 active+clean; 119 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 319 KiB/s rd, 856 KiB/s wr, 85 op/s 2024-03-31T23:22:05.035 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:05.031+0000 7f256b223640 1 -- 172.21.15.46:0/1190376850 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2564101260 msgr2=0x7f256410d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:22:05.035 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:05.031+0000 7f256b223640 1 --2- 172.21.15.46:0/1190376850 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2564101260 0x7f256410d480 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7f255c009ca0 tx=0x7f255c02c0f0 comp rx=0 tx=0).stop 2024-03-31T23:22:05.035 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:05.032+0000 7f256b223640 1 -- 172.21.15.46:0/1190376850 shutdown_connections 2024-03-31T23:22:05.035 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:05.032+0000 7f256b223640 1 --2- 172.21.15.46:0/1190376850 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2564101260 0x7f256410d480 unknown :-1 s=CLOSED pgs=811 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:22:05.035 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:05.032+0000 7f256b223640 1 --2- 172.21.15.46:0/1190376850 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f25641008a0 0x7f2564100d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:22:05.035 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:05.032+0000 7f256b223640 1 --2- 172.21.15.46:0/1190376850 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f25641055f0 0x7f25641059d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:22:05.035 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:05.032+0000 7f256b223640 1 -- 172.21.15.46:0/1190376850 >> 172.21.15.46:0/1190376850 conn(0x7f25640774a0 msgr2=0x7f25640778b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:22:05.036 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:05.032+0000 7f256b223640 1 -- 172.21.15.46:0/1190376850 shutdown_connections 2024-03-31T23:22:05.036 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:05.032+0000 7f256b223640 1 -- 172.21.15.46:0/1190376850 wait complete. 2024-03-31T23:22:05.036 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:05.032+0000 7f256b223640 1 Processor -- start 2024-03-31T23:22:05.036 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:05.032+0000 7f256b223640 1 -- start start 2024-03-31T23:22:05.036 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:05.032+0000 7f256b223640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f25641008a0 0x7f2564197ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:22:05.036 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:05.032+0000 7f256b223640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2564101260 0x7f2564198410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:22:05.036 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:05.032+0000 7f256b223640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f25641055f0 0x7f256419c790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:22:05.036 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:05.032+0000 7f256b223640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f25641109c0 con 0x7f25641055f0 2024-03-31T23:22:05.036 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:05.032+0000 7f256b223640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f2564110840 con 0x7f2564101260 2024-03-31T23:22:05.036 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:05.032+0000 7f256b223640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f2564110b40 con 0x7f25641008a0 2024-03-31T23:22:05.036 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:05.033+0000 7f25697c0640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f25641055f0 0x7f256419c790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:22:05.036 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:05.033+0000 7f2568fbf640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f25641008a0 0x7f2564197ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:22:05.036 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:05.033+0000 7f25697c0640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f25641055f0 0x7f256419c790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37840/0 (socket says 172.21.15.46:37840) 2024-03-31T23:22:05.036 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:05.033+0000 7f25697c0640 1 -- 172.21.15.46:0/1598000157 learned_addr learned my addr 172.21.15.46:0/1598000157 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:22:05.037 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:05.033+0000 7f2563fff640 1 --2- 172.21.15.46:0/1598000157 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2564101260 0x7f2564198410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:22:05.037 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:05.033+0000 7f25697c0640 1 -- 172.21.15.46:0/1598000157 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f25641008a0 msgr2=0x7f2564197ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:22:05.037 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:05.033+0000 7f25697c0640 1 --2- 172.21.15.46:0/1598000157 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f25641008a0 0x7f2564197ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:22:05.037 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:05.033+0000 7f25697c0640 1 -- 172.21.15.46:0/1598000157 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2564101260 msgr2=0x7f2564198410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:22:05.037 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:05.033+0000 7f25697c0640 1 --2- 172.21.15.46:0/1598000157 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2564101260 0x7f2564198410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:22:05.037 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:05.033+0000 7f25697c0640 1 -- 172.21.15.46:0/1598000157 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f256419cec0 con 0x7f25641055f0 2024-03-31T23:22:05.037 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:05.033+0000 7f25697c0640 1 --2- 172.21.15.46:0/1598000157 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f25641055f0 0x7f256419c790 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7f255c02c700 tx=0x7f255c035d60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:22:05.037 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:05.033+0000 7f2561ffb640 1 -- 172.21.15.46:0/1598000157 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f255c039040 con 0x7f25641055f0 2024-03-31T23:22:05.037 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:05.033+0000 7f2561ffb640 1 -- 172.21.15.46:0/1598000157 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f255c044070 con 0x7f25641055f0 2024-03-31T23:22:05.037 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:05.033+0000 7f256b223640 1 -- 172.21.15.46:0/1598000157 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f256419d150 con 0x7f25641055f0 2024-03-31T23:22:05.037 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:05.033+0000 7f2561ffb640 1 -- 172.21.15.46:0/1598000157 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f255c0042a0 con 0x7f25641055f0 2024-03-31T23:22:05.037 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:05.034+0000 7f256b223640 1 -- 172.21.15.46:0/1598000157 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f25641a4a40 con 0x7f25641055f0 2024-03-31T23:22:05.045 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:05.036+0000 7f2561ffb640 1 -- 172.21.15.46:0/1598000157 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 25) ==== 137451+0+0 (secure 0 0 0) 0x7f255c004440 con 0x7f25641055f0 2024-03-31T23:22:05.045 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:05.036+0000 7f256de06740 1 Processor -- start 2024-03-31T23:22:05.045 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:05.036+0000 7f256de06740 1 -- start start 2024-03-31T23:22:05.045 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:05.036+0000 7f256de06740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b28942f6c0 0x55b28942faa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:22:05.045 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:05.036+0000 7f256de06740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55b28942ffe0 0x55b289438870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:22:05.045 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:05.036+0000 7f256de06740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b289438db0 0x55b28943b1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:22:05.045 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:05.036+0000 7f256de06740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55b2893928f0 con 0x55b28942f6c0 2024-03-31T23:22:05.045 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:05.036+0000 7f256de06740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55b28933b8d0 con 0x55b28942ffe0 2024-03-31T23:22:05.045 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:05.036+0000 7f256de06740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55b28938e990 con 0x55b289438db0 2024-03-31T23:22:05.045 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:05.036+0000 7f2561ffb640 1 --2- 172.21.15.46:0/1598000157 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f25540a2730 0x7f25540a4bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:22:05.046 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:05.036+0000 7f2563fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b28942f6c0 0x55b28942faa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:22:05.046 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:05.037+0000 7f2563fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b28942f6c0 0x55b28942faa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37848/0 (socket says 172.21.15.46:37848) 2024-03-31T23:22:05.046 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:05.037+0000 7f2568fbf640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55b28942ffe0 0x55b289438870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:22:05.046 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:05.037+0000 7f2563fff640 1 -- 172.21.15.46:0/52625367 learned_addr learned my addr 172.21.15.46:0/52625367 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:22:05.046 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:05.037+0000 7f2563fff640 1 --2- 172.21.15.46:0/52625367 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b289438db0 0x55b28943b1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:22:05.046 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:05.037+0000 7f2561ffb640 1 -- 172.21.15.46:0/1598000157 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 9554+0+0 (secure 0 0 0) 0x7f255c0f1190 con 0x7f25641055f0 2024-03-31T23:22:05.046 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:05.037+0000 7f25697c0640 1 --2- 172.21.15.46:0/1598000157 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f25540a2730 0x7f25540a4bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:22:05.046 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:05.037+0000 7f2568fbf640 1 -- 172.21.15.46:0/52625367 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b289438db0 msgr2=0x55b28943b1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:22:05.046 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:05.037+0000 7f2568fbf640 1 --2- 172.21.15.46:0/52625367 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b289438db0 0x55b28943b1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:22:05.046 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:05.037+0000 7f2568fbf640 1 -- 172.21.15.46:0/52625367 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b28942f6c0 msgr2=0x55b28942faa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:22:05.046 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:05.037+0000 7f2568fbf640 1 --2- 172.21.15.46:0/52625367 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b28942f6c0 0x55b28942faa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:22:05.046 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:05.037+0000 7f2568fbf640 1 -- 172.21.15.46:0/52625367 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b2890a7210 con 0x55b28942ffe0 2024-03-31T23:22:05.046 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:05.037+0000 7f2568fbf640 1 --2- 172.21.15.46:0/52625367 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55b28942ffe0 0x55b289438870 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f2558009c20 tx=0x7f2558011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8254a7967e3cc63c server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:22:05.047 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:05.037+0000 7f25697c0640 1 --2- 172.21.15.46:0/1598000157 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f25540a2730 0x7f25540a4bf0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f255c0ee640 tx=0x7f255c02c0f0 comp rx=0 tx=0).ready entity=mgr.15654 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:22:05.047 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:05.037+0000 7f254effd640 1 -- 172.21.15.46:0/52625367 <== mon.2 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2558004050 con 0x55b28942ffe0 2024-03-31T23:22:05.047 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:05.037+0000 7f254effd640 1 -- 172.21.15.46:0/52625367 <== mon.2 v2:172.21.15.142:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f25580054d0 con 0x55b28942ffe0 2024-03-31T23:22:05.047 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:05.037+0000 7f254effd640 1 -- 172.21.15.46:0/52625367 <== mon.2 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2558004c60 con 0x55b28942ffe0 2024-03-31T23:22:05.047 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:05.038+0000 7f256de06740 1 -- 172.21.15.46:0/52625367 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55b28942ffe0 msgr2=0x55b289438870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:22:05.047 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:05.038+0000 7f256de06740 1 --2- 172.21.15.46:0/52625367 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55b28942ffe0 0x55b289438870 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f2558009c20 tx=0x7f2558011810 comp rx=0 tx=0).stop 2024-03-31T23:22:05.047 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:05.038+0000 7f256de06740 1 -- 172.21.15.46:0/52625367 shutdown_connections 2024-03-31T23:22:05.047 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:05.038+0000 7f256de06740 1 --2- 172.21.15.46:0/52625367 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b289438db0 0x55b28943b1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:22:05.047 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:05.038+0000 7f256de06740 1 --2- 172.21.15.46:0/52625367 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55b28942ffe0 0x55b289438870 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:22:05.047 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:05.038+0000 7f256de06740 1 --2- 172.21.15.46:0/52625367 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b28942f6c0 0x55b28942faa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:22:05.047 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:05.038+0000 7f256de06740 1 -- 172.21.15.46:0/52625367 >> 172.21.15.46:0/52625367 conn(0x55b289429fe0 msgr2=0x55b28942a3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:22:05.047 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:05.038+0000 7f256de06740 1 -- 172.21.15.46:0/52625367 shutdown_connections 2024-03-31T23:22:05.047 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:05.038+0000 7f256de06740 1 -- 172.21.15.46:0/52625367 wait complete. 2024-03-31T23:22:05.047 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:05.038+0000 7f256de06740 1 -- start start 2024-03-31T23:22:05.048 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:05.038+0000 7f256de06740 1 Processor -- start 2024-03-31T23:22:05.048 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:05.039+0000 7f256de06740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b28942f6c0 0x55b289435d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:22:05.048 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:05.039+0000 7f256de06740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b28942ffe0 0x55b289436250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:22:05.048 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:05.039+0000 7f256de06740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55b289438db0 0x55b289436790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:22:05.048 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:05.039+0000 7f256de06740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55b28943b9e0 con 0x55b28942f6c0 2024-03-31T23:22:05.048 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:05.039+0000 7f256de06740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55b289330f80 con 0x55b289438db0 2024-03-31T23:22:05.048 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:05.039+0000 7f256de06740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55b28943bb60 con 0x55b28942ffe0 2024-03-31T23:22:05.048 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:05.039+0000 7f256de06740 10 client.0 _ll_register_callbacks cb 0x55b289389290 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T23:22:05.048 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:05.039+0000 7f2563fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b28942f6c0 0x55b289435d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:22:05.048 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:05.039+0000 7f2563fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b28942f6c0 0x55b289435d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37854/0 (socket says 172.21.15.46:37854) 2024-03-31T23:22:05.048 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:05.039+0000 7f2563fff640 1 -- 172.21.15.46:0/3801604282 learned_addr learned my addr 172.21.15.46:0/3801604282 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:22:05.048 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:05.039+0000 7f2568fbf640 1 --2- 172.21.15.46:0/3801604282 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b28942ffe0 0x55b289436250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:22:05.048 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:05.039+0000 7f2563fff640 1 --2- 172.21.15.46:0/3801604282 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55b289438db0 0x55b289436790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:22:05.048 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:05.039+0000 7f2568fbf640 1 -- 172.21.15.46:0/3801604282 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55b289438db0 msgr2=0x55b289436790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:22:05.049 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:05.039+0000 7f2568fbf640 1 --2- 172.21.15.46:0/3801604282 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55b289438db0 0x55b289436790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:22:05.049 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:05.039+0000 7f2568fbf640 1 -- 172.21.15.46:0/3801604282 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b28942f6c0 msgr2=0x55b289435d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:22:05.049 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:05.039+0000 7f2568fbf640 1 --2- 172.21.15.46:0/3801604282 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b28942f6c0 0x55b289435d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:22:05.049 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:05.039+0000 7f2568fbf640 1 -- 172.21.15.46:0/3801604282 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b289173620 con 0x55b28942ffe0 2024-03-31T23:22:05.049 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:05.039+0000 7f2563fff640 1 --2- 172.21.15.46:0/3801604282 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b28942f6c0 0x55b289435d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:22:05.049 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:05.040+0000 7f2568fbf640 1 --2- 172.21.15.46:0/3801604282 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b28942ffe0 0x55b289436250 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f2558000c00 tx=0x7f2558003820 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4b1ecb05f8cc1563 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:22:05.049 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:05.040+0000 7f2543fff640 10 client.0 ms_handle_connect on v2:172.21.15.162:3300/0 2024-03-31T23:22:05.049 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:05.040+0000 7f2543fff640 1 -- 172.21.15.46:0/3801604282 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2558003c20 con 0x55b28942ffe0 2024-03-31T23:22:05.049 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:05.040+0000 7f2543fff640 1 -- 172.21.15.46:0/3801604282 <== mon.1 v2:172.21.15.162:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 9554+0+0 (secure 0 0 0) 0x7f2558003dc0 con 0x55b28942ffe0 2024-03-31T23:22:05.049 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:05.040+0000 7f256de06740 1 -- 172.21.15.46:0/3801604282 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f25580054d0 con 0x55b28942ffe0 2024-03-31T23:22:05.049 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:05.040+0000 7f2543fff640 1 -- 172.21.15.46:0/3801604282 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f2558034050 con 0x55b28942ffe0 2024-03-31T23:22:05.049 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:05.040+0000 7f256de06740 10 client.26722 fetch_fsmap learned FSMap version 36 2024-03-31T23:22:05.049 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:05.040+0000 7f256de06740 1 -- 172.21.15.46:0/3801604282 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x55b289447850 con 0x55b28942ffe0 2024-03-31T23:22:05.049 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:05.040+0000 7f2543fff640 1 -- 172.21.15.46:0/3801604282 <== mon.1 v2:172.21.15.162:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f25580054d0 con 0x55b28942ffe0 2024-03-31T23:22:05.050 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:05.040+0000 7f2543fff640 1 -- 172.21.15.46:0/3801604282 <== mon.1 v2:172.21.15.162:3300/0 5 ==== fsmap(e 36) ==== 8362+0+0 (secure 0 0 0) 0x7f2558023430 con 0x55b28942ffe0 2024-03-31T23:22:05.050 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:05.040+0000 7f256de06740 10 client.26722 fetch_fsmap finished waiting for FSMap version 36 2024-03-31T23:22:05.050 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:05.040+0000 7f256de06740 10 client.26722 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-31T23:22:05.050 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:05.040+0000 7f256de06740 20 client.26722 populate_metadata read hostname 'smithi046' 2024-03-31T23:22:05.050 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:05.040+0000 7f256de06740 1 --2- 172.21.15.46:0/3801604282 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55b289448cb0 0x55b28944b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:22:05.050 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:05.040+0000 7f256de06740 4 client.26722 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-31T23:22:05.050 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:05.040+0000 7f256de06740 1 -- 172.21.15.46:0/3801604282 --> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b2893a6120 con 0x55b289448cb0 2024-03-31T23:22:05.050 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:05.041+0000 7f2563fff640 1 --2- 172.21.15.46:0/3801604282 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55b289448cb0 0x55b28944b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:22:05.050 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:05.041+0000 7f2563fff640 1 --2- 172.21.15.46:0/3801604282 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55b289448cb0 0x55b28944b0a0 crc :-1 s=READY pgs=167 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=38dfe7d4845b7e90 server_cookie=bb267e634bd0406c in_seq=0 out_seq=0 2024-03-31T23:22:05.050 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:05.041+0000 7f2543fff640 10 client.26722 ms_handle_connect on v2:172.21.15.142:6838/2519143918 2024-03-31T23:22:05.050 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:05.042+0000 7f2543fff640 1 -- 172.21.15.46:0/3801604282 <== mds.0 v2:172.21.15.142:6838/2519143918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55b2893a6120 con 0x55b289448cb0 2024-03-31T23:22:05.050 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:05.042+0000 7f2543fff640 10 client.26722 handle_command_reply: tid=0 2024-03-31T23:22:05.050 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:05.042+0000 7f256de06740 1 client.26722 shutdown 2024-03-31T23:22:05.050 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:05.042+0000 7f256de06740 1 -- 172.21.15.46:0/3801604282 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b28942ffe0 msgr2=0x55b289436250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:22:05.051 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:05.042+0000 7f256de06740 1 --2- 172.21.15.46:0/3801604282 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b28942ffe0 0x55b289436250 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f2558000c00 tx=0x7f2558003820 comp rx=0 tx=0).stop 2024-03-31T23:22:05.051 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:05.042+0000 7f256de06740 1 -- 172.21.15.46:0/3801604282 shutdown_connections 2024-03-31T23:22:05.051 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:05.042+0000 7f256de06740 1 --2- 172.21.15.46:0/3801604282 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55b289448cb0 0x55b28944b0a0 crc :-1 s=READY pgs=167 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:22:05.051 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:05.042+0000 7f256de06740 1 --2- 172.21.15.46:0/3801604282 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55b289438db0 0x55b289436790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:22:05.051 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:05.042+0000 7f256de06740 1 --2- 172.21.15.46:0/3801604282 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b28942ffe0 0x55b289436250 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:22:05.051 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:05.042+0000 7f256de06740 1 --2- 172.21.15.46:0/3801604282 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b28942f6c0 0x55b289435d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:22:05.051 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:05.042+0000 7f2543fff640 0 client.26722 ms_handle_reset on v2:172.21.15.142:6838/2519143918 2024-03-31T23:22:05.051 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:05.042+0000 7f256de06740 1 -- 172.21.15.46:0/3801604282 >> 172.21.15.46:0/3801604282 conn(0x55b289429fe0 msgr2=0x55b28942bd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:22:05.051 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:05.042+0000 7f256de06740 1 -- 172.21.15.46:0/3801604282 shutdown_connections 2024-03-31T23:22:05.051 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:05.042+0000 7f256de06740 1 -- 172.21.15.46:0/3801604282 wait complete. 2024-03-31T23:22:05.051 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:05.042+0000 7f256de06740 20 client.26722 trim_cache size 0 max 16384 2024-03-31T23:22:05.060 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:05.057+0000 7f256de06740 1 Processor -- start 2024-03-31T23:22:05.060 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:05.057+0000 7f256de06740 1 -- start start 2024-03-31T23:22:05.060 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:05.057+0000 7f256de06740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b28942f6c0 0x55b2894447b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:22:05.060 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:05.057+0000 7f256de06740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55b28942ffe0 0x55b289434e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:22:05.060 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:05.057+0000 7f256de06740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b289438db0 0x55b289450920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:22:05.060 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:05.057+0000 7f256de06740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f25641b1410 con 0x55b289438db0 2024-03-31T23:22:05.061 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:05.057+0000 7f256de06740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f25641aec80 con 0x55b28942ffe0 2024-03-31T23:22:05.061 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:05.057+0000 7f256de06740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f25641acaf0 con 0x55b28942f6c0 2024-03-31T23:22:05.061 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:05.058+0000 7f2563fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b289438db0 0x55b289450920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:22:05.061 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:05.058+0000 7f2568fbf640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55b28942ffe0 0x55b289434e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:22:05.061 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:05.058+0000 7f2568fbf640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55b28942ffe0 0x55b289434e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.46:35694/0 (socket says 172.21.15.46:35694) 2024-03-31T23:22:05.062 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:05.058+0000 7f2568fbf640 1 -- 172.21.15.46:0/577550268 learned_addr learned my addr 172.21.15.46:0/577550268 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:22:05.062 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:05.058+0000 7f2563fff640 1 --2- 172.21.15.46:0/577550268 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b28942f6c0 0x55b2894447b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:22:05.062 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:05.058+0000 7f2568fbf640 1 -- 172.21.15.46:0/577550268 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b28942f6c0 msgr2=0x55b2894447b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:22:05.062 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:05.058+0000 7f2568fbf640 1 --2- 172.21.15.46:0/577550268 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b28942f6c0 0x55b2894447b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:22:05.062 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:05.058+0000 7f2568fbf640 1 -- 172.21.15.46:0/577550268 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b289438db0 msgr2=0x55b289450920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:22:05.063 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:05.058+0000 7f2568fbf640 1 --2- 172.21.15.46:0/577550268 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b289438db0 0x55b289450920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:22:05.063 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:05.058+0000 7f2568fbf640 1 -- 172.21.15.46:0/577550268 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b28942fe10 con 0x55b28942ffe0 2024-03-31T23:22:05.063 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:05.058+0000 7f2568fbf640 1 --2- 172.21.15.46:0/577550268 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55b28942ffe0 0x55b289434e70 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f2558004020 tx=0x7f2558005970 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9ed7d5495c8ab239 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:22:05.063 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:05.058+0000 7f254e7fc640 1 -- 172.21.15.46:0/577550268 <== mon.2 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f255803d070 con 0x55b28942ffe0 2024-03-31T23:22:05.063 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:05.058+0000 7f254e7fc640 1 -- 172.21.15.46:0/577550268 <== mon.2 v2:172.21.15.142:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f2558023430 con 0x55b28942ffe0 2024-03-31T23:22:05.063 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:05.058+0000 7f254e7fc640 1 -- 172.21.15.46:0/577550268 <== mon.2 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2558003c20 con 0x55b28942ffe0 2024-03-31T23:22:05.066 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:05.059+0000 7f256de06740 1 -- 172.21.15.46:0/577550268 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55b28942ffe0 msgr2=0x55b289434e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:22:05.066 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:05.059+0000 7f256de06740 1 --2- 172.21.15.46:0/577550268 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55b28942ffe0 0x55b289434e70 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f2558004020 tx=0x7f2558005970 comp rx=0 tx=0).stop 2024-03-31T23:22:05.066 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:05.059+0000 7f256de06740 1 -- 172.21.15.46:0/577550268 shutdown_connections 2024-03-31T23:22:05.066 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:05.059+0000 7f256de06740 1 --2- 172.21.15.46:0/577550268 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b289438db0 0x55b289450920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:22:05.066 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:05.059+0000 7f256de06740 1 --2- 172.21.15.46:0/577550268 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55b28942ffe0 0x55b289434e70 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:22:05.067 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:05.059+0000 7f256de06740 1 --2- 172.21.15.46:0/577550268 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b28942f6c0 0x55b2894447b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:22:05.067 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:05.059+0000 7f256de06740 1 -- 172.21.15.46:0/577550268 >> 172.21.15.46:0/577550268 conn(0x55b289429fe0 msgr2=0x55b28942a3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:22:05.067 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:05.059+0000 7f256de06740 1 -- 172.21.15.46:0/577550268 shutdown_connections 2024-03-31T23:22:05.067 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:05.059+0000 7f256de06740 1 -- 172.21.15.46:0/577550268 wait complete. 2024-03-31T23:22:05.067 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:05.059+0000 7f256de06740 1 -- start start 2024-03-31T23:22:05.067 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:05.059+0000 7f256de06740 1 Processor -- start 2024-03-31T23:22:05.068 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:05.060+0000 7f256de06740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55b28942f6c0 0x55b289440920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:22:05.068 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:05.060+0000 7f256de06740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b28942ffe0 0x55b289440e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:22:05.068 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:05.060+0000 7f256de06740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b289438db0 0x55b28943bde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:22:05.068 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:05.060+0000 7f256de06740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f25641ad140 con 0x55b289438db0 2024-03-31T23:22:05.068 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:05.060+0000 7f256de06740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f25641ae5b0 con 0x55b28942f6c0 2024-03-31T23:22:05.068 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:05.060+0000 7f256de06740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f25641ad810 con 0x55b28942ffe0 2024-03-31T23:22:05.068 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:05.060+0000 7f256de06740 10 client.0 _ll_register_callbacks cb 0x7f2550009ad0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T23:22:05.068 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:05.060+0000 7f2563fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b289438db0 0x55b28943bde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:22:05.068 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:05.060+0000 7f2563fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b289438db0 0x55b28943bde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37858/0 (socket says 172.21.15.46:37858) 2024-03-31T23:22:05.068 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:05.060+0000 7f2563fff640 1 -- 172.21.15.46:0/1771152491 learned_addr learned my addr 172.21.15.46:0/1771152491 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:22:05.068 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:05.060+0000 7f2563fff640 1 --2- 172.21.15.46:0/1771152491 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55b28942f6c0 0x55b289440920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:22:05.068 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:05.060+0000 7f2568fbf640 1 --2- 172.21.15.46:0/1771152491 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b28942ffe0 0x55b289440e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:22:05.068 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:05.060+0000 7f2563fff640 1 -- 172.21.15.46:0/1771152491 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b28942ffe0 msgr2=0x55b289440e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:22:05.069 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:05.060+0000 7f2563fff640 1 --2- 172.21.15.46:0/1771152491 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b28942ffe0 0x55b289440e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:22:05.069 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:05.060+0000 7f2563fff640 1 -- 172.21.15.46:0/1771152491 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55b28942f6c0 msgr2=0x55b289440920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:22:05.069 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:05.060+0000 7f2563fff640 1 --2- 172.21.15.46:0/1771152491 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55b28942f6c0 0x55b289440920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:22:05.069 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:05.060+0000 7f2563fff640 1 -- 172.21.15.46:0/1771152491 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b28943c520 con 0x55b289438db0 2024-03-31T23:22:05.069 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:05.060+0000 7f2563fff640 1 --2- 172.21.15.46:0/1771152491 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b289438db0 0x55b28943bde0 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x55b28944b860 tx=0x7f2550004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e8e4cdb8e76ce381 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:22:05.069 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:05.060+0000 7f2543fff640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-31T23:22:05.069 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:05.061+0000 7f2543fff640 1 -- 172.21.15.46:0/1771152491 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2550005020 con 0x55b289438db0 2024-03-31T23:22:05.069 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:05.061+0000 7f256de06740 1 -- 172.21.15.46:0/1771152491 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2558023430 con 0x55b289438db0 2024-03-31T23:22:05.069 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:05.061+0000 7f2543fff640 1 -- 172.21.15.46:0/1771152491 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 9554+0+0 (secure 0 0 0) 0x7f25500051c0 con 0x55b289438db0 2024-03-31T23:22:05.069 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:05.061+0000 7f2543fff640 1 -- 172.21.15.46:0/1771152491 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f2550038050 con 0x55b289438db0 2024-03-31T23:22:05.069 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:05.061+0000 7f256de06740 10 client.16998 fetch_fsmap learned FSMap version 36 2024-03-31T23:22:05.069 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:05.061+0000 7f256de06740 1 -- 172.21.15.46:0/1771152491 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x55b2894ae960 con 0x55b289438db0 2024-03-31T23:22:05.070 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:05.061+0000 7f2543fff640 1 -- 172.21.15.46:0/1771152491 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f2558023430 con 0x55b289438db0 2024-03-31T23:22:05.070 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:05.061+0000 7f2543fff640 1 -- 172.21.15.46:0/1771152491 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 36) ==== 8362+0+0 (secure 0 0 0) 0x7f25500036b0 con 0x55b289438db0 2024-03-31T23:22:05.070 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:05.061+0000 7f256de06740 10 client.16998 fetch_fsmap finished waiting for FSMap version 36 2024-03-31T23:22:05.070 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:05.061+0000 7f256de06740 10 client.16998 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-31T23:22:05.070 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:05.061+0000 7f256de06740 20 client.16998 populate_metadata read hostname 'smithi046' 2024-03-31T23:22:05.070 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:05.061+0000 7f256de06740 1 --2- 172.21.15.46:0/1771152491 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55b2894af110 0x55b2894b1530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:22:05.070 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:05.061+0000 7f256de06740 4 client.16998 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-31T23:22:05.070 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:05.061+0000 7f256de06740 1 -- 172.21.15.46:0/1771152491 --> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55b28939fe30 con 0x55b2894af110 2024-03-31T23:22:05.070 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:05.061+0000 7f2568fbf640 1 --2- 172.21.15.46:0/1771152491 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55b2894af110 0x55b2894b1530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:22:05.070 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:05.062+0000 7f2543fff640 10 client.16998 ms_handle_connect on v2:172.21.15.142:6838/2519143918 2024-03-31T23:22:05.071 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:05.062+0000 7f2568fbf640 1 --2- 172.21.15.46:0/1771152491 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55b2894af110 0x55b2894b1530 crc :-1 s=READY pgs=168 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=72d7ee6bb16dab9e server_cookie=b9f49c466cb00bd0 in_seq=0 out_seq=0 2024-03-31T23:22:05.092 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:05.091+0000 7f2543fff640 1 -- 172.21.15.46:0/1771152491 <== mds.0 v2:172.21.15.142:6838/2519143918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55b28939fe30 con 0x55b2894af110 2024-03-31T23:22:05.093 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:05.091+0000 7f2543fff640 10 client.16998 handle_command_reply: tid=0 2024-03-31T23:22:05.093 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:05.091+0000 7f256de06740 1 client.16998 shutdown 2024-03-31T23:22:05.093 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:05.091+0000 7f256de06740 1 -- 172.21.15.46:0/1771152491 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b289438db0 msgr2=0x55b28943bde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:22:05.093 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:05.091+0000 7f256de06740 1 --2- 172.21.15.46:0/1771152491 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b289438db0 0x55b28943bde0 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x55b28944b860 tx=0x7f2550004730 comp rx=0 tx=0).stop 2024-03-31T23:22:05.093 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:05.091+0000 7f256de06740 1 -- 172.21.15.46:0/1771152491 shutdown_connections 2024-03-31T23:22:05.093 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:05.091+0000 7f256de06740 1 --2- 172.21.15.46:0/1771152491 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55b2894af110 0x55b2894b1530 crc :-1 s=READY pgs=168 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:22:05.093 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:05.091+0000 7f256de06740 1 --2- 172.21.15.46:0/1771152491 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b289438db0 0x55b28943bde0 unknown :-1 s=CLOSED pgs=813 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:22:05.093 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:05.091+0000 7f256de06740 1 --2- 172.21.15.46:0/1771152491 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b28942ffe0 0x55b289440e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:22:05.093 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:05.091+0000 7f256de06740 1 --2- 172.21.15.46:0/1771152491 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55b28942f6c0 0x55b289440920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:22:05.093 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:05.091+0000 7f2543fff640 0 client.16998 ms_handle_reset on v2:172.21.15.142:6838/2519143918 2024-03-31T23:22:05.093 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:05.091+0000 7f256de06740 1 -- 172.21.15.46:0/1771152491 >> 172.21.15.46:0/1771152491 conn(0x55b289429fe0 msgr2=0x55b289434e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:22:05.093 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:05.091+0000 7f256de06740 1 -- 172.21.15.46:0/1771152491 shutdown_connections 2024-03-31T23:22:05.094 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:05.091+0000 7f256de06740 1 -- 172.21.15.46:0/1771152491 wait complete. 2024-03-31T23:22:05.094 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:05.091+0000 7f256de06740 20 client.16998 trim_cache size 0 max 16384 2024-03-31T23:22:05.094 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout:{ 2024-03-31T23:22:05.094 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout: "message": "", 2024-03-31T23:22:05.094 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout: "return_code": 0 2024-03-31T23:22:05.094 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout:} 2024-03-31T23:22:05.095 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:05.092+0000 7f256b223640 1 -- 172.21.15.46:0/1598000157 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f25540a2730 msgr2=0x7f25540a4bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:22:05.095 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:05.092+0000 7f256b223640 1 --2- 172.21.15.46:0/1598000157 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f25540a2730 0x7f25540a4bf0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f255c0ee640 tx=0x7f255c02c0f0 comp rx=0 tx=0).stop 2024-03-31T23:22:05.095 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:05.092+0000 7f256b223640 1 -- 172.21.15.46:0/1598000157 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f25641055f0 msgr2=0x7f256419c790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:22:05.095 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:05.092+0000 7f256b223640 1 --2- 172.21.15.46:0/1598000157 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f25641055f0 0x7f256419c790 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7f255c02c700 tx=0x7f255c035d60 comp rx=0 tx=0).stop 2024-03-31T23:22:05.095 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:05.092+0000 7f256b223640 1 -- 172.21.15.46:0/1598000157 shutdown_connections 2024-03-31T23:22:05.095 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:05.092+0000 7f256b223640 1 --2- 172.21.15.46:0/1598000157 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f25540a2730 0x7f25540a4bf0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:22:05.095 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:05.092+0000 7f256b223640 1 --2- 172.21.15.46:0/1598000157 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f25641055f0 0x7f256419c790 unknown :-1 s=CLOSED pgs=812 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:22:05.096 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:05.092+0000 7f256b223640 1 --2- 172.21.15.46:0/1598000157 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2564101260 0x7f2564198410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:22:05.096 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:05.092+0000 7f256b223640 1 --2- 172.21.15.46:0/1598000157 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f25641008a0 0x7f2564197ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:22:05.096 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:05.092+0000 7f256b223640 1 -- 172.21.15.46:0/1598000157 >> 172.21.15.46:0/1598000157 conn(0x7f25640774a0 msgr2=0x7f2564069010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:22:05.096 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:05.092+0000 7f256b223640 1 -- 172.21.15.46:0/1598000157 shutdown_connections 2024-03-31T23:22:05.096 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:05.092+0000 7f256b223640 1 -- 172.21.15.46:0/1598000157 wait complete. 2024-03-31T23:22:05.213 INFO:tasks.workunit.client.0.smithi046.stdout: 1 210192 1.95 MB/sec execute 431 sec latency 30.432 ms 2024-03-31T23:22:06.213 INFO:tasks.workunit.client.0.smithi046.stdout: 1 210688 1.96 MB/sec execute 432 sec latency 16.862 ms 2024-03-31T23:22:06.693 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:22:06 smithi142 ceph-mon[35271]: pgmap v322: 129 pgs: 129 active+clean; 121 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 402 KiB/s rd, 1.1 MiB/s wr, 118 op/s 2024-03-31T23:22:06.795 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:22:06 smithi162 ceph-mon[36237]: pgmap v322: 129 pgs: 129 active+clean; 121 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 402 KiB/s rd, 1.1 MiB/s wr, 118 op/s 2024-03-31T23:22:06.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:22:06 smithi046 ceph-mon[32498]: pgmap v322: 129 pgs: 129 active+clean; 121 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 402 KiB/s rd, 1.1 MiB/s wr, 118 op/s 2024-03-31T23:22:07.213 INFO:tasks.workunit.client.0.smithi046.stdout: 1 211139 1.95 MB/sec execute 433 sec latency 25.456 ms 2024-03-31T23:22:08.213 INFO:tasks.workunit.client.0.smithi046.stdout: 1 211678 1.95 MB/sec execute 434 sec latency 33.629 ms 2024-03-31T23:22:08.693 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:22:08 smithi142 ceph-mon[35271]: pgmap v323: 129 pgs: 129 active+clean; 121 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 179 KiB/s rd, 554 KiB/s wr, 60 op/s 2024-03-31T23:22:08.693 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:22:08 smithi142 ceph-mon[35271]: osdmap e105: 12 total, 12 up, 12 in 2024-03-31T23:22:08.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:22:08 smithi162 ceph-mon[36237]: pgmap v323: 129 pgs: 129 active+clean; 121 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 179 KiB/s rd, 554 KiB/s wr, 60 op/s 2024-03-31T23:22:08.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:22:08 smithi162 ceph-mon[36237]: osdmap e105: 12 total, 12 up, 12 in 2024-03-31T23:22:08.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:22:08 smithi046 ceph-mon[32498]: pgmap v323: 129 pgs: 129 active+clean; 121 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 179 KiB/s rd, 554 KiB/s wr, 60 op/s 2024-03-31T23:22:08.862 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:22:08 smithi046 ceph-mon[32498]: osdmap e105: 12 total, 12 up, 12 in 2024-03-31T23:22:09.213 INFO:tasks.workunit.client.0.smithi046.stdout: 1 212259 1.96 MB/sec execute 435 sec latency 15.849 ms 2024-03-31T23:22:10.213 INFO:tasks.workunit.client.0.smithi046.stdout: 1 212661 1.96 MB/sec execute 436 sec latency 16.014 ms 2024-03-31T23:22:10.663 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:22:10 smithi046 ceph-mon[32498]: pgmap v325: 129 pgs: 129 active+clean; 123 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 385 KiB/s rd, 947 KiB/s wr, 111 op/s 2024-03-31T23:22:10.694 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:22:10 smithi142 ceph-mon[35271]: pgmap v325: 129 pgs: 129 active+clean; 123 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 385 KiB/s rd, 947 KiB/s wr, 111 op/s 2024-03-31T23:22:10.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:22:10 smithi162 ceph-mon[36237]: pgmap v325: 129 pgs: 129 active+clean; 123 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 385 KiB/s rd, 947 KiB/s wr, 111 op/s 2024-03-31T23:22:11.213 INFO:tasks.workunit.client.0.smithi046.stdout: 1 212892 1.96 MB/sec execute 437 sec latency 675.404 ms 2024-03-31T23:22:11.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:22:11 smithi162 ceph-mon[36237]: osdmap e106: 12 total, 12 up, 12 in 2024-03-31T23:22:11.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:22:11 smithi046 ceph-mon[32498]: osdmap e106: 12 total, 12 up, 12 in 2024-03-31T23:22:11.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:22:11 smithi142 ceph-mon[35271]: osdmap e106: 12 total, 12 up, 12 in 2024-03-31T23:22:12.213 INFO:tasks.workunit.client.0.smithi046.stdout: 1 212892 1.96 MB/sec execute 438 sec latency 1675.488 ms 2024-03-31T23:22:12.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:22:12 smithi162 ceph-mon[36237]: pgmap v327: 129 pgs: 129 active+clean; 123 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 482 KiB/s rd, 1.2 MiB/s wr, 139 op/s 2024-03-31T23:22:12.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:22:12 smithi046 ceph-mon[32498]: pgmap v327: 129 pgs: 129 active+clean; 123 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 482 KiB/s rd, 1.2 MiB/s wr, 139 op/s 2024-03-31T23:22:12.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:22:12 smithi142 ceph-mon[35271]: pgmap v327: 129 pgs: 129 active+clean; 123 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 482 KiB/s rd, 1.2 MiB/s wr, 139 op/s 2024-03-31T23:22:13.213 INFO:tasks.workunit.client.0.smithi046.stdout: 1 212959 1.96 MB/sec execute 439 sec latency 2206.440 ms 2024-03-31T23:22:14.214 INFO:tasks.workunit.client.0.smithi046.stdout: 1 212959 1.95 MB/sec execute 440 sec latency 1215.763 ms 2024-03-31T23:22:14.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:22:14 smithi162 ceph-mon[36237]: pgmap v328: 129 pgs: 129 active+clean; 123 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 357 KiB/s rd, 832 KiB/s wr, 89 op/s 2024-03-31T23:22:14.862 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:22:14 smithi046 ceph-mon[32498]: pgmap v328: 129 pgs: 129 active+clean; 123 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 357 KiB/s rd, 832 KiB/s wr, 89 op/s 2024-03-31T23:22:14.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:22:14 smithi142 ceph-mon[35271]: pgmap v328: 129 pgs: 129 active+clean; 123 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 357 KiB/s rd, 832 KiB/s wr, 89 op/s 2024-03-31T23:22:15.214 INFO:tasks.workunit.client.0.smithi046.stdout: 1 212959 1.95 MB/sec execute 441 sec latency 2215.866 ms 2024-03-31T23:22:16.214 INFO:tasks.workunit.client.0.smithi046.stdout: 1 212959 1.94 MB/sec execute 442 sec latency 3215.978 ms 2024-03-31T23:22:16.795 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:22:16 smithi162 ceph-mon[36237]: pgmap v329: 129 pgs: 129 active+clean; 126 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 496 KiB/s rd, 1.3 MiB/s wr, 130 op/s 2024-03-31T23:22:16.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:22:16 smithi046 ceph-mon[32498]: pgmap v329: 129 pgs: 129 active+clean; 126 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 496 KiB/s rd, 1.3 MiB/s wr, 130 op/s 2024-03-31T23:22:16.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:22:16 smithi142 ceph-mon[35271]: pgmap v329: 129 pgs: 129 active+clean; 126 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 496 KiB/s rd, 1.3 MiB/s wr, 130 op/s 2024-03-31T23:22:17.214 INFO:tasks.workunit.client.0.smithi046.stdout: 1 213204 1.94 MB/sec execute 443 sec latency 3301.032 ms 2024-03-31T23:22:18.188 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:18.184+0000 7f16bfb8a640 1 -- 172.21.15.46:0/2514622793 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f16b81066b0 msgr2=0x7f16b8106a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:22:18.188 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:18.184+0000 7f16bfb8a640 1 --2- 172.21.15.46:0/2514622793 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f16b81066b0 0x7f16b8106a90 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f16b4009860 tx=0x7f16b402bb70 comp rx=0 tx=0).stop 2024-03-31T23:22:18.188 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:18.184+0000 7f16bfb8a640 1 -- 172.21.15.46:0/2514622793 shutdown_connections 2024-03-31T23:22:18.188 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:18.184+0000 7f16bfb8a640 1 --2- 172.21.15.46:0/2514622793 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f16b8100fd0 0x7f16b810d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:22:18.188 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:18.184+0000 7f16bfb8a640 1 --2- 172.21.15.46:0/2514622793 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f16b8100610 0x7f16b8100a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:22:18.188 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:18.184+0000 7f16bfb8a640 1 --2- 172.21.15.46:0/2514622793 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f16b81066b0 0x7f16b8106a90 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:22:18.188 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:18.184+0000 7f16bfb8a640 1 -- 172.21.15.46:0/2514622793 >> 172.21.15.46:0/2514622793 conn(0x7f16b8077770 msgr2=0x7f16b8077b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:22:18.188 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:18.184+0000 7f16bfb8a640 1 -- 172.21.15.46:0/2514622793 shutdown_connections 2024-03-31T23:22:18.188 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:18.184+0000 7f16bfb8a640 1 -- 172.21.15.46:0/2514622793 wait complete. 2024-03-31T23:22:18.188 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:18.185+0000 7f16bfb8a640 1 Processor -- start 2024-03-31T23:22:18.188 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:18.185+0000 7f16bfb8a640 1 -- start start 2024-03-31T23:22:18.189 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:18.185+0000 7f16bfb8a640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f16b8100610 0x7f16b8197dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:22:18.189 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:18.185+0000 7f16bfb8a640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f16b8100fd0 0x7f16b8198300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:22:18.189 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:18.185+0000 7f16bfb8a640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f16b81066b0 0x7f16b819c680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:22:18.189 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:18.185+0000 7f16bfb8a640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f16b81119e0 con 0x7f16b81066b0 2024-03-31T23:22:18.189 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:18.185+0000 7f16bfb8a640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f16b8111860 con 0x7f16b8100610 2024-03-31T23:22:18.189 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:18.185+0000 7f16bfb8a640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f16b8111b60 con 0x7f16b8100fd0 2024-03-31T23:22:18.189 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:18.185+0000 7f16be127640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f16b81066b0 0x7f16b819c680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:22:18.189 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:18.185+0000 7f16be127640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f16b81066b0 0x7f16b819c680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40780/0 (socket says 172.21.15.46:40780) 2024-03-31T23:22:18.189 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:18.185+0000 7f16be127640 1 -- 172.21.15.46:0/3931854436 learned_addr learned my addr 172.21.15.46:0/3931854436 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:22:18.189 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:18.185+0000 7f16bd926640 1 --2- 172.21.15.46:0/3931854436 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f16b8100610 0x7f16b8197dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:22:18.189 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:18.185+0000 7f16bd125640 1 --2- 172.21.15.46:0/3931854436 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f16b8100fd0 0x7f16b8198300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:22:18.189 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:18.186+0000 7f16be127640 1 -- 172.21.15.46:0/3931854436 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f16b8100fd0 msgr2=0x7f16b8198300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:22:18.189 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:18.186+0000 7f16be127640 1 --2- 172.21.15.46:0/3931854436 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f16b8100fd0 0x7f16b8198300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:22:18.189 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:18.186+0000 7f16be127640 1 -- 172.21.15.46:0/3931854436 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f16b8100610 msgr2=0x7f16b8197dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:22:18.190 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:18.186+0000 7f16be127640 1 --2- 172.21.15.46:0/3931854436 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f16b8100610 0x7f16b8197dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:22:18.190 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:18.186+0000 7f16be127640 1 -- 172.21.15.46:0/3931854436 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f16b819cdb0 con 0x7f16b81066b0 2024-03-31T23:22:18.190 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:18.186+0000 7f16bd926640 1 --2- 172.21.15.46:0/3931854436 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f16b8100610 0x7f16b8197dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:22:18.190 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:18.186+0000 7f16be127640 1 --2- 172.21.15.46:0/3931854436 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f16b81066b0 0x7f16b819c680 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7f16b000eb10 tx=0x7f16b000ef10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:22:18.190 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:18.186+0000 7f16aeffd640 1 -- 172.21.15.46:0/3931854436 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16b000ce20 con 0x7f16b81066b0 2024-03-31T23:22:18.190 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:18.186+0000 7f16bfb8a640 1 -- 172.21.15.46:0/3931854436 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f16b819d0a0 con 0x7f16b81066b0 2024-03-31T23:22:18.190 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:18.186+0000 7f16aeffd640 1 -- 172.21.15.46:0/3931854436 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f16b0005020 con 0x7f16b81066b0 2024-03-31T23:22:18.190 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:18.186+0000 7f16aeffd640 1 -- 172.21.15.46:0/3931854436 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16b0010690 con 0x7f16b81066b0 2024-03-31T23:22:18.190 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:18.186+0000 7f16bfb8a640 1 -- 172.21.15.46:0/3931854436 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f16b810dea0 con 0x7f16b81066b0 2024-03-31T23:22:18.193 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:18.189+0000 7f16aeffd640 1 -- 172.21.15.46:0/3931854436 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 25) ==== 137451+0+0 (secure 0 0 0) 0x7f16b00051d0 con 0x7f16b81066b0 2024-03-31T23:22:18.193 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:18.189+0000 7f16c2768740 1 Processor -- start 2024-03-31T23:22:18.193 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:18.190+0000 7f16c2768740 1 -- start start 2024-03-31T23:22:18.193 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:18.190+0000 7f16c2768740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555a80aca6c0 0x555a80acaaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:22:18.193 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:18.190+0000 7f16c2768740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x555a80acafe0 0x555a80ad3870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:22:18.193 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:18.190+0000 7f16aeffd640 1 --2- 172.21.15.46:0/3931854436 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f16a40a26b0 0x7f16a40a4b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:22:18.193 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:18.190+0000 7f16c2768740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555a80ad3db0 0x555a80ad61a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:22:18.193 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:18.190+0000 7f16c2768740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x555a80a2d8f0 con 0x555a80ad3db0 2024-03-31T23:22:18.194 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:18.190+0000 7f16c2768740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x555a809d68d0 con 0x555a80acafe0 2024-03-31T23:22:18.194 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:18.190+0000 7f16c2768740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x555a80a29990 con 0x555a80aca6c0 2024-03-31T23:22:18.194 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:18.190+0000 7f16be127640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555a80ad3db0 0x555a80ad61a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:22:18.194 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:18.190+0000 7f16bd926640 1 --2- 172.21.15.46:0/3931854436 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f16a40a26b0 0x7f16a40a4b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:22:18.194 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:18.190+0000 7f16bd125640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555a80aca6c0 0x555a80acaaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:22:18.194 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:18.190+0000 7f16bd125640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555a80aca6c0 0x555a80acaaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.46:52612/0 (socket says 172.21.15.46:52612) 2024-03-31T23:22:18.194 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:18.190+0000 7f16be127640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555a80ad3db0 0x555a80ad61a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40786/0 (socket says 172.21.15.46:40786) 2024-03-31T23:22:18.194 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:18.190+0000 7f16bd125640 1 -- 172.21.15.46:0/2184609426 learned_addr learned my addr 172.21.15.46:0/2184609426 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:22:18.194 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:18.190+0000 7f16aeffd640 1 -- 172.21.15.46:0/3931854436 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 9554+0+0 (secure 0 0 0) 0x7f16b00cf620 con 0x7f16b81066b0 2024-03-31T23:22:18.194 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:18.190+0000 7f16bd125640 1 --2- 172.21.15.46:0/2184609426 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x555a80acafe0 0x555a80ad3870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:22:18.194 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:18.190+0000 7f16bd125640 1 -- 172.21.15.46:0/2184609426 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x555a80acafe0 msgr2=0x555a80ad3870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:22:18.194 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:18.190+0000 7f16bd125640 1 --2- 172.21.15.46:0/2184609426 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x555a80acafe0 0x555a80ad3870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:22:18.194 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:18.190+0000 7f16bd125640 1 -- 172.21.15.46:0/2184609426 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555a80ad3db0 msgr2=0x555a80ad61a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:22:18.195 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:18.190+0000 7f16bd125640 1 --2- 172.21.15.46:0/2184609426 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555a80ad3db0 0x555a80ad61a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:22:18.195 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:18.190+0000 7f16bd125640 1 -- 172.21.15.46:0/2184609426 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555a80742210 con 0x555a80aca6c0 2024-03-31T23:22:18.195 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:18.191+0000 7f16bd926640 1 --2- 172.21.15.46:0/3931854436 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f16a40a26b0 0x7f16a40a4b70 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f16b4009830 tx=0x7f16b40058d0 comp rx=0 tx=0).ready entity=mgr.15654 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:22:18.195 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:18.191+0000 7f16bd125640 1 --2- 172.21.15.46:0/2184609426 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555a80aca6c0 0x555a80acaaa0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f16a80058c0 tx=0x7f16a8005850 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ce0c940d7f69183f server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:22:18.195 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:18.191+0000 7f16a3fff640 1 -- 172.21.15.46:0/2184609426 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16a8025070 con 0x555a80aca6c0 2024-03-31T23:22:18.195 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:18.191+0000 7f16a3fff640 1 -- 172.21.15.46:0/2184609426 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f16a8020420 con 0x555a80aca6c0 2024-03-31T23:22:18.195 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:18.191+0000 7f16a3fff640 1 -- 172.21.15.46:0/2184609426 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16a801f5c0 con 0x555a80aca6c0 2024-03-31T23:22:18.195 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:18.191+0000 7f16c2768740 1 -- 172.21.15.46:0/2184609426 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555a80aca6c0 msgr2=0x555a80acaaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:22:18.195 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:18.191+0000 7f16c2768740 1 --2- 172.21.15.46:0/2184609426 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555a80aca6c0 0x555a80acaaa0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f16a80058c0 tx=0x7f16a8005850 comp rx=0 tx=0).stop 2024-03-31T23:22:18.195 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:18.192+0000 7f16c2768740 1 -- 172.21.15.46:0/2184609426 shutdown_connections 2024-03-31T23:22:18.195 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:18.192+0000 7f16c2768740 1 --2- 172.21.15.46:0/2184609426 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555a80ad3db0 0x555a80ad61a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:22:18.195 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:18.192+0000 7f16c2768740 1 --2- 172.21.15.46:0/2184609426 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x555a80acafe0 0x555a80ad3870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:22:18.195 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:18.192+0000 7f16c2768740 1 --2- 172.21.15.46:0/2184609426 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555a80aca6c0 0x555a80acaaa0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:22:18.195 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:18.192+0000 7f16c2768740 1 -- 172.21.15.46:0/2184609426 >> 172.21.15.46:0/2184609426 conn(0x555a80ac4fe0 msgr2=0x555a80ac53c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:22:18.196 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:18.192+0000 7f16c2768740 1 -- 172.21.15.46:0/2184609426 shutdown_connections 2024-03-31T23:22:18.196 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:18.192+0000 7f16c2768740 1 -- 172.21.15.46:0/2184609426 wait complete. 2024-03-31T23:22:18.196 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:18.192+0000 7f16c2768740 1 -- start start 2024-03-31T23:22:18.196 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:18.192+0000 7f16c2768740 1 Processor -- start 2024-03-31T23:22:18.199 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:18.192+0000 7f16c2768740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x555a80aca6c0 0x555a80ad0e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:22:18.199 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:18.192+0000 7f16c2768740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555a80acafe0 0x555a80ad13b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:22:18.199 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:18.192+0000 7f16c2768740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555a80ad3db0 0x555a80ad18f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:22:18.199 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:18.192+0000 7f16c2768740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x555a80ad69e0 con 0x555a80acafe0 2024-03-31T23:22:18.199 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:18.192+0000 7f16c2768740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x555a809cbf80 con 0x555a80aca6c0 2024-03-31T23:22:18.200 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:18.192+0000 7f16c2768740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x555a80ad6b60 con 0x555a80ad3db0 2024-03-31T23:22:18.200 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:18.192+0000 7f16c2768740 10 client.0 _ll_register_callbacks cb 0x555a80a24290 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T23:22:18.200 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:18.193+0000 7f16bd125640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555a80acafe0 0x555a80ad13b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:22:18.200 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:18.193+0000 7f16bd125640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555a80acafe0 0x555a80ad13b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40794/0 (socket says 172.21.15.46:40794) 2024-03-31T23:22:18.200 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:18.193+0000 7f16bd125640 1 -- 172.21.15.46:0/2010518512 learned_addr learned my addr 172.21.15.46:0/2010518512 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:22:18.200 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:18.193+0000 7f16bd926640 1 --2- 172.21.15.46:0/2010518512 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x555a80aca6c0 0x555a80ad0e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:22:18.200 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:18.193+0000 7f16be127640 1 --2- 172.21.15.46:0/2010518512 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555a80ad3db0 0x555a80ad18f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:22:18.200 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:18.193+0000 7f16bd125640 1 -- 172.21.15.46:0/2010518512 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555a80ad3db0 msgr2=0x555a80ad18f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:22:18.200 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:18.193+0000 7f16bd125640 1 --2- 172.21.15.46:0/2010518512 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555a80ad3db0 0x555a80ad18f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:22:18.200 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:18.193+0000 7f16bd125640 1 -- 172.21.15.46:0/2010518512 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x555a80aca6c0 msgr2=0x555a80ad0e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:22:18.200 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:18.193+0000 7f16bd125640 1 --2- 172.21.15.46:0/2010518512 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x555a80aca6c0 0x555a80ad0e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:22:18.200 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:18.193+0000 7f16bd125640 1 -- 172.21.15.46:0/2010518512 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555a8080e620 con 0x555a80acafe0 2024-03-31T23:22:18.200 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:18.193+0000 7f16bd125640 1 --2- 172.21.15.46:0/2010518512 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555a80acafe0 0x555a80ad13b0 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7f16a8023040 tx=0x7f16a8009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b78389bf5b99584 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:22:18.200 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:18.193+0000 7f16a17fa640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-31T23:22:18.201 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:18.193+0000 7f16a17fa640 1 -- 172.21.15.46:0/2010518512 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16a802e030 con 0x555a80acafe0 2024-03-31T23:22:18.201 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:18.193+0000 7f16a17fa640 1 -- 172.21.15.46:0/2010518512 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 9554+0+0 (secure 0 0 0) 0x7f16a8010d50 con 0x555a80acafe0 2024-03-31T23:22:18.201 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:18.193+0000 7f16c2768740 1 -- 172.21.15.46:0/2010518512 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f16a8020420 con 0x555a80acafe0 2024-03-31T23:22:18.201 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:18.193+0000 7f16a17fa640 1 -- 172.21.15.46:0/2010518512 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f16a8038050 con 0x555a80acafe0 2024-03-31T23:22:18.201 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:18.193+0000 7f16c2768740 10 client.17010 fetch_fsmap learned FSMap version 36 2024-03-31T23:22:18.201 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:18.193+0000 7f16c2768740 1 -- 172.21.15.46:0/2010518512 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x555a80ae2940 con 0x555a80acafe0 2024-03-31T23:22:18.201 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:18.194+0000 7f16a17fa640 1 -- 172.21.15.46:0/2010518512 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f16a8020420 con 0x555a80acafe0 2024-03-31T23:22:18.201 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:18.194+0000 7f16a17fa640 1 -- 172.21.15.46:0/2010518512 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 36) ==== 8362+0+0 (secure 0 0 0) 0x7f16a801c9f0 con 0x555a80acafe0 2024-03-31T23:22:18.201 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:18.194+0000 7f16c2768740 10 client.17010 fetch_fsmap finished waiting for FSMap version 36 2024-03-31T23:22:18.201 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:18.194+0000 7f16c2768740 10 client.17010 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-31T23:22:18.201 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:18.194+0000 7f16c2768740 20 client.17010 populate_metadata read hostname 'smithi046' 2024-03-31T23:22:18.201 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:18.194+0000 7f16c2768740 1 --2- 172.21.15.46:0/2010518512 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x555a80ae3d40 0x555a80ae6130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:22:18.201 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:18.194+0000 7f16c2768740 4 client.17010 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-31T23:22:18.201 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:18.194+0000 7f16c2768740 1 -- 172.21.15.46:0/2010518512 --> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555a80a41120 con 0x555a80ae3d40 2024-03-31T23:22:18.202 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:18.194+0000 7f16bd926640 1 --2- 172.21.15.46:0/2010518512 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x555a80ae3d40 0x555a80ae6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:22:18.202 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:18.194+0000 7f16a17fa640 10 client.17010 ms_handle_connect on v2:172.21.15.142:6838/2519143918 2024-03-31T23:22:18.202 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:18.194+0000 7f16bd926640 1 --2- 172.21.15.46:0/2010518512 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x555a80ae3d40 0x555a80ae6130 crc :-1 s=READY pgs=169 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=17e9fc4e25bf7460 server_cookie=af1e552da45da878 in_seq=0 out_seq=0 2024-03-31T23:22:18.202 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:18.195+0000 7f16a17fa640 1 -- 172.21.15.46:0/2010518512 <== mds.0 v2:172.21.15.142:6838/2519143918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x555a80a41120 con 0x555a80ae3d40 2024-03-31T23:22:18.202 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:18.195+0000 7f16a17fa640 10 client.17010 handle_command_reply: tid=0 2024-03-31T23:22:18.202 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:18.195+0000 7f16c2768740 1 client.17010 shutdown 2024-03-31T23:22:18.202 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:18.195+0000 7f16c2768740 1 -- 172.21.15.46:0/2010518512 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555a80acafe0 msgr2=0x555a80ad13b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:22:18.202 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:18.195+0000 7f16c2768740 1 --2- 172.21.15.46:0/2010518512 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555a80acafe0 0x555a80ad13b0 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7f16a8023040 tx=0x7f16a8009260 comp rx=0 tx=0).stop 2024-03-31T23:22:18.202 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:18.196+0000 7f16c2768740 1 -- 172.21.15.46:0/2010518512 shutdown_connections 2024-03-31T23:22:18.202 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:18.196+0000 7f16c2768740 1 --2- 172.21.15.46:0/2010518512 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x555a80ae3d40 0x555a80ae6130 crc :-1 s=READY pgs=169 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:22:18.202 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:18.196+0000 7f16c2768740 1 --2- 172.21.15.46:0/2010518512 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555a80ad3db0 0x555a80ad18f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:22:18.202 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:18.196+0000 7f16c2768740 1 --2- 172.21.15.46:0/2010518512 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555a80acafe0 0x555a80ad13b0 unknown :-1 s=CLOSED pgs=815 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:22:18.202 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:18.196+0000 7f16c2768740 1 --2- 172.21.15.46:0/2010518512 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x555a80aca6c0 0x555a80ad0e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:22:18.202 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:18.196+0000 7f16a17fa640 0 client.17010 ms_handle_reset on v2:172.21.15.142:6838/2519143918 2024-03-31T23:22:18.203 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:18.196+0000 7f16c2768740 1 -- 172.21.15.46:0/2010518512 >> 172.21.15.46:0/2010518512 conn(0x555a80ac4fe0 msgr2=0x555a80ac6d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:22:18.203 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:18.196+0000 7f16c2768740 1 -- 172.21.15.46:0/2010518512 shutdown_connections 2024-03-31T23:22:18.203 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:18.196+0000 7f16c2768740 1 -- 172.21.15.46:0/2010518512 wait complete. 2024-03-31T23:22:18.203 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:18.196+0000 7f16c2768740 20 client.17010 trim_cache size 0 max 16384 2024-03-31T23:22:18.214 INFO:tasks.workunit.client.0.smithi046.stdout: 1 213480 1.94 MB/sec execute 444 sec latency 21.546 ms 2024-03-31T23:22:18.219 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:18.213+0000 7f16c2768740 1 Processor -- start 2024-03-31T23:22:18.219 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:18.213+0000 7f16c2768740 1 -- start start 2024-03-31T23:22:18.219 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:18.213+0000 7f16c2768740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555a80aca6c0 0x555a80ad35a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:22:18.219 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:18.213+0000 7f16c2768740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x555a80acafe0 0x555a80adf780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:22:18.219 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:18.213+0000 7f16c2768740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555a80ad3db0 0x555a80aeb700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:22:18.220 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:18.213+0000 7f16c2768740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f16b81af310 con 0x555a80aca6c0 2024-03-31T23:22:18.220 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:18.213+0000 7f16c2768740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f16b81acb80 con 0x555a80acafe0 2024-03-31T23:22:18.220 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:18.213+0000 7f16c2768740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f16b81aa9f0 con 0x555a80ad3db0 2024-03-31T23:22:18.220 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:18.213+0000 7f16bd926640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555a80aca6c0 0x555a80ad35a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:22:18.220 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:18.213+0000 7f16bd926640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555a80aca6c0 0x555a80ad35a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40804/0 (socket says 172.21.15.46:40804) 2024-03-31T23:22:18.220 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:18.213+0000 7f16bd926640 1 -- 172.21.15.46:0/2326778086 learned_addr learned my addr 172.21.15.46:0/2326778086 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:22:18.220 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:18.213+0000 7f16bd125640 1 --2- 172.21.15.46:0/2326778086 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x555a80acafe0 0x555a80adf780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:22:18.220 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:18.213+0000 7f16be127640 1 --2- 172.21.15.46:0/2326778086 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555a80ad3db0 0x555a80aeb700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:22:18.221 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:18.213+0000 7f16bd926640 1 -- 172.21.15.46:0/2326778086 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555a80ad3db0 msgr2=0x555a80aeb700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:22:18.221 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:18.213+0000 7f16bd926640 1 --2- 172.21.15.46:0/2326778086 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555a80ad3db0 0x555a80aeb700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:22:18.221 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:18.213+0000 7f16bd926640 1 -- 172.21.15.46:0/2326778086 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x555a80acafe0 msgr2=0x555a80adf780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:22:18.221 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:18.213+0000 7f16bd926640 1 --2- 172.21.15.46:0/2326778086 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x555a80acafe0 0x555a80adf780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:22:18.221 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:18.213+0000 7f16bd926640 1 -- 172.21.15.46:0/2326778086 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555a80acc840 con 0x555a80aca6c0 2024-03-31T23:22:18.221 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:18.214+0000 7f16bd926640 1 --2- 172.21.15.46:0/2326778086 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555a80aca6c0 0x555a80ad35a0 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7f16b4009230 tx=0x7f16b403e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee1d621ddde6ef server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:22:18.221 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:18.214+0000 7f16a37fe640 1 -- 172.21.15.46:0/2326778086 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16b4058070 con 0x555a80aca6c0 2024-03-31T23:22:18.221 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:18.214+0000 7f16a37fe640 1 -- 172.21.15.46:0/2326778086 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f16b4053800 con 0x555a80aca6c0 2024-03-31T23:22:18.221 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:18.214+0000 7f16a37fe640 1 -- 172.21.15.46:0/2326778086 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16b4039040 con 0x555a80aca6c0 2024-03-31T23:22:18.221 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:18.214+0000 7f16c2768740 1 -- 172.21.15.46:0/2326778086 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555a80aca6c0 msgr2=0x555a80ad35a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:22:18.221 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:18.214+0000 7f16c2768740 1 --2- 172.21.15.46:0/2326778086 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555a80aca6c0 0x555a80ad35a0 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7f16b4009230 tx=0x7f16b403e000 comp rx=0 tx=0).stop 2024-03-31T23:22:18.221 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:18.214+0000 7f16c2768740 1 -- 172.21.15.46:0/2326778086 shutdown_connections 2024-03-31T23:22:18.221 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:18.214+0000 7f16c2768740 1 --2- 172.21.15.46:0/2326778086 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555a80ad3db0 0x555a80aeb700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:22:18.221 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:18.214+0000 7f16c2768740 1 --2- 172.21.15.46:0/2326778086 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x555a80acafe0 0x555a80adf780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:22:18.222 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:18.214+0000 7f16c2768740 1 --2- 172.21.15.46:0/2326778086 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555a80aca6c0 0x555a80ad35a0 unknown :-1 s=CLOSED pgs=816 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:22:18.222 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:18.214+0000 7f16c2768740 1 -- 172.21.15.46:0/2326778086 >> 172.21.15.46:0/2326778086 conn(0x555a80ac4fe0 msgr2=0x555a80acc130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:22:18.222 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:18.215+0000 7f16c2768740 1 -- 172.21.15.46:0/2326778086 shutdown_connections 2024-03-31T23:22:18.222 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:18.215+0000 7f16c2768740 1 -- 172.21.15.46:0/2326778086 wait complete. 2024-03-31T23:22:18.222 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:18.215+0000 7f16c2768740 1 -- start start 2024-03-31T23:22:18.222 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:18.215+0000 7f16c2768740 1 Processor -- start 2024-03-31T23:22:18.222 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:18.215+0000 7f16c2768740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555a80aca6c0 0x555a80ad7690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:22:18.222 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:18.215+0000 7f16c2768740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x555a80acafe0 0x555a80ad7bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:22:18.222 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:18.215+0000 7f16c2768740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555a80ad3db0 0x555a80ad8110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:22:18.222 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:18.215+0000 7f16c2768740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f16b81ab040 con 0x555a80aca6c0 2024-03-31T23:22:18.222 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:18.215+0000 7f16c2768740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f16b81ac4b0 con 0x555a80acafe0 2024-03-31T23:22:18.222 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:18.215+0000 7f16c2768740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f16b81ab710 con 0x555a80ad3db0 2024-03-31T23:22:18.222 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:18.215+0000 7f16c2768740 10 client.0 _ll_register_callbacks cb 0x7f16a8023070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T23:22:18.222 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:18.215+0000 7f16bd926640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555a80aca6c0 0x555a80ad7690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:22:18.222 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:18.215+0000 7f16bd926640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555a80aca6c0 0x555a80ad7690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40814/0 (socket says 172.21.15.46:40814) 2024-03-31T23:22:18.223 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:18.215+0000 7f16bd926640 1 -- 172.21.15.46:0/651826047 learned_addr learned my addr 172.21.15.46:0/651826047 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:22:18.223 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:18.215+0000 7f16bd125640 1 --2- 172.21.15.46:0/651826047 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x555a80acafe0 0x555a80ad7bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:22:18.223 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:18.215+0000 7f16be127640 1 --2- 172.21.15.46:0/651826047 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555a80ad3db0 0x555a80ad8110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:22:18.223 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:18.216+0000 7f16bd926640 1 -- 172.21.15.46:0/651826047 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555a80ad3db0 msgr2=0x555a80ad8110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:22:18.223 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:18.216+0000 7f16bd926640 1 --2- 172.21.15.46:0/651826047 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555a80ad3db0 0x555a80ad8110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:22:18.223 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:18.216+0000 7f16bd926640 1 -- 172.21.15.46:0/651826047 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x555a80acafe0 msgr2=0x555a80ad7bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:22:18.223 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:18.216+0000 7f16bd926640 1 --2- 172.21.15.46:0/651826047 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x555a80acafe0 0x555a80ad7bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:22:18.223 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:18.216+0000 7f16bd926640 1 -- 172.21.15.46:0/651826047 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555a80b48d30 con 0x555a80aca6c0 2024-03-31T23:22:18.223 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:18.216+0000 7f16bd926640 1 --2- 172.21.15.46:0/651826047 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555a80aca6c0 0x555a80ad7690 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7f16b4005e00 tx=0x7f16b4005d20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd8cf108aa18bb59 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:22:18.223 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:18.216+0000 7f16a17fa640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-31T23:22:18.223 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:18.216+0000 7f16a17fa640 1 -- 172.21.15.46:0/651826047 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16b4058030 con 0x555a80aca6c0 2024-03-31T23:22:18.223 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:18.216+0000 7f16a17fa640 1 -- 172.21.15.46:0/651826047 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 9554+0+0 (secure 0 0 0) 0x7f16b40533d0 con 0x555a80aca6c0 2024-03-31T23:22:18.223 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:18.216+0000 7f16c2768740 1 -- 172.21.15.46:0/651826047 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x555a80a3ae30 con 0x555a80aca6c0 2024-03-31T23:22:18.223 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:18.216+0000 7f16a17fa640 1 -- 172.21.15.46:0/651826047 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f16b4039040 con 0x555a80aca6c0 2024-03-31T23:22:18.224 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:18.216+0000 7f16c2768740 10 client.17016 fetch_fsmap learned FSMap version 36 2024-03-31T23:22:18.224 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:18.216+0000 7f16c2768740 1 -- 172.21.15.46:0/651826047 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x555a80b4a010 con 0x555a80aca6c0 2024-03-31T23:22:18.224 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:18.216+0000 7f16a17fa640 1 -- 172.21.15.46:0/651826047 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x555a80a3ae30 con 0x555a80aca6c0 2024-03-31T23:22:18.224 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:18.217+0000 7f16a17fa640 1 -- 172.21.15.46:0/651826047 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 36) ==== 8362+0+0 (secure 0 0 0) 0x7f16b4053030 con 0x555a80aca6c0 2024-03-31T23:22:18.224 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:18.217+0000 7f16c2768740 10 client.17016 fetch_fsmap finished waiting for FSMap version 36 2024-03-31T23:22:18.224 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:18.217+0000 7f16c2768740 10 client.17016 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-31T23:22:18.224 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:18.217+0000 7f16c2768740 20 client.17016 populate_metadata read hostname 'smithi046' 2024-03-31T23:22:18.224 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:18.217+0000 7f16c2768740 1 --2- 172.21.15.46:0/651826047 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x555a80b4a7f0 0x555a80b4cc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:22:18.224 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:18.217+0000 7f16c2768740 4 client.17016 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-31T23:22:18.224 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:18.217+0000 7f16c2768740 1 -- 172.21.15.46:0/651826047 --> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] -- command(tid 0: {"prefix": "flush journal"}) -- 0x555a80a35180 con 0x555a80b4a7f0 2024-03-31T23:22:18.224 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:18.217+0000 7f16bd125640 1 --2- 172.21.15.46:0/651826047 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x555a80b4a7f0 0x555a80b4cc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:22:18.224 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:18.217+0000 7f16bd125640 1 --2- 172.21.15.46:0/651826047 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x555a80b4a7f0 0x555a80b4cc10 crc :-1 s=READY pgs=170 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d2f8615f95a36cff server_cookie=8ca802aea1f26dc3 in_seq=0 out_seq=0 2024-03-31T23:22:18.224 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:18.217+0000 7f16a17fa640 10 client.17016 ms_handle_connect on v2:172.21.15.142:6838/2519143918 2024-03-31T23:22:18.243 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:18.241+0000 7f16a17fa640 1 -- 172.21.15.46:0/651826047 <== mds.0 v2:172.21.15.142:6838/2519143918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x555a80a35180 con 0x555a80b4a7f0 2024-03-31T23:22:18.243 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:18.241+0000 7f16a17fa640 10 client.17016 handle_command_reply: tid=0 2024-03-31T23:22:18.243 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:18.241+0000 7f16c2768740 1 client.17016 shutdown 2024-03-31T23:22:18.243 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:18.242+0000 7f16c2768740 1 -- 172.21.15.46:0/651826047 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555a80aca6c0 msgr2=0x555a80ad7690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:22:18.243 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:18.242+0000 7f16c2768740 1 --2- 172.21.15.46:0/651826047 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555a80aca6c0 0x555a80ad7690 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7f16b4005e00 tx=0x7f16b4005d20 comp rx=0 tx=0).stop 2024-03-31T23:22:18.244 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:18.242+0000 7f16c2768740 1 -- 172.21.15.46:0/651826047 shutdown_connections 2024-03-31T23:22:18.244 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:18.242+0000 7f16c2768740 1 --2- 172.21.15.46:0/651826047 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x555a80b4a7f0 0x555a80b4cc10 crc :-1 s=READY pgs=170 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:22:18.244 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:18.242+0000 7f16c2768740 1 --2- 172.21.15.46:0/651826047 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555a80ad3db0 0x555a80ad8110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:22:18.244 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:18.242+0000 7f16c2768740 1 --2- 172.21.15.46:0/651826047 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x555a80acafe0 0x555a80ad7bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:22:18.244 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:18.242+0000 7f16a17fa640 0 client.17016 ms_handle_reset on v2:172.21.15.142:6838/2519143918 2024-03-31T23:22:18.244 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:18.242+0000 7f16c2768740 1 --2- 172.21.15.46:0/651826047 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555a80aca6c0 0x555a80ad7690 unknown :-1 s=CLOSED pgs=817 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:22:18.244 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:18.242+0000 7f16c2768740 1 -- 172.21.15.46:0/651826047 >> 172.21.15.46:0/651826047 conn(0x555a80ac4fe0 msgr2=0x555a80ad0e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:22:18.244 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:18.242+0000 7f16c2768740 1 -- 172.21.15.46:0/651826047 shutdown_connections 2024-03-31T23:22:18.244 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:18.242+0000 7f16c2768740 1 -- 172.21.15.46:0/651826047 wait complete. 2024-03-31T23:22:18.244 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:18.242+0000 7f16c2768740 20 client.17016 trim_cache size 0 max 16384 2024-03-31T23:22:18.244 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout:{ 2024-03-31T23:22:18.245 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout: "message": "", 2024-03-31T23:22:18.245 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout: "return_code": 0 2024-03-31T23:22:18.245 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout:} 2024-03-31T23:22:18.245 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:18.242+0000 7f16bfb8a640 1 -- 172.21.15.46:0/3931854436 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f16a40a26b0 msgr2=0x7f16a40a4b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:22:18.246 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:18.242+0000 7f16bfb8a640 1 --2- 172.21.15.46:0/3931854436 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f16a40a26b0 0x7f16a40a4b70 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f16b4009830 tx=0x7f16b40058d0 comp rx=0 tx=0).stop 2024-03-31T23:22:18.246 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:18.242+0000 7f16bfb8a640 1 -- 172.21.15.46:0/3931854436 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f16b81066b0 msgr2=0x7f16b819c680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:22:18.246 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:18.242+0000 7f16bfb8a640 1 --2- 172.21.15.46:0/3931854436 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f16b81066b0 0x7f16b819c680 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7f16b000eb10 tx=0x7f16b000ef10 comp rx=0 tx=0).stop 2024-03-31T23:22:18.246 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:18.243+0000 7f16bfb8a640 1 -- 172.21.15.46:0/3931854436 shutdown_connections 2024-03-31T23:22:18.246 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:18.243+0000 7f16bfb8a640 1 --2- 172.21.15.46:0/3931854436 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f16a40a26b0 0x7f16a40a4b70 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:22:18.246 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:18.243+0000 7f16bfb8a640 1 --2- 172.21.15.46:0/3931854436 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f16b81066b0 0x7f16b819c680 unknown :-1 s=CLOSED pgs=814 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:22:18.246 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:18.243+0000 7f16bfb8a640 1 --2- 172.21.15.46:0/3931854436 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f16b8100fd0 0x7f16b8198300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:22:18.246 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:18.243+0000 7f16bfb8a640 1 --2- 172.21.15.46:0/3931854436 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f16b8100610 0x7f16b8197dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:22:18.246 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:18.243+0000 7f16bfb8a640 1 -- 172.21.15.46:0/3931854436 >> 172.21.15.46:0/3931854436 conn(0x7f16b8077770 msgr2=0x7f16b80ff120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:22:18.246 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:18.243+0000 7f16bfb8a640 1 -- 172.21.15.46:0/3931854436 shutdown_connections 2024-03-31T23:22:18.246 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:18.243+0000 7f16bfb8a640 1 -- 172.21.15.46:0/3931854436 wait complete. 2024-03-31T23:22:18.795 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:22:18 smithi162 ceph-mon[36237]: pgmap v330: 129 pgs: 129 active+clean; 126 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 419 KiB/s rd, 1.1 MiB/s wr, 110 op/s 2024-03-31T23:22:18.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:22:18 smithi046 ceph-mon[32498]: pgmap v330: 129 pgs: 129 active+clean; 126 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 419 KiB/s rd, 1.1 MiB/s wr, 110 op/s 2024-03-31T23:22:18.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:22:18 smithi142 ceph-mon[35271]: pgmap v330: 129 pgs: 129 active+clean; 126 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 419 KiB/s rd, 1.1 MiB/s wr, 110 op/s 2024-03-31T23:22:19.214 INFO:tasks.workunit.client.0.smithi046.stdout: 1 213927 1.94 MB/sec execute 445 sec latency 16.522 ms 2024-03-31T23:22:20.214 INFO:tasks.workunit.client.0.smithi046.stdout: 1 214432 1.94 MB/sec execute 446 sec latency 22.501 ms 2024-03-31T23:22:20.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:22:20 smithi162 ceph-mon[36237]: pgmap v331: 129 pgs: 129 active+clean; 124 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 303 KiB/s rd, 768 KiB/s wr, 79 op/s 2024-03-31T23:22:20.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:22:20 smithi046 ceph-mon[32498]: pgmap v331: 129 pgs: 129 active+clean; 124 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 303 KiB/s rd, 768 KiB/s wr, 79 op/s 2024-03-31T23:22:20.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:22:20 smithi142 ceph-mon[35271]: pgmap v331: 129 pgs: 129 active+clean; 124 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 303 KiB/s rd, 768 KiB/s wr, 79 op/s 2024-03-31T23:22:21.214 INFO:tasks.workunit.client.0.smithi046.stdout: 1 215014 1.94 MB/sec execute 447 sec latency 22.362 ms 2024-03-31T23:22:22.214 INFO:tasks.workunit.client.0.smithi046.stdout: 1 215460 1.94 MB/sec execute 448 sec latency 21.099 ms 2024-03-31T23:22:22.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:22:22 smithi162 ceph-mon[36237]: pgmap v332: 129 pgs: 129 active+clean; 124 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 294 KiB/s rd, 745 KiB/s wr, 76 op/s 2024-03-31T23:22:22.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:22:22 smithi046 ceph-mon[32498]: pgmap v332: 129 pgs: 129 active+clean; 124 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 294 KiB/s rd, 745 KiB/s wr, 76 op/s 2024-03-31T23:22:22.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:22:22 smithi142 ceph-mon[35271]: pgmap v332: 129 pgs: 129 active+clean; 124 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 294 KiB/s rd, 745 KiB/s wr, 76 op/s 2024-03-31T23:22:23.214 INFO:tasks.workunit.client.0.smithi046.stdout: 1 216164 1.95 MB/sec execute 449 sec latency 15.736 ms 2024-03-31T23:22:24.214 INFO:tasks.workunit.client.0.smithi046.stdout: 1 216491 1.95 MB/sec execute 450 sec latency 85.454 ms 2024-03-31T23:22:24.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:22:24 smithi162 ceph-mon[36237]: pgmap v333: 129 pgs: 129 active+clean; 124 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 253 KiB/s rd, 640 KiB/s wr, 65 op/s 2024-03-31T23:22:24.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:22:24 smithi046 ceph-mon[32498]: pgmap v333: 129 pgs: 129 active+clean; 124 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 253 KiB/s rd, 640 KiB/s wr, 65 op/s 2024-03-31T23:22:24.944 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:22:24 smithi142 ceph-mon[35271]: pgmap v333: 129 pgs: 129 active+clean; 124 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 253 KiB/s rd, 640 KiB/s wr, 65 op/s 2024-03-31T23:22:25.214 INFO:tasks.workunit.client.0.smithi046.stdout: 1 216491 1.95 MB/sec execute 451 sec latency 1085.501 ms 2024-03-31T23:22:26.214 INFO:tasks.workunit.client.0.smithi046.stdout: 1 216491 1.94 MB/sec execute 452 sec latency 2085.591 ms 2024-03-31T23:22:26.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:22:26 smithi162 ceph-mon[36237]: pgmap v334: 129 pgs: 129 active+clean; 126 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 426 KiB/s rd, 1.3 MiB/s wr, 115 op/s 2024-03-31T23:22:26.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:22:26 smithi046 ceph-mon[32498]: pgmap v334: 129 pgs: 129 active+clean; 126 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 426 KiB/s rd, 1.3 MiB/s wr, 115 op/s 2024-03-31T23:22:26.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:22:26 smithi142 ceph-mon[35271]: pgmap v334: 129 pgs: 129 active+clean; 126 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 426 KiB/s rd, 1.3 MiB/s wr, 115 op/s 2024-03-31T23:22:27.214 INFO:tasks.workunit.client.0.smithi046.stdout: 1 216491 1.94 MB/sec execute 453 sec latency 3085.641 ms 2024-03-31T23:22:28.214 INFO:tasks.workunit.client.0.smithi046.stdout: 1 216558 1.93 MB/sec execute 454 sec latency 3617.951 ms 2024-03-31T23:22:28.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:22:28 smithi162 ceph-mon[36237]: pgmap v335: 129 pgs: 129 active+clean; 126 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 334 KiB/s rd, 984 KiB/s wr, 88 op/s 2024-03-31T23:22:28.862 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:22:28 smithi046 ceph-mon[32498]: pgmap v335: 129 pgs: 129 active+clean; 126 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 334 KiB/s rd, 984 KiB/s wr, 88 op/s 2024-03-31T23:22:28.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:22:28 smithi142 ceph-mon[35271]: pgmap v335: 129 pgs: 129 active+clean; 126 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 334 KiB/s rd, 984 KiB/s wr, 88 op/s 2024-03-31T23:22:29.214 INFO:tasks.workunit.client.0.smithi046.stdout: 1 216558 1.93 MB/sec execute 455 sec latency 1212.436 ms 2024-03-31T23:22:30.214 INFO:tasks.workunit.client.0.smithi046.stdout: 1 216558 1.92 MB/sec execute 456 sec latency 2212.492 ms 2024-03-31T23:22:30.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:22:30 smithi162 ceph-mon[36237]: pgmap v336: 129 pgs: 129 active+clean; 126 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 384 KiB/s rd, 1.0 MiB/s wr, 104 op/s 2024-03-31T23:22:30.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:22:30 smithi046 ceph-mon[32498]: pgmap v336: 129 pgs: 129 active+clean; 126 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 384 KiB/s rd, 1.0 MiB/s wr, 104 op/s 2024-03-31T23:22:30.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:22:30 smithi142 ceph-mon[35271]: pgmap v336: 129 pgs: 129 active+clean; 126 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 384 KiB/s rd, 1.0 MiB/s wr, 104 op/s 2024-03-31T23:22:31.214 INFO:tasks.workunit.client.0.smithi046.stdout: 1 216558 1.92 MB/sec execute 457 sec latency 3212.552 ms 2024-03-31T23:22:31.342 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:31.341+0000 7f752e6ae640 1 -- 172.21.15.46:0/3299313391 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f752806a910 msgr2=0x7f752810e5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:22:31.343 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:31.341+0000 7f752e6ae640 1 --2- 172.21.15.46:0/3299313391 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f752806a910 0x7f752810e5e0 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7f7520009cd0 tx=0x7f752002c0f0 comp rx=0 tx=0).stop 2024-03-31T23:22:31.343 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:31.341+0000 7f752e6ae640 1 -- 172.21.15.46:0/3299313391 shutdown_connections 2024-03-31T23:22:31.343 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:31.341+0000 7f752e6ae640 1 --2- 172.21.15.46:0/3299313391 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f752806a910 0x7f752810e5e0 unknown :-1 s=CLOSED pgs=818 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:22:31.343 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:31.341+0000 7f752e6ae640 1 --2- 172.21.15.46:0/3299313391 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7528069f50 0x7f752806a3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:22:31.343 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:31.341+0000 7f752e6ae640 1 --2- 172.21.15.46:0/3299313391 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f75281077d0 0x7f7528107bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:22:31.343 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:31.341+0000 7f752e6ae640 1 -- 172.21.15.46:0/3299313391 >> 172.21.15.46:0/3299313391 conn(0x7f7528077420 msgr2=0x7f7528077830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:22:31.343 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:31.341+0000 7f752e6ae640 1 -- 172.21.15.46:0/3299313391 shutdown_connections 2024-03-31T23:22:31.343 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:31.341+0000 7f752e6ae640 1 -- 172.21.15.46:0/3299313391 wait complete. 2024-03-31T23:22:31.344 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:31.341+0000 7f752e6ae640 1 Processor -- start 2024-03-31T23:22:31.344 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:31.341+0000 7f752e6ae640 1 -- start start 2024-03-31T23:22:31.346 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:31.342+0000 7f752e6ae640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7528069f50 0x7f7528197dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:22:31.346 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:31.342+0000 7f752e6ae640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f752806a910 0x7f7528198300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:22:31.346 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:31.342+0000 7f752e6ae640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f75281077d0 0x7f752819c630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:22:31.346 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:31.342+0000 7f752e6ae640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f75281119a0 con 0x7f752806a910 2024-03-31T23:22:31.346 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:31.342+0000 7f752e6ae640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f7528111820 con 0x7f7528069f50 2024-03-31T23:22:31.346 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:31.342+0000 7f752e6ae640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f7528111b20 con 0x7f75281077d0 2024-03-31T23:22:31.346 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:31.342+0000 7f75277fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f752806a910 0x7f7528198300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:22:31.346 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:31.342+0000 7f75277fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f752806a910 0x7f7528198300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60160/0 (socket says 172.21.15.46:60160) 2024-03-31T23:22:31.346 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:31.342+0000 7f7527fff640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7528069f50 0x7f7528197dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:22:31.346 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:31.342+0000 7f75277fe640 1 -- 172.21.15.46:0/2829807926 learned_addr learned my addr 172.21.15.46:0/2829807926 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:22:31.347 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:31.342+0000 7f752cc4b640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f75281077d0 0x7f752819c630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:22:31.347 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:31.342+0000 7f75277fe640 1 -- 172.21.15.46:0/2829807926 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f75281077d0 msgr2=0x7f752819c630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:22:31.347 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:31.342+0000 7f75277fe640 1 --2- 172.21.15.46:0/2829807926 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f75281077d0 0x7f752819c630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:22:31.347 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:31.342+0000 7f75277fe640 1 -- 172.21.15.46:0/2829807926 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7528069f50 msgr2=0x7f7528197dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:22:31.347 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:31.342+0000 7f75277fe640 1 --2- 172.21.15.46:0/2829807926 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7528069f50 0x7f7528197dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:22:31.347 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:31.342+0000 7f75277fe640 1 -- 172.21.15.46:0/2829807926 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f752819cd60 con 0x7f752806a910 2024-03-31T23:22:31.348 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:31.342+0000 7f7527fff640 1 --2- 172.21.15.46:0/2829807926 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7528069f50 0x7f7528197dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:22:31.348 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:31.342+0000 7f752cc4b640 1 --2- 172.21.15.46:0/2829807926 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f75281077d0 0x7f752819c630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:22:31.348 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:31.343+0000 7f75277fe640 1 --2- 172.21.15.46:0/2829807926 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f752806a910 0x7f7528198300 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7f751c00e9f0 tx=0x7f751c00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:22:31.348 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:31.343+0000 7f75257fa640 1 -- 172.21.15.46:0/2829807926 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f751c00cd80 con 0x7f752806a910 2024-03-31T23:22:31.348 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:31.343+0000 7f75257fa640 1 -- 172.21.15.46:0/2829807926 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f751c0108e0 con 0x7f752806a910 2024-03-31T23:22:31.348 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:31.343+0000 7f752e6ae640 1 -- 172.21.15.46:0/2829807926 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f752819d050 con 0x7f752806a910 2024-03-31T23:22:31.348 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:31.343+0000 7f75257fa640 1 -- 172.21.15.46:0/2829807926 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f751c018920 con 0x7f752806a910 2024-03-31T23:22:31.348 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:31.343+0000 7f752e6ae640 1 -- 172.21.15.46:0/2829807926 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f75281a48e0 con 0x7f752806a910 2024-03-31T23:22:31.357 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:31.345+0000 7f75257fa640 1 -- 172.21.15.46:0/2829807926 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 25) ==== 137451+0+0 (secure 0 0 0) 0x7f751c010a90 con 0x7f752806a910 2024-03-31T23:22:31.357 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:31.345+0000 7f753136d740 1 Processor -- start 2024-03-31T23:22:31.357 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:31.345+0000 7f753136d740 1 -- start start 2024-03-31T23:22:31.357 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:31.346+0000 7f753136d740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55add00db6c0 0x55add00dbaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:22:31.358 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:31.346+0000 7f753136d740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55add00dbfe0 0x55add00e4870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:22:31.358 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:31.346+0000 7f753136d740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55add00e4db0 0x55add00e71a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:22:31.358 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:31.346+0000 7f75257fa640 1 --2- 172.21.15.46:0/2829807926 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f75100a26b0 0x7f75100a4b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:22:31.358 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:31.346+0000 7f753136d740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55add003e8f0 con 0x55add00e4db0 2024-03-31T23:22:31.358 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:31.346+0000 7f753136d740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55adcffe78d0 con 0x55add00db6c0 2024-03-31T23:22:31.358 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:31.346+0000 7f753136d740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55add003a990 con 0x55add00dbfe0 2024-03-31T23:22:31.358 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:31.346+0000 7f75277fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55add00e4db0 0x55add00e71a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:22:31.358 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:31.346+0000 7f752cc4b640 1 --2- 172.21.15.46:0/2829807926 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f75100a26b0 0x7f75100a4b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:22:31.358 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:31.346+0000 7f7527fff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55add00dbfe0 0x55add00e4870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:22:31.358 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:31.346+0000 7f75277fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55add00e4db0 0x55add00e71a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60174/0 (socket says 172.21.15.46:60174) 2024-03-31T23:22:31.359 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:31.346+0000 7f7527fff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55add00dbfe0 0x55add00e4870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.46:60434/0 (socket says 172.21.15.46:60434) 2024-03-31T23:22:31.359 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:31.346+0000 7f75277fe640 1 -- 172.21.15.46:0/1004701270 learned_addr learned my addr 172.21.15.46:0/1004701270 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:22:31.359 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:31.346+0000 7f752cc4b640 1 --2- 172.21.15.46:0/1004701270 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55add00db6c0 0x55add00dbaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:22:31.359 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:31.346+0000 7f75257fa640 1 -- 172.21.15.46:0/2829807926 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 9554+0+0 (secure 0 0 0) 0x7f751c0cf270 con 0x7f752806a910 2024-03-31T23:22:31.359 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:31.346+0000 7f7527fff640 1 -- 172.21.15.46:0/1004701270 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55add00db6c0 msgr2=0x55add00dbaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:22:31.359 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:31.346+0000 7f7527fff640 1 --2- 172.21.15.46:0/1004701270 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55add00db6c0 0x55add00dbaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:22:31.359 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:31.346+0000 7f7527fff640 1 -- 172.21.15.46:0/1004701270 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55add00e4db0 msgr2=0x55add00e71a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:22:31.360 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:31.346+0000 7f7527fff640 1 --2- 172.21.15.46:0/1004701270 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55add00e4db0 0x55add00e71a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:22:31.360 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:31.346+0000 7f7527fff640 1 -- 172.21.15.46:0/1004701270 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55adcfd53210 con 0x55add00dbfe0 2024-03-31T23:22:31.360 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:31.346+0000 7f7527fff640 1 --2- 172.21.15.46:0/1004701270 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55add00dbfe0 0x55add00e4870 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f7518009fd0 tx=0x7f75180118d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f89d10e83d11addc server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:22:31.360 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:31.347+0000 7f752cc4b640 1 --2- 172.21.15.46:0/2829807926 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f75100a26b0 0x7f75100a4b70 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f7520005cc0 tx=0x7f7520032450 comp rx=0 tx=0).ready entity=mgr.15654 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:22:31.360 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:31.347+0000 7f750e7fc640 1 -- 172.21.15.46:0/1004701270 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7518004030 con 0x55add00dbfe0 2024-03-31T23:22:31.360 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:31.347+0000 7f750e7fc640 1 -- 172.21.15.46:0/1004701270 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7518005020 con 0x55add00dbfe0 2024-03-31T23:22:31.360 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:31.347+0000 7f750e7fc640 1 -- 172.21.15.46:0/1004701270 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7518004c10 con 0x55add00dbfe0 2024-03-31T23:22:31.361 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:31.347+0000 7f753136d740 1 -- 172.21.15.46:0/1004701270 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55add00dbfe0 msgr2=0x55add00e4870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:22:31.361 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:31.347+0000 7f753136d740 1 --2- 172.21.15.46:0/1004701270 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55add00dbfe0 0x55add00e4870 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f7518009fd0 tx=0x7f75180118d0 comp rx=0 tx=0).stop 2024-03-31T23:22:31.361 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:31.347+0000 7f753136d740 1 -- 172.21.15.46:0/1004701270 shutdown_connections 2024-03-31T23:22:31.361 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:31.347+0000 7f753136d740 1 --2- 172.21.15.46:0/1004701270 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55add00e4db0 0x55add00e71a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:22:31.361 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:31.347+0000 7f753136d740 1 --2- 172.21.15.46:0/1004701270 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55add00dbfe0 0x55add00e4870 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:22:31.361 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:31.347+0000 7f753136d740 1 --2- 172.21.15.46:0/1004701270 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55add00db6c0 0x55add00dbaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:22:31.361 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:31.347+0000 7f753136d740 1 -- 172.21.15.46:0/1004701270 >> 172.21.15.46:0/1004701270 conn(0x55add00d5fe0 msgr2=0x55add00d63c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:22:31.361 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:31.347+0000 7f753136d740 1 -- 172.21.15.46:0/1004701270 shutdown_connections 2024-03-31T23:22:31.361 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:31.347+0000 7f753136d740 1 -- 172.21.15.46:0/1004701270 wait complete. 2024-03-31T23:22:31.361 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:31.348+0000 7f753136d740 1 -- start start 2024-03-31T23:22:31.361 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:31.348+0000 7f753136d740 1 Processor -- start 2024-03-31T23:22:31.361 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:31.348+0000 7f753136d740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55add00db6c0 0x55add00e1dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:22:31.361 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:31.348+0000 7f753136d740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55add00dbfe0 0x55add00e2300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:22:31.361 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:31.348+0000 7f753136d740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55add00e4db0 0x55add00e2840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:22:31.362 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:31.348+0000 7f753136d740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55add00e79e0 con 0x55add00e4db0 2024-03-31T23:22:31.362 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:31.348+0000 7f753136d740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55adcffdcf80 con 0x55add00dbfe0 2024-03-31T23:22:31.362 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:31.348+0000 7f753136d740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55add00e7b60 con 0x55add00db6c0 2024-03-31T23:22:31.362 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:31.348+0000 7f753136d740 10 client.0 _ll_register_callbacks cb 0x55add0035290 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T23:22:31.362 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:31.348+0000 7f752cc4b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55add00e4db0 0x55add00e2840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:22:31.362 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:31.349+0000 7f7527fff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55add00db6c0 0x55add00e1dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:22:31.362 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:31.348+0000 7f752cc4b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55add00e4db0 0x55add00e2840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60178/0 (socket says 172.21.15.46:60178) 2024-03-31T23:22:31.362 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:31.349+0000 7f752cc4b640 1 -- 172.21.15.46:0/812955157 learned_addr learned my addr 172.21.15.46:0/812955157 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:22:31.362 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:31.349+0000 7f75277fe640 1 --2- 172.21.15.46:0/812955157 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55add00dbfe0 0x55add00e2300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:22:31.362 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:31.349+0000 7f752cc4b640 1 -- 172.21.15.46:0/812955157 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55add00db6c0 msgr2=0x55add00e1dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:22:31.362 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:31.349+0000 7f752cc4b640 1 --2- 172.21.15.46:0/812955157 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55add00db6c0 0x55add00e1dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:22:31.362 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:31.349+0000 7f752cc4b640 1 -- 172.21.15.46:0/812955157 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55add00dbfe0 msgr2=0x55add00e2300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:22:31.363 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:31.349+0000 7f752cc4b640 1 --2- 172.21.15.46:0/812955157 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55add00dbfe0 0x55add00e2300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:22:31.363 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:31.349+0000 7f752cc4b640 1 -- 172.21.15.46:0/812955157 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55adcfe1f620 con 0x55add00e4db0 2024-03-31T23:22:31.363 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:31.349+0000 7f7527fff640 1 --2- 172.21.15.46:0/812955157 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55add00db6c0 0x55add00e1dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:22:31.363 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:31.349+0000 7f75277fe640 1 --2- 172.21.15.46:0/812955157 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55add00dbfe0 0x55add00e2300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:22:31.363 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:31.349+0000 7f752cc4b640 1 --2- 172.21.15.46:0/812955157 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55add00e4db0 0x55add00e2840 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7f7520036000 tx=0x7f7520005c50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=18ffee2530d862a3 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:22:31.363 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:31.349+0000 7f75077fe640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-31T23:22:31.363 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:31.349+0000 7f75077fe640 1 -- 172.21.15.46:0/812955157 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7520039040 con 0x55add00e4db0 2024-03-31T23:22:31.363 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:31.349+0000 7f75077fe640 1 -- 172.21.15.46:0/812955157 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 9554+0+0 (secure 0 0 0) 0x7f7520004b40 con 0x55add00e4db0 2024-03-31T23:22:31.363 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:31.349+0000 7f753136d740 1 -- 172.21.15.46:0/812955157 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7518005020 con 0x55add00e4db0 2024-03-31T23:22:31.363 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:31.349+0000 7f75077fe640 1 -- 172.21.15.46:0/812955157 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f752005b050 con 0x55add00e4db0 2024-03-31T23:22:31.363 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:31.349+0000 7f753136d740 10 client.17028 fetch_fsmap learned FSMap version 36 2024-03-31T23:22:31.364 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:31.349+0000 7f753136d740 1 -- 172.21.15.46:0/812955157 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x55add00f3920 con 0x55add00e4db0 2024-03-31T23:22:31.364 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:31.350+0000 7f75077fe640 1 -- 172.21.15.46:0/812955157 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7518005020 con 0x55add00e4db0 2024-03-31T23:22:31.364 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:31.350+0000 7f75077fe640 1 -- 172.21.15.46:0/812955157 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 36) ==== 8362+0+0 (secure 0 0 0) 0x7f752003f040 con 0x55add00e4db0 2024-03-31T23:22:31.364 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:31.350+0000 7f753136d740 10 client.17028 fetch_fsmap finished waiting for FSMap version 36 2024-03-31T23:22:31.364 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:31.350+0000 7f753136d740 10 client.17028 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-31T23:22:31.364 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:31.350+0000 7f753136d740 20 client.17028 populate_metadata read hostname 'smithi046' 2024-03-31T23:22:31.364 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:31.350+0000 7f753136d740 1 --2- 172.21.15.46:0/812955157 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55add00f4ea0 0x55add00f72b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:22:31.364 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:31.350+0000 7f753136d740 4 client.17028 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-31T23:22:31.364 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:31.350+0000 7f753136d740 1 -- 172.21.15.46:0/812955157 --> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55add0052120 con 0x55add00f4ea0 2024-03-31T23:22:31.364 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:31.350+0000 7f7527fff640 1 --2- 172.21.15.46:0/812955157 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55add00f4ea0 0x55add00f72b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:22:31.364 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:31.350+0000 7f75077fe640 10 client.17028 ms_handle_connect on v2:172.21.15.142:6838/2519143918 2024-03-31T23:22:31.364 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:31.350+0000 7f7527fff640 1 --2- 172.21.15.46:0/812955157 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55add00f4ea0 0x55add00f72b0 crc :-1 s=READY pgs=171 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1f2391f033f6a29d server_cookie=f3bfb905111cd69b in_seq=0 out_seq=0 2024-03-31T23:22:31.365 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:31.351+0000 7f75077fe640 1 -- 172.21.15.46:0/812955157 <== mds.0 v2:172.21.15.142:6838/2519143918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55add0052120 con 0x55add00f4ea0 2024-03-31T23:22:31.365 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:31.351+0000 7f75077fe640 10 client.17028 handle_command_reply: tid=0 2024-03-31T23:22:31.365 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:31.351+0000 7f753136d740 1 client.17028 shutdown 2024-03-31T23:22:31.365 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:31.351+0000 7f753136d740 1 -- 172.21.15.46:0/812955157 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55add00e4db0 msgr2=0x55add00e2840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:22:31.365 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:31.351+0000 7f753136d740 1 --2- 172.21.15.46:0/812955157 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55add00e4db0 0x55add00e2840 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7f7520036000 tx=0x7f7520005c50 comp rx=0 tx=0).stop 2024-03-31T23:22:31.365 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:31.351+0000 7f753136d740 1 -- 172.21.15.46:0/812955157 shutdown_connections 2024-03-31T23:22:31.365 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:31.351+0000 7f753136d740 1 --2- 172.21.15.46:0/812955157 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55add00f4ea0 0x55add00f72b0 crc :-1 s=READY pgs=171 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:22:31.365 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:31.351+0000 7f753136d740 1 --2- 172.21.15.46:0/812955157 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55add00e4db0 0x55add00e2840 unknown :-1 s=CLOSED pgs=820 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:22:31.365 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:31.351+0000 7f753136d740 1 --2- 172.21.15.46:0/812955157 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55add00dbfe0 0x55add00e2300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:22:31.365 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:31.351+0000 7f753136d740 1 --2- 172.21.15.46:0/812955157 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55add00db6c0 0x55add00e1dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:22:31.365 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:31.351+0000 7f75077fe640 0 client.17028 ms_handle_reset on v2:172.21.15.142:6838/2519143918 2024-03-31T23:22:31.365 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:31.351+0000 7f753136d740 1 -- 172.21.15.46:0/812955157 >> 172.21.15.46:0/812955157 conn(0x55add00d5fe0 msgr2=0x55add00d7d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:22:31.365 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:31.351+0000 7f753136d740 1 -- 172.21.15.46:0/812955157 shutdown_connections 2024-03-31T23:22:31.365 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:31.351+0000 7f753136d740 1 -- 172.21.15.46:0/812955157 wait complete. 2024-03-31T23:22:31.366 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:31.352+0000 7f753136d740 20 client.17028 trim_cache size 0 max 16384 2024-03-31T23:22:31.377 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:31.366+0000 7f753136d740 1 Processor -- start 2024-03-31T23:22:31.378 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:31.366+0000 7f753136d740 1 -- start start 2024-03-31T23:22:31.378 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:31.367+0000 7f753136d740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55add00db6c0 0x55add00e44f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:22:31.378 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:31.367+0000 7f753136d740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55add00dbfe0 0x55add00f0780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:22:31.378 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:31.367+0000 7f753136d740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55add00e4db0 0x55add00fcbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:22:31.378 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:31.367+0000 7f753136d740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f75281b13d0 con 0x55add00e4db0 2024-03-31T23:22:31.379 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:31.367+0000 7f753136d740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f75281aec40 con 0x55add00db6c0 2024-03-31T23:22:31.379 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:31.367+0000 7f753136d740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f75281acab0 con 0x55add00dbfe0 2024-03-31T23:22:31.379 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:31.367+0000 7f752cc4b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55add00e4db0 0x55add00fcbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:22:31.379 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:31.367+0000 7f7527fff640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55add00db6c0 0x55add00e44f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:22:31.379 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:31.367+0000 7f752cc4b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55add00e4db0 0x55add00fcbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60184/0 (socket says 172.21.15.46:60184) 2024-03-31T23:22:31.379 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:31.367+0000 7f752cc4b640 1 -- 172.21.15.46:0/3868620700 learned_addr learned my addr 172.21.15.46:0/3868620700 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:22:31.380 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:31.367+0000 7f75277fe640 1 --2- 172.21.15.46:0/3868620700 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55add00dbfe0 0x55add00f0780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:22:31.380 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:31.367+0000 7f7527fff640 1 -- 172.21.15.46:0/3868620700 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55add00dbfe0 msgr2=0x55add00f0780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:22:31.380 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:31.367+0000 7f7527fff640 1 --2- 172.21.15.46:0/3868620700 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55add00dbfe0 0x55add00f0780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:22:31.380 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:31.367+0000 7f7527fff640 1 -- 172.21.15.46:0/3868620700 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55add00e4db0 msgr2=0x55add00fcbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:22:31.380 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:31.367+0000 7f7527fff640 1 --2- 172.21.15.46:0/3868620700 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55add00e4db0 0x55add00fcbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:22:31.380 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:31.367+0000 7f7527fff640 1 -- 172.21.15.46:0/3868620700 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55add00e74d0 con 0x55add00db6c0 2024-03-31T23:22:31.381 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:31.367+0000 7f7527fff640 1 --2- 172.21.15.46:0/3868620700 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55add00db6c0 0x55add00e44f0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f7518000c00 tx=0x7f75180118d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2c88b1175222e846 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:22:31.381 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:31.367+0000 7f750dffb640 1 -- 172.21.15.46:0/3868620700 <== mon.2 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7518009b50 con 0x55add00db6c0 2024-03-31T23:22:31.381 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:31.368+0000 7f750dffb640 1 -- 172.21.15.46:0/3868620700 <== mon.2 v2:172.21.15.142:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7518004480 con 0x55add00db6c0 2024-03-31T23:22:31.381 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:31.368+0000 7f750dffb640 1 -- 172.21.15.46:0/3868620700 <== mon.2 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7518004800 con 0x55add00db6c0 2024-03-31T23:22:31.381 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:31.368+0000 7f753136d740 1 -- 172.21.15.46:0/3868620700 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55add00db6c0 msgr2=0x55add00e44f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:22:31.381 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:31.368+0000 7f753136d740 1 --2- 172.21.15.46:0/3868620700 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55add00db6c0 0x55add00e44f0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f7518000c00 tx=0x7f75180118d0 comp rx=0 tx=0).stop 2024-03-31T23:22:31.382 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:31.368+0000 7f753136d740 1 -- 172.21.15.46:0/3868620700 shutdown_connections 2024-03-31T23:22:31.382 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:31.368+0000 7f753136d740 1 --2- 172.21.15.46:0/3868620700 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55add00e4db0 0x55add00fcbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:22:31.382 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:31.368+0000 7f753136d740 1 --2- 172.21.15.46:0/3868620700 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55add00dbfe0 0x55add00f0780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:22:31.382 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:31.368+0000 7f753136d740 1 --2- 172.21.15.46:0/3868620700 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55add00db6c0 0x55add00e44f0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:22:31.382 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:31.368+0000 7f753136d740 1 -- 172.21.15.46:0/3868620700 >> 172.21.15.46:0/3868620700 conn(0x55add00d5fe0 msgr2=0x55add00dc670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:22:31.383 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:31.368+0000 7f753136d740 1 -- 172.21.15.46:0/3868620700 shutdown_connections 2024-03-31T23:22:31.383 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:31.368+0000 7f753136d740 1 -- 172.21.15.46:0/3868620700 wait complete. 2024-03-31T23:22:31.383 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:31.368+0000 7f753136d740 1 -- start start 2024-03-31T23:22:31.383 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:31.369+0000 7f753136d740 1 Processor -- start 2024-03-31T23:22:31.383 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:31.369+0000 7f753136d740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55add00db6c0 0x55add00ecfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:22:31.383 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:31.369+0000 7f753136d740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55add00dbfe0 0x55add00de610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:22:31.384 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:31.369+0000 7f753136d740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55add00e4db0 0x55add00deb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:22:31.384 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:31.369+0000 7f753136d740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f75281ad100 con 0x55add00db6c0 2024-03-31T23:22:31.384 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:31.369+0000 7f753136d740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f75281ae570 con 0x55add00e4db0 2024-03-31T23:22:31.384 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:31.369+0000 7f753136d740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f75281ad7d0 con 0x55add00dbfe0 2024-03-31T23:22:31.384 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:31.369+0000 7f753136d740 10 client.0 _ll_register_callbacks cb 0x7f7518005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T23:22:31.384 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:31.369+0000 7f7527fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55add00db6c0 0x55add00ecfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:22:31.385 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:31.369+0000 7f7527fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55add00db6c0 0x55add00ecfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60194/0 (socket says 172.21.15.46:60194) 2024-03-31T23:22:31.385 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:31.369+0000 7f7527fff640 1 -- 172.21.15.46:0/881019083 learned_addr learned my addr 172.21.15.46:0/881019083 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:22:31.385 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:31.369+0000 7f75277fe640 1 --2- 172.21.15.46:0/881019083 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55add00dbfe0 0x55add00de610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:22:31.385 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:31.369+0000 7f752cc4b640 1 --2- 172.21.15.46:0/881019083 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55add00e4db0 0x55add00deb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:22:31.385 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:31.369+0000 7f7527fff640 1 -- 172.21.15.46:0/881019083 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55add00dbfe0 msgr2=0x55add00de610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:22:31.385 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:31.369+0000 7f7527fff640 1 --2- 172.21.15.46:0/881019083 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55add00dbfe0 0x55add00de610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:22:31.386 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:31.369+0000 7f7527fff640 1 -- 172.21.15.46:0/881019083 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55add00e4db0 msgr2=0x55add00deb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:22:31.386 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:31.369+0000 7f7527fff640 1 --2- 172.21.15.46:0/881019083 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55add00e4db0 0x55add00deb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:22:31.386 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:31.369+0000 7f7527fff640 1 -- 172.21.15.46:0/881019083 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55add00df1f0 con 0x55add00db6c0 2024-03-31T23:22:31.386 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:31.369+0000 7f752cc4b640 1 --2- 172.21.15.46:0/881019083 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55add00e4db0 0x55add00deb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:22:31.386 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:31.370+0000 7f7527fff640 1 --2- 172.21.15.46:0/881019083 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55add00db6c0 0x55add00ecfa0 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7f7518009560 tx=0x7f7518005680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c1ff99cc5f993073 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:22:31.386 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:31.370+0000 7f75077fe640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-31T23:22:31.387 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:31.370+0000 7f75077fe640 1 -- 172.21.15.46:0/881019083 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7518011dd0 con 0x55add00db6c0 2024-03-31T23:22:31.387 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:31.370+0000 7f75077fe640 1 -- 172.21.15.46:0/881019083 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 9554+0+0 (secure 0 0 0) 0x7f7518029320 con 0x55add00db6c0 2024-03-31T23:22:31.387 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:31.370+0000 7f753136d740 1 -- 172.21.15.46:0/881019083 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7518004480 con 0x55add00db6c0 2024-03-31T23:22:31.387 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:31.370+0000 7f75077fe640 1 -- 172.21.15.46:0/881019083 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f7518043050 con 0x55add00db6c0 2024-03-31T23:22:31.387 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:31.370+0000 7f753136d740 10 client.17034 fetch_fsmap learned FSMap version 36 2024-03-31T23:22:31.387 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:31.370+0000 7f753136d740 1 -- 172.21.15.46:0/881019083 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x55add015b050 con 0x55add00db6c0 2024-03-31T23:22:31.388 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:31.370+0000 7f75077fe640 1 -- 172.21.15.46:0/881019083 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7518004480 con 0x55add00db6c0 2024-03-31T23:22:31.388 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:31.370+0000 7f75077fe640 1 -- 172.21.15.46:0/881019083 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 36) ==== 8362+0+0 (secure 0 0 0) 0x7f7518029950 con 0x55add00db6c0 2024-03-31T23:22:31.388 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:31.370+0000 7f753136d740 10 client.17034 fetch_fsmap finished waiting for FSMap version 36 2024-03-31T23:22:31.388 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:31.370+0000 7f753136d740 10 client.17034 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-31T23:22:31.388 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:31.370+0000 7f753136d740 20 client.17034 populate_metadata read hostname 'smithi046' 2024-03-31T23:22:31.388 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:31.370+0000 7f753136d740 1 --2- 172.21.15.46:0/881019083 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55add015b5f0 0x55add015da10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:22:31.389 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:31.370+0000 7f753136d740 4 client.17034 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-31T23:22:31.389 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:31.370+0000 7f753136d740 1 -- 172.21.15.46:0/881019083 --> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55add004be30 con 0x55add015b5f0 2024-03-31T23:22:31.389 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:31.371+0000 7f75277fe640 1 --2- 172.21.15.46:0/881019083 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55add015b5f0 0x55add015da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:22:31.389 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:31.371+0000 7f75277fe640 1 --2- 172.21.15.46:0/881019083 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55add015b5f0 0x55add015da10 crc :-1 s=READY pgs=172 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e05f651fac5611e4 server_cookie=9c85f7ec4c9a1bf6 in_seq=0 out_seq=0 2024-03-31T23:22:31.389 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:31.371+0000 7f75077fe640 10 client.17034 ms_handle_connect on v2:172.21.15.142:6838/2519143918 2024-03-31T23:22:31.398 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:31.396+0000 7f75077fe640 1 -- 172.21.15.46:0/881019083 <== mds.0 v2:172.21.15.142:6838/2519143918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55add004be30 con 0x55add015b5f0 2024-03-31T23:22:31.398 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:31.396+0000 7f75077fe640 10 client.17034 handle_command_reply: tid=0 2024-03-31T23:22:31.399 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:31.396+0000 7f753136d740 1 client.17034 shutdown 2024-03-31T23:22:31.399 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:31.397+0000 7f753136d740 1 -- 172.21.15.46:0/881019083 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55add00db6c0 msgr2=0x55add00ecfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:22:31.399 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:31.397+0000 7f753136d740 1 --2- 172.21.15.46:0/881019083 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55add00db6c0 0x55add00ecfa0 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7f7518009560 tx=0x7f7518005680 comp rx=0 tx=0).stop 2024-03-31T23:22:31.399 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:31.397+0000 7f753136d740 1 -- 172.21.15.46:0/881019083 shutdown_connections 2024-03-31T23:22:31.399 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:31.397+0000 7f753136d740 1 --2- 172.21.15.46:0/881019083 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55add015b5f0 0x55add015da10 crc :-1 s=READY pgs=172 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:22:31.399 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:31.397+0000 7f753136d740 1 --2- 172.21.15.46:0/881019083 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55add00e4db0 0x55add00deb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:22:31.399 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:31.397+0000 7f75077fe640 0 client.17034 ms_handle_reset on v2:172.21.15.142:6838/2519143918 2024-03-31T23:22:31.399 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:31.397+0000 7f753136d740 1 --2- 172.21.15.46:0/881019083 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55add00dbfe0 0x55add00de610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:22:31.399 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:31.397+0000 7f753136d740 1 --2- 172.21.15.46:0/881019083 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55add00db6c0 0x55add00ecfa0 unknown :-1 s=CLOSED pgs=821 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:22:31.399 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:31.397+0000 7f753136d740 1 -- 172.21.15.46:0/881019083 >> 172.21.15.46:0/881019083 conn(0x55add00d5fe0 msgr2=0x55add00d46f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:22:31.399 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:31.397+0000 7f753136d740 1 -- 172.21.15.46:0/881019083 shutdown_connections 2024-03-31T23:22:31.399 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:31.397+0000 7f753136d740 1 -- 172.21.15.46:0/881019083 wait complete. 2024-03-31T23:22:31.399 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:31.397+0000 7f753136d740 20 client.17034 trim_cache size 0 max 16384 2024-03-31T23:22:31.400 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout:{ 2024-03-31T23:22:31.400 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout: "message": "", 2024-03-31T23:22:31.400 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout: "return_code": 0 2024-03-31T23:22:31.400 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout:} 2024-03-31T23:22:31.401 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:31.397+0000 7f752e6ae640 1 -- 172.21.15.46:0/2829807926 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f75100a26b0 msgr2=0x7f75100a4b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:22:31.401 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:31.397+0000 7f752e6ae640 1 --2- 172.21.15.46:0/2829807926 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f75100a26b0 0x7f75100a4b70 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f7520005cc0 tx=0x7f7520032450 comp rx=0 tx=0).stop 2024-03-31T23:22:31.401 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:31.397+0000 7f752e6ae640 1 -- 172.21.15.46:0/2829807926 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f752806a910 msgr2=0x7f7528198300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:22:31.401 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:31.397+0000 7f752e6ae640 1 --2- 172.21.15.46:0/2829807926 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f752806a910 0x7f7528198300 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7f751c00e9f0 tx=0x7f751c00eeb0 comp rx=0 tx=0).stop 2024-03-31T23:22:31.401 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:31.398+0000 7f752e6ae640 1 -- 172.21.15.46:0/2829807926 shutdown_connections 2024-03-31T23:22:31.401 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:31.398+0000 7f752e6ae640 1 --2- 172.21.15.46:0/2829807926 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f75100a26b0 0x7f75100a4b70 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:22:31.401 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:31.398+0000 7f752e6ae640 1 --2- 172.21.15.46:0/2829807926 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f75281077d0 0x7f752819c630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:22:31.401 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:31.398+0000 7f752e6ae640 1 --2- 172.21.15.46:0/2829807926 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f752806a910 0x7f7528198300 unknown :-1 s=CLOSED pgs=819 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:22:31.401 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:31.398+0000 7f752e6ae640 1 --2- 172.21.15.46:0/2829807926 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7528069f50 0x7f7528197dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:22:31.401 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:31.398+0000 7f752e6ae640 1 -- 172.21.15.46:0/2829807926 >> 172.21.15.46:0/2829807926 conn(0x7f7528077420 msgr2=0x7f75280fe2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:22:31.402 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:31.398+0000 7f752e6ae640 1 -- 172.21.15.46:0/2829807926 shutdown_connections 2024-03-31T23:22:31.402 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:31.398+0000 7f752e6ae640 1 -- 172.21.15.46:0/2829807926 wait complete. 2024-03-31T23:22:32.215 INFO:tasks.workunit.client.0.smithi046.stdout: 1 216558 1.92 MB/sec execute 458 sec latency 4212.653 ms 2024-03-31T23:22:32.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:22:32 smithi046 ceph-mon[32498]: pgmap v337: 129 pgs: 129 active+clean; 126 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 224 KiB/s rd, 722 KiB/s wr, 65 op/s 2024-03-31T23:22:32.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:22:32 smithi142 ceph-mon[35271]: pgmap v337: 129 pgs: 129 active+clean; 126 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 224 KiB/s rd, 722 KiB/s wr, 65 op/s 2024-03-31T23:22:33.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:22:32 smithi162 ceph-mon[36237]: pgmap v337: 129 pgs: 129 active+clean; 126 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 224 KiB/s rd, 722 KiB/s wr, 65 op/s 2024-03-31T23:22:33.215 INFO:tasks.workunit.client.0.smithi046.stdout: 1 216570 1.91 MB/sec execute 459 sec latency 4742.658 ms 2024-03-31T23:22:34.215 INFO:tasks.workunit.client.0.smithi046.stdout: 1 216570 1.91 MB/sec execute 460 sec latency 1409.798 ms 2024-03-31T23:22:34.862 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:22:34 smithi046 ceph-mon[32498]: pgmap v338: 129 pgs: 129 active+clean; 126 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 224 KiB/s rd, 722 KiB/s wr, 65 op/s 2024-03-31T23:22:34.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:22:34 smithi142 ceph-mon[35271]: pgmap v338: 129 pgs: 129 active+clean; 126 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 224 KiB/s rd, 722 KiB/s wr, 65 op/s 2024-03-31T23:22:35.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:22:34 smithi162 ceph-mon[36237]: pgmap v338: 129 pgs: 129 active+clean; 126 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 224 KiB/s rd, 722 KiB/s wr, 65 op/s 2024-03-31T23:22:35.215 INFO:tasks.workunit.client.0.smithi046.stdout: 1 216570 1.90 MB/sec execute 461 sec latency 2409.846 ms 2024-03-31T23:22:36.215 INFO:tasks.workunit.client.0.smithi046.stdout: 1 216733 1.90 MB/sec execute 462 sec latency 2825.726 ms 2024-03-31T23:22:36.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:22:36 smithi046 ceph-mon[32498]: pgmap v339: 129 pgs: 129 active+clean; 123 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 261 KiB/s rd, 768 KiB/s wr, 76 op/s 2024-03-31T23:22:36.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:22:36 smithi142 ceph-mon[35271]: pgmap v339: 129 pgs: 129 active+clean; 123 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 261 KiB/s rd, 768 KiB/s wr, 76 op/s 2024-03-31T23:22:37.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:22:36 smithi162 ceph-mon[36237]: pgmap v339: 129 pgs: 129 active+clean; 123 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 261 KiB/s rd, 768 KiB/s wr, 76 op/s 2024-03-31T23:22:37.215 INFO:tasks.workunit.client.0.smithi046.stdout: 1 217006 1.90 MB/sec execute 463 sec latency 16.116 ms 2024-03-31T23:22:38.215 INFO:tasks.workunit.client.0.smithi046.stdout: 1 217459 1.90 MB/sec execute 464 sec latency 16.717 ms 2024-03-31T23:22:38.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:22:38 smithi046 ceph-mon[32498]: pgmap v340: 129 pgs: 129 active+clean; 123 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 105 KiB/s wr, 26 op/s 2024-03-31T23:22:38.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:22:38 smithi142 ceph-mon[35271]: pgmap v340: 129 pgs: 129 active+clean; 123 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 105 KiB/s wr, 26 op/s 2024-03-31T23:22:39.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:22:38 smithi162 ceph-mon[36237]: pgmap v340: 129 pgs: 129 active+clean; 123 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 105 KiB/s wr, 26 op/s 2024-03-31T23:22:39.215 INFO:tasks.workunit.client.0.smithi046.stdout: 1 217928 1.90 MB/sec execute 465 sec latency 20.438 ms 2024-03-31T23:22:40.215 INFO:tasks.workunit.client.0.smithi046.stdout: 1 218461 1.90 MB/sec execute 466 sec latency 21.283 ms 2024-03-31T23:22:40.862 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:22:40 smithi046 ceph-mon[32498]: pgmap v341: 129 pgs: 129 active+clean; 125 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 241 KiB/s rd, 527 KiB/s wr, 66 op/s 2024-03-31T23:22:40.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:22:40 smithi142 ceph-mon[35271]: pgmap v341: 129 pgs: 129 active+clean; 125 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 241 KiB/s rd, 527 KiB/s wr, 66 op/s 2024-03-31T23:22:41.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:22:40 smithi162 ceph-mon[36237]: pgmap v341: 129 pgs: 129 active+clean; 125 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 241 KiB/s rd, 527 KiB/s wr, 66 op/s 2024-03-31T23:22:41.215 INFO:tasks.workunit.client.0.smithi046.stdout: 1 218914 1.90 MB/sec execute 467 sec latency 24.877 ms 2024-03-31T23:22:42.215 INFO:tasks.workunit.client.0.smithi046.stdout: 1 219687 1.91 MB/sec execute 468 sec latency 16.695 ms 2024-03-31T23:22:42.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:22:42 smithi046 ceph-mon[32498]: pgmap v342: 129 pgs: 129 active+clean; 125 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 191 KiB/s rd, 467 KiB/s wr, 50 op/s 2024-03-31T23:22:42.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:22:42 smithi142 ceph-mon[35271]: pgmap v342: 129 pgs: 129 active+clean; 125 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 191 KiB/s rd, 467 KiB/s wr, 50 op/s 2024-03-31T23:22:43.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:22:42 smithi162 ceph-mon[36237]: pgmap v342: 129 pgs: 129 active+clean; 125 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 191 KiB/s rd, 467 KiB/s wr, 50 op/s 2024-03-31T23:22:43.215 INFO:tasks.workunit.client.0.smithi046.stdout: 1 219968 1.91 MB/sec execute 469 sec latency 89.045 ms 2024-03-31T23:22:44.215 INFO:tasks.workunit.client.0.smithi046.stdout: 1 220037 1.91 MB/sec execute 470 sec latency 874.348 ms 2024-03-31T23:22:44.500 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:44.496+0000 7fa578b48640 1 -- 172.21.15.46:0/2166158874 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa574101260 msgr2=0x7fa57410d460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:22:44.500 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:44.496+0000 7fa578b48640 1 --2- 172.21.15.46:0/2166158874 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa574101260 0x7fa57410d460 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7fa56c009cd0 tx=0x7fa56c02c0f0 comp rx=0 tx=0).stop 2024-03-31T23:22:44.500 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:44.496+0000 7fa578b48640 1 -- 172.21.15.46:0/2166158874 shutdown_connections 2024-03-31T23:22:44.500 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:44.496+0000 7fa578b48640 1 --2- 172.21.15.46:0/2166158874 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa574101260 0x7fa57410d460 unknown :-1 s=CLOSED pgs=822 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:22:44.500 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:44.496+0000 7fa578b48640 1 --2- 172.21.15.46:0/2166158874 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa5741008a0 0x7fa574100d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:22:44.500 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:44.496+0000 7fa578b48640 1 --2- 172.21.15.46:0/2166158874 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa574106750 0x7fa574106b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:22:44.500 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:44.496+0000 7fa578b48640 1 -- 172.21.15.46:0/2166158874 >> 172.21.15.46:0/2166158874 conn(0x7fa574077490 msgr2=0x7fa5740778a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:22:44.500 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:44.496+0000 7fa578b48640 1 -- 172.21.15.46:0/2166158874 shutdown_connections 2024-03-31T23:22:44.500 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:44.496+0000 7fa578b48640 1 -- 172.21.15.46:0/2166158874 wait complete. 2024-03-31T23:22:44.500 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:44.497+0000 7fa578b48640 1 Processor -- start 2024-03-31T23:22:44.501 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:44.497+0000 7fa578b48640 1 -- start start 2024-03-31T23:22:44.501 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:44.497+0000 7fa578b48640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa5741008a0 0x7fa574197e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:22:44.501 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:44.497+0000 7fa578b48640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa574101260 0x7fa574198380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:22:44.501 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:44.497+0000 7fa578b48640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa574106750 0x7fa57419c6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:22:44.501 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:44.497+0000 7fa578b48640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fa574111a60 con 0x7fa574101260 2024-03-31T23:22:44.501 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:44.497+0000 7fa578b48640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fa5741118e0 con 0x7fa574106750 2024-03-31T23:22:44.501 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:44.497+0000 7fa578b48640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fa574111be0 con 0x7fa5741008a0 2024-03-31T23:22:44.501 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:44.498+0000 7fa571d9b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa574101260 0x7fa574198380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:22:44.501 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:44.498+0000 7fa572d9d640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa574106750 0x7fa57419c6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:22:44.501 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:44.498+0000 7fa571d9b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa574101260 0x7fa574198380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:56676/0 (socket says 172.21.15.46:56676) 2024-03-31T23:22:44.501 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:44.498+0000 7fa572d9d640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa574106750 0x7fa57419c6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.46:46568/0 (socket says 172.21.15.46:46568) 2024-03-31T23:22:44.501 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:44.498+0000 7fa571d9b640 1 -- 172.21.15.46:0/1370296064 learned_addr learned my addr 172.21.15.46:0/1370296064 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:22:44.501 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:44.498+0000 7fa57259c640 1 --2- 172.21.15.46:0/1370296064 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa5741008a0 0x7fa574197e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:22:44.502 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:44.498+0000 7fa571d9b640 1 -- 172.21.15.46:0/1370296064 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa5741008a0 msgr2=0x7fa574197e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:22:44.502 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:44.498+0000 7fa571d9b640 1 --2- 172.21.15.46:0/1370296064 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa5741008a0 0x7fa574197e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:22:44.502 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:44.498+0000 7fa571d9b640 1 -- 172.21.15.46:0/1370296064 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa574106750 msgr2=0x7fa57419c6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:22:44.502 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:44.498+0000 7fa571d9b640 1 --2- 172.21.15.46:0/1370296064 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa574106750 0x7fa57419c6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:22:44.502 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:44.498+0000 7fa571d9b640 1 -- 172.21.15.46:0/1370296064 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa57419cde0 con 0x7fa574101260 2024-03-31T23:22:44.502 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:44.498+0000 7fa572d9d640 1 --2- 172.21.15.46:0/1370296064 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa574106750 0x7fa57419c6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:22:44.502 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:44.498+0000 7fa571d9b640 1 --2- 172.21.15.46:0/1370296064 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa574101260 0x7fa574198380 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7fa56800e9f0 tx=0x7fa56800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:22:44.502 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:44.498+0000 7fa5637fe640 1 -- 172.21.15.46:0/1370296064 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa56800cd80 con 0x7fa574101260 2024-03-31T23:22:44.502 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:44.498+0000 7fa5637fe640 1 -- 172.21.15.46:0/1370296064 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa5680054d0 con 0x7fa574101260 2024-03-31T23:22:44.502 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:44.498+0000 7fa578b48640 1 -- 172.21.15.46:0/1370296064 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa57419d0d0 con 0x7fa574101260 2024-03-31T23:22:44.502 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:44.498+0000 7fa5637fe640 1 -- 172.21.15.46:0/1370296064 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa568010490 con 0x7fa574101260 2024-03-31T23:22:44.503 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:44.498+0000 7fa578b48640 1 -- 172.21.15.46:0/1370296064 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa5741a4960 con 0x7fa574101260 2024-03-31T23:22:44.509 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:44.501+0000 7fa5637fe640 1 -- 172.21.15.46:0/1370296064 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 25) ==== 137451+0+0 (secure 0 0 0) 0x7fa568005020 con 0x7fa574101260 2024-03-31T23:22:44.509 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:44.501+0000 7fa57b807740 1 Processor -- start 2024-03-31T23:22:44.509 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:44.501+0000 7fa5637fe640 1 --2- 172.21.15.46:0/1370296064 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fa55c0a26b0 0x7fa55c0a4b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:22:44.509 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:44.501+0000 7fa5637fe640 1 -- 172.21.15.46:0/1370296064 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 9554+0+0 (secure 0 0 0) 0x7fa5680cf2b0 con 0x7fa574101260 2024-03-31T23:22:44.509 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:44.501+0000 7fa572d9d640 1 --2- 172.21.15.46:0/1370296064 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fa55c0a26b0 0x7fa55c0a4b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:22:44.510 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:44.501+0000 7fa57b807740 1 -- start start 2024-03-31T23:22:44.510 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:44.502+0000 7fa57b807740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556ca549a6c0 0x556ca549aaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:22:44.510 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:44.502+0000 7fa57b807740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556ca549afe0 0x556ca54a3870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:22:44.510 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:44.502+0000 7fa57b807740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x556ca54a3db0 0x556ca54a61a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:22:44.510 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:44.502+0000 7fa57b807740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x556ca53fd8f0 con 0x556ca549a6c0 2024-03-31T23:22:44.510 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:44.502+0000 7fa57b807740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x556ca53a68d0 con 0x556ca54a3db0 2024-03-31T23:22:44.510 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:44.502+0000 7fa57b807740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x556ca53f9990 con 0x556ca549afe0 2024-03-31T23:22:44.510 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:44.502+0000 7fa57259c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556ca549a6c0 0x556ca549aaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:22:44.510 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:44.502+0000 7fa57259c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556ca549a6c0 0x556ca549aaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:56682/0 (socket says 172.21.15.46:56682) 2024-03-31T23:22:44.510 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:44.502+0000 7fa57259c640 1 -- 172.21.15.46:0/363471215 learned_addr learned my addr 172.21.15.46:0/363471215 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:22:44.510 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:44.502+0000 7fa572d9d640 1 --2- 172.21.15.46:0/1370296064 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fa55c0a26b0 0x7fa55c0a4b70 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fa56c005cc0 tx=0x7fa56c005c50 comp rx=0 tx=0).ready entity=mgr.15654 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:22:44.510 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:44.502+0000 7fa571d9b640 1 --2- 172.21.15.46:0/363471215 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556ca549afe0 0x556ca54a3870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:22:44.510 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:44.502+0000 7fa572d9d640 1 --2- 172.21.15.46:0/363471215 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x556ca54a3db0 0x556ca54a61a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:22:44.511 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:44.502+0000 7fa57259c640 1 -- 172.21.15.46:0/363471215 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556ca549afe0 msgr2=0x556ca54a3870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:22:44.511 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:44.502+0000 7fa57259c640 1 --2- 172.21.15.46:0/363471215 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556ca549afe0 0x556ca54a3870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:22:44.511 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:44.502+0000 7fa57259c640 1 -- 172.21.15.46:0/363471215 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x556ca54a3db0 msgr2=0x556ca54a61a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:22:44.511 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:44.502+0000 7fa57259c640 1 --2- 172.21.15.46:0/363471215 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x556ca54a3db0 0x556ca54a61a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:22:44.511 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:44.502+0000 7fa57259c640 1 -- 172.21.15.46:0/363471215 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556ca5112210 con 0x556ca549a6c0 2024-03-31T23:22:44.511 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:44.502+0000 7fa571d9b640 1 --2- 172.21.15.46:0/363471215 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556ca549afe0 0x556ca54a3870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:22:44.511 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:44.502+0000 7fa57259c640 1 --2- 172.21.15.46:0/363471215 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556ca549a6c0 0x556ca549aaa0 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7fa564009cf0 tx=0x7fa564011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=38c64acc61990226 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:22:44.511 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:44.503+0000 7fa560ff9640 1 -- 172.21.15.46:0/363471215 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa564004230 con 0x556ca549a6c0 2024-03-31T23:22:44.511 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:44.503+0000 7fa560ff9640 1 -- 172.21.15.46:0/363471215 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa5640043d0 con 0x556ca549a6c0 2024-03-31T23:22:44.511 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:44.503+0000 7fa560ff9640 1 -- 172.21.15.46:0/363471215 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa564019830 con 0x556ca549a6c0 2024-03-31T23:22:44.511 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:44.503+0000 7fa57b807740 1 -- 172.21.15.46:0/363471215 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556ca549a6c0 msgr2=0x556ca549aaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:22:44.511 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:44.503+0000 7fa57b807740 1 --2- 172.21.15.46:0/363471215 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556ca549a6c0 0x556ca549aaa0 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7fa564009cf0 tx=0x7fa564011810 comp rx=0 tx=0).stop 2024-03-31T23:22:44.511 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:44.503+0000 7fa57b807740 1 -- 172.21.15.46:0/363471215 shutdown_connections 2024-03-31T23:22:44.511 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:44.503+0000 7fa57b807740 1 --2- 172.21.15.46:0/363471215 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x556ca54a3db0 0x556ca54a61a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:22:44.512 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:44.503+0000 7fa57b807740 1 --2- 172.21.15.46:0/363471215 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556ca549afe0 0x556ca54a3870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:22:44.512 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:44.503+0000 7fa57b807740 1 --2- 172.21.15.46:0/363471215 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556ca549a6c0 0x556ca549aaa0 unknown :-1 s=CLOSED pgs=824 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:22:44.512 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:44.503+0000 7fa57b807740 1 -- 172.21.15.46:0/363471215 >> 172.21.15.46:0/363471215 conn(0x556ca5494fe0 msgr2=0x556ca54953c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:22:44.512 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:44.504+0000 7fa57b807740 1 -- 172.21.15.46:0/363471215 shutdown_connections 2024-03-31T23:22:44.512 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:44.504+0000 7fa57b807740 1 -- 172.21.15.46:0/363471215 wait complete. 2024-03-31T23:22:44.512 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:44.504+0000 7fa57b807740 1 -- start start 2024-03-31T23:22:44.512 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:44.504+0000 7fa57b807740 1 Processor -- start 2024-03-31T23:22:44.512 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:44.504+0000 7fa57b807740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556ca549a6c0 0x556ca54a0d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:22:44.512 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:44.504+0000 7fa57b807740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556ca549afe0 0x556ca54a1290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:22:44.512 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:44.504+0000 7fa57b807740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x556ca54a3db0 0x556ca54a17d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:22:44.512 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:44.504+0000 7fa57b807740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x556ca54a69e0 con 0x556ca549a6c0 2024-03-31T23:22:44.512 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:44.504+0000 7fa57b807740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x556ca539bf80 con 0x556ca54a3db0 2024-03-31T23:22:44.512 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:44.504+0000 7fa57b807740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x556ca54a6b60 con 0x556ca549afe0 2024-03-31T23:22:44.512 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:44.504+0000 7fa57b807740 10 client.0 _ll_register_callbacks cb 0x556ca53f4290 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T23:22:44.513 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:44.504+0000 7fa57259c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556ca549a6c0 0x556ca54a0d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:22:44.513 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:44.504+0000 7fa57259c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556ca549a6c0 0x556ca54a0d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:56698/0 (socket says 172.21.15.46:56698) 2024-03-31T23:22:44.513 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:44.504+0000 7fa57259c640 1 -- 172.21.15.46:0/1542424334 learned_addr learned my addr 172.21.15.46:0/1542424334 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:22:44.513 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:44.504+0000 7fa572d9d640 1 --2- 172.21.15.46:0/1542424334 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x556ca54a3db0 0x556ca54a17d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:22:44.513 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:44.504+0000 7fa571d9b640 1 --2- 172.21.15.46:0/1542424334 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556ca549afe0 0x556ca54a1290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:22:44.513 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:44.505+0000 7fa57259c640 1 -- 172.21.15.46:0/1542424334 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556ca549afe0 msgr2=0x556ca54a1290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:22:44.513 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:44.505+0000 7fa57259c640 1 --2- 172.21.15.46:0/1542424334 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556ca549afe0 0x556ca54a1290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:22:44.513 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:44.505+0000 7fa57259c640 1 -- 172.21.15.46:0/1542424334 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x556ca54a3db0 msgr2=0x556ca54a17d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:22:44.513 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:44.505+0000 7fa57259c640 1 --2- 172.21.15.46:0/1542424334 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x556ca54a3db0 0x556ca54a17d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:22:44.513 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:44.505+0000 7fa57259c640 1 -- 172.21.15.46:0/1542424334 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556ca51de620 con 0x556ca549a6c0 2024-03-31T23:22:44.513 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:44.505+0000 7fa57259c640 1 --2- 172.21.15.46:0/1542424334 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556ca549a6c0 0x556ca54a0d50 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7fa564004630 tx=0x7fa5640052f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bddcda3965855110 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:22:44.513 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:44.505+0000 7fa559ffb640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-31T23:22:44.513 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:44.505+0000 7fa559ffb640 1 -- 172.21.15.46:0/1542424334 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa564027070 con 0x556ca549a6c0 2024-03-31T23:22:44.514 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:44.505+0000 7fa559ffb640 1 -- 172.21.15.46:0/1542424334 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 9554+0+0 (secure 0 0 0) 0x7fa5640048c0 con 0x556ca549a6c0 2024-03-31T23:22:44.514 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:44.505+0000 7fa57b807740 1 -- 172.21.15.46:0/1542424334 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa5640043d0 con 0x556ca549a6c0 2024-03-31T23:22:44.514 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:44.505+0000 7fa559ffb640 1 -- 172.21.15.46:0/1542424334 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7fa564033050 con 0x556ca549a6c0 2024-03-31T23:22:44.514 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:44.505+0000 7fa57b807740 10 client.17046 fetch_fsmap learned FSMap version 36 2024-03-31T23:22:44.514 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:44.505+0000 7fa57b807740 1 -- 172.21.15.46:0/1542424334 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x556ca54b2810 con 0x556ca549a6c0 2024-03-31T23:22:44.514 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:44.505+0000 7fa559ffb640 1 -- 172.21.15.46:0/1542424334 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa5640043d0 con 0x556ca549a6c0 2024-03-31T23:22:44.514 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:44.506+0000 7fa559ffb640 1 -- 172.21.15.46:0/1542424334 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 36) ==== 8362+0+0 (secure 0 0 0) 0x7fa564003960 con 0x556ca549a6c0 2024-03-31T23:22:44.514 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:44.506+0000 7fa57b807740 10 client.17046 fetch_fsmap finished waiting for FSMap version 36 2024-03-31T23:22:44.514 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:44.506+0000 7fa57b807740 10 client.17046 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-31T23:22:44.514 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:44.506+0000 7fa57b807740 20 client.17046 populate_metadata read hostname 'smithi046' 2024-03-31T23:22:44.514 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:44.506+0000 7fa57b807740 1 --2- 172.21.15.46:0/1542424334 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x556ca54b3c20 0x556ca54b6010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:22:44.514 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:44.506+0000 7fa57b807740 4 client.17046 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-31T23:22:44.514 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:44.506+0000 7fa57b807740 1 -- 172.21.15.46:0/1542424334 --> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556ca5411120 con 0x556ca54b3c20 2024-03-31T23:22:44.515 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:44.506+0000 7fa571d9b640 1 --2- 172.21.15.46:0/1542424334 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x556ca54b3c20 0x556ca54b6010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:22:44.515 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:44.506+0000 7fa559ffb640 10 client.17046 ms_handle_connect on v2:172.21.15.142:6838/2519143918 2024-03-31T23:22:44.515 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:44.506+0000 7fa571d9b640 1 --2- 172.21.15.46:0/1542424334 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x556ca54b3c20 0x556ca54b6010 crc :-1 s=READY pgs=173 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a5b88c57572bcd53 server_cookie=22f97f49c8420eeb in_seq=0 out_seq=0 2024-03-31T23:22:44.515 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:44.507+0000 7fa559ffb640 1 -- 172.21.15.46:0/1542424334 <== mds.0 v2:172.21.15.142:6838/2519143918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x556ca5411120 con 0x556ca54b3c20 2024-03-31T23:22:44.515 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:44.507+0000 7fa559ffb640 10 client.17046 handle_command_reply: tid=0 2024-03-31T23:22:44.515 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:44.507+0000 7fa57b807740 1 client.17046 shutdown 2024-03-31T23:22:44.515 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:44.507+0000 7fa57b807740 1 -- 172.21.15.46:0/1542424334 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556ca549a6c0 msgr2=0x556ca54a0d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:22:44.515 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:44.507+0000 7fa57b807740 1 --2- 172.21.15.46:0/1542424334 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556ca549a6c0 0x556ca54a0d50 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7fa564004630 tx=0x7fa5640052f0 comp rx=0 tx=0).stop 2024-03-31T23:22:44.515 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:44.507+0000 7fa57b807740 1 -- 172.21.15.46:0/1542424334 shutdown_connections 2024-03-31T23:22:44.515 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:44.507+0000 7fa57b807740 1 --2- 172.21.15.46:0/1542424334 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x556ca54b3c20 0x556ca54b6010 crc :-1 s=READY pgs=173 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:22:44.515 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:44.507+0000 7fa57b807740 1 --2- 172.21.15.46:0/1542424334 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x556ca54a3db0 0x556ca54a17d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:22:44.515 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:44.507+0000 7fa57b807740 1 --2- 172.21.15.46:0/1542424334 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556ca549afe0 0x556ca54a1290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:22:44.515 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:44.507+0000 7fa57b807740 1 --2- 172.21.15.46:0/1542424334 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556ca549a6c0 0x556ca54a0d50 unknown :-1 s=CLOSED pgs=825 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:22:44.515 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:44.507+0000 7fa559ffb640 0 client.17046 ms_handle_reset on v2:172.21.15.142:6838/2519143918 2024-03-31T23:22:44.516 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:44.507+0000 7fa57b807740 1 -- 172.21.15.46:0/1542424334 >> 172.21.15.46:0/1542424334 conn(0x556ca5494fe0 msgr2=0x556ca5496d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:22:44.516 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:44.507+0000 7fa57b807740 1 -- 172.21.15.46:0/1542424334 shutdown_connections 2024-03-31T23:22:44.516 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:44.507+0000 7fa57b807740 1 -- 172.21.15.46:0/1542424334 wait complete. 2024-03-31T23:22:44.516 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:44.507+0000 7fa57b807740 20 client.17046 trim_cache size 0 max 16384 2024-03-31T23:22:44.528 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:44.522+0000 7fa57b807740 1 Processor -- start 2024-03-31T23:22:44.528 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:44.523+0000 7fa57b807740 1 -- start start 2024-03-31T23:22:44.528 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:44.523+0000 7fa57b807740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x556ca5494fe0 0x556ca54af780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:22:44.528 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:44.523+0000 7fa57b807740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556ca549a6c0 0x556ca5493e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:22:44.529 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:44.523+0000 7fa57b807740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556ca549afe0 0x556ca54bb7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:22:44.529 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:44.523+0000 7fa57b807740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fa5741b14e0 con 0x556ca549a6c0 2024-03-31T23:22:44.529 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:44.523+0000 7fa57b807740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fa5741aed50 con 0x556ca5494fe0 2024-03-31T23:22:44.529 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:44.523+0000 7fa57b807740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fa5741acbc0 con 0x556ca549afe0 2024-03-31T23:22:44.529 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:44.523+0000 7fa571d9b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556ca549a6c0 0x556ca5493e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:22:44.529 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:44.523+0000 7fa57259c640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x556ca5494fe0 0x556ca54af780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:22:44.529 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:44.523+0000 7fa571d9b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556ca549a6c0 0x556ca5493e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:56708/0 (socket says 172.21.15.46:56708) 2024-03-31T23:22:44.529 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:44.523+0000 7fa571d9b640 1 -- 172.21.15.46:0/1115509768 learned_addr learned my addr 172.21.15.46:0/1115509768 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:22:44.529 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:44.523+0000 7fa572d9d640 1 --2- 172.21.15.46:0/1115509768 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556ca549afe0 0x556ca54bb7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:22:44.529 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:44.523+0000 7fa57259c640 1 -- 172.21.15.46:0/1115509768 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556ca549afe0 msgr2=0x556ca54bb7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:22:44.529 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:44.523+0000 7fa57259c640 1 --2- 172.21.15.46:0/1115509768 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556ca549afe0 0x556ca54bb7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:22:44.530 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:44.523+0000 7fa57259c640 1 -- 172.21.15.46:0/1115509768 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556ca549a6c0 msgr2=0x556ca5493e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:22:44.530 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:44.523+0000 7fa57259c640 1 --2- 172.21.15.46:0/1115509768 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556ca549a6c0 0x556ca5493e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:22:44.530 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:44.523+0000 7fa57259c640 1 -- 172.21.15.46:0/1115509768 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556ca54a61a0 con 0x556ca5494fe0 2024-03-31T23:22:44.530 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:44.523+0000 7fa571d9b640 1 --2- 172.21.15.46:0/1115509768 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556ca549a6c0 0x556ca5493e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:22:44.530 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:44.523+0000 7fa57259c640 1 --2- 172.21.15.46:0/1115509768 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x556ca5494fe0 0x556ca54af780 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fa564000c00 tx=0x7fa564005970 comp rx=0 tx=0).ready entity=mon.2 client_cookie=276659a7c3774ca1 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:22:44.530 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:44.524+0000 7fa55bfff640 1 -- 172.21.15.46:0/1115509768 <== mon.2 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa564030020 con 0x556ca5494fe0 2024-03-31T23:22:44.530 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:44.524+0000 7fa55bfff640 1 -- 172.21.15.46:0/1115509768 <== mon.2 v2:172.21.15.142:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa564011d10 con 0x556ca5494fe0 2024-03-31T23:22:44.530 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:44.524+0000 7fa55bfff640 1 -- 172.21.15.46:0/1115509768 <== mon.2 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa564027070 con 0x556ca5494fe0 2024-03-31T23:22:44.530 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:44.524+0000 7fa57b807740 1 -- 172.21.15.46:0/1115509768 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x556ca5494fe0 msgr2=0x556ca54af780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:22:44.530 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:44.524+0000 7fa57b807740 1 --2- 172.21.15.46:0/1115509768 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x556ca5494fe0 0x556ca54af780 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fa564000c00 tx=0x7fa564005970 comp rx=0 tx=0).stop 2024-03-31T23:22:44.531 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:44.524+0000 7fa57b807740 1 -- 172.21.15.46:0/1115509768 shutdown_connections 2024-03-31T23:22:44.531 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:44.524+0000 7fa57b807740 1 --2- 172.21.15.46:0/1115509768 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556ca549afe0 0x556ca54bb7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:22:44.531 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:44.524+0000 7fa57b807740 1 --2- 172.21.15.46:0/1115509768 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556ca549a6c0 0x556ca5493e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:22:44.531 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:44.524+0000 7fa57b807740 1 --2- 172.21.15.46:0/1115509768 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x556ca5494fe0 0x556ca54af780 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:22:44.531 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:44.524+0000 7fa57b807740 1 -- 172.21.15.46:0/1115509768 >> 172.21.15.46:0/1115509768 conn(0x556ca54b6380 msgr2=0x556ca54936f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:22:44.531 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:44.524+0000 7fa57b807740 1 -- 172.21.15.46:0/1115509768 shutdown_connections 2024-03-31T23:22:44.531 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:44.524+0000 7fa57b807740 1 -- 172.21.15.46:0/1115509768 wait complete. 2024-03-31T23:22:44.531 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:44.525+0000 7fa57b807740 1 -- start start 2024-03-31T23:22:44.531 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:44.525+0000 7fa57b807740 1 Processor -- start 2024-03-31T23:22:44.531 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:44.525+0000 7fa57b807740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x556ca5494fe0 0x556ca54a7920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:22:44.531 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:44.525+0000 7fa57b807740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556ca549a6c0 0x556ca54a7e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:22:44.532 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:44.525+0000 7fa57b807740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556ca549afe0 0x556ca54a83a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:22:44.532 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:44.525+0000 7fa57b807740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fa5741ad210 con 0x556ca549afe0 2024-03-31T23:22:44.532 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:44.525+0000 7fa57b807740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fa5741ae680 con 0x556ca5494fe0 2024-03-31T23:22:44.532 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:44.525+0000 7fa57b807740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fa5741ad8e0 con 0x556ca549a6c0 2024-03-31T23:22:44.532 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:44.525+0000 7fa57b807740 10 client.0 _ll_register_callbacks cb 0x7fa564005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T23:22:44.532 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:44.525+0000 7fa572d9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556ca549afe0 0x556ca54a83a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:22:44.532 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:44.525+0000 7fa572d9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556ca549afe0 0x556ca54a83a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:56722/0 (socket says 172.21.15.46:56722) 2024-03-31T23:22:44.532 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:44.525+0000 7fa57259c640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x556ca5494fe0 0x556ca54a7920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:22:44.532 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:44.525+0000 7fa572d9d640 1 -- 172.21.15.46:0/1307269232 learned_addr learned my addr 172.21.15.46:0/1307269232 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:22:44.532 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:44.525+0000 7fa571d9b640 1 --2- 172.21.15.46:0/1307269232 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556ca549a6c0 0x556ca54a7e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:22:44.533 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:44.525+0000 7fa572d9d640 1 -- 172.21.15.46:0/1307269232 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556ca549a6c0 msgr2=0x556ca54a7e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:22:44.533 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:44.525+0000 7fa572d9d640 1 --2- 172.21.15.46:0/1307269232 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556ca549a6c0 0x556ca54a7e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:22:44.533 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:44.525+0000 7fa572d9d640 1 -- 172.21.15.46:0/1307269232 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x556ca5494fe0 msgr2=0x556ca54a7920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:22:44.533 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:44.525+0000 7fa572d9d640 1 --2- 172.21.15.46:0/1307269232 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x556ca5494fe0 0x556ca54a7920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:22:44.533 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:44.525+0000 7fa572d9d640 1 -- 172.21.15.46:0/1307269232 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556ca5519090 con 0x556ca549afe0 2024-03-31T23:22:44.533 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:44.525+0000 7fa57259c640 1 --2- 172.21.15.46:0/1307269232 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x556ca5494fe0 0x556ca54a7920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:22:44.533 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:44.526+0000 7fa572d9d640 1 --2- 172.21.15.46:0/1307269232 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556ca549afe0 0x556ca54a83a0 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7fa56c02c700 tx=0x7fa56c02c0f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f40fc4149e0de55 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:22:44.533 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:44.526+0000 7fa559ffb640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-31T23:22:44.533 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:44.526+0000 7fa559ffb640 1 -- 172.21.15.46:0/1307269232 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa56c039040 con 0x556ca549afe0 2024-03-31T23:22:44.533 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:44.526+0000 7fa559ffb640 1 -- 172.21.15.46:0/1307269232 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 9554+0+0 (secure 0 0 0) 0x7fa56c0046d0 con 0x556ca549afe0 2024-03-31T23:22:44.534 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:44.526+0000 7fa57b807740 1 -- 172.21.15.46:0/1307269232 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa564011d10 con 0x556ca549afe0 2024-03-31T23:22:44.534 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:44.526+0000 7fa559ffb640 1 -- 172.21.15.46:0/1307269232 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7fa56c034440 con 0x556ca549afe0 2024-03-31T23:22:44.534 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:44.526+0000 7fa57b807740 10 client.17052 fetch_fsmap learned FSMap version 36 2024-03-31T23:22:44.534 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:44.526+0000 7fa57b807740 1 -- 172.21.15.46:0/1307269232 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x556ca551a380 con 0x556ca549afe0 2024-03-31T23:22:44.534 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:44.526+0000 7fa559ffb640 1 -- 172.21.15.46:0/1307269232 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa564011d10 con 0x556ca549afe0 2024-03-31T23:22:44.534 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:44.526+0000 7fa559ffb640 1 -- 172.21.15.46:0/1307269232 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 36) ==== 8362+0+0 (secure 0 0 0) 0x7fa56c055020 con 0x556ca549afe0 2024-03-31T23:22:44.534 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:44.526+0000 7fa57b807740 10 client.17052 fetch_fsmap finished waiting for FSMap version 36 2024-03-31T23:22:44.534 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:44.526+0000 7fa57b807740 10 client.17052 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-31T23:22:44.534 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:44.526+0000 7fa57b807740 20 client.17052 populate_metadata read hostname 'smithi046' 2024-03-31T23:22:44.534 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:44.526+0000 7fa57b807740 1 --2- 172.21.15.46:0/1307269232 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x556ca54a3db0 0x556ca551cba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:22:44.535 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:44.526+0000 7fa57b807740 4 client.17052 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-31T23:22:44.535 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:44.526+0000 7fa57b807740 1 -- 172.21.15.46:0/1307269232 --> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] -- command(tid 0: {"prefix": "flush journal"}) -- 0x556ca540ae30 con 0x556ca54a3db0 2024-03-31T23:22:44.535 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:44.527+0000 7fa57259c640 1 --2- 172.21.15.46:0/1307269232 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x556ca54a3db0 0x556ca551cba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:22:44.535 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:44.527+0000 7fa559ffb640 10 client.17052 ms_handle_connect on v2:172.21.15.142:6838/2519143918 2024-03-31T23:22:44.535 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:44.527+0000 7fa57259c640 1 --2- 172.21.15.46:0/1307269232 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x556ca54a3db0 0x556ca551cba0 crc :-1 s=READY pgs=174 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c0e3085eeeec1d28 server_cookie=fb1c38c8073bddd in_seq=0 out_seq=0 2024-03-31T23:22:44.557 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:44.555+0000 7fa559ffb640 1 -- 172.21.15.46:0/1307269232 <== mds.0 v2:172.21.15.142:6838/2519143918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x556ca540ae30 con 0x556ca54a3db0 2024-03-31T23:22:44.557 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:44.555+0000 7fa559ffb640 10 client.17052 handle_command_reply: tid=0 2024-03-31T23:22:44.557 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:44.555+0000 7fa57b807740 1 client.17052 shutdown 2024-03-31T23:22:44.557 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:44.556+0000 7fa57b807740 1 -- 172.21.15.46:0/1307269232 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556ca549afe0 msgr2=0x556ca54a83a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:22:44.557 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:44.556+0000 7fa57b807740 1 --2- 172.21.15.46:0/1307269232 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556ca549afe0 0x556ca54a83a0 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7fa56c02c700 tx=0x7fa56c02c0f0 comp rx=0 tx=0).stop 2024-03-31T23:22:44.557 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:44.556+0000 7fa57b807740 1 -- 172.21.15.46:0/1307269232 shutdown_connections 2024-03-31T23:22:44.557 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:44.556+0000 7fa57b807740 1 --2- 172.21.15.46:0/1307269232 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x556ca54a3db0 0x556ca551cba0 crc :-1 s=READY pgs=174 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:22:44.557 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:44.556+0000 7fa57b807740 1 --2- 172.21.15.46:0/1307269232 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556ca549afe0 0x556ca54a83a0 unknown :-1 s=CLOSED pgs=826 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:22:44.557 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:44.556+0000 7fa57b807740 1 --2- 172.21.15.46:0/1307269232 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556ca549a6c0 0x556ca54a7e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:22:44.557 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:44.556+0000 7fa559ffb640 0 client.17052 ms_handle_reset on v2:172.21.15.142:6838/2519143918 2024-03-31T23:22:44.558 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:44.556+0000 7fa57b807740 1 --2- 172.21.15.46:0/1307269232 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x556ca5494fe0 0x556ca54a7920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:22:44.558 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:44.556+0000 7fa57b807740 1 -- 172.21.15.46:0/1307269232 >> 172.21.15.46:0/1307269232 conn(0x556ca54b6380 msgr2=0x556ca5493f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:22:44.558 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:44.556+0000 7fa57b807740 1 -- 172.21.15.46:0/1307269232 shutdown_connections 2024-03-31T23:22:44.558 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:44.556+0000 7fa57b807740 1 -- 172.21.15.46:0/1307269232 wait complete. 2024-03-31T23:22:44.558 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:44.556+0000 7fa57b807740 20 client.17052 trim_cache size 0 max 16384 2024-03-31T23:22:44.558 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout:{ 2024-03-31T23:22:44.558 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout: "message": "", 2024-03-31T23:22:44.558 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout: "return_code": 0 2024-03-31T23:22:44.558 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout:} 2024-03-31T23:22:44.559 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:44.556+0000 7fa578b48640 1 -- 172.21.15.46:0/1370296064 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fa55c0a26b0 msgr2=0x7fa55c0a4b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:22:44.559 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:44.556+0000 7fa578b48640 1 --2- 172.21.15.46:0/1370296064 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fa55c0a26b0 0x7fa55c0a4b70 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fa56c005cc0 tx=0x7fa56c005c50 comp rx=0 tx=0).stop 2024-03-31T23:22:44.559 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:44.556+0000 7fa578b48640 1 -- 172.21.15.46:0/1370296064 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa574101260 msgr2=0x7fa574198380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:22:44.559 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:44.556+0000 7fa578b48640 1 --2- 172.21.15.46:0/1370296064 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa574101260 0x7fa574198380 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7fa56800e9f0 tx=0x7fa56800edf0 comp rx=0 tx=0).stop 2024-03-31T23:22:44.559 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:44.556+0000 7fa578b48640 1 -- 172.21.15.46:0/1370296064 shutdown_connections 2024-03-31T23:22:44.560 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:44.556+0000 7fa578b48640 1 --2- 172.21.15.46:0/1370296064 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fa55c0a26b0 0x7fa55c0a4b70 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:22:44.560 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:44.556+0000 7fa578b48640 1 --2- 172.21.15.46:0/1370296064 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa574106750 0x7fa57419c6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:22:44.560 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:44.556+0000 7fa578b48640 1 --2- 172.21.15.46:0/1370296064 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa574101260 0x7fa574198380 unknown :-1 s=CLOSED pgs=823 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:22:44.560 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:44.556+0000 7fa578b48640 1 --2- 172.21.15.46:0/1370296064 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa5741008a0 0x7fa574197e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:22:44.560 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:44.556+0000 7fa578b48640 1 -- 172.21.15.46:0/1370296064 >> 172.21.15.46:0/1370296064 conn(0x7fa574077490 msgr2=0x7fa5740ff490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:22:44.560 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:44.557+0000 7fa578b48640 1 -- 172.21.15.46:0/1370296064 shutdown_connections 2024-03-31T23:22:44.560 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:44.557+0000 7fa578b48640 1 -- 172.21.15.46:0/1370296064 wait complete. 2024-03-31T23:22:44.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:22:44 smithi046 ceph-mon[32498]: pgmap v343: 129 pgs: 129 active+clean; 125 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 191 KiB/s rd, 467 KiB/s wr, 50 op/s 2024-03-31T23:22:44.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:22:44 smithi142 ceph-mon[35271]: pgmap v343: 129 pgs: 129 active+clean; 125 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 191 KiB/s rd, 467 KiB/s wr, 50 op/s 2024-03-31T23:22:45.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:22:44 smithi162 ceph-mon[36237]: pgmap v343: 129 pgs: 129 active+clean; 125 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 191 KiB/s rd, 467 KiB/s wr, 50 op/s 2024-03-31T23:22:45.215 INFO:tasks.workunit.client.0.smithi046.stdout: 1 220037 1.90 MB/sec execute 471 sec latency 1874.454 ms 2024-03-31T23:22:46.215 INFO:tasks.workunit.client.0.smithi046.stdout: 1 220037 1.90 MB/sec execute 472 sec latency 2874.543 ms 2024-03-31T23:22:46.862 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:22:46 smithi046 ceph-mon[32498]: pgmap v344: 129 pgs: 129 active+clean; 126 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 359 KiB/s rd, 1.0 MiB/s wr, 96 op/s 2024-03-31T23:22:46.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:22:46 smithi142 ceph-mon[35271]: pgmap v344: 129 pgs: 129 active+clean; 126 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 359 KiB/s rd, 1.0 MiB/s wr, 96 op/s 2024-03-31T23:22:47.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:22:46 smithi162 ceph-mon[36237]: pgmap v344: 129 pgs: 129 active+clean; 126 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 359 KiB/s rd, 1.0 MiB/s wr, 96 op/s 2024-03-31T23:22:47.215 INFO:tasks.workunit.client.0.smithi046.stdout: 1 220037 1.90 MB/sec execute 473 sec latency 3874.649 ms 2024-03-31T23:22:48.215 INFO:tasks.workunit.client.0.smithi046.stdout: 1 220104 1.89 MB/sec execute 474 sec latency 4404.785 ms 2024-03-31T23:22:48.862 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:22:48 smithi046 ceph-mon[32498]: pgmap v345: 129 pgs: 129 active+clean; 126 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 322 KiB/s rd, 979 KiB/s wr, 84 op/s 2024-03-31T23:22:48.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:22:48 smithi142 ceph-mon[35271]: pgmap v345: 129 pgs: 129 active+clean; 126 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 322 KiB/s rd, 979 KiB/s wr, 84 op/s 2024-03-31T23:22:49.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:22:48 smithi162 ceph-mon[36237]: pgmap v345: 129 pgs: 129 active+clean; 126 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 322 KiB/s rd, 979 KiB/s wr, 84 op/s 2024-03-31T23:22:49.216 INFO:tasks.workunit.client.0.smithi046.stdout: 1 220104 1.89 MB/sec execute 475 sec latency 1218.537 ms 2024-03-31T23:22:50.216 INFO:tasks.workunit.client.0.smithi046.stdout: 1 220104 1.88 MB/sec execute 476 sec latency 2218.655 ms 2024-03-31T23:22:50.847 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:22:50 smithi162 ceph-mon[36237]: pgmap v346: 129 pgs: 129 active+clean; 127 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 368 KiB/s rd, 1.0 MiB/s wr, 100 op/s 2024-03-31T23:22:50.862 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:22:50 smithi046 ceph-mon[32498]: pgmap v346: 129 pgs: 129 active+clean; 127 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 368 KiB/s rd, 1.0 MiB/s wr, 100 op/s 2024-03-31T23:22:50.944 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:22:50 smithi142 ceph-mon[35271]: pgmap v346: 129 pgs: 129 active+clean; 127 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 368 KiB/s rd, 1.0 MiB/s wr, 100 op/s 2024-03-31T23:22:51.216 INFO:tasks.workunit.client.0.smithi046.stdout: 1 220104 1.88 MB/sec execute 477 sec latency 3218.703 ms 2024-03-31T23:22:52.216 INFO:tasks.workunit.client.0.smithi046.stdout: 1 220104 1.88 MB/sec execute 478 sec latency 4218.738 ms 2024-03-31T23:22:52.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:22:52 smithi046 ceph-mon[32498]: pgmap v347: 129 pgs: 129 active+clean; 127 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 214 KiB/s rd, 614 KiB/s wr, 61 op/s 2024-03-31T23:22:52.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:22:52 smithi142 ceph-mon[35271]: pgmap v347: 129 pgs: 129 active+clean; 127 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 214 KiB/s rd, 614 KiB/s wr, 61 op/s 2024-03-31T23:22:53.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:22:52 smithi162 ceph-mon[36237]: pgmap v347: 129 pgs: 129 active+clean; 127 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 214 KiB/s rd, 614 KiB/s wr, 61 op/s 2024-03-31T23:22:53.216 INFO:tasks.workunit.client.0.smithi046.stdout: 1 220238 1.87 MB/sec execute 479 sec latency 4746.864 ms 2024-03-31T23:22:54.216 INFO:tasks.workunit.client.0.smithi046.stdout: 1 220496 1.87 MB/sec execute 480 sec latency 15.714 ms 2024-03-31T23:22:54.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:22:54 smithi046 ceph-mon[32498]: pgmap v348: 129 pgs: 129 active+clean; 127 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 214 KiB/s rd, 614 KiB/s wr, 61 op/s 2024-03-31T23:22:54.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:22:54 smithi142 ceph-mon[35271]: pgmap v348: 129 pgs: 129 active+clean; 127 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 214 KiB/s rd, 614 KiB/s wr, 61 op/s 2024-03-31T23:22:55.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:22:54 smithi162 ceph-mon[36237]: pgmap v348: 129 pgs: 129 active+clean; 127 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 214 KiB/s rd, 614 KiB/s wr, 61 op/s 2024-03-31T23:22:55.216 INFO:tasks.workunit.client.0.smithi046.stdout: 1 220880 1.87 MB/sec execute 481 sec latency 16.179 ms 2024-03-31T23:22:56.218 INFO:tasks.workunit.client.0.smithi046.stdout: 1 221375 1.87 MB/sec execute 482 sec latency 19.191 ms 2024-03-31T23:22:56.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:22:56 smithi046 ceph-mon[32498]: pgmap v349: 129 pgs: 129 active+clean; 128 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 240 KiB/s rd, 795 KiB/s wr, 77 op/s 2024-03-31T23:22:56.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:22:56 smithi142 ceph-mon[35271]: pgmap v349: 129 pgs: 129 active+clean; 128 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 240 KiB/s rd, 795 KiB/s wr, 77 op/s 2024-03-31T23:22:57.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:22:56 smithi162 ceph-mon[36237]: pgmap v349: 129 pgs: 129 active+clean; 128 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 240 KiB/s rd, 795 KiB/s wr, 77 op/s 2024-03-31T23:22:57.217 INFO:tasks.workunit.client.0.smithi046.stdout: 1 221833 1.87 MB/sec execute 483 sec latency 22.351 ms 2024-03-31T23:22:57.658 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:57.655+0000 7f9e3999a640 1 -- 172.21.15.46:0/3186759961 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9e341020e0 msgr2=0x7f9e3410e5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:22:57.658 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:57.655+0000 7f9e3999a640 1 --2- 172.21.15.46:0/3186759961 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9e341020e0 0x7f9e3410e5f0 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7f9e2c009cd0 tx=0x7f9e2c02c0f0 comp rx=0 tx=0).stop 2024-03-31T23:22:57.658 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:57.655+0000 7f9e3999a640 1 -- 172.21.15.46:0/3186759961 shutdown_connections 2024-03-31T23:22:57.659 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:57.655+0000 7f9e3999a640 1 --2- 172.21.15.46:0/3186759961 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9e341020e0 0x7f9e3410e5f0 unknown :-1 s=CLOSED pgs=827 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:22:57.659 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:57.655+0000 7f9e3999a640 1 --2- 172.21.15.46:0/3186759961 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9e34101720 0x7f9e34101ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:22:57.659 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:57.655+0000 7f9e3999a640 1 --2- 172.21.15.46:0/3186759961 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9e34107730 0x7f9e34107b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:22:57.659 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:57.655+0000 7f9e3999a640 1 -- 172.21.15.46:0/3186759961 >> 172.21.15.46:0/3186759961 conn(0x7f9e340fd230 msgr2=0x7f9e340ff650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:22:57.659 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:57.655+0000 7f9e3999a640 1 -- 172.21.15.46:0/3186759961 shutdown_connections 2024-03-31T23:22:57.659 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:57.655+0000 7f9e3999a640 1 -- 172.21.15.46:0/3186759961 wait complete. 2024-03-31T23:22:57.659 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:57.655+0000 7f9e3999a640 1 Processor -- start 2024-03-31T23:22:57.659 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:57.656+0000 7f9e3999a640 1 -- start start 2024-03-31T23:22:57.659 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:57.656+0000 7f9e3999a640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9e34101720 0x7f9e34197ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:22:57.659 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:57.656+0000 7f9e3999a640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9e341020e0 0x7f9e34198410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:22:57.659 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:57.656+0000 7f9e3999a640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9e34107730 0x7f9e3419c740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:22:57.660 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:57.656+0000 7f9e3999a640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f9e341119f0 con 0x7f9e34107730 2024-03-31T23:22:57.660 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:57.656+0000 7f9e3999a640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f9e34111870 con 0x7f9e34101720 2024-03-31T23:22:57.660 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:57.656+0000 7f9e3999a640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f9e34111b70 con 0x7f9e341020e0 2024-03-31T23:22:57.660 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:57.656+0000 7f9e337fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9e34107730 0x7f9e3419c740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:22:57.660 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:57.656+0000 7f9e337fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9e34107730 0x7f9e3419c740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37274/0 (socket says 172.21.15.46:37274) 2024-03-31T23:22:57.660 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:57.656+0000 7f9e32ffd640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9e34101720 0x7f9e34197ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:22:57.660 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:57.656+0000 7f9e337fe640 1 -- 172.21.15.46:0/256367501 learned_addr learned my addr 172.21.15.46:0/256367501 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:22:57.660 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:57.656+0000 7f9e327fc640 1 --2- 172.21.15.46:0/256367501 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9e341020e0 0x7f9e34198410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:22:57.660 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:57.656+0000 7f9e337fe640 1 -- 172.21.15.46:0/256367501 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9e341020e0 msgr2=0x7f9e34198410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:22:57.660 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:57.656+0000 7f9e337fe640 1 --2- 172.21.15.46:0/256367501 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9e341020e0 0x7f9e34198410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:22:57.660 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:57.656+0000 7f9e337fe640 1 -- 172.21.15.46:0/256367501 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9e34101720 msgr2=0x7f9e34197ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:22:57.661 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:57.656+0000 7f9e337fe640 1 --2- 172.21.15.46:0/256367501 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9e34101720 0x7f9e34197ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:22:57.661 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:57.656+0000 7f9e337fe640 1 -- 172.21.15.46:0/256367501 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e3419ce70 con 0x7f9e34107730 2024-03-31T23:22:57.661 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:57.656+0000 7f9e32ffd640 1 --2- 172.21.15.46:0/256367501 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9e34101720 0x7f9e34197ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:22:57.661 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:57.656+0000 7f9e337fe640 1 --2- 172.21.15.46:0/256367501 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9e34107730 0x7f9e3419c740 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7f9e2c004af0 tx=0x7f9e2c035ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:22:57.661 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:57.656+0000 7f9e38998640 1 -- 172.21.15.46:0/256367501 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e2c039040 con 0x7f9e34107730 2024-03-31T23:22:57.661 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:57.657+0000 7f9e3999a640 1 -- 172.21.15.46:0/256367501 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9e3419d100 con 0x7f9e34107730 2024-03-31T23:22:57.661 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:57.657+0000 7f9e38998640 1 -- 172.21.15.46:0/256367501 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9e2c044070 con 0x7f9e34107730 2024-03-31T23:22:57.661 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:57.657+0000 7f9e38998640 1 -- 172.21.15.46:0/256367501 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e2c0042a0 con 0x7f9e34107730 2024-03-31T23:22:57.661 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:57.657+0000 7f9e3999a640 1 -- 172.21.15.46:0/256367501 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9e341a49f0 con 0x7f9e34107730 2024-03-31T23:22:57.668 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:57.659+0000 7f9e38998640 1 -- 172.21.15.46:0/256367501 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 25) ==== 137451+0+0 (secure 0 0 0) 0x7f9e2c003970 con 0x7f9e34107730 2024-03-31T23:22:57.668 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:57.659+0000 7f9e3c580740 1 Processor -- start 2024-03-31T23:22:57.668 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:57.659+0000 7f9e3c580740 1 -- start start 2024-03-31T23:22:57.668 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:57.660+0000 7f9e3c580740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56142a2b26c0 0x56142a2b2aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:22:57.668 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:57.660+0000 7f9e38998640 1 --2- 172.21.15.46:0/256367501 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f9e1c0a26b0 0x7f9e1c0a4b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:22:57.668 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:57.660+0000 7f9e3c580740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56142a2b2fe0 0x56142a2bb870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:22:57.668 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:57.660+0000 7f9e3c580740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56142a2bbdb0 0x56142a2be1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:22:57.669 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:57.660+0000 7f9e3c580740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x56142a2158f0 con 0x56142a2bbdb0 2024-03-31T23:22:57.669 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:57.660+0000 7f9e3c580740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x56142a1be8d0 con 0x56142a2b2fe0 2024-03-31T23:22:57.669 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:57.660+0000 7f9e3c580740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x56142a211990 con 0x56142a2b26c0 2024-03-31T23:22:57.669 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:57.660+0000 7f9e337fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56142a2bbdb0 0x56142a2be1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:22:57.669 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:57.660+0000 7f9e327fc640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56142a2b26c0 0x56142a2b2aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:22:57.669 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:57.660+0000 7f9e337fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56142a2bbdb0 0x56142a2be1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37280/0 (socket says 172.21.15.46:37280) 2024-03-31T23:22:57.669 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:57.660+0000 7f9e337fe640 1 -- 172.21.15.46:0/2529185860 learned_addr learned my addr 172.21.15.46:0/2529185860 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:22:57.669 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:57.660+0000 7f9e32ffd640 1 --2- 172.21.15.46:0/256367501 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f9e1c0a26b0 0x7f9e1c0a4b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:22:57.669 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:57.660+0000 7f9e327fc640 1 --2- 172.21.15.46:0/2529185860 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56142a2b2fe0 0x56142a2bb870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:22:57.669 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:57.660+0000 7f9e38998640 1 -- 172.21.15.46:0/256367501 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 9554+0+0 (secure 0 0 0) 0x7f9e2c0f1210 con 0x7f9e34107730 2024-03-31T23:22:57.669 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:57.660+0000 7f9e327fc640 1 -- 172.21.15.46:0/2529185860 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56142a2b2fe0 msgr2=0x56142a2bb870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:22:57.669 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:57.660+0000 7f9e327fc640 1 --2- 172.21.15.46:0/2529185860 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56142a2b2fe0 0x56142a2bb870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:22:57.670 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:57.660+0000 7f9e327fc640 1 -- 172.21.15.46:0/2529185860 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56142a2bbdb0 msgr2=0x56142a2be1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:22:57.670 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:57.660+0000 7f9e327fc640 1 --2- 172.21.15.46:0/2529185860 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56142a2bbdb0 0x56142a2be1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:22:57.670 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:57.660+0000 7f9e327fc640 1 -- 172.21.15.46:0/2529185860 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561429f2a210 con 0x56142a2b26c0 2024-03-31T23:22:57.670 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:57.660+0000 7f9e337fe640 1 --2- 172.21.15.46:0/2529185860 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56142a2bbdb0 0x56142a2be1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:22:57.670 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:57.660+0000 7f9e32ffd640 1 --2- 172.21.15.46:0/256367501 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f9e1c0a26b0 0x7f9e1c0a4b70 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f9e24009f40 tx=0x7f9e24005f90 comp rx=0 tx=0).ready entity=mgr.15654 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:22:57.670 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:57.660+0000 7f9e327fc640 1 --2- 172.21.15.46:0/2529185860 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56142a2b26c0 0x56142a2b2aa0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f9e28009fd0 tx=0x7f9e28004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=39c6bba4e5786b42 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:22:57.670 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:57.661+0000 7f9e19ffb640 1 -- 172.21.15.46:0/2529185860 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e28004050 con 0x56142a2b26c0 2024-03-31T23:22:57.670 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:57.661+0000 7f9e19ffb640 1 -- 172.21.15.46:0/2529185860 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9e280041f0 con 0x56142a2b26c0 2024-03-31T23:22:57.670 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:57.661+0000 7f9e19ffb640 1 -- 172.21.15.46:0/2529185860 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e2800e610 con 0x56142a2b26c0 2024-03-31T23:22:57.670 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:57.661+0000 7f9e3c580740 1 -- 172.21.15.46:0/2529185860 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56142a2b26c0 msgr2=0x56142a2b2aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:22:57.670 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:57.661+0000 7f9e3c580740 1 --2- 172.21.15.46:0/2529185860 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56142a2b26c0 0x56142a2b2aa0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f9e28009fd0 tx=0x7f9e28004730 comp rx=0 tx=0).stop 2024-03-31T23:22:57.670 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:57.661+0000 7f9e3c580740 1 -- 172.21.15.46:0/2529185860 shutdown_connections 2024-03-31T23:22:57.671 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:57.661+0000 7f9e3c580740 1 --2- 172.21.15.46:0/2529185860 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56142a2bbdb0 0x56142a2be1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:22:57.671 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:57.661+0000 7f9e3c580740 1 --2- 172.21.15.46:0/2529185860 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56142a2b2fe0 0x56142a2bb870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:22:57.671 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:57.661+0000 7f9e3c580740 1 --2- 172.21.15.46:0/2529185860 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56142a2b26c0 0x56142a2b2aa0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:22:57.671 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:57.661+0000 7f9e3c580740 1 -- 172.21.15.46:0/2529185860 >> 172.21.15.46:0/2529185860 conn(0x56142a2acfe0 msgr2=0x56142a2ad3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:22:57.671 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:57.662+0000 7f9e3c580740 1 -- 172.21.15.46:0/2529185860 shutdown_connections 2024-03-31T23:22:57.671 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:57.662+0000 7f9e3c580740 1 -- 172.21.15.46:0/2529185860 wait complete. 2024-03-31T23:22:57.671 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:57.662+0000 7f9e3c580740 1 -- start start 2024-03-31T23:22:57.671 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:57.662+0000 7f9e3c580740 1 Processor -- start 2024-03-31T23:22:57.671 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:57.662+0000 7f9e3c580740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56142a2b26c0 0x56142a2b8db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:22:57.671 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:57.662+0000 7f9e3c580740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56142a2b2fe0 0x56142a2b92f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:22:57.672 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:57.662+0000 7f9e3c580740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56142a2bbdb0 0x56142a2b9830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:22:57.672 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:57.662+0000 7f9e3c580740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x56142a2be980 con 0x56142a2b2fe0 2024-03-31T23:22:57.672 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:57.662+0000 7f9e3c580740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x56142a1b3f80 con 0x56142a2bbdb0 2024-03-31T23:22:57.672 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:57.662+0000 7f9e3c580740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x56142a2beb00 con 0x56142a2b26c0 2024-03-31T23:22:57.672 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:57.663+0000 7f9e3c580740 10 client.0 _ll_register_callbacks cb 0x56142a20c290 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T23:22:57.672 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:57.663+0000 7f9e327fc640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56142a2b2fe0 0x56142a2b92f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:22:57.672 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:57.663+0000 7f9e327fc640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56142a2b2fe0 0x56142a2b92f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37294/0 (socket says 172.21.15.46:37294) 2024-03-31T23:22:57.672 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:57.663+0000 7f9e327fc640 1 -- 172.21.15.46:0/745573995 learned_addr learned my addr 172.21.15.46:0/745573995 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:22:57.672 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:57.663+0000 7f9e337fe640 1 --2- 172.21.15.46:0/745573995 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56142a2bbdb0 0x56142a2b9830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:22:57.672 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:57.663+0000 7f9e32ffd640 1 --2- 172.21.15.46:0/745573995 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56142a2b26c0 0x56142a2b8db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:22:57.672 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:57.663+0000 7f9e337fe640 1 -- 172.21.15.46:0/745573995 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56142a2b26c0 msgr2=0x56142a2b8db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:22:57.672 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:57.663+0000 7f9e337fe640 1 --2- 172.21.15.46:0/745573995 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56142a2b26c0 0x56142a2b8db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:22:57.673 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:57.663+0000 7f9e337fe640 1 -- 172.21.15.46:0/745573995 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56142a2b2fe0 msgr2=0x56142a2b92f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:22:57.673 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:57.663+0000 7f9e337fe640 1 --2- 172.21.15.46:0/745573995 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56142a2b2fe0 0x56142a2b92f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:22:57.673 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:57.663+0000 7f9e337fe640 1 -- 172.21.15.46:0/745573995 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561429ff6620 con 0x56142a2bbdb0 2024-03-31T23:22:57.673 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:57.663+0000 7f9e327fc640 1 --2- 172.21.15.46:0/745573995 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56142a2b2fe0 0x56142a2b92f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:22:57.673 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:57.663+0000 7f9e337fe640 1 --2- 172.21.15.46:0/745573995 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56142a2bbdb0 0x56142a2b9830 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f9e2c0bd620 tx=0x7f9e2c02c0f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=87b947230af894e server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:22:57.673 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:57.663+0000 7f9e12ffd640 10 client.0 ms_handle_connect on v2:172.21.15.142:3300/0 2024-03-31T23:22:57.673 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:57.663+0000 7f9e12ffd640 1 -- 172.21.15.46:0/745573995 <== mon.2 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e2c0bcff0 con 0x56142a2bbdb0 2024-03-31T23:22:57.673 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:57.663+0000 7f9e12ffd640 1 -- 172.21.15.46:0/745573995 <== mon.2 v2:172.21.15.142:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 9554+0+0 (secure 0 0 0) 0x7f9e2c0b9d00 con 0x56142a2bbdb0 2024-03-31T23:22:57.673 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:57.663+0000 7f9e3c580740 1 -- 172.21.15.46:0/745573995 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9e280041f0 con 0x56142a2bbdb0 2024-03-31T23:22:57.673 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:57.663+0000 7f9e12ffd640 1 -- 172.21.15.46:0/745573995 <== mon.2 v2:172.21.15.142:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f9e2c0cff70 con 0x56142a2bbdb0 2024-03-31T23:22:57.673 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:57.663+0000 7f9e3c580740 10 client.26897 fetch_fsmap learned FSMap version 36 2024-03-31T23:22:57.673 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:57.664+0000 7f9e3c580740 1 -- 172.21.15.46:0/745573995 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x56142a2ca810 con 0x56142a2bbdb0 2024-03-31T23:22:57.674 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:57.664+0000 7f9e12ffd640 1 -- 172.21.15.46:0/745573995 <== mon.2 v2:172.21.15.142:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9e280041f0 con 0x56142a2bbdb0 2024-03-31T23:22:57.674 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:57.664+0000 7f9e12ffd640 1 -- 172.21.15.46:0/745573995 <== mon.2 v2:172.21.15.142:3300/0 5 ==== fsmap(e 36) ==== 8362+0+0 (secure 0 0 0) 0x7f9e2c03c030 con 0x56142a2bbdb0 2024-03-31T23:22:57.674 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:57.664+0000 7f9e3c580740 10 client.26897 fetch_fsmap finished waiting for FSMap version 36 2024-03-31T23:22:57.674 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:57.664+0000 7f9e3c580740 10 client.26897 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-31T23:22:57.674 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:57.664+0000 7f9e3c580740 20 client.26897 populate_metadata read hostname 'smithi046' 2024-03-31T23:22:57.674 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:57.664+0000 7f9e3c580740 1 --2- 172.21.15.46:0/745573995 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x56142a2cbd40 0x56142a2ce180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:22:57.674 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:57.664+0000 7f9e3c580740 4 client.26897 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-31T23:22:57.674 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:57.664+0000 7f9e3c580740 1 -- 172.21.15.46:0/745573995 --> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56142a229120 con 0x56142a2cbd40 2024-03-31T23:22:57.674 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:57.664+0000 7f9e32ffd640 1 --2- 172.21.15.46:0/745573995 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x56142a2cbd40 0x56142a2ce180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:22:57.674 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:57.665+0000 7f9e12ffd640 10 client.26897 ms_handle_connect on v2:172.21.15.142:6838/2519143918 2024-03-31T23:22:57.674 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:57.665+0000 7f9e32ffd640 1 --2- 172.21.15.46:0/745573995 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x56142a2cbd40 0x56142a2ce180 crc :-1 s=READY pgs=175 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b1ae49760bfb618e server_cookie=72b856827d8f8724 in_seq=0 out_seq=0 2024-03-31T23:22:57.674 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:57.665+0000 7f9e12ffd640 1 -- 172.21.15.46:0/745573995 <== mds.0 v2:172.21.15.142:6838/2519143918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x56142a229120 con 0x56142a2cbd40 2024-03-31T23:22:57.675 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:57.665+0000 7f9e12ffd640 10 client.26897 handle_command_reply: tid=0 2024-03-31T23:22:57.675 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:57.665+0000 7f9e3c580740 1 client.26897 shutdown 2024-03-31T23:22:57.675 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:57.665+0000 7f9e3c580740 1 -- 172.21.15.46:0/745573995 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56142a2bbdb0 msgr2=0x56142a2b9830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:22:57.675 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:57.665+0000 7f9e3c580740 1 --2- 172.21.15.46:0/745573995 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56142a2bbdb0 0x56142a2b9830 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f9e2c0bd620 tx=0x7f9e2c02c0f0 comp rx=0 tx=0).stop 2024-03-31T23:22:57.675 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:57.665+0000 7f9e3c580740 1 -- 172.21.15.46:0/745573995 shutdown_connections 2024-03-31T23:22:57.675 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:57.665+0000 7f9e3c580740 1 --2- 172.21.15.46:0/745573995 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x56142a2cbd40 0x56142a2ce180 crc :-1 s=READY pgs=175 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:22:57.675 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:57.665+0000 7f9e3c580740 1 --2- 172.21.15.46:0/745573995 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56142a2bbdb0 0x56142a2b9830 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:22:57.675 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:57.665+0000 7f9e3c580740 1 --2- 172.21.15.46:0/745573995 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56142a2b2fe0 0x56142a2b92f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:22:57.675 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:57.665+0000 7f9e3c580740 1 --2- 172.21.15.46:0/745573995 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56142a2b26c0 0x56142a2b8db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:22:57.675 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:57.665+0000 7f9e12ffd640 0 client.26897 ms_handle_reset on v2:172.21.15.142:6838/2519143918 2024-03-31T23:22:57.675 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:57.665+0000 7f9e3c580740 1 -- 172.21.15.46:0/745573995 >> 172.21.15.46:0/745573995 conn(0x56142a2acfe0 msgr2=0x56142a2aed30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:22:57.675 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:57.666+0000 7f9e3c580740 1 -- 172.21.15.46:0/745573995 shutdown_connections 2024-03-31T23:22:57.675 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:57.666+0000 7f9e3c580740 1 -- 172.21.15.46:0/745573995 wait complete. 2024-03-31T23:22:57.676 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:57.666+0000 7f9e3c580740 20 client.26897 trim_cache size 0 max 16384 2024-03-31T23:22:57.695 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:57.689+0000 7f9e3c580740 1 Processor -- start 2024-03-31T23:22:57.696 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:57.689+0000 7f9e3c580740 1 -- start start 2024-03-31T23:22:57.696 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:57.689+0000 7f9e3c580740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56142a2b26c0 0x56142a2bb4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:22:57.696 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:57.689+0000 7f9e3c580740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56142a2b2fe0 0x56142a2c7720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:22:57.696 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:57.689+0000 7f9e3c580740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56142a2bbdb0 0x56142a2a7bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:22:57.696 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:57.689+0000 7f9e3c580740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f9e341b1530 con 0x56142a2bbdb0 2024-03-31T23:22:57.696 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:57.689+0000 7f9e3c580740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f9e341aeda0 con 0x56142a2b2fe0 2024-03-31T23:22:57.696 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:57.689+0000 7f9e3c580740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f9e341acc10 con 0x56142a2b26c0 2024-03-31T23:22:57.696 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:57.690+0000 7f9e337fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56142a2bbdb0 0x56142a2a7bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:22:57.696 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:57.690+0000 7f9e337fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56142a2bbdb0 0x56142a2a7bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37308/0 (socket says 172.21.15.46:37308) 2024-03-31T23:22:57.696 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:57.690+0000 7f9e337fe640 1 -- 172.21.15.46:0/956483057 learned_addr learned my addr 172.21.15.46:0/956483057 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:22:57.696 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:57.690+0000 7f9e327fc640 1 --2- 172.21.15.46:0/956483057 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56142a2b2fe0 0x56142a2c7720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:22:57.696 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:57.690+0000 7f9e32ffd640 1 --2- 172.21.15.46:0/956483057 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56142a2b26c0 0x56142a2bb4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:22:57.697 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:57.690+0000 7f9e337fe640 1 -- 172.21.15.46:0/956483057 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56142a2b26c0 msgr2=0x56142a2bb4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:22:57.697 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:57.690+0000 7f9e337fe640 1 --2- 172.21.15.46:0/956483057 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56142a2b26c0 0x56142a2bb4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:22:57.697 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:57.690+0000 7f9e337fe640 1 -- 172.21.15.46:0/956483057 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56142a2b2fe0 msgr2=0x56142a2c7720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:22:57.697 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:57.690+0000 7f9e337fe640 1 --2- 172.21.15.46:0/956483057 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56142a2b2fe0 0x56142a2c7720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:22:57.697 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:57.690+0000 7f9e337fe640 1 -- 172.21.15.46:0/956483057 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56142a2b7d60 con 0x56142a2bbdb0 2024-03-31T23:22:57.697 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:57.690+0000 7f9e337fe640 1 --2- 172.21.15.46:0/956483057 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56142a2bbdb0 0x56142a2a7bf0 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7f9e2c0c5b70 tx=0x7f9e2c0095c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a0405987406f382 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:22:57.697 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:57.690+0000 7f9e197fa640 1 -- 172.21.15.46:0/956483057 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e2c0d0150 con 0x56142a2bbdb0 2024-03-31T23:22:57.697 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:57.690+0000 7f9e197fa640 1 -- 172.21.15.46:0/956483057 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9e2c034700 con 0x56142a2bbdb0 2024-03-31T23:22:57.697 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:57.690+0000 7f9e197fa640 1 -- 172.21.15.46:0/956483057 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e2c0bcff0 con 0x56142a2bbdb0 2024-03-31T23:22:57.697 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:57.691+0000 7f9e3c580740 1 -- 172.21.15.46:0/956483057 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56142a2bbdb0 msgr2=0x56142a2a7bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:22:57.697 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:57.691+0000 7f9e3c580740 1 --2- 172.21.15.46:0/956483057 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56142a2bbdb0 0x56142a2a7bf0 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7f9e2c0c5b70 tx=0x7f9e2c0095c0 comp rx=0 tx=0).stop 2024-03-31T23:22:57.697 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:57.691+0000 7f9e3c580740 1 -- 172.21.15.46:0/956483057 shutdown_connections 2024-03-31T23:22:57.698 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:57.691+0000 7f9e3c580740 1 --2- 172.21.15.46:0/956483057 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56142a2bbdb0 0x56142a2a7bf0 unknown :-1 s=CLOSED pgs=829 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:22:57.698 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:57.691+0000 7f9e3c580740 1 --2- 172.21.15.46:0/956483057 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56142a2b2fe0 0x56142a2c7720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:22:57.698 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:57.691+0000 7f9e3c580740 1 --2- 172.21.15.46:0/956483057 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56142a2b26c0 0x56142a2bb4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:22:57.698 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:57.691+0000 7f9e3c580740 1 -- 172.21.15.46:0/956483057 >> 172.21.15.46:0/956483057 conn(0x56142a2acfe0 msgr2=0x56142a2abf80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:22:57.698 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:57.691+0000 7f9e3c580740 1 -- 172.21.15.46:0/956483057 shutdown_connections 2024-03-31T23:22:57.698 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:57.691+0000 7f9e3c580740 1 -- 172.21.15.46:0/956483057 wait complete. 2024-03-31T23:22:57.698 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:57.691+0000 7f9e3c580740 1 -- start start 2024-03-31T23:22:57.698 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:57.692+0000 7f9e3c580740 1 Processor -- start 2024-03-31T23:22:57.698 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:57.692+0000 7f9e3c580740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56142a2b26c0 0x56142a2c39f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:22:57.698 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:57.692+0000 7f9e3c580740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56142a2b2fe0 0x56142a2c3f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:22:57.698 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:57.692+0000 7f9e3c580740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56142a2bbdb0 0x56142a2bfe00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:22:57.699 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:57.692+0000 7f9e3c580740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f9e341ad260 con 0x56142a2b26c0 2024-03-31T23:22:57.699 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:57.692+0000 7f9e3c580740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f9e341ae6d0 con 0x56142a2b2fe0 2024-03-31T23:22:57.699 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:57.692+0000 7f9e3c580740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f9e341ad930 con 0x56142a2bbdb0 2024-03-31T23:22:57.699 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:57.692+0000 7f9e3c580740 10 client.0 _ll_register_callbacks cb 0x7f9e2c0ee600 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T23:22:57.699 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:57.692+0000 7f9e32ffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56142a2b26c0 0x56142a2c39f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:22:57.699 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:57.692+0000 7f9e32ffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56142a2b26c0 0x56142a2c39f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37318/0 (socket says 172.21.15.46:37318) 2024-03-31T23:22:57.699 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:57.692+0000 7f9e32ffd640 1 -- 172.21.15.46:0/1365168552 learned_addr learned my addr 172.21.15.46:0/1365168552 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:22:57.699 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:57.692+0000 7f9e327fc640 1 --2- 172.21.15.46:0/1365168552 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56142a2b2fe0 0x56142a2c3f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:22:57.699 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:57.692+0000 7f9e337fe640 1 --2- 172.21.15.46:0/1365168552 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56142a2bbdb0 0x56142a2bfe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:22:57.699 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:57.692+0000 7f9e32ffd640 1 -- 172.21.15.46:0/1365168552 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56142a2bbdb0 msgr2=0x56142a2bfe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:22:57.699 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:57.692+0000 7f9e32ffd640 1 --2- 172.21.15.46:0/1365168552 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56142a2bbdb0 0x56142a2bfe00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:22:57.699 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:57.692+0000 7f9e32ffd640 1 -- 172.21.15.46:0/1365168552 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56142a2b2fe0 msgr2=0x56142a2c3f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:22:57.700 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:57.692+0000 7f9e32ffd640 1 --2- 172.21.15.46:0/1365168552 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56142a2b2fe0 0x56142a2c3f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:22:57.700 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:57.692+0000 7f9e32ffd640 1 -- 172.21.15.46:0/1365168552 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56142a2c0600 con 0x56142a2b26c0 2024-03-31T23:22:57.700 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:57.692+0000 7f9e337fe640 1 --2- 172.21.15.46:0/1365168552 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56142a2bbdb0 0x56142a2bfe00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:22:57.700 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:57.692+0000 7f9e32ffd640 1 --2- 172.21.15.46:0/1365168552 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56142a2b26c0 0x56142a2c39f0 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x56142a2ce9c0 tx=0x7f9e24004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b39b4cc1e32e3b80 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:22:57.700 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:57.692+0000 7f9e12ffd640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-31T23:22:57.700 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:57.693+0000 7f9e12ffd640 1 -- 172.21.15.46:0/1365168552 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e24003e50 con 0x56142a2b26c0 2024-03-31T23:22:57.700 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:57.693+0000 7f9e12ffd640 1 -- 172.21.15.46:0/1365168552 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 9554+0+0 (secure 0 0 0) 0x7f9e24027a30 con 0x56142a2b26c0 2024-03-31T23:22:57.700 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:57.693+0000 7f9e3c580740 1 -- 172.21.15.46:0/1365168552 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x56142a222e30 con 0x56142a2b26c0 2024-03-31T23:22:57.700 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:57.693+0000 7f9e12ffd640 1 -- 172.21.15.46:0/1365168552 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f9e24027c30 con 0x56142a2b26c0 2024-03-31T23:22:57.700 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:57.693+0000 7f9e3c580740 10 client.17070 fetch_fsmap learned FSMap version 36 2024-03-31T23:22:57.700 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:57.693+0000 7f9e3c580740 1 -- 172.21.15.46:0/1365168552 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x56142a332ba0 con 0x56142a2b26c0 2024-03-31T23:22:57.700 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:57.693+0000 7f9e12ffd640 1 -- 172.21.15.46:0/1365168552 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x56142a222e30 con 0x56142a2b26c0 2024-03-31T23:22:57.700 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:57.693+0000 7f9e12ffd640 1 -- 172.21.15.46:0/1365168552 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 36) ==== 8362+0+0 (secure 0 0 0) 0x7f9e2400f480 con 0x56142a2b26c0 2024-03-31T23:22:57.701 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:57.693+0000 7f9e3c580740 10 client.17070 fetch_fsmap finished waiting for FSMap version 36 2024-03-31T23:22:57.701 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:57.693+0000 7f9e3c580740 10 client.17070 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-31T23:22:57.701 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:57.693+0000 7f9e3c580740 20 client.17070 populate_metadata read hostname 'smithi046' 2024-03-31T23:22:57.701 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:57.693+0000 7f9e3c580740 1 --2- 172.21.15.46:0/1365168552 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x56142a333300 0x56142a335720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:22:57.701 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:57.693+0000 7f9e3c580740 4 client.17070 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-31T23:22:57.701 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:57.693+0000 7f9e3c580740 1 -- 172.21.15.46:0/1365168552 --> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] -- command(tid 0: {"prefix": "flush journal"}) -- 0x56142a21d180 con 0x56142a333300 2024-03-31T23:22:57.701 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:57.693+0000 7f9e327fc640 1 --2- 172.21.15.46:0/1365168552 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x56142a333300 0x56142a335720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:22:57.701 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:57.694+0000 7f9e12ffd640 10 client.17070 ms_handle_connect on v2:172.21.15.142:6838/2519143918 2024-03-31T23:22:57.701 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:57.694+0000 7f9e327fc640 1 --2- 172.21.15.46:0/1365168552 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x56142a333300 0x56142a335720 crc :-1 s=READY pgs=176 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1161c2a08cdab760 server_cookie=1f85657edf7fb5b2 in_seq=0 out_seq=0 2024-03-31T23:22:57.722 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:57.719+0000 7f9e12ffd640 1 -- 172.21.15.46:0/1365168552 <== mds.0 v2:172.21.15.142:6838/2519143918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x56142a21d180 con 0x56142a333300 2024-03-31T23:22:57.722 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:57.719+0000 7f9e12ffd640 10 client.17070 handle_command_reply: tid=0 2024-03-31T23:22:57.722 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:57.719+0000 7f9e3c580740 1 client.17070 shutdown 2024-03-31T23:22:57.722 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:57.720+0000 7f9e3c580740 1 -- 172.21.15.46:0/1365168552 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56142a2b26c0 msgr2=0x56142a2c39f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:22:57.722 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:57.720+0000 7f9e3c580740 1 --2- 172.21.15.46:0/1365168552 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56142a2b26c0 0x56142a2c39f0 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x56142a2ce9c0 tx=0x7f9e24004730 comp rx=0 tx=0).stop 2024-03-31T23:22:57.722 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:57.720+0000 7f9e3c580740 1 -- 172.21.15.46:0/1365168552 shutdown_connections 2024-03-31T23:22:57.722 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:57.720+0000 7f9e3c580740 1 --2- 172.21.15.46:0/1365168552 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x56142a333300 0x56142a335720 crc :-1 s=READY pgs=176 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:22:57.722 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:57.720+0000 7f9e3c580740 1 --2- 172.21.15.46:0/1365168552 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56142a2bbdb0 0x56142a2bfe00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:22:57.722 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:57.720+0000 7f9e3c580740 1 --2- 172.21.15.46:0/1365168552 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56142a2b2fe0 0x56142a2c3f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:22:57.722 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:57.720+0000 7f9e12ffd640 0 client.17070 ms_handle_reset on v2:172.21.15.142:6838/2519143918 2024-03-31T23:22:57.723 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:57.720+0000 7f9e3c580740 1 --2- 172.21.15.46:0/1365168552 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56142a2b26c0 0x56142a2c39f0 unknown :-1 s=CLOSED pgs=830 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:22:57.723 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:57.720+0000 7f9e3c580740 1 -- 172.21.15.46:0/1365168552 >> 172.21.15.46:0/1365168552 conn(0x56142a2acfe0 msgr2=0x56142a2abf80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:22:57.723 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:57.720+0000 7f9e3c580740 1 -- 172.21.15.46:0/1365168552 shutdown_connections 2024-03-31T23:22:57.723 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:57.720+0000 7f9e3c580740 1 -- 172.21.15.46:0/1365168552 wait complete. 2024-03-31T23:22:57.723 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:57.720+0000 7f9e3c580740 20 client.17070 trim_cache size 0 max 16384 2024-03-31T23:22:57.723 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout:{ 2024-03-31T23:22:57.724 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout: "message": "", 2024-03-31T23:22:57.724 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout: "return_code": 0 2024-03-31T23:22:57.724 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout:} 2024-03-31T23:22:57.725 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:57.720+0000 7f9e3999a640 1 -- 172.21.15.46:0/256367501 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f9e1c0a26b0 msgr2=0x7f9e1c0a4b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:22:57.725 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:57.720+0000 7f9e3999a640 1 --2- 172.21.15.46:0/256367501 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f9e1c0a26b0 0x7f9e1c0a4b70 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f9e24009f40 tx=0x7f9e24005f90 comp rx=0 tx=0).stop 2024-03-31T23:22:57.725 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:57.720+0000 7f9e3999a640 1 -- 172.21.15.46:0/256367501 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9e34107730 msgr2=0x7f9e3419c740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:22:57.725 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:57.720+0000 7f9e3999a640 1 --2- 172.21.15.46:0/256367501 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9e34107730 0x7f9e3419c740 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7f9e2c004af0 tx=0x7f9e2c035ce0 comp rx=0 tx=0).stop 2024-03-31T23:22:57.725 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:57.720+0000 7f9e3999a640 1 -- 172.21.15.46:0/256367501 shutdown_connections 2024-03-31T23:22:57.725 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:57.720+0000 7f9e3999a640 1 --2- 172.21.15.46:0/256367501 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f9e1c0a26b0 0x7f9e1c0a4b70 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:22:57.725 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:57.720+0000 7f9e3999a640 1 --2- 172.21.15.46:0/256367501 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9e34107730 0x7f9e3419c740 unknown :-1 s=CLOSED pgs=828 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:22:57.726 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:57.720+0000 7f9e3999a640 1 --2- 172.21.15.46:0/256367501 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9e341020e0 0x7f9e34198410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:22:57.726 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:57.721+0000 7f9e3999a640 1 --2- 172.21.15.46:0/256367501 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9e34101720 0x7f9e34197ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:22:57.726 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:57.721+0000 7f9e3999a640 1 -- 172.21.15.46:0/256367501 >> 172.21.15.46:0/256367501 conn(0x7f9e340fd230 msgr2=0x7f9e340fd830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:22:57.726 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:57.721+0000 7f9e3999a640 1 -- 172.21.15.46:0/256367501 shutdown_connections 2024-03-31T23:22:57.726 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:22:57.721+0000 7f9e3999a640 1 -- 172.21.15.46:0/256367501 wait complete. 2024-03-31T23:22:58.216 INFO:tasks.workunit.client.0.smithi046.stdout: 1 222361 1.87 MB/sec execute 484 sec latency 32.524 ms 2024-03-31T23:22:58.862 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:22:58 smithi046 ceph-mon[32498]: pgmap v350: 129 pgs: 129 active+clean; 128 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 237 KiB/s wr, 31 op/s 2024-03-31T23:22:58.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:22:58 smithi142 ceph-mon[35271]: pgmap v350: 129 pgs: 129 active+clean; 128 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 237 KiB/s wr, 31 op/s 2024-03-31T23:22:59.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:22:58 smithi162 ceph-mon[36237]: pgmap v350: 129 pgs: 129 active+clean; 128 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 237 KiB/s wr, 31 op/s 2024-03-31T23:22:59.216 INFO:tasks.workunit.client.0.smithi046.stdout: 1 222949 1.88 MB/sec execute 485 sec latency 16.138 ms 2024-03-31T23:22:59.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:22:59 smithi046 ceph-mon[32498]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-31T23:22:59.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:22:59 smithi142 ceph-mon[35271]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-31T23:23:00.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:22:59 smithi162 ceph-mon[36237]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-31T23:23:00.216 INFO:tasks.workunit.client.0.smithi046.stdout: 1 223328 1.88 MB/sec execute 486 sec latency 16.707 ms 2024-03-31T23:23:00.862 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:23:00 smithi046 ceph-mon[32498]: pgmap v351: 129 pgs: 129 active+clean; 125 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 314 KiB/s rd, 782 KiB/s wr, 90 op/s 2024-03-31T23:23:00.862 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:23:00 smithi046 ceph-mon[32498]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' 2024-03-31T23:23:00.862 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:23:00 smithi046 ceph-mon[32498]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' 2024-03-31T23:23:00.862 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:23:00 smithi046 ceph-mon[32498]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' 2024-03-31T23:23:00.862 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:23:00 smithi046 ceph-mon[32498]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' 2024-03-31T23:23:00.862 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:23:00 smithi046 ceph-mon[32498]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' 2024-03-31T23:23:00.862 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:23:00 smithi046 ceph-mon[32498]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' 2024-03-31T23:23:00.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:23:00 smithi142 ceph-mon[35271]: pgmap v351: 129 pgs: 129 active+clean; 125 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 314 KiB/s rd, 782 KiB/s wr, 90 op/s 2024-03-31T23:23:00.944 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:23:00 smithi142 ceph-mon[35271]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' 2024-03-31T23:23:00.944 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:23:00 smithi142 ceph-mon[35271]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' 2024-03-31T23:23:00.944 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:23:00 smithi142 ceph-mon[35271]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' 2024-03-31T23:23:00.944 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:23:00 smithi142 ceph-mon[35271]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' 2024-03-31T23:23:00.944 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:23:00 smithi142 ceph-mon[35271]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' 2024-03-31T23:23:00.944 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:23:00 smithi142 ceph-mon[35271]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' 2024-03-31T23:23:01.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:23:00 smithi162 ceph-mon[36237]: pgmap v351: 129 pgs: 129 active+clean; 125 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 314 KiB/s rd, 782 KiB/s wr, 90 op/s 2024-03-31T23:23:01.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:23:00 smithi162 ceph-mon[36237]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' 2024-03-31T23:23:01.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:23:00 smithi162 ceph-mon[36237]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' 2024-03-31T23:23:01.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:23:00 smithi162 ceph-mon[36237]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' 2024-03-31T23:23:01.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:23:00 smithi162 ceph-mon[36237]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' 2024-03-31T23:23:01.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:23:00 smithi162 ceph-mon[36237]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' 2024-03-31T23:23:01.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:23:00 smithi162 ceph-mon[36237]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' 2024-03-31T23:23:01.218 INFO:tasks.workunit.client.0.smithi046.stdout: 1 223582 1.88 MB/sec execute 487 sec latency 566.553 ms 2024-03-31T23:23:01.862 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:23:01 smithi046 ceph-mon[32498]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:23:01.862 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:23:01 smithi046 ceph-mon[32498]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-31T23:23:01.862 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:23:01 smithi046 ceph-mon[32498]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' 2024-03-31T23:23:01.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:23:01 smithi142 ceph-mon[35271]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:23:01.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:23:01 smithi142 ceph-mon[35271]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-31T23:23:01.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:23:01 smithi142 ceph-mon[35271]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' 2024-03-31T23:23:02.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:23:01 smithi162 ceph-mon[36237]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:23:02.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:23:01 smithi162 ceph-mon[36237]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-31T23:23:02.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:23:01 smithi162 ceph-mon[36237]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' 2024-03-31T23:23:02.216 INFO:tasks.workunit.client.0.smithi046.stdout: 1 223582 1.88 MB/sec execute 488 sec latency 1566.663 ms 2024-03-31T23:23:02.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:23:02 smithi046 ceph-mon[32498]: pgmap v352: 129 pgs: 129 active+clean; 125 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 268 KiB/s rd, 725 KiB/s wr, 75 op/s 2024-03-31T23:23:02.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:23:02 smithi142 ceph-mon[35271]: pgmap v352: 129 pgs: 129 active+clean; 125 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 268 KiB/s rd, 725 KiB/s wr, 75 op/s 2024-03-31T23:23:03.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:23:02 smithi162 ceph-mon[36237]: pgmap v352: 129 pgs: 129 active+clean; 125 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 268 KiB/s rd, 725 KiB/s wr, 75 op/s 2024-03-31T23:23:03.216 INFO:tasks.workunit.client.0.smithi046.stdout: 1 223649 1.87 MB/sec execute 489 sec latency 2096.090 ms 2024-03-31T23:23:04.217 INFO:tasks.workunit.client.0.smithi046.stdout: 1 223649 1.87 MB/sec execute 490 sec latency 1207.970 ms 2024-03-31T23:23:04.862 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:23:04 smithi046 ceph-mon[32498]: pgmap v353: 129 pgs: 129 active+clean; 125 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 268 KiB/s rd, 726 KiB/s wr, 75 op/s 2024-03-31T23:23:04.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:23:04 smithi142 ceph-mon[35271]: pgmap v353: 129 pgs: 129 active+clean; 125 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 268 KiB/s rd, 726 KiB/s wr, 75 op/s 2024-03-31T23:23:05.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:23:04 smithi162 ceph-mon[36237]: pgmap v353: 129 pgs: 129 active+clean; 125 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 268 KiB/s rd, 726 KiB/s wr, 75 op/s 2024-03-31T23:23:05.217 INFO:tasks.workunit.client.0.smithi046.stdout: 1 223649 1.87 MB/sec execute 491 sec latency 2208.059 ms 2024-03-31T23:23:06.217 INFO:tasks.workunit.client.0.smithi046.stdout: 1 223649 1.86 MB/sec execute 492 sec latency 3208.177 ms 2024-03-31T23:23:06.862 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:23:06 smithi046 ceph-mon[32498]: pgmap v354: 129 pgs: 129 active+clean; 127 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 394 KiB/s rd, 1.1 MiB/s wr, 118 op/s 2024-03-31T23:23:06.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:23:06 smithi142 ceph-mon[35271]: pgmap v354: 129 pgs: 129 active+clean; 127 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 394 KiB/s rd, 1.1 MiB/s wr, 118 op/s 2024-03-31T23:23:07.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:23:06 smithi162 ceph-mon[36237]: pgmap v354: 129 pgs: 129 active+clean; 127 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 394 KiB/s rd, 1.1 MiB/s wr, 118 op/s 2024-03-31T23:23:07.217 INFO:tasks.workunit.client.0.smithi046.stdout: 1 223649 1.86 MB/sec execute 493 sec latency 4208.310 ms 2024-03-31T23:23:08.217 INFO:tasks.workunit.client.0.smithi046.stdout: 1 223784 1.86 MB/sec execute 494 sec latency 4736.911 ms 2024-03-31T23:23:08.863 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:23:08 smithi046 ceph-mon[32498]: pgmap v355: 129 pgs: 129 active+clean; 127 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 369 KiB/s rd, 909 KiB/s wr, 101 op/s 2024-03-31T23:23:08.863 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:23:08 smithi046 ceph-mon[32498]: osdmap e107: 12 total, 12 up, 12 in 2024-03-31T23:23:08.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:23:08 smithi142 ceph-mon[35271]: pgmap v355: 129 pgs: 129 active+clean; 127 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 369 KiB/s rd, 909 KiB/s wr, 101 op/s 2024-03-31T23:23:08.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:23:08 smithi142 ceph-mon[35271]: osdmap e107: 12 total, 12 up, 12 in 2024-03-31T23:23:09.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:23:08 smithi162 ceph-mon[36237]: pgmap v355: 129 pgs: 129 active+clean; 127 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 369 KiB/s rd, 909 KiB/s wr, 101 op/s 2024-03-31T23:23:09.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:23:08 smithi162 ceph-mon[36237]: osdmap e107: 12 total, 12 up, 12 in 2024-03-31T23:23:09.217 INFO:tasks.workunit.client.0.smithi046.stdout: 1 224047 1.85 MB/sec execute 495 sec latency 15.234 ms 2024-03-31T23:23:10.217 INFO:tasks.workunit.client.0.smithi046.stdout: 1 224437 1.85 MB/sec execute 496 sec latency 16.270 ms 2024-03-31T23:23:10.818 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:10.814+0000 7f26e0f6c640 1 -- 172.21.15.46:0/2653414078 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f26dc1008a0 msgr2=0x7f26dc100d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:23:10.818 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:10.814+0000 7f26e0f6c640 1 --2- 172.21.15.46:0/2653414078 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f26dc1008a0 0x7f26dc100d20 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7f26c4009900 tx=0x7f26c402bc30 comp rx=0 tx=0).stop 2024-03-31T23:23:10.818 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:10.814+0000 7f26e0f6c640 1 -- 172.21.15.46:0/2653414078 shutdown_connections 2024-03-31T23:23:10.818 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:10.814+0000 7f26e0f6c640 1 --2- 172.21.15.46:0/2653414078 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f26dc101260 0x7f26dc10d480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:23:10.818 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:10.814+0000 7f26e0f6c640 1 --2- 172.21.15.46:0/2653414078 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f26dc1008a0 0x7f26dc100d20 unknown :-1 s=CLOSED pgs=831 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:23:10.818 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:10.814+0000 7f26e0f6c640 1 --2- 172.21.15.46:0/2653414078 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f26dc1055f0 0x7f26dc1059d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:23:10.818 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:10.814+0000 7f26e0f6c640 1 -- 172.21.15.46:0/2653414078 >> 172.21.15.46:0/2653414078 conn(0x7f26dc0774a0 msgr2=0x7f26dc0778b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:23:10.818 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:10.814+0000 7f26e0f6c640 1 -- 172.21.15.46:0/2653414078 shutdown_connections 2024-03-31T23:23:10.818 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:10.814+0000 7f26e0f6c640 1 -- 172.21.15.46:0/2653414078 wait complete. 2024-03-31T23:23:10.818 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:10.815+0000 7f26e0f6c640 1 Processor -- start 2024-03-31T23:23:10.819 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:10.815+0000 7f26e0f6c640 1 -- start start 2024-03-31T23:23:10.819 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:10.815+0000 7f26e0f6c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f26dc1008a0 0x7f26dc19a130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:23:10.819 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:10.815+0000 7f26e0f6c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f26dc101260 0x7f26dc19a670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:23:10.819 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:10.815+0000 7f26e0f6c640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f26dc1055f0 0x7f26dc19e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:23:10.819 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:10.815+0000 7f26e0f6c640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f26dc110a10 con 0x7f26dc101260 2024-03-31T23:23:10.819 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:10.815+0000 7f26e0f6c640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f26dc110890 con 0x7f26dc1055f0 2024-03-31T23:23:10.819 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:10.815+0000 7f26e0f6c640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f26dc110b90 con 0x7f26dc1008a0 2024-03-31T23:23:10.819 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:10.815+0000 7f26d3fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f26dc101260 0x7f26dc19a670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:23:10.819 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:10.815+0000 7f26d3fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f26dc101260 0x7f26dc19a670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:43848/0 (socket says 172.21.15.46:43848) 2024-03-31T23:23:10.819 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:10.815+0000 7f26d3fff640 1 -- 172.21.15.46:0/3844844193 learned_addr learned my addr 172.21.15.46:0/3844844193 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:23:10.819 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:10.815+0000 7f26da59c640 1 --2- 172.21.15.46:0/3844844193 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f26dc1008a0 0x7f26dc19a130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:23:10.819 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:10.815+0000 7f26dad9d640 1 --2- 172.21.15.46:0/3844844193 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f26dc1055f0 0x7f26dc19e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:23:10.819 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:10.816+0000 7f26d3fff640 1 -- 172.21.15.46:0/3844844193 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f26dc1008a0 msgr2=0x7f26dc19a130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:23:10.819 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:10.816+0000 7f26d3fff640 1 --2- 172.21.15.46:0/3844844193 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f26dc1008a0 0x7f26dc19a130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:23:10.820 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:10.816+0000 7f26d3fff640 1 -- 172.21.15.46:0/3844844193 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f26dc1055f0 msgr2=0x7f26dc19e9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:23:10.820 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:10.816+0000 7f26d3fff640 1 --2- 172.21.15.46:0/3844844193 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f26dc1055f0 0x7f26dc19e9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:23:10.820 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:10.816+0000 7f26d3fff640 1 -- 172.21.15.46:0/3844844193 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f26dc19f120 con 0x7f26dc101260 2024-03-31T23:23:10.820 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:10.816+0000 7f26d3fff640 1 --2- 172.21.15.46:0/3844844193 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f26dc101260 0x7f26dc19a670 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7f26c4000c00 tx=0x7f26c4005dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:23:10.820 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:10.816+0000 7f26d37fe640 1 -- 172.21.15.46:0/3844844193 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26c4043020 con 0x7f26dc101260 2024-03-31T23:23:10.820 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:10.816+0000 7f26d37fe640 1 -- 172.21.15.46:0/3844844193 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f26c402cb50 con 0x7f26dc101260 2024-03-31T23:23:10.820 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:10.816+0000 7f26e0f6c640 1 -- 172.21.15.46:0/3844844193 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f26dc19f3b0 con 0x7f26dc101260 2024-03-31T23:23:10.820 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:10.816+0000 7f26d37fe640 1 -- 172.21.15.46:0/3844844193 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26c4003a50 con 0x7f26dc101260 2024-03-31T23:23:10.820 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:10.816+0000 7f26e0f6c640 1 -- 172.21.15.46:0/3844844193 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f26dc1a6ca0 con 0x7f26dc101260 2024-03-31T23:23:10.827 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:10.819+0000 7f26e3c2b740 1 Processor -- start 2024-03-31T23:23:10.827 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:10.819+0000 7f26e3c2b740 1 -- start start 2024-03-31T23:23:10.827 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:10.819+0000 7f26d37fe640 1 -- 172.21.15.46:0/3844844193 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 25) ==== 137451+0+0 (secure 0 0 0) 0x7f26c4046020 con 0x7f26dc101260 2024-03-31T23:23:10.827 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:10.819+0000 7f26e3c2b740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5654bcc356c0 0x5654bcc35aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:23:10.827 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:10.819+0000 7f26e3c2b740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5654bcc35fe0 0x5654bcc3e870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:23:10.827 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:10.819+0000 7f26e3c2b740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5654bcc3edb0 0x5654bcc411a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:23:10.827 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:10.819+0000 7f26e3c2b740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x5654bcb988f0 con 0x5654bcc356c0 2024-03-31T23:23:10.827 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:10.819+0000 7f26e3c2b740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x5654bcb418d0 con 0x5654bcc35fe0 2024-03-31T23:23:10.827 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:10.819+0000 7f26e3c2b740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5654bcb94990 con 0x5654bcc3edb0 2024-03-31T23:23:10.828 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:10.819+0000 7f26dad9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5654bcc356c0 0x5654bcc35aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:23:10.828 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:10.819+0000 7f26dad9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5654bcc356c0 0x5654bcc35aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:43862/0 (socket says 172.21.15.46:43862) 2024-03-31T23:23:10.828 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:10.819+0000 7f26dad9d640 1 -- 172.21.15.46:0/1637873290 learned_addr learned my addr 172.21.15.46:0/1637873290 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:23:10.828 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:10.819+0000 7f26da59c640 1 --2- 172.21.15.46:0/1637873290 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5654bcc35fe0 0x5654bcc3e870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:23:10.828 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:10.819+0000 7f26d3fff640 1 --2- 172.21.15.46:0/1637873290 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5654bcc3edb0 0x5654bcc411a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:23:10.828 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:10.820+0000 7f26dad9d640 1 -- 172.21.15.46:0/1637873290 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5654bcc3edb0 msgr2=0x5654bcc411a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:23:10.828 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:10.820+0000 7f26dad9d640 1 --2- 172.21.15.46:0/1637873290 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5654bcc3edb0 0x5654bcc411a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:23:10.828 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:10.820+0000 7f26dad9d640 1 -- 172.21.15.46:0/1637873290 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5654bcc35fe0 msgr2=0x5654bcc3e870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:23:10.828 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:10.820+0000 7f26dad9d640 1 --2- 172.21.15.46:0/1637873290 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5654bcc35fe0 0x5654bcc3e870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:23:10.828 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:10.820+0000 7f26dad9d640 1 -- 172.21.15.46:0/1637873290 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5654bc8ad210 con 0x5654bcc356c0 2024-03-31T23:23:10.828 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:10.820+0000 7f26da59c640 1 --2- 172.21.15.46:0/1637873290 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5654bcc35fe0 0x5654bcc3e870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:23:10.828 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:10.820+0000 7f26d37fe640 1 --2- 172.21.15.46:0/3844844193 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f26c80a26b0 0x7f26c80a4b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:23:10.828 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:10.820+0000 7f26dad9d640 1 --2- 172.21.15.46:0/1637873290 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5654bcc356c0 0x5654bcc35aa0 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7f26d4009fc0 tx=0x7f26d40118d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=27287e438a51b57a server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:23:10.829 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:10.820+0000 7f26da59c640 1 --2- 172.21.15.46:0/3844844193 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f26c80a26b0 0x7f26c80a4b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:23:10.829 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:10.820+0000 7f26d37fe640 1 -- 172.21.15.46:0/3844844193 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 9554+0+0 (secure 0 0 0) 0x7f26c40f0f70 con 0x7f26dc101260 2024-03-31T23:23:10.829 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:10.820+0000 7f26d0ff9640 1 -- 172.21.15.46:0/1637873290 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26d4004d20 con 0x5654bcc356c0 2024-03-31T23:23:10.829 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:10.820+0000 7f26d0ff9640 1 -- 172.21.15.46:0/1637873290 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f26d4004050 con 0x5654bcc356c0 2024-03-31T23:23:10.829 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:10.820+0000 7f26d0ff9640 1 -- 172.21.15.46:0/1637873290 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26d4005280 con 0x5654bcc356c0 2024-03-31T23:23:10.829 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:10.821+0000 7f26da59c640 1 --2- 172.21.15.46:0/3844844193 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f26c80a26b0 0x7f26c80a4b70 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x5654bcc35170 tx=0x7f26cc005f90 comp rx=0 tx=0).ready entity=mgr.15654 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:23:10.829 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:10.821+0000 7f26e3c2b740 1 -- 172.21.15.46:0/1637873290 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5654bcc356c0 msgr2=0x5654bcc35aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:23:10.829 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:10.821+0000 7f26e3c2b740 1 --2- 172.21.15.46:0/1637873290 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5654bcc356c0 0x5654bcc35aa0 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7f26d4009fc0 tx=0x7f26d40118d0 comp rx=0 tx=0).stop 2024-03-31T23:23:10.829 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:10.821+0000 7f26e3c2b740 1 -- 172.21.15.46:0/1637873290 shutdown_connections 2024-03-31T23:23:10.829 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:10.821+0000 7f26e3c2b740 1 --2- 172.21.15.46:0/1637873290 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5654bcc3edb0 0x5654bcc411a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:23:10.829 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:10.821+0000 7f26e3c2b740 1 --2- 172.21.15.46:0/1637873290 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5654bcc35fe0 0x5654bcc3e870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:23:10.829 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:10.821+0000 7f26e3c2b740 1 --2- 172.21.15.46:0/1637873290 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5654bcc356c0 0x5654bcc35aa0 unknown :-1 s=CLOSED pgs=833 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:23:10.829 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:10.821+0000 7f26e3c2b740 1 -- 172.21.15.46:0/1637873290 >> 172.21.15.46:0/1637873290 conn(0x5654bcc2ffe0 msgr2=0x5654bcc303c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:23:10.829 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:10.821+0000 7f26e3c2b740 1 -- 172.21.15.46:0/1637873290 shutdown_connections 2024-03-31T23:23:10.830 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:10.821+0000 7f26e3c2b740 1 -- 172.21.15.46:0/1637873290 wait complete. 2024-03-31T23:23:10.830 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:10.821+0000 7f26e3c2b740 1 -- start start 2024-03-31T23:23:10.830 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:10.821+0000 7f26e3c2b740 1 Processor -- start 2024-03-31T23:23:10.830 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:10.822+0000 7f26e3c2b740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5654bcc356c0 0x5654bcc3bda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:23:10.830 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:10.822+0000 7f26e3c2b740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5654bcc35fe0 0x5654bcc3c2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:23:10.830 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:10.822+0000 7f26e3c2b740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5654bcc3edb0 0x5654bcc3c820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:23:10.830 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:10.822+0000 7f26e3c2b740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x5654bcc419e0 con 0x5654bcc3edb0 2024-03-31T23:23:10.830 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:10.822+0000 7f26e3c2b740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x5654bcb36f80 con 0x5654bcc35fe0 2024-03-31T23:23:10.830 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:10.822+0000 7f26e3c2b740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5654bcc41b60 con 0x5654bcc356c0 2024-03-31T23:23:10.830 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:10.822+0000 7f26dad9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5654bcc3edb0 0x5654bcc3c820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:23:10.830 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:10.822+0000 7f26e3c2b740 10 client.0 _ll_register_callbacks cb 0x5654bcb8f290 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T23:23:10.830 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:10.822+0000 7f26dad9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5654bcc3edb0 0x5654bcc3c820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:43876/0 (socket says 172.21.15.46:43876) 2024-03-31T23:23:10.830 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:10.822+0000 7f26dad9d640 1 -- 172.21.15.46:0/1043117342 learned_addr learned my addr 172.21.15.46:0/1043117342 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:23:10.831 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:10.822+0000 7f26da59c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5654bcc356c0 0x5654bcc3bda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:23:10.831 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:10.822+0000 7f26d3fff640 1 --2- 172.21.15.46:0/1043117342 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5654bcc35fe0 0x5654bcc3c2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:23:10.831 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:10.822+0000 7f26dad9d640 1 -- 172.21.15.46:0/1043117342 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5654bcc356c0 msgr2=0x5654bcc3bda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:23:10.831 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:10.822+0000 7f26dad9d640 1 --2- 172.21.15.46:0/1043117342 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5654bcc356c0 0x5654bcc3bda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:23:10.831 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:10.822+0000 7f26dad9d640 1 -- 172.21.15.46:0/1043117342 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5654bcc35fe0 msgr2=0x5654bcc3c2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:23:10.831 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:10.822+0000 7f26dad9d640 1 --2- 172.21.15.46:0/1043117342 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5654bcc35fe0 0x5654bcc3c2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:23:10.831 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:10.822+0000 7f26dad9d640 1 -- 172.21.15.46:0/1043117342 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5654bc979620 con 0x5654bcc3edb0 2024-03-31T23:23:10.831 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:10.822+0000 7f26dad9d640 1 --2- 172.21.15.46:0/1043117342 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5654bcc3edb0 0x5654bcc3c820 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7f26d401b040 tx=0x7f26d4004870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f56eef66b698824 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:23:10.831 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:10.822+0000 7f26c1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-31T23:23:10.831 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:10.822+0000 7f26c1ffb640 1 -- 172.21.15.46:0/1043117342 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26d4028020 con 0x5654bcc3edb0 2024-03-31T23:23:10.831 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:10.822+0000 7f26c1ffb640 1 -- 172.21.15.46:0/1043117342 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(108..108 src has 1..108) ==== 9554+0+0 (secure 0 0 0) 0x7f26d4004510 con 0x5654bcc3edb0 2024-03-31T23:23:10.831 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:10.822+0000 7f26e3c2b740 1 -- 172.21.15.46:0/1043117342 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f26d4004050 con 0x5654bcc3edb0 2024-03-31T23:23:10.831 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:10.823+0000 7f26c1ffb640 1 -- 172.21.15.46:0/1043117342 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f26d4033050 con 0x5654bcc3edb0 2024-03-31T23:23:10.831 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:10.823+0000 7f26e3c2b740 10 client.17082 fetch_fsmap learned FSMap version 36 2024-03-31T23:23:10.832 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:10.823+0000 7f26e3c2b740 1 -- 172.21.15.46:0/1043117342 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x5654bcc4d860 con 0x5654bcc3edb0 2024-03-31T23:23:10.832 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:10.823+0000 7f26c1ffb640 1 -- 172.21.15.46:0/1043117342 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f26d4004050 con 0x5654bcc3edb0 2024-03-31T23:23:10.832 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:10.823+0000 7f26c1ffb640 1 -- 172.21.15.46:0/1043117342 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 36) ==== 8362+0+0 (secure 0 0 0) 0x7f26d4005750 con 0x5654bcc3edb0 2024-03-31T23:23:10.832 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:10.823+0000 7f26e3c2b740 10 client.17082 fetch_fsmap finished waiting for FSMap version 36 2024-03-31T23:23:10.832 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:10.823+0000 7f26e3c2b740 10 client.17082 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-31T23:23:10.832 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:10.823+0000 7f26e3c2b740 20 client.17082 populate_metadata read hostname 'smithi046' 2024-03-31T23:23:10.832 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:10.823+0000 7f26e3c2b740 1 --2- 172.21.15.46:0/1043117342 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x5654bcc4ecc0 0x5654bcc510b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:23:10.832 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:10.823+0000 7f26e3c2b740 4 client.17082 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-31T23:23:10.832 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:10.823+0000 7f26e3c2b740 1 -- 172.21.15.46:0/1043117342 --> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5654bcbac120 con 0x5654bcc4ecc0 2024-03-31T23:23:10.832 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:10.823+0000 7f26da59c640 1 --2- 172.21.15.46:0/1043117342 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x5654bcc4ecc0 0x5654bcc510b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:23:10.832 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:10.824+0000 7f26da59c640 1 --2- 172.21.15.46:0/1043117342 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x5654bcc4ecc0 0x5654bcc510b0 crc :-1 s=READY pgs=177 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=33a6a26a33482a91 server_cookie=b97e4b66bf1a4c3c in_seq=0 out_seq=0 2024-03-31T23:23:10.832 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:10.824+0000 7f26c1ffb640 10 client.17082 ms_handle_connect on v2:172.21.15.142:6838/2519143918 2024-03-31T23:23:10.832 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:10.825+0000 7f26c1ffb640 1 -- 172.21.15.46:0/1043117342 <== mds.0 v2:172.21.15.142:6838/2519143918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5654bcbac120 con 0x5654bcc4ecc0 2024-03-31T23:23:10.832 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:10.825+0000 7f26c1ffb640 10 client.17082 handle_command_reply: tid=0 2024-03-31T23:23:10.833 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:10.825+0000 7f26e3c2b740 1 client.17082 shutdown 2024-03-31T23:23:10.833 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:10.825+0000 7f26e3c2b740 1 -- 172.21.15.46:0/1043117342 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5654bcc3edb0 msgr2=0x5654bcc3c820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:23:10.833 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:10.825+0000 7f26e3c2b740 1 --2- 172.21.15.46:0/1043117342 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5654bcc3edb0 0x5654bcc3c820 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7f26d401b040 tx=0x7f26d4004870 comp rx=0 tx=0).stop 2024-03-31T23:23:10.833 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:10.825+0000 7f26e3c2b740 1 -- 172.21.15.46:0/1043117342 shutdown_connections 2024-03-31T23:23:10.833 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:10.825+0000 7f26e3c2b740 1 --2- 172.21.15.46:0/1043117342 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x5654bcc4ecc0 0x5654bcc510b0 crc :-1 s=READY pgs=177 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:23:10.833 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:10.825+0000 7f26e3c2b740 1 --2- 172.21.15.46:0/1043117342 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5654bcc3edb0 0x5654bcc3c820 unknown :-1 s=CLOSED pgs=834 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:23:10.833 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:10.825+0000 7f26e3c2b740 1 --2- 172.21.15.46:0/1043117342 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5654bcc35fe0 0x5654bcc3c2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:23:10.833 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:10.825+0000 7f26e3c2b740 1 --2- 172.21.15.46:0/1043117342 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5654bcc356c0 0x5654bcc3bda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:23:10.833 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:10.825+0000 7f26c1ffb640 0 client.17082 ms_handle_reset on v2:172.21.15.142:6838/2519143918 2024-03-31T23:23:10.833 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:10.825+0000 7f26e3c2b740 1 -- 172.21.15.46:0/1043117342 >> 172.21.15.46:0/1043117342 conn(0x5654bcc2ffe0 msgr2=0x5654bcc31d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:23:10.833 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:10.825+0000 7f26e3c2b740 1 -- 172.21.15.46:0/1043117342 shutdown_connections 2024-03-31T23:23:10.833 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:10.825+0000 7f26e3c2b740 1 -- 172.21.15.46:0/1043117342 wait complete. 2024-03-31T23:23:10.833 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:10.825+0000 7f26e3c2b740 20 client.17082 trim_cache size 0 max 16384 2024-03-31T23:23:10.847 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:10.842+0000 7f26e3c2b740 1 Processor -- start 2024-03-31T23:23:10.847 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:10.842+0000 7f26e3c2b740 1 -- start start 2024-03-31T23:23:10.847 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:10.842+0000 7f26e3c2b740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5654bcc356c0 0x5654bcc4a780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:23:10.847 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:10.842+0000 7f26e3c2b740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5654bcc35fe0 0x5654bcc3c610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:23:10.848 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:10.842+0000 7f26e3c2b740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5654bcc3edb0 0x5654bcc2ee00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:23:10.848 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:10.842+0000 7f26e3c2b740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f26dc1b3640 con 0x5654bcc356c0 2024-03-31T23:23:10.848 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:10.842+0000 7f26e3c2b740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f26dc1b0eb0 con 0x5654bcc3edb0 2024-03-31T23:23:10.848 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:10.842+0000 7f26e3c2b740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f26dc1aed20 con 0x5654bcc35fe0 2024-03-31T23:23:10.848 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:10.842+0000 7f26da59c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5654bcc356c0 0x5654bcc4a780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:23:10.848 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:10.842+0000 7f26da59c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5654bcc356c0 0x5654bcc4a780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:43892/0 (socket says 172.21.15.46:43892) 2024-03-31T23:23:10.848 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:10.842+0000 7f26da59c640 1 -- 172.21.15.46:0/3282494142 learned_addr learned my addr 172.21.15.46:0/3282494142 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:23:10.848 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:10.842+0000 7f26d3fff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5654bcc35fe0 0x5654bcc3c610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:23:10.848 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:10.842+0000 7f26dad9d640 1 --2- 172.21.15.46:0/3282494142 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5654bcc3edb0 0x5654bcc2ee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:23:10.848 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:10.842+0000 7f26d3fff640 1 -- 172.21.15.46:0/3282494142 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5654bcc3edb0 msgr2=0x5654bcc2ee00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:23:10.848 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:10.842+0000 7f26d3fff640 1 --2- 172.21.15.46:0/3282494142 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5654bcc3edb0 0x5654bcc2ee00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:23:10.848 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:10.842+0000 7f26d3fff640 1 -- 172.21.15.46:0/3282494142 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5654bcc356c0 msgr2=0x5654bcc4a780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:23:10.848 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:10.842+0000 7f26d3fff640 1 --2- 172.21.15.46:0/3282494142 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5654bcc356c0 0x5654bcc4a780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:23:10.848 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:10.842+0000 7f26d3fff640 1 -- 172.21.15.46:0/3282494142 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5654bcc35e10 con 0x5654bcc35fe0 2024-03-31T23:23:10.849 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:10.842+0000 7f26da59c640 1 --2- 172.21.15.46:0/3282494142 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5654bcc356c0 0x5654bcc4a780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:23:10.849 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:10.843+0000 7f26d3fff640 1 --2- 172.21.15.46:0/3282494142 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5654bcc35fe0 0x5654bcc3c610 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f26c40ee3c0 tx=0x7f26c40b9040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8c8679faef4d0f3e server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:23:10.849 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:10.843+0000 7f26c3fff640 1 -- 172.21.15.46:0/3282494142 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26c40b9690 con 0x5654bcc35fe0 2024-03-31T23:23:10.849 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:10.843+0000 7f26c3fff640 1 -- 172.21.15.46:0/3282494142 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f26c40bdaf0 con 0x5654bcc35fe0 2024-03-31T23:23:10.849 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:10.843+0000 7f26c3fff640 1 -- 172.21.15.46:0/3282494142 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26c40bcd50 con 0x5654bcc35fe0 2024-03-31T23:23:10.849 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:10.843+0000 7f26e3c2b740 1 -- 172.21.15.46:0/3282494142 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5654bcc35fe0 msgr2=0x5654bcc3c610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:23:10.849 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:10.843+0000 7f26e3c2b740 1 --2- 172.21.15.46:0/3282494142 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5654bcc35fe0 0x5654bcc3c610 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f26c40ee3c0 tx=0x7f26c40b9040 comp rx=0 tx=0).stop 2024-03-31T23:23:10.849 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:10.843+0000 7f26e3c2b740 1 -- 172.21.15.46:0/3282494142 shutdown_connections 2024-03-31T23:23:10.849 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:10.843+0000 7f26e3c2b740 1 --2- 172.21.15.46:0/3282494142 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5654bcc3edb0 0x5654bcc2ee00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:23:10.849 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:10.843+0000 7f26e3c2b740 1 --2- 172.21.15.46:0/3282494142 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5654bcc35fe0 0x5654bcc3c610 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:23:10.849 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:10.843+0000 7f26e3c2b740 1 --2- 172.21.15.46:0/3282494142 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5654bcc356c0 0x5654bcc4a780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:23:10.849 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:10.843+0000 7f26e3c2b740 1 -- 172.21.15.46:0/3282494142 >> 172.21.15.46:0/3282494142 conn(0x5654bcc2ffe0 msgr2=0x5654bcc2e6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:23:10.849 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:10.844+0000 7f26e3c2b740 1 -- 172.21.15.46:0/3282494142 shutdown_connections 2024-03-31T23:23:10.849 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:10.844+0000 7f26e3c2b740 1 -- 172.21.15.46:0/3282494142 wait complete. 2024-03-31T23:23:10.850 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:10.844+0000 7f26e3c2b740 1 -- start start 2024-03-31T23:23:10.850 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:10.844+0000 7f26e3c2b740 1 Processor -- start 2024-03-31T23:23:10.850 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:10.844+0000 7f26e3c2b740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5654bcc356c0 0x5654bcc46cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:23:10.850 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:10.844+0000 7f26e3c2b740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5654bcc35fe0 0x5654bcc42180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:23:10.850 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:10.844+0000 7f26e3c2b740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5654bcc3edb0 0x5654bcc426c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:23:10.850 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:10.844+0000 7f26e3c2b740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f26dc1af370 con 0x5654bcc356c0 2024-03-31T23:23:10.850 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:10.844+0000 7f26e3c2b740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f26dc1b07e0 con 0x5654bcc35fe0 2024-03-31T23:23:10.850 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:10.844+0000 7f26e3c2b740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f26dc1afa40 con 0x5654bcc3edb0 2024-03-31T23:23:10.850 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:10.844+0000 7f26e3c2b740 10 client.0 _ll_register_callbacks cb 0x7f26cc0097a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T23:23:10.850 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:10.844+0000 7f26da59c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5654bcc356c0 0x5654bcc46cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:23:10.850 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:10.844+0000 7f26da59c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5654bcc356c0 0x5654bcc46cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:43898/0 (socket says 172.21.15.46:43898) 2024-03-31T23:23:10.850 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:10.844+0000 7f26da59c640 1 -- 172.21.15.46:0/3606771792 learned_addr learned my addr 172.21.15.46:0/3606771792 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:23:10.850 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:10.844+0000 7f26dad9d640 1 --2- 172.21.15.46:0/3606771792 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5654bcc3edb0 0x5654bcc426c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:23:10.850 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:10.844+0000 7f26d3fff640 1 --2- 172.21.15.46:0/3606771792 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5654bcc35fe0 0x5654bcc42180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:23:10.851 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:10.845+0000 7f26da59c640 1 -- 172.21.15.46:0/3606771792 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5654bcc3edb0 msgr2=0x5654bcc426c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:23:10.851 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:10.845+0000 7f26da59c640 1 --2- 172.21.15.46:0/3606771792 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5654bcc3edb0 0x5654bcc426c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:23:10.851 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:10.845+0000 7f26da59c640 1 -- 172.21.15.46:0/3606771792 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5654bcc35fe0 msgr2=0x5654bcc42180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:23:10.851 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:10.845+0000 7f26da59c640 1 --2- 172.21.15.46:0/3606771792 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5654bcc35fe0 0x5654bcc42180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:23:10.851 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:10.845+0000 7f26da59c640 1 -- 172.21.15.46:0/3606771792 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5654bcc42c00 con 0x5654bcc356c0 2024-03-31T23:23:10.851 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:10.845+0000 7f26dad9d640 1 --2- 172.21.15.46:0/3606771792 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5654bcc3edb0 0x5654bcc426c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:23:10.851 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:10.845+0000 7f26da59c640 1 --2- 172.21.15.46:0/3606771792 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5654bcc356c0 0x5654bcc46cf0 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7f26cc0045a0 tx=0x7f26cc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=888ed529105ff078 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:23:10.851 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:10.845+0000 7f26c1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-31T23:23:10.851 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:10.845+0000 7f26c1ffb640 1 -- 172.21.15.46:0/3606771792 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26cc02bd10 con 0x5654bcc356c0 2024-03-31T23:23:10.851 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:10.845+0000 7f26c1ffb640 1 -- 172.21.15.46:0/3606771792 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(108..108 src has 1..108) ==== 9554+0+0 (secure 0 0 0) 0x7f26cc005020 con 0x5654bcc356c0 2024-03-31T23:23:10.851 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:10.845+0000 7f26e3c2b740 1 -- 172.21.15.46:0/3606771792 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f26c40bdaf0 con 0x5654bcc356c0 2024-03-31T23:23:10.851 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:10.845+0000 7f26c1ffb640 1 -- 172.21.15.46:0/3606771792 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f26cc005260 con 0x5654bcc356c0 2024-03-31T23:23:10.851 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:10.845+0000 7f26e3c2b740 10 client.17088 fetch_fsmap learned FSMap version 36 2024-03-31T23:23:10.851 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:10.845+0000 7f26e3c2b740 1 -- 172.21.15.46:0/3606771792 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x5654bccb4b90 con 0x5654bcc356c0 2024-03-31T23:23:10.852 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:10.845+0000 7f26c1ffb640 1 -- 172.21.15.46:0/3606771792 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f26c40bdaf0 con 0x5654bcc356c0 2024-03-31T23:23:10.852 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:10.845+0000 7f26c1ffb640 1 -- 172.21.15.46:0/3606771792 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 36) ==== 8362+0+0 (secure 0 0 0) 0x7f26cc0035f0 con 0x5654bcc356c0 2024-03-31T23:23:10.852 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:10.846+0000 7f26e3c2b740 10 client.17088 fetch_fsmap finished waiting for FSMap version 36 2024-03-31T23:23:10.852 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:10.846+0000 7f26e3c2b740 10 client.17088 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-31T23:23:10.852 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:10.846+0000 7f26e3c2b740 20 client.17088 populate_metadata read hostname 'smithi046' 2024-03-31T23:23:10.852 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:10.846+0000 7f26e3c2b740 1 --2- 172.21.15.46:0/3606771792 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x5654bccb53d0 0x5654bccb77f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:23:10.852 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:10.846+0000 7f26e3c2b740 4 client.17088 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-31T23:23:10.852 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:10.846+0000 7f26e3c2b740 1 -- 172.21.15.46:0/3606771792 --> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5654bcba5e30 con 0x5654bccb53d0 2024-03-31T23:23:10.852 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:10.846+0000 7f26d3fff640 1 --2- 172.21.15.46:0/3606771792 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x5654bccb53d0 0x5654bccb77f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:23:10.852 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:10.846+0000 7f26d3fff640 1 --2- 172.21.15.46:0/3606771792 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x5654bccb53d0 0x5654bccb77f0 crc :-1 s=READY pgs=178 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=26c54a882a2bce61 server_cookie=6c5fb74d4e791aaa in_seq=0 out_seq=0 2024-03-31T23:23:10.852 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:10.846+0000 7f26c1ffb640 10 client.17088 ms_handle_connect on v2:172.21.15.142:6838/2519143918 2024-03-31T23:23:10.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:23:10 smithi046 ceph-mon[32498]: pgmap v357: 129 pgs: 129 active+clean; 128 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 197 KiB/s rd, 677 KiB/s wr, 72 op/s 2024-03-31T23:23:10.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:23:10 smithi046 ceph-mon[32498]: osdmap e108: 12 total, 12 up, 12 in 2024-03-31T23:23:10.876 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:10.874+0000 7f26c1ffb640 1 -- 172.21.15.46:0/3606771792 <== mds.0 v2:172.21.15.142:6838/2519143918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5654bcba5e30 con 0x5654bccb53d0 2024-03-31T23:23:10.876 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:10.874+0000 7f26c1ffb640 10 client.17088 handle_command_reply: tid=0 2024-03-31T23:23:10.876 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:10.874+0000 7f26e3c2b740 1 client.17088 shutdown 2024-03-31T23:23:10.876 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:10.874+0000 7f26e3c2b740 1 -- 172.21.15.46:0/3606771792 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5654bcc356c0 msgr2=0x5654bcc46cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:23:10.876 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:10.874+0000 7f26e3c2b740 1 --2- 172.21.15.46:0/3606771792 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5654bcc356c0 0x5654bcc46cf0 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7f26cc0045a0 tx=0x7f26cc0046c0 comp rx=0 tx=0).stop 2024-03-31T23:23:10.876 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:10.874+0000 7f26e3c2b740 1 -- 172.21.15.46:0/3606771792 shutdown_connections 2024-03-31T23:23:10.876 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:10.874+0000 7f26e3c2b740 1 --2- 172.21.15.46:0/3606771792 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x5654bccb53d0 0x5654bccb77f0 crc :-1 s=READY pgs=178 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:23:10.876 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:10.874+0000 7f26e3c2b740 1 --2- 172.21.15.46:0/3606771792 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5654bcc3edb0 0x5654bcc426c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:23:10.876 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:10.874+0000 7f26e3c2b740 1 --2- 172.21.15.46:0/3606771792 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5654bcc35fe0 0x5654bcc42180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:23:10.876 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:10.874+0000 7f26e3c2b740 1 --2- 172.21.15.46:0/3606771792 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5654bcc356c0 0x5654bcc46cf0 unknown :-1 s=CLOSED pgs=835 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:23:10.877 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:10.874+0000 7f26e3c2b740 1 -- 172.21.15.46:0/3606771792 >> 172.21.15.46:0/3606771792 conn(0x5654bcc2ffe0 msgr2=0x5654bcc2ef80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:23:10.877 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:10.874+0000 7f26c1ffb640 0 client.17088 ms_handle_reset on v2:172.21.15.142:6838/2519143918 2024-03-31T23:23:10.877 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:10.874+0000 7f26e3c2b740 1 -- 172.21.15.46:0/3606771792 shutdown_connections 2024-03-31T23:23:10.877 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:10.874+0000 7f26e3c2b740 1 -- 172.21.15.46:0/3606771792 wait complete. 2024-03-31T23:23:10.877 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:10.874+0000 7f26e3c2b740 20 client.17088 trim_cache size 0 max 16384 2024-03-31T23:23:10.877 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout:{ 2024-03-31T23:23:10.877 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout: "message": "", 2024-03-31T23:23:10.877 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout: "return_code": 0 2024-03-31T23:23:10.877 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout:} 2024-03-31T23:23:10.878 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:10.875+0000 7f26e0f6c640 1 -- 172.21.15.46:0/3844844193 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f26c80a26b0 msgr2=0x7f26c80a4b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:23:10.878 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:10.875+0000 7f26e0f6c640 1 --2- 172.21.15.46:0/3844844193 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f26c80a26b0 0x7f26c80a4b70 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x5654bcc35170 tx=0x7f26cc005f90 comp rx=0 tx=0).stop 2024-03-31T23:23:10.878 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:10.875+0000 7f26e0f6c640 1 -- 172.21.15.46:0/3844844193 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f26dc101260 msgr2=0x7f26dc19a670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:23:10.878 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:10.875+0000 7f26e0f6c640 1 --2- 172.21.15.46:0/3844844193 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f26dc101260 0x7f26dc19a670 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7f26c4000c00 tx=0x7f26c4005dd0 comp rx=0 tx=0).stop 2024-03-31T23:23:10.878 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:10.875+0000 7f26e0f6c640 1 -- 172.21.15.46:0/3844844193 shutdown_connections 2024-03-31T23:23:10.878 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:10.875+0000 7f26e0f6c640 1 --2- 172.21.15.46:0/3844844193 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f26c80a26b0 0x7f26c80a4b70 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:23:10.879 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:10.875+0000 7f26e0f6c640 1 --2- 172.21.15.46:0/3844844193 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f26dc1055f0 0x7f26dc19e9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:23:10.879 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:10.875+0000 7f26e0f6c640 1 --2- 172.21.15.46:0/3844844193 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f26dc101260 0x7f26dc19a670 unknown :-1 s=CLOSED pgs=832 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:23:10.879 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:10.875+0000 7f26e0f6c640 1 --2- 172.21.15.46:0/3844844193 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f26dc1008a0 0x7f26dc19a130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:23:10.879 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:10.875+0000 7f26e0f6c640 1 -- 172.21.15.46:0/3844844193 >> 172.21.15.46:0/3844844193 conn(0x7f26dc0774a0 msgr2=0x7f26dc068f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:23:10.879 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:10.875+0000 7f26e0f6c640 1 -- 172.21.15.46:0/3844844193 shutdown_connections 2024-03-31T23:23:10.879 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:10.875+0000 7f26e0f6c640 1 -- 172.21.15.46:0/3844844193 wait complete. 2024-03-31T23:23:10.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:23:10 smithi142 ceph-mon[35271]: pgmap v357: 129 pgs: 129 active+clean; 128 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 197 KiB/s rd, 677 KiB/s wr, 72 op/s 2024-03-31T23:23:10.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:23:10 smithi142 ceph-mon[35271]: osdmap e108: 12 total, 12 up, 12 in 2024-03-31T23:23:11.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:23:10 smithi162 ceph-mon[36237]: pgmap v357: 129 pgs: 129 active+clean; 128 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 197 KiB/s rd, 677 KiB/s wr, 72 op/s 2024-03-31T23:23:11.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:23:10 smithi162 ceph-mon[36237]: osdmap e108: 12 total, 12 up, 12 in 2024-03-31T23:23:11.217 INFO:tasks.workunit.client.0.smithi046.stdout: 1 224916 1.86 MB/sec execute 497 sec latency 16.120 ms 2024-03-31T23:23:12.217 INFO:tasks.workunit.client.0.smithi046.stdout: 1 225374 1.85 MB/sec execute 498 sec latency 24.120 ms 2024-03-31T23:23:12.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:23:12 smithi142 ceph-mon[35271]: pgmap v359: 129 pgs: 129 active+clean; 128 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 247 KiB/s rd, 846 KiB/s wr, 90 op/s 2024-03-31T23:23:13.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:23:12 smithi162 ceph-mon[36237]: pgmap v359: 129 pgs: 129 active+clean; 128 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 247 KiB/s rd, 846 KiB/s wr, 90 op/s 2024-03-31T23:23:13.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:23:12 smithi046 ceph-mon[32498]: pgmap v359: 129 pgs: 129 active+clean; 128 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 247 KiB/s rd, 846 KiB/s wr, 90 op/s 2024-03-31T23:23:13.217 INFO:tasks.workunit.client.0.smithi046.stdout: 1 225899 1.85 MB/sec execute 499 sec latency 22.792 ms 2024-03-31T23:23:14.217 INFO:tasks.workunit.client.0.smithi046.stdout: 1 226477 1.86 MB/sec execute 500 sec latency 16.484 ms 2024-03-31T23:23:14.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:23:14 smithi142 ceph-mon[35271]: pgmap v360: 129 pgs: 129 active+clean; 128 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 300 KiB/s wr, 26 op/s 2024-03-31T23:23:15.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:23:14 smithi162 ceph-mon[36237]: pgmap v360: 129 pgs: 129 active+clean; 128 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 300 KiB/s wr, 26 op/s 2024-03-31T23:23:15.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:23:14 smithi046 ceph-mon[32498]: pgmap v360: 129 pgs: 129 active+clean; 128 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 300 KiB/s wr, 26 op/s 2024-03-31T23:23:15.217 INFO:tasks.workunit.client.0.smithi046.stdout: 1 226869 1.86 MB/sec execute 501 sec latency 16.834 ms 2024-03-31T23:23:16.217 INFO:tasks.workunit.client.0.smithi046.stdout: 1 227128 1.86 MB/sec execute 502 sec latency 556.114 ms 2024-03-31T23:23:16.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:23:16 smithi142 ceph-mon[35271]: pgmap v361: 129 pgs: 129 active+clean; 132 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 460 KiB/s rd, 1.1 MiB/s wr, 114 op/s 2024-03-31T23:23:17.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:23:16 smithi162 ceph-mon[36237]: pgmap v361: 129 pgs: 129 active+clean; 132 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 460 KiB/s rd, 1.1 MiB/s wr, 114 op/s 2024-03-31T23:23:17.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:23:16 smithi046 ceph-mon[32498]: pgmap v361: 129 pgs: 129 active+clean; 132 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 460 KiB/s rd, 1.1 MiB/s wr, 114 op/s 2024-03-31T23:23:17.217 INFO:tasks.workunit.client.0.smithi046.stdout: 1 227128 1.86 MB/sec execute 503 sec latency 1556.170 ms 2024-03-31T23:23:18.217 INFO:tasks.workunit.client.0.smithi046.stdout: 1 227195 1.85 MB/sec execute 504 sec latency 2084.893 ms 2024-03-31T23:23:18.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:23:18 smithi142 ceph-mon[35271]: pgmap v362: 129 pgs: 129 active+clean; 132 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 398 KiB/s rd, 980 KiB/s wr, 99 op/s 2024-03-31T23:23:19.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:23:18 smithi162 ceph-mon[36237]: pgmap v362: 129 pgs: 129 active+clean; 132 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 398 KiB/s rd, 980 KiB/s wr, 99 op/s 2024-03-31T23:23:19.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:23:18 smithi046 ceph-mon[32498]: pgmap v362: 129 pgs: 129 active+clean; 132 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 398 KiB/s rd, 980 KiB/s wr, 99 op/s 2024-03-31T23:23:19.217 INFO:tasks.workunit.client.0.smithi046.stdout: 1 227195 1.85 MB/sec execute 505 sec latency 1218.551 ms 2024-03-31T23:23:20.218 INFO:tasks.workunit.client.0.smithi046.stdout: 1 227195 1.85 MB/sec execute 506 sec latency 2218.635 ms 2024-03-31T23:23:20.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:23:20 smithi142 ceph-mon[35271]: pgmap v363: 129 pgs: 129 active+clean; 134 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 470 KiB/s rd, 1.0 MiB/s wr, 104 op/s 2024-03-31T23:23:21.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:23:20 smithi162 ceph-mon[36237]: pgmap v363: 129 pgs: 129 active+clean; 134 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 470 KiB/s rd, 1.0 MiB/s wr, 104 op/s 2024-03-31T23:23:21.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:23:20 smithi046 ceph-mon[32498]: pgmap v363: 129 pgs: 129 active+clean; 134 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 470 KiB/s rd, 1.0 MiB/s wr, 104 op/s 2024-03-31T23:23:21.217 INFO:tasks.workunit.client.0.smithi046.stdout: 1 227195 1.84 MB/sec execute 507 sec latency 3218.735 ms 2024-03-31T23:23:22.218 INFO:tasks.workunit.client.0.smithi046.stdout: 1 227195 1.84 MB/sec execute 508 sec latency 4218.845 ms 2024-03-31T23:23:22.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:23:22 smithi142 ceph-mon[35271]: pgmap v364: 129 pgs: 129 active+clean; 134 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 455 KiB/s rd, 1.0 MiB/s wr, 100 op/s 2024-03-31T23:23:23.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:23:22 smithi162 ceph-mon[36237]: pgmap v364: 129 pgs: 129 active+clean; 134 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 455 KiB/s rd, 1.0 MiB/s wr, 100 op/s 2024-03-31T23:23:23.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:23:22 smithi046 ceph-mon[32498]: pgmap v364: 129 pgs: 129 active+clean; 134 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 455 KiB/s rd, 1.0 MiB/s wr, 100 op/s 2024-03-31T23:23:23.218 INFO:tasks.workunit.client.0.smithi046.stdout: 1 227331 1.84 MB/sec execute 509 sec latency 4746.132 ms 2024-03-31T23:23:23.976 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:23.972+0000 7f0da55ae640 1 -- 172.21.15.46:0/1959217411 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0da0100fd0 msgr2=0x7f0da01013b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:23:23.976 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:23.972+0000 7f0da55ae640 1 --2- 172.21.15.46:0/1959217411 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0da0100fd0 0x7f0da01013b0 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7f0d90009930 tx=0x7f0d9002bc80 comp rx=0 tx=0).stop 2024-03-31T23:23:23.976 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:23.973+0000 7f0da55ae640 1 -- 172.21.15.46:0/1959217411 shutdown_connections 2024-03-31T23:23:23.976 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:23.973+0000 7f0da55ae640 1 --2- 172.21.15.46:0/1959217411 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0da0105920 0x7f0da010d530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:23:23.976 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:23.973+0000 7f0da55ae640 1 --2- 172.21.15.46:0/1959217411 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0da0101980 0x7f0da01051f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:23:23.976 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:23.973+0000 7f0da55ae640 1 --2- 172.21.15.46:0/1959217411 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0da0100fd0 0x7f0da01013b0 unknown :-1 s=CLOSED pgs=836 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:23:23.976 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:23.973+0000 7f0da55ae640 1 -- 172.21.15.46:0/1959217411 >> 172.21.15.46:0/1959217411 conn(0x7f0da0076be0 msgr2=0x7f0da00fdf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:23:23.976 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:23.973+0000 7f0da55ae640 1 -- 172.21.15.46:0/1959217411 shutdown_connections 2024-03-31T23:23:23.977 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:23.973+0000 7f0da55ae640 1 -- 172.21.15.46:0/1959217411 wait complete. 2024-03-31T23:23:23.977 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:23.973+0000 7f0da55ae640 1 Processor -- start 2024-03-31T23:23:23.977 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:23.973+0000 7f0da55ae640 1 -- start start 2024-03-31T23:23:23.977 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:23.973+0000 7f0da55ae640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0da0100fd0 0x7f0da00fc020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:23:23.977 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:23.973+0000 7f0da55ae640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0da0105920 0x7f0da00fc560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:23:23.977 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:23.973+0000 7f0da55ae640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0da00fcaa0 0x7f0da01a6030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:23:23.977 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:23.973+0000 7f0da55ae640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f0da0110890 con 0x7f0da0100fd0 2024-03-31T23:23:23.977 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:23.973+0000 7f0da55ae640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f0da0110710 con 0x7f0da00fcaa0 2024-03-31T23:23:23.977 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:23.973+0000 7f0da55ae640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f0da0110a10 con 0x7f0da0105920 2024-03-31T23:23:23.977 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:23.974+0000 7f0d9effd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0da0100fd0 0x7f0da00fc020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:23:23.977 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:23.974+0000 7f0d9effd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0da0100fd0 0x7f0da00fc020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:35912/0 (socket says 172.21.15.46:35912) 2024-03-31T23:23:23.977 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:23.974+0000 7f0d9effd640 1 -- 172.21.15.46:0/1240307520 learned_addr learned my addr 172.21.15.46:0/1240307520 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:23:23.977 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:23.974+0000 7f0d97fff640 1 --2- 172.21.15.46:0/1240307520 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0da0105920 0x7f0da00fc560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:23:23.978 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:23.974+0000 7f0d9f7fe640 1 --2- 172.21.15.46:0/1240307520 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0da00fcaa0 0x7f0da01a6030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:23:23.978 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:23.974+0000 7f0d9effd640 1 -- 172.21.15.46:0/1240307520 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0da0105920 msgr2=0x7f0da00fc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:23:23.978 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:23.974+0000 7f0d9effd640 1 --2- 172.21.15.46:0/1240307520 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0da0105920 0x7f0da00fc560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:23:23.978 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:23.974+0000 7f0d9effd640 1 -- 172.21.15.46:0/1240307520 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0da00fcaa0 msgr2=0x7f0da01a6030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:23:23.978 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:23.974+0000 7f0d9effd640 1 --2- 172.21.15.46:0/1240307520 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0da00fcaa0 0x7f0da01a6030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:23:23.978 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:23.974+0000 7f0d9effd640 1 -- 172.21.15.46:0/1240307520 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0da01a6730 con 0x7f0da0100fd0 2024-03-31T23:23:23.978 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:23.974+0000 7f0d9effd640 1 --2- 172.21.15.46:0/1240307520 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0da0100fd0 0x7f0da00fc020 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7f0d90009230 tx=0x7f0d9002c700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:23:23.978 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:23.974+0000 7f0d9cff9640 1 -- 172.21.15.46:0/1240307520 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d90043020 con 0x7f0da0100fd0 2024-03-31T23:23:23.978 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:23.974+0000 7f0d9cff9640 1 -- 172.21.15.46:0/1240307520 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0d9002cae0 con 0x7f0da0100fd0 2024-03-31T23:23:23.978 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:23.974+0000 7f0da55ae640 1 -- 172.21.15.46:0/1240307520 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0da01a69c0 con 0x7f0da0100fd0 2024-03-31T23:23:23.978 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:23.974+0000 7f0d9cff9640 1 -- 172.21.15.46:0/1240307520 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d90003ae0 con 0x7f0da0100fd0 2024-03-31T23:23:23.978 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:23.974+0000 7f0da55ae640 1 -- 172.21.15.46:0/1240307520 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0da01a6f60 con 0x7f0da0100fd0 2024-03-31T23:23:23.984 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:23.976+0000 7f0d9cff9640 1 -- 172.21.15.46:0/1240307520 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 25) ==== 137451+0+0 (secure 0 0 0) 0x7f0d90046020 con 0x7f0da0100fd0 2024-03-31T23:23:23.984 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:23.977+0000 7f0da818a740 1 Processor -- start 2024-03-31T23:23:23.984 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:23.977+0000 7f0da818a740 1 -- start start 2024-03-31T23:23:23.984 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:23.977+0000 7f0d9cff9640 1 --2- 172.21.15.46:0/1240307520 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f0d8c0a2760 0x7f0d8c0a4c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:23:23.984 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:23.977+0000 7f0d9f7fe640 1 --2- 172.21.15.46:0/1240307520 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f0d8c0a2760 0x7f0d8c0a4c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:23:23.984 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:23.977+0000 7f0d9cff9640 1 -- 172.21.15.46:0/1240307520 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 9554+0+0 (secure 0 0 0) 0x7f0d900f0ca0 con 0x7f0da0100fd0 2024-03-31T23:23:23.985 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:23.977+0000 7f0da818a740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55931582b6c0 0x55931582baa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:23:23.985 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:23.977+0000 7f0da818a740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55931582bfe0 0x559315834870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:23:23.985 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:23.977+0000 7f0da818a740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559315834db0 0x5593158371a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:23:23.985 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:23.977+0000 7f0da818a740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55931578e8f0 con 0x55931582b6c0 2024-03-31T23:23:23.985 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:23.977+0000 7f0da818a740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x5593157378d0 con 0x559315834db0 2024-03-31T23:23:23.985 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:23.977+0000 7f0da818a740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55931578a990 con 0x55931582bfe0 2024-03-31T23:23:23.985 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:23.977+0000 7f0d9effd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55931582b6c0 0x55931582baa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:23:23.985 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:23.977+0000 7f0d9effd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55931582b6c0 0x55931582baa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:35918/0 (socket says 172.21.15.46:35918) 2024-03-31T23:23:23.985 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:23.977+0000 7f0d9effd640 1 -- 172.21.15.46:0/3516556292 learned_addr learned my addr 172.21.15.46:0/3516556292 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:23:23.985 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:23.977+0000 7f0d97fff640 1 --2- 172.21.15.46:0/3516556292 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55931582bfe0 0x559315834870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:23:23.985 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:23.978+0000 7f0d9f7fe640 1 --2- 172.21.15.46:0/3516556292 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559315834db0 0x5593158371a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:23:23.985 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:23.978+0000 7f0d9effd640 1 -- 172.21.15.46:0/3516556292 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55931582bfe0 msgr2=0x559315834870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:23:23.985 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:23.978+0000 7f0d9effd640 1 --2- 172.21.15.46:0/3516556292 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55931582bfe0 0x559315834870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:23:23.985 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:23.978+0000 7f0d9effd640 1 -- 172.21.15.46:0/3516556292 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559315834db0 msgr2=0x5593158371a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:23:23.986 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:23.978+0000 7f0d9effd640 1 --2- 172.21.15.46:0/3516556292 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559315834db0 0x5593158371a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:23:23.986 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:23.978+0000 7f0d9effd640 1 -- 172.21.15.46:0/3516556292 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5593154a3210 con 0x55931582b6c0 2024-03-31T23:23:23.986 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:23.978+0000 7f0d9f7fe640 1 --2- 172.21.15.46:0/3516556292 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559315834db0 0x5593158371a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T23:23:23.986 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:23.978+0000 7f0d9f7fe640 1 --2- 172.21.15.46:0/1240307520 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f0d8c0a2760 0x7f0d8c0a4c20 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f0d98005fd0 tx=0x7f0d980107a0 comp rx=0 tx=0).ready entity=mgr.15654 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:23:23.986 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:23.978+0000 7f0d9effd640 1 --2- 172.21.15.46:0/3516556292 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55931582b6c0 0x55931582baa0 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7f0d900bd0b0 tx=0x7f0d900bcb40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f64bd84f81a10441 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:23:23.986 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:23.978+0000 7f0d957fa640 1 -- 172.21.15.46:0/3516556292 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d900c1270 con 0x55931582b6c0 2024-03-31T23:23:23.986 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:23.978+0000 7f0d957fa640 1 -- 172.21.15.46:0/3516556292 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0d900c5b30 con 0x55931582b6c0 2024-03-31T23:23:23.986 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:23.978+0000 7f0d957fa640 1 -- 172.21.15.46:0/3516556292 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d90005490 con 0x55931582b6c0 2024-03-31T23:23:23.986 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:23.979+0000 7f0da818a740 1 -- 172.21.15.46:0/3516556292 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55931582b6c0 msgr2=0x55931582baa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:23:23.986 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:23.979+0000 7f0da818a740 1 --2- 172.21.15.46:0/3516556292 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55931582b6c0 0x55931582baa0 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7f0d900bd0b0 tx=0x7f0d900bcb40 comp rx=0 tx=0).stop 2024-03-31T23:23:23.986 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:23.979+0000 7f0da818a740 1 -- 172.21.15.46:0/3516556292 shutdown_connections 2024-03-31T23:23:23.986 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:23.979+0000 7f0da818a740 1 --2- 172.21.15.46:0/3516556292 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559315834db0 0x5593158371a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:23:23.986 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:23.979+0000 7f0da818a740 1 --2- 172.21.15.46:0/3516556292 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55931582bfe0 0x559315834870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:23:23.986 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:23.979+0000 7f0da818a740 1 --2- 172.21.15.46:0/3516556292 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55931582b6c0 0x55931582baa0 unknown :-1 s=CLOSED pgs=838 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:23:23.987 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:23.979+0000 7f0da818a740 1 -- 172.21.15.46:0/3516556292 >> 172.21.15.46:0/3516556292 conn(0x559315825fe0 msgr2=0x5593158263c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:23:23.987 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:23.979+0000 7f0da818a740 1 -- 172.21.15.46:0/3516556292 shutdown_connections 2024-03-31T23:23:23.987 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:23.979+0000 7f0da818a740 1 -- 172.21.15.46:0/3516556292 wait complete. 2024-03-31T23:23:23.987 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:23.979+0000 7f0da818a740 1 -- start start 2024-03-31T23:23:23.987 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:23.980+0000 7f0da818a740 1 Processor -- start 2024-03-31T23:23:23.987 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:23.980+0000 7f0da818a740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55931582b6c0 0x559315831e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:23:23.987 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:23.980+0000 7f0da818a740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55931582bfe0 0x559315832340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:23:23.987 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:23.980+0000 7f0da818a740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559315834db0 0x559315832880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:23:23.987 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:23.980+0000 7f0da818a740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x5593158377a0 con 0x55931582b6c0 2024-03-31T23:23:23.987 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:23.980+0000 7f0da818a740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55931572cf80 con 0x559315834db0 2024-03-31T23:23:23.988 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:23.980+0000 7f0da818a740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x559315837920 con 0x55931582bfe0 2024-03-31T23:23:23.988 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:23.980+0000 7f0da818a740 10 client.0 _ll_register_callbacks cb 0x559315785290 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T23:23:23.988 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:23.980+0000 7f0d9effd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55931582b6c0 0x559315831e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:23:23.988 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:23.980+0000 7f0d9effd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55931582b6c0 0x559315831e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:35928/0 (socket says 172.21.15.46:35928) 2024-03-31T23:23:23.988 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:23.980+0000 7f0d9effd640 1 -- 172.21.15.46:0/2075258374 learned_addr learned my addr 172.21.15.46:0/2075258374 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:23:23.988 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:23.980+0000 7f0d9f7fe640 1 --2- 172.21.15.46:0/2075258374 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559315834db0 0x559315832880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:23:23.988 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:23.980+0000 7f0d97fff640 1 --2- 172.21.15.46:0/2075258374 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55931582bfe0 0x559315832340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:23:23.988 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:23.981+0000 7f0d9effd640 1 -- 172.21.15.46:0/2075258374 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55931582bfe0 msgr2=0x559315832340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:23:23.988 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:23.981+0000 7f0d9effd640 1 --2- 172.21.15.46:0/2075258374 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55931582bfe0 0x559315832340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:23:23.988 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:23.981+0000 7f0d9effd640 1 -- 172.21.15.46:0/2075258374 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559315834db0 msgr2=0x559315832880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:23:23.988 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:23.981+0000 7f0d9effd640 1 --2- 172.21.15.46:0/2075258374 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559315834db0 0x559315832880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:23:23.988 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:23.981+0000 7f0d9effd640 1 -- 172.21.15.46:0/2075258374 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55931556f620 con 0x55931582b6c0 2024-03-31T23:23:23.989 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:23.981+0000 7f0d9effd640 1 --2- 172.21.15.46:0/2075258374 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55931582b6c0 0x559315831e00 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7f0d900b9af0 tx=0x7f0d90003eb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=15652c4274e52a52 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:23:23.989 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:23.981+0000 7f0d867fc640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-31T23:23:23.989 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:23.981+0000 7f0d867fc640 1 -- 172.21.15.46:0/2075258374 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d900056a0 con 0x55931582b6c0 2024-03-31T23:23:23.989 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:23.981+0000 7f0d867fc640 1 -- 172.21.15.46:0/2075258374 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(108..108 src has 1..108) ==== 9554+0+0 (secure 0 0 0) 0x7f0d900c56a0 con 0x55931582b6c0 2024-03-31T23:23:23.989 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:23.981+0000 7f0da818a740 1 -- 172.21.15.46:0/2075258374 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0d900c5b30 con 0x55931582b6c0 2024-03-31T23:23:23.989 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:23.981+0000 7f0d867fc640 1 -- 172.21.15.46:0/2075258374 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f0d900c1270 con 0x55931582b6c0 2024-03-31T23:23:23.989 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:23.981+0000 7f0da818a740 10 client.17100 fetch_fsmap learned FSMap version 36 2024-03-31T23:23:23.989 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:23.981+0000 7f0da818a740 1 -- 172.21.15.46:0/2075258374 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x5593158377a0 con 0x55931582b6c0 2024-03-31T23:23:23.989 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:23.981+0000 7f0d867fc640 1 -- 172.21.15.46:0/2075258374 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0d900c5b30 con 0x55931582b6c0 2024-03-31T23:23:23.989 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:23.981+0000 7f0d867fc640 1 -- 172.21.15.46:0/2075258374 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 36) ==== 8362+0+0 (secure 0 0 0) 0x7f0d900dbc90 con 0x55931582b6c0 2024-03-31T23:23:23.989 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:23.981+0000 7f0da818a740 10 client.17100 fetch_fsmap finished waiting for FSMap version 36 2024-03-31T23:23:23.989 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:23.981+0000 7f0da818a740 10 client.17100 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-31T23:23:23.989 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:23.981+0000 7f0da818a740 20 client.17100 populate_metadata read hostname 'smithi046' 2024-03-31T23:23:23.989 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:23.981+0000 7f0da818a740 1 --2- 172.21.15.46:0/2075258374 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55931584d0f0 0x55931584f530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:23:23.990 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:23.981+0000 7f0da818a740 4 client.17100 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-31T23:23:23.990 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:23.981+0000 7f0da818a740 1 -- 172.21.15.46:0/2075258374 --> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5593157a2120 con 0x55931584d0f0 2024-03-31T23:23:23.990 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:23.982+0000 7f0d97fff640 1 --2- 172.21.15.46:0/2075258374 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55931584d0f0 0x55931584f530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:23:23.990 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:23.982+0000 7f0d97fff640 1 --2- 172.21.15.46:0/2075258374 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55931584d0f0 0x55931584f530 crc :-1 s=READY pgs=179 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7b815ff2e780dc2c server_cookie=62995c417a8bdcd1 in_seq=0 out_seq=0 2024-03-31T23:23:23.990 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:23.982+0000 7f0d867fc640 10 client.17100 ms_handle_connect on v2:172.21.15.142:6838/2519143918 2024-03-31T23:23:23.990 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:23.983+0000 7f0d867fc640 1 -- 172.21.15.46:0/2075258374 <== mds.0 v2:172.21.15.142:6838/2519143918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5593157a2120 con 0x55931584d0f0 2024-03-31T23:23:23.990 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:23.983+0000 7f0d867fc640 10 client.17100 handle_command_reply: tid=0 2024-03-31T23:23:23.990 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:23.983+0000 7f0da818a740 1 client.17100 shutdown 2024-03-31T23:23:23.990 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:23.983+0000 7f0da818a740 1 -- 172.21.15.46:0/2075258374 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55931582b6c0 msgr2=0x559315831e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:23:23.990 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:23.983+0000 7f0da818a740 1 --2- 172.21.15.46:0/2075258374 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55931582b6c0 0x559315831e00 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7f0d900b9af0 tx=0x7f0d90003eb0 comp rx=0 tx=0).stop 2024-03-31T23:23:23.990 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:23.983+0000 7f0da818a740 1 -- 172.21.15.46:0/2075258374 shutdown_connections 2024-03-31T23:23:23.990 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:23.983+0000 7f0da818a740 1 --2- 172.21.15.46:0/2075258374 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55931584d0f0 0x55931584f530 crc :-1 s=READY pgs=179 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:23:23.990 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:23.983+0000 7f0da818a740 1 --2- 172.21.15.46:0/2075258374 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559315834db0 0x559315832880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:23:23.991 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:23.983+0000 7f0da818a740 1 --2- 172.21.15.46:0/2075258374 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55931582bfe0 0x559315832340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:23:23.991 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:23.983+0000 7f0da818a740 1 --2- 172.21.15.46:0/2075258374 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55931582b6c0 0x559315831e00 unknown :-1 s=CLOSED pgs=839 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:23:23.991 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:23.983+0000 7f0d867fc640 0 client.17100 ms_handle_reset on v2:172.21.15.142:6838/2519143918 2024-03-31T23:23:23.991 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:23.983+0000 7f0da818a740 1 -- 172.21.15.46:0/2075258374 >> 172.21.15.46:0/2075258374 conn(0x559315825fe0 msgr2=0x559315827d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:23:23.991 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:23.983+0000 7f0da818a740 1 -- 172.21.15.46:0/2075258374 shutdown_connections 2024-03-31T23:23:23.991 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:23.983+0000 7f0da818a740 1 -- 172.21.15.46:0/2075258374 wait complete. 2024-03-31T23:23:23.991 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:23.983+0000 7f0da818a740 20 client.17100 trim_cache size 0 max 16384 2024-03-31T23:23:24.001 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:23.998+0000 7f0da818a740 1 Processor -- start 2024-03-31T23:23:24.001 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:23.998+0000 7f0da818a740 1 -- start start 2024-03-31T23:23:24.001 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:23.998+0000 7f0da818a740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55931582b6c0 0x559315834530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:23:24.001 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:23.998+0000 7f0da818a740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55931582bfe0 0x559315848d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:23:24.001 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:23.998+0000 7f0da818a740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559315834db0 0x559315827b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:23:24.001 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:23.998+0000 7f0da818a740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f0da01af280 con 0x55931582bfe0 2024-03-31T23:23:24.001 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:23.998+0000 7f0da818a740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f0da01acaf0 con 0x559315834db0 2024-03-31T23:23:24.001 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:23.998+0000 7f0da818a740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f0da01aa9a0 con 0x55931582b6c0 2024-03-31T23:23:24.001 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:23.999+0000 7f0d97fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55931582bfe0 0x559315848d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:23:24.001 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:23.999+0000 7f0d9f7fe640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559315834db0 0x559315827b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:23:24.001 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:23.999+0000 7f0d9effd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55931582b6c0 0x559315834530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:23:24.001 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:23.999+0000 7f0d97fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55931582bfe0 0x559315848d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:35942/0 (socket says 172.21.15.46:35942) 2024-03-31T23:23:24.001 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:23.999+0000 7f0d9f7fe640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559315834db0 0x559315827b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.46:34772/0 (socket says 172.21.15.46:34772) 2024-03-31T23:23:24.002 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:23.999+0000 7f0d97fff640 1 -- 172.21.15.46:0/3748940071 learned_addr learned my addr 172.21.15.46:0/3748940071 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:23:24.002 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:23.999+0000 7f0d9effd640 1 -- 172.21.15.46:0/3748940071 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559315834db0 msgr2=0x559315827b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:23:24.002 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:23.999+0000 7f0d9effd640 1 --2- 172.21.15.46:0/3748940071 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559315834db0 0x559315827b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:23:24.002 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:23.999+0000 7f0d9effd640 1 -- 172.21.15.46:0/3748940071 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55931582bfe0 msgr2=0x559315848d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:23:24.002 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:23.999+0000 7f0d9effd640 1 --2- 172.21.15.46:0/3748940071 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55931582bfe0 0x559315848d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:23:24.002 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:23.999+0000 7f0d9effd640 1 -- 172.21.15.46:0/3748940071 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5593158371a0 con 0x55931582b6c0 2024-03-31T23:23:24.002 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:23.999+0000 7f0d9f7fe640 1 --2- 172.21.15.46:0/3748940071 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559315834db0 0x559315827b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:23:24.002 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:23.999+0000 7f0d97fff640 1 --2- 172.21.15.46:0/3748940071 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55931582bfe0 0x559315848d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:23:24.002 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:23.999+0000 7f0d9effd640 1 --2- 172.21.15.46:0/3748940071 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55931582b6c0 0x559315834530 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f0d900bd0b0 tx=0x7f0d900035f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fce9563482b786 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:23:24.002 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:23.999+0000 7f0d94ff9640 1 -- 172.21.15.46:0/3748940071 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d900056a0 con 0x55931582b6c0 2024-03-31T23:23:24.002 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:23.999+0000 7f0d94ff9640 1 -- 172.21.15.46:0/3748940071 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0d900c5b30 con 0x55931582b6c0 2024-03-31T23:23:24.002 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:23.999+0000 7f0d94ff9640 1 -- 172.21.15.46:0/3748940071 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d900b9930 con 0x55931582b6c0 2024-03-31T23:23:24.006 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:24.000+0000 7f0da818a740 1 -- 172.21.15.46:0/3748940071 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55931582b6c0 msgr2=0x559315834530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:23:24.006 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:24.000+0000 7f0da818a740 1 --2- 172.21.15.46:0/3748940071 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55931582b6c0 0x559315834530 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f0d900bd0b0 tx=0x7f0d900035f0 comp rx=0 tx=0).stop 2024-03-31T23:23:24.006 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:24.000+0000 7f0da818a740 1 -- 172.21.15.46:0/3748940071 shutdown_connections 2024-03-31T23:23:24.006 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:24.000+0000 7f0da818a740 1 --2- 172.21.15.46:0/3748940071 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559315834db0 0x559315827b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:23:24.006 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:24.000+0000 7f0da818a740 1 --2- 172.21.15.46:0/3748940071 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55931582bfe0 0x559315848d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:23:24.006 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:24.000+0000 7f0da818a740 1 --2- 172.21.15.46:0/3748940071 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55931582b6c0 0x559315834530 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:23:24.006 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:24.000+0000 7f0da818a740 1 -- 172.21.15.46:0/3748940071 >> 172.21.15.46:0/3748940071 conn(0x559315825fe0 msgr2=0x559315827430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:23:24.006 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:24.000+0000 7f0da818a740 1 -- 172.21.15.46:0/3748940071 shutdown_connections 2024-03-31T23:23:24.006 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:24.000+0000 7f0da818a740 1 -- 172.21.15.46:0/3748940071 wait complete. 2024-03-31T23:23:24.006 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:24.001+0000 7f0da818a740 1 -- start start 2024-03-31T23:23:24.006 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:24.001+0000 7f0da818a740 1 Processor -- start 2024-03-31T23:23:24.006 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:24.001+0000 7f0da818a740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55931582b6c0 0x559315845520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:23:24.007 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:24.001+0000 7f0da818a740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55931582bfe0 0x55931582e670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:23:24.007 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:24.001+0000 7f0da818a740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559315834db0 0x55931582ebb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:23:24.007 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:24.001+0000 7f0da818a740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f0da01aaff0 con 0x559315834db0 2024-03-31T23:23:24.007 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:24.001+0000 7f0da818a740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f0da01ac420 con 0x55931582bfe0 2024-03-31T23:23:24.007 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:24.001+0000 7f0da818a740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f0da01ab680 con 0x55931582b6c0 2024-03-31T23:23:24.007 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:24.001+0000 7f0da818a740 10 client.0 _ll_register_callbacks cb 0x7f0d900d5070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T23:23:24.007 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:24.001+0000 7f0d9f7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559315834db0 0x55931582ebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:23:24.007 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:24.001+0000 7f0d9f7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559315834db0 0x55931582ebb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:35950/0 (socket says 172.21.15.46:35950) 2024-03-31T23:23:24.007 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:24.001+0000 7f0d9f7fe640 1 -- 172.21.15.46:0/1988928923 learned_addr learned my addr 172.21.15.46:0/1988928923 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:23:24.007 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:24.001+0000 7f0d9effd640 1 --2- 172.21.15.46:0/1988928923 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55931582b6c0 0x559315845520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:23:24.007 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:24.001+0000 7f0d97fff640 1 --2- 172.21.15.46:0/1988928923 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55931582bfe0 0x55931582e670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:23:24.007 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:24.001+0000 7f0d9f7fe640 1 -- 172.21.15.46:0/1988928923 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55931582b6c0 msgr2=0x559315845520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:23:24.007 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:24.001+0000 7f0d9f7fe640 1 --2- 172.21.15.46:0/1988928923 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55931582b6c0 0x559315845520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:23:24.008 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:24.001+0000 7f0d9f7fe640 1 -- 172.21.15.46:0/1988928923 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55931582bfe0 msgr2=0x55931582e670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:23:24.008 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:24.001+0000 7f0d9f7fe640 1 --2- 172.21.15.46:0/1988928923 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55931582bfe0 0x55931582e670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:23:24.008 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:24.001+0000 7f0d9f7fe640 1 -- 172.21.15.46:0/1988928923 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55931582f3b0 con 0x559315834db0 2024-03-31T23:23:24.008 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:24.002+0000 7f0d9f7fe640 1 --2- 172.21.15.46:0/1988928923 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559315834db0 0x55931582ebb0 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7f0d9801a000 tx=0x7f0d98004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c86329c42b0cbcb1 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:23:24.008 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:24.002+0000 7f0d867fc640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-31T23:23:24.008 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:24.002+0000 7f0d867fc640 1 -- 172.21.15.46:0/1988928923 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d9802d060 con 0x559315834db0 2024-03-31T23:23:24.008 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:24.002+0000 7f0d867fc640 1 -- 172.21.15.46:0/1988928923 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(108..108 src has 1..108) ==== 9554+0+0 (secure 0 0 0) 0x7f0d9801fb80 con 0x559315834db0 2024-03-31T23:23:24.008 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:24.002+0000 7f0da818a740 1 -- 172.21.15.46:0/1988928923 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0d900c5b30 con 0x559315834db0 2024-03-31T23:23:24.008 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:24.002+0000 7f0d867fc640 1 -- 172.21.15.46:0/1988928923 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f0d980132e0 con 0x559315834db0 2024-03-31T23:23:24.008 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:24.002+0000 7f0da818a740 10 client.17106 fetch_fsmap learned FSMap version 36 2024-03-31T23:23:24.008 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:24.002+0000 7f0da818a740 1 -- 172.21.15.46:0/1988928923 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x5593158b3340 con 0x559315834db0 2024-03-31T23:23:24.008 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:24.002+0000 7f0d867fc640 1 -- 172.21.15.46:0/1988928923 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0d900c5b30 con 0x559315834db0 2024-03-31T23:23:24.008 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:24.002+0000 7f0d867fc640 1 -- 172.21.15.46:0/1988928923 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 36) ==== 8362+0+0 (secure 0 0 0) 0x7f0d980044c0 con 0x559315834db0 2024-03-31T23:23:24.008 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:24.002+0000 7f0da818a740 10 client.17106 fetch_fsmap finished waiting for FSMap version 36 2024-03-31T23:23:24.008 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:24.002+0000 7f0da818a740 10 client.17106 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-31T23:23:24.009 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:24.002+0000 7f0da818a740 20 client.17106 populate_metadata read hostname 'smithi046' 2024-03-31T23:23:24.009 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:24.002+0000 7f0da818a740 1 --2- 172.21.15.46:0/1988928923 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x5593158b3b50 0x5593158b5f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:23:24.009 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:24.002+0000 7f0da818a740 4 client.17106 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-31T23:23:24.009 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:24.002+0000 7f0da818a740 1 -- 172.21.15.46:0/1988928923 --> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55931579be30 con 0x5593158b3b50 2024-03-31T23:23:24.009 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:24.002+0000 7f0d9effd640 1 --2- 172.21.15.46:0/1988928923 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x5593158b3b50 0x5593158b5f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:23:24.009 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:24.003+0000 7f0d9effd640 1 --2- 172.21.15.46:0/1988928923 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x5593158b3b50 0x5593158b5f70 crc :-1 s=READY pgs=180 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d28855239efd2f34 server_cookie=e0613b961d9cb3a7 in_seq=0 out_seq=0 2024-03-31T23:23:24.009 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:24.003+0000 7f0d867fc640 10 client.17106 ms_handle_connect on v2:172.21.15.142:6838/2519143918 2024-03-31T23:23:24.033 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:24.031+0000 7f0d867fc640 1 -- 172.21.15.46:0/1988928923 <== mds.0 v2:172.21.15.142:6838/2519143918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55931579be30 con 0x5593158b3b50 2024-03-31T23:23:24.033 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:24.031+0000 7f0d867fc640 10 client.17106 handle_command_reply: tid=0 2024-03-31T23:23:24.033 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:24.031+0000 7f0da818a740 1 client.17106 shutdown 2024-03-31T23:23:24.033 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:24.031+0000 7f0da818a740 1 -- 172.21.15.46:0/1988928923 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559315834db0 msgr2=0x55931582ebb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:23:24.033 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:24.031+0000 7f0da818a740 1 --2- 172.21.15.46:0/1988928923 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559315834db0 0x55931582ebb0 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7f0d9801a000 tx=0x7f0d98004ab0 comp rx=0 tx=0).stop 2024-03-31T23:23:24.034 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:24.031+0000 7f0da818a740 1 -- 172.21.15.46:0/1988928923 shutdown_connections 2024-03-31T23:23:24.034 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:24.031+0000 7f0da818a740 1 --2- 172.21.15.46:0/1988928923 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x5593158b3b50 0x5593158b5f70 crc :-1 s=READY pgs=180 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:23:24.034 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:24.031+0000 7f0da818a740 1 --2- 172.21.15.46:0/1988928923 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559315834db0 0x55931582ebb0 unknown :-1 s=CLOSED pgs=840 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:23:24.034 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:24.031+0000 7f0da818a740 1 --2- 172.21.15.46:0/1988928923 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55931582bfe0 0x55931582e670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:23:24.034 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:24.031+0000 7f0d867fc640 0 client.17106 ms_handle_reset on v2:172.21.15.142:6838/2519143918 2024-03-31T23:23:24.034 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:24.031+0000 7f0da818a740 1 --2- 172.21.15.46:0/1988928923 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55931582b6c0 0x559315845520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:23:24.034 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:24.031+0000 7f0da818a740 1 -- 172.21.15.46:0/1988928923 >> 172.21.15.46:0/1988928923 conn(0x559315825fe0 msgr2=0x559315824f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:23:24.034 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:24.031+0000 7f0da818a740 1 -- 172.21.15.46:0/1988928923 shutdown_connections 2024-03-31T23:23:24.034 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:24.032+0000 7f0da818a740 1 -- 172.21.15.46:0/1988928923 wait complete. 2024-03-31T23:23:24.034 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:24.032+0000 7f0da818a740 20 client.17106 trim_cache size 0 max 16384 2024-03-31T23:23:24.035 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout:{ 2024-03-31T23:23:24.035 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout: "message": "", 2024-03-31T23:23:24.035 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout: "return_code": 0 2024-03-31T23:23:24.035 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout:} 2024-03-31T23:23:24.036 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:24.032+0000 7f0da55ae640 1 -- 172.21.15.46:0/1240307520 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f0d8c0a2760 msgr2=0x7f0d8c0a4c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:23:24.036 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:24.032+0000 7f0da55ae640 1 --2- 172.21.15.46:0/1240307520 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f0d8c0a2760 0x7f0d8c0a4c20 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f0d98005fd0 tx=0x7f0d980107a0 comp rx=0 tx=0).stop 2024-03-31T23:23:24.036 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:24.032+0000 7f0da55ae640 1 -- 172.21.15.46:0/1240307520 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0da0100fd0 msgr2=0x7f0da00fc020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:23:24.036 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:24.032+0000 7f0da55ae640 1 --2- 172.21.15.46:0/1240307520 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0da0100fd0 0x7f0da00fc020 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7f0d90009230 tx=0x7f0d9002c700 comp rx=0 tx=0).stop 2024-03-31T23:23:24.036 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:24.032+0000 7f0da55ae640 1 -- 172.21.15.46:0/1240307520 shutdown_connections 2024-03-31T23:23:24.036 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:24.032+0000 7f0da55ae640 1 --2- 172.21.15.46:0/1240307520 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f0d8c0a2760 0x7f0d8c0a4c20 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:23:24.036 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:24.032+0000 7f0da55ae640 1 --2- 172.21.15.46:0/1240307520 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0da00fcaa0 0x7f0da01a6030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:23:24.036 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:24.032+0000 7f0da55ae640 1 --2- 172.21.15.46:0/1240307520 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0da0105920 0x7f0da00fc560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:23:24.036 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:24.032+0000 7f0da55ae640 1 --2- 172.21.15.46:0/1240307520 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0da0100fd0 0x7f0da00fc020 unknown :-1 s=CLOSED pgs=837 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:23:24.036 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:24.032+0000 7f0da55ae640 1 -- 172.21.15.46:0/1240307520 >> 172.21.15.46:0/1240307520 conn(0x7f0da0076be0 msgr2=0x7f0da0102fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:23:24.036 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:24.032+0000 7f0da55ae640 1 -- 172.21.15.46:0/1240307520 shutdown_connections 2024-03-31T23:23:24.036 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:24.032+0000 7f0da55ae640 1 -- 172.21.15.46:0/1240307520 wait complete. 2024-03-31T23:23:24.218 INFO:tasks.workunit.client.0.smithi046.stdout: 1 227582 1.84 MB/sec execute 510 sec latency 18.441 ms 2024-03-31T23:23:24.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:23:24 smithi142 ceph-mon[35271]: pgmap v365: 129 pgs: 129 active+clean; 134 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 392 KiB/s rd, 883 KiB/s wr, 86 op/s 2024-03-31T23:23:25.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:23:24 smithi162 ceph-mon[36237]: pgmap v365: 129 pgs: 129 active+clean; 134 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 392 KiB/s rd, 883 KiB/s wr, 86 op/s 2024-03-31T23:23:25.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:23:24 smithi046 ceph-mon[32498]: pgmap v365: 129 pgs: 129 active+clean; 134 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 392 KiB/s rd, 883 KiB/s wr, 86 op/s 2024-03-31T23:23:25.218 INFO:tasks.workunit.client.0.smithi046.stdout: 1 227982 1.83 MB/sec execute 511 sec latency 15.693 ms 2024-03-31T23:23:26.218 INFO:tasks.workunit.client.0.smithi046.stdout: 1 228480 1.84 MB/sec execute 512 sec latency 16.950 ms 2024-03-31T23:23:26.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:23:26 smithi142 ceph-mon[35271]: pgmap v366: 129 pgs: 129 active+clean; 136 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 425 KiB/s rd, 1.0 MiB/s wr, 103 op/s 2024-03-31T23:23:27.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:23:26 smithi162 ceph-mon[36237]: pgmap v366: 129 pgs: 129 active+clean; 136 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 425 KiB/s rd, 1.0 MiB/s wr, 103 op/s 2024-03-31T23:23:27.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:23:26 smithi046 ceph-mon[32498]: pgmap v366: 129 pgs: 129 active+clean; 136 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 425 KiB/s rd, 1.0 MiB/s wr, 103 op/s 2024-03-31T23:23:27.218 INFO:tasks.workunit.client.0.smithi046.stdout: 1 228924 1.84 MB/sec execute 513 sec latency 22.944 ms 2024-03-31T23:23:28.218 INFO:tasks.workunit.client.0.smithi046.stdout: 1 229453 1.84 MB/sec execute 514 sec latency 24.219 ms 2024-03-31T23:23:28.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:23:28 smithi142 ceph-mon[35271]: pgmap v367: 129 pgs: 129 active+clean; 136 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 156 KiB/s rd, 517 KiB/s wr, 44 op/s 2024-03-31T23:23:29.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:23:28 smithi162 ceph-mon[36237]: pgmap v367: 129 pgs: 129 active+clean; 136 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 156 KiB/s rd, 517 KiB/s wr, 44 op/s 2024-03-31T23:23:29.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:23:28 smithi046 ceph-mon[32498]: pgmap v367: 129 pgs: 129 active+clean; 136 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 156 KiB/s rd, 517 KiB/s wr, 44 op/s 2024-03-31T23:23:29.218 INFO:tasks.workunit.client.0.smithi046.stdout: 1 230043 1.84 MB/sec execute 515 sec latency 15.887 ms 2024-03-31T23:23:30.218 INFO:tasks.workunit.client.0.smithi046.stdout: 1 230543 1.85 MB/sec execute 516 sec latency 17.003 ms 2024-03-31T23:23:30.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:23:30 smithi142 ceph-mon[35271]: pgmap v368: 129 pgs: 129 active+clean; 133 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 402 KiB/s rd, 1.0 MiB/s wr, 104 op/s 2024-03-31T23:23:31.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:23:30 smithi162 ceph-mon[36237]: pgmap v368: 129 pgs: 129 active+clean; 133 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 402 KiB/s rd, 1.0 MiB/s wr, 104 op/s 2024-03-31T23:23:31.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:23:30 smithi046 ceph-mon[32498]: pgmap v368: 129 pgs: 129 active+clean; 133 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 402 KiB/s rd, 1.0 MiB/s wr, 104 op/s 2024-03-31T23:23:31.218 INFO:tasks.workunit.client.0.smithi046.stdout: 1 230672 1.84 MB/sec execute 517 sec latency 729.070 ms 2024-03-31T23:23:32.218 INFO:tasks.workunit.client.0.smithi046.stdout: 1 230672 1.84 MB/sec execute 518 sec latency 1729.159 ms 2024-03-31T23:23:32.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:23:32 smithi142 ceph-mon[35271]: pgmap v369: 129 pgs: 129 active+clean; 133 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 279 KiB/s rd, 738 KiB/s wr, 77 op/s 2024-03-31T23:23:33.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:23:32 smithi162 ceph-mon[36237]: pgmap v369: 129 pgs: 129 active+clean; 133 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 279 KiB/s rd, 738 KiB/s wr, 77 op/s 2024-03-31T23:23:33.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:23:32 smithi046 ceph-mon[32498]: pgmap v369: 129 pgs: 129 active+clean; 133 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 279 KiB/s rd, 738 KiB/s wr, 77 op/s 2024-03-31T23:23:33.218 INFO:tasks.workunit.client.0.smithi046.stdout: 1 230739 1.84 MB/sec execute 519 sec latency 2258.027 ms 2024-03-31T23:23:34.218 INFO:tasks.workunit.client.0.smithi046.stdout: 1 230739 1.83 MB/sec execute 520 sec latency 1227.309 ms 2024-03-31T23:23:34.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:23:34 smithi142 ceph-mon[35271]: pgmap v370: 129 pgs: 129 active+clean; 133 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 279 KiB/s rd, 738 KiB/s wr, 77 op/s 2024-03-31T23:23:35.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:23:34 smithi162 ceph-mon[36237]: pgmap v370: 129 pgs: 129 active+clean; 133 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 279 KiB/s rd, 738 KiB/s wr, 77 op/s 2024-03-31T23:23:35.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:23:34 smithi046 ceph-mon[32498]: pgmap v370: 129 pgs: 129 active+clean; 133 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 279 KiB/s rd, 738 KiB/s wr, 77 op/s 2024-03-31T23:23:35.218 INFO:tasks.workunit.client.0.smithi046.stdout: 1 230739 1.83 MB/sec execute 521 sec latency 2227.411 ms 2024-03-31T23:23:36.218 INFO:tasks.workunit.client.0.smithi046.stdout: 1 230739 1.83 MB/sec execute 522 sec latency 3227.515 ms 2024-03-31T23:23:36.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:23:36 smithi142 ceph-mon[35271]: pgmap v371: 129 pgs: 129 active+clean; 135 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 398 KiB/s rd, 1.0 MiB/s wr, 103 op/s 2024-03-31T23:23:37.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:23:36 smithi162 ceph-mon[36237]: pgmap v371: 129 pgs: 129 active+clean; 135 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 398 KiB/s rd, 1.0 MiB/s wr, 103 op/s 2024-03-31T23:23:37.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:23:36 smithi046 ceph-mon[32498]: pgmap v371: 129 pgs: 129 active+clean; 135 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 398 KiB/s rd, 1.0 MiB/s wr, 103 op/s 2024-03-31T23:23:37.132 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:37.129+0000 7f3032ca4640 1 -- 172.21.15.46:0/2678755479 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f302c1008a0 msgr2=0x7f302c100d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:23:37.133 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:37.129+0000 7f3032ca4640 1 --2- 172.21.15.46:0/2678755479 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f302c1008a0 0x7f302c100d00 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7f3018009900 tx=0x7f301802bc30 comp rx=0 tx=0).stop 2024-03-31T23:23:37.133 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:37.129+0000 7f3032ca4640 1 -- 172.21.15.46:0/2678755479 shutdown_connections 2024-03-31T23:23:37.133 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:37.129+0000 7f3032ca4640 1 --2- 172.21.15.46:0/2678755479 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f302c101240 0x7f302c10d480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:23:37.133 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:37.129+0000 7f3032ca4640 1 --2- 172.21.15.46:0/2678755479 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f302c1008a0 0x7f302c100d00 unknown :-1 s=CLOSED pgs=841 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:23:37.133 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:37.129+0000 7f3032ca4640 1 --2- 172.21.15.46:0/2678755479 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f302c1055d0 0x7f302c1059b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:23:37.133 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:37.129+0000 7f3032ca4640 1 -- 172.21.15.46:0/2678755479 >> 172.21.15.46:0/2678755479 conn(0x7f302c077420 msgr2=0x7f302c077830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:23:37.133 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:37.129+0000 7f3032ca4640 1 -- 172.21.15.46:0/2678755479 shutdown_connections 2024-03-31T23:23:37.133 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:37.129+0000 7f3032ca4640 1 -- 172.21.15.46:0/2678755479 wait complete. 2024-03-31T23:23:37.133 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:37.130+0000 7f3032ca4640 1 Processor -- start 2024-03-31T23:23:37.133 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:37.130+0000 7f3032ca4640 1 -- start start 2024-03-31T23:23:37.133 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:37.130+0000 7f3032ca4640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f302c1008a0 0x7f302c198000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:23:37.133 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:37.130+0000 7f3032ca4640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f302c101240 0x7f302c198540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:23:37.133 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:37.130+0000 7f3032ca4640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f302c1055d0 0x7f302c19c870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:23:37.134 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:37.130+0000 7f3032ca4640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f302c110950 con 0x7f302c1055d0 2024-03-31T23:23:37.134 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:37.130+0000 7f3032ca4640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f302c1107d0 con 0x7f302c101240 2024-03-31T23:23:37.134 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:37.130+0000 7f3032ca4640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f302c110ad0 con 0x7f302c1008a0 2024-03-31T23:23:37.134 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:37.130+0000 7f3031241640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f302c1055d0 0x7f302c19c870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:23:37.134 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:37.130+0000 7f3031241640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f302c1055d0 0x7f302c19c870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:56162/0 (socket says 172.21.15.46:56162) 2024-03-31T23:23:37.134 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:37.130+0000 7f3031241640 1 -- 172.21.15.46:0/601928763 learned_addr learned my addr 172.21.15.46:0/601928763 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:23:37.134 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:37.130+0000 7f3030a40640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f302c1008a0 0x7f302c198000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:23:37.134 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:37.130+0000 7f302bfff640 1 --2- 172.21.15.46:0/601928763 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f302c101240 0x7f302c198540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:23:37.134 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:37.131+0000 7f3031241640 1 -- 172.21.15.46:0/601928763 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f302c1008a0 msgr2=0x7f302c198000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:23:37.134 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:37.131+0000 7f3031241640 1 --2- 172.21.15.46:0/601928763 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f302c1008a0 0x7f302c198000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:23:37.134 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:37.131+0000 7f3031241640 1 -- 172.21.15.46:0/601928763 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f302c101240 msgr2=0x7f302c198540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:23:37.134 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:37.131+0000 7f3031241640 1 --2- 172.21.15.46:0/601928763 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f302c101240 0x7f302c198540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:23:37.135 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:37.131+0000 7f3031241640 1 -- 172.21.15.46:0/601928763 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f302c19cfa0 con 0x7f302c1055d0 2024-03-31T23:23:37.135 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:37.131+0000 7f3031241640 1 --2- 172.21.15.46:0/601928763 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f302c1055d0 0x7f302c19c870 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7f302400eb10 tx=0x7f302400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:23:37.135 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:37.131+0000 7f3029ffb640 1 -- 172.21.15.46:0/601928763 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3024004b20 con 0x7f302c1055d0 2024-03-31T23:23:37.135 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:37.131+0000 7f3029ffb640 1 -- 172.21.15.46:0/601928763 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3024004cc0 con 0x7f302c1055d0 2024-03-31T23:23:37.135 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:37.131+0000 7f3032ca4640 1 -- 172.21.15.46:0/601928763 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f302c19d290 con 0x7f302c1055d0 2024-03-31T23:23:37.135 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:37.131+0000 7f3029ffb640 1 -- 172.21.15.46:0/601928763 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3024010690 con 0x7f302c1055d0 2024-03-31T23:23:37.135 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:37.131+0000 7f3032ca4640 1 -- 172.21.15.46:0/601928763 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f302c1a4b20 con 0x7f302c1055d0 2024-03-31T23:23:37.142 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:37.133+0000 7f3029ffb640 1 -- 172.21.15.46:0/601928763 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 25) ==== 137451+0+0 (secure 0 0 0) 0x7f3024005020 con 0x7f302c1055d0 2024-03-31T23:23:37.142 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:37.134+0000 7f3029ffb640 1 --2- 172.21.15.46:0/601928763 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f301c0a26b0 0x7f301c0a4b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:23:37.142 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:37.134+0000 7f3035964740 1 Processor -- start 2024-03-31T23:23:37.142 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:37.134+0000 7f3035964740 1 -- start start 2024-03-31T23:23:37.142 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:37.134+0000 7f3029ffb640 1 -- 172.21.15.46:0/601928763 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 9554+0+0 (secure 0 0 0) 0x7f30240cf530 con 0x7f302c1055d0 2024-03-31T23:23:37.142 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:37.134+0000 7f3030a40640 1 --2- 172.21.15.46:0/601928763 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f301c0a26b0 0x7f301c0a4b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:23:37.142 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:37.134+0000 7f3035964740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558e41be46c0 0x558e41be4aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:23:37.142 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:37.134+0000 7f3035964740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558e41be4fe0 0x558e41bed870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:23:37.142 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:37.134+0000 7f3035964740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558e41beddb0 0x558e41bf01a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:23:37.142 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:37.134+0000 7f3035964740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x558e41b478f0 con 0x558e41be46c0 2024-03-31T23:23:37.143 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:37.134+0000 7f3035964740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x558e41af08d0 con 0x558e41be4fe0 2024-03-31T23:23:37.143 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:37.134+0000 7f3035964740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x558e41b43990 con 0x558e41beddb0 2024-03-31T23:23:37.143 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:37.134+0000 7f3030a40640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558e41be46c0 0x558e41be4aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:23:37.143 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:37.135+0000 7f3030a40640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558e41be46c0 0x558e41be4aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:56166/0 (socket says 172.21.15.46:56166) 2024-03-31T23:23:37.143 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:37.135+0000 7f3030a40640 1 -- 172.21.15.46:0/717895065 learned_addr learned my addr 172.21.15.46:0/717895065 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:23:37.143 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:37.135+0000 7f302bfff640 1 --2- 172.21.15.46:0/717895065 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558e41be4fe0 0x558e41bed870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:23:37.143 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:37.135+0000 7f3031241640 1 --2- 172.21.15.46:0/717895065 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558e41beddb0 0x558e41bf01a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:23:37.143 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:37.135+0000 7f302bfff640 1 -- 172.21.15.46:0/717895065 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558e41beddb0 msgr2=0x558e41bf01a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:23:37.143 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:37.135+0000 7f302bfff640 1 --2- 172.21.15.46:0/717895065 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558e41beddb0 0x558e41bf01a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:23:37.143 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:37.135+0000 7f3030a40640 1 --2- 172.21.15.46:0/601928763 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f301c0a26b0 0x7f301c0a4b70 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f30200134b0 tx=0x7f30200138b0 comp rx=0 tx=0).ready entity=mgr.15654 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:23:37.143 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:37.135+0000 7f302bfff640 1 -- 172.21.15.46:0/717895065 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558e41be46c0 msgr2=0x558e41be4aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:23:37.143 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:37.135+0000 7f302bfff640 1 --2- 172.21.15.46:0/717895065 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558e41be46c0 0x558e41be4aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:23:37.143 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:37.135+0000 7f302bfff640 1 -- 172.21.15.46:0/717895065 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558e4185c210 con 0x558e41be4fe0 2024-03-31T23:23:37.144 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:37.135+0000 7f3031241640 1 --2- 172.21.15.46:0/717895065 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558e41beddb0 0x558e41bf01a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:23:37.144 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:37.135+0000 7f3030a40640 1 --2- 172.21.15.46:0/717895065 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558e41be46c0 0x558e41be4aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:23:37.144 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:37.135+0000 7f302bfff640 1 --2- 172.21.15.46:0/717895065 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558e41be4fe0 0x558e41bed870 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f3018000c00 tx=0x7f30180095f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=85511e848c12f418 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:23:37.144 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:37.135+0000 7f3016ffd640 1 -- 172.21.15.46:0/717895065 <== mon.2 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3018045020 con 0x558e41be4fe0 2024-03-31T23:23:37.144 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:37.135+0000 7f3016ffd640 1 -- 172.21.15.46:0/717895065 <== mon.2 v2:172.21.15.142:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f301802c7e0 con 0x558e41be4fe0 2024-03-31T23:23:37.144 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:37.135+0000 7f3016ffd640 1 -- 172.21.15.46:0/717895065 <== mon.2 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3018005500 con 0x558e41be4fe0 2024-03-31T23:23:37.144 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:37.136+0000 7f3035964740 1 -- 172.21.15.46:0/717895065 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558e41be4fe0 msgr2=0x558e41bed870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:23:37.144 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:37.136+0000 7f3035964740 1 --2- 172.21.15.46:0/717895065 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558e41be4fe0 0x558e41bed870 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f3018000c00 tx=0x7f30180095f0 comp rx=0 tx=0).stop 2024-03-31T23:23:37.144 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:37.136+0000 7f3035964740 1 -- 172.21.15.46:0/717895065 shutdown_connections 2024-03-31T23:23:37.144 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:37.136+0000 7f3035964740 1 --2- 172.21.15.46:0/717895065 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558e41beddb0 0x558e41bf01a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:23:37.144 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:37.136+0000 7f3035964740 1 --2- 172.21.15.46:0/717895065 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558e41be4fe0 0x558e41bed870 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:23:37.144 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:37.136+0000 7f3035964740 1 --2- 172.21.15.46:0/717895065 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558e41be46c0 0x558e41be4aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:23:37.144 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:37.136+0000 7f3035964740 1 -- 172.21.15.46:0/717895065 >> 172.21.15.46:0/717895065 conn(0x558e41bdefe0 msgr2=0x558e41bdf3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:23:37.144 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:37.136+0000 7f3035964740 1 -- 172.21.15.46:0/717895065 shutdown_connections 2024-03-31T23:23:37.144 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:37.136+0000 7f3035964740 1 -- 172.21.15.46:0/717895065 wait complete. 2024-03-31T23:23:37.145 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:37.136+0000 7f3035964740 1 -- start start 2024-03-31T23:23:37.145 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:37.137+0000 7f3035964740 1 Processor -- start 2024-03-31T23:23:37.145 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:37.137+0000 7f3035964740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558e41be46c0 0x558e41bead90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:23:37.145 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:37.137+0000 7f3035964740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558e41be4fe0 0x558e41beb2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:23:37.145 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:37.137+0000 7f3035964740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558e41beddb0 0x558e41beb810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:23:37.145 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:37.137+0000 7f3035964740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x558e41bf0980 con 0x558e41be4fe0 2024-03-31T23:23:37.145 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:37.137+0000 7f3035964740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x558e41ae5f80 con 0x558e41be46c0 2024-03-31T23:23:37.145 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:37.137+0000 7f3035964740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x558e41bf0b00 con 0x558e41beddb0 2024-03-31T23:23:37.145 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:37.137+0000 7f3035964740 10 client.0 _ll_register_callbacks cb 0x558e41b3e290 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T23:23:37.145 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:37.137+0000 7f302bfff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558e41be4fe0 0x558e41beb2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:23:37.145 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:37.137+0000 7f302bfff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558e41be4fe0 0x558e41beb2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:56168/0 (socket says 172.21.15.46:56168) 2024-03-31T23:23:37.145 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:37.137+0000 7f3031241640 1 --2- 172.21.15.46:0/672323042 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558e41beddb0 0x558e41beb810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:23:37.145 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:37.137+0000 7f302bfff640 1 -- 172.21.15.46:0/672323042 learned_addr learned my addr 172.21.15.46:0/672323042 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:23:37.145 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:37.137+0000 7f3030a40640 1 --2- 172.21.15.46:0/672323042 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558e41be46c0 0x558e41bead90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:23:37.145 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:37.137+0000 7f302bfff640 1 -- 172.21.15.46:0/672323042 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558e41beddb0 msgr2=0x558e41beb810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:23:37.146 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:37.137+0000 7f302bfff640 1 --2- 172.21.15.46:0/672323042 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558e41beddb0 0x558e41beb810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:23:37.146 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:37.137+0000 7f302bfff640 1 -- 172.21.15.46:0/672323042 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558e41be46c0 msgr2=0x558e41bead90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:23:37.146 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:37.137+0000 7f302bfff640 1 --2- 172.21.15.46:0/672323042 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558e41be46c0 0x558e41bead90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:23:37.146 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:37.137+0000 7f302bfff640 1 -- 172.21.15.46:0/672323042 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558e41928620 con 0x558e41be4fe0 2024-03-31T23:23:37.146 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:37.137+0000 7f3031241640 1 --2- 172.21.15.46:0/672323042 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558e41beddb0 0x558e41beb810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:23:37.146 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:37.138+0000 7f302bfff640 1 --2- 172.21.15.46:0/672323042 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558e41be4fe0 0x558e41beb2d0 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7f30180055e0 tx=0x7f3018009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ab820fb321afabb server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:23:37.146 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:37.138+0000 7f300bfff640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-31T23:23:37.146 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:37.138+0000 7f300bfff640 1 -- 172.21.15.46:0/672323042 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f301804f060 con 0x558e41be4fe0 2024-03-31T23:23:37.146 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:37.138+0000 7f300bfff640 1 -- 172.21.15.46:0/672323042 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(108..108 src has 1..108) ==== 9554+0+0 (secure 0 0 0) 0x7f3018004460 con 0x558e41be4fe0 2024-03-31T23:23:37.146 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:37.138+0000 7f3035964740 1 -- 172.21.15.46:0/672323042 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f301802c7e0 con 0x558e41be4fe0 2024-03-31T23:23:37.146 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:37.138+0000 7f300bfff640 1 -- 172.21.15.46:0/672323042 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f3018056050 con 0x558e41be4fe0 2024-03-31T23:23:37.146 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:37.138+0000 7f3035964740 10 client.17118 fetch_fsmap learned FSMap version 36 2024-03-31T23:23:37.146 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:37.138+0000 7f3035964740 1 -- 172.21.15.46:0/672323042 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x558e41bfc800 con 0x558e41be4fe0 2024-03-31T23:23:37.146 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:37.138+0000 7f300bfff640 1 -- 172.21.15.46:0/672323042 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f301802c7e0 con 0x558e41be4fe0 2024-03-31T23:23:37.147 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:37.138+0000 7f300bfff640 1 -- 172.21.15.46:0/672323042 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 36) ==== 8362+0+0 (secure 0 0 0) 0x7f3018003770 con 0x558e41be4fe0 2024-03-31T23:23:37.147 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:37.138+0000 7f3035964740 10 client.17118 fetch_fsmap finished waiting for FSMap version 36 2024-03-31T23:23:37.147 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:37.138+0000 7f3035964740 10 client.17118 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-31T23:23:37.147 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:37.138+0000 7f3035964740 20 client.17118 populate_metadata read hostname 'smithi046' 2024-03-31T23:23:37.147 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:37.138+0000 7f3035964740 1 --2- 172.21.15.46:0/672323042 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x558e41bfdb50 0x558e41bfff40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:23:37.147 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:37.139+0000 7f3035964740 4 client.17118 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-31T23:23:37.147 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:37.139+0000 7f3035964740 1 -- 172.21.15.46:0/672323042 --> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558e41b5b120 con 0x558e41bfdb50 2024-03-31T23:23:37.147 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:37.139+0000 7f3030a40640 1 --2- 172.21.15.46:0/672323042 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x558e41bfdb50 0x558e41bfff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:23:37.147 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:37.139+0000 7f3030a40640 1 --2- 172.21.15.46:0/672323042 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x558e41bfdb50 0x558e41bfff40 crc :-1 s=READY pgs=181 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c42e8f7988e3ff58 server_cookie=dc2b6780da86609e in_seq=0 out_seq=0 2024-03-31T23:23:37.147 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:37.139+0000 7f300bfff640 10 client.17118 ms_handle_connect on v2:172.21.15.142:6838/2519143918 2024-03-31T23:23:37.147 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:37.140+0000 7f300bfff640 1 -- 172.21.15.46:0/672323042 <== mds.0 v2:172.21.15.142:6838/2519143918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x558e41b5b120 con 0x558e41bfdb50 2024-03-31T23:23:37.147 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:37.140+0000 7f300bfff640 10 client.17118 handle_command_reply: tid=0 2024-03-31T23:23:37.147 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:37.140+0000 7f3035964740 1 client.17118 shutdown 2024-03-31T23:23:37.147 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:37.140+0000 7f3035964740 1 -- 172.21.15.46:0/672323042 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558e41be4fe0 msgr2=0x558e41beb2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:23:37.148 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:37.140+0000 7f3035964740 1 --2- 172.21.15.46:0/672323042 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558e41be4fe0 0x558e41beb2d0 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7f30180055e0 tx=0x7f3018009260 comp rx=0 tx=0).stop 2024-03-31T23:23:37.148 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:37.140+0000 7f3035964740 1 -- 172.21.15.46:0/672323042 shutdown_connections 2024-03-31T23:23:37.148 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:37.140+0000 7f3035964740 1 --2- 172.21.15.46:0/672323042 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x558e41bfdb50 0x558e41bfff40 crc :-1 s=READY pgs=181 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:23:37.148 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:37.140+0000 7f3035964740 1 --2- 172.21.15.46:0/672323042 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558e41beddb0 0x558e41beb810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:23:37.148 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:37.140+0000 7f300bfff640 0 client.17118 ms_handle_reset on v2:172.21.15.142:6838/2519143918 2024-03-31T23:23:37.148 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:37.140+0000 7f3035964740 1 --2- 172.21.15.46:0/672323042 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558e41be4fe0 0x558e41beb2d0 unknown :-1 s=CLOSED pgs=843 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:23:37.148 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:37.140+0000 7f3035964740 1 --2- 172.21.15.46:0/672323042 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558e41be46c0 0x558e41bead90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:23:37.148 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:37.140+0000 7f3035964740 1 -- 172.21.15.46:0/672323042 >> 172.21.15.46:0/672323042 conn(0x558e41bdefe0 msgr2=0x558e41be0d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:23:37.148 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:37.140+0000 7f3035964740 1 -- 172.21.15.46:0/672323042 shutdown_connections 2024-03-31T23:23:37.148 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:37.140+0000 7f3035964740 1 -- 172.21.15.46:0/672323042 wait complete. 2024-03-31T23:23:37.148 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:37.140+0000 7f3035964740 20 client.17118 trim_cache size 0 max 16384 2024-03-31T23:23:37.162 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:37.155+0000 7f3035964740 1 Processor -- start 2024-03-31T23:23:37.162 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:37.155+0000 7f3035964740 1 -- start start 2024-03-31T23:23:37.162 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:37.156+0000 7f3035964740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558e41be46c0 0x558e41bf9720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:23:37.162 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:37.156+0000 7f3035964740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558e41be4fe0 0x558e41c055d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:23:37.162 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:37.156+0000 7f3035964740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558e41beddb0 0x558e41bec4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:23:37.162 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:37.156+0000 7f3035964740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f302c1b1460 con 0x558e41beddb0 2024-03-31T23:23:37.162 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:37.156+0000 7f3035964740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f302c1aecd0 con 0x558e41be4fe0 2024-03-31T23:23:37.162 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:37.156+0000 7f3035964740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f302c1acb40 con 0x558e41be46c0 2024-03-31T23:23:37.162 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:37.156+0000 7f3031241640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558e41beddb0 0x558e41bec4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:23:37.162 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:37.156+0000 7f3031241640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558e41beddb0 0x558e41bec4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:56172/0 (socket says 172.21.15.46:56172) 2024-03-31T23:23:37.162 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:37.156+0000 7f3031241640 1 -- 172.21.15.46:0/2654124277 learned_addr learned my addr 172.21.15.46:0/2654124277 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:23:37.163 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:37.156+0000 7f302bfff640 1 --2- 172.21.15.46:0/2654124277 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558e41be4fe0 0x558e41c055d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:23:37.163 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:37.156+0000 7f3030a40640 1 --2- 172.21.15.46:0/2654124277 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558e41be46c0 0x558e41bf9720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:23:37.163 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:37.156+0000 7f3031241640 1 -- 172.21.15.46:0/2654124277 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558e41be46c0 msgr2=0x558e41bf9720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:23:37.163 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:37.156+0000 7f3031241640 1 --2- 172.21.15.46:0/2654124277 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558e41be46c0 0x558e41bf9720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:23:37.163 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:37.156+0000 7f3031241640 1 -- 172.21.15.46:0/2654124277 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558e41be4fe0 msgr2=0x558e41c055d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:23:37.163 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:37.156+0000 7f3031241640 1 --2- 172.21.15.46:0/2654124277 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558e41be4fe0 0x558e41c055d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:23:37.163 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:37.156+0000 7f3031241640 1 -- 172.21.15.46:0/2654124277 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558e41bf9c60 con 0x558e41beddb0 2024-03-31T23:23:37.163 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:37.156+0000 7f3030a40640 1 --2- 172.21.15.46:0/2654124277 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558e41be46c0 0x558e41bf9720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T23:23:37.163 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:37.156+0000 7f3031241640 1 --2- 172.21.15.46:0/2654124277 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558e41beddb0 0x558e41bec4b0 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7f30240049d0 tx=0x7f3024086040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a2b4dae4944ce7a3 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:23:37.163 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:37.156+0000 7f30167fc640 1 -- 172.21.15.46:0/2654124277 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30240a2f50 con 0x558e41beddb0 2024-03-31T23:23:37.163 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:37.156+0000 7f30167fc640 1 -- 172.21.15.46:0/2654124277 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f302409bf40 con 0x558e41beddb0 2024-03-31T23:23:37.163 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:37.156+0000 7f30167fc640 1 -- 172.21.15.46:0/2654124277 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3024097ae0 con 0x558e41beddb0 2024-03-31T23:23:37.163 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:37.157+0000 7f3035964740 1 -- 172.21.15.46:0/2654124277 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558e41beddb0 msgr2=0x558e41bec4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:23:37.163 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:37.157+0000 7f3035964740 1 --2- 172.21.15.46:0/2654124277 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558e41beddb0 0x558e41bec4b0 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7f30240049d0 tx=0x7f3024086040 comp rx=0 tx=0).stop 2024-03-31T23:23:37.164 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:37.157+0000 7f3035964740 1 -- 172.21.15.46:0/2654124277 shutdown_connections 2024-03-31T23:23:37.164 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:37.157+0000 7f3035964740 1 --2- 172.21.15.46:0/2654124277 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558e41beddb0 0x558e41bec4b0 unknown :-1 s=CLOSED pgs=844 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:23:37.164 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:37.157+0000 7f3035964740 1 --2- 172.21.15.46:0/2654124277 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558e41be4fe0 0x558e41c055d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:23:37.164 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:37.157+0000 7f3035964740 1 --2- 172.21.15.46:0/2654124277 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558e41be46c0 0x558e41bf9720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:23:37.164 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:37.157+0000 7f3035964740 1 -- 172.21.15.46:0/2654124277 >> 172.21.15.46:0/2654124277 conn(0x558e41bdefe0 msgr2=0x558e41be5540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:23:37.164 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:37.157+0000 7f3035964740 1 -- 172.21.15.46:0/2654124277 shutdown_connections 2024-03-31T23:23:37.164 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:37.157+0000 7f3035964740 1 -- 172.21.15.46:0/2654124277 wait complete. 2024-03-31T23:23:37.164 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:37.157+0000 7f3035964740 1 -- start start 2024-03-31T23:23:37.164 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:37.157+0000 7f3035964740 1 Processor -- start 2024-03-31T23:23:37.164 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:37.158+0000 7f3035964740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558e41be46c0 0x558e41bf5d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:23:37.164 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:37.158+0000 7f3035964740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558e41be4fe0 0x558e41bf2d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:23:37.164 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:37.158+0000 7f3035964740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558e41beddb0 0x558e41bf3260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:23:37.164 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:37.158+0000 7f3035964740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f302c1ad190 con 0x558e41be46c0 2024-03-31T23:23:37.164 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:37.158+0000 7f3035964740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f302c1ae600 con 0x558e41be4fe0 2024-03-31T23:23:37.165 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:37.158+0000 7f3035964740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f302c1ad860 con 0x558e41beddb0 2024-03-31T23:23:37.165 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:37.158+0000 7f3035964740 10 client.0 _ll_register_callbacks cb 0x7f302000f540 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T23:23:37.165 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:37.158+0000 7f3030a40640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558e41be46c0 0x558e41bf5d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:23:37.165 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:37.158+0000 7f3030a40640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558e41be46c0 0x558e41bf5d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:56186/0 (socket says 172.21.15.46:56186) 2024-03-31T23:23:37.165 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:37.158+0000 7f3030a40640 1 -- 172.21.15.46:0/1958916109 learned_addr learned my addr 172.21.15.46:0/1958916109 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:23:37.165 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:37.158+0000 7f302bfff640 1 --2- 172.21.15.46:0/1958916109 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558e41be4fe0 0x558e41bf2d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:23:37.165 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:37.158+0000 7f3031241640 1 --2- 172.21.15.46:0/1958916109 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558e41beddb0 0x558e41bf3260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:23:37.165 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:37.158+0000 7f3030a40640 1 -- 172.21.15.46:0/1958916109 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558e41beddb0 msgr2=0x558e41bf3260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:23:37.165 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:37.158+0000 7f3030a40640 1 --2- 172.21.15.46:0/1958916109 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558e41beddb0 0x558e41bf3260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:23:37.165 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:37.158+0000 7f3030a40640 1 -- 172.21.15.46:0/1958916109 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558e41be4fe0 msgr2=0x558e41bf2d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:23:37.165 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:37.158+0000 7f3030a40640 1 --2- 172.21.15.46:0/1958916109 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558e41be4fe0 0x558e41bf2d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:23:37.165 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:37.158+0000 7f3030a40640 1 -- 172.21.15.46:0/1958916109 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558e41bf37a0 con 0x558e41be46c0 2024-03-31T23:23:37.165 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:37.158+0000 7f302bfff640 1 --2- 172.21.15.46:0/1958916109 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558e41be4fe0 0x558e41bf2d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:23:37.165 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:37.158+0000 7f3030a40640 1 --2- 172.21.15.46:0/1958916109 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558e41be46c0 0x558e41bf5d50 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7f3020012270 tx=0x7f302001b000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=887d84b42f32cf94 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:23:37.165 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:37.158+0000 7f300bfff640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-31T23:23:37.166 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:37.159+0000 7f300bfff640 1 -- 172.21.15.46:0/1958916109 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3020030060 con 0x558e41be46c0 2024-03-31T23:23:37.166 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:37.159+0000 7f300bfff640 1 -- 172.21.15.46:0/1958916109 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(108..108 src has 1..108) ==== 9554+0+0 (secure 0 0 0) 0x7f302002a5a0 con 0x558e41be46c0 2024-03-31T23:23:37.166 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:37.159+0000 7f3035964740 1 -- 172.21.15.46:0/1958916109 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f302409bf40 con 0x558e41be46c0 2024-03-31T23:23:37.166 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:37.159+0000 7f300bfff640 1 -- 172.21.15.46:0/1958916109 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f3020037050 con 0x558e41be46c0 2024-03-31T23:23:37.166 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:37.159+0000 7f3035964740 10 client.17124 fetch_fsmap learned FSMap version 36 2024-03-31T23:23:37.166 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:37.159+0000 7f3035964740 1 -- 172.21.15.46:0/1958916109 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x558e41c648e0 con 0x558e41be46c0 2024-03-31T23:23:37.166 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:37.159+0000 7f300bfff640 1 -- 172.21.15.46:0/1958916109 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f302409bf40 con 0x558e41be46c0 2024-03-31T23:23:37.166 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:37.159+0000 7f300bfff640 1 -- 172.21.15.46:0/1958916109 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 36) ==== 8362+0+0 (secure 0 0 0) 0x7f302002a070 con 0x558e41be46c0 2024-03-31T23:23:37.166 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:37.159+0000 7f3035964740 10 client.17124 fetch_fsmap finished waiting for FSMap version 36 2024-03-31T23:23:37.166 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:37.159+0000 7f3035964740 10 client.17124 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-31T23:23:37.166 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:37.159+0000 7f3035964740 20 client.17124 populate_metadata read hostname 'smithi046' 2024-03-31T23:23:37.166 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:37.159+0000 7f3035964740 1 --2- 172.21.15.46:0/1958916109 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x558e41c65010 0x558e41c67430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:23:37.167 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:37.159+0000 7f3035964740 4 client.17124 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-31T23:23:37.167 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:37.159+0000 7f3035964740 1 -- 172.21.15.46:0/1958916109 --> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] -- command(tid 0: {"prefix": "flush journal"}) -- 0x558e41b54e30 con 0x558e41c65010 2024-03-31T23:23:37.167 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:37.159+0000 7f302bfff640 1 --2- 172.21.15.46:0/1958916109 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x558e41c65010 0x558e41c67430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:23:37.167 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:37.160+0000 7f302bfff640 1 --2- 172.21.15.46:0/1958916109 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x558e41c65010 0x558e41c67430 crc :-1 s=READY pgs=182 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=31a15c316ad2b104 server_cookie=b17f4593ef679c0a in_seq=0 out_seq=0 2024-03-31T23:23:37.167 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:37.160+0000 7f300bfff640 10 client.17124 ms_handle_connect on v2:172.21.15.142:6838/2519143918 2024-03-31T23:23:37.188 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:37.186+0000 7f300bfff640 1 -- 172.21.15.46:0/1958916109 <== mds.0 v2:172.21.15.142:6838/2519143918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x558e41b54e30 con 0x558e41c65010 2024-03-31T23:23:37.188 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:37.186+0000 7f300bfff640 10 client.17124 handle_command_reply: tid=0 2024-03-31T23:23:37.188 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:37.186+0000 7f3035964740 1 client.17124 shutdown 2024-03-31T23:23:37.188 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:37.186+0000 7f3035964740 1 -- 172.21.15.46:0/1958916109 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558e41be46c0 msgr2=0x558e41bf5d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:23:37.188 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:37.186+0000 7f3035964740 1 --2- 172.21.15.46:0/1958916109 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558e41be46c0 0x558e41bf5d50 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7f3020012270 tx=0x7f302001b000 comp rx=0 tx=0).stop 2024-03-31T23:23:37.188 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:37.186+0000 7f3035964740 1 -- 172.21.15.46:0/1958916109 shutdown_connections 2024-03-31T23:23:37.189 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:37.186+0000 7f3035964740 1 --2- 172.21.15.46:0/1958916109 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x558e41c65010 0x558e41c67430 crc :-1 s=READY pgs=182 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:23:37.189 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:37.186+0000 7f3035964740 1 --2- 172.21.15.46:0/1958916109 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558e41beddb0 0x558e41bf3260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:23:37.189 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:37.186+0000 7f3035964740 1 --2- 172.21.15.46:0/1958916109 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558e41be4fe0 0x558e41bf2d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:23:37.189 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:37.186+0000 7f300bfff640 0 client.17124 ms_handle_reset on v2:172.21.15.142:6838/2519143918 2024-03-31T23:23:37.189 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:37.186+0000 7f3035964740 1 --2- 172.21.15.46:0/1958916109 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558e41be46c0 0x558e41bf5d50 unknown :-1 s=CLOSED pgs=845 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:23:37.189 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:37.186+0000 7f3035964740 1 -- 172.21.15.46:0/1958916109 >> 172.21.15.46:0/1958916109 conn(0x558e41bdefe0 msgr2=0x558e41bdd6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:23:37.189 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:37.186+0000 7f3035964740 1 -- 172.21.15.46:0/1958916109 shutdown_connections 2024-03-31T23:23:37.189 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:37.186+0000 7f3035964740 1 -- 172.21.15.46:0/1958916109 wait complete. 2024-03-31T23:23:37.189 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:37.186+0000 7f3035964740 20 client.17124 trim_cache size 0 max 16384 2024-03-31T23:23:37.189 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout:{ 2024-03-31T23:23:37.190 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout: "message": "", 2024-03-31T23:23:37.190 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout: "return_code": 0 2024-03-31T23:23:37.190 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout:} 2024-03-31T23:23:37.191 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:37.187+0000 7f3032ca4640 1 -- 172.21.15.46:0/601928763 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f301c0a26b0 msgr2=0x7f301c0a4b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:23:37.191 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:37.187+0000 7f3032ca4640 1 --2- 172.21.15.46:0/601928763 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f301c0a26b0 0x7f301c0a4b70 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f30200134b0 tx=0x7f30200138b0 comp rx=0 tx=0).stop 2024-03-31T23:23:37.191 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:37.187+0000 7f3032ca4640 1 -- 172.21.15.46:0/601928763 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f302c1055d0 msgr2=0x7f302c19c870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:23:37.191 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:37.187+0000 7f3032ca4640 1 --2- 172.21.15.46:0/601928763 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f302c1055d0 0x7f302c19c870 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7f302400eb10 tx=0x7f302400c2d0 comp rx=0 tx=0).stop 2024-03-31T23:23:37.191 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:37.187+0000 7f3032ca4640 1 -- 172.21.15.46:0/601928763 shutdown_connections 2024-03-31T23:23:37.191 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:37.187+0000 7f3032ca4640 1 --2- 172.21.15.46:0/601928763 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f301c0a26b0 0x7f301c0a4b70 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:23:37.191 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:37.187+0000 7f3032ca4640 1 --2- 172.21.15.46:0/601928763 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f302c1055d0 0x7f302c19c870 unknown :-1 s=CLOSED pgs=842 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:23:37.191 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:37.187+0000 7f3032ca4640 1 --2- 172.21.15.46:0/601928763 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f302c101240 0x7f302c198540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:23:37.191 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:37.187+0000 7f3032ca4640 1 --2- 172.21.15.46:0/601928763 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f302c1008a0 0x7f302c198000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:23:37.191 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:37.187+0000 7f3032ca4640 1 -- 172.21.15.46:0/601928763 >> 172.21.15.46:0/601928763 conn(0x7f302c077420 msgr2=0x7f302c06a330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:23:37.191 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:37.187+0000 7f3032ca4640 1 -- 172.21.15.46:0/601928763 shutdown_connections 2024-03-31T23:23:37.191 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:37.187+0000 7f3032ca4640 1 -- 172.21.15.46:0/601928763 wait complete. 2024-03-31T23:23:37.218 INFO:tasks.workunit.client.0.smithi046.stdout: 1 230739 1.82 MB/sec execute 523 sec latency 4227.609 ms 2024-03-31T23:23:38.219 INFO:tasks.workunit.client.0.smithi046.stdout: 1 230872 1.82 MB/sec execute 524 sec latency 4755.300 ms 2024-03-31T23:23:38.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:23:38 smithi142 ceph-mon[35271]: pgmap v372: 129 pgs: 129 active+clean; 135 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 364 KiB/s rd, 865 KiB/s wr, 86 op/s 2024-03-31T23:23:39.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:23:38 smithi162 ceph-mon[36237]: pgmap v372: 129 pgs: 129 active+clean; 135 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 364 KiB/s rd, 865 KiB/s wr, 86 op/s 2024-03-31T23:23:39.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:23:38 smithi046 ceph-mon[32498]: pgmap v372: 129 pgs: 129 active+clean; 135 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 364 KiB/s rd, 865 KiB/s wr, 86 op/s 2024-03-31T23:23:39.218 INFO:tasks.workunit.client.0.smithi046.stdout: 1 231142 1.82 MB/sec execute 525 sec latency 15.157 ms 2024-03-31T23:23:40.219 INFO:tasks.workunit.client.0.smithi046.stdout: 1 231543 1.82 MB/sec execute 526 sec latency 17.501 ms 2024-03-31T23:23:40.913 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:23:40 smithi046 ceph-mon[32498]: pgmap v373: 129 pgs: 129 active+clean; 136 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 411 KiB/s rd, 1.0 MiB/s wr, 107 op/s 2024-03-31T23:23:40.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:23:40 smithi142 ceph-mon[35271]: pgmap v373: 129 pgs: 129 active+clean; 136 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 411 KiB/s rd, 1.0 MiB/s wr, 107 op/s 2024-03-31T23:23:41.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:23:40 smithi162 ceph-mon[36237]: pgmap v373: 129 pgs: 129 active+clean; 136 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 411 KiB/s rd, 1.0 MiB/s wr, 107 op/s 2024-03-31T23:23:41.219 INFO:tasks.workunit.client.0.smithi046.stdout: 1 232057 1.82 MB/sec execute 527 sec latency 22.631 ms 2024-03-31T23:23:42.219 INFO:tasks.workunit.client.0.smithi046.stdout: 1 232520 1.82 MB/sec execute 528 sec latency 23.374 ms 2024-03-31T23:23:42.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:23:42 smithi142 ceph-mon[35271]: pgmap v374: 129 pgs: 129 active+clean; 136 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 165 KiB/s rd, 518 KiB/s wr, 47 op/s 2024-03-31T23:23:43.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:23:42 smithi162 ceph-mon[36237]: pgmap v374: 129 pgs: 129 active+clean; 136 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 165 KiB/s rd, 518 KiB/s wr, 47 op/s 2024-03-31T23:23:43.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:23:42 smithi046 ceph-mon[32498]: pgmap v374: 129 pgs: 129 active+clean; 136 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 165 KiB/s rd, 518 KiB/s wr, 47 op/s 2024-03-31T23:23:43.219 INFO:tasks.workunit.client.0.smithi046.stdout: 1 233038 1.82 MB/sec execute 529 sec latency 22.055 ms 2024-03-31T23:23:44.219 INFO:tasks.workunit.client.0.smithi046.stdout: 1 233772 1.83 MB/sec execute 530 sec latency 18.562 ms 2024-03-31T23:23:44.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:23:44 smithi142 ceph-mon[35271]: pgmap v375: 129 pgs: 129 active+clean; 136 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 165 KiB/s rd, 518 KiB/s wr, 47 op/s 2024-03-31T23:23:45.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:23:44 smithi162 ceph-mon[36237]: pgmap v375: 129 pgs: 129 active+clean; 136 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 165 KiB/s rd, 518 KiB/s wr, 47 op/s 2024-03-31T23:23:45.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:23:44 smithi046 ceph-mon[32498]: pgmap v375: 129 pgs: 129 active+clean; 136 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 165 KiB/s rd, 518 KiB/s wr, 47 op/s 2024-03-31T23:23:45.219 INFO:tasks.workunit.client.0.smithi046.stdout: 1 234124 1.83 MB/sec execute 531 sec latency 21.172 ms 2024-03-31T23:23:46.219 INFO:tasks.workunit.client.0.smithi046.stdout: 1 234220 1.83 MB/sec execute 532 sec latency 805.617 ms 2024-03-31T23:23:46.945 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:23:46 smithi142 ceph-mon[35271]: pgmap v376: 129 pgs: 129 active+clean; 137 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 393 KiB/s rd, 1.0 MiB/s wr, 103 op/s 2024-03-31T23:23:47.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:23:46 smithi162 ceph-mon[36237]: pgmap v376: 129 pgs: 129 active+clean; 137 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 393 KiB/s rd, 1.0 MiB/s wr, 103 op/s 2024-03-31T23:23:47.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:23:46 smithi046 ceph-mon[32498]: pgmap v376: 129 pgs: 129 active+clean; 137 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 393 KiB/s rd, 1.0 MiB/s wr, 103 op/s 2024-03-31T23:23:47.219 INFO:tasks.workunit.client.0.smithi046.stdout: 1 234220 1.83 MB/sec execute 533 sec latency 1805.656 ms 2024-03-31T23:23:48.219 INFO:tasks.workunit.client.0.smithi046.stdout: 1 234287 1.82 MB/sec execute 534 sec latency 2333.693 ms 2024-03-31T23:23:48.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:23:48 smithi142 ceph-mon[35271]: pgmap v377: 129 pgs: 129 active+clean; 137 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 274 KiB/s rd, 737 KiB/s wr, 77 op/s 2024-03-31T23:23:49.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:23:48 smithi162 ceph-mon[36237]: pgmap v377: 129 pgs: 129 active+clean; 137 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 274 KiB/s rd, 737 KiB/s wr, 77 op/s 2024-03-31T23:23:49.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:23:48 smithi046 ceph-mon[32498]: pgmap v377: 129 pgs: 129 active+clean; 137 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 274 KiB/s rd, 737 KiB/s wr, 77 op/s 2024-03-31T23:23:49.219 INFO:tasks.workunit.client.0.smithi046.stdout: 1 234287 1.82 MB/sec execute 535 sec latency 1228.550 ms 2024-03-31T23:23:50.219 INFO:tasks.workunit.client.0.smithi046.stdout: 1 234287 1.82 MB/sec execute 536 sec latency 2228.612 ms 2024-03-31T23:23:50.288 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:50.284+0000 7f352a014640 1 -- 172.21.15.46:0/3342866450 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3524107730 msgr2=0x7f3524107b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:23:50.289 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:50.284+0000 7f352a014640 1 --2- 172.21.15.46:0/3342866450 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3524107730 0x7f3524107b10 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7f3514009930 tx=0x7f351402bc30 comp rx=0 tx=0).stop 2024-03-31T23:23:50.289 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:50.284+0000 7f352a014640 1 -- 172.21.15.46:0/3342866450 shutdown_connections 2024-03-31T23:23:50.289 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:50.284+0000 7f352a014640 1 --2- 172.21.15.46:0/3342866450 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f35241020e0 0x7f352410e5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:23:50.289 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:50.284+0000 7f352a014640 1 --2- 172.21.15.46:0/3342866450 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3524101720 0x7f3524101ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:23:50.289 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:50.284+0000 7f352a014640 1 --2- 172.21.15.46:0/3342866450 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3524107730 0x7f3524107b10 unknown :-1 s=CLOSED pgs=846 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:23:50.289 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:50.284+0000 7f352a014640 1 -- 172.21.15.46:0/3342866450 >> 172.21.15.46:0/3342866450 conn(0x7f35240fd230 msgr2=0x7f35240ff650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:23:50.289 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:50.284+0000 7f352a014640 1 -- 172.21.15.46:0/3342866450 shutdown_connections 2024-03-31T23:23:50.289 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:50.284+0000 7f352a014640 1 -- 172.21.15.46:0/3342866450 wait complete. 2024-03-31T23:23:50.289 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:50.285+0000 7f352a014640 1 Processor -- start 2024-03-31T23:23:50.289 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:50.285+0000 7f352a014640 1 -- start start 2024-03-31T23:23:50.289 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:50.286+0000 7f352a014640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3524101720 0x7f3524197ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:23:50.290 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:50.286+0000 7f352a014640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f35241020e0 0x7f3524198410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:23:50.290 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:50.286+0000 7f352a014640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3524107730 0x7f352419c740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:23:50.290 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:50.286+0000 7f352a014640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f35241119b0 con 0x7f3524101720 2024-03-31T23:23:50.290 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:50.286+0000 7f352a014640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f3524111830 con 0x7f35241020e0 2024-03-31T23:23:50.290 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:50.286+0000 7f352a014640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f3524111b30 con 0x7f3524107730 2024-03-31T23:23:50.290 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:50.286+0000 7f35237fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3524101720 0x7f3524197ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:23:50.290 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:50.286+0000 7f35237fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3524101720 0x7f3524197ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:38064/0 (socket says 172.21.15.46:38064) 2024-03-31T23:23:50.290 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:50.286+0000 7f35237fe640 1 -- 172.21.15.46:0/1065165396 learned_addr learned my addr 172.21.15.46:0/1065165396 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:23:50.290 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:50.286+0000 7f3523fff640 1 --2- 172.21.15.46:0/1065165396 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3524107730 0x7f352419c740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:23:50.291 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:50.286+0000 7f351affd640 1 --2- 172.21.15.46:0/1065165396 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f35241020e0 0x7f3524198410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:23:50.291 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:50.286+0000 7f35237fe640 1 -- 172.21.15.46:0/1065165396 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3524107730 msgr2=0x7f352419c740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:23:50.291 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:50.286+0000 7f35237fe640 1 --2- 172.21.15.46:0/1065165396 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3524107730 0x7f352419c740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:23:50.291 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:50.286+0000 7f35237fe640 1 -- 172.21.15.46:0/1065165396 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f35241020e0 msgr2=0x7f3524198410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:23:50.291 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:50.286+0000 7f35237fe640 1 --2- 172.21.15.46:0/1065165396 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f35241020e0 0x7f3524198410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:23:50.291 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:50.286+0000 7f35237fe640 1 -- 172.21.15.46:0/1065165396 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f352419ce70 con 0x7f3524101720 2024-03-31T23:23:50.291 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:50.286+0000 7f35237fe640 1 --2- 172.21.15.46:0/1065165396 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3524101720 0x7f3524197ed0 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7f3514009230 tx=0x7f351402c5d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:23:50.291 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:50.286+0000 7f35217fa640 1 -- 172.21.15.46:0/1065165396 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3514043020 con 0x7f3524101720 2024-03-31T23:23:50.291 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:50.287+0000 7f352a014640 1 -- 172.21.15.46:0/1065165396 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f352419d100 con 0x7f3524101720 2024-03-31T23:23:50.291 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:50.287+0000 7f35217fa640 1 -- 172.21.15.46:0/1065165396 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f351402ca70 con 0x7f3524101720 2024-03-31T23:23:50.291 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:50.287+0000 7f35217fa640 1 -- 172.21.15.46:0/1065165396 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3514003a50 con 0x7f3524101720 2024-03-31T23:23:50.291 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:50.287+0000 7f352a014640 1 -- 172.21.15.46:0/1065165396 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f35241a49f0 con 0x7f3524101720 2024-03-31T23:23:50.299 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:50.289+0000 7f35217fa640 1 -- 172.21.15.46:0/1065165396 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 25) ==== 137451+0+0 (secure 0 0 0) 0x7f3514046020 con 0x7f3524101720 2024-03-31T23:23:50.299 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:50.289+0000 7f352cbf0740 1 Processor -- start 2024-03-31T23:23:50.299 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:50.289+0000 7f352cbf0740 1 -- start start 2024-03-31T23:23:50.299 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:50.289+0000 7f35217fa640 1 --2- 172.21.15.46:0/1065165396 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f35100a26b0 0x7f35100a4b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:23:50.299 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:50.289+0000 7f352cbf0740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5557321986c0 0x555732198aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:23:50.299 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:50.289+0000 7f352cbf0740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555732198fe0 0x5557321a1870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:23:50.299 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:50.289+0000 7f352cbf0740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5557321a1db0 0x5557321a41a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:23:50.299 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:50.289+0000 7f352cbf0740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x5557320fb8f0 con 0x5557321a1db0 2024-03-31T23:23:50.299 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:50.289+0000 7f352cbf0740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x5557320a48d0 con 0x5557321986c0 2024-03-31T23:23:50.300 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:50.289+0000 7f352cbf0740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5557320f7990 con 0x555732198fe0 2024-03-31T23:23:50.300 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:50.290+0000 7f3523fff640 1 --2- 172.21.15.46:0/1065165396 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f35100a26b0 0x7f35100a4b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:23:50.300 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:50.290+0000 7f35217fa640 1 -- 172.21.15.46:0/1065165396 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 9554+0+0 (secure 0 0 0) 0x7f35140f0de0 con 0x7f3524101720 2024-03-31T23:23:50.300 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:50.290+0000 7f3523fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5557321a1db0 0x5557321a41a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:23:50.300 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:50.290+0000 7f3523fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5557321a1db0 0x5557321a41a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:38068/0 (socket says 172.21.15.46:38068) 2024-03-31T23:23:50.300 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:50.290+0000 7f3523fff640 1 -- 172.21.15.46:0/2026157433 learned_addr learned my addr 172.21.15.46:0/2026157433 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:23:50.300 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:50.290+0000 7f35237fe640 1 --2- 172.21.15.46:0/2026157433 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5557321986c0 0x555732198aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:23:50.300 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:50.290+0000 7f351affd640 1 --2- 172.21.15.46:0/2026157433 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555732198fe0 0x5557321a1870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:23:50.300 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:50.290+0000 7f3523fff640 1 -- 172.21.15.46:0/2026157433 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555732198fe0 msgr2=0x5557321a1870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:23:50.300 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:50.290+0000 7f3523fff640 1 --2- 172.21.15.46:0/2026157433 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555732198fe0 0x5557321a1870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:23:50.300 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:50.290+0000 7f3523fff640 1 -- 172.21.15.46:0/2026157433 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5557321986c0 msgr2=0x555732198aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:23:50.301 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:50.290+0000 7f3523fff640 1 --2- 172.21.15.46:0/2026157433 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5557321986c0 0x555732198aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:23:50.301 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:50.290+0000 7f3523fff640 1 -- 172.21.15.46:0/2026157433 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555731e10210 con 0x5557321a1db0 2024-03-31T23:23:50.301 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:50.290+0000 7f35237fe640 1 --2- 172.21.15.46:0/2026157433 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5557321986c0 0x555732198aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:23:50.301 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:50.290+0000 7f351affd640 1 --2- 172.21.15.46:0/2026157433 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555732198fe0 0x5557321a1870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:23:50.301 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:50.290+0000 7f3523fff640 1 --2- 172.21.15.46:0/2026157433 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5557321a1db0 0x5557321a41a0 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7f351c010dd0 tx=0x7f351c00f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b5745fdb00b38d9f server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:23:50.301 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:50.290+0000 7f3523fff640 1 --2- 172.21.15.46:0/1065165396 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f35100a26b0 0x7f35100a4b70 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f351c00fc90 tx=0x7f351c018000 comp rx=0 tx=0).ready entity=mgr.15654 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:23:50.301 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:50.290+0000 7f3519ffb640 1 -- 172.21.15.46:0/2026157433 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f351c019040 con 0x5557321a1db0 2024-03-31T23:23:50.301 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:50.290+0000 7f3519ffb640 1 -- 172.21.15.46:0/2026157433 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f351c009d80 con 0x5557321a1db0 2024-03-31T23:23:50.301 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:50.290+0000 7f3519ffb640 1 -- 172.21.15.46:0/2026157433 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f351c005490 con 0x5557321a1db0 2024-03-31T23:23:50.301 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:50.291+0000 7f352cbf0740 1 -- 172.21.15.46:0/2026157433 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5557321a1db0 msgr2=0x5557321a41a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:23:50.301 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:50.291+0000 7f352cbf0740 1 --2- 172.21.15.46:0/2026157433 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5557321a1db0 0x5557321a41a0 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7f351c010dd0 tx=0x7f351c00f790 comp rx=0 tx=0).stop 2024-03-31T23:23:50.301 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:50.291+0000 7f352cbf0740 1 -- 172.21.15.46:0/2026157433 shutdown_connections 2024-03-31T23:23:50.302 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:50.291+0000 7f352cbf0740 1 --2- 172.21.15.46:0/2026157433 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5557321a1db0 0x5557321a41a0 unknown :-1 s=CLOSED pgs=848 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:23:50.302 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:50.291+0000 7f352cbf0740 1 --2- 172.21.15.46:0/2026157433 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555732198fe0 0x5557321a1870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:23:50.302 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:50.291+0000 7f352cbf0740 1 --2- 172.21.15.46:0/2026157433 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5557321986c0 0x555732198aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:23:50.302 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:50.291+0000 7f352cbf0740 1 -- 172.21.15.46:0/2026157433 >> 172.21.15.46:0/2026157433 conn(0x555732192fe0 msgr2=0x5557321933c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:23:50.302 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:50.291+0000 7f352cbf0740 1 -- 172.21.15.46:0/2026157433 shutdown_connections 2024-03-31T23:23:50.302 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:50.291+0000 7f352cbf0740 1 -- 172.21.15.46:0/2026157433 wait complete. 2024-03-31T23:23:50.302 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:50.291+0000 7f352cbf0740 1 -- start start 2024-03-31T23:23:50.302 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:50.292+0000 7f352cbf0740 1 Processor -- start 2024-03-31T23:23:50.302 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:50.292+0000 7f352cbf0740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5557321986c0 0x55573219edc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:23:50.302 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:50.292+0000 7f352cbf0740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x555732198fe0 0x55573219f300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:23:50.302 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:50.292+0000 7f352cbf0740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5557321a1db0 0x55573219f840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:23:50.303 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:50.292+0000 7f352cbf0740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x5557321a4980 con 0x5557321a1db0 2024-03-31T23:23:50.303 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:50.292+0000 7f352cbf0740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x555732099f80 con 0x555732198fe0 2024-03-31T23:23:50.303 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:50.292+0000 7f352cbf0740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5557321a4b00 con 0x5557321986c0 2024-03-31T23:23:50.303 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:50.292+0000 7f352cbf0740 10 client.0 _ll_register_callbacks cb 0x5557320f2290 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T23:23:50.303 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:50.292+0000 7f3523fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5557321a1db0 0x55573219f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:23:50.303 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:50.292+0000 7f3523fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5557321a1db0 0x55573219f840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:38076/0 (socket says 172.21.15.46:38076) 2024-03-31T23:23:50.303 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:50.292+0000 7f3523fff640 1 -- 172.21.15.46:0/958195433 learned_addr learned my addr 172.21.15.46:0/958195433 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:23:50.303 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:50.292+0000 7f35237fe640 1 --2- 172.21.15.46:0/958195433 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5557321986c0 0x55573219edc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:23:50.303 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:50.292+0000 7f351affd640 1 --2- 172.21.15.46:0/958195433 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x555732198fe0 0x55573219f300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:23:50.303 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:50.292+0000 7f3523fff640 1 -- 172.21.15.46:0/958195433 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5557321986c0 msgr2=0x55573219edc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:23:50.303 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:50.292+0000 7f3523fff640 1 --2- 172.21.15.46:0/958195433 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5557321986c0 0x55573219edc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:23:50.303 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:50.292+0000 7f3523fff640 1 -- 172.21.15.46:0/958195433 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x555732198fe0 msgr2=0x55573219f300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:23:50.304 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:50.292+0000 7f3523fff640 1 --2- 172.21.15.46:0/958195433 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x555732198fe0 0x55573219f300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:23:50.304 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:50.292+0000 7f3523fff640 1 -- 172.21.15.46:0/958195433 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555731edc620 con 0x5557321a1db0 2024-03-31T23:23:50.304 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:50.293+0000 7f3523fff640 1 --2- 172.21.15.46:0/958195433 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5557321a1db0 0x55573219f840 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7f351c0049a0 tx=0x7f351c009f30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2783f8c9f475f439 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:23:50.304 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:50.293+0000 7f3502ffd640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-31T23:23:50.304 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:50.293+0000 7f3502ffd640 1 -- 172.21.15.46:0/958195433 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f351c034060 con 0x5557321a1db0 2024-03-31T23:23:50.304 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:50.293+0000 7f3502ffd640 1 -- 172.21.15.46:0/958195433 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(108..108 src has 1..108) ==== 9554+0+0 (secure 0 0 0) 0x7f351c005390 con 0x5557321a1db0 2024-03-31T23:23:50.304 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:50.293+0000 7f352cbf0740 1 -- 172.21.15.46:0/958195433 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f351c009d80 con 0x5557321a1db0 2024-03-31T23:23:50.304 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:50.293+0000 7f3502ffd640 1 -- 172.21.15.46:0/958195433 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f351c03b050 con 0x5557321a1db0 2024-03-31T23:23:50.304 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:50.293+0000 7f352cbf0740 10 client.17136 fetch_fsmap learned FSMap version 36 2024-03-31T23:23:50.304 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:50.293+0000 7f352cbf0740 1 -- 172.21.15.46:0/958195433 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x5557321b08a0 con 0x5557321a1db0 2024-03-31T23:23:50.304 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:50.293+0000 7f3502ffd640 1 -- 172.21.15.46:0/958195433 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f351c009d80 con 0x5557321a1db0 2024-03-31T23:23:50.304 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:50.293+0000 7f3502ffd640 1 -- 172.21.15.46:0/958195433 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 36) ==== 8362+0+0 (secure 0 0 0) 0x7f351c026780 con 0x5557321a1db0 2024-03-31T23:23:50.305 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:50.293+0000 7f352cbf0740 10 client.17136 fetch_fsmap finished waiting for FSMap version 36 2024-03-31T23:23:50.305 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:50.293+0000 7f352cbf0740 10 client.17136 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-31T23:23:50.305 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:50.293+0000 7f352cbf0740 20 client.17136 populate_metadata read hostname 'smithi046' 2024-03-31T23:23:50.305 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:50.293+0000 7f352cbf0740 1 --2- 172.21.15.46:0/958195433 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x5557321b1d60 0x5557321b4150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:23:50.305 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:50.293+0000 7f352cbf0740 4 client.17136 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-31T23:23:50.305 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:50.293+0000 7f352cbf0740 1 -- 172.21.15.46:0/958195433 --> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55573210f120 con 0x5557321b1d60 2024-03-31T23:23:50.305 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:50.294+0000 7f35237fe640 1 --2- 172.21.15.46:0/958195433 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x5557321b1d60 0x5557321b4150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:23:50.305 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:50.294+0000 7f3502ffd640 10 client.17136 ms_handle_connect on v2:172.21.15.142:6838/2519143918 2024-03-31T23:23:50.305 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:50.294+0000 7f35237fe640 1 --2- 172.21.15.46:0/958195433 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x5557321b1d60 0x5557321b4150 crc :-1 s=READY pgs=183 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=90fa5f93874a7b64 server_cookie=fd6a45a8a26173ee in_seq=0 out_seq=0 2024-03-31T23:23:50.305 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:50.295+0000 7f3502ffd640 1 -- 172.21.15.46:0/958195433 <== mds.0 v2:172.21.15.142:6838/2519143918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55573210f120 con 0x5557321b1d60 2024-03-31T23:23:50.305 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:50.295+0000 7f3502ffd640 10 client.17136 handle_command_reply: tid=0 2024-03-31T23:23:50.305 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:50.295+0000 7f352cbf0740 1 client.17136 shutdown 2024-03-31T23:23:50.306 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:50.295+0000 7f352cbf0740 1 -- 172.21.15.46:0/958195433 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5557321a1db0 msgr2=0x55573219f840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:23:50.306 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:50.295+0000 7f352cbf0740 1 --2- 172.21.15.46:0/958195433 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5557321a1db0 0x55573219f840 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7f351c0049a0 tx=0x7f351c009f30 comp rx=0 tx=0).stop 2024-03-31T23:23:50.306 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:50.295+0000 7f352cbf0740 1 -- 172.21.15.46:0/958195433 shutdown_connections 2024-03-31T23:23:50.306 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:50.295+0000 7f352cbf0740 1 --2- 172.21.15.46:0/958195433 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x5557321b1d60 0x5557321b4150 crc :-1 s=READY pgs=183 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:23:50.306 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:50.295+0000 7f352cbf0740 1 --2- 172.21.15.46:0/958195433 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5557321a1db0 0x55573219f840 unknown :-1 s=CLOSED pgs=849 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:23:50.306 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:50.295+0000 7f352cbf0740 1 --2- 172.21.15.46:0/958195433 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x555732198fe0 0x55573219f300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:23:50.306 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:50.295+0000 7f3502ffd640 0 client.17136 ms_handle_reset on v2:172.21.15.142:6838/2519143918 2024-03-31T23:23:50.306 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:50.295+0000 7f352cbf0740 1 --2- 172.21.15.46:0/958195433 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5557321986c0 0x55573219edc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:23:50.306 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:50.295+0000 7f352cbf0740 1 -- 172.21.15.46:0/958195433 >> 172.21.15.46:0/958195433 conn(0x555732192fe0 msgr2=0x555732194d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:23:50.306 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:50.295+0000 7f352cbf0740 1 -- 172.21.15.46:0/958195433 shutdown_connections 2024-03-31T23:23:50.306 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:50.295+0000 7f352cbf0740 1 -- 172.21.15.46:0/958195433 wait complete. 2024-03-31T23:23:50.306 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:50.295+0000 7f352cbf0740 20 client.17136 trim_cache size 0 max 16384 2024-03-31T23:23:50.315 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:50.310+0000 7f352cbf0740 1 Processor -- start 2024-03-31T23:23:50.315 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:50.310+0000 7f352cbf0740 1 -- start start 2024-03-31T23:23:50.316 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:50.310+0000 7f352cbf0740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5557321986c0 0x5557321a14f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:23:50.316 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:50.310+0000 7f352cbf0740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555732198fe0 0x55573219df20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:23:50.316 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:50.310+0000 7f352cbf0740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5557321a1db0 0x5557321ad750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:23:50.316 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:50.310+0000 7f352cbf0740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f35241b1520 con 0x555732198fe0 2024-03-31T23:23:50.316 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:50.310+0000 7f352cbf0740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f35241aed90 con 0x5557321a1db0 2024-03-31T23:23:50.316 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:50.310+0000 7f352cbf0740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f35241acc00 con 0x5557321986c0 2024-03-31T23:23:50.316 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:50.310+0000 7f351affd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555732198fe0 0x55573219df20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:23:50.316 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:50.310+0000 7f351affd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555732198fe0 0x55573219df20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:38090/0 (socket says 172.21.15.46:38090) 2024-03-31T23:23:50.316 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:50.310+0000 7f351affd640 1 -- 172.21.15.46:0/3411357491 learned_addr learned my addr 172.21.15.46:0/3411357491 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:23:50.316 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:50.310+0000 7f3523fff640 1 --2- 172.21.15.46:0/3411357491 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5557321a1db0 0x5557321ad750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:23:50.316 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:50.310+0000 7f35237fe640 1 --2- 172.21.15.46:0/3411357491 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5557321986c0 0x5557321a14f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:23:50.316 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:50.310+0000 7f351affd640 1 -- 172.21.15.46:0/3411357491 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5557321986c0 msgr2=0x5557321a14f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:23:50.317 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:50.310+0000 7f351affd640 1 --2- 172.21.15.46:0/3411357491 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5557321986c0 0x5557321a14f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:23:50.317 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:50.310+0000 7f351affd640 1 -- 172.21.15.46:0/3411357491 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5557321a1db0 msgr2=0x5557321ad750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:23:50.317 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:50.310+0000 7f351affd640 1 --2- 172.21.15.46:0/3411357491 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5557321a1db0 0x5557321ad750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:23:50.317 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:50.310+0000 7f351affd640 1 -- 172.21.15.46:0/3411357491 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555732199bb0 con 0x555732198fe0 2024-03-31T23:23:50.317 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:50.310+0000 7f35237fe640 1 --2- 172.21.15.46:0/3411357491 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5557321986c0 0x5557321a14f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T23:23:50.317 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:50.310+0000 7f3523fff640 1 --2- 172.21.15.46:0/3411357491 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5557321a1db0 0x5557321ad750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T23:23:50.317 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:50.311+0000 7f351affd640 1 --2- 172.21.15.46:0/3411357491 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555732198fe0 0x55573219df20 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7f350c009c50 tx=0x7f350c005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b0b994448c86abc4 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:23:50.317 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:50.311+0000 7f35197fa640 1 -- 172.21.15.46:0/3411357491 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f350c009e00 con 0x555732198fe0 2024-03-31T23:23:50.317 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:50.311+0000 7f35197fa640 1 -- 172.21.15.46:0/3411357491 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f350c0054d0 con 0x555732198fe0 2024-03-31T23:23:50.317 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:50.311+0000 7f35197fa640 1 -- 172.21.15.46:0/3411357491 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f350c004d00 con 0x555732198fe0 2024-03-31T23:23:50.317 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:50.311+0000 7f352cbf0740 1 -- 172.21.15.46:0/3411357491 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555732198fe0 msgr2=0x55573219df20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:23:50.317 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:50.311+0000 7f352cbf0740 1 --2- 172.21.15.46:0/3411357491 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555732198fe0 0x55573219df20 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7f350c009c50 tx=0x7f350c005f90 comp rx=0 tx=0).stop 2024-03-31T23:23:50.317 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:50.311+0000 7f352cbf0740 1 -- 172.21.15.46:0/3411357491 shutdown_connections 2024-03-31T23:23:50.317 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:50.311+0000 7f352cbf0740 1 --2- 172.21.15.46:0/3411357491 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5557321a1db0 0x5557321ad750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:23:50.317 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:50.311+0000 7f352cbf0740 1 --2- 172.21.15.46:0/3411357491 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555732198fe0 0x55573219df20 unknown :-1 s=CLOSED pgs=850 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:23:50.318 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:50.311+0000 7f352cbf0740 1 --2- 172.21.15.46:0/3411357491 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5557321986c0 0x5557321a14f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:23:50.318 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:50.311+0000 7f352cbf0740 1 -- 172.21.15.46:0/3411357491 >> 172.21.15.46:0/3411357491 conn(0x555732192fe0 msgr2=0x5557321916f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:23:50.318 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:50.311+0000 7f352cbf0740 1 -- 172.21.15.46:0/3411357491 shutdown_connections 2024-03-31T23:23:50.318 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:50.312+0000 7f352cbf0740 1 -- 172.21.15.46:0/3411357491 wait complete. 2024-03-31T23:23:50.318 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:50.312+0000 7f352cbf0740 1 -- start start 2024-03-31T23:23:50.318 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:50.312+0000 7f352cbf0740 1 Processor -- start 2024-03-31T23:23:50.318 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:50.312+0000 7f352cbf0740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5557321986c0 0x5557321a56c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:23:50.318 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:50.312+0000 7f352cbf0740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5557321a1db0 0x5557321a5c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:23:50.318 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:50.312+0000 7f352cbf0740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5557321a6140 0x555732216e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:23:50.318 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:50.312+0000 7f352cbf0740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f35241ad250 con 0x5557321986c0 2024-03-31T23:23:50.318 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:50.312+0000 7f352cbf0740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f35241ae6c0 con 0x5557321a6140 2024-03-31T23:23:50.318 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:50.312+0000 7f352cbf0740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f35241ad920 con 0x5557321a1db0 2024-03-31T23:23:50.319 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:50.312+0000 7f352cbf0740 10 client.0 _ll_register_callbacks cb 0x7f3514005b50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T23:23:50.319 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:50.312+0000 7f35237fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5557321986c0 0x5557321a56c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:23:50.319 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:50.312+0000 7f35237fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5557321986c0 0x5557321a56c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:38100/0 (socket says 172.21.15.46:38100) 2024-03-31T23:23:50.319 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:50.312+0000 7f35237fe640 1 -- 172.21.15.46:0/1761490444 learned_addr learned my addr 172.21.15.46:0/1761490444 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:23:50.319 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:50.312+0000 7f3523fff640 1 --2- 172.21.15.46:0/1761490444 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5557321a6140 0x555732216e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:23:50.319 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:50.312+0000 7f351affd640 1 --2- 172.21.15.46:0/1761490444 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5557321a1db0 0x5557321a5c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:23:50.319 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:50.313+0000 7f35237fe640 1 -- 172.21.15.46:0/1761490444 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5557321a1db0 msgr2=0x5557321a5c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:23:50.319 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:50.313+0000 7f35237fe640 1 --2- 172.21.15.46:0/1761490444 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5557321a1db0 0x5557321a5c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:23:50.319 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:50.313+0000 7f35237fe640 1 -- 172.21.15.46:0/1761490444 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5557321a6140 msgr2=0x555732216e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:23:50.319 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:50.313+0000 7f35237fe640 1 --2- 172.21.15.46:0/1761490444 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5557321a6140 0x555732216e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:23:50.319 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:50.313+0000 7f35237fe640 1 -- 172.21.15.46:0/1761490444 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555732217380 con 0x5557321986c0 2024-03-31T23:23:50.319 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:50.313+0000 7f35237fe640 1 --2- 172.21.15.46:0/1761490444 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5557321986c0 0x5557321a56c0 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7f35140ee2e0 tx=0x7f35140dd040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a49cc67dcefb146 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:23:50.320 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:50.313+0000 7f3502ffd640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-31T23:23:50.320 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:50.313+0000 7f3502ffd640 1 -- 172.21.15.46:0/1761490444 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35140c11b0 con 0x5557321986c0 2024-03-31T23:23:50.320 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:50.313+0000 7f3502ffd640 1 -- 172.21.15.46:0/1761490444 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(108..108 src has 1..108) ==== 9554+0+0 (secure 0 0 0) 0x7f35140bb560 con 0x5557321986c0 2024-03-31T23:23:50.320 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:50.313+0000 7f352cbf0740 1 -- 172.21.15.46:0/1761490444 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f350c0054d0 con 0x5557321986c0 2024-03-31T23:23:50.320 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:50.313+0000 7f3502ffd640 1 -- 172.21.15.46:0/1761490444 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f35140c5c70 con 0x5557321986c0 2024-03-31T23:23:50.320 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:50.313+0000 7f352cbf0740 10 client.17142 fetch_fsmap learned FSMap version 36 2024-03-31T23:23:50.320 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:50.313+0000 7f352cbf0740 1 -- 172.21.15.46:0/1761490444 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x555732218650 con 0x5557321986c0 2024-03-31T23:23:50.320 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:50.313+0000 7f3502ffd640 1 -- 172.21.15.46:0/1761490444 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f350c0054d0 con 0x5557321986c0 2024-03-31T23:23:50.320 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:50.314+0000 7f3502ffd640 1 -- 172.21.15.46:0/1761490444 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 36) ==== 8362+0+0 (secure 0 0 0) 0x7f35140053d0 con 0x5557321986c0 2024-03-31T23:23:50.320 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:50.314+0000 7f352cbf0740 10 client.17142 fetch_fsmap finished waiting for FSMap version 36 2024-03-31T23:23:50.320 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:50.314+0000 7f352cbf0740 10 client.17142 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-31T23:23:50.320 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:50.314+0000 7f352cbf0740 20 client.17142 populate_metadata read hostname 'smithi046' 2024-03-31T23:23:50.321 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:50.314+0000 7f352cbf0740 1 --2- 172.21.15.46:0/1761490444 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x555732218dd0 0x55573221b1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:23:50.321 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:50.314+0000 7f352cbf0740 4 client.17142 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-31T23:23:50.321 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:50.314+0000 7f352cbf0740 1 -- 172.21.15.46:0/1761490444 --> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] -- command(tid 0: {"prefix": "flush journal"}) -- 0x555732108e30 con 0x555732218dd0 2024-03-31T23:23:50.321 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:50.314+0000 7f351affd640 1 --2- 172.21.15.46:0/1761490444 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x555732218dd0 0x55573221b1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:23:50.321 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:50.314+0000 7f351affd640 1 --2- 172.21.15.46:0/1761490444 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x555732218dd0 0x55573221b1f0 crc :-1 s=READY pgs=184 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d62b15d8f4a5c7e6 server_cookie=f51f274b4c20cbe5 in_seq=0 out_seq=0 2024-03-31T23:23:50.321 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:50.314+0000 7f3502ffd640 10 client.17142 ms_handle_connect on v2:172.21.15.142:6838/2519143918 2024-03-31T23:23:50.337 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:50.335+0000 7f3502ffd640 1 -- 172.21.15.46:0/1761490444 <== mds.0 v2:172.21.15.142:6838/2519143918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x555732108e30 con 0x555732218dd0 2024-03-31T23:23:50.337 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:50.335+0000 7f3502ffd640 10 client.17142 handle_command_reply: tid=0 2024-03-31T23:23:50.337 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:50.335+0000 7f352cbf0740 1 client.17142 shutdown 2024-03-31T23:23:50.337 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:50.335+0000 7f352cbf0740 1 -- 172.21.15.46:0/1761490444 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5557321986c0 msgr2=0x5557321a56c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:23:50.337 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:50.335+0000 7f352cbf0740 1 --2- 172.21.15.46:0/1761490444 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5557321986c0 0x5557321a56c0 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7f35140ee2e0 tx=0x7f35140dd040 comp rx=0 tx=0).stop 2024-03-31T23:23:50.337 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:50.335+0000 7f352cbf0740 1 -- 172.21.15.46:0/1761490444 shutdown_connections 2024-03-31T23:23:50.337 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:50.335+0000 7f352cbf0740 1 --2- 172.21.15.46:0/1761490444 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x555732218dd0 0x55573221b1f0 crc :-1 s=READY pgs=184 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:23:50.337 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:50.335+0000 7f352cbf0740 1 --2- 172.21.15.46:0/1761490444 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5557321a6140 0x555732216e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:23:50.337 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:50.335+0000 7f352cbf0740 1 --2- 172.21.15.46:0/1761490444 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5557321a1db0 0x5557321a5c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:23:50.337 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:50.335+0000 7f3502ffd640 0 client.17142 ms_handle_reset on v2:172.21.15.142:6838/2519143918 2024-03-31T23:23:50.337 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:50.335+0000 7f352cbf0740 1 --2- 172.21.15.46:0/1761490444 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5557321986c0 0x5557321a56c0 unknown :-1 s=CLOSED pgs=851 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:23:50.338 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:50.335+0000 7f352cbf0740 1 -- 172.21.15.46:0/1761490444 >> 172.21.15.46:0/1761490444 conn(0x555732192fe0 msgr2=0x5557321943e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:23:50.338 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:50.335+0000 7f352cbf0740 1 -- 172.21.15.46:0/1761490444 shutdown_connections 2024-03-31T23:23:50.338 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:50.335+0000 7f352cbf0740 1 -- 172.21.15.46:0/1761490444 wait complete. 2024-03-31T23:23:50.338 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:50.335+0000 7f352cbf0740 20 client.17142 trim_cache size 0 max 16384 2024-03-31T23:23:50.338 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout:{ 2024-03-31T23:23:50.338 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout: "message": "", 2024-03-31T23:23:50.338 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout: "return_code": 0 2024-03-31T23:23:50.339 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout:} 2024-03-31T23:23:50.339 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:50.336+0000 7f352a014640 1 -- 172.21.15.46:0/1065165396 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f35100a26b0 msgr2=0x7f35100a4b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:23:50.339 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:50.336+0000 7f352a014640 1 --2- 172.21.15.46:0/1065165396 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f35100a26b0 0x7f35100a4b70 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f351c00fc90 tx=0x7f351c018000 comp rx=0 tx=0).stop 2024-03-31T23:23:50.339 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:50.336+0000 7f352a014640 1 -- 172.21.15.46:0/1065165396 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3524101720 msgr2=0x7f3524197ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:23:50.339 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:50.336+0000 7f352a014640 1 --2- 172.21.15.46:0/1065165396 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3524101720 0x7f3524197ed0 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7f3514009230 tx=0x7f351402c5d0 comp rx=0 tx=0).stop 2024-03-31T23:23:50.339 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:50.336+0000 7f352a014640 1 -- 172.21.15.46:0/1065165396 shutdown_connections 2024-03-31T23:23:50.340 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:50.336+0000 7f352a014640 1 --2- 172.21.15.46:0/1065165396 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f35100a26b0 0x7f35100a4b70 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:23:50.340 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:50.336+0000 7f352a014640 1 --2- 172.21.15.46:0/1065165396 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3524107730 0x7f352419c740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:23:50.340 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:50.336+0000 7f352a014640 1 --2- 172.21.15.46:0/1065165396 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f35241020e0 0x7f3524198410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:23:50.340 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:50.336+0000 7f352a014640 1 --2- 172.21.15.46:0/1065165396 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3524101720 0x7f3524197ed0 unknown :-1 s=CLOSED pgs=847 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:23:50.340 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:50.336+0000 7f352a014640 1 -- 172.21.15.46:0/1065165396 >> 172.21.15.46:0/1065165396 conn(0x7f35240fd230 msgr2=0x7f35240fd990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:23:50.340 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:50.336+0000 7f352a014640 1 -- 172.21.15.46:0/1065165396 shutdown_connections 2024-03-31T23:23:50.340 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:23:50.336+0000 7f352a014640 1 -- 172.21.15.46:0/1065165396 wait complete. 2024-03-31T23:23:50.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:23:50 smithi142 ceph-mon[35271]: pgmap v378: 129 pgs: 129 active+clean; 139 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 386 KiB/s rd, 1.0 MiB/s wr, 103 op/s 2024-03-31T23:23:51.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:23:50 smithi162 ceph-mon[36237]: pgmap v378: 129 pgs: 129 active+clean; 139 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 386 KiB/s rd, 1.0 MiB/s wr, 103 op/s 2024-03-31T23:23:51.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:23:50 smithi046 ceph-mon[32498]: pgmap v378: 129 pgs: 129 active+clean; 139 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 386 KiB/s rd, 1.0 MiB/s wr, 103 op/s 2024-03-31T23:23:51.219 INFO:tasks.workunit.client.0.smithi046.stdout: 1 234287 1.81 MB/sec execute 537 sec latency 3228.689 ms 2024-03-31T23:23:52.219 INFO:tasks.workunit.client.0.smithi046.stdout: 1 234287 1.81 MB/sec execute 538 sec latency 4228.810 ms 2024-03-31T23:23:52.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:23:52 smithi142 ceph-mon[35271]: pgmap v379: 129 pgs: 129 active+clean; 139 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 340 KiB/s rd, 844 KiB/s wr, 81 op/s 2024-03-31T23:23:53.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:23:52 smithi162 ceph-mon[36237]: pgmap v379: 129 pgs: 129 active+clean; 139 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 340 KiB/s rd, 844 KiB/s wr, 81 op/s 2024-03-31T23:23:53.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:23:52 smithi046 ceph-mon[32498]: pgmap v379: 129 pgs: 129 active+clean; 139 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 340 KiB/s rd, 844 KiB/s wr, 81 op/s 2024-03-31T23:23:53.219 INFO:tasks.workunit.client.0.smithi046.stdout: 1 234425 1.81 MB/sec execute 539 sec latency 4754.502 ms 2024-03-31T23:23:54.219 INFO:tasks.workunit.client.0.smithi046.stdout: 1 234679 1.80 MB/sec execute 540 sec latency 15.386 ms 2024-03-31T23:23:54.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:23:54 smithi142 ceph-mon[35271]: pgmap v380: 129 pgs: 129 active+clean; 139 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 340 KiB/s rd, 844 KiB/s wr, 81 op/s 2024-03-31T23:23:55.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:23:54 smithi162 ceph-mon[36237]: pgmap v380: 129 pgs: 129 active+clean; 139 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 340 KiB/s rd, 844 KiB/s wr, 81 op/s 2024-03-31T23:23:55.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:23:54 smithi046 ceph-mon[32498]: pgmap v380: 129 pgs: 129 active+clean; 139 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 340 KiB/s rd, 844 KiB/s wr, 81 op/s 2024-03-31T23:23:55.219 INFO:tasks.workunit.client.0.smithi046.stdout: 1 235085 1.80 MB/sec execute 541 sec latency 16.415 ms 2024-03-31T23:23:56.219 INFO:tasks.workunit.client.0.smithi046.stdout: 1 235582 1.81 MB/sec execute 542 sec latency 18.274 ms 2024-03-31T23:23:56.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:23:56 smithi142 ceph-mon[35271]: pgmap v381: 129 pgs: 129 active+clean; 137 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 391 KiB/s rd, 1.0 MiB/s wr, 103 op/s 2024-03-31T23:23:57.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:23:56 smithi162 ceph-mon[36237]: pgmap v381: 129 pgs: 129 active+clean; 137 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 391 KiB/s rd, 1.0 MiB/s wr, 103 op/s 2024-03-31T23:23:57.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:23:56 smithi046 ceph-mon[32498]: pgmap v381: 129 pgs: 129 active+clean; 137 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 391 KiB/s rd, 1.0 MiB/s wr, 103 op/s 2024-03-31T23:23:57.219 INFO:tasks.workunit.client.0.smithi046.stdout: 1 236051 1.80 MB/sec execute 543 sec latency 23.090 ms 2024-03-31T23:23:58.219 INFO:tasks.workunit.client.0.smithi046.stdout: 1 236561 1.80 MB/sec execute 544 sec latency 24.187 ms 2024-03-31T23:23:58.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:23:58 smithi142 ceph-mon[35271]: pgmap v382: 129 pgs: 129 active+clean; 137 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 163 KiB/s rd, 515 KiB/s wr, 47 op/s 2024-03-31T23:23:59.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:23:58 smithi162 ceph-mon[36237]: pgmap v382: 129 pgs: 129 active+clean; 137 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 163 KiB/s rd, 515 KiB/s wr, 47 op/s 2024-03-31T23:23:59.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:23:58 smithi046 ceph-mon[32498]: pgmap v382: 129 pgs: 129 active+clean; 137 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 163 KiB/s rd, 515 KiB/s wr, 47 op/s 2024-03-31T23:23:59.219 INFO:tasks.workunit.client.0.smithi046.stdout: 1 237135 1.81 MB/sec execute 545 sec latency 16.008 ms 2024-03-31T23:24:00.220 INFO:tasks.workunit.client.0.smithi046.stdout: 1 237532 1.81 MB/sec execute 546 sec latency 16.378 ms 2024-03-31T23:24:00.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:24:00 smithi142 ceph-mon[35271]: pgmap v383: 129 pgs: 129 active+clean; 138 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 391 KiB/s rd, 1.0 MiB/s wr, 102 op/s 2024-03-31T23:24:01.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:24:00 smithi162 ceph-mon[36237]: pgmap v383: 129 pgs: 129 active+clean; 138 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 391 KiB/s rd, 1.0 MiB/s wr, 102 op/s 2024-03-31T23:24:01.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:24:00 smithi046 ceph-mon[32498]: pgmap v383: 129 pgs: 129 active+clean; 138 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 391 KiB/s rd, 1.0 MiB/s wr, 102 op/s 2024-03-31T23:24:01.220 INFO:tasks.workunit.client.0.smithi046.stdout: 1 237764 1.81 MB/sec execute 547 sec latency 669.421 ms 2024-03-31T23:24:01.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:24:01 smithi142 ceph-mon[35271]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-31T23:24:01.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:24:01 smithi142 ceph-mon[35271]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:24:01.944 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:24:01 smithi142 ceph-mon[35271]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-31T23:24:01.944 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:24:01 smithi142 ceph-mon[35271]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' 2024-03-31T23:24:02.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:24:01 smithi162 ceph-mon[36237]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-31T23:24:02.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:24:01 smithi162 ceph-mon[36237]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:24:02.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:24:01 smithi162 ceph-mon[36237]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-31T23:24:02.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:24:01 smithi162 ceph-mon[36237]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' 2024-03-31T23:24:02.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:24:01 smithi046 ceph-mon[32498]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-31T23:24:02.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:24:01 smithi046 ceph-mon[32498]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:24:02.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:24:01 smithi046 ceph-mon[32498]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-31T23:24:02.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:24:01 smithi046 ceph-mon[32498]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' 2024-03-31T23:24:02.220 INFO:tasks.workunit.client.0.smithi046.stdout: 1 237764 1.81 MB/sec execute 548 sec latency 1669.494 ms 2024-03-31T23:24:02.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:24:02 smithi142 ceph-mon[35271]: pgmap v384: 129 pgs: 129 active+clean; 138 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 279 KiB/s rd, 742 KiB/s wr, 77 op/s 2024-03-31T23:24:03.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:24:02 smithi162 ceph-mon[36237]: pgmap v384: 129 pgs: 129 active+clean; 138 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 279 KiB/s rd, 742 KiB/s wr, 77 op/s 2024-03-31T23:24:03.114 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:24:02 smithi046 ceph-mon[32498]: pgmap v384: 129 pgs: 129 active+clean; 138 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 279 KiB/s rd, 742 KiB/s wr, 77 op/s 2024-03-31T23:24:03.220 INFO:tasks.workunit.client.0.smithi046.stdout: 1 237831 1.81 MB/sec execute 549 sec latency 2197.451 ms 2024-03-31T23:24:03.443 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:03.426+0000 7f4cdc755640 1 -- 172.21.15.46:0/3228661411 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4cd4101240 msgr2=0x7f4cd410d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:24:03.443 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:03.426+0000 7f4cdc755640 1 --2- 172.21.15.46:0/3228661411 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4cd4101240 0x7f4cd410d480 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7f4ccc009ca0 tx=0x7f4ccc02c0f0 comp rx=0 tx=0).stop 2024-03-31T23:24:03.443 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:03.426+0000 7f4cdc755640 1 -- 172.21.15.46:0/3228661411 shutdown_connections 2024-03-31T23:24:03.443 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:03.426+0000 7f4cdc755640 1 --2- 172.21.15.46:0/3228661411 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4cd4101240 0x7f4cd410d480 unknown :-1 s=CLOSED pgs=852 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:24:03.443 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:03.426+0000 7f4cdc755640 1 --2- 172.21.15.46:0/3228661411 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4cd41008a0 0x7f4cd4100d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:24:03.444 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:03.426+0000 7f4cdc755640 1 --2- 172.21.15.46:0/3228661411 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4cd41055d0 0x7f4cd41059b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:24:03.444 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:03.426+0000 7f4cdc755640 1 -- 172.21.15.46:0/3228661411 >> 172.21.15.46:0/3228661411 conn(0x7f4cd4077420 msgr2=0x7f4cd4077830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:24:03.444 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:03.426+0000 7f4cdc755640 1 -- 172.21.15.46:0/3228661411 shutdown_connections 2024-03-31T23:24:03.444 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:03.426+0000 7f4cdc755640 1 -- 172.21.15.46:0/3228661411 wait complete. 2024-03-31T23:24:03.444 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:03.427+0000 7f4cdc755640 1 Processor -- start 2024-03-31T23:24:03.444 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:03.427+0000 7f4cdc755640 1 -- start start 2024-03-31T23:24:03.445 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:03.427+0000 7f4cdc755640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4cd41008a0 0x7f4cd4196db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:24:03.445 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:03.427+0000 7f4cdc755640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4cd4101240 0x7f4cd41972f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:24:03.445 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:03.427+0000 7f4cdc755640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4cd41055d0 0x7f4cd419b670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:24:03.445 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:03.427+0000 7f4cdc755640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f4cd4110990 con 0x7f4cd4101240 2024-03-31T23:24:03.445 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:03.427+0000 7f4cdc755640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f4cd4110810 con 0x7f4cd41008a0 2024-03-31T23:24:03.445 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:03.427+0000 7f4cdc755640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f4cd4110b10 con 0x7f4cd41055d0 2024-03-31T23:24:03.446 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:03.427+0000 7f4cd9cf0640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4cd4101240 0x7f4cd41972f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:24:03.446 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:03.427+0000 7f4cd9cf0640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4cd4101240 0x7f4cd41972f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:34180/0 (socket says 172.21.15.46:34180) 2024-03-31T23:24:03.446 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:03.427+0000 7f4cda4f1640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4cd41008a0 0x7f4cd4196db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:24:03.446 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:03.427+0000 7f4cd9cf0640 1 -- 172.21.15.46:0/2970497671 learned_addr learned my addr 172.21.15.46:0/2970497671 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:24:03.446 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:03.427+0000 7f4cdacf2640 1 --2- 172.21.15.46:0/2970497671 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4cd41055d0 0x7f4cd419b670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:24:03.446 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:03.427+0000 7f4cd9cf0640 1 -- 172.21.15.46:0/2970497671 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4cd41055d0 msgr2=0x7f4cd419b670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:24:03.446 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:03.427+0000 7f4cd9cf0640 1 --2- 172.21.15.46:0/2970497671 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4cd41055d0 0x7f4cd419b670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:24:03.446 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:03.427+0000 7f4cd9cf0640 1 -- 172.21.15.46:0/2970497671 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4cd41008a0 msgr2=0x7f4cd4196db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:24:03.447 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:03.427+0000 7f4cd9cf0640 1 --2- 172.21.15.46:0/2970497671 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4cd41008a0 0x7f4cd4196db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:24:03.447 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:03.427+0000 7f4cd9cf0640 1 -- 172.21.15.46:0/2970497671 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4cd419bda0 con 0x7f4cd4101240 2024-03-31T23:24:03.447 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:03.427+0000 7f4cd9cf0640 1 --2- 172.21.15.46:0/2970497671 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4cd4101240 0x7f4cd41972f0 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7f4cc400e9f0 tx=0x7f4cc400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:24:03.447 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:03.428+0000 7f4ccb7fe640 1 -- 172.21.15.46:0/2970497671 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4cc400cd80 con 0x7f4cd4101240 2024-03-31T23:24:03.447 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:03.428+0000 7f4cdc755640 1 -- 172.21.15.46:0/2970497671 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4cd419c090 con 0x7f4cd4101240 2024-03-31T23:24:03.447 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:03.428+0000 7f4ccb7fe640 1 -- 172.21.15.46:0/2970497671 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4cc40054d0 con 0x7f4cd4101240 2024-03-31T23:24:03.447 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:03.428+0000 7f4ccb7fe640 1 -- 172.21.15.46:0/2970497671 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4cc4010490 con 0x7f4cd4101240 2024-03-31T23:24:03.448 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:03.428+0000 7f4cdc755640 1 -- 172.21.15.46:0/2970497671 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4cd41a3920 con 0x7f4cd4101240 2024-03-31T23:24:03.448 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:03.430+0000 7f4ccb7fe640 1 -- 172.21.15.46:0/2970497671 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 25) ==== 137451+0+0 (secure 0 0 0) 0x7f4cc4005020 con 0x7f4cd4101240 2024-03-31T23:24:03.448 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:03.430+0000 7f4cdf40c740 1 Processor -- start 2024-03-31T23:24:03.448 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:03.430+0000 7f4cdf40c740 1 -- start start 2024-03-31T23:24:03.448 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:03.431+0000 7f4cdf40c740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56485d8cb8d0 0x56485d8cbcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:24:03.448 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:03.431+0000 7f4cdf40c740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56485d8cc1f0 0x56485d8d4a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:24:03.448 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:03.431+0000 7f4cdf40c740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56485d8d4fc0 0x56485d8d73b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:24:03.448 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:03.431+0000 7f4cdf40c740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x56485d82c8f0 con 0x56485d8cb8d0 2024-03-31T23:24:03.449 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:03.431+0000 7f4cdf40c740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x56485d7d58d0 con 0x56485d8cc1f0 2024-03-31T23:24:03.449 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:03.431+0000 7f4cdf40c740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x56485d828990 con 0x56485d8d4fc0 2024-03-31T23:24:03.449 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:03.431+0000 7f4ccb7fe640 1 --2- 172.21.15.46:0/2970497671 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f4cc00a26b0 0x7f4cc00a4b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:24:03.449 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:03.431+0000 7f4cdacf2640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56485d8cb8d0 0x56485d8cbcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:24:03.449 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:03.431+0000 7f4cdacf2640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56485d8cb8d0 0x56485d8cbcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:34190/0 (socket says 172.21.15.46:34190) 2024-03-31T23:24:03.449 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:03.431+0000 7f4cdacf2640 1 -- 172.21.15.46:0/4281935740 learned_addr learned my addr 172.21.15.46:0/4281935740 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:24:03.449 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:03.431+0000 7f4cda4f1640 1 --2- 172.21.15.46:0/4281935740 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56485d8cc1f0 0x56485d8d4a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:24:03.449 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:03.431+0000 7f4cd9cf0640 1 --2- 172.21.15.46:0/4281935740 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56485d8d4fc0 0x56485d8d73b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:24:03.450 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:03.431+0000 7f4cdacf2640 1 --2- 172.21.15.46:0/2970497671 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f4cc00a26b0 0x7f4cc00a4b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:24:03.450 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:03.431+0000 7f4ccb7fe640 1 -- 172.21.15.46:0/2970497671 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 9554+0+0 (secure 0 0 0) 0x7f4cc40cf340 con 0x7f4cd4101240 2024-03-31T23:24:03.450 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:03.431+0000 7f4cda4f1640 1 -- 172.21.15.46:0/4281935740 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56485d8d4fc0 msgr2=0x56485d8d73b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:24:03.450 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:03.431+0000 7f4cda4f1640 1 --2- 172.21.15.46:0/4281935740 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56485d8d4fc0 0x56485d8d73b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:24:03.450 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:03.431+0000 7f4cda4f1640 1 -- 172.21.15.46:0/4281935740 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56485d8cb8d0 msgr2=0x56485d8cbcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:24:03.450 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:03.431+0000 7f4cda4f1640 1 --2- 172.21.15.46:0/4281935740 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56485d8cb8d0 0x56485d8cbcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:24:03.450 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:03.431+0000 7f4cda4f1640 1 -- 172.21.15.46:0/4281935740 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56485d541210 con 0x56485d8cc1f0 2024-03-31T23:24:03.451 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:03.431+0000 7f4cdacf2640 1 --2- 172.21.15.46:0/4281935740 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56485d8cb8d0 0x56485d8cbcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:24:03.451 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:03.431+0000 7f4cdacf2640 1 --2- 172.21.15.46:0/2970497671 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f4cc00a26b0 0x7f4cc00a4b70 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f4ccc009a60 tx=0x7f4ccc005c50 comp rx=0 tx=0).ready entity=mgr.15654 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:24:03.451 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:03.431+0000 7f4cda4f1640 1 --2- 172.21.15.46:0/4281935740 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56485d8cc1f0 0x56485d8d4a80 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f4cd0009c20 tx=0x7f4cd0011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c9d9081c67e5886 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:24:03.451 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:03.432+0000 7f4cc8ff9640 1 -- 172.21.15.46:0/4281935740 <== mon.2 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4cd0004050 con 0x56485d8cc1f0 2024-03-31T23:24:03.451 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:03.432+0000 7f4cc8ff9640 1 -- 172.21.15.46:0/4281935740 <== mon.2 v2:172.21.15.142:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4cd00054d0 con 0x56485d8cc1f0 2024-03-31T23:24:03.451 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:03.432+0000 7f4cc8ff9640 1 -- 172.21.15.46:0/4281935740 <== mon.2 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4cd0004cf0 con 0x56485d8cc1f0 2024-03-31T23:24:03.451 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:03.432+0000 7f4cdf40c740 1 -- 172.21.15.46:0/4281935740 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56485d8cc1f0 msgr2=0x56485d8d4a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:24:03.451 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:03.432+0000 7f4cdf40c740 1 --2- 172.21.15.46:0/4281935740 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56485d8cc1f0 0x56485d8d4a80 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f4cd0009c20 tx=0x7f4cd0011810 comp rx=0 tx=0).stop 2024-03-31T23:24:03.451 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:03.432+0000 7f4cdf40c740 1 -- 172.21.15.46:0/4281935740 shutdown_connections 2024-03-31T23:24:03.452 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:03.432+0000 7f4cdf40c740 1 --2- 172.21.15.46:0/4281935740 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56485d8d4fc0 0x56485d8d73b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:24:03.452 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:03.432+0000 7f4cdf40c740 1 --2- 172.21.15.46:0/4281935740 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56485d8cc1f0 0x56485d8d4a80 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:24:03.452 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:03.432+0000 7f4cdf40c740 1 --2- 172.21.15.46:0/4281935740 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56485d8cb8d0 0x56485d8cbcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:24:03.452 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:03.432+0000 7f4cdf40c740 1 -- 172.21.15.46:0/4281935740 >> 172.21.15.46:0/4281935740 conn(0x56485d8c61f0 msgr2=0x56485d8c65d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:24:03.452 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:03.432+0000 7f4cdf40c740 1 -- 172.21.15.46:0/4281935740 shutdown_connections 2024-03-31T23:24:03.452 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:03.432+0000 7f4cdf40c740 1 -- 172.21.15.46:0/4281935740 wait complete. 2024-03-31T23:24:03.452 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:03.433+0000 7f4cdf40c740 1 -- start start 2024-03-31T23:24:03.452 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:03.433+0000 7f4cdf40c740 1 Processor -- start 2024-03-31T23:24:03.452 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:03.433+0000 7f4cdf40c740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56485d8cb8d0 0x56485d8d1f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:24:03.452 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:03.433+0000 7f4cdf40c740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56485d8cc1f0 0x56485d8d2480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:24:03.452 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:03.433+0000 7f4cdf40c740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56485d8d4fc0 0x56485d8d29c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:24:03.452 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:03.433+0000 7f4cdf40c740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x56485d8d7bf0 con 0x56485d8cc1f0 2024-03-31T23:24:03.453 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:03.433+0000 7f4cdf40c740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x56485d7caf80 con 0x56485d8cb8d0 2024-03-31T23:24:03.453 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:03.433+0000 7f4cdf40c740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x56485d8d7d70 con 0x56485d8d4fc0 2024-03-31T23:24:03.453 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:03.433+0000 7f4cdf40c740 10 client.0 _ll_register_callbacks cb 0x56485d823290 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T23:24:03.453 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:03.433+0000 7f4cd9cf0640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56485d8cc1f0 0x56485d8d2480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:24:03.453 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:03.433+0000 7f4cd9cf0640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56485d8cc1f0 0x56485d8d2480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:34196/0 (socket says 172.21.15.46:34196) 2024-03-31T23:24:03.453 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:03.433+0000 7f4cd9cf0640 1 -- 172.21.15.46:0/278033608 learned_addr learned my addr 172.21.15.46:0/278033608 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:24:03.453 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:03.433+0000 7f4cdacf2640 1 --2- 172.21.15.46:0/278033608 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56485d8d4fc0 0x56485d8d29c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:24:03.453 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:03.433+0000 7f4cda4f1640 1 --2- 172.21.15.46:0/278033608 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56485d8cb8d0 0x56485d8d1f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:24:03.453 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:03.433+0000 7f4cd9cf0640 1 -- 172.21.15.46:0/278033608 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56485d8d4fc0 msgr2=0x56485d8d29c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:24:03.453 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:03.433+0000 7f4cd9cf0640 1 --2- 172.21.15.46:0/278033608 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56485d8d4fc0 0x56485d8d29c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:24:03.453 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:03.433+0000 7f4cd9cf0640 1 -- 172.21.15.46:0/278033608 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56485d8cb8d0 msgr2=0x56485d8d1f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:24:03.453 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:03.433+0000 7f4cd9cf0640 1 --2- 172.21.15.46:0/278033608 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56485d8cb8d0 0x56485d8d1f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:24:03.453 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:03.433+0000 7f4cd9cf0640 1 -- 172.21.15.46:0/278033608 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56485d60d620 con 0x56485d8cc1f0 2024-03-31T23:24:03.453 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:03.434+0000 7f4cd9cf0640 1 --2- 172.21.15.46:0/278033608 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56485d8cc1f0 0x56485d8d2480 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7f4cc400cfd0 tx=0x7f4cc401def0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=950545d6464557e8 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:24:03.454 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:03.434+0000 7f4cbdffb640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-31T23:24:03.454 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:03.434+0000 7f4cbdffb640 1 -- 172.21.15.46:0/278033608 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4cc4095060 con 0x56485d8cc1f0 2024-03-31T23:24:03.454 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:03.434+0000 7f4cbdffb640 1 -- 172.21.15.46:0/278033608 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(108..108 src has 1..108) ==== 9554+0+0 (secure 0 0 0) 0x7f4cc40b4210 con 0x56485d8cc1f0 2024-03-31T23:24:03.454 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:03.434+0000 7f4cdf40c740 1 -- 172.21.15.46:0/278033608 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4cd00054d0 con 0x56485d8cc1f0 2024-03-31T23:24:03.454 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:03.434+0000 7f4cbdffb640 1 -- 172.21.15.46:0/278033608 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f4cc40b4070 con 0x56485d8cc1f0 2024-03-31T23:24:03.454 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:03.434+0000 7f4cdf40c740 10 client.17154 fetch_fsmap learned FSMap version 36 2024-03-31T23:24:03.454 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:03.434+0000 7f4cdf40c740 1 -- 172.21.15.46:0/278033608 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x56485d8e3a60 con 0x56485d8cc1f0 2024-03-31T23:24:03.454 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:03.434+0000 7f4cbdffb640 1 -- 172.21.15.46:0/278033608 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4cd00054d0 con 0x56485d8cc1f0 2024-03-31T23:24:03.454 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:03.434+0000 7f4cbdffb640 1 -- 172.21.15.46:0/278033608 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 36) ==== 8362+0+0 (secure 0 0 0) 0x7f4cc40cc8a0 con 0x56485d8cc1f0 2024-03-31T23:24:03.454 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:03.434+0000 7f4cdf40c740 10 client.17154 fetch_fsmap finished waiting for FSMap version 36 2024-03-31T23:24:03.454 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:03.434+0000 7f4cdf40c740 10 client.17154 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-31T23:24:03.454 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:03.434+0000 7f4cdf40c740 20 client.17154 populate_metadata read hostname 'smithi046' 2024-03-31T23:24:03.454 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:03.434+0000 7f4cdf40c740 1 --2- 172.21.15.46:0/278033608 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x56485d8e4f20 0x56485d8e7370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:24:03.454 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:03.434+0000 7f4cdf40c740 4 client.17154 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-31T23:24:03.455 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:03.434+0000 7f4cdf40c740 1 -- 172.21.15.46:0/278033608 --> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56485d840120 con 0x56485d8e4f20 2024-03-31T23:24:03.455 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:03.434+0000 7f4cda4f1640 1 --2- 172.21.15.46:0/278033608 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x56485d8e4f20 0x56485d8e7370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:24:03.455 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:03.435+0000 7f4cbdffb640 10 client.17154 ms_handle_connect on v2:172.21.15.142:6838/2519143918 2024-03-31T23:24:03.455 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:03.435+0000 7f4cda4f1640 1 --2- 172.21.15.46:0/278033608 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x56485d8e4f20 0x56485d8e7370 crc :-1 s=READY pgs=185 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=25f6be1e3c9cf8e9 server_cookie=a9f0daad5f88ba29 in_seq=0 out_seq=0 2024-03-31T23:24:03.455 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:03.435+0000 7f4cbdffb640 1 -- 172.21.15.46:0/278033608 <== mds.0 v2:172.21.15.142:6838/2519143918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x56485d840120 con 0x56485d8e4f20 2024-03-31T23:24:03.455 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:03.435+0000 7f4cbdffb640 10 client.17154 handle_command_reply: tid=0 2024-03-31T23:24:03.455 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:03.435+0000 7f4cdf40c740 1 client.17154 shutdown 2024-03-31T23:24:03.455 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:03.436+0000 7f4cdf40c740 1 -- 172.21.15.46:0/278033608 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56485d8cc1f0 msgr2=0x56485d8d2480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:24:03.455 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:03.436+0000 7f4cdf40c740 1 --2- 172.21.15.46:0/278033608 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56485d8cc1f0 0x56485d8d2480 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7f4cc400cfd0 tx=0x7f4cc401def0 comp rx=0 tx=0).stop 2024-03-31T23:24:03.455 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:03.436+0000 7f4cdf40c740 1 -- 172.21.15.46:0/278033608 shutdown_connections 2024-03-31T23:24:03.455 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:03.436+0000 7f4cdf40c740 1 --2- 172.21.15.46:0/278033608 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x56485d8e4f20 0x56485d8e7370 crc :-1 s=READY pgs=185 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:24:03.455 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:03.436+0000 7f4cdf40c740 1 --2- 172.21.15.46:0/278033608 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56485d8d4fc0 0x56485d8d29c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:24:03.455 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:03.436+0000 7f4cdf40c740 1 --2- 172.21.15.46:0/278033608 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56485d8cc1f0 0x56485d8d2480 unknown :-1 s=CLOSED pgs=854 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:24:03.455 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:03.436+0000 7f4cbdffb640 0 client.17154 ms_handle_reset on v2:172.21.15.142:6838/2519143918 2024-03-31T23:24:03.455 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:03.436+0000 7f4cdf40c740 1 --2- 172.21.15.46:0/278033608 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56485d8cb8d0 0x56485d8d1f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:24:03.455 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:03.436+0000 7f4cdf40c740 1 -- 172.21.15.46:0/278033608 >> 172.21.15.46:0/278033608 conn(0x56485d8c61f0 msgr2=0x56485d8c7f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:24:03.456 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:03.436+0000 7f4cdf40c740 1 -- 172.21.15.46:0/278033608 shutdown_connections 2024-03-31T23:24:03.456 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:03.436+0000 7f4cdf40c740 1 -- 172.21.15.46:0/278033608 wait complete. 2024-03-31T23:24:03.456 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:03.436+0000 7f4cdf40c740 20 client.17154 trim_cache size 0 max 16384 2024-03-31T23:24:03.461 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:03.450+0000 7f4cdf40c740 1 Processor -- start 2024-03-31T23:24:03.461 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:03.450+0000 7f4cdf40c740 1 -- start start 2024-03-31T23:24:03.461 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:03.450+0000 7f4cdf40c740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56485d8cb8d0 0x56485d8d4670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:24:03.461 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:03.450+0000 7f4cdf40c740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56485d8cc1f0 0x56485d8e0990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:24:03.461 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:03.450+0000 7f4cdf40c740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56485d8d4fc0 0x56485d8ecc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:24:03.461 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:03.450+0000 7f4cdf40c740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f4cd41af300 con 0x56485d8cb8d0 2024-03-31T23:24:03.461 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:03.450+0000 7f4cdf40c740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f4cd41acb70 con 0x56485d8cc1f0 2024-03-31T23:24:03.461 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:03.450+0000 7f4cdf40c740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f4cd41aa9e0 con 0x56485d8d4fc0 2024-03-31T23:24:03.462 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:03.451+0000 7f4cda4f1640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56485d8cb8d0 0x56485d8d4670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:24:03.462 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:03.451+0000 7f4cda4f1640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56485d8cb8d0 0x56485d8d4670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:34208/0 (socket says 172.21.15.46:34208) 2024-03-31T23:24:03.462 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:03.451+0000 7f4cda4f1640 1 -- 172.21.15.46:0/2428843481 learned_addr learned my addr 172.21.15.46:0/2428843481 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:24:03.462 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:03.451+0000 7f4cd9cf0640 1 --2- 172.21.15.46:0/2428843481 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56485d8cc1f0 0x56485d8e0990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:24:03.462 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:03.451+0000 7f4cdacf2640 1 --2- 172.21.15.46:0/2428843481 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56485d8d4fc0 0x56485d8ecc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:24:03.462 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:03.451+0000 7f4cda4f1640 1 -- 172.21.15.46:0/2428843481 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56485d8d4fc0 msgr2=0x56485d8ecc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:24:03.462 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:03.451+0000 7f4cda4f1640 1 --2- 172.21.15.46:0/2428843481 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56485d8d4fc0 0x56485d8ecc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:24:03.462 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:03.451+0000 7f4cda4f1640 1 -- 172.21.15.46:0/2428843481 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56485d8cc1f0 msgr2=0x56485d8e0990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:24:03.462 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:03.451+0000 7f4cda4f1640 1 --2- 172.21.15.46:0/2428843481 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56485d8cc1f0 0x56485d8e0990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:24:03.462 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:03.451+0000 7f4cda4f1640 1 -- 172.21.15.46:0/2428843481 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56485d8c6020 con 0x56485d8cb8d0 2024-03-31T23:24:03.462 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:03.451+0000 7f4cdacf2640 1 --2- 172.21.15.46:0/2428843481 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56485d8d4fc0 0x56485d8ecc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T23:24:03.462 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:03.451+0000 7f4cda4f1640 1 --2- 172.21.15.46:0/2428843481 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56485d8cb8d0 0x56485d8d4670 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7f4cd0009250 tx=0x7f4cd0009680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a7b4f03e113fe523 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:24:03.462 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:03.451+0000 7f4cbffff640 1 -- 172.21.15.46:0/2428843481 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4cd002ecb0 con 0x56485d8cb8d0 2024-03-31T23:24:03.463 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:03.451+0000 7f4cbffff640 1 -- 172.21.15.46:0/2428843481 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4cd0005a70 con 0x56485d8cb8d0 2024-03-31T23:24:03.463 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:03.451+0000 7f4cbffff640 1 -- 172.21.15.46:0/2428843481 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4cd0003850 con 0x56485d8cb8d0 2024-03-31T23:24:03.463 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:03.452+0000 7f4cdf40c740 1 -- 172.21.15.46:0/2428843481 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56485d8cb8d0 msgr2=0x56485d8d4670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:24:03.463 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:03.452+0000 7f4cdf40c740 1 --2- 172.21.15.46:0/2428843481 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56485d8cb8d0 0x56485d8d4670 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7f4cd0009250 tx=0x7f4cd0009680 comp rx=0 tx=0).stop 2024-03-31T23:24:03.463 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:03.452+0000 7f4cdf40c740 1 -- 172.21.15.46:0/2428843481 shutdown_connections 2024-03-31T23:24:03.463 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:03.452+0000 7f4cdf40c740 1 --2- 172.21.15.46:0/2428843481 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56485d8d4fc0 0x56485d8ecc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:24:03.463 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:03.452+0000 7f4cdf40c740 1 --2- 172.21.15.46:0/2428843481 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56485d8cc1f0 0x56485d8e0990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:24:03.463 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:03.452+0000 7f4cdf40c740 1 --2- 172.21.15.46:0/2428843481 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56485d8cb8d0 0x56485d8d4670 unknown :-1 s=CLOSED pgs=855 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:24:03.463 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:03.452+0000 7f4cdf40c740 1 -- 172.21.15.46:0/2428843481 >> 172.21.15.46:0/2428843481 conn(0x56485d8c61f0 msgr2=0x56485d8c65d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:24:03.463 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:03.452+0000 7f4cdf40c740 1 -- 172.21.15.46:0/2428843481 shutdown_connections 2024-03-31T23:24:03.463 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:03.452+0000 7f4cdf40c740 1 -- 172.21.15.46:0/2428843481 wait complete. 2024-03-31T23:24:03.463 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:03.452+0000 7f4cdf40c740 1 -- start start 2024-03-31T23:24:03.463 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:03.452+0000 7f4cdf40c740 1 Processor -- start 2024-03-31T23:24:03.463 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:03.452+0000 7f4cdf40c740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56485d8cb8d0 0x56485d8dd290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:24:03.464 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:03.452+0000 7f4cdf40c740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56485d8cc1f0 0x56485d8ce7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:24:03.464 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:03.452+0000 7f4cdf40c740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56485d8d4fc0 0x56485d8ced50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:24:03.464 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:03.452+0000 7f4cdf40c740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f4cd41ab030 con 0x56485d8cb8d0 2024-03-31T23:24:03.464 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:03.452+0000 7f4cdf40c740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f4cd41ac4a0 con 0x56485d8cc1f0 2024-03-31T23:24:03.464 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:03.452+0000 7f4cdf40c740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f4cd41ab700 con 0x56485d8d4fc0 2024-03-31T23:24:03.464 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:03.452+0000 7f4cdf40c740 10 client.0 _ll_register_callbacks cb 0x7f4cc400e940 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T23:24:03.464 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:03.452+0000 7f4cda4f1640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56485d8cb8d0 0x56485d8dd290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:24:03.464 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:03.452+0000 7f4cda4f1640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56485d8cb8d0 0x56485d8dd290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:34210/0 (socket says 172.21.15.46:34210) 2024-03-31T23:24:03.464 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:03.452+0000 7f4cda4f1640 1 -- 172.21.15.46:0/1609658954 learned_addr learned my addr 172.21.15.46:0/1609658954 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:24:03.464 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:03.452+0000 7f4cd9cf0640 1 --2- 172.21.15.46:0/1609658954 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56485d8cc1f0 0x56485d8ce7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:24:03.464 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:03.452+0000 7f4cdacf2640 1 --2- 172.21.15.46:0/1609658954 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56485d8d4fc0 0x56485d8ced50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:24:03.464 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:03.453+0000 7f4cda4f1640 1 -- 172.21.15.46:0/1609658954 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56485d8d4fc0 msgr2=0x56485d8ced50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:24:03.464 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:03.453+0000 7f4cda4f1640 1 --2- 172.21.15.46:0/1609658954 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56485d8d4fc0 0x56485d8ced50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:24:03.464 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:03.453+0000 7f4cda4f1640 1 -- 172.21.15.46:0/1609658954 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56485d8cc1f0 msgr2=0x56485d8ce7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:24:03.465 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:03.453+0000 7f4cda4f1640 1 --2- 172.21.15.46:0/1609658954 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56485d8cc1f0 0x56485d8ce7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:24:03.465 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:03.453+0000 7f4cda4f1640 1 -- 172.21.15.46:0/1609658954 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56485d8d73b0 con 0x56485d8cb8d0 2024-03-31T23:24:03.465 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:03.453+0000 7f4cdacf2640 1 --2- 172.21.15.46:0/1609658954 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56485d8d4fc0 0x56485d8ced50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T23:24:03.465 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:03.453+0000 7f4cda4f1640 1 --2- 172.21.15.46:0/1609658954 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56485d8cb8d0 0x56485d8dd290 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7f4cd0009bf0 tx=0x7f4cd0011dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9838773b65585162 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:24:03.465 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:03.453+0000 7f4cbdffb640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-31T23:24:03.465 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:03.453+0000 7f4cbdffb640 1 -- 172.21.15.46:0/1609658954 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4cd003d060 con 0x56485d8cb8d0 2024-03-31T23:24:03.465 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:03.453+0000 7f4cbdffb640 1 -- 172.21.15.46:0/1609658954 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(108..108 src has 1..108) ==== 9554+0+0 (secure 0 0 0) 0x7f4cd002e3b0 con 0x56485d8cb8d0 2024-03-31T23:24:03.465 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:03.453+0000 7f4cdf40c740 1 -- 172.21.15.46:0/1609658954 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4cd0005a70 con 0x56485d8cb8d0 2024-03-31T23:24:03.465 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:03.453+0000 7f4cbdffb640 1 -- 172.21.15.46:0/1609658954 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f4cd002e5b0 con 0x56485d8cb8d0 2024-03-31T23:24:03.465 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:03.453+0000 7f4cdf40c740 10 client.17160 fetch_fsmap learned FSMap version 36 2024-03-31T23:24:03.465 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:03.453+0000 7f4cdf40c740 1 -- 172.21.15.46:0/1609658954 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x56485d94b170 con 0x56485d8cb8d0 2024-03-31T23:24:03.465 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:03.453+0000 7f4cbdffb640 1 -- 172.21.15.46:0/1609658954 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4cd0005a70 con 0x56485d8cb8d0 2024-03-31T23:24:03.465 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:03.453+0000 7f4cbdffb640 1 -- 172.21.15.46:0/1609658954 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 36) ==== 8362+0+0 (secure 0 0 0) 0x7f4cd0024410 con 0x56485d8cb8d0 2024-03-31T23:24:03.466 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:03.453+0000 7f4cdf40c740 10 client.17160 fetch_fsmap finished waiting for FSMap version 36 2024-03-31T23:24:03.466 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:03.453+0000 7f4cdf40c740 10 client.17160 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-31T23:24:03.466 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:03.453+0000 7f4cdf40c740 20 client.17160 populate_metadata read hostname 'smithi046' 2024-03-31T23:24:03.466 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:03.453+0000 7f4cdf40c740 1 --2- 172.21.15.46:0/1609658954 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x56485d94b690 0x56485d94da80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:24:03.466 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:03.453+0000 7f4cdf40c740 4 client.17160 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-31T23:24:03.466 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:03.453+0000 7f4cdf40c740 1 -- 172.21.15.46:0/1609658954 --> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] -- command(tid 0: {"prefix": "flush journal"}) -- 0x56485d839e30 con 0x56485d94b690 2024-03-31T23:24:03.466 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:03.454+0000 7f4cd9cf0640 1 --2- 172.21.15.46:0/1609658954 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x56485d94b690 0x56485d94da80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:24:03.466 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:03.454+0000 7f4cbdffb640 10 client.17160 ms_handle_connect on v2:172.21.15.142:6838/2519143918 2024-03-31T23:24:03.466 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:03.454+0000 7f4cd9cf0640 1 --2- 172.21.15.46:0/1609658954 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x56485d94b690 0x56485d94da80 crc :-1 s=READY pgs=186 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e75cf7fe73caaa86 server_cookie=563a24f8adbe0a9 in_seq=0 out_seq=0 2024-03-31T23:24:03.485 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:03.484+0000 7f4cbdffb640 1 -- 172.21.15.46:0/1609658954 <== mds.0 v2:172.21.15.142:6838/2519143918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x56485d839e30 con 0x56485d94b690 2024-03-31T23:24:03.485 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:03.484+0000 7f4cbdffb640 10 client.17160 handle_command_reply: tid=0 2024-03-31T23:24:03.485 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:03.484+0000 7f4cdf40c740 1 client.17160 shutdown 2024-03-31T23:24:03.485 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:03.484+0000 7f4cdf40c740 1 -- 172.21.15.46:0/1609658954 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56485d8cb8d0 msgr2=0x56485d8dd290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:24:03.485 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:03.484+0000 7f4cdf40c740 1 --2- 172.21.15.46:0/1609658954 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56485d8cb8d0 0x56485d8dd290 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7f4cd0009bf0 tx=0x7f4cd0011dc0 comp rx=0 tx=0).stop 2024-03-31T23:24:03.485 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:03.484+0000 7f4cdf40c740 1 -- 172.21.15.46:0/1609658954 shutdown_connections 2024-03-31T23:24:03.486 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:03.484+0000 7f4cdf40c740 1 --2- 172.21.15.46:0/1609658954 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x56485d94b690 0x56485d94da80 crc :-1 s=READY pgs=186 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:24:03.486 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:03.485+0000 7f4cdf40c740 1 --2- 172.21.15.46:0/1609658954 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56485d8d4fc0 0x56485d8ced50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:24:03.486 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:03.485+0000 7f4cdf40c740 1 --2- 172.21.15.46:0/1609658954 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56485d8cc1f0 0x56485d8ce7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:24:03.486 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:03.485+0000 7f4cbdffb640 0 client.17160 ms_handle_reset on v2:172.21.15.142:6838/2519143918 2024-03-31T23:24:03.486 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:03.485+0000 7f4cdf40c740 1 --2- 172.21.15.46:0/1609658954 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56485d8cb8d0 0x56485d8dd290 unknown :-1 s=CLOSED pgs=856 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:24:03.486 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:03.485+0000 7f4cdf40c740 1 -- 172.21.15.46:0/1609658954 >> 172.21.15.46:0/1609658954 conn(0x56485d8c61f0 msgr2=0x56485d8c26f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:24:03.486 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:03.485+0000 7f4cdf40c740 1 -- 172.21.15.46:0/1609658954 shutdown_connections 2024-03-31T23:24:03.486 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:03.485+0000 7f4cdf40c740 1 -- 172.21.15.46:0/1609658954 wait complete. 2024-03-31T23:24:03.486 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:03.485+0000 7f4cdf40c740 20 client.17160 trim_cache size 0 max 16384 2024-03-31T23:24:03.487 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout:{ 2024-03-31T23:24:03.487 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout: "message": "", 2024-03-31T23:24:03.487 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout: "return_code": 0 2024-03-31T23:24:03.487 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout:} 2024-03-31T23:24:03.488 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:03.485+0000 7f4cdc755640 1 -- 172.21.15.46:0/2970497671 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f4cc00a26b0 msgr2=0x7f4cc00a4b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:24:03.488 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:03.485+0000 7f4cdc755640 1 --2- 172.21.15.46:0/2970497671 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f4cc00a26b0 0x7f4cc00a4b70 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f4ccc009a60 tx=0x7f4ccc005c50 comp rx=0 tx=0).stop 2024-03-31T23:24:03.488 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:03.485+0000 7f4cdc755640 1 -- 172.21.15.46:0/2970497671 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4cd4101240 msgr2=0x7f4cd41972f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:24:03.488 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:03.485+0000 7f4cdc755640 1 --2- 172.21.15.46:0/2970497671 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4cd4101240 0x7f4cd41972f0 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7f4cc400e9f0 tx=0x7f4cc400edf0 comp rx=0 tx=0).stop 2024-03-31T23:24:03.488 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:03.485+0000 7f4cdc755640 1 -- 172.21.15.46:0/2970497671 shutdown_connections 2024-03-31T23:24:03.488 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:03.485+0000 7f4cdc755640 1 --2- 172.21.15.46:0/2970497671 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f4cc00a26b0 0x7f4cc00a4b70 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:24:03.488 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:03.485+0000 7f4cdc755640 1 --2- 172.21.15.46:0/2970497671 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4cd41055d0 0x7f4cd419b670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:24:03.488 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:03.485+0000 7f4cdc755640 1 --2- 172.21.15.46:0/2970497671 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4cd4101240 0x7f4cd41972f0 unknown :-1 s=CLOSED pgs=853 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:24:03.488 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:03.485+0000 7f4cdc755640 1 --2- 172.21.15.46:0/2970497671 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4cd41008a0 0x7f4cd4196db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:24:03.488 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:03.485+0000 7f4cdc755640 1 -- 172.21.15.46:0/2970497671 >> 172.21.15.46:0/2970497671 conn(0x7f4cd4077420 msgr2=0x7f4cd40fe390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:24:03.488 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:03.486+0000 7f4cdc755640 1 -- 172.21.15.46:0/2970497671 shutdown_connections 2024-03-31T23:24:03.489 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:03.486+0000 7f4cdc755640 1 -- 172.21.15.46:0/2970497671 wait complete. 2024-03-31T23:24:04.220 INFO:tasks.workunit.client.0.smithi046.stdout: 1 237831 1.80 MB/sec execute 550 sec latency 1205.487 ms 2024-03-31T23:24:04.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:24:04 smithi142 ceph-mon[35271]: pgmap v385: 129 pgs: 129 active+clean; 138 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 279 KiB/s rd, 742 KiB/s wr, 77 op/s 2024-03-31T23:24:05.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:24:04 smithi162 ceph-mon[36237]: pgmap v385: 129 pgs: 129 active+clean; 138 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 279 KiB/s rd, 742 KiB/s wr, 77 op/s 2024-03-31T23:24:05.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:24:04 smithi046 ceph-mon[32498]: pgmap v385: 129 pgs: 129 active+clean; 138 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 279 KiB/s rd, 742 KiB/s wr, 77 op/s 2024-03-31T23:24:05.220 INFO:tasks.workunit.client.0.smithi046.stdout: 1 237831 1.80 MB/sec execute 551 sec latency 2205.536 ms 2024-03-31T23:24:06.220 INFO:tasks.workunit.client.0.smithi046.stdout: 1 237831 1.80 MB/sec execute 552 sec latency 3205.640 ms 2024-03-31T23:24:06.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:24:06 smithi142 ceph-mon[35271]: pgmap v386: 129 pgs: 129 active+clean; 136 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 405 KiB/s rd, 1.1 MiB/s wr, 126 op/s 2024-03-31T23:24:07.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:24:06 smithi162 ceph-mon[36237]: pgmap v386: 129 pgs: 129 active+clean; 136 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 405 KiB/s rd, 1.1 MiB/s wr, 126 op/s 2024-03-31T23:24:07.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:24:06 smithi046 ceph-mon[32498]: pgmap v386: 129 pgs: 129 active+clean; 136 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 405 KiB/s rd, 1.1 MiB/s wr, 126 op/s 2024-03-31T23:24:07.220 INFO:tasks.workunit.client.0.smithi046.stdout: 1 237831 1.79 MB/sec execute 553 sec latency 4205.711 ms 2024-03-31T23:24:08.222 INFO:tasks.workunit.client.0.smithi046.stdout: 1 237957 1.79 MB/sec execute 554 sec latency 4732.837 ms 2024-03-31T23:24:08.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:24:08 smithi142 ceph-mon[35271]: pgmap v387: 129 pgs: 129 active+clean; 136 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 354 KiB/s rd, 914 KiB/s wr, 104 op/s 2024-03-31T23:24:08.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:24:08 smithi142 ceph-mon[35271]: osdmap e109: 12 total, 12 up, 12 in 2024-03-31T23:24:09.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:24:08 smithi162 ceph-mon[36237]: pgmap v387: 129 pgs: 129 active+clean; 136 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 354 KiB/s rd, 914 KiB/s wr, 104 op/s 2024-03-31T23:24:09.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:24:08 smithi162 ceph-mon[36237]: osdmap e109: 12 total, 12 up, 12 in 2024-03-31T23:24:09.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:24:08 smithi046 ceph-mon[32498]: pgmap v387: 129 pgs: 129 active+clean; 136 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 354 KiB/s rd, 914 KiB/s wr, 104 op/s 2024-03-31T23:24:09.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:24:08 smithi046 ceph-mon[32498]: osdmap e109: 12 total, 12 up, 12 in 2024-03-31T23:24:09.220 INFO:tasks.workunit.client.0.smithi046.stdout: 1 238184 1.79 MB/sec execute 555 sec latency 15.492 ms 2024-03-31T23:24:10.220 INFO:tasks.workunit.client.0.smithi046.stdout: 1 238570 1.79 MB/sec execute 556 sec latency 17.499 ms 2024-03-31T23:24:10.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:24:10 smithi142 ceph-mon[35271]: pgmap v389: 129 pgs: 129 active+clean; 137 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 220 KiB/s rd, 675 KiB/s wr, 79 op/s 2024-03-31T23:24:10.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:24:10 smithi142 ceph-mon[35271]: osdmap e110: 12 total, 12 up, 12 in 2024-03-31T23:24:11.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:24:10 smithi162 ceph-mon[36237]: pgmap v389: 129 pgs: 129 active+clean; 137 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 220 KiB/s rd, 675 KiB/s wr, 79 op/s 2024-03-31T23:24:11.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:24:10 smithi162 ceph-mon[36237]: osdmap e110: 12 total, 12 up, 12 in 2024-03-31T23:24:11.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:24:10 smithi046 ceph-mon[32498]: pgmap v389: 129 pgs: 129 active+clean; 137 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 220 KiB/s rd, 675 KiB/s wr, 79 op/s 2024-03-31T23:24:11.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:24:10 smithi046 ceph-mon[32498]: osdmap e110: 12 total, 12 up, 12 in 2024-03-31T23:24:11.220 INFO:tasks.workunit.client.0.smithi046.stdout: 1 239079 1.79 MB/sec execute 557 sec latency 16.995 ms 2024-03-31T23:24:12.220 INFO:tasks.workunit.client.0.smithi046.stdout: 1 239510 1.79 MB/sec execute 558 sec latency 19.947 ms 2024-03-31T23:24:12.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:24:12 smithi142 ceph-mon[35271]: pgmap v391: 129 pgs: 129 active+clean; 137 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 275 KiB/s rd, 844 KiB/s wr, 99 op/s 2024-03-31T23:24:13.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:24:12 smithi162 ceph-mon[36237]: pgmap v391: 129 pgs: 129 active+clean; 137 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 275 KiB/s rd, 844 KiB/s wr, 99 op/s 2024-03-31T23:24:13.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:24:12 smithi046 ceph-mon[32498]: pgmap v391: 129 pgs: 129 active+clean; 137 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 275 KiB/s rd, 844 KiB/s wr, 99 op/s 2024-03-31T23:24:13.220 INFO:tasks.workunit.client.0.smithi046.stdout: 1 240058 1.79 MB/sec execute 559 sec latency 24.323 ms 2024-03-31T23:24:14.220 INFO:tasks.workunit.client.0.smithi046.stdout: 1 240583 1.80 MB/sec execute 560 sec latency 16.828 ms 2024-03-31T23:24:14.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:24:14 smithi142 ceph-mon[35271]: pgmap v392: 129 pgs: 129 active+clean; 137 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 276 KiB/s wr, 24 op/s 2024-03-31T23:24:15.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:24:14 smithi162 ceph-mon[36237]: pgmap v392: 129 pgs: 129 active+clean; 137 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 276 KiB/s wr, 24 op/s 2024-03-31T23:24:15.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:24:14 smithi046 ceph-mon[32498]: pgmap v392: 129 pgs: 129 active+clean; 137 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 276 KiB/s wr, 24 op/s 2024-03-31T23:24:15.220 INFO:tasks.workunit.client.0.smithi046.stdout: 1 241017 1.80 MB/sec execute 561 sec latency 15.892 ms 2024-03-31T23:24:16.220 INFO:tasks.workunit.client.0.smithi046.stdout: 1 241310 1.80 MB/sec execute 562 sec latency 404.697 ms 2024-03-31T23:24:16.586 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:16.583+0000 7f5e5b93e640 1 -- 172.21.15.46:0/3306919897 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5e54074420 msgr2=0x7f5e5410da80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:24:16.586 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:16.583+0000 7f5e5b93e640 1 --2- 172.21.15.46:0/3306919897 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5e54074420 0x7f5e5410da80 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f5e44009860 tx=0x7f5e4402bc00 comp rx=0 tx=0).stop 2024-03-31T23:24:16.586 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:16.583+0000 7f5e5b93e640 1 -- 172.21.15.46:0/3306919897 shutdown_connections 2024-03-31T23:24:16.586 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:16.583+0000 7f5e5b93e640 1 --2- 172.21.15.46:0/3306919897 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5e5410dfc0 0x7f5e541103b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:24:16.586 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:16.583+0000 7f5e5b93e640 1 --2- 172.21.15.46:0/3306919897 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5e54074420 0x7f5e5410da80 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:24:16.587 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:16.583+0000 7f5e5b93e640 1 --2- 172.21.15.46:0/3306919897 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5e54073b00 0x7f5e54073ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:24:16.587 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:16.583+0000 7f5e5b93e640 1 -- 172.21.15.46:0/3306919897 >> 172.21.15.46:0/3306919897 conn(0x7f5e540fd210 msgr2=0x7f5e540ff630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:24:16.587 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:16.584+0000 7f5e5b93e640 1 -- 172.21.15.46:0/3306919897 shutdown_connections 2024-03-31T23:24:16.587 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:16.584+0000 7f5e5b93e640 1 -- 172.21.15.46:0/3306919897 wait complete. 2024-03-31T23:24:16.587 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:16.584+0000 7f5e5b93e640 1 Processor -- start 2024-03-31T23:24:16.587 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:16.584+0000 7f5e5b93e640 1 -- start start 2024-03-31T23:24:16.587 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:16.584+0000 7f5e5b93e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5e54073b00 0x7f5e5419a0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:24:16.587 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:16.584+0000 7f5e5b93e640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5e54074420 0x7f5e5419a5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:24:16.587 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:16.584+0000 7f5e5b93e640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5e5410dfc0 0x7f5e5419e960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:24:16.587 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:16.584+0000 7f5e5b93e640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f5e54113b40 con 0x7f5e54073b00 2024-03-31T23:24:16.587 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:16.584+0000 7f5e5b93e640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f5e541139c0 con 0x7f5e5410dfc0 2024-03-31T23:24:16.587 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:16.584+0000 7f5e5b93e640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f5e54113cc0 con 0x7f5e54074420 2024-03-31T23:24:16.587 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:16.584+0000 7f5e596da640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5e54073b00 0x7f5e5419a0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:24:16.588 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:16.585+0000 7f5e59edb640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5e5410dfc0 0x7f5e5419e960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:24:16.588 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:16.585+0000 7f5e596da640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5e54073b00 0x7f5e5419a0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:34670/0 (socket says 172.21.15.46:34670) 2024-03-31T23:24:16.588 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:16.585+0000 7f5e596da640 1 -- 172.21.15.46:0/2054312580 learned_addr learned my addr 172.21.15.46:0/2054312580 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:24:16.588 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:16.585+0000 7f5e58ed9640 1 --2- 172.21.15.46:0/2054312580 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5e54074420 0x7f5e5419a5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:24:16.588 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:16.585+0000 7f5e59edb640 1 -- 172.21.15.46:0/2054312580 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5e54074420 msgr2=0x7f5e5419a5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:24:16.588 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:16.585+0000 7f5e59edb640 1 --2- 172.21.15.46:0/2054312580 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5e54074420 0x7f5e5419a5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:24:16.588 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:16.585+0000 7f5e59edb640 1 -- 172.21.15.46:0/2054312580 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5e54073b00 msgr2=0x7f5e5419a0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:24:16.588 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:16.585+0000 7f5e59edb640 1 --2- 172.21.15.46:0/2054312580 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5e54073b00 0x7f5e5419a0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:24:16.588 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:16.585+0000 7f5e59edb640 1 -- 172.21.15.46:0/2054312580 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5e5419f090 con 0x7f5e5410dfc0 2024-03-31T23:24:16.588 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:16.585+0000 7f5e596da640 1 --2- 172.21.15.46:0/2054312580 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5e54073b00 0x7f5e5419a0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:24:16.588 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:16.585+0000 7f5e59edb640 1 --2- 172.21.15.46:0/2054312580 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5e5410dfc0 0x7f5e5419e960 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f5e4c009fd0 tx=0x7f5e4c00ef10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:24:16.588 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:16.585+0000 7f5e4a7fc640 1 -- 172.21.15.46:0/2054312580 <== mon.2 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e4c00ce20 con 0x7f5e5410dfc0 2024-03-31T23:24:16.588 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:16.585+0000 7f5e5b93e640 1 -- 172.21.15.46:0/2054312580 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5e5419f380 con 0x7f5e5410dfc0 2024-03-31T23:24:16.588 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:16.585+0000 7f5e4a7fc640 1 -- 172.21.15.46:0/2054312580 <== mon.2 v2:172.21.15.142:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5e4c005020 con 0x7f5e5410dfc0 2024-03-31T23:24:16.589 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:16.585+0000 7f5e4a7fc640 1 -- 172.21.15.46:0/2054312580 <== mon.2 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e4c010690 con 0x7f5e5410dfc0 2024-03-31T23:24:16.589 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:16.585+0000 7f5e5b93e640 1 -- 172.21.15.46:0/2054312580 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5e541a6c10 con 0x7f5e5410dfc0 2024-03-31T23:24:16.595 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:16.588+0000 7f5e4a7fc640 1 -- 172.21.15.46:0/2054312580 <== mon.2 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 25) ==== 137451+0+0 (secure 0 0 0) 0x7f5e4c0051d0 con 0x7f5e5410dfc0 2024-03-31T23:24:16.596 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:16.588+0000 7f5e5e5f8740 1 Processor -- start 2024-03-31T23:24:16.596 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:16.588+0000 7f5e5e5f8740 1 -- start start 2024-03-31T23:24:16.596 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:16.588+0000 7f5e5e5f8740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56034139f6c0 0x56034139faa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:24:16.596 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:16.588+0000 7f5e5e5f8740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56034139ffe0 0x5603413a8870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:24:16.596 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:16.588+0000 7f5e5e5f8740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5603413a8db0 0x5603413ab1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:24:16.596 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:16.588+0000 7f5e5e5f8740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x5603413028f0 con 0x56034139ffe0 2024-03-31T23:24:16.596 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:16.588+0000 7f5e5e5f8740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x5603412ab8d0 con 0x5603413a8db0 2024-03-31T23:24:16.596 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:16.588+0000 7f5e5e5f8740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5603412fe990 con 0x56034139f6c0 2024-03-31T23:24:16.596 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:16.589+0000 7f5e596da640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56034139ffe0 0x5603413a8870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:24:16.596 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:16.589+0000 7f5e58ed9640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5603413a8db0 0x5603413ab1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:24:16.596 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:16.589+0000 7f5e596da640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56034139ffe0 0x5603413a8870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:34686/0 (socket says 172.21.15.46:34686) 2024-03-31T23:24:16.597 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:16.589+0000 7f5e596da640 1 -- 172.21.15.46:0/2079799976 learned_addr learned my addr 172.21.15.46:0/2079799976 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:24:16.597 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:16.589+0000 7f5e4a7fc640 1 --2- 172.21.15.46:0/2054312580 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f5e400a26b0 0x7f5e400a4b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:24:16.597 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:16.589+0000 7f5e58ed9640 1 --2- 172.21.15.46:0/2079799976 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56034139f6c0 0x56034139faa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:24:16.597 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:16.589+0000 7f5e59edb640 1 --2- 172.21.15.46:0/2054312580 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f5e400a26b0 0x7f5e400a4b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:24:16.597 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:16.589+0000 7f5e596da640 1 -- 172.21.15.46:0/2079799976 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56034139f6c0 msgr2=0x56034139faa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:24:16.597 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:16.589+0000 7f5e596da640 1 --2- 172.21.15.46:0/2079799976 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56034139f6c0 0x56034139faa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:24:16.597 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:16.589+0000 7f5e596da640 1 -- 172.21.15.46:0/2079799976 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5603413a8db0 msgr2=0x5603413ab1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:24:16.597 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:16.589+0000 7f5e596da640 1 --2- 172.21.15.46:0/2079799976 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5603413a8db0 0x5603413ab1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:24:16.597 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:16.589+0000 7f5e596da640 1 -- 172.21.15.46:0/2079799976 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560341017210 con 0x56034139ffe0 2024-03-31T23:24:16.597 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:16.589+0000 7f5e58ed9640 1 --2- 172.21.15.46:0/2079799976 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5603413a8db0 0x5603413ab1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:24:16.597 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:16.589+0000 7f5e4a7fc640 1 -- 172.21.15.46:0/2054312580 <== mon.2 v2:172.21.15.142:3300/0 5 ==== osd_map(110..110 src has 1..110) ==== 9554+0+0 (secure 0 0 0) 0x7f5e4c0cf560 con 0x7f5e5410dfc0 2024-03-31T23:24:16.597 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:16.589+0000 7f5e596da640 1 --2- 172.21.15.46:0/2079799976 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56034139ffe0 0x5603413a8870 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7f5e500115c0 tx=0x7f5e50011a80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5962f5947a475cb6 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:24:16.598 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:16.589+0000 7f5e59edb640 1 --2- 172.21.15.46:0/2054312580 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f5e400a26b0 0x7f5e400a4b70 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f5e4c0982a0 tx=0x7f5e4c096040 comp rx=0 tx=0).ready entity=mgr.15654 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:24:16.598 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:16.589+0000 7f5e3f7fe640 1 -- 172.21.15.46:0/2079799976 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e5000f6c0 con 0x56034139ffe0 2024-03-31T23:24:16.598 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:16.589+0000 7f5e3f7fe640 1 -- 172.21.15.46:0/2079799976 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5e5000fd10 con 0x56034139ffe0 2024-03-31T23:24:16.598 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:16.589+0000 7f5e3f7fe640 1 -- 172.21.15.46:0/2079799976 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e50019cf0 con 0x56034139ffe0 2024-03-31T23:24:16.598 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:16.590+0000 7f5e5e5f8740 1 -- 172.21.15.46:0/2079799976 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56034139ffe0 msgr2=0x5603413a8870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:24:16.598 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:16.590+0000 7f5e5e5f8740 1 --2- 172.21.15.46:0/2079799976 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56034139ffe0 0x5603413a8870 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7f5e500115c0 tx=0x7f5e50011a80 comp rx=0 tx=0).stop 2024-03-31T23:24:16.598 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:16.590+0000 7f5e5e5f8740 1 -- 172.21.15.46:0/2079799976 shutdown_connections 2024-03-31T23:24:16.598 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:16.590+0000 7f5e5e5f8740 1 --2- 172.21.15.46:0/2079799976 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5603413a8db0 0x5603413ab1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:24:16.598 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:16.590+0000 7f5e5e5f8740 1 --2- 172.21.15.46:0/2079799976 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56034139ffe0 0x5603413a8870 unknown :-1 s=CLOSED pgs=857 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:24:16.598 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:16.590+0000 7f5e5e5f8740 1 --2- 172.21.15.46:0/2079799976 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56034139f6c0 0x56034139faa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:24:16.598 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:16.590+0000 7f5e5e5f8740 1 -- 172.21.15.46:0/2079799976 >> 172.21.15.46:0/2079799976 conn(0x560341399fe0 msgr2=0x56034139a3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:24:16.598 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:16.590+0000 7f5e5e5f8740 1 -- 172.21.15.46:0/2079799976 shutdown_connections 2024-03-31T23:24:16.598 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:16.590+0000 7f5e5e5f8740 1 -- 172.21.15.46:0/2079799976 wait complete. 2024-03-31T23:24:16.599 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:16.590+0000 7f5e5e5f8740 1 -- start start 2024-03-31T23:24:16.599 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:16.591+0000 7f5e5e5f8740 1 Processor -- start 2024-03-31T23:24:16.599 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:16.591+0000 7f5e5e5f8740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56034139f6c0 0x5603413a5d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:24:16.599 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:16.591+0000 7f5e5e5f8740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56034139ffe0 0x5603413a62c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:24:16.599 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:16.591+0000 7f5e5e5f8740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5603413a8db0 0x5603413a6800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:24:16.599 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:16.591+0000 7f5e5e5f8740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x5603413ab9e0 con 0x56034139ffe0 2024-03-31T23:24:16.599 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:16.591+0000 7f5e5e5f8740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x5603412a0f80 con 0x5603413a8db0 2024-03-31T23:24:16.599 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:16.591+0000 7f5e5e5f8740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5603413abb60 con 0x56034139f6c0 2024-03-31T23:24:16.599 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:16.591+0000 7f5e5e5f8740 10 client.0 _ll_register_callbacks cb 0x5603412f9290 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T23:24:16.599 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:16.591+0000 7f5e596da640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56034139ffe0 0x5603413a62c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:24:16.600 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:16.591+0000 7f5e596da640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56034139ffe0 0x5603413a62c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:34688/0 (socket says 172.21.15.46:34688) 2024-03-31T23:24:16.600 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:16.591+0000 7f5e596da640 1 -- 172.21.15.46:0/978161054 learned_addr learned my addr 172.21.15.46:0/978161054 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:24:16.600 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:16.591+0000 7f5e58ed9640 1 --2- 172.21.15.46:0/978161054 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5603413a8db0 0x5603413a6800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:24:16.600 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:16.591+0000 7f5e58ed9640 1 --2- 172.21.15.46:0/978161054 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56034139f6c0 0x5603413a5d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:24:16.600 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:16.592+0000 7f5e596da640 1 -- 172.21.15.46:0/978161054 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56034139f6c0 msgr2=0x5603413a5d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:24:16.600 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:16.592+0000 7f5e596da640 1 --2- 172.21.15.46:0/978161054 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56034139f6c0 0x5603413a5d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:24:16.600 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:16.592+0000 7f5e596da640 1 -- 172.21.15.46:0/978161054 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5603413a8db0 msgr2=0x5603413a6800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:24:16.600 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:16.592+0000 7f5e596da640 1 --2- 172.21.15.46:0/978161054 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5603413a8db0 0x5603413a6800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:24:16.600 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:16.592+0000 7f5e596da640 1 -- 172.21.15.46:0/978161054 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5603410e3620 con 0x56034139ffe0 2024-03-31T23:24:16.600 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:16.592+0000 7f5e596da640 1 --2- 172.21.15.46:0/978161054 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56034139ffe0 0x5603413a62c0 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7f5e5000f8f0 tx=0x7f5e5000f920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=658b75b5b78d975e server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:24:16.600 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:16.592+0000 7f5e3cff9640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-31T23:24:16.600 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:16.592+0000 7f5e3cff9640 1 -- 172.21.15.46:0/978161054 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e50019e60 con 0x56034139ffe0 2024-03-31T23:24:16.601 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:16.592+0000 7f5e3cff9640 1 -- 172.21.15.46:0/978161054 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(110..110 src has 1..110) ==== 9554+0+0 (secure 0 0 0) 0x7f5e50004050 con 0x56034139ffe0 2024-03-31T23:24:16.601 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:16.592+0000 7f5e5e5f8740 1 -- 172.21.15.46:0/978161054 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5e5000fd10 con 0x56034139ffe0 2024-03-31T23:24:16.601 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:16.592+0000 7f5e3cff9640 1 -- 172.21.15.46:0/978161054 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f5e50030050 con 0x56034139ffe0 2024-03-31T23:24:16.601 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:16.592+0000 7f5e5e5f8740 10 client.17172 fetch_fsmap learned FSMap version 36 2024-03-31T23:24:16.601 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:16.592+0000 7f5e5e5f8740 1 -- 172.21.15.46:0/978161054 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x5603413b7830 con 0x56034139ffe0 2024-03-31T23:24:16.601 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:16.592+0000 7f5e3cff9640 1 -- 172.21.15.46:0/978161054 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5e5000fd10 con 0x56034139ffe0 2024-03-31T23:24:16.601 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:16.593+0000 7f5e3cff9640 1 -- 172.21.15.46:0/978161054 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 36) ==== 8362+0+0 (secure 0 0 0) 0x7f5e500046c0 con 0x56034139ffe0 2024-03-31T23:24:16.601 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:16.593+0000 7f5e5e5f8740 10 client.17172 fetch_fsmap finished waiting for FSMap version 36 2024-03-31T23:24:16.601 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:16.593+0000 7f5e5e5f8740 10 client.17172 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-31T23:24:16.601 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:16.593+0000 7f5e5e5f8740 20 client.17172 populate_metadata read hostname 'smithi046' 2024-03-31T23:24:16.601 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:16.593+0000 7f5e5e5f8740 1 --2- 172.21.15.46:0/978161054 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x5603413b8c90 0x5603413bb080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:24:16.601 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:16.593+0000 7f5e5e5f8740 4 client.17172 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-31T23:24:16.601 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:16.593+0000 7f5e5e5f8740 1 -- 172.21.15.46:0/978161054 --> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560341316120 con 0x5603413b8c90 2024-03-31T23:24:16.601 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:16.593+0000 7f5e58ed9640 1 --2- 172.21.15.46:0/978161054 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x5603413b8c90 0x5603413bb080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:24:16.602 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:16.593+0000 7f5e3cff9640 10 client.17172 ms_handle_connect on v2:172.21.15.142:6838/2519143918 2024-03-31T23:24:16.602 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:16.593+0000 7f5e58ed9640 1 --2- 172.21.15.46:0/978161054 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x5603413b8c90 0x5603413bb080 crc :-1 s=READY pgs=187 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=80a7f6bea3d31e14 server_cookie=4dc4cb42aee71f17 in_seq=0 out_seq=0 2024-03-31T23:24:16.602 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:16.594+0000 7f5e3cff9640 1 -- 172.21.15.46:0/978161054 <== mds.0 v2:172.21.15.142:6838/2519143918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x560341316120 con 0x5603413b8c90 2024-03-31T23:24:16.602 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:16.594+0000 7f5e3cff9640 10 client.17172 handle_command_reply: tid=0 2024-03-31T23:24:16.602 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:16.594+0000 7f5e5e5f8740 1 client.17172 shutdown 2024-03-31T23:24:16.602 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:16.594+0000 7f5e5e5f8740 1 -- 172.21.15.46:0/978161054 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56034139ffe0 msgr2=0x5603413a62c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:24:16.602 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:16.594+0000 7f5e5e5f8740 1 --2- 172.21.15.46:0/978161054 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56034139ffe0 0x5603413a62c0 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7f5e5000f8f0 tx=0x7f5e5000f920 comp rx=0 tx=0).stop 2024-03-31T23:24:16.602 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:16.594+0000 7f5e5e5f8740 1 -- 172.21.15.46:0/978161054 shutdown_connections 2024-03-31T23:24:16.602 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:16.594+0000 7f5e5e5f8740 1 --2- 172.21.15.46:0/978161054 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x5603413b8c90 0x5603413bb080 crc :-1 s=READY pgs=187 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:24:16.602 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:16.594+0000 7f5e5e5f8740 1 --2- 172.21.15.46:0/978161054 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5603413a8db0 0x5603413a6800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:24:16.602 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:16.594+0000 7f5e5e5f8740 1 --2- 172.21.15.46:0/978161054 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56034139ffe0 0x5603413a62c0 unknown :-1 s=CLOSED pgs=858 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:24:16.602 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:16.594+0000 7f5e5e5f8740 1 --2- 172.21.15.46:0/978161054 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56034139f6c0 0x5603413a5d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:24:16.603 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:16.594+0000 7f5e3cff9640 0 client.17172 ms_handle_reset on v2:172.21.15.142:6838/2519143918 2024-03-31T23:24:16.603 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:16.594+0000 7f5e5e5f8740 1 -- 172.21.15.46:0/978161054 >> 172.21.15.46:0/978161054 conn(0x560341399fe0 msgr2=0x56034139bd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:24:16.603 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:16.594+0000 7f5e5e5f8740 1 -- 172.21.15.46:0/978161054 shutdown_connections 2024-03-31T23:24:16.603 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:16.594+0000 7f5e5e5f8740 1 -- 172.21.15.46:0/978161054 wait complete. 2024-03-31T23:24:16.603 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:16.594+0000 7f5e5e5f8740 20 client.17172 trim_cache size 0 max 16384 2024-03-31T23:24:16.614 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:16.609+0000 7f5e5e5f8740 1 Processor -- start 2024-03-31T23:24:16.614 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:16.609+0000 7f5e5e5f8740 1 -- start start 2024-03-31T23:24:16.614 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:16.609+0000 7f5e5e5f8740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56034139f6c0 0x5603413a84b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:24:16.614 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:16.609+0000 7f5e5e5f8740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56034139ffe0 0x5603413b4780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:24:16.614 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:16.609+0000 7f5e5e5f8740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5603413a8db0 0x5603413c0930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:24:16.614 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:16.609+0000 7f5e5e5f8740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f5e541b3810 con 0x56034139f6c0 2024-03-31T23:24:16.614 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:16.609+0000 7f5e5e5f8740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f5e541b1080 con 0x5603413a8db0 2024-03-31T23:24:16.617 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:16.609+0000 7f5e5e5f8740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f5e541aeef0 con 0x56034139ffe0 2024-03-31T23:24:16.618 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:16.609+0000 7f5e58ed9640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56034139f6c0 0x5603413a84b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:24:16.618 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:16.610+0000 7f5e58ed9640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56034139f6c0 0x5603413a84b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:34700/0 (socket says 172.21.15.46:34700) 2024-03-31T23:24:16.618 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:16.610+0000 7f5e58ed9640 1 -- 172.21.15.46:0/69199360 learned_addr learned my addr 172.21.15.46:0/69199360 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:24:16.618 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:16.610+0000 7f5e596da640 1 --2- 172.21.15.46:0/69199360 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56034139ffe0 0x5603413b4780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:24:16.618 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:16.610+0000 7f5e58ed9640 1 --2- 172.21.15.46:0/69199360 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5603413a8db0 0x5603413c0930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:24:16.618 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:16.610+0000 7f5e58ed9640 1 -- 172.21.15.46:0/69199360 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56034139ffe0 msgr2=0x5603413b4780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:24:16.618 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:16.610+0000 7f5e58ed9640 1 --2- 172.21.15.46:0/69199360 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56034139ffe0 0x5603413b4780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:24:16.618 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:16.610+0000 7f5e58ed9640 1 -- 172.21.15.46:0/69199360 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5603413a8db0 msgr2=0x5603413c0930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:24:16.618 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:16.610+0000 7f5e58ed9640 1 --2- 172.21.15.46:0/69199360 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5603413a8db0 0x5603413c0930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:24:16.618 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:16.610+0000 7f5e58ed9640 1 -- 172.21.15.46:0/69199360 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56034139f4f0 con 0x56034139f6c0 2024-03-31T23:24:16.618 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:16.610+0000 7f5e58ed9640 1 --2- 172.21.15.46:0/69199360 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56034139f6c0 0x5603413a84b0 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7f5e4402c140 tx=0x7f5e440058b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e8119d4a18981592 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:24:16.618 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:16.610+0000 7f5e3effd640 1 -- 172.21.15.46:0/69199360 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e44055070 con 0x56034139f6c0 2024-03-31T23:24:16.619 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:16.610+0000 7f5e3effd640 1 -- 172.21.15.46:0/69199360 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5e4403a040 con 0x56034139f6c0 2024-03-31T23:24:16.619 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:16.610+0000 7f5e3effd640 1 -- 172.21.15.46:0/69199360 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e44039040 con 0x56034139f6c0 2024-03-31T23:24:16.619 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:16.611+0000 7f5e5e5f8740 1 -- 172.21.15.46:0/69199360 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56034139f6c0 msgr2=0x5603413a84b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:24:16.619 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:16.611+0000 7f5e5e5f8740 1 --2- 172.21.15.46:0/69199360 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56034139f6c0 0x5603413a84b0 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7f5e4402c140 tx=0x7f5e440058b0 comp rx=0 tx=0).stop 2024-03-31T23:24:16.619 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:16.611+0000 7f5e5e5f8740 1 -- 172.21.15.46:0/69199360 shutdown_connections 2024-03-31T23:24:16.619 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:16.611+0000 7f5e5e5f8740 1 --2- 172.21.15.46:0/69199360 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5603413a8db0 0x5603413c0930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:24:16.619 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:16.611+0000 7f5e5e5f8740 1 --2- 172.21.15.46:0/69199360 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56034139ffe0 0x5603413b4780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:24:16.619 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:16.611+0000 7f5e5e5f8740 1 --2- 172.21.15.46:0/69199360 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56034139f6c0 0x5603413a84b0 unknown :-1 s=CLOSED pgs=859 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:24:16.619 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:16.611+0000 7f5e5e5f8740 1 -- 172.21.15.46:0/69199360 >> 172.21.15.46:0/69199360 conn(0x560341399fe0 msgr2=0x56034139a3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:24:16.619 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:16.611+0000 7f5e5e5f8740 1 -- 172.21.15.46:0/69199360 shutdown_connections 2024-03-31T23:24:16.619 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:16.611+0000 7f5e5e5f8740 1 -- 172.21.15.46:0/69199360 wait complete. 2024-03-31T23:24:16.619 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:16.611+0000 7f5e5e5f8740 1 -- start start 2024-03-31T23:24:16.619 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:16.611+0000 7f5e5e5f8740 1 Processor -- start 2024-03-31T23:24:16.620 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:16.612+0000 7f5e5e5f8740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56034139f6c0 0x5603413b0ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:24:16.620 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:16.612+0000 7f5e5e5f8740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56034139ffe0 0x5603413ac350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:24:16.620 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:16.612+0000 7f5e5e5f8740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5603413a8db0 0x5603413ac890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:24:16.620 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:16.612+0000 7f5e5e5f8740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f5e541af540 con 0x5603413a8db0 2024-03-31T23:24:16.620 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:16.612+0000 7f5e5e5f8740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f5e541b09b0 con 0x56034139ffe0 2024-03-31T23:24:16.620 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:16.612+0000 7f5e5e5f8740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f5e541afc10 con 0x56034139f6c0 2024-03-31T23:24:16.620 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:16.612+0000 7f5e5e5f8740 10 client.0 _ll_register_callbacks cb 0x7f5e50011440 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T23:24:16.620 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:16.612+0000 7f5e58ed9640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5603413a8db0 0x5603413ac890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:24:16.620 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:16.612+0000 7f5e58ed9640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5603413a8db0 0x5603413ac890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:34702/0 (socket says 172.21.15.46:34702) 2024-03-31T23:24:16.620 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:16.612+0000 7f5e58ed9640 1 -- 172.21.15.46:0/3849054541 learned_addr learned my addr 172.21.15.46:0/3849054541 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:24:16.620 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:16.612+0000 7f5e596da640 1 --2- 172.21.15.46:0/3849054541 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56034139ffe0 0x5603413ac350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:24:16.620 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:16.612+0000 7f5e58ed9640 1 --2- 172.21.15.46:0/3849054541 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56034139f6c0 0x5603413b0ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:24:16.620 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:16.612+0000 7f5e58ed9640 1 -- 172.21.15.46:0/3849054541 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56034139f6c0 msgr2=0x5603413b0ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:24:16.620 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:16.612+0000 7f5e58ed9640 1 --2- 172.21.15.46:0/3849054541 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56034139f6c0 0x5603413b0ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:24:16.621 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:16.612+0000 7f5e58ed9640 1 -- 172.21.15.46:0/3849054541 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56034139ffe0 msgr2=0x5603413ac350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:24:16.621 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:16.612+0000 7f5e58ed9640 1 --2- 172.21.15.46:0/3849054541 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56034139ffe0 0x5603413ac350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:24:16.621 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:16.612+0000 7f5e58ed9640 1 -- 172.21.15.46:0/3849054541 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5603413aced0 con 0x5603413a8db0 2024-03-31T23:24:16.621 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:16.612+0000 7f5e596da640 1 --2- 172.21.15.46:0/3849054541 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56034139ffe0 0x5603413ac350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:24:16.621 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:16.612+0000 7f5e58ed9640 1 --2- 172.21.15.46:0/3849054541 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5603413a8db0 0x5603413ac890 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7f5e44050890 tx=0x7f5e44050790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd99011ad9ce9699 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:24:16.621 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:16.612+0000 7f5e3cff9640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-31T23:24:16.621 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:16.613+0000 7f5e3cff9640 1 -- 172.21.15.46:0/3849054541 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e44055070 con 0x5603413a8db0 2024-03-31T23:24:16.621 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:16.613+0000 7f5e3cff9640 1 -- 172.21.15.46:0/3849054541 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(110..110 src has 1..110) ==== 9554+0+0 (secure 0 0 0) 0x7f5e440504c0 con 0x5603413a8db0 2024-03-31T23:24:16.621 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:16.613+0000 7f5e5e5f8740 1 -- 172.21.15.46:0/3849054541 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5e4403a040 con 0x5603413a8db0 2024-03-31T23:24:16.621 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:16.613+0000 7f5e3cff9640 1 -- 172.21.15.46:0/3849054541 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f5e44066050 con 0x5603413a8db0 2024-03-31T23:24:16.621 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:16.613+0000 7f5e5e5f8740 10 client.17178 fetch_fsmap learned FSMap version 36 2024-03-31T23:24:16.621 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:16.613+0000 7f5e5e5f8740 1 -- 172.21.15.46:0/3849054541 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x56034141edc0 con 0x5603413a8db0 2024-03-31T23:24:16.621 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:16.613+0000 7f5e3cff9640 1 -- 172.21.15.46:0/3849054541 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5e4403a040 con 0x5603413a8db0 2024-03-31T23:24:16.621 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:16.613+0000 7f5e3cff9640 1 -- 172.21.15.46:0/3849054541 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 36) ==== 8362+0+0 (secure 0 0 0) 0x7f5e4402c5a0 con 0x5603413a8db0 2024-03-31T23:24:16.621 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:16.613+0000 7f5e5e5f8740 10 client.17178 fetch_fsmap finished waiting for FSMap version 36 2024-03-31T23:24:16.622 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:16.613+0000 7f5e5e5f8740 10 client.17178 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-31T23:24:16.622 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:16.613+0000 7f5e5e5f8740 20 client.17178 populate_metadata read hostname 'smithi046' 2024-03-31T23:24:16.622 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:16.613+0000 7f5e5e5f8740 1 --2- 172.21.15.46:0/3849054541 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x56034141f420 0x560341421840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:24:16.622 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:16.613+0000 7f5e5e5f8740 4 client.17178 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-31T23:24:16.622 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:16.613+0000 7f5e5e5f8740 1 -- 172.21.15.46:0/3849054541 --> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] -- command(tid 0: {"prefix": "flush journal"}) -- 0x56034130fe30 con 0x56034141f420 2024-03-31T23:24:16.622 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:16.613+0000 7f5e596da640 1 --2- 172.21.15.46:0/3849054541 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x56034141f420 0x560341421840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:24:16.622 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:16.614+0000 7f5e3cff9640 10 client.17178 ms_handle_connect on v2:172.21.15.142:6838/2519143918 2024-03-31T23:24:16.622 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:16.614+0000 7f5e596da640 1 --2- 172.21.15.46:0/3849054541 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x56034141f420 0x560341421840 crc :-1 s=READY pgs=188 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bf367f902941c6 server_cookie=1d1d03e24360667d in_seq=0 out_seq=0 2024-03-31T23:24:16.646 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:16.645+0000 7f5e3cff9640 1 -- 172.21.15.46:0/3849054541 <== mds.0 v2:172.21.15.142:6838/2519143918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x56034130fe30 con 0x56034141f420 2024-03-31T23:24:16.646 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:16.645+0000 7f5e3cff9640 10 client.17178 handle_command_reply: tid=0 2024-03-31T23:24:16.646 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:16.645+0000 7f5e5e5f8740 1 client.17178 shutdown 2024-03-31T23:24:16.646 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:16.645+0000 7f5e5e5f8740 1 -- 172.21.15.46:0/3849054541 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5603413a8db0 msgr2=0x5603413ac890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:24:16.646 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:16.645+0000 7f5e5e5f8740 1 --2- 172.21.15.46:0/3849054541 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5603413a8db0 0x5603413ac890 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7f5e44050890 tx=0x7f5e44050790 comp rx=0 tx=0).stop 2024-03-31T23:24:16.646 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:16.645+0000 7f5e5e5f8740 1 -- 172.21.15.46:0/3849054541 shutdown_connections 2024-03-31T23:24:16.647 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:16.645+0000 7f5e5e5f8740 1 --2- 172.21.15.46:0/3849054541 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x56034141f420 0x560341421840 crc :-1 s=READY pgs=188 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:24:16.647 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:16.645+0000 7f5e5e5f8740 1 --2- 172.21.15.46:0/3849054541 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5603413a8db0 0x5603413ac890 unknown :-1 s=CLOSED pgs=860 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:24:16.647 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:16.645+0000 7f5e5e5f8740 1 --2- 172.21.15.46:0/3849054541 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56034139ffe0 0x5603413ac350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:24:16.647 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:16.645+0000 7f5e3cff9640 0 client.17178 ms_handle_reset on v2:172.21.15.142:6838/2519143918 2024-03-31T23:24:16.647 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:16.645+0000 7f5e5e5f8740 1 --2- 172.21.15.46:0/3849054541 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56034139f6c0 0x5603413b0ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:24:16.647 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:16.645+0000 7f5e5e5f8740 1 -- 172.21.15.46:0/3849054541 >> 172.21.15.46:0/3849054541 conn(0x560341399fe0 msgr2=0x5603413986f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:24:16.647 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:16.645+0000 7f5e5e5f8740 1 -- 172.21.15.46:0/3849054541 shutdown_connections 2024-03-31T23:24:16.647 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:16.645+0000 7f5e5e5f8740 1 -- 172.21.15.46:0/3849054541 wait complete. 2024-03-31T23:24:16.647 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:16.646+0000 7f5e5e5f8740 20 client.17178 trim_cache size 0 max 16384 2024-03-31T23:24:16.647 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout:{ 2024-03-31T23:24:16.648 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout: "message": "", 2024-03-31T23:24:16.648 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout: "return_code": 0 2024-03-31T23:24:16.648 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout:} 2024-03-31T23:24:16.649 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:16.646+0000 7f5e5b93e640 1 -- 172.21.15.46:0/2054312580 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f5e400a26b0 msgr2=0x7f5e400a4b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:24:16.649 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:16.646+0000 7f5e5b93e640 1 --2- 172.21.15.46:0/2054312580 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f5e400a26b0 0x7f5e400a4b70 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f5e4c0982a0 tx=0x7f5e4c096040 comp rx=0 tx=0).stop 2024-03-31T23:24:16.649 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:16.646+0000 7f5e5b93e640 1 -- 172.21.15.46:0/2054312580 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5e5410dfc0 msgr2=0x7f5e5419e960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:24:16.649 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:16.646+0000 7f5e5b93e640 1 --2- 172.21.15.46:0/2054312580 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5e5410dfc0 0x7f5e5419e960 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f5e4c009fd0 tx=0x7f5e4c00ef10 comp rx=0 tx=0).stop 2024-03-31T23:24:16.649 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:16.646+0000 7f5e5b93e640 1 -- 172.21.15.46:0/2054312580 shutdown_connections 2024-03-31T23:24:16.649 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:16.646+0000 7f5e5b93e640 1 --2- 172.21.15.46:0/2054312580 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f5e400a26b0 0x7f5e400a4b70 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:24:16.649 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:16.646+0000 7f5e5b93e640 1 --2- 172.21.15.46:0/2054312580 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5e5410dfc0 0x7f5e5419e960 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:24:16.649 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:16.646+0000 7f5e5b93e640 1 --2- 172.21.15.46:0/2054312580 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5e54074420 0x7f5e5419a5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:24:16.649 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:16.646+0000 7f5e5b93e640 1 --2- 172.21.15.46:0/2054312580 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5e54073b00 0x7f5e5419a0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:24:16.649 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:16.646+0000 7f5e5b93e640 1 -- 172.21.15.46:0/2054312580 >> 172.21.15.46:0/2054312580 conn(0x7f5e540fd210 msgr2=0x7f5e540fead0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:24:16.649 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:16.646+0000 7f5e5b93e640 1 -- 172.21.15.46:0/2054312580 shutdown_connections 2024-03-31T23:24:16.649 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:16.646+0000 7f5e5b93e640 1 -- 172.21.15.46:0/2054312580 wait complete. 2024-03-31T23:24:16.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:24:16 smithi142 ceph-mon[35271]: pgmap v393: 129 pgs: 129 active+clean; 140 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 425 KiB/s rd, 1.0 MiB/s wr, 106 op/s 2024-03-31T23:24:17.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:24:16 smithi162 ceph-mon[36237]: pgmap v393: 129 pgs: 129 active+clean; 140 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 425 KiB/s rd, 1.0 MiB/s wr, 106 op/s 2024-03-31T23:24:17.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:24:16 smithi046 ceph-mon[32498]: pgmap v393: 129 pgs: 129 active+clean; 140 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 425 KiB/s rd, 1.0 MiB/s wr, 106 op/s 2024-03-31T23:24:17.220 INFO:tasks.workunit.client.0.smithi046.stdout: 1 241310 1.80 MB/sec execute 563 sec latency 1404.761 ms 2024-03-31T23:24:18.220 INFO:tasks.workunit.client.0.smithi046.stdout: 1 241377 1.79 MB/sec execute 564 sec latency 1934.252 ms 2024-03-31T23:24:18.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:24:18 smithi142 ceph-mon[35271]: pgmap v394: 129 pgs: 129 active+clean; 140 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 370 KiB/s rd, 926 KiB/s wr, 92 op/s 2024-03-31T23:24:19.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:24:18 smithi162 ceph-mon[36237]: pgmap v394: 129 pgs: 129 active+clean; 140 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 370 KiB/s rd, 926 KiB/s wr, 92 op/s 2024-03-31T23:24:19.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:24:18 smithi046 ceph-mon[32498]: pgmap v394: 129 pgs: 129 active+clean; 140 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 370 KiB/s rd, 926 KiB/s wr, 92 op/s 2024-03-31T23:24:19.220 INFO:tasks.workunit.client.0.smithi046.stdout: 1 241377 1.79 MB/sec execute 565 sec latency 1215.506 ms 2024-03-31T23:24:20.221 INFO:tasks.workunit.client.0.smithi046.stdout: 1 241377 1.79 MB/sec execute 566 sec latency 2215.541 ms 2024-03-31T23:24:21.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:24:20 smithi162 ceph-mon[36237]: pgmap v395: 129 pgs: 129 active+clean; 143 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 429 KiB/s rd, 1.0 MiB/s wr, 106 op/s 2024-03-31T23:24:21.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:24:20 smithi046 ceph-mon[32498]: pgmap v395: 129 pgs: 129 active+clean; 143 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 429 KiB/s rd, 1.0 MiB/s wr, 106 op/s 2024-03-31T23:24:21.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:24:20 smithi142 ceph-mon[35271]: pgmap v395: 129 pgs: 129 active+clean; 143 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 429 KiB/s rd, 1.0 MiB/s wr, 106 op/s 2024-03-31T23:24:21.220 INFO:tasks.workunit.client.0.smithi046.stdout: 1 241377 1.78 MB/sec execute 567 sec latency 3215.613 ms 2024-03-31T23:24:22.221 INFO:tasks.workunit.client.0.smithi046.stdout: 1 241377 1.78 MB/sec execute 568 sec latency 4215.707 ms 2024-03-31T23:24:22.995 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:24:22 smithi142 ceph-mon[35271]: pgmap v396: 129 pgs: 129 active+clean; 143 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 415 KiB/s rd, 1023 KiB/s wr, 102 op/s 2024-03-31T23:24:23.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:24:22 smithi162 ceph-mon[36237]: pgmap v396: 129 pgs: 129 active+clean; 143 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 415 KiB/s rd, 1023 KiB/s wr, 102 op/s 2024-03-31T23:24:23.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:24:22 smithi046 ceph-mon[32498]: pgmap v396: 129 pgs: 129 active+clean; 143 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 415 KiB/s rd, 1023 KiB/s wr, 102 op/s 2024-03-31T23:24:23.221 INFO:tasks.workunit.client.0.smithi046.stdout: 1 241503 1.78 MB/sec execute 569 sec latency 4742.837 ms 2024-03-31T23:24:24.221 INFO:tasks.workunit.client.0.smithi046.stdout: 1 241744 1.78 MB/sec execute 570 sec latency 14.984 ms 2024-03-31T23:24:25.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:24:24 smithi162 ceph-mon[36237]: pgmap v397: 129 pgs: 129 active+clean; 143 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 357 KiB/s rd, 880 KiB/s wr, 88 op/s 2024-03-31T23:24:25.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:24:24 smithi046 ceph-mon[32498]: pgmap v397: 129 pgs: 129 active+clean; 143 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 357 KiB/s rd, 880 KiB/s wr, 88 op/s 2024-03-31T23:24:25.194 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:24:24 smithi142 ceph-mon[35271]: pgmap v397: 129 pgs: 129 active+clean; 143 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 357 KiB/s rd, 880 KiB/s wr, 88 op/s 2024-03-31T23:24:25.221 INFO:tasks.workunit.client.0.smithi046.stdout: 1 242141 1.77 MB/sec execute 571 sec latency 17.017 ms 2024-03-31T23:24:26.221 INFO:tasks.workunit.client.0.smithi046.stdout: 1 242644 1.78 MB/sec execute 572 sec latency 18.005 ms 2024-03-31T23:24:27.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:24:26 smithi162 ceph-mon[36237]: pgmap v398: 129 pgs: 129 active+clean; 144 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 404 KiB/s rd, 1.0 MiB/s wr, 105 op/s 2024-03-31T23:24:27.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:24:26 smithi046 ceph-mon[32498]: pgmap v398: 129 pgs: 129 active+clean; 144 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 404 KiB/s rd, 1.0 MiB/s wr, 105 op/s 2024-03-31T23:24:27.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:24:26 smithi142 ceph-mon[35271]: pgmap v398: 129 pgs: 129 active+clean; 144 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 404 KiB/s rd, 1.0 MiB/s wr, 105 op/s 2024-03-31T23:24:27.221 INFO:tasks.workunit.client.0.smithi046.stdout: 1 243115 1.78 MB/sec execute 573 sec latency 21.289 ms 2024-03-31T23:24:28.221 INFO:tasks.workunit.client.0.smithi046.stdout: 1 243653 1.78 MB/sec execute 574 sec latency 20.502 ms 2024-03-31T23:24:29.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:24:28 smithi162 ceph-mon[36237]: pgmap v399: 129 pgs: 129 active+clean; 144 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 178 KiB/s rd, 541 KiB/s wr, 51 op/s 2024-03-31T23:24:29.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:24:28 smithi046 ceph-mon[32498]: pgmap v399: 129 pgs: 129 active+clean; 144 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 178 KiB/s rd, 541 KiB/s wr, 51 op/s 2024-03-31T23:24:29.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:24:28 smithi142 ceph-mon[35271]: pgmap v399: 129 pgs: 129 active+clean; 144 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 178 KiB/s rd, 541 KiB/s wr, 51 op/s 2024-03-31T23:24:29.221 INFO:tasks.workunit.client.0.smithi046.stdout: 1 244225 1.78 MB/sec execute 575 sec latency 16.351 ms 2024-03-31T23:24:29.738 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:29.735+0000 7f4b8b969640 1 -- 172.21.15.46:0/107816833 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4b84101260 msgr2=0x7f4b8410d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:24:29.739 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:29.735+0000 7f4b8b969640 1 --2- 172.21.15.46:0/107816833 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4b84101260 0x7f4b8410d480 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7f4b7c009ca0 tx=0x7f4b7c02c0f0 comp rx=0 tx=0).stop 2024-03-31T23:24:29.739 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:29.735+0000 7f4b8b969640 1 -- 172.21.15.46:0/107816833 shutdown_connections 2024-03-31T23:24:29.739 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:29.735+0000 7f4b8b969640 1 --2- 172.21.15.46:0/107816833 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4b84101260 0x7f4b8410d480 unknown :-1 s=CLOSED pgs=861 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:24:29.739 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:29.735+0000 7f4b8b969640 1 --2- 172.21.15.46:0/107816833 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4b841008a0 0x7f4b84100d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:24:29.739 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:29.735+0000 7f4b8b969640 1 --2- 172.21.15.46:0/107816833 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4b841055f0 0x7f4b841059d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:24:29.739 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:29.735+0000 7f4b8b969640 1 -- 172.21.15.46:0/107816833 >> 172.21.15.46:0/107816833 conn(0x7f4b840774a0 msgr2=0x7f4b840778b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:24:29.739 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:29.735+0000 7f4b8b969640 1 -- 172.21.15.46:0/107816833 shutdown_connections 2024-03-31T23:24:29.739 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:29.735+0000 7f4b8b969640 1 -- 172.21.15.46:0/107816833 wait complete. 2024-03-31T23:24:29.739 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:29.736+0000 7f4b8b969640 1 Processor -- start 2024-03-31T23:24:29.739 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:29.736+0000 7f4b8b969640 1 -- start start 2024-03-31T23:24:29.739 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:29.736+0000 7f4b8b969640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4b841008a0 0x7f4b84196d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:24:29.739 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:29.736+0000 7f4b8b969640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4b84101260 0x7f4b841972c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:24:29.739 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:29.736+0000 7f4b8b969640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4b841055f0 0x7f4b8419b640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:24:29.740 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:29.736+0000 7f4b8b969640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f4b841109c0 con 0x7f4b84101260 2024-03-31T23:24:29.740 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:29.736+0000 7f4b8b969640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f4b84110840 con 0x7f4b841008a0 2024-03-31T23:24:29.740 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:29.736+0000 7f4b8b969640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f4b84110b40 con 0x7f4b841055f0 2024-03-31T23:24:29.740 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:29.736+0000 7f4b88f04640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4b84101260 0x7f4b841972c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:24:29.740 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:29.736+0000 7f4b89705640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4b841008a0 0x7f4b84196d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:24:29.740 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:29.736+0000 7f4b88f04640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4b84101260 0x7f4b841972c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:49174/0 (socket says 172.21.15.46:49174) 2024-03-31T23:24:29.740 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:29.736+0000 7f4b88f04640 1 -- 172.21.15.46:0/2847139084 learned_addr learned my addr 172.21.15.46:0/2847139084 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:24:29.740 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:29.737+0000 7f4b89f06640 1 --2- 172.21.15.46:0/2847139084 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4b841055f0 0x7f4b8419b640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:24:29.740 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:29.737+0000 7f4b88f04640 1 -- 172.21.15.46:0/2847139084 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4b841055f0 msgr2=0x7f4b8419b640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:24:29.740 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:29.737+0000 7f4b88f04640 1 --2- 172.21.15.46:0/2847139084 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4b841055f0 0x7f4b8419b640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:24:29.740 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:29.737+0000 7f4b88f04640 1 -- 172.21.15.46:0/2847139084 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4b841008a0 msgr2=0x7f4b84196d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:24:29.740 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:29.737+0000 7f4b88f04640 1 --2- 172.21.15.46:0/2847139084 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4b841008a0 0x7f4b84196d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:24:29.740 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:29.737+0000 7f4b88f04640 1 -- 172.21.15.46:0/2847139084 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b8419bd70 con 0x7f4b84101260 2024-03-31T23:24:29.740 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:29.737+0000 7f4b88f04640 1 --2- 172.21.15.46:0/2847139084 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4b84101260 0x7f4b841972c0 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7f4b7400e9f0 tx=0x7f4b7400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:24:29.740 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:29.737+0000 7f4b7a7fc640 1 -- 172.21.15.46:0/2847139084 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b7400cd80 con 0x7f4b84101260 2024-03-31T23:24:29.741 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:29.737+0000 7f4b7a7fc640 1 -- 172.21.15.46:0/2847139084 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4b740108e0 con 0x7f4b84101260 2024-03-31T23:24:29.741 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:29.737+0000 7f4b7a7fc640 1 -- 172.21.15.46:0/2847139084 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b74018920 con 0x7f4b84101260 2024-03-31T23:24:29.741 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:29.737+0000 7f4b8b969640 1 -- 172.21.15.46:0/2847139084 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4b8419c060 con 0x7f4b84101260 2024-03-31T23:24:29.741 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:29.737+0000 7f4b8b969640 1 -- 172.21.15.46:0/2847139084 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4b841a38f0 con 0x7f4b84101260 2024-03-31T23:24:29.743 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:29.740+0000 7f4b7a7fc640 1 -- 172.21.15.46:0/2847139084 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 25) ==== 137451+0+0 (secure 0 0 0) 0x7f4b74010a90 con 0x7f4b84101260 2024-03-31T23:24:29.744 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:29.740+0000 7f4b8e622740 1 Processor -- start 2024-03-31T23:24:29.744 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:29.740+0000 7f4b8e622740 1 -- start start 2024-03-31T23:24:29.744 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:29.740+0000 7f4b8e622740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e0bb325170 0x55e0bb325550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:24:29.744 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:29.740+0000 7f4b8e622740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e0bb325a90 0x55e0bb330a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:24:29.744 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:29.740+0000 7f4b8e622740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55e0bb330fc0 0x55e0bb3333b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:24:29.744 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:29.740+0000 7f4b8e622740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55e0bb2888f0 con 0x55e0bb325a90 2024-03-31T23:24:29.744 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:29.740+0000 7f4b8e622740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55e0bb2318d0 con 0x55e0bb330fc0 2024-03-31T23:24:29.744 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:29.740+0000 7f4b8e622740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55e0bb284990 con 0x55e0bb325170 2024-03-31T23:24:29.744 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:29.740+0000 7f4b7a7fc640 1 --2- 172.21.15.46:0/2847139084 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f4b700a2730 0x7f4b700a4bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:24:29.744 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:29.740+0000 7f4b89705640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e0bb325a90 0x55e0bb330a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:24:29.744 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:29.740+0000 7f4b88f04640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55e0bb330fc0 0x55e0bb3333b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:24:29.744 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:29.740+0000 7f4b89f06640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e0bb325170 0x55e0bb325550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:24:29.744 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:29.741+0000 7f4b89705640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e0bb325a90 0x55e0bb330a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:49188/0 (socket says 172.21.15.46:49188) 2024-03-31T23:24:29.744 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:29.741+0000 7f4b89705640 1 -- 172.21.15.46:0/1726390399 learned_addr learned my addr 172.21.15.46:0/1726390399 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:24:29.745 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:29.741+0000 7f4b89f06640 1 --2- 172.21.15.46:0/2847139084 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f4b700a2730 0x7f4b700a4bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:24:29.745 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:29.741+0000 7f4b7a7fc640 1 -- 172.21.15.46:0/2847139084 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(110..110 src has 1..110) ==== 9554+0+0 (secure 0 0 0) 0x7f4b740cf190 con 0x7f4b84101260 2024-03-31T23:24:29.745 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:29.741+0000 7f4b88f04640 1 -- 172.21.15.46:0/1726390399 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e0bb325170 msgr2=0x55e0bb325550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:24:29.745 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:29.741+0000 7f4b88f04640 1 --2- 172.21.15.46:0/1726390399 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e0bb325170 0x55e0bb325550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:24:29.745 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:29.741+0000 7f4b88f04640 1 -- 172.21.15.46:0/1726390399 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e0bb325a90 msgr2=0x55e0bb330a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:24:29.745 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:29.741+0000 7f4b88f04640 1 --2- 172.21.15.46:0/1726390399 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e0bb325a90 0x55e0bb330a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:24:29.745 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:29.741+0000 7f4b88f04640 1 -- 172.21.15.46:0/1726390399 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e0baf9d210 con 0x55e0bb330fc0 2024-03-31T23:24:29.745 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:29.741+0000 7f4b89705640 1 --2- 172.21.15.46:0/1726390399 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e0bb325a90 0x55e0bb330a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:24:29.745 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:29.741+0000 7f4b89f06640 1 --2- 172.21.15.46:0/2847139084 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f4b700a2730 0x7f4b700a4bf0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f4b7c005e30 tx=0x7f4b7c005d80 comp rx=0 tx=0).ready entity=mgr.15654 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:24:29.745 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:29.741+0000 7f4b88f04640 1 --2- 172.21.15.46:0/1726390399 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55e0bb330fc0 0x55e0bb3333b0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f4b740cc4d0 tx=0x7f4b7406e6e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f9f39cee69bd3af7 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:24:29.745 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:29.741+0000 7f4b6f7fe640 1 -- 172.21.15.46:0/1726390399 <== mon.2 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b7409f650 con 0x55e0bb330fc0 2024-03-31T23:24:29.745 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:29.741+0000 7f4b6f7fe640 1 -- 172.21.15.46:0/1726390399 <== mon.2 v2:172.21.15.142:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4b740a3f10 con 0x55e0bb330fc0 2024-03-31T23:24:29.745 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:29.741+0000 7f4b6f7fe640 1 -- 172.21.15.46:0/1726390399 <== mon.2 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b740a3ce0 con 0x55e0bb330fc0 2024-03-31T23:24:29.745 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:29.742+0000 7f4b8e622740 1 -- 172.21.15.46:0/1726390399 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55e0bb330fc0 msgr2=0x55e0bb3333b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:24:29.745 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:29.742+0000 7f4b8e622740 1 --2- 172.21.15.46:0/1726390399 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55e0bb330fc0 0x55e0bb3333b0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f4b740cc4d0 tx=0x7f4b7406e6e0 comp rx=0 tx=0).stop 2024-03-31T23:24:29.746 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:29.742+0000 7f4b8e622740 1 -- 172.21.15.46:0/1726390399 shutdown_connections 2024-03-31T23:24:29.746 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:29.742+0000 7f4b8e622740 1 --2- 172.21.15.46:0/1726390399 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55e0bb330fc0 0x55e0bb3333b0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:24:29.746 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:29.742+0000 7f4b8e622740 1 --2- 172.21.15.46:0/1726390399 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e0bb325a90 0x55e0bb330a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:24:29.746 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:29.742+0000 7f4b8e622740 1 --2- 172.21.15.46:0/1726390399 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e0bb325170 0x55e0bb325550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:24:29.746 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:29.742+0000 7f4b8e622740 1 -- 172.21.15.46:0/1726390399 >> 172.21.15.46:0/1726390399 conn(0x55e0bb31ffe0 msgr2=0x55e0bb3203c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:24:29.746 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:29.742+0000 7f4b8e622740 1 -- 172.21.15.46:0/1726390399 shutdown_connections 2024-03-31T23:24:29.746 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:29.742+0000 7f4b8e622740 1 -- 172.21.15.46:0/1726390399 wait complete. 2024-03-31T23:24:29.746 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:29.742+0000 7f4b8e622740 1 -- start start 2024-03-31T23:24:29.746 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:29.743+0000 7f4b8e622740 1 Processor -- start 2024-03-31T23:24:29.749 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:29.743+0000 7f4b8e622740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e0bb325170 0x55e0bb32e050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:24:29.749 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:29.743+0000 7f4b8e622740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55e0bb325a90 0x55e0bb32e590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:24:29.749 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:29.743+0000 7f4b8e622740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e0bb330fc0 0x55e0bb32ead0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:24:29.749 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:29.743+0000 7f4b8e622740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55e0bb333b90 con 0x55e0bb330fc0 2024-03-31T23:24:29.750 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:29.743+0000 7f4b8e622740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55e0bb226f80 con 0x55e0bb325a90 2024-03-31T23:24:29.750 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:29.743+0000 7f4b8e622740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55e0bb333d10 con 0x55e0bb325170 2024-03-31T23:24:29.750 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:29.743+0000 7f4b8e622740 10 client.0 _ll_register_callbacks cb 0x55e0bb27f290 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T23:24:29.750 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:29.743+0000 7f4b89f06640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e0bb330fc0 0x55e0bb32ead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:24:29.750 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:29.743+0000 7f4b89705640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e0bb325170 0x55e0bb32e050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:24:29.750 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:29.743+0000 7f4b89f06640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e0bb330fc0 0x55e0bb32ead0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:49194/0 (socket says 172.21.15.46:49194) 2024-03-31T23:24:29.750 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:29.743+0000 7f4b89f06640 1 -- 172.21.15.46:0/2442064804 learned_addr learned my addr 172.21.15.46:0/2442064804 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:24:29.750 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:29.743+0000 7f4b88f04640 1 --2- 172.21.15.46:0/2442064804 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55e0bb325a90 0x55e0bb32e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:24:29.750 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:29.743+0000 7f4b89f06640 1 -- 172.21.15.46:0/2442064804 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e0bb325170 msgr2=0x55e0bb32e050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:24:29.750 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:29.743+0000 7f4b89f06640 1 --2- 172.21.15.46:0/2442064804 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e0bb325170 0x55e0bb32e050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:24:29.750 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:29.743+0000 7f4b89f06640 1 -- 172.21.15.46:0/2442064804 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55e0bb325a90 msgr2=0x55e0bb32e590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:24:29.750 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:29.743+0000 7f4b89f06640 1 --2- 172.21.15.46:0/2442064804 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55e0bb325a90 0x55e0bb32e590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:24:29.750 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:29.743+0000 7f4b89f06640 1 -- 172.21.15.46:0/2442064804 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e0bb069620 con 0x55e0bb330fc0 2024-03-31T23:24:29.750 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:29.744+0000 7f4b89f06640 1 --2- 172.21.15.46:0/2442064804 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e0bb330fc0 0x55e0bb32ead0 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7f4b7c009ca0 tx=0x7f4b7c0095c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e180bf3e111a0a4 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:24:29.751 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:29.744+0000 7f4b6cff9640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-31T23:24:29.751 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:29.744+0000 7f4b6cff9640 1 -- 172.21.15.46:0/2442064804 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b7c054060 con 0x55e0bb330fc0 2024-03-31T23:24:29.751 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:29.744+0000 7f4b6cff9640 1 -- 172.21.15.46:0/2442064804 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(110..110 src has 1..110) ==== 9554+0+0 (secure 0 0 0) 0x7f4b7c032870 con 0x55e0bb330fc0 2024-03-31T23:24:29.751 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:29.744+0000 7f4b8e622740 1 -- 172.21.15.46:0/2442064804 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4b740a3f10 con 0x55e0bb330fc0 2024-03-31T23:24:29.751 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:29.744+0000 7f4b6cff9640 1 -- 172.21.15.46:0/2442064804 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f4b7c05b050 con 0x55e0bb330fc0 2024-03-31T23:24:29.751 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:29.744+0000 7f4b8e622740 10 client.17190 fetch_fsmap learned FSMap version 36 2024-03-31T23:24:29.751 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:29.744+0000 7f4b8e622740 1 -- 172.21.15.46:0/2442064804 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x55e0bb33fb90 con 0x55e0bb330fc0 2024-03-31T23:24:29.751 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:29.744+0000 7f4b6cff9640 1 -- 172.21.15.46:0/2442064804 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4b740a3f10 con 0x55e0bb330fc0 2024-03-31T23:24:29.751 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:29.744+0000 7f4b6cff9640 1 -- 172.21.15.46:0/2442064804 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 36) ==== 8362+0+0 (secure 0 0 0) 0x7f4b7c04fa10 con 0x55e0bb330fc0 2024-03-31T23:24:29.751 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:29.744+0000 7f4b8e622740 10 client.17190 fetch_fsmap finished waiting for FSMap version 36 2024-03-31T23:24:29.751 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:29.744+0000 7f4b8e622740 10 client.17190 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-31T23:24:29.751 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:29.744+0000 7f4b8e622740 20 client.17190 populate_metadata read hostname 'smithi046' 2024-03-31T23:24:29.751 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:29.744+0000 7f4b8e622740 1 --2- 172.21.15.46:0/2442064804 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55e0bb3410c0 0x55e0bb3434f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:24:29.751 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:29.744+0000 7f4b8e622740 4 client.17190 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-31T23:24:29.751 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:29.744+0000 7f4b8e622740 1 -- 172.21.15.46:0/2442064804 --> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e0bb29c120 con 0x55e0bb3410c0 2024-03-31T23:24:29.751 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:29.745+0000 7f4b89705640 1 --2- 172.21.15.46:0/2442064804 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55e0bb3410c0 0x55e0bb3434f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:24:29.752 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:29.745+0000 7f4b89705640 1 --2- 172.21.15.46:0/2442064804 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55e0bb3410c0 0x55e0bb3434f0 crc :-1 s=READY pgs=189 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=96003b87fa8a32b8 server_cookie=e979591a15afd123 in_seq=0 out_seq=0 2024-03-31T23:24:29.752 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:29.745+0000 7f4b6cff9640 10 client.17190 ms_handle_connect on v2:172.21.15.142:6838/2519143918 2024-03-31T23:24:29.752 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:29.746+0000 7f4b6cff9640 1 -- 172.21.15.46:0/2442064804 <== mds.0 v2:172.21.15.142:6838/2519143918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55e0bb29c120 con 0x55e0bb3410c0 2024-03-31T23:24:29.752 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:29.746+0000 7f4b6cff9640 10 client.17190 handle_command_reply: tid=0 2024-03-31T23:24:29.752 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:29.746+0000 7f4b8e622740 1 client.17190 shutdown 2024-03-31T23:24:29.752 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:29.746+0000 7f4b8e622740 1 -- 172.21.15.46:0/2442064804 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e0bb330fc0 msgr2=0x55e0bb32ead0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:24:29.752 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:29.746+0000 7f4b8e622740 1 --2- 172.21.15.46:0/2442064804 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e0bb330fc0 0x55e0bb32ead0 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7f4b7c009ca0 tx=0x7f4b7c0095c0 comp rx=0 tx=0).stop 2024-03-31T23:24:29.752 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:29.746+0000 7f4b8e622740 1 -- 172.21.15.46:0/2442064804 shutdown_connections 2024-03-31T23:24:29.752 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:29.746+0000 7f4b8e622740 1 --2- 172.21.15.46:0/2442064804 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55e0bb3410c0 0x55e0bb3434f0 crc :-1 s=READY pgs=189 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:24:29.752 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:29.746+0000 7f4b6cff9640 0 client.17190 ms_handle_reset on v2:172.21.15.142:6838/2519143918 2024-03-31T23:24:29.752 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:29.746+0000 7f4b8e622740 1 --2- 172.21.15.46:0/2442064804 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e0bb330fc0 0x55e0bb32ead0 unknown :-1 s=CLOSED pgs=863 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:24:29.752 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:29.746+0000 7f4b8e622740 1 --2- 172.21.15.46:0/2442064804 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55e0bb325a90 0x55e0bb32e590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:24:29.752 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:29.746+0000 7f4b8e622740 1 --2- 172.21.15.46:0/2442064804 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e0bb325170 0x55e0bb32e050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:24:29.752 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:29.746+0000 7f4b8e622740 1 -- 172.21.15.46:0/2442064804 >> 172.21.15.46:0/2442064804 conn(0x55e0bb31ffe0 msgr2=0x55e0bb321d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:24:29.753 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:29.746+0000 7f4b8e622740 1 -- 172.21.15.46:0/2442064804 shutdown_connections 2024-03-31T23:24:29.753 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:29.746+0000 7f4b8e622740 1 -- 172.21.15.46:0/2442064804 wait complete. 2024-03-31T23:24:29.753 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:29.747+0000 7f4b8e622740 20 client.17190 trim_cache size 0 max 16384 2024-03-31T23:24:29.768 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:29.762+0000 7f4b8e622740 1 Processor -- start 2024-03-31T23:24:29.768 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:29.762+0000 7f4b8e622740 1 -- start start 2024-03-31T23:24:29.768 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:29.762+0000 7f4b8e622740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e0bb325170 0x55e0bb33c970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:24:29.768 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:29.762+0000 7f4b8e622740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e0bb325a90 0x55e0bb348e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:24:29.768 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:29.762+0000 7f4b8e622740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55e0bb330fc0 0x55e0bb31e6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:24:29.768 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:29.762+0000 7f4b8e622740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f4b841af2b0 con 0x55e0bb325a90 2024-03-31T23:24:29.768 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:29.762+0000 7f4b8e622740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f4b841acb20 con 0x55e0bb330fc0 2024-03-31T23:24:29.768 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:29.762+0000 7f4b8e622740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f4b841aa990 con 0x55e0bb325170 2024-03-31T23:24:29.768 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:29.762+0000 7f4b88f04640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e0bb325a90 0x55e0bb348e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:24:29.768 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:29.762+0000 7f4b88f04640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e0bb325a90 0x55e0bb348e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:49202/0 (socket says 172.21.15.46:49202) 2024-03-31T23:24:29.768 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:29.762+0000 7f4b88f04640 1 -- 172.21.15.46:0/3727842906 learned_addr learned my addr 172.21.15.46:0/3727842906 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:24:29.769 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:29.762+0000 7f4b89f06640 1 --2- 172.21.15.46:0/3727842906 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55e0bb330fc0 0x55e0bb31e6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:24:29.769 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:29.762+0000 7f4b89705640 1 --2- 172.21.15.46:0/3727842906 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e0bb325170 0x55e0bb33c970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:24:29.769 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:29.763+0000 7f4b88f04640 1 -- 172.21.15.46:0/3727842906 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e0bb325170 msgr2=0x55e0bb33c970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:24:29.769 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:29.763+0000 7f4b88f04640 1 --2- 172.21.15.46:0/3727842906 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e0bb325170 0x55e0bb33c970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:24:29.769 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:29.763+0000 7f4b88f04640 1 -- 172.21.15.46:0/3727842906 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55e0bb330fc0 msgr2=0x55e0bb31e6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:24:29.769 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:29.763+0000 7f4b88f04640 1 --2- 172.21.15.46:0/3727842906 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55e0bb330fc0 0x55e0bb31e6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:24:29.769 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:29.763+0000 7f4b88f04640 1 -- 172.21.15.46:0/3727842906 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e0bb321170 con 0x55e0bb325a90 2024-03-31T23:24:29.769 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:29.763+0000 7f4b88f04640 1 --2- 172.21.15.46:0/3727842906 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e0bb325a90 0x55e0bb348e50 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7f4b740979c0 tx=0x7f4b74000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=41f96e15356fdb57 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:24:29.769 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:29.763+0000 7f4b6effd640 1 -- 172.21.15.46:0/3727842906 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b740a3f10 con 0x55e0bb325a90 2024-03-31T23:24:29.769 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:29.763+0000 7f4b6effd640 1 -- 172.21.15.46:0/3727842906 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4b740cc750 con 0x55e0bb325a90 2024-03-31T23:24:29.769 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:29.763+0000 7f4b6effd640 1 -- 172.21.15.46:0/3727842906 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b7409f1b0 con 0x55e0bb325a90 2024-03-31T23:24:29.769 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:29.764+0000 7f4b8e622740 1 -- 172.21.15.46:0/3727842906 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e0bb325a90 msgr2=0x55e0bb348e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:24:29.769 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:29.764+0000 7f4b8e622740 1 --2- 172.21.15.46:0/3727842906 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e0bb325a90 0x55e0bb348e50 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7f4b740979c0 tx=0x7f4b74000bc0 comp rx=0 tx=0).stop 2024-03-31T23:24:29.769 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:29.764+0000 7f4b8e622740 1 -- 172.21.15.46:0/3727842906 shutdown_connections 2024-03-31T23:24:29.769 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:29.764+0000 7f4b8e622740 1 --2- 172.21.15.46:0/3727842906 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55e0bb330fc0 0x55e0bb31e6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:24:29.770 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:29.764+0000 7f4b8e622740 1 --2- 172.21.15.46:0/3727842906 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e0bb325a90 0x55e0bb348e50 unknown :-1 s=CLOSED pgs=864 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:24:29.770 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:29.764+0000 7f4b8e622740 1 --2- 172.21.15.46:0/3727842906 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e0bb325170 0x55e0bb33c970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:24:29.770 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:29.764+0000 7f4b8e622740 1 -- 172.21.15.46:0/3727842906 >> 172.21.15.46:0/3727842906 conn(0x55e0bb31ffe0 msgr2=0x55e0bb3203c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:24:29.770 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:29.764+0000 7f4b8e622740 1 -- 172.21.15.46:0/3727842906 shutdown_connections 2024-03-31T23:24:29.770 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:29.764+0000 7f4b8e622740 1 -- 172.21.15.46:0/3727842906 wait complete. 2024-03-31T23:24:29.770 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:29.764+0000 7f4b8e622740 1 -- start start 2024-03-31T23:24:29.770 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:29.764+0000 7f4b8e622740 1 Processor -- start 2024-03-31T23:24:29.770 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:29.764+0000 7f4b8e622740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e0bb325170 0x55e0bb334a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:24:29.770 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:29.764+0000 7f4b8e622740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55e0bb325a90 0x55e0bb334f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:24:29.770 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:29.764+0000 7f4b8e622740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e0bb330fc0 0x55e0bb335480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:24:29.770 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:29.764+0000 7f4b8e622740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f4b841aafe0 con 0x55e0bb330fc0 2024-03-31T23:24:29.770 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:29.764+0000 7f4b8e622740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f4b841ac450 con 0x55e0bb325a90 2024-03-31T23:24:29.770 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:29.764+0000 7f4b8e622740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f4b841ab6b0 con 0x55e0bb325170 2024-03-31T23:24:29.770 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:29.765+0000 7f4b8e622740 10 client.0 _ll_register_callbacks cb 0x7f4b80005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T23:24:29.771 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:29.765+0000 7f4b89f06640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e0bb330fc0 0x55e0bb335480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:24:29.771 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:29.765+0000 7f4b88f04640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55e0bb325a90 0x55e0bb334f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:24:29.771 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:29.765+0000 7f4b89705640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e0bb325170 0x55e0bb334a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:24:29.771 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:29.765+0000 7f4b88f04640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55e0bb325a90 0x55e0bb334f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.46:43240/0 (socket says 172.21.15.46:43240) 2024-03-31T23:24:29.771 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:29.765+0000 7f4b88f04640 1 -- 172.21.15.46:0/588485537 learned_addr learned my addr 172.21.15.46:0/588485537 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:24:29.771 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:29.765+0000 7f4b89f06640 1 -- 172.21.15.46:0/588485537 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e0bb325170 msgr2=0x55e0bb334a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:24:29.771 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:29.765+0000 7f4b89f06640 1 --2- 172.21.15.46:0/588485537 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e0bb325170 0x55e0bb334a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:24:29.771 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:29.765+0000 7f4b89f06640 1 -- 172.21.15.46:0/588485537 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55e0bb325a90 msgr2=0x55e0bb334f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:24:29.771 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:29.765+0000 7f4b89f06640 1 --2- 172.21.15.46:0/588485537 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55e0bb325a90 0x55e0bb334f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:24:29.771 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:29.765+0000 7f4b89f06640 1 -- 172.21.15.46:0/588485537 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e0bb326810 con 0x55e0bb330fc0 2024-03-31T23:24:29.771 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:29.765+0000 7f4b88f04640 1 --2- 172.21.15.46:0/588485537 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55e0bb325a90 0x55e0bb334f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:24:29.771 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:29.765+0000 7f4b89705640 1 --2- 172.21.15.46:0/588485537 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e0bb325170 0x55e0bb334a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:24:29.771 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:29.765+0000 7f4b89f06640 1 --2- 172.21.15.46:0/588485537 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e0bb330fc0 0x55e0bb335480 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7f4b7c02c630 tx=0x7f4b7c032f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c6866419a81557ff server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:24:29.772 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:29.765+0000 7f4b6cff9640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-31T23:24:29.772 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:29.765+0000 7f4b6cff9640 1 -- 172.21.15.46:0/588485537 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b7c054040 con 0x55e0bb330fc0 2024-03-31T23:24:29.772 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:29.765+0000 7f4b6cff9640 1 -- 172.21.15.46:0/588485537 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(110..110 src has 1..110) ==== 9554+0+0 (secure 0 0 0) 0x7f4b7c032870 con 0x55e0bb330fc0 2024-03-31T23:24:29.772 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:29.765+0000 7f4b8e622740 1 -- 172.21.15.46:0/588485537 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4b740cc750 con 0x55e0bb330fc0 2024-03-31T23:24:29.772 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:29.765+0000 7f4b6cff9640 1 -- 172.21.15.46:0/588485537 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f4b7c039040 con 0x55e0bb330fc0 2024-03-31T23:24:29.772 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:29.765+0000 7f4b8e622740 10 client.17196 fetch_fsmap learned FSMap version 36 2024-03-31T23:24:29.772 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:29.765+0000 7f4b8e622740 1 -- 172.21.15.46:0/588485537 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x55e0bb327b00 con 0x55e0bb330fc0 2024-03-31T23:24:29.772 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:29.766+0000 7f4b6cff9640 1 -- 172.21.15.46:0/588485537 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4b740cc750 con 0x55e0bb330fc0 2024-03-31T23:24:29.772 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:29.766+0000 7f4b6cff9640 1 -- 172.21.15.46:0/588485537 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 36) ==== 8362+0+0 (secure 0 0 0) 0x7f4b7c04f470 con 0x55e0bb330fc0 2024-03-31T23:24:29.772 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:29.766+0000 7f4b8e622740 10 client.17196 fetch_fsmap finished waiting for FSMap version 36 2024-03-31T23:24:29.772 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:29.766+0000 7f4b8e622740 10 client.17196 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-31T23:24:29.772 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:29.766+0000 7f4b8e622740 20 client.17196 populate_metadata read hostname 'smithi046' 2024-03-31T23:24:29.772 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:29.766+0000 7f4b8e622740 1 --2- 172.21.15.46:0/588485537 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55e0bb3410c0 0x55e0bb328330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:24:29.772 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:29.766+0000 7f4b8e622740 4 client.17196 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-31T23:24:29.772 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:29.766+0000 7f4b8e622740 1 -- 172.21.15.46:0/588485537 --> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55e0bb295e30 con 0x55e0bb3410c0 2024-03-31T23:24:29.772 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:29.766+0000 7f4b89705640 1 --2- 172.21.15.46:0/588485537 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55e0bb3410c0 0x55e0bb328330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:24:29.773 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:29.766+0000 7f4b89705640 1 --2- 172.21.15.46:0/588485537 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55e0bb3410c0 0x55e0bb328330 crc :-1 s=READY pgs=190 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=954dd5d3fcedaff9 server_cookie=a5ac3f4b8830a574 in_seq=0 out_seq=0 2024-03-31T23:24:29.773 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:29.766+0000 7f4b6cff9640 10 client.17196 ms_handle_connect on v2:172.21.15.142:6838/2519143918 2024-03-31T23:24:29.794 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:29.793+0000 7f4b6cff9640 1 -- 172.21.15.46:0/588485537 <== mds.0 v2:172.21.15.142:6838/2519143918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55e0bb295e30 con 0x55e0bb3410c0 2024-03-31T23:24:29.795 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:29.793+0000 7f4b6cff9640 10 client.17196 handle_command_reply: tid=0 2024-03-31T23:24:29.795 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:29.793+0000 7f4b8e622740 1 client.17196 shutdown 2024-03-31T23:24:29.795 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:29.793+0000 7f4b8e622740 1 -- 172.21.15.46:0/588485537 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e0bb330fc0 msgr2=0x55e0bb335480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:24:29.795 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:29.793+0000 7f4b8e622740 1 --2- 172.21.15.46:0/588485537 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e0bb330fc0 0x55e0bb335480 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7f4b7c02c630 tx=0x7f4b7c032f90 comp rx=0 tx=0).stop 2024-03-31T23:24:29.795 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:29.793+0000 7f4b8e622740 1 -- 172.21.15.46:0/588485537 shutdown_connections 2024-03-31T23:24:29.795 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:29.793+0000 7f4b8e622740 1 --2- 172.21.15.46:0/588485537 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55e0bb3410c0 0x55e0bb328330 crc :-1 s=READY pgs=190 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:24:29.795 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:29.793+0000 7f4b8e622740 1 --2- 172.21.15.46:0/588485537 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e0bb330fc0 0x55e0bb335480 unknown :-1 s=CLOSED pgs=865 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:24:29.795 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:29.793+0000 7f4b8e622740 1 --2- 172.21.15.46:0/588485537 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55e0bb325a90 0x55e0bb334f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:24:29.795 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:29.793+0000 7f4b8e622740 1 --2- 172.21.15.46:0/588485537 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e0bb325170 0x55e0bb334a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:24:29.795 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:29.793+0000 7f4b8e622740 1 -- 172.21.15.46:0/588485537 >> 172.21.15.46:0/588485537 conn(0x55e0bb31ffe0 msgr2=0x55e0bb31e6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:24:29.795 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:29.793+0000 7f4b6cff9640 0 client.17196 ms_handle_reset on v2:172.21.15.142:6838/2519143918 2024-03-31T23:24:29.795 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:29.793+0000 7f4b8e622740 1 -- 172.21.15.46:0/588485537 shutdown_connections 2024-03-31T23:24:29.795 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:29.793+0000 7f4b8e622740 1 -- 172.21.15.46:0/588485537 wait complete. 2024-03-31T23:24:29.795 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:29.793+0000 7f4b8e622740 20 client.17196 trim_cache size 0 max 16384 2024-03-31T23:24:29.796 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout:{ 2024-03-31T23:24:29.796 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout: "message": "", 2024-03-31T23:24:29.796 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout: "return_code": 0 2024-03-31T23:24:29.796 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout:} 2024-03-31T23:24:29.797 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:29.794+0000 7f4b8b969640 1 -- 172.21.15.46:0/2847139084 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f4b700a2730 msgr2=0x7f4b700a4bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:24:29.797 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:29.794+0000 7f4b8b969640 1 --2- 172.21.15.46:0/2847139084 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f4b700a2730 0x7f4b700a4bf0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f4b7c005e30 tx=0x7f4b7c005d80 comp rx=0 tx=0).stop 2024-03-31T23:24:29.797 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:29.794+0000 7f4b8b969640 1 -- 172.21.15.46:0/2847139084 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4b84101260 msgr2=0x7f4b841972c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:24:29.797 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:29.794+0000 7f4b8b969640 1 --2- 172.21.15.46:0/2847139084 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4b84101260 0x7f4b841972c0 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7f4b7400e9f0 tx=0x7f4b7400eeb0 comp rx=0 tx=0).stop 2024-03-31T23:24:29.797 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:29.794+0000 7f4b8b969640 1 -- 172.21.15.46:0/2847139084 shutdown_connections 2024-03-31T23:24:29.797 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:29.794+0000 7f4b8b969640 1 --2- 172.21.15.46:0/2847139084 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f4b700a2730 0x7f4b700a4bf0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:24:29.797 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:29.794+0000 7f4b8b969640 1 --2- 172.21.15.46:0/2847139084 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4b841055f0 0x7f4b8419b640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:24:29.797 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:29.794+0000 7f4b8b969640 1 --2- 172.21.15.46:0/2847139084 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4b84101260 0x7f4b841972c0 unknown :-1 s=CLOSED pgs=862 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:24:29.797 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:29.794+0000 7f4b8b969640 1 --2- 172.21.15.46:0/2847139084 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4b841008a0 0x7f4b84196d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:24:29.797 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:29.794+0000 7f4b8b969640 1 -- 172.21.15.46:0/2847139084 >> 172.21.15.46:0/2847139084 conn(0x7f4b840774a0 msgr2=0x7f4b840ff530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:24:29.797 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:29.794+0000 7f4b8b969640 1 -- 172.21.15.46:0/2847139084 shutdown_connections 2024-03-31T23:24:29.798 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:29.794+0000 7f4b8b969640 1 -- 172.21.15.46:0/2847139084 wait complete. 2024-03-31T23:24:30.221 INFO:tasks.workunit.client.0.smithi046.stdout: 1 244612 1.78 MB/sec execute 576 sec latency 26.199 ms 2024-03-31T23:24:31.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:24:30 smithi162 ceph-mon[36237]: pgmap v400: 129 pgs: 129 active+clean; 145 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 404 KiB/s rd, 1.1 MiB/s wr, 106 op/s 2024-03-31T23:24:31.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:24:30 smithi046 ceph-mon[32498]: pgmap v400: 129 pgs: 129 active+clean; 145 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 404 KiB/s rd, 1.1 MiB/s wr, 106 op/s 2024-03-31T23:24:31.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:24:30 smithi142 ceph-mon[35271]: pgmap v400: 129 pgs: 129 active+clean; 145 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 404 KiB/s rd, 1.1 MiB/s wr, 106 op/s 2024-03-31T23:24:31.221 INFO:tasks.workunit.client.0.smithi046.stdout: 1 244855 1.79 MB/sec execute 577 sec latency 622.364 ms 2024-03-31T23:24:32.221 INFO:tasks.workunit.client.0.smithi046.stdout: 1 244855 1.78 MB/sec execute 578 sec latency 1622.425 ms 2024-03-31T23:24:33.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:24:32 smithi162 ceph-mon[36237]: pgmap v401: 129 pgs: 129 active+clean; 145 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 273 KiB/s rd, 720 KiB/s wr, 72 op/s 2024-03-31T23:24:33.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:24:32 smithi046 ceph-mon[32498]: pgmap v401: 129 pgs: 129 active+clean; 145 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 273 KiB/s rd, 720 KiB/s wr, 72 op/s 2024-03-31T23:24:33.194 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:24:32 smithi142 ceph-mon[35271]: pgmap v401: 129 pgs: 129 active+clean; 145 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 273 KiB/s rd, 720 KiB/s wr, 72 op/s 2024-03-31T23:24:33.221 INFO:tasks.workunit.client.0.smithi046.stdout: 1 244922 1.78 MB/sec execute 579 sec latency 2148.513 ms 2024-03-31T23:24:34.221 INFO:tasks.workunit.client.0.smithi046.stdout: 1 244922 1.78 MB/sec execute 580 sec latency 1230.710 ms 2024-03-31T23:24:35.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:24:34 smithi162 ceph-mon[36237]: pgmap v402: 129 pgs: 129 active+clean; 145 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 273 KiB/s rd, 720 KiB/s wr, 72 op/s 2024-03-31T23:24:35.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:24:34 smithi046 ceph-mon[32498]: pgmap v402: 129 pgs: 129 active+clean; 145 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 273 KiB/s rd, 720 KiB/s wr, 72 op/s 2024-03-31T23:24:35.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:24:34 smithi142 ceph-mon[35271]: pgmap v402: 129 pgs: 129 active+clean; 145 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 273 KiB/s rd, 720 KiB/s wr, 72 op/s 2024-03-31T23:24:35.221 INFO:tasks.workunit.client.0.smithi046.stdout: 1 244922 1.77 MB/sec execute 581 sec latency 2230.812 ms 2024-03-31T23:24:36.221 INFO:tasks.workunit.client.0.smithi046.stdout: 1 244922 1.77 MB/sec execute 582 sec latency 3230.923 ms 2024-03-31T23:24:37.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:24:36 smithi162 ceph-mon[36237]: pgmap v403: 129 pgs: 129 active+clean; 143 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 388 KiB/s rd, 1.0 MiB/s wr, 102 op/s 2024-03-31T23:24:37.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:24:36 smithi046 ceph-mon[32498]: pgmap v403: 129 pgs: 129 active+clean; 143 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 388 KiB/s rd, 1.0 MiB/s wr, 102 op/s 2024-03-31T23:24:37.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:24:36 smithi142 ceph-mon[35271]: pgmap v403: 129 pgs: 129 active+clean; 143 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 388 KiB/s rd, 1.0 MiB/s wr, 102 op/s 2024-03-31T23:24:37.222 INFO:tasks.workunit.client.0.smithi046.stdout: 1 244922 1.77 MB/sec execute 583 sec latency 4231.011 ms 2024-03-31T23:24:38.222 INFO:tasks.workunit.client.0.smithi046.stdout: 1 245054 1.77 MB/sec execute 584 sec latency 4755.540 ms 2024-03-31T23:24:39.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:24:38 smithi162 ceph-mon[36237]: pgmap v404: 129 pgs: 129 active+clean; 143 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 341 KiB/s rd, 864 KiB/s wr, 86 op/s 2024-03-31T23:24:39.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:24:38 smithi046 ceph-mon[32498]: pgmap v404: 129 pgs: 129 active+clean; 143 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 341 KiB/s rd, 864 KiB/s wr, 86 op/s 2024-03-31T23:24:39.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:24:38 smithi142 ceph-mon[35271]: pgmap v404: 129 pgs: 129 active+clean; 143 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 341 KiB/s rd, 864 KiB/s wr, 86 op/s 2024-03-31T23:24:39.222 INFO:tasks.workunit.client.0.smithi046.stdout: 1 245312 1.76 MB/sec execute 585 sec latency 14.925 ms 2024-03-31T23:24:40.222 INFO:tasks.workunit.client.0.smithi046.stdout: 1 245725 1.76 MB/sec execute 586 sec latency 16.466 ms 2024-03-31T23:24:41.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:24:40 smithi162 ceph-mon[36237]: pgmap v405: 129 pgs: 129 active+clean; 145 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 396 KiB/s rd, 1.0 MiB/s wr, 103 op/s 2024-03-31T23:24:41.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:24:40 smithi046 ceph-mon[32498]: pgmap v405: 129 pgs: 129 active+clean; 145 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 396 KiB/s rd, 1.0 MiB/s wr, 103 op/s 2024-03-31T23:24:41.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:24:40 smithi142 ceph-mon[35271]: pgmap v405: 129 pgs: 129 active+clean; 145 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 396 KiB/s rd, 1.0 MiB/s wr, 103 op/s 2024-03-31T23:24:41.222 INFO:tasks.workunit.client.0.smithi046.stdout: 1 246221 1.77 MB/sec execute 587 sec latency 22.084 ms 2024-03-31T23:24:42.222 INFO:tasks.workunit.client.0.smithi046.stdout: 1 246701 1.76 MB/sec execute 588 sec latency 27.664 ms 2024-03-31T23:24:42.894 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:42.890+0000 7feeefb93640 1 -- 172.21.15.46:0/1054905398 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7feee8101720 msgr2=0x7feee8101ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:24:42.894 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:42.890+0000 7feeefb93640 1 --2- 172.21.15.46:0/1054905398 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7feee8101720 0x7feee8101ba0 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7feed8009930 tx=0x7feed802bc30 comp rx=0 tx=0).stop 2024-03-31T23:24:42.894 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:42.890+0000 7feeefb93640 1 -- 172.21.15.46:0/1054905398 shutdown_connections 2024-03-31T23:24:42.894 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:42.890+0000 7feeefb93640 1 --2- 172.21.15.46:0/1054905398 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7feee81020e0 0x7feee810e5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:24:42.894 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:42.890+0000 7feeefb93640 1 --2- 172.21.15.46:0/1054905398 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7feee8101720 0x7feee8101ba0 unknown :-1 s=CLOSED pgs=866 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:24:42.894 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:42.890+0000 7feeefb93640 1 --2- 172.21.15.46:0/1054905398 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7feee8107730 0x7feee8107b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:24:42.894 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:42.890+0000 7feeefb93640 1 -- 172.21.15.46:0/1054905398 >> 172.21.15.46:0/1054905398 conn(0x7feee80fd230 msgr2=0x7feee80ff650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:24:42.895 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:42.891+0000 7feeefb93640 1 -- 172.21.15.46:0/1054905398 shutdown_connections 2024-03-31T23:24:42.895 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:42.891+0000 7feeefb93640 1 -- 172.21.15.46:0/1054905398 wait complete. 2024-03-31T23:24:42.895 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:42.891+0000 7feeefb93640 1 Processor -- start 2024-03-31T23:24:42.895 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:42.891+0000 7feeefb93640 1 -- start start 2024-03-31T23:24:42.895 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:42.891+0000 7feeefb93640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7feee8101720 0x7feee8197f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:24:42.895 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:42.891+0000 7feeefb93640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7feee81020e0 0x7feee8198460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:24:42.895 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:42.891+0000 7feeefb93640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7feee8107730 0x7feee819c7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:24:42.895 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:42.891+0000 7feeefb93640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7feee8111a40 con 0x7feee8107730 2024-03-31T23:24:42.895 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:42.891+0000 7feeefb93640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7feee81118c0 con 0x7feee81020e0 2024-03-31T23:24:42.895 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:42.891+0000 7feeefb93640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7feee8111bc0 con 0x7feee8101720 2024-03-31T23:24:42.895 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:42.891+0000 7feeee130640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7feee8107730 0x7feee819c7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:24:42.895 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:42.891+0000 7feeee130640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7feee8107730 0x7feee819c7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:33014/0 (socket says 172.21.15.46:33014) 2024-03-31T23:24:42.895 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:42.891+0000 7feeee130640 1 -- 172.21.15.46:0/3355647752 learned_addr learned my addr 172.21.15.46:0/3355647752 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:24:42.896 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:42.891+0000 7feeed12e640 1 --2- 172.21.15.46:0/3355647752 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7feee81020e0 0x7feee8198460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:24:42.896 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:42.891+0000 7feeed92f640 1 --2- 172.21.15.46:0/3355647752 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7feee8101720 0x7feee8197f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:24:42.896 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:42.892+0000 7feeee130640 1 -- 172.21.15.46:0/3355647752 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7feee8101720 msgr2=0x7feee8197f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:24:42.896 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:42.892+0000 7feeee130640 1 --2- 172.21.15.46:0/3355647752 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7feee8101720 0x7feee8197f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:24:42.896 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:42.892+0000 7feeee130640 1 -- 172.21.15.46:0/3355647752 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7feee81020e0 msgr2=0x7feee8198460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:24:42.896 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:42.892+0000 7feeee130640 1 --2- 172.21.15.46:0/3355647752 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7feee81020e0 0x7feee8198460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:24:42.896 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:42.892+0000 7feeee130640 1 -- 172.21.15.46:0/3355647752 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feee819cf10 con 0x7feee8107730 2024-03-31T23:24:42.896 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:42.892+0000 7feeee130640 1 --2- 172.21.15.46:0/3355647752 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7feee8107730 0x7feee819c7e0 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7feee000eb10 tx=0x7feee000c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:24:42.896 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:42.892+0000 7feedeffd640 1 -- 172.21.15.46:0/3355647752 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feee0004b20 con 0x7feee8107730 2024-03-31T23:24:42.896 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:42.892+0000 7feeefb93640 1 -- 172.21.15.46:0/3355647752 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feee819d200 con 0x7feee8107730 2024-03-31T23:24:42.896 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:42.892+0000 7feedeffd640 1 -- 172.21.15.46:0/3355647752 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7feee0004cc0 con 0x7feee8107730 2024-03-31T23:24:42.896 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:42.892+0000 7feedeffd640 1 -- 172.21.15.46:0/3355647752 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feee0010690 con 0x7feee8107730 2024-03-31T23:24:42.897 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:42.892+0000 7feeefb93640 1 -- 172.21.15.46:0/3355647752 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feee81a4a90 con 0x7feee8107730 2024-03-31T23:24:42.903 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:42.895+0000 7feedeffd640 1 -- 172.21.15.46:0/3355647752 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 25) ==== 137451+0+0 (secure 0 0 0) 0x7feee0005020 con 0x7feee8107730 2024-03-31T23:24:42.903 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:42.895+0000 7feef2779740 1 Processor -- start 2024-03-31T23:24:42.903 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:42.895+0000 7feef2779740 1 -- start start 2024-03-31T23:24:42.903 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:42.895+0000 7feedeffd640 1 --2- 172.21.15.46:0/3355647752 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7feed40a26b0 0x7feed40a4b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:24:42.903 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:42.896+0000 7feef2779740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5595858e26c0 0x5595858e2aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:24:42.903 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:42.896+0000 7feef2779740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5595858e2fe0 0x5595858eb870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:24:42.903 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:42.896+0000 7feef2779740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5595858ebdb0 0x5595858ee1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:24:42.903 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:42.896+0000 7feef2779740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x5595858458f0 con 0x5595858ebdb0 2024-03-31T23:24:42.903 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:42.896+0000 7feef2779740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x5595857ee8d0 con 0x5595858e2fe0 2024-03-31T23:24:42.903 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:42.896+0000 7feef2779740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x559585841990 con 0x5595858e26c0 2024-03-31T23:24:42.903 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:42.896+0000 7feeed12e640 1 --2- 172.21.15.46:0/3355647752 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7feed40a26b0 0x7feed40a4b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:24:42.904 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:42.896+0000 7feeed92f640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5595858e26c0 0x5595858e2aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:24:42.904 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:42.896+0000 7feeee130640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5595858ebdb0 0x5595858ee1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:24:42.904 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:42.896+0000 7feedeffd640 1 -- 172.21.15.46:0/3355647752 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(110..110 src has 1..110) ==== 9554+0+0 (secure 0 0 0) 0x7feee00cf4d0 con 0x7feee8107730 2024-03-31T23:24:42.904 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:42.896+0000 7feeed92f640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5595858e26c0 0x5595858e2aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.46:54576/0 (socket says 172.21.15.46:54576) 2024-03-31T23:24:42.904 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:42.896+0000 7feeed92f640 1 -- 172.21.15.46:0/2236214977 learned_addr learned my addr 172.21.15.46:0/2236214977 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:24:42.904 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:42.896+0000 7feeed12e640 1 --2- 172.21.15.46:0/2236214977 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5595858e2fe0 0x5595858eb870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:24:42.904 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:42.896+0000 7feeed92f640 1 -- 172.21.15.46:0/2236214977 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5595858e2fe0 msgr2=0x5595858eb870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:24:42.904 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:42.896+0000 7feeed92f640 1 --2- 172.21.15.46:0/2236214977 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5595858e2fe0 0x5595858eb870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:24:42.904 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:42.896+0000 7feeed92f640 1 -- 172.21.15.46:0/2236214977 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5595858ebdb0 msgr2=0x5595858ee1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:24:42.904 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:42.896+0000 7feeed92f640 1 --2- 172.21.15.46:0/2236214977 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5595858ebdb0 0x5595858ee1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:24:42.904 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:42.896+0000 7feeed92f640 1 -- 172.21.15.46:0/2236214977 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55958555a210 con 0x5595858e26c0 2024-03-31T23:24:42.904 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:42.896+0000 7feeee130640 1 --2- 172.21.15.46:0/2236214977 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5595858ebdb0 0x5595858ee1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:24:42.904 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:42.896+0000 7feeed12e640 1 --2- 172.21.15.46:0/3355647752 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7feed40a26b0 0x7feed40a4b70 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7feed8009900 tx=0x7feed802c6e0 comp rx=0 tx=0).ready entity=mgr.15654 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:24:42.905 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:42.896+0000 7feeed92f640 1 --2- 172.21.15.46:0/2236214977 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5595858e26c0 0x5595858e2aa0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7feee4009c70 tx=0x7feee4011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=20f3b7493d3dfb33 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:24:42.905 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:42.897+0000 7feed3fff640 1 -- 172.21.15.46:0/2236214977 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feee401e070 con 0x5595858e26c0 2024-03-31T23:24:42.905 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:42.897+0000 7feed3fff640 1 -- 172.21.15.46:0/2236214977 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7feee4009e20 con 0x5595858e26c0 2024-03-31T23:24:42.905 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:42.897+0000 7feed3fff640 1 -- 172.21.15.46:0/2236214977 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feee4019690 con 0x5595858e26c0 2024-03-31T23:24:42.905 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:42.897+0000 7feef2779740 1 -- 172.21.15.46:0/2236214977 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5595858e26c0 msgr2=0x5595858e2aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:24:42.905 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:42.897+0000 7feef2779740 1 --2- 172.21.15.46:0/2236214977 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5595858e26c0 0x5595858e2aa0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7feee4009c70 tx=0x7feee4011810 comp rx=0 tx=0).stop 2024-03-31T23:24:42.905 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:42.898+0000 7feef2779740 1 -- 172.21.15.46:0/2236214977 shutdown_connections 2024-03-31T23:24:42.905 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:42.898+0000 7feef2779740 1 --2- 172.21.15.46:0/2236214977 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5595858ebdb0 0x5595858ee1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:24:42.905 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:42.898+0000 7feef2779740 1 --2- 172.21.15.46:0/2236214977 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5595858e2fe0 0x5595858eb870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:24:42.905 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:42.898+0000 7feef2779740 1 --2- 172.21.15.46:0/2236214977 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5595858e26c0 0x5595858e2aa0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:24:42.905 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:42.898+0000 7feef2779740 1 -- 172.21.15.46:0/2236214977 >> 172.21.15.46:0/2236214977 conn(0x5595858dcfe0 msgr2=0x5595858dd3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:24:42.905 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:42.898+0000 7feef2779740 1 -- 172.21.15.46:0/2236214977 shutdown_connections 2024-03-31T23:24:42.905 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:42.898+0000 7feef2779740 1 -- 172.21.15.46:0/2236214977 wait complete. 2024-03-31T23:24:42.905 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:42.898+0000 7feef2779740 1 -- start start 2024-03-31T23:24:42.906 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:42.898+0000 7feef2779740 1 Processor -- start 2024-03-31T23:24:42.906 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:42.899+0000 7feef2779740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5595858e26c0 0x5595858e8e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:24:42.906 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:42.899+0000 7feef2779740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5595858e2fe0 0x5595858e93c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:24:42.906 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:42.899+0000 7feef2779740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5595858ebdb0 0x5595858e9900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:24:42.906 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:42.899+0000 7feef2779740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x5595858f2df0 con 0x5595858e26c0 2024-03-31T23:24:42.906 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:42.899+0000 7feef2779740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x5595857e3f80 con 0x5595858ebdb0 2024-03-31T23:24:42.906 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:42.899+0000 7feef2779740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5595858f2f70 con 0x5595858e2fe0 2024-03-31T23:24:42.906 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:42.899+0000 7feeed92f640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5595858e26c0 0x5595858e8e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:24:42.906 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:42.899+0000 7feeed12e640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5595858e2fe0 0x5595858e93c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:24:42.906 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:42.899+0000 7feef2779740 10 client.0 _ll_register_callbacks cb 0x55958583c290 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T23:24:42.906 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:42.899+0000 7feeed92f640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5595858e26c0 0x5595858e8e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:33026/0 (socket says 172.21.15.46:33026) 2024-03-31T23:24:42.906 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:42.899+0000 7feeed92f640 1 -- 172.21.15.46:0/771814453 learned_addr learned my addr 172.21.15.46:0/771814453 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:24:42.906 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:42.899+0000 7feeee130640 1 --2- 172.21.15.46:0/771814453 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5595858ebdb0 0x5595858e9900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:24:42.907 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:42.899+0000 7feeed92f640 1 -- 172.21.15.46:0/771814453 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5595858e2fe0 msgr2=0x5595858e93c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:24:42.907 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:42.899+0000 7feeed92f640 1 --2- 172.21.15.46:0/771814453 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5595858e2fe0 0x5595858e93c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:24:42.907 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:42.899+0000 7feeed92f640 1 -- 172.21.15.46:0/771814453 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5595858ebdb0 msgr2=0x5595858e9900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:24:42.907 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:42.899+0000 7feeed92f640 1 --2- 172.21.15.46:0/771814453 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5595858ebdb0 0x5595858e9900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:24:42.907 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:42.899+0000 7feeed92f640 1 -- 172.21.15.46:0/771814453 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559585626620 con 0x5595858e26c0 2024-03-31T23:24:42.907 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:42.900+0000 7feeed92f640 1 --2- 172.21.15.46:0/771814453 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5595858e26c0 0x5595858e8e80 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7feee400a010 tx=0x7feee4003820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=336f49a5d0dbf40b server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:24:42.907 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:42.900+0000 7feed17fa640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-31T23:24:42.907 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:42.900+0000 7feed17fa640 1 -- 172.21.15.46:0/771814453 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feee4027060 con 0x5595858e26c0 2024-03-31T23:24:42.907 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:42.900+0000 7feed17fa640 1 -- 172.21.15.46:0/771814453 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(110..110 src has 1..110) ==== 9554+0+0 (secure 0 0 0) 0x7feee4005030 con 0x5595858e26c0 2024-03-31T23:24:42.907 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:42.900+0000 7feef2779740 1 -- 172.21.15.46:0/771814453 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7feee4009e20 con 0x5595858e26c0 2024-03-31T23:24:42.907 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:42.900+0000 7feed17fa640 1 -- 172.21.15.46:0/771814453 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7feee402e050 con 0x5595858e26c0 2024-03-31T23:24:42.907 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:42.900+0000 7feef2779740 10 client.17208 fetch_fsmap learned FSMap version 36 2024-03-31T23:24:42.907 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:42.900+0000 7feef2779740 1 -- 172.21.15.46:0/771814453 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x5595858fed50 con 0x5595858e26c0 2024-03-31T23:24:42.908 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:42.900+0000 7feed17fa640 1 -- 172.21.15.46:0/771814453 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7feee4009e20 con 0x5595858e26c0 2024-03-31T23:24:42.908 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:42.900+0000 7feed17fa640 1 -- 172.21.15.46:0/771814453 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 36) ==== 8362+0+0 (secure 0 0 0) 0x7feee4021030 con 0x5595858e26c0 2024-03-31T23:24:42.908 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:42.900+0000 7feef2779740 10 client.17208 fetch_fsmap finished waiting for FSMap version 36 2024-03-31T23:24:42.908 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:42.901+0000 7feef2779740 10 client.17208 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-31T23:24:42.908 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:42.901+0000 7feef2779740 20 client.17208 populate_metadata read hostname 'smithi046' 2024-03-31T23:24:42.908 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:42.901+0000 7feef2779740 1 --2- 172.21.15.46:0/771814453 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x559585900150 0x559585902540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:24:42.908 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:42.901+0000 7feef2779740 4 client.17208 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-31T23:24:42.908 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:42.901+0000 7feef2779740 1 -- 172.21.15.46:0/771814453 --> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559585859120 con 0x559585900150 2024-03-31T23:24:42.908 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:42.901+0000 7feeed12e640 1 --2- 172.21.15.46:0/771814453 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x559585900150 0x559585902540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:24:42.908 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:42.901+0000 7feeed12e640 1 --2- 172.21.15.46:0/771814453 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x559585900150 0x559585902540 crc :-1 s=READY pgs=191 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6d4c169f8793e0f6 server_cookie=573555f400681495 in_seq=0 out_seq=0 2024-03-31T23:24:42.908 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:42.901+0000 7feed17fa640 10 client.17208 ms_handle_connect on v2:172.21.15.142:6838/2519143918 2024-03-31T23:24:42.909 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:42.902+0000 7feed17fa640 1 -- 172.21.15.46:0/771814453 <== mds.0 v2:172.21.15.142:6838/2519143918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x559585859120 con 0x559585900150 2024-03-31T23:24:42.909 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:42.902+0000 7feed17fa640 10 client.17208 handle_command_reply: tid=0 2024-03-31T23:24:42.910 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:42.902+0000 7feef2779740 1 client.17208 shutdown 2024-03-31T23:24:42.910 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:42.902+0000 7feef2779740 1 -- 172.21.15.46:0/771814453 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5595858e26c0 msgr2=0x5595858e8e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:24:42.910 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:42.902+0000 7feef2779740 1 --2- 172.21.15.46:0/771814453 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5595858e26c0 0x5595858e8e80 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7feee400a010 tx=0x7feee4003820 comp rx=0 tx=0).stop 2024-03-31T23:24:42.910 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:42.903+0000 7feef2779740 1 -- 172.21.15.46:0/771814453 shutdown_connections 2024-03-31T23:24:42.910 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:42.903+0000 7feef2779740 1 --2- 172.21.15.46:0/771814453 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x559585900150 0x559585902540 crc :-1 s=READY pgs=191 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:24:42.910 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:42.903+0000 7feef2779740 1 --2- 172.21.15.46:0/771814453 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5595858ebdb0 0x5595858e9900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:24:42.910 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:42.903+0000 7feef2779740 1 --2- 172.21.15.46:0/771814453 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5595858e2fe0 0x5595858e93c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:24:42.910 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:42.903+0000 7feef2779740 1 --2- 172.21.15.46:0/771814453 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5595858e26c0 0x5595858e8e80 unknown :-1 s=CLOSED pgs=868 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:24:42.910 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:42.903+0000 7feed17fa640 0 client.17208 ms_handle_reset on v2:172.21.15.142:6838/2519143918 2024-03-31T23:24:42.910 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:42.903+0000 7feef2779740 1 -- 172.21.15.46:0/771814453 >> 172.21.15.46:0/771814453 conn(0x5595858dcfe0 msgr2=0x5595858ded30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:24:42.910 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:42.903+0000 7feef2779740 1 -- 172.21.15.46:0/771814453 shutdown_connections 2024-03-31T23:24:42.911 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:42.903+0000 7feef2779740 1 -- 172.21.15.46:0/771814453 wait complete. 2024-03-31T23:24:42.911 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:42.903+0000 7feef2779740 20 client.17208 trim_cache size 0 max 16384 2024-03-31T23:24:42.925 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:42.919+0000 7feef2779740 1 Processor -- start 2024-03-31T23:24:42.926 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:42.919+0000 7feef2779740 1 -- start start 2024-03-31T23:24:42.926 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:42.920+0000 7feef2779740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5595858e26c0 0x5595858eb5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:24:42.926 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:42.920+0000 7feef2779740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5595858e2fe0 0x5595858fbb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:24:42.926 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:42.920+0000 7feef2779740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5595858ebdb0 0x5595858dd3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:24:42.926 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:42.920+0000 7feef2779740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7feee81af3e0 con 0x5595858ebdb0 2024-03-31T23:24:42.926 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:42.920+0000 7feef2779740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7feee81acc50 con 0x5595858e2fe0 2024-03-31T23:24:42.926 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:42.920+0000 7feef2779740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7feee81aaac0 con 0x5595858e26c0 2024-03-31T23:24:42.926 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:42.920+0000 7feeee130640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5595858ebdb0 0x5595858dd3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:24:42.926 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:42.920+0000 7feeee130640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5595858ebdb0 0x5595858dd3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:33028/0 (socket says 172.21.15.46:33028) 2024-03-31T23:24:42.926 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:42.920+0000 7feeee130640 1 -- 172.21.15.46:0/310549918 learned_addr learned my addr 172.21.15.46:0/310549918 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:24:42.926 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:42.920+0000 7feeed12e640 1 --2- 172.21.15.46:0/310549918 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5595858e2fe0 0x5595858fbb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:24:42.926 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:42.920+0000 7feeed92f640 1 --2- 172.21.15.46:0/310549918 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5595858e26c0 0x5595858eb5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:24:42.927 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:42.920+0000 7feeee130640 1 -- 172.21.15.46:0/310549918 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5595858e26c0 msgr2=0x5595858eb5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:24:42.927 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:42.920+0000 7feeee130640 1 --2- 172.21.15.46:0/310549918 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5595858e26c0 0x5595858eb5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:24:42.927 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:42.920+0000 7feeee130640 1 -- 172.21.15.46:0/310549918 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5595858e2fe0 msgr2=0x5595858fbb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:24:42.927 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:42.920+0000 7feeee130640 1 --2- 172.21.15.46:0/310549918 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5595858e2fe0 0x5595858fbb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:24:42.927 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:42.920+0000 7feeee130640 1 -- 172.21.15.46:0/310549918 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5595858e7e30 con 0x5595858ebdb0 2024-03-31T23:24:42.927 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:42.920+0000 7feeee130640 1 --2- 172.21.15.46:0/310549918 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5595858ebdb0 0x5595858dd3c0 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7feee00cc8c0 tx=0x7feee009b6d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=23cd76247cdac46 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:24:42.927 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:42.920+0000 7feed37fe640 1 -- 172.21.15.46:0/310549918 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feee009b840 con 0x5595858ebdb0 2024-03-31T23:24:42.927 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:42.920+0000 7feed37fe640 1 -- 172.21.15.46:0/310549918 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7feee009f990 con 0x5595858ebdb0 2024-03-31T23:24:42.928 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:42.920+0000 7feed37fe640 1 -- 172.21.15.46:0/310549918 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feee004f2f0 con 0x5595858ebdb0 2024-03-31T23:24:42.928 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:42.921+0000 7feef2779740 1 -- 172.21.15.46:0/310549918 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5595858ebdb0 msgr2=0x5595858dd3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:24:42.928 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:42.921+0000 7feef2779740 1 --2- 172.21.15.46:0/310549918 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5595858ebdb0 0x5595858dd3c0 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7feee00cc8c0 tx=0x7feee009b6d0 comp rx=0 tx=0).stop 2024-03-31T23:24:42.928 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:42.921+0000 7feef2779740 1 -- 172.21.15.46:0/310549918 shutdown_connections 2024-03-31T23:24:42.928 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:42.921+0000 7feef2779740 1 --2- 172.21.15.46:0/310549918 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5595858ebdb0 0x5595858dd3c0 unknown :-1 s=CLOSED pgs=869 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:24:42.928 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:42.921+0000 7feef2779740 1 --2- 172.21.15.46:0/310549918 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5595858e2fe0 0x5595858fbb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:24:42.928 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:42.921+0000 7feef2779740 1 --2- 172.21.15.46:0/310549918 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5595858e26c0 0x5595858eb5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:24:42.928 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:42.921+0000 7feef2779740 1 -- 172.21.15.46:0/310549918 >> 172.21.15.46:0/310549918 conn(0x5595858dcfe0 msgr2=0x5595858dbf80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:24:42.928 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:42.921+0000 7feef2779740 1 -- 172.21.15.46:0/310549918 shutdown_connections 2024-03-31T23:24:42.928 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:42.921+0000 7feef2779740 1 -- 172.21.15.46:0/310549918 wait complete. 2024-03-31T23:24:42.929 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:42.922+0000 7feef2779740 1 -- start start 2024-03-31T23:24:42.929 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:42.922+0000 7feef2779740 1 Processor -- start 2024-03-31T23:24:42.929 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:42.922+0000 7feef2779740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5595858e26c0 0x5595858f3bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:24:42.929 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:42.922+0000 7feef2779740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5595858e2fe0 0x5595858f4100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:24:42.929 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:42.922+0000 7feef2779740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5595858ebdb0 0x5595858f4640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:24:42.929 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:42.922+0000 7feef2779740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7feee81ab110 con 0x5595858e2fe0 2024-03-31T23:24:42.929 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:42.922+0000 7feef2779740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7feee81ac580 con 0x5595858ebdb0 2024-03-31T23:24:42.929 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:42.922+0000 7feef2779740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7feee81ab7e0 con 0x5595858e26c0 2024-03-31T23:24:42.929 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:42.922+0000 7feef2779740 10 client.0 _ll_register_callbacks cb 0x7feee401c070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T23:24:42.929 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:42.922+0000 7feeed12e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5595858e2fe0 0x5595858f4100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:24:42.929 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:42.922+0000 7feeed92f640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5595858e26c0 0x5595858f3bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:24:42.929 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:42.922+0000 7feeed12e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5595858e2fe0 0x5595858f4100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:33042/0 (socket says 172.21.15.46:33042) 2024-03-31T23:24:42.929 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:42.922+0000 7feeed12e640 1 -- 172.21.15.46:0/1418854315 learned_addr learned my addr 172.21.15.46:0/1418854315 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:24:42.930 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:42.922+0000 7feeee130640 1 --2- 172.21.15.46:0/1418854315 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5595858ebdb0 0x5595858f4640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:24:42.930 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:42.923+0000 7feeed12e640 1 -- 172.21.15.46:0/1418854315 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5595858e26c0 msgr2=0x5595858f3bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:24:42.930 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:42.923+0000 7feeed12e640 1 --2- 172.21.15.46:0/1418854315 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5595858e26c0 0x5595858f3bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:24:42.930 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:42.923+0000 7feeed12e640 1 -- 172.21.15.46:0/1418854315 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5595858ebdb0 msgr2=0x5595858f4640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:24:42.930 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:42.923+0000 7feeed12e640 1 --2- 172.21.15.46:0/1418854315 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5595858ebdb0 0x5595858f4640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:24:42.930 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:42.923+0000 7feeed12e640 1 -- 172.21.15.46:0/1418854315 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559585965390 con 0x5595858e2fe0 2024-03-31T23:24:42.930 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:42.923+0000 7feeed92f640 1 --2- 172.21.15.46:0/1418854315 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5595858e26c0 0x5595858f3bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:24:42.930 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:42.923+0000 7feeed12e640 1 --2- 172.21.15.46:0/1418854315 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5595858e2fe0 0x5595858f4100 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7feed8005e90 tx=0x7feed8046040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=30740c38e3cd687a server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:24:42.930 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:42.923+0000 7feed17fa640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-31T23:24:42.930 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:42.923+0000 7feed17fa640 1 -- 172.21.15.46:0/1418854315 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feed805c070 con 0x5595858e2fe0 2024-03-31T23:24:42.930 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:42.923+0000 7feed17fa640 1 -- 172.21.15.46:0/1418854315 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(110..110 src has 1..110) ==== 9554+0+0 (secure 0 0 0) 0x7feed80551c0 con 0x5595858e2fe0 2024-03-31T23:24:42.931 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:42.923+0000 7feef2779740 1 -- 172.21.15.46:0/1418854315 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7feee009f990 con 0x5595858e2fe0 2024-03-31T23:24:42.931 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:42.923+0000 7feed17fa640 1 -- 172.21.15.46:0/1418854315 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7feed8068050 con 0x5595858e2fe0 2024-03-31T23:24:42.931 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:42.923+0000 7feef2779740 10 client.17214 fetch_fsmap learned FSMap version 36 2024-03-31T23:24:42.931 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:42.923+0000 7feef2779740 1 -- 172.21.15.46:0/1418854315 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x559585966700 con 0x5595858e2fe0 2024-03-31T23:24:42.931 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:42.923+0000 7feed17fa640 1 -- 172.21.15.46:0/1418854315 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7feee009f990 con 0x5595858e2fe0 2024-03-31T23:24:42.931 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:42.923+0000 7feed17fa640 1 -- 172.21.15.46:0/1418854315 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 36) ==== 8362+0+0 (secure 0 0 0) 0x7feed804f040 con 0x5595858e2fe0 2024-03-31T23:24:42.931 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:42.924+0000 7feef2779740 10 client.17214 fetch_fsmap finished waiting for FSMap version 36 2024-03-31T23:24:42.931 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:42.924+0000 7feef2779740 10 client.17214 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-31T23:24:42.931 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:42.924+0000 7feef2779740 20 client.17214 populate_metadata read hostname 'smithi046' 2024-03-31T23:24:42.931 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:42.924+0000 7feef2779740 1 --2- 172.21.15.46:0/1418854315 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x559585966ec0 0x5595859692e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:24:42.931 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:42.924+0000 7feef2779740 4 client.17214 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-31T23:24:42.931 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:42.924+0000 7feef2779740 1 -- 172.21.15.46:0/1418854315 --> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] -- command(tid 0: {"prefix": "flush journal"}) -- 0x559585852e30 con 0x559585966ec0 2024-03-31T23:24:42.931 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:42.924+0000 7feeed92f640 1 --2- 172.21.15.46:0/1418854315 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x559585966ec0 0x5595859692e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:24:42.931 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:42.924+0000 7feeed92f640 1 --2- 172.21.15.46:0/1418854315 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x559585966ec0 0x5595859692e0 crc :-1 s=READY pgs=192 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=27cfbd9dd0151429 server_cookie=209744fe54089026 in_seq=0 out_seq=0 2024-03-31T23:24:42.931 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:42.924+0000 7feed17fa640 10 client.17214 ms_handle_connect on v2:172.21.15.142:6838/2519143918 2024-03-31T23:24:42.993 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:42.991+0000 7feed17fa640 1 -- 172.21.15.46:0/1418854315 <== mds.0 v2:172.21.15.142:6838/2519143918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x559585852e30 con 0x559585966ec0 2024-03-31T23:24:42.993 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:42.991+0000 7feed17fa640 10 client.17214 handle_command_reply: tid=0 2024-03-31T23:24:42.993 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:42.991+0000 7feef2779740 1 client.17214 shutdown 2024-03-31T23:24:42.993 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:42.991+0000 7feef2779740 1 -- 172.21.15.46:0/1418854315 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5595858e2fe0 msgr2=0x5595858f4100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:24:42.994 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:42.991+0000 7feef2779740 1 --2- 172.21.15.46:0/1418854315 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5595858e2fe0 0x5595858f4100 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7feed8005e90 tx=0x7feed8046040 comp rx=0 tx=0).stop 2024-03-31T23:24:42.994 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:42.992+0000 7feef2779740 1 -- 172.21.15.46:0/1418854315 shutdown_connections 2024-03-31T23:24:42.994 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:42.992+0000 7feef2779740 1 --2- 172.21.15.46:0/1418854315 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x559585966ec0 0x5595859692e0 crc :-1 s=READY pgs=192 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:24:42.994 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:42.992+0000 7feef2779740 1 --2- 172.21.15.46:0/1418854315 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5595858ebdb0 0x5595858f4640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:24:42.994 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:42.992+0000 7feef2779740 1 --2- 172.21.15.46:0/1418854315 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5595858e2fe0 0x5595858f4100 unknown :-1 s=CLOSED pgs=870 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:24:42.994 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:42.992+0000 7feef2779740 1 --2- 172.21.15.46:0/1418854315 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5595858e26c0 0x5595858f3bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:24:42.994 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:42.992+0000 7feed17fa640 0 client.17214 ms_handle_reset on v2:172.21.15.142:6838/2519143918 2024-03-31T23:24:42.994 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:42.992+0000 7feef2779740 1 -- 172.21.15.46:0/1418854315 >> 172.21.15.46:0/1418854315 conn(0x5595858dcfe0 msgr2=0x5595858dbf80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:24:42.994 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:42.992+0000 7feef2779740 1 -- 172.21.15.46:0/1418854315 shutdown_connections 2024-03-31T23:24:42.994 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:42.992+0000 7feef2779740 1 -- 172.21.15.46:0/1418854315 wait complete. 2024-03-31T23:24:42.994 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:42.992+0000 7feef2779740 20 client.17214 trim_cache size 0 max 16384 2024-03-31T23:24:42.995 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout:{ 2024-03-31T23:24:42.995 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout: "message": "", 2024-03-31T23:24:42.995 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout: "return_code": 0 2024-03-31T23:24:42.995 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout:} 2024-03-31T23:24:42.996 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:42.992+0000 7feeefb93640 1 -- 172.21.15.46:0/3355647752 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7feed40a26b0 msgr2=0x7feed40a4b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:24:42.996 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:42.992+0000 7feeefb93640 1 --2- 172.21.15.46:0/3355647752 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7feed40a26b0 0x7feed40a4b70 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7feed8009900 tx=0x7feed802c6e0 comp rx=0 tx=0).stop 2024-03-31T23:24:42.996 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:42.992+0000 7feeefb93640 1 -- 172.21.15.46:0/3355647752 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7feee8107730 msgr2=0x7feee819c7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:24:42.996 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:42.992+0000 7feeefb93640 1 --2- 172.21.15.46:0/3355647752 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7feee8107730 0x7feee819c7e0 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7feee000eb10 tx=0x7feee000c2d0 comp rx=0 tx=0).stop 2024-03-31T23:24:42.996 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:42.992+0000 7feeefb93640 1 -- 172.21.15.46:0/3355647752 shutdown_connections 2024-03-31T23:24:42.996 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:42.992+0000 7feeefb93640 1 --2- 172.21.15.46:0/3355647752 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7feed40a26b0 0x7feed40a4b70 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:24:42.996 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:42.992+0000 7feeefb93640 1 --2- 172.21.15.46:0/3355647752 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7feee8107730 0x7feee819c7e0 unknown :-1 s=CLOSED pgs=867 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:24:42.996 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:42.992+0000 7feeefb93640 1 --2- 172.21.15.46:0/3355647752 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7feee81020e0 0x7feee8198460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:24:42.996 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:42.992+0000 7feeefb93640 1 --2- 172.21.15.46:0/3355647752 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7feee8101720 0x7feee8197f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:24:42.996 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:42.992+0000 7feeefb93640 1 -- 172.21.15.46:0/3355647752 >> 172.21.15.46:0/3355647752 conn(0x7feee80fd230 msgr2=0x7feee80fd9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:24:42.996 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:42.992+0000 7feeefb93640 1 -- 172.21.15.46:0/3355647752 shutdown_connections 2024-03-31T23:24:42.996 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:42.992+0000 7feeefb93640 1 -- 172.21.15.46:0/3355647752 wait complete. 2024-03-31T23:24:43.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:24:42 smithi162 ceph-mon[36237]: pgmap v406: 129 pgs: 129 active+clean; 145 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 170 KiB/s rd, 520 KiB/s wr, 48 op/s 2024-03-31T23:24:43.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:24:42 smithi046 ceph-mon[32498]: pgmap v406: 129 pgs: 129 active+clean; 145 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 170 KiB/s rd, 520 KiB/s wr, 48 op/s 2024-03-31T23:24:43.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:24:42 smithi142 ceph-mon[35271]: pgmap v406: 129 pgs: 129 active+clean; 145 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 170 KiB/s rd, 520 KiB/s wr, 48 op/s 2024-03-31T23:24:43.222 INFO:tasks.workunit.client.0.smithi046.stdout: 1 247198 1.76 MB/sec execute 589 sec latency 21.548 ms 2024-03-31T23:24:44.222 INFO:tasks.workunit.client.0.smithi046.stdout: 1 247779 1.77 MB/sec execute 590 sec latency 16.458 ms 2024-03-31T23:24:45.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:24:44 smithi162 ceph-mon[36237]: pgmap v407: 129 pgs: 129 active+clean; 145 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 170 KiB/s rd, 520 KiB/s wr, 48 op/s 2024-03-31T23:24:45.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:24:44 smithi046 ceph-mon[32498]: pgmap v407: 129 pgs: 129 active+clean; 145 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 170 KiB/s rd, 520 KiB/s wr, 48 op/s 2024-03-31T23:24:45.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:24:44 smithi142 ceph-mon[35271]: pgmap v407: 129 pgs: 129 active+clean; 145 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 170 KiB/s rd, 520 KiB/s wr, 48 op/s 2024-03-31T23:24:45.222 INFO:tasks.workunit.client.0.smithi046.stdout: 1 248174 1.77 MB/sec execute 591 sec latency 15.941 ms 2024-03-31T23:24:46.222 INFO:tasks.workunit.client.0.smithi046.stdout: 1 248402 1.77 MB/sec execute 592 sec latency 679.586 ms 2024-03-31T23:24:47.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:24:46 smithi162 ceph-mon[36237]: pgmap v408: 129 pgs: 129 active+clean; 146 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 401 KiB/s rd, 1.0 MiB/s wr, 107 op/s 2024-03-31T23:24:47.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:24:46 smithi046 ceph-mon[32498]: pgmap v408: 129 pgs: 129 active+clean; 146 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 401 KiB/s rd, 1.0 MiB/s wr, 107 op/s 2024-03-31T23:24:47.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:24:46 smithi142 ceph-mon[35271]: pgmap v408: 129 pgs: 129 active+clean; 146 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 401 KiB/s rd, 1.0 MiB/s wr, 107 op/s 2024-03-31T23:24:47.222 INFO:tasks.workunit.client.0.smithi046.stdout: 1 248402 1.77 MB/sec execute 593 sec latency 1679.700 ms 2024-03-31T23:24:48.222 INFO:tasks.workunit.client.0.smithi046.stdout: 1 248469 1.77 MB/sec execute 594 sec latency 2204.500 ms 2024-03-31T23:24:49.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:24:48 smithi162 ceph-mon[36237]: pgmap v409: 129 pgs: 129 active+clean; 146 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 285 KiB/s rd, 737 KiB/s wr, 76 op/s 2024-03-31T23:24:49.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:24:48 smithi046 ceph-mon[32498]: pgmap v409: 129 pgs: 129 active+clean; 146 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 285 KiB/s rd, 737 KiB/s wr, 76 op/s 2024-03-31T23:24:49.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:24:48 smithi142 ceph-mon[35271]: pgmap v409: 129 pgs: 129 active+clean; 146 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 285 KiB/s rd, 737 KiB/s wr, 76 op/s 2024-03-31T23:24:49.222 INFO:tasks.workunit.client.0.smithi046.stdout: 1 248469 1.76 MB/sec execute 595 sec latency 1239.236 ms 2024-03-31T23:24:50.222 INFO:tasks.workunit.client.0.smithi046.stdout: 1 248469 1.76 MB/sec execute 596 sec latency 2239.349 ms 2024-03-31T23:24:51.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:24:50 smithi162 ceph-mon[36237]: pgmap v410: 129 pgs: 129 active+clean; 148 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 387 KiB/s rd, 1.0 MiB/s wr, 103 op/s 2024-03-31T23:24:51.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:24:50 smithi046 ceph-mon[32498]: pgmap v410: 129 pgs: 129 active+clean; 148 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 387 KiB/s rd, 1.0 MiB/s wr, 103 op/s 2024-03-31T23:24:51.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:24:50 smithi142 ceph-mon[35271]: pgmap v410: 129 pgs: 129 active+clean; 148 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 387 KiB/s rd, 1.0 MiB/s wr, 103 op/s 2024-03-31T23:24:51.222 INFO:tasks.workunit.client.0.smithi046.stdout: 1 248469 1.76 MB/sec execute 597 sec latency 3239.453 ms 2024-03-31T23:24:52.223 INFO:tasks.workunit.client.0.smithi046.stdout: 1 248469 1.76 MB/sec execute 598 sec latency 4239.574 ms 2024-03-31T23:24:52.995 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:24:52 smithi142 ceph-mon[35271]: pgmap v411: 129 pgs: 129 active+clean; 148 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 333 KiB/s rd, 858 KiB/s wr, 85 op/s 2024-03-31T23:24:53.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:24:52 smithi162 ceph-mon[36237]: pgmap v411: 129 pgs: 129 active+clean; 148 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 333 KiB/s rd, 858 KiB/s wr, 85 op/s 2024-03-31T23:24:53.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:24:52 smithi046 ceph-mon[32498]: pgmap v411: 129 pgs: 129 active+clean; 148 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 333 KiB/s rd, 858 KiB/s wr, 85 op/s 2024-03-31T23:24:53.222 INFO:tasks.workunit.client.0.smithi046.stdout: 1 248601 1.75 MB/sec execute 599 sec latency 4763.130 ms 2024-03-31T23:24:54.223 INFO:tasks.workunit.client.0.smithi046.stdout: 1 cleanup 600 sec 2024-03-31T23:24:55.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:24:54 smithi162 ceph-mon[36237]: pgmap v412: 129 pgs: 129 active+clean; 148 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 333 KiB/s rd, 858 KiB/s wr, 85 op/s 2024-03-31T23:24:55.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:24:54 smithi046 ceph-mon[32498]: pgmap v412: 129 pgs: 129 active+clean; 148 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 333 KiB/s rd, 858 KiB/s wr, 85 op/s 2024-03-31T23:24:55.163 INFO:tasks.workunit.client.0.smithi046.stdout: 0 cleanup 601 sec 2024-03-31T23:24:55.164 INFO:tasks.workunit.client.0.smithi046.stdout: 2024-03-31T23:24:55.164 INFO:tasks.workunit.client.0.smithi046.stdout: Operation Count AvgLat MaxLat 2024-03-31T23:24:55.164 INFO:tasks.workunit.client.0.smithi046.stdout: ---------------------------------------- 2024-03-31T23:24:55.164 INFO:tasks.workunit.client.0.smithi046.stdout: NTCreateX 33688 3.800 239.342 2024-03-31T23:24:55.164 INFO:tasks.workunit.client.0.smithi046.stdout: Close 24732 0.049 1.897 2024-03-31T23:24:55.164 INFO:tasks.workunit.client.0.smithi046.stdout: Rename 1427 124.426 4783.443 2024-03-31T23:24:55.164 INFO:tasks.workunit.client.0.smithi046.stdout: Unlink 6785 4.402 13.369 2024-03-31T23:24:55.164 INFO:tasks.workunit.client.0.smithi046.stdout: Qpathinfo 30533 3.404 2812.293 2024-03-31T23:24:55.164 INFO:tasks.workunit.client.0.smithi046.stdout: Qfileinfo 5327 0.074 0.928 2024-03-31T23:24:55.164 INFO:tasks.workunit.client.0.smithi046.stdout: Qfsinfo 5594 0.300 3.010 2024-03-31T23:24:55.164 INFO:tasks.workunit.client.0.smithi046.stdout: Sfileinfo 2735 1.346 20.073 2024-03-31T23:24:55.164 INFO:tasks.workunit.client.0.smithi046.stdout: Find 11795 12.091 3102.164 2024-03-31T23:24:55.164 INFO:tasks.workunit.client.0.smithi046.stdout: WriteX 16629 0.171 4.037 2024-03-31T23:24:55.165 INFO:tasks.workunit.client.0.smithi046.stdout: ReadX 52871 0.003 0.150 2024-03-31T23:24:55.165 INFO:tasks.workunit.client.0.smithi046.stdout: LockX 110 0.638 0.884 2024-03-31T23:24:55.165 INFO:tasks.workunit.client.0.smithi046.stdout: UnlockX 110 0.588 0.918 2024-03-31T23:24:55.165 INFO:tasks.workunit.client.0.smithi046.stdout: Flush 2343 3.181 24.118 2024-03-31T23:24:55.165 INFO:tasks.workunit.client.0.smithi046.stdout: 2024-03-31T23:24:55.165 INFO:tasks.workunit.client.0.smithi046.stdout:Throughput 1.75395 MB/sec 1 clients 1 procs max_latency=4783.457 ms 2024-03-31T23:24:55.165 INFO:tasks.workunit.client.0.smithi046.stderr:+ dbench 10 2024-03-31T23:24:55.165 INFO:tasks.workunit.client.0.smithi046.stdout:dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 2024-03-31T23:24:55.166 INFO:tasks.workunit.client.0.smithi046.stdout: 2024-03-31T23:24:55.166 INFO:tasks.workunit.client.0.smithi046.stdout:Running for 600 seconds with load '/usr/share/dbench/client.txt' and minimum warmup 120 secs 2024-03-31T23:24:55.166 INFO:tasks.workunit.client.0.smithi046.stdout:8 of 10 processes prepared for launch 0 sec 2024-03-31T23:24:55.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:24:54 smithi142 ceph-mon[35271]: pgmap v412: 129 pgs: 129 active+clean; 148 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 333 KiB/s rd, 858 KiB/s wr, 85 op/s 2024-03-31T23:24:55.265 INFO:tasks.workunit.client.0.smithi046.stdout:10 of 10 processes prepared for launch 0 sec 2024-03-31T23:24:55.265 INFO:tasks.workunit.client.0.smithi046.stdout:releasing clients 2024-03-31T23:24:56.118 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:56.114+0000 7f5ac73e5640 1 -- 172.21.15.46:0/3519214691 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5ac0100d80 msgr2=0x7f5ac0101200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:24:56.118 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:56.114+0000 7f5ac73e5640 1 --2- 172.21.15.46:0/3519214691 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5ac0100d80 0x7f5ac0101200 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7f5ab0009900 tx=0x7f5ab002bc30 comp rx=0 tx=0).stop 2024-03-31T23:24:56.118 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:56.114+0000 7f5ac73e5640 1 -- 172.21.15.46:0/3519214691 shutdown_connections 2024-03-31T23:24:56.118 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:56.114+0000 7f5ac73e5640 1 --2- 172.21.15.46:0/3519214691 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5ac0101740 0x7f5ac010d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:24:56.118 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:56.114+0000 7f5ac73e5640 1 --2- 172.21.15.46:0/3519214691 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5ac0100d80 0x7f5ac0101200 unknown :-1 s=CLOSED pgs=871 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:24:56.119 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:56.114+0000 7f5ac73e5640 1 --2- 172.21.15.46:0/3519214691 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5ac01055c0 0x7f5ac01059a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:24:56.119 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:56.114+0000 7f5ac73e5640 1 -- 172.21.15.46:0/3519214691 >> 172.21.15.46:0/3519214691 conn(0x7f5ac0076be0 msgr2=0x7f5ac00fdf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:24:56.119 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:56.114+0000 7f5ac73e5640 1 -- 172.21.15.46:0/3519214691 shutdown_connections 2024-03-31T23:24:56.119 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:56.114+0000 7f5ac73e5640 1 -- 172.21.15.46:0/3519214691 wait complete. 2024-03-31T23:24:56.119 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:56.115+0000 7f5ac73e5640 1 Processor -- start 2024-03-31T23:24:56.119 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:56.115+0000 7f5ac73e5640 1 -- start start 2024-03-31T23:24:56.119 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:56.115+0000 7f5ac73e5640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5ac0100d80 0x7f5ac0197e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:24:56.119 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:56.115+0000 7f5ac73e5640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5ac0101740 0x7f5ac01983d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:24:56.119 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:56.115+0000 7f5ac73e5640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5ac01055c0 0x7f5ac019c700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:24:56.119 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:56.115+0000 7f5ac73e5640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f5ac01119b0 con 0x7f5ac01055c0 2024-03-31T23:24:56.119 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:56.115+0000 7f5ac73e5640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f5ac0111830 con 0x7f5ac0100d80 2024-03-31T23:24:56.119 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:56.115+0000 7f5ac73e5640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f5ac0111b30 con 0x7f5ac0101740 2024-03-31T23:24:56.119 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:56.115+0000 7f5ac5982640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5ac01055c0 0x7f5ac019c700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:24:56.120 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:56.115+0000 7f5ac5982640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5ac01055c0 0x7f5ac019c700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:36894/0 (socket says 172.21.15.46:36894) 2024-03-31T23:24:56.120 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:56.115+0000 7f5ac5982640 1 -- 172.21.15.46:0/2706076241 learned_addr learned my addr 172.21.15.46:0/2706076241 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:24:56.120 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:56.115+0000 7f5ac4980640 1 --2- 172.21.15.46:0/2706076241 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5ac0101740 0x7f5ac01983d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:24:56.120 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:56.115+0000 7f5ac5181640 1 --2- 172.21.15.46:0/2706076241 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5ac0100d80 0x7f5ac0197e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:24:56.120 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:56.116+0000 7f5ac4980640 1 -- 172.21.15.46:0/2706076241 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5ac0100d80 msgr2=0x7f5ac0197e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:24:56.120 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:56.116+0000 7f5ac4980640 1 --2- 172.21.15.46:0/2706076241 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5ac0100d80 0x7f5ac0197e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:24:56.120 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:56.116+0000 7f5ac4980640 1 -- 172.21.15.46:0/2706076241 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5ac01055c0 msgr2=0x7f5ac019c700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:24:56.120 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:56.116+0000 7f5ac4980640 1 --2- 172.21.15.46:0/2706076241 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5ac01055c0 0x7f5ac019c700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:24:56.120 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:56.116+0000 7f5ac4980640 1 -- 172.21.15.46:0/2706076241 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ac019ce30 con 0x7f5ac0101740 2024-03-31T23:24:56.120 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:56.116+0000 7f5ac5982640 1 --2- 172.21.15.46:0/2706076241 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5ac01055c0 0x7f5ac019c700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:24:56.121 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:56.116+0000 7f5ac4980640 1 --2- 172.21.15.46:0/2706076241 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5ac0101740 0x7f5ac01983d0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f5ab0036040 tx=0x7f5ab002cd20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:24:56.121 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:56.116+0000 7f5ab67fc640 1 -- 172.21.15.46:0/2706076241 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ab0043020 con 0x7f5ac0101740 2024-03-31T23:24:56.121 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:56.116+0000 7f5ac73e5640 1 -- 172.21.15.46:0/2706076241 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5ac019d0c0 con 0x7f5ac0101740 2024-03-31T23:24:56.121 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:56.116+0000 7f5ab67fc640 1 -- 172.21.15.46:0/2706076241 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5ab0004410 con 0x7f5ac0101740 2024-03-31T23:24:56.121 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:56.116+0000 7f5ab67fc640 1 -- 172.21.15.46:0/2706076241 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ab0003b20 con 0x7f5ac0101740 2024-03-31T23:24:56.121 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:56.116+0000 7f5ac73e5640 1 -- 172.21.15.46:0/2706076241 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5ac010de10 con 0x7f5ac0101740 2024-03-31T23:24:56.126 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:56.119+0000 7f5ab67fc640 1 -- 172.21.15.46:0/2706076241 <== mon.1 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 25) ==== 137451+0+0 (secure 0 0 0) 0x7f5ab0046020 con 0x7f5ac0101740 2024-03-31T23:24:56.126 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:56.120+0000 7f5ac9fc7740 1 Processor -- start 2024-03-31T23:24:56.126 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:56.120+0000 7f5ac9fc7740 1 -- start start 2024-03-31T23:24:56.126 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:56.120+0000 7f5ab67fc640 1 --2- 172.21.15.46:0/2706076241 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f5aac0a2730 0x7f5aac0a4bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:24:56.127 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:56.120+0000 7f5ac9fc7740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5634b6d856c0 0x5634b6d85aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:24:56.127 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:56.120+0000 7f5ac9fc7740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5634b6d85fe0 0x5634b6d8e870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:24:56.127 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:56.120+0000 7f5ac9fc7740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5634b6d8edb0 0x5634b6d911a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:24:56.127 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:56.120+0000 7f5ac9fc7740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x5634b6ce88f0 con 0x5634b6d8edb0 2024-03-31T23:24:56.127 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:56.120+0000 7f5ac9fc7740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x5634b6c918d0 con 0x5634b6d856c0 2024-03-31T23:24:56.127 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:56.120+0000 7f5ac9fc7740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5634b6ce4990 con 0x5634b6d85fe0 2024-03-31T23:24:56.127 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:56.120+0000 7f5ab67fc640 1 -- 172.21.15.46:0/2706076241 <== mon.1 v2:172.21.15.162:3300/0 5 ==== osd_map(110..110 src has 1..110) ==== 9554+0+0 (secure 0 0 0) 0x7f5ab00f0a40 con 0x7f5ac0101740 2024-03-31T23:24:56.127 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:56.120+0000 7f5ac4980640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5634b6d85fe0 0x5634b6d8e870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:24:56.127 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:56.120+0000 7f5ac5181640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5634b6d856c0 0x5634b6d85aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:24:56.127 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:56.120+0000 7f5ac5181640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5634b6d856c0 0x5634b6d85aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.46:47104/0 (socket says 172.21.15.46:47104) 2024-03-31T23:24:56.127 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:56.120+0000 7f5ac4980640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5634b6d85fe0 0x5634b6d8e870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.46:54270/0 (socket says 172.21.15.46:54270) 2024-03-31T23:24:56.127 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:56.120+0000 7f5ac5181640 1 -- 172.21.15.46:0/3893658852 learned_addr learned my addr 172.21.15.46:0/3893658852 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:24:56.127 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:56.120+0000 7f5ac5982640 1 --2- 172.21.15.46:0/2706076241 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f5aac0a2730 0x7f5aac0a4bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:24:56.127 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:56.120+0000 7f5ac4980640 1 -- 172.21.15.46:0/3893658852 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5634b6d856c0 msgr2=0x5634b6d85aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:24:56.128 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:56.120+0000 7f5ac4980640 1 --2- 172.21.15.46:0/3893658852 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5634b6d856c0 0x5634b6d85aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:24:56.128 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:56.120+0000 7f5ac4980640 1 -- 172.21.15.46:0/3893658852 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5634b6d8edb0 msgr2=0x5634b6d911a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:24:56.128 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:56.121+0000 7f5ac5982640 1 --2- 172.21.15.46:0/3893658852 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5634b6d8edb0 0x5634b6d911a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:24:56.128 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:56.121+0000 7f5ac4980640 1 --2- 172.21.15.46:0/3893658852 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5634b6d8edb0 0x5634b6d911a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:24:56.128 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:56.121+0000 7f5ac4980640 1 -- 172.21.15.46:0/3893658852 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5634b69fd210 con 0x5634b6d85fe0 2024-03-31T23:24:56.128 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:56.121+0000 7f5ac5982640 1 --2- 172.21.15.46:0/3893658852 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5634b6d8edb0 0x5634b6d911a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T23:24:56.128 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:56.121+0000 7f5ac4980640 1 --2- 172.21.15.46:0/3893658852 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5634b6d85fe0 0x5634b6d8e870 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f5ab00ede90 tx=0x7f5ab00045c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fe2b8051bb40e48e server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:24:56.128 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:56.121+0000 7f5ac5982640 1 --2- 172.21.15.46:0/2706076241 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f5aac0a2730 0x7f5aac0a4bf0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x5634b6d81400 tx=0x7f5ab800f3c0 comp rx=0 tx=0).ready entity=mgr.15654 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:24:56.128 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:56.121+0000 7f5aab7fe640 1 -- 172.21.15.46:0/3893658852 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ab00d5e60 con 0x5634b6d85fe0 2024-03-31T23:24:56.128 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:56.121+0000 7f5aab7fe640 1 -- 172.21.15.46:0/3893658852 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5ab00bcec0 con 0x5634b6d85fe0 2024-03-31T23:24:56.128 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:56.121+0000 7f5aab7fe640 1 -- 172.21.15.46:0/3893658852 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ab00ee000 con 0x5634b6d85fe0 2024-03-31T23:24:56.128 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:56.122+0000 7f5ac9fc7740 1 -- 172.21.15.46:0/3893658852 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5634b6d85fe0 msgr2=0x5634b6d8e870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:24:56.129 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:56.122+0000 7f5ac9fc7740 1 --2- 172.21.15.46:0/3893658852 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5634b6d85fe0 0x5634b6d8e870 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f5ab00ede90 tx=0x7f5ab00045c0 comp rx=0 tx=0).stop 2024-03-31T23:24:56.129 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:56.122+0000 7f5ac9fc7740 1 -- 172.21.15.46:0/3893658852 shutdown_connections 2024-03-31T23:24:56.129 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:56.122+0000 7f5ac9fc7740 1 --2- 172.21.15.46:0/3893658852 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5634b6d8edb0 0x5634b6d911a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:24:56.129 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:56.122+0000 7f5ac9fc7740 1 --2- 172.21.15.46:0/3893658852 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5634b6d85fe0 0x5634b6d8e870 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:24:56.129 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:56.122+0000 7f5ac9fc7740 1 --2- 172.21.15.46:0/3893658852 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5634b6d856c0 0x5634b6d85aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:24:56.129 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:56.122+0000 7f5ac9fc7740 1 -- 172.21.15.46:0/3893658852 >> 172.21.15.46:0/3893658852 conn(0x5634b6d7ffe0 msgr2=0x5634b6d803c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:24:56.129 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:56.122+0000 7f5ac9fc7740 1 -- 172.21.15.46:0/3893658852 shutdown_connections 2024-03-31T23:24:56.129 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:56.122+0000 7f5ac9fc7740 1 -- 172.21.15.46:0/3893658852 wait complete. 2024-03-31T23:24:56.129 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:56.122+0000 7f5ac9fc7740 1 -- start start 2024-03-31T23:24:56.129 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:56.122+0000 7f5ac9fc7740 1 Processor -- start 2024-03-31T23:24:56.129 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:56.123+0000 7f5ac9fc7740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5634b6d856c0 0x5634b6d8bea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:24:56.129 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:56.123+0000 7f5ac9fc7740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5634b6d8edb0 0x5634b6d8c3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:24:56.129 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:56.123+0000 7f5ac9fc7740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5634b6d8c920 0x5634b6d9a9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:24:56.129 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:56.123+0000 7f5ac9fc7740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x5634b6ce4990 con 0x5634b6d8edb0 2024-03-31T23:24:56.129 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:56.123+0000 7f5ac9fc7740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x5634b6c86f80 con 0x5634b6d8c920 2024-03-31T23:24:56.129 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:56.123+0000 7f5ac9fc7740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5634b6d917a0 con 0x5634b6d856c0 2024-03-31T23:24:56.130 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:56.123+0000 7f5ac9fc7740 10 client.0 _ll_register_callbacks cb 0x5634b6cdf290 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T23:24:56.130 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:56.123+0000 7f5ac4980640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5634b6d8edb0 0x5634b6d8c3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:24:56.130 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:56.123+0000 7f5ac4980640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5634b6d8edb0 0x5634b6d8c3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:36910/0 (socket says 172.21.15.46:36910) 2024-03-31T23:24:56.130 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:56.123+0000 7f5ac4980640 1 -- 172.21.15.46:0/1455577055 learned_addr learned my addr 172.21.15.46:0/1455577055 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:24:56.130 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:56.123+0000 7f5ac5181640 1 --2- 172.21.15.46:0/1455577055 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5634b6d856c0 0x5634b6d8bea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:24:56.130 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:56.123+0000 7f5ac5982640 1 --2- 172.21.15.46:0/1455577055 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5634b6d8c920 0x5634b6d9a9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:24:56.130 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:56.123+0000 7f5ac4980640 1 -- 172.21.15.46:0/1455577055 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5634b6d856c0 msgr2=0x5634b6d8bea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:24:56.130 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:56.123+0000 7f5ac4980640 1 --2- 172.21.15.46:0/1455577055 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5634b6d856c0 0x5634b6d8bea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:24:56.130 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:56.123+0000 7f5ac4980640 1 -- 172.21.15.46:0/1455577055 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5634b6d8c920 msgr2=0x5634b6d9a9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:24:56.130 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:56.123+0000 7f5ac4980640 1 --2- 172.21.15.46:0/1455577055 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5634b6d8c920 0x5634b6d9a9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:24:56.130 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:56.123+0000 7f5ac4980640 1 -- 172.21.15.46:0/1455577055 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5634b6ac9620 con 0x5634b6d8edb0 2024-03-31T23:24:56.130 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:56.123+0000 7f5ac5181640 1 --2- 172.21.15.46:0/1455577055 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5634b6d856c0 0x5634b6d8bea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:24:56.130 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:56.123+0000 7f5ac4980640 1 --2- 172.21.15.46:0/1455577055 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5634b6d8edb0 0x5634b6d8c3e0 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7f5ab00c53a0 tx=0x7f5ab002bc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=28fae7e7bd332cc0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:24:56.130 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:56.123+0000 7f5aa8ff9640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-31T23:24:56.131 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:56.124+0000 7f5aa8ff9640 1 -- 172.21.15.46:0/1455577055 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ab00d5e60 con 0x5634b6d8edb0 2024-03-31T23:24:56.131 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:56.124+0000 7f5aa8ff9640 1 -- 172.21.15.46:0/1455577055 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(110..110 src has 1..110) ==== 9554+0+0 (secure 0 0 0) 0x7f5ab00054d0 con 0x5634b6d8edb0 2024-03-31T23:24:56.131 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:56.124+0000 7f5ac9fc7740 1 -- 172.21.15.46:0/1455577055 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5ab00bcec0 con 0x5634b6d8edb0 2024-03-31T23:24:56.131 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:56.124+0000 7f5aa8ff9640 1 -- 172.21.15.46:0/1455577055 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f5ab0003cc0 con 0x5634b6d8edb0 2024-03-31T23:24:56.131 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:56.124+0000 7f5ac9fc7740 10 client.17223 fetch_fsmap learned FSMap version 36 2024-03-31T23:24:56.131 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:56.124+0000 7f5ac9fc7740 1 -- 172.21.15.46:0/1455577055 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x5634b6d9d930 con 0x5634b6d8edb0 2024-03-31T23:24:56.131 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:56.124+0000 7f5aa8ff9640 1 -- 172.21.15.46:0/1455577055 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5ab00bcec0 con 0x5634b6d8edb0 2024-03-31T23:24:56.131 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:56.124+0000 7f5aa8ff9640 1 -- 172.21.15.46:0/1455577055 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 36) ==== 8362+0+0 (secure 0 0 0) 0x7f5ab009bce0 con 0x5634b6d8edb0 2024-03-31T23:24:56.131 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:56.124+0000 7f5ac9fc7740 10 client.17223 fetch_fsmap finished waiting for FSMap version 36 2024-03-31T23:24:56.131 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:56.124+0000 7f5ac9fc7740 10 client.17223 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-31T23:24:56.131 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:56.124+0000 7f5ac9fc7740 20 client.17223 populate_metadata read hostname 'smithi046' 2024-03-31T23:24:56.131 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:56.124+0000 7f5ac9fc7740 1 --2- 172.21.15.46:0/1455577055 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x5634b6d9eab0 0x5634b6da0f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:24:56.131 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:56.124+0000 7f5ac9fc7740 4 client.17223 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-31T23:24:56.132 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:56.124+0000 7f5ac9fc7740 1 -- 172.21.15.46:0/1455577055 --> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5634b6cfc120 con 0x5634b6d9eab0 2024-03-31T23:24:56.132 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:56.125+0000 7f5ac5181640 1 --2- 172.21.15.46:0/1455577055 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x5634b6d9eab0 0x5634b6da0f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:24:56.132 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:56.125+0000 7f5ac5181640 1 --2- 172.21.15.46:0/1455577055 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x5634b6d9eab0 0x5634b6da0f00 crc :-1 s=READY pgs=193 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9110d22e0a6caba4 server_cookie=4d7e082ae49db384 in_seq=0 out_seq=0 2024-03-31T23:24:56.132 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:56.125+0000 7f5aa8ff9640 10 client.17223 ms_handle_connect on v2:172.21.15.142:6838/2519143918 2024-03-31T23:24:56.133 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:56.126+0000 7f5aa8ff9640 1 -- 172.21.15.46:0/1455577055 <== mds.0 v2:172.21.15.142:6838/2519143918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5634b6cfc120 con 0x5634b6d9eab0 2024-03-31T23:24:56.133 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:56.126+0000 7f5aa8ff9640 10 client.17223 handle_command_reply: tid=0 2024-03-31T23:24:56.133 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:56.127+0000 7f5ac9fc7740 1 client.17223 shutdown 2024-03-31T23:24:56.133 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:56.127+0000 7f5ac9fc7740 1 -- 172.21.15.46:0/1455577055 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5634b6d8edb0 msgr2=0x5634b6d8c3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:24:56.133 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:56.127+0000 7f5ac9fc7740 1 --2- 172.21.15.46:0/1455577055 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5634b6d8edb0 0x5634b6d8c3e0 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7f5ab00c53a0 tx=0x7f5ab002bc30 comp rx=0 tx=0).stop 2024-03-31T23:24:56.133 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:56.127+0000 7f5ac9fc7740 1 -- 172.21.15.46:0/1455577055 shutdown_connections 2024-03-31T23:24:56.133 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:56.127+0000 7f5ac9fc7740 1 --2- 172.21.15.46:0/1455577055 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x5634b6d9eab0 0x5634b6da0f00 crc :-1 s=READY pgs=193 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:24:56.133 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:56.127+0000 7f5ac9fc7740 1 --2- 172.21.15.46:0/1455577055 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5634b6d8c920 0x5634b6d9a9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:24:56.133 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:56.127+0000 7f5ac9fc7740 1 --2- 172.21.15.46:0/1455577055 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5634b6d8edb0 0x5634b6d8c3e0 unknown :-1 s=CLOSED pgs=872 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:24:56.134 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:56.127+0000 7f5aa8ff9640 0 client.17223 ms_handle_reset on v2:172.21.15.142:6838/2519143918 2024-03-31T23:24:56.134 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:56.127+0000 7f5ac9fc7740 1 --2- 172.21.15.46:0/1455577055 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5634b6d856c0 0x5634b6d8bea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:24:56.134 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:56.127+0000 7f5ac9fc7740 1 -- 172.21.15.46:0/1455577055 >> 172.21.15.46:0/1455577055 conn(0x5634b6d7ffe0 msgr2=0x5634b6d81d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:24:56.134 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:56.127+0000 7f5ac9fc7740 1 -- 172.21.15.46:0/1455577055 shutdown_connections 2024-03-31T23:24:56.134 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:56.127+0000 7f5ac9fc7740 1 -- 172.21.15.46:0/1455577055 wait complete. 2024-03-31T23:24:56.134 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:56.127+0000 7f5ac9fc7740 20 client.17223 trim_cache size 0 max 16384 2024-03-31T23:24:56.151 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:56.148+0000 7f5ac9fc7740 1 Processor -- start 2024-03-31T23:24:56.151 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:56.148+0000 7f5ac9fc7740 1 -- start start 2024-03-31T23:24:56.151 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:56.148+0000 7f5ac9fc7740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5634b6d856c0 0x5634b6da6820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:24:56.151 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:56.148+0000 7f5ac9fc7740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5634b6d8c920 0x5634b6d8a770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:24:56.151 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:56.148+0000 7f5ac9fc7740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5634b6d8edb0 0x5634b6d8acb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:24:56.151 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:56.148+0000 7f5ac9fc7740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f5ac01af360 con 0x5634b6d8c920 2024-03-31T23:24:56.151 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:56.148+0000 7f5ac9fc7740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f5ac01acbd0 con 0x5634b6d8edb0 2024-03-31T23:24:56.151 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:56.148+0000 7f5ac9fc7740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f5ac01aaa40 con 0x5634b6d856c0 2024-03-31T23:24:56.151 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:56.148+0000 7f5ac4980640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5634b6d8c920 0x5634b6d8a770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:24:56.151 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:56.148+0000 7f5ac4980640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5634b6d8c920 0x5634b6d8a770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:36912/0 (socket says 172.21.15.46:36912) 2024-03-31T23:24:56.151 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:56.148+0000 7f5ac4980640 1 -- 172.21.15.46:0/1792267066 learned_addr learned my addr 172.21.15.46:0/1792267066 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:24:56.151 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:56.148+0000 7f5ac5181640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5634b6d856c0 0x5634b6da6820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:24:56.152 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:56.148+0000 7f5ac5982640 1 --2- 172.21.15.46:0/1792267066 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5634b6d8edb0 0x5634b6d8acb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:24:56.152 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:56.149+0000 7f5ac4980640 1 -- 172.21.15.46:0/1792267066 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5634b6d856c0 msgr2=0x5634b6da6820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:24:56.152 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:56.149+0000 7f5ac4980640 1 --2- 172.21.15.46:0/1792267066 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5634b6d856c0 0x5634b6da6820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:24:56.152 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:56.149+0000 7f5ac4980640 1 -- 172.21.15.46:0/1792267066 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5634b6d8edb0 msgr2=0x5634b6d8acb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:24:56.152 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:56.149+0000 7f5ac4980640 1 --2- 172.21.15.46:0/1792267066 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5634b6d8edb0 0x5634b6d8acb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:24:56.152 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:56.149+0000 7f5ac4980640 1 -- 172.21.15.46:0/1792267066 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5634b6d80bb0 con 0x5634b6d8c920 2024-03-31T23:24:56.152 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:56.149+0000 7f5ac5181640 1 --2- 172.21.15.46:0/1792267066 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5634b6d856c0 0x5634b6da6820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:24:56.152 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:56.149+0000 7f5ac4980640 1 --2- 172.21.15.46:0/1792267066 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5634b6d8c920 0x5634b6d8a770 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7f5ab00b9890 tx=0x7f5ab00045c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d3ed3f26a9e85e80 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:24:56.152 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:56.149+0000 7f5aaaffd640 1 -- 172.21.15.46:0/1792267066 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ab00d5e60 con 0x5634b6d8c920 2024-03-31T23:24:56.152 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:56.149+0000 7f5aaaffd640 1 -- 172.21.15.46:0/1792267066 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5ab00d2020 con 0x5634b6d8c920 2024-03-31T23:24:56.152 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:56.149+0000 7f5aaaffd640 1 -- 172.21.15.46:0/1792267066 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ab00bc5f0 con 0x5634b6d8c920 2024-03-31T23:24:56.155 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:56.150+0000 7f5ac9fc7740 1 -- 172.21.15.46:0/1792267066 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5634b6d8c920 msgr2=0x5634b6d8a770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:24:56.155 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:56.150+0000 7f5ac9fc7740 1 --2- 172.21.15.46:0/1792267066 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5634b6d8c920 0x5634b6d8a770 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7f5ab00b9890 tx=0x7f5ab00045c0 comp rx=0 tx=0).stop 2024-03-31T23:24:56.155 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:56.150+0000 7f5ac9fc7740 1 -- 172.21.15.46:0/1792267066 shutdown_connections 2024-03-31T23:24:56.155 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:56.150+0000 7f5ac9fc7740 1 --2- 172.21.15.46:0/1792267066 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5634b6d8edb0 0x5634b6d8acb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:24:56.155 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:56.150+0000 7f5ac9fc7740 1 --2- 172.21.15.46:0/1792267066 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5634b6d8c920 0x5634b6d8a770 unknown :-1 s=CLOSED pgs=873 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:24:56.155 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:56.150+0000 7f5ac9fc7740 1 --2- 172.21.15.46:0/1792267066 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5634b6d856c0 0x5634b6da6820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:24:56.155 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:56.150+0000 7f5ac9fc7740 1 -- 172.21.15.46:0/1792267066 >> 172.21.15.46:0/1792267066 conn(0x5634b6d7ffe0 msgr2=0x5634b6d803c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:24:56.156 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:56.151+0000 7f5ac9fc7740 1 -- 172.21.15.46:0/1792267066 shutdown_connections 2024-03-31T23:24:56.156 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:56.151+0000 7f5ac9fc7740 1 -- 172.21.15.46:0/1792267066 wait complete. 2024-03-31T23:24:56.156 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:56.151+0000 7f5ac9fc7740 1 -- start start 2024-03-31T23:24:56.156 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:56.151+0000 7f5ac9fc7740 1 Processor -- start 2024-03-31T23:24:56.156 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:56.152+0000 7f5ac9fc7740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5634b6d856c0 0x5634b6d96a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:24:56.156 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:56.152+0000 7f5ac9fc7740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5634b6d8c920 0x5634b6d886f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:24:56.156 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:56.152+0000 7f5ac9fc7740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5634b6d8edb0 0x5634b6d88c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:24:56.156 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:56.152+0000 7f5ac5181640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5634b6d856c0 0x5634b6d96a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:24:56.156 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:56.152+0000 7f5ac5181640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5634b6d856c0 0x5634b6d96a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.46:54310/0 (socket says 172.21.15.46:54310) 2024-03-31T23:24:56.156 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:56.152+0000 7f5ac4980640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5634b6d8c920 0x5634b6d886f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:24:56.156 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:56.152+0000 7f5ac5181640 1 -- 172.21.15.46:0/614185421 learned_addr learned my addr 172.21.15.46:0/614185421 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:24:56.156 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:56.152+0000 7f5ac9fc7740 1 -- 172.21.15.46:0/614185421 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f5ac01ab090 con 0x5634b6d8c920 2024-03-31T23:24:56.156 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:56.152+0000 7f5ac9fc7740 1 -- 172.21.15.46:0/614185421 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f5ac01ac500 con 0x5634b6d8edb0 2024-03-31T23:24:56.157 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:56.152+0000 7f5ac9fc7740 1 -- 172.21.15.46:0/614185421 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f5ac01ab760 con 0x5634b6d856c0 2024-03-31T23:24:56.157 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:56.152+0000 7f5ac5982640 1 --2- 172.21.15.46:0/614185421 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5634b6d8edb0 0x5634b6d88c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:24:56.157 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:56.152+0000 7f5ac9fc7740 10 client.0 _ll_register_callbacks cb 0x7f5ab00c58d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T23:24:56.157 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:56.153+0000 7f5ac5181640 1 -- 172.21.15.46:0/614185421 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5634b6d8edb0 msgr2=0x5634b6d88c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:24:56.157 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:56.153+0000 7f5ac5181640 1 --2- 172.21.15.46:0/614185421 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5634b6d8edb0 0x5634b6d88c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:24:56.157 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:56.153+0000 7f5ac5181640 1 -- 172.21.15.46:0/614185421 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5634b6d8c920 msgr2=0x5634b6d886f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:24:56.157 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:56.153+0000 7f5ac5181640 1 --2- 172.21.15.46:0/614185421 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5634b6d8c920 0x5634b6d886f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:24:56.157 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:56.153+0000 7f5ac5181640 1 -- 172.21.15.46:0/614185421 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5634b6d89220 con 0x5634b6d856c0 2024-03-31T23:24:56.157 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:56.153+0000 7f5ac4980640 1 --2- 172.21.15.46:0/614185421 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5634b6d8c920 0x5634b6d886f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:24:56.157 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:56.154+0000 7f5ac5181640 1 --2- 172.21.15.46:0/614185421 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5634b6d856c0 0x5634b6d96a50 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f5abc004700 tx=0x7f5abc009d40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5685822ad43500c7 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:24:56.157 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:56.154+0000 7f5aa8ff9640 10 client.0 ms_handle_connect on v2:172.21.15.162:3300/0 2024-03-31T23:24:56.157 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:56.154+0000 7f5aa8ff9640 1 -- 172.21.15.46:0/614185421 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5abc0044e0 con 0x5634b6d856c0 2024-03-31T23:24:56.157 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:56.154+0000 7f5aa8ff9640 1 -- 172.21.15.46:0/614185421 <== mon.1 v2:172.21.15.162:3300/0 2 ==== osd_map(110..110 src has 1..110) ==== 9554+0+0 (secure 0 0 0) 0x7f5abc01c3f0 con 0x5634b6d856c0 2024-03-31T23:24:56.157 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:56.154+0000 7f5ac9fc7740 1 -- 172.21.15.46:0/614185421 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5ab00d2020 con 0x5634b6d856c0 2024-03-31T23:24:56.157 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:56.154+0000 7f5aa8ff9640 1 -- 172.21.15.46:0/614185421 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f5abc005020 con 0x5634b6d856c0 2024-03-31T23:24:56.158 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:56.154+0000 7f5ac9fc7740 10 client.26962 fetch_fsmap learned FSMap version 36 2024-03-31T23:24:56.158 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:56.154+0000 7f5ac9fc7740 1 -- 172.21.15.46:0/614185421 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x5634b6e04c80 con 0x5634b6d856c0 2024-03-31T23:24:56.158 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:56.154+0000 7f5aa8ff9640 1 -- 172.21.15.46:0/614185421 <== mon.1 v2:172.21.15.162:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5ab00d2020 con 0x5634b6d856c0 2024-03-31T23:24:56.159 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:56.155+0000 7f5aa8ff9640 1 -- 172.21.15.46:0/614185421 <== mon.1 v2:172.21.15.162:3300/0 5 ==== fsmap(e 36) ==== 8362+0+0 (secure 0 0 0) 0x7f5abc004cd0 con 0x5634b6d856c0 2024-03-31T23:24:56.159 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:56.155+0000 7f5ac9fc7740 10 client.26962 fetch_fsmap finished waiting for FSMap version 36 2024-03-31T23:24:56.159 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:56.155+0000 7f5ac9fc7740 10 client.26962 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-31T23:24:56.159 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:56.155+0000 7f5ac9fc7740 20 client.26962 populate_metadata read hostname 'smithi046' 2024-03-31T23:24:56.159 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:56.155+0000 7f5ac9fc7740 1 --2- 172.21.15.46:0/614185421 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x5634b6e05400 0x5634b6e07820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:24:56.159 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:56.155+0000 7f5ac9fc7740 4 client.26962 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-31T23:24:56.159 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:56.155+0000 7f5ac9fc7740 1 -- 172.21.15.46:0/614185421 --> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5634b6cf5e30 con 0x5634b6e05400 2024-03-31T23:24:56.159 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:56.155+0000 7f5ac4980640 1 --2- 172.21.15.46:0/614185421 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x5634b6e05400 0x5634b6e07820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:24:56.159 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:56.155+0000 7f5ac4980640 1 --2- 172.21.15.46:0/614185421 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x5634b6e05400 0x5634b6e07820 crc :-1 s=READY pgs=194 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4490a425d5cff044 server_cookie=d00be31538a6c8f6 in_seq=0 out_seq=0 2024-03-31T23:24:56.159 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:56.155+0000 7f5aa8ff9640 10 client.26962 ms_handle_connect on v2:172.21.15.142:6838/2519143918 2024-03-31T23:24:56.265 INFO:tasks.workunit.client.0.smithi046.stdout: 10 202 80.00 MB/sec warmup 1 sec latency 133.478 ms 2024-03-31T23:24:56.338 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:56.336+0000 7f5aa8ff9640 1 -- 172.21.15.46:0/614185421 <== mds.0 v2:172.21.15.142:6838/2519143918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5634b6cf5e30 con 0x5634b6e05400 2024-03-31T23:24:56.338 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:56.336+0000 7f5aa8ff9640 10 client.26962 handle_command_reply: tid=0 2024-03-31T23:24:56.338 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:56.336+0000 7f5ac9fc7740 1 client.26962 shutdown 2024-03-31T23:24:56.338 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:56.336+0000 7f5ac9fc7740 1 -- 172.21.15.46:0/614185421 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5634b6d856c0 msgr2=0x5634b6d96a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:24:56.338 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:56.336+0000 7f5ac9fc7740 1 --2- 172.21.15.46:0/614185421 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5634b6d856c0 0x5634b6d96a50 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f5abc004700 tx=0x7f5abc009d40 comp rx=0 tx=0).stop 2024-03-31T23:24:56.338 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:56.336+0000 7f5ac9fc7740 1 -- 172.21.15.46:0/614185421 shutdown_connections 2024-03-31T23:24:56.338 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:56.336+0000 7f5ac9fc7740 1 --2- 172.21.15.46:0/614185421 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x5634b6e05400 0x5634b6e07820 crc :-1 s=READY pgs=194 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:24:56.338 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:56.336+0000 7f5aa8ff9640 0 client.26962 ms_handle_reset on v2:172.21.15.142:6838/2519143918 2024-03-31T23:24:56.338 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:56.336+0000 7f5ac9fc7740 1 --2- 172.21.15.46:0/614185421 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5634b6d8edb0 0x5634b6d88c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:24:56.339 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:56.336+0000 7f5ac9fc7740 1 --2- 172.21.15.46:0/614185421 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5634b6d8c920 0x5634b6d886f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:24:56.339 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:56.336+0000 7f5ac9fc7740 1 --2- 172.21.15.46:0/614185421 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5634b6d856c0 0x5634b6d96a50 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:24:56.339 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:56.336+0000 7f5ac9fc7740 1 -- 172.21.15.46:0/614185421 >> 172.21.15.46:0/614185421 conn(0x5634b6d7ffe0 msgr2=0x5634b6d8a6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:24:56.339 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:56.336+0000 7f5ac9fc7740 1 -- 172.21.15.46:0/614185421 shutdown_connections 2024-03-31T23:24:56.339 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:56.336+0000 7f5ac9fc7740 1 -- 172.21.15.46:0/614185421 wait complete. 2024-03-31T23:24:56.339 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:56.337+0000 7f5ac9fc7740 20 client.26962 trim_cache size 0 max 16384 2024-03-31T23:24:56.339 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout:{ 2024-03-31T23:24:56.339 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout: "message": "", 2024-03-31T23:24:56.339 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout: "return_code": 0 2024-03-31T23:24:56.339 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout:} 2024-03-31T23:24:56.340 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:56.337+0000 7f5ac73e5640 1 -- 172.21.15.46:0/2706076241 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f5aac0a2730 msgr2=0x7f5aac0a4bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:24:56.340 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:56.337+0000 7f5ac73e5640 1 --2- 172.21.15.46:0/2706076241 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f5aac0a2730 0x7f5aac0a4bf0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x5634b6d81400 tx=0x7f5ab800f3c0 comp rx=0 tx=0).stop 2024-03-31T23:24:56.340 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:56.337+0000 7f5ac73e5640 1 -- 172.21.15.46:0/2706076241 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5ac0101740 msgr2=0x7f5ac01983d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:24:56.340 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:56.337+0000 7f5ac73e5640 1 --2- 172.21.15.46:0/2706076241 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5ac0101740 0x7f5ac01983d0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f5ab0036040 tx=0x7f5ab002cd20 comp rx=0 tx=0).stop 2024-03-31T23:24:56.340 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:56.338+0000 7f5ac73e5640 1 -- 172.21.15.46:0/2706076241 shutdown_connections 2024-03-31T23:24:56.340 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:56.338+0000 7f5ac73e5640 1 --2- 172.21.15.46:0/2706076241 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f5aac0a2730 0x7f5aac0a4bf0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:24:56.340 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:56.338+0000 7f5ac73e5640 1 --2- 172.21.15.46:0/2706076241 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5ac01055c0 0x7f5ac019c700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:24:56.340 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:56.338+0000 7f5ac73e5640 1 --2- 172.21.15.46:0/2706076241 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5ac0101740 0x7f5ac01983d0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:24:56.341 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:56.338+0000 7f5ac73e5640 1 --2- 172.21.15.46:0/2706076241 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5ac0100d80 0x7f5ac0197e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:24:56.341 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:56.338+0000 7f5ac73e5640 1 -- 172.21.15.46:0/2706076241 >> 172.21.15.46:0/2706076241 conn(0x7f5ac0076be0 msgr2=0x7f5ac01033c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:24:56.341 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:56.339+0000 7f5ac73e5640 1 -- 172.21.15.46:0/2706076241 shutdown_connections 2024-03-31T23:24:56.341 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:24:56.339+0000 7f5ac73e5640 1 -- 172.21.15.46:0/2706076241 wait complete. 2024-03-31T23:24:57.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:24:56 smithi162 ceph-mon[36237]: pgmap v413: 129 pgs: 129 active+clean; 149 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 396 KiB/s rd, 1.0 MiB/s wr, 103 op/s 2024-03-31T23:24:57.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:24:56 smithi046 ceph-mon[32498]: pgmap v413: 129 pgs: 129 active+clean; 149 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 396 KiB/s rd, 1.0 MiB/s wr, 103 op/s 2024-03-31T23:24:57.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:24:56 smithi142 ceph-mon[35271]: pgmap v413: 129 pgs: 129 active+clean; 149 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 396 KiB/s rd, 1.0 MiB/s wr, 103 op/s 2024-03-31T23:24:57.265 INFO:tasks.workunit.client.0.smithi046.stdout: 10 340 61.46 MB/sec warmup 2 sec latency 195.562 ms 2024-03-31T23:24:58.265 INFO:tasks.workunit.client.0.smithi046.stdout: 10 490 57.12 MB/sec warmup 3 sec latency 120.168 ms 2024-03-31T23:24:59.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:24:58 smithi162 ceph-mon[36237]: pgmap v414: 129 pgs: 129 active+clean; 149 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 165 KiB/s rd, 507 KiB/s wr, 44 op/s 2024-03-31T23:24:59.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:24:58 smithi046 ceph-mon[32498]: pgmap v414: 129 pgs: 129 active+clean; 149 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 165 KiB/s rd, 507 KiB/s wr, 44 op/s 2024-03-31T23:24:59.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:24:58 smithi142 ceph-mon[35271]: pgmap v414: 129 pgs: 129 active+clean; 149 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 165 KiB/s rd, 507 KiB/s wr, 44 op/s 2024-03-31T23:24:59.265 INFO:tasks.workunit.client.0.smithi046.stdout: 10 636 51.47 MB/sec warmup 4 sec latency 154.609 ms 2024-03-31T23:25:00.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:24:59 smithi162 ceph-mon[36237]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-03-31T23:25:00.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:24:59 smithi046 ceph-mon[32498]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-03-31T23:25:00.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:24:59 smithi142 ceph-mon[35271]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-03-31T23:25:00.265 INFO:tasks.workunit.client.0.smithi046.stdout: 10 751 42.66 MB/sec warmup 5 sec latency 161.060 ms 2024-03-31T23:25:01.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:25:00 smithi162 ceph-mon[36237]: pgmap v415: 129 pgs: 129 active+clean; 339 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 492 KiB/s rd, 17 MiB/s wr, 200 op/s 2024-03-31T23:25:01.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:25:00 smithi046 ceph-mon[32498]: pgmap v415: 129 pgs: 129 active+clean; 339 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 492 KiB/s rd, 17 MiB/s wr, 200 op/s 2024-03-31T23:25:01.194 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:25:00 smithi142 ceph-mon[35271]: pgmap v415: 129 pgs: 129 active+clean; 339 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 492 KiB/s rd, 17 MiB/s wr, 200 op/s 2024-03-31T23:25:01.265 INFO:tasks.workunit.client.0.smithi046.stdout: 10 812 35.78 MB/sec warmup 6 sec latency 437.989 ms 2024-03-31T23:25:02.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:25:01 smithi162 ceph-mon[36237]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-31T23:25:02.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:25:01 smithi046 ceph-mon[32498]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-31T23:25:02.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:25:01 smithi142 ceph-mon[35271]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-31T23:25:02.265 INFO:tasks.workunit.client.0.smithi046.stdout: 10 905 31.02 MB/sec warmup 7 sec latency 200.427 ms 2024-03-31T23:25:03.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:25:02 smithi162 ceph-mon[36237]: pgmap v416: 129 pgs: 129 active+clean; 339 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 390 KiB/s rd, 17 MiB/s wr, 173 op/s 2024-03-31T23:25:03.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:25:02 smithi162 ceph-mon[36237]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:25:03.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:25:02 smithi162 ceph-mon[36237]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-31T23:25:03.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:25:02 smithi162 ceph-mon[36237]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' 2024-03-31T23:25:03.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:25:02 smithi046 ceph-mon[32498]: pgmap v416: 129 pgs: 129 active+clean; 339 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 390 KiB/s rd, 17 MiB/s wr, 173 op/s 2024-03-31T23:25:03.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:25:02 smithi046 ceph-mon[32498]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:25:03.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:25:02 smithi046 ceph-mon[32498]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-31T23:25:03.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:25:02 smithi046 ceph-mon[32498]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' 2024-03-31T23:25:03.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:25:02 smithi142 ceph-mon[35271]: pgmap v416: 129 pgs: 129 active+clean; 339 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 390 KiB/s rd, 17 MiB/s wr, 173 op/s 2024-03-31T23:25:03.194 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:25:02 smithi142 ceph-mon[35271]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:25:03.194 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:25:02 smithi142 ceph-mon[35271]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-31T23:25:03.194 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:25:02 smithi142 ceph-mon[35271]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' 2024-03-31T23:25:03.265 INFO:tasks.workunit.client.0.smithi046.stdout: 10 960 27.27 MB/sec warmup 8 sec latency 1090.903 ms 2024-03-31T23:25:04.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:25:03 smithi162 ceph-mon[36237]: osdmap e111: 12 total, 12 up, 12 in 2024-03-31T23:25:04.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:25:03 smithi162 ceph-mon[36237]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-03-31T23:25:04.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:25:03 smithi046 ceph-mon[32498]: osdmap e111: 12 total, 12 up, 12 in 2024-03-31T23:25:04.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:25:03 smithi046 ceph-mon[32498]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-03-31T23:25:04.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:25:03 smithi142 ceph-mon[35271]: osdmap e111: 12 total, 12 up, 12 in 2024-03-31T23:25:04.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:25:03 smithi142 ceph-mon[35271]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-03-31T23:25:04.265 INFO:tasks.workunit.client.0.smithi046.stdout: 10 1069 25.24 MB/sec warmup 9 sec latency 2090.931 ms 2024-03-31T23:25:05.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:25:04 smithi162 ceph-mon[36237]: pgmap v418: 129 pgs: 129 active+clean; 339 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 469 KiB/s rd, 20 MiB/s wr, 208 op/s 2024-03-31T23:25:05.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:25:04 smithi046 ceph-mon[32498]: pgmap v418: 129 pgs: 129 active+clean; 339 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 469 KiB/s rd, 20 MiB/s wr, 208 op/s 2024-03-31T23:25:05.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:25:04 smithi142 ceph-mon[35271]: pgmap v418: 129 pgs: 129 active+clean; 339 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 469 KiB/s rd, 20 MiB/s wr, 208 op/s 2024-03-31T23:25:05.265 INFO:tasks.workunit.client.0.smithi046.stdout: 10 1176 23.09 MB/sec warmup 10 sec latency 3090.960 ms 2024-03-31T23:25:06.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:25:05 smithi162 ceph-mon[36237]: pgmap v419: 129 pgs: 129 active+clean; 355 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 22 MiB/s wr, 294 op/s 2024-03-31T23:25:06.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:25:05 smithi046 ceph-mon[32498]: pgmap v419: 129 pgs: 129 active+clean; 355 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 22 MiB/s wr, 294 op/s 2024-03-31T23:25:06.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:25:05 smithi142 ceph-mon[35271]: pgmap v419: 129 pgs: 129 active+clean; 355 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 22 MiB/s wr, 294 op/s 2024-03-31T23:25:06.265 INFO:tasks.workunit.client.0.smithi046.stdout: 10 1280 21.34 MB/sec warmup 11 sec latency 4090.987 ms 2024-03-31T23:25:07.265 INFO:tasks.workunit.client.0.smithi046.stdout: 10 1431 21.05 MB/sec warmup 12 sec latency 5091.016 ms 2024-03-31T23:25:08.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:25:07 smithi142 ceph-mon[35271]: pgmap v420: 129 pgs: 129 active+clean; 355 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 22 MiB/s wr, 294 op/s 2024-03-31T23:25:08.265 INFO:tasks.workunit.client.0.smithi046.stdout: 10 1542 19.74 MB/sec warmup 13 sec latency 5643.210 ms 2024-03-31T23:25:08.296 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:25:07 smithi162 ceph-mon[36237]: pgmap v420: 129 pgs: 129 active+clean; 355 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 22 MiB/s wr, 294 op/s 2024-03-31T23:25:08.361 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:25:07 smithi046 ceph-mon[32498]: pgmap v420: 129 pgs: 129 active+clean; 355 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 22 MiB/s wr, 294 op/s 2024-03-31T23:25:09.266 INFO:tasks.workunit.client.0.smithi046.stdout: 10 1701 18.48 MB/sec warmup 14 sec latency 1623.836 ms 2024-03-31T23:25:09.454 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:09.452+0000 7f47d496c640 1 -- 172.21.15.46:0/3638685370 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f47d0101720 msgr2=0x7f47d0101ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:25:09.454 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:09.452+0000 7f47d496c640 1 --2- 172.21.15.46:0/3638685370 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f47d0101720 0x7f47d0101ba0 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7f47c4009930 tx=0x7f47c402bc30 comp rx=0 tx=0).stop 2024-03-31T23:25:09.454 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:09.453+0000 7f47d496c640 1 -- 172.21.15.46:0/3638685370 shutdown_connections 2024-03-31T23:25:09.454 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:09.453+0000 7f47d496c640 1 --2- 172.21.15.46:0/3638685370 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f47d01020e0 0x7f47d010e5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:25:09.454 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:09.453+0000 7f47d496c640 1 --2- 172.21.15.46:0/3638685370 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f47d0101720 0x7f47d0101ba0 unknown :-1 s=CLOSED pgs=874 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:25:09.454 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:09.453+0000 7f47d496c640 1 --2- 172.21.15.46:0/3638685370 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f47d0107730 0x7f47d0107b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:25:09.454 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:09.453+0000 7f47d496c640 1 -- 172.21.15.46:0/3638685370 >> 172.21.15.46:0/3638685370 conn(0x7f47d00fd250 msgr2=0x7f47d00ff670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:25:09.455 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:09.453+0000 7f47d496c640 1 -- 172.21.15.46:0/3638685370 shutdown_connections 2024-03-31T23:25:09.455 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:09.453+0000 7f47d496c640 1 -- 172.21.15.46:0/3638685370 wait complete. 2024-03-31T23:25:09.457 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:09.453+0000 7f47d496c640 1 Processor -- start 2024-03-31T23:25:09.457 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:09.454+0000 7f47d496c640 1 -- start start 2024-03-31T23:25:09.457 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:09.454+0000 7f47d496c640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f47d0101720 0x7f47d0197ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:25:09.457 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:09.454+0000 7f47d496c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f47d01020e0 0x7f47d0198410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:25:09.457 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:09.454+0000 7f47d496c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f47d0107730 0x7f47d019c790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:25:09.457 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:09.454+0000 7f47d496c640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f47d01119f0 con 0x7f47d01020e0 2024-03-31T23:25:09.457 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:09.454+0000 7f47d496c640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f47d0111870 con 0x7f47d0101720 2024-03-31T23:25:09.457 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:09.454+0000 7f47d496c640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f47d0111b70 con 0x7f47d0107730 2024-03-31T23:25:09.457 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:09.454+0000 7f47ce59c640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f47d0101720 0x7f47d0197ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:25:09.457 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:09.454+0000 7f47ced9d640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f47d0107730 0x7f47d019c790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:25:09.457 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:09.454+0000 7f47cdd9b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f47d01020e0 0x7f47d0198410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:25:09.457 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:09.454+0000 7f47ce59c640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f47d0101720 0x7f47d0197ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.46:58524/0 (socket says 172.21.15.46:58524) 2024-03-31T23:25:09.457 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:09.454+0000 7f47ce59c640 1 -- 172.21.15.46:0/639820910 learned_addr learned my addr 172.21.15.46:0/639820910 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:25:09.458 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:09.454+0000 7f47ced9d640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f47d0107730 0x7f47d019c790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.46:50846/0 (socket says 172.21.15.46:50846) 2024-03-31T23:25:09.458 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:09.454+0000 7f47ce59c640 1 -- 172.21.15.46:0/639820910 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f47d0107730 msgr2=0x7f47d019c790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:25:09.458 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:09.454+0000 7f47ce59c640 1 --2- 172.21.15.46:0/639820910 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f47d0107730 0x7f47d019c790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:25:09.458 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:09.454+0000 7f47ce59c640 1 -- 172.21.15.46:0/639820910 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f47d01020e0 msgr2=0x7f47d0198410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:25:09.458 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:09.454+0000 7f47ce59c640 1 --2- 172.21.15.46:0/639820910 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f47d01020e0 0x7f47d0198410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:25:09.458 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:09.454+0000 7f47ce59c640 1 -- 172.21.15.46:0/639820910 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f47d019cec0 con 0x7f47d0101720 2024-03-31T23:25:09.458 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:09.454+0000 7f47ced9d640 1 --2- 172.21.15.46:0/639820910 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f47d0107730 0x7f47d019c790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:25:09.458 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:09.455+0000 7f47ce59c640 1 --2- 172.21.15.46:0/639820910 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f47d0101720 0x7f47d0197ed0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f47c000e9f0 tx=0x7f47c000eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:25:09.458 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:09.455+0000 7f47bf7fe640 1 -- 172.21.15.46:0/639820910 <== mon.2 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47c000cd80 con 0x7f47d0101720 2024-03-31T23:25:09.458 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:09.455+0000 7f47d496c640 1 -- 172.21.15.46:0/639820910 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f47d019d1b0 con 0x7f47d0101720 2024-03-31T23:25:09.458 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:09.455+0000 7f47d496c640 1 -- 172.21.15.46:0/639820910 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f47d01a4a40 con 0x7f47d0101720 2024-03-31T23:25:09.458 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:09.455+0000 7f47bf7fe640 1 -- 172.21.15.46:0/639820910 <== mon.2 v2:172.21.15.142:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f47c00108e0 con 0x7f47d0101720 2024-03-31T23:25:09.458 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:09.455+0000 7f47bf7fe640 1 -- 172.21.15.46:0/639820910 <== mon.2 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47c0026060 con 0x7f47d0101720 2024-03-31T23:25:09.460 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:09.458+0000 7f47bf7fe640 1 -- 172.21.15.46:0/639820910 <== mon.2 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 25) ==== 137451+0+0 (secure 0 0 0) 0x7f47c0010430 con 0x7f47d0101720 2024-03-31T23:25:09.460 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:09.458+0000 7f47d762e740 1 Processor -- start 2024-03-31T23:25:09.460 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:09.458+0000 7f47d762e740 1 -- start start 2024-03-31T23:25:09.460 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:09.458+0000 7f47d762e740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55f1d61e91e0 0x55f1d61e95c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:25:09.460 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:09.458+0000 7f47d762e740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f1d61e9b00 0x55f1d61f6c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:25:09.460 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:09.458+0000 7f47d762e740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f1d61f71c0 0x55f1d61f95b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:25:09.460 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:09.458+0000 7f47d762e740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55f1d614c8f0 con 0x55f1d61e9b00 2024-03-31T23:25:09.460 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:09.458+0000 7f47d762e740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55f1d60f58d0 con 0x55f1d61e91e0 2024-03-31T23:25:09.461 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:09.458+0000 7f47d762e740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55f1d6148990 con 0x55f1d61f71c0 2024-03-31T23:25:09.461 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:09.458+0000 7f47ce59c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f1d61e9b00 0x55f1d61f6c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:25:09.461 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:09.459+0000 7f47ced9d640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55f1d61e91e0 0x55f1d61e95c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:25:09.461 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:09.459+0000 7f47ce59c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f1d61e9b00 0x55f1d61f6c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:57698/0 (socket says 172.21.15.46:57698) 2024-03-31T23:25:09.461 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:09.459+0000 7f47ce59c640 1 -- 172.21.15.46:0/2338755597 learned_addr learned my addr 172.21.15.46:0/2338755597 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:25:09.461 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:09.459+0000 7f47cdd9b640 1 --2- 172.21.15.46:0/2338755597 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f1d61f71c0 0x55f1d61f95b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:25:09.461 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:09.459+0000 7f47bf7fe640 1 --2- 172.21.15.46:0/639820910 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f47b80a26b0 0x7f47b80a4b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:25:09.461 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:09.459+0000 7f47ced9d640 1 --2- 172.21.15.46:0/639820910 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f47b80a26b0 0x7f47b80a4b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:25:09.461 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:09.459+0000 7f47ce59c640 1 -- 172.21.15.46:0/2338755597 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f1d61f71c0 msgr2=0x55f1d61f95b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:25:09.461 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:09.459+0000 7f47ce59c640 1 --2- 172.21.15.46:0/2338755597 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f1d61f71c0 0x55f1d61f95b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:25:09.461 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:09.459+0000 7f47ce59c640 1 -- 172.21.15.46:0/2338755597 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55f1d61e91e0 msgr2=0x55f1d61e95c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:25:09.461 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:09.459+0000 7f47ce59c640 1 --2- 172.21.15.46:0/2338755597 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55f1d61e91e0 0x55f1d61e95c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:25:09.461 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:09.459+0000 7f47ce59c640 1 -- 172.21.15.46:0/2338755597 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f1d5e61210 con 0x55f1d61e9b00 2024-03-31T23:25:09.461 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:09.459+0000 7f47bf7fe640 1 -- 172.21.15.46:0/639820910 <== mon.2 v2:172.21.15.142:3300/0 5 ==== osd_map(111..111 src has 1..111) ==== 9642+0+0 (secure 0 0 0) 0x7f47c00cf210 con 0x7f47d0101720 2024-03-31T23:25:09.461 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:09.459+0000 7f47ce59c640 1 --2- 172.21.15.46:0/2338755597 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f1d61e9b00 0x55f1d61f6c80 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7f47c009b2e0 tx=0x7f47c0066530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c221797d3ef6183 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:25:09.465 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:09.459+0000 7f47ced9d640 1 --2- 172.21.15.46:0/639820910 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f47b80a26b0 0x7f47b80a4b70 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f47c80049d0 tx=0x7f47c8004ab0 comp rx=0 tx=0).ready entity=mgr.15654 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:25:09.465 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:09.459+0000 7f47bcff9640 1 -- 172.21.15.46:0/2338755597 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47c0010720 con 0x55f1d61e9b00 2024-03-31T23:25:09.465 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:09.459+0000 7f47bcff9640 1 -- 172.21.15.46:0/2338755597 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f47c009b5e0 con 0x55f1d61e9b00 2024-03-31T23:25:09.465 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:09.459+0000 7f47bcff9640 1 -- 172.21.15.46:0/2338755597 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47c0099970 con 0x55f1d61e9b00 2024-03-31T23:25:09.466 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:09.460+0000 7f47d762e740 1 -- 172.21.15.46:0/2338755597 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f1d61e9b00 msgr2=0x55f1d61f6c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:25:09.466 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:09.460+0000 7f47d762e740 1 --2- 172.21.15.46:0/2338755597 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f1d61e9b00 0x55f1d61f6c80 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7f47c009b2e0 tx=0x7f47c0066530 comp rx=0 tx=0).stop 2024-03-31T23:25:09.466 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:09.460+0000 7f47d762e740 1 -- 172.21.15.46:0/2338755597 shutdown_connections 2024-03-31T23:25:09.466 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:09.460+0000 7f47d762e740 1 --2- 172.21.15.46:0/2338755597 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f1d61f71c0 0x55f1d61f95b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:25:09.466 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:09.460+0000 7f47d762e740 1 --2- 172.21.15.46:0/2338755597 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f1d61e9b00 0x55f1d61f6c80 unknown :-1 s=CLOSED pgs=875 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:25:09.466 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:09.460+0000 7f47d762e740 1 --2- 172.21.15.46:0/2338755597 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55f1d61e91e0 0x55f1d61e95c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:25:09.466 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:09.460+0000 7f47d762e740 1 -- 172.21.15.46:0/2338755597 >> 172.21.15.46:0/2338755597 conn(0x55f1d61e3fe0 msgr2=0x55f1d61e43c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:25:09.466 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:09.460+0000 7f47d762e740 1 -- 172.21.15.46:0/2338755597 shutdown_connections 2024-03-31T23:25:09.466 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:09.460+0000 7f47d762e740 1 -- 172.21.15.46:0/2338755597 wait complete. 2024-03-31T23:25:09.466 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:09.461+0000 7f47d762e740 1 -- start start 2024-03-31T23:25:09.466 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:09.461+0000 7f47d762e740 1 Processor -- start 2024-03-31T23:25:09.466 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:09.461+0000 7f47d762e740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f1d61e91e0 0x55f1d61f4150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:25:09.466 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:09.461+0000 7f47d762e740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f1d61e9b00 0x55f1d61f4690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:25:09.467 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:09.461+0000 7f47d762e740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55f1d61f71c0 0x55f1d61f4bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:25:09.467 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:09.461+0000 7f47d762e740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55f1d61f9d80 con 0x55f1d61e91e0 2024-03-31T23:25:09.467 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:09.461+0000 7f47d762e740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55f1d60eaf80 con 0x55f1d61f71c0 2024-03-31T23:25:09.467 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:09.461+0000 7f47d762e740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55f1d61f9f00 con 0x55f1d61e9b00 2024-03-31T23:25:09.467 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:09.461+0000 7f47d762e740 10 client.0 _ll_register_callbacks cb 0x55f1d6143290 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T23:25:09.467 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:09.461+0000 7f47ce59c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f1d61e91e0 0x55f1d61f4150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:25:09.467 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:09.461+0000 7f47ce59c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f1d61e91e0 0x55f1d61f4150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:57704/0 (socket says 172.21.15.46:57704) 2024-03-31T23:25:09.467 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:09.461+0000 7f47ce59c640 1 -- 172.21.15.46:0/155783737 learned_addr learned my addr 172.21.15.46:0/155783737 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:25:09.467 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:09.461+0000 7f47ced9d640 1 --2- 172.21.15.46:0/155783737 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55f1d61f71c0 0x55f1d61f4bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:25:09.467 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:09.461+0000 7f47cdd9b640 1 --2- 172.21.15.46:0/155783737 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f1d61e9b00 0x55f1d61f4690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:25:09.467 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:09.462+0000 7f47ce59c640 1 -- 172.21.15.46:0/155783737 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f1d61e9b00 msgr2=0x55f1d61f4690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:25:09.467 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:09.462+0000 7f47ce59c640 1 --2- 172.21.15.46:0/155783737 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f1d61e9b00 0x55f1d61f4690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:25:09.467 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:09.462+0000 7f47ce59c640 1 -- 172.21.15.46:0/155783737 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55f1d61f71c0 msgr2=0x55f1d61f4bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:25:09.467 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:09.462+0000 7f47ce59c640 1 --2- 172.21.15.46:0/155783737 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55f1d61f71c0 0x55f1d61f4bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:25:09.468 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:09.462+0000 7f47ce59c640 1 -- 172.21.15.46:0/155783737 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f1d5f2d620 con 0x55f1d61e91e0 2024-03-31T23:25:09.468 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:09.462+0000 7f47ce59c640 1 --2- 172.21.15.46:0/155783737 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f1d61e91e0 0x55f1d61f4150 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7f47c0097fb0 tx=0x7f47c0009700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=35216054eff2115e server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:25:09.468 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:09.462+0000 7f47b5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-31T23:25:09.468 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:09.462+0000 7f47b5ffb640 1 -- 172.21.15.46:0/155783737 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47c0097880 con 0x55f1d61e91e0 2024-03-31T23:25:09.468 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:09.462+0000 7f47b5ffb640 1 -- 172.21.15.46:0/155783737 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(111..111 src has 1..111) ==== 9642+0+0 (secure 0 0 0) 0x7f47c0097330 con 0x55f1d61e91e0 2024-03-31T23:25:09.468 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:09.462+0000 7f47d762e740 1 -- 172.21.15.46:0/155783737 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f47c009b5e0 con 0x55f1d61e91e0 2024-03-31T23:25:09.468 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:09.462+0000 7f47b5ffb640 1 -- 172.21.15.46:0/155783737 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7f47c0010720 con 0x55f1d61e91e0 2024-03-31T23:25:09.468 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:09.462+0000 7f47d762e740 10 client.17241 fetch_fsmap learned FSMap version 38 2024-03-31T23:25:09.468 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:09.462+0000 7f47d762e740 1 -- 172.21.15.46:0/155783737 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x55f1d61f9d80 con 0x55f1d61e91e0 2024-03-31T23:25:09.468 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:09.462+0000 7f47b5ffb640 1 -- 172.21.15.46:0/155783737 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f47c009b5e0 con 0x55f1d61e91e0 2024-03-31T23:25:09.468 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:09.463+0000 7f47b5ffb640 1 -- 172.21.15.46:0/155783737 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 38) ==== 8370+0+0 (secure 0 0 0) 0x7f47c009aa30 con 0x55f1d61e91e0 2024-03-31T23:25:09.468 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:09.463+0000 7f47d762e740 10 client.17241 fetch_fsmap finished waiting for FSMap version 38 2024-03-31T23:25:09.468 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:09.463+0000 7f47d762e740 10 client.17241 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-31T23:25:09.469 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:09.463+0000 7f47d762e740 20 client.17241 populate_metadata read hostname 'smithi046' 2024-03-31T23:25:09.469 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:09.463+0000 7f47d762e740 1 --2- 172.21.15.46:0/155783737 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55f1d6206de0 0x55f1d62091d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:25:09.469 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:09.463+0000 7f47d762e740 4 client.17241 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-31T23:25:09.469 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:09.463+0000 7f47d762e740 1 -- 172.21.15.46:0/155783737 --> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f1d6160120 con 0x55f1d6206de0 2024-03-31T23:25:09.469 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:09.463+0000 7f47cdd9b640 1 --2- 172.21.15.46:0/155783737 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55f1d6206de0 0x55f1d62091d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:25:09.469 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:09.463+0000 7f47b5ffb640 10 client.17241 ms_handle_connect on v2:172.21.15.142:6838/2519143918 2024-03-31T23:25:09.469 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:09.463+0000 7f47cdd9b640 1 --2- 172.21.15.46:0/155783737 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55f1d6206de0 0x55f1d62091d0 crc :-1 s=READY pgs=195 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a2cd6518aa1d76c4 server_cookie=8b726d495d028e54 in_seq=0 out_seq=0 2024-03-31T23:25:09.470 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:09.465+0000 7f47b5ffb640 1 -- 172.21.15.46:0/155783737 <== mds.0 v2:172.21.15.142:6838/2519143918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55f1d6160120 con 0x55f1d6206de0 2024-03-31T23:25:09.470 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:09.465+0000 7f47b5ffb640 10 client.17241 handle_command_reply: tid=0 2024-03-31T23:25:09.470 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:09.465+0000 7f47d762e740 1 client.17241 shutdown 2024-03-31T23:25:09.470 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:09.465+0000 7f47d762e740 1 -- 172.21.15.46:0/155783737 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f1d61e91e0 msgr2=0x55f1d61f4150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:25:09.470 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:09.465+0000 7f47d762e740 1 --2- 172.21.15.46:0/155783737 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f1d61e91e0 0x55f1d61f4150 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7f47c0097fb0 tx=0x7f47c0009700 comp rx=0 tx=0).stop 2024-03-31T23:25:09.470 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:09.465+0000 7f47d762e740 1 -- 172.21.15.46:0/155783737 shutdown_connections 2024-03-31T23:25:09.471 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:09.465+0000 7f47d762e740 1 --2- 172.21.15.46:0/155783737 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55f1d6206de0 0x55f1d62091d0 crc :-1 s=READY pgs=195 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:25:09.471 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:09.465+0000 7f47d762e740 1 --2- 172.21.15.46:0/155783737 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55f1d61f71c0 0x55f1d61f4bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:25:09.471 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:09.465+0000 7f47d762e740 1 --2- 172.21.15.46:0/155783737 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f1d61e9b00 0x55f1d61f4690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:25:09.471 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:09.465+0000 7f47d762e740 1 --2- 172.21.15.46:0/155783737 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f1d61e91e0 0x55f1d61f4150 unknown :-1 s=CLOSED pgs=876 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:25:09.471 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:09.465+0000 7f47d762e740 1 -- 172.21.15.46:0/155783737 >> 172.21.15.46:0/155783737 conn(0x55f1d61e3fe0 msgr2=0x55f1d61e5d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:25:09.471 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:09.465+0000 7f47b5ffb640 0 client.17241 ms_handle_reset on v2:172.21.15.142:6838/2519143918 2024-03-31T23:25:09.471 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:09.465+0000 7f47d762e740 1 -- 172.21.15.46:0/155783737 shutdown_connections 2024-03-31T23:25:09.471 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:09.465+0000 7f47d762e740 1 -- 172.21.15.46:0/155783737 wait complete. 2024-03-31T23:25:09.471 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:09.466+0000 7f47d762e740 20 client.17241 trim_cache size 0 max 16384 2024-03-31T23:25:09.488 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:09.487+0000 7f47d762e740 1 Processor -- start 2024-03-31T23:25:09.489 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:09.487+0000 7f47d762e740 1 -- start start 2024-03-31T23:25:09.489 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:09.487+0000 7f47d762e740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f1d61e91e0 0x55f1d61f6880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:25:09.489 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:09.487+0000 7f47d762e740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55f1d61e9b00 0x55f1d6202b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:25:09.489 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:09.487+0000 7f47d762e740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f1d61f71c0 0x55f1d620e990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:25:09.489 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:09.487+0000 7f47d762e740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f47b0012b30 con 0x55f1d61f71c0 2024-03-31T23:25:09.489 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:09.487+0000 7f47d762e740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f47b0010340 con 0x55f1d61e9b00 2024-03-31T23:25:09.489 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:09.487+0000 7f47d762e740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f47b000e1b0 con 0x55f1d61e91e0 2024-03-31T23:25:09.489 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:09.487+0000 7f47ce59c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f1d61e91e0 0x55f1d61f6880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:25:09.489 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:09.487+0000 7f47ced9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f1d61f71c0 0x55f1d620e990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:25:09.489 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:09.487+0000 7f47ce59c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f1d61e91e0 0x55f1d61f6880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.46:50866/0 (socket says 172.21.15.46:50866) 2024-03-31T23:25:09.489 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:09.487+0000 7f47ce59c640 1 -- 172.21.15.46:0/547607013 learned_addr learned my addr 172.21.15.46:0/547607013 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:25:09.489 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:09.487+0000 7f47cdd9b640 1 --2- 172.21.15.46:0/547607013 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55f1d61e9b00 0x55f1d6202b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:25:09.489 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:09.488+0000 7f47ce59c640 1 -- 172.21.15.46:0/547607013 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55f1d61e9b00 msgr2=0x55f1d6202b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:25:09.490 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:09.488+0000 7f47ce59c640 1 --2- 172.21.15.46:0/547607013 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55f1d61e9b00 0x55f1d6202b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:25:09.490 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:09.488+0000 7f47ce59c640 1 -- 172.21.15.46:0/547607013 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f1d61f71c0 msgr2=0x55f1d620e990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:25:09.490 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:09.488+0000 7f47ce59c640 1 --2- 172.21.15.46:0/547607013 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f1d61f71c0 0x55f1d620e990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:25:09.490 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:09.488+0000 7f47ce59c640 1 -- 172.21.15.46:0/547607013 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f1d6203790 con 0x55f1d61e91e0 2024-03-31T23:25:09.490 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:09.488+0000 7f47ced9d640 1 --2- 172.21.15.46:0/547607013 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f1d61f71c0 0x55f1d620e990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:25:09.493 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:09.488+0000 7f47ce59c640 1 --2- 172.21.15.46:0/547607013 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f1d61e91e0 0x55f1d61f6880 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f47c009b2e0 tx=0x7f47c00bb340 comp rx=0 tx=0).ready entity=mon.1 client_cookie=234c6df9d05f1e20 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:25:09.493 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:09.488+0000 7f47b7fff640 1 -- 172.21.15.46:0/547607013 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47c009b5e0 con 0x55f1d61e91e0 2024-03-31T23:25:09.493 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:09.488+0000 7f47b7fff640 1 -- 172.21.15.46:0/547607013 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f47c009f680 con 0x55f1d61e91e0 2024-03-31T23:25:09.493 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:09.488+0000 7f47b7fff640 1 -- 172.21.15.46:0/547607013 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47c0084e20 con 0x55f1d61e91e0 2024-03-31T23:25:09.493 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:09.489+0000 7f47d762e740 1 -- 172.21.15.46:0/547607013 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f1d61e91e0 msgr2=0x55f1d61f6880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:25:09.493 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:09.489+0000 7f47d762e740 1 --2- 172.21.15.46:0/547607013 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f1d61e91e0 0x55f1d61f6880 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f47c009b2e0 tx=0x7f47c00bb340 comp rx=0 tx=0).stop 2024-03-31T23:25:09.493 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:09.489+0000 7f47d762e740 1 -- 172.21.15.46:0/547607013 shutdown_connections 2024-03-31T23:25:09.493 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:09.489+0000 7f47d762e740 1 --2- 172.21.15.46:0/547607013 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f1d61f71c0 0x55f1d620e990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:25:09.494 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:09.489+0000 7f47d762e740 1 --2- 172.21.15.46:0/547607013 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55f1d61e9b00 0x55f1d6202b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:25:09.494 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:09.489+0000 7f47d762e740 1 --2- 172.21.15.46:0/547607013 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f1d61e91e0 0x55f1d61f6880 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:25:09.494 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:09.489+0000 7f47d762e740 1 -- 172.21.15.46:0/547607013 >> 172.21.15.46:0/547607013 conn(0x55f1d61e3fe0 msgr2=0x55f1d61e2f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:25:09.494 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:09.489+0000 7f47d762e740 1 -- 172.21.15.46:0/547607013 shutdown_connections 2024-03-31T23:25:09.494 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:09.489+0000 7f47d762e740 1 -- 172.21.15.46:0/547607013 wait complete. 2024-03-31T23:25:09.494 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:09.490+0000 7f47d762e740 1 -- start start 2024-03-31T23:25:09.494 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:09.490+0000 7f47d762e740 1 Processor -- start 2024-03-31T23:25:09.494 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:09.490+0000 7f47d762e740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f1d61e91e0 0x55f1d61ff490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:25:09.494 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:09.490+0000 7f47d762e740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f1d61e9b00 0x55f1d61f0a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:25:09.494 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:09.490+0000 7f47d762e740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55f1d61f71c0 0x55f1d61f0f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:25:09.494 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:09.490+0000 7f47d762e740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f47b000e800 con 0x55f1d61e91e0 2024-03-31T23:25:09.494 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:09.490+0000 7f47d762e740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f47b000fc70 con 0x55f1d61f71c0 2024-03-31T23:25:09.494 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:09.490+0000 7f47d762e740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f47b000eed0 con 0x55f1d61e9b00 2024-03-31T23:25:09.494 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:09.490+0000 7f47d762e740 10 client.0 _ll_register_callbacks cb 0x7f47c8009ee0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T23:25:09.494 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:09.490+0000 7f47ce59c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f1d61e91e0 0x55f1d61ff490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:25:09.495 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:09.490+0000 7f47ce59c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f1d61e91e0 0x55f1d61ff490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:57718/0 (socket says 172.21.15.46:57718) 2024-03-31T23:25:09.495 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:09.490+0000 7f47ce59c640 1 -- 172.21.15.46:0/3088093014 learned_addr learned my addr 172.21.15.46:0/3088093014 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:25:09.495 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:09.490+0000 7f47ced9d640 1 --2- 172.21.15.46:0/3088093014 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55f1d61f71c0 0x55f1d61f0f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:25:09.495 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:09.490+0000 7f47cdd9b640 1 --2- 172.21.15.46:0/3088093014 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f1d61e9b00 0x55f1d61f0a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:25:09.495 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:09.491+0000 7f47ce59c640 1 -- 172.21.15.46:0/3088093014 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f1d61e9b00 msgr2=0x55f1d61f0a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:25:09.495 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:09.491+0000 7f47ce59c640 1 --2- 172.21.15.46:0/3088093014 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f1d61e9b00 0x55f1d61f0a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:25:09.495 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:09.491+0000 7f47ce59c640 1 -- 172.21.15.46:0/3088093014 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55f1d61f71c0 msgr2=0x55f1d61f0f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:25:09.495 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:09.491+0000 7f47ce59c640 1 --2- 172.21.15.46:0/3088093014 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55f1d61f71c0 0x55f1d61f0f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:25:09.495 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:09.491+0000 7f47ce59c640 1 -- 172.21.15.46:0/3088093014 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f1d61f17b0 con 0x55f1d61e91e0 2024-03-31T23:25:09.495 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:09.491+0000 7f47ced9d640 1 --2- 172.21.15.46:0/3088093014 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55f1d61f71c0 0x55f1d61f0f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T23:25:09.495 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:09.491+0000 7f47cdd9b640 1 --2- 172.21.15.46:0/3088093014 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f1d61e9b00 0x55f1d61f0a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T23:25:09.495 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:09.491+0000 7f47ce59c640 1 --2- 172.21.15.46:0/3088093014 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f1d61e91e0 0x55f1d61ff490 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7f47c009b570 tx=0x7f47c0000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d6ad47ad39c7ed59 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:25:09.495 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:09.491+0000 7f47b5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-31T23:25:09.495 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:09.491+0000 7f47b5ffb640 1 -- 172.21.15.46:0/3088093014 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47c009b5e0 con 0x55f1d61e91e0 2024-03-31T23:25:09.495 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:09.491+0000 7f47b5ffb640 1 -- 172.21.15.46:0/3088093014 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(111..111 src has 1..111) ==== 9642+0+0 (secure 0 0 0) 0x7f47c0084bc0 con 0x55f1d61e91e0 2024-03-31T23:25:09.496 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:09.491+0000 7f47d762e740 1 -- 172.21.15.46:0/3088093014 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f47c009f680 con 0x55f1d61e91e0 2024-03-31T23:25:09.496 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:09.491+0000 7f47b5ffb640 1 -- 172.21.15.46:0/3088093014 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7f47c0084dc0 con 0x55f1d61e91e0 2024-03-31T23:25:09.496 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:09.491+0000 7f47d762e740 10 client.17247 fetch_fsmap learned FSMap version 38 2024-03-31T23:25:09.496 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:09.491+0000 7f47d762e740 1 -- 172.21.15.46:0/3088093014 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x55f1d61f17b0 con 0x55f1d61e91e0 2024-03-31T23:25:09.496 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:09.492+0000 7f47b5ffb640 1 -- 172.21.15.46:0/3088093014 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f47c009f680 con 0x55f1d61e91e0 2024-03-31T23:25:09.496 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:09.492+0000 7f47b5ffb640 1 -- 172.21.15.46:0/3088093014 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 38) ==== 8370+0+0 (secure 0 0 0) 0x7f47c0010a90 con 0x55f1d61e91e0 2024-03-31T23:25:09.496 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:09.492+0000 7f47d762e740 10 client.17247 fetch_fsmap finished waiting for FSMap version 38 2024-03-31T23:25:09.496 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:09.492+0000 7f47d762e740 10 client.17247 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-31T23:25:09.496 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:09.492+0000 7f47d762e740 20 client.17247 populate_metadata read hostname 'smithi046' 2024-03-31T23:25:09.496 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:09.492+0000 7f47d762e740 1 --2- 172.21.15.46:0/3088093014 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55f1d626d230 0x55f1d626f620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:25:09.496 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:09.492+0000 7f47d762e740 4 client.17247 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-31T23:25:09.497 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:09.492+0000 7f47d762e740 1 -- 172.21.15.46:0/3088093014 --> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55f1d6159e30 con 0x55f1d626d230 2024-03-31T23:25:09.497 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:09.492+0000 7f47cdd9b640 1 --2- 172.21.15.46:0/3088093014 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55f1d626d230 0x55f1d626f620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:25:09.497 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:09.492+0000 7f47cdd9b640 1 --2- 172.21.15.46:0/3088093014 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55f1d626d230 0x55f1d626f620 crc :-1 s=READY pgs=196 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=38e1e9460bacfde1 server_cookie=bdd48e4a244410f2 in_seq=0 out_seq=0 2024-03-31T23:25:09.497 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:09.492+0000 7f47b5ffb640 10 client.17247 ms_handle_connect on v2:172.21.15.142:6838/2519143918 2024-03-31T23:25:10.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:25:09 smithi142 ceph-mon[35271]: pgmap v421: 129 pgs: 129 active+clean; 367 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 3.6 MiB/s wr, 187 op/s 2024-03-31T23:25:10.265 INFO:tasks.workunit.client.0.smithi046.stdout: 10 1843 17.99 MB/sec warmup 15 sec latency 2623.863 ms 2024-03-31T23:25:10.295 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:25:09 smithi162 ceph-mon[36237]: pgmap v421: 129 pgs: 129 active+clean; 367 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 3.6 MiB/s wr, 187 op/s 2024-03-31T23:25:10.361 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:25:09 smithi046 ceph-mon[32498]: pgmap v421: 129 pgs: 129 active+clean; 367 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 3.6 MiB/s wr, 187 op/s 2024-03-31T23:25:11.266 INFO:tasks.workunit.client.0.smithi046.stdout: 10 2010 17.07 MB/sec warmup 16 sec latency 3623.893 ms 2024-03-31T23:25:11.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:25:11 smithi046 ceph-mon[32498]: osdmap e112: 12 total, 12 up, 12 in 2024-03-31T23:25:11.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:25:11 smithi142 ceph-mon[35271]: osdmap e112: 12 total, 12 up, 12 in 2024-03-31T23:25:12.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:25:11 smithi162 ceph-mon[36237]: osdmap e112: 12 total, 12 up, 12 in 2024-03-31T23:25:12.266 INFO:tasks.workunit.client.0.smithi046.stdout: 10 2172 16.37 MB/sec warmup 17 sec latency 4623.935 ms 2024-03-31T23:25:12.544 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:12.542+0000 7f47b5ffb640 1 -- 172.21.15.46:0/3088093014 <== mds.0 v2:172.21.15.142:6838/2519143918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55f1d6159e30 con 0x55f1d626d230 2024-03-31T23:25:12.544 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:12.542+0000 7f47b5ffb640 10 client.17247 handle_command_reply: tid=0 2024-03-31T23:25:12.544 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:12.542+0000 7f47d762e740 1 client.17247 shutdown 2024-03-31T23:25:12.544 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:12.542+0000 7f47d762e740 1 -- 172.21.15.46:0/3088093014 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f1d61e91e0 msgr2=0x55f1d61ff490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:25:12.544 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:12.542+0000 7f47d762e740 1 --2- 172.21.15.46:0/3088093014 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f1d61e91e0 0x55f1d61ff490 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7f47c009b570 tx=0x7f47c0000bc0 comp rx=0 tx=0).stop 2024-03-31T23:25:12.544 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:12.542+0000 7f47d762e740 1 -- 172.21.15.46:0/3088093014 shutdown_connections 2024-03-31T23:25:12.544 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:12.542+0000 7f47d762e740 1 --2- 172.21.15.46:0/3088093014 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55f1d626d230 0x55f1d626f620 crc :-1 s=READY pgs=196 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:25:12.545 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:12.542+0000 7f47d762e740 1 --2- 172.21.15.46:0/3088093014 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55f1d61f71c0 0x55f1d61f0f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:25:12.545 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:12.542+0000 7f47d762e740 1 --2- 172.21.15.46:0/3088093014 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f1d61e9b00 0x55f1d61f0a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:25:12.545 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:12.542+0000 7f47d762e740 1 --2- 172.21.15.46:0/3088093014 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f1d61e91e0 0x55f1d61ff490 unknown :-1 s=CLOSED pgs=877 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:25:12.545 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:12.542+0000 7f47d762e740 1 -- 172.21.15.46:0/3088093014 >> 172.21.15.46:0/3088093014 conn(0x55f1d61e3fe0 msgr2=0x55f1d61f3f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:25:12.545 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:12.542+0000 7f47b5ffb640 0 client.17247 ms_handle_reset on v2:172.21.15.142:6838/2519143918 2024-03-31T23:25:12.545 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:12.542+0000 7f47d762e740 1 -- 172.21.15.46:0/3088093014 shutdown_connections 2024-03-31T23:25:12.545 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:12.542+0000 7f47d762e740 1 -- 172.21.15.46:0/3088093014 wait complete. 2024-03-31T23:25:12.545 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:12.542+0000 7f47d762e740 20 client.17247 trim_cache size 0 max 16384 2024-03-31T23:25:12.545 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout:{ 2024-03-31T23:25:12.545 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout: "message": "", 2024-03-31T23:25:12.545 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout: "return_code": 0 2024-03-31T23:25:12.546 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout:} 2024-03-31T23:25:12.546 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:12.543+0000 7f47d496c640 1 -- 172.21.15.46:0/639820910 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f47b80a26b0 msgr2=0x7f47b80a4b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:25:12.546 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:12.543+0000 7f47d496c640 1 --2- 172.21.15.46:0/639820910 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f47b80a26b0 0x7f47b80a4b70 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f47c80049d0 tx=0x7f47c8004ab0 comp rx=0 tx=0).stop 2024-03-31T23:25:12.547 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:12.543+0000 7f47d496c640 1 -- 172.21.15.46:0/639820910 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f47d0101720 msgr2=0x7f47d0197ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:25:12.547 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:12.543+0000 7f47d496c640 1 --2- 172.21.15.46:0/639820910 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f47d0101720 0x7f47d0197ed0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f47c000e9f0 tx=0x7f47c000eeb0 comp rx=0 tx=0).stop 2024-03-31T23:25:12.547 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:12.543+0000 7f47d496c640 1 -- 172.21.15.46:0/639820910 shutdown_connections 2024-03-31T23:25:12.547 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:12.543+0000 7f47d496c640 1 --2- 172.21.15.46:0/639820910 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f47b80a26b0 0x7f47b80a4b70 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:25:12.547 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:12.543+0000 7f47d496c640 1 --2- 172.21.15.46:0/639820910 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f47d0107730 0x7f47d019c790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:25:12.547 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:12.543+0000 7f47d496c640 1 --2- 172.21.15.46:0/639820910 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f47d01020e0 0x7f47d0198410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:25:12.547 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:12.543+0000 7f47d496c640 1 --2- 172.21.15.46:0/639820910 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f47d0101720 0x7f47d0197ed0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:25:12.547 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:12.543+0000 7f47d496c640 1 -- 172.21.15.46:0/639820910 >> 172.21.15.46:0/639820910 conn(0x7f47d00fd250 msgr2=0x7f47d00fd9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:25:12.547 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:12.543+0000 7f47d496c640 1 -- 172.21.15.46:0/639820910 shutdown_connections 2024-03-31T23:25:12.547 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:12.543+0000 7f47d496c640 1 -- 172.21.15.46:0/639820910 wait complete. 2024-03-31T23:25:12.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:25:12 smithi046 ceph-mon[32498]: pgmap v423: 129 pgs: 129 active+clean; 367 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 4.4 MiB/s wr, 230 op/s 2024-03-31T23:25:12.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:25:12 smithi142 ceph-mon[35271]: pgmap v423: 129 pgs: 129 active+clean; 367 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 4.4 MiB/s wr, 230 op/s 2024-03-31T23:25:13.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:25:12 smithi162 ceph-mon[36237]: pgmap v423: 129 pgs: 129 active+clean; 367 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 4.4 MiB/s wr, 230 op/s 2024-03-31T23:25:13.266 INFO:tasks.workunit.client.0.smithi046.stdout: 10 2311 15.77 MB/sec warmup 18 sec latency 4896.925 ms 2024-03-31T23:25:14.266 INFO:tasks.workunit.client.0.smithi046.stdout: 10 2488 15.28 MB/sec warmup 19 sec latency 1808.661 ms 2024-03-31T23:25:14.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:25:14 smithi046 ceph-mon[32498]: pgmap v424: 129 pgs: 129 active+clean; 367 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 3.6 MiB/s wr, 187 op/s 2024-03-31T23:25:14.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:25:14 smithi142 ceph-mon[35271]: pgmap v424: 129 pgs: 129 active+clean; 367 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 3.6 MiB/s wr, 187 op/s 2024-03-31T23:25:15.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:25:14 smithi162 ceph-mon[36237]: pgmap v424: 129 pgs: 129 active+clean; 367 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 3.6 MiB/s wr, 187 op/s 2024-03-31T23:25:15.266 INFO:tasks.workunit.client.0.smithi046.stdout: 10 2708 15.47 MB/sec warmup 20 sec latency 2808.736 ms 2024-03-31T23:25:16.266 INFO:tasks.workunit.client.0.smithi046.stdout: 10 2945 15.55 MB/sec warmup 21 sec latency 3808.773 ms 2024-03-31T23:25:16.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:25:16 smithi046 ceph-mon[32498]: pgmap v425: 129 pgs: 129 active+clean; 364 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 900 KiB/s rd, 3.0 MiB/s wr, 281 op/s 2024-03-31T23:25:16.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:25:16 smithi142 ceph-mon[35271]: pgmap v425: 129 pgs: 129 active+clean; 364 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 900 KiB/s rd, 3.0 MiB/s wr, 281 op/s 2024-03-31T23:25:17.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:25:16 smithi162 ceph-mon[36237]: pgmap v425: 129 pgs: 129 active+clean; 364 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 900 KiB/s rd, 3.0 MiB/s wr, 281 op/s 2024-03-31T23:25:17.266 INFO:tasks.workunit.client.0.smithi046.stdout: 10 3126 15.43 MB/sec warmup 22 sec latency 4808.807 ms 2024-03-31T23:25:18.266 INFO:tasks.workunit.client.0.smithi046.stdout: 10 3234 14.95 MB/sec warmup 23 sec latency 5337.542 ms 2024-03-31T23:25:18.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:25:18 smithi046 ceph-mon[32498]: pgmap v426: 129 pgs: 129 active+clean; 364 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 900 KiB/s rd, 3.0 MiB/s wr, 281 op/s 2024-03-31T23:25:18.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:25:18 smithi142 ceph-mon[35271]: pgmap v426: 129 pgs: 129 active+clean; 364 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 900 KiB/s rd, 3.0 MiB/s wr, 281 op/s 2024-03-31T23:25:19.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:25:18 smithi162 ceph-mon[36237]: pgmap v426: 129 pgs: 129 active+clean; 364 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 900 KiB/s rd, 3.0 MiB/s wr, 281 op/s 2024-03-31T23:25:19.266 INFO:tasks.workunit.client.0.smithi046.stdout: 10 3413 15.07 MB/sec warmup 24 sec latency 67.965 ms 2024-03-31T23:25:20.266 INFO:tasks.workunit.client.0.smithi046.stdout: 10 3621 15.23 MB/sec warmup 25 sec latency 60.726 ms 2024-03-31T23:25:20.847 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:25:20 smithi162 ceph-mon[36237]: pgmap v427: 129 pgs: 129 active+clean; 384 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 4.7 MiB/s wr, 301 op/s 2024-03-31T23:25:20.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:25:20 smithi046 ceph-mon[32498]: pgmap v427: 129 pgs: 129 active+clean; 384 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 4.7 MiB/s wr, 301 op/s 2024-03-31T23:25:20.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:25:20 smithi142 ceph-mon[35271]: pgmap v427: 129 pgs: 129 active+clean; 384 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 4.7 MiB/s wr, 301 op/s 2024-03-31T23:25:21.266 INFO:tasks.workunit.client.0.smithi046.stdout: 10 3790 14.90 MB/sec warmup 26 sec latency 77.585 ms 2024-03-31T23:25:22.266 INFO:tasks.workunit.client.0.smithi046.stdout: 10 3941 14.66 MB/sec warmup 27 sec latency 177.796 ms 2024-03-31T23:25:22.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:25:22 smithi046 ceph-mon[32498]: pgmap v428: 129 pgs: 129 active+clean; 384 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 4.5 MiB/s wr, 291 op/s 2024-03-31T23:25:22.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:25:22 smithi142 ceph-mon[35271]: pgmap v428: 129 pgs: 129 active+clean; 384 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 4.5 MiB/s wr, 291 op/s 2024-03-31T23:25:23.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:25:22 smithi162 ceph-mon[36237]: pgmap v428: 129 pgs: 129 active+clean; 384 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 4.5 MiB/s wr, 291 op/s 2024-03-31T23:25:23.266 INFO:tasks.workunit.client.0.smithi046.stdout: 10 4053 14.18 MB/sec warmup 28 sec latency 91.104 ms 2024-03-31T23:25:24.266 INFO:tasks.workunit.client.0.smithi046.stdout: 10 4196 13.76 MB/sec warmup 29 sec latency 72.005 ms 2024-03-31T23:25:24.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:25:24 smithi046 ceph-mon[32498]: pgmap v429: 129 pgs: 129 active+clean; 384 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 888 KiB/s rd, 3.9 MiB/s wr, 251 op/s 2024-03-31T23:25:24.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:25:24 smithi142 ceph-mon[35271]: pgmap v429: 129 pgs: 129 active+clean; 384 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 888 KiB/s rd, 3.9 MiB/s wr, 251 op/s 2024-03-31T23:25:25.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:25:24 smithi162 ceph-mon[36237]: pgmap v429: 129 pgs: 129 active+clean; 384 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 888 KiB/s rd, 3.9 MiB/s wr, 251 op/s 2024-03-31T23:25:25.266 INFO:tasks.workunit.client.0.smithi046.stdout: 10 4383 13.64 MB/sec warmup 30 sec latency 106.681 ms 2024-03-31T23:25:25.679 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:25.678+0000 7fa8311e5640 1 -- 172.21.15.46:0/1237004018 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa82c1008a0 msgr2=0x7fa82c100d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:25:25.680 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:25.678+0000 7fa8311e5640 1 --2- 172.21.15.46:0/1237004018 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa82c1008a0 0x7fa82c100d20 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fa814009930 tx=0x7fa81402bc20 comp rx=0 tx=0).stop 2024-03-31T23:25:25.680 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:25.678+0000 7fa8311e5640 1 -- 172.21.15.46:0/1237004018 shutdown_connections 2024-03-31T23:25:25.680 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:25.678+0000 7fa8311e5640 1 --2- 172.21.15.46:0/1237004018 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa82c101260 0x7fa82c10d480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:25:25.680 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:25.678+0000 7fa8311e5640 1 --2- 172.21.15.46:0/1237004018 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa82c1008a0 0x7fa82c100d20 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:25:25.680 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:25.678+0000 7fa8311e5640 1 --2- 172.21.15.46:0/1237004018 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa82c1055f0 0x7fa82c1059d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:25:25.680 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:25.678+0000 7fa8311e5640 1 -- 172.21.15.46:0/1237004018 >> 172.21.15.46:0/1237004018 conn(0x7fa82c0774a0 msgr2=0x7fa82c0778b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:25:25.680 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:25.678+0000 7fa8311e5640 1 -- 172.21.15.46:0/1237004018 shutdown_connections 2024-03-31T23:25:25.680 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:25.678+0000 7fa8311e5640 1 -- 172.21.15.46:0/1237004018 wait complete. 2024-03-31T23:25:25.680 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:25.679+0000 7fa8311e5640 1 Processor -- start 2024-03-31T23:25:25.682 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:25.679+0000 7fa8311e5640 1 -- start start 2024-03-31T23:25:25.683 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:25.679+0000 7fa8311e5640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa82c1008a0 0x7fa82c19a110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:25:25.683 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:25.679+0000 7fa8311e5640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa82c101260 0x7fa82c19a650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:25:25.683 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:25.679+0000 7fa8311e5640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa82c1055f0 0x7fa82c19e9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:25:25.683 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:25.679+0000 7fa8311e5640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fa82c113d10 con 0x7fa82c1008a0 2024-03-31T23:25:25.683 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:25.679+0000 7fa8311e5640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fa82c113b90 con 0x7fa82c1055f0 2024-03-31T23:25:25.683 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:25.679+0000 7fa8311e5640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fa82c113e90 con 0x7fa82c101260 2024-03-31T23:25:25.683 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:25.679+0000 7fa82ad9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa82c1008a0 0x7fa82c19a110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:25:25.683 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:25.679+0000 7fa82a59c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa82c101260 0x7fa82c19a650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:25:25.683 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:25.679+0000 7fa82b59e640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa82c1055f0 0x7fa82c19e9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:25:25.684 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:25.679+0000 7fa82a59c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa82c101260 0x7fa82c19a650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.46:44162/0 (socket says 172.21.15.46:44162) 2024-03-31T23:25:25.684 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:25.679+0000 7fa82a59c640 1 -- 172.21.15.46:0/3767913591 learned_addr learned my addr 172.21.15.46:0/3767913591 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:25:25.684 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:25.680+0000 7fa82b59e640 1 -- 172.21.15.46:0/3767913591 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa82c101260 msgr2=0x7fa82c19a650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:25:25.684 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:25.680+0000 7fa82b59e640 1 --2- 172.21.15.46:0/3767913591 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa82c101260 0x7fa82c19a650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:25:25.684 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:25.680+0000 7fa82b59e640 1 -- 172.21.15.46:0/3767913591 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa82c1008a0 msgr2=0x7fa82c19a110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:25:25.684 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:25.680+0000 7fa82b59e640 1 --2- 172.21.15.46:0/3767913591 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa82c1008a0 0x7fa82c19a110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:25:25.684 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:25.680+0000 7fa82b59e640 1 -- 172.21.15.46:0/3767913591 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa82c19f100 con 0x7fa82c1055f0 2024-03-31T23:25:25.684 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:25.680+0000 7fa82a59c640 1 --2- 172.21.15.46:0/3767913591 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa82c101260 0x7fa82c19a650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:25:25.684 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:25.680+0000 7fa82ad9d640 1 --2- 172.21.15.46:0/3767913591 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa82c1008a0 0x7fa82c19a110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:25:25.684 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:25.680+0000 7fa82b59e640 1 --2- 172.21.15.46:0/3767913591 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa82c1055f0 0x7fa82c19e9d0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fa82400d590 tx=0x7fa82400da50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:25:25.684 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:25.680+0000 7fa823fff640 1 -- 172.21.15.46:0/3767913591 <== mon.2 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa824017070 con 0x7fa82c1055f0 2024-03-31T23:25:25.685 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:25.680+0000 7fa823fff640 1 -- 172.21.15.46:0/3767913591 <== mon.2 v2:172.21.15.142:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa8240054d0 con 0x7fa82c1055f0 2024-03-31T23:25:25.685 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:25.680+0000 7fa823fff640 1 -- 172.21.15.46:0/3767913591 <== mon.2 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8240096d0 con 0x7fa82c1055f0 2024-03-31T23:25:25.685 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:25.680+0000 7fa8311e5640 1 -- 172.21.15.46:0/3767913591 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa82c19f3f0 con 0x7fa82c1055f0 2024-03-31T23:25:25.685 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:25.680+0000 7fa8311e5640 1 -- 172.21.15.46:0/3767913591 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa82c1a6c80 con 0x7fa82c1055f0 2024-03-31T23:25:25.694 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:25.683+0000 7fa833dc8740 1 Processor -- start 2024-03-31T23:25:25.694 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:25.683+0000 7fa823fff640 1 -- 172.21.15.46:0/3767913591 <== mon.2 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 25) ==== 137451+0+0 (secure 0 0 0) 0x7fa824005020 con 0x7fa82c1055f0 2024-03-31T23:25:25.694 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:25.683+0000 7fa833dc8740 1 -- start start 2024-03-31T23:25:25.694 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:25.684+0000 7fa833dc8740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56546625f6c0 0x56546625faa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:25:25.694 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:25.684+0000 7fa833dc8740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56546625ffe0 0x565466271080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:25:25.694 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:25.684+0000 7fa833dc8740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5654662715c0 0x5654662739b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:25:25.694 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:25.684+0000 7fa833dc8740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x5654661c28f0 con 0x5654662715c0 2024-03-31T23:25:25.694 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:25.684+0000 7fa833dc8740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x56546616b8d0 con 0x56546625ffe0 2024-03-31T23:25:25.694 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:25.684+0000 7fa833dc8740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5654661be990 con 0x56546625f6c0 2024-03-31T23:25:25.694 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:25.684+0000 7fa82ad9d640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56546625ffe0 0x565466271080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:25:25.695 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:25.684+0000 7fa82ad9d640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56546625ffe0 0x565466271080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.46:46234/0 (socket says 172.21.15.46:46234) 2024-03-31T23:25:25.695 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:25.684+0000 7fa82ad9d640 1 -- 172.21.15.46:0/846968364 learned_addr learned my addr 172.21.15.46:0/846968364 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:25:25.695 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:25.684+0000 7fa823fff640 1 --2- 172.21.15.46:0/3767913591 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fa8180a26b0 0x7fa8180a4b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:25:25.695 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:25.684+0000 7fa82a59c640 1 --2- 172.21.15.46:0/846968364 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56546625f6c0 0x56546625faa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:25:25.695 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:25.684+0000 7fa82a59c640 1 --2- 172.21.15.46:0/846968364 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5654662715c0 0x5654662739b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:25:25.695 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:25.684+0000 7fa823fff640 1 -- 172.21.15.46:0/3767913591 <== mon.2 v2:172.21.15.142:3300/0 5 ==== osd_map(112..112 src has 1..112) ==== 9554+0+0 (secure 0 0 0) 0x7fa8240ce9f0 con 0x7fa82c1055f0 2024-03-31T23:25:25.695 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:25.684+0000 7fa82b59e640 1 --2- 172.21.15.46:0/3767913591 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fa8180a26b0 0x7fa8180a4b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:25:25.695 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:25.684+0000 7fa82ad9d640 1 -- 172.21.15.46:0/846968364 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56546625f6c0 msgr2=0x56546625faa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:25:25.695 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:25.684+0000 7fa82ad9d640 1 --2- 172.21.15.46:0/846968364 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56546625f6c0 0x56546625faa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:25:25.695 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:25.684+0000 7fa82ad9d640 1 -- 172.21.15.46:0/846968364 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5654662715c0 msgr2=0x5654662739b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:25:25.695 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:25.685+0000 7fa82ad9d640 1 --2- 172.21.15.46:0/846968364 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5654662715c0 0x5654662739b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:25:25.695 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:25.685+0000 7fa82ad9d640 1 -- 172.21.15.46:0/846968364 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x565465ed7210 con 0x56546625ffe0 2024-03-31T23:25:25.695 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:25.685+0000 7fa82ad9d640 1 --2- 172.21.15.46:0/846968364 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56546625ffe0 0x565466271080 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fa81c00b480 tx=0x7fa81c00b710 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1aac1db9c8f722c7 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:25:25.696 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:25.685+0000 7fa82b59e640 1 --2- 172.21.15.46:0/3767913591 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fa8180a26b0 0x7fa8180a4b70 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fa8240cbde0 tx=0x7fa824017eb0 comp rx=0 tx=0).ready entity=mgr.15654 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:25:25.696 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:25.685+0000 7fa8217fa640 1 -- 172.21.15.46:0/846968364 <== mon.2 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa81c016670 con 0x56546625ffe0 2024-03-31T23:25:25.696 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:25.685+0000 7fa8217fa640 1 -- 172.21.15.46:0/846968364 <== mon.2 v2:172.21.15.142:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa81c01b070 con 0x56546625ffe0 2024-03-31T23:25:25.696 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:25.685+0000 7fa8217fa640 1 -- 172.21.15.46:0/846968364 <== mon.2 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa81c015b50 con 0x56546625ffe0 2024-03-31T23:25:25.696 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:25.686+0000 7fa833dc8740 1 -- 172.21.15.46:0/846968364 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56546625ffe0 msgr2=0x565466271080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:25:25.696 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:25.686+0000 7fa833dc8740 1 --2- 172.21.15.46:0/846968364 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56546625ffe0 0x565466271080 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fa81c00b480 tx=0x7fa81c00b710 comp rx=0 tx=0).stop 2024-03-31T23:25:25.696 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:25.686+0000 7fa833dc8740 1 -- 172.21.15.46:0/846968364 shutdown_connections 2024-03-31T23:25:25.696 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:25.686+0000 7fa833dc8740 1 --2- 172.21.15.46:0/846968364 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5654662715c0 0x5654662739b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:25:25.696 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:25.686+0000 7fa833dc8740 1 --2- 172.21.15.46:0/846968364 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56546625ffe0 0x565466271080 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:25:25.696 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:25.686+0000 7fa833dc8740 1 --2- 172.21.15.46:0/846968364 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56546625f6c0 0x56546625faa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:25:25.696 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:25.686+0000 7fa833dc8740 1 -- 172.21.15.46:0/846968364 >> 172.21.15.46:0/846968364 conn(0x565466259fe0 msgr2=0x56546625a3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:25:25.696 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:25.686+0000 7fa833dc8740 1 -- 172.21.15.46:0/846968364 shutdown_connections 2024-03-31T23:25:25.696 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:25.686+0000 7fa833dc8740 1 -- 172.21.15.46:0/846968364 wait complete. 2024-03-31T23:25:25.696 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:25.686+0000 7fa833dc8740 1 -- start start 2024-03-31T23:25:25.696 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:25.687+0000 7fa833dc8740 1 Processor -- start 2024-03-31T23:25:25.697 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:25.687+0000 7fa833dc8740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56546625f6c0 0x56546626e590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:25:25.697 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:25.687+0000 7fa833dc8740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56546625ffe0 0x56546626ead0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:25:25.697 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:25.687+0000 7fa833dc8740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5654662715c0 0x56546626f010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:25:25.697 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:25.687+0000 7fa82a59c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56546625f6c0 0x56546626e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:25:25.697 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:25.687+0000 7fa833dc8740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x5654662741f0 con 0x56546625f6c0 2024-03-31T23:25:25.697 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:25.687+0000 7fa82a59c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56546625f6c0 0x56546626e590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44950/0 (socket says 172.21.15.46:44950) 2024-03-31T23:25:25.697 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:25.687+0000 7fa833dc8740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x565466160f80 con 0x5654662715c0 2024-03-31T23:25:25.697 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:25.687+0000 7fa82a59c640 1 -- 172.21.15.46:0/1602776567 learned_addr learned my addr 172.21.15.46:0/1602776567 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:25:25.697 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:25.687+0000 7fa833dc8740 1 -- 172.21.15.46:0/1602776567 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x565466274370 con 0x56546625ffe0 2024-03-31T23:25:25.697 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:25.687+0000 7fa82ad9d640 1 --2- 172.21.15.46:0/1602776567 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56546625ffe0 0x56546626ead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:25:25.697 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:25.687+0000 7fa833dc8740 10 client.0 _ll_register_callbacks cb 0x5654661b9290 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T23:25:25.697 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:25.687+0000 7fa82a59c640 1 --2- 172.21.15.46:0/1602776567 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5654662715c0 0x56546626f010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:25:25.697 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:25.687+0000 7fa82ad9d640 1 -- 172.21.15.46:0/1602776567 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5654662715c0 msgr2=0x56546626f010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:25:25.698 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:25.687+0000 7fa82ad9d640 1 --2- 172.21.15.46:0/1602776567 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5654662715c0 0x56546626f010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:25:25.698 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:25.687+0000 7fa82ad9d640 1 -- 172.21.15.46:0/1602776567 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56546625f6c0 msgr2=0x56546626e590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:25:25.698 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:25.687+0000 7fa82ad9d640 1 --2- 172.21.15.46:0/1602776567 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56546625f6c0 0x56546626e590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:25:25.698 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:25.687+0000 7fa82ad9d640 1 -- 172.21.15.46:0/1602776567 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x565465fa3620 con 0x56546625ffe0 2024-03-31T23:25:25.698 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:25.687+0000 7fa82a59c640 1 --2- 172.21.15.46:0/1602776567 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5654662715c0 0x56546626f010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T23:25:25.698 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:25.688+0000 7fa82ad9d640 1 --2- 172.21.15.46:0/1602776567 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56546625ffe0 0x56546626ead0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fa81c00d010 tx=0x7fa81c015b80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=85fe1ed8c5400911 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:25:25.698 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:25.688+0000 7fa8127fc640 10 client.0 ms_handle_connect on v2:172.21.15.162:3300/0 2024-03-31T23:25:25.698 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:25.688+0000 7fa8127fc640 1 -- 172.21.15.46:0/1602776567 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa81c004a40 con 0x56546625ffe0 2024-03-31T23:25:25.698 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:25.688+0000 7fa833dc8740 1 -- 172.21.15.46:0/1602776567 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa81c01b070 con 0x56546625ffe0 2024-03-31T23:25:25.698 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:25.688+0000 7fa8127fc640 1 -- 172.21.15.46:0/1602776567 <== mon.1 v2:172.21.15.162:3300/0 2 ==== osd_map(112..112 src has 1..112) ==== 9554+0+0 (secure 0 0 0) 0x7fa81c004be0 con 0x56546625ffe0 2024-03-31T23:25:25.698 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:25.688+0000 7fa8127fc640 1 -- 172.21.15.46:0/1602776567 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7fa81c004de0 con 0x56546625ffe0 2024-03-31T23:25:25.698 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:25.688+0000 7fa833dc8740 10 client.26989 fetch_fsmap learned FSMap version 38 2024-03-31T23:25:25.698 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:25.688+0000 7fa833dc8740 1 -- 172.21.15.46:0/1602776567 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x565466263230 con 0x56546625ffe0 2024-03-31T23:25:25.699 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:25.688+0000 7fa8127fc640 1 -- 172.21.15.46:0/1602776567 <== mon.1 v2:172.21.15.162:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa81c01b070 con 0x56546625ffe0 2024-03-31T23:25:25.699 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:25.689+0000 7fa8127fc640 1 -- 172.21.15.46:0/1602776567 <== mon.1 v2:172.21.15.162:3300/0 5 ==== fsmap(e 38) ==== 8370+0+0 (secure 0 0 0) 0x7fa81c005050 con 0x56546625ffe0 2024-03-31T23:25:25.699 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:25.689+0000 7fa833dc8740 10 client.26989 fetch_fsmap finished waiting for FSMap version 38 2024-03-31T23:25:25.699 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:25.689+0000 7fa833dc8740 10 client.26989 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-31T23:25:25.699 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:25.689+0000 7fa833dc8740 20 client.26989 populate_metadata read hostname 'smithi046' 2024-03-31T23:25:25.699 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:25.689+0000 7fa833dc8740 1 --2- 172.21.15.46:0/1602776567 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x5654662646a0 0x565466266b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:25:25.699 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:25.689+0000 7fa833dc8740 4 client.26989 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-31T23:25:25.699 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:25.689+0000 7fa833dc8740 1 -- 172.21.15.46:0/1602776567 --> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5654661d6120 con 0x5654662646a0 2024-03-31T23:25:25.699 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:25.689+0000 7fa82a59c640 1 --2- 172.21.15.46:0/1602776567 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x5654662646a0 0x565466266b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:25:25.699 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:25.690+0000 7fa82a59c640 1 --2- 172.21.15.46:0/1602776567 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x5654662646a0 0x565466266b60 crc :-1 s=READY pgs=197 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=935b68c1df09f0c6 server_cookie=7256af7de8ddf1f3 in_seq=0 out_seq=0 2024-03-31T23:25:25.699 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:25.690+0000 7fa8127fc640 10 client.26989 ms_handle_connect on v2:172.21.15.142:6838/2519143918 2024-03-31T23:25:25.699 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:25.691+0000 7fa8127fc640 1 -- 172.21.15.46:0/1602776567 <== mds.0 v2:172.21.15.142:6838/2519143918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5654661d6120 con 0x5654662646a0 2024-03-31T23:25:25.699 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:25.691+0000 7fa8127fc640 10 client.26989 handle_command_reply: tid=0 2024-03-31T23:25:25.700 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:25.691+0000 7fa833dc8740 1 client.26989 shutdown 2024-03-31T23:25:25.700 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:25.691+0000 7fa833dc8740 1 -- 172.21.15.46:0/1602776567 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56546625ffe0 msgr2=0x56546626ead0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:25:25.700 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:25.691+0000 7fa833dc8740 1 --2- 172.21.15.46:0/1602776567 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56546625ffe0 0x56546626ead0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fa81c00d010 tx=0x7fa81c015b80 comp rx=0 tx=0).stop 2024-03-31T23:25:25.700 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:25.691+0000 7fa833dc8740 1 -- 172.21.15.46:0/1602776567 shutdown_connections 2024-03-31T23:25:25.700 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:25.691+0000 7fa833dc8740 1 --2- 172.21.15.46:0/1602776567 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x5654662646a0 0x565466266b60 crc :-1 s=READY pgs=197 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:25:25.700 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:25.691+0000 7fa833dc8740 1 --2- 172.21.15.46:0/1602776567 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5654662715c0 0x56546626f010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:25:25.700 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:25.691+0000 7fa833dc8740 1 --2- 172.21.15.46:0/1602776567 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56546625ffe0 0x56546626ead0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:25:25.700 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:25.691+0000 7fa833dc8740 1 --2- 172.21.15.46:0/1602776567 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56546625f6c0 0x56546626e590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:25:25.700 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:25.691+0000 7fa8127fc640 0 client.26989 ms_handle_reset on v2:172.21.15.142:6838/2519143918 2024-03-31T23:25:25.700 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:25.691+0000 7fa833dc8740 1 -- 172.21.15.46:0/1602776567 >> 172.21.15.46:0/1602776567 conn(0x565466259fe0 msgr2=0x56546625bd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:25:25.700 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:25.691+0000 7fa833dc8740 1 -- 172.21.15.46:0/1602776567 shutdown_connections 2024-03-31T23:25:25.700 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:25.691+0000 7fa833dc8740 1 -- 172.21.15.46:0/1602776567 wait complete. 2024-03-31T23:25:25.700 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:25.691+0000 7fa833dc8740 20 client.26989 trim_cache size 0 max 16384 2024-03-31T23:25:25.716 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:25.714+0000 7fa833dc8740 1 Processor -- start 2024-03-31T23:25:25.716 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:25.714+0000 7fa833dc8740 1 -- start start 2024-03-31T23:25:25.716 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:25.715+0000 7fa833dc8740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56546625f6c0 0x565466270cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:25:25.716 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:25.715+0000 7fa833dc8740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56546625ffe0 0x565466254e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:25:25.716 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:25.715+0000 7fa833dc8740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5654662715c0 0x56546625a3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:25:25.716 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:25.715+0000 7fa833dc8740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fa82c1af310 con 0x56546625ffe0 2024-03-31T23:25:25.716 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:25.715+0000 7fa833dc8740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fa82c1acb20 con 0x5654662715c0 2024-03-31T23:25:25.716 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:25.715+0000 7fa833dc8740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fa82c1aa990 con 0x56546625f6c0 2024-03-31T23:25:25.716 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:25.715+0000 7fa82ad9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56546625ffe0 0x565466254e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:25:25.717 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:25.715+0000 7fa82a59c640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5654662715c0 0x56546625a3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:25:25.717 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:25.715+0000 7fa82ad9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56546625ffe0 0x565466254e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44966/0 (socket says 172.21.15.46:44966) 2024-03-31T23:25:25.717 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:25.715+0000 7fa82ad9d640 1 -- 172.21.15.46:0/2105831987 learned_addr learned my addr 172.21.15.46:0/2105831987 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:25:25.717 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:25.715+0000 7fa82a59c640 1 --2- 172.21.15.46:0/2105831987 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56546625f6c0 0x565466270cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:25:25.721 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:25.715+0000 7fa82a59c640 1 -- 172.21.15.46:0/2105831987 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56546625f6c0 msgr2=0x565466270cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:25:25.721 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:25.715+0000 7fa82a59c640 1 --2- 172.21.15.46:0/2105831987 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56546625f6c0 0x565466270cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:25:25.721 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:25.715+0000 7fa82a59c640 1 -- 172.21.15.46:0/2105831987 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56546625ffe0 msgr2=0x565466254e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:25:25.721 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:25.715+0000 7fa82a59c640 1 --2- 172.21.15.46:0/2105831987 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56546625ffe0 0x565466254e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:25:25.721 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:25.715+0000 7fa82a59c640 1 -- 172.21.15.46:0/2105831987 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56546626d540 con 0x5654662715c0 2024-03-31T23:25:25.721 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:25.715+0000 7fa82ad9d640 1 --2- 172.21.15.46:0/2105831987 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56546625ffe0 0x565466254e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:25:25.721 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:25.716+0000 7fa82a59c640 1 --2- 172.21.15.46:0/2105831987 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5654662715c0 0x56546625a3c0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x565466267260 tx=0x7fa814005a20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=548d8787dd4522a4 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:25:25.721 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:25.716+0000 7fa820ff9640 1 -- 172.21.15.46:0/2105831987 <== mon.2 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa81402c930 con 0x5654662715c0 2024-03-31T23:25:25.721 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:25.716+0000 7fa820ff9640 1 -- 172.21.15.46:0/2105831987 <== mon.2 v2:172.21.15.142:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa8140707e0 con 0x5654662715c0 2024-03-31T23:25:25.722 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:25.716+0000 7fa820ff9640 1 -- 172.21.15.46:0/2105831987 <== mon.2 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa814043940 con 0x5654662715c0 2024-03-31T23:25:25.722 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:25.717+0000 7fa833dc8740 1 -- 172.21.15.46:0/2105831987 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5654662715c0 msgr2=0x56546625a3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:25:25.722 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:25.717+0000 7fa833dc8740 1 --2- 172.21.15.46:0/2105831987 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5654662715c0 0x56546625a3c0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x565466267260 tx=0x7fa814005a20 comp rx=0 tx=0).stop 2024-03-31T23:25:25.722 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:25.717+0000 7fa833dc8740 1 -- 172.21.15.46:0/2105831987 shutdown_connections 2024-03-31T23:25:25.722 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:25.717+0000 7fa833dc8740 1 --2- 172.21.15.46:0/2105831987 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5654662715c0 0x56546625a3c0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:25:25.722 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:25.717+0000 7fa833dc8740 1 --2- 172.21.15.46:0/2105831987 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56546625ffe0 0x565466254e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:25:25.722 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:25.717+0000 7fa833dc8740 1 --2- 172.21.15.46:0/2105831987 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56546625f6c0 0x565466270cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:25:25.722 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:25.717+0000 7fa833dc8740 1 -- 172.21.15.46:0/2105831987 >> 172.21.15.46:0/2105831987 conn(0x565466259fe0 msgr2=0x565466282840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:25:25.722 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:25.717+0000 7fa833dc8740 1 -- 172.21.15.46:0/2105831987 shutdown_connections 2024-03-31T23:25:25.722 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:25.717+0000 7fa833dc8740 1 -- 172.21.15.46:0/2105831987 wait complete. 2024-03-31T23:25:25.722 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:25.718+0000 7fa833dc8740 1 -- start start 2024-03-31T23:25:25.722 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:25.718+0000 7fa833dc8740 1 Processor -- start 2024-03-31T23:25:25.722 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:25.718+0000 7fa833dc8740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56546625f6c0 0x565466277550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:25:25.723 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:25.718+0000 7fa833dc8740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56546625ffe0 0x565466277a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:25:25.723 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:25.718+0000 7fa833dc8740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5654662715c0 0x5654662ddc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:25:25.723 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:25.718+0000 7fa833dc8740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fa82c1aafe0 con 0x5654662715c0 2024-03-31T23:25:25.723 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:25.718+0000 7fa833dc8740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fa82c1ac450 con 0x56546625f6c0 2024-03-31T23:25:25.723 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:25.719+0000 7fa82ad9d640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56546625ffe0 0x565466277a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:25:25.723 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:25.719+0000 7fa82a59c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5654662715c0 0x5654662ddc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:25:25.723 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:25.719+0000 7fa82ad9d640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56546625ffe0 0x565466277a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.46:44200/0 (socket says 172.21.15.46:44200) 2024-03-31T23:25:25.723 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:25.719+0000 7fa82ad9d640 1 -- 172.21.15.46:0/1485333247 learned_addr learned my addr 172.21.15.46:0/1485333247 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:25:25.723 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:25.719+0000 7fa833dc8740 1 -- 172.21.15.46:0/1485333247 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fa82c1ab6b0 con 0x56546625ffe0 2024-03-31T23:25:25.723 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:25.719+0000 7fa82a59c640 1 --2- 172.21.15.46:0/1485333247 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56546625f6c0 0x565466277550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:25:25.723 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:25.719+0000 7fa833dc8740 10 client.0 _ll_register_callbacks cb 0x7fa814009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T23:25:25.723 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:25.719+0000 7fa82a59c640 1 -- 172.21.15.46:0/1485333247 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56546625ffe0 msgr2=0x565466277a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:25:25.723 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:25.719+0000 7fa82a59c640 1 --2- 172.21.15.46:0/1485333247 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56546625ffe0 0x565466277a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:25:25.723 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:25.719+0000 7fa82a59c640 1 -- 172.21.15.46:0/1485333247 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56546625f6c0 msgr2=0x565466277550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:25:25.724 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:25.719+0000 7fa82a59c640 1 --2- 172.21.15.46:0/1485333247 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56546625f6c0 0x565466277550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:25:25.724 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:25.719+0000 7fa82a59c640 1 -- 172.21.15.46:0/1485333247 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5654662de1c0 con 0x5654662715c0 2024-03-31T23:25:25.724 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:25.719+0000 7fa82ad9d640 1 --2- 172.21.15.46:0/1485333247 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56546625ffe0 0x565466277a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:25:25.724 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:25.719+0000 7fa82a59c640 1 --2- 172.21.15.46:0/1485333247 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5654662715c0 0x5654662ddc80 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x565466267260 tx=0x7fa8140045e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=abbce498cd96daee server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:25:25.724 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:25.719+0000 7fa8127fc640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-31T23:25:25.724 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:25.720+0000 7fa8127fc640 1 -- 172.21.15.46:0/1485333247 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa81402cab0 con 0x5654662715c0 2024-03-31T23:25:25.724 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:25.720+0000 7fa8127fc640 1 -- 172.21.15.46:0/1485333247 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(112..112 src has 1..112) ==== 9554+0+0 (secure 0 0 0) 0x7fa81407c480 con 0x5654662715c0 2024-03-31T23:25:25.724 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:25.720+0000 7fa833dc8740 1 -- 172.21.15.46:0/1485333247 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa8140707e0 con 0x5654662715c0 2024-03-31T23:25:25.724 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:25.720+0000 7fa8127fc640 1 -- 172.21.15.46:0/1485333247 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7fa81402cc50 con 0x5654662715c0 2024-03-31T23:25:25.724 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:25.720+0000 7fa833dc8740 10 client.17265 fetch_fsmap learned FSMap version 38 2024-03-31T23:25:25.724 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:25.720+0000 7fa833dc8740 1 -- 172.21.15.46:0/1485333247 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x5654662df3d0 con 0x5654662715c0 2024-03-31T23:25:25.724 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:25.720+0000 7fa8127fc640 1 -- 172.21.15.46:0/1485333247 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa8140707e0 con 0x5654662715c0 2024-03-31T23:25:25.725 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:25.720+0000 7fa8127fc640 1 -- 172.21.15.46:0/1485333247 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 38) ==== 8370+0+0 (secure 0 0 0) 0x7fa81407a070 con 0x5654662715c0 2024-03-31T23:25:25.725 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:25.720+0000 7fa833dc8740 10 client.17265 fetch_fsmap finished waiting for FSMap version 38 2024-03-31T23:25:25.726 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:25.720+0000 7fa833dc8740 10 client.17265 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-31T23:25:25.726 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:25.720+0000 7fa833dc8740 20 client.17265 populate_metadata read hostname 'smithi046' 2024-03-31T23:25:25.726 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:25.720+0000 7fa833dc8740 1 --2- 172.21.15.46:0/1485333247 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x5654662df940 0x5654662e1d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:25:25.726 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:25.720+0000 7fa833dc8740 4 client.17265 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-31T23:25:25.726 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:25.720+0000 7fa833dc8740 1 -- 172.21.15.46:0/1485333247 --> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5654661cfe30 con 0x5654662df940 2024-03-31T23:25:25.726 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:25.721+0000 7fa82ad9d640 1 --2- 172.21.15.46:0/1485333247 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x5654662df940 0x5654662e1d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:25:25.726 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:25.722+0000 7fa82ad9d640 1 --2- 172.21.15.46:0/1485333247 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x5654662df940 0x5654662e1d30 crc :-1 s=READY pgs=198 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=277fe1d9db8a412f server_cookie=64a5afdfe0e5e496 in_seq=0 out_seq=0 2024-03-31T23:25:25.726 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:25.722+0000 7fa8127fc640 10 client.17265 ms_handle_connect on v2:172.21.15.142:6838/2519143918 2024-03-31T23:25:25.800 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:25.798+0000 7fa8127fc640 1 -- 172.21.15.46:0/1485333247 <== mds.0 v2:172.21.15.142:6838/2519143918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5654661cfe30 con 0x5654662df940 2024-03-31T23:25:25.800 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:25.798+0000 7fa8127fc640 10 client.17265 handle_command_reply: tid=0 2024-03-31T23:25:25.800 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:25.798+0000 7fa833dc8740 1 client.17265 shutdown 2024-03-31T23:25:25.801 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:25.798+0000 7fa833dc8740 1 -- 172.21.15.46:0/1485333247 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5654662715c0 msgr2=0x5654662ddc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:25:25.801 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:25.798+0000 7fa833dc8740 1 --2- 172.21.15.46:0/1485333247 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5654662715c0 0x5654662ddc80 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x565466267260 tx=0x7fa8140045e0 comp rx=0 tx=0).stop 2024-03-31T23:25:25.801 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:25.798+0000 7fa833dc8740 1 -- 172.21.15.46:0/1485333247 shutdown_connections 2024-03-31T23:25:25.801 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:25.798+0000 7fa833dc8740 1 --2- 172.21.15.46:0/1485333247 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x5654662df940 0x5654662e1d30 crc :-1 s=READY pgs=198 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:25:25.801 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:25.799+0000 7fa8127fc640 0 client.17265 ms_handle_reset on v2:172.21.15.142:6838/2519143918 2024-03-31T23:25:25.801 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:25.799+0000 7fa833dc8740 1 --2- 172.21.15.46:0/1485333247 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5654662715c0 0x5654662ddc80 unknown :-1 s=CLOSED pgs=878 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:25:25.801 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:25.799+0000 7fa833dc8740 1 --2- 172.21.15.46:0/1485333247 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56546625ffe0 0x565466277a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:25:25.801 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:25.799+0000 7fa833dc8740 1 --2- 172.21.15.46:0/1485333247 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56546625f6c0 0x565466277550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:25:25.801 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:25.799+0000 7fa833dc8740 1 -- 172.21.15.46:0/1485333247 >> 172.21.15.46:0/1485333247 conn(0x565466259fe0 msgr2=0x565466282840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:25:25.801 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:25.799+0000 7fa833dc8740 1 -- 172.21.15.46:0/1485333247 shutdown_connections 2024-03-31T23:25:25.801 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:25.799+0000 7fa833dc8740 1 -- 172.21.15.46:0/1485333247 wait complete. 2024-03-31T23:25:25.801 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:25.799+0000 7fa833dc8740 20 client.17265 trim_cache size 0 max 16384 2024-03-31T23:25:25.802 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout:{ 2024-03-31T23:25:25.802 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout: "message": "", 2024-03-31T23:25:25.802 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout: "return_code": 0 2024-03-31T23:25:25.802 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout:} 2024-03-31T23:25:25.803 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:25.800+0000 7fa8311e5640 1 -- 172.21.15.46:0/3767913591 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fa8180a26b0 msgr2=0x7fa8180a4b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:25:25.803 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:25.800+0000 7fa8311e5640 1 --2- 172.21.15.46:0/3767913591 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fa8180a26b0 0x7fa8180a4b70 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fa8240cbde0 tx=0x7fa824017eb0 comp rx=0 tx=0).stop 2024-03-31T23:25:25.803 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:25.800+0000 7fa8311e5640 1 -- 172.21.15.46:0/3767913591 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa82c1055f0 msgr2=0x7fa82c19e9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:25:25.803 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:25.800+0000 7fa8311e5640 1 --2- 172.21.15.46:0/3767913591 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa82c1055f0 0x7fa82c19e9d0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fa82400d590 tx=0x7fa82400da50 comp rx=0 tx=0).stop 2024-03-31T23:25:25.803 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:25.800+0000 7fa8311e5640 1 -- 172.21.15.46:0/3767913591 shutdown_connections 2024-03-31T23:25:25.803 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:25.800+0000 7fa8311e5640 1 --2- 172.21.15.46:0/3767913591 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fa8180a26b0 0x7fa8180a4b70 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:25:25.803 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:25.800+0000 7fa8311e5640 1 --2- 172.21.15.46:0/3767913591 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa82c1055f0 0x7fa82c19e9d0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:25:25.804 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:25.800+0000 7fa8311e5640 1 --2- 172.21.15.46:0/3767913591 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa82c101260 0x7fa82c19a650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:25:25.804 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:25.800+0000 7fa8311e5640 1 --2- 172.21.15.46:0/3767913591 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa82c1008a0 0x7fa82c19a110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:25:25.804 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:25.800+0000 7fa8311e5640 1 -- 172.21.15.46:0/3767913591 >> 172.21.15.46:0/3767913591 conn(0x7fa82c0774a0 msgr2=0x7fa82c0ff5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:25:25.804 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:25.800+0000 7fa8311e5640 1 -- 172.21.15.46:0/3767913591 shutdown_connections 2024-03-31T23:25:25.804 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:25.800+0000 7fa8311e5640 1 -- 172.21.15.46:0/3767913591 wait complete. 2024-03-31T23:25:26.266 INFO:tasks.workunit.client.0.smithi046.stdout: 10 4560 13.60 MB/sec warmup 31 sec latency 91.959 ms 2024-03-31T23:25:26.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:25:26 smithi046 ceph-mon[32498]: pgmap v430: 129 pgs: 129 active+clean; 395 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 6.2 MiB/s wr, 339 op/s 2024-03-31T23:25:26.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:25:26 smithi142 ceph-mon[35271]: pgmap v430: 129 pgs: 129 active+clean; 395 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 6.2 MiB/s wr, 339 op/s 2024-03-31T23:25:27.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:25:26 smithi162 ceph-mon[36237]: pgmap v430: 129 pgs: 129 active+clean; 395 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 6.2 MiB/s wr, 339 op/s 2024-03-31T23:25:27.266 INFO:tasks.workunit.client.0.smithi046.stdout: 10 4715 13.29 MB/sec warmup 32 sec latency 75.730 ms 2024-03-31T23:25:28.266 INFO:tasks.workunit.client.0.smithi046.stdout: 10 4891 13.20 MB/sec warmup 33 sec latency 59.508 ms 2024-03-31T23:25:28.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:25:28 smithi046 ceph-mon[32498]: pgmap v431: 129 pgs: 129 active+clean; 395 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 5.0 MiB/s wr, 169 op/s 2024-03-31T23:25:28.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:25:28 smithi142 ceph-mon[35271]: pgmap v431: 129 pgs: 129 active+clean; 395 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 5.0 MiB/s wr, 169 op/s 2024-03-31T23:25:29.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:25:28 smithi162 ceph-mon[36237]: pgmap v431: 129 pgs: 129 active+clean; 395 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 5.0 MiB/s wr, 169 op/s 2024-03-31T23:25:29.266 INFO:tasks.workunit.client.0.smithi046.stdout: 10 5065 13.02 MB/sec warmup 34 sec latency 62.189 ms 2024-03-31T23:25:30.266 INFO:tasks.workunit.client.0.smithi046.stdout: 10 5256 12.90 MB/sec warmup 35 sec latency 74.192 ms 2024-03-31T23:25:30.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:25:30 smithi046 ceph-mon[32498]: pgmap v432: 129 pgs: 129 active+clean; 384 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 7.0 MiB/s wr, 307 op/s 2024-03-31T23:25:30.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:25:30 smithi142 ceph-mon[35271]: pgmap v432: 129 pgs: 129 active+clean; 384 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 7.0 MiB/s wr, 307 op/s 2024-03-31T23:25:31.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:25:30 smithi162 ceph-mon[36237]: pgmap v432: 129 pgs: 129 active+clean; 384 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 7.0 MiB/s wr, 307 op/s 2024-03-31T23:25:31.266 INFO:tasks.workunit.client.0.smithi046.stdout: 10 5438 12.70 MB/sec warmup 36 sec latency 69.216 ms 2024-03-31T23:25:32.266 INFO:tasks.workunit.client.0.smithi046.stdout: 10 5599 12.50 MB/sec warmup 37 sec latency 87.294 ms 2024-03-31T23:25:32.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:25:32 smithi046 ceph-mon[32498]: pgmap v433: 129 pgs: 129 active+clean; 384 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 4.3 MiB/s wr, 225 op/s 2024-03-31T23:25:32.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:25:32 smithi142 ceph-mon[35271]: pgmap v433: 129 pgs: 129 active+clean; 384 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 4.3 MiB/s wr, 225 op/s 2024-03-31T23:25:33.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:25:32 smithi162 ceph-mon[36237]: pgmap v433: 129 pgs: 129 active+clean; 384 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 4.3 MiB/s wr, 225 op/s 2024-03-31T23:25:33.266 INFO:tasks.workunit.client.0.smithi046.stdout: 10 5785 12.54 MB/sec warmup 38 sec latency 65.864 ms 2024-03-31T23:25:34.266 INFO:tasks.workunit.client.0.smithi046.stdout: 10 5996 12.43 MB/sec warmup 39 sec latency 63.303 ms 2024-03-31T23:25:34.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:25:34 smithi046 ceph-mon[32498]: pgmap v434: 129 pgs: 129 active+clean; 384 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 4.3 MiB/s wr, 225 op/s 2024-03-31T23:25:34.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:25:34 smithi142 ceph-mon[35271]: pgmap v434: 129 pgs: 129 active+clean; 384 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 4.3 MiB/s wr, 225 op/s 2024-03-31T23:25:35.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:25:34 smithi162 ceph-mon[36237]: pgmap v434: 129 pgs: 129 active+clean; 384 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 4.3 MiB/s wr, 225 op/s 2024-03-31T23:25:35.266 INFO:tasks.workunit.client.0.smithi046.stdout: 10 6195 12.61 MB/sec warmup 40 sec latency 56.475 ms 2024-03-31T23:25:36.266 INFO:tasks.workunit.client.0.smithi046.stdout: 10 6355 12.45 MB/sec warmup 41 sec latency 87.834 ms 2024-03-31T23:25:36.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:25:36 smithi046 ceph-mon[32498]: pgmap v435: 129 pgs: 129 active+clean; 392 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 6.1 MiB/s wr, 325 op/s 2024-03-31T23:25:36.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:25:36 smithi142 ceph-mon[35271]: pgmap v435: 129 pgs: 129 active+clean; 392 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 6.1 MiB/s wr, 325 op/s 2024-03-31T23:25:37.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:25:36 smithi162 ceph-mon[36237]: pgmap v435: 129 pgs: 129 active+clean; 392 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 6.1 MiB/s wr, 325 op/s 2024-03-31T23:25:37.266 INFO:tasks.workunit.client.0.smithi046.stdout: 10 6531 12.43 MB/sec warmup 42 sec latency 63.043 ms 2024-03-31T23:25:38.266 INFO:tasks.workunit.client.0.smithi046.stdout: 10 6721 12.44 MB/sec warmup 43 sec latency 63.480 ms 2024-03-31T23:25:38.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:25:38 smithi046 ceph-mon[32498]: pgmap v436: 129 pgs: 129 active+clean; 392 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 3.9 MiB/s wr, 238 op/s 2024-03-31T23:25:38.931 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:38.930+0000 7ffa11725640 1 -- 172.21.15.46:0/4203612136 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffa0c100d60 msgr2=0x7ffa0c1011e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:25:38.931 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:38.930+0000 7ffa11725640 1 --2- 172.21.15.46:0/4203612136 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffa0c100d60 0x7ffa0c1011e0 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7ffa00009930 tx=0x7ffa0002bc30 comp rx=0 tx=0).stop 2024-03-31T23:25:38.931 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:38.930+0000 7ffa11725640 1 -- 172.21.15.46:0/4203612136 shutdown_connections 2024-03-31T23:25:38.931 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:38.930+0000 7ffa11725640 1 --2- 172.21.15.46:0/4203612136 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ffa0c101720 0x7ffa0c10d710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:25:38.931 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:38.930+0000 7ffa11725640 1 --2- 172.21.15.46:0/4203612136 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffa0c100d60 0x7ffa0c1011e0 unknown :-1 s=CLOSED pgs=879 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:25:38.931 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:38.930+0000 7ffa11725640 1 --2- 172.21.15.46:0/4203612136 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ffa0c106850 0x7ffa0c106c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:25:38.931 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:38.930+0000 7ffa11725640 1 -- 172.21.15.46:0/4203612136 >> 172.21.15.46:0/4203612136 conn(0x7ffa0c076c10 msgr2=0x7ffa0c0fdf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:25:38.931 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:38.930+0000 7ffa11725640 1 -- 172.21.15.46:0/4203612136 shutdown_connections 2024-03-31T23:25:38.931 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:38.930+0000 7ffa11725640 1 -- 172.21.15.46:0/4203612136 wait complete. 2024-03-31T23:25:38.931 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:38.931+0000 7ffa11725640 1 Processor -- start 2024-03-31T23:25:38.931 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:38.931+0000 7ffa11725640 1 -- start start 2024-03-31T23:25:38.933 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:38.931+0000 7ffa11725640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ffa0c100d60 0x7ffa0c1a4420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:25:38.933 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:38.931+0000 7ffa11725640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ffa0c101720 0x7ffa0c1a4960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:25:38.933 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:38.931+0000 7ffa11725640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffa0c106850 0x7ffa0c19e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:25:38.934 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:38.931+0000 7ffa11725640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7ffa0c110ad0 con 0x7ffa0c106850 2024-03-31T23:25:38.934 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:38.931+0000 7ffa11725640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7ffa0c110950 con 0x7ffa0c100d60 2024-03-31T23:25:38.934 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:38.931+0000 7ffa11725640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7ffa0c110c50 con 0x7ffa0c101720 2024-03-31T23:25:38.934 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:38.931+0000 7ffa0a7fc640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ffa0c101720 0x7ffa0c1a4960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:25:38.934 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:38.931+0000 7ffa0a7fc640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ffa0c101720 0x7ffa0c1a4960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.46:43396/0 (socket says 172.21.15.46:43396) 2024-03-31T23:25:38.934 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:38.931+0000 7ffa0a7fc640 1 -- 172.21.15.46:0/200947113 learned_addr learned my addr 172.21.15.46:0/200947113 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:25:38.934 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:38.931+0000 7ffa0b7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffa0c106850 0x7ffa0c19e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:25:38.934 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:38.931+0000 7ffa0affd640 1 --2- 172.21.15.46:0/200947113 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ffa0c100d60 0x7ffa0c1a4420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:25:38.934 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:38.932+0000 7ffa0affd640 1 -- 172.21.15.46:0/200947113 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ffa0c101720 msgr2=0x7ffa0c1a4960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:25:38.934 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:38.932+0000 7ffa0affd640 1 --2- 172.21.15.46:0/200947113 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ffa0c101720 0x7ffa0c1a4960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:25:38.934 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:38.932+0000 7ffa0affd640 1 -- 172.21.15.46:0/200947113 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffa0c106850 msgr2=0x7ffa0c19e770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:25:38.934 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:38.932+0000 7ffa0affd640 1 --2- 172.21.15.46:0/200947113 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffa0c106850 0x7ffa0c19e770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:25:38.935 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:38.932+0000 7ffa0affd640 1 -- 172.21.15.46:0/200947113 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa0c19ef40 con 0x7ffa0c100d60 2024-03-31T23:25:38.935 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:38.932+0000 7ffa0a7fc640 1 --2- 172.21.15.46:0/200947113 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ffa0c101720 0x7ffa0c1a4960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:25:38.935 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:38.932+0000 7ffa0b7fe640 1 --2- 172.21.15.46:0/200947113 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffa0c106850 0x7ffa0c19e770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:25:38.935 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:38.932+0000 7ffa0affd640 1 --2- 172.21.15.46:0/200947113 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ffa0c100d60 0x7ffa0c1a4420 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7ff9fc00e9f0 tx=0x7ff9fc00eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:25:38.935 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:38.932+0000 7ff9f3fff640 1 -- 172.21.15.46:0/200947113 <== mon.2 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9fc00ccc0 con 0x7ffa0c100d60 2024-03-31T23:25:38.935 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:38.932+0000 7ffa11725640 1 -- 172.21.15.46:0/200947113 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffa0c19f230 con 0x7ffa0c100d60 2024-03-31T23:25:38.935 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:38.932+0000 7ff9f3fff640 1 -- 172.21.15.46:0/200947113 <== mon.2 v2:172.21.15.142:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff9fc0108e0 con 0x7ffa0c100d60 2024-03-31T23:25:38.935 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:38.932+0000 7ffa11725640 1 -- 172.21.15.46:0/200947113 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffa0c1ab410 con 0x7ffa0c100d60 2024-03-31T23:25:38.935 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:38.932+0000 7ff9f3fff640 1 -- 172.21.15.46:0/200947113 <== mon.2 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9fc018880 con 0x7ffa0c100d60 2024-03-31T23:25:38.937 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:38.935+0000 7ff9f3fff640 1 -- 172.21.15.46:0/200947113 <== mon.2 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 25) ==== 137451+0+0 (secure 0 0 0) 0x7ff9fc010a90 con 0x7ffa0c100d60 2024-03-31T23:25:38.937 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:38.936+0000 7ffa143e4740 1 Processor -- start 2024-03-31T23:25:38.937 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:38.936+0000 7ffa143e4740 1 -- start start 2024-03-31T23:25:38.937 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:38.936+0000 7ffa143e4740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d87340b6c0 0x55d87340baa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:25:38.938 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:38.936+0000 7ffa143e4740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d87340bfe0 0x55d873414870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:25:38.938 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:38.936+0000 7ffa143e4740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d873414db0 0x55d8734171a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:25:38.938 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:38.936+0000 7ffa143e4740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55d87336e8f0 con 0x55d873414db0 2024-03-31T23:25:38.938 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:38.936+0000 7ffa143e4740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55d8733178d0 con 0x55d87340bfe0 2024-03-31T23:25:38.938 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:38.936+0000 7ffa143e4740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55d87336a990 con 0x55d87340b6c0 2024-03-31T23:25:38.938 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:38.936+0000 7ffa0b7fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d87340b6c0 0x55d87340baa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:25:38.938 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:38.936+0000 7ffa0affd640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d87340bfe0 0x55d873414870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:25:38.938 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:38.936+0000 7ffa0a7fc640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d873414db0 0x55d8734171a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:25:38.938 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:38.936+0000 7ffa0b7fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d87340b6c0 0x55d87340baa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.46:43408/0 (socket says 172.21.15.46:43408) 2024-03-31T23:25:38.938 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:38.936+0000 7ffa0b7fe640 1 -- 172.21.15.46:0/1149694653 learned_addr learned my addr 172.21.15.46:0/1149694653 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:25:38.938 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:38.937+0000 7ff9f3fff640 1 --2- 172.21.15.46:0/200947113 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7ff9f40a2730 0x7ff9f40a4bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:25:38.938 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:38.937+0000 7ffa0a7fc640 1 -- 172.21.15.46:0/1149694653 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d87340b6c0 msgr2=0x55d87340baa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:25:38.938 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:38.937+0000 7ffa0a7fc640 1 --2- 172.21.15.46:0/1149694653 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d87340b6c0 0x55d87340baa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:25:38.939 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:38.937+0000 7ffa0a7fc640 1 -- 172.21.15.46:0/1149694653 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d87340bfe0 msgr2=0x55d873414870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:25:38.939 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:38.937+0000 7ffa0a7fc640 1 --2- 172.21.15.46:0/1149694653 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d87340bfe0 0x55d873414870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:25:38.939 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:38.937+0000 7ffa0a7fc640 1 -- 172.21.15.46:0/1149694653 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d873083210 con 0x55d873414db0 2024-03-31T23:25:38.939 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:38.937+0000 7ffa0b7fe640 1 --2- 172.21.15.46:0/1149694653 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d87340b6c0 0x55d87340baa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:25:38.939 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:38.937+0000 7ff9f3fff640 1 -- 172.21.15.46:0/200947113 <== mon.2 v2:172.21.15.142:3300/0 5 ==== osd_map(112..112 src has 1..112) ==== 9554+0+0 (secure 0 0 0) 0x7ff9fc0cf160 con 0x7ffa0c100d60 2024-03-31T23:25:38.939 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:38.937+0000 7ffa0b7fe640 1 --2- 172.21.15.46:0/200947113 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7ff9f40a2730 0x7ff9f40a4bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:25:38.939 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:38.937+0000 7ffa0a7fc640 1 --2- 172.21.15.46:0/1149694653 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d873414db0 0x55d8734171a0 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7ffa00009900 tx=0x7ffa00009670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=34e1b42ec872dc3 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:25:38.939 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:38.937+0000 7ff9f17fa640 1 -- 172.21.15.46:0/1149694653 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa00045070 con 0x55d873414db0 2024-03-31T23:25:38.939 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:38.937+0000 7ff9f17fa640 1 -- 172.21.15.46:0/1149694653 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ffa0002c7e0 con 0x55d873414db0 2024-03-31T23:25:38.939 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:38.937+0000 7ff9f17fa640 1 -- 172.21.15.46:0/1149694653 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa00003b50 con 0x55d873414db0 2024-03-31T23:25:38.943 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:38.938+0000 7ffa143e4740 1 -- 172.21.15.46:0/1149694653 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d873414db0 msgr2=0x55d8734171a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:25:38.943 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:38.938+0000 7ffa143e4740 1 --2- 172.21.15.46:0/1149694653 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d873414db0 0x55d8734171a0 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7ffa00009900 tx=0x7ffa00009670 comp rx=0 tx=0).stop 2024-03-31T23:25:38.943 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:38.938+0000 7ffa0b7fe640 1 --2- 172.21.15.46:0/200947113 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7ff9f40a2730 0x7ff9f40a4bf0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7ffa04003f50 tx=0x7ffa04004a40 comp rx=0 tx=0).ready entity=mgr.15654 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:25:38.943 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:38.938+0000 7ffa143e4740 1 -- 172.21.15.46:0/1149694653 shutdown_connections 2024-03-31T23:25:38.943 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:38.938+0000 7ffa143e4740 1 --2- 172.21.15.46:0/1149694653 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d873414db0 0x55d8734171a0 unknown :-1 s=CLOSED pgs=880 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:25:38.943 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:38.938+0000 7ffa143e4740 1 --2- 172.21.15.46:0/1149694653 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d87340bfe0 0x55d873414870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:25:38.943 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:38.938+0000 7ffa143e4740 1 --2- 172.21.15.46:0/1149694653 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d87340b6c0 0x55d87340baa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:25:38.943 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:38.938+0000 7ffa143e4740 1 -- 172.21.15.46:0/1149694653 >> 172.21.15.46:0/1149694653 conn(0x55d873405fe0 msgr2=0x55d8734063c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:25:38.943 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:38.938+0000 7ffa143e4740 1 -- 172.21.15.46:0/1149694653 shutdown_connections 2024-03-31T23:25:38.944 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:38.938+0000 7ffa143e4740 1 -- 172.21.15.46:0/1149694653 wait complete. 2024-03-31T23:25:38.944 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:38.939+0000 7ffa143e4740 1 -- start start 2024-03-31T23:25:38.944 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:38.939+0000 7ffa143e4740 1 Processor -- start 2024-03-31T23:25:38.944 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:38.939+0000 7ffa143e4740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d87340b6c0 0x55d873411e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:25:38.944 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:38.939+0000 7ffa143e4740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d87340bfe0 0x55d873412370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:25:38.944 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:38.939+0000 7ffa143e4740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d873414db0 0x55d8734128b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:25:38.944 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:38.939+0000 7ffa143e4740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55d873417980 con 0x55d87340b6c0 2024-03-31T23:25:38.944 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:38.939+0000 7ffa143e4740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55d87330cf80 con 0x55d87340bfe0 2024-03-31T23:25:38.944 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:38.939+0000 7ffa143e4740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55d873417b00 con 0x55d873414db0 2024-03-31T23:25:38.944 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:38.939+0000 7ffa0affd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d87340b6c0 0x55d873411e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:25:38.944 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:38.939+0000 7ffa143e4740 10 client.0 _ll_register_callbacks cb 0x55d873365290 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T23:25:38.944 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:38.939+0000 7ffa0affd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d87340b6c0 0x55d873411e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39706/0 (socket says 172.21.15.46:39706) 2024-03-31T23:25:38.944 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:38.939+0000 7ffa0affd640 1 -- 172.21.15.46:0/1992747400 learned_addr learned my addr 172.21.15.46:0/1992747400 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:25:38.944 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:38.939+0000 7ffa0a7fc640 1 --2- 172.21.15.46:0/1992747400 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d87340bfe0 0x55d873412370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:25:38.945 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:38.940+0000 7ffa0b7fe640 1 --2- 172.21.15.46:0/1992747400 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d873414db0 0x55d8734128b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:25:38.945 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:38.940+0000 7ffa0b7fe640 1 -- 172.21.15.46:0/1992747400 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d87340bfe0 msgr2=0x55d873412370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:25:38.945 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:38.940+0000 7ffa0b7fe640 1 --2- 172.21.15.46:0/1992747400 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d87340bfe0 0x55d873412370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:25:38.945 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:38.940+0000 7ffa0b7fe640 1 -- 172.21.15.46:0/1992747400 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d87340b6c0 msgr2=0x55d873411e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:25:38.945 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:38.940+0000 7ffa0b7fe640 1 --2- 172.21.15.46:0/1992747400 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d87340b6c0 0x55d873411e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:25:38.945 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:38.940+0000 7ffa0b7fe640 1 -- 172.21.15.46:0/1992747400 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d87314f620 con 0x55d873414db0 2024-03-31T23:25:38.945 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:38.940+0000 7ffa0affd640 1 --2- 172.21.15.46:0/1992747400 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d87340b6c0 0x55d873411e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:25:38.945 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:38.940+0000 7ffa0a7fc640 1 --2- 172.21.15.46:0/1992747400 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d87340bfe0 0x55d873412370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:25:38.945 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:38.940+0000 7ffa0b7fe640 1 --2- 172.21.15.46:0/1992747400 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d873414db0 0x55d8734128b0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7ffa04004940 tx=0x7ffa04004ab0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9f28d8cb68c13c20 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:25:38.945 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:38.940+0000 7ff9eeff5640 10 client.0 ms_handle_connect on v2:172.21.15.162:3300/0 2024-03-31T23:25:38.945 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:38.940+0000 7ff9eeff5640 1 -- 172.21.15.46:0/1992747400 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa04003c10 con 0x55d873414db0 2024-03-31T23:25:38.945 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:38.941+0000 7ffa143e4740 1 -- 172.21.15.46:0/1992747400 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ffa0002c7e0 con 0x55d873414db0 2024-03-31T23:25:38.945 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:38.941+0000 7ff9eeff5640 1 -- 172.21.15.46:0/1992747400 <== mon.1 v2:172.21.15.162:3300/0 2 ==== osd_map(112..112 src has 1..112) ==== 9554+0+0 (secure 0 0 0) 0x7ffa04004d20 con 0x55d873414db0 2024-03-31T23:25:38.945 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:38.941+0000 7ff9eeff5640 1 -- 172.21.15.46:0/1992747400 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7ffa04033050 con 0x55d873414db0 2024-03-31T23:25:38.945 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:38.941+0000 7ffa143e4740 10 client.27007 fetch_fsmap learned FSMap version 38 2024-03-31T23:25:38.946 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:38.941+0000 7ffa143e4740 1 -- 172.21.15.46:0/1992747400 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x55d873423950 con 0x55d873414db0 2024-03-31T23:25:38.946 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:38.942+0000 7ff9eeff5640 1 -- 172.21.15.46:0/1992747400 <== mon.1 v2:172.21.15.162:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ffa0002c7e0 con 0x55d873414db0 2024-03-31T23:25:38.946 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:38.942+0000 7ff9eeff5640 1 -- 172.21.15.46:0/1992747400 <== mon.1 v2:172.21.15.162:3300/0 5 ==== fsmap(e 38) ==== 8370+0+0 (secure 0 0 0) 0x7ffa04005130 con 0x55d873414db0 2024-03-31T23:25:38.946 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:38.942+0000 7ffa143e4740 10 client.27007 fetch_fsmap finished waiting for FSMap version 38 2024-03-31T23:25:38.946 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:38.942+0000 7ffa143e4740 10 client.27007 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-31T23:25:38.946 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:38.942+0000 7ffa143e4740 20 client.27007 populate_metadata read hostname 'smithi046' 2024-03-31T23:25:38.946 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:38.942+0000 7ffa143e4740 1 --2- 172.21.15.46:0/1992747400 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55d873424e10 0x55d873427260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:25:38.946 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:38.942+0000 7ffa143e4740 4 client.27007 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-31T23:25:38.946 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:38.942+0000 7ffa143e4740 1 -- 172.21.15.46:0/1992747400 --> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d873382120 con 0x55d873424e10 2024-03-31T23:25:38.946 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:38.943+0000 7ffa0affd640 1 --2- 172.21.15.46:0/1992747400 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55d873424e10 0x55d873427260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:25:38.946 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:38.943+0000 7ff9eeff5640 10 client.27007 ms_handle_connect on v2:172.21.15.142:6838/2519143918 2024-03-31T23:25:38.946 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:38.943+0000 7ffa0affd640 1 --2- 172.21.15.46:0/1992747400 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55d873424e10 0x55d873427260 crc :-1 s=READY pgs=199 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=78d6e519655c0a11 server_cookie=34befbeb7a1f0dfa in_seq=0 out_seq=0 2024-03-31T23:25:38.947 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:25:38 smithi142 ceph-mon[35271]: pgmap v436: 129 pgs: 129 active+clean; 392 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 3.9 MiB/s wr, 238 op/s 2024-03-31T23:25:38.948 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:38.944+0000 7ff9eeff5640 1 -- 172.21.15.46:0/1992747400 <== mds.0 v2:172.21.15.142:6838/2519143918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55d873382120 con 0x55d873424e10 2024-03-31T23:25:38.948 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:38.944+0000 7ff9eeff5640 10 client.27007 handle_command_reply: tid=0 2024-03-31T23:25:38.948 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:38.944+0000 7ffa143e4740 1 client.27007 shutdown 2024-03-31T23:25:38.948 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:38.944+0000 7ffa143e4740 1 -- 172.21.15.46:0/1992747400 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d873414db0 msgr2=0x55d8734128b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:25:38.948 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:38.944+0000 7ffa143e4740 1 --2- 172.21.15.46:0/1992747400 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d873414db0 0x55d8734128b0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7ffa04004940 tx=0x7ffa04004ab0 comp rx=0 tx=0).stop 2024-03-31T23:25:38.948 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:38.944+0000 7ffa143e4740 1 -- 172.21.15.46:0/1992747400 shutdown_connections 2024-03-31T23:25:38.948 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:38.945+0000 7ffa143e4740 1 --2- 172.21.15.46:0/1992747400 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55d873424e10 0x55d873427260 crc :-1 s=READY pgs=199 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:25:38.948 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:38.945+0000 7ffa143e4740 1 --2- 172.21.15.46:0/1992747400 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d873414db0 0x55d8734128b0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:25:38.948 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:38.945+0000 7ffa143e4740 1 --2- 172.21.15.46:0/1992747400 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d87340bfe0 0x55d873412370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:25:38.949 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:38.945+0000 7ff9eeff5640 0 client.27007 ms_handle_reset on v2:172.21.15.142:6838/2519143918 2024-03-31T23:25:38.949 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:38.945+0000 7ffa143e4740 1 --2- 172.21.15.46:0/1992747400 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d87340b6c0 0x55d873411e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:25:38.949 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:38.945+0000 7ffa143e4740 1 -- 172.21.15.46:0/1992747400 >> 172.21.15.46:0/1992747400 conn(0x55d873405fe0 msgr2=0x55d873407d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:25:38.949 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:38.945+0000 7ffa143e4740 1 -- 172.21.15.46:0/1992747400 shutdown_connections 2024-03-31T23:25:38.949 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:38.945+0000 7ffa143e4740 1 -- 172.21.15.46:0/1992747400 wait complete. 2024-03-31T23:25:38.949 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:38.945+0000 7ffa143e4740 20 client.27007 trim_cache size 0 max 16384 2024-03-31T23:25:38.971 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:38.970+0000 7ffa143e4740 1 Processor -- start 2024-03-31T23:25:38.971 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:38.970+0000 7ffa143e4740 1 -- start start 2024-03-31T23:25:38.971 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:38.970+0000 7ffa143e4740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d87340b6c0 0x55d873414560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:25:38.972 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:38.970+0000 7ffa143e4740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d87340bfe0 0x55d873420720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:25:38.972 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:38.970+0000 7ffa143e4740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d873414db0 0x55d8734063c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:25:38.972 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:38.970+0000 7ffa143e4740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7ffa0c1b7c60 con 0x55d873414db0 2024-03-31T23:25:38.972 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:38.970+0000 7ffa143e4740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7ffa0c1b5470 con 0x55d87340bfe0 2024-03-31T23:25:38.972 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:38.970+0000 7ffa143e4740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7ffa0c1b32e0 con 0x55d87340b6c0 2024-03-31T23:25:38.972 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:38.970+0000 7ffa0b7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d873414db0 0x55d8734063c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:25:38.972 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:38.970+0000 7ffa0affd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d87340b6c0 0x55d873414560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:25:38.972 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:38.970+0000 7ffa0b7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d873414db0 0x55d8734063c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39720/0 (socket says 172.21.15.46:39720) 2024-03-31T23:25:38.972 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:38.970+0000 7ffa0b7fe640 1 -- 172.21.15.46:0/1502254194 learned_addr learned my addr 172.21.15.46:0/1502254194 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:25:38.972 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:38.971+0000 7ffa0a7fc640 1 --2- 172.21.15.46:0/1502254194 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d87340bfe0 0x55d873420720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:25:38.972 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:38.971+0000 7ffa0affd640 1 -- 172.21.15.46:0/1502254194 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d87340bfe0 msgr2=0x55d873420720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:25:38.972 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:38.971+0000 7ffa0affd640 1 --2- 172.21.15.46:0/1502254194 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d87340bfe0 0x55d873420720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:25:38.972 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:38.971+0000 7ffa0affd640 1 -- 172.21.15.46:0/1502254194 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d873414db0 msgr2=0x55d8734063c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:25:38.972 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:38.971+0000 7ffa0affd640 1 --2- 172.21.15.46:0/1502254194 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d873414db0 0x55d8734063c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:25:38.973 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:38.971+0000 7ffa0affd640 1 -- 172.21.15.46:0/1502254194 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d87340be10 con 0x55d87340b6c0 2024-03-31T23:25:38.973 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:38.971+0000 7ffa0a7fc640 1 --2- 172.21.15.46:0/1502254194 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d87340bfe0 0x55d873420720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:25:38.973 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:38.971+0000 7ffa0b7fe640 1 --2- 172.21.15.46:0/1502254194 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d873414db0 0x55d8734063c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:25:38.973 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:38.971+0000 7ffa0affd640 1 --2- 172.21.15.46:0/1502254194 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d87340b6c0 0x55d873414560 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7ff9fc000c00 tx=0x7ff9fc086040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=301a509a5edb10cd server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:25:38.973 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:38.971+0000 7ff9f0ff9640 1 -- 172.21.15.46:0/1502254194 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9fc09b350 con 0x55d87340b6c0 2024-03-31T23:25:38.973 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:38.971+0000 7ff9f0ff9640 1 -- 172.21.15.46:0/1502254194 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff9fc010e00 con 0x55d87340b6c0 2024-03-31T23:25:38.973 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:38.971+0000 7ff9f0ff9640 1 -- 172.21.15.46:0/1502254194 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9fc0d2e60 con 0x55d87340b6c0 2024-03-31T23:25:38.976 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:38.972+0000 7ffa143e4740 1 -- 172.21.15.46:0/1502254194 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d87340b6c0 msgr2=0x55d873414560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:25:38.976 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:38.972+0000 7ffa143e4740 1 --2- 172.21.15.46:0/1502254194 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d87340b6c0 0x55d873414560 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7ff9fc000c00 tx=0x7ff9fc086040 comp rx=0 tx=0).stop 2024-03-31T23:25:38.976 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:38.972+0000 7ffa143e4740 1 -- 172.21.15.46:0/1502254194 shutdown_connections 2024-03-31T23:25:38.976 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:38.972+0000 7ffa143e4740 1 --2- 172.21.15.46:0/1502254194 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d873414db0 0x55d8734063c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:25:38.976 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:38.972+0000 7ffa143e4740 1 --2- 172.21.15.46:0/1502254194 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d87340bfe0 0x55d873420720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:25:38.976 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:38.972+0000 7ffa143e4740 1 --2- 172.21.15.46:0/1502254194 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d87340b6c0 0x55d873414560 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:25:38.976 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:38.972+0000 7ffa143e4740 1 -- 172.21.15.46:0/1502254194 >> 172.21.15.46:0/1502254194 conn(0x55d873405fe0 msgr2=0x55d873400bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:25:38.976 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:38.972+0000 7ffa143e4740 1 -- 172.21.15.46:0/1502254194 shutdown_connections 2024-03-31T23:25:38.977 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:38.972+0000 7ffa143e4740 1 -- 172.21.15.46:0/1502254194 wait complete. 2024-03-31T23:25:38.977 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:38.973+0000 7ffa143e4740 1 -- start start 2024-03-31T23:25:38.977 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:38.973+0000 7ffa143e4740 1 Processor -- start 2024-03-31T23:25:38.977 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:38.973+0000 7ffa143e4740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d87340b6c0 0x55d87341ce30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:25:38.977 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:38.973+0000 7ffa143e4740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d87340bfe0 0x55d873418310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:25:38.977 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:38.973+0000 7ffa143e4740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d873414db0 0x55d873418850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:25:38.977 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:38.973+0000 7ffa143e4740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7ffa0c1b3930 con 0x55d873414db0 2024-03-31T23:25:38.977 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:38.973+0000 7ffa143e4740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7ffa0c1b4da0 con 0x55d87340b6c0 2024-03-31T23:25:38.977 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:38.973+0000 7ffa143e4740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7ffa0c1b4000 con 0x55d87340bfe0 2024-03-31T23:25:38.977 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:38.973+0000 7ffa143e4740 10 client.0 _ll_register_callbacks cb 0x7ffa040045c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T23:25:38.977 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:38.973+0000 7ffa0b7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d873414db0 0x55d873418850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:25:38.977 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:38.973+0000 7ffa0affd640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d87340b6c0 0x55d87341ce30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:25:38.977 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:38.973+0000 7ffa0affd640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d87340b6c0 0x55d87341ce30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.46:60238/0 (socket says 172.21.15.46:60238) 2024-03-31T23:25:38.978 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:38.973+0000 7ffa0affd640 1 -- 172.21.15.46:0/4266991384 learned_addr learned my addr 172.21.15.46:0/4266991384 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:25:38.978 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:38.973+0000 7ffa0b7fe640 1 --2- 172.21.15.46:0/4266991384 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d873414db0 0x55d873418850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39722/0 (socket says 172.21.15.46:39722) 2024-03-31T23:25:38.978 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:38.973+0000 7ffa0a7fc640 1 --2- 172.21.15.46:0/4266991384 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d87340bfe0 0x55d873418310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:25:38.978 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:38.974+0000 7ffa0b7fe640 1 -- 172.21.15.46:0/4266991384 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d87340bfe0 msgr2=0x55d873418310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:25:38.978 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:38.974+0000 7ffa0b7fe640 1 --2- 172.21.15.46:0/4266991384 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d87340bfe0 0x55d873418310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:25:38.978 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:38.974+0000 7ffa0b7fe640 1 -- 172.21.15.46:0/4266991384 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d87340b6c0 msgr2=0x55d87341ce30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:25:38.978 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:38.974+0000 7ffa0b7fe640 1 --2- 172.21.15.46:0/4266991384 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d87340b6c0 0x55d87341ce30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:25:38.978 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:38.974+0000 7ffa0b7fe640 1 -- 172.21.15.46:0/4266991384 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d873418e90 con 0x55d873414db0 2024-03-31T23:25:38.978 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:38.974+0000 7ffa0affd640 1 --2- 172.21.15.46:0/4266991384 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d87340b6c0 0x55d87341ce30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:25:38.978 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:38.974+0000 7ffa0b7fe640 1 --2- 172.21.15.46:0/4266991384 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d873414db0 0x55d873418850 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7ffa04004590 tx=0x7ffa04003e20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a199327c1b26e66 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:25:38.978 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:38.974+0000 7ff9eeff5640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-31T23:25:38.978 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:38.974+0000 7ff9eeff5640 1 -- 172.21.15.46:0/4266991384 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa04005990 con 0x55d873414db0 2024-03-31T23:25:38.978 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:38.974+0000 7ff9eeff5640 1 -- 172.21.15.46:0/4266991384 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(112..112 src has 1..112) ==== 9554+0+0 (secure 0 0 0) 0x7ffa04004b20 con 0x55d873414db0 2024-03-31T23:25:38.979 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:38.974+0000 7ffa143e4740 1 -- 172.21.15.46:0/4266991384 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55d87337be30 con 0x55d873414db0 2024-03-31T23:25:38.979 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:38.974+0000 7ff9eeff5640 1 -- 172.21.15.46:0/4266991384 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7ffa04004d20 con 0x55d873414db0 2024-03-31T23:25:38.979 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:38.974+0000 7ffa143e4740 10 client.17283 fetch_fsmap learned FSMap version 38 2024-03-31T23:25:38.979 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:38.974+0000 7ffa143e4740 1 -- 172.21.15.46:0/4266991384 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x55d87348afe0 con 0x55d873414db0 2024-03-31T23:25:38.979 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:38.975+0000 7ff9eeff5640 1 -- 172.21.15.46:0/4266991384 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ffa0400f5d0 con 0x55d873414db0 2024-03-31T23:25:38.979 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:38.975+0000 7ff9eeff5640 1 -- 172.21.15.46:0/4266991384 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 38) ==== 8370+0+0 (secure 0 0 0) 0x7ffa04037400 con 0x55d873414db0 2024-03-31T23:25:38.979 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:38.975+0000 7ffa143e4740 10 client.17283 fetch_fsmap finished waiting for FSMap version 38 2024-03-31T23:25:38.979 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:38.975+0000 7ffa143e4740 10 client.17283 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-31T23:25:38.979 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:38.975+0000 7ffa143e4740 20 client.17283 populate_metadata read hostname 'smithi046' 2024-03-31T23:25:38.979 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:38.975+0000 7ffa143e4740 1 --2- 172.21.15.46:0/4266991384 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55d87348b750 0x55d87348db40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:25:38.979 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:38.975+0000 7ffa143e4740 4 client.17283 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-31T23:25:38.979 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:38.975+0000 7ffa143e4740 1 -- 172.21.15.46:0/4266991384 --> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55d87337be30 con 0x55d87348b750 2024-03-31T23:25:38.979 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:38.975+0000 7ffa0affd640 1 --2- 172.21.15.46:0/4266991384 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55d87348b750 0x55d87348db40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:25:38.979 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:38.976+0000 7ffa0affd640 1 --2- 172.21.15.46:0/4266991384 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55d87348b750 0x55d87348db40 crc :-1 s=READY pgs=200 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6d60b98e9ceb4721 server_cookie=7510deacf8d4a555 in_seq=0 out_seq=0 2024-03-31T23:25:38.980 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:38.976+0000 7ff9eeff5640 10 client.17283 ms_handle_connect on v2:172.21.15.142:6838/2519143918 2024-03-31T23:25:39.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:25:38 smithi162 ceph-mon[36237]: pgmap v436: 129 pgs: 129 active+clean; 392 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 3.9 MiB/s wr, 238 op/s 2024-03-31T23:25:39.077 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:39.077+0000 7ff9eeff5640 1 -- 172.21.15.46:0/4266991384 <== mds.0 v2:172.21.15.142:6838/2519143918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55d87337be30 con 0x55d87348b750 2024-03-31T23:25:39.078 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:39.077+0000 7ff9eeff5640 10 client.17283 handle_command_reply: tid=0 2024-03-31T23:25:39.078 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:39.078+0000 7ffa143e4740 1 client.17283 shutdown 2024-03-31T23:25:39.080 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:39.079+0000 7ffa143e4740 1 -- 172.21.15.46:0/4266991384 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d873414db0 msgr2=0x55d873418850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:25:39.080 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:39.079+0000 7ffa143e4740 1 --2- 172.21.15.46:0/4266991384 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d873414db0 0x55d873418850 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7ffa04004590 tx=0x7ffa04003e20 comp rx=0 tx=0).stop 2024-03-31T23:25:39.080 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:39.079+0000 7ffa143e4740 1 -- 172.21.15.46:0/4266991384 shutdown_connections 2024-03-31T23:25:39.080 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:39.079+0000 7ffa143e4740 1 --2- 172.21.15.46:0/4266991384 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55d87348b750 0x55d87348db40 crc :-1 s=READY pgs=200 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:25:39.080 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:39.079+0000 7ffa143e4740 1 --2- 172.21.15.46:0/4266991384 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d873414db0 0x55d873418850 unknown :-1 s=CLOSED pgs=881 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:25:39.080 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:39.079+0000 7ff9eeff5640 0 client.17283 ms_handle_reset on v2:172.21.15.142:6838/2519143918 2024-03-31T23:25:39.080 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:39.079+0000 7ffa143e4740 1 --2- 172.21.15.46:0/4266991384 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d87340bfe0 0x55d873418310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:25:39.080 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:39.079+0000 7ffa143e4740 1 --2- 172.21.15.46:0/4266991384 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d87340b6c0 0x55d87341ce30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:25:39.080 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:39.079+0000 7ffa143e4740 1 -- 172.21.15.46:0/4266991384 >> 172.21.15.46:0/4266991384 conn(0x55d873405fe0 msgr2=0x55d8734063c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:25:39.080 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:39.080+0000 7ffa143e4740 1 -- 172.21.15.46:0/4266991384 shutdown_connections 2024-03-31T23:25:39.080 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:39.080+0000 7ffa143e4740 1 -- 172.21.15.46:0/4266991384 wait complete. 2024-03-31T23:25:39.080 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:39.080+0000 7ffa143e4740 20 client.17283 trim_cache size 0 max 16384 2024-03-31T23:25:39.081 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout:{ 2024-03-31T23:25:39.081 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout: "message": "", 2024-03-31T23:25:39.081 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout: "return_code": 0 2024-03-31T23:25:39.081 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout:} 2024-03-31T23:25:39.082 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:39.082+0000 7ffa11725640 1 -- 172.21.15.46:0/200947113 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7ff9f40a2730 msgr2=0x7ff9f40a4bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:25:39.083 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:39.082+0000 7ffa11725640 1 --2- 172.21.15.46:0/200947113 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7ff9f40a2730 0x7ff9f40a4bf0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7ffa04003f50 tx=0x7ffa04004a40 comp rx=0 tx=0).stop 2024-03-31T23:25:39.083 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:39.082+0000 7ffa11725640 1 -- 172.21.15.46:0/200947113 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ffa0c100d60 msgr2=0x7ffa0c1a4420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:25:39.083 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:39.082+0000 7ffa11725640 1 --2- 172.21.15.46:0/200947113 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ffa0c100d60 0x7ffa0c1a4420 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7ff9fc00e9f0 tx=0x7ff9fc00eeb0 comp rx=0 tx=0).stop 2024-03-31T23:25:39.084 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:39.083+0000 7ffa11725640 1 -- 172.21.15.46:0/200947113 shutdown_connections 2024-03-31T23:25:39.084 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:39.083+0000 7ffa11725640 1 --2- 172.21.15.46:0/200947113 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7ff9f40a2730 0x7ff9f40a4bf0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:25:39.084 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:39.083+0000 7ffa11725640 1 --2- 172.21.15.46:0/200947113 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffa0c106850 0x7ffa0c19e770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:25:39.084 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:39.083+0000 7ffa11725640 1 --2- 172.21.15.46:0/200947113 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ffa0c101720 0x7ffa0c1a4960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:25:39.084 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:39.084+0000 7ffa11725640 1 --2- 172.21.15.46:0/200947113 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ffa0c100d60 0x7ffa0c1a4420 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:25:39.084 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:39.084+0000 7ffa11725640 1 -- 172.21.15.46:0/200947113 >> 172.21.15.46:0/200947113 conn(0x7ffa0c076c10 msgr2=0x7ffa0c0ffd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:25:39.084 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:39.084+0000 7ffa11725640 1 -- 172.21.15.46:0/200947113 shutdown_connections 2024-03-31T23:25:39.085 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:39.085+0000 7ffa11725640 1 -- 172.21.15.46:0/200947113 wait complete. 2024-03-31T23:25:39.266 INFO:tasks.workunit.client.0.smithi046.stdout: 10 6904 12.41 MB/sec warmup 44 sec latency 137.172 ms 2024-03-31T23:25:40.266 INFO:tasks.workunit.client.0.smithi046.stdout: 10 7051 12.31 MB/sec warmup 45 sec latency 61.593 ms 2024-03-31T23:25:40.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:25:40 smithi046 ceph-mon[32498]: pgmap v437: 129 pgs: 129 active+clean; 408 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 6.2 MiB/s wr, 318 op/s 2024-03-31T23:25:40.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:25:40 smithi142 ceph-mon[35271]: pgmap v437: 129 pgs: 129 active+clean; 408 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 6.2 MiB/s wr, 318 op/s 2024-03-31T23:25:41.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:25:40 smithi162 ceph-mon[36237]: pgmap v437: 129 pgs: 129 active+clean; 408 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 6.2 MiB/s wr, 318 op/s 2024-03-31T23:25:41.267 INFO:tasks.workunit.client.0.smithi046.stdout: 10 7177 12.13 MB/sec warmup 46 sec latency 68.544 ms 2024-03-31T23:25:42.267 INFO:tasks.workunit.client.0.smithi046.stdout: 10 7363 12.23 MB/sec warmup 47 sec latency 64.115 ms 2024-03-31T23:25:42.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:25:42 smithi046 ceph-mon[32498]: pgmap v438: 129 pgs: 129 active+clean; 408 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 4.1 MiB/s wr, 181 op/s 2024-03-31T23:25:42.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:25:42 smithi142 ceph-mon[35271]: pgmap v438: 129 pgs: 129 active+clean; 408 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 4.1 MiB/s wr, 181 op/s 2024-03-31T23:25:43.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:25:42 smithi162 ceph-mon[36237]: pgmap v438: 129 pgs: 129 active+clean; 408 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 4.1 MiB/s wr, 181 op/s 2024-03-31T23:25:43.266 INFO:tasks.workunit.client.0.smithi046.stdout: 10 7550 12.19 MB/sec warmup 48 sec latency 76.582 ms 2024-03-31T23:25:44.267 INFO:tasks.workunit.client.0.smithi046.stdout: 10 7745 12.15 MB/sec warmup 49 sec latency 67.778 ms 2024-03-31T23:25:44.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:25:44 smithi046 ceph-mon[32498]: pgmap v439: 129 pgs: 129 active+clean; 408 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 4.1 MiB/s wr, 181 op/s 2024-03-31T23:25:44.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:25:44 smithi142 ceph-mon[35271]: pgmap v439: 129 pgs: 129 active+clean; 408 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 4.1 MiB/s wr, 181 op/s 2024-03-31T23:25:45.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:25:44 smithi162 ceph-mon[36237]: pgmap v439: 129 pgs: 129 active+clean; 408 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 4.1 MiB/s wr, 181 op/s 2024-03-31T23:25:45.267 INFO:tasks.workunit.client.0.smithi046.stdout: 10 7884 12.04 MB/sec warmup 50 sec latency 89.693 ms 2024-03-31T23:25:46.267 INFO:tasks.workunit.client.0.smithi046.stdout: 10 8024 11.83 MB/sec warmup 51 sec latency 73.186 ms 2024-03-31T23:25:46.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:25:46 smithi046 ceph-mon[32498]: pgmap v440: 129 pgs: 129 active+clean; 415 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 6.5 MiB/s wr, 326 op/s 2024-03-31T23:25:46.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:25:46 smithi142 ceph-mon[35271]: pgmap v440: 129 pgs: 129 active+clean; 415 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 6.5 MiB/s wr, 326 op/s 2024-03-31T23:25:47.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:25:46 smithi162 ceph-mon[36237]: pgmap v440: 129 pgs: 129 active+clean; 415 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 6.5 MiB/s wr, 326 op/s 2024-03-31T23:25:47.267 INFO:tasks.workunit.client.0.smithi046.stdout: 10 8218 11.86 MB/sec warmup 52 sec latency 63.869 ms 2024-03-31T23:25:48.267 INFO:tasks.workunit.client.0.smithi046.stdout: 10 8368 11.69 MB/sec warmup 53 sec latency 95.986 ms 2024-03-31T23:25:48.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:25:48 smithi046 ceph-mon[32498]: pgmap v441: 129 pgs: 129 active+clean; 415 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 4.6 MiB/s wr, 225 op/s 2024-03-31T23:25:48.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:25:48 smithi142 ceph-mon[35271]: pgmap v441: 129 pgs: 129 active+clean; 415 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 4.6 MiB/s wr, 225 op/s 2024-03-31T23:25:49.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:25:48 smithi162 ceph-mon[36237]: pgmap v441: 129 pgs: 129 active+clean; 415 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 4.6 MiB/s wr, 225 op/s 2024-03-31T23:25:49.267 INFO:tasks.workunit.client.0.smithi046.stdout: 10 8541 11.62 MB/sec warmup 54 sec latency 92.195 ms 2024-03-31T23:25:50.267 INFO:tasks.workunit.client.0.smithi046.stdout: 10 8711 11.58 MB/sec warmup 55 sec latency 62.925 ms 2024-03-31T23:25:50.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:25:50 smithi046 ceph-mon[32498]: pgmap v442: 129 pgs: 129 active+clean; 417 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 6.3 MiB/s wr, 331 op/s 2024-03-31T23:25:50.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:25:50 smithi142 ceph-mon[35271]: pgmap v442: 129 pgs: 129 active+clean; 417 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 6.3 MiB/s wr, 331 op/s 2024-03-31T23:25:51.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:25:50 smithi162 ceph-mon[36237]: pgmap v442: 129 pgs: 129 active+clean; 417 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 6.3 MiB/s wr, 331 op/s 2024-03-31T23:25:51.267 INFO:tasks.workunit.client.0.smithi046.stdout: 10 8913 11.61 MB/sec warmup 56 sec latency 63.191 ms 2024-03-31T23:25:52.232 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:52.230+0000 7f16f14fa640 1 -- 172.21.15.46:0/1158816596 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f16ec1012d0 msgr2=0x7f16ec10d470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:25:52.232 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:52.230+0000 7f16f14fa640 1 --2- 172.21.15.46:0/1158816596 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f16ec1012d0 0x7f16ec10d470 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7f16e4009ca0 tx=0x7f16e402c0f0 comp rx=0 tx=0).stop 2024-03-31T23:25:52.232 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:52.231+0000 7f16f14fa640 1 -- 172.21.15.46:0/1158816596 shutdown_connections 2024-03-31T23:25:52.232 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:52.231+0000 7f16f14fa640 1 --2- 172.21.15.46:0/1158816596 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f16ec1012d0 0x7f16ec10d470 unknown :-1 s=CLOSED pgs=882 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:25:52.233 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:52.231+0000 7f16f14fa640 1 --2- 172.21.15.46:0/1158816596 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f16ec100930 0x7f16ec100d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:25:52.233 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:52.231+0000 7f16f14fa640 1 --2- 172.21.15.46:0/1158816596 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f16ec106720 0x7f16ec106b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:25:52.233 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:52.231+0000 7f16f14fa640 1 -- 172.21.15.46:0/1158816596 >> 172.21.15.46:0/1158816596 conn(0x7f16ec077630 msgr2=0x7f16ec077a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:25:52.233 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:52.231+0000 7f16f14fa640 1 -- 172.21.15.46:0/1158816596 shutdown_connections 2024-03-31T23:25:52.233 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:52.231+0000 7f16f14fa640 1 -- 172.21.15.46:0/1158816596 wait complete. 2024-03-31T23:25:52.233 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:52.231+0000 7f16f14fa640 1 Processor -- start 2024-03-31T23:25:52.235 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:52.232+0000 7f16f14fa640 1 -- start start 2024-03-31T23:25:52.235 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:52.232+0000 7f16f14fa640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f16ec100930 0x7f16ec19c8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:25:52.235 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:52.232+0000 7f16f14fa640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f16ec1012d0 0x7f16ec19ce00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:25:52.235 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:52.232+0000 7f16f14fa640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f16ec106720 0x7f16ec196a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:25:52.235 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:52.232+0000 7f16f14fa640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f16ec1108b0 con 0x7f16ec106720 2024-03-31T23:25:52.235 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:52.232+0000 7f16f14fa640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f16ec110730 con 0x7f16ec100930 2024-03-31T23:25:52.235 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:52.232+0000 7f16f14fa640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f16ec110a30 con 0x7f16ec1012d0 2024-03-31T23:25:52.235 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:52.232+0000 7f16eaffd640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f16ec100930 0x7f16ec19c8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:25:52.235 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:52.232+0000 7f16eb7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f16ec106720 0x7f16ec196a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:25:52.235 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:52.232+0000 7f16eaffd640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f16ec100930 0x7f16ec19c8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.46:45470/0 (socket says 172.21.15.46:45470) 2024-03-31T23:25:52.235 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:52.232+0000 7f16ea7fc640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f16ec1012d0 0x7f16ec19ce00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:25:52.235 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:52.232+0000 7f16eaffd640 1 -- 172.21.15.46:0/4123151582 learned_addr learned my addr 172.21.15.46:0/4123151582 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:25:52.236 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:52.232+0000 7f16eaffd640 1 -- 172.21.15.46:0/4123151582 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f16ec1012d0 msgr2=0x7f16ec19ce00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:25:52.236 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:52.233+0000 7f16eaffd640 1 --2- 172.21.15.46:0/4123151582 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f16ec1012d0 0x7f16ec19ce00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:25:52.236 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:52.233+0000 7f16eaffd640 1 -- 172.21.15.46:0/4123151582 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f16ec106720 msgr2=0x7f16ec196a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:25:52.236 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:52.233+0000 7f16eaffd640 1 --2- 172.21.15.46:0/4123151582 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f16ec106720 0x7f16ec196a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:25:52.236 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:52.233+0000 7f16eaffd640 1 -- 172.21.15.46:0/4123151582 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f16ec1971f0 con 0x7f16ec100930 2024-03-31T23:25:52.236 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:52.233+0000 7f16ea7fc640 1 --2- 172.21.15.46:0/4123151582 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f16ec1012d0 0x7f16ec19ce00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:25:52.236 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:52.233+0000 7f16eb7fe640 1 --2- 172.21.15.46:0/4123151582 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f16ec106720 0x7f16ec196a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:25:52.236 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:52.233+0000 7f16eaffd640 1 --2- 172.21.15.46:0/4123151582 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f16ec100930 0x7f16ec19c8c0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f16dc00d950 tx=0x7f16dc00de10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:25:52.236 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:52.233+0000 7f16d3fff640 1 -- 172.21.15.46:0/4123151582 <== mon.2 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16dc017030 con 0x7f16ec100930 2024-03-31T23:25:52.236 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:52.233+0000 7f16d3fff640 1 -- 172.21.15.46:0/4123151582 <== mon.2 v2:172.21.15.142:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f16dc004020 con 0x7f16ec100930 2024-03-31T23:25:52.236 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:52.233+0000 7f16f14fa640 1 -- 172.21.15.46:0/4123151582 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f16ec1974e0 con 0x7f16ec100930 2024-03-31T23:25:52.236 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:52.233+0000 7f16d3fff640 1 -- 172.21.15.46:0/4123151582 <== mon.2 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16dc0047a0 con 0x7f16ec100930 2024-03-31T23:25:52.236 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:52.233+0000 7f16f14fa640 1 -- 172.21.15.46:0/4123151582 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f16ec069ef0 con 0x7f16ec100930 2024-03-31T23:25:52.237 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:52.236+0000 7f16d3fff640 1 -- 172.21.15.46:0/4123151582 <== mon.2 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 25) ==== 137451+0+0 (secure 0 0 0) 0x7f16dc0041d0 con 0x7f16ec100930 2024-03-31T23:25:52.237 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:52.236+0000 7f16f41b0740 1 Processor -- start 2024-03-31T23:25:52.238 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:52.237+0000 7f16f41b0740 1 -- start start 2024-03-31T23:25:52.240 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:52.237+0000 7f16f41b0740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x564150db26c0 0x564150db2aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:25:52.240 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:52.237+0000 7f16f41b0740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564150db2fe0 0x564150dbb870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:25:52.240 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:52.237+0000 7f16f41b0740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564150dbbdb0 0x564150dbe1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:25:52.241 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:52.237+0000 7f16f41b0740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x564150d158f0 con 0x564150db2fe0 2024-03-31T23:25:52.241 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:52.237+0000 7f16f41b0740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x564150cbe8d0 con 0x564150db26c0 2024-03-31T23:25:52.241 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:52.237+0000 7f16f41b0740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x564150d11990 con 0x564150dbbdb0 2024-03-31T23:25:52.241 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:52.237+0000 7f16d3fff640 1 --2- 172.21.15.46:0/4123151582 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f16d40a2730 0x7f16d40a4bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:25:52.241 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:52.237+0000 7f16ea7fc640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564150dbbdb0 0x564150dbe1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:25:52.241 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:52.237+0000 7f16ea7fc640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564150dbbdb0 0x564150dbe1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.46:46262/0 (socket says 172.21.15.46:46262) 2024-03-31T23:25:52.241 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:52.237+0000 7f16eb7fe640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x564150db26c0 0x564150db2aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:25:52.241 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:52.237+0000 7f16ea7fc640 1 -- 172.21.15.46:0/676204331 learned_addr learned my addr 172.21.15.46:0/676204331 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:25:52.241 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:52.237+0000 7f16eaffd640 1 --2- 172.21.15.46:0/676204331 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564150db2fe0 0x564150dbb870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:25:52.241 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:52.237+0000 7f16eb7fe640 1 --2- 172.21.15.46:0/4123151582 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f16d40a2730 0x7f16d40a4bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:25:52.241 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:52.237+0000 7f16d3fff640 1 -- 172.21.15.46:0/4123151582 <== mon.2 v2:172.21.15.142:3300/0 5 ==== osd_map(112..112 src has 1..112) ==== 9554+0+0 (secure 0 0 0) 0x7f16dc0ce960 con 0x7f16ec100930 2024-03-31T23:25:52.241 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:52.238+0000 7f16ea7fc640 1 -- 172.21.15.46:0/676204331 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x564150db26c0 msgr2=0x564150db2aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:25:52.241 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:52.238+0000 7f16ea7fc640 1 --2- 172.21.15.46:0/676204331 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x564150db26c0 0x564150db2aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:25:52.241 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:52.238+0000 7f16ea7fc640 1 -- 172.21.15.46:0/676204331 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564150db2fe0 msgr2=0x564150dbb870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:25:52.242 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:52.238+0000 7f16ea7fc640 1 --2- 172.21.15.46:0/676204331 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564150db2fe0 0x564150dbb870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:25:52.242 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:52.238+0000 7f16ea7fc640 1 -- 172.21.15.46:0/676204331 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564150a2a210 con 0x564150dbbdb0 2024-03-31T23:25:52.242 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:52.238+0000 7f16ea7fc640 1 --2- 172.21.15.46:0/676204331 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564150dbbdb0 0x564150dbe1a0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f16e0010480 tx=0x7f16e0010940 comp rx=0 tx=0).ready entity=mon.1 client_cookie=499aa3ea6895b362 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:25:52.242 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:52.238+0000 7f16eb7fe640 1 --2- 172.21.15.46:0/4123151582 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f16d40a2730 0x7f16d40a4bf0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f16e4004af0 tx=0x7f16e4009360 comp rx=0 tx=0).ready entity=mgr.15654 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:25:52.242 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:52.238+0000 7f16d17fa640 1 -- 172.21.15.46:0/676204331 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16e0011760 con 0x564150dbbdb0 2024-03-31T23:25:52.242 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:52.238+0000 7f16d17fa640 1 -- 172.21.15.46:0/676204331 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f16e0011db0 con 0x564150dbbdb0 2024-03-31T23:25:52.242 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:52.238+0000 7f16d17fa640 1 -- 172.21.15.46:0/676204331 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16e000e430 con 0x564150dbbdb0 2024-03-31T23:25:52.242 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:52.239+0000 7f16f41b0740 1 -- 172.21.15.46:0/676204331 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564150dbbdb0 msgr2=0x564150dbe1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:25:52.242 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:52.239+0000 7f16f41b0740 1 --2- 172.21.15.46:0/676204331 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564150dbbdb0 0x564150dbe1a0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f16e0010480 tx=0x7f16e0010940 comp rx=0 tx=0).stop 2024-03-31T23:25:52.242 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:52.239+0000 7f16f41b0740 1 -- 172.21.15.46:0/676204331 shutdown_connections 2024-03-31T23:25:52.242 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:52.239+0000 7f16f41b0740 1 --2- 172.21.15.46:0/676204331 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564150dbbdb0 0x564150dbe1a0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:25:52.242 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:52.239+0000 7f16f41b0740 1 --2- 172.21.15.46:0/676204331 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564150db2fe0 0x564150dbb870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:25:52.242 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:52.239+0000 7f16f41b0740 1 --2- 172.21.15.46:0/676204331 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x564150db26c0 0x564150db2aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:25:52.242 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:52.239+0000 7f16f41b0740 1 -- 172.21.15.46:0/676204331 >> 172.21.15.46:0/676204331 conn(0x564150dacfe0 msgr2=0x564150dad3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:25:52.242 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:52.239+0000 7f16f41b0740 1 -- 172.21.15.46:0/676204331 shutdown_connections 2024-03-31T23:25:52.243 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:52.239+0000 7f16f41b0740 1 -- 172.21.15.46:0/676204331 wait complete. 2024-03-31T23:25:52.243 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:52.240+0000 7f16f41b0740 1 -- start start 2024-03-31T23:25:52.246 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:52.240+0000 7f16f41b0740 1 Processor -- start 2024-03-31T23:25:52.246 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:52.240+0000 7f16f41b0740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564150db26c0 0x564150db8f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:25:52.246 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:52.240+0000 7f16f41b0740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x564150db2fe0 0x564150db9450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:25:52.246 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:52.240+0000 7f16f41b0740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564150dbbdb0 0x564150db9990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:25:52.246 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:52.240+0000 7f16f41b0740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x564150dbe9e0 con 0x564150dbbdb0 2024-03-31T23:25:52.246 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:52.240+0000 7f16f41b0740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x564150cb3f80 con 0x564150db2fe0 2024-03-31T23:25:52.247 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:52.240+0000 7f16f41b0740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x564150dbeb60 con 0x564150db26c0 2024-03-31T23:25:52.247 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:52.240+0000 7f16eaffd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564150db26c0 0x564150db8f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:25:52.247 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:52.240+0000 7f16f41b0740 10 client.0 _ll_register_callbacks cb 0x564150d0c290 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T23:25:52.247 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:52.240+0000 7f16eaffd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564150db26c0 0x564150db8f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.46:46270/0 (socket says 172.21.15.46:46270) 2024-03-31T23:25:52.247 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:52.240+0000 7f16eaffd640 1 -- 172.21.15.46:0/1897776945 learned_addr learned my addr 172.21.15.46:0/1897776945 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:25:52.247 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:52.240+0000 7f16ea7fc640 1 --2- 172.21.15.46:0/1897776945 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x564150db2fe0 0x564150db9450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:25:52.247 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:52.240+0000 7f16eb7fe640 1 --2- 172.21.15.46:0/1897776945 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564150dbbdb0 0x564150db9990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:25:52.247 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:52.240+0000 7f16eaffd640 1 -- 172.21.15.46:0/1897776945 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x564150db2fe0 msgr2=0x564150db9450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:25:52.247 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:52.240+0000 7f16eaffd640 1 --2- 172.21.15.46:0/1897776945 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x564150db2fe0 0x564150db9450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:25:52.247 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:52.240+0000 7f16eaffd640 1 -- 172.21.15.46:0/1897776945 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564150dbbdb0 msgr2=0x564150db9990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:25:52.247 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:52.240+0000 7f16eaffd640 1 --2- 172.21.15.46:0/1897776945 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564150dbbdb0 0x564150db9990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:25:52.247 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:52.240+0000 7f16eaffd640 1 -- 172.21.15.46:0/1897776945 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564150af6620 con 0x564150db26c0 2024-03-31T23:25:52.247 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:52.240+0000 7f16ea7fc640 1 --2- 172.21.15.46:0/1897776945 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x564150db2fe0 0x564150db9450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T23:25:52.247 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:52.241+0000 7f16eaffd640 1 --2- 172.21.15.46:0/1897776945 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564150db26c0 0x564150db8f10 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f16dc004770 tx=0x7f16dc069ba0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5e102a04f62e1b87 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:25:52.248 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:52.241+0000 7f16ceff5640 10 client.0 ms_handle_connect on v2:172.21.15.162:3300/0 2024-03-31T23:25:52.248 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:52.241+0000 7f16ceff5640 1 -- 172.21.15.46:0/1897776945 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16dc09ac70 con 0x564150db26c0 2024-03-31T23:25:52.248 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:52.241+0000 7f16f41b0740 1 -- 172.21.15.46:0/1897776945 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f16e0011db0 con 0x564150db26c0 2024-03-31T23:25:52.248 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:52.241+0000 7f16ceff5640 1 -- 172.21.15.46:0/1897776945 <== mon.1 v2:172.21.15.162:3300/0 2 ==== osd_map(112..112 src has 1..112) ==== 9554+0+0 (secure 0 0 0) 0x7f16dc06ab40 con 0x564150db26c0 2024-03-31T23:25:52.248 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:52.241+0000 7f16ceff5640 1 -- 172.21.15.46:0/1897776945 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7f16dc09edc0 con 0x564150db26c0 2024-03-31T23:25:52.248 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:52.241+0000 7f16f41b0740 10 client.27025 fetch_fsmap learned FSMap version 38 2024-03-31T23:25:52.248 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:52.241+0000 7f16f41b0740 1 -- 172.21.15.46:0/1897776945 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x564150dd31f0 con 0x564150db26c0 2024-03-31T23:25:52.248 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:52.241+0000 7f16ceff5640 1 -- 172.21.15.46:0/1897776945 <== mon.1 v2:172.21.15.162:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f16e0011db0 con 0x564150db26c0 2024-03-31T23:25:52.248 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:52.241+0000 7f16ceff5640 1 -- 172.21.15.46:0/1897776945 <== mon.1 v2:172.21.15.162:3300/0 5 ==== fsmap(e 38) ==== 8370+0+0 (secure 0 0 0) 0x7f16dc0969c0 con 0x564150db26c0 2024-03-31T23:25:52.248 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:52.242+0000 7f16f41b0740 10 client.27025 fetch_fsmap finished waiting for FSMap version 38 2024-03-31T23:25:52.248 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:52.242+0000 7f16f41b0740 10 client.27025 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-31T23:25:52.248 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:52.242+0000 7f16f41b0740 20 client.27025 populate_metadata read hostname 'smithi046' 2024-03-31T23:25:52.248 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:52.242+0000 7f16f41b0740 1 --2- 172.21.15.46:0/1897776945 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x564150dd4b70 0x564150dd7010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:25:52.248 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:52.242+0000 7f16f41b0740 4 client.27025 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-31T23:25:52.248 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:52.242+0000 7f16f41b0740 1 -- 172.21.15.46:0/1897776945 --> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564150d29120 con 0x564150dd4b70 2024-03-31T23:25:52.249 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:52.242+0000 7f16ea7fc640 1 --2- 172.21.15.46:0/1897776945 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x564150dd4b70 0x564150dd7010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:25:52.249 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:52.242+0000 7f16ea7fc640 1 --2- 172.21.15.46:0/1897776945 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x564150dd4b70 0x564150dd7010 crc :-1 s=READY pgs=201 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dfe4f2de607985f7 server_cookie=fdb6782f3642e29d in_seq=0 out_seq=0 2024-03-31T23:25:52.249 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:52.242+0000 7f16ceff5640 10 client.27025 ms_handle_connect on v2:172.21.15.142:6838/2519143918 2024-03-31T23:25:52.249 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:52.243+0000 7f16ceff5640 1 -- 172.21.15.46:0/1897776945 <== mds.0 v2:172.21.15.142:6838/2519143918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x564150d29120 con 0x564150dd4b70 2024-03-31T23:25:52.249 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:52.243+0000 7f16ceff5640 10 client.27025 handle_command_reply: tid=0 2024-03-31T23:25:52.249 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:52.243+0000 7f16f41b0740 1 client.27025 shutdown 2024-03-31T23:25:52.249 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:52.244+0000 7f16f41b0740 1 -- 172.21.15.46:0/1897776945 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564150db26c0 msgr2=0x564150db8f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:25:52.249 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:52.244+0000 7f16f41b0740 1 --2- 172.21.15.46:0/1897776945 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564150db26c0 0x564150db8f10 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f16dc004770 tx=0x7f16dc069ba0 comp rx=0 tx=0).stop 2024-03-31T23:25:52.249 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:52.244+0000 7f16f41b0740 1 -- 172.21.15.46:0/1897776945 shutdown_connections 2024-03-31T23:25:52.249 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:52.244+0000 7f16f41b0740 1 --2- 172.21.15.46:0/1897776945 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x564150dd4b70 0x564150dd7010 crc :-1 s=READY pgs=201 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:25:52.249 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:52.244+0000 7f16f41b0740 1 --2- 172.21.15.46:0/1897776945 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564150dbbdb0 0x564150db9990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:25:52.249 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:52.244+0000 7f16f41b0740 1 --2- 172.21.15.46:0/1897776945 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x564150db2fe0 0x564150db9450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:25:52.249 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:52.244+0000 7f16f41b0740 1 --2- 172.21.15.46:0/1897776945 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564150db26c0 0x564150db8f10 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:25:52.249 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:52.244+0000 7f16ceff5640 0 client.27025 ms_handle_reset on v2:172.21.15.142:6838/2519143918 2024-03-31T23:25:52.249 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:52.244+0000 7f16f41b0740 1 -- 172.21.15.46:0/1897776945 >> 172.21.15.46:0/1897776945 conn(0x564150dacfe0 msgr2=0x564150daed30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:25:52.250 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:52.244+0000 7f16f41b0740 1 -- 172.21.15.46:0/1897776945 shutdown_connections 2024-03-31T23:25:52.250 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:52.244+0000 7f16f41b0740 1 -- 172.21.15.46:0/1897776945 wait complete. 2024-03-31T23:25:52.250 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:52.244+0000 7f16f41b0740 20 client.27025 trim_cache size 0 max 16384 2024-03-31T23:25:52.267 INFO:tasks.workunit.client.0.smithi046.stdout: 10 9066 11.60 MB/sec warmup 57 sec latency 143.127 ms 2024-03-31T23:25:52.268 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:52.267+0000 7f16f41b0740 1 Processor -- start 2024-03-31T23:25:52.268 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:52.267+0000 7f16f41b0740 1 -- start start 2024-03-31T23:25:52.269 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:52.267+0000 7f16f41b0740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x564150db26c0 0x564150dcffd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:25:52.269 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:52.267+0000 7f16f41b0740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564150db2fe0 0x564150db97b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:25:52.269 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:52.267+0000 7f16f41b0740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564150dbbdb0 0x564150daeb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:25:52.269 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:52.267+0000 7f16f41b0740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f16ec1a69d0 con 0x564150dbbdb0 2024-03-31T23:25:52.269 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:52.267+0000 7f16f41b0740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f16ec1a41e0 con 0x564150db26c0 2024-03-31T23:25:52.269 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:52.267+0000 7f16f41b0740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f16ec072250 con 0x564150db2fe0 2024-03-31T23:25:52.269 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:52.268+0000 7f16ea7fc640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564150db2fe0 0x564150db97b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:25:52.269 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:52.268+0000 7f16eaffd640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x564150db26c0 0x564150dcffd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:25:52.269 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:52.268+0000 7f16eb7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564150dbbdb0 0x564150daeb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:25:52.269 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:52.268+0000 7f16ea7fc640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564150db2fe0 0x564150db97b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.46:46272/0 (socket says 172.21.15.46:46272) 2024-03-31T23:25:52.269 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:52.268+0000 7f16ea7fc640 1 -- 172.21.15.46:0/3584949743 learned_addr learned my addr 172.21.15.46:0/3584949743 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:25:52.271 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:52.268+0000 7f16eb7fe640 1 -- 172.21.15.46:0/3584949743 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564150db2fe0 msgr2=0x564150db97b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:25:52.271 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:52.268+0000 7f16eb7fe640 1 --2- 172.21.15.46:0/3584949743 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564150db2fe0 0x564150db97b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:25:52.271 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:52.268+0000 7f16eb7fe640 1 -- 172.21.15.46:0/3584949743 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x564150db26c0 msgr2=0x564150dcffd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:25:52.271 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:52.268+0000 7f16eb7fe640 1 --2- 172.21.15.46:0/3584949743 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x564150db26c0 0x564150dcffd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:25:52.271 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:52.268+0000 7f16eb7fe640 1 -- 172.21.15.46:0/3584949743 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564150db7460 con 0x564150dbbdb0 2024-03-31T23:25:52.271 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:52.268+0000 7f16eb7fe640 1 --2- 172.21.15.46:0/3584949743 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564150dbbdb0 0x564150daeb40 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7f16e402c700 tx=0x7f16e4047000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cfe39c7c10e21593 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:25:52.271 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:52.269+0000 7f16d0ff9640 1 -- 172.21.15.46:0/3584949743 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16e405a040 con 0x564150dbbdb0 2024-03-31T23:25:52.271 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:52.269+0000 7f16d0ff9640 1 -- 172.21.15.46:0/3584949743 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f16e4034bd0 con 0x564150dbbdb0 2024-03-31T23:25:52.271 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:52.269+0000 7f16d0ff9640 1 -- 172.21.15.46:0/3584949743 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16e4039040 con 0x564150dbbdb0 2024-03-31T23:25:52.271 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:52.269+0000 7f16f41b0740 1 -- 172.21.15.46:0/3584949743 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564150dbbdb0 msgr2=0x564150daeb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:25:52.272 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:52.269+0000 7f16f41b0740 1 --2- 172.21.15.46:0/3584949743 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564150dbbdb0 0x564150daeb40 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7f16e402c700 tx=0x7f16e4047000 comp rx=0 tx=0).stop 2024-03-31T23:25:52.272 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:52.270+0000 7f16f41b0740 1 -- 172.21.15.46:0/3584949743 shutdown_connections 2024-03-31T23:25:52.272 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:52.270+0000 7f16f41b0740 1 --2- 172.21.15.46:0/3584949743 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564150dbbdb0 0x564150daeb40 unknown :-1 s=CLOSED pgs=883 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:25:52.272 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:52.270+0000 7f16f41b0740 1 --2- 172.21.15.46:0/3584949743 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564150db2fe0 0x564150db97b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:25:52.272 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:52.270+0000 7f16f41b0740 1 --2- 172.21.15.46:0/3584949743 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x564150db26c0 0x564150dcffd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:25:52.272 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:52.270+0000 7f16f41b0740 1 -- 172.21.15.46:0/3584949743 >> 172.21.15.46:0/3584949743 conn(0x564150dacfe0 msgr2=0x564150dae430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:25:52.272 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:52.270+0000 7f16f41b0740 1 -- 172.21.15.46:0/3584949743 shutdown_connections 2024-03-31T23:25:52.272 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:52.270+0000 7f16f41b0740 1 -- 172.21.15.46:0/3584949743 wait complete. 2024-03-31T23:25:52.272 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:52.270+0000 7f16f41b0740 1 -- start start 2024-03-31T23:25:52.272 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:52.270+0000 7f16f41b0740 1 Processor -- start 2024-03-31T23:25:52.275 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:52.270+0000 7f16f41b0740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x564150db26c0 0x564150dcc600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:25:52.275 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:52.271+0000 7f16f41b0740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564150db2fe0 0x564150db5570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:25:52.275 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:52.271+0000 7f16f41b0740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564150dbbdb0 0x564150db5ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:25:52.275 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:52.271+0000 7f16f41b0740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f16ec0728a0 con 0x564150db2fe0 2024-03-31T23:25:52.275 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:52.271+0000 7f16f41b0740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f16ec1a3b10 con 0x564150db26c0 2024-03-31T23:25:52.275 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:52.271+0000 7f16f41b0740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f16ec072f70 con 0x564150dbbdb0 2024-03-31T23:25:52.275 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:52.271+0000 7f16f41b0740 10 client.0 _ll_register_callbacks cb 0x7f16dc096f10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T23:25:52.275 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:52.271+0000 7f16ea7fc640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564150db2fe0 0x564150db5570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:25:52.275 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:52.271+0000 7f16eaffd640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x564150db26c0 0x564150dcc600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:25:52.275 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:52.271+0000 7f16eaffd640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x564150db26c0 0x564150dcc600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.46:45520/0 (socket says 172.21.15.46:45520) 2024-03-31T23:25:52.275 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:52.271+0000 7f16eaffd640 1 -- 172.21.15.46:0/2793571332 learned_addr learned my addr 172.21.15.46:0/2793571332 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:25:52.275 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:52.271+0000 7f16eb7fe640 1 --2- 172.21.15.46:0/2793571332 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564150dbbdb0 0x564150db5ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:25:52.275 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:52.271+0000 7f16eaffd640 1 -- 172.21.15.46:0/2793571332 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564150dbbdb0 msgr2=0x564150db5ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:25:52.276 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:52.271+0000 7f16eaffd640 1 --2- 172.21.15.46:0/2793571332 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564150dbbdb0 0x564150db5ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:25:52.276 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:52.271+0000 7f16eaffd640 1 -- 172.21.15.46:0/2793571332 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564150db2fe0 msgr2=0x564150db5570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:25:52.276 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:52.271+0000 7f16eaffd640 1 --2- 172.21.15.46:0/2793571332 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564150db2fe0 0x564150db5570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:25:52.276 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:52.271+0000 7f16eaffd640 1 -- 172.21.15.46:0/2793571332 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564150db5ff0 con 0x564150db26c0 2024-03-31T23:25:52.276 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:52.271+0000 7f16eb7fe640 1 --2- 172.21.15.46:0/2793571332 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564150dbbdb0 0x564150db5ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:25:52.276 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:52.271+0000 7f16ea7fc640 1 --2- 172.21.15.46:0/2793571332 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564150db2fe0 0x564150db5570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:25:52.276 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:52.271+0000 7f16eaffd640 1 --2- 172.21.15.46:0/2793571332 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x564150db26c0 0x564150dcc600 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f16dc09ef80 tx=0x7f16dc009d70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=547cda7a4fe5342c server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:25:52.276 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:52.271+0000 7f16ceff5640 10 client.0 ms_handle_connect on v2:172.21.15.142:3300/0 2024-03-31T23:25:52.276 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:52.272+0000 7f16ceff5640 1 -- 172.21.15.46:0/2793571332 <== mon.2 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16dc09a5d0 con 0x564150db26c0 2024-03-31T23:25:52.276 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:52.272+0000 7f16ceff5640 1 -- 172.21.15.46:0/2793571332 <== mon.2 v2:172.21.15.142:3300/0 2 ==== osd_map(112..112 src has 1..112) ==== 9554+0+0 (secure 0 0 0) 0x7f16dc09a2c0 con 0x564150db26c0 2024-03-31T23:25:52.276 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:52.272+0000 7f16f41b0740 1 -- 172.21.15.46:0/2793571332 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f16e4034bd0 con 0x564150db26c0 2024-03-31T23:25:52.276 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:52.272+0000 7f16ceff5640 1 -- 172.21.15.46:0/2793571332 <== mon.2 v2:172.21.15.142:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7f16dc09ac70 con 0x564150db26c0 2024-03-31T23:25:52.276 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:52.272+0000 7f16f41b0740 10 client.27131 fetch_fsmap learned FSMap version 38 2024-03-31T23:25:52.276 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:52.272+0000 7f16f41b0740 1 -- 172.21.15.46:0/2793571332 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x564150e3ac10 con 0x564150db26c0 2024-03-31T23:25:52.276 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:52.272+0000 7f16ceff5640 1 -- 172.21.15.46:0/2793571332 <== mon.2 v2:172.21.15.142:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f16e4034bd0 con 0x564150db26c0 2024-03-31T23:25:52.277 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:52.272+0000 7f16ceff5640 1 -- 172.21.15.46:0/2793571332 <== mon.2 v2:172.21.15.142:3300/0 5 ==== fsmap(e 38) ==== 8370+0+0 (secure 0 0 0) 0x7f16dc085220 con 0x564150db26c0 2024-03-31T23:25:52.277 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:52.272+0000 7f16f41b0740 10 client.27131 fetch_fsmap finished waiting for FSMap version 38 2024-03-31T23:25:52.277 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:52.272+0000 7f16f41b0740 10 client.27131 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-31T23:25:52.277 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:52.272+0000 7f16f41b0740 20 client.27131 populate_metadata read hostname 'smithi046' 2024-03-31T23:25:52.277 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:52.272+0000 7f16f41b0740 1 --2- 172.21.15.46:0/2793571332 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x564150e3b330 0x564150e3d750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:25:52.277 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:52.272+0000 7f16f41b0740 4 client.27131 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-31T23:25:52.277 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:52.272+0000 7f16f41b0740 1 -- 172.21.15.46:0/2793571332 --> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] -- command(tid 0: {"prefix": "flush journal"}) -- 0x564150d22e30 con 0x564150e3b330 2024-03-31T23:25:52.277 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:52.273+0000 7f16ea7fc640 1 --2- 172.21.15.46:0/2793571332 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x564150e3b330 0x564150e3d750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:25:52.277 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:52.273+0000 7f16ceff5640 10 client.27131 ms_handle_connect on v2:172.21.15.142:6838/2519143918 2024-03-31T23:25:52.277 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:52.273+0000 7f16ea7fc640 1 --2- 172.21.15.46:0/2793571332 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x564150e3b330 0x564150e3d750 crc :-1 s=READY pgs=202 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6111764ffe98cef0 server_cookie=b6076c8beab3cc60 in_seq=0 out_seq=0 2024-03-31T23:25:52.399 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:52.397+0000 7f16ceff5640 1 -- 172.21.15.46:0/2793571332 <== mds.0 v2:172.21.15.142:6838/2519143918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x564150d22e30 con 0x564150e3b330 2024-03-31T23:25:52.399 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:52.397+0000 7f16ceff5640 10 client.27131 handle_command_reply: tid=0 2024-03-31T23:25:52.399 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:52.397+0000 7f16f41b0740 1 client.27131 shutdown 2024-03-31T23:25:52.399 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:52.398+0000 7f16f41b0740 1 -- 172.21.15.46:0/2793571332 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x564150db26c0 msgr2=0x564150dcc600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:25:52.399 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:52.398+0000 7f16f41b0740 1 --2- 172.21.15.46:0/2793571332 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x564150db26c0 0x564150dcc600 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f16dc09ef80 tx=0x7f16dc009d70 comp rx=0 tx=0).stop 2024-03-31T23:25:52.399 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:52.398+0000 7f16f41b0740 1 -- 172.21.15.46:0/2793571332 shutdown_connections 2024-03-31T23:25:52.400 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:52.398+0000 7f16f41b0740 1 --2- 172.21.15.46:0/2793571332 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x564150e3b330 0x564150e3d750 crc :-1 s=READY pgs=202 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:25:52.400 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:52.398+0000 7f16f41b0740 1 --2- 172.21.15.46:0/2793571332 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564150dbbdb0 0x564150db5ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:25:52.400 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:52.398+0000 7f16f41b0740 1 --2- 172.21.15.46:0/2793571332 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564150db2fe0 0x564150db5570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:25:52.400 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:52.398+0000 7f16ceff5640 0 client.27131 ms_handle_reset on v2:172.21.15.142:6838/2519143918 2024-03-31T23:25:52.400 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:52.398+0000 7f16f41b0740 1 --2- 172.21.15.46:0/2793571332 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x564150db26c0 0x564150dcc600 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:25:52.400 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:52.398+0000 7f16f41b0740 1 -- 172.21.15.46:0/2793571332 >> 172.21.15.46:0/2793571332 conn(0x564150dacfe0 msgr2=0x564150dabd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:25:52.400 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:52.398+0000 7f16f41b0740 1 -- 172.21.15.46:0/2793571332 shutdown_connections 2024-03-31T23:25:52.400 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:52.398+0000 7f16f41b0740 1 -- 172.21.15.46:0/2793571332 wait complete. 2024-03-31T23:25:52.400 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:52.398+0000 7f16f41b0740 20 client.27131 trim_cache size 0 max 16384 2024-03-31T23:25:52.401 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout:{ 2024-03-31T23:25:52.401 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout: "message": "", 2024-03-31T23:25:52.401 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout: "return_code": 0 2024-03-31T23:25:52.401 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout:} 2024-03-31T23:25:52.401 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:52.399+0000 7f16f14fa640 1 -- 172.21.15.46:0/4123151582 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f16d40a2730 msgr2=0x7f16d40a4bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:25:52.401 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:52.399+0000 7f16f14fa640 1 --2- 172.21.15.46:0/4123151582 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f16d40a2730 0x7f16d40a4bf0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f16e4004af0 tx=0x7f16e4009360 comp rx=0 tx=0).stop 2024-03-31T23:25:52.401 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:52.399+0000 7f16f14fa640 1 -- 172.21.15.46:0/4123151582 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f16ec100930 msgr2=0x7f16ec19c8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:25:52.402 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:52.399+0000 7f16f14fa640 1 --2- 172.21.15.46:0/4123151582 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f16ec100930 0x7f16ec19c8c0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f16dc00d950 tx=0x7f16dc00de10 comp rx=0 tx=0).stop 2024-03-31T23:25:52.402 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:52.400+0000 7f16f14fa640 1 -- 172.21.15.46:0/4123151582 shutdown_connections 2024-03-31T23:25:52.402 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:52.400+0000 7f16f14fa640 1 --2- 172.21.15.46:0/4123151582 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f16d40a2730 0x7f16d40a4bf0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:25:52.402 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:52.400+0000 7f16f14fa640 1 --2- 172.21.15.46:0/4123151582 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f16ec106720 0x7f16ec196a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:25:52.402 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:52.400+0000 7f16f14fa640 1 --2- 172.21.15.46:0/4123151582 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f16ec1012d0 0x7f16ec19ce00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:25:52.402 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:52.400+0000 7f16f14fa640 1 --2- 172.21.15.46:0/4123151582 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f16ec100930 0x7f16ec19c8c0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:25:52.402 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:52.400+0000 7f16f14fa640 1 -- 172.21.15.46:0/4123151582 >> 172.21.15.46:0/4123151582 conn(0x7f16ec077630 msgr2=0x7f16ec0ff450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:25:52.402 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:52.400+0000 7f16f14fa640 1 -- 172.21.15.46:0/4123151582 shutdown_connections 2024-03-31T23:25:52.402 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:25:52.400+0000 7f16f14fa640 1 -- 172.21.15.46:0/4123151582 wait complete. 2024-03-31T23:25:52.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:25:52 smithi046 ceph-mon[32498]: pgmap v443: 129 pgs: 129 active+clean; 417 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 4.0 MiB/s wr, 251 op/s 2024-03-31T23:25:52.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:25:52 smithi142 ceph-mon[35271]: pgmap v443: 129 pgs: 129 active+clean; 417 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 4.0 MiB/s wr, 251 op/s 2024-03-31T23:25:53.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:25:52 smithi162 ceph-mon[36237]: pgmap v443: 129 pgs: 129 active+clean; 417 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 4.0 MiB/s wr, 251 op/s 2024-03-31T23:25:53.267 INFO:tasks.workunit.client.0.smithi046.stdout: 10 9229 11.51 MB/sec warmup 58 sec latency 119.275 ms 2024-03-31T23:25:54.267 INFO:tasks.workunit.client.0.smithi046.stdout: 10 9412 11.51 MB/sec warmup 59 sec latency 78.939 ms 2024-03-31T23:25:54.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:25:54 smithi046 ceph-mon[32498]: pgmap v444: 129 pgs: 129 active+clean; 417 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 4.0 MiB/s wr, 251 op/s 2024-03-31T23:25:54.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:25:54 smithi142 ceph-mon[35271]: pgmap v444: 129 pgs: 129 active+clean; 417 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 4.0 MiB/s wr, 251 op/s 2024-03-31T23:25:55.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:25:54 smithi162 ceph-mon[36237]: pgmap v444: 129 pgs: 129 active+clean; 417 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 4.0 MiB/s wr, 251 op/s 2024-03-31T23:25:55.267 INFO:tasks.workunit.client.0.smithi046.stdout: 10 9616 11.49 MB/sec warmup 60 sec latency 65.302 ms 2024-03-31T23:25:56.267 INFO:tasks.workunit.client.0.smithi046.stdout: 10 9786 11.48 MB/sec warmup 61 sec latency 58.848 ms 2024-03-31T23:25:56.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:25:56 smithi046 ceph-mon[32498]: pgmap v445: 129 pgs: 129 active+clean; 421 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 6.0 MiB/s wr, 396 op/s 2024-03-31T23:25:56.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:25:56 smithi142 ceph-mon[35271]: pgmap v445: 129 pgs: 129 active+clean; 421 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 6.0 MiB/s wr, 396 op/s 2024-03-31T23:25:57.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:25:56 smithi162 ceph-mon[36237]: pgmap v445: 129 pgs: 129 active+clean; 421 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 6.0 MiB/s wr, 396 op/s 2024-03-31T23:25:57.267 INFO:tasks.workunit.client.0.smithi046.stdout: 10 9907 11.38 MB/sec warmup 62 sec latency 168.058 ms 2024-03-31T23:25:58.267 INFO:tasks.workunit.client.0.smithi046.stdout: 10 10067 11.36 MB/sec warmup 63 sec latency 66.407 ms 2024-03-31T23:25:58.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:25:58 smithi046 ceph-mon[32498]: pgmap v446: 129 pgs: 129 active+clean; 421 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 3.7 MiB/s wr, 251 op/s 2024-03-31T23:25:58.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:25:58 smithi142 ceph-mon[35271]: pgmap v446: 129 pgs: 129 active+clean; 421 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 3.7 MiB/s wr, 251 op/s 2024-03-31T23:25:59.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:25:58 smithi162 ceph-mon[36237]: pgmap v446: 129 pgs: 129 active+clean; 421 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 3.7 MiB/s wr, 251 op/s 2024-03-31T23:25:59.267 INFO:tasks.workunit.client.0.smithi046.stdout: 10 10262 11.40 MB/sec warmup 64 sec latency 67.169 ms 2024-03-31T23:26:00.267 INFO:tasks.workunit.client.0.smithi046.stdout: 10 10457 11.37 MB/sec warmup 65 sec latency 66.783 ms 2024-03-31T23:26:00.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:26:00 smithi046 ceph-mon[32498]: pgmap v447: 129 pgs: 129 active+clean; 429 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 5.9 MiB/s wr, 355 op/s 2024-03-31T23:26:00.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:26:00 smithi142 ceph-mon[35271]: pgmap v447: 129 pgs: 129 active+clean; 429 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 5.9 MiB/s wr, 355 op/s 2024-03-31T23:26:01.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:26:00 smithi162 ceph-mon[36237]: pgmap v447: 129 pgs: 129 active+clean; 429 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 5.9 MiB/s wr, 355 op/s 2024-03-31T23:26:01.267 INFO:tasks.workunit.client.0.smithi046.stdout: 10 10621 11.29 MB/sec warmup 66 sec latency 85.531 ms 2024-03-31T23:26:02.267 INFO:tasks.workunit.client.0.smithi046.stdout: 10 10749 11.18 MB/sec warmup 67 sec latency 94.307 ms 2024-03-31T23:26:02.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:26:02 smithi142 ceph-mon[35271]: pgmap v448: 129 pgs: 129 active+clean; 429 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 4.2 MiB/s wr, 249 op/s 2024-03-31T23:26:02.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:26:02 smithi142 ceph-mon[35271]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-31T23:26:03.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:26:02 smithi162 ceph-mon[36237]: pgmap v448: 129 pgs: 129 active+clean; 429 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 4.2 MiB/s wr, 249 op/s 2024-03-31T23:26:03.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:26:02 smithi162 ceph-mon[36237]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-31T23:26:03.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:26:02 smithi046 ceph-mon[32498]: pgmap v448: 129 pgs: 129 active+clean; 429 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 4.2 MiB/s wr, 249 op/s 2024-03-31T23:26:03.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:26:02 smithi046 ceph-mon[32498]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-31T23:26:03.267 INFO:tasks.workunit.client.0.smithi046.stdout: 10 10897 11.14 MB/sec warmup 68 sec latency 107.095 ms 2024-03-31T23:26:03.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:26:03 smithi142 ceph-mon[35271]: osdmap e113: 12 total, 12 up, 12 in 2024-03-31T23:26:03.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:26:03 smithi142 ceph-mon[35271]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:26:03.944 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:26:03 smithi142 ceph-mon[35271]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-31T23:26:03.944 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:26:03 smithi142 ceph-mon[35271]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' 2024-03-31T23:26:04.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:26:03 smithi162 ceph-mon[36237]: osdmap e113: 12 total, 12 up, 12 in 2024-03-31T23:26:04.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:26:03 smithi162 ceph-mon[36237]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:26:04.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:26:03 smithi162 ceph-mon[36237]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-31T23:26:04.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:26:03 smithi162 ceph-mon[36237]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' 2024-03-31T23:26:04.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:26:03 smithi046 ceph-mon[32498]: osdmap e113: 12 total, 12 up, 12 in 2024-03-31T23:26:04.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:26:03 smithi046 ceph-mon[32498]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:26:04.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:26:03 smithi046 ceph-mon[32498]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-31T23:26:04.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:26:03 smithi046 ceph-mon[32498]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' 2024-03-31T23:26:04.267 INFO:tasks.workunit.client.0.smithi046.stdout: 10 11080 11.19 MB/sec warmup 69 sec latency 92.826 ms 2024-03-31T23:26:04.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:26:04 smithi142 ceph-mon[35271]: pgmap v450: 129 pgs: 129 active+clean; 429 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 5.1 MiB/s wr, 299 op/s 2024-03-31T23:26:05.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:26:04 smithi162 ceph-mon[36237]: pgmap v450: 129 pgs: 129 active+clean; 429 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 5.1 MiB/s wr, 299 op/s 2024-03-31T23:26:05.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:26:04 smithi046 ceph-mon[32498]: pgmap v450: 129 pgs: 129 active+clean; 429 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 5.1 MiB/s wr, 299 op/s 2024-03-31T23:26:05.267 INFO:tasks.workunit.client.0.smithi046.stdout: 10 11251 11.17 MB/sec warmup 70 sec latency 133.274 ms 2024-03-31T23:26:05.529 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:05.527+0000 7f9674e23640 1 -- 172.21.15.46:0/1109342851 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9670101740 msgr2=0x7f9670101bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:26:05.529 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:05.527+0000 7f9674e23640 1 --2- 172.21.15.46:0/1109342851 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9670101740 0x7f9670101bc0 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7f9664009930 tx=0x7f966402bc30 comp rx=0 tx=0).stop 2024-03-31T23:26:05.529 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:05.528+0000 7f9674e23640 1 -- 172.21.15.46:0/1109342851 shutdown_connections 2024-03-31T23:26:05.529 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:05.528+0000 7f9674e23640 1 --2- 172.21.15.46:0/1109342851 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9670102100 0x7f967010e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:26:05.529 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:05.528+0000 7f9674e23640 1 --2- 172.21.15.46:0/1109342851 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9670101740 0x7f9670101bc0 unknown :-1 s=CLOSED pgs=884 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:26:05.530 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:05.528+0000 7f9674e23640 1 --2- 172.21.15.46:0/1109342851 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9670107750 0x7f9670107b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:26:05.530 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:05.528+0000 7f9674e23640 1 -- 172.21.15.46:0/1109342851 >> 172.21.15.46:0/1109342851 conn(0x7f96700fd210 msgr2=0x7f96700ff630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:26:05.530 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:05.528+0000 7f9674e23640 1 -- 172.21.15.46:0/1109342851 shutdown_connections 2024-03-31T23:26:05.530 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:05.528+0000 7f9674e23640 1 -- 172.21.15.46:0/1109342851 wait complete. 2024-03-31T23:26:05.531 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:05.529+0000 7f9674e23640 1 Processor -- start 2024-03-31T23:26:05.531 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:05.529+0000 7f9674e23640 1 -- start start 2024-03-31T23:26:05.531 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:05.529+0000 7f9674e23640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9670101740 0x7f9670197f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:26:05.531 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:05.529+0000 7f9674e23640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9670102100 0x7f96701984a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:26:05.531 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:05.529+0000 7f9674e23640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9670107750 0x7f967019c820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:26:05.531 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:05.529+0000 7f9674e23640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f9670111a20 con 0x7f9670101740 2024-03-31T23:26:05.531 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:05.529+0000 7f9674e23640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f96701118a0 con 0x7f9670102100 2024-03-31T23:26:05.531 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:05.529+0000 7f9674e23640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f9670111ba0 con 0x7f9670107750 2024-03-31T23:26:05.531 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:05.530+0000 7f966ed9d640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9670107750 0x7f967019c820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:26:05.531 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:05.530+0000 7f966dd9b640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9670102100 0x7f96701984a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:26:05.531 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:05.530+0000 7f966ed9d640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9670107750 0x7f967019c820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.46:56778/0 (socket says 172.21.15.46:56778) 2024-03-31T23:26:05.531 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:05.530+0000 7f966ed9d640 1 -- 172.21.15.46:0/3958673720 learned_addr learned my addr 172.21.15.46:0/3958673720 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:26:05.532 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:05.530+0000 7f966e59c640 1 --2- 172.21.15.46:0/3958673720 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9670101740 0x7f9670197f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:26:05.532 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:05.530+0000 7f966ed9d640 1 -- 172.21.15.46:0/3958673720 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9670102100 msgr2=0x7f96701984a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:26:05.532 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:05.530+0000 7f966ed9d640 1 --2- 172.21.15.46:0/3958673720 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9670102100 0x7f96701984a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:26:05.532 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:05.530+0000 7f966ed9d640 1 -- 172.21.15.46:0/3958673720 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9670101740 msgr2=0x7f9670197f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:26:05.532 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:05.530+0000 7f966ed9d640 1 --2- 172.21.15.46:0/3958673720 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9670101740 0x7f9670197f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:26:05.532 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:05.530+0000 7f966ed9d640 1 -- 172.21.15.46:0/3958673720 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f967019cf50 con 0x7f9670107750 2024-03-31T23:26:05.533 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:05.530+0000 7f966dd9b640 1 --2- 172.21.15.46:0/3958673720 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9670102100 0x7f96701984a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:26:05.533 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:05.530+0000 7f966ed9d640 1 --2- 172.21.15.46:0/3958673720 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9670107750 0x7f967019c820 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f966800eb10 tx=0x7f966800c2d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:26:05.533 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:05.530+0000 7f965f7fe640 1 -- 172.21.15.46:0/3958673720 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9668004b20 con 0x7f9670107750 2024-03-31T23:26:05.533 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:05.531+0000 7f9674e23640 1 -- 172.21.15.46:0/3958673720 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f967019d240 con 0x7f9670107750 2024-03-31T23:26:05.533 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:05.531+0000 7f9674e23640 1 -- 172.21.15.46:0/3958673720 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9670073980 con 0x7f9670107750 2024-03-31T23:26:05.533 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:05.531+0000 7f965f7fe640 1 -- 172.21.15.46:0/3958673720 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9668004cc0 con 0x7f9670107750 2024-03-31T23:26:05.533 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:05.531+0000 7f965f7fe640 1 -- 172.21.15.46:0/3958673720 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9668010690 con 0x7f9670107750 2024-03-31T23:26:05.534 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:05.533+0000 7f965f7fe640 1 -- 172.21.15.46:0/3958673720 <== mon.1 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 25) ==== 137451+0+0 (secure 0 0 0) 0x7f9668005020 con 0x7f9670107750 2024-03-31T23:26:05.535 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:05.534+0000 7f9677a07740 1 Processor -- start 2024-03-31T23:26:05.536 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:05.534+0000 7f965f7fe640 1 --2- 172.21.15.46:0/3958673720 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f96580a26b0 0x7f96580a4b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:26:05.536 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:05.534+0000 7f9677a07740 1 -- start start 2024-03-31T23:26:05.536 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:05.534+0000 7f966dd9b640 1 --2- 172.21.15.46:0/3958673720 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f96580a26b0 0x7f96580a4b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:26:05.536 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:05.534+0000 7f965f7fe640 1 -- 172.21.15.46:0/3958673720 <== mon.1 v2:172.21.15.162:3300/0 5 ==== osd_map(113..113 src has 1..113) ==== 9642+0+0 (secure 0 0 0) 0x7f96680cf520 con 0x7f9670107750 2024-03-31T23:26:05.536 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:05.534+0000 7f9677a07740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a949bc96c0 0x55a949bc9aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:26:05.536 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:05.534+0000 7f9677a07740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a949bc9fe0 0x55a949bd2870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:26:05.536 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:05.534+0000 7f9677a07740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a949bd2db0 0x55a949bd51a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:26:05.536 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:05.534+0000 7f9677a07740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55a949b2c8f0 con 0x55a949bc9fe0 2024-03-31T23:26:05.536 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:05.534+0000 7f9677a07740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55a949ad58d0 con 0x55a949bd2db0 2024-03-31T23:26:05.536 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:05.534+0000 7f9677a07740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55a949b28990 con 0x55a949bc96c0 2024-03-31T23:26:05.536 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:05.535+0000 7f966e59c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a949bc96c0 0x55a949bc9aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:26:05.537 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:05.535+0000 7f966e59c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a949bc96c0 0x55a949bc9aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.46:56794/0 (socket says 172.21.15.46:56794) 2024-03-31T23:26:05.537 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:05.535+0000 7f966e59c640 1 -- 172.21.15.46:0/2719325020 learned_addr learned my addr 172.21.15.46:0/2719325020 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:26:05.537 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:05.535+0000 7f966dd9b640 1 --2- 172.21.15.46:0/2719325020 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a949bc9fe0 0x55a949bd2870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:26:05.537 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:05.535+0000 7f966ed9d640 1 --2- 172.21.15.46:0/2719325020 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a949bd2db0 0x55a949bd51a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:26:05.537 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:05.535+0000 7f966e59c640 1 -- 172.21.15.46:0/2719325020 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a949bd2db0 msgr2=0x55a949bd51a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:26:05.537 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:05.535+0000 7f966e59c640 1 --2- 172.21.15.46:0/2719325020 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a949bd2db0 0x55a949bd51a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:26:05.537 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:05.535+0000 7f966e59c640 1 -- 172.21.15.46:0/2719325020 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a949bc9fe0 msgr2=0x55a949bd2870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:26:05.537 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:05.535+0000 7f966e59c640 1 --2- 172.21.15.46:0/2719325020 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a949bc9fe0 0x55a949bd2870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:26:05.537 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:05.535+0000 7f966e59c640 1 -- 172.21.15.46:0/2719325020 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a949841210 con 0x55a949bc96c0 2024-03-31T23:26:05.537 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:05.535+0000 7f966ed9d640 1 --2- 172.21.15.46:0/2719325020 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a949bd2db0 0x55a949bd51a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T23:26:05.537 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:05.535+0000 7f966dd9b640 1 --2- 172.21.15.46:0/2719325020 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a949bc9fe0 0x55a949bd2870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T23:26:05.541 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:05.535+0000 7f966dd9b640 1 --2- 172.21.15.46:0/3958673720 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f96580a26b0 0x7f96580a4b70 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f9664009900 tx=0x7f96640043b0 comp rx=0 tx=0).ready entity=mgr.15654 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:26:05.541 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:05.535+0000 7f966e59c640 1 --2- 172.21.15.46:0/2719325020 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a949bc96c0 0x55a949bc9aa0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f966000bc90 tx=0x7f96600107a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7e191d6ecd5418ad server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:26:05.541 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:05.535+0000 7f965cff9640 1 -- 172.21.15.46:0/2719325020 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f966001c480 con 0x55a949bc96c0 2024-03-31T23:26:05.541 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:05.535+0000 7f965cff9640 1 -- 172.21.15.46:0/2719325020 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f966001cad0 con 0x55a949bc96c0 2024-03-31T23:26:05.541 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:05.535+0000 7f965cff9640 1 -- 172.21.15.46:0/2719325020 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9660011a80 con 0x55a949bc96c0 2024-03-31T23:26:05.541 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:05.536+0000 7f9677a07740 1 -- 172.21.15.46:0/2719325020 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a949bc96c0 msgr2=0x55a949bc9aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:26:05.541 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:05.536+0000 7f9677a07740 1 --2- 172.21.15.46:0/2719325020 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a949bc96c0 0x55a949bc9aa0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f966000bc90 tx=0x7f96600107a0 comp rx=0 tx=0).stop 2024-03-31T23:26:05.542 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:05.536+0000 7f9677a07740 1 -- 172.21.15.46:0/2719325020 shutdown_connections 2024-03-31T23:26:05.542 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:05.536+0000 7f9677a07740 1 --2- 172.21.15.46:0/2719325020 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a949bd2db0 0x55a949bd51a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:26:05.542 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:05.536+0000 7f9677a07740 1 --2- 172.21.15.46:0/2719325020 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a949bc9fe0 0x55a949bd2870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:26:05.542 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:05.536+0000 7f9677a07740 1 --2- 172.21.15.46:0/2719325020 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a949bc96c0 0x55a949bc9aa0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:26:05.542 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:05.536+0000 7f9677a07740 1 -- 172.21.15.46:0/2719325020 >> 172.21.15.46:0/2719325020 conn(0x55a949bc3fe0 msgr2=0x55a949bc43c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:26:05.542 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:05.536+0000 7f9677a07740 1 -- 172.21.15.46:0/2719325020 shutdown_connections 2024-03-31T23:26:05.542 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:05.536+0000 7f9677a07740 1 -- 172.21.15.46:0/2719325020 wait complete. 2024-03-31T23:26:05.542 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:05.537+0000 7f9677a07740 1 -- start start 2024-03-31T23:26:05.542 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:05.537+0000 7f9677a07740 1 Processor -- start 2024-03-31T23:26:05.542 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:05.537+0000 7f9677a07740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a949bc96c0 0x55a949bcfe20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:26:05.542 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:05.537+0000 7f9677a07740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a949bc9fe0 0x55a949bd0360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:26:05.542 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:05.537+0000 7f9677a07740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a949bd2db0 0x55a949bd08a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:26:05.542 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:05.537+0000 7f9677a07740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55a949bd59e0 con 0x55a949bc96c0 2024-03-31T23:26:05.542 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:05.537+0000 7f9677a07740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55a949acaf80 con 0x55a949bc9fe0 2024-03-31T23:26:05.543 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:05.537+0000 7f9677a07740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55a949bd5b60 con 0x55a949bd2db0 2024-03-31T23:26:05.543 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:05.537+0000 7f9677a07740 10 client.0 _ll_register_callbacks cb 0x55a949b23290 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T23:26:05.543 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:05.537+0000 7f966e59c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a949bc96c0 0x55a949bcfe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:26:05.543 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:05.537+0000 7f966dd9b640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a949bc9fe0 0x55a949bd0360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:26:05.543 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:05.537+0000 7f966e59c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a949bc96c0 0x55a949bcfe20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37844/0 (socket says 172.21.15.46:37844) 2024-03-31T23:26:05.543 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:05.537+0000 7f966ed9d640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a949bd2db0 0x55a949bd08a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:26:05.543 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:05.537+0000 7f966e59c640 1 -- 172.21.15.46:0/1338524863 learned_addr learned my addr 172.21.15.46:0/1338524863 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:26:05.543 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:05.538+0000 7f966e59c640 1 -- 172.21.15.46:0/1338524863 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a949bd2db0 msgr2=0x55a949bd08a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:26:05.543 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:05.538+0000 7f966e59c640 1 --2- 172.21.15.46:0/1338524863 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a949bd2db0 0x55a949bd08a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:26:05.543 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:05.538+0000 7f966e59c640 1 -- 172.21.15.46:0/1338524863 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a949bc9fe0 msgr2=0x55a949bd0360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:26:05.543 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:05.538+0000 7f966e59c640 1 --2- 172.21.15.46:0/1338524863 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a949bc9fe0 0x55a949bd0360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:26:05.543 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:05.538+0000 7f966e59c640 1 -- 172.21.15.46:0/1338524863 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a94990d620 con 0x55a949bc96c0 2024-03-31T23:26:05.543 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:05.538+0000 7f966ed9d640 1 --2- 172.21.15.46:0/1338524863 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a949bd2db0 0x55a949bd08a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:26:05.543 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:05.538+0000 7f966e59c640 1 --2- 172.21.15.46:0/1338524863 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a949bc96c0 0x55a949bcfe20 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7f9660006010 tx=0x7f966000bec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1cf6e6d1377c4c6c server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:26:05.544 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:05.538+0000 7f9655ffb640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-31T23:26:05.544 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:05.538+0000 7f9655ffb640 1 -- 172.21.15.46:0/1338524863 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9660005040 con 0x55a949bc96c0 2024-03-31T23:26:05.544 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:05.538+0000 7f9655ffb640 1 -- 172.21.15.46:0/1338524863 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(113..113 src has 1..113) ==== 9642+0+0 (secure 0 0 0) 0x7f96600051e0 con 0x55a949bc96c0 2024-03-31T23:26:05.544 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:05.538+0000 7f9677a07740 1 -- 172.21.15.46:0/1338524863 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f966001cad0 con 0x55a949bc96c0 2024-03-31T23:26:05.544 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:05.539+0000 7f9655ffb640 1 -- 172.21.15.46:0/1338524863 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7f9660032050 con 0x55a949bc96c0 2024-03-31T23:26:05.544 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:05.539+0000 7f9677a07740 10 client.17313 fetch_fsmap learned FSMap version 38 2024-03-31T23:26:05.544 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:05.539+0000 7f9677a07740 1 -- 172.21.15.46:0/1338524863 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x55a949be1900 con 0x55a949bc96c0 2024-03-31T23:26:05.544 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:05.539+0000 7f9655ffb640 1 -- 172.21.15.46:0/1338524863 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f966001cad0 con 0x55a949bc96c0 2024-03-31T23:26:05.544 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:05.539+0000 7f9655ffb640 1 -- 172.21.15.46:0/1338524863 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 38) ==== 8370+0+0 (secure 0 0 0) 0x7f96600046c0 con 0x55a949bc96c0 2024-03-31T23:26:05.544 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:05.539+0000 7f9677a07740 10 client.17313 fetch_fsmap finished waiting for FSMap version 38 2024-03-31T23:26:05.544 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:05.539+0000 7f9677a07740 10 client.17313 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-31T23:26:05.544 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:05.539+0000 7f9677a07740 20 client.17313 populate_metadata read hostname 'smithi046' 2024-03-31T23:26:05.544 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:05.539+0000 7f9677a07740 1 --2- 172.21.15.46:0/1338524863 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55a949be3140 0x55a949be55e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:26:05.544 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:05.539+0000 7f9677a07740 4 client.17313 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-31T23:26:05.545 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:05.539+0000 7f9677a07740 1 -- 172.21.15.46:0/1338524863 --> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a949b40120 con 0x55a949be3140 2024-03-31T23:26:05.545 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:05.539+0000 7f966dd9b640 1 --2- 172.21.15.46:0/1338524863 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55a949be3140 0x55a949be55e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:26:05.545 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:05.540+0000 7f966dd9b640 1 --2- 172.21.15.46:0/1338524863 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55a949be3140 0x55a949be55e0 crc :-1 s=READY pgs=203 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=afd2e3833d648e68 server_cookie=adb68be3a05c99a3 in_seq=0 out_seq=0 2024-03-31T23:26:05.545 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:05.540+0000 7f9655ffb640 10 client.17313 ms_handle_connect on v2:172.21.15.142:6838/2519143918 2024-03-31T23:26:05.546 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:05.541+0000 7f9655ffb640 1 -- 172.21.15.46:0/1338524863 <== mds.0 v2:172.21.15.142:6838/2519143918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55a949b40120 con 0x55a949be3140 2024-03-31T23:26:05.546 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:05.541+0000 7f9655ffb640 10 client.17313 handle_command_reply: tid=0 2024-03-31T23:26:05.546 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:05.541+0000 7f9677a07740 1 client.17313 shutdown 2024-03-31T23:26:05.546 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:05.542+0000 7f9677a07740 1 -- 172.21.15.46:0/1338524863 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a949bc96c0 msgr2=0x55a949bcfe20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:26:05.546 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:05.542+0000 7f9677a07740 1 --2- 172.21.15.46:0/1338524863 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a949bc96c0 0x55a949bcfe20 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7f9660006010 tx=0x7f966000bec0 comp rx=0 tx=0).stop 2024-03-31T23:26:05.546 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:05.542+0000 7f9677a07740 1 -- 172.21.15.46:0/1338524863 shutdown_connections 2024-03-31T23:26:05.547 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:05.542+0000 7f9677a07740 1 --2- 172.21.15.46:0/1338524863 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55a949be3140 0x55a949be55e0 crc :-1 s=READY pgs=203 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:26:05.547 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:05.542+0000 7f9677a07740 1 --2- 172.21.15.46:0/1338524863 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a949bd2db0 0x55a949bd08a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:26:05.547 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:05.542+0000 7f9677a07740 1 --2- 172.21.15.46:0/1338524863 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a949bc9fe0 0x55a949bd0360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:26:05.547 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:05.542+0000 7f9655ffb640 0 client.17313 ms_handle_reset on v2:172.21.15.142:6838/2519143918 2024-03-31T23:26:05.547 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:05.542+0000 7f9677a07740 1 --2- 172.21.15.46:0/1338524863 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a949bc96c0 0x55a949bcfe20 unknown :-1 s=CLOSED pgs=885 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:26:05.547 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:05.542+0000 7f9677a07740 1 -- 172.21.15.46:0/1338524863 >> 172.21.15.46:0/1338524863 conn(0x55a949bc3fe0 msgr2=0x55a949bc5d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:26:05.547 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:05.542+0000 7f9677a07740 1 -- 172.21.15.46:0/1338524863 shutdown_connections 2024-03-31T23:26:05.547 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:05.542+0000 7f9677a07740 1 -- 172.21.15.46:0/1338524863 wait complete. 2024-03-31T23:26:05.547 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:05.542+0000 7f9677a07740 20 client.17313 trim_cache size 0 max 16384 2024-03-31T23:26:05.570 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:05.566+0000 7f9677a07740 1 Processor -- start 2024-03-31T23:26:05.571 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:05.566+0000 7f9677a07740 1 -- start start 2024-03-31T23:26:05.571 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:05.566+0000 7f9677a07740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a949bc96c0 0x55a949bd2550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:26:05.571 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:05.566+0000 7f9677a07740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a949bc9fe0 0x55a949bde780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:26:05.571 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:05.566+0000 7f9677a07740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a949bd2db0 0x55a949bc43c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:26:05.571 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:05.566+0000 7f9677a07740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f96701b39d0 con 0x55a949bc96c0 2024-03-31T23:26:05.571 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:05.566+0000 7f9677a07740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f96701b11e0 con 0x55a949bd2db0 2024-03-31T23:26:05.571 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:05.566+0000 7f9677a07740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f96701af050 con 0x55a949bc9fe0 2024-03-31T23:26:05.571 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:05.566+0000 7f966e59c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a949bc96c0 0x55a949bd2550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:26:05.571 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:05.566+0000 7f966e59c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a949bc96c0 0x55a949bd2550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37848/0 (socket says 172.21.15.46:37848) 2024-03-31T23:26:05.571 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:05.566+0000 7f966e59c640 1 -- 172.21.15.46:0/3733704459 learned_addr learned my addr 172.21.15.46:0/3733704459 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:26:05.571 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:05.566+0000 7f966dd9b640 1 --2- 172.21.15.46:0/3733704459 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a949bc9fe0 0x55a949bde780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:26:05.571 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:05.566+0000 7f966ed9d640 1 --2- 172.21.15.46:0/3733704459 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a949bd2db0 0x55a949bc43c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:26:05.572 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:05.567+0000 7f966e59c640 1 -- 172.21.15.46:0/3733704459 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a949bc9fe0 msgr2=0x55a949bde780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:26:05.572 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:05.567+0000 7f966e59c640 1 --2- 172.21.15.46:0/3733704459 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a949bc9fe0 0x55a949bde780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:26:05.572 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:05.567+0000 7f966e59c640 1 -- 172.21.15.46:0/3733704459 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a949bd2db0 msgr2=0x55a949bc43c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:26:05.572 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:05.567+0000 7f966e59c640 1 --2- 172.21.15.46:0/3733704459 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a949bd2db0 0x55a949bc43c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:26:05.572 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:05.567+0000 7f966e59c640 1 -- 172.21.15.46:0/3733704459 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a949bdecc0 con 0x55a949bc96c0 2024-03-31T23:26:05.572 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:05.567+0000 7f966e59c640 1 --2- 172.21.15.46:0/3733704459 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a949bc96c0 0x55a949bd2550 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7f96600107e0 tx=0x7f9660010dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=449a1d6ab19b4d40 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:26:05.572 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:05.567+0000 7f9657fff640 1 -- 172.21.15.46:0/3733704459 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9660000e60 con 0x55a949bc96c0 2024-03-31T23:26:05.572 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:05.567+0000 7f9657fff640 1 -- 172.21.15.46:0/3733704459 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9660004d00 con 0x55a949bc96c0 2024-03-31T23:26:05.572 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:05.567+0000 7f9657fff640 1 -- 172.21.15.46:0/3733704459 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9660011be0 con 0x55a949bc96c0 2024-03-31T23:26:05.572 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:05.568+0000 7f9677a07740 1 -- 172.21.15.46:0/3733704459 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a949bc96c0 msgr2=0x55a949bd2550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:26:05.572 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:05.568+0000 7f9677a07740 1 --2- 172.21.15.46:0/3733704459 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a949bc96c0 0x55a949bd2550 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7f96600107e0 tx=0x7f9660010dd0 comp rx=0 tx=0).stop 2024-03-31T23:26:05.572 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:05.568+0000 7f9677a07740 1 -- 172.21.15.46:0/3733704459 shutdown_connections 2024-03-31T23:26:05.572 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:05.568+0000 7f9677a07740 1 --2- 172.21.15.46:0/3733704459 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a949bd2db0 0x55a949bc43c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:26:05.572 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:05.568+0000 7f9677a07740 1 --2- 172.21.15.46:0/3733704459 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a949bc9fe0 0x55a949bde780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:26:05.572 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:05.568+0000 7f9677a07740 1 --2- 172.21.15.46:0/3733704459 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a949bc96c0 0x55a949bd2550 unknown :-1 s=CLOSED pgs=886 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:26:05.573 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:05.568+0000 7f9677a07740 1 -- 172.21.15.46:0/3733704459 >> 172.21.15.46:0/3733704459 conn(0x55a949bc3fe0 msgr2=0x55a949bbebf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:26:05.573 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:05.568+0000 7f9677a07740 1 -- 172.21.15.46:0/3733704459 shutdown_connections 2024-03-31T23:26:05.573 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:05.568+0000 7f9677a07740 1 -- 172.21.15.46:0/3733704459 wait complete. 2024-03-31T23:26:05.573 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:05.569+0000 7f9677a07740 1 -- start start 2024-03-31T23:26:05.573 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:05.569+0000 7f9677a07740 1 Processor -- start 2024-03-31T23:26:05.573 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:05.569+0000 7f9677a07740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a949bc96c0 0x55a949bd5d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:26:05.573 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:05.569+0000 7f9677a07740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a949bc9fe0 0x55a949bd62b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:26:05.573 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:05.569+0000 7f9677a07740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a949bd2db0 0x55a949bd6830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:26:05.573 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:05.569+0000 7f9677a07740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f96701af6a0 con 0x55a949bd2db0 2024-03-31T23:26:05.573 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:05.569+0000 7f9677a07740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f96701b0b10 con 0x55a949bc9fe0 2024-03-31T23:26:05.573 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:05.569+0000 7f9677a07740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f96701afd70 con 0x55a949bc96c0 2024-03-31T23:26:05.573 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:05.569+0000 7f966ed9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a949bd2db0 0x55a949bd6830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:26:05.573 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:05.569+0000 7f966e59c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a949bc96c0 0x55a949bd5d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:26:05.574 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:05.569+0000 7f9677a07740 10 client.0 _ll_register_callbacks cb 0x7f966000bae0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T23:26:05.574 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:05.569+0000 7f966dd9b640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a949bc9fe0 0x55a949bd62b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:26:05.574 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:05.569+0000 7f966e59c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a949bc96c0 0x55a949bd5d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.46:56816/0 (socket says 172.21.15.46:56816) 2024-03-31T23:26:05.574 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:05.569+0000 7f966e59c640 1 -- 172.21.15.46:0/4173461132 learned_addr learned my addr 172.21.15.46:0/4173461132 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:26:05.574 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:05.569+0000 7f966ed9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a949bd2db0 0x55a949bd6830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37858/0 (socket says 172.21.15.46:37858) 2024-03-31T23:26:05.574 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:05.569+0000 7f966e59c640 1 -- 172.21.15.46:0/4173461132 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a949bc9fe0 msgr2=0x55a949bd62b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:26:05.574 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:05.570+0000 7f966e59c640 1 --2- 172.21.15.46:0/4173461132 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a949bc9fe0 0x55a949bd62b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:26:05.574 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:05.570+0000 7f966e59c640 1 -- 172.21.15.46:0/4173461132 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a949bd2db0 msgr2=0x55a949bd6830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:26:05.574 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:05.570+0000 7f966e59c640 1 --2- 172.21.15.46:0/4173461132 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a949bd2db0 0x55a949bd6830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:26:05.574 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:05.570+0000 7f966e59c640 1 -- 172.21.15.46:0/4173461132 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a949c47810 con 0x55a949bc96c0 2024-03-31T23:26:05.574 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:05.570+0000 7f966e59c640 1 --2- 172.21.15.46:0/4173461132 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a949bc96c0 0x55a949bd5d70 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f9660004c00 tx=0x7f9660011670 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b305c4a9b3505a51 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:26:05.574 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:05.570+0000 7f9655ffb640 10 client.0 ms_handle_connect on v2:172.21.15.162:3300/0 2024-03-31T23:26:05.576 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:05.570+0000 7f9655ffb640 1 -- 172.21.15.46:0/4173461132 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9660011d70 con 0x55a949bc96c0 2024-03-31T23:26:05.576 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:05.570+0000 7f9655ffb640 1 -- 172.21.15.46:0/4173461132 <== mon.1 v2:172.21.15.162:3300/0 2 ==== osd_map(113..113 src has 1..113) ==== 9642+0+0 (secure 0 0 0) 0x7f966001c810 con 0x55a949bc96c0 2024-03-31T23:26:05.576 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:05.570+0000 7f9677a07740 1 -- 172.21.15.46:0/4173461132 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9660004d00 con 0x55a949bc96c0 2024-03-31T23:26:05.576 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:05.570+0000 7f9655ffb640 1 -- 172.21.15.46:0/4173461132 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7f9660000e40 con 0x55a949bc96c0 2024-03-31T23:26:05.576 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:05.570+0000 7f9677a07740 10 client.27049 fetch_fsmap learned FSMap version 38 2024-03-31T23:26:05.576 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:05.570+0000 7f9677a07740 1 -- 172.21.15.46:0/4173461132 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x55a949c48ac0 con 0x55a949bc96c0 2024-03-31T23:26:05.576 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:05.571+0000 7f9655ffb640 1 -- 172.21.15.46:0/4173461132 <== mon.1 v2:172.21.15.162:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9660004d00 con 0x55a949bc96c0 2024-03-31T23:26:05.576 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:05.571+0000 7f9655ffb640 1 -- 172.21.15.46:0/4173461132 <== mon.1 v2:172.21.15.162:3300/0 5 ==== fsmap(e 38) ==== 8370+0+0 (secure 0 0 0) 0x7f9660027950 con 0x55a949bc96c0 2024-03-31T23:26:05.576 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:05.571+0000 7f9677a07740 10 client.27049 fetch_fsmap finished waiting for FSMap version 38 2024-03-31T23:26:05.577 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:05.571+0000 7f9677a07740 10 client.27049 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-31T23:26:05.577 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:05.571+0000 7f9677a07740 20 client.27049 populate_metadata read hostname 'smithi046' 2024-03-31T23:26:05.577 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:05.571+0000 7f9677a07740 1 --2- 172.21.15.46:0/4173461132 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55a949be3140 0x55a949c4b1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:26:05.577 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:05.571+0000 7f9677a07740 4 client.27049 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-31T23:26:05.577 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:05.571+0000 7f9677a07740 1 -- 172.21.15.46:0/4173461132 --> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55a949b39e30 con 0x55a949be3140 2024-03-31T23:26:05.577 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:05.571+0000 7f966dd9b640 1 --2- 172.21.15.46:0/4173461132 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55a949be3140 0x55a949c4b1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:26:05.577 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:05.572+0000 7f9655ffb640 10 client.27049 ms_handle_connect on v2:172.21.15.142:6838/2519143918 2024-03-31T23:26:05.577 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:05.572+0000 7f966dd9b640 1 --2- 172.21.15.46:0/4173461132 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55a949be3140 0x55a949c4b1b0 crc :-1 s=READY pgs=204 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4457051a48c51436 server_cookie=fb7acfb289cc3530 in_seq=0 out_seq=0 2024-03-31T23:26:05.720 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:05.718+0000 7f9655ffb640 1 -- 172.21.15.46:0/4173461132 <== mds.0 v2:172.21.15.142:6838/2519143918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55a949b39e30 con 0x55a949be3140 2024-03-31T23:26:05.720 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:05.718+0000 7f9655ffb640 10 client.27049 handle_command_reply: tid=0 2024-03-31T23:26:05.720 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:05.718+0000 7f9677a07740 1 client.27049 shutdown 2024-03-31T23:26:05.720 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:05.718+0000 7f9677a07740 1 -- 172.21.15.46:0/4173461132 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a949bc96c0 msgr2=0x55a949bd5d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:26:05.720 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:05.718+0000 7f9677a07740 1 --2- 172.21.15.46:0/4173461132 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a949bc96c0 0x55a949bd5d70 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f9660004c00 tx=0x7f9660011670 comp rx=0 tx=0).stop 2024-03-31T23:26:05.720 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:05.718+0000 7f9677a07740 1 -- 172.21.15.46:0/4173461132 shutdown_connections 2024-03-31T23:26:05.720 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:05.718+0000 7f9677a07740 1 --2- 172.21.15.46:0/4173461132 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55a949be3140 0x55a949c4b1b0 crc :-1 s=READY pgs=204 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:26:05.720 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:05.718+0000 7f9677a07740 1 --2- 172.21.15.46:0/4173461132 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a949bd2db0 0x55a949bd6830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:26:05.720 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:05.718+0000 7f9677a07740 1 --2- 172.21.15.46:0/4173461132 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a949bc9fe0 0x55a949bd62b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:26:05.720 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:05.718+0000 7f9677a07740 1 --2- 172.21.15.46:0/4173461132 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a949bc96c0 0x55a949bd5d70 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:26:05.720 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:05.718+0000 7f9677a07740 1 -- 172.21.15.46:0/4173461132 >> 172.21.15.46:0/4173461132 conn(0x55a949bc3fe0 msgr2=0x55a949bc43c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:26:05.721 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:05.718+0000 7f9655ffb640 0 client.27049 ms_handle_reset on v2:172.21.15.142:6838/2519143918 2024-03-31T23:26:05.721 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:05.718+0000 7f9677a07740 1 -- 172.21.15.46:0/4173461132 shutdown_connections 2024-03-31T23:26:05.721 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:05.718+0000 7f9677a07740 1 -- 172.21.15.46:0/4173461132 wait complete. 2024-03-31T23:26:05.721 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:05.719+0000 7f9677a07740 20 client.27049 trim_cache size 0 max 16384 2024-03-31T23:26:05.721 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout:{ 2024-03-31T23:26:05.722 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout: "message": "", 2024-03-31T23:26:05.722 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout: "return_code": 0 2024-03-31T23:26:05.722 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout:} 2024-03-31T23:26:05.723 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:05.719+0000 7f9674e23640 1 -- 172.21.15.46:0/3958673720 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f96580a26b0 msgr2=0x7f96580a4b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:26:05.723 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:05.719+0000 7f9674e23640 1 --2- 172.21.15.46:0/3958673720 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f96580a26b0 0x7f96580a4b70 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f9664009900 tx=0x7f96640043b0 comp rx=0 tx=0).stop 2024-03-31T23:26:05.723 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:05.720+0000 7f9674e23640 1 -- 172.21.15.46:0/3958673720 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9670107750 msgr2=0x7f967019c820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:26:05.723 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:05.720+0000 7f9674e23640 1 --2- 172.21.15.46:0/3958673720 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9670107750 0x7f967019c820 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f966800eb10 tx=0x7f966800c2d0 comp rx=0 tx=0).stop 2024-03-31T23:26:05.723 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:05.720+0000 7f9674e23640 1 -- 172.21.15.46:0/3958673720 shutdown_connections 2024-03-31T23:26:05.723 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:05.720+0000 7f9674e23640 1 --2- 172.21.15.46:0/3958673720 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f96580a26b0 0x7f96580a4b70 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:26:05.723 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:05.720+0000 7f9674e23640 1 --2- 172.21.15.46:0/3958673720 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9670107750 0x7f967019c820 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:26:05.723 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:05.720+0000 7f9674e23640 1 --2- 172.21.15.46:0/3958673720 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9670102100 0x7f96701984a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:26:05.723 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:05.720+0000 7f9674e23640 1 --2- 172.21.15.46:0/3958673720 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9670101740 0x7f9670197f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:26:05.723 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:05.720+0000 7f9674e23640 1 -- 172.21.15.46:0/3958673720 >> 172.21.15.46:0/3958673720 conn(0x7f96700fd210 msgr2=0x7f96700fd8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:26:05.723 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:05.720+0000 7f9674e23640 1 -- 172.21.15.46:0/3958673720 shutdown_connections 2024-03-31T23:26:05.723 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:05.721+0000 7f9674e23640 1 -- 172.21.15.46:0/3958673720 wait complete. 2024-03-31T23:26:06.267 INFO:tasks.workunit.client.0.smithi046.stdout: 10 11409 11.07 MB/sec warmup 71 sec latency 134.925 ms 2024-03-31T23:26:06.296 INFO:journalctl@ceph.mds.i.smithi162.stdout:Mar 31 23:26:05 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mds-i[67378]: ignoring --setuser ceph since I am not root 2024-03-31T23:26:06.296 INFO:journalctl@ceph.mds.i.smithi162.stdout:Mar 31 23:26:05 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mds-i[67378]: ignoring --setgroup ceph since I am not root 2024-03-31T23:26:06.296 INFO:journalctl@ceph.mds.i.smithi162.stdout:Mar 31 23:26:05 smithi162 ceph-mds[67382]: -- 172.21.15.162:0/270303532 <== mon.2 v2:172.21.15.142:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5572eb7dda40 con 0x5572eb7b0800 2024-03-31T23:26:06.296 INFO:journalctl@ceph.mds.i.smithi162.stdout:Mar 31 23:26:05 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mds-i[67378]: starting mds.i at 2024-03-31T23:26:06.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:26:06 smithi142 ceph-mon[35271]: pgmap v451: 129 pgs: 129 active+clean; 435 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 5.2 MiB/s wr, 275 op/s 2024-03-31T23:26:06.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:26:06 smithi142 ceph-mon[35271]: daemon mds.i restarted 2024-03-31T23:26:07.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:26:06 smithi162 ceph-mon[36237]: pgmap v451: 129 pgs: 129 active+clean; 435 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 5.2 MiB/s wr, 275 op/s 2024-03-31T23:26:07.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:26:06 smithi162 ceph-mon[36237]: daemon mds.i restarted 2024-03-31T23:26:07.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:26:06 smithi046 ceph-mon[32498]: pgmap v451: 129 pgs: 129 active+clean; 435 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 5.2 MiB/s wr, 275 op/s 2024-03-31T23:26:07.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:26:06 smithi046 ceph-mon[32498]: daemon mds.i restarted 2024-03-31T23:26:07.267 INFO:tasks.workunit.client.0.smithi046.stdout: 10 11560 11.03 MB/sec warmup 72 sec latency 61.727 ms 2024-03-31T23:26:07.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:26:07 smithi142 ceph-mon[35271]: osdmap e114: 12 total, 12 up, 12 in 2024-03-31T23:26:07.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:26:07 smithi142 ceph-mon[35271]: mds.? [v2:172.21.15.162:6838/1308437877,v1:172.21.15.162:6839/1308437877] up:boot 2024-03-31T23:26:07.944 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:26:07 smithi142 ceph-mon[35271]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=c=up:active} 2 up:standby-replay 7 up:standby 2024-03-31T23:26:07.944 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:26:07 smithi142 ceph-mon[35271]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-31T23:26:07.944 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:26:07 smithi142 ceph-mon[35271]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-03-31T23:26:08.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:26:07 smithi162 ceph-mon[36237]: osdmap e114: 12 total, 12 up, 12 in 2024-03-31T23:26:08.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:26:07 smithi162 ceph-mon[36237]: mds.? [v2:172.21.15.162:6838/1308437877,v1:172.21.15.162:6839/1308437877] up:boot 2024-03-31T23:26:08.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:26:07 smithi162 ceph-mon[36237]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=c=up:active} 2 up:standby-replay 7 up:standby 2024-03-31T23:26:08.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:26:07 smithi162 ceph-mon[36237]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-31T23:26:08.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:26:07 smithi162 ceph-mon[36237]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-03-31T23:26:08.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:26:07 smithi046 ceph-mon[32498]: osdmap e114: 12 total, 12 up, 12 in 2024-03-31T23:26:08.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:26:07 smithi046 ceph-mon[32498]: mds.? [v2:172.21.15.162:6838/1308437877,v1:172.21.15.162:6839/1308437877] up:boot 2024-03-31T23:26:08.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:26:07 smithi046 ceph-mon[32498]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=c=up:active} 2 up:standby-replay 7 up:standby 2024-03-31T23:26:08.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:26:07 smithi046 ceph-mon[32498]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-31T23:26:08.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:26:07 smithi046 ceph-mon[32498]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-03-31T23:26:08.267 INFO:tasks.workunit.client.0.smithi046.stdout: 10 11714 10.95 MB/sec warmup 73 sec latency 88.090 ms 2024-03-31T23:26:08.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:26:08 smithi142 ceph-mon[35271]: pgmap v453: 129 pgs: 129 active+clean; 435 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 3.2 MiB/s wr, 188 op/s 2024-03-31T23:26:09.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:26:08 smithi162 ceph-mon[36237]: pgmap v453: 129 pgs: 129 active+clean; 435 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 3.2 MiB/s wr, 188 op/s 2024-03-31T23:26:09.113 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:26:08 smithi046 ceph-mon[32498]: pgmap v453: 129 pgs: 129 active+clean; 435 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 3.2 MiB/s wr, 188 op/s 2024-03-31T23:26:09.268 INFO:tasks.workunit.client.0.smithi046.stdout: 10 11896 10.92 MB/sec warmup 74 sec latency 64.505 ms 2024-03-31T23:26:10.267 INFO:tasks.workunit.client.0.smithi046.stdout: 10 12066 10.85 MB/sec warmup 75 sec latency 121.720 ms 2024-03-31T23:26:10.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:26:10 smithi142 ceph-mon[35271]: pgmap v454: 129 pgs: 129 active+clean; 444 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 5.9 MiB/s wr, 433 op/s 2024-03-31T23:26:10.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:26:10 smithi142 ceph-mon[35271]: osdmap e115: 12 total, 12 up, 12 in 2024-03-31T23:26:11.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:26:10 smithi162 ceph-mon[36237]: pgmap v454: 129 pgs: 129 active+clean; 444 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 5.9 MiB/s wr, 433 op/s 2024-03-31T23:26:11.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:26:10 smithi162 ceph-mon[36237]: osdmap e115: 12 total, 12 up, 12 in 2024-03-31T23:26:11.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:26:10 smithi046 ceph-mon[32498]: pgmap v454: 129 pgs: 129 active+clean; 444 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 5.9 MiB/s wr, 433 op/s 2024-03-31T23:26:11.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:26:10 smithi046 ceph-mon[32498]: osdmap e115: 12 total, 12 up, 12 in 2024-03-31T23:26:11.267 INFO:tasks.workunit.client.0.smithi046.stdout: 10 12252 10.88 MB/sec warmup 76 sec latency 77.804 ms 2024-03-31T23:26:12.268 INFO:tasks.workunit.client.0.smithi046.stdout: 10 12434 10.92 MB/sec warmup 77 sec latency 135.124 ms 2024-03-31T23:26:12.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:26:12 smithi142 ceph-mon[35271]: pgmap v456: 129 pgs: 129 active+clean; 444 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 5.9 MiB/s wr, 433 op/s 2024-03-31T23:26:13.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:26:12 smithi162 ceph-mon[36237]: pgmap v456: 129 pgs: 129 active+clean; 444 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 5.9 MiB/s wr, 433 op/s 2024-03-31T23:26:13.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:26:12 smithi046 ceph-mon[32498]: pgmap v456: 129 pgs: 129 active+clean; 444 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 5.9 MiB/s wr, 433 op/s 2024-03-31T23:26:13.268 INFO:tasks.workunit.client.0.smithi046.stdout: 10 12586 10.85 MB/sec warmup 78 sec latency 141.223 ms 2024-03-31T23:26:14.268 INFO:tasks.workunit.client.0.smithi046.stdout: 10 12778 10.93 MB/sec warmup 79 sec latency 65.580 ms 2024-03-31T23:26:14.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:26:14 smithi142 ceph-mon[35271]: pgmap v457: 129 pgs: 129 active+clean; 444 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 882 KiB/s rd, 2.6 MiB/s wr, 244 op/s 2024-03-31T23:26:15.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:26:14 smithi162 ceph-mon[36237]: pgmap v457: 129 pgs: 129 active+clean; 444 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 882 KiB/s rd, 2.6 MiB/s wr, 244 op/s 2024-03-31T23:26:15.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:26:14 smithi046 ceph-mon[32498]: pgmap v457: 129 pgs: 129 active+clean; 444 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 882 KiB/s rd, 2.6 MiB/s wr, 244 op/s 2024-03-31T23:26:15.268 INFO:tasks.workunit.client.0.smithi046.stdout: 10 12954 10.90 MB/sec warmup 80 sec latency 69.611 ms 2024-03-31T23:26:16.268 INFO:tasks.workunit.client.0.smithi046.stdout: 10 13126 10.93 MB/sec warmup 81 sec latency 59.243 ms 2024-03-31T23:26:16.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:26:16 smithi142 ceph-mon[35271]: pgmap v458: 129 pgs: 129 active+clean; 460 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 5.8 MiB/s wr, 389 op/s 2024-03-31T23:26:17.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:26:16 smithi162 ceph-mon[36237]: pgmap v458: 129 pgs: 129 active+clean; 460 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 5.8 MiB/s wr, 389 op/s 2024-03-31T23:26:17.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:26:16 smithi046 ceph-mon[32498]: pgmap v458: 129 pgs: 129 active+clean; 460 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 5.8 MiB/s wr, 389 op/s 2024-03-31T23:26:17.268 INFO:tasks.workunit.client.0.smithi046.stdout: 10 13263 10.87 MB/sec warmup 82 sec latency 70.067 ms 2024-03-31T23:26:18.268 INFO:tasks.workunit.client.0.smithi046.stdout: 10 13431 10.86 MB/sec warmup 83 sec latency 71.655 ms 2024-03-31T23:26:18.833 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:18.831+0000 7ff81c2af640 1 -- 172.21.15.46:0/2325883830 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff814107750 msgr2=0x7ff814107b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:26:18.833 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:18.831+0000 7ff81c2af640 1 --2- 172.21.15.46:0/2325883830 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff814107750 0x7ff814107b30 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7ff810009930 tx=0x7ff81002bc30 comp rx=0 tx=0).stop 2024-03-31T23:26:18.833 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:18.831+0000 7ff81c2af640 1 -- 172.21.15.46:0/2325883830 shutdown_connections 2024-03-31T23:26:18.833 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:18.831+0000 7ff81c2af640 1 --2- 172.21.15.46:0/2325883830 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff814102100 0x7ff81410e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:26:18.833 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:18.831+0000 7ff81c2af640 1 --2- 172.21.15.46:0/2325883830 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff814101740 0x7ff814101bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:26:18.833 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:18.831+0000 7ff81c2af640 1 --2- 172.21.15.46:0/2325883830 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff814107750 0x7ff814107b30 unknown :-1 s=CLOSED pgs=887 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:26:18.833 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:18.831+0000 7ff81c2af640 1 -- 172.21.15.46:0/2325883830 >> 172.21.15.46:0/2325883830 conn(0x7ff8140fd210 msgr2=0x7ff8140ff630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:26:18.833 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:18.831+0000 7ff81c2af640 1 -- 172.21.15.46:0/2325883830 shutdown_connections 2024-03-31T23:26:18.833 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:18.832+0000 7ff81c2af640 1 -- 172.21.15.46:0/2325883830 wait complete. 2024-03-31T23:26:18.835 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:18.832+0000 7ff81c2af640 1 Processor -- start 2024-03-31T23:26:18.835 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:18.832+0000 7ff81c2af640 1 -- start start 2024-03-31T23:26:18.835 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:18.832+0000 7ff81c2af640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff814101740 0x7ff81419a0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:26:18.836 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:18.832+0000 7ff81c2af640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff814102100 0x7ff81419a5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:26:18.836 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:18.832+0000 7ff81c2af640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff814107750 0x7ff81419e960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:26:18.836 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:18.833+0000 7ff81c2af640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7ff814111a10 con 0x7ff814107750 2024-03-31T23:26:18.836 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:18.833+0000 7ff81c2af640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7ff814111890 con 0x7ff814102100 2024-03-31T23:26:18.836 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:18.833+0000 7ff81c2af640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7ff814111b90 con 0x7ff814101740 2024-03-31T23:26:18.836 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:18.833+0000 7ff81a84c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff814107750 0x7ff81419e960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:26:18.836 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:18.833+0000 7ff81a04b640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff814101740 0x7ff81419a0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:26:18.836 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:18.833+0000 7ff81a04b640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff814101740 0x7ff81419a0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.46:44802/0 (socket says 172.21.15.46:44802) 2024-03-31T23:26:18.836 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:18.833+0000 7ff81a84c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff814107750 0x7ff81419e960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59902/0 (socket says 172.21.15.46:59902) 2024-03-31T23:26:18.836 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:18.833+0000 7ff81a04b640 1 -- 172.21.15.46:0/2575508639 learned_addr learned my addr 172.21.15.46:0/2575508639 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:26:18.836 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:18.833+0000 7ff81984a640 1 --2- 172.21.15.46:0/2575508639 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff814102100 0x7ff81419a5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:26:18.836 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:18.833+0000 7ff81a04b640 1 -- 172.21.15.46:0/2575508639 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff814102100 msgr2=0x7ff81419a5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:26:18.836 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:18.833+0000 7ff81a04b640 1 --2- 172.21.15.46:0/2575508639 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff814102100 0x7ff81419a5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:26:18.837 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:18.833+0000 7ff81a04b640 1 -- 172.21.15.46:0/2575508639 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff814107750 msgr2=0x7ff81419e960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:26:18.837 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:18.833+0000 7ff81a04b640 1 --2- 172.21.15.46:0/2575508639 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff814107750 0x7ff81419e960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:26:18.837 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:18.833+0000 7ff81a04b640 1 -- 172.21.15.46:0/2575508639 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff81419f090 con 0x7ff814101740 2024-03-31T23:26:18.837 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:18.833+0000 7ff81a84c640 1 --2- 172.21.15.46:0/2575508639 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff814107750 0x7ff81419e960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:26:18.837 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:18.833+0000 7ff81a04b640 1 --2- 172.21.15.46:0/2575508639 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff814101740 0x7ff81419a0a0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7ff810009230 tx=0x7ff81002c5d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:26:18.837 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:18.833+0000 7ff80b7fe640 1 -- 172.21.15.46:0/2575508639 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff810043020 con 0x7ff814101740 2024-03-31T23:26:18.837 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:18.834+0000 7ff81c2af640 1 -- 172.21.15.46:0/2575508639 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff81419f320 con 0x7ff814101740 2024-03-31T23:26:18.837 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:18.834+0000 7ff80b7fe640 1 -- 172.21.15.46:0/2575508639 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff81002ca70 con 0x7ff814101740 2024-03-31T23:26:18.837 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:18.834+0000 7ff80b7fe640 1 -- 172.21.15.46:0/2575508639 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff810003a50 con 0x7ff814101740 2024-03-31T23:26:18.837 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:18.834+0000 7ff81c2af640 1 -- 172.21.15.46:0/2575508639 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff8141a6c10 con 0x7ff814101740 2024-03-31T23:26:18.838 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:18.836+0000 7ff80b7fe640 1 -- 172.21.15.46:0/2575508639 <== mon.1 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 25) ==== 137451+0+0 (secure 0 0 0) 0x7ff810046020 con 0x7ff814101740 2024-03-31T23:26:18.838 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:18.837+0000 7ff80b7fe640 1 --2- 172.21.15.46:0/2575508639 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7ff8000a2730 0x7ff8000a4bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:26:18.838 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:18.837+0000 7ff81ef6a740 1 Processor -- start 2024-03-31T23:26:18.838 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:18.837+0000 7ff81984a640 1 --2- 172.21.15.46:0/2575508639 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7ff8000a2730 0x7ff8000a4bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:26:18.839 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:18.837+0000 7ff80b7fe640 1 -- 172.21.15.46:0/2575508639 <== mon.1 v2:172.21.15.162:3300/0 5 ==== osd_map(115..115 src has 1..115) ==== 9554+0+0 (secure 0 0 0) 0x7ff8100f0d60 con 0x7ff814101740 2024-03-31T23:26:18.839 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:18.837+0000 7ff81ef6a740 1 -- start start 2024-03-31T23:26:18.840 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:18.838+0000 7ff81ef6a740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55613cb356c0 0x55613cb35aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:26:18.840 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:18.838+0000 7ff81ef6a740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55613cb35fe0 0x55613cb3e870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:26:18.840 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:18.838+0000 7ff81ef6a740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55613cb3edb0 0x55613cb411a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:26:18.840 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:18.838+0000 7ff81ef6a740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55613ca988f0 con 0x55613cb3edb0 2024-03-31T23:26:18.840 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:18.838+0000 7ff81ef6a740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55613ca418d0 con 0x55613cb356c0 2024-03-31T23:26:18.841 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:18.838+0000 7ff81ef6a740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55613ca94990 con 0x55613cb35fe0 2024-03-31T23:26:18.841 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:18.838+0000 7ff81a84c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55613cb3edb0 0x55613cb411a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:26:18.841 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:18.838+0000 7ff81a04b640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55613cb356c0 0x55613cb35aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:26:18.841 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:18.838+0000 7ff81a04b640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55613cb356c0 0x55613cb35aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.46:46882/0 (socket says 172.21.15.46:46882) 2024-03-31T23:26:18.841 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:18.838+0000 7ff81a04b640 1 -- 172.21.15.46:0/2963356616 learned_addr learned my addr 172.21.15.46:0/2963356616 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:26:18.841 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:18.838+0000 7ff81984a640 1 --2- 172.21.15.46:0/2575508639 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7ff8000a2730 0x7ff8000a4bf0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7ff81410e940 tx=0x7ff804005f90 comp rx=0 tx=0).ready entity=mgr.15654 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:26:18.841 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:18.838+0000 7ff81984a640 1 --2- 172.21.15.46:0/2963356616 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55613cb35fe0 0x55613cb3e870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:26:18.841 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:18.838+0000 7ff81a04b640 1 -- 172.21.15.46:0/2963356616 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55613cb35fe0 msgr2=0x55613cb3e870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:26:18.841 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:18.838+0000 7ff81a04b640 1 --2- 172.21.15.46:0/2963356616 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55613cb35fe0 0x55613cb3e870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:26:18.841 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:18.838+0000 7ff81a04b640 1 -- 172.21.15.46:0/2963356616 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55613cb3edb0 msgr2=0x55613cb411a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:26:18.841 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:18.838+0000 7ff81a04b640 1 --2- 172.21.15.46:0/2963356616 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55613cb3edb0 0x55613cb411a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:26:18.841 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:18.838+0000 7ff81a04b640 1 -- 172.21.15.46:0/2963356616 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55613c7ad210 con 0x55613cb356c0 2024-03-31T23:26:18.841 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:18.838+0000 7ff81a84c640 1 --2- 172.21.15.46:0/2963356616 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55613cb3edb0 0x55613cb411a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:26:18.842 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:18.839+0000 7ff81a04b640 1 --2- 172.21.15.46:0/2963356616 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55613cb356c0 0x55613cb35aa0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7ff8100b9640 tx=0x7ff81002bc30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ac9bc74d9c070325 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:26:18.842 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:18.839+0000 7ff808bfe640 1 -- 172.21.15.46:0/2963356616 <== mon.2 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8100bd1e0 con 0x55613cb356c0 2024-03-31T23:26:18.842 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:18.839+0000 7ff808bfe640 1 -- 172.21.15.46:0/2963356616 <== mon.2 v2:172.21.15.142:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff8100c1330 con 0x55613cb356c0 2024-03-31T23:26:18.842 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:18.839+0000 7ff808bfe640 1 -- 172.21.15.46:0/2963356616 <== mon.2 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8100053d0 con 0x55613cb356c0 2024-03-31T23:26:18.845 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:18.840+0000 7ff81ef6a740 1 -- 172.21.15.46:0/2963356616 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55613cb356c0 msgr2=0x55613cb35aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:26:18.845 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:18.840+0000 7ff81ef6a740 1 --2- 172.21.15.46:0/2963356616 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55613cb356c0 0x55613cb35aa0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7ff8100b9640 tx=0x7ff81002bc30 comp rx=0 tx=0).stop 2024-03-31T23:26:18.845 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:18.840+0000 7ff81ef6a740 1 -- 172.21.15.46:0/2963356616 shutdown_connections 2024-03-31T23:26:18.845 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:18.840+0000 7ff81ef6a740 1 --2- 172.21.15.46:0/2963356616 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55613cb3edb0 0x55613cb411a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:26:18.845 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:18.840+0000 7ff81ef6a740 1 --2- 172.21.15.46:0/2963356616 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55613cb35fe0 0x55613cb3e870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:26:18.845 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:18.840+0000 7ff81ef6a740 1 --2- 172.21.15.46:0/2963356616 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55613cb356c0 0x55613cb35aa0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:26:18.846 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:18.840+0000 7ff81ef6a740 1 -- 172.21.15.46:0/2963356616 >> 172.21.15.46:0/2963356616 conn(0x55613cb2ffe0 msgr2=0x55613cb303c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:26:18.846 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:18.840+0000 7ff81ef6a740 1 -- 172.21.15.46:0/2963356616 shutdown_connections 2024-03-31T23:26:18.846 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:18.840+0000 7ff81ef6a740 1 -- 172.21.15.46:0/2963356616 wait complete. 2024-03-31T23:26:18.846 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:18.840+0000 7ff81ef6a740 1 -- start start 2024-03-31T23:26:18.846 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:18.840+0000 7ff81ef6a740 1 Processor -- start 2024-03-31T23:26:18.846 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:18.841+0000 7ff81ef6a740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55613cb356c0 0x55613cb3be70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:26:18.846 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:18.841+0000 7ff81ef6a740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55613cb35fe0 0x55613cb3c3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:26:18.846 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:18.841+0000 7ff81ef6a740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55613cb3edb0 0x55613cb3c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:26:18.846 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:18.841+0000 7ff81ef6a740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55613cb417a0 con 0x55613cb3edb0 2024-03-31T23:26:18.846 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:18.841+0000 7ff81ef6a740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55613ca36f80 con 0x55613cb356c0 2024-03-31T23:26:18.846 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:18.841+0000 7ff81ef6a740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55613cb41920 con 0x55613cb35fe0 2024-03-31T23:26:18.846 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:18.841+0000 7ff81a04b640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55613cb356c0 0x55613cb3be70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:26:18.846 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:18.841+0000 7ff81984a640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55613cb35fe0 0x55613cb3c3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:26:18.847 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:18.841+0000 7ff81a84c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55613cb3edb0 0x55613cb3c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:26:18.847 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:18.841+0000 7ff81984a640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55613cb35fe0 0x55613cb3c3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.46:44814/0 (socket says 172.21.15.46:44814) 2024-03-31T23:26:18.847 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:18.841+0000 7ff81a04b640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55613cb356c0 0x55613cb3be70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.46:46894/0 (socket says 172.21.15.46:46894) 2024-03-31T23:26:18.847 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:18.841+0000 7ff81984a640 1 -- 172.21.15.46:0/2887608283 learned_addr learned my addr 172.21.15.46:0/2887608283 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:26:18.847 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:18.841+0000 7ff81ef6a740 10 client.0 _ll_register_callbacks cb 0x55613ca8f290 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T23:26:18.847 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:18.841+0000 7ff81a84c640 1 -- 172.21.15.46:0/2887608283 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55613cb35fe0 msgr2=0x55613cb3c3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:26:18.847 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:18.841+0000 7ff81a84c640 1 --2- 172.21.15.46:0/2887608283 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55613cb35fe0 0x55613cb3c3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:26:18.847 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:18.841+0000 7ff81a84c640 1 -- 172.21.15.46:0/2887608283 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55613cb356c0 msgr2=0x55613cb3be70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:26:18.847 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:18.841+0000 7ff81a84c640 1 --2- 172.21.15.46:0/2887608283 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55613cb356c0 0x55613cb3be70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:26:18.847 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:18.841+0000 7ff81a84c640 1 -- 172.21.15.46:0/2887608283 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55613c879620 con 0x55613cb3edb0 2024-03-31T23:26:18.847 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:18.841+0000 7ff81984a640 1 --2- 172.21.15.46:0/2887608283 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55613cb35fe0 0x55613cb3c3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:26:18.847 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:18.841+0000 7ff81a04b640 1 --2- 172.21.15.46:0/2887608283 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55613cb356c0 0x55613cb3be70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:26:18.847 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:18.842+0000 7ff81a84c640 1 --2- 172.21.15.46:0/2887608283 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55613cb3edb0 0x55613cb3c8f0 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x55613cb31170 tx=0x7ff80c00e780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f66f5ec439b7dba server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:26:18.847 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:18.842+0000 7ff7fdffb640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-31T23:26:18.848 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:18.842+0000 7ff7fdffb640 1 -- 172.21.15.46:0/2887608283 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff80c003970 con 0x55613cb3edb0 2024-03-31T23:26:18.848 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:18.842+0000 7ff7fdffb640 1 -- 172.21.15.46:0/2887608283 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(115..115 src has 1..115) ==== 9554+0+0 (secure 0 0 0) 0x7ff80c003b10 con 0x55613cb3edb0 2024-03-31T23:26:18.848 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:18.842+0000 7ff81ef6a740 1 -- 172.21.15.46:0/2887608283 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff8100c1330 con 0x55613cb3edb0 2024-03-31T23:26:18.848 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:18.842+0000 7ff7fdffb640 1 -- 172.21.15.46:0/2887608283 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7ff80c003d10 con 0x55613cb3edb0 2024-03-31T23:26:18.848 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:18.842+0000 7ff81ef6a740 10 client.17337 fetch_fsmap learned FSMap version 40 2024-03-31T23:26:18.848 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:18.842+0000 7ff81ef6a740 1 -- 172.21.15.46:0/2887608283 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x55613cb4d690 con 0x55613cb3edb0 2024-03-31T23:26:18.848 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:18.843+0000 7ff7fdffb640 1 -- 172.21.15.46:0/2887608283 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff8100c1330 con 0x55613cb3edb0 2024-03-31T23:26:18.848 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:18.843+0000 7ff7fdffb640 1 -- 172.21.15.46:0/2887608283 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 40) ==== 8370+0+0 (secure 0 0 0) 0x7ff80c01a550 con 0x55613cb3edb0 2024-03-31T23:26:18.848 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:18.843+0000 7ff81ef6a740 10 client.17337 fetch_fsmap finished waiting for FSMap version 40 2024-03-31T23:26:18.848 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:18.843+0000 7ff81ef6a740 10 client.17337 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-31T23:26:18.848 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:18.843+0000 7ff81ef6a740 20 client.17337 populate_metadata read hostname 'smithi046' 2024-03-31T23:26:18.848 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:18.843+0000 7ff81ef6a740 1 --2- 172.21.15.46:0/2887608283 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55613cb4eba0 0x55613cb51000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:26:18.848 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:18.843+0000 7ff81ef6a740 4 client.17337 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-31T23:26:18.849 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:18.843+0000 7ff81ef6a740 1 -- 172.21.15.46:0/2887608283 --> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55613caac120 con 0x55613cb4eba0 2024-03-31T23:26:18.849 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:18.843+0000 7ff81a04b640 1 --2- 172.21.15.46:0/2887608283 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55613cb4eba0 0x55613cb51000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:26:18.849 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:18.843+0000 7ff7fdffb640 10 client.17337 ms_handle_connect on v2:172.21.15.142:6838/2519143918 2024-03-31T23:26:18.849 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:18.843+0000 7ff81a04b640 1 --2- 172.21.15.46:0/2887608283 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55613cb4eba0 0x55613cb51000 crc :-1 s=READY pgs=205 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fd237092cedd9949 server_cookie=6936abec63c96c08 in_seq=0 out_seq=0 2024-03-31T23:26:18.850 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:18.845+0000 7ff7fdffb640 1 -- 172.21.15.46:0/2887608283 <== mds.0 v2:172.21.15.142:6838/2519143918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55613caac120 con 0x55613cb4eba0 2024-03-31T23:26:18.850 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:18.845+0000 7ff7fdffb640 10 client.17337 handle_command_reply: tid=0 2024-03-31T23:26:18.850 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:18.845+0000 7ff81ef6a740 1 client.17337 shutdown 2024-03-31T23:26:18.850 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:18.845+0000 7ff81ef6a740 1 -- 172.21.15.46:0/2887608283 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55613cb3edb0 msgr2=0x55613cb3c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:26:18.850 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:18.845+0000 7ff81ef6a740 1 --2- 172.21.15.46:0/2887608283 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55613cb3edb0 0x55613cb3c8f0 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x55613cb31170 tx=0x7ff80c00e780 comp rx=0 tx=0).stop 2024-03-31T23:26:18.850 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:18.845+0000 7ff81ef6a740 1 -- 172.21.15.46:0/2887608283 shutdown_connections 2024-03-31T23:26:18.850 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:18.845+0000 7ff81ef6a740 1 --2- 172.21.15.46:0/2887608283 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55613cb4eba0 0x55613cb51000 crc :-1 s=READY pgs=205 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:26:18.850 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:18.845+0000 7ff81ef6a740 1 --2- 172.21.15.46:0/2887608283 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55613cb3edb0 0x55613cb3c8f0 unknown :-1 s=CLOSED pgs=888 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:26:18.850 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:18.845+0000 7ff81ef6a740 1 --2- 172.21.15.46:0/2887608283 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55613cb35fe0 0x55613cb3c3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:26:18.850 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:18.845+0000 7ff81ef6a740 1 --2- 172.21.15.46:0/2887608283 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55613cb356c0 0x55613cb3be70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:26:18.851 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:18.845+0000 7ff81ef6a740 1 -- 172.21.15.46:0/2887608283 >> 172.21.15.46:0/2887608283 conn(0x55613cb2ffe0 msgr2=0x55613cb31d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:26:18.851 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:18.845+0000 7ff7fdffb640 0 client.17337 ms_handle_reset on v2:172.21.15.142:6838/2519143918 2024-03-31T23:26:18.851 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:18.845+0000 7ff81ef6a740 1 -- 172.21.15.46:0/2887608283 shutdown_connections 2024-03-31T23:26:18.851 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:18.845+0000 7ff81ef6a740 1 -- 172.21.15.46:0/2887608283 wait complete. 2024-03-31T23:26:18.851 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:18.846+0000 7ff81ef6a740 20 client.17337 trim_cache size 0 max 16384 2024-03-31T23:26:18.882 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:18.873+0000 7ff81ef6a740 1 Processor -- start 2024-03-31T23:26:18.882 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:18.873+0000 7ff81ef6a740 1 -- start start 2024-03-31T23:26:18.882 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:18.873+0000 7ff81ef6a740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55613cb356c0 0x55613cb3e5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:26:18.882 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:18.873+0000 7ff81ef6a740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55613cb35fe0 0x55613cb2ae20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:26:18.882 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:18.874+0000 7ff81ef6a740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55613cb3edb0 0x55613cb4a420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:26:18.882 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:18.874+0000 7ff81ef6a740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7ff8141af460 con 0x55613cb3edb0 2024-03-31T23:26:18.882 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:18.874+0000 7ff81ef6a740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7ff8141acc10 con 0x55613cb356c0 2024-03-31T23:26:18.882 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:18.874+0000 7ff81a84c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55613cb3edb0 0x55613cb4a420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:26:18.883 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:18.874+0000 7ff81a84c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55613cb3edb0 0x55613cb4a420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59942/0 (socket says 172.21.15.46:59942) 2024-03-31T23:26:18.883 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:18.874+0000 7ff81a84c640 1 -- 172.21.15.46:0/1680289394 learned_addr learned my addr 172.21.15.46:0/1680289394 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:26:18.883 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:18.874+0000 7ff81ef6a740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7ff8141aaaa0 con 0x55613cb35fe0 2024-03-31T23:26:18.883 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:18.874+0000 7ff81a04b640 1 --2- 172.21.15.46:0/1680289394 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55613cb356c0 0x55613cb3e5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:26:18.883 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:18.874+0000 7ff81a84c640 1 -- 172.21.15.46:0/1680289394 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55613cb35fe0 msgr2=0x55613cb2ae20 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-31T23:26:18.883 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:18.874+0000 7ff81a84c640 1 --2- 172.21.15.46:0/1680289394 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55613cb35fe0 0x55613cb2ae20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:26:18.883 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:18.874+0000 7ff81a84c640 1 -- 172.21.15.46:0/1680289394 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55613cb356c0 msgr2=0x55613cb3e5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:26:18.883 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:18.874+0000 7ff81a84c640 1 --2- 172.21.15.46:0/1680289394 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55613cb356c0 0x55613cb3e5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:26:18.883 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:18.874+0000 7ff81a84c640 1 -- 172.21.15.46:0/1680289394 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55613cb379c0 con 0x55613cb3edb0 2024-03-31T23:26:18.883 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:18.874+0000 7ff81a04b640 1 --2- 172.21.15.46:0/1680289394 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55613cb356c0 0x55613cb3e5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:26:18.883 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:18.874+0000 7ff81a84c640 1 --2- 172.21.15.46:0/1680289394 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55613cb3edb0 0x55613cb4a420 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7ff80c000f80 tx=0x7ff80c009570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac4a276ebf9faf5b server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:26:18.883 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:18.875+0000 7ff7fffff640 1 -- 172.21.15.46:0/1680289394 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff80c003eb0 con 0x55613cb3edb0 2024-03-31T23:26:18.883 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:18.875+0000 7ff7fffff640 1 -- 172.21.15.46:0/1680289394 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff80c037070 con 0x55613cb3edb0 2024-03-31T23:26:18.883 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:18.875+0000 7ff7fffff640 1 -- 172.21.15.46:0/1680289394 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff80c01ae40 con 0x55613cb3edb0 2024-03-31T23:26:18.884 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:18.876+0000 7ff81ef6a740 1 -- 172.21.15.46:0/1680289394 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55613cb3edb0 msgr2=0x55613cb4a420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:26:18.884 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:18.876+0000 7ff81ef6a740 1 --2- 172.21.15.46:0/1680289394 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55613cb3edb0 0x55613cb4a420 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7ff80c000f80 tx=0x7ff80c009570 comp rx=0 tx=0).stop 2024-03-31T23:26:18.884 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:18.876+0000 7ff81ef6a740 1 -- 172.21.15.46:0/1680289394 shutdown_connections 2024-03-31T23:26:18.884 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:18.876+0000 7ff81ef6a740 1 --2- 172.21.15.46:0/1680289394 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55613cb3edb0 0x55613cb4a420 unknown :-1 s=CLOSED pgs=889 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:26:18.884 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:18.876+0000 7ff81ef6a740 1 --2- 172.21.15.46:0/1680289394 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55613cb35fe0 0x55613cb2ae20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:26:18.884 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:18.876+0000 7ff81ef6a740 1 --2- 172.21.15.46:0/1680289394 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55613cb356c0 0x55613cb3e5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:26:18.884 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:18.876+0000 7ff81ef6a740 1 -- 172.21.15.46:0/1680289394 >> 172.21.15.46:0/1680289394 conn(0x55613cb2ffe0 msgr2=0x55613cb3a760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:26:18.884 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:18.876+0000 7ff81ef6a740 1 -- 172.21.15.46:0/1680289394 shutdown_connections 2024-03-31T23:26:18.884 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:18.876+0000 7ff81ef6a740 1 -- 172.21.15.46:0/1680289394 wait complete. 2024-03-31T23:26:18.884 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:18.876+0000 7ff81ef6a740 1 -- start start 2024-03-31T23:26:18.884 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:18.876+0000 7ff81ef6a740 1 Processor -- start 2024-03-31T23:26:18.884 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:18.877+0000 7ff81ef6a740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55613cb356c0 0x55613cb46a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:26:18.884 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:18.877+0000 7ff81ef6a740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55613cb35fe0 0x55613cb38510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:26:18.884 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:18.877+0000 7ff81ef6a740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55613cb3edb0 0x55613cb38a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:26:18.885 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:18.877+0000 7ff81ef6a740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7ff8141ab0f0 con 0x55613cb356c0 2024-03-31T23:26:18.885 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:18.877+0000 7ff81ef6a740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7ff8141ac540 con 0x55613cb3edb0 2024-03-31T23:26:18.885 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:18.877+0000 7ff81ef6a740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7ff8141ab7a0 con 0x55613cb35fe0 2024-03-31T23:26:18.885 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:18.877+0000 7ff81ef6a740 10 client.0 _ll_register_callbacks cb 0x7ff80c00f4a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T23:26:18.885 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:18.877+0000 7ff81a04b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55613cb356c0 0x55613cb46a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:26:18.885 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:18.877+0000 7ff81a04b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55613cb356c0 0x55613cb46a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59948/0 (socket says 172.21.15.46:59948) 2024-03-31T23:26:18.885 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:18.877+0000 7ff81a04b640 1 -- 172.21.15.46:0/3055990536 learned_addr learned my addr 172.21.15.46:0/3055990536 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:26:18.885 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:18.877+0000 7ff81984a640 1 --2- 172.21.15.46:0/3055990536 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55613cb35fe0 0x55613cb38510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:26:18.885 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:18.877+0000 7ff81a84c640 1 --2- 172.21.15.46:0/3055990536 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55613cb3edb0 0x55613cb38a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:26:18.885 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:18.877+0000 7ff81a04b640 1 -- 172.21.15.46:0/3055990536 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55613cb35fe0 msgr2=0x55613cb38510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:26:18.885 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:18.877+0000 7ff81a04b640 1 --2- 172.21.15.46:0/3055990536 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55613cb35fe0 0x55613cb38510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:26:18.885 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:18.877+0000 7ff81a04b640 1 -- 172.21.15.46:0/3055990536 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55613cb3edb0 msgr2=0x55613cb38a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:26:18.885 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:18.877+0000 7ff81a04b640 1 --2- 172.21.15.46:0/3055990536 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55613cb3edb0 0x55613cb38a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:26:18.885 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:18.877+0000 7ff81a04b640 1 -- 172.21.15.46:0/3055990536 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55613cb38f90 con 0x55613cb356c0 2024-03-31T23:26:18.886 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:18.877+0000 7ff81a04b640 1 --2- 172.21.15.46:0/3055990536 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55613cb356c0 0x55613cb46a50 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7ff8100dd1f0 tx=0x7ff8100045e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9694a78098b192a5 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:26:18.886 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:18.877+0000 7ff7fdffb640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-31T23:26:18.886 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:18.878+0000 7ff7fdffb640 1 -- 172.21.15.46:0/3055990536 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8100bd1e0 con 0x55613cb356c0 2024-03-31T23:26:18.886 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:18.878+0000 7ff7fdffb640 1 -- 172.21.15.46:0/3055990536 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(115..115 src has 1..115) ==== 9554+0+0 (secure 0 0 0) 0x7ff8100ee260 con 0x55613cb356c0 2024-03-31T23:26:18.886 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:18.878+0000 7ff81ef6a740 1 -- 172.21.15.46:0/3055990536 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff80c037070 con 0x55613cb356c0 2024-03-31T23:26:18.886 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:18.878+0000 7ff7fdffb640 1 -- 172.21.15.46:0/3055990536 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7ff8100a3050 con 0x55613cb356c0 2024-03-31T23:26:18.886 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:18.878+0000 7ff81ef6a740 10 client.17343 fetch_fsmap learned FSMap version 40 2024-03-31T23:26:18.886 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:18.878+0000 7ff81ef6a740 1 -- 172.21.15.46:0/3055990536 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x55613cbb4b70 con 0x55613cb356c0 2024-03-31T23:26:18.886 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:18.878+0000 7ff7fdffb640 1 -- 172.21.15.46:0/3055990536 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff80c037070 con 0x55613cb356c0 2024-03-31T23:26:18.886 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:18.879+0000 7ff7fdffb640 1 -- 172.21.15.46:0/3055990536 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 40) ==== 8370+0+0 (secure 0 0 0) 0x7ff8100bd440 con 0x55613cb356c0 2024-03-31T23:26:18.886 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:18.879+0000 7ff81ef6a740 10 client.17343 fetch_fsmap finished waiting for FSMap version 40 2024-03-31T23:26:18.886 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:18.879+0000 7ff81ef6a740 10 client.17343 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-31T23:26:18.886 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:18.879+0000 7ff81ef6a740 20 client.17343 populate_metadata read hostname 'smithi046' 2024-03-31T23:26:18.887 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:18.879+0000 7ff81ef6a740 1 --2- 172.21.15.46:0/3055990536 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55613cbb5130 0x55613cbb7550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:26:18.887 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:18.879+0000 7ff81ef6a740 4 client.17343 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-31T23:26:18.887 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:18.879+0000 7ff81ef6a740 1 -- 172.21.15.46:0/3055990536 --> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55613caa5e30 con 0x55613cbb5130 2024-03-31T23:26:18.887 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:18.879+0000 7ff81984a640 1 --2- 172.21.15.46:0/3055990536 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55613cbb5130 0x55613cbb7550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:26:18.887 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:18.879+0000 7ff81984a640 1 --2- 172.21.15.46:0/3055990536 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55613cbb5130 0x55613cbb7550 crc :-1 s=READY pgs=206 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=980c45ea3b559ccb server_cookie=c2354cc269b2cdc6 in_seq=0 out_seq=0 2024-03-31T23:26:18.887 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:18.879+0000 7ff7fdffb640 10 client.17343 ms_handle_connect on v2:172.21.15.142:6838/2519143918 2024-03-31T23:26:18.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:26:18 smithi142 ceph-mon[35271]: pgmap v459: 129 pgs: 129 active+clean; 460 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 4.8 MiB/s wr, 322 op/s 2024-03-31T23:26:19.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:26:18 smithi162 ceph-mon[36237]: pgmap v459: 129 pgs: 129 active+clean; 460 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 4.8 MiB/s wr, 322 op/s 2024-03-31T23:26:19.063 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:19.061+0000 7ff7fdffb640 1 -- 172.21.15.46:0/3055990536 <== mds.0 v2:172.21.15.142:6838/2519143918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55613caa5e30 con 0x55613cbb5130 2024-03-31T23:26:19.063 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:19.061+0000 7ff7fdffb640 10 client.17343 handle_command_reply: tid=0 2024-03-31T23:26:19.063 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:19.061+0000 7ff81ef6a740 1 client.17343 shutdown 2024-03-31T23:26:19.063 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:19.061+0000 7ff81ef6a740 1 -- 172.21.15.46:0/3055990536 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55613cb356c0 msgr2=0x55613cb46a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:26:19.063 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:19.061+0000 7ff81ef6a740 1 --2- 172.21.15.46:0/3055990536 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55613cb356c0 0x55613cb46a50 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7ff8100dd1f0 tx=0x7ff8100045e0 comp rx=0 tx=0).stop 2024-03-31T23:26:19.063 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:19.061+0000 7ff81ef6a740 1 -- 172.21.15.46:0/3055990536 shutdown_connections 2024-03-31T23:26:19.064 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:19.062+0000 7ff81ef6a740 1 --2- 172.21.15.46:0/3055990536 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55613cbb5130 0x55613cbb7550 crc :-1 s=READY pgs=206 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:26:19.064 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:19.062+0000 7ff81ef6a740 1 --2- 172.21.15.46:0/3055990536 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55613cb3edb0 0x55613cb38a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:26:19.064 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:19.062+0000 7ff81ef6a740 1 --2- 172.21.15.46:0/3055990536 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55613cb35fe0 0x55613cb38510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:26:19.064 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:19.062+0000 7ff81ef6a740 1 --2- 172.21.15.46:0/3055990536 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55613cb356c0 0x55613cb46a50 unknown :-1 s=CLOSED pgs=890 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:26:19.064 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:19.062+0000 7ff81ef6a740 1 -- 172.21.15.46:0/3055990536 >> 172.21.15.46:0/3055990536 conn(0x55613cb2ffe0 msgr2=0x55613cb3be10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:26:19.064 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:19.062+0000 7ff7fdffb640 0 client.17343 ms_handle_reset on v2:172.21.15.142:6838/2519143918 2024-03-31T23:26:19.064 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:19.062+0000 7ff81ef6a740 1 -- 172.21.15.46:0/3055990536 shutdown_connections 2024-03-31T23:26:19.064 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:19.062+0000 7ff81ef6a740 1 -- 172.21.15.46:0/3055990536 wait complete. 2024-03-31T23:26:19.064 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:19.062+0000 7ff81ef6a740 20 client.17343 trim_cache size 0 max 16384 2024-03-31T23:26:19.064 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout:{ 2024-03-31T23:26:19.065 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout: "message": "", 2024-03-31T23:26:19.065 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout: "return_code": 0 2024-03-31T23:26:19.065 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout:} 2024-03-31T23:26:19.066 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:19.063+0000 7ff81c2af640 1 -- 172.21.15.46:0/2575508639 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7ff8000a2730 msgr2=0x7ff8000a4bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:26:19.066 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:19.063+0000 7ff81c2af640 1 --2- 172.21.15.46:0/2575508639 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7ff8000a2730 0x7ff8000a4bf0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7ff81410e940 tx=0x7ff804005f90 comp rx=0 tx=0).stop 2024-03-31T23:26:19.066 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:19.063+0000 7ff81c2af640 1 -- 172.21.15.46:0/2575508639 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff814101740 msgr2=0x7ff81419a0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:26:19.066 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:19.063+0000 7ff81c2af640 1 --2- 172.21.15.46:0/2575508639 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff814101740 0x7ff81419a0a0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7ff810009230 tx=0x7ff81002c5d0 comp rx=0 tx=0).stop 2024-03-31T23:26:19.066 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:19.063+0000 7ff81c2af640 1 -- 172.21.15.46:0/2575508639 shutdown_connections 2024-03-31T23:26:19.066 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:19.063+0000 7ff81c2af640 1 --2- 172.21.15.46:0/2575508639 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7ff8000a2730 0x7ff8000a4bf0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:26:19.066 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:19.063+0000 7ff81c2af640 1 --2- 172.21.15.46:0/2575508639 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff814107750 0x7ff81419e960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:26:19.066 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:19.063+0000 7ff81c2af640 1 --2- 172.21.15.46:0/2575508639 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff814102100 0x7ff81419a5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:26:19.066 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:19.063+0000 7ff81c2af640 1 --2- 172.21.15.46:0/2575508639 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff814101740 0x7ff81419a0a0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:26:19.066 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:19.063+0000 7ff81c2af640 1 -- 172.21.15.46:0/2575508639 >> 172.21.15.46:0/2575508639 conn(0x7ff8140fd210 msgr2=0x7ff81410ce10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:26:19.066 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:19.063+0000 7ff81c2af640 1 -- 172.21.15.46:0/2575508639 shutdown_connections 2024-03-31T23:26:19.066 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:19.063+0000 7ff81c2af640 1 -- 172.21.15.46:0/2575508639 wait complete. 2024-03-31T23:26:19.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:26:18 smithi046 ceph-mon[32498]: pgmap v459: 129 pgs: 129 active+clean; 460 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 4.8 MiB/s wr, 322 op/s 2024-03-31T23:26:19.268 INFO:tasks.workunit.client.0.smithi046.stdout: 10 13545 10.76 MB/sec warmup 84 sec latency 129.615 ms 2024-03-31T23:26:20.268 INFO:tasks.workunit.client.0.smithi046.stdout: 10 13704 10.71 MB/sec warmup 85 sec latency 95.675 ms 2024-03-31T23:26:20.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:26:20 smithi142 ceph-mon[35271]: pgmap v460: 129 pgs: 129 active+clean; 470 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 5.5 MiB/s wr, 258 op/s 2024-03-31T23:26:21.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:26:20 smithi162 ceph-mon[36237]: pgmap v460: 129 pgs: 129 active+clean; 470 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 5.5 MiB/s wr, 258 op/s 2024-03-31T23:26:21.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:26:20 smithi046 ceph-mon[32498]: pgmap v460: 129 pgs: 129 active+clean; 470 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 5.5 MiB/s wr, 258 op/s 2024-03-31T23:26:21.268 INFO:tasks.workunit.client.0.smithi046.stdout: 10 13880 10.67 MB/sec warmup 86 sec latency 73.306 ms 2024-03-31T23:26:22.269 INFO:tasks.workunit.client.0.smithi046.stdout: 10 14039 10.62 MB/sec warmup 87 sec latency 88.660 ms 2024-03-31T23:26:23.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:26:22 smithi162 ceph-mon[36237]: pgmap v461: 129 pgs: 129 active+clean; 470 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 5.3 MiB/s wr, 250 op/s 2024-03-31T23:26:23.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:26:22 smithi046 ceph-mon[32498]: pgmap v461: 129 pgs: 129 active+clean; 470 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 5.3 MiB/s wr, 250 op/s 2024-03-31T23:26:23.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:26:22 smithi142 ceph-mon[35271]: pgmap v461: 129 pgs: 129 active+clean; 470 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 5.3 MiB/s wr, 250 op/s 2024-03-31T23:26:23.269 INFO:tasks.workunit.client.0.smithi046.stdout: 10 14215 10.61 MB/sec warmup 88 sec latency 80.963 ms 2024-03-31T23:26:24.269 INFO:tasks.workunit.client.0.smithi046.stdout: 10 14420 10.71 MB/sec warmup 89 sec latency 80.923 ms 2024-03-31T23:26:25.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:26:24 smithi162 ceph-mon[36237]: pgmap v462: 129 pgs: 129 active+clean; 470 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 4.6 MiB/s wr, 215 op/s 2024-03-31T23:26:25.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:26:24 smithi046 ceph-mon[32498]: pgmap v462: 129 pgs: 129 active+clean; 470 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 4.6 MiB/s wr, 215 op/s 2024-03-31T23:26:25.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:26:24 smithi142 ceph-mon[35271]: pgmap v462: 129 pgs: 129 active+clean; 470 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 4.6 MiB/s wr, 215 op/s 2024-03-31T23:26:25.269 INFO:tasks.workunit.client.0.smithi046.stdout: 10 14597 10.70 MB/sec warmup 90 sec latency 58.674 ms 2024-03-31T23:26:26.269 INFO:tasks.workunit.client.0.smithi046.stdout: 10 14772 10.68 MB/sec warmup 91 sec latency 64.143 ms 2024-03-31T23:26:27.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:26:26 smithi162 ceph-mon[36237]: pgmap v463: 129 pgs: 129 active+clean; 472 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 6.8 MiB/s wr, 411 op/s 2024-03-31T23:26:27.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:26:26 smithi046 ceph-mon[32498]: pgmap v463: 129 pgs: 129 active+clean; 472 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 6.8 MiB/s wr, 411 op/s 2024-03-31T23:26:27.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:26:26 smithi142 ceph-mon[35271]: pgmap v463: 129 pgs: 129 active+clean; 472 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 6.8 MiB/s wr, 411 op/s 2024-03-31T23:26:27.269 INFO:tasks.workunit.client.0.smithi046.stdout: 10 14923 10.64 MB/sec warmup 92 sec latency 166.600 ms 2024-03-31T23:26:28.269 INFO:tasks.workunit.client.0.smithi046.stdout: 10 15088 10.60 MB/sec warmup 93 sec latency 287.306 ms 2024-03-31T23:26:29.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:26:28 smithi162 ceph-mon[36237]: pgmap v464: 129 pgs: 129 active+clean; 472 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 4.5 MiB/s wr, 305 op/s 2024-03-31T23:26:29.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:26:28 smithi046 ceph-mon[32498]: pgmap v464: 129 pgs: 129 active+clean; 472 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 4.5 MiB/s wr, 305 op/s 2024-03-31T23:26:29.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:26:28 smithi142 ceph-mon[35271]: pgmap v464: 129 pgs: 129 active+clean; 472 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 4.5 MiB/s wr, 305 op/s 2024-03-31T23:26:29.269 INFO:tasks.workunit.client.0.smithi046.stdout: 10 15246 10.54 MB/sec warmup 94 sec latency 78.007 ms 2024-03-31T23:26:30.269 INFO:tasks.workunit.client.0.smithi046.stdout: 10 15400 10.48 MB/sec warmup 95 sec latency 62.640 ms 2024-03-31T23:26:31.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:26:30 smithi162 ceph-mon[36237]: pgmap v465: 129 pgs: 129 active+clean; 474 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 6.0 MiB/s wr, 424 op/s 2024-03-31T23:26:31.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:26:30 smithi046 ceph-mon[32498]: pgmap v465: 129 pgs: 129 active+clean; 474 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 6.0 MiB/s wr, 424 op/s 2024-03-31T23:26:31.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:26:30 smithi142 ceph-mon[35271]: pgmap v465: 129 pgs: 129 active+clean; 474 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 6.0 MiB/s wr, 424 op/s 2024-03-31T23:26:31.269 INFO:tasks.workunit.client.0.smithi046.stdout: 10 15564 10.46 MB/sec warmup 96 sec latency 93.968 ms 2024-03-31T23:26:32.204 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:32.195+0000 7f8b23fb6640 1 -- 172.21.15.46:0/1023668533 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8b1c100ff0 msgr2=0x7f8b1c10d500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:26:32.204 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:32.195+0000 7f8b23fb6640 1 --2- 172.21.15.46:0/1023668533 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8b1c100ff0 0x7f8b1c10d500 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7f8b0c009cd0 tx=0x7f8b0c02c0f0 comp rx=0 tx=0).stop 2024-03-31T23:26:32.204 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:32.196+0000 7f8b23fb6640 1 -- 172.21.15.46:0/1023668533 shutdown_connections 2024-03-31T23:26:32.205 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:32.196+0000 7f8b23fb6640 1 --2- 172.21.15.46:0/1023668533 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8b1c100ff0 0x7f8b1c10d500 unknown :-1 s=CLOSED pgs=891 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:26:32.205 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:32.196+0000 7f8b23fb6640 1 --2- 172.21.15.46:0/1023668533 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8b1c100630 0x7f8b1c100ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:26:32.205 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:32.196+0000 7f8b23fb6640 1 --2- 172.21.15.46:0/1023668533 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8b1c106640 0x7f8b1c106a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:26:32.205 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:32.196+0000 7f8b23fb6640 1 -- 172.21.15.46:0/1023668533 >> 172.21.15.46:0/1023668533 conn(0x7f8b1c0777b0 msgr2=0x7f8b1c077bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:26:32.205 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:32.196+0000 7f8b23fb6640 1 -- 172.21.15.46:0/1023668533 shutdown_connections 2024-03-31T23:26:32.205 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:32.196+0000 7f8b23fb6640 1 -- 172.21.15.46:0/1023668533 wait complete. 2024-03-31T23:26:32.205 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:32.196+0000 7f8b23fb6640 1 Processor -- start 2024-03-31T23:26:32.205 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:32.197+0000 7f8b23fb6640 1 -- start start 2024-03-31T23:26:32.206 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:32.197+0000 7f8b23fb6640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8b1c100630 0x7f8b1c197e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:26:32.206 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:32.197+0000 7f8b23fb6640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8b1c100ff0 0x7f8b1c1983b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:26:32.206 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:32.197+0000 7f8b23fb6640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8b1c106640 0x7f8b1c19c6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:26:32.206 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:32.197+0000 7f8b21d52640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8b1c100630 0x7f8b1c197e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:26:32.206 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:32.197+0000 7f8b21d52640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8b1c100630 0x7f8b1c197e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:47108/0 (socket says 172.21.15.46:47108) 2024-03-31T23:26:32.206 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:32.197+0000 7f8b21d52640 1 -- 172.21.15.46:0/368812698 learned_addr learned my addr 172.21.15.46:0/368812698 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:26:32.206 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:32.197+0000 7f8b23fb6640 1 -- 172.21.15.46:0/368812698 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f8b1c1108c0 con 0x7f8b1c100630 2024-03-31T23:26:32.206 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:32.197+0000 7f8b23fb6640 1 -- 172.21.15.46:0/368812698 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f8b1c110740 con 0x7f8b1c106640 2024-03-31T23:26:32.206 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:32.197+0000 7f8b23fb6640 1 -- 172.21.15.46:0/368812698 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f8b1c110a40 con 0x7f8b1c100ff0 2024-03-31T23:26:32.207 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:32.197+0000 7f8b21551640 1 --2- 172.21.15.46:0/368812698 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8b1c100ff0 0x7f8b1c1983b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:26:32.207 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:32.198+0000 7f8b22553640 1 --2- 172.21.15.46:0/368812698 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8b1c106640 0x7f8b1c19c6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:26:32.207 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:32.198+0000 7f8b21d52640 1 -- 172.21.15.46:0/368812698 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8b1c100ff0 msgr2=0x7f8b1c1983b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:26:32.207 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:32.198+0000 7f8b21d52640 1 --2- 172.21.15.46:0/368812698 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8b1c100ff0 0x7f8b1c1983b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:26:32.207 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:32.198+0000 7f8b21d52640 1 -- 172.21.15.46:0/368812698 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8b1c106640 msgr2=0x7f8b1c19c6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:26:32.207 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:32.198+0000 7f8b21d52640 1 --2- 172.21.15.46:0/368812698 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8b1c106640 0x7f8b1c19c6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:26:32.207 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:32.198+0000 7f8b21d52640 1 -- 172.21.15.46:0/368812698 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b1c19ce10 con 0x7f8b1c100630 2024-03-31T23:26:32.207 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:32.198+0000 7f8b21d52640 1 --2- 172.21.15.46:0/368812698 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8b1c100630 0x7f8b1c197e70 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7f8b1400ea60 tx=0x7f8b1400ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:26:32.207 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:32.198+0000 7f8b1affd640 1 -- 172.21.15.46:0/368812698 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b1400cde0 con 0x7f8b1c100630 2024-03-31T23:26:32.207 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:32.198+0000 7f8b1affd640 1 -- 172.21.15.46:0/368812698 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8b140035f0 con 0x7f8b1c100630 2024-03-31T23:26:32.207 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:32.198+0000 7f8b1affd640 1 -- 172.21.15.46:0/368812698 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b14010690 con 0x7f8b1c100630 2024-03-31T23:26:32.208 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:32.198+0000 7f8b23fb6640 1 -- 172.21.15.46:0/368812698 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8b1c19d100 con 0x7f8b1c100630 2024-03-31T23:26:32.208 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:32.199+0000 7f8b23fb6640 1 -- 172.21.15.46:0/368812698 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8b1c0fd520 con 0x7f8b1c100630 2024-03-31T23:26:32.208 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:32.202+0000 7f8b1affd640 1 -- 172.21.15.46:0/368812698 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 25) ==== 137451+0+0 (secure 0 0 0) 0x7f8b140037a0 con 0x7f8b1c100630 2024-03-31T23:26:32.214 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:32.202+0000 7f8b26b94740 1 Processor -- start 2024-03-31T23:26:32.214 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:32.202+0000 7f8b26b94740 1 -- start start 2024-03-31T23:26:32.214 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:32.202+0000 7f8b26b94740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55b20149d6c0 0x55b20149daa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:26:32.215 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:32.202+0000 7f8b26b94740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b20149dfe0 0x55b2014af080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:26:32.215 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:32.202+0000 7f8b26b94740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b2014af5c0 0x55b2014b19b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:26:32.215 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:32.202+0000 7f8b26b94740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55b2014008f0 con 0x55b20149dfe0 2024-03-31T23:26:32.215 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:32.202+0000 7f8b26b94740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55b2013a98d0 con 0x55b20149d6c0 2024-03-31T23:26:32.215 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:32.202+0000 7f8b26b94740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55b2013fc990 con 0x55b2014af5c0 2024-03-31T23:26:32.215 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:32.203+0000 7f8b21d52640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b20149dfe0 0x55b2014af080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:26:32.215 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:32.203+0000 7f8b22553640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55b20149d6c0 0x55b20149daa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:26:32.215 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:32.203+0000 7f8b22553640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55b20149d6c0 0x55b20149daa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.46:53688/0 (socket says 172.21.15.46:53688) 2024-03-31T23:26:32.215 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:32.203+0000 7f8b21d52640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b20149dfe0 0x55b2014af080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:47122/0 (socket says 172.21.15.46:47122) 2024-03-31T23:26:32.215 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:32.203+0000 7f8b22553640 1 -- 172.21.15.46:0/1463753221 learned_addr learned my addr 172.21.15.46:0/1463753221 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:26:32.216 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:32.203+0000 7f8b21551640 1 --2- 172.21.15.46:0/1463753221 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b2014af5c0 0x55b2014b19b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:26:32.216 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:32.203+0000 7f8b1affd640 1 --2- 172.21.15.46:0/368812698 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f8b040a2730 0x7f8b040a4bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:26:32.216 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:32.203+0000 7f8b22553640 1 -- 172.21.15.46:0/1463753221 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b2014af5c0 msgr2=0x55b2014b19b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:26:32.216 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:32.203+0000 7f8b22553640 1 --2- 172.21.15.46:0/1463753221 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b2014af5c0 0x55b2014b19b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:26:32.216 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:32.203+0000 7f8b22553640 1 -- 172.21.15.46:0/1463753221 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b20149dfe0 msgr2=0x55b2014af080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:26:32.216 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:32.203+0000 7f8b22553640 1 --2- 172.21.15.46:0/1463753221 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b20149dfe0 0x55b2014af080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:26:32.216 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:32.203+0000 7f8b22553640 1 -- 172.21.15.46:0/1463753221 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b201115210 con 0x55b20149d6c0 2024-03-31T23:26:32.216 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:32.203+0000 7f8b21d52640 1 --2- 172.21.15.46:0/1463753221 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b20149dfe0 0x55b2014af080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:26:32.216 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:32.203+0000 7f8b1affd640 1 -- 172.21.15.46:0/368812698 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(115..115 src has 1..115) ==== 9554+0+0 (secure 0 0 0) 0x7f8b140cf0c0 con 0x7f8b1c100630 2024-03-31T23:26:32.216 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:32.203+0000 7f8b22553640 1 --2- 172.21.15.46:0/368812698 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f8b040a2730 0x7f8b040a4bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:26:32.217 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:32.204+0000 7f8b22553640 1 --2- 172.21.15.46:0/1463753221 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55b20149d6c0 0x55b20149daa0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f8b0c02c700 tx=0x7f8b0c032340 comp rx=0 tx=0).ready entity=mon.2 client_cookie=af620276599a1ad8 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:26:32.217 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:32.204+0000 7f8b03fff640 1 -- 172.21.15.46:0/1463753221 <== mon.2 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b0c039040 con 0x55b20149d6c0 2024-03-31T23:26:32.217 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:32.204+0000 7f8b03fff640 1 -- 172.21.15.46:0/1463753221 <== mon.2 v2:172.21.15.142:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8b0c043050 con 0x55b20149d6c0 2024-03-31T23:26:32.217 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:32.204+0000 7f8b03fff640 1 -- 172.21.15.46:0/1463753221 <== mon.2 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b0c051070 con 0x55b20149d6c0 2024-03-31T23:26:32.217 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:32.205+0000 7f8b26b94740 1 -- 172.21.15.46:0/1463753221 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55b20149d6c0 msgr2=0x55b20149daa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:26:32.217 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:32.205+0000 7f8b26b94740 1 --2- 172.21.15.46:0/1463753221 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55b20149d6c0 0x55b20149daa0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f8b0c02c700 tx=0x7f8b0c032340 comp rx=0 tx=0).stop 2024-03-31T23:26:32.217 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:32.205+0000 7f8b22553640 1 --2- 172.21.15.46:0/368812698 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f8b040a2730 0x7f8b040a4bf0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f8b0c009a60 tx=0x7f8b0c04b000 comp rx=0 tx=0).ready entity=mgr.15654 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:26:32.217 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:32.205+0000 7f8b26b94740 1 -- 172.21.15.46:0/1463753221 shutdown_connections 2024-03-31T23:26:32.217 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:32.205+0000 7f8b26b94740 1 --2- 172.21.15.46:0/1463753221 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b2014af5c0 0x55b2014b19b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:26:32.217 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:32.205+0000 7f8b26b94740 1 --2- 172.21.15.46:0/1463753221 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b20149dfe0 0x55b2014af080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:26:32.217 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:32.205+0000 7f8b26b94740 1 --2- 172.21.15.46:0/1463753221 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55b20149d6c0 0x55b20149daa0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:26:32.217 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:32.205+0000 7f8b26b94740 1 -- 172.21.15.46:0/1463753221 >> 172.21.15.46:0/1463753221 conn(0x55b201497fe0 msgr2=0x55b2014983c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:26:32.217 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:32.205+0000 7f8b26b94740 1 -- 172.21.15.46:0/1463753221 shutdown_connections 2024-03-31T23:26:32.217 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:32.205+0000 7f8b26b94740 1 -- 172.21.15.46:0/1463753221 wait complete. 2024-03-31T23:26:32.218 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:32.206+0000 7f8b26b94740 1 -- start start 2024-03-31T23:26:32.218 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:32.206+0000 7f8b26b94740 1 Processor -- start 2024-03-31T23:26:32.218 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:32.206+0000 7f8b26b94740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55b20149d6c0 0x55b2014ac5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:26:32.218 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:32.206+0000 7f8b26b94740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b20149dfe0 0x55b2014acae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:26:32.218 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:32.206+0000 7f8b26b94740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b2014af5c0 0x55b2014ad020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:26:32.218 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:32.207+0000 7f8b26b94740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55b2014b21f0 con 0x55b20149dfe0 2024-03-31T23:26:32.218 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:32.207+0000 7f8b21d52640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55b20149d6c0 0x55b2014ac5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:26:32.218 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:32.207+0000 7f8b21d52640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55b20149d6c0 0x55b2014ac5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.46:53690/0 (socket says 172.21.15.46:53690) 2024-03-31T23:26:32.218 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:32.207+0000 7f8b21d52640 1 -- 172.21.15.46:0/2936200236 learned_addr learned my addr 172.21.15.46:0/2936200236 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:26:32.218 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:32.207+0000 7f8b26b94740 1 -- 172.21.15.46:0/2936200236 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55b20139ef80 con 0x55b20149d6c0 2024-03-31T23:26:32.218 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:32.207+0000 7f8b26b94740 1 -- 172.21.15.46:0/2936200236 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55b2014b2370 con 0x55b2014af5c0 2024-03-31T23:26:32.219 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:32.207+0000 7f8b22553640 1 --2- 172.21.15.46:0/2936200236 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b2014af5c0 0x55b2014ad020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:26:32.219 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:32.207+0000 7f8b26b94740 10 client.0 _ll_register_callbacks cb 0x55b2013f7290 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T23:26:32.219 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:32.207+0000 7f8b21d52640 1 -- 172.21.15.46:0/2936200236 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b2014af5c0 msgr2=0x55b2014ad020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:26:32.219 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:32.207+0000 7f8b21d52640 1 --2- 172.21.15.46:0/2936200236 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b2014af5c0 0x55b2014ad020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:26:32.219 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:32.207+0000 7f8b21d52640 1 -- 172.21.15.46:0/2936200236 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b20149dfe0 msgr2=0x55b2014acae0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-31T23:26:32.219 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:32.207+0000 7f8b21551640 1 --2- 172.21.15.46:0/2936200236 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b20149dfe0 0x55b2014acae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:26:32.219 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:32.207+0000 7f8b21d52640 1 --2- 172.21.15.46:0/2936200236 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b20149dfe0 0x55b2014acae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:26:32.219 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:32.207+0000 7f8b21d52640 1 -- 172.21.15.46:0/2936200236 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b2011e1620 con 0x55b20149d6c0 2024-03-31T23:26:32.219 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:32.208+0000 7f8b26b94740 1 -- 172.21.15.46:0/2936200236 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8b0c043050 con 0x55b20149d6c0 2024-03-31T23:26:32.219 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:32.208+0000 7f8b21551640 1 --2- 172.21.15.46:0/2936200236 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b20149dfe0 0x55b2014acae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T23:26:32.219 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:32.208+0000 7f8b21d52640 1 --2- 172.21.15.46:0/2936200236 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55b20149d6c0 0x55b2014ac5a0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f8b14009920 tx=0x7f8b14003e30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6dd60bdf47f4445e server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:26:32.219 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:32.208+0000 7f8b017fa640 10 client.27185 ms_handle_connect on v2:172.21.15.142:3300/0 2024-03-31T23:26:32.219 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:32.208+0000 7f8b017fa640 1 -- 172.21.15.46:0/2936200236 <== mon.2 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b140976d0 con 0x55b20149d6c0 2024-03-31T23:26:32.219 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:32.208+0000 7f8b017fa640 1 -- 172.21.15.46:0/2936200236 <== mon.2 v2:172.21.15.142:3300/0 2 ==== osd_map(115..115 src has 1..115) ==== 9554+0+0 (secure 0 0 0) 0x7f8b1409f110 con 0x55b20149d6c0 2024-03-31T23:26:32.220 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:32.208+0000 7f8b017fa640 1 -- 172.21.15.46:0/2936200236 <== mon.2 v2:172.21.15.142:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7f8b1409ad90 con 0x55b20149d6c0 2024-03-31T23:26:32.220 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:32.209+0000 7f8b26b94740 10 client.27185 fetch_fsmap learned FSMap version 40 2024-03-31T23:26:32.220 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:32.209+0000 7f8b26b94740 1 -- 172.21.15.46:0/2936200236 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({fsmap=40}) -- 0x55b2014bdfd0 con 0x55b20149d6c0 2024-03-31T23:26:32.220 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:32.209+0000 7f8b017fa640 1 -- 172.21.15.46:0/2936200236 <== mon.2 v2:172.21.15.142:3300/0 4 ==== fsmap(e 40) ==== 8370+0+0 (secure 0 0 0) 0x7f8b140a39b0 con 0x55b20149d6c0 2024-03-31T23:26:32.220 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:32.209+0000 7f8b26b94740 10 client.27185 fetch_fsmap finished waiting for FSMap version 40 2024-03-31T23:26:32.220 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:32.209+0000 7f8b26b94740 10 client.27185 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-31T23:26:32.220 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:32.209+0000 7f8b26b94740 20 client.27185 populate_metadata read hostname 'smithi046' 2024-03-31T23:26:32.220 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:32.209+0000 7f8b26b94740 1 --2- 172.21.15.46:0/2936200236 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55b2014bf3f0 0x55b2014c1880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:26:32.220 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:32.209+0000 7f8b26b94740 4 client.27185 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-31T23:26:32.220 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:32.209+0000 7f8b26b94740 1 -- 172.21.15.46:0/2936200236 --> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b201414120 con 0x55b2014bf3f0 2024-03-31T23:26:32.220 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:32.210+0000 7f8b21551640 1 --2- 172.21.15.46:0/2936200236 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55b2014bf3f0 0x55b2014c1880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:26:32.220 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:32.210+0000 7f8b21551640 1 --2- 172.21.15.46:0/2936200236 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55b2014bf3f0 0x55b2014c1880 crc :-1 s=READY pgs=207 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c1d6194516d4f443 server_cookie=83c370b5992dc7aa in_seq=0 out_seq=0 2024-03-31T23:26:32.220 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:32.211+0000 7f8b017fa640 10 client.27185 ms_handle_connect on v2:172.21.15.142:6838/2519143918 2024-03-31T23:26:32.221 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:32.214+0000 7f8b017fa640 1 -- 172.21.15.46:0/2936200236 <== mds.0 v2:172.21.15.142:6838/2519143918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55b201414120 con 0x55b2014bf3f0 2024-03-31T23:26:32.221 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:32.214+0000 7f8b017fa640 10 client.27185 handle_command_reply: tid=0 2024-03-31T23:26:32.221 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:32.214+0000 7f8b26b94740 1 client.27185 shutdown 2024-03-31T23:26:32.230 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:32.214+0000 7f8b26b94740 1 -- 172.21.15.46:0/2936200236 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55b20149d6c0 msgr2=0x55b2014ac5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:26:32.231 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:32.214+0000 7f8b26b94740 1 --2- 172.21.15.46:0/2936200236 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55b20149d6c0 0x55b2014ac5a0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f8b14009920 tx=0x7f8b14003e30 comp rx=0 tx=0).stop 2024-03-31T23:26:32.231 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:32.214+0000 7f8b26b94740 1 -- 172.21.15.46:0/2936200236 shutdown_connections 2024-03-31T23:26:32.231 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:32.214+0000 7f8b26b94740 1 --2- 172.21.15.46:0/2936200236 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55b2014bf3f0 0x55b2014c1880 crc :-1 s=READY pgs=207 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:26:32.231 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:32.214+0000 7f8b017fa640 0 client.27185 ms_handle_reset on v2:172.21.15.142:6838/2519143918 2024-03-31T23:26:32.231 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:32.214+0000 7f8b26b94740 1 --2- 172.21.15.46:0/2936200236 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b2014af5c0 0x55b2014ad020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:26:32.231 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:32.214+0000 7f8b26b94740 1 --2- 172.21.15.46:0/2936200236 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b20149dfe0 0x55b2014acae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:26:32.231 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:32.214+0000 7f8b26b94740 1 --2- 172.21.15.46:0/2936200236 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55b20149d6c0 0x55b2014ac5a0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:26:32.231 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:32.214+0000 7f8b26b94740 1 -- 172.21.15.46:0/2936200236 >> 172.21.15.46:0/2936200236 conn(0x55b201497fe0 msgr2=0x55b201499d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:26:32.232 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:32.215+0000 7f8b26b94740 1 -- 172.21.15.46:0/2936200236 shutdown_connections 2024-03-31T23:26:32.232 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:32.215+0000 7f8b26b94740 1 -- 172.21.15.46:0/2936200236 wait complete. 2024-03-31T23:26:32.232 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:32.215+0000 7f8b26b94740 20 client.27185 trim_cache size 0 max 16384 2024-03-31T23:26:32.248 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:32.245+0000 7f8b26b94740 1 Processor -- start 2024-03-31T23:26:32.248 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:32.245+0000 7f8b26b94740 1 -- start start 2024-03-31T23:26:32.248 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:32.245+0000 7f8b26b94740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b20149d6c0 0x55b2014aec60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:26:32.248 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:32.245+0000 7f8b26b94740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b20149dfe0 0x55b2014baf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:26:32.248 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:32.245+0000 7f8b26b94740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55b2014af5c0 0x55b201499b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:26:32.248 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:32.245+0000 7f8b21d52640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b20149d6c0 0x55b2014aec60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:26:32.248 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:32.246+0000 7f8b26b94740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f8b1c1af550 con 0x55b20149d6c0 2024-03-31T23:26:32.248 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:32.246+0000 7f8b26b94740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f8b1c1acd00 con 0x55b2014af5c0 2024-03-31T23:26:32.248 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:32.246+0000 7f8b21d52640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b20149d6c0 0x55b2014aec60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:47138/0 (socket says 172.21.15.46:47138) 2024-03-31T23:26:32.249 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:32.246+0000 7f8b26b94740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f8b1c1aab70 con 0x55b20149dfe0 2024-03-31T23:26:32.249 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:32.246+0000 7f8b21d52640 1 -- 172.21.15.46:0/2758762154 learned_addr learned my addr 172.21.15.46:0/2758762154 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:26:32.249 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:32.246+0000 7f8b21551640 1 --2- 172.21.15.46:0/2758762154 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b20149dfe0 0x55b2014baf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:26:32.249 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:32.246+0000 7f8b22553640 1 --2- 172.21.15.46:0/2758762154 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55b2014af5c0 0x55b201499b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:26:32.249 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:32.246+0000 7f8b21d52640 1 -- 172.21.15.46:0/2758762154 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b20149dfe0 msgr2=0x55b2014baf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:26:32.249 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:32.246+0000 7f8b21d52640 1 --2- 172.21.15.46:0/2758762154 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b20149dfe0 0x55b2014baf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:26:32.249 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:32.246+0000 7f8b21d52640 1 -- 172.21.15.46:0/2758762154 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55b2014af5c0 msgr2=0x55b201499b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:26:32.249 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:32.246+0000 7f8b21d52640 1 --2- 172.21.15.46:0/2758762154 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55b2014af5c0 0x55b201499b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:26:32.249 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:32.246+0000 7f8b21d52640 1 -- 172.21.15.46:0/2758762154 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b2014c1f00 con 0x55b20149d6c0 2024-03-31T23:26:32.249 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:32.246+0000 7f8b21d52640 1 --2- 172.21.15.46:0/2758762154 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b20149d6c0 0x55b2014aec60 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7f8b140cca00 tx=0x7f8b14097000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b730d5ca71873522 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:26:32.254 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:32.246+0000 7f8b037fe640 1 -- 172.21.15.46:0/2758762154 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b1409af70 con 0x55b20149d6c0 2024-03-31T23:26:32.254 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:32.247+0000 7f8b037fe640 1 -- 172.21.15.46:0/2758762154 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8b0c043050 con 0x55b20149d6c0 2024-03-31T23:26:32.254 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:32.247+0000 7f8b037fe640 1 -- 172.21.15.46:0/2758762154 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b1409b200 con 0x55b20149d6c0 2024-03-31T23:26:32.254 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:32.247+0000 7f8b26b94740 1 -- 172.21.15.46:0/2758762154 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b20149d6c0 msgr2=0x55b2014aec60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:26:32.254 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:32.247+0000 7f8b26b94740 1 --2- 172.21.15.46:0/2758762154 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b20149d6c0 0x55b2014aec60 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7f8b140cca00 tx=0x7f8b14097000 comp rx=0 tx=0).stop 2024-03-31T23:26:32.254 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:32.247+0000 7f8b26b94740 1 -- 172.21.15.46:0/2758762154 shutdown_connections 2024-03-31T23:26:32.254 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:32.247+0000 7f8b26b94740 1 --2- 172.21.15.46:0/2758762154 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55b2014af5c0 0x55b201499b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:26:32.255 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:32.247+0000 7f8b26b94740 1 --2- 172.21.15.46:0/2758762154 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b20149dfe0 0x55b2014baf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:26:32.255 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:32.247+0000 7f8b26b94740 1 --2- 172.21.15.46:0/2758762154 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b20149d6c0 0x55b2014aec60 unknown :-1 s=CLOSED pgs=893 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:26:32.255 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:32.247+0000 7f8b26b94740 1 -- 172.21.15.46:0/2758762154 >> 172.21.15.46:0/2758762154 conn(0x55b201497fe0 msgr2=0x55b201499430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:26:32.255 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:32.247+0000 7f8b26b94740 1 -- 172.21.15.46:0/2758762154 shutdown_connections 2024-03-31T23:26:32.255 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:32.247+0000 7f8b26b94740 1 -- 172.21.15.46:0/2758762154 wait complete. 2024-03-31T23:26:32.255 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:32.248+0000 7f8b26b94740 1 -- start start 2024-03-31T23:26:32.255 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:32.248+0000 7f8b26b94740 1 Processor -- start 2024-03-31T23:26:32.255 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:32.248+0000 7f8b26b94740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55b20149d6c0 0x55b2014a9930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:26:32.255 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:32.248+0000 7f8b26b94740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b20149dfe0 0x55b2014a9e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:26:32.255 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:32.248+0000 7f8b26b94740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b2014af5c0 0x55b2015247c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:26:32.255 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:32.248+0000 7f8b26b94740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f8b1c1ab1c0 con 0x55b20149dfe0 2024-03-31T23:26:32.255 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:32.248+0000 7f8b26b94740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f8b1c1ac630 con 0x55b20149d6c0 2024-03-31T23:26:32.255 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:32.248+0000 7f8b26b94740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f8b1c1ab890 con 0x55b2014af5c0 2024-03-31T23:26:32.256 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:32.248+0000 7f8b26b94740 10 client.0 _ll_register_callbacks cb 0x7f8b1400ef90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T23:26:32.256 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:32.249+0000 7f8b21d52640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55b20149d6c0 0x55b2014a9930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:26:32.256 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:32.249+0000 7f8b22553640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b2014af5c0 0x55b2015247c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:26:32.256 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:32.249+0000 7f8b21d52640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55b20149d6c0 0x55b2014a9930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.46:53710/0 (socket says 172.21.15.46:53710) 2024-03-31T23:26:32.256 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:32.249+0000 7f8b21d52640 1 -- 172.21.15.46:0/4173412473 learned_addr learned my addr 172.21.15.46:0/4173412473 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:26:32.256 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:32.249+0000 7f8b21551640 1 --2- 172.21.15.46:0/4173412473 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b20149dfe0 0x55b2014a9e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:26:32.256 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:32.249+0000 7f8b21d52640 1 -- 172.21.15.46:0/4173412473 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b2014af5c0 msgr2=0x55b2015247c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:26:32.256 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:32.249+0000 7f8b21d52640 1 --2- 172.21.15.46:0/4173412473 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b2014af5c0 0x55b2015247c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:26:32.256 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:32.249+0000 7f8b21d52640 1 -- 172.21.15.46:0/4173412473 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b20149dfe0 msgr2=0x55b2014a9e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:26:32.256 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:32.249+0000 7f8b21d52640 1 --2- 172.21.15.46:0/4173412473 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b20149dfe0 0x55b2014a9e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:26:32.256 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:32.249+0000 7f8b21d52640 1 -- 172.21.15.46:0/4173412473 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b201524d80 con 0x55b20149d6c0 2024-03-31T23:26:32.256 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:32.249+0000 7f8b22553640 1 --2- 172.21.15.46:0/4173412473 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b2014af5c0 0x55b2015247c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:26:32.256 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:32.249+0000 7f8b21d52640 1 --2- 172.21.15.46:0/4173412473 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55b20149d6c0 0x55b2014a9930 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f8b14097e00 tx=0x7f8b140d2a30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=752c607d9435dd36 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:26:32.256 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:32.249+0000 7f8b017fa640 10 client.0 ms_handle_connect on v2:172.21.15.142:3300/0 2024-03-31T23:26:32.257 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:32.249+0000 7f8b017fa640 1 -- 172.21.15.46:0/4173412473 <== mon.2 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b140b9020 con 0x55b20149d6c0 2024-03-31T23:26:32.257 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:32.250+0000 7f8b26b94740 1 -- 172.21.15.46:0/4173412473 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8b0c043050 con 0x55b20149d6c0 2024-03-31T23:26:32.257 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:32.250+0000 7f8b017fa640 1 -- 172.21.15.46:0/4173412473 <== mon.2 v2:172.21.15.142:3300/0 2 ==== osd_map(115..115 src has 1..115) ==== 9554+0+0 (secure 0 0 0) 0x7f8b1409a880 con 0x55b20149d6c0 2024-03-31T23:26:32.257 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:32.250+0000 7f8b017fa640 1 -- 172.21.15.46:0/4173412473 <== mon.2 v2:172.21.15.142:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7f8b1409b200 con 0x55b20149d6c0 2024-03-31T23:26:32.257 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:32.250+0000 7f8b26b94740 10 client.27188 fetch_fsmap learned FSMap version 40 2024-03-31T23:26:32.257 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:32.250+0000 7f8b26b94740 1 -- 172.21.15.46:0/4173412473 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x55b201524d80 con 0x55b20149d6c0 2024-03-31T23:26:32.257 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:32.250+0000 7f8b017fa640 1 -- 172.21.15.46:0/4173412473 <== mon.2 v2:172.21.15.142:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8b0c043050 con 0x55b20149d6c0 2024-03-31T23:26:32.257 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:32.250+0000 7f8b017fa640 1 -- 172.21.15.46:0/4173412473 <== mon.2 v2:172.21.15.142:3300/0 5 ==== fsmap(e 40) ==== 8370+0+0 (secure 0 0 0) 0x7f8b14097180 con 0x55b20149d6c0 2024-03-31T23:26:32.257 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:32.250+0000 7f8b26b94740 10 client.27188 fetch_fsmap finished waiting for FSMap version 40 2024-03-31T23:26:32.257 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:32.250+0000 7f8b26b94740 10 client.27188 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-31T23:26:32.257 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:32.250+0000 7f8b26b94740 20 client.27188 populate_metadata read hostname 'smithi046' 2024-03-31T23:26:32.257 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:32.250+0000 7f8b26b94740 1 --2- 172.21.15.46:0/4173412473 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55b201526340 0x55b201528730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:26:32.257 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:32.251+0000 7f8b26b94740 4 client.27188 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-31T23:26:32.257 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:32.251+0000 7f8b26b94740 1 -- 172.21.15.46:0/4173412473 --> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55b20140de30 con 0x55b201526340 2024-03-31T23:26:32.258 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:32.251+0000 7f8b21551640 1 --2- 172.21.15.46:0/4173412473 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55b201526340 0x55b201528730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:26:32.258 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:32.251+0000 7f8b017fa640 10 client.27188 ms_handle_connect on v2:172.21.15.142:6838/2519143918 2024-03-31T23:26:32.258 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:32.251+0000 7f8b21551640 1 --2- 172.21.15.46:0/4173412473 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55b201526340 0x55b201528730 crc :-1 s=READY pgs=208 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6a97515fbe6319af server_cookie=d83580aee0542dda in_seq=0 out_seq=0 2024-03-31T23:26:32.270 INFO:tasks.workunit.client.0.smithi046.stdout: 10 15747 10.48 MB/sec warmup 97 sec latency 85.860 ms 2024-03-31T23:26:32.389 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:32.386+0000 7f8b017fa640 1 -- 172.21.15.46:0/4173412473 <== mds.0 v2:172.21.15.142:6838/2519143918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55b20140de30 con 0x55b201526340 2024-03-31T23:26:32.389 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:32.386+0000 7f8b017fa640 10 client.27188 handle_command_reply: tid=0 2024-03-31T23:26:32.389 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:32.386+0000 7f8b26b94740 1 client.27188 shutdown 2024-03-31T23:26:32.389 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:32.386+0000 7f8b26b94740 1 -- 172.21.15.46:0/4173412473 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55b20149d6c0 msgr2=0x55b2014a9930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:26:32.389 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:32.386+0000 7f8b26b94740 1 --2- 172.21.15.46:0/4173412473 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55b20149d6c0 0x55b2014a9930 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f8b14097e00 tx=0x7f8b140d2a30 comp rx=0 tx=0).stop 2024-03-31T23:26:32.389 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:32.387+0000 7f8b26b94740 1 -- 172.21.15.46:0/4173412473 shutdown_connections 2024-03-31T23:26:32.389 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:32.387+0000 7f8b26b94740 1 --2- 172.21.15.46:0/4173412473 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55b201526340 0x55b201528730 crc :-1 s=READY pgs=208 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:26:32.389 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:32.387+0000 7f8b017fa640 0 client.27188 ms_handle_reset on v2:172.21.15.142:6838/2519143918 2024-03-31T23:26:32.389 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:32.387+0000 7f8b26b94740 1 --2- 172.21.15.46:0/4173412473 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b2014af5c0 0x55b2015247c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:26:32.389 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:32.387+0000 7f8b26b94740 1 --2- 172.21.15.46:0/4173412473 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b20149dfe0 0x55b2014a9e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:26:32.389 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:32.387+0000 7f8b26b94740 1 --2- 172.21.15.46:0/4173412473 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55b20149d6c0 0x55b2014a9930 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:26:32.390 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:32.387+0000 7f8b26b94740 1 -- 172.21.15.46:0/4173412473 >> 172.21.15.46:0/4173412473 conn(0x55b201497fe0 msgr2=0x55b201499cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:26:32.390 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:32.387+0000 7f8b26b94740 1 -- 172.21.15.46:0/4173412473 shutdown_connections 2024-03-31T23:26:32.390 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:32.387+0000 7f8b26b94740 1 -- 172.21.15.46:0/4173412473 wait complete. 2024-03-31T23:26:32.390 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:32.387+0000 7f8b26b94740 20 client.27188 trim_cache size 0 max 16384 2024-03-31T23:26:32.391 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout:{ 2024-03-31T23:26:32.391 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout: "message": "", 2024-03-31T23:26:32.391 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout: "return_code": 0 2024-03-31T23:26:32.391 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout:} 2024-03-31T23:26:32.392 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:32.388+0000 7f8b23fb6640 1 -- 172.21.15.46:0/368812698 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f8b040a2730 msgr2=0x7f8b040a4bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:26:32.392 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:32.388+0000 7f8b23fb6640 1 --2- 172.21.15.46:0/368812698 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f8b040a2730 0x7f8b040a4bf0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f8b0c009a60 tx=0x7f8b0c04b000 comp rx=0 tx=0).stop 2024-03-31T23:26:32.392 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:32.388+0000 7f8b23fb6640 1 -- 172.21.15.46:0/368812698 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8b1c100630 msgr2=0x7f8b1c197e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:26:32.392 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:32.388+0000 7f8b23fb6640 1 --2- 172.21.15.46:0/368812698 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8b1c100630 0x7f8b1c197e70 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7f8b1400ea60 tx=0x7f8b1400ef20 comp rx=0 tx=0).stop 2024-03-31T23:26:32.392 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:32.388+0000 7f8b23fb6640 1 -- 172.21.15.46:0/368812698 shutdown_connections 2024-03-31T23:26:32.392 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:32.388+0000 7f8b23fb6640 1 --2- 172.21.15.46:0/368812698 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f8b040a2730 0x7f8b040a4bf0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:26:32.392 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:32.388+0000 7f8b23fb6640 1 --2- 172.21.15.46:0/368812698 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8b1c106640 0x7f8b1c19c6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:26:32.392 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:32.388+0000 7f8b23fb6640 1 --2- 172.21.15.46:0/368812698 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8b1c100ff0 0x7f8b1c1983b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:26:32.392 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:32.388+0000 7f8b23fb6640 1 --2- 172.21.15.46:0/368812698 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8b1c100630 0x7f8b1c197e70 unknown :-1 s=CLOSED pgs=892 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:26:32.392 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:32.388+0000 7f8b23fb6640 1 -- 172.21.15.46:0/368812698 >> 172.21.15.46:0/368812698 conn(0x7f8b1c0777b0 msgr2=0x7f8b1c069bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:26:32.392 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:32.389+0000 7f8b23fb6640 1 -- 172.21.15.46:0/368812698 shutdown_connections 2024-03-31T23:26:32.393 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:32.389+0000 7f8b23fb6640 1 -- 172.21.15.46:0/368812698 wait complete. 2024-03-31T23:26:33.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:26:32 smithi162 ceph-mon[36237]: pgmap v466: 129 pgs: 129 active+clean; 474 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 3.7 MiB/s wr, 314 op/s 2024-03-31T23:26:33.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:26:32 smithi046 ceph-mon[32498]: pgmap v466: 129 pgs: 129 active+clean; 474 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 3.7 MiB/s wr, 314 op/s 2024-03-31T23:26:33.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:26:32 smithi142 ceph-mon[35271]: pgmap v466: 129 pgs: 129 active+clean; 474 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 3.7 MiB/s wr, 314 op/s 2024-03-31T23:26:33.269 INFO:tasks.workunit.client.0.smithi046.stdout: 10 15914 10.47 MB/sec warmup 98 sec latency 152.896 ms 2024-03-31T23:26:34.269 INFO:tasks.workunit.client.0.smithi046.stdout: 10 16112 10.53 MB/sec warmup 99 sec latency 96.716 ms 2024-03-31T23:26:35.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:26:34 smithi162 ceph-mon[36237]: pgmap v467: 129 pgs: 129 active+clean; 474 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 3.7 MiB/s wr, 314 op/s 2024-03-31T23:26:35.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:26:34 smithi046 ceph-mon[32498]: pgmap v467: 129 pgs: 129 active+clean; 474 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 3.7 MiB/s wr, 314 op/s 2024-03-31T23:26:35.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:26:34 smithi142 ceph-mon[35271]: pgmap v467: 129 pgs: 129 active+clean; 474 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 3.7 MiB/s wr, 314 op/s 2024-03-31T23:26:35.269 INFO:tasks.workunit.client.0.smithi046.stdout: 10 16305 10.53 MB/sec warmup 100 sec latency 71.029 ms 2024-03-31T23:26:36.269 INFO:tasks.workunit.client.0.smithi046.stdout: 10 16460 10.56 MB/sec warmup 101 sec latency 64.382 ms 2024-03-31T23:26:37.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:26:36 smithi162 ceph-mon[36237]: pgmap v468: 129 pgs: 129 active+clean; 483 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 6.1 MiB/s wr, 477 op/s 2024-03-31T23:26:37.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:26:36 smithi046 ceph-mon[32498]: pgmap v468: 129 pgs: 129 active+clean; 483 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 6.1 MiB/s wr, 477 op/s 2024-03-31T23:26:37.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:26:36 smithi142 ceph-mon[35271]: pgmap v468: 129 pgs: 129 active+clean; 483 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 6.1 MiB/s wr, 477 op/s 2024-03-31T23:26:37.269 INFO:tasks.workunit.client.0.smithi046.stdout: 10 16586 10.48 MB/sec warmup 102 sec latency 68.539 ms 2024-03-31T23:26:38.269 INFO:tasks.workunit.client.0.smithi046.stdout: 10 16781 10.56 MB/sec warmup 103 sec latency 74.160 ms 2024-03-31T23:26:39.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:26:38 smithi162 ceph-mon[36237]: pgmap v469: 129 pgs: 129 active+clean; 483 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 3.9 MiB/s wr, 281 op/s 2024-03-31T23:26:39.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:26:38 smithi046 ceph-mon[32498]: pgmap v469: 129 pgs: 129 active+clean; 483 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 3.9 MiB/s wr, 281 op/s 2024-03-31T23:26:39.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:26:38 smithi142 ceph-mon[35271]: pgmap v469: 129 pgs: 129 active+clean; 483 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 3.9 MiB/s wr, 281 op/s 2024-03-31T23:26:39.269 INFO:tasks.workunit.client.0.smithi046.stdout: 10 16926 10.50 MB/sec warmup 104 sec latency 62.008 ms 2024-03-31T23:26:40.269 INFO:tasks.workunit.client.0.smithi046.stdout: 10 17070 10.42 MB/sec warmup 105 sec latency 66.867 ms 2024-03-31T23:26:41.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:26:40 smithi162 ceph-mon[36237]: pgmap v470: 129 pgs: 129 active+clean; 489 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 6.2 MiB/s wr, 398 op/s 2024-03-31T23:26:41.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:26:40 smithi046 ceph-mon[32498]: pgmap v470: 129 pgs: 129 active+clean; 489 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 6.2 MiB/s wr, 398 op/s 2024-03-31T23:26:41.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:26:40 smithi142 ceph-mon[35271]: pgmap v470: 129 pgs: 129 active+clean; 489 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 6.2 MiB/s wr, 398 op/s 2024-03-31T23:26:41.269 INFO:tasks.workunit.client.0.smithi046.stdout: 10 17226 10.38 MB/sec warmup 106 sec latency 83.068 ms 2024-03-31T23:26:42.270 INFO:tasks.workunit.client.0.smithi046.stdout: 10 17381 10.32 MB/sec warmup 107 sec latency 74.605 ms 2024-03-31T23:26:43.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:26:42 smithi162 ceph-mon[36237]: pgmap v471: 129 pgs: 129 active+clean; 489 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 4.7 MiB/s wr, 279 op/s 2024-03-31T23:26:43.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:26:42 smithi046 ceph-mon[32498]: pgmap v471: 129 pgs: 129 active+clean; 489 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 4.7 MiB/s wr, 279 op/s 2024-03-31T23:26:43.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:26:42 smithi142 ceph-mon[35271]: pgmap v471: 129 pgs: 129 active+clean; 489 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 4.7 MiB/s wr, 279 op/s 2024-03-31T23:26:43.269 INFO:tasks.workunit.client.0.smithi046.stdout: 10 17550 10.29 MB/sec warmup 108 sec latency 84.902 ms 2024-03-31T23:26:44.270 INFO:tasks.workunit.client.0.smithi046.stdout: 10 17714 10.29 MB/sec warmup 109 sec latency 86.733 ms 2024-03-31T23:26:45.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:26:44 smithi162 ceph-mon[36237]: pgmap v472: 129 pgs: 129 active+clean; 489 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 4.7 MiB/s wr, 279 op/s 2024-03-31T23:26:45.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:26:44 smithi046 ceph-mon[32498]: pgmap v472: 129 pgs: 129 active+clean; 489 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 4.7 MiB/s wr, 279 op/s 2024-03-31T23:26:45.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:26:44 smithi142 ceph-mon[35271]: pgmap v472: 129 pgs: 129 active+clean; 489 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 4.7 MiB/s wr, 279 op/s 2024-03-31T23:26:45.270 INFO:tasks.workunit.client.0.smithi046.stdout: 10 17884 10.29 MB/sec warmup 110 sec latency 56.446 ms 2024-03-31T23:26:45.519 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:45.518+0000 7efcd3aae640 1 -- 172.21.15.46:0/1244560409 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efccc110300 msgr2=0x7efccc10dff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:26:45.519 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:45.518+0000 7efcd3aae640 1 --2- 172.21.15.46:0/1244560409 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efccc110300 0x7efccc10dff0 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7efcc8009860 tx=0x7efcc802bb70 comp rx=0 tx=0).stop 2024-03-31T23:26:45.519 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:45.518+0000 7efcd3aae640 1 -- 172.21.15.46:0/1244560409 shutdown_connections 2024-03-31T23:26:45.520 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:45.518+0000 7efcd3aae640 1 --2- 172.21.15.46:0/1244560409 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7efccc11af30 0x7efccc11d320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:26:45.520 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:45.518+0000 7efcd3aae640 1 --2- 172.21.15.46:0/1244560409 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efccc110300 0x7efccc10dff0 unknown :-1 s=CLOSED pgs=894 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:26:45.520 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:45.518+0000 7efcd3aae640 1 --2- 172.21.15.46:0/1244560409 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7efccc10fe90 0x7efccc10dab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:26:45.520 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:45.518+0000 7efcd3aae640 1 -- 172.21.15.46:0/1244560409 >> 172.21.15.46:0/1244560409 conn(0x7efccc06c7f0 msgr2=0x7efccc06cc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:26:45.520 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:45.518+0000 7efcd3aae640 1 -- 172.21.15.46:0/1244560409 shutdown_connections 2024-03-31T23:26:45.520 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:45.518+0000 7efcd3aae640 1 -- 172.21.15.46:0/1244560409 wait complete. 2024-03-31T23:26:45.522 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:45.519+0000 7efcd3aae640 1 Processor -- start 2024-03-31T23:26:45.522 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:45.519+0000 7efcd3aae640 1 -- start start 2024-03-31T23:26:45.522 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:45.519+0000 7efcd3aae640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7efccc10fe90 0x7efccc1a6f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:26:45.522 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:45.519+0000 7efcd3aae640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efccc11af30 0x7efccc1a7440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:26:45.522 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:45.519+0000 7efcd3aae640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7efccc1ab7c0 0x7efccc1abc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:26:45.522 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:45.519+0000 7efcd3aae640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7efccc120790 con 0x7efccc11af30 2024-03-31T23:26:45.522 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:45.519+0000 7efcd3aae640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7efccc120610 con 0x7efccc1ab7c0 2024-03-31T23:26:45.522 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:45.519+0000 7efcd3aae640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7efccc120910 con 0x7efccc10fe90 2024-03-31T23:26:45.523 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:45.519+0000 7efcd1049640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efccc11af30 0x7efccc1a7440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:26:45.523 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:45.519+0000 7efcd1049640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efccc11af30 0x7efccc1a7440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:47800/0 (socket says 172.21.15.46:47800) 2024-03-31T23:26:45.523 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:45.519+0000 7efcd1049640 1 -- 172.21.15.46:0/2791718864 learned_addr learned my addr 172.21.15.46:0/2791718864 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:26:45.523 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:45.519+0000 7efcd184a640 1 --2- 172.21.15.46:0/2791718864 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7efccc10fe90 0x7efccc1a6f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:26:45.523 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:45.519+0000 7efcd204b640 1 --2- 172.21.15.46:0/2791718864 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7efccc1ab7c0 0x7efccc1abc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:26:45.523 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:45.519+0000 7efcd1049640 1 -- 172.21.15.46:0/2791718864 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7efccc10fe90 msgr2=0x7efccc1a6f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:26:45.523 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:45.519+0000 7efcd1049640 1 --2- 172.21.15.46:0/2791718864 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7efccc10fe90 0x7efccc1a6f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:26:45.523 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:45.519+0000 7efcd1049640 1 -- 172.21.15.46:0/2791718864 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7efccc1ab7c0 msgr2=0x7efccc1abc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:26:45.523 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:45.519+0000 7efcd1049640 1 --2- 172.21.15.46:0/2791718864 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7efccc1ab7c0 0x7efccc1abc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:26:45.523 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:45.519+0000 7efcd1049640 1 -- 172.21.15.46:0/2791718864 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efccc1ac2b0 con 0x7efccc11af30 2024-03-31T23:26:45.523 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:45.520+0000 7efcd1049640 1 --2- 172.21.15.46:0/2791718864 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efccc11af30 0x7efccc1a7440 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7efcc8037040 tx=0x7efcc8035a10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:26:45.523 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:45.520+0000 7efcc2ffd640 1 -- 172.21.15.46:0/2791718864 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcc8041020 con 0x7efccc11af30 2024-03-31T23:26:45.523 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:45.520+0000 7efcc2ffd640 1 -- 172.21.15.46:0/2791718864 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7efcc8035e00 con 0x7efccc11af30 2024-03-31T23:26:45.523 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:45.520+0000 7efcd3aae640 1 -- 172.21.15.46:0/2791718864 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efccc1ac540 con 0x7efccc11af30 2024-03-31T23:26:45.524 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:45.520+0000 7efcd3aae640 1 -- 172.21.15.46:0/2791718864 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efccc1b3d00 con 0x7efccc11af30 2024-03-31T23:26:45.524 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:45.520+0000 7efcc2ffd640 1 -- 172.21.15.46:0/2791718864 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcc8039040 con 0x7efccc11af30 2024-03-31T23:26:45.524 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:45.523+0000 7efcc2ffd640 1 -- 172.21.15.46:0/2791718864 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 25) ==== 137451+0+0 (secure 0 0 0) 0x7efcc8044020 con 0x7efccc11af30 2024-03-31T23:26:45.526 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:45.525+0000 7efcc2ffd640 1 --2- 172.21.15.46:0/2791718864 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7efcb80a2730 0x7efcb80a4bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:26:45.528 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:45.525+0000 7efcc2ffd640 1 -- 172.21.15.46:0/2791718864 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(115..115 src has 1..115) ==== 9554+0+0 (secure 0 0 0) 0x7efcc80f0430 con 0x7efccc11af30 2024-03-31T23:26:45.528 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:45.525+0000 7efcd184a640 1 --2- 172.21.15.46:0/2791718864 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7efcb80a2730 0x7efcb80a4bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:26:45.528 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:45.525+0000 7efcd676f740 1 Processor -- start 2024-03-31T23:26:45.528 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:45.526+0000 7efcd676f740 1 -- start start 2024-03-31T23:26:45.528 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:45.526+0000 7efcd184a640 1 --2- 172.21.15.46:0/2791718864 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7efcb80a2730 0x7efcb80a4bf0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7efcbc005d00 tx=0x7efcbc00b410 comp rx=0 tx=0).ready entity=mgr.15654 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:26:45.528 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:45.526+0000 7efcd676f740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55c0a797e6c0 0x55c0a797eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:26:45.528 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:45.526+0000 7efcd676f740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c0a797efe0 0x55c0a7987870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:26:45.528 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:45.526+0000 7efcd676f740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c0a7987db0 0x55c0a798a1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:26:45.528 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:45.526+0000 7efcd676f740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55c0a78e18f0 con 0x55c0a797efe0 2024-03-31T23:26:45.529 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:45.526+0000 7efcd676f740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55c0a788a8d0 con 0x55c0a797e6c0 2024-03-31T23:26:45.529 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:45.526+0000 7efcd676f740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55c0a78dd990 con 0x55c0a7987db0 2024-03-31T23:26:45.529 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:45.526+0000 7efcd184a640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55c0a797e6c0 0x55c0a797eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:26:45.529 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:45.526+0000 7efcd184a640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55c0a797e6c0 0x55c0a797eaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.46:46328/0 (socket says 172.21.15.46:46328) 2024-03-31T23:26:45.529 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:45.526+0000 7efcd204b640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c0a7987db0 0x55c0a798a1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:26:45.529 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:45.526+0000 7efcd184a640 1 -- 172.21.15.46:0/952473017 learned_addr learned my addr 172.21.15.46:0/952473017 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:26:45.529 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:45.526+0000 7efcd1049640 1 --2- 172.21.15.46:0/952473017 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c0a797efe0 0x55c0a7987870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:26:45.529 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:45.526+0000 7efcd204b640 1 -- 172.21.15.46:0/952473017 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55c0a797e6c0 msgr2=0x55c0a797eaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:26:45.529 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:45.526+0000 7efcd204b640 1 --2- 172.21.15.46:0/952473017 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55c0a797e6c0 0x55c0a797eaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:26:45.529 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:45.526+0000 7efcd204b640 1 -- 172.21.15.46:0/952473017 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c0a797efe0 msgr2=0x55c0a7987870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:26:45.529 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:45.526+0000 7efcd204b640 1 --2- 172.21.15.46:0/952473017 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c0a797efe0 0x55c0a7987870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:26:45.529 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:45.526+0000 7efcd204b640 1 -- 172.21.15.46:0/952473017 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c0a75f6210 con 0x55c0a7987db0 2024-03-31T23:26:45.529 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:45.527+0000 7efcd184a640 1 --2- 172.21.15.46:0/952473017 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55c0a797e6c0 0x55c0a797eaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:26:45.529 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:45.527+0000 7efcd1049640 1 --2- 172.21.15.46:0/952473017 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c0a797efe0 0x55c0a7987870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:26:45.530 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:45.527+0000 7efcd204b640 1 --2- 172.21.15.46:0/952473017 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c0a7987db0 0x55c0a798a1a0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7efcc4005fd0 tx=0x7efcc40095c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b084daca825fa04a server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:26:45.530 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:45.527+0000 7efcb7fff640 1 -- 172.21.15.46:0/952473017 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcc4004b20 con 0x55c0a7987db0 2024-03-31T23:26:45.530 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:45.527+0000 7efcb7fff640 1 -- 172.21.15.46:0/952473017 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7efcc4004cc0 con 0x55c0a7987db0 2024-03-31T23:26:45.530 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:45.527+0000 7efcb7fff640 1 -- 172.21.15.46:0/952473017 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcc4005480 con 0x55c0a7987db0 2024-03-31T23:26:45.533 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:45.528+0000 7efcd676f740 1 -- 172.21.15.46:0/952473017 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c0a7987db0 msgr2=0x55c0a798a1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:26:45.533 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:45.528+0000 7efcd676f740 1 --2- 172.21.15.46:0/952473017 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c0a7987db0 0x55c0a798a1a0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7efcc4005fd0 tx=0x7efcc40095c0 comp rx=0 tx=0).stop 2024-03-31T23:26:45.533 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:45.528+0000 7efcd676f740 1 -- 172.21.15.46:0/952473017 shutdown_connections 2024-03-31T23:26:45.533 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:45.528+0000 7efcd676f740 1 --2- 172.21.15.46:0/952473017 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c0a7987db0 0x55c0a798a1a0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:26:45.533 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:45.528+0000 7efcd676f740 1 --2- 172.21.15.46:0/952473017 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c0a797efe0 0x55c0a7987870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:26:45.533 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:45.528+0000 7efcd676f740 1 --2- 172.21.15.46:0/952473017 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55c0a797e6c0 0x55c0a797eaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:26:45.533 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:45.528+0000 7efcd676f740 1 -- 172.21.15.46:0/952473017 >> 172.21.15.46:0/952473017 conn(0x55c0a7978fe0 msgr2=0x55c0a79793c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:26:45.534 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:45.528+0000 7efcd676f740 1 -- 172.21.15.46:0/952473017 shutdown_connections 2024-03-31T23:26:45.534 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:45.528+0000 7efcd676f740 1 -- 172.21.15.46:0/952473017 wait complete. 2024-03-31T23:26:45.534 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:45.529+0000 7efcd676f740 1 -- start start 2024-03-31T23:26:45.534 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:45.529+0000 7efcd676f740 1 Processor -- start 2024-03-31T23:26:45.534 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:45.529+0000 7efcd676f740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55c0a797e6c0 0x55c0a7984d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:26:45.534 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:45.529+0000 7efcd676f740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c0a797efe0 0x55c0a7985290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:26:45.534 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:45.529+0000 7efcd1049640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c0a797efe0 0x55c0a7985290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:26:45.534 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:45.529+0000 7efcd1049640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c0a797efe0 0x55c0a7985290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:47828/0 (socket says 172.21.15.46:47828) 2024-03-31T23:26:45.534 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:45.529+0000 7efcd676f740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c0a7987db0 0x55c0a79857d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:26:45.534 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:45.529+0000 7efcd676f740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55c0a798a9e0 con 0x55c0a797efe0 2024-03-31T23:26:45.534 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:45.529+0000 7efcd676f740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55c0a787ff80 con 0x55c0a797e6c0 2024-03-31T23:26:45.534 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:45.529+0000 7efcd676f740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55c0a798ab60 con 0x55c0a7987db0 2024-03-31T23:26:45.534 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:45.529+0000 7efcd184a640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55c0a797e6c0 0x55c0a7984d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:26:45.535 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:45.529+0000 7efcd184a640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55c0a797e6c0 0x55c0a7984d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.46:46334/0 (socket says 172.21.15.46:46334) 2024-03-31T23:26:45.535 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:45.529+0000 7efcd184a640 1 -- 172.21.15.46:0/2821548383 learned_addr learned my addr 172.21.15.46:0/2821548383 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:26:45.535 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:45.529+0000 7efcd676f740 10 client.0 _ll_register_callbacks cb 0x55c0a78d8290 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T23:26:45.535 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:45.530+0000 7efcd204b640 1 --2- 172.21.15.46:0/2821548383 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c0a7987db0 0x55c0a79857d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:26:45.535 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:45.530+0000 7efcd184a640 1 -- 172.21.15.46:0/2821548383 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c0a7987db0 msgr2=0x55c0a79857d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:26:45.535 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:45.530+0000 7efcd184a640 1 --2- 172.21.15.46:0/2821548383 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c0a7987db0 0x55c0a79857d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:26:45.535 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:45.530+0000 7efcd184a640 1 -- 172.21.15.46:0/2821548383 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c0a797efe0 msgr2=0x55c0a7985290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:26:45.535 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:45.530+0000 7efcd184a640 1 --2- 172.21.15.46:0/2821548383 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c0a797efe0 0x55c0a7985290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:26:45.535 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:45.530+0000 7efcd184a640 1 -- 172.21.15.46:0/2821548383 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c0a76c2620 con 0x55c0a797e6c0 2024-03-31T23:26:45.535 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:45.530+0000 7efcd204b640 1 --2- 172.21.15.46:0/2821548383 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c0a7987db0 0x55c0a79857d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T23:26:45.535 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:45.530+0000 7efcd1049640 1 --2- 172.21.15.46:0/2821548383 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c0a797efe0 0x55c0a7985290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:26:45.535 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:45.530+0000 7efcd184a640 1 --2- 172.21.15.46:0/2821548383 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55c0a797e6c0 0x55c0a7984d50 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7efcbc00be40 tx=0x7efcbc004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d0e8abdef338323d server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:26:45.535 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:45.530+0000 7efcb57fa640 10 client.0 ms_handle_connect on v2:172.21.15.142:3300/0 2024-03-31T23:26:45.535 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:45.530+0000 7efcb57fa640 1 -- 172.21.15.46:0/2821548383 <== mon.2 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcbc01fb70 con 0x55c0a797e6c0 2024-03-31T23:26:45.535 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:45.530+0000 7efcb57fa640 1 -- 172.21.15.46:0/2821548383 <== mon.2 v2:172.21.15.142:3300/0 2 ==== osd_map(115..115 src has 1..115) ==== 9554+0+0 (secure 0 0 0) 0x7efcbc01fd10 con 0x55c0a797e6c0 2024-03-31T23:26:45.536 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:45.530+0000 7efcd676f740 1 -- 172.21.15.46:0/2821548383 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efcc4004cc0 con 0x55c0a797e6c0 2024-03-31T23:26:45.536 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:45.531+0000 7efcb57fa640 1 -- 172.21.15.46:0/2821548383 <== mon.2 v2:172.21.15.142:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7efcbc028440 con 0x55c0a797e6c0 2024-03-31T23:26:45.536 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:45.531+0000 7efcd676f740 10 client.27200 fetch_fsmap learned FSMap version 40 2024-03-31T23:26:45.536 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:45.531+0000 7efcd676f740 1 -- 172.21.15.46:0/2821548383 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x55c0a79968c0 con 0x55c0a797e6c0 2024-03-31T23:26:45.536 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:45.531+0000 7efcb57fa640 1 -- 172.21.15.46:0/2821548383 <== mon.2 v2:172.21.15.142:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7efcbc028a90 con 0x55c0a797e6c0 2024-03-31T23:26:45.536 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:45.531+0000 7efcb57fa640 1 -- 172.21.15.46:0/2821548383 <== mon.2 v2:172.21.15.142:3300/0 5 ==== fsmap(e 40) ==== 8370+0+0 (secure 0 0 0) 0x7efcbc026070 con 0x55c0a797e6c0 2024-03-31T23:26:45.536 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:45.531+0000 7efcd676f740 10 client.27200 fetch_fsmap finished waiting for FSMap version 40 2024-03-31T23:26:45.536 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:45.531+0000 7efcd676f740 10 client.27200 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-31T23:26:45.536 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:45.531+0000 7efcd676f740 20 client.27200 populate_metadata read hostname 'smithi046' 2024-03-31T23:26:45.536 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:45.531+0000 7efcd676f740 1 --2- 172.21.15.46:0/2821548383 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55c0a7997ec0 0x55c0a799a360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:26:45.536 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:45.531+0000 7efcd676f740 4 client.27200 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-31T23:26:45.536 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:45.531+0000 7efcd676f740 1 -- 172.21.15.46:0/2821548383 --> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7efcc4004cc0 con 0x55c0a7997ec0 2024-03-31T23:26:45.536 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:45.531+0000 7efcd1049640 1 --2- 172.21.15.46:0/2821548383 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55c0a7997ec0 0x55c0a799a360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:26:45.536 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:45.532+0000 7efcb57fa640 10 client.27200 ms_handle_connect on v2:172.21.15.142:6838/2519143918 2024-03-31T23:26:45.537 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:45.532+0000 7efcd1049640 1 --2- 172.21.15.46:0/2821548383 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55c0a7997ec0 0x55c0a799a360 crc :-1 s=READY pgs=210 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6a36fddbd7ceaaca server_cookie=9bdd698bdb0591e4 in_seq=0 out_seq=0 2024-03-31T23:26:45.537 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:45.532+0000 7efcb57fa640 1 -- 172.21.15.46:0/2821548383 <== mds.0 v2:172.21.15.142:6838/2519143918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x7efcc4004cc0 con 0x55c0a7997ec0 2024-03-31T23:26:45.538 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:45.532+0000 7efcb57fa640 10 client.27200 handle_command_reply: tid=0 2024-03-31T23:26:45.538 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:45.533+0000 7efcd676f740 1 client.27200 shutdown 2024-03-31T23:26:45.538 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:45.533+0000 7efcd676f740 1 -- 172.21.15.46:0/2821548383 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55c0a797e6c0 msgr2=0x55c0a7984d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:26:45.538 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:45.533+0000 7efcd676f740 1 --2- 172.21.15.46:0/2821548383 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55c0a797e6c0 0x55c0a7984d50 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7efcbc00be40 tx=0x7efcbc004730 comp rx=0 tx=0).stop 2024-03-31T23:26:45.538 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:45.533+0000 7efcd676f740 1 -- 172.21.15.46:0/2821548383 shutdown_connections 2024-03-31T23:26:45.538 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:45.533+0000 7efcd676f740 1 --2- 172.21.15.46:0/2821548383 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55c0a7997ec0 0x55c0a799a360 crc :-1 s=READY pgs=210 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:26:45.538 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:45.533+0000 7efcd676f740 1 --2- 172.21.15.46:0/2821548383 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c0a7987db0 0x55c0a79857d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:26:45.538 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:45.533+0000 7efcb57fa640 0 client.27200 ms_handle_reset on v2:172.21.15.142:6838/2519143918 2024-03-31T23:26:45.538 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:45.533+0000 7efcd676f740 1 --2- 172.21.15.46:0/2821548383 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c0a797efe0 0x55c0a7985290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:26:45.538 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:45.533+0000 7efcd676f740 1 --2- 172.21.15.46:0/2821548383 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55c0a797e6c0 0x55c0a7984d50 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:26:45.538 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:45.533+0000 7efcd676f740 1 -- 172.21.15.46:0/2821548383 >> 172.21.15.46:0/2821548383 conn(0x55c0a7978fe0 msgr2=0x55c0a797ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:26:45.538 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:45.533+0000 7efcd676f740 1 -- 172.21.15.46:0/2821548383 shutdown_connections 2024-03-31T23:26:45.538 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:45.533+0000 7efcd676f740 1 -- 172.21.15.46:0/2821548383 wait complete. 2024-03-31T23:26:45.539 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:45.533+0000 7efcd676f740 20 client.27200 trim_cache size 0 max 16384 2024-03-31T23:26:45.562 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:45.560+0000 7efcd676f740 1 Processor -- start 2024-03-31T23:26:45.562 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:45.560+0000 7efcd676f740 1 -- start start 2024-03-31T23:26:45.562 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:45.560+0000 7efcd676f740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c0a797e6c0 0x55c0a7987480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:26:45.562 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:45.560+0000 7efcd676f740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c0a797efe0 0x55c0a7973e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:26:45.562 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:45.560+0000 7efcd676f740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55c0a7987db0 0x55c0a79937b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:26:45.562 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:45.560+0000 7efcd676f740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7efcb0012a70 con 0x55c0a797efe0 2024-03-31T23:26:45.562 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:45.560+0000 7efcd676f740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7efcb0010220 con 0x55c0a7987db0 2024-03-31T23:26:45.563 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:45.560+0000 7efcd676f740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7efcb000e090 con 0x55c0a797e6c0 2024-03-31T23:26:45.563 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:45.560+0000 7efcd184a640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c0a797e6c0 0x55c0a7987480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:26:45.563 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:45.560+0000 7efcd1049640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c0a797efe0 0x55c0a7973e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:26:45.563 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:45.560+0000 7efcd184a640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c0a797e6c0 0x55c0a7987480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.46:39154/0 (socket says 172.21.15.46:39154) 2024-03-31T23:26:45.563 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:45.560+0000 7efcd184a640 1 -- 172.21.15.46:0/3728789163 learned_addr learned my addr 172.21.15.46:0/3728789163 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:26:45.563 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:45.560+0000 7efcd204b640 1 --2- 172.21.15.46:0/3728789163 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55c0a7987db0 0x55c0a79937b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:26:45.563 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:45.560+0000 7efcd184a640 1 -- 172.21.15.46:0/3728789163 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55c0a7987db0 msgr2=0x55c0a79937b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:26:45.563 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:45.560+0000 7efcd184a640 1 --2- 172.21.15.46:0/3728789163 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55c0a7987db0 0x55c0a79937b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:26:45.563 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:45.560+0000 7efcd184a640 1 -- 172.21.15.46:0/3728789163 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c0a797efe0 msgr2=0x55c0a7973e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:26:45.563 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:45.560+0000 7efcd184a640 1 --2- 172.21.15.46:0/3728789163 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c0a797efe0 0x55c0a7973e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:26:45.563 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:45.560+0000 7efcd184a640 1 -- 172.21.15.46:0/3728789163 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c0a797de70 con 0x55c0a797e6c0 2024-03-31T23:26:45.563 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:45.560+0000 7efcd1049640 1 --2- 172.21.15.46:0/3728789163 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c0a797efe0 0x55c0a7973e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:26:45.564 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:45.561+0000 7efcd184a640 1 --2- 172.21.15.46:0/3728789163 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c0a797e6c0 0x55c0a7987480 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7efcbc00ba20 tx=0x7efcbc0114a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7d3c87b12cf0d684 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:26:45.564 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:45.561+0000 7efcb77fe640 1 -- 172.21.15.46:0/3728789163 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcbc027d60 con 0x55c0a797e6c0 2024-03-31T23:26:45.564 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:45.561+0000 7efcb77fe640 1 -- 172.21.15.46:0/3728789163 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7efcbc03b070 con 0x55c0a797e6c0 2024-03-31T23:26:45.564 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:45.561+0000 7efcb77fe640 1 -- 172.21.15.46:0/3728789163 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcbc028840 con 0x55c0a797e6c0 2024-03-31T23:26:45.567 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:45.562+0000 7efcd676f740 1 -- 172.21.15.46:0/3728789163 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c0a797e6c0 msgr2=0x55c0a7987480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:26:45.567 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:45.562+0000 7efcd676f740 1 --2- 172.21.15.46:0/3728789163 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c0a797e6c0 0x55c0a7987480 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7efcbc00ba20 tx=0x7efcbc0114a0 comp rx=0 tx=0).stop 2024-03-31T23:26:45.567 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:45.562+0000 7efcd676f740 1 -- 172.21.15.46:0/3728789163 shutdown_connections 2024-03-31T23:26:45.567 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:45.562+0000 7efcd676f740 1 --2- 172.21.15.46:0/3728789163 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55c0a7987db0 0x55c0a79937b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:26:45.567 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:45.562+0000 7efcd676f740 1 --2- 172.21.15.46:0/3728789163 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c0a797efe0 0x55c0a7973e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:26:45.567 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:45.562+0000 7efcd676f740 1 --2- 172.21.15.46:0/3728789163 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c0a797e6c0 0x55c0a7987480 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:26:45.568 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:45.562+0000 7efcd676f740 1 -- 172.21.15.46:0/3728789163 >> 172.21.15.46:0/3728789163 conn(0x55c0a7978fe0 msgr2=0x55c0a79793c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:26:45.568 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:45.562+0000 7efcd676f740 1 -- 172.21.15.46:0/3728789163 shutdown_connections 2024-03-31T23:26:45.568 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:45.562+0000 7efcd676f740 1 -- 172.21.15.46:0/3728789163 wait complete. 2024-03-31T23:26:45.568 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:45.562+0000 7efcd676f740 1 -- start start 2024-03-31T23:26:45.568 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:45.563+0000 7efcd676f740 1 Processor -- start 2024-03-31T23:26:45.568 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:45.563+0000 7efcd676f740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55c0a797e6c0 0x55c0a798b840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:26:45.568 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:45.563+0000 7efcd676f740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c0a797efe0 0x55c0a798bd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:26:45.568 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:45.564+0000 7efcd184a640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55c0a797e6c0 0x55c0a798b840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:26:45.568 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:45.564+0000 7efcd184a640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55c0a797e6c0 0x55c0a798b840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.46:46342/0 (socket says 172.21.15.46:46342) 2024-03-31T23:26:45.568 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:45.564+0000 7efcd184a640 1 -- 172.21.15.46:0/3423743330 learned_addr learned my addr 172.21.15.46:0/3423743330 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:26:45.568 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:45.564+0000 7efcd676f740 1 --2- 172.21.15.46:0/3423743330 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c0a7987db0 0x55c0a798c2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:26:45.568 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:45.564+0000 7efcd676f740 1 -- 172.21.15.46:0/3423743330 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7efcb000e6e0 con 0x55c0a797efe0 2024-03-31T23:26:45.568 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:45.564+0000 7efcd1049640 1 --2- 172.21.15.46:0/3423743330 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c0a797efe0 0x55c0a798bd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:26:45.569 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:45.564+0000 7efcd676f740 1 -- 172.21.15.46:0/3423743330 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7efcb000fb50 con 0x55c0a797e6c0 2024-03-31T23:26:45.569 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:45.564+0000 7efcd204b640 1 --2- 172.21.15.46:0/3423743330 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c0a7987db0 0x55c0a798c2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:26:45.569 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:45.564+0000 7efcd676f740 1 -- 172.21.15.46:0/3423743330 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7efcb000edb0 con 0x55c0a7987db0 2024-03-31T23:26:45.569 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:45.564+0000 7efcd676f740 10 client.0 _ll_register_callbacks cb 0x7efcbc005d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T23:26:45.569 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:45.565+0000 7efcd184a640 1 -- 172.21.15.46:0/3423743330 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c0a7987db0 msgr2=0x55c0a798c2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:26:45.569 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:45.565+0000 7efcd184a640 1 --2- 172.21.15.46:0/3423743330 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c0a7987db0 0x55c0a798c2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:26:45.569 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:45.565+0000 7efcd184a640 1 -- 172.21.15.46:0/3423743330 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c0a797efe0 msgr2=0x55c0a798bd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:26:45.569 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:45.565+0000 7efcd184a640 1 --2- 172.21.15.46:0/3423743330 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c0a797efe0 0x55c0a798bd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:26:45.569 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:45.565+0000 7efcd184a640 1 -- 172.21.15.46:0/3423743330 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c0a79fd3c0 con 0x55c0a797e6c0 2024-03-31T23:26:45.569 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:45.565+0000 7efcd184a640 1 --2- 172.21.15.46:0/3423743330 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55c0a797e6c0 0x55c0a798b840 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7efcbc027970 tx=0x7efcbc01f370 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dc7d3f2dd38c3ff4 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:26:45.569 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:45.565+0000 7efcb57fa640 10 client.0 ms_handle_connect on v2:172.21.15.142:3300/0 2024-03-31T23:26:45.569 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:45.565+0000 7efcb57fa640 1 -- 172.21.15.46:0/3423743330 <== mon.2 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcbc01fd80 con 0x55c0a797e6c0 2024-03-31T23:26:45.569 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:45.565+0000 7efcb57fa640 1 -- 172.21.15.46:0/3423743330 <== mon.2 v2:172.21.15.142:3300/0 2 ==== osd_map(115..115 src has 1..115) ==== 9554+0+0 (secure 0 0 0) 0x7efcbc0285e0 con 0x55c0a797e6c0 2024-03-31T23:26:45.569 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:45.566+0000 7efcd676f740 1 -- 172.21.15.46:0/3423743330 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efcbc03b070 con 0x55c0a797e6c0 2024-03-31T23:26:45.569 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:45.566+0000 7efcb57fa640 1 -- 172.21.15.46:0/3423743330 <== mon.2 v2:172.21.15.142:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7efcbc005850 con 0x55c0a797e6c0 2024-03-31T23:26:45.570 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:45.566+0000 7efcd676f740 10 client.27206 fetch_fsmap learned FSMap version 40 2024-03-31T23:26:45.570 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:45.566+0000 7efcd676f740 1 -- 172.21.15.46:0/3423743330 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x55c0a79fe6b0 con 0x55c0a797e6c0 2024-03-31T23:26:45.570 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:45.566+0000 7efcb57fa640 1 -- 172.21.15.46:0/3423743330 <== mon.2 v2:172.21.15.142:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7efcbc03b070 con 0x55c0a797e6c0 2024-03-31T23:26:45.570 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:45.567+0000 7efcb57fa640 1 -- 172.21.15.46:0/3423743330 <== mon.2 v2:172.21.15.142:3300/0 5 ==== fsmap(e 40) ==== 8370+0+0 (secure 0 0 0) 0x7efcbc026070 con 0x55c0a797e6c0 2024-03-31T23:26:45.570 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:45.567+0000 7efcd676f740 10 client.27206 fetch_fsmap finished waiting for FSMap version 40 2024-03-31T23:26:45.571 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:45.567+0000 7efcd676f740 10 client.27206 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-31T23:26:45.571 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:45.567+0000 7efcd676f740 20 client.27206 populate_metadata read hostname 'smithi046' 2024-03-31T23:26:45.571 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:45.567+0000 7efcd676f740 1 --2- 172.21.15.46:0/3423743330 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55c0a79fec30 0x55c0a7a01050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:26:45.571 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:45.567+0000 7efcd676f740 4 client.27206 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-31T23:26:45.571 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:45.567+0000 7efcd676f740 1 -- 172.21.15.46:0/3423743330 --> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55c0a78f5120 con 0x55c0a79fec30 2024-03-31T23:26:45.571 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:45.567+0000 7efcd1049640 1 --2- 172.21.15.46:0/3423743330 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55c0a79fec30 0x55c0a7a01050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:26:45.571 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:45.567+0000 7efcb57fa640 10 client.27206 ms_handle_connect on v2:172.21.15.142:6838/2519143918 2024-03-31T23:26:45.571 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:45.567+0000 7efcd1049640 1 --2- 172.21.15.46:0/3423743330 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55c0a79fec30 0x55c0a7a01050 crc :-1 s=READY pgs=211 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ef9e8674e2a484bf server_cookie=dcf51ccfcc9c3743 in_seq=0 out_seq=0 2024-03-31T23:26:45.701 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:45.699+0000 7efcb57fa640 1 -- 172.21.15.46:0/3423743330 <== mds.0 v2:172.21.15.142:6838/2519143918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55c0a78f5120 con 0x55c0a79fec30 2024-03-31T23:26:45.702 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:45.699+0000 7efcb57fa640 10 client.27206 handle_command_reply: tid=0 2024-03-31T23:26:45.702 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:45.699+0000 7efcd676f740 1 client.27206 shutdown 2024-03-31T23:26:45.702 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:45.699+0000 7efcd676f740 1 -- 172.21.15.46:0/3423743330 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55c0a797e6c0 msgr2=0x55c0a798b840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:26:45.702 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:45.699+0000 7efcd676f740 1 --2- 172.21.15.46:0/3423743330 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55c0a797e6c0 0x55c0a798b840 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7efcbc027970 tx=0x7efcbc01f370 comp rx=0 tx=0).stop 2024-03-31T23:26:45.702 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:45.700+0000 7efcd676f740 1 -- 172.21.15.46:0/3423743330 shutdown_connections 2024-03-31T23:26:45.702 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:45.700+0000 7efcd676f740 1 --2- 172.21.15.46:0/3423743330 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55c0a79fec30 0x55c0a7a01050 crc :-1 s=READY pgs=211 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:26:45.702 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:45.700+0000 7efcb57fa640 0 client.27206 ms_handle_reset on v2:172.21.15.142:6838/2519143918 2024-03-31T23:26:45.702 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:45.700+0000 7efcd676f740 1 --2- 172.21.15.46:0/3423743330 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c0a7987db0 0x55c0a798c2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:26:45.702 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:45.700+0000 7efcd676f740 1 --2- 172.21.15.46:0/3423743330 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c0a797efe0 0x55c0a798bd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:26:45.702 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:45.700+0000 7efcd676f740 1 --2- 172.21.15.46:0/3423743330 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55c0a797e6c0 0x55c0a798b840 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:26:45.702 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:45.700+0000 7efcd676f740 1 -- 172.21.15.46:0/3423743330 >> 172.21.15.46:0/3423743330 conn(0x55c0a7978fe0 msgr2=0x55c0a7984cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:26:45.702 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:45.700+0000 7efcd676f740 1 -- 172.21.15.46:0/3423743330 shutdown_connections 2024-03-31T23:26:45.702 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:45.700+0000 7efcd676f740 1 -- 172.21.15.46:0/3423743330 wait complete. 2024-03-31T23:26:45.703 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:45.700+0000 7efcd676f740 20 client.27206 trim_cache size 0 max 16384 2024-03-31T23:26:45.703 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout:{ 2024-03-31T23:26:45.703 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout: "message": "", 2024-03-31T23:26:45.703 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout: "return_code": 0 2024-03-31T23:26:45.703 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout:} 2024-03-31T23:26:45.704 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:45.701+0000 7efcd3aae640 1 -- 172.21.15.46:0/2791718864 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7efcb80a2730 msgr2=0x7efcb80a4bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:26:45.704 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:45.701+0000 7efcd3aae640 1 --2- 172.21.15.46:0/2791718864 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7efcb80a2730 0x7efcb80a4bf0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7efcbc005d00 tx=0x7efcbc00b410 comp rx=0 tx=0).stop 2024-03-31T23:26:45.704 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:45.701+0000 7efcd3aae640 1 -- 172.21.15.46:0/2791718864 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efccc11af30 msgr2=0x7efccc1a7440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:26:45.704 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:45.701+0000 7efcd3aae640 1 --2- 172.21.15.46:0/2791718864 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efccc11af30 0x7efccc1a7440 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7efcc8037040 tx=0x7efcc8035a10 comp rx=0 tx=0).stop 2024-03-31T23:26:45.704 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:45.701+0000 7efcd3aae640 1 -- 172.21.15.46:0/2791718864 shutdown_connections 2024-03-31T23:26:45.704 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:45.701+0000 7efcd3aae640 1 --2- 172.21.15.46:0/2791718864 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7efcb80a2730 0x7efcb80a4bf0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:26:45.704 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:45.701+0000 7efcd3aae640 1 --2- 172.21.15.46:0/2791718864 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7efccc1ab7c0 0x7efccc1abc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:26:45.705 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:45.701+0000 7efcd3aae640 1 --2- 172.21.15.46:0/2791718864 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efccc11af30 0x7efccc1a7440 unknown :-1 s=CLOSED pgs=895 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:26:45.705 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:45.701+0000 7efcd3aae640 1 --2- 172.21.15.46:0/2791718864 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7efccc10fe90 0x7efccc1a6f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:26:45.705 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:45.701+0000 7efcd3aae640 1 -- 172.21.15.46:0/2791718864 >> 172.21.15.46:0/2791718864 conn(0x7efccc06c7f0 msgr2=0x7efccc119aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:26:45.705 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:45.701+0000 7efcd3aae640 1 -- 172.21.15.46:0/2791718864 shutdown_connections 2024-03-31T23:26:45.705 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:45.701+0000 7efcd3aae640 1 -- 172.21.15.46:0/2791718864 wait complete. 2024-03-31T23:26:46.270 INFO:tasks.workunit.client.0.smithi046.stdout: 10 18073 10.33 MB/sec warmup 111 sec latency 133.524 ms 2024-03-31T23:26:47.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:26:46 smithi162 ceph-mon[36237]: pgmap v473: 129 pgs: 129 active+clean; 494 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 6.5 MiB/s wr, 401 op/s 2024-03-31T23:26:47.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:26:46 smithi046 ceph-mon[32498]: pgmap v473: 129 pgs: 129 active+clean; 494 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 6.5 MiB/s wr, 401 op/s 2024-03-31T23:26:47.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:26:46 smithi142 ceph-mon[35271]: pgmap v473: 129 pgs: 129 active+clean; 494 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 6.5 MiB/s wr, 401 op/s 2024-03-31T23:26:47.270 INFO:tasks.workunit.client.0.smithi046.stdout: 10 18257 10.34 MB/sec warmup 112 sec latency 66.543 ms 2024-03-31T23:26:48.270 INFO:tasks.workunit.client.0.smithi046.stdout: 10 18430 10.31 MB/sec warmup 113 sec latency 61.885 ms 2024-03-31T23:26:49.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:26:48 smithi162 ceph-mon[36237]: pgmap v474: 129 pgs: 129 active+clean; 494 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 4.1 MiB/s wr, 238 op/s 2024-03-31T23:26:49.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:26:48 smithi046 ceph-mon[32498]: pgmap v474: 129 pgs: 129 active+clean; 494 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 4.1 MiB/s wr, 238 op/s 2024-03-31T23:26:49.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:26:48 smithi142 ceph-mon[35271]: pgmap v474: 129 pgs: 129 active+clean; 494 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 4.1 MiB/s wr, 238 op/s 2024-03-31T23:26:49.270 INFO:tasks.workunit.client.0.smithi046.stdout: 10 18588 10.28 MB/sec warmup 114 sec latency 65.930 ms 2024-03-31T23:26:50.270 INFO:tasks.workunit.client.0.smithi046.stdout: 10 18742 10.24 MB/sec warmup 115 sec latency 66.798 ms 2024-03-31T23:26:51.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:26:50 smithi162 ceph-mon[36237]: pgmap v475: 129 pgs: 129 active+clean; 494 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 6.1 MiB/s wr, 391 op/s 2024-03-31T23:26:51.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:26:50 smithi046 ceph-mon[32498]: pgmap v475: 129 pgs: 129 active+clean; 494 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 6.1 MiB/s wr, 391 op/s 2024-03-31T23:26:51.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:26:50 smithi142 ceph-mon[35271]: pgmap v475: 129 pgs: 129 active+clean; 494 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 6.1 MiB/s wr, 391 op/s 2024-03-31T23:26:51.270 INFO:tasks.workunit.client.0.smithi046.stdout: 10 18932 10.25 MB/sec warmup 116 sec latency 59.671 ms 2024-03-31T23:26:52.270 INFO:tasks.workunit.client.0.smithi046.stdout: 10 19114 10.25 MB/sec warmup 117 sec latency 71.015 ms 2024-03-31T23:26:52.995 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:26:52 smithi142 ceph-mon[35271]: pgmap v476: 129 pgs: 129 active+clean; 494 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.8 MiB/s wr, 274 op/s 2024-03-31T23:26:53.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:26:52 smithi162 ceph-mon[36237]: pgmap v476: 129 pgs: 129 active+clean; 494 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.8 MiB/s wr, 274 op/s 2024-03-31T23:26:53.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:26:52 smithi046 ceph-mon[32498]: pgmap v476: 129 pgs: 129 active+clean; 494 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.8 MiB/s wr, 274 op/s 2024-03-31T23:26:53.270 INFO:tasks.workunit.client.0.smithi046.stdout: 10 19280 10.22 MB/sec warmup 118 sec latency 80.502 ms 2024-03-31T23:26:54.270 INFO:tasks.workunit.client.0.smithi046.stdout: 10 19463 10.26 MB/sec warmup 119 sec latency 92.590 ms 2024-03-31T23:26:55.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:26:54 smithi162 ceph-mon[36237]: pgmap v477: 129 pgs: 129 active+clean; 494 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.8 MiB/s wr, 274 op/s 2024-03-31T23:26:55.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:26:54 smithi046 ceph-mon[32498]: pgmap v477: 129 pgs: 129 active+clean; 494 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.8 MiB/s wr, 274 op/s 2024-03-31T23:26:55.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:26:54 smithi142 ceph-mon[35271]: pgmap v477: 129 pgs: 129 active+clean; 494 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.8 MiB/s wr, 274 op/s 2024-03-31T23:26:56.271 INFO:tasks.workunit.client.0.smithi046.stdout: 10 19802 11.44 MB/sec execute 1 sec latency 71.693 ms 2024-03-31T23:26:57.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:26:56 smithi162 ceph-mon[36237]: pgmap v478: 129 pgs: 129 active+clean; 506 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 5.8 MiB/s wr, 392 op/s 2024-03-31T23:26:57.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:26:56 smithi046 ceph-mon[32498]: pgmap v478: 129 pgs: 129 active+clean; 506 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 5.8 MiB/s wr, 392 op/s 2024-03-31T23:26:57.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:26:56 smithi142 ceph-mon[35271]: pgmap v478: 129 pgs: 129 active+clean; 506 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 5.8 MiB/s wr, 392 op/s 2024-03-31T23:26:57.272 INFO:tasks.workunit.client.0.smithi046.stdout: 10 19942 8.93 MB/sec execute 2 sec latency 82.146 ms 2024-03-31T23:26:58.271 INFO:tasks.workunit.client.0.smithi046.stdout: 10 20120 9.95 MB/sec execute 3 sec latency 72.086 ms 2024-03-31T23:26:58.825 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:58.825+0000 7f54bb449640 1 -- 172.21.15.46:0/1086428168 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f54b41032e0 msgr2=0x7f54b410b7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:26:58.826 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:58.825+0000 7f54bb449640 1 --2- 172.21.15.46:0/1086428168 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f54b41032e0 0x7f54b410b7c0 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7f54a8009900 tx=0x7f54a802bd60 comp rx=0 tx=0).stop 2024-03-31T23:26:58.826 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:58.825+0000 7f54bb449640 1 -- 172.21.15.46:0/1086428168 shutdown_connections 2024-03-31T23:26:58.826 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:58.825+0000 7f54bb449640 1 --2- 172.21.15.46:0/1086428168 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f54b410bd00 0x7f54b410e1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:26:58.826 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:58.825+0000 7f54bb449640 1 --2- 172.21.15.46:0/1086428168 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f54b41032e0 0x7f54b410b7c0 unknown :-1 s=CLOSED pgs=896 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:26:58.826 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:58.825+0000 7f54bb449640 1 --2- 172.21.15.46:0/1086428168 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f54b41029c0 0x7f54b4102da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:26:58.826 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:58.825+0000 7f54bb449640 1 -- 172.21.15.46:0/1086428168 >> 172.21.15.46:0/1086428168 conn(0x7f54b4077780 msgr2=0x7f54b4077b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:26:58.826 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:58.825+0000 7f54bb449640 1 -- 172.21.15.46:0/1086428168 shutdown_connections 2024-03-31T23:26:58.826 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:58.825+0000 7f54bb449640 1 -- 172.21.15.46:0/1086428168 wait complete. 2024-03-31T23:26:58.828 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:58.826+0000 7f54bb449640 1 Processor -- start 2024-03-31T23:26:58.828 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:58.826+0000 7f54bb449640 1 -- start start 2024-03-31T23:26:58.828 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:58.826+0000 7f54bb449640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f54b41029c0 0x7f54b4197f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:26:58.828 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:58.826+0000 7f54bb449640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f54b41032e0 0x7f54b4198450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:26:58.828 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:58.826+0000 7f54bb449640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f54b410bd00 0x7f54b419c780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:26:58.829 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:58.826+0000 7f54b99e6640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f54b410bd00 0x7f54b419c780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:26:58.829 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:58.826+0000 7f54b99e6640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f54b410bd00 0x7f54b419c780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:51228/0 (socket says 172.21.15.46:51228) 2024-03-31T23:26:58.829 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:58.826+0000 7f54bb449640 1 -- 172.21.15.46:0/3032652889 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f54b41119b0 con 0x7f54b410bd00 2024-03-31T23:26:58.829 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:58.826+0000 7f54b89e4640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f54b41032e0 0x7f54b4198450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:26:58.829 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:58.826+0000 7f54b91e5640 1 --2- 172.21.15.46:0/3032652889 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f54b41029c0 0x7f54b4197f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:26:58.829 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:58.826+0000 7f54b99e6640 1 -- 172.21.15.46:0/3032652889 learned_addr learned my addr 172.21.15.46:0/3032652889 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:26:58.829 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:58.826+0000 7f54bb449640 1 -- 172.21.15.46:0/3032652889 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f54b4111830 con 0x7f54b41032e0 2024-03-31T23:26:58.829 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:58.826+0000 7f54bb449640 1 -- 172.21.15.46:0/3032652889 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f54b4111b30 con 0x7f54b41029c0 2024-03-31T23:26:58.829 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:58.827+0000 7f54b99e6640 1 -- 172.21.15.46:0/3032652889 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f54b41029c0 msgr2=0x7f54b4197f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:26:58.829 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:58.827+0000 7f54b99e6640 1 --2- 172.21.15.46:0/3032652889 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f54b41029c0 0x7f54b4197f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:26:58.829 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:58.827+0000 7f54b99e6640 1 -- 172.21.15.46:0/3032652889 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f54b41032e0 msgr2=0x7f54b4198450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:26:58.829 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:58.827+0000 7f54b99e6640 1 --2- 172.21.15.46:0/3032652889 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f54b41032e0 0x7f54b4198450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:26:58.829 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:58.827+0000 7f54b99e6640 1 -- 172.21.15.46:0/3032652889 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f54b419ceb0 con 0x7f54b410bd00 2024-03-31T23:26:58.829 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:58.827+0000 7f54b99e6640 1 --2- 172.21.15.46:0/3032652889 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f54b410bd00 0x7f54b419c780 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7f54ac00eb10 tx=0x7f54ac00c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:26:58.830 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:58.827+0000 7f54b27fc640 1 -- 172.21.15.46:0/3032652889 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54ac004b20 con 0x7f54b410bd00 2024-03-31T23:26:58.830 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:58.827+0000 7f54b27fc640 1 -- 172.21.15.46:0/3032652889 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f54ac004cc0 con 0x7f54b410bd00 2024-03-31T23:26:58.830 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:58.827+0000 7f54bb449640 1 -- 172.21.15.46:0/3032652889 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f54b419d1a0 con 0x7f54b410bd00 2024-03-31T23:26:58.830 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:58.827+0000 7f54b27fc640 1 -- 172.21.15.46:0/3032652889 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54ac010690 con 0x7f54b410bd00 2024-03-31T23:26:58.830 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:58.827+0000 7f54bb449640 1 -- 172.21.15.46:0/3032652889 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f54b4100540 con 0x7f54b410bd00 2024-03-31T23:26:58.830 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:58.830+0000 7f54b27fc640 1 -- 172.21.15.46:0/3032652889 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 25) ==== 137451+0+0 (secure 0 0 0) 0x7f54ac005020 con 0x7f54b410bd00 2024-03-31T23:26:58.834 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:58.831+0000 7f54be021740 1 Processor -- start 2024-03-31T23:26:58.834 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:58.831+0000 7f54be021740 1 -- start start 2024-03-31T23:26:58.834 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:58.831+0000 7f54b27fc640 1 --2- 172.21.15.46:0/3032652889 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f549c0a26b0 0x7f549c0a4b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:26:58.834 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:58.831+0000 7f54be021740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b8b94f26c0 0x55b8b94f2aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:26:58.834 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:58.831+0000 7f54be021740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b8b94f2fe0 0x55b8b94fb870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:26:58.834 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:58.831+0000 7f54be021740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55b8b94fbdb0 0x55b8b94fe1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:26:58.835 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:58.831+0000 7f54be021740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55b8b94558f0 con 0x55b8b94f2fe0 2024-03-31T23:26:58.835 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:58.831+0000 7f54be021740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55b8b93fe8d0 con 0x55b8b94fbdb0 2024-03-31T23:26:58.835 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:58.831+0000 7f54be021740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55b8b9451990 con 0x55b8b94f26c0 2024-03-31T23:26:58.835 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:58.831+0000 7f54b27fc640 1 -- 172.21.15.46:0/3032652889 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(115..115 src has 1..115) ==== 9554+0+0 (secure 0 0 0) 0x7f54ac0cf350 con 0x7f54b410bd00 2024-03-31T23:26:58.835 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:58.831+0000 7f54b89e4640 1 --2- 172.21.15.46:0/3032652889 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f549c0a26b0 0x7f549c0a4b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:26:58.835 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:58.831+0000 7f54b91e5640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b8b94f26c0 0x55b8b94f2aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:26:58.835 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:58.831+0000 7f54b99e6640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55b8b94fbdb0 0x55b8b94fe1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:26:58.835 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:58.831+0000 7f54b99e6640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55b8b94fbdb0 0x55b8b94fe1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.46:41520/0 (socket says 172.21.15.46:41520) 2024-03-31T23:26:58.835 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:58.831+0000 7f54b99e6640 1 -- 172.21.15.46:0/1222629494 learned_addr learned my addr 172.21.15.46:0/1222629494 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:26:58.835 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:58.831+0000 7f54b91e5640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b8b94f26c0 0x55b8b94f2aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.46:35662/0 (socket says 172.21.15.46:35662) 2024-03-31T23:26:58.835 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:58.831+0000 7f54b89e4640 1 --2- 172.21.15.46:0/1222629494 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b8b94f2fe0 0x55b8b94fb870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:26:58.835 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:58.832+0000 7f54b91e5640 1 -- 172.21.15.46:0/1222629494 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55b8b94fbdb0 msgr2=0x55b8b94fe1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:26:58.836 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:58.832+0000 7f54b91e5640 1 --2- 172.21.15.46:0/1222629494 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55b8b94fbdb0 0x55b8b94fe1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:26:58.836 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:58.832+0000 7f54b91e5640 1 -- 172.21.15.46:0/1222629494 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b8b94f2fe0 msgr2=0x55b8b94fb870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:26:58.836 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:58.832+0000 7f54b91e5640 1 --2- 172.21.15.46:0/1222629494 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b8b94f2fe0 0x55b8b94fb870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:26:58.836 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:58.832+0000 7f54b91e5640 1 -- 172.21.15.46:0/1222629494 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b8b916a210 con 0x55b8b94f26c0 2024-03-31T23:26:58.836 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:58.832+0000 7f54b89e4640 1 --2- 172.21.15.46:0/3032652889 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f549c0a26b0 0x7f549c0a4b70 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f54a8006010 tx=0x7f54a802c740 comp rx=0 tx=0).ready entity=mgr.15654 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:26:58.836 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:58.832+0000 7f54b91e5640 1 --2- 172.21.15.46:0/1222629494 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b8b94f26c0 0x55b8b94f2aa0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f54a4009c20 tx=0x7f54a4011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b080f74f03d7ac13 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:26:58.836 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:58.832+0000 7f549b7fe640 1 -- 172.21.15.46:0/1222629494 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54a4004050 con 0x55b8b94f26c0 2024-03-31T23:26:58.836 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:58.832+0000 7f549b7fe640 1 -- 172.21.15.46:0/1222629494 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f54a40054d0 con 0x55b8b94f26c0 2024-03-31T23:26:58.836 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:58.832+0000 7f549b7fe640 1 -- 172.21.15.46:0/1222629494 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54a4004bc0 con 0x55b8b94f26c0 2024-03-31T23:26:58.836 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:58.833+0000 7f54be021740 1 -- 172.21.15.46:0/1222629494 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b8b94f26c0 msgr2=0x55b8b94f2aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:26:58.836 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:58.833+0000 7f54be021740 1 --2- 172.21.15.46:0/1222629494 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b8b94f26c0 0x55b8b94f2aa0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f54a4009c20 tx=0x7f54a4011810 comp rx=0 tx=0).stop 2024-03-31T23:26:58.836 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:58.833+0000 7f54be021740 1 -- 172.21.15.46:0/1222629494 shutdown_connections 2024-03-31T23:26:58.836 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:58.833+0000 7f54be021740 1 --2- 172.21.15.46:0/1222629494 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55b8b94fbdb0 0x55b8b94fe1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:26:58.836 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:58.833+0000 7f54be021740 1 --2- 172.21.15.46:0/1222629494 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b8b94f2fe0 0x55b8b94fb870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:26:58.837 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:58.833+0000 7f54be021740 1 --2- 172.21.15.46:0/1222629494 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b8b94f26c0 0x55b8b94f2aa0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:26:58.837 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:58.833+0000 7f54be021740 1 -- 172.21.15.46:0/1222629494 >> 172.21.15.46:0/1222629494 conn(0x55b8b94ecfe0 msgr2=0x55b8b94ed3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:26:58.837 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:58.833+0000 7f54be021740 1 -- 172.21.15.46:0/1222629494 shutdown_connections 2024-03-31T23:26:58.837 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:58.833+0000 7f54be021740 1 -- 172.21.15.46:0/1222629494 wait complete. 2024-03-31T23:26:58.837 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:58.834+0000 7f54be021740 1 -- start start 2024-03-31T23:26:58.837 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:58.834+0000 7f54be021740 1 Processor -- start 2024-03-31T23:26:58.841 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:58.834+0000 7f54be021740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55b8b94f26c0 0x55b8b94f8d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:26:58.841 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:58.834+0000 7f54be021740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b8b94f2fe0 0x55b8b94f92a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:26:58.841 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:58.834+0000 7f54be021740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b8b94fbdb0 0x55b8b94f97e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:26:58.841 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:58.834+0000 7f54be021740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55b8b94fe9e0 con 0x55b8b94fbdb0 2024-03-31T23:26:58.841 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:58.834+0000 7f54be021740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55b8b93f3f80 con 0x55b8b94f26c0 2024-03-31T23:26:58.841 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:58.834+0000 7f54be021740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55b8b94feb60 con 0x55b8b94f2fe0 2024-03-31T23:26:58.841 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:58.834+0000 7f54be021740 10 client.0 _ll_register_callbacks cb 0x55b8b944c290 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T23:26:58.841 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:58.835+0000 7f54b91e5640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55b8b94f26c0 0x55b8b94f8d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:26:58.842 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:58.835+0000 7f54b91e5640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55b8b94f26c0 0x55b8b94f8d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.46:41526/0 (socket says 172.21.15.46:41526) 2024-03-31T23:26:58.842 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:58.835+0000 7f54b91e5640 1 -- 172.21.15.46:0/392919126 learned_addr learned my addr 172.21.15.46:0/392919126 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:26:58.842 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:58.835+0000 7f54b99e6640 1 --2- 172.21.15.46:0/392919126 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b8b94fbdb0 0x55b8b94f97e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:26:58.842 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:58.835+0000 7f54b89e4640 1 --2- 172.21.15.46:0/392919126 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b8b94f2fe0 0x55b8b94f92a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:26:58.842 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:58.835+0000 7f54b91e5640 1 -- 172.21.15.46:0/392919126 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b8b94f2fe0 msgr2=0x55b8b94f92a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:26:58.842 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:58.835+0000 7f54b91e5640 1 --2- 172.21.15.46:0/392919126 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b8b94f2fe0 0x55b8b94f92a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:26:58.842 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:58.835+0000 7f54b91e5640 1 -- 172.21.15.46:0/392919126 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b8b94fbdb0 msgr2=0x55b8b94f97e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:26:58.842 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:58.835+0000 7f54b91e5640 1 --2- 172.21.15.46:0/392919126 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b8b94fbdb0 0x55b8b94f97e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:26:58.842 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:58.835+0000 7f54b91e5640 1 -- 172.21.15.46:0/392919126 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b8b9236620 con 0x55b8b94f26c0 2024-03-31T23:26:58.842 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:58.835+0000 7f54b89e4640 1 --2- 172.21.15.46:0/392919126 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b8b94f2fe0 0x55b8b94f92a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:26:58.842 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:58.835+0000 7f54b99e6640 1 --2- 172.21.15.46:0/392919126 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b8b94fbdb0 0x55b8b94f97e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:26:58.842 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:58.835+0000 7f54b91e5640 1 --2- 172.21.15.46:0/392919126 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55b8b94f26c0 0x55b8b94f8d60 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f54a401b040 tx=0x7f54a4004300 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7df590a992049897 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:26:58.842 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:58.835+0000 7f5498ff9640 10 client.0 ms_handle_connect on v2:172.21.15.142:3300/0 2024-03-31T23:26:58.842 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:58.835+0000 7f5498ff9640 1 -- 172.21.15.46:0/392919126 <== mon.2 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54a4029070 con 0x55b8b94f26c0 2024-03-31T23:26:58.843 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:58.836+0000 7f54be021740 1 -- 172.21.15.46:0/392919126 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f54a40054d0 con 0x55b8b94f26c0 2024-03-31T23:26:58.843 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:58.836+0000 7f5498ff9640 1 -- 172.21.15.46:0/392919126 <== mon.2 v2:172.21.15.142:3300/0 2 ==== osd_map(115..115 src has 1..115) ==== 9554+0+0 (secure 0 0 0) 0x7f54a4003a00 con 0x55b8b94f26c0 2024-03-31T23:26:58.843 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:58.836+0000 7f5498ff9640 1 -- 172.21.15.46:0/392919126 <== mon.2 v2:172.21.15.142:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7f54a4035050 con 0x55b8b94f26c0 2024-03-31T23:26:58.843 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:58.836+0000 7f54be021740 10 client.27218 fetch_fsmap learned FSMap version 40 2024-03-31T23:26:58.843 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:58.836+0000 7f54be021740 1 -- 172.21.15.46:0/392919126 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x55b8b950a830 con 0x55b8b94f26c0 2024-03-31T23:26:58.843 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:58.836+0000 7f5498ff9640 1 -- 172.21.15.46:0/392919126 <== mon.2 v2:172.21.15.142:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f54a40054d0 con 0x55b8b94f26c0 2024-03-31T23:26:58.843 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:58.836+0000 7f5498ff9640 1 -- 172.21.15.46:0/392919126 <== mon.2 v2:172.21.15.142:3300/0 5 ==== fsmap(e 40) ==== 8370+0+0 (secure 0 0 0) 0x7f54a40050f0 con 0x55b8b94f26c0 2024-03-31T23:26:58.843 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:58.836+0000 7f54be021740 10 client.27218 fetch_fsmap finished waiting for FSMap version 40 2024-03-31T23:26:58.843 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:58.836+0000 7f54be021740 10 client.27218 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-31T23:26:58.843 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:58.836+0000 7f54be021740 20 client.27218 populate_metadata read hostname 'smithi046' 2024-03-31T23:26:58.843 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:58.836+0000 7f54be021740 1 --2- 172.21.15.46:0/392919126 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55b8b950bc40 0x55b8b950e050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:26:58.843 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:58.836+0000 7f54be021740 4 client.27218 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-31T23:26:58.843 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:58.836+0000 7f54be021740 1 -- 172.21.15.46:0/392919126 --> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b8b9469120 con 0x55b8b950bc40 2024-03-31T23:26:58.843 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:58.837+0000 7f54b89e4640 1 --2- 172.21.15.46:0/392919126 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55b8b950bc40 0x55b8b950e050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:26:58.844 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:58.837+0000 7f54b89e4640 1 --2- 172.21.15.46:0/392919126 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55b8b950bc40 0x55b8b950e050 crc :-1 s=READY pgs=212 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1992d7ae88d0ffd3 server_cookie=5e5c72fa1590d34d in_seq=0 out_seq=0 2024-03-31T23:26:58.844 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:58.837+0000 7f5498ff9640 10 client.27218 ms_handle_connect on v2:172.21.15.142:6838/2519143918 2024-03-31T23:26:58.844 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:58.839+0000 7f5498ff9640 1 -- 172.21.15.46:0/392919126 <== mds.0 v2:172.21.15.142:6838/2519143918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55b8b9469120 con 0x55b8b950bc40 2024-03-31T23:26:58.844 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:58.839+0000 7f5498ff9640 10 client.27218 handle_command_reply: tid=0 2024-03-31T23:26:58.844 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:58.839+0000 7f54be021740 1 client.27218 shutdown 2024-03-31T23:26:58.844 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:58.839+0000 7f54be021740 1 -- 172.21.15.46:0/392919126 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55b8b94f26c0 msgr2=0x55b8b94f8d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:26:58.844 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:58.839+0000 7f54be021740 1 --2- 172.21.15.46:0/392919126 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55b8b94f26c0 0x55b8b94f8d60 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f54a401b040 tx=0x7f54a4004300 comp rx=0 tx=0).stop 2024-03-31T23:26:58.844 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:58.839+0000 7f54be021740 1 -- 172.21.15.46:0/392919126 shutdown_connections 2024-03-31T23:26:58.844 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:58.839+0000 7f54be021740 1 --2- 172.21.15.46:0/392919126 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55b8b950bc40 0x55b8b950e050 crc :-1 s=READY pgs=212 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:26:58.844 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:58.839+0000 7f54be021740 1 --2- 172.21.15.46:0/392919126 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b8b94fbdb0 0x55b8b94f97e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:26:58.844 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:58.839+0000 7f5498ff9640 0 client.27218 ms_handle_reset on v2:172.21.15.142:6838/2519143918 2024-03-31T23:26:58.844 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:58.839+0000 7f54be021740 1 --2- 172.21.15.46:0/392919126 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b8b94f2fe0 0x55b8b94f92a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:26:58.844 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:58.839+0000 7f54be021740 1 --2- 172.21.15.46:0/392919126 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55b8b94f26c0 0x55b8b94f8d60 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:26:58.844 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:58.839+0000 7f54be021740 1 -- 172.21.15.46:0/392919126 >> 172.21.15.46:0/392919126 conn(0x55b8b94ecfe0 msgr2=0x55b8b94eed30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:26:58.845 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:58.839+0000 7f54be021740 1 -- 172.21.15.46:0/392919126 shutdown_connections 2024-03-31T23:26:58.845 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:58.839+0000 7f54be021740 1 -- 172.21.15.46:0/392919126 wait complete. 2024-03-31T23:26:58.845 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:58.839+0000 7f54be021740 20 client.27218 trim_cache size 0 max 16384 2024-03-31T23:26:58.864 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:58.862+0000 7f54be021740 1 Processor -- start 2024-03-31T23:26:58.864 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:58.862+0000 7f54be021740 1 -- start start 2024-03-31T23:26:58.864 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:58.863+0000 7f54be021740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b8b94f26c0 0x55b8b94fb490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:26:58.864 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:58.863+0000 7f54be021740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55b8b94f2fe0 0x55b8b9507780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:26:58.864 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:58.863+0000 7f54be021740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b8b94fbdb0 0x55b8b94f8640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:26:58.864 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:58.863+0000 7f54be021740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f54a0012a70 con 0x55b8b94f26c0 2024-03-31T23:26:58.864 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:58.863+0000 7f54be021740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f54a0010220 con 0x55b8b94f2fe0 2024-03-31T23:26:58.864 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:58.863+0000 7f54be021740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f54a000e090 con 0x55b8b94fbdb0 2024-03-31T23:26:58.864 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:58.863+0000 7f54b91e5640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b8b94f26c0 0x55b8b94fb490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:26:58.864 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:58.863+0000 7f54b91e5640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b8b94f26c0 0x55b8b94fb490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:51256/0 (socket says 172.21.15.46:51256) 2024-03-31T23:26:58.864 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:58.863+0000 7f54b91e5640 1 -- 172.21.15.46:0/3200864964 learned_addr learned my addr 172.21.15.46:0/3200864964 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:26:58.865 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:58.863+0000 7f54b89e4640 1 --2- 172.21.15.46:0/3200864964 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55b8b94f2fe0 0x55b8b9507780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:26:58.865 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:58.863+0000 7f54b99e6640 1 --2- 172.21.15.46:0/3200864964 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b8b94fbdb0 0x55b8b94f8640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:26:58.865 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:58.863+0000 7f54b91e5640 1 -- 172.21.15.46:0/3200864964 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b8b94fbdb0 msgr2=0x55b8b94f8640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:26:58.865 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:58.863+0000 7f54b91e5640 1 --2- 172.21.15.46:0/3200864964 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b8b94fbdb0 0x55b8b94f8640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:26:58.865 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:58.863+0000 7f54b91e5640 1 -- 172.21.15.46:0/3200864964 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55b8b94f2fe0 msgr2=0x55b8b9507780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:26:58.865 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:58.863+0000 7f54b91e5640 1 --2- 172.21.15.46:0/3200864964 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55b8b94f2fe0 0x55b8b9507780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:26:58.865 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:58.863+0000 7f54b91e5640 1 -- 172.21.15.46:0/3200864964 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b8b94fe1a0 con 0x55b8b94f26c0 2024-03-31T23:26:58.865 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:58.863+0000 7f54b91e5640 1 --2- 172.21.15.46:0/3200864964 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b8b94f26c0 0x55b8b94fb490 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7f54a4000c00 tx=0x7f54a40236a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e8c7c011812b90d server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:26:58.865 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:58.864+0000 7f549affd640 1 -- 172.21.15.46:0/3200864964 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54a4032020 con 0x55b8b94f26c0 2024-03-31T23:26:58.865 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:58.864+0000 7f549affd640 1 -- 172.21.15.46:0/3200864964 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f54a40097a0 con 0x55b8b94f26c0 2024-03-31T23:26:58.865 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:58.864+0000 7f549affd640 1 -- 172.21.15.46:0/3200864964 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54a4029040 con 0x55b8b94f26c0 2024-03-31T23:26:58.869 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:58.865+0000 7f54be021740 1 -- 172.21.15.46:0/3200864964 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b8b94f26c0 msgr2=0x55b8b94fb490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:26:58.870 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:58.865+0000 7f54be021740 1 --2- 172.21.15.46:0/3200864964 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b8b94f26c0 0x55b8b94fb490 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7f54a4000c00 tx=0x7f54a40236a0 comp rx=0 tx=0).stop 2024-03-31T23:26:58.870 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:58.865+0000 7f54be021740 1 -- 172.21.15.46:0/3200864964 shutdown_connections 2024-03-31T23:26:58.870 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:58.865+0000 7f54be021740 1 --2- 172.21.15.46:0/3200864964 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b8b94fbdb0 0x55b8b94f8640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:26:58.870 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:58.865+0000 7f54be021740 1 --2- 172.21.15.46:0/3200864964 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55b8b94f2fe0 0x55b8b9507780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:26:58.870 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:58.865+0000 7f54be021740 1 --2- 172.21.15.46:0/3200864964 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b8b94f26c0 0x55b8b94fb490 unknown :-1 s=CLOSED pgs=898 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:26:58.870 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:58.865+0000 7f54be021740 1 -- 172.21.15.46:0/3200864964 >> 172.21.15.46:0/3200864964 conn(0x55b8b94ecfe0 msgr2=0x55b8b94ebf80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:26:58.870 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:58.865+0000 7f54be021740 1 -- 172.21.15.46:0/3200864964 shutdown_connections 2024-03-31T23:26:58.870 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:58.865+0000 7f54be021740 1 -- 172.21.15.46:0/3200864964 wait complete. 2024-03-31T23:26:58.870 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:58.865+0000 7f54be021740 1 -- start start 2024-03-31T23:26:58.870 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:58.865+0000 7f54be021740 1 Processor -- start 2024-03-31T23:26:58.870 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:58.866+0000 7f54be021740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b8b94f26c0 0x55b8b94f62b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:26:58.870 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:58.866+0000 7f54be021740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b8b94f2fe0 0x55b8b94f67f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:26:58.870 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:58.866+0000 7f54be021740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55b8b94fbdb0 0x55b8b94f4fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:26:58.871 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:58.866+0000 7f54be021740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f54a000e6e0 con 0x55b8b94f2fe0 2024-03-31T23:26:58.871 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:58.866+0000 7f54be021740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f54a000fb50 con 0x55b8b94fbdb0 2024-03-31T23:26:58.871 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:58.866+0000 7f54be021740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f54a000edb0 con 0x55b8b94f26c0 2024-03-31T23:26:58.871 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:58.866+0000 7f54be021740 10 client.0 _ll_register_callbacks cb 0x7f54a4005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T23:26:58.871 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:58.866+0000 7f54b89e4640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b8b94f2fe0 0x55b8b94f67f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:26:58.871 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:58.866+0000 7f54b91e5640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b8b94f26c0 0x55b8b94f62b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:26:58.871 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:58.866+0000 7f54b89e4640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b8b94f2fe0 0x55b8b94f67f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:51258/0 (socket says 172.21.15.46:51258) 2024-03-31T23:26:58.871 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:58.866+0000 7f54b89e4640 1 -- 172.21.15.46:0/728518538 learned_addr learned my addr 172.21.15.46:0/728518538 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:26:58.871 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:58.866+0000 7f54b99e6640 1 --2- 172.21.15.46:0/728518538 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55b8b94fbdb0 0x55b8b94f4fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:26:58.871 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:58.867+0000 7f54b89e4640 1 -- 172.21.15.46:0/728518538 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b8b94f26c0 msgr2=0x55b8b94f62b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:26:58.871 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:58.867+0000 7f54b89e4640 1 --2- 172.21.15.46:0/728518538 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b8b94f26c0 0x55b8b94f62b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:26:58.871 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:58.867+0000 7f54b89e4640 1 -- 172.21.15.46:0/728518538 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55b8b94fbdb0 msgr2=0x55b8b94f4fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:26:58.871 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:58.867+0000 7f54b89e4640 1 --2- 172.21.15.46:0/728518538 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55b8b94fbdb0 0x55b8b94f4fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:26:58.872 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:58.867+0000 7f54b89e4640 1 -- 172.21.15.46:0/728518538 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b8b9504480 con 0x55b8b94f2fe0 2024-03-31T23:26:58.872 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:58.867+0000 7f5498ff9640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-31T23:26:58.872 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:58.867+0000 7f54b89e4640 1 --2- 172.21.15.46:0/728518538 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b8b94f2fe0 0x55b8b94f67f0 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7f54a8009850 tx=0x7f54a8005e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d83d2e9bd0a7ea2 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:26:58.872 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:58.867+0000 7f5498ff9640 1 -- 172.21.15.46:0/728518538 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54a80055d0 con 0x55b8b94f2fe0 2024-03-31T23:26:58.872 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:58.867+0000 7f5498ff9640 1 -- 172.21.15.46:0/728518538 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(115..115 src has 1..115) ==== 9554+0+0 (secure 0 0 0) 0x7f54a80542c0 con 0x55b8b94f2fe0 2024-03-31T23:26:58.872 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:58.867+0000 7f54be021740 1 -- 172.21.15.46:0/728518538 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f54a40097a0 con 0x55b8b94f2fe0 2024-03-31T23:26:58.872 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:58.867+0000 7f5498ff9640 1 -- 172.21.15.46:0/728518538 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7f54a8065050 con 0x55b8b94f2fe0 2024-03-31T23:26:58.872 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:58.867+0000 7f54be021740 10 client.17394 fetch_fsmap learned FSMap version 40 2024-03-31T23:26:58.872 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:58.867+0000 7f54be021740 1 -- 172.21.15.46:0/728518538 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x55b8b95714a0 con 0x55b8b94f2fe0 2024-03-31T23:26:58.872 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:58.868+0000 7f5498ff9640 1 -- 172.21.15.46:0/728518538 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f54a40097a0 con 0x55b8b94f2fe0 2024-03-31T23:26:58.872 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:58.868+0000 7f5498ff9640 1 -- 172.21.15.46:0/728518538 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 40) ==== 8370+0+0 (secure 0 0 0) 0x7f54a804f040 con 0x55b8b94f2fe0 2024-03-31T23:26:58.872 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:58.868+0000 7f54be021740 10 client.17394 fetch_fsmap finished waiting for FSMap version 40 2024-03-31T23:26:58.872 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:58.868+0000 7f54be021740 10 client.17394 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-31T23:26:58.872 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:58.868+0000 7f54be021740 20 client.17394 populate_metadata read hostname 'smithi046' 2024-03-31T23:26:58.873 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:58.868+0000 7f54be021740 1 --2- 172.21.15.46:0/728518538 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55b8b950bc40 0x55b8b9573ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:26:58.873 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:58.868+0000 7f54be021740 4 client.17394 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-31T23:26:58.873 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:58.868+0000 7f54be021740 1 -- 172.21.15.46:0/728518538 --> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55b8b9462e30 con 0x55b8b950bc40 2024-03-31T23:26:58.873 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:58.868+0000 7f54b91e5640 1 --2- 172.21.15.46:0/728518538 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55b8b950bc40 0x55b8b9573ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:26:58.873 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:58.869+0000 7f5498ff9640 10 client.17394 ms_handle_connect on v2:172.21.15.142:6838/2519143918 2024-03-31T23:26:58.873 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:58.869+0000 7f54b91e5640 1 --2- 172.21.15.46:0/728518538 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55b8b950bc40 0x55b8b9573ca0 crc :-1 s=READY pgs=213 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4ae15e06f847ad9 server_cookie=542802574cc4d0b7 in_seq=0 out_seq=0 2024-03-31T23:26:59.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:26:58 smithi162 ceph-mon[36237]: pgmap v479: 129 pgs: 129 active+clean; 506 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 4.1 MiB/s wr, 271 op/s 2024-03-31T23:26:59.070 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:59.069+0000 7f5498ff9640 1 -- 172.21.15.46:0/728518538 <== mds.0 v2:172.21.15.142:6838/2519143918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55b8b9462e30 con 0x55b8b950bc40 2024-03-31T23:26:59.070 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:59.069+0000 7f5498ff9640 10 client.17394 handle_command_reply: tid=0 2024-03-31T23:26:59.070 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:59.069+0000 7f54be021740 1 client.17394 shutdown 2024-03-31T23:26:59.070 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:59.069+0000 7f54be021740 1 -- 172.21.15.46:0/728518538 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b8b94f2fe0 msgr2=0x55b8b94f67f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:26:59.070 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:59.069+0000 7f54be021740 1 --2- 172.21.15.46:0/728518538 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b8b94f2fe0 0x55b8b94f67f0 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7f54a8009850 tx=0x7f54a8005e40 comp rx=0 tx=0).stop 2024-03-31T23:26:59.070 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:59.069+0000 7f54be021740 1 -- 172.21.15.46:0/728518538 shutdown_connections 2024-03-31T23:26:59.070 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:59.069+0000 7f54be021740 1 --2- 172.21.15.46:0/728518538 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55b8b950bc40 0x55b8b9573ca0 crc :-1 s=READY pgs=213 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:26:59.070 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:59.069+0000 7f54be021740 1 --2- 172.21.15.46:0/728518538 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55b8b94fbdb0 0x55b8b94f4fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:26:59.070 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:59.069+0000 7f54be021740 1 --2- 172.21.15.46:0/728518538 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b8b94f2fe0 0x55b8b94f67f0 unknown :-1 s=CLOSED pgs=899 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:26:59.071 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:59.069+0000 7f54be021740 1 --2- 172.21.15.46:0/728518538 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b8b94f26c0 0x55b8b94f62b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:26:59.071 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:59.069+0000 7f54be021740 1 -- 172.21.15.46:0/728518538 >> 172.21.15.46:0/728518538 conn(0x55b8b94ecfe0 msgr2=0x55b8b94ebf80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:26:59.071 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:59.069+0000 7f5498ff9640 0 client.17394 ms_handle_reset on v2:172.21.15.142:6838/2519143918 2024-03-31T23:26:59.071 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:59.070+0000 7f54be021740 1 -- 172.21.15.46:0/728518538 shutdown_connections 2024-03-31T23:26:59.071 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:59.070+0000 7f54be021740 1 -- 172.21.15.46:0/728518538 wait complete. 2024-03-31T23:26:59.071 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:59.070+0000 7f54be021740 20 client.17394 trim_cache size 0 max 16384 2024-03-31T23:26:59.071 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout:{ 2024-03-31T23:26:59.071 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout: "message": "", 2024-03-31T23:26:59.071 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout: "return_code": 0 2024-03-31T23:26:59.072 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout:} 2024-03-31T23:26:59.072 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:59.071+0000 7f54bb449640 1 -- 172.21.15.46:0/3032652889 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f549c0a26b0 msgr2=0x7f549c0a4b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:26:59.072 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:59.071+0000 7f54bb449640 1 --2- 172.21.15.46:0/3032652889 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f549c0a26b0 0x7f549c0a4b70 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f54a8006010 tx=0x7f54a802c740 comp rx=0 tx=0).stop 2024-03-31T23:26:59.072 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:59.071+0000 7f54bb449640 1 -- 172.21.15.46:0/3032652889 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f54b410bd00 msgr2=0x7f54b419c780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:26:59.073 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:59.071+0000 7f54bb449640 1 --2- 172.21.15.46:0/3032652889 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f54b410bd00 0x7f54b419c780 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7f54ac00eb10 tx=0x7f54ac00c2d0 comp rx=0 tx=0).stop 2024-03-31T23:26:59.073 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:59.071+0000 7f54bb449640 1 -- 172.21.15.46:0/3032652889 shutdown_connections 2024-03-31T23:26:59.073 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:59.071+0000 7f54bb449640 1 --2- 172.21.15.46:0/3032652889 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f549c0a26b0 0x7f549c0a4b70 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:26:59.073 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:59.071+0000 7f54bb449640 1 --2- 172.21.15.46:0/3032652889 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f54b410bd00 0x7f54b419c780 unknown :-1 s=CLOSED pgs=897 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:26:59.073 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:59.071+0000 7f54bb449640 1 --2- 172.21.15.46:0/3032652889 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f54b41032e0 0x7f54b4198450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:26:59.073 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:59.071+0000 7f54bb449640 1 --2- 172.21.15.46:0/3032652889 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f54b41029c0 0x7f54b4197f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:26:59.073 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:59.071+0000 7f54bb449640 1 -- 172.21.15.46:0/3032652889 >> 172.21.15.46:0/3032652889 conn(0x7f54b4077780 msgr2=0x7f54b40fd7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:26:59.073 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:59.072+0000 7f54bb449640 1 -- 172.21.15.46:0/3032652889 shutdown_connections 2024-03-31T23:26:59.073 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:26:59.072+0000 7f54bb449640 1 -- 172.21.15.46:0/3032652889 wait complete. 2024-03-31T23:26:59.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:26:58 smithi046 ceph-mon[32498]: pgmap v479: 129 pgs: 129 active+clean; 506 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 4.1 MiB/s wr, 271 op/s 2024-03-31T23:26:59.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:26:58 smithi142 ceph-mon[35271]: pgmap v479: 129 pgs: 129 active+clean; 506 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 4.1 MiB/s wr, 271 op/s 2024-03-31T23:26:59.273 INFO:tasks.workunit.client.0.smithi046.stdout: 10 20275 9.19 MB/sec execute 4 sec latency 188.509 ms 2024-03-31T23:27:00.273 INFO:tasks.workunit.client.0.smithi046.stdout: 10 20412 8.12 MB/sec execute 5 sec latency 66.917 ms 2024-03-31T23:27:01.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:27:00 smithi162 ceph-mon[36237]: pgmap v480: 129 pgs: 129 active+clean; 515 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 6.3 MiB/s wr, 386 op/s 2024-03-31T23:27:01.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:27:00 smithi046 ceph-mon[32498]: pgmap v480: 129 pgs: 129 active+clean; 515 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 6.3 MiB/s wr, 386 op/s 2024-03-31T23:27:01.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:27:00 smithi142 ceph-mon[35271]: pgmap v480: 129 pgs: 129 active+clean; 515 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 6.3 MiB/s wr, 386 op/s 2024-03-31T23:27:01.273 INFO:tasks.workunit.client.0.smithi046.stdout: 10 20530 7.76 MB/sec execute 6 sec latency 193.641 ms 2024-03-31T23:27:02.273 INFO:tasks.workunit.client.0.smithi046.stdout: 10 20677 7.52 MB/sec execute 7 sec latency 101.522 ms 2024-03-31T23:27:03.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:27:02 smithi162 ceph-mon[36237]: pgmap v481: 129 pgs: 129 active+clean; 515 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 4.3 MiB/s wr, 233 op/s 2024-03-31T23:27:03.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:27:02 smithi046 ceph-mon[32498]: pgmap v481: 129 pgs: 129 active+clean; 515 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 4.3 MiB/s wr, 233 op/s 2024-03-31T23:27:03.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:27:02 smithi142 ceph-mon[35271]: pgmap v481: 129 pgs: 129 active+clean; 515 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 4.3 MiB/s wr, 233 op/s 2024-03-31T23:27:03.273 INFO:tasks.workunit.client.0.smithi046.stdout: 10 20854 8.11 MB/sec execute 8 sec latency 92.603 ms 2024-03-31T23:27:04.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:27:03 smithi162 ceph-mon[36237]: osdmap e116: 12 total, 12 up, 12 in 2024-03-31T23:27:04.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:27:03 smithi046 ceph-mon[32498]: osdmap e116: 12 total, 12 up, 12 in 2024-03-31T23:27:04.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:27:03 smithi142 ceph-mon[35271]: osdmap e116: 12 total, 12 up, 12 in 2024-03-31T23:27:04.274 INFO:tasks.workunit.client.0.smithi046.stdout: 10 21031 7.90 MB/sec execute 9 sec latency 80.808 ms 2024-03-31T23:27:05.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:27:04 smithi162 ceph-mon[36237]: pgmap v483: 129 pgs: 129 active+clean; 515 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 5.1 MiB/s wr, 279 op/s 2024-03-31T23:27:05.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:27:04 smithi162 ceph-mon[36237]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-31T23:27:05.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:27:04 smithi046 ceph-mon[32498]: pgmap v483: 129 pgs: 129 active+clean; 515 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 5.1 MiB/s wr, 279 op/s 2024-03-31T23:27:05.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:27:04 smithi046 ceph-mon[32498]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-31T23:27:05.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:27:04 smithi142 ceph-mon[35271]: pgmap v483: 129 pgs: 129 active+clean; 515 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 5.1 MiB/s wr, 279 op/s 2024-03-31T23:27:05.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:27:04 smithi142 ceph-mon[35271]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-31T23:27:05.274 INFO:tasks.workunit.client.0.smithi046.stdout: 10 21192 8.07 MB/sec execute 10 sec latency 151.879 ms 2024-03-31T23:27:06.274 INFO:tasks.workunit.client.0.smithi046.stdout: 10 21336 7.83 MB/sec execute 11 sec latency 69.100 ms 2024-03-31T23:27:07.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:27:06 smithi162 ceph-mon[36237]: pgmap v484: 129 pgs: 129 active+clean; 519 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 5.3 MiB/s wr, 367 op/s 2024-03-31T23:27:07.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:27:06 smithi046 ceph-mon[32498]: pgmap v484: 129 pgs: 129 active+clean; 519 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 5.3 MiB/s wr, 367 op/s 2024-03-31T23:27:07.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:27:06 smithi142 ceph-mon[35271]: pgmap v484: 129 pgs: 129 active+clean; 519 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 5.3 MiB/s wr, 367 op/s 2024-03-31T23:27:07.274 INFO:tasks.workunit.client.0.smithi046.stdout: 10 21500 7.72 MB/sec execute 12 sec latency 164.481 ms 2024-03-31T23:27:08.274 INFO:tasks.workunit.client.0.smithi046.stdout: 10 21684 8.26 MB/sec execute 13 sec latency 67.404 ms 2024-03-31T23:27:09.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:27:08 smithi162 ceph-mon[36237]: pgmap v485: 129 pgs: 129 active+clean; 519 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 5.3 MiB/s wr, 367 op/s 2024-03-31T23:27:09.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:27:08 smithi046 ceph-mon[32498]: pgmap v485: 129 pgs: 129 active+clean; 519 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 5.3 MiB/s wr, 367 op/s 2024-03-31T23:27:09.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:27:08 smithi142 ceph-mon[35271]: pgmap v485: 129 pgs: 129 active+clean; 519 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 5.3 MiB/s wr, 367 op/s 2024-03-31T23:27:09.274 INFO:tasks.workunit.client.0.smithi046.stdout: 10 21857 8.23 MB/sec execute 14 sec latency 83.070 ms 2024-03-31T23:27:10.013 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:27:09 smithi046 ceph-mon[32498]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:27:10.013 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:27:09 smithi046 ceph-mon[32498]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-31T23:27:10.013 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:27:09 smithi046 ceph-mon[32498]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' 2024-03-31T23:27:10.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:27:09 smithi162 ceph-mon[36237]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:27:10.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:27:09 smithi162 ceph-mon[36237]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-31T23:27:10.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:27:09 smithi162 ceph-mon[36237]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' 2024-03-31T23:27:10.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:27:09 smithi142 ceph-mon[35271]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:27:10.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:27:09 smithi142 ceph-mon[35271]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-31T23:27:10.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:27:09 smithi142 ceph-mon[35271]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' 2024-03-31T23:27:10.274 INFO:tasks.workunit.client.0.smithi046.stdout: 10 22027 8.05 MB/sec execute 15 sec latency 62.115 ms 2024-03-31T23:27:11.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:27:10 smithi162 ceph-mon[36237]: pgmap v486: 129 pgs: 129 active+clean; 536 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 4.7 MiB/s wr, 368 op/s 2024-03-31T23:27:11.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:27:10 smithi162 ceph-mon[36237]: osdmap e117: 12 total, 12 up, 12 in 2024-03-31T23:27:11.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:27:10 smithi046 ceph-mon[32498]: pgmap v486: 129 pgs: 129 active+clean; 536 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 4.7 MiB/s wr, 368 op/s 2024-03-31T23:27:11.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:27:10 smithi046 ceph-mon[32498]: osdmap e117: 12 total, 12 up, 12 in 2024-03-31T23:27:11.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:27:10 smithi142 ceph-mon[35271]: pgmap v486: 129 pgs: 129 active+clean; 536 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 4.7 MiB/s wr, 368 op/s 2024-03-31T23:27:11.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:27:10 smithi142 ceph-mon[35271]: osdmap e117: 12 total, 12 up, 12 in 2024-03-31T23:27:11.274 INFO:tasks.workunit.client.0.smithi046.stdout: 10 22179 7.99 MB/sec execute 16 sec latency 96.358 ms 2024-03-31T23:27:12.197 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:12.197+0000 7fb8d16a4640 1 -- 172.21.15.46:0/3807281910 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb8cc1125d0 msgr2=0x7fb8cc1149c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:27:12.198 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:12.197+0000 7fb8d16a4640 1 --2- 172.21.15.46:0/3807281910 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb8cc1125d0 0x7fb8cc1149c0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fb8c4009890 tx=0x7fb8c402bb70 comp rx=0 tx=0).stop 2024-03-31T23:27:12.198 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:12.197+0000 7fb8d16a4640 1 -- 172.21.15.46:0/3807281910 shutdown_connections 2024-03-31T23:27:12.198 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:12.197+0000 7fb8d16a4640 1 --2- 172.21.15.46:0/3807281910 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb8cc1125d0 0x7fb8cc1149c0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:27:12.198 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:12.197+0000 7fb8d16a4640 1 --2- 172.21.15.46:0/3807281910 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb8cc0765a0 0x7fb8cc112090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:27:12.198 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:12.197+0000 7fb8d16a4640 1 --2- 172.21.15.46:0/3807281910 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb8cc075c80 0x7fb8cc076060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:27:12.198 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:12.197+0000 7fb8d16a4640 1 -- 172.21.15.46:0/3807281910 >> 172.21.15.46:0/3807281910 conn(0x7fb8cc0ff400 msgr2=0x7fb8cc101820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:27:12.198 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:12.197+0000 7fb8d16a4640 1 -- 172.21.15.46:0/3807281910 shutdown_connections 2024-03-31T23:27:12.198 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:12.197+0000 7fb8d16a4640 1 -- 172.21.15.46:0/3807281910 wait complete. 2024-03-31T23:27:12.198 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:12.198+0000 7fb8d16a4640 1 Processor -- start 2024-03-31T23:27:12.200 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:12.198+0000 7fb8d16a4640 1 -- start start 2024-03-31T23:27:12.200 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:12.198+0000 7fb8d16a4640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb8cc075c80 0x7fb8cc19e4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:27:12.200 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:12.198+0000 7fb8d16a4640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb8cc0765a0 0x7fb8cc19ea00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:27:12.200 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:12.198+0000 7fb8d16a4640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb8cc1125d0 0x7fb8cc1a2d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:27:12.200 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:12.198+0000 7fb8d16a4640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fb8cc117fa0 con 0x7fb8cc1125d0 2024-03-31T23:27:12.200 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:12.198+0000 7fb8d16a4640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fb8cc117e20 con 0x7fb8cc0765a0 2024-03-31T23:27:12.200 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:12.198+0000 7fb8d16a4640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fb8cc118120 con 0x7fb8cc075c80 2024-03-31T23:27:12.200 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:12.198+0000 7fb8cb7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb8cc1125d0 0x7fb8cc1a2d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:27:12.200 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:12.198+0000 7fb8caffd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb8cc075c80 0x7fb8cc19e4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:27:12.200 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:12.198+0000 7fb8cb7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb8cc1125d0 0x7fb8cc1a2d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:52016/0 (socket says 172.21.15.46:52016) 2024-03-31T23:27:12.200 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:12.198+0000 7fb8cb7fe640 1 -- 172.21.15.46:0/55953833 learned_addr learned my addr 172.21.15.46:0/55953833 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:27:12.201 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:12.198+0000 7fb8ca7fc640 1 --2- 172.21.15.46:0/55953833 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb8cc0765a0 0x7fb8cc19ea00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:27:12.201 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:12.199+0000 7fb8cb7fe640 1 -- 172.21.15.46:0/55953833 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb8cc075c80 msgr2=0x7fb8cc19e4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:27:12.201 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:12.199+0000 7fb8cb7fe640 1 --2- 172.21.15.46:0/55953833 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb8cc075c80 0x7fb8cc19e4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:27:12.201 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:12.199+0000 7fb8cb7fe640 1 -- 172.21.15.46:0/55953833 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb8cc0765a0 msgr2=0x7fb8cc19ea00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:27:12.201 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:12.199+0000 7fb8cb7fe640 1 --2- 172.21.15.46:0/55953833 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb8cc0765a0 0x7fb8cc19ea00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:27:12.201 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:12.199+0000 7fb8cb7fe640 1 -- 172.21.15.46:0/55953833 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb8cc1a34b0 con 0x7fb8cc1125d0 2024-03-31T23:27:12.201 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:12.199+0000 7fb8cb7fe640 1 --2- 172.21.15.46:0/55953833 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb8cc1125d0 0x7fb8cc1a2d80 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7fb8c4037040 tx=0x7fb8c4035990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:27:12.201 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:12.199+0000 7fb8b3fff640 1 -- 172.21.15.46:0/55953833 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8c4041020 con 0x7fb8cc1125d0 2024-03-31T23:27:12.201 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:12.199+0000 7fb8d16a4640 1 -- 172.21.15.46:0/55953833 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb8cc1a3740 con 0x7fb8cc1125d0 2024-03-31T23:27:12.201 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:12.199+0000 7fb8d16a4640 1 -- 172.21.15.46:0/55953833 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb8cc1ab030 con 0x7fb8cc1125d0 2024-03-31T23:27:12.201 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:12.200+0000 7fb8b3fff640 1 -- 172.21.15.46:0/55953833 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb8c4035d00 con 0x7fb8cc1125d0 2024-03-31T23:27:12.201 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:12.200+0000 7fb8b3fff640 1 -- 172.21.15.46:0/55953833 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8c4039040 con 0x7fb8cc1125d0 2024-03-31T23:27:12.203 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:12.203+0000 7fb8d4366740 1 Processor -- start 2024-03-31T23:27:12.204 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:12.204+0000 7fb8b3fff640 1 -- 172.21.15.46:0/55953833 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 25) ==== 137451+0+0 (secure 0 0 0) 0x7fb8c4044020 con 0x7fb8cc1125d0 2024-03-31T23:27:12.204 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:12.204+0000 7fb8d4366740 1 -- start start 2024-03-31T23:27:12.206 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:12.204+0000 7fb8b3fff640 1 --2- 172.21.15.46:0/55953833 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fb8b40a2780 0x7fb8b40a4c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:27:12.206 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:12.204+0000 7fb8d4366740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557207c2b6c0 0x557207c2baa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:27:12.206 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:12.205+0000 7fb8d4366740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557207c2bfe0 0x557207c34870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:27:12.206 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:12.205+0000 7fb8d4366740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557207c34db0 0x557207c371a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:27:12.206 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:12.205+0000 7fb8d4366740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x557207b8e8f0 con 0x557207c2b6c0 2024-03-31T23:27:12.206 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:12.205+0000 7fb8d4366740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x557207b378d0 con 0x557207c2bfe0 2024-03-31T23:27:12.206 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:12.205+0000 7fb8d4366740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x557207b8a990 con 0x557207c34db0 2024-03-31T23:27:12.206 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:12.205+0000 7fb8b3fff640 1 -- 172.21.15.46:0/55953833 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(117..117 src has 1..117) ==== 9554+0+0 (secure 0 0 0) 0x7fb8c40f0400 con 0x7fb8cc1125d0 2024-03-31T23:27:12.206 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:12.205+0000 7fb8caffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557207c2b6c0 0x557207c2baa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:27:12.206 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:12.205+0000 7fb8caffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557207c2b6c0 0x557207c2baa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:52020/0 (socket says 172.21.15.46:52020) 2024-03-31T23:27:12.206 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:12.205+0000 7fb8caffd640 1 -- 172.21.15.46:0/2550645803 learned_addr learned my addr 172.21.15.46:0/2550645803 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:27:12.206 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:12.205+0000 7fb8ca7fc640 1 --2- 172.21.15.46:0/55953833 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fb8b40a2780 0x7fb8b40a4c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:27:12.207 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:12.205+0000 7fb8caffd640 1 -- 172.21.15.46:0/2550645803 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557207c34db0 msgr2=0x557207c371a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:27:12.207 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:12.205+0000 7fb8cb7fe640 1 --2- 172.21.15.46:0/2550645803 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557207c34db0 0x557207c371a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:27:12.207 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:12.205+0000 7fb8caffd640 1 --2- 172.21.15.46:0/2550645803 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557207c34db0 0x557207c371a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:27:12.207 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:12.205+0000 7fb8caffd640 1 -- 172.21.15.46:0/2550645803 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557207c2bfe0 msgr2=0x557207c34870 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-31T23:27:12.207 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:12.205+0000 7fb8caffd640 1 --2- 172.21.15.46:0/2550645803 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557207c2bfe0 0x557207c34870 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:27:12.207 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:12.205+0000 7fb8caffd640 1 -- 172.21.15.46:0/2550645803 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5572078a3210 con 0x557207c2b6c0 2024-03-31T23:27:12.207 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:12.206+0000 7fb8caffd640 1 --2- 172.21.15.46:0/2550645803 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557207c2b6c0 0x557207c2baa0 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7fb8c0009cd0 tx=0x7fb8c0011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=50233226acd78b6d server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:27:12.207 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:12.206+0000 7fb8b17fa640 1 -- 172.21.15.46:0/2550645803 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8c001e070 con 0x557207c2b6c0 2024-03-31T23:27:12.207 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:12.206+0000 7fb8b17fa640 1 -- 172.21.15.46:0/2550645803 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb8c0019ae0 con 0x557207c2b6c0 2024-03-31T23:27:12.207 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:12.206+0000 7fb8b17fa640 1 -- 172.21.15.46:0/2550645803 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8c000e970 con 0x557207c2b6c0 2024-03-31T23:27:12.210 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:12.207+0000 7fb8d4366740 1 -- 172.21.15.46:0/2550645803 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557207c2b6c0 msgr2=0x557207c2baa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:27:12.210 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:12.207+0000 7fb8d4366740 1 --2- 172.21.15.46:0/2550645803 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557207c2b6c0 0x557207c2baa0 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7fb8c0009cd0 tx=0x7fb8c0011810 comp rx=0 tx=0).stop 2024-03-31T23:27:12.210 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:12.207+0000 7fb8ca7fc640 1 --2- 172.21.15.46:0/55953833 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fb8b40a2780 0x7fb8b40a4c40 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x557207c2b170 tx=0x7fb8bc009500 comp rx=0 tx=0).ready entity=mgr.15654 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:27:12.210 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:12.207+0000 7fb8d4366740 1 -- 172.21.15.46:0/2550645803 shutdown_connections 2024-03-31T23:27:12.210 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:12.207+0000 7fb8d4366740 1 --2- 172.21.15.46:0/2550645803 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557207c34db0 0x557207c371a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:27:12.211 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:12.207+0000 7fb8d4366740 1 --2- 172.21.15.46:0/2550645803 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557207c2bfe0 0x557207c34870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:27:12.211 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:12.207+0000 7fb8d4366740 1 --2- 172.21.15.46:0/2550645803 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557207c2b6c0 0x557207c2baa0 unknown :-1 s=CLOSED pgs=901 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:27:12.211 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:12.207+0000 7fb8d4366740 1 -- 172.21.15.46:0/2550645803 >> 172.21.15.46:0/2550645803 conn(0x557207c25fe0 msgr2=0x557207c263c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:27:12.211 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:12.207+0000 7fb8d4366740 1 -- 172.21.15.46:0/2550645803 shutdown_connections 2024-03-31T23:27:12.211 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:12.207+0000 7fb8d4366740 1 -- 172.21.15.46:0/2550645803 wait complete. 2024-03-31T23:27:12.211 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:12.208+0000 7fb8d4366740 1 -- start start 2024-03-31T23:27:12.211 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:12.208+0000 7fb8d4366740 1 Processor -- start 2024-03-31T23:27:12.211 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:12.208+0000 7fb8d4366740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557207c2b6c0 0x557207c31d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:27:12.211 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:12.208+0000 7fb8d4366740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557207c2bfe0 0x557207c32270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:27:12.211 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:12.208+0000 7fb8d4366740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557207c34db0 0x557207c327b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:27:12.211 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:12.208+0000 7fb8d4366740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x557207c379e0 con 0x557207c34db0 2024-03-31T23:27:12.211 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:12.208+0000 7fb8d4366740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x557207b2cf80 con 0x557207c2b6c0 2024-03-31T23:27:12.211 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:12.208+0000 7fb8d4366740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x557207c37b60 con 0x557207c2bfe0 2024-03-31T23:27:12.211 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:12.208+0000 7fb8caffd640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557207c2b6c0 0x557207c31d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:27:12.212 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:12.208+0000 7fb8caffd640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557207c2b6c0 0x557207c31d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.46:44122/0 (socket says 172.21.15.46:44122) 2024-03-31T23:27:12.212 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:12.208+0000 7fb8caffd640 1 -- 172.21.15.46:0/1104683387 learned_addr learned my addr 172.21.15.46:0/1104683387 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:27:12.212 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:12.208+0000 7fb8d4366740 10 client.0 _ll_register_callbacks cb 0x557207b85290 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T23:27:12.212 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:12.208+0000 7fb8cb7fe640 1 --2- 172.21.15.46:0/1104683387 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557207c34db0 0x557207c327b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:27:12.212 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:12.209+0000 7fb8ca7fc640 1 --2- 172.21.15.46:0/1104683387 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557207c2bfe0 0x557207c32270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:27:12.212 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:12.209+0000 7fb8cb7fe640 1 -- 172.21.15.46:0/1104683387 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557207c2bfe0 msgr2=0x557207c32270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:27:12.212 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:12.209+0000 7fb8cb7fe640 1 --2- 172.21.15.46:0/1104683387 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557207c2bfe0 0x557207c32270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:27:12.212 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:12.209+0000 7fb8cb7fe640 1 -- 172.21.15.46:0/1104683387 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557207c2b6c0 msgr2=0x557207c31d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:27:12.212 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:12.209+0000 7fb8cb7fe640 1 --2- 172.21.15.46:0/1104683387 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557207c2b6c0 0x557207c31d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:27:12.213 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:12.209+0000 7fb8cb7fe640 1 -- 172.21.15.46:0/1104683387 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55720796f620 con 0x557207c34db0 2024-03-31T23:27:12.213 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:12.209+0000 7fb8caffd640 1 --2- 172.21.15.46:0/1104683387 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557207c2b6c0 0x557207c31d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:27:12.213 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:12.209+0000 7fb8cb7fe640 1 --2- 172.21.15.46:0/1104683387 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557207c34db0 0x557207c327b0 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7fb8c40ed7f0 tx=0x7fb8c40bc6b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f3a5aaf034604714 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:27:12.213 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:12.209+0000 7fb8aeff5640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-31T23:27:12.213 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:12.209+0000 7fb8aeff5640 1 -- 172.21.15.46:0/1104683387 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8c404d4e0 con 0x557207c34db0 2024-03-31T23:27:12.213 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:12.209+0000 7fb8aeff5640 1 -- 172.21.15.46:0/1104683387 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(117..117 src has 1..117) ==== 9554+0+0 (secure 0 0 0) 0x7fb8c40f0170 con 0x557207c34db0 2024-03-31T23:27:12.213 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:12.209+0000 7fb8d4366740 1 -- 172.21.15.46:0/1104683387 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb8c0019ae0 con 0x557207c34db0 2024-03-31T23:27:12.213 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:12.210+0000 7fb8aeff5640 1 -- 172.21.15.46:0/1104683387 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7fb8c40bcf20 con 0x557207c34db0 2024-03-31T23:27:12.213 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:12.210+0000 7fb8d4366740 10 client.17406 fetch_fsmap learned FSMap version 40 2024-03-31T23:27:12.213 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:12.210+0000 7fb8d4366740 1 -- 172.21.15.46:0/1104683387 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x557207c43840 con 0x557207c34db0 2024-03-31T23:27:12.213 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:12.210+0000 7fb8aeff5640 1 -- 172.21.15.46:0/1104683387 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb8c40bc820 con 0x557207c34db0 2024-03-31T23:27:12.213 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:12.210+0000 7fb8aeff5640 1 -- 172.21.15.46:0/1104683387 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 40) ==== 8370+0+0 (secure 0 0 0) 0x7fb8c403c030 con 0x557207c34db0 2024-03-31T23:27:12.213 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:12.210+0000 7fb8d4366740 10 client.17406 fetch_fsmap finished waiting for FSMap version 40 2024-03-31T23:27:12.213 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:12.210+0000 7fb8d4366740 10 client.17406 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-31T23:27:12.214 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:12.210+0000 7fb8d4366740 20 client.17406 populate_metadata read hostname 'smithi046' 2024-03-31T23:27:12.214 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:12.210+0000 7fb8d4366740 1 --2- 172.21.15.46:0/1104683387 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x557207c44eb0 0x557207c47350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:27:12.214 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:12.210+0000 7fb8d4366740 4 client.17406 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-31T23:27:12.214 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:12.210+0000 7fb8d4366740 1 -- 172.21.15.46:0/1104683387 --> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fb8c0019ae0 con 0x557207c44eb0 2024-03-31T23:27:12.215 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:12.211+0000 7fb8caffd640 1 --2- 172.21.15.46:0/1104683387 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x557207c44eb0 0x557207c47350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:27:12.215 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:12.211+0000 7fb8caffd640 1 --2- 172.21.15.46:0/1104683387 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x557207c44eb0 0x557207c47350 crc :-1 s=READY pgs=214 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=80b4aa3827562bc0 server_cookie=b50f647336dceacd in_seq=0 out_seq=0 2024-03-31T23:27:12.215 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:12.211+0000 7fb8aeff5640 10 client.17406 ms_handle_connect on v2:172.21.15.142:6838/2519143918 2024-03-31T23:27:12.215 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:12.212+0000 7fb8aeff5640 1 -- 172.21.15.46:0/1104683387 <== mds.0 v2:172.21.15.142:6838/2519143918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x7fb8c0019ae0 con 0x557207c44eb0 2024-03-31T23:27:12.215 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:12.212+0000 7fb8aeff5640 10 client.17406 handle_command_reply: tid=0 2024-03-31T23:27:12.215 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:12.212+0000 7fb8d4366740 1 client.17406 shutdown 2024-03-31T23:27:12.215 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:12.212+0000 7fb8d4366740 1 -- 172.21.15.46:0/1104683387 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557207c34db0 msgr2=0x557207c327b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:27:12.216 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:12.212+0000 7fb8d4366740 1 --2- 172.21.15.46:0/1104683387 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557207c34db0 0x557207c327b0 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7fb8c40ed7f0 tx=0x7fb8c40bc6b0 comp rx=0 tx=0).stop 2024-03-31T23:27:12.216 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:12.212+0000 7fb8d4366740 1 -- 172.21.15.46:0/1104683387 shutdown_connections 2024-03-31T23:27:12.216 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:12.212+0000 7fb8d4366740 1 --2- 172.21.15.46:0/1104683387 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x557207c44eb0 0x557207c47350 crc :-1 s=READY pgs=214 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:27:12.216 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:12.212+0000 7fb8d4366740 1 --2- 172.21.15.46:0/1104683387 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557207c34db0 0x557207c327b0 unknown :-1 s=CLOSED pgs=902 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:27:12.216 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:12.212+0000 7fb8d4366740 1 --2- 172.21.15.46:0/1104683387 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557207c2bfe0 0x557207c32270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:27:12.216 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:12.212+0000 7fb8d4366740 1 --2- 172.21.15.46:0/1104683387 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557207c2b6c0 0x557207c31d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:27:12.216 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:12.212+0000 7fb8d4366740 1 -- 172.21.15.46:0/1104683387 >> 172.21.15.46:0/1104683387 conn(0x557207c25fe0 msgr2=0x557207c27d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:27:12.216 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:12.212+0000 7fb8aeff5640 0 client.17406 ms_handle_reset on v2:172.21.15.142:6838/2519143918 2024-03-31T23:27:12.216 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:12.212+0000 7fb8d4366740 1 -- 172.21.15.46:0/1104683387 shutdown_connections 2024-03-31T23:27:12.216 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:12.212+0000 7fb8d4366740 1 -- 172.21.15.46:0/1104683387 wait complete. 2024-03-31T23:27:12.216 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:12.213+0000 7fb8d4366740 20 client.17406 trim_cache size 0 max 16384 2024-03-31T23:27:12.243 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:12.241+0000 7fb8d4366740 1 Processor -- start 2024-03-31T23:27:12.244 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:12.241+0000 7fb8d4366740 1 -- start start 2024-03-31T23:27:12.244 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:12.242+0000 7fb8d4366740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557207c2b6c0 0x557207c343f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:27:12.244 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:12.242+0000 7fb8d4366740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557207c2bfe0 0x557207c407b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:27:12.244 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:12.242+0000 7fb8d4366740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557207c34db0 0x557207c4cc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:27:12.244 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:12.242+0000 7fb8d4366740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fb8b8012a70 con 0x557207c34db0 2024-03-31T23:27:12.244 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:12.242+0000 7fb8d4366740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fb8b8010220 con 0x557207c2b6c0 2024-03-31T23:27:12.244 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:12.242+0000 7fb8d4366740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fb8b800e090 con 0x557207c2bfe0 2024-03-31T23:27:12.244 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:12.242+0000 7fb8cb7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557207c34db0 0x557207c4cc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:27:12.244 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:12.242+0000 7fb8cb7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557207c34db0 0x557207c4cc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:52038/0 (socket says 172.21.15.46:52038) 2024-03-31T23:27:12.244 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:12.242+0000 7fb8cb7fe640 1 -- 172.21.15.46:0/4250216035 learned_addr learned my addr 172.21.15.46:0/4250216035 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:27:12.244 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:12.242+0000 7fb8ca7fc640 1 --2- 172.21.15.46:0/4250216035 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557207c2bfe0 0x557207c407b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:27:12.244 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:12.242+0000 7fb8caffd640 1 --2- 172.21.15.46:0/4250216035 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557207c2b6c0 0x557207c343f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:27:12.245 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:12.242+0000 7fb8ca7fc640 1 -- 172.21.15.46:0/4250216035 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557207c2b6c0 msgr2=0x557207c343f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:27:12.245 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:12.242+0000 7fb8ca7fc640 1 --2- 172.21.15.46:0/4250216035 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557207c2b6c0 0x557207c343f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:27:12.245 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:12.242+0000 7fb8ca7fc640 1 -- 172.21.15.46:0/4250216035 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557207c34db0 msgr2=0x557207c4cc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:27:12.245 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:12.242+0000 7fb8ca7fc640 1 --2- 172.21.15.46:0/4250216035 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557207c34db0 0x557207c4cc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:27:12.245 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:12.242+0000 7fb8ca7fc640 1 -- 172.21.15.46:0/4250216035 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557207c30280 con 0x557207c2bfe0 2024-03-31T23:27:12.245 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:12.243+0000 7fb8cb7fe640 1 --2- 172.21.15.46:0/4250216035 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557207c34db0 0x557207c4cc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:27:12.245 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:12.243+0000 7fb8caffd640 1 --2- 172.21.15.46:0/4250216035 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557207c2b6c0 0x557207c343f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:27:12.245 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:12.243+0000 7fb8ca7fc640 1 --2- 172.21.15.46:0/4250216035 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557207c2bfe0 0x557207c407b0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x557207c33300 tx=0x7fb8bc014000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=812efbfee28a631e server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:27:12.245 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:12.243+0000 7fb8b0ff9640 1 -- 172.21.15.46:0/4250216035 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8bc028440 con 0x557207c2bfe0 2024-03-31T23:27:12.245 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:12.243+0000 7fb8b0ff9640 1 -- 172.21.15.46:0/4250216035 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb8bc028a90 con 0x557207c2bfe0 2024-03-31T23:27:12.245 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:12.243+0000 7fb8b0ff9640 1 -- 172.21.15.46:0/4250216035 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8bc01e9c0 con 0x557207c2bfe0 2024-03-31T23:27:12.248 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:12.244+0000 7fb8d4366740 1 -- 172.21.15.46:0/4250216035 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557207c2bfe0 msgr2=0x557207c407b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:27:12.248 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:12.244+0000 7fb8d4366740 1 --2- 172.21.15.46:0/4250216035 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557207c2bfe0 0x557207c407b0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x557207c33300 tx=0x7fb8bc014000 comp rx=0 tx=0).stop 2024-03-31T23:27:12.248 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:12.244+0000 7fb8d4366740 1 -- 172.21.15.46:0/4250216035 shutdown_connections 2024-03-31T23:27:12.249 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:12.244+0000 7fb8d4366740 1 --2- 172.21.15.46:0/4250216035 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557207c34db0 0x557207c4cc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:27:12.249 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:12.244+0000 7fb8d4366740 1 --2- 172.21.15.46:0/4250216035 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557207c2bfe0 0x557207c407b0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:27:12.249 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:12.244+0000 7fb8d4366740 1 --2- 172.21.15.46:0/4250216035 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557207c2b6c0 0x557207c343f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:27:12.249 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:12.244+0000 7fb8d4366740 1 -- 172.21.15.46:0/4250216035 >> 172.21.15.46:0/4250216035 conn(0x557207c25fe0 msgr2=0x557207c2a9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:27:12.249 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:12.244+0000 7fb8d4366740 1 -- 172.21.15.46:0/4250216035 shutdown_connections 2024-03-31T23:27:12.249 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:12.244+0000 7fb8d4366740 1 -- 172.21.15.46:0/4250216035 wait complete. 2024-03-31T23:27:12.249 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:12.245+0000 7fb8d4366740 1 -- start start 2024-03-31T23:27:12.249 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:12.245+0000 7fb8d4366740 1 Processor -- start 2024-03-31T23:27:12.249 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:12.245+0000 7fb8d4366740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557207c2b6c0 0x557207c3cde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:27:12.249 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:12.245+0000 7fb8d4366740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557207c2bfe0 0x557207c2e160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:27:12.249 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:12.245+0000 7fb8d4366740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557207c34db0 0x557207c2e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:27:12.249 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:12.245+0000 7fb8d4366740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fb8b800e6e0 con 0x557207c34db0 2024-03-31T23:27:12.249 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:12.245+0000 7fb8d4366740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fb8b800fb50 con 0x557207c2b6c0 2024-03-31T23:27:12.250 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:12.245+0000 7fb8d4366740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fb8b800edb0 con 0x557207c2bfe0 2024-03-31T23:27:12.250 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:12.245+0000 7fb8d4366740 10 client.0 _ll_register_callbacks cb 0x7fb8c4009490 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T23:27:12.250 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:12.245+0000 7fb8caffd640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557207c2b6c0 0x557207c3cde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:27:12.250 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:12.245+0000 7fb8caffd640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557207c2b6c0 0x557207c3cde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.46:44140/0 (socket says 172.21.15.46:44140) 2024-03-31T23:27:12.250 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:12.245+0000 7fb8caffd640 1 -- 172.21.15.46:0/15142923 learned_addr learned my addr 172.21.15.46:0/15142923 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:27:12.250 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:12.245+0000 7fb8cb7fe640 1 --2- 172.21.15.46:0/15142923 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557207c34db0 0x557207c2e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:27:12.250 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:12.245+0000 7fb8ca7fc640 1 --2- 172.21.15.46:0/15142923 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557207c2bfe0 0x557207c2e160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:27:12.250 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:12.246+0000 7fb8caffd640 1 -- 172.21.15.46:0/15142923 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557207c2bfe0 msgr2=0x557207c2e160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:27:12.250 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:12.246+0000 7fb8caffd640 1 --2- 172.21.15.46:0/15142923 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557207c2bfe0 0x557207c2e160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:27:12.250 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:12.246+0000 7fb8caffd640 1 -- 172.21.15.46:0/15142923 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557207c34db0 msgr2=0x557207c2e6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:27:12.250 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:12.246+0000 7fb8caffd640 1 --2- 172.21.15.46:0/15142923 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557207c34db0 0x557207c2e6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:27:12.250 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:12.246+0000 7fb8caffd640 1 -- 172.21.15.46:0/15142923 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557207c2ebe0 con 0x557207c2b6c0 2024-03-31T23:27:12.250 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:12.246+0000 7fb8cb7fe640 1 --2- 172.21.15.46:0/15142923 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557207c34db0 0x557207c2e6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:27:12.250 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:12.246+0000 7fb8caffd640 1 --2- 172.21.15.46:0/15142923 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557207c2b6c0 0x557207c3cde0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fb8c0009c20 tx=0x7fb8c0009490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c571e2edda0e7c2b server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:27:12.251 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:12.246+0000 7fb8aeff5640 10 client.0 ms_handle_connect on v2:172.21.15.142:3300/0 2024-03-31T23:27:12.251 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:12.246+0000 7fb8aeff5640 1 -- 172.21.15.46:0/15142923 <== mon.2 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8c001e040 con 0x557207c2b6c0 2024-03-31T23:27:12.251 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:12.246+0000 7fb8d4366740 1 -- 172.21.15.46:0/15142923 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb8bc028a90 con 0x557207c2b6c0 2024-03-31T23:27:12.251 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:12.246+0000 7fb8aeff5640 1 -- 172.21.15.46:0/15142923 <== mon.2 v2:172.21.15.142:3300/0 2 ==== osd_map(117..117 src has 1..117) ==== 9554+0+0 (secure 0 0 0) 0x7fb8c00035f0 con 0x557207c2b6c0 2024-03-31T23:27:12.251 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:12.246+0000 7fb8aeff5640 1 -- 172.21.15.46:0/15142923 <== mon.2 v2:172.21.15.142:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7fb8c0037050 con 0x557207c2b6c0 2024-03-31T23:27:12.251 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:12.246+0000 7fb8d4366740 10 client.27239 fetch_fsmap learned FSMap version 40 2024-03-31T23:27:12.251 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:12.246+0000 7fb8d4366740 1 -- 172.21.15.46:0/15142923 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x557207caade0 con 0x557207c2b6c0 2024-03-31T23:27:12.251 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:12.247+0000 7fb8aeff5640 1 -- 172.21.15.46:0/15142923 <== mon.2 v2:172.21.15.142:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb8bc028a90 con 0x557207c2b6c0 2024-03-31T23:27:12.251 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:12.247+0000 7fb8aeff5640 1 -- 172.21.15.46:0/15142923 <== mon.2 v2:172.21.15.142:3300/0 5 ==== fsmap(e 40) ==== 8370+0+0 (secure 0 0 0) 0x7fb8c0019730 con 0x557207c2b6c0 2024-03-31T23:27:12.251 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:12.247+0000 7fb8d4366740 10 client.27239 fetch_fsmap finished waiting for FSMap version 40 2024-03-31T23:27:12.251 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:12.247+0000 7fb8d4366740 10 client.27239 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-31T23:27:12.251 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:12.247+0000 7fb8d4366740 20 client.27239 populate_metadata read hostname 'smithi046' 2024-03-31T23:27:12.251 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:12.247+0000 7fb8d4366740 1 --2- 172.21.15.46:0/15142923 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x557207cab3a0 0x557207cad7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:27:12.251 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:12.247+0000 7fb8d4366740 4 client.27239 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-31T23:27:12.252 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:12.247+0000 7fb8d4366740 1 -- 172.21.15.46:0/15142923 --> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] -- command(tid 0: {"prefix": "flush journal"}) -- 0x557207ba2120 con 0x557207cab3a0 2024-03-31T23:27:12.252 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:12.247+0000 7fb8ca7fc640 1 --2- 172.21.15.46:0/15142923 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x557207cab3a0 0x557207cad7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:27:12.252 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:12.248+0000 7fb8aeff5640 10 client.27239 ms_handle_connect on v2:172.21.15.142:6838/2519143918 2024-03-31T23:27:12.252 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:12.248+0000 7fb8ca7fc640 1 --2- 172.21.15.46:0/15142923 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x557207cab3a0 0x557207cad7c0 crc :-1 s=READY pgs=215 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=44f6d5703e934fb6 server_cookie=8bf87203182ad8fc in_seq=0 out_seq=0 2024-03-31T23:27:12.274 INFO:tasks.workunit.client.0.smithi046.stdout: 10 22379 8.39 MB/sec execute 17 sec latency 81.931 ms 2024-03-31T23:27:12.397 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:12.395+0000 7fb8aeff5640 1 -- 172.21.15.46:0/15142923 <== mds.0 v2:172.21.15.142:6838/2519143918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x557207ba2120 con 0x557207cab3a0 2024-03-31T23:27:12.397 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:12.395+0000 7fb8aeff5640 10 client.27239 handle_command_reply: tid=0 2024-03-31T23:27:12.397 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:12.395+0000 7fb8d4366740 1 client.27239 shutdown 2024-03-31T23:27:12.397 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:12.395+0000 7fb8d4366740 1 -- 172.21.15.46:0/15142923 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557207c2b6c0 msgr2=0x557207c3cde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:27:12.397 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:12.395+0000 7fb8d4366740 1 --2- 172.21.15.46:0/15142923 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557207c2b6c0 0x557207c3cde0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fb8c0009c20 tx=0x7fb8c0009490 comp rx=0 tx=0).stop 2024-03-31T23:27:12.397 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:12.396+0000 7fb8d4366740 1 -- 172.21.15.46:0/15142923 shutdown_connections 2024-03-31T23:27:12.397 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:12.396+0000 7fb8d4366740 1 --2- 172.21.15.46:0/15142923 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x557207cab3a0 0x557207cad7c0 crc :-1 s=READY pgs=215 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:27:12.398 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:12.396+0000 7fb8d4366740 1 --2- 172.21.15.46:0/15142923 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557207c34db0 0x557207c2e6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:27:12.398 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:12.396+0000 7fb8aeff5640 0 client.27239 ms_handle_reset on v2:172.21.15.142:6838/2519143918 2024-03-31T23:27:12.398 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:12.396+0000 7fb8d4366740 1 --2- 172.21.15.46:0/15142923 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557207c2bfe0 0x557207c2e160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:27:12.398 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:12.396+0000 7fb8d4366740 1 --2- 172.21.15.46:0/15142923 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557207c2b6c0 0x557207c3cde0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:27:12.398 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:12.396+0000 7fb8d4366740 1 -- 172.21.15.46:0/15142923 >> 172.21.15.46:0/15142923 conn(0x557207c25fe0 msgr2=0x557207c2d090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:27:12.398 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:12.396+0000 7fb8d4366740 1 -- 172.21.15.46:0/15142923 shutdown_connections 2024-03-31T23:27:12.398 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:12.396+0000 7fb8d4366740 1 -- 172.21.15.46:0/15142923 wait complete. 2024-03-31T23:27:12.398 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:12.396+0000 7fb8d4366740 20 client.27239 trim_cache size 0 max 16384 2024-03-31T23:27:12.398 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout:{ 2024-03-31T23:27:12.398 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout: "message": "", 2024-03-31T23:27:12.399 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout: "return_code": 0 2024-03-31T23:27:12.399 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout:} 2024-03-31T23:27:12.399 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:12.396+0000 7fb8d16a4640 1 -- 172.21.15.46:0/55953833 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fb8b40a2780 msgr2=0x7fb8b40a4c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:27:12.399 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:12.396+0000 7fb8d16a4640 1 --2- 172.21.15.46:0/55953833 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fb8b40a2780 0x7fb8b40a4c40 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x557207c2b170 tx=0x7fb8bc009500 comp rx=0 tx=0).stop 2024-03-31T23:27:12.399 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:12.396+0000 7fb8d16a4640 1 -- 172.21.15.46:0/55953833 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb8cc1125d0 msgr2=0x7fb8cc1a2d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:27:12.399 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:12.396+0000 7fb8d16a4640 1 --2- 172.21.15.46:0/55953833 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb8cc1125d0 0x7fb8cc1a2d80 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7fb8c4037040 tx=0x7fb8c4035990 comp rx=0 tx=0).stop 2024-03-31T23:27:12.400 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:12.397+0000 7fb8d16a4640 1 -- 172.21.15.46:0/55953833 shutdown_connections 2024-03-31T23:27:12.400 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:12.397+0000 7fb8d16a4640 1 --2- 172.21.15.46:0/55953833 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fb8b40a2780 0x7fb8b40a4c40 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:27:12.400 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:12.397+0000 7fb8d16a4640 1 --2- 172.21.15.46:0/55953833 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb8cc1125d0 0x7fb8cc1a2d80 unknown :-1 s=CLOSED pgs=900 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:27:12.400 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:12.397+0000 7fb8d16a4640 1 --2- 172.21.15.46:0/55953833 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb8cc0765a0 0x7fb8cc19ea00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:27:12.400 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:12.397+0000 7fb8d16a4640 1 --2- 172.21.15.46:0/55953833 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb8cc075c80 0x7fb8cc19e4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:27:12.400 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:12.397+0000 7fb8d16a4640 1 -- 172.21.15.46:0/55953833 >> 172.21.15.46:0/55953833 conn(0x7fb8cc0ff400 msgr2=0x7fb8cc113240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:27:12.400 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:12.397+0000 7fb8d16a4640 1 -- 172.21.15.46:0/55953833 shutdown_connections 2024-03-31T23:27:12.400 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:12.397+0000 7fb8d16a4640 1 -- 172.21.15.46:0/55953833 wait complete. 2024-03-31T23:27:13.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:27:12 smithi162 ceph-mon[36237]: pgmap v488: 129 pgs: 129 active+clean; 536 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 5.8 MiB/s wr, 450 op/s 2024-03-31T23:27:13.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:27:12 smithi046 ceph-mon[32498]: pgmap v488: 129 pgs: 129 active+clean; 536 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 5.8 MiB/s wr, 450 op/s 2024-03-31T23:27:13.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:27:12 smithi142 ceph-mon[35271]: pgmap v488: 129 pgs: 129 active+clean; 536 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 5.8 MiB/s wr, 450 op/s 2024-03-31T23:27:13.274 INFO:tasks.workunit.client.0.smithi046.stdout: 10 22544 8.32 MB/sec execute 18 sec latency 136.530 ms 2024-03-31T23:27:14.274 INFO:tasks.workunit.client.0.smithi046.stdout: 10 22695 8.41 MB/sec execute 19 sec latency 73.935 ms 2024-03-31T23:27:15.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:27:14 smithi162 ceph-mon[36237]: pgmap v489: 129 pgs: 129 active+clean; 536 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 4.7 MiB/s wr, 368 op/s 2024-03-31T23:27:15.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:27:14 smithi046 ceph-mon[32498]: pgmap v489: 129 pgs: 129 active+clean; 536 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 4.7 MiB/s wr, 368 op/s 2024-03-31T23:27:15.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:27:14 smithi142 ceph-mon[35271]: pgmap v489: 129 pgs: 129 active+clean; 536 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 4.7 MiB/s wr, 368 op/s 2024-03-31T23:27:15.274 INFO:tasks.workunit.client.0.smithi046.stdout: 10 22865 8.62 MB/sec execute 20 sec latency 70.641 ms 2024-03-31T23:27:16.274 INFO:tasks.workunit.client.0.smithi046.stdout: 10 23054 8.84 MB/sec execute 21 sec latency 200.082 ms 2024-03-31T23:27:17.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:27:16 smithi162 ceph-mon[36237]: pgmap v490: 129 pgs: 129 active+clean; 543 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 4.7 MiB/s wr, 327 op/s 2024-03-31T23:27:17.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:27:16 smithi046 ceph-mon[32498]: pgmap v490: 129 pgs: 129 active+clean; 543 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 4.7 MiB/s wr, 327 op/s 2024-03-31T23:27:17.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:27:16 smithi142 ceph-mon[35271]: pgmap v490: 129 pgs: 129 active+clean; 543 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 4.7 MiB/s wr, 327 op/s 2024-03-31T23:27:17.274 INFO:tasks.workunit.client.0.smithi046.stdout: 10 23220 8.93 MB/sec execute 22 sec latency 73.997 ms 2024-03-31T23:27:18.274 INFO:tasks.workunit.client.0.smithi046.stdout: 10 23357 8.83 MB/sec execute 23 sec latency 103.116 ms 2024-03-31T23:27:19.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:27:18 smithi162 ceph-mon[36237]: pgmap v491: 129 pgs: 129 active+clean; 543 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 4.7 MiB/s wr, 327 op/s 2024-03-31T23:27:19.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:27:18 smithi046 ceph-mon[32498]: pgmap v491: 129 pgs: 129 active+clean; 543 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 4.7 MiB/s wr, 327 op/s 2024-03-31T23:27:19.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:27:18 smithi142 ceph-mon[35271]: pgmap v491: 129 pgs: 129 active+clean; 543 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 4.7 MiB/s wr, 327 op/s 2024-03-31T23:27:19.274 INFO:tasks.workunit.client.0.smithi046.stdout: 10 23506 8.81 MB/sec execute 24 sec latency 75.247 ms 2024-03-31T23:27:20.274 INFO:tasks.workunit.client.0.smithi046.stdout: 10 23655 8.73 MB/sec execute 25 sec latency 124.718 ms 2024-03-31T23:27:21.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:27:20 smithi162 ceph-mon[36237]: pgmap v492: 129 pgs: 129 active+clean; 555 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 5.4 MiB/s wr, 319 op/s 2024-03-31T23:27:21.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:27:20 smithi046 ceph-mon[32498]: pgmap v492: 129 pgs: 129 active+clean; 555 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 5.4 MiB/s wr, 319 op/s 2024-03-31T23:27:21.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:27:20 smithi142 ceph-mon[35271]: pgmap v492: 129 pgs: 129 active+clean; 555 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 5.4 MiB/s wr, 319 op/s 2024-03-31T23:27:21.274 INFO:tasks.workunit.client.0.smithi046.stdout: 10 23796 8.52 MB/sec execute 26 sec latency 60.923 ms 2024-03-31T23:27:22.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:27:21 smithi162 ceph-mon[36237]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-03-31T23:27:22.094 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:27:21 smithi142 ceph-mon[35271]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-03-31T23:27:22.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:27:21 smithi046 ceph-mon[32498]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-03-31T23:27:22.274 INFO:tasks.workunit.client.0.smithi046.stdout: 10 23956 8.50 MB/sec execute 27 sec latency 97.457 ms 2024-03-31T23:27:23.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:27:22 smithi162 ceph-mon[36237]: pgmap v493: 129 pgs: 129 active+clean; 555 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 5.2 MiB/s wr, 309 op/s 2024-03-31T23:27:23.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:27:22 smithi046 ceph-mon[32498]: pgmap v493: 129 pgs: 129 active+clean; 555 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 5.2 MiB/s wr, 309 op/s 2024-03-31T23:27:23.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:27:22 smithi142 ceph-mon[35271]: pgmap v493: 129 pgs: 129 active+clean; 555 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 5.2 MiB/s wr, 309 op/s 2024-03-31T23:27:23.275 INFO:tasks.workunit.client.0.smithi046.stdout: 10 24118 8.54 MB/sec execute 28 sec latency 82.967 ms 2024-03-31T23:27:24.275 INFO:tasks.workunit.client.0.smithi046.stdout: 10 24300 8.57 MB/sec execute 29 sec latency 81.063 ms 2024-03-31T23:27:25.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:27:24 smithi162 ceph-mon[36237]: pgmap v494: 129 pgs: 129 active+clean; 555 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 4.5 MiB/s wr, 266 op/s 2024-03-31T23:27:25.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:27:24 smithi046 ceph-mon[32498]: pgmap v494: 129 pgs: 129 active+clean; 555 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 4.5 MiB/s wr, 266 op/s 2024-03-31T23:27:25.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:27:24 smithi142 ceph-mon[35271]: pgmap v494: 129 pgs: 129 active+clean; 555 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 4.5 MiB/s wr, 266 op/s 2024-03-31T23:27:25.275 INFO:tasks.workunit.client.0.smithi046.stdout: 10 24437 8.47 MB/sec execute 30 sec latency 74.704 ms 2024-03-31T23:27:25.520 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:25.519+0000 7fed5d525640 1 -- 172.21.15.46:0/85215554 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fed581055f0 msgr2=0x7fed581059d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:27:25.521 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:25.519+0000 7fed5d525640 1 --2- 172.21.15.46:0/85215554 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fed581055f0 0x7fed581059d0 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7fed48009900 tx=0x7fed4802bc30 comp rx=0 tx=0).stop 2024-03-31T23:27:25.521 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:25.519+0000 7fed5d525640 1 -- 172.21.15.46:0/85215554 shutdown_connections 2024-03-31T23:27:25.521 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:25.519+0000 7fed5d525640 1 --2- 172.21.15.46:0/85215554 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fed58101260 0x7fed5810d480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:27:25.521 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:25.519+0000 7fed5d525640 1 --2- 172.21.15.46:0/85215554 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fed581008a0 0x7fed58100d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:27:25.521 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:25.519+0000 7fed5d525640 1 --2- 172.21.15.46:0/85215554 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fed581055f0 0x7fed581059d0 unknown :-1 s=CLOSED pgs=903 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:27:25.521 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:25.519+0000 7fed5d525640 1 -- 172.21.15.46:0/85215554 >> 172.21.15.46:0/85215554 conn(0x7fed580774a0 msgr2=0x7fed580778b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:27:25.521 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:25.519+0000 7fed5d525640 1 -- 172.21.15.46:0/85215554 shutdown_connections 2024-03-31T23:27:25.521 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:25.519+0000 7fed5d525640 1 -- 172.21.15.46:0/85215554 wait complete. 2024-03-31T23:27:25.523 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:25.520+0000 7fed5d525640 1 Processor -- start 2024-03-31T23:27:25.523 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:25.520+0000 7fed5d525640 1 -- start start 2024-03-31T23:27:25.523 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:25.520+0000 7fed5d525640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fed581008a0 0x7fed58107500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:27:25.523 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:25.520+0000 7fed5d525640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fed58101260 0x7fed58107a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:27:25.523 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:25.520+0000 7fed5d525640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fed581055f0 0x7fed58107f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:27:25.523 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:25.520+0000 7fed5d525640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fed581109c0 con 0x7fed581055f0 2024-03-31T23:27:25.524 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:25.521+0000 7fed5d525640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fed58110840 con 0x7fed58101260 2024-03-31T23:27:25.524 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:25.521+0000 7fed5d525640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fed58110b40 con 0x7fed581008a0 2024-03-31T23:27:25.524 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:25.521+0000 7fed56ffd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fed581008a0 0x7fed58107500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:27:25.524 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:25.521+0000 7fed577fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fed581055f0 0x7fed58107f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:27:25.524 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:25.521+0000 7fed56ffd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fed581008a0 0x7fed58107500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.46:55012/0 (socket says 172.21.15.46:55012) 2024-03-31T23:27:25.524 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:25.521+0000 7fed577fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fed581055f0 0x7fed58107f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37230/0 (socket says 172.21.15.46:37230) 2024-03-31T23:27:25.524 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:25.521+0000 7fed56ffd640 1 -- 172.21.15.46:0/1342150675 learned_addr learned my addr 172.21.15.46:0/1342150675 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:27:25.524 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:25.521+0000 7fed4ffff640 1 --2- 172.21.15.46:0/1342150675 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fed58101260 0x7fed58107a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:27:25.524 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:25.521+0000 7fed577fe640 1 -- 172.21.15.46:0/1342150675 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fed581008a0 msgr2=0x7fed58107500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:27:25.524 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:25.521+0000 7fed577fe640 1 --2- 172.21.15.46:0/1342150675 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fed581008a0 0x7fed58107500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:27:25.524 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:25.521+0000 7fed577fe640 1 -- 172.21.15.46:0/1342150675 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fed58101260 msgr2=0x7fed58107a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:27:25.524 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:25.521+0000 7fed577fe640 1 --2- 172.21.15.46:0/1342150675 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fed58101260 0x7fed58107a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:27:25.524 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:25.521+0000 7fed577fe640 1 -- 172.21.15.46:0/1342150675 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed581aa7b0 con 0x7fed581055f0 2024-03-31T23:27:25.524 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:25.521+0000 7fed577fe640 1 --2- 172.21.15.46:0/1342150675 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fed581055f0 0x7fed58107f80 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7fed5000e9d0 tx=0x7fed5000ee90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:27:25.525 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:25.522+0000 7fed54ff9640 1 -- 172.21.15.46:0/1342150675 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed5000cde0 con 0x7fed581055f0 2024-03-31T23:27:25.525 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:25.522+0000 7fed5d525640 1 -- 172.21.15.46:0/1342150675 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fed581aaaa0 con 0x7fed581055f0 2024-03-31T23:27:25.525 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:25.522+0000 7fed54ff9640 1 -- 172.21.15.46:0/1342150675 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fed50005020 con 0x7fed581055f0 2024-03-31T23:27:25.525 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:25.522+0000 7fed54ff9640 1 -- 172.21.15.46:0/1342150675 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed50018670 con 0x7fed581055f0 2024-03-31T23:27:25.525 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:25.522+0000 7fed5d525640 1 -- 172.21.15.46:0/1342150675 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fed581aaff0 con 0x7fed581055f0 2024-03-31T23:27:25.526 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:25.525+0000 7fed54ff9640 1 -- 172.21.15.46:0/1342150675 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 25) ==== 137451+0+0 (secure 0 0 0) 0x7fed500051d0 con 0x7fed581055f0 2024-03-31T23:27:25.526 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:25.525+0000 7fed601e3740 1 Processor -- start 2024-03-31T23:27:25.526 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:25.525+0000 7fed601e3740 1 -- start start 2024-03-31T23:27:25.536 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:25.526+0000 7fed601e3740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x556b83e136c0 0x556b83e13aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:27:25.536 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:25.526+0000 7fed601e3740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556b83e13fe0 0x556b83e1c870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:27:25.536 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:25.526+0000 7fed601e3740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556b83e1cdb0 0x556b83e1f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:27:25.536 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:25.526+0000 7fed601e3740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x556b83d768f0 con 0x556b83e13fe0 2024-03-31T23:27:25.536 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:25.526+0000 7fed601e3740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x556b83d1f8d0 con 0x556b83e136c0 2024-03-31T23:27:25.536 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:25.526+0000 7fed601e3740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x556b83d72990 con 0x556b83e1cdb0 2024-03-31T23:27:25.536 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:25.526+0000 7fed54ff9640 1 --2- 172.21.15.46:0/1342150675 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fed440a26b0 0x7fed440a4b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:27:25.537 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:25.526+0000 7fed4ffff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556b83e1cdb0 0x556b83e1f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:27:25.537 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:25.526+0000 7fed56ffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556b83e13fe0 0x556b83e1c870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:27:25.537 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:25.526+0000 7fed4ffff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556b83e1cdb0 0x556b83e1f1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.46:55020/0 (socket says 172.21.15.46:55020) 2024-03-31T23:27:25.537 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:25.526+0000 7fed4ffff640 1 -- 172.21.15.46:0/1957765867 learned_addr learned my addr 172.21.15.46:0/1957765867 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:27:25.537 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:25.526+0000 7fed577fe640 1 --2- 172.21.15.46:0/1342150675 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fed440a26b0 0x7fed440a4b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:27:25.537 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:25.526+0000 7fed4ffff640 1 --2- 172.21.15.46:0/1957765867 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x556b83e136c0 0x556b83e13aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:27:25.537 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:25.526+0000 7fed54ff9640 1 -- 172.21.15.46:0/1342150675 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(117..117 src has 1..117) ==== 9554+0+0 (secure 0 0 0) 0x7fed500cf4d0 con 0x7fed581055f0 2024-03-31T23:27:25.537 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:25.526+0000 7fed56ffd640 1 -- 172.21.15.46:0/1957765867 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556b83e1cdb0 msgr2=0x556b83e1f1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:27:25.537 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:25.526+0000 7fed56ffd640 1 --2- 172.21.15.46:0/1957765867 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556b83e1cdb0 0x556b83e1f1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:27:25.537 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:25.526+0000 7fed56ffd640 1 -- 172.21.15.46:0/1957765867 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x556b83e136c0 msgr2=0x556b83e13aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:27:25.537 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:25.526+0000 7fed56ffd640 1 --2- 172.21.15.46:0/1957765867 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x556b83e136c0 0x556b83e13aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:27:25.537 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:25.526+0000 7fed56ffd640 1 -- 172.21.15.46:0/1957765867 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556b83a8b210 con 0x556b83e13fe0 2024-03-31T23:27:25.537 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:25.526+0000 7fed4ffff640 1 --2- 172.21.15.46:0/1957765867 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556b83e1cdb0 0x556b83e1f1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:27:25.537 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:25.527+0000 7fed577fe640 1 --2- 172.21.15.46:0/1342150675 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fed440a26b0 0x7fed440a4b70 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fed500cc960 tx=0x7fed5001c000 comp rx=0 tx=0).ready entity=mgr.15654 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:27:25.538 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:25.527+0000 7fed56ffd640 1 --2- 172.21.15.46:0/1957765867 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556b83e13fe0 0x556b83e1c870 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7fed48009230 tx=0x7fed48005e10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=897f986009dd8455 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:27:25.538 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:25.527+0000 7fed4d7fa640 1 -- 172.21.15.46:0/1957765867 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed4802c8c0 con 0x556b83e13fe0 2024-03-31T23:27:25.538 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:25.527+0000 7fed4d7fa640 1 -- 172.21.15.46:0/1957765867 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fed4802ca60 con 0x556b83e13fe0 2024-03-31T23:27:25.538 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:25.527+0000 7fed4d7fa640 1 -- 172.21.15.46:0/1957765867 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed480046c0 con 0x556b83e13fe0 2024-03-31T23:27:25.538 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:25.528+0000 7fed601e3740 1 -- 172.21.15.46:0/1957765867 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556b83e13fe0 msgr2=0x556b83e1c870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:27:25.538 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:25.528+0000 7fed601e3740 1 --2- 172.21.15.46:0/1957765867 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556b83e13fe0 0x556b83e1c870 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7fed48009230 tx=0x7fed48005e10 comp rx=0 tx=0).stop 2024-03-31T23:27:25.538 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:25.528+0000 7fed601e3740 1 -- 172.21.15.46:0/1957765867 shutdown_connections 2024-03-31T23:27:25.538 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:25.528+0000 7fed601e3740 1 --2- 172.21.15.46:0/1957765867 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556b83e1cdb0 0x556b83e1f1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:27:25.538 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:25.528+0000 7fed601e3740 1 --2- 172.21.15.46:0/1957765867 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556b83e13fe0 0x556b83e1c870 unknown :-1 s=CLOSED pgs=905 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:27:25.538 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:25.528+0000 7fed601e3740 1 --2- 172.21.15.46:0/1957765867 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x556b83e136c0 0x556b83e13aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:27:25.538 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:25.528+0000 7fed601e3740 1 -- 172.21.15.46:0/1957765867 >> 172.21.15.46:0/1957765867 conn(0x556b83e0dfe0 msgr2=0x556b83e0e3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:27:25.538 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:25.528+0000 7fed601e3740 1 -- 172.21.15.46:0/1957765867 shutdown_connections 2024-03-31T23:27:25.539 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:25.528+0000 7fed601e3740 1 -- 172.21.15.46:0/1957765867 wait complete. 2024-03-31T23:27:25.539 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:25.529+0000 7fed601e3740 1 -- start start 2024-03-31T23:27:25.539 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:25.529+0000 7fed601e3740 1 Processor -- start 2024-03-31T23:27:25.539 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:25.529+0000 7fed601e3740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556b83e136c0 0x556b83e19e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:27:25.539 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:25.529+0000 7fed601e3740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x556b83e13fe0 0x556b83e1a340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:27:25.539 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:25.529+0000 7fed601e3740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556b83e1cdb0 0x556b83e1a880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:27:25.539 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:25.529+0000 7fed601e3740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x556b83e1f980 con 0x556b83e136c0 2024-03-31T23:27:25.539 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:25.529+0000 7fed601e3740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x556b83d14f80 con 0x556b83e13fe0 2024-03-31T23:27:25.539 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:25.529+0000 7fed601e3740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x556b83e1fb00 con 0x556b83e1cdb0 2024-03-31T23:27:25.539 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:25.529+0000 7fed601e3740 10 client.0 _ll_register_callbacks cb 0x556b83d6d290 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T23:27:25.539 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:25.529+0000 7fed4ffff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556b83e136c0 0x556b83e19e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:27:25.539 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:25.529+0000 7fed4ffff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556b83e136c0 0x556b83e19e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37248/0 (socket says 172.21.15.46:37248) 2024-03-31T23:27:25.539 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:25.529+0000 7fed4ffff640 1 -- 172.21.15.46:0/966706097 learned_addr learned my addr 172.21.15.46:0/966706097 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:27:25.540 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:25.529+0000 7fed56ffd640 1 --2- 172.21.15.46:0/966706097 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x556b83e13fe0 0x556b83e1a340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:27:25.540 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:25.529+0000 7fed4ffff640 1 --2- 172.21.15.46:0/966706097 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556b83e1cdb0 0x556b83e1a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:27:25.540 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:25.530+0000 7fed4ffff640 1 -- 172.21.15.46:0/966706097 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556b83e1cdb0 msgr2=0x556b83e1a880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:27:25.540 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:25.530+0000 7fed4ffff640 1 --2- 172.21.15.46:0/966706097 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556b83e1cdb0 0x556b83e1a880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:27:25.540 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:25.530+0000 7fed4ffff640 1 -- 172.21.15.46:0/966706097 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x556b83e13fe0 msgr2=0x556b83e1a340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:27:25.540 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:25.530+0000 7fed4ffff640 1 --2- 172.21.15.46:0/966706097 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x556b83e13fe0 0x556b83e1a340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:27:25.540 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:25.530+0000 7fed4ffff640 1 -- 172.21.15.46:0/966706097 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556b83b57620 con 0x556b83e136c0 2024-03-31T23:27:25.540 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:25.530+0000 7fed56ffd640 1 --2- 172.21.15.46:0/966706097 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x556b83e13fe0 0x556b83e1a340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:27:25.540 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:25.530+0000 7fed4ffff640 1 --2- 172.21.15.46:0/966706097 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556b83e136c0 0x556b83e19e00 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7fed40003730 tx=0x7fed40005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2680e95f18afbd41 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:27:25.540 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:25.530+0000 7fed3e7fc640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-31T23:27:25.540 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:25.530+0000 7fed3e7fc640 1 -- 172.21.15.46:0/966706097 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed40011bf0 con 0x556b83e136c0 2024-03-31T23:27:25.540 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:25.530+0000 7fed601e3740 1 -- 172.21.15.46:0/966706097 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fed4802ca60 con 0x556b83e136c0 2024-03-31T23:27:25.540 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:25.530+0000 7fed3e7fc640 1 -- 172.21.15.46:0/966706097 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(117..117 src has 1..117) ==== 9554+0+0 (secure 0 0 0) 0x7fed40011d90 con 0x556b83e136c0 2024-03-31T23:27:25.540 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:25.530+0000 7fed3e7fc640 1 -- 172.21.15.46:0/966706097 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=41) ==== 24+0+0 (secure 0 0 0) 0x7fed40005020 con 0x556b83e136c0 2024-03-31T23:27:25.540 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:25.530+0000 7fed601e3740 10 client.17424 fetch_fsmap learned FSMap version 41 2024-03-31T23:27:25.541 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:25.530+0000 7fed601e3740 1 -- 172.21.15.46:0/966706097 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=41,monmap=4+}) -- 0x556b83e2b960 con 0x556b83e136c0 2024-03-31T23:27:25.541 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:25.531+0000 7fed3e7fc640 1 -- 172.21.15.46:0/966706097 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fed4802ca60 con 0x556b83e136c0 2024-03-31T23:27:25.541 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:25.531+0000 7fed3e7fc640 1 -- 172.21.15.46:0/966706097 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 41) ==== 8366+0+0 (secure 0 0 0) 0x7fed40004bf0 con 0x556b83e136c0 2024-03-31T23:27:25.541 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:25.531+0000 7fed601e3740 10 client.17424 fetch_fsmap finished waiting for FSMap version 41 2024-03-31T23:27:25.541 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:25.531+0000 7fed601e3740 10 client.17424 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-31T23:27:25.541 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:25.531+0000 7fed601e3740 20 client.17424 populate_metadata read hostname 'smithi046' 2024-03-31T23:27:25.541 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:25.531+0000 7fed601e3740 1 --2- 172.21.15.46:0/966706097 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x556b83e2cdd0 0x556b83e2f1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:27:25.541 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:25.531+0000 7fed601e3740 4 client.17424 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-31T23:27:25.541 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:25.531+0000 7fed601e3740 1 -- 172.21.15.46:0/966706097 --> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556b83d8a120 con 0x556b83e2cdd0 2024-03-31T23:27:25.541 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:25.531+0000 7fed56ffd640 1 --2- 172.21.15.46:0/966706097 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x556b83e2cdd0 0x556b83e2f1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:27:25.541 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:25.532+0000 7fed56ffd640 1 --2- 172.21.15.46:0/966706097 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x556b83e2cdd0 0x556b83e2f1c0 crc :-1 s=READY pgs=216 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e5e40f0a27a234d6 server_cookie=ae9db5392d554dc7 in_seq=0 out_seq=0 2024-03-31T23:27:25.541 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:25.532+0000 7fed3e7fc640 10 client.17424 ms_handle_connect on v2:172.21.15.142:6838/2519143918 2024-03-31T23:27:25.541 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:25.534+0000 7fed3e7fc640 1 -- 172.21.15.46:0/966706097 <== mds.0 v2:172.21.15.142:6838/2519143918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x556b83d8a120 con 0x556b83e2cdd0 2024-03-31T23:27:25.541 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:25.534+0000 7fed3e7fc640 10 client.17424 handle_command_reply: tid=0 2024-03-31T23:27:25.542 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:25.534+0000 7fed601e3740 1 client.17424 shutdown 2024-03-31T23:27:25.542 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:25.534+0000 7fed601e3740 1 -- 172.21.15.46:0/966706097 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556b83e136c0 msgr2=0x556b83e19e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:27:25.542 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:25.534+0000 7fed601e3740 1 --2- 172.21.15.46:0/966706097 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556b83e136c0 0x556b83e19e00 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7fed40003730 tx=0x7fed40005f90 comp rx=0 tx=0).stop 2024-03-31T23:27:25.542 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:25.534+0000 7fed601e3740 1 -- 172.21.15.46:0/966706097 shutdown_connections 2024-03-31T23:27:25.542 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:25.534+0000 7fed601e3740 1 --2- 172.21.15.46:0/966706097 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x556b83e2cdd0 0x556b83e2f1c0 crc :-1 s=READY pgs=216 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:27:25.542 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:25.534+0000 7fed601e3740 1 --2- 172.21.15.46:0/966706097 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556b83e1cdb0 0x556b83e1a880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:27:25.542 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:25.534+0000 7fed3e7fc640 0 client.17424 ms_handle_reset on v2:172.21.15.142:6838/2519143918 2024-03-31T23:27:25.542 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:25.534+0000 7fed601e3740 1 --2- 172.21.15.46:0/966706097 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x556b83e13fe0 0x556b83e1a340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:27:25.542 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:25.534+0000 7fed601e3740 1 --2- 172.21.15.46:0/966706097 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556b83e136c0 0x556b83e19e00 secure :-1 s=CLOSED pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7fed40003730 tx=0x7fed40005f90 comp rx=0 tx=0).stop 2024-03-31T23:27:25.542 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:25.534+0000 7fed601e3740 1 -- 172.21.15.46:0/966706097 >> 172.21.15.46:0/966706097 conn(0x556b83e0dfe0 msgr2=0x556b83e0fd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:27:25.543 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:25.535+0000 7fed601e3740 1 -- 172.21.15.46:0/966706097 shutdown_connections 2024-03-31T23:27:25.543 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:25.535+0000 7fed601e3740 1 -- 172.21.15.46:0/966706097 wait complete. 2024-03-31T23:27:25.543 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:25.535+0000 7fed601e3740 20 client.17424 trim_cache size 0 max 16384 2024-03-31T23:27:25.563 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:25.561+0000 7fed601e3740 1 Processor -- start 2024-03-31T23:27:25.564 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:25.561+0000 7fed601e3740 1 -- start start 2024-03-31T23:27:25.564 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:25.561+0000 7fed601e3740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556b83e13fe0 0x556b83e1c530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:27:25.564 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:25.561+0000 7fed601e3740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556b83e1cdb0 0x556b83e28720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:27:25.564 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:25.561+0000 7fed601e3740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x556b83e2cdd0 0x556b83e14670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:27:25.564 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:25.561+0000 7fed601e3740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fed581b7af0 con 0x556b83e1cdb0 2024-03-31T23:27:25.564 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:25.561+0000 7fed601e3740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fed581b52d0 con 0x556b83e2cdd0 2024-03-31T23:27:25.564 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:25.561+0000 7fed601e3740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fed581b3140 con 0x556b83e13fe0 2024-03-31T23:27:25.564 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:25.562+0000 7fed56ffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556b83e1cdb0 0x556b83e28720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:27:25.564 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:25.562+0000 7fed56ffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556b83e1cdb0 0x556b83e28720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37264/0 (socket says 172.21.15.46:37264) 2024-03-31T23:27:25.564 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:25.562+0000 7fed56ffd640 1 -- 172.21.15.46:0/3441022122 learned_addr learned my addr 172.21.15.46:0/3441022122 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:27:25.564 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:25.562+0000 7fed4ffff640 1 --2- 172.21.15.46:0/3441022122 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556b83e13fe0 0x556b83e1c530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:27:25.565 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:25.562+0000 7fed4ffff640 1 --2- 172.21.15.46:0/3441022122 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x556b83e2cdd0 0x556b83e14670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:27:25.565 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:25.562+0000 7fed56ffd640 1 -- 172.21.15.46:0/3441022122 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556b83e13fe0 msgr2=0x556b83e1c530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:27:25.565 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:25.562+0000 7fed56ffd640 1 --2- 172.21.15.46:0/3441022122 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556b83e13fe0 0x556b83e1c530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:27:25.565 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:25.562+0000 7fed56ffd640 1 -- 172.21.15.46:0/3441022122 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x556b83e2cdd0 msgr2=0x556b83e14670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:27:25.565 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:25.562+0000 7fed56ffd640 1 --2- 172.21.15.46:0/3441022122 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x556b83e2cdd0 0x556b83e14670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:27:25.565 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:25.562+0000 7fed56ffd640 1 -- 172.21.15.46:0/3441022122 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556b83e2b770 con 0x556b83e1cdb0 2024-03-31T23:27:25.565 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:25.562+0000 7fed4ffff640 1 --2- 172.21.15.46:0/3441022122 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x556b83e2cdd0 0x556b83e14670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T23:27:25.565 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:25.562+0000 7fed56ffd640 1 --2- 172.21.15.46:0/3441022122 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556b83e1cdb0 0x556b83e28720 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 crypto rx=0x7fed48036040 tx=0x7fed48009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e6bc92eeb1b64341 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:27:25.565 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:25.562+0000 7fed4cff9640 1 -- 172.21.15.46:0/3441022122 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed4802ca00 con 0x556b83e1cdb0 2024-03-31T23:27:25.565 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:25.562+0000 7fed4cff9640 1 -- 172.21.15.46:0/3441022122 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fed4802cba0 con 0x556b83e1cdb0 2024-03-31T23:27:25.565 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:25.562+0000 7fed4cff9640 1 -- 172.21.15.46:0/3441022122 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed4804e750 con 0x556b83e1cdb0 2024-03-31T23:27:25.568 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:25.563+0000 7fed601e3740 1 -- 172.21.15.46:0/3441022122 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556b83e1cdb0 msgr2=0x556b83e28720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:27:25.569 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:25.563+0000 7fed601e3740 1 --2- 172.21.15.46:0/3441022122 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556b83e1cdb0 0x556b83e28720 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 crypto rx=0x7fed48036040 tx=0x7fed48009260 comp rx=0 tx=0).stop 2024-03-31T23:27:25.569 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:25.564+0000 7fed601e3740 1 -- 172.21.15.46:0/3441022122 shutdown_connections 2024-03-31T23:27:25.569 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:25.564+0000 7fed601e3740 1 --2- 172.21.15.46:0/3441022122 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x556b83e2cdd0 0x556b83e14670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:27:25.569 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:25.564+0000 7fed601e3740 1 --2- 172.21.15.46:0/3441022122 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556b83e1cdb0 0x556b83e28720 unknown :-1 s=CLOSED pgs=907 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:27:25.569 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:25.564+0000 7fed601e3740 1 --2- 172.21.15.46:0/3441022122 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556b83e13fe0 0x556b83e1c530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:27:25.569 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:25.564+0000 7fed601e3740 1 -- 172.21.15.46:0/3441022122 >> 172.21.15.46:0/3441022122 conn(0x556b83e0dfe0 msgr2=0x556b83e0e3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:27:25.569 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:25.564+0000 7fed601e3740 1 -- 172.21.15.46:0/3441022122 shutdown_connections 2024-03-31T23:27:25.569 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:25.564+0000 7fed601e3740 1 -- 172.21.15.46:0/3441022122 wait complete. 2024-03-31T23:27:25.569 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:25.564+0000 7fed601e3740 1 -- start start 2024-03-31T23:27:25.569 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:25.564+0000 7fed601e3740 1 Processor -- start 2024-03-31T23:27:25.569 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:25.564+0000 7fed601e3740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x556b83e13fe0 0x556b83e24d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:27:25.569 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:25.564+0000 7fed601e3740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556b83e1cdb0 0x556b83e16480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:27:25.569 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:25.564+0000 7fed601e3740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556b83e2cdd0 0x556b83e169c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:27:25.570 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:25.564+0000 7fed601e3740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fed581b3790 con 0x556b83e1cdb0 2024-03-31T23:27:25.570 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:25.564+0000 7fed601e3740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fed581b4c00 con 0x556b83e13fe0 2024-03-31T23:27:25.570 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:25.564+0000 7fed601e3740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fed581b3e60 con 0x556b83e2cdd0 2024-03-31T23:27:25.570 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:25.565+0000 7fed601e3740 10 client.0 _ll_register_callbacks cb 0x7fed400035f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T23:27:25.570 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:25.565+0000 7fed4ffff640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x556b83e13fe0 0x556b83e24d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:27:25.570 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:25.565+0000 7fed4ffff640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x556b83e13fe0 0x556b83e24d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.46:47540/0 (socket says 172.21.15.46:47540) 2024-03-31T23:27:25.570 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:25.565+0000 7fed4ffff640 1 -- 172.21.15.46:0/1900114325 learned_addr learned my addr 172.21.15.46:0/1900114325 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:27:25.570 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:25.565+0000 7fed56ffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556b83e1cdb0 0x556b83e16480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:27:25.570 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:25.565+0000 7fed4ffff640 1 --2- 172.21.15.46:0/1900114325 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556b83e2cdd0 0x556b83e169c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:27:25.570 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:25.565+0000 7fed56ffd640 1 -- 172.21.15.46:0/1900114325 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556b83e2cdd0 msgr2=0x556b83e169c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:27:25.570 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:25.565+0000 7fed56ffd640 1 --2- 172.21.15.46:0/1900114325 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556b83e2cdd0 0x556b83e169c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:27:25.570 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:25.565+0000 7fed56ffd640 1 -- 172.21.15.46:0/1900114325 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x556b83e13fe0 msgr2=0x556b83e24d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:27:25.570 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:25.565+0000 7fed56ffd640 1 --2- 172.21.15.46:0/1900114325 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x556b83e13fe0 0x556b83e24d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:27:25.571 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:25.565+0000 7fed56ffd640 1 -- 172.21.15.46:0/1900114325 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556b83e16f30 con 0x556b83e1cdb0 2024-03-31T23:27:25.571 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:25.565+0000 7fed4ffff640 1 --2- 172.21.15.46:0/1900114325 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556b83e2cdd0 0x556b83e169c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:27:25.571 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:25.565+0000 7fed3e7fc640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-31T23:27:25.571 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:25.565+0000 7fed56ffd640 1 --2- 172.21.15.46:0/1900114325 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556b83e1cdb0 0x556b83e16480 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x556b83e1aee0 tx=0x7fed480058c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=acb431f921ab9b51 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:27:25.571 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:25.565+0000 7fed3e7fc640 1 -- 172.21.15.46:0/1900114325 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed4804e0c0 con 0x556b83e1cdb0 2024-03-31T23:27:25.571 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:25.565+0000 7fed3e7fc640 1 -- 172.21.15.46:0/1900114325 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(117..117 src has 1..117) ==== 9554+0+0 (secure 0 0 0) 0x7fed4804e4d0 con 0x556b83e1cdb0 2024-03-31T23:27:25.571 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:25.565+0000 7fed601e3740 1 -- 172.21.15.46:0/1900114325 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fed4802cba0 con 0x556b83e1cdb0 2024-03-31T23:27:25.571 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:25.566+0000 7fed3e7fc640 1 -- 172.21.15.46:0/1900114325 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=41) ==== 24+0+0 (secure 0 0 0) 0x7fed4804e6d0 con 0x556b83e1cdb0 2024-03-31T23:27:25.571 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:25.566+0000 7fed601e3740 10 client.17430 fetch_fsmap learned FSMap version 41 2024-03-31T23:27:25.571 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:25.566+0000 7fed601e3740 1 -- 172.21.15.46:0/1900114325 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=41,monmap=4+}) -- 0x556b83e92e90 con 0x556b83e1cdb0 2024-03-31T23:27:25.571 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:25.566+0000 7fed3e7fc640 1 -- 172.21.15.46:0/1900114325 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fed4802cba0 con 0x556b83e1cdb0 2024-03-31T23:27:25.571 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:25.566+0000 7fed3e7fc640 1 -- 172.21.15.46:0/1900114325 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 41) ==== 8366+0+0 (secure 0 0 0) 0x7fed48003a70 con 0x556b83e1cdb0 2024-03-31T23:27:25.571 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:25.566+0000 7fed601e3740 10 client.17430 fetch_fsmap finished waiting for FSMap version 41 2024-03-31T23:27:25.571 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:25.566+0000 7fed601e3740 10 client.17430 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-31T23:27:25.572 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:25.566+0000 7fed601e3740 20 client.17430 populate_metadata read hostname 'smithi046' 2024-03-31T23:27:25.572 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:25.566+0000 7fed601e3740 1 --2- 172.21.15.46:0/1900114325 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x556b83e93100 0x556b83e954f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:27:25.572 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:25.566+0000 7fed601e3740 4 client.17430 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-31T23:27:25.572 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:25.566+0000 7fed601e3740 1 -- 172.21.15.46:0/1900114325 --> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] -- command(tid 0: {"prefix": "flush journal"}) -- 0x556b83d83e30 con 0x556b83e93100 2024-03-31T23:27:25.572 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:25.566+0000 7fed4ffff640 1 --2- 172.21.15.46:0/1900114325 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x556b83e93100 0x556b83e954f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:27:25.572 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:25.567+0000 7fed3e7fc640 10 client.17430 ms_handle_connect on v2:172.21.15.142:6838/2519143918 2024-03-31T23:27:25.572 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:25.567+0000 7fed4ffff640 1 --2- 172.21.15.46:0/1900114325 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x556b83e93100 0x556b83e954f0 crc :-1 s=READY pgs=217 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3b47ecf2f10b9759 server_cookie=2bff75e7712e3bd9 in_seq=0 out_seq=0 2024-03-31T23:27:25.683 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:25.682+0000 7fed3e7fc640 1 -- 172.21.15.46:0/1900114325 <== mds.0 v2:172.21.15.142:6838/2519143918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x556b83d83e30 con 0x556b83e93100 2024-03-31T23:27:25.684 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:25.682+0000 7fed3e7fc640 10 client.17430 handle_command_reply: tid=0 2024-03-31T23:27:25.684 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:25.682+0000 7fed601e3740 1 client.17430 shutdown 2024-03-31T23:27:25.684 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:25.682+0000 7fed601e3740 1 -- 172.21.15.46:0/1900114325 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556b83e1cdb0 msgr2=0x556b83e16480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:27:25.684 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:25.682+0000 7fed601e3740 1 --2- 172.21.15.46:0/1900114325 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556b83e1cdb0 0x556b83e16480 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x556b83e1aee0 tx=0x7fed480058c0 comp rx=0 tx=0).stop 2024-03-31T23:27:25.684 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:25.682+0000 7fed601e3740 1 -- 172.21.15.46:0/1900114325 shutdown_connections 2024-03-31T23:27:25.684 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:25.682+0000 7fed601e3740 1 --2- 172.21.15.46:0/1900114325 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x556b83e93100 0x556b83e954f0 crc :-1 s=READY pgs=217 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:27:25.684 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:25.682+0000 7fed601e3740 1 --2- 172.21.15.46:0/1900114325 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556b83e2cdd0 0x556b83e169c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:27:25.684 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:25.682+0000 7fed601e3740 1 --2- 172.21.15.46:0/1900114325 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556b83e1cdb0 0x556b83e16480 unknown :-1 s=CLOSED pgs=908 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:27:25.684 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:25.682+0000 7fed3e7fc640 0 client.17430 ms_handle_reset on v2:172.21.15.142:6838/2519143918 2024-03-31T23:27:25.684 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:25.682+0000 7fed601e3740 1 --2- 172.21.15.46:0/1900114325 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x556b83e13fe0 0x556b83e24d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:27:25.684 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:25.682+0000 7fed601e3740 1 -- 172.21.15.46:0/1900114325 >> 172.21.15.46:0/1900114325 conn(0x556b83e0dfe0 msgr2=0x556b83e14670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:27:25.684 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:25.682+0000 7fed601e3740 1 -- 172.21.15.46:0/1900114325 shutdown_connections 2024-03-31T23:27:25.685 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:25.682+0000 7fed601e3740 1 -- 172.21.15.46:0/1900114325 wait complete. 2024-03-31T23:27:25.685 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:25.682+0000 7fed601e3740 20 client.17430 trim_cache size 0 max 16384 2024-03-31T23:27:25.685 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout:{ 2024-03-31T23:27:25.685 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout: "message": "", 2024-03-31T23:27:25.685 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout: "return_code": 0 2024-03-31T23:27:25.685 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout:} 2024-03-31T23:27:25.686 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:25.683+0000 7fed5d525640 1 -- 172.21.15.46:0/1342150675 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fed440a26b0 msgr2=0x7fed440a4b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:27:25.686 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:25.683+0000 7fed5d525640 1 --2- 172.21.15.46:0/1342150675 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fed440a26b0 0x7fed440a4b70 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fed500cc960 tx=0x7fed5001c000 comp rx=0 tx=0).stop 2024-03-31T23:27:25.686 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:25.683+0000 7fed5d525640 1 -- 172.21.15.46:0/1342150675 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fed581055f0 msgr2=0x7fed58107f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:27:25.686 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:25.683+0000 7fed5d525640 1 --2- 172.21.15.46:0/1342150675 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fed581055f0 0x7fed58107f80 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7fed5000e9d0 tx=0x7fed5000ee90 comp rx=0 tx=0).stop 2024-03-31T23:27:25.686 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:25.683+0000 7fed5d525640 1 -- 172.21.15.46:0/1342150675 shutdown_connections 2024-03-31T23:27:25.686 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:25.683+0000 7fed5d525640 1 --2- 172.21.15.46:0/1342150675 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fed440a26b0 0x7fed440a4b70 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:27:25.686 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:25.683+0000 7fed5d525640 1 --2- 172.21.15.46:0/1342150675 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fed581055f0 0x7fed58107f80 unknown :-1 s=CLOSED pgs=904 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:27:25.686 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:25.683+0000 7fed5d525640 1 --2- 172.21.15.46:0/1342150675 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fed58101260 0x7fed58107a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:27:25.686 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:25.683+0000 7fed5d525640 1 --2- 172.21.15.46:0/1342150675 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fed581008a0 0x7fed58107500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:27:25.687 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:25.683+0000 7fed5d525640 1 -- 172.21.15.46:0/1342150675 >> 172.21.15.46:0/1342150675 conn(0x7fed580774a0 msgr2=0x7fed5810bc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:27:25.687 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:25.683+0000 7fed5d525640 1 -- 172.21.15.46:0/1342150675 shutdown_connections 2024-03-31T23:27:25.687 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:25.683+0000 7fed5d525640 1 -- 172.21.15.46:0/1342150675 wait complete. 2024-03-31T23:27:26.275 INFO:tasks.workunit.client.0.smithi046.stdout: 10 24584 8.46 MB/sec execute 31 sec latency 124.365 ms 2024-03-31T23:27:27.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:27:26 smithi162 ceph-mon[36237]: pgmap v495: 129 pgs: 129 active+clean; 561 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 6.7 MiB/s wr, 386 op/s 2024-03-31T23:27:27.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:27:26 smithi046 ceph-mon[32498]: pgmap v495: 129 pgs: 129 active+clean; 561 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 6.7 MiB/s wr, 386 op/s 2024-03-31T23:27:27.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:27:26 smithi142 ceph-mon[35271]: pgmap v495: 129 pgs: 129 active+clean; 561 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 6.7 MiB/s wr, 386 op/s 2024-03-31T23:27:27.275 INFO:tasks.workunit.client.0.smithi046.stdout: 10 24768 8.61 MB/sec execute 32 sec latency 73.747 ms 2024-03-31T23:27:28.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:27:27 smithi046 ceph-mon[32498]: pgmap v496: 129 pgs: 129 active+clean; 561 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 4.5 MiB/s wr, 229 op/s 2024-03-31T23:27:28.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:27:27 smithi142 ceph-mon[35271]: pgmap v496: 129 pgs: 129 active+clean; 561 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 4.5 MiB/s wr, 229 op/s 2024-03-31T23:27:28.275 INFO:tasks.workunit.client.0.smithi046.stdout: 10 24958 8.68 MB/sec execute 33 sec latency 67.180 ms 2024-03-31T23:27:28.295 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:27:27 smithi162 ceph-mon[36237]: pgmap v496: 129 pgs: 129 active+clean; 561 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 4.5 MiB/s wr, 229 op/s 2024-03-31T23:27:29.276 INFO:tasks.workunit.client.0.smithi046.stdout: 10 25103 8.53 MB/sec execute 34 sec latency 68.629 ms 2024-03-31T23:27:30.276 INFO:tasks.workunit.client.0.smithi046.stdout: 10 25248 8.39 MB/sec execute 35 sec latency 79.483 ms 2024-03-31T23:27:30.295 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:27:29 smithi162 ceph-mon[36237]: pgmap v497: 129 pgs: 129 active+clean; 558 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 6.3 MiB/s wr, 381 op/s 2024-03-31T23:27:30.362 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:27:29 smithi046 ceph-mon[32498]: pgmap v497: 129 pgs: 129 active+clean; 558 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 6.3 MiB/s wr, 381 op/s 2024-03-31T23:27:30.443 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:27:29 smithi142 ceph-mon[35271]: pgmap v497: 129 pgs: 129 active+clean; 558 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 6.3 MiB/s wr, 381 op/s 2024-03-31T23:27:31.276 INFO:tasks.workunit.client.0.smithi046.stdout: 10 25418 8.40 MB/sec execute 36 sec latency 60.852 ms 2024-03-31T23:27:32.276 INFO:tasks.workunit.client.0.smithi046.stdout: 10 25613 8.51 MB/sec execute 37 sec latency 82.040 ms 2024-03-31T23:27:32.295 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:27:31 smithi162 ceph-mon[36237]: pgmap v498: 129 pgs: 129 active+clean; 558 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 4.0 MiB/s wr, 271 op/s 2024-03-31T23:27:32.361 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:27:31 smithi046 ceph-mon[32498]: pgmap v498: 129 pgs: 129 active+clean; 558 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 4.0 MiB/s wr, 271 op/s 2024-03-31T23:27:32.443 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:27:31 smithi142 ceph-mon[35271]: pgmap v498: 129 pgs: 129 active+clean; 558 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 4.0 MiB/s wr, 271 op/s 2024-03-31T23:27:33.276 INFO:tasks.workunit.client.0.smithi046.stdout: 10 25817 8.58 MB/sec execute 38 sec latency 69.577 ms 2024-03-31T23:27:34.277 INFO:tasks.workunit.client.0.smithi046.stdout: 10 25966 8.52 MB/sec execute 39 sec latency 73.108 ms 2024-03-31T23:27:34.295 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:27:33 smithi162 ceph-mon[36237]: pgmap v499: 129 pgs: 129 active+clean; 558 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 4.0 MiB/s wr, 271 op/s 2024-03-31T23:27:34.361 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:27:33 smithi046 ceph-mon[32498]: pgmap v499: 129 pgs: 129 active+clean; 558 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 4.0 MiB/s wr, 271 op/s 2024-03-31T23:27:34.443 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:27:33 smithi142 ceph-mon[35271]: pgmap v499: 129 pgs: 129 active+clean; 558 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 4.0 MiB/s wr, 271 op/s 2024-03-31T23:27:35.276 INFO:tasks.workunit.client.0.smithi046.stdout: 10 26106 8.53 MB/sec execute 40 sec latency 81.381 ms 2024-03-31T23:27:36.277 INFO:tasks.workunit.client.0.smithi046.stdout: 10 26309 8.78 MB/sec execute 41 sec latency 67.036 ms 2024-03-31T23:27:36.295 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:27:35 smithi162 ceph-mon[36237]: pgmap v500: 129 pgs: 129 active+clean; 571 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 6.2 MiB/s wr, 383 op/s 2024-03-31T23:27:36.361 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:27:35 smithi046 ceph-mon[32498]: pgmap v500: 129 pgs: 129 active+clean; 571 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 6.2 MiB/s wr, 383 op/s 2024-03-31T23:27:36.443 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:27:35 smithi142 ceph-mon[35271]: pgmap v500: 129 pgs: 129 active+clean; 571 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 6.2 MiB/s wr, 383 op/s 2024-03-31T23:27:37.277 INFO:tasks.workunit.client.0.smithi046.stdout: 10 26519 8.84 MB/sec execute 42 sec latency 74.888 ms 2024-03-31T23:27:38.277 INFO:tasks.workunit.client.0.smithi046.stdout: 10 26663 8.87 MB/sec execute 43 sec latency 73.619 ms 2024-03-31T23:27:38.295 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:27:37 smithi162 ceph-mon[36237]: pgmap v501: 129 pgs: 129 active+clean; 571 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.9 MiB/s wr, 263 op/s 2024-03-31T23:27:38.361 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:27:37 smithi046 ceph-mon[32498]: pgmap v501: 129 pgs: 129 active+clean; 571 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.9 MiB/s wr, 263 op/s 2024-03-31T23:27:38.443 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:27:37 smithi142 ceph-mon[35271]: pgmap v501: 129 pgs: 129 active+clean; 571 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.9 MiB/s wr, 263 op/s 2024-03-31T23:27:38.816 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:38.815+0000 7fc6ddcb5640 1 -- 172.21.15.46:0/2578739385 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc6d8101260 msgr2=0x7fc6d810d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:27:38.817 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:38.815+0000 7fc6ddcb5640 1 --2- 172.21.15.46:0/2578739385 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc6d8101260 0x7fc6d810d480 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 crypto rx=0x7fc6d0009ca0 tx=0x7fc6d002c0f0 comp rx=0 tx=0).stop 2024-03-31T23:27:38.817 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:38.815+0000 7fc6ddcb5640 1 -- 172.21.15.46:0/2578739385 shutdown_connections 2024-03-31T23:27:38.817 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:38.815+0000 7fc6ddcb5640 1 --2- 172.21.15.46:0/2578739385 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc6d8101260 0x7fc6d810d480 unknown :-1 s=CLOSED pgs=909 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:27:38.817 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:38.815+0000 7fc6ddcb5640 1 --2- 172.21.15.46:0/2578739385 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc6d81008a0 0x7fc6d8100d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:27:38.817 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:38.815+0000 7fc6ddcb5640 1 --2- 172.21.15.46:0/2578739385 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc6d81055f0 0x7fc6d81059d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:27:38.817 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:38.815+0000 7fc6ddcb5640 1 -- 172.21.15.46:0/2578739385 >> 172.21.15.46:0/2578739385 conn(0x7fc6d80774a0 msgr2=0x7fc6d80778b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:27:38.817 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:38.815+0000 7fc6ddcb5640 1 -- 172.21.15.46:0/2578739385 shutdown_connections 2024-03-31T23:27:38.817 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:38.815+0000 7fc6ddcb5640 1 -- 172.21.15.46:0/2578739385 wait complete. 2024-03-31T23:27:38.817 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:38.816+0000 7fc6ddcb5640 1 Processor -- start 2024-03-31T23:27:38.819 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:38.816+0000 7fc6ddcb5640 1 -- start start 2024-03-31T23:27:38.819 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:38.816+0000 7fc6ddcb5640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc6d81008a0 0x7fc6d819a090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:27:38.819 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:38.816+0000 7fc6ddcb5640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc6d8101260 0x7fc6d819a5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:27:38.819 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:38.816+0000 7fc6ddcb5640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc6d81055f0 0x7fc6d819e900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:27:38.819 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:38.816+0000 7fc6ddcb5640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fc6d81109c0 con 0x7fc6d8101260 2024-03-31T23:27:38.819 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:38.816+0000 7fc6ddcb5640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fc6d8110840 con 0x7fc6d81055f0 2024-03-31T23:27:38.819 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:38.816+0000 7fc6ddcb5640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fc6d8110b40 con 0x7fc6d81008a0 2024-03-31T23:27:38.820 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:38.816+0000 7fc6d6ffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc6d8101260 0x7fc6d819a5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:27:38.820 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:38.816+0000 7fc6d6ffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc6d8101260 0x7fc6d819a5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:45834/0 (socket says 172.21.15.46:45834) 2024-03-31T23:27:38.820 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:38.816+0000 7fc6d6ffd640 1 -- 172.21.15.46:0/2896388918 learned_addr learned my addr 172.21.15.46:0/2896388918 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:27:38.820 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:38.816+0000 7fc6d77fe640 1 --2- 172.21.15.46:0/2896388918 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc6d81008a0 0x7fc6d819a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:27:38.820 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:38.816+0000 7fc6d7fff640 1 --2- 172.21.15.46:0/2896388918 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc6d81055f0 0x7fc6d819e900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:27:38.820 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:38.817+0000 7fc6d6ffd640 1 -- 172.21.15.46:0/2896388918 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc6d81008a0 msgr2=0x7fc6d819a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:27:38.820 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:38.817+0000 7fc6d6ffd640 1 --2- 172.21.15.46:0/2896388918 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc6d81008a0 0x7fc6d819a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:27:38.820 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:38.817+0000 7fc6d6ffd640 1 -- 172.21.15.46:0/2896388918 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc6d81055f0 msgr2=0x7fc6d819e900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:27:38.820 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:38.817+0000 7fc6d6ffd640 1 --2- 172.21.15.46:0/2896388918 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc6d81055f0 0x7fc6d819e900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:27:38.820 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:38.817+0000 7fc6d6ffd640 1 -- 172.21.15.46:0/2896388918 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc6d819f030 con 0x7fc6d8101260 2024-03-31T23:27:38.820 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:38.817+0000 7fc6d77fe640 1 --2- 172.21.15.46:0/2896388918 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc6d81008a0 0x7fc6d819a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:27:38.820 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:38.817+0000 7fc6d6ffd640 1 --2- 172.21.15.46:0/2896388918 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc6d8101260 0x7fc6d819a5d0 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7fc6cc00ea60 tx=0x7fc6cc00ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:27:38.820 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:38.817+0000 7fc6d4ff9640 1 -- 172.21.15.46:0/2896388918 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6cc00cde0 con 0x7fc6d8101260 2024-03-31T23:27:38.821 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:38.817+0000 7fc6d4ff9640 1 -- 172.21.15.46:0/2896388918 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc6cc0035f0 con 0x7fc6d8101260 2024-03-31T23:27:38.821 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:38.817+0000 7fc6ddcb5640 1 -- 172.21.15.46:0/2896388918 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc6d819f320 con 0x7fc6d8101260 2024-03-31T23:27:38.821 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:38.817+0000 7fc6ddcb5640 1 -- 172.21.15.46:0/2896388918 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc6d81a6bb0 con 0x7fc6d8101260 2024-03-31T23:27:38.821 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:38.818+0000 7fc6d4ff9640 1 -- 172.21.15.46:0/2896388918 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6cc010690 con 0x7fc6d8101260 2024-03-31T23:27:38.821 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:38.820+0000 7fc6d4ff9640 1 -- 172.21.15.46:0/2896388918 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 25) ==== 137451+0+0 (secure 0 0 0) 0x7fc6cc004000 con 0x7fc6d8101260 2024-03-31T23:27:38.823 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:38.821+0000 7fc6e0972740 1 Processor -- start 2024-03-31T23:27:38.823 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:38.821+0000 7fc6e0972740 1 -- start start 2024-03-31T23:27:38.823 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:38.821+0000 7fc6d4ff9640 1 --2- 172.21.15.46:0/2896388918 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fc6c00a2730 0x7fc6c00a4bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:27:38.823 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:38.821+0000 7fc6e0972740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x562b0f7606c0 0x562b0f760aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:27:38.824 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:38.821+0000 7fc6e0972740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562b0f760fe0 0x562b0f769870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:27:38.824 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:38.821+0000 7fc6e0972740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562b0f769db0 0x562b0f76c1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:27:38.824 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:38.821+0000 7fc6e0972740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x562b0f6c38f0 con 0x562b0f769db0 2024-03-31T23:27:38.824 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:38.821+0000 7fc6e0972740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x562b0f66c8d0 con 0x562b0f7606c0 2024-03-31T23:27:38.824 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:38.821+0000 7fc6e0972740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x562b0f6bf990 con 0x562b0f760fe0 2024-03-31T23:27:38.824 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:38.821+0000 7fc6d4ff9640 1 -- 172.21.15.46:0/2896388918 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(117..117 src has 1..117) ==== 9554+0+0 (secure 0 0 0) 0x7fc6cc0cf070 con 0x7fc6d8101260 2024-03-31T23:27:38.824 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:38.821+0000 7fc6d7fff640 1 --2- 172.21.15.46:0/2896388918 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fc6c00a2730 0x7fc6c00a4bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:27:38.824 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:38.821+0000 7fc6d6ffd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562b0f760fe0 0x562b0f769870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:27:38.824 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:38.821+0000 7fc6d77fe640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x562b0f7606c0 0x562b0f760aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:27:38.824 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:38.821+0000 7fc6d6ffd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562b0f760fe0 0x562b0f769870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.46:40056/0 (socket says 172.21.15.46:40056) 2024-03-31T23:27:38.824 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:38.821+0000 7fc6d6ffd640 1 -- 172.21.15.46:0/4174314406 learned_addr learned my addr 172.21.15.46:0/4174314406 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:27:38.824 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:38.822+0000 7fc6d6ffd640 1 -- 172.21.15.46:0/4174314406 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x562b0f7606c0 msgr2=0x562b0f760aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:27:38.824 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:38.822+0000 7fc6d6ffd640 1 --2- 172.21.15.46:0/4174314406 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x562b0f7606c0 0x562b0f760aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:27:38.824 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:38.822+0000 7fc6d6ffd640 1 -- 172.21.15.46:0/4174314406 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562b0f769db0 msgr2=0x562b0f76c1a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-31T23:27:38.825 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:38.822+0000 7fc6d7fff640 1 --2- 172.21.15.46:0/4174314406 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562b0f769db0 0x562b0f76c1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:27:38.825 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:38.822+0000 7fc6d6ffd640 1 --2- 172.21.15.46:0/4174314406 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562b0f769db0 0x562b0f76c1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:27:38.825 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:38.822+0000 7fc6d6ffd640 1 -- 172.21.15.46:0/4174314406 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562b0f3d8210 con 0x562b0f760fe0 2024-03-31T23:27:38.825 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:38.822+0000 7fc6d7fff640 1 --2- 172.21.15.46:0/4174314406 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562b0f769db0 0x562b0f76c1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T23:27:38.825 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:38.822+0000 7fc6d6ffd640 1 --2- 172.21.15.46:0/4174314406 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562b0f760fe0 0x562b0f769870 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fc6cc097d50 tx=0x7fc6cc09ace0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3d07e7302f97676c server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:27:38.825 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:38.822+0000 7fc6d7fff640 1 --2- 172.21.15.46:0/2896388918 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fc6c00a2730 0x7fc6c00a4bf0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fc6d0004af0 tx=0x7fc6d0005f30 comp rx=0 tx=0).ready entity=mgr.15654 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:27:38.825 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:38.822+0000 7fc6bdffb640 1 -- 172.21.15.46:0/4174314406 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6cc09ba80 con 0x562b0f760fe0 2024-03-31T23:27:38.825 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:38.822+0000 7fc6bdffb640 1 -- 172.21.15.46:0/4174314406 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc6cc011ad0 con 0x562b0f760fe0 2024-03-31T23:27:38.825 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:38.822+0000 7fc6bdffb640 1 -- 172.21.15.46:0/4174314406 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6cc0979f0 con 0x562b0f760fe0 2024-03-31T23:27:38.829 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:38.823+0000 7fc6e0972740 1 -- 172.21.15.46:0/4174314406 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562b0f760fe0 msgr2=0x562b0f769870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:27:38.829 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:38.823+0000 7fc6e0972740 1 --2- 172.21.15.46:0/4174314406 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562b0f760fe0 0x562b0f769870 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fc6cc097d50 tx=0x7fc6cc09ace0 comp rx=0 tx=0).stop 2024-03-31T23:27:38.829 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:38.823+0000 7fc6e0972740 1 -- 172.21.15.46:0/4174314406 shutdown_connections 2024-03-31T23:27:38.829 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:38.823+0000 7fc6e0972740 1 --2- 172.21.15.46:0/4174314406 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562b0f769db0 0x562b0f76c1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:27:38.829 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:38.823+0000 7fc6e0972740 1 --2- 172.21.15.46:0/4174314406 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562b0f760fe0 0x562b0f769870 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:27:38.829 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:38.823+0000 7fc6e0972740 1 --2- 172.21.15.46:0/4174314406 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x562b0f7606c0 0x562b0f760aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:27:38.829 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:38.823+0000 7fc6e0972740 1 -- 172.21.15.46:0/4174314406 >> 172.21.15.46:0/4174314406 conn(0x562b0f75afe0 msgr2=0x562b0f75b3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:27:38.829 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:38.823+0000 7fc6e0972740 1 -- 172.21.15.46:0/4174314406 shutdown_connections 2024-03-31T23:27:38.829 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:38.823+0000 7fc6e0972740 1 -- 172.21.15.46:0/4174314406 wait complete. 2024-03-31T23:27:38.829 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:38.823+0000 7fc6e0972740 1 -- start start 2024-03-31T23:27:38.830 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:38.824+0000 7fc6e0972740 1 Processor -- start 2024-03-31T23:27:38.830 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:38.824+0000 7fc6e0972740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562b0f7606c0 0x562b0f766d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:27:38.830 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:38.824+0000 7fc6e0972740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x562b0f760fe0 0x562b0f7672b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:27:38.830 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:38.824+0000 7fc6e0972740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562b0f769db0 0x562b0f7677f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:27:38.830 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:38.824+0000 7fc6e0972740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x562b0f76c9e0 con 0x562b0f7606c0 2024-03-31T23:27:38.830 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:38.824+0000 7fc6e0972740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x562b0f661f80 con 0x562b0f760fe0 2024-03-31T23:27:38.830 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:38.824+0000 7fc6e0972740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x562b0f76cb60 con 0x562b0f769db0 2024-03-31T23:27:38.830 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:38.824+0000 7fc6e0972740 10 client.0 _ll_register_callbacks cb 0x562b0f6ba290 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T23:27:38.830 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:38.824+0000 7fc6d77fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562b0f7606c0 0x562b0f766d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:27:38.830 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:38.824+0000 7fc6d77fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562b0f7606c0 0x562b0f766d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:45848/0 (socket says 172.21.15.46:45848) 2024-03-31T23:27:38.830 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:38.824+0000 7fc6d77fe640 1 -- 172.21.15.46:0/3518322529 learned_addr learned my addr 172.21.15.46:0/3518322529 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:27:38.830 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:38.824+0000 7fc6d7fff640 1 --2- 172.21.15.46:0/3518322529 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562b0f769db0 0x562b0f7677f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:27:38.830 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:38.824+0000 7fc6d6ffd640 1 --2- 172.21.15.46:0/3518322529 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x562b0f760fe0 0x562b0f7672b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:27:38.831 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:38.825+0000 7fc6d77fe640 1 -- 172.21.15.46:0/3518322529 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562b0f769db0 msgr2=0x562b0f7677f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:27:38.831 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:38.825+0000 7fc6d77fe640 1 --2- 172.21.15.46:0/3518322529 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562b0f769db0 0x562b0f7677f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:27:38.831 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:38.825+0000 7fc6d77fe640 1 -- 172.21.15.46:0/3518322529 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x562b0f760fe0 msgr2=0x562b0f7672b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:27:38.831 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:38.825+0000 7fc6d77fe640 1 --2- 172.21.15.46:0/3518322529 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x562b0f760fe0 0x562b0f7672b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:27:38.831 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:38.825+0000 7fc6d77fe640 1 -- 172.21.15.46:0/3518322529 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562b0f4a4620 con 0x562b0f7606c0 2024-03-31T23:27:38.831 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:38.825+0000 7fc6d77fe640 1 --2- 172.21.15.46:0/3518322529 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562b0f7606c0 0x562b0f766d70 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7fc6c8012d50 tx=0x7fc6c800f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a2f113397688af73 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:27:38.831 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:38.825+0000 7fc6b6ffd640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-31T23:27:38.831 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:38.825+0000 7fc6b6ffd640 1 -- 172.21.15.46:0/3518322529 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6c80120d0 con 0x562b0f7606c0 2024-03-31T23:27:38.831 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:38.825+0000 7fc6b6ffd640 1 -- 172.21.15.46:0/3518322529 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(117..117 src has 1..117) ==== 9554+0+0 (secure 0 0 0) 0x7fc6c8010430 con 0x562b0f7606c0 2024-03-31T23:27:38.831 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:38.825+0000 7fc6e0972740 1 -- 172.21.15.46:0/3518322529 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc6cc011ad0 con 0x562b0f7606c0 2024-03-31T23:27:38.831 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:38.825+0000 7fc6b6ffd640 1 -- 172.21.15.46:0/3518322529 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=41) ==== 24+0+0 (secure 0 0 0) 0x7fc6c8012270 con 0x562b0f7606c0 2024-03-31T23:27:38.831 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:38.825+0000 7fc6e0972740 10 client.17439 fetch_fsmap learned FSMap version 41 2024-03-31T23:27:38.831 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:38.825+0000 7fc6e0972740 1 -- 172.21.15.46:0/3518322529 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=41,monmap=4+}) -- 0x562b0f778820 con 0x562b0f7606c0 2024-03-31T23:27:38.831 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:38.826+0000 7fc6b6ffd640 1 -- 172.21.15.46:0/3518322529 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc6cc011ad0 con 0x562b0f7606c0 2024-03-31T23:27:38.832 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:38.826+0000 7fc6b6ffd640 1 -- 172.21.15.46:0/3518322529 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 41) ==== 8366+0+0 (secure 0 0 0) 0x7fc6c801b6f0 con 0x562b0f7606c0 2024-03-31T23:27:38.832 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:38.826+0000 7fc6e0972740 10 client.17439 fetch_fsmap finished waiting for FSMap version 41 2024-03-31T23:27:38.832 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:38.826+0000 7fc6e0972740 10 client.17439 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-31T23:27:38.832 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:38.826+0000 7fc6e0972740 20 client.17439 populate_metadata read hostname 'smithi046' 2024-03-31T23:27:38.832 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:38.826+0000 7fc6e0972740 1 --2- 172.21.15.46:0/3518322529 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x562b0f779e70 0x562b0f77c2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:27:38.832 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:38.826+0000 7fc6e0972740 4 client.17439 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-31T23:27:38.832 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:38.826+0000 7fc6e0972740 1 -- 172.21.15.46:0/3518322529 --> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562b0f6d7120 con 0x562b0f779e70 2024-03-31T23:27:38.832 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:38.826+0000 7fc6d6ffd640 1 --2- 172.21.15.46:0/3518322529 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x562b0f779e70 0x562b0f77c2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:27:38.832 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:38.827+0000 7fc6b6ffd640 10 client.17439 ms_handle_connect on v2:172.21.15.142:6838/2519143918 2024-03-31T23:27:38.832 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:38.827+0000 7fc6d6ffd640 1 --2- 172.21.15.46:0/3518322529 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x562b0f779e70 0x562b0f77c2d0 crc :-1 s=READY pgs=218 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cb92c4e7f333045e server_cookie=c9356320fb2be96b in_seq=0 out_seq=0 2024-03-31T23:27:38.833 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:38.829+0000 7fc6b6ffd640 1 -- 172.21.15.46:0/3518322529 <== mds.0 v2:172.21.15.142:6838/2519143918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x562b0f6d7120 con 0x562b0f779e70 2024-03-31T23:27:38.833 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:38.829+0000 7fc6b6ffd640 10 client.17439 handle_command_reply: tid=0 2024-03-31T23:27:38.833 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:38.829+0000 7fc6e0972740 1 client.17439 shutdown 2024-03-31T23:27:38.834 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:38.829+0000 7fc6e0972740 1 -- 172.21.15.46:0/3518322529 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562b0f7606c0 msgr2=0x562b0f766d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:27:38.834 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:38.829+0000 7fc6e0972740 1 --2- 172.21.15.46:0/3518322529 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562b0f7606c0 0x562b0f766d70 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7fc6c8012d50 tx=0x7fc6c800f790 comp rx=0 tx=0).stop 2024-03-31T23:27:38.834 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:38.829+0000 7fc6e0972740 1 -- 172.21.15.46:0/3518322529 shutdown_connections 2024-03-31T23:27:38.834 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:38.829+0000 7fc6e0972740 1 --2- 172.21.15.46:0/3518322529 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x562b0f779e70 0x562b0f77c2d0 crc :-1 s=READY pgs=218 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:27:38.834 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:38.829+0000 7fc6e0972740 1 --2- 172.21.15.46:0/3518322529 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562b0f769db0 0x562b0f7677f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:27:38.834 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:38.829+0000 7fc6e0972740 1 --2- 172.21.15.46:0/3518322529 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x562b0f760fe0 0x562b0f7672b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:27:38.834 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:38.829+0000 7fc6b6ffd640 0 client.17439 ms_handle_reset on v2:172.21.15.142:6838/2519143918 2024-03-31T23:27:38.834 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:38.829+0000 7fc6e0972740 1 --2- 172.21.15.46:0/3518322529 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562b0f7606c0 0x562b0f766d70 unknown :-1 s=CLOSED pgs=911 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:27:38.834 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:38.829+0000 7fc6e0972740 1 -- 172.21.15.46:0/3518322529 >> 172.21.15.46:0/3518322529 conn(0x562b0f75afe0 msgr2=0x562b0f75cd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:27:38.834 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:38.829+0000 7fc6e0972740 1 -- 172.21.15.46:0/3518322529 shutdown_connections 2024-03-31T23:27:38.834 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:38.829+0000 7fc6e0972740 1 -- 172.21.15.46:0/3518322529 wait complete. 2024-03-31T23:27:38.834 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:38.829+0000 7fc6e0972740 20 client.17439 trim_cache size 0 max 16384 2024-03-31T23:27:38.859 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:38.852+0000 7fc6e0972740 1 Processor -- start 2024-03-31T23:27:38.859 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:38.852+0000 7fc6e0972740 1 -- start start 2024-03-31T23:27:38.859 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:38.852+0000 7fc6e0972740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562b0f7606c0 0x562b0f7694a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:27:38.859 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:38.852+0000 7fc6e0972740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x562b0f760fe0 0x562b0f775780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:27:38.859 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:38.852+0000 7fc6e0972740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562b0f769db0 0x562b0f781ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:27:38.859 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:38.852+0000 7fc6e0972740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fc6c40129f0 con 0x562b0f769db0 2024-03-31T23:27:38.859 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:38.852+0000 7fc6e0972740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fc6c40101d0 con 0x562b0f760fe0 2024-03-31T23:27:38.860 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:38.852+0000 7fc6e0972740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fc6c400e040 con 0x562b0f7606c0 2024-03-31T23:27:38.860 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:38.852+0000 7fc6d7fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562b0f769db0 0x562b0f781ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:27:38.860 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:38.852+0000 7fc6d77fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562b0f7606c0 0x562b0f7694a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:27:38.860 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:38.852+0000 7fc6d7fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562b0f769db0 0x562b0f781ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:45858/0 (socket says 172.21.15.46:45858) 2024-03-31T23:27:38.860 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:38.852+0000 7fc6d77fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562b0f7606c0 0x562b0f7694a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.46:40086/0 (socket says 172.21.15.46:40086) 2024-03-31T23:27:38.860 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:38.852+0000 7fc6d7fff640 1 -- 172.21.15.46:0/3960546473 learned_addr learned my addr 172.21.15.46:0/3960546473 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:27:38.860 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:38.853+0000 7fc6d6ffd640 1 --2- 172.21.15.46:0/3960546473 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x562b0f760fe0 0x562b0f775780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:27:38.860 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:38.853+0000 7fc6d77fe640 1 -- 172.21.15.46:0/3960546473 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x562b0f760fe0 msgr2=0x562b0f775780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:27:38.860 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:38.853+0000 7fc6d77fe640 1 --2- 172.21.15.46:0/3960546473 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x562b0f760fe0 0x562b0f775780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:27:38.860 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:38.853+0000 7fc6d77fe640 1 -- 172.21.15.46:0/3960546473 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562b0f769db0 msgr2=0x562b0f781ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:27:38.860 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:38.853+0000 7fc6d77fe640 1 --2- 172.21.15.46:0/3960546473 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562b0f769db0 0x562b0f781ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:27:38.860 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:38.853+0000 7fc6d77fe640 1 -- 172.21.15.46:0/3960546473 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562b0f75f9a0 con 0x562b0f7606c0 2024-03-31T23:27:38.860 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:38.853+0000 7fc6d6ffd640 1 --2- 172.21.15.46:0/3960546473 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x562b0f760fe0 0x562b0f775780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T23:27:38.861 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:38.853+0000 7fc6d7fff640 1 --2- 172.21.15.46:0/3960546473 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562b0f769db0 0x562b0f781ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:27:38.861 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:38.853+0000 7fc6d77fe640 1 --2- 172.21.15.46:0/3960546473 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562b0f7606c0 0x562b0f7694a0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fc6c801bda0 tx=0x7fc6c8005970 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e7d7dd087a84fde8 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:27:38.861 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:38.853+0000 7fc6bd7fa640 1 -- 172.21.15.46:0/3960546473 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6c801b410 con 0x562b0f7606c0 2024-03-31T23:27:38.861 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:38.853+0000 7fc6bd7fa640 1 -- 172.21.15.46:0/3960546473 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc6c8005070 con 0x562b0f7606c0 2024-03-31T23:27:38.861 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:38.853+0000 7fc6bd7fa640 1 -- 172.21.15.46:0/3960546473 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6c80048c0 con 0x562b0f7606c0 2024-03-31T23:27:38.861 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:38.854+0000 7fc6e0972740 1 -- 172.21.15.46:0/3960546473 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562b0f7606c0 msgr2=0x562b0f7694a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:27:38.861 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:38.854+0000 7fc6e0972740 1 --2- 172.21.15.46:0/3960546473 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562b0f7606c0 0x562b0f7694a0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fc6c801bda0 tx=0x7fc6c8005970 comp rx=0 tx=0).stop 2024-03-31T23:27:38.861 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:38.854+0000 7fc6e0972740 1 -- 172.21.15.46:0/3960546473 shutdown_connections 2024-03-31T23:27:38.861 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:38.854+0000 7fc6e0972740 1 --2- 172.21.15.46:0/3960546473 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562b0f769db0 0x562b0f781ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:27:38.861 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:38.854+0000 7fc6e0972740 1 --2- 172.21.15.46:0/3960546473 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x562b0f760fe0 0x562b0f775780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:27:38.861 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:38.854+0000 7fc6e0972740 1 --2- 172.21.15.46:0/3960546473 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562b0f7606c0 0x562b0f7694a0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:27:38.861 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:38.854+0000 7fc6e0972740 1 -- 172.21.15.46:0/3960546473 >> 172.21.15.46:0/3960546473 conn(0x562b0f75afe0 msgr2=0x562b0f755bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:27:38.861 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:38.855+0000 7fc6e0972740 1 -- 172.21.15.46:0/3960546473 shutdown_connections 2024-03-31T23:27:38.861 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:38.855+0000 7fc6e0972740 1 -- 172.21.15.46:0/3960546473 wait complete. 2024-03-31T23:27:38.861 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:38.855+0000 7fc6e0972740 1 -- start start 2024-03-31T23:27:38.862 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:38.855+0000 7fc6e0972740 1 Processor -- start 2024-03-31T23:27:38.862 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:38.856+0000 7fc6e0972740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x562b0f7606c0 0x562b0f771ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:27:38.862 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:38.856+0000 7fc6e0972740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562b0f760fe0 0x562b0f76d370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:27:38.862 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:38.856+0000 7fc6e0972740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562b0f769db0 0x562b0f76d8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:27:38.862 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:38.856+0000 7fc6e0972740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fc6c400e690 con 0x562b0f769db0 2024-03-31T23:27:38.862 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:38.856+0000 7fc6e0972740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fc6c400fb00 con 0x562b0f7606c0 2024-03-31T23:27:38.862 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:38.856+0000 7fc6e0972740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fc6c400ed60 con 0x562b0f760fe0 2024-03-31T23:27:38.862 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:38.856+0000 7fc6e0972740 10 client.0 _ll_register_callbacks cb 0x7fc6c8012930 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T23:27:38.862 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:38.856+0000 7fc6d7fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562b0f769db0 0x562b0f76d8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:27:38.863 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:38.856+0000 7fc6d6ffd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562b0f760fe0 0x562b0f76d370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:27:38.863 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:38.856+0000 7fc6d6ffd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562b0f760fe0 0x562b0f76d370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.46:40100/0 (socket says 172.21.15.46:40100) 2024-03-31T23:27:38.863 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:38.856+0000 7fc6d77fe640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x562b0f7606c0 0x562b0f771ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:27:38.863 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:38.856+0000 7fc6d6ffd640 1 -- 172.21.15.46:0/3291558791 learned_addr learned my addr 172.21.15.46:0/3291558791 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:27:38.863 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:38.856+0000 7fc6d6ffd640 1 -- 172.21.15.46:0/3291558791 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x562b0f7606c0 msgr2=0x562b0f771ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:27:38.863 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:38.856+0000 7fc6d6ffd640 1 --2- 172.21.15.46:0/3291558791 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x562b0f7606c0 0x562b0f771ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:27:38.863 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:38.856+0000 7fc6d6ffd640 1 -- 172.21.15.46:0/3291558791 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562b0f769db0 msgr2=0x562b0f76d8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:27:38.863 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:38.856+0000 7fc6d6ffd640 1 --2- 172.21.15.46:0/3291558791 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562b0f769db0 0x562b0f76d8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:27:38.863 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:38.856+0000 7fc6d6ffd640 1 -- 172.21.15.46:0/3291558791 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562b0f76df50 con 0x562b0f760fe0 2024-03-31T23:27:38.863 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:38.857+0000 7fc6d6ffd640 1 --2- 172.21.15.46:0/3291558791 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562b0f760fe0 0x562b0f76d370 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fc6cc097d50 tx=0x7fc6cc097750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bc7cad0bc899f800 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:27:38.863 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:38.857+0000 7fc6b6ffd640 10 client.0 ms_handle_connect on v2:172.21.15.162:3300/0 2024-03-31T23:27:38.863 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:38.857+0000 7fc6b6ffd640 1 -- 172.21.15.46:0/3291558791 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6cc09ba80 con 0x562b0f760fe0 2024-03-31T23:27:38.865 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:38.857+0000 7fc6b6ffd640 1 -- 172.21.15.46:0/3291558791 <== mon.1 v2:172.21.15.162:3300/0 2 ==== osd_map(117..117 src has 1..117) ==== 9554+0+0 (secure 0 0 0) 0x7fc6cc0979f0 con 0x562b0f760fe0 2024-03-31T23:27:38.866 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:38.857+0000 7fc6e0972740 1 -- 172.21.15.46:0/3291558791 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc6c8005070 con 0x562b0f760fe0 2024-03-31T23:27:38.866 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:38.857+0000 7fc6b6ffd640 1 -- 172.21.15.46:0/3291558791 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_get_version_reply(handle=1 version=41) ==== 24+0+0 (secure 0 0 0) 0x7fc6cc011d80 con 0x562b0f760fe0 2024-03-31T23:27:38.866 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:38.857+0000 7fc6e0972740 10 client.27175 fetch_fsmap learned FSMap version 41 2024-03-31T23:27:38.866 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:38.857+0000 7fc6e0972740 1 -- 172.21.15.46:0/3291558791 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,fsmap=41,monmap=4+}) -- 0x562b0f7e0070 con 0x562b0f760fe0 2024-03-31T23:27:38.866 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:38.857+0000 7fc6b6ffd640 1 -- 172.21.15.46:0/3291558791 <== mon.1 v2:172.21.15.162:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc6c8005070 con 0x562b0f760fe0 2024-03-31T23:27:38.866 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:38.858+0000 7fc6b6ffd640 1 -- 172.21.15.46:0/3291558791 <== mon.1 v2:172.21.15.162:3300/0 5 ==== fsmap(e 41) ==== 8366+0+0 (secure 0 0 0) 0x7fc6cc080b90 con 0x562b0f760fe0 2024-03-31T23:27:38.866 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:38.858+0000 7fc6e0972740 10 client.27175 fetch_fsmap finished waiting for FSMap version 41 2024-03-31T23:27:38.866 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:38.858+0000 7fc6e0972740 10 client.27175 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-31T23:27:38.866 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:38.858+0000 7fc6e0972740 20 client.27175 populate_metadata read hostname 'smithi046' 2024-03-31T23:27:38.866 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:38.858+0000 7fc6e0972740 1 --2- 172.21.15.46:0/3291558791 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x562b0f7e03b0 0x562b0f7e27a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:27:38.867 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:38.858+0000 7fc6e0972740 4 client.27175 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-31T23:27:38.867 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:38.858+0000 7fc6e0972740 1 -- 172.21.15.46:0/3291558791 --> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] -- command(tid 0: {"prefix": "flush journal"}) -- 0x562b0f6d0e30 con 0x562b0f7e03b0 2024-03-31T23:27:38.867 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:38.858+0000 7fc6d77fe640 1 --2- 172.21.15.46:0/3291558791 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x562b0f7e03b0 0x562b0f7e27a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:27:38.867 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:38.858+0000 7fc6b6ffd640 10 client.27175 ms_handle_connect on v2:172.21.15.142:6838/2519143918 2024-03-31T23:27:38.867 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:38.858+0000 7fc6d77fe640 1 --2- 172.21.15.46:0/3291558791 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x562b0f7e03b0 0x562b0f7e27a0 crc :-1 s=READY pgs=219 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c2b05e548743da1e server_cookie=a511579716d84d28 in_seq=0 out_seq=0 2024-03-31T23:27:39.045 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:39.043+0000 7fc6b6ffd640 1 -- 172.21.15.46:0/3291558791 <== mds.0 v2:172.21.15.142:6838/2519143918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x562b0f6d0e30 con 0x562b0f7e03b0 2024-03-31T23:27:39.045 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:39.043+0000 7fc6b6ffd640 10 client.27175 handle_command_reply: tid=0 2024-03-31T23:27:39.045 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:39.043+0000 7fc6e0972740 1 client.27175 shutdown 2024-03-31T23:27:39.045 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:39.043+0000 7fc6e0972740 1 -- 172.21.15.46:0/3291558791 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562b0f760fe0 msgr2=0x562b0f76d370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:27:39.045 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:39.043+0000 7fc6e0972740 1 --2- 172.21.15.46:0/3291558791 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562b0f760fe0 0x562b0f76d370 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fc6cc097d50 tx=0x7fc6cc097750 comp rx=0 tx=0).stop 2024-03-31T23:27:39.045 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:39.043+0000 7fc6e0972740 1 -- 172.21.15.46:0/3291558791 shutdown_connections 2024-03-31T23:27:39.045 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:39.043+0000 7fc6e0972740 1 --2- 172.21.15.46:0/3291558791 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x562b0f7e03b0 0x562b0f7e27a0 crc :-1 s=READY pgs=219 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:27:39.045 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:39.044+0000 7fc6e0972740 1 --2- 172.21.15.46:0/3291558791 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562b0f769db0 0x562b0f76d8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:27:39.045 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:39.044+0000 7fc6e0972740 1 --2- 172.21.15.46:0/3291558791 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562b0f760fe0 0x562b0f76d370 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:27:39.046 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:39.044+0000 7fc6b6ffd640 0 client.27175 ms_handle_reset on v2:172.21.15.142:6838/2519143918 2024-03-31T23:27:39.046 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:39.044+0000 7fc6e0972740 1 --2- 172.21.15.46:0/3291558791 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x562b0f7606c0 0x562b0f771ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:27:39.046 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:39.044+0000 7fc6e0972740 1 -- 172.21.15.46:0/3291558791 >> 172.21.15.46:0/3291558791 conn(0x562b0f75afe0 msgr2=0x562b0f7596f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:27:39.046 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:39.044+0000 7fc6e0972740 1 -- 172.21.15.46:0/3291558791 shutdown_connections 2024-03-31T23:27:39.046 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:39.044+0000 7fc6e0972740 1 -- 172.21.15.46:0/3291558791 wait complete. 2024-03-31T23:27:39.046 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:39.044+0000 7fc6e0972740 20 client.27175 trim_cache size 0 max 16384 2024-03-31T23:27:39.046 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout:{ 2024-03-31T23:27:39.046 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout: "message": "", 2024-03-31T23:27:39.046 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout: "return_code": 0 2024-03-31T23:27:39.046 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout:} 2024-03-31T23:27:39.047 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:39.044+0000 7fc6ddcb5640 1 -- 172.21.15.46:0/2896388918 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fc6c00a2730 msgr2=0x7fc6c00a4bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:27:39.047 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:39.044+0000 7fc6ddcb5640 1 --2- 172.21.15.46:0/2896388918 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fc6c00a2730 0x7fc6c00a4bf0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fc6d0004af0 tx=0x7fc6d0005f30 comp rx=0 tx=0).stop 2024-03-31T23:27:39.047 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:39.044+0000 7fc6ddcb5640 1 -- 172.21.15.46:0/2896388918 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc6d8101260 msgr2=0x7fc6d819a5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:27:39.047 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:39.044+0000 7fc6ddcb5640 1 --2- 172.21.15.46:0/2896388918 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc6d8101260 0x7fc6d819a5d0 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7fc6cc00ea60 tx=0x7fc6cc00ef20 comp rx=0 tx=0).stop 2024-03-31T23:27:39.047 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:39.045+0000 7fc6ddcb5640 1 -- 172.21.15.46:0/2896388918 shutdown_connections 2024-03-31T23:27:39.048 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:39.045+0000 7fc6ddcb5640 1 --2- 172.21.15.46:0/2896388918 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fc6c00a2730 0x7fc6c00a4bf0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:27:39.048 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:39.045+0000 7fc6ddcb5640 1 --2- 172.21.15.46:0/2896388918 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc6d81055f0 0x7fc6d819e900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:27:39.048 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:39.045+0000 7fc6ddcb5640 1 --2- 172.21.15.46:0/2896388918 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc6d8101260 0x7fc6d819a5d0 unknown :-1 s=CLOSED pgs=910 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:27:39.048 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:39.045+0000 7fc6ddcb5640 1 --2- 172.21.15.46:0/2896388918 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc6d81008a0 0x7fc6d819a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:27:39.048 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:39.045+0000 7fc6ddcb5640 1 -- 172.21.15.46:0/2896388918 >> 172.21.15.46:0/2896388918 conn(0x7fc6d80774a0 msgr2=0x7fc6d8068ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:27:39.048 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:39.045+0000 7fc6ddcb5640 1 -- 172.21.15.46:0/2896388918 shutdown_connections 2024-03-31T23:27:39.048 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:39.045+0000 7fc6ddcb5640 1 -- 172.21.15.46:0/2896388918 wait complete. 2024-03-31T23:27:39.277 INFO:tasks.workunit.client.0.smithi046.stdout: 10 26775 8.76 MB/sec execute 44 sec latency 135.302 ms 2024-03-31T23:27:40.277 INFO:tasks.workunit.client.0.smithi046.stdout: 10 26974 8.88 MB/sec execute 45 sec latency 150.356 ms 2024-03-31T23:27:40.295 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:27:39 smithi162 ceph-mon[36237]: pgmap v502: 129 pgs: 129 active+clean; 581 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 6.2 MiB/s wr, 380 op/s 2024-03-31T23:27:40.361 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:27:39 smithi046 ceph-mon[32498]: pgmap v502: 129 pgs: 129 active+clean; 581 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 6.2 MiB/s wr, 380 op/s 2024-03-31T23:27:40.443 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:27:39 smithi142 ceph-mon[35271]: pgmap v502: 129 pgs: 129 active+clean; 581 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 6.2 MiB/s wr, 380 op/s 2024-03-31T23:27:41.278 INFO:tasks.workunit.client.0.smithi046.stdout: 10 27121 8.83 MB/sec execute 46 sec latency 116.611 ms 2024-03-31T23:27:42.277 INFO:tasks.workunit.client.0.smithi046.stdout: 10 27271 8.75 MB/sec execute 47 sec latency 73.742 ms 2024-03-31T23:27:42.295 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:27:41 smithi162 ceph-mon[36237]: pgmap v503: 129 pgs: 129 active+clean; 581 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 4.4 MiB/s wr, 229 op/s 2024-03-31T23:27:42.295 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:27:41 smithi162 ceph-mon[36237]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-03-31T23:27:42.361 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:27:41 smithi046 ceph-mon[32498]: pgmap v503: 129 pgs: 129 active+clean; 581 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 4.4 MiB/s wr, 229 op/s 2024-03-31T23:27:42.361 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:27:41 smithi046 ceph-mon[32498]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-03-31T23:27:42.443 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:27:41 smithi142 ceph-mon[35271]: pgmap v503: 129 pgs: 129 active+clean; 581 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 4.4 MiB/s wr, 229 op/s 2024-03-31T23:27:42.443 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:27:41 smithi142 ceph-mon[35271]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-03-31T23:27:43.277 INFO:tasks.workunit.client.0.smithi046.stdout: 10 27452 8.75 MB/sec execute 48 sec latency 68.397 ms 2024-03-31T23:27:44.277 INFO:tasks.workunit.client.0.smithi046.stdout: 10 27592 8.68 MB/sec execute 49 sec latency 82.928 ms 2024-03-31T23:27:44.295 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:27:43 smithi162 ceph-mon[36237]: pgmap v504: 129 pgs: 129 active+clean; 581 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 4.4 MiB/s wr, 229 op/s 2024-03-31T23:27:44.361 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:27:43 smithi046 ceph-mon[32498]: pgmap v504: 129 pgs: 129 active+clean; 581 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 4.4 MiB/s wr, 229 op/s 2024-03-31T23:27:44.443 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:27:43 smithi142 ceph-mon[35271]: pgmap v504: 129 pgs: 129 active+clean; 581 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 4.4 MiB/s wr, 229 op/s 2024-03-31T23:27:45.277 INFO:tasks.workunit.client.0.smithi046.stdout: 10 27734 8.64 MB/sec execute 50 sec latency 81.602 ms 2024-03-31T23:27:46.277 INFO:tasks.workunit.client.0.smithi046.stdout: 10 27914 8.73 MB/sec execute 51 sec latency 103.842 ms 2024-03-31T23:27:46.295 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:27:46 smithi162 ceph-mon[36237]: pgmap v505: 129 pgs: 129 active+clean; 584 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 6.6 MiB/s wr, 409 op/s 2024-03-31T23:27:46.361 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:27:46 smithi046 ceph-mon[32498]: pgmap v505: 129 pgs: 129 active+clean; 584 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 6.6 MiB/s wr, 409 op/s 2024-03-31T23:27:46.443 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:27:46 smithi142 ceph-mon[35271]: pgmap v505: 129 pgs: 129 active+clean; 584 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 6.6 MiB/s wr, 409 op/s 2024-03-31T23:27:47.277 INFO:tasks.workunit.client.0.smithi046.stdout: 10 28082 8.71 MB/sec execute 52 sec latency 90.843 ms 2024-03-31T23:27:48.277 INFO:tasks.workunit.client.0.smithi046.stdout: 10 28241 8.68 MB/sec execute 53 sec latency 105.527 ms 2024-03-31T23:27:48.295 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:27:48 smithi162 ceph-mon[36237]: pgmap v506: 129 pgs: 129 active+clean; 584 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 4.5 MiB/s wr, 297 op/s 2024-03-31T23:27:48.361 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:27:48 smithi046 ceph-mon[32498]: pgmap v506: 129 pgs: 129 active+clean; 584 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 4.5 MiB/s wr, 297 op/s 2024-03-31T23:27:48.443 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:27:48 smithi142 ceph-mon[35271]: pgmap v506: 129 pgs: 129 active+clean; 584 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 4.5 MiB/s wr, 297 op/s 2024-03-31T23:27:49.277 INFO:tasks.workunit.client.0.smithi046.stdout: 10 28412 8.66 MB/sec execute 54 sec latency 73.135 ms 2024-03-31T23:27:50.278 INFO:tasks.workunit.client.0.smithi046.stdout: 10 28574 8.64 MB/sec execute 55 sec latency 96.034 ms 2024-03-31T23:27:50.295 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:27:50 smithi162 ceph-mon[36237]: pgmap v507: 129 pgs: 129 active+clean; 589 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 6.5 MiB/s wr, 417 op/s 2024-03-31T23:27:50.361 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:27:50 smithi046 ceph-mon[32498]: pgmap v507: 129 pgs: 129 active+clean; 589 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 6.5 MiB/s wr, 417 op/s 2024-03-31T23:27:50.443 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:27:50 smithi142 ceph-mon[35271]: pgmap v507: 129 pgs: 129 active+clean; 589 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 6.5 MiB/s wr, 417 op/s 2024-03-31T23:27:51.278 INFO:tasks.workunit.client.0.smithi046.stdout: 10 28726 8.56 MB/sec execute 56 sec latency 81.969 ms 2024-03-31T23:27:52.168 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:52.164+0000 7f47c498a640 1 -- 172.21.15.46:0/2515007766 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f47c01008a0 msgr2=0x7f47c0100d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:27:52.168 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:52.164+0000 7f47c498a640 1 --2- 172.21.15.46:0/2515007766 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f47c01008a0 0x7f47c0100d20 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7f47b4009900 tx=0x7f47b402bc30 comp rx=0 tx=0).stop 2024-03-31T23:27:52.168 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:52.164+0000 7f47c498a640 1 -- 172.21.15.46:0/2515007766 shutdown_connections 2024-03-31T23:27:52.168 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:52.164+0000 7f47c498a640 1 --2- 172.21.15.46:0/2515007766 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f47c0101260 0x7f47c010d480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:27:52.169 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:52.164+0000 7f47c498a640 1 --2- 172.21.15.46:0/2515007766 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f47c01008a0 0x7f47c0100d20 unknown :-1 s=CLOSED pgs=912 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:27:52.169 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:52.164+0000 7f47c498a640 1 --2- 172.21.15.46:0/2515007766 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f47c01055f0 0x7f47c01059d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:27:52.169 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:52.164+0000 7f47c498a640 1 -- 172.21.15.46:0/2515007766 >> 172.21.15.46:0/2515007766 conn(0x7f47c00774a0 msgr2=0x7f47c00778b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:27:52.169 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:52.164+0000 7f47c498a640 1 -- 172.21.15.46:0/2515007766 shutdown_connections 2024-03-31T23:27:52.169 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:52.164+0000 7f47c498a640 1 -- 172.21.15.46:0/2515007766 wait complete. 2024-03-31T23:27:52.169 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:52.165+0000 7f47c498a640 1 Processor -- start 2024-03-31T23:27:52.169 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:52.165+0000 7f47c498a640 1 -- start start 2024-03-31T23:27:52.169 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:52.165+0000 7f47c498a640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f47c01008a0 0x7f47c019a1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:27:52.169 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:52.165+0000 7f47c498a640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f47c0101260 0x7f47c019a6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:27:52.169 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:52.165+0000 7f47c498a640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f47c01055f0 0x7f47c019ea60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:27:52.169 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:52.165+0000 7f47c498a640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f47c01109c0 con 0x7f47c0101260 2024-03-31T23:27:52.169 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:52.165+0000 7f47c498a640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f47c0110840 con 0x7f47c01055f0 2024-03-31T23:27:52.169 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:52.165+0000 7f47c498a640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f47c0110b40 con 0x7f47c01008a0 2024-03-31T23:27:52.170 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:52.165+0000 7f47bdd9b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f47c0101260 0x7f47c019a6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:27:52.170 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:52.165+0000 7f47be59c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f47c01008a0 0x7f47c019a1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:27:52.170 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:52.165+0000 7f47bdd9b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f47c0101260 0x7f47c019a6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:34796/0 (socket says 172.21.15.46:34796) 2024-03-31T23:27:52.170 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:52.165+0000 7f47bdd9b640 1 -- 172.21.15.46:0/3862485176 learned_addr learned my addr 172.21.15.46:0/3862485176 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:27:52.170 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:52.165+0000 7f47bed9d640 1 --2- 172.21.15.46:0/3862485176 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f47c01055f0 0x7f47c019ea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:27:52.170 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:52.166+0000 7f47be59c640 1 -- 172.21.15.46:0/3862485176 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f47c01055f0 msgr2=0x7f47c019ea60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:27:52.170 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:52.166+0000 7f47be59c640 1 --2- 172.21.15.46:0/3862485176 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f47c01055f0 0x7f47c019ea60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:27:52.170 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:52.166+0000 7f47be59c640 1 -- 172.21.15.46:0/3862485176 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f47c0101260 msgr2=0x7f47c019a6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:27:52.170 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:52.166+0000 7f47be59c640 1 --2- 172.21.15.46:0/3862485176 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f47c0101260 0x7f47c019a6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:27:52.170 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:52.166+0000 7f47be59c640 1 -- 172.21.15.46:0/3862485176 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f47c019f190 con 0x7f47c01008a0 2024-03-31T23:27:52.170 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:52.166+0000 7f47bed9d640 1 --2- 172.21.15.46:0/3862485176 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f47c01055f0 0x7f47c019ea60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T23:27:52.171 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:52.166+0000 7f47be59c640 1 --2- 172.21.15.46:0/3862485176 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f47c01008a0 0x7f47c019a1a0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f47b000e9f0 tx=0x7f47b000eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:27:52.171 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:52.166+0000 7f47af7fe640 1 -- 172.21.15.46:0/3862485176 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47b000ccc0 con 0x7f47c01008a0 2024-03-31T23:27:52.171 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:52.166+0000 7f47af7fe640 1 -- 172.21.15.46:0/3862485176 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f47b0005020 con 0x7f47c01008a0 2024-03-31T23:27:52.171 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:52.166+0000 7f47c498a640 1 -- 172.21.15.46:0/3862485176 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f47c019f480 con 0x7f47c01008a0 2024-03-31T23:27:52.171 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:52.166+0000 7f47af7fe640 1 -- 172.21.15.46:0/3862485176 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47b0004a00 con 0x7f47c01008a0 2024-03-31T23:27:52.171 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:52.166+0000 7f47c498a640 1 -- 172.21.15.46:0/3862485176 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f47c01a6d10 con 0x7f47c01008a0 2024-03-31T23:27:52.177 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:52.169+0000 7f47af7fe640 1 -- 172.21.15.46:0/3862485176 <== mon.1 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 25) ==== 137451+0+0 (secure 0 0 0) 0x7f47b0004ba0 con 0x7f47c01008a0 2024-03-31T23:27:52.177 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:52.169+0000 7f47c756f740 1 Processor -- start 2024-03-31T23:27:52.177 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:52.170+0000 7f47c756f740 1 -- start start 2024-03-31T23:27:52.177 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:52.170+0000 7f47c756f740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56282ac52ed0 0x56282ac532b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:27:52.177 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:52.170+0000 7f47c756f740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56282ac537f0 0x56282ac5c080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:27:52.177 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:52.170+0000 7f47c756f740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56282ac5c5c0 0x56282ac5e9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:27:52.177 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:52.170+0000 7f47c756f740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x56282abad8f0 con 0x56282ac52ed0 2024-03-31T23:27:52.177 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:52.170+0000 7f47c756f740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x56282ab568d0 con 0x56282ac537f0 2024-03-31T23:27:52.177 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:52.170+0000 7f47c756f740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x56282aba9990 con 0x56282ac5c5c0 2024-03-31T23:27:52.177 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:52.170+0000 7f47bed9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56282ac52ed0 0x56282ac532b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:27:52.177 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:52.170+0000 7f47bed9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56282ac52ed0 0x56282ac532b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:34810/0 (socket says 172.21.15.46:34810) 2024-03-31T23:27:52.177 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:52.170+0000 7f47bed9d640 1 -- 172.21.15.46:0/397760086 learned_addr learned my addr 172.21.15.46:0/397760086 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:27:52.178 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:52.170+0000 7f47be59c640 1 --2- 172.21.15.46:0/397760086 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56282ac537f0 0x56282ac5c080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:27:52.178 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:52.170+0000 7f47bdd9b640 1 --2- 172.21.15.46:0/397760086 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56282ac5c5c0 0x56282ac5e9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:27:52.178 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:52.170+0000 7f47af7fe640 1 --2- 172.21.15.46:0/3862485176 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f47a80a2730 0x7f47a80a4bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:27:52.178 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:52.170+0000 7f47bdd9b640 1 -- 172.21.15.46:0/397760086 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56282ac537f0 msgr2=0x56282ac5c080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:27:52.178 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:52.170+0000 7f47bdd9b640 1 --2- 172.21.15.46:0/397760086 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56282ac537f0 0x56282ac5c080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:27:52.178 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:52.170+0000 7f47bdd9b640 1 -- 172.21.15.46:0/397760086 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56282ac52ed0 msgr2=0x56282ac532b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:27:52.178 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:52.170+0000 7f47bdd9b640 1 --2- 172.21.15.46:0/397760086 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56282ac52ed0 0x56282ac532b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:27:52.178 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:52.170+0000 7f47bdd9b640 1 -- 172.21.15.46:0/397760086 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56282a8c2210 con 0x56282ac5c5c0 2024-03-31T23:27:52.178 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:52.170+0000 7f47bed9d640 1 --2- 172.21.15.46:0/397760086 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56282ac52ed0 0x56282ac532b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:27:52.178 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:52.170+0000 7f47bed9d640 1 --2- 172.21.15.46:0/3862485176 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f47a80a2730 0x7f47a80a4bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:27:52.178 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:52.170+0000 7f47af7fe640 1 -- 172.21.15.46:0/3862485176 <== mon.1 v2:172.21.15.162:3300/0 5 ==== osd_map(117..117 src has 1..117) ==== 9554+0+0 (secure 0 0 0) 0x7f47b00cee60 con 0x7f47c01008a0 2024-03-31T23:27:52.178 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:52.171+0000 7f47bdd9b640 1 --2- 172.21.15.46:0/397760086 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56282ac5c5c0 0x56282ac5e9b0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f47b4009230 tx=0x7f47b4005e10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=795a17db2e41fcac server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:27:52.179 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:52.171+0000 7f47acff9640 1 -- 172.21.15.46:0/397760086 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47b402c8c0 con 0x56282ac5c5c0 2024-03-31T23:27:52.179 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:52.171+0000 7f47bed9d640 1 --2- 172.21.15.46:0/3862485176 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f47a80a2730 0x7f47a80a4bf0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f47b8005c40 tx=0x7f47b8004a40 comp rx=0 tx=0).ready entity=mgr.15654 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:27:52.179 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:52.171+0000 7f47acff9640 1 -- 172.21.15.46:0/397760086 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f47b402ca60 con 0x56282ac5c5c0 2024-03-31T23:27:52.179 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:52.171+0000 7f47acff9640 1 -- 172.21.15.46:0/397760086 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47b40046c0 con 0x56282ac5c5c0 2024-03-31T23:27:52.179 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:52.172+0000 7f47c756f740 1 -- 172.21.15.46:0/397760086 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56282ac5c5c0 msgr2=0x56282ac5e9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:27:52.179 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:52.172+0000 7f47c756f740 1 --2- 172.21.15.46:0/397760086 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56282ac5c5c0 0x56282ac5e9b0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f47b4009230 tx=0x7f47b4005e10 comp rx=0 tx=0).stop 2024-03-31T23:27:52.179 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:52.172+0000 7f47c756f740 1 -- 172.21.15.46:0/397760086 shutdown_connections 2024-03-31T23:27:52.179 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:52.172+0000 7f47c756f740 1 --2- 172.21.15.46:0/397760086 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56282ac5c5c0 0x56282ac5e9b0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:27:52.179 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:52.172+0000 7f47c756f740 1 --2- 172.21.15.46:0/397760086 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56282ac537f0 0x56282ac5c080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:27:52.179 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:52.172+0000 7f47c756f740 1 --2- 172.21.15.46:0/397760086 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56282ac52ed0 0x56282ac532b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:27:52.179 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:52.172+0000 7f47c756f740 1 -- 172.21.15.46:0/397760086 >> 172.21.15.46:0/397760086 conn(0x56282ac4d7f0 msgr2=0x56282ac4dbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:27:52.179 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:52.172+0000 7f47c756f740 1 -- 172.21.15.46:0/397760086 shutdown_connections 2024-03-31T23:27:52.179 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:52.172+0000 7f47c756f740 1 -- 172.21.15.46:0/397760086 wait complete. 2024-03-31T23:27:52.179 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:52.172+0000 7f47c756f740 1 -- start start 2024-03-31T23:27:52.179 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:52.173+0000 7f47c756f740 1 Processor -- start 2024-03-31T23:27:52.180 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:52.173+0000 7f47c756f740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56282ac52ed0 0x56282ac595d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:27:52.180 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:52.173+0000 7f47c756f740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56282ac537f0 0x56282ac59b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:27:52.180 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:52.173+0000 7f47c756f740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56282ac5c5c0 0x56282ac5a050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:27:52.180 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:52.173+0000 7f47c756f740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x56282ac5f190 con 0x56282ac537f0 2024-03-31T23:27:52.180 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:52.173+0000 7f47c756f740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x56282ab4bf80 con 0x56282ac52ed0 2024-03-31T23:27:52.180 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:52.173+0000 7f47c756f740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x56282ac5f310 con 0x56282ac5c5c0 2024-03-31T23:27:52.180 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:52.173+0000 7f47c756f740 10 client.0 _ll_register_callbacks cb 0x56282aba4290 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T23:27:52.180 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:52.173+0000 7f47be59c640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56282ac52ed0 0x56282ac595d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:27:52.180 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:52.173+0000 7f47bed9d640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56282ac5c5c0 0x56282ac5a050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:27:52.180 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:52.173+0000 7f47be59c640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56282ac52ed0 0x56282ac595d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.46:46752/0 (socket says 172.21.15.46:46752) 2024-03-31T23:27:52.180 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:52.173+0000 7f47be59c640 1 -- 172.21.15.46:0/3445152815 learned_addr learned my addr 172.21.15.46:0/3445152815 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:27:52.180 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:52.173+0000 7f47bdd9b640 1 --2- 172.21.15.46:0/3445152815 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56282ac537f0 0x56282ac59b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:27:52.181 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:52.174+0000 7f47be59c640 1 -- 172.21.15.46:0/3445152815 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56282ac5c5c0 msgr2=0x56282ac5a050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:27:52.181 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:52.174+0000 7f47be59c640 1 --2- 172.21.15.46:0/3445152815 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56282ac5c5c0 0x56282ac5a050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:27:52.181 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:52.174+0000 7f47be59c640 1 -- 172.21.15.46:0/3445152815 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56282ac537f0 msgr2=0x56282ac59b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:27:52.181 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:52.174+0000 7f47be59c640 1 --2- 172.21.15.46:0/3445152815 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56282ac537f0 0x56282ac59b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:27:52.181 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:52.174+0000 7f47be59c640 1 -- 172.21.15.46:0/3445152815 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56282a98e620 con 0x56282ac52ed0 2024-03-31T23:27:52.181 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:52.174+0000 7f47be59c640 1 --2- 172.21.15.46:0/3445152815 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56282ac52ed0 0x56282ac595d0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f47b000c6a0 tx=0x7f47b0009f60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=edb1290e5d3c0fa4 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:27:52.181 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:52.174+0000 7f47a5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.142:3300/0 2024-03-31T23:27:52.182 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:52.174+0000 7f47a5ffb640 1 -- 172.21.15.46:0/3445152815 <== mon.2 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47b001c040 con 0x56282ac52ed0 2024-03-31T23:27:52.182 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:52.174+0000 7f47a5ffb640 1 -- 172.21.15.46:0/3445152815 <== mon.2 v2:172.21.15.142:3300/0 2 ==== osd_map(117..117 src has 1..117) ==== 9554+0+0 (secure 0 0 0) 0x7f47b0097840 con 0x56282ac52ed0 2024-03-31T23:27:52.182 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:52.174+0000 7f47c756f740 1 -- 172.21.15.46:0/3445152815 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f47b402ca60 con 0x56282ac52ed0 2024-03-31T23:27:52.182 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:52.174+0000 7f47a5ffb640 1 -- 172.21.15.46:0/3445152815 <== mon.2 v2:172.21.15.142:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) ==== 24+0+0 (secure 0 0 0) 0x7f47b009ab30 con 0x56282ac52ed0 2024-03-31T23:27:52.182 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:52.174+0000 7f47c756f740 10 client.27287 fetch_fsmap learned FSMap version 42 2024-03-31T23:27:52.182 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:52.174+0000 7f47c756f740 1 -- 172.21.15.46:0/3445152815 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=4+}) -- 0x56282ac6ae80 con 0x56282ac52ed0 2024-03-31T23:27:52.182 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:52.175+0000 7f47a5ffb640 1 -- 172.21.15.46:0/3445152815 <== mon.2 v2:172.21.15.142:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f47b402ca60 con 0x56282ac52ed0 2024-03-31T23:27:52.182 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:52.175+0000 7f47a5ffb640 1 -- 172.21.15.46:0/3445152815 <== mon.2 v2:172.21.15.142:3300/0 5 ==== fsmap(e 42) ==== 8362+0+0 (secure 0 0 0) 0x7f47b00a3750 con 0x56282ac52ed0 2024-03-31T23:27:52.182 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:52.175+0000 7f47c756f740 10 client.27287 fetch_fsmap finished waiting for FSMap version 42 2024-03-31T23:27:52.182 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:52.175+0000 7f47c756f740 10 client.27287 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-31T23:27:52.182 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:52.175+0000 7f47c756f740 20 client.27287 populate_metadata read hostname 'smithi046' 2024-03-31T23:27:52.182 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:52.175+0000 7f47c756f740 1 --2- 172.21.15.46:0/3445152815 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x56282ac6c340 0x56282ac6e790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:27:52.183 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:52.175+0000 7f47c756f740 4 client.27287 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-31T23:27:52.183 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:52.175+0000 7f47c756f740 1 -- 172.21.15.46:0/3445152815 --> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56282abc1120 con 0x56282ac6c340 2024-03-31T23:27:52.183 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:52.175+0000 7f47bdd9b640 1 --2- 172.21.15.46:0/3445152815 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x56282ac6c340 0x56282ac6e790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:27:52.184 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:52.176+0000 7f47a5ffb640 10 client.27287 ms_handle_connect on v2:172.21.15.142:6838/2519143918 2024-03-31T23:27:52.184 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:52.176+0000 7f47bdd9b640 1 --2- 172.21.15.46:0/3445152815 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x56282ac6c340 0x56282ac6e790 crc :-1 s=READY pgs=220 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=787be6493aa9f194 server_cookie=4f1645c84226db88 in_seq=0 out_seq=0 2024-03-31T23:27:52.184 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:52.177+0000 7f47a5ffb640 1 -- 172.21.15.46:0/3445152815 <== mds.0 v2:172.21.15.142:6838/2519143918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x56282abc1120 con 0x56282ac6c340 2024-03-31T23:27:52.184 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:52.177+0000 7f47a5ffb640 10 client.27287 handle_command_reply: tid=0 2024-03-31T23:27:52.184 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:52.177+0000 7f47c756f740 1 client.27287 shutdown 2024-03-31T23:27:52.184 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:52.178+0000 7f47c756f740 1 -- 172.21.15.46:0/3445152815 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56282ac52ed0 msgr2=0x56282ac595d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:27:52.184 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:52.178+0000 7f47c756f740 1 --2- 172.21.15.46:0/3445152815 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56282ac52ed0 0x56282ac595d0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f47b000c6a0 tx=0x7f47b0009f60 comp rx=0 tx=0).stop 2024-03-31T23:27:52.185 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:52.178+0000 7f47c756f740 1 -- 172.21.15.46:0/3445152815 shutdown_connections 2024-03-31T23:27:52.185 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:52.178+0000 7f47c756f740 1 --2- 172.21.15.46:0/3445152815 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x56282ac6c340 0x56282ac6e790 crc :-1 s=READY pgs=220 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:27:52.185 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:52.178+0000 7f47c756f740 1 --2- 172.21.15.46:0/3445152815 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56282ac5c5c0 0x56282ac5a050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:27:52.185 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:52.178+0000 7f47a5ffb640 0 client.27287 ms_handle_reset on v2:172.21.15.142:6838/2519143918 2024-03-31T23:27:52.185 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:52.178+0000 7f47c756f740 1 --2- 172.21.15.46:0/3445152815 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56282ac537f0 0x56282ac59b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:27:52.185 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:52.178+0000 7f47c756f740 1 --2- 172.21.15.46:0/3445152815 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56282ac52ed0 0x56282ac595d0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:27:52.185 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:52.178+0000 7f47c756f740 1 -- 172.21.15.46:0/3445152815 >> 172.21.15.46:0/3445152815 conn(0x56282ac4d7f0 msgr2=0x56282ac4f540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:27:52.185 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:52.178+0000 7f47c756f740 1 -- 172.21.15.46:0/3445152815 shutdown_connections 2024-03-31T23:27:52.185 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:52.178+0000 7f47c756f740 1 -- 172.21.15.46:0/3445152815 wait complete. 2024-03-31T23:27:52.185 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:52.178+0000 7f47c756f740 20 client.27287 trim_cache size 0 max 16384 2024-03-31T23:27:52.206 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:52.203+0000 7f47c756f740 1 Processor -- start 2024-03-31T23:27:52.206 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:52.203+0000 7f47c756f740 1 -- start start 2024-03-31T23:27:52.206 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:52.204+0000 7f47c756f740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56282ac52ed0 0x56282ac5bd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:27:52.206 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:52.204+0000 7f47c756f740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56282ac537f0 0x56282ac3fe20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:27:52.206 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:52.204+0000 7f47c756f740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56282ac5c5c0 0x56282ac67f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:27:52.206 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:52.204+0000 7f47c756f740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f47c01af270 con 0x56282ac52ed0 2024-03-31T23:27:52.206 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:52.204+0000 7f47c756f740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f47c01aca80 con 0x56282ac537f0 2024-03-31T23:27:52.206 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:52.204+0000 7f47c756f740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f47c01aa8f0 con 0x56282ac5c5c0 2024-03-31T23:27:52.206 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:52.204+0000 7f47be59c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56282ac52ed0 0x56282ac5bd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:27:52.207 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:52.204+0000 7f47be59c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56282ac52ed0 0x56282ac5bd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:34828/0 (socket says 172.21.15.46:34828) 2024-03-31T23:27:52.207 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:52.204+0000 7f47be59c640 1 -- 172.21.15.46:0/1401489078 learned_addr learned my addr 172.21.15.46:0/1401489078 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:27:52.207 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:52.204+0000 7f47bdd9b640 1 --2- 172.21.15.46:0/1401489078 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56282ac537f0 0x56282ac3fe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:27:52.207 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:52.204+0000 7f47bed9d640 1 --2- 172.21.15.46:0/1401489078 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56282ac5c5c0 0x56282ac67f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:27:52.207 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:52.204+0000 7f47be59c640 1 -- 172.21.15.46:0/1401489078 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56282ac5c5c0 msgr2=0x56282ac67f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:27:52.207 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:52.204+0000 7f47be59c640 1 --2- 172.21.15.46:0/1401489078 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56282ac5c5c0 0x56282ac67f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:27:52.207 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:52.204+0000 7f47be59c640 1 -- 172.21.15.46:0/1401489078 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56282ac537f0 msgr2=0x56282ac3fe20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:27:52.207 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:52.204+0000 7f47be59c640 1 --2- 172.21.15.46:0/1401489078 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56282ac537f0 0x56282ac3fe20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:27:52.207 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:52.204+0000 7f47be59c640 1 -- 172.21.15.46:0/1401489078 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56282ac44700 con 0x56282ac52ed0 2024-03-31T23:27:52.207 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:52.204+0000 7f47be59c640 1 --2- 172.21.15.46:0/1401489078 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56282ac52ed0 0x56282ac5bd00 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7f47b009aed0 tx=0x7f47b00996e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=29e69b2e9e125538 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:27:52.207 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:52.205+0000 7f47a7fff640 1 -- 172.21.15.46:0/1401489078 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47b00b5020 con 0x56282ac52ed0 2024-03-31T23:27:52.207 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:52.205+0000 7f47a7fff640 1 -- 172.21.15.46:0/1401489078 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f47b00a3c10 con 0x56282ac52ed0 2024-03-31T23:27:52.207 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:52.205+0000 7f47a7fff640 1 -- 172.21.15.46:0/1401489078 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47b0097950 con 0x56282ac52ed0 2024-03-31T23:27:52.210 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:52.205+0000 7f47c756f740 1 -- 172.21.15.46:0/1401489078 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56282ac52ed0 msgr2=0x56282ac5bd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:27:52.210 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:52.205+0000 7f47c756f740 1 --2- 172.21.15.46:0/1401489078 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56282ac52ed0 0x56282ac5bd00 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7f47b009aed0 tx=0x7f47b00996e0 comp rx=0 tx=0).stop 2024-03-31T23:27:52.210 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:52.206+0000 7f47c756f740 1 -- 172.21.15.46:0/1401489078 shutdown_connections 2024-03-31T23:27:52.210 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:52.206+0000 7f47c756f740 1 --2- 172.21.15.46:0/1401489078 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56282ac5c5c0 0x56282ac67f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:27:52.210 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:52.206+0000 7f47c756f740 1 --2- 172.21.15.46:0/1401489078 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56282ac537f0 0x56282ac3fe20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:27:52.210 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:52.206+0000 7f47c756f740 1 --2- 172.21.15.46:0/1401489078 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56282ac52ed0 0x56282ac5bd00 unknown :-1 s=CLOSED pgs=913 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:27:52.210 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:52.206+0000 7f47c756f740 1 -- 172.21.15.46:0/1401489078 >> 172.21.15.46:0/1401489078 conn(0x56282ac4d7f0 msgr2=0x56282ac54940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:27:52.210 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:52.206+0000 7f47c756f740 1 -- 172.21.15.46:0/1401489078 shutdown_connections 2024-03-31T23:27:52.210 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:52.206+0000 7f47c756f740 1 -- 172.21.15.46:0/1401489078 wait complete. 2024-03-31T23:27:52.211 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:52.206+0000 7f47c756f740 1 -- start start 2024-03-31T23:27:52.211 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:52.206+0000 7f47c756f740 1 Processor -- start 2024-03-31T23:27:52.211 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:52.207+0000 7f47c756f740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56282ac537f0 0x56282ac562b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:27:52.211 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:52.207+0000 7f47c756f740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56282ac5c5c0 0x56282ac567f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:27:52.211 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:52.207+0000 7f47c756f740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56282ac56d30 0x56282acd16d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:27:52.211 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:52.207+0000 7f47c756f740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f47c01aaf40 con 0x56282ac56d30 2024-03-31T23:27:52.211 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:52.207+0000 7f47c756f740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f47c01ac3b0 con 0x56282ac537f0 2024-03-31T23:27:52.211 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:52.207+0000 7f47c756f740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f47c01ab610 con 0x56282ac5c5c0 2024-03-31T23:27:52.211 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:52.207+0000 7f47c756f740 10 client.0 _ll_register_callbacks cb 0x7f47b00046c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T23:27:52.211 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:52.207+0000 7f47bdd9b640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56282ac5c5c0 0x56282ac567f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:27:52.211 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:52.207+0000 7f47be59c640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56282ac537f0 0x56282ac562b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:27:52.211 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:52.207+0000 7f47bed9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56282ac56d30 0x56282acd16d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:27:52.211 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:52.207+0000 7f47bdd9b640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56282ac5c5c0 0x56282ac567f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.46:54820/0 (socket says 172.21.15.46:54820) 2024-03-31T23:27:52.211 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:52.207+0000 7f47bdd9b640 1 -- 172.21.15.46:0/2293929336 learned_addr learned my addr 172.21.15.46:0/2293929336 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:27:52.212 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:52.207+0000 7f47bed9d640 1 -- 172.21.15.46:0/2293929336 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56282ac5c5c0 msgr2=0x56282ac567f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:27:52.212 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:52.207+0000 7f47bed9d640 1 --2- 172.21.15.46:0/2293929336 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56282ac5c5c0 0x56282ac567f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:27:52.212 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:52.207+0000 7f47bed9d640 1 -- 172.21.15.46:0/2293929336 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56282ac537f0 msgr2=0x56282ac562b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:27:52.212 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:52.207+0000 7f47bed9d640 1 --2- 172.21.15.46:0/2293929336 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56282ac537f0 0x56282ac562b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:27:52.212 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:52.207+0000 7f47bed9d640 1 -- 172.21.15.46:0/2293929336 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56282acd1c90 con 0x56282ac56d30 2024-03-31T23:27:52.212 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:52.207+0000 7f47be59c640 1 --2- 172.21.15.46:0/2293929336 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56282ac537f0 0x56282ac562b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:27:52.212 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:52.207+0000 7f47bed9d640 1 --2- 172.21.15.46:0/2293929336 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56282ac56d30 0x56282acd16d0 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7f47b8005fd0 tx=0x7f47b8004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=51c16688196095c5 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:27:52.212 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:52.208+0000 7f47a5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-31T23:27:52.212 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:52.208+0000 7f47a5ffb640 1 -- 172.21.15.46:0/2293929336 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47b802f060 con 0x56282ac56d30 2024-03-31T23:27:52.212 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:52.208+0000 7f47c756f740 1 -- 172.21.15.46:0/2293929336 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x56282abbae30 con 0x56282ac56d30 2024-03-31T23:27:52.212 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:52.208+0000 7f47a5ffb640 1 -- 172.21.15.46:0/2293929336 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(117..117 src has 1..117) ==== 9554+0+0 (secure 0 0 0) 0x7f47b8020420 con 0x56282ac56d30 2024-03-31T23:27:52.212 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:52.208+0000 7f47a5ffb640 1 -- 172.21.15.46:0/2293929336 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) ==== 24+0+0 (secure 0 0 0) 0x7f47b8020620 con 0x56282ac56d30 2024-03-31T23:27:52.212 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:52.208+0000 7f47c756f740 10 client.17463 fetch_fsmap learned FSMap version 42 2024-03-31T23:27:52.212 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:52.208+0000 7f47c756f740 1 -- 172.21.15.46:0/2293929336 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=4+}) -- 0x56282acd2ff0 con 0x56282ac56d30 2024-03-31T23:27:52.212 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:52.209+0000 7f47a5ffb640 1 -- 172.21.15.46:0/2293929336 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x56282abbae30 con 0x56282ac56d30 2024-03-31T23:27:52.213 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:52.209+0000 7f47a5ffb640 1 -- 172.21.15.46:0/2293929336 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 42) ==== 8362+0+0 (secure 0 0 0) 0x7f47b8029030 con 0x56282ac56d30 2024-03-31T23:27:52.213 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:52.209+0000 7f47c756f740 10 client.17463 fetch_fsmap finished waiting for FSMap version 42 2024-03-31T23:27:52.213 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:52.209+0000 7f47c756f740 10 client.17463 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-31T23:27:52.213 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:52.209+0000 7f47c756f740 20 client.17463 populate_metadata read hostname 'smithi046' 2024-03-31T23:27:52.213 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:52.209+0000 7f47c756f740 1 --2- 172.21.15.46:0/2293929336 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x56282acd3780 0x56282acd5ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:27:52.213 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:52.209+0000 7f47c756f740 4 client.17463 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-31T23:27:52.213 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:52.209+0000 7f47c756f740 1 -- 172.21.15.46:0/2293929336 --> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] -- command(tid 0: {"prefix": "flush journal"}) -- 0x56282abb5180 con 0x56282acd3780 2024-03-31T23:27:52.213 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:52.209+0000 7f47be59c640 1 --2- 172.21.15.46:0/2293929336 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x56282acd3780 0x56282acd5ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:27:52.214 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:52.210+0000 7f47a5ffb640 10 client.17463 ms_handle_connect on v2:172.21.15.142:6838/2519143918 2024-03-31T23:27:52.214 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:52.210+0000 7f47be59c640 1 --2- 172.21.15.46:0/2293929336 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x56282acd3780 0x56282acd5ba0 crc :-1 s=READY pgs=221 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3a99607a6cb55251 server_cookie=1fed11348a557ab9 in_seq=0 out_seq=0 2024-03-31T23:27:52.280 INFO:tasks.workunit.client.0.smithi046.stdout: 10 28903 8.65 MB/sec execute 57 sec latency 83.915 ms 2024-03-31T23:27:52.365 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:52.363+0000 7f47a5ffb640 1 -- 172.21.15.46:0/2293929336 <== mds.0 v2:172.21.15.142:6838/2519143918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x56282abb5180 con 0x56282acd3780 2024-03-31T23:27:52.365 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:52.363+0000 7f47a5ffb640 10 client.17463 handle_command_reply: tid=0 2024-03-31T23:27:52.365 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:52.363+0000 7f47c756f740 1 client.17463 shutdown 2024-03-31T23:27:52.365 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:52.363+0000 7f47c756f740 1 -- 172.21.15.46:0/2293929336 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56282ac56d30 msgr2=0x56282acd16d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:27:52.365 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:52.363+0000 7f47c756f740 1 --2- 172.21.15.46:0/2293929336 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56282ac56d30 0x56282acd16d0 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7f47b8005fd0 tx=0x7f47b8004ab0 comp rx=0 tx=0).stop 2024-03-31T23:27:52.365 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:52.363+0000 7f47c756f740 1 -- 172.21.15.46:0/2293929336 shutdown_connections 2024-03-31T23:27:52.365 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:52.363+0000 7f47c756f740 1 --2- 172.21.15.46:0/2293929336 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x56282acd3780 0x56282acd5ba0 crc :-1 s=READY pgs=221 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:27:52.365 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:52.363+0000 7f47c756f740 1 --2- 172.21.15.46:0/2293929336 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56282ac56d30 0x56282acd16d0 unknown :-1 s=CLOSED pgs=914 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:27:52.365 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:52.363+0000 7f47a5ffb640 0 client.17463 ms_handle_reset on v2:172.21.15.142:6838/2519143918 2024-03-31T23:27:52.365 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:52.363+0000 7f47c756f740 1 --2- 172.21.15.46:0/2293929336 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56282ac5c5c0 0x56282ac567f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:27:52.365 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:52.363+0000 7f47c756f740 1 --2- 172.21.15.46:0/2293929336 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56282ac537f0 0x56282ac562b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:27:52.365 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:52.363+0000 7f47c756f740 1 -- 172.21.15.46:0/2293929336 >> 172.21.15.46:0/2293929336 conn(0x56282ac4d7f0 msgr2=0x56282ac59590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:27:52.366 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:52.364+0000 7f47c756f740 1 -- 172.21.15.46:0/2293929336 shutdown_connections 2024-03-31T23:27:52.366 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:52.364+0000 7f47c756f740 1 -- 172.21.15.46:0/2293929336 wait complete. 2024-03-31T23:27:52.366 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:52.364+0000 7f47c756f740 20 client.17463 trim_cache size 0 max 16384 2024-03-31T23:27:52.366 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout:{ 2024-03-31T23:27:52.366 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout: "message": "", 2024-03-31T23:27:52.366 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout: "return_code": 0 2024-03-31T23:27:52.366 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout:} 2024-03-31T23:27:52.367 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:52.365+0000 7f47c498a640 1 -- 172.21.15.46:0/3862485176 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f47a80a2730 msgr2=0x7f47a80a4bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:27:52.368 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:52.365+0000 7f47c498a640 1 --2- 172.21.15.46:0/3862485176 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f47a80a2730 0x7f47a80a4bf0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f47b8005c40 tx=0x7f47b8004a40 comp rx=0 tx=0).stop 2024-03-31T23:27:52.368 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:52.365+0000 7f47c498a640 1 -- 172.21.15.46:0/3862485176 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f47c01008a0 msgr2=0x7f47c019a1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:27:52.368 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:52.365+0000 7f47c498a640 1 --2- 172.21.15.46:0/3862485176 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f47c01008a0 0x7f47c019a1a0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f47b000e9f0 tx=0x7f47b000eeb0 comp rx=0 tx=0).stop 2024-03-31T23:27:52.368 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:52.365+0000 7f47c498a640 1 -- 172.21.15.46:0/3862485176 shutdown_connections 2024-03-31T23:27:52.368 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:52.365+0000 7f47c498a640 1 --2- 172.21.15.46:0/3862485176 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f47a80a2730 0x7f47a80a4bf0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:27:52.368 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:52.365+0000 7f47c498a640 1 --2- 172.21.15.46:0/3862485176 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f47c01055f0 0x7f47c019ea60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:27:52.369 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:52.365+0000 7f47c498a640 1 --2- 172.21.15.46:0/3862485176 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f47c0101260 0x7f47c019a6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:27:52.370 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:52.365+0000 7f47c498a640 1 --2- 172.21.15.46:0/3862485176 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f47c01008a0 0x7f47c019a1a0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:27:52.370 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:52.365+0000 7f47c498a640 1 -- 172.21.15.46:0/3862485176 >> 172.21.15.46:0/3862485176 conn(0x7f47c00774a0 msgr2=0x7f47c010bb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:27:52.370 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:52.366+0000 7f47c498a640 1 -- 172.21.15.46:0/3862485176 shutdown_connections 2024-03-31T23:27:52.371 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:27:52.366+0000 7f47c498a640 1 -- 172.21.15.46:0/3862485176 wait complete. 2024-03-31T23:27:52.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:27:52 smithi046 ceph-mon[32498]: pgmap v508: 129 pgs: 129 active+clean; 589 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 4.3 MiB/s wr, 301 op/s 2024-03-31T23:27:52.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:27:52 smithi142 ceph-mon[35271]: pgmap v508: 129 pgs: 129 active+clean; 589 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 4.3 MiB/s wr, 301 op/s 2024-03-31T23:27:53.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:27:52 smithi162 ceph-mon[36237]: pgmap v508: 129 pgs: 129 active+clean; 589 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 4.3 MiB/s wr, 301 op/s 2024-03-31T23:27:53.280 INFO:tasks.workunit.client.0.smithi046.stdout: 10 29091 8.69 MB/sec execute 58 sec latency 154.788 ms 2024-03-31T23:27:54.280 INFO:tasks.workunit.client.0.smithi046.stdout: 10 29291 8.78 MB/sec execute 59 sec latency 64.072 ms 2024-03-31T23:27:54.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:27:54 smithi046 ceph-mon[32498]: pgmap v509: 129 pgs: 129 active+clean; 589 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 4.3 MiB/s wr, 301 op/s 2024-03-31T23:27:54.944 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:27:54 smithi142 ceph-mon[35271]: pgmap v509: 129 pgs: 129 active+clean; 589 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 4.3 MiB/s wr, 301 op/s 2024-03-31T23:27:55.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:27:54 smithi162 ceph-mon[36237]: pgmap v509: 129 pgs: 129 active+clean; 589 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 4.3 MiB/s wr, 301 op/s 2024-03-31T23:27:55.280 INFO:tasks.workunit.client.0.smithi046.stdout: 10 29438 8.73 MB/sec execute 60 sec latency 76.294 ms 2024-03-31T23:27:56.280 INFO:tasks.workunit.client.0.smithi046.stdout: 10 29591 8.75 MB/sec execute 61 sec latency 76.846 ms 2024-03-31T23:27:56.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:27:56 smithi046 ceph-mon[32498]: pgmap v510: 129 pgs: 129 active+clean; 593 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 6.4 MiB/s wr, 460 op/s 2024-03-31T23:27:56.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:27:56 smithi142 ceph-mon[35271]: pgmap v510: 129 pgs: 129 active+clean; 593 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 6.4 MiB/s wr, 460 op/s 2024-03-31T23:27:57.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:27:56 smithi162 ceph-mon[36237]: pgmap v510: 129 pgs: 129 active+clean; 593 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 6.4 MiB/s wr, 460 op/s 2024-03-31T23:27:57.280 INFO:tasks.workunit.client.0.smithi046.stdout: 10 29764 8.77 MB/sec execute 62 sec latency 85.086 ms 2024-03-31T23:27:58.280 INFO:tasks.workunit.client.0.smithi046.stdout: 10 29959 8.80 MB/sec execute 63 sec latency 72.393 ms 2024-03-31T23:27:58.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:27:58 smithi046 ceph-mon[32498]: pgmap v511: 129 pgs: 129 active+clean; 593 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 4.1 MiB/s wr, 280 op/s 2024-03-31T23:27:58.862 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:27:58 smithi046 ceph-mon[32498]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-03-31T23:27:58.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:27:58 smithi142 ceph-mon[35271]: pgmap v511: 129 pgs: 129 active+clean; 593 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 4.1 MiB/s wr, 280 op/s 2024-03-31T23:27:58.944 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:27:58 smithi142 ceph-mon[35271]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-03-31T23:27:59.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:27:58 smithi162 ceph-mon[36237]: pgmap v511: 129 pgs: 129 active+clean; 593 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 4.1 MiB/s wr, 280 op/s 2024-03-31T23:27:59.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:27:58 smithi162 ceph-mon[36237]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-03-31T23:27:59.280 INFO:tasks.workunit.client.0.smithi046.stdout: 10 30095 8.78 MB/sec execute 64 sec latency 75.606 ms 2024-03-31T23:28:00.281 INFO:tasks.workunit.client.0.smithi046.stdout: 10 30220 8.73 MB/sec execute 65 sec latency 101.891 ms 2024-03-31T23:28:00.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:28:00 smithi046 ceph-mon[32498]: pgmap v512: 129 pgs: 129 active+clean; 605 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 6.2 MiB/s wr, 392 op/s 2024-03-31T23:28:00.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:28:00 smithi142 ceph-mon[35271]: pgmap v512: 129 pgs: 129 active+clean; 605 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 6.2 MiB/s wr, 392 op/s 2024-03-31T23:28:01.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:28:00 smithi162 ceph-mon[36237]: pgmap v512: 129 pgs: 129 active+clean; 605 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 6.2 MiB/s wr, 392 op/s 2024-03-31T23:28:01.280 INFO:tasks.workunit.client.0.smithi046.stdout: 10 30418 8.84 MB/sec execute 66 sec latency 77.735 ms 2024-03-31T23:28:02.280 INFO:tasks.workunit.client.0.smithi046.stdout: 10 30594 8.89 MB/sec execute 67 sec latency 61.579 ms 2024-03-31T23:28:02.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:28:02 smithi142 ceph-mon[35271]: pgmap v513: 129 pgs: 129 active+clean; 605 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 4.1 MiB/s wr, 272 op/s 2024-03-31T23:28:03.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:28:02 smithi162 ceph-mon[36237]: pgmap v513: 129 pgs: 129 active+clean; 605 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 4.1 MiB/s wr, 272 op/s 2024-03-31T23:28:03.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:28:02 smithi046 ceph-mon[32498]: pgmap v513: 129 pgs: 129 active+clean; 605 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 4.1 MiB/s wr, 272 op/s 2024-03-31T23:28:03.280 INFO:tasks.workunit.client.0.smithi046.stdout: 10 30742 8.84 MB/sec execute 68 sec latency 101.031 ms 2024-03-31T23:28:03.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:28:03 smithi142 ceph-mon[35271]: osdmap e118: 12 total, 12 up, 12 in 2024-03-31T23:28:03.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:28:03 smithi142 ceph-mon[35271]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-03-31T23:28:04.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:28:03 smithi162 ceph-mon[36237]: osdmap e118: 12 total, 12 up, 12 in 2024-03-31T23:28:04.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:28:03 smithi162 ceph-mon[36237]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-03-31T23:28:04.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:28:03 smithi046 ceph-mon[32498]: osdmap e118: 12 total, 12 up, 12 in 2024-03-31T23:28:04.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:28:03 smithi046 ceph-mon[32498]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-03-31T23:28:04.280 INFO:tasks.workunit.client.0.smithi046.stdout: 10 30876 8.79 MB/sec execute 69 sec latency 77.751 ms 2024-03-31T23:28:04.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:28:04 smithi142 ceph-mon[35271]: pgmap v515: 129 pgs: 129 active+clean; 605 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 5.0 MiB/s wr, 326 op/s 2024-03-31T23:28:05.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:28:04 smithi162 ceph-mon[36237]: pgmap v515: 129 pgs: 129 active+clean; 605 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 5.0 MiB/s wr, 326 op/s 2024-03-31T23:28:05.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:28:04 smithi046 ceph-mon[32498]: pgmap v515: 129 pgs: 129 active+clean; 605 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 5.0 MiB/s wr, 326 op/s 2024-03-31T23:28:05.282 INFO:tasks.workunit.client.0.smithi046.stdout: 10 31037 8.77 MB/sec execute 70 sec latency 71.238 ms 2024-03-31T23:28:05.505 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:05.503+0000 7ff77744e640 1 -- 172.21.15.46:0/2583625931 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff770106680 msgr2=0x7ff770106a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:28:05.505 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:05.503+0000 7ff77744e640 1 --2- 172.21.15.46:0/2583625931 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff770106680 0x7ff770106a60 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7ff764009930 tx=0x7ff76402bc30 comp rx=0 tx=0).stop 2024-03-31T23:28:05.505 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:05.503+0000 7ff77744e640 1 -- 172.21.15.46:0/2583625931 shutdown_connections 2024-03-31T23:28:05.505 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:05.503+0000 7ff77744e640 1 --2- 172.21.15.46:0/2583625931 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff770100fa0 0x7ff77010d4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:28:05.505 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:05.503+0000 7ff77744e640 1 --2- 172.21.15.46:0/2583625931 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff7701005e0 0x7ff770100a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:28:05.505 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:05.503+0000 7ff77744e640 1 --2- 172.21.15.46:0/2583625931 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff770106680 0x7ff770106a60 unknown :-1 s=CLOSED pgs=915 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:28:05.505 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:05.503+0000 7ff77744e640 1 -- 172.21.15.46:0/2583625931 >> 172.21.15.46:0/2583625931 conn(0x7ff770077710 msgr2=0x7ff770077b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:28:05.505 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:05.504+0000 7ff77744e640 1 -- 172.21.15.46:0/2583625931 shutdown_connections 2024-03-31T23:28:05.505 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:05.504+0000 7ff77744e640 1 -- 172.21.15.46:0/2583625931 wait complete. 2024-03-31T23:28:05.507 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:05.504+0000 7ff77744e640 1 Processor -- start 2024-03-31T23:28:05.507 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:05.504+0000 7ff77744e640 1 -- start start 2024-03-31T23:28:05.508 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:05.505+0000 7ff77744e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff7701005e0 0x7ff77019d860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:28:05.508 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:05.505+0000 7ff77744e640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff770100fa0 0x7ff77019dda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:28:05.508 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:05.505+0000 7ff77744e640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff770106680 0x7ff770197a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:28:05.508 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:05.505+0000 7ff77744e640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7ff7701119b0 con 0x7ff7701005e0 2024-03-31T23:28:05.508 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:05.505+0000 7ff77744e640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7ff770111830 con 0x7ff770106680 2024-03-31T23:28:05.508 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:05.505+0000 7ff77744e640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7ff770111b30 con 0x7ff770100fa0 2024-03-31T23:28:05.508 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:05.505+0000 7ff7751ea640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff7701005e0 0x7ff77019d860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:28:05.508 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:05.505+0000 7ff7751ea640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff7701005e0 0x7ff77019d860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:52064/0 (socket says 172.21.15.46:52064) 2024-03-31T23:28:05.508 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:05.505+0000 7ff7751ea640 1 -- 172.21.15.46:0/78119239 learned_addr learned my addr 172.21.15.46:0/78119239 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:28:05.508 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:05.505+0000 7ff7759eb640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff770106680 0x7ff770197a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:28:05.508 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:05.505+0000 7ff7749e9640 1 --2- 172.21.15.46:0/78119239 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff770100fa0 0x7ff77019dda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:28:05.508 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:05.505+0000 7ff7751ea640 1 -- 172.21.15.46:0/78119239 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff770100fa0 msgr2=0x7ff77019dda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:28:05.509 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:05.505+0000 7ff7751ea640 1 --2- 172.21.15.46:0/78119239 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff770100fa0 0x7ff77019dda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:28:05.509 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:05.505+0000 7ff7751ea640 1 -- 172.21.15.46:0/78119239 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff770106680 msgr2=0x7ff770197a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:28:05.509 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:05.505+0000 7ff7751ea640 1 --2- 172.21.15.46:0/78119239 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff770106680 0x7ff770197a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:28:05.509 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:05.505+0000 7ff7751ea640 1 -- 172.21.15.46:0/78119239 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff770198190 con 0x7ff7701005e0 2024-03-31T23:28:05.509 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:05.505+0000 7ff7759eb640 1 --2- 172.21.15.46:0/78119239 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff770106680 0x7ff770197a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:28:05.509 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:05.505+0000 7ff7751ea640 1 --2- 172.21.15.46:0/78119239 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff7701005e0 0x7ff77019d860 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7ff764009900 tx=0x7ff76402ca90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:28:05.509 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:05.506+0000 7ff7627fc640 1 -- 172.21.15.46:0/78119239 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff764042020 con 0x7ff7701005e0 2024-03-31T23:28:05.509 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:05.506+0000 7ff77744e640 1 -- 172.21.15.46:0/78119239 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff770198420 con 0x7ff7701005e0 2024-03-31T23:28:05.509 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:05.506+0000 7ff77744e640 1 -- 172.21.15.46:0/78119239 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff77010de70 con 0x7ff7701005e0 2024-03-31T23:28:05.509 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:05.506+0000 7ff7627fc640 1 -- 172.21.15.46:0/78119239 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff76402cc80 con 0x7ff7701005e0 2024-03-31T23:28:05.509 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:05.506+0000 7ff7627fc640 1 -- 172.21.15.46:0/78119239 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff764003b00 con 0x7ff7701005e0 2024-03-31T23:28:05.511 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:05.509+0000 7ff7627fc640 1 -- 172.21.15.46:0/78119239 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 25) ==== 137451+0+0 (secure 0 0 0) 0x7ff764045020 con 0x7ff7701005e0 2024-03-31T23:28:05.512 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:05.509+0000 7ff77a033740 1 Processor -- start 2024-03-31T23:28:05.512 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:05.509+0000 7ff77a033740 1 -- start start 2024-03-31T23:28:05.512 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:05.509+0000 7ff77a033740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5569000eb6c0 0x5569000ebaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:28:05.512 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:05.509+0000 7ff77a033740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5569000ebfe0 0x5569000f4870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:28:05.512 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:05.509+0000 7ff77a033740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5569000f4db0 0x5569000f71a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:28:05.512 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:05.509+0000 7ff77a033740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55690004e8f0 con 0x5569000eb6c0 2024-03-31T23:28:05.512 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:05.509+0000 7ff77a033740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x5568ffff78d0 con 0x5569000ebfe0 2024-03-31T23:28:05.512 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:05.509+0000 7ff77a033740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55690004a990 con 0x5569000f4db0 2024-03-31T23:28:05.512 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:05.510+0000 7ff7759eb640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5569000eb6c0 0x5569000ebaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:28:05.513 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:05.510+0000 7ff7627fc640 1 --2- 172.21.15.46:0/78119239 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7ff75c0a2730 0x7ff75c0a4bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:28:05.513 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:05.510+0000 7ff7759eb640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5569000eb6c0 0x5569000ebaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:52076/0 (socket says 172.21.15.46:52076) 2024-03-31T23:28:05.513 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:05.510+0000 7ff7759eb640 1 -- 172.21.15.46:0/672291075 learned_addr learned my addr 172.21.15.46:0/672291075 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:28:05.513 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:05.510+0000 7ff7751ea640 1 --2- 172.21.15.46:0/672291075 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5569000ebfe0 0x5569000f4870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:28:05.513 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:05.510+0000 7ff7749e9640 1 --2- 172.21.15.46:0/672291075 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5569000f4db0 0x5569000f71a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:28:05.513 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:05.510+0000 7ff7759eb640 1 --2- 172.21.15.46:0/78119239 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7ff75c0a2730 0x7ff75c0a4bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:28:05.513 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:05.510+0000 7ff7759eb640 1 -- 172.21.15.46:0/672291075 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5569000f4db0 msgr2=0x5569000f71a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:28:05.513 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:05.510+0000 7ff7759eb640 1 --2- 172.21.15.46:0/672291075 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5569000f4db0 0x5569000f71a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:28:05.513 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:05.510+0000 7ff7759eb640 1 -- 172.21.15.46:0/672291075 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5569000ebfe0 msgr2=0x5569000f4870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:28:05.513 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:05.510+0000 7ff7759eb640 1 --2- 172.21.15.46:0/672291075 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5569000ebfe0 0x5569000f4870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:28:05.514 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:05.510+0000 7ff7759eb640 1 -- 172.21.15.46:0/672291075 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5568ffd63210 con 0x5569000eb6c0 2024-03-31T23:28:05.514 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:05.510+0000 7ff7627fc640 1 -- 172.21.15.46:0/78119239 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(118..118 src has 1..118) ==== 9642+0+0 (secure 0 0 0) 0x7ff7640f0e80 con 0x7ff7701005e0 2024-03-31T23:28:05.514 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:05.510+0000 7ff7759eb640 1 --2- 172.21.15.46:0/672291075 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5569000eb6c0 0x5569000ebaa0 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7ff768009fd0 tx=0x7ff768013830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=10af034ce147ae06 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:28:05.514 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:05.511+0000 7ff75b7fe640 1 -- 172.21.15.46:0/672291075 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff768022020 con 0x5569000eb6c0 2024-03-31T23:28:05.514 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:05.511+0000 7ff75b7fe640 1 -- 172.21.15.46:0/672291075 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff768010430 con 0x5569000eb6c0 2024-03-31T23:28:05.514 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:05.511+0000 7ff7759eb640 1 --2- 172.21.15.46:0/78119239 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7ff75c0a2730 0x7ff75c0a4bf0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7ff768023000 tx=0x7ff768021000 comp rx=0 tx=0).ready entity=mgr.15654 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:28:05.514 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:05.511+0000 7ff75b7fe640 1 -- 172.21.15.46:0/672291075 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff76801e660 con 0x5569000eb6c0 2024-03-31T23:28:05.520 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:05.512+0000 7ff77a033740 1 -- 172.21.15.46:0/672291075 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5569000eb6c0 msgr2=0x5569000ebaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:28:05.520 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:05.512+0000 7ff77a033740 1 --2- 172.21.15.46:0/672291075 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5569000eb6c0 0x5569000ebaa0 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7ff768009fd0 tx=0x7ff768013830 comp rx=0 tx=0).stop 2024-03-31T23:28:05.520 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:05.512+0000 7ff77a033740 1 -- 172.21.15.46:0/672291075 shutdown_connections 2024-03-31T23:28:05.521 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:05.512+0000 7ff77a033740 1 --2- 172.21.15.46:0/672291075 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5569000f4db0 0x5569000f71a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:28:05.521 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:05.512+0000 7ff77a033740 1 --2- 172.21.15.46:0/672291075 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5569000ebfe0 0x5569000f4870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:28:05.521 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:05.512+0000 7ff77a033740 1 --2- 172.21.15.46:0/672291075 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5569000eb6c0 0x5569000ebaa0 unknown :-1 s=CLOSED pgs=917 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:28:05.521 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:05.512+0000 7ff77a033740 1 -- 172.21.15.46:0/672291075 >> 172.21.15.46:0/672291075 conn(0x5569000e5fe0 msgr2=0x5569000e63c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:28:05.521 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:05.512+0000 7ff77a033740 1 -- 172.21.15.46:0/672291075 shutdown_connections 2024-03-31T23:28:05.521 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:05.512+0000 7ff77a033740 1 -- 172.21.15.46:0/672291075 wait complete. 2024-03-31T23:28:05.521 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:05.512+0000 7ff77a033740 1 -- start start 2024-03-31T23:28:05.521 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:05.512+0000 7ff77a033740 1 Processor -- start 2024-03-31T23:28:05.521 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:05.513+0000 7ff77a033740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5569000eb6c0 0x5569000f1df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:28:05.521 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:05.513+0000 7ff77a033740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5569000ebfe0 0x5569000f2330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:28:05.522 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:05.513+0000 7ff77a033740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5569000f4db0 0x5569000f2870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:28:05.522 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:05.513+0000 7ff77a033740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x5569000f79e0 con 0x5569000eb6c0 2024-03-31T23:28:05.522 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:05.513+0000 7ff77a033740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x5568fffecf80 con 0x5569000f4db0 2024-03-31T23:28:05.522 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:05.513+0000 7ff77a033740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5569000f7b60 con 0x5569000ebfe0 2024-03-31T23:28:05.522 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:05.513+0000 7ff7751ea640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5569000eb6c0 0x5569000f1df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:28:05.522 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:05.513+0000 7ff7749e9640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5569000ebfe0 0x5569000f2330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:28:05.522 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:05.513+0000 7ff7751ea640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5569000eb6c0 0x5569000f1df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:52090/0 (socket says 172.21.15.46:52090) 2024-03-31T23:28:05.522 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:05.513+0000 7ff7749e9640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5569000ebfe0 0x5569000f2330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.46:53942/0 (socket says 172.21.15.46:53942) 2024-03-31T23:28:05.523 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:05.513+0000 7ff7749e9640 1 -- 172.21.15.46:0/1242566092 learned_addr learned my addr 172.21.15.46:0/1242566092 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:28:05.523 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:05.513+0000 7ff7759eb640 1 --2- 172.21.15.46:0/1242566092 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5569000f4db0 0x5569000f2870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:28:05.523 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:05.513+0000 7ff77a033740 10 client.0 _ll_register_callbacks cb 0x556900045290 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T23:28:05.523 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:05.514+0000 7ff7751ea640 1 -- 172.21.15.46:0/1242566092 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5569000ebfe0 msgr2=0x5569000f2330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:28:05.523 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:05.514+0000 7ff7751ea640 1 --2- 172.21.15.46:0/1242566092 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5569000ebfe0 0x5569000f2330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:28:05.523 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:05.514+0000 7ff7751ea640 1 -- 172.21.15.46:0/1242566092 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5569000f4db0 msgr2=0x5569000f2870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:28:05.523 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:05.514+0000 7ff7751ea640 1 --2- 172.21.15.46:0/1242566092 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5569000f4db0 0x5569000f2870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:28:05.523 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:05.514+0000 7ff7751ea640 1 -- 172.21.15.46:0/1242566092 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5568ffe2f620 con 0x5569000eb6c0 2024-03-31T23:28:05.523 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:05.514+0000 7ff7749e9640 1 --2- 172.21.15.46:0/1242566092 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5569000ebfe0 0x5569000f2330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:28:05.523 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:05.514+0000 7ff7751ea640 1 --2- 172.21.15.46:0/1242566092 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5569000eb6c0 0x5569000f1df0 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7ff7640ee250 tx=0x7ff764009680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=37a8313f27d2773f server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:28:05.524 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:05.514+0000 7ff758ff9640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-31T23:28:05.524 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:05.514+0000 7ff758ff9640 1 -- 172.21.15.46:0/1242566092 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7640bd2b0 con 0x5569000eb6c0 2024-03-31T23:28:05.524 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:05.514+0000 7ff758ff9640 1 -- 172.21.15.46:0/1242566092 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(118..118 src has 1..118) ==== 9642+0+0 (secure 0 0 0) 0x7ff7640db980 con 0x5569000eb6c0 2024-03-31T23:28:05.524 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:05.514+0000 7ff77a033740 1 -- 172.21.15.46:0/1242566092 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff768010430 con 0x5569000eb6c0 2024-03-31T23:28:05.524 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:05.514+0000 7ff758ff9640 1 -- 172.21.15.46:0/1242566092 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=44) ==== 24+0+0 (secure 0 0 0) 0x7ff7640c1400 con 0x5569000eb6c0 2024-03-31T23:28:05.524 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:05.514+0000 7ff77a033740 10 client.17475 fetch_fsmap learned FSMap version 44 2024-03-31T23:28:05.524 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:05.514+0000 7ff77a033740 1 -- 172.21.15.46:0/1242566092 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=44,monmap=4+}) -- 0x556900103920 con 0x5569000eb6c0 2024-03-31T23:28:05.524 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:05.515+0000 7ff758ff9640 1 -- 172.21.15.46:0/1242566092 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff768010430 con 0x5569000eb6c0 2024-03-31T23:28:05.524 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:05.515+0000 7ff758ff9640 1 -- 172.21.15.46:0/1242566092 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 44) ==== 8354+0+0 (secure 0 0 0) 0x7ff7640c5830 con 0x5569000eb6c0 2024-03-31T23:28:05.524 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:05.515+0000 7ff77a033740 10 client.17475 fetch_fsmap finished waiting for FSMap version 44 2024-03-31T23:28:05.525 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:05.515+0000 7ff77a033740 10 client.17475 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-31T23:28:05.525 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:05.515+0000 7ff77a033740 20 client.17475 populate_metadata read hostname 'smithi046' 2024-03-31T23:28:05.525 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:05.515+0000 7ff77a033740 1 --2- 172.21.15.46:0/1242566092 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x556900104e00 0x5569001072a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:28:05.525 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:05.515+0000 7ff77a033740 4 client.17475 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-31T23:28:05.525 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:05.515+0000 7ff77a033740 1 -- 172.21.15.46:0/1242566092 --> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556900062120 con 0x556900104e00 2024-03-31T23:28:05.525 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:05.515+0000 7ff7749e9640 1 --2- 172.21.15.46:0/1242566092 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x556900104e00 0x5569001072a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:28:05.525 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:05.516+0000 7ff758ff9640 10 client.17475 ms_handle_connect on v2:172.21.15.142:6838/2519143918 2024-03-31T23:28:05.525 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:05.516+0000 7ff7749e9640 1 --2- 172.21.15.46:0/1242566092 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x556900104e00 0x5569001072a0 crc :-1 s=READY pgs=222 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8a9af03f59bc4a93 server_cookie=ce1a54cb0d03c7a in_seq=0 out_seq=0 2024-03-31T23:28:05.525 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:05.517+0000 7ff758ff9640 1 -- 172.21.15.46:0/1242566092 <== mds.0 v2:172.21.15.142:6838/2519143918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x556900062120 con 0x556900104e00 2024-03-31T23:28:05.525 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:05.517+0000 7ff758ff9640 10 client.17475 handle_command_reply: tid=0 2024-03-31T23:28:05.526 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:05.517+0000 7ff77a033740 1 client.17475 shutdown 2024-03-31T23:28:05.526 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:05.517+0000 7ff77a033740 1 -- 172.21.15.46:0/1242566092 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5569000eb6c0 msgr2=0x5569000f1df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:28:05.526 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:05.517+0000 7ff77a033740 1 --2- 172.21.15.46:0/1242566092 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5569000eb6c0 0x5569000f1df0 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7ff7640ee250 tx=0x7ff764009680 comp rx=0 tx=0).stop 2024-03-31T23:28:05.526 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:05.518+0000 7ff77a033740 1 -- 172.21.15.46:0/1242566092 shutdown_connections 2024-03-31T23:28:05.526 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:05.518+0000 7ff77a033740 1 --2- 172.21.15.46:0/1242566092 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x556900104e00 0x5569001072a0 crc :-1 s=READY pgs=222 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:28:05.526 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:05.518+0000 7ff77a033740 1 --2- 172.21.15.46:0/1242566092 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5569000f4db0 0x5569000f2870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:28:05.526 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:05.518+0000 7ff77a033740 1 --2- 172.21.15.46:0/1242566092 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5569000ebfe0 0x5569000f2330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:28:05.526 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:05.518+0000 7ff77a033740 1 --2- 172.21.15.46:0/1242566092 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5569000eb6c0 0x5569000f1df0 unknown :-1 s=CLOSED pgs=918 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:28:05.526 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:05.518+0000 7ff77a033740 1 -- 172.21.15.46:0/1242566092 >> 172.21.15.46:0/1242566092 conn(0x5569000e5fe0 msgr2=0x5569000e7d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:28:05.526 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:05.518+0000 7ff758ff9640 0 client.17475 ms_handle_reset on v2:172.21.15.142:6838/2519143918 2024-03-31T23:28:05.526 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:05.518+0000 7ff77a033740 1 -- 172.21.15.46:0/1242566092 shutdown_connections 2024-03-31T23:28:05.526 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:05.518+0000 7ff77a033740 1 -- 172.21.15.46:0/1242566092 wait complete. 2024-03-31T23:28:05.526 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:05.518+0000 7ff77a033740 20 client.17475 trim_cache size 0 max 16384 2024-03-31T23:28:05.547 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:05.544+0000 7ff77a033740 1 Processor -- start 2024-03-31T23:28:05.547 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:05.544+0000 7ff77a033740 1 -- start start 2024-03-31T23:28:05.547 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:05.544+0000 7ff77a033740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5569000eb6c0 0x5569000f4520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:28:05.547 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:05.544+0000 7ff77a033740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5569000ebfe0 0x556900101700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:28:05.547 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:05.544+0000 7ff77a033740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5569000f4db0 0x5569001007b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:28:05.547 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:05.544+0000 7ff77a033740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7ff754012a60 con 0x5569000eb6c0 2024-03-31T23:28:05.547 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:05.544+0000 7ff77a033740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7ff7540102d0 con 0x5569000f4db0 2024-03-31T23:28:05.548 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:05.544+0000 7ff77a033740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7ff75400e140 con 0x5569000ebfe0 2024-03-31T23:28:05.548 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:05.544+0000 7ff7751ea640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5569000eb6c0 0x5569000f4520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:28:05.548 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:05.545+0000 7ff7759eb640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5569000f4db0 0x5569001007b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:28:05.548 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:05.545+0000 7ff7759eb640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5569000f4db0 0x5569001007b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.46:58910/0 (socket says 172.21.15.46:58910) 2024-03-31T23:28:05.548 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:05.545+0000 7ff7759eb640 1 -- 172.21.15.46:0/3304285808 learned_addr learned my addr 172.21.15.46:0/3304285808 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:28:05.548 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:05.545+0000 7ff7749e9640 1 --2- 172.21.15.46:0/3304285808 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5569000ebfe0 0x556900101700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:28:05.548 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:05.545+0000 7ff7759eb640 1 -- 172.21.15.46:0/3304285808 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5569000ebfe0 msgr2=0x556900101700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:28:05.548 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:05.545+0000 7ff7759eb640 1 --2- 172.21.15.46:0/3304285808 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5569000ebfe0 0x556900101700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:28:05.549 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:05.545+0000 7ff7759eb640 1 -- 172.21.15.46:0/3304285808 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5569000eb6c0 msgr2=0x5569000f4520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:28:05.549 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:05.545+0000 7ff7759eb640 1 --2- 172.21.15.46:0/3304285808 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5569000eb6c0 0x5569000f4520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:28:05.549 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:05.545+0000 7ff7759eb640 1 -- 172.21.15.46:0/3304285808 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5569000f71a0 con 0x5569000f4db0 2024-03-31T23:28:05.549 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:05.545+0000 7ff7751ea640 1 --2- 172.21.15.46:0/3304285808 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5569000eb6c0 0x5569000f4520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:28:05.550 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:05.545+0000 7ff7759eb640 1 --2- 172.21.15.46:0/3304285808 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5569000f4db0 0x5569001007b0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7ff768010db0 tx=0x7ff768010de0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c5f5d1ab035c77fa server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:28:05.550 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:05.546+0000 7ff75affd640 1 -- 172.21.15.46:0/3304285808 <== mon.2 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff768022040 con 0x5569000f4db0 2024-03-31T23:28:05.550 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:05.546+0000 7ff75affd640 1 -- 172.21.15.46:0/3304285808 <== mon.2 v2:172.21.15.142:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff768012050 con 0x5569000f4db0 2024-03-31T23:28:05.550 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:05.546+0000 7ff75affd640 1 -- 172.21.15.46:0/3304285808 <== mon.2 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff768010430 con 0x5569000f4db0 2024-03-31T23:28:05.555 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:05.547+0000 7ff77a033740 1 -- 172.21.15.46:0/3304285808 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5569000f4db0 msgr2=0x5569001007b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:28:05.555 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:05.547+0000 7ff77a033740 1 --2- 172.21.15.46:0/3304285808 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5569000f4db0 0x5569001007b0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7ff768010db0 tx=0x7ff768010de0 comp rx=0 tx=0).stop 2024-03-31T23:28:05.555 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:05.547+0000 7ff77a033740 1 -- 172.21.15.46:0/3304285808 shutdown_connections 2024-03-31T23:28:05.555 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:05.547+0000 7ff77a033740 1 --2- 172.21.15.46:0/3304285808 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5569000f4db0 0x5569001007b0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:28:05.555 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:05.547+0000 7ff77a033740 1 --2- 172.21.15.46:0/3304285808 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5569000ebfe0 0x556900101700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:28:05.556 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:05.547+0000 7ff77a033740 1 --2- 172.21.15.46:0/3304285808 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5569000eb6c0 0x5569000f4520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:28:05.556 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:05.547+0000 7ff77a033740 1 -- 172.21.15.46:0/3304285808 >> 172.21.15.46:0/3304285808 conn(0x5569000e5fe0 msgr2=0x556900100ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:28:05.556 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:05.547+0000 7ff77a033740 1 -- 172.21.15.46:0/3304285808 shutdown_connections 2024-03-31T23:28:05.556 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:05.547+0000 7ff77a033740 1 -- 172.21.15.46:0/3304285808 wait complete. 2024-03-31T23:28:05.556 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:05.547+0000 7ff77a033740 1 -- start start 2024-03-31T23:28:05.556 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:05.548+0000 7ff77a033740 1 Processor -- start 2024-03-31T23:28:05.556 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:05.548+0000 7ff77a033740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5569000eb6c0 0x5569000fcdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:28:05.556 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:05.548+0000 7ff77a033740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5569000ebfe0 0x5569000f83a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:28:05.556 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:05.548+0000 7ff77a033740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5569000f4db0 0x5569000f88e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:28:05.556 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:05.549+0000 7ff77a033740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7ff75400e790 con 0x5569000eb6c0 2024-03-31T23:28:05.556 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:05.549+0000 7ff77a033740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7ff75400fc00 con 0x5569000ebfe0 2024-03-31T23:28:05.556 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:05.549+0000 7ff77a033740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7ff75400ee60 con 0x5569000f4db0 2024-03-31T23:28:05.556 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:05.549+0000 7ff7751ea640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5569000eb6c0 0x5569000fcdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:28:05.557 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:05.549+0000 7ff77a033740 10 client.0 _ll_register_callbacks cb 0x7ff764005e10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T23:28:05.557 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:05.549+0000 7ff7751ea640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5569000eb6c0 0x5569000fcdf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:52118/0 (socket says 172.21.15.46:52118) 2024-03-31T23:28:05.557 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:05.549+0000 7ff7751ea640 1 -- 172.21.15.46:0/2671976185 learned_addr learned my addr 172.21.15.46:0/2671976185 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:28:05.557 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:05.549+0000 7ff7749e9640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5569000ebfe0 0x5569000f83a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:28:05.557 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:05.549+0000 7ff7759eb640 1 --2- 172.21.15.46:0/2671976185 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5569000f4db0 0x5569000f88e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:28:05.557 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:05.549+0000 7ff7749e9640 1 -- 172.21.15.46:0/2671976185 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5569000f4db0 msgr2=0x5569000f88e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:28:05.557 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:05.549+0000 7ff7749e9640 1 --2- 172.21.15.46:0/2671976185 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5569000f4db0 0x5569000f88e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:28:05.557 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:05.549+0000 7ff7749e9640 1 -- 172.21.15.46:0/2671976185 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5569000eb6c0 msgr2=0x5569000fcdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:28:05.557 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:05.549+0000 7ff7749e9640 1 --2- 172.21.15.46:0/2671976185 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5569000eb6c0 0x5569000fcdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:28:05.557 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:05.549+0000 7ff7749e9640 1 -- 172.21.15.46:0/2671976185 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5569000f8ea0 con 0x5569000ebfe0 2024-03-31T23:28:05.557 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:05.549+0000 7ff7759eb640 1 --2- 172.21.15.46:0/2671976185 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5569000f4db0 0x5569000f88e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:28:05.557 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:05.549+0000 7ff7751ea640 1 --2- 172.21.15.46:0/2671976185 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5569000eb6c0 0x5569000fcdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:28:05.557 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:05.549+0000 7ff7749e9640 1 --2- 172.21.15.46:0/2671976185 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5569000ebfe0 0x5569000f83a0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x556900107af0 tx=0x7ff76c012170 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f89037982626ecaf server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:28:05.557 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:05.549+0000 7ff758ff9640 10 client.0 ms_handle_connect on v2:172.21.15.142:3300/0 2024-03-31T23:28:05.558 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:05.550+0000 7ff758ff9640 1 -- 172.21.15.46:0/2671976185 <== mon.2 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff76c004030 con 0x5569000ebfe0 2024-03-31T23:28:05.558 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:05.550+0000 7ff758ff9640 1 -- 172.21.15.46:0/2671976185 <== mon.2 v2:172.21.15.142:3300/0 2 ==== osd_map(118..118 src has 1..118) ==== 9642+0+0 (secure 0 0 0) 0x7ff76c0041d0 con 0x5569000ebfe0 2024-03-31T23:28:05.558 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:05.550+0000 7ff77a033740 1 -- 172.21.15.46:0/2671976185 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff768012050 con 0x5569000ebfe0 2024-03-31T23:28:05.558 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:05.550+0000 7ff758ff9640 1 -- 172.21.15.46:0/2671976185 <== mon.2 v2:172.21.15.142:3300/0 3 ==== mon_get_version_reply(handle=1 version=44) ==== 24+0+0 (secure 0 0 0) 0x7ff76c033050 con 0x5569000ebfe0 2024-03-31T23:28:05.558 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:05.550+0000 7ff77a033740 10 client.27311 fetch_fsmap learned FSMap version 44 2024-03-31T23:28:05.558 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:05.550+0000 7ff77a033740 1 -- 172.21.15.46:0/2671976185 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,fsmap=44,monmap=4+}) -- 0x55690016b020 con 0x5569000ebfe0 2024-03-31T23:28:05.558 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:05.550+0000 7ff758ff9640 1 -- 172.21.15.46:0/2671976185 <== mon.2 v2:172.21.15.142:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff768012050 con 0x5569000ebfe0 2024-03-31T23:28:05.558 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:05.550+0000 7ff758ff9640 1 -- 172.21.15.46:0/2671976185 <== mon.2 v2:172.21.15.142:3300/0 5 ==== fsmap(e 44) ==== 8354+0+0 (secure 0 0 0) 0x7ff76c003680 con 0x5569000ebfe0 2024-03-31T23:28:05.558 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:05.550+0000 7ff77a033740 10 client.27311 fetch_fsmap finished waiting for FSMap version 44 2024-03-31T23:28:05.558 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:05.550+0000 7ff77a033740 10 client.27311 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-31T23:28:05.558 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:05.551+0000 7ff77a033740 20 client.27311 populate_metadata read hostname 'smithi046' 2024-03-31T23:28:05.558 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:05.551+0000 7ff77a033740 1 --2- 172.21.15.46:0/2671976185 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55690016b7d0 0x55690016dbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:28:05.559 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:05.551+0000 7ff77a033740 4 client.27311 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-31T23:28:05.559 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:05.551+0000 7ff77a033740 1 -- 172.21.15.46:0/2671976185 --> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55690005be30 con 0x55690016b7d0 2024-03-31T23:28:05.559 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:05.551+0000 7ff7751ea640 1 --2- 172.21.15.46:0/2671976185 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55690016b7d0 0x55690016dbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:28:05.559 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:05.551+0000 7ff758ff9640 10 client.27311 ms_handle_connect on v2:172.21.15.142:6838/2519143918 2024-03-31T23:28:05.559 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:05.551+0000 7ff7751ea640 1 --2- 172.21.15.46:0/2671976185 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55690016b7d0 0x55690016dbf0 crc :-1 s=READY pgs=223 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=56891564a9943d8f server_cookie=3659dcc231f212c6 in_seq=0 out_seq=0 2024-03-31T23:28:05.714 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:05.712+0000 7ff758ff9640 1 -- 172.21.15.46:0/2671976185 <== mds.0 v2:172.21.15.142:6838/2519143918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55690005be30 con 0x55690016b7d0 2024-03-31T23:28:05.714 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:05.712+0000 7ff758ff9640 10 client.27311 handle_command_reply: tid=0 2024-03-31T23:28:05.714 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:05.712+0000 7ff77a033740 1 client.27311 shutdown 2024-03-31T23:28:05.714 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:05.712+0000 7ff77a033740 1 -- 172.21.15.46:0/2671976185 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5569000ebfe0 msgr2=0x5569000f83a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:28:05.714 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:05.712+0000 7ff77a033740 1 --2- 172.21.15.46:0/2671976185 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5569000ebfe0 0x5569000f83a0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x556900107af0 tx=0x7ff76c012170 comp rx=0 tx=0).stop 2024-03-31T23:28:05.715 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:05.712+0000 7ff77a033740 1 -- 172.21.15.46:0/2671976185 shutdown_connections 2024-03-31T23:28:05.715 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:05.712+0000 7ff77a033740 1 --2- 172.21.15.46:0/2671976185 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55690016b7d0 0x55690016dbf0 crc :-1 s=READY pgs=223 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:28:05.715 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:05.712+0000 7ff77a033740 1 --2- 172.21.15.46:0/2671976185 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5569000f4db0 0x5569000f88e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:28:05.715 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:05.712+0000 7ff758ff9640 0 client.27311 ms_handle_reset on v2:172.21.15.142:6838/2519143918 2024-03-31T23:28:05.715 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:05.712+0000 7ff77a033740 1 --2- 172.21.15.46:0/2671976185 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5569000ebfe0 0x5569000f83a0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:28:05.715 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:05.712+0000 7ff77a033740 1 --2- 172.21.15.46:0/2671976185 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5569000eb6c0 0x5569000fcdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:28:05.715 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:05.712+0000 7ff77a033740 1 -- 172.21.15.46:0/2671976185 >> 172.21.15.46:0/2671976185 conn(0x5569000e5fe0 msgr2=0x5569000e46f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:28:05.715 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:05.713+0000 7ff77a033740 1 -- 172.21.15.46:0/2671976185 shutdown_connections 2024-03-31T23:28:05.715 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:05.713+0000 7ff77a033740 1 -- 172.21.15.46:0/2671976185 wait complete. 2024-03-31T23:28:05.715 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:05.713+0000 7ff77a033740 20 client.27311 trim_cache size 0 max 16384 2024-03-31T23:28:05.716 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout:{ 2024-03-31T23:28:05.716 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout: "message": "", 2024-03-31T23:28:05.716 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout: "return_code": 0 2024-03-31T23:28:05.716 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout:} 2024-03-31T23:28:05.717 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:05.714+0000 7ff77744e640 1 -- 172.21.15.46:0/78119239 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7ff75c0a2730 msgr2=0x7ff75c0a4bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:28:05.717 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:05.715+0000 7ff77744e640 1 --2- 172.21.15.46:0/78119239 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7ff75c0a2730 0x7ff75c0a4bf0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7ff768023000 tx=0x7ff768021000 comp rx=0 tx=0).stop 2024-03-31T23:28:05.717 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:05.715+0000 7ff77744e640 1 -- 172.21.15.46:0/78119239 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff7701005e0 msgr2=0x7ff77019d860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:28:05.717 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:05.715+0000 7ff77744e640 1 --2- 172.21.15.46:0/78119239 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff7701005e0 0x7ff77019d860 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7ff764009900 tx=0x7ff76402ca90 comp rx=0 tx=0).stop 2024-03-31T23:28:05.717 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:05.715+0000 7ff77744e640 1 -- 172.21.15.46:0/78119239 shutdown_connections 2024-03-31T23:28:05.717 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:05.715+0000 7ff77744e640 1 --2- 172.21.15.46:0/78119239 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7ff75c0a2730 0x7ff75c0a4bf0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:28:05.717 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:05.715+0000 7ff77744e640 1 --2- 172.21.15.46:0/78119239 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff770106680 0x7ff770197a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:28:05.717 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:05.715+0000 7ff77744e640 1 --2- 172.21.15.46:0/78119239 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff770100fa0 0x7ff77019dda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:28:05.717 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:05.715+0000 7ff77744e640 1 --2- 172.21.15.46:0/78119239 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff7701005e0 0x7ff77019d860 unknown :-1 s=CLOSED pgs=916 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:28:05.717 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:05.715+0000 7ff77744e640 1 -- 172.21.15.46:0/78119239 >> 172.21.15.46:0/78119239 conn(0x7ff770077710 msgr2=0x7ff7701042a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:28:05.718 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:05.715+0000 7ff77744e640 1 -- 172.21.15.46:0/78119239 shutdown_connections 2024-03-31T23:28:05.718 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:05.715+0000 7ff77744e640 1 -- 172.21.15.46:0/78119239 wait complete. 2024-03-31T23:28:06.280 INFO:tasks.workunit.client.0.smithi046.stdout: 10 31199 8.79 MB/sec execute 71 sec latency 152.057 ms 2024-03-31T23:28:06.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:28:06 smithi142 ceph-mon[35271]: pgmap v516: 129 pgs: 129 active+clean; 622 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 5.5 MiB/s wr, 290 op/s 2024-03-31T23:28:07.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:28:06 smithi162 ceph-mon[36237]: pgmap v516: 129 pgs: 129 active+clean; 622 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 5.5 MiB/s wr, 290 op/s 2024-03-31T23:28:07.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:28:06 smithi046 ceph-mon[32498]: pgmap v516: 129 pgs: 129 active+clean; 622 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 5.5 MiB/s wr, 290 op/s 2024-03-31T23:28:07.280 INFO:tasks.workunit.client.0.smithi046.stdout: 10 31355 8.74 MB/sec execute 72 sec latency 103.855 ms 2024-03-31T23:28:08.280 INFO:tasks.workunit.client.0.smithi046.stdout: 10 31497 8.68 MB/sec execute 73 sec latency 69.911 ms 2024-03-31T23:28:08.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:28:08 smithi142 ceph-mon[35271]: pgmap v517: 129 pgs: 129 active+clean; 622 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 5.5 MiB/s wr, 290 op/s 2024-03-31T23:28:09.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:28:08 smithi162 ceph-mon[36237]: pgmap v517: 129 pgs: 129 active+clean; 622 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 5.5 MiB/s wr, 290 op/s 2024-03-31T23:28:09.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:28:08 smithi046 ceph-mon[32498]: pgmap v517: 129 pgs: 129 active+clean; 622 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 5.5 MiB/s wr, 290 op/s 2024-03-31T23:28:09.280 INFO:tasks.workunit.client.0.smithi046.stdout: 10 31654 8.65 MB/sec execute 74 sec latency 75.528 ms 2024-03-31T23:28:09.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:28:09 smithi142 ceph-mon[35271]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-31T23:28:10.012 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:28:09 smithi046 ceph-mon[32498]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-31T23:28:10.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:28:09 smithi162 ceph-mon[36237]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-31T23:28:10.280 INFO:tasks.workunit.client.0.smithi046.stdout: 10 31834 8.72 MB/sec execute 75 sec latency 72.556 ms 2024-03-31T23:28:10.944 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:28:10 smithi142 ceph-mon[35271]: pgmap v518: 129 pgs: 129 active+clean; 630 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 5.6 MiB/s wr, 340 op/s 2024-03-31T23:28:10.944 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:28:10 smithi142 ceph-mon[35271]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:28:10.944 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:28:10 smithi142 ceph-mon[35271]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-31T23:28:10.944 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:28:10 smithi142 ceph-mon[35271]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' 2024-03-31T23:28:10.944 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:28:10 smithi142 ceph-mon[35271]: osdmap e119: 12 total, 12 up, 12 in 2024-03-31T23:28:11.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:28:10 smithi162 ceph-mon[36237]: pgmap v518: 129 pgs: 129 active+clean; 630 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 5.6 MiB/s wr, 340 op/s 2024-03-31T23:28:11.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:28:10 smithi162 ceph-mon[36237]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:28:11.047 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:28:10 smithi162 ceph-mon[36237]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-31T23:28:11.047 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:28:10 smithi162 ceph-mon[36237]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' 2024-03-31T23:28:11.047 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:28:10 smithi162 ceph-mon[36237]: osdmap e119: 12 total, 12 up, 12 in 2024-03-31T23:28:11.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:28:10 smithi046 ceph-mon[32498]: pgmap v518: 129 pgs: 129 active+clean; 630 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 5.6 MiB/s wr, 340 op/s 2024-03-31T23:28:11.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:28:10 smithi046 ceph-mon[32498]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:28:11.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:28:10 smithi046 ceph-mon[32498]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-31T23:28:11.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:28:10 smithi046 ceph-mon[32498]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' 2024-03-31T23:28:11.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:28:10 smithi046 ceph-mon[32498]: osdmap e119: 12 total, 12 up, 12 in 2024-03-31T23:28:11.280 INFO:tasks.workunit.client.0.smithi046.stdout: 10 31985 8.67 MB/sec execute 76 sec latency 77.350 ms 2024-03-31T23:28:12.280 INFO:tasks.workunit.client.0.smithi046.stdout: 10 32174 8.66 MB/sec execute 77 sec latency 63.514 ms 2024-03-31T23:28:12.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:28:12 smithi142 ceph-mon[35271]: pgmap v520: 129 pgs: 129 active+clean; 630 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 6.8 MiB/s wr, 409 op/s 2024-03-31T23:28:13.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:28:12 smithi162 ceph-mon[36237]: pgmap v520: 129 pgs: 129 active+clean; 630 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 6.8 MiB/s wr, 409 op/s 2024-03-31T23:28:13.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:28:12 smithi046 ceph-mon[32498]: pgmap v520: 129 pgs: 129 active+clean; 630 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 6.8 MiB/s wr, 409 op/s 2024-03-31T23:28:13.280 INFO:tasks.workunit.client.0.smithi046.stdout: 10 32342 8.67 MB/sec execute 78 sec latency 99.874 ms 2024-03-31T23:28:14.280 INFO:tasks.workunit.client.0.smithi046.stdout: 10 32523 8.69 MB/sec execute 79 sec latency 61.690 ms 2024-03-31T23:28:14.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:28:14 smithi142 ceph-mon[35271]: pgmap v521: 129 pgs: 129 active+clean; 630 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 5.6 MiB/s wr, 340 op/s 2024-03-31T23:28:15.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:28:14 smithi162 ceph-mon[36237]: pgmap v521: 129 pgs: 129 active+clean; 630 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 5.6 MiB/s wr, 340 op/s 2024-03-31T23:28:15.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:28:14 smithi046 ceph-mon[32498]: pgmap v521: 129 pgs: 129 active+clean; 630 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 5.6 MiB/s wr, 340 op/s 2024-03-31T23:28:15.280 INFO:tasks.workunit.client.0.smithi046.stdout: 10 32688 8.72 MB/sec execute 80 sec latency 131.213 ms 2024-03-31T23:28:16.281 INFO:tasks.workunit.client.0.smithi046.stdout: 10 32833 8.74 MB/sec execute 81 sec latency 131.861 ms 2024-03-31T23:28:16.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:28:16 smithi142 ceph-mon[35271]: pgmap v522: 129 pgs: 129 active+clean; 640 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 4.7 MiB/s wr, 329 op/s 2024-03-31T23:28:17.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:28:16 smithi162 ceph-mon[36237]: pgmap v522: 129 pgs: 129 active+clean; 640 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 4.7 MiB/s wr, 329 op/s 2024-03-31T23:28:17.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:28:16 smithi046 ceph-mon[32498]: pgmap v522: 129 pgs: 129 active+clean; 640 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 4.7 MiB/s wr, 329 op/s 2024-03-31T23:28:17.280 INFO:tasks.workunit.client.0.smithi046.stdout: 10 33015 8.80 MB/sec execute 82 sec latency 66.283 ms 2024-03-31T23:28:18.280 INFO:tasks.workunit.client.0.smithi046.stdout: 10 33219 8.84 MB/sec execute 83 sec latency 70.419 ms 2024-03-31T23:28:18.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:28:18 smithi142 ceph-mon[35271]: pgmap v523: 129 pgs: 129 active+clean; 640 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 4.7 MiB/s wr, 329 op/s 2024-03-31T23:28:19.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:28:18 smithi162 ceph-mon[36237]: pgmap v523: 129 pgs: 129 active+clean; 640 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 4.7 MiB/s wr, 329 op/s 2024-03-31T23:28:19.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:28:18 smithi046 ceph-mon[32498]: pgmap v523: 129 pgs: 129 active+clean; 640 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 4.7 MiB/s wr, 329 op/s 2024-03-31T23:28:19.281 INFO:tasks.workunit.client.0.smithi046.stdout: 10 33378 8.82 MB/sec execute 84 sec latency 69.720 ms 2024-03-31T23:28:19.547 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:19.545+0000 7fe4dacae640 1 -- 172.21.15.46:0/155918498 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe4d4102970 msgr2=0x7fe4d4102d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:28:19.547 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:19.545+0000 7fe4dacae640 1 --2- 172.21.15.46:0/155918498 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe4d4102970 0x7fe4d4102d50 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x7fe4c8009900 tx=0x7fe4c802bcd0 comp rx=0 tx=0).stop 2024-03-31T23:28:19.547 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:19.545+0000 7fe4dacae640 1 -- 172.21.15.46:0/155918498 shutdown_connections 2024-03-31T23:28:19.548 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:19.545+0000 7fe4dacae640 1 --2- 172.21.15.46:0/155918498 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe4d410bd40 0x7fe4d410e1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:28:19.548 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:19.545+0000 7fe4dacae640 1 --2- 172.21.15.46:0/155918498 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe4d4103290 0x7fe4d410b770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:28:19.548 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:19.545+0000 7fe4dacae640 1 --2- 172.21.15.46:0/155918498 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe4d4102970 0x7fe4d4102d50 unknown :-1 s=CLOSED pgs=919 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:28:19.548 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:19.545+0000 7fe4dacae640 1 -- 172.21.15.46:0/155918498 >> 172.21.15.46:0/155918498 conn(0x7fe4d4077860 msgr2=0x7fe4d4077c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:28:19.548 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:19.545+0000 7fe4dacae640 1 -- 172.21.15.46:0/155918498 shutdown_connections 2024-03-31T23:28:19.548 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:19.545+0000 7fe4dacae640 1 -- 172.21.15.46:0/155918498 wait complete. 2024-03-31T23:28:19.548 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:19.546+0000 7fe4dacae640 1 Processor -- start 2024-03-31T23:28:19.548 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:19.546+0000 7fe4dacae640 1 -- start start 2024-03-31T23:28:19.548 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:19.546+0000 7fe4dacae640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe4d4102970 0x7fe4d4197ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:28:19.548 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:19.546+0000 7fe4dacae640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe4d4103290 0x7fe4d4198410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:28:19.548 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:19.546+0000 7fe4dacae640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe4d410bd40 0x7fe4d419c740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:28:19.548 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:19.546+0000 7fe4dacae640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fe4d41119d0 con 0x7fe4d410bd40 2024-03-31T23:28:19.549 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:19.546+0000 7fe4dacae640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fe4d4111850 con 0x7fe4d4103290 2024-03-31T23:28:19.549 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:19.546+0000 7fe4dacae640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fe4d4111b50 con 0x7fe4d4102970 2024-03-31T23:28:19.549 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:19.546+0000 7fe4d924b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe4d410bd40 0x7fe4d419c740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:28:19.549 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:19.546+0000 7fe4d8a4a640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe4d4102970 0x7fe4d4197ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:28:19.549 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:19.546+0000 7fe4d924b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe4d410bd40 0x7fe4d419c740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:35274/0 (socket says 172.21.15.46:35274) 2024-03-31T23:28:19.549 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:19.546+0000 7fe4d924b640 1 -- 172.21.15.46:0/2446905229 learned_addr learned my addr 172.21.15.46:0/2446905229 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:28:19.549 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:19.546+0000 7fe4d3fff640 1 --2- 172.21.15.46:0/2446905229 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe4d4103290 0x7fe4d4198410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:28:19.551 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:19.547+0000 7fe4d924b640 1 -- 172.21.15.46:0/2446905229 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe4d4102970 msgr2=0x7fe4d4197ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:28:19.551 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:19.547+0000 7fe4d924b640 1 --2- 172.21.15.46:0/2446905229 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe4d4102970 0x7fe4d4197ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:28:19.551 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:19.547+0000 7fe4d924b640 1 -- 172.21.15.46:0/2446905229 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe4d4103290 msgr2=0x7fe4d4198410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:28:19.551 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:19.547+0000 7fe4d924b640 1 --2- 172.21.15.46:0/2446905229 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe4d4103290 0x7fe4d4198410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:28:19.551 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:19.547+0000 7fe4d924b640 1 -- 172.21.15.46:0/2446905229 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe4d419ce70 con 0x7fe4d410bd40 2024-03-31T23:28:19.551 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:19.547+0000 7fe4d8a4a640 1 --2- 172.21.15.46:0/2446905229 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe4d4102970 0x7fe4d4197ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:28:19.551 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:19.547+0000 7fe4d3fff640 1 --2- 172.21.15.46:0/2446905229 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe4d4103290 0x7fe4d4198410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:28:19.551 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:19.547+0000 7fe4d924b640 1 --2- 172.21.15.46:0/2446905229 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe4d410bd40 0x7fe4d419c740 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7fe4cc00ea60 tx=0x7fe4cc00ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:28:19.551 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:19.547+0000 7fe4d1ffb640 1 -- 172.21.15.46:0/2446905229 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4cc00ce40 con 0x7fe4d410bd40 2024-03-31T23:28:19.551 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:19.547+0000 7fe4d1ffb640 1 -- 172.21.15.46:0/2446905229 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe4cc005020 con 0x7fe4d410bd40 2024-03-31T23:28:19.551 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:19.547+0000 7fe4dacae640 1 -- 172.21.15.46:0/2446905229 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe4d419d160 con 0x7fe4d410bd40 2024-03-31T23:28:19.551 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:19.547+0000 7fe4d1ffb640 1 -- 172.21.15.46:0/2446905229 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4cc010670 con 0x7fe4d410bd40 2024-03-31T23:28:19.551 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:19.548+0000 7fe4dacae640 1 -- 172.21.15.46:0/2446905229 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe4d41a49f0 con 0x7fe4d410bd40 2024-03-31T23:28:19.552 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:19.550+0000 7fe4d1ffb640 1 -- 172.21.15.46:0/2446905229 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 25) ==== 137451+0+0 (secure 0 0 0) 0x7fe4cc0051d0 con 0x7fe4d410bd40 2024-03-31T23:28:19.553 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:19.550+0000 7fe4d1ffb640 1 --2- 172.21.15.46:0/2446905229 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fe4c40a2730 0x7fe4c40a4bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:28:19.553 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:19.551+0000 7fe4d8a4a640 1 --2- 172.21.15.46:0/2446905229 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fe4c40a2730 0x7fe4c40a4bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:28:19.553 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:19.551+0000 7fe4d1ffb640 1 -- 172.21.15.46:0/2446905229 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(119..119 src has 1..119) ==== 9554+0+0 (secure 0 0 0) 0x7fe4cc0cf2d0 con 0x7fe4d410bd40 2024-03-31T23:28:19.553 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:19.551+0000 7fe4d8a4a640 1 --2- 172.21.15.46:0/2446905229 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fe4c40a2730 0x7fe4c40a4bf0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fe4c8006010 tx=0x7fe4c8005e40 comp rx=0 tx=0).ready entity=mgr.15654 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:28:19.588 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:19.586+0000 7fe4dd96c740 1 Processor -- start 2024-03-31T23:28:19.588 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:19.587+0000 7fe4dd96c740 1 -- start start 2024-03-31T23:28:19.588 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:19.587+0000 7fe4dd96c740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561f0ba9d6c0 0x561f0ba9daa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:28:19.589 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:19.587+0000 7fe4dd96c740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x561f0ba9dfe0 0x561f0baa6870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:28:19.589 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:19.587+0000 7fe4dd96c740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561f0baa6db0 0x561f0baa91a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:28:19.589 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:19.587+0000 7fe4dd96c740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x561f0ba008f0 con 0x561f0baa6db0 2024-03-31T23:28:19.589 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:19.587+0000 7fe4dd96c740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x561f0b9a98d0 con 0x561f0ba9dfe0 2024-03-31T23:28:19.589 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:19.587+0000 7fe4dd96c740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x561f0b9fc990 con 0x561f0ba9d6c0 2024-03-31T23:28:19.589 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:19.587+0000 7fe4d924b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561f0baa6db0 0x561f0baa91a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:28:19.589 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:19.587+0000 7fe4d924b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561f0baa6db0 0x561f0baa91a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:35282/0 (socket says 172.21.15.46:35282) 2024-03-31T23:28:19.590 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:19.587+0000 7fe4d924b640 1 -- 172.21.15.46:0/1637536298 learned_addr learned my addr 172.21.15.46:0/1637536298 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:28:19.590 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:19.587+0000 7fe4d3fff640 1 --2- 172.21.15.46:0/1637536298 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x561f0ba9dfe0 0x561f0baa6870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:28:19.590 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:19.587+0000 7fe4d8a4a640 1 --2- 172.21.15.46:0/1637536298 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561f0ba9d6c0 0x561f0ba9daa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:28:19.590 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:19.588+0000 7fe4d924b640 1 -- 172.21.15.46:0/1637536298 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561f0ba9d6c0 msgr2=0x561f0ba9daa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:28:19.590 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:19.588+0000 7fe4d924b640 1 --2- 172.21.15.46:0/1637536298 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561f0ba9d6c0 0x561f0ba9daa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:28:19.590 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:19.588+0000 7fe4d924b640 1 -- 172.21.15.46:0/1637536298 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x561f0ba9dfe0 msgr2=0x561f0baa6870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:28:19.590 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:19.588+0000 7fe4d924b640 1 --2- 172.21.15.46:0/1637536298 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x561f0ba9dfe0 0x561f0baa6870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:28:19.590 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:19.588+0000 7fe4d924b640 1 -- 172.21.15.46:0/1637536298 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561f0b715210 con 0x561f0baa6db0 2024-03-31T23:28:19.590 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:19.588+0000 7fe4d8a4a640 1 --2- 172.21.15.46:0/1637536298 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561f0ba9d6c0 0x561f0ba9daa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T23:28:19.590 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:19.588+0000 7fe4d924b640 1 --2- 172.21.15.46:0/1637536298 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561f0baa6db0 0x561f0baa91a0 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7fe4cc0a3b20 tx=0x7fe4cc09afa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2ff3a12c87e710d9 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:28:19.590 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:19.588+0000 7fe4beffd640 1 -- 172.21.15.46:0/1637536298 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4cc09f790 con 0x561f0baa6db0 2024-03-31T23:28:19.590 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:19.588+0000 7fe4beffd640 1 -- 172.21.15.46:0/1637536298 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe4cc0a4050 con 0x561f0baa6db0 2024-03-31T23:28:19.590 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:19.588+0000 7fe4beffd640 1 -- 172.21.15.46:0/1637536298 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4cc0b4210 con 0x561f0baa6db0 2024-03-31T23:28:19.594 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:19.589+0000 7fe4dd96c740 1 -- 172.21.15.46:0/1637536298 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561f0baa6db0 msgr2=0x561f0baa91a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:28:19.594 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:19.589+0000 7fe4dd96c740 1 --2- 172.21.15.46:0/1637536298 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561f0baa6db0 0x561f0baa91a0 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7fe4cc0a3b20 tx=0x7fe4cc09afa0 comp rx=0 tx=0).stop 2024-03-31T23:28:19.594 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:19.589+0000 7fe4dd96c740 1 -- 172.21.15.46:0/1637536298 shutdown_connections 2024-03-31T23:28:19.594 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:19.589+0000 7fe4dd96c740 1 --2- 172.21.15.46:0/1637536298 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561f0baa6db0 0x561f0baa91a0 unknown :-1 s=CLOSED pgs=921 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:28:19.594 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:19.589+0000 7fe4dd96c740 1 --2- 172.21.15.46:0/1637536298 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x561f0ba9dfe0 0x561f0baa6870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:28:19.594 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:19.589+0000 7fe4dd96c740 1 --2- 172.21.15.46:0/1637536298 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561f0ba9d6c0 0x561f0ba9daa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:28:19.595 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:19.589+0000 7fe4dd96c740 1 -- 172.21.15.46:0/1637536298 >> 172.21.15.46:0/1637536298 conn(0x561f0ba97fe0 msgr2=0x561f0ba983c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:28:19.595 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:19.589+0000 7fe4dd96c740 1 -- 172.21.15.46:0/1637536298 shutdown_connections 2024-03-31T23:28:19.595 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:19.589+0000 7fe4dd96c740 1 -- 172.21.15.46:0/1637536298 wait complete. 2024-03-31T23:28:19.595 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:19.589+0000 7fe4dd96c740 1 -- start start 2024-03-31T23:28:19.595 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:19.590+0000 7fe4dd96c740 1 Processor -- start 2024-03-31T23:28:19.595 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:19.590+0000 7fe4dd96c740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x561f0ba9d6c0 0x561f0baa3ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:28:19.595 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:19.590+0000 7fe4dd96c740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561f0ba9dfe0 0x561f0baa43e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:28:19.595 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:19.590+0000 7fe4dd96c740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561f0baa6db0 0x561f0baa4920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:28:19.595 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:19.590+0000 7fe4dd96c740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x561f0baadca0 con 0x561f0baa6db0 2024-03-31T23:28:19.595 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:19.590+0000 7fe4dd96c740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x561f0b99ef80 con 0x561f0ba9d6c0 2024-03-31T23:28:19.595 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:19.590+0000 7fe4dd96c740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x561f0baade20 con 0x561f0ba9dfe0 2024-03-31T23:28:19.595 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:19.590+0000 7fe4dd96c740 10 client.0 _ll_register_callbacks cb 0x561f0b9f7290 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T23:28:19.596 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:19.590+0000 7fe4d924b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561f0baa6db0 0x561f0baa4920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:28:19.596 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:19.590+0000 7fe4d924b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561f0baa6db0 0x561f0baa4920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:35294/0 (socket says 172.21.15.46:35294) 2024-03-31T23:28:19.596 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:19.590+0000 7fe4d924b640 1 -- 172.21.15.46:0/1647366262 learned_addr learned my addr 172.21.15.46:0/1647366262 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:28:19.596 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:19.590+0000 7fe4d8a4a640 1 --2- 172.21.15.46:0/1647366262 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x561f0ba9d6c0 0x561f0baa3ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:28:19.596 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:19.590+0000 7fe4d3fff640 1 --2- 172.21.15.46:0/1647366262 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561f0ba9dfe0 0x561f0baa43e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:28:19.596 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:19.591+0000 7fe4d924b640 1 -- 172.21.15.46:0/1647366262 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561f0ba9dfe0 msgr2=0x561f0baa43e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:28:19.596 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:19.591+0000 7fe4d924b640 1 --2- 172.21.15.46:0/1647366262 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561f0ba9dfe0 0x561f0baa43e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:28:19.596 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:19.591+0000 7fe4d924b640 1 -- 172.21.15.46:0/1647366262 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x561f0ba9d6c0 msgr2=0x561f0baa3ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:28:19.596 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:19.591+0000 7fe4d924b640 1 --2- 172.21.15.46:0/1647366262 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x561f0ba9d6c0 0x561f0baa3ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:28:19.596 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:19.591+0000 7fe4d924b640 1 -- 172.21.15.46:0/1647366262 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561f0b7e1620 con 0x561f0baa6db0 2024-03-31T23:28:19.596 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:19.591+0000 7fe4d8a4a640 1 --2- 172.21.15.46:0/1647366262 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x561f0ba9d6c0 0x561f0baa3ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:28:19.596 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:19.591+0000 7fe4d924b640 1 --2- 172.21.15.46:0/1647366262 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561f0baa6db0 0x561f0baa4920 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 crypto rx=0x7fe4cc0cc6c0 tx=0x7fe4cc005750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3927872dbf9a5b39 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:28:19.596 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:19.591+0000 7fe4b3fff640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-31T23:28:19.596 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:19.591+0000 7fe4b3fff640 1 -- 172.21.15.46:0/1647366262 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4cc0a3bc0 con 0x561f0baa6db0 2024-03-31T23:28:19.597 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:19.591+0000 7fe4dd96c740 1 -- 172.21.15.46:0/1647366262 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe4cc0a4050 con 0x561f0baa6db0 2024-03-31T23:28:19.597 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:19.591+0000 7fe4b3fff640 1 -- 172.21.15.46:0/1647366262 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(119..119 src has 1..119) ==== 9554+0+0 (secure 0 0 0) 0x7fe4cc0cc890 con 0x561f0baa6db0 2024-03-31T23:28:19.597 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:19.591+0000 7fe4b3fff640 1 -- 172.21.15.46:0/1647366262 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=44) ==== 24+0+0 (secure 0 0 0) 0x7fe4cc0a3d60 con 0x561f0baa6db0 2024-03-31T23:28:19.597 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:19.591+0000 7fe4dd96c740 10 client.17493 fetch_fsmap learned FSMap version 44 2024-03-31T23:28:19.597 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:19.591+0000 7fe4dd96c740 1 -- 172.21.15.46:0/1647366262 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=44,monmap=4+}) -- 0x561f0bab9d50 con 0x561f0baa6db0 2024-03-31T23:28:19.597 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:19.592+0000 7fe4b3fff640 1 -- 172.21.15.46:0/1647366262 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe4cc0a4050 con 0x561f0baa6db0 2024-03-31T23:28:19.597 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:19.592+0000 7fe4b3fff640 1 -- 172.21.15.46:0/1647366262 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 44) ==== 8354+0+0 (secure 0 0 0) 0x7fe4cc082030 con 0x561f0baa6db0 2024-03-31T23:28:19.597 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:19.592+0000 7fe4dd96c740 10 client.17493 fetch_fsmap finished waiting for FSMap version 44 2024-03-31T23:28:19.597 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:19.592+0000 7fe4dd96c740 10 client.17493 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-31T23:28:19.597 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:19.592+0000 7fe4dd96c740 20 client.17493 populate_metadata read hostname 'smithi046' 2024-03-31T23:28:19.597 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:19.592+0000 7fe4dd96c740 1 --2- 172.21.15.46:0/1647366262 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x561f0babb0a0 0x561f0babd490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:28:19.597 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:19.592+0000 7fe4dd96c740 4 client.17493 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-31T23:28:19.597 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:19.592+0000 7fe4dd96c740 1 -- 172.21.15.46:0/1647366262 --> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561f0ba14120 con 0x561f0babb0a0 2024-03-31T23:28:19.597 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:19.592+0000 7fe4d8a4a640 1 --2- 172.21.15.46:0/1647366262 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x561f0babb0a0 0x561f0babd490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:28:19.598 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:19.593+0000 7fe4d8a4a640 1 --2- 172.21.15.46:0/1647366262 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x561f0babb0a0 0x561f0babd490 crc :-1 s=READY pgs=224 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=afdecf98daa60ed1 server_cookie=7b515f2549377814 in_seq=0 out_seq=0 2024-03-31T23:28:19.598 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:19.593+0000 7fe4b3fff640 10 client.17493 ms_handle_connect on v2:172.21.15.142:6838/2519143918 2024-03-31T23:28:19.599 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:19.594+0000 7fe4b3fff640 1 -- 172.21.15.46:0/1647366262 <== mds.0 v2:172.21.15.142:6838/2519143918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x561f0ba14120 con 0x561f0babb0a0 2024-03-31T23:28:19.599 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:19.594+0000 7fe4b3fff640 10 client.17493 handle_command_reply: tid=0 2024-03-31T23:28:19.599 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:19.594+0000 7fe4dd96c740 1 client.17493 shutdown 2024-03-31T23:28:19.599 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:19.594+0000 7fe4dd96c740 1 -- 172.21.15.46:0/1647366262 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561f0baa6db0 msgr2=0x561f0baa4920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:28:19.599 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:19.594+0000 7fe4dd96c740 1 --2- 172.21.15.46:0/1647366262 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561f0baa6db0 0x561f0baa4920 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 crypto rx=0x7fe4cc0cc6c0 tx=0x7fe4cc005750 comp rx=0 tx=0).stop 2024-03-31T23:28:19.599 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:19.595+0000 7fe4dd96c740 1 -- 172.21.15.46:0/1647366262 shutdown_connections 2024-03-31T23:28:19.599 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:19.595+0000 7fe4dd96c740 1 --2- 172.21.15.46:0/1647366262 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x561f0babb0a0 0x561f0babd490 crc :-1 s=READY pgs=224 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:28:19.599 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:19.595+0000 7fe4dd96c740 1 --2- 172.21.15.46:0/1647366262 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561f0baa6db0 0x561f0baa4920 unknown :-1 s=CLOSED pgs=922 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:28:19.599 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:19.595+0000 7fe4dd96c740 1 --2- 172.21.15.46:0/1647366262 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561f0ba9dfe0 0x561f0baa43e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:28:19.599 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:19.595+0000 7fe4b3fff640 0 client.17493 ms_handle_reset on v2:172.21.15.142:6838/2519143918 2024-03-31T23:28:19.599 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:19.595+0000 7fe4dd96c740 1 --2- 172.21.15.46:0/1647366262 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x561f0ba9d6c0 0x561f0baa3ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:28:19.600 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:19.595+0000 7fe4dd96c740 1 -- 172.21.15.46:0/1647366262 >> 172.21.15.46:0/1647366262 conn(0x561f0ba97fe0 msgr2=0x561f0ba99d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:28:19.600 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:19.595+0000 7fe4dd96c740 1 -- 172.21.15.46:0/1647366262 shutdown_connections 2024-03-31T23:28:19.600 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:19.595+0000 7fe4dd96c740 1 -- 172.21.15.46:0/1647366262 wait complete. 2024-03-31T23:28:19.600 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:19.595+0000 7fe4dd96c740 20 client.17493 trim_cache size 0 max 16384 2024-03-31T23:28:19.626 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:19.622+0000 7fe4dd96c740 1 Processor -- start 2024-03-31T23:28:19.626 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:19.622+0000 7fe4dd96c740 1 -- start start 2024-03-31T23:28:19.626 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:19.622+0000 7fe4dd96c740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561f0ba9d6c0 0x561f0baa65d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:28:19.626 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:19.622+0000 7fe4dd96c740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561f0ba9dfe0 0x561f0bab6b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:28:19.627 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:19.622+0000 7fe4dd96c740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x561f0baa6db0 0x561f0baa2e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:28:19.627 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:19.622+0000 7fe4dd96c740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fe4d41af490 con 0x561f0ba9d6c0 2024-03-31T23:28:19.627 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:19.622+0000 7fe4dd96c740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fe4d41acd00 con 0x561f0baa6db0 2024-03-31T23:28:19.627 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:19.622+0000 7fe4dd96c740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fe4d41aab70 con 0x561f0ba9dfe0 2024-03-31T23:28:19.627 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:19.622+0000 7fe4d8a4a640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561f0ba9d6c0 0x561f0baa65d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:28:19.627 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:19.622+0000 7fe4d3fff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561f0ba9dfe0 0x561f0bab6b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:28:19.628 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:19.622+0000 7fe4d8a4a640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561f0ba9d6c0 0x561f0baa65d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:35302/0 (socket says 172.21.15.46:35302) 2024-03-31T23:28:19.628 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:19.622+0000 7fe4d8a4a640 1 -- 172.21.15.46:0/1847005543 learned_addr learned my addr 172.21.15.46:0/1847005543 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:28:19.628 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:19.622+0000 7fe4d924b640 1 --2- 172.21.15.46:0/1847005543 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x561f0baa6db0 0x561f0baa2e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:28:19.628 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:19.623+0000 7fe4d8a4a640 1 -- 172.21.15.46:0/1847005543 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561f0ba9dfe0 msgr2=0x561f0bab6b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:28:19.628 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:19.623+0000 7fe4d8a4a640 1 --2- 172.21.15.46:0/1847005543 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561f0ba9dfe0 0x561f0bab6b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:28:19.628 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:19.623+0000 7fe4d8a4a640 1 -- 172.21.15.46:0/1847005543 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x561f0baa6db0 msgr2=0x561f0baa2e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:28:19.628 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:19.623+0000 7fe4d8a4a640 1 --2- 172.21.15.46:0/1847005543 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x561f0baa6db0 0x561f0baa2e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:28:19.628 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:19.623+0000 7fe4d8a4a640 1 -- 172.21.15.46:0/1847005543 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561f0bab7060 con 0x561f0ba9d6c0 2024-03-31T23:28:19.628 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:19.623+0000 7fe4d3fff640 1 --2- 172.21.15.46:0/1847005543 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561f0ba9dfe0 0x561f0bab6b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:28:19.628 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:19.623+0000 7fe4d8a4a640 1 --2- 172.21.15.46:0/1847005543 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561f0ba9d6c0 0x561f0baa65d0 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 crypto rx=0x7fe4c802c210 tx=0x7fe4c8042000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=930e0d1a32c679e6 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:28:19.629 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:19.623+0000 7fe4be7fc640 1 -- 172.21.15.46:0/1847005543 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4c805a070 con 0x561f0ba9d6c0 2024-03-31T23:28:19.629 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:19.623+0000 7fe4be7fc640 1 -- 172.21.15.46:0/1847005543 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe4c80542f0 con 0x561f0ba9d6c0 2024-03-31T23:28:19.629 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:19.623+0000 7fe4be7fc640 1 -- 172.21.15.46:0/1847005543 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4c8003b40 con 0x561f0ba9d6c0 2024-03-31T23:28:19.629 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:19.624+0000 7fe4dd96c740 1 -- 172.21.15.46:0/1847005543 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561f0ba9d6c0 msgr2=0x561f0baa65d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:28:19.629 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:19.624+0000 7fe4dd96c740 1 --2- 172.21.15.46:0/1847005543 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561f0ba9d6c0 0x561f0baa65d0 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 crypto rx=0x7fe4c802c210 tx=0x7fe4c8042000 comp rx=0 tx=0).stop 2024-03-31T23:28:19.629 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:19.624+0000 7fe4dd96c740 1 -- 172.21.15.46:0/1847005543 shutdown_connections 2024-03-31T23:28:19.629 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:19.624+0000 7fe4dd96c740 1 --2- 172.21.15.46:0/1847005543 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x561f0baa6db0 0x561f0baa2e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:28:19.629 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:19.624+0000 7fe4dd96c740 1 --2- 172.21.15.46:0/1847005543 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561f0ba9dfe0 0x561f0bab6b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:28:19.629 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:19.624+0000 7fe4dd96c740 1 --2- 172.21.15.46:0/1847005543 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561f0ba9d6c0 0x561f0baa65d0 unknown :-1 s=CLOSED pgs=923 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:28:19.629 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:19.624+0000 7fe4dd96c740 1 -- 172.21.15.46:0/1847005543 >> 172.21.15.46:0/1847005543 conn(0x561f0ba97fe0 msgr2=0x561f0baa2770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:28:19.629 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:19.624+0000 7fe4dd96c740 1 -- 172.21.15.46:0/1847005543 shutdown_connections 2024-03-31T23:28:19.630 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:19.624+0000 7fe4dd96c740 1 -- 172.21.15.46:0/1847005543 wait complete. 2024-03-31T23:28:19.630 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:19.625+0000 7fe4dd96c740 1 -- start start 2024-03-31T23:28:19.630 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:19.625+0000 7fe4dd96c740 1 Processor -- start 2024-03-31T23:28:19.634 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:19.625+0000 7fe4dd96c740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561f0ba9d6c0 0x561f0bab3150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:28:19.634 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:19.625+0000 7fe4dd96c740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561f0ba9dfe0 0x561f0baa0520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:28:19.634 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:19.625+0000 7fe4dd96c740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x561f0baa6db0 0x561f0baa0a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:28:19.635 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:19.625+0000 7fe4dd96c740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fe4d41ab1c0 con 0x561f0ba9dfe0 2024-03-31T23:28:19.635 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:19.625+0000 7fe4dd96c740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fe4d41ac630 con 0x561f0baa6db0 2024-03-31T23:28:19.635 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:19.625+0000 7fe4dd96c740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fe4d41ab890 con 0x561f0ba9d6c0 2024-03-31T23:28:19.635 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:19.625+0000 7fe4dd96c740 10 client.0 _ll_register_callbacks cb 0x7fe4c0009570 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T23:28:19.635 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:19.625+0000 7fe4d3fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561f0ba9dfe0 0x561f0baa0520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:28:19.635 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:19.625+0000 7fe4d3fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561f0ba9dfe0 0x561f0baa0520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:35306/0 (socket says 172.21.15.46:35306) 2024-03-31T23:28:19.635 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:19.625+0000 7fe4d3fff640 1 -- 172.21.15.46:0/4087341399 learned_addr learned my addr 172.21.15.46:0/4087341399 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:28:19.635 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:19.625+0000 7fe4d8a4a640 1 --2- 172.21.15.46:0/4087341399 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561f0ba9d6c0 0x561f0bab3150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:28:19.635 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:19.625+0000 7fe4d924b640 1 --2- 172.21.15.46:0/4087341399 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x561f0baa6db0 0x561f0baa0a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:28:19.635 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:19.626+0000 7fe4d924b640 1 -- 172.21.15.46:0/4087341399 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561f0ba9d6c0 msgr2=0x561f0bab3150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:28:19.635 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:19.626+0000 7fe4d924b640 1 --2- 172.21.15.46:0/4087341399 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561f0ba9d6c0 0x561f0bab3150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:28:19.636 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:19.626+0000 7fe4d924b640 1 -- 172.21.15.46:0/4087341399 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561f0ba9dfe0 msgr2=0x561f0baa0520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:28:19.636 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:19.626+0000 7fe4d924b640 1 --2- 172.21.15.46:0/4087341399 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561f0ba9dfe0 0x561f0baa0520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:28:19.636 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:19.626+0000 7fe4d924b640 1 -- 172.21.15.46:0/4087341399 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561f0baa0fa0 con 0x561f0baa6db0 2024-03-31T23:28:19.636 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:19.626+0000 7fe4d8a4a640 1 --2- 172.21.15.46:0/4087341399 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561f0ba9d6c0 0x561f0bab3150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:28:19.636 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:19.626+0000 7fe4d3fff640 1 --2- 172.21.15.46:0/4087341399 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561f0ba9dfe0 0x561f0baa0520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:28:19.636 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:19.627+0000 7fe4d924b640 1 --2- 172.21.15.46:0/4087341399 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x561f0baa6db0 0x561f0baa0a60 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fe4cc0998f0 tx=0x7fe4cc084f10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=181dab90a75f254e server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:28:19.636 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:19.627+0000 7fe4b3fff640 10 client.0 ms_handle_connect on v2:172.21.15.142:3300/0 2024-03-31T23:28:19.636 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:19.628+0000 7fe4b3fff640 1 -- 172.21.15.46:0/4087341399 <== mon.2 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4cc0a3bc0 con 0x561f0baa6db0 2024-03-31T23:28:19.637 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:19.628+0000 7fe4b3fff640 1 -- 172.21.15.46:0/4087341399 <== mon.2 v2:172.21.15.142:3300/0 2 ==== osd_map(119..119 src has 1..119) ==== 9554+0+0 (secure 0 0 0) 0x7fe4cc097390 con 0x561f0baa6db0 2024-03-31T23:28:19.637 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:19.628+0000 7fe4dd96c740 1 -- 172.21.15.46:0/4087341399 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe4c80542f0 con 0x561f0baa6db0 2024-03-31T23:28:19.637 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:19.628+0000 7fe4b3fff640 1 -- 172.21.15.46:0/4087341399 <== mon.2 v2:172.21.15.142:3300/0 3 ==== mon_get_version_reply(handle=1 version=44) ==== 24+0+0 (secure 0 0 0) 0x7fe4cc09b410 con 0x561f0baa6db0 2024-03-31T23:28:19.637 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:19.628+0000 7fe4dd96c740 10 client.27329 fetch_fsmap learned FSMap version 44 2024-03-31T23:28:19.637 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:19.628+0000 7fe4dd96c740 1 -- 172.21.15.46:0/4087341399 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,fsmap=44,monmap=4+}) -- 0x561f0bb20e30 con 0x561f0baa6db0 2024-03-31T23:28:19.637 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:19.628+0000 7fe4b3fff640 1 -- 172.21.15.46:0/4087341399 <== mon.2 v2:172.21.15.142:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe4c80542f0 con 0x561f0baa6db0 2024-03-31T23:28:19.637 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:19.628+0000 7fe4b3fff640 1 -- 172.21.15.46:0/4087341399 <== mon.2 v2:172.21.15.142:3300/0 5 ==== fsmap(e 44) ==== 8354+0+0 (secure 0 0 0) 0x7fe4cc082040 con 0x561f0baa6db0 2024-03-31T23:28:19.637 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:19.629+0000 7fe4dd96c740 10 client.27329 fetch_fsmap finished waiting for FSMap version 44 2024-03-31T23:28:19.637 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:19.629+0000 7fe4dd96c740 10 client.27329 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-31T23:28:19.637 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:19.629+0000 7fe4dd96c740 20 client.27329 populate_metadata read hostname 'smithi046' 2024-03-31T23:28:19.637 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:19.629+0000 7fe4dd96c740 1 --2- 172.21.15.46:0/4087341399 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x561f0bb21340 0x561f0bb23760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:28:19.638 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:19.629+0000 7fe4dd96c740 4 client.27329 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-31T23:28:19.638 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:19.629+0000 7fe4dd96c740 1 -- 172.21.15.46:0/4087341399 --> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] -- command(tid 0: {"prefix": "flush journal"}) -- 0x561f0ba0de30 con 0x561f0bb21340 2024-03-31T23:28:19.638 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:19.629+0000 7fe4d8a4a640 1 --2- 172.21.15.46:0/4087341399 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x561f0bb21340 0x561f0bb23760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:28:19.638 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:19.629+0000 7fe4b3fff640 10 client.27329 ms_handle_connect on v2:172.21.15.142:6838/2519143918 2024-03-31T23:28:19.638 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:19.629+0000 7fe4d8a4a640 1 --2- 172.21.15.46:0/4087341399 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x561f0bb21340 0x561f0bb23760 crc :-1 s=READY pgs=225 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=830950e8a51aaea9 server_cookie=96ae851e5c937c6c in_seq=0 out_seq=0 2024-03-31T23:28:19.823 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:19.821+0000 7fe4b3fff640 1 -- 172.21.15.46:0/4087341399 <== mds.0 v2:172.21.15.142:6838/2519143918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x561f0ba0de30 con 0x561f0bb21340 2024-03-31T23:28:19.823 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:19.821+0000 7fe4b3fff640 10 client.27329 handle_command_reply: tid=0 2024-03-31T23:28:19.823 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:19.821+0000 7fe4dd96c740 1 client.27329 shutdown 2024-03-31T23:28:19.824 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:19.821+0000 7fe4dd96c740 1 -- 172.21.15.46:0/4087341399 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x561f0baa6db0 msgr2=0x561f0baa0a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:28:19.824 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:19.821+0000 7fe4dd96c740 1 --2- 172.21.15.46:0/4087341399 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x561f0baa6db0 0x561f0baa0a60 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fe4cc0998f0 tx=0x7fe4cc084f10 comp rx=0 tx=0).stop 2024-03-31T23:28:19.824 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:19.821+0000 7fe4dd96c740 1 -- 172.21.15.46:0/4087341399 shutdown_connections 2024-03-31T23:28:19.824 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:19.821+0000 7fe4dd96c740 1 --2- 172.21.15.46:0/4087341399 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x561f0bb21340 0x561f0bb23760 crc :-1 s=READY pgs=225 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:28:19.824 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:19.821+0000 7fe4dd96c740 1 --2- 172.21.15.46:0/4087341399 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x561f0baa6db0 0x561f0baa0a60 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:28:19.824 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:19.821+0000 7fe4dd96c740 1 --2- 172.21.15.46:0/4087341399 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561f0ba9dfe0 0x561f0baa0520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:28:19.824 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:19.821+0000 7fe4dd96c740 1 --2- 172.21.15.46:0/4087341399 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561f0ba9d6c0 0x561f0bab3150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:28:19.824 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:19.821+0000 7fe4b3fff640 0 client.27329 ms_handle_reset on v2:172.21.15.142:6838/2519143918 2024-03-31T23:28:19.824 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:19.821+0000 7fe4dd96c740 1 -- 172.21.15.46:0/4087341399 >> 172.21.15.46:0/4087341399 conn(0x561f0ba97fe0 msgr2=0x561f0ba966f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:28:19.824 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:19.822+0000 7fe4dd96c740 1 -- 172.21.15.46:0/4087341399 shutdown_connections 2024-03-31T23:28:19.824 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:19.822+0000 7fe4dd96c740 1 -- 172.21.15.46:0/4087341399 wait complete. 2024-03-31T23:28:19.824 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:19.822+0000 7fe4dd96c740 20 client.27329 trim_cache size 0 max 16384 2024-03-31T23:28:19.825 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout:{ 2024-03-31T23:28:19.825 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout: "message": "", 2024-03-31T23:28:19.825 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout: "return_code": 0 2024-03-31T23:28:19.825 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout:} 2024-03-31T23:28:19.826 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:19.823+0000 7fe4dacae640 1 -- 172.21.15.46:0/2446905229 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fe4c40a2730 msgr2=0x7fe4c40a4bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:28:19.826 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:19.823+0000 7fe4dacae640 1 --2- 172.21.15.46:0/2446905229 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fe4c40a2730 0x7fe4c40a4bf0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fe4c8006010 tx=0x7fe4c8005e40 comp rx=0 tx=0).stop 2024-03-31T23:28:19.826 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:19.823+0000 7fe4dacae640 1 -- 172.21.15.46:0/2446905229 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe4d410bd40 msgr2=0x7fe4d419c740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:28:19.826 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:19.823+0000 7fe4dacae640 1 --2- 172.21.15.46:0/2446905229 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe4d410bd40 0x7fe4d419c740 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7fe4cc00ea60 tx=0x7fe4cc00ef20 comp rx=0 tx=0).stop 2024-03-31T23:28:19.826 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:19.823+0000 7fe4dacae640 1 -- 172.21.15.46:0/2446905229 shutdown_connections 2024-03-31T23:28:19.826 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:19.823+0000 7fe4dacae640 1 --2- 172.21.15.46:0/2446905229 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fe4c40a2730 0x7fe4c40a4bf0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:28:19.826 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:19.823+0000 7fe4dacae640 1 --2- 172.21.15.46:0/2446905229 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe4d410bd40 0x7fe4d419c740 unknown :-1 s=CLOSED pgs=920 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:28:19.826 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:19.823+0000 7fe4dacae640 1 --2- 172.21.15.46:0/2446905229 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe4d4103290 0x7fe4d4198410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:28:19.826 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:19.823+0000 7fe4dacae640 1 --2- 172.21.15.46:0/2446905229 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe4d4102970 0x7fe4d4197ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:28:19.826 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:19.823+0000 7fe4dacae640 1 -- 172.21.15.46:0/2446905229 >> 172.21.15.46:0/2446905229 conn(0x7fe4d4077860 msgr2=0x7fe4d40fe7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:28:19.827 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:19.823+0000 7fe4dacae640 1 -- 172.21.15.46:0/2446905229 shutdown_connections 2024-03-31T23:28:19.827 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:19.823+0000 7fe4dacae640 1 -- 172.21.15.46:0/2446905229 wait complete. 2024-03-31T23:28:20.281 INFO:tasks.workunit.client.0.smithi046.stdout: 10 33480 8.73 MB/sec execute 85 sec latency 137.727 ms 2024-03-31T23:28:20.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:28:20 smithi142 ceph-mon[35271]: pgmap v524: 129 pgs: 129 active+clean; 657 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 5.2 MiB/s wr, 282 op/s 2024-03-31T23:28:21.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:28:20 smithi162 ceph-mon[36237]: pgmap v524: 129 pgs: 129 active+clean; 657 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 5.2 MiB/s wr, 282 op/s 2024-03-31T23:28:21.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:28:20 smithi046 ceph-mon[32498]: pgmap v524: 129 pgs: 129 active+clean; 657 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 5.2 MiB/s wr, 282 op/s 2024-03-31T23:28:21.281 INFO:tasks.workunit.client.0.smithi046.stdout: 10 33625 8.74 MB/sec execute 86 sec latency 83.381 ms 2024-03-31T23:28:22.281 INFO:tasks.workunit.client.0.smithi046.stdout: 10 33815 8.77 MB/sec execute 87 sec latency 70.664 ms 2024-03-31T23:28:22.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:28:22 smithi142 ceph-mon[35271]: pgmap v525: 129 pgs: 129 active+clean; 657 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 5.1 MiB/s wr, 273 op/s 2024-03-31T23:28:23.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:28:22 smithi162 ceph-mon[36237]: pgmap v525: 129 pgs: 129 active+clean; 657 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 5.1 MiB/s wr, 273 op/s 2024-03-31T23:28:23.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:28:22 smithi046 ceph-mon[32498]: pgmap v525: 129 pgs: 129 active+clean; 657 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 5.1 MiB/s wr, 273 op/s 2024-03-31T23:28:23.281 INFO:tasks.workunit.client.0.smithi046.stdout: 10 33964 8.75 MB/sec execute 88 sec latency 73.698 ms 2024-03-31T23:28:24.281 INFO:tasks.workunit.client.0.smithi046.stdout: 10 34123 8.77 MB/sec execute 89 sec latency 89.678 ms 2024-03-31T23:28:24.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:28:24 smithi142 ceph-mon[35271]: pgmap v526: 129 pgs: 129 active+clean; 657 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 4.4 MiB/s wr, 235 op/s 2024-03-31T23:28:25.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:28:24 smithi162 ceph-mon[36237]: pgmap v526: 129 pgs: 129 active+clean; 657 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 4.4 MiB/s wr, 235 op/s 2024-03-31T23:28:25.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:28:24 smithi046 ceph-mon[32498]: pgmap v526: 129 pgs: 129 active+clean; 657 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 4.4 MiB/s wr, 235 op/s 2024-03-31T23:28:25.281 INFO:tasks.workunit.client.0.smithi046.stdout: 10 34282 8.79 MB/sec execute 90 sec latency 60.370 ms 2024-03-31T23:28:26.281 INFO:tasks.workunit.client.0.smithi046.stdout: 10 34461 8.82 MB/sec execute 91 sec latency 65.322 ms 2024-03-31T23:28:26.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:28:26 smithi142 ceph-mon[35271]: pgmap v527: 129 pgs: 129 active+clean; 658 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 6.7 MiB/s wr, 423 op/s 2024-03-31T23:28:27.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:28:26 smithi162 ceph-mon[36237]: pgmap v527: 129 pgs: 129 active+clean; 658 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 6.7 MiB/s wr, 423 op/s 2024-03-31T23:28:27.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:28:26 smithi046 ceph-mon[32498]: pgmap v527: 129 pgs: 129 active+clean; 658 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 6.7 MiB/s wr, 423 op/s 2024-03-31T23:28:27.281 INFO:tasks.workunit.client.0.smithi046.stdout: 10 34588 8.77 MB/sec execute 92 sec latency 102.471 ms 2024-03-31T23:28:28.281 INFO:tasks.workunit.client.0.smithi046.stdout: 10 34715 8.69 MB/sec execute 93 sec latency 117.790 ms 2024-03-31T23:28:28.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:28:28 smithi142 ceph-mon[35271]: pgmap v528: 129 pgs: 129 active+clean; 658 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 4.9 MiB/s wr, 302 op/s 2024-03-31T23:28:29.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:28:28 smithi162 ceph-mon[36237]: pgmap v528: 129 pgs: 129 active+clean; 658 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 4.9 MiB/s wr, 302 op/s 2024-03-31T23:28:29.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:28:28 smithi046 ceph-mon[32498]: pgmap v528: 129 pgs: 129 active+clean; 658 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 4.9 MiB/s wr, 302 op/s 2024-03-31T23:28:29.281 INFO:tasks.workunit.client.0.smithi046.stdout: 10 34885 8.70 MB/sec execute 94 sec latency 100.586 ms 2024-03-31T23:28:30.281 INFO:tasks.workunit.client.0.smithi046.stdout: 10 35052 8.67 MB/sec execute 95 sec latency 68.302 ms 2024-03-31T23:28:30.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:28:30 smithi142 ceph-mon[35271]: pgmap v529: 129 pgs: 129 active+clean; 664 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 6.7 MiB/s wr, 415 op/s 2024-03-31T23:28:31.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:28:30 smithi162 ceph-mon[36237]: pgmap v529: 129 pgs: 129 active+clean; 664 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 6.7 MiB/s wr, 415 op/s 2024-03-31T23:28:31.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:28:30 smithi046 ceph-mon[32498]: pgmap v529: 129 pgs: 129 active+clean; 664 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 6.7 MiB/s wr, 415 op/s 2024-03-31T23:28:31.282 INFO:tasks.workunit.client.0.smithi046.stdout: 10 35206 8.67 MB/sec execute 96 sec latency 64.348 ms 2024-03-31T23:28:32.283 INFO:tasks.workunit.client.0.smithi046.stdout: 10 35357 8.67 MB/sec execute 97 sec latency 79.979 ms 2024-03-31T23:28:32.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:28:32 smithi142 ceph-mon[35271]: pgmap v530: 129 pgs: 129 active+clean; 664 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 4.2 MiB/s wr, 300 op/s 2024-03-31T23:28:33.002 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:33.002+0000 7f86e9237640 1 -- 172.21.15.46:0/2466670108 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86dc033710 con 0x7f86e4069770 2024-03-31T23:28:33.003 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:33.003+0000 7f86ec49d640 1 -- 172.21.15.46:0/2466670108 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f86e4069770 msgr2=0x7f86e410e5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:28:33.003 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:33.003+0000 7f86ec49d640 1 --2- 172.21.15.46:0/2466670108 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f86e4069770 0x7f86e410e5e0 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 crypto rx=0x7f86dc009be0 tx=0x7f86dc02bef0 comp rx=0 tx=0).stop 2024-03-31T23:28:33.003 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:33.003+0000 7f86ec49d640 1 -- 172.21.15.46:0/2466670108 shutdown_connections 2024-03-31T23:28:33.003 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:33.003+0000 7f86ec49d640 1 --2- 172.21.15.46:0/2466670108 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f86e4069770 0x7f86e410e5e0 unknown :-1 s=CLOSED pgs=924 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:28:33.004 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:33.003+0000 7f86ec49d640 1 --2- 172.21.15.46:0/2466670108 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f86e4068dd0 0x7f86e4069230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:28:33.004 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:33.003+0000 7f86ec49d640 1 --2- 172.21.15.46:0/2466670108 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f86e41077b0 0x7f86e4107b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:28:33.004 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:33.003+0000 7f86ec49d640 1 -- 172.21.15.46:0/2466670108 >> 172.21.15.46:0/2466670108 conn(0x7f86e40774a0 msgr2=0x7f86e40778b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:28:33.004 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:33.003+0000 7f86ec49d640 1 -- 172.21.15.46:0/2466670108 shutdown_connections 2024-03-31T23:28:33.004 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:33.003+0000 7f86ec49d640 1 -- 172.21.15.46:0/2466670108 wait complete. 2024-03-31T23:28:33.007 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:33.004+0000 7f86ec49d640 1 Processor -- start 2024-03-31T23:28:33.007 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:33.004+0000 7f86ec49d640 1 -- start start 2024-03-31T23:28:33.007 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:33.004+0000 7f86ec49d640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f86e4068dd0 0x7f86e419a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:28:33.007 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:33.004+0000 7f86ec49d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f86e4069770 0x7f86e419a820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:28:33.007 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:33.004+0000 7f86ec49d640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f86e41077b0 0x7f86e419e680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:28:33.007 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:33.004+0000 7f86ec49d640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f86e4101280 con 0x7f86e4069770 2024-03-31T23:28:33.007 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:33.004+0000 7f86ec49d640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f86e4101100 con 0x7f86e4068dd0 2024-03-31T23:28:33.008 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:33.004+0000 7f86ec49d640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f86e4101400 con 0x7f86e41077b0 2024-03-31T23:28:33.008 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:33.004+0000 7f86ea239640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f86e4068dd0 0x7f86e419a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:28:33.008 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:33.004+0000 7f86eaa3a640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f86e41077b0 0x7f86e419e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:28:33.008 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:33.004+0000 7f86eaa3a640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f86e41077b0 0x7f86e419e680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.46:44278/0 (socket says 172.21.15.46:44278) 2024-03-31T23:28:33.008 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:33.004+0000 7f86ea239640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f86e4068dd0 0x7f86e419a2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.46:50160/0 (socket says 172.21.15.46:50160) 2024-03-31T23:28:33.008 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:33.004+0000 7f86eaa3a640 1 -- 172.21.15.46:0/997540115 learned_addr learned my addr 172.21.15.46:0/997540115 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:28:33.008 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:33.004+0000 7f86e9a38640 1 --2- 172.21.15.46:0/997540115 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f86e4069770 0x7f86e419a820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:28:33.008 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:33.004+0000 7f86eaa3a640 1 -- 172.21.15.46:0/997540115 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f86e4068dd0 msgr2=0x7f86e419a2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:28:33.008 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:33.004+0000 7f86eaa3a640 1 --2- 172.21.15.46:0/997540115 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f86e4068dd0 0x7f86e419a2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:28:33.008 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:33.004+0000 7f86eaa3a640 1 -- 172.21.15.46:0/997540115 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f86e4069770 msgr2=0x7f86e419a820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:28:33.008 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:33.004+0000 7f86eaa3a640 1 --2- 172.21.15.46:0/997540115 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f86e4069770 0x7f86e419a820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:28:33.008 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:33.004+0000 7f86eaa3a640 1 -- 172.21.15.46:0/997540115 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f86e419edb0 con 0x7f86e41077b0 2024-03-31T23:28:33.009 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:33.005+0000 7f86eaa3a640 1 --2- 172.21.15.46:0/997540115 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f86e41077b0 0x7f86e419e680 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f86dc036040 tx=0x7f86dc033d10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:28:33.009 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:33.005+0000 7f86db7fe640 1 -- 172.21.15.46:0/997540115 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86dc043020 con 0x7f86e41077b0 2024-03-31T23:28:33.009 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:33.005+0000 7f86db7fe640 1 -- 172.21.15.46:0/997540115 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f86dc003b80 con 0x7f86e41077b0 2024-03-31T23:28:33.009 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:33.005+0000 7f86db7fe640 1 -- 172.21.15.46:0/997540115 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86dc034690 con 0x7f86e41077b0 2024-03-31T23:28:33.009 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:33.005+0000 7f86ec49d640 1 -- 172.21.15.46:0/997540115 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f86e419f040 con 0x7f86e41077b0 2024-03-31T23:28:33.009 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:33.005+0000 7f86ec49d640 1 -- 172.21.15.46:0/997540115 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f86e41a6930 con 0x7f86e41077b0 2024-03-31T23:28:33.012 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:33.008+0000 7f86db7fe640 1 -- 172.21.15.46:0/997540115 <== mon.1 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 25) ==== 137451+0+0 (secure 0 0 0) 0x7f86dc046020 con 0x7f86e41077b0 2024-03-31T23:28:33.012 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:33.009+0000 7f86db7fe640 1 --2- 172.21.15.46:0/997540115 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f86d00a2730 0x7f86d00a4bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:28:33.012 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:33.009+0000 7f86ef084740 1 Processor -- start 2024-03-31T23:28:33.012 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:33.010+0000 7f86ea239640 1 --2- 172.21.15.46:0/997540115 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f86d00a2730 0x7f86d00a4bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:28:33.012 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:33.010+0000 7f86ef084740 1 -- start start 2024-03-31T23:28:33.012 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:33.010+0000 7f86db7fe640 1 -- 172.21.15.46:0/997540115 <== mon.1 v2:172.21.15.162:3300/0 5 ==== osd_map(119..119 src has 1..119) ==== 9554+0+0 (secure 0 0 0) 0x7f86dc0f1520 con 0x7f86e41077b0 2024-03-31T23:28:33.012 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:33.010+0000 7f86ef084740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5629c4f826c0 0x5629c4f82aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:28:33.013 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:33.010+0000 7f86ef084740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5629c4f82fe0 0x5629c4f94080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:28:33.013 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:33.010+0000 7f86ef084740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5629c4f945c0 0x5629c4f969b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:28:33.013 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:33.010+0000 7f86ef084740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x5629c4ee58f0 con 0x5629c4f826c0 2024-03-31T23:28:33.013 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:33.010+0000 7f86ef084740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x5629c4e8e8d0 con 0x5629c4f945c0 2024-03-31T23:28:33.013 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:33.010+0000 7f86ef084740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5629c4ee1990 con 0x5629c4f82fe0 2024-03-31T23:28:33.013 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:33.010+0000 7f86ea239640 1 --2- 172.21.15.46:0/997540115 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f86d00a2730 0x7f86d00a4bf0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f86e0009530 tx=0x7f86e0009260 comp rx=0 tx=0).ready entity=mgr.15654 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:28:33.013 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:33.010+0000 7f86eaa3a640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5629c4f945c0 0x5629c4f969b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:28:33.013 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:33.010+0000 7f86ea239640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5629c4f826c0 0x5629c4f82aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:28:33.013 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:33.011+0000 7f86ea239640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5629c4f826c0 0x5629c4f82aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37022/0 (socket says 172.21.15.46:37022) 2024-03-31T23:28:33.013 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:33.011+0000 7f86ea239640 1 -- 172.21.15.46:0/525708001 learned_addr learned my addr 172.21.15.46:0/525708001 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:28:33.013 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:33.010+0000 7f86eaa3a640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5629c4f945c0 0x5629c4f969b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.46:50172/0 (socket says 172.21.15.46:50172) 2024-03-31T23:28:33.013 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:33.011+0000 7f86ea239640 1 -- 172.21.15.46:0/525708001 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5629c4f82fe0 msgr2=0x5629c4f94080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:28:33.014 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:33.011+0000 7f86ea239640 1 --2- 172.21.15.46:0/525708001 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5629c4f82fe0 0x5629c4f94080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:28:33.014 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:33.011+0000 7f86ea239640 1 -- 172.21.15.46:0/525708001 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5629c4f945c0 msgr2=0x5629c4f969b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:28:33.014 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:33.011+0000 7f86ea239640 1 --2- 172.21.15.46:0/525708001 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5629c4f945c0 0x5629c4f969b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:28:33.014 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:33.011+0000 7f86ea239640 1 -- 172.21.15.46:0/525708001 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5629c4bfa210 con 0x5629c4f826c0 2024-03-31T23:28:33.014 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:33.011+0000 7f86ea239640 1 --2- 172.21.15.46:0/525708001 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5629c4f826c0 0x5629c4f82aa0 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 crypto rx=0x7f86e0009f20 tx=0x7f86e0014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae92c7ff92ca9d8 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:28:33.014 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:33.012+0000 7f86d8bfe640 1 -- 172.21.15.46:0/525708001 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86e0005220 con 0x5629c4f826c0 2024-03-31T23:28:33.014 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:33.012+0000 7f86d8bfe640 1 -- 172.21.15.46:0/525708001 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f86e0004c50 con 0x5629c4f826c0 2024-03-31T23:28:33.014 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:33.012+0000 7f86d8bfe640 1 -- 172.21.15.46:0/525708001 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86e0026820 con 0x5629c4f826c0 2024-03-31T23:28:33.019 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:33.013+0000 7f86ef084740 1 -- 172.21.15.46:0/525708001 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5629c4f826c0 msgr2=0x5629c4f82aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:28:33.019 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:33.013+0000 7f86ef084740 1 --2- 172.21.15.46:0/525708001 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5629c4f826c0 0x5629c4f82aa0 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 crypto rx=0x7f86e0009f20 tx=0x7f86e0014000 comp rx=0 tx=0).stop 2024-03-31T23:28:33.019 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:33.013+0000 7f86ef084740 1 -- 172.21.15.46:0/525708001 shutdown_connections 2024-03-31T23:28:33.019 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:33.013+0000 7f86ef084740 1 --2- 172.21.15.46:0/525708001 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5629c4f945c0 0x5629c4f969b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:28:33.019 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:33.013+0000 7f86ef084740 1 --2- 172.21.15.46:0/525708001 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5629c4f82fe0 0x5629c4f94080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:28:33.019 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:33.013+0000 7f86ef084740 1 --2- 172.21.15.46:0/525708001 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5629c4f826c0 0x5629c4f82aa0 unknown :-1 s=CLOSED pgs=925 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:28:33.019 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:33.013+0000 7f86ef084740 1 -- 172.21.15.46:0/525708001 >> 172.21.15.46:0/525708001 conn(0x5629c4f7cfe0 msgr2=0x5629c4f7d3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:28:33.019 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:33.013+0000 7f86ef084740 1 -- 172.21.15.46:0/525708001 shutdown_connections 2024-03-31T23:28:33.020 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:33.013+0000 7f86ef084740 1 -- 172.21.15.46:0/525708001 wait complete. 2024-03-31T23:28:33.020 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:33.014+0000 7f86ef084740 1 -- start start 2024-03-31T23:28:33.020 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:33.014+0000 7f86ef084740 1 Processor -- start 2024-03-31T23:28:33.020 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:33.014+0000 7f86ef084740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5629c4f826c0 0x5629c4f91660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:28:33.020 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:33.014+0000 7f86ef084740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5629c4f82fe0 0x5629c4f91ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:28:33.020 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:33.014+0000 7f86ef084740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5629c4f945c0 0x5629c4f920e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:28:33.020 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:33.014+0000 7f86ef084740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x5629c4f971f0 con 0x5629c4f826c0 2024-03-31T23:28:33.020 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:33.014+0000 7f86ef084740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x5629c4e83f80 con 0x5629c4f945c0 2024-03-31T23:28:33.020 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:33.014+0000 7f86ef084740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5629c4f97370 con 0x5629c4f82fe0 2024-03-31T23:28:33.020 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:33.014+0000 7f86ef084740 10 client.0 _ll_register_callbacks cb 0x5629c4edc290 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T23:28:33.020 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:33.014+0000 7f86ea239640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5629c4f826c0 0x5629c4f91660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:28:33.020 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:33.014+0000 7f86eaa3a640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5629c4f945c0 0x5629c4f920e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:28:33.020 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:33.014+0000 7f86ea239640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5629c4f826c0 0x5629c4f91660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37028/0 (socket says 172.21.15.46:37028) 2024-03-31T23:28:33.020 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:33.014+0000 7f86ea239640 1 -- 172.21.15.46:0/4075145161 learned_addr learned my addr 172.21.15.46:0/4075145161 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:28:33.021 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:33.014+0000 7f86e9a38640 1 --2- 172.21.15.46:0/4075145161 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5629c4f82fe0 0x5629c4f91ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:28:33.021 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:33.015+0000 7f86ea239640 1 -- 172.21.15.46:0/4075145161 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5629c4f82fe0 msgr2=0x5629c4f91ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:28:33.021 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:33.015+0000 7f86ea239640 1 --2- 172.21.15.46:0/4075145161 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5629c4f82fe0 0x5629c4f91ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:28:33.021 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:33.015+0000 7f86ea239640 1 -- 172.21.15.46:0/4075145161 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5629c4f945c0 msgr2=0x5629c4f920e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:28:33.021 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:33.015+0000 7f86ea239640 1 --2- 172.21.15.46:0/4075145161 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5629c4f945c0 0x5629c4f920e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:28:33.021 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:33.015+0000 7f86ea239640 1 -- 172.21.15.46:0/4075145161 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5629c4cc6620 con 0x5629c4f826c0 2024-03-31T23:28:33.021 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:33.015+0000 7f86eaa3a640 1 --2- 172.21.15.46:0/4075145161 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5629c4f945c0 0x5629c4f920e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:28:33.021 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:33.015+0000 7f86ea239640 1 --2- 172.21.15.46:0/4075145161 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5629c4f826c0 0x5629c4f91660 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 crypto rx=0x7f86e0028040 tx=0x7f86e0005c50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b176c8af150a52cf server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:28:33.021 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:33.015+0000 7f86cdffb640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-31T23:28:33.021 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:33.016+0000 7f86cdffb640 1 -- 172.21.15.46:0/4075145161 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86e0035020 con 0x5629c4f826c0 2024-03-31T23:28:33.021 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:33.016+0000 7f86cdffb640 1 -- 172.21.15.46:0/4075145161 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(119..119 src has 1..119) ==== 9554+0+0 (secure 0 0 0) 0x7f86e0026d20 con 0x5629c4f826c0 2024-03-31T23:28:33.021 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:33.016+0000 7f86ef084740 1 -- 172.21.15.46:0/4075145161 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f86e0004c50 con 0x5629c4f826c0 2024-03-31T23:28:33.021 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:33.016+0000 7f86cdffb640 1 -- 172.21.15.46:0/4075145161 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=44) ==== 24+0+0 (secure 0 0 0) 0x7f86e0040050 con 0x5629c4f826c0 2024-03-31T23:28:33.021 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:33.016+0000 7f86ef084740 10 client.17511 fetch_fsmap learned FSMap version 44 2024-03-31T23:28:33.022 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:33.016+0000 7f86ef084740 1 -- 172.21.15.46:0/4075145161 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=44,monmap=4+}) -- 0x5629c4fa3120 con 0x5629c4f826c0 2024-03-31T23:28:33.022 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:33.016+0000 7f86cdffb640 1 -- 172.21.15.46:0/4075145161 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f86e0004c50 con 0x5629c4f826c0 2024-03-31T23:28:33.022 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:33.017+0000 7f86cdffb640 1 -- 172.21.15.46:0/4075145161 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 44) ==== 8354+0+0 (secure 0 0 0) 0x7f86e0031470 con 0x5629c4f826c0 2024-03-31T23:28:33.022 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:33.017+0000 7f86ef084740 10 client.17511 fetch_fsmap finished waiting for FSMap version 44 2024-03-31T23:28:33.022 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:33.017+0000 7f86ef084740 10 client.17511 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-31T23:28:33.022 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:33.017+0000 7f86ef084740 20 client.17511 populate_metadata read hostname 'smithi046' 2024-03-31T23:28:33.022 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:33.017+0000 7f86ef084740 1 --2- 172.21.15.46:0/4075145161 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x5629c4fa4470 0x5629c4fa6860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:28:33.022 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:33.017+0000 7f86ef084740 4 client.17511 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-31T23:28:33.022 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:33.017+0000 7f86ef084740 1 -- 172.21.15.46:0/4075145161 --> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5629c4ef9120 con 0x5629c4fa4470 2024-03-31T23:28:33.022 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:33.017+0000 7f86e9a38640 1 --2- 172.21.15.46:0/4075145161 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x5629c4fa4470 0x5629c4fa6860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:28:33.022 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:33.018+0000 7f86e9a38640 1 --2- 172.21.15.46:0/4075145161 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x5629c4fa4470 0x5629c4fa6860 crc :-1 s=READY pgs=226 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bca4ecd76ce18b7c server_cookie=988acea39fe00f6c in_seq=0 out_seq=0 2024-03-31T23:28:33.022 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:33.018+0000 7f86cdffb640 10 client.17511 ms_handle_connect on v2:172.21.15.142:6838/2519143918 2024-03-31T23:28:33.022 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:33.019+0000 7f86cdffb640 1 -- 172.21.15.46:0/4075145161 <== mds.0 v2:172.21.15.142:6838/2519143918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5629c4ef9120 con 0x5629c4fa4470 2024-03-31T23:28:33.023 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:33.019+0000 7f86cdffb640 10 client.17511 handle_command_reply: tid=0 2024-03-31T23:28:33.023 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:33.019+0000 7f86ef084740 1 client.17511 shutdown 2024-03-31T23:28:33.024 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:33.019+0000 7f86ef084740 1 -- 172.21.15.46:0/4075145161 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5629c4f826c0 msgr2=0x5629c4f91660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:28:33.024 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:33.019+0000 7f86ef084740 1 --2- 172.21.15.46:0/4075145161 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5629c4f826c0 0x5629c4f91660 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 crypto rx=0x7f86e0028040 tx=0x7f86e0005c50 comp rx=0 tx=0).stop 2024-03-31T23:28:33.024 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:33.020+0000 7f86ef084740 1 -- 172.21.15.46:0/4075145161 shutdown_connections 2024-03-31T23:28:33.024 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:33.020+0000 7f86ef084740 1 --2- 172.21.15.46:0/4075145161 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x5629c4fa4470 0x5629c4fa6860 crc :-1 s=READY pgs=226 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:28:33.024 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:33.020+0000 7f86cdffb640 0 client.17511 ms_handle_reset on v2:172.21.15.142:6838/2519143918 2024-03-31T23:28:33.024 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:33.020+0000 7f86ef084740 1 --2- 172.21.15.46:0/4075145161 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5629c4f945c0 0x5629c4f920e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:28:33.024 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:33.020+0000 7f86ef084740 1 --2- 172.21.15.46:0/4075145161 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5629c4f82fe0 0x5629c4f91ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:28:33.024 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:33.020+0000 7f86ef084740 1 --2- 172.21.15.46:0/4075145161 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5629c4f826c0 0x5629c4f91660 unknown :-1 s=CLOSED pgs=926 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:28:33.024 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:33.020+0000 7f86ef084740 1 -- 172.21.15.46:0/4075145161 >> 172.21.15.46:0/4075145161 conn(0x5629c4f7cfe0 msgr2=0x5629c4f7ed30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:28:33.024 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:33.020+0000 7f86ef084740 1 -- 172.21.15.46:0/4075145161 shutdown_connections 2024-03-31T23:28:33.024 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:33.020+0000 7f86ef084740 1 -- 172.21.15.46:0/4075145161 wait complete. 2024-03-31T23:28:33.025 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:33.020+0000 7f86ef084740 20 client.17511 trim_cache size 0 max 16384 2024-03-31T23:28:33.046 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:33.044+0000 7f86ef084740 1 Processor -- start 2024-03-31T23:28:33.046 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:33.044+0000 7f86ef084740 1 -- start start 2024-03-31T23:28:33.046 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:33.045+0000 7f86ef084740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5629c4f826c0 0x5629c4f93d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:28:33.046 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:33.045+0000 7f86ef084740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5629c4f82fe0 0x5629c4f9ff90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:28:33.046 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:33.045+0000 7f86ef084740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5629c4f945c0 0x5629c4f7d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:28:33.046 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:33.045+0000 7f86ef084740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f86e41af0c0 con 0x5629c4f945c0 2024-03-31T23:28:33.046 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:33.045+0000 7f86ef084740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f86e41ac930 con 0x5629c4f826c0 2024-03-31T23:28:33.046 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:33.045+0000 7f86ef084740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f86e41aa7e0 con 0x5629c4f82fe0 2024-03-31T23:28:33.046 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:33.045+0000 7f86ea239640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5629c4f826c0 0x5629c4f93d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:28:33.047 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:33.045+0000 7f86e9a38640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5629c4f82fe0 0x5629c4f9ff90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:28:33.047 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:33.045+0000 7f86eaa3a640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5629c4f945c0 0x5629c4f7d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:28:33.047 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:33.045+0000 7f86ea239640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5629c4f826c0 0x5629c4f93d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.46:50194/0 (socket says 172.21.15.46:50194) 2024-03-31T23:28:33.047 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:33.045+0000 7f86ea239640 1 -- 172.21.15.46:0/171150806 learned_addr learned my addr 172.21.15.46:0/171150806 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:28:33.047 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:33.045+0000 7f86e9a38640 1 -- 172.21.15.46:0/171150806 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5629c4f826c0 msgr2=0x5629c4f93d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:28:33.047 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:33.045+0000 7f86e9a38640 1 --2- 172.21.15.46:0/171150806 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5629c4f826c0 0x5629c4f93d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:28:33.047 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:33.045+0000 7f86e9a38640 1 -- 172.21.15.46:0/171150806 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5629c4f945c0 msgr2=0x5629c4f7d3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:28:33.047 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:33.045+0000 7f86e9a38640 1 --2- 172.21.15.46:0/171150806 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5629c4f945c0 0x5629c4f7d3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:28:33.047 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:33.045+0000 7f86e9a38640 1 -- 172.21.15.46:0/171150806 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5629c4f7e020 con 0x5629c4f82fe0 2024-03-31T23:28:33.047 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:33.045+0000 7f86eaa3a640 1 --2- 172.21.15.46:0/171150806 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5629c4f945c0 0x5629c4f7d3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:28:33.047 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:33.045+0000 7f86ea239640 1 --2- 172.21.15.46:0/171150806 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5629c4f826c0 0x5629c4f93d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:28:33.047 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:33.046+0000 7f86e9a38640 1 --2- 172.21.15.46:0/171150806 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5629c4f82fe0 0x5629c4f9ff90 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x5629c4fa6f90 tx=0x7f86d40044d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7d9439afa65f85ef server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:28:33.048 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:33.046+0000 7f86cffff640 1 -- 172.21.15.46:0/171150806 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86d4003e20 con 0x5629c4f82fe0 2024-03-31T23:28:33.048 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:33.046+0000 7f86cffff640 1 -- 172.21.15.46:0/171150806 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f86d4022070 con 0x5629c4f82fe0 2024-03-31T23:28:33.048 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:33.046+0000 7f86cffff640 1 -- 172.21.15.46:0/171150806 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86d4010990 con 0x5629c4f82fe0 2024-03-31T23:28:33.048 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:28:32 smithi162 ceph-mon[36237]: pgmap v530: 129 pgs: 129 active+clean; 664 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 4.2 MiB/s wr, 300 op/s 2024-03-31T23:28:33.052 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:33.047+0000 7f86ef084740 1 -- 172.21.15.46:0/171150806 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5629c4f82fe0 msgr2=0x5629c4f9ff90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:28:33.052 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:33.047+0000 7f86ef084740 1 --2- 172.21.15.46:0/171150806 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5629c4f82fe0 0x5629c4f9ff90 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x5629c4fa6f90 tx=0x7f86d40044d0 comp rx=0 tx=0).stop 2024-03-31T23:28:33.052 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:33.047+0000 7f86ef084740 1 -- 172.21.15.46:0/171150806 shutdown_connections 2024-03-31T23:28:33.052 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:33.047+0000 7f86ef084740 1 --2- 172.21.15.46:0/171150806 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5629c4f945c0 0x5629c4f7d3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:28:33.052 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:33.047+0000 7f86ef084740 1 --2- 172.21.15.46:0/171150806 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5629c4f82fe0 0x5629c4f9ff90 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:28:33.052 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:33.047+0000 7f86ef084740 1 --2- 172.21.15.46:0/171150806 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5629c4f826c0 0x5629c4f93d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:28:33.052 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:33.047+0000 7f86ef084740 1 -- 172.21.15.46:0/171150806 >> 172.21.15.46:0/171150806 conn(0x5629c4f7cfe0 msgr2=0x5629c4f77bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:28:33.052 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:33.047+0000 7f86ef084740 1 -- 172.21.15.46:0/171150806 shutdown_connections 2024-03-31T23:28:33.052 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:33.047+0000 7f86ef084740 1 -- 172.21.15.46:0/171150806 wait complete. 2024-03-31T23:28:33.053 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:33.047+0000 7f86ef084740 1 -- start start 2024-03-31T23:28:33.053 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:33.047+0000 7f86ef084740 1 Processor -- start 2024-03-31T23:28:33.053 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:33.048+0000 7f86ef084740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5629c4f826c0 0x5629c4f9c5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:28:33.053 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:33.048+0000 7f86ef084740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5629c4f82fe0 0x5629c4f8dce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:28:33.053 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:33.048+0000 7f86ef084740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5629c4f945c0 0x5629c4f8e220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:28:33.053 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:33.048+0000 7f86ef084740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f86e41aae30 con 0x5629c4f945c0 2024-03-31T23:28:33.053 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:33.048+0000 7f86ef084740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f86e41ac260 con 0x5629c4f82fe0 2024-03-31T23:28:33.053 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:33.048+0000 7f86ef084740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f86e41ab4c0 con 0x5629c4f826c0 2024-03-31T23:28:33.053 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:33.048+0000 7f86ef084740 10 client.0 _ll_register_callbacks cb 0x7f86dc0048c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T23:28:33.053 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:33.048+0000 7f86ea239640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5629c4f826c0 0x5629c4f9c5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:28:33.053 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:33.048+0000 7f86eaa3a640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5629c4f945c0 0x5629c4f8e220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:28:33.053 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:33.048+0000 7f86e9a38640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5629c4f82fe0 0x5629c4f8dce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:28:33.053 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:33.048+0000 7f86ea239640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5629c4f826c0 0x5629c4f9c5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.46:44320/0 (socket says 172.21.15.46:44320) 2024-03-31T23:28:33.054 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:33.048+0000 7f86ea239640 1 -- 172.21.15.46:0/2337966402 learned_addr learned my addr 172.21.15.46:0/2337966402 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:28:33.054 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:33.048+0000 7f86eaa3a640 1 -- 172.21.15.46:0/2337966402 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5629c4f826c0 msgr2=0x5629c4f9c5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:28:33.054 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:33.048+0000 7f86eaa3a640 1 --2- 172.21.15.46:0/2337966402 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5629c4f826c0 0x5629c4f9c5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:28:33.054 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:33.048+0000 7f86eaa3a640 1 -- 172.21.15.46:0/2337966402 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5629c4f82fe0 msgr2=0x5629c4f8dce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:28:33.054 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:33.048+0000 7f86eaa3a640 1 --2- 172.21.15.46:0/2337966402 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5629c4f82fe0 0x5629c4f8dce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:28:33.054 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:33.048+0000 7f86eaa3a640 1 -- 172.21.15.46:0/2337966402 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5629c4f8e760 con 0x5629c4f945c0 2024-03-31T23:28:33.054 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:33.048+0000 7f86eaa3a640 1 --2- 172.21.15.46:0/2337966402 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5629c4f945c0 0x5629c4f8e220 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7f86dc0ee970 tx=0x7f86dc0b9670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7eb8eac641993b1c server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:28:33.054 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:33.048+0000 7f86cdffb640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-31T23:28:33.054 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:33.049+0000 7f86cdffb640 1 -- 172.21.15.46:0/2337966402 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86dc0b9ca0 con 0x5629c4f945c0 2024-03-31T23:28:33.054 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:33.049+0000 7f86cdffb640 1 -- 172.21.15.46:0/2337966402 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(119..119 src has 1..119) ==== 9554+0+0 (secure 0 0 0) 0x7f86dc04d200 con 0x5629c4f945c0 2024-03-31T23:28:33.054 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:33.049+0000 7f86ef084740 1 -- 172.21.15.46:0/2337966402 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f86d4022070 con 0x5629c4f945c0 2024-03-31T23:28:33.054 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:33.049+0000 7f86cdffb640 1 -- 172.21.15.46:0/2337966402 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=44) ==== 24+0+0 (secure 0 0 0) 0x7f86dc0be0a0 con 0x5629c4f945c0 2024-03-31T23:28:33.054 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:33.049+0000 7f86ef084740 10 client.17517 fetch_fsmap learned FSMap version 44 2024-03-31T23:28:33.055 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:33.049+0000 7f86ef084740 1 -- 172.21.15.46:0/2337966402 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=44,monmap=4+}) -- 0x5629c500a380 con 0x5629c4f945c0 2024-03-31T23:28:33.055 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:33.049+0000 7f86cdffb640 1 -- 172.21.15.46:0/2337966402 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f86dc031930 con 0x5629c4f945c0 2024-03-31T23:28:33.055 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:33.049+0000 7f86cdffb640 1 -- 172.21.15.46:0/2337966402 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 44) ==== 8354+0+0 (secure 0 0 0) 0x7f86dc0c5f20 con 0x5629c4f945c0 2024-03-31T23:28:33.055 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:33.049+0000 7f86ef084740 10 client.17517 fetch_fsmap finished waiting for FSMap version 44 2024-03-31T23:28:33.055 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:33.049+0000 7f86ef084740 10 client.17517 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-31T23:28:33.055 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:33.049+0000 7f86ef084740 20 client.17517 populate_metadata read hostname 'smithi046' 2024-03-31T23:28:33.055 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:33.049+0000 7f86ef084740 1 --2- 172.21.15.46:0/2337966402 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x5629c4fa4470 0x5629c500c930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:28:33.055 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:33.049+0000 7f86ef084740 4 client.17517 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-31T23:28:33.055 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:33.049+0000 7f86ef084740 1 -- 172.21.15.46:0/2337966402 --> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] -- command(tid 0: {"prefix": "flush journal"}) -- 0x7f86d4022070 con 0x5629c4fa4470 2024-03-31T23:28:33.055 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:33.050+0000 7f86ea239640 1 --2- 172.21.15.46:0/2337966402 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x5629c4fa4470 0x5629c500c930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:28:33.055 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:33.050+0000 7f86cdffb640 10 client.17517 ms_handle_connect on v2:172.21.15.142:6838/2519143918 2024-03-31T23:28:33.055 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:33.050+0000 7f86ea239640 1 --2- 172.21.15.46:0/2337966402 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x5629c4fa4470 0x5629c500c930 crc :-1 s=READY pgs=227 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9d655bb4dd3c8e50 server_cookie=4d81356805aefe29 in_seq=0 out_seq=0 2024-03-31T23:28:33.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:28:32 smithi046 ceph-mon[32498]: pgmap v530: 129 pgs: 129 active+clean; 664 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 4.2 MiB/s wr, 300 op/s 2024-03-31T23:28:33.208 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:33.207+0000 7f86cdffb640 1 -- 172.21.15.46:0/2337966402 <== mds.0 v2:172.21.15.142:6838/2519143918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x7f86d4022070 con 0x5629c4fa4470 2024-03-31T23:28:33.208 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:33.207+0000 7f86cdffb640 10 client.17517 handle_command_reply: tid=0 2024-03-31T23:28:33.208 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:33.207+0000 7f86ef084740 1 client.17517 shutdown 2024-03-31T23:28:33.208 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:33.207+0000 7f86ef084740 1 -- 172.21.15.46:0/2337966402 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5629c4f945c0 msgr2=0x5629c4f8e220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:28:33.208 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:33.207+0000 7f86ef084740 1 --2- 172.21.15.46:0/2337966402 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5629c4f945c0 0x5629c4f8e220 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7f86dc0ee970 tx=0x7f86dc0b9670 comp rx=0 tx=0).stop 2024-03-31T23:28:33.208 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:33.208+0000 7f86ef084740 1 -- 172.21.15.46:0/2337966402 shutdown_connections 2024-03-31T23:28:33.209 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:33.208+0000 7f86ef084740 1 --2- 172.21.15.46:0/2337966402 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x5629c4fa4470 0x5629c500c930 crc :-1 s=READY pgs=227 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:28:33.209 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:33.208+0000 7f86ef084740 1 --2- 172.21.15.46:0/2337966402 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5629c4f945c0 0x5629c4f8e220 unknown :-1 s=CLOSED pgs=927 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:28:33.209 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:33.208+0000 7f86ef084740 1 --2- 172.21.15.46:0/2337966402 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5629c4f82fe0 0x5629c4f8dce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:28:33.209 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:33.208+0000 7f86ef084740 1 --2- 172.21.15.46:0/2337966402 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5629c4f826c0 0x5629c4f9c5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:28:33.209 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:33.208+0000 7f86ef084740 1 -- 172.21.15.46:0/2337966402 >> 172.21.15.46:0/2337966402 conn(0x5629c4f7cfe0 msgr2=0x5629c4f7d3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:28:33.209 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:33.208+0000 7f86cdffb640 0 client.17517 ms_handle_reset on v2:172.21.15.142:6838/2519143918 2024-03-31T23:28:33.209 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:33.208+0000 7f86ef084740 1 -- 172.21.15.46:0/2337966402 shutdown_connections 2024-03-31T23:28:33.209 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:33.208+0000 7f86ef084740 1 -- 172.21.15.46:0/2337966402 wait complete. 2024-03-31T23:28:33.209 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:33.208+0000 7f86ef084740 20 client.17517 trim_cache size 0 max 16384 2024-03-31T23:28:33.209 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout:{ 2024-03-31T23:28:33.209 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout: "message": "", 2024-03-31T23:28:33.210 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout: "return_code": 0 2024-03-31T23:28:33.210 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout:} 2024-03-31T23:28:33.210 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:33.209+0000 7f86ec49d640 1 -- 172.21.15.46:0/997540115 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f86d00a2730 msgr2=0x7f86d00a4bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:28:33.211 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:33.209+0000 7f86ec49d640 1 --2- 172.21.15.46:0/997540115 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f86d00a2730 0x7f86d00a4bf0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f86e0009530 tx=0x7f86e0009260 comp rx=0 tx=0).stop 2024-03-31T23:28:33.211 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:33.209+0000 7f86ec49d640 1 -- 172.21.15.46:0/997540115 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f86e41077b0 msgr2=0x7f86e419e680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:28:33.211 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:33.209+0000 7f86ec49d640 1 --2- 172.21.15.46:0/997540115 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f86e41077b0 0x7f86e419e680 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f86dc036040 tx=0x7f86dc033d10 comp rx=0 tx=0).stop 2024-03-31T23:28:33.211 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:33.209+0000 7f86ec49d640 1 -- 172.21.15.46:0/997540115 shutdown_connections 2024-03-31T23:28:33.211 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:33.209+0000 7f86ec49d640 1 --2- 172.21.15.46:0/997540115 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f86d00a2730 0x7f86d00a4bf0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:28:33.211 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:33.209+0000 7f86ec49d640 1 --2- 172.21.15.46:0/997540115 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f86e41077b0 0x7f86e419e680 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:28:33.211 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:33.209+0000 7f86ec49d640 1 --2- 172.21.15.46:0/997540115 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f86e4069770 0x7f86e419a820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:28:33.211 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:33.209+0000 7f86ec49d640 1 --2- 172.21.15.46:0/997540115 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f86e4068dd0 0x7f86e419a2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:28:33.211 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:33.209+0000 7f86ec49d640 1 -- 172.21.15.46:0/997540115 >> 172.21.15.46:0/997540115 conn(0x7f86e40774a0 msgr2=0x7f86e40fd880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:28:33.211 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:33.209+0000 7f86ec49d640 1 -- 172.21.15.46:0/997540115 shutdown_connections 2024-03-31T23:28:33.211 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:33.209+0000 7f86ec49d640 1 -- 172.21.15.46:0/997540115 wait complete. 2024-03-31T23:28:33.283 INFO:tasks.workunit.client.0.smithi046.stdout: 10 35502 8.64 MB/sec execute 98 sec latency 159.772 ms 2024-03-31T23:28:34.283 INFO:tasks.workunit.client.0.smithi046.stdout: 10 35667 8.66 MB/sec execute 99 sec latency 77.118 ms 2024-03-31T23:28:34.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:28:34 smithi142 ceph-mon[35271]: pgmap v531: 129 pgs: 129 active+clean; 664 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 4.2 MiB/s wr, 300 op/s 2024-03-31T23:28:35.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:28:34 smithi162 ceph-mon[36237]: pgmap v531: 129 pgs: 129 active+clean; 664 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 4.2 MiB/s wr, 300 op/s 2024-03-31T23:28:35.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:28:34 smithi046 ceph-mon[32498]: pgmap v531: 129 pgs: 129 active+clean; 664 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 4.2 MiB/s wr, 300 op/s 2024-03-31T23:28:35.283 INFO:tasks.workunit.client.0.smithi046.stdout: 10 35864 8.69 MB/sec execute 100 sec latency 131.154 ms 2024-03-31T23:28:36.283 INFO:tasks.workunit.client.0.smithi046.stdout: 10 36031 8.66 MB/sec execute 101 sec latency 124.595 ms 2024-03-31T23:28:36.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:28:36 smithi142 ceph-mon[35271]: pgmap v532: 129 pgs: 129 active+clean; 666 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 6.3 MiB/s wr, 467 op/s 2024-03-31T23:28:37.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:28:36 smithi162 ceph-mon[36237]: pgmap v532: 129 pgs: 129 active+clean; 666 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 6.3 MiB/s wr, 467 op/s 2024-03-31T23:28:37.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:28:36 smithi046 ceph-mon[32498]: pgmap v532: 129 pgs: 129 active+clean; 666 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 6.3 MiB/s wr, 467 op/s 2024-03-31T23:28:37.283 INFO:tasks.workunit.client.0.smithi046.stdout: 10 36181 8.65 MB/sec execute 102 sec latency 83.142 ms 2024-03-31T23:28:38.283 INFO:tasks.workunit.client.0.smithi046.stdout: 10 36336 8.68 MB/sec execute 103 sec latency 91.197 ms 2024-03-31T23:28:38.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:28:38 smithi142 ceph-mon[35271]: pgmap v533: 129 pgs: 129 active+clean; 666 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 3.9 MiB/s wr, 279 op/s 2024-03-31T23:28:39.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:28:38 smithi162 ceph-mon[36237]: pgmap v533: 129 pgs: 129 active+clean; 666 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 3.9 MiB/s wr, 279 op/s 2024-03-31T23:28:39.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:28:38 smithi046 ceph-mon[32498]: pgmap v533: 129 pgs: 129 active+clean; 666 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 3.9 MiB/s wr, 279 op/s 2024-03-31T23:28:39.283 INFO:tasks.workunit.client.0.smithi046.stdout: 10 36520 8.75 MB/sec execute 104 sec latency 62.390 ms 2024-03-31T23:28:40.283 INFO:tasks.workunit.client.0.smithi046.stdout: 10 36709 8.78 MB/sec execute 105 sec latency 81.614 ms 2024-03-31T23:28:40.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:28:40 smithi142 ceph-mon[35271]: pgmap v534: 129 pgs: 129 active+clean; 675 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 6.0 MiB/s wr, 389 op/s 2024-03-31T23:28:41.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:28:40 smithi162 ceph-mon[36237]: pgmap v534: 129 pgs: 129 active+clean; 675 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 6.0 MiB/s wr, 389 op/s 2024-03-31T23:28:41.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:28:40 smithi046 ceph-mon[32498]: pgmap v534: 129 pgs: 129 active+clean; 675 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 6.0 MiB/s wr, 389 op/s 2024-03-31T23:28:41.283 INFO:tasks.workunit.client.0.smithi046.stdout: 10 36845 8.77 MB/sec execute 106 sec latency 206.525 ms 2024-03-31T23:28:42.284 INFO:tasks.workunit.client.0.smithi046.stdout: 10 36968 8.76 MB/sec execute 107 sec latency 112.754 ms 2024-03-31T23:28:42.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:28:42 smithi142 ceph-mon[35271]: pgmap v535: 129 pgs: 129 active+clean; 675 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 4.2 MiB/s wr, 276 op/s 2024-03-31T23:28:43.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:28:42 smithi162 ceph-mon[36237]: pgmap v535: 129 pgs: 129 active+clean; 675 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 4.2 MiB/s wr, 276 op/s 2024-03-31T23:28:43.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:28:42 smithi046 ceph-mon[32498]: pgmap v535: 129 pgs: 129 active+clean; 675 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 4.2 MiB/s wr, 276 op/s 2024-03-31T23:28:43.283 INFO:tasks.workunit.client.0.smithi046.stdout: 10 37114 8.73 MB/sec execute 108 sec latency 57.957 ms 2024-03-31T23:28:44.283 INFO:tasks.workunit.client.0.smithi046.stdout: 10 37274 8.74 MB/sec execute 109 sec latency 69.136 ms 2024-03-31T23:28:44.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:28:44 smithi142 ceph-mon[35271]: pgmap v536: 129 pgs: 129 active+clean; 675 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 4.2 MiB/s wr, 276 op/s 2024-03-31T23:28:45.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:28:44 smithi162 ceph-mon[36237]: pgmap v536: 129 pgs: 129 active+clean; 675 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 4.2 MiB/s wr, 276 op/s 2024-03-31T23:28:45.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:28:44 smithi046 ceph-mon[32498]: pgmap v536: 129 pgs: 129 active+clean; 675 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 4.2 MiB/s wr, 276 op/s 2024-03-31T23:28:45.283 INFO:tasks.workunit.client.0.smithi046.stdout: 10 37407 8.70 MB/sec execute 110 sec latency 76.339 ms 2024-03-31T23:28:46.286 INFO:tasks.workunit.client.0.smithi046.stdout: 10 37558 8.71 MB/sec execute 111 sec latency 64.132 ms 2024-03-31T23:28:46.539 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:46.537+0000 7fd45f2eb640 1 -- 172.21.15.46:0/1795057686 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd4581008a0 msgr2=0x7fd458100d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:28:46.539 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:46.537+0000 7fd45f2eb640 1 --2- 172.21.15.46:0/1795057686 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd4581008a0 0x7fd458100d20 secure :-1 s=READY pgs=928 cs=0 l=1 rev1=1 crypto rx=0x7fd448009900 tx=0x7fd44802bc30 comp rx=0 tx=0).stop 2024-03-31T23:28:46.539 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:46.537+0000 7fd45f2eb640 1 -- 172.21.15.46:0/1795057686 shutdown_connections 2024-03-31T23:28:46.539 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:46.537+0000 7fd45f2eb640 1 --2- 172.21.15.46:0/1795057686 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd458101260 0x7fd45810d480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:28:46.539 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:46.537+0000 7fd45f2eb640 1 --2- 172.21.15.46:0/1795057686 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd4581008a0 0x7fd458100d20 unknown :-1 s=CLOSED pgs=928 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:28:46.539 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:46.537+0000 7fd45f2eb640 1 --2- 172.21.15.46:0/1795057686 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd4581055f0 0x7fd4581059d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:28:46.539 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:46.537+0000 7fd45f2eb640 1 -- 172.21.15.46:0/1795057686 >> 172.21.15.46:0/1795057686 conn(0x7fd4580774a0 msgr2=0x7fd4580778b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:28:46.539 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:46.537+0000 7fd45f2eb640 1 -- 172.21.15.46:0/1795057686 shutdown_connections 2024-03-31T23:28:46.539 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:46.537+0000 7fd45f2eb640 1 -- 172.21.15.46:0/1795057686 wait complete. 2024-03-31T23:28:46.540 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:46.538+0000 7fd45f2eb640 1 Processor -- start 2024-03-31T23:28:46.540 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:46.538+0000 7fd45f2eb640 1 -- start start 2024-03-31T23:28:46.541 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:46.538+0000 7fd45f2eb640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd4581008a0 0x7fd458197f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:28:46.542 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:46.538+0000 7fd45f2eb640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd458101260 0x7fd458198460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:28:46.542 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:46.538+0000 7fd45f2eb640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd4581055f0 0x7fd45819c7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:28:46.542 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:46.538+0000 7fd45f2eb640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fd458111ae0 con 0x7fd4581008a0 2024-03-31T23:28:46.542 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:46.538+0000 7fd45d087640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd4581008a0 0x7fd458197f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:28:46.542 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:46.538+0000 7fd45d087640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd4581008a0 0x7fd458197f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40098/0 (socket says 172.21.15.46:40098) 2024-03-31T23:28:46.542 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:46.538+0000 7fd45d087640 1 -- 172.21.15.46:0/930065440 learned_addr learned my addr 172.21.15.46:0/930065440 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:28:46.542 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:46.538+0000 7fd45f2eb640 1 -- 172.21.15.46:0/930065440 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fd458111960 con 0x7fd4581055f0 2024-03-31T23:28:46.542 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:46.538+0000 7fd45f2eb640 1 -- 172.21.15.46:0/930065440 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fd458111c60 con 0x7fd458101260 2024-03-31T23:28:46.542 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:46.538+0000 7fd45c886640 1 --2- 172.21.15.46:0/930065440 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd458101260 0x7fd458198460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:28:46.542 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:46.538+0000 7fd45d888640 1 --2- 172.21.15.46:0/930065440 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd4581055f0 0x7fd45819c7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:28:46.542 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:46.539+0000 7fd45c886640 1 -- 172.21.15.46:0/930065440 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd4581055f0 msgr2=0x7fd45819c7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:28:46.543 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:46.539+0000 7fd45c886640 1 --2- 172.21.15.46:0/930065440 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd4581055f0 0x7fd45819c7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:28:46.543 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:46.539+0000 7fd45c886640 1 -- 172.21.15.46:0/930065440 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd4581008a0 msgr2=0x7fd458197f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:28:46.543 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:46.539+0000 7fd45c886640 1 --2- 172.21.15.46:0/930065440 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd4581008a0 0x7fd458197f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:28:46.543 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:46.539+0000 7fd45c886640 1 -- 172.21.15.46:0/930065440 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd45819cf10 con 0x7fd458101260 2024-03-31T23:28:46.543 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:46.539+0000 7fd45d087640 1 --2- 172.21.15.46:0/930065440 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd4581008a0 0x7fd458197f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:28:46.543 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:46.539+0000 7fd45c886640 1 --2- 172.21.15.46:0/930065440 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd458101260 0x7fd458198460 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fd448036040 tx=0x7fd44802cd20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:28:46.543 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:46.539+0000 7fd44e7fc640 1 -- 172.21.15.46:0/930065440 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd448043020 con 0x7fd458101260 2024-03-31T23:28:46.543 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:46.540+0000 7fd44e7fc640 1 -- 172.21.15.46:0/930065440 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd448004410 con 0x7fd458101260 2024-03-31T23:28:46.543 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:46.540+0000 7fd45f2eb640 1 -- 172.21.15.46:0/930065440 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd45819d1a0 con 0x7fd458101260 2024-03-31T23:28:46.543 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:46.540+0000 7fd45f2eb640 1 -- 172.21.15.46:0/930065440 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd4581a4a90 con 0x7fd458101260 2024-03-31T23:28:46.543 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:46.540+0000 7fd44e7fc640 1 -- 172.21.15.46:0/930065440 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd448003b20 con 0x7fd458101260 2024-03-31T23:28:46.544 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:46.543+0000 7fd44e7fc640 1 -- 172.21.15.46:0/930065440 <== mon.1 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 25) ==== 137451+0+0 (secure 0 0 0) 0x7fd448046020 con 0x7fd458101260 2024-03-31T23:28:46.544 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:46.543+0000 7fd461fa7740 1 Processor -- start 2024-03-31T23:28:46.544 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:46.544+0000 7fd461fa7740 1 -- start start 2024-03-31T23:28:46.546 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:46.544+0000 7fd461fa7740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5632ecb696c0 0x5632ecb69aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:28:46.546 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:46.544+0000 7fd461fa7740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5632ecb69fe0 0x5632ecb72870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:28:46.546 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:46.544+0000 7fd461fa7740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5632ecb72db0 0x5632ecb751a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:28:46.546 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:46.544+0000 7fd461fa7740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x5632ecacc8f0 con 0x5632ecb696c0 2024-03-31T23:28:46.546 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:46.544+0000 7fd461fa7740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x5632eca758d0 con 0x5632ecb72db0 2024-03-31T23:28:46.547 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:46.544+0000 7fd461fa7740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5632ecac8990 con 0x5632ecb69fe0 2024-03-31T23:28:46.547 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:46.544+0000 7fd45d888640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5632ecb696c0 0x5632ecb69aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:28:46.547 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:46.544+0000 7fd45d888640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5632ecb696c0 0x5632ecb69aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40114/0 (socket says 172.21.15.46:40114) 2024-03-31T23:28:46.547 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:46.544+0000 7fd45d888640 1 -- 172.21.15.46:0/817124228 learned_addr learned my addr 172.21.15.46:0/817124228 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:28:46.547 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:46.544+0000 7fd44e7fc640 1 --2- 172.21.15.46:0/930065440 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fd4440a27b0 0x7fd4440a4c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:28:46.547 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:46.544+0000 7fd45d087640 1 --2- 172.21.15.46:0/817124228 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5632ecb69fe0 0x5632ecb72870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:28:46.547 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:46.544+0000 7fd45c886640 1 --2- 172.21.15.46:0/817124228 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5632ecb72db0 0x5632ecb751a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:28:46.547 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:46.545+0000 7fd44e7fc640 1 -- 172.21.15.46:0/930065440 <== mon.1 v2:172.21.15.162:3300/0 5 ==== osd_map(119..119 src has 1..119) ==== 9554+0+0 (secure 0 0 0) 0x7fd4480f0880 con 0x7fd458101260 2024-03-31T23:28:46.547 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:46.545+0000 7fd45d888640 1 -- 172.21.15.46:0/817124228 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5632ecb69fe0 msgr2=0x5632ecb72870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:28:46.547 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:46.545+0000 7fd45d888640 1 --2- 172.21.15.46:0/817124228 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5632ecb69fe0 0x5632ecb72870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:28:46.547 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:46.545+0000 7fd45d888640 1 -- 172.21.15.46:0/817124228 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5632ecb72db0 msgr2=0x5632ecb751a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:28:46.548 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:46.545+0000 7fd45d888640 1 --2- 172.21.15.46:0/817124228 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5632ecb72db0 0x5632ecb751a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:28:46.548 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:46.545+0000 7fd45d888640 1 -- 172.21.15.46:0/817124228 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5632ec7e1210 con 0x5632ecb696c0 2024-03-31T23:28:46.548 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:46.545+0000 7fd45d888640 1 --2- 172.21.15.46:0/930065440 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fd4440a27b0 0x7fd4440a4c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:28:46.548 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:46.545+0000 7fd45d888640 1 --2- 172.21.15.46:0/817124228 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5632ecb696c0 0x5632ecb69aa0 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 crypto rx=0x7fd450013560 tx=0x7fd450013a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c1f5654fa3ca4eec server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:28:46.548 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:46.545+0000 7fd4437fe640 1 -- 172.21.15.46:0/817124228 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd450003e50 con 0x5632ecb696c0 2024-03-31T23:28:46.552 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:46.546+0000 7fd4437fe640 1 -- 172.21.15.46:0/817124228 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd450004620 con 0x5632ecb696c0 2024-03-31T23:28:46.552 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:46.546+0000 7fd4437fe640 1 -- 172.21.15.46:0/817124228 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd450026ab0 con 0x5632ecb696c0 2024-03-31T23:28:46.552 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:46.546+0000 7fd461fa7740 1 -- 172.21.15.46:0/817124228 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5632ecb696c0 msgr2=0x5632ecb69aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:28:46.552 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:46.546+0000 7fd45d888640 1 --2- 172.21.15.46:0/930065440 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fd4440a27b0 0x7fd4440a4c70 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fd450021000 tx=0x7fd45001e000 comp rx=0 tx=0).ready entity=mgr.15654 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:28:46.552 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:46.546+0000 7fd461fa7740 1 --2- 172.21.15.46:0/817124228 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5632ecb696c0 0x5632ecb69aa0 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 crypto rx=0x7fd450013560 tx=0x7fd450013a20 comp rx=0 tx=0).stop 2024-03-31T23:28:46.552 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:46.547+0000 7fd461fa7740 1 -- 172.21.15.46:0/817124228 shutdown_connections 2024-03-31T23:28:46.553 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:46.547+0000 7fd461fa7740 1 --2- 172.21.15.46:0/817124228 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5632ecb72db0 0x5632ecb751a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:28:46.553 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:46.547+0000 7fd461fa7740 1 --2- 172.21.15.46:0/817124228 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5632ecb69fe0 0x5632ecb72870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:28:46.553 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:46.547+0000 7fd461fa7740 1 --2- 172.21.15.46:0/817124228 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5632ecb696c0 0x5632ecb69aa0 unknown :-1 s=CLOSED pgs=929 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:28:46.553 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:46.547+0000 7fd461fa7740 1 -- 172.21.15.46:0/817124228 >> 172.21.15.46:0/817124228 conn(0x5632ecb63fe0 msgr2=0x5632ecb643c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:28:46.553 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:46.547+0000 7fd461fa7740 1 -- 172.21.15.46:0/817124228 shutdown_connections 2024-03-31T23:28:46.553 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:46.547+0000 7fd461fa7740 1 -- 172.21.15.46:0/817124228 wait complete. 2024-03-31T23:28:46.553 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:46.547+0000 7fd461fa7740 1 -- start start 2024-03-31T23:28:46.553 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:46.547+0000 7fd461fa7740 1 Processor -- start 2024-03-31T23:28:46.553 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:46.548+0000 7fd461fa7740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5632ecb696c0 0x5632ecb6fd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:28:46.553 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:46.548+0000 7fd461fa7740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5632ecb69fe0 0x5632ecb70260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:28:46.554 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:46.548+0000 7fd461fa7740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5632ecb72db0 0x5632ecb707a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:28:46.554 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:46.548+0000 7fd461fa7740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x5632ecb79df0 con 0x5632ecb72db0 2024-03-31T23:28:46.554 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:46.548+0000 7fd461fa7740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x5632eca6af80 con 0x5632ecb696c0 2024-03-31T23:28:46.554 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:46.548+0000 7fd461fa7740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5632ecb79f70 con 0x5632ecb69fe0 2024-03-31T23:28:46.554 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:46.548+0000 7fd45c886640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5632ecb69fe0 0x5632ecb70260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:28:46.554 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:46.548+0000 7fd45c886640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5632ecb69fe0 0x5632ecb70260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.46:39530/0 (socket says 172.21.15.46:39530) 2024-03-31T23:28:46.554 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:46.548+0000 7fd45c886640 1 -- 172.21.15.46:0/2132093240 learned_addr learned my addr 172.21.15.46:0/2132093240 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:28:46.554 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:46.548+0000 7fd461fa7740 10 client.0 _ll_register_callbacks cb 0x5632ecac3290 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T23:28:46.554 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:46.548+0000 7fd45d087640 1 --2- 172.21.15.46:0/2132093240 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5632ecb696c0 0x5632ecb6fd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:28:46.554 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:46.548+0000 7fd45d888640 1 --2- 172.21.15.46:0/2132093240 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5632ecb72db0 0x5632ecb707a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:28:46.554 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:46.548+0000 7fd45c886640 1 -- 172.21.15.46:0/2132093240 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5632ecb696c0 msgr2=0x5632ecb6fd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:28:46.554 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:46.548+0000 7fd45c886640 1 --2- 172.21.15.46:0/2132093240 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5632ecb696c0 0x5632ecb6fd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:28:46.554 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:46.548+0000 7fd45c886640 1 -- 172.21.15.46:0/2132093240 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5632ecb72db0 msgr2=0x5632ecb707a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:28:46.555 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:46.549+0000 7fd45c886640 1 --2- 172.21.15.46:0/2132093240 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5632ecb72db0 0x5632ecb707a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:28:46.555 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:46.549+0000 7fd45c886640 1 -- 172.21.15.46:0/2132093240 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5632ec8ad620 con 0x5632ecb69fe0 2024-03-31T23:28:46.555 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:46.549+0000 7fd45d888640 1 --2- 172.21.15.46:0/2132093240 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5632ecb72db0 0x5632ecb707a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:28:46.555 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:46.549+0000 7fd45c886640 1 --2- 172.21.15.46:0/2132093240 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5632ecb69fe0 0x5632ecb70260 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fd448004650 tx=0x7fd4480bcb30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b296259674a8779 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:28:46.555 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:46.549+0000 7fd440ff9640 10 client.0 ms_handle_connect on v2:172.21.15.162:3300/0 2024-03-31T23:28:46.555 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:46.549+0000 7fd440ff9640 1 -- 172.21.15.46:0/2132093240 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4480d4020 con 0x5632ecb69fe0 2024-03-31T23:28:46.555 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:46.549+0000 7fd440ff9640 1 -- 172.21.15.46:0/2132093240 <== mon.1 v2:172.21.15.162:3300/0 2 ==== osd_map(119..119 src has 1..119) ==== 9554+0+0 (secure 0 0 0) 0x7fd448003cc0 con 0x5632ecb69fe0 2024-03-31T23:28:46.555 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:46.549+0000 7fd461fa7740 1 -- 172.21.15.46:0/2132093240 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd450004620 con 0x5632ecb69fe0 2024-03-31T23:28:46.555 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:46.549+0000 7fd440ff9640 1 -- 172.21.15.46:0/2132093240 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_get_version_reply(handle=1 version=44) ==== 24+0+0 (secure 0 0 0) 0x7fd4480bcca0 con 0x5632ecb69fe0 2024-03-31T23:28:46.555 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:46.549+0000 7fd461fa7740 10 client.27256 fetch_fsmap learned FSMap version 44 2024-03-31T23:28:46.555 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:46.549+0000 7fd461fa7740 1 -- 172.21.15.46:0/2132093240 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,fsmap=44,monmap=4+}) -- 0x5632ecb79df0 con 0x5632ecb69fe0 2024-03-31T23:28:46.556 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:46.550+0000 7fd440ff9640 1 -- 172.21.15.46:0/2132093240 <== mon.1 v2:172.21.15.162:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd450004620 con 0x5632ecb69fe0 2024-03-31T23:28:46.556 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:46.550+0000 7fd440ff9640 1 -- 172.21.15.46:0/2132093240 <== mon.1 v2:172.21.15.162:3300/0 5 ==== fsmap(e 44) ==== 8354+0+0 (secure 0 0 0) 0x7fd448050060 con 0x5632ecb69fe0 2024-03-31T23:28:46.556 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:46.550+0000 7fd461fa7740 10 client.27256 fetch_fsmap finished waiting for FSMap version 44 2024-03-31T23:28:46.556 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:46.550+0000 7fd461fa7740 10 client.27256 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-31T23:28:46.556 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:46.550+0000 7fd461fa7740 20 client.27256 populate_metadata read hostname 'smithi046' 2024-03-31T23:28:46.556 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:46.550+0000 7fd461fa7740 1 --2- 172.21.15.46:0/2132093240 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x5632ecb870e0 0x5632ecb89560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:28:46.556 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:46.550+0000 7fd461fa7740 4 client.27256 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-31T23:28:46.556 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:46.550+0000 7fd461fa7740 1 -- 172.21.15.46:0/2132093240 --> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5632ecae0120 con 0x5632ecb870e0 2024-03-31T23:28:46.556 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:46.551+0000 7fd45d087640 1 --2- 172.21.15.46:0/2132093240 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x5632ecb870e0 0x5632ecb89560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:28:46.556 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:46.551+0000 7fd440ff9640 10 client.27256 ms_handle_connect on v2:172.21.15.142:6838/2519143918 2024-03-31T23:28:46.556 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:46.551+0000 7fd45d087640 1 --2- 172.21.15.46:0/2132093240 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x5632ecb870e0 0x5632ecb89560 crc :-1 s=READY pgs=228 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d7aafd8bd72f5c2e server_cookie=6bbf9e4a33655628 in_seq=0 out_seq=0 2024-03-31T23:28:46.558 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:46.552+0000 7fd440ff9640 1 -- 172.21.15.46:0/2132093240 <== mds.0 v2:172.21.15.142:6838/2519143918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5632ecae0120 con 0x5632ecb870e0 2024-03-31T23:28:46.558 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:46.552+0000 7fd440ff9640 10 client.27256 handle_command_reply: tid=0 2024-03-31T23:28:46.558 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:46.552+0000 7fd461fa7740 1 client.27256 shutdown 2024-03-31T23:28:46.558 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:46.552+0000 7fd461fa7740 1 -- 172.21.15.46:0/2132093240 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5632ecb69fe0 msgr2=0x5632ecb70260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:28:46.558 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:46.552+0000 7fd461fa7740 1 --2- 172.21.15.46:0/2132093240 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5632ecb69fe0 0x5632ecb70260 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fd448004650 tx=0x7fd4480bcb30 comp rx=0 tx=0).stop 2024-03-31T23:28:46.558 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:46.552+0000 7fd461fa7740 1 -- 172.21.15.46:0/2132093240 shutdown_connections 2024-03-31T23:28:46.558 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:46.552+0000 7fd461fa7740 1 --2- 172.21.15.46:0/2132093240 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x5632ecb870e0 0x5632ecb89560 crc :-1 s=READY pgs=228 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:28:46.558 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:46.552+0000 7fd461fa7740 1 --2- 172.21.15.46:0/2132093240 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5632ecb72db0 0x5632ecb707a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:28:46.559 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:46.552+0000 7fd440ff9640 0 client.27256 ms_handle_reset on v2:172.21.15.142:6838/2519143918 2024-03-31T23:28:46.559 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:46.552+0000 7fd461fa7740 1 --2- 172.21.15.46:0/2132093240 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5632ecb69fe0 0x5632ecb70260 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:28:46.559 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:46.552+0000 7fd461fa7740 1 --2- 172.21.15.46:0/2132093240 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5632ecb696c0 0x5632ecb6fd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:28:46.559 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:46.552+0000 7fd461fa7740 1 -- 172.21.15.46:0/2132093240 >> 172.21.15.46:0/2132093240 conn(0x5632ecb63fe0 msgr2=0x5632ecb65d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:28:46.559 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:46.552+0000 7fd461fa7740 1 -- 172.21.15.46:0/2132093240 shutdown_connections 2024-03-31T23:28:46.559 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:46.553+0000 7fd461fa7740 1 -- 172.21.15.46:0/2132093240 wait complete. 2024-03-31T23:28:46.559 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:46.553+0000 7fd461fa7740 20 client.27256 trim_cache size 0 max 16384 2024-03-31T23:28:46.578 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:46.574+0000 7fd461fa7740 1 Processor -- start 2024-03-31T23:28:46.578 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:46.574+0000 7fd461fa7740 1 -- start start 2024-03-31T23:28:46.578 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:46.575+0000 7fd461fa7740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5632ecb696c0 0x5632ecb72450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:28:46.578 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:46.575+0000 7fd461fa7740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5632ecb69fe0 0x5632ecb82b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:28:46.578 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:46.575+0000 7fd461fa7740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5632ecb72db0 0x5632ecb8ee70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:28:46.578 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:46.575+0000 7fd461fa7740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fd43c012960 con 0x5632ecb69fe0 2024-03-31T23:28:46.578 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:46.575+0000 7fd461fa7740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fd43c0101d0 con 0x5632ecb72db0 2024-03-31T23:28:46.578 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:46.575+0000 7fd461fa7740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fd43c00e040 con 0x5632ecb696c0 2024-03-31T23:28:46.578 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:46.575+0000 7fd45c886640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5632ecb69fe0 0x5632ecb82b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:28:46.578 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:46.575+0000 7fd45c886640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5632ecb69fe0 0x5632ecb82b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40132/0 (socket says 172.21.15.46:40132) 2024-03-31T23:28:46.579 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:46.575+0000 7fd45c886640 1 -- 172.21.15.46:0/2204220966 learned_addr learned my addr 172.21.15.46:0/2204220966 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:28:46.579 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:46.575+0000 7fd45d087640 1 --2- 172.21.15.46:0/2204220966 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5632ecb696c0 0x5632ecb72450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:28:46.579 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:46.575+0000 7fd45d888640 1 --2- 172.21.15.46:0/2204220966 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5632ecb72db0 0x5632ecb8ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:28:46.579 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:46.575+0000 7fd45c886640 1 -- 172.21.15.46:0/2204220966 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5632ecb696c0 msgr2=0x5632ecb72450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:28:46.579 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:46.575+0000 7fd45c886640 1 --2- 172.21.15.46:0/2204220966 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5632ecb696c0 0x5632ecb72450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:28:46.579 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:46.575+0000 7fd45c886640 1 -- 172.21.15.46:0/2204220966 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5632ecb72db0 msgr2=0x5632ecb8ee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:28:46.579 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:46.575+0000 7fd45c886640 1 --2- 172.21.15.46:0/2204220966 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5632ecb72db0 0x5632ecb8ee70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:28:46.579 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:46.575+0000 7fd45c886640 1 -- 172.21.15.46:0/2204220966 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5632ecb751a0 con 0x5632ecb69fe0 2024-03-31T23:28:46.579 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:46.575+0000 7fd45c886640 1 --2- 172.21.15.46:0/2204220966 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5632ecb69fe0 0x5632ecb82b90 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7fd4480bc9a0 tx=0x7fd4480bb1b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a312c3e2bdac3b6f server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:28:46.579 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:46.576+0000 7fd442ffd640 1 -- 172.21.15.46:0/2204220966 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4480d4070 con 0x5632ecb69fe0 2024-03-31T23:28:46.579 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:46.576+0000 7fd442ffd640 1 -- 172.21.15.46:0/2204220966 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd4480f03b0 con 0x5632ecb69fe0 2024-03-31T23:28:46.579 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:46.576+0000 7fd442ffd640 1 -- 172.21.15.46:0/2204220966 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4480c5220 con 0x5632ecb69fe0 2024-03-31T23:28:46.579 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:46.576+0000 7fd461fa7740 1 -- 172.21.15.46:0/2204220966 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5632ecb69fe0 msgr2=0x5632ecb82b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:28:46.580 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:46.576+0000 7fd461fa7740 1 --2- 172.21.15.46:0/2204220966 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5632ecb69fe0 0x5632ecb82b90 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7fd4480bc9a0 tx=0x7fd4480bb1b0 comp rx=0 tx=0).stop 2024-03-31T23:28:46.580 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:46.576+0000 7fd461fa7740 1 -- 172.21.15.46:0/2204220966 shutdown_connections 2024-03-31T23:28:46.580 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:46.576+0000 7fd461fa7740 1 --2- 172.21.15.46:0/2204220966 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5632ecb72db0 0x5632ecb8ee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:28:46.580 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:46.576+0000 7fd461fa7740 1 --2- 172.21.15.46:0/2204220966 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5632ecb69fe0 0x5632ecb82b90 unknown :-1 s=CLOSED pgs=930 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:28:46.580 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:46.576+0000 7fd461fa7740 1 --2- 172.21.15.46:0/2204220966 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5632ecb696c0 0x5632ecb72450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:28:46.580 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:46.576+0000 7fd461fa7740 1 -- 172.21.15.46:0/2204220966 >> 172.21.15.46:0/2204220966 conn(0x5632ecb63fe0 msgr2=0x5632ecb643c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:28:46.580 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:46.577+0000 7fd461fa7740 1 -- 172.21.15.46:0/2204220966 shutdown_connections 2024-03-31T23:28:46.580 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:46.577+0000 7fd461fa7740 1 -- 172.21.15.46:0/2204220966 wait complete. 2024-03-31T23:28:46.583 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:46.577+0000 7fd461fa7740 1 -- start start 2024-03-31T23:28:46.583 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:46.577+0000 7fd461fa7740 1 Processor -- start 2024-03-31T23:28:46.583 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:46.578+0000 7fd461fa7740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5632ecb696c0 0x5632ecb6c970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:28:46.583 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:46.579+0000 7fd461fa7740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5632ecb72db0 0x5632ecb6ceb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:28:46.583 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:46.579+0000 7fd461fa7740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5632ecb870e0 0x5632ecb7e1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:28:46.584 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:46.579+0000 7fd45d087640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5632ecb696c0 0x5632ecb6c970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:28:46.584 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:46.579+0000 7fd461fa7740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fd43c00e690 con 0x5632ecb696c0 2024-03-31T23:28:46.584 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:46.579+0000 7fd45d087640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5632ecb696c0 0x5632ecb6c970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40142/0 (socket says 172.21.15.46:40142) 2024-03-31T23:28:46.584 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:46.579+0000 7fd45d087640 1 -- 172.21.15.46:0/1486631497 learned_addr learned my addr 172.21.15.46:0/1486631497 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:28:46.584 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:46.579+0000 7fd461fa7740 1 -- 172.21.15.46:0/1486631497 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fd43c00fb00 con 0x5632ecb72db0 2024-03-31T23:28:46.584 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:46.579+0000 7fd461fa7740 1 -- 172.21.15.46:0/1486631497 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fd43c00ed60 con 0x5632ecb870e0 2024-03-31T23:28:46.584 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:46.579+0000 7fd461fa7740 10 client.0 _ll_register_callbacks cb 0x7fd4480b8f40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T23:28:46.584 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:46.580+0000 7fd45d087640 1 -- 172.21.15.46:0/1486631497 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5632ecb870e0 msgr2=0x5632ecb7e1b0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-31T23:28:46.584 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:46.580+0000 7fd45c886640 1 --2- 172.21.15.46:0/1486631497 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5632ecb72db0 0x5632ecb6ceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:28:46.584 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:46.580+0000 7fd45d888640 1 --2- 172.21.15.46:0/1486631497 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5632ecb870e0 0x5632ecb7e1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:28:46.585 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:46.580+0000 7fd45d087640 1 --2- 172.21.15.46:0/1486631497 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5632ecb870e0 0x5632ecb7e1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:28:46.585 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:46.580+0000 7fd45d087640 1 -- 172.21.15.46:0/1486631497 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5632ecb72db0 msgr2=0x5632ecb6ceb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:28:46.585 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:46.580+0000 7fd45d087640 1 --2- 172.21.15.46:0/1486631497 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5632ecb72db0 0x5632ecb6ceb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:28:46.585 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:46.580+0000 7fd45d087640 1 -- 172.21.15.46:0/1486631497 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5632ecb7e6f0 con 0x5632ecb696c0 2024-03-31T23:28:46.585 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:46.580+0000 7fd45c886640 1 --2- 172.21.15.46:0/1486631497 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5632ecb72db0 0x5632ecb6ceb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T23:28:46.585 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:46.580+0000 7fd45d087640 1 --2- 172.21.15.46:0/1486631497 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5632ecb696c0 0x5632ecb6c970 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x5632ecb89e10 tx=0x7fd4540227f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=523a48c318162c4f server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:28:46.585 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:46.580+0000 7fd440ff9640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-31T23:28:46.585 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:46.581+0000 7fd440ff9640 1 -- 172.21.15.46:0/1486631497 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd454012ca0 con 0x5632ecb696c0 2024-03-31T23:28:46.585 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:46.581+0000 7fd440ff9640 1 -- 172.21.15.46:0/1486631497 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(119..119 src has 1..119) ==== 9554+0+0 (secure 0 0 0) 0x7fd454005020 con 0x5632ecb696c0 2024-03-31T23:28:46.585 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:46.581+0000 7fd461fa7740 1 -- 172.21.15.46:0/1486631497 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd4480f03b0 con 0x5632ecb696c0 2024-03-31T23:28:46.586 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:46.581+0000 7fd440ff9640 1 -- 172.21.15.46:0/1486631497 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=44) ==== 24+0+0 (secure 0 0 0) 0x7fd454012e40 con 0x5632ecb696c0 2024-03-31T23:28:46.586 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:46.581+0000 7fd461fa7740 10 client.17535 fetch_fsmap learned FSMap version 44 2024-03-31T23:28:46.586 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:46.581+0000 7fd461fa7740 1 -- 172.21.15.46:0/1486631497 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=44,monmap=4+}) -- 0x5632ecbf1c90 con 0x5632ecb696c0 2024-03-31T23:28:46.586 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:46.582+0000 7fd440ff9640 1 -- 172.21.15.46:0/1486631497 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd4480f03b0 con 0x5632ecb696c0 2024-03-31T23:28:46.586 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:46.582+0000 7fd440ff9640 1 -- 172.21.15.46:0/1486631497 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 44) ==== 8354+0+0 (secure 0 0 0) 0x7fd4540035f0 con 0x5632ecb696c0 2024-03-31T23:28:46.586 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:46.582+0000 7fd461fa7740 10 client.17535 fetch_fsmap finished waiting for FSMap version 44 2024-03-31T23:28:46.586 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:46.582+0000 7fd461fa7740 10 client.17535 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-31T23:28:46.586 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:46.582+0000 7fd461fa7740 20 client.17535 populate_metadata read hostname 'smithi046' 2024-03-31T23:28:46.586 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:46.582+0000 7fd461fa7740 1 --2- 172.21.15.46:0/1486631497 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x5632ecbf22e0 0x5632ecbf4700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:28:46.586 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:46.582+0000 7fd461fa7740 4 client.17535 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-31T23:28:46.586 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:46.582+0000 7fd461fa7740 1 -- 172.21.15.46:0/1486631497 --> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5632ecad9e30 con 0x5632ecbf22e0 2024-03-31T23:28:46.587 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:46.583+0000 7fd45c886640 1 --2- 172.21.15.46:0/1486631497 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x5632ecbf22e0 0x5632ecbf4700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:28:46.587 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:46.584+0000 7fd45c886640 1 --2- 172.21.15.46:0/1486631497 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x5632ecbf22e0 0x5632ecbf4700 crc :-1 s=READY pgs=229 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b848059a1fb33d6 server_cookie=3de269053f30eede in_seq=0 out_seq=0 2024-03-31T23:28:46.587 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:46.584+0000 7fd440ff9640 10 client.17535 ms_handle_connect on v2:172.21.15.142:6838/2519143918 2024-03-31T23:28:46.751 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:46.749+0000 7fd440ff9640 1 -- 172.21.15.46:0/1486631497 <== mds.0 v2:172.21.15.142:6838/2519143918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5632ecad9e30 con 0x5632ecbf22e0 2024-03-31T23:28:46.751 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:46.749+0000 7fd440ff9640 10 client.17535 handle_command_reply: tid=0 2024-03-31T23:28:46.751 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:46.749+0000 7fd461fa7740 1 client.17535 shutdown 2024-03-31T23:28:46.751 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:46.749+0000 7fd461fa7740 1 -- 172.21.15.46:0/1486631497 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5632ecb696c0 msgr2=0x5632ecb6c970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:28:46.751 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:46.749+0000 7fd461fa7740 1 --2- 172.21.15.46:0/1486631497 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5632ecb696c0 0x5632ecb6c970 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x5632ecb89e10 tx=0x7fd4540227f0 comp rx=0 tx=0).stop 2024-03-31T23:28:46.751 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:46.749+0000 7fd461fa7740 1 -- 172.21.15.46:0/1486631497 shutdown_connections 2024-03-31T23:28:46.751 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:46.749+0000 7fd461fa7740 1 --2- 172.21.15.46:0/1486631497 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x5632ecbf22e0 0x5632ecbf4700 crc :-1 s=READY pgs=229 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:28:46.751 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:46.749+0000 7fd461fa7740 1 --2- 172.21.15.46:0/1486631497 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5632ecb870e0 0x5632ecb7e1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:28:46.752 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:46.749+0000 7fd461fa7740 1 --2- 172.21.15.46:0/1486631497 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5632ecb72db0 0x5632ecb6ceb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:28:46.752 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:46.749+0000 7fd461fa7740 1 --2- 172.21.15.46:0/1486631497 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5632ecb696c0 0x5632ecb6c970 unknown :-1 s=CLOSED pgs=931 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:28:46.752 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:46.749+0000 7fd440ff9640 0 client.17535 ms_handle_reset on v2:172.21.15.142:6838/2519143918 2024-03-31T23:28:46.752 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:46.749+0000 7fd461fa7740 1 -- 172.21.15.46:0/1486631497 >> 172.21.15.46:0/1486631497 conn(0x5632ecb63fe0 msgr2=0x5632ecb65360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:28:46.752 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:46.749+0000 7fd461fa7740 1 -- 172.21.15.46:0/1486631497 shutdown_connections 2024-03-31T23:28:46.752 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:46.749+0000 7fd461fa7740 1 -- 172.21.15.46:0/1486631497 wait complete. 2024-03-31T23:28:46.752 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:46.749+0000 7fd461fa7740 20 client.17535 trim_cache size 0 max 16384 2024-03-31T23:28:46.752 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout:{ 2024-03-31T23:28:46.752 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout: "message": "", 2024-03-31T23:28:46.752 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout: "return_code": 0 2024-03-31T23:28:46.752 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout:} 2024-03-31T23:28:46.753 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:46.750+0000 7fd45f2eb640 1 -- 172.21.15.46:0/930065440 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fd4440a27b0 msgr2=0x7fd4440a4c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:28:46.753 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:46.750+0000 7fd45f2eb640 1 --2- 172.21.15.46:0/930065440 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fd4440a27b0 0x7fd4440a4c70 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fd450021000 tx=0x7fd45001e000 comp rx=0 tx=0).stop 2024-03-31T23:28:46.753 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:46.750+0000 7fd45f2eb640 1 -- 172.21.15.46:0/930065440 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd458101260 msgr2=0x7fd458198460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:28:46.753 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:46.750+0000 7fd45f2eb640 1 --2- 172.21.15.46:0/930065440 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd458101260 0x7fd458198460 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fd448036040 tx=0x7fd44802cd20 comp rx=0 tx=0).stop 2024-03-31T23:28:46.754 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:46.750+0000 7fd45f2eb640 1 -- 172.21.15.46:0/930065440 shutdown_connections 2024-03-31T23:28:46.754 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:46.750+0000 7fd45f2eb640 1 --2- 172.21.15.46:0/930065440 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fd4440a27b0 0x7fd4440a4c70 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:28:46.754 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:46.750+0000 7fd45f2eb640 1 --2- 172.21.15.46:0/930065440 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd4581055f0 0x7fd45819c7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:28:46.754 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:46.750+0000 7fd45f2eb640 1 --2- 172.21.15.46:0/930065440 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd458101260 0x7fd458198460 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:28:46.754 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:46.750+0000 7fd45f2eb640 1 --2- 172.21.15.46:0/930065440 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd4581008a0 0x7fd458197f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:28:46.754 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:46.750+0000 7fd45f2eb640 1 -- 172.21.15.46:0/930065440 >> 172.21.15.46:0/930065440 conn(0x7fd4580774a0 msgr2=0x7fd4580ff5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:28:46.754 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:46.750+0000 7fd45f2eb640 1 -- 172.21.15.46:0/930065440 shutdown_connections 2024-03-31T23:28:46.754 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:46.750+0000 7fd45f2eb640 1 -- 172.21.15.46:0/930065440 wait complete. 2024-03-31T23:28:46.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:28:46 smithi142 ceph-mon[35271]: pgmap v537: 129 pgs: 129 active+clean; 692 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 6.4 MiB/s wr, 370 op/s 2024-03-31T23:28:47.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:28:46 smithi162 ceph-mon[36237]: pgmap v537: 129 pgs: 129 active+clean; 692 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 6.4 MiB/s wr, 370 op/s 2024-03-31T23:28:47.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:28:46 smithi046 ceph-mon[32498]: pgmap v537: 129 pgs: 129 active+clean; 692 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 6.4 MiB/s wr, 370 op/s 2024-03-31T23:28:47.286 INFO:tasks.workunit.client.0.smithi046.stdout: 10 37694 8.70 MB/sec execute 112 sec latency 144.812 ms 2024-03-31T23:28:48.287 INFO:tasks.workunit.client.0.smithi046.stdout: 10 37836 8.70 MB/sec execute 113 sec latency 99.704 ms 2024-03-31T23:28:48.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:28:48 smithi142 ceph-mon[35271]: pgmap v538: 129 pgs: 129 active+clean; 692 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 4.3 MiB/s wr, 203 op/s 2024-03-31T23:28:49.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:28:48 smithi162 ceph-mon[36237]: pgmap v538: 129 pgs: 129 active+clean; 692 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 4.3 MiB/s wr, 203 op/s 2024-03-31T23:28:49.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:28:48 smithi046 ceph-mon[32498]: pgmap v538: 129 pgs: 129 active+clean; 692 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 4.3 MiB/s wr, 203 op/s 2024-03-31T23:28:49.287 INFO:tasks.workunit.client.0.smithi046.stdout: 10 37992 8.68 MB/sec execute 114 sec latency 72.272 ms 2024-03-31T23:28:50.287 INFO:tasks.workunit.client.0.smithi046.stdout: 10 38118 8.63 MB/sec execute 115 sec latency 139.298 ms 2024-03-31T23:28:50.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:28:50 smithi142 ceph-mon[35271]: pgmap v539: 129 pgs: 129 active+clean; 692 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 6.5 MiB/s wr, 364 op/s 2024-03-31T23:28:51.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:28:50 smithi162 ceph-mon[36237]: pgmap v539: 129 pgs: 129 active+clean; 692 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 6.5 MiB/s wr, 364 op/s 2024-03-31T23:28:51.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:28:50 smithi046 ceph-mon[32498]: pgmap v539: 129 pgs: 129 active+clean; 692 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 6.5 MiB/s wr, 364 op/s 2024-03-31T23:28:51.287 INFO:tasks.workunit.client.0.smithi046.stdout: 10 38282 8.60 MB/sec execute 116 sec latency 66.088 ms 2024-03-31T23:28:52.288 INFO:tasks.workunit.client.0.smithi046.stdout: 10 38444 8.58 MB/sec execute 117 sec latency 68.213 ms 2024-03-31T23:28:52.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:28:52 smithi142 ceph-mon[35271]: pgmap v540: 129 pgs: 129 active+clean; 692 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 4.4 MiB/s wr, 254 op/s 2024-03-31T23:28:53.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:28:52 smithi162 ceph-mon[36237]: pgmap v540: 129 pgs: 129 active+clean; 692 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 4.4 MiB/s wr, 254 op/s 2024-03-31T23:28:53.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:28:52 smithi046 ceph-mon[32498]: pgmap v540: 129 pgs: 129 active+clean; 692 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 4.4 MiB/s wr, 254 op/s 2024-03-31T23:28:53.287 INFO:tasks.workunit.client.0.smithi046.stdout: 10 38609 8.63 MB/sec execute 118 sec latency 72.421 ms 2024-03-31T23:28:54.287 INFO:tasks.workunit.client.0.smithi046.stdout: 10 38757 8.62 MB/sec execute 119 sec latency 67.221 ms 2024-03-31T23:28:55.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:28:54 smithi162 ceph-mon[36237]: pgmap v541: 129 pgs: 129 active+clean; 692 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 4.4 MiB/s wr, 254 op/s 2024-03-31T23:28:55.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:28:54 smithi162 ceph-mon[36237]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-03-31T23:28:55.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:28:54 smithi046 ceph-mon[32498]: pgmap v541: 129 pgs: 129 active+clean; 692 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 4.4 MiB/s wr, 254 op/s 2024-03-31T23:28:55.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:28:54 smithi046 ceph-mon[32498]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-03-31T23:28:55.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:28:54 smithi142 ceph-mon[35271]: pgmap v541: 129 pgs: 129 active+clean; 692 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 4.4 MiB/s wr, 254 op/s 2024-03-31T23:28:55.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:28:54 smithi142 ceph-mon[35271]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-03-31T23:28:55.287 INFO:tasks.workunit.client.0.smithi046.stdout: 10 38923 8.62 MB/sec execute 120 sec latency 65.485 ms 2024-03-31T23:28:56.287 INFO:tasks.workunit.client.0.smithi046.stdout: 10 39104 8.65 MB/sec execute 121 sec latency 70.905 ms 2024-03-31T23:28:57.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:28:56 smithi162 ceph-mon[36237]: pgmap v542: 129 pgs: 129 active+clean; 692 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 6.2 MiB/s wr, 378 op/s 2024-03-31T23:28:57.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:28:56 smithi046 ceph-mon[32498]: pgmap v542: 129 pgs: 129 active+clean; 692 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 6.2 MiB/s wr, 378 op/s 2024-03-31T23:28:57.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:28:56 smithi142 ceph-mon[35271]: pgmap v542: 129 pgs: 129 active+clean; 692 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 6.2 MiB/s wr, 378 op/s 2024-03-31T23:28:57.287 INFO:tasks.workunit.client.0.smithi046.stdout: 10 39276 8.63 MB/sec execute 122 sec latency 70.276 ms 2024-03-31T23:28:58.288 INFO:tasks.workunit.client.0.smithi046.stdout: 10 39451 8.62 MB/sec execute 123 sec latency 75.068 ms 2024-03-31T23:28:59.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:28:58 smithi162 ceph-mon[36237]: pgmap v543: 129 pgs: 129 active+clean; 692 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 4.1 MiB/s wr, 284 op/s 2024-03-31T23:28:59.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:28:58 smithi046 ceph-mon[32498]: pgmap v543: 129 pgs: 129 active+clean; 692 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 4.1 MiB/s wr, 284 op/s 2024-03-31T23:28:59.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:28:58 smithi142 ceph-mon[35271]: pgmap v543: 129 pgs: 129 active+clean; 692 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 4.1 MiB/s wr, 284 op/s 2024-03-31T23:28:59.288 INFO:tasks.workunit.client.0.smithi046.stdout: 10 39557 8.57 MB/sec execute 124 sec latency 81.772 ms 2024-03-31T23:28:59.914 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:59.913+0000 7fee2cd02640 1 -- 172.21.15.46:0/2820576976 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fee280696f0 msgr2=0x7fee2810df10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:28:59.915 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:59.913+0000 7fee2cd02640 1 --2- 172.21.15.46:0/2820576976 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fee280696f0 0x7fee2810df10 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 crypto rx=0x7fee10009900 tx=0x7fee1002bcd0 comp rx=0 tx=0).stop 2024-03-31T23:28:59.915 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:59.913+0000 7fee2cd02640 1 -- 172.21.15.46:0/2820576976 shutdown_connections 2024-03-31T23:28:59.915 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:59.913+0000 7fee2cd02640 1 --2- 172.21.15.46:0/2820576976 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fee2810e450 0x7fee28110840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:28:59.915 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:59.913+0000 7fee2cd02640 1 --2- 172.21.15.46:0/2820576976 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fee280696f0 0x7fee2810df10 unknown :-1 s=CLOSED pgs=932 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:28:59.915 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:59.913+0000 7fee2cd02640 1 --2- 172.21.15.46:0/2820576976 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fee28068dd0 0x7fee280691b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:28:59.915 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:59.913+0000 7fee2cd02640 1 -- 172.21.15.46:0/2820576976 >> 172.21.15.46:0/2820576976 conn(0x7fee280774a0 msgr2=0x7fee280778b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:28:59.915 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:59.913+0000 7fee2cd02640 1 -- 172.21.15.46:0/2820576976 shutdown_connections 2024-03-31T23:28:59.915 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:59.913+0000 7fee2cd02640 1 -- 172.21.15.46:0/2820576976 wait complete. 2024-03-31T23:28:59.917 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:59.914+0000 7fee2cd02640 1 Processor -- start 2024-03-31T23:28:59.917 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:59.914+0000 7fee2cd02640 1 -- start start 2024-03-31T23:28:59.918 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:59.914+0000 7fee2cd02640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fee28068dd0 0x7fee2819a030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:28:59.918 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:59.914+0000 7fee2cd02640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fee280696f0 0x7fee2819a570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:28:59.918 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:59.914+0000 7fee2cd02640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fee2810e450 0x7fee2819e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:28:59.918 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:59.915+0000 7fee2cd02640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fee28113bd0 con 0x7fee2810e450 2024-03-31T23:28:59.918 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:59.915+0000 7fee2cd02640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fee28113a50 con 0x7fee280696f0 2024-03-31T23:28:59.918 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:59.915+0000 7fee2cd02640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fee28113d50 con 0x7fee28068dd0 2024-03-31T23:28:59.918 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:59.915+0000 7fee26d9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fee2810e450 0x7fee2819e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:28:59.918 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:59.915+0000 7fee26d9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fee2810e450 0x7fee2819e8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:34482/0 (socket says 172.21.15.46:34482) 2024-03-31T23:28:59.918 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:59.915+0000 7fee2659c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fee28068dd0 0x7fee2819a030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:28:59.918 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:59.915+0000 7fee1ffff640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fee280696f0 0x7fee2819a570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:28:59.919 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:59.915+0000 7fee26d9d640 1 -- 172.21.15.46:0/3611058017 learned_addr learned my addr 172.21.15.46:0/3611058017 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:28:59.919 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:59.915+0000 7fee26d9d640 1 -- 172.21.15.46:0/3611058017 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fee28068dd0 msgr2=0x7fee2819a030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:28:59.919 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:59.915+0000 7fee26d9d640 1 --2- 172.21.15.46:0/3611058017 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fee28068dd0 0x7fee2819a030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:28:59.919 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:59.915+0000 7fee26d9d640 1 -- 172.21.15.46:0/3611058017 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fee280696f0 msgr2=0x7fee2819a570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:28:59.919 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:59.915+0000 7fee26d9d640 1 --2- 172.21.15.46:0/3611058017 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fee280696f0 0x7fee2819a570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:28:59.919 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:59.915+0000 7fee26d9d640 1 -- 172.21.15.46:0/3611058017 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee2819f020 con 0x7fee2810e450 2024-03-31T23:28:59.919 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:59.915+0000 7fee2659c640 1 --2- 172.21.15.46:0/3611058017 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fee28068dd0 0x7fee2819a030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:28:59.919 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:59.915+0000 7fee26d9d640 1 --2- 172.21.15.46:0/3611058017 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fee2810e450 0x7fee2819e8f0 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7fee2000eab0 tx=0x7fee20009f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:28:59.919 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:59.915+0000 7fee1f7fe640 1 -- 172.21.15.46:0/3611058017 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee2000ce40 con 0x7fee2810e450 2024-03-31T23:28:59.919 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:59.916+0000 7fee2cd02640 1 -- 172.21.15.46:0/3611058017 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fee2819f310 con 0x7fee2810e450 2024-03-31T23:28:59.919 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:59.916+0000 7fee1f7fe640 1 -- 172.21.15.46:0/3611058017 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fee20005020 con 0x7fee2810e450 2024-03-31T23:28:59.919 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:59.916+0000 7fee2cd02640 1 -- 172.21.15.46:0/3611058017 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fee28102ab0 con 0x7fee2810e450 2024-03-31T23:28:59.919 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:59.916+0000 7fee1f7fe640 1 -- 172.21.15.46:0/3611058017 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee20010690 con 0x7fee2810e450 2024-03-31T23:28:59.923 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:59.919+0000 7fee1f7fe640 1 -- 172.21.15.46:0/3611058017 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 25) ==== 137451+0+0 (secure 0 0 0) 0x7fee20004a40 con 0x7fee2810e450 2024-03-31T23:28:59.923 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:59.919+0000 7fee2f9c4740 1 Processor -- start 2024-03-31T23:28:59.923 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:59.919+0000 7fee2f9c4740 1 -- start start 2024-03-31T23:28:59.923 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:59.919+0000 7fee2f9c4740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a2532de6c0 0x55a2532deaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:28:59.923 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:59.919+0000 7fee2f9c4740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a2532defe0 0x55a2532e7870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:28:59.924 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:59.919+0000 7fee2f9c4740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a2532e7db0 0x55a2532ea1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:28:59.924 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:59.919+0000 7fee2f9c4740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55a2532418f0 con 0x55a2532defe0 2024-03-31T23:28:59.924 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:59.919+0000 7fee2f9c4740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55a2531ea8d0 con 0x55a2532de6c0 2024-03-31T23:28:59.924 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:59.919+0000 7fee2f9c4740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55a25323d990 con 0x55a2532e7db0 2024-03-31T23:28:59.924 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:59.920+0000 7fee2659c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a2532defe0 0x55a2532e7870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:28:59.924 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:59.920+0000 7fee1ffff640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a2532de6c0 0x55a2532deaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:28:59.924 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:59.920+0000 7fee1ffff640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a2532de6c0 0x55a2532deaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.46:42456/0 (socket says 172.21.15.46:42456) 2024-03-31T23:28:59.924 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:59.920+0000 7fee2659c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a2532defe0 0x55a2532e7870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:34490/0 (socket says 172.21.15.46:34490) 2024-03-31T23:28:59.924 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:59.920+0000 7fee1ffff640 1 -- 172.21.15.46:0/1206537328 learned_addr learned my addr 172.21.15.46:0/1206537328 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:28:59.924 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:59.920+0000 7fee1ffff640 1 --2- 172.21.15.46:0/1206537328 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a2532e7db0 0x55a2532ea1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:28:59.924 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:59.920+0000 7fee1f7fe640 1 --2- 172.21.15.46:0/3611058017 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fee140a26b0 0x7fee140a4b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:28:59.924 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:59.920+0000 7fee2659c640 1 -- 172.21.15.46:0/1206537328 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a2532e7db0 msgr2=0x55a2532ea1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:28:59.924 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:59.920+0000 7fee2659c640 1 --2- 172.21.15.46:0/1206537328 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a2532e7db0 0x55a2532ea1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:28:59.925 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:59.920+0000 7fee2659c640 1 -- 172.21.15.46:0/1206537328 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a2532de6c0 msgr2=0x55a2532deaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:28:59.925 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:59.920+0000 7fee2659c640 1 --2- 172.21.15.46:0/1206537328 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a2532de6c0 0x55a2532deaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:28:59.925 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:59.920+0000 7fee2659c640 1 -- 172.21.15.46:0/1206537328 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a252f56210 con 0x55a2532defe0 2024-03-31T23:28:59.925 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:59.920+0000 7fee26d9d640 1 --2- 172.21.15.46:0/3611058017 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fee140a26b0 0x7fee140a4b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:28:59.925 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:59.920+0000 7fee2659c640 1 --2- 172.21.15.46:0/1206537328 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a2532defe0 0x55a2532e7870 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 crypto rx=0x7fee18009fd0 tx=0x7fee180118d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ef56e204a4e415e server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:28:59.925 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:59.920+0000 7fee1f7fe640 1 -- 172.21.15.46:0/3611058017 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(119..119 src has 1..119) ==== 9554+0+0 (secure 0 0 0) 0x7fee200cf510 con 0x7fee2810e450 2024-03-31T23:28:59.925 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:59.921+0000 7fee1cff9640 1 -- 172.21.15.46:0/1206537328 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee1801e070 con 0x55a2532defe0 2024-03-31T23:28:59.925 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:59.921+0000 7fee1cff9640 1 -- 172.21.15.46:0/1206537328 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fee18004310 con 0x55a2532defe0 2024-03-31T23:28:59.925 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:59.921+0000 7fee1cff9640 1 -- 172.21.15.46:0/1206537328 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee1800f840 con 0x55a2532defe0 2024-03-31T23:28:59.925 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:59.922+0000 7fee2f9c4740 1 -- 172.21.15.46:0/1206537328 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a2532defe0 msgr2=0x55a2532e7870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:28:59.925 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:59.922+0000 7fee26d9d640 1 --2- 172.21.15.46:0/3611058017 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fee140a26b0 0x7fee140a4b70 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fee200cc9a0 tx=0x7fee2006d040 comp rx=0 tx=0).ready entity=mgr.15654 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:28:59.925 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:59.922+0000 7fee2f9c4740 1 --2- 172.21.15.46:0/1206537328 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a2532defe0 0x55a2532e7870 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 crypto rx=0x7fee18009fd0 tx=0x7fee180118d0 comp rx=0 tx=0).stop 2024-03-31T23:28:59.925 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:59.922+0000 7fee2f9c4740 1 -- 172.21.15.46:0/1206537328 shutdown_connections 2024-03-31T23:28:59.925 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:59.922+0000 7fee2f9c4740 1 --2- 172.21.15.46:0/1206537328 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a2532e7db0 0x55a2532ea1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:28:59.926 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:59.922+0000 7fee2f9c4740 1 --2- 172.21.15.46:0/1206537328 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a2532defe0 0x55a2532e7870 unknown :-1 s=CLOSED pgs=934 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:28:59.926 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:59.922+0000 7fee2f9c4740 1 --2- 172.21.15.46:0/1206537328 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a2532de6c0 0x55a2532deaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:28:59.926 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:59.922+0000 7fee2f9c4740 1 -- 172.21.15.46:0/1206537328 >> 172.21.15.46:0/1206537328 conn(0x55a2532d8fe0 msgr2=0x55a2532d93c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:28:59.926 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:59.922+0000 7fee2f9c4740 1 -- 172.21.15.46:0/1206537328 shutdown_connections 2024-03-31T23:28:59.926 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:59.922+0000 7fee2f9c4740 1 -- 172.21.15.46:0/1206537328 wait complete. 2024-03-31T23:28:59.926 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:59.922+0000 7fee2f9c4740 1 -- start start 2024-03-31T23:28:59.926 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:59.922+0000 7fee2f9c4740 1 Processor -- start 2024-03-31T23:28:59.930 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:59.923+0000 7fee2f9c4740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a2532de6c0 0x55a2532e4d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:28:59.930 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:59.923+0000 7fee2f9c4740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a2532defe0 0x55a2532e5290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:28:59.930 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:59.923+0000 7fee2f9c4740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a2532e7db0 0x55a2532e57d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:28:59.930 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:59.923+0000 7fee2f9c4740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55a2532ea9e0 con 0x55a2532defe0 2024-03-31T23:28:59.930 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:59.923+0000 7fee2f9c4740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55a2531dff80 con 0x55a2532de6c0 2024-03-31T23:28:59.930 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:59.923+0000 7fee2f9c4740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55a2532eab60 con 0x55a2532e7db0 2024-03-31T23:28:59.930 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:59.923+0000 7fee2f9c4740 10 client.0 _ll_register_callbacks cb 0x55a253238290 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T23:28:59.930 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:59.923+0000 7fee2659c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a2532defe0 0x55a2532e5290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:28:59.931 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:59.923+0000 7fee2659c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a2532defe0 0x55a2532e5290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:34506/0 (socket says 172.21.15.46:34506) 2024-03-31T23:28:59.931 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:59.923+0000 7fee2659c640 1 -- 172.21.15.46:0/1424483008 learned_addr learned my addr 172.21.15.46:0/1424483008 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:28:59.931 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:59.923+0000 7fee1ffff640 1 --2- 172.21.15.46:0/1424483008 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a2532de6c0 0x55a2532e4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:28:59.931 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:59.923+0000 7fee1ffff640 1 --2- 172.21.15.46:0/1424483008 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a2532e7db0 0x55a2532e57d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:28:59.931 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:59.923+0000 7fee2659c640 1 -- 172.21.15.46:0/1424483008 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a2532e7db0 msgr2=0x55a2532e57d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:28:59.931 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:59.923+0000 7fee2659c640 1 --2- 172.21.15.46:0/1424483008 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a2532e7db0 0x55a2532e57d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:28:59.931 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:59.923+0000 7fee2659c640 1 -- 172.21.15.46:0/1424483008 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a2532de6c0 msgr2=0x55a2532e4d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:28:59.931 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:59.923+0000 7fee2659c640 1 --2- 172.21.15.46:0/1424483008 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a2532de6c0 0x55a2532e4d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:28:59.931 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:59.923+0000 7fee2659c640 1 -- 172.21.15.46:0/1424483008 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a253022620 con 0x55a2532defe0 2024-03-31T23:28:59.931 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:59.924+0000 7fee2659c640 1 --2- 172.21.15.46:0/1424483008 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a2532defe0 0x55a2532e5290 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7fee18003f80 tx=0x7fee18005db0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=742655edb10f602f server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:28:59.931 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:59.924+0000 7fee0dffb640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-31T23:28:59.932 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:59.924+0000 7fee0dffb640 1 -- 172.21.15.46:0/1424483008 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee18027020 con 0x55a2532defe0 2024-03-31T23:28:59.932 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:59.924+0000 7fee0dffb640 1 -- 172.21.15.46:0/1424483008 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(119..119 src has 1..119) ==== 9554+0+0 (secure 0 0 0) 0x7fee18005040 con 0x55a2532defe0 2024-03-31T23:28:59.932 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:59.924+0000 7fee2f9c4740 1 -- 172.21.15.46:0/1424483008 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fee18004310 con 0x55a2532defe0 2024-03-31T23:28:59.932 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:59.924+0000 7fee0dffb640 1 -- 172.21.15.46:0/1424483008 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=45) ==== 24+0+0 (secure 0 0 0) 0x7fee18032050 con 0x55a2532defe0 2024-03-31T23:28:59.932 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:59.924+0000 7fee2f9c4740 10 client.17547 fetch_fsmap learned FSMap version 45 2024-03-31T23:28:59.932 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:59.924+0000 7fee2f9c4740 1 -- 172.21.15.46:0/1424483008 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=45,monmap=4+}) -- 0x55a2532f68b0 con 0x55a2532defe0 2024-03-31T23:28:59.933 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:59.924+0000 7fee0dffb640 1 -- 172.21.15.46:0/1424483008 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fee18004310 con 0x55a2532defe0 2024-03-31T23:28:59.933 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:59.924+0000 7fee0dffb640 1 -- 172.21.15.46:0/1424483008 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 45) ==== 8350+0+0 (secure 0 0 0) 0x7fee18004bd0 con 0x55a2532defe0 2024-03-31T23:28:59.933 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:59.924+0000 7fee2f9c4740 10 client.17547 fetch_fsmap finished waiting for FSMap version 45 2024-03-31T23:28:59.933 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:59.924+0000 7fee2f9c4740 10 client.17547 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-31T23:28:59.933 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:59.924+0000 7fee2f9c4740 20 client.17547 populate_metadata read hostname 'smithi046' 2024-03-31T23:28:59.933 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:59.925+0000 7fee2f9c4740 1 --2- 172.21.15.46:0/1424483008 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55a2532f7cb0 0x55a2532fa0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:28:59.933 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:59.925+0000 7fee2f9c4740 4 client.17547 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-31T23:28:59.933 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:59.925+0000 7fee2f9c4740 1 -- 172.21.15.46:0/1424483008 --> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a253255120 con 0x55a2532f7cb0 2024-03-31T23:28:59.933 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:59.925+0000 7fee1ffff640 1 --2- 172.21.15.46:0/1424483008 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55a2532f7cb0 0x55a2532fa0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:28:59.933 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:59.925+0000 7fee1ffff640 1 --2- 172.21.15.46:0/1424483008 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55a2532f7cb0 0x55a2532fa0a0 crc :-1 s=READY pgs=230 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3cbd56a7f213c62a server_cookie=5e3808a9e8a4f3a in_seq=0 out_seq=0 2024-03-31T23:28:59.934 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:59.925+0000 7fee0dffb640 10 client.17547 ms_handle_connect on v2:172.21.15.142:6838/2519143918 2024-03-31T23:28:59.934 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:59.926+0000 7fee0dffb640 1 -- 172.21.15.46:0/1424483008 <== mds.0 v2:172.21.15.142:6838/2519143918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55a253255120 con 0x55a2532f7cb0 2024-03-31T23:28:59.934 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:59.926+0000 7fee0dffb640 10 client.17547 handle_command_reply: tid=0 2024-03-31T23:28:59.934 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:59.926+0000 7fee2f9c4740 1 client.17547 shutdown 2024-03-31T23:28:59.934 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:59.926+0000 7fee2f9c4740 1 -- 172.21.15.46:0/1424483008 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a2532defe0 msgr2=0x55a2532e5290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:28:59.934 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:59.926+0000 7fee2f9c4740 1 --2- 172.21.15.46:0/1424483008 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a2532defe0 0x55a2532e5290 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7fee18003f80 tx=0x7fee18005db0 comp rx=0 tx=0).stop 2024-03-31T23:28:59.934 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:59.926+0000 7fee2f9c4740 1 -- 172.21.15.46:0/1424483008 shutdown_connections 2024-03-31T23:28:59.934 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:59.926+0000 7fee2f9c4740 1 --2- 172.21.15.46:0/1424483008 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55a2532f7cb0 0x55a2532fa0a0 crc :-1 s=READY pgs=230 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:28:59.934 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:59.926+0000 7fee2f9c4740 1 --2- 172.21.15.46:0/1424483008 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a2532e7db0 0x55a2532e57d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:28:59.934 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:59.926+0000 7fee2f9c4740 1 --2- 172.21.15.46:0/1424483008 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a2532defe0 0x55a2532e5290 unknown :-1 s=CLOSED pgs=935 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:28:59.935 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:59.926+0000 7fee2f9c4740 1 --2- 172.21.15.46:0/1424483008 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a2532de6c0 0x55a2532e4d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:28:59.935 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:59.926+0000 7fee0dffb640 0 client.17547 ms_handle_reset on v2:172.21.15.142:6838/2519143918 2024-03-31T23:28:59.935 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:59.926+0000 7fee2f9c4740 1 -- 172.21.15.46:0/1424483008 >> 172.21.15.46:0/1424483008 conn(0x55a2532d8fe0 msgr2=0x55a2532dad30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:28:59.935 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:59.926+0000 7fee2f9c4740 1 -- 172.21.15.46:0/1424483008 shutdown_connections 2024-03-31T23:28:59.935 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:59.926+0000 7fee2f9c4740 1 -- 172.21.15.46:0/1424483008 wait complete. 2024-03-31T23:28:59.935 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:59.926+0000 7fee2f9c4740 20 client.17547 trim_cache size 0 max 16384 2024-03-31T23:28:59.951 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:59.949+0000 7fee2f9c4740 1 Processor -- start 2024-03-31T23:28:59.951 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:59.949+0000 7fee2f9c4740 1 -- start start 2024-03-31T23:28:59.952 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:59.949+0000 7fee2f9c4740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a2532de6c0 0x55a2532e7480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:28:59.952 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:59.949+0000 7fee2f9c4740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a2532defe0 0x55a2532f37e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:28:59.952 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:59.949+0000 7fee2f9c4740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a2532e7db0 0x55a2532ff8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:28:59.952 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:59.949+0000 7fee2f9c4740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fee08012980 con 0x55a2532defe0 2024-03-31T23:28:59.952 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:59.949+0000 7fee2f9c4740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fee08010220 con 0x55a2532de6c0 2024-03-31T23:28:59.952 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:59.949+0000 7fee2f9c4740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fee0800e090 con 0x55a2532e7db0 2024-03-31T23:28:59.952 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:59.949+0000 7fee1ffff640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a2532de6c0 0x55a2532e7480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:28:59.952 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:59.949+0000 7fee2659c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a2532defe0 0x55a2532f37e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:28:59.952 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:59.949+0000 7fee1ffff640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a2532de6c0 0x55a2532e7480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.46:42460/0 (socket says 172.21.15.46:42460) 2024-03-31T23:28:59.952 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:59.949+0000 7fee2659c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a2532defe0 0x55a2532f37e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:34520/0 (socket says 172.21.15.46:34520) 2024-03-31T23:28:59.952 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:59.949+0000 7fee1ffff640 1 -- 172.21.15.46:0/1380221207 learned_addr learned my addr 172.21.15.46:0/1380221207 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:28:59.952 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:59.949+0000 7fee1ffff640 1 --2- 172.21.15.46:0/1380221207 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a2532e7db0 0x55a2532ff8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:28:59.953 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:59.949+0000 7fee1ffff640 1 -- 172.21.15.46:0/1380221207 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a2532e7db0 msgr2=0x55a2532ff8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:28:59.953 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:59.949+0000 7fee1ffff640 1 --2- 172.21.15.46:0/1380221207 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a2532e7db0 0x55a2532ff8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:28:59.953 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:59.949+0000 7fee1ffff640 1 -- 172.21.15.46:0/1380221207 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a2532defe0 msgr2=0x55a2532f37e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:28:59.953 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:59.949+0000 7fee1ffff640 1 --2- 172.21.15.46:0/1380221207 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a2532defe0 0x55a2532f37e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:28:59.953 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:59.949+0000 7fee1ffff640 1 -- 172.21.15.46:0/1380221207 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a2532fa410 con 0x55a2532de6c0 2024-03-31T23:28:59.954 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:59.949+0000 7fee2659c640 1 --2- 172.21.15.46:0/1380221207 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a2532defe0 0x55a2532f37e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:28:59.954 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:59.950+0000 7fee1ffff640 1 --2- 172.21.15.46:0/1380221207 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a2532de6c0 0x55a2532e7480 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fee10006010 tx=0x7fee10009680 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9a24514fa41ddac8 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:28:59.954 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:59.950+0000 7fee0ffff640 1 -- 172.21.15.46:0/1380221207 <== mon.2 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee10004380 con 0x55a2532de6c0 2024-03-31T23:28:59.954 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:59.950+0000 7fee0ffff640 1 -- 172.21.15.46:0/1380221207 <== mon.2 v2:172.21.15.142:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fee10005050 con 0x55a2532de6c0 2024-03-31T23:28:59.954 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:59.950+0000 7fee0ffff640 1 -- 172.21.15.46:0/1380221207 <== mon.2 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee1002cb70 con 0x55a2532de6c0 2024-03-31T23:28:59.958 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:59.951+0000 7fee2f9c4740 1 -- 172.21.15.46:0/1380221207 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a2532de6c0 msgr2=0x55a2532e7480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:28:59.958 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:59.951+0000 7fee2f9c4740 1 --2- 172.21.15.46:0/1380221207 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a2532de6c0 0x55a2532e7480 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fee10006010 tx=0x7fee10009680 comp rx=0 tx=0).stop 2024-03-31T23:28:59.958 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:59.951+0000 7fee2f9c4740 1 -- 172.21.15.46:0/1380221207 shutdown_connections 2024-03-31T23:28:59.958 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:59.951+0000 7fee2f9c4740 1 --2- 172.21.15.46:0/1380221207 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a2532e7db0 0x55a2532ff8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:28:59.958 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:59.951+0000 7fee2f9c4740 1 --2- 172.21.15.46:0/1380221207 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a2532defe0 0x55a2532f37e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:28:59.958 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:59.951+0000 7fee2f9c4740 1 --2- 172.21.15.46:0/1380221207 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a2532de6c0 0x55a2532e7480 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:28:59.958 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:59.951+0000 7fee2f9c4740 1 -- 172.21.15.46:0/1380221207 >> 172.21.15.46:0/1380221207 conn(0x55a2532d8fe0 msgr2=0x55a2532d93c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:28:59.958 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:59.951+0000 7fee2f9c4740 1 -- 172.21.15.46:0/1380221207 shutdown_connections 2024-03-31T23:28:59.958 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:59.951+0000 7fee2f9c4740 1 -- 172.21.15.46:0/1380221207 wait complete. 2024-03-31T23:28:59.958 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:59.951+0000 7fee2f9c4740 1 -- start start 2024-03-31T23:28:59.958 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:59.952+0000 7fee2f9c4740 1 Processor -- start 2024-03-31T23:28:59.959 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:59.952+0000 7fee2f9c4740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a2532de6c0 0x55a2532f01a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:28:59.959 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:59.952+0000 7fee2f9c4740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a2532defe0 0x55a2532eb4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:28:59.959 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:59.952+0000 7fee2f9c4740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a2532e7db0 0x55a2532eba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:28:59.959 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:59.952+0000 7fee2f9c4740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fee0800e6e0 con 0x55a2532de6c0 2024-03-31T23:28:59.959 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:59.952+0000 7fee2f9c4740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fee0800fb50 con 0x55a2532defe0 2024-03-31T23:28:59.959 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:59.952+0000 7fee1ffff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a2532de6c0 0x55a2532f01a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:28:59.959 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:59.952+0000 7fee2f9c4740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fee0800edb0 con 0x55a2532e7db0 2024-03-31T23:28:59.959 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:59.952+0000 7fee2f9c4740 10 client.0 _ll_register_callbacks cb 0x7fee1801c070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T23:28:59.959 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:59.952+0000 7fee1ffff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a2532de6c0 0x55a2532f01a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:34534/0 (socket says 172.21.15.46:34534) 2024-03-31T23:28:59.959 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:59.952+0000 7fee1ffff640 1 -- 172.21.15.46:0/2801267879 learned_addr learned my addr 172.21.15.46:0/2801267879 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:28:59.959 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:59.953+0000 7fee1ffff640 1 -- 172.21.15.46:0/2801267879 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a2532e7db0 msgr2=0x55a2532eba60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-31T23:28:59.959 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:59.953+0000 7fee1ffff640 1 --2- 172.21.15.46:0/2801267879 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a2532e7db0 0x55a2532eba60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:28:59.959 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:59.953+0000 7fee1ffff640 1 -- 172.21.15.46:0/2801267879 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a2532defe0 msgr2=0x55a2532eb4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:28:59.960 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:59.953+0000 7fee1ffff640 1 --2- 172.21.15.46:0/2801267879 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a2532defe0 0x55a2532eb4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:28:59.960 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:59.953+0000 7fee1ffff640 1 -- 172.21.15.46:0/2801267879 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a2532ec290 con 0x55a2532de6c0 2024-03-31T23:28:59.960 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:59.953+0000 7fee1ffff640 1 --2- 172.21.15.46:0/2801267879 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a2532de6c0 0x55a2532f01a0 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7fee1002c820 tx=0x7fee100041c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4555d72e77ee4ca9 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:28:59.960 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:59.953+0000 7fee0dffb640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-31T23:28:59.960 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:59.953+0000 7fee0dffb640 1 -- 172.21.15.46:0/2801267879 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee10063060 con 0x55a2532de6c0 2024-03-31T23:28:59.960 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:59.953+0000 7fee0dffb640 1 -- 172.21.15.46:0/2801267879 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(119..119 src has 1..119) ==== 9554+0+0 (secure 0 0 0) 0x7fee10004380 con 0x55a2532de6c0 2024-03-31T23:28:59.960 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:59.953+0000 7fee2f9c4740 1 -- 172.21.15.46:0/2801267879 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fee10005050 con 0x55a2532de6c0 2024-03-31T23:28:59.960 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:59.954+0000 7fee0dffb640 1 -- 172.21.15.46:0/2801267879 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=45) ==== 24+0+0 (secure 0 0 0) 0x7fee1006a050 con 0x55a2532de6c0 2024-03-31T23:28:59.960 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:59.954+0000 7fee2f9c4740 10 client.17553 fetch_fsmap learned FSMap version 45 2024-03-31T23:28:59.960 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:59.954+0000 7fee2f9c4740 1 -- 172.21.15.46:0/2801267879 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=45,monmap=4+}) -- 0x55a25335dfd0 con 0x55a2532de6c0 2024-03-31T23:28:59.960 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:59.954+0000 7fee0dffb640 1 -- 172.21.15.46:0/2801267879 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fee10005200 con 0x55a2532de6c0 2024-03-31T23:28:59.960 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:59.954+0000 7fee0dffb640 1 -- 172.21.15.46:0/2801267879 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 45) ==== 8350+0+0 (secure 0 0 0) 0x7fee1002cc20 con 0x55a2532de6c0 2024-03-31T23:28:59.960 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:59.954+0000 7fee2f9c4740 10 client.17553 fetch_fsmap finished waiting for FSMap version 45 2024-03-31T23:28:59.960 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:59.954+0000 7fee2f9c4740 10 client.17553 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-31T23:28:59.961 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:59.954+0000 7fee2f9c4740 20 client.17553 populate_metadata read hostname 'smithi046' 2024-03-31T23:28:59.961 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:59.954+0000 7fee2f9c4740 1 --2- 172.21.15.46:0/2801267879 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55a25335e5c0 0x55a2533609e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:28:59.961 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:59.954+0000 7fee2f9c4740 4 client.17553 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-31T23:28:59.961 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:59.954+0000 7fee2f9c4740 1 -- 172.21.15.46:0/2801267879 --> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] -- command(tid 0: {"prefix": "flush journal"}) -- 0x7fee10005050 con 0x55a25335e5c0 2024-03-31T23:28:59.961 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:59.954+0000 7fee2659c640 1 --2- 172.21.15.46:0/2801267879 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55a25335e5c0 0x55a2533609e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:28:59.961 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:59.955+0000 7fee2659c640 1 --2- 172.21.15.46:0/2801267879 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55a25335e5c0 0x55a2533609e0 crc :-1 s=READY pgs=231 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=910ad07f75d3e3ad server_cookie=2828c966fada9944 in_seq=0 out_seq=0 2024-03-31T23:28:59.961 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:28:59.955+0000 7fee0dffb640 10 client.17553 ms_handle_connect on v2:172.21.15.142:6838/2519143918 2024-03-31T23:29:00.123 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:00.122+0000 7fee0dffb640 1 -- 172.21.15.46:0/2801267879 <== mds.0 v2:172.21.15.142:6838/2519143918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x7fee10005050 con 0x55a25335e5c0 2024-03-31T23:29:00.124 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:00.122+0000 7fee0dffb640 10 client.17553 handle_command_reply: tid=0 2024-03-31T23:29:00.124 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:00.122+0000 7fee2f9c4740 1 client.17553 shutdown 2024-03-31T23:29:00.124 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:00.122+0000 7fee2f9c4740 1 -- 172.21.15.46:0/2801267879 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a2532de6c0 msgr2=0x55a2532f01a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:29:00.124 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:00.122+0000 7fee2f9c4740 1 --2- 172.21.15.46:0/2801267879 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a2532de6c0 0x55a2532f01a0 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7fee1002c820 tx=0x7fee100041c0 comp rx=0 tx=0).stop 2024-03-31T23:29:00.124 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:00.122+0000 7fee2f9c4740 1 -- 172.21.15.46:0/2801267879 shutdown_connections 2024-03-31T23:29:00.124 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:00.122+0000 7fee2f9c4740 1 --2- 172.21.15.46:0/2801267879 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55a25335e5c0 0x55a2533609e0 crc :-1 s=READY pgs=231 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:29:00.124 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:00.122+0000 7fee2f9c4740 1 --2- 172.21.15.46:0/2801267879 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a2532e7db0 0x55a2532eba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:29:00.124 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:00.122+0000 7fee2f9c4740 1 --2- 172.21.15.46:0/2801267879 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a2532defe0 0x55a2532eb4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:29:00.124 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:00.122+0000 7fee2f9c4740 1 --2- 172.21.15.46:0/2801267879 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a2532de6c0 0x55a2532f01a0 unknown :-1 s=CLOSED pgs=936 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:29:00.124 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:00.122+0000 7fee2f9c4740 1 -- 172.21.15.46:0/2801267879 >> 172.21.15.46:0/2801267879 conn(0x55a2532d8fe0 msgr2=0x55a2532e4cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:29:00.124 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:00.122+0000 7fee0dffb640 0 client.17553 ms_handle_reset on v2:172.21.15.142:6838/2519143918 2024-03-31T23:29:00.124 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:00.122+0000 7fee2f9c4740 1 -- 172.21.15.46:0/2801267879 shutdown_connections 2024-03-31T23:29:00.124 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:00.122+0000 7fee2f9c4740 1 -- 172.21.15.46:0/2801267879 wait complete. 2024-03-31T23:29:00.125 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:00.122+0000 7fee2f9c4740 20 client.17553 trim_cache size 0 max 16384 2024-03-31T23:29:00.125 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout:{ 2024-03-31T23:29:00.125 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout: "message": "", 2024-03-31T23:29:00.125 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout: "return_code": 0 2024-03-31T23:29:00.125 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout:} 2024-03-31T23:29:00.126 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:00.123+0000 7fee2cd02640 1 -- 172.21.15.46:0/3611058017 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fee140a26b0 msgr2=0x7fee140a4b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:29:00.126 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:00.123+0000 7fee2cd02640 1 --2- 172.21.15.46:0/3611058017 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fee140a26b0 0x7fee140a4b70 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fee200cc9a0 tx=0x7fee2006d040 comp rx=0 tx=0).stop 2024-03-31T23:29:00.126 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:00.123+0000 7fee2cd02640 1 -- 172.21.15.46:0/3611058017 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fee2810e450 msgr2=0x7fee2819e8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:29:00.126 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:00.123+0000 7fee2cd02640 1 --2- 172.21.15.46:0/3611058017 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fee2810e450 0x7fee2819e8f0 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7fee2000eab0 tx=0x7fee20009f80 comp rx=0 tx=0).stop 2024-03-31T23:29:00.127 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:00.124+0000 7fee2cd02640 1 -- 172.21.15.46:0/3611058017 shutdown_connections 2024-03-31T23:29:00.127 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:00.124+0000 7fee2cd02640 1 --2- 172.21.15.46:0/3611058017 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fee140a26b0 0x7fee140a4b70 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:29:00.127 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:00.124+0000 7fee2cd02640 1 --2- 172.21.15.46:0/3611058017 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fee2810e450 0x7fee2819e8f0 unknown :-1 s=CLOSED pgs=933 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:29:00.127 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:00.124+0000 7fee2cd02640 1 --2- 172.21.15.46:0/3611058017 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fee280696f0 0x7fee2819a570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:29:00.127 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:00.124+0000 7fee2cd02640 1 --2- 172.21.15.46:0/3611058017 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fee28068dd0 0x7fee2819a030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:29:00.127 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:00.124+0000 7fee2cd02640 1 -- 172.21.15.46:0/3611058017 >> 172.21.15.46:0/3611058017 conn(0x7fee280774a0 msgr2=0x7fee2810f800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:29:00.127 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:00.124+0000 7fee2cd02640 1 -- 172.21.15.46:0/3611058017 shutdown_connections 2024-03-31T23:29:00.127 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:00.124+0000 7fee2cd02640 1 -- 172.21.15.46:0/3611058017 wait complete. 2024-03-31T23:29:00.288 INFO:tasks.workunit.client.0.smithi046.stdout: 10 39710 8.60 MB/sec execute 125 sec latency 143.468 ms 2024-03-31T23:29:00.796 INFO:journalctl@ceph.mds.i.smithi162.stdout:Mar 31 23:29:00 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mds-i[67378]: ignoring --setuser ceph since I am not root 2024-03-31T23:29:00.796 INFO:journalctl@ceph.mds.i.smithi162.stdout:Mar 31 23:29:00 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mds-i[67378]: ignoring --setgroup ceph since I am not root 2024-03-31T23:29:00.796 INFO:journalctl@ceph.mds.i.smithi162.stdout:Mar 31 23:29:00 smithi162 ceph-mds[67382]: -- 172.21.15.162:0/1705822213 <== mon.1 v2:172.21.15.162:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5618cbd2da40 con 0x5618cbd01000 2024-03-31T23:29:00.796 INFO:journalctl@ceph.mds.i.smithi162.stdout:Mar 31 23:29:00 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mds-i[67378]: starting mds.i at 2024-03-31T23:29:00.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:29:00 smithi162 ceph-mon[36237]: pgmap v544: 129 pgs: 129 active+clean; 704 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 5.9 MiB/s wr, 395 op/s 2024-03-31T23:29:00.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:29:00 smithi162 ceph-mon[36237]: daemon mds.i restarted 2024-03-31T23:29:00.796 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:29:00 smithi162 ceph-mon[36237]: osdmap e120: 12 total, 12 up, 12 in 2024-03-31T23:29:00.797 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:29:00 smithi162 ceph-mon[36237]: mds.? [v2:172.21.15.162:6838/1588535467,v1:172.21.15.162:6839/1588535467] up:boot 2024-03-31T23:29:00.797 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:29:00 smithi162 ceph-mon[36237]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-03-31T23:29:00.797 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:29:00 smithi162 ceph-mon[36237]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-31T23:29:01.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:29:00 smithi046 ceph-mon[32498]: pgmap v544: 129 pgs: 129 active+clean; 704 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 5.9 MiB/s wr, 395 op/s 2024-03-31T23:29:01.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:29:00 smithi046 ceph-mon[32498]: daemon mds.i restarted 2024-03-31T23:29:01.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:29:00 smithi046 ceph-mon[32498]: osdmap e120: 12 total, 12 up, 12 in 2024-03-31T23:29:01.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:29:00 smithi046 ceph-mon[32498]: mds.? [v2:172.21.15.162:6838/1588535467,v1:172.21.15.162:6839/1588535467] up:boot 2024-03-31T23:29:01.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:29:00 smithi046 ceph-mon[32498]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-03-31T23:29:01.113 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:29:00 smithi046 ceph-mon[32498]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-31T23:29:01.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:29:00 smithi142 ceph-mon[35271]: pgmap v544: 129 pgs: 129 active+clean; 704 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 5.9 MiB/s wr, 395 op/s 2024-03-31T23:29:01.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:29:00 smithi142 ceph-mon[35271]: daemon mds.i restarted 2024-03-31T23:29:01.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:29:00 smithi142 ceph-mon[35271]: osdmap e120: 12 total, 12 up, 12 in 2024-03-31T23:29:01.194 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:29:00 smithi142 ceph-mon[35271]: mds.? [v2:172.21.15.162:6838/1588535467,v1:172.21.15.162:6839/1588535467] up:boot 2024-03-31T23:29:01.194 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:29:00 smithi142 ceph-mon[35271]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-03-31T23:29:01.194 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:29:00 smithi142 ceph-mon[35271]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-31T23:29:01.288 INFO:tasks.workunit.client.0.smithi046.stdout: 10 39848 8.63 MB/sec execute 126 sec latency 381.442 ms 2024-03-31T23:29:02.288 INFO:tasks.workunit.client.0.smithi046.stdout: 10 40051 8.67 MB/sec execute 127 sec latency 69.273 ms 2024-03-31T23:29:03.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:29:02 smithi162 ceph-mon[36237]: pgmap v546: 129 pgs: 129 active+clean; 704 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 4.4 MiB/s wr, 281 op/s 2024-03-31T23:29:03.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:29:02 smithi046 ceph-mon[32498]: pgmap v546: 129 pgs: 129 active+clean; 704 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 4.4 MiB/s wr, 281 op/s 2024-03-31T23:29:03.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:29:02 smithi142 ceph-mon[35271]: pgmap v546: 129 pgs: 129 active+clean; 704 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 4.4 MiB/s wr, 281 op/s 2024-03-31T23:29:03.288 INFO:tasks.workunit.client.0.smithi046.stdout: 10 40219 8.67 MB/sec execute 128 sec latency 107.951 ms 2024-03-31T23:29:04.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:29:03 smithi162 ceph-mon[36237]: osdmap e121: 12 total, 12 up, 12 in 2024-03-31T23:29:04.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:29:03 smithi046 ceph-mon[32498]: osdmap e121: 12 total, 12 up, 12 in 2024-03-31T23:29:04.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:29:03 smithi142 ceph-mon[35271]: osdmap e121: 12 total, 12 up, 12 in 2024-03-31T23:29:04.288 INFO:tasks.workunit.client.0.smithi046.stdout: 10 40374 8.67 MB/sec execute 129 sec latency 88.010 ms 2024-03-31T23:29:05.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:29:04 smithi162 ceph-mon[36237]: pgmap v548: 129 pgs: 129 active+clean; 704 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 966 KiB/s rd, 2.8 MiB/s wr, 166 op/s 2024-03-31T23:29:05.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:29:04 smithi162 ceph-mon[36237]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-03-31T23:29:05.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:29:04 smithi046 ceph-mon[32498]: pgmap v548: 129 pgs: 129 active+clean; 704 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 966 KiB/s rd, 2.8 MiB/s wr, 166 op/s 2024-03-31T23:29:05.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:29:04 smithi046 ceph-mon[32498]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-03-31T23:29:05.195 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:29:04 smithi142 ceph-mon[35271]: pgmap v548: 129 pgs: 129 active+clean; 704 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 966 KiB/s rd, 2.8 MiB/s wr, 166 op/s 2024-03-31T23:29:05.195 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:29:04 smithi142 ceph-mon[35271]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-03-31T23:29:05.288 INFO:tasks.workunit.client.0.smithi046.stdout: 10 40525 8.68 MB/sec execute 130 sec latency 65.914 ms 2024-03-31T23:29:06.288 INFO:tasks.workunit.client.0.smithi046.stdout: 10 40686 8.69 MB/sec execute 131 sec latency 66.597 ms 2024-03-31T23:29:07.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:29:06 smithi162 ceph-mon[36237]: pgmap v549: 129 pgs: 129 active+clean; 717 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 6.7 MiB/s wr, 429 op/s 2024-03-31T23:29:07.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:29:06 smithi046 ceph-mon[32498]: pgmap v549: 129 pgs: 129 active+clean; 717 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 6.7 MiB/s wr, 429 op/s 2024-03-31T23:29:07.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:29:06 smithi142 ceph-mon[35271]: pgmap v549: 129 pgs: 129 active+clean; 717 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 6.7 MiB/s wr, 429 op/s 2024-03-31T23:29:07.288 INFO:tasks.workunit.client.0.smithi046.stdout: 10 40837 8.68 MB/sec execute 132 sec latency 85.458 ms 2024-03-31T23:29:08.288 INFO:tasks.workunit.client.0.smithi046.stdout: 10 40988 8.67 MB/sec execute 133 sec latency 69.871 ms 2024-03-31T23:29:09.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:29:08 smithi162 ceph-mon[36237]: pgmap v550: 129 pgs: 129 active+clean; 717 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 3.9 MiB/s wr, 263 op/s 2024-03-31T23:29:09.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:29:08 smithi046 ceph-mon[32498]: pgmap v550: 129 pgs: 129 active+clean; 717 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 3.9 MiB/s wr, 263 op/s 2024-03-31T23:29:09.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:29:08 smithi142 ceph-mon[35271]: pgmap v550: 129 pgs: 129 active+clean; 717 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 3.9 MiB/s wr, 263 op/s 2024-03-31T23:29:09.288 INFO:tasks.workunit.client.0.smithi046.stdout: 10 41134 8.65 MB/sec execute 134 sec latency 67.385 ms 2024-03-31T23:29:10.288 INFO:tasks.workunit.client.0.smithi046.stdout: 10 41298 8.66 MB/sec execute 135 sec latency 88.707 ms 2024-03-31T23:29:11.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:29:10 smithi162 ceph-mon[36237]: pgmap v551: 129 pgs: 129 active+clean; 736 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 7.3 MiB/s wr, 416 op/s 2024-03-31T23:29:11.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:29:10 smithi162 ceph-mon[36237]: osdmap e122: 12 total, 12 up, 12 in 2024-03-31T23:29:11.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:29:10 smithi046 ceph-mon[32498]: pgmap v551: 129 pgs: 129 active+clean; 736 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 7.3 MiB/s wr, 416 op/s 2024-03-31T23:29:11.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:29:10 smithi046 ceph-mon[32498]: osdmap e122: 12 total, 12 up, 12 in 2024-03-31T23:29:11.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:29:10 smithi142 ceph-mon[35271]: pgmap v551: 129 pgs: 129 active+clean; 736 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 7.3 MiB/s wr, 416 op/s 2024-03-31T23:29:11.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:29:10 smithi142 ceph-mon[35271]: osdmap e122: 12 total, 12 up, 12 in 2024-03-31T23:29:11.288 INFO:tasks.workunit.client.0.smithi046.stdout: 10 41397 8.61 MB/sec execute 136 sec latency 289.227 ms 2024-03-31T23:29:12.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:29:11 smithi162 ceph-mon[36237]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-31T23:29:12.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:29:11 smithi046 ceph-mon[32498]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-31T23:29:12.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:29:11 smithi142 ceph-mon[35271]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-31T23:29:12.288 INFO:tasks.workunit.client.0.smithi046.stdout: 10 41555 8.60 MB/sec execute 137 sec latency 101.769 ms 2024-03-31T23:29:13.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:29:12 smithi162 ceph-mon[36237]: pgmap v553: 129 pgs: 129 active+clean; 736 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 7.4 MiB/s wr, 421 op/s 2024-03-31T23:29:13.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:29:12 smithi046 ceph-mon[32498]: pgmap v553: 129 pgs: 129 active+clean; 736 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 7.4 MiB/s wr, 421 op/s 2024-03-31T23:29:13.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:29:12 smithi142 ceph-mon[35271]: pgmap v553: 129 pgs: 129 active+clean; 736 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 7.4 MiB/s wr, 421 op/s 2024-03-31T23:29:13.260 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:13.256+0000 7f2c7ab8c640 1 -- 172.21.15.46:0/68028593 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2c7410bd00 msgr2=0x7f2c7410e1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:29:13.260 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:13.256+0000 7f2c7ab8c640 1 --2- 172.21.15.46:0/68028593 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2c7410bd00 0x7f2c7410e1a0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f2c6c009860 tx=0x7f2c6c02bca0 comp rx=0 tx=0).stop 2024-03-31T23:29:13.260 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:13.256+0000 7f2c7ab8c640 1 -- 172.21.15.46:0/68028593 shutdown_connections 2024-03-31T23:29:13.260 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:13.256+0000 7f2c7ab8c640 1 --2- 172.21.15.46:0/68028593 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2c7410bd00 0x7f2c7410e1a0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:29:13.260 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:13.256+0000 7f2c7ab8c640 1 --2- 172.21.15.46:0/68028593 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2c741032e0 0x7f2c7410b7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:29:13.260 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:13.256+0000 7f2c7ab8c640 1 --2- 172.21.15.46:0/68028593 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2c741029c0 0x7f2c74102da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:29:13.260 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:13.256+0000 7f2c7ab8c640 1 -- 172.21.15.46:0/68028593 >> 172.21.15.46:0/68028593 conn(0x7f2c74077770 msgr2=0x7f2c74077b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:29:13.260 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:13.257+0000 7f2c7ab8c640 1 -- 172.21.15.46:0/68028593 shutdown_connections 2024-03-31T23:29:13.260 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:13.257+0000 7f2c7ab8c640 1 -- 172.21.15.46:0/68028593 wait complete. 2024-03-31T23:29:13.261 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:13.257+0000 7f2c7ab8c640 1 Processor -- start 2024-03-31T23:29:13.261 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:13.257+0000 7f2c7ab8c640 1 -- start start 2024-03-31T23:29:13.261 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:13.257+0000 7f2c7ab8c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2c741029c0 0x7f2c7419a0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:29:13.261 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:13.258+0000 7f2c7ab8c640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2c741032e0 0x7f2c7419a5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:29:13.261 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:13.258+0000 7f2c7ab8c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2c7410bd00 0x7f2c7419e970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:29:13.261 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:13.258+0000 7f2c7ab8c640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f2c741119b0 con 0x7f2c741029c0 2024-03-31T23:29:13.261 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:13.258+0000 7f2c7ab8c640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f2c74111830 con 0x7f2c741032e0 2024-03-31T23:29:13.261 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:13.258+0000 7f2c7ab8c640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f2c74111b30 con 0x7f2c7410bd00 2024-03-31T23:29:13.261 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:13.258+0000 7f2c78928640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2c741029c0 0x7f2c7419a0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:29:13.261 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:13.258+0000 7f2c78928640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2c741029c0 0x7f2c7419a0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:35164/0 (socket says 172.21.15.46:35164) 2024-03-31T23:29:13.261 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:13.258+0000 7f2c78928640 1 -- 172.21.15.46:0/2526803272 learned_addr learned my addr 172.21.15.46:0/2526803272 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:29:13.261 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:13.258+0000 7f2c73fff640 1 --2- 172.21.15.46:0/2526803272 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2c741032e0 0x7f2c7419a5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:29:13.261 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:13.258+0000 7f2c79129640 1 --2- 172.21.15.46:0/2526803272 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2c7410bd00 0x7f2c7419e970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:29:13.261 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:13.258+0000 7f2c78928640 1 -- 172.21.15.46:0/2526803272 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2c7410bd00 msgr2=0x7f2c7419e970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:29:13.262 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:13.258+0000 7f2c78928640 1 --2- 172.21.15.46:0/2526803272 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2c7410bd00 0x7f2c7419e970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:29:13.262 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:13.258+0000 7f2c78928640 1 -- 172.21.15.46:0/2526803272 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2c741032e0 msgr2=0x7f2c7419a5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:29:13.262 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:13.258+0000 7f2c78928640 1 --2- 172.21.15.46:0/2526803272 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2c741032e0 0x7f2c7419a5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:29:13.262 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:13.258+0000 7f2c78928640 1 -- 172.21.15.46:0/2526803272 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2c7419f0a0 con 0x7f2c741029c0 2024-03-31T23:29:13.262 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:13.258+0000 7f2c73fff640 1 --2- 172.21.15.46:0/2526803272 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2c741032e0 0x7f2c7419a5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:29:13.262 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:13.258+0000 7f2c78928640 1 --2- 172.21.15.46:0/2526803272 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2c741029c0 0x7f2c7419a0b0 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x7f2c6400ebc0 tx=0x7f2c6400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:29:13.262 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:13.259+0000 7f2c71ffb640 1 -- 172.21.15.46:0/2526803272 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c64010430 con 0x7f2c741029c0 2024-03-31T23:29:13.262 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:13.259+0000 7f2c7ab8c640 1 -- 172.21.15.46:0/2526803272 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2c7419f390 con 0x7f2c741029c0 2024-03-31T23:29:13.262 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:13.259+0000 7f2c71ffb640 1 -- 172.21.15.46:0/2526803272 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f2c64010a80 con 0x7f2c741029c0 2024-03-31T23:29:13.262 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:13.259+0000 7f2c71ffb640 1 -- 172.21.15.46:0/2526803272 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c64018a50 con 0x7f2c741029c0 2024-03-31T23:29:13.262 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:13.259+0000 7f2c7ab8c640 1 -- 172.21.15.46:0/2526803272 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2c741a6c20 con 0x7f2c741029c0 2024-03-31T23:29:13.263 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:13.262+0000 7f2c71ffb640 1 -- 172.21.15.46:0/2526803272 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 25) ==== 137451+0+0 (secure 0 0 0) 0x7f2c64018bf0 con 0x7f2c741029c0 2024-03-31T23:29:13.265 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:13.262+0000 7f2c7d771740 1 Processor -- start 2024-03-31T23:29:13.265 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:13.262+0000 7f2c7d771740 1 -- start start 2024-03-31T23:29:13.265 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:13.263+0000 7f2c7d771740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5557443386c0 0x555744338aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:29:13.265 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:13.263+0000 7f2c7d771740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555744338fe0 0x555744341870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:29:13.265 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:13.263+0000 7f2c7d771740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x555744341db0 0x5557443441a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:29:13.265 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:13.263+0000 7f2c7d771740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55574429b8f0 con 0x555744338fe0 2024-03-31T23:29:13.265 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:13.263+0000 7f2c7d771740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x5557442448d0 con 0x555744341db0 2024-03-31T23:29:13.266 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:13.263+0000 7f2c7d771740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x555744297990 con 0x5557443386c0 2024-03-31T23:29:13.266 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:13.263+0000 7f2c71ffb640 1 --2- 172.21.15.46:0/2526803272 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f2c600a26b0 0x7f2c600a4b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:29:13.266 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:13.263+0000 7f2c78928640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555744338fe0 0x555744341870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:29:13.266 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:13.263+0000 7f2c78928640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555744338fe0 0x555744341870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:35176/0 (socket says 172.21.15.46:35176) 2024-03-31T23:29:13.266 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:13.263+0000 7f2c78928640 1 -- 172.21.15.46:0/754170612 learned_addr learned my addr 172.21.15.46:0/754170612 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:29:13.266 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:13.263+0000 7f2c79129640 1 --2- 172.21.15.46:0/754170612 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5557443386c0 0x555744338aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:29:13.266 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:13.263+0000 7f2c79129640 1 --2- 172.21.15.46:0/2526803272 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f2c600a26b0 0x7f2c600a4b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:29:13.266 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:13.263+0000 7f2c73fff640 1 --2- 172.21.15.46:0/754170612 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x555744341db0 0x5557443441a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:29:13.266 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:13.263+0000 7f2c78928640 1 -- 172.21.15.46:0/754170612 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5557443386c0 msgr2=0x555744338aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:29:13.266 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:13.263+0000 7f2c78928640 1 --2- 172.21.15.46:0/754170612 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5557443386c0 0x555744338aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:29:13.266 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:13.263+0000 7f2c78928640 1 -- 172.21.15.46:0/754170612 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x555744341db0 msgr2=0x5557443441a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:29:13.266 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:13.263+0000 7f2c78928640 1 --2- 172.21.15.46:0/754170612 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x555744341db0 0x5557443441a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:29:13.267 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:13.263+0000 7f2c78928640 1 -- 172.21.15.46:0/754170612 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555743fb0210 con 0x555744338fe0 2024-03-31T23:29:13.267 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:13.264+0000 7f2c71ffb640 1 -- 172.21.15.46:0/2526803272 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(122..122 src has 1..122) ==== 9554+0+0 (secure 0 0 0) 0x7f2c640cf620 con 0x7f2c741029c0 2024-03-31T23:29:13.267 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:13.264+0000 7f2c79129640 1 --2- 172.21.15.46:0/2526803272 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f2c600a26b0 0x7f2c600a4b70 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f2c6c02cf50 tx=0x7f2c6c005a90 comp rx=0 tx=0).ready entity=mgr.15654 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:29:13.267 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:13.264+0000 7f2c78928640 1 --2- 172.21.15.46:0/754170612 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555744338fe0 0x555744341870 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 crypto rx=0x7f2c64097df0 tx=0x7f2c64009a70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e37edaf9acd8e97 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:29:13.267 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:13.264+0000 7f2c6affd640 1 -- 172.21.15.46:0/754170612 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c6409c090 con 0x555744338fe0 2024-03-31T23:29:13.267 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:13.264+0000 7f2c6affd640 1 -- 172.21.15.46:0/754170612 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f2c6409b990 con 0x555744338fe0 2024-03-31T23:29:13.267 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:13.264+0000 7f2c6affd640 1 -- 172.21.15.46:0/754170612 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c64098000 con 0x555744338fe0 2024-03-31T23:29:13.271 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:13.265+0000 7f2c7d771740 1 -- 172.21.15.46:0/754170612 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555744338fe0 msgr2=0x555744341870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:29:13.271 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:13.265+0000 7f2c7d771740 1 --2- 172.21.15.46:0/754170612 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555744338fe0 0x555744341870 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 crypto rx=0x7f2c64097df0 tx=0x7f2c64009a70 comp rx=0 tx=0).stop 2024-03-31T23:29:13.271 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:13.265+0000 7f2c7d771740 1 -- 172.21.15.46:0/754170612 shutdown_connections 2024-03-31T23:29:13.271 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:13.265+0000 7f2c7d771740 1 --2- 172.21.15.46:0/754170612 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x555744341db0 0x5557443441a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:29:13.271 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:13.265+0000 7f2c7d771740 1 --2- 172.21.15.46:0/754170612 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555744338fe0 0x555744341870 unknown :-1 s=CLOSED pgs=938 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:29:13.271 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:13.265+0000 7f2c7d771740 1 --2- 172.21.15.46:0/754170612 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5557443386c0 0x555744338aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:29:13.271 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:13.265+0000 7f2c7d771740 1 -- 172.21.15.46:0/754170612 >> 172.21.15.46:0/754170612 conn(0x555744332fe0 msgr2=0x5557443333c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:29:13.271 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:13.265+0000 7f2c7d771740 1 -- 172.21.15.46:0/754170612 shutdown_connections 2024-03-31T23:29:13.272 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:13.265+0000 7f2c7d771740 1 -- 172.21.15.46:0/754170612 wait complete. 2024-03-31T23:29:13.272 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:13.266+0000 7f2c7d771740 1 -- start start 2024-03-31T23:29:13.272 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:13.266+0000 7f2c7d771740 1 Processor -- start 2024-03-31T23:29:13.272 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:13.266+0000 7f2c7d771740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5557443386c0 0x55574433ee00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:29:13.272 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:13.266+0000 7f2c7d771740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x555744338fe0 0x55574433f340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:29:13.272 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:13.266+0000 7f2c7d771740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555744341db0 0x55574433f880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:29:13.272 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:13.267+0000 7f2c7d771740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x555744344980 con 0x5557443386c0 2024-03-31T23:29:13.272 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:13.267+0000 7f2c73fff640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x555744338fe0 0x55574433f340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:29:13.272 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:13.267+0000 7f2c73fff640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x555744338fe0 0x55574433f340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.46:48764/0 (socket says 172.21.15.46:48764) 2024-03-31T23:29:13.272 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:13.267+0000 7f2c78928640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5557443386c0 0x55574433ee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:29:13.272 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:13.267+0000 7f2c73fff640 1 -- 172.21.15.46:0/2010486143 learned_addr learned my addr 172.21.15.46:0/2010486143 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:29:13.272 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:13.267+0000 7f2c7d771740 1 -- 172.21.15.46:0/2010486143 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x555744239f80 con 0x555744338fe0 2024-03-31T23:29:13.272 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:13.267+0000 7f2c7d771740 1 -- 172.21.15.46:0/2010486143 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x555744344b00 con 0x555744341db0 2024-03-31T23:29:13.273 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:13.267+0000 7f2c79129640 1 --2- 172.21.15.46:0/2010486143 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555744341db0 0x55574433f880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:29:13.273 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:13.267+0000 7f2c73fff640 1 -- 172.21.15.46:0/2010486143 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555744341db0 msgr2=0x55574433f880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:29:13.273 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:13.267+0000 7f2c73fff640 1 --2- 172.21.15.46:0/2010486143 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555744341db0 0x55574433f880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:29:13.273 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:13.267+0000 7f2c73fff640 1 -- 172.21.15.46:0/2010486143 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5557443386c0 msgr2=0x55574433ee00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:29:13.273 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:13.267+0000 7f2c7d771740 10 client.0 _ll_register_callbacks cb 0x555744292290 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T23:29:13.273 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:13.267+0000 7f2c73fff640 1 --2- 172.21.15.46:0/2010486143 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5557443386c0 0x55574433ee00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:29:13.273 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:13.267+0000 7f2c73fff640 1 -- 172.21.15.46:0/2010486143 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55574407c620 con 0x555744338fe0 2024-03-31T23:29:13.273 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:13.267+0000 7f2c78928640 1 --2- 172.21.15.46:0/2010486143 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5557443386c0 0x55574433ee00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:29:13.273 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:13.267+0000 7f2c7d771740 1 -- 172.21.15.46:0/2010486143 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2c6409b990 con 0x555744338fe0 2024-03-31T23:29:13.273 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:13.268+0000 7f2c73fff640 1 --2- 172.21.15.46:0/2010486143 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x555744338fe0 0x55574433f340 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f2c5c012d20 tx=0x7f2c5c00f790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5c7317689e40e4c8 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:29:13.273 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:13.268+0000 7f2c53fff640 10 client.27395 ms_handle_connect on v2:172.21.15.142:3300/0 2024-03-31T23:29:13.273 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:13.268+0000 7f2c53fff640 1 -- 172.21.15.46:0/2010486143 <== mon.2 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c5c012110 con 0x555744338fe0 2024-03-31T23:29:13.273 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:13.268+0000 7f2c53fff640 1 -- 172.21.15.46:0/2010486143 <== mon.2 v2:172.21.15.142:3300/0 2 ==== osd_map(122..122 src has 1..122) ==== 9554+0+0 (secure 0 0 0) 0x7f2c5c004050 con 0x555744338fe0 2024-03-31T23:29:13.274 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:13.268+0000 7f2c53fff640 1 -- 172.21.15.46:0/2010486143 <== mon.2 v2:172.21.15.142:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) ==== 24+0+0 (secure 0 0 0) 0x7f2c5c0122b0 con 0x555744338fe0 2024-03-31T23:29:13.274 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:13.268+0000 7f2c7d771740 10 client.27395 fetch_fsmap learned FSMap version 47 2024-03-31T23:29:13.274 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:13.268+0000 7f2c7d771740 1 -- 172.21.15.46:0/2010486143 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({fsmap=47}) -- 0x555744350810 con 0x555744338fe0 2024-03-31T23:29:13.274 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:13.269+0000 7f2c53fff640 1 -- 172.21.15.46:0/2010486143 <== mon.2 v2:172.21.15.142:3300/0 4 ==== fsmap(e 47) ==== 8346+0+0 (secure 0 0 0) 0x7f2c5c004290 con 0x555744338fe0 2024-03-31T23:29:13.274 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:13.269+0000 7f2c7d771740 10 client.27395 fetch_fsmap finished waiting for FSMap version 47 2024-03-31T23:29:13.274 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:13.269+0000 7f2c7d771740 10 client.27395 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-31T23:29:13.274 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:13.269+0000 7f2c7d771740 20 client.27395 populate_metadata read hostname 'smithi046' 2024-03-31T23:29:13.274 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:13.269+0000 7f2c7d771740 1 --2- 172.21.15.46:0/2010486143 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x555744351c10 0x5557443540d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:29:13.274 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:13.269+0000 7f2c7d771740 4 client.27395 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-31T23:29:13.274 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:13.269+0000 7f2c7d771740 1 -- 172.21.15.46:0/2010486143 --> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f2c6409b990 con 0x555744351c10 2024-03-31T23:29:13.274 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:13.269+0000 7f2c78928640 1 --2- 172.21.15.46:0/2010486143 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x555744351c10 0x5557443540d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:29:13.274 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:13.270+0000 7f2c78928640 1 --2- 172.21.15.46:0/2010486143 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x555744351c10 0x5557443540d0 crc :-1 s=READY pgs=232 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a0350b3474138a8d server_cookie=c994d0ba6bd724ac in_seq=0 out_seq=0 2024-03-31T23:29:13.274 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:13.270+0000 7f2c53fff640 10 client.27395 ms_handle_connect on v2:172.21.15.142:6838/2519143918 2024-03-31T23:29:13.275 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:13.272+0000 7f2c53fff640 1 -- 172.21.15.46:0/2010486143 <== mds.0 v2:172.21.15.142:6838/2519143918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x7f2c6409b990 con 0x555744351c10 2024-03-31T23:29:13.276 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:13.272+0000 7f2c53fff640 10 client.27395 handle_command_reply: tid=0 2024-03-31T23:29:13.276 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:13.272+0000 7f2c7d771740 1 client.27395 shutdown 2024-03-31T23:29:13.276 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:13.272+0000 7f2c7d771740 1 -- 172.21.15.46:0/2010486143 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x555744338fe0 msgr2=0x55574433f340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:29:13.276 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:13.272+0000 7f2c7d771740 1 --2- 172.21.15.46:0/2010486143 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x555744338fe0 0x55574433f340 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f2c5c012d20 tx=0x7f2c5c00f790 comp rx=0 tx=0).stop 2024-03-31T23:29:13.276 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:13.272+0000 7f2c7d771740 1 -- 172.21.15.46:0/2010486143 shutdown_connections 2024-03-31T23:29:13.276 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:13.272+0000 7f2c7d771740 1 --2- 172.21.15.46:0/2010486143 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x555744351c10 0x5557443540d0 crc :-1 s=READY pgs=232 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:29:13.276 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:13.272+0000 7f2c7d771740 1 --2- 172.21.15.46:0/2010486143 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555744341db0 0x55574433f880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:29:13.276 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:13.272+0000 7f2c7d771740 1 --2- 172.21.15.46:0/2010486143 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x555744338fe0 0x55574433f340 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:29:13.276 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:13.272+0000 7f2c7d771740 1 --2- 172.21.15.46:0/2010486143 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5557443386c0 0x55574433ee00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:29:13.276 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:13.272+0000 7f2c53fff640 0 client.27395 ms_handle_reset on v2:172.21.15.142:6838/2519143918 2024-03-31T23:29:13.276 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:13.272+0000 7f2c7d771740 1 -- 172.21.15.46:0/2010486143 >> 172.21.15.46:0/2010486143 conn(0x555744332fe0 msgr2=0x555744334d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:29:13.276 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:13.272+0000 7f2c7d771740 1 -- 172.21.15.46:0/2010486143 shutdown_connections 2024-03-31T23:29:13.276 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:13.272+0000 7f2c7d771740 1 -- 172.21.15.46:0/2010486143 wait complete. 2024-03-31T23:29:13.276 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:13.272+0000 7f2c7d771740 20 client.27395 trim_cache size 0 max 16384 2024-03-31T23:29:13.288 INFO:tasks.workunit.client.0.smithi046.stdout: 10 41683 8.56 MB/sec execute 138 sec latency 77.154 ms 2024-03-31T23:29:13.298 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:13.296+0000 7f2c7d771740 1 Processor -- start 2024-03-31T23:29:13.298 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:13.296+0000 7f2c7d771740 1 -- start start 2024-03-31T23:29:13.298 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:13.296+0000 7f2c7d771740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5557443386c0 0x555744341530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:29:13.298 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:13.296+0000 7f2c7d771740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555744338fe0 0x55574434d750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:29:13.298 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:13.296+0000 7f2c7d771740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555744341db0 0x55574433d6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:29:13.298 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:13.296+0000 7f2c7d771740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f2c741b37b0 con 0x555744338fe0 2024-03-31T23:29:13.298 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:13.296+0000 7f2c7d771740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f2c741b1080 con 0x5557443386c0 2024-03-31T23:29:13.298 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:13.296+0000 7f2c7d771740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f2c741aeef0 con 0x555744341db0 2024-03-31T23:29:13.298 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:13.297+0000 7f2c73fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555744338fe0 0x55574434d750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:29:13.298 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:13.297+0000 7f2c78928640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5557443386c0 0x555744341530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:29:13.298 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:13.297+0000 7f2c78928640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5557443386c0 0x555744341530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.46:48768/0 (socket says 172.21.15.46:48768) 2024-03-31T23:29:13.298 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:13.297+0000 7f2c78928640 1 -- 172.21.15.46:0/583450134 learned_addr learned my addr 172.21.15.46:0/583450134 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:29:13.298 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:13.297+0000 7f2c79129640 1 --2- 172.21.15.46:0/583450134 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555744341db0 0x55574433d6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:29:13.303 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:13.297+0000 7f2c78928640 1 -- 172.21.15.46:0/583450134 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555744341db0 msgr2=0x55574433d6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:29:13.303 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:13.297+0000 7f2c78928640 1 --2- 172.21.15.46:0/583450134 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555744341db0 0x55574433d6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:29:13.303 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:13.297+0000 7f2c78928640 1 -- 172.21.15.46:0/583450134 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555744338fe0 msgr2=0x55574434d750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:29:13.303 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:13.297+0000 7f2c78928640 1 --2- 172.21.15.46:0/583450134 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555744338fe0 0x55574434d750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:29:13.303 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:13.297+0000 7f2c78928640 1 -- 172.21.15.46:0/583450134 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555744337e50 con 0x5557443386c0 2024-03-31T23:29:13.303 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:13.297+0000 7f2c73fff640 1 --2- 172.21.15.46:0/583450134 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555744338fe0 0x55574434d750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:29:13.304 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:13.297+0000 7f2c78928640 1 --2- 172.21.15.46:0/583450134 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5557443386c0 0x555744341530 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f2c6400c7d0 tx=0x7f2c6407a040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9b62a280e3ad6738 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:29:13.304 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:13.298+0000 7f2c6a7fc640 1 -- 172.21.15.46:0/583450134 <== mon.2 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c640c4040 con 0x5557443386c0 2024-03-31T23:29:13.304 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:13.298+0000 7f2c6a7fc640 1 -- 172.21.15.46:0/583450134 <== mon.2 v2:172.21.15.142:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f2c6409c090 con 0x5557443386c0 2024-03-31T23:29:13.304 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:13.298+0000 7f2c6a7fc640 1 -- 172.21.15.46:0/583450134 <== mon.2 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c64098000 con 0x5557443386c0 2024-03-31T23:29:13.304 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:13.299+0000 7f2c7d771740 1 -- 172.21.15.46:0/583450134 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5557443386c0 msgr2=0x555744341530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:29:13.304 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:13.299+0000 7f2c7d771740 1 --2- 172.21.15.46:0/583450134 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5557443386c0 0x555744341530 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f2c6400c7d0 tx=0x7f2c6407a040 comp rx=0 tx=0).stop 2024-03-31T23:29:13.304 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:13.299+0000 7f2c7d771740 1 -- 172.21.15.46:0/583450134 shutdown_connections 2024-03-31T23:29:13.304 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:13.299+0000 7f2c7d771740 1 --2- 172.21.15.46:0/583450134 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555744341db0 0x55574433d6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:29:13.304 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:13.299+0000 7f2c7d771740 1 --2- 172.21.15.46:0/583450134 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555744338fe0 0x55574434d750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:29:13.304 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:13.299+0000 7f2c7d771740 1 --2- 172.21.15.46:0/583450134 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5557443386c0 0x555744341530 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:29:13.304 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:13.299+0000 7f2c7d771740 1 -- 172.21.15.46:0/583450134 >> 172.21.15.46:0/583450134 conn(0x555744332fe0 msgr2=0x5557443333c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:29:13.304 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:13.299+0000 7f2c7d771740 1 -- 172.21.15.46:0/583450134 shutdown_connections 2024-03-31T23:29:13.305 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:13.299+0000 7f2c7d771740 1 -- 172.21.15.46:0/583450134 wait complete. 2024-03-31T23:29:13.305 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:13.299+0000 7f2c7d771740 1 -- start start 2024-03-31T23:29:13.305 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:13.299+0000 7f2c7d771740 1 Processor -- start 2024-03-31T23:29:13.305 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:13.299+0000 7f2c7d771740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5557443386c0 0x55574433b960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:29:13.305 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:13.299+0000 7f2c7d771740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555744338fe0 0x55574433bea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:29:13.305 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:13.300+0000 7f2c7d771740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x555744341db0 0x55574433c3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:29:13.305 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:13.300+0000 7f2c7d771740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f2c741af540 con 0x5557443386c0 2024-03-31T23:29:13.305 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:13.300+0000 7f2c7d771740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f2c741b09b0 con 0x555744341db0 2024-03-31T23:29:13.305 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:13.300+0000 7f2c7d771740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f2c741afc10 con 0x555744338fe0 2024-03-31T23:29:13.305 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:13.300+0000 7f2c78928640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5557443386c0 0x55574433b960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:29:13.305 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:13.300+0000 7f2c78928640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5557443386c0 0x55574433b960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:35192/0 (socket says 172.21.15.46:35192) 2024-03-31T23:29:13.305 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:13.300+0000 7f2c78928640 1 -- 172.21.15.46:0/1996627417 learned_addr learned my addr 172.21.15.46:0/1996627417 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:29:13.305 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:13.300+0000 7f2c7d771740 10 client.0 _ll_register_callbacks cb 0x7f2c5c009d20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T23:29:13.305 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:13.300+0000 7f2c73fff640 1 --2- 172.21.15.46:0/1996627417 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555744338fe0 0x55574433bea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:29:13.306 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:13.300+0000 7f2c79129640 1 --2- 172.21.15.46:0/1996627417 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x555744341db0 0x55574433c3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:29:13.306 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:13.300+0000 7f2c78928640 1 -- 172.21.15.46:0/1996627417 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555744338fe0 msgr2=0x55574433bea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:29:13.306 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:13.300+0000 7f2c78928640 1 --2- 172.21.15.46:0/1996627417 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555744338fe0 0x55574433bea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:29:13.306 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:13.300+0000 7f2c78928640 1 -- 172.21.15.46:0/1996627417 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x555744341db0 msgr2=0x55574433c3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:29:13.306 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:13.300+0000 7f2c78928640 1 --2- 172.21.15.46:0/1996627417 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x555744341db0 0x55574433c3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:29:13.306 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:13.300+0000 7f2c78928640 1 -- 172.21.15.46:0/1996627417 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5557443bf820 con 0x5557443386c0 2024-03-31T23:29:13.306 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:13.300+0000 7f2c78928640 1 --2- 172.21.15.46:0/1996627417 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5557443386c0 0x55574433b960 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 crypto rx=0x7f2c640c42a0 tx=0x7f2c6400cf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1294b246f636323c server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:29:13.306 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:13.300+0000 7f2c53fff640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-31T23:29:13.306 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:13.300+0000 7f2c53fff640 1 -- 172.21.15.46:0/1996627417 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c640b8070 con 0x5557443386c0 2024-03-31T23:29:13.306 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:13.300+0000 7f2c53fff640 1 -- 172.21.15.46:0/1996627417 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(122..122 src has 1..122) ==== 9554+0+0 (secure 0 0 0) 0x7f2c64098000 con 0x5557443386c0 2024-03-31T23:29:13.306 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:13.300+0000 7f2c7d771740 1 -- 172.21.15.46:0/1996627417 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2c6409c090 con 0x5557443386c0 2024-03-31T23:29:13.306 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:13.301+0000 7f2c53fff640 1 -- 172.21.15.46:0/1996627417 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) ==== 24+0+0 (secure 0 0 0) 0x7f2c6409fae0 con 0x5557443386c0 2024-03-31T23:29:13.306 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:13.301+0000 7f2c7d771740 10 client.17574 fetch_fsmap learned FSMap version 47 2024-03-31T23:29:13.306 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:13.301+0000 7f2c7d771740 1 -- 172.21.15.46:0/1996627417 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) -- 0x5557443c0ae0 con 0x5557443386c0 2024-03-31T23:29:13.307 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:13.301+0000 7f2c53fff640 1 -- 172.21.15.46:0/1996627417 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f2c6409c090 con 0x5557443386c0 2024-03-31T23:29:13.307 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:13.301+0000 7f2c53fff640 1 -- 172.21.15.46:0/1996627417 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 47) ==== 8346+0+0 (secure 0 0 0) 0x7f2c6409ade0 con 0x5557443386c0 2024-03-31T23:29:13.307 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:13.301+0000 7f2c7d771740 10 client.17574 fetch_fsmap finished waiting for FSMap version 47 2024-03-31T23:29:13.307 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:13.301+0000 7f2c7d771740 10 client.17574 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-31T23:29:13.307 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:13.301+0000 7f2c7d771740 20 client.17574 populate_metadata read hostname 'smithi046' 2024-03-31T23:29:13.307 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:13.301+0000 7f2c7d771740 1 --2- 172.21.15.46:0/1996627417 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x5557443c0dc0 0x5557443c31b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:29:13.307 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:13.301+0000 7f2c7d771740 4 client.17574 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-31T23:29:13.307 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:13.301+0000 7f2c7d771740 1 -- 172.21.15.46:0/1996627417 --> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5557442af120 con 0x5557443c0dc0 2024-03-31T23:29:13.307 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:13.302+0000 7f2c73fff640 1 --2- 172.21.15.46:0/1996627417 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x5557443c0dc0 0x5557443c31b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:29:13.307 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:13.302+0000 7f2c73fff640 1 --2- 172.21.15.46:0/1996627417 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x5557443c0dc0 0x5557443c31b0 crc :-1 s=READY pgs=233 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fec9be56e2bb61a2 server_cookie=7c9afbd8d3ae9fc9 in_seq=0 out_seq=0 2024-03-31T23:29:13.307 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:13.302+0000 7f2c53fff640 10 client.17574 ms_handle_connect on v2:172.21.15.142:6838/2519143918 2024-03-31T23:29:13.519 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:13.518+0000 7f2c53fff640 1 -- 172.21.15.46:0/1996627417 <== mds.0 v2:172.21.15.142:6838/2519143918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5557442af120 con 0x5557443c0dc0 2024-03-31T23:29:13.520 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:13.518+0000 7f2c53fff640 10 client.17574 handle_command_reply: tid=0 2024-03-31T23:29:13.520 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:13.518+0000 7f2c7d771740 1 client.17574 shutdown 2024-03-31T23:29:13.520 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:13.518+0000 7f2c7d771740 1 -- 172.21.15.46:0/1996627417 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5557443386c0 msgr2=0x55574433b960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:29:13.520 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:13.518+0000 7f2c7d771740 1 --2- 172.21.15.46:0/1996627417 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5557443386c0 0x55574433b960 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 crypto rx=0x7f2c640c42a0 tx=0x7f2c6400cf10 comp rx=0 tx=0).stop 2024-03-31T23:29:13.520 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:13.518+0000 7f2c7d771740 1 -- 172.21.15.46:0/1996627417 shutdown_connections 2024-03-31T23:29:13.520 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:13.518+0000 7f2c7d771740 1 --2- 172.21.15.46:0/1996627417 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x5557443c0dc0 0x5557443c31b0 crc :-1 s=READY pgs=233 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:29:13.520 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:13.518+0000 7f2c7d771740 1 --2- 172.21.15.46:0/1996627417 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x555744341db0 0x55574433c3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:29:13.520 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:13.518+0000 7f2c7d771740 1 --2- 172.21.15.46:0/1996627417 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555744338fe0 0x55574433bea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:29:13.520 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:13.518+0000 7f2c7d771740 1 --2- 172.21.15.46:0/1996627417 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5557443386c0 0x55574433b960 unknown :-1 s=CLOSED pgs=939 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:29:13.520 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:13.518+0000 7f2c7d771740 1 -- 172.21.15.46:0/1996627417 >> 172.21.15.46:0/1996627417 conn(0x555744332fe0 msgr2=0x55574433edc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:29:13.520 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:13.518+0000 7f2c53fff640 0 client.17574 ms_handle_reset on v2:172.21.15.142:6838/2519143918 2024-03-31T23:29:13.521 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:13.518+0000 7f2c7d771740 1 -- 172.21.15.46:0/1996627417 shutdown_connections 2024-03-31T23:29:13.521 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:13.518+0000 7f2c7d771740 1 -- 172.21.15.46:0/1996627417 wait complete. 2024-03-31T23:29:13.521 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:13.518+0000 7f2c7d771740 20 client.17574 trim_cache size 0 max 16384 2024-03-31T23:29:13.521 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout:{ 2024-03-31T23:29:13.521 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout: "message": "", 2024-03-31T23:29:13.521 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout: "return_code": 0 2024-03-31T23:29:13.521 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout:} 2024-03-31T23:29:13.522 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:13.519+0000 7f2c7ab8c640 1 -- 172.21.15.46:0/2526803272 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f2c600a26b0 msgr2=0x7f2c600a4b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:29:13.522 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:13.519+0000 7f2c7ab8c640 1 --2- 172.21.15.46:0/2526803272 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f2c600a26b0 0x7f2c600a4b70 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f2c6c02cf50 tx=0x7f2c6c005a90 comp rx=0 tx=0).stop 2024-03-31T23:29:13.522 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:13.519+0000 7f2c7ab8c640 1 -- 172.21.15.46:0/2526803272 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2c741029c0 msgr2=0x7f2c7419a0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:29:13.522 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:13.519+0000 7f2c7ab8c640 1 --2- 172.21.15.46:0/2526803272 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2c741029c0 0x7f2c7419a0b0 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x7f2c6400ebc0 tx=0x7f2c6400c2d0 comp rx=0 tx=0).stop 2024-03-31T23:29:13.522 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:13.519+0000 7f2c7ab8c640 1 -- 172.21.15.46:0/2526803272 shutdown_connections 2024-03-31T23:29:13.522 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:13.519+0000 7f2c7ab8c640 1 --2- 172.21.15.46:0/2526803272 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f2c600a26b0 0x7f2c600a4b70 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:29:13.522 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:13.519+0000 7f2c7ab8c640 1 --2- 172.21.15.46:0/2526803272 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2c7410bd00 0x7f2c7419e970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:29:13.523 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:13.519+0000 7f2c7ab8c640 1 --2- 172.21.15.46:0/2526803272 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2c741032e0 0x7f2c7419a5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:29:13.523 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:13.519+0000 7f2c7ab8c640 1 --2- 172.21.15.46:0/2526803272 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2c741029c0 0x7f2c7419a0b0 unknown :-1 s=CLOSED pgs=937 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:29:13.523 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:13.519+0000 7f2c7ab8c640 1 -- 172.21.15.46:0/2526803272 >> 172.21.15.46:0/2526803272 conn(0x7f2c74077770 msgr2=0x7f2c74107c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:29:13.523 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:13.519+0000 7f2c7ab8c640 1 -- 172.21.15.46:0/2526803272 shutdown_connections 2024-03-31T23:29:13.523 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:13.519+0000 7f2c7ab8c640 1 -- 172.21.15.46:0/2526803272 wait complete. 2024-03-31T23:29:14.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:29:13 smithi162 ceph-mon[36237]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:29:14.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:29:13 smithi162 ceph-mon[36237]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-31T23:29:14.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:29:13 smithi162 ceph-mon[36237]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' 2024-03-31T23:29:14.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:29:13 smithi046 ceph-mon[32498]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:29:14.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:29:13 smithi046 ceph-mon[32498]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-31T23:29:14.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:29:13 smithi046 ceph-mon[32498]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' 2024-03-31T23:29:14.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:29:13 smithi142 ceph-mon[35271]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:29:14.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:29:13 smithi142 ceph-mon[35271]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-31T23:29:14.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:29:13 smithi142 ceph-mon[35271]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' 2024-03-31T23:29:14.288 INFO:tasks.workunit.client.0.smithi046.stdout: 10 41823 8.54 MB/sec execute 139 sec latency 187.708 ms 2024-03-31T23:29:15.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:29:14 smithi162 ceph-mon[36237]: pgmap v554: 129 pgs: 129 active+clean; 736 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 6.1 MiB/s wr, 347 op/s 2024-03-31T23:29:15.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:29:14 smithi046 ceph-mon[32498]: pgmap v554: 129 pgs: 129 active+clean; 736 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 6.1 MiB/s wr, 347 op/s 2024-03-31T23:29:15.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:29:14 smithi142 ceph-mon[35271]: pgmap v554: 129 pgs: 129 active+clean; 736 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 6.1 MiB/s wr, 347 op/s 2024-03-31T23:29:15.288 INFO:tasks.workunit.client.0.smithi046.stdout: 10 42006 8.56 MB/sec execute 140 sec latency 89.944 ms 2024-03-31T23:29:16.288 INFO:tasks.workunit.client.0.smithi046.stdout: 10 42162 8.56 MB/sec execute 141 sec latency 77.594 ms 2024-03-31T23:29:17.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:29:16 smithi162 ceph-mon[36237]: pgmap v555: 129 pgs: 129 active+clean; 739 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 5.1 MiB/s wr, 330 op/s 2024-03-31T23:29:17.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:29:16 smithi046 ceph-mon[32498]: pgmap v555: 129 pgs: 129 active+clean; 739 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 5.1 MiB/s wr, 330 op/s 2024-03-31T23:29:17.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:29:16 smithi142 ceph-mon[35271]: pgmap v555: 129 pgs: 129 active+clean; 739 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 5.1 MiB/s wr, 330 op/s 2024-03-31T23:29:17.288 INFO:tasks.workunit.client.0.smithi046.stdout: 10 42344 8.57 MB/sec execute 142 sec latency 75.302 ms 2024-03-31T23:29:18.289 INFO:tasks.workunit.client.0.smithi046.stdout: 10 42505 8.56 MB/sec execute 143 sec latency 70.497 ms 2024-03-31T23:29:19.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:29:18 smithi162 ceph-mon[36237]: pgmap v556: 129 pgs: 129 active+clean; 739 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 5.1 MiB/s wr, 330 op/s 2024-03-31T23:29:19.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:29:18 smithi046 ceph-mon[32498]: pgmap v556: 129 pgs: 129 active+clean; 739 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 5.1 MiB/s wr, 330 op/s 2024-03-31T23:29:19.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:29:18 smithi142 ceph-mon[35271]: pgmap v556: 129 pgs: 129 active+clean; 739 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 5.1 MiB/s wr, 330 op/s 2024-03-31T23:29:19.288 INFO:tasks.workunit.client.0.smithi046.stdout: 10 42690 8.60 MB/sec execute 144 sec latency 78.439 ms 2024-03-31T23:29:20.289 INFO:tasks.workunit.client.0.smithi046.stdout: 10 42846 8.60 MB/sec execute 145 sec latency 93.731 ms 2024-03-31T23:29:21.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:29:20 smithi162 ceph-mon[36237]: pgmap v557: 129 pgs: 129 active+clean; 750 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 4.8 MiB/s wr, 340 op/s 2024-03-31T23:29:21.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:29:20 smithi046 ceph-mon[32498]: pgmap v557: 129 pgs: 129 active+clean; 750 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 4.8 MiB/s wr, 340 op/s 2024-03-31T23:29:21.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:29:20 smithi142 ceph-mon[35271]: pgmap v557: 129 pgs: 129 active+clean; 750 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 4.8 MiB/s wr, 340 op/s 2024-03-31T23:29:21.289 INFO:tasks.workunit.client.0.smithi046.stdout: 10 42977 8.58 MB/sec execute 146 sec latency 210.737 ms 2024-03-31T23:29:22.289 INFO:tasks.workunit.client.0.smithi046.stdout: 10 43121 8.58 MB/sec execute 147 sec latency 83.778 ms 2024-03-31T23:29:22.994 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:29:22 smithi142 ceph-mon[35271]: pgmap v558: 129 pgs: 129 active+clean; 750 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 4.6 MiB/s wr, 329 op/s 2024-03-31T23:29:23.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:29:22 smithi162 ceph-mon[36237]: pgmap v558: 129 pgs: 129 active+clean; 750 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 4.6 MiB/s wr, 329 op/s 2024-03-31T23:29:23.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:29:22 smithi046 ceph-mon[32498]: pgmap v558: 129 pgs: 129 active+clean; 750 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 4.6 MiB/s wr, 329 op/s 2024-03-31T23:29:23.289 INFO:tasks.workunit.client.0.smithi046.stdout: 10 43293 8.61 MB/sec execute 148 sec latency 88.662 ms 2024-03-31T23:29:24.289 INFO:tasks.workunit.client.0.smithi046.stdout: 10 43504 8.64 MB/sec execute 149 sec latency 67.816 ms 2024-03-31T23:29:25.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:29:24 smithi162 ceph-mon[36237]: pgmap v559: 129 pgs: 129 active+clean; 750 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 4.0 MiB/s wr, 284 op/s 2024-03-31T23:29:25.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:29:24 smithi046 ceph-mon[32498]: pgmap v559: 129 pgs: 129 active+clean; 750 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 4.0 MiB/s wr, 284 op/s 2024-03-31T23:29:25.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:29:24 smithi142 ceph-mon[35271]: pgmap v559: 129 pgs: 129 active+clean; 750 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 4.0 MiB/s wr, 284 op/s 2024-03-31T23:29:25.289 INFO:tasks.workunit.client.0.smithi046.stdout: 10 43647 8.63 MB/sec execute 150 sec latency 89.226 ms 2024-03-31T23:29:26.289 INFO:tasks.workunit.client.0.smithi046.stdout: 10 43748 8.59 MB/sec execute 151 sec latency 75.277 ms 2024-03-31T23:29:26.662 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:26.661+0000 7f12bf4d4640 1 -- 172.21.15.46:0/3330385410 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f12b8103250 msgr2=0x7f12b810b730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:29:26.662 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:26.661+0000 7f12bf4d4640 1 --2- 172.21.15.46:0/3330385410 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f12b8103250 0x7f12b810b730 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x7f12a8009900 tx=0x7f12a802bc30 comp rx=0 tx=0).stop 2024-03-31T23:29:26.662 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:26.661+0000 7f12bf4d4640 1 -- 172.21.15.46:0/3330385410 shutdown_connections 2024-03-31T23:29:26.662 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:26.661+0000 7f12bf4d4640 1 --2- 172.21.15.46:0/3330385410 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f12b810bda0 0x7f12b810e210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:29:26.663 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:26.661+0000 7f12bf4d4640 1 --2- 172.21.15.46:0/3330385410 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f12b8103250 0x7f12b810b730 unknown :-1 s=CLOSED pgs=940 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:29:26.663 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:26.661+0000 7f12bf4d4640 1 --2- 172.21.15.46:0/3330385410 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f12b8102930 0x7f12b8102d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:29:26.663 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:26.661+0000 7f12bf4d4640 1 -- 172.21.15.46:0/3330385410 >> 172.21.15.46:0/3330385410 conn(0x7f12b8077780 msgr2=0x7f12b8077b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:29:26.663 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:26.661+0000 7f12bf4d4640 1 -- 172.21.15.46:0/3330385410 shutdown_connections 2024-03-31T23:29:26.663 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:26.661+0000 7f12bf4d4640 1 -- 172.21.15.46:0/3330385410 wait complete. 2024-03-31T23:29:26.664 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:26.662+0000 7f12bf4d4640 1 Processor -- start 2024-03-31T23:29:26.664 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:26.662+0000 7f12bf4d4640 1 -- start start 2024-03-31T23:29:26.664 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:26.662+0000 7f12bf4d4640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f12b8102930 0x7f12b8197ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:29:26.664 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:26.662+0000 7f12bf4d4640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f12b8103250 0x7f12b8198420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:29:26.665 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:26.662+0000 7f12bf4d4640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f12b810bda0 0x7f12b819c750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:29:26.665 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:26.662+0000 7f12bf4d4640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f12b8111a20 con 0x7f12b8103250 2024-03-31T23:29:26.665 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:26.662+0000 7f12bf4d4640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f12b81118a0 con 0x7f12b8102930 2024-03-31T23:29:26.665 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:26.662+0000 7f12bf4d4640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f12b8111ba0 con 0x7f12b810bda0 2024-03-31T23:29:26.665 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:26.662+0000 7f12bca6f640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f12b8103250 0x7f12b8198420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:29:26.665 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:26.662+0000 7f12bca6f640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f12b8103250 0x7f12b8198420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:51192/0 (socket says 172.21.15.46:51192) 2024-03-31T23:29:26.665 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:26.662+0000 7f12bca6f640 1 -- 172.21.15.46:0/3595153635 learned_addr learned my addr 172.21.15.46:0/3595153635 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:29:26.665 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:26.662+0000 7f12bda71640 1 --2- 172.21.15.46:0/3595153635 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f12b810bda0 0x7f12b819c750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:29:26.665 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:26.663+0000 7f12bd270640 1 --2- 172.21.15.46:0/3595153635 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f12b8102930 0x7f12b8197ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:29:26.665 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:26.663+0000 7f12bca6f640 1 -- 172.21.15.46:0/3595153635 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f12b810bda0 msgr2=0x7f12b819c750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:29:26.665 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:26.663+0000 7f12bca6f640 1 --2- 172.21.15.46:0/3595153635 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f12b810bda0 0x7f12b819c750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:29:26.665 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:26.663+0000 7f12bca6f640 1 -- 172.21.15.46:0/3595153635 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f12b8102930 msgr2=0x7f12b8197ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:29:26.665 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:26.663+0000 7f12bca6f640 1 --2- 172.21.15.46:0/3595153635 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f12b8102930 0x7f12b8197ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:29:26.665 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:26.663+0000 7f12bca6f640 1 -- 172.21.15.46:0/3595153635 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f12b819ce80 con 0x7f12b8103250 2024-03-31T23:29:26.666 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:26.663+0000 7f12bca6f640 1 --2- 172.21.15.46:0/3595153635 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f12b8103250 0x7f12b8198420 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 crypto rx=0x7f12a8005940 tx=0x7f12a802c5f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:29:26.666 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:26.663+0000 7f12ae7fc640 1 -- 172.21.15.46:0/3595153635 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12a8043020 con 0x7f12b8103250 2024-03-31T23:29:26.666 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:26.663+0000 7f12ae7fc640 1 -- 172.21.15.46:0/3595153635 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f12a802ca70 con 0x7f12b8103250 2024-03-31T23:29:26.666 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:26.663+0000 7f12ae7fc640 1 -- 172.21.15.46:0/3595153635 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12a8003a80 con 0x7f12b8103250 2024-03-31T23:29:26.666 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:26.663+0000 7f12bf4d4640 1 -- 172.21.15.46:0/3595153635 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f12b819d110 con 0x7f12b8103250 2024-03-31T23:29:26.666 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:26.664+0000 7f12bf4d4640 1 -- 172.21.15.46:0/3595153635 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f12b81a4a00 con 0x7f12b8103250 2024-03-31T23:29:26.668 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:26.667+0000 7f12ae7fc640 1 -- 172.21.15.46:0/3595153635 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 25) ==== 137451+0+0 (secure 0 0 0) 0x7f12a8046020 con 0x7f12b8103250 2024-03-31T23:29:26.668 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:26.667+0000 7f12c2199740 1 Processor -- start 2024-03-31T23:29:26.668 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:26.667+0000 7f12c2199740 1 -- start start 2024-03-31T23:29:26.669 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:26.667+0000 7f12c2199740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559c46e7e6c0 0x559c46e7eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:29:26.669 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:26.667+0000 7f12c2199740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559c46e7efe0 0x559c46e87870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:29:26.669 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:26.667+0000 7f12c2199740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559c46e87db0 0x559c46e8a1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:29:26.669 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:26.667+0000 7f12c2199740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x559c46de18f0 con 0x559c46e7efe0 2024-03-31T23:29:26.669 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:26.667+0000 7f12c2199740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x559c46d8a8d0 con 0x559c46e87db0 2024-03-31T23:29:26.669 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:26.667+0000 7f12c2199740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x559c46ddd990 con 0x559c46e7e6c0 2024-03-31T23:29:26.669 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:26.668+0000 7f12bd270640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559c46e7efe0 0x559c46e87870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:29:26.669 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:26.668+0000 7f12bda71640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559c46e7e6c0 0x559c46e7eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:29:26.669 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:26.668+0000 7f12bd270640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559c46e7efe0 0x559c46e87870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:51198/0 (socket says 172.21.15.46:51198) 2024-03-31T23:29:26.669 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:26.668+0000 7f12bd270640 1 -- 172.21.15.46:0/2672197708 learned_addr learned my addr 172.21.15.46:0/2672197708 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:29:26.669 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:26.668+0000 7f12bca6f640 1 --2- 172.21.15.46:0/2672197708 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559c46e87db0 0x559c46e8a1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:29:26.674 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:26.668+0000 7f12bd270640 1 -- 172.21.15.46:0/2672197708 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559c46e7e6c0 msgr2=0x559c46e7eaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:29:26.674 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:26.668+0000 7f12bd270640 1 --2- 172.21.15.46:0/2672197708 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559c46e7e6c0 0x559c46e7eaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:29:26.674 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:26.668+0000 7f12bd270640 1 -- 172.21.15.46:0/2672197708 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559c46e87db0 msgr2=0x559c46e8a1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:29:26.674 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:26.668+0000 7f12bd270640 1 --2- 172.21.15.46:0/2672197708 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559c46e87db0 0x559c46e8a1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:29:26.674 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:26.668+0000 7f12bd270640 1 -- 172.21.15.46:0/2672197708 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559c46af6210 con 0x559c46e7efe0 2024-03-31T23:29:26.675 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:26.668+0000 7f12bda71640 1 --2- 172.21.15.46:0/2672197708 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559c46e7e6c0 0x559c46e7eaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:29:26.675 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:26.668+0000 7f12bd270640 1 --2- 172.21.15.46:0/2672197708 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559c46e7efe0 0x559c46e87870 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x7f12b4009c80 tx=0x7f12b40117b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=73eb992f117f13bc server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:29:26.675 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:26.669+0000 7f12ae7fc640 1 --2- 172.21.15.46:0/3595153635 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f12a40a26b0 0x7f12a40a4b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:29:26.675 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:26.669+0000 7f12a37fe640 1 -- 172.21.15.46:0/2672197708 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12b4012450 con 0x559c46e7efe0 2024-03-31T23:29:26.675 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:26.669+0000 7f12a37fe640 1 -- 172.21.15.46:0/2672197708 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f12b4012aa0 con 0x559c46e7efe0 2024-03-31T23:29:26.675 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:26.669+0000 7f12a37fe640 1 -- 172.21.15.46:0/2672197708 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12b400fb90 con 0x559c46e7efe0 2024-03-31T23:29:26.675 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:26.669+0000 7f12ae7fc640 1 -- 172.21.15.46:0/3595153635 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(122..122 src has 1..122) ==== 9554+0+0 (secure 0 0 0) 0x7f12a80f0fc0 con 0x7f12b8103250 2024-03-31T23:29:26.675 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:26.669+0000 7f12bda71640 1 --2- 172.21.15.46:0/3595153635 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f12a40a26b0 0x7f12a40a4b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:29:26.675 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:26.670+0000 7f12bda71640 1 --2- 172.21.15.46:0/3595153635 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f12a40a26b0 0x7f12a40a4b70 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f12b0009fd0 tx=0x7f12b0009500 comp rx=0 tx=0).ready entity=mgr.15654 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:29:26.675 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:26.670+0000 7f12c2199740 1 -- 172.21.15.46:0/2672197708 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559c46e7efe0 msgr2=0x559c46e87870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:29:26.675 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:26.670+0000 7f12c2199740 1 --2- 172.21.15.46:0/2672197708 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559c46e7efe0 0x559c46e87870 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x7f12b4009c80 tx=0x7f12b40117b0 comp rx=0 tx=0).stop 2024-03-31T23:29:26.675 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:26.670+0000 7f12c2199740 1 -- 172.21.15.46:0/2672197708 shutdown_connections 2024-03-31T23:29:26.675 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:26.670+0000 7f12c2199740 1 --2- 172.21.15.46:0/2672197708 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559c46e87db0 0x559c46e8a1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:29:26.676 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:26.670+0000 7f12c2199740 1 --2- 172.21.15.46:0/2672197708 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559c46e7efe0 0x559c46e87870 unknown :-1 s=CLOSED pgs=942 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:29:26.676 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:26.670+0000 7f12c2199740 1 --2- 172.21.15.46:0/2672197708 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559c46e7e6c0 0x559c46e7eaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:29:26.676 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:26.670+0000 7f12c2199740 1 -- 172.21.15.46:0/2672197708 >> 172.21.15.46:0/2672197708 conn(0x559c46e78fe0 msgr2=0x559c46e793c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:29:26.676 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:26.670+0000 7f12c2199740 1 -- 172.21.15.46:0/2672197708 shutdown_connections 2024-03-31T23:29:26.676 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:26.670+0000 7f12c2199740 1 -- 172.21.15.46:0/2672197708 wait complete. 2024-03-31T23:29:26.676 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:26.670+0000 7f12c2199740 1 -- start start 2024-03-31T23:29:26.676 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:26.670+0000 7f12c2199740 1 Processor -- start 2024-03-31T23:29:26.676 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:26.671+0000 7f12c2199740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559c46e7e6c0 0x559c46e84ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:29:26.676 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:26.671+0000 7f12c2199740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559c46e7efe0 0x559c46e85430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:29:26.676 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:26.671+0000 7f12c2199740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559c46e87db0 0x559c46e85970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:29:26.676 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:26.671+0000 7f12bda71640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559c46e87db0 0x559c46e85970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:29:26.676 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:26.671+0000 7f12bd270640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559c46e7e6c0 0x559c46e84ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:29:26.676 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:26.671+0000 7f12bda71640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559c46e87db0 0x559c46e85970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:51204/0 (socket says 172.21.15.46:51204) 2024-03-31T23:29:26.677 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:26.671+0000 7f12bda71640 1 -- 172.21.15.46:0/1215725065 learned_addr learned my addr 172.21.15.46:0/1215725065 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:29:26.677 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:26.671+0000 7f12c2199740 1 -- 172.21.15.46:0/1215725065 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x559c46e8a9e0 con 0x559c46e87db0 2024-03-31T23:29:26.677 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:26.671+0000 7f12c2199740 1 -- 172.21.15.46:0/1215725065 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x559c46d7ff80 con 0x559c46e7e6c0 2024-03-31T23:29:26.677 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:26.671+0000 7f12c2199740 1 -- 172.21.15.46:0/1215725065 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x559c46e8ab60 con 0x559c46e7efe0 2024-03-31T23:29:26.677 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:26.671+0000 7f12bca6f640 1 --2- 172.21.15.46:0/1215725065 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559c46e7efe0 0x559c46e85430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:29:26.677 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:26.671+0000 7f12c2199740 10 client.0 _ll_register_callbacks cb 0x559c46dd8290 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T23:29:26.677 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:26.671+0000 7f12bda71640 1 -- 172.21.15.46:0/1215725065 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559c46e7efe0 msgr2=0x559c46e85430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:29:26.677 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:26.671+0000 7f12bda71640 1 --2- 172.21.15.46:0/1215725065 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559c46e7efe0 0x559c46e85430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:29:26.677 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:26.671+0000 7f12bda71640 1 -- 172.21.15.46:0/1215725065 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559c46e7e6c0 msgr2=0x559c46e84ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:29:26.677 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:26.671+0000 7f12bda71640 1 --2- 172.21.15.46:0/1215725065 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559c46e7e6c0 0x559c46e84ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:29:26.677 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:26.671+0000 7f12bda71640 1 -- 172.21.15.46:0/1215725065 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559c46bc2620 con 0x559c46e87db0 2024-03-31T23:29:26.677 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:26.672+0000 7f12bda71640 1 --2- 172.21.15.46:0/1215725065 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559c46e87db0 0x559c46e85970 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 crypto rx=0x7f12b000f6a0 tx=0x7f12b0004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f0a97ef5ccfa5e1b server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:29:26.677 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:26.672+0000 7f12a0ff9640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-31T23:29:26.677 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:26.672+0000 7f12a0ff9640 1 -- 172.21.15.46:0/1215725065 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12b002c060 con 0x559c46e87db0 2024-03-31T23:29:26.678 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:26.672+0000 7f12a0ff9640 1 -- 172.21.15.46:0/1215725065 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(122..122 src has 1..122) ==== 9554+0+0 (secure 0 0 0) 0x7f12b0017030 con 0x559c46e87db0 2024-03-31T23:29:26.678 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:26.672+0000 7f12c2199740 1 -- 172.21.15.46:0/1215725065 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f12b4012aa0 con 0x559c46e87db0 2024-03-31T23:29:26.678 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:26.672+0000 7f12a0ff9640 1 -- 172.21.15.46:0/1215725065 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) ==== 24+0+0 (secure 0 0 0) 0x7f12b0033050 con 0x559c46e87db0 2024-03-31T23:29:26.678 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:26.672+0000 7f12c2199740 10 client.17586 fetch_fsmap learned FSMap version 47 2024-03-31T23:29:26.678 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:26.672+0000 7f12c2199740 1 -- 172.21.15.46:0/1215725065 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) -- 0x559c46e96a40 con 0x559c46e87db0 2024-03-31T23:29:26.678 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:26.672+0000 7f12a0ff9640 1 -- 172.21.15.46:0/1215725065 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f12b4012aa0 con 0x559c46e87db0 2024-03-31T23:29:26.678 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:26.673+0000 7f12a0ff9640 1 -- 172.21.15.46:0/1215725065 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 47) ==== 8346+0+0 (secure 0 0 0) 0x7f12b0027920 con 0x559c46e87db0 2024-03-31T23:29:26.678 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:26.673+0000 7f12c2199740 10 client.17586 fetch_fsmap finished waiting for FSMap version 47 2024-03-31T23:29:26.678 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:26.673+0000 7f12c2199740 10 client.17586 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-31T23:29:26.678 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:26.673+0000 7f12c2199740 20 client.17586 populate_metadata read hostname 'smithi046' 2024-03-31T23:29:26.678 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:26.673+0000 7f12c2199740 1 --2- 172.21.15.46:0/1215725065 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x559c46e97f00 0x559c46e9a330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:29:26.678 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:26.673+0000 7f12c2199740 4 client.17586 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-31T23:29:26.678 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:26.673+0000 7f12c2199740 1 -- 172.21.15.46:0/1215725065 --> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559c46df5120 con 0x559c46e97f00 2024-03-31T23:29:26.679 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:26.673+0000 7f12bd270640 1 --2- 172.21.15.46:0/1215725065 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x559c46e97f00 0x559c46e9a330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:29:26.679 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:26.673+0000 7f12a0ff9640 10 client.17586 ms_handle_connect on v2:172.21.15.142:6838/2519143918 2024-03-31T23:29:26.679 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:26.673+0000 7f12bd270640 1 --2- 172.21.15.46:0/1215725065 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x559c46e97f00 0x559c46e9a330 crc :-1 s=READY pgs=234 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c51df4114e909a98 server_cookie=709e6f1b20a8aa91 in_seq=0 out_seq=0 2024-03-31T23:29:26.679 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:26.678+0000 7f12a0ff9640 1 -- 172.21.15.46:0/1215725065 <== mds.0 v2:172.21.15.142:6838/2519143918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x559c46df5120 con 0x559c46e97f00 2024-03-31T23:29:26.680 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:26.678+0000 7f12a0ff9640 10 client.17586 handle_command_reply: tid=0 2024-03-31T23:29:26.680 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:26.678+0000 7f12c2199740 1 client.17586 shutdown 2024-03-31T23:29:26.680 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:26.678+0000 7f12c2199740 1 -- 172.21.15.46:0/1215725065 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559c46e87db0 msgr2=0x559c46e85970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:29:26.680 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:26.678+0000 7f12c2199740 1 --2- 172.21.15.46:0/1215725065 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559c46e87db0 0x559c46e85970 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 crypto rx=0x7f12b000f6a0 tx=0x7f12b0004ab0 comp rx=0 tx=0).stop 2024-03-31T23:29:26.680 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:26.679+0000 7f12c2199740 1 -- 172.21.15.46:0/1215725065 shutdown_connections 2024-03-31T23:29:26.680 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:26.679+0000 7f12c2199740 1 --2- 172.21.15.46:0/1215725065 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x559c46e97f00 0x559c46e9a330 crc :-1 s=READY pgs=234 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:29:26.681 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:26.679+0000 7f12c2199740 1 --2- 172.21.15.46:0/1215725065 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559c46e87db0 0x559c46e85970 unknown :-1 s=CLOSED pgs=943 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:29:26.681 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:26.679+0000 7f12a0ff9640 0 client.17586 ms_handle_reset on v2:172.21.15.142:6838/2519143918 2024-03-31T23:29:26.681 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:26.679+0000 7f12c2199740 1 --2- 172.21.15.46:0/1215725065 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559c46e7efe0 0x559c46e85430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:29:26.681 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:26.679+0000 7f12c2199740 1 --2- 172.21.15.46:0/1215725065 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559c46e7e6c0 0x559c46e84ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:29:26.681 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:26.679+0000 7f12c2199740 1 -- 172.21.15.46:0/1215725065 >> 172.21.15.46:0/1215725065 conn(0x559c46e78fe0 msgr2=0x559c46e7ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:29:26.681 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:26.679+0000 7f12c2199740 1 -- 172.21.15.46:0/1215725065 shutdown_connections 2024-03-31T23:29:26.681 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:26.679+0000 7f12c2199740 1 -- 172.21.15.46:0/1215725065 wait complete. 2024-03-31T23:29:26.681 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:26.680+0000 7f12c2199740 20 client.17586 trim_cache size 0 max 16384 2024-03-31T23:29:26.706 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:26.705+0000 7f12c2199740 1 Processor -- start 2024-03-31T23:29:26.706 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:26.705+0000 7f12c2199740 1 -- start start 2024-03-31T23:29:26.706 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:26.705+0000 7f12c2199740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559c46e7e6c0 0x559c46e87620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:29:26.706 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:26.705+0000 7f12c2199740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559c46e7efe0 0x559c46e93780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:29:26.706 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:26.705+0000 7f12c2199740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559c46e87db0 0x559c46e7ab10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:29:26.706 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:26.705+0000 7f12c2199740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f12b81b14b0 con 0x559c46e87db0 2024-03-31T23:29:26.706 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:26.705+0000 7f12c2199740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f12b81aed80 con 0x559c46e7efe0 2024-03-31T23:29:26.707 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:26.705+0000 7f12c2199740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f12b81acbf0 con 0x559c46e7e6c0 2024-03-31T23:29:26.707 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:26.705+0000 7f12bda71640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559c46e87db0 0x559c46e7ab10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:29:26.707 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:26.705+0000 7f12bd270640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559c46e7e6c0 0x559c46e87620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:29:26.707 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:26.705+0000 7f12bd270640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559c46e7e6c0 0x559c46e87620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.46:58146/0 (socket says 172.21.15.46:58146) 2024-03-31T23:29:26.707 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:26.705+0000 7f12bd270640 1 -- 172.21.15.46:0/1421368112 learned_addr learned my addr 172.21.15.46:0/1421368112 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:29:26.707 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:26.705+0000 7f12bca6f640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559c46e7efe0 0x559c46e93780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:29:26.707 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:26.705+0000 7f12bda71640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559c46e87db0 0x559c46e7ab10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:51216/0 (socket says 172.21.15.46:51216) 2024-03-31T23:29:26.711 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:26.705+0000 7f12bda71640 1 -- 172.21.15.46:0/1421368112 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559c46e7e6c0 msgr2=0x559c46e87620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:29:26.711 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:26.705+0000 7f12bda71640 1 --2- 172.21.15.46:0/1421368112 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559c46e7e6c0 0x559c46e87620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:29:26.711 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:26.705+0000 7f12bda71640 1 -- 172.21.15.46:0/1421368112 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559c46e7efe0 msgr2=0x559c46e93780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:29:26.712 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:26.705+0000 7f12bda71640 1 --2- 172.21.15.46:0/1421368112 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559c46e7efe0 0x559c46e93780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:29:26.712 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:26.705+0000 7f12bda71640 1 -- 172.21.15.46:0/1421368112 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559c46e7a1a0 con 0x559c46e87db0 2024-03-31T23:29:26.712 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:26.706+0000 7f12bda71640 1 --2- 172.21.15.46:0/1421368112 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559c46e87db0 0x559c46e7ab10 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 crypto rx=0x7f12b0000f80 tx=0x7f12b001c660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b8714f4b4c5dc1f server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:29:26.712 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:26.706+0000 7f12a2ffd640 1 -- 172.21.15.46:0/1421368112 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12b001f230 con 0x559c46e87db0 2024-03-31T23:29:26.712 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:26.706+0000 7f12a2ffd640 1 -- 172.21.15.46:0/1421368112 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f12b001fc30 con 0x559c46e87db0 2024-03-31T23:29:26.712 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:26.706+0000 7f12a2ffd640 1 -- 172.21.15.46:0/1421368112 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12b0027a20 con 0x559c46e87db0 2024-03-31T23:29:26.712 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:26.707+0000 7f12c2199740 1 -- 172.21.15.46:0/1421368112 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559c46e87db0 msgr2=0x559c46e7ab10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:29:26.712 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:26.707+0000 7f12c2199740 1 --2- 172.21.15.46:0/1421368112 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559c46e87db0 0x559c46e7ab10 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 crypto rx=0x7f12b0000f80 tx=0x7f12b001c660 comp rx=0 tx=0).stop 2024-03-31T23:29:26.712 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:26.707+0000 7f12c2199740 1 -- 172.21.15.46:0/1421368112 shutdown_connections 2024-03-31T23:29:26.712 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:26.707+0000 7f12c2199740 1 --2- 172.21.15.46:0/1421368112 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559c46e87db0 0x559c46e7ab10 unknown :-1 s=CLOSED pgs=944 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:29:26.712 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:26.707+0000 7f12c2199740 1 --2- 172.21.15.46:0/1421368112 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559c46e7efe0 0x559c46e93780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:29:26.712 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:26.707+0000 7f12c2199740 1 --2- 172.21.15.46:0/1421368112 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559c46e7e6c0 0x559c46e87620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:29:26.713 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:26.707+0000 7f12c2199740 1 -- 172.21.15.46:0/1421368112 >> 172.21.15.46:0/1421368112 conn(0x559c46e78fe0 msgr2=0x559c46e7a400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:29:26.713 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:26.707+0000 7f12c2199740 1 -- 172.21.15.46:0/1421368112 shutdown_connections 2024-03-31T23:29:26.713 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:26.707+0000 7f12c2199740 1 -- 172.21.15.46:0/1421368112 wait complete. 2024-03-31T23:29:26.713 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:26.708+0000 7f12c2199740 1 -- start start 2024-03-31T23:29:26.713 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:26.708+0000 7f12c2199740 1 Processor -- start 2024-03-31T23:29:26.713 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:26.708+0000 7f12c2199740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559c46e7e6c0 0x559c46e8edb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:29:26.713 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:26.708+0000 7f12c2199740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559c46e7efe0 0x559c46e8f2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:29:26.713 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:26.708+0000 7f12c2199740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559c46e87db0 0x559c46e8fa30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:29:26.713 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:26.708+0000 7f12c2199740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f12b81ad240 con 0x559c46e7e6c0 2024-03-31T23:29:26.713 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:26.708+0000 7f12c2199740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f12b81ae6b0 con 0x559c46e7efe0 2024-03-31T23:29:26.713 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:26.708+0000 7f12c2199740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f12b81ad910 con 0x559c46e87db0 2024-03-31T23:29:26.713 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:26.708+0000 7f12c2199740 10 client.0 _ll_register_callbacks cb 0x7f12b0009be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T23:29:26.713 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:26.708+0000 7f12bd270640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559c46e7e6c0 0x559c46e8edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:29:26.713 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:26.708+0000 7f12bd270640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559c46e7e6c0 0x559c46e8edb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:51218/0 (socket says 172.21.15.46:51218) 2024-03-31T23:29:26.714 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:26.708+0000 7f12bd270640 1 -- 172.21.15.46:0/2010621983 learned_addr learned my addr 172.21.15.46:0/2010621983 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:29:26.714 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:26.708+0000 7f12bda71640 1 --2- 172.21.15.46:0/2010621983 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559c46e87db0 0x559c46e8fa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:29:26.714 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:26.708+0000 7f12bca6f640 1 --2- 172.21.15.46:0/2010621983 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559c46e7efe0 0x559c46e8f2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:29:26.714 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:26.709+0000 7f12bd270640 1 -- 172.21.15.46:0/2010621983 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559c46e87db0 msgr2=0x559c46e8fa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:29:26.714 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:26.709+0000 7f12bd270640 1 --2- 172.21.15.46:0/2010621983 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559c46e87db0 0x559c46e8fa30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:29:26.714 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:26.709+0000 7f12bd270640 1 -- 172.21.15.46:0/2010621983 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559c46e7efe0 msgr2=0x559c46e8f2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:29:26.714 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:26.709+0000 7f12bd270640 1 --2- 172.21.15.46:0/2010621983 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559c46e7efe0 0x559c46e8f2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:29:26.714 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:26.709+0000 7f12bd270640 1 -- 172.21.15.46:0/2010621983 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559c46e90230 con 0x559c46e7e6c0 2024-03-31T23:29:26.714 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:26.709+0000 7f12bda71640 1 --2- 172.21.15.46:0/2010621983 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559c46e87db0 0x559c46e8fa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:29:26.714 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:26.709+0000 7f12bd270640 1 --2- 172.21.15.46:0/2010621983 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559c46e7e6c0 0x559c46e8edb0 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 crypto rx=0x7f12b4009c80 tx=0x7f12b4009e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e3240a1d19aeba3 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:29:26.714 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:26.709+0000 7f12a0ff9640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-31T23:29:26.714 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:26.709+0000 7f12a0ff9640 1 -- 172.21.15.46:0/2010621983 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12b40050b0 con 0x559c46e7e6c0 2024-03-31T23:29:26.714 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:26.709+0000 7f12a0ff9640 1 -- 172.21.15.46:0/2010621983 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(122..122 src has 1..122) ==== 9554+0+0 (secure 0 0 0) 0x7f12b4005250 con 0x559c46e7e6c0 2024-03-31T23:29:26.715 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:26.709+0000 7f12c2199740 1 -- 172.21.15.46:0/2010621983 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f12b001fc30 con 0x559c46e7e6c0 2024-03-31T23:29:26.715 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:26.709+0000 7f12a0ff9640 1 -- 172.21.15.46:0/2010621983 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) ==== 24+0+0 (secure 0 0 0) 0x7f12b4039050 con 0x559c46e7e6c0 2024-03-31T23:29:26.715 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:26.709+0000 7f12c2199740 10 client.17592 fetch_fsmap learned FSMap version 47 2024-03-31T23:29:26.715 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:26.709+0000 7f12c2199740 1 -- 172.21.15.46:0/2010621983 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) -- 0x559c46f033e0 con 0x559c46e7e6c0 2024-03-31T23:29:26.715 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:26.709+0000 7f12a0ff9640 1 -- 172.21.15.46:0/2010621983 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f12b001fc30 con 0x559c46e7e6c0 2024-03-31T23:29:26.715 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:26.710+0000 7f12a0ff9640 1 -- 172.21.15.46:0/2010621983 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 47) ==== 8346+0+0 (secure 0 0 0) 0x7f12b400fa80 con 0x559c46e7e6c0 2024-03-31T23:29:26.715 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:26.710+0000 7f12c2199740 10 client.17592 fetch_fsmap finished waiting for FSMap version 47 2024-03-31T23:29:26.715 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:26.710+0000 7f12c2199740 10 client.17592 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-31T23:29:26.715 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:26.710+0000 7f12c2199740 20 client.17592 populate_metadata read hostname 'smithi046' 2024-03-31T23:29:26.715 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:26.710+0000 7f12c2199740 1 --2- 172.21.15.46:0/2010621983 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x559c46e97f00 0x559c46f058c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:29:26.715 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:26.710+0000 7f12c2199740 4 client.17592 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-31T23:29:26.715 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:26.710+0000 7f12c2199740 1 -- 172.21.15.46:0/2010621983 --> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] -- command(tid 0: {"prefix": "flush journal"}) -- 0x559c46deee30 con 0x559c46e97f00 2024-03-31T23:29:26.715 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:26.710+0000 7f12bca6f640 1 --2- 172.21.15.46:0/2010621983 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x559c46e97f00 0x559c46f058c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:29:26.715 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:26.710+0000 7f12a0ff9640 10 client.17592 ms_handle_connect on v2:172.21.15.142:6838/2519143918 2024-03-31T23:29:26.715 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:26.710+0000 7f12bca6f640 1 --2- 172.21.15.46:0/2010621983 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x559c46e97f00 0x559c46f058c0 crc :-1 s=READY pgs=235 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=efe94fb23d3ce7ad server_cookie=8887ec9a055f8a1b in_seq=0 out_seq=0 2024-03-31T23:29:26.898 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:26.896+0000 7f12a0ff9640 1 -- 172.21.15.46:0/2010621983 <== mds.0 v2:172.21.15.142:6838/2519143918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x559c46deee30 con 0x559c46e97f00 2024-03-31T23:29:26.898 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:26.896+0000 7f12a0ff9640 10 client.17592 handle_command_reply: tid=0 2024-03-31T23:29:26.898 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:26.896+0000 7f12c2199740 1 client.17592 shutdown 2024-03-31T23:29:26.898 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:26.896+0000 7f12c2199740 1 -- 172.21.15.46:0/2010621983 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559c46e7e6c0 msgr2=0x559c46e8edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:29:26.898 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:26.896+0000 7f12c2199740 1 --2- 172.21.15.46:0/2010621983 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559c46e7e6c0 0x559c46e8edb0 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 crypto rx=0x7f12b4009c80 tx=0x7f12b4009e90 comp rx=0 tx=0).stop 2024-03-31T23:29:26.898 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:26.896+0000 7f12c2199740 1 -- 172.21.15.46:0/2010621983 shutdown_connections 2024-03-31T23:29:26.898 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:26.896+0000 7f12c2199740 1 --2- 172.21.15.46:0/2010621983 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x559c46e97f00 0x559c46f058c0 crc :-1 s=READY pgs=235 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:29:26.898 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:26.896+0000 7f12c2199740 1 --2- 172.21.15.46:0/2010621983 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559c46e87db0 0x559c46e8fa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:29:26.898 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:26.896+0000 7f12c2199740 1 --2- 172.21.15.46:0/2010621983 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x559c46e7efe0 0x559c46e8f2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:29:26.899 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:26.896+0000 7f12c2199740 1 --2- 172.21.15.46:0/2010621983 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559c46e7e6c0 0x559c46e8edb0 unknown :-1 s=CLOSED pgs=945 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:29:26.899 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:26.896+0000 7f12c2199740 1 -- 172.21.15.46:0/2010621983 >> 172.21.15.46:0/2010621983 conn(0x559c46e78fe0 msgr2=0x559c46e77f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:29:26.899 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:26.896+0000 7f12a0ff9640 0 client.17592 ms_handle_reset on v2:172.21.15.142:6838/2519143918 2024-03-31T23:29:26.899 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:26.897+0000 7f12c2199740 1 -- 172.21.15.46:0/2010621983 shutdown_connections 2024-03-31T23:29:26.899 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:26.897+0000 7f12c2199740 1 -- 172.21.15.46:0/2010621983 wait complete. 2024-03-31T23:29:26.899 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:26.897+0000 7f12c2199740 20 client.17592 trim_cache size 0 max 16384 2024-03-31T23:29:26.899 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout:{ 2024-03-31T23:29:26.899 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout: "message": "", 2024-03-31T23:29:26.899 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout: "return_code": 0 2024-03-31T23:29:26.899 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout:} 2024-03-31T23:29:26.900 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:26.897+0000 7f12bf4d4640 1 -- 172.21.15.46:0/3595153635 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f12a40a26b0 msgr2=0x7f12a40a4b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:29:26.900 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:26.898+0000 7f12bf4d4640 1 --2- 172.21.15.46:0/3595153635 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f12a40a26b0 0x7f12a40a4b70 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f12b0009fd0 tx=0x7f12b0009500 comp rx=0 tx=0).stop 2024-03-31T23:29:26.900 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:26.898+0000 7f12bf4d4640 1 -- 172.21.15.46:0/3595153635 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f12b8103250 msgr2=0x7f12b8198420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:29:26.901 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:26.898+0000 7f12bf4d4640 1 --2- 172.21.15.46:0/3595153635 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f12b8103250 0x7f12b8198420 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 crypto rx=0x7f12a8005940 tx=0x7f12a802c5f0 comp rx=0 tx=0).stop 2024-03-31T23:29:26.901 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:26.898+0000 7f12bf4d4640 1 -- 172.21.15.46:0/3595153635 shutdown_connections 2024-03-31T23:29:26.901 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:26.898+0000 7f12bf4d4640 1 --2- 172.21.15.46:0/3595153635 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f12a40a26b0 0x7f12a40a4b70 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:29:26.901 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:26.898+0000 7f12bf4d4640 1 --2- 172.21.15.46:0/3595153635 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f12b810bda0 0x7f12b819c750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:29:26.901 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:26.898+0000 7f12bf4d4640 1 --2- 172.21.15.46:0/3595153635 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f12b8103250 0x7f12b8198420 unknown :-1 s=CLOSED pgs=941 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:29:26.901 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:26.898+0000 7f12bf4d4640 1 --2- 172.21.15.46:0/3595153635 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f12b8102930 0x7f12b8197ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:29:26.901 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:26.898+0000 7f12bf4d4640 1 -- 172.21.15.46:0/3595153635 >> 172.21.15.46:0/3595153635 conn(0x7f12b8077780 msgr2=0x7f12b80fe950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:29:26.901 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:26.898+0000 7f12bf4d4640 1 -- 172.21.15.46:0/3595153635 shutdown_connections 2024-03-31T23:29:26.901 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:26.898+0000 7f12bf4d4640 1 -- 172.21.15.46:0/3595153635 wait complete. 2024-03-31T23:29:27.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:29:26 smithi162 ceph-mon[36237]: pgmap v560: 129 pgs: 129 active+clean; 764 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 6.2 MiB/s wr, 387 op/s 2024-03-31T23:29:27.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:29:26 smithi046 ceph-mon[32498]: pgmap v560: 129 pgs: 129 active+clean; 764 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 6.2 MiB/s wr, 387 op/s 2024-03-31T23:29:27.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:29:26 smithi142 ceph-mon[35271]: pgmap v560: 129 pgs: 129 active+clean; 764 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 6.2 MiB/s wr, 387 op/s 2024-03-31T23:29:27.289 INFO:tasks.workunit.client.0.smithi046.stdout: 10 43886 8.61 MB/sec execute 152 sec latency 169.073 ms 2024-03-31T23:29:28.289 INFO:tasks.workunit.client.0.smithi046.stdout: 10 44061 8.63 MB/sec execute 153 sec latency 113.112 ms 2024-03-31T23:29:29.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:29:28 smithi162 ceph-mon[36237]: pgmap v561: 129 pgs: 129 active+clean; 764 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 4.4 MiB/s wr, 226 op/s 2024-03-31T23:29:29.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:29:28 smithi046 ceph-mon[32498]: pgmap v561: 129 pgs: 129 active+clean; 764 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 4.4 MiB/s wr, 226 op/s 2024-03-31T23:29:29.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:29:28 smithi142 ceph-mon[35271]: pgmap v561: 129 pgs: 129 active+clean; 764 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 4.4 MiB/s wr, 226 op/s 2024-03-31T23:29:29.289 INFO:tasks.workunit.client.0.smithi046.stdout: 10 44230 8.61 MB/sec execute 154 sec latency 69.389 ms 2024-03-31T23:29:30.289 INFO:tasks.workunit.client.0.smithi046.stdout: 10 44361 8.59 MB/sec execute 155 sec latency 71.199 ms 2024-03-31T23:29:31.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:29:30 smithi162 ceph-mon[36237]: pgmap v562: 129 pgs: 129 active+clean; 767 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 6.9 MiB/s wr, 392 op/s 2024-03-31T23:29:31.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:29:30 smithi046 ceph-mon[32498]: pgmap v562: 129 pgs: 129 active+clean; 767 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 6.9 MiB/s wr, 392 op/s 2024-03-31T23:29:31.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:29:30 smithi142 ceph-mon[35271]: pgmap v562: 129 pgs: 129 active+clean; 767 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 6.9 MiB/s wr, 392 op/s 2024-03-31T23:29:31.289 INFO:tasks.workunit.client.0.smithi046.stdout: 10 44470 8.57 MB/sec execute 156 sec latency 155.833 ms 2024-03-31T23:29:32.289 INFO:tasks.workunit.client.0.smithi046.stdout: 10 44626 8.58 MB/sec execute 157 sec latency 138.487 ms 2024-03-31T23:29:33.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:29:32 smithi162 ceph-mon[36237]: pgmap v563: 129 pgs: 129 active+clean; 767 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 4.6 MiB/s wr, 269 op/s 2024-03-31T23:29:33.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:29:32 smithi046 ceph-mon[32498]: pgmap v563: 129 pgs: 129 active+clean; 767 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 4.6 MiB/s wr, 269 op/s 2024-03-31T23:29:33.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:29:32 smithi142 ceph-mon[35271]: pgmap v563: 129 pgs: 129 active+clean; 767 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 4.6 MiB/s wr, 269 op/s 2024-03-31T23:29:33.289 INFO:tasks.workunit.client.0.smithi046.stdout: 10 44785 8.57 MB/sec execute 158 sec latency 63.474 ms 2024-03-31T23:29:34.289 INFO:tasks.workunit.client.0.smithi046.stdout: 10 44953 8.57 MB/sec execute 159 sec latency 110.983 ms 2024-03-31T23:29:35.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:29:34 smithi162 ceph-mon[36237]: pgmap v564: 129 pgs: 129 active+clean; 767 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 4.6 MiB/s wr, 269 op/s 2024-03-31T23:29:35.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:29:34 smithi046 ceph-mon[32498]: pgmap v564: 129 pgs: 129 active+clean; 767 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 4.6 MiB/s wr, 269 op/s 2024-03-31T23:29:35.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:29:34 smithi142 ceph-mon[35271]: pgmap v564: 129 pgs: 129 active+clean; 767 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 4.6 MiB/s wr, 269 op/s 2024-03-31T23:29:35.289 INFO:tasks.workunit.client.0.smithi046.stdout: 10 45118 8.56 MB/sec execute 160 sec latency 82.535 ms 2024-03-31T23:29:36.289 INFO:tasks.workunit.client.0.smithi046.stdout: 10 45276 8.55 MB/sec execute 161 sec latency 73.812 ms 2024-03-31T23:29:37.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:29:36 smithi162 ceph-mon[36237]: pgmap v565: 129 pgs: 129 active+clean; 770 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 6.5 MiB/s wr, 380 op/s 2024-03-31T23:29:37.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:29:36 smithi046 ceph-mon[32498]: pgmap v565: 129 pgs: 129 active+clean; 770 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 6.5 MiB/s wr, 380 op/s 2024-03-31T23:29:37.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:29:36 smithi142 ceph-mon[35271]: pgmap v565: 129 pgs: 129 active+clean; 770 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 6.5 MiB/s wr, 380 op/s 2024-03-31T23:29:37.289 INFO:tasks.workunit.client.0.smithi046.stdout: 10 45430 8.56 MB/sec execute 162 sec latency 62.761 ms 2024-03-31T23:29:38.289 INFO:tasks.workunit.client.0.smithi046.stdout: 10 45571 8.53 MB/sec execute 163 sec latency 77.926 ms 2024-03-31T23:29:39.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:29:38 smithi162 ceph-mon[36237]: pgmap v566: 129 pgs: 129 active+clean; 770 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 4.3 MiB/s wr, 276 op/s 2024-03-31T23:29:39.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:29:38 smithi046 ceph-mon[32498]: pgmap v566: 129 pgs: 129 active+clean; 770 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 4.3 MiB/s wr, 276 op/s 2024-03-31T23:29:39.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:29:38 smithi142 ceph-mon[35271]: pgmap v566: 129 pgs: 129 active+clean; 770 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 4.3 MiB/s wr, 276 op/s 2024-03-31T23:29:39.290 INFO:tasks.workunit.client.0.smithi046.stdout: 10 45745 8.53 MB/sec execute 164 sec latency 69.673 ms 2024-03-31T23:29:40.024 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:40.022+0000 7f4276769640 1 -- 172.21.15.46:0/1050412771 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4270069770 msgr2=0x7f427010e5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:29:40.025 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:40.022+0000 7f4276769640 1 --2- 172.21.15.46:0/1050412771 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4270069770 0x7f427010e5e0 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 crypto rx=0x7f4268009cd0 tx=0x7f426802c0f0 comp rx=0 tx=0).stop 2024-03-31T23:29:40.025 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:40.022+0000 7f4276769640 1 -- 172.21.15.46:0/1050412771 shutdown_connections 2024-03-31T23:29:40.025 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:40.022+0000 7f4276769640 1 --2- 172.21.15.46:0/1050412771 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4270069770 0x7f427010e5e0 unknown :-1 s=CLOSED pgs=946 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:29:40.025 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:40.022+0000 7f4276769640 1 --2- 172.21.15.46:0/1050412771 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4270068dd0 0x7f4270069230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:29:40.025 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:40.022+0000 7f4276769640 1 --2- 172.21.15.46:0/1050412771 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4270107820 0x7f4270107c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:29:40.025 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:40.023+0000 7f4276769640 1 -- 172.21.15.46:0/1050412771 >> 172.21.15.46:0/1050412771 conn(0x7f42700774a0 msgr2=0x7f42700778b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:29:40.025 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:40.023+0000 7f4276769640 1 -- 172.21.15.46:0/1050412771 shutdown_connections 2024-03-31T23:29:40.025 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:40.023+0000 7f4276769640 1 -- 172.21.15.46:0/1050412771 wait complete. 2024-03-31T23:29:40.025 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:40.023+0000 7f4276769640 1 Processor -- start 2024-03-31T23:29:40.025 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:40.023+0000 7f4276769640 1 -- start start 2024-03-31T23:29:40.027 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:40.024+0000 7f4276769640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4270068dd0 0x7f4270100970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:29:40.027 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:40.024+0000 7f4276769640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4270069770 0x7f4270100eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:29:40.027 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:40.024+0000 7f4276769640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4270107820 0x7f42701013f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:29:40.027 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:40.024+0000 7f4276769640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f42701119c0 con 0x7f4270068dd0 2024-03-31T23:29:40.027 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:40.024+0000 7f4276769640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f4270111840 con 0x7f4270069770 2024-03-31T23:29:40.028 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:40.024+0000 7f4276769640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f4270111b40 con 0x7f4270107820 2024-03-31T23:29:40.028 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:40.024+0000 7f426ffff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4270068dd0 0x7f4270100970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:29:40.028 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:40.024+0000 7f426ffff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4270068dd0 0x7f4270100970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:58470/0 (socket says 172.21.15.46:58470) 2024-03-31T23:29:40.028 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:40.024+0000 7f426ffff640 1 -- 172.21.15.46:0/3106041902 learned_addr learned my addr 172.21.15.46:0/3106041902 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:29:40.028 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:40.024+0000 7f4274d06640 1 --2- 172.21.15.46:0/3106041902 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4270107820 0x7f42701013f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:29:40.028 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:40.024+0000 7f426f7fe640 1 --2- 172.21.15.46:0/3106041902 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4270069770 0x7f4270100eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:29:40.028 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:40.024+0000 7f426ffff640 1 -- 172.21.15.46:0/3106041902 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4270107820 msgr2=0x7f42701013f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:29:40.028 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:40.024+0000 7f426ffff640 1 --2- 172.21.15.46:0/3106041902 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4270107820 0x7f42701013f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:29:40.028 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:40.024+0000 7f426ffff640 1 -- 172.21.15.46:0/3106041902 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4270069770 msgr2=0x7f4270100eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:29:40.028 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:40.024+0000 7f426ffff640 1 --2- 172.21.15.46:0/3106041902 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4270069770 0x7f4270100eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:29:40.028 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:40.024+0000 7f426ffff640 1 -- 172.21.15.46:0/3106041902 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f42701a6260 con 0x7f4270068dd0 2024-03-31T23:29:40.028 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:40.025+0000 7f426ffff640 1 --2- 172.21.15.46:0/3106041902 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4270068dd0 0x7f4270100970 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 crypto rx=0x7f426000e9f0 tx=0x7f426000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:29:40.028 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:40.025+0000 7f426d7fa640 1 -- 172.21.15.46:0/3106041902 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f426000cd80 con 0x7f4270068dd0 2024-03-31T23:29:40.029 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:40.025+0000 7f4276769640 1 -- 172.21.15.46:0/3106041902 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f42701a6550 con 0x7f4270068dd0 2024-03-31T23:29:40.029 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:40.025+0000 7f4276769640 1 -- 172.21.15.46:0/3106041902 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f42701a6af0 con 0x7f4270068dd0 2024-03-31T23:29:40.029 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:40.025+0000 7f426d7fa640 1 -- 172.21.15.46:0/3106041902 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f42600108e0 con 0x7f4270068dd0 2024-03-31T23:29:40.029 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:40.026+0000 7f426d7fa640 1 -- 172.21.15.46:0/3106041902 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4260018920 con 0x7f4270068dd0 2024-03-31T23:29:40.029 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:40.028+0000 7f426d7fa640 1 -- 172.21.15.46:0/3106041902 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 25) ==== 137451+0+0 (secure 0 0 0) 0x7f4260018ac0 con 0x7f4270068dd0 2024-03-31T23:29:40.029 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:40.028+0000 7f4279422740 1 Processor -- start 2024-03-31T23:29:40.030 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:40.029+0000 7f4279422740 1 -- start start 2024-03-31T23:29:40.030 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:40.029+0000 7f426d7fa640 1 --2- 172.21.15.46:0/3106041902 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f42580a2760 0x7f42580a4c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:29:40.030 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:40.029+0000 7f4279422740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563a657171e0 0x563a657175c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:29:40.030 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:40.029+0000 7f4279422740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563a65717b00 0x563a65729080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:29:40.030 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:40.029+0000 7f4279422740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x563a657295c0 0x563a6572b9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:29:40.030 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:40.029+0000 7f4279422740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x563a6567a8f0 con 0x563a657171e0 2024-03-31T23:29:40.030 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:40.029+0000 7f4279422740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x563a656238d0 con 0x563a657295c0 2024-03-31T23:29:40.031 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:40.029+0000 7f4279422740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x563a65676990 con 0x563a65717b00 2024-03-31T23:29:40.031 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:40.029+0000 7f426d7fa640 1 -- 172.21.15.46:0/3106041902 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(122..122 src has 1..122) ==== 9554+0+0 (secure 0 0 0) 0x7f42600cf050 con 0x7f4270068dd0 2024-03-31T23:29:40.031 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:40.029+0000 7f4274d06640 1 --2- 172.21.15.46:0/3106041902 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f42580a2760 0x7f42580a4c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:29:40.031 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:40.029+0000 7f4274d06640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x563a657295c0 0x563a6572b9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:29:40.031 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:40.029+0000 7f4274d06640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x563a657295c0 0x563a6572b9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.46:56284/0 (socket says 172.21.15.46:56284) 2024-03-31T23:29:40.031 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:40.029+0000 7f4274d06640 1 -- 172.21.15.46:0/239874371 learned_addr learned my addr 172.21.15.46:0/239874371 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:29:40.032 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:40.030+0000 7f426ffff640 1 --2- 172.21.15.46:0/239874371 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563a657171e0 0x563a657175c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:29:40.032 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:40.030+0000 7f4274d06640 1 --2- 172.21.15.46:0/3106041902 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f42580a2760 0x7f42580a4c20 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f4268005e30 tx=0x7f4268032e60 comp rx=0 tx=0).ready entity=mgr.15654 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:29:40.032 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:40.030+0000 7f4274d06640 1 -- 172.21.15.46:0/239874371 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563a65717b00 msgr2=0x563a65729080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:29:40.032 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:40.030+0000 7f426f7fe640 1 --2- 172.21.15.46:0/239874371 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563a65717b00 0x563a65729080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:29:40.032 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:40.030+0000 7f4274d06640 1 --2- 172.21.15.46:0/239874371 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563a65717b00 0x563a65729080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:29:40.032 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:40.030+0000 7f4274d06640 1 -- 172.21.15.46:0/239874371 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563a657171e0 msgr2=0x563a657175c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:29:40.032 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:40.030+0000 7f4274d06640 1 --2- 172.21.15.46:0/239874371 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563a657171e0 0x563a657175c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:29:40.033 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:40.030+0000 7f4274d06640 1 -- 172.21.15.46:0/239874371 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563a6538f210 con 0x563a657295c0 2024-03-31T23:29:40.033 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:40.030+0000 7f426ffff640 1 --2- 172.21.15.46:0/239874371 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563a657171e0 0x563a657175c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:29:40.033 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:40.030+0000 7f426f7fe640 1 --2- 172.21.15.46:0/239874371 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563a65717b00 0x563a65729080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T23:29:40.033 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:40.031+0000 7f4274d06640 1 --2- 172.21.15.46:0/239874371 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x563a657295c0 0x563a6572b9b0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f4268009ca0 tx=0x7f4268048000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e057ac47b6841b6c server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:29:40.033 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:40.031+0000 7f42567fc640 1 -- 172.21.15.46:0/239874371 <== mon.2 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4268039040 con 0x563a657295c0 2024-03-31T23:29:40.033 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:40.031+0000 7f42567fc640 1 -- 172.21.15.46:0/239874371 <== mon.2 v2:172.21.15.142:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4268052070 con 0x563a657295c0 2024-03-31T23:29:40.033 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:40.031+0000 7f42567fc640 1 -- 172.21.15.46:0/239874371 <== mon.2 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4268034a40 con 0x563a657295c0 2024-03-31T23:29:40.036 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:40.032+0000 7f4279422740 1 -- 172.21.15.46:0/239874371 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x563a657295c0 msgr2=0x563a6572b9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:29:40.036 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:40.032+0000 7f4279422740 1 --2- 172.21.15.46:0/239874371 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x563a657295c0 0x563a6572b9b0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f4268009ca0 tx=0x7f4268048000 comp rx=0 tx=0).stop 2024-03-31T23:29:40.037 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:40.032+0000 7f4279422740 1 -- 172.21.15.46:0/239874371 shutdown_connections 2024-03-31T23:29:40.037 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:40.032+0000 7f4279422740 1 --2- 172.21.15.46:0/239874371 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x563a657295c0 0x563a6572b9b0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:29:40.037 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:40.032+0000 7f4279422740 1 --2- 172.21.15.46:0/239874371 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563a65717b00 0x563a65729080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:29:40.037 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:40.032+0000 7f4279422740 1 --2- 172.21.15.46:0/239874371 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563a657171e0 0x563a657175c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:29:40.037 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:40.032+0000 7f4279422740 1 -- 172.21.15.46:0/239874371 >> 172.21.15.46:0/239874371 conn(0x563a65711fe0 msgr2=0x563a657123c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:29:40.037 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:40.032+0000 7f4279422740 1 -- 172.21.15.46:0/239874371 shutdown_connections 2024-03-31T23:29:40.037 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:40.032+0000 7f4279422740 1 -- 172.21.15.46:0/239874371 wait complete. 2024-03-31T23:29:40.037 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:40.033+0000 7f4279422740 1 -- start start 2024-03-31T23:29:40.037 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:40.033+0000 7f4279422740 1 Processor -- start 2024-03-31T23:29:40.037 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:40.033+0000 7f4279422740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563a657171e0 0x563a65726540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:29:40.037 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:40.033+0000 7f4279422740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563a65717b00 0x563a65726a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:29:40.037 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:40.033+0000 7f4279422740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x563a657295c0 0x563a65726fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:29:40.037 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:40.033+0000 7f4279422740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x563a6572c120 con 0x563a657171e0 2024-03-31T23:29:40.037 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:40.033+0000 7f4279422740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x563a65618f80 con 0x563a657295c0 2024-03-31T23:29:40.038 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:40.033+0000 7f4279422740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x563a6572c2a0 con 0x563a65717b00 2024-03-31T23:29:40.038 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:40.033+0000 7f4279422740 10 client.0 _ll_register_callbacks cb 0x563a65671290 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T23:29:40.038 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:40.033+0000 7f426ffff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563a657171e0 0x563a65726540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:29:40.038 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:40.033+0000 7f426ffff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563a657171e0 0x563a65726540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:58496/0 (socket says 172.21.15.46:58496) 2024-03-31T23:29:40.038 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:40.033+0000 7f426ffff640 1 -- 172.21.15.46:0/3881171744 learned_addr learned my addr 172.21.15.46:0/3881171744 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:29:40.038 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:40.033+0000 7f426f7fe640 1 --2- 172.21.15.46:0/3881171744 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563a65717b00 0x563a65726a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:29:40.038 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:40.034+0000 7f4274d06640 1 --2- 172.21.15.46:0/3881171744 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x563a657295c0 0x563a65726fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:29:40.038 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:40.034+0000 7f426ffff640 1 -- 172.21.15.46:0/3881171744 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563a65717b00 msgr2=0x563a65726a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:29:40.038 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:40.034+0000 7f426ffff640 1 --2- 172.21.15.46:0/3881171744 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563a65717b00 0x563a65726a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:29:40.038 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:40.034+0000 7f426ffff640 1 -- 172.21.15.46:0/3881171744 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x563a657295c0 msgr2=0x563a65726fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:29:40.038 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:40.034+0000 7f426ffff640 1 --2- 172.21.15.46:0/3881171744 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x563a657295c0 0x563a65726fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:29:40.038 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:40.034+0000 7f426ffff640 1 -- 172.21.15.46:0/3881171744 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563a6545b620 con 0x563a657171e0 2024-03-31T23:29:40.038 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:40.034+0000 7f426ffff640 1 --2- 172.21.15.46:0/3881171744 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563a657171e0 0x563a65726540 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 crypto rx=0x7f426000c6a0 tx=0x7f426009f5d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e3a05878dd82f server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:29:40.038 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:40.034+0000 7f424f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-31T23:29:40.039 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:40.034+0000 7f424f7fe640 1 -- 172.21.15.46:0/3881171744 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42600b4060 con 0x563a657171e0 2024-03-31T23:29:40.039 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:40.034+0000 7f424f7fe640 1 -- 172.21.15.46:0/3881171744 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(122..122 src has 1..122) ==== 9554+0+0 (secure 0 0 0) 0x7f4260010430 con 0x563a657171e0 2024-03-31T23:29:40.039 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:40.034+0000 7f4279422740 1 -- 172.21.15.46:0/3881171744 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4268052070 con 0x563a657171e0 2024-03-31T23:29:40.039 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:40.034+0000 7f424f7fe640 1 -- 172.21.15.46:0/3881171744 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) ==== 24+0+0 (secure 0 0 0) 0x7f4260097c60 con 0x563a657171e0 2024-03-31T23:29:40.039 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:40.034+0000 7f4279422740 10 client.17604 fetch_fsmap learned FSMap version 47 2024-03-31T23:29:40.039 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:40.035+0000 7f4279422740 1 -- 172.21.15.46:0/3881171744 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) -- 0x563a65738010 con 0x563a657171e0 2024-03-31T23:29:40.039 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:40.035+0000 7f424f7fe640 1 -- 172.21.15.46:0/3881171744 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4268052070 con 0x563a657171e0 2024-03-31T23:29:40.039 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:40.035+0000 7f424f7fe640 1 -- 172.21.15.46:0/3881171744 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 47) ==== 8346+0+0 (secure 0 0 0) 0x7f42600ccc70 con 0x563a657171e0 2024-03-31T23:29:40.039 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:40.035+0000 7f4279422740 10 client.17604 fetch_fsmap finished waiting for FSMap version 47 2024-03-31T23:29:40.039 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:40.035+0000 7f4279422740 10 client.17604 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-31T23:29:40.039 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:40.035+0000 7f4279422740 20 client.17604 populate_metadata read hostname 'smithi046' 2024-03-31T23:29:40.039 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:40.035+0000 7f4279422740 1 --2- 172.21.15.46:0/3881171744 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x563a65739520 0x563a6573b910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:29:40.039 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:40.035+0000 7f4279422740 4 client.17604 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-31T23:29:40.040 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:40.035+0000 7f4279422740 1 -- 172.21.15.46:0/3881171744 --> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563a6568e120 con 0x563a65739520 2024-03-31T23:29:40.040 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:40.035+0000 7f426f7fe640 1 --2- 172.21.15.46:0/3881171744 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x563a65739520 0x563a6573b910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:29:40.040 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:40.036+0000 7f426f7fe640 1 --2- 172.21.15.46:0/3881171744 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x563a65739520 0x563a6573b910 crc :-1 s=READY pgs=236 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a798fdc23c16592e server_cookie=29f1bfa9d4f5d1d1 in_seq=0 out_seq=0 2024-03-31T23:29:40.040 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:40.036+0000 7f424f7fe640 10 client.17604 ms_handle_connect on v2:172.21.15.142:6838/2519143918 2024-03-31T23:29:40.041 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:40.037+0000 7f424f7fe640 1 -- 172.21.15.46:0/3881171744 <== mds.0 v2:172.21.15.142:6838/2519143918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x563a6568e120 con 0x563a65739520 2024-03-31T23:29:40.041 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:40.037+0000 7f424f7fe640 10 client.17604 handle_command_reply: tid=0 2024-03-31T23:29:40.041 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:40.037+0000 7f4279422740 1 client.17604 shutdown 2024-03-31T23:29:40.041 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:40.037+0000 7f4279422740 1 -- 172.21.15.46:0/3881171744 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563a657171e0 msgr2=0x563a65726540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:29:40.041 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:40.037+0000 7f4279422740 1 --2- 172.21.15.46:0/3881171744 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563a657171e0 0x563a65726540 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 crypto rx=0x7f426000c6a0 tx=0x7f426009f5d0 comp rx=0 tx=0).stop 2024-03-31T23:29:40.042 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:40.037+0000 7f4279422740 1 -- 172.21.15.46:0/3881171744 shutdown_connections 2024-03-31T23:29:40.042 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:40.037+0000 7f4279422740 1 --2- 172.21.15.46:0/3881171744 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x563a65739520 0x563a6573b910 crc :-1 s=READY pgs=236 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:29:40.042 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:40.037+0000 7f4279422740 1 --2- 172.21.15.46:0/3881171744 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x563a657295c0 0x563a65726fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:29:40.042 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:40.037+0000 7f424f7fe640 0 client.17604 ms_handle_reset on v2:172.21.15.142:6838/2519143918 2024-03-31T23:29:40.042 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:40.037+0000 7f4279422740 1 --2- 172.21.15.46:0/3881171744 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563a65717b00 0x563a65726a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:29:40.042 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:40.037+0000 7f4279422740 1 --2- 172.21.15.46:0/3881171744 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563a657171e0 0x563a65726540 unknown :-1 s=CLOSED pgs=948 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:29:40.042 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:40.037+0000 7f4279422740 1 -- 172.21.15.46:0/3881171744 >> 172.21.15.46:0/3881171744 conn(0x563a65711fe0 msgr2=0x563a65713d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:29:40.042 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:40.037+0000 7f4279422740 1 -- 172.21.15.46:0/3881171744 shutdown_connections 2024-03-31T23:29:40.042 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:40.037+0000 7f4279422740 1 -- 172.21.15.46:0/3881171744 wait complete. 2024-03-31T23:29:40.042 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:40.037+0000 7f4279422740 20 client.17604 trim_cache size 0 max 16384 2024-03-31T23:29:40.061 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:40.059+0000 7f4279422740 1 Processor -- start 2024-03-31T23:29:40.061 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:40.059+0000 7f4279422740 1 -- start start 2024-03-31T23:29:40.061 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:40.059+0000 7f4279422740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563a657171e0 0x563a65734f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:29:40.061 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:40.059+0000 7f4279422740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563a65717b00 0x563a65741200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:29:40.061 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:40.059+0000 7f4279422740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x563a657295c0 0x563a657106f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:29:40.061 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:40.059+0000 7f4279422740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f425c012950 con 0x563a65717b00 2024-03-31T23:29:40.061 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:40.059+0000 7f4279422740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f425c010220 con 0x563a657295c0 2024-03-31T23:29:40.061 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:40.059+0000 7f4279422740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f425c00e090 con 0x563a657171e0 2024-03-31T23:29:40.062 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:40.059+0000 7f426f7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563a65717b00 0x563a65741200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:29:40.062 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:40.059+0000 7f426f7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563a65717b00 0x563a65741200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:58500/0 (socket says 172.21.15.46:58500) 2024-03-31T23:29:40.062 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:40.059+0000 7f426f7fe640 1 -- 172.21.15.46:0/2211250744 learned_addr learned my addr 172.21.15.46:0/2211250744 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:29:40.062 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:40.059+0000 7f4274d06640 1 --2- 172.21.15.46:0/2211250744 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x563a657295c0 0x563a657106f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:29:40.062 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:40.059+0000 7f426ffff640 1 --2- 172.21.15.46:0/2211250744 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563a657171e0 0x563a65734f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:29:40.062 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:40.059+0000 7f426f7fe640 1 -- 172.21.15.46:0/2211250744 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563a657171e0 msgr2=0x563a65734f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:29:40.062 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:40.059+0000 7f426f7fe640 1 --2- 172.21.15.46:0/2211250744 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563a657171e0 0x563a65734f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:29:40.062 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:40.059+0000 7f426f7fe640 1 -- 172.21.15.46:0/2211250744 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x563a657295c0 msgr2=0x563a657106f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:29:40.062 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:40.059+0000 7f426f7fe640 1 --2- 172.21.15.46:0/2211250744 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x563a657295c0 0x563a657106f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:29:40.062 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:40.060+0000 7f426f7fe640 1 -- 172.21.15.46:0/2211250744 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563a6573bf70 con 0x563a65717b00 2024-03-31T23:29:40.062 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:40.060+0000 7f426f7fe640 1 --2- 172.21.15.46:0/2211250744 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563a65717b00 0x563a65741200 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 crypto rx=0x7f42640095f0 tx=0x7f4264012820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e29c6fba0e683459 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:29:40.062 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:40.060+0000 7f4255ffb640 1 -- 172.21.15.46:0/2211250744 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f426401dd20 con 0x563a65717b00 2024-03-31T23:29:40.062 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:40.060+0000 7f4255ffb640 1 -- 172.21.15.46:0/2211250744 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4264022070 con 0x563a65717b00 2024-03-31T23:29:40.063 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:40.060+0000 7f4255ffb640 1 -- 172.21.15.46:0/2211250744 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4264003970 con 0x563a65717b00 2024-03-31T23:29:40.066 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:40.061+0000 7f4279422740 1 -- 172.21.15.46:0/2211250744 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563a65717b00 msgr2=0x563a65741200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:29:40.066 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:40.061+0000 7f4279422740 1 --2- 172.21.15.46:0/2211250744 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563a65717b00 0x563a65741200 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 crypto rx=0x7f42640095f0 tx=0x7f4264012820 comp rx=0 tx=0).stop 2024-03-31T23:29:40.066 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:40.061+0000 7f4279422740 1 -- 172.21.15.46:0/2211250744 shutdown_connections 2024-03-31T23:29:40.066 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:40.061+0000 7f4279422740 1 --2- 172.21.15.46:0/2211250744 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x563a657295c0 0x563a657106f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:29:40.066 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:40.061+0000 7f4279422740 1 --2- 172.21.15.46:0/2211250744 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563a65717b00 0x563a65741200 unknown :-1 s=CLOSED pgs=949 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:29:40.066 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:40.061+0000 7f4279422740 1 --2- 172.21.15.46:0/2211250744 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563a657171e0 0x563a65734f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:29:40.066 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:40.061+0000 7f4279422740 1 -- 172.21.15.46:0/2211250744 >> 172.21.15.46:0/2211250744 conn(0x563a65711fe0 msgr2=0x563a657123c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:29:40.066 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:40.061+0000 7f4279422740 1 -- 172.21.15.46:0/2211250744 shutdown_connections 2024-03-31T23:29:40.066 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:40.061+0000 7f4279422740 1 -- 172.21.15.46:0/2211250744 wait complete. 2024-03-31T23:29:40.066 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:40.061+0000 7f4279422740 1 -- start start 2024-03-31T23:29:40.066 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:40.062+0000 7f4279422740 1 Processor -- start 2024-03-31T23:29:40.066 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:40.062+0000 7f4279422740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563a657171e0 0x563a657232b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:29:40.067 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:40.062+0000 7f4279422740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563a65717b00 0x563a657237f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:29:40.067 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:40.062+0000 7f4279422740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x563a657295c0 0x563a65723d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:29:40.067 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:40.062+0000 7f4279422740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f425c00e6e0 con 0x563a657171e0 2024-03-31T23:29:40.067 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:40.062+0000 7f4279422740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f425c00fb50 con 0x563a657295c0 2024-03-31T23:29:40.067 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:40.062+0000 7f4279422740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f425c00edb0 con 0x563a65717b00 2024-03-31T23:29:40.067 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:40.062+0000 7f4279422740 10 client.0 _ll_register_callbacks cb 0x7f426009b000 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T23:29:40.067 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:40.062+0000 7f426ffff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563a657171e0 0x563a657232b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:29:40.067 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:40.062+0000 7f426f7fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563a65717b00 0x563a657237f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:29:40.067 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:40.062+0000 7f426ffff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563a657171e0 0x563a657232b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:58504/0 (socket says 172.21.15.46:58504) 2024-03-31T23:29:40.067 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:40.062+0000 7f426ffff640 1 -- 172.21.15.46:0/3355375328 learned_addr learned my addr 172.21.15.46:0/3355375328 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:29:40.067 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:40.062+0000 7f4274d06640 1 --2- 172.21.15.46:0/3355375328 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x563a657295c0 0x563a65723d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:29:40.067 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:40.062+0000 7f426ffff640 1 -- 172.21.15.46:0/3355375328 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563a65717b00 msgr2=0x563a657237f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:29:40.067 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:40.062+0000 7f426ffff640 1 --2- 172.21.15.46:0/3355375328 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563a65717b00 0x563a657237f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:29:40.068 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:40.062+0000 7f426ffff640 1 -- 172.21.15.46:0/3355375328 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x563a657295c0 msgr2=0x563a65723d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:29:40.068 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:40.062+0000 7f426ffff640 1 --2- 172.21.15.46:0/3355375328 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x563a657295c0 0x563a65723d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:29:40.068 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:40.062+0000 7f426ffff640 1 -- 172.21.15.46:0/3355375328 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563a65716d70 con 0x563a657171e0 2024-03-31T23:29:40.068 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:40.063+0000 7f4274d06640 1 --2- 172.21.15.46:0/3355375328 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x563a657295c0 0x563a65723d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T23:29:40.068 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:40.063+0000 7f426ffff640 1 --2- 172.21.15.46:0/3355375328 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563a657171e0 0x563a657232b0 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 crypto rx=0x7f426009b400 tx=0x7f426009b0d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=56944938e856b4e2 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:29:40.068 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:40.063+0000 7f424f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-31T23:29:40.068 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:40.063+0000 7f424f7fe640 1 -- 172.21.15.46:0/3355375328 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4260010630 con 0x563a657171e0 2024-03-31T23:29:40.068 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:40.063+0000 7f424f7fe640 1 -- 172.21.15.46:0/3355375328 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(122..122 src has 1..122) ==== 9554+0+0 (secure 0 0 0) 0x7f42600ccae0 con 0x563a657171e0 2024-03-31T23:29:40.068 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:40.063+0000 7f4279422740 1 -- 172.21.15.46:0/3355375328 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4264022070 con 0x563a657171e0 2024-03-31T23:29:40.068 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:40.063+0000 7f424f7fe640 1 -- 172.21.15.46:0/3355375328 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) ==== 24+0+0 (secure 0 0 0) 0x7f42600a3dd0 con 0x563a657171e0 2024-03-31T23:29:40.068 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:40.063+0000 7f4279422740 10 client.17610 fetch_fsmap learned FSMap version 47 2024-03-31T23:29:40.068 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:40.063+0000 7f4279422740 1 -- 172.21.15.46:0/3355375328 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) -- 0x563a6579fb30 con 0x563a657171e0 2024-03-31T23:29:40.068 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:40.063+0000 7f424f7fe640 1 -- 172.21.15.46:0/3355375328 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4264022070 con 0x563a657171e0 2024-03-31T23:29:40.069 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:40.064+0000 7f424f7fe640 1 -- 172.21.15.46:0/3355375328 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 47) ==== 8346+0+0 (secure 0 0 0) 0x7f42600af120 con 0x563a657171e0 2024-03-31T23:29:40.069 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:40.064+0000 7f4279422740 10 client.17610 fetch_fsmap finished waiting for FSMap version 47 2024-03-31T23:29:40.069 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:40.064+0000 7f4279422740 10 client.17610 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-31T23:29:40.069 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:40.064+0000 7f4279422740 20 client.17610 populate_metadata read hostname 'smithi046' 2024-03-31T23:29:40.069 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:40.064+0000 7f4279422740 1 --2- 172.21.15.46:0/3355375328 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x563a657a0280 0x563a657a26a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:29:40.069 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:40.064+0000 7f4279422740 4 client.17610 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-31T23:29:40.069 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:40.064+0000 7f4279422740 1 -- 172.21.15.46:0/3355375328 --> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] -- command(tid 0: {"prefix": "flush journal"}) -- 0x563a65687e30 con 0x563a657a0280 2024-03-31T23:29:40.069 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:40.064+0000 7f426f7fe640 1 --2- 172.21.15.46:0/3355375328 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x563a657a0280 0x563a657a26a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:29:40.069 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:40.064+0000 7f426f7fe640 1 --2- 172.21.15.46:0/3355375328 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x563a657a0280 0x563a657a26a0 crc :-1 s=READY pgs=237 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=837bbbb3d2d36ecc server_cookie=cd45058427c98733 in_seq=0 out_seq=0 2024-03-31T23:29:40.069 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:40.064+0000 7f424f7fe640 10 client.17610 ms_handle_connect on v2:172.21.15.142:6838/2519143918 2024-03-31T23:29:40.258 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:40.257+0000 7f424f7fe640 1 -- 172.21.15.46:0/3355375328 <== mds.0 v2:172.21.15.142:6838/2519143918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x563a65687e30 con 0x563a657a0280 2024-03-31T23:29:40.259 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:40.257+0000 7f424f7fe640 10 client.17610 handle_command_reply: tid=0 2024-03-31T23:29:40.259 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:40.257+0000 7f4279422740 1 client.17610 shutdown 2024-03-31T23:29:40.259 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:40.258+0000 7f4279422740 1 -- 172.21.15.46:0/3355375328 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563a657171e0 msgr2=0x563a657232b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:29:40.260 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:40.258+0000 7f4279422740 1 --2- 172.21.15.46:0/3355375328 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563a657171e0 0x563a657232b0 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 crypto rx=0x7f426009b400 tx=0x7f426009b0d0 comp rx=0 tx=0).stop 2024-03-31T23:29:40.260 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:40.258+0000 7f4279422740 1 -- 172.21.15.46:0/3355375328 shutdown_connections 2024-03-31T23:29:40.260 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:40.258+0000 7f4279422740 1 --2- 172.21.15.46:0/3355375328 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x563a657a0280 0x563a657a26a0 crc :-1 s=READY pgs=237 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:29:40.260 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:40.258+0000 7f4279422740 1 --2- 172.21.15.46:0/3355375328 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x563a657295c0 0x563a65723d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:29:40.260 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:40.258+0000 7f4279422740 1 --2- 172.21.15.46:0/3355375328 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563a65717b00 0x563a657237f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:29:40.260 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:40.258+0000 7f4279422740 1 --2- 172.21.15.46:0/3355375328 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563a657171e0 0x563a657232b0 unknown :-1 s=CLOSED pgs=950 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:29:40.260 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:40.258+0000 7f424f7fe640 0 client.17610 ms_handle_reset on v2:172.21.15.142:6838/2519143918 2024-03-31T23:29:40.260 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:40.258+0000 7f4279422740 1 -- 172.21.15.46:0/3355375328 >> 172.21.15.46:0/3355375328 conn(0x563a65711fe0 msgr2=0x563a657106f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:29:40.260 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:40.259+0000 7f4279422740 1 -- 172.21.15.46:0/3355375328 shutdown_connections 2024-03-31T23:29:40.261 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:40.259+0000 7f4279422740 1 -- 172.21.15.46:0/3355375328 wait complete. 2024-03-31T23:29:40.261 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:40.260+0000 7f4279422740 20 client.17610 trim_cache size 0 max 16384 2024-03-31T23:29:40.261 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout:{ 2024-03-31T23:29:40.261 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout: "message": "", 2024-03-31T23:29:40.261 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout: "return_code": 0 2024-03-31T23:29:40.262 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout:} 2024-03-31T23:29:40.263 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:40.261+0000 7f4276769640 1 -- 172.21.15.46:0/3106041902 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f42580a2760 msgr2=0x7f42580a4c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:29:40.263 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:40.261+0000 7f4276769640 1 --2- 172.21.15.46:0/3106041902 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f42580a2760 0x7f42580a4c20 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f4268005e30 tx=0x7f4268032e60 comp rx=0 tx=0).stop 2024-03-31T23:29:40.263 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:40.261+0000 7f4276769640 1 -- 172.21.15.46:0/3106041902 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4270068dd0 msgr2=0x7f4270100970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:29:40.263 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:40.262+0000 7f4276769640 1 --2- 172.21.15.46:0/3106041902 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4270068dd0 0x7f4270100970 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 crypto rx=0x7f426000e9f0 tx=0x7f426000eeb0 comp rx=0 tx=0).stop 2024-03-31T23:29:40.263 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:40.262+0000 7f4276769640 1 -- 172.21.15.46:0/3106041902 shutdown_connections 2024-03-31T23:29:40.263 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:40.262+0000 7f4276769640 1 --2- 172.21.15.46:0/3106041902 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f42580a2760 0x7f42580a4c20 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:29:40.263 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:40.262+0000 7f4276769640 1 --2- 172.21.15.46:0/3106041902 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4270107820 0x7f42701013f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:29:40.263 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:40.262+0000 7f4276769640 1 --2- 172.21.15.46:0/3106041902 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4270069770 0x7f4270100eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:29:40.263 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:40.262+0000 7f4276769640 1 --2- 172.21.15.46:0/3106041902 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4270068dd0 0x7f4270100970 unknown :-1 s=CLOSED pgs=947 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:29:40.263 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:40.262+0000 7f4276769640 1 -- 172.21.15.46:0/3106041902 >> 172.21.15.46:0/3106041902 conn(0x7f42700774a0 msgr2=0x7f42700fe850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:29:40.264 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:40.262+0000 7f4276769640 1 -- 172.21.15.46:0/3106041902 shutdown_connections 2024-03-31T23:29:40.264 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:40.262+0000 7f4276769640 1 -- 172.21.15.46:0/3106041902 wait complete. 2024-03-31T23:29:40.289 INFO:tasks.workunit.client.0.smithi046.stdout: 10 45906 8.53 MB/sec execute 165 sec latency 186.415 ms 2024-03-31T23:29:41.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:29:40 smithi162 ceph-mon[36237]: pgmap v567: 129 pgs: 129 active+clean; 781 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 6.3 MiB/s wr, 388 op/s 2024-03-31T23:29:41.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:29:40 smithi046 ceph-mon[32498]: pgmap v567: 129 pgs: 129 active+clean; 781 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 6.3 MiB/s wr, 388 op/s 2024-03-31T23:29:41.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:29:40 smithi142 ceph-mon[35271]: pgmap v567: 129 pgs: 129 active+clean; 781 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 6.3 MiB/s wr, 388 op/s 2024-03-31T23:29:41.289 INFO:tasks.workunit.client.0.smithi046.stdout: 10 46075 8.52 MB/sec execute 166 sec latency 96.105 ms 2024-03-31T23:29:42.289 INFO:tasks.workunit.client.0.smithi046.stdout: 10 46251 8.55 MB/sec execute 167 sec latency 101.620 ms 2024-03-31T23:29:43.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:29:42 smithi162 ceph-mon[36237]: pgmap v568: 129 pgs: 129 active+clean; 781 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 3.8 MiB/s wr, 222 op/s 2024-03-31T23:29:43.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:29:42 smithi046 ceph-mon[32498]: pgmap v568: 129 pgs: 129 active+clean; 781 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 3.8 MiB/s wr, 222 op/s 2024-03-31T23:29:43.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:29:42 smithi142 ceph-mon[35271]: pgmap v568: 129 pgs: 129 active+clean; 781 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 3.8 MiB/s wr, 222 op/s 2024-03-31T23:29:43.289 INFO:tasks.workunit.client.0.smithi046.stdout: 10 46386 8.54 MB/sec execute 168 sec latency 71.952 ms 2024-03-31T23:29:44.289 INFO:tasks.workunit.client.0.smithi046.stdout: 10 46583 8.60 MB/sec execute 169 sec latency 64.334 ms 2024-03-31T23:29:45.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:29:44 smithi162 ceph-mon[36237]: pgmap v569: 129 pgs: 129 active+clean; 781 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 3.8 MiB/s wr, 222 op/s 2024-03-31T23:29:45.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:29:44 smithi046 ceph-mon[32498]: pgmap v569: 129 pgs: 129 active+clean; 781 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 3.8 MiB/s wr, 222 op/s 2024-03-31T23:29:45.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:29:44 smithi142 ceph-mon[35271]: pgmap v569: 129 pgs: 129 active+clean; 781 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 3.8 MiB/s wr, 222 op/s 2024-03-31T23:29:45.289 INFO:tasks.workunit.client.0.smithi046.stdout: 10 46785 8.63 MB/sec execute 170 sec latency 69.299 ms 2024-03-31T23:29:46.289 INFO:tasks.workunit.client.0.smithi046.stdout: 10 46951 8.63 MB/sec execute 171 sec latency 76.058 ms 2024-03-31T23:29:47.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:29:46 smithi162 ceph-mon[36237]: pgmap v570: 129 pgs: 129 active+clean; 790 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 6.2 MiB/s wr, 390 op/s 2024-03-31T23:29:47.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:29:46 smithi046 ceph-mon[32498]: pgmap v570: 129 pgs: 129 active+clean; 790 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 6.2 MiB/s wr, 390 op/s 2024-03-31T23:29:47.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:29:46 smithi142 ceph-mon[35271]: pgmap v570: 129 pgs: 129 active+clean; 790 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 6.2 MiB/s wr, 390 op/s 2024-03-31T23:29:47.289 INFO:tasks.workunit.client.0.smithi046.stdout: 10 47072 8.60 MB/sec execute 172 sec latency 70.174 ms 2024-03-31T23:29:48.289 INFO:tasks.workunit.client.0.smithi046.stdout: 10 47224 8.62 MB/sec execute 173 sec latency 68.824 ms 2024-03-31T23:29:49.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:29:48 smithi162 ceph-mon[36237]: pgmap v571: 129 pgs: 129 active+clean; 790 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 4.4 MiB/s wr, 280 op/s 2024-03-31T23:29:49.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:29:48 smithi046 ceph-mon[32498]: pgmap v571: 129 pgs: 129 active+clean; 790 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 4.4 MiB/s wr, 280 op/s 2024-03-31T23:29:49.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:29:48 smithi142 ceph-mon[35271]: pgmap v571: 129 pgs: 129 active+clean; 790 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 4.4 MiB/s wr, 280 op/s 2024-03-31T23:29:49.289 INFO:tasks.workunit.client.0.smithi046.stdout: 10 47377 8.61 MB/sec execute 174 sec latency 75.747 ms 2024-03-31T23:29:50.290 INFO:tasks.workunit.client.0.smithi046.stdout: 10 47503 8.59 MB/sec execute 175 sec latency 161.485 ms 2024-03-31T23:29:51.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:29:50 smithi162 ceph-mon[36237]: pgmap v572: 129 pgs: 129 active+clean; 802 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 6.7 MiB/s wr, 388 op/s 2024-03-31T23:29:51.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:29:50 smithi046 ceph-mon[32498]: pgmap v572: 129 pgs: 129 active+clean; 802 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 6.7 MiB/s wr, 388 op/s 2024-03-31T23:29:51.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:29:50 smithi142 ceph-mon[35271]: pgmap v572: 129 pgs: 129 active+clean; 802 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 6.7 MiB/s wr, 388 op/s 2024-03-31T23:29:51.290 INFO:tasks.workunit.client.0.smithi046.stdout: 10 47627 8.56 MB/sec execute 176 sec latency 86.654 ms 2024-03-31T23:29:52.290 INFO:tasks.workunit.client.0.smithi046.stdout: 10 47776 8.57 MB/sec execute 177 sec latency 115.065 ms 2024-03-31T23:29:53.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:29:52 smithi162 ceph-mon[36237]: pgmap v573: 129 pgs: 129 active+clean; 802 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 4.7 MiB/s wr, 275 op/s 2024-03-31T23:29:53.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:29:52 smithi046 ceph-mon[32498]: pgmap v573: 129 pgs: 129 active+clean; 802 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 4.7 MiB/s wr, 275 op/s 2024-03-31T23:29:53.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:29:52 smithi142 ceph-mon[35271]: pgmap v573: 129 pgs: 129 active+clean; 802 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 4.7 MiB/s wr, 275 op/s 2024-03-31T23:29:53.290 INFO:tasks.workunit.client.0.smithi046.stdout: 10 47927 8.55 MB/sec execute 178 sec latency 120.194 ms 2024-03-31T23:29:53.389 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:53.388+0000 7f949b43b640 1 -- 172.21.15.46:0/2919585290 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9494107750 msgr2=0x7f9494107b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:29:53.389 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:53.388+0000 7f949b43b640 1 --2- 172.21.15.46:0/2919585290 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9494107750 0x7f9494107b30 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 crypto rx=0x7f9490009930 tx=0x7f949002bc30 comp rx=0 tx=0).stop 2024-03-31T23:29:53.389 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:53.388+0000 7f949b43b640 1 -- 172.21.15.46:0/2919585290 shutdown_connections 2024-03-31T23:29:53.389 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:53.388+0000 7f949b43b640 1 --2- 172.21.15.46:0/2919585290 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9494102100 0x7f949410e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:29:53.389 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:53.388+0000 7f949b43b640 1 --2- 172.21.15.46:0/2919585290 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9494101740 0x7f9494101bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:29:53.389 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:53.388+0000 7f949b43b640 1 --2- 172.21.15.46:0/2919585290 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9494107750 0x7f9494107b30 unknown :-1 s=CLOSED pgs=951 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:29:53.389 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:53.388+0000 7f949b43b640 1 -- 172.21.15.46:0/2919585290 >> 172.21.15.46:0/2919585290 conn(0x7f94940fd210 msgr2=0x7f94940ff630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:29:53.389 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:53.388+0000 7f949b43b640 1 -- 172.21.15.46:0/2919585290 shutdown_connections 2024-03-31T23:29:53.389 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:53.389+0000 7f949b43b640 1 -- 172.21.15.46:0/2919585290 wait complete. 2024-03-31T23:29:53.391 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:53.389+0000 7f949b43b640 1 Processor -- start 2024-03-31T23:29:53.391 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:53.389+0000 7f949b43b640 1 -- start start 2024-03-31T23:29:53.392 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:53.389+0000 7f949b43b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9494101740 0x7f9494197f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:29:53.392 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:53.389+0000 7f949b43b640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9494102100 0x7f9494198460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:29:53.392 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:53.390+0000 7f949b43b640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9494107750 0x7f949419c790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:29:53.392 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:53.390+0000 7f949b43b640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f9494111a60 con 0x7f9494101740 2024-03-31T23:29:53.392 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:53.390+0000 7f949b43b640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f94941118e0 con 0x7f9494107750 2024-03-31T23:29:53.392 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:53.390+0000 7f949b43b640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f9494111be0 con 0x7f9494102100 2024-03-31T23:29:53.392 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:53.390+0000 7f94991d7640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9494101740 0x7f9494197f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:29:53.392 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:53.390+0000 7f94991d7640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9494101740 0x7f9494197f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:58322/0 (socket says 172.21.15.46:58322) 2024-03-31T23:29:53.392 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:53.390+0000 7f94991d7640 1 -- 172.21.15.46:0/3659115069 learned_addr learned my addr 172.21.15.46:0/3659115069 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:29:53.392 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:53.390+0000 7f94989d6640 1 --2- 172.21.15.46:0/3659115069 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9494102100 0x7f9494198460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:29:53.392 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:53.390+0000 7f94999d8640 1 --2- 172.21.15.46:0/3659115069 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9494107750 0x7f949419c790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:29:53.392 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:53.390+0000 7f94989d6640 1 -- 172.21.15.46:0/3659115069 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9494107750 msgr2=0x7f949419c790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:29:53.392 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:53.390+0000 7f94989d6640 1 --2- 172.21.15.46:0/3659115069 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9494107750 0x7f949419c790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:29:53.393 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:53.390+0000 7f94989d6640 1 -- 172.21.15.46:0/3659115069 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9494101740 msgr2=0x7f9494197f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:29:53.393 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:53.390+0000 7f94989d6640 1 --2- 172.21.15.46:0/3659115069 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9494101740 0x7f9494197f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:29:53.393 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:53.390+0000 7f94989d6640 1 -- 172.21.15.46:0/3659115069 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f949419cec0 con 0x7f9494102100 2024-03-31T23:29:53.393 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:53.390+0000 7f94991d7640 1 --2- 172.21.15.46:0/3659115069 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9494101740 0x7f9494197f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:29:53.393 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:53.390+0000 7f94989d6640 1 --2- 172.21.15.46:0/3659115069 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9494102100 0x7f9494198460 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f948400d6d0 tx=0x7f948400db90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:29:53.393 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:53.391+0000 7f948a7fc640 1 -- 172.21.15.46:0/3659115069 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9484017070 con 0x7f9494102100 2024-03-31T23:29:53.393 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:53.391+0000 7f948a7fc640 1 -- 172.21.15.46:0/3659115069 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9484009820 con 0x7f9494102100 2024-03-31T23:29:53.393 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:53.391+0000 7f949b43b640 1 -- 172.21.15.46:0/3659115069 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f949419d1b0 con 0x7f9494102100 2024-03-31T23:29:53.393 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:53.391+0000 7f948a7fc640 1 -- 172.21.15.46:0/3659115069 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9484004c40 con 0x7f9494102100 2024-03-31T23:29:53.393 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:53.391+0000 7f949b43b640 1 -- 172.21.15.46:0/3659115069 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f94941a4a40 con 0x7f9494102100 2024-03-31T23:29:53.395 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:53.393+0000 7f948a7fc640 1 -- 172.21.15.46:0/3659115069 <== mon.1 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 25) ==== 137451+0+0 (secure 0 0 0) 0x7f94840099d0 con 0x7f9494102100 2024-03-31T23:29:53.395 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:53.394+0000 7f948a7fc640 1 --2- 172.21.15.46:0/3659115069 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f94800a2730 0x7f94800a4bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:29:53.395 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:53.394+0000 7f949e016740 1 Processor -- start 2024-03-31T23:29:53.396 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:53.394+0000 7f949e016740 1 -- start start 2024-03-31T23:29:53.396 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:53.394+0000 7f94991d7640 1 --2- 172.21.15.46:0/3659115069 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f94800a2730 0x7f94800a4bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:29:53.396 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:53.394+0000 7f948a7fc640 1 -- 172.21.15.46:0/3659115069 <== mon.1 v2:172.21.15.162:3300/0 5 ==== osd_map(122..122 src has 1..122) ==== 9554+0+0 (secure 0 0 0) 0x7f94840ce910 con 0x7f9494102100 2024-03-31T23:29:53.396 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:53.394+0000 7f949e016740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x562691dfed40 0x562691dff120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:29:53.396 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:53.394+0000 7f949e016740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562691dff6f0 0x562691deba90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:29:53.396 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:53.394+0000 7f949e016740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562691dec0b0 0x562691dec510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:29:53.396 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:53.394+0000 7f949e016740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x562691d548f0 con 0x562691dec0b0 2024-03-31T23:29:53.396 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:53.394+0000 7f949e016740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x562691cfd8d0 con 0x562691dfed40 2024-03-31T23:29:53.396 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:53.394+0000 7f949e016740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x562691d50990 con 0x562691dff6f0 2024-03-31T23:29:53.396 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:53.394+0000 7f94989d6640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562691dff6f0 0x562691deba90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:29:53.396 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:53.394+0000 7f94999d8640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562691dec0b0 0x562691dec510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:29:53.396 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:53.394+0000 7f94989d6640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562691dff6f0 0x562691deba90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.46:60954/0 (socket says 172.21.15.46:60954) 2024-03-31T23:29:53.396 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:53.394+0000 7f94989d6640 1 -- 172.21.15.46:0/4197442340 learned_addr learned my addr 172.21.15.46:0/4197442340 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:29:53.396 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:53.394+0000 7f94991d7640 1 --2- 172.21.15.46:0/4197442340 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x562691dfed40 0x562691dff120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:29:53.397 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:53.395+0000 7f94989d6640 1 -- 172.21.15.46:0/4197442340 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x562691dfed40 msgr2=0x562691dff120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:29:53.397 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:53.395+0000 7f94989d6640 1 --2- 172.21.15.46:0/4197442340 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x562691dfed40 0x562691dff120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:29:53.397 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:53.395+0000 7f94989d6640 1 -- 172.21.15.46:0/4197442340 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562691dec0b0 msgr2=0x562691dec510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:29:53.397 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:53.395+0000 7f94989d6640 1 --2- 172.21.15.46:0/4197442340 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562691dec0b0 0x562691dec510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:29:53.397 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:53.395+0000 7f94989d6640 1 -- 172.21.15.46:0/4197442340 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562691a69210 con 0x562691dff6f0 2024-03-31T23:29:53.397 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:53.395+0000 7f94991d7640 1 --2- 172.21.15.46:0/3659115069 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f94800a2730 0x7f94800a4bf0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f9490006010 tx=0x7f9490005a20 comp rx=0 tx=0).ready entity=mgr.15654 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:29:53.397 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:53.395+0000 7f94989d6640 1 --2- 172.21.15.46:0/4197442340 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562691dff6f0 0x562691deba90 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f9484097700 tx=0x7f948409adf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=58f5bfa8f81e43b3 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:29:53.397 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:53.395+0000 7f947f7fe640 1 -- 172.21.15.46:0/4197442340 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9484020040 con 0x562691dff6f0 2024-03-31T23:29:53.397 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:53.395+0000 7f947f7fe640 1 -- 172.21.15.46:0/4197442340 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9484099030 con 0x562691dff6f0 2024-03-31T23:29:53.397 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:53.395+0000 7f947f7fe640 1 -- 172.21.15.46:0/4197442340 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9484096df0 con 0x562691dff6f0 2024-03-31T23:29:53.401 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:53.396+0000 7f949e016740 1 -- 172.21.15.46:0/4197442340 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562691dff6f0 msgr2=0x562691deba90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:29:53.402 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:53.396+0000 7f949e016740 1 --2- 172.21.15.46:0/4197442340 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562691dff6f0 0x562691deba90 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f9484097700 tx=0x7f948409adf0 comp rx=0 tx=0).stop 2024-03-31T23:29:53.402 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:53.396+0000 7f949e016740 1 -- 172.21.15.46:0/4197442340 shutdown_connections 2024-03-31T23:29:53.402 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:53.396+0000 7f949e016740 1 --2- 172.21.15.46:0/4197442340 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562691dec0b0 0x562691dec510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:29:53.402 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:53.396+0000 7f949e016740 1 --2- 172.21.15.46:0/4197442340 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562691dff6f0 0x562691deba90 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:29:53.402 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:53.396+0000 7f949e016740 1 --2- 172.21.15.46:0/4197442340 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x562691dfed40 0x562691dff120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:29:53.402 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:53.396+0000 7f949e016740 1 -- 172.21.15.46:0/4197442340 >> 172.21.15.46:0/4197442340 conn(0x562691dee550 msgr2=0x562691dee930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:29:53.402 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:53.396+0000 7f949e016740 1 -- 172.21.15.46:0/4197442340 shutdown_connections 2024-03-31T23:29:53.402 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:53.396+0000 7f949e016740 1 -- 172.21.15.46:0/4197442340 wait complete. 2024-03-31T23:29:53.402 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:53.396+0000 7f949e016740 1 -- start start 2024-03-31T23:29:53.402 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:53.397+0000 7f949e016740 1 Processor -- start 2024-03-31T23:29:53.402 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:53.397+0000 7f949e016740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562691dec0b0 0x562691e02da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:29:53.402 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:53.397+0000 7f949e016740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562691dfed40 0x562691e032e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:29:53.403 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:53.397+0000 7f94991d7640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562691dec0b0 0x562691e02da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:29:53.403 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:53.397+0000 7f94991d7640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562691dec0b0 0x562691e02da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:58348/0 (socket says 172.21.15.46:58348) 2024-03-31T23:29:53.403 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:53.397+0000 7f94991d7640 1 -- 172.21.15.46:0/3481262229 learned_addr learned my addr 172.21.15.46:0/3481262229 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:29:53.403 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:53.397+0000 7f949e016740 1 --2- 172.21.15.46:0/3481262229 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x562691dff6f0 0x562691e03820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:29:53.403 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:53.397+0000 7f949e016740 1 -- 172.21.15.46:0/3481262229 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x562691e06180 con 0x562691dec0b0 2024-03-31T23:29:53.403 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:53.397+0000 7f949e016740 1 -- 172.21.15.46:0/3481262229 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x562691cf2f80 con 0x562691dff6f0 2024-03-31T23:29:53.403 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:53.397+0000 7f949e016740 1 -- 172.21.15.46:0/3481262229 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x562691e06300 con 0x562691dfed40 2024-03-31T23:29:53.403 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:53.397+0000 7f94989d6640 1 --2- 172.21.15.46:0/3481262229 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562691dfed40 0x562691e032e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:29:53.403 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:53.397+0000 7f94999d8640 1 --2- 172.21.15.46:0/3481262229 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x562691dff6f0 0x562691e03820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:29:53.403 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:53.397+0000 7f949e016740 10 client.0 _ll_register_callbacks cb 0x562691d4b290 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T23:29:53.403 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:53.398+0000 7f94989d6640 1 -- 172.21.15.46:0/3481262229 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x562691dff6f0 msgr2=0x562691e03820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:29:53.403 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:53.398+0000 7f94989d6640 1 --2- 172.21.15.46:0/3481262229 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x562691dff6f0 0x562691e03820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:29:53.403 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:53.398+0000 7f94989d6640 1 -- 172.21.15.46:0/3481262229 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562691dec0b0 msgr2=0x562691e02da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:29:53.404 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:53.398+0000 7f94989d6640 1 --2- 172.21.15.46:0/3481262229 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562691dec0b0 0x562691e02da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:29:53.404 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:53.398+0000 7f94989d6640 1 -- 172.21.15.46:0/3481262229 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562691b35620 con 0x562691dfed40 2024-03-31T23:29:53.404 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:53.398+0000 7f94999d8640 1 --2- 172.21.15.46:0/3481262229 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x562691dff6f0 0x562691e03820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T23:29:53.404 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:53.398+0000 7f94989d6640 1 --2- 172.21.15.46:0/3481262229 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562691dfed40 0x562691e032e0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x562691dfa730 tx=0x7f9484004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8ab1e1139d9a63f2 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:29:53.404 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:53.398+0000 7f947cff9640 10 client.0 ms_handle_connect on v2:172.21.15.162:3300/0 2024-03-31T23:29:53.404 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:53.398+0000 7f947cff9640 1 -- 172.21.15.46:0/3481262229 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94840b7070 con 0x562691dfed40 2024-03-31T23:29:53.404 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:53.398+0000 7f949e016740 1 -- 172.21.15.46:0/3481262229 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9484099030 con 0x562691dfed40 2024-03-31T23:29:53.404 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:53.398+0000 7f947cff9640 1 -- 172.21.15.46:0/3481262229 <== mon.1 v2:172.21.15.162:3300/0 2 ==== osd_map(122..122 src has 1..122) ==== 9554+0+0 (secure 0 0 0) 0x7f94840b92c0 con 0x562691dfed40 2024-03-31T23:29:53.404 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:53.398+0000 7f947cff9640 1 -- 172.21.15.46:0/3481262229 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) ==== 24+0+0 (secure 0 0 0) 0x7f9484020050 con 0x562691dfed40 2024-03-31T23:29:53.404 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:53.398+0000 7f949e016740 10 client.27343 fetch_fsmap learned FSMap version 47 2024-03-31T23:29:53.404 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:53.398+0000 7f949e016740 1 -- 172.21.15.46:0/3481262229 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) -- 0x562691e06180 con 0x562691dfed40 2024-03-31T23:29:53.404 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:53.398+0000 7f947cff9640 1 -- 172.21.15.46:0/3481262229 <== mon.1 v2:172.21.15.162:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9484099030 con 0x562691dfed40 2024-03-31T23:29:53.404 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:53.399+0000 7f947cff9640 1 -- 172.21.15.46:0/3481262229 <== mon.1 v2:172.21.15.162:3300/0 5 ==== fsmap(e 47) ==== 8346+0+0 (secure 0 0 0) 0x7f948409a210 con 0x562691dfed40 2024-03-31T23:29:53.405 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:53.399+0000 7f949e016740 10 client.27343 fetch_fsmap finished waiting for FSMap version 47 2024-03-31T23:29:53.405 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:53.399+0000 7f949e016740 10 client.27343 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-31T23:29:53.405 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:53.399+0000 7f949e016740 20 client.27343 populate_metadata read hostname 'smithi046' 2024-03-31T23:29:53.405 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:53.399+0000 7f949e016740 1 --2- 172.21.15.46:0/3481262229 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x562691e13340 0x562691e15730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:29:53.405 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:53.399+0000 7f949e016740 4 client.27343 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-31T23:29:53.405 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:53.399+0000 7f949e016740 1 -- 172.21.15.46:0/3481262229 --> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562691d68120 con 0x562691e13340 2024-03-31T23:29:53.405 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:53.399+0000 7f94991d7640 1 --2- 172.21.15.46:0/3481262229 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x562691e13340 0x562691e15730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:29:53.405 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:53.399+0000 7f94991d7640 1 --2- 172.21.15.46:0/3481262229 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x562691e13340 0x562691e15730 crc :-1 s=READY pgs=238 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6050be25a8a5d261 server_cookie=cf430f213a23a7a0 in_seq=0 out_seq=0 2024-03-31T23:29:53.405 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:53.399+0000 7f947cff9640 10 client.27343 ms_handle_connect on v2:172.21.15.142:6838/2519143918 2024-03-31T23:29:53.405 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:53.401+0000 7f947cff9640 1 -- 172.21.15.46:0/3481262229 <== mds.0 v2:172.21.15.142:6838/2519143918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x562691d68120 con 0x562691e13340 2024-03-31T23:29:53.405 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:53.401+0000 7f947cff9640 10 client.27343 handle_command_reply: tid=0 2024-03-31T23:29:53.405 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:53.401+0000 7f949e016740 1 client.27343 shutdown 2024-03-31T23:29:53.405 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:53.401+0000 7f949e016740 1 -- 172.21.15.46:0/3481262229 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562691dfed40 msgr2=0x562691e032e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:29:53.406 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:53.401+0000 7f949e016740 1 --2- 172.21.15.46:0/3481262229 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562691dfed40 0x562691e032e0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x562691dfa730 tx=0x7f9484004730 comp rx=0 tx=0).stop 2024-03-31T23:29:53.406 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:53.401+0000 7f949e016740 1 -- 172.21.15.46:0/3481262229 shutdown_connections 2024-03-31T23:29:53.406 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:53.401+0000 7f949e016740 1 --2- 172.21.15.46:0/3481262229 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x562691e13340 0x562691e15730 crc :-1 s=READY pgs=238 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:29:53.406 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:53.401+0000 7f949e016740 1 --2- 172.21.15.46:0/3481262229 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x562691dff6f0 0x562691e03820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:29:53.406 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:53.401+0000 7f947cff9640 0 client.27343 ms_handle_reset on v2:172.21.15.142:6838/2519143918 2024-03-31T23:29:53.406 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:53.401+0000 7f949e016740 1 --2- 172.21.15.46:0/3481262229 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562691dfed40 0x562691e032e0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:29:53.406 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:53.401+0000 7f949e016740 1 --2- 172.21.15.46:0/3481262229 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562691dec0b0 0x562691e02da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:29:53.406 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:53.401+0000 7f949e016740 1 -- 172.21.15.46:0/3481262229 >> 172.21.15.46:0/3481262229 conn(0x562691dee550 msgr2=0x562691dedcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:29:53.406 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:53.401+0000 7f949e016740 1 -- 172.21.15.46:0/3481262229 shutdown_connections 2024-03-31T23:29:53.406 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:53.401+0000 7f949e016740 1 -- 172.21.15.46:0/3481262229 wait complete. 2024-03-31T23:29:53.406 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:53.401+0000 7f949e016740 20 client.27343 trim_cache size 0 max 16384 2024-03-31T23:29:53.427 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:53.425+0000 7f949e016740 1 Processor -- start 2024-03-31T23:29:53.428 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:53.425+0000 7f949e016740 1 -- start start 2024-03-31T23:29:53.428 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:53.426+0000 7f949e016740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562691dec0b0 0x562691e054d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:29:53.428 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:53.426+0000 7f949e016740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562691dfed40 0x562691e0efb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:29:53.428 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:53.426+0000 7f949e016740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x562691dff6f0 0x562691e01670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:29:53.428 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:53.426+0000 7f949e016740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f94941af390 con 0x562691dfed40 2024-03-31T23:29:53.428 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:53.426+0000 7f949e016740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f94941acc60 con 0x562691dff6f0 2024-03-31T23:29:53.428 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:53.426+0000 7f949e016740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f94941aaad0 con 0x562691dec0b0 2024-03-31T23:29:53.428 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:53.426+0000 7f94989d6640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562691dfed40 0x562691e0efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:29:53.428 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:53.426+0000 7f94989d6640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562691dfed40 0x562691e0efb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:58362/0 (socket says 172.21.15.46:58362) 2024-03-31T23:29:53.429 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:53.426+0000 7f94989d6640 1 -- 172.21.15.46:0/4127739706 learned_addr learned my addr 172.21.15.46:0/4127739706 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:29:53.429 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:53.426+0000 7f94991d7640 1 --2- 172.21.15.46:0/4127739706 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562691dec0b0 0x562691e054d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:29:53.429 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:53.426+0000 7f94999d8640 1 --2- 172.21.15.46:0/4127739706 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x562691dff6f0 0x562691e01670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:29:53.429 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:53.426+0000 7f94989d6640 1 -- 172.21.15.46:0/4127739706 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562691dec0b0 msgr2=0x562691e054d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:29:53.429 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:53.426+0000 7f94989d6640 1 --2- 172.21.15.46:0/4127739706 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562691dec0b0 0x562691e054d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:29:53.429 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:53.426+0000 7f94989d6640 1 -- 172.21.15.46:0/4127739706 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x562691dff6f0 msgr2=0x562691e01670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:29:53.429 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:53.426+0000 7f94989d6640 1 --2- 172.21.15.46:0/4127739706 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x562691dff6f0 0x562691e01670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:29:53.429 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:53.426+0000 7f94989d6640 1 -- 172.21.15.46:0/4127739706 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562691deb700 con 0x562691dfed40 2024-03-31T23:29:53.429 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:53.426+0000 7f94999d8640 1 --2- 172.21.15.46:0/4127739706 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x562691dff6f0 0x562691e01670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T23:29:53.429 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:53.426+0000 7f94989d6640 1 --2- 172.21.15.46:0/4127739706 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562691dfed40 0x562691e0efb0 secure :-1 s=READY pgs=952 cs=0 l=1 rev1=1 crypto rx=0x7f9484099c90 tx=0x7f948409adf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d8dc697d79cf8f8 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:29:53.429 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:53.426+0000 7f947effd640 1 -- 172.21.15.46:0/4127739706 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94840b7070 con 0x562691dfed40 2024-03-31T23:29:53.429 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:53.427+0000 7f947effd640 1 -- 172.21.15.46:0/4127739706 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9484099030 con 0x562691dfed40 2024-03-31T23:29:53.429 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:53.427+0000 7f947effd640 1 -- 172.21.15.46:0/4127739706 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f948409a180 con 0x562691dfed40 2024-03-31T23:29:53.430 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:53.427+0000 7f949e016740 1 -- 172.21.15.46:0/4127739706 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562691dfed40 msgr2=0x562691e0efb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:29:53.430 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:53.427+0000 7f949e016740 1 --2- 172.21.15.46:0/4127739706 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562691dfed40 0x562691e0efb0 secure :-1 s=READY pgs=952 cs=0 l=1 rev1=1 crypto rx=0x7f9484099c90 tx=0x7f948409adf0 comp rx=0 tx=0).stop 2024-03-31T23:29:53.430 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:53.427+0000 7f949e016740 1 -- 172.21.15.46:0/4127739706 shutdown_connections 2024-03-31T23:29:53.430 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:53.427+0000 7f949e016740 1 --2- 172.21.15.46:0/4127739706 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x562691dff6f0 0x562691e01670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:29:53.430 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:53.427+0000 7f949e016740 1 --2- 172.21.15.46:0/4127739706 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562691dfed40 0x562691e0efb0 unknown :-1 s=CLOSED pgs=952 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:29:53.430 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:53.427+0000 7f949e016740 1 --2- 172.21.15.46:0/4127739706 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562691dec0b0 0x562691e054d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:29:53.430 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:53.428+0000 7f949e016740 1 -- 172.21.15.46:0/4127739706 >> 172.21.15.46:0/4127739706 conn(0x562691dee550 msgr2=0x562691def520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:29:53.430 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:53.428+0000 7f949e016740 1 -- 172.21.15.46:0/4127739706 shutdown_connections 2024-03-31T23:29:53.430 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:53.428+0000 7f949e016740 1 -- 172.21.15.46:0/4127739706 wait complete. 2024-03-31T23:29:53.433 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:53.428+0000 7f949e016740 1 -- start start 2024-03-31T23:29:53.433 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:53.428+0000 7f949e016740 1 Processor -- start 2024-03-31T23:29:53.433 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:53.428+0000 7f949e016740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x562691dec0b0 0x562691dfb6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:29:53.433 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:53.428+0000 7f949e016740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562691dfed40 0x562691dfbbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:29:53.433 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:53.429+0000 7f949e016740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562691dff6f0 0x562691e1a380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:29:53.433 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:53.429+0000 7f949e016740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f94941ab120 con 0x562691dff6f0 2024-03-31T23:29:53.434 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:53.429+0000 7f949e016740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f94941ac590 con 0x562691dec0b0 2024-03-31T23:29:53.434 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:53.429+0000 7f949e016740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f94941ab7f0 con 0x562691dfed40 2024-03-31T23:29:53.434 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:53.429+0000 7f949e016740 10 client.0 _ll_register_callbacks cb 0x7f948c009b30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T23:29:53.434 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:53.429+0000 7f94991d7640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x562691dec0b0 0x562691dfb6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:29:53.434 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:53.429+0000 7f94991d7640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x562691dec0b0 0x562691dfb6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.46:48748/0 (socket says 172.21.15.46:48748) 2024-03-31T23:29:53.434 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:53.429+0000 7f94991d7640 1 -- 172.21.15.46:0/3005782558 learned_addr learned my addr 172.21.15.46:0/3005782558 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:29:53.434 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:53.429+0000 7f94989d6640 1 --2- 172.21.15.46:0/3005782558 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562691dfed40 0x562691dfbbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:29:53.434 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:53.429+0000 7f94999d8640 1 --2- 172.21.15.46:0/3005782558 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562691dff6f0 0x562691e1a380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:29:53.435 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:53.429+0000 7f94989d6640 1 -- 172.21.15.46:0/3005782558 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x562691dec0b0 msgr2=0x562691dfb6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:29:53.435 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:53.429+0000 7f94989d6640 1 --2- 172.21.15.46:0/3005782558 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x562691dec0b0 0x562691dfb6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:29:53.435 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:53.429+0000 7f94989d6640 1 -- 172.21.15.46:0/3005782558 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562691dff6f0 msgr2=0x562691e1a380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:29:53.435 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:53.429+0000 7f94989d6640 1 --2- 172.21.15.46:0/3005782558 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562691dff6f0 0x562691e1a380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:29:53.435 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:53.429+0000 7f94989d6640 1 -- 172.21.15.46:0/3005782558 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562691e1a8c0 con 0x562691dfed40 2024-03-31T23:29:53.435 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:53.429+0000 7f94991d7640 1 --2- 172.21.15.46:0/3005782558 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x562691dec0b0 0x562691dfb6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:29:53.435 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:53.429+0000 7f94999d8640 1 --2- 172.21.15.46:0/3005782558 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562691dff6f0 0x562691e1a380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T23:29:53.435 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:53.429+0000 7f94989d6640 1 --2- 172.21.15.46:0/3005782558 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562691dfed40 0x562691dfbbf0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f948409acc0 tx=0x7f94840a37c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=faff765d476de320 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:29:53.435 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:53.429+0000 7f947cff9640 10 client.0 ms_handle_connect on v2:172.21.15.162:3300/0 2024-03-31T23:29:53.435 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:53.429+0000 7f947cff9640 1 -- 172.21.15.46:0/3005782558 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94840ab060 con 0x562691dfed40 2024-03-31T23:29:53.435 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:53.430+0000 7f947cff9640 1 -- 172.21.15.46:0/3005782558 <== mon.1 v2:172.21.15.162:3300/0 2 ==== osd_map(122..122 src has 1..122) ==== 9554+0+0 (secure 0 0 0) 0x7f9484079d50 con 0x562691dfed40 2024-03-31T23:29:53.435 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:53.430+0000 7f949e016740 1 -- 172.21.15.46:0/3005782558 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9484099030 con 0x562691dfed40 2024-03-31T23:29:53.436 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:53.430+0000 7f947cff9640 1 -- 172.21.15.46:0/3005782558 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) ==== 24+0+0 (secure 0 0 0) 0x7f94840b7070 con 0x562691dfed40 2024-03-31T23:29:53.436 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:53.430+0000 7f949e016740 10 client.27349 fetch_fsmap learned FSMap version 47 2024-03-31T23:29:53.436 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:53.430+0000 7f949e016740 1 -- 172.21.15.46:0/3005782558 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) -- 0x562691e1a8c0 con 0x562691dfed40 2024-03-31T23:29:53.436 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:53.430+0000 7f947cff9640 1 -- 172.21.15.46:0/3005782558 <== mon.1 v2:172.21.15.162:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f948409a390 con 0x562691dfed40 2024-03-31T23:29:53.436 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:53.430+0000 7f947cff9640 1 -- 172.21.15.46:0/3005782558 <== mon.1 v2:172.21.15.162:3300/0 5 ==== fsmap(e 47) ==== 8346+0+0 (secure 0 0 0) 0x7f94840a0160 con 0x562691dfed40 2024-03-31T23:29:53.436 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:53.430+0000 7f949e016740 10 client.27349 fetch_fsmap finished waiting for FSMap version 47 2024-03-31T23:29:53.436 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:53.430+0000 7f949e016740 10 client.27349 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-31T23:29:53.436 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:53.430+0000 7f949e016740 20 client.27349 populate_metadata read hostname 'smithi046' 2024-03-31T23:29:53.436 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:53.430+0000 7f949e016740 1 --2- 172.21.15.46:0/3005782558 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x562691e13340 0x562691e7bcc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:29:53.437 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:53.431+0000 7f949e016740 4 client.27349 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-31T23:29:53.437 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:53.431+0000 7f949e016740 1 -- 172.21.15.46:0/3005782558 --> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] -- command(tid 0: {"prefix": "flush journal"}) -- 0x7f9484099030 con 0x562691e13340 2024-03-31T23:29:53.437 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:53.431+0000 7f94991d7640 1 --2- 172.21.15.46:0/3005782558 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x562691e13340 0x562691e7bcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:29:53.437 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:53.431+0000 7f947cff9640 10 client.27349 ms_handle_connect on v2:172.21.15.142:6838/2519143918 2024-03-31T23:29:53.437 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:53.431+0000 7f94991d7640 1 --2- 172.21.15.46:0/3005782558 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x562691e13340 0x562691e7bcc0 crc :-1 s=READY pgs=239 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3e7c592b3193cc78 server_cookie=5c93fc267ccffc30 in_seq=0 out_seq=0 2024-03-31T23:29:53.588 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:53.587+0000 7f947cff9640 1 -- 172.21.15.46:0/3005782558 <== mds.0 v2:172.21.15.142:6838/2519143918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x7f9484099030 con 0x562691e13340 2024-03-31T23:29:53.588 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:53.587+0000 7f947cff9640 10 client.27349 handle_command_reply: tid=0 2024-03-31T23:29:53.588 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:53.587+0000 7f949e016740 1 client.27349 shutdown 2024-03-31T23:29:53.588 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:53.587+0000 7f949e016740 1 -- 172.21.15.46:0/3005782558 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562691dfed40 msgr2=0x562691dfbbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:29:53.588 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:53.587+0000 7f949e016740 1 --2- 172.21.15.46:0/3005782558 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562691dfed40 0x562691dfbbf0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f948409acc0 tx=0x7f94840a37c0 comp rx=0 tx=0).stop 2024-03-31T23:29:53.588 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:53.587+0000 7f949e016740 1 -- 172.21.15.46:0/3005782558 shutdown_connections 2024-03-31T23:29:53.588 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:53.587+0000 7f949e016740 1 --2- 172.21.15.46:0/3005782558 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x562691e13340 0x562691e7bcc0 crc :-1 s=READY pgs=239 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:29:53.588 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:53.587+0000 7f949e016740 1 --2- 172.21.15.46:0/3005782558 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562691dff6f0 0x562691e1a380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:29:53.588 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:53.587+0000 7f949e016740 1 --2- 172.21.15.46:0/3005782558 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562691dfed40 0x562691dfbbf0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:29:53.588 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:53.587+0000 7f949e016740 1 --2- 172.21.15.46:0/3005782558 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x562691dec0b0 0x562691dfb6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:29:53.589 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:53.587+0000 7f949e016740 1 -- 172.21.15.46:0/3005782558 >> 172.21.15.46:0/3005782558 conn(0x562691dee550 msgr2=0x562691e01630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:29:53.589 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:53.587+0000 7f947cff9640 0 client.27349 ms_handle_reset on v2:172.21.15.142:6838/2519143918 2024-03-31T23:29:53.589 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:53.587+0000 7f949e016740 1 -- 172.21.15.46:0/3005782558 shutdown_connections 2024-03-31T23:29:53.589 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:53.587+0000 7f949e016740 1 -- 172.21.15.46:0/3005782558 wait complete. 2024-03-31T23:29:53.589 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:53.587+0000 7f949e016740 20 client.27349 trim_cache size 0 max 16384 2024-03-31T23:29:53.589 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout:{ 2024-03-31T23:29:53.589 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout: "message": "", 2024-03-31T23:29:53.589 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout: "return_code": 0 2024-03-31T23:29:53.589 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout:} 2024-03-31T23:29:53.590 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:53.588+0000 7f949b43b640 1 -- 172.21.15.46:0/3659115069 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f94800a2730 msgr2=0x7f94800a4bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:29:53.590 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:53.588+0000 7f949b43b640 1 --2- 172.21.15.46:0/3659115069 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f94800a2730 0x7f94800a4bf0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f9490006010 tx=0x7f9490005a20 comp rx=0 tx=0).stop 2024-03-31T23:29:53.590 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:53.588+0000 7f949b43b640 1 -- 172.21.15.46:0/3659115069 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9494102100 msgr2=0x7f9494198460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:29:53.591 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:53.588+0000 7f949b43b640 1 --2- 172.21.15.46:0/3659115069 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9494102100 0x7f9494198460 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f948400d6d0 tx=0x7f948400db90 comp rx=0 tx=0).stop 2024-03-31T23:29:53.591 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:53.589+0000 7f949b43b640 1 -- 172.21.15.46:0/3659115069 shutdown_connections 2024-03-31T23:29:53.591 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:53.589+0000 7f949b43b640 1 --2- 172.21.15.46:0/3659115069 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f94800a2730 0x7f94800a4bf0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:29:53.591 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:53.589+0000 7f949b43b640 1 --2- 172.21.15.46:0/3659115069 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9494107750 0x7f949419c790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:29:53.591 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:53.589+0000 7f949b43b640 1 --2- 172.21.15.46:0/3659115069 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9494102100 0x7f9494198460 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:29:53.591 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:53.589+0000 7f949b43b640 1 --2- 172.21.15.46:0/3659115069 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9494101740 0x7f9494197f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:29:53.591 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:53.589+0000 7f949b43b640 1 -- 172.21.15.46:0/3659115069 >> 172.21.15.46:0/3659115069 conn(0x7f94940fd210 msgr2=0x7f94940fd8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:29:53.591 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:53.589+0000 7f949b43b640 1 -- 172.21.15.46:0/3659115069 shutdown_connections 2024-03-31T23:29:53.591 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:29:53.589+0000 7f949b43b640 1 -- 172.21.15.46:0/3659115069 wait complete. 2024-03-31T23:29:54.290 INFO:tasks.workunit.client.0.smithi046.stdout: 10 48068 8.54 MB/sec execute 179 sec latency 151.580 ms 2024-03-31T23:29:55.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:29:54 smithi162 ceph-mon[36237]: pgmap v574: 129 pgs: 129 active+clean; 802 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 4.7 MiB/s wr, 275 op/s 2024-03-31T23:29:55.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:29:54 smithi046 ceph-mon[32498]: pgmap v574: 129 pgs: 129 active+clean; 802 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 4.7 MiB/s wr, 275 op/s 2024-03-31T23:29:55.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:29:54 smithi142 ceph-mon[35271]: pgmap v574: 129 pgs: 129 active+clean; 802 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 4.7 MiB/s wr, 275 op/s 2024-03-31T23:29:55.290 INFO:tasks.workunit.client.0.smithi046.stdout: 10 48211 8.52 MB/sec execute 180 sec latency 186.286 ms 2024-03-31T23:29:56.290 INFO:tasks.workunit.client.0.smithi046.stdout: 10 48383 8.54 MB/sec execute 181 sec latency 151.160 ms 2024-03-31T23:29:57.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:29:56 smithi162 ceph-mon[36237]: pgmap v575: 129 pgs: 129 active+clean; 802 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 6.8 MiB/s wr, 426 op/s 2024-03-31T23:29:57.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:29:56 smithi046 ceph-mon[32498]: pgmap v575: 129 pgs: 129 active+clean; 802 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 6.8 MiB/s wr, 426 op/s 2024-03-31T23:29:57.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:29:56 smithi142 ceph-mon[35271]: pgmap v575: 129 pgs: 129 active+clean; 802 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 6.8 MiB/s wr, 426 op/s 2024-03-31T23:29:57.290 INFO:tasks.workunit.client.0.smithi046.stdout: 10 48564 8.55 MB/sec execute 182 sec latency 58.369 ms 2024-03-31T23:29:58.290 INFO:tasks.workunit.client.0.smithi046.stdout: 10 48725 8.56 MB/sec execute 183 sec latency 105.106 ms 2024-03-31T23:29:59.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:29:58 smithi162 ceph-mon[36237]: pgmap v576: 129 pgs: 129 active+clean; 802 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 4.4 MiB/s wr, 258 op/s 2024-03-31T23:29:59.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:29:58 smithi046 ceph-mon[32498]: pgmap v576: 129 pgs: 129 active+clean; 802 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 4.4 MiB/s wr, 258 op/s 2024-03-31T23:29:59.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:29:58 smithi142 ceph-mon[35271]: pgmap v576: 129 pgs: 129 active+clean; 802 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 4.4 MiB/s wr, 258 op/s 2024-03-31T23:29:59.290 INFO:tasks.workunit.client.0.smithi046.stdout: 10 48873 8.56 MB/sec execute 184 sec latency 111.796 ms 2024-03-31T23:30:00.290 INFO:tasks.workunit.client.0.smithi046.stdout: 10 49026 8.54 MB/sec execute 185 sec latency 68.220 ms 2024-03-31T23:30:00.361 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:30:00 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mon-a[32494]: 2024-03-31T23:29:59.999+0000 7fdb0b79b700 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR 1 MDSs report damaged metadata 2024-03-31T23:30:01.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:30:00 smithi162 ceph-mon[36237]: pgmap v577: 129 pgs: 129 active+clean; 805 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 6.5 MiB/s wr, 378 op/s 2024-03-31T23:30:01.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:30:00 smithi162 ceph-mon[36237]: overall HEALTH_ERR 1 MDSs report damaged metadata 2024-03-31T23:30:01.114 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:30:00 smithi046 ceph-mon[32498]: pgmap v577: 129 pgs: 129 active+clean; 805 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 6.5 MiB/s wr, 378 op/s 2024-03-31T23:30:01.114 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:30:00 smithi046 ceph-mon[32498]: overall HEALTH_ERR 1 MDSs report damaged metadata 2024-03-31T23:30:01.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:30:00 smithi142 ceph-mon[35271]: pgmap v577: 129 pgs: 129 active+clean; 805 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 6.5 MiB/s wr, 378 op/s 2024-03-31T23:30:01.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:30:00 smithi142 ceph-mon[35271]: overall HEALTH_ERR 1 MDSs report damaged metadata 2024-03-31T23:30:01.290 INFO:tasks.workunit.client.0.smithi046.stdout: 10 49186 8.53 MB/sec execute 186 sec latency 106.137 ms 2024-03-31T23:30:02.290 INFO:tasks.workunit.client.0.smithi046.stdout: 10 49385 8.54 MB/sec execute 187 sec latency 87.616 ms 2024-03-31T23:30:03.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:30:02 smithi162 ceph-mon[36237]: pgmap v578: 129 pgs: 129 active+clean; 805 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 4.2 MiB/s wr, 271 op/s 2024-03-31T23:30:03.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:30:02 smithi046 ceph-mon[32498]: pgmap v578: 129 pgs: 129 active+clean; 805 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 4.2 MiB/s wr, 271 op/s 2024-03-31T23:30:03.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:30:02 smithi142 ceph-mon[35271]: pgmap v578: 129 pgs: 129 active+clean; 805 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 4.2 MiB/s wr, 271 op/s 2024-03-31T23:30:03.290 INFO:tasks.workunit.client.0.smithi046.stdout: 10 49528 8.52 MB/sec execute 188 sec latency 107.618 ms 2024-03-31T23:30:04.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:30:03 smithi046 ceph-mon[32498]: osdmap e123: 12 total, 12 up, 12 in 2024-03-31T23:30:04.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:30:03 smithi142 ceph-mon[35271]: osdmap e123: 12 total, 12 up, 12 in 2024-03-31T23:30:04.290 INFO:tasks.workunit.client.0.smithi046.stdout: 10 49646 8.50 MB/sec execute 189 sec latency 75.547 ms 2024-03-31T23:30:04.295 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:30:03 smithi162 ceph-mon[36237]: osdmap e123: 12 total, 12 up, 12 in 2024-03-31T23:30:05.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:30:04 smithi046 ceph-mon[32498]: pgmap v580: 129 pgs: 129 active+clean; 805 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 5.1 MiB/s wr, 325 op/s 2024-03-31T23:30:05.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:30:04 smithi142 ceph-mon[35271]: pgmap v580: 129 pgs: 129 active+clean; 805 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 5.1 MiB/s wr, 325 op/s 2024-03-31T23:30:05.290 INFO:tasks.workunit.client.0.smithi046.stdout: 10 49809 8.54 MB/sec execute 190 sec latency 94.031 ms 2024-03-31T23:30:05.295 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:30:04 smithi162 ceph-mon[36237]: pgmap v580: 129 pgs: 129 active+clean; 805 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 5.1 MiB/s wr, 325 op/s 2024-03-31T23:30:06.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:30:05 smithi046 ceph-mon[32498]: pgmap v581: 129 pgs: 129 active+clean; 802 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 5.1 MiB/s wr, 292 op/s 2024-03-31T23:30:06.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:30:05 smithi142 ceph-mon[35271]: pgmap v581: 129 pgs: 129 active+clean; 802 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 5.1 MiB/s wr, 292 op/s 2024-03-31T23:30:06.290 INFO:tasks.workunit.client.0.smithi046.stdout: 10 50011 8.57 MB/sec execute 191 sec latency 74.326 ms 2024-03-31T23:30:06.295 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:30:05 smithi162 ceph-mon[36237]: pgmap v581: 129 pgs: 129 active+clean; 802 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 5.1 MiB/s wr, 292 op/s 2024-03-31T23:30:06.729 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:06.727+0000 7f2004db5640 1 -- 172.21.15.46:0/735199202 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2000069310 msgr2=0x7f2000069750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:30:06.729 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:06.727+0000 7f2004db5640 1 --2- 172.21.15.46:0/735199202 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2000069310 0x7f2000069750 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 crypto rx=0x7f1ff4009900 tx=0x7f1ff402bcd0 comp rx=0 tx=0).stop 2024-03-31T23:30:06.730 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:06.727+0000 7f2004db5640 1 -- 172.21.15.46:0/735199202 shutdown_connections 2024-03-31T23:30:06.730 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:06.727+0000 7f2004db5640 1 --2- 172.21.15.46:0/735199202 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2000104160 0x7f20001045c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:30:06.730 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:06.727+0000 7f2004db5640 1 --2- 172.21.15.46:0/735199202 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2000069310 0x7f2000069750 unknown :-1 s=CLOSED pgs=953 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:30:06.730 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:06.727+0000 7f2004db5640 1 --2- 172.21.15.46:0/735199202 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2000106710 0x7f2000068dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:30:06.730 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:06.727+0000 7f2004db5640 1 -- 172.21.15.46:0/735199202 >> 172.21.15.46:0/735199202 conn(0x7f20000774a0 msgr2=0x7f20000778b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:30:06.730 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:06.728+0000 7f2004db5640 1 -- 172.21.15.46:0/735199202 shutdown_connections 2024-03-31T23:30:06.730 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:06.728+0000 7f2004db5640 1 -- 172.21.15.46:0/735199202 wait complete. 2024-03-31T23:30:06.730 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:06.728+0000 7f2004db5640 1 Processor -- start 2024-03-31T23:30:06.730 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:06.728+0000 7f2004db5640 1 -- start start 2024-03-31T23:30:06.732 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:06.729+0000 7f2004db5640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2000069310 0x7f200019e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:30:06.732 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:06.729+0000 7f2004db5640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2000104160 0x7f200019e920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:30:06.732 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:06.729+0000 7f2004db5640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2000106710 0x7f20001a2ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:30:06.732 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:06.729+0000 7f2004db5640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f2000117f60 con 0x7f2000069310 2024-03-31T23:30:06.732 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:06.729+0000 7f2004db5640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f2000117de0 con 0x7f2000104160 2024-03-31T23:30:06.732 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:06.729+0000 7f2004db5640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f20001180e0 con 0x7f2000106710 2024-03-31T23:30:06.732 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:06.729+0000 7f1ffe59c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2000069310 0x7f200019e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:30:06.732 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:06.729+0000 7f1ffe59c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2000069310 0x7f200019e3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:48152/0 (socket says 172.21.15.46:48152) 2024-03-31T23:30:06.732 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:06.729+0000 7f1ffe59c640 1 -- 172.21.15.46:0/2371452066 learned_addr learned my addr 172.21.15.46:0/2371452066 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:30:06.732 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:06.729+0000 7f1ffed9d640 1 --2- 172.21.15.46:0/2371452066 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2000106710 0x7f20001a2ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:30:06.732 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:06.729+0000 7f1ffdd9b640 1 --2- 172.21.15.46:0/2371452066 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2000104160 0x7f200019e920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:30:06.732 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:06.729+0000 7f1ffe59c640 1 -- 172.21.15.46:0/2371452066 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2000106710 msgr2=0x7f20001a2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:30:06.733 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:06.729+0000 7f1ffe59c640 1 --2- 172.21.15.46:0/2371452066 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2000106710 0x7f20001a2ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:30:06.733 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:06.729+0000 7f1ffe59c640 1 -- 172.21.15.46:0/2371452066 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2000104160 msgr2=0x7f200019e920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:30:06.733 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:06.729+0000 7f1ffe59c640 1 --2- 172.21.15.46:0/2371452066 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2000104160 0x7f200019e920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:30:06.733 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:06.729+0000 7f1ffe59c640 1 -- 172.21.15.46:0/2371452066 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f20001a33d0 con 0x7f2000069310 2024-03-31T23:30:06.733 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:06.730+0000 7f1ffe59c640 1 --2- 172.21.15.46:0/2371452066 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2000069310 0x7f200019e3e0 secure :-1 s=READY pgs=954 cs=0 l=1 rev1=1 crypto rx=0x7f1ff000e9f0 tx=0x7f1ff000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:30:06.733 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:06.730+0000 7f1fef7fe640 1 -- 172.21.15.46:0/2371452066 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ff000cd80 con 0x7f2000069310 2024-03-31T23:30:06.733 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:06.730+0000 7f1fef7fe640 1 -- 172.21.15.46:0/2371452066 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1ff00054d0 con 0x7f2000069310 2024-03-31T23:30:06.733 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:06.730+0000 7f1fef7fe640 1 -- 172.21.15.46:0/2371452066 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ff0010490 con 0x7f2000069310 2024-03-31T23:30:06.733 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:06.730+0000 7f2004db5640 1 -- 172.21.15.46:0/2371452066 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f20001a36c0 con 0x7f2000069310 2024-03-31T23:30:06.733 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:06.731+0000 7f2004db5640 1 -- 172.21.15.46:0/2371452066 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f20001aaf50 con 0x7f2000069310 2024-03-31T23:30:06.737 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:06.736+0000 7f2007998740 1 Processor -- start 2024-03-31T23:30:06.738 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:06.737+0000 7f1fef7fe640 1 -- 172.21.15.46:0/2371452066 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 25) ==== 137451+0+0 (secure 0 0 0) 0x7f1ff0005000 con 0x7f2000069310 2024-03-31T23:30:06.739 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:06.737+0000 7f2007998740 1 -- start start 2024-03-31T23:30:06.739 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:06.738+0000 7f2007998740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557f797316c0 0x557f79731aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:30:06.740 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:06.738+0000 7f2007998740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557f79731fe0 0x557f7973a870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:30:06.740 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:06.738+0000 7f2007998740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557f7973adb0 0x557f7973d1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:30:06.740 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:06.738+0000 7f2007998740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x557f796948f0 con 0x557f79731fe0 2024-03-31T23:30:06.740 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:06.738+0000 7f2007998740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x557f7963d8d0 con 0x557f7973adb0 2024-03-31T23:30:06.740 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:06.738+0000 7f2007998740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x557f79690990 con 0x557f797316c0 2024-03-31T23:30:06.740 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:06.738+0000 7f1ffdd9b640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557f7973adb0 0x557f7973d1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:30:06.740 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:06.738+0000 7f1ffdd9b640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557f7973adb0 0x557f7973d1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.46:60546/0 (socket says 172.21.15.46:60546) 2024-03-31T23:30:06.740 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:06.738+0000 7f1ffdd9b640 1 -- 172.21.15.46:0/3655873217 learned_addr learned my addr 172.21.15.46:0/3655873217 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:30:06.741 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:06.738+0000 7f1ffed9d640 1 --2- 172.21.15.46:0/3655873217 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557f797316c0 0x557f79731aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:30:06.741 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:06.739+0000 7f1fef7fe640 1 --2- 172.21.15.46:0/2371452066 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f1fe80a2730 0x7f1fe80a4bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:30:06.742 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:06.740+0000 7f1ffe59c640 1 --2- 172.21.15.46:0/3655873217 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557f79731fe0 0x557f7973a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:30:06.742 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:06.740+0000 7f1ffed9d640 1 --2- 172.21.15.46:0/2371452066 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f1fe80a2730 0x7f1fe80a4bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:30:06.742 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:06.740+0000 7f1ffdd9b640 1 -- 172.21.15.46:0/3655873217 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557f797316c0 msgr2=0x557f79731aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:30:06.742 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:06.740+0000 7f1ffdd9b640 1 --2- 172.21.15.46:0/3655873217 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557f797316c0 0x557f79731aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:30:06.743 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:06.740+0000 7f1ffdd9b640 1 -- 172.21.15.46:0/3655873217 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557f79731fe0 msgr2=0x557f7973a870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:30:06.743 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:06.740+0000 7f1ffdd9b640 1 --2- 172.21.15.46:0/3655873217 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557f79731fe0 0x557f7973a870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:30:06.743 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:06.740+0000 7f1ffdd9b640 1 -- 172.21.15.46:0/3655873217 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557f793a9210 con 0x557f7973adb0 2024-03-31T23:30:06.743 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:06.741+0000 7f1fef7fe640 1 -- 172.21.15.46:0/2371452066 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(123..123 src has 1..123) ==== 9642+0+0 (secure 0 0 0) 0x7f1ff00cf2c0 con 0x7f2000069310 2024-03-31T23:30:06.744 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:06.741+0000 7f1ffdd9b640 1 --2- 172.21.15.46:0/3655873217 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557f7973adb0 0x557f7973d1a0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f1ff4009230 tx=0x7f1ff402c770 comp rx=0 tx=0).ready entity=mon.2 client_cookie=229b22ade3977572 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:30:06.744 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:06.741+0000 7f1ffe59c640 1 --2- 172.21.15.46:0/3655873217 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557f79731fe0 0x557f7973a870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T23:30:06.744 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:06.741+0000 7f1fecff9640 1 -- 172.21.15.46:0/3655873217 <== mon.2 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ff4004180 con 0x557f7973adb0 2024-03-31T23:30:06.744 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:06.742+0000 7f1fecff9640 1 -- 172.21.15.46:0/3655873217 <== mon.2 v2:172.21.15.142:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1ff402cd30 con 0x557f7973adb0 2024-03-31T23:30:06.744 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:06.742+0000 7f1fecff9640 1 -- 172.21.15.46:0/3655873217 <== mon.2 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ff4005640 con 0x557f7973adb0 2024-03-31T23:30:06.744 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:06.743+0000 7f2007998740 1 -- 172.21.15.46:0/3655873217 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557f7973adb0 msgr2=0x557f7973d1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:30:06.744 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:06.743+0000 7f2007998740 1 --2- 172.21.15.46:0/3655873217 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557f7973adb0 0x557f7973d1a0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f1ff4009230 tx=0x7f1ff402c770 comp rx=0 tx=0).stop 2024-03-31T23:30:06.745 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:06.744+0000 7f1ffed9d640 1 --2- 172.21.15.46:0/2371452066 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f1fe80a2730 0x7f1fe80a4bf0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f1ff8005fd0 tx=0x7f1ff8009ba0 comp rx=0 tx=0).ready entity=mgr.15654 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:30:06.745 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:06.744+0000 7f2007998740 1 -- 172.21.15.46:0/3655873217 shutdown_connections 2024-03-31T23:30:06.745 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:06.744+0000 7f2007998740 1 --2- 172.21.15.46:0/3655873217 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557f7973adb0 0x557f7973d1a0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:30:06.745 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:06.744+0000 7f2007998740 1 --2- 172.21.15.46:0/3655873217 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557f79731fe0 0x557f7973a870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:30:06.745 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:06.744+0000 7f2007998740 1 --2- 172.21.15.46:0/3655873217 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557f797316c0 0x557f79731aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:30:06.745 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:06.744+0000 7f2007998740 1 -- 172.21.15.46:0/3655873217 >> 172.21.15.46:0/3655873217 conn(0x557f7972bfe0 msgr2=0x557f7972c3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:30:06.746 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:06.745+0000 7f2007998740 1 -- 172.21.15.46:0/3655873217 shutdown_connections 2024-03-31T23:30:06.746 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:06.745+0000 7f2007998740 1 -- 172.21.15.46:0/3655873217 wait complete. 2024-03-31T23:30:06.746 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:06.745+0000 7f2007998740 1 -- start start 2024-03-31T23:30:06.746 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:06.746+0000 7f2007998740 1 Processor -- start 2024-03-31T23:30:06.747 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:06.746+0000 7f2007998740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557f797316c0 0x557f79737e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:30:06.748 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:06.746+0000 7f2007998740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557f79731fe0 0x557f79738340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:30:06.748 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:06.746+0000 7f2007998740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557f7973adb0 0x557f79738880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:30:06.748 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:06.746+0000 7f2007998740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x557f7974e990 con 0x557f7973adb0 2024-03-31T23:30:06.748 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:06.746+0000 7f2007998740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x557f79632f80 con 0x557f797316c0 2024-03-31T23:30:06.748 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:06.746+0000 7f2007998740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x557f7974eb10 con 0x557f79731fe0 2024-03-31T23:30:06.748 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:06.746+0000 7f2007998740 10 client.0 _ll_register_callbacks cb 0x557f7968b290 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T23:30:06.748 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:06.747+0000 7f1ffed9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557f7973adb0 0x557f79738880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:30:06.748 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:06.747+0000 7f1ffed9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557f7973adb0 0x557f79738880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:48164/0 (socket says 172.21.15.46:48164) 2024-03-31T23:30:06.748 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:06.747+0000 7f1ffed9d640 1 -- 172.21.15.46:0/2175521927 learned_addr learned my addr 172.21.15.46:0/2175521927 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:30:06.748 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:06.747+0000 7f1ffdd9b640 1 --2- 172.21.15.46:0/2175521927 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557f79731fe0 0x557f79738340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:30:06.749 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:06.747+0000 7f1ffed9d640 1 -- 172.21.15.46:0/2175521927 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557f79731fe0 msgr2=0x557f79738340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:30:06.749 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:06.747+0000 7f1ffed9d640 1 --2- 172.21.15.46:0/2175521927 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557f79731fe0 0x557f79738340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:30:06.749 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:06.747+0000 7f1ffed9d640 1 -- 172.21.15.46:0/2175521927 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557f797316c0 msgr2=0x557f79737e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:30:06.749 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:06.747+0000 7f1ffed9d640 1 --2- 172.21.15.46:0/2175521927 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557f797316c0 0x557f79737e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:30:06.750 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:06.747+0000 7f1ffed9d640 1 -- 172.21.15.46:0/2175521927 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557f79475620 con 0x557f7973adb0 2024-03-31T23:30:06.750 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:06.748+0000 7f1ffed9d640 1 --2- 172.21.15.46:0/2175521927 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557f7973adb0 0x557f79738880 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 crypto rx=0x7f1ff8013a70 tx=0x7f1ff801d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8614cb0d65ecadd5 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:30:06.750 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:06.748+0000 7f1fe5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-31T23:30:06.750 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:06.748+0000 7f1fe5ffb640 1 -- 172.21.15.46:0/2175521927 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ff8004b20 con 0x557f7973adb0 2024-03-31T23:30:06.750 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:06.748+0000 7f2007998740 1 -- 172.21.15.46:0/2175521927 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1ff402cd30 con 0x557f7973adb0 2024-03-31T23:30:06.750 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:06.749+0000 7f1fe5ffb640 1 -- 172.21.15.46:0/2175521927 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(123..123 src has 1..123) ==== 9642+0+0 (secure 0 0 0) 0x7f1ff8004cc0 con 0x557f7973adb0 2024-03-31T23:30:06.750 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:06.749+0000 7f1fe5ffb640 1 -- 172.21.15.46:0/2175521927 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) ==== 24+0+0 (secure 0 0 0) 0x7f1ff8032050 con 0x557f7973adb0 2024-03-31T23:30:06.753 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:06.753+0000 7f2007998740 10 client.17637 fetch_fsmap learned FSMap version 47 2024-03-31T23:30:06.754 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:06.753+0000 7f2007998740 1 -- 172.21.15.46:0/2175521927 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) -- 0x557f7975a7f0 con 0x557f7973adb0 2024-03-31T23:30:06.755 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:06.754+0000 7f1fe5ffb640 1 -- 172.21.15.46:0/2175521927 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1ff402cd30 con 0x557f7973adb0 2024-03-31T23:30:06.756 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:06.755+0000 7f1fe5ffb640 1 -- 172.21.15.46:0/2175521927 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 47) ==== 8346+0+0 (secure 0 0 0) 0x7f1ff8028030 con 0x557f7973adb0 2024-03-31T23:30:06.756 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:06.755+0000 7f2007998740 10 client.17637 fetch_fsmap finished waiting for FSMap version 47 2024-03-31T23:30:06.756 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:06.755+0000 7f2007998740 10 client.17637 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-31T23:30:06.757 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:06.755+0000 7f2007998740 20 client.17637 populate_metadata read hostname 'smithi046' 2024-03-31T23:30:06.757 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:06.755+0000 7f2007998740 1 --2- 172.21.15.46:0/2175521927 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x557f7975c0e0 0x557f7975e570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:30:06.757 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:06.756+0000 7f2007998740 4 client.17637 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-31T23:30:06.757 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:06.756+0000 7f2007998740 1 -- 172.21.15.46:0/2175521927 --> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557f796a8120 con 0x557f7975c0e0 2024-03-31T23:30:06.760 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:06.759+0000 7f1ffe59c640 1 --2- 172.21.15.46:0/2175521927 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x557f7975c0e0 0x557f7975e570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:30:06.762 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:06.761+0000 7f1ffe59c640 1 --2- 172.21.15.46:0/2175521927 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x557f7975c0e0 0x557f7975e570 crc :-1 s=READY pgs=240 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ea48ef500b8edf94 server_cookie=1dc34a08895661c9 in_seq=0 out_seq=0 2024-03-31T23:30:06.762 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:06.761+0000 7f1fe5ffb640 10 client.17637 ms_handle_connect on v2:172.21.15.142:6838/2519143918 2024-03-31T23:30:06.764 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:06.763+0000 7f1fe5ffb640 1 -- 172.21.15.46:0/2175521927 <== mds.0 v2:172.21.15.142:6838/2519143918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x557f796a8120 con 0x557f7975c0e0 2024-03-31T23:30:06.764 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:06.763+0000 7f1fe5ffb640 10 client.17637 handle_command_reply: tid=0 2024-03-31T23:30:06.764 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:06.763+0000 7f2007998740 1 client.17637 shutdown 2024-03-31T23:30:06.765 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:06.764+0000 7f2007998740 1 -- 172.21.15.46:0/2175521927 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557f7973adb0 msgr2=0x557f79738880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:30:06.765 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:06.764+0000 7f2007998740 1 --2- 172.21.15.46:0/2175521927 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557f7973adb0 0x557f79738880 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 crypto rx=0x7f1ff8013a70 tx=0x7f1ff801d000 comp rx=0 tx=0).stop 2024-03-31T23:30:06.765 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:06.764+0000 7f2007998740 1 -- 172.21.15.46:0/2175521927 shutdown_connections 2024-03-31T23:30:06.765 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:06.764+0000 7f2007998740 1 --2- 172.21.15.46:0/2175521927 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x557f7975c0e0 0x557f7975e570 crc :-1 s=READY pgs=240 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:30:06.765 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:06.764+0000 7f2007998740 1 --2- 172.21.15.46:0/2175521927 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557f7973adb0 0x557f79738880 unknown :-1 s=CLOSED pgs=955 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:30:06.766 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:06.764+0000 7f2007998740 1 --2- 172.21.15.46:0/2175521927 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557f79731fe0 0x557f79738340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:30:06.766 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:06.764+0000 7f2007998740 1 --2- 172.21.15.46:0/2175521927 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557f797316c0 0x557f79737e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:30:06.766 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:06.764+0000 7f1fe5ffb640 0 client.17637 ms_handle_reset on v2:172.21.15.142:6838/2519143918 2024-03-31T23:30:06.766 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:06.764+0000 7f2007998740 1 -- 172.21.15.46:0/2175521927 >> 172.21.15.46:0/2175521927 conn(0x557f7972bfe0 msgr2=0x557f7972dd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:30:06.766 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:06.764+0000 7f2007998740 1 -- 172.21.15.46:0/2175521927 shutdown_connections 2024-03-31T23:30:06.766 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:06.764+0000 7f2007998740 1 -- 172.21.15.46:0/2175521927 wait complete. 2024-03-31T23:30:06.766 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:06.764+0000 7f2007998740 20 client.17637 trim_cache size 0 max 16384 2024-03-31T23:30:06.787 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:06.785+0000 7f2007998740 1 Processor -- start 2024-03-31T23:30:06.787 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:06.785+0000 7f2007998740 1 -- start start 2024-03-31T23:30:06.787 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:06.785+0000 7f2007998740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557f797316c0 0x557f7973a530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:30:06.787 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:06.785+0000 7f2007998740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557f79731fe0 0x557f79757730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:30:06.787 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:06.785+0000 7f2007998740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557f7973adb0 0x557f79763e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:30:06.787 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:06.785+0000 7f2007998740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f20001af280 con 0x557f79731fe0 2024-03-31T23:30:06.788 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:06.785+0000 7f2007998740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f20001acb50 con 0x557f797316c0 2024-03-31T23:30:06.788 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:06.785+0000 7f2007998740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f200019f730 con 0x557f7973adb0 2024-03-31T23:30:06.788 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:06.785+0000 7f1ffdd9b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557f79731fe0 0x557f79757730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:30:06.788 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:06.786+0000 7f1ffdd9b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557f79731fe0 0x557f79757730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:48166/0 (socket says 172.21.15.46:48166) 2024-03-31T23:30:06.788 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:06.786+0000 7f1ffdd9b640 1 -- 172.21.15.46:0/79760421 learned_addr learned my addr 172.21.15.46:0/79760421 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:30:06.788 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:06.786+0000 7f1ffed9d640 1 --2- 172.21.15.46:0/79760421 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557f7973adb0 0x557f79763e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:30:06.788 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:06.786+0000 7f1ffe59c640 1 --2- 172.21.15.46:0/79760421 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557f797316c0 0x557f7973a530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:30:06.788 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:06.786+0000 7f1ffdd9b640 1 -- 172.21.15.46:0/79760421 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557f7973adb0 msgr2=0x557f79763e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:30:06.788 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:06.786+0000 7f1ffdd9b640 1 --2- 172.21.15.46:0/79760421 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557f7973adb0 0x557f79763e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:30:06.788 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:06.786+0000 7f1ffdd9b640 1 -- 172.21.15.46:0/79760421 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557f797316c0 msgr2=0x557f7973a530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:30:06.788 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:06.786+0000 7f1ffdd9b640 1 --2- 172.21.15.46:0/79760421 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557f797316c0 0x557f7973a530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:30:06.788 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:06.786+0000 7f1ffdd9b640 1 -- 172.21.15.46:0/79760421 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557f79731dd0 con 0x557f79731fe0 2024-03-31T23:30:06.788 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:06.786+0000 7f1ffed9d640 1 --2- 172.21.15.46:0/79760421 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557f7973adb0 0x557f79763e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:30:06.788 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:06.786+0000 7f1ffe59c640 1 --2- 172.21.15.46:0/79760421 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557f797316c0 0x557f7973a530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:30:06.789 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:06.786+0000 7f1ffdd9b640 1 --2- 172.21.15.46:0/79760421 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557f79731fe0 0x557f79757730 secure :-1 s=READY pgs=956 cs=0 l=1 rev1=1 crypto rx=0x7f1ff4009230 tx=0x7f1ff4005fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=12667b5542bcbbbe server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:30:06.789 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:06.786+0000 7f1fe7fff640 1 -- 172.21.15.46:0/79760421 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ff402cc20 con 0x557f79731fe0 2024-03-31T23:30:06.789 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:06.786+0000 7f1fe7fff640 1 -- 172.21.15.46:0/79760421 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1ff404b070 con 0x557f79731fe0 2024-03-31T23:30:06.789 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:06.786+0000 7f1fe7fff640 1 -- 172.21.15.46:0/79760421 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ff4004b80 con 0x557f79731fe0 2024-03-31T23:30:06.792 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:06.787+0000 7f2007998740 1 -- 172.21.15.46:0/79760421 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557f79731fe0 msgr2=0x557f79757730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:30:06.792 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:06.787+0000 7f2007998740 1 --2- 172.21.15.46:0/79760421 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557f79731fe0 0x557f79757730 secure :-1 s=READY pgs=956 cs=0 l=1 rev1=1 crypto rx=0x7f1ff4009230 tx=0x7f1ff4005fd0 comp rx=0 tx=0).stop 2024-03-31T23:30:06.792 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:06.787+0000 7f2007998740 1 -- 172.21.15.46:0/79760421 shutdown_connections 2024-03-31T23:30:06.792 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:06.787+0000 7f2007998740 1 --2- 172.21.15.46:0/79760421 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557f7973adb0 0x557f79763e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:30:06.792 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:06.787+0000 7f2007998740 1 --2- 172.21.15.46:0/79760421 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557f79731fe0 0x557f79757730 unknown :-1 s=CLOSED pgs=956 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:30:06.792 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:06.787+0000 7f2007998740 1 --2- 172.21.15.46:0/79760421 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557f797316c0 0x557f7973a530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:30:06.792 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:06.787+0000 7f2007998740 1 -- 172.21.15.46:0/79760421 >> 172.21.15.46:0/79760421 conn(0x557f7972bfe0 msgr2=0x557f7972c3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:30:06.792 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:06.787+0000 7f2007998740 1 -- 172.21.15.46:0/79760421 shutdown_connections 2024-03-31T23:30:06.792 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:06.787+0000 7f2007998740 1 -- 172.21.15.46:0/79760421 wait complete. 2024-03-31T23:30:06.793 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:06.788+0000 7f2007998740 1 -- start start 2024-03-31T23:30:06.793 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:06.788+0000 7f2007998740 1 Processor -- start 2024-03-31T23:30:06.793 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:06.788+0000 7f2007998740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557f797316c0 0x557f79735500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:30:06.793 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:06.788+0000 7f2007998740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557f79731fe0 0x557f79735a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:30:06.793 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:06.788+0000 7f2007998740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557f7973adb0 0x557f79734070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:30:06.793 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:06.788+0000 7f2007998740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f20001ab120 con 0x557f7973adb0 2024-03-31T23:30:06.793 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:06.788+0000 7f2007998740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f20001ac480 con 0x557f79731fe0 2024-03-31T23:30:06.793 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:06.788+0000 7f2007998740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f20001ab6e0 con 0x557f797316c0 2024-03-31T23:30:06.793 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:06.788+0000 7f2007998740 10 client.0 _ll_register_callbacks cb 0x7f1ff8013ca0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T23:30:06.793 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:06.788+0000 7f1ffe59c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557f797316c0 0x557f79735500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:30:06.793 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:06.788+0000 7f1ffed9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557f7973adb0 0x557f79734070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:30:06.793 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:06.788+0000 7f1ffdd9b640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557f79731fe0 0x557f79735a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:30:06.793 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:06.788+0000 7f1ffe59c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557f797316c0 0x557f79735500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.46:60028/0 (socket says 172.21.15.46:60028) 2024-03-31T23:30:06.794 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:06.788+0000 7f1ffe59c640 1 -- 172.21.15.46:0/2630726486 learned_addr learned my addr 172.21.15.46:0/2630726486 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:30:06.794 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:06.789+0000 7f1ffed9d640 1 -- 172.21.15.46:0/2630726486 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557f797316c0 msgr2=0x557f79735500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:30:06.794 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:06.789+0000 7f1ffed9d640 1 --2- 172.21.15.46:0/2630726486 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557f797316c0 0x557f79735500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:30:06.794 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:06.789+0000 7f1ffed9d640 1 -- 172.21.15.46:0/2630726486 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557f79731fe0 msgr2=0x557f79735a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:30:06.794 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:06.789+0000 7f1ffed9d640 1 --2- 172.21.15.46:0/2630726486 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557f79731fe0 0x557f79735a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:30:06.794 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:06.789+0000 7f1ffed9d640 1 -- 172.21.15.46:0/2630726486 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557f797345b0 con 0x557f7973adb0 2024-03-31T23:30:06.794 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:06.789+0000 7f1ffe59c640 1 --2- 172.21.15.46:0/2630726486 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557f797316c0 0x557f79735500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:30:06.794 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:06.789+0000 7f1ffdd9b640 1 --2- 172.21.15.46:0/2630726486 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557f79731fe0 0x557f79735a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:30:06.794 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:06.789+0000 7f1ffed9d640 1 --2- 172.21.15.46:0/2630726486 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557f7973adb0 0x557f79734070 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 crypto rx=0x7f1ff8011fd0 tx=0x7f1ff8004d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c440feecfd40514e server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:30:06.794 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:06.789+0000 7f1fe5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-31T23:30:06.794 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:06.789+0000 7f1fe5ffb640 1 -- 172.21.15.46:0/2630726486 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ff8004e40 con 0x557f7973adb0 2024-03-31T23:30:06.794 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:06.789+0000 7f1fe5ffb640 1 -- 172.21.15.46:0/2630726486 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(123..123 src has 1..123) ==== 9642+0+0 (secure 0 0 0) 0x7f1ff8020cf0 con 0x557f7973adb0 2024-03-31T23:30:06.794 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:06.789+0000 7f2007998740 1 -- 172.21.15.46:0/2630726486 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1ff404b070 con 0x557f7973adb0 2024-03-31T23:30:06.795 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:06.789+0000 7f1fe5ffb640 1 -- 172.21.15.46:0/2630726486 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) ==== 24+0+0 (secure 0 0 0) 0x7f1ff8040050 con 0x557f7973adb0 2024-03-31T23:30:06.795 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:06.789+0000 7f2007998740 10 client.17643 fetch_fsmap learned FSMap version 47 2024-03-31T23:30:06.795 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:06.789+0000 7f2007998740 1 -- 172.21.15.46:0/2630726486 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) -- 0x557f797c1c00 con 0x557f7973adb0 2024-03-31T23:30:06.795 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:06.790+0000 7f1fe5ffb640 1 -- 172.21.15.46:0/2630726486 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1ff404b070 con 0x557f7973adb0 2024-03-31T23:30:06.795 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:06.790+0000 7f1fe5ffb640 1 -- 172.21.15.46:0/2630726486 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 47) ==== 8346+0+0 (secure 0 0 0) 0x7f1ff8035020 con 0x557f7973adb0 2024-03-31T23:30:06.795 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:06.790+0000 7f2007998740 10 client.17643 fetch_fsmap finished waiting for FSMap version 47 2024-03-31T23:30:06.795 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:06.790+0000 7f2007998740 10 client.17643 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-31T23:30:06.795 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:06.790+0000 7f2007998740 20 client.17643 populate_metadata read hostname 'smithi046' 2024-03-31T23:30:06.795 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:06.790+0000 7f2007998740 1 --2- 172.21.15.46:0/2630726486 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x557f797c2260 0x557f797c4680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:30:06.795 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:06.790+0000 7f2007998740 4 client.17643 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-31T23:30:06.795 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:06.790+0000 7f2007998740 1 -- 172.21.15.46:0/2630726486 --> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] -- command(tid 0: {"prefix": "flush journal"}) -- 0x557f796a1e30 con 0x557f797c2260 2024-03-31T23:30:06.795 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:06.790+0000 7f1ffe59c640 1 --2- 172.21.15.46:0/2630726486 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x557f797c2260 0x557f797c4680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:30:06.795 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:06.791+0000 7f1ffe59c640 1 --2- 172.21.15.46:0/2630726486 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x557f797c2260 0x557f797c4680 crc :-1 s=READY pgs=241 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=321b2637e0686ae2 server_cookie=38fd0f91f821848 in_seq=0 out_seq=0 2024-03-31T23:30:06.796 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:06.791+0000 7f1fe5ffb640 10 client.17643 ms_handle_connect on v2:172.21.15.142:6838/2519143918 2024-03-31T23:30:06.985 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:06.984+0000 7f1fe5ffb640 1 -- 172.21.15.46:0/2630726486 <== mds.0 v2:172.21.15.142:6838/2519143918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x557f796a1e30 con 0x557f797c2260 2024-03-31T23:30:06.985 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:06.984+0000 7f1fe5ffb640 10 client.17643 handle_command_reply: tid=0 2024-03-31T23:30:06.985 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:06.984+0000 7f2007998740 1 client.17643 shutdown 2024-03-31T23:30:06.986 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:06.984+0000 7f2007998740 1 -- 172.21.15.46:0/2630726486 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557f7973adb0 msgr2=0x557f79734070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:30:06.986 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:06.984+0000 7f2007998740 1 --2- 172.21.15.46:0/2630726486 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557f7973adb0 0x557f79734070 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 crypto rx=0x7f1ff8011fd0 tx=0x7f1ff8004d50 comp rx=0 tx=0).stop 2024-03-31T23:30:06.986 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:06.984+0000 7f2007998740 1 -- 172.21.15.46:0/2630726486 shutdown_connections 2024-03-31T23:30:06.986 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:06.984+0000 7f2007998740 1 --2- 172.21.15.46:0/2630726486 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x557f797c2260 0x557f797c4680 crc :-1 s=READY pgs=241 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:30:06.986 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:06.984+0000 7f2007998740 1 --2- 172.21.15.46:0/2630726486 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557f7973adb0 0x557f79734070 unknown :-1 s=CLOSED pgs=957 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:30:06.986 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:06.984+0000 7f2007998740 1 --2- 172.21.15.46:0/2630726486 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557f79731fe0 0x557f79735a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:30:06.986 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:06.984+0000 7f2007998740 1 --2- 172.21.15.46:0/2630726486 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557f797316c0 0x557f79735500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:30:06.986 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:06.984+0000 7f1fe5ffb640 0 client.17643 ms_handle_reset on v2:172.21.15.142:6838/2519143918 2024-03-31T23:30:06.986 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:06.984+0000 7f2007998740 1 -- 172.21.15.46:0/2630726486 >> 172.21.15.46:0/2630726486 conn(0x557f7972bfe0 msgr2=0x557f79736630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:30:06.986 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:06.984+0000 7f2007998740 1 -- 172.21.15.46:0/2630726486 shutdown_connections 2024-03-31T23:30:06.986 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:06.984+0000 7f2007998740 1 -- 172.21.15.46:0/2630726486 wait complete. 2024-03-31T23:30:06.987 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:06.984+0000 7f2007998740 20 client.17643 trim_cache size 0 max 16384 2024-03-31T23:30:06.987 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout:{ 2024-03-31T23:30:06.987 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout: "message": "", 2024-03-31T23:30:06.987 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout: "return_code": 0 2024-03-31T23:30:06.987 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout:} 2024-03-31T23:30:06.988 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:06.984+0000 7f2004db5640 1 -- 172.21.15.46:0/2371452066 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f1fe80a2730 msgr2=0x7f1fe80a4bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:30:06.988 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:06.985+0000 7f2004db5640 1 --2- 172.21.15.46:0/2371452066 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f1fe80a2730 0x7f1fe80a4bf0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f1ff8005fd0 tx=0x7f1ff8009ba0 comp rx=0 tx=0).stop 2024-03-31T23:30:06.988 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:06.985+0000 7f2004db5640 1 -- 172.21.15.46:0/2371452066 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2000069310 msgr2=0x7f200019e3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:30:06.988 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:06.985+0000 7f2004db5640 1 --2- 172.21.15.46:0/2371452066 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2000069310 0x7f200019e3e0 secure :-1 s=READY pgs=954 cs=0 l=1 rev1=1 crypto rx=0x7f1ff000e9f0 tx=0x7f1ff000edf0 comp rx=0 tx=0).stop 2024-03-31T23:30:06.989 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:06.985+0000 7f2004db5640 1 -- 172.21.15.46:0/2371452066 shutdown_connections 2024-03-31T23:30:06.989 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:06.985+0000 7f2004db5640 1 --2- 172.21.15.46:0/2371452066 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f1fe80a2730 0x7f1fe80a4bf0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:30:06.989 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:06.985+0000 7f2004db5640 1 --2- 172.21.15.46:0/2371452066 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2000106710 0x7f20001a2ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:30:06.989 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:06.985+0000 7f2004db5640 1 --2- 172.21.15.46:0/2371452066 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2000104160 0x7f200019e920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:30:06.989 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:06.985+0000 7f2004db5640 1 --2- 172.21.15.46:0/2371452066 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2000069310 0x7f200019e3e0 unknown :-1 s=CLOSED pgs=954 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:30:06.989 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:06.985+0000 7f2004db5640 1 -- 172.21.15.46:0/2371452066 >> 172.21.15.46:0/2371452066 conn(0x7f20000774a0 msgr2=0x7f20000ff890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:30:06.989 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:06.985+0000 7f2004db5640 1 -- 172.21.15.46:0/2371452066 shutdown_connections 2024-03-31T23:30:06.989 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:06.985+0000 7f2004db5640 1 -- 172.21.15.46:0/2371452066 wait complete. 2024-03-31T23:30:07.290 INFO:tasks.workunit.client.0.smithi046.stdout: 10 50206 8.58 MB/sec execute 192 sec latency 137.026 ms 2024-03-31T23:30:08.290 INFO:tasks.workunit.client.0.smithi046.stdout: 10 50349 8.59 MB/sec execute 193 sec latency 80.713 ms 2024-03-31T23:30:08.295 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:30:08 smithi162 ceph-mon[36237]: pgmap v582: 129 pgs: 129 active+clean; 802 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 5.1 MiB/s wr, 292 op/s 2024-03-31T23:30:08.361 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:30:08 smithi046 ceph-mon[32498]: pgmap v582: 129 pgs: 129 active+clean; 802 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 5.1 MiB/s wr, 292 op/s 2024-03-31T23:30:08.443 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:30:08 smithi142 ceph-mon[35271]: pgmap v582: 129 pgs: 129 active+clean; 802 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 5.1 MiB/s wr, 292 op/s 2024-03-31T23:30:09.290 INFO:tasks.workunit.client.0.smithi046.stdout: 10 50457 8.57 MB/sec execute 194 sec latency 80.098 ms 2024-03-31T23:30:10.290 INFO:tasks.workunit.client.0.smithi046.stdout: 10 50641 8.60 MB/sec execute 195 sec latency 83.402 ms 2024-03-31T23:30:10.295 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:30:10 smithi162 ceph-mon[36237]: pgmap v583: 129 pgs: 129 active+clean; 825 MiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 5.8 MiB/s wr, 347 op/s 2024-03-31T23:30:10.361 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:30:10 smithi046 ceph-mon[32498]: pgmap v583: 129 pgs: 129 active+clean; 825 MiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 5.8 MiB/s wr, 347 op/s 2024-03-31T23:30:10.443 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:30:10 smithi142 ceph-mon[35271]: pgmap v583: 129 pgs: 129 active+clean; 825 MiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 5.8 MiB/s wr, 347 op/s 2024-03-31T23:30:11.291 INFO:tasks.workunit.client.0.smithi046.stdout: 10 50802 8.60 MB/sec execute 196 sec latency 117.230 ms 2024-03-31T23:30:11.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:30:11 smithi142 ceph-mon[35271]: osdmap e124: 12 total, 12 up, 12 in 2024-03-31T23:30:12.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:30:11 smithi162 ceph-mon[36237]: osdmap e124: 12 total, 12 up, 12 in 2024-03-31T23:30:12.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:30:11 smithi046 ceph-mon[32498]: osdmap e124: 12 total, 12 up, 12 in 2024-03-31T23:30:12.290 INFO:tasks.workunit.client.0.smithi046.stdout: 10 50942 8.59 MB/sec execute 197 sec latency 96.651 ms 2024-03-31T23:30:12.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:30:12 smithi142 ceph-mon[35271]: pgmap v585: 129 pgs: 129 active+clean; 825 MiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 7.2 MiB/s wr, 424 op/s 2024-03-31T23:30:13.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:30:12 smithi162 ceph-mon[36237]: pgmap v585: 129 pgs: 129 active+clean; 825 MiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 7.2 MiB/s wr, 424 op/s 2024-03-31T23:30:13.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:30:12 smithi046 ceph-mon[32498]: pgmap v585: 129 pgs: 129 active+clean; 825 MiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 7.2 MiB/s wr, 424 op/s 2024-03-31T23:30:13.290 INFO:tasks.workunit.client.0.smithi046.stdout: 10 51079 8.56 MB/sec execute 198 sec latency 93.330 ms 2024-03-31T23:30:13.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:30:13 smithi142 ceph-mon[35271]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-31T23:30:14.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:30:13 smithi162 ceph-mon[36237]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-31T23:30:14.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:30:13 smithi046 ceph-mon[32498]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-31T23:30:14.290 INFO:tasks.workunit.client.0.smithi046.stdout: 10 51200 8.54 MB/sec execute 199 sec latency 81.041 ms 2024-03-31T23:30:14.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:30:14 smithi142 ceph-mon[35271]: pgmap v586: 129 pgs: 129 active+clean; 825 MiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 5.8 MiB/s wr, 347 op/s 2024-03-31T23:30:14.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:30:14 smithi142 ceph-mon[35271]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:30:14.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:30:14 smithi142 ceph-mon[35271]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-31T23:30:14.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:30:14 smithi142 ceph-mon[35271]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' 2024-03-31T23:30:15.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:30:14 smithi162 ceph-mon[36237]: pgmap v586: 129 pgs: 129 active+clean; 825 MiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 5.8 MiB/s wr, 347 op/s 2024-03-31T23:30:15.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:30:14 smithi162 ceph-mon[36237]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:30:15.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:30:14 smithi162 ceph-mon[36237]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-31T23:30:15.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:30:14 smithi162 ceph-mon[36237]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' 2024-03-31T23:30:15.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:30:14 smithi046 ceph-mon[32498]: pgmap v586: 129 pgs: 129 active+clean; 825 MiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 5.8 MiB/s wr, 347 op/s 2024-03-31T23:30:15.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:30:14 smithi046 ceph-mon[32498]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:30:15.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:30:14 smithi046 ceph-mon[32498]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-31T23:30:15.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:30:14 smithi046 ceph-mon[32498]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' 2024-03-31T23:30:15.290 INFO:tasks.workunit.client.0.smithi046.stdout: 10 51338 8.54 MB/sec execute 200 sec latency 221.649 ms 2024-03-31T23:30:16.291 INFO:tasks.workunit.client.0.smithi046.stdout: 10 51499 8.52 MB/sec execute 201 sec latency 105.222 ms 2024-03-31T23:30:16.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:30:16 smithi142 ceph-mon[35271]: pgmap v587: 129 pgs: 129 active+clean; 835 MiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 6.1 MiB/s wr, 340 op/s 2024-03-31T23:30:17.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:30:16 smithi162 ceph-mon[36237]: pgmap v587: 129 pgs: 129 active+clean; 835 MiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 6.1 MiB/s wr, 340 op/s 2024-03-31T23:30:17.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:30:16 smithi046 ceph-mon[32498]: pgmap v587: 129 pgs: 129 active+clean; 835 MiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 6.1 MiB/s wr, 340 op/s 2024-03-31T23:30:17.290 INFO:tasks.workunit.client.0.smithi046.stdout: 10 51653 8.50 MB/sec execute 202 sec latency 77.122 ms 2024-03-31T23:30:18.291 INFO:tasks.workunit.client.0.smithi046.stdout: 10 51801 8.49 MB/sec execute 203 sec latency 97.501 ms 2024-03-31T23:30:18.944 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:30:18 smithi142 ceph-mon[35271]: pgmap v588: 129 pgs: 129 active+clean; 835 MiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 6.1 MiB/s wr, 340 op/s 2024-03-31T23:30:19.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:30:18 smithi162 ceph-mon[36237]: pgmap v588: 129 pgs: 129 active+clean; 835 MiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 6.1 MiB/s wr, 340 op/s 2024-03-31T23:30:19.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:30:18 smithi046 ceph-mon[32498]: pgmap v588: 129 pgs: 129 active+clean; 835 MiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 6.1 MiB/s wr, 340 op/s 2024-03-31T23:30:19.291 INFO:tasks.workunit.client.0.smithi046.stdout: 10 51991 8.53 MB/sec execute 204 sec latency 66.431 ms 2024-03-31T23:30:20.116 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:20.113+0000 7fa4e9780640 1 -- 172.21.15.46:0/1981796223 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa4e4101980 msgr2=0x7fa4e41051f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:30:20.117 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:20.113+0000 7fa4e9780640 1 --2- 172.21.15.46:0/1981796223 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa4e4101980 0x7fa4e41051f0 secure :-1 s=READY pgs=958 cs=0 l=1 rev1=1 crypto rx=0x7fa4d8009930 tx=0x7fa4d802bc80 comp rx=0 tx=0).stop 2024-03-31T23:30:20.117 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:20.113+0000 7fa4e9780640 1 -- 172.21.15.46:0/1981796223 shutdown_connections 2024-03-31T23:30:20.117 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:20.113+0000 7fa4e9780640 1 --2- 172.21.15.46:0/1981796223 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa4e4105920 0x7fa4e410d530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:30:20.117 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:20.113+0000 7fa4e9780640 1 --2- 172.21.15.46:0/1981796223 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa4e4101980 0x7fa4e41051f0 unknown :-1 s=CLOSED pgs=958 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:30:20.117 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:20.113+0000 7fa4e9780640 1 --2- 172.21.15.46:0/1981796223 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa4e4100fd0 0x7fa4e41013b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:30:20.117 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:20.113+0000 7fa4e9780640 1 -- 172.21.15.46:0/1981796223 >> 172.21.15.46:0/1981796223 conn(0x7fa4e4076be0 msgr2=0x7fa4e40fdf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:30:20.117 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:20.113+0000 7fa4e9780640 1 -- 172.21.15.46:0/1981796223 shutdown_connections 2024-03-31T23:30:20.117 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:20.113+0000 7fa4e9780640 1 -- 172.21.15.46:0/1981796223 wait complete. 2024-03-31T23:30:20.117 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:20.114+0000 7fa4e9780640 1 Processor -- start 2024-03-31T23:30:20.117 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:20.114+0000 7fa4e9780640 1 -- start start 2024-03-31T23:30:20.117 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:20.114+0000 7fa4e9780640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa4e4100fd0 0x7fa4e4198cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:30:20.117 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:20.114+0000 7fa4e9780640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa4e4105920 0x7fa4e4199210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:30:20.118 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:20.114+0000 7fa4e9780640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa4e4199750 0x7fa4e41a3d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:30:20.118 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:20.114+0000 7fa4e9780640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fa4e4110850 con 0x7fa4e4100fd0 2024-03-31T23:30:20.118 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:20.114+0000 7fa4e9780640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fa4e41106d0 con 0x7fa4e4105920 2024-03-31T23:30:20.118 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:20.114+0000 7fa4e9780640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fa4e41109d0 con 0x7fa4e4199750 2024-03-31T23:30:20.118 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:20.114+0000 7fa4e2ffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa4e4100fd0 0x7fa4e4198cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:30:20.118 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:20.114+0000 7fa4e2ffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa4e4100fd0 0x7fa4e4198cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:54566/0 (socket says 172.21.15.46:54566) 2024-03-31T23:30:20.118 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:20.114+0000 7fa4e2ffd640 1 -- 172.21.15.46:0/56565598 learned_addr learned my addr 172.21.15.46:0/56565598 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:30:20.118 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:20.114+0000 7fa4e27fc640 1 --2- 172.21.15.46:0/56565598 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa4e4105920 0x7fa4e4199210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:30:20.118 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:20.114+0000 7fa4e37fe640 1 --2- 172.21.15.46:0/56565598 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa4e4199750 0x7fa4e41a3d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:30:20.118 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:20.114+0000 7fa4e2ffd640 1 -- 172.21.15.46:0/56565598 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa4e4199750 msgr2=0x7fa4e41a3d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:30:20.118 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:20.114+0000 7fa4e2ffd640 1 --2- 172.21.15.46:0/56565598 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa4e4199750 0x7fa4e41a3d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:30:20.118 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:20.114+0000 7fa4e2ffd640 1 -- 172.21.15.46:0/56565598 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa4e4105920 msgr2=0x7fa4e4199210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:30:20.118 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:20.114+0000 7fa4e2ffd640 1 --2- 172.21.15.46:0/56565598 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa4e4105920 0x7fa4e4199210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:30:20.119 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:20.114+0000 7fa4e2ffd640 1 -- 172.21.15.46:0/56565598 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4e41a44d0 con 0x7fa4e4100fd0 2024-03-31T23:30:20.119 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:20.115+0000 7fa4e2ffd640 1 --2- 172.21.15.46:0/56565598 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa4e4100fd0 0x7fa4e4198cd0 secure :-1 s=READY pgs=959 cs=0 l=1 rev1=1 crypto rx=0x7fa4d400d950 tx=0x7fa4d400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:30:20.119 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:20.115+0000 7fa4cbfff640 1 -- 172.21.15.46:0/56565598 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4d4017030 con 0x7fa4e4100fd0 2024-03-31T23:30:20.119 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:20.115+0000 7fa4cbfff640 1 -- 172.21.15.46:0/56565598 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa4d4004020 con 0x7fa4e4100fd0 2024-03-31T23:30:20.119 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:20.115+0000 7fa4e9780640 1 -- 172.21.15.46:0/56565598 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa4e41a47c0 con 0x7fa4e4100fd0 2024-03-31T23:30:20.119 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:20.115+0000 7fa4cbfff640 1 -- 172.21.15.46:0/56565598 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4d40047a0 con 0x7fa4e4100fd0 2024-03-31T23:30:20.119 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:20.115+0000 7fa4e9780640 1 -- 172.21.15.46:0/56565598 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa4e41a4d30 con 0x7fa4e4100fd0 2024-03-31T23:30:20.121 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:20.117+0000 7fa4cbfff640 1 -- 172.21.15.46:0/56565598 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 25) ==== 137451+0+0 (secure 0 0 0) 0x7fa4d40041d0 con 0x7fa4e4100fd0 2024-03-31T23:30:20.121 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:20.118+0000 7fa4ec435740 1 Processor -- start 2024-03-31T23:30:20.121 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:20.118+0000 7fa4cbfff640 1 --2- 172.21.15.46:0/56565598 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fa4cc0a2730 0x7fa4cc0a4bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:30:20.121 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:20.118+0000 7fa4ec435740 1 -- start start 2024-03-31T23:30:20.121 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:20.118+0000 7fa4e37fe640 1 --2- 172.21.15.46:0/56565598 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fa4cc0a2730 0x7fa4cc0a4bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:30:20.122 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:20.118+0000 7fa4cbfff640 1 -- 172.21.15.46:0/56565598 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(124..124 src has 1..124) ==== 9554+0+0 (secure 0 0 0) 0x7fa4d40ce880 con 0x7fa4e4100fd0 2024-03-31T23:30:20.122 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:20.118+0000 7fa4ec435740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557a980f8ad0 0x557a980f8eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:30:20.122 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:20.119+0000 7fa4ec435740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557a980f93f0 0x557a98101c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:30:20.122 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:20.119+0000 7fa4ec435740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557a981021c0 0x557a981045b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:30:20.122 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:20.119+0000 7fa4ec435740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x557a980578f0 con 0x557a981021c0 2024-03-31T23:30:20.122 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:20.119+0000 7fa4ec435740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x557a980008d0 con 0x557a980f93f0 2024-03-31T23:30:20.122 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:20.119+0000 7fa4ec435740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x557a98053990 con 0x557a980f8ad0 2024-03-31T23:30:20.122 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:20.119+0000 7fa4e2ffd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557a980f8ad0 0x557a980f8eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:30:20.122 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:20.119+0000 7fa4e27fc640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557a980f93f0 0x557a98101c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:30:20.122 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:20.119+0000 7fa4e2ffd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557a980f8ad0 0x557a980f8eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.46:34292/0 (socket says 172.21.15.46:34292) 2024-03-31T23:30:20.122 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:20.119+0000 7fa4e2ffd640 1 -- 172.21.15.46:0/1049202175 learned_addr learned my addr 172.21.15.46:0/1049202175 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:30:20.123 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:20.119+0000 7fa4e37fe640 1 --2- 172.21.15.46:0/1049202175 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557a981021c0 0x557a981045b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:30:20.123 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:20.119+0000 7fa4e37fe640 1 --2- 172.21.15.46:0/56565598 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fa4cc0a2730 0x7fa4cc0a4bf0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fa4dc005fd0 tx=0x7fa4dc011810 comp rx=0 tx=0).ready entity=mgr.15654 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:30:20.123 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:20.119+0000 7fa4e2ffd640 1 -- 172.21.15.46:0/1049202175 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557a980f93f0 msgr2=0x557a98101c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:30:20.123 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:20.119+0000 7fa4e2ffd640 1 --2- 172.21.15.46:0/1049202175 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557a980f93f0 0x557a98101c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:30:20.123 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:20.119+0000 7fa4e2ffd640 1 -- 172.21.15.46:0/1049202175 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557a981021c0 msgr2=0x557a981045b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:30:20.123 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:20.119+0000 7fa4e2ffd640 1 --2- 172.21.15.46:0/1049202175 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557a981021c0 0x557a981045b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:30:20.123 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:20.119+0000 7fa4e2ffd640 1 -- 172.21.15.46:0/1049202175 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557a97d6c210 con 0x557a980f8ad0 2024-03-31T23:30:20.123 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:20.119+0000 7fa4e2ffd640 1 --2- 172.21.15.46:0/1049202175 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557a980f8ad0 0x557a980f8eb0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fa4d4097050 tx=0x7fa4d409a860 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4b0c9fcca76d2a31 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:30:20.123 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:20.120+0000 7fa4c97fa640 1 -- 172.21.15.46:0/1049202175 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4d40a3600 con 0x557a980f8ad0 2024-03-31T23:30:20.123 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:20.120+0000 7fa4c97fa640 1 -- 172.21.15.46:0/1049202175 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa4d40cbe40 con 0x557a980f8ad0 2024-03-31T23:30:20.123 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:20.120+0000 7fa4c97fa640 1 -- 172.21.15.46:0/1049202175 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4d4096940 con 0x557a980f8ad0 2024-03-31T23:30:20.128 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:20.121+0000 7fa4ec435740 1 -- 172.21.15.46:0/1049202175 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557a980f8ad0 msgr2=0x557a980f8eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:30:20.128 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:20.121+0000 7fa4ec435740 1 --2- 172.21.15.46:0/1049202175 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557a980f8ad0 0x557a980f8eb0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fa4d4097050 tx=0x7fa4d409a860 comp rx=0 tx=0).stop 2024-03-31T23:30:20.128 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:20.121+0000 7fa4ec435740 1 -- 172.21.15.46:0/1049202175 shutdown_connections 2024-03-31T23:30:20.128 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:20.121+0000 7fa4ec435740 1 --2- 172.21.15.46:0/1049202175 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557a981021c0 0x557a981045b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:30:20.128 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:20.121+0000 7fa4ec435740 1 --2- 172.21.15.46:0/1049202175 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557a980f93f0 0x557a98101c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:30:20.128 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:20.121+0000 7fa4ec435740 1 --2- 172.21.15.46:0/1049202175 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557a980f8ad0 0x557a980f8eb0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:30:20.128 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:20.121+0000 7fa4ec435740 1 -- 172.21.15.46:0/1049202175 >> 172.21.15.46:0/1049202175 conn(0x557a980f33f0 msgr2=0x557a980f37d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:30:20.129 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:20.121+0000 7fa4ec435740 1 -- 172.21.15.46:0/1049202175 shutdown_connections 2024-03-31T23:30:20.129 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:20.121+0000 7fa4ec435740 1 -- 172.21.15.46:0/1049202175 wait complete. 2024-03-31T23:30:20.129 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:20.121+0000 7fa4ec435740 1 -- start start 2024-03-31T23:30:20.129 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:20.121+0000 7fa4ec435740 1 Processor -- start 2024-03-31T23:30:20.129 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:20.122+0000 7fa4ec435740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557a980f8ad0 0x557a980ff2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:30:20.129 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:20.122+0000 7fa4ec435740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557a980f93f0 0x557a980ff810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:30:20.129 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:20.122+0000 7fa4ec435740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557a981021c0 0x557a980ffd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:30:20.129 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:20.122+0000 7fa4ec435740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x557a98104d30 con 0x557a980f93f0 2024-03-31T23:30:20.129 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:20.122+0000 7fa4ec435740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x557a97ff5f80 con 0x557a980f8ad0 2024-03-31T23:30:20.129 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:20.122+0000 7fa4ec435740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x557a98104eb0 con 0x557a981021c0 2024-03-31T23:30:20.129 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:20.122+0000 7fa4e27fc640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557a980f93f0 0x557a980ff810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:30:20.129 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:20.122+0000 7fa4e27fc640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557a980f93f0 0x557a980ff810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:54596/0 (socket says 172.21.15.46:54596) 2024-03-31T23:30:20.129 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:20.122+0000 7fa4e2ffd640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557a980f8ad0 0x557a980ff2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:30:20.129 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:20.122+0000 7fa4e27fc640 1 -- 172.21.15.46:0/4246421272 learned_addr learned my addr 172.21.15.46:0/4246421272 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:30:20.130 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:20.122+0000 7fa4ec435740 10 client.0 _ll_register_callbacks cb 0x557a9804e290 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T23:30:20.130 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:20.122+0000 7fa4e37fe640 1 --2- 172.21.15.46:0/4246421272 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557a981021c0 0x557a980ffd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:30:20.130 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:20.122+0000 7fa4e2ffd640 1 -- 172.21.15.46:0/4246421272 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557a981021c0 msgr2=0x557a980ffd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:30:20.130 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:20.122+0000 7fa4e2ffd640 1 --2- 172.21.15.46:0/4246421272 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557a981021c0 0x557a980ffd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:30:20.130 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:20.122+0000 7fa4e2ffd640 1 -- 172.21.15.46:0/4246421272 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557a980f93f0 msgr2=0x557a980ff810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:30:20.130 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:20.122+0000 7fa4e2ffd640 1 --2- 172.21.15.46:0/4246421272 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557a980f93f0 0x557a980ff810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:30:20.130 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:20.122+0000 7fa4e2ffd640 1 -- 172.21.15.46:0/4246421272 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557a97e38620 con 0x557a980f8ad0 2024-03-31T23:30:20.130 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:20.122+0000 7fa4e27fc640 1 --2- 172.21.15.46:0/4246421272 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557a980f93f0 0x557a980ff810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:30:20.130 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:20.122+0000 7fa4ec435740 1 -- 172.21.15.46:0/4246421272 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa4d40cbe40 con 0x557a980f8ad0 2024-03-31T23:30:20.130 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:20.122+0000 7fa4e2ffd640 1 --2- 172.21.15.46:0/4246421272 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557a980f8ad0 0x557a980ff2d0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fa4d40975c0 tx=0x7fa4d409ad60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6dd01730ba063e67 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:30:20.130 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:20.123+0000 7fa4c6ff5640 10 client.27482 ms_handle_connect on v2:172.21.15.142:3300/0 2024-03-31T23:30:20.130 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:20.123+0000 7fa4c6ff5640 1 -- 172.21.15.46:0/4246421272 <== mon.2 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4d407e050 con 0x557a980f8ad0 2024-03-31T23:30:20.130 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:20.123+0000 7fa4c6ff5640 1 -- 172.21.15.46:0/4246421272 <== mon.2 v2:172.21.15.142:3300/0 2 ==== osd_map(124..124 src has 1..124) ==== 9554+0+0 (secure 0 0 0) 0x7fa4d40b92a0 con 0x557a980f8ad0 2024-03-31T23:30:20.130 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:20.123+0000 7fa4c6ff5640 1 -- 172.21.15.46:0/4246421272 <== mon.2 v2:172.21.15.142:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) ==== 24+0+0 (secure 0 0 0) 0x7fa4d40a3600 con 0x557a980f8ad0 2024-03-31T23:30:20.131 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:20.123+0000 7fa4ec435740 10 client.27482 fetch_fsmap learned FSMap version 47 2024-03-31T23:30:20.131 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:20.123+0000 7fa4ec435740 1 -- 172.21.15.46:0/4246421272 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({fsmap=47}) -- 0x557a98110be0 con 0x557a980f8ad0 2024-03-31T23:30:20.131 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:20.123+0000 7fa4c6ff5640 1 -- 172.21.15.46:0/4246421272 <== mon.2 v2:172.21.15.142:3300/0 4 ==== fsmap(e 47) ==== 8346+0+0 (secure 0 0 0) 0x7fa4d40b3b90 con 0x557a980f8ad0 2024-03-31T23:30:20.131 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:20.124+0000 7fa4ec435740 10 client.27482 fetch_fsmap finished waiting for FSMap version 47 2024-03-31T23:30:20.131 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:20.124+0000 7fa4ec435740 10 client.27482 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-31T23:30:20.131 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:20.124+0000 7fa4ec435740 20 client.27482 populate_metadata read hostname 'smithi046' 2024-03-31T23:30:20.131 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:20.124+0000 7fa4ec435740 1 --2- 172.21.15.46:0/4246421272 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x557a98111f30 0x557a98114320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:30:20.131 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:20.124+0000 7fa4ec435740 4 client.27482 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-31T23:30:20.131 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:20.124+0000 7fa4ec435740 1 -- 172.21.15.46:0/4246421272 --> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557a9806b120 con 0x557a98111f30 2024-03-31T23:30:20.131 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:20.124+0000 7fa4e27fc640 1 --2- 172.21.15.46:0/4246421272 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x557a98111f30 0x557a98114320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:30:20.131 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:20.125+0000 7fa4c6ff5640 10 client.27482 ms_handle_connect on v2:172.21.15.142:6838/2519143918 2024-03-31T23:30:20.131 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:20.125+0000 7fa4e27fc640 1 --2- 172.21.15.46:0/4246421272 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x557a98111f30 0x557a98114320 crc :-1 s=READY pgs=242 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=685857acde6850a3 server_cookie=a511c25bcd99b775 in_seq=0 out_seq=0 2024-03-31T23:30:20.131 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:20.126+0000 7fa4c6ff5640 1 -- 172.21.15.46:0/4246421272 <== mds.0 v2:172.21.15.142:6838/2519143918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x557a9806b120 con 0x557a98111f30 2024-03-31T23:30:20.131 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:20.126+0000 7fa4c6ff5640 10 client.27482 handle_command_reply: tid=0 2024-03-31T23:30:20.132 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:20.126+0000 7fa4ec435740 1 client.27482 shutdown 2024-03-31T23:30:20.132 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:20.126+0000 7fa4ec435740 1 -- 172.21.15.46:0/4246421272 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557a980f8ad0 msgr2=0x557a980ff2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:30:20.132 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:20.126+0000 7fa4ec435740 1 --2- 172.21.15.46:0/4246421272 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557a980f8ad0 0x557a980ff2d0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fa4d40975c0 tx=0x7fa4d409ad60 comp rx=0 tx=0).stop 2024-03-31T23:30:20.132 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:20.126+0000 7fa4ec435740 1 -- 172.21.15.46:0/4246421272 shutdown_connections 2024-03-31T23:30:20.132 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:20.126+0000 7fa4ec435740 1 --2- 172.21.15.46:0/4246421272 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x557a98111f30 0x557a98114320 crc :-1 s=READY pgs=242 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:30:20.132 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:20.126+0000 7fa4ec435740 1 --2- 172.21.15.46:0/4246421272 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557a981021c0 0x557a980ffd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:30:20.132 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:20.126+0000 7fa4c6ff5640 0 client.27482 ms_handle_reset on v2:172.21.15.142:6838/2519143918 2024-03-31T23:30:20.132 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:20.126+0000 7fa4ec435740 1 --2- 172.21.15.46:0/4246421272 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557a980f93f0 0x557a980ff810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:30:20.132 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:20.126+0000 7fa4ec435740 1 --2- 172.21.15.46:0/4246421272 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557a980f8ad0 0x557a980ff2d0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:30:20.132 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:20.126+0000 7fa4ec435740 1 -- 172.21.15.46:0/4246421272 >> 172.21.15.46:0/4246421272 conn(0x557a980f33f0 msgr2=0x557a980f5140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:30:20.132 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:20.127+0000 7fa4ec435740 1 -- 172.21.15.46:0/4246421272 shutdown_connections 2024-03-31T23:30:20.132 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:20.127+0000 7fa4ec435740 1 -- 172.21.15.46:0/4246421272 wait complete. 2024-03-31T23:30:20.132 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:20.127+0000 7fa4ec435740 20 client.27482 trim_cache size 0 max 16384 2024-03-31T23:30:20.154 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:20.151+0000 7fa4ec435740 1 Processor -- start 2024-03-31T23:30:20.154 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:20.151+0000 7fa4ec435740 1 -- start start 2024-03-31T23:30:20.154 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:20.152+0000 7fa4ec435740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557a980f8ad0 0x557a98101a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:30:20.154 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:20.152+0000 7fa4ec435740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557a980f93f0 0x557a980ede00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:30:20.154 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:20.152+0000 7fa4ec435740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557a981021c0 0x557a98117340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:30:20.154 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:20.152+0000 7fa4ec435740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fa4e41af2d0 con 0x557a981021c0 2024-03-31T23:30:20.154 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:20.152+0000 7fa4ec435740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fa4e41acba0 con 0x557a980f93f0 2024-03-31T23:30:20.155 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:20.152+0000 7fa4ec435740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fa4e41aaa10 con 0x557a980f8ad0 2024-03-31T23:30:20.155 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:20.152+0000 7fa4e27fc640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557a980f93f0 0x557a980ede00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:30:20.155 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:20.152+0000 7fa4e37fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557a981021c0 0x557a98117340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:30:20.155 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:20.152+0000 7fa4e2ffd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557a980f8ad0 0x557a98101a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:30:20.155 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:20.152+0000 7fa4e2ffd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557a980f8ad0 0x557a98101a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.46:34312/0 (socket says 172.21.15.46:34312) 2024-03-31T23:30:20.155 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:20.152+0000 7fa4e27fc640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557a980f93f0 0x557a980ede00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.46:53210/0 (socket says 172.21.15.46:53210) 2024-03-31T23:30:20.155 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:20.152+0000 7fa4e2ffd640 1 -- 172.21.15.46:0/238950515 learned_addr learned my addr 172.21.15.46:0/238950515 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:30:20.155 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:20.152+0000 7fa4e2ffd640 1 -- 172.21.15.46:0/238950515 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557a980f93f0 msgr2=0x557a980ede00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:30:20.155 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:20.152+0000 7fa4e2ffd640 1 --2- 172.21.15.46:0/238950515 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557a980f93f0 0x557a980ede00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:30:20.155 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:20.152+0000 7fa4e2ffd640 1 -- 172.21.15.46:0/238950515 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557a981021c0 msgr2=0x557a98117340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:30:20.155 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:20.152+0000 7fa4e2ffd640 1 --2- 172.21.15.46:0/238950515 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557a981021c0 0x557a98117340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:30:20.155 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:20.152+0000 7fa4e2ffd640 1 -- 172.21.15.46:0/238950515 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557a980f7db0 con 0x557a980f8ad0 2024-03-31T23:30:20.155 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:20.152+0000 7fa4e37fe640 1 --2- 172.21.15.46:0/238950515 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557a981021c0 0x557a98117340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:30:20.155 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:20.153+0000 7fa4e2ffd640 1 --2- 172.21.15.46:0/238950515 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557a980f8ad0 0x557a98101a00 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fa4d4000c00 tx=0x7fa4d4014000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8043f23358966ada server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:30:20.156 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:20.153+0000 7fa4c8ff9640 1 -- 172.21.15.46:0/238950515 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4d407e040 con 0x557a980f8ad0 2024-03-31T23:30:20.156 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:20.153+0000 7fa4c8ff9640 1 -- 172.21.15.46:0/238950515 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa4d40cbe40 con 0x557a980f8ad0 2024-03-31T23:30:20.156 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:20.153+0000 7fa4c8ff9640 1 -- 172.21.15.46:0/238950515 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4d40b3b90 con 0x557a980f8ad0 2024-03-31T23:30:20.159 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:20.154+0000 7fa4ec435740 1 -- 172.21.15.46:0/238950515 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557a980f8ad0 msgr2=0x557a98101a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:30:20.159 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:20.154+0000 7fa4ec435740 1 --2- 172.21.15.46:0/238950515 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557a980f8ad0 0x557a98101a00 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fa4d4000c00 tx=0x7fa4d4014000 comp rx=0 tx=0).stop 2024-03-31T23:30:20.159 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:20.154+0000 7fa4ec435740 1 -- 172.21.15.46:0/238950515 shutdown_connections 2024-03-31T23:30:20.159 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:20.154+0000 7fa4ec435740 1 --2- 172.21.15.46:0/238950515 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557a981021c0 0x557a98117340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:30:20.159 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:20.154+0000 7fa4ec435740 1 --2- 172.21.15.46:0/238950515 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557a980f93f0 0x557a980ede00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:30:20.159 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:20.154+0000 7fa4ec435740 1 --2- 172.21.15.46:0/238950515 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557a980f8ad0 0x557a98101a00 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:30:20.159 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:20.154+0000 7fa4ec435740 1 -- 172.21.15.46:0/238950515 >> 172.21.15.46:0/238950515 conn(0x557a980f33f0 msgr2=0x557a980ed6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:30:20.159 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:20.154+0000 7fa4ec435740 1 -- 172.21.15.46:0/238950515 shutdown_connections 2024-03-31T23:30:20.159 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:20.154+0000 7fa4ec435740 1 -- 172.21.15.46:0/238950515 wait complete. 2024-03-31T23:30:20.159 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:20.154+0000 7fa4ec435740 1 -- start start 2024-03-31T23:30:20.159 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:20.154+0000 7fa4ec435740 1 Processor -- start 2024-03-31T23:30:20.159 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:20.155+0000 7fa4ec435740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557a980f8ad0 0x557a98114ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:30:20.160 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:20.155+0000 7fa4ec435740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557a980f93f0 0x557a98115530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:30:20.160 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:20.155+0000 7fa4ec435740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557a981021c0 0x557a98115ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:30:20.160 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:20.155+0000 7fa4ec435740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fa4e41ab060 con 0x557a981021c0 2024-03-31T23:30:20.160 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:20.155+0000 7fa4ec435740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fa4e41ac4d0 con 0x557a980f93f0 2024-03-31T23:30:20.160 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:20.155+0000 7fa4ec435740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fa4e41ab730 con 0x557a980f8ad0 2024-03-31T23:30:20.160 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:20.155+0000 7fa4ec435740 10 client.0 _ll_register_callbacks cb 0x7fa4dc009f50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T23:30:20.161 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:20.155+0000 7fa4e27fc640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557a980f93f0 0x557a98115530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:30:20.161 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:20.155+0000 7fa4e2ffd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557a980f8ad0 0x557a98114ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:30:20.161 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:20.155+0000 7fa4e27fc640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557a980f93f0 0x557a98115530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.46:53222/0 (socket says 172.21.15.46:53222) 2024-03-31T23:30:20.162 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:20.155+0000 7fa4e27fc640 1 -- 172.21.15.46:0/1779815161 learned_addr learned my addr 172.21.15.46:0/1779815161 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:30:20.162 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:20.155+0000 7fa4e37fe640 1 --2- 172.21.15.46:0/1779815161 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557a981021c0 0x557a98115ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:30:20.162 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:20.155+0000 7fa4e27fc640 1 -- 172.21.15.46:0/1779815161 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557a980f8ad0 msgr2=0x557a98114ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:30:20.162 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:20.155+0000 7fa4e27fc640 1 --2- 172.21.15.46:0/1779815161 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557a980f8ad0 0x557a98114ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:30:20.162 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:20.155+0000 7fa4e27fc640 1 -- 172.21.15.46:0/1779815161 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557a981021c0 msgr2=0x557a98115ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:30:20.162 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:20.155+0000 7fa4e27fc640 1 --2- 172.21.15.46:0/1779815161 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557a981021c0 0x557a98115ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:30:20.162 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:20.155+0000 7fa4e27fc640 1 -- 172.21.15.46:0/1779815161 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557a981771c0 con 0x557a980f93f0 2024-03-31T23:30:20.162 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:20.155+0000 7fa4e37fe640 1 --2- 172.21.15.46:0/1779815161 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557a981021c0 0x557a98115ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:30:20.162 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:20.155+0000 7fa4e27fc640 1 --2- 172.21.15.46:0/1779815161 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557a980f93f0 0x557a98115530 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fa4d802c1c0 tx=0x7fa4d802c5c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9336acf0084f2fec server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:30:20.162 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:20.155+0000 7fa4c6ff5640 10 client.0 ms_handle_connect on v2:172.21.15.142:3300/0 2024-03-31T23:30:20.162 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:20.156+0000 7fa4c6ff5640 1 -- 172.21.15.46:0/1779815161 <== mon.2 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4d804f070 con 0x557a980f93f0 2024-03-31T23:30:20.162 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:20.156+0000 7fa4ec435740 1 -- 172.21.15.46:0/1779815161 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa4d40cbe40 con 0x557a980f93f0 2024-03-31T23:30:20.162 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:20.156+0000 7fa4c6ff5640 1 -- 172.21.15.46:0/1779815161 <== mon.2 v2:172.21.15.142:3300/0 2 ==== osd_map(124..124 src has 1..124) ==== 9554+0+0 (secure 0 0 0) 0x7fa4d804bcc0 con 0x557a980f93f0 2024-03-31T23:30:20.162 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:20.156+0000 7fa4c6ff5640 1 -- 172.21.15.46:0/1779815161 <== mon.2 v2:172.21.15.142:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) ==== 24+0+0 (secure 0 0 0) 0x7fa4d805b050 con 0x557a980f93f0 2024-03-31T23:30:20.163 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:20.156+0000 7fa4ec435740 10 client.27488 fetch_fsmap learned FSMap version 47 2024-03-31T23:30:20.163 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:20.156+0000 7fa4ec435740 1 -- 172.21.15.46:0/1779815161 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) -- 0x557a98178530 con 0x557a980f93f0 2024-03-31T23:30:20.163 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:20.156+0000 7fa4c6ff5640 1 -- 172.21.15.46:0/1779815161 <== mon.2 v2:172.21.15.142:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa4d40cbe40 con 0x557a980f93f0 2024-03-31T23:30:20.163 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:20.156+0000 7fa4c6ff5640 1 -- 172.21.15.46:0/1779815161 <== mon.2 v2:172.21.15.142:3300/0 5 ==== fsmap(e 47) ==== 8346+0+0 (secure 0 0 0) 0x7fa4d8005040 con 0x557a980f93f0 2024-03-31T23:30:20.163 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:20.157+0000 7fa4ec435740 10 client.27488 fetch_fsmap finished waiting for FSMap version 47 2024-03-31T23:30:20.163 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:20.157+0000 7fa4ec435740 10 client.27488 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-31T23:30:20.163 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:20.157+0000 7fa4ec435740 20 client.27488 populate_metadata read hostname 'smithi046' 2024-03-31T23:30:20.163 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:20.157+0000 7fa4ec435740 1 --2- 172.21.15.46:0/1779815161 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x557a98178c30 0x557a9817b050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:30:20.163 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:20.157+0000 7fa4ec435740 4 client.27488 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-31T23:30:20.163 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:20.157+0000 7fa4ec435740 1 -- 172.21.15.46:0/1779815161 --> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] -- command(tid 0: {"prefix": "flush journal"}) -- 0x557a98064e30 con 0x557a98178c30 2024-03-31T23:30:20.164 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:20.157+0000 7fa4e2ffd640 1 --2- 172.21.15.46:0/1779815161 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x557a98178c30 0x557a9817b050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:30:20.164 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:20.157+0000 7fa4c6ff5640 10 client.27488 ms_handle_connect on v2:172.21.15.142:6838/2519143918 2024-03-31T23:30:20.164 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:20.157+0000 7fa4e2ffd640 1 --2- 172.21.15.46:0/1779815161 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x557a98178c30 0x557a9817b050 crc :-1 s=READY pgs=243 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=50b45ef920be385b server_cookie=2f3005b4b6cdb760 in_seq=0 out_seq=0 2024-03-31T23:30:20.291 INFO:tasks.workunit.client.0.smithi046.stdout: 10 52137 8.52 MB/sec execute 205 sec latency 131.137 ms 2024-03-31T23:30:20.342 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:20.340+0000 7fa4c6ff5640 1 -- 172.21.15.46:0/1779815161 <== mds.0 v2:172.21.15.142:6838/2519143918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x557a98064e30 con 0x557a98178c30 2024-03-31T23:30:20.342 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:20.340+0000 7fa4c6ff5640 10 client.27488 handle_command_reply: tid=0 2024-03-31T23:30:20.343 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:20.341+0000 7fa4ec435740 1 client.27488 shutdown 2024-03-31T23:30:20.343 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:20.341+0000 7fa4ec435740 1 -- 172.21.15.46:0/1779815161 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557a980f93f0 msgr2=0x557a98115530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:30:20.343 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:20.341+0000 7fa4ec435740 1 --2- 172.21.15.46:0/1779815161 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557a980f93f0 0x557a98115530 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fa4d802c1c0 tx=0x7fa4d802c5c0 comp rx=0 tx=0).stop 2024-03-31T23:30:20.343 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:20.341+0000 7fa4ec435740 1 -- 172.21.15.46:0/1779815161 shutdown_connections 2024-03-31T23:30:20.343 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:20.341+0000 7fa4ec435740 1 --2- 172.21.15.46:0/1779815161 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x557a98178c30 0x557a9817b050 crc :-1 s=READY pgs=243 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:30:20.343 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:20.341+0000 7fa4ec435740 1 --2- 172.21.15.46:0/1779815161 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557a981021c0 0x557a98115ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:30:20.343 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:20.341+0000 7fa4ec435740 1 --2- 172.21.15.46:0/1779815161 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557a980f93f0 0x557a98115530 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:30:20.343 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:20.341+0000 7fa4ec435740 1 --2- 172.21.15.46:0/1779815161 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557a980f8ad0 0x557a98114ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:30:20.343 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:20.341+0000 7fa4c6ff5640 0 client.27488 ms_handle_reset on v2:172.21.15.142:6838/2519143918 2024-03-31T23:30:20.343 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:20.341+0000 7fa4ec435740 1 -- 172.21.15.46:0/1779815161 >> 172.21.15.46:0/1779815161 conn(0x557a980f33f0 msgr2=0x557a980edf80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:30:20.343 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:20.341+0000 7fa4ec435740 1 -- 172.21.15.46:0/1779815161 shutdown_connections 2024-03-31T23:30:20.343 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:20.341+0000 7fa4ec435740 1 -- 172.21.15.46:0/1779815161 wait complete. 2024-03-31T23:30:20.343 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:20.341+0000 7fa4ec435740 20 client.27488 trim_cache size 0 max 16384 2024-03-31T23:30:20.344 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout:{ 2024-03-31T23:30:20.344 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout: "message": "", 2024-03-31T23:30:20.344 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout: "return_code": 0 2024-03-31T23:30:20.344 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout:} 2024-03-31T23:30:20.345 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:20.342+0000 7fa4e9780640 1 -- 172.21.15.46:0/56565598 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fa4cc0a2730 msgr2=0x7fa4cc0a4bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:30:20.345 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:20.342+0000 7fa4e9780640 1 --2- 172.21.15.46:0/56565598 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fa4cc0a2730 0x7fa4cc0a4bf0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fa4dc005fd0 tx=0x7fa4dc011810 comp rx=0 tx=0).stop 2024-03-31T23:30:20.345 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:20.342+0000 7fa4e9780640 1 -- 172.21.15.46:0/56565598 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa4e4100fd0 msgr2=0x7fa4e4198cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:30:20.345 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:20.342+0000 7fa4e9780640 1 --2- 172.21.15.46:0/56565598 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa4e4100fd0 0x7fa4e4198cd0 secure :-1 s=READY pgs=959 cs=0 l=1 rev1=1 crypto rx=0x7fa4d400d950 tx=0x7fa4d400de10 comp rx=0 tx=0).stop 2024-03-31T23:30:20.345 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:20.342+0000 7fa4e9780640 1 -- 172.21.15.46:0/56565598 shutdown_connections 2024-03-31T23:30:20.345 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:20.342+0000 7fa4e9780640 1 --2- 172.21.15.46:0/56565598 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fa4cc0a2730 0x7fa4cc0a4bf0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:30:20.345 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:20.342+0000 7fa4e9780640 1 --2- 172.21.15.46:0/56565598 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa4e4199750 0x7fa4e41a3d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:30:20.345 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:20.342+0000 7fa4e9780640 1 --2- 172.21.15.46:0/56565598 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa4e4105920 0x7fa4e4199210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:30:20.345 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:20.342+0000 7fa4e9780640 1 --2- 172.21.15.46:0/56565598 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa4e4100fd0 0x7fa4e4198cd0 unknown :-1 s=CLOSED pgs=959 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:30:20.345 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:20.342+0000 7fa4e9780640 1 -- 172.21.15.46:0/56565598 >> 172.21.15.46:0/56565598 conn(0x7fa4e4076be0 msgr2=0x7fa4e40fdf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:30:20.345 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:20.342+0000 7fa4e9780640 1 -- 172.21.15.46:0/56565598 shutdown_connections 2024-03-31T23:30:20.345 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:20.342+0000 7fa4e9780640 1 -- 172.21.15.46:0/56565598 wait complete. 2024-03-31T23:30:20.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:30:20 smithi142 ceph-mon[35271]: pgmap v589: 129 pgs: 129 active+clean; 838 MiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 5.1 MiB/s wr, 285 op/s 2024-03-31T23:30:21.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:30:20 smithi162 ceph-mon[36237]: pgmap v589: 129 pgs: 129 active+clean; 838 MiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 5.1 MiB/s wr, 285 op/s 2024-03-31T23:30:21.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:30:20 smithi046 ceph-mon[32498]: pgmap v589: 129 pgs: 129 active+clean; 838 MiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 5.1 MiB/s wr, 285 op/s 2024-03-31T23:30:21.291 INFO:tasks.workunit.client.0.smithi046.stdout: 10 52323 8.53 MB/sec execute 206 sec latency 162.373 ms 2024-03-31T23:30:22.291 INFO:tasks.workunit.client.0.smithi046.stdout: 10 52483 8.54 MB/sec execute 207 sec latency 89.774 ms 2024-03-31T23:30:22.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:30:22 smithi142 ceph-mon[35271]: pgmap v590: 129 pgs: 129 active+clean; 838 MiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 4.9 MiB/s wr, 276 op/s 2024-03-31T23:30:23.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:30:22 smithi162 ceph-mon[36237]: pgmap v590: 129 pgs: 129 active+clean; 838 MiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 4.9 MiB/s wr, 276 op/s 2024-03-31T23:30:23.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:30:22 smithi046 ceph-mon[32498]: pgmap v590: 129 pgs: 129 active+clean; 838 MiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 4.9 MiB/s wr, 276 op/s 2024-03-31T23:30:23.291 INFO:tasks.workunit.client.0.smithi046.stdout: 10 52611 8.52 MB/sec execute 208 sec latency 154.471 ms 2024-03-31T23:30:24.291 INFO:tasks.workunit.client.0.smithi046.stdout: 10 52797 8.54 MB/sec execute 209 sec latency 64.329 ms 2024-03-31T23:30:24.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:30:24 smithi142 ceph-mon[35271]: pgmap v591: 129 pgs: 129 active+clean; 838 MiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 4.2 MiB/s wr, 238 op/s 2024-03-31T23:30:25.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:30:24 smithi162 ceph-mon[36237]: pgmap v591: 129 pgs: 129 active+clean; 838 MiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 4.2 MiB/s wr, 238 op/s 2024-03-31T23:30:25.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:30:24 smithi046 ceph-mon[32498]: pgmap v591: 129 pgs: 129 active+clean; 838 MiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 4.2 MiB/s wr, 238 op/s 2024-03-31T23:30:25.291 INFO:tasks.workunit.client.0.smithi046.stdout: 10 52958 8.52 MB/sec execute 210 sec latency 82.940 ms 2024-03-31T23:30:26.291 INFO:tasks.workunit.client.0.smithi046.stdout: 10 53109 8.53 MB/sec execute 211 sec latency 109.978 ms 2024-03-31T23:30:26.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:30:26 smithi142 ceph-mon[35271]: pgmap v592: 129 pgs: 129 active+clean; 844 MiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 6.3 MiB/s wr, 393 op/s 2024-03-31T23:30:27.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:30:26 smithi162 ceph-mon[36237]: pgmap v592: 129 pgs: 129 active+clean; 844 MiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 6.3 MiB/s wr, 393 op/s 2024-03-31T23:30:27.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:30:26 smithi046 ceph-mon[32498]: pgmap v592: 129 pgs: 129 active+clean; 844 MiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 6.3 MiB/s wr, 393 op/s 2024-03-31T23:30:27.291 INFO:tasks.workunit.client.0.smithi046.stdout: 10 53241 8.53 MB/sec execute 212 sec latency 179.422 ms 2024-03-31T23:30:28.291 INFO:tasks.workunit.client.0.smithi046.stdout: 10 53420 8.55 MB/sec execute 213 sec latency 68.339 ms 2024-03-31T23:30:28.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:30:28 smithi142 ceph-mon[35271]: pgmap v593: 129 pgs: 129 active+clean; 844 MiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 4.0 MiB/s wr, 275 op/s 2024-03-31T23:30:29.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:30:28 smithi162 ceph-mon[36237]: pgmap v593: 129 pgs: 129 active+clean; 844 MiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 4.0 MiB/s wr, 275 op/s 2024-03-31T23:30:29.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:30:28 smithi046 ceph-mon[32498]: pgmap v593: 129 pgs: 129 active+clean; 844 MiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 4.0 MiB/s wr, 275 op/s 2024-03-31T23:30:29.291 INFO:tasks.workunit.client.0.smithi046.stdout: 10 53572 8.56 MB/sec execute 214 sec latency 107.978 ms 2024-03-31T23:30:30.291 INFO:tasks.workunit.client.0.smithi046.stdout: 10 53728 8.55 MB/sec execute 215 sec latency 95.671 ms 2024-03-31T23:30:30.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:30:30 smithi142 ceph-mon[35271]: pgmap v594: 129 pgs: 129 active+clean; 861 MiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 6.5 MiB/s wr, 380 op/s 2024-03-31T23:30:31.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:30:30 smithi162 ceph-mon[36237]: pgmap v594: 129 pgs: 129 active+clean; 861 MiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 6.5 MiB/s wr, 380 op/s 2024-03-31T23:30:31.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:30:30 smithi046 ceph-mon[32498]: pgmap v594: 129 pgs: 129 active+clean; 861 MiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 6.5 MiB/s wr, 380 op/s 2024-03-31T23:30:31.291 INFO:tasks.workunit.client.0.smithi046.stdout: 10 53881 8.56 MB/sec execute 216 sec latency 130.395 ms 2024-03-31T23:30:32.291 INFO:tasks.workunit.client.0.smithi046.stdout: 10 54012 8.55 MB/sec execute 217 sec latency 101.432 ms 2024-03-31T23:30:32.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:30:32 smithi142 ceph-mon[35271]: pgmap v595: 129 pgs: 129 active+clean; 861 MiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 4.6 MiB/s wr, 260 op/s 2024-03-31T23:30:33.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:30:32 smithi162 ceph-mon[36237]: pgmap v595: 129 pgs: 129 active+clean; 861 MiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 4.6 MiB/s wr, 260 op/s 2024-03-31T23:30:33.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:30:32 smithi046 ceph-mon[32498]: pgmap v595: 129 pgs: 129 active+clean; 861 MiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 4.6 MiB/s wr, 260 op/s 2024-03-31T23:30:33.291 INFO:tasks.workunit.client.0.smithi046.stdout: 10 54147 8.54 MB/sec execute 218 sec latency 118.203 ms 2024-03-31T23:30:33.484 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:33.481+0000 7fbaf566b640 1 -- 172.21.15.46:0/3197385215 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbae8094430 msgr2=0x7fbae8094810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:30:33.485 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:33.481+0000 7fbaf566b640 1 --2- 172.21.15.46:0/3197385215 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbae8094430 0x7fbae8094810 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fbae0009900 tx=0x7fbae002bc30 comp rx=0 tx=0).stop 2024-03-31T23:30:33.485 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:33.482+0000 7fbaf566b640 1 -- 172.21.15.46:0/3197385215 shutdown_connections 2024-03-31T23:30:33.485 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:33.482+0000 7fbaf566b640 1 --2- 172.21.15.46:0/3197385215 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbae80a03a0 0x7fbae80a2800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:30:33.485 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:33.482+0000 7fbaf566b640 1 --2- 172.21.15.46:0/3197385215 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbae8094d50 0x7fbae809fdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:30:33.485 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:33.482+0000 7fbaf566b640 1 --2- 172.21.15.46:0/3197385215 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbae8094430 0x7fbae8094810 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:30:33.485 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:33.482+0000 7fbaf566b640 1 -- 172.21.15.46:0/3197385215 >> 172.21.15.46:0/3197385215 conn(0x7fbae8009ab0 msgr2=0x7fbae8009ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:30:33.485 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:33.482+0000 7fbaf566b640 1 -- 172.21.15.46:0/3197385215 shutdown_connections 2024-03-31T23:30:33.485 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:33.482+0000 7fbaf566b640 1 -- 172.21.15.46:0/3197385215 wait complete. 2024-03-31T23:30:33.485 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:33.482+0000 7fbaf566b640 1 Processor -- start 2024-03-31T23:30:33.485 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:33.483+0000 7fbaf566b640 1 -- start start 2024-03-31T23:30:33.485 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:33.483+0000 7fbaf566b640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbae8094430 0x7fbae81343e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:30:33.485 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:33.483+0000 7fbaf566b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbae8094d50 0x7fbae8134920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:30:33.486 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:33.483+0000 7fbaf566b640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbae80a03a0 0x7fbae812e590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:30:33.487 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:33.483+0000 7fbaf566b640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fbae80a6070 con 0x7fbae8094d50 2024-03-31T23:30:33.487 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:33.483+0000 7fbaf566b640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fbae80a5ef0 con 0x7fbae8094430 2024-03-31T23:30:33.487 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:33.483+0000 7fbaf566b640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fbae80a61f0 con 0x7fbae80a03a0 2024-03-31T23:30:33.487 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:33.483+0000 7fbaee7fc640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbae8094d50 0x7fbae8134920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:30:33.487 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:33.483+0000 7fbaee7fc640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbae8094d50 0x7fbae8134920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:42790/0 (socket says 172.21.15.46:42790) 2024-03-31T23:30:33.488 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:33.483+0000 7fbaee7fc640 1 -- 172.21.15.46:0/3161150558 learned_addr learned my addr 172.21.15.46:0/3161150558 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:30:33.488 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:33.483+0000 7fbaeeffd640 1 --2- 172.21.15.46:0/3161150558 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbae8094430 0x7fbae81343e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:30:33.488 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:33.483+0000 7fbaef7fe640 1 --2- 172.21.15.46:0/3161150558 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbae80a03a0 0x7fbae812e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:30:33.488 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:33.484+0000 7fbaee7fc640 1 -- 172.21.15.46:0/3161150558 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbae80a03a0 msgr2=0x7fbae812e590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:30:33.488 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:33.484+0000 7fbaee7fc640 1 --2- 172.21.15.46:0/3161150558 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbae80a03a0 0x7fbae812e590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:30:33.488 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:33.484+0000 7fbaee7fc640 1 -- 172.21.15.46:0/3161150558 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbae8094430 msgr2=0x7fbae81343e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:30:33.488 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:33.484+0000 7fbaee7fc640 1 --2- 172.21.15.46:0/3161150558 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbae8094430 0x7fbae81343e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:30:33.488 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:33.484+0000 7fbaee7fc640 1 -- 172.21.15.46:0/3161150558 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbae812ed10 con 0x7fbae8094d50 2024-03-31T23:30:33.488 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:33.484+0000 7fbaef7fe640 1 --2- 172.21.15.46:0/3161150558 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbae80a03a0 0x7fbae812e590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T23:30:33.488 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:33.484+0000 7fbaee7fc640 1 --2- 172.21.15.46:0/3161150558 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbae8094d50 0x7fbae8134920 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 crypto rx=0x7fbae400e9f0 tx=0x7fbae400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:30:33.488 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:33.484+0000 7fbad7fff640 1 -- 172.21.15.46:0/3161150558 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbae400ccc0 con 0x7fbae8094d50 2024-03-31T23:30:33.488 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:33.484+0000 7fbaf566b640 1 -- 172.21.15.46:0/3161150558 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbae812f000 con 0x7fbae8094d50 2024-03-31T23:30:33.489 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:33.484+0000 7fbaf566b640 1 -- 172.21.15.46:0/3161150558 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbae8097d70 con 0x7fbae8094d50 2024-03-31T23:30:33.489 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:33.484+0000 7fbad7fff640 1 -- 172.21.15.46:0/3161150558 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fbae40218f0 con 0x7fbae8094d50 2024-03-31T23:30:33.489 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:33.484+0000 7fbad7fff640 1 -- 172.21.15.46:0/3161150558 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbae4010860 con 0x7fbae8094d50 2024-03-31T23:30:33.490 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:33.487+0000 7fbad7fff640 1 -- 172.21.15.46:0/3161150558 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 25) ==== 137451+0+0 (secure 0 0 0) 0x7fbae4010a00 con 0x7fbae8094d50 2024-03-31T23:30:33.491 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:33.488+0000 7fbaf8249740 1 Processor -- start 2024-03-31T23:30:33.491 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:33.488+0000 7fbaf8249740 1 -- start start 2024-03-31T23:30:33.491 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:33.488+0000 7fbaf8249740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55bf838e88d0 0x55bf838e8cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:30:33.491 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:33.488+0000 7fbaf8249740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bf838e91f0 0x55bf838f1a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:30:33.491 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:33.488+0000 7fbaf8249740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55bf838f1fc0 0x55bf838f43b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:30:33.491 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:33.488+0000 7fbaf8249740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55bf8384b8f0 con 0x55bf838e88d0 2024-03-31T23:30:33.491 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:33.488+0000 7fbaf8249740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55bf837f48d0 con 0x55bf838f1fc0 2024-03-31T23:30:33.492 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:33.488+0000 7fbaf8249740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55bf83847990 con 0x55bf838e91f0 2024-03-31T23:30:33.492 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:33.489+0000 7fbad7fff640 1 --2- 172.21.15.46:0/3161150558 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fbad80a2690 0x7fbad80a4b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:30:33.492 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:33.489+0000 7fbaeeffd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bf838e91f0 0x55bf838f1a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:30:33.492 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:33.489+0000 7fbaeeffd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bf838e91f0 0x55bf838f1a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.46:49332/0 (socket says 172.21.15.46:49332) 2024-03-31T23:30:33.492 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:33.489+0000 7fbaeeffd640 1 -- 172.21.15.46:0/3727580825 learned_addr learned my addr 172.21.15.46:0/3727580825 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:30:33.492 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:33.489+0000 7fbaee7fc640 1 --2- 172.21.15.46:0/3727580825 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55bf838f1fc0 0x55bf838f43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:30:33.492 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:33.489+0000 7fbaef7fe640 1 --2- 172.21.15.46:0/3727580825 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55bf838e88d0 0x55bf838e8cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:30:33.492 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:33.489+0000 7fbad7fff640 1 -- 172.21.15.46:0/3161150558 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(124..124 src has 1..124) ==== 9554+0+0 (secure 0 0 0) 0x7fbae40cf2e0 con 0x7fbae8094d50 2024-03-31T23:30:33.492 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:33.489+0000 7fbaeeffd640 1 -- 172.21.15.46:0/3727580825 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55bf838f1fc0 msgr2=0x55bf838f43b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:30:33.492 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:33.489+0000 7fbaeeffd640 1 --2- 172.21.15.46:0/3727580825 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55bf838f1fc0 0x55bf838f43b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:30:33.492 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:33.489+0000 7fbaeeffd640 1 -- 172.21.15.46:0/3727580825 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55bf838e88d0 msgr2=0x55bf838e8cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:30:33.492 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:33.489+0000 7fbaeeffd640 1 --2- 172.21.15.46:0/3727580825 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55bf838e88d0 0x55bf838e8cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:30:33.492 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:33.489+0000 7fbaeeffd640 1 -- 172.21.15.46:0/3727580825 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bf83560210 con 0x55bf838e91f0 2024-03-31T23:30:33.493 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:33.489+0000 7fbaef7fe640 1 --2- 172.21.15.46:0/3161150558 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fbad80a2690 0x7fbad80a4b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:30:33.493 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:33.489+0000 7fbaeeffd640 1 --2- 172.21.15.46:0/3727580825 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bf838e91f0 0x55bf838f1a80 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fbae0006010 tx=0x7fbae0009630 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e55f8d8bb3110a3d server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:30:33.493 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:33.490+0000 7fbad57fa640 1 -- 172.21.15.46:0/3727580825 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbae0050070 con 0x55bf838e91f0 2024-03-31T23:30:33.493 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:33.490+0000 7fbad57fa640 1 -- 172.21.15.46:0/3727580825 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fbae0004040 con 0x55bf838e91f0 2024-03-31T23:30:33.493 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:33.490+0000 7fbad57fa640 1 -- 172.21.15.46:0/3727580825 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbae0005040 con 0x55bf838e91f0 2024-03-31T23:30:33.499 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:33.490+0000 7fbaef7fe640 1 --2- 172.21.15.46:0/3161150558 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fbad80a2690 0x7fbad80a4b50 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fbaf0067000 tx=0x7fbaf0067690 comp rx=0 tx=0).ready entity=mgr.15654 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:30:33.499 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:33.490+0000 7fbaf8249740 1 -- 172.21.15.46:0/3727580825 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bf838e91f0 msgr2=0x55bf838f1a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:30:33.499 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:33.490+0000 7fbaf8249740 1 --2- 172.21.15.46:0/3727580825 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bf838e91f0 0x55bf838f1a80 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fbae0006010 tx=0x7fbae0009630 comp rx=0 tx=0).stop 2024-03-31T23:30:33.499 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:33.490+0000 7fbaf8249740 1 -- 172.21.15.46:0/3727580825 shutdown_connections 2024-03-31T23:30:33.500 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:33.490+0000 7fbaf8249740 1 --2- 172.21.15.46:0/3727580825 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55bf838f1fc0 0x55bf838f43b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:30:33.500 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:33.490+0000 7fbaf8249740 1 --2- 172.21.15.46:0/3727580825 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bf838e91f0 0x55bf838f1a80 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:30:33.500 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:33.490+0000 7fbaf8249740 1 --2- 172.21.15.46:0/3727580825 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55bf838e88d0 0x55bf838e8cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:30:33.500 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:33.490+0000 7fbaf8249740 1 -- 172.21.15.46:0/3727580825 >> 172.21.15.46:0/3727580825 conn(0x55bf838e31f0 msgr2=0x55bf838e35d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:30:33.500 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:33.491+0000 7fbaf8249740 1 -- 172.21.15.46:0/3727580825 shutdown_connections 2024-03-31T23:30:33.500 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:33.491+0000 7fbaf8249740 1 -- 172.21.15.46:0/3727580825 wait complete. 2024-03-31T23:30:33.500 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:33.491+0000 7fbaf8249740 1 -- start start 2024-03-31T23:30:33.500 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:33.491+0000 7fbaf8249740 1 Processor -- start 2024-03-31T23:30:33.500 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:33.491+0000 7fbaf8249740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bf838e88d0 0x55bf838eef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:30:33.500 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:33.492+0000 7fbaf8249740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55bf838e91f0 0x55bf838ef460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:30:33.500 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:33.492+0000 7fbaf8249740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55bf838f1fc0 0x55bf838ef9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:30:33.500 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:33.492+0000 7fbaf8249740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55bf838f4b90 con 0x55bf838f1fc0 2024-03-31T23:30:33.500 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:33.492+0000 7fbaf8249740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55bf837e9f80 con 0x55bf838e91f0 2024-03-31T23:30:33.500 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:33.492+0000 7fbaf8249740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55bf838f4d10 con 0x55bf838e88d0 2024-03-31T23:30:33.501 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:33.492+0000 7fbaeeffd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bf838e88d0 0x55bf838eef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:30:33.501 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:33.492+0000 7fbaee7fc640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55bf838e91f0 0x55bf838ef460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:30:33.501 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:33.492+0000 7fbaee7fc640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55bf838e91f0 0x55bf838ef460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.46:54190/0 (socket says 172.21.15.46:54190) 2024-03-31T23:30:33.501 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:33.492+0000 7fbaf8249740 10 client.0 _ll_register_callbacks cb 0x55bf83842290 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T23:30:33.501 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:33.492+0000 7fbaef7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55bf838f1fc0 0x55bf838ef9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:30:33.501 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:33.492+0000 7fbaee7fc640 1 -- 172.21.15.46:0/2132269828 learned_addr learned my addr 172.21.15.46:0/2132269828 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:30:33.501 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:33.492+0000 7fbaeeffd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bf838e88d0 0x55bf838eef20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.46:49342/0 (socket says 172.21.15.46:49342) 2024-03-31T23:30:33.501 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:33.492+0000 7fbaee7fc640 1 -- 172.21.15.46:0/2132269828 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bf838e88d0 msgr2=0x55bf838eef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:30:33.501 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:33.492+0000 7fbaee7fc640 1 --2- 172.21.15.46:0/2132269828 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bf838e88d0 0x55bf838eef20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:30:33.501 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:33.492+0000 7fbaee7fc640 1 -- 172.21.15.46:0/2132269828 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55bf838f1fc0 msgr2=0x55bf838ef9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:30:33.501 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:33.492+0000 7fbaee7fc640 1 --2- 172.21.15.46:0/2132269828 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55bf838f1fc0 0x55bf838ef9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:30:33.501 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:33.492+0000 7fbaee7fc640 1 -- 172.21.15.46:0/2132269828 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bf8362c620 con 0x55bf838e91f0 2024-03-31T23:30:33.501 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:33.492+0000 7fbaeeffd640 1 --2- 172.21.15.46:0/2132269828 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bf838e88d0 0x55bf838eef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:30:33.502 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:33.492+0000 7fbaee7fc640 1 --2- 172.21.15.46:0/2132269828 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55bf838e91f0 0x55bf838ef460 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fbae400cfd0 tx=0x7fbae409af50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a23b647549959c6b server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:30:33.502 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:33.492+0000 7fbad2ff5640 10 client.0 ms_handle_connect on v2:172.21.15.142:3300/0 2024-03-31T23:30:33.502 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:33.493+0000 7fbad2ff5640 1 -- 172.21.15.46:0/2132269828 <== mon.2 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbae40cca40 con 0x55bf838e91f0 2024-03-31T23:30:33.502 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:33.493+0000 7fbad2ff5640 1 -- 172.21.15.46:0/2132269828 <== mon.2 v2:172.21.15.142:3300/0 2 ==== osd_map(124..124 src has 1..124) ==== 9554+0+0 (secure 0 0 0) 0x7fbae40a3b70 con 0x55bf838e91f0 2024-03-31T23:30:33.502 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:33.493+0000 7fbaf8249740 1 -- 172.21.15.46:0/2132269828 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbae0004040 con 0x55bf838e91f0 2024-03-31T23:30:33.502 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:33.493+0000 7fbad2ff5640 1 -- 172.21.15.46:0/2132269828 <== mon.2 v2:172.21.15.142:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) ==== 24+0+0 (secure 0 0 0) 0x7fbae409f740 con 0x55bf838e91f0 2024-03-31T23:30:33.502 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:33.493+0000 7fbaf8249740 10 client.27500 fetch_fsmap learned FSMap version 47 2024-03-31T23:30:33.502 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:33.493+0000 7fbaf8249740 1 -- 172.21.15.46:0/2132269828 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) -- 0x55bf83900ab0 con 0x55bf838e91f0 2024-03-31T23:30:33.502 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:33.493+0000 7fbad2ff5640 1 -- 172.21.15.46:0/2132269828 <== mon.2 v2:172.21.15.142:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fbae0004040 con 0x55bf838e91f0 2024-03-31T23:30:33.502 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:33.494+0000 7fbad2ff5640 1 -- 172.21.15.46:0/2132269828 <== mon.2 v2:172.21.15.142:3300/0 5 ==== fsmap(e 47) ==== 8346+0+0 (secure 0 0 0) 0x7fbae4084b30 con 0x55bf838e91f0 2024-03-31T23:30:33.502 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:33.494+0000 7fbaf8249740 10 client.27500 fetch_fsmap finished waiting for FSMap version 47 2024-03-31T23:30:33.502 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:33.494+0000 7fbaf8249740 10 client.27500 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-31T23:30:33.503 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:33.494+0000 7fbaf8249740 20 client.27500 populate_metadata read hostname 'smithi046' 2024-03-31T23:30:33.503 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:33.494+0000 7fbaf8249740 1 --2- 172.21.15.46:0/2132269828 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55bf83901f70 0x55bf83904360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:30:33.503 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:33.494+0000 7fbaf8249740 4 client.27500 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-31T23:30:33.503 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:33.494+0000 7fbaf8249740 1 -- 172.21.15.46:0/2132269828 --> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55bf8385f120 con 0x55bf83901f70 2024-03-31T23:30:33.503 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:33.494+0000 7fbaeeffd640 1 --2- 172.21.15.46:0/2132269828 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55bf83901f70 0x55bf83904360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:30:33.503 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:33.495+0000 7fbad2ff5640 10 client.27500 ms_handle_connect on v2:172.21.15.142:6838/2519143918 2024-03-31T23:30:33.503 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:33.495+0000 7fbaeeffd640 1 --2- 172.21.15.46:0/2132269828 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55bf83901f70 0x55bf83904360 crc :-1 s=READY pgs=244 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b0d342c2b83d964a server_cookie=cbf1037c4876d786 in_seq=0 out_seq=0 2024-03-31T23:30:33.503 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:33.496+0000 7fbad2ff5640 1 -- 172.21.15.46:0/2132269828 <== mds.0 v2:172.21.15.142:6838/2519143918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55bf8385f120 con 0x55bf83901f70 2024-03-31T23:30:33.503 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:33.496+0000 7fbad2ff5640 10 client.27500 handle_command_reply: tid=0 2024-03-31T23:30:33.503 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:33.496+0000 7fbaf8249740 1 client.27500 shutdown 2024-03-31T23:30:33.503 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:33.496+0000 7fbaf8249740 1 -- 172.21.15.46:0/2132269828 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55bf838e91f0 msgr2=0x55bf838ef460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:30:33.503 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:33.496+0000 7fbaf8249740 1 --2- 172.21.15.46:0/2132269828 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55bf838e91f0 0x55bf838ef460 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fbae400cfd0 tx=0x7fbae409af50 comp rx=0 tx=0).stop 2024-03-31T23:30:33.503 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:33.498+0000 7fbaf8249740 1 -- 172.21.15.46:0/2132269828 shutdown_connections 2024-03-31T23:30:33.503 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:33.498+0000 7fbaf8249740 1 --2- 172.21.15.46:0/2132269828 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55bf83901f70 0x55bf83904360 crc :-1 s=READY pgs=244 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:30:33.504 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:33.498+0000 7fbad2ff5640 0 client.27500 ms_handle_reset on v2:172.21.15.142:6838/2519143918 2024-03-31T23:30:33.504 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:33.498+0000 7fbaf8249740 1 --2- 172.21.15.46:0/2132269828 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55bf838f1fc0 0x55bf838ef9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:30:33.504 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:33.498+0000 7fbaf8249740 1 --2- 172.21.15.46:0/2132269828 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55bf838e91f0 0x55bf838ef460 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:30:33.504 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:33.498+0000 7fbaf8249740 1 --2- 172.21.15.46:0/2132269828 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bf838e88d0 0x55bf838eef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:30:33.504 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:33.498+0000 7fbaf8249740 1 -- 172.21.15.46:0/2132269828 >> 172.21.15.46:0/2132269828 conn(0x55bf838e31f0 msgr2=0x55bf838e4f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:30:33.504 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:33.498+0000 7fbaf8249740 1 -- 172.21.15.46:0/2132269828 shutdown_connections 2024-03-31T23:30:33.504 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:33.498+0000 7fbaf8249740 1 -- 172.21.15.46:0/2132269828 wait complete. 2024-03-31T23:30:33.504 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:33.498+0000 7fbaf8249740 20 client.27500 trim_cache size 0 max 16384 2024-03-31T23:30:33.524 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:33.522+0000 7fbaf8249740 1 Processor -- start 2024-03-31T23:30:33.524 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:33.522+0000 7fbaf8249740 1 -- start start 2024-03-31T23:30:33.524 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:33.522+0000 7fbaf8249740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55bf838e88d0 0x55bf838fd930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:30:33.525 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:33.522+0000 7fbaf8249740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bf838e91f0 0x55bf838e35d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:30:33.525 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:33.522+0000 7fbaf8249740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55bf838f1fc0 0x55bf838ef7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:30:33.525 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:33.522+0000 7fbaf8249740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fbadc0128b0 con 0x55bf838f1fc0 2024-03-31T23:30:33.525 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:33.522+0000 7fbaf8249740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fbadc010180 con 0x55bf838e88d0 2024-03-31T23:30:33.525 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:33.522+0000 7fbaf8249740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fbadc00dff0 con 0x55bf838e91f0 2024-03-31T23:30:33.525 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:33.522+0000 7fbaef7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55bf838f1fc0 0x55bf838ef7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:30:33.525 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:33.523+0000 7fbaee7fc640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bf838e91f0 0x55bf838e35d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:30:33.525 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:33.523+0000 7fbaeeffd640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55bf838e88d0 0x55bf838fd930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:30:33.525 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:33.523+0000 7fbaef7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55bf838f1fc0 0x55bf838ef7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:42822/0 (socket says 172.21.15.46:42822) 2024-03-31T23:30:33.525 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:33.523+0000 7fbaef7fe640 1 -- 172.21.15.46:0/4040442040 learned_addr learned my addr 172.21.15.46:0/4040442040 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:30:33.525 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:33.523+0000 7fbaee7fc640 1 -- 172.21.15.46:0/4040442040 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55bf838e88d0 msgr2=0x55bf838fd930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:30:33.525 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:33.523+0000 7fbaee7fc640 1 --2- 172.21.15.46:0/4040442040 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55bf838e88d0 0x55bf838fd930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:30:33.525 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:33.523+0000 7fbaee7fc640 1 -- 172.21.15.46:0/4040442040 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55bf838f1fc0 msgr2=0x55bf838ef7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:30:33.526 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:33.523+0000 7fbaee7fc640 1 --2- 172.21.15.46:0/4040442040 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55bf838f1fc0 0x55bf838ef7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:30:33.526 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:33.523+0000 7fbaee7fc640 1 -- 172.21.15.46:0/4040442040 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bf838eded0 con 0x55bf838e91f0 2024-03-31T23:30:33.526 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:33.523+0000 7fbaef7fe640 1 --2- 172.21.15.46:0/4040442040 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55bf838f1fc0 0x55bf838ef7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:30:33.526 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:33.523+0000 7fbaee7fc640 1 --2- 172.21.15.46:0/4040442040 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bf838e91f0 0x55bf838e35d0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fbae409b2f0 tx=0x7fbae409b770 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c5136959e689600 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:30:33.526 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:33.523+0000 7fbad4ff9640 1 -- 172.21.15.46:0/4040442040 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbae40cca40 con 0x55bf838e91f0 2024-03-31T23:30:33.526 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:33.524+0000 7fbad4ff9640 1 -- 172.21.15.46:0/4040442040 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fbae409ad40 con 0x55bf838e91f0 2024-03-31T23:30:33.526 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:33.524+0000 7fbad4ff9640 1 -- 172.21.15.46:0/4040442040 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbae40b4360 con 0x55bf838e91f0 2024-03-31T23:30:33.529 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:33.525+0000 7fbaf8249740 1 -- 172.21.15.46:0/4040442040 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bf838e91f0 msgr2=0x55bf838e35d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:30:33.529 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:33.525+0000 7fbaf8249740 1 --2- 172.21.15.46:0/4040442040 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bf838e91f0 0x55bf838e35d0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fbae409b2f0 tx=0x7fbae409b770 comp rx=0 tx=0).stop 2024-03-31T23:30:33.529 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:33.525+0000 7fbaf8249740 1 -- 172.21.15.46:0/4040442040 shutdown_connections 2024-03-31T23:30:33.529 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:33.525+0000 7fbaf8249740 1 --2- 172.21.15.46:0/4040442040 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55bf838f1fc0 0x55bf838ef7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:30:33.529 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:33.525+0000 7fbaf8249740 1 --2- 172.21.15.46:0/4040442040 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bf838e91f0 0x55bf838e35d0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:30:33.529 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:33.525+0000 7fbaf8249740 1 --2- 172.21.15.46:0/4040442040 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55bf838e88d0 0x55bf838fd930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:30:33.529 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:33.525+0000 7fbaf8249740 1 -- 172.21.15.46:0/4040442040 >> 172.21.15.46:0/4040442040 conn(0x55bf838e31f0 msgr2=0x55bf838e2190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:30:33.529 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:33.525+0000 7fbaf8249740 1 -- 172.21.15.46:0/4040442040 shutdown_connections 2024-03-31T23:30:33.529 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:33.525+0000 7fbaf8249740 1 -- 172.21.15.46:0/4040442040 wait complete. 2024-03-31T23:30:33.529 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:33.525+0000 7fbaf8249740 1 -- start start 2024-03-31T23:30:33.529 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:33.526+0000 7fbaf8249740 1 Processor -- start 2024-03-31T23:30:33.530 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:33.526+0000 7fbaf8249740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bf838e88d0 0x55bf838f58e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:30:33.530 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:33.526+0000 7fbaf8249740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55bf838e91f0 0x55bf838f5e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:30:33.530 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:33.526+0000 7fbaeeffd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bf838e88d0 0x55bf838f58e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:30:33.530 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:33.526+0000 7fbaeeffd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bf838e88d0 0x55bf838f58e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.46:49356/0 (socket says 172.21.15.46:49356) 2024-03-31T23:30:33.530 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:33.526+0000 7fbaeeffd640 1 -- 172.21.15.46:0/1534054334 learned_addr learned my addr 172.21.15.46:0/1534054334 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:30:33.530 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:33.526+0000 7fbaf8249740 1 --2- 172.21.15.46:0/1534054334 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55bf838f1fc0 0x55bf838f6360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:30:33.530 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:33.526+0000 7fbaee7fc640 1 --2- 172.21.15.46:0/1534054334 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55bf838e91f0 0x55bf838f5e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:30:33.530 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:33.526+0000 7fbaf8249740 1 -- 172.21.15.46:0/1534054334 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fbadc00e640 con 0x55bf838e91f0 2024-03-31T23:30:33.530 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:33.526+0000 7fbaf8249740 1 -- 172.21.15.46:0/1534054334 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fbadc00fab0 con 0x55bf838f1fc0 2024-03-31T23:30:33.530 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:33.526+0000 7fbaf8249740 1 -- 172.21.15.46:0/1534054334 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fbadc00ed10 con 0x55bf838e88d0 2024-03-31T23:30:33.530 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:33.526+0000 7fbaef7fe640 1 --2- 172.21.15.46:0/1534054334 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55bf838f1fc0 0x55bf838f6360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:30:33.530 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:33.527+0000 7fbaeeffd640 1 -- 172.21.15.46:0/1534054334 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55bf838f1fc0 msgr2=0x55bf838f6360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:30:33.531 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:33.527+0000 7fbaeeffd640 1 --2- 172.21.15.46:0/1534054334 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55bf838f1fc0 0x55bf838f6360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:30:33.531 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:33.527+0000 7fbaeeffd640 1 -- 172.21.15.46:0/1534054334 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55bf838e91f0 msgr2=0x55bf838f5e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:30:33.531 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:33.527+0000 7fbaeeffd640 1 --2- 172.21.15.46:0/1534054334 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55bf838e91f0 0x55bf838f5e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:30:33.531 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:33.527+0000 7fbaeeffd640 1 -- 172.21.15.46:0/1534054334 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bf838e9020 con 0x55bf838e88d0 2024-03-31T23:30:33.531 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:33.527+0000 7fbaf8249740 10 client.0 _ll_register_callbacks cb 0x7fbae400e940 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T23:30:33.531 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:33.527+0000 7fbaeeffd640 1 --2- 172.21.15.46:0/1534054334 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bf838e88d0 0x55bf838f58e0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fbae0000c00 tx=0x7fbae002c710 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1a164eabf4b51e21 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:30:33.531 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:33.527+0000 7fbaf8249740 1 -- 172.21.15.46:0/1534054334 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbae409ad40 con 0x55bf838e88d0 2024-03-31T23:30:33.531 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:33.527+0000 7fbad2ff5640 10 client.27400 ms_handle_connect on v2:172.21.15.162:3300/0 2024-03-31T23:30:33.531 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:33.527+0000 7fbad2ff5640 1 -- 172.21.15.46:0/1534054334 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbae0050040 con 0x55bf838e88d0 2024-03-31T23:30:33.531 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:33.528+0000 7fbad2ff5640 1 -- 172.21.15.46:0/1534054334 <== mon.1 v2:172.21.15.162:3300/0 2 ==== osd_map(124..124 src has 1..124) ==== 9554+0+0 (secure 0 0 0) 0x7fbae002c7e0 con 0x55bf838e88d0 2024-03-31T23:30:33.531 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:33.528+0000 7fbad2ff5640 1 -- 172.21.15.46:0/1534054334 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) ==== 24+0+0 (secure 0 0 0) 0x7fbae0058be0 con 0x55bf838e88d0 2024-03-31T23:30:33.531 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:33.528+0000 7fbaf8249740 10 client.27400 fetch_fsmap learned FSMap version 47 2024-03-31T23:30:33.531 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:33.528+0000 7fbaf8249740 1 -- 172.21.15.46:0/1534054334 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({fsmap=47}) -- 0x55bf83968310 con 0x55bf838e88d0 2024-03-31T23:30:33.533 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:33.528+0000 7fbad2ff5640 1 -- 172.21.15.46:0/1534054334 <== mon.1 v2:172.21.15.162:3300/0 4 ==== fsmap(e 47) ==== 8346+0+0 (secure 0 0 0) 0x7fbae00409d0 con 0x55bf838e88d0 2024-03-31T23:30:33.533 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:33.528+0000 7fbaf8249740 10 client.27400 fetch_fsmap finished waiting for FSMap version 47 2024-03-31T23:30:33.533 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:33.528+0000 7fbaf8249740 10 client.27400 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-31T23:30:33.533 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:33.528+0000 7fbaf8249740 20 client.27400 populate_metadata read hostname 'smithi046' 2024-03-31T23:30:33.533 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:33.528+0000 7fbaf8249740 1 --2- 172.21.15.46:0/1534054334 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55bf83901f70 0x55bf8396a570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:30:33.533 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:33.528+0000 7fbaf8249740 4 client.27400 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-31T23:30:33.533 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:33.528+0000 7fbaf8249740 1 -- 172.21.15.46:0/1534054334 --> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55bf83858e30 con 0x55bf83901f70 2024-03-31T23:30:33.534 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:33.528+0000 7fbaee7fc640 1 --2- 172.21.15.46:0/1534054334 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55bf83901f70 0x55bf8396a570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:30:33.534 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:33.529+0000 7fbaee7fc640 1 --2- 172.21.15.46:0/1534054334 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55bf83901f70 0x55bf8396a570 crc :-1 s=READY pgs=245 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ba24cdf7240bfc2c server_cookie=c8f70ced030a6223 in_seq=0 out_seq=0 2024-03-31T23:30:33.534 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:33.529+0000 7fbad2ff5640 10 client.27400 ms_handle_connect on v2:172.21.15.142:6838/2519143918 2024-03-31T23:30:33.743 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:33.741+0000 7fbad2ff5640 1 -- 172.21.15.46:0/1534054334 <== mds.0 v2:172.21.15.142:6838/2519143918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55bf83858e30 con 0x55bf83901f70 2024-03-31T23:30:33.743 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:33.741+0000 7fbad2ff5640 10 client.27400 handle_command_reply: tid=0 2024-03-31T23:30:33.743 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:33.742+0000 7fbaf8249740 1 client.27400 shutdown 2024-03-31T23:30:33.744 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:33.743+0000 7fbaf8249740 1 -- 172.21.15.46:0/1534054334 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bf838e88d0 msgr2=0x55bf838f58e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:30:33.744 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:33.743+0000 7fbaf8249740 1 --2- 172.21.15.46:0/1534054334 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bf838e88d0 0x55bf838f58e0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fbae0000c00 tx=0x7fbae002c710 comp rx=0 tx=0).stop 2024-03-31T23:30:33.744 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:33.743+0000 7fbaf8249740 1 -- 172.21.15.46:0/1534054334 shutdown_connections 2024-03-31T23:30:33.744 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:33.743+0000 7fbaf8249740 1 --2- 172.21.15.46:0/1534054334 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55bf83901f70 0x55bf8396a570 crc :-1 s=READY pgs=245 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:30:33.744 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:33.743+0000 7fbaf8249740 1 --2- 172.21.15.46:0/1534054334 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55bf838f1fc0 0x55bf838f6360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:30:33.744 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:33.743+0000 7fbaf8249740 1 --2- 172.21.15.46:0/1534054334 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55bf838e91f0 0x55bf838f5e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:30:33.744 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:33.743+0000 7fbaf8249740 1 --2- 172.21.15.46:0/1534054334 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bf838e88d0 0x55bf838f58e0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:30:33.744 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:33.743+0000 7fbaf8249740 1 -- 172.21.15.46:0/1534054334 >> 172.21.15.46:0/1534054334 conn(0x55bf838e31f0 msgr2=0x55bf838e2190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:30:33.745 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:33.744+0000 7fbad2ff5640 0 client.27400 ms_handle_reset on v2:172.21.15.142:6838/2519143918 2024-03-31T23:30:33.745 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:33.744+0000 7fbaf8249740 1 -- 172.21.15.46:0/1534054334 shutdown_connections 2024-03-31T23:30:33.745 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:33.744+0000 7fbaf8249740 1 -- 172.21.15.46:0/1534054334 wait complete. 2024-03-31T23:30:33.745 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:33.744+0000 7fbaf8249740 20 client.27400 trim_cache size 0 max 16384 2024-03-31T23:30:33.746 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout:{ 2024-03-31T23:30:33.746 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout: "message": "", 2024-03-31T23:30:33.746 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout: "return_code": 0 2024-03-31T23:30:33.746 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout:} 2024-03-31T23:30:33.746 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:33.745+0000 7fbaf566b640 1 -- 172.21.15.46:0/3161150558 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fbad80a2690 msgr2=0x7fbad80a4b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:30:33.746 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:33.745+0000 7fbaf566b640 1 --2- 172.21.15.46:0/3161150558 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fbad80a2690 0x7fbad80a4b50 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fbaf0067000 tx=0x7fbaf0067690 comp rx=0 tx=0).stop 2024-03-31T23:30:33.747 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:33.745+0000 7fbaf566b640 1 -- 172.21.15.46:0/3161150558 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbae8094d50 msgr2=0x7fbae8134920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:30:33.747 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:33.745+0000 7fbaf566b640 1 --2- 172.21.15.46:0/3161150558 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbae8094d50 0x7fbae8134920 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 crypto rx=0x7fbae400e9f0 tx=0x7fbae400eeb0 comp rx=0 tx=0).stop 2024-03-31T23:30:33.747 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:33.746+0000 7fbaf566b640 1 -- 172.21.15.46:0/3161150558 shutdown_connections 2024-03-31T23:30:33.747 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:33.746+0000 7fbaf566b640 1 --2- 172.21.15.46:0/3161150558 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fbad80a2690 0x7fbad80a4b50 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:30:33.747 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:33.746+0000 7fbaf566b640 1 --2- 172.21.15.46:0/3161150558 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbae80a03a0 0x7fbae812e590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:30:33.747 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:33.746+0000 7fbaf566b640 1 --2- 172.21.15.46:0/3161150558 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbae8094d50 0x7fbae8134920 unknown :-1 s=CLOSED pgs=960 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:30:33.748 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:33.746+0000 7fbaf566b640 1 --2- 172.21.15.46:0/3161150558 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbae8094430 0x7fbae81343e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:30:33.748 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:33.746+0000 7fbaf566b640 1 -- 172.21.15.46:0/3161150558 >> 172.21.15.46:0/3161150558 conn(0x7fbae8009ab0 msgr2=0x7fbae8096d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:30:33.748 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:33.746+0000 7fbaf566b640 1 -- 172.21.15.46:0/3161150558 shutdown_connections 2024-03-31T23:30:33.748 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:33.746+0000 7fbaf566b640 1 -- 172.21.15.46:0/3161150558 wait complete. 2024-03-31T23:30:34.291 INFO:tasks.workunit.client.0.smithi046.stdout: 10 54284 8.55 MB/sec execute 219 sec latency 183.969 ms 2024-03-31T23:30:34.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:30:34 smithi142 ceph-mon[35271]: pgmap v596: 129 pgs: 129 active+clean; 861 MiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 4.6 MiB/s wr, 260 op/s 2024-03-31T23:30:35.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:30:34 smithi162 ceph-mon[36237]: pgmap v596: 129 pgs: 129 active+clean; 861 MiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 4.6 MiB/s wr, 260 op/s 2024-03-31T23:30:35.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:30:34 smithi046 ceph-mon[32498]: pgmap v596: 129 pgs: 129 active+clean; 861 MiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 4.6 MiB/s wr, 260 op/s 2024-03-31T23:30:35.291 INFO:tasks.workunit.client.0.smithi046.stdout: 10 54411 8.52 MB/sec execute 220 sec latency 132.799 ms 2024-03-31T23:30:36.291 INFO:tasks.workunit.client.0.smithi046.stdout: 10 54566 8.53 MB/sec execute 221 sec latency 86.812 ms 2024-03-31T23:30:36.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:30:36 smithi142 ceph-mon[35271]: pgmap v597: 129 pgs: 129 active+clean; 868 MiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 7.0 MiB/s wr, 422 op/s 2024-03-31T23:30:37.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:30:36 smithi162 ceph-mon[36237]: pgmap v597: 129 pgs: 129 active+clean; 868 MiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 7.0 MiB/s wr, 422 op/s 2024-03-31T23:30:37.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:30:36 smithi046 ceph-mon[32498]: pgmap v597: 129 pgs: 129 active+clean; 868 MiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 7.0 MiB/s wr, 422 op/s 2024-03-31T23:30:37.291 INFO:tasks.workunit.client.0.smithi046.stdout: 10 54722 8.53 MB/sec execute 222 sec latency 71.869 ms 2024-03-31T23:30:38.291 INFO:tasks.workunit.client.0.smithi046.stdout: 10 54862 8.50 MB/sec execute 223 sec latency 97.663 ms 2024-03-31T23:30:38.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:30:38 smithi142 ceph-mon[35271]: pgmap v598: 129 pgs: 129 active+clean; 868 MiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 4.9 MiB/s wr, 267 op/s 2024-03-31T23:30:39.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:30:38 smithi162 ceph-mon[36237]: pgmap v598: 129 pgs: 129 active+clean; 868 MiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 4.9 MiB/s wr, 267 op/s 2024-03-31T23:30:39.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:30:38 smithi046 ceph-mon[32498]: pgmap v598: 129 pgs: 129 active+clean; 868 MiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 4.9 MiB/s wr, 267 op/s 2024-03-31T23:30:39.291 INFO:tasks.workunit.client.0.smithi046.stdout: 10 55018 8.50 MB/sec execute 224 sec latency 100.736 ms 2024-03-31T23:30:40.292 INFO:tasks.workunit.client.0.smithi046.stdout: 10 55185 8.49 MB/sec execute 225 sec latency 84.500 ms 2024-03-31T23:30:40.913 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:30:40 smithi046 ceph-mon[32498]: pgmap v599: 129 pgs: 129 active+clean; 871 MiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 6.9 MiB/s wr, 397 op/s 2024-03-31T23:30:40.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:30:40 smithi142 ceph-mon[35271]: pgmap v599: 129 pgs: 129 active+clean; 871 MiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 6.9 MiB/s wr, 397 op/s 2024-03-31T23:30:41.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:30:40 smithi162 ceph-mon[36237]: pgmap v599: 129 pgs: 129 active+clean; 871 MiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 6.9 MiB/s wr, 397 op/s 2024-03-31T23:30:41.292 INFO:tasks.workunit.client.0.smithi046.stdout: 10 55330 8.47 MB/sec execute 226 sec latency 98.875 ms 2024-03-31T23:30:42.292 INFO:tasks.workunit.client.0.smithi046.stdout: 10 55472 8.46 MB/sec execute 227 sec latency 131.355 ms 2024-03-31T23:30:42.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:30:42 smithi142 ceph-mon[35271]: pgmap v600: 129 pgs: 129 active+clean; 871 MiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 4.4 MiB/s wr, 292 op/s 2024-03-31T23:30:43.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:30:42 smithi162 ceph-mon[36237]: pgmap v600: 129 pgs: 129 active+clean; 871 MiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 4.4 MiB/s wr, 292 op/s 2024-03-31T23:30:43.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:30:42 smithi046 ceph-mon[32498]: pgmap v600: 129 pgs: 129 active+clean; 871 MiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 4.4 MiB/s wr, 292 op/s 2024-03-31T23:30:43.292 INFO:tasks.workunit.client.0.smithi046.stdout: 10 55630 8.48 MB/sec execute 228 sec latency 158.850 ms 2024-03-31T23:30:44.292 INFO:tasks.workunit.client.0.smithi046.stdout: 10 55788 8.47 MB/sec execute 229 sec latency 80.603 ms 2024-03-31T23:30:44.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:30:44 smithi142 ceph-mon[35271]: pgmap v601: 129 pgs: 129 active+clean; 871 MiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 4.4 MiB/s wr, 293 op/s 2024-03-31T23:30:45.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:30:44 smithi162 ceph-mon[36237]: pgmap v601: 129 pgs: 129 active+clean; 871 MiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 4.4 MiB/s wr, 293 op/s 2024-03-31T23:30:45.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:30:44 smithi046 ceph-mon[32498]: pgmap v601: 129 pgs: 129 active+clean; 871 MiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 4.4 MiB/s wr, 293 op/s 2024-03-31T23:30:45.292 INFO:tasks.workunit.client.0.smithi046.stdout: 10 55977 8.49 MB/sec execute 230 sec latency 93.935 ms 2024-03-31T23:30:46.296 INFO:tasks.workunit.client.0.smithi046.stdout: 10 56164 8.50 MB/sec execute 231 sec latency 73.454 ms 2024-03-31T23:30:46.884 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:46.881+0000 7f834358c640 1 -- 172.21.15.46:0/1268758233 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f833c100610 msgr2=0x7f833c10e5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:30:46.885 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:46.881+0000 7f834358c640 1 --2- 172.21.15.46:0/1268758233 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f833c100610 0x7f833c10e5f0 secure :-1 s=READY pgs=961 cs=0 l=1 rev1=1 crypto rx=0x7f8334009cd0 tx=0x7f833402c0f0 comp rx=0 tx=0).stop 2024-03-31T23:30:46.885 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:46.881+0000 7f834358c640 1 -- 172.21.15.46:0/1268758233 shutdown_connections 2024-03-31T23:30:46.885 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:46.881+0000 7f834358c640 1 --2- 172.21.15.46:0/1268758233 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f833c100610 0x7f833c10e5f0 unknown :-1 s=CLOSED pgs=961 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:30:46.885 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:46.881+0000 7f834358c640 1 --2- 172.21.15.46:0/1268758233 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f833c0ffc50 0x7f833c1000d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:30:46.885 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:46.881+0000 7f834358c640 1 --2- 172.21.15.46:0/1268758233 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f833c107730 0x7f833c107b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:30:46.885 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:46.881+0000 7f834358c640 1 -- 172.21.15.46:0/1268758233 >> 172.21.15.46:0/1268758233 conn(0x7f833c076c10 msgr2=0x7f833c0fdf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:30:46.885 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:46.881+0000 7f834358c640 1 -- 172.21.15.46:0/1268758233 shutdown_connections 2024-03-31T23:30:46.885 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:46.881+0000 7f834358c640 1 -- 172.21.15.46:0/1268758233 wait complete. 2024-03-31T23:30:46.885 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:46.881+0000 7f834358c640 1 Processor -- start 2024-03-31T23:30:46.885 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:46.882+0000 7f834358c640 1 -- start start 2024-03-31T23:30:46.885 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:46.882+0000 7f834358c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f833c0ffc50 0x7f833c197e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:30:46.886 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:46.882+0000 7f834358c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f833c100610 0x7f833c198350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:30:46.886 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:46.882+0000 7f834358c640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f833c107730 0x7f833c19c6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:30:46.886 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:46.882+0000 7f834358c640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f833c1119f0 con 0x7f833c0ffc50 2024-03-31T23:30:46.886 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:46.882+0000 7f834358c640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f833c111870 con 0x7f833c107730 2024-03-31T23:30:46.886 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:46.882+0000 7f834358c640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f833c111b70 con 0x7f833c100610 2024-03-31T23:30:46.886 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:46.882+0000 7f8341328640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f833c0ffc50 0x7f833c197e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:30:46.886 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:46.882+0000 7f8341328640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f833c0ffc50 0x7f833c197e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:50318/0 (socket says 172.21.15.46:50318) 2024-03-31T23:30:46.886 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:46.882+0000 7f8341328640 1 -- 172.21.15.46:0/2168925525 learned_addr learned my addr 172.21.15.46:0/2168925525 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:30:46.886 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:46.882+0000 7f8340b27640 1 --2- 172.21.15.46:0/2168925525 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f833c100610 0x7f833c198350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:30:46.886 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:46.882+0000 7f8341b29640 1 --2- 172.21.15.46:0/2168925525 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f833c107730 0x7f833c19c6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:30:46.886 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:46.882+0000 7f8341328640 1 -- 172.21.15.46:0/2168925525 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f833c100610 msgr2=0x7f833c198350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:30:46.886 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:46.882+0000 7f8341328640 1 --2- 172.21.15.46:0/2168925525 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f833c100610 0x7f833c198350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:30:46.886 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:46.882+0000 7f8341328640 1 -- 172.21.15.46:0/2168925525 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f833c107730 msgr2=0x7f833c19c6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:30:46.887 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:46.882+0000 7f8341328640 1 --2- 172.21.15.46:0/2168925525 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f833c107730 0x7f833c19c6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:30:46.887 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:46.882+0000 7f8341328640 1 -- 172.21.15.46:0/2168925525 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f833c19ce00 con 0x7f833c0ffc50 2024-03-31T23:30:46.887 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:46.882+0000 7f8340b27640 1 --2- 172.21.15.46:0/2168925525 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f833c100610 0x7f833c198350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:30:46.887 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:46.882+0000 7f8341b29640 1 --2- 172.21.15.46:0/2168925525 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f833c107730 0x7f833c19c6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:30:46.887 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:46.883+0000 7f8341328640 1 --2- 172.21.15.46:0/2168925525 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f833c0ffc50 0x7f833c197e10 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 crypto rx=0x7f833800ea60 tx=0x7f833800ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:30:46.887 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:46.883+0000 7f83327fc640 1 -- 172.21.15.46:0/2168925525 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f833800cde0 con 0x7f833c0ffc50 2024-03-31T23:30:46.887 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:46.883+0000 7f83327fc640 1 -- 172.21.15.46:0/2168925525 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f83380035f0 con 0x7f833c0ffc50 2024-03-31T23:30:46.887 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:46.883+0000 7f83327fc640 1 -- 172.21.15.46:0/2168925525 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8338010690 con 0x7f833c0ffc50 2024-03-31T23:30:46.887 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:46.883+0000 7f834358c640 1 -- 172.21.15.46:0/2168925525 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f833c19d0f0 con 0x7f833c0ffc50 2024-03-31T23:30:46.887 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:46.883+0000 7f834358c640 1 -- 172.21.15.46:0/2168925525 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f833c1a4980 con 0x7f833c0ffc50 2024-03-31T23:30:46.892 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:46.886+0000 7f83327fc640 1 -- 172.21.15.46:0/2168925525 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 25) ==== 137451+0+0 (secure 0 0 0) 0x7f83380037a0 con 0x7f833c0ffc50 2024-03-31T23:30:46.892 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:46.886+0000 7f8346173740 1 Processor -- start 2024-03-31T23:30:46.892 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:46.886+0000 7f8346173740 1 -- start start 2024-03-31T23:30:46.892 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:46.887+0000 7f83327fc640 1 --2- 172.21.15.46:0/2168925525 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f83280a2730 0x7f83280a4bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:30:46.892 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:46.887+0000 7f8346173740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a262ce7110 0x55a262ce74f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:30:46.892 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:46.887+0000 7f8346173740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a262ce7ac0 0x55a262ce7f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:30:46.893 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:46.887+0000 7f8346173740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a262cf9450 0x55a262cfb840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:30:46.893 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:46.887+0000 7f8346173740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55a262c4a8f0 con 0x55a262ce7110 2024-03-31T23:30:46.893 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:46.887+0000 7f8346173740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55a262bf38d0 con 0x55a262ce7ac0 2024-03-31T23:30:46.893 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:46.887+0000 7f8346173740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55a262c46990 con 0x55a262cf9450 2024-03-31T23:30:46.893 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:46.887+0000 7f83327fc640 1 -- 172.21.15.46:0/2168925525 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(124..124 src has 1..124) ==== 9554+0+0 (secure 0 0 0) 0x7f83380cf0d0 con 0x7f833c0ffc50 2024-03-31T23:30:46.893 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:46.887+0000 7f8341328640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a262ce7110 0x55a262ce74f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:30:46.893 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:46.887+0000 7f8341328640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a262ce7110 0x55a262ce74f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:50332/0 (socket says 172.21.15.46:50332) 2024-03-31T23:30:46.893 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:46.887+0000 7f8341328640 1 -- 172.21.15.46:0/3527314486 learned_addr learned my addr 172.21.15.46:0/3527314486 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:30:46.893 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:46.887+0000 7f8340b27640 1 --2- 172.21.15.46:0/3527314486 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a262ce7ac0 0x55a262ce7f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:30:46.893 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:46.887+0000 7f8341b29640 1 --2- 172.21.15.46:0/2168925525 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f83280a2730 0x7f83280a4bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:30:46.893 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:46.887+0000 7f8341328640 1 -- 172.21.15.46:0/3527314486 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a262cf9450 msgr2=0x55a262cfb840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:30:46.893 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:46.887+0000 7f8341328640 1 --2- 172.21.15.46:0/3527314486 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a262cf9450 0x55a262cfb840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:30:46.893 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:46.887+0000 7f8341328640 1 -- 172.21.15.46:0/3527314486 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a262ce7ac0 msgr2=0x55a262ce7f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:30:46.894 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:46.887+0000 7f8341328640 1 --2- 172.21.15.46:0/3527314486 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a262ce7ac0 0x55a262ce7f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:30:46.894 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:46.887+0000 7f8341328640 1 -- 172.21.15.46:0/3527314486 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a26295f210 con 0x55a262ce7110 2024-03-31T23:30:46.894 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:46.887+0000 7f8340b27640 1 --2- 172.21.15.46:0/3527314486 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a262ce7ac0 0x55a262ce7f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:30:46.894 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:46.888+0000 7f8341328640 1 --2- 172.21.15.46:0/3527314486 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a262ce7110 0x55a262ce74f0 secure :-1 s=READY pgs=963 cs=0 l=1 rev1=1 crypto rx=0x7f83380978a0 tx=0x7f8338097810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b07af1a56d344c8e server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:30:46.894 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:46.888+0000 7f8341b29640 1 --2- 172.21.15.46:0/2168925525 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f83280a2730 0x7f83280a4bf0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f8334005cc0 tx=0x7f83340099a0 comp rx=0 tx=0).ready entity=mgr.15654 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:30:46.894 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:46.888+0000 7f83277fe640 1 -- 172.21.15.46:0/3527314486 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8338051020 con 0x55a262ce7110 2024-03-31T23:30:46.894 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:46.888+0000 7f83277fe640 1 -- 172.21.15.46:0/3527314486 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8338003b30 con 0x55a262ce7110 2024-03-31T23:30:46.894 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:46.888+0000 7f83277fe640 1 -- 172.21.15.46:0/3527314486 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8338097ab0 con 0x55a262ce7110 2024-03-31T23:30:46.894 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:46.889+0000 7f8346173740 1 -- 172.21.15.46:0/3527314486 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a262ce7110 msgr2=0x55a262ce74f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:30:46.894 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:46.889+0000 7f8346173740 1 --2- 172.21.15.46:0/3527314486 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a262ce7110 0x55a262ce74f0 secure :-1 s=READY pgs=963 cs=0 l=1 rev1=1 crypto rx=0x7f83380978a0 tx=0x7f8338097810 comp rx=0 tx=0).stop 2024-03-31T23:30:46.894 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:46.889+0000 7f8346173740 1 -- 172.21.15.46:0/3527314486 shutdown_connections 2024-03-31T23:30:46.894 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:46.889+0000 7f8346173740 1 --2- 172.21.15.46:0/3527314486 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a262cf9450 0x55a262cfb840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:30:46.894 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:46.889+0000 7f8346173740 1 --2- 172.21.15.46:0/3527314486 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a262ce7ac0 0x55a262ce7f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:30:46.895 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:46.889+0000 7f8346173740 1 --2- 172.21.15.46:0/3527314486 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a262ce7110 0x55a262ce74f0 unknown :-1 s=CLOSED pgs=963 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:30:46.895 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:46.889+0000 7f8346173740 1 -- 172.21.15.46:0/3527314486 >> 172.21.15.46:0/3527314486 conn(0x55a262ce1fe0 msgr2=0x55a262ce23c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:30:46.895 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:46.889+0000 7f8346173740 1 -- 172.21.15.46:0/3527314486 shutdown_connections 2024-03-31T23:30:46.895 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:46.889+0000 7f8346173740 1 -- 172.21.15.46:0/3527314486 wait complete. 2024-03-31T23:30:46.895 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:46.889+0000 7f8346173740 1 -- start start 2024-03-31T23:30:46.895 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:46.889+0000 7f8346173740 1 Processor -- start 2024-03-31T23:30:46.895 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:46.889+0000 7f8346173740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a262ce7110 0x55a262ce9860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:30:46.895 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:46.889+0000 7f8346173740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a262ce7ac0 0x55a262ce9da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:30:46.895 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:46.889+0000 7f8346173740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a262cf9450 0x55a262cea2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:30:46.895 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:46.890+0000 7f8341328640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a262ce7110 0x55a262ce9860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:30:46.895 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:46.890+0000 7f8346173740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55a262cfc120 con 0x55a262cf9450 2024-03-31T23:30:46.895 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:46.890+0000 7f8346173740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55a262be8f80 con 0x55a262ce7110 2024-03-31T23:30:46.895 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:46.890+0000 7f8346173740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55a262cfc2a0 con 0x55a262ce7ac0 2024-03-31T23:30:46.895 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:46.890+0000 7f8340b27640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a262ce7ac0 0x55a262ce9da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:30:46.896 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:46.890+0000 7f8341b29640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a262cf9450 0x55a262cea2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:30:46.896 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:46.890+0000 7f8341328640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a262ce7110 0x55a262ce9860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.46:39560/0 (socket says 172.21.15.46:39560) 2024-03-31T23:30:46.896 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:46.890+0000 7f8346173740 10 client.0 _ll_register_callbacks cb 0x55a262c41290 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T23:30:46.896 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:46.890+0000 7f8341328640 1 -- 172.21.15.46:0/833225299 learned_addr learned my addr 172.21.15.46:0/833225299 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:30:46.896 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:46.890+0000 7f8340b27640 1 -- 172.21.15.46:0/833225299 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a262ce7110 msgr2=0x55a262ce9860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:30:46.896 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:46.890+0000 7f8340b27640 1 --2- 172.21.15.46:0/833225299 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a262ce7110 0x55a262ce9860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:30:46.896 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:46.890+0000 7f8340b27640 1 -- 172.21.15.46:0/833225299 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a262cf9450 msgr2=0x55a262cea2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:30:46.896 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:46.890+0000 7f8340b27640 1 --2- 172.21.15.46:0/833225299 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a262cf9450 0x55a262cea2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:30:46.896 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:46.890+0000 7f8340b27640 1 -- 172.21.15.46:0/833225299 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a262a2b620 con 0x55a262ce7ac0 2024-03-31T23:30:46.896 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:46.890+0000 7f8341b29640 1 --2- 172.21.15.46:0/833225299 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a262cf9450 0x55a262cea2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:30:46.896 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:46.890+0000 7f8340b27640 1 --2- 172.21.15.46:0/833225299 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a262ce7ac0 0x55a262ce9da0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f833c106440 tx=0x7f832c00f790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=97c57a7eecc6ccef server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:30:46.896 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:46.890+0000 7f8324ff9640 10 client.0 ms_handle_connect on v2:172.21.15.162:3300/0 2024-03-31T23:30:46.896 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:46.890+0000 7f8324ff9640 1 -- 172.21.15.46:0/833225299 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f832c0120d0 con 0x55a262ce7ac0 2024-03-31T23:30:46.896 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:46.891+0000 7f8346173740 1 -- 172.21.15.46:0/833225299 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55a262c5e120 con 0x55a262ce7ac0 2024-03-31T23:30:46.897 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:46.891+0000 7f8324ff9640 1 -- 172.21.15.46:0/833225299 <== mon.1 v2:172.21.15.162:3300/0 2 ==== osd_map(124..124 src has 1..124) ==== 9554+0+0 (secure 0 0 0) 0x7f832c004020 con 0x55a262ce7ac0 2024-03-31T23:30:46.897 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:46.891+0000 7f8324ff9640 1 -- 172.21.15.46:0/833225299 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) ==== 24+0+0 (secure 0 0 0) 0x7f832c012270 con 0x55a262ce7ac0 2024-03-31T23:30:46.897 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:46.891+0000 7f8346173740 10 client.27409 fetch_fsmap learned FSMap version 47 2024-03-31T23:30:46.897 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:46.891+0000 7f8346173740 1 -- 172.21.15.46:0/833225299 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) -- 0x55a262d08060 con 0x55a262ce7ac0 2024-03-31T23:30:46.897 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:46.891+0000 7f8324ff9640 1 -- 172.21.15.46:0/833225299 <== mon.1 v2:172.21.15.162:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x55a262c5e120 con 0x55a262ce7ac0 2024-03-31T23:30:46.897 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:46.891+0000 7f8324ff9640 1 -- 172.21.15.46:0/833225299 <== mon.1 v2:172.21.15.162:3300/0 5 ==== fsmap(e 47) ==== 8346+0+0 (secure 0 0 0) 0x7f832c01c470 con 0x55a262ce7ac0 2024-03-31T23:30:46.897 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:46.891+0000 7f8346173740 10 client.27409 fetch_fsmap finished waiting for FSMap version 47 2024-03-31T23:30:46.897 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:46.891+0000 7f8346173740 10 client.27409 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-31T23:30:46.897 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:46.891+0000 7f8346173740 20 client.27409 populate_metadata read hostname 'smithi046' 2024-03-31T23:30:46.897 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:46.891+0000 7f8346173740 1 --2- 172.21.15.46:0/833225299 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55a262d09940 0x55a262d0bd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:30:46.897 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:46.891+0000 7f8346173740 4 client.27409 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-31T23:30:46.897 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:46.891+0000 7f8346173740 1 -- 172.21.15.46:0/833225299 --> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a262c57e30 con 0x55a262d09940 2024-03-31T23:30:46.897 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:46.891+0000 7f8341328640 1 --2- 172.21.15.46:0/833225299 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55a262d09940 0x55a262d0bd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:30:46.899 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:46.892+0000 7f8341328640 1 --2- 172.21.15.46:0/833225299 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55a262d09940 0x55a262d0bd80 crc :-1 s=READY pgs=246 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ec747baa74f7c377 server_cookie=9518de70d3068425 in_seq=0 out_seq=0 2024-03-31T23:30:46.899 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:46.892+0000 7f8324ff9640 10 client.27409 ms_handle_connect on v2:172.21.15.142:6838/2519143918 2024-03-31T23:30:46.899 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:46.893+0000 7f8324ff9640 1 -- 172.21.15.46:0/833225299 <== mds.0 v2:172.21.15.142:6838/2519143918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55a262c57e30 con 0x55a262d09940 2024-03-31T23:30:46.899 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:46.893+0000 7f8324ff9640 10 client.27409 handle_command_reply: tid=0 2024-03-31T23:30:46.899 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:46.893+0000 7f8346173740 1 client.27409 shutdown 2024-03-31T23:30:46.899 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:46.893+0000 7f8346173740 1 -- 172.21.15.46:0/833225299 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a262ce7ac0 msgr2=0x55a262ce9da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:30:46.899 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:46.893+0000 7f8346173740 1 --2- 172.21.15.46:0/833225299 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a262ce7ac0 0x55a262ce9da0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f833c106440 tx=0x7f832c00f790 comp rx=0 tx=0).stop 2024-03-31T23:30:46.899 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:46.893+0000 7f8346173740 1 -- 172.21.15.46:0/833225299 shutdown_connections 2024-03-31T23:30:46.899 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:46.893+0000 7f8346173740 1 --2- 172.21.15.46:0/833225299 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55a262d09940 0x55a262d0bd80 crc :-1 s=READY pgs=246 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:30:46.900 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:46.893+0000 7f8346173740 1 --2- 172.21.15.46:0/833225299 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a262cf9450 0x55a262cea2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:30:46.900 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:46.893+0000 7f8346173740 1 --2- 172.21.15.46:0/833225299 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a262ce7ac0 0x55a262ce9da0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:30:46.900 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:46.893+0000 7f8324ff9640 0 client.27409 ms_handle_reset on v2:172.21.15.142:6838/2519143918 2024-03-31T23:30:46.900 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:46.893+0000 7f8346173740 1 --2- 172.21.15.46:0/833225299 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a262ce7110 0x55a262ce9860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:30:46.900 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:46.893+0000 7f8346173740 1 -- 172.21.15.46:0/833225299 >> 172.21.15.46:0/833225299 conn(0x55a262ce1fe0 msgr2=0x55a262ce3d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:30:46.900 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:46.893+0000 7f8346173740 1 -- 172.21.15.46:0/833225299 shutdown_connections 2024-03-31T23:30:46.900 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:46.893+0000 7f8346173740 1 -- 172.21.15.46:0/833225299 wait complete. 2024-03-31T23:30:46.900 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:46.893+0000 7f8346173740 20 client.27409 trim_cache size 0 max 16384 2024-03-31T23:30:46.920 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:46.915+0000 7f8346173740 1 Processor -- start 2024-03-31T23:30:46.920 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:46.915+0000 7f8346173740 1 -- start start 2024-03-31T23:30:46.920 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:46.916+0000 7f8346173740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a262ce7110 0x55a262cebf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:30:46.921 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:46.916+0000 7f8346173740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a262ce7ac0 0x55a262cdccf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:30:46.921 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:46.916+0000 7f8346173740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a262cf9450 0x55a262d04f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:30:46.921 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:46.916+0000 7f8346173740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f833c1af3a0 con 0x55a262ce7ac0 2024-03-31T23:30:46.921 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:46.916+0000 7f8346173740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f833c1acc70 con 0x55a262ce7110 2024-03-31T23:30:46.921 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:46.916+0000 7f8346173740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f833c1aaae0 con 0x55a262cf9450 2024-03-31T23:30:46.921 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:46.916+0000 7f8340b27640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a262ce7ac0 0x55a262cdccf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:30:46.921 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:46.916+0000 7f8340b27640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a262ce7ac0 0x55a262cdccf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:50350/0 (socket says 172.21.15.46:50350) 2024-03-31T23:30:46.921 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:46.916+0000 7f8341b29640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a262cf9450 0x55a262d04f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:30:46.921 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:46.916+0000 7f8340b27640 1 -- 172.21.15.46:0/526416827 learned_addr learned my addr 172.21.15.46:0/526416827 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:30:46.921 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:46.916+0000 7f8341328640 1 --2- 172.21.15.46:0/526416827 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a262ce7110 0x55a262cebf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:30:46.921 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:46.916+0000 7f8341b29640 1 -- 172.21.15.46:0/526416827 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a262ce7110 msgr2=0x55a262cebf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:30:46.922 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:46.916+0000 7f8341b29640 1 --2- 172.21.15.46:0/526416827 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a262ce7110 0x55a262cebf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:30:46.922 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:46.916+0000 7f8341b29640 1 -- 172.21.15.46:0/526416827 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a262ce7ac0 msgr2=0x55a262cdccf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:30:46.922 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:46.916+0000 7f8341b29640 1 --2- 172.21.15.46:0/526416827 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a262ce7ac0 0x55a262cdccf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:30:46.922 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:46.916+0000 7f8341b29640 1 -- 172.21.15.46:0/526416827 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a262ce3210 con 0x55a262cf9450 2024-03-31T23:30:46.922 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:46.916+0000 7f8340b27640 1 --2- 172.21.15.46:0/526416827 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a262ce7ac0 0x55a262cdccf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:30:46.922 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:46.917+0000 7f8341b29640 1 --2- 172.21.15.46:0/526416827 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a262cf9450 0x55a262d04f00 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f833402c630 tx=0x7f833403e000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7144f83988f6e98d server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:30:46.922 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:46.917+0000 7f8326ffd640 1 -- 172.21.15.46:0/526416827 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8334050070 con 0x55a262cf9450 2024-03-31T23:30:46.922 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:46.917+0000 7f8326ffd640 1 -- 172.21.15.46:0/526416827 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f83340047b0 con 0x55a262cf9450 2024-03-31T23:30:46.922 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:46.917+0000 7f8326ffd640 1 -- 172.21.15.46:0/526416827 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8334044070 con 0x55a262cf9450 2024-03-31T23:30:46.922 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:46.918+0000 7f8346173740 1 -- 172.21.15.46:0/526416827 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a262cf9450 msgr2=0x55a262d04f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:30:46.922 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:46.918+0000 7f8346173740 1 --2- 172.21.15.46:0/526416827 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a262cf9450 0x55a262d04f00 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f833402c630 tx=0x7f833403e000 comp rx=0 tx=0).stop 2024-03-31T23:30:46.922 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:46.918+0000 7f8346173740 1 -- 172.21.15.46:0/526416827 shutdown_connections 2024-03-31T23:30:46.922 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:46.918+0000 7f8346173740 1 --2- 172.21.15.46:0/526416827 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a262cf9450 0x55a262d04f00 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:30:46.923 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:46.918+0000 7f8346173740 1 --2- 172.21.15.46:0/526416827 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a262ce7ac0 0x55a262cdccf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:30:46.923 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:46.918+0000 7f8346173740 1 --2- 172.21.15.46:0/526416827 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a262ce7110 0x55a262cebf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:30:46.923 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:46.918+0000 7f8346173740 1 -- 172.21.15.46:0/526416827 >> 172.21.15.46:0/526416827 conn(0x55a262ce1fe0 msgr2=0x55a262ce90f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:30:46.923 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:46.918+0000 7f8346173740 1 -- 172.21.15.46:0/526416827 shutdown_connections 2024-03-31T23:30:46.923 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:46.918+0000 7f8346173740 1 -- 172.21.15.46:0/526416827 wait complete. 2024-03-31T23:30:46.923 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:46.918+0000 7f8346173740 1 -- start start 2024-03-31T23:30:46.923 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:46.918+0000 7f8346173740 1 Processor -- start 2024-03-31T23:30:46.924 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:46.919+0000 7f8346173740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a262ce7110 0x55a262d01530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:30:46.924 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:46.919+0000 7f8346173740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a262ce7ac0 0x55a262cf6800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:30:46.924 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:46.919+0000 7f8346173740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a262cf9450 0x55a262cf6d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:30:46.924 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:46.919+0000 7f8346173740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f833c1ab130 con 0x55a262cf9450 2024-03-31T23:30:46.924 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:46.919+0000 7f8346173740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f833c1ac5a0 con 0x55a262ce7ac0 2024-03-31T23:30:46.924 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:46.919+0000 7f8346173740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f833c1ab800 con 0x55a262ce7110 2024-03-31T23:30:46.924 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:46.919+0000 7f8341328640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a262ce7110 0x55a262d01530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:30:46.925 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:46.919+0000 7f8346173740 10 client.0 _ll_register_callbacks cb 0x7f832c009ce0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T23:30:46.925 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:46.919+0000 7f8341328640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a262ce7110 0x55a262d01530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.46:34748/0 (socket says 172.21.15.46:34748) 2024-03-31T23:30:46.925 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:46.919+0000 7f8341328640 1 -- 172.21.15.46:0/3951675358 learned_addr learned my addr 172.21.15.46:0/3951675358 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:30:46.925 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:46.919+0000 7f8340b27640 1 --2- 172.21.15.46:0/3951675358 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a262ce7ac0 0x55a262cf6800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:30:46.925 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:46.919+0000 7f8341b29640 1 --2- 172.21.15.46:0/3951675358 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a262cf9450 0x55a262cf6d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:30:46.925 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:46.919+0000 7f8341328640 1 -- 172.21.15.46:0/3951675358 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a262ce7ac0 msgr2=0x55a262cf6800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:30:46.925 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:46.919+0000 7f8341328640 1 --2- 172.21.15.46:0/3951675358 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a262ce7ac0 0x55a262cf6800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:30:46.925 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:46.919+0000 7f8341328640 1 -- 172.21.15.46:0/3951675358 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a262cf9450 msgr2=0x55a262cf6d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:30:46.925 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:46.919+0000 7f8341328640 1 --2- 172.21.15.46:0/3951675358 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a262cf9450 0x55a262cf6d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:30:46.925 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:46.919+0000 7f8341328640 1 -- 172.21.15.46:0/3951675358 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a262cfc290 con 0x55a262ce7110 2024-03-31T23:30:46.925 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:46.919+0000 7f8341b29640 1 --2- 172.21.15.46:0/3951675358 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a262cf9450 0x55a262cf6d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T23:30:46.925 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:46.919+0000 7f8341328640 1 --2- 172.21.15.46:0/3951675358 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a262ce7110 0x55a262d01530 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f83380978a0 tx=0x7f833809bcb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6de12e997df1bdd3 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:30:46.925 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:46.919+0000 7f8324ff9640 10 client.0 ms_handle_connect on v2:172.21.15.162:3300/0 2024-03-31T23:30:46.927 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:46.920+0000 7f8324ff9640 1 -- 172.21.15.46:0/3951675358 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f833809f590 con 0x55a262ce7110 2024-03-31T23:30:46.927 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:46.920+0000 7f8324ff9640 1 -- 172.21.15.46:0/3951675358 <== mon.1 v2:172.21.15.162:3300/0 2 ==== osd_map(124..124 src has 1..124) ==== 9554+0+0 (secure 0 0 0) 0x7f8338097ab0 con 0x55a262ce7110 2024-03-31T23:30:46.927 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:46.920+0000 7f8346173740 1 -- 172.21.15.46:0/3951675358 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f83340047b0 con 0x55a262ce7110 2024-03-31T23:30:46.927 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:46.920+0000 7f8324ff9640 1 -- 172.21.15.46:0/3951675358 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) ==== 24+0+0 (secure 0 0 0) 0x7f83380b4150 con 0x55a262ce7110 2024-03-31T23:30:46.927 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:46.920+0000 7f8346173740 10 client.27415 fetch_fsmap learned FSMap version 47 2024-03-31T23:30:46.927 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:46.920+0000 7f8346173740 1 -- 172.21.15.46:0/3951675358 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) -- 0x55a262d6f610 con 0x55a262ce7110 2024-03-31T23:30:46.928 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:46.920+0000 7f8324ff9640 1 -- 172.21.15.46:0/3951675358 <== mon.1 v2:172.21.15.162:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f83340047b0 con 0x55a262ce7110 2024-03-31T23:30:46.928 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:46.920+0000 7f8324ff9640 1 -- 172.21.15.46:0/3951675358 <== mon.1 v2:172.21.15.162:3300/0 5 ==== fsmap(e 47) ==== 8346+0+0 (secure 0 0 0) 0x7f83380ccb60 con 0x55a262ce7110 2024-03-31T23:30:46.928 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:46.920+0000 7f8346173740 10 client.27415 fetch_fsmap finished waiting for FSMap version 47 2024-03-31T23:30:46.928 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:46.920+0000 7f8346173740 10 client.27415 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-31T23:30:46.928 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:46.920+0000 7f8346173740 20 client.27415 populate_metadata read hostname 'smithi046' 2024-03-31T23:30:46.928 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:46.920+0000 7f8346173740 1 --2- 172.21.15.46:0/3951675358 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55a262d6f900 0x55a262d71cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:30:46.928 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:46.921+0000 7f8346173740 4 client.27415 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-31T23:30:46.928 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:46.921+0000 7f8346173740 1 -- 172.21.15.46:0/3951675358 --> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55a262c52180 con 0x55a262d6f900 2024-03-31T23:30:46.928 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:46.921+0000 7f8340b27640 1 --2- 172.21.15.46:0/3951675358 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55a262d6f900 0x55a262d71cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:30:46.928 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:46.921+0000 7f8340b27640 1 --2- 172.21.15.46:0/3951675358 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55a262d6f900 0x55a262d71cf0 crc :-1 s=READY pgs=247 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=82258ce455f5b6f2 server_cookie=7d7af0792e997894 in_seq=0 out_seq=0 2024-03-31T23:30:46.928 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:46.921+0000 7f8324ff9640 10 client.27415 ms_handle_connect on v2:172.21.15.142:6838/2519143918 2024-03-31T23:30:46.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:30:46 smithi142 ceph-mon[35271]: pgmap v602: 129 pgs: 129 active+clean; 876 MiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 6.4 MiB/s wr, 404 op/s 2024-03-31T23:30:47.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:30:46 smithi162 ceph-mon[36237]: pgmap v602: 129 pgs: 129 active+clean; 876 MiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 6.4 MiB/s wr, 404 op/s 2024-03-31T23:30:47.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:30:46 smithi046 ceph-mon[32498]: pgmap v602: 129 pgs: 129 active+clean; 876 MiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 6.4 MiB/s wr, 404 op/s 2024-03-31T23:30:47.134 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:47.132+0000 7f8324ff9640 1 -- 172.21.15.46:0/3951675358 <== mds.0 v2:172.21.15.142:6838/2519143918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55a262c52180 con 0x55a262d6f900 2024-03-31T23:30:47.134 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:47.132+0000 7f8324ff9640 10 client.27415 handle_command_reply: tid=0 2024-03-31T23:30:47.134 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:47.132+0000 7f8346173740 1 client.27415 shutdown 2024-03-31T23:30:47.134 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:47.132+0000 7f8346173740 1 -- 172.21.15.46:0/3951675358 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a262ce7110 msgr2=0x55a262d01530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:30:47.134 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:47.132+0000 7f8346173740 1 --2- 172.21.15.46:0/3951675358 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a262ce7110 0x55a262d01530 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f83380978a0 tx=0x7f833809bcb0 comp rx=0 tx=0).stop 2024-03-31T23:30:47.134 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:47.132+0000 7f8346173740 1 -- 172.21.15.46:0/3951675358 shutdown_connections 2024-03-31T23:30:47.134 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:47.132+0000 7f8346173740 1 --2- 172.21.15.46:0/3951675358 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55a262d6f900 0x55a262d71cf0 crc :-1 s=READY pgs=247 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:30:47.134 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:47.132+0000 7f8346173740 1 --2- 172.21.15.46:0/3951675358 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a262cf9450 0x55a262cf6d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:30:47.134 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:47.132+0000 7f8346173740 1 --2- 172.21.15.46:0/3951675358 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a262ce7ac0 0x55a262cf6800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:30:47.135 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:47.132+0000 7f8324ff9640 0 client.27415 ms_handle_reset on v2:172.21.15.142:6838/2519143918 2024-03-31T23:30:47.135 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:47.132+0000 7f8346173740 1 --2- 172.21.15.46:0/3951675358 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a262ce7110 0x55a262d01530 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:30:47.135 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:47.132+0000 7f8346173740 1 -- 172.21.15.46:0/3951675358 >> 172.21.15.46:0/3951675358 conn(0x55a262ce1fe0 msgr2=0x55a262ce23c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:30:47.135 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:47.132+0000 7f8346173740 1 -- 172.21.15.46:0/3951675358 shutdown_connections 2024-03-31T23:30:47.135 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:47.132+0000 7f8346173740 1 -- 172.21.15.46:0/3951675358 wait complete. 2024-03-31T23:30:47.135 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:47.133+0000 7f8346173740 20 client.27415 trim_cache size 0 max 16384 2024-03-31T23:30:47.135 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout:{ 2024-03-31T23:30:47.135 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout: "message": "", 2024-03-31T23:30:47.135 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout: "return_code": 0 2024-03-31T23:30:47.136 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout:} 2024-03-31T23:30:47.136 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:47.133+0000 7f834358c640 1 -- 172.21.15.46:0/2168925525 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f83280a2730 msgr2=0x7f83280a4bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:30:47.137 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:47.133+0000 7f834358c640 1 --2- 172.21.15.46:0/2168925525 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f83280a2730 0x7f83280a4bf0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f8334005cc0 tx=0x7f83340099a0 comp rx=0 tx=0).stop 2024-03-31T23:30:47.137 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:47.133+0000 7f834358c640 1 -- 172.21.15.46:0/2168925525 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f833c0ffc50 msgr2=0x7f833c197e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:30:47.137 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:47.133+0000 7f834358c640 1 --2- 172.21.15.46:0/2168925525 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f833c0ffc50 0x7f833c197e10 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 crypto rx=0x7f833800ea60 tx=0x7f833800ef20 comp rx=0 tx=0).stop 2024-03-31T23:30:47.137 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:47.133+0000 7f834358c640 1 -- 172.21.15.46:0/2168925525 shutdown_connections 2024-03-31T23:30:47.137 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:47.133+0000 7f834358c640 1 --2- 172.21.15.46:0/2168925525 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f83280a2730 0x7f83280a4bf0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:30:47.137 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:47.133+0000 7f834358c640 1 --2- 172.21.15.46:0/2168925525 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f833c107730 0x7f833c19c6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:30:47.137 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:47.133+0000 7f834358c640 1 --2- 172.21.15.46:0/2168925525 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f833c100610 0x7f833c198350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:30:47.137 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:47.133+0000 7f834358c640 1 --2- 172.21.15.46:0/2168925525 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f833c0ffc50 0x7f833c197e10 unknown :-1 s=CLOSED pgs=962 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:30:47.137 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:47.133+0000 7f834358c640 1 -- 172.21.15.46:0/2168925525 >> 172.21.15.46:0/2168925525 conn(0x7f833c076c10 msgr2=0x7f833c0fe130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:30:47.137 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:47.134+0000 7f834358c640 1 -- 172.21.15.46:0/2168925525 shutdown_connections 2024-03-31T23:30:47.137 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:30:47.134+0000 7f834358c640 1 -- 172.21.15.46:0/2168925525 wait complete. 2024-03-31T23:30:47.292 INFO:tasks.workunit.client.0.smithi046.stdout: 10 56314 8.50 MB/sec execute 232 sec latency 191.527 ms 2024-03-31T23:30:48.292 INFO:tasks.workunit.client.0.smithi046.stdout: 10 56478 8.51 MB/sec execute 233 sec latency 107.425 ms 2024-03-31T23:30:48.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:30:48 smithi142 ceph-mon[35271]: pgmap v603: 129 pgs: 129 active+clean; 876 MiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 3.9 MiB/s wr, 241 op/s 2024-03-31T23:30:49.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:30:48 smithi162 ceph-mon[36237]: pgmap v603: 129 pgs: 129 active+clean; 876 MiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 3.9 MiB/s wr, 241 op/s 2024-03-31T23:30:49.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:30:48 smithi046 ceph-mon[32498]: pgmap v603: 129 pgs: 129 active+clean; 876 MiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 3.9 MiB/s wr, 241 op/s 2024-03-31T23:30:49.292 INFO:tasks.workunit.client.0.smithi046.stdout: 10 56605 8.50 MB/sec execute 234 sec latency 64.364 ms 2024-03-31T23:30:50.292 INFO:tasks.workunit.client.0.smithi046.stdout: 10 56751 8.52 MB/sec execute 235 sec latency 196.961 ms 2024-03-31T23:30:50.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:30:50 smithi142 ceph-mon[35271]: pgmap v604: 129 pgs: 129 active+clean; 888 MiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 6.3 MiB/s wr, 404 op/s 2024-03-31T23:30:51.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:30:50 smithi162 ceph-mon[36237]: pgmap v604: 129 pgs: 129 active+clean; 888 MiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 6.3 MiB/s wr, 404 op/s 2024-03-31T23:30:51.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:30:50 smithi046 ceph-mon[32498]: pgmap v604: 129 pgs: 129 active+clean; 888 MiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 6.3 MiB/s wr, 404 op/s 2024-03-31T23:30:51.292 INFO:tasks.workunit.client.0.smithi046.stdout: 10 56928 8.54 MB/sec execute 236 sec latency 68.749 ms 2024-03-31T23:30:52.292 INFO:tasks.workunit.client.0.smithi046.stdout: 10 57067 8.52 MB/sec execute 237 sec latency 141.950 ms 2024-03-31T23:30:52.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:30:52 smithi142 ceph-mon[35271]: pgmap v605: 129 pgs: 129 active+clean; 888 MiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 4.3 MiB/s wr, 273 op/s 2024-03-31T23:30:53.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:30:52 smithi162 ceph-mon[36237]: pgmap v605: 129 pgs: 129 active+clean; 888 MiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 4.3 MiB/s wr, 273 op/s 2024-03-31T23:30:53.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:30:52 smithi046 ceph-mon[32498]: pgmap v605: 129 pgs: 129 active+clean; 888 MiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 4.3 MiB/s wr, 273 op/s 2024-03-31T23:30:53.292 INFO:tasks.workunit.client.0.smithi046.stdout: 10 57202 8.52 MB/sec execute 238 sec latency 170.606 ms 2024-03-31T23:30:54.293 INFO:tasks.workunit.client.0.smithi046.stdout: 10 57338 8.51 MB/sec execute 239 sec latency 87.711 ms 2024-03-31T23:30:54.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:30:54 smithi142 ceph-mon[35271]: pgmap v606: 129 pgs: 129 active+clean; 888 MiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 4.3 MiB/s wr, 273 op/s 2024-03-31T23:30:55.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:30:54 smithi162 ceph-mon[36237]: pgmap v606: 129 pgs: 129 active+clean; 888 MiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 4.3 MiB/s wr, 273 op/s 2024-03-31T23:30:55.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:30:54 smithi046 ceph-mon[32498]: pgmap v606: 129 pgs: 129 active+clean; 888 MiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 4.3 MiB/s wr, 273 op/s 2024-03-31T23:30:55.292 INFO:tasks.workunit.client.0.smithi046.stdout: 10 57510 8.52 MB/sec execute 240 sec latency 64.178 ms 2024-03-31T23:30:56.292 INFO:tasks.workunit.client.0.smithi046.stdout: 10 57648 8.51 MB/sec execute 241 sec latency 117.829 ms 2024-03-31T23:30:56.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:30:56 smithi142 ceph-mon[35271]: pgmap v607: 129 pgs: 129 active+clean; 902 MiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 6.8 MiB/s wr, 367 op/s 2024-03-31T23:30:57.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:30:56 smithi162 ceph-mon[36237]: pgmap v607: 129 pgs: 129 active+clean; 902 MiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 6.8 MiB/s wr, 367 op/s 2024-03-31T23:30:57.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:30:56 smithi046 ceph-mon[32498]: pgmap v607: 129 pgs: 129 active+clean; 902 MiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 6.8 MiB/s wr, 367 op/s 2024-03-31T23:30:57.292 INFO:tasks.workunit.client.0.smithi046.stdout: 10 57797 8.51 MB/sec execute 242 sec latency 112.204 ms 2024-03-31T23:30:58.292 INFO:tasks.workunit.client.0.smithi046.stdout: 10 57949 8.50 MB/sec execute 243 sec latency 91.029 ms 2024-03-31T23:30:58.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:30:58 smithi142 ceph-mon[35271]: pgmap v608: 129 pgs: 129 active+clean; 902 MiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 4.8 MiB/s wr, 256 op/s 2024-03-31T23:30:59.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:30:58 smithi162 ceph-mon[36237]: pgmap v608: 129 pgs: 129 active+clean; 902 MiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 4.8 MiB/s wr, 256 op/s 2024-03-31T23:30:59.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:30:58 smithi046 ceph-mon[32498]: pgmap v608: 129 pgs: 129 active+clean; 902 MiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 4.8 MiB/s wr, 256 op/s 2024-03-31T23:30:59.292 INFO:tasks.workunit.client.0.smithi046.stdout: 10 58094 8.50 MB/sec execute 244 sec latency 101.371 ms 2024-03-31T23:31:00.274 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:00.271+0000 7fee7a713640 1 -- 172.21.15.46:0/2946120954 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fee741055f0 msgr2=0x7fee741059d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:31:00.275 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:00.271+0000 7fee7a713640 1 --2- 172.21.15.46:0/2946120954 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fee741055f0 0x7fee741059d0 secure :-1 s=READY pgs=964 cs=0 l=1 rev1=1 crypto rx=0x7fee64009900 tx=0x7fee6402bc30 comp rx=0 tx=0).stop 2024-03-31T23:31:00.275 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:00.271+0000 7fee7a713640 1 -- 172.21.15.46:0/2946120954 shutdown_connections 2024-03-31T23:31:00.275 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:00.271+0000 7fee7a713640 1 --2- 172.21.15.46:0/2946120954 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fee74101260 0x7fee7410d480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:31:00.275 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:00.271+0000 7fee7a713640 1 --2- 172.21.15.46:0/2946120954 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fee741008a0 0x7fee74100d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:31:00.275 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:00.271+0000 7fee7a713640 1 --2- 172.21.15.46:0/2946120954 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fee741055f0 0x7fee741059d0 unknown :-1 s=CLOSED pgs=964 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:31:00.275 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:00.271+0000 7fee7a713640 1 -- 172.21.15.46:0/2946120954 >> 172.21.15.46:0/2946120954 conn(0x7fee740774a0 msgr2=0x7fee740778b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:31:00.275 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:00.271+0000 7fee7a713640 1 -- 172.21.15.46:0/2946120954 shutdown_connections 2024-03-31T23:31:00.275 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:00.271+0000 7fee7a713640 1 -- 172.21.15.46:0/2946120954 wait complete. 2024-03-31T23:31:00.275 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:00.272+0000 7fee7a713640 1 Processor -- start 2024-03-31T23:31:00.275 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:00.272+0000 7fee7a713640 1 -- start start 2024-03-31T23:31:00.275 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:00.272+0000 7fee7a713640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fee741008a0 0x7fee7410dc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:31:00.275 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:00.272+0000 7fee7a713640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fee74101260 0x7fee7410e150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:31:00.276 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:00.272+0000 7fee7a713640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fee741055f0 0x7fee7410e690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:31:00.276 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:00.272+0000 7fee7a713640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fee74111a70 con 0x7fee741055f0 2024-03-31T23:31:00.276 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:00.272+0000 7fee7a713640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fee741118f0 con 0x7fee741008a0 2024-03-31T23:31:00.276 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:00.272+0000 7fee7a713640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fee74111bf0 con 0x7fee74101260 2024-03-31T23:31:00.276 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:00.272+0000 7fee78cb0640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fee741055f0 0x7fee7410e690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:31:00.276 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:00.272+0000 7fee73fff640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fee741008a0 0x7fee7410dc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:31:00.276 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:00.272+0000 7fee737fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fee74101260 0x7fee7410e150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:31:00.276 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:00.272+0000 7fee78cb0640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fee741055f0 0x7fee7410e690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:58828/0 (socket says 172.21.15.46:58828) 2024-03-31T23:31:00.276 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:00.272+0000 7fee73fff640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fee741008a0 0x7fee7410dc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.46:58128/0 (socket says 172.21.15.46:58128) 2024-03-31T23:31:00.276 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:00.272+0000 7fee78cb0640 1 -- 172.21.15.46:0/2180369821 learned_addr learned my addr 172.21.15.46:0/2180369821 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:31:00.276 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:00.273+0000 7fee78cb0640 1 -- 172.21.15.46:0/2180369821 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fee74101260 msgr2=0x7fee7410e150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:31:00.276 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:00.273+0000 7fee78cb0640 1 --2- 172.21.15.46:0/2180369821 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fee74101260 0x7fee7410e150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:31:00.276 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:00.273+0000 7fee78cb0640 1 -- 172.21.15.46:0/2180369821 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fee741008a0 msgr2=0x7fee7410dc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:31:00.277 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:00.273+0000 7fee78cb0640 1 --2- 172.21.15.46:0/2180369821 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fee741008a0 0x7fee7410dc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:31:00.277 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:00.273+0000 7fee78cb0640 1 -- 172.21.15.46:0/2180369821 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee741a63f0 con 0x7fee741055f0 2024-03-31T23:31:00.277 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:00.273+0000 7fee737fe640 1 --2- 172.21.15.46:0/2180369821 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fee74101260 0x7fee7410e150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:31:00.277 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:00.273+0000 7fee78cb0640 1 --2- 172.21.15.46:0/2180369821 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fee741055f0 0x7fee7410e690 secure :-1 s=READY pgs=965 cs=0 l=1 rev1=1 crypto rx=0x7fee6c00e960 tx=0x7fee6c00ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:31:00.277 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:00.273+0000 7fee717fa640 1 -- 172.21.15.46:0/2180369821 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee6c004d20 con 0x7fee741055f0 2024-03-31T23:31:00.277 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:00.273+0000 7fee7a713640 1 -- 172.21.15.46:0/2180369821 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fee741a66e0 con 0x7fee741055f0 2024-03-31T23:31:00.277 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:00.273+0000 7fee717fa640 1 -- 172.21.15.46:0/2180369821 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fee6c005020 con 0x7fee741055f0 2024-03-31T23:31:00.277 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:00.273+0000 7fee717fa640 1 -- 172.21.15.46:0/2180369821 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee6c0116d0 con 0x7fee741055f0 2024-03-31T23:31:00.277 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:00.273+0000 7fee7a713640 1 -- 172.21.15.46:0/2180369821 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fee741a6c30 con 0x7fee741055f0 2024-03-31T23:31:00.280 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:00.276+0000 7fee717fa640 1 -- 172.21.15.46:0/2180369821 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 25) ==== 137451+0+0 (secure 0 0 0) 0x7fee6c0051d0 con 0x7fee741055f0 2024-03-31T23:31:00.280 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:00.277+0000 7fee7d3d3740 1 Processor -- start 2024-03-31T23:31:00.280 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:00.277+0000 7fee7d3d3740 1 -- start start 2024-03-31T23:31:00.280 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:00.277+0000 7fee717fa640 1 --2- 172.21.15.46:0/2180369821 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fee5c0a2730 0x7fee5c0a4bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:31:00.280 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:00.277+0000 7fee7d3d3740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55eea03c66c0 0x55eea03c6aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:31:00.280 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:00.277+0000 7fee7d3d3740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55eea03c6fe0 0x55eea03cf870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:31:00.280 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:00.278+0000 7fee7d3d3740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55eea03cfdb0 0x55eea03d21a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:31:00.280 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:00.278+0000 7fee7d3d3740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55eea03298f0 con 0x55eea03c66c0 2024-03-31T23:31:00.280 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:00.278+0000 7fee717fa640 1 -- 172.21.15.46:0/2180369821 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(124..124 src has 1..124) ==== 9554+0+0 (secure 0 0 0) 0x7fee6c0cefe0 con 0x7fee741055f0 2024-03-31T23:31:00.281 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:00.278+0000 7fee7d3d3740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55eea02d28d0 con 0x55eea03cfdb0 2024-03-31T23:31:00.281 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:00.278+0000 7fee7d3d3740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55eea0325990 con 0x55eea03c6fe0 2024-03-31T23:31:00.281 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:00.278+0000 7fee73fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55eea03c66c0 0x55eea03c6aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:31:00.281 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:00.278+0000 7fee73fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55eea03c66c0 0x55eea03c6aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:58840/0 (socket says 172.21.15.46:58840) 2024-03-31T23:31:00.281 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:00.278+0000 7fee73fff640 1 -- 172.21.15.46:0/2252614961 learned_addr learned my addr 172.21.15.46:0/2252614961 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:31:00.281 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:00.278+0000 7fee737fe640 1 --2- 172.21.15.46:0/2180369821 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fee5c0a2730 0x7fee5c0a4bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:31:00.281 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:00.278+0000 7fee78cb0640 1 --2- 172.21.15.46:0/2252614961 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55eea03cfdb0 0x55eea03d21a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:31:00.281 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:00.278+0000 7fee737fe640 1 --2- 172.21.15.46:0/2252614961 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55eea03c6fe0 0x55eea03cf870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:31:00.281 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:00.278+0000 7fee78cb0640 1 -- 172.21.15.46:0/2252614961 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55eea03c6fe0 msgr2=0x55eea03cf870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:31:00.281 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:00.278+0000 7fee78cb0640 1 --2- 172.21.15.46:0/2252614961 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55eea03c6fe0 0x55eea03cf870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:31:00.281 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:00.278+0000 7fee78cb0640 1 -- 172.21.15.46:0/2252614961 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55eea03c66c0 msgr2=0x55eea03c6aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:31:00.281 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:00.278+0000 7fee78cb0640 1 --2- 172.21.15.46:0/2252614961 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55eea03c66c0 0x55eea03c6aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:31:00.281 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:00.278+0000 7fee78cb0640 1 -- 172.21.15.46:0/2252614961 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55eea003e210 con 0x55eea03cfdb0 2024-03-31T23:31:00.281 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:00.278+0000 7fee73fff640 1 --2- 172.21.15.46:0/2252614961 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55eea03c66c0 0x55eea03c6aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:31:00.282 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:00.278+0000 7fee737fe640 1 --2- 172.21.15.46:0/2252614961 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55eea03c6fe0 0x55eea03cf870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T23:31:00.282 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:00.279+0000 7fee737fe640 1 --2- 172.21.15.46:0/2180369821 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fee5c0a2730 0x7fee5c0a4bf0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fee7410f6a0 tx=0x7fee68012c40 comp rx=0 tx=0).ready entity=mgr.15654 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:31:00.282 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:00.279+0000 7fee78cb0640 1 --2- 172.21.15.46:0/2252614961 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55eea03cfdb0 0x55eea03d21a0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fee6c0cc2c0 tx=0x7fee6c09b3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=83e605485533b1a6 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:31:00.282 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:00.279+0000 7fee5a7fc640 1 -- 172.21.15.46:0/2252614961 <== mon.2 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee6c01f040 con 0x55eea03cfdb0 2024-03-31T23:31:00.282 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:00.279+0000 7fee5a7fc640 1 -- 172.21.15.46:0/2252614961 <== mon.2 v2:172.21.15.142:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fee6c0996a0 con 0x55eea03cfdb0 2024-03-31T23:31:00.282 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:00.279+0000 7fee5a7fc640 1 -- 172.21.15.46:0/2252614961 <== mon.2 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee6c0cc540 con 0x55eea03cfdb0 2024-03-31T23:31:00.289 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:00.280+0000 7fee7d3d3740 1 -- 172.21.15.46:0/2252614961 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55eea03cfdb0 msgr2=0x55eea03d21a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:31:00.289 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:00.280+0000 7fee7d3d3740 1 --2- 172.21.15.46:0/2252614961 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55eea03cfdb0 0x55eea03d21a0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fee6c0cc2c0 tx=0x7fee6c09b3a0 comp rx=0 tx=0).stop 2024-03-31T23:31:00.289 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:00.280+0000 7fee7d3d3740 1 -- 172.21.15.46:0/2252614961 shutdown_connections 2024-03-31T23:31:00.289 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:00.280+0000 7fee7d3d3740 1 --2- 172.21.15.46:0/2252614961 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55eea03cfdb0 0x55eea03d21a0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:31:00.289 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:00.280+0000 7fee7d3d3740 1 --2- 172.21.15.46:0/2252614961 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55eea03c6fe0 0x55eea03cf870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:31:00.289 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:00.280+0000 7fee7d3d3740 1 --2- 172.21.15.46:0/2252614961 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55eea03c66c0 0x55eea03c6aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:31:00.289 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:00.280+0000 7fee7d3d3740 1 -- 172.21.15.46:0/2252614961 >> 172.21.15.46:0/2252614961 conn(0x55eea03c0fe0 msgr2=0x55eea03c13c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:31:00.289 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:00.280+0000 7fee7d3d3740 1 -- 172.21.15.46:0/2252614961 shutdown_connections 2024-03-31T23:31:00.289 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:00.280+0000 7fee7d3d3740 1 -- 172.21.15.46:0/2252614961 wait complete. 2024-03-31T23:31:00.289 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:00.280+0000 7fee7d3d3740 1 -- start start 2024-03-31T23:31:00.290 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:00.281+0000 7fee7d3d3740 1 Processor -- start 2024-03-31T23:31:00.290 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:00.281+0000 7fee7d3d3740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55eea03c66c0 0x55eea03cceb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:31:00.290 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:00.281+0000 7fee7d3d3740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55eea03c6fe0 0x55eea03cd3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:31:00.290 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:00.281+0000 7fee7d3d3740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55eea03cfdb0 0x55eea03cd930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:31:00.290 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:00.281+0000 7fee7d3d3740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55eea03d29e0 con 0x55eea03c6fe0 2024-03-31T23:31:00.290 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:00.281+0000 7fee737fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55eea03c6fe0 0x55eea03cd3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:31:00.290 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:00.281+0000 7fee7d3d3740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55eea02c7f80 con 0x55eea03c66c0 2024-03-31T23:31:00.290 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:00.281+0000 7fee737fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55eea03c6fe0 0x55eea03cd3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:58842/0 (socket says 172.21.15.46:58842) 2024-03-31T23:31:00.290 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:00.281+0000 7fee73fff640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55eea03c66c0 0x55eea03cceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:31:00.290 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:00.281+0000 7fee737fe640 1 -- 172.21.15.46:0/2262754941 learned_addr learned my addr 172.21.15.46:0/2262754941 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:31:00.290 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:00.281+0000 7fee7d3d3740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55eea03d2b60 con 0x55eea03cfdb0 2024-03-31T23:31:00.290 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:00.281+0000 7fee78cb0640 1 --2- 172.21.15.46:0/2262754941 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55eea03cfdb0 0x55eea03cd930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:31:00.290 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:00.281+0000 7fee7d3d3740 10 client.0 _ll_register_callbacks cb 0x55eea0320290 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T23:31:00.291 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:00.281+0000 7fee73fff640 1 -- 172.21.15.46:0/2262754941 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55eea03cfdb0 msgr2=0x55eea03cd930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:31:00.291 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:00.281+0000 7fee73fff640 1 --2- 172.21.15.46:0/2262754941 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55eea03cfdb0 0x55eea03cd930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:31:00.291 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:00.282+0000 7fee73fff640 1 -- 172.21.15.46:0/2262754941 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55eea03c6fe0 msgr2=0x55eea03cd3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:31:00.291 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:00.282+0000 7fee73fff640 1 --2- 172.21.15.46:0/2262754941 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55eea03c6fe0 0x55eea03cd3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:31:00.291 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:00.282+0000 7fee73fff640 1 -- 172.21.15.46:0/2262754941 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55eea010a620 con 0x55eea03c66c0 2024-03-31T23:31:00.291 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:00.282+0000 7fee737fe640 1 --2- 172.21.15.46:0/2262754941 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55eea03c6fe0 0x55eea03cd3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:31:00.291 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:00.282+0000 7fee73fff640 1 --2- 172.21.15.46:0/2262754941 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55eea03c66c0 0x55eea03cceb0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fee64036040 tx=0x7fee64009680 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8873661f7e9ffb1d server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:31:00.291 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:00.282+0000 7fee537fe640 10 client.0 ms_handle_connect on v2:172.21.15.142:3300/0 2024-03-31T23:31:00.292 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:00.282+0000 7fee537fe640 1 -- 172.21.15.46:0/2262754941 <== mon.2 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee64005030 con 0x55eea03c66c0 2024-03-31T23:31:00.292 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:00.282+0000 7fee537fe640 1 -- 172.21.15.46:0/2262754941 <== mon.2 v2:172.21.15.142:3300/0 2 ==== osd_map(124..124 src has 1..124) ==== 9554+0+0 (secure 0 0 0) 0x7fee640051d0 con 0x55eea03c66c0 2024-03-31T23:31:00.292 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:00.282+0000 7fee7d3d3740 1 -- 172.21.15.46:0/2262754941 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fee6c0996a0 con 0x55eea03c66c0 2024-03-31T23:31:00.292 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:00.282+0000 7fee537fe640 1 -- 172.21.15.46:0/2262754941 <== mon.2 v2:172.21.15.142:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) ==== 24+0+0 (secure 0 0 0) 0x7fee640053d0 con 0x55eea03c66c0 2024-03-31T23:31:00.292 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:00.283+0000 7fee7d3d3740 10 client.27536 fetch_fsmap learned FSMap version 47 2024-03-31T23:31:00.292 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:00.283+0000 7fee7d3d3740 1 -- 172.21.15.46:0/2262754941 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) -- 0x55eea03de950 con 0x55eea03c66c0 2024-03-31T23:31:00.292 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:00.283+0000 7fee537fe640 1 -- 172.21.15.46:0/2262754941 <== mon.2 v2:172.21.15.142:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fee6c0996a0 con 0x55eea03c66c0 2024-03-31T23:31:00.292 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:00.283+0000 7fee537fe640 1 -- 172.21.15.46:0/2262754941 <== mon.2 v2:172.21.15.142:3300/0 5 ==== fsmap(e 47) ==== 8346+0+0 (secure 0 0 0) 0x7fee64003850 con 0x55eea03c66c0 2024-03-31T23:31:00.292 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:00.283+0000 7fee7d3d3740 10 client.27536 fetch_fsmap finished waiting for FSMap version 47 2024-03-31T23:31:00.292 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:00.283+0000 7fee7d3d3740 10 client.27536 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-31T23:31:00.292 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:00.283+0000 7fee7d3d3740 20 client.27536 populate_metadata read hostname 'smithi046' 2024-03-31T23:31:00.292 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:00.283+0000 7fee7d3d3740 1 --2- 172.21.15.46:0/2262754941 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55eea03dfe10 0x55eea03e2260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:31:00.292 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:00.283+0000 7fee7d3d3740 4 client.27536 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-31T23:31:00.292 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:00.283+0000 7fee7d3d3740 1 -- 172.21.15.46:0/2262754941 --> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55eea033d120 con 0x55eea03dfe10 2024-03-31T23:31:00.293 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:00.283+0000 7fee737fe640 1 --2- 172.21.15.46:0/2262754941 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55eea03dfe10 0x55eea03e2260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:31:00.293 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:00.284+0000 7fee537fe640 10 client.27536 ms_handle_connect on v2:172.21.15.142:6838/2519143918 2024-03-31T23:31:00.293 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:00.284+0000 7fee737fe640 1 --2- 172.21.15.46:0/2262754941 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55eea03dfe10 0x55eea03e2260 crc :-1 s=READY pgs=248 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ce61c11a2e1583b3 server_cookie=7e3d6cabec5f3774 in_seq=0 out_seq=0 2024-03-31T23:31:00.293 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:00.285+0000 7fee537fe640 1 -- 172.21.15.46:0/2262754941 <== mds.0 v2:172.21.15.142:6838/2519143918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55eea033d120 con 0x55eea03dfe10 2024-03-31T23:31:00.293 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:00.285+0000 7fee537fe640 10 client.27536 handle_command_reply: tid=0 2024-03-31T23:31:00.293 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:00.285+0000 7fee7d3d3740 1 client.27536 shutdown 2024-03-31T23:31:00.293 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:00.285+0000 7fee7d3d3740 1 -- 172.21.15.46:0/2262754941 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55eea03c66c0 msgr2=0x55eea03cceb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:31:00.293 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:00.285+0000 7fee7d3d3740 1 --2- 172.21.15.46:0/2262754941 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55eea03c66c0 0x55eea03cceb0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fee64036040 tx=0x7fee64009680 comp rx=0 tx=0).stop 2024-03-31T23:31:00.293 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:00.286+0000 7fee7d3d3740 1 -- 172.21.15.46:0/2262754941 shutdown_connections 2024-03-31T23:31:00.293 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:00.286+0000 7fee7d3d3740 1 --2- 172.21.15.46:0/2262754941 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55eea03dfe10 0x55eea03e2260 crc :-1 s=READY pgs=248 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:31:00.293 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:00.286+0000 7fee7d3d3740 1 --2- 172.21.15.46:0/2262754941 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55eea03cfdb0 0x55eea03cd930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:31:00.293 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:00.286+0000 7fee537fe640 0 client.27536 ms_handle_reset on v2:172.21.15.142:6838/2519143918 2024-03-31T23:31:00.293 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:00.286+0000 7fee7d3d3740 1 --2- 172.21.15.46:0/2262754941 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55eea03c6fe0 0x55eea03cd3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:31:00.293 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:00.286+0000 7fee7d3d3740 1 --2- 172.21.15.46:0/2262754941 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55eea03c66c0 0x55eea03cceb0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:31:00.294 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:00.286+0000 7fee7d3d3740 1 -- 172.21.15.46:0/2262754941 >> 172.21.15.46:0/2262754941 conn(0x55eea03c0fe0 msgr2=0x55eea03c2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:31:00.294 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:00.286+0000 7fee7d3d3740 1 -- 172.21.15.46:0/2262754941 shutdown_connections 2024-03-31T23:31:00.294 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:00.286+0000 7fee7d3d3740 1 -- 172.21.15.46:0/2262754941 wait complete. 2024-03-31T23:31:00.294 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:00.286+0000 7fee7d3d3740 20 client.27536 trim_cache size 0 max 16384 2024-03-31T23:31:00.294 INFO:tasks.workunit.client.0.smithi046.stdout: 10 58229 8.49 MB/sec execute 245 sec latency 89.695 ms 2024-03-31T23:31:00.314 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:00.313+0000 7fee7d3d3740 1 Processor -- start 2024-03-31T23:31:00.315 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:00.313+0000 7fee7d3d3740 1 -- start start 2024-03-31T23:31:00.315 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:00.313+0000 7fee7d3d3740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55eea03c66c0 0x55eea03cf5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:31:00.315 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:00.313+0000 7fee7d3d3740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55eea03c6fe0 0x55eea03db780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:31:00.315 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:00.313+0000 7fee7d3d3740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55eea03cfdb0 0x55eea03e7bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:31:00.315 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:00.313+0000 7fee7d3d3740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fee741b36a0 con 0x55eea03c6fe0 2024-03-31T23:31:00.315 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:00.313+0000 7fee7d3d3740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fee741b0f70 con 0x55eea03c66c0 2024-03-31T23:31:00.315 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:00.313+0000 7fee7d3d3740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fee741aede0 con 0x55eea03cfdb0 2024-03-31T23:31:00.315 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:00.313+0000 7fee737fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55eea03c6fe0 0x55eea03db780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:31:00.315 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:00.313+0000 7fee73fff640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55eea03c66c0 0x55eea03cf5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:31:00.315 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:00.313+0000 7fee73fff640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55eea03c66c0 0x55eea03cf5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.46:58150/0 (socket says 172.21.15.46:58150) 2024-03-31T23:31:00.315 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:00.313+0000 7fee73fff640 1 -- 172.21.15.46:0/2826799383 learned_addr learned my addr 172.21.15.46:0/2826799383 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:31:00.315 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:00.313+0000 7fee78cb0640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55eea03cfdb0 0x55eea03e7bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:31:00.316 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:00.314+0000 7fee73fff640 1 -- 172.21.15.46:0/2826799383 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55eea03cfdb0 msgr2=0x55eea03e7bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:31:00.316 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:00.314+0000 7fee73fff640 1 --2- 172.21.15.46:0/2826799383 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55eea03cfdb0 0x55eea03e7bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:31:00.316 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:00.314+0000 7fee73fff640 1 -- 172.21.15.46:0/2826799383 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55eea03c6fe0 msgr2=0x55eea03db780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:31:00.316 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:00.314+0000 7fee73fff640 1 --2- 172.21.15.46:0/2826799383 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55eea03c6fe0 0x55eea03db780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:31:00.316 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:00.314+0000 7fee73fff640 1 -- 172.21.15.46:0/2826799383 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55eea03d2510 con 0x55eea03c66c0 2024-03-31T23:31:00.320 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:00.314+0000 7fee73fff640 1 --2- 172.21.15.46:0/2826799383 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55eea03c66c0 0x55eea03cf5e0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fee64005d00 tx=0x7fee6402c3d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=15bbc2dadcfcbc40 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:31:00.320 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:00.314+0000 7fee59ffb640 1 -- 172.21.15.46:0/2826799383 <== mon.2 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee64059070 con 0x55eea03c66c0 2024-03-31T23:31:00.320 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:00.314+0000 7fee59ffb640 1 -- 172.21.15.46:0/2826799383 <== mon.2 v2:172.21.15.142:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fee64003e90 con 0x55eea03c66c0 2024-03-31T23:31:00.320 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:00.314+0000 7fee59ffb640 1 -- 172.21.15.46:0/2826799383 <== mon.2 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee6403f8b0 con 0x55eea03c66c0 2024-03-31T23:31:00.320 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:00.315+0000 7fee7d3d3740 1 -- 172.21.15.46:0/2826799383 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55eea03c66c0 msgr2=0x55eea03cf5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:31:00.320 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:00.315+0000 7fee7d3d3740 1 --2- 172.21.15.46:0/2826799383 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55eea03c66c0 0x55eea03cf5e0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fee64005d00 tx=0x7fee6402c3d0 comp rx=0 tx=0).stop 2024-03-31T23:31:00.320 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:00.315+0000 7fee7d3d3740 1 -- 172.21.15.46:0/2826799383 shutdown_connections 2024-03-31T23:31:00.320 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:00.315+0000 7fee7d3d3740 1 --2- 172.21.15.46:0/2826799383 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55eea03cfdb0 0x55eea03e7bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:31:00.320 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:00.315+0000 7fee7d3d3740 1 --2- 172.21.15.46:0/2826799383 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55eea03c6fe0 0x55eea03db780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:31:00.320 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:00.315+0000 7fee7d3d3740 1 --2- 172.21.15.46:0/2826799383 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55eea03c66c0 0x55eea03cf5e0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:31:00.320 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:00.315+0000 7fee7d3d3740 1 -- 172.21.15.46:0/2826799383 >> 172.21.15.46:0/2826799383 conn(0x55eea03c0fe0 msgr2=0x55eea03c7670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:31:00.320 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:00.316+0000 7fee7d3d3740 1 -- 172.21.15.46:0/2826799383 shutdown_connections 2024-03-31T23:31:00.321 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:00.316+0000 7fee7d3d3740 1 -- 172.21.15.46:0/2826799383 wait complete. 2024-03-31T23:31:00.321 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:00.316+0000 7fee7d3d3740 1 -- start start 2024-03-31T23:31:00.321 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:00.316+0000 7fee7d3d3740 1 Processor -- start 2024-03-31T23:31:00.321 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:00.317+0000 7fee7d3d3740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55eea03c66c0 0x55eea03c99a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:31:00.321 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:00.317+0000 7fee7d3d3740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55eea03c6fe0 0x55eea03c9ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:31:00.321 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:00.317+0000 7fee7d3d3740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55eea03cfdb0 0x55eea03ca420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:31:00.321 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:00.317+0000 7fee7d3d3740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fee741af430 con 0x55eea03cfdb0 2024-03-31T23:31:00.321 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:00.317+0000 7fee7d3d3740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fee741b08a0 con 0x55eea03c66c0 2024-03-31T23:31:00.321 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:00.317+0000 7fee7d3d3740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fee741afb00 con 0x55eea03c6fe0 2024-03-31T23:31:00.321 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:00.317+0000 7fee7d3d3740 10 client.0 _ll_register_callbacks cb 0x7fee64009560 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T23:31:00.321 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:00.317+0000 7fee73fff640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55eea03c66c0 0x55eea03c99a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:31:00.321 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:00.317+0000 7fee73fff640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55eea03c66c0 0x55eea03c99a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.46:58158/0 (socket says 172.21.15.46:58158) 2024-03-31T23:31:00.321 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:00.317+0000 7fee73fff640 1 -- 172.21.15.46:0/2707499264 learned_addr learned my addr 172.21.15.46:0/2707499264 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:31:00.321 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:00.317+0000 7fee737fe640 1 --2- 172.21.15.46:0/2707499264 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55eea03c6fe0 0x55eea03c9ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:31:00.322 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:00.317+0000 7fee78cb0640 1 --2- 172.21.15.46:0/2707499264 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55eea03cfdb0 0x55eea03ca420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:31:00.322 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:00.317+0000 7fee73fff640 1 -- 172.21.15.46:0/2707499264 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55eea03c6fe0 msgr2=0x55eea03c9ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:31:00.322 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:00.317+0000 7fee73fff640 1 --2- 172.21.15.46:0/2707499264 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55eea03c6fe0 0x55eea03c9ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:31:00.322 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:00.317+0000 7fee73fff640 1 -- 172.21.15.46:0/2707499264 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55eea03cfdb0 msgr2=0x55eea03ca420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:31:00.322 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:00.317+0000 7fee73fff640 1 --2- 172.21.15.46:0/2707499264 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55eea03cfdb0 0x55eea03ca420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:31:00.322 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:00.317+0000 7fee73fff640 1 -- 172.21.15.46:0/2707499264 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55eea044da00 con 0x55eea03c66c0 2024-03-31T23:31:00.322 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:00.317+0000 7fee78cb0640 1 --2- 172.21.15.46:0/2707499264 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55eea03cfdb0 0x55eea03ca420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T23:31:00.322 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:00.318+0000 7fee73fff640 1 --2- 172.21.15.46:0/2707499264 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55eea03c66c0 0x55eea03c99a0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fee64000c00 tx=0x7fee640058c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f6866307878945ad server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:31:00.322 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:00.318+0000 7fee537fe640 10 client.0 ms_handle_connect on v2:172.21.15.142:3300/0 2024-03-31T23:31:00.323 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:00.318+0000 7fee537fe640 1 -- 172.21.15.46:0/2707499264 <== mon.2 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee64059070 con 0x55eea03c66c0 2024-03-31T23:31:00.323 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:00.318+0000 7fee7d3d3740 1 -- 172.21.15.46:0/2707499264 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fee64003e90 con 0x55eea03c66c0 2024-03-31T23:31:00.323 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:00.318+0000 7fee537fe640 1 -- 172.21.15.46:0/2707499264 <== mon.2 v2:172.21.15.142:3300/0 2 ==== osd_map(124..124 src has 1..124) ==== 9554+0+0 (secure 0 0 0) 0x7fee6403f540 con 0x55eea03c66c0 2024-03-31T23:31:00.323 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:00.318+0000 7fee537fe640 1 -- 172.21.15.46:0/2707499264 <== mon.2 v2:172.21.15.142:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) ==== 24+0+0 (secure 0 0 0) 0x7fee64066050 con 0x55eea03c66c0 2024-03-31T23:31:00.323 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:00.318+0000 7fee7d3d3740 10 client.27542 fetch_fsmap learned FSMap version 47 2024-03-31T23:31:00.323 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:00.318+0000 7fee7d3d3740 1 -- 172.21.15.46:0/2707499264 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) -- 0x55eea044ec90 con 0x55eea03c66c0 2024-03-31T23:31:00.323 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:00.318+0000 7fee537fe640 1 -- 172.21.15.46:0/2707499264 <== mon.2 v2:172.21.15.142:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fee64003e90 con 0x55eea03c66c0 2024-03-31T23:31:00.323 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:00.319+0000 7fee537fe640 1 -- 172.21.15.46:0/2707499264 <== mon.2 v2:172.21.15.142:3300/0 5 ==== fsmap(e 47) ==== 8346+0+0 (secure 0 0 0) 0x7fee64054930 con 0x55eea03c66c0 2024-03-31T23:31:00.323 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:00.319+0000 7fee7d3d3740 10 client.27542 fetch_fsmap finished waiting for FSMap version 47 2024-03-31T23:31:00.323 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:00.319+0000 7fee7d3d3740 10 client.27542 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-31T23:31:00.323 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:00.319+0000 7fee7d3d3740 20 client.27542 populate_metadata read hostname 'smithi046' 2024-03-31T23:31:00.323 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:00.319+0000 7fee7d3d3740 1 --2- 172.21.15.46:0/2707499264 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55eea03dfe10 0x55eea0451390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:31:00.324 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:00.319+0000 7fee7d3d3740 4 client.27542 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-31T23:31:00.324 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:00.319+0000 7fee7d3d3740 1 -- 172.21.15.46:0/2707499264 --> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55eea0336e30 con 0x55eea03dfe10 2024-03-31T23:31:00.324 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:00.319+0000 7fee737fe640 1 --2- 172.21.15.46:0/2707499264 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55eea03dfe10 0x55eea0451390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:31:00.324 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:00.320+0000 7fee737fe640 1 --2- 172.21.15.46:0/2707499264 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55eea03dfe10 0x55eea0451390 crc :-1 s=READY pgs=249 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fca903a005b9c18d server_cookie=473715b5c609769c in_seq=0 out_seq=0 2024-03-31T23:31:00.324 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:00.320+0000 7fee537fe640 10 client.27542 ms_handle_connect on v2:172.21.15.142:6838/2519143918 2024-03-31T23:31:00.486 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:00.484+0000 7fee537fe640 1 -- 172.21.15.46:0/2707499264 <== mds.0 v2:172.21.15.142:6838/2519143918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55eea0336e30 con 0x55eea03dfe10 2024-03-31T23:31:00.486 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:00.484+0000 7fee537fe640 10 client.27542 handle_command_reply: tid=0 2024-03-31T23:31:00.486 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:00.484+0000 7fee7d3d3740 1 client.27542 shutdown 2024-03-31T23:31:00.486 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:00.484+0000 7fee7d3d3740 1 -- 172.21.15.46:0/2707499264 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55eea03c66c0 msgr2=0x55eea03c99a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:31:00.486 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:00.484+0000 7fee7d3d3740 1 --2- 172.21.15.46:0/2707499264 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55eea03c66c0 0x55eea03c99a0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fee64000c00 tx=0x7fee640058c0 comp rx=0 tx=0).stop 2024-03-31T23:31:00.486 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:00.484+0000 7fee7d3d3740 1 -- 172.21.15.46:0/2707499264 shutdown_connections 2024-03-31T23:31:00.486 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:00.484+0000 7fee7d3d3740 1 --2- 172.21.15.46:0/2707499264 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55eea03dfe10 0x55eea0451390 crc :-1 s=READY pgs=249 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:31:00.486 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:00.484+0000 7fee7d3d3740 1 --2- 172.21.15.46:0/2707499264 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55eea03cfdb0 0x55eea03ca420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:31:00.486 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:00.484+0000 7fee7d3d3740 1 --2- 172.21.15.46:0/2707499264 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55eea03c6fe0 0x55eea03c9ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:31:00.486 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:00.484+0000 7fee7d3d3740 1 --2- 172.21.15.46:0/2707499264 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55eea03c66c0 0x55eea03c99a0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:31:00.487 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:00.484+0000 7fee7d3d3740 1 -- 172.21.15.46:0/2707499264 >> 172.21.15.46:0/2707499264 conn(0x55eea03c0fe0 msgr2=0x55eea03bf6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:31:00.487 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:00.484+0000 7fee537fe640 0 client.27542 ms_handle_reset on v2:172.21.15.142:6838/2519143918 2024-03-31T23:31:00.487 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:00.484+0000 7fee7d3d3740 1 -- 172.21.15.46:0/2707499264 shutdown_connections 2024-03-31T23:31:00.487 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:00.484+0000 7fee7d3d3740 1 -- 172.21.15.46:0/2707499264 wait complete. 2024-03-31T23:31:00.487 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:00.484+0000 7fee7d3d3740 20 client.27542 trim_cache size 0 max 16384 2024-03-31T23:31:00.487 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout:{ 2024-03-31T23:31:00.487 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout: "message": "", 2024-03-31T23:31:00.487 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout: "return_code": 0 2024-03-31T23:31:00.487 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout:} 2024-03-31T23:31:00.488 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:00.485+0000 7fee7a713640 1 -- 172.21.15.46:0/2180369821 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fee5c0a2730 msgr2=0x7fee5c0a4bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:31:00.488 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:00.485+0000 7fee7a713640 1 --2- 172.21.15.46:0/2180369821 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fee5c0a2730 0x7fee5c0a4bf0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fee7410f6a0 tx=0x7fee68012c40 comp rx=0 tx=0).stop 2024-03-31T23:31:00.488 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:00.485+0000 7fee7a713640 1 -- 172.21.15.46:0/2180369821 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fee741055f0 msgr2=0x7fee7410e690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:31:00.488 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:00.485+0000 7fee7a713640 1 --2- 172.21.15.46:0/2180369821 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fee741055f0 0x7fee7410e690 secure :-1 s=READY pgs=965 cs=0 l=1 rev1=1 crypto rx=0x7fee6c00e960 tx=0x7fee6c00ee20 comp rx=0 tx=0).stop 2024-03-31T23:31:00.489 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:00.485+0000 7fee7a713640 1 -- 172.21.15.46:0/2180369821 shutdown_connections 2024-03-31T23:31:00.489 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:00.485+0000 7fee7a713640 1 --2- 172.21.15.46:0/2180369821 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fee5c0a2730 0x7fee5c0a4bf0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:31:00.489 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:00.485+0000 7fee7a713640 1 --2- 172.21.15.46:0/2180369821 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fee741055f0 0x7fee7410e690 unknown :-1 s=CLOSED pgs=965 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:31:00.489 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:00.485+0000 7fee7a713640 1 --2- 172.21.15.46:0/2180369821 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fee74101260 0x7fee7410e150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:31:00.489 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:00.485+0000 7fee7a713640 1 --2- 172.21.15.46:0/2180369821 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fee741008a0 0x7fee7410dc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:31:00.489 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:00.485+0000 7fee7a713640 1 -- 172.21.15.46:0/2180369821 >> 172.21.15.46:0/2180369821 conn(0x7fee740774a0 msgr2=0x7fee740fe8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:31:00.489 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:00.485+0000 7fee7a713640 1 -- 172.21.15.46:0/2180369821 shutdown_connections 2024-03-31T23:31:00.489 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:00.485+0000 7fee7a713640 1 -- 172.21.15.46:0/2180369821 wait complete. 2024-03-31T23:31:00.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:31:00 smithi142 ceph-mon[35271]: pgmap v609: 129 pgs: 129 active+clean; 908 MiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 7.1 MiB/s wr, 381 op/s 2024-03-31T23:31:01.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:31:00 smithi162 ceph-mon[36237]: pgmap v609: 129 pgs: 129 active+clean; 908 MiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 7.1 MiB/s wr, 381 op/s 2024-03-31T23:31:01.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:31:00 smithi046 ceph-mon[32498]: pgmap v609: 129 pgs: 129 active+clean; 908 MiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 7.1 MiB/s wr, 381 op/s 2024-03-31T23:31:01.292 INFO:tasks.workunit.client.0.smithi046.stdout: 10 58321 8.46 MB/sec execute 246 sec latency 354.097 ms 2024-03-31T23:31:02.293 INFO:tasks.workunit.client.0.smithi046.stdout: 10 58479 8.45 MB/sec execute 247 sec latency 91.282 ms 2024-03-31T23:31:02.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:31:02 smithi142 ceph-mon[35271]: pgmap v610: 129 pgs: 129 active+clean; 908 MiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 4.7 MiB/s wr, 219 op/s 2024-03-31T23:31:03.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:31:02 smithi162 ceph-mon[36237]: pgmap v610: 129 pgs: 129 active+clean; 908 MiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 4.7 MiB/s wr, 219 op/s 2024-03-31T23:31:03.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:31:02 smithi046 ceph-mon[32498]: pgmap v610: 129 pgs: 129 active+clean; 908 MiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 4.7 MiB/s wr, 219 op/s 2024-03-31T23:31:03.292 INFO:tasks.workunit.client.0.smithi046.stdout: 10 58611 8.43 MB/sec execute 248 sec latency 169.055 ms 2024-03-31T23:31:04.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:31:03 smithi162 ceph-mon[36237]: osdmap e125: 12 total, 12 up, 12 in 2024-03-31T23:31:04.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:31:03 smithi046 ceph-mon[32498]: osdmap e125: 12 total, 12 up, 12 in 2024-03-31T23:31:04.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:31:03 smithi142 ceph-mon[35271]: osdmap e125: 12 total, 12 up, 12 in 2024-03-31T23:31:04.293 INFO:tasks.workunit.client.0.smithi046.stdout: 10 58797 8.45 MB/sec execute 249 sec latency 84.488 ms 2024-03-31T23:31:05.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:31:04 smithi162 ceph-mon[36237]: pgmap v612: 129 pgs: 129 active+clean; 908 MiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 5.7 MiB/s wr, 263 op/s 2024-03-31T23:31:05.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:31:04 smithi046 ceph-mon[32498]: pgmap v612: 129 pgs: 129 active+clean; 908 MiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 5.7 MiB/s wr, 263 op/s 2024-03-31T23:31:05.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:31:04 smithi142 ceph-mon[35271]: pgmap v612: 129 pgs: 129 active+clean; 908 MiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 5.7 MiB/s wr, 263 op/s 2024-03-31T23:31:05.292 INFO:tasks.workunit.client.0.smithi046.stdout: 10 58963 8.45 MB/sec execute 250 sec latency 64.036 ms 2024-03-31T23:31:06.293 INFO:tasks.workunit.client.0.smithi046.stdout: 10 59105 8.45 MB/sec execute 251 sec latency 70.542 ms 2024-03-31T23:31:07.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:31:06 smithi162 ceph-mon[36237]: pgmap v613: 129 pgs: 129 active+clean; 898 MiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 4.9 MiB/s wr, 364 op/s 2024-03-31T23:31:07.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:31:06 smithi046 ceph-mon[32498]: pgmap v613: 129 pgs: 129 active+clean; 898 MiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 4.9 MiB/s wr, 364 op/s 2024-03-31T23:31:07.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:31:06 smithi142 ceph-mon[35271]: pgmap v613: 129 pgs: 129 active+clean; 898 MiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 4.9 MiB/s wr, 364 op/s 2024-03-31T23:31:07.293 INFO:tasks.workunit.client.0.smithi046.stdout: 10 59279 8.46 MB/sec execute 252 sec latency 70.689 ms 2024-03-31T23:31:08.293 INFO:tasks.workunit.client.0.smithi046.stdout: 10 59485 8.47 MB/sec execute 253 sec latency 76.359 ms 2024-03-31T23:31:09.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:31:08 smithi162 ceph-mon[36237]: pgmap v614: 129 pgs: 129 active+clean; 898 MiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 4.9 MiB/s wr, 364 op/s 2024-03-31T23:31:09.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:31:08 smithi046 ceph-mon[32498]: pgmap v614: 129 pgs: 129 active+clean; 898 MiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 4.9 MiB/s wr, 364 op/s 2024-03-31T23:31:09.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:31:08 smithi142 ceph-mon[35271]: pgmap v614: 129 pgs: 129 active+clean; 898 MiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 4.9 MiB/s wr, 364 op/s 2024-03-31T23:31:09.293 INFO:tasks.workunit.client.0.smithi046.stdout: 10 59689 8.48 MB/sec execute 254 sec latency 82.171 ms 2024-03-31T23:31:10.293 INFO:tasks.workunit.client.0.smithi046.stdout: 10 59808 8.46 MB/sec execute 255 sec latency 84.578 ms 2024-03-31T23:31:11.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:31:10 smithi162 ceph-mon[36237]: pgmap v615: 129 pgs: 129 active+clean; 923 MiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 5.0 MiB/s wr, 343 op/s 2024-03-31T23:31:11.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:31:10 smithi162 ceph-mon[36237]: osdmap e126: 12 total, 12 up, 12 in 2024-03-31T23:31:11.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:31:10 smithi046 ceph-mon[32498]: pgmap v615: 129 pgs: 129 active+clean; 923 MiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 5.0 MiB/s wr, 343 op/s 2024-03-31T23:31:11.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:31:10 smithi046 ceph-mon[32498]: osdmap e126: 12 total, 12 up, 12 in 2024-03-31T23:31:11.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:31:10 smithi142 ceph-mon[35271]: pgmap v615: 129 pgs: 129 active+clean; 923 MiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 5.0 MiB/s wr, 343 op/s 2024-03-31T23:31:11.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:31:10 smithi142 ceph-mon[35271]: osdmap e126: 12 total, 12 up, 12 in 2024-03-31T23:31:11.293 INFO:tasks.workunit.client.0.smithi046.stdout: 10 59931 8.46 MB/sec execute 256 sec latency 114.031 ms 2024-03-31T23:31:12.293 INFO:tasks.workunit.client.0.smithi046.stdout: 10 60103 8.49 MB/sec execute 257 sec latency 117.350 ms 2024-03-31T23:31:13.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:31:12 smithi162 ceph-mon[36237]: pgmap v617: 129 pgs: 129 active+clean; 923 MiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 6.1 MiB/s wr, 414 op/s 2024-03-31T23:31:13.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:31:12 smithi046 ceph-mon[32498]: pgmap v617: 129 pgs: 129 active+clean; 923 MiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 6.1 MiB/s wr, 414 op/s 2024-03-31T23:31:13.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:31:12 smithi142 ceph-mon[35271]: pgmap v617: 129 pgs: 129 active+clean; 923 MiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 6.1 MiB/s wr, 414 op/s 2024-03-31T23:31:13.293 INFO:tasks.workunit.client.0.smithi046.stdout: 10 60301 8.51 MB/sec execute 258 sec latency 59.970 ms 2024-03-31T23:31:13.612 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:13.609+0000 7f5a04209640 1 -- 172.21.15.46:0/3528312279 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f59fc100ff0 msgr2=0x7f59fc10d500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:31:13.612 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:13.609+0000 7f5a04209640 1 --2- 172.21.15.46:0/3528312279 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f59fc100ff0 0x7f59fc10d500 secure :-1 s=READY pgs=966 cs=0 l=1 rev1=1 crypto rx=0x7f59f4009c10 tx=0x7f59f4030300 comp rx=0 tx=0).stop 2024-03-31T23:31:13.612 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:13.609+0000 7f5a04209640 1 -- 172.21.15.46:0/3528312279 shutdown_connections 2024-03-31T23:31:13.612 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:13.609+0000 7f5a04209640 1 --2- 172.21.15.46:0/3528312279 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f59fc100ff0 0x7f59fc10d500 unknown :-1 s=CLOSED pgs=966 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:31:13.612 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:13.609+0000 7f5a04209640 1 --2- 172.21.15.46:0/3528312279 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f59fc100630 0x7f59fc100ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:31:13.613 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:13.609+0000 7f5a04209640 1 --2- 172.21.15.46:0/3528312279 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f59fc106640 0x7f59fc106a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:31:13.613 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:13.609+0000 7f5a04209640 1 -- 172.21.15.46:0/3528312279 >> 172.21.15.46:0/3528312279 conn(0x7f59fc0777b0 msgr2=0x7f59fc077bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:31:13.613 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:13.610+0000 7f5a04209640 1 -- 172.21.15.46:0/3528312279 shutdown_connections 2024-03-31T23:31:13.613 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:13.610+0000 7f5a04209640 1 -- 172.21.15.46:0/3528312279 wait complete. 2024-03-31T23:31:13.613 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:13.610+0000 7f5a04209640 1 Processor -- start 2024-03-31T23:31:13.613 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:13.610+0000 7f5a04209640 1 -- start start 2024-03-31T23:31:13.613 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:13.611+0000 7f5a04209640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f59fc100630 0x7f59fc19b900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:31:13.613 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:13.611+0000 7f5a04209640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f59fc100ff0 0x7f59fc19be40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:31:13.613 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:13.611+0000 7f5a04209640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f59fc106640 0x7f59fc195ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:31:13.613 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:13.611+0000 7f5a04209640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f59fc10f7f0 con 0x7f59fc100630 2024-03-31T23:31:13.613 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:13.611+0000 7f5a04209640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f59fc10f670 con 0x7f59fc100ff0 2024-03-31T23:31:13.613 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:13.611+0000 7f5a04209640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f59fc10f970 con 0x7f59fc106640 2024-03-31T23:31:13.613 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:13.611+0000 7f5a01fa5640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f59fc100630 0x7f59fc19b900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:31:13.613 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:13.611+0000 7f5a01fa5640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f59fc100630 0x7f59fc19b900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:42386/0 (socket says 172.21.15.46:42386) 2024-03-31T23:31:13.614 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:13.611+0000 7f5a017a4640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f59fc100ff0 0x7f59fc19be40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:31:13.614 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:13.611+0000 7f5a01fa5640 1 -- 172.21.15.46:0/2387295311 learned_addr learned my addr 172.21.15.46:0/2387295311 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:31:13.614 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:13.611+0000 7f5a027a6640 1 --2- 172.21.15.46:0/2387295311 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f59fc106640 0x7f59fc195ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:31:13.614 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:13.611+0000 7f5a01fa5640 1 -- 172.21.15.46:0/2387295311 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f59fc106640 msgr2=0x7f59fc195ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:31:13.614 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:13.611+0000 7f5a01fa5640 1 --2- 172.21.15.46:0/2387295311 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f59fc106640 0x7f59fc195ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:31:13.614 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:13.611+0000 7f5a01fa5640 1 -- 172.21.15.46:0/2387295311 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f59fc100ff0 msgr2=0x7f59fc19be40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:31:13.614 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:13.611+0000 7f5a01fa5640 1 --2- 172.21.15.46:0/2387295311 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f59fc100ff0 0x7f59fc19be40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:31:13.614 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:13.611+0000 7f5a01fa5640 1 -- 172.21.15.46:0/2387295311 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59fc196230 con 0x7f59fc100630 2024-03-31T23:31:13.614 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:13.611+0000 7f5a01fa5640 1 --2- 172.21.15.46:0/2387295311 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f59fc100630 0x7f59fc19b900 secure :-1 s=READY pgs=967 cs=0 l=1 rev1=1 crypto rx=0x7f59f800e9c0 tx=0x7f59f800ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:31:13.615 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:13.612+0000 7f59f2ffd640 1 -- 172.21.15.46:0/2387295311 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59f800ccc0 con 0x7f59fc100630 2024-03-31T23:31:13.615 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:13.612+0000 7f59f2ffd640 1 -- 172.21.15.46:0/2387295311 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f59f80054d0 con 0x7f59fc100630 2024-03-31T23:31:13.615 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:13.612+0000 7f5a04209640 1 -- 172.21.15.46:0/2387295311 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f59fc196520 con 0x7f59fc100630 2024-03-31T23:31:13.615 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:13.612+0000 7f59f2ffd640 1 -- 172.21.15.46:0/2387295311 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59f8010430 con 0x7f59fc100630 2024-03-31T23:31:13.615 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:13.612+0000 7f5a04209640 1 -- 172.21.15.46:0/2387295311 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f59fc1a27c0 con 0x7f59fc100630 2024-03-31T23:31:13.616 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:13.615+0000 7f59f2ffd640 1 -- 172.21.15.46:0/2387295311 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 25) ==== 137451+0+0 (secure 0 0 0) 0x7f59f80046c0 con 0x7f59fc100630 2024-03-31T23:31:13.617 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:13.615+0000 7f5a06de1740 1 Processor -- start 2024-03-31T23:31:13.617 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:13.615+0000 7f5a06de1740 1 -- start start 2024-03-31T23:31:13.617 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:13.615+0000 7f5a06de1740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5628ab4b46c0 0x5628ab4b4aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:31:13.617 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:13.615+0000 7f5a06de1740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5628ab4b4fe0 0x5628ab4bd870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:31:13.617 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:13.615+0000 7f5a06de1740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5628ab4bddb0 0x5628ab4c01a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:31:13.617 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:13.615+0000 7f5a06de1740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x5628ab4178f0 con 0x5628ab4bddb0 2024-03-31T23:31:13.617 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:13.615+0000 7f5a06de1740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x5628ab3c08d0 con 0x5628ab4b46c0 2024-03-31T23:31:13.617 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:13.615+0000 7f5a06de1740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5628ab413990 con 0x5628ab4b4fe0 2024-03-31T23:31:13.617 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:13.615+0000 7f5a017a4640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5628ab4bddb0 0x5628ab4c01a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:31:13.617 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:13.615+0000 7f5a01fa5640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5628ab4b4fe0 0x5628ab4bd870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:31:13.617 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:13.615+0000 7f5a01fa5640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5628ab4b4fe0 0x5628ab4bd870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.46:34752/0 (socket says 172.21.15.46:34752) 2024-03-31T23:31:13.617 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:13.615+0000 7f5a01fa5640 1 -- 172.21.15.46:0/1188423452 learned_addr learned my addr 172.21.15.46:0/1188423452 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:31:13.617 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:13.615+0000 7f5a017a4640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5628ab4bddb0 0x5628ab4c01a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:42392/0 (socket says 172.21.15.46:42392) 2024-03-31T23:31:13.617 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:13.615+0000 7f5a027a6640 1 --2- 172.21.15.46:0/1188423452 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5628ab4b46c0 0x5628ab4b4aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:31:13.618 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:13.615+0000 7f59f2ffd640 1 --2- 172.21.15.46:0/2387295311 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f59e80a26b0 0x7f59e80a4b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:31:13.618 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:13.615+0000 7f5a01fa5640 1 -- 172.21.15.46:0/1188423452 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5628ab4b46c0 msgr2=0x5628ab4b4aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:31:13.618 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:13.615+0000 7f5a01fa5640 1 --2- 172.21.15.46:0/1188423452 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5628ab4b46c0 0x5628ab4b4aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:31:13.618 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:13.615+0000 7f5a01fa5640 1 -- 172.21.15.46:0/1188423452 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5628ab4bddb0 msgr2=0x5628ab4c01a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:31:13.618 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:13.615+0000 7f5a01fa5640 1 --2- 172.21.15.46:0/1188423452 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5628ab4bddb0 0x5628ab4c01a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:31:13.618 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:13.615+0000 7f5a01fa5640 1 -- 172.21.15.46:0/1188423452 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5628ab12c210 con 0x5628ab4b4fe0 2024-03-31T23:31:13.618 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:13.616+0000 7f5a027a6640 1 --2- 172.21.15.46:0/2387295311 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f59e80a26b0 0x7f59e80a4b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:31:13.618 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:13.616+0000 7f5a01fa5640 1 --2- 172.21.15.46:0/1188423452 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5628ab4b4fe0 0x5628ab4bd870 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f59f80cc280 tx=0x7f59f80976d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=467e6e67130f84d server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:31:13.621 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:13.616+0000 7f59e7fff640 1 -- 172.21.15.46:0/1188423452 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59f809b140 con 0x5628ab4b4fe0 2024-03-31T23:31:13.621 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:13.616+0000 7f59e7fff640 1 -- 172.21.15.46:0/1188423452 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f59f8005680 con 0x5628ab4b4fe0 2024-03-31T23:31:13.621 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:13.616+0000 7f59e7fff640 1 -- 172.21.15.46:0/1188423452 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59f8082890 con 0x5628ab4b4fe0 2024-03-31T23:31:13.621 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:13.616+0000 7f59f2ffd640 1 -- 172.21.15.46:0/2387295311 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(126..126 src has 1..126) ==== 9554+0+0 (secure 0 0 0) 0x7f59f80cf060 con 0x7f59fc100630 2024-03-31T23:31:13.621 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:13.617+0000 7f5a06de1740 1 -- 172.21.15.46:0/1188423452 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5628ab4b4fe0 msgr2=0x5628ab4bd870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:31:13.622 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:13.617+0000 7f5a06de1740 1 --2- 172.21.15.46:0/1188423452 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5628ab4b4fe0 0x5628ab4bd870 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f59f80cc280 tx=0x7f59f80976d0 comp rx=0 tx=0).stop 2024-03-31T23:31:13.622 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:13.617+0000 7f5a027a6640 1 --2- 172.21.15.46:0/2387295311 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f59e80a26b0 0x7f59e80a4b70 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f59f4037ff0 tx=0x7f59f4030300 comp rx=0 tx=0).ready entity=mgr.15654 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:31:13.622 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:13.617+0000 7f5a06de1740 1 -- 172.21.15.46:0/1188423452 shutdown_connections 2024-03-31T23:31:13.622 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:13.617+0000 7f5a06de1740 1 --2- 172.21.15.46:0/1188423452 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5628ab4bddb0 0x5628ab4c01a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:31:13.622 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:13.617+0000 7f5a06de1740 1 --2- 172.21.15.46:0/1188423452 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5628ab4b4fe0 0x5628ab4bd870 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:31:13.622 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:13.617+0000 7f5a06de1740 1 --2- 172.21.15.46:0/1188423452 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5628ab4b46c0 0x5628ab4b4aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:31:13.622 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:13.617+0000 7f5a06de1740 1 -- 172.21.15.46:0/1188423452 >> 172.21.15.46:0/1188423452 conn(0x5628ab4aefe0 msgr2=0x5628ab4af3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:31:13.622 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:13.617+0000 7f5a06de1740 1 -- 172.21.15.46:0/1188423452 shutdown_connections 2024-03-31T23:31:13.622 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:13.617+0000 7f5a06de1740 1 -- 172.21.15.46:0/1188423452 wait complete. 2024-03-31T23:31:13.622 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:13.617+0000 7f5a06de1740 1 -- start start 2024-03-31T23:31:13.622 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:13.617+0000 7f5a06de1740 1 Processor -- start 2024-03-31T23:31:13.622 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:13.618+0000 7f5a06de1740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5628ab4b46c0 0x5628ab4bae90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:31:13.622 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:13.618+0000 7f5a06de1740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5628ab4b4fe0 0x5628ab4bb3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:31:13.623 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:13.618+0000 7f5a06de1740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5628ab4bddb0 0x5628ab4bb910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:31:13.623 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:13.618+0000 7f5a06de1740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x5628ab4c09e0 con 0x5628ab4bddb0 2024-03-31T23:31:13.623 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:13.618+0000 7f5a06de1740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x5628ab3b5f80 con 0x5628ab4b46c0 2024-03-31T23:31:13.623 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:13.618+0000 7f5a06de1740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5628ab4c0b60 con 0x5628ab4b4fe0 2024-03-31T23:31:13.623 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:13.618+0000 7f5a06de1740 10 client.0 _ll_register_callbacks cb 0x5628ab40e290 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T23:31:13.623 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:13.618+0000 7f5a027a6640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5628ab4bddb0 0x5628ab4bb910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:31:13.623 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:13.618+0000 7f5a01fa5640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5628ab4b46c0 0x5628ab4bae90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:31:13.623 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:13.618+0000 7f5a017a4640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5628ab4b4fe0 0x5628ab4bb3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:31:13.623 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:13.618+0000 7f5a027a6640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5628ab4bddb0 0x5628ab4bb910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:42408/0 (socket says 172.21.15.46:42408) 2024-03-31T23:31:13.623 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:13.618+0000 7f5a027a6640 1 -- 172.21.15.46:0/1233873235 learned_addr learned my addr 172.21.15.46:0/1233873235 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:31:13.623 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:13.618+0000 7f5a01fa5640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5628ab4b46c0 0x5628ab4bae90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.46:48128/0 (socket says 172.21.15.46:48128) 2024-03-31T23:31:13.623 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:13.618+0000 7f5a01fa5640 1 -- 172.21.15.46:0/1233873235 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5628ab4b4fe0 msgr2=0x5628ab4bb3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:31:13.623 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:13.618+0000 7f5a01fa5640 1 --2- 172.21.15.46:0/1233873235 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5628ab4b4fe0 0x5628ab4bb3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:31:13.623 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:13.618+0000 7f5a01fa5640 1 -- 172.21.15.46:0/1233873235 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5628ab4bddb0 msgr2=0x5628ab4bb910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:31:13.623 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:13.618+0000 7f5a01fa5640 1 --2- 172.21.15.46:0/1233873235 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5628ab4bddb0 0x5628ab4bb910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:31:13.624 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:13.618+0000 7f5a01fa5640 1 -- 172.21.15.46:0/1233873235 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5628ab1f8620 con 0x5628ab4b46c0 2024-03-31T23:31:13.624 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:13.618+0000 7f5a027a6640 1 --2- 172.21.15.46:0/1233873235 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5628ab4bddb0 0x5628ab4bb910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:31:13.624 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:13.618+0000 7f5a017a4640 1 --2- 172.21.15.46:0/1233873235 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5628ab4b4fe0 0x5628ab4bb3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T23:31:13.624 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:13.618+0000 7f5a01fa5640 1 --2- 172.21.15.46:0/1233873235 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5628ab4b46c0 0x5628ab4bae90 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f59f801b590 tx=0x7f59f80049b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5a2e115e2435c3eb server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:31:13.624 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:13.618+0000 7f59e57fa640 10 client.0 ms_handle_connect on v2:172.21.15.142:3300/0 2024-03-31T23:31:13.624 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:13.619+0000 7f59e57fa640 1 -- 172.21.15.46:0/1233873235 <== mon.2 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59f8082a60 con 0x5628ab4b46c0 2024-03-31T23:31:13.624 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:13.619+0000 7f5a06de1740 1 -- 172.21.15.46:0/1233873235 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f59f8005680 con 0x5628ab4b46c0 2024-03-31T23:31:13.624 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:13.619+0000 7f59e57fa640 1 -- 172.21.15.46:0/1233873235 <== mon.2 v2:172.21.15.142:3300/0 2 ==== osd_map(126..126 src has 1..126) ==== 9554+0+0 (secure 0 0 0) 0x7f59f8007620 con 0x5628ab4b46c0 2024-03-31T23:31:13.624 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:13.619+0000 7f59e57fa640 1 -- 172.21.15.46:0/1233873235 <== mon.2 v2:172.21.15.142:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) ==== 24+0+0 (secure 0 0 0) 0x7f59f80817c0 con 0x5628ab4b46c0 2024-03-31T23:31:13.624 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:13.619+0000 7f5a06de1740 10 client.27554 fetch_fsmap learned FSMap version 47 2024-03-31T23:31:13.624 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:13.619+0000 7f5a06de1740 1 -- 172.21.15.46:0/1233873235 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) -- 0x5628ab4cc9e0 con 0x5628ab4b46c0 2024-03-31T23:31:13.624 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:13.619+0000 7f59e57fa640 1 -- 172.21.15.46:0/1233873235 <== mon.2 v2:172.21.15.142:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f59f8005680 con 0x5628ab4b46c0 2024-03-31T23:31:13.624 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:13.619+0000 7f59e57fa640 1 -- 172.21.15.46:0/1233873235 <== mon.2 v2:172.21.15.142:3300/0 5 ==== fsmap(e 47) ==== 8346+0+0 (secure 0 0 0) 0x7f59f809f0e0 con 0x5628ab4b46c0 2024-03-31T23:31:13.624 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:13.619+0000 7f5a06de1740 10 client.27554 fetch_fsmap finished waiting for FSMap version 47 2024-03-31T23:31:13.625 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:13.619+0000 7f5a06de1740 10 client.27554 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-31T23:31:13.625 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:13.619+0000 7f5a06de1740 20 client.27554 populate_metadata read hostname 'smithi046' 2024-03-31T23:31:13.625 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:13.619+0000 7f5a06de1740 1 --2- 172.21.15.46:0/1233873235 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x5628ab4ce100 0x5628ab4d05a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:31:13.625 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:13.619+0000 7f5a06de1740 4 client.27554 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-31T23:31:13.625 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:13.619+0000 7f5a06de1740 1 -- 172.21.15.46:0/1233873235 --> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5628ab42b120 con 0x5628ab4ce100 2024-03-31T23:31:13.625 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:13.620+0000 7f5a017a4640 1 --2- 172.21.15.46:0/1233873235 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x5628ab4ce100 0x5628ab4d05a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:31:13.625 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:13.620+0000 7f59e57fa640 10 client.27554 ms_handle_connect on v2:172.21.15.142:6838/2519143918 2024-03-31T23:31:13.625 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:13.620+0000 7f5a017a4640 1 --2- 172.21.15.46:0/1233873235 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x5628ab4ce100 0x5628ab4d05a0 crc :-1 s=READY pgs=250 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ac5964269019649b server_cookie=ec44b3b9f3d7e641 in_seq=0 out_seq=0 2024-03-31T23:31:13.626 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:13.621+0000 7f59e57fa640 1 -- 172.21.15.46:0/1233873235 <== mds.0 v2:172.21.15.142:6838/2519143918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5628ab42b120 con 0x5628ab4ce100 2024-03-31T23:31:13.626 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:13.621+0000 7f59e57fa640 10 client.27554 handle_command_reply: tid=0 2024-03-31T23:31:13.626 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:13.621+0000 7f5a06de1740 1 client.27554 shutdown 2024-03-31T23:31:13.626 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:13.622+0000 7f5a06de1740 1 -- 172.21.15.46:0/1233873235 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5628ab4b46c0 msgr2=0x5628ab4bae90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:31:13.627 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:13.622+0000 7f5a06de1740 1 --2- 172.21.15.46:0/1233873235 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5628ab4b46c0 0x5628ab4bae90 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f59f801b590 tx=0x7f59f80049b0 comp rx=0 tx=0).stop 2024-03-31T23:31:13.627 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:13.622+0000 7f5a06de1740 1 -- 172.21.15.46:0/1233873235 shutdown_connections 2024-03-31T23:31:13.627 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:13.622+0000 7f5a06de1740 1 --2- 172.21.15.46:0/1233873235 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x5628ab4ce100 0x5628ab4d05a0 crc :-1 s=READY pgs=250 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:31:13.627 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:13.622+0000 7f5a06de1740 1 --2- 172.21.15.46:0/1233873235 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5628ab4bddb0 0x5628ab4bb910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:31:13.627 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:13.622+0000 7f5a06de1740 1 --2- 172.21.15.46:0/1233873235 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5628ab4b4fe0 0x5628ab4bb3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:31:13.627 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:13.622+0000 7f5a06de1740 1 --2- 172.21.15.46:0/1233873235 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5628ab4b46c0 0x5628ab4bae90 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:31:13.627 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:13.622+0000 7f5a06de1740 1 -- 172.21.15.46:0/1233873235 >> 172.21.15.46:0/1233873235 conn(0x5628ab4aefe0 msgr2=0x5628ab4b0d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:31:13.627 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:13.622+0000 7f59e57fa640 0 client.27554 ms_handle_reset on v2:172.21.15.142:6838/2519143918 2024-03-31T23:31:13.627 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:13.622+0000 7f5a06de1740 1 -- 172.21.15.46:0/1233873235 shutdown_connections 2024-03-31T23:31:13.627 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:13.622+0000 7f5a06de1740 1 -- 172.21.15.46:0/1233873235 wait complete. 2024-03-31T23:31:13.627 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:13.622+0000 7f5a06de1740 20 client.27554 trim_cache size 0 max 16384 2024-03-31T23:31:13.648 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:13.646+0000 7f5a06de1740 1 Processor -- start 2024-03-31T23:31:13.648 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:13.646+0000 7f5a06de1740 1 -- start start 2024-03-31T23:31:13.648 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:13.646+0000 7f5a06de1740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5628ab4b46c0 0x5628ab4bd5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:31:13.649 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:13.646+0000 7f5a06de1740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5628ab4b4fe0 0x5628ab4c97b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:31:13.649 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:13.646+0000 7f5a06de1740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5628ab4bddb0 0x5628ab4b9e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:31:13.649 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:13.646+0000 7f5a06de1740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f59e0012900 con 0x5628ab4bddb0 2024-03-31T23:31:13.649 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:13.646+0000 7f5a06de1740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f59e00101d0 con 0x5628ab4b4fe0 2024-03-31T23:31:13.649 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:13.646+0000 7f5a06de1740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f59e000e040 con 0x5628ab4b46c0 2024-03-31T23:31:13.649 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:13.646+0000 7f5a027a6640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5628ab4bddb0 0x5628ab4b9e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:31:13.649 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:13.646+0000 7f5a027a6640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5628ab4bddb0 0x5628ab4b9e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:42422/0 (socket says 172.21.15.46:42422) 2024-03-31T23:31:13.649 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:13.646+0000 7f5a01fa5640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5628ab4b46c0 0x5628ab4bd5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:31:13.649 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:13.646+0000 7f5a027a6640 1 -- 172.21.15.46:0/4010053454 learned_addr learned my addr 172.21.15.46:0/4010053454 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:31:13.649 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:13.646+0000 7f5a017a4640 1 --2- 172.21.15.46:0/4010053454 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5628ab4b4fe0 0x5628ab4c97b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:31:13.649 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:13.647+0000 7f5a017a4640 1 -- 172.21.15.46:0/4010053454 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5628ab4b46c0 msgr2=0x5628ab4bd5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:31:13.649 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:13.647+0000 7f5a017a4640 1 --2- 172.21.15.46:0/4010053454 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5628ab4b46c0 0x5628ab4bd5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:31:13.649 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:13.647+0000 7f5a017a4640 1 -- 172.21.15.46:0/4010053454 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5628ab4bddb0 msgr2=0x5628ab4b9e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:31:13.649 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:13.647+0000 7f5a017a4640 1 --2- 172.21.15.46:0/4010053454 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5628ab4bddb0 0x5628ab4b9e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:31:13.650 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:13.647+0000 7f5a017a4640 1 -- 172.21.15.46:0/4010053454 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5628ab4c9cf0 con 0x5628ab4b4fe0 2024-03-31T23:31:13.650 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:13.647+0000 7f5a027a6640 1 --2- 172.21.15.46:0/4010053454 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5628ab4bddb0 0x5628ab4b9e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:31:13.650 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:13.647+0000 7f5a01fa5640 1 --2- 172.21.15.46:0/4010053454 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5628ab4b46c0 0x5628ab4bd5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:31:13.650 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:13.647+0000 7f5a017a4640 1 --2- 172.21.15.46:0/4010053454 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5628ab4b4fe0 0x5628ab4c97b0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x5628ab4ba6f0 tx=0x7f59ec0046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4a45be39a881bbe7 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:31:13.650 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:13.647+0000 7f59e77fe640 1 -- 172.21.15.46:0/4010053454 <== mon.2 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59ec025070 con 0x5628ab4b4fe0 2024-03-31T23:31:13.650 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:13.647+0000 7f59e77fe640 1 -- 172.21.15.46:0/4010053454 <== mon.2 v2:172.21.15.142:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f59ec003d80 con 0x5628ab4b4fe0 2024-03-31T23:31:13.650 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:13.647+0000 7f59e77fe640 1 -- 172.21.15.46:0/4010053454 <== mon.2 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59ec010a70 con 0x5628ab4b4fe0 2024-03-31T23:31:13.654 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:13.648+0000 7f5a06de1740 1 -- 172.21.15.46:0/4010053454 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5628ab4b4fe0 msgr2=0x5628ab4c97b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:31:13.654 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:13.648+0000 7f5a06de1740 1 --2- 172.21.15.46:0/4010053454 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5628ab4b4fe0 0x5628ab4c97b0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x5628ab4ba6f0 tx=0x7f59ec0046c0 comp rx=0 tx=0).stop 2024-03-31T23:31:13.654 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:13.648+0000 7f5a06de1740 1 -- 172.21.15.46:0/4010053454 shutdown_connections 2024-03-31T23:31:13.654 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:13.648+0000 7f5a06de1740 1 --2- 172.21.15.46:0/4010053454 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5628ab4bddb0 0x5628ab4b9e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:31:13.654 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:13.648+0000 7f5a06de1740 1 --2- 172.21.15.46:0/4010053454 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5628ab4b4fe0 0x5628ab4c97b0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:31:13.654 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:13.648+0000 7f5a06de1740 1 --2- 172.21.15.46:0/4010053454 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5628ab4b46c0 0x5628ab4bd5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:31:13.654 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:13.648+0000 7f5a06de1740 1 -- 172.21.15.46:0/4010053454 >> 172.21.15.46:0/4010053454 conn(0x5628ab4aefe0 msgr2=0x5628ab4b9760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:31:13.654 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:13.648+0000 7f5a06de1740 1 -- 172.21.15.46:0/4010053454 shutdown_connections 2024-03-31T23:31:13.654 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:13.648+0000 7f5a06de1740 1 -- 172.21.15.46:0/4010053454 wait complete. 2024-03-31T23:31:13.654 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:13.649+0000 7f5a06de1740 1 -- start start 2024-03-31T23:31:13.654 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:13.649+0000 7f5a06de1740 1 Processor -- start 2024-03-31T23:31:13.654 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:13.649+0000 7f5a06de1740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5628ab4b46c0 0x5628ab4c0da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:31:13.655 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:13.649+0000 7f5a06de1740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5628ab4b4fe0 0x5628ab4c12e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:31:13.655 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:13.649+0000 7f5a06de1740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5628ab4bddb0 0x5628ab4c1820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:31:13.655 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:13.649+0000 7f5a06de1740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f59e000e690 con 0x5628ab4bddb0 2024-03-31T23:31:13.655 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:13.649+0000 7f5a06de1740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f59e000fb00 con 0x5628ab4b46c0 2024-03-31T23:31:13.655 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:13.649+0000 7f5a06de1740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f59e000ed60 con 0x5628ab4b4fe0 2024-03-31T23:31:13.655 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:13.649+0000 7f5a06de1740 10 client.0 _ll_register_callbacks cb 0x7f59ec009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T23:31:13.655 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:13.649+0000 7f5a027a6640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5628ab4bddb0 0x5628ab4c1820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:31:13.655 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:13.649+0000 7f5a027a6640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5628ab4bddb0 0x5628ab4c1820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:42436/0 (socket says 172.21.15.46:42436) 2024-03-31T23:31:13.655 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:13.649+0000 7f5a027a6640 1 -- 172.21.15.46:0/3998957911 learned_addr learned my addr 172.21.15.46:0/3998957911 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:31:13.655 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:13.649+0000 7f5a01fa5640 1 --2- 172.21.15.46:0/3998957911 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5628ab4b46c0 0x5628ab4c0da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:31:13.655 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:13.649+0000 7f5a017a4640 1 --2- 172.21.15.46:0/3998957911 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5628ab4b4fe0 0x5628ab4c12e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:31:13.655 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:13.650+0000 7f5a027a6640 1 -- 172.21.15.46:0/3998957911 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5628ab4b4fe0 msgr2=0x5628ab4c12e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:31:13.655 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:13.650+0000 7f5a027a6640 1 --2- 172.21.15.46:0/3998957911 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5628ab4b4fe0 0x5628ab4c12e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:31:13.655 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:13.650+0000 7f5a027a6640 1 -- 172.21.15.46:0/3998957911 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5628ab4b46c0 msgr2=0x5628ab4c0da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:31:13.656 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:13.650+0000 7f5a027a6640 1 --2- 172.21.15.46:0/3998957911 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5628ab4b46c0 0x5628ab4c0da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:31:13.656 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:13.650+0000 7f5a027a6640 1 -- 172.21.15.46:0/3998957911 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5628ab5324c0 con 0x5628ab4bddb0 2024-03-31T23:31:13.656 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:13.650+0000 7f5a01fa5640 1 --2- 172.21.15.46:0/3998957911 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5628ab4b46c0 0x5628ab4c0da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:31:13.656 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:13.650+0000 7f5a027a6640 1 --2- 172.21.15.46:0/3998957911 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5628ab4bddb0 0x5628ab4c1820 secure :-1 s=READY pgs=968 cs=0 l=1 rev1=1 crypto rx=0x7f59f40099a0 tx=0x7f59f4009500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae1bd43791a67e53 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:31:13.656 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:13.650+0000 7f59e57fa640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-31T23:31:13.656 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:13.650+0000 7f59e57fa640 1 -- 172.21.15.46:0/3998957911 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59f4057070 con 0x5628ab4bddb0 2024-03-31T23:31:13.656 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:13.650+0000 7f59e57fa640 1 -- 172.21.15.46:0/3998957911 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(126..126 src has 1..126) ==== 9554+0+0 (secure 0 0 0) 0x7f59f403e040 con 0x5628ab4bddb0 2024-03-31T23:31:13.656 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:13.650+0000 7f5a06de1740 1 -- 172.21.15.46:0/3998957911 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f59ec003d80 con 0x5628ab4bddb0 2024-03-31T23:31:13.656 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:13.650+0000 7f59e57fa640 1 -- 172.21.15.46:0/3998957911 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) ==== 24+0+0 (secure 0 0 0) 0x7f59f4064050 con 0x5628ab4bddb0 2024-03-31T23:31:13.656 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:13.650+0000 7f5a06de1740 10 client.17730 fetch_fsmap learned FSMap version 47 2024-03-31T23:31:13.656 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:13.651+0000 7f5a06de1740 1 -- 172.21.15.46:0/3998957911 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) -- 0x5628ab533790 con 0x5628ab4bddb0 2024-03-31T23:31:13.656 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:13.651+0000 7f59e57fa640 1 -- 172.21.15.46:0/3998957911 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f59ec003d80 con 0x5628ab4bddb0 2024-03-31T23:31:13.656 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:13.651+0000 7f59e57fa640 1 -- 172.21.15.46:0/3998957911 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 47) ==== 8346+0+0 (secure 0 0 0) 0x7f59f4043870 con 0x5628ab4bddb0 2024-03-31T23:31:13.656 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:13.651+0000 7f5a06de1740 10 client.17730 fetch_fsmap finished waiting for FSMap version 47 2024-03-31T23:31:13.656 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:13.651+0000 7f5a06de1740 10 client.17730 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-31T23:31:13.657 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:13.651+0000 7f5a06de1740 20 client.17730 populate_metadata read hostname 'smithi046' 2024-03-31T23:31:13.657 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:13.651+0000 7f5a06de1740 1 --2- 172.21.15.46:0/3998957911 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x5628ab533ee0 0x5628ab536300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:31:13.657 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:13.651+0000 7f5a06de1740 4 client.17730 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-31T23:31:13.657 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:13.651+0000 7f5a06de1740 1 -- 172.21.15.46:0/3998957911 --> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5628ab424e30 con 0x5628ab533ee0 2024-03-31T23:31:13.657 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:13.651+0000 7f5a01fa5640 1 --2- 172.21.15.46:0/3998957911 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x5628ab533ee0 0x5628ab536300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:31:13.657 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:13.652+0000 7f5a01fa5640 1 --2- 172.21.15.46:0/3998957911 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x5628ab533ee0 0x5628ab536300 crc :-1 s=READY pgs=251 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=361b0c2e7b0e0bbd server_cookie=6f6c1767053a334f in_seq=0 out_seq=0 2024-03-31T23:31:13.657 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:13.652+0000 7f59e57fa640 10 client.17730 ms_handle_connect on v2:172.21.15.142:6838/2519143918 2024-03-31T23:31:13.881 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:13.879+0000 7f59e57fa640 1 -- 172.21.15.46:0/3998957911 <== mds.0 v2:172.21.15.142:6838/2519143918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5628ab424e30 con 0x5628ab533ee0 2024-03-31T23:31:13.881 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:13.879+0000 7f59e57fa640 10 client.17730 handle_command_reply: tid=0 2024-03-31T23:31:13.881 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:13.879+0000 7f5a06de1740 1 client.17730 shutdown 2024-03-31T23:31:13.882 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:13.880+0000 7f5a06de1740 1 -- 172.21.15.46:0/3998957911 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5628ab4bddb0 msgr2=0x5628ab4c1820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:31:13.882 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:13.880+0000 7f5a06de1740 1 --2- 172.21.15.46:0/3998957911 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5628ab4bddb0 0x5628ab4c1820 secure :-1 s=READY pgs=968 cs=0 l=1 rev1=1 crypto rx=0x7f59f40099a0 tx=0x7f59f4009500 comp rx=0 tx=0).stop 2024-03-31T23:31:13.882 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:13.880+0000 7f5a06de1740 1 -- 172.21.15.46:0/3998957911 shutdown_connections 2024-03-31T23:31:13.882 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:13.880+0000 7f5a06de1740 1 --2- 172.21.15.46:0/3998957911 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x5628ab533ee0 0x5628ab536300 crc :-1 s=READY pgs=251 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:31:13.882 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:13.880+0000 7f5a06de1740 1 --2- 172.21.15.46:0/3998957911 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5628ab4bddb0 0x5628ab4c1820 unknown :-1 s=CLOSED pgs=968 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:31:13.882 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:13.880+0000 7f5a06de1740 1 --2- 172.21.15.46:0/3998957911 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5628ab4b4fe0 0x5628ab4c12e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:31:13.882 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:13.880+0000 7f5a06de1740 1 --2- 172.21.15.46:0/3998957911 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5628ab4b46c0 0x5628ab4c0da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:31:13.882 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:13.880+0000 7f5a06de1740 1 -- 172.21.15.46:0/3998957911 >> 172.21.15.46:0/3998957911 conn(0x5628ab4aefe0 msgr2=0x5628ab4b0400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:31:13.882 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:13.880+0000 7f59e57fa640 0 client.17730 ms_handle_reset on v2:172.21.15.142:6838/2519143918 2024-03-31T23:31:13.882 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:13.880+0000 7f5a06de1740 1 -- 172.21.15.46:0/3998957911 shutdown_connections 2024-03-31T23:31:13.882 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:13.880+0000 7f5a06de1740 1 -- 172.21.15.46:0/3998957911 wait complete. 2024-03-31T23:31:13.882 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:13.880+0000 7f5a06de1740 20 client.17730 trim_cache size 0 max 16384 2024-03-31T23:31:13.883 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout:{ 2024-03-31T23:31:13.883 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout: "message": "", 2024-03-31T23:31:13.883 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout: "return_code": 0 2024-03-31T23:31:13.883 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout:} 2024-03-31T23:31:13.884 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:13.881+0000 7f5a04209640 1 -- 172.21.15.46:0/2387295311 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f59e80a26b0 msgr2=0x7f59e80a4b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:31:13.884 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:13.881+0000 7f5a04209640 1 --2- 172.21.15.46:0/2387295311 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f59e80a26b0 0x7f59e80a4b70 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f59f4037ff0 tx=0x7f59f4030300 comp rx=0 tx=0).stop 2024-03-31T23:31:13.884 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:13.881+0000 7f5a04209640 1 -- 172.21.15.46:0/2387295311 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f59fc100630 msgr2=0x7f59fc19b900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:31:13.884 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:13.881+0000 7f5a04209640 1 --2- 172.21.15.46:0/2387295311 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f59fc100630 0x7f59fc19b900 secure :-1 s=READY pgs=967 cs=0 l=1 rev1=1 crypto rx=0x7f59f800e9c0 tx=0x7f59f800ee80 comp rx=0 tx=0).stop 2024-03-31T23:31:13.884 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:13.881+0000 7f5a04209640 1 -- 172.21.15.46:0/2387295311 shutdown_connections 2024-03-31T23:31:13.884 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:13.881+0000 7f5a04209640 1 --2- 172.21.15.46:0/2387295311 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f59e80a26b0 0x7f59e80a4b70 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:31:13.884 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:13.881+0000 7f5a04209640 1 --2- 172.21.15.46:0/2387295311 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f59fc106640 0x7f59fc195ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:31:13.884 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:13.881+0000 7f5a04209640 1 --2- 172.21.15.46:0/2387295311 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f59fc100ff0 0x7f59fc19be40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:31:13.884 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:13.881+0000 7f5a04209640 1 --2- 172.21.15.46:0/2387295311 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f59fc100630 0x7f59fc19b900 unknown :-1 s=CLOSED pgs=967 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:31:13.884 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:13.881+0000 7f5a04209640 1 -- 172.21.15.46:0/2387295311 >> 172.21.15.46:0/2387295311 conn(0x7f59fc0777b0 msgr2=0x7f59fc1031a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:31:13.884 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:13.881+0000 7f5a04209640 1 -- 172.21.15.46:0/2387295311 shutdown_connections 2024-03-31T23:31:13.884 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:13.881+0000 7f5a04209640 1 -- 172.21.15.46:0/2387295311 wait complete. 2024-03-31T23:31:14.293 INFO:tasks.workunit.client.0.smithi046.stdout: 10 60439 8.51 MB/sec execute 259 sec latency 181.998 ms 2024-03-31T23:31:15.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:31:14 smithi162 ceph-mon[36237]: pgmap v618: 129 pgs: 129 active+clean; 923 MiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 5.0 MiB/s wr, 343 op/s 2024-03-31T23:31:15.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:31:14 smithi162 ceph-mon[36237]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-31T23:31:15.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:31:14 smithi046 ceph-mon[32498]: pgmap v618: 129 pgs: 129 active+clean; 923 MiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 5.0 MiB/s wr, 343 op/s 2024-03-31T23:31:15.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:31:14 smithi046 ceph-mon[32498]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-31T23:31:15.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:31:14 smithi142 ceph-mon[35271]: pgmap v618: 129 pgs: 129 active+clean; 923 MiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 5.0 MiB/s wr, 343 op/s 2024-03-31T23:31:15.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:31:14 smithi142 ceph-mon[35271]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-31T23:31:15.293 INFO:tasks.workunit.client.0.smithi046.stdout: 10 60553 8.49 MB/sec execute 260 sec latency 134.848 ms 2024-03-31T23:31:16.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:31:15 smithi162 ceph-mon[36237]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:31:16.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:31:15 smithi162 ceph-mon[36237]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-31T23:31:16.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:31:15 smithi162 ceph-mon[36237]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' 2024-03-31T23:31:16.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:31:15 smithi046 ceph-mon[32498]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:31:16.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:31:15 smithi046 ceph-mon[32498]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-31T23:31:16.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:31:15 smithi046 ceph-mon[32498]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' 2024-03-31T23:31:16.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:31:15 smithi142 ceph-mon[35271]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:31:16.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:31:15 smithi142 ceph-mon[35271]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-31T23:31:16.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:31:15 smithi142 ceph-mon[35271]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' 2024-03-31T23:31:16.293 INFO:tasks.workunit.client.0.smithi046.stdout: 10 60661 8.47 MB/sec execute 261 sec latency 75.990 ms 2024-03-31T23:31:17.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:31:16 smithi162 ceph-mon[36237]: pgmap v619: 129 pgs: 129 active+clean; 942 MiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 6.2 MiB/s wr, 311 op/s 2024-03-31T23:31:17.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:31:16 smithi046 ceph-mon[32498]: pgmap v619: 129 pgs: 129 active+clean; 942 MiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 6.2 MiB/s wr, 311 op/s 2024-03-31T23:31:17.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:31:16 smithi142 ceph-mon[35271]: pgmap v619: 129 pgs: 129 active+clean; 942 MiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 6.2 MiB/s wr, 311 op/s 2024-03-31T23:31:17.293 INFO:tasks.workunit.client.0.smithi046.stdout: 10 60828 8.47 MB/sec execute 262 sec latency 81.510 ms 2024-03-31T23:31:18.293 INFO:tasks.workunit.client.0.smithi046.stdout: 10 60954 8.48 MB/sec execute 263 sec latency 296.877 ms 2024-03-31T23:31:19.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:31:18 smithi162 ceph-mon[36237]: pgmap v620: 129 pgs: 129 active+clean; 942 MiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 6.2 MiB/s wr, 311 op/s 2024-03-31T23:31:19.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:31:18 smithi162 ceph-mon[36237]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-03-31T23:31:19.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:31:18 smithi046 ceph-mon[32498]: pgmap v620: 129 pgs: 129 active+clean; 942 MiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 6.2 MiB/s wr, 311 op/s 2024-03-31T23:31:19.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:31:18 smithi046 ceph-mon[32498]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-03-31T23:31:19.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:31:18 smithi142 ceph-mon[35271]: pgmap v620: 129 pgs: 129 active+clean; 942 MiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 6.2 MiB/s wr, 311 op/s 2024-03-31T23:31:19.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:31:18 smithi142 ceph-mon[35271]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-03-31T23:31:19.293 INFO:tasks.workunit.client.0.smithi046.stdout: 10 61109 8.46 MB/sec execute 264 sec latency 97.834 ms 2024-03-31T23:31:20.293 INFO:tasks.workunit.client.0.smithi046.stdout: 10 61253 8.47 MB/sec execute 265 sec latency 64.837 ms 2024-03-31T23:31:21.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:31:20 smithi162 ceph-mon[36237]: pgmap v621: 129 pgs: 129 active+clean; 947 MiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 5.8 MiB/s wr, 347 op/s 2024-03-31T23:31:21.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:31:20 smithi046 ceph-mon[32498]: pgmap v621: 129 pgs: 129 active+clean; 947 MiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 5.8 MiB/s wr, 347 op/s 2024-03-31T23:31:21.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:31:20 smithi142 ceph-mon[35271]: pgmap v621: 129 pgs: 129 active+clean; 947 MiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 5.8 MiB/s wr, 347 op/s 2024-03-31T23:31:21.293 INFO:tasks.workunit.client.0.smithi046.stdout: 10 61385 8.47 MB/sec execute 266 sec latency 117.425 ms 2024-03-31T23:31:22.293 INFO:tasks.workunit.client.0.smithi046.stdout: 10 61523 8.47 MB/sec execute 267 sec latency 102.079 ms 2024-03-31T23:31:22.995 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:31:22 smithi142 ceph-mon[35271]: pgmap v622: 129 pgs: 129 active+clean; 947 MiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 5.6 MiB/s wr, 335 op/s 2024-03-31T23:31:23.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:31:22 smithi162 ceph-mon[36237]: pgmap v622: 129 pgs: 129 active+clean; 947 MiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 5.6 MiB/s wr, 335 op/s 2024-03-31T23:31:23.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:31:22 smithi046 ceph-mon[32498]: pgmap v622: 129 pgs: 129 active+clean; 947 MiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 5.6 MiB/s wr, 335 op/s 2024-03-31T23:31:23.293 INFO:tasks.workunit.client.0.smithi046.stdout: 10 61676 8.46 MB/sec execute 268 sec latency 127.139 ms 2024-03-31T23:31:24.293 INFO:tasks.workunit.client.0.smithi046.stdout: 10 61837 8.44 MB/sec execute 269 sec latency 82.797 ms 2024-03-31T23:31:25.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:31:24 smithi162 ceph-mon[36237]: pgmap v623: 129 pgs: 129 active+clean; 947 MiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 4.8 MiB/s wr, 289 op/s 2024-03-31T23:31:25.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:31:24 smithi046 ceph-mon[32498]: pgmap v623: 129 pgs: 129 active+clean; 947 MiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 4.8 MiB/s wr, 289 op/s 2024-03-31T23:31:25.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:31:24 smithi142 ceph-mon[35271]: pgmap v623: 129 pgs: 129 active+clean; 947 MiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 4.8 MiB/s wr, 289 op/s 2024-03-31T23:31:25.293 INFO:tasks.workunit.client.0.smithi046.stdout: 10 61949 8.42 MB/sec execute 270 sec latency 263.001 ms 2024-03-31T23:31:26.293 INFO:tasks.workunit.client.0.smithi046.stdout: 10 62093 8.41 MB/sec execute 271 sec latency 94.045 ms 2024-03-31T23:31:27.029 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:27.028+0000 7fbb39d1e640 1 -- 172.21.15.46:0/831247550 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbb3411af30 msgr2=0x7fbb3411d320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:31:27.029 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:27.028+0000 7fbb39d1e640 1 --2- 172.21.15.46:0/831247550 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbb3411af30 0x7fbb3411d320 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fbb2401a910 tx=0x7fbb2403cc60 comp rx=0 tx=0).stop 2024-03-31T23:31:27.032 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:27.032+0000 7fbb39d1e640 1 -- 172.21.15.46:0/831247550 shutdown_connections 2024-03-31T23:31:27.032 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:27.032+0000 7fbb39d1e640 1 --2- 172.21.15.46:0/831247550 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbb3411af30 0x7fbb3411d320 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:31:27.032 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:27.032+0000 7fbb39d1e640 1 --2- 172.21.15.46:0/831247550 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbb34110300 0x7fbb3410dff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:31:27.032 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:27.032+0000 7fbb39d1e640 1 --2- 172.21.15.46:0/831247550 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbb3410fe90 0x7fbb3410dab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:31:27.032 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:27.032+0000 7fbb39d1e640 1 -- 172.21.15.46:0/831247550 >> 172.21.15.46:0/831247550 conn(0x7fbb3406c7f0 msgr2=0x7fbb3406cc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:31:27.041 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:27.042+0000 7fbb39d1e640 1 -- 172.21.15.46:0/831247550 shutdown_connections 2024-03-31T23:31:27.042 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:27.042+0000 7fbb39d1e640 1 -- 172.21.15.46:0/831247550 wait complete. 2024-03-31T23:31:27.043 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:27.043+0000 7fbb39d1e640 1 Processor -- start 2024-03-31T23:31:27.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:31:26 smithi162 ceph-mon[36237]: pgmap v624: 129 pgs: 129 active+clean; 957 MiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 6.9 MiB/s wr, 407 op/s 2024-03-31T23:31:27.046 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:27.045+0000 7fbb39d1e640 1 -- start start 2024-03-31T23:31:27.046 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:27.045+0000 7fbb39d1e640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbb3410fe90 0x7fbb3411a490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:31:27.046 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:27.045+0000 7fbb39d1e640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbb34110300 0x7fbb3411a9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:31:27.046 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:27.045+0000 7fbb39d1e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbb3411af30 0x7fbb34116730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:31:27.046 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:27.045+0000 7fbb39d1e640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fbb34112f50 con 0x7fbb3411af30 2024-03-31T23:31:27.047 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:27.045+0000 7fbb39d1e640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fbb34112dd0 con 0x7fbb34110300 2024-03-31T23:31:27.047 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:27.045+0000 7fbb39d1e640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fbb341130d0 con 0x7fbb3410fe90 2024-03-31T23:31:27.047 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:27.046+0000 7fbb32ffd640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbb34110300 0x7fbb3411a9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:31:27.048 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:27.046+0000 7fbb337fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbb3410fe90 0x7fbb3411a490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:31:27.048 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:27.046+0000 7fbb32ffd640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbb34110300 0x7fbb3411a9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.46:53278/0 (socket says 172.21.15.46:53278) 2024-03-31T23:31:27.048 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:27.046+0000 7fbb32ffd640 1 -- 172.21.15.46:0/2856950726 learned_addr learned my addr 172.21.15.46:0/2856950726 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:31:27.048 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:27.047+0000 7fbb33fff640 1 --2- 172.21.15.46:0/2856950726 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbb3411af30 0x7fbb34116730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:31:27.048 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:27.048+0000 7fbb33fff640 1 -- 172.21.15.46:0/2856950726 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbb3410fe90 msgr2=0x7fbb3411a490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:31:27.048 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:27.048+0000 7fbb33fff640 1 --2- 172.21.15.46:0/2856950726 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbb3410fe90 0x7fbb3411a490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:31:27.048 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:27.048+0000 7fbb33fff640 1 -- 172.21.15.46:0/2856950726 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbb34110300 msgr2=0x7fbb3411a9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:31:27.048 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:27.048+0000 7fbb33fff640 1 --2- 172.21.15.46:0/2856950726 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbb34110300 0x7fbb3411a9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:31:27.048 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:27.048+0000 7fbb33fff640 1 -- 172.21.15.46:0/2856950726 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb34116ff0 con 0x7fbb3411af30 2024-03-31T23:31:27.049 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:27.048+0000 7fbb33fff640 1 --2- 172.21.15.46:0/2856950726 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbb3411af30 0x7fbb34116730 secure :-1 s=READY pgs=969 cs=0 l=1 rev1=1 crypto rx=0x7fbb24047040 tx=0x7fbb24004840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:31:27.049 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:27.049+0000 7fbb30ff9640 1 -- 172.21.15.46:0/2856950726 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb24052020 con 0x7fbb3411af30 2024-03-31T23:31:27.049 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:27.049+0000 7fbb30ff9640 1 -- 172.21.15.46:0/2856950726 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fbb24004c00 con 0x7fbb3411af30 2024-03-31T23:31:27.050 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:27.049+0000 7fbb30ff9640 1 -- 172.21.15.46:0/2856950726 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb24003d00 con 0x7fbb3411af30 2024-03-31T23:31:27.050 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:27.049+0000 7fbb39d1e640 1 -- 172.21.15.46:0/2856950726 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbb34117280 con 0x7fbb3411af30 2024-03-31T23:31:27.050 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:27.050+0000 7fbb39d1e640 1 -- 172.21.15.46:0/2856950726 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbb341aaeb0 con 0x7fbb3411af30 2024-03-31T23:31:27.055 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:27.055+0000 7fbb3c9db740 1 Processor -- start 2024-03-31T23:31:27.055 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:27.055+0000 7fbb3c9db740 1 -- start start 2024-03-31T23:31:27.055 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:27.055+0000 7fbb3c9db740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a41aba26c0 0x55a41aba2aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:31:27.055 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:27.055+0000 7fbb3c9db740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a41aba2fe0 0x55a41abab870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:31:27.055 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:27.055+0000 7fbb3c9db740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a41ababdb0 0x55a41abae1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:31:27.055 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:27.055+0000 7fbb3c9db740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55a41ab058f0 con 0x55a41aba2fe0 2024-03-31T23:31:27.055 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:27.055+0000 7fbb3c9db740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55a41aaae8d0 con 0x55a41aba26c0 2024-03-31T23:31:27.055 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:27.055+0000 7fbb3c9db740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55a41ab01990 con 0x55a41ababdb0 2024-03-31T23:31:27.056 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:27.055+0000 7fbb337fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a41aba2fe0 0x55a41abab870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:31:27.056 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:27.055+0000 7fbb337fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a41aba2fe0 0x55a41abab870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59954/0 (socket says 172.21.15.46:59954) 2024-03-31T23:31:27.056 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:27.055+0000 7fbb337fe640 1 -- 172.21.15.46:0/2847942265 learned_addr learned my addr 172.21.15.46:0/2847942265 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:31:27.056 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:27.056+0000 7fbb337fe640 1 -- 172.21.15.46:0/2847942265 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a41ababdb0 msgr2=0x55a41abae1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:31:27.056 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:27.056+0000 7fbb337fe640 1 --2- 172.21.15.46:0/2847942265 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a41ababdb0 0x55a41abae1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:31:27.056 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:27.056+0000 7fbb337fe640 1 -- 172.21.15.46:0/2847942265 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a41aba26c0 msgr2=0x55a41aba2aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:31:27.057 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:27.056+0000 7fbb337fe640 1 --2- 172.21.15.46:0/2847942265 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a41aba26c0 0x55a41aba2aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:31:27.057 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:27.056+0000 7fbb337fe640 1 -- 172.21.15.46:0/2847942265 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a41a81a210 con 0x55a41aba2fe0 2024-03-31T23:31:27.057 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:27.056+0000 7fbb337fe640 1 --2- 172.21.15.46:0/2847942265 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a41aba2fe0 0x55a41abab870 secure :-1 s=READY pgs=970 cs=0 l=1 rev1=1 crypto rx=0x7fbb2c009f60 tx=0x7fbb2c009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=202a981c1826f063 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:31:27.057 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:27.058+0000 7fbb19ffb640 1 -- 172.21.15.46:0/2847942265 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb2c016070 con 0x55a41aba2fe0 2024-03-31T23:31:27.057 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:27.058+0000 7fbb19ffb640 1 -- 172.21.15.46:0/2847942265 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fbb2c01bb10 con 0x55a41aba2fe0 2024-03-31T23:31:27.058 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:27.058+0000 7fbb19ffb640 1 -- 172.21.15.46:0/2847942265 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb2c012c10 con 0x55a41aba2fe0 2024-03-31T23:31:27.058 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:27.059+0000 7fbb3c9db740 1 -- 172.21.15.46:0/2847942265 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a41aba2fe0 msgr2=0x55a41abab870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:31:27.058 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:27.059+0000 7fbb3c9db740 1 --2- 172.21.15.46:0/2847942265 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a41aba2fe0 0x55a41abab870 secure :-1 s=READY pgs=970 cs=0 l=1 rev1=1 crypto rx=0x7fbb2c009f60 tx=0x7fbb2c009f90 comp rx=0 tx=0).stop 2024-03-31T23:31:27.060 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:27.059+0000 7fbb30ff9640 1 -- 172.21.15.46:0/2856950726 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 25) ==== 137451+0+0 (secure 0 0 0) 0x7fbb24055020 con 0x7fbb3411af30 2024-03-31T23:31:27.060 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:27.059+0000 7fbb3c9db740 1 -- 172.21.15.46:0/2847942265 shutdown_connections 2024-03-31T23:31:27.060 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:27.059+0000 7fbb3c9db740 1 --2- 172.21.15.46:0/2847942265 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a41ababdb0 0x55a41abae1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:31:27.060 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:27.059+0000 7fbb3c9db740 1 --2- 172.21.15.46:0/2847942265 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a41aba2fe0 0x55a41abab870 unknown :-1 s=CLOSED pgs=970 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:31:27.060 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:27.059+0000 7fbb3c9db740 1 --2- 172.21.15.46:0/2847942265 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a41aba26c0 0x55a41aba2aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:31:27.060 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:27.059+0000 7fbb3c9db740 1 -- 172.21.15.46:0/2847942265 >> 172.21.15.46:0/2847942265 conn(0x55a41ab9cfe0 msgr2=0x55a41ab9d3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:31:27.060 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:27.059+0000 7fbb3c9db740 1 -- 172.21.15.46:0/2847942265 shutdown_connections 2024-03-31T23:31:27.060 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:27.059+0000 7fbb30ff9640 1 --2- 172.21.15.46:0/2856950726 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fbb1c0a2790 0x7fbb1c0a4c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:31:27.061 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:27.060+0000 7fbb3c9db740 1 -- 172.21.15.46:0/2847942265 wait complete. 2024-03-31T23:31:27.061 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:27.060+0000 7fbb30ff9640 1 -- 172.21.15.46:0/2856950726 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(126..126 src has 1..126) ==== 9554+0+0 (secure 0 0 0) 0x7fbb24101c00 con 0x7fbb3411af30 2024-03-31T23:31:27.061 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:27.060+0000 7fbb337fe640 1 --2- 172.21.15.46:0/2856950726 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fbb1c0a2790 0x7fbb1c0a4c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:31:27.061 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:27.060+0000 7fbb3c9db740 1 -- start start 2024-03-31T23:31:27.061 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:27.060+0000 7fbb3c9db740 1 Processor -- start 2024-03-31T23:31:27.062 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:27.061+0000 7fbb337fe640 1 --2- 172.21.15.46:0/2856950726 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fbb1c0a2790 0x7fbb1c0a4c50 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fbb2c012790 tx=0x7fbb2c012560 comp rx=0 tx=0).ready entity=mgr.15654 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:31:27.062 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:27.061+0000 7fbb3c9db740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a41aba26c0 0x55a41aba8e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:31:27.062 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:27.061+0000 7fbb3c9db740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a41aba2fe0 0x55a41aba9370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:31:27.063 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:27.061+0000 7fbb337fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a41aba26c0 0x55a41aba8e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:31:27.063 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:27.061+0000 7fbb3c9db740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a41ababdb0 0x55a41aba98b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:31:27.063 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:27.061+0000 7fbb3c9db740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55a41abae9e0 con 0x55a41aba26c0 2024-03-31T23:31:27.063 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:27.061+0000 7fbb3c9db740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55a41aaa3f80 con 0x55a41aba2fe0 2024-03-31T23:31:27.063 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:27.061+0000 7fbb3c9db740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55a41abaeb60 con 0x55a41ababdb0 2024-03-31T23:31:27.063 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:27.061+0000 7fbb337fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a41aba26c0 0x55a41aba8e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59966/0 (socket says 172.21.15.46:59966) 2024-03-31T23:31:27.063 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:27.061+0000 7fbb337fe640 1 -- 172.21.15.46:0/1488925841 learned_addr learned my addr 172.21.15.46:0/1488925841 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:31:27.063 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:27.061+0000 7fbb3c9db740 10 client.0 _ll_register_callbacks cb 0x55a41aafc290 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T23:31:27.063 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:27.061+0000 7fbb32ffd640 1 --2- 172.21.15.46:0/1488925841 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a41aba2fe0 0x55a41aba9370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:31:27.063 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:27.061+0000 7fbb33fff640 1 --2- 172.21.15.46:0/1488925841 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a41ababdb0 0x55a41aba98b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:31:27.063 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:27.062+0000 7fbb337fe640 1 -- 172.21.15.46:0/1488925841 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a41ababdb0 msgr2=0x55a41aba98b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:31:27.063 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:27.062+0000 7fbb337fe640 1 --2- 172.21.15.46:0/1488925841 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a41ababdb0 0x55a41aba98b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:31:27.063 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:27.062+0000 7fbb337fe640 1 -- 172.21.15.46:0/1488925841 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a41aba2fe0 msgr2=0x55a41aba9370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:31:27.064 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:27.062+0000 7fbb337fe640 1 --2- 172.21.15.46:0/1488925841 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a41aba2fe0 0x55a41aba9370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:31:27.064 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:27.062+0000 7fbb337fe640 1 -- 172.21.15.46:0/1488925841 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a41a8e6620 con 0x55a41aba26c0 2024-03-31T23:31:27.064 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:27.062+0000 7fbb33fff640 1 --2- 172.21.15.46:0/1488925841 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a41ababdb0 0x55a41aba98b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:31:27.064 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:27.062+0000 7fbb32ffd640 1 --2- 172.21.15.46:0/1488925841 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a41aba2fe0 0x55a41aba9370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:31:27.064 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:27.062+0000 7fbb337fe640 1 --2- 172.21.15.46:0/1488925841 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a41aba26c0 0x55a41aba8e30 secure :-1 s=READY pgs=971 cs=0 l=1 rev1=1 crypto rx=0x55a41aba2170 tx=0x7fbb2c009860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea8892e12ea75abf server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:31:27.064 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:27.062+0000 7fbb12ffd640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-31T23:31:27.064 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:27.062+0000 7fbb12ffd640 1 -- 172.21.15.46:0/1488925841 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb2c016040 con 0x55a41aba26c0 2024-03-31T23:31:27.064 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:27.062+0000 7fbb12ffd640 1 -- 172.21.15.46:0/1488925841 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(126..126 src has 1..126) ==== 9554+0+0 (secure 0 0 0) 0x7fbb2c003970 con 0x55a41aba26c0 2024-03-31T23:31:27.064 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:27.062+0000 7fbb3c9db740 1 -- 172.21.15.46:0/1488925841 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbb2c01bb10 con 0x55a41aba26c0 2024-03-31T23:31:27.064 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:27.063+0000 7fbb12ffd640 1 -- 172.21.15.46:0/1488925841 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=48) ==== 24+0+0 (secure 0 0 0) 0x7fbb2c003b70 con 0x55a41aba26c0 2024-03-31T23:31:27.064 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:27.063+0000 7fbb3c9db740 10 client.17742 fetch_fsmap learned FSMap version 48 2024-03-31T23:31:27.064 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:27.063+0000 7fbb3c9db740 1 -- 172.21.15.46:0/1488925841 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=48,monmap=4+}) -- 0x55a41abba900 con 0x55a41aba26c0 2024-03-31T23:31:27.065 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:27.063+0000 7fbb12ffd640 1 -- 172.21.15.46:0/1488925841 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fbb2c01bb10 con 0x55a41aba26c0 2024-03-31T23:31:27.065 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:27.063+0000 7fbb12ffd640 1 -- 172.21.15.46:0/1488925841 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 48) ==== 8350+0+0 (secure 0 0 0) 0x7fbb2c00f040 con 0x55a41aba26c0 2024-03-31T23:31:27.065 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:27.063+0000 7fbb3c9db740 10 client.17742 fetch_fsmap finished waiting for FSMap version 48 2024-03-31T23:31:27.065 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:27.063+0000 7fbb3c9db740 10 client.17742 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-31T23:31:27.066 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:27.063+0000 7fbb3c9db740 20 client.17742 populate_metadata read hostname 'smithi046' 2024-03-31T23:31:27.066 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:27.063+0000 7fbb3c9db740 1 --2- 172.21.15.46:0/1488925841 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55a41abbbd60 0x55a41abbe190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:31:27.066 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:27.063+0000 7fbb3c9db740 4 client.17742 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-31T23:31:27.066 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:27.063+0000 7fbb3c9db740 1 -- 172.21.15.46:0/1488925841 --> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a41ab19120 con 0x55a41abbbd60 2024-03-31T23:31:27.066 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:27.063+0000 7fbb32ffd640 1 --2- 172.21.15.46:0/1488925841 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55a41abbbd60 0x55a41abbe190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:31:27.066 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:27.064+0000 7fbb12ffd640 10 client.17742 ms_handle_connect on v2:172.21.15.142:6838/2519143918 2024-03-31T23:31:27.066 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:27.064+0000 7fbb32ffd640 1 --2- 172.21.15.46:0/1488925841 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55a41abbbd60 0x55a41abbe190 crc :-1 s=READY pgs=252 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=946e2521c4d985a0 server_cookie=b2f62ef73534fece in_seq=0 out_seq=0 2024-03-31T23:31:27.067 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:27.066+0000 7fbb12ffd640 1 -- 172.21.15.46:0/1488925841 <== mds.0 v2:172.21.15.142:6838/2519143918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55a41ab19120 con 0x55a41abbbd60 2024-03-31T23:31:27.067 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:27.066+0000 7fbb12ffd640 10 client.17742 handle_command_reply: tid=0 2024-03-31T23:31:27.068 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:27.066+0000 7fbb3c9db740 1 client.17742 shutdown 2024-03-31T23:31:27.068 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:27.066+0000 7fbb3c9db740 1 -- 172.21.15.46:0/1488925841 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a41aba26c0 msgr2=0x55a41aba8e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:31:27.068 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:27.066+0000 7fbb3c9db740 1 --2- 172.21.15.46:0/1488925841 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a41aba26c0 0x55a41aba8e30 secure :-1 s=READY pgs=971 cs=0 l=1 rev1=1 crypto rx=0x55a41aba2170 tx=0x7fbb2c009860 comp rx=0 tx=0).stop 2024-03-31T23:31:27.068 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:27.066+0000 7fbb3c9db740 1 -- 172.21.15.46:0/1488925841 shutdown_connections 2024-03-31T23:31:27.068 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:27.066+0000 7fbb3c9db740 1 --2- 172.21.15.46:0/1488925841 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55a41abbbd60 0x55a41abbe190 crc :-1 s=READY pgs=252 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:31:27.068 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:27.066+0000 7fbb3c9db740 1 --2- 172.21.15.46:0/1488925841 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a41ababdb0 0x55a41aba98b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:31:27.068 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:27.066+0000 7fbb12ffd640 0 client.17742 ms_handle_reset on v2:172.21.15.142:6838/2519143918 2024-03-31T23:31:27.068 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:27.067+0000 7fbb3c9db740 1 --2- 172.21.15.46:0/1488925841 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a41aba2fe0 0x55a41aba9370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:31:27.068 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:27.067+0000 7fbb3c9db740 1 --2- 172.21.15.46:0/1488925841 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a41aba26c0 0x55a41aba8e30 unknown :-1 s=CLOSED pgs=971 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:31:27.068 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:27.067+0000 7fbb3c9db740 1 -- 172.21.15.46:0/1488925841 >> 172.21.15.46:0/1488925841 conn(0x55a41ab9cfe0 msgr2=0x55a41ab9ed30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:31:27.068 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:27.067+0000 7fbb3c9db740 1 -- 172.21.15.46:0/1488925841 shutdown_connections 2024-03-31T23:31:27.068 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:27.067+0000 7fbb3c9db740 1 -- 172.21.15.46:0/1488925841 wait complete. 2024-03-31T23:31:27.068 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:27.067+0000 7fbb3c9db740 20 client.17742 trim_cache size 0 max 16384 2024-03-31T23:31:27.092 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:27.092+0000 7fbb3c9db740 1 Processor -- start 2024-03-31T23:31:27.092 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:27.092+0000 7fbb3c9db740 1 -- start start 2024-03-31T23:31:27.093 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:27.092+0000 7fbb3c9db740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a41aba26c0 0x55a41abb7780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:31:27.094 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:27.092+0000 7fbb3c9db740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a41aba2fe0 0x55a41aba35f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:31:27.094 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:27.092+0000 7fbb337fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a41aba26c0 0x55a41abb7780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:31:27.094 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:27.092+0000 7fbb337fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a41aba26c0 0x55a41abb7780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59974/0 (socket says 172.21.15.46:59974) 2024-03-31T23:31:27.094 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:27.092+0000 7fbb337fe640 1 -- 172.21.15.46:0/2796249569 learned_addr learned my addr 172.21.15.46:0/2796249569 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:31:27.094 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:27.092+0000 7fbb3c9db740 1 --2- 172.21.15.46:0/2796249569 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a41ababdb0 0x55a41abc3a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:31:27.094 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:27.093+0000 7fbb32ffd640 1 --2- 172.21.15.46:0/2796249569 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a41aba2fe0 0x55a41aba35f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:31:27.094 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:27.093+0000 7fbb3c9db740 1 -- 172.21.15.46:0/2796249569 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fbb341b7b40 con 0x55a41aba26c0 2024-03-31T23:31:27.094 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:27.093+0000 7fbb3c9db740 1 -- 172.21.15.46:0/2796249569 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fbb341b53e0 con 0x55a41ababdb0 2024-03-31T23:31:27.094 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:27.093+0000 7fbb3c9db740 1 -- 172.21.15.46:0/2796249569 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fbb341b3250 con 0x55a41aba2fe0 2024-03-31T23:31:27.094 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:27.093+0000 7fbb33fff640 1 --2- 172.21.15.46:0/2796249569 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a41ababdb0 0x55a41abc3a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:31:27.094 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:27.093+0000 7fbb32ffd640 1 -- 172.21.15.46:0/2796249569 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a41ababdb0 msgr2=0x55a41abc3a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:31:27.094 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:27.093+0000 7fbb32ffd640 1 --2- 172.21.15.46:0/2796249569 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a41ababdb0 0x55a41abc3a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:31:27.094 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:27.093+0000 7fbb32ffd640 1 -- 172.21.15.46:0/2796249569 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a41aba26c0 msgr2=0x55a41abb7780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:31:27.094 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:27.093+0000 7fbb32ffd640 1 --2- 172.21.15.46:0/2796249569 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a41aba26c0 0x55a41abb7780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:31:27.095 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:27.093+0000 7fbb32ffd640 1 -- 172.21.15.46:0/2796249569 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a41aba24f0 con 0x55a41aba2fe0 2024-03-31T23:31:27.095 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:27.093+0000 7fbb337fe640 1 --2- 172.21.15.46:0/2796249569 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a41aba26c0 0x55a41abb7780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:31:27.095 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:27.093+0000 7fbb32ffd640 1 --2- 172.21.15.46:0/2796249569 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a41aba2fe0 0x55a41aba35f0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fbb28005d80 tx=0x7fbb2800bc00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=62f85aeab251f50c server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:31:27.095 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:27.094+0000 7fbb197fa640 1 -- 172.21.15.46:0/2796249569 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb28004bf0 con 0x55a41aba2fe0 2024-03-31T23:31:27.095 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:27.094+0000 7fbb197fa640 1 -- 172.21.15.46:0/2796249569 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fbb28004d90 con 0x55a41aba2fe0 2024-03-31T23:31:27.095 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:27.094+0000 7fbb197fa640 1 -- 172.21.15.46:0/2796249569 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb28011aa0 con 0x55a41aba2fe0 2024-03-31T23:31:27.097 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:27.094+0000 7fbb3c9db740 1 -- 172.21.15.46:0/2796249569 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a41aba2fe0 msgr2=0x55a41aba35f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:31:27.098 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:27.095+0000 7fbb3c9db740 1 --2- 172.21.15.46:0/2796249569 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a41aba2fe0 0x55a41aba35f0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fbb28005d80 tx=0x7fbb2800bc00 comp rx=0 tx=0).stop 2024-03-31T23:31:27.098 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:27.095+0000 7fbb3c9db740 1 -- 172.21.15.46:0/2796249569 shutdown_connections 2024-03-31T23:31:27.098 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:27.095+0000 7fbb3c9db740 1 --2- 172.21.15.46:0/2796249569 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a41ababdb0 0x55a41abc3a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:31:27.098 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:27.095+0000 7fbb3c9db740 1 --2- 172.21.15.46:0/2796249569 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a41aba2fe0 0x55a41aba35f0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:31:27.098 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:27.095+0000 7fbb3c9db740 1 --2- 172.21.15.46:0/2796249569 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a41aba26c0 0x55a41abb7780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:31:27.098 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:27.095+0000 7fbb3c9db740 1 -- 172.21.15.46:0/2796249569 >> 172.21.15.46:0/2796249569 conn(0x55a41ab9cfe0 msgr2=0x55a41ab9d3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:31:27.098 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:27.095+0000 7fbb3c9db740 1 -- 172.21.15.46:0/2796249569 shutdown_connections 2024-03-31T23:31:27.098 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:27.095+0000 7fbb3c9db740 1 -- 172.21.15.46:0/2796249569 wait complete. 2024-03-31T23:31:27.098 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:27.095+0000 7fbb3c9db740 1 -- start start 2024-03-31T23:31:27.098 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:27.095+0000 7fbb3c9db740 1 Processor -- start 2024-03-31T23:31:27.098 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:27.096+0000 7fbb3c9db740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a41aba26c0 0x55a41abb4110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:31:27.098 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:27.096+0000 7fbb3c9db740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a41aba2fe0 0x55a41aba5740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:31:27.098 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:27.096+0000 7fbb337fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a41aba26c0 0x55a41abb4110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:31:27.099 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:27.096+0000 7fbb3c9db740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a41ababdb0 0x55a41aba5ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:31:27.099 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:27.096+0000 7fbb32ffd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a41aba2fe0 0x55a41aba5740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:31:27.099 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:27.096+0000 7fbb337fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a41aba26c0 0x55a41abb4110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59984/0 (socket says 172.21.15.46:59984) 2024-03-31T23:31:27.099 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:27.096+0000 7fbb32ffd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a41aba2fe0 0x55a41aba5740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.46:44350/0 (socket says 172.21.15.46:44350) 2024-03-31T23:31:27.099 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:27.096+0000 7fbb3c9db740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fbb341b38a0 con 0x55a41aba26c0 2024-03-31T23:31:27.099 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:27.096+0000 7fbb3c9db740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fbb341b4d10 con 0x55a41ababdb0 2024-03-31T23:31:27.099 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:27.096+0000 7fbb3c9db740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fbb341b3f70 con 0x55a41aba2fe0 2024-03-31T23:31:27.099 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:27.096+0000 7fbb33fff640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a41ababdb0 0x55a41aba5ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:31:27.099 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:27.096+0000 7fbb337fe640 1 -- 172.21.15.46:0/280588598 learned_addr learned my addr 172.21.15.46:0/280588598 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:31:27.099 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:27.096+0000 7fbb3c9db740 10 client.0 _ll_register_callbacks cb 0x7fbb2c009de0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T23:31:27.099 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:27.096+0000 7fbb33fff640 1 -- 172.21.15.46:0/280588598 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a41aba2fe0 msgr2=0x55a41aba5740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:31:27.099 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:27.097+0000 7fbb33fff640 1 --2- 172.21.15.46:0/280588598 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a41aba2fe0 0x55a41aba5740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:31:27.099 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:27.097+0000 7fbb33fff640 1 -- 172.21.15.46:0/280588598 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a41aba26c0 msgr2=0x55a41abb4110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:31:27.100 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:27.097+0000 7fbb33fff640 1 --2- 172.21.15.46:0/280588598 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a41aba26c0 0x55a41abb4110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:31:27.100 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:27.097+0000 7fbb33fff640 1 -- 172.21.15.46:0/280588598 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a41abb7cc0 con 0x55a41ababdb0 2024-03-31T23:31:27.100 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:27.097+0000 7fbb32ffd640 1 --2- 172.21.15.46:0/280588598 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a41aba2fe0 0x55a41aba5740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:31:27.100 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:27.097+0000 7fbb33fff640 1 --2- 172.21.15.46:0/280588598 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a41ababdb0 0x55a41aba5ce0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fbb240ff050 tx=0x7fbb2403cc60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bdae5e438ea6db6a server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:31:27.100 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:27.097+0000 7fbb12ffd640 10 client.0 ms_handle_connect on v2:172.21.15.142:3300/0 2024-03-31T23:31:27.100 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:27.097+0000 7fbb12ffd640 1 -- 172.21.15.46:0/280588598 <== mon.2 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb240cc1c0 con 0x55a41ababdb0 2024-03-31T23:31:27.100 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:27.097+0000 7fbb12ffd640 1 -- 172.21.15.46:0/280588598 <== mon.2 v2:172.21.15.142:3300/0 2 ==== osd_map(126..126 src has 1..126) ==== 9554+0+0 (secure 0 0 0) 0x7fbb240c8b60 con 0x55a41ababdb0 2024-03-31T23:31:27.100 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:27.097+0000 7fbb3c9db740 1 -- 172.21.15.46:0/280588598 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbb28004d90 con 0x55a41ababdb0 2024-03-31T23:31:27.100 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:27.098+0000 7fbb12ffd640 1 -- 172.21.15.46:0/280588598 <== mon.2 v2:172.21.15.142:3300/0 3 ==== mon_get_version_reply(handle=1 version=48) ==== 24+0+0 (secure 0 0 0) 0x7fbb2403d810 con 0x55a41ababdb0 2024-03-31T23:31:27.100 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:27.098+0000 7fbb3c9db740 10 client.27575 fetch_fsmap learned FSMap version 48 2024-03-31T23:31:27.100 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:27.098+0000 7fbb3c9db740 1 -- 172.21.15.46:0/280588598 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,fsmap=48,monmap=4+}) -- 0x55a41ac21fb0 con 0x55a41ababdb0 2024-03-31T23:31:27.100 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:27.098+0000 7fbb12ffd640 1 -- 172.21.15.46:0/280588598 <== mon.2 v2:172.21.15.142:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fbb240ce780 con 0x55a41ababdb0 2024-03-31T23:31:27.101 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:27.098+0000 7fbb12ffd640 1 -- 172.21.15.46:0/280588598 <== mon.2 v2:172.21.15.142:3300/0 5 ==== fsmap(e 48) ==== 8350+0+0 (secure 0 0 0) 0x7fbb240d6ce0 con 0x55a41ababdb0 2024-03-31T23:31:27.102 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:27.098+0000 7fbb3c9db740 10 client.27575 fetch_fsmap finished waiting for FSMap version 48 2024-03-31T23:31:27.102 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:27.098+0000 7fbb3c9db740 10 client.27575 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-31T23:31:27.102 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:27.098+0000 7fbb3c9db740 20 client.27575 populate_metadata read hostname 'smithi046' 2024-03-31T23:31:27.102 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:27.098+0000 7fbb3c9db740 1 --2- 172.21.15.46:0/280588598 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55a41abbbd60 0x55a41ac246a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:31:27.102 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:27.098+0000 7fbb3c9db740 4 client.27575 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-31T23:31:27.102 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:27.098+0000 7fbb3c9db740 1 -- 172.21.15.46:0/280588598 --> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] -- command(tid 0: {"prefix": "flush journal"}) -- 0x7fbb28004d90 con 0x55a41abbbd60 2024-03-31T23:31:27.102 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:27.099+0000 7fbb337fe640 1 --2- 172.21.15.46:0/280588598 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55a41abbbd60 0x55a41ac246a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:31:27.102 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:27.099+0000 7fbb337fe640 1 --2- 172.21.15.46:0/280588598 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55a41abbbd60 0x55a41ac246a0 crc :-1 s=READY pgs=253 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3090c4299e3aa59c server_cookie=4740be0c39f86327 in_seq=0 out_seq=0 2024-03-31T23:31:27.102 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:27.099+0000 7fbb12ffd640 10 client.27575 ms_handle_connect on v2:172.21.15.142:6838/2519143918 2024-03-31T23:31:27.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:31:26 smithi046 ceph-mon[32498]: pgmap v624: 129 pgs: 129 active+clean; 957 MiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 6.9 MiB/s wr, 407 op/s 2024-03-31T23:31:27.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:31:26 smithi142 ceph-mon[35271]: pgmap v624: 129 pgs: 129 active+clean; 957 MiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 6.9 MiB/s wr, 407 op/s 2024-03-31T23:31:27.293 INFO:tasks.workunit.client.0.smithi046.stdout: 10 62235 8.41 MB/sec execute 272 sec latency 150.883 ms 2024-03-31T23:31:27.300 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:27.299+0000 7fbb12ffd640 1 -- 172.21.15.46:0/280588598 <== mds.0 v2:172.21.15.142:6838/2519143918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x7fbb28004d90 con 0x55a41abbbd60 2024-03-31T23:31:27.300 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:27.299+0000 7fbb12ffd640 10 client.27575 handle_command_reply: tid=0 2024-03-31T23:31:27.300 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:27.299+0000 7fbb3c9db740 1 client.27575 shutdown 2024-03-31T23:31:27.300 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:27.299+0000 7fbb3c9db740 1 -- 172.21.15.46:0/280588598 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a41ababdb0 msgr2=0x55a41aba5ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:31:27.300 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:27.299+0000 7fbb3c9db740 1 --2- 172.21.15.46:0/280588598 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a41ababdb0 0x55a41aba5ce0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fbb240ff050 tx=0x7fbb2403cc60 comp rx=0 tx=0).stop 2024-03-31T23:31:27.300 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:27.299+0000 7fbb3c9db740 1 -- 172.21.15.46:0/280588598 shutdown_connections 2024-03-31T23:31:27.300 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:27.299+0000 7fbb3c9db740 1 --2- 172.21.15.46:0/280588598 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55a41abbbd60 0x55a41ac246a0 crc :-1 s=READY pgs=253 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:31:27.300 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:27.299+0000 7fbb3c9db740 1 --2- 172.21.15.46:0/280588598 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55a41ababdb0 0x55a41aba5ce0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:31:27.300 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:27.299+0000 7fbb3c9db740 1 --2- 172.21.15.46:0/280588598 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a41aba2fe0 0x55a41aba5740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:31:27.300 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:27.299+0000 7fbb3c9db740 1 --2- 172.21.15.46:0/280588598 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a41aba26c0 0x55a41abb4110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:31:27.301 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:27.299+0000 7fbb3c9db740 1 -- 172.21.15.46:0/280588598 >> 172.21.15.46:0/280588598 conn(0x55a41ab9cfe0 msgr2=0x55a41aba35f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:31:27.301 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:27.299+0000 7fbb12ffd640 0 client.27575 ms_handle_reset on v2:172.21.15.142:6838/2519143918 2024-03-31T23:31:27.301 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:27.300+0000 7fbb3c9db740 1 -- 172.21.15.46:0/280588598 shutdown_connections 2024-03-31T23:31:27.301 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:27.300+0000 7fbb3c9db740 1 -- 172.21.15.46:0/280588598 wait complete. 2024-03-31T23:31:27.301 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:27.300+0000 7fbb3c9db740 20 client.27575 trim_cache size 0 max 16384 2024-03-31T23:31:27.301 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout:{ 2024-03-31T23:31:27.301 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout: "message": "", 2024-03-31T23:31:27.301 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout: "return_code": 0 2024-03-31T23:31:27.301 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout:} 2024-03-31T23:31:27.302 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:27.301+0000 7fbb39d1e640 1 -- 172.21.15.46:0/2856950726 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fbb1c0a2790 msgr2=0x7fbb1c0a4c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:31:27.302 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:27.301+0000 7fbb39d1e640 1 --2- 172.21.15.46:0/2856950726 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fbb1c0a2790 0x7fbb1c0a4c50 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fbb2c012790 tx=0x7fbb2c012560 comp rx=0 tx=0).stop 2024-03-31T23:31:27.302 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:27.301+0000 7fbb39d1e640 1 -- 172.21.15.46:0/2856950726 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbb3411af30 msgr2=0x7fbb34116730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:31:27.302 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:27.301+0000 7fbb39d1e640 1 --2- 172.21.15.46:0/2856950726 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbb3411af30 0x7fbb34116730 secure :-1 s=READY pgs=969 cs=0 l=1 rev1=1 crypto rx=0x7fbb24047040 tx=0x7fbb24004840 comp rx=0 tx=0).stop 2024-03-31T23:31:27.302 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:27.303+0000 7fbb39d1e640 1 -- 172.21.15.46:0/2856950726 shutdown_connections 2024-03-31T23:31:27.303 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:27.303+0000 7fbb39d1e640 1 --2- 172.21.15.46:0/2856950726 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fbb1c0a2790 0x7fbb1c0a4c50 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:31:27.303 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:27.303+0000 7fbb39d1e640 1 --2- 172.21.15.46:0/2856950726 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbb3411af30 0x7fbb34116730 unknown :-1 s=CLOSED pgs=969 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:31:27.303 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:27.303+0000 7fbb39d1e640 1 --2- 172.21.15.46:0/2856950726 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbb34110300 0x7fbb3411a9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:31:27.303 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:27.303+0000 7fbb39d1e640 1 --2- 172.21.15.46:0/2856950726 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbb3410fe90 0x7fbb3411a490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:31:27.303 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:27.303+0000 7fbb39d1e640 1 -- 172.21.15.46:0/2856950726 >> 172.21.15.46:0/2856950726 conn(0x7fbb3406c7f0 msgr2=0x7fbb3411bdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:31:27.304 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:27.304+0000 7fbb39d1e640 1 -- 172.21.15.46:0/2856950726 shutdown_connections 2024-03-31T23:31:27.304 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:27.304+0000 7fbb39d1e640 1 -- 172.21.15.46:0/2856950726 wait complete. 2024-03-31T23:31:28.293 INFO:tasks.workunit.client.0.smithi046.stdout: 10 62418 8.43 MB/sec execute 273 sec latency 73.868 ms 2024-03-31T23:31:29.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:31:28 smithi162 ceph-mon[36237]: pgmap v625: 129 pgs: 129 active+clean; 957 MiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 4.2 MiB/s wr, 255 op/s 2024-03-31T23:31:29.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:31:28 smithi046 ceph-mon[32498]: pgmap v625: 129 pgs: 129 active+clean; 957 MiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 4.2 MiB/s wr, 255 op/s 2024-03-31T23:31:29.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:31:28 smithi142 ceph-mon[35271]: pgmap v625: 129 pgs: 129 active+clean; 957 MiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 4.2 MiB/s wr, 255 op/s 2024-03-31T23:31:29.293 INFO:tasks.workunit.client.0.smithi046.stdout: 10 62595 8.44 MB/sec execute 274 sec latency 81.902 ms 2024-03-31T23:31:30.293 INFO:tasks.workunit.client.0.smithi046.stdout: 10 62780 8.45 MB/sec execute 275 sec latency 77.559 ms 2024-03-31T23:31:31.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:31:30 smithi162 ceph-mon[36237]: pgmap v626: 129 pgs: 129 active+clean; 956 MiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 6.3 MiB/s wr, 415 op/s 2024-03-31T23:31:31.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:31:30 smithi046 ceph-mon[32498]: pgmap v626: 129 pgs: 129 active+clean; 956 MiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 6.3 MiB/s wr, 415 op/s 2024-03-31T23:31:31.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:31:30 smithi142 ceph-mon[35271]: pgmap v626: 129 pgs: 129 active+clean; 956 MiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 6.3 MiB/s wr, 415 op/s 2024-03-31T23:31:31.293 INFO:tasks.workunit.client.0.smithi046.stdout: 10 62928 8.44 MB/sec execute 276 sec latency 111.311 ms 2024-03-31T23:31:32.293 INFO:tasks.workunit.client.0.smithi046.stdout: 10 63072 8.44 MB/sec execute 277 sec latency 98.832 ms 2024-03-31T23:31:33.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:31:32 smithi162 ceph-mon[36237]: pgmap v627: 129 pgs: 129 active+clean; 956 MiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 4.3 MiB/s wr, 277 op/s 2024-03-31T23:31:33.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:31:32 smithi046 ceph-mon[32498]: pgmap v627: 129 pgs: 129 active+clean; 956 MiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 4.3 MiB/s wr, 277 op/s 2024-03-31T23:31:33.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:31:32 smithi142 ceph-mon[35271]: pgmap v627: 129 pgs: 129 active+clean; 956 MiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 4.3 MiB/s wr, 277 op/s 2024-03-31T23:31:33.294 INFO:tasks.workunit.client.0.smithi046.stdout: 10 63234 8.46 MB/sec execute 278 sec latency 95.127 ms 2024-03-31T23:31:34.293 INFO:tasks.workunit.client.0.smithi046.stdout: 10 63402 8.46 MB/sec execute 279 sec latency 69.647 ms 2024-03-31T23:31:35.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:31:34 smithi162 ceph-mon[36237]: pgmap v628: 129 pgs: 129 active+clean; 956 MiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 4.3 MiB/s wr, 277 op/s 2024-03-31T23:31:35.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:31:34 smithi046 ceph-mon[32498]: pgmap v628: 129 pgs: 129 active+clean; 956 MiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 4.3 MiB/s wr, 277 op/s 2024-03-31T23:31:35.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:31:34 smithi142 ceph-mon[35271]: pgmap v628: 129 pgs: 129 active+clean; 956 MiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 4.3 MiB/s wr, 277 op/s 2024-03-31T23:31:35.294 INFO:tasks.workunit.client.0.smithi046.stdout: 10 63551 8.46 MB/sec execute 280 sec latency 81.540 ms 2024-03-31T23:31:36.294 INFO:tasks.workunit.client.0.smithi046.stdout: 10 63720 8.46 MB/sec execute 281 sec latency 114.803 ms 2024-03-31T23:31:37.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:31:36 smithi162 ceph-mon[36237]: pgmap v629: 129 pgs: 129 active+clean; 970 MiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 6.6 MiB/s wr, 381 op/s 2024-03-31T23:31:37.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:31:36 smithi046 ceph-mon[32498]: pgmap v629: 129 pgs: 129 active+clean; 970 MiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 6.6 MiB/s wr, 381 op/s 2024-03-31T23:31:37.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:31:36 smithi142 ceph-mon[35271]: pgmap v629: 129 pgs: 129 active+clean; 970 MiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 6.6 MiB/s wr, 381 op/s 2024-03-31T23:31:37.294 INFO:tasks.workunit.client.0.smithi046.stdout: 10 63878 8.45 MB/sec execute 282 sec latency 69.414 ms 2024-03-31T23:31:38.294 INFO:tasks.workunit.client.0.smithi046.stdout: 10 64003 8.44 MB/sec execute 283 sec latency 80.960 ms 2024-03-31T23:31:39.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:31:38 smithi162 ceph-mon[36237]: pgmap v630: 129 pgs: 129 active+clean; 970 MiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 4.5 MiB/s wr, 263 op/s 2024-03-31T23:31:39.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:31:38 smithi046 ceph-mon[32498]: pgmap v630: 129 pgs: 129 active+clean; 970 MiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 4.5 MiB/s wr, 263 op/s 2024-03-31T23:31:39.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:31:38 smithi142 ceph-mon[35271]: pgmap v630: 129 pgs: 129 active+clean; 970 MiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 4.5 MiB/s wr, 263 op/s 2024-03-31T23:31:39.294 INFO:tasks.workunit.client.0.smithi046.stdout: 10 64126 8.44 MB/sec execute 284 sec latency 147.928 ms 2024-03-31T23:31:40.294 INFO:tasks.workunit.client.0.smithi046.stdout: 10 64275 8.44 MB/sec execute 285 sec latency 81.209 ms 2024-03-31T23:31:40.431 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:40.430+0000 7fba7ca22640 1 -- 172.21.15.46:0/3082205121 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fba781064f0 msgr2=0x7fba781068d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:31:40.432 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:40.430+0000 7fba7ca22640 1 --2- 172.21.15.46:0/3082205121 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fba781064f0 0x7fba781068d0 secure :-1 s=READY pgs=972 cs=0 l=1 rev1=1 crypto rx=0x7fba68009be0 tx=0x7fba6802bef0 comp rx=0 tx=0).stop 2024-03-31T23:31:40.432 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:40.430+0000 7fba7ca22640 1 -- 172.21.15.46:0/3082205121 shutdown_connections 2024-03-31T23:31:40.432 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:40.430+0000 7fba7ca22640 1 --2- 172.21.15.46:0/3082205121 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fba78100e10 0x7fba780697c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:31:40.432 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:40.430+0000 7fba7ca22640 1 --2- 172.21.15.46:0/3082205121 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fba78100450 0x7fba781008d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:31:40.432 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:40.430+0000 7fba7ca22640 1 --2- 172.21.15.46:0/3082205121 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fba781064f0 0x7fba781068d0 unknown :-1 s=CLOSED pgs=972 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:31:40.432 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:40.430+0000 7fba7ca22640 1 -- 172.21.15.46:0/3082205121 >> 172.21.15.46:0/3082205121 conn(0x7fba780fbf60 msgr2=0x7fba780fe380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:31:40.432 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:40.430+0000 7fba7ca22640 1 -- 172.21.15.46:0/3082205121 shutdown_connections 2024-03-31T23:31:40.432 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:40.430+0000 7fba7ca22640 1 -- 172.21.15.46:0/3082205121 wait complete. 2024-03-31T23:31:40.434 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:40.431+0000 7fba7ca22640 1 Processor -- start 2024-03-31T23:31:40.434 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:40.432+0000 7fba7ca22640 1 -- start start 2024-03-31T23:31:40.434 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:40.432+0000 7fba7ca22640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fba78100450 0x7fba78199f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:31:40.434 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:40.432+0000 7fba7ca22640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fba78100e10 0x7fba7819a490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:31:40.435 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:40.432+0000 7fba7ca22640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fba781064f0 0x7fba7819e810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:31:40.435 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:40.432+0000 7fba7ca22640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fba78113ac0 con 0x7fba78100e10 2024-03-31T23:31:40.435 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:40.432+0000 7fba7ca22640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fba78113940 con 0x7fba78100450 2024-03-31T23:31:40.435 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:40.432+0000 7fba7ca22640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fba78113c40 con 0x7fba781064f0 2024-03-31T23:31:40.435 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:40.432+0000 7fba7759e640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fba781064f0 0x7fba7819e810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:31:40.435 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:40.432+0000 7fba7759e640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fba781064f0 0x7fba7819e810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.46:49104/0 (socket says 172.21.15.46:49104) 2024-03-31T23:31:40.435 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:40.432+0000 7fba7759e640 1 -- 172.21.15.46:0/933513772 learned_addr learned my addr 172.21.15.46:0/933513772 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:31:40.435 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:40.433+0000 7fba7659c640 1 --2- 172.21.15.46:0/933513772 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fba78100e10 0x7fba7819a490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:31:40.435 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:40.433+0000 7fba76d9d640 1 --2- 172.21.15.46:0/933513772 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fba78100450 0x7fba78199f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:31:40.435 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:40.433+0000 7fba7759e640 1 -- 172.21.15.46:0/933513772 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fba78100450 msgr2=0x7fba78199f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:31:40.436 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:40.433+0000 7fba7759e640 1 --2- 172.21.15.46:0/933513772 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fba78100450 0x7fba78199f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:31:40.436 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:40.433+0000 7fba7759e640 1 -- 172.21.15.46:0/933513772 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fba78100e10 msgr2=0x7fba7819a490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:31:40.436 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:40.433+0000 7fba7759e640 1 --2- 172.21.15.46:0/933513772 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fba78100e10 0x7fba7819a490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:31:40.436 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:40.433+0000 7fba7759e640 1 -- 172.21.15.46:0/933513772 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fba7819ef40 con 0x7fba781064f0 2024-03-31T23:31:40.436 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:40.433+0000 7fba76d9d640 1 --2- 172.21.15.46:0/933513772 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fba78100450 0x7fba78199f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T23:31:40.436 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:40.433+0000 7fba7659c640 1 --2- 172.21.15.46:0/933513772 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fba78100e10 0x7fba7819a490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:31:40.437 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:40.434+0000 7fba7759e640 1 --2- 172.21.15.46:0/933513772 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fba781064f0 0x7fba7819e810 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fba7000e970 tx=0x7fba7000ee30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:31:40.437 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:40.434+0000 7fba6ffff640 1 -- 172.21.15.46:0/933513772 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba7000ccc0 con 0x7fba781064f0 2024-03-31T23:31:40.437 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:40.434+0000 7fba7ca22640 1 -- 172.21.15.46:0/933513772 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fba7819f230 con 0x7fba781064f0 2024-03-31T23:31:40.437 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:40.434+0000 7fba7ca22640 1 -- 172.21.15.46:0/933513772 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fba781a6ac0 con 0x7fba781064f0 2024-03-31T23:31:40.437 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:40.434+0000 7fba6ffff640 1 -- 172.21.15.46:0/933513772 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fba700054d0 con 0x7fba781064f0 2024-03-31T23:31:40.437 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:40.434+0000 7fba6ffff640 1 -- 172.21.15.46:0/933513772 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba70010430 con 0x7fba781064f0 2024-03-31T23:31:40.438 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:40.437+0000 7fba6ffff640 1 -- 172.21.15.46:0/933513772 <== mon.1 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 25) ==== 137451+0+0 (secure 0 0 0) 0x7fba70005020 con 0x7fba781064f0 2024-03-31T23:31:40.439 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:40.438+0000 7fba7fe08740 1 Processor -- start 2024-03-31T23:31:40.439 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:40.438+0000 7fba7fe08740 1 -- start start 2024-03-31T23:31:40.445 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:40.438+0000 7fba7fe08740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55fa15f0e8d0 0x55fa15f0ecb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:31:40.446 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:40.438+0000 7fba7fe08740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fa15f0f1f0 0x55fa15f17a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:31:40.446 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:40.438+0000 7fba7fe08740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fa15f207d0 0x55fa15f22bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:31:40.446 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:40.438+0000 7fba7fe08740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55fa15e718f0 con 0x55fa15f0f1f0 2024-03-31T23:31:40.446 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:40.438+0000 7fba7fe08740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55fa15e1a8d0 con 0x55fa15f0e8d0 2024-03-31T23:31:40.446 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:40.439+0000 7fba7fe08740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55fa15e6d990 con 0x55fa15f207d0 2024-03-31T23:31:40.446 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:40.439+0000 7fba76d9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fa15f0f1f0 0x55fa15f17a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:31:40.446 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:40.440+0000 7fba6ffff640 1 --2- 172.21.15.46:0/933513772 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fba640a2700 0x7fba640a4bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:31:40.446 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:40.440+0000 7fba7759e640 1 --2- 172.21.15.46:0/933513772 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fba640a2700 0x7fba640a4bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:31:40.446 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:40.441+0000 7fba7659c640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55fa15f0e8d0 0x55fa15f0ecb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:31:40.446 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:40.441+0000 7fba7659c640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55fa15f0e8d0 0x55fa15f0ecb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.46:36712/0 (socket says 172.21.15.46:36712) 2024-03-31T23:31:40.447 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:40.441+0000 7fba7659c640 1 -- 172.21.15.46:0/2377959424 learned_addr learned my addr 172.21.15.46:0/2377959424 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:31:40.447 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:40.441+0000 7fba7659c640 1 --2- 172.21.15.46:0/2377959424 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fa15f207d0 0x55fa15f22bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:31:40.447 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:40.442+0000 7fba7659c640 1 -- 172.21.15.46:0/2377959424 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fa15f207d0 msgr2=0x55fa15f22bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:31:40.447 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:40.442+0000 7fba7659c640 1 --2- 172.21.15.46:0/2377959424 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fa15f207d0 0x55fa15f22bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:31:40.447 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:40.442+0000 7fba7659c640 1 -- 172.21.15.46:0/2377959424 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fa15f0f1f0 msgr2=0x55fa15f17a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:31:40.447 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:40.442+0000 7fba7659c640 1 --2- 172.21.15.46:0/2377959424 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fa15f0f1f0 0x55fa15f17a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:31:40.447 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:40.442+0000 7fba7659c640 1 -- 172.21.15.46:0/2377959424 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fa15b86210 con 0x55fa15f0e8d0 2024-03-31T23:31:40.447 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:40.442+0000 7fba7759e640 1 --2- 172.21.15.46:0/933513772 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fba640a2700 0x7fba640a4bc0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fba7009b260 tx=0x7fba70097170 comp rx=0 tx=0).ready entity=mgr.15654 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:31:40.447 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:40.442+0000 7fba6ffff640 1 -- 172.21.15.46:0/933513772 <== mon.1 v2:172.21.15.162:3300/0 5 ==== osd_map(126..126 src has 1..126) ==== 9554+0+0 (secure 0 0 0) 0x7fba700cf250 con 0x7fba781064f0 2024-03-31T23:31:40.447 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:40.444+0000 7fba7659c640 1 --2- 172.21.15.46:0/2377959424 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55fa15f0e8d0 0x55fa15f0ecb0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fba60004650 tx=0x7fba60009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7ff42c83ef5e35a6 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:31:40.447 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:40.444+0000 7fba6d7fa640 1 -- 172.21.15.46:0/2377959424 <== mon.2 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba6000fc50 con 0x55fa15f0e8d0 2024-03-31T23:31:40.447 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:40.444+0000 7fba6d7fa640 1 -- 172.21.15.46:0/2377959424 <== mon.2 v2:172.21.15.142:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fba6000fdf0 con 0x55fa15f0e8d0 2024-03-31T23:31:40.448 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:40.445+0000 7fba7fe08740 1 -- 172.21.15.46:0/2377959424 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55fa15f0e8d0 msgr2=0x55fa15f0ecb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:31:40.448 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:40.445+0000 7fba7fe08740 1 --2- 172.21.15.46:0/2377959424 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55fa15f0e8d0 0x55fa15f0ecb0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fba60004650 tx=0x7fba60009b80 comp rx=0 tx=0).stop 2024-03-31T23:31:40.448 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:40.446+0000 7fba7fe08740 1 -- 172.21.15.46:0/2377959424 shutdown_connections 2024-03-31T23:31:40.448 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:40.446+0000 7fba7fe08740 1 --2- 172.21.15.46:0/2377959424 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fa15f207d0 0x55fa15f22bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:31:40.448 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:40.446+0000 7fba7fe08740 1 --2- 172.21.15.46:0/2377959424 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fa15f0f1f0 0x55fa15f17a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:31:40.449 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:40.446+0000 7fba7fe08740 1 --2- 172.21.15.46:0/2377959424 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55fa15f0e8d0 0x55fa15f0ecb0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:31:40.449 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:40.446+0000 7fba7fe08740 1 -- 172.21.15.46:0/2377959424 >> 172.21.15.46:0/2377959424 conn(0x55fa15f091f0 msgr2=0x55fa15f095d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:31:40.450 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:40.448+0000 7fba7fe08740 1 -- 172.21.15.46:0/2377959424 shutdown_connections 2024-03-31T23:31:40.450 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:40.448+0000 7fba7fe08740 1 -- 172.21.15.46:0/2377959424 wait complete. 2024-03-31T23:31:40.450 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:40.448+0000 7fba7fe08740 1 -- start start 2024-03-31T23:31:40.450 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:40.448+0000 7fba7fe08740 1 Processor -- start 2024-03-31T23:31:40.450 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:40.449+0000 7fba7fe08740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55fa15f0e8d0 0x55fa15f14fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:31:40.450 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:40.449+0000 7fba7fe08740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fa15f0f1f0 0x55fa15f15500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:31:40.450 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:40.449+0000 7fba7fe08740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fa15f207d0 0x55fa15f15a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:31:40.450 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:40.449+0000 7fba7fe08740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55fa15f23400 con 0x55fa15f0f1f0 2024-03-31T23:31:40.450 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:40.449+0000 7fba7fe08740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55fa15e0ff80 con 0x55fa15f0e8d0 2024-03-31T23:31:40.451 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:40.449+0000 7fba7fe08740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55fa15f23580 con 0x55fa15f207d0 2024-03-31T23:31:40.451 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:40.449+0000 7fba7fe08740 10 client.0 _ll_register_callbacks cb 0x55fa15e68290 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T23:31:40.451 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:40.449+0000 7fba7659c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fa15f207d0 0x55fa15f15a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:31:40.452 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:40.449+0000 7fba7659c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fa15f207d0 0x55fa15f15a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.46:49116/0 (socket says 172.21.15.46:49116) 2024-03-31T23:31:40.452 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:40.449+0000 7fba7659c640 1 -- 172.21.15.46:0/2635016099 learned_addr learned my addr 172.21.15.46:0/2635016099 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:31:40.452 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:40.450+0000 7fba7659c640 1 -- 172.21.15.46:0/2635016099 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55fa15f0e8d0 msgr2=0x55fa15f14fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:31:40.452 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:40.450+0000 7fba7659c640 1 --2- 172.21.15.46:0/2635016099 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55fa15f0e8d0 0x55fa15f14fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:31:40.452 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:40.450+0000 7fba7659c640 1 -- 172.21.15.46:0/2635016099 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fa15f0f1f0 msgr2=0x55fa15f15500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:31:40.452 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:40.450+0000 7fba7659c640 1 --2- 172.21.15.46:0/2635016099 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fa15f0f1f0 0x55fa15f15500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:31:40.452 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:40.450+0000 7fba7659c640 1 -- 172.21.15.46:0/2635016099 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fa15c52620 con 0x55fa15f207d0 2024-03-31T23:31:40.452 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:40.451+0000 7fba7659c640 1 --2- 172.21.15.46:0/2635016099 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fa15f207d0 0x55fa15f15a40 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fba600114f0 tx=0x7fba60011520 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e4026aaba54c5417 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:31:40.452 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:40.451+0000 7fba5e7fc640 10 client.0 ms_handle_connect on v2:172.21.15.162:3300/0 2024-03-31T23:31:40.454 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:40.451+0000 7fba5e7fc640 1 -- 172.21.15.46:0/2635016099 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba6002e060 con 0x55fa15f207d0 2024-03-31T23:31:40.454 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:40.451+0000 7fba7fe08740 1 -- 172.21.15.46:0/2635016099 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fba6000fdf0 con 0x55fa15f207d0 2024-03-31T23:31:40.454 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:40.451+0000 7fba5e7fc640 1 -- 172.21.15.46:0/2635016099 <== mon.1 v2:172.21.15.162:3300/0 2 ==== osd_map(126..126 src has 1..126) ==== 9554+0+0 (secure 0 0 0) 0x7fba60019040 con 0x55fa15f207d0 2024-03-31T23:31:40.454 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:40.451+0000 7fba5e7fc640 1 -- 172.21.15.46:0/2635016099 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_get_version_reply(handle=1 version=48) ==== 24+0+0 (secure 0 0 0) 0x7fba60035050 con 0x55fa15f207d0 2024-03-31T23:31:40.454 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:40.451+0000 7fba7fe08740 10 client.27478 fetch_fsmap learned FSMap version 48 2024-03-31T23:31:40.454 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:40.451+0000 7fba7fe08740 1 -- 172.21.15.46:0/2635016099 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,fsmap=48,monmap=4+}) -- 0x55fa15f2f3b0 con 0x55fa15f207d0 2024-03-31T23:31:40.454 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:40.452+0000 7fba5e7fc640 1 -- 172.21.15.46:0/2635016099 <== mon.1 v2:172.21.15.162:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fba6000fdf0 con 0x55fa15f207d0 2024-03-31T23:31:40.455 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:40.453+0000 7fba5e7fc640 1 -- 172.21.15.46:0/2635016099 <== mon.1 v2:172.21.15.162:3300/0 5 ==== fsmap(e 48) ==== 8350+0+0 (secure 0 0 0) 0x7fba6001d020 con 0x55fa15f207d0 2024-03-31T23:31:40.455 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:40.453+0000 7fba7fe08740 10 client.27478 fetch_fsmap finished waiting for FSMap version 48 2024-03-31T23:31:40.455 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:40.453+0000 7fba7fe08740 10 client.27478 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-31T23:31:40.455 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:40.453+0000 7fba7fe08740 20 client.27478 populate_metadata read hostname 'smithi046' 2024-03-31T23:31:40.455 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:40.453+0000 7fba7fe08740 1 --2- 172.21.15.46:0/2635016099 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55fa15f30af0 0x55fa15f32f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:31:40.455 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:40.453+0000 7fba7fe08740 4 client.27478 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-31T23:31:40.455 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:40.453+0000 7fba7fe08740 1 -- 172.21.15.46:0/2635016099 --> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55fa15e85120 con 0x55fa15f30af0 2024-03-31T23:31:40.455 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:40.453+0000 7fba76d9d640 1 --2- 172.21.15.46:0/2635016099 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55fa15f30af0 0x55fa15f32f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:31:40.459 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:40.454+0000 7fba5e7fc640 10 client.27478 ms_handle_connect on v2:172.21.15.142:6838/2519143918 2024-03-31T23:31:40.459 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:40.454+0000 7fba76d9d640 1 --2- 172.21.15.46:0/2635016099 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55fa15f30af0 0x55fa15f32f70 crc :-1 s=READY pgs=254 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=27a8da415e7f05c4 server_cookie=376b4f2eee8aacbb in_seq=0 out_seq=0 2024-03-31T23:31:40.459 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:40.454+0000 7fba5e7fc640 1 -- 172.21.15.46:0/2635016099 <== mds.0 v2:172.21.15.142:6838/2519143918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55fa15e85120 con 0x55fa15f30af0 2024-03-31T23:31:40.459 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:40.454+0000 7fba5e7fc640 10 client.27478 handle_command_reply: tid=0 2024-03-31T23:31:40.459 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:40.454+0000 7fba7fe08740 1 client.27478 shutdown 2024-03-31T23:31:40.460 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:40.455+0000 7fba7fe08740 1 -- 172.21.15.46:0/2635016099 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fa15f207d0 msgr2=0x55fa15f15a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:31:40.460 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:40.455+0000 7fba7fe08740 1 --2- 172.21.15.46:0/2635016099 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fa15f207d0 0x55fa15f15a40 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fba600114f0 tx=0x7fba60011520 comp rx=0 tx=0).stop 2024-03-31T23:31:40.460 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:40.455+0000 7fba7fe08740 1 -- 172.21.15.46:0/2635016099 shutdown_connections 2024-03-31T23:31:40.460 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:40.455+0000 7fba7fe08740 1 --2- 172.21.15.46:0/2635016099 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55fa15f30af0 0x55fa15f32f70 crc :-1 s=READY pgs=254 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:31:40.460 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:40.455+0000 7fba7fe08740 1 --2- 172.21.15.46:0/2635016099 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fa15f207d0 0x55fa15f15a40 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:31:40.460 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:40.455+0000 7fba7fe08740 1 --2- 172.21.15.46:0/2635016099 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fa15f0f1f0 0x55fa15f15500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:31:40.460 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:40.455+0000 7fba7fe08740 1 --2- 172.21.15.46:0/2635016099 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55fa15f0e8d0 0x55fa15f14fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:31:40.460 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:40.455+0000 7fba7fe08740 1 -- 172.21.15.46:0/2635016099 >> 172.21.15.46:0/2635016099 conn(0x55fa15f091f0 msgr2=0x55fa15f0af40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:31:40.460 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:40.455+0000 7fba5e7fc640 0 client.27478 ms_handle_reset on v2:172.21.15.142:6838/2519143918 2024-03-31T23:31:40.460 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:40.455+0000 7fba7fe08740 1 -- 172.21.15.46:0/2635016099 shutdown_connections 2024-03-31T23:31:40.460 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:40.455+0000 7fba7fe08740 1 -- 172.21.15.46:0/2635016099 wait complete. 2024-03-31T23:31:40.460 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:40.455+0000 7fba7fe08740 20 client.27478 trim_cache size 0 max 16384 2024-03-31T23:31:40.479 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:40.476+0000 7fba7fe08740 1 Processor -- start 2024-03-31T23:31:40.479 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:40.476+0000 7fba7fe08740 1 -- start start 2024-03-31T23:31:40.479 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:40.477+0000 7fba7fe08740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fa15f0e8d0 0x55fa15f176f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:31:40.480 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:40.477+0000 7fba7fe08740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fa15f0f1f0 0x55fa15f04030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:31:40.480 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:40.477+0000 7fba7fe08740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55fa15f207d0 0x55fa15f2c280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:31:40.480 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:40.477+0000 7fba7fe08740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fba58012930 con 0x55fa15f0e8d0 2024-03-31T23:31:40.480 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:40.477+0000 7fba7fe08740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fba580101d0 con 0x55fa15f207d0 2024-03-31T23:31:40.480 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:40.477+0000 7fba7fe08740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fba5800e040 con 0x55fa15f0f1f0 2024-03-31T23:31:40.480 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:40.477+0000 7fba7659c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fa15f0e8d0 0x55fa15f176f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:31:40.480 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:40.477+0000 7fba7659c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fa15f0e8d0 0x55fa15f176f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:54286/0 (socket says 172.21.15.46:54286) 2024-03-31T23:31:40.480 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:40.477+0000 7fba76d9d640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fa15f0f1f0 0x55fa15f04030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:31:40.480 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:40.477+0000 7fba7659c640 1 -- 172.21.15.46:0/348613977 learned_addr learned my addr 172.21.15.46:0/348613977 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:31:40.480 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:40.477+0000 7fba7659c640 1 --2- 172.21.15.46:0/348613977 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55fa15f207d0 0x55fa15f2c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:31:40.480 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:40.477+0000 7fba7659c640 1 -- 172.21.15.46:0/348613977 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fa15f0f1f0 msgr2=0x55fa15f04030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:31:40.480 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:40.477+0000 7fba7659c640 1 --2- 172.21.15.46:0/348613977 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fa15f0f1f0 0x55fa15f04030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:31:40.481 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:40.477+0000 7fba7659c640 1 -- 172.21.15.46:0/348613977 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55fa15f207d0 msgr2=0x55fa15f2c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:31:40.481 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:40.477+0000 7fba7659c640 1 --2- 172.21.15.46:0/348613977 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55fa15f207d0 0x55fa15f2c280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:31:40.481 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:40.477+0000 7fba7659c640 1 -- 172.21.15.46:0/348613977 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fa15f0f020 con 0x55fa15f0e8d0 2024-03-31T23:31:40.481 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:40.478+0000 7fba7659c640 1 --2- 172.21.15.46:0/348613977 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fa15f0e8d0 0x55fa15f176f0 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 crypto rx=0x7fba60011660 tx=0x7fba60005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fed0d16d038b2805 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:31:40.481 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:40.478+0000 7fba6cff9640 1 -- 172.21.15.46:0/348613977 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba6002e040 con 0x55fa15f0e8d0 2024-03-31T23:31:40.481 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:40.478+0000 7fba6cff9640 1 -- 172.21.15.46:0/348613977 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fba60005a70 con 0x55fa15f0e8d0 2024-03-31T23:31:40.481 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:40.478+0000 7fba6cff9640 1 -- 172.21.15.46:0/348613977 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba600050f0 con 0x55fa15f0e8d0 2024-03-31T23:31:40.486 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:40.479+0000 7fba7fe08740 1 -- 172.21.15.46:0/348613977 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fa15f0e8d0 msgr2=0x55fa15f176f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:31:40.486 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:40.479+0000 7fba7fe08740 1 --2- 172.21.15.46:0/348613977 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fa15f0e8d0 0x55fa15f176f0 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 crypto rx=0x7fba60011660 tx=0x7fba60005f90 comp rx=0 tx=0).stop 2024-03-31T23:31:40.486 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:40.479+0000 7fba7fe08740 1 -- 172.21.15.46:0/348613977 shutdown_connections 2024-03-31T23:31:40.486 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:40.479+0000 7fba7fe08740 1 --2- 172.21.15.46:0/348613977 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55fa15f207d0 0x55fa15f2c280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:31:40.486 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:40.479+0000 7fba7fe08740 1 --2- 172.21.15.46:0/348613977 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fa15f0f1f0 0x55fa15f04030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:31:40.486 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:40.479+0000 7fba7fe08740 1 --2- 172.21.15.46:0/348613977 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fa15f0e8d0 0x55fa15f176f0 unknown :-1 s=CLOSED pgs=973 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:31:40.486 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:40.479+0000 7fba7fe08740 1 -- 172.21.15.46:0/348613977 >> 172.21.15.46:0/348613977 conn(0x55fa15f091f0 msgr2=0x55fa15f10340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:31:40.486 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:40.479+0000 7fba7fe08740 1 -- 172.21.15.46:0/348613977 shutdown_connections 2024-03-31T23:31:40.486 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:40.479+0000 7fba7fe08740 1 -- 172.21.15.46:0/348613977 wait complete. 2024-03-31T23:31:40.487 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:40.480+0000 7fba7fe08740 1 -- start start 2024-03-31T23:31:40.487 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:40.480+0000 7fba7fe08740 1 Processor -- start 2024-03-31T23:31:40.487 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:40.480+0000 7fba7fe08740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fa15f0e8d0 0x55fa15f28b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:31:40.487 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:40.480+0000 7fba7659c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fa15f0e8d0 0x55fa15f28b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:31:40.487 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:40.480+0000 7fba7659c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fa15f0e8d0 0x55fa15f28b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:54296/0 (socket says 172.21.15.46:54296) 2024-03-31T23:31:40.487 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:40.480+0000 7fba7fe08740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55fa15f0f1f0 0x55fa15f11850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:31:40.487 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:40.480+0000 7fba7659c640 1 -- 172.21.15.46:0/3271874772 learned_addr learned my addr 172.21.15.46:0/3271874772 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:31:40.487 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:40.480+0000 7fba7fe08740 1 --2- 172.21.15.46:0/3271874772 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fa15f207d0 0x55fa15f11d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:31:40.487 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:40.480+0000 7fba7fe08740 1 -- 172.21.15.46:0/3271874772 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fba5800e690 con 0x55fa15f0e8d0 2024-03-31T23:31:40.487 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:40.480+0000 7fba7fe08740 1 -- 172.21.15.46:0/3271874772 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fba5800fb00 con 0x55fa15f0f1f0 2024-03-31T23:31:40.487 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:40.480+0000 7fba7fe08740 1 -- 172.21.15.46:0/3271874772 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fba5800ed60 con 0x55fa15f207d0 2024-03-31T23:31:40.488 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:40.481+0000 7fba76d9d640 1 --2- 172.21.15.46:0/3271874772 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55fa15f0f1f0 0x55fa15f11850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:31:40.488 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:40.481+0000 7fba7fe08740 10 client.0 _ll_register_callbacks cb 0x7fba600116d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T23:31:40.488 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:40.481+0000 7fba7659c640 1 --2- 172.21.15.46:0/3271874772 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fa15f207d0 0x55fa15f11d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:31:40.488 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:40.481+0000 7fba7659c640 1 -- 172.21.15.46:0/3271874772 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fa15f207d0 msgr2=0x55fa15f11d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:31:40.488 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:40.481+0000 7fba7659c640 1 --2- 172.21.15.46:0/3271874772 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fa15f207d0 0x55fa15f11d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:31:40.488 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:40.481+0000 7fba7659c640 1 -- 172.21.15.46:0/3271874772 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55fa15f0f1f0 msgr2=0x55fa15f11850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:31:40.488 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:40.481+0000 7fba7659c640 1 --2- 172.21.15.46:0/3271874772 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55fa15f0f1f0 0x55fa15f11850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:31:40.488 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:40.481+0000 7fba7659c640 1 -- 172.21.15.46:0/3271874772 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fa15f125c0 con 0x55fa15f0e8d0 2024-03-31T23:31:40.488 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:40.481+0000 7fba7659c640 1 --2- 172.21.15.46:0/3271874772 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fa15f0e8d0 0x55fa15f28b10 secure :-1 s=READY pgs=974 cs=0 l=1 rev1=1 crypto rx=0x7fba6000fc60 tx=0x7fba60005970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e9f598494def2238 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:31:40.488 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:40.481+0000 7fba5e7fc640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-31T23:31:40.488 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:40.481+0000 7fba5e7fc640 1 -- 172.21.15.46:0/3271874772 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba6000ee60 con 0x55fa15f0e8d0 2024-03-31T23:31:40.488 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:40.481+0000 7fba7fe08740 1 -- 172.21.15.46:0/3271874772 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fba60005a70 con 0x55fa15f0e8d0 2024-03-31T23:31:40.489 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:40.481+0000 7fba5e7fc640 1 -- 172.21.15.46:0/3271874772 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(126..126 src has 1..126) ==== 9554+0+0 (secure 0 0 0) 0x7fba600050f0 con 0x55fa15f0e8d0 2024-03-31T23:31:40.489 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:40.482+0000 7fba5e7fc640 1 -- 172.21.15.46:0/3271874772 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=48) ==== 24+0+0 (secure 0 0 0) 0x7fba60009b80 con 0x55fa15f0e8d0 2024-03-31T23:31:40.489 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:40.482+0000 7fba7fe08740 10 client.17763 fetch_fsmap learned FSMap version 48 2024-03-31T23:31:40.489 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:40.482+0000 7fba7fe08740 1 -- 172.21.15.46:0/3271874772 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=48,monmap=4+}) -- 0x55fa15f96e40 con 0x55fa15f0e8d0 2024-03-31T23:31:40.489 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:40.482+0000 7fba5e7fc640 1 -- 172.21.15.46:0/3271874772 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fba60005a70 con 0x55fa15f0e8d0 2024-03-31T23:31:40.489 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:40.482+0000 7fba5e7fc640 1 -- 172.21.15.46:0/3271874772 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 48) ==== 8350+0+0 (secure 0 0 0) 0x7fba60038040 con 0x55fa15f0e8d0 2024-03-31T23:31:40.489 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:40.482+0000 7fba7fe08740 10 client.17763 fetch_fsmap finished waiting for FSMap version 48 2024-03-31T23:31:40.489 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:40.482+0000 7fba7fe08740 10 client.17763 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-31T23:31:40.489 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:40.482+0000 7fba7fe08740 20 client.17763 populate_metadata read hostname 'smithi046' 2024-03-31T23:31:40.489 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:40.482+0000 7fba7fe08740 1 --2- 172.21.15.46:0/3271874772 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55fa15f973d0 0x55fa15f997f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:31:40.489 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:40.482+0000 7fba7fe08740 4 client.17763 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-31T23:31:40.489 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:40.482+0000 7fba7fe08740 1 -- 172.21.15.46:0/3271874772 --> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55fa15e7ee30 con 0x55fa15f973d0 2024-03-31T23:31:40.489 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:40.482+0000 7fba76d9d640 1 --2- 172.21.15.46:0/3271874772 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55fa15f973d0 0x55fa15f997f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:31:40.490 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:40.483+0000 7fba5e7fc640 10 client.17763 ms_handle_connect on v2:172.21.15.142:6838/2519143918 2024-03-31T23:31:40.490 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:40.483+0000 7fba76d9d640 1 --2- 172.21.15.46:0/3271874772 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55fa15f973d0 0x55fa15f997f0 crc :-1 s=READY pgs=255 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4e9eab41eb819e52 server_cookie=5fd00ef6a9ce2147 in_seq=0 out_seq=0 2024-03-31T23:31:40.699 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:40.697+0000 7fba5e7fc640 1 -- 172.21.15.46:0/3271874772 <== mds.0 v2:172.21.15.142:6838/2519143918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55fa15e7ee30 con 0x55fa15f973d0 2024-03-31T23:31:40.699 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:40.697+0000 7fba5e7fc640 10 client.17763 handle_command_reply: tid=0 2024-03-31T23:31:40.699 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:40.697+0000 7fba7fe08740 1 client.17763 shutdown 2024-03-31T23:31:40.699 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:40.698+0000 7fba7fe08740 1 -- 172.21.15.46:0/3271874772 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fa15f0e8d0 msgr2=0x55fa15f28b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:31:40.699 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:40.698+0000 7fba7fe08740 1 --2- 172.21.15.46:0/3271874772 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fa15f0e8d0 0x55fa15f28b10 secure :-1 s=READY pgs=974 cs=0 l=1 rev1=1 crypto rx=0x7fba6000fc60 tx=0x7fba60005970 comp rx=0 tx=0).stop 2024-03-31T23:31:40.699 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:40.698+0000 7fba7fe08740 1 -- 172.21.15.46:0/3271874772 shutdown_connections 2024-03-31T23:31:40.699 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:40.698+0000 7fba7fe08740 1 --2- 172.21.15.46:0/3271874772 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55fa15f973d0 0x55fa15f997f0 crc :-1 s=READY pgs=255 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:31:40.699 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:40.698+0000 7fba7fe08740 1 --2- 172.21.15.46:0/3271874772 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fa15f207d0 0x55fa15f11d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:31:40.699 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:40.698+0000 7fba7fe08740 1 --2- 172.21.15.46:0/3271874772 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55fa15f0f1f0 0x55fa15f11850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:31:40.699 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:40.698+0000 7fba7fe08740 1 --2- 172.21.15.46:0/3271874772 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fa15f0e8d0 0x55fa15f28b10 secure :-1 s=CLOSED pgs=974 cs=0 l=1 rev1=1 crypto rx=0x7fba6000fc60 tx=0x7fba60005970 comp rx=0 tx=0).stop 2024-03-31T23:31:40.700 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:40.698+0000 7fba7fe08740 1 -- 172.21.15.46:0/3271874772 >> 172.21.15.46:0/3271874772 conn(0x55fa15f091f0 msgr2=0x55fa15f0a5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:31:40.700 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:40.698+0000 7fba5e7fc640 0 client.17763 ms_handle_reset on v2:172.21.15.142:6838/2519143918 2024-03-31T23:31:40.703 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:40.700+0000 7fba7fe08740 1 -- 172.21.15.46:0/3271874772 shutdown_connections 2024-03-31T23:31:40.703 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:40.702+0000 7fba7fe08740 1 -- 172.21.15.46:0/3271874772 wait complete. 2024-03-31T23:31:40.703 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:40.702+0000 7fba7fe08740 20 client.17763 trim_cache size 0 max 16384 2024-03-31T23:31:40.703 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout:{ 2024-03-31T23:31:40.703 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout: "message": "", 2024-03-31T23:31:40.703 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout: "return_code": 0 2024-03-31T23:31:40.703 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout:} 2024-03-31T23:31:40.704 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:40.702+0000 7fba7ca22640 1 -- 172.21.15.46:0/933513772 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fba640a2700 msgr2=0x7fba640a4bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:31:40.704 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:40.702+0000 7fba7ca22640 1 --2- 172.21.15.46:0/933513772 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fba640a2700 0x7fba640a4bc0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fba7009b260 tx=0x7fba70097170 comp rx=0 tx=0).stop 2024-03-31T23:31:40.704 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:40.702+0000 7fba7ca22640 1 -- 172.21.15.46:0/933513772 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fba781064f0 msgr2=0x7fba7819e810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:31:40.704 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:40.702+0000 7fba7ca22640 1 --2- 172.21.15.46:0/933513772 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fba781064f0 0x7fba7819e810 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fba7000e970 tx=0x7fba7000ee30 comp rx=0 tx=0).stop 2024-03-31T23:31:40.704 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:40.703+0000 7fba7ca22640 1 -- 172.21.15.46:0/933513772 shutdown_connections 2024-03-31T23:31:40.704 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:40.703+0000 7fba7ca22640 1 --2- 172.21.15.46:0/933513772 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fba640a2700 0x7fba640a4bc0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:31:40.704 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:40.703+0000 7fba7ca22640 1 --2- 172.21.15.46:0/933513772 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fba781064f0 0x7fba7819e810 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:31:40.704 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:40.703+0000 7fba7ca22640 1 --2- 172.21.15.46:0/933513772 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fba78100e10 0x7fba7819a490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:31:40.705 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:40.703+0000 7fba7ca22640 1 --2- 172.21.15.46:0/933513772 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fba78100450 0x7fba78199f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:31:40.705 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:40.703+0000 7fba7ca22640 1 -- 172.21.15.46:0/933513772 >> 172.21.15.46:0/933513772 conn(0x7fba780fbf60 msgr2=0x7fba781072b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:31:40.705 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:40.703+0000 7fba7ca22640 1 -- 172.21.15.46:0/933513772 shutdown_connections 2024-03-31T23:31:40.705 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:40.703+0000 7fba7ca22640 1 -- 172.21.15.46:0/933513772 wait complete. 2024-03-31T23:31:41.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:31:40 smithi162 ceph-mon[36237]: pgmap v631: 129 pgs: 129 active+clean; 985 MiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 6.9 MiB/s wr, 367 op/s 2024-03-31T23:31:41.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:31:40 smithi046 ceph-mon[32498]: pgmap v631: 129 pgs: 129 active+clean; 985 MiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 6.9 MiB/s wr, 367 op/s 2024-03-31T23:31:41.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:31:40 smithi142 ceph-mon[35271]: pgmap v631: 129 pgs: 129 active+clean; 985 MiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 6.9 MiB/s wr, 367 op/s 2024-03-31T23:31:41.294 INFO:tasks.workunit.client.0.smithi046.stdout: 10 64427 8.44 MB/sec execute 286 sec latency 209.238 ms 2024-03-31T23:31:42.294 INFO:tasks.workunit.client.0.smithi046.stdout: 10 64591 8.46 MB/sec execute 287 sec latency 95.158 ms 2024-03-31T23:31:43.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:31:42 smithi162 ceph-mon[36237]: pgmap v632: 129 pgs: 129 active+clean; 985 MiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 4.7 MiB/s wr, 208 op/s 2024-03-31T23:31:43.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:31:42 smithi046 ceph-mon[32498]: pgmap v632: 129 pgs: 129 active+clean; 985 MiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 4.7 MiB/s wr, 208 op/s 2024-03-31T23:31:43.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:31:42 smithi142 ceph-mon[35271]: pgmap v632: 129 pgs: 129 active+clean; 985 MiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 4.7 MiB/s wr, 208 op/s 2024-03-31T23:31:43.294 INFO:tasks.workunit.client.0.smithi046.stdout: 10 64736 8.44 MB/sec execute 288 sec latency 85.822 ms 2024-03-31T23:31:44.295 INFO:tasks.workunit.client.0.smithi046.stdout: 10 64896 8.44 MB/sec execute 289 sec latency 93.062 ms 2024-03-31T23:31:45.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:31:44 smithi162 ceph-mon[36237]: pgmap v633: 129 pgs: 129 active+clean; 985 MiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 4.7 MiB/s wr, 208 op/s 2024-03-31T23:31:45.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:31:44 smithi046 ceph-mon[32498]: pgmap v633: 129 pgs: 129 active+clean; 985 MiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 4.7 MiB/s wr, 208 op/s 2024-03-31T23:31:45.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:31:44 smithi142 ceph-mon[35271]: pgmap v633: 129 pgs: 129 active+clean; 985 MiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 4.7 MiB/s wr, 208 op/s 2024-03-31T23:31:45.294 INFO:tasks.workunit.client.0.smithi046.stdout: 10 65027 8.43 MB/sec execute 290 sec latency 120.390 ms 2024-03-31T23:31:46.295 INFO:tasks.workunit.client.0.smithi046.stdout: 10 65187 8.42 MB/sec execute 291 sec latency 69.642 ms 2024-03-31T23:31:47.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:31:46 smithi162 ceph-mon[36237]: pgmap v634: 129 pgs: 129 active+clean; 984 MiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 7.1 MiB/s wr, 382 op/s 2024-03-31T23:31:47.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:31:46 smithi046 ceph-mon[32498]: pgmap v634: 129 pgs: 129 active+clean; 984 MiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 7.1 MiB/s wr, 382 op/s 2024-03-31T23:31:47.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:31:46 smithi142 ceph-mon[35271]: pgmap v634: 129 pgs: 129 active+clean; 984 MiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 7.1 MiB/s wr, 382 op/s 2024-03-31T23:31:47.294 INFO:tasks.workunit.client.0.smithi046.stdout: 10 65354 8.42 MB/sec execute 292 sec latency 87.458 ms 2024-03-31T23:31:48.295 INFO:tasks.workunit.client.0.smithi046.stdout: 10 65500 8.41 MB/sec execute 293 sec latency 110.295 ms 2024-03-31T23:31:49.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:31:48 smithi162 ceph-mon[36237]: pgmap v635: 129 pgs: 129 active+clean; 984 MiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 4.7 MiB/s wr, 278 op/s 2024-03-31T23:31:49.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:31:48 smithi046 ceph-mon[32498]: pgmap v635: 129 pgs: 129 active+clean; 984 MiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 4.7 MiB/s wr, 278 op/s 2024-03-31T23:31:49.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:31:48 smithi142 ceph-mon[35271]: pgmap v635: 129 pgs: 129 active+clean; 984 MiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 4.7 MiB/s wr, 278 op/s 2024-03-31T23:31:49.295 INFO:tasks.workunit.client.0.smithi046.stdout: 10 65651 8.41 MB/sec execute 294 sec latency 152.601 ms 2024-03-31T23:31:50.295 INFO:tasks.workunit.client.0.smithi046.stdout: 10 65852 8.43 MB/sec execute 295 sec latency 84.912 ms 2024-03-31T23:31:51.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:31:50 smithi162 ceph-mon[36237]: pgmap v636: 129 pgs: 129 active+clean; 991 MiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 6.8 MiB/s wr, 396 op/s 2024-03-31T23:31:51.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:31:50 smithi046 ceph-mon[32498]: pgmap v636: 129 pgs: 129 active+clean; 991 MiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 6.8 MiB/s wr, 396 op/s 2024-03-31T23:31:51.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:31:50 smithi142 ceph-mon[35271]: pgmap v636: 129 pgs: 129 active+clean; 991 MiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 6.8 MiB/s wr, 396 op/s 2024-03-31T23:31:51.295 INFO:tasks.workunit.client.0.smithi046.stdout: 10 66025 8.44 MB/sec execute 296 sec latency 88.500 ms 2024-03-31T23:31:52.295 INFO:tasks.workunit.client.0.smithi046.stdout: 10 66182 8.44 MB/sec execute 297 sec latency 105.659 ms 2024-03-31T23:31:53.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:31:52 smithi162 ceph-mon[36237]: pgmap v637: 129 pgs: 129 active+clean; 991 MiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 4.4 MiB/s wr, 291 op/s 2024-03-31T23:31:53.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:31:52 smithi046 ceph-mon[32498]: pgmap v637: 129 pgs: 129 active+clean; 991 MiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 4.4 MiB/s wr, 291 op/s 2024-03-31T23:31:53.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:31:52 smithi142 ceph-mon[35271]: pgmap v637: 129 pgs: 129 active+clean; 991 MiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 4.4 MiB/s wr, 291 op/s 2024-03-31T23:31:53.295 INFO:tasks.workunit.client.0.smithi046.stdout: 10 66359 8.44 MB/sec execute 298 sec latency 95.758 ms 2024-03-31T23:31:53.834 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:53.832+0000 7efe62047640 1 -- 172.21.15.46:0/4073511446 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7efe5c069770 msgr2=0x7efe5c10e5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:31:53.834 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:53.832+0000 7efe62047640 1 --2- 172.21.15.46:0/4073511446 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7efe5c069770 0x7efe5c10e5e0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7efe54009ca0 tx=0x7efe5402c100 comp rx=0 tx=0).stop 2024-03-31T23:31:53.834 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:53.832+0000 7efe62047640 1 -- 172.21.15.46:0/4073511446 shutdown_connections 2024-03-31T23:31:53.834 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:53.832+0000 7efe62047640 1 --2- 172.21.15.46:0/4073511446 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7efe5c069770 0x7efe5c10e5e0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:31:53.834 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:53.832+0000 7efe62047640 1 --2- 172.21.15.46:0/4073511446 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7efe5c068dd0 0x7efe5c069230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:31:53.834 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:53.832+0000 7efe62047640 1 --2- 172.21.15.46:0/4073511446 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efe5c107840 0x7efe5c107c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:31:53.834 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:53.832+0000 7efe62047640 1 -- 172.21.15.46:0/4073511446 >> 172.21.15.46:0/4073511446 conn(0x7efe5c0774a0 msgr2=0x7efe5c0778b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:31:53.834 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:53.832+0000 7efe62047640 1 -- 172.21.15.46:0/4073511446 shutdown_connections 2024-03-31T23:31:53.834 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:53.832+0000 7efe62047640 1 -- 172.21.15.46:0/4073511446 wait complete. 2024-03-31T23:31:53.835 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:53.833+0000 7efe62047640 1 Processor -- start 2024-03-31T23:31:53.835 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:53.833+0000 7efe62047640 1 -- start start 2024-03-31T23:31:53.836 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:53.833+0000 7efe62047640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7efe5c068dd0 0x7efe5c100970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:31:53.836 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:53.833+0000 7efe62047640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7efe5c069770 0x7efe5c100eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:31:53.837 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:53.833+0000 7efe62047640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efe5c107840 0x7efe5c1013f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:31:53.837 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:53.833+0000 7efe62047640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7efe5c111a70 con 0x7efe5c107840 2024-03-31T23:31:53.837 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:53.833+0000 7efe62047640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7efe5c1118f0 con 0x7efe5c069770 2024-03-31T23:31:53.837 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:53.833+0000 7efe62047640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7efe5c111bf0 con 0x7efe5c068dd0 2024-03-31T23:31:53.837 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:53.833+0000 7efe5bfff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efe5c107840 0x7efe5c1013f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:31:53.837 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:53.833+0000 7efe5affd640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7efe5c069770 0x7efe5c100eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:31:53.837 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:53.833+0000 7efe5b7fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7efe5c068dd0 0x7efe5c100970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:31:53.837 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:53.833+0000 7efe5bfff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efe5c107840 0x7efe5c1013f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55292/0 (socket says 172.21.15.46:55292) 2024-03-31T23:31:53.837 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:53.833+0000 7efe5b7fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7efe5c068dd0 0x7efe5c100970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.46:59200/0 (socket says 172.21.15.46:59200) 2024-03-31T23:31:53.837 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:53.833+0000 7efe5bfff640 1 -- 172.21.15.46:0/2761461825 learned_addr learned my addr 172.21.15.46:0/2761461825 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:31:53.837 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:53.833+0000 7efe5affd640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7efe5c069770 0x7efe5c100eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.46:39984/0 (socket says 172.21.15.46:39984) 2024-03-31T23:31:53.837 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:53.834+0000 7efe5b7fe640 1 -- 172.21.15.46:0/2761461825 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7efe5c069770 msgr2=0x7efe5c100eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:31:53.838 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:53.834+0000 7efe5b7fe640 1 --2- 172.21.15.46:0/2761461825 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7efe5c069770 0x7efe5c100eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:31:53.838 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:53.834+0000 7efe5b7fe640 1 -- 172.21.15.46:0/2761461825 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efe5c107840 msgr2=0x7efe5c1013f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:31:53.838 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:53.834+0000 7efe5b7fe640 1 --2- 172.21.15.46:0/2761461825 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efe5c107840 0x7efe5c1013f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:31:53.838 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:53.834+0000 7efe5b7fe640 1 -- 172.21.15.46:0/2761461825 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efe5c1a62b0 con 0x7efe5c068dd0 2024-03-31T23:31:53.838 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:53.834+0000 7efe5b7fe640 1 --2- 172.21.15.46:0/2761461825 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7efe5c068dd0 0x7efe5c100970 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7efe4c00e9f0 tx=0x7efe4c00eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:31:53.838 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:53.834+0000 7efe58ff9640 1 -- 172.21.15.46:0/2761461825 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe4c00cd80 con 0x7efe5c068dd0 2024-03-31T23:31:53.838 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:53.834+0000 7efe58ff9640 1 -- 172.21.15.46:0/2761461825 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7efe4c0035f0 con 0x7efe5c068dd0 2024-03-31T23:31:53.838 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:53.834+0000 7efe58ff9640 1 -- 172.21.15.46:0/2761461825 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe4c010670 con 0x7efe5c068dd0 2024-03-31T23:31:53.838 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:53.834+0000 7efe62047640 1 -- 172.21.15.46:0/2761461825 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efe5c1a65a0 con 0x7efe5c068dd0 2024-03-31T23:31:53.838 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:53.834+0000 7efe62047640 1 -- 172.21.15.46:0/2761461825 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efe5c1a6af0 con 0x7efe5c068dd0 2024-03-31T23:31:53.841 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:53.837+0000 7efe64c2d740 1 Processor -- start 2024-03-31T23:31:53.841 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:53.837+0000 7efe58ff9640 1 -- 172.21.15.46:0/2761461825 <== mon.1 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 25) ==== 137451+0+0 (secure 0 0 0) 0x7efe4c0037a0 con 0x7efe5c068dd0 2024-03-31T23:31:53.841 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:53.837+0000 7efe64c2d740 1 -- start start 2024-03-31T23:31:53.842 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:53.838+0000 7efe64c2d740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5574ac4fc6c0 0x5574ac4fcaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:31:53.842 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:53.838+0000 7efe64c2d740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5574ac4fcfe0 0x5574ac505870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:31:53.842 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:53.838+0000 7efe64c2d740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5574ac505db0 0x5574ac5081a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:31:53.842 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:53.838+0000 7efe64c2d740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x5574ac45f8f0 con 0x5574ac4fcfe0 2024-03-31T23:31:53.842 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:53.838+0000 7efe64c2d740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x5574ac4088d0 con 0x5574ac505db0 2024-03-31T23:31:53.842 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:53.838+0000 7efe64c2d740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5574ac45b990 con 0x5574ac4fc6c0 2024-03-31T23:31:53.842 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:53.838+0000 7efe5b7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5574ac4fcfe0 0x5574ac505870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:31:53.842 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:53.838+0000 7efe5bfff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5574ac4fc6c0 0x5574ac4fcaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:31:53.842 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:53.838+0000 7efe5bfff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5574ac4fc6c0 0x5574ac4fcaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.46:59216/0 (socket says 172.21.15.46:59216) 2024-03-31T23:31:53.842 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:53.838+0000 7efe5bfff640 1 -- 172.21.15.46:0/831783129 learned_addr learned my addr 172.21.15.46:0/831783129 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:31:53.842 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:53.838+0000 7efe5affd640 1 --2- 172.21.15.46:0/831783129 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5574ac505db0 0x5574ac5081a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:31:53.842 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:53.838+0000 7efe5bfff640 1 -- 172.21.15.46:0/831783129 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5574ac505db0 msgr2=0x5574ac5081a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:31:53.843 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:53.838+0000 7efe5bfff640 1 --2- 172.21.15.46:0/831783129 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5574ac505db0 0x5574ac5081a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:31:53.843 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:53.838+0000 7efe5bfff640 1 -- 172.21.15.46:0/831783129 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5574ac4fcfe0 msgr2=0x5574ac505870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:31:53.843 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:53.838+0000 7efe5bfff640 1 --2- 172.21.15.46:0/831783129 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5574ac4fcfe0 0x5574ac505870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:31:53.843 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:53.838+0000 7efe58ff9640 1 --2- 172.21.15.46:0/2761461825 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7efe440a2760 0x7efe440a4c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:31:53.843 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:53.838+0000 7efe5bfff640 1 -- 172.21.15.46:0/831783129 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5574ac174210 con 0x5574ac4fc6c0 2024-03-31T23:31:53.843 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:53.838+0000 7efe5affd640 1 --2- 172.21.15.46:0/831783129 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5574ac505db0 0x5574ac5081a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:31:53.843 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:53.838+0000 7efe5affd640 1 --2- 172.21.15.46:0/2761461825 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7efe440a2760 0x7efe440a4c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:31:53.843 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:53.839+0000 7efe58ff9640 1 -- 172.21.15.46:0/2761461825 <== mon.1 v2:172.21.15.162:3300/0 5 ==== osd_map(126..126 src has 1..126) ==== 9554+0+0 (secure 0 0 0) 0x7efe4c0cf0d0 con 0x7efe5c068dd0 2024-03-31T23:31:53.843 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:53.839+0000 7efe5bfff640 1 --2- 172.21.15.46:0/831783129 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5574ac4fc6c0 0x5574ac4fcaa0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7efe54006010 tx=0x7efe540323c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d162bb5d8c6e68cb server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:31:53.843 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:53.839+0000 7efe41ffb640 1 -- 172.21.15.46:0/831783129 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe54043070 con 0x5574ac4fc6c0 2024-03-31T23:31:53.843 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:53.839+0000 7efe41ffb640 1 -- 172.21.15.46:0/831783129 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7efe54032e80 con 0x5574ac4fc6c0 2024-03-31T23:31:53.843 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:53.839+0000 7efe41ffb640 1 -- 172.21.15.46:0/831783129 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe54039040 con 0x5574ac4fc6c0 2024-03-31T23:31:53.844 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:53.840+0000 7efe64c2d740 1 -- 172.21.15.46:0/831783129 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5574ac4fc6c0 msgr2=0x5574ac4fcaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:31:53.844 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:53.840+0000 7efe5affd640 1 --2- 172.21.15.46:0/2761461825 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7efe440a2760 0x7efe440a4c20 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x5574ac4f8400 tx=0x7efe50005f90 comp rx=0 tx=0).ready entity=mgr.15654 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:31:53.844 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:53.840+0000 7efe64c2d740 1 --2- 172.21.15.46:0/831783129 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5574ac4fc6c0 0x5574ac4fcaa0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7efe54006010 tx=0x7efe540323c0 comp rx=0 tx=0).stop 2024-03-31T23:31:53.844 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:53.840+0000 7efe64c2d740 1 -- 172.21.15.46:0/831783129 shutdown_connections 2024-03-31T23:31:53.844 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:53.840+0000 7efe64c2d740 1 --2- 172.21.15.46:0/831783129 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5574ac505db0 0x5574ac5081a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:31:53.844 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:53.840+0000 7efe64c2d740 1 --2- 172.21.15.46:0/831783129 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5574ac4fcfe0 0x5574ac505870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:31:53.844 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:53.840+0000 7efe64c2d740 1 --2- 172.21.15.46:0/831783129 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5574ac4fc6c0 0x5574ac4fcaa0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:31:53.844 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:53.840+0000 7efe64c2d740 1 -- 172.21.15.46:0/831783129 >> 172.21.15.46:0/831783129 conn(0x5574ac4f6fe0 msgr2=0x5574ac4f73c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:31:53.844 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:53.840+0000 7efe64c2d740 1 -- 172.21.15.46:0/831783129 shutdown_connections 2024-03-31T23:31:53.844 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:53.840+0000 7efe64c2d740 1 -- 172.21.15.46:0/831783129 wait complete. 2024-03-31T23:31:53.844 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:53.840+0000 7efe64c2d740 1 -- start start 2024-03-31T23:31:53.844 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:53.840+0000 7efe64c2d740 1 Processor -- start 2024-03-31T23:31:53.848 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:53.841+0000 7efe64c2d740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5574ac4fc6c0 0x5574ac502e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:31:53.848 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:53.841+0000 7efe64c2d740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5574ac4fcfe0 0x5574ac5033a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:31:53.848 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:53.841+0000 7efe64c2d740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5574ac505db0 0x5574ac5038e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:31:53.848 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:53.841+0000 7efe5b7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5574ac4fc6c0 0x5574ac502e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:31:53.848 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:53.841+0000 7efe64c2d740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x5574ac508980 con 0x5574ac4fc6c0 2024-03-31T23:31:53.848 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:53.841+0000 7efe5b7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5574ac4fc6c0 0x5574ac502e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55302/0 (socket says 172.21.15.46:55302) 2024-03-31T23:31:53.848 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:53.841+0000 7efe64c2d740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x5574ac3fdf80 con 0x5574ac4fcfe0 2024-03-31T23:31:53.848 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:53.841+0000 7efe5b7fe640 1 -- 172.21.15.46:0/2972095865 learned_addr learned my addr 172.21.15.46:0/2972095865 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:31:53.849 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:53.841+0000 7efe64c2d740 1 -- 172.21.15.46:0/2972095865 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5574ac508b00 con 0x5574ac505db0 2024-03-31T23:31:53.849 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:53.841+0000 7efe5affd640 1 --2- 172.21.15.46:0/2972095865 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5574ac4fcfe0 0x5574ac5033a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:31:53.849 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:53.841+0000 7efe5bfff640 1 --2- 172.21.15.46:0/2972095865 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5574ac505db0 0x5574ac5038e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:31:53.849 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:53.841+0000 7efe64c2d740 10 client.0 _ll_register_callbacks cb 0x5574ac456290 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T23:31:53.849 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:53.841+0000 7efe5b7fe640 1 -- 172.21.15.46:0/2972095865 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5574ac505db0 msgr2=0x5574ac5038e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:31:53.849 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:53.841+0000 7efe5b7fe640 1 --2- 172.21.15.46:0/2972095865 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5574ac505db0 0x5574ac5038e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:31:53.849 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:53.841+0000 7efe5b7fe640 1 -- 172.21.15.46:0/2972095865 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5574ac4fcfe0 msgr2=0x5574ac5033a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:31:53.849 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:53.841+0000 7efe5b7fe640 1 --2- 172.21.15.46:0/2972095865 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5574ac4fcfe0 0x5574ac5033a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:31:53.849 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:53.841+0000 7efe5b7fe640 1 -- 172.21.15.46:0/2972095865 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5574ac240620 con 0x5574ac4fc6c0 2024-03-31T23:31:53.849 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:53.842+0000 7efe5b7fe640 1 --2- 172.21.15.46:0/2972095865 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5574ac4fc6c0 0x5574ac502e60 secure :-1 s=READY pgs=975 cs=0 l=1 rev1=1 crypto rx=0x7efe4c09b0e0 tx=0x7efe4c097080 comp rx=0 tx=0).ready entity=mon.0 client_cookie=788aaddfcf11a6a4 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:31:53.849 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:53.842+0000 7efe3affd640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-31T23:31:53.849 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:53.842+0000 7efe3affd640 1 -- 172.21.15.46:0/2972095865 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe4c003b30 con 0x5574ac4fc6c0 2024-03-31T23:31:53.850 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:53.842+0000 7efe3affd640 1 -- 172.21.15.46:0/2972095865 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(126..126 src has 1..126) ==== 9554+0+0 (secure 0 0 0) 0x7efe4c097a50 con 0x5574ac4fc6c0 2024-03-31T23:31:53.850 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:53.842+0000 7efe64c2d740 1 -- 172.21.15.46:0/2972095865 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efe54032e80 con 0x5574ac4fc6c0 2024-03-31T23:31:53.850 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:53.842+0000 7efe3affd640 1 -- 172.21.15.46:0/2972095865 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=48) ==== 24+0+0 (secure 0 0 0) 0x7efe4c09bae0 con 0x5574ac4fc6c0 2024-03-31T23:31:53.850 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:53.842+0000 7efe64c2d740 10 client.17775 fetch_fsmap learned FSMap version 48 2024-03-31T23:31:53.850 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:53.842+0000 7efe64c2d740 1 -- 172.21.15.46:0/2972095865 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=48,monmap=4+}) -- 0x5574ac5148c0 con 0x5574ac4fc6c0 2024-03-31T23:31:53.850 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:53.842+0000 7efe3affd640 1 -- 172.21.15.46:0/2972095865 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7efe4c09b3e0 con 0x5574ac4fc6c0 2024-03-31T23:31:53.850 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:53.843+0000 7efe3affd640 1 -- 172.21.15.46:0/2972095865 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 48) ==== 8350+0+0 (secure 0 0 0) 0x7efe4c0cc630 con 0x5574ac4fc6c0 2024-03-31T23:31:53.850 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:53.843+0000 7efe64c2d740 10 client.17775 fetch_fsmap finished waiting for FSMap version 48 2024-03-31T23:31:53.850 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:53.843+0000 7efe64c2d740 10 client.17775 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-31T23:31:53.850 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:53.843+0000 7efe64c2d740 20 client.17775 populate_metadata read hostname 'smithi046' 2024-03-31T23:31:53.850 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:53.843+0000 7efe64c2d740 1 --2- 172.21.15.46:0/2972095865 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x5574ac515d30 0x5574ac518140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:31:53.850 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:53.843+0000 7efe64c2d740 4 client.17775 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-31T23:31:53.851 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:53.843+0000 7efe64c2d740 1 -- 172.21.15.46:0/2972095865 --> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7efe54032e80 con 0x5574ac515d30 2024-03-31T23:31:53.851 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:53.843+0000 7efe5affd640 1 --2- 172.21.15.46:0/2972095865 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x5574ac515d30 0x5574ac518140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:31:53.851 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:53.843+0000 7efe3affd640 10 client.17775 ms_handle_connect on v2:172.21.15.142:6838/2519143918 2024-03-31T23:31:53.851 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:53.843+0000 7efe5affd640 1 --2- 172.21.15.46:0/2972095865 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x5574ac515d30 0x5574ac518140 crc :-1 s=READY pgs=256 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=36040faab131c323 server_cookie=9710586487f7610c in_seq=0 out_seq=0 2024-03-31T23:31:53.851 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:53.844+0000 7efe3affd640 1 -- 172.21.15.46:0/2972095865 <== mds.0 v2:172.21.15.142:6838/2519143918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x7efe54032e80 con 0x5574ac515d30 2024-03-31T23:31:53.851 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:53.844+0000 7efe3affd640 10 client.17775 handle_command_reply: tid=0 2024-03-31T23:31:53.851 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:53.844+0000 7efe64c2d740 1 client.17775 shutdown 2024-03-31T23:31:53.851 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:53.845+0000 7efe64c2d740 1 -- 172.21.15.46:0/2972095865 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5574ac4fc6c0 msgr2=0x5574ac502e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:31:53.851 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:53.845+0000 7efe64c2d740 1 --2- 172.21.15.46:0/2972095865 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5574ac4fc6c0 0x5574ac502e60 secure :-1 s=READY pgs=975 cs=0 l=1 rev1=1 crypto rx=0x7efe4c09b0e0 tx=0x7efe4c097080 comp rx=0 tx=0).stop 2024-03-31T23:31:53.851 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:53.845+0000 7efe64c2d740 1 -- 172.21.15.46:0/2972095865 shutdown_connections 2024-03-31T23:31:53.851 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:53.845+0000 7efe64c2d740 1 --2- 172.21.15.46:0/2972095865 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x5574ac515d30 0x5574ac518140 crc :-1 s=READY pgs=256 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:31:53.851 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:53.845+0000 7efe64c2d740 1 --2- 172.21.15.46:0/2972095865 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5574ac505db0 0x5574ac5038e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:31:53.851 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:53.845+0000 7efe64c2d740 1 --2- 172.21.15.46:0/2972095865 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5574ac4fcfe0 0x5574ac5033a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:31:53.852 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:53.845+0000 7efe64c2d740 1 --2- 172.21.15.46:0/2972095865 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5574ac4fc6c0 0x5574ac502e60 unknown :-1 s=CLOSED pgs=975 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:31:53.852 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:53.845+0000 7efe3affd640 0 client.17775 ms_handle_reset on v2:172.21.15.142:6838/2519143918 2024-03-31T23:31:53.852 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:53.845+0000 7efe64c2d740 1 -- 172.21.15.46:0/2972095865 >> 172.21.15.46:0/2972095865 conn(0x5574ac4f6fe0 msgr2=0x5574ac4f8d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:31:53.852 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:53.845+0000 7efe64c2d740 1 -- 172.21.15.46:0/2972095865 shutdown_connections 2024-03-31T23:31:53.852 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:53.845+0000 7efe64c2d740 1 -- 172.21.15.46:0/2972095865 wait complete. 2024-03-31T23:31:53.852 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:53.845+0000 7efe64c2d740 20 client.17775 trim_cache size 0 max 16384 2024-03-31T23:31:53.870 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:53.867+0000 7efe64c2d740 1 Processor -- start 2024-03-31T23:31:53.870 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:53.867+0000 7efe64c2d740 1 -- start start 2024-03-31T23:31:53.870 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:53.867+0000 7efe64c2d740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5574ac4fc6c0 0x5574ac505590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:31:53.870 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:53.867+0000 7efe64c2d740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5574ac4fcfe0 0x5574ac511720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:31:53.870 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:53.868+0000 7efe64c2d740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5574ac505db0 0x5574ac51da50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:31:53.870 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:53.868+0000 7efe64c2d740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7efe5c1b3780 con 0x5574ac505db0 2024-03-31T23:31:53.870 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:53.868+0000 7efe64c2d740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7efe5c1b1020 con 0x5574ac4fc6c0 2024-03-31T23:31:53.870 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:53.868+0000 7efe64c2d740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7efe5c1aee90 con 0x5574ac4fcfe0 2024-03-31T23:31:53.870 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:53.868+0000 7efe5bfff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5574ac505db0 0x5574ac51da50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:31:53.870 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:53.868+0000 7efe5bfff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5574ac505db0 0x5574ac51da50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55314/0 (socket says 172.21.15.46:55314) 2024-03-31T23:31:53.871 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:53.868+0000 7efe5bfff640 1 -- 172.21.15.46:0/336382556 learned_addr learned my addr 172.21.15.46:0/336382556 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:31:53.871 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:53.868+0000 7efe5affd640 1 --2- 172.21.15.46:0/336382556 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5574ac4fcfe0 0x5574ac511720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:31:53.871 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:53.868+0000 7efe5b7fe640 1 --2- 172.21.15.46:0/336382556 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5574ac4fc6c0 0x5574ac505590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:31:53.871 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:53.868+0000 7efe5bfff640 1 -- 172.21.15.46:0/336382556 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5574ac4fcfe0 msgr2=0x5574ac511720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:31:53.871 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:53.868+0000 7efe5bfff640 1 --2- 172.21.15.46:0/336382556 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5574ac4fcfe0 0x5574ac511720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:31:53.871 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:53.868+0000 7efe5bfff640 1 -- 172.21.15.46:0/336382556 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5574ac4fc6c0 msgr2=0x5574ac505590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:31:53.871 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:53.868+0000 7efe5bfff640 1 --2- 172.21.15.46:0/336382556 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5574ac4fc6c0 0x5574ac505590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:31:53.871 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:53.868+0000 7efe5bfff640 1 -- 172.21.15.46:0/336382556 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5574ac501e10 con 0x5574ac505db0 2024-03-31T23:31:53.871 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:53.868+0000 7efe5bfff640 1 --2- 172.21.15.46:0/336382556 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5574ac505db0 0x5574ac51da50 secure :-1 s=READY pgs=976 cs=0 l=1 rev1=1 crypto rx=0x7efe5402c710 tx=0x7efe5402c100 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b72580c921c0de27 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:31:53.871 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:53.869+0000 7efe417fa640 1 -- 172.21.15.46:0/336382556 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe5404c070 con 0x5574ac505db0 2024-03-31T23:31:53.871 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:53.869+0000 7efe417fa640 1 -- 172.21.15.46:0/336382556 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7efe54032e80 con 0x5574ac505db0 2024-03-31T23:31:53.871 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:53.869+0000 7efe417fa640 1 -- 172.21.15.46:0/336382556 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe54039070 con 0x5574ac505db0 2024-03-31T23:31:53.875 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:53.870+0000 7efe64c2d740 1 -- 172.21.15.46:0/336382556 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5574ac505db0 msgr2=0x5574ac51da50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:31:53.875 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:53.870+0000 7efe64c2d740 1 --2- 172.21.15.46:0/336382556 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5574ac505db0 0x5574ac51da50 secure :-1 s=READY pgs=976 cs=0 l=1 rev1=1 crypto rx=0x7efe5402c710 tx=0x7efe5402c100 comp rx=0 tx=0).stop 2024-03-31T23:31:53.875 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:53.870+0000 7efe64c2d740 1 -- 172.21.15.46:0/336382556 shutdown_connections 2024-03-31T23:31:53.875 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:53.870+0000 7efe64c2d740 1 --2- 172.21.15.46:0/336382556 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5574ac505db0 0x5574ac51da50 unknown :-1 s=CLOSED pgs=976 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:31:53.875 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:53.870+0000 7efe64c2d740 1 --2- 172.21.15.46:0/336382556 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5574ac4fcfe0 0x5574ac511720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:31:53.875 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:53.870+0000 7efe64c2d740 1 --2- 172.21.15.46:0/336382556 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5574ac4fc6c0 0x5574ac505590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:31:53.875 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:53.870+0000 7efe64c2d740 1 -- 172.21.15.46:0/336382556 >> 172.21.15.46:0/336382556 conn(0x5574ac4f6fe0 msgr2=0x5574ac4f5f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:31:53.875 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:53.870+0000 7efe64c2d740 1 -- 172.21.15.46:0/336382556 shutdown_connections 2024-03-31T23:31:53.875 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:53.870+0000 7efe64c2d740 1 -- 172.21.15.46:0/336382556 wait complete. 2024-03-31T23:31:53.875 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:53.870+0000 7efe64c2d740 1 -- start start 2024-03-31T23:31:53.876 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:53.870+0000 7efe64c2d740 1 Processor -- start 2024-03-31T23:31:53.876 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:53.871+0000 7efe64c2d740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5574ac4fc6c0 0x5574ac50dd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:31:53.876 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:53.871+0000 7efe64c2d740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5574ac4fcfe0 0x5574ac50ad20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:31:53.876 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:53.871+0000 7efe64c2d740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5574ac505db0 0x5574ac50b260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:31:53.876 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:53.871+0000 7efe64c2d740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7efe5c1af4e0 con 0x5574ac4fc6c0 2024-03-31T23:31:53.876 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:53.871+0000 7efe64c2d740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7efe5c1b0950 con 0x5574ac505db0 2024-03-31T23:31:53.876 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:53.871+0000 7efe64c2d740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7efe5c1afbb0 con 0x5574ac4fcfe0 2024-03-31T23:31:53.876 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:53.871+0000 7efe64c2d740 10 client.0 _ll_register_callbacks cb 0x7efe4c00e940 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T23:31:53.876 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:53.871+0000 7efe5b7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5574ac4fc6c0 0x5574ac50dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:31:53.876 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:53.871+0000 7efe5affd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5574ac4fcfe0 0x5574ac50ad20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:31:53.876 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:53.871+0000 7efe5b7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5574ac4fc6c0 0x5574ac50dd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55330/0 (socket says 172.21.15.46:55330) 2024-03-31T23:31:53.877 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:53.871+0000 7efe5affd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5574ac4fcfe0 0x5574ac50ad20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.46:59252/0 (socket says 172.21.15.46:59252) 2024-03-31T23:31:53.877 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:53.871+0000 7efe5b7fe640 1 -- 172.21.15.46:0/1828718674 learned_addr learned my addr 172.21.15.46:0/1828718674 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:31:53.877 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:53.871+0000 7efe5bfff640 1 --2- 172.21.15.46:0/1828718674 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5574ac505db0 0x5574ac50b260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:31:53.877 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:53.871+0000 7efe5affd640 1 -- 172.21.15.46:0/1828718674 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5574ac505db0 msgr2=0x5574ac50b260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:31:53.877 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:53.871+0000 7efe5affd640 1 --2- 172.21.15.46:0/1828718674 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5574ac505db0 0x5574ac50b260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:31:53.877 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:53.871+0000 7efe5affd640 1 -- 172.21.15.46:0/1828718674 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5574ac4fc6c0 msgr2=0x5574ac50dd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:31:53.877 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:53.871+0000 7efe5affd640 1 --2- 172.21.15.46:0/1828718674 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5574ac4fc6c0 0x5574ac50dd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:31:53.877 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:53.871+0000 7efe5affd640 1 -- 172.21.15.46:0/1828718674 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5574ac50e310 con 0x5574ac4fcfe0 2024-03-31T23:31:53.877 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:53.871+0000 7efe5b7fe640 1 --2- 172.21.15.46:0/1828718674 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5574ac4fc6c0 0x5574ac50dd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:31:53.877 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:53.872+0000 7efe5affd640 1 --2- 172.21.15.46:0/1828718674 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5574ac4fcfe0 0x5574ac50ad20 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x5574ac5026c0 tx=0x7efe500046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=eee6082bc4b0a7e8 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:31:53.877 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:53.872+0000 7efe3affd640 10 client.0 ms_handle_connect on v2:172.21.15.162:3300/0 2024-03-31T23:31:53.877 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:53.872+0000 7efe3affd640 1 -- 172.21.15.46:0/1828718674 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe5001ddd0 con 0x5574ac4fcfe0 2024-03-31T23:31:53.878 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:53.872+0000 7efe64c2d740 1 -- 172.21.15.46:0/1828718674 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efe54032e80 con 0x5574ac4fcfe0 2024-03-31T23:31:53.878 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:53.872+0000 7efe3affd640 1 -- 172.21.15.46:0/1828718674 <== mon.1 v2:172.21.15.162:3300/0 2 ==== osd_map(126..126 src has 1..126) ==== 9554+0+0 (secure 0 0 0) 0x7efe50005020 con 0x5574ac4fcfe0 2024-03-31T23:31:53.878 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:53.872+0000 7efe3affd640 1 -- 172.21.15.46:0/1828718674 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_get_version_reply(handle=1 version=48) ==== 24+0+0 (secure 0 0 0) 0x7efe5003b050 con 0x5574ac4fcfe0 2024-03-31T23:31:53.878 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:53.872+0000 7efe64c2d740 10 client.27502 fetch_fsmap learned FSMap version 48 2024-03-31T23:31:53.878 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:53.872+0000 7efe64c2d740 1 -- 172.21.15.46:0/1828718674 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,fsmap=48,monmap=4+}) -- 0x5574ac57cad0 con 0x5574ac4fcfe0 2024-03-31T23:31:53.878 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:53.872+0000 7efe3affd640 1 -- 172.21.15.46:0/1828718674 <== mon.1 v2:172.21.15.162:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7efe54032e80 con 0x5574ac4fcfe0 2024-03-31T23:31:53.878 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:53.872+0000 7efe3affd640 1 -- 172.21.15.46:0/1828718674 <== mon.1 v2:172.21.15.162:3300/0 5 ==== fsmap(e 48) ==== 8350+0+0 (secure 0 0 0) 0x7efe5002b050 con 0x5574ac4fcfe0 2024-03-31T23:31:53.878 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:53.872+0000 7efe64c2d740 10 client.27502 fetch_fsmap finished waiting for FSMap version 48 2024-03-31T23:31:53.878 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:53.872+0000 7efe64c2d740 10 client.27502 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-31T23:31:53.878 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:53.872+0000 7efe64c2d740 20 client.27502 populate_metadata read hostname 'smithi046' 2024-03-31T23:31:53.878 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:53.873+0000 7efe64c2d740 1 --2- 172.21.15.46:0/1828718674 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x5574ac515d30 0x5574ac57f270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:31:53.878 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:53.873+0000 7efe64c2d740 4 client.27502 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-31T23:31:53.878 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:53.873+0000 7efe64c2d740 1 -- 172.21.15.46:0/1828718674 --> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5574ac473120 con 0x5574ac515d30 2024-03-31T23:31:53.878 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:53.873+0000 7efe5b7fe640 1 --2- 172.21.15.46:0/1828718674 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x5574ac515d30 0x5574ac57f270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:31:53.879 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:53.873+0000 7efe3affd640 10 client.27502 ms_handle_connect on v2:172.21.15.142:6838/2519143918 2024-03-31T23:31:53.879 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:53.873+0000 7efe5b7fe640 1 --2- 172.21.15.46:0/1828718674 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x5574ac515d30 0x5574ac57f270 crc :-1 s=READY pgs=257 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=23c71895b92243b6 server_cookie=acbdd7ce145189ed in_seq=0 out_seq=0 2024-03-31T23:31:54.076 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:54.074+0000 7efe3affd640 1 -- 172.21.15.46:0/1828718674 <== mds.0 v2:172.21.15.142:6838/2519143918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5574ac473120 con 0x5574ac515d30 2024-03-31T23:31:54.076 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:54.074+0000 7efe3affd640 10 client.27502 handle_command_reply: tid=0 2024-03-31T23:31:54.076 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:54.074+0000 7efe64c2d740 1 client.27502 shutdown 2024-03-31T23:31:54.076 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:54.074+0000 7efe64c2d740 1 -- 172.21.15.46:0/1828718674 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5574ac4fcfe0 msgr2=0x5574ac50ad20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:31:54.076 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:54.074+0000 7efe64c2d740 1 --2- 172.21.15.46:0/1828718674 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5574ac4fcfe0 0x5574ac50ad20 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x5574ac5026c0 tx=0x7efe500046c0 comp rx=0 tx=0).stop 2024-03-31T23:31:54.076 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:54.074+0000 7efe64c2d740 1 -- 172.21.15.46:0/1828718674 shutdown_connections 2024-03-31T23:31:54.076 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:54.074+0000 7efe64c2d740 1 --2- 172.21.15.46:0/1828718674 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x5574ac515d30 0x5574ac57f270 crc :-1 s=READY pgs=257 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:31:54.076 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:54.074+0000 7efe64c2d740 1 --2- 172.21.15.46:0/1828718674 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5574ac505db0 0x5574ac50b260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:31:54.076 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:54.074+0000 7efe64c2d740 1 --2- 172.21.15.46:0/1828718674 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5574ac4fcfe0 0x5574ac50ad20 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:31:54.077 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:54.074+0000 7efe64c2d740 1 --2- 172.21.15.46:0/1828718674 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5574ac4fc6c0 0x5574ac50dd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:31:54.077 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:54.074+0000 7efe3affd640 0 client.27502 ms_handle_reset on v2:172.21.15.142:6838/2519143918 2024-03-31T23:31:54.077 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:54.074+0000 7efe64c2d740 1 -- 172.21.15.46:0/1828718674 >> 172.21.15.46:0/1828718674 conn(0x5574ac4f6fe0 msgr2=0x5574ac4f8430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:31:54.077 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:54.075+0000 7efe64c2d740 1 -- 172.21.15.46:0/1828718674 shutdown_connections 2024-03-31T23:31:54.077 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:54.075+0000 7efe64c2d740 1 -- 172.21.15.46:0/1828718674 wait complete. 2024-03-31T23:31:54.077 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:54.075+0000 7efe64c2d740 20 client.27502 trim_cache size 0 max 16384 2024-03-31T23:31:54.077 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout:{ 2024-03-31T23:31:54.077 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout: "message": "", 2024-03-31T23:31:54.077 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout: "return_code": 0 2024-03-31T23:31:54.077 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout:} 2024-03-31T23:31:54.078 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:54.075+0000 7efe62047640 1 -- 172.21.15.46:0/2761461825 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7efe440a2760 msgr2=0x7efe440a4c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:31:54.078 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:54.075+0000 7efe62047640 1 --2- 172.21.15.46:0/2761461825 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7efe440a2760 0x7efe440a4c20 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x5574ac4f8400 tx=0x7efe50005f90 comp rx=0 tx=0).stop 2024-03-31T23:31:54.078 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:54.075+0000 7efe62047640 1 -- 172.21.15.46:0/2761461825 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7efe5c068dd0 msgr2=0x7efe5c100970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:31:54.078 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:54.075+0000 7efe62047640 1 --2- 172.21.15.46:0/2761461825 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7efe5c068dd0 0x7efe5c100970 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7efe4c00e9f0 tx=0x7efe4c00eeb0 comp rx=0 tx=0).stop 2024-03-31T23:31:54.078 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:54.076+0000 7efe62047640 1 -- 172.21.15.46:0/2761461825 shutdown_connections 2024-03-31T23:31:54.078 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:54.076+0000 7efe62047640 1 --2- 172.21.15.46:0/2761461825 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7efe440a2760 0x7efe440a4c20 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:31:54.078 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:54.076+0000 7efe62047640 1 --2- 172.21.15.46:0/2761461825 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efe5c107840 0x7efe5c1013f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:31:54.079 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:54.076+0000 7efe62047640 1 --2- 172.21.15.46:0/2761461825 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7efe5c069770 0x7efe5c100eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:31:54.079 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:54.076+0000 7efe62047640 1 --2- 172.21.15.46:0/2761461825 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7efe5c068dd0 0x7efe5c100970 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:31:54.079 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:54.076+0000 7efe62047640 1 -- 172.21.15.46:0/2761461825 >> 172.21.15.46:0/2761461825 conn(0x7efe5c0774a0 msgr2=0x7efe5c0fe8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:31:54.085 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:54.084+0000 7efe62047640 1 -- 172.21.15.46:0/2761461825 shutdown_connections 2024-03-31T23:31:54.085 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:31:54.084+0000 7efe62047640 1 -- 172.21.15.46:0/2761461825 wait complete. 2024-03-31T23:31:54.295 INFO:tasks.workunit.client.0.smithi046.stdout: 10 66486 8.44 MB/sec execute 299 sec latency 156.015 ms 2024-03-31T23:31:55.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:31:54 smithi162 ceph-mon[36237]: pgmap v638: 129 pgs: 129 active+clean; 991 MiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 4.4 MiB/s wr, 291 op/s 2024-03-31T23:31:55.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:31:54 smithi046 ceph-mon[32498]: pgmap v638: 129 pgs: 129 active+clean; 991 MiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 4.4 MiB/s wr, 291 op/s 2024-03-31T23:31:55.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:31:54 smithi142 ceph-mon[35271]: pgmap v638: 129 pgs: 129 active+clean; 991 MiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 4.4 MiB/s wr, 291 op/s 2024-03-31T23:31:55.295 INFO:tasks.workunit.client.0.smithi046.stdout: 10 66594 8.43 MB/sec execute 300 sec latency 130.834 ms 2024-03-31T23:31:56.295 INFO:tasks.workunit.client.0.smithi046.stdout: 10 66767 8.44 MB/sec execute 301 sec latency 70.960 ms 2024-03-31T23:31:57.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:31:56 smithi162 ceph-mon[36237]: pgmap v639: 129 pgs: 129 active+clean; 1007 MiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 6.7 MiB/s wr, 398 op/s 2024-03-31T23:31:57.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:31:56 smithi046 ceph-mon[32498]: pgmap v639: 129 pgs: 129 active+clean; 1007 MiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 6.7 MiB/s wr, 398 op/s 2024-03-31T23:31:57.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:31:56 smithi142 ceph-mon[35271]: pgmap v639: 129 pgs: 129 active+clean; 1007 MiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 6.7 MiB/s wr, 398 op/s 2024-03-31T23:31:57.295 INFO:tasks.workunit.client.0.smithi046.stdout: 10 66928 8.46 MB/sec execute 302 sec latency 93.426 ms 2024-03-31T23:31:58.295 INFO:tasks.workunit.client.0.smithi046.stdout: 10 67091 8.45 MB/sec execute 303 sec latency 93.999 ms 2024-03-31T23:31:59.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:31:58 smithi162 ceph-mon[36237]: pgmap v640: 129 pgs: 129 active+clean; 1007 MiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 4.3 MiB/s wr, 224 op/s 2024-03-31T23:31:59.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:31:58 smithi046 ceph-mon[32498]: pgmap v640: 129 pgs: 129 active+clean; 1007 MiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 4.3 MiB/s wr, 224 op/s 2024-03-31T23:31:59.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:31:58 smithi142 ceph-mon[35271]: pgmap v640: 129 pgs: 129 active+clean; 1007 MiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 4.3 MiB/s wr, 224 op/s 2024-03-31T23:31:59.295 INFO:tasks.workunit.client.0.smithi046.stdout: 10 67241 8.45 MB/sec execute 304 sec latency 67.519 ms 2024-03-31T23:32:00.295 INFO:tasks.workunit.client.0.smithi046.stdout: 10 67372 8.43 MB/sec execute 305 sec latency 171.986 ms 2024-03-31T23:32:01.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:32:00 smithi162 ceph-mon[36237]: pgmap v641: 129 pgs: 129 active+clean; 1018 MiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 7.0 MiB/s wr, 379 op/s 2024-03-31T23:32:01.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:32:00 smithi046 ceph-mon[32498]: pgmap v641: 129 pgs: 129 active+clean; 1018 MiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 7.0 MiB/s wr, 379 op/s 2024-03-31T23:32:01.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:32:00 smithi142 ceph-mon[35271]: pgmap v641: 129 pgs: 129 active+clean; 1018 MiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 7.0 MiB/s wr, 379 op/s 2024-03-31T23:32:01.295 INFO:tasks.workunit.client.0.smithi046.stdout: 10 67509 8.43 MB/sec execute 306 sec latency 99.763 ms 2024-03-31T23:32:02.295 INFO:tasks.workunit.client.0.smithi046.stdout: 10 67668 8.43 MB/sec execute 307 sec latency 80.341 ms 2024-03-31T23:32:03.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:32:02 smithi162 ceph-mon[36237]: pgmap v642: 129 pgs: 129 active+clean; 1018 MiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 4.9 MiB/s wr, 262 op/s 2024-03-31T23:32:03.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:32:02 smithi046 ceph-mon[32498]: pgmap v642: 129 pgs: 129 active+clean; 1018 MiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 4.9 MiB/s wr, 262 op/s 2024-03-31T23:32:03.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:32:02 smithi142 ceph-mon[35271]: pgmap v642: 129 pgs: 129 active+clean; 1018 MiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 4.9 MiB/s wr, 262 op/s 2024-03-31T23:32:03.295 INFO:tasks.workunit.client.0.smithi046.stdout: 10 67778 8.42 MB/sec execute 308 sec latency 148.400 ms 2024-03-31T23:32:04.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:32:03 smithi162 ceph-mon[36237]: osdmap e127: 12 total, 12 up, 12 in 2024-03-31T23:32:04.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:32:03 smithi046 ceph-mon[32498]: osdmap e127: 12 total, 12 up, 12 in 2024-03-31T23:32:04.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:32:03 smithi142 ceph-mon[35271]: osdmap e127: 12 total, 12 up, 12 in 2024-03-31T23:32:04.295 INFO:tasks.workunit.client.0.smithi046.stdout: 10 67922 8.42 MB/sec execute 309 sec latency 78.209 ms 2024-03-31T23:32:05.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:32:04 smithi162 ceph-mon[36237]: pgmap v644: 129 pgs: 129 active+clean; 1018 MiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 5.9 MiB/s wr, 314 op/s 2024-03-31T23:32:05.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:32:04 smithi046 ceph-mon[32498]: pgmap v644: 129 pgs: 129 active+clean; 1018 MiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 5.9 MiB/s wr, 314 op/s 2024-03-31T23:32:05.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:32:04 smithi142 ceph-mon[35271]: pgmap v644: 129 pgs: 129 active+clean; 1018 MiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 5.9 MiB/s wr, 314 op/s 2024-03-31T23:32:05.295 INFO:tasks.workunit.client.0.smithi046.stdout: 10 68089 8.43 MB/sec execute 310 sec latency 74.636 ms 2024-03-31T23:32:06.295 INFO:tasks.workunit.client.0.smithi046.stdout: 10 68240 8.43 MB/sec execute 311 sec latency 118.275 ms 2024-03-31T23:32:07.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:32:06 smithi046 ceph-mon[32498]: pgmap v645: 129 pgs: 129 active+clean; 1006 MiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 5.9 MiB/s wr, 349 op/s 2024-03-31T23:32:07.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:32:06 smithi142 ceph-mon[35271]: pgmap v645: 129 pgs: 129 active+clean; 1006 MiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 5.9 MiB/s wr, 349 op/s 2024-03-31T23:32:07.213 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:07.211+0000 7fcc009a0640 1 -- 172.21.15.46:0/1558223704 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcbfc100e10 msgr2=0x7fcbfc1052d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:32:07.213 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:07.211+0000 7fcc009a0640 1 --2- 172.21.15.46:0/1558223704 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcbfc100e10 0x7fcbfc1052d0 secure :-1 s=READY pgs=977 cs=0 l=1 rev1=1 crypto rx=0x7fcbf0009930 tx=0x7fcbf002bc30 comp rx=0 tx=0).stop 2024-03-31T23:32:07.213 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:07.211+0000 7fcc009a0640 1 -- 172.21.15.46:0/1558223704 shutdown_connections 2024-03-31T23:32:07.213 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:07.211+0000 7fcc009a0640 1 --2- 172.21.15.46:0/1558223704 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcbfc105a00 0x7fcbfc105e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:32:07.213 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:07.211+0000 7fcc009a0640 1 --2- 172.21.15.46:0/1558223704 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcbfc100e10 0x7fcbfc1052d0 unknown :-1 s=CLOSED pgs=977 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:32:07.213 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:07.211+0000 7fcc009a0640 1 --2- 172.21.15.46:0/1558223704 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcbfc1004f0 0x7fcbfc1008d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:32:07.213 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:07.211+0000 7fcc009a0640 1 -- 172.21.15.46:0/1558223704 >> 172.21.15.46:0/1558223704 conn(0x7fcbfc0fbf60 msgr2=0x7fcbfc0fe380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:32:07.213 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:07.211+0000 7fcc009a0640 1 -- 172.21.15.46:0/1558223704 shutdown_connections 2024-03-31T23:32:07.213 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:07.211+0000 7fcc009a0640 1 -- 172.21.15.46:0/1558223704 wait complete. 2024-03-31T23:32:07.214 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:07.212+0000 7fcc009a0640 1 Processor -- start 2024-03-31T23:32:07.214 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:07.212+0000 7fcc009a0640 1 -- start start 2024-03-31T23:32:07.215 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:07.212+0000 7fcc009a0640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcbfc1004f0 0x7fcbfc19e400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:32:07.215 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:07.212+0000 7fcc009a0640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcbfc100e10 0x7fcbfc19e940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:32:07.215 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:07.212+0000 7fcc009a0640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcbfc105a00 0x7fcbfc1a2cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:32:07.215 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:07.212+0000 7fcc009a0640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fcbfc117e00 con 0x7fcbfc105a00 2024-03-31T23:32:07.216 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:07.212+0000 7fcc009a0640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fcbfc117c80 con 0x7fcbfc1004f0 2024-03-31T23:32:07.216 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:07.212+0000 7fcc009a0640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fcbfc117f80 con 0x7fcbfc100e10 2024-03-31T23:32:07.216 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:07.213+0000 7fcbfbfff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcbfc105a00 0x7fcbfc1a2cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:32:07.216 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:07.213+0000 7fcbfaffd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcbfc100e10 0x7fcbfc19e940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:32:07.216 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:07.213+0000 7fcbfb7fe640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcbfc1004f0 0x7fcbfc19e400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:32:07.216 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:07.213+0000 7fcbfbfff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcbfc105a00 0x7fcbfc1a2cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:48268/0 (socket says 172.21.15.46:48268) 2024-03-31T23:32:07.216 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:07.213+0000 7fcbfbfff640 1 -- 172.21.15.46:0/3438549834 learned_addr learned my addr 172.21.15.46:0/3438549834 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:32:07.216 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:07.213+0000 7fcbfaffd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcbfc100e10 0x7fcbfc19e940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.46:51592/0 (socket says 172.21.15.46:51592) 2024-03-31T23:32:07.216 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:07.213+0000 7fcbfaffd640 1 -- 172.21.15.46:0/3438549834 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcbfc1004f0 msgr2=0x7fcbfc19e400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:32:07.216 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:07.213+0000 7fcbfaffd640 1 --2- 172.21.15.46:0/3438549834 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcbfc1004f0 0x7fcbfc19e400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:32:07.216 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:07.213+0000 7fcbfaffd640 1 -- 172.21.15.46:0/3438549834 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcbfc105a00 msgr2=0x7fcbfc1a2cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:32:07.216 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:07.213+0000 7fcbfaffd640 1 --2- 172.21.15.46:0/3438549834 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcbfc105a00 0x7fcbfc1a2cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:32:07.216 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:07.213+0000 7fcbfaffd640 1 -- 172.21.15.46:0/3438549834 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcbfc1a33f0 con 0x7fcbfc100e10 2024-03-31T23:32:07.216 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:07.213+0000 7fcbfb7fe640 1 --2- 172.21.15.46:0/3438549834 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcbfc1004f0 0x7fcbfc19e400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:32:07.216 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:07.213+0000 7fcbfaffd640 1 --2- 172.21.15.46:0/3438549834 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcbfc100e10 0x7fcbfc19e940 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fcbf0009230 tx=0x7fcbf00045e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:32:07.217 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:07.214+0000 7fcbf8ff9640 1 -- 172.21.15.46:0/3438549834 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbf0043020 con 0x7fcbfc100e10 2024-03-31T23:32:07.217 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:07.214+0000 7fcbf8ff9640 1 -- 172.21.15.46:0/3438549834 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fcbf002ca70 con 0x7fcbfc100e10 2024-03-31T23:32:07.217 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:07.214+0000 7fcc009a0640 1 -- 172.21.15.46:0/3438549834 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcbfc1a3680 con 0x7fcbfc100e10 2024-03-31T23:32:07.217 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:07.214+0000 7fcbf8ff9640 1 -- 172.21.15.46:0/3438549834 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbf0003ac0 con 0x7fcbfc100e10 2024-03-31T23:32:07.217 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:07.214+0000 7fcc009a0640 1 -- 172.21.15.46:0/3438549834 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcbfc1aaf70 con 0x7fcbfc100e10 2024-03-31T23:32:07.218 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:07.217+0000 7fcbf8ff9640 1 -- 172.21.15.46:0/3438549834 <== mon.1 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 25) ==== 137451+0+0 (secure 0 0 0) 0x7fcbf0046020 con 0x7fcbfc100e10 2024-03-31T23:32:07.218 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:07.217+0000 7fcc047e4740 1 Processor -- start 2024-03-31T23:32:07.218 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:07.217+0000 7fcc047e4740 1 -- start start 2024-03-31T23:32:07.220 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:07.218+0000 7fcc047e4740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56549b7198d0 0x56549b719cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:32:07.220 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:07.218+0000 7fcc047e4740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56549b71a1f0 0x56549b722a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:32:07.220 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:07.218+0000 7fcc047e4740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56549b722fc0 0x56549b7253b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:32:07.220 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:07.218+0000 7fcc047e4740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x56549b67c8f0 con 0x56549b7198d0 2024-03-31T23:32:07.220 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:07.218+0000 7fcc047e4740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x56549b6258d0 con 0x56549b722fc0 2024-03-31T23:32:07.220 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:07.218+0000 7fcc047e4740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x56549b678990 con 0x56549b71a1f0 2024-03-31T23:32:07.220 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:07.218+0000 7fcbfbfff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56549b7198d0 0x56549b719cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:32:07.220 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:07.218+0000 7fcbfb7fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56549b71a1f0 0x56549b722a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:32:07.220 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:07.218+0000 7fcbfbfff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56549b7198d0 0x56549b719cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:48272/0 (socket says 172.21.15.46:48272) 2024-03-31T23:32:07.221 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:07.218+0000 7fcbfb7fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56549b71a1f0 0x56549b722a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.46:51608/0 (socket says 172.21.15.46:51608) 2024-03-31T23:32:07.221 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:07.218+0000 7fcbfbfff640 1 -- 172.21.15.46:0/2064966600 learned_addr learned my addr 172.21.15.46:0/2064966600 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:32:07.221 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:07.218+0000 7fcbf8ff9640 1 --2- 172.21.15.46:0/3438549834 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fcbe40a2600 0x7fcbe40a4ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:32:07.221 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:07.218+0000 7fcbfaffd640 1 --2- 172.21.15.46:0/2064966600 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56549b722fc0 0x56549b7253b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:32:07.221 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:07.218+0000 7fcbf8ff9640 1 -- 172.21.15.46:0/3438549834 <== mon.1 v2:172.21.15.162:3300/0 5 ==== osd_map(127..127 src has 1..127) ==== 9642+0+0 (secure 0 0 0) 0x7fcbf00f0e50 con 0x7fcbfc100e10 2024-03-31T23:32:07.221 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:07.218+0000 7fcbfbfff640 1 --2- 172.21.15.46:0/3438549834 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fcbe40a2600 0x7fcbe40a4ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:32:07.221 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:07.218+0000 7fcbfb7fe640 1 -- 172.21.15.46:0/2064966600 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56549b722fc0 msgr2=0x56549b7253b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:32:07.221 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:07.218+0000 7fcbfb7fe640 1 --2- 172.21.15.46:0/2064966600 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56549b722fc0 0x56549b7253b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:32:07.221 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:07.218+0000 7fcbfb7fe640 1 -- 172.21.15.46:0/2064966600 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56549b7198d0 msgr2=0x56549b719cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:32:07.221 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:07.218+0000 7fcbfb7fe640 1 --2- 172.21.15.46:0/2064966600 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56549b7198d0 0x56549b719cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:32:07.221 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:07.218+0000 7fcbfb7fe640 1 -- 172.21.15.46:0/2064966600 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56549b391210 con 0x56549b71a1f0 2024-03-31T23:32:07.221 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:07.219+0000 7fcbfb7fe640 1 --2- 172.21.15.46:0/2064966600 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56549b71a1f0 0x56549b722a80 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fcbec010710 tx=0x7fcbec010bd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3d57f4cb3db1a78c server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:32:07.221 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:07.219+0000 7fcbfbfff640 1 --2- 172.21.15.46:0/3438549834 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fcbe40a2600 0x7fcbe40a4ac0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fcbf4013ac0 tx=0x7fcbf4004730 comp rx=0 tx=0).ready entity=mgr.15654 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:32:07.221 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:07.219+0000 7fcbe1ffb640 1 -- 172.21.15.46:0/2064966600 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbec003970 con 0x56549b71a1f0 2024-03-31T23:32:07.222 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:07.219+0000 7fcbe1ffb640 1 -- 172.21.15.46:0/2064966600 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fcbec00e8d0 con 0x56549b71a1f0 2024-03-31T23:32:07.222 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:07.219+0000 7fcbe1ffb640 1 -- 172.21.15.46:0/2064966600 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbec0045f0 con 0x56549b71a1f0 2024-03-31T23:32:07.225 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:07.220+0000 7fcc047e4740 1 -- 172.21.15.46:0/2064966600 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56549b71a1f0 msgr2=0x56549b722a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:32:07.225 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:07.220+0000 7fcc047e4740 1 --2- 172.21.15.46:0/2064966600 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56549b71a1f0 0x56549b722a80 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fcbec010710 tx=0x7fcbec010bd0 comp rx=0 tx=0).stop 2024-03-31T23:32:07.225 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:07.220+0000 7fcc047e4740 1 -- 172.21.15.46:0/2064966600 shutdown_connections 2024-03-31T23:32:07.225 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:07.220+0000 7fcc047e4740 1 --2- 172.21.15.46:0/2064966600 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56549b722fc0 0x56549b7253b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:32:07.225 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:07.220+0000 7fcc047e4740 1 --2- 172.21.15.46:0/2064966600 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56549b71a1f0 0x56549b722a80 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:32:07.225 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:07.220+0000 7fcc047e4740 1 --2- 172.21.15.46:0/2064966600 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56549b7198d0 0x56549b719cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:32:07.225 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:07.220+0000 7fcc047e4740 1 -- 172.21.15.46:0/2064966600 >> 172.21.15.46:0/2064966600 conn(0x56549b7141f0 msgr2=0x56549b7145d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:32:07.225 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:07.220+0000 7fcc047e4740 1 -- 172.21.15.46:0/2064966600 shutdown_connections 2024-03-31T23:32:07.225 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:07.220+0000 7fcc047e4740 1 -- 172.21.15.46:0/2064966600 wait complete. 2024-03-31T23:32:07.226 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:07.220+0000 7fcc047e4740 1 -- start start 2024-03-31T23:32:07.226 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:07.220+0000 7fcc047e4740 1 Processor -- start 2024-03-31T23:32:07.226 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:07.221+0000 7fcc047e4740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56549b7198d0 0x56549b720010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:32:07.226 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:07.221+0000 7fcc047e4740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56549b71a1f0 0x56549b720550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:32:07.226 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:07.221+0000 7fcc047e4740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56549b722fc0 0x56549b720a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:32:07.226 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:07.221+0000 7fcc047e4740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x56549b725bf0 con 0x56549b71a1f0 2024-03-31T23:32:07.226 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:07.221+0000 7fcc047e4740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x56549b61af80 con 0x56549b722fc0 2024-03-31T23:32:07.226 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:07.221+0000 7fcc047e4740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x56549b725d70 con 0x56549b7198d0 2024-03-31T23:32:07.226 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:07.221+0000 7fcbfb7fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56549b7198d0 0x56549b720010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:32:07.226 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:07.221+0000 7fcbfaffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56549b71a1f0 0x56549b720550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:32:07.226 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:07.221+0000 7fcbfb7fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56549b7198d0 0x56549b720010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.46:51622/0 (socket says 172.21.15.46:51622) 2024-03-31T23:32:07.226 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:07.221+0000 7fcbfb7fe640 1 -- 172.21.15.46:0/3444073408 learned_addr learned my addr 172.21.15.46:0/3444073408 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:32:07.227 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:07.221+0000 7fcc047e4740 10 client.0 _ll_register_callbacks cb 0x56549b673290 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T23:32:07.227 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:07.221+0000 7fcbfbfff640 1 --2- 172.21.15.46:0/3444073408 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56549b722fc0 0x56549b720a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:32:07.227 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:07.221+0000 7fcbfaffd640 1 -- 172.21.15.46:0/3444073408 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56549b7198d0 msgr2=0x56549b720010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:32:07.227 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:07.221+0000 7fcbfaffd640 1 --2- 172.21.15.46:0/3444073408 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56549b7198d0 0x56549b720010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:32:07.227 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:07.221+0000 7fcbfaffd640 1 -- 172.21.15.46:0/3444073408 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56549b722fc0 msgr2=0x56549b720a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:32:07.227 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:07.221+0000 7fcbfaffd640 1 --2- 172.21.15.46:0/3444073408 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56549b722fc0 0x56549b720a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:32:07.227 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:07.221+0000 7fcbfaffd640 1 -- 172.21.15.46:0/3444073408 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56549b45d620 con 0x56549b71a1f0 2024-03-31T23:32:07.227 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:07.222+0000 7fcbfaffd640 1 --2- 172.21.15.46:0/3444073408 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56549b71a1f0 0x56549b720550 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 crypto rx=0x7fcbf00ee220 tx=0x7fcbf008c1b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e5e624973945623 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:32:07.227 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:07.222+0000 7fcbdaffd640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-31T23:32:07.227 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:07.222+0000 7fcbdaffd640 1 -- 172.21.15.46:0/3444073408 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbf00bd280 con 0x56549b71a1f0 2024-03-31T23:32:07.227 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:07.222+0000 7fcbdaffd640 1 -- 172.21.15.46:0/3444073408 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(127..127 src has 1..127) ==== 9642+0+0 (secure 0 0 0) 0x7fcbf00bb320 con 0x56549b71a1f0 2024-03-31T23:32:07.227 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:07.222+0000 7fcc047e4740 1 -- 172.21.15.46:0/3444073408 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcbec00e8d0 con 0x56549b71a1f0 2024-03-31T23:32:07.227 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:07.222+0000 7fcbdaffd640 1 -- 172.21.15.46:0/3444073408 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=48) ==== 24+0+0 (secure 0 0 0) 0x7fcbf00c13d0 con 0x56549b71a1f0 2024-03-31T23:32:07.227 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:07.222+0000 7fcc047e4740 10 client.17793 fetch_fsmap learned FSMap version 48 2024-03-31T23:32:07.228 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:07.222+0000 7fcc047e4740 1 -- 172.21.15.46:0/3444073408 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=48,monmap=4+}) -- 0x56549b731ac0 con 0x56549b71a1f0 2024-03-31T23:32:07.228 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:07.222+0000 7fcbdaffd640 1 -- 172.21.15.46:0/3444073408 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fcbec00e8d0 con 0x56549b71a1f0 2024-03-31T23:32:07.228 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:07.222+0000 7fcbdaffd640 1 -- 172.21.15.46:0/3444073408 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 48) ==== 8350+0+0 (secure 0 0 0) 0x7fcbf00053b0 con 0x56549b71a1f0 2024-03-31T23:32:07.228 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:07.223+0000 7fcc047e4740 10 client.17793 fetch_fsmap finished waiting for FSMap version 48 2024-03-31T23:32:07.228 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:07.223+0000 7fcc047e4740 10 client.17793 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-31T23:32:07.228 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:07.223+0000 7fcc047e4740 20 client.17793 populate_metadata read hostname 'smithi046' 2024-03-31T23:32:07.228 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:07.223+0000 7fcc047e4740 1 --2- 172.21.15.46:0/3444073408 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x56549b732f50 0x56549b7353e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:32:07.228 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:07.223+0000 7fcc047e4740 4 client.17793 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-31T23:32:07.228 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:07.223+0000 7fcc047e4740 1 -- 172.21.15.46:0/3444073408 --> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56549b690120 con 0x56549b732f50 2024-03-31T23:32:07.228 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:07.223+0000 7fcbfb7fe640 1 --2- 172.21.15.46:0/3444073408 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x56549b732f50 0x56549b7353e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:32:07.228 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:07.223+0000 7fcbfb7fe640 1 --2- 172.21.15.46:0/3444073408 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x56549b732f50 0x56549b7353e0 crc :-1 s=READY pgs=258 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7b66eb465626dcb4 server_cookie=3914b8ee1aa0800 in_seq=0 out_seq=0 2024-03-31T23:32:07.228 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:07.223+0000 7fcbdaffd640 10 client.17793 ms_handle_connect on v2:172.21.15.142:6838/2519143918 2024-03-31T23:32:07.229 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:07.225+0000 7fcbdaffd640 1 -- 172.21.15.46:0/3444073408 <== mds.0 v2:172.21.15.142:6838/2519143918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x56549b690120 con 0x56549b732f50 2024-03-31T23:32:07.229 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:07.225+0000 7fcbdaffd640 10 client.17793 handle_command_reply: tid=0 2024-03-31T23:32:07.229 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:07.226+0000 7fcc047e4740 1 client.17793 shutdown 2024-03-31T23:32:07.229 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:07.226+0000 7fcc047e4740 1 -- 172.21.15.46:0/3444073408 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56549b71a1f0 msgr2=0x56549b720550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:32:07.229 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:07.226+0000 7fcc047e4740 1 --2- 172.21.15.46:0/3444073408 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56549b71a1f0 0x56549b720550 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 crypto rx=0x7fcbf00ee220 tx=0x7fcbf008c1b0 comp rx=0 tx=0).stop 2024-03-31T23:32:07.229 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:07.226+0000 7fcc047e4740 1 -- 172.21.15.46:0/3444073408 shutdown_connections 2024-03-31T23:32:07.230 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:07.226+0000 7fcc047e4740 1 --2- 172.21.15.46:0/3444073408 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x56549b732f50 0x56549b7353e0 crc :-1 s=READY pgs=258 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:32:07.230 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:07.226+0000 7fcbdaffd640 0 client.17793 ms_handle_reset on v2:172.21.15.142:6838/2519143918 2024-03-31T23:32:07.230 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:07.226+0000 7fcc047e4740 1 --2- 172.21.15.46:0/3444073408 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56549b722fc0 0x56549b720a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:32:07.230 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:07.226+0000 7fcc047e4740 1 --2- 172.21.15.46:0/3444073408 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56549b71a1f0 0x56549b720550 unknown :-1 s=CLOSED pgs=978 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:32:07.230 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:07.226+0000 7fcc047e4740 1 --2- 172.21.15.46:0/3444073408 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56549b7198d0 0x56549b720010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:32:07.230 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:07.226+0000 7fcc047e4740 1 -- 172.21.15.46:0/3444073408 >> 172.21.15.46:0/3444073408 conn(0x56549b7141f0 msgr2=0x56549b715f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:32:07.230 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:07.226+0000 7fcc047e4740 1 -- 172.21.15.46:0/3444073408 shutdown_connections 2024-03-31T23:32:07.230 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:07.226+0000 7fcc047e4740 1 -- 172.21.15.46:0/3444073408 wait complete. 2024-03-31T23:32:07.230 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:07.226+0000 7fcc047e4740 20 client.17793 trim_cache size 0 max 16384 2024-03-31T23:32:07.251 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:07.249+0000 7fcc047e4740 1 Processor -- start 2024-03-31T23:32:07.252 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:07.249+0000 7fcc047e4740 1 -- start start 2024-03-31T23:32:07.252 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:07.249+0000 7fcc047e4740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56549b7198d0 0x56549b722740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:32:07.252 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:07.249+0000 7fcc047e4740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56549b71a1f0 0x56549b72e990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:32:07.252 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:07.249+0000 7fcc047e4740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56549b722fc0 0x56549b73acf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:32:07.252 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:07.249+0000 7fcc047e4740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fcbfc1b7b20 con 0x56549b722fc0 2024-03-31T23:32:07.252 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:07.249+0000 7fcc047e4740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fcbfc1b53c0 con 0x56549b71a1f0 2024-03-31T23:32:07.252 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:07.249+0000 7fcc047e4740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fcbfc1b3230 con 0x56549b7198d0 2024-03-31T23:32:07.252 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:07.249+0000 7fcbfbfff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56549b722fc0 0x56549b73acf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:32:07.252 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:07.249+0000 7fcbfb7fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56549b7198d0 0x56549b722740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:32:07.252 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:07.249+0000 7fcbfb7fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56549b7198d0 0x56549b722740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.46:51634/0 (socket says 172.21.15.46:51634) 2024-03-31T23:32:07.252 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:07.249+0000 7fcbfb7fe640 1 -- 172.21.15.46:0/199210984 learned_addr learned my addr 172.21.15.46:0/199210984 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:32:07.252 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:07.249+0000 7fcbfbfff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56549b722fc0 0x56549b73acf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:48292/0 (socket says 172.21.15.46:48292) 2024-03-31T23:32:07.252 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:07.249+0000 7fcbfaffd640 1 --2- 172.21.15.46:0/199210984 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56549b71a1f0 0x56549b72e990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:32:07.253 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:07.250+0000 7fcbfb7fe640 1 -- 172.21.15.46:0/199210984 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56549b71a1f0 msgr2=0x56549b72e990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:32:07.253 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:07.250+0000 7fcbfb7fe640 1 --2- 172.21.15.46:0/199210984 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56549b71a1f0 0x56549b72e990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:32:07.253 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:07.250+0000 7fcbfb7fe640 1 -- 172.21.15.46:0/199210984 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56549b722fc0 msgr2=0x56549b73acf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:32:07.253 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:07.250+0000 7fcbfb7fe640 1 --2- 172.21.15.46:0/199210984 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56549b722fc0 0x56549b73acf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:32:07.253 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:07.250+0000 7fcbfb7fe640 1 -- 172.21.15.46:0/199210984 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56549b71e560 con 0x56549b7198d0 2024-03-31T23:32:07.253 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:07.250+0000 7fcbfb7fe640 1 --2- 172.21.15.46:0/199210984 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56549b7198d0 0x56549b722740 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fcbec0042d0 tx=0x7fcbec009680 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e77f1a37f52bfbc8 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:32:07.253 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:07.250+0000 7fcbe17fa640 1 -- 172.21.15.46:0/199210984 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbec00ec20 con 0x56549b7198d0 2024-03-31T23:32:07.253 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:07.250+0000 7fcbe17fa640 1 -- 172.21.15.46:0/199210984 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fcbec005830 con 0x56549b7198d0 2024-03-31T23:32:07.253 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:07.250+0000 7fcbe17fa640 1 -- 172.21.15.46:0/199210984 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbec02a6f0 con 0x56549b7198d0 2024-03-31T23:32:07.257 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:07.251+0000 7fcc047e4740 1 -- 172.21.15.46:0/199210984 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56549b7198d0 msgr2=0x56549b722740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:32:07.257 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:07.251+0000 7fcc047e4740 1 --2- 172.21.15.46:0/199210984 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56549b7198d0 0x56549b722740 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fcbec0042d0 tx=0x7fcbec009680 comp rx=0 tx=0).stop 2024-03-31T23:32:07.257 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:07.251+0000 7fcc047e4740 1 -- 172.21.15.46:0/199210984 shutdown_connections 2024-03-31T23:32:07.257 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:07.251+0000 7fcc047e4740 1 --2- 172.21.15.46:0/199210984 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56549b722fc0 0x56549b73acf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:32:07.257 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:07.251+0000 7fcc047e4740 1 --2- 172.21.15.46:0/199210984 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56549b71a1f0 0x56549b72e990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:32:07.257 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:07.251+0000 7fcc047e4740 1 --2- 172.21.15.46:0/199210984 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56549b7198d0 0x56549b722740 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:32:07.257 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:07.251+0000 7fcc047e4740 1 -- 172.21.15.46:0/199210984 >> 172.21.15.46:0/199210984 conn(0x56549b7141f0 msgr2=0x56549b71a800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:32:07.257 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:07.251+0000 7fcc047e4740 1 -- 172.21.15.46:0/199210984 shutdown_connections 2024-03-31T23:32:07.257 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:07.251+0000 7fcc047e4740 1 -- 172.21.15.46:0/199210984 wait complete. 2024-03-31T23:32:07.257 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:07.252+0000 7fcc047e4740 1 -- start start 2024-03-31T23:32:07.258 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:07.252+0000 7fcc047e4740 1 Processor -- start 2024-03-31T23:32:07.258 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:07.253+0000 7fcc047e4740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56549b7198d0 0x56549b72b3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:32:07.258 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:07.253+0000 7fcc047e4740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56549b71a1f0 0x56549b71c9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:32:07.258 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:07.253+0000 7fcc047e4740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56549b722fc0 0x56549b71d0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:32:07.258 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:07.253+0000 7fcc047e4740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fcbfc1b3880 con 0x56549b7198d0 2024-03-31T23:32:07.258 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:07.253+0000 7fcc047e4740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fcbfc1b4cf0 con 0x56549b71a1f0 2024-03-31T23:32:07.259 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:07.253+0000 7fcc047e4740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fcbfc1b3f50 con 0x56549b722fc0 2024-03-31T23:32:07.259 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:07.253+0000 7fcc047e4740 10 client.0 _ll_register_callbacks cb 0x7fcbf0009880 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T23:32:07.259 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:07.253+0000 7fcbfb7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56549b7198d0 0x56549b72b3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:32:07.259 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:07.253+0000 7fcbfaffd640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56549b71a1f0 0x56549b71c9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:32:07.259 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:07.253+0000 7fcbfbfff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56549b722fc0 0x56549b71d0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:32:07.259 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:07.253+0000 7fcbfb7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56549b7198d0 0x56549b72b3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:48302/0 (socket says 172.21.15.46:48302) 2024-03-31T23:32:07.259 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:07.253+0000 7fcbfb7fe640 1 -- 172.21.15.46:0/2564216579 learned_addr learned my addr 172.21.15.46:0/2564216579 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:32:07.259 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:07.253+0000 7fcbfaffd640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56549b71a1f0 0x56549b71c9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.46:41742/0 (socket says 172.21.15.46:41742) 2024-03-31T23:32:07.259 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:07.253+0000 7fcbfb7fe640 1 -- 172.21.15.46:0/2564216579 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56549b722fc0 msgr2=0x56549b71d0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:32:07.259 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:07.253+0000 7fcbfb7fe640 1 --2- 172.21.15.46:0/2564216579 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56549b722fc0 0x56549b71d0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:32:07.259 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:07.253+0000 7fcbfb7fe640 1 -- 172.21.15.46:0/2564216579 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56549b71a1f0 msgr2=0x56549b71c9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:32:07.259 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:07.253+0000 7fcbfb7fe640 1 --2- 172.21.15.46:0/2564216579 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56549b71a1f0 0x56549b71c9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:32:07.260 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:07.253+0000 7fcbfb7fe640 1 -- 172.21.15.46:0/2564216579 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56549b798320 con 0x56549b7198d0 2024-03-31T23:32:07.260 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:07.254+0000 7fcbfb7fe640 1 --2- 172.21.15.46:0/2564216579 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56549b7198d0 0x56549b72b3f0 secure :-1 s=READY pgs=979 cs=0 l=1 rev1=1 crypto rx=0x7fcbec010660 tx=0x7fcbec00eb20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=21c6126c1eaf3d29 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:32:07.260 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:07.254+0000 7fcbdaffd640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-31T23:32:07.260 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:07.254+0000 7fcbdaffd640 1 -- 172.21.15.46:0/2564216579 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbec039060 con 0x56549b7198d0 2024-03-31T23:32:07.260 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:07.254+0000 7fcc047e4740 1 -- 172.21.15.46:0/2564216579 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcbec005830 con 0x56549b7198d0 2024-03-31T23:32:07.260 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:07.254+0000 7fcbdaffd640 1 -- 172.21.15.46:0/2564216579 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(127..127 src has 1..127) ==== 9642+0+0 (secure 0 0 0) 0x7fcbec005380 con 0x56549b7198d0 2024-03-31T23:32:07.260 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:07.254+0000 7fcbdaffd640 1 -- 172.21.15.46:0/2564216579 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=48) ==== 24+0+0 (secure 0 0 0) 0x7fcbec040050 con 0x56549b7198d0 2024-03-31T23:32:07.260 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:07.254+0000 7fcc047e4740 10 client.17799 fetch_fsmap learned FSMap version 48 2024-03-31T23:32:07.260 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:07.254+0000 7fcc047e4740 1 -- 172.21.15.46:0/2564216579 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=48,monmap=4+}) -- 0x56549b799410 con 0x56549b7198d0 2024-03-31T23:32:07.260 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:07.255+0000 7fcbdaffd640 1 -- 172.21.15.46:0/2564216579 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fcbec005830 con 0x56549b7198d0 2024-03-31T23:32:07.261 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:07.255+0000 7fcbdaffd640 1 -- 172.21.15.46:0/2564216579 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 48) ==== 8350+0+0 (secure 0 0 0) 0x7fcbec00f040 con 0x56549b7198d0 2024-03-31T23:32:07.261 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:07.255+0000 7fcc047e4740 10 client.17799 fetch_fsmap finished waiting for FSMap version 48 2024-03-31T23:32:07.261 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:07.255+0000 7fcc047e4740 10 client.17799 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-31T23:32:07.261 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:07.255+0000 7fcc047e4740 20 client.17799 populate_metadata read hostname 'smithi046' 2024-03-31T23:32:07.261 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:07.255+0000 7fcc047e4740 1 --2- 172.21.15.46:0/2564216579 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x56549b799930 0x56549b79bd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:32:07.261 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:07.255+0000 7fcc047e4740 4 client.17799 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-31T23:32:07.261 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:07.255+0000 7fcc047e4740 1 -- 172.21.15.46:0/2564216579 --> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] -- command(tid 0: {"prefix": "flush journal"}) -- 0x56549b689e30 con 0x56549b799930 2024-03-31T23:32:07.261 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:07.255+0000 7fcbfaffd640 1 --2- 172.21.15.46:0/2564216579 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x56549b799930 0x56549b79bd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:32:07.261 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:07.255+0000 7fcbdaffd640 10 client.17799 ms_handle_connect on v2:172.21.15.142:6838/2519143918 2024-03-31T23:32:07.261 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:07.256+0000 7fcbfaffd640 1 --2- 172.21.15.46:0/2564216579 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x56549b799930 0x56549b79bd20 crc :-1 s=READY pgs=259 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a35990808cd83b64 server_cookie=8ad8376946dd1950 in_seq=0 out_seq=0 2024-03-31T23:32:07.295 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:32:06 smithi162 ceph-mon[36237]: pgmap v645: 129 pgs: 129 active+clean; 1006 MiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 5.9 MiB/s wr, 349 op/s 2024-03-31T23:32:07.295 INFO:tasks.workunit.client.0.smithi046.stdout: 10 68409 8.41 MB/sec execute 312 sec latency 83.089 ms 2024-03-31T23:32:07.485 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:07.483+0000 7fcbdaffd640 1 -- 172.21.15.46:0/2564216579 <== mds.0 v2:172.21.15.142:6838/2519143918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x56549b689e30 con 0x56549b799930 2024-03-31T23:32:07.485 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:07.483+0000 7fcbdaffd640 10 client.17799 handle_command_reply: tid=0 2024-03-31T23:32:07.485 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:07.483+0000 7fcc047e4740 1 client.17799 shutdown 2024-03-31T23:32:07.485 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:07.483+0000 7fcc047e4740 1 -- 172.21.15.46:0/2564216579 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56549b7198d0 msgr2=0x56549b72b3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:32:07.485 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:07.483+0000 7fcc047e4740 1 --2- 172.21.15.46:0/2564216579 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56549b7198d0 0x56549b72b3f0 secure :-1 s=READY pgs=979 cs=0 l=1 rev1=1 crypto rx=0x7fcbec010660 tx=0x7fcbec00eb20 comp rx=0 tx=0).stop 2024-03-31T23:32:07.485 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:07.483+0000 7fcc047e4740 1 -- 172.21.15.46:0/2564216579 shutdown_connections 2024-03-31T23:32:07.485 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:07.483+0000 7fcc047e4740 1 --2- 172.21.15.46:0/2564216579 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x56549b799930 0x56549b79bd20 crc :-1 s=READY pgs=259 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:32:07.485 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:07.483+0000 7fcc047e4740 1 --2- 172.21.15.46:0/2564216579 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56549b722fc0 0x56549b71d0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:32:07.485 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:07.483+0000 7fcc047e4740 1 --2- 172.21.15.46:0/2564216579 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56549b71a1f0 0x56549b71c9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:32:07.486 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:07.483+0000 7fcc047e4740 1 --2- 172.21.15.46:0/2564216579 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56549b7198d0 0x56549b72b3f0 unknown :-1 s=CLOSED pgs=979 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:32:07.486 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:07.483+0000 7fcc047e4740 1 -- 172.21.15.46:0/2564216579 >> 172.21.15.46:0/2564216579 conn(0x56549b7141f0 msgr2=0x56549b71ffd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:32:07.486 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:07.483+0000 7fcbdaffd640 0 client.17799 ms_handle_reset on v2:172.21.15.142:6838/2519143918 2024-03-31T23:32:07.486 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:07.483+0000 7fcc047e4740 1 -- 172.21.15.46:0/2564216579 shutdown_connections 2024-03-31T23:32:07.486 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:07.483+0000 7fcc047e4740 1 -- 172.21.15.46:0/2564216579 wait complete. 2024-03-31T23:32:07.486 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:07.484+0000 7fcc047e4740 20 client.17799 trim_cache size 0 max 16384 2024-03-31T23:32:07.486 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout:{ 2024-03-31T23:32:07.486 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout: "message": "", 2024-03-31T23:32:07.486 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout: "return_code": 0 2024-03-31T23:32:07.486 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout:} 2024-03-31T23:32:07.487 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:07.484+0000 7fcc009a0640 1 -- 172.21.15.46:0/3438549834 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fcbe40a2600 msgr2=0x7fcbe40a4ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:32:07.487 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:07.484+0000 7fcc009a0640 1 --2- 172.21.15.46:0/3438549834 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fcbe40a2600 0x7fcbe40a4ac0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fcbf4013ac0 tx=0x7fcbf4004730 comp rx=0 tx=0).stop 2024-03-31T23:32:07.487 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:07.484+0000 7fcc009a0640 1 -- 172.21.15.46:0/3438549834 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcbfc100e10 msgr2=0x7fcbfc19e940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:32:07.488 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:07.484+0000 7fcc009a0640 1 --2- 172.21.15.46:0/3438549834 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcbfc100e10 0x7fcbfc19e940 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fcbf0009230 tx=0x7fcbf00045e0 comp rx=0 tx=0).stop 2024-03-31T23:32:07.488 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:07.484+0000 7fcc009a0640 1 -- 172.21.15.46:0/3438549834 shutdown_connections 2024-03-31T23:32:07.488 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:07.484+0000 7fcc009a0640 1 --2- 172.21.15.46:0/3438549834 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fcbe40a2600 0x7fcbe40a4ac0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:32:07.488 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:07.484+0000 7fcc009a0640 1 --2- 172.21.15.46:0/3438549834 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcbfc105a00 0x7fcbfc1a2cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:32:07.488 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:07.484+0000 7fcc009a0640 1 --2- 172.21.15.46:0/3438549834 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcbfc100e10 0x7fcbfc19e940 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:32:07.488 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:07.484+0000 7fcc009a0640 1 --2- 172.21.15.46:0/3438549834 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcbfc1004f0 0x7fcbfc19e400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:32:07.488 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:07.484+0000 7fcc009a0640 1 -- 172.21.15.46:0/3438549834 >> 172.21.15.46:0/3438549834 conn(0x7fcbfc0fbf60 msgr2=0x7fcbfc0fd050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:32:07.488 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:07.485+0000 7fcc009a0640 1 -- 172.21.15.46:0/3438549834 shutdown_connections 2024-03-31T23:32:07.488 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:07.485+0000 7fcc009a0640 1 -- 172.21.15.46:0/3438549834 wait complete. 2024-03-31T23:32:08.295 INFO:tasks.workunit.client.0.smithi046.stdout: 10 68533 8.40 MB/sec execute 313 sec latency 194.277 ms 2024-03-31T23:32:09.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:32:08 smithi046 ceph-mon[32498]: pgmap v646: 129 pgs: 129 active+clean; 1006 MiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 5.9 MiB/s wr, 349 op/s 2024-03-31T23:32:09.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:32:08 smithi142 ceph-mon[35271]: pgmap v646: 129 pgs: 129 active+clean; 1006 MiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 5.9 MiB/s wr, 349 op/s 2024-03-31T23:32:09.295 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:32:08 smithi162 ceph-mon[36237]: pgmap v646: 129 pgs: 129 active+clean; 1006 MiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 5.9 MiB/s wr, 349 op/s 2024-03-31T23:32:09.296 INFO:tasks.workunit.client.0.smithi046.stdout: 10 68677 8.39 MB/sec execute 314 sec latency 132.334 ms 2024-03-31T23:32:10.295 INFO:tasks.workunit.client.0.smithi046.stdout: 10 68847 8.40 MB/sec execute 315 sec latency 81.723 ms 2024-03-31T23:32:11.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:32:10 smithi046 ceph-mon[32498]: pgmap v647: 129 pgs: 129 active+clean; 1006 MiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 5.1 MiB/s wr, 347 op/s 2024-03-31T23:32:11.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:32:10 smithi046 ceph-mon[32498]: osdmap e128: 12 total, 12 up, 12 in 2024-03-31T23:32:11.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:32:10 smithi142 ceph-mon[35271]: pgmap v647: 129 pgs: 129 active+clean; 1006 MiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 5.1 MiB/s wr, 347 op/s 2024-03-31T23:32:11.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:32:10 smithi142 ceph-mon[35271]: osdmap e128: 12 total, 12 up, 12 in 2024-03-31T23:32:11.295 INFO:tasks.workunit.client.0.smithi046.stdout: 10 69028 8.41 MB/sec execute 316 sec latency 80.211 ms 2024-03-31T23:32:11.296 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:32:10 smithi162 ceph-mon[36237]: pgmap v647: 129 pgs: 129 active+clean; 1006 MiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 5.1 MiB/s wr, 347 op/s 2024-03-31T23:32:11.296 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:32:10 smithi162 ceph-mon[36237]: osdmap e128: 12 total, 12 up, 12 in 2024-03-31T23:32:12.295 INFO:tasks.workunit.client.0.smithi046.stdout: 10 69204 8.42 MB/sec execute 317 sec latency 75.500 ms 2024-03-31T23:32:13.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:32:12 smithi046 ceph-mon[32498]: pgmap v649: 129 pgs: 129 active+clean; 1006 MiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 6.2 MiB/s wr, 423 op/s 2024-03-31T23:32:13.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:32:12 smithi142 ceph-mon[35271]: pgmap v649: 129 pgs: 129 active+clean; 1006 MiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 6.2 MiB/s wr, 423 op/s 2024-03-31T23:32:13.295 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:32:12 smithi162 ceph-mon[36237]: pgmap v649: 129 pgs: 129 active+clean; 1006 MiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 6.2 MiB/s wr, 423 op/s 2024-03-31T23:32:13.296 INFO:tasks.workunit.client.0.smithi046.stdout: 10 69365 8.43 MB/sec execute 318 sec latency 75.227 ms 2024-03-31T23:32:14.295 INFO:tasks.workunit.client.0.smithi046.stdout: 10 69538 8.43 MB/sec execute 319 sec latency 103.293 ms 2024-03-31T23:32:15.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:32:14 smithi046 ceph-mon[32498]: pgmap v650: 129 pgs: 129 active+clean; 1006 MiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 5.1 MiB/s wr, 347 op/s 2024-03-31T23:32:15.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:32:14 smithi142 ceph-mon[35271]: pgmap v650: 129 pgs: 129 active+clean; 1006 MiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 5.1 MiB/s wr, 347 op/s 2024-03-31T23:32:15.295 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:32:14 smithi162 ceph-mon[36237]: pgmap v650: 129 pgs: 129 active+clean; 1006 MiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 5.1 MiB/s wr, 347 op/s 2024-03-31T23:32:15.296 INFO:tasks.workunit.client.0.smithi046.stdout: 10 69680 8.42 MB/sec execute 320 sec latency 92.801 ms 2024-03-31T23:32:16.296 INFO:tasks.workunit.client.0.smithi046.stdout: 10 69835 8.42 MB/sec execute 321 sec latency 140.996 ms 2024-03-31T23:32:17.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:32:16 smithi046 ceph-mon[32498]: pgmap v651: 129 pgs: 129 active+clean; 1.0 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 5.5 MiB/s wr, 326 op/s 2024-03-31T23:32:17.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:32:16 smithi046 ceph-mon[32498]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-31T23:32:17.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:32:16 smithi142 ceph-mon[35271]: pgmap v651: 129 pgs: 129 active+clean; 1.0 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 5.5 MiB/s wr, 326 op/s 2024-03-31T23:32:17.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:32:16 smithi142 ceph-mon[35271]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-31T23:32:17.295 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:32:16 smithi162 ceph-mon[36237]: pgmap v651: 129 pgs: 129 active+clean; 1.0 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 5.5 MiB/s wr, 326 op/s 2024-03-31T23:32:17.295 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:32:16 smithi162 ceph-mon[36237]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-31T23:32:17.296 INFO:tasks.workunit.client.0.smithi046.stdout: 10 69990 8.42 MB/sec execute 322 sec latency 66.625 ms 2024-03-31T23:32:18.296 INFO:tasks.workunit.client.0.smithi046.stdout: 10 70130 8.41 MB/sec execute 323 sec latency 147.252 ms 2024-03-31T23:32:18.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:32:18 smithi142 ceph-mon[35271]: pgmap v652: 129 pgs: 129 active+clean; 1.0 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 5.5 MiB/s wr, 326 op/s 2024-03-31T23:32:19.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:32:18 smithi162 ceph-mon[36237]: pgmap v652: 129 pgs: 129 active+clean; 1.0 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 5.5 MiB/s wr, 326 op/s 2024-03-31T23:32:19.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:32:18 smithi046 ceph-mon[32498]: pgmap v652: 129 pgs: 129 active+clean; 1.0 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 5.5 MiB/s wr, 326 op/s 2024-03-31T23:32:19.296 INFO:tasks.workunit.client.0.smithi046.stdout: 10 70249 8.41 MB/sec execute 324 sec latency 148.724 ms 2024-03-31T23:32:20.296 INFO:tasks.workunit.client.0.smithi046.stdout: 10 70363 8.40 MB/sec execute 325 sec latency 128.743 ms 2024-03-31T23:32:20.593 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:20.591+0000 7f43134cd640 1 -- 172.21.15.46:0/3094151787 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f430c105650 msgr2=0x7f430c105a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:32:20.593 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:20.591+0000 7f43134cd640 1 --2- 172.21.15.46:0/3094151787 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f430c105650 0x7f430c105a30 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f4308009900 tx=0x7f430802bc30 comp rx=0 tx=0).stop 2024-03-31T23:32:20.593 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:20.591+0000 7f43134cd640 1 -- 172.21.15.46:0/3094151787 shutdown_connections 2024-03-31T23:32:20.593 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:20.591+0000 7f43134cd640 1 --2- 172.21.15.46:0/3094151787 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f430c101b80 0x7f430c10d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:32:20.593 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:20.591+0000 7f43134cd640 1 --2- 172.21.15.46:0/3094151787 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f430c1011c0 0x7f430c101640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:32:20.593 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:20.591+0000 7f43134cd640 1 --2- 172.21.15.46:0/3094151787 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f430c105650 0x7f430c105a30 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:32:20.594 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:20.591+0000 7f43134cd640 1 -- 172.21.15.46:0/3094151787 >> 172.21.15.46:0/3094151787 conn(0x7f430c0767e0 msgr2=0x7f430c0fdb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:32:20.594 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:20.592+0000 7f43134cd640 1 -- 172.21.15.46:0/3094151787 shutdown_connections 2024-03-31T23:32:20.594 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:20.592+0000 7f43134cd640 1 -- 172.21.15.46:0/3094151787 wait complete. 2024-03-31T23:32:20.596 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:20.592+0000 7f43134cd640 1 Processor -- start 2024-03-31T23:32:20.596 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:20.592+0000 7f43134cd640 1 -- start start 2024-03-31T23:32:20.596 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:20.593+0000 7f43134cd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f430c1011c0 0x7f430c19b8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:32:20.596 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:20.593+0000 7f43134cd640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f430c101b80 0x7f430c19be00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:32:20.596 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:20.593+0000 7f43134cd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f430c105650 0x7f430c195a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:32:20.596 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:20.593+0000 7f43134cd640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f430c10f830 con 0x7f430c105650 2024-03-31T23:32:20.596 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:20.593+0000 7f43134cd640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f430c10f6b0 con 0x7f430c101b80 2024-03-31T23:32:20.596 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:20.593+0000 7f43134cd640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f430c10f9b0 con 0x7f430c1011c0 2024-03-31T23:32:20.596 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:20.593+0000 7f4311269640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f430c1011c0 0x7f430c19b8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:32:20.596 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:20.593+0000 7f4311a6a640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f430c105650 0x7f430c195a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:32:20.596 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:20.593+0000 7f4311269640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f430c1011c0 0x7f430c19b8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.46:34332/0 (socket says 172.21.15.46:34332) 2024-03-31T23:32:20.597 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:20.593+0000 7f4311269640 1 -- 172.21.15.46:0/4062245025 learned_addr learned my addr 172.21.15.46:0/4062245025 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:32:20.597 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:20.593+0000 7f4311a6a640 1 --2- 172.21.15.46:0/4062245025 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f430c105650 0x7f430c195a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:43808/0 (socket says 172.21.15.46:43808) 2024-03-31T23:32:20.597 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:20.593+0000 7f4310a68640 1 --2- 172.21.15.46:0/4062245025 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f430c101b80 0x7f430c19be00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:32:20.597 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:20.593+0000 7f4311269640 1 -- 172.21.15.46:0/4062245025 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f430c101b80 msgr2=0x7f430c19be00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:32:20.597 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:20.593+0000 7f4311269640 1 --2- 172.21.15.46:0/4062245025 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f430c101b80 0x7f430c19be00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:32:20.597 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:20.593+0000 7f4311269640 1 -- 172.21.15.46:0/4062245025 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f430c105650 msgr2=0x7f430c195a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:32:20.597 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:20.593+0000 7f4311269640 1 --2- 172.21.15.46:0/4062245025 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f430c105650 0x7f430c195a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:32:20.597 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:20.593+0000 7f4311269640 1 -- 172.21.15.46:0/4062245025 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f430c1961f0 con 0x7f430c1011c0 2024-03-31T23:32:20.597 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:20.593+0000 7f4311a6a640 1 --2- 172.21.15.46:0/4062245025 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f430c105650 0x7f430c195a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:32:20.597 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:20.594+0000 7f4311269640 1 --2- 172.21.15.46:0/4062245025 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f430c1011c0 0x7f430c19b8c0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f4308006010 tx=0x7f4308005d90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:32:20.597 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:20.594+0000 7f43027fc640 1 -- 172.21.15.46:0/4062245025 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4308043020 con 0x7f430c1011c0 2024-03-31T23:32:20.597 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:20.594+0000 7f43027fc640 1 -- 172.21.15.46:0/4062245025 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4308004350 con 0x7f430c1011c0 2024-03-31T23:32:20.597 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:20.594+0000 7f43027fc640 1 -- 172.21.15.46:0/4062245025 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4308003970 con 0x7f430c1011c0 2024-03-31T23:32:20.598 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:20.594+0000 7f43134cd640 1 -- 172.21.15.46:0/4062245025 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f430c196480 con 0x7f430c1011c0 2024-03-31T23:32:20.598 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:20.594+0000 7f43134cd640 1 -- 172.21.15.46:0/4062245025 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f430c1a2780 con 0x7f430c1011c0 2024-03-31T23:32:20.600 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:20.597+0000 7f4316193740 1 Processor -- start 2024-03-31T23:32:20.600 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:20.597+0000 7f43027fc640 1 -- 172.21.15.46:0/4062245025 <== mon.1 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 25) ==== 137451+0+0 (secure 0 0 0) 0x7f4308046020 con 0x7f430c1011c0 2024-03-31T23:32:20.601 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:20.598+0000 7f4316193740 1 -- start start 2024-03-31T23:32:20.601 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:20.598+0000 7f4316193740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56549806a1a0 0x56549806a580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:32:20.601 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:20.598+0000 7f4316193740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56549806aac0 0x565498077c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:32:20.601 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:20.598+0000 7f4316193740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5654980781c0 0x56549807a5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:32:20.601 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:20.598+0000 7f4316193740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x565497fcd8f0 con 0x56549806a1a0 2024-03-31T23:32:20.601 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:20.598+0000 7f4316193740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x565497f768d0 con 0x5654980781c0 2024-03-31T23:32:20.601 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:20.598+0000 7f4316193740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x565497fc9990 con 0x56549806aac0 2024-03-31T23:32:20.601 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:20.598+0000 7f4311a6a640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56549806a1a0 0x56549806a580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:32:20.601 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:20.598+0000 7f4311a6a640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56549806a1a0 0x56549806a580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:43814/0 (socket says 172.21.15.46:43814) 2024-03-31T23:32:20.601 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:20.598+0000 7f4311a6a640 1 -- 172.21.15.46:0/1454721215 learned_addr learned my addr 172.21.15.46:0/1454721215 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:32:20.601 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:20.598+0000 7f4311269640 1 --2- 172.21.15.46:0/1454721215 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56549806aac0 0x565498077c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:32:20.601 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:20.598+0000 7f4310a68640 1 --2- 172.21.15.46:0/1454721215 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5654980781c0 0x56549807a5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:32:20.601 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:20.598+0000 7f4311a6a640 1 -- 172.21.15.46:0/1454721215 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56549806aac0 msgr2=0x565498077c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:32:20.601 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:20.598+0000 7f4311a6a640 1 --2- 172.21.15.46:0/1454721215 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56549806aac0 0x565498077c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:32:20.602 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:20.598+0000 7f4311a6a640 1 -- 172.21.15.46:0/1454721215 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5654980781c0 msgr2=0x56549807a5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:32:20.602 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:20.598+0000 7f4311a6a640 1 --2- 172.21.15.46:0/1454721215 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5654980781c0 0x56549807a5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:32:20.602 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:20.598+0000 7f4311a6a640 1 -- 172.21.15.46:0/1454721215 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x565497ce2210 con 0x56549806a1a0 2024-03-31T23:32:20.602 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:20.599+0000 7f4310a68640 1 --2- 172.21.15.46:0/1454721215 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5654980781c0 0x56549807a5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T23:32:20.602 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:20.599+0000 7f43027fc640 1 --2- 172.21.15.46:0/4062245025 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f42f80a27b0 0x7f42f80a4c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:32:20.602 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:20.599+0000 7f4311a6a640 1 --2- 172.21.15.46:0/1454721215 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56549806a1a0 0x56549806a580 secure :-1 s=READY pgs=980 cs=0 l=1 rev1=1 crypto rx=0x7f430400c620 tx=0x7f4304009500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c94db5b57475c5f1 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:32:20.602 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:20.599+0000 7f43027fc640 1 -- 172.21.15.46:0/4062245025 <== mon.1 v2:172.21.15.162:3300/0 5 ==== osd_map(128..128 src has 1..128) ==== 9554+0+0 (secure 0 0 0) 0x7f43080f0800 con 0x7f430c1011c0 2024-03-31T23:32:20.602 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:20.599+0000 7f4311269640 1 --2- 172.21.15.46:0/4062245025 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f42f80a27b0 0x7f42f80a4c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:32:20.602 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:20.600+0000 7f4311269640 1 --2- 172.21.15.46:0/4062245025 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f42f80a27b0 0x7f42f80a4c70 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x56549806b7c0 tx=0x7f4308005e80 comp rx=0 tx=0).ready entity=mgr.15654 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:32:20.638 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:20.637+0000 7f42f77fe640 1 -- 172.21.15.46:0/1454721215 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f430401b030 con 0x56549806a1a0 2024-03-31T23:32:20.638 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:20.637+0000 7f42f77fe640 1 -- 172.21.15.46:0/1454721215 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4304003970 con 0x56549806a1a0 2024-03-31T23:32:20.639 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:20.637+0000 7f42f77fe640 1 -- 172.21.15.46:0/1454721215 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4304004630 con 0x56549806a1a0 2024-03-31T23:32:20.639 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:20.638+0000 7f4316193740 1 -- 172.21.15.46:0/1454721215 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56549806a1a0 msgr2=0x56549806a580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:32:20.639 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:20.638+0000 7f4316193740 1 --2- 172.21.15.46:0/1454721215 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56549806a1a0 0x56549806a580 secure :-1 s=READY pgs=980 cs=0 l=1 rev1=1 crypto rx=0x7f430400c620 tx=0x7f4304009500 comp rx=0 tx=0).stop 2024-03-31T23:32:20.639 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:20.638+0000 7f4316193740 1 -- 172.21.15.46:0/1454721215 shutdown_connections 2024-03-31T23:32:20.640 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:20.638+0000 7f4316193740 1 --2- 172.21.15.46:0/1454721215 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5654980781c0 0x56549807a5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:32:20.640 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:20.638+0000 7f4316193740 1 --2- 172.21.15.46:0/1454721215 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56549806aac0 0x565498077c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:32:20.640 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:20.638+0000 7f4316193740 1 --2- 172.21.15.46:0/1454721215 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56549806a1a0 0x56549806a580 unknown :-1 s=CLOSED pgs=980 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:32:20.640 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:20.638+0000 7f4316193740 1 -- 172.21.15.46:0/1454721215 >> 172.21.15.46:0/1454721215 conn(0x565498064fe0 msgr2=0x5654980653c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:32:20.640 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:20.639+0000 7f4316193740 1 -- 172.21.15.46:0/1454721215 shutdown_connections 2024-03-31T23:32:20.641 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:20.639+0000 7f4316193740 1 -- 172.21.15.46:0/1454721215 wait complete. 2024-03-31T23:32:20.641 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:20.639+0000 7f4316193740 1 -- start start 2024-03-31T23:32:20.641 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:20.640+0000 7f4316193740 1 Processor -- start 2024-03-31T23:32:20.643 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:20.640+0000 7f4316193740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56549806a1a0 0x5654980751a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:32:20.643 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:20.640+0000 7f4311a6a640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56549806a1a0 0x5654980751a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:32:20.643 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:20.640+0000 7f4311a6a640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56549806a1a0 0x5654980751a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:43824/0 (socket says 172.21.15.46:43824) 2024-03-31T23:32:20.643 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:20.640+0000 7f4311a6a640 1 -- 172.21.15.46:0/112456554 learned_addr learned my addr 172.21.15.46:0/112456554 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:32:20.643 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:20.640+0000 7f4316193740 1 --2- 172.21.15.46:0/112456554 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56549806aac0 0x5654980756e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:32:20.643 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:20.640+0000 7f4316193740 1 --2- 172.21.15.46:0/112456554 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5654980781c0 0x565498075c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:32:20.643 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:20.640+0000 7f4316193740 1 -- 172.21.15.46:0/112456554 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x56549807ad90 con 0x56549806a1a0 2024-03-31T23:32:20.643 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:20.640+0000 7f4316193740 1 -- 172.21.15.46:0/112456554 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x565497f6bf80 con 0x56549806aac0 2024-03-31T23:32:20.643 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:20.640+0000 7f4316193740 1 -- 172.21.15.46:0/112456554 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x56549807af10 con 0x5654980781c0 2024-03-31T23:32:20.643 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:20.641+0000 7f4310a68640 1 --2- 172.21.15.46:0/112456554 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56549806aac0 0x5654980756e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:32:20.643 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:20.641+0000 7f4311a6a640 1 -- 172.21.15.46:0/112456554 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5654980781c0 msgr2=0x565498075c20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-31T23:32:20.643 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:20.641+0000 7f4311a6a640 1 --2- 172.21.15.46:0/112456554 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5654980781c0 0x565498075c20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:32:20.643 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:20.641+0000 7f4311a6a640 1 -- 172.21.15.46:0/112456554 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56549806aac0 msgr2=0x5654980756e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:32:20.644 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:20.641+0000 7f4311a6a640 1 --2- 172.21.15.46:0/112456554 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56549806aac0 0x5654980756e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:32:20.644 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:20.641+0000 7f4311a6a640 1 -- 172.21.15.46:0/112456554 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x565497dae620 con 0x56549806a1a0 2024-03-31T23:32:20.644 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:20.641+0000 7f4316193740 10 client.0 _ll_register_callbacks cb 0x565497fc4290 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T23:32:20.644 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:20.641+0000 7f4316193740 1 -- 172.21.15.46:0/112456554 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4304003970 con 0x56549806a1a0 2024-03-31T23:32:20.644 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:20.641+0000 7f4311a6a640 1 --2- 172.21.15.46:0/112456554 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56549806a1a0 0x5654980751a0 secure :-1 s=READY pgs=981 cs=0 l=1 rev1=1 crypto rx=0x7f4304005f40 tx=0x7f4304003f50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e9922c3b510c6f server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:32:20.644 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:20.641+0000 7f42f4ff9640 10 client.17811 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-31T23:32:20.644 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:20.642+0000 7f42f4ff9640 1 -- 172.21.15.46:0/112456554 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f430402f050 con 0x56549806a1a0 2024-03-31T23:32:20.644 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:20.642+0000 7f42f4ff9640 1 -- 172.21.15.46:0/112456554 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(128..128 src has 1..128) ==== 9554+0+0 (secure 0 0 0) 0x7f4304004530 con 0x56549806a1a0 2024-03-31T23:32:20.644 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:20.642+0000 7f42f4ff9640 1 -- 172.21.15.46:0/112456554 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=48) ==== 24+0+0 (secure 0 0 0) 0x7f4304004770 con 0x56549806a1a0 2024-03-31T23:32:20.644 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:20.642+0000 7f4316193740 10 client.17811 fetch_fsmap learned FSMap version 48 2024-03-31T23:32:20.644 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:20.642+0000 7f4316193740 1 -- 172.21.15.46:0/112456554 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({fsmap=48}) -- 0x565498086ad0 con 0x56549806a1a0 2024-03-31T23:32:20.646 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:20.642+0000 7f42f4ff9640 1 -- 172.21.15.46:0/112456554 <== mon.0 v2:172.21.15.46:3300/0 4 ==== fsmap(e 48) ==== 8350+0+0 (secure 0 0 0) 0x7f4304032020 con 0x56549806a1a0 2024-03-31T23:32:20.647 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:20.642+0000 7f4316193740 10 client.17811 fetch_fsmap finished waiting for FSMap version 48 2024-03-31T23:32:20.647 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:20.642+0000 7f4316193740 10 client.17811 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-31T23:32:20.647 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:20.642+0000 7f4316193740 20 client.17811 populate_metadata read hostname 'smithi046' 2024-03-31T23:32:20.647 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:20.642+0000 7f4316193740 1 --2- 172.21.15.46:0/112456554 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x565498087e20 0x56549808a210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:32:20.647 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:20.642+0000 7f4316193740 4 client.17811 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-31T23:32:20.647 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:20.642+0000 7f4316193740 1 -- 172.21.15.46:0/112456554 --> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x565497fe1120 con 0x565498087e20 2024-03-31T23:32:20.647 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:20.642+0000 7f4310a68640 1 --2- 172.21.15.46:0/112456554 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x565498087e20 0x56549808a210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:32:20.647 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:20.643+0000 7f4310a68640 1 --2- 172.21.15.46:0/112456554 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x565498087e20 0x56549808a210 crc :-1 s=READY pgs=260 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=89ebfb7b9fe9c09 server_cookie=adfcc722a4faf721 in_seq=0 out_seq=0 2024-03-31T23:32:20.647 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:20.643+0000 7f42f4ff9640 10 client.17811 ms_handle_connect on v2:172.21.15.142:6838/2519143918 2024-03-31T23:32:20.647 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:20.644+0000 7f42f4ff9640 1 -- 172.21.15.46:0/112456554 <== mds.0 v2:172.21.15.142:6838/2519143918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x565497fe1120 con 0x565498087e20 2024-03-31T23:32:20.647 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:20.644+0000 7f42f4ff9640 10 client.17811 handle_command_reply: tid=0 2024-03-31T23:32:20.647 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:20.644+0000 7f4316193740 1 client.17811 shutdown 2024-03-31T23:32:20.647 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:20.644+0000 7f4316193740 1 -- 172.21.15.46:0/112456554 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56549806a1a0 msgr2=0x5654980751a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:32:20.647 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:20.644+0000 7f4316193740 1 --2- 172.21.15.46:0/112456554 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56549806a1a0 0x5654980751a0 secure :-1 s=READY pgs=981 cs=0 l=1 rev1=1 crypto rx=0x7f4304005f40 tx=0x7f4304003f50 comp rx=0 tx=0).stop 2024-03-31T23:32:20.648 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:20.645+0000 7f4316193740 1 -- 172.21.15.46:0/112456554 shutdown_connections 2024-03-31T23:32:20.648 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:20.645+0000 7f4316193740 1 --2- 172.21.15.46:0/112456554 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x565498087e20 0x56549808a210 crc :-1 s=READY pgs=260 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:32:20.648 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:20.645+0000 7f42f4ff9640 0 client.17811 ms_handle_reset on v2:172.21.15.142:6838/2519143918 2024-03-31T23:32:20.648 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:20.645+0000 7f4316193740 1 --2- 172.21.15.46:0/112456554 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5654980781c0 0x565498075c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:32:20.648 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:20.645+0000 7f4316193740 1 --2- 172.21.15.46:0/112456554 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56549806aac0 0x5654980756e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:32:20.648 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:20.645+0000 7f4316193740 1 --2- 172.21.15.46:0/112456554 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56549806a1a0 0x5654980751a0 unknown :-1 s=CLOSED pgs=981 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:32:20.648 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:20.645+0000 7f4316193740 1 -- 172.21.15.46:0/112456554 >> 172.21.15.46:0/112456554 conn(0x565498064fe0 msgr2=0x565498066d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:32:20.648 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:20.645+0000 7f4316193740 1 -- 172.21.15.46:0/112456554 shutdown_connections 2024-03-31T23:32:20.648 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:20.645+0000 7f4316193740 1 -- 172.21.15.46:0/112456554 wait complete. 2024-03-31T23:32:20.648 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:20.645+0000 7f4316193740 20 client.17811 trim_cache size 0 max 16384 2024-03-31T23:32:20.675 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:20.674+0000 7f4316193740 1 Processor -- start 2024-03-31T23:32:20.675 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:20.674+0000 7f4316193740 1 -- start start 2024-03-31T23:32:20.675 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:20.674+0000 7f4316193740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56549806a1a0 0x5654980778d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:32:20.675 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:20.674+0000 7f4316193740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56549806aac0 0x565498083b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:32:20.675 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:20.674+0000 7f4316193740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5654980781c0 0x56549806f830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:32:20.675 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:20.674+0000 7f4316193740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f430c1af310 con 0x56549806a1a0 2024-03-31T23:32:20.676 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:20.674+0000 7f4316193740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f430c1acbb0 con 0x5654980781c0 2024-03-31T23:32:20.676 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:20.674+0000 7f4316193740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f430c1aaa20 con 0x56549806aac0 2024-03-31T23:32:20.676 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:20.674+0000 7f4311a6a640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56549806a1a0 0x5654980778d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:32:20.677 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:20.674+0000 7f4311a6a640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56549806a1a0 0x5654980778d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:43832/0 (socket says 172.21.15.46:43832) 2024-03-31T23:32:20.677 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:20.674+0000 7f4310a68640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56549806aac0 0x565498083b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:32:20.677 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:20.674+0000 7f4311a6a640 1 -- 172.21.15.46:0/348021065 learned_addr learned my addr 172.21.15.46:0/348021065 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:32:20.677 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:20.674+0000 7f4311a6a640 1 --2- 172.21.15.46:0/348021065 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5654980781c0 0x56549806f830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:32:20.678 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:20.675+0000 7f4311a6a640 1 -- 172.21.15.46:0/348021065 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56549806aac0 msgr2=0x565498083b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:32:20.678 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:20.675+0000 7f4311a6a640 1 --2- 172.21.15.46:0/348021065 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56549806aac0 0x565498083b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:32:20.678 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:20.675+0000 7f4311a6a640 1 -- 172.21.15.46:0/348021065 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5654980781c0 msgr2=0x56549806f830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:32:20.678 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:20.675+0000 7f4311a6a640 1 --2- 172.21.15.46:0/348021065 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5654980781c0 0x56549806f830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:32:20.678 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:20.675+0000 7f4311a6a640 1 -- 172.21.15.46:0/348021065 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56549806a8f0 con 0x56549806a1a0 2024-03-31T23:32:20.678 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:20.675+0000 7f4311a6a640 1 --2- 172.21.15.46:0/348021065 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56549806a1a0 0x5654980778d0 secure :-1 s=READY pgs=982 cs=0 l=1 rev1=1 crypto rx=0x7f4304002230 tx=0x7f4304021000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=810ab925c9d9820 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:32:20.678 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:20.675+0000 7f42f6ffd640 1 -- 172.21.15.46:0/348021065 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f430401b070 con 0x56549806a1a0 2024-03-31T23:32:20.678 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:20.675+0000 7f42f6ffd640 1 -- 172.21.15.46:0/348021065 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f430401c040 con 0x56549806a1a0 2024-03-31T23:32:20.678 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:20.675+0000 7f42f6ffd640 1 -- 172.21.15.46:0/348021065 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f430402f040 con 0x56549806a1a0 2024-03-31T23:32:20.678 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:20.676+0000 7f4316193740 1 -- 172.21.15.46:0/348021065 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56549806a1a0 msgr2=0x5654980778d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:32:20.678 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:20.676+0000 7f4316193740 1 --2- 172.21.15.46:0/348021065 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56549806a1a0 0x5654980778d0 secure :-1 s=READY pgs=982 cs=0 l=1 rev1=1 crypto rx=0x7f4304002230 tx=0x7f4304021000 comp rx=0 tx=0).stop 2024-03-31T23:32:20.678 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:20.676+0000 7f4316193740 1 -- 172.21.15.46:0/348021065 shutdown_connections 2024-03-31T23:32:20.678 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:20.676+0000 7f4316193740 1 --2- 172.21.15.46:0/348021065 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5654980781c0 0x56549806f830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:32:20.679 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:20.676+0000 7f4316193740 1 --2- 172.21.15.46:0/348021065 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56549806aac0 0x565498083b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:32:20.679 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:20.676+0000 7f4316193740 1 --2- 172.21.15.46:0/348021065 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56549806a1a0 0x5654980778d0 unknown :-1 s=CLOSED pgs=982 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:32:20.679 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:20.676+0000 7f4316193740 1 -- 172.21.15.46:0/348021065 >> 172.21.15.46:0/348021065 conn(0x565498064fe0 msgr2=0x5654980653c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:32:20.679 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:20.677+0000 7f4316193740 1 -- 172.21.15.46:0/348021065 shutdown_connections 2024-03-31T23:32:20.679 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:20.677+0000 7f4316193740 1 -- 172.21.15.46:0/348021065 wait complete. 2024-03-31T23:32:20.681 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:20.677+0000 7f4316193740 1 -- start start 2024-03-31T23:32:20.681 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:20.677+0000 7f4316193740 1 Processor -- start 2024-03-31T23:32:20.681 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:20.678+0000 7f4316193740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56549806a1a0 0x565498071e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:32:20.682 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:20.678+0000 7f4316193740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56549806aac0 0x565498072390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:32:20.682 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:20.678+0000 7f4311a6a640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56549806a1a0 0x565498071e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:32:20.682 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:20.678+0000 7f4311a6a640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56549806a1a0 0x565498071e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.46:34366/0 (socket says 172.21.15.46:34366) 2024-03-31T23:32:20.682 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:20.678+0000 7f4311a6a640 1 -- 172.21.15.46:0/4085694273 learned_addr learned my addr 172.21.15.46:0/4085694273 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:32:20.682 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:20.678+0000 7f4316193740 1 --2- 172.21.15.46:0/4085694273 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5654980781c0 0x56549807f190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:32:20.682 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:20.678+0000 7f4316193740 1 -- 172.21.15.46:0/4085694273 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f430c1ab070 con 0x56549806aac0 2024-03-31T23:32:20.682 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:20.678+0000 7f4310a68640 1 --2- 172.21.15.46:0/4085694273 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56549806aac0 0x565498072390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:32:20.682 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:20.678+0000 7f4316193740 1 -- 172.21.15.46:0/4085694273 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f430c1ac4e0 con 0x5654980781c0 2024-03-31T23:32:20.682 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:20.678+0000 7f4316193740 1 -- 172.21.15.46:0/4085694273 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f430c1ab740 con 0x56549806a1a0 2024-03-31T23:32:20.682 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:20.679+0000 7f4316193740 10 client.0 _ll_register_callbacks cb 0x7f430400c4f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T23:32:20.682 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:20.679+0000 7f4311a6a640 1 --2- 172.21.15.46:0/4085694273 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5654980781c0 0x56549807f190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:32:20.682 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:20.679+0000 7f4311a6a640 1 -- 172.21.15.46:0/4085694273 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5654980781c0 msgr2=0x56549807f190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:32:20.682 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:20.679+0000 7f4311a6a640 1 --2- 172.21.15.46:0/4085694273 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5654980781c0 0x56549807f190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:32:20.682 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:20.679+0000 7f4311a6a640 1 -- 172.21.15.46:0/4085694273 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56549806aac0 msgr2=0x565498072390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:32:20.683 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:20.679+0000 7f4311a6a640 1 --2- 172.21.15.46:0/4085694273 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56549806aac0 0x565498072390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:32:20.683 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:20.679+0000 7f4311a6a640 1 -- 172.21.15.46:0/4085694273 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56549807f6d0 con 0x56549806a1a0 2024-03-31T23:32:20.683 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:20.679+0000 7f4311a6a640 1 --2- 172.21.15.46:0/4085694273 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56549806a1a0 0x565498071e50 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f4304003e90 tx=0x7f4304009b10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4d188392d46b972c server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:32:20.683 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:20.679+0000 7f42f4ff9640 10 client.0 ms_handle_connect on v2:172.21.15.162:3300/0 2024-03-31T23:32:20.683 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:20.679+0000 7f42f4ff9640 1 -- 172.21.15.46:0/4085694273 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f430401b040 con 0x56549806a1a0 2024-03-31T23:32:20.683 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:20.679+0000 7f4316193740 1 -- 172.21.15.46:0/4085694273 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f430401c040 con 0x56549806a1a0 2024-03-31T23:32:20.683 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:20.679+0000 7f42f4ff9640 1 -- 172.21.15.46:0/4085694273 <== mon.1 v2:172.21.15.162:3300/0 2 ==== osd_map(128..128 src has 1..128) ==== 9554+0+0 (secure 0 0 0) 0x7f4304017960 con 0x56549806a1a0 2024-03-31T23:32:20.683 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:20.679+0000 7f42f4ff9640 1 -- 172.21.15.46:0/4085694273 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_get_version_reply(handle=1 version=48) ==== 24+0+0 (secure 0 0 0) 0x7f4304017b60 con 0x56549806a1a0 2024-03-31T23:32:20.683 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:20.679+0000 7f4316193740 10 client.27535 fetch_fsmap learned FSMap version 48 2024-03-31T23:32:20.683 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:20.679+0000 7f4316193740 1 -- 172.21.15.46:0/4085694273 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,fsmap=48,monmap=4+}) -- 0x565498080860 con 0x56549806a1a0 2024-03-31T23:32:20.683 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:20.680+0000 7f42f4ff9640 1 -- 172.21.15.46:0/4085694273 <== mon.1 v2:172.21.15.162:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f430401c040 con 0x56549806a1a0 2024-03-31T23:32:20.683 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:20.680+0000 7f42f4ff9640 1 -- 172.21.15.46:0/4085694273 <== mon.1 v2:172.21.15.162:3300/0 5 ==== fsmap(e 48) ==== 8350+0+0 (secure 0 0 0) 0x7f4304043070 con 0x56549806a1a0 2024-03-31T23:32:20.683 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:20.680+0000 7f4316193740 10 client.27535 fetch_fsmap finished waiting for FSMap version 48 2024-03-31T23:32:20.683 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:20.680+0000 7f4316193740 10 client.27535 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-31T23:32:20.684 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:20.680+0000 7f4316193740 20 client.27535 populate_metadata read hostname 'smithi046' 2024-03-31T23:32:20.684 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:20.680+0000 7f4316193740 1 --2- 172.21.15.46:0/4085694273 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x5654980f2980 0x5654980f4d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:32:20.684 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:20.680+0000 7f4316193740 4 client.27535 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-31T23:32:20.684 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:20.680+0000 7f4316193740 1 -- 172.21.15.46:0/4085694273 --> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] -- command(tid 0: {"prefix": "flush journal"}) -- 0x565497fdae30 con 0x5654980f2980 2024-03-31T23:32:20.684 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:20.680+0000 7f4310a68640 1 --2- 172.21.15.46:0/4085694273 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x5654980f2980 0x5654980f4d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:32:20.684 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:20.681+0000 7f4310a68640 1 --2- 172.21.15.46:0/4085694273 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x5654980f2980 0x5654980f4d70 crc :-1 s=READY pgs=261 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bd4d83f5e2f1d67a server_cookie=210f6d56f36983b6 in_seq=0 out_seq=0 2024-03-31T23:32:20.684 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:20.681+0000 7f42f4ff9640 10 client.27535 ms_handle_connect on v2:172.21.15.142:6838/2519143918 2024-03-31T23:32:20.847 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:32:20 smithi162 ceph-mon[36237]: pgmap v653: 129 pgs: 129 active+clean; 1.0 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 5.9 MiB/s wr, 264 op/s 2024-03-31T23:32:20.847 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:32:20 smithi162 ceph-mon[36237]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' 2024-03-31T23:32:20.847 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:32:20 smithi162 ceph-mon[36237]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' 2024-03-31T23:32:20.848 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:32:20 smithi162 ceph-mon[36237]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-31T23:32:20.848 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:32:20 smithi162 ceph-mon[36237]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-31T23:32:20.848 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:32:20 smithi162 ceph-mon[36237]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-31T23:32:20.848 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:32:20 smithi162 ceph-mon[36237]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-31T23:32:20.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:32:20 smithi046 ceph-mon[32498]: pgmap v653: 129 pgs: 129 active+clean; 1.0 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 5.9 MiB/s wr, 264 op/s 2024-03-31T23:32:20.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:32:20 smithi046 ceph-mon[32498]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' 2024-03-31T23:32:20.862 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:32:20 smithi046 ceph-mon[32498]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' 2024-03-31T23:32:20.862 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:32:20 smithi046 ceph-mon[32498]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-31T23:32:20.862 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:32:20 smithi046 ceph-mon[32498]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-31T23:32:20.862 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:32:20 smithi046 ceph-mon[32498]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-31T23:32:20.862 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:32:20 smithi046 ceph-mon[32498]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-31T23:32:20.935 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:20.933+0000 7f42f4ff9640 1 -- 172.21.15.46:0/4085694273 <== mds.0 v2:172.21.15.142:6838/2519143918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x565497fdae30 con 0x5654980f2980 2024-03-31T23:32:20.935 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:20.933+0000 7f42f4ff9640 10 client.27535 handle_command_reply: tid=0 2024-03-31T23:32:20.935 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:20.933+0000 7f4316193740 1 client.27535 shutdown 2024-03-31T23:32:20.935 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:20.933+0000 7f4316193740 1 -- 172.21.15.46:0/4085694273 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56549806a1a0 msgr2=0x565498071e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:32:20.935 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:20.933+0000 7f4316193740 1 --2- 172.21.15.46:0/4085694273 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56549806a1a0 0x565498071e50 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f4304003e90 tx=0x7f4304009b10 comp rx=0 tx=0).stop 2024-03-31T23:32:20.935 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:20.933+0000 7f4316193740 1 -- 172.21.15.46:0/4085694273 shutdown_connections 2024-03-31T23:32:20.935 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:20.934+0000 7f4316193740 1 --2- 172.21.15.46:0/4085694273 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x5654980f2980 0x5654980f4d70 crc :-1 s=READY pgs=261 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:32:20.935 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:20.934+0000 7f4316193740 1 --2- 172.21.15.46:0/4085694273 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5654980781c0 0x56549807f190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:32:20.935 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:20.934+0000 7f4316193740 1 --2- 172.21.15.46:0/4085694273 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56549806aac0 0x565498072390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:32:20.936 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:20.934+0000 7f42f4ff9640 0 client.27535 ms_handle_reset on v2:172.21.15.142:6838/2519143918 2024-03-31T23:32:20.936 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:20.934+0000 7f4316193740 1 --2- 172.21.15.46:0/4085694273 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56549806a1a0 0x565498071e50 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:32:20.936 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:20.934+0000 7f4316193740 1 -- 172.21.15.46:0/4085694273 >> 172.21.15.46:0/4085694273 conn(0x565498064fe0 msgr2=0x565498066430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:32:20.936 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:20.934+0000 7f4316193740 1 -- 172.21.15.46:0/4085694273 shutdown_connections 2024-03-31T23:32:20.936 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:20.934+0000 7f4316193740 1 -- 172.21.15.46:0/4085694273 wait complete. 2024-03-31T23:32:20.936 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:20.934+0000 7f4316193740 20 client.27535 trim_cache size 0 max 16384 2024-03-31T23:32:20.936 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout:{ 2024-03-31T23:32:20.936 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout: "message": "", 2024-03-31T23:32:20.936 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout: "return_code": 0 2024-03-31T23:32:20.936 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout:} 2024-03-31T23:32:20.937 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:20.935+0000 7f43134cd640 1 -- 172.21.15.46:0/4062245025 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f42f80a27b0 msgr2=0x7f42f80a4c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:32:20.937 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:20.935+0000 7f43134cd640 1 --2- 172.21.15.46:0/4062245025 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f42f80a27b0 0x7f42f80a4c70 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x56549806b7c0 tx=0x7f4308005e80 comp rx=0 tx=0).stop 2024-03-31T23:32:20.938 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:20.935+0000 7f43134cd640 1 -- 172.21.15.46:0/4062245025 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f430c1011c0 msgr2=0x7f430c19b8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:32:20.938 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:20.935+0000 7f43134cd640 1 --2- 172.21.15.46:0/4062245025 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f430c1011c0 0x7f430c19b8c0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f4308006010 tx=0x7f4308005d90 comp rx=0 tx=0).stop 2024-03-31T23:32:20.938 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:20.935+0000 7f43134cd640 1 -- 172.21.15.46:0/4062245025 shutdown_connections 2024-03-31T23:32:20.938 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:20.935+0000 7f43134cd640 1 --2- 172.21.15.46:0/4062245025 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f42f80a27b0 0x7f42f80a4c70 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:32:20.938 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:20.935+0000 7f43134cd640 1 --2- 172.21.15.46:0/4062245025 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f430c105650 0x7f430c195a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:32:20.938 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:20.935+0000 7f43134cd640 1 --2- 172.21.15.46:0/4062245025 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f430c101b80 0x7f430c19be00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:32:20.938 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:20.935+0000 7f43134cd640 1 --2- 172.21.15.46:0/4062245025 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f430c1011c0 0x7f430c19b8c0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:32:20.938 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:20.935+0000 7f43134cd640 1 -- 172.21.15.46:0/4062245025 >> 172.21.15.46:0/4062245025 conn(0x7f430c0767e0 msgr2=0x7f430c103750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:32:20.938 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:20.935+0000 7f43134cd640 1 -- 172.21.15.46:0/4062245025 shutdown_connections 2024-03-31T23:32:20.938 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:20.935+0000 7f43134cd640 1 -- 172.21.15.46:0/4062245025 wait complete. 2024-03-31T23:32:20.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:32:20 smithi142 ceph-mon[35271]: pgmap v653: 129 pgs: 129 active+clean; 1.0 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 5.9 MiB/s wr, 264 op/s 2024-03-31T23:32:20.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:32:20 smithi142 ceph-mon[35271]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' 2024-03-31T23:32:20.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:32:20 smithi142 ceph-mon[35271]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' 2024-03-31T23:32:20.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:32:20 smithi142 ceph-mon[35271]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-31T23:32:20.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:32:20 smithi142 ceph-mon[35271]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-31T23:32:20.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:32:20 smithi142 ceph-mon[35271]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-31T23:32:20.944 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:32:20 smithi142 ceph-mon[35271]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-31T23:32:21.296 INFO:tasks.workunit.client.0.smithi046.stdout: 10 70472 8.40 MB/sec execute 326 sec latency 187.238 ms 2024-03-31T23:32:22.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:32:21 smithi162 ceph-mon[36237]: Adjusting osd_memory_target on smithi162 to 176.2M 2024-03-31T23:32:22.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:32:21 smithi162 ceph-mon[36237]: Unable to set osd_memory_target on smithi162 to 184788889: error parsing value: Value '184788889' is below minimum 939524096 2024-03-31T23:32:22.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:32:21 smithi162 ceph-mon[36237]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' 2024-03-31T23:32:22.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:32:21 smithi162 ceph-mon[36237]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' 2024-03-31T23:32:22.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:32:21 smithi162 ceph-mon[36237]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-03-31T23:32:22.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:32:21 smithi162 ceph-mon[36237]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-31T23:32:22.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:32:21 smithi162 ceph-mon[36237]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-31T23:32:22.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:32:21 smithi162 ceph-mon[36237]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-31T23:32:22.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:32:21 smithi162 ceph-mon[36237]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-31T23:32:22.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:32:21 smithi046 ceph-mon[32498]: Adjusting osd_memory_target on smithi162 to 176.2M 2024-03-31T23:32:22.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:32:21 smithi046 ceph-mon[32498]: Unable to set osd_memory_target on smithi162 to 184788889: error parsing value: Value '184788889' is below minimum 939524096 2024-03-31T23:32:22.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:32:21 smithi046 ceph-mon[32498]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' 2024-03-31T23:32:22.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:32:21 smithi046 ceph-mon[32498]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' 2024-03-31T23:32:22.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:32:21 smithi046 ceph-mon[32498]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-03-31T23:32:22.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:32:21 smithi046 ceph-mon[32498]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-31T23:32:22.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:32:21 smithi046 ceph-mon[32498]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-31T23:32:22.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:32:21 smithi046 ceph-mon[32498]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-31T23:32:22.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:32:21 smithi046 ceph-mon[32498]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-31T23:32:22.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:32:21 smithi142 ceph-mon[35271]: Adjusting osd_memory_target on smithi162 to 176.2M 2024-03-31T23:32:22.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:32:21 smithi142 ceph-mon[35271]: Unable to set osd_memory_target on smithi162 to 184788889: error parsing value: Value '184788889' is below minimum 939524096 2024-03-31T23:32:22.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:32:21 smithi142 ceph-mon[35271]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' 2024-03-31T23:32:22.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:32:21 smithi142 ceph-mon[35271]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' 2024-03-31T23:32:22.194 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:32:21 smithi142 ceph-mon[35271]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-03-31T23:32:22.194 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:32:21 smithi142 ceph-mon[35271]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-31T23:32:22.194 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:32:21 smithi142 ceph-mon[35271]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-31T23:32:22.194 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:32:21 smithi142 ceph-mon[35271]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-31T23:32:22.194 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:32:21 smithi142 ceph-mon[35271]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-31T23:32:22.296 INFO:tasks.workunit.client.0.smithi046.stdout: 10 70616 8.41 MB/sec execute 327 sec latency 194.480 ms 2024-03-31T23:32:23.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:32:22 smithi162 ceph-mon[36237]: pgmap v654: 129 pgs: 129 active+clean; 1.0 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 5.7 MiB/s wr, 255 op/s 2024-03-31T23:32:23.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:32:22 smithi162 ceph-mon[36237]: Adjusting osd_memory_target on smithi142 to 176.2M 2024-03-31T23:32:23.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:32:22 smithi162 ceph-mon[36237]: Unable to set osd_memory_target on smithi142 to 184770252: error parsing value: Value '184770252' is below minimum 939524096 2024-03-31T23:32:23.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:32:22 smithi046 ceph-mon[32498]: pgmap v654: 129 pgs: 129 active+clean; 1.0 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 5.7 MiB/s wr, 255 op/s 2024-03-31T23:32:23.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:32:22 smithi046 ceph-mon[32498]: Adjusting osd_memory_target on smithi142 to 176.2M 2024-03-31T23:32:23.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:32:22 smithi046 ceph-mon[32498]: Unable to set osd_memory_target on smithi142 to 184770252: error parsing value: Value '184770252' is below minimum 939524096 2024-03-31T23:32:23.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:32:22 smithi142 ceph-mon[35271]: pgmap v654: 129 pgs: 129 active+clean; 1.0 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 5.7 MiB/s wr, 255 op/s 2024-03-31T23:32:23.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:32:22 smithi142 ceph-mon[35271]: Adjusting osd_memory_target on smithi142 to 176.2M 2024-03-31T23:32:23.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:32:22 smithi142 ceph-mon[35271]: Unable to set osd_memory_target on smithi142 to 184770252: error parsing value: Value '184770252' is below minimum 939524096 2024-03-31T23:32:23.296 INFO:tasks.workunit.client.0.smithi046.stdout: 10 70760 8.40 MB/sec execute 328 sec latency 187.265 ms 2024-03-31T23:32:24.296 INFO:tasks.workunit.client.0.smithi046.stdout: 10 70929 8.41 MB/sec execute 329 sec latency 72.735 ms 2024-03-31T23:32:25.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:32:24 smithi162 ceph-mon[36237]: pgmap v655: 129 pgs: 129 active+clean; 1.0 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 4.9 MiB/s wr, 220 op/s 2024-03-31T23:32:25.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:32:24 smithi046 ceph-mon[32498]: pgmap v655: 129 pgs: 129 active+clean; 1.0 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 4.9 MiB/s wr, 220 op/s 2024-03-31T23:32:25.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:32:24 smithi142 ceph-mon[35271]: pgmap v655: 129 pgs: 129 active+clean; 1.0 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 4.9 MiB/s wr, 220 op/s 2024-03-31T23:32:25.296 INFO:tasks.workunit.client.0.smithi046.stdout: 10 71044 8.40 MB/sec execute 330 sec latency 110.561 ms 2024-03-31T23:32:26.296 INFO:tasks.workunit.client.0.smithi046.stdout: 10 71158 8.40 MB/sec execute 331 sec latency 148.824 ms 2024-03-31T23:32:27.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:32:26 smithi162 ceph-mon[36237]: pgmap v656: 129 pgs: 129 active+clean; 1.0 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 7.2 MiB/s wr, 390 op/s 2024-03-31T23:32:27.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:32:26 smithi162 ceph-mon[36237]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-03-31T23:32:27.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:32:26 smithi046 ceph-mon[32498]: pgmap v656: 129 pgs: 129 active+clean; 1.0 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 7.2 MiB/s wr, 390 op/s 2024-03-31T23:32:27.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:32:26 smithi046 ceph-mon[32498]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-03-31T23:32:27.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:32:26 smithi142 ceph-mon[35271]: pgmap v656: 129 pgs: 129 active+clean; 1.0 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 7.2 MiB/s wr, 390 op/s 2024-03-31T23:32:27.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:32:26 smithi142 ceph-mon[35271]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-03-31T23:32:27.296 INFO:tasks.workunit.client.0.smithi046.stdout: 10 71265 8.39 MB/sec execute 332 sec latency 189.203 ms 2024-03-31T23:32:28.296 INFO:tasks.workunit.client.0.smithi046.stdout: 10 71350 8.37 MB/sec execute 333 sec latency 211.807 ms 2024-03-31T23:32:29.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:32:28 smithi162 ceph-mon[36237]: pgmap v657: 129 pgs: 129 active+clean; 1.0 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 4.5 MiB/s wr, 272 op/s 2024-03-31T23:32:29.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:32:28 smithi046 ceph-mon[32498]: pgmap v657: 129 pgs: 129 active+clean; 1.0 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 4.5 MiB/s wr, 272 op/s 2024-03-31T23:32:29.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:32:28 smithi142 ceph-mon[35271]: pgmap v657: 129 pgs: 129 active+clean; 1.0 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 4.5 MiB/s wr, 272 op/s 2024-03-31T23:32:29.296 INFO:tasks.workunit.client.0.smithi046.stdout: 10 71421 8.36 MB/sec execute 334 sec latency 446.277 ms 2024-03-31T23:32:30.296 INFO:tasks.workunit.client.0.smithi046.stdout: 10 71552 8.34 MB/sec execute 335 sec latency 83.414 ms 2024-03-31T23:32:30.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:32:30 smithi142 ceph-mon[35271]: pgmap v658: 129 pgs: 129 active+clean; 1.0 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 6.0 MiB/s wr, 362 op/s 2024-03-31T23:32:30.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:32:30 smithi142 ceph-mon[35271]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' 2024-03-31T23:32:30.944 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:32:30 smithi142 ceph-mon[35271]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' 2024-03-31T23:32:30.944 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:32:30 smithi142 ceph-mon[35271]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-31T23:32:30.944 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:32:30 smithi142 ceph-mon[35271]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-31T23:32:30.944 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:32:30 smithi142 ceph-mon[35271]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-31T23:32:30.944 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:32:30 smithi142 ceph-mon[35271]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-31T23:32:30.944 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:32:30 smithi142 ceph-mon[35271]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:32:30.944 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:32:30 smithi142 ceph-mon[35271]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-31T23:32:30.944 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:32:30 smithi142 ceph-mon[35271]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' 2024-03-31T23:32:31.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:32:30 smithi162 ceph-mon[36237]: pgmap v658: 129 pgs: 129 active+clean; 1.0 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 6.0 MiB/s wr, 362 op/s 2024-03-31T23:32:31.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:32:30 smithi162 ceph-mon[36237]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' 2024-03-31T23:32:31.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:32:30 smithi162 ceph-mon[36237]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' 2024-03-31T23:32:31.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:32:30 smithi162 ceph-mon[36237]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-31T23:32:31.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:32:30 smithi162 ceph-mon[36237]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-31T23:32:31.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:32:30 smithi162 ceph-mon[36237]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-31T23:32:31.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:32:30 smithi162 ceph-mon[36237]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-31T23:32:31.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:32:30 smithi162 ceph-mon[36237]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:32:31.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:32:30 smithi162 ceph-mon[36237]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-31T23:32:31.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:32:30 smithi162 ceph-mon[36237]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' 2024-03-31T23:32:31.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:32:30 smithi046 ceph-mon[32498]: pgmap v658: 129 pgs: 129 active+clean; 1.0 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 6.0 MiB/s wr, 362 op/s 2024-03-31T23:32:31.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:32:30 smithi046 ceph-mon[32498]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' 2024-03-31T23:32:31.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:32:30 smithi046 ceph-mon[32498]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' 2024-03-31T23:32:31.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:32:30 smithi046 ceph-mon[32498]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-31T23:32:31.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:32:30 smithi046 ceph-mon[32498]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-31T23:32:31.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:32:30 smithi046 ceph-mon[32498]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-31T23:32:31.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:32:30 smithi046 ceph-mon[32498]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-31T23:32:31.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:32:30 smithi046 ceph-mon[32498]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:32:31.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:32:30 smithi046 ceph-mon[32498]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-31T23:32:31.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:32:30 smithi046 ceph-mon[32498]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' 2024-03-31T23:32:31.296 INFO:tasks.workunit.client.0.smithi046.stdout: 10 71707 8.34 MB/sec execute 336 sec latency 91.953 ms 2024-03-31T23:32:31.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:32:31 smithi142 ceph-mon[35271]: Adjusting osd_memory_target on smithi046 to 177.2M 2024-03-31T23:32:31.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:32:31 smithi142 ceph-mon[35271]: Unable to set osd_memory_target on smithi046 to 185899212: error parsing value: Value '185899212' is below minimum 939524096 2024-03-31T23:32:32.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:32:31 smithi162 ceph-mon[36237]: Adjusting osd_memory_target on smithi046 to 177.2M 2024-03-31T23:32:32.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:32:31 smithi162 ceph-mon[36237]: Unable to set osd_memory_target on smithi046 to 185899212: error parsing value: Value '185899212' is below minimum 939524096 2024-03-31T23:32:32.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:32:31 smithi046 ceph-mon[32498]: Adjusting osd_memory_target on smithi046 to 177.2M 2024-03-31T23:32:32.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:32:31 smithi046 ceph-mon[32498]: Unable to set osd_memory_target on smithi046 to 185899212: error parsing value: Value '185899212' is below minimum 939524096 2024-03-31T23:32:32.296 INFO:tasks.workunit.client.0.smithi046.stdout: 10 71855 8.33 MB/sec execute 337 sec latency 95.995 ms 2024-03-31T23:32:32.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:32:32 smithi142 ceph-mon[35271]: pgmap v659: 129 pgs: 129 active+clean; 1.0 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 3.7 MiB/s wr, 260 op/s 2024-03-31T23:32:33.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:32:32 smithi162 ceph-mon[36237]: pgmap v659: 129 pgs: 129 active+clean; 1.0 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 3.7 MiB/s wr, 260 op/s 2024-03-31T23:32:33.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:32:32 smithi046 ceph-mon[32498]: pgmap v659: 129 pgs: 129 active+clean; 1.0 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 3.7 MiB/s wr, 260 op/s 2024-03-31T23:32:33.296 INFO:tasks.workunit.client.0.smithi046.stdout: 10 72008 8.33 MB/sec execute 338 sec latency 104.621 ms 2024-03-31T23:32:34.067 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:34.065+0000 7f16d9454640 1 -- 172.21.15.46:0/3792916330 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f16d41020c0 msgr2=0x7f16d410e5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:32:34.067 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:34.065+0000 7f16d9454640 1 --2- 172.21.15.46:0/3792916330 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f16d41020c0 0x7f16d410e5d0 secure :-1 s=READY pgs=983 cs=0 l=1 rev1=1 crypto rx=0x7f16cc009cd0 tx=0x7f16cc02c0f0 comp rx=0 tx=0).stop 2024-03-31T23:32:34.067 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:34.065+0000 7f16d9454640 1 -- 172.21.15.46:0/3792916330 shutdown_connections 2024-03-31T23:32:34.067 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:34.065+0000 7f16d9454640 1 --2- 172.21.15.46:0/3792916330 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f16d41020c0 0x7f16d410e5d0 unknown :-1 s=CLOSED pgs=983 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:32:34.067 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:34.065+0000 7f16d9454640 1 --2- 172.21.15.46:0/3792916330 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f16d4101700 0x7f16d4101b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:32:34.067 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:34.065+0000 7f16d9454640 1 --2- 172.21.15.46:0/3792916330 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f16d4107710 0x7f16d4107af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:32:34.068 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:34.065+0000 7f16d9454640 1 -- 172.21.15.46:0/3792916330 >> 172.21.15.46:0/3792916330 conn(0x7f16d40fd1d0 msgr2=0x7f16d40ff5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:32:34.068 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:34.065+0000 7f16d9454640 1 -- 172.21.15.46:0/3792916330 shutdown_connections 2024-03-31T23:32:34.068 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:34.065+0000 7f16d9454640 1 -- 172.21.15.46:0/3792916330 wait complete. 2024-03-31T23:32:34.068 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:34.066+0000 7f16d9454640 1 Processor -- start 2024-03-31T23:32:34.070 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:34.066+0000 7f16d9454640 1 -- start start 2024-03-31T23:32:34.070 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:34.067+0000 7f16d9454640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f16d4101700 0x7f16d4073a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:32:34.070 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:34.067+0000 7f16d9454640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f16d41020c0 0x7f16d4073f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:32:34.070 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:34.067+0000 7f16d9454640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f16d4107710 0x7f16d40744d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:32:34.070 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:34.067+0000 7f16d9454640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f16d41119d0 con 0x7f16d41020c0 2024-03-31T23:32:34.070 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:34.067+0000 7f16d9454640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f16d4111850 con 0x7f16d4107710 2024-03-31T23:32:34.070 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:34.067+0000 7f16d9454640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f16d4111b50 con 0x7f16d4101700 2024-03-31T23:32:34.070 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:34.067+0000 7f16d27fc640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f16d41020c0 0x7f16d4073f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:32:34.070 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:34.067+0000 7f16d27fc640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f16d41020c0 0x7f16d4073f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39230/0 (socket says 172.21.15.46:39230) 2024-03-31T23:32:34.071 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:34.067+0000 7f16d27fc640 1 -- 172.21.15.46:0/594411510 learned_addr learned my addr 172.21.15.46:0/594411510 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:32:34.071 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:34.067+0000 7f16d2ffd640 1 --2- 172.21.15.46:0/594411510 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f16d4101700 0x7f16d4073a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:32:34.071 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:34.067+0000 7f16d37fe640 1 --2- 172.21.15.46:0/594411510 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f16d4107710 0x7f16d40744d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:32:34.071 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:34.067+0000 7f16d27fc640 1 -- 172.21.15.46:0/594411510 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f16d4101700 msgr2=0x7f16d4073a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:32:34.071 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:34.067+0000 7f16d27fc640 1 --2- 172.21.15.46:0/594411510 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f16d4101700 0x7f16d4073a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:32:34.071 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:34.067+0000 7f16d27fc640 1 -- 172.21.15.46:0/594411510 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f16d4107710 msgr2=0x7f16d40744d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:32:34.071 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:34.067+0000 7f16d27fc640 1 --2- 172.21.15.46:0/594411510 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f16d4107710 0x7f16d40744d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:32:34.071 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:34.067+0000 7f16d27fc640 1 -- 172.21.15.46:0/594411510 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f16d41a6370 con 0x7f16d41020c0 2024-03-31T23:32:34.071 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:34.067+0000 7f16d2ffd640 1 --2- 172.21.15.46:0/594411510 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f16d4101700 0x7f16d4073a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:32:34.071 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:34.067+0000 7f16d27fc640 1 --2- 172.21.15.46:0/594411510 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f16d41020c0 0x7f16d4073f90 secure :-1 s=READY pgs=984 cs=0 l=1 rev1=1 crypto rx=0x7f16c800ea60 tx=0x7f16c800ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:32:34.071 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:34.068+0000 7f16bbfff640 1 -- 172.21.15.46:0/594411510 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16c800cde0 con 0x7f16d41020c0 2024-03-31T23:32:34.071 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:34.068+0000 7f16bbfff640 1 -- 172.21.15.46:0/594411510 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f16c80035f0 con 0x7f16d41020c0 2024-03-31T23:32:34.071 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:34.068+0000 7f16d9454640 1 -- 172.21.15.46:0/594411510 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f16d41a6660 con 0x7f16d41020c0 2024-03-31T23:32:34.071 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:34.068+0000 7f16bbfff640 1 -- 172.21.15.46:0/594411510 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16c8010690 con 0x7f16d41020c0 2024-03-31T23:32:34.071 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:34.068+0000 7f16d9454640 1 -- 172.21.15.46:0/594411510 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f16d41a6c00 con 0x7f16d41020c0 2024-03-31T23:32:34.072 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:34.070+0000 7f16bbfff640 1 -- 172.21.15.46:0/594411510 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 25) ==== 137451+0+0 (secure 0 0 0) 0x7f16c80037a0 con 0x7f16d41020c0 2024-03-31T23:32:34.072 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:34.071+0000 7f16bbfff640 1 --2- 172.21.15.46:0/594411510 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f16bc0a2730 0x7f16bc0a4bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:32:34.072 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:34.071+0000 7f16bbfff640 1 -- 172.21.15.46:0/594411510 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(128..128 src has 1..128) ==== 9554+0+0 (secure 0 0 0) 0x7f16c80cf140 con 0x7f16d41020c0 2024-03-31T23:32:34.072 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:34.071+0000 7f16d2ffd640 1 --2- 172.21.15.46:0/594411510 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f16bc0a2730 0x7f16bc0a4bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:32:34.072 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:34.071+0000 7f16d2ffd640 1 --2- 172.21.15.46:0/594411510 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f16bc0a2730 0x7f16bc0a4bf0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f16c4009f40 tx=0x7f16c4005f90 comp rx=0 tx=0).ready entity=mgr.15654 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:32:34.072 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:34.071+0000 7f16dc030740 1 Processor -- start 2024-03-31T23:32:34.073 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:34.072+0000 7f16dc030740 1 -- start start 2024-03-31T23:32:34.074 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:34.072+0000 7f16dc030740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cbbbf156c0 0x55cbbbf15aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:32:34.074 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:34.072+0000 7f16dc030740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55cbbbf15fe0 0x55cbbbf1e870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:32:34.074 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:34.072+0000 7f16dc030740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55cbbbf1edb0 0x55cbbbf211a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:32:34.074 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:34.072+0000 7f16dc030740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55cbbbe788f0 con 0x55cbbbf15fe0 2024-03-31T23:32:34.074 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:34.072+0000 7f16dc030740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55cbbbe218d0 con 0x55cbbbf1edb0 2024-03-31T23:32:34.074 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:34.072+0000 7f16dc030740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55cbbbe74990 con 0x55cbbbf156c0 2024-03-31T23:32:34.074 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:34.073+0000 7f16d27fc640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55cbbbf15fe0 0x55cbbbf1e870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:32:34.075 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:34.073+0000 7f16d27fc640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55cbbbf15fe0 0x55cbbbf1e870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39238/0 (socket says 172.21.15.46:39238) 2024-03-31T23:32:34.075 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:34.073+0000 7f16d27fc640 1 -- 172.21.15.46:0/532345782 learned_addr learned my addr 172.21.15.46:0/532345782 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:32:34.075 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:34.073+0000 7f16d37fe640 1 --2- 172.21.15.46:0/532345782 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55cbbbf1edb0 0x55cbbbf211a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:32:34.075 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:34.073+0000 7f16d2ffd640 1 --2- 172.21.15.46:0/532345782 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cbbbf156c0 0x55cbbbf15aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:32:34.075 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:34.073+0000 7f16d27fc640 1 -- 172.21.15.46:0/532345782 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cbbbf156c0 msgr2=0x55cbbbf15aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:32:34.075 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:34.073+0000 7f16d27fc640 1 --2- 172.21.15.46:0/532345782 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cbbbf156c0 0x55cbbbf15aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:32:34.075 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:34.073+0000 7f16d27fc640 1 -- 172.21.15.46:0/532345782 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55cbbbf1edb0 msgr2=0x55cbbbf211a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:32:34.075 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:34.073+0000 7f16d27fc640 1 --2- 172.21.15.46:0/532345782 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55cbbbf1edb0 0x55cbbbf211a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:32:34.075 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:34.073+0000 7f16d27fc640 1 -- 172.21.15.46:0/532345782 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cbbbb8d210 con 0x55cbbbf15fe0 2024-03-31T23:32:34.075 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:34.073+0000 7f16d27fc640 1 --2- 172.21.15.46:0/532345782 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55cbbbf15fe0 0x55cbbbf1e870 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 crypto rx=0x7f16c8097910 tx=0x7f16c8097860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=edb5603d06b0ce99 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:32:34.077 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:34.073+0000 7f16b97fa640 1 -- 172.21.15.46:0/532345782 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16c80b4070 con 0x55cbbbf15fe0 2024-03-31T23:32:34.077 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:34.074+0000 7f16b97fa640 1 -- 172.21.15.46:0/532345782 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f16c809bbb0 con 0x55cbbbf15fe0 2024-03-31T23:32:34.077 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:34.074+0000 7f16b97fa640 1 -- 172.21.15.46:0/532345782 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16c80cc700 con 0x55cbbbf15fe0 2024-03-31T23:32:34.077 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:34.074+0000 7f16dc030740 1 -- 172.21.15.46:0/532345782 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55cbbbf15fe0 msgr2=0x55cbbbf1e870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:32:34.077 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:34.074+0000 7f16dc030740 1 --2- 172.21.15.46:0/532345782 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55cbbbf15fe0 0x55cbbbf1e870 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 crypto rx=0x7f16c8097910 tx=0x7f16c8097860 comp rx=0 tx=0).stop 2024-03-31T23:32:34.078 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:34.075+0000 7f16dc030740 1 -- 172.21.15.46:0/532345782 shutdown_connections 2024-03-31T23:32:34.078 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:34.075+0000 7f16dc030740 1 --2- 172.21.15.46:0/532345782 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55cbbbf1edb0 0x55cbbbf211a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:32:34.078 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:34.075+0000 7f16dc030740 1 --2- 172.21.15.46:0/532345782 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55cbbbf15fe0 0x55cbbbf1e870 unknown :-1 s=CLOSED pgs=985 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:32:34.078 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:34.075+0000 7f16dc030740 1 --2- 172.21.15.46:0/532345782 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cbbbf156c0 0x55cbbbf15aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:32:34.078 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:34.075+0000 7f16dc030740 1 -- 172.21.15.46:0/532345782 >> 172.21.15.46:0/532345782 conn(0x55cbbbf0ffe0 msgr2=0x55cbbbf103c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:32:34.078 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:34.075+0000 7f16dc030740 1 -- 172.21.15.46:0/532345782 shutdown_connections 2024-03-31T23:32:34.078 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:34.075+0000 7f16dc030740 1 -- 172.21.15.46:0/532345782 wait complete. 2024-03-31T23:32:34.078 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:34.075+0000 7f16dc030740 1 -- start start 2024-03-31T23:32:34.078 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:34.075+0000 7f16dc030740 1 Processor -- start 2024-03-31T23:32:34.078 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:34.075+0000 7f16dc030740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cbbbf156c0 0x55cbbbf1bf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:32:34.078 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:34.075+0000 7f16dc030740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55cbbbf15fe0 0x55cbbbf1c4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:32:34.078 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:34.075+0000 7f16dc030740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55cbbbf1edb0 0x55cbbbf1c9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:32:34.078 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:34.075+0000 7f16dc030740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55cbbbf219e0 con 0x55cbbbf15fe0 2024-03-31T23:32:34.079 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:34.075+0000 7f16dc030740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55cbbbe16f80 con 0x55cbbbf1edb0 2024-03-31T23:32:34.079 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:34.075+0000 7f16dc030740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55cbbbf21b60 con 0x55cbbbf156c0 2024-03-31T23:32:34.079 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:34.075+0000 7f16dc030740 10 client.0 _ll_register_callbacks cb 0x55cbbbe6f290 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T23:32:34.079 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:34.076+0000 7f16d2ffd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cbbbf156c0 0x55cbbbf1bf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:32:34.079 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:34.076+0000 7f16d27fc640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55cbbbf15fe0 0x55cbbbf1c4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:32:34.079 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:34.076+0000 7f16d2ffd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cbbbf156c0 0x55cbbbf1bf70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.46:50052/0 (socket says 172.21.15.46:50052) 2024-03-31T23:32:34.081 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:34.076+0000 7f16d2ffd640 1 -- 172.21.15.46:0/2801194274 learned_addr learned my addr 172.21.15.46:0/2801194274 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:32:34.081 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:34.076+0000 7f16d37fe640 1 --2- 172.21.15.46:0/2801194274 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55cbbbf1edb0 0x55cbbbf1c9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:32:34.081 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:34.076+0000 7f16d27fc640 1 -- 172.21.15.46:0/2801194274 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cbbbf156c0 msgr2=0x55cbbbf1bf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:32:34.082 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:34.076+0000 7f16d27fc640 1 --2- 172.21.15.46:0/2801194274 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cbbbf156c0 0x55cbbbf1bf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:32:34.082 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:34.076+0000 7f16d27fc640 1 -- 172.21.15.46:0/2801194274 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55cbbbf1edb0 msgr2=0x55cbbbf1c9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:32:34.082 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:34.076+0000 7f16d27fc640 1 --2- 172.21.15.46:0/2801194274 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55cbbbf1edb0 0x55cbbbf1c9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:32:34.082 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:34.076+0000 7f16d27fc640 1 -- 172.21.15.46:0/2801194274 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cbbbc59620 con 0x55cbbbf15fe0 2024-03-31T23:32:34.082 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:34.076+0000 7f16d2ffd640 1 --2- 172.21.15.46:0/2801194274 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cbbbf156c0 0x55cbbbf1bf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:32:34.082 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:34.076+0000 7f16d37fe640 1 --2- 172.21.15.46:0/2801194274 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55cbbbf1edb0 0x55cbbbf1c9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T23:32:34.082 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:34.076+0000 7f16d27fc640 1 --2- 172.21.15.46:0/2801194274 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55cbbbf15fe0 0x55cbbbf1c4b0 secure :-1 s=READY pgs=986 cs=0 l=1 rev1=1 crypto rx=0x7f16c809bd60 tx=0x7f16c8004230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce338e2efcac8a96 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:32:34.082 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:34.076+0000 7f16b6ff5640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-31T23:32:34.082 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:34.076+0000 7f16b6ff5640 1 -- 172.21.15.46:0/2801194274 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16c80b4040 con 0x55cbbbf15fe0 2024-03-31T23:32:34.082 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:34.076+0000 7f16b6ff5640 1 -- 172.21.15.46:0/2801194274 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(128..128 src has 1..128) ==== 9554+0+0 (secure 0 0 0) 0x7f16c8097b20 con 0x55cbbbf15fe0 2024-03-31T23:32:34.082 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:34.076+0000 7f16dc030740 1 -- 172.21.15.46:0/2801194274 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f16c809bbb0 con 0x55cbbbf15fe0 2024-03-31T23:32:34.082 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:34.077+0000 7f16b6ff5640 1 -- 172.21.15.46:0/2801194274 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=50) ==== 24+0+0 (secure 0 0 0) 0x7f16c809f600 con 0x55cbbbf15fe0 2024-03-31T23:32:34.082 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:34.077+0000 7f16dc030740 10 client.17829 fetch_fsmap learned FSMap version 50 2024-03-31T23:32:34.082 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:34.077+0000 7f16dc030740 1 -- 172.21.15.46:0/2801194274 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=50,monmap=4+}) -- 0x55cbbbf2d840 con 0x55cbbbf15fe0 2024-03-31T23:32:34.084 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:34.077+0000 7f16b6ff5640 1 -- 172.21.15.46:0/2801194274 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f16c80bb110 con 0x55cbbbf15fe0 2024-03-31T23:32:34.084 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:34.077+0000 7f16b6ff5640 1 -- 172.21.15.46:0/2801194274 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 50) ==== 8358+0+0 (secure 0 0 0) 0x7f16c809f170 con 0x55cbbbf15fe0 2024-03-31T23:32:34.084 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:34.077+0000 7f16dc030740 10 client.17829 fetch_fsmap finished waiting for FSMap version 50 2024-03-31T23:32:34.084 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:34.077+0000 7f16dc030740 10 client.17829 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-31T23:32:34.084 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:34.077+0000 7f16dc030740 20 client.17829 populate_metadata read hostname 'smithi046' 2024-03-31T23:32:34.084 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:34.077+0000 7f16dc030740 1 --2- 172.21.15.46:0/2801194274 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55cbbbf2eb90 0x55cbbbf30f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:32:34.084 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:34.077+0000 7f16dc030740 4 client.17829 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-31T23:32:34.085 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:34.077+0000 7f16dc030740 1 -- 172.21.15.46:0/2801194274 --> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f16c809bbb0 con 0x55cbbbf2eb90 2024-03-31T23:32:34.085 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:34.077+0000 7f16d2ffd640 1 --2- 172.21.15.46:0/2801194274 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55cbbbf2eb90 0x55cbbbf30f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:32:34.085 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:34.078+0000 7f16b6ff5640 10 client.17829 ms_handle_connect on v2:172.21.15.142:6838/2519143918 2024-03-31T23:32:34.085 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:34.078+0000 7f16d2ffd640 1 --2- 172.21.15.46:0/2801194274 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55cbbbf2eb90 0x55cbbbf30f80 crc :-1 s=READY pgs=262 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=743fba03b69d90a5 server_cookie=3611fbded516a639 in_seq=0 out_seq=0 2024-03-31T23:32:34.085 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:34.080+0000 7f16b6ff5640 1 -- 172.21.15.46:0/2801194274 <== mds.0 v2:172.21.15.142:6838/2519143918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x7f16c809bbb0 con 0x55cbbbf2eb90 2024-03-31T23:32:34.085 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:34.080+0000 7f16b6ff5640 10 client.17829 handle_command_reply: tid=0 2024-03-31T23:32:34.085 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:34.080+0000 7f16dc030740 1 client.17829 shutdown 2024-03-31T23:32:34.085 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:34.080+0000 7f16dc030740 1 -- 172.21.15.46:0/2801194274 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55cbbbf15fe0 msgr2=0x55cbbbf1c4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:32:34.085 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:34.080+0000 7f16dc030740 1 --2- 172.21.15.46:0/2801194274 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55cbbbf15fe0 0x55cbbbf1c4b0 secure :-1 s=READY pgs=986 cs=0 l=1 rev1=1 crypto rx=0x7f16c809bd60 tx=0x7f16c8004230 comp rx=0 tx=0).stop 2024-03-31T23:32:34.085 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:34.080+0000 7f16dc030740 1 -- 172.21.15.46:0/2801194274 shutdown_connections 2024-03-31T23:32:34.085 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:34.080+0000 7f16dc030740 1 --2- 172.21.15.46:0/2801194274 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55cbbbf2eb90 0x55cbbbf30f80 crc :-1 s=READY pgs=262 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:32:34.085 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:34.080+0000 7f16dc030740 1 --2- 172.21.15.46:0/2801194274 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55cbbbf1edb0 0x55cbbbf1c9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:32:34.085 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:34.080+0000 7f16dc030740 1 --2- 172.21.15.46:0/2801194274 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55cbbbf15fe0 0x55cbbbf1c4b0 unknown :-1 s=CLOSED pgs=986 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:32:34.085 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:34.080+0000 7f16dc030740 1 --2- 172.21.15.46:0/2801194274 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cbbbf156c0 0x55cbbbf1bf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:32:34.085 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:34.080+0000 7f16dc030740 1 -- 172.21.15.46:0/2801194274 >> 172.21.15.46:0/2801194274 conn(0x55cbbbf0ffe0 msgr2=0x55cbbbf11d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:32:34.086 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:34.080+0000 7f16b6ff5640 0 client.17829 ms_handle_reset on v2:172.21.15.142:6838/2519143918 2024-03-31T23:32:34.086 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:34.080+0000 7f16dc030740 1 -- 172.21.15.46:0/2801194274 shutdown_connections 2024-03-31T23:32:34.086 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:34.080+0000 7f16dc030740 1 -- 172.21.15.46:0/2801194274 wait complete. 2024-03-31T23:32:34.086 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:34.080+0000 7f16dc030740 20 client.17829 trim_cache size 0 max 16384 2024-03-31T23:32:34.105 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:34.103+0000 7f16dc030740 1 Processor -- start 2024-03-31T23:32:34.105 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:34.103+0000 7f16dc030740 1 -- start start 2024-03-31T23:32:34.105 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:34.103+0000 7f16dc030740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cbbbf156c0 0x55cbbbf1e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:32:34.105 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:34.103+0000 7f16dc030740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55cbbbf1edb0 0x55cbbbf1b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:32:34.106 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:34.103+0000 7f16dc030740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55cbbbf178e0 0x55cbbbf1c640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:32:34.106 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:34.103+0000 7f16dc030740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f16d41af460 con 0x55cbbbf178e0 2024-03-31T23:32:34.106 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:34.103+0000 7f16dc030740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f16d41acca0 con 0x55cbbbf1edb0 2024-03-31T23:32:34.106 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:34.103+0000 7f16dc030740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f16d41aab10 con 0x55cbbbf156c0 2024-03-31T23:32:34.106 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:34.104+0000 7f16d2ffd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cbbbf156c0 0x55cbbbf1e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:32:34.106 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:34.104+0000 7f16d27fc640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55cbbbf1edb0 0x55cbbbf1b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:32:34.106 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:34.104+0000 7f16d37fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55cbbbf178e0 0x55cbbbf1c640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:32:34.106 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:34.104+0000 7f16d27fc640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55cbbbf1edb0 0x55cbbbf1b0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.46:33454/0 (socket says 172.21.15.46:33454) 2024-03-31T23:32:34.106 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:34.104+0000 7f16d27fc640 1 -- 172.21.15.46:0/3754300764 learned_addr learned my addr 172.21.15.46:0/3754300764 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:32:34.106 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:34.104+0000 7f16d2ffd640 1 -- 172.21.15.46:0/3754300764 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55cbbbf1edb0 msgr2=0x55cbbbf1b0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:32:34.106 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:34.104+0000 7f16d2ffd640 1 --2- 172.21.15.46:0/3754300764 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55cbbbf1edb0 0x55cbbbf1b0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:32:34.106 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:34.104+0000 7f16d2ffd640 1 -- 172.21.15.46:0/3754300764 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55cbbbf178e0 msgr2=0x55cbbbf1c640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:32:34.106 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:34.104+0000 7f16d2ffd640 1 --2- 172.21.15.46:0/3754300764 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55cbbbf178e0 0x55cbbbf1c640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:32:34.107 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:34.104+0000 7f16d2ffd640 1 -- 172.21.15.46:0/3754300764 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cbbbf2bb60 con 0x55cbbbf156c0 2024-03-31T23:32:34.107 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:34.104+0000 7f16d2ffd640 1 --2- 172.21.15.46:0/3754300764 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cbbbf156c0 0x55cbbbf1e6a0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f16c4011ca0 tx=0x7f16c4017000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c1ce086e73e21114 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:32:34.107 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:34.104+0000 7f16b8ff9640 1 -- 172.21.15.46:0/3754300764 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16c4039070 con 0x55cbbbf156c0 2024-03-31T23:32:34.107 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:34.105+0000 7f16b8ff9640 1 -- 172.21.15.46:0/3754300764 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f16c4004030 con 0x55cbbbf156c0 2024-03-31T23:32:34.107 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:34.105+0000 7f16b8ff9640 1 -- 172.21.15.46:0/3754300764 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16c40345c0 con 0x55cbbbf156c0 2024-03-31T23:32:34.110 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:34.105+0000 7f16dc030740 1 -- 172.21.15.46:0/3754300764 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cbbbf156c0 msgr2=0x55cbbbf1e6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:32:34.110 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:34.105+0000 7f16dc030740 1 --2- 172.21.15.46:0/3754300764 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cbbbf156c0 0x55cbbbf1e6a0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f16c4011ca0 tx=0x7f16c4017000 comp rx=0 tx=0).stop 2024-03-31T23:32:34.110 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:34.105+0000 7f16dc030740 1 -- 172.21.15.46:0/3754300764 shutdown_connections 2024-03-31T23:32:34.111 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:34.105+0000 7f16dc030740 1 --2- 172.21.15.46:0/3754300764 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55cbbbf178e0 0x55cbbbf1c640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:32:34.111 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:34.105+0000 7f16dc030740 1 --2- 172.21.15.46:0/3754300764 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55cbbbf1edb0 0x55cbbbf1b0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:32:34.111 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:34.105+0000 7f16dc030740 1 --2- 172.21.15.46:0/3754300764 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cbbbf156c0 0x55cbbbf1e6a0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:32:34.111 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:34.106+0000 7f16dc030740 1 -- 172.21.15.46:0/3754300764 >> 172.21.15.46:0/3754300764 conn(0x55cbbbf0ffe0 msgr2=0x55cbbbf1bf30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:32:34.111 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:34.106+0000 7f16dc030740 1 -- 172.21.15.46:0/3754300764 shutdown_connections 2024-03-31T23:32:34.111 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:34.106+0000 7f16dc030740 1 -- 172.21.15.46:0/3754300764 wait complete. 2024-03-31T23:32:34.111 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:34.106+0000 7f16dc030740 1 -- start start 2024-03-31T23:32:34.111 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:34.106+0000 7f16dc030740 1 Processor -- start 2024-03-31T23:32:34.111 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:34.106+0000 7f16dc030740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55cbbbf156c0 0x55cbbbf33d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:32:34.111 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:34.106+0000 7f16dc030740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55cbbbf178e0 0x55cbbbf311c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:32:34.111 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:34.106+0000 7f16dc030740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cbbbf1edb0 0x55cbbbf31700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:32:34.111 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:34.106+0000 7f16dc030740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f16d41ab160 con 0x55cbbbf178e0 2024-03-31T23:32:34.111 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:34.106+0000 7f16dc030740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f16d41ac5d0 con 0x55cbbbf156c0 2024-03-31T23:32:34.112 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:34.106+0000 7f16dc030740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f16d41ab830 con 0x55cbbbf1edb0 2024-03-31T23:32:34.112 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:34.106+0000 7f16d27fc640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55cbbbf178e0 0x55cbbbf311c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:32:34.112 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:34.106+0000 7f16d2ffd640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55cbbbf156c0 0x55cbbbf33d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:32:34.112 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:34.106+0000 7f16d27fc640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55cbbbf178e0 0x55cbbbf311c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39258/0 (socket says 172.21.15.46:39258) 2024-03-31T23:32:34.112 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:34.106+0000 7f16d27fc640 1 -- 172.21.15.46:0/2908630937 learned_addr learned my addr 172.21.15.46:0/2908630937 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:32:34.112 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:34.107+0000 7f16d37fe640 1 --2- 172.21.15.46:0/2908630937 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cbbbf1edb0 0x55cbbbf31700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:32:34.112 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:34.107+0000 7f16dc030740 10 client.0 _ll_register_callbacks cb 0x7f16c80b2070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T23:32:34.112 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:34.107+0000 7f16d2ffd640 1 -- 172.21.15.46:0/2908630937 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cbbbf1edb0 msgr2=0x55cbbbf31700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:32:34.112 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:34.107+0000 7f16d2ffd640 1 --2- 172.21.15.46:0/2908630937 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cbbbf1edb0 0x55cbbbf31700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:32:34.112 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:34.107+0000 7f16d2ffd640 1 -- 172.21.15.46:0/2908630937 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55cbbbf178e0 msgr2=0x55cbbbf311c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:32:34.112 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:34.107+0000 7f16d2ffd640 1 --2- 172.21.15.46:0/2908630937 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55cbbbf178e0 0x55cbbbf311c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:32:34.112 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:34.107+0000 7f16d2ffd640 1 -- 172.21.15.46:0/2908630937 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cbbbf31e40 con 0x55cbbbf156c0 2024-03-31T23:32:34.113 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:34.107+0000 7f16d27fc640 1 --2- 172.21.15.46:0/2908630937 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55cbbbf178e0 0x55cbbbf311c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:32:34.113 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:34.107+0000 7f16d2ffd640 1 --2- 172.21.15.46:0/2908630937 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55cbbbf156c0 0x55cbbbf33d70 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f16c4009e00 tx=0x7f16c40056d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c5c21975fd261973 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:32:34.113 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:34.107+0000 7f16b6ff5640 10 client.0 ms_handle_connect on v2:172.21.15.142:3300/0 2024-03-31T23:32:34.113 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:34.107+0000 7f16b6ff5640 1 -- 172.21.15.46:0/2908630937 <== mon.2 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16c4039040 con 0x55cbbbf156c0 2024-03-31T23:32:34.113 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:34.107+0000 7f16b6ff5640 1 -- 172.21.15.46:0/2908630937 <== mon.2 v2:172.21.15.142:3300/0 2 ==== osd_map(128..128 src has 1..128) ==== 9554+0+0 (secure 0 0 0) 0x7f16c4003a40 con 0x55cbbbf156c0 2024-03-31T23:32:34.113 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:34.107+0000 7f16dc030740 1 -- 172.21.15.46:0/2908630937 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f16c4004030 con 0x55cbbbf156c0 2024-03-31T23:32:34.113 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:34.108+0000 7f16b6ff5640 1 -- 172.21.15.46:0/2908630937 <== mon.2 v2:172.21.15.142:3300/0 3 ==== mon_get_version_reply(handle=1 version=50) ==== 24+0+0 (secure 0 0 0) 0x7f16c4049050 con 0x55cbbbf156c0 2024-03-31T23:32:34.113 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:34.108+0000 7f16dc030740 10 client.27659 fetch_fsmap learned FSMap version 50 2024-03-31T23:32:34.113 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:34.108+0000 7f16dc030740 1 -- 172.21.15.46:0/2908630937 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,fsmap=50,monmap=4+}) -- 0x55cbbbf94a30 con 0x55cbbbf156c0 2024-03-31T23:32:34.113 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:34.108+0000 7f16b6ff5640 1 -- 172.21.15.46:0/2908630937 <== mon.2 v2:172.21.15.142:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f16c4004030 con 0x55cbbbf156c0 2024-03-31T23:32:34.113 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:34.108+0000 7f16b6ff5640 1 -- 172.21.15.46:0/2908630937 <== mon.2 v2:172.21.15.142:3300/0 5 ==== fsmap(e 50) ==== 8358+0+0 (secure 0 0 0) 0x7f16c4040480 con 0x55cbbbf156c0 2024-03-31T23:32:34.113 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:34.108+0000 7f16dc030740 10 client.27659 fetch_fsmap finished waiting for FSMap version 50 2024-03-31T23:32:34.113 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:34.108+0000 7f16dc030740 10 client.27659 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-31T23:32:34.113 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:34.108+0000 7f16dc030740 20 client.27659 populate_metadata read hostname 'smithi046' 2024-03-31T23:32:34.113 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:34.109+0000 7f16dc030740 1 --2- 172.21.15.46:0/2908630937 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55cbbbf951e0 0x55cbbbf97600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:32:34.114 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:34.109+0000 7f16dc030740 4 client.27659 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-31T23:32:34.114 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:34.109+0000 7f16dc030740 1 -- 172.21.15.46:0/2908630937 --> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55cbbbe8c120 con 0x55cbbbf951e0 2024-03-31T23:32:34.114 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:34.109+0000 7f16d27fc640 1 --2- 172.21.15.46:0/2908630937 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55cbbbf951e0 0x55cbbbf97600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:32:34.114 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:34.109+0000 7f16d27fc640 1 --2- 172.21.15.46:0/2908630937 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55cbbbf951e0 0x55cbbbf97600 crc :-1 s=READY pgs=263 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ae0829257b6cd50b server_cookie=6074147f480c94e4 in_seq=0 out_seq=0 2024-03-31T23:32:34.114 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:34.109+0000 7f16b6ff5640 10 client.27659 ms_handle_connect on v2:172.21.15.142:6838/2519143918 2024-03-31T23:32:34.296 INFO:tasks.workunit.client.0.smithi046.stdout: 10 72172 8.34 MB/sec execute 339 sec latency 168.136 ms 2024-03-31T23:32:34.343 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:34.342+0000 7f16b6ff5640 1 -- 172.21.15.46:0/2908630937 <== mds.0 v2:172.21.15.142:6838/2519143918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55cbbbe8c120 con 0x55cbbbf951e0 2024-03-31T23:32:34.343 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:34.342+0000 7f16b6ff5640 10 client.27659 handle_command_reply: tid=0 2024-03-31T23:32:34.343 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:34.342+0000 7f16dc030740 1 client.27659 shutdown 2024-03-31T23:32:34.344 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:34.343+0000 7f16dc030740 1 -- 172.21.15.46:0/2908630937 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55cbbbf156c0 msgr2=0x55cbbbf33d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:32:34.344 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:34.343+0000 7f16dc030740 1 --2- 172.21.15.46:0/2908630937 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55cbbbf156c0 0x55cbbbf33d70 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f16c4009e00 tx=0x7f16c40056d0 comp rx=0 tx=0).stop 2024-03-31T23:32:34.346 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:34.345+0000 7f16dc030740 1 -- 172.21.15.46:0/2908630937 shutdown_connections 2024-03-31T23:32:34.346 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:34.345+0000 7f16dc030740 1 --2- 172.21.15.46:0/2908630937 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55cbbbf951e0 0x55cbbbf97600 crc :-1 s=READY pgs=263 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:32:34.346 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:34.345+0000 7f16dc030740 1 --2- 172.21.15.46:0/2908630937 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cbbbf1edb0 0x55cbbbf31700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:32:34.346 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:34.345+0000 7f16dc030740 1 --2- 172.21.15.46:0/2908630937 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55cbbbf178e0 0x55cbbbf311c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:32:34.346 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:34.345+0000 7f16b6ff5640 0 client.27659 ms_handle_reset on v2:172.21.15.142:6838/2519143918 2024-03-31T23:32:34.346 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:34.345+0000 7f16dc030740 1 --2- 172.21.15.46:0/2908630937 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55cbbbf156c0 0x55cbbbf33d70 secure :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f16c4009e00 tx=0x7f16c40056d0 comp rx=0 tx=0).stop 2024-03-31T23:32:34.346 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:34.345+0000 7f16dc030740 1 -- 172.21.15.46:0/2908630937 >> 172.21.15.46:0/2908630937 conn(0x55cbbbf0ffe0 msgr2=0x55cbbbf0ef80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:32:34.348 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:34.346+0000 7f16dc030740 1 -- 172.21.15.46:0/2908630937 shutdown_connections 2024-03-31T23:32:34.348 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:34.346+0000 7f16dc030740 1 -- 172.21.15.46:0/2908630937 wait complete. 2024-03-31T23:32:34.348 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:34.346+0000 7f16dc030740 20 client.27659 trim_cache size 0 max 16384 2024-03-31T23:32:34.348 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout:{ 2024-03-31T23:32:34.348 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout: "message": "", 2024-03-31T23:32:34.348 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout: "return_code": 0 2024-03-31T23:32:34.348 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout:} 2024-03-31T23:32:34.349 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:34.347+0000 7f16d9454640 1 -- 172.21.15.46:0/594411510 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f16bc0a2730 msgr2=0x7f16bc0a4bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:32:34.349 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:34.347+0000 7f16d9454640 1 --2- 172.21.15.46:0/594411510 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f16bc0a2730 0x7f16bc0a4bf0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f16c4009f40 tx=0x7f16c4005f90 comp rx=0 tx=0).stop 2024-03-31T23:32:34.349 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:34.347+0000 7f16d9454640 1 -- 172.21.15.46:0/594411510 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f16d41020c0 msgr2=0x7f16d4073f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:32:34.349 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:34.347+0000 7f16d9454640 1 --2- 172.21.15.46:0/594411510 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f16d41020c0 0x7f16d4073f90 secure :-1 s=READY pgs=984 cs=0 l=1 rev1=1 crypto rx=0x7f16c800ea60 tx=0x7f16c800ef20 comp rx=0 tx=0).stop 2024-03-31T23:32:34.350 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:34.347+0000 7f16d9454640 1 -- 172.21.15.46:0/594411510 shutdown_connections 2024-03-31T23:32:34.350 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:34.348+0000 7f16d9454640 1 --2- 172.21.15.46:0/594411510 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f16bc0a2730 0x7f16bc0a4bf0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:32:34.350 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:34.348+0000 7f16d9454640 1 --2- 172.21.15.46:0/594411510 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f16d4107710 0x7f16d40744d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:32:34.350 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:34.348+0000 7f16d9454640 1 --2- 172.21.15.46:0/594411510 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f16d41020c0 0x7f16d4073f90 unknown :-1 s=CLOSED pgs=984 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:32:34.350 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:34.348+0000 7f16d9454640 1 --2- 172.21.15.46:0/594411510 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f16d4101700 0x7f16d4073a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:32:34.350 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:34.348+0000 7f16d9454640 1 -- 172.21.15.46:0/594411510 >> 172.21.15.46:0/594411510 conn(0x7f16d40fd1d0 msgr2=0x7f16d40fd950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:32:34.350 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:34.348+0000 7f16d9454640 1 -- 172.21.15.46:0/594411510 shutdown_connections 2024-03-31T23:32:34.350 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:34.348+0000 7f16d9454640 1 -- 172.21.15.46:0/594411510 wait complete. 2024-03-31T23:32:34.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:32:34 smithi142 ceph-mon[35271]: pgmap v660: 129 pgs: 129 active+clean; 1.0 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 3.7 MiB/s wr, 260 op/s 2024-03-31T23:32:35.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:32:34 smithi162 ceph-mon[36237]: pgmap v660: 129 pgs: 129 active+clean; 1.0 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 3.7 MiB/s wr, 260 op/s 2024-03-31T23:32:35.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:32:34 smithi046 ceph-mon[32498]: pgmap v660: 129 pgs: 129 active+clean; 1.0 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 3.7 MiB/s wr, 260 op/s 2024-03-31T23:32:35.296 INFO:tasks.workunit.client.0.smithi046.stdout: 10 72336 8.34 MB/sec execute 340 sec latency 210.647 ms 2024-03-31T23:32:36.296 INFO:tasks.workunit.client.0.smithi046.stdout: 10 72510 8.36 MB/sec execute 341 sec latency 150.327 ms 2024-03-31T23:32:36.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:32:36 smithi142 ceph-mon[35271]: pgmap v661: 129 pgs: 129 active+clean; 1.0 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 5.9 MiB/s wr, 367 op/s 2024-03-31T23:32:37.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:32:36 smithi162 ceph-mon[36237]: pgmap v661: 129 pgs: 129 active+clean; 1.0 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 5.9 MiB/s wr, 367 op/s 2024-03-31T23:32:37.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:32:36 smithi046 ceph-mon[32498]: pgmap v661: 129 pgs: 129 active+clean; 1.0 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 5.9 MiB/s wr, 367 op/s 2024-03-31T23:32:37.296 INFO:tasks.workunit.client.0.smithi046.stdout: 10 72687 8.36 MB/sec execute 342 sec latency 111.031 ms 2024-03-31T23:32:38.296 INFO:tasks.workunit.client.0.smithi046.stdout: 10 72849 8.37 MB/sec execute 343 sec latency 63.179 ms 2024-03-31T23:32:38.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:32:38 smithi142 ceph-mon[35271]: pgmap v662: 129 pgs: 129 active+clean; 1.0 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 3.7 MiB/s wr, 197 op/s 2024-03-31T23:32:39.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:32:38 smithi162 ceph-mon[36237]: pgmap v662: 129 pgs: 129 active+clean; 1.0 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 3.7 MiB/s wr, 197 op/s 2024-03-31T23:32:39.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:32:38 smithi046 ceph-mon[32498]: pgmap v662: 129 pgs: 129 active+clean; 1.0 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 3.7 MiB/s wr, 197 op/s 2024-03-31T23:32:39.296 INFO:tasks.workunit.client.0.smithi046.stdout: 10 72994 8.37 MB/sec execute 344 sec latency 95.393 ms 2024-03-31T23:32:40.296 INFO:tasks.workunit.client.0.smithi046.stdout: 10 73125 8.35 MB/sec execute 345 sec latency 95.356 ms 2024-03-31T23:32:40.913 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:32:40 smithi046 ceph-mon[32498]: pgmap v663: 129 pgs: 129 active+clean; 1.0 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 6.2 MiB/s wr, 353 op/s 2024-03-31T23:32:40.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:32:40 smithi142 ceph-mon[35271]: pgmap v663: 129 pgs: 129 active+clean; 1.0 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 6.2 MiB/s wr, 353 op/s 2024-03-31T23:32:41.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:32:40 smithi162 ceph-mon[36237]: pgmap v663: 129 pgs: 129 active+clean; 1.0 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 6.2 MiB/s wr, 353 op/s 2024-03-31T23:32:41.297 INFO:tasks.workunit.client.0.smithi046.stdout: 10 73272 8.35 MB/sec execute 346 sec latency 119.913 ms 2024-03-31T23:32:42.296 INFO:tasks.workunit.client.0.smithi046.stdout: 10 73406 8.34 MB/sec execute 347 sec latency 108.175 ms 2024-03-31T23:32:42.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:32:42 smithi142 ceph-mon[35271]: pgmap v664: 129 pgs: 129 active+clean; 1.0 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 4.7 MiB/s wr, 262 op/s 2024-03-31T23:32:43.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:32:42 smithi162 ceph-mon[36237]: pgmap v664: 129 pgs: 129 active+clean; 1.0 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 4.7 MiB/s wr, 262 op/s 2024-03-31T23:32:43.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:32:42 smithi046 ceph-mon[32498]: pgmap v664: 129 pgs: 129 active+clean; 1.0 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 4.7 MiB/s wr, 262 op/s 2024-03-31T23:32:43.297 INFO:tasks.workunit.client.0.smithi046.stdout: 10 73569 8.34 MB/sec execute 348 sec latency 89.320 ms 2024-03-31T23:32:44.297 INFO:tasks.workunit.client.0.smithi046.stdout: 10 73743 8.34 MB/sec execute 349 sec latency 118.918 ms 2024-03-31T23:32:44.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:32:44 smithi142 ceph-mon[35271]: pgmap v665: 129 pgs: 129 active+clean; 1.0 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 4.7 MiB/s wr, 262 op/s 2024-03-31T23:32:45.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:32:44 smithi162 ceph-mon[36237]: pgmap v665: 129 pgs: 129 active+clean; 1.0 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 4.7 MiB/s wr, 262 op/s 2024-03-31T23:32:45.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:32:44 smithi046 ceph-mon[32498]: pgmap v665: 129 pgs: 129 active+clean; 1.0 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 4.7 MiB/s wr, 262 op/s 2024-03-31T23:32:45.297 INFO:tasks.workunit.client.0.smithi046.stdout: 10 73912 8.34 MB/sec execute 350 sec latency 104.804 ms 2024-03-31T23:32:46.297 INFO:tasks.workunit.client.0.smithi046.stdout: 10 74052 8.35 MB/sec execute 351 sec latency 82.727 ms 2024-03-31T23:32:46.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:32:46 smithi142 ceph-mon[35271]: pgmap v666: 129 pgs: 129 active+clean; 1.0 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 7.0 MiB/s wr, 370 op/s 2024-03-31T23:32:47.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:32:46 smithi162 ceph-mon[36237]: pgmap v666: 129 pgs: 129 active+clean; 1.0 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 7.0 MiB/s wr, 370 op/s 2024-03-31T23:32:47.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:32:46 smithi046 ceph-mon[32498]: pgmap v666: 129 pgs: 129 active+clean; 1.0 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 7.0 MiB/s wr, 370 op/s 2024-03-31T23:32:47.297 INFO:tasks.workunit.client.0.smithi046.stdout: 10 74205 8.36 MB/sec execute 352 sec latency 86.019 ms 2024-03-31T23:32:47.467 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:47.466+0000 7f8541d1e640 1 -- 172.21.15.46:0/491123079 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f853c1089a0 msgr2=0x7f853c108d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:32:47.467 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:47.466+0000 7f8541d1e640 1 --2- 172.21.15.46:0/491123079 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f853c1089a0 0x7f853c108d80 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f852c009900 tx=0x7f852c02bc40 comp rx=0 tx=0).stop 2024-03-31T23:32:47.467 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:47.467+0000 7f8541d1e640 1 -- 172.21.15.46:0/491123079 shutdown_connections 2024-03-31T23:32:47.467 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:47.467+0000 7f8541d1e640 1 --2- 172.21.15.46:0/491123079 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f853c116ca0 0x7f853c119090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:32:47.467 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:47.467+0000 7f8541d1e640 1 --2- 172.21.15.46:0/491123079 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f853c1092c0 0x7f853c116760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:32:47.467 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:47.467+0000 7f8541d1e640 1 --2- 172.21.15.46:0/491123079 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f853c1089a0 0x7f853c108d80 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:32:47.467 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:47.467+0000 7f8541d1e640 1 -- 172.21.15.46:0/491123079 >> 172.21.15.46:0/491123079 conn(0x7f853c06c7f0 msgr2=0x7f853c06cc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:32:47.467 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:47.467+0000 7f8541d1e640 1 -- 172.21.15.46:0/491123079 shutdown_connections 2024-03-31T23:32:47.468 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:47.467+0000 7f8541d1e640 1 -- 172.21.15.46:0/491123079 wait complete. 2024-03-31T23:32:47.469 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:47.467+0000 7f8541d1e640 1 Processor -- start 2024-03-31T23:32:47.469 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:47.468+0000 7f8541d1e640 1 -- start start 2024-03-31T23:32:47.470 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:47.468+0000 7f8541d1e640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f853c1089a0 0x7f853c1ac990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:32:47.470 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:47.468+0000 7f8541d1e640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f853c1092c0 0x7f853c1aced0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:32:47.470 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:47.468+0000 7f8541d1e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f853c116ca0 0x7f853c1a70b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:32:47.470 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:47.468+0000 7f8541d1e640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f853c112bd0 con 0x7f853c116ca0 2024-03-31T23:32:47.470 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:47.468+0000 7f853bfff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f853c116ca0 0x7f853c1a70b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:32:47.470 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:47.468+0000 7f853bfff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f853c116ca0 0x7f853c1a70b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:50044/0 (socket says 172.21.15.46:50044) 2024-03-31T23:32:47.470 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:47.468+0000 7f853affd640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f853c1092c0 0x7f853c1aced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:32:47.470 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:47.468+0000 7f853bfff640 1 -- 172.21.15.46:0/3269205777 learned_addr learned my addr 172.21.15.46:0/3269205777 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:32:47.470 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:47.468+0000 7f853b7fe640 1 --2- 172.21.15.46:0/3269205777 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f853c1089a0 0x7f853c1ac990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:32:47.470 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:47.468+0000 7f8541d1e640 1 -- 172.21.15.46:0/3269205777 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f853c112a50 con 0x7f853c1092c0 2024-03-31T23:32:47.470 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:47.468+0000 7f8541d1e640 1 -- 172.21.15.46:0/3269205777 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f853c112d50 con 0x7f853c1089a0 2024-03-31T23:32:47.471 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:47.469+0000 7f853bfff640 1 -- 172.21.15.46:0/3269205777 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f853c1089a0 msgr2=0x7f853c1ac990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:32:47.471 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:47.469+0000 7f853bfff640 1 --2- 172.21.15.46:0/3269205777 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f853c1089a0 0x7f853c1ac990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:32:47.471 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:47.469+0000 7f853bfff640 1 -- 172.21.15.46:0/3269205777 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f853c1092c0 msgr2=0x7f853c1aced0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:32:47.471 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:47.469+0000 7f853bfff640 1 --2- 172.21.15.46:0/3269205777 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f853c1092c0 0x7f853c1aced0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:32:47.471 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:47.469+0000 7f853bfff640 1 -- 172.21.15.46:0/3269205777 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f853c1a7880 con 0x7f853c116ca0 2024-03-31T23:32:47.471 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:47.469+0000 7f853b7fe640 1 --2- 172.21.15.46:0/3269205777 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f853c1089a0 0x7f853c1ac990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:32:47.471 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:47.469+0000 7f853bfff640 1 --2- 172.21.15.46:0/3269205777 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f853c116ca0 0x7f853c1a70b0 secure :-1 s=READY pgs=987 cs=0 l=1 rev1=1 crypto rx=0x7f853400eb10 tx=0x7f853400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:32:47.471 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:47.469+0000 7f8538ff9640 1 -- 172.21.15.46:0/3269205777 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8534004b20 con 0x7f853c116ca0 2024-03-31T23:32:47.471 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:47.469+0000 7f8538ff9640 1 -- 172.21.15.46:0/3269205777 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8534004cc0 con 0x7f853c116ca0 2024-03-31T23:32:47.471 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:47.469+0000 7f8541d1e640 1 -- 172.21.15.46:0/3269205777 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f853c1a7b70 con 0x7f853c116ca0 2024-03-31T23:32:47.471 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:47.469+0000 7f8538ff9640 1 -- 172.21.15.46:0/3269205777 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8534010690 con 0x7f853c116ca0 2024-03-31T23:32:47.471 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:47.470+0000 7f8541d1e640 1 -- 172.21.15.46:0/3269205777 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f853c1b37e0 con 0x7f853c116ca0 2024-03-31T23:32:47.473 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:47.473+0000 7f8538ff9640 1 -- 172.21.15.46:0/3269205777 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 25) ==== 137451+0+0 (secure 0 0 0) 0x7f8534010830 con 0x7f853c116ca0 2024-03-31T23:32:47.473 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:47.473+0000 7f85449d9740 1 Processor -- start 2024-03-31T23:32:47.473 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:47.473+0000 7f85449d9740 1 -- start start 2024-03-31T23:32:47.474 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:47.473+0000 7f85449d9740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c363acf6c0 0x55c363acfaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:32:47.474 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:47.473+0000 7f85449d9740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c363acffe0 0x55c363ad8870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:32:47.474 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:47.473+0000 7f85449d9740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55c363ad8db0 0x55c363adb1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:32:47.474 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:47.473+0000 7f85449d9740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55c363a328f0 con 0x55c363acf6c0 2024-03-31T23:32:47.474 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:47.473+0000 7f85449d9740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55c3639db8d0 con 0x55c363ad8db0 2024-03-31T23:32:47.475 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:47.473+0000 7f85449d9740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55c363a2e990 con 0x55c363acffe0 2024-03-31T23:32:47.475 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:47.474+0000 7f853affd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c363acf6c0 0x55c363acfaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:32:47.475 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:47.474+0000 7f853b7fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c363acffe0 0x55c363ad8870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:32:47.475 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:47.474+0000 7f853affd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c363acf6c0 0x55c363acfaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:50046/0 (socket says 172.21.15.46:50046) 2024-03-31T23:32:47.475 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:47.474+0000 7f853affd640 1 -- 172.21.15.46:0/3407588882 learned_addr learned my addr 172.21.15.46:0/3407588882 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:32:47.475 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:47.474+0000 7f8538ff9640 1 --2- 172.21.15.46:0/3269205777 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f85240a26b0 0x7f85240a4b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:32:47.475 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:47.474+0000 7f853bfff640 1 --2- 172.21.15.46:0/3269205777 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f85240a26b0 0x7f85240a4b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:32:47.475 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:47.474+0000 7f853b7fe640 1 -- 172.21.15.46:0/3407588882 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55c363ad8db0 msgr2=0x55c363adb1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:32:47.475 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:47.474+0000 7f853b7fe640 1 --2- 172.21.15.46:0/3407588882 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55c363ad8db0 0x55c363adb1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:32:47.475 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:47.474+0000 7f853b7fe640 1 -- 172.21.15.46:0/3407588882 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c363acf6c0 msgr2=0x55c363acfaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:32:47.475 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:47.474+0000 7f853b7fe640 1 --2- 172.21.15.46:0/3407588882 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c363acf6c0 0x55c363acfaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:32:47.475 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:47.474+0000 7f853b7fe640 1 -- 172.21.15.46:0/3407588882 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c363747210 con 0x55c363acffe0 2024-03-31T23:32:47.475 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:47.474+0000 7f8538ff9640 1 -- 172.21.15.46:0/3269205777 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(128..128 src has 1..128) ==== 9554+0+0 (secure 0 0 0) 0x7f85340cf510 con 0x7f853c116ca0 2024-03-31T23:32:47.476 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:47.475+0000 7f853bfff640 1 --2- 172.21.15.46:0/3269205777 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f85240a26b0 0x7f85240a4b70 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f85340cc9a0 tx=0x7f8534097ca0 comp rx=0 tx=0).ready entity=mgr.15654 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:32:47.477 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:47.475+0000 7f853b7fe640 1 --2- 172.21.15.46:0/3407588882 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c363acffe0 0x55c363ad8870 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f852c02c180 tx=0x7f852c02c680 comp rx=0 tx=0).ready entity=mon.1 client_cookie=13c6a37fdd1618f0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:32:47.477 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:47.475+0000 7f8521ffb640 1 -- 172.21.15.46:0/3407588882 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f852c057070 con 0x55c363acffe0 2024-03-31T23:32:47.477 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:47.476+0000 7f8521ffb640 1 -- 172.21.15.46:0/3407588882 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f852c02c7f0 con 0x55c363acffe0 2024-03-31T23:32:47.477 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:47.476+0000 7f8521ffb640 1 -- 172.21.15.46:0/3407588882 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f852c003c50 con 0x55c363acffe0 2024-03-31T23:32:47.477 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:47.477+0000 7f85449d9740 1 -- 172.21.15.46:0/3407588882 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c363acffe0 msgr2=0x55c363ad8870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:32:47.477 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:47.477+0000 7f85449d9740 1 --2- 172.21.15.46:0/3407588882 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c363acffe0 0x55c363ad8870 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f852c02c180 tx=0x7f852c02c680 comp rx=0 tx=0).stop 2024-03-31T23:32:47.477 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:47.477+0000 7f85449d9740 1 -- 172.21.15.46:0/3407588882 shutdown_connections 2024-03-31T23:32:47.477 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:47.477+0000 7f85449d9740 1 --2- 172.21.15.46:0/3407588882 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55c363ad8db0 0x55c363adb1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:32:47.477 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:47.477+0000 7f85449d9740 1 --2- 172.21.15.46:0/3407588882 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c363acffe0 0x55c363ad8870 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:32:47.478 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:47.477+0000 7f85449d9740 1 --2- 172.21.15.46:0/3407588882 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c363acf6c0 0x55c363acfaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:32:47.478 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:47.477+0000 7f85449d9740 1 -- 172.21.15.46:0/3407588882 >> 172.21.15.46:0/3407588882 conn(0x55c363ac9fe0 msgr2=0x55c363aca3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:32:47.479 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:47.479+0000 7f85449d9740 1 -- 172.21.15.46:0/3407588882 shutdown_connections 2024-03-31T23:32:47.479 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:47.479+0000 7f85449d9740 1 -- 172.21.15.46:0/3407588882 wait complete. 2024-03-31T23:32:47.480 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:47.480+0000 7f85449d9740 1 -- start start 2024-03-31T23:32:47.480 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:47.480+0000 7f85449d9740 1 Processor -- start 2024-03-31T23:32:47.481 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:47.481+0000 7f85449d9740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55c363acf6c0 0x55c363ad5e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:32:47.481 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:47.481+0000 7f85449d9740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c363acffe0 0x55c363ad63d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:32:47.482 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:47.481+0000 7f85449d9740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c363ad8db0 0x55c363ad6910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:32:47.482 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:47.481+0000 7f853affd640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55c363acf6c0 0x55c363ad5e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:32:47.482 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:47.481+0000 7f853affd640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55c363acf6c0 0x55c363ad5e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.46:33948/0 (socket says 172.21.15.46:33948) 2024-03-31T23:32:47.482 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:47.481+0000 7f853affd640 1 -- 172.21.15.46:0/1723539907 learned_addr learned my addr 172.21.15.46:0/1723539907 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:32:47.482 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:47.481+0000 7f853b7fe640 1 --2- 172.21.15.46:0/1723539907 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c363acffe0 0x55c363ad63d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:32:47.482 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:47.481+0000 7f85449d9740 1 -- 172.21.15.46:0/1723539907 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55c363adb980 con 0x55c363ad8db0 2024-03-31T23:32:47.482 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:47.481+0000 7f85449d9740 1 -- 172.21.15.46:0/1723539907 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55c3639d0f80 con 0x55c363acf6c0 2024-03-31T23:32:47.482 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:47.481+0000 7f85449d9740 1 -- 172.21.15.46:0/1723539907 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55c363adbb00 con 0x55c363acffe0 2024-03-31T23:32:47.482 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:47.481+0000 7f85449d9740 10 client.0 _ll_register_callbacks cb 0x55c363a29290 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T23:32:47.482 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:47.482+0000 7f853affd640 1 --2- 172.21.15.46:0/1723539907 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c363ad8db0 0x55c363ad6910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:32:47.484 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:47.482+0000 7f853affd640 1 -- 172.21.15.46:0/1723539907 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c363acffe0 msgr2=0x55c363ad63d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:32:47.484 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:47.482+0000 7f853affd640 1 --2- 172.21.15.46:0/1723539907 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c363acffe0 0x55c363ad63d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:32:47.485 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:47.482+0000 7f853affd640 1 -- 172.21.15.46:0/1723539907 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c363ad8db0 msgr2=0x55c363ad6910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:32:47.485 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:47.482+0000 7f853affd640 1 --2- 172.21.15.46:0/1723539907 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c363ad8db0 0x55c363ad6910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:32:47.485 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:47.482+0000 7f853affd640 1 -- 172.21.15.46:0/1723539907 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c363813620 con 0x55c363acf6c0 2024-03-31T23:32:47.485 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:47.482+0000 7f853affd640 1 --2- 172.21.15.46:0/1723539907 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55c363acf6c0 0x55c363ad5e90 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f8530018cf0 tx=0x7f853000c390 comp rx=0 tx=0).ready entity=mon.2 client_cookie=37ab24072ae8fd12 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:32:47.485 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:47.482+0000 7f851affd640 10 client.0 ms_handle_connect on v2:172.21.15.142:3300/0 2024-03-31T23:32:47.485 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:47.483+0000 7f851affd640 1 -- 172.21.15.46:0/1723539907 <== mon.2 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f853001e060 con 0x55c363acf6c0 2024-03-31T23:32:47.485 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:47.483+0000 7f85449d9740 1 -- 172.21.15.46:0/1723539907 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f852c02c7f0 con 0x55c363acf6c0 2024-03-31T23:32:47.485 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:47.483+0000 7f851affd640 1 -- 172.21.15.46:0/1723539907 <== mon.2 v2:172.21.15.142:3300/0 2 ==== osd_map(128..128 src has 1..128) ==== 9554+0+0 (secure 0 0 0) 0x7f8530016e00 con 0x55c363acf6c0 2024-03-31T23:32:47.485 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:47.483+0000 7f851affd640 1 -- 172.21.15.46:0/1723539907 <== mon.2 v2:172.21.15.142:3300/0 3 ==== mon_get_version_reply(handle=1 version=50) ==== 24+0+0 (secure 0 0 0) 0x7f8530025050 con 0x55c363acf6c0 2024-03-31T23:32:47.485 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:47.483+0000 7f85449d9740 10 client.27665 fetch_fsmap learned FSMap version 50 2024-03-31T23:32:47.485 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:47.483+0000 7f85449d9740 1 -- 172.21.15.46:0/1723539907 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,fsmap=50,monmap=4+}) -- 0x55c363ae7970 con 0x55c363acf6c0 2024-03-31T23:32:47.485 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:47.483+0000 7f851affd640 1 -- 172.21.15.46:0/1723539907 <== mon.2 v2:172.21.15.142:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8530014a90 con 0x55c363acf6c0 2024-03-31T23:32:47.485 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:47.483+0000 7f851affd640 1 -- 172.21.15.46:0/1723539907 <== mon.2 v2:172.21.15.142:3300/0 5 ==== fsmap(e 50) ==== 8358+0+0 (secure 0 0 0) 0x7f8530015040 con 0x55c363acf6c0 2024-03-31T23:32:47.486 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:47.484+0000 7f85449d9740 10 client.27665 fetch_fsmap finished waiting for FSMap version 50 2024-03-31T23:32:47.486 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:47.484+0000 7f85449d9740 10 client.27665 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-31T23:32:47.486 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:47.484+0000 7f85449d9740 20 client.27665 populate_metadata read hostname 'smithi046' 2024-03-31T23:32:47.486 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:47.484+0000 7f85449d9740 1 --2- 172.21.15.46:0/1723539907 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55c363ae8e30 0x55c363aeb220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:32:47.486 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:47.484+0000 7f85449d9740 4 client.27665 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-31T23:32:47.486 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:47.484+0000 7f85449d9740 1 -- 172.21.15.46:0/1723539907 --> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f852c02c7f0 con 0x55c363ae8e30 2024-03-31T23:32:47.486 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:47.484+0000 7f853b7fe640 1 --2- 172.21.15.46:0/1723539907 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55c363ae8e30 0x55c363aeb220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:32:47.486 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:47.484+0000 7f853b7fe640 1 --2- 172.21.15.46:0/1723539907 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55c363ae8e30 0x55c363aeb220 crc :-1 s=READY pgs=264 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1c2ac44a06d1cc2 server_cookie=81a05b57871a7e4d in_seq=0 out_seq=0 2024-03-31T23:32:47.486 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:47.484+0000 7f851affd640 10 client.27665 ms_handle_connect on v2:172.21.15.142:6838/2519143918 2024-03-31T23:32:47.487 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:47.485+0000 7f851affd640 1 -- 172.21.15.46:0/1723539907 <== mds.0 v2:172.21.15.142:6838/2519143918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x7f852c02c7f0 con 0x55c363ae8e30 2024-03-31T23:32:47.487 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:47.485+0000 7f851affd640 10 client.27665 handle_command_reply: tid=0 2024-03-31T23:32:47.487 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:47.485+0000 7f85449d9740 1 client.27665 shutdown 2024-03-31T23:32:47.488 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:47.485+0000 7f85449d9740 1 -- 172.21.15.46:0/1723539907 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55c363acf6c0 msgr2=0x55c363ad5e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:32:47.488 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:47.485+0000 7f85449d9740 1 --2- 172.21.15.46:0/1723539907 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55c363acf6c0 0x55c363ad5e90 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f8530018cf0 tx=0x7f853000c390 comp rx=0 tx=0).stop 2024-03-31T23:32:47.488 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:47.485+0000 7f85449d9740 1 -- 172.21.15.46:0/1723539907 shutdown_connections 2024-03-31T23:32:47.488 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:47.485+0000 7f85449d9740 1 --2- 172.21.15.46:0/1723539907 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55c363ae8e30 0x55c363aeb220 crc :-1 s=READY pgs=264 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:32:47.488 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:47.485+0000 7f85449d9740 1 --2- 172.21.15.46:0/1723539907 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c363ad8db0 0x55c363ad6910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:32:47.488 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:47.485+0000 7f851affd640 0 client.27665 ms_handle_reset on v2:172.21.15.142:6838/2519143918 2024-03-31T23:32:47.488 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:47.485+0000 7f85449d9740 1 --2- 172.21.15.46:0/1723539907 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c363acffe0 0x55c363ad63d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:32:47.488 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:47.485+0000 7f85449d9740 1 --2- 172.21.15.46:0/1723539907 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55c363acf6c0 0x55c363ad5e90 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:32:47.488 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:47.485+0000 7f85449d9740 1 -- 172.21.15.46:0/1723539907 >> 172.21.15.46:0/1723539907 conn(0x55c363ac9fe0 msgr2=0x55c363acbd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:32:47.488 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:47.485+0000 7f85449d9740 1 -- 172.21.15.46:0/1723539907 shutdown_connections 2024-03-31T23:32:47.488 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:47.486+0000 7f85449d9740 1 -- 172.21.15.46:0/1723539907 wait complete. 2024-03-31T23:32:47.488 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:47.486+0000 7f85449d9740 20 client.27665 trim_cache size 0 max 16384 2024-03-31T23:32:47.513 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:47.512+0000 7f85449d9740 1 Processor -- start 2024-03-31T23:32:47.513 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:47.512+0000 7f85449d9740 1 -- start start 2024-03-31T23:32:47.513 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:47.512+0000 7f85449d9740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c363acf6c0 0x55c363ad85c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:32:47.513 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:47.512+0000 7f85449d9740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55c363acffe0 0x55c363ae4720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:32:47.513 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:47.512+0000 7f85449d9740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c363ad8db0 0x55c363af0ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:32:47.513 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:47.512+0000 7f85449d9740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f853c1b7ae0 con 0x55c363ad8db0 2024-03-31T23:32:47.513 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:47.512+0000 7f85449d9740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f853c1b5320 con 0x55c363acffe0 2024-03-31T23:32:47.513 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:47.512+0000 7f85449d9740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f853c04f4b0 con 0x55c363acf6c0 2024-03-31T23:32:47.513 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:47.512+0000 7f853affd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c363ad8db0 0x55c363af0ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:32:47.514 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:47.512+0000 7f853affd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c363ad8db0 0x55c363af0ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:50074/0 (socket says 172.21.15.46:50074) 2024-03-31T23:32:47.514 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:47.512+0000 7f853affd640 1 -- 172.21.15.46:0/881870118 learned_addr learned my addr 172.21.15.46:0/881870118 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:32:47.514 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:47.513+0000 7f853affd640 1 --2- 172.21.15.46:0/881870118 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c363acf6c0 0x55c363ad85c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:32:47.514 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:47.513+0000 7f853b7fe640 1 --2- 172.21.15.46:0/881870118 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55c363acffe0 0x55c363ae4720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:32:47.514 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:47.513+0000 7f853affd640 1 -- 172.21.15.46:0/881870118 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c363acf6c0 msgr2=0x55c363ad85c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:32:47.514 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:47.513+0000 7f853affd640 1 --2- 172.21.15.46:0/881870118 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c363acf6c0 0x55c363ad85c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:32:47.514 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:47.513+0000 7f853affd640 1 -- 172.21.15.46:0/881870118 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55c363acffe0 msgr2=0x55c363ae4720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:32:47.514 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:47.513+0000 7f853affd640 1 --2- 172.21.15.46:0/881870118 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55c363acffe0 0x55c363ae4720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:32:47.514 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:47.513+0000 7f853affd640 1 -- 172.21.15.46:0/881870118 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c363adb510 con 0x55c363ad8db0 2024-03-31T23:32:47.514 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:47.513+0000 7f853b7fe640 1 --2- 172.21.15.46:0/881870118 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55c363acffe0 0x55c363ae4720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T23:32:47.515 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:47.513+0000 7f853affd640 1 --2- 172.21.15.46:0/881870118 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c363ad8db0 0x55c363af0ac0 secure :-1 s=READY pgs=988 cs=0 l=1 rev1=1 crypto rx=0x7f8530006b50 tx=0x7f85300047d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f996c8974e3f5472 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:32:47.515 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:47.514+0000 7f85217fa640 1 -- 172.21.15.46:0/881870118 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8530016110 con 0x55c363ad8db0 2024-03-31T23:32:47.516 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:47.514+0000 7f85217fa640 1 -- 172.21.15.46:0/881870118 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f85300140e0 con 0x55c363ad8db0 2024-03-31T23:32:47.516 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:47.514+0000 7f85217fa640 1 -- 172.21.15.46:0/881870118 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8530016bd0 con 0x55c363ad8db0 2024-03-31T23:32:47.516 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:47.515+0000 7f85449d9740 1 -- 172.21.15.46:0/881870118 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c363ad8db0 msgr2=0x55c363af0ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:32:47.516 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:47.515+0000 7f85449d9740 1 --2- 172.21.15.46:0/881870118 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c363ad8db0 0x55c363af0ac0 secure :-1 s=READY pgs=988 cs=0 l=1 rev1=1 crypto rx=0x7f8530006b50 tx=0x7f85300047d0 comp rx=0 tx=0).stop 2024-03-31T23:32:47.516 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:47.515+0000 7f85449d9740 1 -- 172.21.15.46:0/881870118 shutdown_connections 2024-03-31T23:32:47.516 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:47.515+0000 7f85449d9740 1 --2- 172.21.15.46:0/881870118 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c363ad8db0 0x55c363af0ac0 unknown :-1 s=CLOSED pgs=988 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:32:47.516 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:47.515+0000 7f85449d9740 1 --2- 172.21.15.46:0/881870118 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55c363acffe0 0x55c363ae4720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:32:47.516 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:47.515+0000 7f85449d9740 1 --2- 172.21.15.46:0/881870118 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c363acf6c0 0x55c363ad85c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:32:47.516 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:47.515+0000 7f85449d9740 1 -- 172.21.15.46:0/881870118 >> 172.21.15.46:0/881870118 conn(0x55c363ac9fe0 msgr2=0x55c363aca3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:32:47.516 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:47.515+0000 7f85449d9740 1 -- 172.21.15.46:0/881870118 shutdown_connections 2024-03-31T23:32:47.516 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:47.515+0000 7f85449d9740 1 -- 172.21.15.46:0/881870118 wait complete. 2024-03-31T23:32:47.516 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:47.516+0000 7f85449d9740 1 -- start start 2024-03-31T23:32:47.519 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:47.516+0000 7f85449d9740 1 Processor -- start 2024-03-31T23:32:47.519 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:47.517+0000 7f85449d9740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55c363acf6c0 0x55c363adc640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:32:47.519 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:47.517+0000 7f85449d9740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c363acffe0 0x55c363adcb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:32:47.519 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:47.517+0000 7f85449d9740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c363ad8db0 0x55c363add0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:32:47.519 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:47.517+0000 7f85449d9740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f853c122120 con 0x55c363ad8db0 2024-03-31T23:32:47.519 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:47.517+0000 7f85449d9740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f853c1b4c50 con 0x55c363acf6c0 2024-03-31T23:32:47.520 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:47.517+0000 7f85449d9740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f853c1b3eb0 con 0x55c363acffe0 2024-03-31T23:32:47.520 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:47.517+0000 7f85449d9740 10 client.0 _ll_register_callbacks cb 0x7f853000c840 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T23:32:47.520 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:47.517+0000 7f853affd640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55c363acf6c0 0x55c363adc640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:32:47.520 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:47.517+0000 7f853affd640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55c363acf6c0 0x55c363adc640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.46:33978/0 (socket says 172.21.15.46:33978) 2024-03-31T23:32:47.520 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:47.517+0000 7f853affd640 1 -- 172.21.15.46:0/2912685154 learned_addr learned my addr 172.21.15.46:0/2912685154 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:32:47.520 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:47.517+0000 7f853b7fe640 1 --2- 172.21.15.46:0/2912685154 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c363acffe0 0x55c363adcb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:32:47.520 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:47.517+0000 7f853affd640 1 --2- 172.21.15.46:0/2912685154 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c363ad8db0 0x55c363add0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:32:47.520 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:47.517+0000 7f853affd640 1 -- 172.21.15.46:0/2912685154 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c363acffe0 msgr2=0x55c363adcb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:32:47.520 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:47.517+0000 7f853affd640 1 --2- 172.21.15.46:0/2912685154 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c363acffe0 0x55c363adcb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:32:47.520 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:47.517+0000 7f853affd640 1 -- 172.21.15.46:0/2912685154 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c363ad8db0 msgr2=0x55c363add0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:32:47.520 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:47.517+0000 7f853affd640 1 --2- 172.21.15.46:0/2912685154 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c363ad8db0 0x55c363add0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:32:47.520 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:47.517+0000 7f853affd640 1 -- 172.21.15.46:0/2912685154 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c363b4e0f0 con 0x55c363acf6c0 2024-03-31T23:32:47.520 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:47.517+0000 7f853b7fe640 1 --2- 172.21.15.46:0/2912685154 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c363acffe0 0x55c363adcb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T23:32:47.520 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:47.518+0000 7f853affd640 1 --2- 172.21.15.46:0/2912685154 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55c363acf6c0 0x55c363adc640 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f8530018870 tx=0x7f853000c460 comp rx=0 tx=0).ready entity=mon.2 client_cookie=faf6c62f52c36bcb server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:32:47.521 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:47.518+0000 7f851affd640 10 client.0 ms_handle_connect on v2:172.21.15.142:3300/0 2024-03-31T23:32:47.521 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:47.518+0000 7f851affd640 1 -- 172.21.15.46:0/2912685154 <== mon.2 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f853001e040 con 0x55c363acf6c0 2024-03-31T23:32:47.521 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:47.518+0000 7f85449d9740 1 -- 172.21.15.46:0/2912685154 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f85300140e0 con 0x55c363acf6c0 2024-03-31T23:32:47.521 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:47.518+0000 7f851affd640 1 -- 172.21.15.46:0/2912685154 <== mon.2 v2:172.21.15.142:3300/0 2 ==== osd_map(128..128 src has 1..128) ==== 9554+0+0 (secure 0 0 0) 0x7f85300139a0 con 0x55c363acf6c0 2024-03-31T23:32:47.521 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:47.518+0000 7f851affd640 1 -- 172.21.15.46:0/2912685154 <== mon.2 v2:172.21.15.142:3300/0 3 ==== mon_get_version_reply(handle=1 version=50) ==== 24+0+0 (secure 0 0 0) 0x7f8530016bd0 con 0x55c363acf6c0 2024-03-31T23:32:47.521 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:47.518+0000 7f85449d9740 10 client.27668 fetch_fsmap learned FSMap version 50 2024-03-31T23:32:47.521 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:47.518+0000 7f85449d9740 1 -- 172.21.15.46:0/2912685154 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,fsmap=50,monmap=4+}) -- 0x55c363b4f310 con 0x55c363acf6c0 2024-03-31T23:32:47.521 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:47.519+0000 7f851affd640 1 -- 172.21.15.46:0/2912685154 <== mon.2 v2:172.21.15.142:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f85300140e0 con 0x55c363acf6c0 2024-03-31T23:32:47.521 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:47.519+0000 7f851affd640 1 -- 172.21.15.46:0/2912685154 <== mon.2 v2:172.21.15.142:3300/0 5 ==== fsmap(e 50) ==== 8358+0+0 (secure 0 0 0) 0x7f8530006030 con 0x55c363acf6c0 2024-03-31T23:32:47.521 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:47.519+0000 7f85449d9740 10 client.27668 fetch_fsmap finished waiting for FSMap version 50 2024-03-31T23:32:47.521 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:47.519+0000 7f85449d9740 10 client.27668 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-31T23:32:47.521 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:47.519+0000 7f85449d9740 20 client.27668 populate_metadata read hostname 'smithi046' 2024-03-31T23:32:47.521 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:47.519+0000 7f85449d9740 1 --2- 172.21.15.46:0/2912685154 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55c363b4f980 0x55c363b51d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:32:47.521 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:47.519+0000 7f85449d9740 4 client.27668 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-31T23:32:47.521 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:47.519+0000 7f85449d9740 1 -- 172.21.15.46:0/2912685154 --> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55c363a46120 con 0x55c363b4f980 2024-03-31T23:32:47.522 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:47.519+0000 7f853b7fe640 1 --2- 172.21.15.46:0/2912685154 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55c363b4f980 0x55c363b51d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:32:47.522 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:47.520+0000 7f851affd640 10 client.27668 ms_handle_connect on v2:172.21.15.142:6838/2519143918 2024-03-31T23:32:47.522 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:47.520+0000 7f853b7fe640 1 --2- 172.21.15.46:0/2912685154 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55c363b4f980 0x55c363b51d70 crc :-1 s=READY pgs=265 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5069ea60f18ebc1f server_cookie=2d9b3fc3a083f6b in_seq=0 out_seq=0 2024-03-31T23:32:47.743 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:47.742+0000 7f851affd640 1 -- 172.21.15.46:0/2912685154 <== mds.0 v2:172.21.15.142:6838/2519143918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55c363a46120 con 0x55c363b4f980 2024-03-31T23:32:47.743 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:47.742+0000 7f851affd640 10 client.27668 handle_command_reply: tid=0 2024-03-31T23:32:47.743 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:47.742+0000 7f85449d9740 1 client.27668 shutdown 2024-03-31T23:32:47.743 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:47.742+0000 7f85449d9740 1 -- 172.21.15.46:0/2912685154 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55c363acf6c0 msgr2=0x55c363adc640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:32:47.743 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:47.742+0000 7f85449d9740 1 --2- 172.21.15.46:0/2912685154 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55c363acf6c0 0x55c363adc640 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f8530018870 tx=0x7f853000c460 comp rx=0 tx=0).stop 2024-03-31T23:32:47.743 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:47.742+0000 7f85449d9740 1 -- 172.21.15.46:0/2912685154 shutdown_connections 2024-03-31T23:32:47.743 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:47.742+0000 7f85449d9740 1 --2- 172.21.15.46:0/2912685154 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55c363b4f980 0x55c363b51d70 crc :-1 s=READY pgs=265 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:32:47.743 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:47.742+0000 7f85449d9740 1 --2- 172.21.15.46:0/2912685154 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c363ad8db0 0x55c363add0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:32:47.743 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:47.742+0000 7f85449d9740 1 --2- 172.21.15.46:0/2912685154 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c363acffe0 0x55c363adcb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:32:47.744 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:47.742+0000 7f851affd640 0 client.27668 ms_handle_reset on v2:172.21.15.142:6838/2519143918 2024-03-31T23:32:47.744 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:47.742+0000 7f85449d9740 1 --2- 172.21.15.46:0/2912685154 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55c363acf6c0 0x55c363adc640 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:32:47.744 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:47.742+0000 7f85449d9740 1 -- 172.21.15.46:0/2912685154 >> 172.21.15.46:0/2912685154 conn(0x55c363ac9fe0 msgr2=0x55c363ac86f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:32:47.744 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:47.742+0000 7f85449d9740 1 -- 172.21.15.46:0/2912685154 shutdown_connections 2024-03-31T23:32:47.744 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:47.742+0000 7f85449d9740 1 -- 172.21.15.46:0/2912685154 wait complete. 2024-03-31T23:32:47.744 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:47.742+0000 7f85449d9740 20 client.27668 trim_cache size 0 max 16384 2024-03-31T23:32:47.744 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout:{ 2024-03-31T23:32:47.744 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout: "message": "", 2024-03-31T23:32:47.744 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout: "return_code": 0 2024-03-31T23:32:47.744 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout:} 2024-03-31T23:32:47.745 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:47.743+0000 7f8541d1e640 1 -- 172.21.15.46:0/3269205777 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f85240a26b0 msgr2=0x7f85240a4b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:32:47.745 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:47.743+0000 7f8541d1e640 1 --2- 172.21.15.46:0/3269205777 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f85240a26b0 0x7f85240a4b70 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f85340cc9a0 tx=0x7f8534097ca0 comp rx=0 tx=0).stop 2024-03-31T23:32:47.745 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:47.743+0000 7f8541d1e640 1 -- 172.21.15.46:0/3269205777 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f853c116ca0 msgr2=0x7f853c1a70b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:32:47.745 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:47.743+0000 7f8541d1e640 1 --2- 172.21.15.46:0/3269205777 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f853c116ca0 0x7f853c1a70b0 secure :-1 s=READY pgs=987 cs=0 l=1 rev1=1 crypto rx=0x7f853400eb10 tx=0x7f853400c2d0 comp rx=0 tx=0).stop 2024-03-31T23:32:47.745 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:47.743+0000 7f8541d1e640 1 -- 172.21.15.46:0/3269205777 shutdown_connections 2024-03-31T23:32:47.746 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:47.743+0000 7f8541d1e640 1 --2- 172.21.15.46:0/3269205777 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f85240a26b0 0x7f85240a4b70 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:32:47.746 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:47.743+0000 7f8541d1e640 1 --2- 172.21.15.46:0/3269205777 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f853c116ca0 0x7f853c1a70b0 unknown :-1 s=CLOSED pgs=987 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:32:47.746 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:47.743+0000 7f8541d1e640 1 --2- 172.21.15.46:0/3269205777 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f853c1092c0 0x7f853c1aced0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:32:47.746 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:47.743+0000 7f8541d1e640 1 --2- 172.21.15.46:0/3269205777 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f853c1089a0 0x7f853c1ac990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:32:47.746 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:47.743+0000 7f8541d1e640 1 -- 172.21.15.46:0/3269205777 >> 172.21.15.46:0/3269205777 conn(0x7f853c06c7f0 msgr2=0x7f853c1171c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:32:47.746 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:47.743+0000 7f8541d1e640 1 -- 172.21.15.46:0/3269205777 shutdown_connections 2024-03-31T23:32:47.746 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:32:47.743+0000 7f8541d1e640 1 -- 172.21.15.46:0/3269205777 wait complete. 2024-03-31T23:32:48.297 INFO:tasks.workunit.client.0.smithi046.stdout: 10 74315 8.35 MB/sec execute 353 sec latency 192.838 ms 2024-03-31T23:32:48.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:32:48 smithi142 ceph-mon[35271]: pgmap v667: 129 pgs: 129 active+clean; 1.0 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 4.8 MiB/s wr, 263 op/s 2024-03-31T23:32:49.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:32:48 smithi162 ceph-mon[36237]: pgmap v667: 129 pgs: 129 active+clean; 1.0 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 4.8 MiB/s wr, 263 op/s 2024-03-31T23:32:49.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:32:48 smithi046 ceph-mon[32498]: pgmap v667: 129 pgs: 129 active+clean; 1.0 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 4.8 MiB/s wr, 263 op/s 2024-03-31T23:32:49.297 INFO:tasks.workunit.client.0.smithi046.stdout: 10 74467 8.36 MB/sec execute 354 sec latency 101.966 ms 2024-03-31T23:32:50.297 INFO:tasks.workunit.client.0.smithi046.stdout: 10 74647 8.37 MB/sec execute 355 sec latency 86.977 ms 2024-03-31T23:32:50.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:32:50 smithi142 ceph-mon[35271]: pgmap v668: 129 pgs: 129 active+clean; 1.0 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 7.0 MiB/s wr, 429 op/s 2024-03-31T23:32:51.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:32:50 smithi162 ceph-mon[36237]: pgmap v668: 129 pgs: 129 active+clean; 1.0 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 7.0 MiB/s wr, 429 op/s 2024-03-31T23:32:51.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:32:50 smithi046 ceph-mon[32498]: pgmap v668: 129 pgs: 129 active+clean; 1.0 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 7.0 MiB/s wr, 429 op/s 2024-03-31T23:32:51.297 INFO:tasks.workunit.client.0.smithi046.stdout: 10 74781 8.35 MB/sec execute 356 sec latency 71.460 ms 2024-03-31T23:32:52.297 INFO:tasks.workunit.client.0.smithi046.stdout: 10 74920 8.35 MB/sec execute 357 sec latency 81.409 ms 2024-03-31T23:32:52.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:32:52 smithi142 ceph-mon[35271]: pgmap v669: 129 pgs: 129 active+clean; 1.0 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 4.5 MiB/s wr, 273 op/s 2024-03-31T23:32:53.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:32:52 smithi162 ceph-mon[36237]: pgmap v669: 129 pgs: 129 active+clean; 1.0 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 4.5 MiB/s wr, 273 op/s 2024-03-31T23:32:53.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:32:52 smithi046 ceph-mon[32498]: pgmap v669: 129 pgs: 129 active+clean; 1.0 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 4.5 MiB/s wr, 273 op/s 2024-03-31T23:32:53.298 INFO:tasks.workunit.client.0.smithi046.stdout: 10 75055 8.34 MB/sec execute 358 sec latency 88.138 ms 2024-03-31T23:32:54.297 INFO:tasks.workunit.client.0.smithi046.stdout: 10 75211 8.33 MB/sec execute 359 sec latency 123.706 ms 2024-03-31T23:32:54.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:32:54 smithi142 ceph-mon[35271]: pgmap v670: 129 pgs: 129 active+clean; 1.0 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 4.5 MiB/s wr, 273 op/s 2024-03-31T23:32:55.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:32:54 smithi162 ceph-mon[36237]: pgmap v670: 129 pgs: 129 active+clean; 1.0 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 4.5 MiB/s wr, 273 op/s 2024-03-31T23:32:55.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:32:54 smithi046 ceph-mon[32498]: pgmap v670: 129 pgs: 129 active+clean; 1.0 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 4.5 MiB/s wr, 273 op/s 2024-03-31T23:32:55.298 INFO:tasks.workunit.client.0.smithi046.stdout: 10 75389 8.33 MB/sec execute 360 sec latency 98.832 ms 2024-03-31T23:32:56.298 INFO:tasks.workunit.client.0.smithi046.stdout: 10 75523 8.33 MB/sec execute 361 sec latency 158.241 ms 2024-03-31T23:32:56.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:32:56 smithi142 ceph-mon[35271]: pgmap v671: 129 pgs: 129 active+clean; 1.1 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 6.8 MiB/s wr, 384 op/s 2024-03-31T23:32:57.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:32:56 smithi162 ceph-mon[36237]: pgmap v671: 129 pgs: 129 active+clean; 1.1 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 6.8 MiB/s wr, 384 op/s 2024-03-31T23:32:57.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:32:56 smithi046 ceph-mon[32498]: pgmap v671: 129 pgs: 129 active+clean; 1.1 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 6.8 MiB/s wr, 384 op/s 2024-03-31T23:32:57.298 INFO:tasks.workunit.client.0.smithi046.stdout: 10 75688 8.33 MB/sec execute 362 sec latency 72.402 ms 2024-03-31T23:32:58.298 INFO:tasks.workunit.client.0.smithi046.stdout: 10 75850 8.33 MB/sec execute 363 sec latency 101.104 ms 2024-03-31T23:32:58.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:32:58 smithi142 ceph-mon[35271]: pgmap v672: 129 pgs: 129 active+clean; 1.1 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 4.5 MiB/s wr, 276 op/s 2024-03-31T23:32:59.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:32:58 smithi162 ceph-mon[36237]: pgmap v672: 129 pgs: 129 active+clean; 1.1 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 4.5 MiB/s wr, 276 op/s 2024-03-31T23:32:59.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:32:58 smithi046 ceph-mon[32498]: pgmap v672: 129 pgs: 129 active+clean; 1.1 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 4.5 MiB/s wr, 276 op/s 2024-03-31T23:32:59.298 INFO:tasks.workunit.client.0.smithi046.stdout: 10 75996 8.34 MB/sec execute 364 sec latency 146.560 ms 2024-03-31T23:33:00.298 INFO:tasks.workunit.client.0.smithi046.stdout: 10 76157 8.35 MB/sec execute 365 sec latency 86.189 ms 2024-03-31T23:33:00.859 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:00.859+0000 7f76d453e640 1 -- 172.21.15.46:0/1559928694 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f76cc1055f0 msgr2=0x7f76cc1059d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:33:00.859 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:00.859+0000 7f76d453e640 1 --2- 172.21.15.46:0/1559928694 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f76cc1055f0 0x7f76cc1059d0 secure :-1 s=READY pgs=989 cs=0 l=1 rev1=1 crypto rx=0x7f76c8009900 tx=0x7f76c802bc30 comp rx=0 tx=0).stop 2024-03-31T23:33:00.859 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:00.859+0000 7f76d453e640 1 -- 172.21.15.46:0/1559928694 shutdown_connections 2024-03-31T23:33:00.859 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:00.859+0000 7f76d453e640 1 --2- 172.21.15.46:0/1559928694 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f76cc101260 0x7f76cc10d480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:33:00.859 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:00.859+0000 7f76d453e640 1 --2- 172.21.15.46:0/1559928694 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f76cc1008a0 0x7f76cc100d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:33:00.860 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:00.859+0000 7f76d453e640 1 --2- 172.21.15.46:0/1559928694 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f76cc1055f0 0x7f76cc1059d0 unknown :-1 s=CLOSED pgs=989 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:33:00.860 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:00.859+0000 7f76d453e640 1 -- 172.21.15.46:0/1559928694 >> 172.21.15.46:0/1559928694 conn(0x7f76cc0774a0 msgr2=0x7f76cc0778b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:33:00.860 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:00.859+0000 7f76d453e640 1 -- 172.21.15.46:0/1559928694 shutdown_connections 2024-03-31T23:33:00.860 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:00.859+0000 7f76d453e640 1 -- 172.21.15.46:0/1559928694 wait complete. 2024-03-31T23:33:00.862 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:00.859+0000 7f76d453e640 1 Processor -- start 2024-03-31T23:33:00.862 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:00.859+0000 7f76d453e640 1 -- start start 2024-03-31T23:33:00.862 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:00.860+0000 7f76d453e640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f76cc1008a0 0x7f76cc197f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:33:00.862 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:00.860+0000 7f76d453e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f76cc101260 0x7f76cc1984b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:33:00.862 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:00.860+0000 7f76d453e640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f76cc1055f0 0x7f76cc19c7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:33:00.862 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:00.860+0000 7f76d453e640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f76cc1109d0 con 0x7f76cc101260 2024-03-31T23:33:00.862 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:00.860+0000 7f76d453e640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f76cc110850 con 0x7f76cc1055f0 2024-03-31T23:33:00.862 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:00.860+0000 7f76d453e640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f76cc110b50 con 0x7f76cc1008a0 2024-03-31T23:33:00.862 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:00.860+0000 7f76d1ad9640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f76cc101260 0x7f76cc1984b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:33:00.862 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:00.860+0000 7f76d22da640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f76cc1008a0 0x7f76cc197f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:33:00.862 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:00.860+0000 7f76d1ad9640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f76cc101260 0x7f76cc1984b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60480/0 (socket says 172.21.15.46:60480) 2024-03-31T23:33:00.863 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:00.860+0000 7f76d1ad9640 1 -- 172.21.15.46:0/256600364 learned_addr learned my addr 172.21.15.46:0/256600364 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:33:00.863 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:00.860+0000 7f76d2adb640 1 --2- 172.21.15.46:0/256600364 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f76cc1055f0 0x7f76cc19c7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:33:00.863 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:00.860+0000 7f76d1ad9640 1 -- 172.21.15.46:0/256600364 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f76cc1008a0 msgr2=0x7f76cc197f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:33:00.863 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:00.860+0000 7f76d1ad9640 1 --2- 172.21.15.46:0/256600364 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f76cc1008a0 0x7f76cc197f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:33:00.863 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:00.860+0000 7f76d1ad9640 1 -- 172.21.15.46:0/256600364 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f76cc1055f0 msgr2=0x7f76cc19c7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:33:00.863 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:00.860+0000 7f76d1ad9640 1 --2- 172.21.15.46:0/256600364 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f76cc1055f0 0x7f76cc19c7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:33:00.863 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:00.860+0000 7f76d1ad9640 1 -- 172.21.15.46:0/256600364 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f76cc19cf10 con 0x7f76cc101260 2024-03-31T23:33:00.863 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:00.860+0000 7f76d2adb640 1 --2- 172.21.15.46:0/256600364 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f76cc1055f0 0x7f76cc19c7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:33:00.863 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:00.860+0000 7f76d1ad9640 1 --2- 172.21.15.46:0/256600364 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f76cc101260 0x7f76cc1984b0 secure :-1 s=READY pgs=990 cs=0 l=1 rev1=1 crypto rx=0x7f76bc00e9f0 tx=0x7f76bc00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:33:00.863 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:00.861+0000 7f76c37fe640 1 -- 172.21.15.46:0/256600364 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76bc00cd80 con 0x7f76cc101260 2024-03-31T23:33:00.863 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:00.861+0000 7f76d453e640 1 -- 172.21.15.46:0/256600364 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f76cc19d200 con 0x7f76cc101260 2024-03-31T23:33:00.863 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:00.861+0000 7f76d453e640 1 -- 172.21.15.46:0/256600364 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f76cc1a4a90 con 0x7f76cc101260 2024-03-31T23:33:00.863 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:00.861+0000 7f76c37fe640 1 -- 172.21.15.46:0/256600364 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f76bc0108e0 con 0x7f76cc101260 2024-03-31T23:33:00.864 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:00.861+0000 7f76c37fe640 1 -- 172.21.15.46:0/256600364 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76bc018920 con 0x7f76cc101260 2024-03-31T23:33:00.866 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:00.863+0000 7f76c37fe640 1 -- 172.21.15.46:0/256600364 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 25) ==== 137451+0+0 (secure 0 0 0) 0x7f76bc010430 con 0x7f76cc101260 2024-03-31T23:33:00.866 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:00.864+0000 7f76d7203740 1 Processor -- start 2024-03-31T23:33:00.866 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:00.864+0000 7f76d7203740 1 -- start start 2024-03-31T23:33:00.866 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:00.864+0000 7f76c37fe640 1 --2- 172.21.15.46:0/256600364 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f76b80a2730 0x7f76b80a4bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:33:00.866 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:00.864+0000 7f76d7203740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fde868f6c0 0x55fde868faa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:33:00.866 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:00.865+0000 7f76d7203740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55fde868ffe0 0x55fde8698870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:33:00.866 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:00.865+0000 7f76d7203740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fde8698db0 0x55fde869b1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:33:00.866 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:00.865+0000 7f76d7203740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55fde85f28f0 con 0x55fde868f6c0 2024-03-31T23:33:00.866 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:00.865+0000 7f76d2adb640 1 --2- 172.21.15.46:0/256600364 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f76b80a2730 0x7f76b80a4bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:33:00.866 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:00.865+0000 7f76d22da640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fde868f6c0 0x55fde868faa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:33:00.866 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:00.865+0000 7f76c37fe640 1 -- 172.21.15.46:0/256600364 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(128..128 src has 1..128) ==== 9554+0+0 (secure 0 0 0) 0x7f76bc0cf090 con 0x7f76cc101260 2024-03-31T23:33:00.866 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:00.865+0000 7f76d22da640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fde868f6c0 0x55fde868faa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60486/0 (socket says 172.21.15.46:60486) 2024-03-31T23:33:00.867 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:00.865+0000 7f76d22da640 1 -- 172.21.15.46:0/4081357688 learned_addr learned my addr 172.21.15.46:0/4081357688 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:33:00.867 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:00.865+0000 7f76d7203740 1 -- 172.21.15.46:0/4081357688 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55fde859b8d0 con 0x55fde868ffe0 2024-03-31T23:33:00.867 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:00.865+0000 7f76d7203740 1 -- 172.21.15.46:0/4081357688 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55fde85ee990 con 0x55fde8698db0 2024-03-31T23:33:00.867 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:00.865+0000 7f76d2adb640 1 --2- 172.21.15.46:0/4081357688 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fde8698db0 0x55fde869b1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:33:00.867 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:00.865+0000 7f76d1ad9640 1 --2- 172.21.15.46:0/4081357688 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55fde868ffe0 0x55fde8698870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:33:00.867 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:00.865+0000 7f76d22da640 1 -- 172.21.15.46:0/4081357688 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fde8698db0 msgr2=0x55fde869b1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:33:00.867 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:00.865+0000 7f76d22da640 1 --2- 172.21.15.46:0/4081357688 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fde8698db0 0x55fde869b1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:33:00.867 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:00.865+0000 7f76d22da640 1 -- 172.21.15.46:0/4081357688 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55fde868ffe0 msgr2=0x55fde8698870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:33:00.867 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:00.865+0000 7f76d22da640 1 --2- 172.21.15.46:0/4081357688 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55fde868ffe0 0x55fde8698870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:33:00.867 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:00.865+0000 7f76d22da640 1 -- 172.21.15.46:0/4081357688 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fde8307210 con 0x55fde868f6c0 2024-03-31T23:33:00.868 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:00.865+0000 7f76d2adb640 1 --2- 172.21.15.46:0/4081357688 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fde8698db0 0x55fde869b1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T23:33:00.868 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:00.865+0000 7f76d1ad9640 1 --2- 172.21.15.46:0/4081357688 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55fde868ffe0 0x55fde8698870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:33:00.868 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:00.865+0000 7f76d2adb640 1 --2- 172.21.15.46:0/256600364 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f76b80a2730 0x7f76b80a4bf0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f76c4005fd0 tx=0x7f76c4011f80 comp rx=0 tx=0).ready entity=mgr.15654 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:33:00.868 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:00.865+0000 7f76d22da640 1 --2- 172.21.15.46:0/4081357688 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fde868f6c0 0x55fde868faa0 secure :-1 s=READY pgs=991 cs=0 l=1 rev1=1 crypto rx=0x7f76c8000c00 tx=0x7f76c8005e10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c6d3b5c44f393317 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:33:00.868 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:00.866+0000 7f76c0bfe640 1 -- 172.21.15.46:0/4081357688 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76c8045020 con 0x55fde868f6c0 2024-03-31T23:33:00.868 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:00.866+0000 7f76c0bfe640 1 -- 172.21.15.46:0/4081357688 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f76c802c780 con 0x55fde868f6c0 2024-03-31T23:33:00.868 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:00.866+0000 7f76c0bfe640 1 -- 172.21.15.46:0/4081357688 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76c8005620 con 0x55fde868f6c0 2024-03-31T23:33:00.871 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:00.866+0000 7f76d7203740 1 -- 172.21.15.46:0/4081357688 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fde868f6c0 msgr2=0x55fde868faa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:33:00.871 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:00.867+0000 7f76d7203740 1 --2- 172.21.15.46:0/4081357688 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fde868f6c0 0x55fde868faa0 secure :-1 s=READY pgs=991 cs=0 l=1 rev1=1 crypto rx=0x7f76c8000c00 tx=0x7f76c8005e10 comp rx=0 tx=0).stop 2024-03-31T23:33:00.872 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:00.867+0000 7f76d7203740 1 -- 172.21.15.46:0/4081357688 shutdown_connections 2024-03-31T23:33:00.872 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:00.867+0000 7f76d7203740 1 --2- 172.21.15.46:0/4081357688 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fde8698db0 0x55fde869b1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:33:00.872 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:00.867+0000 7f76d7203740 1 --2- 172.21.15.46:0/4081357688 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55fde868ffe0 0x55fde8698870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:33:00.872 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:00.867+0000 7f76d7203740 1 --2- 172.21.15.46:0/4081357688 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fde868f6c0 0x55fde868faa0 unknown :-1 s=CLOSED pgs=991 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:33:00.872 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:00.867+0000 7f76d7203740 1 -- 172.21.15.46:0/4081357688 >> 172.21.15.46:0/4081357688 conn(0x55fde8689fe0 msgr2=0x55fde868a3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:33:00.872 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:00.867+0000 7f76d7203740 1 -- 172.21.15.46:0/4081357688 shutdown_connections 2024-03-31T23:33:00.872 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:00.867+0000 7f76d7203740 1 -- 172.21.15.46:0/4081357688 wait complete. 2024-03-31T23:33:00.872 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:00.867+0000 7f76d7203740 1 -- start start 2024-03-31T23:33:00.872 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:00.867+0000 7f76d7203740 1 Processor -- start 2024-03-31T23:33:00.872 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:00.868+0000 7f76d7203740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fde868f6c0 0x55fde8695de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:33:00.872 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:00.868+0000 7f76d7203740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55fde868ffe0 0x55fde8696320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:33:00.872 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:00.868+0000 7f76d7203740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fde8698db0 0x55fde8696860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:33:00.872 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:00.868+0000 7f76d7203740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55fde869fd90 con 0x55fde8698db0 2024-03-31T23:33:00.872 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:00.868+0000 7f76d7203740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55fde8590f80 con 0x55fde868ffe0 2024-03-31T23:33:00.873 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:00.868+0000 7f76d7203740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55fde869ff10 con 0x55fde868f6c0 2024-03-31T23:33:00.873 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:00.868+0000 7f76d7203740 10 client.0 _ll_register_callbacks cb 0x55fde85e9290 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T23:33:00.873 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:00.868+0000 7f76d2adb640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fde8698db0 0x55fde8696860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:33:00.873 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:00.868+0000 7f76d22da640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fde868f6c0 0x55fde8695de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:33:00.873 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:00.868+0000 7f76d2adb640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fde8698db0 0x55fde8696860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60492/0 (socket says 172.21.15.46:60492) 2024-03-31T23:33:00.873 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:00.868+0000 7f76d2adb640 1 -- 172.21.15.46:0/3270096522 learned_addr learned my addr 172.21.15.46:0/3270096522 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:33:00.873 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:00.868+0000 7f76d1ad9640 1 --2- 172.21.15.46:0/3270096522 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55fde868ffe0 0x55fde8696320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:33:00.873 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:00.868+0000 7f76d2adb640 1 -- 172.21.15.46:0/3270096522 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fde868f6c0 msgr2=0x55fde8695de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:33:00.873 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:00.868+0000 7f76d2adb640 1 --2- 172.21.15.46:0/3270096522 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fde868f6c0 0x55fde8695de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:33:00.873 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:00.868+0000 7f76d2adb640 1 -- 172.21.15.46:0/3270096522 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55fde868ffe0 msgr2=0x55fde8696320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:33:00.873 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:00.868+0000 7f76d2adb640 1 --2- 172.21.15.46:0/3270096522 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55fde868ffe0 0x55fde8696320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:33:00.873 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:00.868+0000 7f76d2adb640 1 -- 172.21.15.46:0/3270096522 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fde83d3620 con 0x55fde8698db0 2024-03-31T23:33:00.873 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:00.868+0000 7f76d2adb640 1 --2- 172.21.15.46:0/3270096522 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fde8698db0 0x55fde8696860 secure :-1 s=READY pgs=992 cs=0 l=1 rev1=1 crypto rx=0x7f76c400fe80 tx=0x7f76c4004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c4f831eff14f99e8 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:33:00.873 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:00.868+0000 7f76b5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-31T23:33:00.874 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:00.868+0000 7f76b5ffb640 1 -- 172.21.15.46:0/3270096522 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76c401a040 con 0x55fde8698db0 2024-03-31T23:33:00.874 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:00.868+0000 7f76b5ffb640 1 -- 172.21.15.46:0/3270096522 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(128..128 src has 1..128) ==== 9554+0+0 (secure 0 0 0) 0x7f76c4029420 con 0x55fde8698db0 2024-03-31T23:33:00.874 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:00.869+0000 7f76d7203740 1 -- 172.21.15.46:0/3270096522 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f76c802c780 con 0x55fde8698db0 2024-03-31T23:33:00.874 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:00.869+0000 7f76b5ffb640 1 -- 172.21.15.46:0/3270096522 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=50) ==== 24+0+0 (secure 0 0 0) 0x7f76c4031050 con 0x55fde8698db0 2024-03-31T23:33:00.874 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:00.869+0000 7f76d7203740 10 client.17865 fetch_fsmap learned FSMap version 50 2024-03-31T23:33:00.874 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:00.869+0000 7f76d7203740 1 -- 172.21.15.46:0/3270096522 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=50,monmap=4+}) -- 0x55fde86abac0 con 0x55fde8698db0 2024-03-31T23:33:00.874 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:00.869+0000 7f76b5ffb640 1 -- 172.21.15.46:0/3270096522 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f76c802c780 con 0x55fde8698db0 2024-03-31T23:33:00.874 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:00.869+0000 7f76b5ffb640 1 -- 172.21.15.46:0/3270096522 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 50) ==== 8358+0+0 (secure 0 0 0) 0x7f76c40283d0 con 0x55fde8698db0 2024-03-31T23:33:00.874 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:00.869+0000 7f76d7203740 10 client.17865 fetch_fsmap finished waiting for FSMap version 50 2024-03-31T23:33:00.874 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:00.869+0000 7f76d7203740 10 client.17865 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-31T23:33:00.874 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:00.869+0000 7f76d7203740 20 client.17865 populate_metadata read hostname 'smithi046' 2024-03-31T23:33:00.874 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:00.869+0000 7f76d7203740 1 --2- 172.21.15.46:0/3270096522 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55fde86ad040 0x55fde86af470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:33:00.874 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:00.869+0000 7f76d7203740 4 client.17865 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-31T23:33:00.874 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:00.869+0000 7f76d7203740 1 -- 172.21.15.46:0/3270096522 --> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55fde8606120 con 0x55fde86ad040 2024-03-31T23:33:00.875 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:00.869+0000 7f76d22da640 1 --2- 172.21.15.46:0/3270096522 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55fde86ad040 0x55fde86af470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:33:00.875 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:00.870+0000 7f76d22da640 1 --2- 172.21.15.46:0/3270096522 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55fde86ad040 0x55fde86af470 crc :-1 s=READY pgs=266 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c8722489d9b44a57 server_cookie=991028bf83882e67 in_seq=0 out_seq=0 2024-03-31T23:33:00.875 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:00.870+0000 7f76b5ffb640 10 client.17865 ms_handle_connect on v2:172.21.15.142:6838/2519143918 2024-03-31T23:33:00.875 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:00.870+0000 7f76b5ffb640 1 -- 172.21.15.46:0/3270096522 <== mds.0 v2:172.21.15.142:6838/2519143918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55fde8606120 con 0x55fde86ad040 2024-03-31T23:33:00.875 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:00.870+0000 7f76b5ffb640 10 client.17865 handle_command_reply: tid=0 2024-03-31T23:33:00.875 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:00.870+0000 7f76d7203740 1 client.17865 shutdown 2024-03-31T23:33:00.875 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:00.871+0000 7f76d7203740 1 -- 172.21.15.46:0/3270096522 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fde8698db0 msgr2=0x55fde8696860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:33:00.875 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:00.871+0000 7f76d7203740 1 --2- 172.21.15.46:0/3270096522 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fde8698db0 0x55fde8696860 secure :-1 s=READY pgs=992 cs=0 l=1 rev1=1 crypto rx=0x7f76c400fe80 tx=0x7f76c4004ab0 comp rx=0 tx=0).stop 2024-03-31T23:33:00.875 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:00.871+0000 7f76d7203740 1 -- 172.21.15.46:0/3270096522 shutdown_connections 2024-03-31T23:33:00.875 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:00.871+0000 7f76d7203740 1 --2- 172.21.15.46:0/3270096522 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55fde86ad040 0x55fde86af470 crc :-1 s=READY pgs=266 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:33:00.875 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:00.871+0000 7f76b5ffb640 0 client.17865 ms_handle_reset on v2:172.21.15.142:6838/2519143918 2024-03-31T23:33:00.875 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:00.871+0000 7f76d7203740 1 --2- 172.21.15.46:0/3270096522 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fde8698db0 0x55fde8696860 unknown :-1 s=CLOSED pgs=992 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:33:00.875 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:00.871+0000 7f76d7203740 1 --2- 172.21.15.46:0/3270096522 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55fde868ffe0 0x55fde8696320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:33:00.875 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:00.871+0000 7f76d7203740 1 --2- 172.21.15.46:0/3270096522 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fde868f6c0 0x55fde8695de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:33:00.875 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:00.871+0000 7f76d7203740 1 -- 172.21.15.46:0/3270096522 >> 172.21.15.46:0/3270096522 conn(0x55fde8689fe0 msgr2=0x55fde868bd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:33:00.876 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:00.871+0000 7f76d7203740 1 -- 172.21.15.46:0/3270096522 shutdown_connections 2024-03-31T23:33:00.876 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:00.871+0000 7f76d7203740 1 -- 172.21.15.46:0/3270096522 wait complete. 2024-03-31T23:33:00.876 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:00.871+0000 7f76d7203740 20 client.17865 trim_cache size 0 max 16384 2024-03-31T23:33:00.894 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:00.893+0000 7f76d7203740 1 Processor -- start 2024-03-31T23:33:00.894 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:00.893+0000 7f76d7203740 1 -- start start 2024-03-31T23:33:00.894 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:00.893+0000 7f76d7203740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55fde868f6c0 0x55fde8698510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:33:00.894 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:00.893+0000 7f76d7203740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fde868ffe0 0x55fde86a8b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:33:00.894 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:00.893+0000 7f76d7203740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fde8698db0 0x55fde8690540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:33:00.894 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:00.893+0000 7f76d7203740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f76b00129e0 con 0x55fde868ffe0 2024-03-31T23:33:00.894 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:00.893+0000 7f76d7203740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f76b0010220 con 0x55fde868f6c0 2024-03-31T23:33:00.895 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:00.893+0000 7f76d7203740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f76b000e090 con 0x55fde8698db0 2024-03-31T23:33:00.895 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:00.893+0000 7f76d1ad9640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fde868ffe0 0x55fde86a8b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:33:00.895 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:00.893+0000 7f76d1ad9640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fde868ffe0 0x55fde86a8b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60506/0 (socket says 172.21.15.46:60506) 2024-03-31T23:33:00.895 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:00.893+0000 7f76d1ad9640 1 -- 172.21.15.46:0/1042515184 learned_addr learned my addr 172.21.15.46:0/1042515184 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:33:00.895 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:00.893+0000 7f76d2adb640 1 --2- 172.21.15.46:0/1042515184 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fde8698db0 0x55fde8690540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:33:00.895 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:00.893+0000 7f76d22da640 1 --2- 172.21.15.46:0/1042515184 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55fde868f6c0 0x55fde8698510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:33:00.895 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:00.894+0000 7f76d1ad9640 1 -- 172.21.15.46:0/1042515184 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fde8698db0 msgr2=0x55fde8690540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:33:00.895 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:00.894+0000 7f76d1ad9640 1 --2- 172.21.15.46:0/1042515184 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fde8698db0 0x55fde8690540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:33:00.895 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:00.894+0000 7f76d1ad9640 1 -- 172.21.15.46:0/1042515184 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55fde868f6c0 msgr2=0x55fde8698510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:33:00.895 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:00.894+0000 7f76d1ad9640 1 --2- 172.21.15.46:0/1042515184 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55fde868f6c0 0x55fde8698510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:33:00.895 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:00.894+0000 7f76d1ad9640 1 -- 172.21.15.46:0/1042515184 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fde8694330 con 0x55fde868ffe0 2024-03-31T23:33:00.895 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:00.894+0000 7f76d1ad9640 1 --2- 172.21.15.46:0/1042515184 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fde868ffe0 0x55fde86a8b60 secure :-1 s=READY pgs=993 cs=0 l=1 rev1=1 crypto rx=0x7f76bc018530 tx=0x7f76bc01a040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a4fec75011a9576 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:33:00.895 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:00.894+0000 7f76b7fff640 1 -- 172.21.15.46:0/1042515184 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76bc0cc650 con 0x55fde868ffe0 2024-03-31T23:33:00.895 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:00.894+0000 7f76b7fff640 1 -- 172.21.15.46:0/1042515184 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f76bc097b20 con 0x55fde868ffe0 2024-03-31T23:33:00.896 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:00.894+0000 7f76b7fff640 1 -- 172.21.15.46:0/1042515184 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76bc0ccb20 con 0x55fde868ffe0 2024-03-31T23:33:00.899 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:00.895+0000 7f76d7203740 1 -- 172.21.15.46:0/1042515184 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fde868ffe0 msgr2=0x55fde86a8b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:33:00.899 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:00.895+0000 7f76d7203740 1 --2- 172.21.15.46:0/1042515184 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fde868ffe0 0x55fde86a8b60 secure :-1 s=READY pgs=993 cs=0 l=1 rev1=1 crypto rx=0x7f76bc018530 tx=0x7f76bc01a040 comp rx=0 tx=0).stop 2024-03-31T23:33:00.899 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:00.895+0000 7f76d7203740 1 -- 172.21.15.46:0/1042515184 shutdown_connections 2024-03-31T23:33:00.899 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:00.895+0000 7f76d7203740 1 --2- 172.21.15.46:0/1042515184 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fde8698db0 0x55fde8690540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:33:00.899 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:00.895+0000 7f76d7203740 1 --2- 172.21.15.46:0/1042515184 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fde868ffe0 0x55fde86a8b60 unknown :-1 s=CLOSED pgs=993 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:33:00.899 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:00.895+0000 7f76d7203740 1 --2- 172.21.15.46:0/1042515184 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55fde868f6c0 0x55fde8698510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:33:00.899 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:00.895+0000 7f76d7203740 1 -- 172.21.15.46:0/1042515184 >> 172.21.15.46:0/1042515184 conn(0x55fde8689fe0 msgr2=0x55fde868a3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:33:00.899 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:00.895+0000 7f76d7203740 1 -- 172.21.15.46:0/1042515184 shutdown_connections 2024-03-31T23:33:00.899 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:00.895+0000 7f76d7203740 1 -- 172.21.15.46:0/1042515184 wait complete. 2024-03-31T23:33:00.899 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:00.896+0000 7f76d7203740 1 -- start start 2024-03-31T23:33:00.900 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:00.896+0000 7f76d7203740 1 Processor -- start 2024-03-31T23:33:00.900 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:00.896+0000 7f76d7203740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55fde868f6c0 0x55fde869be90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:33:00.900 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:00.896+0000 7f76d7203740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fde868ffe0 0x55fde869c3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:33:00.900 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:00.896+0000 7f76d7203740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fde8698db0 0x55fde86a4180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:33:00.900 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:00.896+0000 7f76d7203740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f76b000e6e0 con 0x55fde868ffe0 2024-03-31T23:33:00.900 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:00.896+0000 7f76d7203740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f76b000fb50 con 0x55fde868f6c0 2024-03-31T23:33:00.900 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:00.896+0000 7f76d7203740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f76b000edb0 con 0x55fde8698db0 2024-03-31T23:33:00.900 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:00.896+0000 7f76d7203740 10 client.0 _ll_register_callbacks cb 0x7f76c4011440 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T23:33:00.900 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:00.896+0000 7f76d1ad9640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fde868ffe0 0x55fde869c3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:33:00.900 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:00.896+0000 7f76d1ad9640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fde868ffe0 0x55fde869c3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60508/0 (socket says 172.21.15.46:60508) 2024-03-31T23:33:00.900 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:00.896+0000 7f76d1ad9640 1 -- 172.21.15.46:0/3600611027 learned_addr learned my addr 172.21.15.46:0/3600611027 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:33:00.900 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:00.896+0000 7f76d22da640 1 --2- 172.21.15.46:0/3600611027 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55fde868f6c0 0x55fde869be90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:33:00.900 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:00.896+0000 7f76d2adb640 1 --2- 172.21.15.46:0/3600611027 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fde8698db0 0x55fde86a4180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:33:00.900 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:00.896+0000 7f76d1ad9640 1 -- 172.21.15.46:0/3600611027 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fde8698db0 msgr2=0x55fde86a4180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:33:00.900 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:00.896+0000 7f76d1ad9640 1 --2- 172.21.15.46:0/3600611027 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fde8698db0 0x55fde86a4180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:33:00.901 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:00.896+0000 7f76d1ad9640 1 -- 172.21.15.46:0/3600611027 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55fde868f6c0 msgr2=0x55fde869be90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:33:00.901 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:00.896+0000 7f76d1ad9640 1 --2- 172.21.15.46:0/3600611027 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55fde868f6c0 0x55fde869be90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:33:00.901 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:00.896+0000 7f76d1ad9640 1 -- 172.21.15.46:0/3600611027 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fde869ccd0 con 0x55fde868ffe0 2024-03-31T23:33:00.901 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:00.897+0000 7f76d22da640 1 --2- 172.21.15.46:0/3600611027 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55fde868f6c0 0x55fde869be90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:33:00.901 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:00.897+0000 7f76d1ad9640 1 --2- 172.21.15.46:0/3600611027 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fde868ffe0 0x55fde869c3d0 secure :-1 s=READY pgs=994 cs=0 l=1 rev1=1 crypto rx=0x7f76bc0cc480 tx=0x7f76bc005c50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=850af3db0ff6a75 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:33:00.901 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:00.897+0000 7f76b5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-31T23:33:00.901 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:00.897+0000 7f76b5ffb640 1 -- 172.21.15.46:0/3600611027 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76bc09a900 con 0x55fde868ffe0 2024-03-31T23:33:00.901 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:00.897+0000 7f76b5ffb640 1 -- 172.21.15.46:0/3600611027 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(128..128 src has 1..128) ==== 9554+0+0 (secure 0 0 0) 0x7f76bc09aaa0 con 0x55fde868ffe0 2024-03-31T23:33:00.901 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:00.897+0000 7f76d7203740 1 -- 172.21.15.46:0/3600611027 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f76bc097b20 con 0x55fde868ffe0 2024-03-31T23:33:00.901 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:00.897+0000 7f76b5ffb640 1 -- 172.21.15.46:0/3600611027 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=50) ==== 24+0+0 (secure 0 0 0) 0x7f76bc0ccb20 con 0x55fde868ffe0 2024-03-31T23:33:00.901 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:00.897+0000 7f76d7203740 10 client.17871 fetch_fsmap learned FSMap version 50 2024-03-31T23:33:00.901 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:00.897+0000 7f76d7203740 1 -- 172.21.15.46:0/3600611027 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=50,monmap=4+}) -- 0x55fde86a5800 con 0x55fde868ffe0 2024-03-31T23:33:00.901 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:00.897+0000 7f76b5ffb640 1 -- 172.21.15.46:0/3600611027 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f76bc097b20 con 0x55fde868ffe0 2024-03-31T23:33:00.901 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:00.898+0000 7f76b5ffb640 1 -- 172.21.15.46:0/3600611027 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 50) ==== 8358+0+0 (secure 0 0 0) 0x7f76bc018bc0 con 0x55fde868ffe0 2024-03-31T23:33:00.902 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:00.898+0000 7f76d7203740 10 client.17871 fetch_fsmap finished waiting for FSMap version 50 2024-03-31T23:33:00.902 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:00.898+0000 7f76d7203740 10 client.17871 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-31T23:33:00.902 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:00.898+0000 7f76d7203740 20 client.17871 populate_metadata read hostname 'smithi046' 2024-03-31T23:33:00.902 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:00.898+0000 7f76d7203740 1 --2- 172.21.15.46:0/3600611027 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55fde86ad040 0x55fde871a000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:33:00.902 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:00.898+0000 7f76d7203740 4 client.17871 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-31T23:33:00.902 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:00.898+0000 7f76d7203740 1 -- 172.21.15.46:0/3600611027 --> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55fde85ffe30 con 0x55fde86ad040 2024-03-31T23:33:00.902 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:00.898+0000 7f76d22da640 1 --2- 172.21.15.46:0/3600611027 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55fde86ad040 0x55fde871a000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:33:00.902 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:00.898+0000 7f76b5ffb640 10 client.17871 ms_handle_connect on v2:172.21.15.142:6838/2519143918 2024-03-31T23:33:00.902 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:00.898+0000 7f76d22da640 1 --2- 172.21.15.46:0/3600611027 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55fde86ad040 0x55fde871a000 crc :-1 s=READY pgs=267 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d452c9283ce1497d server_cookie=512253a704bbfd18 in_seq=0 out_seq=0 2024-03-31T23:33:00.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:33:00 smithi142 ceph-mon[35271]: pgmap v673: 129 pgs: 129 active+clean; 1.1 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 7.0 MiB/s wr, 388 op/s 2024-03-31T23:33:01.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:33:00 smithi162 ceph-mon[36237]: pgmap v673: 129 pgs: 129 active+clean; 1.1 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 7.0 MiB/s wr, 388 op/s 2024-03-31T23:33:01.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:33:00 smithi046 ceph-mon[32498]: pgmap v673: 129 pgs: 129 active+clean; 1.1 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 7.0 MiB/s wr, 388 op/s 2024-03-31T23:33:01.158 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:01.157+0000 7f76b5ffb640 1 -- 172.21.15.46:0/3600611027 <== mds.0 v2:172.21.15.142:6838/2519143918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55fde85ffe30 con 0x55fde86ad040 2024-03-31T23:33:01.158 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:01.157+0000 7f76b5ffb640 10 client.17871 handle_command_reply: tid=0 2024-03-31T23:33:01.158 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:01.157+0000 7f76d7203740 1 client.17871 shutdown 2024-03-31T23:33:01.158 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:01.158+0000 7f76d7203740 1 -- 172.21.15.46:0/3600611027 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fde868ffe0 msgr2=0x55fde869c3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:33:01.158 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:01.158+0000 7f76d7203740 1 --2- 172.21.15.46:0/3600611027 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fde868ffe0 0x55fde869c3d0 secure :-1 s=READY pgs=994 cs=0 l=1 rev1=1 crypto rx=0x7f76bc0cc480 tx=0x7f76bc005c50 comp rx=0 tx=0).stop 2024-03-31T23:33:01.158 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:01.158+0000 7f76d7203740 1 -- 172.21.15.46:0/3600611027 shutdown_connections 2024-03-31T23:33:01.159 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:01.158+0000 7f76d7203740 1 --2- 172.21.15.46:0/3600611027 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55fde86ad040 0x55fde871a000 crc :-1 s=READY pgs=267 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:33:01.159 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:01.158+0000 7f76d7203740 1 --2- 172.21.15.46:0/3600611027 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fde8698db0 0x55fde86a4180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:33:01.159 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:01.158+0000 7f76d7203740 1 --2- 172.21.15.46:0/3600611027 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fde868ffe0 0x55fde869c3d0 unknown :-1 s=CLOSED pgs=994 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:33:01.159 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:01.158+0000 7f76d7203740 1 --2- 172.21.15.46:0/3600611027 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55fde868f6c0 0x55fde869be90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:33:01.159 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:01.158+0000 7f76d7203740 1 -- 172.21.15.46:0/3600611027 >> 172.21.15.46:0/3600611027 conn(0x55fde8689fe0 msgr2=0x55fde8690540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:33:01.159 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:01.158+0000 7f76b5ffb640 0 client.17871 ms_handle_reset on v2:172.21.15.142:6838/2519143918 2024-03-31T23:33:01.159 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:01.158+0000 7f76d7203740 1 -- 172.21.15.46:0/3600611027 shutdown_connections 2024-03-31T23:33:01.159 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:01.158+0000 7f76d7203740 1 -- 172.21.15.46:0/3600611027 wait complete. 2024-03-31T23:33:01.159 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:01.158+0000 7f76d7203740 20 client.17871 trim_cache size 0 max 16384 2024-03-31T23:33:01.159 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout:{ 2024-03-31T23:33:01.160 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout: "message": "", 2024-03-31T23:33:01.160 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout: "return_code": 0 2024-03-31T23:33:01.160 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout:} 2024-03-31T23:33:01.160 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:01.159+0000 7f76d453e640 1 -- 172.21.15.46:0/256600364 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f76b80a2730 msgr2=0x7f76b80a4bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:33:01.160 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:01.159+0000 7f76d453e640 1 --2- 172.21.15.46:0/256600364 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f76b80a2730 0x7f76b80a4bf0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f76c4005fd0 tx=0x7f76c4011f80 comp rx=0 tx=0).stop 2024-03-31T23:33:01.161 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:01.159+0000 7f76d453e640 1 -- 172.21.15.46:0/256600364 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f76cc101260 msgr2=0x7f76cc1984b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:33:01.161 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:01.159+0000 7f76d453e640 1 --2- 172.21.15.46:0/256600364 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f76cc101260 0x7f76cc1984b0 secure :-1 s=READY pgs=990 cs=0 l=1 rev1=1 crypto rx=0x7f76bc00e9f0 tx=0x7f76bc00eeb0 comp rx=0 tx=0).stop 2024-03-31T23:33:01.161 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:01.159+0000 7f76d453e640 1 -- 172.21.15.46:0/256600364 shutdown_connections 2024-03-31T23:33:01.161 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:01.159+0000 7f76d453e640 1 --2- 172.21.15.46:0/256600364 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f76b80a2730 0x7f76b80a4bf0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:33:01.161 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:01.159+0000 7f76d453e640 1 --2- 172.21.15.46:0/256600364 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f76cc1055f0 0x7f76cc19c7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:33:01.161 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:01.159+0000 7f76d453e640 1 --2- 172.21.15.46:0/256600364 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f76cc101260 0x7f76cc1984b0 unknown :-1 s=CLOSED pgs=990 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:33:01.161 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:01.159+0000 7f76d453e640 1 --2- 172.21.15.46:0/256600364 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f76cc1008a0 0x7f76cc197f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:33:01.161 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:01.159+0000 7f76d453e640 1 -- 172.21.15.46:0/256600364 >> 172.21.15.46:0/256600364 conn(0x7f76cc0774a0 msgr2=0x7f76cc069050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:33:01.161 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:01.159+0000 7f76d453e640 1 -- 172.21.15.46:0/256600364 shutdown_connections 2024-03-31T23:33:01.161 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:01.159+0000 7f76d453e640 1 -- 172.21.15.46:0/256600364 wait complete. 2024-03-31T23:33:01.298 INFO:tasks.workunit.client.0.smithi046.stdout: 10 76271 8.35 MB/sec execute 366 sec latency 305.079 ms 2024-03-31T23:33:02.298 INFO:tasks.workunit.client.0.smithi046.stdout: 10 76275 8.33 MB/sec execute 367 sec latency 1019.114 ms 2024-03-31T23:33:02.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:33:02 smithi142 ceph-mon[35271]: pgmap v674: 129 pgs: 129 active+clean; 1.1 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 4.8 MiB/s wr, 222 op/s 2024-03-31T23:33:03.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:33:02 smithi162 ceph-mon[36237]: pgmap v674: 129 pgs: 129 active+clean; 1.1 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 4.8 MiB/s wr, 222 op/s 2024-03-31T23:33:03.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:33:02 smithi046 ceph-mon[32498]: pgmap v674: 129 pgs: 129 active+clean; 1.1 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 4.8 MiB/s wr, 222 op/s 2024-03-31T23:33:03.298 INFO:tasks.workunit.client.0.smithi046.stdout: 10 76346 8.31 MB/sec execute 368 sec latency 1519.368 ms 2024-03-31T23:33:04.298 INFO:tasks.workunit.client.0.smithi046.stdout: 10 76488 8.30 MB/sec execute 369 sec latency 86.393 ms 2024-03-31T23:33:04.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:33:04 smithi142 ceph-mon[35271]: pgmap v675: 129 pgs: 129 active+clean; 1.1 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 4.8 MiB/s wr, 222 op/s 2024-03-31T23:33:05.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:33:04 smithi162 ceph-mon[36237]: pgmap v675: 129 pgs: 129 active+clean; 1.1 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 4.8 MiB/s wr, 222 op/s 2024-03-31T23:33:05.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:33:04 smithi046 ceph-mon[32498]: pgmap v675: 129 pgs: 129 active+clean; 1.1 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 4.8 MiB/s wr, 222 op/s 2024-03-31T23:33:05.298 INFO:tasks.workunit.client.0.smithi046.stdout: 10 76669 8.30 MB/sec execute 370 sec latency 74.997 ms 2024-03-31T23:33:06.298 INFO:tasks.workunit.client.0.smithi046.stdout: 10 76798 8.29 MB/sec execute 371 sec latency 111.715 ms 2024-03-31T23:33:06.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:33:06 smithi142 ceph-mon[35271]: pgmap v676: 129 pgs: 129 active+clean; 1.1 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 6.5 MiB/s wr, 376 op/s 2024-03-31T23:33:07.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:33:06 smithi162 ceph-mon[36237]: pgmap v676: 129 pgs: 129 active+clean; 1.1 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 6.5 MiB/s wr, 376 op/s 2024-03-31T23:33:07.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:33:06 smithi046 ceph-mon[32498]: pgmap v676: 129 pgs: 129 active+clean; 1.1 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 6.5 MiB/s wr, 376 op/s 2024-03-31T23:33:07.298 INFO:tasks.workunit.client.0.smithi046.stdout: 10 76950 8.29 MB/sec execute 372 sec latency 90.915 ms 2024-03-31T23:33:08.298 INFO:tasks.workunit.client.0.smithi046.stdout: 10 77058 8.28 MB/sec execute 373 sec latency 252.113 ms 2024-03-31T23:33:08.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:33:08 smithi142 ceph-mon[35271]: pgmap v677: 129 pgs: 129 active+clean; 1.1 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 4.2 MiB/s wr, 265 op/s 2024-03-31T23:33:08.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:33:08 smithi142 ceph-mon[35271]: osdmap e129: 12 total, 12 up, 12 in 2024-03-31T23:33:09.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:33:08 smithi162 ceph-mon[36237]: pgmap v677: 129 pgs: 129 active+clean; 1.1 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 4.2 MiB/s wr, 265 op/s 2024-03-31T23:33:09.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:33:08 smithi162 ceph-mon[36237]: osdmap e129: 12 total, 12 up, 12 in 2024-03-31T23:33:09.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:33:08 smithi046 ceph-mon[32498]: pgmap v677: 129 pgs: 129 active+clean; 1.1 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 4.2 MiB/s wr, 265 op/s 2024-03-31T23:33:09.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:33:08 smithi046 ceph-mon[32498]: osdmap e129: 12 total, 12 up, 12 in 2024-03-31T23:33:09.298 INFO:tasks.workunit.client.0.smithi046.stdout: 10 77221 8.28 MB/sec execute 374 sec latency 147.477 ms 2024-03-31T23:33:10.299 INFO:tasks.workunit.client.0.smithi046.stdout: 10 77412 8.29 MB/sec execute 375 sec latency 133.500 ms 2024-03-31T23:33:10.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:33:10 smithi142 ceph-mon[35271]: pgmap v679: 129 pgs: 129 active+clean; 1.1 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 4.7 MiB/s wr, 305 op/s 2024-03-31T23:33:10.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:33:10 smithi142 ceph-mon[35271]: osdmap e130: 12 total, 12 up, 12 in 2024-03-31T23:33:11.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:33:10 smithi162 ceph-mon[36237]: pgmap v679: 129 pgs: 129 active+clean; 1.1 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 4.7 MiB/s wr, 305 op/s 2024-03-31T23:33:11.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:33:10 smithi162 ceph-mon[36237]: osdmap e130: 12 total, 12 up, 12 in 2024-03-31T23:33:11.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:33:10 smithi046 ceph-mon[32498]: pgmap v679: 129 pgs: 129 active+clean; 1.1 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 4.7 MiB/s wr, 305 op/s 2024-03-31T23:33:11.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:33:10 smithi046 ceph-mon[32498]: osdmap e130: 12 total, 12 up, 12 in 2024-03-31T23:33:11.301 INFO:tasks.workunit.client.0.smithi046.stdout: 10 77597 8.30 MB/sec execute 376 sec latency 91.569 ms 2024-03-31T23:33:12.301 INFO:tasks.workunit.client.0.smithi046.stdout: 10 77742 8.31 MB/sec execute 377 sec latency 96.683 ms 2024-03-31T23:33:12.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:33:12 smithi142 ceph-mon[35271]: pgmap v681: 129 pgs: 129 active+clean; 1.1 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 5.8 MiB/s wr, 382 op/s 2024-03-31T23:33:13.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:33:12 smithi162 ceph-mon[36237]: pgmap v681: 129 pgs: 129 active+clean; 1.1 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 5.8 MiB/s wr, 382 op/s 2024-03-31T23:33:13.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:33:12 smithi046 ceph-mon[32498]: pgmap v681: 129 pgs: 129 active+clean; 1.1 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 5.8 MiB/s wr, 382 op/s 2024-03-31T23:33:13.302 INFO:tasks.workunit.client.0.smithi046.stdout: 10 77881 8.30 MB/sec execute 378 sec latency 66.658 ms 2024-03-31T23:33:14.287 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:14.286+0000 7f26f589a640 1 -- 172.21.15.46:0/3373458635 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f26f0102100 msgr2=0x7f26f010e610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:33:14.287 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:14.286+0000 7f26f589a640 1 --2- 172.21.15.46:0/3373458635 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f26f0102100 0x7f26f010e610 secure :-1 s=READY pgs=995 cs=0 l=1 rev1=1 crypto rx=0x7f26e8009cd0 tx=0x7f26e802c0f0 comp rx=0 tx=0).stop 2024-03-31T23:33:14.287 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:14.286+0000 7f26f589a640 1 -- 172.21.15.46:0/3373458635 shutdown_connections 2024-03-31T23:33:14.288 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:14.286+0000 7f26f589a640 1 --2- 172.21.15.46:0/3373458635 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f26f0102100 0x7f26f010e610 unknown :-1 s=CLOSED pgs=995 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:33:14.288 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:14.286+0000 7f26f589a640 1 --2- 172.21.15.46:0/3373458635 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f26f0101740 0x7f26f0101bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:33:14.288 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:14.286+0000 7f26f589a640 1 --2- 172.21.15.46:0/3373458635 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f26f0107750 0x7f26f0107b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:33:14.288 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:14.286+0000 7f26f589a640 1 -- 172.21.15.46:0/3373458635 >> 172.21.15.46:0/3373458635 conn(0x7f26f00fd210 msgr2=0x7f26f00ff630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:33:14.288 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:14.286+0000 7f26f589a640 1 -- 172.21.15.46:0/3373458635 shutdown_connections 2024-03-31T23:33:14.288 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:14.286+0000 7f26f589a640 1 -- 172.21.15.46:0/3373458635 wait complete. 2024-03-31T23:33:14.290 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:14.287+0000 7f26f589a640 1 Processor -- start 2024-03-31T23:33:14.290 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:14.287+0000 7f26f589a640 1 -- start start 2024-03-31T23:33:14.290 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:14.288+0000 7f26f589a640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f26f0101740 0x7f26f0197e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:33:14.290 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:14.288+0000 7f26f589a640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f26f0102100 0x7f26f01983d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:33:14.290 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:14.288+0000 7f26f589a640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f26f0107750 0x7f26f019c700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:33:14.290 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:14.288+0000 7f26f589a640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f26f0111a10 con 0x7f26f0102100 2024-03-31T23:33:14.290 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:14.288+0000 7f26f589a640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f26f0111890 con 0x7f26f0107750 2024-03-31T23:33:14.290 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:14.288+0000 7f26f589a640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f26f0111b90 con 0x7f26f0101740 2024-03-31T23:33:14.290 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:14.288+0000 7f26ee7fc640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f26f0102100 0x7f26f01983d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:33:14.290 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:14.288+0000 7f26eeffd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f26f0101740 0x7f26f0197e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:33:14.290 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:14.288+0000 7f26eeffd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f26f0101740 0x7f26f0197e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.46:48178/0 (socket says 172.21.15.46:48178) 2024-03-31T23:33:14.291 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:14.288+0000 7f26eeffd640 1 -- 172.21.15.46:0/2694531006 learned_addr learned my addr 172.21.15.46:0/2694531006 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:33:14.291 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:14.288+0000 7f26ef7fe640 1 --2- 172.21.15.46:0/2694531006 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f26f0107750 0x7f26f019c700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:33:14.291 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:14.288+0000 7f26eeffd640 1 -- 172.21.15.46:0/2694531006 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f26f0107750 msgr2=0x7f26f019c700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:33:14.291 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:14.288+0000 7f26eeffd640 1 --2- 172.21.15.46:0/2694531006 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f26f0107750 0x7f26f019c700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:33:14.291 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:14.288+0000 7f26eeffd640 1 -- 172.21.15.46:0/2694531006 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f26f0102100 msgr2=0x7f26f01983d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:33:14.291 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:14.288+0000 7f26eeffd640 1 --2- 172.21.15.46:0/2694531006 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f26f0102100 0x7f26f01983d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:33:14.291 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:14.288+0000 7f26eeffd640 1 -- 172.21.15.46:0/2694531006 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f26f019ce30 con 0x7f26f0101740 2024-03-31T23:33:14.291 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:14.288+0000 7f26ef7fe640 1 --2- 172.21.15.46:0/2694531006 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f26f0107750 0x7f26f019c700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:33:14.291 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:14.288+0000 7f26ee7fc640 1 --2- 172.21.15.46:0/2694531006 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f26f0102100 0x7f26f01983d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:33:14.291 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:14.288+0000 7f26eeffd640 1 --2- 172.21.15.46:0/2694531006 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f26f0101740 0x7f26f0197e90 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f26e000e9f0 tx=0x7f26e000eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:33:14.291 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:14.289+0000 7f26f4898640 1 -- 172.21.15.46:0/2694531006 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26e000cd80 con 0x7f26f0101740 2024-03-31T23:33:14.291 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:14.289+0000 7f26f589a640 1 -- 172.21.15.46:0/2694531006 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f26f019d120 con 0x7f26f0101740 2024-03-31T23:33:14.291 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:14.289+0000 7f26f4898640 1 -- 172.21.15.46:0/2694531006 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f26e00108e0 con 0x7f26f0101740 2024-03-31T23:33:14.291 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:14.289+0000 7f26f4898640 1 -- 172.21.15.46:0/2694531006 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26e0018920 con 0x7f26f0101740 2024-03-31T23:33:14.292 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:14.289+0000 7f26f589a640 1 -- 172.21.15.46:0/2694531006 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f26f0073980 con 0x7f26f0101740 2024-03-31T23:33:14.293 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:14.292+0000 7f26f4898640 1 -- 172.21.15.46:0/2694531006 <== mon.1 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 25) ==== 137451+0+0 (secure 0 0 0) 0x7f26e0010a90 con 0x7f26f0101740 2024-03-31T23:33:14.293 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:14.292+0000 7f26f847e740 1 Processor -- start 2024-03-31T23:33:14.293 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:14.292+0000 7f26f847e740 1 -- start start 2024-03-31T23:33:14.295 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:14.292+0000 7f26f847e740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56324914b6c0 0x56324914baa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:33:14.295 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:14.292+0000 7f26f847e740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56324914bfe0 0x563249154870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:33:14.295 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:14.292+0000 7f26f847e740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563249154db0 0x5632491571a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:33:14.295 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:14.292+0000 7f26f4898640 1 --2- 172.21.15.46:0/2694531006 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f26d80a26b0 0x7f26d80a4b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:33:14.295 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:14.293+0000 7f26f847e740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x5632490ae8f0 con 0x563249154db0 2024-03-31T23:33:14.295 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:14.293+0000 7f26f847e740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x5632490578d0 con 0x56324914bfe0 2024-03-31T23:33:14.295 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:14.293+0000 7f26f847e740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5632490aa990 con 0x56324914b6c0 2024-03-31T23:33:14.295 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:14.293+0000 7f26ee7fc640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563249154db0 0x5632491571a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:33:14.295 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:14.293+0000 7f26ef7fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56324914b6c0 0x56324914baa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:33:14.295 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:14.293+0000 7f26ee7fc640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563249154db0 0x5632491571a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:51056/0 (socket says 172.21.15.46:51056) 2024-03-31T23:33:14.296 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:14.293+0000 7f26f4898640 1 -- 172.21.15.46:0/2694531006 <== mon.1 v2:172.21.15.162:3300/0 5 ==== osd_map(130..130 src has 1..130) ==== 9554+0+0 (secure 0 0 0) 0x7f26e00cf220 con 0x7f26f0101740 2024-03-31T23:33:14.296 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:14.293+0000 7f26ee7fc640 1 -- 172.21.15.46:0/649742812 learned_addr learned my addr 172.21.15.46:0/649742812 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:33:14.296 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:14.293+0000 7f26eeffd640 1 --2- 172.21.15.46:0/649742812 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56324914bfe0 0x563249154870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:33:14.296 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:14.293+0000 7f26ef7fe640 1 --2- 172.21.15.46:0/2694531006 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f26d80a26b0 0x7f26d80a4b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:33:14.296 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:14.293+0000 7f26ee7fc640 1 -- 172.21.15.46:0/649742812 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56324914b6c0 msgr2=0x56324914baa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:33:14.296 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:14.293+0000 7f26ee7fc640 1 --2- 172.21.15.46:0/649742812 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56324914b6c0 0x56324914baa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:33:14.296 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:14.293+0000 7f26ee7fc640 1 -- 172.21.15.46:0/649742812 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56324914bfe0 msgr2=0x563249154870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:33:14.296 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:14.293+0000 7f26ee7fc640 1 --2- 172.21.15.46:0/649742812 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56324914bfe0 0x563249154870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:33:14.296 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:14.293+0000 7f26ee7fc640 1 -- 172.21.15.46:0/649742812 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563248dc3210 con 0x563249154db0 2024-03-31T23:33:14.296 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:14.293+0000 7f26eeffd640 1 --2- 172.21.15.46:0/649742812 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56324914bfe0 0x563249154870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:33:14.296 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:14.293+0000 7f26ef7fe640 1 --2- 172.21.15.46:0/2694531006 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f26d80a26b0 0x7f26d80a4b70 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f26e802c630 tx=0x7f26e8005d80 comp rx=0 tx=0).ready entity=mgr.15654 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:33:14.296 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:14.293+0000 7f26ee7fc640 1 --2- 172.21.15.46:0/649742812 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563249154db0 0x5632491571a0 secure :-1 s=READY pgs=996 cs=0 l=1 rev1=1 crypto rx=0x7f26e4011470 tx=0x7f26e4011930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1414439eac75fd90 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:33:14.296 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:14.294+0000 7f26d5ffb640 1 -- 172.21.15.46:0/649742812 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26e401e070 con 0x563249154db0 2024-03-31T23:33:14.297 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:14.294+0000 7f26d5ffb640 1 -- 172.21.15.46:0/649742812 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f26e40035f0 con 0x563249154db0 2024-03-31T23:33:14.297 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:14.294+0000 7f26d5ffb640 1 -- 172.21.15.46:0/649742812 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26e400f7a0 con 0x563249154db0 2024-03-31T23:33:14.300 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:14.295+0000 7f26f847e740 1 -- 172.21.15.46:0/649742812 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563249154db0 msgr2=0x5632491571a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:33:14.300 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:14.295+0000 7f26f847e740 1 --2- 172.21.15.46:0/649742812 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563249154db0 0x5632491571a0 secure :-1 s=READY pgs=996 cs=0 l=1 rev1=1 crypto rx=0x7f26e4011470 tx=0x7f26e4011930 comp rx=0 tx=0).stop 2024-03-31T23:33:14.300 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:14.295+0000 7f26f847e740 1 -- 172.21.15.46:0/649742812 shutdown_connections 2024-03-31T23:33:14.300 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:14.295+0000 7f26f847e740 1 --2- 172.21.15.46:0/649742812 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563249154db0 0x5632491571a0 unknown :-1 s=CLOSED pgs=996 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:33:14.300 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:14.295+0000 7f26f847e740 1 --2- 172.21.15.46:0/649742812 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56324914bfe0 0x563249154870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:33:14.300 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:14.295+0000 7f26f847e740 1 --2- 172.21.15.46:0/649742812 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56324914b6c0 0x56324914baa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:33:14.300 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:14.295+0000 7f26f847e740 1 -- 172.21.15.46:0/649742812 >> 172.21.15.46:0/649742812 conn(0x563249145fe0 msgr2=0x5632491463c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:33:14.300 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:14.295+0000 7f26f847e740 1 -- 172.21.15.46:0/649742812 shutdown_connections 2024-03-31T23:33:14.300 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:14.295+0000 7f26f847e740 1 -- 172.21.15.46:0/649742812 wait complete. 2024-03-31T23:33:14.301 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:14.296+0000 7f26f847e740 1 -- start start 2024-03-31T23:33:14.301 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:14.296+0000 7f26f847e740 1 Processor -- start 2024-03-31T23:33:14.301 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:14.296+0000 7f26f847e740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56324914b6c0 0x563249151e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:33:14.301 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:14.296+0000 7f26f847e740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56324914bfe0 0x563249152370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:33:14.301 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:14.296+0000 7f26f847e740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563249154db0 0x5632491528b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:33:14.301 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:14.297+0000 7f26ee7fc640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56324914bfe0 0x563249152370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:33:14.301 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:14.297+0000 7f26f847e740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x5632491579e0 con 0x56324914bfe0 2024-03-31T23:33:14.301 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:14.297+0000 7f26eeffd640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56324914b6c0 0x563249151e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:33:14.301 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:14.297+0000 7f26ee7fc640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56324914bfe0 0x563249152370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:51070/0 (socket says 172.21.15.46:51070) 2024-03-31T23:33:14.301 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:14.297+0000 7f26ee7fc640 1 -- 172.21.15.46:0/2682451207 learned_addr learned my addr 172.21.15.46:0/2682451207 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:33:14.301 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:14.297+0000 7f26f847e740 1 -- 172.21.15.46:0/2682451207 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x56324904cf80 con 0x56324914b6c0 2024-03-31T23:33:14.301 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:14.297+0000 7f26f847e740 1 -- 172.21.15.46:0/2682451207 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x563249157b60 con 0x563249154db0 2024-03-31T23:33:14.301 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:14.297+0000 7f26ef7fe640 1 --2- 172.21.15.46:0/2682451207 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563249154db0 0x5632491528b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:33:14.302 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:14.297+0000 7f26f847e740 10 client.0 _ll_register_callbacks cb 0x5632490a5290 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T23:33:14.302 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:14.297+0000 7f26ee7fc640 1 -- 172.21.15.46:0/2682451207 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563249154db0 msgr2=0x5632491528b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:33:14.302 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:14.297+0000 7f26ee7fc640 1 --2- 172.21.15.46:0/2682451207 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563249154db0 0x5632491528b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:33:14.302 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:14.297+0000 7f26ee7fc640 1 -- 172.21.15.46:0/2682451207 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56324914b6c0 msgr2=0x563249151e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:33:14.302 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:14.297+0000 7f26ee7fc640 1 --2- 172.21.15.46:0/2682451207 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56324914b6c0 0x563249151e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:33:14.302 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:14.297+0000 7f26ee7fc640 1 -- 172.21.15.46:0/2682451207 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563248e8f620 con 0x56324914bfe0 2024-03-31T23:33:14.302 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:14.297+0000 7f26ee7fc640 1 --2- 172.21.15.46:0/2682451207 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56324914bfe0 0x563249152370 secure :-1 s=READY pgs=997 cs=0 l=1 rev1=1 crypto rx=0x7f26e4011e30 tx=0x7f26e40039d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a8c7644f51fa2626 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:33:14.302 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:14.297+0000 7f26ceffd640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-31T23:33:14.302 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:14.297+0000 7f26ceffd640 1 -- 172.21.15.46:0/2682451207 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26e401e040 con 0x56324914bfe0 2024-03-31T23:33:14.302 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:14.297+0000 7f26ceffd640 1 -- 172.21.15.46:0/2682451207 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(130..130 src has 1..130) ==== 9554+0+0 (secure 0 0 0) 0x7f26e4003bc0 con 0x56324914bfe0 2024-03-31T23:33:14.302 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:14.297+0000 7f26f847e740 1 -- 172.21.15.46:0/2682451207 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f26e40035f0 con 0x56324914bfe0 2024-03-31T23:33:14.302 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:14.298+0000 7f26ceffd640 1 -- 172.21.15.46:0/2682451207 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=50) ==== 24+0+0 (secure 0 0 0) 0x7f26e4032050 con 0x56324914bfe0 2024-03-31T23:33:14.302 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:14.298+0000 7f26f847e740 10 client.17883 fetch_fsmap learned FSMap version 50 2024-03-31T23:33:14.302 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:14.298+0000 7f26f847e740 1 -- 172.21.15.46:0/2682451207 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=50,monmap=4+}) -- 0x563249163940 con 0x56324914bfe0 2024-03-31T23:33:14.303 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:14.298+0000 7f26ceffd640 1 -- 172.21.15.46:0/2682451207 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f26e40035f0 con 0x56324914bfe0 2024-03-31T23:33:14.303 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:14.298+0000 7f26ceffd640 1 -- 172.21.15.46:0/2682451207 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 50) ==== 8358+0+0 (secure 0 0 0) 0x7f26e4021030 con 0x56324914bfe0 2024-03-31T23:33:14.303 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:14.298+0000 7f26f847e740 10 client.17883 fetch_fsmap finished waiting for FSMap version 50 2024-03-31T23:33:14.303 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:14.298+0000 7f26f847e740 10 client.17883 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-31T23:33:14.303 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:14.298+0000 7f26f847e740 20 client.17883 populate_metadata read hostname 'smithi046' 2024-03-31T23:33:14.303 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:14.298+0000 7f26f847e740 1 --2- 172.21.15.46:0/2682451207 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x563249164cf0 0x5632491670e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:33:14.303 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:14.298+0000 7f26f847e740 4 client.17883 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-31T23:33:14.303 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:14.298+0000 7f26f847e740 1 -- 172.21.15.46:0/2682451207 --> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5632490c2120 con 0x563249164cf0 2024-03-31T23:33:14.303 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:14.299+0000 7f26eeffd640 1 --2- 172.21.15.46:0/2682451207 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x563249164cf0 0x5632491670e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:33:14.303 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:14.299+0000 7f26ceffd640 10 client.17883 ms_handle_connect on v2:172.21.15.142:6838/2519143918 2024-03-31T23:33:14.303 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:14.299+0000 7f26eeffd640 1 --2- 172.21.15.46:0/2682451207 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x563249164cf0 0x5632491670e0 crc :-1 s=READY pgs=268 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b9809b726ebb5211 server_cookie=c16ba19cfcd36546 in_seq=0 out_seq=0 2024-03-31T23:33:14.304 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:14.300+0000 7f26ceffd640 1 -- 172.21.15.46:0/2682451207 <== mds.0 v2:172.21.15.142:6838/2519143918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5632490c2120 con 0x563249164cf0 2024-03-31T23:33:14.304 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:14.300+0000 7f26ceffd640 10 client.17883 handle_command_reply: tid=0 2024-03-31T23:33:14.304 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:14.300+0000 7f26f847e740 1 client.17883 shutdown 2024-03-31T23:33:14.305 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:14.300+0000 7f26f847e740 1 -- 172.21.15.46:0/2682451207 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56324914bfe0 msgr2=0x563249152370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:33:14.305 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:14.300+0000 7f26f847e740 1 --2- 172.21.15.46:0/2682451207 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56324914bfe0 0x563249152370 secure :-1 s=READY pgs=997 cs=0 l=1 rev1=1 crypto rx=0x7f26e4011e30 tx=0x7f26e40039d0 comp rx=0 tx=0).stop 2024-03-31T23:33:14.305 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:14.300+0000 7f26f847e740 1 -- 172.21.15.46:0/2682451207 shutdown_connections 2024-03-31T23:33:14.305 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:14.300+0000 7f26f847e740 1 --2- 172.21.15.46:0/2682451207 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x563249164cf0 0x5632491670e0 crc :-1 s=READY pgs=268 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:33:14.305 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:14.300+0000 7f26f847e740 1 --2- 172.21.15.46:0/2682451207 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563249154db0 0x5632491528b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:33:14.305 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:14.300+0000 7f26f847e740 1 --2- 172.21.15.46:0/2682451207 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56324914bfe0 0x563249152370 unknown :-1 s=CLOSED pgs=997 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:33:14.305 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:14.300+0000 7f26f847e740 1 --2- 172.21.15.46:0/2682451207 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56324914b6c0 0x563249151e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:33:14.305 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:14.300+0000 7f26ceffd640 0 client.17883 ms_handle_reset on v2:172.21.15.142:6838/2519143918 2024-03-31T23:33:14.305 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:14.300+0000 7f26f847e740 1 -- 172.21.15.46:0/2682451207 >> 172.21.15.46:0/2682451207 conn(0x563249145fe0 msgr2=0x563249147d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:33:14.305 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:14.300+0000 7f26f847e740 1 -- 172.21.15.46:0/2682451207 shutdown_connections 2024-03-31T23:33:14.305 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:14.300+0000 7f26f847e740 1 -- 172.21.15.46:0/2682451207 wait complete. 2024-03-31T23:33:14.305 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:14.301+0000 7f26f847e740 20 client.17883 trim_cache size 0 max 16384 2024-03-31T23:33:14.305 INFO:tasks.workunit.client.0.smithi046.stdout: 10 78033 8.31 MB/sec execute 379 sec latency 94.269 ms 2024-03-31T23:33:14.328 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:14.326+0000 7f26f847e740 1 Processor -- start 2024-03-31T23:33:14.328 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:14.327+0000 7f26f847e740 1 -- start start 2024-03-31T23:33:14.328 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:14.327+0000 7f26f847e740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56324914b6c0 0x563249154560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:33:14.329 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:14.327+0000 7f26f847e740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56324914bfe0 0x563249160780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:33:14.329 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:14.327+0000 7f26f847e740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563249154db0 0x56324916c8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:33:14.329 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:14.327+0000 7f26f847e740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f26f01b3860 con 0x56324914bfe0 2024-03-31T23:33:14.329 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:14.327+0000 7f26f847e740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f26f01b10a0 con 0x56324914b6c0 2024-03-31T23:33:14.329 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:14.327+0000 7f26f847e740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f26f01aef10 con 0x563249154db0 2024-03-31T23:33:14.329 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:14.327+0000 7f26ee7fc640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56324914bfe0 0x563249160780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:33:14.329 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:14.327+0000 7f26ef7fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563249154db0 0x56324916c8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:33:14.329 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:14.327+0000 7f26ef7fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563249154db0 0x56324916c8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.46:48200/0 (socket says 172.21.15.46:48200) 2024-03-31T23:33:14.329 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:14.327+0000 7f26ee7fc640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56324914bfe0 0x563249160780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:51086/0 (socket says 172.21.15.46:51086) 2024-03-31T23:33:14.329 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:14.327+0000 7f26ef7fe640 1 -- 172.21.15.46:0/533549869 learned_addr learned my addr 172.21.15.46:0/533549869 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:33:14.329 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:14.327+0000 7f26eeffd640 1 --2- 172.21.15.46:0/533549869 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56324914b6c0 0x563249154560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:33:14.329 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:14.328+0000 7f26ee7fc640 1 -- 172.21.15.46:0/533549869 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563249154db0 msgr2=0x56324916c8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:33:14.329 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:14.328+0000 7f26ee7fc640 1 --2- 172.21.15.46:0/533549869 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563249154db0 0x56324916c8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:33:14.330 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:14.328+0000 7f26ee7fc640 1 -- 172.21.15.46:0/533549869 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56324914b6c0 msgr2=0x563249154560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:33:14.330 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:14.328+0000 7f26ee7fc640 1 --2- 172.21.15.46:0/533549869 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56324914b6c0 0x563249154560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:33:14.330 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:14.328+0000 7f26ee7fc640 1 -- 172.21.15.46:0/533549869 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5632491503c0 con 0x56324914bfe0 2024-03-31T23:33:14.333 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:14.328+0000 7f26ee7fc640 1 --2- 172.21.15.46:0/533549869 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56324914bfe0 0x563249160780 secure :-1 s=READY pgs=998 cs=0 l=1 rev1=1 crypto rx=0x7f26e4004ab0 tx=0x7f26e4011930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb2f6ea281568361 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:33:14.333 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:14.328+0000 7f26d57fa640 1 -- 172.21.15.46:0/533549869 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26e401e040 con 0x56324914bfe0 2024-03-31T23:33:14.333 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:14.328+0000 7f26d57fa640 1 -- 172.21.15.46:0/533549869 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f26e4016070 con 0x56324914bfe0 2024-03-31T23:33:14.333 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:14.328+0000 7f26d57fa640 1 -- 172.21.15.46:0/533549869 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26e4011290 con 0x56324914bfe0 2024-03-31T23:33:14.333 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:14.329+0000 7f26f847e740 1 -- 172.21.15.46:0/533549869 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56324914bfe0 msgr2=0x563249160780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:33:14.333 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:14.329+0000 7f26f847e740 1 --2- 172.21.15.46:0/533549869 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56324914bfe0 0x563249160780 secure :-1 s=READY pgs=998 cs=0 l=1 rev1=1 crypto rx=0x7f26e4004ab0 tx=0x7f26e4011930 comp rx=0 tx=0).stop 2024-03-31T23:33:14.333 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:14.329+0000 7f26f847e740 1 -- 172.21.15.46:0/533549869 shutdown_connections 2024-03-31T23:33:14.334 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:14.329+0000 7f26f847e740 1 --2- 172.21.15.46:0/533549869 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563249154db0 0x56324916c8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:33:14.334 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:14.329+0000 7f26f847e740 1 --2- 172.21.15.46:0/533549869 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56324914bfe0 0x563249160780 unknown :-1 s=CLOSED pgs=998 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:33:14.334 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:14.329+0000 7f26f847e740 1 --2- 172.21.15.46:0/533549869 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56324914b6c0 0x563249154560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:33:14.334 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:14.329+0000 7f26f847e740 1 -- 172.21.15.46:0/533549869 >> 172.21.15.46:0/533549869 conn(0x563249145fe0 msgr2=0x563249166d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:33:14.334 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:14.329+0000 7f26f847e740 1 -- 172.21.15.46:0/533549869 shutdown_connections 2024-03-31T23:33:14.334 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:14.329+0000 7f26f847e740 1 -- 172.21.15.46:0/533549869 wait complete. 2024-03-31T23:33:14.334 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:14.330+0000 7f26f847e740 1 -- start start 2024-03-31T23:33:14.334 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:14.330+0000 7f26f847e740 1 Processor -- start 2024-03-31T23:33:14.334 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:14.330+0000 7f26f847e740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56324914b6c0 0x56324914f570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:33:14.334 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:14.330+0000 7f26f847e740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56324914bfe0 0x56324914fab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:33:14.334 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:14.330+0000 7f26f847e740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x563249154db0 0x56324914e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:33:14.334 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:14.330+0000 7f26f847e740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f26f01af560 con 0x56324914b6c0 2024-03-31T23:33:14.334 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:14.330+0000 7f26f847e740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f26f01b09d0 con 0x563249154db0 2024-03-31T23:33:14.335 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:14.330+0000 7f26f847e740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f26f01afc30 con 0x56324914bfe0 2024-03-31T23:33:14.335 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:14.330+0000 7f26f847e740 10 client.0 _ll_register_callbacks cb 0x7f26e401c070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T23:33:14.335 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:14.330+0000 7f26eeffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56324914b6c0 0x56324914f570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:33:14.335 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:14.330+0000 7f26eeffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56324914b6c0 0x56324914f570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:51102/0 (socket says 172.21.15.46:51102) 2024-03-31T23:33:14.335 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:14.330+0000 7f26eeffd640 1 -- 172.21.15.46:0/2398330246 learned_addr learned my addr 172.21.15.46:0/2398330246 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:33:14.335 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:14.330+0000 7f26ef7fe640 1 --2- 172.21.15.46:0/2398330246 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x563249154db0 0x56324914e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:33:14.335 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:14.331+0000 7f26ee7fc640 1 --2- 172.21.15.46:0/2398330246 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56324914bfe0 0x56324914fab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:33:14.335 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:14.331+0000 7f26eeffd640 1 -- 172.21.15.46:0/2398330246 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56324914bfe0 msgr2=0x56324914fab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:33:14.335 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:14.331+0000 7f26eeffd640 1 --2- 172.21.15.46:0/2398330246 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56324914bfe0 0x56324914fab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:33:14.335 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:14.331+0000 7f26eeffd640 1 -- 172.21.15.46:0/2398330246 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x563249154db0 msgr2=0x56324914e0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:33:14.335 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:14.331+0000 7f26eeffd640 1 --2- 172.21.15.46:0/2398330246 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x563249154db0 0x56324914e0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:33:14.335 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:14.331+0000 7f26eeffd640 1 -- 172.21.15.46:0/2398330246 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56324914e620 con 0x56324914b6c0 2024-03-31T23:33:14.335 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:14.331+0000 7f26eeffd640 1 --2- 172.21.15.46:0/2398330246 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56324914b6c0 0x56324914f570 secure :-1 s=READY pgs=999 cs=0 l=1 rev1=1 crypto rx=0x7f26e0097fb0 tx=0x7f26e001a040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a32f1ec24865064 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:33:14.335 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:14.331+0000 7f26ceffd640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-31T23:33:14.335 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:14.331+0000 7f26ceffd640 1 -- 172.21.15.46:0/2398330246 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26e00b9060 con 0x56324914b6c0 2024-03-31T23:33:14.335 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:14.331+0000 7f26ceffd640 1 -- 172.21.15.46:0/2398330246 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(130..130 src has 1..130) ==== 9554+0+0 (secure 0 0 0) 0x7f26e00b4270 con 0x56324914b6c0 2024-03-31T23:33:14.336 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:14.331+0000 7f26f847e740 1 -- 172.21.15.46:0/2398330246 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f26e4016070 con 0x56324914b6c0 2024-03-31T23:33:14.336 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:14.331+0000 7f26ceffd640 1 -- 172.21.15.46:0/2398330246 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=50) ==== 24+0+0 (secure 0 0 0) 0x7f26e00b3030 con 0x56324914b6c0 2024-03-31T23:33:14.336 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:14.331+0000 7f26f847e740 10 client.17889 fetch_fsmap learned FSMap version 50 2024-03-31T23:33:14.336 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:14.331+0000 7f26f847e740 1 -- 172.21.15.46:0/2398330246 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=50,monmap=4+}) -- 0x5632491d2ec0 con 0x56324914b6c0 2024-03-31T23:33:14.336 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:14.332+0000 7f26ceffd640 1 -- 172.21.15.46:0/2398330246 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f26e4016070 con 0x56324914b6c0 2024-03-31T23:33:14.336 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:14.332+0000 7f26ceffd640 1 -- 172.21.15.46:0/2398330246 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 50) ==== 8358+0+0 (secure 0 0 0) 0x7f26e00cc780 con 0x56324914b6c0 2024-03-31T23:33:14.336 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:14.332+0000 7f26f847e740 10 client.17889 fetch_fsmap finished waiting for FSMap version 50 2024-03-31T23:33:14.336 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:14.332+0000 7f26f847e740 10 client.17889 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-31T23:33:14.336 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:14.332+0000 7f26f847e740 20 client.17889 populate_metadata read hostname 'smithi046' 2024-03-31T23:33:14.336 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:14.332+0000 7f26f847e740 1 --2- 172.21.15.46:0/2398330246 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x5632491d3570 0x5632491d59b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:33:14.336 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:14.332+0000 7f26f847e740 4 client.17889 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-31T23:33:14.336 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:14.332+0000 7f26f847e740 1 -- 172.21.15.46:0/2398330246 --> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5632490bbe30 con 0x5632491d3570 2024-03-31T23:33:14.336 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:14.332+0000 7f26ee7fc640 1 --2- 172.21.15.46:0/2398330246 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x5632491d3570 0x5632491d59b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:33:14.337 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:14.333+0000 7f26ceffd640 10 client.17889 ms_handle_connect on v2:172.21.15.142:6838/2519143918 2024-03-31T23:33:14.337 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:14.333+0000 7f26ee7fc640 1 --2- 172.21.15.46:0/2398330246 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x5632491d3570 0x5632491d59b0 crc :-1 s=READY pgs=269 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f7b3578cdfa845f9 server_cookie=61c9209b2fd46a5a in_seq=0 out_seq=0 2024-03-31T23:33:14.520 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:14.519+0000 7f26ceffd640 1 -- 172.21.15.46:0/2398330246 <== mds.0 v2:172.21.15.142:6838/2519143918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5632490bbe30 con 0x5632491d3570 2024-03-31T23:33:14.521 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:14.519+0000 7f26ceffd640 10 client.17889 handle_command_reply: tid=0 2024-03-31T23:33:14.521 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:14.519+0000 7f26f847e740 1 client.17889 shutdown 2024-03-31T23:33:14.521 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:14.519+0000 7f26f847e740 1 -- 172.21.15.46:0/2398330246 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56324914b6c0 msgr2=0x56324914f570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:33:14.521 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:14.519+0000 7f26f847e740 1 --2- 172.21.15.46:0/2398330246 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56324914b6c0 0x56324914f570 secure :-1 s=READY pgs=999 cs=0 l=1 rev1=1 crypto rx=0x7f26e0097fb0 tx=0x7f26e001a040 comp rx=0 tx=0).stop 2024-03-31T23:33:14.521 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:14.519+0000 7f26f847e740 1 -- 172.21.15.46:0/2398330246 shutdown_connections 2024-03-31T23:33:14.521 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:14.519+0000 7f26f847e740 1 --2- 172.21.15.46:0/2398330246 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x5632491d3570 0x5632491d59b0 crc :-1 s=READY pgs=269 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:33:14.521 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:14.519+0000 7f26ceffd640 0 client.17889 ms_handle_reset on v2:172.21.15.142:6838/2519143918 2024-03-31T23:33:14.521 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:14.519+0000 7f26f847e740 1 --2- 172.21.15.46:0/2398330246 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x563249154db0 0x56324914e0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:33:14.521 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:14.519+0000 7f26f847e740 1 --2- 172.21.15.46:0/2398330246 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56324914bfe0 0x56324914fab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:33:14.521 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:14.519+0000 7f26f847e740 1 --2- 172.21.15.46:0/2398330246 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56324914b6c0 0x56324914f570 unknown :-1 s=CLOSED pgs=999 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:33:14.521 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:14.519+0000 7f26f847e740 1 -- 172.21.15.46:0/2398330246 >> 172.21.15.46:0/2398330246 conn(0x563249145fe0 msgr2=0x563249150700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:33:14.521 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:14.519+0000 7f26f847e740 1 -- 172.21.15.46:0/2398330246 shutdown_connections 2024-03-31T23:33:14.522 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:14.519+0000 7f26f847e740 1 -- 172.21.15.46:0/2398330246 wait complete. 2024-03-31T23:33:14.522 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:14.519+0000 7f26f847e740 20 client.17889 trim_cache size 0 max 16384 2024-03-31T23:33:14.522 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout:{ 2024-03-31T23:33:14.522 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout: "message": "", 2024-03-31T23:33:14.522 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout: "return_code": 0 2024-03-31T23:33:14.522 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout:} 2024-03-31T23:33:14.523 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:14.520+0000 7f26f589a640 1 -- 172.21.15.46:0/2694531006 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f26d80a26b0 msgr2=0x7f26d80a4b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:33:14.523 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:14.520+0000 7f26f589a640 1 --2- 172.21.15.46:0/2694531006 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f26d80a26b0 0x7f26d80a4b70 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f26e802c630 tx=0x7f26e8005d80 comp rx=0 tx=0).stop 2024-03-31T23:33:14.523 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:14.520+0000 7f26f589a640 1 -- 172.21.15.46:0/2694531006 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f26f0101740 msgr2=0x7f26f0197e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:33:14.523 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:14.520+0000 7f26f589a640 1 --2- 172.21.15.46:0/2694531006 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f26f0101740 0x7f26f0197e90 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f26e000e9f0 tx=0x7f26e000eeb0 comp rx=0 tx=0).stop 2024-03-31T23:33:14.523 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:14.520+0000 7f26f589a640 1 -- 172.21.15.46:0/2694531006 shutdown_connections 2024-03-31T23:33:14.523 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:14.520+0000 7f26f589a640 1 --2- 172.21.15.46:0/2694531006 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f26d80a26b0 0x7f26d80a4b70 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:33:14.523 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:14.520+0000 7f26f589a640 1 --2- 172.21.15.46:0/2694531006 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f26f0107750 0x7f26f019c700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:33:14.523 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:14.520+0000 7f26f589a640 1 --2- 172.21.15.46:0/2694531006 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f26f0102100 0x7f26f01983d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:33:14.523 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:14.521+0000 7f26f589a640 1 --2- 172.21.15.46:0/2694531006 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f26f0101740 0x7f26f0197e90 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:33:14.523 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:14.521+0000 7f26f589a640 1 -- 172.21.15.46:0/2694531006 >> 172.21.15.46:0/2694531006 conn(0x7f26f00fd210 msgr2=0x7f26f00fd7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:33:14.523 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:14.521+0000 7f26f589a640 1 -- 172.21.15.46:0/2694531006 shutdown_connections 2024-03-31T23:33:14.524 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:14.521+0000 7f26f589a640 1 -- 172.21.15.46:0/2694531006 wait complete. 2024-03-31T23:33:14.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:33:14 smithi142 ceph-mon[35271]: pgmap v682: 129 pgs: 129 active+clean; 1.1 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.3 MiB/s wr, 152 op/s 2024-03-31T23:33:15.008 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:33:14 smithi162 ceph-mon[36237]: pgmap v682: 129 pgs: 129 active+clean; 1.1 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.3 MiB/s wr, 152 op/s 2024-03-31T23:33:15.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:33:14 smithi046 ceph-mon[32498]: pgmap v682: 129 pgs: 129 active+clean; 1.1 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.3 MiB/s wr, 152 op/s 2024-03-31T23:33:15.296 INFO:journalctl@ceph.mds.i.smithi162.stdout:Mar 31 23:33:15 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mds-i[67378]: ignoring --setuser ceph since I am not root 2024-03-31T23:33:15.296 INFO:journalctl@ceph.mds.i.smithi162.stdout:Mar 31 23:33:15 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mds-i[67378]: ignoring --setgroup ceph since I am not root 2024-03-31T23:33:15.296 INFO:journalctl@ceph.mds.i.smithi162.stdout:Mar 31 23:33:15 smithi162 ceph-mds[67382]: -- 172.21.15.162:0/2604502379 <== mon.1 v2:172.21.15.162:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55e96a3c1a40 con 0x55e96a394800 2024-03-31T23:33:15.296 INFO:journalctl@ceph.mds.i.smithi162.stdout:Mar 31 23:33:15 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mds-i[67378]: starting mds.i at 2024-03-31T23:33:15.302 INFO:tasks.workunit.client.0.smithi046.stdout: 10 78162 8.30 MB/sec execute 380 sec latency 184.375 ms 2024-03-31T23:33:15.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:33:15 smithi142 ceph-mon[35271]: daemon mds.i restarted 2024-03-31T23:33:15.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:33:15 smithi142 ceph-mon[35271]: osdmap e131: 12 total, 12 up, 12 in 2024-03-31T23:33:15.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:33:15 smithi142 ceph-mon[35271]: mds.? [v2:172.21.15.162:6838/2570930841,v1:172.21.15.162:6839/2570930841] up:boot 2024-03-31T23:33:15.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:33:15 smithi142 ceph-mon[35271]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-03-31T23:33:15.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:33:15 smithi142 ceph-mon[35271]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-31T23:33:16.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:33:15 smithi162 ceph-mon[36237]: daemon mds.i restarted 2024-03-31T23:33:16.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:33:15 smithi162 ceph-mon[36237]: osdmap e131: 12 total, 12 up, 12 in 2024-03-31T23:33:16.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:33:15 smithi162 ceph-mon[36237]: mds.? [v2:172.21.15.162:6838/2570930841,v1:172.21.15.162:6839/2570930841] up:boot 2024-03-31T23:33:16.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:33:15 smithi162 ceph-mon[36237]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-03-31T23:33:16.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:33:15 smithi162 ceph-mon[36237]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-31T23:33:16.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:33:15 smithi046 ceph-mon[32498]: daemon mds.i restarted 2024-03-31T23:33:16.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:33:15 smithi046 ceph-mon[32498]: osdmap e131: 12 total, 12 up, 12 in 2024-03-31T23:33:16.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:33:15 smithi046 ceph-mon[32498]: mds.? [v2:172.21.15.162:6838/2570930841,v1:172.21.15.162:6839/2570930841] up:boot 2024-03-31T23:33:16.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:33:15 smithi046 ceph-mon[32498]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-03-31T23:33:16.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:33:15 smithi046 ceph-mon[32498]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-31T23:33:16.302 INFO:tasks.workunit.client.0.smithi046.stdout: 10 78320 8.30 MB/sec execute 381 sec latency 82.253 ms 2024-03-31T23:33:17.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:33:16 smithi162 ceph-mon[36237]: pgmap v683: 129 pgs: 129 active+clean; 1.1 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 7.0 MiB/s wr, 324 op/s 2024-03-31T23:33:17.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:33:16 smithi046 ceph-mon[32498]: pgmap v683: 129 pgs: 129 active+clean; 1.1 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 7.0 MiB/s wr, 324 op/s 2024-03-31T23:33:17.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:33:16 smithi142 ceph-mon[35271]: pgmap v683: 129 pgs: 129 active+clean; 1.1 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 7.0 MiB/s wr, 324 op/s 2024-03-31T23:33:17.302 INFO:tasks.workunit.client.0.smithi046.stdout: 10 78492 8.30 MB/sec execute 382 sec latency 76.917 ms 2024-03-31T23:33:18.303 INFO:tasks.workunit.client.0.smithi046.stdout: 10 78647 8.29 MB/sec execute 383 sec latency 116.410 ms 2024-03-31T23:33:19.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:33:18 smithi162 ceph-mon[36237]: pgmap v685: 129 pgs: 129 active+clean; 1.1 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.7 MiB/s wr, 171 op/s 2024-03-31T23:33:19.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:33:18 smithi046 ceph-mon[32498]: pgmap v685: 129 pgs: 129 active+clean; 1.1 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.7 MiB/s wr, 171 op/s 2024-03-31T23:33:19.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:33:18 smithi142 ceph-mon[35271]: pgmap v685: 129 pgs: 129 active+clean; 1.1 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.7 MiB/s wr, 171 op/s 2024-03-31T23:33:19.302 INFO:tasks.workunit.client.0.smithi046.stdout: 10 78771 8.29 MB/sec execute 384 sec latency 85.551 ms 2024-03-31T23:33:20.303 INFO:tasks.workunit.client.0.smithi046.stdout: 10 78913 8.29 MB/sec execute 385 sec latency 139.641 ms 2024-03-31T23:33:21.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:33:20 smithi162 ceph-mon[36237]: pgmap v686: 129 pgs: 129 active+clean; 1.1 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 7.1 MiB/s wr, 395 op/s 2024-03-31T23:33:21.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:33:20 smithi046 ceph-mon[32498]: pgmap v686: 129 pgs: 129 active+clean; 1.1 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 7.1 MiB/s wr, 395 op/s 2024-03-31T23:33:21.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:33:20 smithi142 ceph-mon[35271]: pgmap v686: 129 pgs: 129 active+clean; 1.1 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 7.1 MiB/s wr, 395 op/s 2024-03-31T23:33:21.303 INFO:tasks.workunit.client.0.smithi046.stdout: 10 79100 8.30 MB/sec execute 386 sec latency 73.088 ms 2024-03-31T23:33:22.303 INFO:tasks.workunit.client.0.smithi046.stdout: 10 79267 8.31 MB/sec execute 387 sec latency 87.784 ms 2024-03-31T23:33:22.994 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:33:22 smithi142 ceph-mon[35271]: pgmap v687: 129 pgs: 129 active+clean; 1.1 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 5.9 MiB/s wr, 330 op/s 2024-03-31T23:33:23.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:33:22 smithi162 ceph-mon[36237]: pgmap v687: 129 pgs: 129 active+clean; 1.1 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 5.9 MiB/s wr, 330 op/s 2024-03-31T23:33:23.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:33:22 smithi046 ceph-mon[32498]: pgmap v687: 129 pgs: 129 active+clean; 1.1 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 5.9 MiB/s wr, 330 op/s 2024-03-31T23:33:23.303 INFO:tasks.workunit.client.0.smithi046.stdout: 10 79413 8.30 MB/sec execute 388 sec latency 77.231 ms 2024-03-31T23:33:24.303 INFO:tasks.workunit.client.0.smithi046.stdout: 10 79559 8.30 MB/sec execute 389 sec latency 69.562 ms 2024-03-31T23:33:25.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:33:24 smithi162 ceph-mon[36237]: pgmap v688: 129 pgs: 129 active+clean; 1.1 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 5.9 MiB/s wr, 330 op/s 2024-03-31T23:33:25.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:33:24 smithi046 ceph-mon[32498]: pgmap v688: 129 pgs: 129 active+clean; 1.1 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 5.9 MiB/s wr, 330 op/s 2024-03-31T23:33:25.194 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:33:24 smithi142 ceph-mon[35271]: pgmap v688: 129 pgs: 129 active+clean; 1.1 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 5.9 MiB/s wr, 330 op/s 2024-03-31T23:33:25.303 INFO:tasks.workunit.client.0.smithi046.stdout: 10 79697 8.29 MB/sec execute 390 sec latency 82.189 ms 2024-03-31T23:33:26.303 INFO:tasks.workunit.client.0.smithi046.stdout: 10 79858 8.30 MB/sec execute 391 sec latency 85.628 ms 2024-03-31T23:33:27.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:33:26 smithi162 ceph-mon[36237]: pgmap v689: 129 pgs: 129 active+clean; 1.1 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 6.0 MiB/s wr, 329 op/s 2024-03-31T23:33:27.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:33:26 smithi046 ceph-mon[32498]: pgmap v689: 129 pgs: 129 active+clean; 1.1 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 6.0 MiB/s wr, 329 op/s 2024-03-31T23:33:27.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:33:26 smithi142 ceph-mon[35271]: pgmap v689: 129 pgs: 129 active+clean; 1.1 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 6.0 MiB/s wr, 329 op/s 2024-03-31T23:33:27.303 INFO:tasks.workunit.client.0.smithi046.stdout: 10 79986 8.29 MB/sec execute 392 sec latency 97.176 ms 2024-03-31T23:33:27.645 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:27.641+0000 7f4bc872c640 1 -- 172.21.15.46:0/3436225889 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4bc0100610 msgr2=0x7f4bc010e5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:33:27.645 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:27.641+0000 7f4bc872c640 1 --2- 172.21.15.46:0/3436225889 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4bc0100610 0x7f4bc010e5f0 secure :-1 s=READY pgs=1000 cs=0 l=1 rev1=1 crypto rx=0x7f4bb8009cd0 tx=0x7f4bb802c0f0 comp rx=0 tx=0).stop 2024-03-31T23:33:27.645 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:27.641+0000 7f4bc872c640 1 -- 172.21.15.46:0/3436225889 shutdown_connections 2024-03-31T23:33:27.645 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:27.641+0000 7f4bc872c640 1 --2- 172.21.15.46:0/3436225889 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4bc0100610 0x7f4bc010e5f0 unknown :-1 s=CLOSED pgs=1000 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:33:27.645 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:27.641+0000 7f4bc872c640 1 --2- 172.21.15.46:0/3436225889 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4bc00ffc50 0x7f4bc01000d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:33:27.645 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:27.641+0000 7f4bc872c640 1 --2- 172.21.15.46:0/3436225889 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4bc0107730 0x7f4bc0107b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:33:27.645 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:27.641+0000 7f4bc872c640 1 -- 172.21.15.46:0/3436225889 >> 172.21.15.46:0/3436225889 conn(0x7f4bc0076b90 msgr2=0x7f4bc00fdf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:33:27.645 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:27.641+0000 7f4bc872c640 1 -- 172.21.15.46:0/3436225889 shutdown_connections 2024-03-31T23:33:27.645 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:27.641+0000 7f4bc872c640 1 -- 172.21.15.46:0/3436225889 wait complete. 2024-03-31T23:33:27.645 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:27.642+0000 7f4bc872c640 1 Processor -- start 2024-03-31T23:33:27.646 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:27.642+0000 7f4bc872c640 1 -- start start 2024-03-31T23:33:27.646 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:27.642+0000 7f4bc872c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4bc00ffc50 0x7f4bc0197e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:33:27.646 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:27.642+0000 7f4bc872c640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4bc0100610 0x7f4bc0198370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:33:27.646 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:27.642+0000 7f4bc872c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4bc0107730 0x7f4bc019c6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:33:27.646 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:27.642+0000 7f4bc872c640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f4bc01119f0 con 0x7f4bc00ffc50 2024-03-31T23:33:27.646 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:27.642+0000 7f4bc872c640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f4bc0111870 con 0x7f4bc0100610 2024-03-31T23:33:27.646 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:27.642+0000 7f4bc872c640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f4bc0111b70 con 0x7f4bc0107730 2024-03-31T23:33:27.646 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:27.642+0000 7f4bc64c8640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4bc00ffc50 0x7f4bc0197e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:33:27.646 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:27.642+0000 7f4bc64c8640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4bc00ffc50 0x7f4bc0197e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60584/0 (socket says 172.21.15.46:60584) 2024-03-31T23:33:27.646 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:27.642+0000 7f4bc64c8640 1 -- 172.21.15.46:0/3610136790 learned_addr learned my addr 172.21.15.46:0/3610136790 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:33:27.646 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:27.643+0000 7f4bc5cc7640 1 --2- 172.21.15.46:0/3610136790 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4bc0100610 0x7f4bc0198370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:33:27.646 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:27.643+0000 7f4bc6cc9640 1 --2- 172.21.15.46:0/3610136790 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4bc0107730 0x7f4bc019c6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:33:27.646 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:27.643+0000 7f4bc64c8640 1 -- 172.21.15.46:0/3610136790 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4bc0107730 msgr2=0x7f4bc019c6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:33:27.646 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:27.643+0000 7f4bc64c8640 1 --2- 172.21.15.46:0/3610136790 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4bc0107730 0x7f4bc019c6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:33:27.647 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:27.643+0000 7f4bc64c8640 1 -- 172.21.15.46:0/3610136790 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4bc0100610 msgr2=0x7f4bc0198370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:33:27.647 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:27.643+0000 7f4bc64c8640 1 --2- 172.21.15.46:0/3610136790 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4bc0100610 0x7f4bc0198370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:33:27.647 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:27.643+0000 7f4bc64c8640 1 -- 172.21.15.46:0/3610136790 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4bc019ce20 con 0x7f4bc00ffc50 2024-03-31T23:33:27.647 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:27.643+0000 7f4bc5cc7640 1 --2- 172.21.15.46:0/3610136790 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4bc0100610 0x7f4bc0198370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:33:27.647 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:27.643+0000 7f4bc64c8640 1 --2- 172.21.15.46:0/3610136790 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4bc00ffc50 0x7f4bc0197e30 secure :-1 s=READY pgs=1001 cs=0 l=1 rev1=1 crypto rx=0x7f4bbc00e9f0 tx=0x7f4bbc00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:33:27.647 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:27.643+0000 7f4bb77fe640 1 -- 172.21.15.46:0/3610136790 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bbc00cd80 con 0x7f4bc00ffc50 2024-03-31T23:33:27.647 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:27.644+0000 7f4bb77fe640 1 -- 172.21.15.46:0/3610136790 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4bbc0108e0 con 0x7f4bc00ffc50 2024-03-31T23:33:27.647 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:27.644+0000 7f4bc872c640 1 -- 172.21.15.46:0/3610136790 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4bc019d110 con 0x7f4bc00ffc50 2024-03-31T23:33:27.647 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:27.644+0000 7f4bc872c640 1 -- 172.21.15.46:0/3610136790 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4bc01a49a0 con 0x7f4bc00ffc50 2024-03-31T23:33:27.647 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:27.644+0000 7f4bb77fe640 1 -- 172.21.15.46:0/3610136790 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bbc018920 con 0x7f4bc00ffc50 2024-03-31T23:33:27.648 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:27.647+0000 7f4bb77fe640 1 -- 172.21.15.46:0/3610136790 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 25) ==== 137451+0+0 (secure 0 0 0) 0x7f4bbc018ac0 con 0x7f4bc00ffc50 2024-03-31T23:33:27.648 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:27.647+0000 7f4bcb3ef740 1 Processor -- start 2024-03-31T23:33:27.648 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:27.648+0000 7f4bcb3ef740 1 -- start start 2024-03-31T23:33:27.649 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:27.648+0000 7f4bcb3ef740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5644612316c0 0x564461231aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:33:27.649 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:27.648+0000 7f4bcb3ef740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564461231fe0 0x56446123a870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:33:27.649 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:27.648+0000 7f4bcb3ef740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56446123adb0 0x56446123d1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:33:27.649 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:27.648+0000 7f4bcb3ef740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x5644611948f0 con 0x564461231fe0 2024-03-31T23:33:27.649 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:27.648+0000 7f4bcb3ef740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x56446113d8d0 con 0x56446123adb0 2024-03-31T23:33:27.649 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:27.648+0000 7f4bcb3ef740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x564461190990 con 0x5644612316c0 2024-03-31T23:33:27.649 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:27.648+0000 7f4bc6cc9640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5644612316c0 0x564461231aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:33:27.649 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:27.648+0000 7f4bc64c8640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564461231fe0 0x56446123a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:33:27.649 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:27.648+0000 7f4bc6cc9640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5644612316c0 0x564461231aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.46:52960/0 (socket says 172.21.15.46:52960) 2024-03-31T23:33:27.650 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:27.648+0000 7f4bc6cc9640 1 -- 172.21.15.46:0/3082641438 learned_addr learned my addr 172.21.15.46:0/3082641438 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:33:27.650 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:27.648+0000 7f4bb77fe640 1 --2- 172.21.15.46:0/3610136790 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f4bac0a26b0 0x7f4bac0a4b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:33:27.653 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:27.648+0000 7f4bc5cc7640 1 --2- 172.21.15.46:0/3082641438 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56446123adb0 0x56446123d1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:33:27.653 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:27.648+0000 7f4bc6cc9640 1 -- 172.21.15.46:0/3082641438 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56446123adb0 msgr2=0x56446123d1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:33:27.653 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:27.648+0000 7f4bc6cc9640 1 --2- 172.21.15.46:0/3082641438 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56446123adb0 0x56446123d1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:33:27.653 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:27.648+0000 7f4bc6cc9640 1 -- 172.21.15.46:0/3082641438 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564461231fe0 msgr2=0x56446123a870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:33:27.653 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:27.648+0000 7f4bc6cc9640 1 --2- 172.21.15.46:0/3082641438 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564461231fe0 0x56446123a870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:33:27.653 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:27.649+0000 7f4bc6cc9640 1 -- 172.21.15.46:0/3082641438 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564460ea9210 con 0x5644612316c0 2024-03-31T23:33:27.654 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:27.649+0000 7f4bc64c8640 1 --2- 172.21.15.46:0/3082641438 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564461231fe0 0x56446123a870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:33:27.654 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:27.649+0000 7f4bc6cc9640 1 --2- 172.21.15.46:0/3610136790 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f4bac0a26b0 0x7f4bac0a4b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:33:27.654 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:27.649+0000 7f4bb77fe640 1 -- 172.21.15.46:0/3610136790 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(131..131 src has 1..131) ==== 9466+0+0 (secure 0 0 0) 0x7f4bbc0cf0c0 con 0x7f4bc00ffc50 2024-03-31T23:33:27.654 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:27.649+0000 7f4bc6cc9640 1 --2- 172.21.15.46:0/3082641438 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5644612316c0 0x564461231aa0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f4bb8004af0 tx=0x7f4bb8032390 comp rx=0 tx=0).ready entity=mon.1 client_cookie=149c5116f2ae8c71 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:33:27.654 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:27.649+0000 7f4bc6cc9640 1 --2- 172.21.15.46:0/3610136790 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f4bac0a26b0 0x7f4bac0a4b70 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f4bb804c000 tx=0x7f4bb8045000 comp rx=0 tx=0).ready entity=mgr.15654 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:33:27.654 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:27.649+0000 7f4bb4ff9640 1 -- 172.21.15.46:0/3082641438 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bb8039040 con 0x5644612316c0 2024-03-31T23:33:27.654 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:27.649+0000 7f4bb4ff9640 1 -- 172.21.15.46:0/3082641438 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4bb804a8f0 con 0x5644612316c0 2024-03-31T23:33:27.654 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:27.649+0000 7f4bb4ff9640 1 -- 172.21.15.46:0/3082641438 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bb80347e0 con 0x5644612316c0 2024-03-31T23:33:27.654 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:27.650+0000 7f4bcb3ef740 1 -- 172.21.15.46:0/3082641438 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5644612316c0 msgr2=0x564461231aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:33:27.654 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:27.650+0000 7f4bcb3ef740 1 --2- 172.21.15.46:0/3082641438 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5644612316c0 0x564461231aa0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f4bb8004af0 tx=0x7f4bb8032390 comp rx=0 tx=0).stop 2024-03-31T23:33:27.654 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:27.650+0000 7f4bcb3ef740 1 -- 172.21.15.46:0/3082641438 shutdown_connections 2024-03-31T23:33:27.654 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:27.650+0000 7f4bcb3ef740 1 --2- 172.21.15.46:0/3082641438 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56446123adb0 0x56446123d1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:33:27.654 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:27.650+0000 7f4bcb3ef740 1 --2- 172.21.15.46:0/3082641438 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564461231fe0 0x56446123a870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:33:27.655 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:27.650+0000 7f4bcb3ef740 1 --2- 172.21.15.46:0/3082641438 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5644612316c0 0x564461231aa0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:33:27.655 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:27.650+0000 7f4bcb3ef740 1 -- 172.21.15.46:0/3082641438 >> 172.21.15.46:0/3082641438 conn(0x56446122bfe0 msgr2=0x56446122c3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:33:27.655 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:27.650+0000 7f4bcb3ef740 1 -- 172.21.15.46:0/3082641438 shutdown_connections 2024-03-31T23:33:27.655 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:27.650+0000 7f4bcb3ef740 1 -- 172.21.15.46:0/3082641438 wait complete. 2024-03-31T23:33:27.655 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:27.651+0000 7f4bcb3ef740 1 -- start start 2024-03-31T23:33:27.655 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:27.651+0000 7f4bcb3ef740 1 Processor -- start 2024-03-31T23:33:27.657 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:27.651+0000 7f4bcb3ef740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5644612316c0 0x564461237e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:33:27.657 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:27.651+0000 7f4bcb3ef740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x564461231fe0 0x564461238360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:33:27.657 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:27.651+0000 7f4bcb3ef740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56446123adb0 0x5644612388a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:33:27.658 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:27.651+0000 7f4bcb3ef740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x564461241df0 con 0x5644612316c0 2024-03-31T23:33:27.658 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:27.651+0000 7f4bcb3ef740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x564461132f80 con 0x564461231fe0 2024-03-31T23:33:27.658 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:27.651+0000 7f4bcb3ef740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x564461241f70 con 0x56446123adb0 2024-03-31T23:33:27.658 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:27.651+0000 7f4bcb3ef740 10 client.0 _ll_register_callbacks cb 0x56446118b290 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T23:33:27.658 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:27.651+0000 7f4bc64c8640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5644612316c0 0x564461237e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:33:27.658 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:27.651+0000 7f4bc64c8640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5644612316c0 0x564461237e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60598/0 (socket says 172.21.15.46:60598) 2024-03-31T23:33:27.658 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:27.651+0000 7f4bc64c8640 1 -- 172.21.15.46:0/1855486359 learned_addr learned my addr 172.21.15.46:0/1855486359 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:33:27.658 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:27.651+0000 7f4bc5cc7640 1 --2- 172.21.15.46:0/1855486359 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x564461231fe0 0x564461238360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:33:27.658 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:27.651+0000 7f4bc6cc9640 1 --2- 172.21.15.46:0/1855486359 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56446123adb0 0x5644612388a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:33:27.658 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:27.652+0000 7f4bc64c8640 1 -- 172.21.15.46:0/1855486359 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56446123adb0 msgr2=0x5644612388a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:33:27.658 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:27.652+0000 7f4bc64c8640 1 --2- 172.21.15.46:0/1855486359 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56446123adb0 0x5644612388a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:33:27.658 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:27.652+0000 7f4bc64c8640 1 -- 172.21.15.46:0/1855486359 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x564461231fe0 msgr2=0x564461238360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:33:27.658 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:27.652+0000 7f4bc64c8640 1 --2- 172.21.15.46:0/1855486359 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x564461231fe0 0x564461238360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:33:27.659 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:27.652+0000 7f4bc64c8640 1 -- 172.21.15.46:0/1855486359 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564460f75620 con 0x5644612316c0 2024-03-31T23:33:27.659 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:27.652+0000 7f4bc64c8640 1 --2- 172.21.15.46:0/1855486359 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5644612316c0 0x564461237e20 secure :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 crypto rx=0x7f4bbc00e9c0 tx=0x7f4bbc009310 comp rx=0 tx=0).ready entity=mon.0 client_cookie=849f096f37cedaaa server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:33:27.659 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:27.652+0000 7f4ba9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-31T23:33:27.659 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:27.652+0000 7f4ba9ffb640 1 -- 172.21.15.46:0/1855486359 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bbc0bb2b0 con 0x5644612316c0 2024-03-31T23:33:27.659 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:27.652+0000 7f4ba9ffb640 1 -- 172.21.15.46:0/1855486359 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(131..131 src has 1..131) ==== 9466+0+0 (secure 0 0 0) 0x7f4bbc09f150 con 0x5644612316c0 2024-03-31T23:33:27.659 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:27.652+0000 7f4bcb3ef740 1 -- 172.21.15.46:0/1855486359 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4bb804a8f0 con 0x5644612316c0 2024-03-31T23:33:27.659 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:27.652+0000 7f4ba9ffb640 1 -- 172.21.15.46:0/1855486359 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=51) ==== 24+0+0 (secure 0 0 0) 0x7f4bbc09f5e0 con 0x5644612316c0 2024-03-31T23:33:27.659 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:27.652+0000 7f4bcb3ef740 10 client.17907 fetch_fsmap learned FSMap version 51 2024-03-31T23:33:27.659 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:27.652+0000 7f4bcb3ef740 1 -- 172.21.15.46:0/1855486359 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=51,monmap=4+}) -- 0x56446124dd80 con 0x5644612316c0 2024-03-31T23:33:27.659 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:27.653+0000 7f4ba9ffb640 1 -- 172.21.15.46:0/1855486359 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4bb804a8f0 con 0x5644612316c0 2024-03-31T23:33:27.659 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:27.653+0000 7f4ba9ffb640 1 -- 172.21.15.46:0/1855486359 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 51) ==== 8358+0+0 (secure 0 0 0) 0x7f4bbc0d1110 con 0x5644612316c0 2024-03-31T23:33:27.659 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:27.653+0000 7f4bcb3ef740 10 client.17907 fetch_fsmap finished waiting for FSMap version 51 2024-03-31T23:33:27.659 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:27.653+0000 7f4bcb3ef740 10 client.17907 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-31T23:33:27.659 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:27.653+0000 7f4bcb3ef740 20 client.17907 populate_metadata read hostname 'smithi046' 2024-03-31T23:33:27.660 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:27.653+0000 7f4bcb3ef740 1 --2- 172.21.15.46:0/1855486359 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x56446124f290 0x5644612516e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:33:27.660 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:27.653+0000 7f4bcb3ef740 4 client.17907 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-31T23:33:27.660 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:27.653+0000 7f4bcb3ef740 1 -- 172.21.15.46:0/1855486359 --> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5644611a8120 con 0x56446124f290 2024-03-31T23:33:27.660 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:27.654+0000 7f4bc5cc7640 1 --2- 172.21.15.46:0/1855486359 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x56446124f290 0x5644612516e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:33:27.660 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:27.654+0000 7f4bc5cc7640 1 --2- 172.21.15.46:0/1855486359 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x56446124f290 0x5644612516e0 crc :-1 s=READY pgs=270 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f54b0e47500484ea server_cookie=2fe0218f87f6d663 in_seq=0 out_seq=0 2024-03-31T23:33:27.660 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:27.654+0000 7f4ba9ffb640 10 client.17907 ms_handle_connect on v2:172.21.15.142:6838/2519143918 2024-03-31T23:33:27.661 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:27.657+0000 7f4ba9ffb640 1 -- 172.21.15.46:0/1855486359 <== mds.0 v2:172.21.15.142:6838/2519143918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5644611a8120 con 0x56446124f290 2024-03-31T23:33:27.661 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:27.657+0000 7f4ba9ffb640 10 client.17907 handle_command_reply: tid=0 2024-03-31T23:33:27.661 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:27.657+0000 7f4bcb3ef740 1 client.17907 shutdown 2024-03-31T23:33:27.661 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:27.657+0000 7f4bcb3ef740 1 -- 172.21.15.46:0/1855486359 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5644612316c0 msgr2=0x564461237e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:33:27.661 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:27.657+0000 7f4bcb3ef740 1 --2- 172.21.15.46:0/1855486359 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5644612316c0 0x564461237e20 secure :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 crypto rx=0x7f4bbc00e9c0 tx=0x7f4bbc009310 comp rx=0 tx=0).stop 2024-03-31T23:33:27.661 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:27.657+0000 7f4bcb3ef740 1 -- 172.21.15.46:0/1855486359 shutdown_connections 2024-03-31T23:33:27.661 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:27.657+0000 7f4bcb3ef740 1 --2- 172.21.15.46:0/1855486359 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x56446124f290 0x5644612516e0 crc :-1 s=READY pgs=270 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:33:27.661 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:27.657+0000 7f4bcb3ef740 1 --2- 172.21.15.46:0/1855486359 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56446123adb0 0x5644612388a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:33:27.661 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:27.657+0000 7f4bcb3ef740 1 --2- 172.21.15.46:0/1855486359 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x564461231fe0 0x564461238360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:33:27.661 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:27.657+0000 7f4bcb3ef740 1 --2- 172.21.15.46:0/1855486359 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5644612316c0 0x564461237e20 unknown :-1 s=CLOSED pgs=1002 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:33:27.661 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:27.657+0000 7f4bcb3ef740 1 -- 172.21.15.46:0/1855486359 >> 172.21.15.46:0/1855486359 conn(0x56446122bfe0 msgr2=0x56446122dd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:33:27.661 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:27.657+0000 7f4ba9ffb640 0 client.17907 ms_handle_reset on v2:172.21.15.142:6838/2519143918 2024-03-31T23:33:27.662 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:27.658+0000 7f4bcb3ef740 1 -- 172.21.15.46:0/1855486359 shutdown_connections 2024-03-31T23:33:27.662 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:27.658+0000 7f4bcb3ef740 1 -- 172.21.15.46:0/1855486359 wait complete. 2024-03-31T23:33:27.662 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:27.658+0000 7f4bcb3ef740 20 client.17907 trim_cache size 0 max 16384 2024-03-31T23:33:27.680 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:27.679+0000 7f4bcb3ef740 1 Processor -- start 2024-03-31T23:33:27.681 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:27.679+0000 7f4bcb3ef740 1 -- start start 2024-03-31T23:33:27.681 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:27.679+0000 7f4bcb3ef740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5644612316c0 0x56446123a550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:33:27.681 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:27.679+0000 7f4bcb3ef740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564461231fe0 0x56446124abc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:33:27.681 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:27.679+0000 7f4bcb3ef740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56446123adb0 0x564461256ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:33:27.681 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:27.679+0000 7f4bcb3ef740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f4ba40128c0 con 0x564461231fe0 2024-03-31T23:33:27.681 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:27.679+0000 7f4bcb3ef740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f4ba4010100 con 0x5644612316c0 2024-03-31T23:33:27.681 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:27.679+0000 7f4bcb3ef740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f4ba400df50 con 0x56446123adb0 2024-03-31T23:33:27.681 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:27.679+0000 7f4bc5cc7640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564461231fe0 0x56446124abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:33:27.681 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:27.679+0000 7f4bc5cc7640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564461231fe0 0x56446124abc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60600/0 (socket says 172.21.15.46:60600) 2024-03-31T23:33:27.681 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:27.679+0000 7f4bc5cc7640 1 -- 172.21.15.46:0/1785518493 learned_addr learned my addr 172.21.15.46:0/1785518493 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:33:27.681 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:27.679+0000 7f4bc64c8640 1 --2- 172.21.15.46:0/1785518493 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5644612316c0 0x56446123a550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:33:27.681 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:27.679+0000 7f4bc6cc9640 1 --2- 172.21.15.46:0/1785518493 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56446123adb0 0x564461256ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:33:27.682 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:27.680+0000 7f4bc5cc7640 1 -- 172.21.15.46:0/1785518493 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56446123adb0 msgr2=0x564461256ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:33:27.682 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:27.680+0000 7f4bc5cc7640 1 --2- 172.21.15.46:0/1785518493 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56446123adb0 0x564461256ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:33:27.682 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:27.680+0000 7f4bc5cc7640 1 -- 172.21.15.46:0/1785518493 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5644612316c0 msgr2=0x56446123a550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:33:27.682 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:27.680+0000 7f4bc5cc7640 1 --2- 172.21.15.46:0/1785518493 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5644612316c0 0x56446123a550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:33:27.682 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:27.680+0000 7f4bc5cc7640 1 -- 172.21.15.46:0/1785518493 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564461251d60 con 0x564461231fe0 2024-03-31T23:33:27.682 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:27.680+0000 7f4bc6cc9640 1 --2- 172.21.15.46:0/1785518493 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56446123adb0 0x564461256ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T23:33:27.682 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:27.680+0000 7f4bc5cc7640 1 --2- 172.21.15.46:0/1785518493 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564461231fe0 0x56446124abc0 secure :-1 s=READY pgs=1003 cs=0 l=1 rev1=1 crypto rx=0x7f4bb0000c00 tx=0x7f4bb0005e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c4544c0ae34ab468 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:33:27.685 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:27.680+0000 7f4babfff640 1 -- 172.21.15.46:0/1785518493 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bb0004250 con 0x564461231fe0 2024-03-31T23:33:27.685 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:27.680+0000 7f4babfff640 1 -- 172.21.15.46:0/1785518493 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4bb0025070 con 0x564461231fe0 2024-03-31T23:33:27.686 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:27.680+0000 7f4babfff640 1 -- 172.21.15.46:0/1785518493 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bb00038e0 con 0x564461231fe0 2024-03-31T23:33:27.686 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:27.681+0000 7f4bcb3ef740 1 -- 172.21.15.46:0/1785518493 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564461231fe0 msgr2=0x56446124abc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:33:27.686 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:27.681+0000 7f4bcb3ef740 1 --2- 172.21.15.46:0/1785518493 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564461231fe0 0x56446124abc0 secure :-1 s=READY pgs=1003 cs=0 l=1 rev1=1 crypto rx=0x7f4bb0000c00 tx=0x7f4bb0005e60 comp rx=0 tx=0).stop 2024-03-31T23:33:27.686 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:27.681+0000 7f4bcb3ef740 1 -- 172.21.15.46:0/1785518493 shutdown_connections 2024-03-31T23:33:27.686 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:27.681+0000 7f4bcb3ef740 1 --2- 172.21.15.46:0/1785518493 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56446123adb0 0x564461256ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:33:27.686 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:27.681+0000 7f4bcb3ef740 1 --2- 172.21.15.46:0/1785518493 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564461231fe0 0x56446124abc0 unknown :-1 s=CLOSED pgs=1003 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:33:27.686 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:27.681+0000 7f4bcb3ef740 1 --2- 172.21.15.46:0/1785518493 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5644612316c0 0x56446123a550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:33:27.686 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:27.681+0000 7f4bcb3ef740 1 -- 172.21.15.46:0/1785518493 >> 172.21.15.46:0/1785518493 conn(0x56446122bfe0 msgr2=0x56446122af80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:33:27.686 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:27.681+0000 7f4bcb3ef740 1 -- 172.21.15.46:0/1785518493 shutdown_connections 2024-03-31T23:33:27.686 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:27.681+0000 7f4bcb3ef740 1 -- 172.21.15.46:0/1785518493 wait complete. 2024-03-31T23:33:27.686 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:27.682+0000 7f4bcb3ef740 1 -- start start 2024-03-31T23:33:27.686 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:27.682+0000 7f4bcb3ef740 1 Processor -- start 2024-03-31T23:33:27.687 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:27.682+0000 7f4bcb3ef740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5644612316c0 0x564461241560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:33:27.687 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:27.682+0000 7f4bcb3ef740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x564461231fe0 0x5644612471f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:33:27.687 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:27.682+0000 7f4bcb3ef740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56446123adb0 0x56446123e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:33:27.687 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:27.682+0000 7f4bcb3ef740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f4ba400e5c0 con 0x56446123adb0 2024-03-31T23:33:27.687 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:27.682+0000 7f4bcb3ef740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f4ba400fa30 con 0x564461231fe0 2024-03-31T23:33:27.687 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:27.682+0000 7f4bcb3ef740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f4ba400ec90 con 0x5644612316c0 2024-03-31T23:33:27.687 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:27.682+0000 7f4bc64c8640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5644612316c0 0x564461241560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:33:27.687 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:27.682+0000 7f4bc5cc7640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x564461231fe0 0x5644612471f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:33:27.688 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:27.682+0000 7f4bc5cc7640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x564461231fe0 0x5644612471f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.46:56516/0 (socket says 172.21.15.46:56516) 2024-03-31T23:33:27.688 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:27.682+0000 7f4bc64c8640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5644612316c0 0x564461241560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.46:52976/0 (socket says 172.21.15.46:52976) 2024-03-31T23:33:27.688 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:27.683+0000 7f4bc64c8640 1 -- 172.21.15.46:0/1882906443 learned_addr learned my addr 172.21.15.46:0/1882906443 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:33:27.688 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:27.683+0000 7f4bcb3ef740 10 client.0 _ll_register_callbacks cb 0x7f4bbc00cf20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T23:33:27.688 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:27.683+0000 7f4bc64c8640 1 -- 172.21.15.46:0/1882906443 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x564461231fe0 msgr2=0x5644612471f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:33:27.688 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:27.683+0000 7f4bc64c8640 1 --2- 172.21.15.46:0/1882906443 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x564461231fe0 0x5644612471f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:33:27.688 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:27.683+0000 7f4bc64c8640 1 -- 172.21.15.46:0/1882906443 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56446123adb0 msgr2=0x56446123e8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:33:27.688 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:27.683+0000 7f4bc64c8640 1 --2- 172.21.15.46:0/1882906443 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56446123adb0 0x56446123e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:33:27.688 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:27.683+0000 7f4bc64c8640 1 -- 172.21.15.46:0/1882906443 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56446123ee30 con 0x5644612316c0 2024-03-31T23:33:27.688 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:27.683+0000 7f4bc64c8640 1 --2- 172.21.15.46:0/1882906443 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5644612316c0 0x564461241560 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f4bbc009380 tx=0x7f4bbc099a50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f121cfad810f9a21 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:33:27.688 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:27.683+0000 7f4ba9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.162:3300/0 2024-03-31T23:33:27.688 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:27.683+0000 7f4ba9ffb640 1 -- 172.21.15.46:0/1882906443 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bbc0a3ea0 con 0x5644612316c0 2024-03-31T23:33:27.689 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:27.683+0000 7f4ba9ffb640 1 -- 172.21.15.46:0/1882906443 <== mon.1 v2:172.21.15.162:3300/0 2 ==== osd_map(131..131 src has 1..131) ==== 9466+0+0 (secure 0 0 0) 0x7f4bbc0bb2c0 con 0x5644612316c0 2024-03-31T23:33:27.689 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:27.683+0000 7f4bcb3ef740 1 -- 172.21.15.46:0/1882906443 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4bb0025070 con 0x5644612316c0 2024-03-31T23:33:27.689 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:27.684+0000 7f4ba9ffb640 1 -- 172.21.15.46:0/1882906443 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_get_version_reply(handle=1 version=51) ==== 24+0+0 (secure 0 0 0) 0x7f4bbc09f150 con 0x5644612316c0 2024-03-31T23:33:27.689 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:27.684+0000 7f4bcb3ef740 10 client.27631 fetch_fsmap learned FSMap version 51 2024-03-31T23:33:27.689 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:27.684+0000 7f4bcb3ef740 1 -- 172.21.15.46:0/1882906443 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,fsmap=51,monmap=4+}) -- 0x5644612ba5e0 con 0x5644612316c0 2024-03-31T23:33:27.689 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:27.684+0000 7f4ba9ffb640 1 -- 172.21.15.46:0/1882906443 <== mon.1 v2:172.21.15.162:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4bb0025070 con 0x5644612316c0 2024-03-31T23:33:27.689 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:27.684+0000 7f4ba9ffb640 1 -- 172.21.15.46:0/1882906443 <== mon.1 v2:172.21.15.162:3300/0 5 ==== fsmap(e 51) ==== 8358+0+0 (secure 0 0 0) 0x7f4bbc097390 con 0x5644612316c0 2024-03-31T23:33:27.689 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:27.684+0000 7f4bcb3ef740 10 client.27631 fetch_fsmap finished waiting for FSMap version 51 2024-03-31T23:33:27.689 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:27.684+0000 7f4bcb3ef740 10 client.27631 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-31T23:33:27.689 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:27.684+0000 7f4bcb3ef740 20 client.27631 populate_metadata read hostname 'smithi046' 2024-03-31T23:33:27.689 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:27.684+0000 7f4bcb3ef740 1 --2- 172.21.15.46:0/1882906443 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x5644612bad30 0x5644612bd1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:33:27.689 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:27.684+0000 7f4bcb3ef740 4 client.27631 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-31T23:33:27.689 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:27.684+0000 7f4bcb3ef740 1 -- 172.21.15.46:0/1882906443 --> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5644611a1e30 con 0x5644612bad30 2024-03-31T23:33:27.689 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:27.685+0000 7f4bc5cc7640 1 --2- 172.21.15.46:0/1882906443 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x5644612bad30 0x5644612bd1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:33:27.690 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:27.685+0000 7f4ba9ffb640 10 client.27631 ms_handle_connect on v2:172.21.15.142:6838/2519143918 2024-03-31T23:33:27.690 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:27.685+0000 7f4bc5cc7640 1 --2- 172.21.15.46:0/1882906443 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x5644612bad30 0x5644612bd1d0 crc :-1 s=READY pgs=271 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ffa9344d7cb32fd server_cookie=eff9811e8b35b69b in_seq=0 out_seq=0 2024-03-31T23:33:27.934 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:27.933+0000 7f4ba9ffb640 1 -- 172.21.15.46:0/1882906443 <== mds.0 v2:172.21.15.142:6838/2519143918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5644611a1e30 con 0x5644612bad30 2024-03-31T23:33:27.935 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:27.933+0000 7f4ba9ffb640 10 client.27631 handle_command_reply: tid=0 2024-03-31T23:33:27.935 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:27.933+0000 7f4bcb3ef740 1 client.27631 shutdown 2024-03-31T23:33:27.935 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:27.933+0000 7f4bcb3ef740 1 -- 172.21.15.46:0/1882906443 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5644612316c0 msgr2=0x564461241560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:33:27.935 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:27.933+0000 7f4bcb3ef740 1 --2- 172.21.15.46:0/1882906443 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5644612316c0 0x564461241560 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f4bbc009380 tx=0x7f4bbc099a50 comp rx=0 tx=0).stop 2024-03-31T23:33:27.935 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:27.933+0000 7f4bcb3ef740 1 -- 172.21.15.46:0/1882906443 shutdown_connections 2024-03-31T23:33:27.935 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:27.933+0000 7f4bcb3ef740 1 --2- 172.21.15.46:0/1882906443 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x5644612bad30 0x5644612bd1d0 crc :-1 s=READY pgs=271 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:33:27.935 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:27.933+0000 7f4bcb3ef740 1 --2- 172.21.15.46:0/1882906443 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56446123adb0 0x56446123e8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:33:27.935 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:27.933+0000 7f4ba9ffb640 0 client.27631 ms_handle_reset on v2:172.21.15.142:6838/2519143918 2024-03-31T23:33:27.935 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:27.933+0000 7f4bcb3ef740 1 --2- 172.21.15.46:0/1882906443 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x564461231fe0 0x5644612471f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:33:27.935 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:27.933+0000 7f4bcb3ef740 1 --2- 172.21.15.46:0/1882906443 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5644612316c0 0x564461241560 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:33:27.935 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:27.933+0000 7f4bcb3ef740 1 -- 172.21.15.46:0/1882906443 >> 172.21.15.46:0/1882906443 conn(0x56446122bfe0 msgr2=0x56446122d430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:33:27.935 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:27.933+0000 7f4bcb3ef740 1 -- 172.21.15.46:0/1882906443 shutdown_connections 2024-03-31T23:33:27.936 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:27.933+0000 7f4bcb3ef740 1 -- 172.21.15.46:0/1882906443 wait complete. 2024-03-31T23:33:27.936 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:27.933+0000 7f4bcb3ef740 20 client.27631 trim_cache size 0 max 16384 2024-03-31T23:33:27.936 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout:{ 2024-03-31T23:33:27.936 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout: "message": "", 2024-03-31T23:33:27.936 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout: "return_code": 0 2024-03-31T23:33:27.936 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout:} 2024-03-31T23:33:27.937 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:27.934+0000 7f4bc872c640 1 -- 172.21.15.46:0/3610136790 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f4bac0a26b0 msgr2=0x7f4bac0a4b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:33:27.937 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:27.934+0000 7f4bc872c640 1 --2- 172.21.15.46:0/3610136790 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f4bac0a26b0 0x7f4bac0a4b70 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f4bb804c000 tx=0x7f4bb8045000 comp rx=0 tx=0).stop 2024-03-31T23:33:27.937 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:27.934+0000 7f4bc872c640 1 -- 172.21.15.46:0/3610136790 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4bc00ffc50 msgr2=0x7f4bc0197e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:33:27.937 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:27.934+0000 7f4bc872c640 1 --2- 172.21.15.46:0/3610136790 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4bc00ffc50 0x7f4bc0197e30 secure :-1 s=READY pgs=1001 cs=0 l=1 rev1=1 crypto rx=0x7f4bbc00e9f0 tx=0x7f4bbc00eeb0 comp rx=0 tx=0).stop 2024-03-31T23:33:27.937 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:27.934+0000 7f4bc872c640 1 -- 172.21.15.46:0/3610136790 shutdown_connections 2024-03-31T23:33:27.937 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:27.934+0000 7f4bc872c640 1 --2- 172.21.15.46:0/3610136790 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f4bac0a26b0 0x7f4bac0a4b70 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:33:27.937 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:27.934+0000 7f4bc872c640 1 --2- 172.21.15.46:0/3610136790 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4bc0107730 0x7f4bc019c6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:33:27.937 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:27.934+0000 7f4bc872c640 1 --2- 172.21.15.46:0/3610136790 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4bc0100610 0x7f4bc0198370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:33:27.937 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:27.934+0000 7f4bc872c640 1 --2- 172.21.15.46:0/3610136790 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4bc00ffc50 0x7f4bc0197e30 unknown :-1 s=CLOSED pgs=1001 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:33:27.937 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:27.934+0000 7f4bc872c640 1 -- 172.21.15.46:0/3610136790 >> 172.21.15.46:0/3610136790 conn(0x7f4bc0076b90 msgr2=0x7f4bc00fe170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:33:27.937 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:27.934+0000 7f4bc872c640 1 -- 172.21.15.46:0/3610136790 shutdown_connections 2024-03-31T23:33:27.938 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:27.934+0000 7f4bc872c640 1 -- 172.21.15.46:0/3610136790 wait complete. 2024-03-31T23:33:28.304 INFO:tasks.workunit.client.0.smithi046.stdout: 10 80103 8.28 MB/sec execute 393 sec latency 191.697 ms 2024-03-31T23:33:29.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:33:28 smithi162 ceph-mon[36237]: pgmap v690: 129 pgs: 129 active+clean; 1.1 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 5.3 MiB/s wr, 289 op/s 2024-03-31T23:33:29.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:33:28 smithi046 ceph-mon[32498]: pgmap v690: 129 pgs: 129 active+clean; 1.1 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 5.3 MiB/s wr, 289 op/s 2024-03-31T23:33:29.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:33:28 smithi142 ceph-mon[35271]: pgmap v690: 129 pgs: 129 active+clean; 1.1 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 5.3 MiB/s wr, 289 op/s 2024-03-31T23:33:29.304 INFO:tasks.workunit.client.0.smithi046.stdout: 10 80261 8.28 MB/sec execute 394 sec latency 89.458 ms 2024-03-31T23:33:30.304 INFO:tasks.workunit.client.0.smithi046.stdout: 10 80431 8.29 MB/sec execute 395 sec latency 85.184 ms 2024-03-31T23:33:31.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:33:30 smithi162 ceph-mon[36237]: pgmap v691: 129 pgs: 129 active+clean; 1.1 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 7.3 MiB/s wr, 458 op/s 2024-03-31T23:33:31.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:33:30 smithi046 ceph-mon[32498]: pgmap v691: 129 pgs: 129 active+clean; 1.1 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 7.3 MiB/s wr, 458 op/s 2024-03-31T23:33:31.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:33:30 smithi142 ceph-mon[35271]: pgmap v691: 129 pgs: 129 active+clean; 1.1 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 7.3 MiB/s wr, 458 op/s 2024-03-31T23:33:31.305 INFO:tasks.workunit.client.0.smithi046.stdout: 10 80578 8.28 MB/sec execute 396 sec latency 99.199 ms 2024-03-31T23:33:32.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:33:31 smithi162 ceph-mon[36237]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-31T23:33:32.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:33:31 smithi162 ceph-mon[36237]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' 2024-03-31T23:33:32.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:33:31 smithi162 ceph-mon[36237]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' 2024-03-31T23:33:32.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:33:31 smithi162 ceph-mon[36237]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' 2024-03-31T23:33:32.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:33:31 smithi162 ceph-mon[36237]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' 2024-03-31T23:33:32.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:33:31 smithi162 ceph-mon[36237]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' 2024-03-31T23:33:32.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:33:31 smithi162 ceph-mon[36237]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' 2024-03-31T23:33:32.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:33:31 smithi046 ceph-mon[32498]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-31T23:33:32.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:33:31 smithi046 ceph-mon[32498]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' 2024-03-31T23:33:32.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:33:31 smithi046 ceph-mon[32498]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' 2024-03-31T23:33:32.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:33:31 smithi046 ceph-mon[32498]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' 2024-03-31T23:33:32.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:33:31 smithi046 ceph-mon[32498]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' 2024-03-31T23:33:32.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:33:31 smithi046 ceph-mon[32498]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' 2024-03-31T23:33:32.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:33:31 smithi046 ceph-mon[32498]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' 2024-03-31T23:33:32.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:33:31 smithi142 ceph-mon[35271]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-31T23:33:32.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:33:31 smithi142 ceph-mon[35271]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' 2024-03-31T23:33:32.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:33:31 smithi142 ceph-mon[35271]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' 2024-03-31T23:33:32.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:33:31 smithi142 ceph-mon[35271]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' 2024-03-31T23:33:32.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:33:31 smithi142 ceph-mon[35271]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' 2024-03-31T23:33:32.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:33:31 smithi142 ceph-mon[35271]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' 2024-03-31T23:33:32.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:33:31 smithi142 ceph-mon[35271]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' 2024-03-31T23:33:32.306 INFO:tasks.workunit.client.0.smithi046.stdout: 10 80717 8.28 MB/sec execute 397 sec latency 110.875 ms 2024-03-31T23:33:33.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:33:32 smithi162 ceph-mon[36237]: pgmap v692: 129 pgs: 129 active+clean; 1.1 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 4.9 MiB/s wr, 297 op/s 2024-03-31T23:33:33.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:33:32 smithi162 ceph-mon[36237]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:33:33.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:33:32 smithi162 ceph-mon[36237]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-31T23:33:33.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:33:32 smithi162 ceph-mon[36237]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' 2024-03-31T23:33:33.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:33:32 smithi046 ceph-mon[32498]: pgmap v692: 129 pgs: 129 active+clean; 1.1 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 4.9 MiB/s wr, 297 op/s 2024-03-31T23:33:33.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:33:32 smithi046 ceph-mon[32498]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:33:33.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:33:32 smithi046 ceph-mon[32498]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-31T23:33:33.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:33:32 smithi046 ceph-mon[32498]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' 2024-03-31T23:33:33.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:33:32 smithi142 ceph-mon[35271]: pgmap v692: 129 pgs: 129 active+clean; 1.1 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 4.9 MiB/s wr, 297 op/s 2024-03-31T23:33:33.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:33:32 smithi142 ceph-mon[35271]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:33:33.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:33:32 smithi142 ceph-mon[35271]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-31T23:33:33.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:33:32 smithi142 ceph-mon[35271]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' 2024-03-31T23:33:33.308 INFO:tasks.workunit.client.0.smithi046.stdout: 10 80856 8.27 MB/sec execute 398 sec latency 105.200 ms 2024-03-31T23:33:34.308 INFO:tasks.workunit.client.0.smithi046.stdout: 10 81047 8.29 MB/sec execute 399 sec latency 111.099 ms 2024-03-31T23:33:35.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:33:34 smithi162 ceph-mon[36237]: pgmap v693: 129 pgs: 129 active+clean; 1.1 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 4.9 MiB/s wr, 297 op/s 2024-03-31T23:33:35.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:33:34 smithi046 ceph-mon[32498]: pgmap v693: 129 pgs: 129 active+clean; 1.1 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 4.9 MiB/s wr, 297 op/s 2024-03-31T23:33:35.194 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:33:34 smithi142 ceph-mon[35271]: pgmap v693: 129 pgs: 129 active+clean; 1.1 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 4.9 MiB/s wr, 297 op/s 2024-03-31T23:33:35.308 INFO:tasks.workunit.client.0.smithi046.stdout: 10 81202 8.28 MB/sec execute 400 sec latency 104.948 ms 2024-03-31T23:33:36.308 INFO:tasks.workunit.client.0.smithi046.stdout: 10 81361 8.29 MB/sec execute 401 sec latency 103.666 ms 2024-03-31T23:33:37.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:33:36 smithi162 ceph-mon[36237]: pgmap v694: 129 pgs: 129 active+clean; 1.1 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 6.9 MiB/s wr, 405 op/s 2024-03-31T23:33:37.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:33:36 smithi046 ceph-mon[32498]: pgmap v694: 129 pgs: 129 active+clean; 1.1 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 6.9 MiB/s wr, 405 op/s 2024-03-31T23:33:37.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:33:36 smithi142 ceph-mon[35271]: pgmap v694: 129 pgs: 129 active+clean; 1.1 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 6.9 MiB/s wr, 405 op/s 2024-03-31T23:33:37.308 INFO:tasks.workunit.client.0.smithi046.stdout: 10 81508 8.29 MB/sec execute 402 sec latency 96.902 ms 2024-03-31T23:33:38.308 INFO:tasks.workunit.client.0.smithi046.stdout: 10 81655 8.29 MB/sec execute 403 sec latency 159.272 ms 2024-03-31T23:33:39.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:33:38 smithi162 ceph-mon[36237]: pgmap v695: 129 pgs: 129 active+clean; 1.1 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 4.3 MiB/s wr, 291 op/s 2024-03-31T23:33:39.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:33:38 smithi046 ceph-mon[32498]: pgmap v695: 129 pgs: 129 active+clean; 1.1 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 4.3 MiB/s wr, 291 op/s 2024-03-31T23:33:39.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:33:38 smithi142 ceph-mon[35271]: pgmap v695: 129 pgs: 129 active+clean; 1.1 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 4.3 MiB/s wr, 291 op/s 2024-03-31T23:33:39.308 INFO:tasks.workunit.client.0.smithi046.stdout: 10 81821 8.29 MB/sec execute 404 sec latency 88.192 ms 2024-03-31T23:33:40.308 INFO:tasks.workunit.client.0.smithi046.stdout: 10 81973 8.29 MB/sec execute 405 sec latency 135.239 ms 2024-03-31T23:33:41.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:33:40 smithi162 ceph-mon[36237]: pgmap v696: 129 pgs: 129 active+clean; 1.1 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 6.9 MiB/s wr, 392 op/s 2024-03-31T23:33:41.064 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:41.062+0000 7f9060b8c640 1 -- 172.21.15.46:0/3673986733 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f905c1008a0 msgr2=0x7f905c100d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:33:41.064 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:41.062+0000 7f9060b8c640 1 --2- 172.21.15.46:0/3673986733 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f905c1008a0 0x7f905c100d20 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f9044009890 tx=0x7f904402bb70 comp rx=0 tx=0).stop 2024-03-31T23:33:41.064 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:41.063+0000 7f9060b8c640 1 -- 172.21.15.46:0/3673986733 shutdown_connections 2024-03-31T23:33:41.064 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:41.063+0000 7f9060b8c640 1 --2- 172.21.15.46:0/3673986733 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f905c101260 0x7f905c10d480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:33:41.064 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:41.063+0000 7f9060b8c640 1 --2- 172.21.15.46:0/3673986733 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f905c1008a0 0x7f905c100d20 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:33:41.064 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:41.063+0000 7f9060b8c640 1 --2- 172.21.15.46:0/3673986733 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f905c1055f0 0x7f905c1059d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:33:41.064 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:41.063+0000 7f9060b8c640 1 -- 172.21.15.46:0/3673986733 >> 172.21.15.46:0/3673986733 conn(0x7f905c0774a0 msgr2=0x7f905c0778b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:33:41.064 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:41.063+0000 7f9060b8c640 1 -- 172.21.15.46:0/3673986733 shutdown_connections 2024-03-31T23:33:41.065 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:41.063+0000 7f9060b8c640 1 -- 172.21.15.46:0/3673986733 wait complete. 2024-03-31T23:33:41.067 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:41.063+0000 7f9060b8c640 1 Processor -- start 2024-03-31T23:33:41.067 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:41.064+0000 7f9060b8c640 1 -- start start 2024-03-31T23:33:41.067 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:41.064+0000 7f9060b8c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f905c1008a0 0x7f905c197f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:33:41.067 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:41.064+0000 7f9060b8c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f905c101260 0x7f905c198490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:33:41.067 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:41.064+0000 7f9060b8c640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f905c1055f0 0x7f905c19c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:33:41.067 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:41.064+0000 7f9060b8c640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f905c1109c0 con 0x7f905c101260 2024-03-31T23:33:41.067 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:41.064+0000 7f9060b8c640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f905c110840 con 0x7f905c1055f0 2024-03-31T23:33:41.068 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:41.064+0000 7f9060b8c640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f905c110b40 con 0x7f905c1008a0 2024-03-31T23:33:41.068 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:41.064+0000 7f9059d9b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f905c101260 0x7f905c198490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:33:41.068 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:41.064+0000 7f905ad9d640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f905c1055f0 0x7f905c19c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:33:41.068 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:41.064+0000 7f905ad9d640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f905c1055f0 0x7f905c19c7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.46:59142/0 (socket says 172.21.15.46:59142) 2024-03-31T23:33:41.068 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:41.064+0000 7f9059d9b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f905c101260 0x7f905c198490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39390/0 (socket says 172.21.15.46:39390) 2024-03-31T23:33:41.068 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:41.064+0000 7f905ad9d640 1 -- 172.21.15.46:0/1481688269 learned_addr learned my addr 172.21.15.46:0/1481688269 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:33:41.068 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:41.064+0000 7f905a59c640 1 --2- 172.21.15.46:0/1481688269 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f905c1008a0 0x7f905c197f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:33:41.068 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:41.064+0000 7f9059d9b640 1 -- 172.21.15.46:0/1481688269 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f905c1008a0 msgr2=0x7f905c197f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:33:41.068 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:41.064+0000 7f9059d9b640 1 --2- 172.21.15.46:0/1481688269 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f905c1008a0 0x7f905c197f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:33:41.068 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:41.064+0000 7f9059d9b640 1 -- 172.21.15.46:0/1481688269 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f905c1055f0 msgr2=0x7f905c19c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:33:41.068 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:41.064+0000 7f9059d9b640 1 --2- 172.21.15.46:0/1481688269 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f905c1055f0 0x7f905c19c7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:33:41.069 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:41.064+0000 7f9059d9b640 1 -- 172.21.15.46:0/1481688269 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f905c19cef0 con 0x7f905c101260 2024-03-31T23:33:41.069 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:41.065+0000 7f9059d9b640 1 --2- 172.21.15.46:0/1481688269 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f905c101260 0x7f905c198490 secure :-1 s=READY pgs=1004 cs=0 l=1 rev1=1 crypto rx=0x7f904402c0b0 tx=0x7f90440045b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:33:41.069 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:41.065+0000 7f90537fe640 1 -- 172.21.15.46:0/1481688269 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9044041020 con 0x7f905c101260 2024-03-31T23:33:41.069 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:41.065+0000 7f90537fe640 1 -- 172.21.15.46:0/1481688269 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f904402c9c0 con 0x7f905c101260 2024-03-31T23:33:41.069 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:41.065+0000 7f9060b8c640 1 -- 172.21.15.46:0/1481688269 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f905c19d180 con 0x7f905c101260 2024-03-31T23:33:41.069 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:41.065+0000 7f9060b8c640 1 -- 172.21.15.46:0/1481688269 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f905c1a4a70 con 0x7f905c101260 2024-03-31T23:33:41.069 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:41.065+0000 7f90537fe640 1 -- 172.21.15.46:0/1481688269 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9044039040 con 0x7f905c101260 2024-03-31T23:33:41.072 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:41.068+0000 7f9063771740 1 Processor -- start 2024-03-31T23:33:41.072 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:41.068+0000 7f9063771740 1 -- start start 2024-03-31T23:33:41.072 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:41.069+0000 7f90537fe640 1 -- 172.21.15.46:0/1481688269 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 25) ==== 137451+0+0 (secure 0 0 0) 0x7f9044044020 con 0x7f905c101260 2024-03-31T23:33:41.073 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:41.069+0000 7f9063771740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ef70ffa6c0 0x55ef70ffaaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:33:41.073 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:41.069+0000 7f9063771740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ef70ffafe0 0x55ef71003870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:33:41.073 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:41.069+0000 7f905ad9d640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ef70ffa6c0 0x55ef70ffaaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:33:41.073 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:41.069+0000 7f905ad9d640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ef70ffa6c0 0x55ef70ffaaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.46:52574/0 (socket says 172.21.15.46:52574) 2024-03-31T23:33:41.073 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:41.069+0000 7f905a59c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ef70ffafe0 0x55ef71003870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:33:41.073 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:41.069+0000 7f905a59c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ef70ffafe0 0x55ef71003870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39394/0 (socket says 172.21.15.46:39394) 2024-03-31T23:33:41.073 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:41.069+0000 7f905ad9d640 1 -- 172.21.15.46:0/3157470168 learned_addr learned my addr 172.21.15.46:0/3157470168 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:33:41.073 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:41.069+0000 7f9063771740 1 --2- 172.21.15.46:0/3157470168 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55ef71003db0 0x55ef710061a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:33:41.073 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:41.069+0000 7f9063771740 1 -- 172.21.15.46:0/3157470168 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55ef70f5d8f0 con 0x55ef70ffafe0 2024-03-31T23:33:41.073 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:41.069+0000 7f9063771740 1 -- 172.21.15.46:0/3157470168 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55ef70f068d0 con 0x55ef71003db0 2024-03-31T23:33:41.073 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:41.069+0000 7f9063771740 1 -- 172.21.15.46:0/3157470168 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55ef70f59990 con 0x55ef70ffa6c0 2024-03-31T23:33:41.073 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:41.069+0000 7f9059d9b640 1 --2- 172.21.15.46:0/3157470168 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55ef71003db0 0x55ef710061a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:33:41.074 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:41.069+0000 7f905ad9d640 1 -- 172.21.15.46:0/3157470168 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55ef71003db0 msgr2=0x55ef710061a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:33:41.074 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:41.069+0000 7f905ad9d640 1 --2- 172.21.15.46:0/3157470168 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55ef71003db0 0x55ef710061a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:33:41.074 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:41.069+0000 7f905ad9d640 1 -- 172.21.15.46:0/3157470168 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ef70ffafe0 msgr2=0x55ef71003870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:33:41.074 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:41.070+0000 7f905ad9d640 1 --2- 172.21.15.46:0/3157470168 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ef70ffafe0 0x55ef71003870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:33:41.074 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:41.070+0000 7f905ad9d640 1 -- 172.21.15.46:0/3157470168 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ef70c72210 con 0x55ef70ffa6c0 2024-03-31T23:33:41.074 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:41.070+0000 7f90537fe640 1 --2- 172.21.15.46:0/1481688269 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f90480a26b0 0x7f90480a4b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:33:41.074 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:41.070+0000 7f9059d9b640 1 --2- 172.21.15.46:0/3157470168 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55ef71003db0 0x55ef710061a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T23:33:41.074 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:41.070+0000 7f905a59c640 1 --2- 172.21.15.46:0/3157470168 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ef70ffafe0 0x55ef71003870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:33:41.074 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:41.070+0000 7f905a59c640 1 --2- 172.21.15.46:0/1481688269 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f90480a26b0 0x7f90480a4b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:33:41.074 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:41.070+0000 7f905ad9d640 1 --2- 172.21.15.46:0/3157470168 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ef70ffa6c0 0x55ef70ffaaa0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f904c00e770 tx=0x7f904c0095c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=14d38070bfdd4aae server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:33:41.074 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:41.070+0000 7f90537fe640 1 -- 172.21.15.46:0/1481688269 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(131..131 src has 1..131) ==== 9466+0+0 (secure 0 0 0) 0x7f90440f0490 con 0x7f905c101260 2024-03-31T23:33:41.074 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:41.070+0000 7f9050ff9640 1 -- 172.21.15.46:0/3157470168 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f904c00d9c0 con 0x55ef70ffa6c0 2024-03-31T23:33:41.074 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:41.070+0000 7f9050ff9640 1 -- 172.21.15.46:0/3157470168 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f904c003970 con 0x55ef70ffa6c0 2024-03-31T23:33:41.075 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:41.070+0000 7f9050ff9640 1 -- 172.21.15.46:0/3157470168 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f904c004490 con 0x55ef70ffa6c0 2024-03-31T23:33:41.075 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:41.071+0000 7f9063771740 1 -- 172.21.15.46:0/3157470168 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ef70ffa6c0 msgr2=0x55ef70ffaaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:33:41.075 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:41.071+0000 7f905a59c640 1 --2- 172.21.15.46:0/1481688269 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f90480a26b0 0x7f90480a4b70 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x55ef70ffa170 tx=0x7f9054005f90 comp rx=0 tx=0).ready entity=mgr.15654 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:33:41.075 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:41.071+0000 7f9063771740 1 --2- 172.21.15.46:0/3157470168 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ef70ffa6c0 0x55ef70ffaaa0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f904c00e770 tx=0x7f904c0095c0 comp rx=0 tx=0).stop 2024-03-31T23:33:41.075 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:41.071+0000 7f9063771740 1 -- 172.21.15.46:0/3157470168 shutdown_connections 2024-03-31T23:33:41.075 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:41.071+0000 7f9063771740 1 --2- 172.21.15.46:0/3157470168 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55ef71003db0 0x55ef710061a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:33:41.075 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:41.071+0000 7f9063771740 1 --2- 172.21.15.46:0/3157470168 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ef70ffafe0 0x55ef71003870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:33:41.075 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:41.071+0000 7f9063771740 1 --2- 172.21.15.46:0/3157470168 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ef70ffa6c0 0x55ef70ffaaa0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:33:41.075 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:41.071+0000 7f9063771740 1 -- 172.21.15.46:0/3157470168 >> 172.21.15.46:0/3157470168 conn(0x55ef70ff4fe0 msgr2=0x55ef70ff53c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:33:41.075 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:41.071+0000 7f9063771740 1 -- 172.21.15.46:0/3157470168 shutdown_connections 2024-03-31T23:33:41.075 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:41.071+0000 7f9063771740 1 -- 172.21.15.46:0/3157470168 wait complete. 2024-03-31T23:33:41.075 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:41.072+0000 7f9063771740 1 -- start start 2024-03-31T23:33:41.076 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:41.072+0000 7f9063771740 1 Processor -- start 2024-03-31T23:33:41.079 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:41.072+0000 7f9063771740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ef70ffa6c0 0x55ef71000e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:33:41.079 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:41.072+0000 7f9063771740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55ef70ffafe0 0x55ef71001390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:33:41.080 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:41.072+0000 7f9063771740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ef71003db0 0x55ef710018d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:33:41.080 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:41.072+0000 7f9063771740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55ef710069e0 con 0x55ef71003db0 2024-03-31T23:33:41.080 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:41.072+0000 7f9063771740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55ef70efbf80 con 0x55ef70ffafe0 2024-03-31T23:33:41.080 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:41.072+0000 7f9063771740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55ef71006b60 con 0x55ef70ffa6c0 2024-03-31T23:33:41.080 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:41.072+0000 7f9063771740 10 client.0 _ll_register_callbacks cb 0x55ef70f54290 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T23:33:41.080 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:41.072+0000 7f905a59c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ef70ffa6c0 0x55ef71000e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:33:41.080 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:41.072+0000 7f905a59c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ef70ffa6c0 0x55ef71000e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.46:52576/0 (socket says 172.21.15.46:52576) 2024-03-31T23:33:41.080 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:41.072+0000 7f905a59c640 1 -- 172.21.15.46:0/2317593907 learned_addr learned my addr 172.21.15.46:0/2317593907 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:33:41.080 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:41.072+0000 7f9059d9b640 1 --2- 172.21.15.46:0/2317593907 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55ef70ffafe0 0x55ef71001390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:33:41.080 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:41.072+0000 7f905ad9d640 1 --2- 172.21.15.46:0/2317593907 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ef71003db0 0x55ef710018d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:33:41.080 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:41.073+0000 7f905a59c640 1 -- 172.21.15.46:0/2317593907 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55ef70ffafe0 msgr2=0x55ef71001390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:33:41.080 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:41.073+0000 7f905a59c640 1 --2- 172.21.15.46:0/2317593907 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55ef70ffafe0 0x55ef71001390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:33:41.081 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:41.073+0000 7f905a59c640 1 -- 172.21.15.46:0/2317593907 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ef71003db0 msgr2=0x55ef710018d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:33:41.081 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:41.073+0000 7f905a59c640 1 --2- 172.21.15.46:0/2317593907 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ef71003db0 0x55ef710018d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:33:41.081 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:41.073+0000 7f905a59c640 1 -- 172.21.15.46:0/2317593907 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ef70d3e620 con 0x55ef70ffa6c0 2024-03-31T23:33:41.081 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:41.073+0000 7f905a59c640 1 --2- 172.21.15.46:0/2317593907 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ef70ffa6c0 0x55ef71000e50 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f9054004620 tx=0x7f9054004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=61ec0026924d6b51 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:33:41.081 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:41.073+0000 7f9041ffb640 10 client.0 ms_handle_connect on v2:172.21.15.162:3300/0 2024-03-31T23:33:41.081 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:41.073+0000 7f9041ffb640 1 -- 172.21.15.46:0/2317593907 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9054009be0 con 0x55ef70ffa6c0 2024-03-31T23:33:41.081 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:41.073+0000 7f9041ffb640 1 -- 172.21.15.46:0/2317593907 <== mon.1 v2:172.21.15.162:3300/0 2 ==== osd_map(131..131 src has 1..131) ==== 9466+0+0 (secure 0 0 0) 0x7f905401f640 con 0x55ef70ffa6c0 2024-03-31T23:33:41.081 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:41.073+0000 7f9063771740 1 -- 172.21.15.46:0/2317593907 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f904c003970 con 0x55ef70ffa6c0 2024-03-31T23:33:41.081 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:41.073+0000 7f9041ffb640 1 -- 172.21.15.46:0/2317593907 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_get_version_reply(handle=1 version=51) ==== 24+0+0 (secure 0 0 0) 0x7f905401f840 con 0x55ef70ffa6c0 2024-03-31T23:33:41.081 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:41.073+0000 7f9063771740 10 client.27643 fetch_fsmap learned FSMap version 51 2024-03-31T23:33:41.081 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:41.073+0000 7f9063771740 1 -- 172.21.15.46:0/2317593907 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,fsmap=51,monmap=4+}) -- 0x55ef710128e0 con 0x55ef70ffa6c0 2024-03-31T23:33:41.081 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:41.074+0000 7f9041ffb640 1 -- 172.21.15.46:0/2317593907 <== mon.1 v2:172.21.15.162:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f904c003970 con 0x55ef70ffa6c0 2024-03-31T23:33:41.081 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:41.074+0000 7f9041ffb640 1 -- 172.21.15.46:0/2317593907 <== mon.1 v2:172.21.15.162:3300/0 5 ==== fsmap(e 51) ==== 8358+0+0 (secure 0 0 0) 0x7f9054027070 con 0x55ef70ffa6c0 2024-03-31T23:33:41.082 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:41.074+0000 7f9063771740 10 client.27643 fetch_fsmap finished waiting for FSMap version 51 2024-03-31T23:33:41.082 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:41.074+0000 7f9063771740 10 client.27643 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-31T23:33:41.082 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:41.074+0000 7f9063771740 20 client.27643 populate_metadata read hostname 'smithi046' 2024-03-31T23:33:41.082 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:41.074+0000 7f9063771740 1 --2- 172.21.15.46:0/2317593907 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55ef71013df0 0x55ef710161e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:33:41.082 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:41.074+0000 7f9063771740 4 client.27643 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-31T23:33:41.082 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:41.074+0000 7f9063771740 1 -- 172.21.15.46:0/2317593907 --> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ef70f71120 con 0x55ef71013df0 2024-03-31T23:33:41.082 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:41.074+0000 7f9059d9b640 1 --2- 172.21.15.46:0/2317593907 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55ef71013df0 0x55ef710161e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:33:41.082 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:41.075+0000 7f9041ffb640 10 client.27643 ms_handle_connect on v2:172.21.15.142:6838/2519143918 2024-03-31T23:33:41.082 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:41.075+0000 7f9059d9b640 1 --2- 172.21.15.46:0/2317593907 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55ef71013df0 0x55ef710161e0 crc :-1 s=READY pgs=272 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8ad4ddb08efbc078 server_cookie=135223249dba2b85 in_seq=0 out_seq=0 2024-03-31T23:33:41.082 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:41.076+0000 7f9041ffb640 1 -- 172.21.15.46:0/2317593907 <== mds.0 v2:172.21.15.142:6838/2519143918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55ef70f71120 con 0x55ef71013df0 2024-03-31T23:33:41.082 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:41.076+0000 7f9041ffb640 10 client.27643 handle_command_reply: tid=0 2024-03-31T23:33:41.082 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:41.076+0000 7f9063771740 1 client.27643 shutdown 2024-03-31T23:33:41.082 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:41.076+0000 7f9063771740 1 -- 172.21.15.46:0/2317593907 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ef70ffa6c0 msgr2=0x55ef71000e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:33:41.082 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:41.076+0000 7f9063771740 1 --2- 172.21.15.46:0/2317593907 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ef70ffa6c0 0x55ef71000e50 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f9054004620 tx=0x7f9054004730 comp rx=0 tx=0).stop 2024-03-31T23:33:41.083 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:41.076+0000 7f9063771740 1 -- 172.21.15.46:0/2317593907 shutdown_connections 2024-03-31T23:33:41.083 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:41.076+0000 7f9063771740 1 --2- 172.21.15.46:0/2317593907 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55ef71013df0 0x55ef710161e0 crc :-1 s=READY pgs=272 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:33:41.083 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:41.076+0000 7f9063771740 1 --2- 172.21.15.46:0/2317593907 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ef71003db0 0x55ef710018d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:33:41.083 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:41.076+0000 7f9063771740 1 --2- 172.21.15.46:0/2317593907 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55ef70ffafe0 0x55ef71001390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:33:41.083 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:41.076+0000 7f9041ffb640 0 client.27643 ms_handle_reset on v2:172.21.15.142:6838/2519143918 2024-03-31T23:33:41.083 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:41.076+0000 7f9063771740 1 --2- 172.21.15.46:0/2317593907 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ef70ffa6c0 0x55ef71000e50 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:33:41.083 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:41.076+0000 7f9063771740 1 -- 172.21.15.46:0/2317593907 >> 172.21.15.46:0/2317593907 conn(0x55ef70ff4fe0 msgr2=0x55ef70ff6d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:33:41.083 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:41.076+0000 7f9063771740 1 -- 172.21.15.46:0/2317593907 shutdown_connections 2024-03-31T23:33:41.083 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:41.076+0000 7f9063771740 1 -- 172.21.15.46:0/2317593907 wait complete. 2024-03-31T23:33:41.083 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:41.076+0000 7f9063771740 20 client.27643 trim_cache size 0 max 16384 2024-03-31T23:33:41.102 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:41.100+0000 7f9063771740 1 Processor -- start 2024-03-31T23:33:41.102 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:41.100+0000 7f9063771740 1 -- start start 2024-03-31T23:33:41.102 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:41.100+0000 7f9063771740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55ef70ffa6c0 0x55ef71003580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:33:41.102 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:41.100+0000 7f9063771740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ef70ffafe0 0x55ef70fefe20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:33:41.102 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:41.100+0000 7f9063771740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ef71003db0 0x55ef7100f780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:33:41.102 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:41.100+0000 7f9063771740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f905c1b1400 con 0x55ef71003db0 2024-03-31T23:33:41.102 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:41.100+0000 7f9063771740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f905c1aec40 con 0x55ef70ffa6c0 2024-03-31T23:33:41.102 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:41.100+0000 7f9063771740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f905c1aca90 con 0x55ef70ffafe0 2024-03-31T23:33:41.102 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:41.100+0000 7f905ad9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ef71003db0 0x55ef7100f780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:33:41.103 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:41.100+0000 7f905ad9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ef71003db0 0x55ef7100f780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39424/0 (socket says 172.21.15.46:39424) 2024-03-31T23:33:41.103 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:41.100+0000 7f905a59c640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55ef70ffa6c0 0x55ef71003580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:33:41.103 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:41.100+0000 7f905ad9d640 1 -- 172.21.15.46:0/1131173315 learned_addr learned my addr 172.21.15.46:0/1131173315 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:33:41.103 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:41.100+0000 7f9059d9b640 1 --2- 172.21.15.46:0/1131173315 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ef70ffafe0 0x55ef70fefe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:33:41.103 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:41.100+0000 7f9059d9b640 1 -- 172.21.15.46:0/1131173315 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55ef70ffa6c0 msgr2=0x55ef71003580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:33:41.103 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:41.100+0000 7f9059d9b640 1 --2- 172.21.15.46:0/1131173315 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55ef70ffa6c0 0x55ef71003580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:33:41.103 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:41.100+0000 7f9059d9b640 1 -- 172.21.15.46:0/1131173315 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ef71003db0 msgr2=0x55ef7100f780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:33:41.103 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:41.100+0000 7f9059d9b640 1 --2- 172.21.15.46:0/1131173315 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ef71003db0 0x55ef7100f780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:33:41.103 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:41.100+0000 7f9059d9b640 1 -- 172.21.15.46:0/1131173315 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ef71016510 con 0x55ef70ffafe0 2024-03-31T23:33:41.103 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:41.101+0000 7f905ad9d640 1 --2- 172.21.15.46:0/1131173315 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ef71003db0 0x55ef7100f780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:33:41.103 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:41.101+0000 7f9059d9b640 1 --2- 172.21.15.46:0/1131173315 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ef70ffafe0 0x55ef70fefe20 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f90440ed940 tx=0x7f90440f29a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2342bd37d81ec002 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:33:41.103 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:41.101+0000 7f9043fff640 1 -- 172.21.15.46:0/1131173315 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9044003e20 con 0x55ef70ffafe0 2024-03-31T23:33:41.103 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:41.101+0000 7f9043fff640 1 -- 172.21.15.46:0/1131173315 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f90440bc970 con 0x55ef70ffafe0 2024-03-31T23:33:41.104 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:41.101+0000 7f9043fff640 1 -- 172.21.15.46:0/1131173315 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90440b8fe0 con 0x55ef70ffafe0 2024-03-31T23:33:41.106 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:41.102+0000 7f9063771740 1 -- 172.21.15.46:0/1131173315 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ef70ffafe0 msgr2=0x55ef70fefe20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:33:41.107 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:41.102+0000 7f9063771740 1 --2- 172.21.15.46:0/1131173315 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ef70ffafe0 0x55ef70fefe20 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f90440ed940 tx=0x7f90440f29a0 comp rx=0 tx=0).stop 2024-03-31T23:33:41.107 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:41.102+0000 7f9063771740 1 -- 172.21.15.46:0/1131173315 shutdown_connections 2024-03-31T23:33:41.107 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:41.102+0000 7f9063771740 1 --2- 172.21.15.46:0/1131173315 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ef71003db0 0x55ef7100f780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:33:41.107 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:41.102+0000 7f9063771740 1 --2- 172.21.15.46:0/1131173315 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ef70ffafe0 0x55ef70fefe20 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:33:41.107 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:41.102+0000 7f9063771740 1 --2- 172.21.15.46:0/1131173315 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55ef70ffa6c0 0x55ef71003580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:33:41.107 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:41.102+0000 7f9063771740 1 -- 172.21.15.46:0/1131173315 >> 172.21.15.46:0/1131173315 conn(0x55ef70ff4fe0 msgr2=0x55ef70ffb5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:33:41.107 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:41.102+0000 7f9063771740 1 -- 172.21.15.46:0/1131173315 shutdown_connections 2024-03-31T23:33:41.107 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:41.102+0000 7f9063771740 1 -- 172.21.15.46:0/1131173315 wait complete. 2024-03-31T23:33:41.107 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:41.102+0000 7f9063771740 1 -- start start 2024-03-31T23:33:41.107 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:41.102+0000 7f9063771740 1 Processor -- start 2024-03-31T23:33:41.107 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:41.103+0000 7f9063771740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55ef70ffa6c0 0x55ef7100bdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:33:41.107 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:41.103+0000 7f9063771740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ef70ffafe0 0x55ef710756d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:33:41.108 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:41.103+0000 7f9063771740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ef71003db0 0x55ef71072a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:33:41.108 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:41.103+0000 7f9059d9b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ef70ffafe0 0x55ef710756d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:33:41.108 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:41.103+0000 7f9059d9b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ef70ffafe0 0x55ef710756d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39436/0 (socket says 172.21.15.46:39436) 2024-03-31T23:33:41.108 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:41.103+0000 7f9059d9b640 1 -- 172.21.15.46:0/1251866125 learned_addr learned my addr 172.21.15.46:0/1251866125 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:33:41.108 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:41.103+0000 7f905a59c640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55ef70ffa6c0 0x55ef7100bdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:33:41.108 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:41.103+0000 7f9063771740 1 -- 172.21.15.46:0/1251866125 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f905c1ad100 con 0x55ef70ffafe0 2024-03-31T23:33:41.108 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:41.103+0000 7f9063771740 1 -- 172.21.15.46:0/1251866125 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f905c1ae570 con 0x55ef70ffa6c0 2024-03-31T23:33:41.108 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:41.103+0000 7f9063771740 1 -- 172.21.15.46:0/1251866125 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f905c1ad7d0 con 0x55ef71003db0 2024-03-31T23:33:41.108 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:41.103+0000 7f9063771740 10 client.0 _ll_register_callbacks cb 0x7f9054009910 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T23:33:41.108 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:41.103+0000 7f905ad9d640 1 --2- 172.21.15.46:0/1251866125 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ef71003db0 0x55ef71072a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:33:41.108 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:41.103+0000 7f9059d9b640 1 -- 172.21.15.46:0/1251866125 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ef71003db0 msgr2=0x55ef71072a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:33:41.108 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:41.103+0000 7f9059d9b640 1 --2- 172.21.15.46:0/1251866125 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ef71003db0 0x55ef71072a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:33:41.108 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:41.103+0000 7f9059d9b640 1 -- 172.21.15.46:0/1251866125 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55ef70ffa6c0 msgr2=0x55ef7100bdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:33:41.109 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:41.103+0000 7f9059d9b640 1 --2- 172.21.15.46:0/1251866125 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55ef70ffa6c0 0x55ef7100bdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:33:41.109 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:41.103+0000 7f9059d9b640 1 -- 172.21.15.46:0/1251866125 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ef71075c10 con 0x55ef70ffafe0 2024-03-31T23:33:41.109 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:41.103+0000 7f905a59c640 1 --2- 172.21.15.46:0/1251866125 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55ef70ffa6c0 0x55ef7100bdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:33:41.109 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:41.103+0000 7f9059d9b640 1 --2- 172.21.15.46:0/1251866125 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ef70ffafe0 0x55ef710756d0 secure :-1 s=READY pgs=1005 cs=0 l=1 rev1=1 crypto rx=0x7f90440b8dd0 tx=0x7f9044005900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c6516de77eca9755 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:33:41.109 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:41.103+0000 7f9041ffb640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-31T23:33:41.109 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:41.103+0000 7f9041ffb640 1 -- 172.21.15.46:0/1251866125 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90440c0ac0 con 0x55ef70ffafe0 2024-03-31T23:33:41.109 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:41.104+0000 7f9041ffb640 1 -- 172.21.15.46:0/1251866125 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(131..131 src has 1..131) ==== 9466+0+0 (secure 0 0 0) 0x7f90440d5660 con 0x55ef70ffafe0 2024-03-31T23:33:41.109 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:41.104+0000 7f9063771740 1 -- 172.21.15.46:0/1251866125 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f90440bc970 con 0x55ef70ffafe0 2024-03-31T23:33:41.109 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:41.104+0000 7f9041ffb640 1 -- 172.21.15.46:0/1251866125 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=51) ==== 24+0+0 (secure 0 0 0) 0x7f90440da050 con 0x55ef70ffafe0 2024-03-31T23:33:41.109 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:41.104+0000 7f9063771740 10 client.17928 fetch_fsmap learned FSMap version 51 2024-03-31T23:33:41.109 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:41.104+0000 7f9063771740 1 -- 172.21.15.46:0/1251866125 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=51,monmap=4+}) -- 0x55ef7107e860 con 0x55ef70ffafe0 2024-03-31T23:33:41.110 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:41.104+0000 7f9041ffb640 1 -- 172.21.15.46:0/1251866125 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f90440c5380 con 0x55ef70ffafe0 2024-03-31T23:33:41.110 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:41.104+0000 7f9041ffb640 1 -- 172.21.15.46:0/1251866125 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 51) ==== 8358+0+0 (secure 0 0 0) 0x7f904403c030 con 0x55ef70ffafe0 2024-03-31T23:33:41.110 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:41.104+0000 7f9063771740 10 client.17928 fetch_fsmap finished waiting for FSMap version 51 2024-03-31T23:33:41.110 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:41.104+0000 7f9063771740 10 client.17928 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-31T23:33:41.110 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:41.104+0000 7f9063771740 20 client.17928 populate_metadata read hostname 'smithi046' 2024-03-31T23:33:41.110 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:41.104+0000 7f9063771740 1 --2- 172.21.15.46:0/1251866125 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55ef7107ee60 0x55ef71081280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:33:41.110 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:41.104+0000 7f9063771740 4 client.17928 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-31T23:33:41.110 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:41.104+0000 7f9063771740 1 -- 172.21.15.46:0/1251866125 --> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] -- command(tid 0: {"prefix": "flush journal"}) -- 0x7f90440bc970 con 0x55ef7107ee60 2024-03-31T23:33:41.110 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:41.104+0000 7f905a59c640 1 --2- 172.21.15.46:0/1251866125 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55ef7107ee60 0x55ef71081280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:33:41.110 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:41.105+0000 7f905a59c640 1 --2- 172.21.15.46:0/1251866125 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55ef7107ee60 0x55ef71081280 crc :-1 s=READY pgs=273 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a43b2c0240a2743a server_cookie=1f356f995d8e3089 in_seq=0 out_seq=0 2024-03-31T23:33:41.110 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:41.105+0000 7f9041ffb640 10 client.17928 ms_handle_connect on v2:172.21.15.142:6838/2519143918 2024-03-31T23:33:41.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:33:40 smithi046 ceph-mon[32498]: pgmap v696: 129 pgs: 129 active+clean; 1.1 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 6.9 MiB/s wr, 392 op/s 2024-03-31T23:33:41.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:33:40 smithi142 ceph-mon[35271]: pgmap v696: 129 pgs: 129 active+clean; 1.1 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 6.9 MiB/s wr, 392 op/s 2024-03-31T23:33:41.305 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:41.304+0000 7f9041ffb640 1 -- 172.21.15.46:0/1251866125 <== mds.0 v2:172.21.15.142:6838/2519143918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x7f90440bc970 con 0x55ef7107ee60 2024-03-31T23:33:41.305 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:41.304+0000 7f9041ffb640 10 client.17928 handle_command_reply: tid=0 2024-03-31T23:33:41.305 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:41.304+0000 7f9063771740 1 client.17928 shutdown 2024-03-31T23:33:41.305 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:41.304+0000 7f9063771740 1 -- 172.21.15.46:0/1251866125 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ef70ffafe0 msgr2=0x55ef710756d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:33:41.305 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:41.304+0000 7f9063771740 1 --2- 172.21.15.46:0/1251866125 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ef70ffafe0 0x55ef710756d0 secure :-1 s=READY pgs=1005 cs=0 l=1 rev1=1 crypto rx=0x7f90440b8dd0 tx=0x7f9044005900 comp rx=0 tx=0).stop 2024-03-31T23:33:41.306 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:41.305+0000 7f9063771740 1 -- 172.21.15.46:0/1251866125 shutdown_connections 2024-03-31T23:33:41.306 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:41.305+0000 7f9063771740 1 --2- 172.21.15.46:0/1251866125 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55ef7107ee60 0x55ef71081280 crc :-1 s=READY pgs=273 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:33:41.306 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:41.305+0000 7f9041ffb640 0 client.17928 ms_handle_reset on v2:172.21.15.142:6838/2519143918 2024-03-31T23:33:41.306 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:41.305+0000 7f9063771740 1 --2- 172.21.15.46:0/1251866125 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ef71003db0 0x55ef71072a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:33:41.306 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:41.305+0000 7f9063771740 1 --2- 172.21.15.46:0/1251866125 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ef70ffafe0 0x55ef710756d0 unknown :-1 s=CLOSED pgs=1005 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:33:41.306 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:41.305+0000 7f9063771740 1 --2- 172.21.15.46:0/1251866125 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55ef70ffa6c0 0x55ef7100bdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:33:41.306 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:41.305+0000 7f9063771740 1 -- 172.21.15.46:0/1251866125 >> 172.21.15.46:0/1251866125 conn(0x55ef70ff4fe0 msgr2=0x55ef71000df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:33:41.307 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:41.306+0000 7f9063771740 1 -- 172.21.15.46:0/1251866125 shutdown_connections 2024-03-31T23:33:41.307 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:41.306+0000 7f9063771740 1 -- 172.21.15.46:0/1251866125 wait complete. 2024-03-31T23:33:41.308 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:41.307+0000 7f9063771740 20 client.17928 trim_cache size 0 max 16384 2024-03-31T23:33:41.308 INFO:tasks.workunit.client.0.smithi046.stdout: 10 82092 8.28 MB/sec execute 406 sec latency 200.923 ms 2024-03-31T23:33:41.308 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout:{ 2024-03-31T23:33:41.309 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout: "message": "", 2024-03-31T23:33:41.309 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout: "return_code": 0 2024-03-31T23:33:41.309 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout:} 2024-03-31T23:33:41.309 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:41.308+0000 7f9060b8c640 1 -- 172.21.15.46:0/1481688269 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f90480a26b0 msgr2=0x7f90480a4b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:33:41.310 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:41.308+0000 7f9060b8c640 1 --2- 172.21.15.46:0/1481688269 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f90480a26b0 0x7f90480a4b70 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x55ef70ffa170 tx=0x7f9054005f90 comp rx=0 tx=0).stop 2024-03-31T23:33:41.310 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:41.309+0000 7f9060b8c640 1 -- 172.21.15.46:0/1481688269 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f905c101260 msgr2=0x7f905c198490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:33:41.310 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:41.309+0000 7f9060b8c640 1 --2- 172.21.15.46:0/1481688269 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f905c101260 0x7f905c198490 secure :-1 s=READY pgs=1004 cs=0 l=1 rev1=1 crypto rx=0x7f904402c0b0 tx=0x7f90440045b0 comp rx=0 tx=0).stop 2024-03-31T23:33:41.310 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:41.309+0000 7f9060b8c640 1 -- 172.21.15.46:0/1481688269 shutdown_connections 2024-03-31T23:33:41.310 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:41.309+0000 7f9060b8c640 1 --2- 172.21.15.46:0/1481688269 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f90480a26b0 0x7f90480a4b70 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:33:41.311 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:41.309+0000 7f9060b8c640 1 --2- 172.21.15.46:0/1481688269 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f905c1055f0 0x7f905c19c7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:33:41.311 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:41.309+0000 7f9060b8c640 1 --2- 172.21.15.46:0/1481688269 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f905c101260 0x7f905c198490 unknown :-1 s=CLOSED pgs=1004 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:33:41.311 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:41.309+0000 7f9060b8c640 1 --2- 172.21.15.46:0/1481688269 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f905c1008a0 0x7f905c197f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:33:41.311 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:41.309+0000 7f9060b8c640 1 -- 172.21.15.46:0/1481688269 >> 172.21.15.46:0/1481688269 conn(0x7f905c0774a0 msgr2=0x7f905c104ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:33:41.311 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:41.309+0000 7f9060b8c640 1 -- 172.21.15.46:0/1481688269 shutdown_connections 2024-03-31T23:33:41.311 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:41.310+0000 7f9060b8c640 1 -- 172.21.15.46:0/1481688269 wait complete. 2024-03-31T23:33:42.308 INFO:tasks.workunit.client.0.smithi046.stdout: 10 82246 8.29 MB/sec execute 407 sec latency 89.405 ms 2024-03-31T23:33:43.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:33:42 smithi162 ceph-mon[36237]: pgmap v697: 129 pgs: 129 active+clean; 1.1 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 4.6 MiB/s wr, 209 op/s 2024-03-31T23:33:43.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:33:42 smithi046 ceph-mon[32498]: pgmap v697: 129 pgs: 129 active+clean; 1.1 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 4.6 MiB/s wr, 209 op/s 2024-03-31T23:33:43.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:33:42 smithi142 ceph-mon[35271]: pgmap v697: 129 pgs: 129 active+clean; 1.1 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 4.6 MiB/s wr, 209 op/s 2024-03-31T23:33:43.309 INFO:tasks.workunit.client.0.smithi046.stdout: 10 82404 8.29 MB/sec execute 408 sec latency 138.276 ms 2024-03-31T23:33:44.309 INFO:tasks.workunit.client.0.smithi046.stdout: 10 82545 8.29 MB/sec execute 409 sec latency 101.704 ms 2024-03-31T23:33:45.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:33:44 smithi162 ceph-mon[36237]: pgmap v698: 129 pgs: 129 active+clean; 1.1 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 4.6 MiB/s wr, 209 op/s 2024-03-31T23:33:45.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:33:44 smithi046 ceph-mon[32498]: pgmap v698: 129 pgs: 129 active+clean; 1.1 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 4.6 MiB/s wr, 209 op/s 2024-03-31T23:33:45.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:33:44 smithi142 ceph-mon[35271]: pgmap v698: 129 pgs: 129 active+clean; 1.1 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 4.6 MiB/s wr, 209 op/s 2024-03-31T23:33:45.309 INFO:tasks.workunit.client.0.smithi046.stdout: 10 82697 8.30 MB/sec execute 410 sec latency 105.064 ms 2024-03-31T23:33:46.309 INFO:tasks.workunit.client.0.smithi046.stdout: 10 82846 8.30 MB/sec execute 411 sec latency 80.366 ms 2024-03-31T23:33:47.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:33:46 smithi162 ceph-mon[36237]: pgmap v699: 129 pgs: 129 active+clean; 1.1 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 7.1 MiB/s wr, 370 op/s 2024-03-31T23:33:47.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:33:46 smithi046 ceph-mon[32498]: pgmap v699: 129 pgs: 129 active+clean; 1.1 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 7.1 MiB/s wr, 370 op/s 2024-03-31T23:33:47.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:33:46 smithi142 ceph-mon[35271]: pgmap v699: 129 pgs: 129 active+clean; 1.1 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 7.1 MiB/s wr, 370 op/s 2024-03-31T23:33:47.309 INFO:tasks.workunit.client.0.smithi046.stdout: 10 83004 8.30 MB/sec execute 412 sec latency 87.412 ms 2024-03-31T23:33:48.309 INFO:tasks.workunit.client.0.smithi046.stdout: 10 83127 8.29 MB/sec execute 413 sec latency 90.367 ms 2024-03-31T23:33:49.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:33:48 smithi162 ceph-mon[36237]: pgmap v700: 129 pgs: 129 active+clean; 1.1 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 5.1 MiB/s wr, 262 op/s 2024-03-31T23:33:49.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:33:48 smithi046 ceph-mon[32498]: pgmap v700: 129 pgs: 129 active+clean; 1.1 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 5.1 MiB/s wr, 262 op/s 2024-03-31T23:33:49.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:33:48 smithi142 ceph-mon[35271]: pgmap v700: 129 pgs: 129 active+clean; 1.1 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 5.1 MiB/s wr, 262 op/s 2024-03-31T23:33:49.309 INFO:tasks.workunit.client.0.smithi046.stdout: 10 83269 8.29 MB/sec execute 414 sec latency 65.922 ms 2024-03-31T23:33:50.309 INFO:tasks.workunit.client.0.smithi046.stdout: 10 83409 8.28 MB/sec execute 415 sec latency 139.476 ms 2024-03-31T23:33:51.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:33:50 smithi162 ceph-mon[36237]: pgmap v701: 129 pgs: 129 active+clean; 1.1 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 7.3 MiB/s wr, 384 op/s 2024-03-31T23:33:51.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:33:50 smithi046 ceph-mon[32498]: pgmap v701: 129 pgs: 129 active+clean; 1.1 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 7.3 MiB/s wr, 384 op/s 2024-03-31T23:33:51.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:33:50 smithi142 ceph-mon[35271]: pgmap v701: 129 pgs: 129 active+clean; 1.1 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 7.3 MiB/s wr, 384 op/s 2024-03-31T23:33:51.309 INFO:tasks.workunit.client.0.smithi046.stdout: 10 83552 8.28 MB/sec execute 416 sec latency 105.891 ms 2024-03-31T23:33:52.309 INFO:tasks.workunit.client.0.smithi046.stdout: 10 83707 8.27 MB/sec execute 417 sec latency 93.575 ms 2024-03-31T23:33:53.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:33:52 smithi162 ceph-mon[36237]: pgmap v702: 129 pgs: 129 active+clean; 1.1 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 4.8 MiB/s wr, 282 op/s 2024-03-31T23:33:53.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:33:52 smithi046 ceph-mon[32498]: pgmap v702: 129 pgs: 129 active+clean; 1.1 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 4.8 MiB/s wr, 282 op/s 2024-03-31T23:33:53.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:33:52 smithi142 ceph-mon[35271]: pgmap v702: 129 pgs: 129 active+clean; 1.1 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 4.8 MiB/s wr, 282 op/s 2024-03-31T23:33:53.309 INFO:tasks.workunit.client.0.smithi046.stdout: 10 83853 8.27 MB/sec execute 418 sec latency 74.975 ms 2024-03-31T23:33:54.309 INFO:tasks.workunit.client.0.smithi046.stdout: 10 84006 8.26 MB/sec execute 419 sec latency 99.338 ms 2024-03-31T23:33:54.449 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:54.448+0000 7f6807e72640 1 -- 172.21.15.46:0/2422221690 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6800101740 msgr2=0x7f6800101bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:33:54.449 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:54.448+0000 7f6807e72640 1 --2- 172.21.15.46:0/2422221690 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6800101740 0x7f6800101bc0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f67f0009860 tx=0x7f67f002bb70 comp rx=0 tx=0).stop 2024-03-31T23:33:54.449 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:54.448+0000 7f6807e72640 1 -- 172.21.15.46:0/2422221690 shutdown_connections 2024-03-31T23:33:54.449 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:54.448+0000 7f6807e72640 1 --2- 172.21.15.46:0/2422221690 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6800102100 0x7f680010e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:33:54.449 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:54.448+0000 7f6807e72640 1 --2- 172.21.15.46:0/2422221690 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6800101740 0x7f6800101bc0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:33:54.450 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:54.448+0000 7f6807e72640 1 --2- 172.21.15.46:0/2422221690 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6800107750 0x7f6800107b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:33:54.450 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:54.448+0000 7f6807e72640 1 -- 172.21.15.46:0/2422221690 >> 172.21.15.46:0/2422221690 conn(0x7f68000fd210 msgr2=0x7f68000ff630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:33:54.450 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:54.448+0000 7f6807e72640 1 -- 172.21.15.46:0/2422221690 shutdown_connections 2024-03-31T23:33:54.450 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:54.448+0000 7f6807e72640 1 -- 172.21.15.46:0/2422221690 wait complete. 2024-03-31T23:33:54.452 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:54.449+0000 7f6807e72640 1 Processor -- start 2024-03-31T23:33:54.452 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:54.449+0000 7f6807e72640 1 -- start start 2024-03-31T23:33:54.452 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:54.449+0000 7f6807e72640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6800101740 0x7f680019a980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:33:54.452 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:54.449+0000 7f6807e72640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6800102100 0x7f680019aec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:33:54.452 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:54.449+0000 7f6807e72640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6800107750 0x7f680019e960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:33:54.452 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:54.449+0000 7f6807e72640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f6800073d10 con 0x7f6800107750 2024-03-31T23:33:54.453 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:54.449+0000 7f6807e72640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f6800073b90 con 0x7f6800101740 2024-03-31T23:33:54.453 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:54.449+0000 7f6807e72640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f6800073e90 con 0x7f6800102100 2024-03-31T23:33:54.453 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:54.449+0000 7f680640f640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6800107750 0x7f680019e960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:33:54.453 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:54.449+0000 7f680640f640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6800107750 0x7f680019e960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:56742/0 (socket says 172.21.15.46:56742) 2024-03-31T23:33:54.453 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:54.449+0000 7f680640f640 1 -- 172.21.15.46:0/3647210548 learned_addr learned my addr 172.21.15.46:0/3647210548 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:33:54.453 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:54.449+0000 7f6805c0e640 1 --2- 172.21.15.46:0/3647210548 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6800101740 0x7f680019a980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:33:54.453 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:54.449+0000 7f680540d640 1 --2- 172.21.15.46:0/3647210548 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6800102100 0x7f680019aec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:33:54.453 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:54.450+0000 7f680640f640 1 -- 172.21.15.46:0/3647210548 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6800102100 msgr2=0x7f680019aec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:33:54.453 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:54.450+0000 7f680640f640 1 --2- 172.21.15.46:0/3647210548 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6800102100 0x7f680019aec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:33:54.453 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:54.450+0000 7f680640f640 1 -- 172.21.15.46:0/3647210548 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6800101740 msgr2=0x7f680019a980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:33:54.453 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:54.450+0000 7f680640f640 1 --2- 172.21.15.46:0/3647210548 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6800101740 0x7f680019a980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:33:54.453 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:54.450+0000 7f680640f640 1 -- 172.21.15.46:0/3647210548 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f680019f090 con 0x7f6800107750 2024-03-31T23:33:54.453 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:54.450+0000 7f6805c0e640 1 --2- 172.21.15.46:0/3647210548 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6800101740 0x7f680019a980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:33:54.453 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:54.450+0000 7f680540d640 1 --2- 172.21.15.46:0/3647210548 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6800102100 0x7f680019aec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:33:54.453 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:54.450+0000 7f680640f640 1 --2- 172.21.15.46:0/3647210548 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6800107750 0x7f680019e960 secure :-1 s=READY pgs=1006 cs=0 l=1 rev1=1 crypto rx=0x7f67f800efc0 tx=0x7f67f800c800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:33:54.454 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:54.450+0000 7f67f6ffd640 1 -- 172.21.15.46:0/3647210548 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67f8003970 con 0x7f6800107750 2024-03-31T23:33:54.454 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:54.450+0000 7f6807e72640 1 -- 172.21.15.46:0/3647210548 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f680019f380 con 0x7f6800107750 2024-03-31T23:33:54.454 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:54.450+0000 7f6807e72640 1 -- 172.21.15.46:0/3647210548 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f68001a6c10 con 0x7f6800107750 2024-03-31T23:33:54.454 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:54.450+0000 7f67f6ffd640 1 -- 172.21.15.46:0/3647210548 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f67f8004020 con 0x7f6800107750 2024-03-31T23:33:54.454 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:54.450+0000 7f67f6ffd640 1 -- 172.21.15.46:0/3647210548 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67f800b940 con 0x7f6800107750 2024-03-31T23:33:54.456 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:54.453+0000 7f680aa4d740 1 Processor -- start 2024-03-31T23:33:54.456 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:54.453+0000 7f67f6ffd640 1 -- 172.21.15.46:0/3647210548 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 25) ==== 137451+0+0 (secure 0 0 0) 0x7f67f801d020 con 0x7f6800107750 2024-03-31T23:33:54.456 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:54.453+0000 7f680aa4d740 1 -- start start 2024-03-31T23:33:54.456 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:54.453+0000 7f680aa4d740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55580cfbd6c0 0x55580cfbdaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:33:54.456 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:54.453+0000 7f680aa4d740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55580cfbdfe0 0x55580cfc6870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:33:54.456 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:54.454+0000 7f680aa4d740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55580cfc6db0 0x55580cfc91a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:33:54.457 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:54.454+0000 7f680aa4d740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55580cf208f0 con 0x55580cfbd6c0 2024-03-31T23:33:54.457 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:54.454+0000 7f680aa4d740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55580cec98d0 con 0x55580cfbdfe0 2024-03-31T23:33:54.457 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:54.454+0000 7f680aa4d740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55580cf1c990 con 0x55580cfc6db0 2024-03-31T23:33:54.457 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:54.454+0000 7f680540d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55580cfbd6c0 0x55580cfbdaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:33:54.457 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:54.454+0000 7f6805c0e640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55580cfbdfe0 0x55580cfc6870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:33:54.457 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:54.454+0000 7f6805c0e640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55580cfbdfe0 0x55580cfc6870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.46:46134/0 (socket says 172.21.15.46:46134) 2024-03-31T23:33:54.457 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:54.454+0000 7f680540d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55580cfbd6c0 0x55580cfbdaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:56756/0 (socket says 172.21.15.46:56756) 2024-03-31T23:33:54.457 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:54.454+0000 7f6805c0e640 1 -- 172.21.15.46:0/2499075328 learned_addr learned my addr 172.21.15.46:0/2499075328 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:33:54.457 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:54.454+0000 7f67f6ffd640 1 --2- 172.21.15.46:0/3647210548 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f67ec0a2730 0x7f67ec0a4bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:33:54.457 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:54.454+0000 7f67f6ffd640 1 -- 172.21.15.46:0/3647210548 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(131..131 src has 1..131) ==== 9466+0+0 (secure 0 0 0) 0x7f67f80ceb60 con 0x7f6800107750 2024-03-31T23:33:54.457 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:54.454+0000 7f680640f640 1 --2- 172.21.15.46:0/3647210548 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f67ec0a2730 0x7f67ec0a4bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:33:54.457 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:54.454+0000 7f680540d640 1 -- 172.21.15.46:0/2499075328 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55580cfc6db0 msgr2=0x55580cfc91a0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-31T23:33:54.457 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:54.454+0000 7f680540d640 1 --2- 172.21.15.46:0/2499075328 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55580cfc6db0 0x55580cfc91a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:33:54.458 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:54.454+0000 7f680540d640 1 -- 172.21.15.46:0/2499075328 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55580cfbdfe0 msgr2=0x55580cfc6870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:33:54.458 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:54.454+0000 7f680540d640 1 --2- 172.21.15.46:0/2499075328 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55580cfbdfe0 0x55580cfc6870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:33:54.458 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:54.454+0000 7f680540d640 1 -- 172.21.15.46:0/2499075328 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55580cc35210 con 0x55580cfbd6c0 2024-03-31T23:33:54.458 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:54.454+0000 7f6805c0e640 1 --2- 172.21.15.46:0/2499075328 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55580cfbdfe0 0x55580cfc6870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:33:54.458 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:54.455+0000 7f680540d640 1 --2- 172.21.15.46:0/2499075328 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55580cfbd6c0 0x55580cfbdaa0 secure :-1 s=READY pgs=1007 cs=0 l=1 rev1=1 crypto rx=0x7f67f0005940 tx=0x7f67f00347f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff1c7ff6d3bc3a37 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:33:54.458 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:54.455+0000 7f680640f640 1 --2- 172.21.15.46:0/3647210548 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f67ec0a2730 0x7f67ec0a4bf0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f67f809aec0 tx=0x7f67f800ceb0 comp rx=0 tx=0).ready entity=mgr.15654 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:33:54.458 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:54.455+0000 7f67ebfff640 1 -- 172.21.15.46:0/2499075328 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67f0039040 con 0x55580cfbd6c0 2024-03-31T23:33:54.458 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:54.455+0000 7f67ebfff640 1 -- 172.21.15.46:0/2499075328 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f67f00044d0 con 0x55580cfbd6c0 2024-03-31T23:33:54.458 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:54.455+0000 7f67ebfff640 1 -- 172.21.15.46:0/2499075328 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67f002cb50 con 0x55580cfbd6c0 2024-03-31T23:33:54.461 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:54.456+0000 7f680aa4d740 1 -- 172.21.15.46:0/2499075328 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55580cfbd6c0 msgr2=0x55580cfbdaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:33:54.461 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:54.456+0000 7f680aa4d740 1 --2- 172.21.15.46:0/2499075328 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55580cfbd6c0 0x55580cfbdaa0 secure :-1 s=READY pgs=1007 cs=0 l=1 rev1=1 crypto rx=0x7f67f0005940 tx=0x7f67f00347f0 comp rx=0 tx=0).stop 2024-03-31T23:33:54.462 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:54.456+0000 7f680aa4d740 1 -- 172.21.15.46:0/2499075328 shutdown_connections 2024-03-31T23:33:54.462 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:54.456+0000 7f680aa4d740 1 --2- 172.21.15.46:0/2499075328 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55580cfc6db0 0x55580cfc91a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:33:54.462 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:54.456+0000 7f680aa4d740 1 --2- 172.21.15.46:0/2499075328 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55580cfbdfe0 0x55580cfc6870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:33:54.462 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:54.456+0000 7f680aa4d740 1 --2- 172.21.15.46:0/2499075328 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55580cfbd6c0 0x55580cfbdaa0 unknown :-1 s=CLOSED pgs=1007 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:33:54.462 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:54.456+0000 7f680aa4d740 1 -- 172.21.15.46:0/2499075328 >> 172.21.15.46:0/2499075328 conn(0x55580cfb7fe0 msgr2=0x55580cfb83c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:33:54.462 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:54.456+0000 7f680aa4d740 1 -- 172.21.15.46:0/2499075328 shutdown_connections 2024-03-31T23:33:54.462 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:54.456+0000 7f680aa4d740 1 -- 172.21.15.46:0/2499075328 wait complete. 2024-03-31T23:33:54.462 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:54.457+0000 7f680aa4d740 1 -- start start 2024-03-31T23:33:54.462 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:54.457+0000 7f680aa4d740 1 Processor -- start 2024-03-31T23:33:54.462 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:54.457+0000 7f680aa4d740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55580cfbd6c0 0x55580cfc3de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:33:54.462 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:54.458+0000 7f680aa4d740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55580cfbdfe0 0x55580cfc4320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:33:54.462 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:54.458+0000 7f680aa4d740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55580cfc6db0 0x55580cfc4860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:33:54.463 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:54.458+0000 7f680aa4d740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55580cfc9980 con 0x55580cfbdfe0 2024-03-31T23:33:54.463 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:54.458+0000 7f680aa4d740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55580cebef80 con 0x55580cfc6db0 2024-03-31T23:33:54.463 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:54.458+0000 7f680aa4d740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55580cfc9b00 con 0x55580cfbd6c0 2024-03-31T23:33:54.463 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:54.458+0000 7f6805c0e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55580cfbdfe0 0x55580cfc4320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:33:54.463 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:54.458+0000 7f680aa4d740 10 client.0 _ll_register_callbacks cb 0x55580cf17290 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T23:33:54.463 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:54.458+0000 7f680540d640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55580cfbd6c0 0x55580cfc3de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:33:54.463 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:54.458+0000 7f680540d640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55580cfbd6c0 0x55580cfc3de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.46:49408/0 (socket says 172.21.15.46:49408) 2024-03-31T23:33:54.463 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:54.458+0000 7f680540d640 1 -- 172.21.15.46:0/3259743163 learned_addr learned my addr 172.21.15.46:0/3259743163 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:33:54.463 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:54.458+0000 7f680540d640 1 --2- 172.21.15.46:0/3259743163 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55580cfc6db0 0x55580cfc4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:33:54.463 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:54.458+0000 7f680540d640 1 -- 172.21.15.46:0/3259743163 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55580cfc6db0 msgr2=0x55580cfc4860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:33:54.463 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:54.458+0000 7f680540d640 1 --2- 172.21.15.46:0/3259743163 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55580cfc6db0 0x55580cfc4860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:33:54.463 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:54.458+0000 7f680540d640 1 -- 172.21.15.46:0/3259743163 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55580cfbdfe0 msgr2=0x55580cfc4320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:33:54.463 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:54.458+0000 7f680540d640 1 --2- 172.21.15.46:0/3259743163 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55580cfbdfe0 0x55580cfc4320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:33:54.463 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:54.458+0000 7f680540d640 1 -- 172.21.15.46:0/3259743163 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55580cd01620 con 0x55580cfbd6c0 2024-03-31T23:33:54.464 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:54.459+0000 7f680540d640 1 --2- 172.21.15.46:0/3259743163 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55580cfbd6c0 0x55580cfc3de0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f67f0005e00 tx=0x7f67f0005d20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bcee2edbfed6ee94 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:33:54.464 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:54.459+0000 7f67e97fa640 10 client.0 ms_handle_connect on v2:172.21.15.162:3300/0 2024-03-31T23:33:54.464 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:54.459+0000 7f67e97fa640 1 -- 172.21.15.46:0/3259743163 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67f0039040 con 0x55580cfbd6c0 2024-03-31T23:33:54.464 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:54.459+0000 7f67e97fa640 1 -- 172.21.15.46:0/3259743163 <== mon.1 v2:172.21.15.162:3300/0 2 ==== osd_map(131..131 src has 1..131) ==== 9466+0+0 (secure 0 0 0) 0x7f67f0034bb0 con 0x55580cfbd6c0 2024-03-31T23:33:54.464 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:54.459+0000 7f680aa4d740 1 -- 172.21.15.46:0/3259743163 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f67f00044d0 con 0x55580cfbd6c0 2024-03-31T23:33:54.464 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:54.459+0000 7f67e97fa640 1 -- 172.21.15.46:0/3259743163 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_get_version_reply(handle=1 version=51) ==== 24+0+0 (secure 0 0 0) 0x7f67f00035f0 con 0x55580cfbd6c0 2024-03-31T23:33:54.464 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:54.459+0000 7f680aa4d740 10 client.27658 fetch_fsmap learned FSMap version 51 2024-03-31T23:33:54.464 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:54.459+0000 7f680aa4d740 1 -- 172.21.15.46:0/3259743163 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,fsmap=51,monmap=4+}) -- 0x55580cfc9980 con 0x55580cfbd6c0 2024-03-31T23:33:54.464 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:54.459+0000 7f67e97fa640 1 -- 172.21.15.46:0/3259743163 <== mon.1 v2:172.21.15.162:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f67f00044d0 con 0x55580cfbd6c0 2024-03-31T23:33:54.464 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:54.460+0000 7f67e97fa640 1 -- 172.21.15.46:0/3259743163 <== mon.1 v2:172.21.15.162:3300/0 5 ==== fsmap(e 51) ==== 8358+0+0 (secure 0 0 0) 0x7f67f003f740 con 0x55580cfbd6c0 2024-03-31T23:33:54.464 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:54.460+0000 7f680aa4d740 10 client.27658 fetch_fsmap finished waiting for FSMap version 51 2024-03-31T23:33:54.464 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:54.460+0000 7f680aa4d740 10 client.27658 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-31T23:33:54.464 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:54.460+0000 7f680aa4d740 20 client.27658 populate_metadata read hostname 'smithi046' 2024-03-31T23:33:54.465 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:54.460+0000 7f680aa4d740 1 --2- 172.21.15.46:0/3259743163 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55580cfd6a10 0x55580cfd8e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:33:54.465 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:54.460+0000 7f680aa4d740 4 client.27658 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-31T23:33:54.465 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:54.460+0000 7f680aa4d740 1 -- 172.21.15.46:0/3259743163 --> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55580cf34120 con 0x55580cfd6a10 2024-03-31T23:33:54.465 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:54.460+0000 7f6805c0e640 1 --2- 172.21.15.46:0/3259743163 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55580cfd6a10 0x55580cfd8e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:33:54.465 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:54.461+0000 7f67e97fa640 10 client.27658 ms_handle_connect on v2:172.21.15.142:6838/2519143918 2024-03-31T23:33:54.465 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:54.461+0000 7f6805c0e640 1 --2- 172.21.15.46:0/3259743163 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55580cfd6a10 0x55580cfd8e00 crc :-1 s=READY pgs=274 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2f3c16b166fc178f server_cookie=f9f1edc08c9c0da9 in_seq=0 out_seq=0 2024-03-31T23:33:54.466 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:54.462+0000 7f67e97fa640 1 -- 172.21.15.46:0/3259743163 <== mds.0 v2:172.21.15.142:6838/2519143918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55580cf34120 con 0x55580cfd6a10 2024-03-31T23:33:54.466 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:54.462+0000 7f67e97fa640 10 client.27658 handle_command_reply: tid=0 2024-03-31T23:33:54.466 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:54.462+0000 7f680aa4d740 1 client.27658 shutdown 2024-03-31T23:33:54.466 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:54.462+0000 7f680aa4d740 1 -- 172.21.15.46:0/3259743163 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55580cfbd6c0 msgr2=0x55580cfc3de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:33:54.466 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:54.462+0000 7f680aa4d740 1 --2- 172.21.15.46:0/3259743163 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55580cfbd6c0 0x55580cfc3de0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f67f0005e00 tx=0x7f67f0005d20 comp rx=0 tx=0).stop 2024-03-31T23:33:54.466 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:54.462+0000 7f680aa4d740 1 -- 172.21.15.46:0/3259743163 shutdown_connections 2024-03-31T23:33:54.466 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:54.462+0000 7f680aa4d740 1 --2- 172.21.15.46:0/3259743163 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55580cfd6a10 0x55580cfd8e00 crc :-1 s=READY pgs=274 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:33:54.466 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:54.462+0000 7f680aa4d740 1 --2- 172.21.15.46:0/3259743163 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55580cfc6db0 0x55580cfc4860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:33:54.467 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:54.462+0000 7f680aa4d740 1 --2- 172.21.15.46:0/3259743163 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55580cfbdfe0 0x55580cfc4320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:33:54.467 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:54.462+0000 7f680aa4d740 1 --2- 172.21.15.46:0/3259743163 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55580cfbd6c0 0x55580cfc3de0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:33:54.467 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:54.462+0000 7f67e97fa640 0 client.27658 ms_handle_reset on v2:172.21.15.142:6838/2519143918 2024-03-31T23:33:54.467 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:54.462+0000 7f680aa4d740 1 -- 172.21.15.46:0/3259743163 >> 172.21.15.46:0/3259743163 conn(0x55580cfb7fe0 msgr2=0x55580cfb9d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:33:54.467 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:54.462+0000 7f680aa4d740 1 -- 172.21.15.46:0/3259743163 shutdown_connections 2024-03-31T23:33:54.467 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:54.462+0000 7f680aa4d740 1 -- 172.21.15.46:0/3259743163 wait complete. 2024-03-31T23:33:54.467 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:54.462+0000 7f680aa4d740 20 client.27658 trim_cache size 0 max 16384 2024-03-31T23:33:54.489 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:54.488+0000 7f680aa4d740 1 Processor -- start 2024-03-31T23:33:54.489 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:54.488+0000 7f680aa4d740 1 -- start start 2024-03-31T23:33:54.489 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:54.488+0000 7f680aa4d740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55580cfbd6c0 0x55580cfc6510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:33:54.489 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:54.488+0000 7f680aa4d740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55580cfbdfe0 0x55580cfd2720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:33:54.489 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:54.488+0000 7f680aa4d740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55580cfc6db0 0x55580cfde590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:33:54.489 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:54.488+0000 7f680aa4d740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f67e40128f0 con 0x55580cfbd6c0 2024-03-31T23:33:54.489 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:54.488+0000 7f680aa4d740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f67e4010130 con 0x55580cfc6db0 2024-03-31T23:33:54.489 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:54.488+0000 7f680aa4d740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f67e400df80 con 0x55580cfbdfe0 2024-03-31T23:33:54.490 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:54.488+0000 7f680540d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55580cfbd6c0 0x55580cfc6510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:33:54.491 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:54.488+0000 7f680540d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55580cfbd6c0 0x55580cfc6510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:56778/0 (socket says 172.21.15.46:56778) 2024-03-31T23:33:54.491 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:54.488+0000 7f680540d640 1 -- 172.21.15.46:0/3473752126 learned_addr learned my addr 172.21.15.46:0/3473752126 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:33:54.491 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:54.489+0000 7f6805c0e640 1 --2- 172.21.15.46:0/3473752126 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55580cfbdfe0 0x55580cfd2720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:33:54.491 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:54.489+0000 7f680540d640 1 --2- 172.21.15.46:0/3473752126 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55580cfc6db0 0x55580cfde590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:33:54.491 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:54.489+0000 7f680540d640 1 -- 172.21.15.46:0/3473752126 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55580cfbdfe0 msgr2=0x55580cfd2720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:33:54.491 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:54.489+0000 7f680540d640 1 --2- 172.21.15.46:0/3473752126 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55580cfbdfe0 0x55580cfd2720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:33:54.491 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:54.489+0000 7f680540d640 1 -- 172.21.15.46:0/3473752126 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55580cfc6db0 msgr2=0x55580cfde590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:33:54.491 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:54.489+0000 7f680540d640 1 --2- 172.21.15.46:0/3473752126 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55580cfc6db0 0x55580cfde590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:33:54.491 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:54.489+0000 7f680540d640 1 -- 172.21.15.46:0/3473752126 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55580cfc2330 con 0x55580cfbd6c0 2024-03-31T23:33:54.491 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:54.489+0000 7f680540d640 1 --2- 172.21.15.46:0/3473752126 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55580cfbd6c0 0x55580cfc6510 secure :-1 s=READY pgs=1008 cs=0 l=1 rev1=1 crypto rx=0x7f67f002cb30 tx=0x7f67f002bb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=709d975531fff954 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:33:54.491 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:54.489+0000 7f67eb7fe640 1 -- 172.21.15.46:0/3473752126 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67f0039070 con 0x55580cfbd6c0 2024-03-31T23:33:54.491 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:54.490+0000 7f67eb7fe640 1 -- 172.21.15.46:0/3473752126 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f67f004d070 con 0x55580cfbd6c0 2024-03-31T23:33:54.491 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:54.490+0000 7f67eb7fe640 1 -- 172.21.15.46:0/3473752126 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67f0034c10 con 0x55580cfbd6c0 2024-03-31T23:33:54.493 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:54.491+0000 7f680aa4d740 1 -- 172.21.15.46:0/3473752126 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55580cfbd6c0 msgr2=0x55580cfc6510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:33:54.493 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:54.491+0000 7f680aa4d740 1 --2- 172.21.15.46:0/3473752126 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55580cfbd6c0 0x55580cfc6510 secure :-1 s=READY pgs=1008 cs=0 l=1 rev1=1 crypto rx=0x7f67f002cb30 tx=0x7f67f002bb70 comp rx=0 tx=0).stop 2024-03-31T23:33:54.493 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:54.491+0000 7f680aa4d740 1 -- 172.21.15.46:0/3473752126 shutdown_connections 2024-03-31T23:33:54.493 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:54.491+0000 7f680aa4d740 1 --2- 172.21.15.46:0/3473752126 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55580cfc6db0 0x55580cfde590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:33:54.493 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:54.491+0000 7f680aa4d740 1 --2- 172.21.15.46:0/3473752126 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55580cfbdfe0 0x55580cfd2720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:33:54.493 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:54.491+0000 7f680aa4d740 1 --2- 172.21.15.46:0/3473752126 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55580cfbd6c0 0x55580cfc6510 unknown :-1 s=CLOSED pgs=1008 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:33:54.493 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:54.491+0000 7f680aa4d740 1 -- 172.21.15.46:0/3473752126 >> 172.21.15.46:0/3473752126 conn(0x55580cfb7fe0 msgr2=0x55580cfbf170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:33:54.493 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:54.491+0000 7f680aa4d740 1 -- 172.21.15.46:0/3473752126 shutdown_connections 2024-03-31T23:33:54.493 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:54.491+0000 7f680aa4d740 1 -- 172.21.15.46:0/3473752126 wait complete. 2024-03-31T23:33:54.495 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:54.491+0000 7f680aa4d740 1 -- start start 2024-03-31T23:33:54.495 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:54.492+0000 7f680aa4d740 1 Processor -- start 2024-03-31T23:33:54.495 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:54.492+0000 7f680aa4d740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55580cfbd6c0 0x55580cfca710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:33:54.495 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:54.492+0000 7f680aa4d740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55580cfbdfe0 0x55580cfcac50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:33:54.495 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:54.492+0000 7f680540d640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55580cfbd6c0 0x55580cfca710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:33:54.495 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:54.492+0000 7f6805c0e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55580cfbdfe0 0x55580cfcac50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:33:54.495 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:54.492+0000 7f680540d640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55580cfbd6c0 0x55580cfca710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.46:46154/0 (socket says 172.21.15.46:46154) 2024-03-31T23:33:54.495 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:54.492+0000 7f6805c0e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55580cfbdfe0 0x55580cfcac50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:56782/0 (socket says 172.21.15.46:56782) 2024-03-31T23:33:54.495 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:54.492+0000 7f680540d640 1 -- 172.21.15.46:0/2844659477 learned_addr learned my addr 172.21.15.46:0/2844659477 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:33:54.496 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:54.492+0000 7f680aa4d740 1 --2- 172.21.15.46:0/2844659477 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55580cfc6db0 0x55580cfcb190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:33:54.496 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:54.493+0000 7f680aa4d740 1 -- 172.21.15.46:0/2844659477 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f67e400e5f0 con 0x55580cfbdfe0 2024-03-31T23:33:54.496 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:54.493+0000 7f680aa4d740 1 -- 172.21.15.46:0/2844659477 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f67e400fa60 con 0x55580cfbd6c0 2024-03-31T23:33:54.496 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:54.493+0000 7f680aa4d740 1 -- 172.21.15.46:0/2844659477 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f67e400ecc0 con 0x55580cfc6db0 2024-03-31T23:33:54.496 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:54.493+0000 7f6805c0e640 1 -- 172.21.15.46:0/2844659477 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55580cfc6db0 msgr2=0x55580cfcb190 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-31T23:33:54.496 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:54.493+0000 7f6805c0e640 1 --2- 172.21.15.46:0/2844659477 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55580cfc6db0 0x55580cfcb190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:33:54.496 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:54.493+0000 7f6805c0e640 1 -- 172.21.15.46:0/2844659477 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55580cfbd6c0 msgr2=0x55580cfca710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:33:54.496 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:54.493+0000 7f6805c0e640 1 --2- 172.21.15.46:0/2844659477 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55580cfbd6c0 0x55580cfca710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:33:54.496 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:54.493+0000 7f6805c0e640 1 -- 172.21.15.46:0/2844659477 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55580d036c40 con 0x55580cfbdfe0 2024-03-31T23:33:54.496 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:54.493+0000 7f680aa4d740 10 client.0 _ll_register_callbacks cb 0x7f67fc009b50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T23:33:54.496 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:54.493+0000 7f680aa4d740 1 -- 172.21.15.46:0/2844659477 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f67f004d070 con 0x55580cfbdfe0 2024-03-31T23:33:54.496 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:54.494+0000 7f6805c0e640 1 --2- 172.21.15.46:0/2844659477 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55580cfbdfe0 0x55580cfcac50 secure :-1 s=READY pgs=1009 cs=0 l=1 rev1=1 crypto rx=0x55580cfd94d0 tx=0x7f67fc01e700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=691bcd4c9c1931ac server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:33:54.496 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:54.494+0000 7f67e97fa640 10 client.17946 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-31T23:33:54.496 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:54.494+0000 7f67e97fa640 1 -- 172.21.15.46:0/2844659477 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67fc044020 con 0x55580cfbdfe0 2024-03-31T23:33:54.496 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:54.494+0000 7f67e97fa640 1 -- 172.21.15.46:0/2844659477 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(131..131 src has 1..131) ==== 9466+0+0 (secure 0 0 0) 0x7f67fc040db0 con 0x55580cfbdfe0 2024-03-31T23:33:54.497 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:54.494+0000 7f67e97fa640 1 -- 172.21.15.46:0/2844659477 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=51) ==== 24+0+0 (secure 0 0 0) 0x7f67fc0044c0 con 0x55580cfbdfe0 2024-03-31T23:33:54.497 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:54.494+0000 7f680aa4d740 10 client.17946 fetch_fsmap learned FSMap version 51 2024-03-31T23:33:54.497 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:54.494+0000 7f680aa4d740 1 -- 172.21.15.46:0/2844659477 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({fsmap=51}) -- 0x55580d037d30 con 0x55580cfbdfe0 2024-03-31T23:33:54.497 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:54.495+0000 7f67e97fa640 1 -- 172.21.15.46:0/2844659477 <== mon.0 v2:172.21.15.46:3300/0 4 ==== fsmap(e 51) ==== 8358+0+0 (secure 0 0 0) 0x7f67fc010b20 con 0x55580cfbdfe0 2024-03-31T23:33:54.497 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:54.495+0000 7f680aa4d740 10 client.17946 fetch_fsmap finished waiting for FSMap version 51 2024-03-31T23:33:54.498 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:54.495+0000 7f680aa4d740 10 client.17946 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-31T23:33:54.498 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:54.495+0000 7f680aa4d740 20 client.17946 populate_metadata read hostname 'smithi046' 2024-03-31T23:33:54.498 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:54.495+0000 7f680aa4d740 1 --2- 172.21.15.46:0/2844659477 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55580d0383e0 0x55580d03a870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:33:54.498 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:54.495+0000 7f680aa4d740 4 client.17946 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-31T23:33:54.498 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:54.495+0000 7f680aa4d740 1 -- 172.21.15.46:0/2844659477 --> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55580cf2de30 con 0x55580d0383e0 2024-03-31T23:33:54.498 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:54.495+0000 7f680540d640 1 --2- 172.21.15.46:0/2844659477 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55580d0383e0 0x55580d03a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:33:54.498 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:54.495+0000 7f680540d640 1 --2- 172.21.15.46:0/2844659477 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55580d0383e0 0x55580d03a870 crc :-1 s=READY pgs=275 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2be85e0d994cc8bd server_cookie=9f39b08d8c0821eb in_seq=0 out_seq=0 2024-03-31T23:33:54.498 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:54.495+0000 7f67e97fa640 10 client.17946 ms_handle_connect on v2:172.21.15.142:6838/2519143918 2024-03-31T23:33:54.696 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:54.694+0000 7f67e97fa640 1 -- 172.21.15.46:0/2844659477 <== mds.0 v2:172.21.15.142:6838/2519143918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55580cf2de30 con 0x55580d0383e0 2024-03-31T23:33:54.696 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:54.694+0000 7f67e97fa640 10 client.17946 handle_command_reply: tid=0 2024-03-31T23:33:54.696 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:54.694+0000 7f680aa4d740 1 client.17946 shutdown 2024-03-31T23:33:54.696 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:54.695+0000 7f680aa4d740 1 -- 172.21.15.46:0/2844659477 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55580cfbdfe0 msgr2=0x55580cfcac50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:33:54.697 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:54.695+0000 7f680aa4d740 1 --2- 172.21.15.46:0/2844659477 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55580cfbdfe0 0x55580cfcac50 secure :-1 s=READY pgs=1009 cs=0 l=1 rev1=1 crypto rx=0x55580cfd94d0 tx=0x7f67fc01e700 comp rx=0 tx=0).stop 2024-03-31T23:33:54.697 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:54.695+0000 7f680aa4d740 1 -- 172.21.15.46:0/2844659477 shutdown_connections 2024-03-31T23:33:54.697 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:54.695+0000 7f680aa4d740 1 --2- 172.21.15.46:0/2844659477 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55580d0383e0 0x55580d03a870 crc :-1 s=READY pgs=275 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:33:54.697 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:54.695+0000 7f680aa4d740 1 --2- 172.21.15.46:0/2844659477 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55580cfc6db0 0x55580cfcb190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:33:54.697 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:54.695+0000 7f680aa4d740 1 --2- 172.21.15.46:0/2844659477 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55580cfbdfe0 0x55580cfcac50 unknown :-1 s=CLOSED pgs=1009 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:33:54.697 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:54.695+0000 7f680aa4d740 1 --2- 172.21.15.46:0/2844659477 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55580cfbd6c0 0x55580cfca710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:33:54.697 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:54.695+0000 7f680aa4d740 1 -- 172.21.15.46:0/2844659477 >> 172.21.15.46:0/2844659477 conn(0x55580cfb7fe0 msgr2=0x55580cfb66f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:33:54.697 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:54.695+0000 7f67e97fa640 0 client.17946 ms_handle_reset on v2:172.21.15.142:6838/2519143918 2024-03-31T23:33:54.697 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:54.695+0000 7f680aa4d740 1 -- 172.21.15.46:0/2844659477 shutdown_connections 2024-03-31T23:33:54.697 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:54.695+0000 7f680aa4d740 1 -- 172.21.15.46:0/2844659477 wait complete. 2024-03-31T23:33:54.697 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:54.695+0000 7f680aa4d740 20 client.17946 trim_cache size 0 max 16384 2024-03-31T23:33:54.698 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout:{ 2024-03-31T23:33:54.698 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout: "message": "", 2024-03-31T23:33:54.698 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout: "return_code": 0 2024-03-31T23:33:54.698 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout:} 2024-03-31T23:33:54.698 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:54.696+0000 7f6807e72640 1 -- 172.21.15.46:0/3647210548 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f67ec0a2730 msgr2=0x7f67ec0a4bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:33:54.699 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:54.696+0000 7f6807e72640 1 --2- 172.21.15.46:0/3647210548 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f67ec0a2730 0x7f67ec0a4bf0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f67f809aec0 tx=0x7f67f800ceb0 comp rx=0 tx=0).stop 2024-03-31T23:33:54.699 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:54.696+0000 7f6807e72640 1 -- 172.21.15.46:0/3647210548 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6800107750 msgr2=0x7f680019e960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:33:54.699 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:54.696+0000 7f6807e72640 1 --2- 172.21.15.46:0/3647210548 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6800107750 0x7f680019e960 secure :-1 s=READY pgs=1006 cs=0 l=1 rev1=1 crypto rx=0x7f67f800efc0 tx=0x7f67f800c800 comp rx=0 tx=0).stop 2024-03-31T23:33:54.699 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:54.696+0000 7f6807e72640 1 -- 172.21.15.46:0/3647210548 shutdown_connections 2024-03-31T23:33:54.699 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:54.696+0000 7f6807e72640 1 --2- 172.21.15.46:0/3647210548 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f67ec0a2730 0x7f67ec0a4bf0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:33:54.699 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:54.696+0000 7f6807e72640 1 --2- 172.21.15.46:0/3647210548 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6800107750 0x7f680019e960 unknown :-1 s=CLOSED pgs=1006 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:33:54.699 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:54.696+0000 7f6807e72640 1 --2- 172.21.15.46:0/3647210548 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6800102100 0x7f680019aec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:33:54.699 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:54.696+0000 7f6807e72640 1 --2- 172.21.15.46:0/3647210548 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6800101740 0x7f680019a980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:33:54.699 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:54.696+0000 7f6807e72640 1 -- 172.21.15.46:0/3647210548 >> 172.21.15.46:0/3647210548 conn(0x7f68000fd210 msgr2=0x7f6800074650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:33:54.699 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:54.696+0000 7f6807e72640 1 -- 172.21.15.46:0/3647210548 shutdown_connections 2024-03-31T23:33:54.699 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:33:54.696+0000 7f6807e72640 1 -- 172.21.15.46:0/3647210548 wait complete. 2024-03-31T23:33:55.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:33:54 smithi162 ceph-mon[36237]: pgmap v703: 129 pgs: 129 active+clean; 1.1 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 4.8 MiB/s wr, 282 op/s 2024-03-31T23:33:55.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:33:54 smithi046 ceph-mon[32498]: pgmap v703: 129 pgs: 129 active+clean; 1.1 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 4.8 MiB/s wr, 282 op/s 2024-03-31T23:33:55.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:33:54 smithi142 ceph-mon[35271]: pgmap v703: 129 pgs: 129 active+clean; 1.1 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 4.8 MiB/s wr, 282 op/s 2024-03-31T23:33:55.310 INFO:tasks.workunit.client.0.smithi046.stdout: 10 84140 8.26 MB/sec execute 420 sec latency 191.290 ms 2024-03-31T23:33:56.310 INFO:tasks.workunit.client.0.smithi046.stdout: 10 84305 8.27 MB/sec execute 421 sec latency 143.570 ms 2024-03-31T23:33:57.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:33:56 smithi162 ceph-mon[36237]: pgmap v704: 129 pgs: 129 active+clean; 1.1 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 7.1 MiB/s wr, 400 op/s 2024-03-31T23:33:57.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:33:56 smithi046 ceph-mon[32498]: pgmap v704: 129 pgs: 129 active+clean; 1.1 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 7.1 MiB/s wr, 400 op/s 2024-03-31T23:33:57.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:33:56 smithi142 ceph-mon[35271]: pgmap v704: 129 pgs: 129 active+clean; 1.1 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 7.1 MiB/s wr, 400 op/s 2024-03-31T23:33:57.310 INFO:tasks.workunit.client.0.smithi046.stdout: 10 84473 8.27 MB/sec execute 422 sec latency 114.165 ms 2024-03-31T23:33:58.310 INFO:tasks.workunit.client.0.smithi046.stdout: 10 84605 8.27 MB/sec execute 423 sec latency 319.178 ms 2024-03-31T23:33:59.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:33:58 smithi162 ceph-mon[36237]: pgmap v705: 129 pgs: 129 active+clean; 1.1 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 4.6 MiB/s wr, 238 op/s 2024-03-31T23:33:59.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:33:58 smithi046 ceph-mon[32498]: pgmap v705: 129 pgs: 129 active+clean; 1.1 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 4.6 MiB/s wr, 238 op/s 2024-03-31T23:33:59.194 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:33:58 smithi142 ceph-mon[35271]: pgmap v705: 129 pgs: 129 active+clean; 1.1 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 4.6 MiB/s wr, 238 op/s 2024-03-31T23:33:59.310 INFO:tasks.workunit.client.0.smithi046.stdout: 10 84743 8.26 MB/sec execute 424 sec latency 94.203 ms 2024-03-31T23:34:00.310 INFO:tasks.workunit.client.0.smithi046.stdout: 10 84929 8.27 MB/sec execute 425 sec latency 88.007 ms 2024-03-31T23:34:01.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:34:00 smithi162 ceph-mon[36237]: pgmap v706: 129 pgs: 129 active+clean; 1.2 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 6.7 MiB/s wr, 399 op/s 2024-03-31T23:34:01.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:34:00 smithi046 ceph-mon[32498]: pgmap v706: 129 pgs: 129 active+clean; 1.2 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 6.7 MiB/s wr, 399 op/s 2024-03-31T23:34:01.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:34:00 smithi142 ceph-mon[35271]: pgmap v706: 129 pgs: 129 active+clean; 1.2 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 6.7 MiB/s wr, 399 op/s 2024-03-31T23:34:01.310 INFO:tasks.workunit.client.0.smithi046.stdout: 10 85120 8.28 MB/sec execute 426 sec latency 192.251 ms 2024-03-31T23:34:02.310 INFO:tasks.workunit.client.0.smithi046.stdout: 10 85236 8.27 MB/sec execute 427 sec latency 87.412 ms 2024-03-31T23:34:03.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:34:02 smithi162 ceph-mon[36237]: pgmap v707: 129 pgs: 129 active+clean; 1.2 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 4.4 MiB/s wr, 278 op/s 2024-03-31T23:34:03.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:34:02 smithi046 ceph-mon[32498]: pgmap v707: 129 pgs: 129 active+clean; 1.2 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 4.4 MiB/s wr, 278 op/s 2024-03-31T23:34:03.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:34:02 smithi142 ceph-mon[35271]: pgmap v707: 129 pgs: 129 active+clean; 1.2 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 4.4 MiB/s wr, 278 op/s 2024-03-31T23:34:03.310 INFO:tasks.workunit.client.0.smithi046.stdout: 10 85338 8.26 MB/sec execute 428 sec latency 191.065 ms 2024-03-31T23:34:04.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:34:03 smithi162 ceph-mon[36237]: osdmap e132: 12 total, 12 up, 12 in 2024-03-31T23:34:04.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:34:03 smithi046 ceph-mon[32498]: osdmap e132: 12 total, 12 up, 12 in 2024-03-31T23:34:04.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:34:03 smithi142 ceph-mon[35271]: osdmap e132: 12 total, 12 up, 12 in 2024-03-31T23:34:04.310 INFO:tasks.workunit.client.0.smithi046.stdout: 10 85500 8.27 MB/sec execute 429 sec latency 75.647 ms 2024-03-31T23:34:05.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:34:04 smithi162 ceph-mon[36237]: pgmap v709: 129 pgs: 129 active+clean; 1.2 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 5.3 MiB/s wr, 334 op/s 2024-03-31T23:34:05.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:34:04 smithi046 ceph-mon[32498]: pgmap v709: 129 pgs: 129 active+clean; 1.2 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 5.3 MiB/s wr, 334 op/s 2024-03-31T23:34:05.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:34:04 smithi142 ceph-mon[35271]: pgmap v709: 129 pgs: 129 active+clean; 1.2 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 5.3 MiB/s wr, 334 op/s 2024-03-31T23:34:05.310 INFO:tasks.workunit.client.0.smithi046.stdout: 10 85675 8.27 MB/sec execute 430 sec latency 91.621 ms 2024-03-31T23:34:06.310 INFO:tasks.workunit.client.0.smithi046.stdout: 10 85823 8.28 MB/sec execute 431 sec latency 101.020 ms 2024-03-31T23:34:07.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:34:06 smithi162 ceph-mon[36237]: pgmap v710: 129 pgs: 129 active+clean; 1.1 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 5.6 MiB/s wr, 317 op/s 2024-03-31T23:34:07.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:34:06 smithi046 ceph-mon[32498]: pgmap v710: 129 pgs: 129 active+clean; 1.1 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 5.6 MiB/s wr, 317 op/s 2024-03-31T23:34:07.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:34:06 smithi142 ceph-mon[35271]: pgmap v710: 129 pgs: 129 active+clean; 1.1 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 5.6 MiB/s wr, 317 op/s 2024-03-31T23:34:07.310 INFO:tasks.workunit.client.0.smithi046.stdout: 10 86001 8.28 MB/sec execute 432 sec latency 68.610 ms 2024-03-31T23:34:07.814 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:07.811+0000 7f858e464640 1 -- 172.21.15.46:0/1207605614 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f85881055f0 msgr2=0x7f85881059d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:34:07.814 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:07.811+0000 7f858e464640 1 --2- 172.21.15.46:0/1207605614 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f85881055f0 0x7f85881059d0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f8578009930 tx=0x7f857802bc30 comp rx=0 tx=0).stop 2024-03-31T23:34:07.814 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:07.811+0000 7f858e464640 1 -- 172.21.15.46:0/1207605614 shutdown_connections 2024-03-31T23:34:07.814 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:07.811+0000 7f858e464640 1 --2- 172.21.15.46:0/1207605614 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8588101260 0x7f858810d480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:34:07.814 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:07.811+0000 7f858e464640 1 --2- 172.21.15.46:0/1207605614 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f85881008a0 0x7f8588100d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:34:07.814 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:07.811+0000 7f858e464640 1 --2- 172.21.15.46:0/1207605614 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f85881055f0 0x7f85881059d0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:34:07.814 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:07.811+0000 7f858e464640 1 -- 172.21.15.46:0/1207605614 >> 172.21.15.46:0/1207605614 conn(0x7f85880774a0 msgr2=0x7f85880778b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:34:07.814 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:07.811+0000 7f858e464640 1 -- 172.21.15.46:0/1207605614 shutdown_connections 2024-03-31T23:34:07.815 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:07.811+0000 7f858e464640 1 -- 172.21.15.46:0/1207605614 wait complete. 2024-03-31T23:34:07.815 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:07.812+0000 7f858e464640 1 Processor -- start 2024-03-31T23:34:07.815 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:07.812+0000 7f858e464640 1 -- start start 2024-03-31T23:34:07.815 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:07.812+0000 7f858e464640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f85881008a0 0x7f858810ee80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:34:07.815 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:07.812+0000 7f858e464640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8588101260 0x7f858810f3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:34:07.815 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:07.812+0000 7f858e464640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f85881055f0 0x7f858810f900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:34:07.815 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:07.812+0000 7f858e464640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f8588111b10 con 0x7f8588101260 2024-03-31T23:34:07.815 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:07.812+0000 7f858e464640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f8588111990 con 0x7f85881008a0 2024-03-31T23:34:07.815 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:07.812+0000 7f858e464640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f8588111c90 con 0x7f85881055f0 2024-03-31T23:34:07.815 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:07.812+0000 7f8587fff640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f85881008a0 0x7f858810ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:34:07.815 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:07.812+0000 7f8587fff640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f85881008a0 0x7f858810ee80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.46:52268/0 (socket says 172.21.15.46:52268) 2024-03-31T23:34:07.815 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:07.812+0000 7f85877fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8588101260 0x7f858810f3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:34:07.816 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:07.812+0000 7f8587fff640 1 -- 172.21.15.46:0/3210375767 learned_addr learned my addr 172.21.15.46:0/3210375767 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:34:07.816 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:07.812+0000 7f858ca01640 1 --2- 172.21.15.46:0/3210375767 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f85881055f0 0x7f858810f900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:34:07.816 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:07.813+0000 7f8587fff640 1 -- 172.21.15.46:0/3210375767 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f85881055f0 msgr2=0x7f858810f900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:34:07.816 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:07.813+0000 7f8587fff640 1 --2- 172.21.15.46:0/3210375767 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f85881055f0 0x7f858810f900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:34:07.816 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:07.813+0000 7f8587fff640 1 -- 172.21.15.46:0/3210375767 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8588101260 msgr2=0x7f858810f3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:34:07.816 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:07.813+0000 7f8587fff640 1 --2- 172.21.15.46:0/3210375767 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8588101260 0x7f858810f3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:34:07.816 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:07.813+0000 7f8587fff640 1 -- 172.21.15.46:0/3210375767 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f85881a64b0 con 0x7f85881008a0 2024-03-31T23:34:07.816 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:07.813+0000 7f85877fe640 1 --2- 172.21.15.46:0/3210375767 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8588101260 0x7f858810f3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:34:07.816 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:07.813+0000 7f858ca01640 1 --2- 172.21.15.46:0/3210375767 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f85881055f0 0x7f858810f900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:34:07.816 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:07.813+0000 7f8587fff640 1 --2- 172.21.15.46:0/3210375767 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f85881008a0 0x7f858810ee80 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f8578000c00 tx=0x7f85780045e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:34:07.817 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:07.813+0000 7f85857fa640 1 -- 172.21.15.46:0/3210375767 <== mon.2 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8578043020 con 0x7f85881008a0 2024-03-31T23:34:07.817 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:07.813+0000 7f85857fa640 1 -- 172.21.15.46:0/3210375767 <== mon.2 v2:172.21.15.142:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f857802cb70 con 0x7f85881008a0 2024-03-31T23:34:07.817 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:07.813+0000 7f858e464640 1 -- 172.21.15.46:0/3210375767 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f85881a6740 con 0x7f85881008a0 2024-03-31T23:34:07.817 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:07.813+0000 7f85857fa640 1 -- 172.21.15.46:0/3210375767 <== mon.2 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8578003b80 con 0x7f85881008a0 2024-03-31T23:34:07.817 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:07.813+0000 7f858e464640 1 -- 172.21.15.46:0/3210375767 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f85881a6c90 con 0x7f85881008a0 2024-03-31T23:34:07.819 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:07.817+0000 7f85857fa640 1 -- 172.21.15.46:0/3210375767 <== mon.2 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 25) ==== 137451+0+0 (secure 0 0 0) 0x7f8578046020 con 0x7f85881008a0 2024-03-31T23:34:07.819 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:07.817+0000 7f8591043740 1 Processor -- start 2024-03-31T23:34:07.820 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:07.817+0000 7f8591043740 1 -- start start 2024-03-31T23:34:07.820 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:07.817+0000 7f8591043740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55ceb25606c0 0x55ceb2560aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:34:07.820 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:07.817+0000 7f8591043740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ceb2560fe0 0x55ceb2569870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:34:07.820 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:07.817+0000 7f8591043740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ceb2569db0 0x55ceb256c1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:34:07.820 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:07.817+0000 7f8591043740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55ceb24c38f0 con 0x55ceb2569db0 2024-03-31T23:34:07.820 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:07.817+0000 7f8591043740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55ceb246c8d0 con 0x55ceb25606c0 2024-03-31T23:34:07.820 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:07.817+0000 7f8591043740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55ceb24bf990 con 0x55ceb2560fe0 2024-03-31T23:34:07.820 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:07.817+0000 7f85877fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ceb2569db0 0x55ceb256c1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:34:07.820 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:07.817+0000 7f8587fff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ceb2560fe0 0x55ceb2569870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:34:07.820 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:07.817+0000 7f8587fff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ceb2560fe0 0x55ceb2569870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.46:52656/0 (socket says 172.21.15.46:52656) 2024-03-31T23:34:07.820 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:07.817+0000 7f8587fff640 1 -- 172.21.15.46:0/366216923 learned_addr learned my addr 172.21.15.46:0/366216923 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:34:07.820 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:07.817+0000 7f858ca01640 1 --2- 172.21.15.46:0/366216923 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55ceb25606c0 0x55ceb2560aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:34:07.820 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:07.818+0000 7f85857fa640 1 --2- 172.21.15.46:0/3210375767 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f85700a26b0 0x7f85700a4b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:34:07.821 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:07.818+0000 7f85877fe640 1 -- 172.21.15.46:0/366216923 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ceb2560fe0 msgr2=0x55ceb2569870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:34:07.821 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:07.818+0000 7f85877fe640 1 --2- 172.21.15.46:0/366216923 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ceb2560fe0 0x55ceb2569870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:34:07.821 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:07.818+0000 7f85877fe640 1 -- 172.21.15.46:0/366216923 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55ceb25606c0 msgr2=0x55ceb2560aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:34:07.821 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:07.818+0000 7f85877fe640 1 --2- 172.21.15.46:0/366216923 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55ceb25606c0 0x55ceb2560aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:34:07.821 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:07.818+0000 7f85877fe640 1 -- 172.21.15.46:0/366216923 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ceb21d8210 con 0x55ceb2569db0 2024-03-31T23:34:07.821 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:07.818+0000 7f8587fff640 1 --2- 172.21.15.46:0/366216923 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ceb2560fe0 0x55ceb2569870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:34:07.821 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:07.818+0000 7f858ca01640 1 --2- 172.21.15.46:0/3210375767 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f85700a26b0 0x7f85700a4b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:34:07.821 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:07.818+0000 7f85857fa640 1 -- 172.21.15.46:0/3210375767 <== mon.2 v2:172.21.15.142:3300/0 5 ==== osd_map(132..132 src has 1..132) ==== 9642+0+0 (secure 0 0 0) 0x7f85780f0f20 con 0x7f85881008a0 2024-03-31T23:34:07.821 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:07.818+0000 7f85877fe640 1 --2- 172.21.15.46:0/366216923 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ceb2569db0 0x55ceb256c1a0 secure :-1 s=READY pgs=1010 cs=0 l=1 rev1=1 crypto rx=0x7f857c009fa0 tx=0x7f857c0118d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=505e2dc06ca4142e server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:34:07.821 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:07.818+0000 7f856e7fc640 1 -- 172.21.15.46:0/366216923 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f857c004030 con 0x55ceb2569db0 2024-03-31T23:34:07.821 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:07.818+0000 7f856e7fc640 1 -- 172.21.15.46:0/366216923 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f857c0054d0 con 0x55ceb2569db0 2024-03-31T23:34:07.821 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:07.818+0000 7f856e7fc640 1 -- 172.21.15.46:0/366216923 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f857c004ce0 con 0x55ceb2569db0 2024-03-31T23:34:07.824 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:07.819+0000 7f8591043740 1 -- 172.21.15.46:0/366216923 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ceb2569db0 msgr2=0x55ceb256c1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:34:07.824 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:07.819+0000 7f858ca01640 1 --2- 172.21.15.46:0/3210375767 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f85700a26b0 0x7f85700a4b70 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f8580005fd0 tx=0x7f8580004a40 comp rx=0 tx=0).ready entity=mgr.15654 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:34:07.824 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:07.819+0000 7f8591043740 1 --2- 172.21.15.46:0/366216923 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ceb2569db0 0x55ceb256c1a0 secure :-1 s=READY pgs=1010 cs=0 l=1 rev1=1 crypto rx=0x7f857c009fa0 tx=0x7f857c0118d0 comp rx=0 tx=0).stop 2024-03-31T23:34:07.825 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:07.819+0000 7f8591043740 1 -- 172.21.15.46:0/366216923 shutdown_connections 2024-03-31T23:34:07.825 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:07.820+0000 7f8591043740 1 --2- 172.21.15.46:0/366216923 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ceb2569db0 0x55ceb256c1a0 unknown :-1 s=CLOSED pgs=1010 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:34:07.825 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:07.820+0000 7f8591043740 1 --2- 172.21.15.46:0/366216923 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ceb2560fe0 0x55ceb2569870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:34:07.825 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:07.820+0000 7f8591043740 1 --2- 172.21.15.46:0/366216923 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55ceb25606c0 0x55ceb2560aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:34:07.825 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:07.820+0000 7f8591043740 1 -- 172.21.15.46:0/366216923 >> 172.21.15.46:0/366216923 conn(0x55ceb255afe0 msgr2=0x55ceb255b3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:34:07.825 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:07.820+0000 7f8591043740 1 -- 172.21.15.46:0/366216923 shutdown_connections 2024-03-31T23:34:07.825 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:07.820+0000 7f8591043740 1 -- 172.21.15.46:0/366216923 wait complete. 2024-03-31T23:34:07.825 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:07.820+0000 7f8591043740 1 -- start start 2024-03-31T23:34:07.825 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:07.820+0000 7f8591043740 1 Processor -- start 2024-03-31T23:34:07.825 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:07.821+0000 7f8591043740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ceb25606c0 0x55ceb2566db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:34:07.825 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:07.821+0000 7f8591043740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ceb2560fe0 0x55ceb25672f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:34:07.825 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:07.821+0000 7f8591043740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55ceb2569db0 0x55ceb2567830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:34:07.825 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:07.821+0000 7f8591043740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55ceb256c9e0 con 0x55ceb25606c0 2024-03-31T23:34:07.825 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:07.821+0000 7f8591043740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55ceb2461f80 con 0x55ceb2569db0 2024-03-31T23:34:07.826 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:07.821+0000 7f8591043740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55ceb256cb60 con 0x55ceb2560fe0 2024-03-31T23:34:07.826 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:07.821+0000 7f8591043740 10 client.0 _ll_register_callbacks cb 0x55ceb24ba290 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T23:34:07.826 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:07.821+0000 7f85877fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ceb2560fe0 0x55ceb25672f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:34:07.826 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:07.821+0000 7f8587fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ceb25606c0 0x55ceb2566db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:34:07.826 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:07.821+0000 7f85877fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ceb2560fe0 0x55ceb25672f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.46:52668/0 (socket says 172.21.15.46:52668) 2024-03-31T23:34:07.826 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:07.821+0000 7f85877fe640 1 -- 172.21.15.46:0/3404788149 learned_addr learned my addr 172.21.15.46:0/3404788149 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:34:07.826 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:07.821+0000 7f858ca01640 1 --2- 172.21.15.46:0/3404788149 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55ceb2569db0 0x55ceb2567830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:34:07.826 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:07.821+0000 7f85877fe640 1 -- 172.21.15.46:0/3404788149 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55ceb2569db0 msgr2=0x55ceb2567830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:34:07.826 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:07.821+0000 7f85877fe640 1 --2- 172.21.15.46:0/3404788149 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55ceb2569db0 0x55ceb2567830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:34:07.826 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:07.821+0000 7f85877fe640 1 -- 172.21.15.46:0/3404788149 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ceb25606c0 msgr2=0x55ceb2566db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:34:07.826 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:07.821+0000 7f85877fe640 1 --2- 172.21.15.46:0/3404788149 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ceb25606c0 0x55ceb2566db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:34:07.826 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:07.821+0000 7f85877fe640 1 -- 172.21.15.46:0/3404788149 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ceb22a4620 con 0x55ceb2560fe0 2024-03-31T23:34:07.826 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:07.821+0000 7f858ca01640 1 --2- 172.21.15.46:0/3404788149 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55ceb2569db0 0x55ceb2567830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T23:34:07.827 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:07.822+0000 7f85877fe640 1 --2- 172.21.15.46:0/3404788149 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ceb2560fe0 0x55ceb25672f0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f857c009350 tx=0x7f857c003820 comp rx=0 tx=0).ready entity=mon.1 client_cookie=998d2fd7a250146 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:34:07.827 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:07.822+0000 7f85677fe640 10 client.0 ms_handle_connect on v2:172.21.15.162:3300/0 2024-03-31T23:34:07.827 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:07.822+0000 7f85677fe640 1 -- 172.21.15.46:0/3404788149 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f857c027020 con 0x55ceb2560fe0 2024-03-31T23:34:07.827 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:07.822+0000 7f8591043740 1 -- 172.21.15.46:0/3404788149 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f857c0054d0 con 0x55ceb2560fe0 2024-03-31T23:34:07.827 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:07.822+0000 7f85677fe640 1 -- 172.21.15.46:0/3404788149 <== mon.1 v2:172.21.15.162:3300/0 2 ==== osd_map(132..132 src has 1..132) ==== 9642+0+0 (secure 0 0 0) 0x7f857c003a10 con 0x55ceb2560fe0 2024-03-31T23:34:07.827 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:07.822+0000 7f85677fe640 1 -- 172.21.15.46:0/3404788149 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_get_version_reply(handle=1 version=51) ==== 24+0+0 (secure 0 0 0) 0x7f857c032050 con 0x55ceb2560fe0 2024-03-31T23:34:07.827 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:07.822+0000 7f8591043740 10 client.27673 fetch_fsmap learned FSMap version 51 2024-03-31T23:34:07.827 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:07.822+0000 7f8591043740 1 -- 172.21.15.46:0/3404788149 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,fsmap=51,monmap=4+}) -- 0x55ceb25788c0 con 0x55ceb2560fe0 2024-03-31T23:34:07.827 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:07.822+0000 7f85677fe640 1 -- 172.21.15.46:0/3404788149 <== mon.1 v2:172.21.15.162:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f857c0054d0 con 0x55ceb2560fe0 2024-03-31T23:34:07.827 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:07.823+0000 7f85677fe640 1 -- 172.21.15.46:0/3404788149 <== mon.1 v2:172.21.15.162:3300/0 5 ==== fsmap(e 51) ==== 8358+0+0 (secure 0 0 0) 0x7f857c00fa50 con 0x55ceb2560fe0 2024-03-31T23:34:07.828 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:07.823+0000 7f8591043740 10 client.27673 fetch_fsmap finished waiting for FSMap version 51 2024-03-31T23:34:07.828 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:07.823+0000 7f8591043740 10 client.27673 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-31T23:34:07.828 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:07.823+0000 7f8591043740 20 client.27673 populate_metadata read hostname 'smithi046' 2024-03-31T23:34:07.828 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:07.823+0000 7f8591043740 1 --2- 172.21.15.46:0/3404788149 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55ceb257a170 0x55ceb257c630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:34:07.828 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:07.823+0000 7f8591043740 4 client.27673 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-31T23:34:07.828 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:07.823+0000 7f8591043740 1 -- 172.21.15.46:0/3404788149 --> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ceb24d7120 con 0x55ceb257a170 2024-03-31T23:34:07.828 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:07.823+0000 7f8587fff640 1 --2- 172.21.15.46:0/3404788149 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55ceb257a170 0x55ceb257c630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:34:07.828 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:07.823+0000 7f85677fe640 10 client.27673 ms_handle_connect on v2:172.21.15.142:6838/2519143918 2024-03-31T23:34:07.828 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:07.823+0000 7f8587fff640 1 --2- 172.21.15.46:0/3404788149 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55ceb257a170 0x55ceb257c630 crc :-1 s=READY pgs=276 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7ac686a214499702 server_cookie=df31aa85eea1522a in_seq=0 out_seq=0 2024-03-31T23:34:07.829 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:07.826+0000 7f85677fe640 1 -- 172.21.15.46:0/3404788149 <== mds.0 v2:172.21.15.142:6838/2519143918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55ceb24d7120 con 0x55ceb257a170 2024-03-31T23:34:07.830 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:07.826+0000 7f85677fe640 10 client.27673 handle_command_reply: tid=0 2024-03-31T23:34:07.830 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:07.826+0000 7f8591043740 1 client.27673 shutdown 2024-03-31T23:34:07.830 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:07.827+0000 7f8591043740 1 -- 172.21.15.46:0/3404788149 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ceb2560fe0 msgr2=0x55ceb25672f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:34:07.830 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:07.827+0000 7f8591043740 1 --2- 172.21.15.46:0/3404788149 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ceb2560fe0 0x55ceb25672f0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f857c009350 tx=0x7f857c003820 comp rx=0 tx=0).stop 2024-03-31T23:34:07.830 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:07.827+0000 7f8591043740 1 -- 172.21.15.46:0/3404788149 shutdown_connections 2024-03-31T23:34:07.830 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:07.827+0000 7f8591043740 1 --2- 172.21.15.46:0/3404788149 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55ceb257a170 0x55ceb257c630 crc :-1 s=READY pgs=276 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:34:07.830 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:07.827+0000 7f8591043740 1 --2- 172.21.15.46:0/3404788149 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55ceb2569db0 0x55ceb2567830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:34:07.830 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:07.827+0000 7f8591043740 1 --2- 172.21.15.46:0/3404788149 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ceb2560fe0 0x55ceb25672f0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:34:07.830 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:07.827+0000 7f8591043740 1 --2- 172.21.15.46:0/3404788149 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ceb25606c0 0x55ceb2566db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:34:07.831 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:07.827+0000 7f8591043740 1 -- 172.21.15.46:0/3404788149 >> 172.21.15.46:0/3404788149 conn(0x55ceb255afe0 msgr2=0x55ceb255cd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:34:07.831 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:07.827+0000 7f85677fe640 0 client.27673 ms_handle_reset on v2:172.21.15.142:6838/2519143918 2024-03-31T23:34:07.831 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:07.827+0000 7f8591043740 1 -- 172.21.15.46:0/3404788149 shutdown_connections 2024-03-31T23:34:07.831 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:07.827+0000 7f8591043740 1 -- 172.21.15.46:0/3404788149 wait complete. 2024-03-31T23:34:07.831 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:07.827+0000 7f8591043740 20 client.27673 trim_cache size 0 max 16384 2024-03-31T23:34:07.848 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:07.846+0000 7f8591043740 1 Processor -- start 2024-03-31T23:34:07.849 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:07.846+0000 7f8591043740 1 -- start start 2024-03-31T23:34:07.849 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:07.846+0000 7f8591043740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ceb25606c0 0x55ceb25694e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:34:07.849 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:07.846+0000 7f8591043740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55ceb2560fe0 0x55ceb25757b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:34:07.849 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:07.846+0000 7f8591043740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ceb2569db0 0x55ceb2581f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:34:07.849 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:07.846+0000 7f8591043740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f85881ab0e0 con 0x55ceb25606c0 2024-03-31T23:34:07.849 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:07.846+0000 7f8591043740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f85881a8920 con 0x55ceb2560fe0 2024-03-31T23:34:07.849 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:07.846+0000 7f8591043740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f85880732a0 con 0x55ceb2569db0 2024-03-31T23:34:07.849 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:07.846+0000 7f8587fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ceb25606c0 0x55ceb25694e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:34:07.849 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:07.846+0000 7f8587fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ceb25606c0 0x55ceb25694e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37654/0 (socket says 172.21.15.46:37654) 2024-03-31T23:34:07.849 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:07.846+0000 7f8587fff640 1 -- 172.21.15.46:0/2237116493 learned_addr learned my addr 172.21.15.46:0/2237116493 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:34:07.849 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:07.846+0000 7f858ca01640 1 --2- 172.21.15.46:0/2237116493 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ceb2569db0 0x55ceb2581f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:34:07.849 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:07.846+0000 7f85877fe640 1 --2- 172.21.15.46:0/2237116493 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55ceb2560fe0 0x55ceb25757b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:34:07.850 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:07.846+0000 7f8587fff640 1 -- 172.21.15.46:0/2237116493 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ceb2569db0 msgr2=0x55ceb2581f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:34:07.850 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:07.846+0000 7f8587fff640 1 --2- 172.21.15.46:0/2237116493 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ceb2569db0 0x55ceb2581f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:34:07.850 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:07.846+0000 7f8587fff640 1 -- 172.21.15.46:0/2237116493 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55ceb2560fe0 msgr2=0x55ceb25757b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:34:07.850 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:07.846+0000 7f8587fff640 1 --2- 172.21.15.46:0/2237116493 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55ceb2560fe0 0x55ceb25757b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:34:07.850 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:07.846+0000 7f8587fff640 1 -- 172.21.15.46:0/2237116493 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ceb2560dd0 con 0x55ceb25606c0 2024-03-31T23:34:07.850 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:07.846+0000 7f8587fff640 1 --2- 172.21.15.46:0/2237116493 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ceb25606c0 0x55ceb25694e0 secure :-1 s=READY pgs=1011 cs=0 l=1 rev1=1 crypto rx=0x7f85780bd2e0 tx=0x7f85780dd040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2becca3b5def980e server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:34:07.850 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:07.847+0000 7f856dffb640 1 -- 172.21.15.46:0/2237116493 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85780a29c0 con 0x55ceb25606c0 2024-03-31T23:34:07.850 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:07.847+0000 7f856dffb640 1 -- 172.21.15.46:0/2237116493 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f85780bda50 con 0x55ceb25606c0 2024-03-31T23:34:07.850 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:07.847+0000 7f856dffb640 1 -- 172.21.15.46:0/2237116493 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85780c14a0 con 0x55ceb25606c0 2024-03-31T23:34:07.850 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:07.848+0000 7f8591043740 1 -- 172.21.15.46:0/2237116493 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ceb25606c0 msgr2=0x55ceb25694e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:34:07.850 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:07.848+0000 7f8591043740 1 --2- 172.21.15.46:0/2237116493 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ceb25606c0 0x55ceb25694e0 secure :-1 s=READY pgs=1011 cs=0 l=1 rev1=1 crypto rx=0x7f85780bd2e0 tx=0x7f85780dd040 comp rx=0 tx=0).stop 2024-03-31T23:34:07.850 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:07.848+0000 7f8591043740 1 -- 172.21.15.46:0/2237116493 shutdown_connections 2024-03-31T23:34:07.851 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:07.848+0000 7f8591043740 1 --2- 172.21.15.46:0/2237116493 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ceb2569db0 0x55ceb2581f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:34:07.851 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:07.848+0000 7f8591043740 1 --2- 172.21.15.46:0/2237116493 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55ceb2560fe0 0x55ceb25757b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:34:07.851 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:07.848+0000 7f8591043740 1 --2- 172.21.15.46:0/2237116493 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ceb25606c0 0x55ceb25694e0 unknown :-1 s=CLOSED pgs=1011 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:34:07.851 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:07.848+0000 7f8591043740 1 -- 172.21.15.46:0/2237116493 >> 172.21.15.46:0/2237116493 conn(0x55ceb255afe0 msgr2=0x55ceb25613c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:34:07.851 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:07.848+0000 7f8591043740 1 -- 172.21.15.46:0/2237116493 shutdown_connections 2024-03-31T23:34:07.851 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:07.848+0000 7f8591043740 1 -- 172.21.15.46:0/2237116493 wait complete. 2024-03-31T23:34:07.854 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:07.848+0000 7f8591043740 1 -- start start 2024-03-31T23:34:07.854 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:07.848+0000 7f8591043740 1 Processor -- start 2024-03-31T23:34:07.854 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:07.849+0000 7f8591043740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ceb25606c0 0x55ceb256cda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:34:07.854 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:07.849+0000 7f8591043740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55ceb2560fe0 0x55ceb256d2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:34:07.854 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:07.849+0000 7f8591043740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ceb2569db0 0x55ceb256d820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:34:07.854 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:07.849+0000 7f8591043740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f85881a6e60 con 0x55ceb2569db0 2024-03-31T23:34:07.854 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:07.849+0000 7f8591043740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f85881a8250 con 0x55ceb2560fe0 2024-03-31T23:34:07.854 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:07.849+0000 7f8591043740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f85881a74b0 con 0x55ceb25606c0 2024-03-31T23:34:07.854 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:07.849+0000 7f8591043740 10 client.0 _ll_register_callbacks cb 0x7f8580009da0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T23:34:07.854 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:07.850+0000 7f8587fff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ceb25606c0 0x55ceb256cda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:34:07.854 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:07.850+0000 7f8587fff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ceb25606c0 0x55ceb256cda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.46:52676/0 (socket says 172.21.15.46:52676) 2024-03-31T23:34:07.855 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:07.850+0000 7f8587fff640 1 -- 172.21.15.46:0/2559479559 learned_addr learned my addr 172.21.15.46:0/2559479559 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:34:07.855 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:07.850+0000 7f858ca01640 1 --2- 172.21.15.46:0/2559479559 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ceb2569db0 0x55ceb256d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:34:07.855 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:07.850+0000 7f85877fe640 1 --2- 172.21.15.46:0/2559479559 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55ceb2560fe0 0x55ceb256d2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:34:07.855 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:07.850+0000 7f8587fff640 1 -- 172.21.15.46:0/2559479559 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55ceb2560fe0 msgr2=0x55ceb256d2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:34:07.855 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:07.850+0000 7f8587fff640 1 --2- 172.21.15.46:0/2559479559 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55ceb2560fe0 0x55ceb256d2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:34:07.855 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:07.850+0000 7f8587fff640 1 -- 172.21.15.46:0/2559479559 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ceb2569db0 msgr2=0x55ceb256d820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:34:07.855 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:07.850+0000 7f8587fff640 1 --2- 172.21.15.46:0/2559479559 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ceb2569db0 0x55ceb256d820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:34:07.855 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:07.850+0000 7f8587fff640 1 -- 172.21.15.46:0/2559479559 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ceb25de520 con 0x55ceb25606c0 2024-03-31T23:34:07.855 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:07.850+0000 7f8587fff640 1 --2- 172.21.15.46:0/2559479559 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ceb25606c0 0x55ceb256cda0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f85780b97b0 tx=0x7f85780bd390 comp rx=0 tx=0).ready entity=mon.1 client_cookie=da3eaaa7a774bceb server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:34:07.855 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:07.851+0000 7f85677fe640 10 client.0 ms_handle_connect on v2:172.21.15.162:3300/0 2024-03-31T23:34:07.855 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:07.851+0000 7f85677fe640 1 -- 172.21.15.46:0/2559479559 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85780bd120 con 0x55ceb25606c0 2024-03-31T23:34:07.855 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:07.851+0000 7f85677fe640 1 -- 172.21.15.46:0/2559479559 <== mon.1 v2:172.21.15.162:3300/0 2 ==== osd_map(132..132 src has 1..132) ==== 9642+0+0 (secure 0 0 0) 0x7f85780dd0b0 con 0x55ceb25606c0 2024-03-31T23:34:07.855 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:07.851+0000 7f8591043740 1 -- 172.21.15.46:0/2559479559 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f85780bda50 con 0x55ceb25606c0 2024-03-31T23:34:07.855 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:07.851+0000 7f85677fe640 1 -- 172.21.15.46:0/2559479559 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_get_version_reply(handle=1 version=51) ==== 24+0+0 (secure 0 0 0) 0x7f857809d020 con 0x55ceb25606c0 2024-03-31T23:34:07.856 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:07.851+0000 7f8591043740 10 client.27679 fetch_fsmap learned FSMap version 51 2024-03-31T23:34:07.856 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:07.851+0000 7f8591043740 1 -- 172.21.15.46:0/2559479559 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,fsmap=51,monmap=4+}) -- 0x55ceb25df770 con 0x55ceb25606c0 2024-03-31T23:34:07.856 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:07.852+0000 7f85677fe640 1 -- 172.21.15.46:0/2559479559 <== mon.1 v2:172.21.15.162:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f85780bda50 con 0x55ceb25606c0 2024-03-31T23:34:07.856 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:07.852+0000 7f85677fe640 1 -- 172.21.15.46:0/2559479559 <== mon.1 v2:172.21.15.162:3300/0 5 ==== fsmap(e 51) ==== 8358+0+0 (secure 0 0 0) 0x7f85780d7060 con 0x55ceb25606c0 2024-03-31T23:34:07.856 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:07.852+0000 7f8591043740 10 client.27679 fetch_fsmap finished waiting for FSMap version 51 2024-03-31T23:34:07.856 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:07.852+0000 7f8591043740 10 client.27679 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-31T23:34:07.856 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:07.852+0000 7f8591043740 20 client.27679 populate_metadata read hostname 'smithi046' 2024-03-31T23:34:07.856 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:07.852+0000 7f8591043740 1 --2- 172.21.15.46:0/2559479559 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55ceb25dfdd0 0x55ceb25e21f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:34:07.856 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:07.852+0000 7f8591043740 4 client.27679 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-31T23:34:07.856 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:07.852+0000 7f8591043740 1 -- 172.21.15.46:0/2559479559 --> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55ceb24d0e30 con 0x55ceb25dfdd0 2024-03-31T23:34:07.856 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:07.852+0000 7f85877fe640 1 --2- 172.21.15.46:0/2559479559 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55ceb25dfdd0 0x55ceb25e21f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:34:07.856 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:07.853+0000 7f85877fe640 1 --2- 172.21.15.46:0/2559479559 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55ceb25dfdd0 0x55ceb25e21f0 crc :-1 s=READY pgs=277 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fcc61fe8739d0820 server_cookie=dfa60b289db8aac0 in_seq=0 out_seq=0 2024-03-31T23:34:07.856 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:07.853+0000 7f85677fe640 10 client.27679 ms_handle_connect on v2:172.21.15.142:6838/2519143918 2024-03-31T23:34:08.287 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:08.286+0000 7f85677fe640 1 -- 172.21.15.46:0/2559479559 <== mds.0 v2:172.21.15.142:6838/2519143918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55ceb24d0e30 con 0x55ceb25dfdd0 2024-03-31T23:34:08.287 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:08.286+0000 7f85677fe640 10 client.27679 handle_command_reply: tid=0 2024-03-31T23:34:08.287 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:08.286+0000 7f8591043740 1 client.27679 shutdown 2024-03-31T23:34:08.288 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:08.286+0000 7f8591043740 1 -- 172.21.15.46:0/2559479559 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ceb25606c0 msgr2=0x55ceb256cda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:34:08.288 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:08.286+0000 7f8591043740 1 --2- 172.21.15.46:0/2559479559 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ceb25606c0 0x55ceb256cda0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f85780b97b0 tx=0x7f85780bd390 comp rx=0 tx=0).stop 2024-03-31T23:34:08.288 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:08.286+0000 7f8591043740 1 -- 172.21.15.46:0/2559479559 shutdown_connections 2024-03-31T23:34:08.288 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:08.286+0000 7f8591043740 1 --2- 172.21.15.46:0/2559479559 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55ceb25dfdd0 0x55ceb25e21f0 crc :-1 s=READY pgs=277 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:34:08.288 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:08.286+0000 7f8591043740 1 --2- 172.21.15.46:0/2559479559 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ceb2569db0 0x55ceb256d820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:34:08.288 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:08.286+0000 7f8591043740 1 --2- 172.21.15.46:0/2559479559 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55ceb2560fe0 0x55ceb256d2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:34:08.288 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:08.286+0000 7f85677fe640 0 client.27679 ms_handle_reset on v2:172.21.15.142:6838/2519143918 2024-03-31T23:34:08.288 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:08.286+0000 7f8591043740 1 --2- 172.21.15.46:0/2559479559 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ceb25606c0 0x55ceb256cda0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:34:08.288 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:08.286+0000 7f8591043740 1 -- 172.21.15.46:0/2559479559 >> 172.21.15.46:0/2559479559 conn(0x55ceb255afe0 msgr2=0x55ceb2565600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:34:08.288 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:08.286+0000 7f8591043740 1 -- 172.21.15.46:0/2559479559 shutdown_connections 2024-03-31T23:34:08.288 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:08.286+0000 7f8591043740 1 -- 172.21.15.46:0/2559479559 wait complete. 2024-03-31T23:34:08.288 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:08.286+0000 7f8591043740 20 client.27679 trim_cache size 0 max 16384 2024-03-31T23:34:08.289 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout:{ 2024-03-31T23:34:08.289 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout: "message": "", 2024-03-31T23:34:08.289 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout: "return_code": 0 2024-03-31T23:34:08.289 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout:} 2024-03-31T23:34:08.290 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:08.287+0000 7f858e464640 1 -- 172.21.15.46:0/3210375767 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f85700a26b0 msgr2=0x7f85700a4b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:34:08.290 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:08.287+0000 7f858e464640 1 --2- 172.21.15.46:0/3210375767 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f85700a26b0 0x7f85700a4b70 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f8580005fd0 tx=0x7f8580004a40 comp rx=0 tx=0).stop 2024-03-31T23:34:08.290 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:08.287+0000 7f858e464640 1 -- 172.21.15.46:0/3210375767 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f85881008a0 msgr2=0x7f858810ee80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:34:08.290 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:08.287+0000 7f858e464640 1 --2- 172.21.15.46:0/3210375767 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f85881008a0 0x7f858810ee80 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f8578000c00 tx=0x7f85780045e0 comp rx=0 tx=0).stop 2024-03-31T23:34:08.290 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:08.287+0000 7f858e464640 1 -- 172.21.15.46:0/3210375767 shutdown_connections 2024-03-31T23:34:08.290 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:08.287+0000 7f858e464640 1 --2- 172.21.15.46:0/3210375767 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f85700a26b0 0x7f85700a4b70 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:34:08.290 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:08.287+0000 7f858e464640 1 --2- 172.21.15.46:0/3210375767 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f85881055f0 0x7f858810f900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:34:08.290 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:08.287+0000 7f858e464640 1 --2- 172.21.15.46:0/3210375767 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8588101260 0x7f858810f3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:34:08.290 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:08.287+0000 7f858e464640 1 --2- 172.21.15.46:0/3210375767 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f85881008a0 0x7f858810ee80 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:34:08.290 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:08.287+0000 7f858e464640 1 -- 172.21.15.46:0/3210375767 >> 172.21.15.46:0/3210375767 conn(0x7f85880774a0 msgr2=0x7f85880fe950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:34:08.290 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:08.287+0000 7f858e464640 1 -- 172.21.15.46:0/3210375767 shutdown_connections 2024-03-31T23:34:08.290 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:08.287+0000 7f858e464640 1 -- 172.21.15.46:0/3210375767 wait complete. 2024-03-31T23:34:08.310 INFO:tasks.workunit.client.0.smithi046.stdout: 10 86102 8.28 MB/sec execute 433 sec latency 224.943 ms 2024-03-31T23:34:09.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:34:08 smithi162 ceph-mon[36237]: pgmap v711: 129 pgs: 129 active+clean; 1.1 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 5.6 MiB/s wr, 317 op/s 2024-03-31T23:34:09.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:34:08 smithi046 ceph-mon[32498]: pgmap v711: 129 pgs: 129 active+clean; 1.1 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 5.6 MiB/s wr, 317 op/s 2024-03-31T23:34:09.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:34:08 smithi142 ceph-mon[35271]: pgmap v711: 129 pgs: 129 active+clean; 1.1 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 5.6 MiB/s wr, 317 op/s 2024-03-31T23:34:09.310 INFO:tasks.workunit.client.0.smithi046.stdout: 10 86213 8.26 MB/sec execute 434 sec latency 88.294 ms 2024-03-31T23:34:10.310 INFO:tasks.workunit.client.0.smithi046.stdout: 10 86361 8.27 MB/sec execute 435 sec latency 99.252 ms 2024-03-31T23:34:11.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:34:10 smithi162 ceph-mon[36237]: pgmap v712: 129 pgs: 129 active+clean; 1.2 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 6.3 MiB/s wr, 344 op/s 2024-03-31T23:34:11.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:34:10 smithi162 ceph-mon[36237]: osdmap e133: 12 total, 12 up, 12 in 2024-03-31T23:34:11.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:34:10 smithi046 ceph-mon[32498]: pgmap v712: 129 pgs: 129 active+clean; 1.2 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 6.3 MiB/s wr, 344 op/s 2024-03-31T23:34:11.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:34:10 smithi046 ceph-mon[32498]: osdmap e133: 12 total, 12 up, 12 in 2024-03-31T23:34:11.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:34:10 smithi142 ceph-mon[35271]: pgmap v712: 129 pgs: 129 active+clean; 1.2 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 6.3 MiB/s wr, 344 op/s 2024-03-31T23:34:11.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:34:10 smithi142 ceph-mon[35271]: osdmap e133: 12 total, 12 up, 12 in 2024-03-31T23:34:11.310 INFO:tasks.workunit.client.0.smithi046.stdout: 10 86507 8.27 MB/sec execute 436 sec latency 71.907 ms 2024-03-31T23:34:12.310 INFO:tasks.workunit.client.0.smithi046.stdout: 10 86633 8.27 MB/sec execute 437 sec latency 134.479 ms 2024-03-31T23:34:13.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:34:12 smithi162 ceph-mon[36237]: pgmap v714: 129 pgs: 129 active+clean; 1.2 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 7.6 MiB/s wr, 417 op/s 2024-03-31T23:34:13.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:34:12 smithi046 ceph-mon[32498]: pgmap v714: 129 pgs: 129 active+clean; 1.2 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 7.6 MiB/s wr, 417 op/s 2024-03-31T23:34:13.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:34:12 smithi142 ceph-mon[35271]: pgmap v714: 129 pgs: 129 active+clean; 1.2 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 7.6 MiB/s wr, 417 op/s 2024-03-31T23:34:13.310 INFO:tasks.workunit.client.0.smithi046.stdout: 10 86768 8.26 MB/sec execute 438 sec latency 94.875 ms 2024-03-31T23:34:14.310 INFO:tasks.workunit.client.0.smithi046.stdout: 10 86911 8.26 MB/sec execute 439 sec latency 67.411 ms 2024-03-31T23:34:15.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:34:14 smithi162 ceph-mon[36237]: pgmap v715: 129 pgs: 129 active+clean; 1.2 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 6.3 MiB/s wr, 344 op/s 2024-03-31T23:34:15.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:34:14 smithi046 ceph-mon[32498]: pgmap v715: 129 pgs: 129 active+clean; 1.2 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 6.3 MiB/s wr, 344 op/s 2024-03-31T23:34:15.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:34:14 smithi142 ceph-mon[35271]: pgmap v715: 129 pgs: 129 active+clean; 1.2 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 6.3 MiB/s wr, 344 op/s 2024-03-31T23:34:15.310 INFO:tasks.workunit.client.0.smithi046.stdout: 10 87062 8.26 MB/sec execute 440 sec latency 100.391 ms 2024-03-31T23:34:16.310 INFO:tasks.workunit.client.0.smithi046.stdout: 10 87188 8.25 MB/sec execute 441 sec latency 143.342 ms 2024-03-31T23:34:17.048 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:34:16 smithi162 ceph-mon[36237]: pgmap v716: 129 pgs: 129 active+clean; 1.2 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 6.0 MiB/s wr, 398 op/s 2024-03-31T23:34:17.113 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:34:16 smithi046 ceph-mon[32498]: pgmap v716: 129 pgs: 129 active+clean; 1.2 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 6.0 MiB/s wr, 398 op/s 2024-03-31T23:34:17.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:34:16 smithi142 ceph-mon[35271]: pgmap v716: 129 pgs: 129 active+clean; 1.2 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 6.0 MiB/s wr, 398 op/s 2024-03-31T23:34:17.311 INFO:tasks.workunit.client.0.smithi046.stdout: 10 87353 8.25 MB/sec execute 442 sec latency 105.283 ms 2024-03-31T23:34:18.310 INFO:tasks.workunit.client.0.smithi046.stdout: 10 87511 8.25 MB/sec execute 443 sec latency 95.948 ms 2024-03-31T23:34:19.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:34:18 smithi046 ceph-mon[32498]: pgmap v717: 129 pgs: 129 active+clean; 1.2 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 6.0 MiB/s wr, 398 op/s 2024-03-31T23:34:19.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:34:18 smithi046 ceph-mon[32498]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-03-31T23:34:19.194 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:34:18 smithi142 ceph-mon[35271]: pgmap v717: 129 pgs: 129 active+clean; 1.2 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 6.0 MiB/s wr, 398 op/s 2024-03-31T23:34:19.194 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:34:18 smithi142 ceph-mon[35271]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-03-31T23:34:19.295 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:34:18 smithi162 ceph-mon[36237]: pgmap v717: 129 pgs: 129 active+clean; 1.2 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 6.0 MiB/s wr, 398 op/s 2024-03-31T23:34:19.295 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:34:18 smithi162 ceph-mon[36237]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-03-31T23:34:19.310 INFO:tasks.workunit.client.0.smithi046.stdout: 10 87653 8.24 MB/sec execute 444 sec latency 117.871 ms 2024-03-31T23:34:20.310 INFO:tasks.workunit.client.0.smithi046.stdout: 10 87808 8.24 MB/sec execute 445 sec latency 104.955 ms 2024-03-31T23:34:21.097 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:34:20 smithi162 ceph-mon[36237]: pgmap v718: 129 pgs: 129 active+clean; 1.2 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 5.4 MiB/s wr, 325 op/s 2024-03-31T23:34:21.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:34:20 smithi046 ceph-mon[32498]: pgmap v718: 129 pgs: 129 active+clean; 1.2 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 5.4 MiB/s wr, 325 op/s 2024-03-31T23:34:21.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:34:20 smithi142 ceph-mon[35271]: pgmap v718: 129 pgs: 129 active+clean; 1.2 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 5.4 MiB/s wr, 325 op/s 2024-03-31T23:34:21.311 INFO:tasks.workunit.client.0.smithi046.stdout: 10 87958 8.25 MB/sec execute 446 sec latency 130.570 ms 2024-03-31T23:34:21.415 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:21.415+0000 7f26080cd640 1 -- 172.21.15.46:0/3793883258 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2600101260 msgr2=0x7f260010d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:34:21.415 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:21.415+0000 7f26080cd640 1 --2- 172.21.15.46:0/3793883258 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2600101260 0x7f260010d480 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f25f8009cd0 tx=0x7f25f802c0f0 comp rx=0 tx=0).stop 2024-03-31T23:34:21.416 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:21.415+0000 7f26080cd640 1 -- 172.21.15.46:0/3793883258 shutdown_connections 2024-03-31T23:34:21.416 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:21.415+0000 7f26080cd640 1 --2- 172.21.15.46:0/3793883258 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2600101260 0x7f260010d480 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:34:21.416 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:21.415+0000 7f26080cd640 1 --2- 172.21.15.46:0/3793883258 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f26001008a0 0x7f2600100d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:34:21.416 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:21.415+0000 7f26080cd640 1 --2- 172.21.15.46:0/3793883258 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f26001055f0 0x7f26001059d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:34:21.416 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:21.415+0000 7f26080cd640 1 -- 172.21.15.46:0/3793883258 >> 172.21.15.46:0/3793883258 conn(0x7f26000774a0 msgr2=0x7f26000778b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:34:21.416 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:21.415+0000 7f26080cd640 1 -- 172.21.15.46:0/3793883258 shutdown_connections 2024-03-31T23:34:21.416 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:21.415+0000 7f26080cd640 1 -- 172.21.15.46:0/3793883258 wait complete. 2024-03-31T23:34:21.418 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:21.416+0000 7f26080cd640 1 Processor -- start 2024-03-31T23:34:21.418 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:21.416+0000 7f26080cd640 1 -- start start 2024-03-31T23:34:21.418 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:21.416+0000 7f26080cd640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f26001008a0 0x7f260019a100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:34:21.419 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:21.416+0000 7f26080cd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2600101260 0x7f260019a640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:34:21.419 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:21.416+0000 7f26080cd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f26001055f0 0x7f260019e9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:34:21.419 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:21.416+0000 7f26080cd640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f260010f8c0 con 0x7f26001055f0 2024-03-31T23:34:21.419 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:21.416+0000 7f26080cd640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f260010f740 con 0x7f26001008a0 2024-03-31T23:34:21.419 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:21.416+0000 7f26080cd640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f260010fa40 con 0x7f2600101260 2024-03-31T23:34:21.419 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:21.416+0000 7f260666a640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f26001055f0 0x7f260019e9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:34:21.419 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:21.416+0000 7f2605668640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2600101260 0x7f260019a640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:34:21.419 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:21.416+0000 7f2605668640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2600101260 0x7f260019a640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.46:54232/0 (socket says 172.21.15.46:54232) 2024-03-31T23:34:21.419 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:21.416+0000 7f2605668640 1 -- 172.21.15.46:0/1502313830 learned_addr learned my addr 172.21.15.46:0/1502313830 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:34:21.419 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:21.416+0000 7f2605e69640 1 --2- 172.21.15.46:0/1502313830 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f26001008a0 0x7f260019a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:34:21.419 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:21.417+0000 7f260666a640 1 -- 172.21.15.46:0/1502313830 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2600101260 msgr2=0x7f260019a640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:34:21.419 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:21.417+0000 7f260666a640 1 --2- 172.21.15.46:0/1502313830 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2600101260 0x7f260019a640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:34:21.420 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:21.417+0000 7f260666a640 1 -- 172.21.15.46:0/1502313830 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f26001008a0 msgr2=0x7f260019a100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:34:21.420 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:21.417+0000 7f260666a640 1 --2- 172.21.15.46:0/1502313830 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f26001008a0 0x7f260019a100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:34:21.420 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:21.417+0000 7f260666a640 1 -- 172.21.15.46:0/1502313830 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f260019f0f0 con 0x7f26001055f0 2024-03-31T23:34:21.420 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:21.417+0000 7f2605668640 1 --2- 172.21.15.46:0/1502313830 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2600101260 0x7f260019a640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:34:21.420 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:21.417+0000 7f260666a640 1 --2- 172.21.15.46:0/1502313830 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f26001055f0 0x7f260019e9c0 secure :-1 s=READY pgs=1012 cs=0 l=1 rev1=1 crypto rx=0x7f25f802c630 tx=0x7f25f8035d60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:34:21.420 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:21.417+0000 7f25f6ffd640 1 -- 172.21.15.46:0/1502313830 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25f8039040 con 0x7f26001055f0 2024-03-31T23:34:21.420 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:21.417+0000 7f26080cd640 1 -- 172.21.15.46:0/1502313830 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f260019f380 con 0x7f26001055f0 2024-03-31T23:34:21.420 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:21.417+0000 7f25f6ffd640 1 -- 172.21.15.46:0/1502313830 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f25f8044070 con 0x7f26001055f0 2024-03-31T23:34:21.420 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:21.417+0000 7f26080cd640 1 -- 172.21.15.46:0/1502313830 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f26001a6c70 con 0x7f26001055f0 2024-03-31T23:34:21.420 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:21.417+0000 7f25f6ffd640 1 -- 172.21.15.46:0/1502313830 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25f80042a0 con 0x7f26001055f0 2024-03-31T23:34:21.423 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:21.421+0000 7f260ad91740 1 Processor -- start 2024-03-31T23:34:21.423 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:21.421+0000 7f25f6ffd640 1 -- 172.21.15.46:0/1502313830 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 25) ==== 137451+0+0 (secure 0 0 0) 0x7f25f8004440 con 0x7f26001055f0 2024-03-31T23:34:21.423 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:21.421+0000 7f260ad91740 1 -- start start 2024-03-31T23:34:21.423 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:21.421+0000 7f260ad91740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5640deca46c0 0x5640deca4aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:34:21.423 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:21.421+0000 7f260ad91740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5640deca4fe0 0x5640decad870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:34:21.423 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:21.421+0000 7f260ad91740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5640decaddb0 0x5640decb01a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:34:21.423 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:21.421+0000 7f260ad91740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x5640dec078f0 con 0x5640deca46c0 2024-03-31T23:34:21.423 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:21.421+0000 7f260ad91740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x5640debb08d0 con 0x5640deca4fe0 2024-03-31T23:34:21.423 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:21.421+0000 7f260ad91740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5640dec03990 con 0x5640decaddb0 2024-03-31T23:34:21.423 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:21.422+0000 7f2605668640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5640deca46c0 0x5640deca4aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:34:21.423 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:21.422+0000 7f2605e69640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5640deca4fe0 0x5640decad870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:34:21.423 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:21.422+0000 7f2605e69640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5640deca4fe0 0x5640decad870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.46:36382/0 (socket says 172.21.15.46:36382) 2024-03-31T23:34:21.424 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:21.422+0000 7f2605e69640 1 -- 172.21.15.46:0/3082101817 learned_addr learned my addr 172.21.15.46:0/3082101817 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:34:21.424 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:21.422+0000 7f2605668640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5640deca46c0 0x5640deca4aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:42794/0 (socket says 172.21.15.46:42794) 2024-03-31T23:34:21.424 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:21.422+0000 7f2605668640 1 --2- 172.21.15.46:0/3082101817 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5640decaddb0 0x5640decb01a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:34:21.424 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:21.422+0000 7f25f6ffd640 1 --2- 172.21.15.46:0/1502313830 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f25ec0a26b0 0x7f25ec0a4b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:34:21.424 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:21.422+0000 7f260666a640 1 --2- 172.21.15.46:0/1502313830 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f25ec0a26b0 0x7f25ec0a4b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:34:21.424 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:21.422+0000 7f2605668640 1 -- 172.21.15.46:0/3082101817 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5640decaddb0 msgr2=0x5640decb01a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:34:21.424 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:21.422+0000 7f2605668640 1 --2- 172.21.15.46:0/3082101817 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5640decaddb0 0x5640decb01a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:34:21.424 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:21.422+0000 7f2605668640 1 -- 172.21.15.46:0/3082101817 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5640deca4fe0 msgr2=0x5640decad870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:34:21.424 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:21.422+0000 7f2605668640 1 --2- 172.21.15.46:0/3082101817 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5640deca4fe0 0x5640decad870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:34:21.424 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:21.422+0000 7f25f6ffd640 1 -- 172.21.15.46:0/1502313830 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(133..133 src has 1..133) ==== 9554+0+0 (secure 0 0 0) 0x7f25f80f0e00 con 0x7f26001055f0 2024-03-31T23:34:21.424 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:21.422+0000 7f2605668640 1 -- 172.21.15.46:0/3082101817 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5640de91c210 con 0x5640deca46c0 2024-03-31T23:34:21.424 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:21.422+0000 7f2605668640 1 --2- 172.21.15.46:0/3082101817 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5640deca46c0 0x5640deca4aa0 secure :-1 s=READY pgs=1013 cs=0 l=1 rev1=1 crypto rx=0x7f260010d8c0 tx=0x7f25fc005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b5f4791332a2c1f8 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:34:21.425 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:21.423+0000 7f260666a640 1 --2- 172.21.15.46:0/1502313830 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f25ec0a26b0 0x7f25ec0a4b70 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f25f80ee300 tx=0x7f25f80bd100 comp rx=0 tx=0).ready entity=mgr.15654 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:34:21.425 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:21.423+0000 7f25ebfff640 1 -- 172.21.15.46:0/3082101817 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25fc015d40 con 0x5640deca46c0 2024-03-31T23:34:21.425 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:21.423+0000 7f25ebfff640 1 -- 172.21.15.46:0/3082101817 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f25fc004030 con 0x5640deca46c0 2024-03-31T23:34:21.425 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:21.423+0000 7f25ebfff640 1 -- 172.21.15.46:0/3082101817 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25fc005280 con 0x5640deca46c0 2024-03-31T23:34:21.428 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:21.424+0000 7f260ad91740 1 -- 172.21.15.46:0/3082101817 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5640deca46c0 msgr2=0x5640deca4aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:34:21.428 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:21.424+0000 7f260ad91740 1 --2- 172.21.15.46:0/3082101817 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5640deca46c0 0x5640deca4aa0 secure :-1 s=READY pgs=1013 cs=0 l=1 rev1=1 crypto rx=0x7f260010d8c0 tx=0x7f25fc005f90 comp rx=0 tx=0).stop 2024-03-31T23:34:21.429 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:21.424+0000 7f260ad91740 1 -- 172.21.15.46:0/3082101817 shutdown_connections 2024-03-31T23:34:21.429 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:21.424+0000 7f260ad91740 1 --2- 172.21.15.46:0/3082101817 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5640decaddb0 0x5640decb01a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:34:21.429 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:21.424+0000 7f260ad91740 1 --2- 172.21.15.46:0/3082101817 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5640deca4fe0 0x5640decad870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:34:21.429 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:21.424+0000 7f260ad91740 1 --2- 172.21.15.46:0/3082101817 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5640deca46c0 0x5640deca4aa0 unknown :-1 s=CLOSED pgs=1013 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:34:21.429 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:21.424+0000 7f260ad91740 1 -- 172.21.15.46:0/3082101817 >> 172.21.15.46:0/3082101817 conn(0x5640dec9efe0 msgr2=0x5640dec9f3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:34:21.429 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:21.424+0000 7f260ad91740 1 -- 172.21.15.46:0/3082101817 shutdown_connections 2024-03-31T23:34:21.429 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:21.424+0000 7f260ad91740 1 -- 172.21.15.46:0/3082101817 wait complete. 2024-03-31T23:34:21.429 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:21.424+0000 7f260ad91740 1 -- start start 2024-03-31T23:34:21.429 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:21.424+0000 7f260ad91740 1 Processor -- start 2024-03-31T23:34:21.430 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:21.425+0000 7f260ad91740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5640deca46c0 0x5640decaade0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:34:21.430 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:21.425+0000 7f260ad91740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5640deca4fe0 0x5640decab320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:34:21.430 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:21.425+0000 7f260ad91740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5640decaddb0 0x5640decab860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:34:21.430 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:21.425+0000 7f260ad91740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x5640decb0980 con 0x5640deca46c0 2024-03-31T23:34:21.430 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:21.425+0000 7f260ad91740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x5640deba5f80 con 0x5640decaddb0 2024-03-31T23:34:21.430 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:21.425+0000 7f260ad91740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5640decb0b00 con 0x5640deca4fe0 2024-03-31T23:34:21.430 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:21.425+0000 7f260ad91740 10 client.0 _ll_register_callbacks cb 0x5640debfe290 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T23:34:21.430 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:21.425+0000 7f2605e69640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5640deca4fe0 0x5640decab320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:34:21.430 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:21.425+0000 7f2605e69640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5640deca4fe0 0x5640decab320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.46:54258/0 (socket says 172.21.15.46:54258) 2024-03-31T23:34:21.430 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:21.425+0000 7f2605e69640 1 -- 172.21.15.46:0/4066706594 learned_addr learned my addr 172.21.15.46:0/4066706594 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:34:21.430 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:21.425+0000 7f2605668640 1 --2- 172.21.15.46:0/4066706594 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5640deca46c0 0x5640decaade0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:34:21.431 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:21.425+0000 7f2605e69640 1 -- 172.21.15.46:0/4066706594 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5640decaddb0 msgr2=0x5640decab860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:34:21.431 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:21.425+0000 7f2605e69640 1 --2- 172.21.15.46:0/4066706594 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5640decaddb0 0x5640decab860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:34:21.431 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:21.425+0000 7f2605e69640 1 -- 172.21.15.46:0/4066706594 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5640deca46c0 msgr2=0x5640decaade0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:34:21.431 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:21.425+0000 7f2605e69640 1 --2- 172.21.15.46:0/4066706594 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5640deca46c0 0x5640decaade0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:34:21.431 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:21.425+0000 7f2605e69640 1 -- 172.21.15.46:0/4066706594 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5640de9e8620 con 0x5640deca4fe0 2024-03-31T23:34:21.431 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:21.426+0000 7f2605e69640 1 --2- 172.21.15.46:0/4066706594 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5640deca4fe0 0x5640decab320 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f25f0012870 tx=0x7f25f0012d30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6d162b56609782a5 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:34:21.431 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:21.426+0000 7f25e97fa640 10 client.0 ms_handle_connect on v2:172.21.15.162:3300/0 2024-03-31T23:34:21.431 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:21.426+0000 7f25e97fa640 1 -- 172.21.15.46:0/4066706594 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25f0004050 con 0x5640deca4fe0 2024-03-31T23:34:21.431 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:21.426+0000 7f25e97fa640 1 -- 172.21.15.46:0/4066706594 <== mon.1 v2:172.21.15.162:3300/0 2 ==== osd_map(133..133 src has 1..133) ==== 9554+0+0 (secure 0 0 0) 0x7f25f00041f0 con 0x5640deca4fe0 2024-03-31T23:34:21.431 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:21.426+0000 7f260ad91740 1 -- 172.21.15.46:0/4066706594 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f25fc004030 con 0x5640deca4fe0 2024-03-31T23:34:21.431 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:21.426+0000 7f25e97fa640 1 -- 172.21.15.46:0/4066706594 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_get_version_reply(handle=1 version=52) ==== 24+0+0 (secure 0 0 0) 0x7f25f00043f0 con 0x5640deca4fe0 2024-03-31T23:34:21.431 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:21.426+0000 7f260ad91740 10 client.27691 fetch_fsmap learned FSMap version 52 2024-03-31T23:34:21.432 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:21.426+0000 7f260ad91740 1 -- 172.21.15.46:0/4066706594 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,fsmap=52,monmap=4+}) -- 0x5640decbc980 con 0x5640deca4fe0 2024-03-31T23:34:21.432 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:21.427+0000 7f25e97fa640 1 -- 172.21.15.46:0/4066706594 <== mon.1 v2:172.21.15.162:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f25fc004030 con 0x5640deca4fe0 2024-03-31T23:34:21.432 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:21.427+0000 7f25e97fa640 1 -- 172.21.15.46:0/4066706594 <== mon.1 v2:172.21.15.162:3300/0 5 ==== fsmap(e 52) ==== 8354+0+0 (secure 0 0 0) 0x7f25f0003b10 con 0x5640deca4fe0 2024-03-31T23:34:21.432 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:21.427+0000 7f260ad91740 10 client.27691 fetch_fsmap finished waiting for FSMap version 52 2024-03-31T23:34:21.432 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:21.427+0000 7f260ad91740 10 client.27691 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-31T23:34:21.432 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:21.427+0000 7f260ad91740 20 client.27691 populate_metadata read hostname 'smithi046' 2024-03-31T23:34:21.432 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:21.427+0000 7f260ad91740 1 --2- 172.21.15.46:0/4066706594 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x5640decbdf00 0x5640decc0330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:34:21.432 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:21.427+0000 7f260ad91740 4 client.27691 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-31T23:34:21.432 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:21.427+0000 7f260ad91740 1 -- 172.21.15.46:0/4066706594 --> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5640dec1b120 con 0x5640decbdf00 2024-03-31T23:34:21.432 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:21.427+0000 7f2605668640 1 --2- 172.21.15.46:0/4066706594 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x5640decbdf00 0x5640decc0330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:34:21.432 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:21.428+0000 7f2605668640 1 --2- 172.21.15.46:0/4066706594 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x5640decbdf00 0x5640decc0330 crc :-1 s=READY pgs=278 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=535d4fbfd8c9a99c server_cookie=a4b87162ff5925c5 in_seq=0 out_seq=0 2024-03-31T23:34:21.432 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:21.428+0000 7f25e97fa640 10 client.27691 ms_handle_connect on v2:172.21.15.142:6838/2519143918 2024-03-31T23:34:21.434 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:21.431+0000 7f25e97fa640 1 -- 172.21.15.46:0/4066706594 <== mds.0 v2:172.21.15.142:6838/2519143918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5640dec1b120 con 0x5640decbdf00 2024-03-31T23:34:21.434 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:21.431+0000 7f25e97fa640 10 client.27691 handle_command_reply: tid=0 2024-03-31T23:34:21.434 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:21.431+0000 7f260ad91740 1 client.27691 shutdown 2024-03-31T23:34:21.435 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:21.431+0000 7f260ad91740 1 -- 172.21.15.46:0/4066706594 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5640deca4fe0 msgr2=0x5640decab320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:34:21.435 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:21.431+0000 7f260ad91740 1 --2- 172.21.15.46:0/4066706594 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5640deca4fe0 0x5640decab320 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f25f0012870 tx=0x7f25f0012d30 comp rx=0 tx=0).stop 2024-03-31T23:34:21.435 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:21.432+0000 7f260ad91740 1 -- 172.21.15.46:0/4066706594 shutdown_connections 2024-03-31T23:34:21.435 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:21.432+0000 7f260ad91740 1 --2- 172.21.15.46:0/4066706594 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x5640decbdf00 0x5640decc0330 crc :-1 s=READY pgs=278 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:34:21.435 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:21.432+0000 7f260ad91740 1 --2- 172.21.15.46:0/4066706594 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5640decaddb0 0x5640decab860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:34:21.435 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:21.432+0000 7f25e97fa640 0 client.27691 ms_handle_reset on v2:172.21.15.142:6838/2519143918 2024-03-31T23:34:21.435 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:21.432+0000 7f260ad91740 1 --2- 172.21.15.46:0/4066706594 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5640deca4fe0 0x5640decab320 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:34:21.435 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:21.432+0000 7f260ad91740 1 --2- 172.21.15.46:0/4066706594 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5640deca46c0 0x5640decaade0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:34:21.435 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:21.432+0000 7f260ad91740 1 -- 172.21.15.46:0/4066706594 >> 172.21.15.46:0/4066706594 conn(0x5640dec9efe0 msgr2=0x5640deca0d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:34:21.435 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:21.432+0000 7f260ad91740 1 -- 172.21.15.46:0/4066706594 shutdown_connections 2024-03-31T23:34:21.435 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:21.432+0000 7f260ad91740 1 -- 172.21.15.46:0/4066706594 wait complete. 2024-03-31T23:34:21.435 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:21.432+0000 7f260ad91740 20 client.27691 trim_cache size 0 max 16384 2024-03-31T23:34:21.470 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:21.470+0000 7f260ad91740 1 Processor -- start 2024-03-31T23:34:21.470 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:21.470+0000 7f260ad91740 1 -- start start 2024-03-31T23:34:21.470 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:21.470+0000 7f260ad91740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5640deca46c0 0x5640decad510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:34:21.470 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:21.470+0000 7f260ad91740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5640deca4fe0 0x5640decb9780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:34:21.471 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:21.470+0000 7f260ad91740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5640decaddb0 0x5640dec9f3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:34:21.471 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:21.470+0000 7f260ad91740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f25e40129b0 con 0x5640deca4fe0 2024-03-31T23:34:21.471 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:21.470+0000 7f260ad91740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f25e4010220 con 0x5640decaddb0 2024-03-31T23:34:21.472 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:21.470+0000 7f260ad91740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f25e400e090 con 0x5640deca46c0 2024-03-31T23:34:21.472 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:21.470+0000 7f2605e69640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5640deca4fe0 0x5640decb9780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:34:21.472 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:21.470+0000 7f2605e69640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5640deca4fe0 0x5640decb9780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39286/0 (socket says 172.21.15.46:39286) 2024-03-31T23:34:21.472 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:21.470+0000 7f2605e69640 1 -- 172.21.15.46:0/2072794633 learned_addr learned my addr 172.21.15.46:0/2072794633 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:34:21.472 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:21.470+0000 7f2605668640 1 --2- 172.21.15.46:0/2072794633 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5640deca46c0 0x5640decad510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:34:21.472 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:21.470+0000 7f2605668640 1 --2- 172.21.15.46:0/2072794633 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5640decaddb0 0x5640dec9f3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:34:21.472 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:21.471+0000 7f2605668640 1 -- 172.21.15.46:0/2072794633 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5640decaddb0 msgr2=0x5640dec9f3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:34:21.472 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:21.471+0000 7f2605668640 1 --2- 172.21.15.46:0/2072794633 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5640decaddb0 0x5640dec9f3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:34:21.472 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:21.471+0000 7f2605668640 1 -- 172.21.15.46:0/2072794633 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5640deca4fe0 msgr2=0x5640decb9780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:34:21.472 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:21.471+0000 7f2605668640 1 --2- 172.21.15.46:0/2072794633 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5640deca4fe0 0x5640decb9780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:34:21.472 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:21.471+0000 7f2605668640 1 -- 172.21.15.46:0/2072794633 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5640deca9d90 con 0x5640deca46c0 2024-03-31T23:34:21.472 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:21.471+0000 7f2605668640 1 --2- 172.21.15.46:0/2072794633 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5640deca46c0 0x5640decad510 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f25fc00b4b0 tx=0x7f25fc015d40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b96325ffd31dc75a server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:34:21.472 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:21.471+0000 7f25eb7fe640 1 -- 172.21.15.46:0/2072794633 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25fc009dd0 con 0x5640deca46c0 2024-03-31T23:34:21.472 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:21.471+0000 7f25eb7fe640 1 -- 172.21.15.46:0/2072794633 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f25fc02d1d0 con 0x5640deca46c0 2024-03-31T23:34:21.472 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:21.471+0000 7f25eb7fe640 1 -- 172.21.15.46:0/2072794633 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25fc003850 con 0x5640deca46c0 2024-03-31T23:34:21.476 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:21.472+0000 7f260ad91740 1 -- 172.21.15.46:0/2072794633 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5640deca46c0 msgr2=0x5640decad510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:34:21.476 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:21.472+0000 7f260ad91740 1 --2- 172.21.15.46:0/2072794633 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5640deca46c0 0x5640decad510 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f25fc00b4b0 tx=0x7f25fc015d40 comp rx=0 tx=0).stop 2024-03-31T23:34:21.476 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:21.472+0000 7f260ad91740 1 -- 172.21.15.46:0/2072794633 shutdown_connections 2024-03-31T23:34:21.476 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:21.472+0000 7f260ad91740 1 --2- 172.21.15.46:0/2072794633 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5640decaddb0 0x5640dec9f3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:34:21.476 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:21.472+0000 7f260ad91740 1 --2- 172.21.15.46:0/2072794633 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5640deca4fe0 0x5640decb9780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:34:21.476 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:21.472+0000 7f260ad91740 1 --2- 172.21.15.46:0/2072794633 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5640deca46c0 0x5640decad510 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:34:21.476 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:21.473+0000 7f260ad91740 1 -- 172.21.15.46:0/2072794633 >> 172.21.15.46:0/2072794633 conn(0x5640dec9efe0 msgr2=0x5640dec9df80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:34:21.476 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:21.473+0000 7f260ad91740 1 -- 172.21.15.46:0/2072794633 shutdown_connections 2024-03-31T23:34:21.476 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:21.473+0000 7f260ad91740 1 -- 172.21.15.46:0/2072794633 wait complete. 2024-03-31T23:34:21.476 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:21.473+0000 7f260ad91740 1 -- start start 2024-03-31T23:34:21.476 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:21.473+0000 7f260ad91740 1 Processor -- start 2024-03-31T23:34:21.476 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:21.473+0000 7f260ad91740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5640deca46c0 0x5640decb1e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:34:21.476 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:21.474+0000 7f260ad91740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5640deca4fe0 0x5640decb2390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:34:21.477 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:21.474+0000 7f260ad91740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5640decaddb0 0x5640ded23270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:34:21.477 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:21.474+0000 7f260ad91740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f25e400e6e0 con 0x5640deca46c0 2024-03-31T23:34:21.477 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:21.474+0000 7f2605668640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5640deca46c0 0x5640decb1e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:34:21.477 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:21.474+0000 7f260ad91740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f25e400fb50 con 0x5640decaddb0 2024-03-31T23:34:21.477 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:21.474+0000 7f2605668640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5640deca46c0 0x5640decb1e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39302/0 (socket says 172.21.15.46:39302) 2024-03-31T23:34:21.477 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:21.474+0000 7f2605668640 1 -- 172.21.15.46:0/3190605706 learned_addr learned my addr 172.21.15.46:0/3190605706 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:34:21.477 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:21.474+0000 7f260ad91740 1 -- 172.21.15.46:0/3190605706 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f25e400edb0 con 0x5640deca4fe0 2024-03-31T23:34:21.477 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:21.474+0000 7f2605e69640 1 --2- 172.21.15.46:0/3190605706 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5640deca4fe0 0x5640decb2390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:34:21.477 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:21.474+0000 7f260ad91740 10 client.0 _ll_register_callbacks cb 0x7f25f0009870 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T23:34:21.477 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:21.474+0000 7f2605668640 1 --2- 172.21.15.46:0/3190605706 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5640decaddb0 0x5640ded23270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:34:21.477 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:21.474+0000 7f2605668640 1 -- 172.21.15.46:0/3190605706 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5640deca4fe0 msgr2=0x5640decb2390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:34:21.477 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:21.474+0000 7f2605668640 1 --2- 172.21.15.46:0/3190605706 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5640deca4fe0 0x5640decb2390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:34:21.477 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:21.474+0000 7f2605668640 1 -- 172.21.15.46:0/3190605706 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5640decaddb0 msgr2=0x5640ded23270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:34:21.478 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:21.474+0000 7f2605668640 1 --2- 172.21.15.46:0/3190605706 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5640decaddb0 0x5640ded23270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:34:21.478 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:21.474+0000 7f2605668640 1 -- 172.21.15.46:0/3190605706 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5640ded23830 con 0x5640deca46c0 2024-03-31T23:34:21.478 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:21.474+0000 7f2605668640 1 --2- 172.21.15.46:0/3190605706 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5640deca46c0 0x5640decb1e50 secure :-1 s=READY pgs=1014 cs=0 l=1 rev1=1 crypto rx=0x7f25fc00b800 tx=0x7f25fc000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac65fa8d84b3936e server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:34:21.478 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:21.474+0000 7f25e97fa640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-31T23:34:21.478 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:21.475+0000 7f25e97fa640 1 -- 172.21.15.46:0/3190605706 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25fc005250 con 0x5640deca46c0 2024-03-31T23:34:21.478 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:21.475+0000 7f25e97fa640 1 -- 172.21.15.46:0/3190605706 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(133..133 src has 1..133) ==== 9554+0+0 (secure 0 0 0) 0x7f25fc0053f0 con 0x5640deca46c0 2024-03-31T23:34:21.478 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:21.475+0000 7f260ad91740 1 -- 172.21.15.46:0/3190605706 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f25fc02d1d0 con 0x5640deca46c0 2024-03-31T23:34:21.478 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:21.475+0000 7f25e97fa640 1 -- 172.21.15.46:0/3190605706 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=52) ==== 24+0+0 (secure 0 0 0) 0x7f25fc0055f0 con 0x5640deca46c0 2024-03-31T23:34:21.478 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:21.475+0000 7f260ad91740 10 client.17982 fetch_fsmap learned FSMap version 52 2024-03-31T23:34:21.478 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:21.475+0000 7f260ad91740 1 -- 172.21.15.46:0/3190605706 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=52,monmap=4+}) -- 0x5640ded24b00 con 0x5640deca46c0 2024-03-31T23:34:21.478 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:21.475+0000 7f25e97fa640 1 -- 172.21.15.46:0/3190605706 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f25fc02d1d0 con 0x5640deca46c0 2024-03-31T23:34:21.478 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:21.476+0000 7f25e97fa640 1 -- 172.21.15.46:0/3190605706 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 52) ==== 8354+0+0 (secure 0 0 0) 0x7f25fc038020 con 0x5640deca46c0 2024-03-31T23:34:21.478 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:21.476+0000 7f260ad91740 10 client.17982 fetch_fsmap finished waiting for FSMap version 52 2024-03-31T23:34:21.478 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:21.476+0000 7f260ad91740 10 client.17982 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-31T23:34:21.478 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:21.476+0000 7f260ad91740 20 client.17982 populate_metadata read hostname 'smithi046' 2024-03-31T23:34:21.479 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:21.476+0000 7f260ad91740 1 --2- 172.21.15.46:0/3190605706 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x5640decbdf00 0x5640ded27180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:34:21.479 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:21.476+0000 7f260ad91740 4 client.17982 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-31T23:34:21.479 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:21.476+0000 7f260ad91740 1 -- 172.21.15.46:0/3190605706 --> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5640dec14e30 con 0x5640decbdf00 2024-03-31T23:34:21.479 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:21.476+0000 7f2605e69640 1 --2- 172.21.15.46:0/3190605706 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x5640decbdf00 0x5640ded27180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:34:21.479 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:21.476+0000 7f25e97fa640 10 client.17982 ms_handle_connect on v2:172.21.15.142:6838/2519143918 2024-03-31T23:34:21.479 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:21.476+0000 7f2605e69640 1 --2- 172.21.15.46:0/3190605706 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x5640decbdf00 0x5640ded27180 crc :-1 s=READY pgs=279 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=db625344bb0bd278 server_cookie=f9a107770f61df97 in_seq=0 out_seq=0 2024-03-31T23:34:21.720 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:21.719+0000 7f25e97fa640 1 -- 172.21.15.46:0/3190605706 <== mds.0 v2:172.21.15.142:6838/2519143918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5640dec14e30 con 0x5640decbdf00 2024-03-31T23:34:21.720 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:21.719+0000 7f25e97fa640 10 client.17982 handle_command_reply: tid=0 2024-03-31T23:34:21.720 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:21.719+0000 7f260ad91740 1 client.17982 shutdown 2024-03-31T23:34:21.720 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:21.719+0000 7f260ad91740 1 -- 172.21.15.46:0/3190605706 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5640deca46c0 msgr2=0x5640decb1e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:34:21.720 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:21.719+0000 7f260ad91740 1 --2- 172.21.15.46:0/3190605706 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5640deca46c0 0x5640decb1e50 secure :-1 s=READY pgs=1014 cs=0 l=1 rev1=1 crypto rx=0x7f25fc00b800 tx=0x7f25fc000bc0 comp rx=0 tx=0).stop 2024-03-31T23:34:21.720 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:21.719+0000 7f260ad91740 1 -- 172.21.15.46:0/3190605706 shutdown_connections 2024-03-31T23:34:21.720 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:21.719+0000 7f260ad91740 1 --2- 172.21.15.46:0/3190605706 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x5640decbdf00 0x5640ded27180 crc :-1 s=READY pgs=279 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:34:21.720 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:21.719+0000 7f260ad91740 1 --2- 172.21.15.46:0/3190605706 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5640decaddb0 0x5640ded23270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:34:21.721 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:21.719+0000 7f260ad91740 1 --2- 172.21.15.46:0/3190605706 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5640deca4fe0 0x5640decb2390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:34:21.721 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:21.719+0000 7f260ad91740 1 --2- 172.21.15.46:0/3190605706 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5640deca46c0 0x5640decb1e50 unknown :-1 s=CLOSED pgs=1014 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:34:21.721 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:21.719+0000 7f260ad91740 1 -- 172.21.15.46:0/3190605706 >> 172.21.15.46:0/3190605706 conn(0x5640dec9efe0 msgr2=0x5640dec9df80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:34:21.721 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:21.719+0000 7f25e97fa640 0 client.17982 ms_handle_reset on v2:172.21.15.142:6838/2519143918 2024-03-31T23:34:21.721 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:21.719+0000 7f260ad91740 1 -- 172.21.15.46:0/3190605706 shutdown_connections 2024-03-31T23:34:21.721 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:21.720+0000 7f260ad91740 1 -- 172.21.15.46:0/3190605706 wait complete. 2024-03-31T23:34:21.721 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:21.720+0000 7f260ad91740 20 client.17982 trim_cache size 0 max 16384 2024-03-31T23:34:21.721 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout:{ 2024-03-31T23:34:21.721 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout: "message": "", 2024-03-31T23:34:21.721 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout: "return_code": 0 2024-03-31T23:34:21.721 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout:} 2024-03-31T23:34:21.722 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:21.720+0000 7f26080cd640 1 -- 172.21.15.46:0/1502313830 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f25ec0a26b0 msgr2=0x7f25ec0a4b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:34:21.722 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:21.720+0000 7f26080cd640 1 --2- 172.21.15.46:0/1502313830 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f25ec0a26b0 0x7f25ec0a4b70 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f25f80ee300 tx=0x7f25f80bd100 comp rx=0 tx=0).stop 2024-03-31T23:34:21.722 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:21.720+0000 7f26080cd640 1 -- 172.21.15.46:0/1502313830 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f26001055f0 msgr2=0x7f260019e9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:34:21.722 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:21.720+0000 7f26080cd640 1 --2- 172.21.15.46:0/1502313830 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f26001055f0 0x7f260019e9c0 secure :-1 s=READY pgs=1012 cs=0 l=1 rev1=1 crypto rx=0x7f25f802c630 tx=0x7f25f8035d60 comp rx=0 tx=0).stop 2024-03-31T23:34:21.723 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:21.721+0000 7f26080cd640 1 -- 172.21.15.46:0/1502313830 shutdown_connections 2024-03-31T23:34:21.723 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:21.721+0000 7f26080cd640 1 --2- 172.21.15.46:0/1502313830 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f25ec0a26b0 0x7f25ec0a4b70 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:34:21.723 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:21.721+0000 7f26080cd640 1 --2- 172.21.15.46:0/1502313830 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f26001055f0 0x7f260019e9c0 unknown :-1 s=CLOSED pgs=1012 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:34:21.723 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:21.721+0000 7f26080cd640 1 --2- 172.21.15.46:0/1502313830 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2600101260 0x7f260019a640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:34:21.723 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:21.721+0000 7f26080cd640 1 --2- 172.21.15.46:0/1502313830 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f26001008a0 0x7f260019a100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:34:21.723 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:21.721+0000 7f26080cd640 1 -- 172.21.15.46:0/1502313830 >> 172.21.15.46:0/1502313830 conn(0x7f26000774a0 msgr2=0x7f26000fe770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:34:21.723 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:21.721+0000 7f26080cd640 1 -- 172.21.15.46:0/1502313830 shutdown_connections 2024-03-31T23:34:21.723 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:21.721+0000 7f26080cd640 1 -- 172.21.15.46:0/1502313830 wait complete. 2024-03-31T23:34:22.310 INFO:tasks.workunit.client.0.smithi046.stdout: 10 88095 8.24 MB/sec execute 447 sec latency 194.404 ms 2024-03-31T23:34:23.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:34:22 smithi046 ceph-mon[32498]: pgmap v719: 129 pgs: 129 active+clean; 1.2 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 5.2 MiB/s wr, 313 op/s 2024-03-31T23:34:23.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:34:22 smithi142 ceph-mon[35271]: pgmap v719: 129 pgs: 129 active+clean; 1.2 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 5.2 MiB/s wr, 313 op/s 2024-03-31T23:34:23.295 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:34:22 smithi162 ceph-mon[36237]: pgmap v719: 129 pgs: 129 active+clean; 1.2 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 5.2 MiB/s wr, 313 op/s 2024-03-31T23:34:23.311 INFO:tasks.workunit.client.0.smithi046.stdout: 10 88254 8.25 MB/sec execute 448 sec latency 99.020 ms 2024-03-31T23:34:24.311 INFO:tasks.workunit.client.0.smithi046.stdout: 10 88406 8.25 MB/sec execute 449 sec latency 98.132 ms 2024-03-31T23:34:25.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:34:24 smithi046 ceph-mon[32498]: pgmap v720: 129 pgs: 129 active+clean; 1.2 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 4.5 MiB/s wr, 270 op/s 2024-03-31T23:34:25.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:34:24 smithi142 ceph-mon[35271]: pgmap v720: 129 pgs: 129 active+clean; 1.2 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 4.5 MiB/s wr, 270 op/s 2024-03-31T23:34:25.295 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:34:24 smithi162 ceph-mon[36237]: pgmap v720: 129 pgs: 129 active+clean; 1.2 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 4.5 MiB/s wr, 270 op/s 2024-03-31T23:34:25.311 INFO:tasks.workunit.client.0.smithi046.stdout: 10 88566 8.25 MB/sec execute 450 sec latency 93.187 ms 2024-03-31T23:34:26.311 INFO:tasks.workunit.client.0.smithi046.stdout: 10 88764 8.26 MB/sec execute 451 sec latency 119.173 ms 2024-03-31T23:34:27.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:34:26 smithi046 ceph-mon[32498]: pgmap v721: 129 pgs: 129 active+clean; 1.2 GiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 6.8 MiB/s wr, 426 op/s 2024-03-31T23:34:27.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:34:26 smithi142 ceph-mon[35271]: pgmap v721: 129 pgs: 129 active+clean; 1.2 GiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 6.8 MiB/s wr, 426 op/s 2024-03-31T23:34:27.295 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:34:26 smithi162 ceph-mon[36237]: pgmap v721: 129 pgs: 129 active+clean; 1.2 GiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 6.8 MiB/s wr, 426 op/s 2024-03-31T23:34:27.311 INFO:tasks.workunit.client.0.smithi046.stdout: 10 88938 8.25 MB/sec execute 452 sec latency 107.967 ms 2024-03-31T23:34:28.311 INFO:tasks.workunit.client.0.smithi046.stdout: 10 89068 8.25 MB/sec execute 453 sec latency 91.468 ms 2024-03-31T23:34:29.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:34:28 smithi046 ceph-mon[32498]: pgmap v722: 129 pgs: 129 active+clean; 1.2 GiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 4.4 MiB/s wr, 278 op/s 2024-03-31T23:34:29.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:34:28 smithi142 ceph-mon[35271]: pgmap v722: 129 pgs: 129 active+clean; 1.2 GiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 4.4 MiB/s wr, 278 op/s 2024-03-31T23:34:29.295 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:34:28 smithi162 ceph-mon[36237]: pgmap v722: 129 pgs: 129 active+clean; 1.2 GiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 4.4 MiB/s wr, 278 op/s 2024-03-31T23:34:29.311 INFO:tasks.workunit.client.0.smithi046.stdout: 10 89234 8.25 MB/sec execute 454 sec latency 80.494 ms 2024-03-31T23:34:30.311 INFO:tasks.workunit.client.0.smithi046.stdout: 10 89376 8.26 MB/sec execute 455 sec latency 90.867 ms 2024-03-31T23:34:31.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:34:30 smithi046 ceph-mon[32498]: pgmap v723: 129 pgs: 129 active+clean; 1.2 GiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 7.0 MiB/s wr, 371 op/s 2024-03-31T23:34:31.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:34:30 smithi142 ceph-mon[35271]: pgmap v723: 129 pgs: 129 active+clean; 1.2 GiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 7.0 MiB/s wr, 371 op/s 2024-03-31T23:34:31.295 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:34:30 smithi162 ceph-mon[36237]: pgmap v723: 129 pgs: 129 active+clean; 1.2 GiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 7.0 MiB/s wr, 371 op/s 2024-03-31T23:34:31.311 INFO:tasks.workunit.client.0.smithi046.stdout: 10 89511 8.25 MB/sec execute 456 sec latency 101.229 ms 2024-03-31T23:34:32.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:34:31 smithi046 ceph-mon[32498]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-03-31T23:34:32.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:34:31 smithi142 ceph-mon[35271]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-03-31T23:34:32.295 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:34:31 smithi162 ceph-mon[36237]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-03-31T23:34:32.311 INFO:tasks.workunit.client.0.smithi046.stdout: 10 89660 8.27 MB/sec execute 457 sec latency 113.708 ms 2024-03-31T23:34:33.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:34:32 smithi046 ceph-mon[32498]: pgmap v724: 129 pgs: 129 active+clean; 1.2 GiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 4.8 MiB/s wr, 248 op/s 2024-03-31T23:34:33.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:34:32 smithi046 ceph-mon[32498]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-31T23:34:33.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:34:32 smithi142 ceph-mon[35271]: pgmap v724: 129 pgs: 129 active+clean; 1.2 GiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 4.8 MiB/s wr, 248 op/s 2024-03-31T23:34:33.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:34:32 smithi142 ceph-mon[35271]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-31T23:34:33.295 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:34:32 smithi162 ceph-mon[36237]: pgmap v724: 129 pgs: 129 active+clean; 1.2 GiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 4.8 MiB/s wr, 248 op/s 2024-03-31T23:34:33.295 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:34:32 smithi162 ceph-mon[36237]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-31T23:34:33.311 INFO:tasks.workunit.client.0.smithi046.stdout: 10 89800 8.26 MB/sec execute 458 sec latency 79.054 ms 2024-03-31T23:34:34.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:34:33 smithi046 ceph-mon[32498]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:34:34.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:34:33 smithi046 ceph-mon[32498]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-31T23:34:34.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:34:33 smithi046 ceph-mon[32498]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' 2024-03-31T23:34:34.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:34:33 smithi142 ceph-mon[35271]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:34:34.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:34:33 smithi142 ceph-mon[35271]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-31T23:34:34.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:34:33 smithi142 ceph-mon[35271]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' 2024-03-31T23:34:34.295 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:34:33 smithi162 ceph-mon[36237]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:34:34.296 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:34:33 smithi162 ceph-mon[36237]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-31T23:34:34.296 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:34:33 smithi162 ceph-mon[36237]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' 2024-03-31T23:34:34.311 INFO:tasks.workunit.client.0.smithi046.stdout: 10 89955 8.26 MB/sec execute 459 sec latency 120.960 ms 2024-03-31T23:34:34.844 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:34.841+0000 7fc4f2bac640 1 -- 172.21.15.46:0/2330106001 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc4ec1055f0 msgr2=0x7fc4ec1059d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:34:34.844 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:34.841+0000 7fc4f2bac640 1 --2- 172.21.15.46:0/2330106001 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc4ec1055f0 0x7fc4ec1059d0 secure :-1 s=READY pgs=1015 cs=0 l=1 rev1=1 crypto rx=0x7fc4e0009900 tx=0x7fc4e002bc30 comp rx=0 tx=0).stop 2024-03-31T23:34:34.844 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:34.841+0000 7fc4f2bac640 1 -- 172.21.15.46:0/2330106001 shutdown_connections 2024-03-31T23:34:34.844 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:34.841+0000 7fc4f2bac640 1 --2- 172.21.15.46:0/2330106001 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc4ec101260 0x7fc4ec10d480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:34:34.844 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:34.841+0000 7fc4f2bac640 1 --2- 172.21.15.46:0/2330106001 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc4ec1008a0 0x7fc4ec100d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:34:34.844 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:34.841+0000 7fc4f2bac640 1 --2- 172.21.15.46:0/2330106001 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc4ec1055f0 0x7fc4ec1059d0 unknown :-1 s=CLOSED pgs=1015 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:34:34.844 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:34.841+0000 7fc4f2bac640 1 -- 172.21.15.46:0/2330106001 >> 172.21.15.46:0/2330106001 conn(0x7fc4ec0774a0 msgr2=0x7fc4ec0778b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:34:34.844 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:34.841+0000 7fc4f2bac640 1 -- 172.21.15.46:0/2330106001 shutdown_connections 2024-03-31T23:34:34.845 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:34.841+0000 7fc4f2bac640 1 -- 172.21.15.46:0/2330106001 wait complete. 2024-03-31T23:34:34.845 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:34.841+0000 7fc4f2bac640 1 Processor -- start 2024-03-31T23:34:34.845 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:34.842+0000 7fc4f2bac640 1 -- start start 2024-03-31T23:34:34.845 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:34.842+0000 7fc4f2bac640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc4ec1008a0 0x7fc4ec19a0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:34:34.845 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:34.842+0000 7fc4f2bac640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc4ec101260 0x7fc4ec19a630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:34:34.845 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:34.842+0000 7fc4f2bac640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc4ec1055f0 0x7fc4ec19e9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:34:34.845 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:34.842+0000 7fc4f2bac640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fc4ec111a90 con 0x7fc4ec1055f0 2024-03-31T23:34:34.845 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:34.842+0000 7fc4f2bac640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fc4ec111910 con 0x7fc4ec1008a0 2024-03-31T23:34:34.845 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:34.842+0000 7fc4f2bac640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fc4ec111c10 con 0x7fc4ec101260 2024-03-31T23:34:34.845 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:34.842+0000 7fc4ebfff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc4ec101260 0x7fc4ec19a630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:34:34.845 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:34.842+0000 7fc4f0948640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc4ec1008a0 0x7fc4ec19a0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:34:34.845 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:34.842+0000 7fc4ebfff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc4ec101260 0x7fc4ec19a630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.46:41322/0 (socket says 172.21.15.46:41322) 2024-03-31T23:34:34.845 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:34.842+0000 7fc4ebfff640 1 -- 172.21.15.46:0/1227385449 learned_addr learned my addr 172.21.15.46:0/1227385449 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:34:34.846 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:34.842+0000 7fc4f0948640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc4ec1008a0 0x7fc4ec19a0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.46:60774/0 (socket says 172.21.15.46:60774) 2024-03-31T23:34:34.846 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:34.842+0000 7fc4f1149640 1 --2- 172.21.15.46:0/1227385449 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc4ec1055f0 0x7fc4ec19e9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:34:34.846 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:34.842+0000 7fc4ebfff640 1 -- 172.21.15.46:0/1227385449 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc4ec1008a0 msgr2=0x7fc4ec19a0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:34:34.846 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:34.842+0000 7fc4ebfff640 1 --2- 172.21.15.46:0/1227385449 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc4ec1008a0 0x7fc4ec19a0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:34:34.846 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:34.842+0000 7fc4ebfff640 1 -- 172.21.15.46:0/1227385449 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc4ec1055f0 msgr2=0x7fc4ec19e9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:34:34.846 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:34.842+0000 7fc4ebfff640 1 --2- 172.21.15.46:0/1227385449 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc4ec1055f0 0x7fc4ec19e9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:34:34.846 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:34.842+0000 7fc4ebfff640 1 -- 172.21.15.46:0/1227385449 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc4ec19f0e0 con 0x7fc4ec101260 2024-03-31T23:34:34.846 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:34.843+0000 7fc4f1149640 1 --2- 172.21.15.46:0/1227385449 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc4ec1055f0 0x7fc4ec19e9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:34:34.846 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:34.843+0000 7fc4ebfff640 1 --2- 172.21.15.46:0/1227385449 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc4ec101260 0x7fc4ec19a630 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fc4d800e9f0 tx=0x7fc4d800eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:34:34.846 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:34.843+0000 7fc4e9ffb640 1 -- 172.21.15.46:0/1227385449 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4d800cd80 con 0x7fc4ec101260 2024-03-31T23:34:34.846 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:34.843+0000 7fc4f2bac640 1 -- 172.21.15.46:0/1227385449 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc4ec19f3d0 con 0x7fc4ec101260 2024-03-31T23:34:34.846 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:34.843+0000 7fc4e9ffb640 1 -- 172.21.15.46:0/1227385449 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc4d80108e0 con 0x7fc4ec101260 2024-03-31T23:34:34.846 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:34.843+0000 7fc4e9ffb640 1 -- 172.21.15.46:0/1227385449 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4d8018850 con 0x7fc4ec101260 2024-03-31T23:34:34.846 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:34.843+0000 7fc4f2bac640 1 -- 172.21.15.46:0/1227385449 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc4ec1a6c60 con 0x7fc4ec101260 2024-03-31T23:34:34.847 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:34.846+0000 7fc4e9ffb640 1 -- 172.21.15.46:0/1227385449 <== mon.1 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 25) ==== 137451+0+0 (secure 0 0 0) 0x7fc4d8010a90 con 0x7fc4ec101260 2024-03-31T23:34:34.847 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:34.846+0000 7fc4f578e740 1 Processor -- start 2024-03-31T23:34:34.848 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:34.846+0000 7fc4f578e740 1 -- start start 2024-03-31T23:34:34.848 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:34.847+0000 7fc4f578e740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d6b55ac6c0 0x55d6b55acaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:34:34.848 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:34.847+0000 7fc4f578e740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d6b55acfe0 0x55d6b55b5870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:34:34.848 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:34.847+0000 7fc4f578e740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d6b55b5db0 0x55d6b55b81a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:34:34.848 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:34.847+0000 7fc4f578e740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55d6b550f8f0 con 0x55d6b55ac6c0 2024-03-31T23:34:34.848 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:34.847+0000 7fc4f578e740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55d6b54b88d0 con 0x55d6b55b5db0 2024-03-31T23:34:34.848 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:34.847+0000 7fc4f578e740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55d6b550b990 con 0x55d6b55acfe0 2024-03-31T23:34:34.849 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:34.847+0000 7fc4f1149640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d6b55ac6c0 0x55d6b55acaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:34:34.849 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:34.847+0000 7fc4f1149640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d6b55ac6c0 0x55d6b55acaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:56822/0 (socket says 172.21.15.46:56822) 2024-03-31T23:34:34.849 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:34.847+0000 7fc4f1149640 1 -- 172.21.15.46:0/3818625948 learned_addr learned my addr 172.21.15.46:0/3818625948 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:34:34.849 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:34.847+0000 7fc4ebfff640 1 --2- 172.21.15.46:0/3818625948 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d6b55b5db0 0x55d6b55b81a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:34:34.849 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:34.847+0000 7fc4f0948640 1 --2- 172.21.15.46:0/3818625948 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d6b55acfe0 0x55d6b55b5870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:34:34.849 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:34.847+0000 7fc4f1149640 1 -- 172.21.15.46:0/3818625948 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d6b55acfe0 msgr2=0x55d6b55b5870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:34:34.849 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:34.847+0000 7fc4f1149640 1 --2- 172.21.15.46:0/3818625948 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d6b55acfe0 0x55d6b55b5870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:34:34.850 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:34.847+0000 7fc4f1149640 1 -- 172.21.15.46:0/3818625948 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d6b55b5db0 msgr2=0x55d6b55b81a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:34:34.850 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:34.847+0000 7fc4f1149640 1 --2- 172.21.15.46:0/3818625948 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d6b55b5db0 0x55d6b55b81a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:34:34.850 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:34.847+0000 7fc4f1149640 1 -- 172.21.15.46:0/3818625948 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d6b5224210 con 0x55d6b55ac6c0 2024-03-31T23:34:34.850 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:34.847+0000 7fc4e9ffb640 1 --2- 172.21.15.46:0/1227385449 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fc4dc0a2730 0x7fc4dc0a4bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:34:34.850 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:34.848+0000 7fc4f1149640 1 --2- 172.21.15.46:0/3818625948 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d6b55ac6c0 0x55d6b55acaa0 secure :-1 s=READY pgs=1016 cs=0 l=1 rev1=1 crypto rx=0x7fc4e40117f0 tx=0x7fc4e4011cb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f88dcc32689439b server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:34:34.850 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:34.848+0000 7fc4f0948640 1 --2- 172.21.15.46:0/1227385449 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fc4dc0a2730 0x7fc4dc0a4bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:34:34.850 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:34.848+0000 7fc4e9ffb640 1 -- 172.21.15.46:0/1227385449 <== mon.1 v2:172.21.15.162:3300/0 5 ==== osd_map(133..133 src has 1..133) ==== 9554+0+0 (secure 0 0 0) 0x7fc4d80cf220 con 0x7fc4ec101260 2024-03-31T23:34:34.850 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:34.848+0000 7fc4d6ffd640 1 -- 172.21.15.46:0/3818625948 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4e40112f0 con 0x55d6b55ac6c0 2024-03-31T23:34:34.850 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:34.848+0000 7fc4d6ffd640 1 -- 172.21.15.46:0/3818625948 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc4e4003e20 con 0x55d6b55ac6c0 2024-03-31T23:34:34.850 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:34.848+0000 7fc4d6ffd640 1 -- 172.21.15.46:0/3818625948 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4e4004440 con 0x55d6b55ac6c0 2024-03-31T23:34:34.852 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:34.849+0000 7fc4f578e740 1 -- 172.21.15.46:0/3818625948 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d6b55ac6c0 msgr2=0x55d6b55acaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:34:34.852 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:34.849+0000 7fc4f0948640 1 --2- 172.21.15.46:0/1227385449 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fc4dc0a2730 0x7fc4dc0a4bf0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fc4e0006010 tx=0x7fc4e002c6c0 comp rx=0 tx=0).ready entity=mgr.15654 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:34:34.853 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:34.849+0000 7fc4f578e740 1 --2- 172.21.15.46:0/3818625948 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d6b55ac6c0 0x55d6b55acaa0 secure :-1 s=READY pgs=1016 cs=0 l=1 rev1=1 crypto rx=0x7fc4e40117f0 tx=0x7fc4e4011cb0 comp rx=0 tx=0).stop 2024-03-31T23:34:34.853 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:34.849+0000 7fc4f578e740 1 -- 172.21.15.46:0/3818625948 shutdown_connections 2024-03-31T23:34:34.853 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:34.849+0000 7fc4f578e740 1 --2- 172.21.15.46:0/3818625948 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d6b55b5db0 0x55d6b55b81a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:34:34.853 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:34.849+0000 7fc4f578e740 1 --2- 172.21.15.46:0/3818625948 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d6b55acfe0 0x55d6b55b5870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:34:34.853 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:34.849+0000 7fc4f578e740 1 --2- 172.21.15.46:0/3818625948 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d6b55ac6c0 0x55d6b55acaa0 unknown :-1 s=CLOSED pgs=1016 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:34:34.853 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:34.849+0000 7fc4f578e740 1 -- 172.21.15.46:0/3818625948 >> 172.21.15.46:0/3818625948 conn(0x55d6b55a6fe0 msgr2=0x55d6b55a73c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:34:34.853 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:34.849+0000 7fc4f578e740 1 -- 172.21.15.46:0/3818625948 shutdown_connections 2024-03-31T23:34:34.853 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:34.849+0000 7fc4f578e740 1 -- 172.21.15.46:0/3818625948 wait complete. 2024-03-31T23:34:34.853 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:34.849+0000 7fc4f578e740 1 -- start start 2024-03-31T23:34:34.853 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:34.850+0000 7fc4f578e740 1 Processor -- start 2024-03-31T23:34:34.853 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:34.850+0000 7fc4f578e740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d6b55ac6c0 0x55d6b55b2e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:34:34.853 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:34.850+0000 7fc4f578e740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d6b55acfe0 0x55d6b55b3380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:34:34.853 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:34.850+0000 7fc4f578e740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d6b55b5db0 0x55d6b55b38c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:34:34.854 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:34.850+0000 7fc4f578e740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55d6b55b89e0 con 0x55d6b55ac6c0 2024-03-31T23:34:34.854 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:34.850+0000 7fc4f578e740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55d6b54adf80 con 0x55d6b55b5db0 2024-03-31T23:34:34.854 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:34.850+0000 7fc4f578e740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55d6b55b8b60 con 0x55d6b55acfe0 2024-03-31T23:34:34.854 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:34.850+0000 7fc4f578e740 10 client.0 _ll_register_callbacks cb 0x55d6b5506290 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T23:34:34.854 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:34.850+0000 7fc4f0948640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d6b55ac6c0 0x55d6b55b2e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:34:34.854 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:34.850+0000 7fc4ebfff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d6b55acfe0 0x55d6b55b3380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:34:34.854 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:34.850+0000 7fc4f0948640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d6b55ac6c0 0x55d6b55b2e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:56834/0 (socket says 172.21.15.46:56834) 2024-03-31T23:34:34.854 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:34.850+0000 7fc4f0948640 1 -- 172.21.15.46:0/1444928953 learned_addr learned my addr 172.21.15.46:0/1444928953 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:34:34.854 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:34.850+0000 7fc4ebfff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d6b55acfe0 0x55d6b55b3380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.46:41340/0 (socket says 172.21.15.46:41340) 2024-03-31T23:34:34.854 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:34.850+0000 7fc4f1149640 1 --2- 172.21.15.46:0/1444928953 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d6b55b5db0 0x55d6b55b38c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:34:34.854 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:34.851+0000 7fc4f1149640 1 -- 172.21.15.46:0/1444928953 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d6b55acfe0 msgr2=0x55d6b55b3380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:34:34.854 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:34.851+0000 7fc4f1149640 1 --2- 172.21.15.46:0/1444928953 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d6b55acfe0 0x55d6b55b3380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:34:34.854 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:34.851+0000 7fc4f1149640 1 -- 172.21.15.46:0/1444928953 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d6b55ac6c0 msgr2=0x55d6b55b2e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:34:34.854 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:34.851+0000 7fc4f1149640 1 --2- 172.21.15.46:0/1444928953 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d6b55ac6c0 0x55d6b55b2e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:34:34.855 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:34.851+0000 7fc4f1149640 1 -- 172.21.15.46:0/1444928953 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d6b52f0620 con 0x55d6b55b5db0 2024-03-31T23:34:34.855 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:34.851+0000 7fc4f0948640 1 --2- 172.21.15.46:0/1444928953 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d6b55ac6c0 0x55d6b55b2e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:34:34.855 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:34.851+0000 7fc4f1149640 1 --2- 172.21.15.46:0/1444928953 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d6b55b5db0 0x55d6b55b38c0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fc4e4005c10 tx=0x7fc4e4011390 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a78b74bccb0eab7 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:34:34.855 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:34.851+0000 7fc4cbfff640 10 client.0 ms_handle_connect on v2:172.21.15.142:3300/0 2024-03-31T23:34:34.855 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:34.852+0000 7fc4cbfff640 1 -- 172.21.15.46:0/1444928953 <== mon.2 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4e4004e50 con 0x55d6b55b5db0 2024-03-31T23:34:34.855 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:34.852+0000 7fc4f578e740 1 -- 172.21.15.46:0/1444928953 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc4e4003e20 con 0x55d6b55b5db0 2024-03-31T23:34:34.855 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:34.852+0000 7fc4cbfff640 1 -- 172.21.15.46:0/1444928953 <== mon.2 v2:172.21.15.142:3300/0 2 ==== osd_map(133..133 src has 1..133) ==== 9554+0+0 (secure 0 0 0) 0x7fc4e400f9a0 con 0x55d6b55b5db0 2024-03-31T23:34:34.857 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:34.852+0000 7fc4cbfff640 1 -- 172.21.15.46:0/1444928953 <== mon.2 v2:172.21.15.142:3300/0 3 ==== mon_get_version_reply(handle=1 version=53) ==== 24+0+0 (secure 0 0 0) 0x7fc4e400fba0 con 0x55d6b55b5db0 2024-03-31T23:34:34.857 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:34.852+0000 7fc4f578e740 10 client.27806 fetch_fsmap learned FSMap version 53 2024-03-31T23:34:34.857 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:34.852+0000 7fc4f578e740 1 -- 172.21.15.46:0/1444928953 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,fsmap=53,monmap=4+}) -- 0x55d6b55c4910 con 0x55d6b55b5db0 2024-03-31T23:34:34.857 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:34.852+0000 7fc4cbfff640 1 -- 172.21.15.46:0/1444928953 <== mon.2 v2:172.21.15.142:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc4e4003e20 con 0x55d6b55b5db0 2024-03-31T23:34:34.857 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:34.852+0000 7fc4cbfff640 1 -- 172.21.15.46:0/1444928953 <== mon.2 v2:172.21.15.142:3300/0 5 ==== fsmap(e 53) ==== 8350+0+0 (secure 0 0 0) 0x7fc4e4018020 con 0x55d6b55b5db0 2024-03-31T23:34:34.857 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:34.853+0000 7fc4f578e740 10 client.27806 fetch_fsmap finished waiting for FSMap version 53 2024-03-31T23:34:34.857 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:34.853+0000 7fc4f578e740 10 client.27806 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-31T23:34:34.857 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:34.853+0000 7fc4f578e740 20 client.27806 populate_metadata read hostname 'smithi046' 2024-03-31T23:34:34.857 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:34.853+0000 7fc4f578e740 1 --2- 172.21.15.46:0/1444928953 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55d6b55c5c60 0x55d6b55c8050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:34:34.857 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:34.853+0000 7fc4f578e740 4 client.27806 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-31T23:34:34.857 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:34.853+0000 7fc4f578e740 1 -- 172.21.15.46:0/1444928953 --> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d6b5523120 con 0x55d6b55c5c60 2024-03-31T23:34:34.858 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:34.853+0000 7fc4f0948640 1 --2- 172.21.15.46:0/1444928953 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55d6b55c5c60 0x55d6b55c8050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:34:34.858 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:34.853+0000 7fc4f0948640 1 --2- 172.21.15.46:0/1444928953 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55d6b55c5c60 0x55d6b55c8050 crc :-1 s=READY pgs=280 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=26552034241695a6 server_cookie=d125797121fd54af in_seq=0 out_seq=0 2024-03-31T23:34:34.858 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:34.853+0000 7fc4cbfff640 10 client.27806 ms_handle_connect on v2:172.21.15.142:6838/2519143918 2024-03-31T23:34:34.858 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:34.855+0000 7fc4cbfff640 1 -- 172.21.15.46:0/1444928953 <== mds.0 v2:172.21.15.142:6838/2519143918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55d6b5523120 con 0x55d6b55c5c60 2024-03-31T23:34:34.858 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:34.855+0000 7fc4cbfff640 10 client.27806 handle_command_reply: tid=0 2024-03-31T23:34:34.858 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:34.855+0000 7fc4f578e740 1 client.27806 shutdown 2024-03-31T23:34:34.858 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:34.855+0000 7fc4f578e740 1 -- 172.21.15.46:0/1444928953 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d6b55b5db0 msgr2=0x55d6b55b38c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:34:34.858 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:34.855+0000 7fc4f578e740 1 --2- 172.21.15.46:0/1444928953 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d6b55b5db0 0x55d6b55b38c0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fc4e4005c10 tx=0x7fc4e4011390 comp rx=0 tx=0).stop 2024-03-31T23:34:34.858 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:34.855+0000 7fc4f578e740 1 -- 172.21.15.46:0/1444928953 shutdown_connections 2024-03-31T23:34:34.858 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:34.855+0000 7fc4f578e740 1 --2- 172.21.15.46:0/1444928953 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55d6b55c5c60 0x55d6b55c8050 crc :-1 s=READY pgs=280 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:34:34.858 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:34.855+0000 7fc4f578e740 1 --2- 172.21.15.46:0/1444928953 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d6b55b5db0 0x55d6b55b38c0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:34:34.858 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:34.855+0000 7fc4f578e740 1 --2- 172.21.15.46:0/1444928953 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d6b55acfe0 0x55d6b55b3380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:34:34.858 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:34.855+0000 7fc4f578e740 1 --2- 172.21.15.46:0/1444928953 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d6b55ac6c0 0x55d6b55b2e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:34:34.859 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:34.855+0000 7fc4cbfff640 0 client.27806 ms_handle_reset on v2:172.21.15.142:6838/2519143918 2024-03-31T23:34:34.859 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:34.855+0000 7fc4f578e740 1 -- 172.21.15.46:0/1444928953 >> 172.21.15.46:0/1444928953 conn(0x55d6b55a6fe0 msgr2=0x55d6b55a8d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:34:34.859 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:34.855+0000 7fc4f578e740 1 -- 172.21.15.46:0/1444928953 shutdown_connections 2024-03-31T23:34:34.859 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:34.855+0000 7fc4f578e740 1 -- 172.21.15.46:0/1444928953 wait complete. 2024-03-31T23:34:34.859 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:34.855+0000 7fc4f578e740 20 client.27806 trim_cache size 0 max 16384 2024-03-31T23:34:34.883 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:34.881+0000 7fc4f578e740 1 Processor -- start 2024-03-31T23:34:34.883 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:34.881+0000 7fc4f578e740 1 -- start start 2024-03-31T23:34:34.883 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:34.881+0000 7fc4f578e740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d6b55ac6c0 0x55d6b55b5570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:34:34.883 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:34.881+0000 7fc4f578e740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d6b55acfe0 0x55d6b55c1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:34:34.884 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:34.881+0000 7fc4f578e740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d6b55b5db0 0x55d6b55a5e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:34:34.884 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:34.881+0000 7fc4f578e740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fc4ec1b36d0 con 0x55d6b55acfe0 2024-03-31T23:34:34.884 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:34.881+0000 7fc4f578e740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fc4ec1b0f70 con 0x55d6b55ac6c0 2024-03-31T23:34:34.884 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:34.881+0000 7fc4f578e740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fc4ec1aede0 con 0x55d6b55b5db0 2024-03-31T23:34:34.884 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:34.881+0000 7fc4ebfff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d6b55acfe0 0x55d6b55c1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:34:34.884 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:34.881+0000 7fc4ebfff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d6b55acfe0 0x55d6b55c1780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:56842/0 (socket says 172.21.15.46:56842) 2024-03-31T23:34:34.884 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:34.881+0000 7fc4f0948640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d6b55ac6c0 0x55d6b55b5570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:34:34.884 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:34.882+0000 7fc4ebfff640 1 -- 172.21.15.46:0/1222533195 learned_addr learned my addr 172.21.15.46:0/1222533195 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:34:34.884 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:34.882+0000 7fc4f1149640 1 --2- 172.21.15.46:0/1222533195 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d6b55b5db0 0x55d6b55a5e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:34:34.884 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:34.882+0000 7fc4ebfff640 1 -- 172.21.15.46:0/1222533195 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d6b55b5db0 msgr2=0x55d6b55a5e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:34:34.884 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:34.882+0000 7fc4ebfff640 1 --2- 172.21.15.46:0/1222533195 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d6b55b5db0 0x55d6b55a5e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:34:34.884 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:34.882+0000 7fc4ebfff640 1 -- 172.21.15.46:0/1222533195 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d6b55ac6c0 msgr2=0x55d6b55b5570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:34:34.884 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:34.882+0000 7fc4ebfff640 1 --2- 172.21.15.46:0/1222533195 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d6b55ac6c0 0x55d6b55b5570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:34:34.884 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:34.882+0000 7fc4ebfff640 1 -- 172.21.15.46:0/1222533195 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d6b55c1cc0 con 0x55d6b55acfe0 2024-03-31T23:34:34.885 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:34.882+0000 7fc4ebfff640 1 --2- 172.21.15.46:0/1222533195 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d6b55acfe0 0x55d6b55c1780 secure :-1 s=READY pgs=1017 cs=0 l=1 rev1=1 crypto rx=0x7fc4d809b2e0 tx=0x7fc4d8086040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b2f6a9bff42d1643 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:34:34.885 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:34.882+0000 7fc4d67fc640 1 -- 172.21.15.46:0/1222533195 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4d809b3b0 con 0x55d6b55acfe0 2024-03-31T23:34:34.885 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:34.882+0000 7fc4d67fc640 1 -- 172.21.15.46:0/1222533195 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc4d8010de0 con 0x55d6b55acfe0 2024-03-31T23:34:34.885 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:34.882+0000 7fc4d67fc640 1 -- 172.21.15.46:0/1222533195 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4d809af40 con 0x55d6b55acfe0 2024-03-31T23:34:34.888 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:34.883+0000 7fc4f578e740 1 -- 172.21.15.46:0/1222533195 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d6b55acfe0 msgr2=0x55d6b55c1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:34:34.888 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:34.883+0000 7fc4f578e740 1 --2- 172.21.15.46:0/1222533195 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d6b55acfe0 0x55d6b55c1780 secure :-1 s=READY pgs=1017 cs=0 l=1 rev1=1 crypto rx=0x7fc4d809b2e0 tx=0x7fc4d8086040 comp rx=0 tx=0).stop 2024-03-31T23:34:34.888 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:34.883+0000 7fc4f578e740 1 -- 172.21.15.46:0/1222533195 shutdown_connections 2024-03-31T23:34:34.888 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:34.883+0000 7fc4f578e740 1 --2- 172.21.15.46:0/1222533195 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d6b55b5db0 0x55d6b55a5e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:34:34.888 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:34.883+0000 7fc4f578e740 1 --2- 172.21.15.46:0/1222533195 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d6b55acfe0 0x55d6b55c1780 unknown :-1 s=CLOSED pgs=1017 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:34:34.888 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:34.883+0000 7fc4f578e740 1 --2- 172.21.15.46:0/1222533195 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d6b55ac6c0 0x55d6b55b5570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:34:34.888 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:34.883+0000 7fc4f578e740 1 -- 172.21.15.46:0/1222533195 >> 172.21.15.46:0/1222533195 conn(0x55d6b55a6fe0 msgr2=0x55d6b55a56f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:34:34.888 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:34.883+0000 7fc4f578e740 1 -- 172.21.15.46:0/1222533195 shutdown_connections 2024-03-31T23:34:34.888 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:34.883+0000 7fc4f578e740 1 -- 172.21.15.46:0/1222533195 wait complete. 2024-03-31T23:34:34.888 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:34.884+0000 7fc4f578e740 1 -- start start 2024-03-31T23:34:34.888 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:34.884+0000 7fc4f578e740 1 Processor -- start 2024-03-31T23:34:34.888 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:34.884+0000 7fc4f578e740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d6b55ac6c0 0x55d6b55bddb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:34:34.889 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:34.884+0000 7fc4f578e740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d6b55acfe0 0x55d6b55af4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:34:34.889 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:34.884+0000 7fc4f578e740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d6b55b5db0 0x55d6b55af9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:34:34.889 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:34.884+0000 7fc4f578e740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fc4ec1af430 con 0x55d6b55b5db0 2024-03-31T23:34:34.889 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:34.884+0000 7fc4f578e740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fc4ec1b08a0 con 0x55d6b55acfe0 2024-03-31T23:34:34.889 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:34.884+0000 7fc4f578e740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fc4ec1afb00 con 0x55d6b55ac6c0 2024-03-31T23:34:34.889 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:34.884+0000 7fc4f578e740 10 client.0 _ll_register_callbacks cb 0x7fc4d80d2ba0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T23:34:34.889 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:34.884+0000 7fc4f1149640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d6b55b5db0 0x55d6b55af9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:34:34.889 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:34.884+0000 7fc4f0948640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d6b55ac6c0 0x55d6b55bddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:34:34.889 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:34.884+0000 7fc4ebfff640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d6b55acfe0 0x55d6b55af4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:34:34.889 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:34.884+0000 7fc4f1149640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d6b55b5db0 0x55d6b55af9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:56846/0 (socket says 172.21.15.46:56846) 2024-03-31T23:34:34.889 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:34.884+0000 7fc4f1149640 1 -- 172.21.15.46:0/886612533 learned_addr learned my addr 172.21.15.46:0/886612533 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:34:34.889 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:34.885+0000 7fc4f1149640 1 -- 172.21.15.46:0/886612533 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d6b55ac6c0 msgr2=0x55d6b55bddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:34:34.889 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:34.885+0000 7fc4f1149640 1 --2- 172.21.15.46:0/886612533 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d6b55ac6c0 0x55d6b55bddb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:34:34.889 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:34.885+0000 7fc4f1149640 1 -- 172.21.15.46:0/886612533 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d6b55acfe0 msgr2=0x55d6b55af4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:34:34.890 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:34.885+0000 7fc4f1149640 1 --2- 172.21.15.46:0/886612533 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d6b55acfe0 0x55d6b55af4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:34:34.890 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:34.885+0000 7fc4f1149640 1 -- 172.21.15.46:0/886612533 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d6b55aff20 con 0x55d6b55b5db0 2024-03-31T23:34:34.890 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:34.885+0000 7fc4f1149640 1 --2- 172.21.15.46:0/886612533 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d6b55b5db0 0x55d6b55af9e0 secure :-1 s=READY pgs=1018 cs=0 l=1 rev1=1 crypto rx=0x7fc4e4011cf0 tx=0x7fc4e4005cc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f37fed0d594c3a4f server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:34:34.890 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:34.885+0000 7fc4cbfff640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-31T23:34:34.890 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:34.885+0000 7fc4cbfff640 1 -- 172.21.15.46:0/886612533 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4e400f9c0 con 0x55d6b55b5db0 2024-03-31T23:34:34.890 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:34.885+0000 7fc4f578e740 1 -- 172.21.15.46:0/886612533 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55d6b551ce30 con 0x55d6b55b5db0 2024-03-31T23:34:34.890 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:34.885+0000 7fc4cbfff640 1 -- 172.21.15.46:0/886612533 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(133..133 src has 1..133) ==== 9554+0+0 (secure 0 0 0) 0x7fc4e4004c50 con 0x55d6b55b5db0 2024-03-31T23:34:34.890 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:34.885+0000 7fc4cbfff640 1 -- 172.21.15.46:0/886612533 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=53) ==== 24+0+0 (secure 0 0 0) 0x7fc4e4004e50 con 0x55d6b55b5db0 2024-03-31T23:34:34.890 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:34.885+0000 7fc4f578e740 10 client.18000 fetch_fsmap learned FSMap version 53 2024-03-31T23:34:34.890 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:34.885+0000 7fc4f578e740 1 -- 172.21.15.46:0/886612533 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=53,monmap=4+}) -- 0x55d6b562ba40 con 0x55d6b55b5db0 2024-03-31T23:34:34.890 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:34.885+0000 7fc4cbfff640 1 -- 172.21.15.46:0/886612533 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x55d6b551ce30 con 0x55d6b55b5db0 2024-03-31T23:34:34.890 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:34.886+0000 7fc4cbfff640 1 -- 172.21.15.46:0/886612533 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 53) ==== 8350+0+0 (secure 0 0 0) 0x7fc4e4033030 con 0x55d6b55b5db0 2024-03-31T23:34:34.890 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:34.886+0000 7fc4f578e740 10 client.18000 fetch_fsmap finished waiting for FSMap version 53 2024-03-31T23:34:34.890 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:34.886+0000 7fc4f578e740 10 client.18000 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-31T23:34:34.891 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:34.886+0000 7fc4f578e740 20 client.18000 populate_metadata read hostname 'smithi046' 2024-03-31T23:34:34.891 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:34.886+0000 7fc4f578e740 1 --2- 172.21.15.46:0/886612533 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55d6b562c040 0x55d6b562e460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:34:34.891 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:34.886+0000 7fc4f578e740 4 client.18000 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-31T23:34:34.891 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:34.886+0000 7fc4f578e740 1 -- 172.21.15.46:0/886612533 --> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55d6b5517180 con 0x55d6b562c040 2024-03-31T23:34:34.891 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:34.886+0000 7fc4f0948640 1 --2- 172.21.15.46:0/886612533 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55d6b562c040 0x55d6b562e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:34:34.891 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:34.886+0000 7fc4cbfff640 10 client.18000 ms_handle_connect on v2:172.21.15.142:6838/2519143918 2024-03-31T23:34:34.891 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:34.886+0000 7fc4f0948640 1 --2- 172.21.15.46:0/886612533 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55d6b562c040 0x55d6b562e460 crc :-1 s=READY pgs=281 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5ebf21ff31aa4fa server_cookie=a7c1761afd5eae12 in_seq=0 out_seq=0 2024-03-31T23:34:35.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:34:34 smithi046 ceph-mon[32498]: pgmap v725: 129 pgs: 129 active+clean; 1.2 GiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 4.8 MiB/s wr, 248 op/s 2024-03-31T23:34:35.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:34:34 smithi142 ceph-mon[35271]: pgmap v725: 129 pgs: 129 active+clean; 1.2 GiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 4.8 MiB/s wr, 248 op/s 2024-03-31T23:34:35.231 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:35.230+0000 7fc4cbfff640 1 -- 172.21.15.46:0/886612533 <== mds.0 v2:172.21.15.142:6838/2519143918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55d6b5517180 con 0x55d6b562c040 2024-03-31T23:34:35.232 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:35.230+0000 7fc4cbfff640 10 client.18000 handle_command_reply: tid=0 2024-03-31T23:34:35.232 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:35.230+0000 7fc4f578e740 1 client.18000 shutdown 2024-03-31T23:34:35.232 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:35.230+0000 7fc4f578e740 1 -- 172.21.15.46:0/886612533 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d6b55b5db0 msgr2=0x55d6b55af9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:34:35.232 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:35.230+0000 7fc4f578e740 1 --2- 172.21.15.46:0/886612533 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d6b55b5db0 0x55d6b55af9e0 secure :-1 s=READY pgs=1018 cs=0 l=1 rev1=1 crypto rx=0x7fc4e4011cf0 tx=0x7fc4e4005cc0 comp rx=0 tx=0).stop 2024-03-31T23:34:35.232 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:35.230+0000 7fc4f578e740 1 -- 172.21.15.46:0/886612533 shutdown_connections 2024-03-31T23:34:35.232 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:35.230+0000 7fc4f578e740 1 --2- 172.21.15.46:0/886612533 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55d6b562c040 0x55d6b562e460 crc :-1 s=READY pgs=281 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:34:35.232 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:35.230+0000 7fc4f578e740 1 --2- 172.21.15.46:0/886612533 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d6b55b5db0 0x55d6b55af9e0 unknown :-1 s=CLOSED pgs=1018 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:34:35.232 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:35.230+0000 7fc4f578e740 1 --2- 172.21.15.46:0/886612533 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d6b55acfe0 0x55d6b55af4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:34:35.232 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:35.230+0000 7fc4f578e740 1 --2- 172.21.15.46:0/886612533 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d6b55ac6c0 0x55d6b55bddb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:34:35.232 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:35.230+0000 7fc4f578e740 1 -- 172.21.15.46:0/886612533 >> 172.21.15.46:0/886612533 conn(0x55d6b55a6fe0 msgr2=0x55d6b55a5f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:34:35.232 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:35.230+0000 7fc4cbfff640 0 client.18000 ms_handle_reset on v2:172.21.15.142:6838/2519143918 2024-03-31T23:34:35.232 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:35.230+0000 7fc4f578e740 1 -- 172.21.15.46:0/886612533 shutdown_connections 2024-03-31T23:34:35.233 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:35.230+0000 7fc4f578e740 1 -- 172.21.15.46:0/886612533 wait complete. 2024-03-31T23:34:35.233 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:35.230+0000 7fc4f578e740 20 client.18000 trim_cache size 0 max 16384 2024-03-31T23:34:35.233 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout:{ 2024-03-31T23:34:35.233 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout: "message": "", 2024-03-31T23:34:35.233 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout: "return_code": 0 2024-03-31T23:34:35.233 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout:} 2024-03-31T23:34:35.234 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:35.231+0000 7fc4f2bac640 1 -- 172.21.15.46:0/1227385449 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fc4dc0a2730 msgr2=0x7fc4dc0a4bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:34:35.234 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:35.231+0000 7fc4f2bac640 1 --2- 172.21.15.46:0/1227385449 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fc4dc0a2730 0x7fc4dc0a4bf0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fc4e0006010 tx=0x7fc4e002c6c0 comp rx=0 tx=0).stop 2024-03-31T23:34:35.234 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:35.231+0000 7fc4f2bac640 1 -- 172.21.15.46:0/1227385449 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc4ec101260 msgr2=0x7fc4ec19a630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:34:35.234 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:35.231+0000 7fc4f2bac640 1 --2- 172.21.15.46:0/1227385449 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc4ec101260 0x7fc4ec19a630 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fc4d800e9f0 tx=0x7fc4d800eeb0 comp rx=0 tx=0).stop 2024-03-31T23:34:35.234 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:35.231+0000 7fc4f2bac640 1 -- 172.21.15.46:0/1227385449 shutdown_connections 2024-03-31T23:34:35.234 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:35.231+0000 7fc4f2bac640 1 --2- 172.21.15.46:0/1227385449 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fc4dc0a2730 0x7fc4dc0a4bf0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:34:35.234 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:35.231+0000 7fc4f2bac640 1 --2- 172.21.15.46:0/1227385449 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc4ec1055f0 0x7fc4ec19e9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:34:35.234 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:35.231+0000 7fc4f2bac640 1 --2- 172.21.15.46:0/1227385449 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc4ec101260 0x7fc4ec19a630 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:34:35.235 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:35.231+0000 7fc4f2bac640 1 --2- 172.21.15.46:0/1227385449 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc4ec1008a0 0x7fc4ec19a0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:34:35.235 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:35.231+0000 7fc4f2bac640 1 -- 172.21.15.46:0/1227385449 >> 172.21.15.46:0/1227385449 conn(0x7fc4ec0774a0 msgr2=0x7fc4ec0fe710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:34:35.235 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:35.231+0000 7fc4f2bac640 1 -- 172.21.15.46:0/1227385449 shutdown_connections 2024-03-31T23:34:35.235 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:35.231+0000 7fc4f2bac640 1 -- 172.21.15.46:0/1227385449 wait complete. 2024-03-31T23:34:35.295 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:34:34 smithi162 ceph-mon[36237]: pgmap v725: 129 pgs: 129 active+clean; 1.2 GiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 4.8 MiB/s wr, 248 op/s 2024-03-31T23:34:35.311 INFO:tasks.workunit.client.0.smithi046.stdout: 10 90070 8.26 MB/sec execute 460 sec latency 232.986 ms 2024-03-31T23:34:36.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:34:35 smithi142 ceph-mon[35271]: pgmap v726: 129 pgs: 129 active+clean; 1.2 GiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 7.4 MiB/s wr, 359 op/s 2024-03-31T23:34:36.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:34:35 smithi142 ceph-mon[35271]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-03-31T23:34:36.295 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:34:35 smithi162 ceph-mon[36237]: pgmap v726: 129 pgs: 129 active+clean; 1.2 GiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 7.4 MiB/s wr, 359 op/s 2024-03-31T23:34:36.295 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:34:35 smithi162 ceph-mon[36237]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-03-31T23:34:36.311 INFO:tasks.workunit.client.0.smithi046.stdout: 10 90198 8.25 MB/sec execute 461 sec latency 78.936 ms 2024-03-31T23:34:36.361 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:34:35 smithi046 ceph-mon[32498]: pgmap v726: 129 pgs: 129 active+clean; 1.2 GiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 7.4 MiB/s wr, 359 op/s 2024-03-31T23:34:36.361 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:34:35 smithi046 ceph-mon[32498]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-03-31T23:34:37.311 INFO:tasks.workunit.client.0.smithi046.stdout: 10 90325 8.24 MB/sec execute 462 sec latency 107.052 ms 2024-03-31T23:34:38.312 INFO:tasks.workunit.client.0.smithi046.stdout: 10 90485 8.24 MB/sec execute 463 sec latency 140.355 ms 2024-03-31T23:34:38.361 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:34:38 smithi046 ceph-mon[32498]: pgmap v727: 129 pgs: 129 active+clean; 1.2 GiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 5.2 MiB/s wr, 203 op/s 2024-03-31T23:34:38.443 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:34:38 smithi142 ceph-mon[35271]: pgmap v727: 129 pgs: 129 active+clean; 1.2 GiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 5.2 MiB/s wr, 203 op/s 2024-03-31T23:34:38.545 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:34:38 smithi162 ceph-mon[36237]: pgmap v727: 129 pgs: 129 active+clean; 1.2 GiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 5.2 MiB/s wr, 203 op/s 2024-03-31T23:34:39.312 INFO:tasks.workunit.client.0.smithi046.stdout: 10 90643 8.24 MB/sec execute 464 sec latency 88.608 ms 2024-03-31T23:34:40.312 INFO:tasks.workunit.client.0.smithi046.stdout: 10 90781 8.25 MB/sec execute 465 sec latency 138.686 ms 2024-03-31T23:34:40.361 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:34:40 smithi046 ceph-mon[32498]: pgmap v728: 129 pgs: 129 active+clean; 1.2 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 7.3 MiB/s wr, 391 op/s 2024-03-31T23:34:40.443 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:34:40 smithi142 ceph-mon[35271]: pgmap v728: 129 pgs: 129 active+clean; 1.2 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 7.3 MiB/s wr, 391 op/s 2024-03-31T23:34:40.545 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:34:40 smithi162 ceph-mon[36237]: pgmap v728: 129 pgs: 129 active+clean; 1.2 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 7.3 MiB/s wr, 391 op/s 2024-03-31T23:34:41.312 INFO:tasks.workunit.client.0.smithi046.stdout: 10 90916 8.24 MB/sec execute 466 sec latency 67.857 ms 2024-03-31T23:34:42.312 INFO:tasks.workunit.client.0.smithi046.stdout: 10 91069 8.25 MB/sec execute 467 sec latency 70.614 ms 2024-03-31T23:34:42.361 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:34:42 smithi046 ceph-mon[32498]: pgmap v729: 129 pgs: 129 active+clean; 1.2 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 4.7 MiB/s wr, 298 op/s 2024-03-31T23:34:42.443 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:34:42 smithi142 ceph-mon[35271]: pgmap v729: 129 pgs: 129 active+clean; 1.2 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 4.7 MiB/s wr, 298 op/s 2024-03-31T23:34:42.545 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:34:42 smithi162 ceph-mon[36237]: pgmap v729: 129 pgs: 129 active+clean; 1.2 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 4.7 MiB/s wr, 298 op/s 2024-03-31T23:34:43.312 INFO:tasks.workunit.client.0.smithi046.stdout: 10 91203 8.25 MB/sec execute 468 sec latency 217.137 ms 2024-03-31T23:34:44.312 INFO:tasks.workunit.client.0.smithi046.stdout: 10 91335 8.24 MB/sec execute 469 sec latency 118.056 ms 2024-03-31T23:34:44.361 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:34:44 smithi046 ceph-mon[32498]: pgmap v730: 129 pgs: 129 active+clean; 1.2 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 4.7 MiB/s wr, 298 op/s 2024-03-31T23:34:44.443 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:34:44 smithi142 ceph-mon[35271]: pgmap v730: 129 pgs: 129 active+clean; 1.2 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 4.7 MiB/s wr, 298 op/s 2024-03-31T23:34:44.545 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:34:44 smithi162 ceph-mon[36237]: pgmap v730: 129 pgs: 129 active+clean; 1.2 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 4.7 MiB/s wr, 298 op/s 2024-03-31T23:34:45.312 INFO:tasks.workunit.client.0.smithi046.stdout: 10 91445 8.23 MB/sec execute 470 sec latency 81.929 ms 2024-03-31T23:34:46.312 INFO:tasks.workunit.client.0.smithi046.stdout: 10 91596 8.23 MB/sec execute 471 sec latency 72.780 ms 2024-03-31T23:34:46.361 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:34:46 smithi046 ceph-mon[32498]: pgmap v731: 129 pgs: 129 active+clean; 1.2 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 7.1 MiB/s wr, 411 op/s 2024-03-31T23:34:46.443 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:34:46 smithi142 ceph-mon[35271]: pgmap v731: 129 pgs: 129 active+clean; 1.2 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 7.1 MiB/s wr, 411 op/s 2024-03-31T23:34:46.545 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:34:46 smithi162 ceph-mon[36237]: pgmap v731: 129 pgs: 129 active+clean; 1.2 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 7.1 MiB/s wr, 411 op/s 2024-03-31T23:34:47.312 INFO:tasks.workunit.client.0.smithi046.stdout: 10 91749 8.22 MB/sec execute 472 sec latency 75.142 ms 2024-03-31T23:34:48.312 INFO:tasks.workunit.client.0.smithi046.stdout: 10 91909 8.23 MB/sec execute 473 sec latency 97.277 ms 2024-03-31T23:34:48.359 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:48.358+0000 7fa66c24e640 1 -- 172.21.15.46:0/3948736964 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa6641007b0 msgr2=0x7fa6641103e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:34:48.360 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:48.358+0000 7fa66c24e640 1 --2- 172.21.15.46:0/3948736964 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa6641007b0 0x7fa6641103e0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fa65c009ca0 tx=0x7fa65c02c190 comp rx=0 tx=0).stop 2024-03-31T23:34:48.360 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:48.358+0000 7fa66c24e640 1 -- 172.21.15.46:0/3948736964 shutdown_connections 2024-03-31T23:34:48.360 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:48.358+0000 7fa66c24e640 1 --2- 172.21.15.46:0/3948736964 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa6641007b0 0x7fa6641103e0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:34:48.360 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:48.358+0000 7fa66c24e640 1 --2- 172.21.15.46:0/3948736964 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa664102a50 0x7fa664100190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:34:48.360 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:48.358+0000 7fa66c24e640 1 --2- 172.21.15.46:0/3948736964 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa664102670 0x7fa6640ffc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:34:48.360 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:48.358+0000 7fa66c24e640 1 -- 172.21.15.46:0/3948736964 >> 172.21.15.46:0/3948736964 conn(0x7fa664076ba0 msgr2=0x7fa6640fdf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:34:48.360 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:48.359+0000 7fa66c24e640 1 -- 172.21.15.46:0/3948736964 shutdown_connections 2024-03-31T23:34:48.360 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:48.359+0000 7fa66c24e640 1 -- 172.21.15.46:0/3948736964 wait complete. 2024-03-31T23:34:48.362 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:48.359+0000 7fa66c24e640 1 Processor -- start 2024-03-31T23:34:48.362 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:48.359+0000 7fa66c24e640 1 -- start start 2024-03-31T23:34:48.362 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:48.359+0000 7fa66c24e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa6641007b0 0x7fa66419a030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:34:48.362 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:48.359+0000 7fa66c24e640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa664102670 0x7fa66419a570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:34:48.362 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:48.359+0000 7fa66c24e640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa664102a50 0x7fa66419e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:34:48.362 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:48.359+0000 7fa66c24e640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fa664113bd0 con 0x7fa6641007b0 2024-03-31T23:34:48.362 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:48.359+0000 7fa66c24e640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fa664113a50 con 0x7fa664102a50 2024-03-31T23:34:48.362 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:48.359+0000 7fa66c24e640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fa664113d50 con 0x7fa664102670 2024-03-31T23:34:48.362 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:48.360+0000 7fa669fea640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa6641007b0 0x7fa66419a030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:34:48.362 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:48.360+0000 7fa669fea640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa6641007b0 0x7fa66419a030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:43938/0 (socket says 172.21.15.46:43938) 2024-03-31T23:34:48.362 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:48.360+0000 7fa669fea640 1 -- 172.21.15.46:0/3987817368 learned_addr learned my addr 172.21.15.46:0/3987817368 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:34:48.363 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:48.360+0000 7fa6697e9640 1 --2- 172.21.15.46:0/3987817368 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa664102670 0x7fa66419a570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:34:48.363 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:48.360+0000 7fa66a7eb640 1 --2- 172.21.15.46:0/3987817368 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa664102a50 0x7fa66419e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:34:48.363 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:48.360+0000 7fa669fea640 1 -- 172.21.15.46:0/3987817368 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa664102670 msgr2=0x7fa66419a570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:34:48.363 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:48.360+0000 7fa669fea640 1 --2- 172.21.15.46:0/3987817368 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa664102670 0x7fa66419a570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:34:48.363 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:48.360+0000 7fa669fea640 1 -- 172.21.15.46:0/3987817368 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa664102a50 msgr2=0x7fa66419e8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:34:48.363 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:48.360+0000 7fa669fea640 1 --2- 172.21.15.46:0/3987817368 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa664102a50 0x7fa66419e8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:34:48.363 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:48.360+0000 7fa669fea640 1 -- 172.21.15.46:0/3987817368 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa66419f020 con 0x7fa6641007b0 2024-03-31T23:34:48.363 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:48.360+0000 7fa669fea640 1 --2- 172.21.15.46:0/3987817368 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa6641007b0 0x7fa66419a030 secure :-1 s=READY pgs=1019 cs=0 l=1 rev1=1 crypto rx=0x7fa66000ea60 tx=0x7fa66000ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:34:48.363 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:34:48 smithi046 ceph-mon[32498]: pgmap v732: 129 pgs: 129 active+clean; 1.2 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 4.5 MiB/s wr, 300 op/s 2024-03-31T23:34:48.364 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:48.360+0000 7fa65affd640 1 -- 172.21.15.46:0/3987817368 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa66000cde0 con 0x7fa6641007b0 2024-03-31T23:34:48.364 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:48.360+0000 7fa65affd640 1 -- 172.21.15.46:0/3987817368 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa6600035f0 con 0x7fa6641007b0 2024-03-31T23:34:48.364 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:48.360+0000 7fa66c24e640 1 -- 172.21.15.46:0/3987817368 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa66419f310 con 0x7fa6641007b0 2024-03-31T23:34:48.364 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:48.360+0000 7fa65affd640 1 -- 172.21.15.46:0/3987817368 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa660010690 con 0x7fa6641007b0 2024-03-31T23:34:48.364 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:48.361+0000 7fa66c24e640 1 -- 172.21.15.46:0/3987817368 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa6641a6ba0 con 0x7fa6641007b0 2024-03-31T23:34:48.364 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:48.363+0000 7fa65affd640 1 -- 172.21.15.46:0/3987817368 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 25) ==== 137451+0+0 (secure 0 0 0) 0x7fa6600037a0 con 0x7fa6641007b0 2024-03-31T23:34:48.367 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:48.364+0000 7fa65affd640 1 --2- 172.21.15.46:0/3987817368 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fa6500a26b0 0x7fa6500a4b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:34:48.367 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:48.364+0000 7fa66ee32740 1 Processor -- start 2024-03-31T23:34:48.367 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:48.364+0000 7fa65affd640 1 -- 172.21.15.46:0/3987817368 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(133..133 src has 1..133) ==== 9554+0+0 (secure 0 0 0) 0x7fa6600cf190 con 0x7fa6641007b0 2024-03-31T23:34:48.367 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:48.364+0000 7fa6697e9640 1 --2- 172.21.15.46:0/3987817368 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fa6500a26b0 0x7fa6500a4b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:34:48.367 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:48.364+0000 7fa66ee32740 1 -- start start 2024-03-31T23:34:48.367 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:48.364+0000 7fa66ee32740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d5c26996c0 0x55d5c2699aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:34:48.367 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:48.364+0000 7fa66ee32740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d5c2699fe0 0x55d5c26a2870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:34:48.367 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:48.364+0000 7fa66ee32740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d5c26a2db0 0x55d5c26a51a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:34:48.367 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:48.365+0000 7fa66ee32740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55d5c25fc8f0 con 0x55d5c26996c0 2024-03-31T23:34:48.367 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:48.365+0000 7fa66ee32740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55d5c25a58d0 con 0x55d5c2699fe0 2024-03-31T23:34:48.367 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:48.365+0000 7fa66ee32740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55d5c25f8990 con 0x55d5c26a2db0 2024-03-31T23:34:48.367 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:48.365+0000 7fa669fea640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d5c26996c0 0x55d5c2699aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:34:48.368 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:48.365+0000 7fa66a7eb640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d5c26a2db0 0x55d5c26a51a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:34:48.368 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:48.365+0000 7fa669fea640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d5c26996c0 0x55d5c2699aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:43952/0 (socket says 172.21.15.46:43952) 2024-03-31T23:34:48.368 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:48.365+0000 7fa66a7eb640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d5c26a2db0 0x55d5c26a51a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.46:47068/0 (socket says 172.21.15.46:47068) 2024-03-31T23:34:48.368 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:48.365+0000 7fa669fea640 1 -- 172.21.15.46:0/2145270981 learned_addr learned my addr 172.21.15.46:0/2145270981 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:34:48.368 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:48.365+0000 7fa6697e9640 1 --2- 172.21.15.46:0/2145270981 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d5c2699fe0 0x55d5c26a2870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:34:48.368 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:48.365+0000 7fa669fea640 1 -- 172.21.15.46:0/2145270981 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d5c26a2db0 msgr2=0x55d5c26a51a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:34:48.368 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:48.365+0000 7fa669fea640 1 --2- 172.21.15.46:0/2145270981 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d5c26a2db0 0x55d5c26a51a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:34:48.368 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:48.365+0000 7fa669fea640 1 -- 172.21.15.46:0/2145270981 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d5c2699fe0 msgr2=0x55d5c26a2870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:34:48.368 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:48.365+0000 7fa669fea640 1 --2- 172.21.15.46:0/2145270981 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d5c2699fe0 0x55d5c26a2870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:34:48.368 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:48.365+0000 7fa669fea640 1 -- 172.21.15.46:0/2145270981 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d5c2311210 con 0x55d5c26996c0 2024-03-31T23:34:48.368 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:48.365+0000 7fa6697e9640 1 --2- 172.21.15.46:0/3987817368 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fa6500a26b0 0x7fa6500a4b70 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fa6540095f0 tx=0x7fa654005f90 comp rx=0 tx=0).ready entity=mgr.15654 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:34:48.368 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:48.365+0000 7fa66a7eb640 1 --2- 172.21.15.46:0/2145270981 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d5c26a2db0 0x55d5c26a51a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:34:48.369 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:48.365+0000 7fa669fea640 1 --2- 172.21.15.46:0/2145270981 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d5c26996c0 0x55d5c2699aa0 secure :-1 s=READY pgs=1020 cs=0 l=1 rev1=1 crypto rx=0x7fa66009b200 tx=0x7fa6600978d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b388e3c1f220e16d server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:34:48.369 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:48.365+0000 7fa64ffff640 1 -- 172.21.15.46:0/2145270981 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa66009bc00 con 0x55d5c26996c0 2024-03-31T23:34:48.369 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:48.366+0000 7fa64ffff640 1 -- 172.21.15.46:0/2145270981 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa66009b500 con 0x55d5c26996c0 2024-03-31T23:34:48.369 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:48.366+0000 7fa64ffff640 1 -- 172.21.15.46:0/2145270981 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa66009f160 con 0x55d5c26996c0 2024-03-31T23:34:48.374 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:48.367+0000 7fa66ee32740 1 -- 172.21.15.46:0/2145270981 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d5c26996c0 msgr2=0x55d5c2699aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:34:48.374 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:48.367+0000 7fa66ee32740 1 --2- 172.21.15.46:0/2145270981 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d5c26996c0 0x55d5c2699aa0 secure :-1 s=READY pgs=1020 cs=0 l=1 rev1=1 crypto rx=0x7fa66009b200 tx=0x7fa6600978d0 comp rx=0 tx=0).stop 2024-03-31T23:34:48.374 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:48.367+0000 7fa66ee32740 1 -- 172.21.15.46:0/2145270981 shutdown_connections 2024-03-31T23:34:48.374 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:48.367+0000 7fa66ee32740 1 --2- 172.21.15.46:0/2145270981 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d5c26a2db0 0x55d5c26a51a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:34:48.374 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:48.367+0000 7fa66ee32740 1 --2- 172.21.15.46:0/2145270981 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d5c2699fe0 0x55d5c26a2870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:34:48.374 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:48.367+0000 7fa66ee32740 1 --2- 172.21.15.46:0/2145270981 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d5c26996c0 0x55d5c2699aa0 unknown :-1 s=CLOSED pgs=1020 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:34:48.374 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:48.367+0000 7fa66ee32740 1 -- 172.21.15.46:0/2145270981 >> 172.21.15.46:0/2145270981 conn(0x55d5c2693fe0 msgr2=0x55d5c26943c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:34:48.374 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:48.367+0000 7fa66ee32740 1 -- 172.21.15.46:0/2145270981 shutdown_connections 2024-03-31T23:34:48.374 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:48.367+0000 7fa66ee32740 1 -- 172.21.15.46:0/2145270981 wait complete. 2024-03-31T23:34:48.375 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:48.367+0000 7fa66ee32740 1 -- start start 2024-03-31T23:34:48.375 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:48.368+0000 7fa66ee32740 1 Processor -- start 2024-03-31T23:34:48.375 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:48.368+0000 7fa66ee32740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d5c26996c0 0x55d5c269fde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:34:48.375 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:48.368+0000 7fa66ee32740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d5c2699fe0 0x55d5c26a0320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:34:48.375 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:48.368+0000 7fa66ee32740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d5c26a2db0 0x55d5c26a0860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:34:48.375 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:48.368+0000 7fa66ee32740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55d5c26a59e0 con 0x55d5c26996c0 2024-03-31T23:34:48.375 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:48.368+0000 7fa66ee32740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55d5c259af80 con 0x55d5c26a2db0 2024-03-31T23:34:48.375 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:48.368+0000 7fa66ee32740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55d5c26a5b60 con 0x55d5c2699fe0 2024-03-31T23:34:48.375 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:48.368+0000 7fa669fea640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d5c26996c0 0x55d5c269fde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:34:48.375 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:48.368+0000 7fa66ee32740 10 client.0 _ll_register_callbacks cb 0x55d5c25f3290 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T23:34:48.375 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:48.368+0000 7fa669fea640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d5c26996c0 0x55d5c269fde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:43962/0 (socket says 172.21.15.46:43962) 2024-03-31T23:34:48.375 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:48.368+0000 7fa669fea640 1 -- 172.21.15.46:0/2610664725 learned_addr learned my addr 172.21.15.46:0/2610664725 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:34:48.375 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:48.368+0000 7fa66a7eb640 1 --2- 172.21.15.46:0/2610664725 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d5c26a2db0 0x55d5c26a0860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:34:48.375 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:48.368+0000 7fa6697e9640 1 --2- 172.21.15.46:0/2610664725 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d5c2699fe0 0x55d5c26a0320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:34:48.376 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:48.369+0000 7fa669fea640 1 -- 172.21.15.46:0/2610664725 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d5c2699fe0 msgr2=0x55d5c26a0320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:34:48.376 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:48.369+0000 7fa669fea640 1 --2- 172.21.15.46:0/2610664725 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d5c2699fe0 0x55d5c26a0320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:34:48.376 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:48.369+0000 7fa669fea640 1 -- 172.21.15.46:0/2610664725 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d5c26a2db0 msgr2=0x55d5c26a0860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:34:48.376 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:48.369+0000 7fa669fea640 1 --2- 172.21.15.46:0/2610664725 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d5c26a2db0 0x55d5c26a0860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:34:48.376 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:48.369+0000 7fa669fea640 1 -- 172.21.15.46:0/2610664725 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d5c23dd620 con 0x55d5c26996c0 2024-03-31T23:34:48.376 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:48.369+0000 7fa6697e9640 1 --2- 172.21.15.46:0/2610664725 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d5c2699fe0 0x55d5c26a0320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T23:34:48.376 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:48.369+0000 7fa66a7eb640 1 --2- 172.21.15.46:0/2610664725 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d5c26a2db0 0x55d5c26a0860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:34:48.376 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:48.369+0000 7fa669fea640 1 --2- 172.21.15.46:0/2610664725 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d5c26996c0 0x55d5c269fde0 secure :-1 s=READY pgs=1021 cs=0 l=1 rev1=1 crypto rx=0x7fa6600cc580 tx=0x7fa66009af50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7adc1dc826dffbba server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:34:48.376 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:48.369+0000 7fa64d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-31T23:34:48.376 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:48.369+0000 7fa64d7fa640 1 -- 172.21.15.46:0/2610664725 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6600a3f10 con 0x55d5c26996c0 2024-03-31T23:34:48.376 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:48.369+0000 7fa66ee32740 1 -- 172.21.15.46:0/2610664725 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa66009b500 con 0x55d5c26996c0 2024-03-31T23:34:48.376 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:48.369+0000 7fa64d7fa640 1 -- 172.21.15.46:0/2610664725 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(133..133 src has 1..133) ==== 9554+0+0 (secure 0 0 0) 0x7fa660097b70 con 0x55d5c26996c0 2024-03-31T23:34:48.376 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:48.369+0000 7fa64d7fa640 1 -- 172.21.15.46:0/2610664725 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=54) ==== 24+0+0 (secure 0 0 0) 0x7fa660083050 con 0x55d5c26996c0 2024-03-31T23:34:48.376 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:48.369+0000 7fa66ee32740 10 client.18012 fetch_fsmap learned FSMap version 54 2024-03-31T23:34:48.377 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:48.369+0000 7fa66ee32740 1 -- 172.21.15.46:0/2610664725 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=54,monmap=4+}) -- 0x55d5c26b18b0 con 0x55d5c26996c0 2024-03-31T23:34:48.377 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:48.370+0000 7fa64d7fa640 1 -- 172.21.15.46:0/2610664725 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa66009b500 con 0x55d5c26996c0 2024-03-31T23:34:48.377 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:48.370+0000 7fa64d7fa640 1 -- 172.21.15.46:0/2610664725 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 54) ==== 8346+0+0 (secure 0 0 0) 0x7fa660004000 con 0x55d5c26996c0 2024-03-31T23:34:48.377 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:48.370+0000 7fa66ee32740 10 client.18012 fetch_fsmap finished waiting for FSMap version 54 2024-03-31T23:34:48.377 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:48.370+0000 7fa66ee32740 10 client.18012 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-31T23:34:48.377 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:48.370+0000 7fa66ee32740 20 client.18012 populate_metadata read hostname 'smithi046' 2024-03-31T23:34:48.377 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:48.370+0000 7fa66ee32740 1 --2- 172.21.15.46:0/2610664725 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55d5c26b2c00 0x55d5c26b4ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:34:48.377 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:48.370+0000 7fa66ee32740 4 client.18012 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-31T23:34:48.377 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:48.370+0000 7fa66ee32740 1 -- 172.21.15.46:0/2610664725 --> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d5c2610120 con 0x55d5c26b2c00 2024-03-31T23:34:48.377 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:48.370+0000 7fa6697e9640 1 --2- 172.21.15.46:0/2610664725 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55d5c26b2c00 0x55d5c26b4ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:34:48.377 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:48.371+0000 7fa64d7fa640 10 client.18012 ms_handle_connect on v2:172.21.15.142:6838/2519143918 2024-03-31T23:34:48.377 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:48.371+0000 7fa6697e9640 1 --2- 172.21.15.46:0/2610664725 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55d5c26b2c00 0x55d5c26b4ff0 crc :-1 s=READY pgs=282 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5508570558f255e9 server_cookie=3d44c5fce7327447 in_seq=0 out_seq=0 2024-03-31T23:34:48.377 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:48.372+0000 7fa64d7fa640 1 -- 172.21.15.46:0/2610664725 <== mds.0 v2:172.21.15.142:6838/2519143918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55d5c2610120 con 0x55d5c26b2c00 2024-03-31T23:34:48.377 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:48.372+0000 7fa64d7fa640 10 client.18012 handle_command_reply: tid=0 2024-03-31T23:34:48.377 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:48.372+0000 7fa66ee32740 1 client.18012 shutdown 2024-03-31T23:34:48.378 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:48.372+0000 7fa66ee32740 1 -- 172.21.15.46:0/2610664725 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d5c26996c0 msgr2=0x55d5c269fde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:34:48.378 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:48.372+0000 7fa66ee32740 1 --2- 172.21.15.46:0/2610664725 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d5c26996c0 0x55d5c269fde0 secure :-1 s=READY pgs=1021 cs=0 l=1 rev1=1 crypto rx=0x7fa6600cc580 tx=0x7fa66009af50 comp rx=0 tx=0).stop 2024-03-31T23:34:48.378 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:48.372+0000 7fa66ee32740 1 -- 172.21.15.46:0/2610664725 shutdown_connections 2024-03-31T23:34:48.378 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:48.372+0000 7fa66ee32740 1 --2- 172.21.15.46:0/2610664725 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55d5c26b2c00 0x55d5c26b4ff0 crc :-1 s=READY pgs=282 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:34:48.378 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:48.372+0000 7fa66ee32740 1 --2- 172.21.15.46:0/2610664725 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d5c26a2db0 0x55d5c26a0860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:34:48.378 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:48.372+0000 7fa66ee32740 1 --2- 172.21.15.46:0/2610664725 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d5c2699fe0 0x55d5c26a0320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:34:48.378 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:48.373+0000 7fa66ee32740 1 --2- 172.21.15.46:0/2610664725 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d5c26996c0 0x55d5c269fde0 unknown :-1 s=CLOSED pgs=1021 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:34:48.378 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:48.373+0000 7fa66ee32740 1 -- 172.21.15.46:0/2610664725 >> 172.21.15.46:0/2610664725 conn(0x55d5c2693fe0 msgr2=0x55d5c2695d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:34:48.378 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:48.373+0000 7fa64d7fa640 0 client.18012 ms_handle_reset on v2:172.21.15.142:6838/2519143918 2024-03-31T23:34:48.378 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:48.373+0000 7fa66ee32740 1 -- 172.21.15.46:0/2610664725 shutdown_connections 2024-03-31T23:34:48.378 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:48.373+0000 7fa66ee32740 1 -- 172.21.15.46:0/2610664725 wait complete. 2024-03-31T23:34:48.378 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:48.373+0000 7fa66ee32740 20 client.18012 trim_cache size 0 max 16384 2024-03-31T23:34:48.397 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:48.396+0000 7fa66ee32740 1 Processor -- start 2024-03-31T23:34:48.397 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:48.396+0000 7fa66ee32740 1 -- start start 2024-03-31T23:34:48.398 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:48.397+0000 7fa66ee32740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d5c26996c0 0x55d5c26a2510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:34:48.398 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:48.397+0000 7fa66ee32740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d5c2699fe0 0x55d5c269d040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:34:48.398 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:48.397+0000 7fa66ee32740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d5c26a2db0 0x55d5c269d580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:34:48.398 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:48.397+0000 7fa66ee32740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fa6641b37d0 con 0x55d5c26a2db0 2024-03-31T23:34:48.398 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:48.397+0000 7fa66ee32740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fa6641b10a0 con 0x55d5c2699fe0 2024-03-31T23:34:48.398 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:48.397+0000 7fa66ee32740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fa6641aef10 con 0x55d5c26996c0 2024-03-31T23:34:48.398 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:48.397+0000 7fa66a7eb640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d5c26a2db0 0x55d5c269d580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:34:48.398 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:48.397+0000 7fa66a7eb640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d5c26a2db0 0x55d5c269d580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:43976/0 (socket says 172.21.15.46:43976) 2024-03-31T23:34:48.399 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:48.397+0000 7fa66a7eb640 1 -- 172.21.15.46:0/2979838787 learned_addr learned my addr 172.21.15.46:0/2979838787 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:34:48.399 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:48.397+0000 7fa66a7eb640 1 -- 172.21.15.46:0/2979838787 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d5c26996c0 msgr2=0x55d5c26a2510 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-31T23:34:48.399 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:48.397+0000 7fa66a7eb640 1 --2- 172.21.15.46:0/2979838787 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d5c26996c0 0x55d5c26a2510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:34:48.399 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:48.397+0000 7fa66a7eb640 1 -- 172.21.15.46:0/2979838787 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d5c2699fe0 msgr2=0x55d5c269d040 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-31T23:34:48.399 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:48.397+0000 7fa66a7eb640 1 --2- 172.21.15.46:0/2979838787 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d5c2699fe0 0x55d5c269d040 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:34:48.400 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:48.397+0000 7fa66a7eb640 1 -- 172.21.15.46:0/2979838787 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d5c269ed20 con 0x55d5c26a2db0 2024-03-31T23:34:48.400 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:48.398+0000 7fa66a7eb640 1 --2- 172.21.15.46:0/2979838787 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d5c26a2db0 0x55d5c269d580 secure :-1 s=READY pgs=1022 cs=0 l=1 rev1=1 crypto rx=0x7fa65c009ca0 tx=0x7fa65c005c50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ea518f3d77e739e server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:34:48.400 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:48.398+0000 7fa64f7fe640 1 -- 172.21.15.46:0/2979838787 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa65c045070 con 0x55d5c26a2db0 2024-03-31T23:34:48.400 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:48.398+0000 7fa64f7fe640 1 -- 172.21.15.46:0/2979838787 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa65c003b70 con 0x55d5c26a2db0 2024-03-31T23:34:48.400 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:48.398+0000 7fa64f7fe640 1 -- 172.21.15.46:0/2979838787 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa65c039040 con 0x55d5c26a2db0 2024-03-31T23:34:48.401 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:48.399+0000 7fa66ee32740 1 -- 172.21.15.46:0/2979838787 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d5c26a2db0 msgr2=0x55d5c269d580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:34:48.401 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:48.399+0000 7fa66ee32740 1 --2- 172.21.15.46:0/2979838787 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d5c26a2db0 0x55d5c269d580 secure :-1 s=READY pgs=1022 cs=0 l=1 rev1=1 crypto rx=0x7fa65c009ca0 tx=0x7fa65c005c50 comp rx=0 tx=0).stop 2024-03-31T23:34:48.401 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:48.399+0000 7fa66ee32740 1 -- 172.21.15.46:0/2979838787 shutdown_connections 2024-03-31T23:34:48.401 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:48.399+0000 7fa66ee32740 1 --2- 172.21.15.46:0/2979838787 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d5c26a2db0 0x55d5c269d580 unknown :-1 s=CLOSED pgs=1022 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:34:48.401 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:48.399+0000 7fa66ee32740 1 --2- 172.21.15.46:0/2979838787 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d5c2699fe0 0x55d5c269d040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:34:48.401 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:48.399+0000 7fa66ee32740 1 --2- 172.21.15.46:0/2979838787 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d5c26996c0 0x55d5c26a2510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:34:48.401 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:48.399+0000 7fa66ee32740 1 -- 172.21.15.46:0/2979838787 >> 172.21.15.46:0/2979838787 conn(0x55d5c2693fe0 msgr2=0x55d5c269b130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:34:48.401 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:48.399+0000 7fa66ee32740 1 -- 172.21.15.46:0/2979838787 shutdown_connections 2024-03-31T23:34:48.401 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:48.399+0000 7fa66ee32740 1 -- 172.21.15.46:0/2979838787 wait complete. 2024-03-31T23:34:48.401 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:48.400+0000 7fa66ee32740 1 -- start start 2024-03-31T23:34:48.401 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:48.400+0000 7fa66ee32740 1 Processor -- start 2024-03-31T23:34:48.403 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:48.401+0000 7fa66ee32740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d5c26996c0 0x55d5c26b55f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:34:48.403 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:48.401+0000 7fa66ee32740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d5c2699fe0 0x55d5c26b5b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:34:48.403 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:48.401+0000 7fa66ee32740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d5c26a2db0 0x55d5c26b6070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:34:48.403 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:48.401+0000 7fa66ee32740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fa6641af560 con 0x55d5c2699fe0 2024-03-31T23:34:48.403 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:48.401+0000 7fa66ee32740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fa6641b09d0 con 0x55d5c26996c0 2024-03-31T23:34:48.403 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:48.401+0000 7fa66ee32740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fa6641afc30 con 0x55d5c26a2db0 2024-03-31T23:34:48.403 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:48.401+0000 7fa66ee32740 10 client.0 _ll_register_callbacks cb 0x7fa660005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T23:34:48.403 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:48.401+0000 7fa6697e9640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d5c2699fe0 0x55d5c26b5b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:34:48.403 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:48.401+0000 7fa6697e9640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d5c2699fe0 0x55d5c26b5b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:43986/0 (socket says 172.21.15.46:43986) 2024-03-31T23:34:48.403 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:48.401+0000 7fa6697e9640 1 -- 172.21.15.46:0/920698072 learned_addr learned my addr 172.21.15.46:0/920698072 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:34:48.403 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:48.401+0000 7fa66a7eb640 1 --2- 172.21.15.46:0/920698072 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d5c26a2db0 0x55d5c26b6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:34:48.404 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:48.401+0000 7fa6697e9640 1 -- 172.21.15.46:0/920698072 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d5c26a2db0 msgr2=0x55d5c26b6070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:34:48.404 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:48.401+0000 7fa6697e9640 1 --2- 172.21.15.46:0/920698072 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d5c26a2db0 0x55d5c26b6070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:34:48.404 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:48.401+0000 7fa6697e9640 1 -- 172.21.15.46:0/920698072 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d5c26996c0 msgr2=0x55d5c26b55f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:34:48.404 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:48.401+0000 7fa6697e9640 1 --2- 172.21.15.46:0/920698072 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d5c26996c0 0x55d5c26b55f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:34:48.404 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:48.401+0000 7fa6697e9640 1 -- 172.21.15.46:0/920698072 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d5c2720460 con 0x55d5c2699fe0 2024-03-31T23:34:48.404 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:48.402+0000 7fa6697e9640 1 --2- 172.21.15.46:0/920698072 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d5c2699fe0 0x55d5c26b5b30 secure :-1 s=READY pgs=1023 cs=0 l=1 rev1=1 crypto rx=0x55d5c269f640 tx=0x7fa654004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4bead2f09bd35f7d server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:34:48.404 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:48.402+0000 7fa64d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-31T23:34:48.404 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:48.402+0000 7fa64d7fa640 1 -- 172.21.15.46:0/920698072 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa654022040 con 0x55d5c2699fe0 2024-03-31T23:34:48.404 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:48.402+0000 7fa66ee32740 1 -- 172.21.15.46:0/920698072 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa65c003b70 con 0x55d5c2699fe0 2024-03-31T23:34:48.404 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:48.402+0000 7fa64d7fa640 1 -- 172.21.15.46:0/920698072 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(133..133 src has 1..133) ==== 9554+0+0 (secure 0 0 0) 0x7fa65401cce0 con 0x55d5c2699fe0 2024-03-31T23:34:48.405 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:48.402+0000 7fa64d7fa640 1 -- 172.21.15.46:0/920698072 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=54) ==== 24+0+0 (secure 0 0 0) 0x7fa654039050 con 0x55d5c2699fe0 2024-03-31T23:34:48.405 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:48.402+0000 7fa66ee32740 10 client.18018 fetch_fsmap learned FSMap version 54 2024-03-31T23:34:48.406 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:48.402+0000 7fa66ee32740 1 -- 172.21.15.46:0/920698072 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=54,monmap=4+}) -- 0x55d5c27217c0 con 0x55d5c2699fe0 2024-03-31T23:34:48.406 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:48.403+0000 7fa64d7fa640 1 -- 172.21.15.46:0/920698072 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa65c003b70 con 0x55d5c2699fe0 2024-03-31T23:34:48.406 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:48.403+0000 7fa64d7fa640 1 -- 172.21.15.46:0/920698072 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 54) ==== 8346+0+0 (secure 0 0 0) 0x7fa65401a040 con 0x55d5c2699fe0 2024-03-31T23:34:48.406 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:48.403+0000 7fa66ee32740 10 client.18018 fetch_fsmap finished waiting for FSMap version 54 2024-03-31T23:34:48.406 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:48.403+0000 7fa66ee32740 10 client.18018 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-31T23:34:48.406 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:48.403+0000 7fa66ee32740 20 client.18018 populate_metadata read hostname 'smithi046' 2024-03-31T23:34:48.406 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:48.403+0000 7fa66ee32740 1 --2- 172.21.15.46:0/920698072 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55d5c2721f70 0x55d5c2724390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:34:48.406 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:48.403+0000 7fa66ee32740 4 client.18018 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-31T23:34:48.406 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:48.403+0000 7fa66ee32740 1 -- 172.21.15.46:0/920698072 --> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55d5c2609e30 con 0x55d5c2721f70 2024-03-31T23:34:48.406 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:48.403+0000 7fa669fea640 1 --2- 172.21.15.46:0/920698072 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55d5c2721f70 0x55d5c2724390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:34:48.406 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:48.404+0000 7fa64d7fa640 10 client.18018 ms_handle_connect on v2:172.21.15.142:6838/2519143918 2024-03-31T23:34:48.406 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:48.404+0000 7fa669fea640 1 --2- 172.21.15.46:0/920698072 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55d5c2721f70 0x55d5c2724390 crc :-1 s=READY pgs=283 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d18c07ffba95b76f server_cookie=db2ffbd569c07d89 in_seq=0 out_seq=0 2024-03-31T23:34:48.443 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:34:48 smithi142 ceph-mon[35271]: pgmap v732: 129 pgs: 129 active+clean; 1.2 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 4.5 MiB/s wr, 300 op/s 2024-03-31T23:34:48.545 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:34:48 smithi162 ceph-mon[36237]: pgmap v732: 129 pgs: 129 active+clean; 1.2 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 4.5 MiB/s wr, 300 op/s 2024-03-31T23:34:48.675 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:48.674+0000 7fa64d7fa640 1 -- 172.21.15.46:0/920698072 <== mds.0 v2:172.21.15.142:6838/2519143918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55d5c2609e30 con 0x55d5c2721f70 2024-03-31T23:34:48.675 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:48.674+0000 7fa64d7fa640 10 client.18018 handle_command_reply: tid=0 2024-03-31T23:34:48.675 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:48.674+0000 7fa66ee32740 1 client.18018 shutdown 2024-03-31T23:34:48.676 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:48.674+0000 7fa66ee32740 1 -- 172.21.15.46:0/920698072 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d5c2699fe0 msgr2=0x55d5c26b5b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:34:48.677 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:48.674+0000 7fa66ee32740 1 --2- 172.21.15.46:0/920698072 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d5c2699fe0 0x55d5c26b5b30 secure :-1 s=READY pgs=1023 cs=0 l=1 rev1=1 crypto rx=0x55d5c269f640 tx=0x7fa654004730 comp rx=0 tx=0).stop 2024-03-31T23:34:48.677 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:48.675+0000 7fa66ee32740 1 -- 172.21.15.46:0/920698072 shutdown_connections 2024-03-31T23:34:48.677 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:48.675+0000 7fa66ee32740 1 --2- 172.21.15.46:0/920698072 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55d5c2721f70 0x55d5c2724390 crc :-1 s=READY pgs=283 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:34:48.677 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:48.675+0000 7fa66ee32740 1 --2- 172.21.15.46:0/920698072 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d5c26a2db0 0x55d5c26b6070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:34:48.677 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:48.675+0000 7fa66ee32740 1 --2- 172.21.15.46:0/920698072 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d5c2699fe0 0x55d5c26b5b30 unknown :-1 s=CLOSED pgs=1023 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:34:48.677 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:48.675+0000 7fa66ee32740 1 --2- 172.21.15.46:0/920698072 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55d5c26996c0 0x55d5c26b55f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:34:48.677 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:48.675+0000 7fa66ee32740 1 -- 172.21.15.46:0/920698072 >> 172.21.15.46:0/920698072 conn(0x55d5c2693fe0 msgr2=0x55d5c26926f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:34:48.677 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:48.675+0000 7fa64d7fa640 0 client.18018 ms_handle_reset on v2:172.21.15.142:6838/2519143918 2024-03-31T23:34:48.677 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:48.675+0000 7fa66ee32740 1 -- 172.21.15.46:0/920698072 shutdown_connections 2024-03-31T23:34:48.677 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:48.675+0000 7fa66ee32740 1 -- 172.21.15.46:0/920698072 wait complete. 2024-03-31T23:34:48.677 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:48.675+0000 7fa66ee32740 20 client.18018 trim_cache size 0 max 16384 2024-03-31T23:34:48.678 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout:{ 2024-03-31T23:34:48.678 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout: "message": "", 2024-03-31T23:34:48.678 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout: "return_code": 0 2024-03-31T23:34:48.678 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout:} 2024-03-31T23:34:48.680 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:48.677+0000 7fa66c24e640 1 -- 172.21.15.46:0/3987817368 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fa6500a26b0 msgr2=0x7fa6500a4b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:34:48.680 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:48.678+0000 7fa66c24e640 1 --2- 172.21.15.46:0/3987817368 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fa6500a26b0 0x7fa6500a4b70 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fa6540095f0 tx=0x7fa654005f90 comp rx=0 tx=0).stop 2024-03-31T23:34:48.680 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:48.678+0000 7fa66c24e640 1 -- 172.21.15.46:0/3987817368 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa6641007b0 msgr2=0x7fa66419a030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:34:48.680 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:48.678+0000 7fa66c24e640 1 --2- 172.21.15.46:0/3987817368 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa6641007b0 0x7fa66419a030 secure :-1 s=READY pgs=1019 cs=0 l=1 rev1=1 crypto rx=0x7fa66000ea60 tx=0x7fa66000ef20 comp rx=0 tx=0).stop 2024-03-31T23:34:48.680 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:48.678+0000 7fa66c24e640 1 -- 172.21.15.46:0/3987817368 shutdown_connections 2024-03-31T23:34:48.680 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:48.678+0000 7fa66c24e640 1 --2- 172.21.15.46:0/3987817368 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fa6500a26b0 0x7fa6500a4b70 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:34:48.680 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:48.678+0000 7fa66c24e640 1 --2- 172.21.15.46:0/3987817368 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa664102a50 0x7fa66419e8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:34:48.680 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:48.678+0000 7fa66c24e640 1 --2- 172.21.15.46:0/3987817368 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa664102670 0x7fa66419a570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:34:48.680 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:48.678+0000 7fa66c24e640 1 --2- 172.21.15.46:0/3987817368 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa6641007b0 0x7fa66419a030 unknown :-1 s=CLOSED pgs=1019 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:34:48.680 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:48.678+0000 7fa66c24e640 1 -- 172.21.15.46:0/3987817368 >> 172.21.15.46:0/3987817368 conn(0x7fa664076ba0 msgr2=0x7fa6640fd140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:34:48.680 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:48.679+0000 7fa66c24e640 1 -- 172.21.15.46:0/3987817368 shutdown_connections 2024-03-31T23:34:48.680 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:34:48.679+0000 7fa66c24e640 1 -- 172.21.15.46:0/3987817368 wait complete. 2024-03-31T23:34:49.312 INFO:tasks.workunit.client.0.smithi046.stdout: 10 92024 8.22 MB/sec execute 474 sec latency 215.610 ms 2024-03-31T23:34:50.312 INFO:tasks.workunit.client.0.smithi046.stdout: 10 92167 8.21 MB/sec execute 475 sec latency 102.582 ms 2024-03-31T23:34:50.361 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:34:50 smithi046 ceph-mon[32498]: pgmap v733: 129 pgs: 129 active+clean; 1.2 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 6.7 MiB/s wr, 455 op/s 2024-03-31T23:34:50.443 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:34:50 smithi142 ceph-mon[35271]: pgmap v733: 129 pgs: 129 active+clean; 1.2 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 6.7 MiB/s wr, 455 op/s 2024-03-31T23:34:50.545 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:34:50 smithi162 ceph-mon[36237]: pgmap v733: 129 pgs: 129 active+clean; 1.2 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 6.7 MiB/s wr, 455 op/s 2024-03-31T23:34:51.312 INFO:tasks.workunit.client.0.smithi046.stdout: 10 92321 8.22 MB/sec execute 476 sec latency 107.018 ms 2024-03-31T23:34:52.312 INFO:tasks.workunit.client.0.smithi046.stdout: 10 92514 8.23 MB/sec execute 477 sec latency 95.903 ms 2024-03-31T23:34:52.361 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:34:52 smithi046 ceph-mon[32498]: pgmap v734: 129 pgs: 129 active+clean; 1.2 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 4.5 MiB/s wr, 267 op/s 2024-03-31T23:34:52.443 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:34:52 smithi142 ceph-mon[35271]: pgmap v734: 129 pgs: 129 active+clean; 1.2 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 4.5 MiB/s wr, 267 op/s 2024-03-31T23:34:52.545 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:34:52 smithi162 ceph-mon[36237]: pgmap v734: 129 pgs: 129 active+clean; 1.2 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 4.5 MiB/s wr, 267 op/s 2024-03-31T23:34:53.312 INFO:tasks.workunit.client.0.smithi046.stdout: 10 92682 8.23 MB/sec execute 478 sec latency 97.188 ms 2024-03-31T23:34:54.312 INFO:tasks.workunit.client.0.smithi046.stdout: 10 92823 8.23 MB/sec execute 479 sec latency 99.497 ms 2024-03-31T23:34:54.443 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:34:54 smithi142 ceph-mon[35271]: pgmap v735: 129 pgs: 129 active+clean; 1.2 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 4.5 MiB/s wr, 267 op/s 2024-03-31T23:34:54.545 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:34:54 smithi162 ceph-mon[36237]: pgmap v735: 129 pgs: 129 active+clean; 1.2 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 4.5 MiB/s wr, 267 op/s 2024-03-31T23:34:54.611 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:34:54 smithi046 ceph-mon[32498]: pgmap v735: 129 pgs: 129 active+clean; 1.2 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 4.5 MiB/s wr, 267 op/s 2024-03-31T23:34:55.312 INFO:tasks.workunit.client.0.smithi046.stdout: 10 92972 8.22 MB/sec execute 480 sec latency 82.001 ms 2024-03-31T23:34:56.312 INFO:tasks.workunit.client.0.smithi046.stdout: 10 93126 8.23 MB/sec execute 481 sec latency 88.565 ms 2024-03-31T23:34:56.443 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:34:56 smithi142 ceph-mon[35271]: pgmap v736: 129 pgs: 129 active+clean; 1.2 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 6.9 MiB/s wr, 378 op/s 2024-03-31T23:34:56.545 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:34:56 smithi162 ceph-mon[36237]: pgmap v736: 129 pgs: 129 active+clean; 1.2 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 6.9 MiB/s wr, 378 op/s 2024-03-31T23:34:56.611 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:34:56 smithi046 ceph-mon[32498]: pgmap v736: 129 pgs: 129 active+clean; 1.2 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 6.9 MiB/s wr, 378 op/s 2024-03-31T23:34:57.312 INFO:tasks.workunit.client.0.smithi046.stdout: 10 93261 8.22 MB/sec execute 482 sec latency 121.463 ms 2024-03-31T23:34:58.312 INFO:tasks.workunit.client.0.smithi046.stdout: 10 93394 8.23 MB/sec execute 483 sec latency 149.945 ms 2024-03-31T23:34:58.443 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:34:58 smithi142 ceph-mon[35271]: pgmap v737: 129 pgs: 129 active+clean; 1.2 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 4.5 MiB/s wr, 265 op/s 2024-03-31T23:34:58.545 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:34:58 smithi162 ceph-mon[36237]: pgmap v737: 129 pgs: 129 active+clean; 1.2 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 4.5 MiB/s wr, 265 op/s 2024-03-31T23:34:58.611 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:34:58 smithi046 ceph-mon[32498]: pgmap v737: 129 pgs: 129 active+clean; 1.2 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 4.5 MiB/s wr, 265 op/s 2024-03-31T23:34:59.312 INFO:tasks.workunit.client.0.smithi046.stdout: 10 93534 8.22 MB/sec execute 484 sec latency 100.584 ms 2024-03-31T23:35:00.314 INFO:tasks.workunit.client.0.smithi046.stdout: 10 93710 8.23 MB/sec execute 485 sec latency 101.025 ms 2024-03-31T23:35:00.443 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:35:00 smithi142 ceph-mon[35271]: pgmap v738: 129 pgs: 129 active+clean; 1.2 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 6.9 MiB/s wr, 364 op/s 2024-03-31T23:35:00.545 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:35:00 smithi162 ceph-mon[36237]: pgmap v738: 129 pgs: 129 active+clean; 1.2 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 6.9 MiB/s wr, 364 op/s 2024-03-31T23:35:00.611 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:35:00 smithi046 ceph-mon[32498]: pgmap v738: 129 pgs: 129 active+clean; 1.2 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 6.9 MiB/s wr, 364 op/s 2024-03-31T23:35:01.312 INFO:tasks.workunit.client.0.smithi046.stdout: 10 93765 8.22 MB/sec execute 486 sec latency 433.155 ms 2024-03-31T23:35:01.781 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:01.778+0000 7f220e847640 1 -- 172.21.15.46:0/2607742727 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2208107730 msgr2=0x7f2208107b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:35:01.781 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:01.778+0000 7f220e847640 1 --2- 172.21.15.46:0/2607742727 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2208107730 0x7f2208107b10 secure :-1 s=READY pgs=1024 cs=0 l=1 rev1=1 crypto rx=0x7f21f8009930 tx=0x7f21f802bc30 comp rx=0 tx=0).stop 2024-03-31T23:35:01.781 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:01.778+0000 7f220e847640 1 -- 172.21.15.46:0/2607742727 shutdown_connections 2024-03-31T23:35:01.781 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:01.778+0000 7f220e847640 1 --2- 172.21.15.46:0/2607742727 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2208069750 0x7f220810e5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:35:01.782 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:01.778+0000 7f220e847640 1 --2- 172.21.15.46:0/2607742727 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2208068dd0 0x7f2208069210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:35:01.782 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:01.778+0000 7f220e847640 1 --2- 172.21.15.46:0/2607742727 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2208107730 0x7f2208107b10 unknown :-1 s=CLOSED pgs=1024 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:35:01.782 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:01.778+0000 7f220e847640 1 -- 172.21.15.46:0/2607742727 >> 172.21.15.46:0/2607742727 conn(0x7f22080774a0 msgr2=0x7f22080778b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:35:01.782 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:01.778+0000 7f220e847640 1 -- 172.21.15.46:0/2607742727 shutdown_connections 2024-03-31T23:35:01.782 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:01.778+0000 7f220e847640 1 -- 172.21.15.46:0/2607742727 wait complete. 2024-03-31T23:35:01.782 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:01.779+0000 7f220e847640 1 Processor -- start 2024-03-31T23:35:01.782 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:01.779+0000 7f220e847640 1 -- start start 2024-03-31T23:35:01.782 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:01.779+0000 7f220e847640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2208068dd0 0x7f2208197ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:35:01.782 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:01.779+0000 7f220e847640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2208069750 0x7f2208198420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:35:01.782 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:01.779+0000 7f220e847640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2208107730 0x7f220819c7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:35:01.782 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:01.779+0000 7f220e847640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f22081119e0 con 0x7f2208107730 2024-03-31T23:35:01.783 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:01.779+0000 7f220e847640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f2208111860 con 0x7f2208069750 2024-03-31T23:35:01.783 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:01.779+0000 7f220e847640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f2208111b60 con 0x7f2208068dd0 2024-03-31T23:35:01.783 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:01.779+0000 7f220cde4640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2208107730 0x7f220819c7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:35:01.783 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:01.779+0000 7f22077fe640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2208069750 0x7f2208198420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:35:01.783 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:01.779+0000 7f220cde4640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2208107730 0x7f220819c7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:49958/0 (socket says 172.21.15.46:49958) 2024-03-31T23:35:01.783 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:01.779+0000 7f220cde4640 1 -- 172.21.15.46:0/3618567887 learned_addr learned my addr 172.21.15.46:0/3618567887 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:35:01.783 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:01.779+0000 7f2207fff640 1 --2- 172.21.15.46:0/3618567887 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2208068dd0 0x7f2208197ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:35:01.783 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:01.780+0000 7f220cde4640 1 -- 172.21.15.46:0/3618567887 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2208068dd0 msgr2=0x7f2208197ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:35:01.783 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:01.780+0000 7f220cde4640 1 --2- 172.21.15.46:0/3618567887 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2208068dd0 0x7f2208197ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:35:01.783 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:01.780+0000 7f220cde4640 1 -- 172.21.15.46:0/3618567887 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2208069750 msgr2=0x7f2208198420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:35:01.783 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:01.780+0000 7f220cde4640 1 --2- 172.21.15.46:0/3618567887 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2208069750 0x7f2208198420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:35:01.783 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:01.780+0000 7f220cde4640 1 -- 172.21.15.46:0/3618567887 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f220819ced0 con 0x7f2208107730 2024-03-31T23:35:01.783 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:01.780+0000 7f22077fe640 1 --2- 172.21.15.46:0/3618567887 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2208069750 0x7f2208198420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:35:01.783 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:01.780+0000 7f220cde4640 1 --2- 172.21.15.46:0/3618567887 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2208107730 0x7f220819c7a0 secure :-1 s=READY pgs=1025 cs=0 l=1 rev1=1 crypto rx=0x7f220000ea60 tx=0x7f220000ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:35:01.783 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:01.780+0000 7f22057fa640 1 -- 172.21.15.46:0/3618567887 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f220000ce40 con 0x7f2208107730 2024-03-31T23:35:01.784 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:01.780+0000 7f22057fa640 1 -- 172.21.15.46:0/3618567887 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f2200005020 con 0x7f2208107730 2024-03-31T23:35:01.784 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:01.780+0000 7f220e847640 1 -- 172.21.15.46:0/3618567887 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f220819d1c0 con 0x7f2208107730 2024-03-31T23:35:01.784 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:01.780+0000 7f22057fa640 1 -- 172.21.15.46:0/3618567887 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2200010670 con 0x7f2208107730 2024-03-31T23:35:01.784 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:01.780+0000 7f220e847640 1 -- 172.21.15.46:0/3618567887 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f22081a4a50 con 0x7f2208107730 2024-03-31T23:35:01.790 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:01.782+0000 7f22057fa640 1 -- 172.21.15.46:0/3618567887 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 25) ==== 137451+0+0 (secure 0 0 0) 0x7f22000051d0 con 0x7f2208107730 2024-03-31T23:35:01.790 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:01.783+0000 7f221142e740 1 Processor -- start 2024-03-31T23:35:01.791 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:01.783+0000 7f221142e740 1 -- start start 2024-03-31T23:35:01.791 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:01.783+0000 7f22057fa640 1 --2- 172.21.15.46:0/3618567887 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f21f00a26b0 0x7f21f00a4b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:35:01.791 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:01.783+0000 7f22057fa640 1 -- 172.21.15.46:0/3618567887 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(133..133 src has 1..133) ==== 9554+0+0 (secure 0 0 0) 0x7f22000cf380 con 0x7f2208107730 2024-03-31T23:35:01.791 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:01.783+0000 7f22077fe640 1 --2- 172.21.15.46:0/3618567887 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f21f00a26b0 0x7f21f00a4b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:35:01.791 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:01.783+0000 7f221142e740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e1189ad160 0x55e1189ad540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:35:01.791 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:01.783+0000 7f221142e740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e1189ada80 0x55e1189bac80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:35:01.791 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:01.783+0000 7f221142e740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55e1189bb1c0 0x55e1189bd5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:35:01.791 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:01.783+0000 7f221142e740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55e1189108f0 con 0x55e1189ad160 2024-03-31T23:35:01.791 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:01.783+0000 7f221142e740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55e1188b98d0 con 0x55e1189bb1c0 2024-03-31T23:35:01.791 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:01.783+0000 7f221142e740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55e11890c990 con 0x55e1189ada80 2024-03-31T23:35:01.791 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:01.783+0000 7f2207fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e1189ad160 0x55e1189ad540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:35:01.791 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:01.783+0000 7f2207fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e1189ad160 0x55e1189ad540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:49966/0 (socket says 172.21.15.46:49966) 2024-03-31T23:35:01.791 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:01.783+0000 7f2207fff640 1 -- 172.21.15.46:0/2122975233 learned_addr learned my addr 172.21.15.46:0/2122975233 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:35:01.792 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:01.783+0000 7f220cde4640 1 --2- 172.21.15.46:0/2122975233 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55e1189bb1c0 0x55e1189bd5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:35:01.792 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:01.783+0000 7f22077fe640 1 --2- 172.21.15.46:0/2122975233 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e1189ada80 0x55e1189bac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:35:01.792 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:01.783+0000 7f2207fff640 1 -- 172.21.15.46:0/2122975233 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e1189ada80 msgr2=0x55e1189bac80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:35:01.792 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:01.783+0000 7f2207fff640 1 --2- 172.21.15.46:0/2122975233 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e1189ada80 0x55e1189bac80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:35:01.792 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:01.783+0000 7f2207fff640 1 -- 172.21.15.46:0/2122975233 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55e1189bb1c0 msgr2=0x55e1189bd5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:35:01.792 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:01.783+0000 7f2207fff640 1 --2- 172.21.15.46:0/2122975233 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55e1189bb1c0 0x55e1189bd5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:35:01.792 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:01.783+0000 7f2207fff640 1 -- 172.21.15.46:0/2122975233 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e118625210 con 0x55e1189ad160 2024-03-31T23:35:01.792 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:01.784+0000 7f22077fe640 1 --2- 172.21.15.46:0/2122975233 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e1189ada80 0x55e1189bac80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T23:35:01.792 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:01.784+0000 7f22077fe640 1 --2- 172.21.15.46:0/3618567887 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f21f00a26b0 0x7f21f00a4b70 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f2208077ba0 tx=0x7f21fc011810 comp rx=0 tx=0).ready entity=mgr.15654 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:35:01.792 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:01.784+0000 7f2207fff640 1 --2- 172.21.15.46:0/2122975233 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e1189ad160 0x55e1189ad540 secure :-1 s=READY pgs=1026 cs=0 l=1 rev1=1 crypto rx=0x7f21f8009930 tx=0x7f21f8009560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da5c543a5dcdb8e2 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:35:01.792 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:01.784+0000 7f21ee7fc640 1 -- 172.21.15.46:0/2122975233 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21f8043070 con 0x55e1189ad160 2024-03-31T23:35:01.792 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:01.784+0000 7f21ee7fc640 1 -- 172.21.15.46:0/2122975233 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f21f802c7e0 con 0x55e1189ad160 2024-03-31T23:35:01.792 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:01.784+0000 7f21ee7fc640 1 -- 172.21.15.46:0/2122975233 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21f80054f0 con 0x55e1189ad160 2024-03-31T23:35:01.792 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:01.785+0000 7f221142e740 1 -- 172.21.15.46:0/2122975233 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e1189ad160 msgr2=0x55e1189ad540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:35:01.793 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:01.785+0000 7f221142e740 1 --2- 172.21.15.46:0/2122975233 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e1189ad160 0x55e1189ad540 secure :-1 s=READY pgs=1026 cs=0 l=1 rev1=1 crypto rx=0x7f21f8009930 tx=0x7f21f8009560 comp rx=0 tx=0).stop 2024-03-31T23:35:01.793 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:01.785+0000 7f221142e740 1 -- 172.21.15.46:0/2122975233 shutdown_connections 2024-03-31T23:35:01.793 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:01.785+0000 7f221142e740 1 --2- 172.21.15.46:0/2122975233 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55e1189bb1c0 0x55e1189bd5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:35:01.793 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:01.785+0000 7f221142e740 1 --2- 172.21.15.46:0/2122975233 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e1189ada80 0x55e1189bac80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:35:01.793 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:01.785+0000 7f221142e740 1 --2- 172.21.15.46:0/2122975233 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e1189ad160 0x55e1189ad540 unknown :-1 s=CLOSED pgs=1026 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:35:01.793 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:01.785+0000 7f221142e740 1 -- 172.21.15.46:0/2122975233 >> 172.21.15.46:0/2122975233 conn(0x55e1189a7fe0 msgr2=0x55e1189a83c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:35:01.793 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:01.785+0000 7f221142e740 1 -- 172.21.15.46:0/2122975233 shutdown_connections 2024-03-31T23:35:01.793 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:01.785+0000 7f221142e740 1 -- 172.21.15.46:0/2122975233 wait complete. 2024-03-31T23:35:01.793 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:01.785+0000 7f221142e740 1 -- start start 2024-03-31T23:35:01.793 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:01.785+0000 7f221142e740 1 Processor -- start 2024-03-31T23:35:01.793 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:01.786+0000 7f221142e740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e1189ad160 0x55e1189b8220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:35:01.794 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:01.786+0000 7f221142e740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55e1189ada80 0x55e1189b8760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:35:01.794 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:01.786+0000 7f221142e740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e1189bb1c0 0x55e1189b8ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:35:01.794 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:01.786+0000 7f221142e740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55e1189bdd90 con 0x55e1189bb1c0 2024-03-31T23:35:01.794 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:01.786+0000 7f221142e740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55e1188aef80 con 0x55e1189ada80 2024-03-31T23:35:01.794 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:01.786+0000 7f221142e740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55e1189bdf10 con 0x55e1189ad160 2024-03-31T23:35:01.794 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:01.786+0000 7f221142e740 10 client.0 _ll_register_callbacks cb 0x55e118907290 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T23:35:01.794 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:01.786+0000 7f220cde4640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e1189bb1c0 0x55e1189b8ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:35:01.794 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:01.786+0000 7f220cde4640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e1189bb1c0 0x55e1189b8ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:49976/0 (socket says 172.21.15.46:49976) 2024-03-31T23:35:01.794 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:01.786+0000 7f220cde4640 1 -- 172.21.15.46:0/2978533009 learned_addr learned my addr 172.21.15.46:0/2978533009 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:35:01.794 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:01.786+0000 7f22077fe640 1 --2- 172.21.15.46:0/2978533009 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55e1189ada80 0x55e1189b8760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:35:01.794 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:01.786+0000 7f2207fff640 1 --2- 172.21.15.46:0/2978533009 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e1189ad160 0x55e1189b8220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:35:01.794 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:01.786+0000 7f220cde4640 1 -- 172.21.15.46:0/2978533009 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e1189ad160 msgr2=0x55e1189b8220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:35:01.794 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:01.786+0000 7f220cde4640 1 --2- 172.21.15.46:0/2978533009 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e1189ad160 0x55e1189b8220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:35:01.795 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:01.786+0000 7f220cde4640 1 -- 172.21.15.46:0/2978533009 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55e1189ada80 msgr2=0x55e1189b8760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:35:01.795 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:01.786+0000 7f220cde4640 1 --2- 172.21.15.46:0/2978533009 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55e1189ada80 0x55e1189b8760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:35:01.795 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:01.786+0000 7f220cde4640 1 -- 172.21.15.46:0/2978533009 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e1186f1620 con 0x55e1189bb1c0 2024-03-31T23:35:01.795 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:01.786+0000 7f22077fe640 1 --2- 172.21.15.46:0/2978533009 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55e1189ada80 0x55e1189b8760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:35:01.795 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:01.786+0000 7f220cde4640 1 --2- 172.21.15.46:0/2978533009 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e1189bb1c0 0x55e1189b8ca0 secure :-1 s=READY pgs=1027 cs=0 l=1 rev1=1 crypto rx=0x7f22000cc720 tx=0x7f220009bf20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b7ca50e9a807c50 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:35:01.795 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:01.786+0000 7f21e77fe640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-31T23:35:01.795 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:01.786+0000 7f21e77fe640 1 -- 172.21.15.46:0/2978533009 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22000b3070 con 0x55e1189bb1c0 2024-03-31T23:35:01.795 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:01.786+0000 7f21e77fe640 1 -- 172.21.15.46:0/2978533009 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(133..133 src has 1..133) ==== 9554+0+0 (secure 0 0 0) 0x7f2200097d60 con 0x55e1189bb1c0 2024-03-31T23:35:01.795 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:01.786+0000 7f221142e740 1 -- 172.21.15.46:0/2978533009 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f21f802c7e0 con 0x55e1189bb1c0 2024-03-31T23:35:01.795 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:01.787+0000 7f21e77fe640 1 -- 172.21.15.46:0/2978533009 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=54) ==== 24+0+0 (secure 0 0 0) 0x7f22000a4100 con 0x55e1189bb1c0 2024-03-31T23:35:01.795 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:01.787+0000 7f221142e740 10 client.18030 fetch_fsmap learned FSMap version 54 2024-03-31T23:35:01.796 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:01.787+0000 7f221142e740 1 -- 172.21.15.46:0/2978533009 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=54,monmap=4+}) -- 0x55e1189c9d10 con 0x55e1189bb1c0 2024-03-31T23:35:01.796 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:01.787+0000 7f21e77fe640 1 -- 172.21.15.46:0/2978533009 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f21f802c7e0 con 0x55e1189bb1c0 2024-03-31T23:35:01.796 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:01.787+0000 7f21e77fe640 1 -- 172.21.15.46:0/2978533009 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 54) ==== 8346+0+0 (secure 0 0 0) 0x7f22000a3c70 con 0x55e1189bb1c0 2024-03-31T23:35:01.796 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:01.787+0000 7f221142e740 10 client.18030 fetch_fsmap finished waiting for FSMap version 54 2024-03-31T23:35:01.796 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:01.787+0000 7f221142e740 10 client.18030 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-31T23:35:01.796 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:01.787+0000 7f221142e740 20 client.18030 populate_metadata read hostname 'smithi046' 2024-03-31T23:35:01.796 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:01.787+0000 7f221142e740 1 --2- 172.21.15.46:0/2978533009 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55e1189cb1d0 0x55e1189cd5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:35:01.796 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:01.787+0000 7f221142e740 4 client.18030 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-31T23:35:01.796 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:01.787+0000 7f221142e740 1 -- 172.21.15.46:0/2978533009 --> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e118924120 con 0x55e1189cb1d0 2024-03-31T23:35:01.796 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:01.787+0000 7f2207fff640 1 --2- 172.21.15.46:0/2978533009 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55e1189cb1d0 0x55e1189cd5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:35:01.796 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:01.788+0000 7f2207fff640 1 --2- 172.21.15.46:0/2978533009 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55e1189cb1d0 0x55e1189cd5c0 crc :-1 s=READY pgs=284 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c39466de798cf0ec server_cookie=27b34db0ba96728a in_seq=0 out_seq=0 2024-03-31T23:35:01.796 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:01.788+0000 7f21e77fe640 10 client.18030 ms_handle_connect on v2:172.21.15.142:6838/2519143918 2024-03-31T23:35:01.797 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:01.788+0000 7f21e77fe640 1 -- 172.21.15.46:0/2978533009 <== mds.0 v2:172.21.15.142:6838/2519143918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55e118924120 con 0x55e1189cb1d0 2024-03-31T23:35:01.797 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:01.788+0000 7f21e77fe640 10 client.18030 handle_command_reply: tid=0 2024-03-31T23:35:01.797 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:01.788+0000 7f221142e740 1 client.18030 shutdown 2024-03-31T23:35:01.797 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:01.788+0000 7f221142e740 1 -- 172.21.15.46:0/2978533009 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e1189bb1c0 msgr2=0x55e1189b8ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:35:01.797 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:01.788+0000 7f221142e740 1 --2- 172.21.15.46:0/2978533009 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e1189bb1c0 0x55e1189b8ca0 secure :-1 s=READY pgs=1027 cs=0 l=1 rev1=1 crypto rx=0x7f22000cc720 tx=0x7f220009bf20 comp rx=0 tx=0).stop 2024-03-31T23:35:01.797 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:01.788+0000 7f221142e740 1 -- 172.21.15.46:0/2978533009 shutdown_connections 2024-03-31T23:35:01.797 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:01.788+0000 7f221142e740 1 --2- 172.21.15.46:0/2978533009 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55e1189cb1d0 0x55e1189cd5c0 crc :-1 s=READY pgs=284 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:35:01.797 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:01.788+0000 7f221142e740 1 --2- 172.21.15.46:0/2978533009 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e1189bb1c0 0x55e1189b8ca0 unknown :-1 s=CLOSED pgs=1027 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:35:01.797 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:01.788+0000 7f221142e740 1 --2- 172.21.15.46:0/2978533009 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55e1189ada80 0x55e1189b8760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:35:01.797 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:01.788+0000 7f21e77fe640 0 client.18030 ms_handle_reset on v2:172.21.15.142:6838/2519143918 2024-03-31T23:35:01.797 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:01.788+0000 7f221142e740 1 --2- 172.21.15.46:0/2978533009 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e1189ad160 0x55e1189b8220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:35:01.797 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:01.788+0000 7f221142e740 1 -- 172.21.15.46:0/2978533009 >> 172.21.15.46:0/2978533009 conn(0x55e1189a7fe0 msgr2=0x55e1189a9d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:35:01.797 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:01.789+0000 7f221142e740 1 -- 172.21.15.46:0/2978533009 shutdown_connections 2024-03-31T23:35:01.797 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:01.789+0000 7f221142e740 1 -- 172.21.15.46:0/2978533009 wait complete. 2024-03-31T23:35:01.798 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:01.789+0000 7f221142e740 20 client.18030 trim_cache size 0 max 16384 2024-03-31T23:35:01.809 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:01.803+0000 7f221142e740 1 Processor -- start 2024-03-31T23:35:01.809 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:01.803+0000 7f221142e740 1 -- start start 2024-03-31T23:35:01.809 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:01.804+0000 7f221142e740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55e1189ad160 0x55e1189ba950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:35:01.810 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:01.804+0000 7f221142e740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e1189ada80 0x55e1189c6b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:35:01.810 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:01.804+0000 7f221142e740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e1189bb1c0 0x55e1189d2ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:35:01.810 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:01.804+0000 7f221142e740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f22081af330 con 0x55e1189bb1c0 2024-03-31T23:35:01.810 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:01.804+0000 7f221142e740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f22081acc00 con 0x55e1189ad160 2024-03-31T23:35:01.810 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:01.804+0000 7f221142e740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f22081aaa70 con 0x55e1189ada80 2024-03-31T23:35:01.810 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:01.804+0000 7f220cde4640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e1189bb1c0 0x55e1189d2ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:35:01.810 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:01.804+0000 7f220cde4640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e1189bb1c0 0x55e1189d2ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:49984/0 (socket says 172.21.15.46:49984) 2024-03-31T23:35:01.810 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:01.804+0000 7f220cde4640 1 -- 172.21.15.46:0/907904902 learned_addr learned my addr 172.21.15.46:0/907904902 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:35:01.810 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:01.804+0000 7f2207fff640 1 --2- 172.21.15.46:0/907904902 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55e1189ad160 0x55e1189ba950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:35:01.810 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:01.804+0000 7f22077fe640 1 --2- 172.21.15.46:0/907904902 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e1189ada80 0x55e1189c6b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:35:01.810 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:01.804+0000 7f220cde4640 1 -- 172.21.15.46:0/907904902 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e1189ada80 msgr2=0x55e1189c6b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:35:01.810 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:01.804+0000 7f220cde4640 1 --2- 172.21.15.46:0/907904902 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e1189ada80 0x55e1189c6b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:35:01.811 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:01.804+0000 7f220cde4640 1 -- 172.21.15.46:0/907904902 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55e1189ad160 msgr2=0x55e1189ba950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:35:01.811 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:01.804+0000 7f220cde4640 1 --2- 172.21.15.46:0/907904902 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55e1189ad160 0x55e1189ba950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:35:01.811 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:01.804+0000 7f220cde4640 1 -- 172.21.15.46:0/907904902 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e1189cd930 con 0x55e1189bb1c0 2024-03-31T23:35:01.811 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:01.804+0000 7f220cde4640 1 --2- 172.21.15.46:0/907904902 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e1189bb1c0 0x55e1189d2ed0 secure :-1 s=READY pgs=1028 cs=0 l=1 rev1=1 crypto rx=0x7f22000cccc0 tx=0x7f2200005730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b44c0a7107582e5 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:35:01.811 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:01.804+0000 7f21edffb640 1 -- 172.21.15.46:0/907904902 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22000b3070 con 0x55e1189bb1c0 2024-03-31T23:35:01.811 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:01.804+0000 7f21edffb640 1 -- 172.21.15.46:0/907904902 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f22000cf0b0 con 0x55e1189bb1c0 2024-03-31T23:35:01.811 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:01.804+0000 7f21edffb640 1 -- 172.21.15.46:0/907904902 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22000a3c70 con 0x55e1189bb1c0 2024-03-31T23:35:01.811 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:01.805+0000 7f221142e740 1 -- 172.21.15.46:0/907904902 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e1189bb1c0 msgr2=0x55e1189d2ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:35:01.811 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:01.805+0000 7f221142e740 1 --2- 172.21.15.46:0/907904902 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e1189bb1c0 0x55e1189d2ed0 secure :-1 s=READY pgs=1028 cs=0 l=1 rev1=1 crypto rx=0x7f22000cccc0 tx=0x7f2200005730 comp rx=0 tx=0).stop 2024-03-31T23:35:01.811 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:01.805+0000 7f221142e740 1 -- 172.21.15.46:0/907904902 shutdown_connections 2024-03-31T23:35:01.811 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:01.805+0000 7f221142e740 1 --2- 172.21.15.46:0/907904902 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e1189bb1c0 0x55e1189d2ed0 unknown :-1 s=CLOSED pgs=1028 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:35:01.811 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:01.805+0000 7f221142e740 1 --2- 172.21.15.46:0/907904902 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e1189ada80 0x55e1189c6b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:35:01.812 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:01.805+0000 7f221142e740 1 --2- 172.21.15.46:0/907904902 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55e1189ad160 0x55e1189ba950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:35:01.812 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:01.805+0000 7f221142e740 1 -- 172.21.15.46:0/907904902 >> 172.21.15.46:0/907904902 conn(0x55e1189a7fe0 msgr2=0x55e1189a83c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:35:01.812 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:01.805+0000 7f221142e740 1 -- 172.21.15.46:0/907904902 shutdown_connections 2024-03-31T23:35:01.812 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:01.805+0000 7f221142e740 1 -- 172.21.15.46:0/907904902 wait complete. 2024-03-31T23:35:01.812 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:01.805+0000 7f221142e740 1 -- start start 2024-03-31T23:35:01.812 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:01.806+0000 7f221142e740 1 Processor -- start 2024-03-31T23:35:01.812 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:01.806+0000 7f221142e740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55e1189ad160 0x55e1189b4ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:35:01.812 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:01.806+0000 7f221142e740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e1189ada80 0x55e1189b5430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:35:01.812 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:01.806+0000 7f221142e740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e1189bb1c0 0x55e1189b5970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:35:01.812 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:01.806+0000 7f221142e740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f22081ab0c0 con 0x55e1189bb1c0 2024-03-31T23:35:01.812 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:01.806+0000 7f221142e740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f22081ac530 con 0x55e1189ad160 2024-03-31T23:35:01.812 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:01.806+0000 7f221142e740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f22081ab790 con 0x55e1189ada80 2024-03-31T23:35:01.812 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:01.806+0000 7f221142e740 10 client.0 _ll_register_callbacks cb 0x7f220009b550 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T23:35:01.813 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:01.806+0000 7f220cde4640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e1189bb1c0 0x55e1189b5970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:35:01.813 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:01.806+0000 7f220cde4640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e1189bb1c0 0x55e1189b5970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:50000/0 (socket says 172.21.15.46:50000) 2024-03-31T23:35:01.813 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:01.806+0000 7f220cde4640 1 -- 172.21.15.46:0/1426101449 learned_addr learned my addr 172.21.15.46:0/1426101449 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:35:01.813 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:01.806+0000 7f22077fe640 1 --2- 172.21.15.46:0/1426101449 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e1189ada80 0x55e1189b5430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:35:01.813 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:01.806+0000 7f2207fff640 1 --2- 172.21.15.46:0/1426101449 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55e1189ad160 0x55e1189b4ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:35:01.813 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:01.806+0000 7f220cde4640 1 -- 172.21.15.46:0/1426101449 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e1189ada80 msgr2=0x55e1189b5430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:35:01.813 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:01.806+0000 7f220cde4640 1 --2- 172.21.15.46:0/1426101449 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e1189ada80 0x55e1189b5430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:35:01.813 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:01.806+0000 7f220cde4640 1 -- 172.21.15.46:0/1426101449 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55e1189ad160 msgr2=0x55e1189b4ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:35:01.813 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:01.806+0000 7f220cde4640 1 --2- 172.21.15.46:0/1426101449 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55e1189ad160 0x55e1189b4ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:35:01.813 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:01.806+0000 7f220cde4640 1 -- 172.21.15.46:0/1426101449 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e118a30660 con 0x55e1189bb1c0 2024-03-31T23:35:01.813 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:01.806+0000 7f220cde4640 1 --2- 172.21.15.46:0/1426101449 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e1189bb1c0 0x55e1189b5970 secure :-1 s=READY pgs=1029 cs=0 l=1 rev1=1 crypto rx=0x7f22000a3bd0 tx=0x7f220009bf20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f5aaa5cdd63025ad server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:35:01.814 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:01.806+0000 7f21e77fe640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-31T23:35:01.814 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:01.807+0000 7f21e77fe640 1 -- 172.21.15.46:0/1426101449 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22000b3050 con 0x55e1189bb1c0 2024-03-31T23:35:01.814 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:01.807+0000 7f21e77fe640 1 -- 172.21.15.46:0/1426101449 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(133..133 src has 1..133) ==== 9554+0+0 (secure 0 0 0) 0x7f22000a3e70 con 0x55e1189bb1c0 2024-03-31T23:35:01.814 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:01.807+0000 7f221142e740 1 -- 172.21.15.46:0/1426101449 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f22000cf0b0 con 0x55e1189bb1c0 2024-03-31T23:35:01.814 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:01.807+0000 7f21e77fe640 1 -- 172.21.15.46:0/1426101449 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=54) ==== 24+0+0 (secure 0 0 0) 0x7f22000b4470 con 0x55e1189bb1c0 2024-03-31T23:35:01.814 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:01.807+0000 7f221142e740 10 client.18036 fetch_fsmap learned FSMap version 54 2024-03-31T23:35:01.814 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:01.807+0000 7f221142e740 1 -- 172.21.15.46:0/1426101449 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=54,monmap=4+}) -- 0x55e118a318f0 con 0x55e1189bb1c0 2024-03-31T23:35:01.814 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:01.807+0000 7f21e77fe640 1 -- 172.21.15.46:0/1426101449 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f22000cf0b0 con 0x55e1189bb1c0 2024-03-31T23:35:01.814 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:01.807+0000 7f21e77fe640 1 -- 172.21.15.46:0/1426101449 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 54) ==== 8346+0+0 (secure 0 0 0) 0x7f2200083020 con 0x55e1189bb1c0 2024-03-31T23:35:01.814 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:01.807+0000 7f221142e740 10 client.18036 fetch_fsmap finished waiting for FSMap version 54 2024-03-31T23:35:01.815 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:01.807+0000 7f221142e740 10 client.18036 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-31T23:35:01.815 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:01.807+0000 7f221142e740 20 client.18036 populate_metadata read hostname 'smithi046' 2024-03-31T23:35:01.815 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:01.807+0000 7f221142e740 1 --2- 172.21.15.46:0/1426101449 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55e118a31cd0 0x55e118a340c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:35:01.815 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:01.807+0000 7f221142e740 4 client.18036 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-31T23:35:01.815 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:01.807+0000 7f221142e740 1 -- 172.21.15.46:0/1426101449 --> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55e11891de30 con 0x55e118a31cd0 2024-03-31T23:35:01.815 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:01.807+0000 7f2207fff640 1 --2- 172.21.15.46:0/1426101449 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55e118a31cd0 0x55e118a340c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:35:01.815 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:01.808+0000 7f2207fff640 1 --2- 172.21.15.46:0/1426101449 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55e118a31cd0 0x55e118a340c0 crc :-1 s=READY pgs=285 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a887b1de2743d21 server_cookie=57120fad693fb090 in_seq=0 out_seq=0 2024-03-31T23:35:01.815 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:01.808+0000 7f21e77fe640 10 client.18036 ms_handle_connect on v2:172.21.15.142:6838/2519143918 2024-03-31T23:35:02.068 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:02.066+0000 7f21e77fe640 1 -- 172.21.15.46:0/1426101449 <== mds.0 v2:172.21.15.142:6838/2519143918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55e11891de30 con 0x55e118a31cd0 2024-03-31T23:35:02.068 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:02.066+0000 7f21e77fe640 10 client.18036 handle_command_reply: tid=0 2024-03-31T23:35:02.068 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:02.066+0000 7f221142e740 1 client.18036 shutdown 2024-03-31T23:35:02.069 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:02.067+0000 7f221142e740 1 -- 172.21.15.46:0/1426101449 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e1189bb1c0 msgr2=0x55e1189b5970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:35:02.069 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:02.067+0000 7f221142e740 1 --2- 172.21.15.46:0/1426101449 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e1189bb1c0 0x55e1189b5970 secure :-1 s=READY pgs=1029 cs=0 l=1 rev1=1 crypto rx=0x7f22000a3bd0 tx=0x7f220009bf20 comp rx=0 tx=0).stop 2024-03-31T23:35:02.069 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:02.067+0000 7f221142e740 1 -- 172.21.15.46:0/1426101449 shutdown_connections 2024-03-31T23:35:02.069 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:02.067+0000 7f221142e740 1 --2- 172.21.15.46:0/1426101449 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55e118a31cd0 0x55e118a340c0 crc :-1 s=READY pgs=285 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:35:02.069 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:02.067+0000 7f221142e740 1 --2- 172.21.15.46:0/1426101449 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e1189bb1c0 0x55e1189b5970 unknown :-1 s=CLOSED pgs=1029 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:35:02.069 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:02.067+0000 7f221142e740 1 --2- 172.21.15.46:0/1426101449 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e1189ada80 0x55e1189b5430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:35:02.069 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:02.067+0000 7f21e77fe640 0 client.18036 ms_handle_reset on v2:172.21.15.142:6838/2519143918 2024-03-31T23:35:02.069 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:02.067+0000 7f221142e740 1 --2- 172.21.15.46:0/1426101449 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55e1189ad160 0x55e1189b4ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:35:02.069 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:02.067+0000 7f221142e740 1 -- 172.21.15.46:0/1426101449 >> 172.21.15.46:0/1426101449 conn(0x55e1189a7fe0 msgr2=0x55e1189b81e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:35:02.069 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:02.067+0000 7f221142e740 1 -- 172.21.15.46:0/1426101449 shutdown_connections 2024-03-31T23:35:02.069 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:02.067+0000 7f221142e740 1 -- 172.21.15.46:0/1426101449 wait complete. 2024-03-31T23:35:02.069 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:02.067+0000 7f221142e740 20 client.18036 trim_cache size 0 max 16384 2024-03-31T23:35:02.070 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout:{ 2024-03-31T23:35:02.070 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout: "message": "", 2024-03-31T23:35:02.070 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout: "return_code": 0 2024-03-31T23:35:02.070 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout:} 2024-03-31T23:35:02.071 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:02.067+0000 7f220e847640 1 -- 172.21.15.46:0/3618567887 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f21f00a26b0 msgr2=0x7f21f00a4b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:35:02.071 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:02.067+0000 7f220e847640 1 --2- 172.21.15.46:0/3618567887 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f21f00a26b0 0x7f21f00a4b70 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f2208077ba0 tx=0x7f21fc011810 comp rx=0 tx=0).stop 2024-03-31T23:35:02.071 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:02.068+0000 7f220e847640 1 -- 172.21.15.46:0/3618567887 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2208107730 msgr2=0x7f220819c7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:35:02.071 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:02.068+0000 7f220e847640 1 --2- 172.21.15.46:0/3618567887 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2208107730 0x7f220819c7a0 secure :-1 s=READY pgs=1025 cs=0 l=1 rev1=1 crypto rx=0x7f220000ea60 tx=0x7f220000ef20 comp rx=0 tx=0).stop 2024-03-31T23:35:02.071 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:02.068+0000 7f220e847640 1 -- 172.21.15.46:0/3618567887 shutdown_connections 2024-03-31T23:35:02.071 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:02.068+0000 7f220e847640 1 --2- 172.21.15.46:0/3618567887 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f21f00a26b0 0x7f21f00a4b70 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:35:02.071 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:02.068+0000 7f220e847640 1 --2- 172.21.15.46:0/3618567887 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2208107730 0x7f220819c7a0 unknown :-1 s=CLOSED pgs=1025 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:35:02.071 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:02.068+0000 7f220e847640 1 --2- 172.21.15.46:0/3618567887 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2208069750 0x7f2208198420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:35:02.072 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:02.068+0000 7f220e847640 1 --2- 172.21.15.46:0/3618567887 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2208068dd0 0x7f2208197ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:35:02.072 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:02.068+0000 7f220e847640 1 -- 172.21.15.46:0/3618567887 >> 172.21.15.46:0/3618567887 conn(0x7f22080774a0 msgr2=0x7f22080ff0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:35:02.072 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:02.068+0000 7f220e847640 1 -- 172.21.15.46:0/3618567887 shutdown_connections 2024-03-31T23:35:02.072 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:02.068+0000 7f220e847640 1 -- 172.21.15.46:0/3618567887 wait complete. 2024-03-31T23:35:02.312 INFO:tasks.workunit.client.0.smithi046.stdout: 10 93832 8.21 MB/sec execute 487 sec latency 1144.612 ms 2024-03-31T23:35:02.443 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:35:02 smithi142 ceph-mon[35271]: pgmap v739: 129 pgs: 129 active+clean; 1.2 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 4.7 MiB/s wr, 209 op/s 2024-03-31T23:35:02.545 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:35:02 smithi162 ceph-mon[36237]: pgmap v739: 129 pgs: 129 active+clean; 1.2 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 4.7 MiB/s wr, 209 op/s 2024-03-31T23:35:02.611 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:35:02 smithi046 ceph-mon[32498]: pgmap v739: 129 pgs: 129 active+clean; 1.2 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 4.7 MiB/s wr, 209 op/s 2024-03-31T23:35:03.314 INFO:tasks.workunit.client.0.smithi046.stdout: 10 93969 8.20 MB/sec execute 488 sec latency 154.460 ms 2024-03-31T23:35:04.314 INFO:tasks.workunit.client.0.smithi046.stdout: 10 94081 8.20 MB/sec execute 489 sec latency 177.385 ms 2024-03-31T23:35:04.443 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:35:04 smithi142 ceph-mon[35271]: pgmap v740: 129 pgs: 129 active+clean; 1.2 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 4.7 MiB/s wr, 209 op/s 2024-03-31T23:35:04.443 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:35:04 smithi142 ceph-mon[35271]: osdmap e134: 12 total, 12 up, 12 in 2024-03-31T23:35:04.545 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:35:04 smithi162 ceph-mon[36237]: pgmap v740: 129 pgs: 129 active+clean; 1.2 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 4.7 MiB/s wr, 209 op/s 2024-03-31T23:35:04.545 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:35:04 smithi162 ceph-mon[36237]: osdmap e134: 12 total, 12 up, 12 in 2024-03-31T23:35:04.611 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:35:04 smithi046 ceph-mon[32498]: pgmap v740: 129 pgs: 129 active+clean; 1.2 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 4.7 MiB/s wr, 209 op/s 2024-03-31T23:35:04.611 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:35:04 smithi046 ceph-mon[32498]: osdmap e134: 12 total, 12 up, 12 in 2024-03-31T23:35:05.314 INFO:tasks.workunit.client.0.smithi046.stdout: 10 94225 8.21 MB/sec execute 490 sec latency 64.932 ms 2024-03-31T23:35:06.314 INFO:tasks.workunit.client.0.smithi046.stdout: 10 94414 8.22 MB/sec execute 491 sec latency 95.899 ms 2024-03-31T23:35:06.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:35:06 smithi142 ceph-mon[35271]: pgmap v742: 129 pgs: 129 active+clean; 1.2 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 5.1 MiB/s wr, 304 op/s 2024-03-31T23:35:06.944 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:35:06 smithi142 ceph-mon[35271]: osdmap e135: 12 total, 12 up, 12 in 2024-03-31T23:35:07.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:35:06 smithi162 ceph-mon[36237]: pgmap v742: 129 pgs: 129 active+clean; 1.2 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 5.1 MiB/s wr, 304 op/s 2024-03-31T23:35:07.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:35:06 smithi162 ceph-mon[36237]: osdmap e135: 12 total, 12 up, 12 in 2024-03-31T23:35:07.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:35:06 smithi046 ceph-mon[32498]: pgmap v742: 129 pgs: 129 active+clean; 1.2 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 5.1 MiB/s wr, 304 op/s 2024-03-31T23:35:07.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:35:06 smithi046 ceph-mon[32498]: osdmap e135: 12 total, 12 up, 12 in 2024-03-31T23:35:07.314 INFO:tasks.workunit.client.0.smithi046.stdout: 10 94561 8.22 MB/sec execute 492 sec latency 161.904 ms 2024-03-31T23:35:08.314 INFO:tasks.workunit.client.0.smithi046.stdout: 10 94675 8.21 MB/sec execute 493 sec latency 99.704 ms 2024-03-31T23:35:08.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:35:08 smithi142 ceph-mon[35271]: pgmap v744: 129 pgs: 129 active+clean; 1.2 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.9 MiB/s wr, 232 op/s 2024-03-31T23:35:09.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:35:08 smithi162 ceph-mon[36237]: pgmap v744: 129 pgs: 129 active+clean; 1.2 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.9 MiB/s wr, 232 op/s 2024-03-31T23:35:09.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:35:08 smithi046 ceph-mon[32498]: pgmap v744: 129 pgs: 129 active+clean; 1.2 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.9 MiB/s wr, 232 op/s 2024-03-31T23:35:09.314 INFO:tasks.workunit.client.0.smithi046.stdout: 10 94802 8.21 MB/sec execute 494 sec latency 75.755 ms 2024-03-31T23:35:10.314 INFO:tasks.workunit.client.0.smithi046.stdout: 10 94938 8.21 MB/sec execute 495 sec latency 97.298 ms 2024-03-31T23:35:10.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:35:10 smithi142 ceph-mon[35271]: pgmap v745: 129 pgs: 129 active+clean; 1.2 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 6.8 MiB/s wr, 403 op/s 2024-03-31T23:35:11.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:35:10 smithi162 ceph-mon[36237]: pgmap v745: 129 pgs: 129 active+clean; 1.2 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 6.8 MiB/s wr, 403 op/s 2024-03-31T23:35:11.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:35:10 smithi046 ceph-mon[32498]: pgmap v745: 129 pgs: 129 active+clean; 1.2 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 6.8 MiB/s wr, 403 op/s 2024-03-31T23:35:11.314 INFO:tasks.workunit.client.0.smithi046.stdout: 10 95067 8.20 MB/sec execute 496 sec latency 70.224 ms 2024-03-31T23:35:12.314 INFO:tasks.workunit.client.0.smithi046.stdout: 10 95228 8.20 MB/sec execute 497 sec latency 82.129 ms 2024-03-31T23:35:12.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:35:12 smithi142 ceph-mon[35271]: pgmap v746: 129 pgs: 129 active+clean; 1.2 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 6.8 MiB/s wr, 403 op/s 2024-03-31T23:35:13.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:35:12 smithi162 ceph-mon[36237]: pgmap v746: 129 pgs: 129 active+clean; 1.2 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 6.8 MiB/s wr, 403 op/s 2024-03-31T23:35:13.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:35:12 smithi046 ceph-mon[32498]: pgmap v746: 129 pgs: 129 active+clean; 1.2 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 6.8 MiB/s wr, 403 op/s 2024-03-31T23:35:13.314 INFO:tasks.workunit.client.0.smithi046.stdout: 10 95374 8.19 MB/sec execute 498 sec latency 72.947 ms 2024-03-31T23:35:14.314 INFO:tasks.workunit.client.0.smithi046.stdout: 10 95503 8.18 MB/sec execute 499 sec latency 87.625 ms 2024-03-31T23:35:15.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:35:14 smithi162 ceph-mon[36237]: pgmap v747: 129 pgs: 129 active+clean; 1.2 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 5.5 MiB/s wr, 325 op/s 2024-03-31T23:35:15.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:35:14 smithi046 ceph-mon[32498]: pgmap v747: 129 pgs: 129 active+clean; 1.2 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 5.5 MiB/s wr, 325 op/s 2024-03-31T23:35:15.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:35:14 smithi142 ceph-mon[35271]: pgmap v747: 129 pgs: 129 active+clean; 1.2 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 5.5 MiB/s wr, 325 op/s 2024-03-31T23:35:15.199 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:15.198+0000 7f98159bf640 1 -- 172.21.15.46:0/3659472854 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f98101016f0 msgr2=0x7f9810101b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:35:15.199 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:15.198+0000 7f98159bf640 1 --2- 172.21.15.46:0/3659472854 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f98101016f0 0x7f9810101b70 secure :-1 s=READY pgs=1030 cs=0 l=1 rev1=1 crypto rx=0x7f9804009930 tx=0x7f980402bc30 comp rx=0 tx=0).stop 2024-03-31T23:35:15.199 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:15.198+0000 7f98159bf640 1 -- 172.21.15.46:0/3659472854 shutdown_connections 2024-03-31T23:35:15.199 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:15.198+0000 7f98159bf640 1 --2- 172.21.15.46:0/3659472854 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f98101020b0 0x7f981010e5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:35:15.199 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:15.198+0000 7f98159bf640 1 --2- 172.21.15.46:0/3659472854 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f98101016f0 0x7f9810101b70 unknown :-1 s=CLOSED pgs=1030 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:35:15.199 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:15.198+0000 7f98159bf640 1 --2- 172.21.15.46:0/3659472854 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9810107700 0x7f9810107ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:35:15.199 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:15.198+0000 7f98159bf640 1 -- 172.21.15.46:0/3659472854 >> 172.21.15.46:0/3659472854 conn(0x7f98100fd1e0 msgr2=0x7f98100ff600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:35:15.199 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:15.198+0000 7f98159bf640 1 -- 172.21.15.46:0/3659472854 shutdown_connections 2024-03-31T23:35:15.199 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:15.198+0000 7f98159bf640 1 -- 172.21.15.46:0/3659472854 wait complete. 2024-03-31T23:35:15.200 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:15.199+0000 7f98159bf640 1 Processor -- start 2024-03-31T23:35:15.200 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:15.199+0000 7f98159bf640 1 -- start start 2024-03-31T23:35:15.201 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:15.200+0000 7f98159bf640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f98101016f0 0x7f9810073a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:35:15.201 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:15.200+0000 7f98159bf640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f98101020b0 0x7f9810073f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:35:15.201 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:15.200+0000 7f98159bf640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9810107700 0x7f98100744d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:35:15.201 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:15.200+0000 7f98159bf640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f98101119c0 con 0x7f98101016f0 2024-03-31T23:35:15.201 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:15.200+0000 7f98159bf640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f9810111840 con 0x7f98101020b0 2024-03-31T23:35:15.201 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:15.200+0000 7f98159bf640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f9810111b40 con 0x7f9810107700 2024-03-31T23:35:15.201 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:15.200+0000 7f980effd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f98101016f0 0x7f9810073a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:35:15.201 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:15.200+0000 7f980effd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f98101016f0 0x7f9810073a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:56714/0 (socket says 172.21.15.46:56714) 2024-03-31T23:35:15.201 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:15.200+0000 7f980effd640 1 -- 172.21.15.46:0/1763003844 learned_addr learned my addr 172.21.15.46:0/1763003844 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:35:15.202 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:15.200+0000 7f980e7fc640 1 --2- 172.21.15.46:0/1763003844 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f98101020b0 0x7f9810073f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:35:15.202 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:15.200+0000 7f980f7fe640 1 --2- 172.21.15.46:0/1763003844 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9810107700 0x7f98100744d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:35:15.202 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:15.201+0000 7f980e7fc640 1 -- 172.21.15.46:0/1763003844 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9810107700 msgr2=0x7f98100744d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:35:15.203 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:15.201+0000 7f980e7fc640 1 --2- 172.21.15.46:0/1763003844 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9810107700 0x7f98100744d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:35:15.203 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:15.201+0000 7f980e7fc640 1 -- 172.21.15.46:0/1763003844 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f98101016f0 msgr2=0x7f9810073a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:35:15.203 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:15.201+0000 7f980e7fc640 1 --2- 172.21.15.46:0/1763003844 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f98101016f0 0x7f9810073a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:35:15.203 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:15.201+0000 7f980e7fc640 1 -- 172.21.15.46:0/1763003844 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f98101a63a0 con 0x7f98101020b0 2024-03-31T23:35:15.203 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:15.201+0000 7f980effd640 1 --2- 172.21.15.46:0/1763003844 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f98101016f0 0x7f9810073a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:35:15.203 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:15.201+0000 7f980f7fe640 1 --2- 172.21.15.46:0/1763003844 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9810107700 0x7f98100744d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T23:35:15.203 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:15.201+0000 7f980e7fc640 1 --2- 172.21.15.46:0/1763003844 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f98101020b0 0x7f9810073f90 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f9804009230 tx=0x7f98040045e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:35:15.203 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:15.201+0000 7f98149bd640 1 -- 172.21.15.46:0/1763003844 <== mon.2 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9804043020 con 0x7f98101020b0 2024-03-31T23:35:15.203 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:15.201+0000 7f98149bd640 1 -- 172.21.15.46:0/1763003844 <== mon.2 v2:172.21.15.142:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f980402ca70 con 0x7f98101020b0 2024-03-31T23:35:15.203 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:15.201+0000 7f98149bd640 1 -- 172.21.15.46:0/1763003844 <== mon.2 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98040039b0 con 0x7f98101020b0 2024-03-31T23:35:15.203 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:15.201+0000 7f98159bf640 1 -- 172.21.15.46:0/1763003844 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f98101a6630 con 0x7f98101020b0 2024-03-31T23:35:15.203 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:15.202+0000 7f98159bf640 1 -- 172.21.15.46:0/1763003844 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f98101a6a40 con 0x7f98101020b0 2024-03-31T23:35:15.205 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:15.205+0000 7f98149bd640 1 -- 172.21.15.46:0/1763003844 <== mon.2 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 25) ==== 137451+0+0 (secure 0 0 0) 0x7f9804046020 con 0x7f98101020b0 2024-03-31T23:35:15.206 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:15.205+0000 7f9818595740 1 Processor -- start 2024-03-31T23:35:15.206 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:15.205+0000 7f9818595740 1 -- start start 2024-03-31T23:35:15.208 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:15.205+0000 7f9818595740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56142968b6c0 0x56142968baa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:35:15.208 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:15.205+0000 7f9818595740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56142968bfe0 0x561429694870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:35:15.208 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:15.205+0000 7f9818595740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561429694db0 0x5614296971a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:35:15.208 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:15.206+0000 7f9818595740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x5614295ee8f0 con 0x56142968b6c0 2024-03-31T23:35:15.208 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:15.206+0000 7f9818595740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x5614295978d0 con 0x56142968bfe0 2024-03-31T23:35:15.208 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:15.206+0000 7f9818595740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5614295ea990 con 0x561429694db0 2024-03-31T23:35:15.208 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:15.206+0000 7f980f7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56142968b6c0 0x56142968baa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:35:15.208 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:15.206+0000 7f980effd640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56142968bfe0 0x561429694870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:35:15.208 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:15.206+0000 7f980effd640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56142968bfe0 0x561429694870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.46:46948/0 (socket says 172.21.15.46:46948) 2024-03-31T23:35:15.208 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:15.206+0000 7f980effd640 1 -- 172.21.15.46:0/2543221830 learned_addr learned my addr 172.21.15.46:0/2543221830 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:35:15.208 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:15.206+0000 7f980e7fc640 1 --2- 172.21.15.46:0/2543221830 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561429694db0 0x5614296971a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:35:15.209 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:15.206+0000 7f98149bd640 1 --2- 172.21.15.46:0/1763003844 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f97f80a2850 0x7f97f80a4d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:35:15.209 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:15.206+0000 7f980effd640 1 -- 172.21.15.46:0/2543221830 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561429694db0 msgr2=0x5614296971a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:35:15.209 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:15.206+0000 7f980effd640 1 --2- 172.21.15.46:0/2543221830 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561429694db0 0x5614296971a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:35:15.209 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:15.206+0000 7f980effd640 1 -- 172.21.15.46:0/2543221830 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56142968b6c0 msgr2=0x56142968baa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:35:15.209 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:15.206+0000 7f980effd640 1 --2- 172.21.15.46:0/2543221830 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56142968b6c0 0x56142968baa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:35:15.209 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:15.206+0000 7f980effd640 1 -- 172.21.15.46:0/2543221830 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561429303210 con 0x56142968bfe0 2024-03-31T23:35:15.209 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:15.206+0000 7f980f7fe640 1 --2- 172.21.15.46:0/1763003844 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f97f80a2850 0x7f97f80a4d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:35:15.209 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:15.206+0000 7f98149bd640 1 -- 172.21.15.46:0/1763003844 <== mon.2 v2:172.21.15.142:3300/0 5 ==== osd_map(135..135 src has 1..135) ==== 9554+0+0 (secure 0 0 0) 0x7f98040f0cd0 con 0x7f98101020b0 2024-03-31T23:35:15.209 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:15.206+0000 7f980effd640 1 --2- 172.21.15.46:0/2543221830 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56142968bfe0 0x561429694870 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f98000115f0 tx=0x7f9800011ab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3adbbf15764dfc63 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:35:15.209 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:15.207+0000 7f97f5ffb640 1 -- 172.21.15.46:0/2543221830 <== mon.2 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f980001e070 con 0x56142968bfe0 2024-03-31T23:35:15.209 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:15.207+0000 7f980f7fe640 1 --2- 172.21.15.46:0/1763003844 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f97f80a2850 0x7f97f80a4d10 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f9808009fd0 tx=0x7f9808009bc0 comp rx=0 tx=0).ready entity=mgr.15654 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:35:15.209 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:15.207+0000 7f97f5ffb640 1 -- 172.21.15.46:0/2543221830 <== mon.2 v2:172.21.15.142:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f980000fda0 con 0x56142968bfe0 2024-03-31T23:35:15.209 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:15.207+0000 7f97f5ffb640 1 -- 172.21.15.46:0/2543221830 <== mon.2 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9800019e50 con 0x56142968bfe0 2024-03-31T23:35:15.213 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:15.208+0000 7f9818595740 1 -- 172.21.15.46:0/2543221830 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56142968bfe0 msgr2=0x561429694870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:35:15.213 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:15.208+0000 7f9818595740 1 --2- 172.21.15.46:0/2543221830 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56142968bfe0 0x561429694870 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f98000115f0 tx=0x7f9800011ab0 comp rx=0 tx=0).stop 2024-03-31T23:35:15.213 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:15.208+0000 7f9818595740 1 -- 172.21.15.46:0/2543221830 shutdown_connections 2024-03-31T23:35:15.213 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:15.208+0000 7f9818595740 1 --2- 172.21.15.46:0/2543221830 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561429694db0 0x5614296971a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:35:15.213 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:15.208+0000 7f9818595740 1 --2- 172.21.15.46:0/2543221830 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56142968bfe0 0x561429694870 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:35:15.213 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:15.208+0000 7f9818595740 1 --2- 172.21.15.46:0/2543221830 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56142968b6c0 0x56142968baa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:35:15.213 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:15.208+0000 7f9818595740 1 -- 172.21.15.46:0/2543221830 >> 172.21.15.46:0/2543221830 conn(0x561429685fe0 msgr2=0x5614296863c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:35:15.214 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:15.208+0000 7f9818595740 1 -- 172.21.15.46:0/2543221830 shutdown_connections 2024-03-31T23:35:15.214 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:15.208+0000 7f9818595740 1 -- 172.21.15.46:0/2543221830 wait complete. 2024-03-31T23:35:15.214 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:15.208+0000 7f9818595740 1 -- start start 2024-03-31T23:35:15.214 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:15.208+0000 7f9818595740 1 Processor -- start 2024-03-31T23:35:15.214 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:15.209+0000 7f9818595740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56142968b6c0 0x561429691e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:35:15.214 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:15.209+0000 7f9818595740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56142968bfe0 0x5614296923b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:35:15.214 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:15.209+0000 7f9818595740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x561429694db0 0x5614296928f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:35:15.214 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:15.209+0000 7f9818595740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x5614296979e0 con 0x56142968bfe0 2024-03-31T23:35:15.214 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:15.209+0000 7f9818595740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x56142958cf80 con 0x561429694db0 2024-03-31T23:35:15.214 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:15.209+0000 7f9818595740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x561429697b60 con 0x56142968b6c0 2024-03-31T23:35:15.214 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:15.209+0000 7f9818595740 10 client.0 _ll_register_callbacks cb 0x5614295e5290 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T23:35:15.214 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:15.209+0000 7f980f7fe640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x561429694db0 0x5614296928f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:35:15.214 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:15.209+0000 7f980e7fc640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56142968bfe0 0x5614296923b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:35:15.215 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:15.209+0000 7f980f7fe640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x561429694db0 0x5614296928f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.46:46950/0 (socket says 172.21.15.46:46950) 2024-03-31T23:35:15.215 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:15.209+0000 7f980f7fe640 1 -- 172.21.15.46:0/2107443316 learned_addr learned my addr 172.21.15.46:0/2107443316 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:35:15.215 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:15.209+0000 7f980effd640 1 --2- 172.21.15.46:0/2107443316 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56142968b6c0 0x561429691e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:35:15.215 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:15.209+0000 7f980f7fe640 1 -- 172.21.15.46:0/2107443316 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56142968b6c0 msgr2=0x561429691e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:35:15.215 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:15.209+0000 7f980f7fe640 1 --2- 172.21.15.46:0/2107443316 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56142968b6c0 0x561429691e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:35:15.215 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:15.209+0000 7f980f7fe640 1 -- 172.21.15.46:0/2107443316 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56142968bfe0 msgr2=0x5614296923b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:35:15.215 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:15.209+0000 7f980f7fe640 1 --2- 172.21.15.46:0/2107443316 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56142968bfe0 0x5614296923b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:35:15.215 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:15.209+0000 7f980f7fe640 1 -- 172.21.15.46:0/2107443316 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5614293cf620 con 0x561429694db0 2024-03-31T23:35:15.215 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:15.209+0000 7f980e7fc640 1 --2- 172.21.15.46:0/2107443316 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56142968bfe0 0x5614296923b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:35:15.215 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:15.210+0000 7f980f7fe640 1 --2- 172.21.15.46:0/2107443316 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x561429694db0 0x5614296928f0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f9808012370 tx=0x7f9808009a30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a18877c686bfe5c3 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:35:15.215 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:15.210+0000 7f97eeffd640 10 client.0 ms_handle_connect on v2:172.21.15.142:3300/0 2024-03-31T23:35:15.215 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:15.210+0000 7f97eeffd640 1 -- 172.21.15.46:0/2107443316 <== mon.2 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9808028430 con 0x561429694db0 2024-03-31T23:35:15.215 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:15.210+0000 7f97eeffd640 1 -- 172.21.15.46:0/2107443316 <== mon.2 v2:172.21.15.142:3300/0 2 ==== osd_map(135..135 src has 1..135) ==== 9554+0+0 (secure 0 0 0) 0x7f980801e400 con 0x561429694db0 2024-03-31T23:35:15.215 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:15.210+0000 7f9818595740 1 -- 172.21.15.46:0/2107443316 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f980000fda0 con 0x561429694db0 2024-03-31T23:35:15.216 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:15.210+0000 7f97eeffd640 1 -- 172.21.15.46:0/2107443316 <== mon.2 v2:172.21.15.142:3300/0 3 ==== mon_get_version_reply(handle=1 version=54) ==== 24+0+0 (secure 0 0 0) 0x7f980801e640 con 0x561429694db0 2024-03-31T23:35:15.216 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:15.210+0000 7f9818595740 10 client.27854 fetch_fsmap learned FSMap version 54 2024-03-31T23:35:15.216 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:15.210+0000 7f9818595740 1 -- 172.21.15.46:0/2107443316 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,fsmap=54,monmap=4+}) -- 0x5614296a3940 con 0x561429694db0 2024-03-31T23:35:15.216 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:15.210+0000 7f97eeffd640 1 -- 172.21.15.46:0/2107443316 <== mon.2 v2:172.21.15.142:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f980000fda0 con 0x561429694db0 2024-03-31T23:35:15.216 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:15.211+0000 7f97eeffd640 1 -- 172.21.15.46:0/2107443316 <== mon.2 v2:172.21.15.142:3300/0 5 ==== fsmap(e 54) ==== 8346+0+0 (secure 0 0 0) 0x7f98080289f0 con 0x561429694db0 2024-03-31T23:35:15.216 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:15.211+0000 7f9818595740 10 client.27854 fetch_fsmap finished waiting for FSMap version 54 2024-03-31T23:35:15.216 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:15.211+0000 7f9818595740 10 client.27854 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-31T23:35:15.216 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:15.211+0000 7f9818595740 20 client.27854 populate_metadata read hostname 'smithi046' 2024-03-31T23:35:15.216 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:15.211+0000 7f9818595740 1 --2- 172.21.15.46:0/2107443316 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x5614296a5260 0x5614296a76e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:35:15.216 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:15.211+0000 7f9818595740 4 client.27854 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-31T23:35:15.216 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:15.211+0000 7f9818595740 1 -- 172.21.15.46:0/2107443316 --> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561429602120 con 0x5614296a5260 2024-03-31T23:35:15.216 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:15.211+0000 7f980effd640 1 --2- 172.21.15.46:0/2107443316 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x5614296a5260 0x5614296a76e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:35:15.216 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:15.211+0000 7f97eeffd640 10 client.27854 ms_handle_connect on v2:172.21.15.142:6838/2519143918 2024-03-31T23:35:15.217 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:15.212+0000 7f980effd640 1 --2- 172.21.15.46:0/2107443316 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x5614296a5260 0x5614296a76e0 crc :-1 s=READY pgs=286 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=99d0ab433df77555 server_cookie=f91832e6630e7bc6 in_seq=0 out_seq=0 2024-03-31T23:35:15.218 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:15.213+0000 7f97eeffd640 1 -- 172.21.15.46:0/2107443316 <== mds.0 v2:172.21.15.142:6838/2519143918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x561429602120 con 0x5614296a5260 2024-03-31T23:35:15.218 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:15.213+0000 7f97eeffd640 10 client.27854 handle_command_reply: tid=0 2024-03-31T23:35:15.218 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:15.213+0000 7f9818595740 1 client.27854 shutdown 2024-03-31T23:35:15.218 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:15.213+0000 7f9818595740 1 -- 172.21.15.46:0/2107443316 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x561429694db0 msgr2=0x5614296928f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:35:15.218 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:15.213+0000 7f9818595740 1 --2- 172.21.15.46:0/2107443316 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x561429694db0 0x5614296928f0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f9808012370 tx=0x7f9808009a30 comp rx=0 tx=0).stop 2024-03-31T23:35:15.218 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:15.214+0000 7f9818595740 1 -- 172.21.15.46:0/2107443316 shutdown_connections 2024-03-31T23:35:15.218 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:15.214+0000 7f9818595740 1 --2- 172.21.15.46:0/2107443316 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x5614296a5260 0x5614296a76e0 crc :-1 s=READY pgs=286 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:35:15.218 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:15.214+0000 7f9818595740 1 --2- 172.21.15.46:0/2107443316 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x561429694db0 0x5614296928f0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:35:15.218 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:15.214+0000 7f9818595740 1 --2- 172.21.15.46:0/2107443316 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56142968bfe0 0x5614296923b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:35:15.218 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:15.214+0000 7f9818595740 1 --2- 172.21.15.46:0/2107443316 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56142968b6c0 0x561429691e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:35:15.218 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:15.214+0000 7f97eeffd640 0 client.27854 ms_handle_reset on v2:172.21.15.142:6838/2519143918 2024-03-31T23:35:15.218 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:15.214+0000 7f9818595740 1 -- 172.21.15.46:0/2107443316 >> 172.21.15.46:0/2107443316 conn(0x561429685fe0 msgr2=0x561429687d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:35:15.219 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:15.214+0000 7f9818595740 1 -- 172.21.15.46:0/2107443316 shutdown_connections 2024-03-31T23:35:15.219 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:15.214+0000 7f9818595740 1 -- 172.21.15.46:0/2107443316 wait complete. 2024-03-31T23:35:15.219 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:15.214+0000 7f9818595740 20 client.27854 trim_cache size 0 max 16384 2024-03-31T23:35:15.237 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:15.236+0000 7f9818595740 1 Processor -- start 2024-03-31T23:35:15.237 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:15.236+0000 7f9818595740 1 -- start start 2024-03-31T23:35:15.237 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:15.236+0000 7f9818595740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56142968b6c0 0x5614296945a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:35:15.238 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:15.236+0000 7f9818595740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56142968bfe0 0x5614296a0780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:35:15.238 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:15.236+0000 7f9818595740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561429694db0 0x56142968c670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:35:15.238 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:15.236+0000 7f9818595740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f98101b3630 con 0x561429694db0 2024-03-31T23:35:15.238 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:15.236+0000 7f9818595740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f98101b0f00 con 0x56142968b6c0 2024-03-31T23:35:15.238 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:15.236+0000 7f9818595740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f98101aed70 con 0x56142968bfe0 2024-03-31T23:35:15.238 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:15.236+0000 7f980f7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561429694db0 0x56142968c670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:35:15.238 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:15.237+0000 7f980e7fc640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56142968bfe0 0x5614296a0780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:35:15.238 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:15.237+0000 7f980effd640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56142968b6c0 0x5614296945a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:35:15.238 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:15.237+0000 7f980e7fc640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56142968bfe0 0x5614296a0780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.46:54508/0 (socket says 172.21.15.46:54508) 2024-03-31T23:35:15.238 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:15.237+0000 7f980e7fc640 1 -- 172.21.15.46:0/2181677211 learned_addr learned my addr 172.21.15.46:0/2181677211 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:35:15.238 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:15.237+0000 7f980effd640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56142968b6c0 0x5614296945a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.46:46958/0 (socket says 172.21.15.46:46958) 2024-03-31T23:35:15.241 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:15.237+0000 7f980e7fc640 1 -- 172.21.15.46:0/2181677211 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56142968b6c0 msgr2=0x5614296945a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:35:15.241 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:15.237+0000 7f980e7fc640 1 --2- 172.21.15.46:0/2181677211 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56142968b6c0 0x5614296945a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:35:15.241 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:15.237+0000 7f980e7fc640 1 -- 172.21.15.46:0/2181677211 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561429694db0 msgr2=0x56142968c670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:35:15.241 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:15.237+0000 7f980e7fc640 1 --2- 172.21.15.46:0/2181677211 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561429694db0 0x56142968c670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:35:15.241 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:15.237+0000 7f980e7fc640 1 -- 172.21.15.46:0/2181677211 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5614296882d0 con 0x56142968bfe0 2024-03-31T23:35:15.241 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:15.237+0000 7f980f7fe640 1 --2- 172.21.15.46:0/2181677211 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561429694db0 0x56142968c670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:35:15.241 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:15.237+0000 7f980e7fc640 1 --2- 172.21.15.46:0/2181677211 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56142968bfe0 0x5614296a0780 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f98040bd0e0 tx=0x7f98040f4820 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1f26c13ad6156da9 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:35:15.241 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:15.237+0000 7f97f57fa640 1 -- 172.21.15.46:0/2181677211 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98040b93f0 con 0x56142968bfe0 2024-03-31T23:35:15.241 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:15.238+0000 7f97f57fa640 1 -- 172.21.15.46:0/2181677211 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f98040f2040 con 0x56142968bfe0 2024-03-31T23:35:15.242 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:15.238+0000 7f97f57fa640 1 -- 172.21.15.46:0/2181677211 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98040bcab0 con 0x56142968bfe0 2024-03-31T23:35:15.242 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:15.238+0000 7f9818595740 1 -- 172.21.15.46:0/2181677211 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56142968bfe0 msgr2=0x5614296a0780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:35:15.242 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:15.238+0000 7f9818595740 1 --2- 172.21.15.46:0/2181677211 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56142968bfe0 0x5614296a0780 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f98040bd0e0 tx=0x7f98040f4820 comp rx=0 tx=0).stop 2024-03-31T23:35:15.242 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:15.238+0000 7f9818595740 1 -- 172.21.15.46:0/2181677211 shutdown_connections 2024-03-31T23:35:15.242 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:15.238+0000 7f9818595740 1 --2- 172.21.15.46:0/2181677211 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561429694db0 0x56142968c670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:35:15.242 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:15.238+0000 7f9818595740 1 --2- 172.21.15.46:0/2181677211 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56142968bfe0 0x5614296a0780 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:35:15.242 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:15.238+0000 7f9818595740 1 --2- 172.21.15.46:0/2181677211 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56142968b6c0 0x5614296945a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:35:15.242 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:15.238+0000 7f9818595740 1 -- 172.21.15.46:0/2181677211 >> 172.21.15.46:0/2181677211 conn(0x561429685fe0 msgr2=0x5614296863c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:35:15.242 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:15.239+0000 7f9818595740 1 -- 172.21.15.46:0/2181677211 shutdown_connections 2024-03-31T23:35:15.242 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:15.239+0000 7f9818595740 1 -- 172.21.15.46:0/2181677211 wait complete. 2024-03-31T23:35:15.242 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:15.239+0000 7f9818595740 1 -- start start 2024-03-31T23:35:15.242 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:15.239+0000 7f9818595740 1 Processor -- start 2024-03-31T23:35:15.242 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:15.240+0000 7f9818595740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56142968b6c0 0x56142968f5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:35:15.243 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:15.240+0000 7f9818595740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56142968bfe0 0x56142968fb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:35:15.243 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:15.240+0000 7f9818595740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561429694db0 0x56142968e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:35:15.243 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:15.240+0000 7f9818595740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f98101af3c0 con 0x561429694db0 2024-03-31T23:35:15.243 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:15.240+0000 7f9818595740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f98101b0830 con 0x56142968b6c0 2024-03-31T23:35:15.243 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:15.240+0000 7f9818595740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f98101afa90 con 0x56142968bfe0 2024-03-31T23:35:15.243 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:15.240+0000 7f9818595740 10 client.0 _ll_register_callbacks cb 0x7f980800ed80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T23:35:15.243 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:15.240+0000 7f980effd640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56142968b6c0 0x56142968f5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:35:15.243 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:15.240+0000 7f980e7fc640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56142968bfe0 0x56142968fb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:35:15.243 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:15.240+0000 7f980effd640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56142968b6c0 0x56142968f5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.46:46968/0 (socket says 172.21.15.46:46968) 2024-03-31T23:35:15.243 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:15.240+0000 7f980f7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561429694db0 0x56142968e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:35:15.243 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:15.240+0000 7f980effd640 1 -- 172.21.15.46:0/1308111019 learned_addr learned my addr 172.21.15.46:0/1308111019 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:35:15.243 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:15.240+0000 7f980e7fc640 1 -- 172.21.15.46:0/1308111019 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56142968b6c0 msgr2=0x56142968f5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:35:15.243 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:15.240+0000 7f980e7fc640 1 --2- 172.21.15.46:0/1308111019 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56142968b6c0 0x56142968f5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:35:15.244 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:15.240+0000 7f980e7fc640 1 -- 172.21.15.46:0/1308111019 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561429694db0 msgr2=0x56142968e0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:35:15.244 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:15.240+0000 7f980e7fc640 1 --2- 172.21.15.46:0/1308111019 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561429694db0 0x56142968e0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:35:15.244 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:15.240+0000 7f980e7fc640 1 -- 172.21.15.46:0/1308111019 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56142968e620 con 0x56142968bfe0 2024-03-31T23:35:15.244 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:15.240+0000 7f980f7fe640 1 --2- 172.21.15.46:0/1308111019 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561429694db0 0x56142968e0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:35:15.245 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:15.240+0000 7f980e7fc640 1 --2- 172.21.15.46:0/1308111019 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56142968bfe0 0x56142968fb10 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f98040bd310 tx=0x7f98040ee0c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a70ac6cbccda51db server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:35:15.245 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:15.241+0000 7f97eeffd640 10 client.0 ms_handle_connect on v2:172.21.15.162:3300/0 2024-03-31T23:35:15.246 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:15.241+0000 7f97eeffd640 1 -- 172.21.15.46:0/1308111019 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98040bcc90 con 0x56142968bfe0 2024-03-31T23:35:15.246 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:15.241+0000 7f97eeffd640 1 -- 172.21.15.46:0/1308111019 <== mon.1 v2:172.21.15.162:3300/0 2 ==== osd_map(135..135 src has 1..135) ==== 9554+0+0 (secure 0 0 0) 0x7f98040c58d0 con 0x56142968bfe0 2024-03-31T23:35:15.246 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:15.241+0000 7f9818595740 1 -- 172.21.15.46:0/1308111019 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f98040f2040 con 0x56142968bfe0 2024-03-31T23:35:15.246 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:15.241+0000 7f97eeffd640 1 -- 172.21.15.46:0/1308111019 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_get_version_reply(handle=1 version=54) ==== 24+0+0 (secure 0 0 0) 0x7f98040bcf20 con 0x56142968bfe0 2024-03-31T23:35:15.246 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:15.241+0000 7f9818595740 10 client.27766 fetch_fsmap learned FSMap version 54 2024-03-31T23:35:15.246 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:15.241+0000 7f9818595740 1 -- 172.21.15.46:0/1308111019 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,fsmap=54,monmap=4+}) -- 0x56142970ad30 con 0x56142968bfe0 2024-03-31T23:35:15.246 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:15.241+0000 7f97eeffd640 1 -- 172.21.15.46:0/1308111019 <== mon.1 v2:172.21.15.162:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f98040d8020 con 0x56142968bfe0 2024-03-31T23:35:15.246 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:15.242+0000 7f97eeffd640 1 -- 172.21.15.46:0/1308111019 <== mon.1 v2:172.21.15.162:3300/0 5 ==== fsmap(e 54) ==== 8346+0+0 (secure 0 0 0) 0x7f98040dbb30 con 0x56142968bfe0 2024-03-31T23:35:15.246 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:15.242+0000 7f9818595740 10 client.27766 fetch_fsmap finished waiting for FSMap version 54 2024-03-31T23:35:15.246 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:15.242+0000 7f9818595740 10 client.27766 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-31T23:35:15.246 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:15.242+0000 7f9818595740 20 client.27766 populate_metadata read hostname 'smithi046' 2024-03-31T23:35:15.246 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:15.242+0000 7f9818595740 1 --2- 172.21.15.46:0/1308111019 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x56142970b3e0 0x56142970d800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:35:15.247 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:15.242+0000 7f9818595740 4 client.27766 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-31T23:35:15.247 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:15.242+0000 7f9818595740 1 -- 172.21.15.46:0/1308111019 --> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5614295fbe30 con 0x56142970b3e0 2024-03-31T23:35:15.247 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:15.242+0000 7f980effd640 1 --2- 172.21.15.46:0/1308111019 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x56142970b3e0 0x56142970d800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:35:15.247 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:15.243+0000 7f980effd640 1 --2- 172.21.15.46:0/1308111019 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x56142970b3e0 0x56142970d800 crc :-1 s=READY pgs=287 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=43aee2f30eb13f44 server_cookie=caa5512fac582695 in_seq=0 out_seq=0 2024-03-31T23:35:15.247 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:15.243+0000 7f97eeffd640 10 client.27766 ms_handle_connect on v2:172.21.15.142:6838/2519143918 2024-03-31T23:35:15.315 INFO:tasks.workunit.client.0.smithi046.stdout: 10 95654 8.18 MB/sec execute 500 sec latency 96.138 ms 2024-03-31T23:35:15.467 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:15.466+0000 7f97eeffd640 1 -- 172.21.15.46:0/1308111019 <== mds.0 v2:172.21.15.142:6838/2519143918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5614295fbe30 con 0x56142970b3e0 2024-03-31T23:35:15.467 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:15.466+0000 7f97eeffd640 10 client.27766 handle_command_reply: tid=0 2024-03-31T23:35:15.467 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:15.466+0000 7f9818595740 1 client.27766 shutdown 2024-03-31T23:35:15.468 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:15.467+0000 7f9818595740 1 -- 172.21.15.46:0/1308111019 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56142968bfe0 msgr2=0x56142968fb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:35:15.468 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:15.468+0000 7f9818595740 1 --2- 172.21.15.46:0/1308111019 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56142968bfe0 0x56142968fb10 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f98040bd310 tx=0x7f98040ee0c0 comp rx=0 tx=0).stop 2024-03-31T23:35:15.470 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:15.469+0000 7f9818595740 1 -- 172.21.15.46:0/1308111019 shutdown_connections 2024-03-31T23:35:15.470 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:15.469+0000 7f9818595740 1 --2- 172.21.15.46:0/1308111019 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x56142970b3e0 0x56142970d800 crc :-1 s=READY pgs=287 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:35:15.470 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:15.469+0000 7f9818595740 1 --2- 172.21.15.46:0/1308111019 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561429694db0 0x56142968e0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:35:15.471 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:15.469+0000 7f97eeffd640 0 client.27766 ms_handle_reset on v2:172.21.15.142:6838/2519143918 2024-03-31T23:35:15.471 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:15.469+0000 7f9818595740 1 --2- 172.21.15.46:0/1308111019 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56142968bfe0 0x56142968fb10 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:35:15.471 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:15.469+0000 7f9818595740 1 --2- 172.21.15.46:0/1308111019 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56142968b6c0 0x56142968f5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:35:15.471 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:15.469+0000 7f9818595740 1 -- 172.21.15.46:0/1308111019 >> 172.21.15.46:0/1308111019 conn(0x561429685fe0 msgr2=0x56142968c670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:35:15.472 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:15.471+0000 7f9818595740 1 -- 172.21.15.46:0/1308111019 shutdown_connections 2024-03-31T23:35:15.472 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:15.471+0000 7f9818595740 1 -- 172.21.15.46:0/1308111019 wait complete. 2024-03-31T23:35:15.472 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:15.471+0000 7f9818595740 20 client.27766 trim_cache size 0 max 16384 2024-03-31T23:35:15.473 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout:{ 2024-03-31T23:35:15.473 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout: "message": "", 2024-03-31T23:35:15.473 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout: "return_code": 0 2024-03-31T23:35:15.473 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout:} 2024-03-31T23:35:15.474 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:15.473+0000 7f98159bf640 1 -- 172.21.15.46:0/1763003844 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f97f80a2850 msgr2=0x7f97f80a4d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:35:15.474 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:15.473+0000 7f98159bf640 1 --2- 172.21.15.46:0/1763003844 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f97f80a2850 0x7f97f80a4d10 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f9808009fd0 tx=0x7f9808009bc0 comp rx=0 tx=0).stop 2024-03-31T23:35:15.474 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:15.473+0000 7f98159bf640 1 -- 172.21.15.46:0/1763003844 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f98101020b0 msgr2=0x7f9810073f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:35:15.474 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:15.473+0000 7f98159bf640 1 --2- 172.21.15.46:0/1763003844 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f98101020b0 0x7f9810073f90 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f9804009230 tx=0x7f98040045e0 comp rx=0 tx=0).stop 2024-03-31T23:35:15.475 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:15.474+0000 7f98159bf640 1 -- 172.21.15.46:0/1763003844 shutdown_connections 2024-03-31T23:35:15.476 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:15.474+0000 7f98159bf640 1 --2- 172.21.15.46:0/1763003844 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f97f80a2850 0x7f97f80a4d10 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:35:15.476 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:15.474+0000 7f98159bf640 1 --2- 172.21.15.46:0/1763003844 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9810107700 0x7f98100744d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:35:15.476 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:15.474+0000 7f98159bf640 1 --2- 172.21.15.46:0/1763003844 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f98101020b0 0x7f9810073f90 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:35:15.476 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:15.474+0000 7f98159bf640 1 --2- 172.21.15.46:0/1763003844 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f98101016f0 0x7f9810073a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:35:15.476 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:15.474+0000 7f98159bf640 1 -- 172.21.15.46:0/1763003844 >> 172.21.15.46:0/1763003844 conn(0x7f98100fd1e0 msgr2=0x7f98100fd8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:35:15.476 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:15.474+0000 7f98159bf640 1 -- 172.21.15.46:0/1763003844 shutdown_connections 2024-03-31T23:35:15.476 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:15.474+0000 7f98159bf640 1 -- 172.21.15.46:0/1763003844 wait complete. 2024-03-31T23:35:16.315 INFO:tasks.workunit.client.0.smithi046.stdout: 10 95802 8.18 MB/sec execute 501 sec latency 198.475 ms 2024-03-31T23:35:17.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:35:16 smithi162 ceph-mon[36237]: pgmap v748: 129 pgs: 129 active+clean; 1.3 GiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 6.1 MiB/s wr, 268 op/s 2024-03-31T23:35:17.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:35:16 smithi046 ceph-mon[32498]: pgmap v748: 129 pgs: 129 active+clean; 1.3 GiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 6.1 MiB/s wr, 268 op/s 2024-03-31T23:35:17.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:35:16 smithi142 ceph-mon[35271]: pgmap v748: 129 pgs: 129 active+clean; 1.3 GiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 6.1 MiB/s wr, 268 op/s 2024-03-31T23:35:17.315 INFO:tasks.workunit.client.0.smithi046.stdout: 10 95962 8.19 MB/sec execute 502 sec latency 97.133 ms 2024-03-31T23:35:18.315 INFO:tasks.workunit.client.0.smithi046.stdout: 10 96107 8.19 MB/sec execute 503 sec latency 99.665 ms 2024-03-31T23:35:19.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:35:18 smithi162 ceph-mon[36237]: pgmap v749: 129 pgs: 129 active+clean; 1.3 GiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 5.4 MiB/s wr, 236 op/s 2024-03-31T23:35:19.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:35:18 smithi046 ceph-mon[32498]: pgmap v749: 129 pgs: 129 active+clean; 1.3 GiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 5.4 MiB/s wr, 236 op/s 2024-03-31T23:35:19.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:35:18 smithi142 ceph-mon[35271]: pgmap v749: 129 pgs: 129 active+clean; 1.3 GiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 5.4 MiB/s wr, 236 op/s 2024-03-31T23:35:19.315 INFO:tasks.workunit.client.0.smithi046.stdout: 10 96277 8.20 MB/sec execute 504 sec latency 122.356 ms 2024-03-31T23:35:20.315 INFO:tasks.workunit.client.0.smithi046.stdout: 10 96422 8.19 MB/sec execute 505 sec latency 92.117 ms 2024-03-31T23:35:21.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:35:20 smithi162 ceph-mon[36237]: pgmap v750: 129 pgs: 129 active+clean; 1.3 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 7.6 MiB/s wr, 391 op/s 2024-03-31T23:35:21.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:35:20 smithi046 ceph-mon[32498]: pgmap v750: 129 pgs: 129 active+clean; 1.3 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 7.6 MiB/s wr, 391 op/s 2024-03-31T23:35:21.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:35:20 smithi142 ceph-mon[35271]: pgmap v750: 129 pgs: 129 active+clean; 1.3 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 7.6 MiB/s wr, 391 op/s 2024-03-31T23:35:21.315 INFO:tasks.workunit.client.0.smithi046.stdout: 10 96557 8.18 MB/sec execute 506 sec latency 82.319 ms 2024-03-31T23:35:22.315 INFO:tasks.workunit.client.0.smithi046.stdout: 10 96746 8.19 MB/sec execute 507 sec latency 83.549 ms 2024-03-31T23:35:22.995 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:35:22 smithi142 ceph-mon[35271]: pgmap v751: 129 pgs: 129 active+clean; 1.3 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 5.0 MiB/s wr, 277 op/s 2024-03-31T23:35:23.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:35:22 smithi162 ceph-mon[36237]: pgmap v751: 129 pgs: 129 active+clean; 1.3 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 5.0 MiB/s wr, 277 op/s 2024-03-31T23:35:23.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:35:22 smithi046 ceph-mon[32498]: pgmap v751: 129 pgs: 129 active+clean; 1.3 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 5.0 MiB/s wr, 277 op/s 2024-03-31T23:35:23.315 INFO:tasks.workunit.client.0.smithi046.stdout: 10 96883 8.17 MB/sec execute 508 sec latency 95.790 ms 2024-03-31T23:35:24.315 INFO:tasks.workunit.client.0.smithi046.stdout: 10 97033 8.17 MB/sec execute 509 sec latency 87.103 ms 2024-03-31T23:35:25.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:35:24 smithi162 ceph-mon[36237]: pgmap v752: 129 pgs: 129 active+clean; 1.3 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 5.0 MiB/s wr, 277 op/s 2024-03-31T23:35:25.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:35:24 smithi046 ceph-mon[32498]: pgmap v752: 129 pgs: 129 active+clean; 1.3 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 5.0 MiB/s wr, 277 op/s 2024-03-31T23:35:25.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:35:24 smithi142 ceph-mon[35271]: pgmap v752: 129 pgs: 129 active+clean; 1.3 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 5.0 MiB/s wr, 277 op/s 2024-03-31T23:35:25.315 INFO:tasks.workunit.client.0.smithi046.stdout: 10 97199 8.18 MB/sec execute 510 sec latency 64.384 ms 2024-03-31T23:35:26.315 INFO:tasks.workunit.client.0.smithi046.stdout: 10 97375 8.19 MB/sec execute 511 sec latency 72.666 ms 2024-03-31T23:35:27.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:35:26 smithi162 ceph-mon[36237]: pgmap v753: 129 pgs: 129 active+clean; 1.3 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 6.9 MiB/s wr, 378 op/s 2024-03-31T23:35:27.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:35:26 smithi046 ceph-mon[32498]: pgmap v753: 129 pgs: 129 active+clean; 1.3 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 6.9 MiB/s wr, 378 op/s 2024-03-31T23:35:27.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:35:26 smithi142 ceph-mon[35271]: pgmap v753: 129 pgs: 129 active+clean; 1.3 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 6.9 MiB/s wr, 378 op/s 2024-03-31T23:35:27.315 INFO:tasks.workunit.client.0.smithi046.stdout: 10 97568 8.20 MB/sec execute 512 sec latency 65.848 ms 2024-03-31T23:35:28.315 INFO:tasks.workunit.client.0.smithi046.stdout: 10 97739 8.21 MB/sec execute 513 sec latency 125.807 ms 2024-03-31T23:35:28.616 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:28.615+0000 7f1e0a743640 1 -- 172.21.15.46:0/427474919 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1e0410dfa0 msgr2=0x7f1e04110390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:35:28.616 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:28.615+0000 7f1e0a743640 1 --2- 172.21.15.46:0/427474919 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1e0410dfa0 0x7f1e04110390 secure :-1 s=READY pgs=1031 cs=0 l=1 rev1=1 crypto rx=0x7f1dfc009ca0 tx=0x7f1dfc02c0f0 comp rx=0 tx=0).stop 2024-03-31T23:35:28.617 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:28.615+0000 7f1e0a743640 1 -- 172.21.15.46:0/427474919 shutdown_connections 2024-03-31T23:35:28.617 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:28.615+0000 7f1e0a743640 1 --2- 172.21.15.46:0/427474919 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1e0410dfa0 0x7f1e04110390 unknown :-1 s=CLOSED pgs=1031 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:35:28.617 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:28.615+0000 7f1e0a743640 1 --2- 172.21.15.46:0/427474919 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1e0406a430 0x7f1e0406a870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:35:28.617 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:28.615+0000 7f1e0a743640 1 --2- 172.21.15.46:0/427474919 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1e04102630 0x7f1e04069ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:35:28.617 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:28.615+0000 7f1e0a743640 1 -- 172.21.15.46:0/427474919 >> 172.21.15.46:0/427474919 conn(0x7f1e04077630 msgr2=0x7f1e04077a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:35:28.617 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:28.615+0000 7f1e0a743640 1 -- 172.21.15.46:0/427474919 shutdown_connections 2024-03-31T23:35:28.617 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:28.615+0000 7f1e0a743640 1 -- 172.21.15.46:0/427474919 wait complete. 2024-03-31T23:35:28.617 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:28.616+0000 7f1e0a743640 1 Processor -- start 2024-03-31T23:35:28.617 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:28.616+0000 7f1e0a743640 1 -- start start 2024-03-31T23:35:28.618 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:28.616+0000 7f1e0a743640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1e0406a430 0x7f1e0410d940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:35:28.619 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:28.616+0000 7f1e0a743640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1e04102630 0x7f1e0410aa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:35:28.619 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:28.616+0000 7f1e0a743640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1e0410dfa0 0x7f1e0410b000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:35:28.619 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:28.616+0000 7f1e0a743640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f1e04113ba0 con 0x7f1e0406a430 2024-03-31T23:35:28.619 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:28.616+0000 7f1e0a743640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f1e04113a20 con 0x7f1e04102630 2024-03-31T23:35:28.619 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:28.616+0000 7f1e0a743640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f1e04113d20 con 0x7f1e0410dfa0 2024-03-31T23:35:28.619 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:28.616+0000 7f1e03fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1e0406a430 0x7f1e0410d940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:35:28.619 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:28.616+0000 7f1e03fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1e0406a430 0x7f1e0410d940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:42170/0 (socket says 172.21.15.46:42170) 2024-03-31T23:35:28.619 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:28.616+0000 7f1e03fff640 1 -- 172.21.15.46:0/2412191340 learned_addr learned my addr 172.21.15.46:0/2412191340 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:35:28.619 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:28.616+0000 7f1e08ce0640 1 --2- 172.21.15.46:0/2412191340 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1e0410dfa0 0x7f1e0410b000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:35:28.619 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:28.616+0000 7f1e037fe640 1 --2- 172.21.15.46:0/2412191340 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1e04102630 0x7f1e0410aa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:35:28.619 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:28.617+0000 7f1e03fff640 1 -- 172.21.15.46:0/2412191340 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1e0410dfa0 msgr2=0x7f1e0410b000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:35:28.619 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:28.617+0000 7f1e03fff640 1 --2- 172.21.15.46:0/2412191340 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1e0410dfa0 0x7f1e0410b000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:35:28.619 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:28.617+0000 7f1e03fff640 1 -- 172.21.15.46:0/2412191340 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1e04102630 msgr2=0x7f1e0410aa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:35:28.620 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:28.617+0000 7f1e03fff640 1 --2- 172.21.15.46:0/2412191340 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1e04102630 0x7f1e0410aa80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:35:28.620 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:28.617+0000 7f1e03fff640 1 -- 172.21.15.46:0/2412191340 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1e0410b8c0 con 0x7f1e0406a430 2024-03-31T23:35:28.620 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:28.617+0000 7f1e03fff640 1 --2- 172.21.15.46:0/2412191340 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1e0406a430 0x7f1e0410d940 secure :-1 s=READY pgs=1032 cs=0 l=1 rev1=1 crypto rx=0x7f1df400d950 tx=0x7f1df400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:35:28.620 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:28.617+0000 7f1e017fa640 1 -- 172.21.15.46:0/2412191340 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1df4017030 con 0x7f1e0406a430 2024-03-31T23:35:28.620 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:28.617+0000 7f1e017fa640 1 -- 172.21.15.46:0/2412191340 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1df4004020 con 0x7f1e0406a430 2024-03-31T23:35:28.620 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:28.617+0000 7f1e017fa640 1 -- 172.21.15.46:0/2412191340 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1df40047a0 con 0x7f1e0406a430 2024-03-31T23:35:28.620 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:28.617+0000 7f1e0a743640 1 -- 172.21.15.46:0/2412191340 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1e041aa9e0 con 0x7f1e0406a430 2024-03-31T23:35:28.620 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:28.617+0000 7f1e0a743640 1 -- 172.21.15.46:0/2412191340 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1e041aaee0 con 0x7f1e0406a430 2024-03-31T23:35:28.622 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:28.620+0000 7f1e0d3f3740 1 Processor -- start 2024-03-31T23:35:28.622 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:28.620+0000 7f1e017fa640 1 -- 172.21.15.46:0/2412191340 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 25) ==== 137451+0+0 (secure 0 0 0) 0x7f1df40041d0 con 0x7f1e0406a430 2024-03-31T23:35:28.622 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:28.620+0000 7f1e0d3f3740 1 -- start start 2024-03-31T23:35:28.622 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:28.621+0000 7f1e0d3f3740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557071b8e6c0 0x557071b8eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:35:28.622 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:28.621+0000 7f1e0d3f3740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557071b8efe0 0x557071b97870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:35:28.623 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:28.621+0000 7f1e0d3f3740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557071b97db0 0x557071b9a1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:35:28.623 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:28.621+0000 7f1e0d3f3740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x557071af18f0 con 0x557071b8efe0 2024-03-31T23:35:28.623 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:28.621+0000 7f1e0d3f3740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x557071a9a8d0 con 0x557071b97db0 2024-03-31T23:35:28.623 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:28.621+0000 7f1e0d3f3740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x557071aed990 con 0x557071b8e6c0 2024-03-31T23:35:28.623 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:28.621+0000 7f1e08ce0640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557071b8e6c0 0x557071b8eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:35:28.623 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:28.621+0000 7f1e017fa640 1 --2- 172.21.15.46:0/2412191340 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f1dec0a26b0 0x7f1dec0a4b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:35:28.623 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:28.621+0000 7f1e08ce0640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557071b8e6c0 0x557071b8eaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.46:48480/0 (socket says 172.21.15.46:48480) 2024-03-31T23:35:28.623 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:28.621+0000 7f1e08ce0640 1 -- 172.21.15.46:0/1784310937 learned_addr learned my addr 172.21.15.46:0/1784310937 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:35:28.623 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:28.621+0000 7f1e03fff640 1 --2- 172.21.15.46:0/1784310937 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557071b8efe0 0x557071b97870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:35:28.623 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:28.621+0000 7f1e037fe640 1 --2- 172.21.15.46:0/1784310937 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557071b97db0 0x557071b9a1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:35:28.623 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:28.621+0000 7f1e08ce0640 1 --2- 172.21.15.46:0/2412191340 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f1dec0a26b0 0x7f1dec0a4b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:35:28.623 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:28.621+0000 7f1e017fa640 1 -- 172.21.15.46:0/2412191340 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(135..135 src has 1..135) ==== 9554+0+0 (secure 0 0 0) 0x7f1df40ce880 con 0x7f1e0406a430 2024-03-31T23:35:28.623 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:28.621+0000 7f1e08ce0640 1 -- 172.21.15.46:0/1784310937 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557071b97db0 msgr2=0x557071b9a1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:35:28.623 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:28.621+0000 7f1e08ce0640 1 --2- 172.21.15.46:0/1784310937 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557071b97db0 0x557071b9a1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:35:28.623 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:28.621+0000 7f1e08ce0640 1 -- 172.21.15.46:0/1784310937 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557071b8efe0 msgr2=0x557071b97870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:35:28.624 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:28.621+0000 7f1e08ce0640 1 --2- 172.21.15.46:0/1784310937 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557071b8efe0 0x557071b97870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:35:28.624 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:28.621+0000 7f1e08ce0640 1 -- 172.21.15.46:0/1784310937 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557071806210 con 0x557071b8e6c0 2024-03-31T23:35:28.624 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:28.621+0000 7f1e03fff640 1 --2- 172.21.15.46:0/1784310937 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557071b8efe0 0x557071b97870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:35:28.627 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:28.622+0000 7f1e08ce0640 1 --2- 172.21.15.46:0/1784310937 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557071b8e6c0 0x557071b8eaa0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f1dfc004af0 tx=0x7f1dfc0327c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=753588cb467b6dcc server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:35:28.627 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:28.622+0000 7f1e08ce0640 1 --2- 172.21.15.46:0/2412191340 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f1dec0a26b0 0x7f1dec0a4b70 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f1e0406aed0 tx=0x7f1dfc02e000 comp rx=0 tx=0).ready entity=mgr.15654 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:35:28.628 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:28.622+0000 7f1dea7fc640 1 -- 172.21.15.46:0/1784310937 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1dfc051070 con 0x557071b8e6c0 2024-03-31T23:35:28.628 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:28.622+0000 7f1dea7fc640 1 -- 172.21.15.46:0/1784310937 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1dfc004d40 con 0x557071b8e6c0 2024-03-31T23:35:28.628 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:28.622+0000 7f1dea7fc640 1 -- 172.21.15.46:0/1784310937 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1dfc039040 con 0x557071b8e6c0 2024-03-31T23:35:28.628 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:28.623+0000 7f1e0d3f3740 1 -- 172.21.15.46:0/1784310937 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557071b8e6c0 msgr2=0x557071b8eaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:35:28.628 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:28.623+0000 7f1e0d3f3740 1 --2- 172.21.15.46:0/1784310937 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557071b8e6c0 0x557071b8eaa0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f1dfc004af0 tx=0x7f1dfc0327c0 comp rx=0 tx=0).stop 2024-03-31T23:35:28.628 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:28.623+0000 7f1e0d3f3740 1 -- 172.21.15.46:0/1784310937 shutdown_connections 2024-03-31T23:35:28.628 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:28.623+0000 7f1e0d3f3740 1 --2- 172.21.15.46:0/1784310937 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557071b97db0 0x557071b9a1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:35:28.628 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:28.623+0000 7f1e0d3f3740 1 --2- 172.21.15.46:0/1784310937 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557071b8efe0 0x557071b97870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:35:28.628 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:28.623+0000 7f1e0d3f3740 1 --2- 172.21.15.46:0/1784310937 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557071b8e6c0 0x557071b8eaa0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:35:28.628 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:28.623+0000 7f1e0d3f3740 1 -- 172.21.15.46:0/1784310937 >> 172.21.15.46:0/1784310937 conn(0x557071b88fe0 msgr2=0x557071b893c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:35:28.628 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:28.623+0000 7f1e0d3f3740 1 -- 172.21.15.46:0/1784310937 shutdown_connections 2024-03-31T23:35:28.628 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:28.623+0000 7f1e0d3f3740 1 -- 172.21.15.46:0/1784310937 wait complete. 2024-03-31T23:35:28.628 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:28.624+0000 7f1e0d3f3740 1 -- start start 2024-03-31T23:35:28.628 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:28.624+0000 7f1e0d3f3740 1 Processor -- start 2024-03-31T23:35:28.629 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:28.624+0000 7f1e0d3f3740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557071b8e6c0 0x557071b94ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:35:28.629 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:28.624+0000 7f1e0d3f3740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557071b8efe0 0x557071b95410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:35:28.629 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:28.624+0000 7f1e0d3f3740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557071b97db0 0x557071b95950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:35:28.629 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:28.624+0000 7f1e0d3f3740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x557071b9a980 con 0x557071b8e6c0 2024-03-31T23:35:28.629 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:28.624+0000 7f1e0d3f3740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x557071a8ff80 con 0x557071b8efe0 2024-03-31T23:35:28.629 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:28.624+0000 7f1e0d3f3740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x557071b9ab00 con 0x557071b97db0 2024-03-31T23:35:28.629 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:28.624+0000 7f1e0d3f3740 10 client.0 _ll_register_callbacks cb 0x557071ae8290 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T23:35:28.629 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:28.624+0000 7f1e03fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557071b8e6c0 0x557071b94ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:35:28.629 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:28.624+0000 7f1e03fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557071b8e6c0 0x557071b94ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:42194/0 (socket says 172.21.15.46:42194) 2024-03-31T23:35:28.629 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:28.624+0000 7f1e03fff640 1 -- 172.21.15.46:0/3489956100 learned_addr learned my addr 172.21.15.46:0/3489956100 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:35:28.629 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:28.625+0000 7f1e037fe640 1 --2- 172.21.15.46:0/3489956100 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557071b8efe0 0x557071b95410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:35:28.629 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:28.625+0000 7f1e08ce0640 1 --2- 172.21.15.46:0/3489956100 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557071b97db0 0x557071b95950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:35:28.629 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:28.625+0000 7f1e03fff640 1 -- 172.21.15.46:0/3489956100 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557071b97db0 msgr2=0x557071b95950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:35:28.629 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:28.625+0000 7f1e03fff640 1 --2- 172.21.15.46:0/3489956100 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557071b97db0 0x557071b95950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:35:28.629 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:28.625+0000 7f1e03fff640 1 -- 172.21.15.46:0/3489956100 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557071b8efe0 msgr2=0x557071b95410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:35:28.630 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:28.625+0000 7f1e03fff640 1 --2- 172.21.15.46:0/3489956100 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557071b8efe0 0x557071b95410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:35:28.630 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:28.625+0000 7f1e03fff640 1 -- 172.21.15.46:0/3489956100 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5570718d2620 con 0x557071b8e6c0 2024-03-31T23:35:28.630 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:28.625+0000 7f1e03fff640 1 --2- 172.21.15.46:0/3489956100 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557071b8e6c0 0x557071b94ed0 secure :-1 s=READY pgs=1033 cs=0 l=1 rev1=1 crypto rx=0x7f1df40975c0 tx=0x7f1df4084130 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d9e97312f3616c81 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:35:28.630 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:28.625+0000 7f1de37fe640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-31T23:35:28.630 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:28.625+0000 7f1de37fe640 1 -- 172.21.15.46:0/3489956100 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1df40b3060 con 0x557071b8e6c0 2024-03-31T23:35:28.630 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:28.625+0000 7f1e0d3f3740 1 -- 172.21.15.46:0/3489956100 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1dfc004d40 con 0x557071b8e6c0 2024-03-31T23:35:28.630 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:28.625+0000 7f1de37fe640 1 -- 172.21.15.46:0/3489956100 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(135..135 src has 1..135) ==== 9554+0+0 (secure 0 0 0) 0x7f1df4085390 con 0x557071b8e6c0 2024-03-31T23:35:28.630 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:28.625+0000 7f1de37fe640 1 -- 172.21.15.46:0/3489956100 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=54) ==== 24+0+0 (secure 0 0 0) 0x7f1df409ed40 con 0x557071b8e6c0 2024-03-31T23:35:28.630 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:28.625+0000 7f1e0d3f3740 10 client.18066 fetch_fsmap learned FSMap version 54 2024-03-31T23:35:28.630 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:28.625+0000 7f1e0d3f3740 1 -- 172.21.15.46:0/3489956100 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=54,monmap=4+}) -- 0x557071ba6990 con 0x557071b8e6c0 2024-03-31T23:35:28.630 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:28.626+0000 7f1de37fe640 1 -- 172.21.15.46:0/3489956100 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1dfc004d40 con 0x557071b8e6c0 2024-03-31T23:35:28.630 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:28.626+0000 7f1de37fe640 1 -- 172.21.15.46:0/3489956100 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 54) ==== 8346+0+0 (secure 0 0 0) 0x7f1df40ce450 con 0x557071b8e6c0 2024-03-31T23:35:28.630 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:28.626+0000 7f1e0d3f3740 10 client.18066 fetch_fsmap finished waiting for FSMap version 54 2024-03-31T23:35:28.630 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:28.626+0000 7f1e0d3f3740 10 client.18066 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-31T23:35:28.631 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:28.626+0000 7f1e0d3f3740 20 client.18066 populate_metadata read hostname 'smithi046' 2024-03-31T23:35:28.631 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:28.626+0000 7f1e0d3f3740 1 --2- 172.21.15.46:0/3489956100 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x557071ba7e00 0x557071baa1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:35:28.631 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:28.626+0000 7f1e0d3f3740 4 client.18066 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-31T23:35:28.631 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:28.626+0000 7f1e0d3f3740 1 -- 172.21.15.46:0/3489956100 --> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557071b05120 con 0x557071ba7e00 2024-03-31T23:35:28.631 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:28.626+0000 7f1e037fe640 1 --2- 172.21.15.46:0/3489956100 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x557071ba7e00 0x557071baa1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:35:28.631 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:28.627+0000 7f1de37fe640 10 client.18066 ms_handle_connect on v2:172.21.15.142:6838/2519143918 2024-03-31T23:35:28.631 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:28.627+0000 7f1e037fe640 1 --2- 172.21.15.46:0/3489956100 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x557071ba7e00 0x557071baa1f0 crc :-1 s=READY pgs=288 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ff159930c9a632f4 server_cookie=fe02cd155bc51f67 in_seq=0 out_seq=0 2024-03-31T23:35:28.632 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:28.628+0000 7f1de37fe640 1 -- 172.21.15.46:0/3489956100 <== mds.0 v2:172.21.15.142:6838/2519143918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x557071b05120 con 0x557071ba7e00 2024-03-31T23:35:28.632 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:28.628+0000 7f1de37fe640 10 client.18066 handle_command_reply: tid=0 2024-03-31T23:35:28.632 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:28.628+0000 7f1e0d3f3740 1 client.18066 shutdown 2024-03-31T23:35:28.632 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:28.628+0000 7f1e0d3f3740 1 -- 172.21.15.46:0/3489956100 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557071b8e6c0 msgr2=0x557071b94ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:35:28.632 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:28.628+0000 7f1e0d3f3740 1 --2- 172.21.15.46:0/3489956100 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557071b8e6c0 0x557071b94ed0 secure :-1 s=READY pgs=1033 cs=0 l=1 rev1=1 crypto rx=0x7f1df40975c0 tx=0x7f1df4084130 comp rx=0 tx=0).stop 2024-03-31T23:35:28.632 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:28.628+0000 7f1e0d3f3740 1 -- 172.21.15.46:0/3489956100 shutdown_connections 2024-03-31T23:35:28.632 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:28.628+0000 7f1e0d3f3740 1 --2- 172.21.15.46:0/3489956100 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x557071ba7e00 0x557071baa1f0 crc :-1 s=READY pgs=288 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:35:28.632 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:28.628+0000 7f1de37fe640 0 client.18066 ms_handle_reset on v2:172.21.15.142:6838/2519143918 2024-03-31T23:35:28.632 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:28.628+0000 7f1e0d3f3740 1 --2- 172.21.15.46:0/3489956100 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557071b97db0 0x557071b95950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:35:28.632 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:28.628+0000 7f1e0d3f3740 1 --2- 172.21.15.46:0/3489956100 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557071b8efe0 0x557071b95410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:35:28.633 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:28.628+0000 7f1e0d3f3740 1 --2- 172.21.15.46:0/3489956100 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557071b8e6c0 0x557071b94ed0 unknown :-1 s=CLOSED pgs=1033 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:35:28.633 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:28.628+0000 7f1e0d3f3740 1 -- 172.21.15.46:0/3489956100 >> 172.21.15.46:0/3489956100 conn(0x557071b88fe0 msgr2=0x557071b8ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:35:28.633 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:28.628+0000 7f1e0d3f3740 1 -- 172.21.15.46:0/3489956100 shutdown_connections 2024-03-31T23:35:28.633 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:28.628+0000 7f1e0d3f3740 1 -- 172.21.15.46:0/3489956100 wait complete. 2024-03-31T23:35:28.633 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:28.629+0000 7f1e0d3f3740 20 client.18066 trim_cache size 0 max 16384 2024-03-31T23:35:28.651 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:28.649+0000 7f1e0d3f3740 1 Processor -- start 2024-03-31T23:35:28.651 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:28.649+0000 7f1e0d3f3740 1 -- start start 2024-03-31T23:35:28.651 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:28.649+0000 7f1e0d3f3740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557071b8e6c0 0x557071b97600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:35:28.651 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:28.649+0000 7f1e0d3f3740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557071b8efe0 0x557071ba3720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:35:28.651 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:28.649+0000 7f1e0d3f3740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557071b97db0 0x557071bafac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:35:28.651 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:28.649+0000 7f1e08ce0640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557071b97db0 0x557071bafac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:35:28.651 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:28.649+0000 7f1e08ce0640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557071b97db0 0x557071bafac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:42208/0 (socket says 172.21.15.46:42208) 2024-03-31T23:35:28.651 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:28.649+0000 7f1e08ce0640 1 -- 172.21.15.46:0/2309324585 learned_addr learned my addr 172.21.15.46:0/2309324585 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:35:28.652 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:28.649+0000 7f1e037fe640 1 --2- 172.21.15.46:0/2309324585 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557071b8efe0 0x557071ba3720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:35:28.652 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:28.649+0000 7f1e0d3f3740 1 -- 172.21.15.46:0/2309324585 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f1e041b7ba0 con 0x557071b97db0 2024-03-31T23:35:28.652 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:28.649+0000 7f1e0d3f3740 1 -- 172.21.15.46:0/2309324585 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f1e041b5470 con 0x557071b8efe0 2024-03-31T23:35:28.652 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:28.649+0000 7f1e0d3f3740 1 -- 172.21.15.46:0/2309324585 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f1e041b32e0 con 0x557071b8e6c0 2024-03-31T23:35:28.652 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:28.650+0000 7f1e08ce0640 1 -- 172.21.15.46:0/2309324585 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557071b8e6c0 msgr2=0x557071b97600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:35:28.652 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:28.650+0000 7f1e08ce0640 1 --2- 172.21.15.46:0/2309324585 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557071b8e6c0 0x557071b97600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:35:28.652 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:28.650+0000 7f1e08ce0640 1 -- 172.21.15.46:0/2309324585 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557071b8efe0 msgr2=0x557071ba3720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:35:28.652 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:28.650+0000 7f1e08ce0640 1 --2- 172.21.15.46:0/2309324585 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557071b8efe0 0x557071ba3720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:35:28.652 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:28.650+0000 7f1e08ce0640 1 -- 172.21.15.46:0/2309324585 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557071baa560 con 0x557071b97db0 2024-03-31T23:35:28.652 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:28.650+0000 7f1e037fe640 1 --2- 172.21.15.46:0/2309324585 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557071b8efe0 0x557071ba3720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:35:28.652 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:28.650+0000 7f1e08ce0640 1 --2- 172.21.15.46:0/2309324585 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557071b97db0 0x557071bafac0 secure :-1 s=READY pgs=1034 cs=0 l=1 rev1=1 crypto rx=0x7f1dfc02c7e0 tx=0x7f1dfc02c770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=604946bad4618c6b server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:35:28.653 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:28.651+0000 7f1de9ffb640 1 -- 172.21.15.46:0/2309324585 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1dfc039070 con 0x557071b97db0 2024-03-31T23:35:28.654 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:28.651+0000 7f1de9ffb640 1 -- 172.21.15.46:0/2309324585 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1dfc05b070 con 0x557071b97db0 2024-03-31T23:35:28.654 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:28.651+0000 7f1de9ffb640 1 -- 172.21.15.46:0/2309324585 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1dfc051040 con 0x557071b97db0 2024-03-31T23:35:28.654 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:28.651+0000 7f1e0d3f3740 1 -- 172.21.15.46:0/2309324585 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557071b97db0 msgr2=0x557071bafac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:35:28.654 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:28.651+0000 7f1e0d3f3740 1 --2- 172.21.15.46:0/2309324585 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557071b97db0 0x557071bafac0 secure :-1 s=READY pgs=1034 cs=0 l=1 rev1=1 crypto rx=0x7f1dfc02c7e0 tx=0x7f1dfc02c770 comp rx=0 tx=0).stop 2024-03-31T23:35:28.654 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:28.652+0000 7f1e0d3f3740 1 -- 172.21.15.46:0/2309324585 shutdown_connections 2024-03-31T23:35:28.654 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:28.652+0000 7f1e0d3f3740 1 --2- 172.21.15.46:0/2309324585 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557071b97db0 0x557071bafac0 unknown :-1 s=CLOSED pgs=1034 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:35:28.654 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:28.652+0000 7f1e0d3f3740 1 --2- 172.21.15.46:0/2309324585 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557071b8efe0 0x557071ba3720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:35:28.654 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:28.652+0000 7f1e0d3f3740 1 --2- 172.21.15.46:0/2309324585 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557071b8e6c0 0x557071b97600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:35:28.654 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:28.652+0000 7f1e0d3f3740 1 -- 172.21.15.46:0/2309324585 >> 172.21.15.46:0/2309324585 conn(0x557071b88fe0 msgr2=0x557071b8f670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:35:28.654 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:28.652+0000 7f1e0d3f3740 1 -- 172.21.15.46:0/2309324585 shutdown_connections 2024-03-31T23:35:28.654 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:28.652+0000 7f1e0d3f3740 1 -- 172.21.15.46:0/2309324585 wait complete. 2024-03-31T23:35:28.654 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:28.652+0000 7f1e0d3f3740 1 -- start start 2024-03-31T23:35:28.654 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:28.653+0000 7f1e0d3f3740 1 Processor -- start 2024-03-31T23:35:28.657 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:28.653+0000 7f1e0d3f3740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557071b8e6c0 0x557071b91a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:35:28.657 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:28.653+0000 7f1e0d3f3740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557071b8efe0 0x557071b91f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:35:28.657 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:28.653+0000 7f1e0d3f3740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557071b97db0 0x557071b924a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:35:28.657 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:28.653+0000 7f1e03fff640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557071b8e6c0 0x557071b91a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:35:28.657 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:28.653+0000 7f1e03fff640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557071b8e6c0 0x557071b91a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.46:35718/0 (socket says 172.21.15.46:35718) 2024-03-31T23:35:28.657 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:28.653+0000 7f1e037fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557071b8efe0 0x557071b91f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:35:28.657 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:28.654+0000 7f1e037fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557071b8efe0 0x557071b91f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.46:48504/0 (socket says 172.21.15.46:48504) 2024-03-31T23:35:28.657 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:28.654+0000 7f1e08ce0640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557071b97db0 0x557071b924a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:35:28.657 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:28.654+0000 7f1e08ce0640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557071b97db0 0x557071b924a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:42222/0 (socket says 172.21.15.46:42222) 2024-03-31T23:35:28.658 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:28.654+0000 7f1e03fff640 1 -- 172.21.15.46:0/3367693448 learned_addr learned my addr 172.21.15.46:0/3367693448 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:35:28.658 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:28.654+0000 7f1e0d3f3740 1 -- 172.21.15.46:0/3367693448 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f1e041b3930 con 0x557071b97db0 2024-03-31T23:35:28.658 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:28.654+0000 7f1e0d3f3740 1 -- 172.21.15.46:0/3367693448 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f1e041b4da0 con 0x557071b8e6c0 2024-03-31T23:35:28.658 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:28.654+0000 7f1e0d3f3740 1 -- 172.21.15.46:0/3367693448 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f1e041b4000 con 0x557071b8efe0 2024-03-31T23:35:28.658 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:28.654+0000 7f1e0d3f3740 10 client.0 _ll_register_callbacks cb 0x7f1df409a730 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T23:35:28.658 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:28.654+0000 7f1e03fff640 1 -- 172.21.15.46:0/3367693448 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557071b8efe0 msgr2=0x557071b91f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:35:28.658 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:28.654+0000 7f1e03fff640 1 --2- 172.21.15.46:0/3367693448 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557071b8efe0 0x557071b91f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:35:28.658 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:28.654+0000 7f1e03fff640 1 -- 172.21.15.46:0/3367693448 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557071b97db0 msgr2=0x557071b924a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:35:28.658 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:28.654+0000 7f1e03fff640 1 --2- 172.21.15.46:0/3367693448 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557071b97db0 0x557071b924a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:35:28.658 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:28.654+0000 7f1e03fff640 1 -- 172.21.15.46:0/3367693448 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557071c0d0f0 con 0x557071b8e6c0 2024-03-31T23:35:28.658 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:28.654+0000 7f1e037fe640 1 --2- 172.21.15.46:0/3367693448 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557071b8efe0 0x557071b91f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:35:28.658 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:28.654+0000 7f1e08ce0640 1 --2- 172.21.15.46:0/3367693448 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557071b97db0 0x557071b924a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:35:28.658 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:28.654+0000 7f1e03fff640 1 --2- 172.21.15.46:0/3367693448 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557071b8e6c0 0x557071b91a20 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f1df40a30d0 tx=0x7f1df40855c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8cab1750b6654a9a server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:35:28.658 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:28.654+0000 7f1de37fe640 10 client.0 ms_handle_connect on v2:172.21.15.142:3300/0 2024-03-31T23:35:28.659 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:28.655+0000 7f1de37fe640 1 -- 172.21.15.46:0/3367693448 <== mon.2 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1df40b3040 con 0x557071b8e6c0 2024-03-31T23:35:28.659 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:28.655+0000 7f1de37fe640 1 -- 172.21.15.46:0/3367693448 <== mon.2 v2:172.21.15.142:3300/0 2 ==== osd_map(135..135 src has 1..135) ==== 9554+0+0 (secure 0 0 0) 0x7f1df4096940 con 0x557071b8e6c0 2024-03-31T23:35:28.659 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:28.655+0000 7f1e0d3f3740 1 -- 172.21.15.46:0/3367693448 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1dfc05b070 con 0x557071b8e6c0 2024-03-31T23:35:28.659 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:28.655+0000 7f1de37fe640 1 -- 172.21.15.46:0/3367693448 <== mon.2 v2:172.21.15.142:3300/0 3 ==== mon_get_version_reply(handle=1 version=54) ==== 24+0+0 (secure 0 0 0) 0x7f1df4014070 con 0x557071b8e6c0 2024-03-31T23:35:28.659 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:28.655+0000 7f1e0d3f3740 10 client.27878 fetch_fsmap learned FSMap version 54 2024-03-31T23:35:28.659 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:28.655+0000 7f1e0d3f3740 1 -- 172.21.15.46:0/3367693448 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,fsmap=54,monmap=4+}) -- 0x557071c0e380 con 0x557071b8e6c0 2024-03-31T23:35:28.659 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:28.655+0000 7f1de37fe640 1 -- 172.21.15.46:0/3367693448 <== mon.2 v2:172.21.15.142:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1dfc05b070 con 0x557071b8e6c0 2024-03-31T23:35:28.659 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:28.655+0000 7f1de37fe640 1 -- 172.21.15.46:0/3367693448 <== mon.2 v2:172.21.15.142:3300/0 5 ==== fsmap(e 54) ==== 8346+0+0 (secure 0 0 0) 0x7f1df4099ab0 con 0x557071b8e6c0 2024-03-31T23:35:28.659 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:28.655+0000 7f1e0d3f3740 10 client.27878 fetch_fsmap finished waiting for FSMap version 54 2024-03-31T23:35:28.659 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:28.655+0000 7f1e0d3f3740 10 client.27878 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-31T23:35:28.659 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:28.655+0000 7f1e0d3f3740 20 client.27878 populate_metadata read hostname 'smithi046' 2024-03-31T23:35:28.659 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:28.655+0000 7f1e0d3f3740 1 --2- 172.21.15.46:0/3367693448 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x557071ba7e00 0x557071c10980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:35:28.659 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:28.655+0000 7f1e0d3f3740 4 client.27878 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-31T23:35:28.659 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:28.655+0000 7f1e0d3f3740 1 -- 172.21.15.46:0/3367693448 --> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] -- command(tid 0: {"prefix": "flush journal"}) -- 0x557071afee30 con 0x557071ba7e00 2024-03-31T23:35:28.660 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:28.656+0000 7f1e037fe640 1 --2- 172.21.15.46:0/3367693448 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x557071ba7e00 0x557071c10980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:35:28.660 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:28.656+0000 7f1de37fe640 10 client.27878 ms_handle_connect on v2:172.21.15.142:6838/2519143918 2024-03-31T23:35:28.660 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:28.656+0000 7f1e037fe640 1 --2- 172.21.15.46:0/3367693448 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x557071ba7e00 0x557071c10980 crc :-1 s=READY pgs=289 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a4cabee0e24ac49e server_cookie=306998bb07ae8c03 in_seq=0 out_seq=0 2024-03-31T23:35:28.856 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:28.854+0000 7f1de37fe640 1 -- 172.21.15.46:0/3367693448 <== mds.0 v2:172.21.15.142:6838/2519143918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x557071afee30 con 0x557071ba7e00 2024-03-31T23:35:28.856 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:28.854+0000 7f1de37fe640 10 client.27878 handle_command_reply: tid=0 2024-03-31T23:35:28.856 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:28.854+0000 7f1e0d3f3740 1 client.27878 shutdown 2024-03-31T23:35:28.856 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:28.855+0000 7f1e0d3f3740 1 -- 172.21.15.46:0/3367693448 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557071b8e6c0 msgr2=0x557071b91a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:35:28.856 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:28.855+0000 7f1e0d3f3740 1 --2- 172.21.15.46:0/3367693448 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557071b8e6c0 0x557071b91a20 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f1df40a30d0 tx=0x7f1df40855c0 comp rx=0 tx=0).stop 2024-03-31T23:35:28.856 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:28.855+0000 7f1e0d3f3740 1 -- 172.21.15.46:0/3367693448 shutdown_connections 2024-03-31T23:35:28.856 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:28.855+0000 7f1e0d3f3740 1 --2- 172.21.15.46:0/3367693448 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x557071ba7e00 0x557071c10980 crc :-1 s=READY pgs=289 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:35:28.856 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:28.855+0000 7f1e0d3f3740 1 --2- 172.21.15.46:0/3367693448 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557071b97db0 0x557071b924a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:35:28.856 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:28.855+0000 7f1e0d3f3740 1 --2- 172.21.15.46:0/3367693448 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557071b8efe0 0x557071b91f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:35:28.856 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:28.855+0000 7f1de37fe640 0 client.27878 ms_handle_reset on v2:172.21.15.142:6838/2519143918 2024-03-31T23:35:28.857 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:28.855+0000 7f1e0d3f3740 1 --2- 172.21.15.46:0/3367693448 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557071b8e6c0 0x557071b91a20 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:35:28.857 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:28.855+0000 7f1e0d3f3740 1 -- 172.21.15.46:0/3367693448 >> 172.21.15.46:0/3367693448 conn(0x557071b88fe0 msgr2=0x557071b8a430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:35:28.857 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:28.855+0000 7f1e0d3f3740 1 -- 172.21.15.46:0/3367693448 shutdown_connections 2024-03-31T23:35:28.857 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:28.855+0000 7f1e0d3f3740 1 -- 172.21.15.46:0/3367693448 wait complete. 2024-03-31T23:35:28.857 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:28.856+0000 7f1e0d3f3740 20 client.27878 trim_cache size 0 max 16384 2024-03-31T23:35:28.858 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout:{ 2024-03-31T23:35:28.858 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout: "message": "", 2024-03-31T23:35:28.858 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout: "return_code": 0 2024-03-31T23:35:28.858 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout:} 2024-03-31T23:35:28.859 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:28.858+0000 7f1e0a743640 1 -- 172.21.15.46:0/2412191340 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f1dec0a26b0 msgr2=0x7f1dec0a4b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:35:28.859 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:28.858+0000 7f1e0a743640 1 --2- 172.21.15.46:0/2412191340 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f1dec0a26b0 0x7f1dec0a4b70 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f1e0406aed0 tx=0x7f1dfc02e000 comp rx=0 tx=0).stop 2024-03-31T23:35:28.859 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:28.858+0000 7f1e0a743640 1 -- 172.21.15.46:0/2412191340 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1e0406a430 msgr2=0x7f1e0410d940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:35:28.859 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:28.858+0000 7f1e0a743640 1 --2- 172.21.15.46:0/2412191340 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1e0406a430 0x7f1e0410d940 secure :-1 s=READY pgs=1032 cs=0 l=1 rev1=1 crypto rx=0x7f1df400d950 tx=0x7f1df400de10 comp rx=0 tx=0).stop 2024-03-31T23:35:28.860 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:28.859+0000 7f1e0a743640 1 -- 172.21.15.46:0/2412191340 shutdown_connections 2024-03-31T23:35:28.860 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:28.859+0000 7f1e0a743640 1 --2- 172.21.15.46:0/2412191340 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f1dec0a26b0 0x7f1dec0a4b70 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:35:28.860 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:28.859+0000 7f1e0a743640 1 --2- 172.21.15.46:0/2412191340 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1e0410dfa0 0x7f1e0410b000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:35:28.860 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:28.859+0000 7f1e0a743640 1 --2- 172.21.15.46:0/2412191340 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1e04102630 0x7f1e0410aa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:35:28.860 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:28.859+0000 7f1e0a743640 1 --2- 172.21.15.46:0/2412191340 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1e0406a430 0x7f1e0410d940 unknown :-1 s=CLOSED pgs=1032 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:35:28.860 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:28.859+0000 7f1e0a743640 1 -- 172.21.15.46:0/2412191340 >> 172.21.15.46:0/2412191340 conn(0x7f1e04077630 msgr2=0x7f1e040ff920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:35:28.860 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:28.859+0000 7f1e0a743640 1 -- 172.21.15.46:0/2412191340 shutdown_connections 2024-03-31T23:35:28.860 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:28.859+0000 7f1e0a743640 1 -- 172.21.15.46:0/2412191340 wait complete. 2024-03-31T23:35:29.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:35:28 smithi162 ceph-mon[36237]: pgmap v754: 129 pgs: 129 active+clean; 1.3 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 4.4 MiB/s wr, 269 op/s 2024-03-31T23:35:29.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:35:28 smithi046 ceph-mon[32498]: pgmap v754: 129 pgs: 129 active+clean; 1.3 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 4.4 MiB/s wr, 269 op/s 2024-03-31T23:35:29.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:35:28 smithi142 ceph-mon[35271]: pgmap v754: 129 pgs: 129 active+clean; 1.3 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 4.4 MiB/s wr, 269 op/s 2024-03-31T23:35:29.315 INFO:tasks.workunit.client.0.smithi046.stdout: 10 97851 8.21 MB/sec execute 514 sec latency 155.777 ms 2024-03-31T23:35:30.315 INFO:tasks.workunit.client.0.smithi046.stdout: 10 98024 8.22 MB/sec execute 515 sec latency 76.766 ms 2024-03-31T23:35:31.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:35:30 smithi162 ceph-mon[36237]: pgmap v755: 129 pgs: 129 active+clean; 1.2 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 7.3 MiB/s wr, 422 op/s 2024-03-31T23:35:31.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:35:30 smithi046 ceph-mon[32498]: pgmap v755: 129 pgs: 129 active+clean; 1.2 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 7.3 MiB/s wr, 422 op/s 2024-03-31T23:35:31.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:35:30 smithi142 ceph-mon[35271]: pgmap v755: 129 pgs: 129 active+clean; 1.2 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 7.3 MiB/s wr, 422 op/s 2024-03-31T23:35:31.315 INFO:tasks.workunit.client.0.smithi046.stdout: 10 98153 8.21 MB/sec execute 516 sec latency 80.782 ms 2024-03-31T23:35:32.315 INFO:tasks.workunit.client.0.smithi046.stdout: 10 98269 8.20 MB/sec execute 517 sec latency 71.486 ms 2024-03-31T23:35:33.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:35:32 smithi162 ceph-mon[36237]: pgmap v756: 129 pgs: 129 active+clean; 1.2 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 4.8 MiB/s wr, 254 op/s 2024-03-31T23:35:33.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:35:32 smithi046 ceph-mon[32498]: pgmap v756: 129 pgs: 129 active+clean; 1.2 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 4.8 MiB/s wr, 254 op/s 2024-03-31T23:35:33.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:35:32 smithi142 ceph-mon[35271]: pgmap v756: 129 pgs: 129 active+clean; 1.2 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 4.8 MiB/s wr, 254 op/s 2024-03-31T23:35:33.315 INFO:tasks.workunit.client.0.smithi046.stdout: 10 98420 8.21 MB/sec execute 518 sec latency 71.513 ms 2024-03-31T23:35:34.315 INFO:tasks.workunit.client.0.smithi046.stdout: 10 98549 8.21 MB/sec execute 519 sec latency 135.555 ms 2024-03-31T23:35:35.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:35:34 smithi162 ceph-mon[36237]: pgmap v757: 129 pgs: 129 active+clean; 1.2 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 4.8 MiB/s wr, 254 op/s 2024-03-31T23:35:35.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:35:34 smithi162 ceph-mon[36237]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-31T23:35:35.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:35:34 smithi046 ceph-mon[32498]: pgmap v757: 129 pgs: 129 active+clean; 1.2 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 4.8 MiB/s wr, 254 op/s 2024-03-31T23:35:35.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:35:34 smithi046 ceph-mon[32498]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-31T23:35:35.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:35:34 smithi142 ceph-mon[35271]: pgmap v757: 129 pgs: 129 active+clean; 1.2 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 4.8 MiB/s wr, 254 op/s 2024-03-31T23:35:35.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:35:34 smithi142 ceph-mon[35271]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-31T23:35:35.315 INFO:tasks.workunit.client.0.smithi046.stdout: 10 98678 8.20 MB/sec execute 520 sec latency 107.086 ms 2024-03-31T23:35:36.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:35:35 smithi162 ceph-mon[36237]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:35:36.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:35:35 smithi162 ceph-mon[36237]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-31T23:35:36.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:35:35 smithi162 ceph-mon[36237]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' 2024-03-31T23:35:36.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:35:35 smithi046 ceph-mon[32498]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:35:36.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:35:35 smithi046 ceph-mon[32498]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-31T23:35:36.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:35:35 smithi046 ceph-mon[32498]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' 2024-03-31T23:35:36.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:35:35 smithi142 ceph-mon[35271]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:35:36.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:35:35 smithi142 ceph-mon[35271]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-31T23:35:36.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:35:35 smithi142 ceph-mon[35271]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' 2024-03-31T23:35:36.315 INFO:tasks.workunit.client.0.smithi046.stdout: 10 98801 8.19 MB/sec execute 521 sec latency 128.101 ms 2024-03-31T23:35:37.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:35:36 smithi162 ceph-mon[36237]: pgmap v758: 129 pgs: 129 active+clean; 1.3 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 7.4 MiB/s wr, 368 op/s 2024-03-31T23:35:37.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:35:36 smithi046 ceph-mon[32498]: pgmap v758: 129 pgs: 129 active+clean; 1.3 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 7.4 MiB/s wr, 368 op/s 2024-03-31T23:35:37.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:35:36 smithi142 ceph-mon[35271]: pgmap v758: 129 pgs: 129 active+clean; 1.3 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 7.4 MiB/s wr, 368 op/s 2024-03-31T23:35:37.315 INFO:tasks.workunit.client.0.smithi046.stdout: 10 98931 8.19 MB/sec execute 522 sec latency 122.979 ms 2024-03-31T23:35:38.316 INFO:tasks.workunit.client.0.smithi046.stdout: 10 99088 8.19 MB/sec execute 523 sec latency 135.710 ms 2024-03-31T23:35:39.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:35:38 smithi162 ceph-mon[36237]: pgmap v759: 129 pgs: 129 active+clean; 1.3 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 5.5 MiB/s wr, 267 op/s 2024-03-31T23:35:39.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:35:38 smithi046 ceph-mon[32498]: pgmap v759: 129 pgs: 129 active+clean; 1.3 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 5.5 MiB/s wr, 267 op/s 2024-03-31T23:35:39.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:35:38 smithi142 ceph-mon[35271]: pgmap v759: 129 pgs: 129 active+clean; 1.3 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 5.5 MiB/s wr, 267 op/s 2024-03-31T23:35:39.315 INFO:tasks.workunit.client.0.smithi046.stdout: 10 99240 8.19 MB/sec execute 524 sec latency 84.853 ms 2024-03-31T23:35:40.316 INFO:tasks.workunit.client.0.smithi046.stdout: 10 99375 8.18 MB/sec execute 525 sec latency 81.753 ms 2024-03-31T23:35:41.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:35:40 smithi162 ceph-mon[36237]: pgmap v760: 129 pgs: 129 active+clean; 1.3 GiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 8.0 MiB/s wr, 392 op/s 2024-03-31T23:35:41.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:35:40 smithi046 ceph-mon[32498]: pgmap v760: 129 pgs: 129 active+clean; 1.3 GiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 8.0 MiB/s wr, 392 op/s 2024-03-31T23:35:41.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:35:40 smithi142 ceph-mon[35271]: pgmap v760: 129 pgs: 129 active+clean; 1.3 GiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 8.0 MiB/s wr, 392 op/s 2024-03-31T23:35:41.316 INFO:tasks.workunit.client.0.smithi046.stdout: 10 99530 8.19 MB/sec execute 526 sec latency 82.598 ms 2024-03-31T23:35:41.993 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:41.992+0000 7efe3b007640 1 -- 172.21.15.46:0/407216478 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efe34101260 msgr2=0x7efe3410d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:35:41.993 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:41.992+0000 7efe3b007640 1 --2- 172.21.15.46:0/407216478 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efe34101260 0x7efe3410d480 secure :-1 s=READY pgs=1035 cs=0 l=1 rev1=1 crypto rx=0x7efe2c009ca0 tx=0x7efe2c02c0f0 comp rx=0 tx=0).stop 2024-03-31T23:35:41.994 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:41.992+0000 7efe3b007640 1 -- 172.21.15.46:0/407216478 shutdown_connections 2024-03-31T23:35:41.994 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:41.992+0000 7efe3b007640 1 --2- 172.21.15.46:0/407216478 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efe34101260 0x7efe3410d480 unknown :-1 s=CLOSED pgs=1035 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:35:41.994 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:41.992+0000 7efe3b007640 1 --2- 172.21.15.46:0/407216478 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7efe341008a0 0x7efe34100d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:35:41.994 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:41.992+0000 7efe3b007640 1 --2- 172.21.15.46:0/407216478 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7efe341055f0 0x7efe341059d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:35:41.994 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:41.992+0000 7efe3b007640 1 -- 172.21.15.46:0/407216478 >> 172.21.15.46:0/407216478 conn(0x7efe340774a0 msgr2=0x7efe340778b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:35:41.994 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:41.992+0000 7efe3b007640 1 -- 172.21.15.46:0/407216478 shutdown_connections 2024-03-31T23:35:41.994 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:41.992+0000 7efe3b007640 1 -- 172.21.15.46:0/407216478 wait complete. 2024-03-31T23:35:41.994 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:41.993+0000 7efe3b007640 1 Processor -- start 2024-03-31T23:35:41.994 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:41.993+0000 7efe3b007640 1 -- start start 2024-03-31T23:35:41.996 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:41.993+0000 7efe3b007640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efe341008a0 0x7efe3419a080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:35:41.996 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:41.993+0000 7efe3b007640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7efe34101260 0x7efe3419a5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:35:41.996 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:41.993+0000 7efe3b007640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7efe341055f0 0x7efe3419e940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:35:41.996 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:41.993+0000 7efe3b007640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7efe3410f8c0 con 0x7efe341008a0 2024-03-31T23:35:41.996 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:41.993+0000 7efe3b007640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7efe3410f740 con 0x7efe34101260 2024-03-31T23:35:41.997 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:41.993+0000 7efe3b007640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7efe3410fa40 con 0x7efe341055f0 2024-03-31T23:35:41.997 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:41.993+0000 7efe38da3640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efe341008a0 0x7efe3419a080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:35:41.997 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:41.993+0000 7efe2bfff640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7efe34101260 0x7efe3419a5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:35:41.997 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:41.993+0000 7efe395a4640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7efe341055f0 0x7efe3419e940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:35:41.997 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:41.993+0000 7efe38da3640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efe341008a0 0x7efe3419a080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60340/0 (socket says 172.21.15.46:60340) 2024-03-31T23:35:41.997 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:41.993+0000 7efe38da3640 1 -- 172.21.15.46:0/658657595 learned_addr learned my addr 172.21.15.46:0/658657595 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:35:41.997 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:41.993+0000 7efe395a4640 1 -- 172.21.15.46:0/658657595 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7efe34101260 msgr2=0x7efe3419a5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:35:41.997 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:41.993+0000 7efe395a4640 1 --2- 172.21.15.46:0/658657595 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7efe34101260 0x7efe3419a5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:35:41.997 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:41.993+0000 7efe395a4640 1 -- 172.21.15.46:0/658657595 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efe341008a0 msgr2=0x7efe3419a080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:35:41.997 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:41.993+0000 7efe395a4640 1 --2- 172.21.15.46:0/658657595 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efe341008a0 0x7efe3419a080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:35:41.997 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:41.993+0000 7efe395a4640 1 -- 172.21.15.46:0/658657595 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efe3419f070 con 0x7efe341055f0 2024-03-31T23:35:41.998 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:41.993+0000 7efe38da3640 1 --2- 172.21.15.46:0/658657595 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efe341008a0 0x7efe3419a080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:35:41.998 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:41.994+0000 7efe395a4640 1 --2- 172.21.15.46:0/658657595 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7efe341055f0 0x7efe3419e940 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7efe2c02c700 tx=0x7efe2c035d60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:35:41.998 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:41.994+0000 7efe29ffb640 1 -- 172.21.15.46:0/658657595 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe2c039040 con 0x7efe341055f0 2024-03-31T23:35:41.998 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:41.994+0000 7efe3b007640 1 -- 172.21.15.46:0/658657595 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efe3419f300 con 0x7efe341055f0 2024-03-31T23:35:41.998 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:41.994+0000 7efe29ffb640 1 -- 172.21.15.46:0/658657595 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7efe2c044070 con 0x7efe341055f0 2024-03-31T23:35:41.998 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:41.994+0000 7efe29ffb640 1 -- 172.21.15.46:0/658657595 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe2c0042a0 con 0x7efe341055f0 2024-03-31T23:35:41.998 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:41.994+0000 7efe3b007640 1 -- 172.21.15.46:0/658657595 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efe341a6bf0 con 0x7efe341055f0 2024-03-31T23:35:42.002 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:41.997+0000 7efe29ffb640 1 -- 172.21.15.46:0/658657595 <== mon.1 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 25) ==== 137451+0+0 (secure 0 0 0) 0x7efe2c004440 con 0x7efe341055f0 2024-03-31T23:35:42.002 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:41.997+0000 7efe3dbee740 1 Processor -- start 2024-03-31T23:35:42.002 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:41.997+0000 7efe3dbee740 1 -- start start 2024-03-31T23:35:42.002 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:41.998+0000 7efe3dbee740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5570578596c0 0x557057859aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:35:42.002 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:41.998+0000 7efe3dbee740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557057859fe0 0x55705786b080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:35:42.002 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:41.998+0000 7efe3dbee740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55705786b5c0 0x55705786d9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:35:42.002 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:41.998+0000 7efe3dbee740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x5570577bc8f0 con 0x557057859fe0 2024-03-31T23:35:42.002 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:41.998+0000 7efe3dbee740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x5570577658d0 con 0x5570578596c0 2024-03-31T23:35:42.002 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:41.998+0000 7efe3dbee740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5570577b8990 con 0x55705786b5c0 2024-03-31T23:35:42.002 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:41.998+0000 7efe29ffb640 1 --2- 172.21.15.46:0/658657595 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7efe1c0a27b0 0x7efe1c0a4c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:35:42.002 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:41.998+0000 7efe38da3640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557057859fe0 0x55705786b080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:35:42.002 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:41.998+0000 7efe38da3640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557057859fe0 0x55705786b080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60346/0 (socket says 172.21.15.46:60346) 2024-03-31T23:35:42.003 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:41.998+0000 7efe38da3640 1 -- 172.21.15.46:0/14680989 learned_addr learned my addr 172.21.15.46:0/14680989 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:35:42.003 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:41.998+0000 7efe2bfff640 1 --2- 172.21.15.46:0/14680989 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5570578596c0 0x557057859aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:35:42.003 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:41.998+0000 7efe2bfff640 1 --2- 172.21.15.46:0/14680989 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55705786b5c0 0x55705786d9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:35:42.003 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:41.998+0000 7efe395a4640 1 --2- 172.21.15.46:0/658657595 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7efe1c0a27b0 0x7efe1c0a4c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:35:42.003 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:41.998+0000 7efe38da3640 1 -- 172.21.15.46:0/14680989 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55705786b5c0 msgr2=0x55705786d9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:35:42.003 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:41.998+0000 7efe38da3640 1 --2- 172.21.15.46:0/14680989 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55705786b5c0 0x55705786d9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:35:42.003 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:41.998+0000 7efe38da3640 1 -- 172.21.15.46:0/14680989 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5570578596c0 msgr2=0x557057859aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:35:42.003 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:41.998+0000 7efe38da3640 1 --2- 172.21.15.46:0/14680989 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5570578596c0 0x557057859aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:35:42.003 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:41.998+0000 7efe38da3640 1 -- 172.21.15.46:0/14680989 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5570574d1210 con 0x557057859fe0 2024-03-31T23:35:42.003 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:41.998+0000 7efe2bfff640 1 --2- 172.21.15.46:0/14680989 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55705786b5c0 0x55705786d9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T23:35:42.003 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:41.999+0000 7efe38da3640 1 --2- 172.21.15.46:0/14680989 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557057859fe0 0x55705786b080 secure :-1 s=READY pgs=1036 cs=0 l=1 rev1=1 crypto rx=0x7efe30011650 tx=0x7efe30009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e092a269738265d7 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:35:42.003 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:41.999+0000 7efe29ffb640 1 -- 172.21.15.46:0/658657595 <== mon.1 v2:172.21.15.162:3300/0 5 ==== osd_map(135..135 src has 1..135) ==== 9554+0+0 (secure 0 0 0) 0x7efe2c0f1190 con 0x7efe341055f0 2024-03-31T23:35:42.003 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:41.999+0000 7efe395a4640 1 --2- 172.21.15.46:0/658657595 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7efe1c0a27b0 0x7efe1c0a4c70 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7efe2c0ee5b0 tx=0x7efe2c0f48d0 comp rx=0 tx=0).ready entity=mgr.15654 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:35:42.003 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:41.999+0000 7efe1affd640 1 -- 172.21.15.46:0/14680989 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe3001e070 con 0x557057859fe0 2024-03-31T23:35:42.004 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:41.999+0000 7efe1affd640 1 -- 172.21.15.46:0/14680989 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7efe3000fb70 con 0x557057859fe0 2024-03-31T23:35:42.004 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:41.999+0000 7efe1affd640 1 -- 172.21.15.46:0/14680989 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe30019c60 con 0x557057859fe0 2024-03-31T23:35:42.004 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:42.000+0000 7efe3dbee740 1 -- 172.21.15.46:0/14680989 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557057859fe0 msgr2=0x55705786b080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:35:42.004 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:42.000+0000 7efe3dbee740 1 --2- 172.21.15.46:0/14680989 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557057859fe0 0x55705786b080 secure :-1 s=READY pgs=1036 cs=0 l=1 rev1=1 crypto rx=0x7efe30011650 tx=0x7efe30009f90 comp rx=0 tx=0).stop 2024-03-31T23:35:42.004 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:42.000+0000 7efe3dbee740 1 -- 172.21.15.46:0/14680989 shutdown_connections 2024-03-31T23:35:42.004 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:42.000+0000 7efe3dbee740 1 --2- 172.21.15.46:0/14680989 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55705786b5c0 0x55705786d9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:35:42.004 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:42.000+0000 7efe3dbee740 1 --2- 172.21.15.46:0/14680989 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557057859fe0 0x55705786b080 unknown :-1 s=CLOSED pgs=1036 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:35:42.004 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:42.000+0000 7efe3dbee740 1 --2- 172.21.15.46:0/14680989 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5570578596c0 0x557057859aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:35:42.004 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:42.000+0000 7efe3dbee740 1 -- 172.21.15.46:0/14680989 >> 172.21.15.46:0/14680989 conn(0x557057853fe0 msgr2=0x5570578543c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:35:42.004 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:42.000+0000 7efe3dbee740 1 -- 172.21.15.46:0/14680989 shutdown_connections 2024-03-31T23:35:42.004 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:42.001+0000 7efe3dbee740 1 -- 172.21.15.46:0/14680989 wait complete. 2024-03-31T23:35:42.008 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:42.001+0000 7efe3dbee740 1 -- start start 2024-03-31T23:35:42.008 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:42.001+0000 7efe3dbee740 1 Processor -- start 2024-03-31T23:35:42.008 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:42.002+0000 7efe3dbee740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5570578596c0 0x5570578686e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:35:42.008 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:42.002+0000 7efe3dbee740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557057859fe0 0x557057868c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:35:42.008 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:42.002+0000 7efe3dbee740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55705786b5c0 0x557057869160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:35:42.009 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:42.002+0000 7efe3dbee740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55705786e1f0 con 0x55705786b5c0 2024-03-31T23:35:42.009 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:42.002+0000 7efe3dbee740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55705775af80 con 0x5570578596c0 2024-03-31T23:35:42.009 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:42.002+0000 7efe3dbee740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55705786e370 con 0x557057859fe0 2024-03-31T23:35:42.009 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:42.002+0000 7efe3dbee740 10 client.0 _ll_register_callbacks cb 0x5570577b3290 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T23:35:42.009 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:42.002+0000 7efe38da3640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557057859fe0 0x557057868c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:35:42.009 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:42.002+0000 7efe38da3640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557057859fe0 0x557057868c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.46:60890/0 (socket says 172.21.15.46:60890) 2024-03-31T23:35:42.009 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:42.002+0000 7efe38da3640 1 -- 172.21.15.46:0/2484860442 learned_addr learned my addr 172.21.15.46:0/2484860442 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:35:42.009 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:42.002+0000 7efe2bfff640 1 --2- 172.21.15.46:0/2484860442 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55705786b5c0 0x557057869160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:35:42.009 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:42.002+0000 7efe2bfff640 1 --2- 172.21.15.46:0/2484860442 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5570578596c0 0x5570578686e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:35:42.009 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:42.002+0000 7efe38da3640 1 -- 172.21.15.46:0/2484860442 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5570578596c0 msgr2=0x5570578686e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:35:42.009 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:42.002+0000 7efe38da3640 1 --2- 172.21.15.46:0/2484860442 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5570578596c0 0x5570578686e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:35:42.009 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:42.002+0000 7efe38da3640 1 -- 172.21.15.46:0/2484860442 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55705786b5c0 msgr2=0x557057869160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:35:42.009 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:42.002+0000 7efe38da3640 1 --2- 172.21.15.46:0/2484860442 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55705786b5c0 0x557057869160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:35:42.009 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:42.002+0000 7efe38da3640 1 -- 172.21.15.46:0/2484860442 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55705759d620 con 0x557057859fe0 2024-03-31T23:35:42.010 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:42.002+0000 7efe2bfff640 1 --2- 172.21.15.46:0/2484860442 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55705786b5c0 0x557057869160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:35:42.010 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:42.003+0000 7efe38da3640 1 --2- 172.21.15.46:0/2484860442 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557057859fe0 0x557057868c20 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7efe30019880 tx=0x7efe30009ef0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=26a41993a4868a12 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:35:42.010 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:42.003+0000 7efe13fff640 10 client.0 ms_handle_connect on v2:172.21.15.162:3300/0 2024-03-31T23:35:42.010 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:42.003+0000 7efe13fff640 1 -- 172.21.15.46:0/2484860442 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe30027060 con 0x557057859fe0 2024-03-31T23:35:42.010 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:42.003+0000 7efe3dbee740 1 -- 172.21.15.46:0/2484860442 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efe3000fb70 con 0x557057859fe0 2024-03-31T23:35:42.010 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:42.003+0000 7efe13fff640 1 -- 172.21.15.46:0/2484860442 <== mon.1 v2:172.21.15.162:3300/0 2 ==== osd_map(135..135 src has 1..135) ==== 9554+0+0 (secure 0 0 0) 0x7efe3000f6c0 con 0x557057859fe0 2024-03-31T23:35:42.010 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:42.003+0000 7efe13fff640 1 -- 172.21.15.46:0/2484860442 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_get_version_reply(handle=1 version=54) ==== 24+0+0 (secure 0 0 0) 0x7efe3002e050 con 0x557057859fe0 2024-03-31T23:35:42.010 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:42.003+0000 7efe3dbee740 10 client.27793 fetch_fsmap learned FSMap version 54 2024-03-31T23:35:42.010 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:42.003+0000 7efe3dbee740 1 -- 172.21.15.46:0/2484860442 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,fsmap=54,monmap=4+}) -- 0x55705787a1b0 con 0x557057859fe0 2024-03-31T23:35:42.010 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:42.004+0000 7efe13fff640 1 -- 172.21.15.46:0/2484860442 <== mon.1 v2:172.21.15.162:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7efe3000fb70 con 0x557057859fe0 2024-03-31T23:35:42.010 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:42.004+0000 7efe13fff640 1 -- 172.21.15.46:0/2484860442 <== mon.1 v2:172.21.15.162:3300/0 5 ==== fsmap(e 54) ==== 8346+0+0 (secure 0 0 0) 0x7efe30021030 con 0x557057859fe0 2024-03-31T23:35:42.010 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:42.004+0000 7efe3dbee740 10 client.27793 fetch_fsmap finished waiting for FSMap version 54 2024-03-31T23:35:42.010 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:42.004+0000 7efe3dbee740 10 client.27793 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-31T23:35:42.010 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:42.004+0000 7efe3dbee740 20 client.27793 populate_metadata read hostname 'smithi046' 2024-03-31T23:35:42.011 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:42.004+0000 7efe3dbee740 1 --2- 172.21.15.46:0/2484860442 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55705787b970 0x55705787ddd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:35:42.011 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:42.004+0000 7efe3dbee740 4 client.27793 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-31T23:35:42.011 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:42.004+0000 7efe3dbee740 1 -- 172.21.15.46:0/2484860442 --> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5570577d0120 con 0x55705787b970 2024-03-31T23:35:42.011 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:42.004+0000 7efe2bfff640 1 --2- 172.21.15.46:0/2484860442 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55705787b970 0x55705787ddd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:35:42.011 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:42.005+0000 7efe2bfff640 1 --2- 172.21.15.46:0/2484860442 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55705787b970 0x55705787ddd0 crc :-1 s=READY pgs=290 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bf50de9e2f4e46c0 server_cookie=5794da6dfccece3b in_seq=0 out_seq=0 2024-03-31T23:35:42.011 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:42.005+0000 7efe13fff640 10 client.27793 ms_handle_connect on v2:172.21.15.142:6838/2519143918 2024-03-31T23:35:42.011 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:42.006+0000 7efe13fff640 1 -- 172.21.15.46:0/2484860442 <== mds.0 v2:172.21.15.142:6838/2519143918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5570577d0120 con 0x55705787b970 2024-03-31T23:35:42.011 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:42.006+0000 7efe13fff640 10 client.27793 handle_command_reply: tid=0 2024-03-31T23:35:42.011 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:42.006+0000 7efe3dbee740 1 client.27793 shutdown 2024-03-31T23:35:42.011 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:42.006+0000 7efe3dbee740 1 -- 172.21.15.46:0/2484860442 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557057859fe0 msgr2=0x557057868c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:35:42.011 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:42.006+0000 7efe3dbee740 1 --2- 172.21.15.46:0/2484860442 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557057859fe0 0x557057868c20 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7efe30019880 tx=0x7efe30009ef0 comp rx=0 tx=0).stop 2024-03-31T23:35:42.011 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:42.006+0000 7efe3dbee740 1 -- 172.21.15.46:0/2484860442 shutdown_connections 2024-03-31T23:35:42.011 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:42.006+0000 7efe3dbee740 1 --2- 172.21.15.46:0/2484860442 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55705787b970 0x55705787ddd0 crc :-1 s=READY pgs=290 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:35:42.012 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:42.006+0000 7efe3dbee740 1 --2- 172.21.15.46:0/2484860442 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55705786b5c0 0x557057869160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:35:42.012 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:42.006+0000 7efe3dbee740 1 --2- 172.21.15.46:0/2484860442 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557057859fe0 0x557057868c20 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:35:42.012 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:42.006+0000 7efe13fff640 0 client.27793 ms_handle_reset on v2:172.21.15.142:6838/2519143918 2024-03-31T23:35:42.012 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:42.006+0000 7efe3dbee740 1 --2- 172.21.15.46:0/2484860442 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5570578596c0 0x5570578686e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:35:42.012 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:42.006+0000 7efe3dbee740 1 -- 172.21.15.46:0/2484860442 >> 172.21.15.46:0/2484860442 conn(0x557057853fe0 msgr2=0x557057855d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:35:42.012 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:42.006+0000 7efe3dbee740 1 -- 172.21.15.46:0/2484860442 shutdown_connections 2024-03-31T23:35:42.012 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:42.006+0000 7efe3dbee740 1 -- 172.21.15.46:0/2484860442 wait complete. 2024-03-31T23:35:42.012 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:42.007+0000 7efe3dbee740 20 client.27793 trim_cache size 0 max 16384 2024-03-31T23:35:42.031 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:42.029+0000 7efe3dbee740 1 Processor -- start 2024-03-31T23:35:42.032 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:42.029+0000 7efe3dbee740 1 -- start start 2024-03-31T23:35:42.032 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:42.029+0000 7efe3dbee740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5570578596c0 0x55705786ae10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:35:42.032 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:42.029+0000 7efe3dbee740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557057859fe0 0x557057876f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:35:42.032 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:42.029+0000 7efe3dbee740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55705786b5c0 0x5570578836e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:35:42.032 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:42.029+0000 7efe3dbee740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7efe341af270 con 0x5570578596c0 2024-03-31T23:35:42.032 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:42.029+0000 7efe3dbee740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7efe341acb40 con 0x557057859fe0 2024-03-31T23:35:42.032 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:42.029+0000 7efe3dbee740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7efe341aaa10 con 0x55705786b5c0 2024-03-31T23:35:42.032 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:42.030+0000 7efe2bfff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5570578596c0 0x55705786ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:35:42.032 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:42.030+0000 7efe2bfff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5570578596c0 0x55705786ae10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60362/0 (socket says 172.21.15.46:60362) 2024-03-31T23:35:42.032 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:42.030+0000 7efe38da3640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557057859fe0 0x557057876f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:35:42.032 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:42.030+0000 7efe2bfff640 1 -- 172.21.15.46:0/710923298 learned_addr learned my addr 172.21.15.46:0/710923298 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:35:42.032 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:42.030+0000 7efe2bfff640 1 --2- 172.21.15.46:0/710923298 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55705786b5c0 0x5570578836e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:35:42.032 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:42.030+0000 7efe2bfff640 1 -- 172.21.15.46:0/710923298 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55705786b5c0 msgr2=0x5570578836e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:35:42.033 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:42.030+0000 7efe2bfff640 1 --2- 172.21.15.46:0/710923298 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55705786b5c0 0x5570578836e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:35:42.033 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:42.030+0000 7efe2bfff640 1 -- 172.21.15.46:0/710923298 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557057859fe0 msgr2=0x557057876f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:35:42.033 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:42.030+0000 7efe2bfff640 1 --2- 172.21.15.46:0/710923298 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557057859fe0 0x557057876f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:35:42.033 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:42.030+0000 7efe2bfff640 1 -- 172.21.15.46:0/710923298 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55705785a5f0 con 0x5570578596c0 2024-03-31T23:35:42.033 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:42.030+0000 7efe38da3640 1 --2- 172.21.15.46:0/710923298 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557057859fe0 0x557057876f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:35:42.033 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:42.030+0000 7efe2bfff640 1 --2- 172.21.15.46:0/710923298 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5570578596c0 0x55705786ae10 secure :-1 s=READY pgs=1037 cs=0 l=1 rev1=1 crypto rx=0x7efe24000c00 tx=0x7efe240096c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d83f69c1757604bd server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:35:42.033 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:42.030+0000 7efe1a7fc640 1 -- 172.21.15.46:0/710923298 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe24021d90 con 0x5570578596c0 2024-03-31T23:35:42.033 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:42.030+0000 7efe1a7fc640 1 -- 172.21.15.46:0/710923298 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7efe240120c0 con 0x5570578596c0 2024-03-31T23:35:42.033 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:42.030+0000 7efe1a7fc640 1 -- 172.21.15.46:0/710923298 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe240256c0 con 0x5570578596c0 2024-03-31T23:35:42.036 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:42.031+0000 7efe3dbee740 1 -- 172.21.15.46:0/710923298 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5570578596c0 msgr2=0x55705786ae10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:35:42.036 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:42.031+0000 7efe3dbee740 1 --2- 172.21.15.46:0/710923298 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5570578596c0 0x55705786ae10 secure :-1 s=READY pgs=1037 cs=0 l=1 rev1=1 crypto rx=0x7efe24000c00 tx=0x7efe240096c0 comp rx=0 tx=0).stop 2024-03-31T23:35:42.036 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:42.031+0000 7efe3dbee740 1 -- 172.21.15.46:0/710923298 shutdown_connections 2024-03-31T23:35:42.036 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:42.031+0000 7efe3dbee740 1 --2- 172.21.15.46:0/710923298 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55705786b5c0 0x5570578836e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:35:42.036 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:42.031+0000 7efe3dbee740 1 --2- 172.21.15.46:0/710923298 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557057859fe0 0x557057876f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:35:42.036 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:42.031+0000 7efe3dbee740 1 --2- 172.21.15.46:0/710923298 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5570578596c0 0x55705786ae10 unknown :-1 s=CLOSED pgs=1037 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:35:42.036 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:42.031+0000 7efe3dbee740 1 -- 172.21.15.46:0/710923298 >> 172.21.15.46:0/710923298 conn(0x557057853fe0 msgr2=0x5570578543c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:35:42.036 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:42.031+0000 7efe3dbee740 1 -- 172.21.15.46:0/710923298 shutdown_connections 2024-03-31T23:35:42.036 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:42.031+0000 7efe3dbee740 1 -- 172.21.15.46:0/710923298 wait complete. 2024-03-31T23:35:42.036 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:42.032+0000 7efe3dbee740 1 -- start start 2024-03-31T23:35:42.037 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:42.032+0000 7efe3dbee740 1 Processor -- start 2024-03-31T23:35:42.037 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:42.032+0000 7efe3dbee740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5570578596c0 0x5570578735c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:35:42.037 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:42.032+0000 7efe3dbee740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557057859fe0 0x557057864990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:35:42.037 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:42.032+0000 7efe3dbee740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55705786b5c0 0x557057864ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:35:42.037 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:42.032+0000 7efe3dbee740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7efe341ab060 con 0x557057859fe0 2024-03-31T23:35:42.037 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:42.032+0000 7efe3dbee740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7efe341ac470 con 0x55705786b5c0 2024-03-31T23:35:42.037 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:42.032+0000 7efe3dbee740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7efe341ab6d0 con 0x5570578596c0 2024-03-31T23:35:42.037 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:42.032+0000 7efe3dbee740 10 client.0 _ll_register_callbacks cb 0x7efe3001c070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T23:35:42.037 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:42.033+0000 7efe38da3640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557057859fe0 0x557057864990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:35:42.037 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:42.033+0000 7efe38da3640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557057859fe0 0x557057864990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60370/0 (socket says 172.21.15.46:60370) 2024-03-31T23:35:42.037 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:42.033+0000 7efe38da3640 1 -- 172.21.15.46:0/2855266511 learned_addr learned my addr 172.21.15.46:0/2855266511 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:35:42.037 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:42.033+0000 7efe2bfff640 1 --2- 172.21.15.46:0/2855266511 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55705786b5c0 0x557057864ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:35:42.037 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:42.033+0000 7efe2bfff640 1 --2- 172.21.15.46:0/2855266511 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5570578596c0 0x5570578735c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:35:42.038 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:42.033+0000 7efe38da3640 1 -- 172.21.15.46:0/2855266511 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5570578596c0 msgr2=0x5570578735c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:35:42.038 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:42.033+0000 7efe38da3640 1 --2- 172.21.15.46:0/2855266511 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5570578596c0 0x5570578735c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:35:42.038 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:42.033+0000 7efe38da3640 1 -- 172.21.15.46:0/2855266511 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55705786b5c0 msgr2=0x557057864ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:35:42.038 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:42.033+0000 7efe38da3640 1 --2- 172.21.15.46:0/2855266511 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55705786b5c0 0x557057864ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:35:42.038 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:42.033+0000 7efe38da3640 1 -- 172.21.15.46:0/2855266511 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557057873bb0 con 0x557057859fe0 2024-03-31T23:35:42.038 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:42.033+0000 7efe2bfff640 1 --2- 172.21.15.46:0/2855266511 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55705786b5c0 0x557057864ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:35:42.038 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:42.033+0000 7efe38da3640 1 --2- 172.21.15.46:0/2855266511 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557057859fe0 0x557057864990 secure :-1 s=READY pgs=1038 cs=0 l=1 rev1=1 crypto rx=0x7efe30006010 tx=0x7efe30005d30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1398ec386b158796 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:35:42.038 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:42.033+0000 7efe13fff640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-31T23:35:42.038 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:42.033+0000 7efe13fff640 1 -- 172.21.15.46:0/2855266511 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe30036070 con 0x557057859fe0 2024-03-31T23:35:42.038 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:42.033+0000 7efe13fff640 1 -- 172.21.15.46:0/2855266511 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(135..135 src has 1..135) ==== 9554+0+0 (secure 0 0 0) 0x7efe3000f8c0 con 0x557057859fe0 2024-03-31T23:35:42.038 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:42.033+0000 7efe3dbee740 1 -- 172.21.15.46:0/2855266511 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efe240120c0 con 0x557057859fe0 2024-03-31T23:35:42.038 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:42.034+0000 7efe13fff640 1 -- 172.21.15.46:0/2855266511 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=54) ==== 24+0+0 (secure 0 0 0) 0x7efe3003f050 con 0x557057859fe0 2024-03-31T23:35:42.038 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:42.034+0000 7efe3dbee740 10 client.18090 fetch_fsmap learned FSMap version 54 2024-03-31T23:35:42.038 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:42.034+0000 7efe3dbee740 1 -- 172.21.15.46:0/2855266511 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=54,monmap=4+}) -- 0x5570578e16e0 con 0x557057859fe0 2024-03-31T23:35:42.039 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:42.034+0000 7efe13fff640 1 -- 172.21.15.46:0/2855266511 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7efe240120c0 con 0x557057859fe0 2024-03-31T23:35:42.039 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:42.034+0000 7efe13fff640 1 -- 172.21.15.46:0/2855266511 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 54) ==== 8346+0+0 (secure 0 0 0) 0x7efe30039020 con 0x557057859fe0 2024-03-31T23:35:42.039 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:42.034+0000 7efe3dbee740 10 client.18090 fetch_fsmap finished waiting for FSMap version 54 2024-03-31T23:35:42.039 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:42.034+0000 7efe3dbee740 10 client.18090 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-31T23:35:42.039 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:42.034+0000 7efe3dbee740 20 client.18090 populate_metadata read hostname 'smithi046' 2024-03-31T23:35:42.039 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:42.034+0000 7efe3dbee740 1 --2- 172.21.15.46:0/2855266511 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55705787b970 0x5570578e3f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:35:42.039 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:42.034+0000 7efe3dbee740 4 client.18090 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-31T23:35:42.039 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:42.034+0000 7efe3dbee740 1 -- 172.21.15.46:0/2855266511 --> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5570577c9e30 con 0x55705787b970 2024-03-31T23:35:42.039 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:42.034+0000 7efe2bfff640 1 --2- 172.21.15.46:0/2855266511 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55705787b970 0x5570578e3f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:35:42.039 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:42.035+0000 7efe13fff640 10 client.18090 ms_handle_connect on v2:172.21.15.142:6838/2519143918 2024-03-31T23:35:42.039 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:42.035+0000 7efe2bfff640 1 --2- 172.21.15.46:0/2855266511 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55705787b970 0x5570578e3f10 crc :-1 s=READY pgs=291 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=519e1fcc01fbef49 server_cookie=682db9d60351ecf7 in_seq=0 out_seq=0 2024-03-31T23:35:42.305 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:42.303+0000 7efe13fff640 1 -- 172.21.15.46:0/2855266511 <== mds.0 v2:172.21.15.142:6838/2519143918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5570577c9e30 con 0x55705787b970 2024-03-31T23:35:42.305 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:42.303+0000 7efe13fff640 10 client.18090 handle_command_reply: tid=0 2024-03-31T23:35:42.305 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:42.303+0000 7efe3dbee740 1 client.18090 shutdown 2024-03-31T23:35:42.305 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:42.303+0000 7efe3dbee740 1 -- 172.21.15.46:0/2855266511 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557057859fe0 msgr2=0x557057864990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:35:42.305 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:42.303+0000 7efe3dbee740 1 --2- 172.21.15.46:0/2855266511 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557057859fe0 0x557057864990 secure :-1 s=READY pgs=1038 cs=0 l=1 rev1=1 crypto rx=0x7efe30006010 tx=0x7efe30005d30 comp rx=0 tx=0).stop 2024-03-31T23:35:42.305 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:42.303+0000 7efe3dbee740 1 -- 172.21.15.46:0/2855266511 shutdown_connections 2024-03-31T23:35:42.305 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:42.303+0000 7efe3dbee740 1 --2- 172.21.15.46:0/2855266511 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x55705787b970 0x5570578e3f10 crc :-1 s=READY pgs=291 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:35:42.305 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:42.303+0000 7efe3dbee740 1 --2- 172.21.15.46:0/2855266511 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55705786b5c0 0x557057864ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:35:42.305 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:42.303+0000 7efe13fff640 0 client.18090 ms_handle_reset on v2:172.21.15.142:6838/2519143918 2024-03-31T23:35:42.306 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:42.303+0000 7efe3dbee740 1 --2- 172.21.15.46:0/2855266511 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557057859fe0 0x557057864990 unknown :-1 s=CLOSED pgs=1038 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:35:42.306 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:42.303+0000 7efe3dbee740 1 --2- 172.21.15.46:0/2855266511 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5570578596c0 0x5570578735c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:35:42.306 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:42.303+0000 7efe3dbee740 1 -- 172.21.15.46:0/2855266511 >> 172.21.15.46:0/2855266511 conn(0x557057853fe0 msgr2=0x5570578553c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:35:42.306 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:42.304+0000 7efe3dbee740 1 -- 172.21.15.46:0/2855266511 shutdown_connections 2024-03-31T23:35:42.306 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:42.304+0000 7efe3dbee740 1 -- 172.21.15.46:0/2855266511 wait complete. 2024-03-31T23:35:42.306 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:42.304+0000 7efe3dbee740 20 client.18090 trim_cache size 0 max 16384 2024-03-31T23:35:42.306 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout:{ 2024-03-31T23:35:42.306 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout: "message": "", 2024-03-31T23:35:42.306 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout: "return_code": 0 2024-03-31T23:35:42.306 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout:} 2024-03-31T23:35:42.307 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:42.304+0000 7efe3b007640 1 -- 172.21.15.46:0/658657595 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7efe1c0a27b0 msgr2=0x7efe1c0a4c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:35:42.307 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:42.304+0000 7efe3b007640 1 --2- 172.21.15.46:0/658657595 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7efe1c0a27b0 0x7efe1c0a4c70 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7efe2c0ee5b0 tx=0x7efe2c0f48d0 comp rx=0 tx=0).stop 2024-03-31T23:35:42.307 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:42.304+0000 7efe3b007640 1 -- 172.21.15.46:0/658657595 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7efe341055f0 msgr2=0x7efe3419e940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:35:42.307 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:42.304+0000 7efe3b007640 1 --2- 172.21.15.46:0/658657595 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7efe341055f0 0x7efe3419e940 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7efe2c02c700 tx=0x7efe2c035d60 comp rx=0 tx=0).stop 2024-03-31T23:35:42.307 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:42.305+0000 7efe3b007640 1 -- 172.21.15.46:0/658657595 shutdown_connections 2024-03-31T23:35:42.308 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:42.305+0000 7efe3b007640 1 --2- 172.21.15.46:0/658657595 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7efe1c0a27b0 0x7efe1c0a4c70 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:35:42.308 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:42.305+0000 7efe3b007640 1 --2- 172.21.15.46:0/658657595 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7efe341055f0 0x7efe3419e940 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:35:42.308 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:42.305+0000 7efe3b007640 1 --2- 172.21.15.46:0/658657595 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7efe34101260 0x7efe3419a5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:35:42.308 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:42.305+0000 7efe3b007640 1 --2- 172.21.15.46:0/658657595 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efe341008a0 0x7efe3419a080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:35:42.308 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:42.305+0000 7efe3b007640 1 -- 172.21.15.46:0/658657595 >> 172.21.15.46:0/658657595 conn(0x7efe340774a0 msgr2=0x7efe340ff360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:35:42.308 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:42.305+0000 7efe3b007640 1 -- 172.21.15.46:0/658657595 shutdown_connections 2024-03-31T23:35:42.308 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:42.305+0000 7efe3b007640 1 -- 172.21.15.46:0/658657595 wait complete. 2024-03-31T23:35:42.315 INFO:tasks.workunit.client.0.smithi046.stdout: 10 99671 8.18 MB/sec execute 527 sec latency 259.902 ms 2024-03-31T23:35:43.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:35:42 smithi162 ceph-mon[36237]: pgmap v761: 129 pgs: 129 active+clean; 1.3 GiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 5.0 MiB/s wr, 238 op/s 2024-03-31T23:35:43.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:35:42 smithi046 ceph-mon[32498]: pgmap v761: 129 pgs: 129 active+clean; 1.3 GiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 5.0 MiB/s wr, 238 op/s 2024-03-31T23:35:43.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:35:42 smithi142 ceph-mon[35271]: pgmap v761: 129 pgs: 129 active+clean; 1.3 GiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 5.0 MiB/s wr, 238 op/s 2024-03-31T23:35:43.316 INFO:tasks.workunit.client.0.smithi046.stdout: 10 99817 8.18 MB/sec execute 528 sec latency 93.476 ms 2024-03-31T23:35:44.316 INFO:tasks.workunit.client.0.smithi046.stdout: 10 99977 8.17 MB/sec execute 529 sec latency 118.709 ms 2024-03-31T23:35:45.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:35:44 smithi162 ceph-mon[36237]: pgmap v762: 129 pgs: 129 active+clean; 1.3 GiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 5.0 MiB/s wr, 238 op/s 2024-03-31T23:35:45.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:35:44 smithi046 ceph-mon[32498]: pgmap v762: 129 pgs: 129 active+clean; 1.3 GiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 5.0 MiB/s wr, 238 op/s 2024-03-31T23:35:45.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:35:44 smithi142 ceph-mon[35271]: pgmap v762: 129 pgs: 129 active+clean; 1.3 GiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 5.0 MiB/s wr, 238 op/s 2024-03-31T23:35:45.316 INFO:tasks.workunit.client.0.smithi046.stdout: 10 100126 8.17 MB/sec execute 530 sec latency 89.735 ms 2024-03-31T23:35:46.316 INFO:tasks.workunit.client.0.smithi046.stdout: 10 100268 8.16 MB/sec execute 531 sec latency 87.849 ms 2024-03-31T23:35:47.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:35:46 smithi162 ceph-mon[36237]: pgmap v763: 129 pgs: 129 active+clean; 1.3 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 7.1 MiB/s wr, 412 op/s 2024-03-31T23:35:47.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:35:46 smithi046 ceph-mon[32498]: pgmap v763: 129 pgs: 129 active+clean; 1.3 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 7.1 MiB/s wr, 412 op/s 2024-03-31T23:35:47.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:35:46 smithi142 ceph-mon[35271]: pgmap v763: 129 pgs: 129 active+clean; 1.3 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 7.1 MiB/s wr, 412 op/s 2024-03-31T23:35:47.316 INFO:tasks.workunit.client.0.smithi046.stdout: 10 100429 8.16 MB/sec execute 532 sec latency 75.404 ms 2024-03-31T23:35:48.316 INFO:tasks.workunit.client.0.smithi046.stdout: 10 100604 8.17 MB/sec execute 533 sec latency 112.850 ms 2024-03-31T23:35:49.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:35:48 smithi162 ceph-mon[36237]: pgmap v764: 129 pgs: 129 active+clean; 1.3 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 4.6 MiB/s wr, 299 op/s 2024-03-31T23:35:49.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:35:48 smithi046 ceph-mon[32498]: pgmap v764: 129 pgs: 129 active+clean; 1.3 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 4.6 MiB/s wr, 299 op/s 2024-03-31T23:35:49.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:35:48 smithi142 ceph-mon[35271]: pgmap v764: 129 pgs: 129 active+clean; 1.3 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 4.6 MiB/s wr, 299 op/s 2024-03-31T23:35:49.316 INFO:tasks.workunit.client.0.smithi046.stdout: 10 100794 8.18 MB/sec execute 534 sec latency 74.927 ms 2024-03-31T23:35:50.316 INFO:tasks.workunit.client.0.smithi046.stdout: 10 100969 8.18 MB/sec execute 535 sec latency 64.322 ms 2024-03-31T23:35:51.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:35:50 smithi162 ceph-mon[36237]: pgmap v765: 129 pgs: 129 active+clean; 1.3 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 6.9 MiB/s wr, 410 op/s 2024-03-31T23:35:51.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:35:50 smithi046 ceph-mon[32498]: pgmap v765: 129 pgs: 129 active+clean; 1.3 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 6.9 MiB/s wr, 410 op/s 2024-03-31T23:35:51.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:35:50 smithi142 ceph-mon[35271]: pgmap v765: 129 pgs: 129 active+clean; 1.3 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 6.9 MiB/s wr, 410 op/s 2024-03-31T23:35:51.316 INFO:tasks.workunit.client.0.smithi046.stdout: 10 101150 8.19 MB/sec execute 536 sec latency 76.196 ms 2024-03-31T23:35:52.316 INFO:tasks.workunit.client.0.smithi046.stdout: 10 101326 8.20 MB/sec execute 537 sec latency 75.862 ms 2024-03-31T23:35:53.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:35:52 smithi162 ceph-mon[36237]: pgmap v766: 129 pgs: 129 active+clean; 1.3 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 4.4 MiB/s wr, 285 op/s 2024-03-31T23:35:53.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:35:52 smithi046 ceph-mon[32498]: pgmap v766: 129 pgs: 129 active+clean; 1.3 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 4.4 MiB/s wr, 285 op/s 2024-03-31T23:35:53.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:35:52 smithi142 ceph-mon[35271]: pgmap v766: 129 pgs: 129 active+clean; 1.3 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 4.4 MiB/s wr, 285 op/s 2024-03-31T23:35:53.316 INFO:tasks.workunit.client.0.smithi046.stdout: 10 101479 8.20 MB/sec execute 538 sec latency 81.015 ms 2024-03-31T23:35:54.316 INFO:tasks.workunit.client.0.smithi046.stdout: 10 101600 8.20 MB/sec execute 539 sec latency 64.484 ms 2024-03-31T23:35:55.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:35:54 smithi162 ceph-mon[36237]: pgmap v767: 129 pgs: 129 active+clean; 1.3 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 4.4 MiB/s wr, 285 op/s 2024-03-31T23:35:55.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:35:54 smithi046 ceph-mon[32498]: pgmap v767: 129 pgs: 129 active+clean; 1.3 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 4.4 MiB/s wr, 285 op/s 2024-03-31T23:35:55.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:35:54 smithi142 ceph-mon[35271]: pgmap v767: 129 pgs: 129 active+clean; 1.3 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 4.4 MiB/s wr, 285 op/s 2024-03-31T23:35:55.316 INFO:tasks.workunit.client.0.smithi046.stdout: 10 101773 8.21 MB/sec execute 540 sec latency 90.994 ms 2024-03-31T23:35:55.435 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:55.435+0000 7f3100a6f640 1 -- 172.21.15.46:0/973783686 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f30fc107840 msgr2=0x7f30fc107c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:35:55.436 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:55.435+0000 7f3100a6f640 1 --2- 172.21.15.46:0/973783686 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f30fc107840 0x7f30fc107c20 secure :-1 s=READY pgs=1039 cs=0 l=1 rev1=1 crypto rx=0x7f30ec009900 tx=0x7f30ec02bc30 comp rx=0 tx=0).stop 2024-03-31T23:35:55.436 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:55.435+0000 7f3100a6f640 1 -- 172.21.15.46:0/973783686 shutdown_connections 2024-03-31T23:35:55.436 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:55.435+0000 7f3100a6f640 1 --2- 172.21.15.46:0/973783686 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f30fc069770 0x7f30fc10e5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:35:55.436 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:55.435+0000 7f3100a6f640 1 --2- 172.21.15.46:0/973783686 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f30fc068dd0 0x7f30fc069230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:35:55.436 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:55.435+0000 7f3100a6f640 1 --2- 172.21.15.46:0/973783686 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f30fc107840 0x7f30fc107c20 unknown :-1 s=CLOSED pgs=1039 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:35:55.436 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:55.435+0000 7f3100a6f640 1 -- 172.21.15.46:0/973783686 >> 172.21.15.46:0/973783686 conn(0x7f30fc0774a0 msgr2=0x7f30fc0778b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:35:55.436 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:55.435+0000 7f3100a6f640 1 -- 172.21.15.46:0/973783686 shutdown_connections 2024-03-31T23:35:55.436 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:55.435+0000 7f3100a6f640 1 -- 172.21.15.46:0/973783686 wait complete. 2024-03-31T23:35:55.439 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:55.436+0000 7f3100a6f640 1 Processor -- start 2024-03-31T23:35:55.439 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:55.436+0000 7f3100a6f640 1 -- start start 2024-03-31T23:35:55.439 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:55.436+0000 7f3100a6f640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f30fc068dd0 0x7f30fc197f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:35:55.439 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:55.436+0000 7f3100a6f640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f30fc069770 0x7f30fc198460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:35:55.439 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:55.436+0000 7f3100a6f640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f30fc107840 0x7f30fc19c7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:35:55.439 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:55.436+0000 7f3100a6f640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f30fc111ac0 con 0x7f30fc107840 2024-03-31T23:35:55.439 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:55.436+0000 7f3100a6f640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f30fc111940 con 0x7f30fc069770 2024-03-31T23:35:55.439 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:55.436+0000 7f3100a6f640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f30fc111c40 con 0x7f30fc068dd0 2024-03-31T23:35:55.439 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:55.437+0000 7f30fa59c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f30fc068dd0 0x7f30fc197f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:35:55.440 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:55.437+0000 7f30fad9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f30fc107840 0x7f30fc19c7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:35:55.440 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:55.437+0000 7f30fa59c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f30fc068dd0 0x7f30fc197f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.46:40246/0 (socket says 172.21.15.46:40246) 2024-03-31T23:35:55.440 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:55.437+0000 7f30fa59c640 1 -- 172.21.15.46:0/647004957 learned_addr learned my addr 172.21.15.46:0/647004957 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:35:55.440 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:55.437+0000 7f30f9d9b640 1 --2- 172.21.15.46:0/647004957 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f30fc069770 0x7f30fc198460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:35:55.440 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:55.437+0000 7f30fad9d640 1 -- 172.21.15.46:0/647004957 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f30fc068dd0 msgr2=0x7f30fc197f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:35:55.440 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:55.437+0000 7f30fad9d640 1 --2- 172.21.15.46:0/647004957 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f30fc068dd0 0x7f30fc197f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:35:55.440 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:55.437+0000 7f30fad9d640 1 -- 172.21.15.46:0/647004957 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f30fc069770 msgr2=0x7f30fc198460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:35:55.440 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:55.437+0000 7f30fad9d640 1 --2- 172.21.15.46:0/647004957 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f30fc069770 0x7f30fc198460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:35:55.440 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:55.437+0000 7f30fad9d640 1 -- 172.21.15.46:0/647004957 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f30fc19cf10 con 0x7f30fc107840 2024-03-31T23:35:55.440 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:55.437+0000 7f30fad9d640 1 --2- 172.21.15.46:0/647004957 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f30fc107840 0x7f30fc19c7e0 secure :-1 s=READY pgs=1040 cs=0 l=1 rev1=1 crypto rx=0x7f30f400ebc0 tx=0x7f30f400c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:35:55.441 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:55.437+0000 7f30eb7fe640 1 -- 172.21.15.46:0/647004957 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30f4003b70 con 0x7f30fc107840 2024-03-31T23:35:55.441 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:55.438+0000 7f30eb7fe640 1 -- 172.21.15.46:0/647004957 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f30f4003d10 con 0x7f30fc107840 2024-03-31T23:35:55.441 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:55.438+0000 7f30eb7fe640 1 -- 172.21.15.46:0/647004957 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30f4010670 con 0x7f30fc107840 2024-03-31T23:35:55.441 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:55.438+0000 7f3100a6f640 1 -- 172.21.15.46:0/647004957 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f30fc19d200 con 0x7f30fc107840 2024-03-31T23:35:55.441 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:55.438+0000 7f3100a6f640 1 -- 172.21.15.46:0/647004957 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f30fc1a4a90 con 0x7f30fc107840 2024-03-31T23:35:55.449 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:55.440+0000 7f30eb7fe640 1 -- 172.21.15.46:0/647004957 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 25) ==== 137451+0+0 (secure 0 0 0) 0x7f30f4004020 con 0x7f30fc107840 2024-03-31T23:35:55.450 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:55.441+0000 7f3103652740 1 Processor -- start 2024-03-31T23:35:55.450 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:55.441+0000 7f3103652740 1 -- start start 2024-03-31T23:35:55.450 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:55.441+0000 7f3103652740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x563f2c9dc6c0 0x563f2c9dcaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:35:55.450 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:55.441+0000 7f3103652740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563f2c9dcfe0 0x563f2c9e5870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:35:55.450 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:55.441+0000 7f3103652740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563f2c9e5db0 0x563f2c9e81a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:35:55.450 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:55.441+0000 7f3103652740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x563f2c93f8f0 con 0x563f2c9dcfe0 2024-03-31T23:35:55.450 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:55.441+0000 7f3103652740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x563f2c8e88d0 con 0x563f2c9dc6c0 2024-03-31T23:35:55.450 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:55.441+0000 7f3103652740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x563f2c93b990 con 0x563f2c9e5db0 2024-03-31T23:35:55.450 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:55.441+0000 7f30eb7fe640 1 --2- 172.21.15.46:0/647004957 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f30e40a26b0 0x7f30e40a4b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:35:55.451 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:55.441+0000 7f30f9d9b640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x563f2c9dc6c0 0x563f2c9dcaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:35:55.451 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:55.441+0000 7f30f9d9b640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x563f2c9dc6c0 0x563f2c9dcaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.46:58448/0 (socket says 172.21.15.46:58448) 2024-03-31T23:35:55.451 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:55.441+0000 7f30f9d9b640 1 -- 172.21.15.46:0/2252212239 learned_addr learned my addr 172.21.15.46:0/2252212239 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:35:55.451 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:55.441+0000 7f30f9d9b640 1 --2- 172.21.15.46:0/2252212239 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563f2c9e5db0 0x563f2c9e81a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:35:55.451 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:55.441+0000 7f30fad9d640 1 --2- 172.21.15.46:0/647004957 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f30e40a26b0 0x7f30e40a4b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:35:55.451 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:55.442+0000 7f30fa59c640 1 --2- 172.21.15.46:0/2252212239 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563f2c9dcfe0 0x563f2c9e5870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:35:55.451 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:55.442+0000 7f30eb7fe640 1 -- 172.21.15.46:0/647004957 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(135..135 src has 1..135) ==== 9554+0+0 (secure 0 0 0) 0x7f30f40cf3c0 con 0x7f30fc107840 2024-03-31T23:35:55.451 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:55.442+0000 7f30f9d9b640 1 -- 172.21.15.46:0/2252212239 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563f2c9e5db0 msgr2=0x563f2c9e81a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:35:55.451 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:55.442+0000 7f30f9d9b640 1 --2- 172.21.15.46:0/2252212239 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563f2c9e5db0 0x563f2c9e81a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:35:55.451 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:55.442+0000 7f30f9d9b640 1 -- 172.21.15.46:0/2252212239 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563f2c9dcfe0 msgr2=0x563f2c9e5870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:35:55.451 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:55.442+0000 7f30f9d9b640 1 --2- 172.21.15.46:0/2252212239 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563f2c9dcfe0 0x563f2c9e5870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:35:55.451 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:55.442+0000 7f30f9d9b640 1 -- 172.21.15.46:0/2252212239 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563f2c654210 con 0x563f2c9dc6c0 2024-03-31T23:35:55.452 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:55.442+0000 7f30fa59c640 1 --2- 172.21.15.46:0/2252212239 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563f2c9dcfe0 0x563f2c9e5870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T23:35:55.452 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:55.442+0000 7f30fad9d640 1 --2- 172.21.15.46:0/647004957 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f30e40a26b0 0x7f30e40a4b70 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f30f409b430 tx=0x7f30f406d6f0 comp rx=0 tx=0).ready entity=mgr.15654 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:35:55.452 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:55.442+0000 7f30f9d9b640 1 --2- 172.21.15.46:0/2252212239 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x563f2c9dc6c0 0x563f2c9dcaa0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f30f0009fd0 tx=0x7f30f0005850 comp rx=0 tx=0).ready entity=mon.2 client_cookie=952e8b01f099e17 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:35:55.452 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:55.442+0000 7f30e8ff9640 1 -- 172.21.15.46:0/2252212239 <== mon.2 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30f0004030 con 0x563f2c9dc6c0 2024-03-31T23:35:55.452 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:55.443+0000 7f30e8ff9640 1 -- 172.21.15.46:0/2252212239 <== mon.2 v2:172.21.15.142:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f30f0005020 con 0x563f2c9dc6c0 2024-03-31T23:35:55.452 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:55.443+0000 7f30e8ff9640 1 -- 172.21.15.46:0/2252212239 <== mon.2 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30f0004b20 con 0x563f2c9dc6c0 2024-03-31T23:35:55.452 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:55.443+0000 7f3103652740 1 -- 172.21.15.46:0/2252212239 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x563f2c9dc6c0 msgr2=0x563f2c9dcaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:35:55.452 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:55.443+0000 7f3103652740 1 --2- 172.21.15.46:0/2252212239 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x563f2c9dc6c0 0x563f2c9dcaa0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f30f0009fd0 tx=0x7f30f0005850 comp rx=0 tx=0).stop 2024-03-31T23:35:55.452 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:55.444+0000 7f3103652740 1 -- 172.21.15.46:0/2252212239 shutdown_connections 2024-03-31T23:35:55.452 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:55.444+0000 7f3103652740 1 --2- 172.21.15.46:0/2252212239 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563f2c9e5db0 0x563f2c9e81a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:35:55.452 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:55.444+0000 7f3103652740 1 --2- 172.21.15.46:0/2252212239 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563f2c9dcfe0 0x563f2c9e5870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:35:55.452 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:55.444+0000 7f3103652740 1 --2- 172.21.15.46:0/2252212239 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x563f2c9dc6c0 0x563f2c9dcaa0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:35:55.453 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:55.444+0000 7f3103652740 1 -- 172.21.15.46:0/2252212239 >> 172.21.15.46:0/2252212239 conn(0x563f2c9d6fe0 msgr2=0x563f2c9d73c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:35:55.453 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:55.444+0000 7f3103652740 1 -- 172.21.15.46:0/2252212239 shutdown_connections 2024-03-31T23:35:55.453 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:55.444+0000 7f3103652740 1 -- 172.21.15.46:0/2252212239 wait complete. 2024-03-31T23:35:55.453 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:55.444+0000 7f3103652740 1 -- start start 2024-03-31T23:35:55.453 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:55.444+0000 7f3103652740 1 Processor -- start 2024-03-31T23:35:55.453 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:55.445+0000 7f3103652740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563f2c9dc6c0 0x563f2c9e2e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:35:55.453 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:55.445+0000 7f3103652740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563f2c9dcfe0 0x563f2c9e3350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:35:55.453 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:55.445+0000 7f3103652740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x563f2c9e5db0 0x563f2c9e3890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:35:55.453 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:55.445+0000 7f3103652740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x563f2c9e89e0 con 0x563f2c9dc6c0 2024-03-31T23:35:55.453 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:55.445+0000 7f30f9d9b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563f2c9dc6c0 0x563f2c9e2e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:35:55.454 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:55.445+0000 7f30f9d9b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563f2c9dc6c0 0x563f2c9e2e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:42784/0 (socket says 172.21.15.46:42784) 2024-03-31T23:35:55.454 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:55.445+0000 7f30f9d9b640 1 -- 172.21.15.46:0/407020383 learned_addr learned my addr 172.21.15.46:0/407020383 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:35:55.454 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:55.445+0000 7f3103652740 1 -- 172.21.15.46:0/407020383 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x563f2c8ddf80 con 0x563f2c9e5db0 2024-03-31T23:35:55.454 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:55.445+0000 7f30fa59c640 1 --2- 172.21.15.46:0/407020383 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563f2c9dcfe0 0x563f2c9e3350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:35:55.454 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:55.445+0000 7f3103652740 1 -- 172.21.15.46:0/407020383 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x563f2c9e8b60 con 0x563f2c9dcfe0 2024-03-31T23:35:55.454 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:55.445+0000 7f30f9d9b640 1 --2- 172.21.15.46:0/407020383 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x563f2c9e5db0 0x563f2c9e3890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:35:55.454 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:55.445+0000 7f3103652740 10 client.0 _ll_register_callbacks cb 0x563f2c936290 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T23:35:55.454 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:55.445+0000 7f30f9d9b640 1 -- 172.21.15.46:0/407020383 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563f2c9dcfe0 msgr2=0x563f2c9e3350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:35:55.454 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:55.445+0000 7f30f9d9b640 1 --2- 172.21.15.46:0/407020383 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563f2c9dcfe0 0x563f2c9e3350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:35:55.454 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:55.445+0000 7f30f9d9b640 1 -- 172.21.15.46:0/407020383 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x563f2c9e5db0 msgr2=0x563f2c9e3890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:35:55.454 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:55.445+0000 7f30f9d9b640 1 --2- 172.21.15.46:0/407020383 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x563f2c9e5db0 0x563f2c9e3890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:35:55.455 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:55.445+0000 7f30f9d9b640 1 -- 172.21.15.46:0/407020383 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563f2c720620 con 0x563f2c9dc6c0 2024-03-31T23:35:55.455 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:55.446+0000 7f30f9d9b640 1 --2- 172.21.15.46:0/407020383 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563f2c9dc6c0 0x563f2c9e2e10 secure :-1 s=READY pgs=1041 cs=0 l=1 rev1=1 crypto rx=0x7f30f0009800 tx=0x7f30f0009d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a4a38da948ea877 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:35:55.455 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:55.446+0000 7f30e1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-31T23:35:55.455 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:55.446+0000 7f30e1ffb640 1 -- 172.21.15.46:0/407020383 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30f0028040 con 0x563f2c9dc6c0 2024-03-31T23:35:55.455 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:55.446+0000 7f30e1ffb640 1 -- 172.21.15.46:0/407020383 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(135..135 src has 1..135) ==== 9554+0+0 (secure 0 0 0) 0x7f30f0004040 con 0x563f2c9dc6c0 2024-03-31T23:35:55.455 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:55.446+0000 7f3103652740 1 -- 172.21.15.46:0/407020383 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f30f0005020 con 0x563f2c9dc6c0 2024-03-31T23:35:55.455 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:55.446+0000 7f30e1ffb640 1 -- 172.21.15.46:0/407020383 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=54) ==== 24+0+0 (secure 0 0 0) 0x7f30f0035050 con 0x563f2c9dc6c0 2024-03-31T23:35:55.455 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:55.446+0000 7f3103652740 10 client.18102 fetch_fsmap learned FSMap version 54 2024-03-31T23:35:55.455 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:55.446+0000 7f3103652740 1 -- 172.21.15.46:0/407020383 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=54,monmap=4+}) -- 0x563f2c9f48e0 con 0x563f2c9dc6c0 2024-03-31T23:35:55.455 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:55.446+0000 7f30e1ffb640 1 -- 172.21.15.46:0/407020383 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f30f0005020 con 0x563f2c9dc6c0 2024-03-31T23:35:55.455 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:55.446+0000 7f30e1ffb640 1 -- 172.21.15.46:0/407020383 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 54) ==== 8346+0+0 (secure 0 0 0) 0x7f30f000fb30 con 0x563f2c9dc6c0 2024-03-31T23:35:55.456 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:55.447+0000 7f3103652740 10 client.18102 fetch_fsmap finished waiting for FSMap version 54 2024-03-31T23:35:55.456 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:55.447+0000 7f3103652740 10 client.18102 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-31T23:35:55.456 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:55.447+0000 7f3103652740 20 client.18102 populate_metadata read hostname 'smithi046' 2024-03-31T23:35:55.456 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:55.447+0000 7f3103652740 1 --2- 172.21.15.46:0/407020383 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x563f2c9f5c30 0x563f2c9f8020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:35:55.456 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:55.447+0000 7f3103652740 4 client.18102 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-31T23:35:55.456 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:55.447+0000 7f3103652740 1 -- 172.21.15.46:0/407020383 --> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563f2c953120 con 0x563f2c9f5c30 2024-03-31T23:35:55.456 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:55.447+0000 7f30fa59c640 1 --2- 172.21.15.46:0/407020383 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x563f2c9f5c30 0x563f2c9f8020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:35:55.456 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:55.447+0000 7f30e1ffb640 10 client.18102 ms_handle_connect on v2:172.21.15.142:6838/2519143918 2024-03-31T23:35:55.456 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:55.447+0000 7f30fa59c640 1 --2- 172.21.15.46:0/407020383 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x563f2c9f5c30 0x563f2c9f8020 crc :-1 s=READY pgs=292 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=393accbe21c56892 server_cookie=a01b47f03db54ccb in_seq=0 out_seq=0 2024-03-31T23:35:55.456 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:55.448+0000 7f30e1ffb640 1 -- 172.21.15.46:0/407020383 <== mds.0 v2:172.21.15.142:6838/2519143918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x563f2c953120 con 0x563f2c9f5c30 2024-03-31T23:35:55.456 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:55.448+0000 7f30e1ffb640 10 client.18102 handle_command_reply: tid=0 2024-03-31T23:35:55.456 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:55.448+0000 7f3103652740 1 client.18102 shutdown 2024-03-31T23:35:55.457 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:55.448+0000 7f3103652740 1 -- 172.21.15.46:0/407020383 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563f2c9dc6c0 msgr2=0x563f2c9e2e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:35:55.457 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:55.448+0000 7f3103652740 1 --2- 172.21.15.46:0/407020383 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563f2c9dc6c0 0x563f2c9e2e10 secure :-1 s=READY pgs=1041 cs=0 l=1 rev1=1 crypto rx=0x7f30f0009800 tx=0x7f30f0009d10 comp rx=0 tx=0).stop 2024-03-31T23:35:55.457 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:55.448+0000 7f3103652740 1 -- 172.21.15.46:0/407020383 shutdown_connections 2024-03-31T23:35:55.457 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:55.448+0000 7f3103652740 1 --2- 172.21.15.46:0/407020383 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x563f2c9f5c30 0x563f2c9f8020 crc :-1 s=READY pgs=292 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:35:55.457 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:55.448+0000 7f30e1ffb640 0 client.18102 ms_handle_reset on v2:172.21.15.142:6838/2519143918 2024-03-31T23:35:55.457 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:55.448+0000 7f3103652740 1 --2- 172.21.15.46:0/407020383 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x563f2c9e5db0 0x563f2c9e3890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:35:55.457 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:55.448+0000 7f3103652740 1 --2- 172.21.15.46:0/407020383 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563f2c9dcfe0 0x563f2c9e3350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:35:55.457 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:55.448+0000 7f3103652740 1 --2- 172.21.15.46:0/407020383 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563f2c9dc6c0 0x563f2c9e2e10 unknown :-1 s=CLOSED pgs=1041 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:35:55.457 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:55.448+0000 7f3103652740 1 -- 172.21.15.46:0/407020383 >> 172.21.15.46:0/407020383 conn(0x563f2c9d6fe0 msgr2=0x563f2c9d8d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:35:55.457 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:55.448+0000 7f3103652740 1 -- 172.21.15.46:0/407020383 shutdown_connections 2024-03-31T23:35:55.457 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:55.449+0000 7f3103652740 1 -- 172.21.15.46:0/407020383 wait complete. 2024-03-31T23:35:55.458 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:55.449+0000 7f3103652740 20 client.18102 trim_cache size 0 max 16384 2024-03-31T23:35:55.473 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:55.471+0000 7f3103652740 1 Processor -- start 2024-03-31T23:35:55.473 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:55.471+0000 7f3103652740 1 -- start start 2024-03-31T23:35:55.473 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:55.471+0000 7f3103652740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x563f2c9dcfe0 0x563f2c9f1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:35:55.473 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:55.471+0000 7f3103652740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563f2c9e5db0 0x563f2c9fd7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:35:55.473 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:55.471+0000 7f3103652740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563f2c9fdd30 0x563f2c9e16e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:35:55.473 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:55.471+0000 7f3103652740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f30fc1b15d0 con 0x563f2c9e5db0 2024-03-31T23:35:55.473 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:55.471+0000 7f3103652740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f30fc1aeea0 con 0x563f2c9dcfe0 2024-03-31T23:35:55.473 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:55.471+0000 7f3103652740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f30fc1acd10 con 0x563f2c9fdd30 2024-03-31T23:35:55.473 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:55.472+0000 7f30fa59c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563f2c9e5db0 0x563f2c9fd7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:35:55.474 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:55.472+0000 7f30fa59c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563f2c9e5db0 0x563f2c9fd7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:42796/0 (socket says 172.21.15.46:42796) 2024-03-31T23:35:55.474 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:55.472+0000 7f30fa59c640 1 -- 172.21.15.46:0/4039535938 learned_addr learned my addr 172.21.15.46:0/4039535938 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:35:55.474 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:55.472+0000 7f30f9d9b640 1 --2- 172.21.15.46:0/4039535938 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563f2c9fdd30 0x563f2c9e16e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:35:55.474 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:55.472+0000 7f30f9d9b640 1 --2- 172.21.15.46:0/4039535938 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x563f2c9dcfe0 0x563f2c9f1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:35:55.474 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:55.472+0000 7f30fa59c640 1 -- 172.21.15.46:0/4039535938 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563f2c9fdd30 msgr2=0x563f2c9e16e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:35:55.474 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:55.472+0000 7f30fa59c640 1 --2- 172.21.15.46:0/4039535938 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563f2c9fdd30 0x563f2c9e16e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:35:55.474 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:55.472+0000 7f30fa59c640 1 -- 172.21.15.46:0/4039535938 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x563f2c9dcfe0 msgr2=0x563f2c9f1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:35:55.474 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:55.472+0000 7f30fa59c640 1 --2- 172.21.15.46:0/4039535938 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x563f2c9dcfe0 0x563f2c9f1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:35:55.474 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:55.472+0000 7f30fa59c640 1 -- 172.21.15.46:0/4039535938 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563f2c9d7d70 con 0x563f2c9e5db0 2024-03-31T23:35:55.474 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:55.472+0000 7f30f9d9b640 1 --2- 172.21.15.46:0/4039535938 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563f2c9fdd30 0x563f2c9e16e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T23:35:55.474 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:55.472+0000 7f30fa59c640 1 --2- 172.21.15.46:0/4039535938 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563f2c9e5db0 0x563f2c9fd7f0 secure :-1 s=READY pgs=1042 cs=0 l=1 rev1=1 crypto rx=0x7f30ec02c240 tx=0x7f30ec005a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=134ff5f87ffb98cb server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:35:55.474 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:55.472+0000 7f30e3fff640 1 -- 172.21.15.46:0/4039535938 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30ec04f070 con 0x563f2c9e5db0 2024-03-31T23:35:55.475 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:55.472+0000 7f30e3fff640 1 -- 172.21.15.46:0/4039535938 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f30ec004360 con 0x563f2c9e5db0 2024-03-31T23:35:55.475 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:55.472+0000 7f30e3fff640 1 -- 172.21.15.46:0/4039535938 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30ec02c870 con 0x563f2c9e5db0 2024-03-31T23:35:55.479 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:55.473+0000 7f3103652740 1 -- 172.21.15.46:0/4039535938 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563f2c9e5db0 msgr2=0x563f2c9fd7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:35:55.479 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:55.473+0000 7f3103652740 1 --2- 172.21.15.46:0/4039535938 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563f2c9e5db0 0x563f2c9fd7f0 secure :-1 s=READY pgs=1042 cs=0 l=1 rev1=1 crypto rx=0x7f30ec02c240 tx=0x7f30ec005a20 comp rx=0 tx=0).stop 2024-03-31T23:35:55.479 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:55.473+0000 7f3103652740 1 -- 172.21.15.46:0/4039535938 shutdown_connections 2024-03-31T23:35:55.479 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:55.473+0000 7f3103652740 1 --2- 172.21.15.46:0/4039535938 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563f2c9fdd30 0x563f2c9e16e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:35:55.479 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:55.473+0000 7f3103652740 1 --2- 172.21.15.46:0/4039535938 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563f2c9e5db0 0x563f2c9fd7f0 unknown :-1 s=CLOSED pgs=1042 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:35:55.479 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:55.473+0000 7f3103652740 1 --2- 172.21.15.46:0/4039535938 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x563f2c9dcfe0 0x563f2c9f1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:35:55.479 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:55.473+0000 7f3103652740 1 -- 172.21.15.46:0/4039535938 >> 172.21.15.46:0/4039535938 conn(0x563f2c9d6fe0 msgr2=0x563f2c9d73c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:35:55.479 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:55.473+0000 7f3103652740 1 -- 172.21.15.46:0/4039535938 shutdown_connections 2024-03-31T23:35:55.479 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:55.473+0000 7f3103652740 1 -- 172.21.15.46:0/4039535938 wait complete. 2024-03-31T23:35:55.479 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:55.474+0000 7f3103652740 1 -- start start 2024-03-31T23:35:55.480 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:55.474+0000 7f3103652740 1 Processor -- start 2024-03-31T23:35:55.480 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:55.474+0000 7f3103652740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x563f2c9dcfe0 0x563f2c9ee120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:35:55.480 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:55.474+0000 7f3103652740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563f2c9e5db0 0x563f2c9e9490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:35:55.480 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:55.474+0000 7f3103652740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563f2c9fdd30 0x563f2c9e9aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:35:55.480 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:55.474+0000 7f3103652740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f30fc1ad360 con 0x563f2c9fdd30 2024-03-31T23:35:55.480 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:55.474+0000 7f3103652740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f30fc1ae7d0 con 0x563f2c9dcfe0 2024-03-31T23:35:55.480 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:55.474+0000 7f3103652740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f30fc1ada30 con 0x563f2c9e5db0 2024-03-31T23:35:55.480 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:55.474+0000 7f30fa59c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563f2c9e5db0 0x563f2c9e9490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:35:55.480 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:55.474+0000 7f30fa59c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563f2c9e5db0 0x563f2c9e9490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.46:40300/0 (socket says 172.21.15.46:40300) 2024-03-31T23:35:55.480 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:55.474+0000 7f30fa59c640 1 -- 172.21.15.46:0/3568405965 learned_addr learned my addr 172.21.15.46:0/3568405965 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:35:55.480 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:55.474+0000 7f30f9d9b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563f2c9fdd30 0x563f2c9e9aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:35:55.480 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:55.475+0000 7f3103652740 10 client.0 _ll_register_callbacks cb 0x7f30f0009ea0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T23:35:55.481 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:55.475+0000 7f30f9d9b640 1 --2- 172.21.15.46:0/3568405965 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x563f2c9dcfe0 0x563f2c9ee120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:35:55.481 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:55.475+0000 7f30fa59c640 1 -- 172.21.15.46:0/3568405965 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x563f2c9dcfe0 msgr2=0x563f2c9ee120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:35:55.481 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:55.475+0000 7f30fa59c640 1 --2- 172.21.15.46:0/3568405965 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x563f2c9dcfe0 0x563f2c9ee120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:35:55.481 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:55.475+0000 7f30fa59c640 1 -- 172.21.15.46:0/3568405965 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563f2c9fdd30 msgr2=0x563f2c9e9aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:35:55.481 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:55.475+0000 7f30fa59c640 1 --2- 172.21.15.46:0/3568405965 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563f2c9fdd30 0x563f2c9e9aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:35:55.481 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:55.475+0000 7f30fa59c640 1 -- 172.21.15.46:0/3568405965 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563f2c9ea2a0 con 0x563f2c9e5db0 2024-03-31T23:35:55.481 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:55.475+0000 7f30fa59c640 1 --2- 172.21.15.46:0/3568405965 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563f2c9e5db0 0x563f2c9e9490 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f30ec04d040 tx=0x7f30ec004510 comp rx=0 tx=0).ready entity=mon.1 client_cookie=33d515e005caab37 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:35:55.481 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:55.475+0000 7f30e1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.162:3300/0 2024-03-31T23:35:55.481 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:55.475+0000 7f30e1ffb640 1 -- 172.21.15.46:0/3568405965 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30ec04f030 con 0x563f2c9e5db0 2024-03-31T23:35:55.481 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:55.475+0000 7f30e1ffb640 1 -- 172.21.15.46:0/3568405965 <== mon.1 v2:172.21.15.162:3300/0 2 ==== osd_map(135..135 src has 1..135) ==== 9554+0+0 (secure 0 0 0) 0x7f30ec047040 con 0x563f2c9e5db0 2024-03-31T23:35:55.481 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:55.475+0000 7f3103652740 1 -- 172.21.15.46:0/3568405965 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f30ec004360 con 0x563f2c9e5db0 2024-03-31T23:35:55.481 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:55.476+0000 7f30e1ffb640 1 -- 172.21.15.46:0/3568405965 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_get_version_reply(handle=1 version=54) ==== 24+0+0 (secure 0 0 0) 0x7f30ec04aab0 con 0x563f2c9e5db0 2024-03-31T23:35:55.482 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:55.476+0000 7f3103652740 10 client.27817 fetch_fsmap learned FSMap version 54 2024-03-31T23:35:55.482 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:55.476+0000 7f3103652740 1 -- 172.21.15.46:0/3568405965 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,fsmap=54,monmap=4+}) -- 0x563f2ca5bee0 con 0x563f2c9e5db0 2024-03-31T23:35:55.482 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:55.476+0000 7f30e1ffb640 1 -- 172.21.15.46:0/3568405965 <== mon.1 v2:172.21.15.162:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f30ec004360 con 0x563f2c9e5db0 2024-03-31T23:35:55.482 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:55.476+0000 7f30e1ffb640 1 -- 172.21.15.46:0/3568405965 <== mon.1 v2:172.21.15.162:3300/0 5 ==== fsmap(e 54) ==== 8346+0+0 (secure 0 0 0) 0x7f30ec005220 con 0x563f2c9e5db0 2024-03-31T23:35:55.482 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:55.476+0000 7f3103652740 10 client.27817 fetch_fsmap finished waiting for FSMap version 54 2024-03-31T23:35:55.482 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:55.476+0000 7f3103652740 10 client.27817 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-31T23:35:55.482 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:55.476+0000 7f3103652740 20 client.27817 populate_metadata read hostname 'smithi046' 2024-03-31T23:35:55.482 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:55.476+0000 7f3103652740 1 --2- 172.21.15.46:0/3568405965 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x563f2ca5c680 0x563f2ca5eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:35:55.482 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:55.476+0000 7f3103652740 4 client.27817 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-31T23:35:55.482 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:55.476+0000 7f3103652740 1 -- 172.21.15.46:0/3568405965 --> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] -- command(tid 0: {"prefix": "flush journal"}) -- 0x563f2c94ce30 con 0x563f2ca5c680 2024-03-31T23:35:55.482 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:55.476+0000 7f30f9d9b640 1 --2- 172.21.15.46:0/3568405965 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x563f2ca5c680 0x563f2ca5eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:35:55.483 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:55.477+0000 7f30e1ffb640 10 client.27817 ms_handle_connect on v2:172.21.15.142:6838/2519143918 2024-03-31T23:35:55.483 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:55.477+0000 7f30f9d9b640 1 --2- 172.21.15.46:0/3568405965 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x563f2ca5c680 0x563f2ca5eaa0 crc :-1 s=READY pgs=293 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7966a42c1dafff23 server_cookie=b3de22fbaa1f94e3 in_seq=0 out_seq=0 2024-03-31T23:35:55.805 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:55.805+0000 7f30e1ffb640 1 -- 172.21.15.46:0/3568405965 <== mds.0 v2:172.21.15.142:6838/2519143918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x563f2c94ce30 con 0x563f2ca5c680 2024-03-31T23:35:55.805 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:55.805+0000 7f30e1ffb640 10 client.27817 handle_command_reply: tid=0 2024-03-31T23:35:55.806 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:55.805+0000 7f3103652740 1 client.27817 shutdown 2024-03-31T23:35:55.806 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:55.805+0000 7f3103652740 1 -- 172.21.15.46:0/3568405965 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563f2c9e5db0 msgr2=0x563f2c9e9490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:35:55.806 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:55.805+0000 7f3103652740 1 --2- 172.21.15.46:0/3568405965 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563f2c9e5db0 0x563f2c9e9490 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f30ec04d040 tx=0x7f30ec004510 comp rx=0 tx=0).stop 2024-03-31T23:35:55.806 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:55.805+0000 7f3103652740 1 -- 172.21.15.46:0/3568405965 shutdown_connections 2024-03-31T23:35:55.806 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:55.805+0000 7f3103652740 1 --2- 172.21.15.46:0/3568405965 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x563f2ca5c680 0x563f2ca5eaa0 crc :-1 s=READY pgs=293 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:35:55.806 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:55.805+0000 7f3103652740 1 --2- 172.21.15.46:0/3568405965 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563f2c9fdd30 0x563f2c9e9aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:35:55.806 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:55.805+0000 7f30e1ffb640 0 client.27817 ms_handle_reset on v2:172.21.15.142:6838/2519143918 2024-03-31T23:35:55.806 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:55.805+0000 7f3103652740 1 --2- 172.21.15.46:0/3568405965 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563f2c9e5db0 0x563f2c9e9490 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:35:55.806 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:55.805+0000 7f3103652740 1 --2- 172.21.15.46:0/3568405965 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x563f2c9dcfe0 0x563f2c9ee120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:35:55.806 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:55.805+0000 7f3103652740 1 -- 172.21.15.46:0/3568405965 >> 172.21.15.46:0/3568405965 conn(0x563f2c9d6fe0 msgr2=0x563f2c9e16a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:35:55.806 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:55.805+0000 7f3103652740 1 -- 172.21.15.46:0/3568405965 shutdown_connections 2024-03-31T23:35:55.806 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:55.805+0000 7f3103652740 1 -- 172.21.15.46:0/3568405965 wait complete. 2024-03-31T23:35:55.806 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:55.805+0000 7f3103652740 20 client.27817 trim_cache size 0 max 16384 2024-03-31T23:35:55.807 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout:{ 2024-03-31T23:35:55.807 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout: "message": "", 2024-03-31T23:35:55.807 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout: "return_code": 0 2024-03-31T23:35:55.807 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout:} 2024-03-31T23:35:55.808 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:55.806+0000 7f3100a6f640 1 -- 172.21.15.46:0/647004957 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f30e40a26b0 msgr2=0x7f30e40a4b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:35:55.808 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:55.806+0000 7f3100a6f640 1 --2- 172.21.15.46:0/647004957 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f30e40a26b0 0x7f30e40a4b70 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f30f409b430 tx=0x7f30f406d6f0 comp rx=0 tx=0).stop 2024-03-31T23:35:55.808 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:55.806+0000 7f3100a6f640 1 -- 172.21.15.46:0/647004957 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f30fc107840 msgr2=0x7f30fc19c7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:35:55.808 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:55.806+0000 7f3100a6f640 1 --2- 172.21.15.46:0/647004957 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f30fc107840 0x7f30fc19c7e0 secure :-1 s=READY pgs=1040 cs=0 l=1 rev1=1 crypto rx=0x7f30f400ebc0 tx=0x7f30f400c6a0 comp rx=0 tx=0).stop 2024-03-31T23:35:55.808 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:55.806+0000 7f3100a6f640 1 -- 172.21.15.46:0/647004957 shutdown_connections 2024-03-31T23:35:55.808 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:55.806+0000 7f3100a6f640 1 --2- 172.21.15.46:0/647004957 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f30e40a26b0 0x7f30e40a4b70 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:35:55.808 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:55.806+0000 7f3100a6f640 1 --2- 172.21.15.46:0/647004957 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f30fc107840 0x7f30fc19c7e0 unknown :-1 s=CLOSED pgs=1040 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:35:55.808 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:55.806+0000 7f3100a6f640 1 --2- 172.21.15.46:0/647004957 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f30fc069770 0x7f30fc198460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:35:55.808 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:55.806+0000 7f3100a6f640 1 --2- 172.21.15.46:0/647004957 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f30fc068dd0 0x7f30fc197f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:35:55.808 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:55.806+0000 7f3100a6f640 1 -- 172.21.15.46:0/647004957 >> 172.21.15.46:0/647004957 conn(0x7f30fc0774a0 msgr2=0x7f30fc0fe8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:35:55.809 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:55.806+0000 7f3100a6f640 1 -- 172.21.15.46:0/647004957 shutdown_connections 2024-03-31T23:35:55.809 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:35:55.806+0000 7f3100a6f640 1 -- 172.21.15.46:0/647004957 wait complete. 2024-03-31T23:35:56.316 INFO:tasks.workunit.client.0.smithi046.stdout: 10 101881 8.20 MB/sec execute 541 sec latency 210.823 ms 2024-03-31T23:35:57.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:35:56 smithi162 ceph-mon[36237]: pgmap v768: 129 pgs: 129 active+clean; 1.3 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 7.1 MiB/s wr, 383 op/s 2024-03-31T23:35:57.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:35:56 smithi046 ceph-mon[32498]: pgmap v768: 129 pgs: 129 active+clean; 1.3 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 7.1 MiB/s wr, 383 op/s 2024-03-31T23:35:57.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:35:56 smithi142 ceph-mon[35271]: pgmap v768: 129 pgs: 129 active+clean; 1.3 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 7.1 MiB/s wr, 383 op/s 2024-03-31T23:35:57.316 INFO:tasks.workunit.client.0.smithi046.stdout: 10 102035 8.20 MB/sec execute 542 sec latency 94.178 ms 2024-03-31T23:35:58.316 INFO:tasks.workunit.client.0.smithi046.stdout: 10 102147 8.20 MB/sec execute 543 sec latency 87.722 ms 2024-03-31T23:35:59.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:35:58 smithi162 ceph-mon[36237]: pgmap v769: 129 pgs: 129 active+clean; 1.3 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 5.0 MiB/s wr, 209 op/s 2024-03-31T23:35:59.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:35:58 smithi046 ceph-mon[32498]: pgmap v769: 129 pgs: 129 active+clean; 1.3 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 5.0 MiB/s wr, 209 op/s 2024-03-31T23:35:59.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:35:58 smithi142 ceph-mon[35271]: pgmap v769: 129 pgs: 129 active+clean; 1.3 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 5.0 MiB/s wr, 209 op/s 2024-03-31T23:35:59.316 INFO:tasks.workunit.client.0.smithi046.stdout: 10 102295 8.21 MB/sec execute 544 sec latency 74.633 ms 2024-03-31T23:36:00.316 INFO:tasks.workunit.client.0.smithi046.stdout: 10 102443 8.20 MB/sec execute 545 sec latency 94.853 ms 2024-03-31T23:36:01.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:36:00 smithi162 ceph-mon[36237]: pgmap v770: 129 pgs: 129 active+clean; 1.3 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 7.9 MiB/s wr, 379 op/s 2024-03-31T23:36:01.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:36:00 smithi046 ceph-mon[32498]: pgmap v770: 129 pgs: 129 active+clean; 1.3 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 7.9 MiB/s wr, 379 op/s 2024-03-31T23:36:01.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:36:00 smithi142 ceph-mon[35271]: pgmap v770: 129 pgs: 129 active+clean; 1.3 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 7.9 MiB/s wr, 379 op/s 2024-03-31T23:36:01.316 INFO:tasks.workunit.client.0.smithi046.stdout: 10 102503 8.19 MB/sec execute 546 sec latency 449.983 ms 2024-03-31T23:36:02.318 INFO:tasks.workunit.client.0.smithi046.stdout: 10 102503 8.18 MB/sec execute 547 sec latency 1450.069 ms 2024-03-31T23:36:03.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:36:02 smithi162 ceph-mon[36237]: pgmap v771: 129 pgs: 129 active+clean; 1.3 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 5.6 MiB/s wr, 268 op/s 2024-03-31T23:36:03.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:36:02 smithi046 ceph-mon[32498]: pgmap v771: 129 pgs: 129 active+clean; 1.3 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 5.6 MiB/s wr, 268 op/s 2024-03-31T23:36:03.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:36:02 smithi142 ceph-mon[35271]: pgmap v771: 129 pgs: 129 active+clean; 1.3 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 5.6 MiB/s wr, 268 op/s 2024-03-31T23:36:03.317 INFO:tasks.workunit.client.0.smithi046.stdout: 10 102513 8.16 MB/sec execute 548 sec latency 2387.242 ms 2024-03-31T23:36:04.317 INFO:tasks.workunit.client.0.smithi046.stdout: 10 102519 8.15 MB/sec execute 549 sec latency 976.975 ms 2024-03-31T23:36:05.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:36:04 smithi162 ceph-mon[36237]: pgmap v772: 129 pgs: 129 active+clean; 1.3 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 5.6 MiB/s wr, 268 op/s 2024-03-31T23:36:05.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:36:04 smithi046 ceph-mon[32498]: pgmap v772: 129 pgs: 129 active+clean; 1.3 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 5.6 MiB/s wr, 268 op/s 2024-03-31T23:36:05.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:36:04 smithi142 ceph-mon[35271]: pgmap v772: 129 pgs: 129 active+clean; 1.3 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 5.6 MiB/s wr, 268 op/s 2024-03-31T23:36:05.317 INFO:tasks.workunit.client.0.smithi046.stdout: 10 102519 8.13 MB/sec execute 550 sec latency 1977.099 ms 2024-03-31T23:36:06.317 INFO:tasks.workunit.client.0.smithi046.stdout: 10 102587 8.12 MB/sec execute 551 sec latency 2963.795 ms 2024-03-31T23:36:07.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:36:06 smithi162 ceph-mon[36237]: pgmap v773: 129 pgs: 129 active+clean; 1.3 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 6.7 MiB/s wr, 344 op/s 2024-03-31T23:36:07.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:36:06 smithi046 ceph-mon[32498]: pgmap v773: 129 pgs: 129 active+clean; 1.3 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 6.7 MiB/s wr, 344 op/s 2024-03-31T23:36:07.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:36:06 smithi142 ceph-mon[35271]: pgmap v773: 129 pgs: 129 active+clean; 1.3 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 6.7 MiB/s wr, 344 op/s 2024-03-31T23:36:07.317 INFO:tasks.workunit.client.0.smithi046.stdout: 10 102738 8.12 MB/sec execute 552 sec latency 3963.857 ms 2024-03-31T23:36:08.317 INFO:tasks.workunit.client.0.smithi046.stdout: 10 102838 8.11 MB/sec execute 553 sec latency 4963.890 ms 2024-03-31T23:36:08.931 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:08.930+0000 7f0795556640 1 -- 172.21.15.46:0/3663175968 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f07901008a0 msgr2=0x7f0790100d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:36:08.931 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:08.930+0000 7f0795556640 1 --2- 172.21.15.46:0/3663175968 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f07901008a0 0x7f0790100d20 secure :-1 s=READY pgs=1043 cs=0 l=1 rev1=1 crypto rx=0x7f0784009900 tx=0x7f078402bc30 comp rx=0 tx=0).stop 2024-03-31T23:36:08.931 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:08.930+0000 7f0795556640 1 -- 172.21.15.46:0/3663175968 shutdown_connections 2024-03-31T23:36:08.931 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:08.930+0000 7f0795556640 1 --2- 172.21.15.46:0/3663175968 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0790101260 0x7f079010d480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:36:08.931 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:08.930+0000 7f0795556640 1 --2- 172.21.15.46:0/3663175968 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f07901008a0 0x7f0790100d20 unknown :-1 s=CLOSED pgs=1043 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:36:08.932 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:08.930+0000 7f0795556640 1 --2- 172.21.15.46:0/3663175968 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f07901055f0 0x7f07901059d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:36:08.932 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:08.930+0000 7f0795556640 1 -- 172.21.15.46:0/3663175968 >> 172.21.15.46:0/3663175968 conn(0x7f07900774a0 msgr2=0x7f07900778b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:36:08.932 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:08.930+0000 7f0795556640 1 -- 172.21.15.46:0/3663175968 shutdown_connections 2024-03-31T23:36:08.932 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:08.930+0000 7f0795556640 1 -- 172.21.15.46:0/3663175968 wait complete. 2024-03-31T23:36:08.934 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:08.930+0000 7f0795556640 1 Processor -- start 2024-03-31T23:36:08.934 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:08.931+0000 7f0795556640 1 -- start start 2024-03-31T23:36:08.934 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:08.931+0000 7f0795556640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f07901008a0 0x7f0790197f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:36:08.934 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:08.931+0000 7f0795556640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0790101260 0x7f0790198490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:36:08.934 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:08.931+0000 7f0795556640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f07901055f0 0x7f079019c810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:36:08.934 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:08.931+0000 7f0795556640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f0790110a10 con 0x7f07901055f0 2024-03-31T23:36:08.934 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:08.931+0000 7f0795556640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f0790110890 con 0x7f07901008a0 2024-03-31T23:36:08.934 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:08.931+0000 7f0795556640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f0790110b90 con 0x7f0790101260 2024-03-31T23:36:08.934 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:08.931+0000 7f078e7fc640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0790101260 0x7f0790198490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:36:08.934 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:08.931+0000 7f078e7fc640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0790101260 0x7f0790198490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.46:52740/0 (socket says 172.21.15.46:52740) 2024-03-31T23:36:08.935 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:08.931+0000 7f078e7fc640 1 -- 172.21.15.46:0/1101444690 learned_addr learned my addr 172.21.15.46:0/1101444690 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:36:08.935 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:08.931+0000 7f078effd640 1 --2- 172.21.15.46:0/1101444690 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f07901008a0 0x7f0790197f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:36:08.935 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:08.932+0000 7f078f7fe640 1 --2- 172.21.15.46:0/1101444690 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f07901055f0 0x7f079019c810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:36:08.935 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:08.932+0000 7f078e7fc640 1 -- 172.21.15.46:0/1101444690 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f07901008a0 msgr2=0x7f0790197f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:36:08.935 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:08.932+0000 7f078e7fc640 1 --2- 172.21.15.46:0/1101444690 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f07901008a0 0x7f0790197f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:36:08.935 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:08.932+0000 7f078e7fc640 1 -- 172.21.15.46:0/1101444690 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f07901055f0 msgr2=0x7f079019c810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:36:08.935 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:08.932+0000 7f078e7fc640 1 --2- 172.21.15.46:0/1101444690 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f07901055f0 0x7f079019c810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:36:08.935 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:08.932+0000 7f078e7fc640 1 -- 172.21.15.46:0/1101444690 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f079019cf40 con 0x7f0790101260 2024-03-31T23:36:08.935 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:08.932+0000 7f078e7fc640 1 --2- 172.21.15.46:0/1101444690 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0790101260 0x7f0790198490 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f0784000c00 tx=0x7f0784005dd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:36:08.935 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:08.932+0000 7f0777fff640 1 -- 172.21.15.46:0/1101444690 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f078402cb50 con 0x7f0790101260 2024-03-31T23:36:08.935 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:08.932+0000 7f0795556640 1 -- 172.21.15.46:0/1101444690 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f079019d1d0 con 0x7f0790101260 2024-03-31T23:36:08.935 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:08.932+0000 7f0795556640 1 -- 172.21.15.46:0/1101444690 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f07901021f0 con 0x7f0790101260 2024-03-31T23:36:08.935 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:08.932+0000 7f0777fff640 1 -- 172.21.15.46:0/1101444690 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f078403eb00 con 0x7f0790101260 2024-03-31T23:36:08.936 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:08.932+0000 7f0777fff640 1 -- 172.21.15.46:0/1101444690 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07840054b0 con 0x7f0790101260 2024-03-31T23:36:08.937 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:08.936+0000 7f0798214740 1 Processor -- start 2024-03-31T23:36:08.937 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:08.936+0000 7f0777fff640 1 -- 172.21.15.46:0/1101444690 <== mon.1 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 25) ==== 137451+0+0 (secure 0 0 0) 0x7f078403ecb0 con 0x7f0790101260 2024-03-31T23:36:08.937 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:08.936+0000 7f0798214740 1 -- start start 2024-03-31T23:36:08.938 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:08.937+0000 7f0798214740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558fe691ded0 0x558fe691e2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:36:08.939 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:08.937+0000 7f0798214740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558fe691e7f0 0x558fe6927080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:36:08.939 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:08.937+0000 7f0798214740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558fe69275c0 0x558fe69299b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:36:08.939 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:08.937+0000 7f0777fff640 1 --2- 172.21.15.46:0/1101444690 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f07780a2730 0x7f07780a4bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:36:08.939 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:08.937+0000 7f078e7fc640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558fe69275c0 0x558fe69299b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:36:08.939 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:08.937+0000 7f078effd640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558fe691e7f0 0x558fe6927080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:36:08.939 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:08.937+0000 7f078e7fc640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558fe69275c0 0x558fe69299b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39512/0 (socket says 172.21.15.46:39512) 2024-03-31T23:36:08.939 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:08.937+0000 7f078e7fc640 1 -- 172.21.15.46:0/2071229801 learned_addr learned my addr 172.21.15.46:0/2071229801 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:36:08.939 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:08.937+0000 7f0798214740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x558fe68788f0 con 0x558fe69275c0 2024-03-31T23:36:08.939 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:08.937+0000 7f0798214740 1 -- 172.21.15.46:0/2071229801 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x558fe68218d0 con 0x558fe691e7f0 2024-03-31T23:36:08.940 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:08.937+0000 7f0798214740 1 -- 172.21.15.46:0/2071229801 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x558fe6874990 con 0x558fe691ded0 2024-03-31T23:36:08.940 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:08.937+0000 7f078f7fe640 1 --2- 172.21.15.46:0/1101444690 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f07780a2730 0x7f07780a4bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:36:08.944 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:08.938+0000 7f0777fff640 1 -- 172.21.15.46:0/1101444690 <== mon.1 v2:172.21.15.162:3300/0 5 ==== osd_map(136..136 src has 1..136) ==== 9642+0+0 (secure 0 0 0) 0x7f07840f07b0 con 0x7f0790101260 2024-03-31T23:36:08.944 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:08.938+0000 7f078e7fc640 1 -- 172.21.15.46:0/2071229801 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558fe691ded0 msgr2=0x558fe691e2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:36:08.944 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:08.938+0000 7f078e7fc640 1 --2- 172.21.15.46:0/2071229801 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558fe691ded0 0x558fe691e2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:36:08.944 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:08.938+0000 7f078e7fc640 1 -- 172.21.15.46:0/2071229801 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558fe691e7f0 msgr2=0x558fe6927080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:36:08.944 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:08.938+0000 7f078e7fc640 1 --2- 172.21.15.46:0/2071229801 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558fe691e7f0 0x558fe6927080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:36:08.944 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:08.938+0000 7f078e7fc640 1 -- 172.21.15.46:0/2071229801 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558fe658d210 con 0x558fe69275c0 2024-03-31T23:36:08.944 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:08.938+0000 7f078f7fe640 1 --2- 172.21.15.46:0/1101444690 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f07780a2730 0x7f07780a4bf0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f0788005fd0 tx=0x7f07880098e0 comp rx=0 tx=0).ready entity=mgr.15654 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:36:08.944 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:08.938+0000 7f078e7fc640 1 --2- 172.21.15.46:0/2071229801 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558fe69275c0 0x558fe69299b0 secure :-1 s=READY pgs=1044 cs=0 l=1 rev1=1 crypto rx=0x7f07840ee0a0 tx=0x7f0784009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a2f7161d8be5607 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:36:08.944 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:08.939+0000 7f07757fa640 1 -- 172.21.15.46:0/2071229801 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f078403a240 con 0x558fe69275c0 2024-03-31T23:36:08.944 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:08.939+0000 7f07757fa640 1 -- 172.21.15.46:0/2071229801 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f07840c16f0 con 0x558fe69275c0 2024-03-31T23:36:08.944 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:08.939+0000 7f07757fa640 1 -- 172.21.15.46:0/2071229801 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07840bd660 con 0x558fe69275c0 2024-03-31T23:36:08.945 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:08.940+0000 7f0798214740 1 -- 172.21.15.46:0/2071229801 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558fe69275c0 msgr2=0x558fe69299b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:36:08.945 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:08.940+0000 7f0798214740 1 --2- 172.21.15.46:0/2071229801 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558fe69275c0 0x558fe69299b0 secure :-1 s=READY pgs=1044 cs=0 l=1 rev1=1 crypto rx=0x7f07840ee0a0 tx=0x7f0784009260 comp rx=0 tx=0).stop 2024-03-31T23:36:08.945 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:08.940+0000 7f0798214740 1 -- 172.21.15.46:0/2071229801 shutdown_connections 2024-03-31T23:36:08.945 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:08.940+0000 7f0798214740 1 --2- 172.21.15.46:0/2071229801 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558fe69275c0 0x558fe69299b0 unknown :-1 s=CLOSED pgs=1044 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:36:08.945 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:08.940+0000 7f0798214740 1 --2- 172.21.15.46:0/2071229801 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558fe691e7f0 0x558fe6927080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:36:08.945 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:08.940+0000 7f0798214740 1 --2- 172.21.15.46:0/2071229801 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558fe691ded0 0x558fe691e2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:36:08.945 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:08.940+0000 7f0798214740 1 -- 172.21.15.46:0/2071229801 >> 172.21.15.46:0/2071229801 conn(0x558fe69187f0 msgr2=0x558fe6918bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:36:08.945 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:08.940+0000 7f0798214740 1 -- 172.21.15.46:0/2071229801 shutdown_connections 2024-03-31T23:36:08.945 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:08.940+0000 7f0798214740 1 -- 172.21.15.46:0/2071229801 wait complete. 2024-03-31T23:36:08.945 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:08.940+0000 7f0798214740 1 -- start start 2024-03-31T23:36:08.945 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:08.940+0000 7f0798214740 1 Processor -- start 2024-03-31T23:36:08.945 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:08.941+0000 7f0798214740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558fe691ded0 0x558fe69245c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:36:08.945 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:08.941+0000 7f0798214740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558fe691e7f0 0x558fe6924b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:36:08.945 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:08.941+0000 7f0798214740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558fe69275c0 0x558fe6925040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:36:08.946 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:08.941+0000 7f0798214740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x558fe6929fb0 con 0x558fe69275c0 2024-03-31T23:36:08.946 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:08.941+0000 7f0798214740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x558fe6816f80 con 0x558fe691e7f0 2024-03-31T23:36:08.946 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:08.941+0000 7f0798214740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x558fe692a130 con 0x558fe691ded0 2024-03-31T23:36:08.946 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:08.941+0000 7f078effd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558fe691ded0 0x558fe69245c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:36:08.946 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:08.941+0000 7f0798214740 10 client.0 _ll_register_callbacks cb 0x558fe686f290 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T23:36:08.946 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:08.941+0000 7f078effd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558fe691ded0 0x558fe69245c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.46:52746/0 (socket says 172.21.15.46:52746) 2024-03-31T23:36:08.946 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:08.941+0000 7f078effd640 1 -- 172.21.15.46:0/3416491053 learned_addr learned my addr 172.21.15.46:0/3416491053 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:36:08.946 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:08.941+0000 7f078e7fc640 1 --2- 172.21.15.46:0/3416491053 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558fe691e7f0 0x558fe6924b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:36:08.946 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:08.941+0000 7f078f7fe640 1 --2- 172.21.15.46:0/3416491053 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558fe69275c0 0x558fe6925040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:36:08.946 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:08.941+0000 7f078effd640 1 -- 172.21.15.46:0/3416491053 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558fe691e7f0 msgr2=0x558fe6924b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:36:08.946 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:08.941+0000 7f078effd640 1 --2- 172.21.15.46:0/3416491053 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558fe691e7f0 0x558fe6924b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:36:08.946 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:08.941+0000 7f078effd640 1 -- 172.21.15.46:0/3416491053 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558fe69275c0 msgr2=0x558fe6925040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:36:08.946 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:08.941+0000 7f078effd640 1 --2- 172.21.15.46:0/3416491053 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558fe69275c0 0x558fe6925040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:36:08.947 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:08.941+0000 7f078effd640 1 -- 172.21.15.46:0/3416491053 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558fe6659620 con 0x558fe691ded0 2024-03-31T23:36:08.947 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:08.941+0000 7f078e7fc640 1 --2- 172.21.15.46:0/3416491053 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558fe691e7f0 0x558fe6924b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:36:08.947 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:08.942+0000 7f078effd640 1 --2- 172.21.15.46:0/3416491053 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558fe691ded0 0x558fe69245c0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f0780012740 tx=0x7f0780012c00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=62b007f51af6594b server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:36:08.947 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:08.942+0000 7f0772ff5640 10 client.0 ms_handle_connect on v2:172.21.15.162:3300/0 2024-03-31T23:36:08.947 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:08.942+0000 7f0772ff5640 1 -- 172.21.15.46:0/3416491053 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f078000f920 con 0x558fe691ded0 2024-03-31T23:36:08.947 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:08.942+0000 7f0772ff5640 1 -- 172.21.15.46:0/3416491053 <== mon.1 v2:172.21.15.162:3300/0 2 ==== osd_map(136..136 src has 1..136) ==== 9642+0+0 (secure 0 0 0) 0x7f078000fac0 con 0x558fe691ded0 2024-03-31T23:36:08.947 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:08.942+0000 7f0798214740 1 -- 172.21.15.46:0/3416491053 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f07840c16f0 con 0x558fe691ded0 2024-03-31T23:36:08.947 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:08.942+0000 7f0772ff5640 1 -- 172.21.15.46:0/3416491053 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_get_version_reply(handle=1 version=54) ==== 24+0+0 (secure 0 0 0) 0x7f078000fd00 con 0x558fe691ded0 2024-03-31T23:36:08.947 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:08.942+0000 7f0798214740 10 client.27826 fetch_fsmap learned FSMap version 54 2024-03-31T23:36:08.947 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:08.942+0000 7f0798214740 1 -- 172.21.15.46:0/3416491053 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,fsmap=54,monmap=4+}) -- 0x558fe6935e50 con 0x558fe691ded0 2024-03-31T23:36:08.947 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:08.942+0000 7f0772ff5640 1 -- 172.21.15.46:0/3416491053 <== mon.1 v2:172.21.15.162:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f07840c16f0 con 0x558fe691ded0 2024-03-31T23:36:08.947 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:08.943+0000 7f0772ff5640 1 -- 172.21.15.46:0/3416491053 <== mon.1 v2:172.21.15.162:3300/0 5 ==== fsmap(e 54) ==== 8346+0+0 (secure 0 0 0) 0x7f078001c420 con 0x558fe691ded0 2024-03-31T23:36:08.947 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:08.943+0000 7f0798214740 10 client.27826 fetch_fsmap finished waiting for FSMap version 54 2024-03-31T23:36:08.947 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:08.943+0000 7f0798214740 10 client.27826 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-31T23:36:08.948 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:08.943+0000 7f0798214740 20 client.27826 populate_metadata read hostname 'smithi046' 2024-03-31T23:36:08.948 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:08.943+0000 7f0798214740 1 --2- 172.21.15.46:0/3416491053 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x558fe6937550 0x558fe6939940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:36:08.948 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:08.943+0000 7f0798214740 4 client.27826 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-31T23:36:08.948 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:08.943+0000 7f0798214740 1 -- 172.21.15.46:0/3416491053 --> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558fe688c120 con 0x558fe6937550 2024-03-31T23:36:08.948 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:08.943+0000 7f078e7fc640 1 --2- 172.21.15.46:0/3416491053 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x558fe6937550 0x558fe6939940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:36:08.950 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:08.943+0000 7f078e7fc640 1 --2- 172.21.15.46:0/3416491053 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x558fe6937550 0x558fe6939940 crc :-1 s=READY pgs=294 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ffc4a8d59142285a server_cookie=928dc9ba6b392148 in_seq=0 out_seq=0 2024-03-31T23:36:08.950 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:08.943+0000 7f0772ff5640 10 client.27826 ms_handle_connect on v2:172.21.15.142:6838/2519143918 2024-03-31T23:36:08.950 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:08.945+0000 7f0772ff5640 1 -- 172.21.15.46:0/3416491053 <== mds.0 v2:172.21.15.142:6838/2519143918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x558fe688c120 con 0x558fe6937550 2024-03-31T23:36:08.950 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:08.945+0000 7f0772ff5640 10 client.27826 handle_command_reply: tid=0 2024-03-31T23:36:08.950 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:08.945+0000 7f0798214740 1 client.27826 shutdown 2024-03-31T23:36:08.950 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:08.945+0000 7f0798214740 1 -- 172.21.15.46:0/3416491053 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558fe691ded0 msgr2=0x558fe69245c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:36:08.950 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:08.945+0000 7f0798214740 1 --2- 172.21.15.46:0/3416491053 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558fe691ded0 0x558fe69245c0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f0780012740 tx=0x7f0780012c00 comp rx=0 tx=0).stop 2024-03-31T23:36:08.950 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:08.945+0000 7f0798214740 1 -- 172.21.15.46:0/3416491053 shutdown_connections 2024-03-31T23:36:08.950 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:08.945+0000 7f0798214740 1 --2- 172.21.15.46:0/3416491053 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x558fe6937550 0x558fe6939940 crc :-1 s=READY pgs=294 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:36:08.950 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:08.945+0000 7f0798214740 1 --2- 172.21.15.46:0/3416491053 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558fe69275c0 0x558fe6925040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:36:08.951 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:08.945+0000 7f0798214740 1 --2- 172.21.15.46:0/3416491053 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558fe691e7f0 0x558fe6924b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:36:08.951 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:08.945+0000 7f0772ff5640 0 client.27826 ms_handle_reset on v2:172.21.15.142:6838/2519143918 2024-03-31T23:36:08.951 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:08.945+0000 7f0798214740 1 --2- 172.21.15.46:0/3416491053 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558fe691ded0 0x558fe69245c0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:36:08.951 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:08.945+0000 7f0798214740 1 -- 172.21.15.46:0/3416491053 >> 172.21.15.46:0/3416491053 conn(0x558fe69187f0 msgr2=0x558fe691a540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:36:08.951 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:08.945+0000 7f0798214740 1 -- 172.21.15.46:0/3416491053 shutdown_connections 2024-03-31T23:36:08.951 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:08.945+0000 7f0798214740 1 -- 172.21.15.46:0/3416491053 wait complete. 2024-03-31T23:36:08.951 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:08.945+0000 7f0798214740 20 client.27826 trim_cache size 0 max 16384 2024-03-31T23:36:08.971 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:08.969+0000 7f0798214740 1 Processor -- start 2024-03-31T23:36:08.971 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:08.969+0000 7f0798214740 1 -- start start 2024-03-31T23:36:08.971 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:08.969+0000 7f0798214740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558fe691ded0 0x558fe6926cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:36:08.971 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:08.969+0000 7f0798214740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558fe691e7f0 0x558fe690ae20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:36:08.971 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:08.969+0000 7f0798214740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558fe69275c0 0x558fe6932c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:36:08.971 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:08.969+0000 7f0798214740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f077c012a70 con 0x558fe691e7f0 2024-03-31T23:36:08.971 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:08.969+0000 7f0798214740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f077c010340 con 0x558fe691ded0 2024-03-31T23:36:08.972 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:08.969+0000 7f0798214740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f077c00e1b0 con 0x558fe69275c0 2024-03-31T23:36:08.972 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:08.969+0000 7f078e7fc640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558fe691e7f0 0x558fe690ae20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:36:08.972 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:08.969+0000 7f078e7fc640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558fe691e7f0 0x558fe690ae20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39526/0 (socket says 172.21.15.46:39526) 2024-03-31T23:36:08.972 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:08.969+0000 7f078e7fc640 1 -- 172.21.15.46:0/3461093759 learned_addr learned my addr 172.21.15.46:0/3461093759 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:36:08.972 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:08.969+0000 7f078effd640 1 --2- 172.21.15.46:0/3461093759 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558fe691ded0 0x558fe6926cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:36:08.972 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:08.969+0000 7f078f7fe640 1 --2- 172.21.15.46:0/3461093759 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558fe69275c0 0x558fe6932c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:36:08.972 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:08.969+0000 7f078e7fc640 1 -- 172.21.15.46:0/3461093759 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558fe69275c0 msgr2=0x558fe6932c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:36:08.972 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:08.969+0000 7f078e7fc640 1 --2- 172.21.15.46:0/3461093759 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558fe69275c0 0x558fe6932c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:36:08.972 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:08.969+0000 7f078e7fc640 1 -- 172.21.15.46:0/3461093759 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558fe691ded0 msgr2=0x558fe6926cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:36:08.972 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:08.969+0000 7f078e7fc640 1 --2- 172.21.15.46:0/3461093759 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558fe691ded0 0x558fe6926cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:36:08.972 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:08.969+0000 7f078e7fc640 1 -- 172.21.15.46:0/3461093759 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558fe69331a0 con 0x558fe691e7f0 2024-03-31T23:36:08.972 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:08.970+0000 7f078e7fc640 1 --2- 172.21.15.46:0/3461093759 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558fe691e7f0 0x558fe690ae20 secure :-1 s=READY pgs=1045 cs=0 l=1 rev1=1 crypto rx=0x7f07840ee0a0 tx=0x7f0784040040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b8948dc95f81e345 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:36:08.972 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:08.970+0000 7f0774ff9640 1 -- 172.21.15.46:0/3461093759 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07840c0dc0 con 0x558fe691e7f0 2024-03-31T23:36:08.972 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:08.970+0000 7f0774ff9640 1 -- 172.21.15.46:0/3461093759 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f07840c5140 con 0x558fe691e7f0 2024-03-31T23:36:08.973 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:08.970+0000 7f0774ff9640 1 -- 172.21.15.46:0/3461093759 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07840bd790 con 0x558fe691e7f0 2024-03-31T23:36:08.976 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:08.971+0000 7f0798214740 1 -- 172.21.15.46:0/3461093759 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558fe691e7f0 msgr2=0x558fe690ae20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:36:08.977 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:08.971+0000 7f0798214740 1 --2- 172.21.15.46:0/3461093759 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558fe691e7f0 0x558fe690ae20 secure :-1 s=READY pgs=1045 cs=0 l=1 rev1=1 crypto rx=0x7f07840ee0a0 tx=0x7f0784040040 comp rx=0 tx=0).stop 2024-03-31T23:36:08.977 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:08.971+0000 7f0798214740 1 -- 172.21.15.46:0/3461093759 shutdown_connections 2024-03-31T23:36:08.977 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:08.971+0000 7f0798214740 1 --2- 172.21.15.46:0/3461093759 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558fe69275c0 0x558fe6932c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:36:08.977 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:08.971+0000 7f0798214740 1 --2- 172.21.15.46:0/3461093759 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558fe691e7f0 0x558fe690ae20 unknown :-1 s=CLOSED pgs=1045 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:36:08.977 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:08.971+0000 7f0798214740 1 --2- 172.21.15.46:0/3461093759 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558fe691ded0 0x558fe6926cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:36:08.977 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:08.971+0000 7f0798214740 1 -- 172.21.15.46:0/3461093759 >> 172.21.15.46:0/3461093759 conn(0x558fe69187f0 msgr2=0x558fe6918bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:36:08.977 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:08.971+0000 7f0798214740 1 -- 172.21.15.46:0/3461093759 shutdown_connections 2024-03-31T23:36:08.977 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:08.971+0000 7f0798214740 1 -- 172.21.15.46:0/3461093759 wait complete. 2024-03-31T23:36:08.977 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:08.971+0000 7f0798214740 1 -- start start 2024-03-31T23:36:08.977 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:08.972+0000 7f0798214740 1 Processor -- start 2024-03-31T23:36:08.977 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:08.972+0000 7f0798214740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558fe691ded0 0x558fe692aea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:36:08.977 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:08.972+0000 7f0798214740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558fe691e7f0 0x558fe692b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:36:08.977 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:08.972+0000 7f0798214740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558fe69275c0 0x558fe699c750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:36:08.977 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:08.972+0000 7f078effd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558fe691ded0 0x558fe692aea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:36:08.978 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:08.972+0000 7f078effd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558fe691ded0 0x558fe692aea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39540/0 (socket says 172.21.15.46:39540) 2024-03-31T23:36:08.978 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:08.972+0000 7f078effd640 1 -- 172.21.15.46:0/870166915 learned_addr learned my addr 172.21.15.46:0/870166915 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:36:08.978 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:08.972+0000 7f078e7fc640 1 --2- 172.21.15.46:0/870166915 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558fe691e7f0 0x558fe692b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:36:08.978 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:08.972+0000 7f078f7fe640 1 --2- 172.21.15.46:0/870166915 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558fe69275c0 0x558fe699c750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:36:08.978 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:08.972+0000 7f0798214740 1 -- 172.21.15.46:0/870166915 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f077c00e800 con 0x558fe691ded0 2024-03-31T23:36:08.978 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:08.973+0000 7f0798214740 1 -- 172.21.15.46:0/870166915 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f077c00fc70 con 0x558fe69275c0 2024-03-31T23:36:08.978 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:08.973+0000 7f0798214740 1 -- 172.21.15.46:0/870166915 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f077c00eed0 con 0x558fe691e7f0 2024-03-31T23:36:08.978 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:08.973+0000 7f0798214740 10 client.0 _ll_register_callbacks cb 0x7f0780009910 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T23:36:08.978 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:08.973+0000 7f078e7fc640 1 -- 172.21.15.46:0/870166915 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558fe69275c0 msgr2=0x558fe699c750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:36:08.978 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:08.973+0000 7f078e7fc640 1 --2- 172.21.15.46:0/870166915 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558fe69275c0 0x558fe699c750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:36:08.978 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:08.973+0000 7f078e7fc640 1 -- 172.21.15.46:0/870166915 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558fe691ded0 msgr2=0x558fe692aea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:36:08.978 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:08.973+0000 7f078e7fc640 1 --2- 172.21.15.46:0/870166915 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558fe691ded0 0x558fe692aea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:36:08.978 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:08.973+0000 7f078e7fc640 1 -- 172.21.15.46:0/870166915 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558fe692bc70 con 0x558fe691e7f0 2024-03-31T23:36:08.978 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:08.973+0000 7f078effd640 1 --2- 172.21.15.46:0/870166915 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558fe691ded0 0x558fe692aea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:36:08.979 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:08.973+0000 7f078e7fc640 1 --2- 172.21.15.46:0/870166915 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558fe691e7f0 0x558fe692b3e0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x558fe691f4d0 tx=0x7f078402ccf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2a0b2903dd379595 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:36:08.979 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:08.973+0000 7f0772ff5640 10 client.0 ms_handle_connect on v2:172.21.15.162:3300/0 2024-03-31T23:36:08.979 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:08.973+0000 7f0772ff5640 1 -- 172.21.15.46:0/870166915 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07840a4060 con 0x558fe691e7f0 2024-03-31T23:36:08.979 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:08.974+0000 7f0772ff5640 1 -- 172.21.15.46:0/870166915 <== mon.1 v2:172.21.15.162:3300/0 2 ==== osd_map(136..136 src has 1..136) ==== 9642+0+0 (secure 0 0 0) 0x7f0784005650 con 0x558fe691e7f0 2024-03-31T23:36:08.979 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:08.974+0000 7f0798214740 1 -- 172.21.15.46:0/870166915 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x558fe6885e30 con 0x558fe691e7f0 2024-03-31T23:36:08.979 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:08.974+0000 7f0772ff5640 1 -- 172.21.15.46:0/870166915 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_get_version_reply(handle=1 version=54) ==== 24+0+0 (secure 0 0 0) 0x7f07840c16f0 con 0x558fe691e7f0 2024-03-31T23:36:08.979 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:08.974+0000 7f0798214740 10 client.27832 fetch_fsmap learned FSMap version 54 2024-03-31T23:36:08.979 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:08.974+0000 7f0798214740 1 -- 172.21.15.46:0/870166915 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,fsmap=54,monmap=4+}) -- 0x558fe699dd70 con 0x558fe691e7f0 2024-03-31T23:36:08.979 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:08.974+0000 7f0772ff5640 1 -- 172.21.15.46:0/870166915 <== mon.1 v2:172.21.15.162:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f078403a240 con 0x558fe691e7f0 2024-03-31T23:36:08.979 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:08.974+0000 7f0772ff5640 1 -- 172.21.15.46:0/870166915 <== mon.1 v2:172.21.15.162:3300/0 5 ==== fsmap(e 54) ==== 8346+0+0 (secure 0 0 0) 0x7f07840bfaa0 con 0x558fe691e7f0 2024-03-31T23:36:08.979 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:08.974+0000 7f0798214740 10 client.27832 fetch_fsmap finished waiting for FSMap version 54 2024-03-31T23:36:08.979 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:08.975+0000 7f0798214740 10 client.27832 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-31T23:36:08.979 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:08.975+0000 7f0798214740 20 client.27832 populate_metadata read hostname 'smithi046' 2024-03-31T23:36:08.979 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:08.975+0000 7f0798214740 1 --2- 172.21.15.46:0/870166915 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x558fe6937550 0x558fe69a0070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:36:08.980 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:08.975+0000 7f0798214740 4 client.27832 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-31T23:36:08.980 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:08.975+0000 7f0798214740 1 -- 172.21.15.46:0/870166915 --> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] -- command(tid 0: {"prefix": "flush journal"}) -- 0x558fe6885e30 con 0x558fe6937550 2024-03-31T23:36:08.980 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:08.975+0000 7f078effd640 1 --2- 172.21.15.46:0/870166915 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x558fe6937550 0x558fe69a0070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:36:08.980 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:08.975+0000 7f0772ff5640 10 client.27832 ms_handle_connect on v2:172.21.15.142:6838/2519143918 2024-03-31T23:36:08.980 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:08.975+0000 7f078effd640 1 --2- 172.21.15.46:0/870166915 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x558fe6937550 0x558fe69a0070 crc :-1 s=READY pgs=295 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d1dd7de201747116 server_cookie=e34a4df3b961de2a in_seq=0 out_seq=0 2024-03-31T23:36:09.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:36:08 smithi162 ceph-mon[36237]: pgmap v774: 129 pgs: 129 active+clean; 1.3 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 4.0 MiB/s wr, 246 op/s 2024-03-31T23:36:09.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:36:08 smithi162 ceph-mon[36237]: osdmap e136: 12 total, 12 up, 12 in 2024-03-31T23:36:09.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:36:08 smithi046 ceph-mon[32498]: pgmap v774: 129 pgs: 129 active+clean; 1.3 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 4.0 MiB/s wr, 246 op/s 2024-03-31T23:36:09.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:36:08 smithi046 ceph-mon[32498]: osdmap e136: 12 total, 12 up, 12 in 2024-03-31T23:36:09.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:36:08 smithi142 ceph-mon[35271]: pgmap v774: 129 pgs: 129 active+clean; 1.3 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 4.0 MiB/s wr, 246 op/s 2024-03-31T23:36:09.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:36:08 smithi142 ceph-mon[35271]: osdmap e136: 12 total, 12 up, 12 in 2024-03-31T23:36:09.317 INFO:tasks.workunit.client.0.smithi046.stdout: 10 102962 8.10 MB/sec execute 554 sec latency 5963.921 ms 2024-03-31T23:36:09.334 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:09.332+0000 7f0772ff5640 1 -- 172.21.15.46:0/870166915 <== mds.0 v2:172.21.15.142:6838/2519143918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x558fe6885e30 con 0x558fe6937550 2024-03-31T23:36:09.334 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:09.332+0000 7f0772ff5640 10 client.27832 handle_command_reply: tid=0 2024-03-31T23:36:09.335 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:09.333+0000 7f0798214740 1 client.27832 shutdown 2024-03-31T23:36:09.335 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:09.333+0000 7f0798214740 1 -- 172.21.15.46:0/870166915 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558fe691e7f0 msgr2=0x558fe692b3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:36:09.335 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:09.333+0000 7f0798214740 1 --2- 172.21.15.46:0/870166915 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558fe691e7f0 0x558fe692b3e0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x558fe691f4d0 tx=0x7f078402ccf0 comp rx=0 tx=0).stop 2024-03-31T23:36:09.335 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:09.333+0000 7f0798214740 1 -- 172.21.15.46:0/870166915 shutdown_connections 2024-03-31T23:36:09.335 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:09.333+0000 7f0798214740 1 --2- 172.21.15.46:0/870166915 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x558fe6937550 0x558fe69a0070 crc :-1 s=READY pgs=295 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:36:09.335 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:09.333+0000 7f0798214740 1 --2- 172.21.15.46:0/870166915 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558fe69275c0 0x558fe699c750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:36:09.335 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:09.333+0000 7f0798214740 1 --2- 172.21.15.46:0/870166915 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558fe691e7f0 0x558fe692b3e0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:36:09.335 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:09.333+0000 7f0772ff5640 0 client.27832 ms_handle_reset on v2:172.21.15.142:6838/2519143918 2024-03-31T23:36:09.335 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:09.333+0000 7f0798214740 1 --2- 172.21.15.46:0/870166915 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558fe691ded0 0x558fe692aea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:36:09.335 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:09.333+0000 7f0798214740 1 -- 172.21.15.46:0/870166915 >> 172.21.15.46:0/870166915 conn(0x558fe69187f0 msgr2=0x558fe6924560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:36:09.335 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:09.333+0000 7f0798214740 1 -- 172.21.15.46:0/870166915 shutdown_connections 2024-03-31T23:36:09.335 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:09.333+0000 7f0798214740 1 -- 172.21.15.46:0/870166915 wait complete. 2024-03-31T23:36:09.336 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:09.333+0000 7f0798214740 20 client.27832 trim_cache size 0 max 16384 2024-03-31T23:36:09.336 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout:{ 2024-03-31T23:36:09.336 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout: "message": "", 2024-03-31T23:36:09.336 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout: "return_code": 0 2024-03-31T23:36:09.336 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout:} 2024-03-31T23:36:09.337 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:09.334+0000 7f0795556640 1 -- 172.21.15.46:0/1101444690 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f07780a2730 msgr2=0x7f07780a4bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:36:09.337 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:09.334+0000 7f0795556640 1 --2- 172.21.15.46:0/1101444690 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f07780a2730 0x7f07780a4bf0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f0788005fd0 tx=0x7f07880098e0 comp rx=0 tx=0).stop 2024-03-31T23:36:09.337 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:09.334+0000 7f0795556640 1 -- 172.21.15.46:0/1101444690 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0790101260 msgr2=0x7f0790198490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:36:09.337 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:09.334+0000 7f0795556640 1 --2- 172.21.15.46:0/1101444690 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0790101260 0x7f0790198490 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f0784000c00 tx=0x7f0784005dd0 comp rx=0 tx=0).stop 2024-03-31T23:36:09.337 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:09.334+0000 7f0795556640 1 -- 172.21.15.46:0/1101444690 shutdown_connections 2024-03-31T23:36:09.337 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:09.334+0000 7f0795556640 1 --2- 172.21.15.46:0/1101444690 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f07780a2730 0x7f07780a4bf0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:36:09.338 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:09.334+0000 7f0795556640 1 --2- 172.21.15.46:0/1101444690 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f07901055f0 0x7f079019c810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:36:09.338 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:09.334+0000 7f0795556640 1 --2- 172.21.15.46:0/1101444690 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0790101260 0x7f0790198490 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:36:09.338 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:09.334+0000 7f0795556640 1 --2- 172.21.15.46:0/1101444690 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f07901008a0 0x7f0790197f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:36:09.338 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:09.334+0000 7f0795556640 1 -- 172.21.15.46:0/1101444690 >> 172.21.15.46:0/1101444690 conn(0x7f07900774a0 msgr2=0x7f0790104500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:36:09.338 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:09.334+0000 7f0795556640 1 -- 172.21.15.46:0/1101444690 shutdown_connections 2024-03-31T23:36:09.338 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:09.334+0000 7f0795556640 1 -- 172.21.15.46:0/1101444690 wait complete. 2024-03-31T23:36:10.317 INFO:tasks.workunit.client.0.smithi046.stdout: 10 103111 8.10 MB/sec execute 555 sec latency 6964.205 ms 2024-03-31T23:36:11.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:36:10 smithi162 ceph-mon[36237]: pgmap v776: 129 pgs: 129 active+clean; 1.3 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.7 MiB/s wr, 181 op/s 2024-03-31T23:36:11.046 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:36:10 smithi162 ceph-mon[36237]: osdmap e137: 12 total, 12 up, 12 in 2024-03-31T23:36:11.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:36:10 smithi046 ceph-mon[32498]: pgmap v776: 129 pgs: 129 active+clean; 1.3 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.7 MiB/s wr, 181 op/s 2024-03-31T23:36:11.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:36:10 smithi046 ceph-mon[32498]: osdmap e137: 12 total, 12 up, 12 in 2024-03-31T23:36:11.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:36:10 smithi142 ceph-mon[35271]: pgmap v776: 129 pgs: 129 active+clean; 1.3 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.7 MiB/s wr, 181 op/s 2024-03-31T23:36:11.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:36:10 smithi142 ceph-mon[35271]: osdmap e137: 12 total, 12 up, 12 in 2024-03-31T23:36:11.317 INFO:tasks.workunit.client.0.smithi046.stdout: 10 103264 8.10 MB/sec execute 556 sec latency 7964.239 ms 2024-03-31T23:36:12.317 INFO:tasks.workunit.client.0.smithi046.stdout: 10 103418 8.10 MB/sec execute 557 sec latency 8964.270 ms 2024-03-31T23:36:13.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:36:12 smithi162 ceph-mon[36237]: pgmap v778: 129 pgs: 129 active+clean; 1.3 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 3.3 MiB/s wr, 226 op/s 2024-03-31T23:36:13.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:36:12 smithi046 ceph-mon[32498]: pgmap v778: 129 pgs: 129 active+clean; 1.3 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 3.3 MiB/s wr, 226 op/s 2024-03-31T23:36:13.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:36:12 smithi142 ceph-mon[35271]: pgmap v778: 129 pgs: 129 active+clean; 1.3 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 3.3 MiB/s wr, 226 op/s 2024-03-31T23:36:13.317 INFO:tasks.workunit.client.0.smithi046.stdout: 10 103558 8.09 MB/sec execute 558 sec latency 9964.303 ms 2024-03-31T23:36:14.317 INFO:tasks.workunit.client.0.smithi046.stdout: 10 103730 8.10 MB/sec execute 559 sec latency 10964.334 ms 2024-03-31T23:36:15.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:36:14 smithi162 ceph-mon[36237]: pgmap v779: 129 pgs: 129 active+clean; 1.3 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.6 MiB/s wr, 112 op/s 2024-03-31T23:36:15.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:36:14 smithi046 ceph-mon[32498]: pgmap v779: 129 pgs: 129 active+clean; 1.3 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.6 MiB/s wr, 112 op/s 2024-03-31T23:36:15.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:36:14 smithi142 ceph-mon[35271]: pgmap v779: 129 pgs: 129 active+clean; 1.3 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.6 MiB/s wr, 112 op/s 2024-03-31T23:36:15.317 INFO:tasks.workunit.client.0.smithi046.stdout: 10 103909 8.10 MB/sec execute 560 sec latency 11964.361 ms 2024-03-31T23:36:16.317 INFO:tasks.workunit.client.0.smithi046.stdout: 10 104082 8.10 MB/sec execute 561 sec latency 12964.412 ms 2024-03-31T23:36:17.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:36:16 smithi162 ceph-mon[36237]: pgmap v780: 129 pgs: 129 active+clean; 1.3 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 5.0 MiB/s wr, 334 op/s 2024-03-31T23:36:17.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:36:16 smithi046 ceph-mon[32498]: pgmap v780: 129 pgs: 129 active+clean; 1.3 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 5.0 MiB/s wr, 334 op/s 2024-03-31T23:36:17.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:36:16 smithi142 ceph-mon[35271]: pgmap v780: 129 pgs: 129 active+clean; 1.3 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 5.0 MiB/s wr, 334 op/s 2024-03-31T23:36:17.317 INFO:tasks.workunit.client.0.smithi046.stdout: 10 104223 8.10 MB/sec execute 562 sec latency 13964.447 ms 2024-03-31T23:36:18.318 INFO:tasks.workunit.client.0.smithi046.stdout: 10 104369 8.10 MB/sec execute 563 sec latency 14494.269 ms 2024-03-31T23:36:19.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:36:18 smithi162 ceph-mon[36237]: pgmap v781: 129 pgs: 129 active+clean; 1.3 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 4.3 MiB/s wr, 287 op/s 2024-03-31T23:36:19.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:36:18 smithi046 ceph-mon[32498]: pgmap v781: 129 pgs: 129 active+clean; 1.3 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 4.3 MiB/s wr, 287 op/s 2024-03-31T23:36:19.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:36:18 smithi142 ceph-mon[35271]: pgmap v781: 129 pgs: 129 active+clean; 1.3 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 4.3 MiB/s wr, 287 op/s 2024-03-31T23:36:19.317 INFO:tasks.workunit.client.0.smithi046.stdout: 10 104537 8.11 MB/sec execute 564 sec latency 72.111 ms 2024-03-31T23:36:20.318 INFO:tasks.workunit.client.0.smithi046.stdout: 10 104683 8.11 MB/sec execute 565 sec latency 77.050 ms 2024-03-31T23:36:21.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:36:20 smithi162 ceph-mon[36237]: pgmap v782: 129 pgs: 129 active+clean; 1.3 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 5.8 MiB/s wr, 303 op/s 2024-03-31T23:36:21.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:36:20 smithi046 ceph-mon[32498]: pgmap v782: 129 pgs: 129 active+clean; 1.3 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 5.8 MiB/s wr, 303 op/s 2024-03-31T23:36:21.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:36:20 smithi142 ceph-mon[35271]: pgmap v782: 129 pgs: 129 active+clean; 1.3 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 5.8 MiB/s wr, 303 op/s 2024-03-31T23:36:21.317 INFO:tasks.workunit.client.0.smithi046.stdout: 10 104840 8.12 MB/sec execute 566 sec latency 76.320 ms 2024-03-31T23:36:22.317 INFO:tasks.workunit.client.0.smithi046.stdout: 10 105011 8.12 MB/sec execute 567 sec latency 71.173 ms 2024-03-31T23:36:22.469 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:22.468+0000 7f78b4d3e640 1 -- 172.21.15.46:0/3926999345 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f78b0106640 msgr2=0x7f78b0106a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:36:22.469 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:22.468+0000 7f78b4d3e640 1 --2- 172.21.15.46:0/3926999345 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f78b0106640 0x7f78b0106a20 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f78a0009930 tx=0x7f78a002bc30 comp rx=0 tx=0).stop 2024-03-31T23:36:22.469 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:22.468+0000 7f78b4d3e640 1 -- 172.21.15.46:0/3926999345 shutdown_connections 2024-03-31T23:36:22.469 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:22.468+0000 7f78b4d3e640 1 --2- 172.21.15.46:0/3926999345 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f78b0100ff0 0x7f78b010d500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:36:22.469 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:22.468+0000 7f78b4d3e640 1 --2- 172.21.15.46:0/3926999345 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f78b0100630 0x7f78b0100ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:36:22.469 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:22.468+0000 7f78b4d3e640 1 --2- 172.21.15.46:0/3926999345 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f78b0106640 0x7f78b0106a20 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:36:22.470 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:22.468+0000 7f78b4d3e640 1 -- 172.21.15.46:0/3926999345 >> 172.21.15.46:0/3926999345 conn(0x7f78b00777b0 msgr2=0x7f78b0077bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:36:22.470 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:22.468+0000 7f78b4d3e640 1 -- 172.21.15.46:0/3926999345 shutdown_connections 2024-03-31T23:36:22.470 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:22.468+0000 7f78b4d3e640 1 -- 172.21.15.46:0/3926999345 wait complete. 2024-03-31T23:36:22.471 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:22.469+0000 7f78b4d3e640 1 Processor -- start 2024-03-31T23:36:22.471 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:22.469+0000 7f78b4d3e640 1 -- start start 2024-03-31T23:36:22.472 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:22.469+0000 7f78b4d3e640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f78b0100630 0x7f78b019e790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:36:22.472 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:22.469+0000 7f78b4d3e640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f78b0100ff0 0x7f78b019ecd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:36:22.472 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:22.469+0000 7f78b4d3e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f78b0106640 0x7f78b01a2dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:36:22.472 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:22.469+0000 7f78b4d3e640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f78b0110910 con 0x7f78b0106640 2024-03-31T23:36:22.472 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:22.469+0000 7f78b4d3e640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f78b0110790 con 0x7f78b0100630 2024-03-31T23:36:22.472 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:22.469+0000 7f78b4d3e640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f78b0110a90 con 0x7f78b0100ff0 2024-03-31T23:36:22.472 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:22.469+0000 7f78aed9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f78b0106640 0x7f78b01a2dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:36:22.472 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:22.469+0000 7f78aed9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f78b0106640 0x7f78b01a2dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41754/0 (socket says 172.21.15.46:41754) 2024-03-31T23:36:22.472 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:22.469+0000 7f78aed9d640 1 -- 172.21.15.46:0/4072287645 learned_addr learned my addr 172.21.15.46:0/4072287645 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:36:22.472 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:22.469+0000 7f78ae59c640 1 --2- 172.21.15.46:0/4072287645 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f78b0100630 0x7f78b019e790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:36:22.472 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:22.469+0000 7f78add9b640 1 --2- 172.21.15.46:0/4072287645 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f78b0100ff0 0x7f78b019ecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:36:22.472 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:22.470+0000 7f78ae59c640 1 -- 172.21.15.46:0/4072287645 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f78b0100ff0 msgr2=0x7f78b019ecd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:36:22.472 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:22.470+0000 7f78ae59c640 1 --2- 172.21.15.46:0/4072287645 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f78b0100ff0 0x7f78b019ecd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:36:22.473 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:22.470+0000 7f78ae59c640 1 -- 172.21.15.46:0/4072287645 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f78b0106640 msgr2=0x7f78b01a2dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:36:22.473 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:22.470+0000 7f78ae59c640 1 --2- 172.21.15.46:0/4072287645 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f78b0106640 0x7f78b01a2dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:36:22.473 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:22.470+0000 7f78ae59c640 1 -- 172.21.15.46:0/4072287645 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78b01a3500 con 0x7f78b0100630 2024-03-31T23:36:22.473 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:22.470+0000 7f78ae59c640 1 --2- 172.21.15.46:0/4072287645 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f78b0100630 0x7f78b019e790 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f78a0036040 tx=0x7f78a00045e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:36:22.473 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:22.470+0000 7f789f7fe640 1 -- 172.21.15.46:0/4072287645 <== mon.2 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78a0043020 con 0x7f78b0100630 2024-03-31T23:36:22.473 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:22.470+0000 7f78b4d3e640 1 -- 172.21.15.46:0/4072287645 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f78b01a3790 con 0x7f78b0100630 2024-03-31T23:36:22.473 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:22.470+0000 7f789f7fe640 1 -- 172.21.15.46:0/4072287645 <== mon.2 v2:172.21.15.142:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f78a002cb70 con 0x7f78b0100630 2024-03-31T23:36:22.473 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:22.471+0000 7f789f7fe640 1 -- 172.21.15.46:0/4072287645 <== mon.2 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78a0003ae0 con 0x7f78b0100630 2024-03-31T23:36:22.473 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:22.471+0000 7f78b4d3e640 1 -- 172.21.15.46:0/4072287645 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f78b01a3bf0 con 0x7f78b0100630 2024-03-31T23:36:22.474 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:22.473+0000 7f789f7fe640 1 -- 172.21.15.46:0/4072287645 <== mon.2 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 25) ==== 137451+0+0 (secure 0 0 0) 0x7f78a0046020 con 0x7f78b0100630 2024-03-31T23:36:22.475 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:22.474+0000 7f78b7a03740 1 Processor -- start 2024-03-31T23:36:22.475 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:22.474+0000 7f78b7a03740 1 -- start start 2024-03-31T23:36:22.475 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:22.474+0000 7f78b7a03740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561cea97a6c0 0x561cea97aaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:36:22.476 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:22.474+0000 7f78b7a03740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561cea97afe0 0x561cea983870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:36:22.476 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:22.474+0000 7f78b7a03740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x561cea983db0 0x561cea9861a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:36:22.476 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:22.474+0000 7f78b7a03740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x561cea8dd8f0 con 0x561cea97a6c0 2024-03-31T23:36:22.476 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:22.474+0000 7f78b7a03740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x561cea8868d0 con 0x561cea983db0 2024-03-31T23:36:22.476 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:22.474+0000 7f78b7a03740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x561cea8d9990 con 0x561cea97afe0 2024-03-31T23:36:22.476 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:22.475+0000 7f78aed9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561cea97a6c0 0x561cea97aaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:36:22.476 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:22.475+0000 7f78aed9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561cea97a6c0 0x561cea97aaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41758/0 (socket says 172.21.15.46:41758) 2024-03-31T23:36:22.476 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:22.475+0000 7f78aed9d640 1 -- 172.21.15.46:0/3288009290 learned_addr learned my addr 172.21.15.46:0/3288009290 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:36:22.476 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:22.475+0000 7f78ae59c640 1 --2- 172.21.15.46:0/3288009290 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561cea97afe0 0x561cea983870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:36:22.476 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:22.475+0000 7f78add9b640 1 --2- 172.21.15.46:0/3288009290 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x561cea983db0 0x561cea9861a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:36:22.477 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:22.475+0000 7f78aed9d640 1 -- 172.21.15.46:0/3288009290 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561cea97afe0 msgr2=0x561cea983870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:36:22.478 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:22.475+0000 7f789f7fe640 1 --2- 172.21.15.46:0/4072287645 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f78980a2730 0x7f78980a4bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:36:22.478 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:22.475+0000 7f78aed9d640 1 --2- 172.21.15.46:0/3288009290 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561cea97afe0 0x561cea983870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:36:22.478 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:22.475+0000 7f78aed9d640 1 -- 172.21.15.46:0/3288009290 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x561cea983db0 msgr2=0x561cea9861a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:36:22.478 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:22.475+0000 7f78aed9d640 1 --2- 172.21.15.46:0/3288009290 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x561cea983db0 0x561cea9861a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:36:22.478 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:22.475+0000 7f78aed9d640 1 -- 172.21.15.46:0/3288009290 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561cea5f2210 con 0x561cea97a6c0 2024-03-31T23:36:22.478 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:22.475+0000 7f78add9b640 1 --2- 172.21.15.46:0/3288009290 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x561cea983db0 0x561cea9861a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:36:22.478 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:22.476+0000 7f789f7fe640 1 -- 172.21.15.46:0/4072287645 <== mon.2 v2:172.21.15.142:3300/0 5 ==== osd_map(137..137 src has 1..137) ==== 9554+0+0 (secure 0 0 0) 0x7f78a00f0e10 con 0x7f78b0100630 2024-03-31T23:36:22.478 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:22.476+0000 7f78aed9d640 1 --2- 172.21.15.46:0/4072287645 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f78980a2730 0x7f78980a4bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:36:22.478 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:22.476+0000 7f78aed9d640 1 --2- 172.21.15.46:0/4072287645 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f78980a2730 0x7f78980a4bf0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f78a8007fd0 tx=0x7f78a800a000 comp rx=0 tx=0).ready entity=mgr.15654 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:36:22.478 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:22.476+0000 7f78aed9d640 1 --2- 172.21.15.46:0/3288009290 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561cea97a6c0 0x561cea97aaa0 secure :-1 s=READY pgs=1046 cs=0 l=1 rev1=1 crypto rx=0x7f78a8007f40 tx=0x7f78a800f850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f5f9f51d82b2c179 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:36:22.478 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:22.477+0000 7f789cff9640 1 -- 172.21.15.46:0/3288009290 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78a801a6d0 con 0x561cea97a6c0 2024-03-31T23:36:22.478 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:22.477+0000 7f789cff9640 1 -- 172.21.15.46:0/3288009290 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f78a8010820 con 0x561cea97a6c0 2024-03-31T23:36:22.478 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:22.477+0000 7f789cff9640 1 -- 172.21.15.46:0/3288009290 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78a8019bb0 con 0x561cea97a6c0 2024-03-31T23:36:22.480 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:22.478+0000 7f78b7a03740 1 -- 172.21.15.46:0/3288009290 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561cea97a6c0 msgr2=0x561cea97aaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:36:22.480 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:22.478+0000 7f78b7a03740 1 --2- 172.21.15.46:0/3288009290 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561cea97a6c0 0x561cea97aaa0 secure :-1 s=READY pgs=1046 cs=0 l=1 rev1=1 crypto rx=0x7f78a8007f40 tx=0x7f78a800f850 comp rx=0 tx=0).stop 2024-03-31T23:36:22.480 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:22.478+0000 7f78b7a03740 1 -- 172.21.15.46:0/3288009290 shutdown_connections 2024-03-31T23:36:22.480 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:22.478+0000 7f78b7a03740 1 --2- 172.21.15.46:0/3288009290 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x561cea983db0 0x561cea9861a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:36:22.480 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:22.478+0000 7f78b7a03740 1 --2- 172.21.15.46:0/3288009290 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561cea97afe0 0x561cea983870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:36:22.480 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:22.478+0000 7f78b7a03740 1 --2- 172.21.15.46:0/3288009290 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561cea97a6c0 0x561cea97aaa0 unknown :-1 s=CLOSED pgs=1046 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:36:22.480 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:22.478+0000 7f78b7a03740 1 -- 172.21.15.46:0/3288009290 >> 172.21.15.46:0/3288009290 conn(0x561cea974fe0 msgr2=0x561cea9753c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:36:22.480 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:22.478+0000 7f78b7a03740 1 -- 172.21.15.46:0/3288009290 shutdown_connections 2024-03-31T23:36:22.480 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:22.478+0000 7f78b7a03740 1 -- 172.21.15.46:0/3288009290 wait complete. 2024-03-31T23:36:22.480 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:22.478+0000 7f78b7a03740 1 -- start start 2024-03-31T23:36:22.480 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:22.479+0000 7f78b7a03740 1 Processor -- start 2024-03-31T23:36:22.481 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:22.479+0000 7f78b7a03740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x561cea97a6c0 0x561cea980e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:36:22.481 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:22.479+0000 7f78b7a03740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561cea97afe0 0x561cea981350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:36:22.481 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:22.479+0000 7f78b7a03740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561cea983db0 0x561cea981890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:36:22.483 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:22.479+0000 7f78b7a03740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x561cea986a30 con 0x561cea983db0 2024-03-31T23:36:22.483 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:22.479+0000 7f78b7a03740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x561cea87bf80 con 0x561cea97a6c0 2024-03-31T23:36:22.483 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:22.479+0000 7f78b7a03740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x561cea986bb0 con 0x561cea97afe0 2024-03-31T23:36:22.484 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:22.479+0000 7f78add9b640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561cea97afe0 0x561cea981350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:36:22.484 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:22.479+0000 7f78b7a03740 10 client.0 _ll_register_callbacks cb 0x561cea8d4290 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T23:36:22.484 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:22.479+0000 7f78aed9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561cea983db0 0x561cea981890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:36:22.484 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:22.479+0000 7f78ae59c640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x561cea97a6c0 0x561cea980e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:36:22.484 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:22.479+0000 7f78aed9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561cea983db0 0x561cea981890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41774/0 (socket says 172.21.15.46:41774) 2024-03-31T23:36:22.484 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:22.479+0000 7f78aed9d640 1 -- 172.21.15.46:0/3567990801 learned_addr learned my addr 172.21.15.46:0/3567990801 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:36:22.484 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:22.480+0000 7f78aed9d640 1 -- 172.21.15.46:0/3567990801 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561cea97afe0 msgr2=0x561cea981350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:36:22.484 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:22.480+0000 7f78aed9d640 1 --2- 172.21.15.46:0/3567990801 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561cea97afe0 0x561cea981350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:36:22.484 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:22.480+0000 7f78aed9d640 1 -- 172.21.15.46:0/3567990801 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x561cea97a6c0 msgr2=0x561cea980e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:36:22.484 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:22.480+0000 7f78aed9d640 1 --2- 172.21.15.46:0/3567990801 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x561cea97a6c0 0x561cea980e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:36:22.484 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:22.480+0000 7f78aed9d640 1 -- 172.21.15.46:0/3567990801 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561cea6be620 con 0x561cea983db0 2024-03-31T23:36:22.484 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:22.480+0000 7f78aed9d640 1 --2- 172.21.15.46:0/3567990801 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561cea983db0 0x561cea981890 secure :-1 s=READY pgs=1047 cs=0 l=1 rev1=1 crypto rx=0x7f78a8006010 tx=0x7f78a8019520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4eb52b1af56e9e23 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:36:22.484 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:22.480+0000 7f7895ffb640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-31T23:36:22.485 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:22.480+0000 7f7895ffb640 1 -- 172.21.15.46:0/3567990801 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78a8032060 con 0x561cea983db0 2024-03-31T23:36:22.485 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:22.480+0000 7f78b7a03740 1 -- 172.21.15.46:0/3567990801 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x561cea8f1120 con 0x561cea983db0 2024-03-31T23:36:22.485 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:22.480+0000 7f7895ffb640 1 -- 172.21.15.46:0/3567990801 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(137..137 src has 1..137) ==== 9554+0+0 (secure 0 0 0) 0x7f78a801a6e0 con 0x561cea983db0 2024-03-31T23:36:22.485 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:22.481+0000 7f7895ffb640 1 -- 172.21.15.46:0/3567990801 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=54) ==== 24+0+0 (secure 0 0 0) 0x7f78a801a8e0 con 0x561cea983db0 2024-03-31T23:36:22.485 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:22.481+0000 7f78b7a03740 10 client.18135 fetch_fsmap learned FSMap version 54 2024-03-31T23:36:22.485 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:22.481+0000 7f78b7a03740 1 -- 172.21.15.46:0/3567990801 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=54,monmap=4+}) -- 0x561cea9929a0 con 0x561cea983db0 2024-03-31T23:36:22.485 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:22.481+0000 7f7895ffb640 1 -- 172.21.15.46:0/3567990801 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x561cea8f1120 con 0x561cea983db0 2024-03-31T23:36:22.485 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:22.481+0000 7f7895ffb640 1 -- 172.21.15.46:0/3567990801 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 54) ==== 8346+0+0 (secure 0 0 0) 0x7f78a80248c0 con 0x561cea983db0 2024-03-31T23:36:22.485 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:22.481+0000 7f78b7a03740 10 client.18135 fetch_fsmap finished waiting for FSMap version 54 2024-03-31T23:36:22.485 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:22.481+0000 7f78b7a03740 10 client.18135 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-31T23:36:22.485 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:22.481+0000 7f78b7a03740 20 client.18135 populate_metadata read hostname 'smithi046' 2024-03-31T23:36:22.485 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:22.481+0000 7f78b7a03740 1 --2- 172.21.15.46:0/3567990801 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x561cea9941d0 0x561cea996630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:36:22.485 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:22.481+0000 7f78b7a03740 4 client.18135 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-31T23:36:22.485 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:22.481+0000 7f78b7a03740 1 -- 172.21.15.46:0/3567990801 --> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561cea8eae30 con 0x561cea9941d0 2024-03-31T23:36:22.486 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:22.482+0000 7f78ae59c640 1 --2- 172.21.15.46:0/3567990801 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x561cea9941d0 0x561cea996630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:36:22.486 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:22.482+0000 7f7895ffb640 10 client.18135 ms_handle_connect on v2:172.21.15.142:6838/2519143918 2024-03-31T23:36:22.486 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:22.482+0000 7f78ae59c640 1 --2- 172.21.15.46:0/3567990801 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x561cea9941d0 0x561cea996630 crc :-1 s=READY pgs=296 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6da83327c4cdb8a4 server_cookie=df1c45aef5d826ee in_seq=0 out_seq=0 2024-03-31T23:36:22.487 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:22.484+0000 7f7895ffb640 1 -- 172.21.15.46:0/3567990801 <== mds.0 v2:172.21.15.142:6838/2519143918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x561cea8eae30 con 0x561cea9941d0 2024-03-31T23:36:22.487 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:22.484+0000 7f7895ffb640 10 client.18135 handle_command_reply: tid=0 2024-03-31T23:36:22.487 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:22.484+0000 7f78b7a03740 1 client.18135 shutdown 2024-03-31T23:36:22.487 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:22.485+0000 7f78b7a03740 1 -- 172.21.15.46:0/3567990801 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561cea983db0 msgr2=0x561cea981890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:36:22.487 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:22.485+0000 7f78b7a03740 1 --2- 172.21.15.46:0/3567990801 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561cea983db0 0x561cea981890 secure :-1 s=READY pgs=1047 cs=0 l=1 rev1=1 crypto rx=0x7f78a8006010 tx=0x7f78a8019520 comp rx=0 tx=0).stop 2024-03-31T23:36:22.487 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:22.485+0000 7f78b7a03740 1 -- 172.21.15.46:0/3567990801 shutdown_connections 2024-03-31T23:36:22.488 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:22.485+0000 7f78b7a03740 1 --2- 172.21.15.46:0/3567990801 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x561cea9941d0 0x561cea996630 crc :-1 s=READY pgs=296 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:36:22.488 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:22.485+0000 7f78b7a03740 1 --2- 172.21.15.46:0/3567990801 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561cea983db0 0x561cea981890 unknown :-1 s=CLOSED pgs=1047 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:36:22.488 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:22.485+0000 7f7895ffb640 0 client.18135 ms_handle_reset on v2:172.21.15.142:6838/2519143918 2024-03-31T23:36:22.488 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:22.485+0000 7f78b7a03740 1 --2- 172.21.15.46:0/3567990801 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561cea97afe0 0x561cea981350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:36:22.488 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:22.485+0000 7f78b7a03740 1 --2- 172.21.15.46:0/3567990801 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x561cea97a6c0 0x561cea980e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:36:22.488 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:22.485+0000 7f78b7a03740 1 -- 172.21.15.46:0/3567990801 >> 172.21.15.46:0/3567990801 conn(0x561cea974fe0 msgr2=0x561cea976d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:36:22.488 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:22.485+0000 7f78b7a03740 1 -- 172.21.15.46:0/3567990801 shutdown_connections 2024-03-31T23:36:22.488 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:22.485+0000 7f78b7a03740 1 -- 172.21.15.46:0/3567990801 wait complete. 2024-03-31T23:36:22.488 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:22.485+0000 7f78b7a03740 20 client.18135 trim_cache size 0 max 16384 2024-03-31T23:36:22.515 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:22.509+0000 7f78b7a03740 1 Processor -- start 2024-03-31T23:36:22.515 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:22.509+0000 7f78b7a03740 1 -- start start 2024-03-31T23:36:22.515 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:22.509+0000 7f78b7a03740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x561cea97a6c0 0x561cea983540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:36:22.515 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:22.509+0000 7f78b7a03740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561cea97afe0 0x561cea97ff70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:36:22.515 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:22.510+0000 7f78b7a03740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561cea983db0 0x561cea98f800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:36:22.516 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:22.510+0000 7f78b7a03740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f78b01b7a00 con 0x561cea97afe0 2024-03-31T23:36:22.516 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:22.510+0000 7f78b7a03740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f78b01b52d0 con 0x561cea97a6c0 2024-03-31T23:36:22.516 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:22.510+0000 7f78b7a03740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f78b01b3140 con 0x561cea983db0 2024-03-31T23:36:22.516 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:22.510+0000 7f78add9b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561cea97afe0 0x561cea97ff70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:36:22.516 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:22.510+0000 7f78ae59c640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x561cea97a6c0 0x561cea983540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:36:22.516 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:22.510+0000 7f78add9b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561cea97afe0 0x561cea97ff70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41780/0 (socket says 172.21.15.46:41780) 2024-03-31T23:36:22.516 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:22.510+0000 7f78add9b640 1 -- 172.21.15.46:0/525892439 learned_addr learned my addr 172.21.15.46:0/525892439 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:36:22.516 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:22.510+0000 7f78aed9d640 1 --2- 172.21.15.46:0/525892439 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561cea983db0 0x561cea98f800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:36:22.516 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:22.510+0000 7f78ae59c640 1 -- 172.21.15.46:0/525892439 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561cea983db0 msgr2=0x561cea98f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:36:22.516 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:22.510+0000 7f78ae59c640 1 --2- 172.21.15.46:0/525892439 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561cea983db0 0x561cea98f800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:36:22.516 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:22.510+0000 7f78ae59c640 1 -- 172.21.15.46:0/525892439 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561cea97afe0 msgr2=0x561cea97ff70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:36:22.516 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:22.510+0000 7f78ae59c640 1 --2- 172.21.15.46:0/525892439 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561cea97afe0 0x561cea97ff70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:36:22.516 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:22.510+0000 7f78ae59c640 1 -- 172.21.15.46:0/525892439 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561cea9861a0 con 0x561cea97a6c0 2024-03-31T23:36:22.516 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:22.510+0000 7f78add9b640 1 --2- 172.21.15.46:0/525892439 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561cea97afe0 0x561cea97ff70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:36:22.517 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:22.510+0000 7f78ae59c640 1 --2- 172.21.15.46:0/525892439 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x561cea97a6c0 0x561cea983540 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f78a00ee200 tx=0x7f78a00dd040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=49f8674efeac9c4f server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:36:22.517 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:22.510+0000 7f7897fff640 1 -- 172.21.15.46:0/525892439 <== mon.2 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78a00dd0b0 con 0x561cea97a6c0 2024-03-31T23:36:22.517 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:22.511+0000 7f7897fff640 1 -- 172.21.15.46:0/525892439 <== mon.2 v2:172.21.15.142:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f78a00bd930 con 0x561cea97a6c0 2024-03-31T23:36:22.517 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:22.511+0000 7f7897fff640 1 -- 172.21.15.46:0/525892439 <== mon.2 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78a00ee370 con 0x561cea97a6c0 2024-03-31T23:36:22.517 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:22.511+0000 7f78b7a03740 1 -- 172.21.15.46:0/525892439 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x561cea97a6c0 msgr2=0x561cea983540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:36:22.517 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:22.511+0000 7f78b7a03740 1 --2- 172.21.15.46:0/525892439 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x561cea97a6c0 0x561cea983540 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f78a00ee200 tx=0x7f78a00dd040 comp rx=0 tx=0).stop 2024-03-31T23:36:22.517 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:22.511+0000 7f78b7a03740 1 -- 172.21.15.46:0/525892439 shutdown_connections 2024-03-31T23:36:22.517 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:22.511+0000 7f78b7a03740 1 --2- 172.21.15.46:0/525892439 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561cea983db0 0x561cea98f800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:36:22.517 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:22.511+0000 7f78b7a03740 1 --2- 172.21.15.46:0/525892439 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561cea97afe0 0x561cea97ff70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:36:22.518 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:22.511+0000 7f78b7a03740 1 --2- 172.21.15.46:0/525892439 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x561cea97a6c0 0x561cea983540 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:36:22.518 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:22.511+0000 7f78b7a03740 1 -- 172.21.15.46:0/525892439 >> 172.21.15.46:0/525892439 conn(0x561cea974fe0 msgr2=0x561cea9753c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:36:22.518 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:22.512+0000 7f78b7a03740 1 -- 172.21.15.46:0/525892439 shutdown_connections 2024-03-31T23:36:22.518 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:22.512+0000 7f78b7a03740 1 -- 172.21.15.46:0/525892439 wait complete. 2024-03-31T23:36:22.518 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:22.512+0000 7f78b7a03740 1 -- start start 2024-03-31T23:36:22.518 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:22.512+0000 7f78b7a03740 1 Processor -- start 2024-03-31T23:36:22.518 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:22.512+0000 7f78b7a03740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561cea97a6c0 0x561cea986fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:36:22.518 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:22.512+0000 7f78b7a03740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x561cea97afe0 0x561cea987520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:36:22.518 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:22.512+0000 7f78b7a03740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561cea983db0 0x561cea9f8520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:36:22.518 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:22.512+0000 7f78b7a03740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f78b01b3790 con 0x561cea983db0 2024-03-31T23:36:22.518 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:22.512+0000 7f78b7a03740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f78b01b4c00 con 0x561cea97afe0 2024-03-31T23:36:22.518 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:22.512+0000 7f78b7a03740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f78b01b3e60 con 0x561cea97a6c0 2024-03-31T23:36:22.518 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:22.512+0000 7f78aed9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561cea983db0 0x561cea9f8520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:36:22.519 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:22.512+0000 7f78ae59c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561cea97a6c0 0x561cea986fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:36:22.519 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:22.512+0000 7f78aed9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561cea983db0 0x561cea9f8520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41796/0 (socket says 172.21.15.46:41796) 2024-03-31T23:36:22.519 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:22.512+0000 7f78b7a03740 10 client.0 _ll_register_callbacks cb 0x7f78a40097a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T23:36:22.519 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:22.512+0000 7f78aed9d640 1 -- 172.21.15.46:0/2814829235 learned_addr learned my addr 172.21.15.46:0/2814829235 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:36:22.519 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:22.513+0000 7f78add9b640 1 --2- 172.21.15.46:0/2814829235 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x561cea97afe0 0x561cea987520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:36:22.519 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:22.513+0000 7f78ae59c640 1 -- 172.21.15.46:0/2814829235 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x561cea97afe0 msgr2=0x561cea987520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:36:22.519 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:22.513+0000 7f78ae59c640 1 --2- 172.21.15.46:0/2814829235 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x561cea97afe0 0x561cea987520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:36:22.519 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:22.513+0000 7f78ae59c640 1 -- 172.21.15.46:0/2814829235 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561cea983db0 msgr2=0x561cea9f8520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:36:22.519 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:22.513+0000 7f78ae59c640 1 --2- 172.21.15.46:0/2814829235 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561cea983db0 0x561cea9f8520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:36:22.519 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:22.513+0000 7f78ae59c640 1 -- 172.21.15.46:0/2814829235 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561cea9f8a60 con 0x561cea97a6c0 2024-03-31T23:36:22.519 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:22.513+0000 7f78aed9d640 1 --2- 172.21.15.46:0/2814829235 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561cea983db0 0x561cea9f8520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:36:22.519 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:22.513+0000 7f78add9b640 1 --2- 172.21.15.46:0/2814829235 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x561cea97afe0 0x561cea987520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T23:36:22.519 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:22.513+0000 7f78ae59c640 1 --2- 172.21.15.46:0/2814829235 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561cea97a6c0 0x561cea986fe0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f78a00bdae0 tx=0x7f78a0005fd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6a23fdf2195bf312 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:36:22.519 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:22.513+0000 7f7895ffb640 10 client.0 ms_handle_connect on v2:172.21.15.162:3300/0 2024-03-31T23:36:22.520 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:22.513+0000 7f7895ffb640 1 -- 172.21.15.46:0/2814829235 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78a00d5070 con 0x561cea97a6c0 2024-03-31T23:36:22.520 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:22.513+0000 7f7895ffb640 1 -- 172.21.15.46:0/2814829235 <== mon.1 v2:172.21.15.162:3300/0 2 ==== osd_map(137..137 src has 1..137) ==== 9554+0+0 (secure 0 0 0) 0x7f78a0005520 con 0x561cea97a6c0 2024-03-31T23:36:22.520 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:22.513+0000 7f78b7a03740 1 -- 172.21.15.46:0/2814829235 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f78a00bd930 con 0x561cea97a6c0 2024-03-31T23:36:22.520 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:22.513+0000 7f7895ffb640 1 -- 172.21.15.46:0/2814829235 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_get_version_reply(handle=1 version=54) ==== 24+0+0 (secure 0 0 0) 0x7f78a00c1170 con 0x561cea97a6c0 2024-03-31T23:36:22.520 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:22.513+0000 7f78b7a03740 10 client.27850 fetch_fsmap learned FSMap version 54 2024-03-31T23:36:22.520 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:22.513+0000 7f78b7a03740 1 -- 172.21.15.46:0/2814829235 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,fsmap=54,monmap=4+}) -- 0x561cea9f9c50 con 0x561cea97a6c0 2024-03-31T23:36:22.520 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:22.514+0000 7f7895ffb640 1 -- 172.21.15.46:0/2814829235 <== mon.1 v2:172.21.15.162:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f78a00bd930 con 0x561cea97a6c0 2024-03-31T23:36:22.520 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:22.514+0000 7f7895ffb640 1 -- 172.21.15.46:0/2814829235 <== mon.1 v2:172.21.15.162:3300/0 5 ==== fsmap(e 54) ==== 8346+0+0 (secure 0 0 0) 0x7f78a00b98a0 con 0x561cea97a6c0 2024-03-31T23:36:22.520 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:22.514+0000 7f78b7a03740 10 client.27850 fetch_fsmap finished waiting for FSMap version 54 2024-03-31T23:36:22.520 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:22.514+0000 7f78b7a03740 10 client.27850 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-31T23:36:22.520 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:22.514+0000 7f78b7a03740 20 client.27850 populate_metadata read hostname 'smithi046' 2024-03-31T23:36:22.520 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:22.514+0000 7f78b7a03740 1 --2- 172.21.15.46:0/2814829235 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x561cea9941d0 0x561cea9fc2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:36:22.520 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:22.514+0000 7f78b7a03740 4 client.27850 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-31T23:36:22.520 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:22.514+0000 7f78b7a03740 1 -- 172.21.15.46:0/2814829235 --> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] -- command(tid 0: {"prefix": "flush journal"}) -- 0x561cea8e5180 con 0x561cea9941d0 2024-03-31T23:36:22.521 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:22.514+0000 7f78add9b640 1 --2- 172.21.15.46:0/2814829235 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x561cea9941d0 0x561cea9fc2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:36:22.521 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:22.515+0000 7f7895ffb640 10 client.27850 ms_handle_connect on v2:172.21.15.142:6838/2519143918 2024-03-31T23:36:22.521 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:22.515+0000 7f78add9b640 1 --2- 172.21.15.46:0/2814829235 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x561cea9941d0 0x561cea9fc2f0 crc :-1 s=READY pgs=297 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d65de71604687abc server_cookie=5a0d618d60cafe5f in_seq=0 out_seq=0 2024-03-31T23:36:22.857 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:22.855+0000 7f7895ffb640 1 -- 172.21.15.46:0/2814829235 <== mds.0 v2:172.21.15.142:6838/2519143918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x561cea8e5180 con 0x561cea9941d0 2024-03-31T23:36:22.857 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:22.856+0000 7f7895ffb640 10 client.27850 handle_command_reply: tid=0 2024-03-31T23:36:22.857 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:22.856+0000 7f78b7a03740 1 client.27850 shutdown 2024-03-31T23:36:22.858 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:22.856+0000 7f78b7a03740 1 -- 172.21.15.46:0/2814829235 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561cea97a6c0 msgr2=0x561cea986fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:36:22.858 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:22.856+0000 7f78b7a03740 1 --2- 172.21.15.46:0/2814829235 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561cea97a6c0 0x561cea986fe0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f78a00bdae0 tx=0x7f78a0005fd0 comp rx=0 tx=0).stop 2024-03-31T23:36:22.858 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:22.856+0000 7f78b7a03740 1 -- 172.21.15.46:0/2814829235 shutdown_connections 2024-03-31T23:36:22.858 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:22.856+0000 7f78b7a03740 1 --2- 172.21.15.46:0/2814829235 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x561cea9941d0 0x561cea9fc2f0 crc :-1 s=READY pgs=297 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:36:22.858 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:22.856+0000 7f78b7a03740 1 --2- 172.21.15.46:0/2814829235 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561cea983db0 0x561cea9f8520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:36:22.858 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:22.856+0000 7f78b7a03740 1 --2- 172.21.15.46:0/2814829235 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x561cea97afe0 0x561cea987520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:36:22.858 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:22.856+0000 7f78b7a03740 1 --2- 172.21.15.46:0/2814829235 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561cea97a6c0 0x561cea986fe0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:36:22.858 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:22.856+0000 7f78b7a03740 1 -- 172.21.15.46:0/2814829235 >> 172.21.15.46:0/2814829235 conn(0x561cea974fe0 msgr2=0x561cea97ff70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:36:22.858 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:22.856+0000 7f7895ffb640 0 client.27850 ms_handle_reset on v2:172.21.15.142:6838/2519143918 2024-03-31T23:36:22.858 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:22.856+0000 7f78b7a03740 1 -- 172.21.15.46:0/2814829235 shutdown_connections 2024-03-31T23:36:22.858 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:22.856+0000 7f78b7a03740 1 -- 172.21.15.46:0/2814829235 wait complete. 2024-03-31T23:36:22.858 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:22.857+0000 7f78b7a03740 20 client.27850 trim_cache size 0 max 16384 2024-03-31T23:36:22.859 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout:{ 2024-03-31T23:36:22.859 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout: "message": "", 2024-03-31T23:36:22.859 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout: "return_code": 0 2024-03-31T23:36:22.859 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout:} 2024-03-31T23:36:22.860 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:22.857+0000 7f78b4d3e640 1 -- 172.21.15.46:0/4072287645 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f78980a2730 msgr2=0x7f78980a4bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:36:22.860 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:22.857+0000 7f78b4d3e640 1 --2- 172.21.15.46:0/4072287645 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f78980a2730 0x7f78980a4bf0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f78a8007fd0 tx=0x7f78a800a000 comp rx=0 tx=0).stop 2024-03-31T23:36:22.860 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:22.857+0000 7f78b4d3e640 1 -- 172.21.15.46:0/4072287645 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f78b0100630 msgr2=0x7f78b019e790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:36:22.860 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:22.857+0000 7f78b4d3e640 1 --2- 172.21.15.46:0/4072287645 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f78b0100630 0x7f78b019e790 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f78a0036040 tx=0x7f78a00045e0 comp rx=0 tx=0).stop 2024-03-31T23:36:22.860 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:22.857+0000 7f78b4d3e640 1 -- 172.21.15.46:0/4072287645 shutdown_connections 2024-03-31T23:36:22.860 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:22.857+0000 7f78b4d3e640 1 --2- 172.21.15.46:0/4072287645 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f78980a2730 0x7f78980a4bf0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:36:22.860 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:22.857+0000 7f78b4d3e640 1 --2- 172.21.15.46:0/4072287645 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f78b0106640 0x7f78b01a2dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:36:22.860 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:22.857+0000 7f78b4d3e640 1 --2- 172.21.15.46:0/4072287645 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f78b0100ff0 0x7f78b019ecd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:36:22.860 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:22.857+0000 7f78b4d3e640 1 --2- 172.21.15.46:0/4072287645 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f78b0100630 0x7f78b019e790 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:36:22.860 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:22.857+0000 7f78b4d3e640 1 -- 172.21.15.46:0/4072287645 >> 172.21.15.46:0/4072287645 conn(0x7f78b00777b0 msgr2=0x7f78b010bba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:36:22.861 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:22.858+0000 7f78b4d3e640 1 -- 172.21.15.46:0/4072287645 shutdown_connections 2024-03-31T23:36:22.861 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:22.858+0000 7f78b4d3e640 1 -- 172.21.15.46:0/4072287645 wait complete. 2024-03-31T23:36:23.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:36:22 smithi162 ceph-mon[36237]: pgmap v783: 129 pgs: 129 active+clean; 1.3 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 5.6 MiB/s wr, 293 op/s 2024-03-31T23:36:23.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:36:22 smithi046 ceph-mon[32498]: pgmap v783: 129 pgs: 129 active+clean; 1.3 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 5.6 MiB/s wr, 293 op/s 2024-03-31T23:36:23.194 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:36:22 smithi142 ceph-mon[35271]: pgmap v783: 129 pgs: 129 active+clean; 1.3 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 5.6 MiB/s wr, 293 op/s 2024-03-31T23:36:23.318 INFO:tasks.workunit.client.0.smithi046.stdout: 10 105128 8.12 MB/sec execute 568 sec latency 197.367 ms 2024-03-31T23:36:24.318 INFO:tasks.workunit.client.0.smithi046.stdout: 10 105243 8.11 MB/sec execute 569 sec latency 105.487 ms 2024-03-31T23:36:25.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:36:24 smithi162 ceph-mon[36237]: pgmap v784: 129 pgs: 129 active+clean; 1.3 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 4.9 MiB/s wr, 253 op/s 2024-03-31T23:36:25.113 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:36:24 smithi046 ceph-mon[32498]: pgmap v784: 129 pgs: 129 active+clean; 1.3 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 4.9 MiB/s wr, 253 op/s 2024-03-31T23:36:25.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:36:24 smithi142 ceph-mon[35271]: pgmap v784: 129 pgs: 129 active+clean; 1.3 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 4.9 MiB/s wr, 253 op/s 2024-03-31T23:36:25.318 INFO:tasks.workunit.client.0.smithi046.stdout: 10 105388 8.11 MB/sec execute 570 sec latency 105.365 ms 2024-03-31T23:36:26.318 INFO:tasks.workunit.client.0.smithi046.stdout: 10 105555 8.12 MB/sec execute 571 sec latency 101.930 ms 2024-03-31T23:36:27.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:36:26 smithi046 ceph-mon[32498]: pgmap v785: 129 pgs: 129 active+clean; 1.3 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 7.7 MiB/s wr, 408 op/s 2024-03-31T23:36:27.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:36:26 smithi142 ceph-mon[35271]: pgmap v785: 129 pgs: 129 active+clean; 1.3 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 7.7 MiB/s wr, 408 op/s 2024-03-31T23:36:27.295 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:36:26 smithi162 ceph-mon[36237]: pgmap v785: 129 pgs: 129 active+clean; 1.3 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 7.7 MiB/s wr, 408 op/s 2024-03-31T23:36:27.318 INFO:tasks.workunit.client.0.smithi046.stdout: 10 105678 8.12 MB/sec execute 572 sec latency 74.504 ms 2024-03-31T23:36:28.319 INFO:tasks.workunit.client.0.smithi046.stdout: 10 105795 8.12 MB/sec execute 573 sec latency 179.149 ms 2024-03-31T23:36:29.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:36:28 smithi046 ceph-mon[32498]: pgmap v786: 129 pgs: 129 active+clean; 1.3 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 5.4 MiB/s wr, 260 op/s 2024-03-31T23:36:29.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:36:28 smithi142 ceph-mon[35271]: pgmap v786: 129 pgs: 129 active+clean; 1.3 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 5.4 MiB/s wr, 260 op/s 2024-03-31T23:36:29.295 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:36:28 smithi162 ceph-mon[36237]: pgmap v786: 129 pgs: 129 active+clean; 1.3 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 5.4 MiB/s wr, 260 op/s 2024-03-31T23:36:29.318 INFO:tasks.workunit.client.0.smithi046.stdout: 10 105923 8.11 MB/sec execute 574 sec latency 180.473 ms 2024-03-31T23:36:30.318 INFO:tasks.workunit.client.0.smithi046.stdout: 10 106054 8.10 MB/sec execute 575 sec latency 91.416 ms 2024-03-31T23:36:31.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:36:30 smithi046 ceph-mon[32498]: pgmap v787: 129 pgs: 129 active+clean; 1.3 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 8.0 MiB/s wr, 382 op/s 2024-03-31T23:36:31.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:36:30 smithi142 ceph-mon[35271]: pgmap v787: 129 pgs: 129 active+clean; 1.3 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 8.0 MiB/s wr, 382 op/s 2024-03-31T23:36:31.295 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:36:30 smithi162 ceph-mon[36237]: pgmap v787: 129 pgs: 129 active+clean; 1.3 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 8.0 MiB/s wr, 382 op/s 2024-03-31T23:36:31.318 INFO:tasks.workunit.client.0.smithi046.stdout: 10 106198 8.10 MB/sec execute 576 sec latency 87.034 ms 2024-03-31T23:36:32.318 INFO:tasks.workunit.client.0.smithi046.stdout: 10 106352 8.09 MB/sec execute 577 sec latency 105.272 ms 2024-03-31T23:36:33.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:36:32 smithi046 ceph-mon[32498]: pgmap v788: 129 pgs: 129 active+clean; 1.3 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 5.4 MiB/s wr, 277 op/s 2024-03-31T23:36:33.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:36:32 smithi142 ceph-mon[35271]: pgmap v788: 129 pgs: 129 active+clean; 1.3 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 5.4 MiB/s wr, 277 op/s 2024-03-31T23:36:33.295 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:36:32 smithi162 ceph-mon[36237]: pgmap v788: 129 pgs: 129 active+clean; 1.3 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 5.4 MiB/s wr, 277 op/s 2024-03-31T23:36:33.318 INFO:tasks.workunit.client.0.smithi046.stdout: 10 106487 8.08 MB/sec execute 578 sec latency 96.342 ms 2024-03-31T23:36:34.318 INFO:tasks.workunit.client.0.smithi046.stdout: 10 106632 8.08 MB/sec execute 579 sec latency 162.797 ms 2024-03-31T23:36:35.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:36:34 smithi046 ceph-mon[32498]: pgmap v789: 129 pgs: 129 active+clean; 1.3 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 5.4 MiB/s wr, 277 op/s 2024-03-31T23:36:35.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:36:34 smithi142 ceph-mon[35271]: pgmap v789: 129 pgs: 129 active+clean; 1.3 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 5.4 MiB/s wr, 277 op/s 2024-03-31T23:36:35.295 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:36:34 smithi162 ceph-mon[36237]: pgmap v789: 129 pgs: 129 active+clean; 1.3 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 5.4 MiB/s wr, 277 op/s 2024-03-31T23:36:35.318 INFO:tasks.workunit.client.0.smithi046.stdout: 10 106794 8.08 MB/sec execute 580 sec latency 124.285 ms 2024-03-31T23:36:35.998 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:35.996+0000 7fbd51e6f640 1 -- 172.21.15.46:0/817631672 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbd4c106640 msgr2=0x7fbd4c106a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:36:35.998 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:35.996+0000 7fbd51e6f640 1 --2- 172.21.15.46:0/817631672 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbd4c106640 0x7fbd4c106a20 secure :-1 s=READY pgs=1048 cs=0 l=1 rev1=1 crypto rx=0x7fbd3c009930 tx=0x7fbd3c02bc30 comp rx=0 tx=0).stop 2024-03-31T23:36:35.999 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:35.996+0000 7fbd51e6f640 1 -- 172.21.15.46:0/817631672 shutdown_connections 2024-03-31T23:36:35.999 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:35.996+0000 7fbd51e6f640 1 --2- 172.21.15.46:0/817631672 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbd4c100ff0 0x7fbd4c10d500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:36:35.999 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:35.996+0000 7fbd51e6f640 1 --2- 172.21.15.46:0/817631672 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbd4c100630 0x7fbd4c100ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:36:35.999 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:35.996+0000 7fbd51e6f640 1 --2- 172.21.15.46:0/817631672 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbd4c106640 0x7fbd4c106a20 unknown :-1 s=CLOSED pgs=1048 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:36:35.999 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:35.996+0000 7fbd51e6f640 1 -- 172.21.15.46:0/817631672 >> 172.21.15.46:0/817631672 conn(0x7fbd4c0777b0 msgr2=0x7fbd4c077bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:36:35.999 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:35.996+0000 7fbd51e6f640 1 -- 172.21.15.46:0/817631672 shutdown_connections 2024-03-31T23:36:35.999 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:35.996+0000 7fbd51e6f640 1 -- 172.21.15.46:0/817631672 wait complete. 2024-03-31T23:36:36.002 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:35.997+0000 7fbd51e6f640 1 Processor -- start 2024-03-31T23:36:36.002 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:35.997+0000 7fbd51e6f640 1 -- start start 2024-03-31T23:36:36.003 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:35.998+0000 7fbd51e6f640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbd4c100630 0x7fbd4c0fd5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:36:36.003 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:35.998+0000 7fbd51e6f640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbd4c100ff0 0x7fbd4c0fdb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:36:36.003 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:35.998+0000 7fbd51e6f640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbd4c106640 0x7fbd4c0fe070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:36:36.003 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:35.998+0000 7fbd51e6f640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fbd4c110950 con 0x7fbd4c106640 2024-03-31T23:36:36.003 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:35.998+0000 7fbd51e6f640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fbd4c1107d0 con 0x7fbd4c100630 2024-03-31T23:36:36.003 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:35.998+0000 7fbd51e6f640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fbd4c110ad0 con 0x7fbd4c100ff0 2024-03-31T23:36:36.003 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:35.998+0000 7fbd4bfff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbd4c106640 0x7fbd4c0fe070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:36:36.003 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:35.998+0000 7fbd4bfff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbd4c106640 0x7fbd4c0fe070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:42630/0 (socket says 172.21.15.46:42630) 2024-03-31T23:36:36.003 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:35.998+0000 7fbd4bfff640 1 -- 172.21.15.46:0/3488251958 learned_addr learned my addr 172.21.15.46:0/3488251958 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:36:36.004 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:35.998+0000 7fbd4affd640 1 --2- 172.21.15.46:0/3488251958 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbd4c100ff0 0x7fbd4c0fdb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:36:36.004 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:35.998+0000 7fbd4b7fe640 1 --2- 172.21.15.46:0/3488251958 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbd4c100630 0x7fbd4c0fd5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:36:36.004 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:35.998+0000 7fbd4bfff640 1 -- 172.21.15.46:0/3488251958 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbd4c100ff0 msgr2=0x7fbd4c0fdb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:36:36.004 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:35.998+0000 7fbd4bfff640 1 --2- 172.21.15.46:0/3488251958 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbd4c100ff0 0x7fbd4c0fdb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:36:36.004 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:35.998+0000 7fbd4bfff640 1 -- 172.21.15.46:0/3488251958 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbd4c100630 msgr2=0x7fbd4c0fd5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:36:36.004 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:35.998+0000 7fbd4bfff640 1 --2- 172.21.15.46:0/3488251958 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbd4c100630 0x7fbd4c0fd5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:36:36.004 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:35.998+0000 7fbd4bfff640 1 -- 172.21.15.46:0/3488251958 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbd4c1a6480 con 0x7fbd4c106640 2024-03-31T23:36:36.005 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:35.999+0000 7fbd4bfff640 1 --2- 172.21.15.46:0/3488251958 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbd4c106640 0x7fbd4c0fe070 secure :-1 s=READY pgs=1049 cs=0 l=1 rev1=1 crypto rx=0x7fbd44009fd0 tx=0x7fbd4400b260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:36:36.005 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:35.999+0000 7fbd48ff9640 1 -- 172.21.15.46:0/3488251958 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd44017070 con 0x7fbd4c106640 2024-03-31T23:36:36.005 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:35.999+0000 7fbd48ff9640 1 -- 172.21.15.46:0/3488251958 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fbd44003e20 con 0x7fbd4c106640 2024-03-31T23:36:36.005 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:35.999+0000 7fbd48ff9640 1 -- 172.21.15.46:0/3488251958 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd44010460 con 0x7fbd4c106640 2024-03-31T23:36:36.005 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:35.999+0000 7fbd51e6f640 1 -- 172.21.15.46:0/3488251958 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbd4c1a6770 con 0x7fbd4c106640 2024-03-31T23:36:36.005 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:35.999+0000 7fbd51e6f640 1 -- 172.21.15.46:0/3488251958 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbd4c1a6c70 con 0x7fbd4c106640 2024-03-31T23:36:36.015 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:36.002+0000 7fbd48ff9640 1 -- 172.21.15.46:0/3488251958 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 25) ==== 137451+0+0 (secure 0 0 0) 0x7fbd44003970 con 0x7fbd4c106640 2024-03-31T23:36:36.015 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:36.003+0000 7fbd54a52740 1 Processor -- start 2024-03-31T23:36:36.015 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:36.003+0000 7fbd48ff9640 1 --2- 172.21.15.46:0/3488251958 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fbd340a2730 0x7fbd340a4bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:36:36.016 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:36.003+0000 7fbd54a52740 1 -- start start 2024-03-31T23:36:36.016 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:36.003+0000 7fbd48ff9640 1 -- 172.21.15.46:0/3488251958 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(137..137 src has 1..137) ==== 9554+0+0 (secure 0 0 0) 0x7fbd440cec20 con 0x7fbd4c106640 2024-03-31T23:36:36.016 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:36.003+0000 7fbd4affd640 1 --2- 172.21.15.46:0/3488251958 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fbd340a2730 0x7fbd340a4bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:36:36.016 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:36.003+0000 7fbd54a52740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5572a7ad36c0 0x5572a7ad3aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:36:36.016 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:36.003+0000 7fbd54a52740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5572a7ad3fe0 0x5572a7adc870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:36:36.016 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:36.003+0000 7fbd54a52740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5572a7adcdb0 0x5572a7adf1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:36:36.017 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:36.003+0000 7fbd54a52740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x5572a7a368f0 con 0x5572a7adcdb0 2024-03-31T23:36:36.017 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:36.003+0000 7fbd54a52740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x5572a79df8d0 con 0x5572a7ad3fe0 2024-03-31T23:36:36.017 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:36.003+0000 7fbd54a52740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5572a7a32990 con 0x5572a7ad36c0 2024-03-31T23:36:36.017 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:36.003+0000 7fbd4bfff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5572a7adcdb0 0x5572a7adf1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:36:36.017 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:36.003+0000 7fbd4b7fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5572a7ad36c0 0x5572a7ad3aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:36:36.017 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:36.003+0000 7fbd4b7fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5572a7ad36c0 0x5572a7ad3aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.46:35692/0 (socket says 172.21.15.46:35692) 2024-03-31T23:36:36.017 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:36.003+0000 7fbd4bfff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5572a7adcdb0 0x5572a7adf1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:42642/0 (socket says 172.21.15.46:42642) 2024-03-31T23:36:36.017 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:36.003+0000 7fbd4b7fe640 1 -- 172.21.15.46:0/1478920309 learned_addr learned my addr 172.21.15.46:0/1478920309 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:36:36.018 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:36.003+0000 7fbd4affd640 1 --2- 172.21.15.46:0/1478920309 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5572a7ad3fe0 0x5572a7adc870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:36:36.018 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:36.004+0000 7fbd4b7fe640 1 -- 172.21.15.46:0/1478920309 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5572a7ad3fe0 msgr2=0x5572a7adc870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:36:36.018 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:36.004+0000 7fbd4b7fe640 1 --2- 172.21.15.46:0/1478920309 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5572a7ad3fe0 0x5572a7adc870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:36:36.018 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:36.004+0000 7fbd4b7fe640 1 -- 172.21.15.46:0/1478920309 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5572a7adcdb0 msgr2=0x5572a7adf1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:36:36.018 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:36.004+0000 7fbd4b7fe640 1 --2- 172.21.15.46:0/1478920309 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5572a7adcdb0 0x5572a7adf1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:36:36.018 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:36.004+0000 7fbd4b7fe640 1 -- 172.21.15.46:0/1478920309 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5572a774b210 con 0x5572a7ad36c0 2024-03-31T23:36:36.018 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:36.004+0000 7fbd4affd640 1 --2- 172.21.15.46:0/3488251958 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fbd340a2730 0x7fbd340a4bf0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fbd4c0ff080 tx=0x7fbd40013830 comp rx=0 tx=0).ready entity=mgr.15654 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:36:36.019 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:36.004+0000 7fbd4b7fe640 1 --2- 172.21.15.46:0/1478920309 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5572a7ad36c0 0x5572a7ad3aa0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fbd3c005ab0 tx=0x7fbd3c02c650 comp rx=0 tx=0).ready entity=mon.1 client_cookie=24d8b3b58a9a7c0f server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:36:36.019 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:36.004+0000 7fbd31ffb640 1 -- 172.21.15.46:0/1478920309 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd3c045020 con 0x5572a7ad36c0 2024-03-31T23:36:36.019 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:36.004+0000 7fbd31ffb640 1 -- 172.21.15.46:0/1478920309 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fbd3c02c7e0 con 0x5572a7ad36c0 2024-03-31T23:36:36.019 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:36.005+0000 7fbd31ffb640 1 -- 172.21.15.46:0/1478920309 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd3c005570 con 0x5572a7ad36c0 2024-03-31T23:36:36.019 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:36.005+0000 7fbd54a52740 1 -- 172.21.15.46:0/1478920309 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5572a7ad36c0 msgr2=0x5572a7ad3aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:36:36.019 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:36.005+0000 7fbd54a52740 1 --2- 172.21.15.46:0/1478920309 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5572a7ad36c0 0x5572a7ad3aa0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fbd3c005ab0 tx=0x7fbd3c02c650 comp rx=0 tx=0).stop 2024-03-31T23:36:36.019 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:36.005+0000 7fbd54a52740 1 -- 172.21.15.46:0/1478920309 shutdown_connections 2024-03-31T23:36:36.020 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:36.005+0000 7fbd54a52740 1 --2- 172.21.15.46:0/1478920309 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5572a7adcdb0 0x5572a7adf1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:36:36.020 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:36.005+0000 7fbd54a52740 1 --2- 172.21.15.46:0/1478920309 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5572a7ad3fe0 0x5572a7adc870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:36:36.020 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:36.005+0000 7fbd54a52740 1 --2- 172.21.15.46:0/1478920309 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5572a7ad36c0 0x5572a7ad3aa0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:36:36.020 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:36.005+0000 7fbd54a52740 1 -- 172.21.15.46:0/1478920309 >> 172.21.15.46:0/1478920309 conn(0x5572a7acdfe0 msgr2=0x5572a7ace3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:36:36.020 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:36.006+0000 7fbd54a52740 1 -- 172.21.15.46:0/1478920309 shutdown_connections 2024-03-31T23:36:36.020 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:36.006+0000 7fbd54a52740 1 -- 172.21.15.46:0/1478920309 wait complete. 2024-03-31T23:36:36.020 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:36.006+0000 7fbd54a52740 1 -- start start 2024-03-31T23:36:36.020 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:36.006+0000 7fbd54a52740 1 Processor -- start 2024-03-31T23:36:36.021 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:36.007+0000 7fbd54a52740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5572a7ad36c0 0x5572a7ad9e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:36:36.021 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:36.007+0000 7fbd54a52740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5572a7ad3fe0 0x5572a7ada340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:36:36.021 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:36.007+0000 7fbd54a52740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5572a7adcdb0 0x5572a7ada880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:36:36.021 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:36.007+0000 7fbd54a52740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x5572a7adf980 con 0x5572a7ad36c0 2024-03-31T23:36:36.021 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:36.007+0000 7fbd54a52740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x5572a79d4f80 con 0x5572a7ad3fe0 2024-03-31T23:36:36.021 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:36.007+0000 7fbd54a52740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5572a7adfb00 con 0x5572a7adcdb0 2024-03-31T23:36:36.021 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:36.007+0000 7fbd54a52740 10 client.0 _ll_register_callbacks cb 0x5572a7a2d290 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T23:36:36.022 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:36.007+0000 7fbd4b7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5572a7ad36c0 0x5572a7ad9e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:36:36.022 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:36.007+0000 7fbd4b7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5572a7ad36c0 0x5572a7ad9e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:42652/0 (socket says 172.21.15.46:42652) 2024-03-31T23:36:36.022 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:36.007+0000 7fbd4b7fe640 1 -- 172.21.15.46:0/2156607426 learned_addr learned my addr 172.21.15.46:0/2156607426 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:36:36.022 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:36.007+0000 7fbd4affd640 1 --2- 172.21.15.46:0/2156607426 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5572a7ad3fe0 0x5572a7ada340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:36:36.022 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:36.007+0000 7fbd4bfff640 1 --2- 172.21.15.46:0/2156607426 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5572a7adcdb0 0x5572a7ada880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:36:36.022 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:36.007+0000 7fbd4b7fe640 1 -- 172.21.15.46:0/2156607426 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5572a7adcdb0 msgr2=0x5572a7ada880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:36:36.022 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:36.007+0000 7fbd4b7fe640 1 --2- 172.21.15.46:0/2156607426 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5572a7adcdb0 0x5572a7ada880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:36:36.023 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:36.007+0000 7fbd4b7fe640 1 -- 172.21.15.46:0/2156607426 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5572a7ad3fe0 msgr2=0x5572a7ada340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:36:36.023 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:36.007+0000 7fbd4b7fe640 1 --2- 172.21.15.46:0/2156607426 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5572a7ad3fe0 0x5572a7ada340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:36:36.023 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:36.007+0000 7fbd4b7fe640 1 -- 172.21.15.46:0/2156607426 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5572a7817620 con 0x5572a7ad36c0 2024-03-31T23:36:36.023 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:36.008+0000 7fbd4b7fe640 1 --2- 172.21.15.46:0/2156607426 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5572a7ad36c0 0x5572a7ad9e00 secure :-1 s=READY pgs=1050 cs=0 l=1 rev1=1 crypto rx=0x7fbd3c006010 tx=0x7fbd3c0043f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=46ae300490ed7de6 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:36:36.023 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:36.008+0000 7fbd2affd640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-31T23:36:36.023 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:36.008+0000 7fbd2affd640 1 -- 172.21.15.46:0/2156607426 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd3c04e070 con 0x5572a7ad36c0 2024-03-31T23:36:36.023 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:36.008+0000 7fbd2affd640 1 -- 172.21.15.46:0/2156607426 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(137..137 src has 1..137) ==== 9554+0+0 (secure 0 0 0) 0x7fbd3c0037f0 con 0x5572a7ad36c0 2024-03-31T23:36:36.024 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:36.008+0000 7fbd54a52740 1 -- 172.21.15.46:0/2156607426 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbd3c02c7e0 con 0x5572a7ad36c0 2024-03-31T23:36:36.024 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:36.008+0000 7fbd2affd640 1 -- 172.21.15.46:0/2156607426 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=54) ==== 24+0+0 (secure 0 0 0) 0x7fbd3c057050 con 0x5572a7ad36c0 2024-03-31T23:36:36.024 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:36.008+0000 7fbd54a52740 10 client.18153 fetch_fsmap learned FSMap version 54 2024-03-31T23:36:36.024 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:36.008+0000 7fbd54a52740 1 -- 172.21.15.46:0/2156607426 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=54,monmap=4+}) -- 0x5572a7aeb8b0 con 0x5572a7ad36c0 2024-03-31T23:36:36.024 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:36.009+0000 7fbd2affd640 1 -- 172.21.15.46:0/2156607426 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fbd3c02c7e0 con 0x5572a7ad36c0 2024-03-31T23:36:36.024 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:36.009+0000 7fbd2affd640 1 -- 172.21.15.46:0/2156607426 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 54) ==== 8346+0+0 (secure 0 0 0) 0x7fbd3c051020 con 0x5572a7ad36c0 2024-03-31T23:36:36.024 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:36.009+0000 7fbd54a52740 10 client.18153 fetch_fsmap finished waiting for FSMap version 54 2024-03-31T23:36:36.024 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:36.009+0000 7fbd54a52740 10 client.18153 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-31T23:36:36.025 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:36.009+0000 7fbd54a52740 20 client.18153 populate_metadata read hostname 'smithi046' 2024-03-31T23:36:36.025 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:36.009+0000 7fbd54a52740 1 --2- 172.21.15.46:0/2156607426 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x5572a7aecc00 0x5572a7aeeff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:36:36.025 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:36.009+0000 7fbd54a52740 4 client.18153 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-31T23:36:36.025 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:36.009+0000 7fbd54a52740 1 -- 172.21.15.46:0/2156607426 --> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5572a7a4a120 con 0x5572a7aecc00 2024-03-31T23:36:36.025 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:36.009+0000 7fbd4affd640 1 --2- 172.21.15.46:0/2156607426 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x5572a7aecc00 0x5572a7aeeff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:36:36.025 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:36.010+0000 7fbd4affd640 1 --2- 172.21.15.46:0/2156607426 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x5572a7aecc00 0x5572a7aeeff0 crc :-1 s=READY pgs=298 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dd890f9d2ccf85 server_cookie=a483de8688ffdc52 in_seq=0 out_seq=0 2024-03-31T23:36:36.025 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:36.010+0000 7fbd2affd640 10 client.18153 ms_handle_connect on v2:172.21.15.142:6838/2519143918 2024-03-31T23:36:36.026 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:36.010+0000 7fbd2affd640 1 -- 172.21.15.46:0/2156607426 <== mds.0 v2:172.21.15.142:6838/2519143918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5572a7a4a120 con 0x5572a7aecc00 2024-03-31T23:36:36.026 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:36.010+0000 7fbd2affd640 10 client.18153 handle_command_reply: tid=0 2024-03-31T23:36:36.026 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:36.010+0000 7fbd54a52740 1 client.18153 shutdown 2024-03-31T23:36:36.026 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:36.011+0000 7fbd54a52740 1 -- 172.21.15.46:0/2156607426 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5572a7ad36c0 msgr2=0x5572a7ad9e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:36:36.026 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:36.011+0000 7fbd54a52740 1 --2- 172.21.15.46:0/2156607426 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5572a7ad36c0 0x5572a7ad9e00 secure :-1 s=READY pgs=1050 cs=0 l=1 rev1=1 crypto rx=0x7fbd3c006010 tx=0x7fbd3c0043f0 comp rx=0 tx=0).stop 2024-03-31T23:36:36.026 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:36.011+0000 7fbd54a52740 1 -- 172.21.15.46:0/2156607426 shutdown_connections 2024-03-31T23:36:36.026 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:36.011+0000 7fbd54a52740 1 --2- 172.21.15.46:0/2156607426 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x5572a7aecc00 0x5572a7aeeff0 crc :-1 s=READY pgs=298 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:36:36.027 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:36.011+0000 7fbd54a52740 1 --2- 172.21.15.46:0/2156607426 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5572a7adcdb0 0x5572a7ada880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:36:36.027 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:36.011+0000 7fbd54a52740 1 --2- 172.21.15.46:0/2156607426 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5572a7ad3fe0 0x5572a7ada340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:36:36.027 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:36.011+0000 7fbd54a52740 1 --2- 172.21.15.46:0/2156607426 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5572a7ad36c0 0x5572a7ad9e00 unknown :-1 s=CLOSED pgs=1050 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:36:36.027 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:36.011+0000 7fbd54a52740 1 -- 172.21.15.46:0/2156607426 >> 172.21.15.46:0/2156607426 conn(0x5572a7acdfe0 msgr2=0x5572a7acfd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:36:36.027 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:36.011+0000 7fbd2affd640 0 client.18153 ms_handle_reset on v2:172.21.15.142:6838/2519143918 2024-03-31T23:36:36.027 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:36.011+0000 7fbd54a52740 1 -- 172.21.15.46:0/2156607426 shutdown_connections 2024-03-31T23:36:36.027 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:36.011+0000 7fbd54a52740 1 -- 172.21.15.46:0/2156607426 wait complete. 2024-03-31T23:36:36.027 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:36.011+0000 7fbd54a52740 20 client.18153 trim_cache size 0 max 16384 2024-03-31T23:36:36.047 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:36.043+0000 7fbd54a52740 1 Processor -- start 2024-03-31T23:36:36.047 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:36.043+0000 7fbd54a52740 1 -- start start 2024-03-31T23:36:36.047 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:36.043+0000 7fbd54a52740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5572a7ad36c0 0x5572a7adc530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:36:36.047 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:36.043+0000 7fbd54a52740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5572a7ad3fe0 0x5572a7ae8720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:36:36.047 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:36.043+0000 7fbd54a52740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5572a7adcdb0 0x5572a7af45c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:36:36.047 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:36.043+0000 7fbd54a52740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fbd4c1b37a0 con 0x5572a7ad36c0 2024-03-31T23:36:36.048 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:36.043+0000 7fbd54a52740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fbd4c1b1070 con 0x5572a7adcdb0 2024-03-31T23:36:36.048 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:36.043+0000 7fbd54a52740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fbd4c1aeee0 con 0x5572a7ad3fe0 2024-03-31T23:36:36.048 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:36.043+0000 7fbd4b7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5572a7ad36c0 0x5572a7adc530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:36:36.048 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:36.043+0000 7fbd4b7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5572a7ad36c0 0x5572a7adc530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:42660/0 (socket says 172.21.15.46:42660) 2024-03-31T23:36:36.048 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:36.043+0000 7fbd4b7fe640 1 -- 172.21.15.46:0/3086576841 learned_addr learned my addr 172.21.15.46:0/3086576841 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:36:36.048 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:36.043+0000 7fbd4affd640 1 --2- 172.21.15.46:0/3086576841 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5572a7ad3fe0 0x5572a7ae8720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:36:36.048 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:36.043+0000 7fbd4bfff640 1 --2- 172.21.15.46:0/3086576841 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5572a7adcdb0 0x5572a7af45c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:36:36.048 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:36.044+0000 7fbd4b7fe640 1 -- 172.21.15.46:0/3086576841 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5572a7ad3fe0 msgr2=0x5572a7ae8720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:36:36.048 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:36.044+0000 7fbd4b7fe640 1 --2- 172.21.15.46:0/3086576841 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5572a7ad3fe0 0x5572a7ae8720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:36:36.049 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:36.044+0000 7fbd4b7fe640 1 -- 172.21.15.46:0/3086576841 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5572a7adcdb0 msgr2=0x5572a7af45c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:36:36.049 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:36.044+0000 7fbd4b7fe640 1 --2- 172.21.15.46:0/3086576841 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5572a7adcdb0 0x5572a7af45c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:36:36.049 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:36.044+0000 7fbd4b7fe640 1 -- 172.21.15.46:0/3086576841 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5572a7af4b00 con 0x5572a7ad36c0 2024-03-31T23:36:36.049 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:36.044+0000 7fbd4b7fe640 1 --2- 172.21.15.46:0/3086576841 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5572a7ad36c0 0x5572a7adc530 secure :-1 s=READY pgs=1051 cs=0 l=1 rev1=1 crypto rx=0x7fbd3c02c170 tx=0x7fbd3c02c650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=954fe25989ed9f12 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:36:36.049 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:36.044+0000 7fbd317fa640 1 -- 172.21.15.46:0/3086576841 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd3c04e040 con 0x5572a7ad36c0 2024-03-31T23:36:36.049 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:36.044+0000 7fbd317fa640 1 -- 172.21.15.46:0/3086576841 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fbd3c02c7e0 con 0x5572a7ad36c0 2024-03-31T23:36:36.049 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:36.044+0000 7fbd317fa640 1 -- 172.21.15.46:0/3086576841 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd3c045070 con 0x5572a7ad36c0 2024-03-31T23:36:36.049 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:36.046+0000 7fbd54a52740 1 -- 172.21.15.46:0/3086576841 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5572a7ad36c0 msgr2=0x5572a7adc530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:36:36.049 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:36.046+0000 7fbd54a52740 1 --2- 172.21.15.46:0/3086576841 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5572a7ad36c0 0x5572a7adc530 secure :-1 s=READY pgs=1051 cs=0 l=1 rev1=1 crypto rx=0x7fbd3c02c170 tx=0x7fbd3c02c650 comp rx=0 tx=0).stop 2024-03-31T23:36:36.049 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:36.046+0000 7fbd54a52740 1 -- 172.21.15.46:0/3086576841 shutdown_connections 2024-03-31T23:36:36.050 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:36.046+0000 7fbd54a52740 1 --2- 172.21.15.46:0/3086576841 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5572a7adcdb0 0x5572a7af45c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:36:36.050 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:36.046+0000 7fbd54a52740 1 --2- 172.21.15.46:0/3086576841 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5572a7ad3fe0 0x5572a7ae8720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:36:36.050 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:36.046+0000 7fbd54a52740 1 --2- 172.21.15.46:0/3086576841 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5572a7ad36c0 0x5572a7adc530 unknown :-1 s=CLOSED pgs=1051 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:36:36.050 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:36.046+0000 7fbd54a52740 1 -- 172.21.15.46:0/3086576841 >> 172.21.15.46:0/3086576841 conn(0x5572a7acdfe0 msgr2=0x5572a7ace3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:36:36.050 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:36.046+0000 7fbd54a52740 1 -- 172.21.15.46:0/3086576841 shutdown_connections 2024-03-31T23:36:36.050 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:36.046+0000 7fbd54a52740 1 -- 172.21.15.46:0/3086576841 wait complete. 2024-03-31T23:36:36.053 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:36.046+0000 7fbd54a52740 1 -- start start 2024-03-31T23:36:36.053 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:36.047+0000 7fbd54a52740 1 Processor -- start 2024-03-31T23:36:36.053 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:36.047+0000 7fbd54a52740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5572a7ad36c0 0x5572a7ae4f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:36:36.053 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:36.047+0000 7fbd54a52740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5572a7ad3fe0 0x5572a7ad6630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:36:36.053 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:36.047+0000 7fbd54a52740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5572a7adcdb0 0x5572a7ad6b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:36:36.054 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:36.047+0000 7fbd54a52740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fbd4c1af530 con 0x5572a7ad36c0 2024-03-31T23:36:36.054 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:36.047+0000 7fbd4b7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5572a7ad36c0 0x5572a7ae4f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:36:36.054 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:36.047+0000 7fbd54a52740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fbd4c1b09a0 con 0x5572a7adcdb0 2024-03-31T23:36:36.054 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:36.047+0000 7fbd54a52740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fbd4c1afc00 con 0x5572a7ad3fe0 2024-03-31T23:36:36.054 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:36.047+0000 7fbd4b7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5572a7ad36c0 0x5572a7ae4f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:42668/0 (socket says 172.21.15.46:42668) 2024-03-31T23:36:36.054 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:36.047+0000 7fbd4b7fe640 1 -- 172.21.15.46:0/1959197015 learned_addr learned my addr 172.21.15.46:0/1959197015 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:36:36.054 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:36.047+0000 7fbd4affd640 1 --2- 172.21.15.46:0/1959197015 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5572a7ad3fe0 0x5572a7ad6630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:36:36.054 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:36.048+0000 7fbd54a52740 10 client.0 _ll_register_callbacks cb 0x7fbd44004a40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T23:36:36.055 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:36.048+0000 7fbd4bfff640 1 --2- 172.21.15.46:0/1959197015 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5572a7adcdb0 0x5572a7ad6b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:36:36.055 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:36.048+0000 7fbd4affd640 1 -- 172.21.15.46:0/1959197015 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5572a7adcdb0 msgr2=0x5572a7ad6b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:36:36.055 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:36.048+0000 7fbd4affd640 1 --2- 172.21.15.46:0/1959197015 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5572a7adcdb0 0x5572a7ad6b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:36:36.055 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:36.048+0000 7fbd4affd640 1 -- 172.21.15.46:0/1959197015 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5572a7ad36c0 msgr2=0x5572a7ae4f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:36:36.055 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:36.048+0000 7fbd4affd640 1 --2- 172.21.15.46:0/1959197015 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5572a7ad36c0 0x5572a7ae4f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:36:36.055 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:36.048+0000 7fbd4affd640 1 -- 172.21.15.46:0/1959197015 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5572a7ad7230 con 0x5572a7ad3fe0 2024-03-31T23:36:36.055 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:36.048+0000 7fbd4affd640 1 --2- 172.21.15.46:0/1959197015 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5572a7ad3fe0 0x5572a7ad6630 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x5572a7aef530 tx=0x7fbd40004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a0eb4c25ebfde597 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:36:36.055 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:36.048+0000 7fbd2affd640 10 client.0 ms_handle_connect on v2:172.21.15.162:3300/0 2024-03-31T23:36:36.055 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:36.048+0000 7fbd2affd640 1 -- 172.21.15.46:0/1959197015 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd400295c0 con 0x5572a7ad3fe0 2024-03-31T23:36:36.056 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:36.048+0000 7fbd2affd640 1 -- 172.21.15.46:0/1959197015 <== mon.1 v2:172.21.15.162:3300/0 2 ==== osd_map(137..137 src has 1..137) ==== 9554+0+0 (secure 0 0 0) 0x7fbd40029760 con 0x5572a7ad3fe0 2024-03-31T23:36:36.056 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:36.049+0000 7fbd54a52740 1 -- 172.21.15.46:0/1959197015 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbd3c02c7e0 con 0x5572a7ad3fe0 2024-03-31T23:36:36.056 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:36.049+0000 7fbd2affd640 1 -- 172.21.15.46:0/1959197015 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_get_version_reply(handle=1 version=54) ==== 24+0+0 (secure 0 0 0) 0x7fbd4000f930 con 0x5572a7ad3fe0 2024-03-31T23:36:36.056 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:36.049+0000 7fbd54a52740 10 client.27868 fetch_fsmap learned FSMap version 54 2024-03-31T23:36:36.056 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:36.049+0000 7fbd54a52740 1 -- 172.21.15.46:0/1959197015 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,fsmap=54,monmap=4+}) -- 0x5572a7b5b360 con 0x5572a7ad3fe0 2024-03-31T23:36:36.056 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:36.049+0000 7fbd2affd640 1 -- 172.21.15.46:0/1959197015 <== mon.1 v2:172.21.15.162:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fbd3c02c7e0 con 0x5572a7ad3fe0 2024-03-31T23:36:36.056 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:36.049+0000 7fbd2affd640 1 -- 172.21.15.46:0/1959197015 <== mon.1 v2:172.21.15.162:3300/0 5 ==== fsmap(e 54) ==== 8346+0+0 (secure 0 0 0) 0x7fbd4001f070 con 0x5572a7ad3fe0 2024-03-31T23:36:36.056 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:36.049+0000 7fbd54a52740 10 client.27868 fetch_fsmap finished waiting for FSMap version 54 2024-03-31T23:36:36.057 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:36.049+0000 7fbd54a52740 10 client.27868 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-31T23:36:36.057 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:36.049+0000 7fbd54a52740 20 client.27868 populate_metadata read hostname 'smithi046' 2024-03-31T23:36:36.057 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:36.049+0000 7fbd54a52740 1 --2- 172.21.15.46:0/1959197015 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x5572a7b5bae0 0x5572a7b5df00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:36:36.057 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:36.049+0000 7fbd54a52740 4 client.27868 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-31T23:36:36.057 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:36.049+0000 7fbd54a52740 1 -- 172.21.15.46:0/1959197015 --> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5572a7a43e30 con 0x5572a7b5bae0 2024-03-31T23:36:36.057 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:36.050+0000 7fbd4b7fe640 1 --2- 172.21.15.46:0/1959197015 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x5572a7b5bae0 0x5572a7b5df00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:36:36.057 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:36.050+0000 7fbd4b7fe640 1 --2- 172.21.15.46:0/1959197015 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x5572a7b5bae0 0x5572a7b5df00 crc :-1 s=READY pgs=299 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=80b8320376d9127 server_cookie=e9d2aff266f3b5a in_seq=0 out_seq=0 2024-03-31T23:36:36.057 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:36.050+0000 7fbd2affd640 10 client.27868 ms_handle_connect on v2:172.21.15.142:6838/2519143918 2024-03-31T23:36:36.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:36:35 smithi046 ceph-mon[32498]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-31T23:36:36.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:36:35 smithi142 ceph-mon[35271]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-31T23:36:36.295 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:36:35 smithi162 ceph-mon[36237]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-31T23:36:36.305 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:36.303+0000 7fbd2affd640 1 -- 172.21.15.46:0/1959197015 <== mds.0 v2:172.21.15.142:6838/2519143918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5572a7a43e30 con 0x5572a7b5bae0 2024-03-31T23:36:36.305 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:36.303+0000 7fbd2affd640 10 client.27868 handle_command_reply: tid=0 2024-03-31T23:36:36.305 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:36.303+0000 7fbd54a52740 1 client.27868 shutdown 2024-03-31T23:36:36.305 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:36.304+0000 7fbd54a52740 1 -- 172.21.15.46:0/1959197015 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5572a7ad3fe0 msgr2=0x5572a7ad6630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:36:36.305 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:36.304+0000 7fbd54a52740 1 --2- 172.21.15.46:0/1959197015 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5572a7ad3fe0 0x5572a7ad6630 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x5572a7aef530 tx=0x7fbd40004730 comp rx=0 tx=0).stop 2024-03-31T23:36:36.305 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:36.304+0000 7fbd54a52740 1 -- 172.21.15.46:0/1959197015 shutdown_connections 2024-03-31T23:36:36.305 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:36.304+0000 7fbd54a52740 1 --2- 172.21.15.46:0/1959197015 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x5572a7b5bae0 0x5572a7b5df00 crc :-1 s=READY pgs=299 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:36:36.305 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:36.304+0000 7fbd54a52740 1 --2- 172.21.15.46:0/1959197015 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5572a7adcdb0 0x5572a7ad6b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:36:36.306 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:36.304+0000 7fbd54a52740 1 --2- 172.21.15.46:0/1959197015 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5572a7ad3fe0 0x5572a7ad6630 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:36:36.306 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:36.304+0000 7fbd54a52740 1 --2- 172.21.15.46:0/1959197015 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5572a7ad36c0 0x5572a7ae4f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:36:36.306 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:36.304+0000 7fbd54a52740 1 -- 172.21.15.46:0/1959197015 >> 172.21.15.46:0/1959197015 conn(0x5572a7acdfe0 msgr2=0x5572a7ad5130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:36:36.306 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:36.304+0000 7fbd2affd640 0 client.27868 ms_handle_reset on v2:172.21.15.142:6838/2519143918 2024-03-31T23:36:36.306 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:36.304+0000 7fbd54a52740 1 -- 172.21.15.46:0/1959197015 shutdown_connections 2024-03-31T23:36:36.306 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:36.304+0000 7fbd54a52740 1 -- 172.21.15.46:0/1959197015 wait complete. 2024-03-31T23:36:36.306 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:36.304+0000 7fbd54a52740 20 client.27868 trim_cache size 0 max 16384 2024-03-31T23:36:36.306 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout:{ 2024-03-31T23:36:36.306 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout: "message": "", 2024-03-31T23:36:36.306 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout: "return_code": 0 2024-03-31T23:36:36.307 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout:} 2024-03-31T23:36:36.307 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:36.305+0000 7fbd51e6f640 1 -- 172.21.15.46:0/3488251958 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fbd340a2730 msgr2=0x7fbd340a4bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:36:36.307 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:36.305+0000 7fbd51e6f640 1 --2- 172.21.15.46:0/3488251958 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fbd340a2730 0x7fbd340a4bf0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fbd4c0ff080 tx=0x7fbd40013830 comp rx=0 tx=0).stop 2024-03-31T23:36:36.307 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:36.305+0000 7fbd51e6f640 1 -- 172.21.15.46:0/3488251958 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbd4c106640 msgr2=0x7fbd4c0fe070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:36:36.308 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:36.305+0000 7fbd51e6f640 1 --2- 172.21.15.46:0/3488251958 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbd4c106640 0x7fbd4c0fe070 secure :-1 s=READY pgs=1049 cs=0 l=1 rev1=1 crypto rx=0x7fbd44009fd0 tx=0x7fbd4400b260 comp rx=0 tx=0).stop 2024-03-31T23:36:36.308 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:36.305+0000 7fbd51e6f640 1 -- 172.21.15.46:0/3488251958 shutdown_connections 2024-03-31T23:36:36.308 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:36.305+0000 7fbd51e6f640 1 --2- 172.21.15.46:0/3488251958 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fbd340a2730 0x7fbd340a4bf0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:36:36.308 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:36.305+0000 7fbd51e6f640 1 --2- 172.21.15.46:0/3488251958 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbd4c106640 0x7fbd4c0fe070 unknown :-1 s=CLOSED pgs=1049 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:36:36.308 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:36.305+0000 7fbd51e6f640 1 --2- 172.21.15.46:0/3488251958 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbd4c100ff0 0x7fbd4c0fdb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:36:36.308 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:36.305+0000 7fbd51e6f640 1 --2- 172.21.15.46:0/3488251958 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbd4c100630 0x7fbd4c0fd5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:36:36.308 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:36.305+0000 7fbd51e6f640 1 -- 172.21.15.46:0/3488251958 >> 172.21.15.46:0/3488251958 conn(0x7fbd4c0777b0 msgr2=0x7fbd4c069ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:36:36.308 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:36.305+0000 7fbd51e6f640 1 -- 172.21.15.46:0/3488251958 shutdown_connections 2024-03-31T23:36:36.308 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:36.305+0000 7fbd51e6f640 1 -- 172.21.15.46:0/3488251958 wait complete. 2024-03-31T23:36:36.318 INFO:tasks.workunit.client.0.smithi046.stdout: 10 106928 8.08 MB/sec execute 581 sec latency 180.703 ms 2024-03-31T23:36:37.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:36:36 smithi046 ceph-mon[32498]: pgmap v790: 129 pgs: 129 active+clean; 1.3 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 7.2 MiB/s wr, 390 op/s 2024-03-31T23:36:37.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:36:36 smithi046 ceph-mon[32498]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:36:37.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:36:36 smithi046 ceph-mon[32498]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-31T23:36:37.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:36:36 smithi046 ceph-mon[32498]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' 2024-03-31T23:36:37.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:36:36 smithi142 ceph-mon[35271]: pgmap v790: 129 pgs: 129 active+clean; 1.3 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 7.2 MiB/s wr, 390 op/s 2024-03-31T23:36:37.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:36:36 smithi142 ceph-mon[35271]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:36:37.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:36:36 smithi142 ceph-mon[35271]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-31T23:36:37.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:36:36 smithi142 ceph-mon[35271]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' 2024-03-31T23:36:37.295 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:36:36 smithi162 ceph-mon[36237]: pgmap v790: 129 pgs: 129 active+clean; 1.3 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 7.2 MiB/s wr, 390 op/s 2024-03-31T23:36:37.295 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:36:36 smithi162 ceph-mon[36237]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:36:37.295 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:36:36 smithi162 ceph-mon[36237]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-31T23:36:37.296 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:36:36 smithi162 ceph-mon[36237]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' 2024-03-31T23:36:37.318 INFO:tasks.workunit.client.0.smithi046.stdout: 10 107097 8.09 MB/sec execute 582 sec latency 86.189 ms 2024-03-31T23:36:38.318 INFO:tasks.workunit.client.0.smithi046.stdout: 10 107282 8.09 MB/sec execute 583 sec latency 95.141 ms 2024-03-31T23:36:39.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:36:38 smithi046 ceph-mon[32498]: pgmap v791: 129 pgs: 129 active+clean; 1.3 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 4.4 MiB/s wr, 235 op/s 2024-03-31T23:36:39.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:36:38 smithi142 ceph-mon[35271]: pgmap v791: 129 pgs: 129 active+clean; 1.3 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 4.4 MiB/s wr, 235 op/s 2024-03-31T23:36:39.295 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:36:38 smithi162 ceph-mon[36237]: pgmap v791: 129 pgs: 129 active+clean; 1.3 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 4.4 MiB/s wr, 235 op/s 2024-03-31T23:36:39.318 INFO:tasks.workunit.client.0.smithi046.stdout: 10 107449 8.10 MB/sec execute 584 sec latency 93.307 ms 2024-03-31T23:36:40.318 INFO:tasks.workunit.client.0.smithi046.stdout: 10 107571 8.10 MB/sec execute 585 sec latency 168.541 ms 2024-03-31T23:36:41.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:36:40 smithi046 ceph-mon[32498]: pgmap v792: 129 pgs: 129 active+clean; 1.3 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 7.1 MiB/s wr, 388 op/s 2024-03-31T23:36:41.196 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:36:40 smithi142 ceph-mon[35271]: pgmap v792: 129 pgs: 129 active+clean; 1.3 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 7.1 MiB/s wr, 388 op/s 2024-03-31T23:36:41.296 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:36:40 smithi162 ceph-mon[36237]: pgmap v792: 129 pgs: 129 active+clean; 1.3 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 7.1 MiB/s wr, 388 op/s 2024-03-31T23:36:41.319 INFO:tasks.workunit.client.0.smithi046.stdout: 10 107739 8.10 MB/sec execute 586 sec latency 176.795 ms 2024-03-31T23:36:42.318 INFO:tasks.workunit.client.0.smithi046.stdout: 10 107873 8.10 MB/sec execute 587 sec latency 124.013 ms 2024-03-31T23:36:43.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:36:42 smithi046 ceph-mon[32498]: pgmap v793: 129 pgs: 129 active+clean; 1.3 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 4.5 MiB/s wr, 267 op/s 2024-03-31T23:36:43.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:36:42 smithi142 ceph-mon[35271]: pgmap v793: 129 pgs: 129 active+clean; 1.3 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 4.5 MiB/s wr, 267 op/s 2024-03-31T23:36:43.295 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:36:42 smithi162 ceph-mon[36237]: pgmap v793: 129 pgs: 129 active+clean; 1.3 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 4.5 MiB/s wr, 267 op/s 2024-03-31T23:36:43.319 INFO:tasks.workunit.client.0.smithi046.stdout: 10 107982 8.09 MB/sec execute 588 sec latency 156.979 ms 2024-03-31T23:36:44.318 INFO:tasks.workunit.client.0.smithi046.stdout: 10 108144 8.09 MB/sec execute 589 sec latency 96.145 ms 2024-03-31T23:36:45.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:36:44 smithi046 ceph-mon[32498]: pgmap v794: 129 pgs: 129 active+clean; 1.3 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 4.5 MiB/s wr, 267 op/s 2024-03-31T23:36:45.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:36:44 smithi142 ceph-mon[35271]: pgmap v794: 129 pgs: 129 active+clean; 1.3 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 4.5 MiB/s wr, 267 op/s 2024-03-31T23:36:45.295 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:36:44 smithi162 ceph-mon[36237]: pgmap v794: 129 pgs: 129 active+clean; 1.3 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 4.5 MiB/s wr, 267 op/s 2024-03-31T23:36:45.318 INFO:tasks.workunit.client.0.smithi046.stdout: 10 108306 8.10 MB/sec execute 590 sec latency 97.539 ms 2024-03-31T23:36:46.318 INFO:tasks.workunit.client.0.smithi046.stdout: 10 108442 8.09 MB/sec execute 591 sec latency 80.544 ms 2024-03-31T23:36:47.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:36:46 smithi046 ceph-mon[32498]: pgmap v795: 129 pgs: 129 active+clean; 1.3 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 6.7 MiB/s wr, 352 op/s 2024-03-31T23:36:47.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:36:46 smithi142 ceph-mon[35271]: pgmap v795: 129 pgs: 129 active+clean; 1.3 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 6.7 MiB/s wr, 352 op/s 2024-03-31T23:36:47.295 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:36:46 smithi162 ceph-mon[36237]: pgmap v795: 129 pgs: 129 active+clean; 1.3 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 6.7 MiB/s wr, 352 op/s 2024-03-31T23:36:47.318 INFO:tasks.workunit.client.0.smithi046.stdout: 10 108577 8.10 MB/sec execute 592 sec latency 87.233 ms 2024-03-31T23:36:48.319 INFO:tasks.workunit.client.0.smithi046.stdout: 10 108751 8.11 MB/sec execute 593 sec latency 94.621 ms 2024-03-31T23:36:49.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:36:48 smithi046 ceph-mon[32498]: pgmap v796: 129 pgs: 129 active+clean; 1.3 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 4.8 MiB/s wr, 239 op/s 2024-03-31T23:36:49.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:36:48 smithi142 ceph-mon[35271]: pgmap v796: 129 pgs: 129 active+clean; 1.3 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 4.8 MiB/s wr, 239 op/s 2024-03-31T23:36:49.295 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:36:48 smithi162 ceph-mon[36237]: pgmap v796: 129 pgs: 129 active+clean; 1.3 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 4.8 MiB/s wr, 239 op/s 2024-03-31T23:36:49.319 INFO:tasks.workunit.client.0.smithi046.stdout: 10 108904 8.11 MB/sec execute 594 sec latency 94.479 ms 2024-03-31T23:36:49.424 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:49.423+0000 7f2ff4ddd640 1 -- 172.21.15.46:0/3681267101 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2ff0101260 msgr2=0x7f2ff010d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:36:49.425 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:49.423+0000 7f2ff4ddd640 1 --2- 172.21.15.46:0/3681267101 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2ff0101260 0x7f2ff010d480 secure :-1 s=READY pgs=1052 cs=0 l=1 rev1=1 crypto rx=0x7f2fe8009ca0 tx=0x7f2fe802c0f0 comp rx=0 tx=0).stop 2024-03-31T23:36:49.425 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:49.423+0000 7f2ff4ddd640 1 -- 172.21.15.46:0/3681267101 shutdown_connections 2024-03-31T23:36:49.425 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:49.423+0000 7f2ff4ddd640 1 --2- 172.21.15.46:0/3681267101 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2ff0101260 0x7f2ff010d480 unknown :-1 s=CLOSED pgs=1052 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:36:49.425 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:49.423+0000 7f2ff4ddd640 1 --2- 172.21.15.46:0/3681267101 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2ff01008a0 0x7f2ff0100d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:36:49.425 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:49.423+0000 7f2ff4ddd640 1 --2- 172.21.15.46:0/3681267101 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2ff01055f0 0x7f2ff01059d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:36:49.425 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:49.423+0000 7f2ff4ddd640 1 -- 172.21.15.46:0/3681267101 >> 172.21.15.46:0/3681267101 conn(0x7f2ff00774a0 msgr2=0x7f2ff00778b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:36:49.425 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:49.423+0000 7f2ff4ddd640 1 -- 172.21.15.46:0/3681267101 shutdown_connections 2024-03-31T23:36:49.425 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:49.423+0000 7f2ff4ddd640 1 -- 172.21.15.46:0/3681267101 wait complete. 2024-03-31T23:36:49.434 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:49.424+0000 7f2ff4ddd640 1 Processor -- start 2024-03-31T23:36:49.434 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:49.424+0000 7f2ff4ddd640 1 -- start start 2024-03-31T23:36:49.434 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:49.424+0000 7f2ff4ddd640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2ff01008a0 0x7f2ff0197ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:36:49.434 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:49.424+0000 7f2ff4ddd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2ff0101260 0x7f2ff01983e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:36:49.435 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:49.424+0000 7f2ff4ddd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2ff01055f0 0x7f2ff019c760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:36:49.435 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:49.424+0000 7f2ff4ddd640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f2ff01109c0 con 0x7f2ff01055f0 2024-03-31T23:36:49.435 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:49.424+0000 7f2ff4ddd640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f2ff0110840 con 0x7f2ff01008a0 2024-03-31T23:36:49.435 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:49.424+0000 7f2ff4ddd640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f2ff0110b40 con 0x7f2ff0101260 2024-03-31T23:36:49.435 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:49.424+0000 7f2feed9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2ff01055f0 0x7f2ff019c760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:36:49.435 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:49.424+0000 7f2feed9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2ff01055f0 0x7f2ff019c760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39280/0 (socket says 172.21.15.46:39280) 2024-03-31T23:36:49.435 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:49.424+0000 7f2feed9d640 1 -- 172.21.15.46:0/3442418639 learned_addr learned my addr 172.21.15.46:0/3442418639 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:36:49.435 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:49.424+0000 7f2fe7fff640 1 --2- 172.21.15.46:0/3442418639 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2ff0101260 0x7f2ff01983e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:36:49.435 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:49.424+0000 7f2fee59c640 1 --2- 172.21.15.46:0/3442418639 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2ff01008a0 0x7f2ff0197ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:36:49.435 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:49.425+0000 7f2feed9d640 1 -- 172.21.15.46:0/3442418639 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2ff0101260 msgr2=0x7f2ff01983e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:36:49.435 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:49.425+0000 7f2feed9d640 1 --2- 172.21.15.46:0/3442418639 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2ff0101260 0x7f2ff01983e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:36:49.435 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:49.425+0000 7f2feed9d640 1 -- 172.21.15.46:0/3442418639 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2ff01008a0 msgr2=0x7f2ff0197ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:36:49.435 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:49.425+0000 7f2feed9d640 1 --2- 172.21.15.46:0/3442418639 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2ff01008a0 0x7f2ff0197ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:36:49.435 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:49.425+0000 7f2feed9d640 1 -- 172.21.15.46:0/3442418639 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ff019ce90 con 0x7f2ff01055f0 2024-03-31T23:36:49.436 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:49.425+0000 7f2fe7fff640 1 --2- 172.21.15.46:0/3442418639 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2ff0101260 0x7f2ff01983e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:36:49.436 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:49.425+0000 7f2feed9d640 1 --2- 172.21.15.46:0/3442418639 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2ff01055f0 0x7f2ff019c760 secure :-1 s=READY pgs=1053 cs=0 l=1 rev1=1 crypto rx=0x7f2fe802c700 tx=0x7f2fe8035d60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:36:49.436 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:49.425+0000 7f2fe77fe640 1 -- 172.21.15.46:0/3442418639 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fe8039040 con 0x7f2ff01055f0 2024-03-31T23:36:49.436 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:49.425+0000 7f2fe77fe640 1 -- 172.21.15.46:0/3442418639 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f2fe8044070 con 0x7f2ff01055f0 2024-03-31T23:36:49.436 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:49.425+0000 7f2ff4ddd640 1 -- 172.21.15.46:0/3442418639 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2ff019d120 con 0x7f2ff01055f0 2024-03-31T23:36:49.436 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:49.425+0000 7f2fe77fe640 1 -- 172.21.15.46:0/3442418639 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fe80042a0 con 0x7f2ff01055f0 2024-03-31T23:36:49.436 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:49.425+0000 7f2ff4ddd640 1 -- 172.21.15.46:0/3442418639 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2ff01a4a10 con 0x7f2ff01055f0 2024-03-31T23:36:49.436 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:49.427+0000 7f2fe77fe640 1 -- 172.21.15.46:0/3442418639 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 25) ==== 137451+0+0 (secure 0 0 0) 0x7f2fe8004440 con 0x7f2ff01055f0 2024-03-31T23:36:49.436 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:49.428+0000 7f2ff79b6740 1 Processor -- start 2024-03-31T23:36:49.436 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:49.428+0000 7f2ff79b6740 1 -- start start 2024-03-31T23:36:49.436 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:49.428+0000 7f2fe77fe640 1 --2- 172.21.15.46:0/3442418639 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f2fdc0a26b0 0x7f2fdc0a4b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:36:49.436 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:49.428+0000 7f2ff79b6740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5587435346c0 0x558743534aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:36:49.436 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:49.428+0000 7f2ff79b6740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558743534fe0 0x55874353d870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:36:49.436 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:49.428+0000 7f2fe7fff640 1 --2- 172.21.15.46:0/3442418639 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f2fdc0a26b0 0x7f2fdc0a4b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:36:49.436 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:49.428+0000 7f2fe77fe640 1 -- 172.21.15.46:0/3442418639 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(137..137 src has 1..137) ==== 9554+0+0 (secure 0 0 0) 0x7f2fe80f1270 con 0x7f2ff01055f0 2024-03-31T23:36:49.437 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:49.428+0000 7f2ff79b6740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55874353ddb0 0x5587435401a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:36:49.437 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:49.428+0000 7f2ff79b6740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x5587434978f0 con 0x5587435346c0 2024-03-31T23:36:49.437 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:49.428+0000 7f2ff79b6740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x5587434408d0 con 0x55874353ddb0 2024-03-31T23:36:49.437 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:49.428+0000 7f2ff79b6740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x558743493990 con 0x558743534fe0 2024-03-31T23:36:49.437 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:49.428+0000 7f2fee59c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5587435346c0 0x558743534aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:36:49.437 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:49.428+0000 7f2fee59c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5587435346c0 0x558743534aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39292/0 (socket says 172.21.15.46:39292) 2024-03-31T23:36:49.437 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:49.428+0000 7f2fee59c640 1 -- 172.21.15.46:0/780011775 learned_addr learned my addr 172.21.15.46:0/780011775 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:36:49.437 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:49.429+0000 7f2fe7fff640 1 --2- 172.21.15.46:0/780011775 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558743534fe0 0x55874353d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:36:49.437 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:49.429+0000 7f2feed9d640 1 --2- 172.21.15.46:0/780011775 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55874353ddb0 0x5587435401a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:36:49.437 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:49.429+0000 7f2fee59c640 1 -- 172.21.15.46:0/780011775 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558743534fe0 msgr2=0x55874353d870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:36:49.437 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:49.429+0000 7f2fee59c640 1 --2- 172.21.15.46:0/780011775 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558743534fe0 0x55874353d870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:36:49.437 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:49.429+0000 7f2fee59c640 1 -- 172.21.15.46:0/780011775 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55874353ddb0 msgr2=0x5587435401a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:36:49.437 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:49.429+0000 7f2fee59c640 1 --2- 172.21.15.46:0/780011775 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55874353ddb0 0x5587435401a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:36:49.438 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:49.429+0000 7f2fee59c640 1 -- 172.21.15.46:0/780011775 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5587431ac210 con 0x5587435346c0 2024-03-31T23:36:49.438 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:49.429+0000 7f2feed9d640 1 --2- 172.21.15.46:0/780011775 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55874353ddb0 0x5587435401a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T23:36:49.438 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:49.429+0000 7f2fe7fff640 1 --2- 172.21.15.46:0/780011775 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558743534fe0 0x55874353d870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T23:36:49.438 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:49.429+0000 7f2fe7fff640 1 --2- 172.21.15.46:0/3442418639 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f2fdc0a26b0 0x7f2fdc0a4b70 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f2fd8009fd0 tx=0x7f2fd8011c90 comp rx=0 tx=0).ready entity=mgr.15654 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:36:49.438 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:49.429+0000 7f2fee59c640 1 --2- 172.21.15.46:0/780011775 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5587435346c0 0x558743534aa0 secure :-1 s=READY pgs=1054 cs=0 l=1 rev1=1 crypto rx=0x7f2fe0009cf0 tx=0x7f2fe0011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65c9192ed405c397 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:36:49.438 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:49.429+0000 7f2fe4ff9640 1 -- 172.21.15.46:0/780011775 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fe0004220 con 0x5587435346c0 2024-03-31T23:36:49.438 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:49.429+0000 7f2fe4ff9640 1 -- 172.21.15.46:0/780011775 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f2fe00043c0 con 0x5587435346c0 2024-03-31T23:36:49.438 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:49.429+0000 7f2fe4ff9640 1 -- 172.21.15.46:0/780011775 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fe0018830 con 0x5587435346c0 2024-03-31T23:36:49.438 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:49.430+0000 7f2ff79b6740 1 -- 172.21.15.46:0/780011775 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5587435346c0 msgr2=0x558743534aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:36:49.438 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:49.430+0000 7f2ff79b6740 1 --2- 172.21.15.46:0/780011775 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5587435346c0 0x558743534aa0 secure :-1 s=READY pgs=1054 cs=0 l=1 rev1=1 crypto rx=0x7f2fe0009cf0 tx=0x7f2fe0011810 comp rx=0 tx=0).stop 2024-03-31T23:36:49.438 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:49.430+0000 7f2ff79b6740 1 -- 172.21.15.46:0/780011775 shutdown_connections 2024-03-31T23:36:49.438 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:49.430+0000 7f2ff79b6740 1 --2- 172.21.15.46:0/780011775 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55874353ddb0 0x5587435401a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:36:49.438 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:49.430+0000 7f2ff79b6740 1 --2- 172.21.15.46:0/780011775 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558743534fe0 0x55874353d870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:36:49.438 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:49.430+0000 7f2ff79b6740 1 --2- 172.21.15.46:0/780011775 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5587435346c0 0x558743534aa0 unknown :-1 s=CLOSED pgs=1054 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:36:49.439 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:49.430+0000 7f2ff79b6740 1 -- 172.21.15.46:0/780011775 >> 172.21.15.46:0/780011775 conn(0x55874352efe0 msgr2=0x55874352f3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:36:49.439 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:49.430+0000 7f2ff79b6740 1 -- 172.21.15.46:0/780011775 shutdown_connections 2024-03-31T23:36:49.439 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:49.430+0000 7f2ff79b6740 1 -- 172.21.15.46:0/780011775 wait complete. 2024-03-31T23:36:49.439 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:49.430+0000 7f2ff79b6740 1 -- start start 2024-03-31T23:36:49.439 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:49.430+0000 7f2ff79b6740 1 Processor -- start 2024-03-31T23:36:49.439 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:49.431+0000 7f2ff79b6740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5587435346c0 0x55874353d7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:36:49.439 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:49.431+0000 7f2ff79b6740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558743534fe0 0x55874353add0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:36:49.439 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:49.431+0000 7f2ff79b6740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55874353ddb0 0x55874353b310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:36:49.439 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:49.431+0000 7f2ff79b6740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x5587435409e0 con 0x558743534fe0 2024-03-31T23:36:49.439 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:49.431+0000 7f2ff79b6740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x558743435f80 con 0x55874353ddb0 2024-03-31T23:36:49.439 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:49.431+0000 7f2ff79b6740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x558743540b60 con 0x5587435346c0 2024-03-31T23:36:49.439 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:49.431+0000 7f2ff79b6740 10 client.0 _ll_register_callbacks cb 0x55874348e290 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T23:36:49.439 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:49.431+0000 7f2fe7fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558743534fe0 0x55874353add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:36:49.439 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:49.431+0000 7f2fe7fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558743534fe0 0x55874353add0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39296/0 (socket says 172.21.15.46:39296) 2024-03-31T23:36:49.439 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:49.431+0000 7f2fee59c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5587435346c0 0x55874353d7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:36:49.439 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:49.431+0000 7f2fe7fff640 1 -- 172.21.15.46:0/2118106628 learned_addr learned my addr 172.21.15.46:0/2118106628 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:36:49.440 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:49.431+0000 7f2feed9d640 1 --2- 172.21.15.46:0/2118106628 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55874353ddb0 0x55874353b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:36:49.440 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:49.431+0000 7f2fe7fff640 1 -- 172.21.15.46:0/2118106628 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5587435346c0 msgr2=0x55874353d7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:36:49.440 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:49.431+0000 7f2fe7fff640 1 --2- 172.21.15.46:0/2118106628 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5587435346c0 0x55874353d7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:36:49.440 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:49.431+0000 7f2fe7fff640 1 -- 172.21.15.46:0/2118106628 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55874353ddb0 msgr2=0x55874353b310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:36:49.440 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:49.431+0000 7f2fe7fff640 1 --2- 172.21.15.46:0/2118106628 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55874353ddb0 0x55874353b310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:36:49.440 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:49.431+0000 7f2fe7fff640 1 -- 172.21.15.46:0/2118106628 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558743278620 con 0x558743534fe0 2024-03-31T23:36:49.440 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:49.432+0000 7f2fe7fff640 1 --2- 172.21.15.46:0/2118106628 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558743534fe0 0x55874353add0 secure :-1 s=READY pgs=1055 cs=0 l=1 rev1=1 crypto rx=0x7f2fd800fb30 tx=0x7f2fd8004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f3c278e07a159e91 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:36:49.440 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:49.432+0000 7f2fd5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-31T23:36:49.440 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:49.432+0000 7f2fd5ffb640 1 -- 172.21.15.46:0/2118106628 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fd801a040 con 0x558743534fe0 2024-03-31T23:36:49.440 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:49.432+0000 7f2fd5ffb640 1 -- 172.21.15.46:0/2118106628 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(137..137 src has 1..137) ==== 9554+0+0 (secure 0 0 0) 0x7f2fd8004030 con 0x558743534fe0 2024-03-31T23:36:49.440 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:49.432+0000 7f2ff79b6740 1 -- 172.21.15.46:0/2118106628 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2fe00043c0 con 0x558743534fe0 2024-03-31T23:36:49.440 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:49.432+0000 7f2fd5ffb640 1 -- 172.21.15.46:0/2118106628 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=54) ==== 24+0+0 (secure 0 0 0) 0x7f2fd8032050 con 0x558743534fe0 2024-03-31T23:36:49.440 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:49.432+0000 7f2ff79b6740 10 client.18171 fetch_fsmap learned FSMap version 54 2024-03-31T23:36:49.441 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:49.432+0000 7f2ff79b6740 1 -- 172.21.15.46:0/2118106628 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=54,monmap=4+}) -- 0x558743555050 con 0x558743534fe0 2024-03-31T23:36:49.441 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:49.432+0000 7f2fd5ffb640 1 -- 172.21.15.46:0/2118106628 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f2fe00043c0 con 0x558743534fe0 2024-03-31T23:36:49.441 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:49.432+0000 7f2fd5ffb640 1 -- 172.21.15.46:0/2118106628 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 54) ==== 8346+0+0 (secure 0 0 0) 0x7f2fd8020420 con 0x558743534fe0 2024-03-31T23:36:49.441 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:49.433+0000 7f2ff79b6740 10 client.18171 fetch_fsmap finished waiting for FSMap version 54 2024-03-31T23:36:49.441 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:49.433+0000 7f2ff79b6740 10 client.18171 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-31T23:36:49.441 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:49.433+0000 7f2ff79b6740 20 client.18171 populate_metadata read hostname 'smithi046' 2024-03-31T23:36:49.441 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:49.433+0000 7f2ff79b6740 1 --2- 172.21.15.46:0/2118106628 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x558743556560 0x558743558950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:36:49.441 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:49.433+0000 7f2ff79b6740 4 client.18171 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-31T23:36:49.441 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:49.433+0000 7f2ff79b6740 1 -- 172.21.15.46:0/2118106628 --> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5587434ab120 con 0x558743556560 2024-03-31T23:36:49.441 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:49.433+0000 7f2fee59c640 1 --2- 172.21.15.46:0/2118106628 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x558743556560 0x558743558950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:36:49.441 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:49.433+0000 7f2fee59c640 1 --2- 172.21.15.46:0/2118106628 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x558743556560 0x558743558950 crc :-1 s=READY pgs=300 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5357f6421a72512a server_cookie=13cc7541d91b010a in_seq=0 out_seq=0 2024-03-31T23:36:49.441 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:49.434+0000 7f2fd5ffb640 10 client.18171 ms_handle_connect on v2:172.21.15.142:6838/2519143918 2024-03-31T23:36:49.442 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:49.435+0000 7f2fd5ffb640 1 -- 172.21.15.46:0/2118106628 <== mds.0 v2:172.21.15.142:6838/2519143918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5587434ab120 con 0x558743556560 2024-03-31T23:36:49.443 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:49.435+0000 7f2fd5ffb640 10 client.18171 handle_command_reply: tid=0 2024-03-31T23:36:49.443 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:49.435+0000 7f2ff79b6740 1 client.18171 shutdown 2024-03-31T23:36:49.443 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:49.436+0000 7f2ff79b6740 1 -- 172.21.15.46:0/2118106628 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558743534fe0 msgr2=0x55874353add0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:36:49.443 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:49.436+0000 7f2ff79b6740 1 --2- 172.21.15.46:0/2118106628 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558743534fe0 0x55874353add0 secure :-1 s=READY pgs=1055 cs=0 l=1 rev1=1 crypto rx=0x7f2fd800fb30 tx=0x7f2fd8004730 comp rx=0 tx=0).stop 2024-03-31T23:36:49.443 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:49.436+0000 7f2ff79b6740 1 -- 172.21.15.46:0/2118106628 shutdown_connections 2024-03-31T23:36:49.443 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:49.436+0000 7f2ff79b6740 1 --2- 172.21.15.46:0/2118106628 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x558743556560 0x558743558950 crc :-1 s=READY pgs=300 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:36:49.443 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:49.436+0000 7f2ff79b6740 1 --2- 172.21.15.46:0/2118106628 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55874353ddb0 0x55874353b310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:36:49.443 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:49.436+0000 7f2ff79b6740 1 --2- 172.21.15.46:0/2118106628 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558743534fe0 0x55874353add0 unknown :-1 s=CLOSED pgs=1055 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:36:49.443 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:49.436+0000 7f2ff79b6740 1 --2- 172.21.15.46:0/2118106628 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5587435346c0 0x55874353d7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:36:49.443 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:49.436+0000 7f2fd5ffb640 0 client.18171 ms_handle_reset on v2:172.21.15.142:6838/2519143918 2024-03-31T23:36:49.443 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:49.436+0000 7f2ff79b6740 1 -- 172.21.15.46:0/2118106628 >> 172.21.15.46:0/2118106628 conn(0x55874352efe0 msgr2=0x558743530d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:36:49.443 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:49.436+0000 7f2ff79b6740 1 -- 172.21.15.46:0/2118106628 shutdown_connections 2024-03-31T23:36:49.443 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:49.436+0000 7f2ff79b6740 1 -- 172.21.15.46:0/2118106628 wait complete. 2024-03-31T23:36:49.443 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:49.436+0000 7f2ff79b6740 20 client.18171 trim_cache size 0 max 16384 2024-03-31T23:36:49.465 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:49.458+0000 7f2ff79b6740 1 Processor -- start 2024-03-31T23:36:49.465 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:49.458+0000 7f2ff79b6740 1 -- start start 2024-03-31T23:36:49.465 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:49.458+0000 7f2ff79b6740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5587435346c0 0x55874353d500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:36:49.466 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:49.458+0000 7f2ff79b6740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558743534fe0 0x55874352de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:36:49.466 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:49.459+0000 7f2fe7fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558743534fe0 0x55874352de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:36:49.466 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:49.459+0000 7f2fee59c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5587435346c0 0x55874353d500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:36:49.466 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:49.459+0000 7f2fe7fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558743534fe0 0x55874352de00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39308/0 (socket says 172.21.15.46:39308) 2024-03-31T23:36:49.466 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:49.459+0000 7f2fe7fff640 1 -- 172.21.15.46:0/1105741223 learned_addr learned my addr 172.21.15.46:0/1105741223 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:36:49.466 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:49.459+0000 7f2ff79b6740 1 --2- 172.21.15.46:0/1105741223 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55874353ddb0 0x55874355e260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:36:49.466 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:49.459+0000 7f2ff79b6740 1 -- 172.21.15.46:0/1105741223 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f2ff01af2c0 con 0x558743534fe0 2024-03-31T23:36:49.466 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:49.459+0000 7f2ff79b6740 1 -- 172.21.15.46:0/1105741223 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f2ff01acb90 con 0x55874353ddb0 2024-03-31T23:36:49.466 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:49.459+0000 7f2ff79b6740 1 -- 172.21.15.46:0/1105741223 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f2ff01aaa00 con 0x5587435346c0 2024-03-31T23:36:49.466 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:49.459+0000 7f2feed9d640 1 --2- 172.21.15.46:0/1105741223 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55874353ddb0 0x55874355e260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:36:49.466 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:49.459+0000 7f2fee59c640 1 -- 172.21.15.46:0/1105741223 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55874353ddb0 msgr2=0x55874355e260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:36:49.466 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:49.459+0000 7f2fee59c640 1 --2- 172.21.15.46:0/1105741223 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55874353ddb0 0x55874355e260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:36:49.467 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:49.459+0000 7f2fee59c640 1 -- 172.21.15.46:0/1105741223 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558743534fe0 msgr2=0x55874352de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:36:49.467 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:49.459+0000 7f2fee59c640 1 --2- 172.21.15.46:0/1105741223 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558743534fe0 0x55874352de00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:36:49.467 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:49.459+0000 7f2fee59c640 1 -- 172.21.15.46:0/1105741223 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5587435404d0 con 0x5587435346c0 2024-03-31T23:36:49.467 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:49.459+0000 7f2feed9d640 1 --2- 172.21.15.46:0/1105741223 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55874353ddb0 0x55874355e260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T23:36:49.467 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:49.459+0000 7f2fe7fff640 1 --2- 172.21.15.46:0/1105741223 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558743534fe0 0x55874352de00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:36:49.467 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:49.459+0000 7f2fee59c640 1 --2- 172.21.15.46:0/1105741223 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5587435346c0 0x55874353d500 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f2fe00045b0 tx=0x7f2fe0018470 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4c46018fdf0a7876 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:36:49.467 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:49.459+0000 7f2fd7fff640 1 -- 172.21.15.46:0/1105741223 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fe00097a0 con 0x5587435346c0 2024-03-31T23:36:49.467 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:49.460+0000 7f2fd7fff640 1 -- 172.21.15.46:0/1105741223 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f2fe00043d0 con 0x5587435346c0 2024-03-31T23:36:49.467 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:49.460+0000 7f2fd7fff640 1 -- 172.21.15.46:0/1105741223 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fe00035f0 con 0x5587435346c0 2024-03-31T23:36:49.467 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:49.460+0000 7f2ff79b6740 1 -- 172.21.15.46:0/1105741223 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5587435346c0 msgr2=0x55874353d500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:36:49.467 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:49.460+0000 7f2ff79b6740 1 --2- 172.21.15.46:0/1105741223 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5587435346c0 0x55874353d500 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f2fe00045b0 tx=0x7f2fe0018470 comp rx=0 tx=0).stop 2024-03-31T23:36:49.467 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:49.460+0000 7f2ff79b6740 1 -- 172.21.15.46:0/1105741223 shutdown_connections 2024-03-31T23:36:49.467 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:49.460+0000 7f2ff79b6740 1 --2- 172.21.15.46:0/1105741223 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55874353ddb0 0x55874355e260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:36:49.467 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:49.460+0000 7f2ff79b6740 1 --2- 172.21.15.46:0/1105741223 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558743534fe0 0x55874352de00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:36:49.468 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:49.460+0000 7f2ff79b6740 1 --2- 172.21.15.46:0/1105741223 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5587435346c0 0x55874353d500 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:36:49.468 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:49.461+0000 7f2ff79b6740 1 -- 172.21.15.46:0/1105741223 >> 172.21.15.46:0/1105741223 conn(0x55874352efe0 msgr2=0x55874352d6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:36:49.468 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:49.461+0000 7f2ff79b6740 1 -- 172.21.15.46:0/1105741223 shutdown_connections 2024-03-31T23:36:49.468 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:49.461+0000 7f2ff79b6740 1 -- 172.21.15.46:0/1105741223 wait complete. 2024-03-31T23:36:49.468 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:49.461+0000 7f2ff79b6740 1 -- start start 2024-03-31T23:36:49.468 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:49.461+0000 7f2ff79b6740 1 Processor -- start 2024-03-31T23:36:49.468 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:49.461+0000 7f2ff79b6740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5587435346c0 0x558743553430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:36:49.468 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:49.461+0000 7f2ff79b6740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558743534fe0 0x558743537db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:36:49.468 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:49.461+0000 7f2ff79b6740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55874353ddb0 0x5587435384f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:36:49.468 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:49.462+0000 7f2ff79b6740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f2ff01ab050 con 0x558743534fe0 2024-03-31T23:36:49.468 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:49.462+0000 7f2ff79b6740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f2ff01ac4c0 con 0x55874353ddb0 2024-03-31T23:36:49.468 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:49.462+0000 7f2ff79b6740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f2ff01ab720 con 0x5587435346c0 2024-03-31T23:36:49.468 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:49.462+0000 7f2ff79b6740 10 client.0 _ll_register_callbacks cb 0x7f2fd80115b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T23:36:49.469 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:49.462+0000 7f2fee59c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5587435346c0 0x558743553430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:36:49.469 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:49.462+0000 7f2fe7fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558743534fe0 0x558743537db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:36:49.469 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:49.462+0000 7f2fee59c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5587435346c0 0x558743553430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.46:34432/0 (socket says 172.21.15.46:34432) 2024-03-31T23:36:49.469 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:49.462+0000 7f2fee59c640 1 -- 172.21.15.46:0/3774388371 learned_addr learned my addr 172.21.15.46:0/3774388371 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:36:49.469 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:49.462+0000 7f2feed9d640 1 --2- 172.21.15.46:0/3774388371 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55874353ddb0 0x5587435384f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:36:49.469 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:49.462+0000 7f2fee59c640 1 -- 172.21.15.46:0/3774388371 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55874353ddb0 msgr2=0x5587435384f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:36:49.469 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:49.462+0000 7f2fee59c640 1 --2- 172.21.15.46:0/3774388371 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55874353ddb0 0x5587435384f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:36:49.469 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:49.462+0000 7f2fee59c640 1 -- 172.21.15.46:0/3774388371 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558743534fe0 msgr2=0x558743537db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T23:36:49.469 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:49.462+0000 7f2fee59c640 1 --2- 172.21.15.46:0/3774388371 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558743534fe0 0x558743537db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:36:49.469 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:49.462+0000 7f2fee59c640 1 -- 172.21.15.46:0/3774388371 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5587435bb890 con 0x5587435346c0 2024-03-31T23:36:49.469 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:49.462+0000 7f2fee59c640 1 --2- 172.21.15.46:0/3774388371 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5587435346c0 0x558743553430 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f2fe0011e70 tx=0x7f2fe0011ea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=626f40e3625fe02 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:36:49.469 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:49.462+0000 7f2fd5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.162:3300/0 2024-03-31T23:36:49.469 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:49.462+0000 7f2fd5ffb640 1 -- 172.21.15.46:0/3774388371 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fe001e040 con 0x5587435346c0 2024-03-31T23:36:49.469 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:49.463+0000 7f2fd5ffb640 1 -- 172.21.15.46:0/3774388371 <== mon.1 v2:172.21.15.162:3300/0 2 ==== osd_map(137..137 src has 1..137) ==== 9554+0+0 (secure 0 0 0) 0x7f2fe0003f20 con 0x5587435346c0 2024-03-31T23:36:49.470 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:49.463+0000 7f2ff79b6740 1 -- 172.21.15.46:0/3774388371 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5587434a4e30 con 0x5587435346c0 2024-03-31T23:36:49.470 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:49.463+0000 7f2fd5ffb640 1 -- 172.21.15.46:0/3774388371 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_get_version_reply(handle=1 version=54) ==== 24+0+0 (secure 0 0 0) 0x7f2fe003e050 con 0x5587435346c0 2024-03-31T23:36:49.470 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:49.463+0000 7f2ff79b6740 10 client.27886 fetch_fsmap learned FSMap version 54 2024-03-31T23:36:49.470 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:49.463+0000 7f2ff79b6740 1 -- 172.21.15.46:0/3774388371 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,fsmap=54,monmap=4+}) -- 0x5587435bc8e0 con 0x5587435346c0 2024-03-31T23:36:49.470 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:49.463+0000 7f2fd5ffb640 1 -- 172.21.15.46:0/3774388371 <== mon.1 v2:172.21.15.162:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x5587434a4e30 con 0x5587435346c0 2024-03-31T23:36:49.470 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:49.463+0000 7f2fd5ffb640 1 -- 172.21.15.46:0/3774388371 <== mon.1 v2:172.21.15.162:3300/0 5 ==== fsmap(e 54) ==== 8346+0+0 (secure 0 0 0) 0x7f2fe00187e0 con 0x5587435346c0 2024-03-31T23:36:49.470 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:49.463+0000 7f2ff79b6740 10 client.27886 fetch_fsmap finished waiting for FSMap version 54 2024-03-31T23:36:49.470 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:49.463+0000 7f2ff79b6740 10 client.27886 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-31T23:36:49.470 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:49.463+0000 7f2ff79b6740 20 client.27886 populate_metadata read hostname 'smithi046' 2024-03-31T23:36:49.470 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:49.463+0000 7f2ff79b6740 1 --2- 172.21.15.46:0/3774388371 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x5587435bcd00 0x5587435bf0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:36:49.470 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:49.463+0000 7f2ff79b6740 4 client.27886 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-31T23:36:49.470 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:49.463+0000 7f2ff79b6740 1 -- 172.21.15.46:0/3774388371 --> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55874349f180 con 0x5587435bcd00 2024-03-31T23:36:49.470 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:49.464+0000 7f2fe7fff640 1 --2- 172.21.15.46:0/3774388371 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x5587435bcd00 0x5587435bf0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:36:49.470 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:49.464+0000 7f2fd5ffb640 10 client.27886 ms_handle_connect on v2:172.21.15.142:6838/2519143918 2024-03-31T23:36:49.471 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:49.464+0000 7f2fe7fff640 1 --2- 172.21.15.46:0/3774388371 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x5587435bcd00 0x5587435bf0f0 crc :-1 s=READY pgs=301 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2bffa49f0923a364 server_cookie=b8975456cbe9b0af in_seq=0 out_seq=0 2024-03-31T23:36:49.769 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:49.768+0000 7f2fd5ffb640 1 -- 172.21.15.46:0/3774388371 <== mds.0 v2:172.21.15.142:6838/2519143918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55874349f180 con 0x5587435bcd00 2024-03-31T23:36:49.770 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:49.768+0000 7f2fd5ffb640 10 client.27886 handle_command_reply: tid=0 2024-03-31T23:36:49.770 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:49.768+0000 7f2ff79b6740 1 client.27886 shutdown 2024-03-31T23:36:49.770 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:49.768+0000 7f2ff79b6740 1 -- 172.21.15.46:0/3774388371 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5587435346c0 msgr2=0x558743553430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:36:49.770 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:49.768+0000 7f2ff79b6740 1 --2- 172.21.15.46:0/3774388371 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5587435346c0 0x558743553430 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f2fe0011e70 tx=0x7f2fe0011ea0 comp rx=0 tx=0).stop 2024-03-31T23:36:49.770 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:49.768+0000 7f2ff79b6740 1 -- 172.21.15.46:0/3774388371 shutdown_connections 2024-03-31T23:36:49.770 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:49.768+0000 7f2ff79b6740 1 --2- 172.21.15.46:0/3774388371 >> [v2:172.21.15.142:6838/2519143918,v1:172.21.15.142:6839/2519143918] conn(0x5587435bcd00 0x5587435bf0f0 crc :-1 s=READY pgs=301 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:36:49.770 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:49.768+0000 7f2ff79b6740 1 --2- 172.21.15.46:0/3774388371 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55874353ddb0 0x5587435384f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:36:49.770 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:49.768+0000 7f2fd5ffb640 0 client.27886 ms_handle_reset on v2:172.21.15.142:6838/2519143918 2024-03-31T23:36:49.770 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:49.768+0000 7f2ff79b6740 1 --2- 172.21.15.46:0/3774388371 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558743534fe0 0x558743537db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:36:49.770 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:49.768+0000 7f2ff79b6740 1 --2- 172.21.15.46:0/3774388371 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5587435346c0 0x558743553430 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:36:49.770 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:49.768+0000 7f2ff79b6740 1 -- 172.21.15.46:0/3774388371 >> 172.21.15.46:0/3774388371 conn(0x55874352efe0 msgr2=0x558743539280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:36:49.771 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:49.768+0000 7f2ff79b6740 1 -- 172.21.15.46:0/3774388371 shutdown_connections 2024-03-31T23:36:49.771 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:49.769+0000 7f2ff79b6740 1 -- 172.21.15.46:0/3774388371 wait complete. 2024-03-31T23:36:49.771 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:49.769+0000 7f2ff79b6740 20 client.27886 trim_cache size 0 max 16384 2024-03-31T23:36:49.771 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout:{ 2024-03-31T23:36:49.771 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout: "message": "", 2024-03-31T23:36:49.771 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout: "return_code": 0 2024-03-31T23:36:49.771 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stdout:} 2024-03-31T23:36:49.772 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:49.771+0000 7f2ff4ddd640 1 -- 172.21.15.46:0/3442418639 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f2fdc0a26b0 msgr2=0x7f2fdc0a4b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:36:49.772 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:49.771+0000 7f2ff4ddd640 1 --2- 172.21.15.46:0/3442418639 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f2fdc0a26b0 0x7f2fdc0a4b70 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f2fd8009fd0 tx=0x7f2fd8011c90 comp rx=0 tx=0).stop 2024-03-31T23:36:49.772 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:49.771+0000 7f2ff4ddd640 1 -- 172.21.15.46:0/3442418639 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2ff01055f0 msgr2=0x7f2ff019c760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:36:49.772 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:49.771+0000 7f2ff4ddd640 1 --2- 172.21.15.46:0/3442418639 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2ff01055f0 0x7f2ff019c760 secure :-1 s=READY pgs=1053 cs=0 l=1 rev1=1 crypto rx=0x7f2fe802c700 tx=0x7f2fe8035d60 comp rx=0 tx=0).stop 2024-03-31T23:36:49.772 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:49.771+0000 7f2ff4ddd640 1 -- 172.21.15.46:0/3442418639 shutdown_connections 2024-03-31T23:36:49.772 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:49.771+0000 7f2ff4ddd640 1 --2- 172.21.15.46:0/3442418639 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f2fdc0a26b0 0x7f2fdc0a4b70 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:36:49.772 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:49.771+0000 7f2ff4ddd640 1 --2- 172.21.15.46:0/3442418639 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2ff01055f0 0x7f2ff019c760 unknown :-1 s=CLOSED pgs=1053 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:36:49.772 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:49.771+0000 7f2ff4ddd640 1 --2- 172.21.15.46:0/3442418639 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2ff0101260 0x7f2ff01983e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:36:49.772 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:49.771+0000 7f2ff4ddd640 1 --2- 172.21.15.46:0/3442418639 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2ff01008a0 0x7f2ff0197ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:36:49.772 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:49.771+0000 7f2ff4ddd640 1 -- 172.21.15.46:0/3442418639 >> 172.21.15.46:0/3442418639 conn(0x7f2ff00774a0 msgr2=0x7f2ff0069660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:36:49.773 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:49.772+0000 7f2ff4ddd640 1 -- 172.21.15.46:0/3442418639 shutdown_connections 2024-03-31T23:36:49.773 INFO:teuthology.task.background_exec.ubuntu@smithi046.front.sepia.ceph.com.smithi046.stderr:2024-03-31T23:36:49.772+0000 7f2ff4ddd640 1 -- 172.21.15.46:0/3442418639 wait complete. 2024-03-31T23:36:50.319 INFO:tasks.workunit.client.0.smithi046.stdout: 10 109010 8.10 MB/sec execute 595 sec latency 335.500 ms 2024-03-31T23:36:51.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:36:50 smithi046 ceph-mon[32498]: pgmap v797: 129 pgs: 129 active+clean; 1.3 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 7.9 MiB/s wr, 348 op/s 2024-03-31T23:36:51.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:36:50 smithi142 ceph-mon[35271]: pgmap v797: 129 pgs: 129 active+clean; 1.3 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 7.9 MiB/s wr, 348 op/s 2024-03-31T23:36:51.295 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:36:50 smithi162 ceph-mon[36237]: pgmap v797: 129 pgs: 129 active+clean; 1.3 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 7.9 MiB/s wr, 348 op/s 2024-03-31T23:36:51.319 INFO:tasks.workunit.client.0.smithi046.stdout: 10 109107 8.10 MB/sec execute 596 sec latency 175.627 ms 2024-03-31T23:36:52.319 INFO:tasks.workunit.client.0.smithi046.stdout: 10 109244 8.10 MB/sec execute 597 sec latency 97.741 ms 2024-03-31T23:36:53.097 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:36:52 smithi162 ceph-mon[36237]: pgmap v798: 129 pgs: 129 active+clean; 1.3 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 5.2 MiB/s wr, 195 op/s 2024-03-31T23:36:53.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:36:52 smithi046 ceph-mon[32498]: pgmap v798: 129 pgs: 129 active+clean; 1.3 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 5.2 MiB/s wr, 195 op/s 2024-03-31T23:36:53.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:36:52 smithi142 ceph-mon[35271]: pgmap v798: 129 pgs: 129 active+clean; 1.3 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 5.2 MiB/s wr, 195 op/s 2024-03-31T23:36:53.319 INFO:tasks.workunit.client.0.smithi046.stdout: 10 109386 8.09 MB/sec execute 598 sec latency 86.820 ms 2024-03-31T23:36:54.319 INFO:tasks.workunit.client.0.smithi046.stdout: 10 109532 8.09 MB/sec execute 599 sec latency 108.718 ms 2024-03-31T23:36:55.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:36:54 smithi046 ceph-mon[32498]: pgmap v799: 129 pgs: 129 active+clean; 1.3 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 5.2 MiB/s wr, 195 op/s 2024-03-31T23:36:55.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:36:54 smithi142 ceph-mon[35271]: pgmap v799: 129 pgs: 129 active+clean; 1.3 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 5.2 MiB/s wr, 195 op/s 2024-03-31T23:36:55.295 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:36:54 smithi162 ceph-mon[36237]: pgmap v799: 129 pgs: 129 active+clean; 1.3 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 5.2 MiB/s wr, 195 op/s 2024-03-31T23:36:55.319 INFO:tasks.workunit.client.0.smithi046.stdout: 10 cleanup 600 sec 2024-03-31T23:36:56.319 INFO:tasks.workunit.client.0.smithi046.stdout: 10 cleanup 601 sec 2024-03-31T23:36:57.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:36:56 smithi046 ceph-mon[32498]: pgmap v800: 129 pgs: 129 active+clean; 1.3 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 7.3 MiB/s wr, 376 op/s 2024-03-31T23:36:57.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:36:56 smithi142 ceph-mon[35271]: pgmap v800: 129 pgs: 129 active+clean; 1.3 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 7.3 MiB/s wr, 376 op/s 2024-03-31T23:36:57.296 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:36:56 smithi162 ceph-mon[36237]: pgmap v800: 129 pgs: 129 active+clean; 1.3 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 7.3 MiB/s wr, 376 op/s 2024-03-31T23:36:57.319 INFO:tasks.workunit.client.0.smithi046.stdout: 5 cleanup 602 sec 2024-03-31T23:36:58.319 INFO:tasks.workunit.client.0.smithi046.stdout: 2 cleanup 603 sec 2024-03-31T23:36:59.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:36:58 smithi046 ceph-mon[32498]: pgmap v801: 129 pgs: 129 active+clean; 1.3 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 5.2 MiB/s wr, 290 op/s 2024-03-31T23:36:59.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:36:58 smithi046 ceph-mon[32498]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-03-31T23:36:59.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:36:58 smithi142 ceph-mon[35271]: pgmap v801: 129 pgs: 129 active+clean; 1.3 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 5.2 MiB/s wr, 290 op/s 2024-03-31T23:36:59.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:36:58 smithi142 ceph-mon[35271]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-03-31T23:36:59.298 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:36:58 smithi162 ceph-mon[36237]: pgmap v801: 129 pgs: 129 active+clean; 1.3 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 5.2 MiB/s wr, 290 op/s 2024-03-31T23:36:59.298 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:36:58 smithi162 ceph-mon[36237]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-03-31T23:36:59.319 INFO:tasks.workunit.client.0.smithi046.stdout: 1 cleanup 604 sec 2024-03-31T23:36:59.335 INFO:tasks.workunit.client.0.smithi046.stdout: 0 cleanup 604 sec 2024-03-31T23:36:59.335 INFO:tasks.workunit.client.0.smithi046.stdout: 2024-03-31T23:36:59.335 INFO:tasks.workunit.client.0.smithi046.stdout: Operation Count AvgLat MaxLat 2024-03-31T23:36:59.335 INFO:tasks.workunit.client.0.smithi046.stdout: ---------------------------------------- 2024-03-31T23:36:59.335 INFO:tasks.workunit.client.0.smithi046.stdout: NTCreateX 155838 11.156 2432.390 2024-03-31T23:36:59.335 INFO:tasks.workunit.client.0.smithi046.stdout: Close 114386 0.167 58.566 2024-03-31T23:36:59.335 INFO:tasks.workunit.client.0.smithi046.stdout: Rename 6595 29.741 14494.263 2024-03-31T23:36:59.336 INFO:tasks.workunit.client.0.smithi046.stdout: Unlink 31558 10.535 2416.430 2024-03-31T23:36:59.336 INFO:tasks.workunit.client.0.smithi046.stdout: Qpathinfo 141147 8.084 2363.289 2024-03-31T23:36:59.336 INFO:tasks.workunit.client.0.smithi046.stdout: Qfileinfo 24557 0.301 35.261 2024-03-31T23:36:59.336 INFO:tasks.workunit.client.0.smithi046.stdout: Qfsinfo 25948 0.911 193.304 2024-03-31T23:36:59.336 INFO:tasks.workunit.client.0.smithi046.stdout: Sfileinfo 12680 2.901 164.078 2024-03-31T23:36:59.336 INFO:tasks.workunit.client.0.smithi046.stdout: Find 54591 43.003 2454.861 2024-03-31T23:36:59.336 INFO:tasks.workunit.client.0.smithi046.stdout: WriteX 76885 0.579 213.800 2024-03-31T23:36:59.336 INFO:tasks.workunit.client.0.smithi046.stdout: ReadX 244017 0.006 6.246 2024-03-31T23:36:59.336 INFO:tasks.workunit.client.0.smithi046.stdout: LockX 504 2.430 44.257 2024-03-31T23:36:59.336 INFO:tasks.workunit.client.0.smithi046.stdout: UnlockX 504 2.298 13.350 2024-03-31T23:36:59.336 INFO:tasks.workunit.client.0.smithi046.stdout: Flush 10923 9.732 271.652 2024-03-31T23:36:59.336 INFO:tasks.workunit.client.0.smithi046.stdout: 2024-03-31T23:36:59.336 INFO:tasks.workunit.client.0.smithi046.stdout:Throughput 8.09366 MB/sec 10 clients 10 procs max_latency=14494.269 ms 2024-03-31T23:36:59.337 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-03-31T23:36:59.337 DEBUG:teuthology.orchestra.run.smithi046:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-03-31T23:37:01.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:37:00 smithi046 ceph-mon[32498]: pgmap v802: 129 pgs: 129 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 7.4 MiB/s wr, 462 op/s 2024-03-31T23:37:01.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:37:00 smithi142 ceph-mon[35271]: pgmap v802: 129 pgs: 129 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 7.4 MiB/s wr, 462 op/s 2024-03-31T23:37:01.209 INFO:tasks.workunit:Stopping ['suites/dbench.sh'] on client.0... 2024-03-31T23:37:01.210 DEBUG:teuthology.orchestra.run.smithi046:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-03-31T23:37:01.295 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:37:00 smithi162 ceph-mon[36237]: pgmap v802: 129 pgs: 129 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 7.4 MiB/s wr, 462 op/s 2024-03-31T23:37:01.588 DEBUG:teuthology.parallel:result is None 2024-03-31T23:37:01.589 DEBUG:teuthology.orchestra.run.smithi046:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0 2024-03-31T23:37:01.630 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.0/client.0 2024-03-31T23:37:01.631 DEBUG:teuthology.run_tasks:Unwinding manager background_exec 2024-03-31T23:37:01.642 INFO:teuthology.task.background_exec:Stopping background command on ubuntu@smithi046.front.sepia.ceph.com 2024-03-31T23:37:01.679 DEBUG:teuthology.run_tasks:Unwinding manager full_sequential_finally 2024-03-31T23:37:01.689 INFO:teuthology.task.full_sequential_finally:In full_sequential_finally, running task exec... 2024-03-31T23:37:01.689 INFO:teuthology.task.exec:Executing custom commands... 2024-03-31T23:37:01.689 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi046.front.sepia.ceph.com 2024-03-31T23:37:01.689 DEBUG:teuthology.orchestra.run.smithi046:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'date +%s > END_TIME' 2024-03-31T23:37:01.755 DEBUG:teuthology.orchestra.run.smithi046:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'START_TIME=$(cat START_TIME); END_TIME=$(cat END_TIME); DIFF_TIME=$((600-(END_TIME-START_TIME))); if [ "$DIFF_TIME" -gt 0 ]; then sleep "$DIFF_TIME"; fi' 2024-03-31T23:37:01.830 DEBUG:teuthology.orchestra.run.smithi046:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule status --fs=cephfs --path=/' 2024-03-31T23:37:01.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:01.981+0000 7ff2d76b1640 1 -- 172.21.15.46:0/2594411226 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff2d01055d0 msgr2=0x7ff2d01059b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:37:01.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:01.981+0000 7ff2d76b1640 1 --2- 172.21.15.46:0/2594411226 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff2d01055d0 0x7ff2d01059b0 secure :-1 s=READY pgs=1056 cs=0 l=1 rev1=1 crypto rx=0x7ff2c8009900 tx=0x7ff2c802bc30 comp rx=0 tx=0).stop 2024-03-31T23:37:01.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:01.981+0000 7ff2d76b1640 1 -- 172.21.15.46:0/2594411226 shutdown_connections 2024-03-31T23:37:01.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:01.981+0000 7ff2d76b1640 1 --2- 172.21.15.46:0/2594411226 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff2d0101240 0x7ff2d010d480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:37:01.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:01.981+0000 7ff2d76b1640 1 --2- 172.21.15.46:0/2594411226 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff2d01008a0 0x7ff2d0100d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:37:01.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:01.981+0000 7ff2d76b1640 1 --2- 172.21.15.46:0/2594411226 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff2d01055d0 0x7ff2d01059b0 unknown :-1 s=CLOSED pgs=1056 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:37:01.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:01.981+0000 7ff2d76b1640 1 -- 172.21.15.46:0/2594411226 >> 172.21.15.46:0/2594411226 conn(0x7ff2d0077420 msgr2=0x7ff2d0077830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:37:01.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:01.981+0000 7ff2d76b1640 1 -- 172.21.15.46:0/2594411226 shutdown_connections 2024-03-31T23:37:01.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:01.981+0000 7ff2d76b1640 1 -- 172.21.15.46:0/2594411226 wait complete. 2024-03-31T23:37:01.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:01.981+0000 7ff2d76b1640 1 Processor -- start 2024-03-31T23:37:01.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:01.981+0000 7ff2d76b1640 1 -- start start 2024-03-31T23:37:01.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:01.982+0000 7ff2d76b1640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff2d01008a0 0x7ff2d010df90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:37:01.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:01.982+0000 7ff2d76b1640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff2d0101240 0x7ff2d010e4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:37:01.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:01.982+0000 7ff2d76b1640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff2d01055d0 0x7ff2d010ea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:37:01.986 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:01.982+0000 7ff2d76b1640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7ff2d0111ab0 con 0x7ff2d01055d0 2024-03-31T23:37:01.986 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:01.982+0000 7ff2d76b1640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7ff2d0111930 con 0x7ff2d01008a0 2024-03-31T23:37:01.986 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:01.982+0000 7ff2d76b1640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7ff2d0111c30 con 0x7ff2d0101240 2024-03-31T23:37:01.986 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:01.982+0000 7ff2d5c4e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff2d01055d0 0x7ff2d010ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:37:01.986 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:01.982+0000 7ff2d5c4e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff2d01055d0 0x7ff2d010ea10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:48792/0 (socket says 172.21.15.46:48792) 2024-03-31T23:37:01.986 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:01.982+0000 7ff2d5c4e640 1 -- 172.21.15.46:0/1658046241 learned_addr learned my addr 172.21.15.46:0/1658046241 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:37:01.986 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:01.982+0000 7ff2d4c4c640 1 --2- 172.21.15.46:0/1658046241 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff2d0101240 0x7ff2d010e4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:37:01.986 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:01.982+0000 7ff2d544d640 1 --2- 172.21.15.46:0/1658046241 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff2d01008a0 0x7ff2d010df90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:37:01.986 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:01.982+0000 7ff2d5c4e640 1 -- 172.21.15.46:0/1658046241 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff2d0101240 msgr2=0x7ff2d010e4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:37:01.986 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:01.982+0000 7ff2d5c4e640 1 --2- 172.21.15.46:0/1658046241 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff2d0101240 0x7ff2d010e4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:37:01.986 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:01.982+0000 7ff2d5c4e640 1 -- 172.21.15.46:0/1658046241 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff2d01008a0 msgr2=0x7ff2d010df90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:37:01.987 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:01.982+0000 7ff2d5c4e640 1 --2- 172.21.15.46:0/1658046241 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff2d01008a0 0x7ff2d010df90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:37:01.987 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:01.982+0000 7ff2d5c4e640 1 -- 172.21.15.46:0/1658046241 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff2d01a6390 con 0x7ff2d01055d0 2024-03-31T23:37:01.987 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:01.982+0000 7ff2d544d640 1 --2- 172.21.15.46:0/1658046241 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff2d01008a0 0x7ff2d010df90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T23:37:01.987 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:01.982+0000 7ff2d5c4e640 1 --2- 172.21.15.46:0/1658046241 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff2d01055d0 0x7ff2d010ea10 secure :-1 s=READY pgs=1057 cs=0 l=1 rev1=1 crypto rx=0x7ff2c000eb10 tx=0x7ff2c000c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:37:01.987 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:01.982+0000 7ff2ce7fc640 1 -- 172.21.15.46:0/1658046241 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2c0004b20 con 0x7ff2d01055d0 2024-03-31T23:37:01.987 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:01.982+0000 7ff2ce7fc640 1 -- 172.21.15.46:0/1658046241 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff2c0004cc0 con 0x7ff2d01055d0 2024-03-31T23:37:01.987 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:01.982+0000 7ff2ce7fc640 1 -- 172.21.15.46:0/1658046241 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2c0010690 con 0x7ff2d01055d0 2024-03-31T23:37:01.987 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:01.982+0000 7ff2d76b1640 1 -- 172.21.15.46:0/1658046241 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff2d01a6680 con 0x7ff2d01055d0 2024-03-31T23:37:01.987 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:01.983+0000 7ff2d76b1640 1 -- 172.21.15.46:0/1658046241 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff2d01a6bd0 con 0x7ff2d01055d0 2024-03-31T23:37:01.987 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:01.983+0000 7ff2d76b1640 1 -- 172.21.15.46:0/1658046241 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff2d0101d60 con 0x7ff2d01055d0 2024-03-31T23:37:01.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:01.985+0000 7ff2ce7fc640 1 -- 172.21.15.46:0/1658046241 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 25) ==== 137451+0+0 (secure 0 0 0) 0x7ff2c0005000 con 0x7ff2d01055d0 2024-03-31T23:37:01.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:01.985+0000 7ff2ce7fc640 1 --2- 172.21.15.46:0/1658046241 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7ff2b80a26e0 0x7ff2b80a4ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:37:01.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:01.985+0000 7ff2ce7fc640 1 -- 172.21.15.46:0/1658046241 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(137..137 src has 1..137) ==== 9554+0+0 (secure 0 0 0) 0x7ff2c00cf660 con 0x7ff2d01055d0 2024-03-31T23:37:01.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:01.985+0000 7ff2d544d640 1 --2- 172.21.15.46:0/1658046241 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7ff2b80a26e0 0x7ff2b80a4ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:37:01.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:01.986+0000 7ff2d544d640 1 --2- 172.21.15.46:0/1658046241 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7ff2b80a26e0 0x7ff2b80a4ba0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7ff2c8005ab0 tx=0x7ff2c8005a20 comp rx=0 tx=0).ready entity=mgr.15654 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:37:01.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:01.989+0000 7ff2ce7fc640 1 -- 172.21.15.46:0/1658046241 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7ff2c0053280 con 0x7ff2d01055d0 2024-03-31T23:37:02.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:37:01 smithi046 ceph-mon[32498]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-03-31T23:37:02.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:37:01 smithi046 ceph-mon[32498]: pgmap v803: 129 pgs: 129 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 4.3 MiB/s wr, 352 op/s 2024-03-31T23:37:02.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:37:01 smithi142 ceph-mon[35271]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-03-31T23:37:02.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:37:01 smithi142 ceph-mon[35271]: pgmap v803: 129 pgs: 129 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 4.3 MiB/s wr, 352 op/s 2024-03-31T23:37:02.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:02.195+0000 7ff2d76b1640 1 -- 172.21.15.46:0/1658046241 --> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}) -- 0x7ff2d0108b90 con 0x7ff2b80a26e0 2024-03-31T23:37:02.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:02.205+0000 7ff2ce7fc640 1 -- 172.21.15.46:0/1658046241 <== mgr.15654 v2:172.21.15.46:6800/685378567 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+351 (secure 0 0 0) 0x7ff2c8004210 con 0x7ff2b80a26e0 2024-03-31T23:37:02.206 INFO:teuthology.orchestra.run.smithi046.stdout:{"fs": "cephfs", "subvol": null, "group": null, "path": "/", "rel_path": "/", "schedule": "1m", "retention": {"m": 6, "h": 3}, "start": "2024-03-31T00:00:00", "created": "2024-03-31T23:11:43", "first": "2024-03-31T23:12:00", "last": "2024-03-31T23:37:00", "last_pruned": "2024-03-31T23:37:00", "created_count": 26, "pruned_count": 20, "active": true} 2024-03-31T23:37:02.209 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:02.207+0000 7ff2d76b1640 1 -- 172.21.15.46:0/1658046241 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7ff2b80a26e0 msgr2=0x7ff2b80a4ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:37:02.209 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:02.207+0000 7ff2d76b1640 1 --2- 172.21.15.46:0/1658046241 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7ff2b80a26e0 0x7ff2b80a4ba0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7ff2c8005ab0 tx=0x7ff2c8005a20 comp rx=0 tx=0).stop 2024-03-31T23:37:02.209 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:02.207+0000 7ff2d76b1640 1 -- 172.21.15.46:0/1658046241 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff2d01055d0 msgr2=0x7ff2d010ea10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:37:02.209 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:02.207+0000 7ff2d76b1640 1 --2- 172.21.15.46:0/1658046241 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff2d01055d0 0x7ff2d010ea10 secure :-1 s=READY pgs=1057 cs=0 l=1 rev1=1 crypto rx=0x7ff2c000eb10 tx=0x7ff2c000c2d0 comp rx=0 tx=0).stop 2024-03-31T23:37:02.209 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:02.207+0000 7ff2d76b1640 1 -- 172.21.15.46:0/1658046241 shutdown_connections 2024-03-31T23:37:02.209 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:02.207+0000 7ff2d76b1640 1 --2- 172.21.15.46:0/1658046241 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7ff2b80a26e0 0x7ff2b80a4ba0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:37:02.209 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:02.207+0000 7ff2d76b1640 1 --2- 172.21.15.46:0/1658046241 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff2d01055d0 0x7ff2d010ea10 unknown :-1 s=CLOSED pgs=1057 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:37:02.209 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:02.207+0000 7ff2d76b1640 1 --2- 172.21.15.46:0/1658046241 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff2d0101240 0x7ff2d010e4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:37:02.209 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:02.207+0000 7ff2d76b1640 1 --2- 172.21.15.46:0/1658046241 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff2d01008a0 0x7ff2d010df90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:37:02.210 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:02.207+0000 7ff2d76b1640 1 -- 172.21.15.46:0/1658046241 >> 172.21.15.46:0/1658046241 conn(0x7ff2d0077420 msgr2=0x7ff2d00fe4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:37:02.210 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:02.207+0000 7ff2d76b1640 1 -- 172.21.15.46:0/1658046241 shutdown_connections 2024-03-31T23:37:02.210 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:02.207+0000 7ff2d76b1640 1 -- 172.21.15.46:0/1658046241 wait complete. 2024-03-31T23:37:02.219 DEBUG:teuthology.orchestra.run.smithi046:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true' 2024-03-31T23:37:02.295 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:37:01 smithi162 ceph-mon[36237]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-03-31T23:37:02.295 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:37:01 smithi162 ceph-mon[36237]: pgmap v803: 129 pgs: 129 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 4.3 MiB/s wr, 352 op/s 2024-03-31T23:37:02.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:02.323+0000 7f7c59a8f640 1 -- 172.21.15.46:0/2967155181 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7c54105600 msgr2=0x7f7c541059e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:37:02.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:02.323+0000 7f7c59a8f640 1 --2- 172.21.15.46:0/2967155181 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7c54105600 0x7f7c541059e0 secure :-1 s=READY pgs=1058 cs=0 l=1 rev1=1 crypto rx=0x7f7c44009900 tx=0x7f7c4402bc30 comp rx=0 tx=0).stop 2024-03-31T23:37:02.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:02.323+0000 7f7c59a8f640 1 -- 172.21.15.46:0/2967155181 shutdown_connections 2024-03-31T23:37:02.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:02.323+0000 7f7c59a8f640 1 --2- 172.21.15.46:0/2967155181 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7c54101780 0x7f7c5410d490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:37:02.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:02.323+0000 7f7c59a8f640 1 --2- 172.21.15.46:0/2967155181 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7c54100dc0 0x7f7c54101240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:37:02.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:02.323+0000 7f7c59a8f640 1 --2- 172.21.15.46:0/2967155181 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7c54105600 0x7f7c541059e0 unknown :-1 s=CLOSED pgs=1058 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:37:02.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:02.323+0000 7f7c59a8f640 1 -- 172.21.15.46:0/2967155181 >> 172.21.15.46:0/2967155181 conn(0x7f7c54076be0 msgr2=0x7f7c540fdf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:37:02.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:02.323+0000 7f7c59a8f640 1 -- 172.21.15.46:0/2967155181 shutdown_connections 2024-03-31T23:37:02.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:02.323+0000 7f7c59a8f640 1 -- 172.21.15.46:0/2967155181 wait complete. 2024-03-31T23:37:02.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:02.323+0000 7f7c59a8f640 1 Processor -- start 2024-03-31T23:37:02.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:02.323+0000 7f7c59a8f640 1 -- start start 2024-03-31T23:37:02.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:02.324+0000 7f7c59a8f640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7c54100dc0 0x7f7c5419c940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:37:02.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:02.324+0000 7f7c59a8f640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7c54101780 0x7f7c5419ce80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:37:02.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:02.324+0000 7f7c59a8f640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7c54105600 0x7f7c54196af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:37:02.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:02.324+0000 7f7c59a8f640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f7c541108f0 con 0x7f7c54100dc0 2024-03-31T23:37:02.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:02.324+0000 7f7c59a8f640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f7c54110770 con 0x7f7c54101780 2024-03-31T23:37:02.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:02.324+0000 7f7c59a8f640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f7c54110a70 con 0x7f7c54105600 2024-03-31T23:37:02.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:02.324+0000 7f7c537fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7c54100dc0 0x7f7c5419c940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:37:02.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:02.324+0000 7f7c537fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7c54100dc0 0x7f7c5419c940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:48810/0 (socket says 172.21.15.46:48810) 2024-03-31T23:37:02.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:02.324+0000 7f7c537fe640 1 -- 172.21.15.46:0/2213962201 learned_addr learned my addr 172.21.15.46:0/2213962201 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:37:02.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:02.324+0000 7f7c52ffd640 1 --2- 172.21.15.46:0/2213962201 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7c54101780 0x7f7c5419ce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:37:02.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:02.324+0000 7f7c53fff640 1 --2- 172.21.15.46:0/2213962201 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7c54105600 0x7f7c54196af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:37:02.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:02.324+0000 7f7c537fe640 1 -- 172.21.15.46:0/2213962201 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7c54105600 msgr2=0x7f7c54196af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:37:02.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:02.324+0000 7f7c537fe640 1 --2- 172.21.15.46:0/2213962201 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7c54105600 0x7f7c54196af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:37:02.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:02.324+0000 7f7c537fe640 1 -- 172.21.15.46:0/2213962201 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7c54101780 msgr2=0x7f7c5419ce80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:37:02.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:02.324+0000 7f7c537fe640 1 --2- 172.21.15.46:0/2213962201 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7c54101780 0x7f7c5419ce80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:37:02.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:02.324+0000 7f7c537fe640 1 -- 172.21.15.46:0/2213962201 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c54197270 con 0x7f7c54100dc0 2024-03-31T23:37:02.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:02.324+0000 7f7c537fe640 1 --2- 172.21.15.46:0/2213962201 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7c54100dc0 0x7f7c5419c940 secure :-1 s=READY pgs=1059 cs=0 l=1 rev1=1 crypto rx=0x7f7c44006010 tx=0x7f7c44005d90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:37:02.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:02.324+0000 7f7c50ff9640 1 -- 172.21.15.46:0/2213962201 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c44043020 con 0x7f7c54100dc0 2024-03-31T23:37:02.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:02.324+0000 7f7c50ff9640 1 -- 172.21.15.46:0/2213962201 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7c44004350 con 0x7f7c54100dc0 2024-03-31T23:37:02.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:02.325+0000 7f7c59a8f640 1 -- 172.21.15.46:0/2213962201 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7c54197500 con 0x7f7c54100dc0 2024-03-31T23:37:02.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:02.325+0000 7f7c50ff9640 1 -- 172.21.15.46:0/2213962201 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c44003a10 con 0x7f7c54100dc0 2024-03-31T23:37:02.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:02.325+0000 7f7c59a8f640 1 -- 172.21.15.46:0/2213962201 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7c541a3800 con 0x7f7c54100dc0 2024-03-31T23:37:02.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:02.325+0000 7f7c59a8f640 1 -- 172.21.15.46:0/2213962201 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7c5404f4b0 con 0x7f7c54100dc0 2024-03-31T23:37:02.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:02.327+0000 7f7c50ff9640 1 -- 172.21.15.46:0/2213962201 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 25) ==== 137451+0+0 (secure 0 0 0) 0x7f7c44046020 con 0x7f7c54100dc0 2024-03-31T23:37:02.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:02.327+0000 7f7c50ff9640 1 --2- 172.21.15.46:0/2213962201 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f7c3c0a26b0 0x7f7c3c0a4b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:37:02.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:02.327+0000 7f7c52ffd640 1 --2- 172.21.15.46:0/2213962201 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f7c3c0a26b0 0x7f7c3c0a4b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:37:02.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:02.327+0000 7f7c50ff9640 1 -- 172.21.15.46:0/2213962201 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(137..137 src has 1..137) ==== 9554+0+0 (secure 0 0 0) 0x7f7c440f0ea0 con 0x7f7c54100dc0 2024-03-31T23:37:02.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:02.328+0000 7f7c52ffd640 1 --2- 172.21.15.46:0/2213962201 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f7c3c0a26b0 0x7f7c3c0a4b70 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f7c540ff0c0 tx=0x7f7c48005f90 comp rx=0 tx=0).ready entity=mgr.15654 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:37:02.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:02.331+0000 7f7c50ff9640 1 -- 172.21.15.46:0/2213962201 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f7c4404d070 con 0x7f7c54100dc0 2024-03-31T23:37:02.537 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:02.536+0000 7f7c59a8f640 1 -- 172.21.15.46:0/2213962201 --> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}) -- 0x7f7c54103960 con 0x7f7c3c0a26b0 2024-03-31T23:37:02.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:02.547+0000 7f7c50ff9640 1 -- 172.21.15.46:0/2213962201 <== mgr.15654 v2:172.21.15.46:6800/685378567 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+10 (secure 0 0 0) 0x7f7c54103960 con 0x7f7c3c0a26b0 2024-03-31T23:37:02.548 INFO:teuthology.orchestra.run.smithi046.stdout:/ 1m 6m3h 2024-03-31T23:37:02.550 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:02.549+0000 7f7c59a8f640 1 -- 172.21.15.46:0/2213962201 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f7c3c0a26b0 msgr2=0x7f7c3c0a4b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:37:02.550 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:02.549+0000 7f7c59a8f640 1 --2- 172.21.15.46:0/2213962201 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f7c3c0a26b0 0x7f7c3c0a4b70 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f7c540ff0c0 tx=0x7f7c48005f90 comp rx=0 tx=0).stop 2024-03-31T23:37:02.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:02.549+0000 7f7c59a8f640 1 -- 172.21.15.46:0/2213962201 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7c54100dc0 msgr2=0x7f7c5419c940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:37:02.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:02.549+0000 7f7c59a8f640 1 --2- 172.21.15.46:0/2213962201 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7c54100dc0 0x7f7c5419c940 secure :-1 s=READY pgs=1059 cs=0 l=1 rev1=1 crypto rx=0x7f7c44006010 tx=0x7f7c44005d90 comp rx=0 tx=0).stop 2024-03-31T23:37:02.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:02.549+0000 7f7c59a8f640 1 -- 172.21.15.46:0/2213962201 shutdown_connections 2024-03-31T23:37:02.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:02.549+0000 7f7c59a8f640 1 --2- 172.21.15.46:0/2213962201 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f7c3c0a26b0 0x7f7c3c0a4b70 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:37:02.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:02.549+0000 7f7c59a8f640 1 --2- 172.21.15.46:0/2213962201 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7c54105600 0x7f7c54196af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:37:02.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:02.549+0000 7f7c59a8f640 1 --2- 172.21.15.46:0/2213962201 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7c54101780 0x7f7c5419ce80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:37:02.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:02.549+0000 7f7c59a8f640 1 --2- 172.21.15.46:0/2213962201 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7c54100dc0 0x7f7c5419c940 unknown :-1 s=CLOSED pgs=1059 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:37:02.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:02.549+0000 7f7c59a8f640 1 -- 172.21.15.46:0/2213962201 >> 172.21.15.46:0/2213962201 conn(0x7f7c54076be0 msgr2=0x7f7c54103250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:37:02.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:02.549+0000 7f7c59a8f640 1 -- 172.21.15.46:0/2213962201 shutdown_connections 2024-03-31T23:37:02.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:02.549+0000 7f7c59a8f640 1 -- 172.21.15.46:0/2213962201 wait complete. 2024-03-31T23:37:02.561 DEBUG:teuthology.run_tasks:Unwinding manager fwd_scrub 2024-03-31T23:37:02.576 INFO:tasks.fwd_scrub:joining ForwardScrubbers 2024-03-31T23:37:02.576 ERROR:teuthology.run_tasks:Manager failed: fwd_scrub Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph_384f31f3d3aee7f8ba3f60846b8844c049165583/qa/tasks/fwd_scrub.py", line 164, in task stop_all_fwd_scrubbers(ctx.ceph[config['cluster']].thrashers) File "/home/teuthworker/src/git.ceph.com_ceph_384f31f3d3aee7f8ba3f60846b8844c049165583/qa/tasks/fwd_scrub.py", line 99, in stop_all_fwd_scrubbers raise RuntimeError(f"error during scrub thrashing: {thrasher.exception}") RuntimeError: error during scrub thrashing: rank damage found: {'backtrace'} 2024-03-31T23:37:02.577 DEBUG:teuthology.run_tasks:Unwinding manager check-counter 2024-03-31T23:37:02.586 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-31T23:37:02.693 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:02.690+0000 7f5ffb0a4640 1 -- 172.21.15.46:0/1073212302 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5ff4102a00 msgr2=0x7f5ff4102de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:37:02.693 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:02.690+0000 7f5ffb0a4640 1 --2- 172.21.15.46:0/1073212302 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5ff4102a00 0x7f5ff4102de0 secure :-1 s=READY pgs=1060 cs=0 l=1 rev1=1 crypto rx=0x7f5ff0009900 tx=0x7f5ff002bcd0 comp rx=0 tx=0).stop 2024-03-31T23:37:02.693 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:02.690+0000 7f5ffb0a4640 1 -- 172.21.15.46:0/1073212302 shutdown_connections 2024-03-31T23:37:02.693 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:02.690+0000 7f5ffb0a4640 1 --2- 172.21.15.46:0/1073212302 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5ff410bdd0 0x7f5ff410e200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:37:02.694 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:02.690+0000 7f5ffb0a4640 1 --2- 172.21.15.46:0/1073212302 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5ff4103320 0x7f5ff410b800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:37:02.694 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:02.690+0000 7f5ffb0a4640 1 --2- 172.21.15.46:0/1073212302 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5ff4102a00 0x7f5ff4102de0 unknown :-1 s=CLOSED pgs=1060 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:37:02.694 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:02.690+0000 7f5ffb0a4640 1 -- 172.21.15.46:0/1073212302 >> 172.21.15.46:0/1073212302 conn(0x7f5ff40777b0 msgr2=0x7f5ff4077bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:37:02.694 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:02.690+0000 7f5ffb0a4640 1 -- 172.21.15.46:0/1073212302 shutdown_connections 2024-03-31T23:37:02.694 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:02.690+0000 7f5ffb0a4640 1 -- 172.21.15.46:0/1073212302 wait complete. 2024-03-31T23:37:02.694 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:02.691+0000 7f5ffb0a4640 1 Processor -- start 2024-03-31T23:37:02.694 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:02.691+0000 7f5ffb0a4640 1 -- start start 2024-03-31T23:37:02.694 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:02.691+0000 7f5ffb0a4640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5ff4102a00 0x7f5ff4198050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:37:02.694 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:02.691+0000 7f5ffb0a4640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5ff4103320 0x7f5ff4198590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:37:02.694 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:02.691+0000 7f5ffb0a4640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5ff410bdd0 0x7f5ff419c910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:37:02.694 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:02.691+0000 7f5ffb0a4640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f5ff4111a60 con 0x7f5ff410bdd0 2024-03-31T23:37:02.694 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:02.691+0000 7f5ffb0a4640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f5ff41118e0 con 0x7f5ff4103320 2024-03-31T23:37:02.694 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:02.691+0000 7f5ffb0a4640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f5ff4111be0 con 0x7f5ff4102a00 2024-03-31T23:37:02.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:02.691+0000 7f5ff9641640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5ff410bdd0 0x7f5ff419c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:37:02.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:02.691+0000 7f5ff9641640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5ff410bdd0 0x7f5ff419c910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:48826/0 (socket says 172.21.15.46:48826) 2024-03-31T23:37:02.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:02.691+0000 7f5ff9641640 1 -- 172.21.15.46:0/3035498447 learned_addr learned my addr 172.21.15.46:0/3035498447 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:37:02.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:02.691+0000 7f5febfff640 1 --2- 172.21.15.46:0/3035498447 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5ff4103320 0x7f5ff4198590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:37:02.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:02.691+0000 7f5ff8e40640 1 --2- 172.21.15.46:0/3035498447 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5ff4102a00 0x7f5ff4198050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:37:02.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:02.691+0000 7f5ff9641640 1 -- 172.21.15.46:0/3035498447 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5ff4102a00 msgr2=0x7f5ff4198050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:37:02.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:02.691+0000 7f5ff9641640 1 --2- 172.21.15.46:0/3035498447 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5ff4102a00 0x7f5ff4198050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:37:02.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:02.691+0000 7f5ff9641640 1 -- 172.21.15.46:0/3035498447 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5ff4103320 msgr2=0x7f5ff4198590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:37:02.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:02.691+0000 7f5ff9641640 1 --2- 172.21.15.46:0/3035498447 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5ff4103320 0x7f5ff4198590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:37:02.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:02.691+0000 7f5ff9641640 1 -- 172.21.15.46:0/3035498447 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ff419d040 con 0x7f5ff410bdd0 2024-03-31T23:37:02.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:02.692+0000 7f5ff9641640 1 --2- 172.21.15.46:0/3035498447 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5ff410bdd0 0x7f5ff419c910 secure :-1 s=READY pgs=1061 cs=0 l=1 rev1=1 crypto rx=0x7f5fec00eb10 tx=0x7f5fec00c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:37:02.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:02.692+0000 7f5fe9ffb640 1 -- 172.21.15.46:0/3035498447 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fec004b20 con 0x7f5ff410bdd0 2024-03-31T23:37:02.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:02.692+0000 7f5fe9ffb640 1 -- 172.21.15.46:0/3035498447 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5fec004cc0 con 0x7f5ff410bdd0 2024-03-31T23:37:02.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:02.692+0000 7f5ffb0a4640 1 -- 172.21.15.46:0/3035498447 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5ff419d330 con 0x7f5ff410bdd0 2024-03-31T23:37:02.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:02.692+0000 7f5fe9ffb640 1 -- 172.21.15.46:0/3035498447 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fec010690 con 0x7f5ff410bdd0 2024-03-31T23:37:02.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:02.692+0000 7f5ffb0a4640 1 -- 172.21.15.46:0/3035498447 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5ff41a4bc0 con 0x7f5ff410bdd0 2024-03-31T23:37:02.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:02.693+0000 7f5ffb0a4640 1 -- 172.21.15.46:0/3035498447 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5ff4068990 con 0x7f5ff410bdd0 2024-03-31T23:37:02.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:02.694+0000 7f5fe9ffb640 1 -- 172.21.15.46:0/3035498447 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 25) ==== 137451+0+0 (secure 0 0 0) 0x7f5fec005000 con 0x7f5ff410bdd0 2024-03-31T23:37:02.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:02.695+0000 7f5fe9ffb640 1 --2- 172.21.15.46:0/3035498447 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f5fdc0a2610 0x7f5fdc0a4ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:37:02.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:02.695+0000 7f5ff8e40640 1 --2- 172.21.15.46:0/3035498447 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f5fdc0a2610 0x7f5fdc0a4ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:37:02.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:02.695+0000 7f5fe9ffb640 1 -- 172.21.15.46:0/3035498447 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(137..137 src has 1..137) ==== 9554+0+0 (secure 0 0 0) 0x7f5fec0cf970 con 0x7f5ff410bdd0 2024-03-31T23:37:02.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:02.695+0000 7f5ff8e40640 1 --2- 172.21.15.46:0/3035498447 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f5fdc0a2610 0x7f5fdc0a4ad0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f5ff0005ab0 tx=0x7f5ff0005a20 comp rx=0 tx=0).ready entity=mgr.15654 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:37:02.699 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:02.698+0000 7f5fe9ffb640 1 -- 172.21.15.46:0/3035498447 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f5fec09c380 con 0x7f5ff410bdd0 2024-03-31T23:37:02.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:02.907+0000 7f5ffb0a4640 1 -- 172.21.15.46:0/3035498447 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f5ff4106f20 con 0x7f5ff410bdd0 2024-03-31T23:37:02.911 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:02.911+0000 7f5fe9ffb640 1 -- 172.21.15.46:0/3035498447 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v138) ==== 74+0+25517 (secure 0 0 0) 0x7f5fec09bc80 con 0x7f5ff410bdd0 2024-03-31T23:37:02.912 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-31T23:37:02.912 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":138,"fsid":"6a9d8f08-efb2-11ee-b647-cb9ed24678a4","created":"2024-03-31T23:01:15.463711+0000","modified":"2024-03-31T23:37:02.854718+0000","last_up_change":"2024-03-31T23:08:20.199888+0000","last_in_change":"2024-03-31T23:07:59.438035+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-31T23:04:32.823751+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-31T23:09:09.145711+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-31T23:09:10.148579+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"138","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":47,"snap_epoch":138,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-31T23:09:13.419519+0000","flags":8197,"flags_names":"hashpspool,ec_overwrites,selfmanaged_snaps","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"138","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":47,"snap_epoch":138,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"47429095-3f46-4ed7-b7c4-504464719815","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":1824772695},{"type":"v1","addr":"172.21.15.46:6803","nonce":1824772695}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":1824772695},{"type":"v1","addr":"172.21.15.46:6805","nonce":1824772695}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":1824772695},{"type":"v1","addr":"172.21.15.46:6809","nonce":1824772695}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":1824772695},{"type":"v1","addr":"172.21.15.46:6807","nonce":1824772695}]},"public_addr":"172.21.15.46:6803/1824772695","cluster_addr":"172.21.15.46:6805/1824772695","heartbeat_back_addr":"172.21.15.46:6809/1824772695","heartbeat_front_addr":"172.21.15.46:6807/1824772695","state":["exists","up"]},{"osd":1,"uuid":"be56f883-411e-4dbe-b50e-a158134df4a6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6800","nonce":973693922},{"type":"v1","addr":"172.21.15.142:6801","nonce":973693922}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6802","nonce":973693922},{"type":"v1","addr":"172.21.15.142:6803","nonce":973693922}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6806","nonce":973693922},{"type":"v1","addr":"172.21.15.142:6807","nonce":973693922}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6804","nonce":973693922},{"type":"v1","addr":"172.21.15.142:6805","nonce":973693922}]},"public_addr":"172.21.15.142:6801/973693922","cluster_addr":"172.21.15.142:6803/973693922","heartbeat_back_addr":"172.21.15.142:6807/973693922","heartbeat_front_addr":"172.21.15.142:6805/973693922","state":["exists","up"]},{"osd":2,"uuid":"c99a8794-ab19-4260-9a8d-4bc0bd2c8175","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6800","nonce":4084261813},{"type":"v1","addr":"172.21.15.162:6801","nonce":4084261813}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6802","nonce":4084261813},{"type":"v1","addr":"172.21.15.162:6803","nonce":4084261813}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6806","nonce":4084261813},{"type":"v1","addr":"172.21.15.162:6807","nonce":4084261813}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6804","nonce":4084261813},{"type":"v1","addr":"172.21.15.162:6805","nonce":4084261813}]},"public_addr":"172.21.15.162:6801/4084261813","cluster_addr":"172.21.15.162:6803/4084261813","heartbeat_back_addr":"172.21.15.162:6807/4084261813","heartbeat_front_addr":"172.21.15.162:6805/4084261813","state":["exists","up"]},{"osd":3,"uuid":"f877814e-9ef6-4872-a081-5609bb9ebc06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":3496317970},{"type":"v1","addr":"172.21.15.46:6811","nonce":3496317970}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":3496317970},{"type":"v1","addr":"172.21.15.46:6813","nonce":3496317970}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":3496317970},{"type":"v1","addr":"172.21.15.46:6817","nonce":3496317970}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":3496317970},{"type":"v1","addr":"172.21.15.46:6815","nonce":3496317970}]},"public_addr":"172.21.15.46:6811/3496317970","cluster_addr":"172.21.15.46:6813/3496317970","heartbeat_back_addr":"172.21.15.46:6817/3496317970","heartbeat_front_addr":"172.21.15.46:6815/3496317970","state":["exists","up"]},{"osd":4,"uuid":"beecec0a-8280-44fe-8f5f-8498a35eca8f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6808","nonce":2996266929},{"type":"v1","addr":"172.21.15.142:6809","nonce":2996266929}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6810","nonce":2996266929},{"type":"v1","addr":"172.21.15.142:6811","nonce":2996266929}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6814","nonce":2996266929},{"type":"v1","addr":"172.21.15.142:6815","nonce":2996266929}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6812","nonce":2996266929},{"type":"v1","addr":"172.21.15.142:6813","nonce":2996266929}]},"public_addr":"172.21.15.142:6809/2996266929","cluster_addr":"172.21.15.142:6811/2996266929","heartbeat_back_addr":"172.21.15.142:6815/2996266929","heartbeat_front_addr":"172.21.15.142:6813/2996266929","state":["exists","up"]},{"osd":5,"uuid":"9da7ff75-f658-45ea-a02f-1943fdda76b9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6808","nonce":914208293},{"type":"v1","addr":"172.21.15.162:6809","nonce":914208293}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6810","nonce":914208293},{"type":"v1","addr":"172.21.15.162:6811","nonce":914208293}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6814","nonce":914208293},{"type":"v1","addr":"172.21.15.162:6815","nonce":914208293}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6812","nonce":914208293},{"type":"v1","addr":"172.21.15.162:6813","nonce":914208293}]},"public_addr":"172.21.15.162:6809/914208293","cluster_addr":"172.21.15.162:6811/914208293","heartbeat_back_addr":"172.21.15.162:6815/914208293","heartbeat_front_addr":"172.21.15.162:6813/914208293","state":["exists","up"]},{"osd":6,"uuid":"8fa11224-cb26-4d3f-87d1-6a8f31de2a19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":56291966},{"type":"v1","addr":"172.21.15.46:6819","nonce":56291966}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":56291966},{"type":"v1","addr":"172.21.15.46:6821","nonce":56291966}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6824","nonce":56291966},{"type":"v1","addr":"172.21.15.46:6825","nonce":56291966}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":56291966},{"type":"v1","addr":"172.21.15.46:6823","nonce":56291966}]},"public_addr":"172.21.15.46:6819/56291966","cluster_addr":"172.21.15.46:6821/56291966","heartbeat_back_addr":"172.21.15.46:6825/56291966","heartbeat_front_addr":"172.21.15.46:6823/56291966","state":["exists","up"]},{"osd":7,"uuid":"0d0338b7-4d0a-4bb0-b152-8f5224871d9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6816","nonce":2146598446},{"type":"v1","addr":"172.21.15.142:6817","nonce":2146598446}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6818","nonce":2146598446},{"type":"v1","addr":"172.21.15.142:6819","nonce":2146598446}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6822","nonce":2146598446},{"type":"v1","addr":"172.21.15.142:6823","nonce":2146598446}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6820","nonce":2146598446},{"type":"v1","addr":"172.21.15.142:6821","nonce":2146598446}]},"public_addr":"172.21.15.142:6817/2146598446","cluster_addr":"172.21.15.142:6819/2146598446","heartbeat_back_addr":"172.21.15.142:6823/2146598446","heartbeat_front_addr":"172.21.15.142:6821/2146598446","state":["exists","up"]},{"osd":8,"uuid":"8b6f6037-892a-416c-babb-7552648d0b3a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6816","nonce":1173078318},{"type":"v1","addr":"172.21.15.162:6817","nonce":1173078318}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6818","nonce":1173078318},{"type":"v1","addr":"172.21.15.162:6819","nonce":1173078318}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6822","nonce":1173078318},{"type":"v1","addr":"172.21.15.162:6823","nonce":1173078318}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6820","nonce":1173078318},{"type":"v1","addr":"172.21.15.162:6821","nonce":1173078318}]},"public_addr":"172.21.15.162:6817/1173078318","cluster_addr":"172.21.15.162:6819/1173078318","heartbeat_back_addr":"172.21.15.162:6823/1173078318","heartbeat_front_addr":"172.21.15.162:6821/1173078318","state":["exists","up"]},{"osd":9,"uuid":"faccce25-6bf2-4220-8858-cda82b72be68","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":532708881},{"type":"v1","addr":"172.21.15.46:6827","nonce":532708881}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6828","nonce":532708881},{"type":"v1","addr":"172.21.15.46:6829","nonce":532708881}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":532708881},{"type":"v1","addr":"172.21.15.46:6833","nonce":532708881}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":532708881},{"type":"v1","addr":"172.21.15.46:6831","nonce":532708881}]},"public_addr":"172.21.15.46:6827/532708881","cluster_addr":"172.21.15.46:6829/532708881","heartbeat_back_addr":"172.21.15.46:6833/532708881","heartbeat_front_addr":"172.21.15.46:6831/532708881","state":["exists","up"]},{"osd":10,"uuid":"cdd356db-f9eb-4080-95ab-db5d8223e572","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6824","nonce":2845222366},{"type":"v1","addr":"172.21.15.142:6825","nonce":2845222366}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6826","nonce":2845222366},{"type":"v1","addr":"172.21.15.142:6827","nonce":2845222366}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6830","nonce":2845222366},{"type":"v1","addr":"172.21.15.142:6831","nonce":2845222366}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6828","nonce":2845222366},{"type":"v1","addr":"172.21.15.142:6829","nonce":2845222366}]},"public_addr":"172.21.15.142:6825/2845222366","cluster_addr":"172.21.15.142:6827/2845222366","heartbeat_back_addr":"172.21.15.142:6831/2845222366","heartbeat_front_addr":"172.21.15.142:6829/2845222366","state":["exists","up"]},{"osd":11,"uuid":"72104d1c-3ea2-45ad-bc0b-0500f146c06b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6824","nonce":1986323778},{"type":"v1","addr":"172.21.15.162:6825","nonce":1986323778}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6826","nonce":1986323778},{"type":"v1","addr":"172.21.15.162:6827","nonce":1986323778}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6830","nonce":1986323778},{"type":"v1","addr":"172.21.15.162:6831","nonce":1986323778}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6828","nonce":1986323778},{"type":"v1","addr":"172.21.15.162:6829","nonce":1986323778}]},"public_addr":"172.21.15.162:6825/1986323778","cluster_addr":"172.21.15.162:6827/1986323778","heartbeat_back_addr":"172.21.15.162:6831/1986323778","heartbeat_front_addr":"172.21.15.162:6829/1986323778","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:03:44.996331+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:04:07.994738+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:04:30.942255+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:04:55.588884+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:05:19.900600+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:05:43.807442+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:06:09.026976+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:06:34.586784+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:06:59.751605+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:07:26.144495+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:07:52.326909+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:08:17.423042+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.b","mappings":[{"from":8,"to":11}]},{"pgid":"3.6","mappings":[{"from":4,"to":7}]},{"pgid":"3.c","mappings":[{"from":3,"to":9}]},{"pgid":"3.1a","mappings":[{"from":5,"to":6}]},{"pgid":"3.1b","mappings":[{"from":4,"to":8}]},{"pgid":"3.1d","mappings":[{"from":4,"to":8}]},{"pgid":"3.2a","mappings":[{"from":3,"to":6}]},{"pgid":"3.33","mappings":[{"from":5,"to":9}]},{"pgid":"3.36","mappings":[{"from":4,"to":9}]},{"pgid":"4.6","mappings":[{"from":5,"to":8}]},{"pgid":"4.17","mappings":[{"from":9,"to":8}]},{"pgid":"4.18","mappings":[{"from":9,"to":8}]},{"pgid":"4.1b","mappings":[{"from":9,"to":10}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.46:0/249968062":"2024-04-01T23:11:42.678401+0000","172.21.15.46:6801/162289569":"2024-04-01T23:11:42.678401+0000","172.21.15.46:6800/162289569":"2024-04-01T23:11:42.678401+0000","172.21.15.46:0/34745898":"2024-04-01T23:11:42.678401+0000","172.21.15.46:0/3306820105":"2024-04-01T23:11:42.678401+0000","172.21.15.46:0/4021039857":"2024-04-01T23:02:08.493619+0000","172.21.15.46:6801/3681182647":"2024-04-01T23:02:08.493619+0000","172.21.15.46:0/4022214806":"2024-04-01T23:02:08.493619+0000","172.21.15.46:6800/1515528215":"2024-04-01T23:01:41.910162+0000","172.21.15.46:0/1146542737":"2024-04-01T23:11:42.678401+0000","172.21.15.46:6800/3681182647":"2024-04-01T23:02:08.493619+0000","172.21.15.46:0/143885707":"2024-04-01T23:01:41.910162+0000","172.21.15.46:6801/1515528215":"2024-04-01T23:01:41.910162+0000","172.21.15.46:0/1680389778":"2024-04-01T23:01:41.910162+0000","172.21.15.46:0/1782133966":"2024-04-01T23:01:41.910162+0000","172.21.15.46:0/3925209376":"2024-04-01T23:02:08.493619+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":34,"length":1},{"begin":47,"length":1}]},{"pool":4,"snaps":[{"begin":34,"length":1},{"begin":47,"length":1}]}],"new_removed_snaps":[{"pool":3,"snaps":[{"begin":34,"length":1},{"begin":47,"length":1}]},{"pool":4,"snaps":[{"begin":34,"length":1},{"begin":47,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-31T23:37:02.916 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:02.915+0000 7f5ffb0a4640 1 -- 172.21.15.46:0/3035498447 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f5fdc0a2610 msgr2=0x7f5fdc0a4ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:37:02.916 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:02.915+0000 7f5ffb0a4640 1 --2- 172.21.15.46:0/3035498447 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f5fdc0a2610 0x7f5fdc0a4ad0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f5ff0005ab0 tx=0x7f5ff0005a20 comp rx=0 tx=0).stop 2024-03-31T23:37:02.916 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:02.915+0000 7f5ffb0a4640 1 -- 172.21.15.46:0/3035498447 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5ff410bdd0 msgr2=0x7f5ff419c910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:37:02.916 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:02.915+0000 7f5ffb0a4640 1 --2- 172.21.15.46:0/3035498447 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5ff410bdd0 0x7f5ff419c910 secure :-1 s=READY pgs=1061 cs=0 l=1 rev1=1 crypto rx=0x7f5fec00eb10 tx=0x7f5fec00c2d0 comp rx=0 tx=0).stop 2024-03-31T23:37:02.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:02.916+0000 7f5ffb0a4640 1 -- 172.21.15.46:0/3035498447 shutdown_connections 2024-03-31T23:37:02.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:02.916+0000 7f5ffb0a4640 1 --2- 172.21.15.46:0/3035498447 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f5fdc0a2610 0x7f5fdc0a4ad0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:37:02.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:02.916+0000 7f5ffb0a4640 1 --2- 172.21.15.46:0/3035498447 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5ff410bdd0 0x7f5ff419c910 unknown :-1 s=CLOSED pgs=1061 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:37:02.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:02.916+0000 7f5ffb0a4640 1 --2- 172.21.15.46:0/3035498447 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5ff4103320 0x7f5ff4198590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:37:02.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:02.916+0000 7f5ffb0a4640 1 --2- 172.21.15.46:0/3035498447 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5ff4102a00 0x7f5ff4198050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:37:02.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:02.916+0000 7f5ffb0a4640 1 -- 172.21.15.46:0/3035498447 >> 172.21.15.46:0/3035498447 conn(0x7f5ff40777b0 msgr2=0x7f5ff40feb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:37:02.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:02.917+0000 7f5ffb0a4640 1 -- 172.21.15.46:0/3035498447 shutdown_connections 2024-03-31T23:37:02.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:02.917+0000 7f5ffb0a4640 1 -- 172.21.15.46:0/3035498447 wait complete. 2024-03-31T23:37:02.937 INFO:tasks.check_counter.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-31T23:04:32.823751+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-31T23:09:09.145711+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-31T23:09:10.148579+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '138', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 47, 'snap_epoch': 138, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-31T23:09:13.419519+0000', 'flags': 8197, 'flags_names': 'hashpspool,ec_overwrites,selfmanaged_snaps', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '138', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 47, 'snap_epoch': 138, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-31T23:37:02.937 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-31T23:37:03.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:37:02 smithi046 ceph-mon[32498]: from='client.18183 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-03-31T23:37:03.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:37:02 smithi046 ceph-mon[32498]: from='client.18189 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-31T23:37:03.123 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:03.121+0000 7f5e2b7fd640 1 -- 172.21.15.46:0/1584263286 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5e241008a0 msgr2=0x7f5e24100d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:37:03.123 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:03.121+0000 7f5e2b7fd640 1 --2- 172.21.15.46:0/1584263286 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5e241008a0 0x7f5e24100d20 secure :-1 s=READY pgs=1062 cs=0 l=1 rev1=1 crypto rx=0x7f5e14009900 tx=0x7f5e1402bc50 comp rx=0 tx=0).stop 2024-03-31T23:37:03.123 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:03.121+0000 7f5e2b7fd640 1 -- 172.21.15.46:0/1584263286 shutdown_connections 2024-03-31T23:37:03.123 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:03.121+0000 7f5e2b7fd640 1 --2- 172.21.15.46:0/1584263286 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5e24101260 0x7f5e24069b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:37:03.123 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:03.121+0000 7f5e2b7fd640 1 --2- 172.21.15.46:0/1584263286 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5e241008a0 0x7f5e24100d20 secure :-1 s=CLOSED pgs=1062 cs=0 l=1 rev1=1 crypto rx=0x7f5e14009900 tx=0x7f5e1402bc50 comp rx=0 tx=0).stop 2024-03-31T23:37:03.123 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:03.121+0000 7f5e2b7fd640 1 --2- 172.21.15.46:0/1584263286 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5e241055c0 0x7f5e241059a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:37:03.123 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:03.121+0000 7f5e2b7fd640 1 -- 172.21.15.46:0/1584263286 >> 172.21.15.46:0/1584263286 conn(0x7f5e240773e0 msgr2=0x7f5e240777f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:37:03.128 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:03.127+0000 7f5e2b7fd640 1 -- 172.21.15.46:0/1584263286 shutdown_connections 2024-03-31T23:37:03.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:03.128+0000 7f5e2b7fd640 1 -- 172.21.15.46:0/1584263286 wait complete. 2024-03-31T23:37:03.130 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:03.129+0000 7f5e2b7fd640 1 Processor -- start 2024-03-31T23:37:03.130 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:03.129+0000 7f5e2b7fd640 1 -- start start 2024-03-31T23:37:03.130 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:03.129+0000 7f5e2b7fd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5e24101260 0x7f5e2419a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:37:03.130 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:03.129+0000 7f5e2b7fd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5e241055c0 0x7f5e2419a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:37:03.130 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:03.129+0000 7f5e2b7fd640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5e2419ecc0 0x7f5e2419f170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:37:03.130 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:03.129+0000 7f5e2b7fd640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f5e24113da0 con 0x7f5e241055c0 2024-03-31T23:37:03.130 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:03.129+0000 7f5e2b7fd640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f5e24113c20 con 0x7f5e2419ecc0 2024-03-31T23:37:03.130 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:03.129+0000 7f5e2b7fd640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f5e24113f20 con 0x7f5e24101260 2024-03-31T23:37:03.131 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:03.130+0000 7f5e29d9a640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5e2419ecc0 0x7f5e2419f170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:37:03.131 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:03.130+0000 7f5e29d9a640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5e2419ecc0 0x7f5e2419f170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.46:56286/0 (socket says 172.21.15.46:56286) 2024-03-31T23:37:03.131 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:03.130+0000 7f5e29d9a640 1 -- 172.21.15.46:0/747611063 learned_addr learned my addr 172.21.15.46:0/747611063 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:37:03.131 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:03.130+0000 7f5e29599640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5e24101260 0x7f5e2419a400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:37:03.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:03.131+0000 7f5e29d9a640 1 -- 172.21.15.46:0/747611063 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5e24101260 msgr2=0x7f5e2419a400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:37:03.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:03.131+0000 7f5e29d9a640 1 --2- 172.21.15.46:0/747611063 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5e24101260 0x7f5e2419a400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:37:03.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:03.131+0000 7f5e29d9a640 1 -- 172.21.15.46:0/747611063 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5e241055c0 msgr2=0x7f5e2419a940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:37:03.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:03.131+0000 7f5e29d9a640 1 --2- 172.21.15.46:0/747611063 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5e241055c0 0x7f5e2419a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:37:03.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:03.131+0000 7f5e29d9a640 1 -- 172.21.15.46:0/747611063 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5e2419f7b0 con 0x7f5e2419ecc0 2024-03-31T23:37:03.135 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:03.134+0000 7f5e29d9a640 1 --2- 172.21.15.46:0/747611063 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5e2419ecc0 0x7f5e2419f170 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f5e1c00b970 tx=0x7f5e1c00be30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:37:03.137 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:03.136+0000 7f5e1a7fc640 1 -- 172.21.15.46:0/747611063 <== mon.2 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e1c019070 con 0x7f5e2419ecc0 2024-03-31T23:37:03.137 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:03.136+0000 7f5e1a7fc640 1 -- 172.21.15.46:0/747611063 <== mon.2 v2:172.21.15.142:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5e1c014440 con 0x7f5e2419ecc0 2024-03-31T23:37:03.137 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:03.136+0000 7f5e1a7fc640 1 -- 172.21.15.46:0/747611063 <== mon.2 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e1c0075d0 con 0x7f5e2419ecc0 2024-03-31T23:37:03.137 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:03.136+0000 7f5e2b7fd640 1 -- 172.21.15.46:0/747611063 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5e2419faa0 con 0x7f5e2419ecc0 2024-03-31T23:37:03.138 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:03.137+0000 7f5e2b7fd640 1 -- 172.21.15.46:0/747611063 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5e241a7240 con 0x7f5e2419ecc0 2024-03-31T23:37:03.140 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:03.139+0000 7f5e2b7fd640 1 -- 172.21.15.46:0/747611063 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5e24067cb0 con 0x7f5e2419ecc0 2024-03-31T23:37:03.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:03.141+0000 7f5e1a7fc640 1 -- 172.21.15.46:0/747611063 <== mon.2 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 25) ==== 137451+0+0 (secure 0 0 0) 0x7f5e1c0145f0 con 0x7f5e2419ecc0 2024-03-31T23:37:03.143 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:03.142+0000 7f5e1a7fc640 1 --2- 172.21.15.46:0/747611063 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f5e100a2690 0x7f5e100a4b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:37:03.146 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:03.142+0000 7f5e1a7fc640 1 -- 172.21.15.46:0/747611063 <== mon.2 v2:172.21.15.142:3300/0 5 ==== osd_map(138..138 src has 1..138) ==== 9642+0+0 (secure 0 0 0) 0x7f5e1c098a00 con 0x7f5e2419ecc0 2024-03-31T23:37:03.147 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:03.143+0000 7f5e29599640 1 --2- 172.21.15.46:0/747611063 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f5e100a2690 0x7f5e100a4b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:37:03.149 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:03.148+0000 7f5e29599640 1 --2- 172.21.15.46:0/747611063 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f5e100a2690 0x7f5e100a4b50 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f5e20006fd0 tx=0x7f5e20008040 comp rx=0 tx=0).ready entity=mgr.15654 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:37:03.156 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:03.155+0000 7f5e1a7fc640 1 -- 172.21.15.46:0/747611063 <== mon.2 v2:172.21.15.142:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f5e1c09b060 con 0x7f5e2419ecc0 2024-03-31T23:37:03.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:37:02 smithi142 ceph-mon[35271]: from='client.18183 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-03-31T23:37:03.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:37:02 smithi142 ceph-mon[35271]: from='client.18189 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-31T23:37:03.295 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:37:02 smithi162 ceph-mon[36237]: from='client.18183 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-03-31T23:37:03.295 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:37:02 smithi162 ceph-mon[36237]: from='client.18189 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-31T23:37:03.529 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:03.527+0000 7f5e2b7fd640 1 -- 172.21.15.46:0/747611063 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f5e241a76c0 con 0x7f5e2419ecc0 2024-03-31T23:37:03.529 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:03.529+0000 7f5e1a7fc640 1 -- 172.21.15.46:0/747611063 <== mon.2 v2:172.21.15.142:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v138) ==== 93+0+10 (secure 0 0 0) 0x7f5e1c09b060 con 0x7f5e2419ecc0 2024-03-31T23:37:03.530 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 1 2024-03-31T23:37:03.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:03.533+0000 7f5e2b7fd640 1 -- 172.21.15.46:0/747611063 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f5e100a2690 msgr2=0x7f5e100a4b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:37:03.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:03.533+0000 7f5e2b7fd640 1 --2- 172.21.15.46:0/747611063 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f5e100a2690 0x7f5e100a4b50 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f5e20006fd0 tx=0x7f5e20008040 comp rx=0 tx=0).stop 2024-03-31T23:37:03.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:03.533+0000 7f5e2b7fd640 1 -- 172.21.15.46:0/747611063 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5e2419ecc0 msgr2=0x7f5e2419f170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:37:03.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:03.533+0000 7f5e2b7fd640 1 --2- 172.21.15.46:0/747611063 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5e2419ecc0 0x7f5e2419f170 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f5e1c00b970 tx=0x7f5e1c00be30 comp rx=0 tx=0).stop 2024-03-31T23:37:03.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:03.534+0000 7f5e2b7fd640 1 -- 172.21.15.46:0/747611063 shutdown_connections 2024-03-31T23:37:03.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:03.534+0000 7f5e2b7fd640 1 --2- 172.21.15.46:0/747611063 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f5e100a2690 0x7f5e100a4b50 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:37:03.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:03.534+0000 7f5e2b7fd640 1 --2- 172.21.15.46:0/747611063 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5e2419ecc0 0x7f5e2419f170 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:37:03.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:03.534+0000 7f5e2b7fd640 1 --2- 172.21.15.46:0/747611063 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5e241055c0 0x7f5e2419a940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:37:03.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:03.534+0000 7f5e2b7fd640 1 --2- 172.21.15.46:0/747611063 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5e24101260 0x7f5e2419a400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:37:03.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:03.534+0000 7f5e2b7fd640 1 -- 172.21.15.46:0/747611063 >> 172.21.15.46:0/747611063 conn(0x7f5e240773e0 msgr2=0x7f5e240ff890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:37:03.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:03.534+0000 7f5e2b7fd640 1 -- 172.21.15.46:0/747611063 shutdown_connections 2024-03-31T23:37:03.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:03.534+0000 7f5e2b7fd640 1 -- 172.21.15.46:0/747611063 wait complete. 2024-03-31T23:37:03.553 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-31T23:37:03.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:03.708+0000 7fa73378c640 1 -- 172.21.15.46:0/3303652847 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa72c106660 msgr2=0x7fa72c106a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:37:03.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:03.708+0000 7fa73378c640 1 --2- 172.21.15.46:0/3303652847 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa72c106660 0x7fa72c106a40 secure :-1 s=READY pgs=1063 cs=0 l=1 rev1=1 crypto rx=0x7fa728009930 tx=0x7fa72802bc30 comp rx=0 tx=0).stop 2024-03-31T23:37:03.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:03.708+0000 7fa73378c640 1 -- 172.21.15.46:0/3303652847 shutdown_connections 2024-03-31T23:37:03.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:03.708+0000 7fa73378c640 1 --2- 172.21.15.46:0/3303652847 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa72c100f80 0x7fa72c10d490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:37:03.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:03.708+0000 7fa73378c640 1 --2- 172.21.15.46:0/3303652847 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa72c1005c0 0x7fa72c100a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:37:03.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:03.708+0000 7fa73378c640 1 --2- 172.21.15.46:0/3303652847 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa72c106660 0x7fa72c106a40 unknown :-1 s=CLOSED pgs=1063 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:37:03.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:03.708+0000 7fa73378c640 1 -- 172.21.15.46:0/3303652847 >> 172.21.15.46:0/3303652847 conn(0x7fa72c077750 msgr2=0x7fa72c077b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:37:03.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:03.708+0000 7fa73378c640 1 -- 172.21.15.46:0/3303652847 shutdown_connections 2024-03-31T23:37:03.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:03.708+0000 7fa73378c640 1 -- 172.21.15.46:0/3303652847 wait complete. 2024-03-31T23:37:03.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:03.708+0000 7fa73378c640 1 Processor -- start 2024-03-31T23:37:03.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:03.709+0000 7fa73378c640 1 -- start start 2024-03-31T23:37:03.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:03.709+0000 7fa73378c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa72c1005c0 0x7fa72c102260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:37:03.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:03.709+0000 7fa73378c640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa72c100f80 0x7fa72c1027a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:37:03.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:03.709+0000 7fa73378c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa72c106660 0x7fa72c102ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:37:03.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:03.709+0000 7fa73378c640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fa72c110850 con 0x7fa72c106660 2024-03-31T23:37:03.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:03.709+0000 7fa73378c640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fa72c1106d0 con 0x7fa72c100f80 2024-03-31T23:37:03.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:03.709+0000 7fa73378c640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fa72c1109d0 con 0x7fa72c1005c0 2024-03-31T23:37:03.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:03.709+0000 7fa731528640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa72c1005c0 0x7fa72c102260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:37:03.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:03.709+0000 7fa731d29640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa72c106660 0x7fa72c102ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:37:03.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:03.709+0000 7fa731528640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa72c1005c0 0x7fa72c102260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.46:47862/0 (socket says 172.21.15.46:47862) 2024-03-31T23:37:03.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:03.709+0000 7fa731528640 1 -- 172.21.15.46:0/4104412423 learned_addr learned my addr 172.21.15.46:0/4104412423 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:37:03.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:03.709+0000 7fa730d27640 1 --2- 172.21.15.46:0/4104412423 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa72c100f80 0x7fa72c1027a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:37:03.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:03.709+0000 7fa731d29640 1 -- 172.21.15.46:0/4104412423 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa72c1005c0 msgr2=0x7fa72c102260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:37:03.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:03.709+0000 7fa731d29640 1 --2- 172.21.15.46:0/4104412423 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa72c1005c0 0x7fa72c102260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:37:03.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:03.709+0000 7fa731d29640 1 -- 172.21.15.46:0/4104412423 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa72c100f80 msgr2=0x7fa72c1027a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:37:03.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:03.709+0000 7fa731d29640 1 --2- 172.21.15.46:0/4104412423 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa72c100f80 0x7fa72c1027a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:37:03.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:03.709+0000 7fa731d29640 1 -- 172.21.15.46:0/4104412423 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa72c1a63a0 con 0x7fa72c106660 2024-03-31T23:37:03.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:03.709+0000 7fa731528640 1 --2- 172.21.15.46:0/4104412423 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa72c1005c0 0x7fa72c102260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:37:03.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:03.709+0000 7fa731d29640 1 --2- 172.21.15.46:0/4104412423 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa72c106660 0x7fa72c102ce0 secure :-1 s=READY pgs=1064 cs=0 l=1 rev1=1 crypto rx=0x7fa724009fd0 tx=0x7fa72400ef10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:37:03.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:03.710+0000 7fa7227fc640 1 -- 172.21.15.46:0/4104412423 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa72400ce20 con 0x7fa72c106660 2024-03-31T23:37:03.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:03.710+0000 7fa7227fc640 1 -- 172.21.15.46:0/4104412423 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa724005020 con 0x7fa72c106660 2024-03-31T23:37:03.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:03.710+0000 7fa7227fc640 1 -- 172.21.15.46:0/4104412423 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa724010690 con 0x7fa72c106660 2024-03-31T23:37:03.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:03.710+0000 7fa73378c640 1 -- 172.21.15.46:0/4104412423 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa72c1a6690 con 0x7fa72c106660 2024-03-31T23:37:03.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:03.710+0000 7fa73378c640 1 -- 172.21.15.46:0/4104412423 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa72c1a6b90 con 0x7fa72c106660 2024-03-31T23:37:03.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:03.711+0000 7fa73378c640 1 -- 172.21.15.46:0/4104412423 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa72c068540 con 0x7fa72c106660 2024-03-31T23:37:03.715 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:03.712+0000 7fa7227fc640 1 -- 172.21.15.46:0/4104412423 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 25) ==== 137451+0+0 (secure 0 0 0) 0x7fa7240051d0 con 0x7fa72c106660 2024-03-31T23:37:03.715 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:03.713+0000 7fa7227fc640 1 --2- 172.21.15.46:0/4104412423 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fa7180a26c0 0x7fa7180a4b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:37:03.715 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:03.713+0000 7fa7227fc640 1 -- 172.21.15.46:0/4104412423 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(138..138 src has 1..138) ==== 9642+0+0 (secure 0 0 0) 0x7fa7240cf520 con 0x7fa72c106660 2024-03-31T23:37:03.715 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:03.713+0000 7fa731528640 1 --2- 172.21.15.46:0/4104412423 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fa7180a26c0 0x7fa7180a4b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:37:03.715 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:03.714+0000 7fa731528640 1 --2- 172.21.15.46:0/4104412423 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fa7180a26c0 0x7fa7180a4b80 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fa728005ec0 tx=0x7fa728009620 comp rx=0 tx=0).ready entity=mgr.15654 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:37:03.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:03.716+0000 7fa7227fc640 1 -- 172.21.15.46:0/4104412423 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fa72409bf40 con 0x7fa72c106660 2024-03-31T23:37:03.902 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:03.901+0000 7fa73378c640 1 -- 172.21.15.46:0/4104412423 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fa72c1a6f50 con 0x7fa72c106660 2024-03-31T23:37:03.903 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:03.902+0000 7fa7227fc640 1 -- 172.21.15.46:0/4104412423 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v138) ==== 104+0+11 (secure 0 0 0) 0x7fa72409b840 con 0x7fa72c106660 2024-03-31T23:37:03.903 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 32 2024-03-31T23:37:03.905 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:03.904+0000 7fa73378c640 1 -- 172.21.15.46:0/4104412423 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fa7180a26c0 msgr2=0x7fa7180a4b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:37:03.905 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:03.904+0000 7fa73378c640 1 --2- 172.21.15.46:0/4104412423 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fa7180a26c0 0x7fa7180a4b80 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fa728005ec0 tx=0x7fa728009620 comp rx=0 tx=0).stop 2024-03-31T23:37:03.905 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:03.904+0000 7fa73378c640 1 -- 172.21.15.46:0/4104412423 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa72c106660 msgr2=0x7fa72c102ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:37:03.905 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:03.904+0000 7fa73378c640 1 --2- 172.21.15.46:0/4104412423 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa72c106660 0x7fa72c102ce0 secure :-1 s=READY pgs=1064 cs=0 l=1 rev1=1 crypto rx=0x7fa724009fd0 tx=0x7fa72400ef10 comp rx=0 tx=0).stop 2024-03-31T23:37:03.905 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:03.904+0000 7fa73378c640 1 -- 172.21.15.46:0/4104412423 shutdown_connections 2024-03-31T23:37:03.905 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:03.904+0000 7fa73378c640 1 --2- 172.21.15.46:0/4104412423 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fa7180a26c0 0x7fa7180a4b80 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:37:03.905 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:03.904+0000 7fa73378c640 1 --2- 172.21.15.46:0/4104412423 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa72c106660 0x7fa72c102ce0 unknown :-1 s=CLOSED pgs=1064 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:37:03.905 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:03.904+0000 7fa73378c640 1 --2- 172.21.15.46:0/4104412423 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa72c100f80 0x7fa72c1027a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:37:03.905 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:03.904+0000 7fa73378c640 1 --2- 172.21.15.46:0/4104412423 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa72c1005c0 0x7fa72c102260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:37:03.906 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:03.904+0000 7fa73378c640 1 -- 172.21.15.46:0/4104412423 >> 172.21.15.46:0/4104412423 conn(0x7fa72c077750 msgr2=0x7fa72c0ff930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:37:03.906 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:03.904+0000 7fa73378c640 1 -- 172.21.15.46:0/4104412423 shutdown_connections 2024-03-31T23:37:03.906 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:03.904+0000 7fa73378c640 1 -- 172.21.15.46:0/4104412423 wait complete. 2024-03-31T23:37:03.916 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-31T23:37:04.024 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:04.021+0000 7f336a548640 1 -- 172.21.15.46:0/1438100241 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f33641016c0 msgr2=0x7f3364101b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:37:04.025 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:04.021+0000 7f336a548640 1 --2- 172.21.15.46:0/1438100241 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f33641016c0 0x7f3364101b40 secure :-1 s=READY pgs=1065 cs=0 l=1 rev1=1 crypto rx=0x7f334c009930 tx=0x7f334c02bc30 comp rx=0 tx=0).stop 2024-03-31T23:37:04.025 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:04.021+0000 7f336a548640 1 -- 172.21.15.46:0/1438100241 shutdown_connections 2024-03-31T23:37:04.025 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:04.021+0000 7f336a548640 1 --2- 172.21.15.46:0/1438100241 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3364102080 0x7f336410e590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:37:04.025 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:04.021+0000 7f336a548640 1 --2- 172.21.15.46:0/1438100241 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f33641016c0 0x7f3364101b40 unknown :-1 s=CLOSED pgs=1065 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:37:04.025 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:04.021+0000 7f336a548640 1 --2- 172.21.15.46:0/1438100241 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f33641076d0 0x7f3364107ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:37:04.025 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:04.021+0000 7f336a548640 1 -- 172.21.15.46:0/1438100241 >> 172.21.15.46:0/1438100241 conn(0x7f33640fd1f0 msgr2=0x7f33640ff610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:37:04.025 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:04.021+0000 7f336a548640 1 -- 172.21.15.46:0/1438100241 shutdown_connections 2024-03-31T23:37:04.025 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:04.021+0000 7f336a548640 1 -- 172.21.15.46:0/1438100241 wait complete. 2024-03-31T23:37:04.025 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:04.022+0000 7f336a548640 1 Processor -- start 2024-03-31T23:37:04.025 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:04.022+0000 7f336a548640 1 -- start start 2024-03-31T23:37:04.025 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:04.022+0000 7f336a548640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f33641016c0 0x7f3364197f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:37:04.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:04.022+0000 7f336a548640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3364102080 0x7f33641984a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:37:04.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:04.022+0000 7f336a548640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f33641076d0 0x7f336419c820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:37:04.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:04.022+0000 7f336a548640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f3364111990 con 0x7f33641016c0 2024-03-31T23:37:04.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:04.022+0000 7f336a548640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f3364111810 con 0x7f3364102080 2024-03-31T23:37:04.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:04.022+0000 7f336a548640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f3364111b10 con 0x7f33641076d0 2024-03-31T23:37:04.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:04.022+0000 7f3363fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f33641016c0 0x7f3364197f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:37:04.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:04.022+0000 7f3363fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f33641016c0 0x7f3364197f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:48874/0 (socket says 172.21.15.46:48874) 2024-03-31T23:37:04.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:04.022+0000 7f3363fff640 1 -- 172.21.15.46:0/2992532885 learned_addr learned my addr 172.21.15.46:0/2992532885 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:37:04.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:04.023+0000 7f33637fe640 1 --2- 172.21.15.46:0/2992532885 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3364102080 0x7f33641984a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:37:04.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:04.023+0000 7f3368ae5640 1 --2- 172.21.15.46:0/2992532885 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f33641076d0 0x7f336419c820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:37:04.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:04.023+0000 7f3363fff640 1 -- 172.21.15.46:0/2992532885 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f33641076d0 msgr2=0x7f336419c820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:37:04.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:04.023+0000 7f3363fff640 1 --2- 172.21.15.46:0/2992532885 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f33641076d0 0x7f336419c820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:37:04.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:04.023+0000 7f3363fff640 1 -- 172.21.15.46:0/2992532885 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3364102080 msgr2=0x7f33641984a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:37:04.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:04.023+0000 7f3363fff640 1 --2- 172.21.15.46:0/2992532885 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3364102080 0x7f33641984a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:37:04.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:04.023+0000 7f3363fff640 1 -- 172.21.15.46:0/2992532885 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f336419cf50 con 0x7f33641016c0 2024-03-31T23:37:04.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:04.023+0000 7f3363fff640 1 --2- 172.21.15.46:0/2992532885 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f33641016c0 0x7f3364197f60 secure :-1 s=READY pgs=1066 cs=0 l=1 rev1=1 crypto rx=0x7f335400e9f0 tx=0x7f335400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:37:04.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:04.023+0000 7f33617fa640 1 -- 172.21.15.46:0/2992532885 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f335400cd80 con 0x7f33641016c0 2024-03-31T23:37:04.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:04.023+0000 7f33617fa640 1 -- 172.21.15.46:0/2992532885 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f33540054d0 con 0x7f33641016c0 2024-03-31T23:37:04.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:04.023+0000 7f336a548640 1 -- 172.21.15.46:0/2992532885 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f336419d240 con 0x7f33641016c0 2024-03-31T23:37:04.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:04.023+0000 7f33617fa640 1 -- 172.21.15.46:0/2992532885 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3354010490 con 0x7f33641016c0 2024-03-31T23:37:04.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:04.023+0000 7f336a548640 1 -- 172.21.15.46:0/2992532885 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f33641a4ad0 con 0x7f33641016c0 2024-03-31T23:37:04.028 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:04.024+0000 7f336a548640 1 -- 172.21.15.46:0/2992532885 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3364068990 con 0x7f33641016c0 2024-03-31T23:37:04.028 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:04.025+0000 7f33617fa640 1 -- 172.21.15.46:0/2992532885 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 25) ==== 137451+0+0 (secure 0 0 0) 0x7f3354005020 con 0x7f33641016c0 2024-03-31T23:37:04.028 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:04.026+0000 7f33617fa640 1 --2- 172.21.15.46:0/2992532885 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f33500a2610 0x7f33500a4ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:37:04.028 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:04.026+0000 7f33637fe640 1 --2- 172.21.15.46:0/2992532885 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f33500a2610 0x7f33500a4ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:37:04.028 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:04.026+0000 7f33617fa640 1 -- 172.21.15.46:0/2992532885 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(138..138 src has 1..138) ==== 9642+0+0 (secure 0 0 0) 0x7f33540cf340 con 0x7f33641016c0 2024-03-31T23:37:04.028 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:04.026+0000 7f33637fe640 1 --2- 172.21.15.46:0/2992532885 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f33500a2610 0x7f33500a4ad0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f334c005ab0 tx=0x7f334c005fd0 comp rx=0 tx=0).ready entity=mgr.15654 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:37:04.031 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:04.030+0000 7f33617fa640 1 -- 172.21.15.46:0/2992532885 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f335409bd00 con 0x7f33641016c0 2024-03-31T23:37:04.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:37:03 smithi046 ceph-mon[32498]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-03-31T23:37:04.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:37:03 smithi046 ceph-mon[32498]: osdmap e138: 12 total, 12 up, 12 in 2024-03-31T23:37:04.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:37:03 smithi046 ceph-mon[32498]: from='client.? 172.21.15.46:0/3035498447' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-31T23:37:04.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:37:03 smithi046 ceph-mon[32498]: pgmap v805: 129 pgs: 129 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 5.2 MiB/s wr, 423 op/s 2024-03-31T23:37:04.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:37:03 smithi046 ceph-mon[32498]: from='client.? 172.21.15.46:0/747611063' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-31T23:37:04.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:37:03 smithi142 ceph-mon[35271]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-03-31T23:37:04.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:37:03 smithi142 ceph-mon[35271]: osdmap e138: 12 total, 12 up, 12 in 2024-03-31T23:37:04.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:37:03 smithi142 ceph-mon[35271]: from='client.? 172.21.15.46:0/3035498447' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-31T23:37:04.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:37:03 smithi142 ceph-mon[35271]: pgmap v805: 129 pgs: 129 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 5.2 MiB/s wr, 423 op/s 2024-03-31T23:37:04.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:37:03 smithi142 ceph-mon[35271]: from='client.? 172.21.15.46:0/747611063' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-31T23:37:04.213 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:04.212+0000 7f336a548640 1 -- 172.21.15.46:0/2992532885 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f3364109c30 con 0x7f33641016c0 2024-03-31T23:37:04.213 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:04.212+0000 7f33617fa640 1 -- 172.21.15.46:0/2992532885 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v138) ==== 100+0+11 (secure 0 0 0) 0x7f335409b600 con 0x7f33641016c0 2024-03-31T23:37:04.213 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 64 2024-03-31T23:37:04.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:04.214+0000 7f336a548640 1 -- 172.21.15.46:0/2992532885 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f33500a2610 msgr2=0x7f33500a4ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:37:04.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:04.214+0000 7f336a548640 1 --2- 172.21.15.46:0/2992532885 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f33500a2610 0x7f33500a4ad0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f334c005ab0 tx=0x7f334c005fd0 comp rx=0 tx=0).stop 2024-03-31T23:37:04.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:04.214+0000 7f336a548640 1 -- 172.21.15.46:0/2992532885 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f33641016c0 msgr2=0x7f3364197f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:37:04.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:04.214+0000 7f336a548640 1 --2- 172.21.15.46:0/2992532885 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f33641016c0 0x7f3364197f60 secure :-1 s=READY pgs=1066 cs=0 l=1 rev1=1 crypto rx=0x7f335400e9f0 tx=0x7f335400edf0 comp rx=0 tx=0).stop 2024-03-31T23:37:04.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:04.214+0000 7f336a548640 1 -- 172.21.15.46:0/2992532885 shutdown_connections 2024-03-31T23:37:04.216 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:04.214+0000 7f336a548640 1 --2- 172.21.15.46:0/2992532885 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f33500a2610 0x7f33500a4ad0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:37:04.216 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:04.214+0000 7f336a548640 1 --2- 172.21.15.46:0/2992532885 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f33641076d0 0x7f336419c820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:37:04.216 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:04.214+0000 7f336a548640 1 --2- 172.21.15.46:0/2992532885 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3364102080 0x7f33641984a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:37:04.216 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:04.214+0000 7f336a548640 1 --2- 172.21.15.46:0/2992532885 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f33641016c0 0x7f3364197f60 unknown :-1 s=CLOSED pgs=1066 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:37:04.216 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:04.214+0000 7f336a548640 1 -- 172.21.15.46:0/2992532885 >> 172.21.15.46:0/2992532885 conn(0x7f33640fd1f0 msgr2=0x7f33640fdb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:37:04.216 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:04.214+0000 7f336a548640 1 -- 172.21.15.46:0/2992532885 shutdown_connections 2024-03-31T23:37:04.216 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:04.214+0000 7f336a548640 1 -- 172.21.15.46:0/2992532885 wait complete. 2024-03-31T23:37:04.226 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-31T23:37:04.295 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:37:03 smithi162 ceph-mon[36237]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-03-31T23:37:04.295 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:37:03 smithi162 ceph-mon[36237]: osdmap e138: 12 total, 12 up, 12 in 2024-03-31T23:37:04.296 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:37:03 smithi162 ceph-mon[36237]: from='client.? 172.21.15.46:0/3035498447' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-31T23:37:04.296 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:37:03 smithi162 ceph-mon[36237]: pgmap v805: 129 pgs: 129 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 5.2 MiB/s wr, 423 op/s 2024-03-31T23:37:04.296 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:37:03 smithi162 ceph-mon[36237]: from='client.? 172.21.15.46:0/747611063' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-31T23:37:04.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:04.332+0000 7fb0c2ef4640 1 -- 172.21.15.46:0/1347188293 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb0bc1020a0 msgr2=0x7fb0bc10e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:37:04.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:04.332+0000 7fb0c2ef4640 1 --2- 172.21.15.46:0/1347188293 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb0bc1020a0 0x7fb0bc10e5b0 secure :-1 s=READY pgs=1067 cs=0 l=1 rev1=1 crypto rx=0x7fb0b4009cd0 tx=0x7fb0b402c0f0 comp rx=0 tx=0).stop 2024-03-31T23:37:04.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:04.332+0000 7fb0c2ef4640 1 -- 172.21.15.46:0/1347188293 shutdown_connections 2024-03-31T23:37:04.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:04.332+0000 7fb0c2ef4640 1 --2- 172.21.15.46:0/1347188293 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb0bc1020a0 0x7fb0bc10e5b0 unknown :-1 s=CLOSED pgs=1067 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:37:04.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:04.332+0000 7fb0c2ef4640 1 --2- 172.21.15.46:0/1347188293 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb0bc1016e0 0x7fb0bc101b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:37:04.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:04.332+0000 7fb0c2ef4640 1 --2- 172.21.15.46:0/1347188293 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb0bc1076f0 0x7fb0bc107ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:37:04.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:04.332+0000 7fb0c2ef4640 1 -- 172.21.15.46:0/1347188293 >> 172.21.15.46:0/1347188293 conn(0x7fb0bc078b50 msgr2=0x7fb0bc078f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:37:04.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:04.332+0000 7fb0c2ef4640 1 -- 172.21.15.46:0/1347188293 shutdown_connections 2024-03-31T23:37:04.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:04.332+0000 7fb0c2ef4640 1 -- 172.21.15.46:0/1347188293 wait complete. 2024-03-31T23:37:04.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:04.333+0000 7fb0c2ef4640 1 Processor -- start 2024-03-31T23:37:04.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:04.333+0000 7fb0c2ef4640 1 -- start start 2024-03-31T23:37:04.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:04.333+0000 7fb0c2ef4640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb0bc1016e0 0x7fb0bc073a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:37:04.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:04.333+0000 7fb0c2ef4640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb0bc1020a0 0x7fb0bc073f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:37:04.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:04.333+0000 7fb0c2ef4640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb0bc1076f0 0x7fb0bc0744d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:37:04.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:04.333+0000 7fb0c2ef4640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fb0bc1119b0 con 0x7fb0bc1020a0 2024-03-31T23:37:04.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:04.333+0000 7fb0c2ef4640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fb0bc111830 con 0x7fb0bc1076f0 2024-03-31T23:37:04.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:04.333+0000 7fb0c2ef4640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fb0bc111b30 con 0x7fb0bc1016e0 2024-03-31T23:37:04.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:04.333+0000 7fb0bbfff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb0bc1020a0 0x7fb0bc073f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:37:04.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:04.333+0000 7fb0bbfff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb0bc1020a0 0x7fb0bc073f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:48892/0 (socket says 172.21.15.46:48892) 2024-03-31T23:37:04.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:04.333+0000 7fb0bbfff640 1 -- 172.21.15.46:0/4032938391 learned_addr learned my addr 172.21.15.46:0/4032938391 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:37:04.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:04.333+0000 7fb0c0c90640 1 --2- 172.21.15.46:0/4032938391 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb0bc1016e0 0x7fb0bc073a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:37:04.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:04.333+0000 7fb0c1491640 1 --2- 172.21.15.46:0/4032938391 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb0bc1076f0 0x7fb0bc0744d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:37:04.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:04.333+0000 7fb0bbfff640 1 -- 172.21.15.46:0/4032938391 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb0bc1016e0 msgr2=0x7fb0bc073a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:37:04.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:04.333+0000 7fb0bbfff640 1 --2- 172.21.15.46:0/4032938391 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb0bc1016e0 0x7fb0bc073a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:37:04.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:04.333+0000 7fb0bbfff640 1 -- 172.21.15.46:0/4032938391 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb0bc1076f0 msgr2=0x7fb0bc0744d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:37:04.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:04.333+0000 7fb0bbfff640 1 --2- 172.21.15.46:0/4032938391 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb0bc1076f0 0x7fb0bc0744d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:37:04.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:04.333+0000 7fb0bbfff640 1 -- 172.21.15.46:0/4032938391 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb0bc1a63b0 con 0x7fb0bc1020a0 2024-03-31T23:37:04.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:04.333+0000 7fb0bbfff640 1 --2- 172.21.15.46:0/4032938391 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb0bc1020a0 0x7fb0bc073f90 secure :-1 s=READY pgs=1068 cs=0 l=1 rev1=1 crypto rx=0x7fb0a800e9f0 tx=0x7fb0a800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:37:04.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:04.334+0000 7fb0b9ffb640 1 -- 172.21.15.46:0/4032938391 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0a800cd80 con 0x7fb0bc1020a0 2024-03-31T23:37:04.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:04.334+0000 7fb0b9ffb640 1 -- 172.21.15.46:0/4032938391 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb0a80108e0 con 0x7fb0bc1020a0 2024-03-31T23:37:04.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:04.334+0000 7fb0b9ffb640 1 -- 172.21.15.46:0/4032938391 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0a8018920 con 0x7fb0bc1020a0 2024-03-31T23:37:04.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:04.334+0000 7fb0c2ef4640 1 -- 172.21.15.46:0/4032938391 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb0bc1a66a0 con 0x7fb0bc1020a0 2024-03-31T23:37:04.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:04.334+0000 7fb0c2ef4640 1 -- 172.21.15.46:0/4032938391 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb0bc1a6c40 con 0x7fb0bc1020a0 2024-03-31T23:37:04.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:04.335+0000 7fb0c2ef4640 1 -- 172.21.15.46:0/4032938391 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb0bc068990 con 0x7fb0bc1020a0 2024-03-31T23:37:04.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:04.336+0000 7fb0b9ffb640 1 -- 172.21.15.46:0/4032938391 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 25) ==== 137451+0+0 (secure 0 0 0) 0x7fb0a8010a90 con 0x7fb0bc1020a0 2024-03-31T23:37:04.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:04.337+0000 7fb0b9ffb640 1 --2- 172.21.15.46:0/4032938391 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fb0ac0a2690 0x7fb0ac0a4b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:37:04.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:04.337+0000 7fb0c0c90640 1 --2- 172.21.15.46:0/4032938391 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fb0ac0a2690 0x7fb0ac0a4b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:37:04.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:04.337+0000 7fb0b9ffb640 1 -- 172.21.15.46:0/4032938391 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(138..138 src has 1..138) ==== 9642+0+0 (secure 0 0 0) 0x7fb0a80cf3b0 con 0x7fb0bc1020a0 2024-03-31T23:37:04.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:04.337+0000 7fb0c0c90640 1 --2- 172.21.15.46:0/4032938391 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fb0ac0a2690 0x7fb0ac0a4b50 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fb0b0009ae0 tx=0x7fb0b0005f90 comp rx=0 tx=0).ready entity=mgr.15654 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:37:04.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:04.341+0000 7fb0b9ffb640 1 -- 172.21.15.46:0/4032938391 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fb0a809bd70 con 0x7fb0bc1020a0 2024-03-31T23:37:04.524 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:04.523+0000 7fb0c2ef4640 1 -- 172.21.15.46:0/4032938391 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7fb0bc109c50 con 0x7fb0bc1020a0 2024-03-31T23:37:04.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:04.524+0000 7fb0b9ffb640 1 -- 172.21.15.46:0/4032938391 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v138) ==== 103+0+11 (secure 0 0 0) 0x7fb0a809b670 con 0x7fb0bc1020a0 2024-03-31T23:37:04.525 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 32 2024-03-31T23:37:04.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:04.526+0000 7fb0c2ef4640 1 -- 172.21.15.46:0/4032938391 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fb0ac0a2690 msgr2=0x7fb0ac0a4b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:37:04.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:04.526+0000 7fb0c2ef4640 1 --2- 172.21.15.46:0/4032938391 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fb0ac0a2690 0x7fb0ac0a4b50 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fb0b0009ae0 tx=0x7fb0b0005f90 comp rx=0 tx=0).stop 2024-03-31T23:37:04.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:04.526+0000 7fb0c2ef4640 1 -- 172.21.15.46:0/4032938391 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb0bc1020a0 msgr2=0x7fb0bc073f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:37:04.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:04.526+0000 7fb0c2ef4640 1 --2- 172.21.15.46:0/4032938391 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb0bc1020a0 0x7fb0bc073f90 secure :-1 s=READY pgs=1068 cs=0 l=1 rev1=1 crypto rx=0x7fb0a800e9f0 tx=0x7fb0a800eeb0 comp rx=0 tx=0).stop 2024-03-31T23:37:04.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:04.526+0000 7fb0c2ef4640 1 -- 172.21.15.46:0/4032938391 shutdown_connections 2024-03-31T23:37:04.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:04.526+0000 7fb0c2ef4640 1 --2- 172.21.15.46:0/4032938391 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fb0ac0a2690 0x7fb0ac0a4b50 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:37:04.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:04.526+0000 7fb0c2ef4640 1 --2- 172.21.15.46:0/4032938391 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb0bc1076f0 0x7fb0bc0744d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:37:04.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:04.526+0000 7fb0c2ef4640 1 --2- 172.21.15.46:0/4032938391 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb0bc1020a0 0x7fb0bc073f90 unknown :-1 s=CLOSED pgs=1068 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:37:04.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:04.526+0000 7fb0c2ef4640 1 --2- 172.21.15.46:0/4032938391 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb0bc1016e0 0x7fb0bc073a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:37:04.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:04.526+0000 7fb0c2ef4640 1 -- 172.21.15.46:0/4032938391 >> 172.21.15.46:0/4032938391 conn(0x7fb0bc078b50 msgr2=0x7fb0bc1041f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:37:04.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:04.526+0000 7fb0c2ef4640 1 -- 172.21.15.46:0/4032938391 shutdown_connections 2024-03-31T23:37:04.528 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:04.526+0000 7fb0c2ef4640 1 -- 172.21.15.46:0/4032938391 wait complete. 2024-03-31T23:37:04.538 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json-pretty 2024-03-31T23:37:04.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:04.640+0000 7fb5a38d4640 1 -- 172.21.15.46:0/2028106521 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb59c1076f0 msgr2=0x7fb59c107ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:37:04.644 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:04.640+0000 7fb5a38d4640 1 --2- 172.21.15.46:0/2028106521 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb59c1076f0 0x7fb59c107ad0 secure :-1 s=READY pgs=1069 cs=0 l=1 rev1=1 crypto rx=0x7fb598009930 tx=0x7fb59802bc30 comp rx=0 tx=0).stop 2024-03-31T23:37:04.644 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:04.640+0000 7fb5a38d4640 1 -- 172.21.15.46:0/2028106521 shutdown_connections 2024-03-31T23:37:04.644 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:04.640+0000 7fb5a38d4640 1 --2- 172.21.15.46:0/2028106521 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb59c1020a0 0x7fb59c10e5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:37:04.644 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:04.640+0000 7fb5a38d4640 1 --2- 172.21.15.46:0/2028106521 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb59c1016e0 0x7fb59c101b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:37:04.644 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:04.640+0000 7fb5a38d4640 1 --2- 172.21.15.46:0/2028106521 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb59c1076f0 0x7fb59c107ad0 unknown :-1 s=CLOSED pgs=1069 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:37:04.644 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:04.640+0000 7fb5a38d4640 1 -- 172.21.15.46:0/2028106521 >> 172.21.15.46:0/2028106521 conn(0x7fb59c0fd1f0 msgr2=0x7fb59c0ff610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:37:04.644 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:04.640+0000 7fb5a38d4640 1 -- 172.21.15.46:0/2028106521 shutdown_connections 2024-03-31T23:37:04.644 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:04.640+0000 7fb5a38d4640 1 -- 172.21.15.46:0/2028106521 wait complete. 2024-03-31T23:37:04.644 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:04.641+0000 7fb5a38d4640 1 Processor -- start 2024-03-31T23:37:04.644 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:04.641+0000 7fb5a38d4640 1 -- start start 2024-03-31T23:37:04.644 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:04.641+0000 7fb5a38d4640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb59c1016e0 0x7fb59c197f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:37:04.644 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:04.641+0000 7fb5a38d4640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb59c1020a0 0x7fb59c198480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:37:04.644 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:04.641+0000 7fb5a38d4640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb59c1076f0 0x7fb59c19c800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:37:04.644 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:04.641+0000 7fb5a38d4640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fb59c1119c0 con 0x7fb59c1076f0 2024-03-31T23:37:04.645 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:04.641+0000 7fb5a38d4640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fb59c111840 con 0x7fb59c1016e0 2024-03-31T23:37:04.645 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:04.641+0000 7fb5a38d4640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fb59c111b40 con 0x7fb59c1020a0 2024-03-31T23:37:04.645 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:04.641+0000 7fb5a1e71640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb59c1076f0 0x7fb59c19c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:37:04.645 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:04.641+0000 7fb5a1670640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb59c1016e0 0x7fb59c197f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:37:04.645 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:04.641+0000 7fb5a1e71640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb59c1076f0 0x7fb59c19c800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:48906/0 (socket says 172.21.15.46:48906) 2024-03-31T23:37:04.645 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:04.641+0000 7fb5a1e71640 1 -- 172.21.15.46:0/4154080297 learned_addr learned my addr 172.21.15.46:0/4154080297 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:37:04.645 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:04.641+0000 7fb5a0e6f640 1 --2- 172.21.15.46:0/4154080297 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb59c1020a0 0x7fb59c198480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:37:04.645 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:04.641+0000 7fb5a1e71640 1 -- 172.21.15.46:0/4154080297 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb59c1020a0 msgr2=0x7fb59c198480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:37:04.645 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:04.641+0000 7fb5a1e71640 1 --2- 172.21.15.46:0/4154080297 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb59c1020a0 0x7fb59c198480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:37:04.645 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:04.641+0000 7fb5a1e71640 1 -- 172.21.15.46:0/4154080297 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb59c1016e0 msgr2=0x7fb59c197f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:37:04.645 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:04.641+0000 7fb5a1e71640 1 --2- 172.21.15.46:0/4154080297 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb59c1016e0 0x7fb59c197f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:37:04.645 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:04.641+0000 7fb5a1e71640 1 -- 172.21.15.46:0/4154080297 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb59c19cf30 con 0x7fb59c1076f0 2024-03-31T23:37:04.645 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:04.641+0000 7fb5a1670640 1 --2- 172.21.15.46:0/4154080297 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb59c1016e0 0x7fb59c197f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:37:04.645 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:04.642+0000 7fb5a1e71640 1 --2- 172.21.15.46:0/4154080297 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb59c1076f0 0x7fb59c19c800 secure :-1 s=READY pgs=1070 cs=0 l=1 rev1=1 crypto rx=0x7fb59400eb10 tx=0x7fb59400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:37:04.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:04.642+0000 7fb5927fc640 1 -- 172.21.15.46:0/4154080297 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb594004b20 con 0x7fb59c1076f0 2024-03-31T23:37:04.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:04.642+0000 7fb5927fc640 1 -- 172.21.15.46:0/4154080297 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb594004cc0 con 0x7fb59c1076f0 2024-03-31T23:37:04.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:04.642+0000 7fb5927fc640 1 -- 172.21.15.46:0/4154080297 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb594010690 con 0x7fb59c1076f0 2024-03-31T23:37:04.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:04.642+0000 7fb5a38d4640 1 -- 172.21.15.46:0/4154080297 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb59c19d220 con 0x7fb59c1076f0 2024-03-31T23:37:04.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:04.642+0000 7fb5a38d4640 1 -- 172.21.15.46:0/4154080297 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb59c1a4ab0 con 0x7fb59c1076f0 2024-03-31T23:37:04.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:04.643+0000 7fb5a38d4640 1 -- 172.21.15.46:0/4154080297 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb59c068990 con 0x7fb59c1076f0 2024-03-31T23:37:04.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:04.644+0000 7fb5927fc640 1 -- 172.21.15.46:0/4154080297 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 25) ==== 137451+0+0 (secure 0 0 0) 0x7fb594005020 con 0x7fb59c1076f0 2024-03-31T23:37:04.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:04.644+0000 7fb5927fc640 1 --2- 172.21.15.46:0/4154080297 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fb5880a2610 0x7fb5880a4ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:37:04.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:04.644+0000 7fb5a1670640 1 --2- 172.21.15.46:0/4154080297 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fb5880a2610 0x7fb5880a4ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:37:04.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:04.644+0000 7fb5927fc640 1 -- 172.21.15.46:0/4154080297 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(138..138 src has 1..138) ==== 9642+0+0 (secure 0 0 0) 0x7fb5940cf4b0 con 0x7fb59c1076f0 2024-03-31T23:37:04.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:04.645+0000 7fb5a1670640 1 --2- 172.21.15.46:0/4154080297 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fb5880a2610 0x7fb5880a4ad0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fb598009900 tx=0x7fb598005a20 comp rx=0 tx=0).ready entity=mgr.15654 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:37:04.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:04.649+0000 7fb5927fc640 1 -- 172.21.15.46:0/4154080297 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fb59409bed0 con 0x7fb59c1076f0 2024-03-31T23:37:04.884 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:04.883+0000 7fb5a38d4640 1 -- 172.21.15.46:0/4154080297 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json-pretty"} v 0) -- 0x7fb59c1096c0 con 0x7fb59c1076f0 2024-03-31T23:37:04.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:04.909+0000 7fb5927fc640 1 -- 172.21.15.46:0/4154080297 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json-pretty"}]=0 v25) ==== 81+0+738935 (secure 0 0 0) 0x7fb5940bb070 con 0x7fb59c1076f0 2024-03-31T23:37:04.919 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-31T23:37:04.919 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-03-31T23:37:04.919 INFO:teuthology.orchestra.run.smithi046.stdout: "epoch": 25, 2024-03-31T23:37:04.919 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:04.919 INFO:teuthology.orchestra.run.smithi046.stdout: "active_gid": 15654, 2024-03-31T23:37:04.920 INFO:teuthology.orchestra.run.smithi046.stdout: "active_name": "x", 2024-03-31T23:37:04.920 INFO:teuthology.orchestra.run.smithi046.stdout: "active_addrs": { 2024-03-31T23:37:04.920 INFO:teuthology.orchestra.run.smithi046.stdout: "addrvec": [ 2024-03-31T23:37:04.920 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-03-31T23:37:04.920 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v2", 2024-03-31T23:37:04.920 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6800", 2024-03-31T23:37:04.920 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 685378567 2024-03-31T23:37:04.920 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:04.920 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-03-31T23:37:04.920 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-03-31T23:37:04.920 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6801", 2024-03-31T23:37:04.920 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 685378567 2024-03-31T23:37:04.920 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-31T23:37:04.920 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-03-31T23:37:04.921 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:04.921 INFO:teuthology.orchestra.run.smithi046.stdout: "active_addr": "172.21.15.46:6801/685378567", 2024-03-31T23:37:04.921 INFO:teuthology.orchestra.run.smithi046.stdout: "active_change": "2024-03-31T23:11:42.678637+0000", 2024-03-31T23:37:04.921 INFO:teuthology.orchestra.run.smithi046.stdout: "active_mgr_features": 4540701547738038271, 2024-03-31T23:37:04.921 INFO:teuthology.orchestra.run.smithi046.stdout: "available": true, 2024-03-31T23:37:04.921 INFO:teuthology.orchestra.run.smithi046.stdout: "standbys": [ 2024-03-31T23:37:04.921 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-03-31T23:37:04.921 INFO:teuthology.orchestra.run.smithi046.stdout: "gid": 25390, 2024-03-31T23:37:04.921 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "z", 2024-03-31T23:37:04.921 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr_features": 4540701547738038271, 2024-03-31T23:37:04.921 INFO:teuthology.orchestra.run.smithi046.stdout: "available_modules": [ 2024-03-31T23:37:04.921 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-03-31T23:37:04.921 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "alerts", 2024-03-31T23:37:04.921 INFO:teuthology.orchestra.run.smithi046.stdout: "can_run": true, 2024-03-31T23:37:04.921 INFO:teuthology.orchestra.run.smithi046.stdout: "error_string": "", 2024-03-31T23:37:04.922 INFO:teuthology.orchestra.run.smithi046.stdout: "module_options": { 2024-03-31T23:37:04.922 INFO:teuthology.orchestra.run.smithi046.stdout: "interval": { 2024-03-31T23:37:04.922 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "interval", 2024-03-31T23:37:04.922 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "secs", 2024-03-31T23:37:04.922 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:04.922 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:04.922 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "60", 2024-03-31T23:37:04.922 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:04.922 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:04.922 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:04.922 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "How frequently to reexamine health status", 2024-03-31T23:37:04.922 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:04.922 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:04.922 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:04.922 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:04.923 INFO:teuthology.orchestra.run.smithi046.stdout: "log_level": { 2024-03-31T23:37:04.923 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_level", 2024-03-31T23:37:04.923 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:04.923 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:04.923 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:04.923 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-31T23:37:04.923 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:04.923 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:04.923 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-31T23:37:04.923 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-31T23:37:04.923 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-31T23:37:04.923 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-31T23:37:04.923 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-31T23:37:04.923 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-31T23:37:04.924 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-31T23:37:04.924 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-31T23:37:04.924 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:04.924 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:04.924 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:04.924 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:04.924 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:04.924 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster": { 2024-03-31T23:37:04.924 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster", 2024-03-31T23:37:04.924 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:04.924 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:04.924 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:04.924 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-31T23:37:04.924 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:04.925 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:04.925 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:04.925 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:04.925 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:04.925 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:04.925 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:04.925 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:04.925 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster_level": { 2024-03-31T23:37:04.925 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster_level", 2024-03-31T23:37:04.925 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:04.925 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:04.925 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:04.925 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "info", 2024-03-31T23:37:04.926 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:04.926 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:04.926 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-31T23:37:04.926 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-31T23:37:04.926 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-31T23:37:04.926 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-31T23:37:04.926 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-31T23:37:04.926 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-31T23:37:04.926 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-31T23:37:04.926 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-31T23:37:04.926 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:04.926 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:04.926 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:04.926 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:04.926 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:04.927 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_file": { 2024-03-31T23:37:04.927 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_file", 2024-03-31T23:37:04.927 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:04.927 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:04.927 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:04.927 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-31T23:37:04.927 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:04.927 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:04.927 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:04.927 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:04.927 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:04.927 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:04.927 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:04.927 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:04.927 INFO:teuthology.orchestra.run.smithi046.stdout: "smtp_destination": { 2024-03-31T23:37:04.928 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "smtp_destination", 2024-03-31T23:37:04.928 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:04.928 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:04.928 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:04.928 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-31T23:37:04.928 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:04.928 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:04.928 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:04.928 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Email address to send alerts to", 2024-03-31T23:37:04.928 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:04.928 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:04.928 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:04.928 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:04.928 INFO:teuthology.orchestra.run.smithi046.stdout: "smtp_from_name": { 2024-03-31T23:37:04.928 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "smtp_from_name", 2024-03-31T23:37:04.929 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:04.929 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:04.929 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:04.929 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "Ceph", 2024-03-31T23:37:04.929 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:04.929 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:04.929 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:04.929 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Email From: name", 2024-03-31T23:37:04.929 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:04.929 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:04.929 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:04.929 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:04.929 INFO:teuthology.orchestra.run.smithi046.stdout: "smtp_host": { 2024-03-31T23:37:04.929 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "smtp_host", 2024-03-31T23:37:04.929 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:04.930 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:04.930 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:04.930 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-31T23:37:04.930 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:04.930 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:04.930 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:04.930 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "SMTP server", 2024-03-31T23:37:04.930 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:04.930 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:04.930 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:04.930 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:04.930 INFO:teuthology.orchestra.run.smithi046.stdout: "smtp_password": { 2024-03-31T23:37:04.930 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "smtp_password", 2024-03-31T23:37:04.930 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:04.930 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:04.931 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:04.931 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-31T23:37:04.931 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:04.931 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:04.931 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:04.931 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Password to authenticate with", 2024-03-31T23:37:04.931 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:04.931 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:04.931 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:04.931 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:04.931 INFO:teuthology.orchestra.run.smithi046.stdout: "smtp_port": { 2024-03-31T23:37:04.931 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "smtp_port", 2024-03-31T23:37:04.931 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "int", 2024-03-31T23:37:04.931 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:04.931 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:04.932 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "465", 2024-03-31T23:37:04.932 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:04.932 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:04.932 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:04.932 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "SMTP port", 2024-03-31T23:37:04.932 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:04.932 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:04.932 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:04.932 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:04.932 INFO:teuthology.orchestra.run.smithi046.stdout: "smtp_sender": { 2024-03-31T23:37:04.932 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "smtp_sender", 2024-03-31T23:37:04.932 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:04.932 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:04.932 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:04.933 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-31T23:37:04.933 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:04.933 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:04.933 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:04.933 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "SMTP envelope sender", 2024-03-31T23:37:04.933 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:04.933 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:04.933 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:04.933 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:04.933 INFO:teuthology.orchestra.run.smithi046.stdout: "smtp_ssl": { 2024-03-31T23:37:04.933 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "smtp_ssl", 2024-03-31T23:37:04.933 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:04.933 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:04.934 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:04.934 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "True", 2024-03-31T23:37:04.934 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:04.934 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:04.934 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:04.934 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Use SSL to connect to SMTP server", 2024-03-31T23:37:04.934 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:04.934 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:04.934 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:04.934 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:04.934 INFO:teuthology.orchestra.run.smithi046.stdout: "smtp_user": { 2024-03-31T23:37:04.934 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "smtp_user", 2024-03-31T23:37:04.934 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:04.934 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:04.935 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:04.935 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-31T23:37:04.935 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:04.935 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:04.935 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:04.935 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "User to authenticate as", 2024-03-31T23:37:04.935 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:04.935 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:04.935 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:04.935 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-31T23:37:04.935 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-31T23:37:04.935 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:04.935 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-03-31T23:37:04.935 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "balancer", 2024-03-31T23:37:04.936 INFO:teuthology.orchestra.run.smithi046.stdout: "can_run": true, 2024-03-31T23:37:04.936 INFO:teuthology.orchestra.run.smithi046.stdout: "error_string": "", 2024-03-31T23:37:04.936 INFO:teuthology.orchestra.run.smithi046.stdout: "module_options": { 2024-03-31T23:37:04.936 INFO:teuthology.orchestra.run.smithi046.stdout: "active": { 2024-03-31T23:37:04.936 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "active", 2024-03-31T23:37:04.936 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:04.936 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:04.936 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:04.936 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "True", 2024-03-31T23:37:04.936 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:04.936 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:04.936 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:04.936 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "automatically balance PGs across cluster", 2024-03-31T23:37:04.936 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:04.936 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:04.937 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:04.937 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:04.937 INFO:teuthology.orchestra.run.smithi046.stdout: "begin_time": { 2024-03-31T23:37:04.937 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "begin_time", 2024-03-31T23:37:04.937 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:04.937 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:04.937 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:04.937 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "0000", 2024-03-31T23:37:04.937 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:04.937 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:04.937 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:04.937 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "beginning time of day to automatically balance", 2024-03-31T23:37:04.937 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-31T23:37:04.937 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:04.937 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:04.938 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:04.938 INFO:teuthology.orchestra.run.smithi046.stdout: "begin_weekday": { 2024-03-31T23:37:04.938 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "begin_weekday", 2024-03-31T23:37:04.938 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "uint", 2024-03-31T23:37:04.938 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:04.938 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:04.938 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "0", 2024-03-31T23:37:04.938 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "0", 2024-03-31T23:37:04.938 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "6", 2024-03-31T23:37:04.938 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:04.938 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-03-31T23:37:04.938 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-31T23:37:04.938 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:04.938 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:04.939 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:04.939 INFO:teuthology.orchestra.run.smithi046.stdout: "crush_compat_max_iterations": { 2024-03-31T23:37:04.939 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "crush_compat_max_iterations", 2024-03-31T23:37:04.939 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "uint", 2024-03-31T23:37:04.939 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:04.939 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:04.939 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "25", 2024-03-31T23:37:04.939 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "1", 2024-03-31T23:37:04.939 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "250", 2024-03-31T23:37:04.939 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:04.939 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-03-31T23:37:04.939 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:04.939 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:04.939 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:04.939 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:04.940 INFO:teuthology.orchestra.run.smithi046.stdout: "crush_compat_metrics": { 2024-03-31T23:37:04.940 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "crush_compat_metrics", 2024-03-31T23:37:04.940 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:04.940 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:04.940 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:04.940 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "pgs,objects,bytes", 2024-03-31T23:37:04.940 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:04.940 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:04.940 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:04.940 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-03-31T23:37:04.940 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-03-31T23:37:04.940 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:04.940 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:04.940 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:04.940 INFO:teuthology.orchestra.run.smithi046.stdout: "crush_compat_step": { 2024-03-31T23:37:04.941 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "crush_compat_step", 2024-03-31T23:37:04.941 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "float", 2024-03-31T23:37:04.941 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:04.941 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:04.941 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "0.5", 2024-03-31T23:37:04.941 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "0.001", 2024-03-31T23:37:04.941 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "0.999", 2024-03-31T23:37:04.941 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:04.941 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "aggressiveness of optimization", 2024-03-31T23:37:04.941 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-03-31T23:37:04.941 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:04.941 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:04.941 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:04.941 INFO:teuthology.orchestra.run.smithi046.stdout: "end_time": { 2024-03-31T23:37:04.941 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "end_time", 2024-03-31T23:37:04.942 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:04.942 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:04.942 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:04.942 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "2359", 2024-03-31T23:37:04.942 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:04.942 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:04.942 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:04.942 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "ending time of day to automatically balance", 2024-03-31T23:37:04.942 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-31T23:37:04.942 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:04.942 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:04.942 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:04.942 INFO:teuthology.orchestra.run.smithi046.stdout: "end_weekday": { 2024-03-31T23:37:04.942 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "end_weekday", 2024-03-31T23:37:04.942 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "uint", 2024-03-31T23:37:04.943 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:04.943 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:04.943 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "0", 2024-03-31T23:37:04.943 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "0", 2024-03-31T23:37:04.943 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "6", 2024-03-31T23:37:04.943 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:04.943 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-03-31T23:37:04.943 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-31T23:37:04.943 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:04.943 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:04.943 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:04.944 INFO:teuthology.orchestra.run.smithi046.stdout: "log_level": { 2024-03-31T23:37:04.944 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_level", 2024-03-31T23:37:04.944 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:04.944 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:04.944 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:04.944 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-31T23:37:04.944 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:04.944 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:04.944 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-31T23:37:04.944 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-31T23:37:04.944 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-31T23:37:04.944 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-31T23:37:04.944 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-31T23:37:04.944 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-31T23:37:04.944 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-31T23:37:04.945 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-31T23:37:04.945 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:04.945 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:04.945 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:04.945 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:04.945 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:04.945 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster": { 2024-03-31T23:37:04.945 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster", 2024-03-31T23:37:04.945 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:04.945 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:04.945 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:04.945 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-31T23:37:04.945 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:04.946 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:04.946 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:04.946 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:04.946 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:04.946 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:04.946 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:04.946 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:04.946 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster_level": { 2024-03-31T23:37:04.946 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster_level", 2024-03-31T23:37:04.946 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:04.946 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:04.946 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:04.946 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "info", 2024-03-31T23:37:04.946 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:04.947 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:04.947 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-31T23:37:04.947 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-31T23:37:04.947 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-31T23:37:04.947 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-31T23:37:04.947 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-31T23:37:04.947 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-31T23:37:04.947 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-31T23:37:04.947 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-31T23:37:04.947 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:04.947 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:04.947 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:04.947 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:04.947 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:04.947 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_file": { 2024-03-31T23:37:04.948 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_file", 2024-03-31T23:37:04.948 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:04.948 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:04.948 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:04.948 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-31T23:37:04.948 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:04.948 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:04.948 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:04.948 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:04.948 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:04.948 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:04.948 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:04.948 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:04.948 INFO:teuthology.orchestra.run.smithi046.stdout: "min_score": { 2024-03-31T23:37:04.949 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "min_score", 2024-03-31T23:37:04.949 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "float", 2024-03-31T23:37:04.949 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:04.949 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:04.949 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "0", 2024-03-31T23:37:04.949 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:04.949 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:04.949 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:04.949 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-03-31T23:37:04.949 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:04.949 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:04.949 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:04.949 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:04.949 INFO:teuthology.orchestra.run.smithi046.stdout: "mode": { 2024-03-31T23:37:04.950 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "mode", 2024-03-31T23:37:04.950 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:04.950 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:04.950 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:04.950 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "upmap", 2024-03-31T23:37:04.950 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:04.950 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:04.950 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-31T23:37:04.950 INFO:teuthology.orchestra.run.smithi046.stdout: "crush-compat", 2024-03-31T23:37:04.950 INFO:teuthology.orchestra.run.smithi046.stdout: "none", 2024-03-31T23:37:04.950 INFO:teuthology.orchestra.run.smithi046.stdout: "read", 2024-03-31T23:37:04.950 INFO:teuthology.orchestra.run.smithi046.stdout: "upmap", 2024-03-31T23:37:04.950 INFO:teuthology.orchestra.run.smithi046.stdout: "upmap-read" 2024-03-31T23:37:04.950 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-31T23:37:04.951 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Balancer mode", 2024-03-31T23:37:04.951 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:04.951 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:04.951 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:04.951 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:04.951 INFO:teuthology.orchestra.run.smithi046.stdout: "pool_ids": { 2024-03-31T23:37:04.951 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "pool_ids", 2024-03-31T23:37:04.951 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:04.951 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:04.951 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:04.951 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-31T23:37:04.951 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:04.951 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:04.951 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:04.952 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-03-31T23:37:04.952 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:04.952 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:04.952 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:04.952 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:04.952 INFO:teuthology.orchestra.run.smithi046.stdout: "sleep_interval": { 2024-03-31T23:37:04.952 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "sleep_interval", 2024-03-31T23:37:04.952 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "secs", 2024-03-31T23:37:04.952 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:04.952 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:04.952 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "60", 2024-03-31T23:37:04.952 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:04.952 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:04.952 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:04.952 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-03-31T23:37:04.953 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:04.953 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:04.953 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:04.953 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:04.953 INFO:teuthology.orchestra.run.smithi046.stdout: "upmap_max_deviation": { 2024-03-31T23:37:04.953 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "upmap_max_deviation", 2024-03-31T23:37:04.953 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "int", 2024-03-31T23:37:04.953 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:04.953 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:04.953 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "5", 2024-03-31T23:37:04.953 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "1", 2024-03-31T23:37:04.953 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:04.953 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:04.953 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "deviation below which no optimization is attempted", 2024-03-31T23:37:04.954 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-03-31T23:37:04.954 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:04.954 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:04.954 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:04.954 INFO:teuthology.orchestra.run.smithi046.stdout: "upmap_max_optimizations": { 2024-03-31T23:37:04.954 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "upmap_max_optimizations", 2024-03-31T23:37:04.954 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "uint", 2024-03-31T23:37:04.954 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:04.954 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:04.954 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "10", 2024-03-31T23:37:04.954 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:04.954 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:04.954 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:04.954 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-03-31T23:37:04.955 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:04.955 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:04.955 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:04.955 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-31T23:37:04.955 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-31T23:37:04.955 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:04.955 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-03-31T23:37:04.955 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "cephadm", 2024-03-31T23:37:04.955 INFO:teuthology.orchestra.run.smithi046.stdout: "can_run": true, 2024-03-31T23:37:04.955 INFO:teuthology.orchestra.run.smithi046.stdout: "error_string": "", 2024-03-31T23:37:04.955 INFO:teuthology.orchestra.run.smithi046.stdout: "module_options": { 2024-03-31T23:37:04.955 INFO:teuthology.orchestra.run.smithi046.stdout: "agent_down_multiplier": { 2024-03-31T23:37:04.955 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "agent_down_multiplier", 2024-03-31T23:37:04.955 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "float", 2024-03-31T23:37:04.955 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:04.956 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:04.956 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "3.0", 2024-03-31T23:37:04.956 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:04.956 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:04.956 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:04.956 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-03-31T23:37:04.956 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:04.956 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:04.956 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:04.956 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:04.956 INFO:teuthology.orchestra.run.smithi046.stdout: "agent_refresh_rate": { 2024-03-31T23:37:04.956 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "agent_refresh_rate", 2024-03-31T23:37:04.956 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "secs", 2024-03-31T23:37:04.956 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:04.957 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:04.957 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "20", 2024-03-31T23:37:04.957 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:04.957 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:04.957 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:04.957 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-03-31T23:37:04.957 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:04.957 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:04.957 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:04.957 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:04.957 INFO:teuthology.orchestra.run.smithi046.stdout: "agent_starting_port": { 2024-03-31T23:37:04.957 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "agent_starting_port", 2024-03-31T23:37:04.957 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "int", 2024-03-31T23:37:04.957 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:04.957 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:04.958 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "4721", 2024-03-31T23:37:04.958 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:04.958 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:04.958 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:04.958 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-03-31T23:37:04.958 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:04.958 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:04.958 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:04.958 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:04.958 INFO:teuthology.orchestra.run.smithi046.stdout: "allow_ptrace": { 2024-03-31T23:37:04.958 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "allow_ptrace", 2024-03-31T23:37:04.958 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:04.958 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:04.959 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:04.959 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-31T23:37:04.959 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:04.959 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:04.959 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:04.959 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-03-31T23:37:04.959 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-03-31T23:37:04.959 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:04.959 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:04.959 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:04.959 INFO:teuthology.orchestra.run.smithi046.stdout: "autotune_interval": { 2024-03-31T23:37:04.959 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "autotune_interval", 2024-03-31T23:37:04.959 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "secs", 2024-03-31T23:37:04.959 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:04.959 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:04.960 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "600", 2024-03-31T23:37:04.960 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:04.960 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:04.960 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:04.960 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "how frequently to autotune daemon memory", 2024-03-31T23:37:04.960 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:04.960 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:04.960 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:04.960 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:04.960 INFO:teuthology.orchestra.run.smithi046.stdout: "autotune_memory_target_ratio": { 2024-03-31T23:37:04.960 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "autotune_memory_target_ratio", 2024-03-31T23:37:04.960 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "float", 2024-03-31T23:37:04.960 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:04.960 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:04.961 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "0.7", 2024-03-31T23:37:04.961 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:04.961 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:04.961 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:04.961 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-03-31T23:37:04.961 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:04.961 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:04.961 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:04.961 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:04.961 INFO:teuthology.orchestra.run.smithi046.stdout: "cephadm_log_destination": { 2024-03-31T23:37:04.961 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "cephadm_log_destination", 2024-03-31T23:37:04.961 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:04.961 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:04.962 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:04.962 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-31T23:37:04.962 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:04.962 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:04.962 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-31T23:37:04.962 INFO:teuthology.orchestra.run.smithi046.stdout: "file", 2024-03-31T23:37:04.962 INFO:teuthology.orchestra.run.smithi046.stdout: "file,syslog", 2024-03-31T23:37:04.962 INFO:teuthology.orchestra.run.smithi046.stdout: "syslog" 2024-03-31T23:37:04.962 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-31T23:37:04.962 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-03-31T23:37:04.962 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:04.962 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:04.962 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:04.962 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:04.963 INFO:teuthology.orchestra.run.smithi046.stdout: "cgroups_split": { 2024-03-31T23:37:04.963 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "cgroups_split", 2024-03-31T23:37:04.963 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:04.963 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:04.963 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:04.963 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "True", 2024-03-31T23:37:04.963 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:04.963 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:04.963 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:04.963 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-03-31T23:37:04.963 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:04.963 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:04.963 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:04.963 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:04.964 INFO:teuthology.orchestra.run.smithi046.stdout: "config_checks_enabled": { 2024-03-31T23:37:04.964 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "config_checks_enabled", 2024-03-31T23:37:04.964 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:04.964 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:04.964 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:04.964 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-31T23:37:04.964 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:04.964 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:04.964 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:04.964 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-03-31T23:37:04.964 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:04.964 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:04.964 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:04.964 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:04.965 INFO:teuthology.orchestra.run.smithi046.stdout: "config_dashboard": { 2024-03-31T23:37:04.965 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "config_dashboard", 2024-03-31T23:37:04.965 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:04.965 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:04.965 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:04.965 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "True", 2024-03-31T23:37:04.965 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:04.965 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:04.965 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:04.965 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-03-31T23:37:04.965 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:04.965 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:04.965 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:04.965 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:04.965 INFO:teuthology.orchestra.run.smithi046.stdout: "container_image_alertmanager": { 2024-03-31T23:37:04.966 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "container_image_alertmanager", 2024-03-31T23:37:04.966 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:04.966 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:04.966 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:04.966 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-03-31T23:37:04.966 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:04.966 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:04.966 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:04.966 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Prometheus container image", 2024-03-31T23:37:04.966 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:04.966 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:04.966 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:04.966 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:04.967 INFO:teuthology.orchestra.run.smithi046.stdout: "container_image_base": { 2024-03-31T23:37:04.967 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "container_image_base", 2024-03-31T23:37:04.967 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:04.967 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:04.967 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:04.967 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "quay.io/ceph/ceph", 2024-03-31T23:37:04.967 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:04.967 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:04.967 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:04.967 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Container image name, without the tag", 2024-03-31T23:37:04.967 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:04.967 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:04.967 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:04.967 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:04.968 INFO:teuthology.orchestra.run.smithi046.stdout: "container_image_elasticsearch": { 2024-03-31T23:37:04.968 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "container_image_elasticsearch", 2024-03-31T23:37:04.968 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:04.968 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:04.968 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:04.968 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-03-31T23:37:04.968 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:04.968 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:04.968 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:04.968 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "elasticsearch container image", 2024-03-31T23:37:04.968 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:04.968 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:04.968 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:04.968 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:04.969 INFO:teuthology.orchestra.run.smithi046.stdout: "container_image_grafana": { 2024-03-31T23:37:04.969 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "container_image_grafana", 2024-03-31T23:37:04.969 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:04.969 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:04.969 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:04.969 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "quay.io/ceph/grafana:9.4.12", 2024-03-31T23:37:04.969 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:04.969 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:04.969 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:04.969 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Prometheus container image", 2024-03-31T23:37:04.969 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:04.969 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:04.969 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:04.969 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:04.969 INFO:teuthology.orchestra.run.smithi046.stdout: "container_image_haproxy": { 2024-03-31T23:37:04.970 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "container_image_haproxy", 2024-03-31T23:37:04.970 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:04.970 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:04.970 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:04.970 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-03-31T23:37:04.970 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:04.970 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:04.970 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:04.970 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "HAproxy container image", 2024-03-31T23:37:04.970 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:04.970 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:04.970 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:04.970 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:04.971 INFO:teuthology.orchestra.run.smithi046.stdout: "container_image_jaeger_agent": { 2024-03-31T23:37:04.971 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "container_image_jaeger_agent", 2024-03-31T23:37:04.971 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:04.971 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:04.971 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:04.971 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-03-31T23:37:04.971 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:04.971 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:04.971 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:04.971 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Jaeger agent container image", 2024-03-31T23:37:04.971 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:04.971 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:04.971 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:04.971 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:04.972 INFO:teuthology.orchestra.run.smithi046.stdout: "container_image_jaeger_collector": { 2024-03-31T23:37:04.972 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "container_image_jaeger_collector", 2024-03-31T23:37:04.972 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:04.972 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:04.972 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:04.972 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-03-31T23:37:04.972 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:04.972 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:04.972 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:04.972 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Jaeger collector container image", 2024-03-31T23:37:04.972 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:04.972 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:04.972 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:04.972 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:04.973 INFO:teuthology.orchestra.run.smithi046.stdout: "container_image_jaeger_query": { 2024-03-31T23:37:04.973 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "container_image_jaeger_query", 2024-03-31T23:37:04.973 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:04.973 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:04.973 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:04.973 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-03-31T23:37:04.973 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:04.973 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:04.973 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:04.973 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Jaeger query container image", 2024-03-31T23:37:04.973 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:04.973 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:04.973 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:04.973 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:04.973 INFO:teuthology.orchestra.run.smithi046.stdout: "container_image_keepalived": { 2024-03-31T23:37:04.974 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "container_image_keepalived", 2024-03-31T23:37:04.974 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:04.974 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:04.974 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:04.974 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-03-31T23:37:04.974 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:04.974 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:04.974 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:04.974 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Keepalived container image", 2024-03-31T23:37:04.974 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:04.974 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:04.974 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:04.974 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:04.974 INFO:teuthology.orchestra.run.smithi046.stdout: "container_image_loki": { 2024-03-31T23:37:04.975 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "container_image_loki", 2024-03-31T23:37:04.975 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:04.975 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:04.975 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:04.975 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-03-31T23:37:04.975 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:04.975 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:04.975 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:04.975 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Loki container image", 2024-03-31T23:37:04.975 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:04.975 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:04.975 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:04.975 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:04.975 INFO:teuthology.orchestra.run.smithi046.stdout: "container_image_node_exporter": { 2024-03-31T23:37:04.976 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "container_image_node_exporter", 2024-03-31T23:37:04.976 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:04.976 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:04.976 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:04.976 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-03-31T23:37:04.976 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:04.976 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:04.976 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:04.976 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Prometheus container image", 2024-03-31T23:37:04.976 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:04.976 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:04.976 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:04.976 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:04.976 INFO:teuthology.orchestra.run.smithi046.stdout: "container_image_nvmeof": { 2024-03-31T23:37:04.976 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "container_image_nvmeof", 2024-03-31T23:37:04.977 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:04.977 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:04.977 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:04.977 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "quay.io/ceph/nvmeof:latest", 2024-03-31T23:37:04.977 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:04.977 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:04.977 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:04.977 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Nvme-of container image", 2024-03-31T23:37:04.977 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:04.977 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:04.977 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:04.977 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:04.977 INFO:teuthology.orchestra.run.smithi046.stdout: "container_image_prometheus": { 2024-03-31T23:37:04.977 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "container_image_prometheus", 2024-03-31T23:37:04.977 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:04.978 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:04.978 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:04.978 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-03-31T23:37:04.978 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:04.978 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:04.978 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:04.978 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Prometheus container image", 2024-03-31T23:37:04.978 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:04.978 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:04.978 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:04.978 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:04.978 INFO:teuthology.orchestra.run.smithi046.stdout: "container_image_promtail": { 2024-03-31T23:37:04.978 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "container_image_promtail", 2024-03-31T23:37:04.978 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:04.979 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:04.979 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:04.979 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-03-31T23:37:04.979 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:04.979 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:04.979 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:04.979 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Promtail container image", 2024-03-31T23:37:04.979 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:04.979 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:04.979 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:04.979 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:04.979 INFO:teuthology.orchestra.run.smithi046.stdout: "container_image_snmp_gateway": { 2024-03-31T23:37:04.979 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "container_image_snmp_gateway", 2024-03-31T23:37:04.979 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:04.979 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:04.980 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:04.980 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-03-31T23:37:04.980 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:04.980 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:04.980 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:04.980 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "SNMP Gateway container image", 2024-03-31T23:37:04.980 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:04.980 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:04.980 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:04.980 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:04.980 INFO:teuthology.orchestra.run.smithi046.stdout: "container_init": { 2024-03-31T23:37:04.980 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "container_init", 2024-03-31T23:37:04.980 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:04.980 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:04.980 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:04.981 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "True", 2024-03-31T23:37:04.981 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:04.981 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:04.981 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:04.981 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Run podman/docker with `--init`", 2024-03-31T23:37:04.981 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:04.981 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:04.981 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:04.981 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:04.981 INFO:teuthology.orchestra.run.smithi046.stdout: "daemon_cache_timeout": { 2024-03-31T23:37:04.981 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "daemon_cache_timeout", 2024-03-31T23:37:04.981 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "secs", 2024-03-31T23:37:04.981 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:04.981 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:04.981 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "600", 2024-03-31T23:37:04.982 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:04.982 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:04.982 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:04.982 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-03-31T23:37:04.982 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:04.982 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:04.982 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:04.982 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:04.982 INFO:teuthology.orchestra.run.smithi046.stdout: "default_cephadm_command_timeout": { 2024-03-31T23:37:04.982 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "default_cephadm_command_timeout", 2024-03-31T23:37:04.982 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "int", 2024-03-31T23:37:04.982 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:04.982 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:04.983 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "900", 2024-03-31T23:37:04.983 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:04.983 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:04.983 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:04.983 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-03-31T23:37:04.983 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:04.983 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:04.983 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:04.983 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:04.983 INFO:teuthology.orchestra.run.smithi046.stdout: "default_registry": { 2024-03-31T23:37:04.983 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "default_registry", 2024-03-31T23:37:04.983 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:04.983 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:04.983 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:04.983 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "docker.io", 2024-03-31T23:37:04.984 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:04.984 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:04.984 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:04.984 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-03-31T23:37:04.984 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:04.984 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:04.984 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:04.984 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:04.984 INFO:teuthology.orchestra.run.smithi046.stdout: "device_cache_timeout": { 2024-03-31T23:37:04.984 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "device_cache_timeout", 2024-03-31T23:37:04.984 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "secs", 2024-03-31T23:37:04.984 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:04.984 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:04.984 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "1800", 2024-03-31T23:37:04.984 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:04.985 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:04.985 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:04.985 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "seconds to cache device inventory", 2024-03-31T23:37:04.985 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:04.985 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:04.985 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:04.985 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:04.985 INFO:teuthology.orchestra.run.smithi046.stdout: "device_enhanced_scan": { 2024-03-31T23:37:04.985 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "device_enhanced_scan", 2024-03-31T23:37:04.985 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:04.985 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:04.985 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:04.985 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-31T23:37:04.985 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:04.985 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:04.986 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:04.986 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Use libstoragemgmt during device scans", 2024-03-31T23:37:04.986 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:04.986 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:04.986 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:04.986 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:04.986 INFO:teuthology.orchestra.run.smithi046.stdout: "facts_cache_timeout": { 2024-03-31T23:37:04.986 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "facts_cache_timeout", 2024-03-31T23:37:04.986 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "secs", 2024-03-31T23:37:04.986 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:04.986 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:04.986 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "60", 2024-03-31T23:37:04.986 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:04.987 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:04.987 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:04.987 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "seconds to cache host facts data", 2024-03-31T23:37:04.987 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:04.987 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:04.987 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:04.987 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:04.987 INFO:teuthology.orchestra.run.smithi046.stdout: "grafana_dashboards_path": { 2024-03-31T23:37:04.987 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "grafana_dashboards_path", 2024-03-31T23:37:04.987 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:04.987 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:04.987 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:04.987 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "/etc/grafana/dashboards/ceph-dashboard/", 2024-03-31T23:37:04.987 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:04.987 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:04.988 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:04.988 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "location of dashboards to include in grafana deployments", 2024-03-31T23:37:04.988 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:04.988 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:04.988 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:04.988 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:04.988 INFO:teuthology.orchestra.run.smithi046.stdout: "host_check_interval": { 2024-03-31T23:37:04.988 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "host_check_interval", 2024-03-31T23:37:04.988 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "secs", 2024-03-31T23:37:04.988 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:04.988 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:04.988 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "600", 2024-03-31T23:37:04.988 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:04.988 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:04.988 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:04.989 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "how frequently to perform a host check", 2024-03-31T23:37:04.989 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:04.989 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:04.989 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:04.989 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:04.989 INFO:teuthology.orchestra.run.smithi046.stdout: "hw_monitoring": { 2024-03-31T23:37:04.989 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "hw_monitoring", 2024-03-31T23:37:04.989 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:04.989 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:04.989 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:04.989 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-31T23:37:04.989 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:04.989 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:04.989 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:04.989 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-03-31T23:37:04.990 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:04.990 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:04.990 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:04.990 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:04.990 INFO:teuthology.orchestra.run.smithi046.stdout: "inventory_list_all": { 2024-03-31T23:37:04.990 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "inventory_list_all", 2024-03-31T23:37:04.990 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:04.990 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:04.990 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:04.990 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-31T23:37:04.990 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:04.990 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:04.990 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:04.991 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-03-31T23:37:04.991 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:04.991 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:04.991 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:04.991 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:04.991 INFO:teuthology.orchestra.run.smithi046.stdout: "log_level": { 2024-03-31T23:37:04.991 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_level", 2024-03-31T23:37:04.991 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:04.991 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:04.991 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:04.991 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-31T23:37:04.991 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:04.991 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:04.991 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-31T23:37:04.991 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-31T23:37:04.992 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-31T23:37:04.992 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-31T23:37:04.992 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-31T23:37:04.992 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-31T23:37:04.992 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-31T23:37:04.992 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-31T23:37:04.992 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:04.992 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:04.992 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:04.992 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:04.992 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:04.992 INFO:teuthology.orchestra.run.smithi046.stdout: "log_refresh_metadata": { 2024-03-31T23:37:04.992 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_refresh_metadata", 2024-03-31T23:37:04.992 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:04.993 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:04.993 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:04.993 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-31T23:37:04.993 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:04.993 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:04.993 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:04.993 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-03-31T23:37:04.993 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:04.993 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:04.993 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:04.993 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:04.993 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster": { 2024-03-31T23:37:04.993 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster", 2024-03-31T23:37:04.993 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:04.993 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:04.994 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:04.994 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "True", 2024-03-31T23:37:04.994 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:04.994 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:04.994 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:04.994 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-03-31T23:37:04.994 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:04.994 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:04.994 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:04.994 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:04.994 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster_level": { 2024-03-31T23:37:04.994 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster_level", 2024-03-31T23:37:04.994 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:04.994 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:04.995 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:04.995 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "info", 2024-03-31T23:37:04.995 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:04.995 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:04.995 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-31T23:37:04.995 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-31T23:37:04.995 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-31T23:37:04.995 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-31T23:37:04.995 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-31T23:37:04.995 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-31T23:37:04.995 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-31T23:37:04.995 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-31T23:37:04.995 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:04.995 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:04.995 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:04.996 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:04.996 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:04.996 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_file": { 2024-03-31T23:37:04.996 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_file", 2024-03-31T23:37:04.996 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:04.996 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:04.996 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:04.996 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-31T23:37:04.996 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:04.996 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:04.996 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:04.996 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:04.996 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:04.996 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:04.996 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:04.997 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:04.997 INFO:teuthology.orchestra.run.smithi046.stdout: "manage_etc_ceph_ceph_conf": { 2024-03-31T23:37:04.997 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-03-31T23:37:04.997 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:04.997 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:04.997 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:04.997 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-31T23:37:04.997 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:04.997 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:04.997 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:04.997 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-03-31T23:37:04.997 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:04.997 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:04.997 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:04.997 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:04.998 INFO:teuthology.orchestra.run.smithi046.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-03-31T23:37:04.998 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-03-31T23:37:04.998 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:04.998 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:04.998 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:04.998 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "*", 2024-03-31T23:37:04.998 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:04.998 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:04.998 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:04.998 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-03-31T23:37:04.998 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:04.998 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:04.998 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:04.998 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:04.999 INFO:teuthology.orchestra.run.smithi046.stdout: "max_count_per_host": { 2024-03-31T23:37:04.999 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "max_count_per_host", 2024-03-31T23:37:04.999 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "int", 2024-03-31T23:37:04.999 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:04.999 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:04.999 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "10", 2024-03-31T23:37:04.999 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:04.999 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:04.999 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:04.999 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "max number of daemons per service per host", 2024-03-31T23:37:04.999 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:04.999 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:04.999 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:04.999 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:04.999 INFO:teuthology.orchestra.run.smithi046.stdout: "max_osd_draining_count": { 2024-03-31T23:37:05.000 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "max_osd_draining_count", 2024-03-31T23:37:05.000 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "int", 2024-03-31T23:37:05.000 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.000 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.000 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "10", 2024-03-31T23:37:05.000 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.000 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.000 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.000 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-03-31T23:37:05.000 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.000 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.000 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.000 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.000 INFO:teuthology.orchestra.run.smithi046.stdout: "migration_current": { 2024-03-31T23:37:05.000 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "migration_current", 2024-03-31T23:37:05.001 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "int", 2024-03-31T23:37:05.001 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.001 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.001 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-31T23:37:05.001 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.001 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.001 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.001 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "internal - do not modify", 2024-03-31T23:37:05.001 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.001 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.001 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.001 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.001 INFO:teuthology.orchestra.run.smithi046.stdout: "mode": { 2024-03-31T23:37:05.001 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "mode", 2024-03-31T23:37:05.001 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.002 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.002 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.002 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "root", 2024-03-31T23:37:05.002 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.002 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.002 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-31T23:37:05.002 INFO:teuthology.orchestra.run.smithi046.stdout: "cephadm-package", 2024-03-31T23:37:05.002 INFO:teuthology.orchestra.run.smithi046.stdout: "root" 2024-03-31T23:37:05.002 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-31T23:37:05.002 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "mode for remote execution of cephadm", 2024-03-31T23:37:05.002 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.002 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.002 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.002 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.002 INFO:teuthology.orchestra.run.smithi046.stdout: "oob_default_addr": { 2024-03-31T23:37:05.003 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "oob_default_addr", 2024-03-31T23:37:05.003 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.003 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.003 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.003 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "169.254.1.1", 2024-03-31T23:37:05.003 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.003 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.003 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.003 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Default address for RedFish API (oob management).", 2024-03-31T23:37:05.003 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.003 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.003 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.003 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.003 INFO:teuthology.orchestra.run.smithi046.stdout: "prometheus_alerts_path": { 2024-03-31T23:37:05.004 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "prometheus_alerts_path", 2024-03-31T23:37:05.004 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.004 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.004 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.004 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-03-31T23:37:05.004 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.004 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.004 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.004 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-03-31T23:37:05.004 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.004 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.004 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.004 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.004 INFO:teuthology.orchestra.run.smithi046.stdout: "registry_insecure": { 2024-03-31T23:37:05.004 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "registry_insecure", 2024-03-31T23:37:05.005 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.005 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.005 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.005 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-31T23:37:05.005 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.005 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.005 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.005 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-03-31T23:37:05.005 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.005 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.005 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.005 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.005 INFO:teuthology.orchestra.run.smithi046.stdout: "registry_password": { 2024-03-31T23:37:05.005 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "registry_password", 2024-03-31T23:37:05.006 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.006 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.006 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.006 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-31T23:37:05.006 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.006 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.006 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.006 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-03-31T23:37:05.006 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.006 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.006 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.006 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.007 INFO:teuthology.orchestra.run.smithi046.stdout: "registry_url": { 2024-03-31T23:37:05.007 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "registry_url", 2024-03-31T23:37:05.007 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.007 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.007 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.007 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-31T23:37:05.007 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.007 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.007 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.007 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-03-31T23:37:05.007 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.007 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.007 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.007 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.008 INFO:teuthology.orchestra.run.smithi046.stdout: "registry_username": { 2024-03-31T23:37:05.008 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "registry_username", 2024-03-31T23:37:05.008 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.008 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.008 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.008 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-31T23:37:05.008 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.008 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.008 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.008 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-03-31T23:37:05.008 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.008 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.008 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.008 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.008 INFO:teuthology.orchestra.run.smithi046.stdout: "secure_monitoring_stack": { 2024-03-31T23:37:05.009 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "secure_monitoring_stack", 2024-03-31T23:37:05.009 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.009 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.009 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.009 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-31T23:37:05.009 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.009 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.009 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.009 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-03-31T23:37:05.009 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.009 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.009 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.009 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.009 INFO:teuthology.orchestra.run.smithi046.stdout: "service_discovery_port": { 2024-03-31T23:37:05.009 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "service_discovery_port", 2024-03-31T23:37:05.010 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "int", 2024-03-31T23:37:05.010 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.010 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.010 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "8765", 2024-03-31T23:37:05.010 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.010 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.010 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.010 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "cephadm service discovery port", 2024-03-31T23:37:05.010 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.010 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.010 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.010 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.010 INFO:teuthology.orchestra.run.smithi046.stdout: "ssh_config_file": { 2024-03-31T23:37:05.010 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "ssh_config_file", 2024-03-31T23:37:05.011 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.011 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.011 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.011 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-31T23:37:05.011 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.011 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.011 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.011 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-03-31T23:37:05.011 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.011 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.011 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.011 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.011 INFO:teuthology.orchestra.run.smithi046.stdout: "use_agent": { 2024-03-31T23:37:05.011 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "use_agent", 2024-03-31T23:37:05.012 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.012 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.012 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.012 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-31T23:37:05.012 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.012 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.012 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.012 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-03-31T23:37:05.012 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.012 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.012 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.012 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.012 INFO:teuthology.orchestra.run.smithi046.stdout: "use_repo_digest": { 2024-03-31T23:37:05.012 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "use_repo_digest", 2024-03-31T23:37:05.012 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.013 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.013 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.013 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "True", 2024-03-31T23:37:05.013 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.013 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.013 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.013 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-03-31T23:37:05.013 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.013 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.013 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.013 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.013 INFO:teuthology.orchestra.run.smithi046.stdout: "warn_on_failed_host_check": { 2024-03-31T23:37:05.013 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "warn_on_failed_host_check", 2024-03-31T23:37:05.013 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.014 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.014 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.014 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "True", 2024-03-31T23:37:05.014 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.014 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.014 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.014 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "raise a health warning if the host check fails", 2024-03-31T23:37:05.014 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.014 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.014 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.014 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.014 INFO:teuthology.orchestra.run.smithi046.stdout: "warn_on_stray_daemons": { 2024-03-31T23:37:05.014 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "warn_on_stray_daemons", 2024-03-31T23:37:05.014 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.014 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.015 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.015 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "True", 2024-03-31T23:37:05.015 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.015 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.015 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.015 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-03-31T23:37:05.015 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.015 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.015 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.015 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.015 INFO:teuthology.orchestra.run.smithi046.stdout: "warn_on_stray_hosts": { 2024-03-31T23:37:05.015 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "warn_on_stray_hosts", 2024-03-31T23:37:05.015 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.015 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.018 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.018 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "True", 2024-03-31T23:37:05.018 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.018 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.018 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.018 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-03-31T23:37:05.018 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.018 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.018 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.018 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-31T23:37:05.019 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-31T23:37:05.019 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.019 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-03-31T23:37:05.019 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "crash", 2024-03-31T23:37:05.019 INFO:teuthology.orchestra.run.smithi046.stdout: "can_run": true, 2024-03-31T23:37:05.019 INFO:teuthology.orchestra.run.smithi046.stdout: "error_string": "", 2024-03-31T23:37:05.019 INFO:teuthology.orchestra.run.smithi046.stdout: "module_options": { 2024-03-31T23:37:05.019 INFO:teuthology.orchestra.run.smithi046.stdout: "log_level": { 2024-03-31T23:37:05.019 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_level", 2024-03-31T23:37:05.019 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.019 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.019 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.019 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-31T23:37:05.019 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.019 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.019 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-31T23:37:05.020 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-31T23:37:05.020 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-31T23:37:05.020 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-31T23:37:05.020 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-31T23:37:05.020 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-31T23:37:05.020 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-31T23:37:05.020 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-31T23:37:05.020 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.020 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.020 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.020 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.020 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.020 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster": { 2024-03-31T23:37:05.020 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster", 2024-03-31T23:37:05.021 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.021 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.021 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.021 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-31T23:37:05.021 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.021 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.021 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.021 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.021 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.021 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.021 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.021 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.021 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster_level": { 2024-03-31T23:37:05.021 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster_level", 2024-03-31T23:37:05.022 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.022 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.022 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.022 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "info", 2024-03-31T23:37:05.022 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.022 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.022 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-31T23:37:05.022 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-31T23:37:05.022 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-31T23:37:05.022 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-31T23:37:05.022 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-31T23:37:05.022 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-31T23:37:05.022 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-31T23:37:05.022 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-31T23:37:05.023 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.023 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.023 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.023 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.023 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.023 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_file": { 2024-03-31T23:37:05.023 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_file", 2024-03-31T23:37:05.023 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.023 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.023 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.023 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-31T23:37:05.023 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.023 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.023 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.023 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.024 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.024 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.024 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.024 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.024 INFO:teuthology.orchestra.run.smithi046.stdout: "retain_interval": { 2024-03-31T23:37:05.024 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "retain_interval", 2024-03-31T23:37:05.024 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "secs", 2024-03-31T23:37:05.024 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.024 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.024 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "31536000", 2024-03-31T23:37:05.024 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.024 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.024 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.024 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "how long to retain crashes before pruning them", 2024-03-31T23:37:05.025 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.025 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.025 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.025 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.025 INFO:teuthology.orchestra.run.smithi046.stdout: "warn_recent_interval": { 2024-03-31T23:37:05.025 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "warn_recent_interval", 2024-03-31T23:37:05.025 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "secs", 2024-03-31T23:37:05.025 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.025 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.025 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "1209600", 2024-03-31T23:37:05.025 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.025 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.025 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.025 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "time interval in which to warn about recent crashes", 2024-03-31T23:37:05.025 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.026 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.026 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.026 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-31T23:37:05.026 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-31T23:37:05.026 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.026 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-03-31T23:37:05.026 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "dashboard", 2024-03-31T23:37:05.026 INFO:teuthology.orchestra.run.smithi046.stdout: "can_run": true, 2024-03-31T23:37:05.026 INFO:teuthology.orchestra.run.smithi046.stdout: "error_string": "", 2024-03-31T23:37:05.026 INFO:teuthology.orchestra.run.smithi046.stdout: "module_options": { 2024-03-31T23:37:05.026 INFO:teuthology.orchestra.run.smithi046.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-03-31T23:37:05.026 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-03-31T23:37:05.026 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "int", 2024-03-31T23:37:05.026 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.027 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.027 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "10", 2024-03-31T23:37:05.027 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.027 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.027 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.027 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.027 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.027 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.027 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.027 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.027 INFO:teuthology.orchestra.run.smithi046.stdout: "ALERTMANAGER_API_HOST": { 2024-03-31T23:37:05.027 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "ALERTMANAGER_API_HOST", 2024-03-31T23:37:05.027 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.027 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.028 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.028 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-31T23:37:05.028 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.028 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.028 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.028 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.028 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.028 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.028 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.028 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.028 INFO:teuthology.orchestra.run.smithi046.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-03-31T23:37:05.028 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-03-31T23:37:05.028 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.028 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.028 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.029 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "True", 2024-03-31T23:37:05.029 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.029 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.029 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.029 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.029 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.029 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.029 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.029 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.029 INFO:teuthology.orchestra.run.smithi046.stdout: "AUDIT_API_ENABLED": { 2024-03-31T23:37:05.029 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "AUDIT_API_ENABLED", 2024-03-31T23:37:05.029 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.029 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.029 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.029 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-31T23:37:05.029 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.030 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.030 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.030 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.030 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.030 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.030 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.030 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.030 INFO:teuthology.orchestra.run.smithi046.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-03-31T23:37:05.030 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-03-31T23:37:05.030 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.030 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.030 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.030 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "True", 2024-03-31T23:37:05.030 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.031 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.031 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.031 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.031 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.031 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.031 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.031 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.031 INFO:teuthology.orchestra.run.smithi046.stdout: "ENABLE_BROWSABLE_API": { 2024-03-31T23:37:05.031 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "ENABLE_BROWSABLE_API", 2024-03-31T23:37:05.031 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.031 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.031 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.031 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "True", 2024-03-31T23:37:05.031 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.032 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.032 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.032 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.032 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.032 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.032 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.032 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.032 INFO:teuthology.orchestra.run.smithi046.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-03-31T23:37:05.032 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-03-31T23:37:05.032 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.032 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.032 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.032 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "True", 2024-03-31T23:37:05.032 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.033 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.033 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.033 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.033 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.033 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.033 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.033 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.033 INFO:teuthology.orchestra.run.smithi046.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-03-31T23:37:05.033 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-03-31T23:37:05.033 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.033 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.033 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.033 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "True", 2024-03-31T23:37:05.034 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.034 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.034 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.034 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.034 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.034 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.034 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.034 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.034 INFO:teuthology.orchestra.run.smithi046.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-03-31T23:37:05.034 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-03-31T23:37:05.034 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.034 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.034 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.034 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "True", 2024-03-31T23:37:05.034 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.035 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.035 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.035 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.035 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.035 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.035 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.035 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.035 INFO:teuthology.orchestra.run.smithi046.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-03-31T23:37:05.035 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-03-31T23:37:05.035 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.035 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.035 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.035 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "True", 2024-03-31T23:37:05.035 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.036 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.036 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.036 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.036 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.036 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.036 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.036 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.036 INFO:teuthology.orchestra.run.smithi046.stdout: "FEATURE_TOGGLE_NFS": { 2024-03-31T23:37:05.036 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-03-31T23:37:05.036 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.036 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.036 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.036 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "True", 2024-03-31T23:37:05.036 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.036 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.037 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.037 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.037 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.037 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.037 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.037 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.037 INFO:teuthology.orchestra.run.smithi046.stdout: "FEATURE_TOGGLE_RBD": { 2024-03-31T23:37:05.037 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-03-31T23:37:05.037 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.037 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.037 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.037 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "True", 2024-03-31T23:37:05.037 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.037 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.038 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.038 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.038 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.038 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.038 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.038 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.038 INFO:teuthology.orchestra.run.smithi046.stdout: "FEATURE_TOGGLE_RGW": { 2024-03-31T23:37:05.038 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-03-31T23:37:05.038 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.038 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.038 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.038 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "True", 2024-03-31T23:37:05.038 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.038 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.039 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.039 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.039 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.039 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.039 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.039 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.039 INFO:teuthology.orchestra.run.smithi046.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-03-31T23:37:05.039 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-03-31T23:37:05.039 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.039 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.039 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.039 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-31T23:37:05.039 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.039 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.039 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.040 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.040 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.040 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.040 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.040 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.040 INFO:teuthology.orchestra.run.smithi046.stdout: "GRAFANA_API_PASSWORD": { 2024-03-31T23:37:05.040 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "GRAFANA_API_PASSWORD", 2024-03-31T23:37:05.040 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.040 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.040 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.040 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "admin", 2024-03-31T23:37:05.040 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.040 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.040 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.040 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.041 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.041 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.041 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.041 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.041 INFO:teuthology.orchestra.run.smithi046.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-03-31T23:37:05.041 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-03-31T23:37:05.041 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.041 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.041 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.041 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "True", 2024-03-31T23:37:05.041 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.041 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.041 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.041 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.041 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.042 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.042 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.042 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.042 INFO:teuthology.orchestra.run.smithi046.stdout: "GRAFANA_API_URL": { 2024-03-31T23:37:05.042 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "GRAFANA_API_URL", 2024-03-31T23:37:05.042 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.042 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.042 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.042 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-31T23:37:05.042 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.042 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.042 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.042 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.042 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.043 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.043 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.043 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.043 INFO:teuthology.orchestra.run.smithi046.stdout: "GRAFANA_API_USERNAME": { 2024-03-31T23:37:05.043 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "GRAFANA_API_USERNAME", 2024-03-31T23:37:05.043 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.043 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.043 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.043 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "admin", 2024-03-31T23:37:05.043 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.043 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.043 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.043 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.043 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.043 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.044 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.044 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.044 INFO:teuthology.orchestra.run.smithi046.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-03-31T23:37:05.044 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-03-31T23:37:05.044 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.044 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.044 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.044 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-31T23:37:05.044 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.044 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.044 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.044 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.044 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.044 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.044 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.045 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.045 INFO:teuthology.orchestra.run.smithi046.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-03-31T23:37:05.045 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-03-31T23:37:05.045 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.045 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.045 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.045 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-31T23:37:05.045 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.045 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.045 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.045 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.045 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.045 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.045 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.045 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.046 INFO:teuthology.orchestra.run.smithi046.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-03-31T23:37:05.046 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-03-31T23:37:05.046 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.046 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.046 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.046 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "True", 2024-03-31T23:37:05.046 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.046 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.046 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.046 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.046 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.046 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.046 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.047 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.047 INFO:teuthology.orchestra.run.smithi046.stdout: "ISSUE_TRACKER_API_KEY": { 2024-03-31T23:37:05.047 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-03-31T23:37:05.047 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.047 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.047 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.047 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-31T23:37:05.047 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.047 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.047 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.047 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.047 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.047 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.047 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.048 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.048 INFO:teuthology.orchestra.run.smithi046.stdout: "PROMETHEUS_API_HOST": { 2024-03-31T23:37:05.048 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "PROMETHEUS_API_HOST", 2024-03-31T23:37:05.048 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.048 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.048 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.048 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-31T23:37:05.048 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.048 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.048 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.048 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.048 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.048 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.048 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.048 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.049 INFO:teuthology.orchestra.run.smithi046.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-03-31T23:37:05.049 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-03-31T23:37:05.049 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.049 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.049 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.049 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "True", 2024-03-31T23:37:05.049 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.049 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.049 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.049 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.049 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.049 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.049 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.049 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.050 INFO:teuthology.orchestra.run.smithi046.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-03-31T23:37:05.050 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-03-31T23:37:05.050 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.050 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.050 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.050 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-31T23:37:05.050 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.050 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.050 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.050 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.050 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.050 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.050 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.050 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.051 INFO:teuthology.orchestra.run.smithi046.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-03-31T23:37:05.051 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-03-31T23:37:05.051 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.051 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.051 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.051 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-31T23:37:05.051 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.051 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.051 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.051 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.051 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.051 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.051 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.051 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.051 INFO:teuthology.orchestra.run.smithi046.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-03-31T23:37:05.052 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-03-31T23:37:05.052 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.052 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.052 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.052 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "True", 2024-03-31T23:37:05.052 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.052 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.052 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.052 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.052 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.052 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.052 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.052 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.052 INFO:teuthology.orchestra.run.smithi046.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-03-31T23:37:05.052 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-03-31T23:37:05.053 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.053 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.053 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.053 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "True", 2024-03-31T23:37:05.053 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.053 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.053 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.053 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.053 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.053 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.053 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.053 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.053 INFO:teuthology.orchestra.run.smithi046.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-03-31T23:37:05.053 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-03-31T23:37:05.054 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.054 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.054 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.054 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-31T23:37:05.054 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.054 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.054 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.054 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.054 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.054 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.054 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.054 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.054 INFO:teuthology.orchestra.run.smithi046.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-03-31T23:37:05.054 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-03-31T23:37:05.054 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.055 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.055 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.055 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-31T23:37:05.055 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.055 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.055 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.055 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.055 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.055 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.055 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.055 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.055 INFO:teuthology.orchestra.run.smithi046.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-03-31T23:37:05.055 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-03-31T23:37:05.055 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.055 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.056 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.056 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-31T23:37:05.056 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.056 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.056 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.056 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.056 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.056 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.056 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.056 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.056 INFO:teuthology.orchestra.run.smithi046.stdout: "PWD_POLICY_ENABLED": { 2024-03-31T23:37:05.056 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "PWD_POLICY_ENABLED", 2024-03-31T23:37:05.056 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.056 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.057 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.057 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "True", 2024-03-31T23:37:05.057 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.057 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.057 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.057 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.057 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.057 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.057 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.057 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.057 INFO:teuthology.orchestra.run.smithi046.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-03-31T23:37:05.057 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-03-31T23:37:05.057 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.057 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.058 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.058 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-03-31T23:37:05.058 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.058 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.058 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.058 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.058 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.058 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.058 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.058 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.058 INFO:teuthology.orchestra.run.smithi046.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-03-31T23:37:05.058 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-03-31T23:37:05.058 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "int", 2024-03-31T23:37:05.059 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.059 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.059 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "10", 2024-03-31T23:37:05.059 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.059 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.059 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.059 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.059 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.059 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.059 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.059 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.059 INFO:teuthology.orchestra.run.smithi046.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-03-31T23:37:05.059 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-03-31T23:37:05.059 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "int", 2024-03-31T23:37:05.059 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.060 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.060 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "8", 2024-03-31T23:37:05.060 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.060 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.060 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.060 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.060 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.060 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.060 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.060 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.060 INFO:teuthology.orchestra.run.smithi046.stdout: "REST_REQUESTS_TIMEOUT": { 2024-03-31T23:37:05.060 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-03-31T23:37:05.060 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "int", 2024-03-31T23:37:05.060 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.060 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.061 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "45", 2024-03-31T23:37:05.061 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.061 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.061 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.061 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.061 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.061 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.061 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.061 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.061 INFO:teuthology.orchestra.run.smithi046.stdout: "RGW_API_ACCESS_KEY": { 2024-03-31T23:37:05.061 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "RGW_API_ACCESS_KEY", 2024-03-31T23:37:05.061 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.061 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.061 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.061 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-31T23:37:05.062 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.062 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.062 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.062 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.062 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.062 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.062 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.062 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.062 INFO:teuthology.orchestra.run.smithi046.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-03-31T23:37:05.062 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-03-31T23:37:05.062 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.062 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.062 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.062 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "admin", 2024-03-31T23:37:05.063 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.063 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.063 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.063 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.063 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.063 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.063 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.063 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.063 INFO:teuthology.orchestra.run.smithi046.stdout: "RGW_API_SECRET_KEY": { 2024-03-31T23:37:05.063 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "RGW_API_SECRET_KEY", 2024-03-31T23:37:05.063 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.063 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.063 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.063 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-31T23:37:05.063 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.064 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.064 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.064 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.064 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.064 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.064 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.064 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.064 INFO:teuthology.orchestra.run.smithi046.stdout: "RGW_API_SSL_VERIFY": { 2024-03-31T23:37:05.064 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "RGW_API_SSL_VERIFY", 2024-03-31T23:37:05.064 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.064 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.064 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.064 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "True", 2024-03-31T23:37:05.064 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.065 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.065 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.065 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.065 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.065 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.065 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.065 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.065 INFO:teuthology.orchestra.run.smithi046.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-03-31T23:37:05.065 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-03-31T23:37:05.065 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "int", 2024-03-31T23:37:05.065 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.065 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.065 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "0", 2024-03-31T23:37:05.065 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.066 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.066 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.066 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.066 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.066 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.066 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.066 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.066 INFO:teuthology.orchestra.run.smithi046.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-03-31T23:37:05.066 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-03-31T23:37:05.066 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "int", 2024-03-31T23:37:05.066 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.066 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.066 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "10", 2024-03-31T23:37:05.066 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.066 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.067 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.067 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.067 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.067 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.067 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.067 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.067 INFO:teuthology.orchestra.run.smithi046.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-03-31T23:37:05.067 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-03-31T23:37:05.067 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "int", 2024-03-31T23:37:05.067 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.067 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.067 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "5", 2024-03-31T23:37:05.067 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.067 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.068 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.068 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.068 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.068 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.068 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.068 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.068 INFO:teuthology.orchestra.run.smithi046.stdout: "cross_origin_url": { 2024-03-31T23:37:05.068 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "cross_origin_url", 2024-03-31T23:37:05.068 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.068 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.068 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.068 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-31T23:37:05.068 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.068 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.068 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.069 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.069 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.069 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.069 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.069 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.069 INFO:teuthology.orchestra.run.smithi046.stdout: "crt_file": { 2024-03-31T23:37:05.069 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "crt_file", 2024-03-31T23:37:05.069 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.069 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.069 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.069 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-31T23:37:05.069 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.069 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.069 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.070 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.070 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.070 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.070 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.070 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.070 INFO:teuthology.orchestra.run.smithi046.stdout: "debug": { 2024-03-31T23:37:05.070 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "debug", 2024-03-31T23:37:05.070 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.070 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.070 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.070 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-31T23:37:05.070 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.070 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.070 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.071 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Enable/disable debug options", 2024-03-31T23:37:05.071 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.071 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.071 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.071 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.071 INFO:teuthology.orchestra.run.smithi046.stdout: "jwt_token_ttl": { 2024-03-31T23:37:05.071 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "jwt_token_ttl", 2024-03-31T23:37:05.071 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "int", 2024-03-31T23:37:05.071 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.071 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.071 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "28800", 2024-03-31T23:37:05.071 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.071 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.072 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.072 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.072 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.072 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.072 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.072 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.072 INFO:teuthology.orchestra.run.smithi046.stdout: "key_file": { 2024-03-31T23:37:05.072 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "key_file", 2024-03-31T23:37:05.072 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.072 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.072 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.072 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-31T23:37:05.072 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.072 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.072 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.073 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.073 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.073 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.073 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.073 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.073 INFO:teuthology.orchestra.run.smithi046.stdout: "log_level": { 2024-03-31T23:37:05.073 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_level", 2024-03-31T23:37:05.073 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.073 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.073 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.073 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-31T23:37:05.073 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.073 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.073 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-31T23:37:05.074 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-31T23:37:05.074 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-31T23:37:05.074 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-31T23:37:05.074 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-31T23:37:05.074 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-31T23:37:05.074 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-31T23:37:05.074 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-31T23:37:05.074 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.074 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.074 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.074 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.074 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.074 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster": { 2024-03-31T23:37:05.074 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster", 2024-03-31T23:37:05.075 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.075 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.075 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.075 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-31T23:37:05.075 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.075 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.075 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.075 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.075 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.075 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.075 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.075 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.075 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster_level": { 2024-03-31T23:37:05.076 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster_level", 2024-03-31T23:37:05.076 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.076 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.076 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.076 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "info", 2024-03-31T23:37:05.076 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.076 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.076 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-31T23:37:05.076 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-31T23:37:05.076 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-31T23:37:05.076 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-31T23:37:05.076 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-31T23:37:05.076 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-31T23:37:05.076 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-31T23:37:05.077 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-31T23:37:05.077 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.077 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.077 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.077 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.077 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.077 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_file": { 2024-03-31T23:37:05.077 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_file", 2024-03-31T23:37:05.077 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.077 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.077 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.077 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-31T23:37:05.077 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.077 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.078 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.078 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.078 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.078 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.078 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.078 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.078 INFO:teuthology.orchestra.run.smithi046.stdout: "motd": { 2024-03-31T23:37:05.078 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "motd", 2024-03-31T23:37:05.078 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.078 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.078 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.078 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-31T23:37:05.078 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.078 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.078 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.079 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "The message of the day", 2024-03-31T23:37:05.079 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.079 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.079 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.079 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.079 INFO:teuthology.orchestra.run.smithi046.stdout: "redirect_resolve_ip_addr": { 2024-03-31T23:37:05.079 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "redirect_resolve_ip_addr", 2024-03-31T23:37:05.079 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.079 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.079 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.079 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-31T23:37:05.079 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.079 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.079 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.079 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.080 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.080 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.080 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.080 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.080 INFO:teuthology.orchestra.run.smithi046.stdout: "server_addr": { 2024-03-31T23:37:05.080 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "server_addr", 2024-03-31T23:37:05.080 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.080 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.080 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.080 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "::", 2024-03-31T23:37:05.080 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.080 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.080 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.080 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.080 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.081 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.081 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.081 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.081 INFO:teuthology.orchestra.run.smithi046.stdout: "server_port": { 2024-03-31T23:37:05.081 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "server_port", 2024-03-31T23:37:05.081 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "int", 2024-03-31T23:37:05.081 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.081 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.081 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "8080", 2024-03-31T23:37:05.081 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.081 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.081 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.081 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.081 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.081 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.082 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.082 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.082 INFO:teuthology.orchestra.run.smithi046.stdout: "ssl": { 2024-03-31T23:37:05.082 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "ssl", 2024-03-31T23:37:05.082 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.082 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.082 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.082 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "True", 2024-03-31T23:37:05.082 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.082 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.082 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.082 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.082 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.082 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.083 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.083 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.083 INFO:teuthology.orchestra.run.smithi046.stdout: "ssl_server_port": { 2024-03-31T23:37:05.083 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "ssl_server_port", 2024-03-31T23:37:05.083 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "int", 2024-03-31T23:37:05.083 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.083 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.083 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "8443", 2024-03-31T23:37:05.083 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.083 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.083 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.083 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.083 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.083 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.084 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.084 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.084 INFO:teuthology.orchestra.run.smithi046.stdout: "standby_behaviour": { 2024-03-31T23:37:05.084 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "standby_behaviour", 2024-03-31T23:37:05.084 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.084 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.084 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.084 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "redirect", 2024-03-31T23:37:05.084 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.084 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.084 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-31T23:37:05.084 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-31T23:37:05.084 INFO:teuthology.orchestra.run.smithi046.stdout: "redirect" 2024-03-31T23:37:05.084 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-31T23:37:05.084 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.085 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.085 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.085 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.085 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.085 INFO:teuthology.orchestra.run.smithi046.stdout: "standby_error_status_code": { 2024-03-31T23:37:05.085 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "standby_error_status_code", 2024-03-31T23:37:05.085 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "int", 2024-03-31T23:37:05.085 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.085 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.085 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "500", 2024-03-31T23:37:05.085 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "400", 2024-03-31T23:37:05.085 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "599", 2024-03-31T23:37:05.085 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.086 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.086 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.086 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.086 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.086 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.086 INFO:teuthology.orchestra.run.smithi046.stdout: "url_prefix": { 2024-03-31T23:37:05.086 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "url_prefix", 2024-03-31T23:37:05.086 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.086 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.086 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.086 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-31T23:37:05.086 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.086 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.087 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.087 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.087 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.087 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.087 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.087 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-31T23:37:05.087 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-31T23:37:05.087 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.087 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-03-31T23:37:05.087 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "devicehealth", 2024-03-31T23:37:05.087 INFO:teuthology.orchestra.run.smithi046.stdout: "can_run": true, 2024-03-31T23:37:05.087 INFO:teuthology.orchestra.run.smithi046.stdout: "error_string": "", 2024-03-31T23:37:05.087 INFO:teuthology.orchestra.run.smithi046.stdout: "module_options": { 2024-03-31T23:37:05.087 INFO:teuthology.orchestra.run.smithi046.stdout: "enable_monitoring": { 2024-03-31T23:37:05.087 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "enable_monitoring", 2024-03-31T23:37:05.088 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.088 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.088 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.088 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "True", 2024-03-31T23:37:05.088 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.088 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.088 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.088 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "monitor device health metrics", 2024-03-31T23:37:05.088 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.088 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.088 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.088 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.088 INFO:teuthology.orchestra.run.smithi046.stdout: "log_level": { 2024-03-31T23:37:05.088 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_level", 2024-03-31T23:37:05.089 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.089 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.089 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.089 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-31T23:37:05.089 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.089 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.089 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-31T23:37:05.089 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-31T23:37:05.089 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-31T23:37:05.089 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-31T23:37:05.089 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-31T23:37:05.089 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-31T23:37:05.089 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-31T23:37:05.089 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-31T23:37:05.090 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.090 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.090 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.090 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.090 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.090 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster": { 2024-03-31T23:37:05.090 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster", 2024-03-31T23:37:05.090 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.090 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.090 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.090 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-31T23:37:05.090 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.090 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.090 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.090 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.091 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.091 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.091 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.091 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.091 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster_level": { 2024-03-31T23:37:05.091 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster_level", 2024-03-31T23:37:05.091 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.091 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.091 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.091 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "info", 2024-03-31T23:37:05.091 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.091 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.091 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-31T23:37:05.091 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-31T23:37:05.091 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-31T23:37:05.092 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-31T23:37:05.092 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-31T23:37:05.092 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-31T23:37:05.092 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-31T23:37:05.092 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-31T23:37:05.092 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.092 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.092 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.092 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.092 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.092 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_file": { 2024-03-31T23:37:05.092 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_file", 2024-03-31T23:37:05.092 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.092 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.093 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.093 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-31T23:37:05.093 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.093 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.093 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.093 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.093 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.093 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.093 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.093 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.093 INFO:teuthology.orchestra.run.smithi046.stdout: "mark_out_threshold": { 2024-03-31T23:37:05.093 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "mark_out_threshold", 2024-03-31T23:37:05.093 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "secs", 2024-03-31T23:37:05.093 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.093 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.094 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "2419200", 2024-03-31T23:37:05.094 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.094 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.094 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.094 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-03-31T23:37:05.094 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.094 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.094 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.094 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.094 INFO:teuthology.orchestra.run.smithi046.stdout: "pool_name": { 2024-03-31T23:37:05.094 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "pool_name", 2024-03-31T23:37:05.094 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.094 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.094 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.095 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "device_health_metrics", 2024-03-31T23:37:05.095 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.095 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.095 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.095 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "name of pool in which to store device health metrics", 2024-03-31T23:37:05.095 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.095 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.095 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.095 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.095 INFO:teuthology.orchestra.run.smithi046.stdout: "retention_period": { 2024-03-31T23:37:05.095 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "retention_period", 2024-03-31T23:37:05.095 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "secs", 2024-03-31T23:37:05.095 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.095 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.095 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "15552000", 2024-03-31T23:37:05.096 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.096 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.096 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.096 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "how long to retain device health metrics", 2024-03-31T23:37:05.096 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.096 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.096 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.096 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.096 INFO:teuthology.orchestra.run.smithi046.stdout: "scrape_frequency": { 2024-03-31T23:37:05.096 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "scrape_frequency", 2024-03-31T23:37:05.096 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "secs", 2024-03-31T23:37:05.096 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.096 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.096 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "86400", 2024-03-31T23:37:05.097 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.097 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.097 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.097 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "how frequently to scrape device health metrics", 2024-03-31T23:37:05.097 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.097 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.097 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.097 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.097 INFO:teuthology.orchestra.run.smithi046.stdout: "self_heal": { 2024-03-31T23:37:05.097 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "self_heal", 2024-03-31T23:37:05.097 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.097 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.097 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.097 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "True", 2024-03-31T23:37:05.097 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.098 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.098 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.098 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-03-31T23:37:05.098 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.098 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.098 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.098 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.098 INFO:teuthology.orchestra.run.smithi046.stdout: "sleep_interval": { 2024-03-31T23:37:05.098 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "sleep_interval", 2024-03-31T23:37:05.098 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "secs", 2024-03-31T23:37:05.098 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.098 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.098 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "600", 2024-03-31T23:37:05.099 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.099 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.099 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.099 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "how frequently to wake up and check device health", 2024-03-31T23:37:05.099 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.099 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.099 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.099 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.099 INFO:teuthology.orchestra.run.smithi046.stdout: "warn_threshold": { 2024-03-31T23:37:05.099 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "warn_threshold", 2024-03-31T23:37:05.099 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "secs", 2024-03-31T23:37:05.099 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.099 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.099 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "7257600", 2024-03-31T23:37:05.100 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.100 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.100 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.100 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-03-31T23:37:05.100 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.100 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.100 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.100 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-31T23:37:05.100 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-31T23:37:05.100 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.100 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-03-31T23:37:05.100 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "diskprediction_local", 2024-03-31T23:37:05.100 INFO:teuthology.orchestra.run.smithi046.stdout: "can_run": true, 2024-03-31T23:37:05.100 INFO:teuthology.orchestra.run.smithi046.stdout: "error_string": "", 2024-03-31T23:37:05.100 INFO:teuthology.orchestra.run.smithi046.stdout: "module_options": { 2024-03-31T23:37:05.101 INFO:teuthology.orchestra.run.smithi046.stdout: "log_level": { 2024-03-31T23:37:05.101 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_level", 2024-03-31T23:37:05.101 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.101 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.101 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.101 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-31T23:37:05.101 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.101 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.101 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-31T23:37:05.101 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-31T23:37:05.101 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-31T23:37:05.101 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-31T23:37:05.101 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-31T23:37:05.101 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-31T23:37:05.101 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-31T23:37:05.102 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-31T23:37:05.102 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.102 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.102 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.102 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.102 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.102 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster": { 2024-03-31T23:37:05.102 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster", 2024-03-31T23:37:05.102 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.102 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.102 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.102 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-31T23:37:05.102 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.102 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.103 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.103 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.103 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.103 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.103 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.103 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.103 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster_level": { 2024-03-31T23:37:05.103 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster_level", 2024-03-31T23:37:05.103 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.103 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.103 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.103 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "info", 2024-03-31T23:37:05.103 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.103 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.104 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-31T23:37:05.104 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-31T23:37:05.104 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-31T23:37:05.104 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-31T23:37:05.104 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-31T23:37:05.104 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-31T23:37:05.104 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-31T23:37:05.104 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-31T23:37:05.104 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.104 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.104 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.104 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.104 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.104 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_file": { 2024-03-31T23:37:05.105 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_file", 2024-03-31T23:37:05.105 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.105 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.105 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.105 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-31T23:37:05.105 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.105 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.105 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.105 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.105 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.105 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.105 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.105 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.105 INFO:teuthology.orchestra.run.smithi046.stdout: "predict_interval": { 2024-03-31T23:37:05.105 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "predict_interval", 2024-03-31T23:37:05.106 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.106 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.106 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.106 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "86400", 2024-03-31T23:37:05.106 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.106 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.106 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.106 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.106 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.106 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.106 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.106 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.106 INFO:teuthology.orchestra.run.smithi046.stdout: "predictor_model": { 2024-03-31T23:37:05.106 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "predictor_model", 2024-03-31T23:37:05.107 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.107 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.107 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.107 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "prophetstor", 2024-03-31T23:37:05.107 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.107 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.107 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.107 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.107 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.107 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.107 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.107 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.107 INFO:teuthology.orchestra.run.smithi046.stdout: "sleep_interval": { 2024-03-31T23:37:05.107 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "sleep_interval", 2024-03-31T23:37:05.108 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.108 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.108 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.108 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "600", 2024-03-31T23:37:05.108 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.108 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.108 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.108 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.108 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.108 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.108 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.108 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-31T23:37:05.108 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-31T23:37:05.108 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.108 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-03-31T23:37:05.109 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "influx", 2024-03-31T23:37:05.109 INFO:teuthology.orchestra.run.smithi046.stdout: "can_run": false, 2024-03-31T23:37:05.109 INFO:teuthology.orchestra.run.smithi046.stdout: "error_string": "influxdb python module not found", 2024-03-31T23:37:05.109 INFO:teuthology.orchestra.run.smithi046.stdout: "module_options": { 2024-03-31T23:37:05.109 INFO:teuthology.orchestra.run.smithi046.stdout: "batch_size": { 2024-03-31T23:37:05.109 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "batch_size", 2024-03-31T23:37:05.109 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "int", 2024-03-31T23:37:05.109 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.109 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.109 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "5000", 2024-03-31T23:37:05.109 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.109 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.109 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.109 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-03-31T23:37:05.109 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.110 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.110 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.110 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.110 INFO:teuthology.orchestra.run.smithi046.stdout: "database": { 2024-03-31T23:37:05.110 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "database", 2024-03-31T23:37:05.110 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.110 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.110 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.110 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "ceph", 2024-03-31T23:37:05.110 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.110 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.110 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.110 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-03-31T23:37:05.110 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.110 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.111 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.111 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.111 INFO:teuthology.orchestra.run.smithi046.stdout: "hostname": { 2024-03-31T23:37:05.111 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "hostname", 2024-03-31T23:37:05.111 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.111 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.111 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.111 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-31T23:37:05.111 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.111 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.111 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.111 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "InfluxDB server hostname", 2024-03-31T23:37:05.111 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.112 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.112 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.112 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.112 INFO:teuthology.orchestra.run.smithi046.stdout: "interval": { 2024-03-31T23:37:05.112 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "interval", 2024-03-31T23:37:05.112 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "secs", 2024-03-31T23:37:05.112 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.112 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.112 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "30", 2024-03-31T23:37:05.112 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "5", 2024-03-31T23:37:05.112 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.112 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.112 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-03-31T23:37:05.112 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.112 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.113 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.113 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.113 INFO:teuthology.orchestra.run.smithi046.stdout: "log_level": { 2024-03-31T23:37:05.113 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_level", 2024-03-31T23:37:05.113 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.113 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.113 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.113 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-31T23:37:05.113 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.113 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.113 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-31T23:37:05.113 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-31T23:37:05.113 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-31T23:37:05.113 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-31T23:37:05.113 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-31T23:37:05.114 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-31T23:37:05.114 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-31T23:37:05.114 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-31T23:37:05.114 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.114 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.114 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.114 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.114 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.114 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster": { 2024-03-31T23:37:05.114 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster", 2024-03-31T23:37:05.114 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.114 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.114 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.114 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-31T23:37:05.115 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.115 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.115 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.115 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.115 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.115 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.115 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.115 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.115 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster_level": { 2024-03-31T23:37:05.115 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster_level", 2024-03-31T23:37:05.115 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.115 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.115 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.115 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "info", 2024-03-31T23:37:05.115 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.116 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.116 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-31T23:37:05.116 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-31T23:37:05.116 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-31T23:37:05.116 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-31T23:37:05.116 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-31T23:37:05.116 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-31T23:37:05.116 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-31T23:37:05.116 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-31T23:37:05.116 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.116 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.116 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.116 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.116 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.116 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_file": { 2024-03-31T23:37:05.117 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_file", 2024-03-31T23:37:05.117 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.117 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.117 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.117 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-31T23:37:05.117 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.117 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.117 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.117 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.117 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.117 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.117 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.117 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.117 INFO:teuthology.orchestra.run.smithi046.stdout: "password": { 2024-03-31T23:37:05.117 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "password", 2024-03-31T23:37:05.118 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.118 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.118 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.118 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-31T23:37:05.118 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.118 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.118 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.118 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "password of InfluxDB server user", 2024-03-31T23:37:05.118 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.118 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.118 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.118 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.118 INFO:teuthology.orchestra.run.smithi046.stdout: "port": { 2024-03-31T23:37:05.118 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "port", 2024-03-31T23:37:05.119 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "int", 2024-03-31T23:37:05.119 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.119 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.119 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "8086", 2024-03-31T23:37:05.119 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.119 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.119 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.119 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "InfluxDB server port", 2024-03-31T23:37:05.119 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.119 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.119 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.119 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.119 INFO:teuthology.orchestra.run.smithi046.stdout: "ssl": { 2024-03-31T23:37:05.119 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "ssl", 2024-03-31T23:37:05.119 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.120 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.120 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.120 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "false", 2024-03-31T23:37:05.120 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.120 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.120 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.120 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-03-31T23:37:05.120 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.120 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.120 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.120 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.120 INFO:teuthology.orchestra.run.smithi046.stdout: "threads": { 2024-03-31T23:37:05.120 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "threads", 2024-03-31T23:37:05.120 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "int", 2024-03-31T23:37:05.120 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.121 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.121 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "5", 2024-03-31T23:37:05.121 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "1", 2024-03-31T23:37:05.121 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "32", 2024-03-31T23:37:05.121 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.121 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-03-31T23:37:05.121 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.121 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.121 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.121 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.121 INFO:teuthology.orchestra.run.smithi046.stdout: "username": { 2024-03-31T23:37:05.121 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "username", 2024-03-31T23:37:05.121 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.121 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.121 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.122 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-31T23:37:05.122 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.122 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.122 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.122 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "username of InfluxDB server user", 2024-03-31T23:37:05.122 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.122 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.122 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.122 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.122 INFO:teuthology.orchestra.run.smithi046.stdout: "verify_ssl": { 2024-03-31T23:37:05.122 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "verify_ssl", 2024-03-31T23:37:05.122 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.122 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.122 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.122 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "true", 2024-03-31T23:37:05.123 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.123 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.123 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.123 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-03-31T23:37:05.123 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.123 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.123 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.123 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-31T23:37:05.123 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-31T23:37:05.123 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.123 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-03-31T23:37:05.123 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "insights", 2024-03-31T23:37:05.123 INFO:teuthology.orchestra.run.smithi046.stdout: "can_run": true, 2024-03-31T23:37:05.124 INFO:teuthology.orchestra.run.smithi046.stdout: "error_string": "", 2024-03-31T23:37:05.124 INFO:teuthology.orchestra.run.smithi046.stdout: "module_options": { 2024-03-31T23:37:05.124 INFO:teuthology.orchestra.run.smithi046.stdout: "log_level": { 2024-03-31T23:37:05.124 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_level", 2024-03-31T23:37:05.124 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.124 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.124 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.124 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-31T23:37:05.124 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.124 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.124 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-31T23:37:05.124 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-31T23:37:05.124 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-31T23:37:05.124 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-31T23:37:05.124 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-31T23:37:05.125 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-31T23:37:05.125 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-31T23:37:05.125 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-31T23:37:05.125 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.125 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.125 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.125 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.125 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.125 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster": { 2024-03-31T23:37:05.125 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster", 2024-03-31T23:37:05.125 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.125 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.125 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.125 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-31T23:37:05.125 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.126 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.126 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.126 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.126 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.126 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.126 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.126 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.126 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster_level": { 2024-03-31T23:37:05.126 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster_level", 2024-03-31T23:37:05.126 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.126 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.126 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.126 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "info", 2024-03-31T23:37:05.127 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.127 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.127 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-31T23:37:05.127 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-31T23:37:05.127 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-31T23:37:05.127 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-31T23:37:05.127 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-31T23:37:05.127 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-31T23:37:05.127 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-31T23:37:05.127 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-31T23:37:05.127 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.127 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.127 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.128 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.128 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.128 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_file": { 2024-03-31T23:37:05.128 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_file", 2024-03-31T23:37:05.128 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.128 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.128 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.128 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-31T23:37:05.128 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.128 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.128 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.128 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.128 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.128 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.128 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.129 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-31T23:37:05.129 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-31T23:37:05.129 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.129 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-03-31T23:37:05.129 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "iostat", 2024-03-31T23:37:05.129 INFO:teuthology.orchestra.run.smithi046.stdout: "can_run": true, 2024-03-31T23:37:05.129 INFO:teuthology.orchestra.run.smithi046.stdout: "error_string": "", 2024-03-31T23:37:05.129 INFO:teuthology.orchestra.run.smithi046.stdout: "module_options": { 2024-03-31T23:37:05.129 INFO:teuthology.orchestra.run.smithi046.stdout: "log_level": { 2024-03-31T23:37:05.129 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_level", 2024-03-31T23:37:05.129 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.129 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.129 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.129 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-31T23:37:05.130 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.130 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.130 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-31T23:37:05.130 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-31T23:37:05.130 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-31T23:37:05.130 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-31T23:37:05.130 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-31T23:37:05.130 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-31T23:37:05.130 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-31T23:37:05.130 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-31T23:37:05.130 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.130 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.130 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.130 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.130 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.131 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster": { 2024-03-31T23:37:05.131 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster", 2024-03-31T23:37:05.131 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.131 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.131 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.131 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-31T23:37:05.131 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.131 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.131 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.131 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.131 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.131 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.131 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.131 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.131 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster_level": { 2024-03-31T23:37:05.132 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster_level", 2024-03-31T23:37:05.132 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.132 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.132 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.132 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "info", 2024-03-31T23:37:05.132 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.132 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.132 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-31T23:37:05.132 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-31T23:37:05.132 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-31T23:37:05.132 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-31T23:37:05.132 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-31T23:37:05.132 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-31T23:37:05.132 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-31T23:37:05.132 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-31T23:37:05.133 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.133 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.133 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.133 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.133 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.133 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_file": { 2024-03-31T23:37:05.133 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_file", 2024-03-31T23:37:05.133 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.133 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.133 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.133 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-31T23:37:05.133 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.133 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.133 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.133 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.134 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.134 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.134 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.134 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-31T23:37:05.134 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-31T23:37:05.134 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.134 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-03-31T23:37:05.134 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "k8sevents", 2024-03-31T23:37:05.134 INFO:teuthology.orchestra.run.smithi046.stdout: "can_run": true, 2024-03-31T23:37:05.134 INFO:teuthology.orchestra.run.smithi046.stdout: "error_string": "", 2024-03-31T23:37:05.134 INFO:teuthology.orchestra.run.smithi046.stdout: "module_options": { 2024-03-31T23:37:05.134 INFO:teuthology.orchestra.run.smithi046.stdout: "ceph_event_retention_days": { 2024-03-31T23:37:05.134 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "ceph_event_retention_days", 2024-03-31T23:37:05.134 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "int", 2024-03-31T23:37:05.135 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.135 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.135 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "7", 2024-03-31T23:37:05.135 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.135 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.135 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.135 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Days to hold ceph event information within local cache", 2024-03-31T23:37:05.135 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.135 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.135 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.135 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.135 INFO:teuthology.orchestra.run.smithi046.stdout: "config_check_secs": { 2024-03-31T23:37:05.135 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "config_check_secs", 2024-03-31T23:37:05.135 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "int", 2024-03-31T23:37:05.135 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.136 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.136 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "10", 2024-03-31T23:37:05.136 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "10", 2024-03-31T23:37:05.136 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.136 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.136 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-03-31T23:37:05.136 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.136 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.136 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.136 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.136 INFO:teuthology.orchestra.run.smithi046.stdout: "log_level": { 2024-03-31T23:37:05.136 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_level", 2024-03-31T23:37:05.136 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.136 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.137 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.137 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-31T23:37:05.137 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.137 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.137 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-31T23:37:05.137 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-31T23:37:05.137 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-31T23:37:05.137 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-31T23:37:05.137 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-31T23:37:05.137 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-31T23:37:05.137 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-31T23:37:05.138 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-31T23:37:05.138 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.138 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.138 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.138 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.138 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.138 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster": { 2024-03-31T23:37:05.138 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster", 2024-03-31T23:37:05.138 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.138 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.138 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.138 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-31T23:37:05.138 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.138 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.139 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.139 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.139 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.139 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.139 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.139 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.139 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster_level": { 2024-03-31T23:37:05.139 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster_level", 2024-03-31T23:37:05.139 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.139 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.139 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.139 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "info", 2024-03-31T23:37:05.139 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.139 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.140 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-31T23:37:05.140 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-31T23:37:05.140 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-31T23:37:05.140 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-31T23:37:05.140 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-31T23:37:05.140 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-31T23:37:05.140 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-31T23:37:05.140 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-31T23:37:05.140 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.140 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.140 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.140 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.140 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.140 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_file": { 2024-03-31T23:37:05.141 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_file", 2024-03-31T23:37:05.141 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.141 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.141 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.141 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-31T23:37:05.141 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.141 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.141 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.141 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.141 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.141 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.141 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.141 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-31T23:37:05.141 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-31T23:37:05.142 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.142 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-03-31T23:37:05.142 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "localpool", 2024-03-31T23:37:05.142 INFO:teuthology.orchestra.run.smithi046.stdout: "can_run": true, 2024-03-31T23:37:05.142 INFO:teuthology.orchestra.run.smithi046.stdout: "error_string": "", 2024-03-31T23:37:05.142 INFO:teuthology.orchestra.run.smithi046.stdout: "module_options": { 2024-03-31T23:37:05.142 INFO:teuthology.orchestra.run.smithi046.stdout: "failure_domain": { 2024-03-31T23:37:05.142 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "failure_domain", 2024-03-31T23:37:05.142 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.142 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.142 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.142 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "host", 2024-03-31T23:37:05.142 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.143 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.143 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.143 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "failure domain for any created local pool", 2024-03-31T23:37:05.143 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-03-31T23:37:05.143 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.143 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.143 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.143 INFO:teuthology.orchestra.run.smithi046.stdout: "log_level": { 2024-03-31T23:37:05.143 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_level", 2024-03-31T23:37:05.143 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.143 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.143 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.143 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-31T23:37:05.143 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.144 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.144 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-31T23:37:05.144 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-31T23:37:05.144 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-31T23:37:05.144 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-31T23:37:05.144 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-31T23:37:05.144 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-31T23:37:05.144 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-31T23:37:05.144 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-31T23:37:05.144 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.144 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.144 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.144 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.144 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.144 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster": { 2024-03-31T23:37:05.145 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster", 2024-03-31T23:37:05.145 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.145 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.145 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.145 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-31T23:37:05.145 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.145 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.145 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.145 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.145 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.145 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.145 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.145 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.145 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster_level": { 2024-03-31T23:37:05.146 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster_level", 2024-03-31T23:37:05.146 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.146 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.146 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.146 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "info", 2024-03-31T23:37:05.146 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.146 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.146 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-31T23:37:05.146 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-31T23:37:05.146 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-31T23:37:05.146 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-31T23:37:05.146 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-31T23:37:05.146 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-31T23:37:05.146 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-31T23:37:05.146 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-31T23:37:05.147 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.147 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.147 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.147 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.147 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.147 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_file": { 2024-03-31T23:37:05.147 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_file", 2024-03-31T23:37:05.147 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.147 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.147 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.147 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-31T23:37:05.147 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.147 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.147 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.148 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.148 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.148 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.148 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.148 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.148 INFO:teuthology.orchestra.run.smithi046.stdout: "min_size": { 2024-03-31T23:37:05.148 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "min_size", 2024-03-31T23:37:05.148 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "int", 2024-03-31T23:37:05.148 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.148 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.148 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-31T23:37:05.148 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.148 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.148 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.148 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "default min_size for any created local pool", 2024-03-31T23:37:05.149 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-03-31T23:37:05.149 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.149 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.149 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.149 INFO:teuthology.orchestra.run.smithi046.stdout: "num_rep": { 2024-03-31T23:37:05.149 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "num_rep", 2024-03-31T23:37:05.149 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "int", 2024-03-31T23:37:05.149 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.149 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.149 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "3", 2024-03-31T23:37:05.149 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.149 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.149 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.149 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "default replica count for any created local pool", 2024-03-31T23:37:05.150 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.150 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.150 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.150 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.150 INFO:teuthology.orchestra.run.smithi046.stdout: "pg_num": { 2024-03-31T23:37:05.150 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "pg_num", 2024-03-31T23:37:05.150 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "int", 2024-03-31T23:37:05.150 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.150 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.150 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "128", 2024-03-31T23:37:05.150 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.150 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.150 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.151 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "default pg_num for any created local pool", 2024-03-31T23:37:05.151 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.151 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.151 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.151 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.151 INFO:teuthology.orchestra.run.smithi046.stdout: "prefix": { 2024-03-31T23:37:05.151 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "prefix", 2024-03-31T23:37:05.151 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.151 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.151 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.151 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-31T23:37:05.151 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.151 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.151 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.152 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "name prefix for any created local pool", 2024-03-31T23:37:05.152 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.152 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.152 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.152 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.152 INFO:teuthology.orchestra.run.smithi046.stdout: "subtree": { 2024-03-31T23:37:05.152 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "subtree", 2024-03-31T23:37:05.152 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.152 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.152 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.152 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "rack", 2024-03-31T23:37:05.152 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.152 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.152 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.152 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "CRUSH level for which to create a local pool", 2024-03-31T23:37:05.153 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-03-31T23:37:05.153 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.153 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.153 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-31T23:37:05.153 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-31T23:37:05.153 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.153 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-03-31T23:37:05.153 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "mds_autoscaler", 2024-03-31T23:37:05.153 INFO:teuthology.orchestra.run.smithi046.stdout: "can_run": true, 2024-03-31T23:37:05.153 INFO:teuthology.orchestra.run.smithi046.stdout: "error_string": "", 2024-03-31T23:37:05.153 INFO:teuthology.orchestra.run.smithi046.stdout: "module_options": { 2024-03-31T23:37:05.153 INFO:teuthology.orchestra.run.smithi046.stdout: "log_level": { 2024-03-31T23:37:05.153 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_level", 2024-03-31T23:37:05.153 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.153 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.154 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.154 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-31T23:37:05.154 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.154 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.154 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-31T23:37:05.154 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-31T23:37:05.154 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-31T23:37:05.154 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-31T23:37:05.154 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-31T23:37:05.154 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-31T23:37:05.154 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-31T23:37:05.154 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-31T23:37:05.154 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.154 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.155 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.155 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.155 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.155 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster": { 2024-03-31T23:37:05.155 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster", 2024-03-31T23:37:05.155 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.155 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.155 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.155 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-31T23:37:05.155 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.155 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.155 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.155 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.155 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.156 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.156 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.156 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.156 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster_level": { 2024-03-31T23:37:05.156 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster_level", 2024-03-31T23:37:05.156 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.156 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.156 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.156 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "info", 2024-03-31T23:37:05.156 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.156 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.156 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-31T23:37:05.156 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-31T23:37:05.156 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-31T23:37:05.156 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-31T23:37:05.157 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-31T23:37:05.157 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-31T23:37:05.157 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-31T23:37:05.157 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-31T23:37:05.157 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.157 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.157 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.157 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.157 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.157 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_file": { 2024-03-31T23:37:05.157 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_file", 2024-03-31T23:37:05.157 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.157 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.157 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.157 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-31T23:37:05.158 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.158 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.158 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.158 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.158 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.158 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.158 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.158 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-31T23:37:05.158 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-31T23:37:05.158 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.158 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-03-31T23:37:05.158 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "mirroring", 2024-03-31T23:37:05.158 INFO:teuthology.orchestra.run.smithi046.stdout: "can_run": true, 2024-03-31T23:37:05.158 INFO:teuthology.orchestra.run.smithi046.stdout: "error_string": "", 2024-03-31T23:37:05.158 INFO:teuthology.orchestra.run.smithi046.stdout: "module_options": { 2024-03-31T23:37:05.159 INFO:teuthology.orchestra.run.smithi046.stdout: "log_level": { 2024-03-31T23:37:05.159 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_level", 2024-03-31T23:37:05.159 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.159 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.159 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.159 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-31T23:37:05.159 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.159 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.159 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-31T23:37:05.159 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-31T23:37:05.159 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-31T23:37:05.159 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-31T23:37:05.159 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-31T23:37:05.159 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-31T23:37:05.159 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-31T23:37:05.160 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-31T23:37:05.160 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.160 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.160 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.160 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.160 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.160 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster": { 2024-03-31T23:37:05.160 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster", 2024-03-31T23:37:05.160 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.160 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.160 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.160 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-31T23:37:05.160 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.160 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.160 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.161 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.161 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.161 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.161 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.161 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.161 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster_level": { 2024-03-31T23:37:05.161 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster_level", 2024-03-31T23:37:05.161 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.161 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.161 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.161 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "info", 2024-03-31T23:37:05.161 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.161 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.161 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-31T23:37:05.161 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-31T23:37:05.162 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-31T23:37:05.162 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-31T23:37:05.162 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-31T23:37:05.162 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-31T23:37:05.162 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-31T23:37:05.162 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-31T23:37:05.162 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.162 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.162 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.162 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.162 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.162 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_file": { 2024-03-31T23:37:05.162 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_file", 2024-03-31T23:37:05.162 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.162 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.163 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.163 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-31T23:37:05.163 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.163 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.163 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.163 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.163 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.163 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.163 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.163 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-31T23:37:05.163 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-31T23:37:05.163 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.163 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-03-31T23:37:05.163 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "nfs", 2024-03-31T23:37:05.163 INFO:teuthology.orchestra.run.smithi046.stdout: "can_run": true, 2024-03-31T23:37:05.164 INFO:teuthology.orchestra.run.smithi046.stdout: "error_string": "", 2024-03-31T23:37:05.164 INFO:teuthology.orchestra.run.smithi046.stdout: "module_options": { 2024-03-31T23:37:05.164 INFO:teuthology.orchestra.run.smithi046.stdout: "log_level": { 2024-03-31T23:37:05.164 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_level", 2024-03-31T23:37:05.164 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.164 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.164 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.164 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-31T23:37:05.164 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.164 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.164 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-31T23:37:05.164 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-31T23:37:05.164 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-31T23:37:05.165 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-31T23:37:05.165 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-31T23:37:05.165 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-31T23:37:05.165 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-31T23:37:05.165 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-31T23:37:05.165 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.165 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.165 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.165 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.165 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.165 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster": { 2024-03-31T23:37:05.165 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster", 2024-03-31T23:37:05.165 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.165 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.165 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.166 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-31T23:37:05.166 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.166 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.166 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.166 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.166 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.166 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.166 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.166 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.166 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster_level": { 2024-03-31T23:37:05.166 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster_level", 2024-03-31T23:37:05.166 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.166 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.166 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.166 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "info", 2024-03-31T23:37:05.167 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.167 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.167 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-31T23:37:05.167 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-31T23:37:05.167 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-31T23:37:05.167 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-31T23:37:05.167 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-31T23:37:05.167 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-31T23:37:05.167 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-31T23:37:05.167 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-31T23:37:05.167 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.167 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.167 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.167 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.168 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.168 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_file": { 2024-03-31T23:37:05.168 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_file", 2024-03-31T23:37:05.168 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.168 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.168 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.168 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-31T23:37:05.168 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.168 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.168 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.168 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.168 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.168 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.168 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.168 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-31T23:37:05.169 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-31T23:37:05.169 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.169 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-03-31T23:37:05.169 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "orchestrator", 2024-03-31T23:37:05.169 INFO:teuthology.orchestra.run.smithi046.stdout: "can_run": true, 2024-03-31T23:37:05.169 INFO:teuthology.orchestra.run.smithi046.stdout: "error_string": "", 2024-03-31T23:37:05.169 INFO:teuthology.orchestra.run.smithi046.stdout: "module_options": { 2024-03-31T23:37:05.169 INFO:teuthology.orchestra.run.smithi046.stdout: "fail_fs": { 2024-03-31T23:37:05.169 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "fail_fs", 2024-03-31T23:37:05.169 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.169 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.169 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.169 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-31T23:37:05.169 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.169 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.170 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.170 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-03-31T23:37:05.170 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.170 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.170 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.170 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.170 INFO:teuthology.orchestra.run.smithi046.stdout: "log_level": { 2024-03-31T23:37:05.170 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_level", 2024-03-31T23:37:05.170 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.170 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.170 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.170 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-31T23:37:05.170 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.170 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.170 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-31T23:37:05.171 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-31T23:37:05.171 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-31T23:37:05.171 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-31T23:37:05.171 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-31T23:37:05.171 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-31T23:37:05.171 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-31T23:37:05.171 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-31T23:37:05.171 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.171 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.171 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.171 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.171 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.171 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster": { 2024-03-31T23:37:05.171 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster", 2024-03-31T23:37:05.172 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.172 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.172 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.172 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-31T23:37:05.172 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.172 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.172 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.172 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.172 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.172 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.172 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.172 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.172 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster_level": { 2024-03-31T23:37:05.172 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster_level", 2024-03-31T23:37:05.172 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.173 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.173 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.173 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "info", 2024-03-31T23:37:05.173 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.173 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.173 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-31T23:37:05.173 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-31T23:37:05.173 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-31T23:37:05.173 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-31T23:37:05.173 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-31T23:37:05.173 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-31T23:37:05.173 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-31T23:37:05.173 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-31T23:37:05.173 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.173 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.173 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.174 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.174 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.174 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_file": { 2024-03-31T23:37:05.174 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_file", 2024-03-31T23:37:05.174 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.174 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.174 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.174 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-31T23:37:05.174 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.174 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.174 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.174 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.174 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.174 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.175 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.175 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.175 INFO:teuthology.orchestra.run.smithi046.stdout: "orchestrator": { 2024-03-31T23:37:05.175 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "orchestrator", 2024-03-31T23:37:05.175 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.175 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.175 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.175 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-31T23:37:05.175 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.175 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.175 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-31T23:37:05.175 INFO:teuthology.orchestra.run.smithi046.stdout: "cephadm", 2024-03-31T23:37:05.175 INFO:teuthology.orchestra.run.smithi046.stdout: "rook", 2024-03-31T23:37:05.175 INFO:teuthology.orchestra.run.smithi046.stdout: "test_orchestrator" 2024-03-31T23:37:05.175 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-31T23:37:05.176 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Orchestrator backend", 2024-03-31T23:37:05.176 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.176 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.176 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.176 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-31T23:37:05.176 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-31T23:37:05.176 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.176 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-03-31T23:37:05.176 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "osd_perf_query", 2024-03-31T23:37:05.176 INFO:teuthology.orchestra.run.smithi046.stdout: "can_run": true, 2024-03-31T23:37:05.176 INFO:teuthology.orchestra.run.smithi046.stdout: "error_string": "", 2024-03-31T23:37:05.176 INFO:teuthology.orchestra.run.smithi046.stdout: "module_options": { 2024-03-31T23:37:05.176 INFO:teuthology.orchestra.run.smithi046.stdout: "log_level": { 2024-03-31T23:37:05.176 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_level", 2024-03-31T23:37:05.176 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.177 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.177 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.177 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-31T23:37:05.177 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.177 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.177 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-31T23:37:05.177 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-31T23:37:05.177 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-31T23:37:05.177 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-31T23:37:05.177 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-31T23:37:05.177 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-31T23:37:05.177 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-31T23:37:05.177 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-31T23:37:05.178 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.178 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.178 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.178 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.178 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.178 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster": { 2024-03-31T23:37:05.178 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster", 2024-03-31T23:37:05.178 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.178 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.178 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.178 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-31T23:37:05.178 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.178 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.178 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.179 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.179 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.179 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.179 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.179 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.179 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster_level": { 2024-03-31T23:37:05.179 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster_level", 2024-03-31T23:37:05.179 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.179 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.179 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.179 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "info", 2024-03-31T23:37:05.179 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.179 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.179 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-31T23:37:05.179 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-31T23:37:05.180 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-31T23:37:05.180 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-31T23:37:05.180 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-31T23:37:05.180 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-31T23:37:05.180 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-31T23:37:05.180 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-31T23:37:05.180 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.180 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.180 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.180 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.180 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.180 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_file": { 2024-03-31T23:37:05.180 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_file", 2024-03-31T23:37:05.180 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.181 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.181 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.181 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-31T23:37:05.181 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.181 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.181 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.181 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.181 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.181 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.181 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.181 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-31T23:37:05.181 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-31T23:37:05.181 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.181 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-03-31T23:37:05.181 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "osd_support", 2024-03-31T23:37:05.182 INFO:teuthology.orchestra.run.smithi046.stdout: "can_run": true, 2024-03-31T23:37:05.182 INFO:teuthology.orchestra.run.smithi046.stdout: "error_string": "", 2024-03-31T23:37:05.182 INFO:teuthology.orchestra.run.smithi046.stdout: "module_options": { 2024-03-31T23:37:05.182 INFO:teuthology.orchestra.run.smithi046.stdout: "log_level": { 2024-03-31T23:37:05.182 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_level", 2024-03-31T23:37:05.182 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.182 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.182 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.182 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-31T23:37:05.182 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.182 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.182 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-31T23:37:05.182 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-31T23:37:05.182 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-31T23:37:05.183 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-31T23:37:05.183 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-31T23:37:05.183 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-31T23:37:05.183 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-31T23:37:05.183 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-31T23:37:05.183 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.183 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.183 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.183 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.183 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.183 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster": { 2024-03-31T23:37:05.183 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster", 2024-03-31T23:37:05.183 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.183 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.184 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.184 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-31T23:37:05.184 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.184 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.184 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.184 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.184 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.184 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.184 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.184 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.184 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster_level": { 2024-03-31T23:37:05.184 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster_level", 2024-03-31T23:37:05.184 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.184 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.185 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.185 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "info", 2024-03-31T23:37:05.185 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.185 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.185 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-31T23:37:05.185 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-31T23:37:05.185 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-31T23:37:05.185 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-31T23:37:05.185 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-31T23:37:05.185 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-31T23:37:05.185 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-31T23:37:05.185 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-31T23:37:05.185 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.185 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.185 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.186 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.186 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.186 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_file": { 2024-03-31T23:37:05.186 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_file", 2024-03-31T23:37:05.186 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.186 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.186 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.186 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-31T23:37:05.186 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.186 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.186 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.186 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.186 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.187 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.187 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.187 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-31T23:37:05.187 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-31T23:37:05.187 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.187 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-03-31T23:37:05.187 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "pg_autoscaler", 2024-03-31T23:37:05.187 INFO:teuthology.orchestra.run.smithi046.stdout: "can_run": true, 2024-03-31T23:37:05.187 INFO:teuthology.orchestra.run.smithi046.stdout: "error_string": "", 2024-03-31T23:37:05.187 INFO:teuthology.orchestra.run.smithi046.stdout: "module_options": { 2024-03-31T23:37:05.187 INFO:teuthology.orchestra.run.smithi046.stdout: "log_level": { 2024-03-31T23:37:05.187 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_level", 2024-03-31T23:37:05.187 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.187 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.188 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.188 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-31T23:37:05.188 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.188 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.188 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-31T23:37:05.188 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-31T23:37:05.188 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-31T23:37:05.188 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-31T23:37:05.188 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-31T23:37:05.188 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-31T23:37:05.188 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-31T23:37:05.188 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-31T23:37:05.188 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.188 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.188 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.189 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.189 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.189 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster": { 2024-03-31T23:37:05.189 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster", 2024-03-31T23:37:05.189 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.189 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.189 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.189 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-31T23:37:05.189 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.189 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.189 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.189 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.189 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.189 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.190 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.190 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.190 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster_level": { 2024-03-31T23:37:05.190 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster_level", 2024-03-31T23:37:05.190 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.190 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.190 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.190 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "info", 2024-03-31T23:37:05.190 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.190 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.190 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-31T23:37:05.190 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-31T23:37:05.190 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-31T23:37:05.190 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-31T23:37:05.190 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-31T23:37:05.191 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-31T23:37:05.191 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-31T23:37:05.191 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-31T23:37:05.191 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.191 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.191 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.191 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.191 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.191 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_file": { 2024-03-31T23:37:05.191 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_file", 2024-03-31T23:37:05.191 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.191 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.191 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.192 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-31T23:37:05.192 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.192 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.192 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.192 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.192 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.192 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.192 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.192 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.192 INFO:teuthology.orchestra.run.smithi046.stdout: "sleep_interval": { 2024-03-31T23:37:05.192 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "sleep_interval", 2024-03-31T23:37:05.192 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "secs", 2024-03-31T23:37:05.192 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.192 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.192 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "60", 2024-03-31T23:37:05.193 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.193 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.193 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.193 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.193 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.193 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.193 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.193 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.193 INFO:teuthology.orchestra.run.smithi046.stdout: "threshold": { 2024-03-31T23:37:05.193 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "threshold", 2024-03-31T23:37:05.193 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "float", 2024-03-31T23:37:05.193 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.193 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.193 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "3.0", 2024-03-31T23:37:05.193 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "1.0", 2024-03-31T23:37:05.194 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.194 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.194 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "scaling threshold", 2024-03-31T23:37:05.194 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-03-31T23:37:05.194 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.194 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.194 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-31T23:37:05.194 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-31T23:37:05.194 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.194 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-03-31T23:37:05.194 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "progress", 2024-03-31T23:37:05.194 INFO:teuthology.orchestra.run.smithi046.stdout: "can_run": true, 2024-03-31T23:37:05.194 INFO:teuthology.orchestra.run.smithi046.stdout: "error_string": "", 2024-03-31T23:37:05.194 INFO:teuthology.orchestra.run.smithi046.stdout: "module_options": { 2024-03-31T23:37:05.195 INFO:teuthology.orchestra.run.smithi046.stdout: "allow_pg_recovery_event": { 2024-03-31T23:37:05.195 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "allow_pg_recovery_event", 2024-03-31T23:37:05.195 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.195 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.195 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.195 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-31T23:37:05.195 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.195 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.195 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.195 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "allow the module to show pg recovery progress", 2024-03-31T23:37:05.195 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.195 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.195 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.195 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.195 INFO:teuthology.orchestra.run.smithi046.stdout: "enabled": { 2024-03-31T23:37:05.196 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "enabled", 2024-03-31T23:37:05.196 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.196 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.196 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.196 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "True", 2024-03-31T23:37:05.196 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.196 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.196 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.196 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.196 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.196 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.196 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.196 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.196 INFO:teuthology.orchestra.run.smithi046.stdout: "log_level": { 2024-03-31T23:37:05.197 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_level", 2024-03-31T23:37:05.197 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.197 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.197 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.197 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-31T23:37:05.197 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.197 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.197 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-31T23:37:05.197 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-31T23:37:05.197 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-31T23:37:05.197 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-31T23:37:05.197 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-31T23:37:05.197 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-31T23:37:05.197 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-31T23:37:05.197 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-31T23:37:05.198 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.198 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.198 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.198 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.198 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.198 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster": { 2024-03-31T23:37:05.198 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster", 2024-03-31T23:37:05.198 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.198 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.198 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.198 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-31T23:37:05.198 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.198 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.198 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.198 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.199 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.199 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.199 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.199 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.199 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster_level": { 2024-03-31T23:37:05.199 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster_level", 2024-03-31T23:37:05.199 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.199 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.199 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.199 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "info", 2024-03-31T23:37:05.199 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.199 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.199 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-31T23:37:05.199 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-31T23:37:05.199 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-31T23:37:05.200 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-31T23:37:05.200 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-31T23:37:05.200 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-31T23:37:05.200 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-31T23:37:05.200 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-31T23:37:05.200 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.200 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.200 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.200 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.200 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.200 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_file": { 2024-03-31T23:37:05.200 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_file", 2024-03-31T23:37:05.200 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.200 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.200 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.201 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-31T23:37:05.201 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.201 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.201 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.201 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.201 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.201 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.201 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.201 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.201 INFO:teuthology.orchestra.run.smithi046.stdout: "max_completed_events": { 2024-03-31T23:37:05.201 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "max_completed_events", 2024-03-31T23:37:05.201 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "int", 2024-03-31T23:37:05.201 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.201 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.201 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "50", 2024-03-31T23:37:05.202 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.202 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.202 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.202 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "number of past completed events to remember", 2024-03-31T23:37:05.202 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.202 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.202 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.202 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.202 INFO:teuthology.orchestra.run.smithi046.stdout: "sleep_interval": { 2024-03-31T23:37:05.202 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "sleep_interval", 2024-03-31T23:37:05.202 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "secs", 2024-03-31T23:37:05.202 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.202 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.202 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "5", 2024-03-31T23:37:05.202 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.203 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.203 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.203 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "how long the module is going to sleep", 2024-03-31T23:37:05.203 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.203 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.203 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.203 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-31T23:37:05.203 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-31T23:37:05.203 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.203 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-03-31T23:37:05.203 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "prometheus", 2024-03-31T23:37:05.203 INFO:teuthology.orchestra.run.smithi046.stdout: "can_run": true, 2024-03-31T23:37:05.203 INFO:teuthology.orchestra.run.smithi046.stdout: "error_string": "", 2024-03-31T23:37:05.203 INFO:teuthology.orchestra.run.smithi046.stdout: "module_options": { 2024-03-31T23:37:05.203 INFO:teuthology.orchestra.run.smithi046.stdout: "cache": { 2024-03-31T23:37:05.204 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "cache", 2024-03-31T23:37:05.204 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.204 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.204 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.204 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "True", 2024-03-31T23:37:05.204 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.204 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.204 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.204 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.204 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.204 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.204 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.204 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.205 INFO:teuthology.orchestra.run.smithi046.stdout: "exclude_perf_counters": { 2024-03-31T23:37:05.205 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "exclude_perf_counters", 2024-03-31T23:37:05.205 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.205 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.205 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.205 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "True", 2024-03-31T23:37:05.205 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.205 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.205 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.205 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-03-31T23:37:05.205 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-03-31T23:37:05.205 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.205 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.205 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.206 INFO:teuthology.orchestra.run.smithi046.stdout: "log_level": { 2024-03-31T23:37:05.206 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_level", 2024-03-31T23:37:05.206 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.206 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.206 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.206 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-31T23:37:05.206 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.206 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.206 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-31T23:37:05.206 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-31T23:37:05.206 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-31T23:37:05.206 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-31T23:37:05.206 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-31T23:37:05.206 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-31T23:37:05.207 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-31T23:37:05.207 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-31T23:37:05.207 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.207 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.207 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.207 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.207 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.207 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster": { 2024-03-31T23:37:05.207 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster", 2024-03-31T23:37:05.207 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.207 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.207 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.207 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-31T23:37:05.207 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.208 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.208 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.208 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.208 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.208 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.208 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.208 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.208 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster_level": { 2024-03-31T23:37:05.208 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster_level", 2024-03-31T23:37:05.208 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.208 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.208 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.208 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "info", 2024-03-31T23:37:05.208 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.208 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.209 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-31T23:37:05.209 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-31T23:37:05.209 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-31T23:37:05.209 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-31T23:37:05.209 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-31T23:37:05.209 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-31T23:37:05.209 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-31T23:37:05.209 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-31T23:37:05.209 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.209 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.209 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.209 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.209 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.209 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_file": { 2024-03-31T23:37:05.209 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_file", 2024-03-31T23:37:05.210 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.210 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.210 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.210 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-31T23:37:05.210 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.210 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.210 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.210 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.210 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.210 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.210 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.210 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.210 INFO:teuthology.orchestra.run.smithi046.stdout: "rbd_stats_pools": { 2024-03-31T23:37:05.210 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "rbd_stats_pools", 2024-03-31T23:37:05.210 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.211 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.211 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.211 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-31T23:37:05.211 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.211 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.211 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.211 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.211 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.211 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.211 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.211 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.211 INFO:teuthology.orchestra.run.smithi046.stdout: "rbd_stats_pools_refresh_interval": { 2024-03-31T23:37:05.211 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-03-31T23:37:05.211 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "int", 2024-03-31T23:37:05.211 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.212 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.212 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "300", 2024-03-31T23:37:05.212 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.212 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.212 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.212 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.212 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.212 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.212 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.212 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.212 INFO:teuthology.orchestra.run.smithi046.stdout: "scrape_interval": { 2024-03-31T23:37:05.212 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "scrape_interval", 2024-03-31T23:37:05.212 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "float", 2024-03-31T23:37:05.212 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.212 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.213 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "15.0", 2024-03-31T23:37:05.213 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.213 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.213 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.213 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.213 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.213 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.213 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.213 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.213 INFO:teuthology.orchestra.run.smithi046.stdout: "server_addr": { 2024-03-31T23:37:05.213 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "server_addr", 2024-03-31T23:37:05.213 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.213 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.213 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.213 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "::", 2024-03-31T23:37:05.214 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.214 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.214 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.214 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-03-31T23:37:05.214 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.214 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.214 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.214 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.214 INFO:teuthology.orchestra.run.smithi046.stdout: "server_port": { 2024-03-31T23:37:05.214 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "server_port", 2024-03-31T23:37:05.214 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "int", 2024-03-31T23:37:05.214 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.214 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.214 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "9283", 2024-03-31T23:37:05.215 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.215 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.215 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.215 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-03-31T23:37:05.215 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.215 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.215 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.215 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.215 INFO:teuthology.orchestra.run.smithi046.stdout: "stale_cache_strategy": { 2024-03-31T23:37:05.215 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "stale_cache_strategy", 2024-03-31T23:37:05.215 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.215 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.215 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.215 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "log", 2024-03-31T23:37:05.215 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.216 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.216 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.216 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.216 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.216 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.216 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.216 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.216 INFO:teuthology.orchestra.run.smithi046.stdout: "standby_behaviour": { 2024-03-31T23:37:05.216 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "standby_behaviour", 2024-03-31T23:37:05.216 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.216 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.216 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.216 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "default", 2024-03-31T23:37:05.216 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.217 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.217 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-31T23:37:05.217 INFO:teuthology.orchestra.run.smithi046.stdout: "default", 2024-03-31T23:37:05.217 INFO:teuthology.orchestra.run.smithi046.stdout: "error" 2024-03-31T23:37:05.217 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-31T23:37:05.217 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.217 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.217 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.217 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.217 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.217 INFO:teuthology.orchestra.run.smithi046.stdout: "standby_error_status_code": { 2024-03-31T23:37:05.217 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "standby_error_status_code", 2024-03-31T23:37:05.217 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "int", 2024-03-31T23:37:05.217 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.217 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.218 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "500", 2024-03-31T23:37:05.218 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "400", 2024-03-31T23:37:05.218 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "599", 2024-03-31T23:37:05.218 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.218 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.218 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.218 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.218 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.218 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-31T23:37:05.218 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-31T23:37:05.218 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.218 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-03-31T23:37:05.218 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "rbd_support", 2024-03-31T23:37:05.218 INFO:teuthology.orchestra.run.smithi046.stdout: "can_run": true, 2024-03-31T23:37:05.218 INFO:teuthology.orchestra.run.smithi046.stdout: "error_string": "", 2024-03-31T23:37:05.219 INFO:teuthology.orchestra.run.smithi046.stdout: "module_options": { 2024-03-31T23:37:05.219 INFO:teuthology.orchestra.run.smithi046.stdout: "log_level": { 2024-03-31T23:37:05.219 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_level", 2024-03-31T23:37:05.219 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.219 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.219 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.219 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-31T23:37:05.219 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.219 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.219 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-31T23:37:05.219 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-31T23:37:05.219 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-31T23:37:05.219 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-31T23:37:05.219 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-31T23:37:05.220 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-31T23:37:05.220 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-31T23:37:05.220 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-31T23:37:05.220 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.220 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.220 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.220 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.220 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.220 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster": { 2024-03-31T23:37:05.220 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster", 2024-03-31T23:37:05.220 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.220 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.220 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.220 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-31T23:37:05.220 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.221 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.221 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.221 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.221 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.221 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.221 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.221 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.221 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster_level": { 2024-03-31T23:37:05.221 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster_level", 2024-03-31T23:37:05.221 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.221 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.221 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.221 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "info", 2024-03-31T23:37:05.221 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.221 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.222 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-31T23:37:05.222 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-31T23:37:05.222 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-31T23:37:05.222 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-31T23:37:05.222 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-31T23:37:05.222 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-31T23:37:05.222 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-31T23:37:05.222 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-31T23:37:05.222 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.222 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.222 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.222 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.222 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.222 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_file": { 2024-03-31T23:37:05.223 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_file", 2024-03-31T23:37:05.223 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.223 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.223 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.223 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-31T23:37:05.223 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.223 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.223 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.223 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.223 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.223 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.223 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.223 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.223 INFO:teuthology.orchestra.run.smithi046.stdout: "max_concurrent_snap_create": { 2024-03-31T23:37:05.223 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "max_concurrent_snap_create", 2024-03-31T23:37:05.224 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "int", 2024-03-31T23:37:05.224 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.224 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.224 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "10", 2024-03-31T23:37:05.224 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.224 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.224 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.224 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.224 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.224 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.224 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.224 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.224 INFO:teuthology.orchestra.run.smithi046.stdout: "mirror_snapshot_schedule": { 2024-03-31T23:37:05.224 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "mirror_snapshot_schedule", 2024-03-31T23:37:05.224 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.225 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.225 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.225 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-31T23:37:05.225 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.225 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.225 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.225 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.225 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.225 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.225 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.225 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.225 INFO:teuthology.orchestra.run.smithi046.stdout: "trash_purge_schedule": { 2024-03-31T23:37:05.225 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "trash_purge_schedule", 2024-03-31T23:37:05.225 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.225 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.226 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.226 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-31T23:37:05.226 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.226 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.226 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.226 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.226 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.226 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.226 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.226 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-31T23:37:05.226 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-31T23:37:05.226 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.226 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-03-31T23:37:05.226 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "restful", 2024-03-31T23:37:05.226 INFO:teuthology.orchestra.run.smithi046.stdout: "can_run": true, 2024-03-31T23:37:05.227 INFO:teuthology.orchestra.run.smithi046.stdout: "error_string": "", 2024-03-31T23:37:05.227 INFO:teuthology.orchestra.run.smithi046.stdout: "module_options": { 2024-03-31T23:37:05.227 INFO:teuthology.orchestra.run.smithi046.stdout: "enable_auth": { 2024-03-31T23:37:05.227 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "enable_auth", 2024-03-31T23:37:05.227 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.227 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.227 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.227 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "True", 2024-03-31T23:37:05.227 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.227 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.227 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.227 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.227 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.227 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.227 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.228 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.228 INFO:teuthology.orchestra.run.smithi046.stdout: "key_file": { 2024-03-31T23:37:05.228 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "key_file", 2024-03-31T23:37:05.228 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.228 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.228 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.228 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-31T23:37:05.228 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.228 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.228 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.228 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.228 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.228 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.228 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.228 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.229 INFO:teuthology.orchestra.run.smithi046.stdout: "log_level": { 2024-03-31T23:37:05.229 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_level", 2024-03-31T23:37:05.229 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.229 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.229 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.229 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-31T23:37:05.229 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.229 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.229 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-31T23:37:05.229 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-31T23:37:05.229 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-31T23:37:05.229 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-31T23:37:05.229 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-31T23:37:05.229 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-31T23:37:05.230 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-31T23:37:05.230 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-31T23:37:05.230 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.230 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.230 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.230 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.230 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.230 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster": { 2024-03-31T23:37:05.230 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster", 2024-03-31T23:37:05.230 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.230 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.231 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.231 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-31T23:37:05.231 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.231 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.231 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.231 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.231 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.231 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.231 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.231 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.231 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster_level": { 2024-03-31T23:37:05.231 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster_level", 2024-03-31T23:37:05.231 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.231 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.231 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.232 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "info", 2024-03-31T23:37:05.232 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.232 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.232 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-31T23:37:05.232 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-31T23:37:05.232 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-31T23:37:05.232 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-31T23:37:05.232 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-31T23:37:05.232 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-31T23:37:05.232 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-31T23:37:05.232 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-31T23:37:05.232 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.232 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.232 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.232 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.233 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.233 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_file": { 2024-03-31T23:37:05.233 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_file", 2024-03-31T23:37:05.233 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.233 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.233 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.233 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-31T23:37:05.233 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.233 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.233 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.233 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.233 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.233 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.233 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.233 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.234 INFO:teuthology.orchestra.run.smithi046.stdout: "server_addr": { 2024-03-31T23:37:05.234 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "server_addr", 2024-03-31T23:37:05.234 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.234 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.234 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.234 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-31T23:37:05.234 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.234 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.234 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.234 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.234 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.234 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.234 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.234 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.235 INFO:teuthology.orchestra.run.smithi046.stdout: "server_port": { 2024-03-31T23:37:05.235 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "server_port", 2024-03-31T23:37:05.235 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.235 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.235 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.235 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-31T23:37:05.235 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.235 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.235 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.235 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.235 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.235 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.235 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.235 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-31T23:37:05.236 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-31T23:37:05.236 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.236 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-03-31T23:37:05.236 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "rgw", 2024-03-31T23:37:05.236 INFO:teuthology.orchestra.run.smithi046.stdout: "can_run": true, 2024-03-31T23:37:05.236 INFO:teuthology.orchestra.run.smithi046.stdout: "error_string": "", 2024-03-31T23:37:05.236 INFO:teuthology.orchestra.run.smithi046.stdout: "module_options": { 2024-03-31T23:37:05.236 INFO:teuthology.orchestra.run.smithi046.stdout: "log_level": { 2024-03-31T23:37:05.236 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_level", 2024-03-31T23:37:05.236 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.236 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.236 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.236 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-31T23:37:05.236 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.237 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.237 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-31T23:37:05.237 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-31T23:37:05.237 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-31T23:37:05.237 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-31T23:37:05.237 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-31T23:37:05.237 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-31T23:37:05.237 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-31T23:37:05.237 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-31T23:37:05.237 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.237 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.237 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.237 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.237 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.237 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster": { 2024-03-31T23:37:05.238 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster", 2024-03-31T23:37:05.238 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.238 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.238 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.238 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-31T23:37:05.238 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.238 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.238 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.238 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.238 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.238 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.238 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.238 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.238 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster_level": { 2024-03-31T23:37:05.238 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster_level", 2024-03-31T23:37:05.239 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.239 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.239 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.239 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "info", 2024-03-31T23:37:05.239 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.239 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.239 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-31T23:37:05.239 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-31T23:37:05.239 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-31T23:37:05.239 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-31T23:37:05.239 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-31T23:37:05.239 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-31T23:37:05.239 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-31T23:37:05.239 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-31T23:37:05.239 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.240 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.240 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.240 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.240 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.240 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_file": { 2024-03-31T23:37:05.240 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_file", 2024-03-31T23:37:05.240 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.240 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.240 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.240 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-31T23:37:05.240 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.240 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.240 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.240 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.240 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.241 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.241 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.241 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-31T23:37:05.241 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-31T23:37:05.241 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.241 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-03-31T23:37:05.241 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "rook", 2024-03-31T23:37:05.241 INFO:teuthology.orchestra.run.smithi046.stdout: "can_run": true, 2024-03-31T23:37:05.241 INFO:teuthology.orchestra.run.smithi046.stdout: "error_string": "", 2024-03-31T23:37:05.241 INFO:teuthology.orchestra.run.smithi046.stdout: "module_options": { 2024-03-31T23:37:05.241 INFO:teuthology.orchestra.run.smithi046.stdout: "log_level": { 2024-03-31T23:37:05.241 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_level", 2024-03-31T23:37:05.241 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.241 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.242 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.242 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-31T23:37:05.242 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.242 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.242 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-31T23:37:05.242 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-31T23:37:05.242 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-31T23:37:05.242 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-31T23:37:05.242 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-31T23:37:05.242 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-31T23:37:05.242 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-31T23:37:05.242 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-31T23:37:05.242 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.242 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.243 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.243 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.243 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.243 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster": { 2024-03-31T23:37:05.243 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster", 2024-03-31T23:37:05.243 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.243 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.243 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.243 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-31T23:37:05.243 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.243 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.243 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.243 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.243 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.244 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.244 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.244 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.244 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster_level": { 2024-03-31T23:37:05.244 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster_level", 2024-03-31T23:37:05.244 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.244 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.244 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.244 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "info", 2024-03-31T23:37:05.244 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.244 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.244 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-31T23:37:05.244 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-31T23:37:05.244 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-31T23:37:05.245 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-31T23:37:05.245 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-31T23:37:05.245 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-31T23:37:05.245 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-31T23:37:05.245 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-31T23:37:05.245 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.245 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.245 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.245 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.245 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.245 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_file": { 2024-03-31T23:37:05.245 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_file", 2024-03-31T23:37:05.245 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.245 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.246 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.246 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-31T23:37:05.246 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.246 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.246 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.246 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.246 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.246 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.246 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.246 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.246 INFO:teuthology.orchestra.run.smithi046.stdout: "storage_class": { 2024-03-31T23:37:05.246 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "storage_class", 2024-03-31T23:37:05.246 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.246 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.247 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.247 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "local", 2024-03-31T23:37:05.247 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.247 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.247 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.247 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-03-31T23:37:05.247 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.247 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.247 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.247 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-31T23:37:05.247 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-31T23:37:05.247 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.247 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-03-31T23:37:05.247 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "selftest", 2024-03-31T23:37:05.248 INFO:teuthology.orchestra.run.smithi046.stdout: "can_run": true, 2024-03-31T23:37:05.248 INFO:teuthology.orchestra.run.smithi046.stdout: "error_string": "", 2024-03-31T23:37:05.248 INFO:teuthology.orchestra.run.smithi046.stdout: "module_options": { 2024-03-31T23:37:05.248 INFO:teuthology.orchestra.run.smithi046.stdout: "log_level": { 2024-03-31T23:37:05.248 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_level", 2024-03-31T23:37:05.248 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.248 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.248 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.248 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-31T23:37:05.248 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.248 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.248 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-31T23:37:05.248 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-31T23:37:05.248 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-31T23:37:05.248 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-31T23:37:05.249 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-31T23:37:05.249 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-31T23:37:05.249 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-31T23:37:05.249 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-31T23:37:05.249 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.249 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.249 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.249 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.249 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.249 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster": { 2024-03-31T23:37:05.249 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster", 2024-03-31T23:37:05.249 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.249 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.249 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.249 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-31T23:37:05.250 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.250 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.250 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.250 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.250 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.250 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.250 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.250 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.250 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster_level": { 2024-03-31T23:37:05.250 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster_level", 2024-03-31T23:37:05.250 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.250 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.250 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.250 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "info", 2024-03-31T23:37:05.250 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.251 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.251 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-31T23:37:05.251 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-31T23:37:05.251 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-31T23:37:05.251 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-31T23:37:05.251 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-31T23:37:05.251 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-31T23:37:05.251 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-31T23:37:05.251 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-31T23:37:05.251 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.251 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.251 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.251 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.251 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.251 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_file": { 2024-03-31T23:37:05.252 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_file", 2024-03-31T23:37:05.252 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.252 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.252 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.252 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-31T23:37:05.252 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.252 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.252 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.252 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.252 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.252 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.252 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.252 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.252 INFO:teuthology.orchestra.run.smithi046.stdout: "roption1": { 2024-03-31T23:37:05.253 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "roption1", 2024-03-31T23:37:05.253 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.253 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.253 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.253 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-31T23:37:05.253 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.253 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.253 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.253 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.253 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.253 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.253 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.253 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.253 INFO:teuthology.orchestra.run.smithi046.stdout: "roption2": { 2024-03-31T23:37:05.253 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "roption2", 2024-03-31T23:37:05.253 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.254 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.254 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.254 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "xyz", 2024-03-31T23:37:05.254 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.254 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.254 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.254 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.254 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.254 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.254 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.254 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.254 INFO:teuthology.orchestra.run.smithi046.stdout: "rwoption1": { 2024-03-31T23:37:05.254 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "rwoption1", 2024-03-31T23:37:05.254 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.255 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.255 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.255 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-31T23:37:05.255 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.255 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.255 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.255 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.255 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.255 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.255 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.255 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.255 INFO:teuthology.orchestra.run.smithi046.stdout: "rwoption2": { 2024-03-31T23:37:05.255 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "rwoption2", 2024-03-31T23:37:05.255 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "int", 2024-03-31T23:37:05.255 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.256 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.256 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-31T23:37:05.256 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.256 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.256 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.256 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.256 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.256 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.256 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.256 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.256 INFO:teuthology.orchestra.run.smithi046.stdout: "rwoption3": { 2024-03-31T23:37:05.256 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "rwoption3", 2024-03-31T23:37:05.256 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "float", 2024-03-31T23:37:05.256 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.256 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.257 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-31T23:37:05.257 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.257 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.257 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.257 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.257 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.257 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.257 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.257 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.257 INFO:teuthology.orchestra.run.smithi046.stdout: "rwoption4": { 2024-03-31T23:37:05.257 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "rwoption4", 2024-03-31T23:37:05.257 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.257 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.258 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.258 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-31T23:37:05.258 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.258 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.258 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.258 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.258 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.258 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.258 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.258 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.258 INFO:teuthology.orchestra.run.smithi046.stdout: "rwoption5": { 2024-03-31T23:37:05.258 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "rwoption5", 2024-03-31T23:37:05.258 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.258 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.258 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.259 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-31T23:37:05.259 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.259 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.259 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.259 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.259 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.259 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.259 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.259 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.259 INFO:teuthology.orchestra.run.smithi046.stdout: "rwoption6": { 2024-03-31T23:37:05.259 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "rwoption6", 2024-03-31T23:37:05.259 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.259 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.259 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.260 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "True", 2024-03-31T23:37:05.260 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.260 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.260 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.260 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.260 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.260 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.260 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.260 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.260 INFO:teuthology.orchestra.run.smithi046.stdout: "rwoption7": { 2024-03-31T23:37:05.260 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "rwoption7", 2024-03-31T23:37:05.260 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "int", 2024-03-31T23:37:05.260 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.260 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.260 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-31T23:37:05.261 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "1", 2024-03-31T23:37:05.261 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "42", 2024-03-31T23:37:05.261 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.261 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.261 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.261 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.261 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.261 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.261 INFO:teuthology.orchestra.run.smithi046.stdout: "testkey": { 2024-03-31T23:37:05.261 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "testkey", 2024-03-31T23:37:05.261 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.261 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.261 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.261 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-31T23:37:05.261 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.262 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.262 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.262 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.262 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.262 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.262 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.262 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.262 INFO:teuthology.orchestra.run.smithi046.stdout: "testlkey": { 2024-03-31T23:37:05.262 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "testlkey", 2024-03-31T23:37:05.262 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.262 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.262 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.262 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-31T23:37:05.262 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.263 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.263 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.263 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.263 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.263 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.263 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.263 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.263 INFO:teuthology.orchestra.run.smithi046.stdout: "testnewline": { 2024-03-31T23:37:05.263 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "testnewline", 2024-03-31T23:37:05.263 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.263 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.263 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.263 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-31T23:37:05.263 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.264 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.264 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.264 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.264 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.264 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.264 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.264 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-31T23:37:05.264 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-31T23:37:05.264 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.264 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-03-31T23:37:05.264 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "snap_schedule", 2024-03-31T23:37:05.264 INFO:teuthology.orchestra.run.smithi046.stdout: "can_run": true, 2024-03-31T23:37:05.264 INFO:teuthology.orchestra.run.smithi046.stdout: "error_string": "", 2024-03-31T23:37:05.264 INFO:teuthology.orchestra.run.smithi046.stdout: "module_options": { 2024-03-31T23:37:05.264 INFO:teuthology.orchestra.run.smithi046.stdout: "allow_m_granularity": { 2024-03-31T23:37:05.265 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "allow_m_granularity", 2024-03-31T23:37:05.265 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.265 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.265 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.265 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-31T23:37:05.265 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.265 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.265 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.265 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "allow minute scheduled snapshots", 2024-03-31T23:37:05.265 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.265 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.265 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.265 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.265 INFO:teuthology.orchestra.run.smithi046.stdout: "dump_on_update": { 2024-03-31T23:37:05.265 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "dump_on_update", 2024-03-31T23:37:05.266 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.266 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.266 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.266 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-31T23:37:05.266 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.266 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.266 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.266 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "dump database to debug log on update", 2024-03-31T23:37:05.266 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.266 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.266 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.266 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.266 INFO:teuthology.orchestra.run.smithi046.stdout: "log_level": { 2024-03-31T23:37:05.266 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_level", 2024-03-31T23:37:05.267 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.267 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.267 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.267 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-31T23:37:05.267 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.267 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.267 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-31T23:37:05.267 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-31T23:37:05.267 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-31T23:37:05.267 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-31T23:37:05.267 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-31T23:37:05.267 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-31T23:37:05.267 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-31T23:37:05.267 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-31T23:37:05.267 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.268 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.268 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.268 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.268 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.268 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster": { 2024-03-31T23:37:05.268 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster", 2024-03-31T23:37:05.268 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.268 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.268 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.268 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-31T23:37:05.268 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.268 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.268 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.268 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.268 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.269 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.269 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.269 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.269 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster_level": { 2024-03-31T23:37:05.269 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster_level", 2024-03-31T23:37:05.269 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.269 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.269 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.269 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "info", 2024-03-31T23:37:05.269 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.269 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.269 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-31T23:37:05.269 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-31T23:37:05.269 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-31T23:37:05.270 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-31T23:37:05.270 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-31T23:37:05.270 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-31T23:37:05.270 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-31T23:37:05.270 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-31T23:37:05.270 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.270 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.270 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.270 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.270 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.270 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_file": { 2024-03-31T23:37:05.270 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_file", 2024-03-31T23:37:05.270 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.271 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.271 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.271 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-31T23:37:05.271 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.271 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.271 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.271 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.271 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.271 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.271 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.271 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-31T23:37:05.271 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-31T23:37:05.271 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.271 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-03-31T23:37:05.272 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "stats", 2024-03-31T23:37:05.272 INFO:teuthology.orchestra.run.smithi046.stdout: "can_run": true, 2024-03-31T23:37:05.272 INFO:teuthology.orchestra.run.smithi046.stdout: "error_string": "", 2024-03-31T23:37:05.272 INFO:teuthology.orchestra.run.smithi046.stdout: "module_options": { 2024-03-31T23:37:05.272 INFO:teuthology.orchestra.run.smithi046.stdout: "log_level": { 2024-03-31T23:37:05.272 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_level", 2024-03-31T23:37:05.272 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.272 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.272 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.272 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-31T23:37:05.272 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.272 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.272 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-31T23:37:05.272 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-31T23:37:05.273 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-31T23:37:05.273 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-31T23:37:05.273 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-31T23:37:05.273 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-31T23:37:05.273 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-31T23:37:05.273 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-31T23:37:05.273 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.273 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.273 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.273 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.273 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.273 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster": { 2024-03-31T23:37:05.273 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster", 2024-03-31T23:37:05.273 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.274 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.274 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.274 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-31T23:37:05.274 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.274 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.274 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.274 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.274 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.274 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.274 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.274 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.274 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster_level": { 2024-03-31T23:37:05.274 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster_level", 2024-03-31T23:37:05.274 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.275 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.275 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.275 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "info", 2024-03-31T23:37:05.275 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.275 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.275 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-31T23:37:05.275 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-31T23:37:05.275 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-31T23:37:05.275 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-31T23:37:05.275 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-31T23:37:05.275 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-31T23:37:05.275 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-31T23:37:05.275 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-31T23:37:05.275 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.276 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.276 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.276 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.276 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.276 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_file": { 2024-03-31T23:37:05.276 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_file", 2024-03-31T23:37:05.276 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.276 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.276 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.276 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-31T23:37:05.276 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.276 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.276 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.276 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.277 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.277 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.277 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.277 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-31T23:37:05.277 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-31T23:37:05.277 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.277 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-03-31T23:37:05.277 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "status", 2024-03-31T23:37:05.277 INFO:teuthology.orchestra.run.smithi046.stdout: "can_run": true, 2024-03-31T23:37:05.277 INFO:teuthology.orchestra.run.smithi046.stdout: "error_string": "", 2024-03-31T23:37:05.277 INFO:teuthology.orchestra.run.smithi046.stdout: "module_options": { 2024-03-31T23:37:05.277 INFO:teuthology.orchestra.run.smithi046.stdout: "log_level": { 2024-03-31T23:37:05.277 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_level", 2024-03-31T23:37:05.277 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.278 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.278 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.278 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-31T23:37:05.278 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.278 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.278 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-31T23:37:05.278 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-31T23:37:05.278 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-31T23:37:05.278 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-31T23:37:05.278 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-31T23:37:05.278 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-31T23:37:05.278 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-31T23:37:05.278 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-31T23:37:05.278 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.279 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.279 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.279 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.279 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.279 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster": { 2024-03-31T23:37:05.279 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster", 2024-03-31T23:37:05.279 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.279 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.279 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.279 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-31T23:37:05.279 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.279 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.279 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.279 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.280 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.280 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.280 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.280 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.280 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster_level": { 2024-03-31T23:37:05.280 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster_level", 2024-03-31T23:37:05.280 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.280 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.280 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.280 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "info", 2024-03-31T23:37:05.280 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.280 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.280 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-31T23:37:05.280 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-31T23:37:05.280 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-31T23:37:05.281 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-31T23:37:05.281 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-31T23:37:05.281 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-31T23:37:05.281 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-31T23:37:05.281 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-31T23:37:05.281 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.281 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.281 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.281 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.281 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.281 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_file": { 2024-03-31T23:37:05.281 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_file", 2024-03-31T23:37:05.281 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.281 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.281 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.282 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-31T23:37:05.282 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.282 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.282 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.282 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.282 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.282 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.282 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.282 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-31T23:37:05.282 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-31T23:37:05.282 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.282 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-03-31T23:37:05.282 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "telegraf", 2024-03-31T23:37:05.283 INFO:teuthology.orchestra.run.smithi046.stdout: "can_run": true, 2024-03-31T23:37:05.283 INFO:teuthology.orchestra.run.smithi046.stdout: "error_string": "", 2024-03-31T23:37:05.283 INFO:teuthology.orchestra.run.smithi046.stdout: "module_options": { 2024-03-31T23:37:05.283 INFO:teuthology.orchestra.run.smithi046.stdout: "address": { 2024-03-31T23:37:05.283 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "address", 2024-03-31T23:37:05.283 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.283 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.283 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.283 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-03-31T23:37:05.283 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.283 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.283 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.283 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.283 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.283 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.284 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.284 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.284 INFO:teuthology.orchestra.run.smithi046.stdout: "interval": { 2024-03-31T23:37:05.284 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "interval", 2024-03-31T23:37:05.284 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "secs", 2024-03-31T23:37:05.284 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.284 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.284 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "15", 2024-03-31T23:37:05.284 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.284 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.284 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.284 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.284 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.285 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.285 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.285 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.285 INFO:teuthology.orchestra.run.smithi046.stdout: "log_level": { 2024-03-31T23:37:05.285 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_level", 2024-03-31T23:37:05.285 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.285 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.285 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.285 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-31T23:37:05.285 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.285 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.285 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-31T23:37:05.285 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-31T23:37:05.285 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-31T23:37:05.285 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-31T23:37:05.286 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-31T23:37:05.286 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-31T23:37:05.286 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-31T23:37:05.286 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-31T23:37:05.286 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.286 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.286 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.286 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.286 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.286 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster": { 2024-03-31T23:37:05.286 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster", 2024-03-31T23:37:05.286 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.286 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.286 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.287 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-31T23:37:05.287 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.287 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.287 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.287 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.287 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.287 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.287 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.287 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.287 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster_level": { 2024-03-31T23:37:05.287 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster_level", 2024-03-31T23:37:05.287 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.287 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.287 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.287 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "info", 2024-03-31T23:37:05.288 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.288 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.288 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-31T23:37:05.288 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-31T23:37:05.288 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-31T23:37:05.288 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-31T23:37:05.288 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-31T23:37:05.288 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-31T23:37:05.288 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-31T23:37:05.288 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-31T23:37:05.288 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.288 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.288 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.288 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.289 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.289 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_file": { 2024-03-31T23:37:05.289 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_file", 2024-03-31T23:37:05.289 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.289 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.289 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.289 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-31T23:37:05.289 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.289 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.289 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.289 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.289 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.289 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.289 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.289 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-31T23:37:05.290 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-31T23:37:05.290 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.290 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-03-31T23:37:05.290 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "telemetry", 2024-03-31T23:37:05.290 INFO:teuthology.orchestra.run.smithi046.stdout: "can_run": true, 2024-03-31T23:37:05.290 INFO:teuthology.orchestra.run.smithi046.stdout: "error_string": "", 2024-03-31T23:37:05.290 INFO:teuthology.orchestra.run.smithi046.stdout: "module_options": { 2024-03-31T23:37:05.290 INFO:teuthology.orchestra.run.smithi046.stdout: "channel_basic": { 2024-03-31T23:37:05.290 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "channel_basic", 2024-03-31T23:37:05.290 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.290 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.290 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.290 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "True", 2024-03-31T23:37:05.290 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.290 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.291 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.291 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Share basic cluster information (size, version)", 2024-03-31T23:37:05.291 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.291 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.291 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.291 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.291 INFO:teuthology.orchestra.run.smithi046.stdout: "channel_crash": { 2024-03-31T23:37:05.291 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "channel_crash", 2024-03-31T23:37:05.291 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.291 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.291 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.291 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "True", 2024-03-31T23:37:05.291 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.291 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.291 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.292 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-03-31T23:37:05.292 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.292 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.292 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.292 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.292 INFO:teuthology.orchestra.run.smithi046.stdout: "channel_device": { 2024-03-31T23:37:05.292 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "channel_device", 2024-03-31T23:37:05.292 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.292 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.292 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.292 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "True", 2024-03-31T23:37:05.292 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.292 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.292 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.293 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-03-31T23:37:05.293 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.293 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.293 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.293 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.293 INFO:teuthology.orchestra.run.smithi046.stdout: "channel_ident": { 2024-03-31T23:37:05.293 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "channel_ident", 2024-03-31T23:37:05.293 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.293 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.293 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.293 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-31T23:37:05.293 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.293 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.293 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.293 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-03-31T23:37:05.294 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.294 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.294 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.294 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.294 INFO:teuthology.orchestra.run.smithi046.stdout: "channel_perf": { 2024-03-31T23:37:05.294 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "channel_perf", 2024-03-31T23:37:05.294 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.294 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.294 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.294 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-31T23:37:05.294 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.294 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.294 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.294 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Share various performance metrics of a cluster", 2024-03-31T23:37:05.295 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.295 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.295 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.295 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.295 INFO:teuthology.orchestra.run.smithi046.stdout: "contact": { 2024-03-31T23:37:05.295 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "contact", 2024-03-31T23:37:05.295 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.295 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.295 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.295 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-31T23:37:05.295 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.295 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.295 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.295 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.295 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.296 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.296 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.296 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.296 INFO:teuthology.orchestra.run.smithi046.stdout: "description": { 2024-03-31T23:37:05.296 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "description", 2024-03-31T23:37:05.296 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.296 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.296 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.296 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-31T23:37:05.296 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.296 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.296 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.296 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.296 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.296 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.297 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.297 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.297 INFO:teuthology.orchestra.run.smithi046.stdout: "device_url": { 2024-03-31T23:37:05.297 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "device_url", 2024-03-31T23:37:05.297 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.297 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.297 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.297 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-03-31T23:37:05.297 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.297 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.297 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.297 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.297 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.297 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.298 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.298 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.298 INFO:teuthology.orchestra.run.smithi046.stdout: "enabled": { 2024-03-31T23:37:05.298 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "enabled", 2024-03-31T23:37:05.298 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.298 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.298 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.298 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-31T23:37:05.298 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.298 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.298 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.298 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.298 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.298 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.299 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.299 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.299 INFO:teuthology.orchestra.run.smithi046.stdout: "interval": { 2024-03-31T23:37:05.299 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "interval", 2024-03-31T23:37:05.299 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "int", 2024-03-31T23:37:05.299 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.299 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.299 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "24", 2024-03-31T23:37:05.299 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "8", 2024-03-31T23:37:05.299 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.299 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.299 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.299 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.299 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.299 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.300 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.300 INFO:teuthology.orchestra.run.smithi046.stdout: "last_opt_revision": { 2024-03-31T23:37:05.300 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "last_opt_revision", 2024-03-31T23:37:05.300 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "int", 2024-03-31T23:37:05.300 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.300 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.300 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "1", 2024-03-31T23:37:05.300 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.300 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.300 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.300 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.300 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.300 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.300 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.300 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.301 INFO:teuthology.orchestra.run.smithi046.stdout: "leaderboard": { 2024-03-31T23:37:05.301 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "leaderboard", 2024-03-31T23:37:05.301 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.301 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.301 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.301 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-31T23:37:05.301 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.301 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.301 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.301 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.301 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.301 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.301 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.301 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.301 INFO:teuthology.orchestra.run.smithi046.stdout: "leaderboard_description": { 2024-03-31T23:37:05.302 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "leaderboard_description", 2024-03-31T23:37:05.302 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.302 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.302 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.302 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-31T23:37:05.302 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.302 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.302 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.302 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.302 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.302 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.302 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.302 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.302 INFO:teuthology.orchestra.run.smithi046.stdout: "log_level": { 2024-03-31T23:37:05.303 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_level", 2024-03-31T23:37:05.303 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.303 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.303 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.303 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-31T23:37:05.303 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.303 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.303 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-31T23:37:05.303 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-31T23:37:05.303 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-31T23:37:05.303 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-31T23:37:05.303 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-31T23:37:05.303 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-31T23:37:05.303 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-31T23:37:05.303 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-31T23:37:05.304 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.304 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.304 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.304 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.304 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.304 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster": { 2024-03-31T23:37:05.304 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster", 2024-03-31T23:37:05.304 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.304 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.304 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.304 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-31T23:37:05.304 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.304 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.304 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.304 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.305 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.305 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.305 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.305 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.305 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster_level": { 2024-03-31T23:37:05.305 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster_level", 2024-03-31T23:37:05.305 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.305 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.305 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.305 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "info", 2024-03-31T23:37:05.305 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.305 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.305 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-31T23:37:05.305 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-31T23:37:05.305 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-31T23:37:05.306 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-31T23:37:05.306 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-31T23:37:05.306 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-31T23:37:05.306 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-31T23:37:05.306 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-31T23:37:05.306 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.306 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.306 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.306 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.306 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.306 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_file": { 2024-03-31T23:37:05.306 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_file", 2024-03-31T23:37:05.306 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.306 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.306 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.307 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-31T23:37:05.307 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.307 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.307 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.307 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.307 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.307 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.307 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.307 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.307 INFO:teuthology.orchestra.run.smithi046.stdout: "organization": { 2024-03-31T23:37:05.307 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "organization", 2024-03-31T23:37:05.307 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.307 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.307 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.307 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-31T23:37:05.308 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.308 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.308 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.308 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.308 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.308 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.308 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.308 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.308 INFO:teuthology.orchestra.run.smithi046.stdout: "proxy": { 2024-03-31T23:37:05.308 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "proxy", 2024-03-31T23:37:05.308 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.308 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.308 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.308 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-31T23:37:05.308 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.309 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.309 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.309 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.309 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.309 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.309 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.309 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.309 INFO:teuthology.orchestra.run.smithi046.stdout: "url": { 2024-03-31T23:37:05.309 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "url", 2024-03-31T23:37:05.309 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.309 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.309 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.309 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-03-31T23:37:05.309 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.309 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.310 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.310 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.310 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.310 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.310 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.310 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-31T23:37:05.310 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-31T23:37:05.310 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.310 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-03-31T23:37:05.310 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "test_orchestrator", 2024-03-31T23:37:05.310 INFO:teuthology.orchestra.run.smithi046.stdout: "can_run": true, 2024-03-31T23:37:05.310 INFO:teuthology.orchestra.run.smithi046.stdout: "error_string": "", 2024-03-31T23:37:05.310 INFO:teuthology.orchestra.run.smithi046.stdout: "module_options": { 2024-03-31T23:37:05.311 INFO:teuthology.orchestra.run.smithi046.stdout: "log_level": { 2024-03-31T23:37:05.311 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_level", 2024-03-31T23:37:05.311 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.311 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.311 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.311 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-31T23:37:05.311 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.311 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.311 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-31T23:37:05.311 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-31T23:37:05.311 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-31T23:37:05.311 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-31T23:37:05.311 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-31T23:37:05.311 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-31T23:37:05.311 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-31T23:37:05.312 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-31T23:37:05.312 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.312 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.312 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.312 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.312 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.312 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster": { 2024-03-31T23:37:05.312 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster", 2024-03-31T23:37:05.312 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.312 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.312 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.312 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-31T23:37:05.312 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.312 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.313 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.313 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.313 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.313 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.313 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.313 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.313 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster_level": { 2024-03-31T23:37:05.313 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster_level", 2024-03-31T23:37:05.313 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.313 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.313 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.313 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "info", 2024-03-31T23:37:05.313 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.313 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.313 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-31T23:37:05.314 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-31T23:37:05.314 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-31T23:37:05.314 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-31T23:37:05.314 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-31T23:37:05.314 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-31T23:37:05.314 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-31T23:37:05.314 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-31T23:37:05.314 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.314 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.314 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.314 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.314 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.314 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_file": { 2024-03-31T23:37:05.314 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_file", 2024-03-31T23:37:05.315 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.315 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.315 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.315 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-31T23:37:05.315 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.315 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.315 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.315 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.315 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.315 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.315 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.315 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-31T23:37:05.315 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-31T23:37:05.315 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.315 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-03-31T23:37:05.315 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "volumes", 2024-03-31T23:37:05.316 INFO:teuthology.orchestra.run.smithi046.stdout: "can_run": true, 2024-03-31T23:37:05.316 INFO:teuthology.orchestra.run.smithi046.stdout: "error_string": "", 2024-03-31T23:37:05.316 INFO:teuthology.orchestra.run.smithi046.stdout: "module_options": { 2024-03-31T23:37:05.316 INFO:teuthology.orchestra.run.smithi046.stdout: "log_level": { 2024-03-31T23:37:05.316 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_level", 2024-03-31T23:37:05.316 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.316 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.316 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.316 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-31T23:37:05.316 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.316 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.316 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-31T23:37:05.316 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-31T23:37:05.316 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-31T23:37:05.317 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-31T23:37:05.317 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-31T23:37:05.317 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-31T23:37:05.317 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-31T23:37:05.317 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-31T23:37:05.317 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.317 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.317 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.317 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.317 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.317 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster": { 2024-03-31T23:37:05.317 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster", 2024-03-31T23:37:05.317 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.317 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.317 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.317 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-31T23:37:05.318 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.318 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.318 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.318 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.318 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.318 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.318 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.318 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.318 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster_level": { 2024-03-31T23:37:05.318 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster_level", 2024-03-31T23:37:05.318 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.318 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.318 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.318 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "info", 2024-03-31T23:37:05.318 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.319 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.319 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-31T23:37:05.319 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-31T23:37:05.319 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-31T23:37:05.319 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-31T23:37:05.319 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-31T23:37:05.319 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-31T23:37:05.319 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-31T23:37:05.319 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-31T23:37:05.319 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.319 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.319 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.319 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.319 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.319 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_file": { 2024-03-31T23:37:05.320 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_file", 2024-03-31T23:37:05.320 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.320 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.320 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.320 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-31T23:37:05.320 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.320 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.320 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.320 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.320 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.320 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.320 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.320 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.320 INFO:teuthology.orchestra.run.smithi046.stdout: "max_concurrent_clones": { 2024-03-31T23:37:05.320 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "max_concurrent_clones", 2024-03-31T23:37:05.321 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "int", 2024-03-31T23:37:05.321 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.321 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.321 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "4", 2024-03-31T23:37:05.321 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.321 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.321 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.321 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Number of asynchronous cloner threads", 2024-03-31T23:37:05.321 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.321 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.321 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.321 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.321 INFO:teuthology.orchestra.run.smithi046.stdout: "periodic_async_work": { 2024-03-31T23:37:05.321 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "periodic_async_work", 2024-03-31T23:37:05.321 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.322 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.322 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.322 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-31T23:37:05.322 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.322 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.322 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.322 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Periodically check for async work", 2024-03-31T23:37:05.322 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.322 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.322 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.322 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.322 INFO:teuthology.orchestra.run.smithi046.stdout: "snapshot_clone_delay": { 2024-03-31T23:37:05.322 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "snapshot_clone_delay", 2024-03-31T23:37:05.322 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "int", 2024-03-31T23:37:05.322 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.323 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.323 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "0", 2024-03-31T23:37:05.323 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.323 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.323 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.323 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-03-31T23:37:05.323 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.323 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.323 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.323 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.323 INFO:teuthology.orchestra.run.smithi046.stdout: "snapshot_clone_no_wait": { 2024-03-31T23:37:05.323 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "snapshot_clone_no_wait", 2024-03-31T23:37:05.323 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.323 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.324 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.324 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "True", 2024-03-31T23:37:05.324 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.324 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.324 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.324 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-03-31T23:37:05.324 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.324 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.324 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.324 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-31T23:37:05.324 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-31T23:37:05.324 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.324 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-03-31T23:37:05.324 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "zabbix", 2024-03-31T23:37:05.324 INFO:teuthology.orchestra.run.smithi046.stdout: "can_run": true, 2024-03-31T23:37:05.325 INFO:teuthology.orchestra.run.smithi046.stdout: "error_string": "", 2024-03-31T23:37:05.325 INFO:teuthology.orchestra.run.smithi046.stdout: "module_options": { 2024-03-31T23:37:05.325 INFO:teuthology.orchestra.run.smithi046.stdout: "discovery_interval": { 2024-03-31T23:37:05.325 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "discovery_interval", 2024-03-31T23:37:05.325 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "uint", 2024-03-31T23:37:05.325 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.325 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.325 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "100", 2024-03-31T23:37:05.325 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.325 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.325 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.325 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.325 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.325 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.325 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.326 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.326 INFO:teuthology.orchestra.run.smithi046.stdout: "identifier": { 2024-03-31T23:37:05.326 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "identifier", 2024-03-31T23:37:05.326 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.326 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.326 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.326 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-31T23:37:05.326 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.326 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.326 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.326 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.326 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.326 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.326 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.327 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.327 INFO:teuthology.orchestra.run.smithi046.stdout: "interval": { 2024-03-31T23:37:05.327 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "interval", 2024-03-31T23:37:05.327 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "secs", 2024-03-31T23:37:05.327 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.327 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.327 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "60", 2024-03-31T23:37:05.327 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.327 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.327 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.327 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.327 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.327 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.327 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.328 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.328 INFO:teuthology.orchestra.run.smithi046.stdout: "log_level": { 2024-03-31T23:37:05.328 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_level", 2024-03-31T23:37:05.328 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.328 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.328 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.328 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-31T23:37:05.328 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.328 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.328 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-31T23:37:05.328 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-31T23:37:05.328 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-31T23:37:05.328 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-31T23:37:05.328 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-31T23:37:05.328 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-31T23:37:05.329 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-31T23:37:05.329 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-31T23:37:05.329 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.329 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.329 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.329 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.329 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.329 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster": { 2024-03-31T23:37:05.329 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster", 2024-03-31T23:37:05.329 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.329 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.329 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.329 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-31T23:37:05.329 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.329 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.330 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.330 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.330 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.330 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.330 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.330 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.330 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster_level": { 2024-03-31T23:37:05.330 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster_level", 2024-03-31T23:37:05.330 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.330 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.330 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.330 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "info", 2024-03-31T23:37:05.330 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.330 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.331 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-31T23:37:05.331 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-31T23:37:05.331 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-31T23:37:05.331 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-31T23:37:05.331 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-31T23:37:05.331 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-31T23:37:05.331 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-31T23:37:05.331 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-31T23:37:05.331 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.331 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.331 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.331 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.331 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.331 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_file": { 2024-03-31T23:37:05.331 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_file", 2024-03-31T23:37:05.332 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.332 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.332 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.332 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-31T23:37:05.332 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.332 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.332 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.332 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.332 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.332 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.332 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.332 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.332 INFO:teuthology.orchestra.run.smithi046.stdout: "zabbix_host": { 2024-03-31T23:37:05.332 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "zabbix_host", 2024-03-31T23:37:05.332 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.333 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.333 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.333 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-31T23:37:05.333 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.333 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.333 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.333 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.333 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.333 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.333 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.333 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.333 INFO:teuthology.orchestra.run.smithi046.stdout: "zabbix_port": { 2024-03-31T23:37:05.333 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "zabbix_port", 2024-03-31T23:37:05.333 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "int", 2024-03-31T23:37:05.333 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.334 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.334 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "10051", 2024-03-31T23:37:05.334 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.334 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.334 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.334 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.334 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.334 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.334 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.334 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.334 INFO:teuthology.orchestra.run.smithi046.stdout: "zabbix_sender": { 2024-03-31T23:37:05.334 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "zabbix_sender", 2024-03-31T23:37:05.334 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.334 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.334 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.335 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-03-31T23:37:05.335 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.335 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.335 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.335 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.335 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.335 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.335 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.335 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-31T23:37:05.335 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-31T23:37:05.335 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-31T23:37:05.335 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-03-31T23:37:05.335 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.335 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-03-31T23:37:05.335 INFO:teuthology.orchestra.run.smithi046.stdout: "gid": 25493, 2024-03-31T23:37:05.336 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "y", 2024-03-31T23:37:05.336 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr_features": 4540701547738038271, 2024-03-31T23:37:05.336 INFO:teuthology.orchestra.run.smithi046.stdout: "available_modules": [ 2024-03-31T23:37:05.336 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-03-31T23:37:05.336 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "alerts", 2024-03-31T23:37:05.336 INFO:teuthology.orchestra.run.smithi046.stdout: "can_run": true, 2024-03-31T23:37:05.336 INFO:teuthology.orchestra.run.smithi046.stdout: "error_string": "", 2024-03-31T23:37:05.336 INFO:teuthology.orchestra.run.smithi046.stdout: "module_options": { 2024-03-31T23:37:05.336 INFO:teuthology.orchestra.run.smithi046.stdout: "interval": { 2024-03-31T23:37:05.336 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "interval", 2024-03-31T23:37:05.336 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "secs", 2024-03-31T23:37:05.336 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.336 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.336 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "60", 2024-03-31T23:37:05.337 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.337 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.337 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.337 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "How frequently to reexamine health status", 2024-03-31T23:37:05.337 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.337 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.337 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.337 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.337 INFO:teuthology.orchestra.run.smithi046.stdout: "log_level": { 2024-03-31T23:37:05.337 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_level", 2024-03-31T23:37:05.337 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.337 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.337 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.337 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-31T23:37:05.338 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.338 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.338 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-31T23:37:05.338 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-31T23:37:05.338 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-31T23:37:05.338 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-31T23:37:05.338 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-31T23:37:05.338 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-31T23:37:05.338 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-31T23:37:05.338 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-31T23:37:05.338 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.338 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.338 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.338 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.339 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.339 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster": { 2024-03-31T23:37:05.339 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster", 2024-03-31T23:37:05.339 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.339 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.339 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.339 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-31T23:37:05.339 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.339 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.339 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.339 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.339 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.339 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.339 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.339 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.340 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster_level": { 2024-03-31T23:37:05.340 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster_level", 2024-03-31T23:37:05.340 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.340 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.340 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.340 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "info", 2024-03-31T23:37:05.340 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.340 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.340 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-31T23:37:05.340 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-31T23:37:05.340 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-31T23:37:05.340 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-31T23:37:05.340 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-31T23:37:05.340 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-31T23:37:05.341 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-31T23:37:05.341 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-31T23:37:05.341 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.341 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.341 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.341 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.341 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.341 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_file": { 2024-03-31T23:37:05.341 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_file", 2024-03-31T23:37:05.341 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.341 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.341 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.341 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-31T23:37:05.341 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.342 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.342 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.342 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.342 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.342 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.342 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.342 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.342 INFO:teuthology.orchestra.run.smithi046.stdout: "smtp_destination": { 2024-03-31T23:37:05.342 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "smtp_destination", 2024-03-31T23:37:05.342 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.342 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.342 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.342 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-31T23:37:05.342 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.342 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.343 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.343 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Email address to send alerts to", 2024-03-31T23:37:05.343 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.343 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.343 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.343 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.343 INFO:teuthology.orchestra.run.smithi046.stdout: "smtp_from_name": { 2024-03-31T23:37:05.343 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "smtp_from_name", 2024-03-31T23:37:05.343 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.343 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.343 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.343 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "Ceph", 2024-03-31T23:37:05.343 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.343 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.343 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.344 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Email From: name", 2024-03-31T23:37:05.344 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.344 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.344 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.344 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.344 INFO:teuthology.orchestra.run.smithi046.stdout: "smtp_host": { 2024-03-31T23:37:05.344 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "smtp_host", 2024-03-31T23:37:05.344 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.344 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.344 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.344 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-31T23:37:05.344 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.344 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.344 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.344 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "SMTP server", 2024-03-31T23:37:05.345 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.345 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.345 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.345 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.345 INFO:teuthology.orchestra.run.smithi046.stdout: "smtp_password": { 2024-03-31T23:37:05.345 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "smtp_password", 2024-03-31T23:37:05.345 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.345 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.345 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.345 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-31T23:37:05.345 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.345 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.345 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.345 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Password to authenticate with", 2024-03-31T23:37:05.345 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.346 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.346 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.346 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.346 INFO:teuthology.orchestra.run.smithi046.stdout: "smtp_port": { 2024-03-31T23:37:05.346 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "smtp_port", 2024-03-31T23:37:05.346 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "int", 2024-03-31T23:37:05.346 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.346 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.346 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "465", 2024-03-31T23:37:05.346 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.346 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.346 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.346 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "SMTP port", 2024-03-31T23:37:05.346 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.346 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.347 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.347 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.347 INFO:teuthology.orchestra.run.smithi046.stdout: "smtp_sender": { 2024-03-31T23:37:05.347 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "smtp_sender", 2024-03-31T23:37:05.347 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.347 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.347 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.347 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-31T23:37:05.347 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.347 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.347 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.347 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "SMTP envelope sender", 2024-03-31T23:37:05.347 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.348 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.348 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.348 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.348 INFO:teuthology.orchestra.run.smithi046.stdout: "smtp_ssl": { 2024-03-31T23:37:05.348 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "smtp_ssl", 2024-03-31T23:37:05.348 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.348 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.348 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.348 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "True", 2024-03-31T23:37:05.348 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.348 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.349 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.349 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Use SSL to connect to SMTP server", 2024-03-31T23:37:05.349 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.349 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.349 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.349 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.349 INFO:teuthology.orchestra.run.smithi046.stdout: "smtp_user": { 2024-03-31T23:37:05.349 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "smtp_user", 2024-03-31T23:37:05.349 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.349 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.349 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.349 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-31T23:37:05.349 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.349 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.350 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.350 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "User to authenticate as", 2024-03-31T23:37:05.350 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.350 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.350 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.350 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-31T23:37:05.350 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-31T23:37:05.350 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.350 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-03-31T23:37:05.350 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "balancer", 2024-03-31T23:37:05.350 INFO:teuthology.orchestra.run.smithi046.stdout: "can_run": true, 2024-03-31T23:37:05.350 INFO:teuthology.orchestra.run.smithi046.stdout: "error_string": "", 2024-03-31T23:37:05.350 INFO:teuthology.orchestra.run.smithi046.stdout: "module_options": { 2024-03-31T23:37:05.350 INFO:teuthology.orchestra.run.smithi046.stdout: "active": { 2024-03-31T23:37:05.351 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "active", 2024-03-31T23:37:05.351 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.351 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.351 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.351 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "True", 2024-03-31T23:37:05.351 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.351 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.351 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.351 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "automatically balance PGs across cluster", 2024-03-31T23:37:05.351 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.351 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.351 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.351 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.351 INFO:teuthology.orchestra.run.smithi046.stdout: "begin_time": { 2024-03-31T23:37:05.352 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "begin_time", 2024-03-31T23:37:05.352 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.352 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.352 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.352 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "0000", 2024-03-31T23:37:05.352 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.352 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.352 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.352 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "beginning time of day to automatically balance", 2024-03-31T23:37:05.352 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-31T23:37:05.352 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.352 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.352 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.352 INFO:teuthology.orchestra.run.smithi046.stdout: "begin_weekday": { 2024-03-31T23:37:05.352 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "begin_weekday", 2024-03-31T23:37:05.353 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "uint", 2024-03-31T23:37:05.353 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.353 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.353 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "0", 2024-03-31T23:37:05.353 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "0", 2024-03-31T23:37:05.353 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "6", 2024-03-31T23:37:05.353 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.353 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-03-31T23:37:05.353 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-31T23:37:05.353 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.353 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.353 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.353 INFO:teuthology.orchestra.run.smithi046.stdout: "crush_compat_max_iterations": { 2024-03-31T23:37:05.354 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "crush_compat_max_iterations", 2024-03-31T23:37:05.354 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "uint", 2024-03-31T23:37:05.354 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.354 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.354 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "25", 2024-03-31T23:37:05.354 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "1", 2024-03-31T23:37:05.354 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "250", 2024-03-31T23:37:05.354 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.354 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-03-31T23:37:05.354 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.354 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.354 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.354 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.354 INFO:teuthology.orchestra.run.smithi046.stdout: "crush_compat_metrics": { 2024-03-31T23:37:05.355 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "crush_compat_metrics", 2024-03-31T23:37:05.355 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.355 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.355 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.355 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "pgs,objects,bytes", 2024-03-31T23:37:05.355 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.355 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.355 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.355 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-03-31T23:37:05.355 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-03-31T23:37:05.355 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.355 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.355 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.355 INFO:teuthology.orchestra.run.smithi046.stdout: "crush_compat_step": { 2024-03-31T23:37:05.355 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "crush_compat_step", 2024-03-31T23:37:05.356 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "float", 2024-03-31T23:37:05.356 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.356 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.356 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "0.5", 2024-03-31T23:37:05.356 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "0.001", 2024-03-31T23:37:05.356 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "0.999", 2024-03-31T23:37:05.356 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.356 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "aggressiveness of optimization", 2024-03-31T23:37:05.356 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-03-31T23:37:05.356 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.356 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.356 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.356 INFO:teuthology.orchestra.run.smithi046.stdout: "end_time": { 2024-03-31T23:37:05.356 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "end_time", 2024-03-31T23:37:05.357 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.357 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.357 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.357 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "2359", 2024-03-31T23:37:05.357 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.357 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.357 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.357 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "ending time of day to automatically balance", 2024-03-31T23:37:05.357 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-31T23:37:05.357 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.357 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.357 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.357 INFO:teuthology.orchestra.run.smithi046.stdout: "end_weekday": { 2024-03-31T23:37:05.357 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "end_weekday", 2024-03-31T23:37:05.358 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "uint", 2024-03-31T23:37:05.358 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.358 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.358 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "0", 2024-03-31T23:37:05.358 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "0", 2024-03-31T23:37:05.358 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "6", 2024-03-31T23:37:05.358 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.358 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-03-31T23:37:05.358 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-31T23:37:05.358 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.358 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.358 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.358 INFO:teuthology.orchestra.run.smithi046.stdout: "log_level": { 2024-03-31T23:37:05.358 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_level", 2024-03-31T23:37:05.358 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.359 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.359 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.359 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-31T23:37:05.359 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.359 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.359 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-31T23:37:05.359 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-31T23:37:05.359 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-31T23:37:05.359 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-31T23:37:05.359 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-31T23:37:05.359 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-31T23:37:05.359 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-31T23:37:05.359 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-31T23:37:05.359 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.360 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.360 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.360 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.360 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.360 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster": { 2024-03-31T23:37:05.360 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster", 2024-03-31T23:37:05.360 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.360 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.360 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.360 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-31T23:37:05.360 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.360 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.360 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.360 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.360 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.360 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.361 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.361 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.361 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster_level": { 2024-03-31T23:37:05.361 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster_level", 2024-03-31T23:37:05.361 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.361 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.361 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.361 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "info", 2024-03-31T23:37:05.361 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.361 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.361 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-31T23:37:05.361 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-31T23:37:05.361 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-31T23:37:05.361 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-31T23:37:05.361 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-31T23:37:05.362 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-31T23:37:05.362 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-31T23:37:05.362 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-31T23:37:05.362 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.362 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.362 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.362 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.362 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.362 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_file": { 2024-03-31T23:37:05.362 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_file", 2024-03-31T23:37:05.362 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.362 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.362 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.362 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-31T23:37:05.362 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.363 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.363 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.363 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.363 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.363 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.363 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.363 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.363 INFO:teuthology.orchestra.run.smithi046.stdout: "min_score": { 2024-03-31T23:37:05.363 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "min_score", 2024-03-31T23:37:05.363 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "float", 2024-03-31T23:37:05.363 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.363 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.363 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "0", 2024-03-31T23:37:05.363 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.364 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.364 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.364 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-03-31T23:37:05.364 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.364 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.364 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.364 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.364 INFO:teuthology.orchestra.run.smithi046.stdout: "mode": { 2024-03-31T23:37:05.364 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "mode", 2024-03-31T23:37:05.364 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.364 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.364 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.364 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "upmap", 2024-03-31T23:37:05.364 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.364 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.365 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-31T23:37:05.365 INFO:teuthology.orchestra.run.smithi046.stdout: "crush-compat", 2024-03-31T23:37:05.365 INFO:teuthology.orchestra.run.smithi046.stdout: "none", 2024-03-31T23:37:05.365 INFO:teuthology.orchestra.run.smithi046.stdout: "read", 2024-03-31T23:37:05.365 INFO:teuthology.orchestra.run.smithi046.stdout: "upmap", 2024-03-31T23:37:05.365 INFO:teuthology.orchestra.run.smithi046.stdout: "upmap-read" 2024-03-31T23:37:05.365 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-31T23:37:05.365 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Balancer mode", 2024-03-31T23:37:05.365 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.365 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.365 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.365 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.365 INFO:teuthology.orchestra.run.smithi046.stdout: "pool_ids": { 2024-03-31T23:37:05.365 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "pool_ids", 2024-03-31T23:37:05.366 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.366 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.366 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.366 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-31T23:37:05.366 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.366 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.366 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.366 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-03-31T23:37:05.366 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.366 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.366 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.366 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.366 INFO:teuthology.orchestra.run.smithi046.stdout: "sleep_interval": { 2024-03-31T23:37:05.366 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "sleep_interval", 2024-03-31T23:37:05.367 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "secs", 2024-03-31T23:37:05.367 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.367 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.367 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "60", 2024-03-31T23:37:05.367 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.367 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.367 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.367 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-03-31T23:37:05.367 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.367 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.367 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.367 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.367 INFO:teuthology.orchestra.run.smithi046.stdout: "upmap_max_deviation": { 2024-03-31T23:37:05.367 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "upmap_max_deviation", 2024-03-31T23:37:05.367 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "int", 2024-03-31T23:37:05.368 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.368 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.368 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "5", 2024-03-31T23:37:05.368 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "1", 2024-03-31T23:37:05.368 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.368 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.368 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "deviation below which no optimization is attempted", 2024-03-31T23:37:05.368 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-03-31T23:37:05.368 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.368 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.368 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.368 INFO:teuthology.orchestra.run.smithi046.stdout: "upmap_max_optimizations": { 2024-03-31T23:37:05.368 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "upmap_max_optimizations", 2024-03-31T23:37:05.368 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "uint", 2024-03-31T23:37:05.369 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.369 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.369 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "10", 2024-03-31T23:37:05.369 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.369 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.369 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.369 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-03-31T23:37:05.369 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.369 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.369 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.369 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-31T23:37:05.369 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-31T23:37:05.369 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.369 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-03-31T23:37:05.369 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "cephadm", 2024-03-31T23:37:05.370 INFO:teuthology.orchestra.run.smithi046.stdout: "can_run": true, 2024-03-31T23:37:05.370 INFO:teuthology.orchestra.run.smithi046.stdout: "error_string": "", 2024-03-31T23:37:05.370 INFO:teuthology.orchestra.run.smithi046.stdout: "module_options": { 2024-03-31T23:37:05.370 INFO:teuthology.orchestra.run.smithi046.stdout: "agent_down_multiplier": { 2024-03-31T23:37:05.370 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "agent_down_multiplier", 2024-03-31T23:37:05.370 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "float", 2024-03-31T23:37:05.370 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.370 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.370 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "3.0", 2024-03-31T23:37:05.370 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.370 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.370 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.370 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-03-31T23:37:05.370 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.370 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.371 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.371 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.371 INFO:teuthology.orchestra.run.smithi046.stdout: "agent_refresh_rate": { 2024-03-31T23:37:05.371 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "agent_refresh_rate", 2024-03-31T23:37:05.371 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "secs", 2024-03-31T23:37:05.371 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.371 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.371 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "20", 2024-03-31T23:37:05.371 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.371 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.371 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.371 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-03-31T23:37:05.371 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.371 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.371 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.372 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.372 INFO:teuthology.orchestra.run.smithi046.stdout: "agent_starting_port": { 2024-03-31T23:37:05.372 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "agent_starting_port", 2024-03-31T23:37:05.372 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "int", 2024-03-31T23:37:05.372 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.372 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.372 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "4721", 2024-03-31T23:37:05.372 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.372 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.372 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.372 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-03-31T23:37:05.372 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.372 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.372 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.373 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.373 INFO:teuthology.orchestra.run.smithi046.stdout: "allow_ptrace": { 2024-03-31T23:37:05.373 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "allow_ptrace", 2024-03-31T23:37:05.373 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.373 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.373 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.373 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-31T23:37:05.373 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.373 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.373 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.373 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-03-31T23:37:05.373 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-03-31T23:37:05.373 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.373 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.373 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.374 INFO:teuthology.orchestra.run.smithi046.stdout: "autotune_interval": { 2024-03-31T23:37:05.374 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "autotune_interval", 2024-03-31T23:37:05.374 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "secs", 2024-03-31T23:37:05.374 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.374 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.374 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "600", 2024-03-31T23:37:05.374 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.374 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.374 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.374 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "how frequently to autotune daemon memory", 2024-03-31T23:37:05.374 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.374 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.374 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.374 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.375 INFO:teuthology.orchestra.run.smithi046.stdout: "autotune_memory_target_ratio": { 2024-03-31T23:37:05.375 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "autotune_memory_target_ratio", 2024-03-31T23:37:05.375 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "float", 2024-03-31T23:37:05.375 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.375 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.375 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "0.7", 2024-03-31T23:37:05.375 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.375 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.375 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.375 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-03-31T23:37:05.375 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.375 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.375 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.375 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.376 INFO:teuthology.orchestra.run.smithi046.stdout: "cephadm_log_destination": { 2024-03-31T23:37:05.376 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "cephadm_log_destination", 2024-03-31T23:37:05.376 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.376 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.376 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.376 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-31T23:37:05.376 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.376 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.376 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-31T23:37:05.376 INFO:teuthology.orchestra.run.smithi046.stdout: "file", 2024-03-31T23:37:05.376 INFO:teuthology.orchestra.run.smithi046.stdout: "file,syslog", 2024-03-31T23:37:05.376 INFO:teuthology.orchestra.run.smithi046.stdout: "syslog" 2024-03-31T23:37:05.376 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-31T23:37:05.376 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-03-31T23:37:05.377 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.377 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.377 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.377 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.377 INFO:teuthology.orchestra.run.smithi046.stdout: "cgroups_split": { 2024-03-31T23:37:05.377 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "cgroups_split", 2024-03-31T23:37:05.377 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.377 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.377 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.377 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "True", 2024-03-31T23:37:05.377 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.377 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.377 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.377 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-03-31T23:37:05.377 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.378 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.378 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.378 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.378 INFO:teuthology.orchestra.run.smithi046.stdout: "config_checks_enabled": { 2024-03-31T23:37:05.378 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "config_checks_enabled", 2024-03-31T23:37:05.378 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.378 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.378 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.378 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-31T23:37:05.378 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.378 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.378 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.378 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-03-31T23:37:05.378 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.379 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.379 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.379 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.379 INFO:teuthology.orchestra.run.smithi046.stdout: "config_dashboard": { 2024-03-31T23:37:05.379 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "config_dashboard", 2024-03-31T23:37:05.379 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.379 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.379 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.379 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "True", 2024-03-31T23:37:05.379 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.379 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.379 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.379 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-03-31T23:37:05.379 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.379 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.380 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.380 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.380 INFO:teuthology.orchestra.run.smithi046.stdout: "container_image_alertmanager": { 2024-03-31T23:37:05.380 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "container_image_alertmanager", 2024-03-31T23:37:05.380 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.380 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.380 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.380 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-03-31T23:37:05.380 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.380 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.380 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.380 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Prometheus container image", 2024-03-31T23:37:05.380 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.380 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.380 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.381 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.381 INFO:teuthology.orchestra.run.smithi046.stdout: "container_image_base": { 2024-03-31T23:37:05.381 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "container_image_base", 2024-03-31T23:37:05.381 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.381 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.381 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.381 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "quay.io/ceph/ceph", 2024-03-31T23:37:05.381 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.381 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.381 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.381 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Container image name, without the tag", 2024-03-31T23:37:05.381 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.381 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.381 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.381 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.382 INFO:teuthology.orchestra.run.smithi046.stdout: "container_image_elasticsearch": { 2024-03-31T23:37:05.382 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "container_image_elasticsearch", 2024-03-31T23:37:05.382 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.382 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.382 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.382 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-03-31T23:37:05.382 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.382 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.382 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.382 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "elasticsearch container image", 2024-03-31T23:37:05.382 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.382 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.382 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.382 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.383 INFO:teuthology.orchestra.run.smithi046.stdout: "container_image_grafana": { 2024-03-31T23:37:05.383 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "container_image_grafana", 2024-03-31T23:37:05.383 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.383 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.383 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.383 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "quay.io/ceph/grafana:9.4.12", 2024-03-31T23:37:05.383 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.383 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.383 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.383 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Prometheus container image", 2024-03-31T23:37:05.383 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.383 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.383 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.383 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.383 INFO:teuthology.orchestra.run.smithi046.stdout: "container_image_haproxy": { 2024-03-31T23:37:05.384 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "container_image_haproxy", 2024-03-31T23:37:05.384 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.384 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.384 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.384 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-03-31T23:37:05.384 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.384 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.384 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.384 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "HAproxy container image", 2024-03-31T23:37:05.384 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.384 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.384 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.384 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.384 INFO:teuthology.orchestra.run.smithi046.stdout: "container_image_jaeger_agent": { 2024-03-31T23:37:05.384 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "container_image_jaeger_agent", 2024-03-31T23:37:05.385 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.385 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.385 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.385 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-03-31T23:37:05.385 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.385 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.385 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.385 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Jaeger agent container image", 2024-03-31T23:37:05.385 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.385 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.385 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.385 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.385 INFO:teuthology.orchestra.run.smithi046.stdout: "container_image_jaeger_collector": { 2024-03-31T23:37:05.385 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "container_image_jaeger_collector", 2024-03-31T23:37:05.385 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.386 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.386 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.386 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-03-31T23:37:05.386 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.386 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.386 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.386 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Jaeger collector container image", 2024-03-31T23:37:05.386 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.386 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.386 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.386 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.386 INFO:teuthology.orchestra.run.smithi046.stdout: "container_image_jaeger_query": { 2024-03-31T23:37:05.386 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "container_image_jaeger_query", 2024-03-31T23:37:05.387 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.387 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.387 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.387 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-03-31T23:37:05.387 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.387 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.387 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.387 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Jaeger query container image", 2024-03-31T23:37:05.387 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.387 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.387 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.387 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.387 INFO:teuthology.orchestra.run.smithi046.stdout: "container_image_keepalived": { 2024-03-31T23:37:05.387 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "container_image_keepalived", 2024-03-31T23:37:05.387 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.388 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.388 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.388 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-03-31T23:37:05.388 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.388 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.388 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.388 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Keepalived container image", 2024-03-31T23:37:05.388 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.388 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.388 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.388 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.388 INFO:teuthology.orchestra.run.smithi046.stdout: "container_image_loki": { 2024-03-31T23:37:05.388 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "container_image_loki", 2024-03-31T23:37:05.388 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.389 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.389 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.389 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-03-31T23:37:05.389 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.389 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.389 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.389 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Loki container image", 2024-03-31T23:37:05.389 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.389 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.389 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.389 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.389 INFO:teuthology.orchestra.run.smithi046.stdout: "container_image_node_exporter": { 2024-03-31T23:37:05.389 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "container_image_node_exporter", 2024-03-31T23:37:05.389 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.389 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.390 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.390 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-03-31T23:37:05.390 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.390 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.390 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.390 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Prometheus container image", 2024-03-31T23:37:05.390 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.390 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.390 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.390 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.390 INFO:teuthology.orchestra.run.smithi046.stdout: "container_image_nvmeof": { 2024-03-31T23:37:05.390 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "container_image_nvmeof", 2024-03-31T23:37:05.390 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.390 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.391 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.391 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "quay.io/ceph/nvmeof:latest", 2024-03-31T23:37:05.391 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.391 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.391 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.391 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Nvme-of container image", 2024-03-31T23:37:05.391 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.391 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.391 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.391 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.391 INFO:teuthology.orchestra.run.smithi046.stdout: "container_image_prometheus": { 2024-03-31T23:37:05.391 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "container_image_prometheus", 2024-03-31T23:37:05.391 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.391 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.391 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.392 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-03-31T23:37:05.392 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.392 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.392 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.392 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Prometheus container image", 2024-03-31T23:37:05.392 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.392 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.392 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.392 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.392 INFO:teuthology.orchestra.run.smithi046.stdout: "container_image_promtail": { 2024-03-31T23:37:05.392 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "container_image_promtail", 2024-03-31T23:37:05.392 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.392 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.392 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.392 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-03-31T23:37:05.393 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.393 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.393 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.393 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Promtail container image", 2024-03-31T23:37:05.393 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.393 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.393 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.393 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.393 INFO:teuthology.orchestra.run.smithi046.stdout: "container_image_snmp_gateway": { 2024-03-31T23:37:05.393 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "container_image_snmp_gateway", 2024-03-31T23:37:05.393 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.393 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.393 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.393 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-03-31T23:37:05.394 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.394 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.394 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.394 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "SNMP Gateway container image", 2024-03-31T23:37:05.394 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.394 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.394 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.394 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.394 INFO:teuthology.orchestra.run.smithi046.stdout: "container_init": { 2024-03-31T23:37:05.394 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "container_init", 2024-03-31T23:37:05.394 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.394 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.394 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.394 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "True", 2024-03-31T23:37:05.394 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.395 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.395 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.395 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Run podman/docker with `--init`", 2024-03-31T23:37:05.395 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.395 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.395 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.395 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.395 INFO:teuthology.orchestra.run.smithi046.stdout: "daemon_cache_timeout": { 2024-03-31T23:37:05.395 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "daemon_cache_timeout", 2024-03-31T23:37:05.395 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "secs", 2024-03-31T23:37:05.395 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.395 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.395 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "600", 2024-03-31T23:37:05.395 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.395 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.396 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.396 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-03-31T23:37:05.396 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.396 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.396 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.396 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.396 INFO:teuthology.orchestra.run.smithi046.stdout: "default_cephadm_command_timeout": { 2024-03-31T23:37:05.396 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "default_cephadm_command_timeout", 2024-03-31T23:37:05.396 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "int", 2024-03-31T23:37:05.396 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.396 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.396 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "900", 2024-03-31T23:37:05.396 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.396 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.396 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.397 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-03-31T23:37:05.397 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.397 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.397 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.397 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.397 INFO:teuthology.orchestra.run.smithi046.stdout: "default_registry": { 2024-03-31T23:37:05.397 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "default_registry", 2024-03-31T23:37:05.397 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.397 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.397 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.397 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "docker.io", 2024-03-31T23:37:05.397 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.397 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.397 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.398 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-03-31T23:37:05.398 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.398 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.398 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.398 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.398 INFO:teuthology.orchestra.run.smithi046.stdout: "device_cache_timeout": { 2024-03-31T23:37:05.398 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "device_cache_timeout", 2024-03-31T23:37:05.398 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "secs", 2024-03-31T23:37:05.398 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.398 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.398 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "1800", 2024-03-31T23:37:05.398 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.398 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.398 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.399 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "seconds to cache device inventory", 2024-03-31T23:37:05.399 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.399 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.399 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.399 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.399 INFO:teuthology.orchestra.run.smithi046.stdout: "device_enhanced_scan": { 2024-03-31T23:37:05.399 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "device_enhanced_scan", 2024-03-31T23:37:05.399 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.399 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.399 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.399 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-31T23:37:05.399 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.399 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.399 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.399 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Use libstoragemgmt during device scans", 2024-03-31T23:37:05.400 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.400 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.400 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.400 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.400 INFO:teuthology.orchestra.run.smithi046.stdout: "facts_cache_timeout": { 2024-03-31T23:37:05.400 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "facts_cache_timeout", 2024-03-31T23:37:05.400 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "secs", 2024-03-31T23:37:05.400 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.400 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.400 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "60", 2024-03-31T23:37:05.400 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.400 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.400 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.400 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "seconds to cache host facts data", 2024-03-31T23:37:05.400 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.401 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.401 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.401 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.401 INFO:teuthology.orchestra.run.smithi046.stdout: "grafana_dashboards_path": { 2024-03-31T23:37:05.401 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "grafana_dashboards_path", 2024-03-31T23:37:05.401 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.401 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.401 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.401 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "/etc/grafana/dashboards/ceph-dashboard/", 2024-03-31T23:37:05.401 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.401 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.401 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.401 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "location of dashboards to include in grafana deployments", 2024-03-31T23:37:05.401 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.402 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.402 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.402 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.402 INFO:teuthology.orchestra.run.smithi046.stdout: "host_check_interval": { 2024-03-31T23:37:05.402 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "host_check_interval", 2024-03-31T23:37:05.402 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "secs", 2024-03-31T23:37:05.402 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.402 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.402 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "600", 2024-03-31T23:37:05.402 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.402 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.402 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.402 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "how frequently to perform a host check", 2024-03-31T23:37:05.402 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.403 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.403 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.403 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.403 INFO:teuthology.orchestra.run.smithi046.stdout: "hw_monitoring": { 2024-03-31T23:37:05.403 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "hw_monitoring", 2024-03-31T23:37:05.403 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.403 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.403 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.403 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-31T23:37:05.403 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.403 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.403 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.403 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-03-31T23:37:05.403 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.403 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.403 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.404 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.404 INFO:teuthology.orchestra.run.smithi046.stdout: "inventory_list_all": { 2024-03-31T23:37:05.404 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "inventory_list_all", 2024-03-31T23:37:05.404 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.404 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.404 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.404 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-31T23:37:05.404 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.404 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.404 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.404 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-03-31T23:37:05.404 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.404 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.404 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.404 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.405 INFO:teuthology.orchestra.run.smithi046.stdout: "log_level": { 2024-03-31T23:37:05.405 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_level", 2024-03-31T23:37:05.405 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.405 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.405 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.405 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-31T23:37:05.405 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.405 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.405 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-31T23:37:05.405 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-31T23:37:05.405 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-31T23:37:05.405 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-31T23:37:05.405 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-31T23:37:05.405 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-31T23:37:05.405 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-31T23:37:05.406 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-31T23:37:05.406 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.406 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.406 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.406 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.406 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.406 INFO:teuthology.orchestra.run.smithi046.stdout: "log_refresh_metadata": { 2024-03-31T23:37:05.406 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_refresh_metadata", 2024-03-31T23:37:05.406 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.406 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.406 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.406 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-31T23:37:05.406 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.406 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.407 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.407 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-03-31T23:37:05.407 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.407 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.407 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.407 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.407 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster": { 2024-03-31T23:37:05.407 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster", 2024-03-31T23:37:05.407 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.407 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.407 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.407 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "True", 2024-03-31T23:37:05.407 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.407 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.407 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.408 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-03-31T23:37:05.408 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.408 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.408 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.408 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.408 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster_level": { 2024-03-31T23:37:05.408 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster_level", 2024-03-31T23:37:05.408 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.408 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.408 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.408 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "info", 2024-03-31T23:37:05.408 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.408 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.408 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-31T23:37:05.409 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-31T23:37:05.409 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-31T23:37:05.409 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-31T23:37:05.409 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-31T23:37:05.409 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-31T23:37:05.409 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-31T23:37:05.409 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-31T23:37:05.409 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.409 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.409 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.409 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.409 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.409 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_file": { 2024-03-31T23:37:05.410 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_file", 2024-03-31T23:37:05.410 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.410 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.410 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.410 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-31T23:37:05.410 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.410 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.410 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.410 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.410 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.410 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.410 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.410 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.410 INFO:teuthology.orchestra.run.smithi046.stdout: "manage_etc_ceph_ceph_conf": { 2024-03-31T23:37:05.411 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-03-31T23:37:05.411 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.411 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.411 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.411 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-31T23:37:05.411 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.411 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.411 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.411 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-03-31T23:37:05.411 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.411 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.411 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.411 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.411 INFO:teuthology.orchestra.run.smithi046.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-03-31T23:37:05.411 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-03-31T23:37:05.412 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.412 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.412 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.412 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "*", 2024-03-31T23:37:05.412 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.412 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.412 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.412 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-03-31T23:37:05.412 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.412 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.412 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.412 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.412 INFO:teuthology.orchestra.run.smithi046.stdout: "max_count_per_host": { 2024-03-31T23:37:05.412 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "max_count_per_host", 2024-03-31T23:37:05.412 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "int", 2024-03-31T23:37:05.413 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.413 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.413 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "10", 2024-03-31T23:37:05.413 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.413 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.413 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.413 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "max number of daemons per service per host", 2024-03-31T23:37:05.413 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.413 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.413 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.413 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.413 INFO:teuthology.orchestra.run.smithi046.stdout: "max_osd_draining_count": { 2024-03-31T23:37:05.413 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "max_osd_draining_count", 2024-03-31T23:37:05.413 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "int", 2024-03-31T23:37:05.414 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.414 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.414 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "10", 2024-03-31T23:37:05.414 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.414 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.414 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.414 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-03-31T23:37:05.414 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.414 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.414 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.414 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.414 INFO:teuthology.orchestra.run.smithi046.stdout: "migration_current": { 2024-03-31T23:37:05.414 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "migration_current", 2024-03-31T23:37:05.414 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "int", 2024-03-31T23:37:05.415 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.415 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.415 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-31T23:37:05.415 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.415 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.415 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.415 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "internal - do not modify", 2024-03-31T23:37:05.415 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.415 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.415 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.415 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.415 INFO:teuthology.orchestra.run.smithi046.stdout: "mode": { 2024-03-31T23:37:05.415 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "mode", 2024-03-31T23:37:05.415 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.416 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.416 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.416 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "root", 2024-03-31T23:37:05.416 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.416 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.416 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-31T23:37:05.416 INFO:teuthology.orchestra.run.smithi046.stdout: "cephadm-package", 2024-03-31T23:37:05.416 INFO:teuthology.orchestra.run.smithi046.stdout: "root" 2024-03-31T23:37:05.416 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-31T23:37:05.416 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "mode for remote execution of cephadm", 2024-03-31T23:37:05.416 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.416 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.416 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.416 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.416 INFO:teuthology.orchestra.run.smithi046.stdout: "oob_default_addr": { 2024-03-31T23:37:05.417 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "oob_default_addr", 2024-03-31T23:37:05.417 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.417 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.417 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.417 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "169.254.1.1", 2024-03-31T23:37:05.417 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.417 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.417 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.417 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Default address for RedFish API (oob management).", 2024-03-31T23:37:05.417 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.417 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.417 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.417 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.417 INFO:teuthology.orchestra.run.smithi046.stdout: "prometheus_alerts_path": { 2024-03-31T23:37:05.417 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "prometheus_alerts_path", 2024-03-31T23:37:05.418 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.418 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.418 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.418 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-03-31T23:37:05.418 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.418 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.418 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.418 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-03-31T23:37:05.418 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.418 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.418 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.418 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.418 INFO:teuthology.orchestra.run.smithi046.stdout: "registry_insecure": { 2024-03-31T23:37:05.418 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "registry_insecure", 2024-03-31T23:37:05.419 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.419 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.419 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.419 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-31T23:37:05.419 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.419 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.419 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.419 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-03-31T23:37:05.419 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.419 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.419 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.419 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.419 INFO:teuthology.orchestra.run.smithi046.stdout: "registry_password": { 2024-03-31T23:37:05.419 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "registry_password", 2024-03-31T23:37:05.419 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.420 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.420 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.420 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-31T23:37:05.420 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.420 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.420 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.420 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-03-31T23:37:05.420 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.420 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.420 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.420 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.420 INFO:teuthology.orchestra.run.smithi046.stdout: "registry_url": { 2024-03-31T23:37:05.420 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "registry_url", 2024-03-31T23:37:05.420 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.420 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.420 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.421 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-31T23:37:05.421 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.421 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.421 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.421 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-03-31T23:37:05.421 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.421 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.421 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.421 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.421 INFO:teuthology.orchestra.run.smithi046.stdout: "registry_username": { 2024-03-31T23:37:05.421 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "registry_username", 2024-03-31T23:37:05.421 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.421 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.421 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.422 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-31T23:37:05.422 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.422 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.422 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.422 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-03-31T23:37:05.422 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.422 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.422 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.422 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.422 INFO:teuthology.orchestra.run.smithi046.stdout: "secure_monitoring_stack": { 2024-03-31T23:37:05.422 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "secure_monitoring_stack", 2024-03-31T23:37:05.422 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.422 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.422 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.423 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-31T23:37:05.423 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.423 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.423 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.423 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-03-31T23:37:05.423 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.423 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.423 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.423 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.423 INFO:teuthology.orchestra.run.smithi046.stdout: "service_discovery_port": { 2024-03-31T23:37:05.423 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "service_discovery_port", 2024-03-31T23:37:05.423 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "int", 2024-03-31T23:37:05.423 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.423 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.424 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "8765", 2024-03-31T23:37:05.424 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.424 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.424 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.424 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "cephadm service discovery port", 2024-03-31T23:37:05.424 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.424 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.424 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.424 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.424 INFO:teuthology.orchestra.run.smithi046.stdout: "ssh_config_file": { 2024-03-31T23:37:05.424 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "ssh_config_file", 2024-03-31T23:37:05.424 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.424 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.424 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.424 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-31T23:37:05.425 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.425 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.425 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.425 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-03-31T23:37:05.425 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.425 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.425 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.425 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.425 INFO:teuthology.orchestra.run.smithi046.stdout: "use_agent": { 2024-03-31T23:37:05.425 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "use_agent", 2024-03-31T23:37:05.425 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.425 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.425 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.425 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-31T23:37:05.425 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.426 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.426 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.426 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-03-31T23:37:05.426 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.426 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.426 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.426 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.426 INFO:teuthology.orchestra.run.smithi046.stdout: "use_repo_digest": { 2024-03-31T23:37:05.426 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "use_repo_digest", 2024-03-31T23:37:05.426 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.426 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.426 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.426 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "True", 2024-03-31T23:37:05.426 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.427 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.427 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.427 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-03-31T23:37:05.427 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.427 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.427 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.427 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.427 INFO:teuthology.orchestra.run.smithi046.stdout: "warn_on_failed_host_check": { 2024-03-31T23:37:05.427 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "warn_on_failed_host_check", 2024-03-31T23:37:05.427 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.427 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.427 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.427 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "True", 2024-03-31T23:37:05.427 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.427 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.428 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.428 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "raise a health warning if the host check fails", 2024-03-31T23:37:05.428 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.428 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.428 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.428 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.428 INFO:teuthology.orchestra.run.smithi046.stdout: "warn_on_stray_daemons": { 2024-03-31T23:37:05.428 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "warn_on_stray_daemons", 2024-03-31T23:37:05.428 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.428 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.428 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.428 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "True", 2024-03-31T23:37:05.428 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.428 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.428 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.429 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-03-31T23:37:05.429 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.429 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.429 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.429 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.429 INFO:teuthology.orchestra.run.smithi046.stdout: "warn_on_stray_hosts": { 2024-03-31T23:37:05.429 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "warn_on_stray_hosts", 2024-03-31T23:37:05.429 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.429 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.429 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.429 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "True", 2024-03-31T23:37:05.429 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.429 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.429 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.429 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-03-31T23:37:05.430 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.430 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.430 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.430 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-31T23:37:05.430 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-31T23:37:05.430 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.430 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-03-31T23:37:05.430 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "crash", 2024-03-31T23:37:05.430 INFO:teuthology.orchestra.run.smithi046.stdout: "can_run": true, 2024-03-31T23:37:05.430 INFO:teuthology.orchestra.run.smithi046.stdout: "error_string": "", 2024-03-31T23:37:05.430 INFO:teuthology.orchestra.run.smithi046.stdout: "module_options": { 2024-03-31T23:37:05.430 INFO:teuthology.orchestra.run.smithi046.stdout: "log_level": { 2024-03-31T23:37:05.430 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_level", 2024-03-31T23:37:05.430 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.430 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.431 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.431 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-31T23:37:05.431 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.431 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.431 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-31T23:37:05.431 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-31T23:37:05.431 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-31T23:37:05.431 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-31T23:37:05.431 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-31T23:37:05.431 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-31T23:37:05.431 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-31T23:37:05.431 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-31T23:37:05.431 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.431 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.432 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.432 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.432 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.432 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster": { 2024-03-31T23:37:05.432 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster", 2024-03-31T23:37:05.432 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.432 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.432 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.432 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-31T23:37:05.432 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.432 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.432 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.432 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.432 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.432 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.433 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.433 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.433 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster_level": { 2024-03-31T23:37:05.433 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster_level", 2024-03-31T23:37:05.433 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.433 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.433 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.433 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "info", 2024-03-31T23:37:05.433 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.433 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.433 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-31T23:37:05.433 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-31T23:37:05.433 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-31T23:37:05.434 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-31T23:37:05.434 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-31T23:37:05.434 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-31T23:37:05.434 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-31T23:37:05.434 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-31T23:37:05.434 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.434 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.434 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.434 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.434 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.434 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_file": { 2024-03-31T23:37:05.434 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_file", 2024-03-31T23:37:05.434 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.434 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.434 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.435 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-31T23:37:05.435 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.435 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.435 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.435 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.435 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.435 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.435 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.435 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.435 INFO:teuthology.orchestra.run.smithi046.stdout: "retain_interval": { 2024-03-31T23:37:05.435 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "retain_interval", 2024-03-31T23:37:05.435 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "secs", 2024-03-31T23:37:05.435 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.435 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.435 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "31536000", 2024-03-31T23:37:05.436 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.436 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.436 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.436 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "how long to retain crashes before pruning them", 2024-03-31T23:37:05.436 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.436 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.436 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.436 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.436 INFO:teuthology.orchestra.run.smithi046.stdout: "warn_recent_interval": { 2024-03-31T23:37:05.436 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "warn_recent_interval", 2024-03-31T23:37:05.436 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "secs", 2024-03-31T23:37:05.436 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.436 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.436 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "1209600", 2024-03-31T23:37:05.437 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.437 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.437 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.437 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "time interval in which to warn about recent crashes", 2024-03-31T23:37:05.437 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.437 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.437 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.437 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-31T23:37:05.437 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-31T23:37:05.437 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.437 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-03-31T23:37:05.437 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "dashboard", 2024-03-31T23:37:05.437 INFO:teuthology.orchestra.run.smithi046.stdout: "can_run": true, 2024-03-31T23:37:05.438 INFO:teuthology.orchestra.run.smithi046.stdout: "error_string": "", 2024-03-31T23:37:05.438 INFO:teuthology.orchestra.run.smithi046.stdout: "module_options": { 2024-03-31T23:37:05.438 INFO:teuthology.orchestra.run.smithi046.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-03-31T23:37:05.438 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-03-31T23:37:05.438 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "int", 2024-03-31T23:37:05.438 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.438 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.438 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "10", 2024-03-31T23:37:05.438 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.438 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.438 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.438 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.438 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.438 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.438 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.439 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.439 INFO:teuthology.orchestra.run.smithi046.stdout: "ALERTMANAGER_API_HOST": { 2024-03-31T23:37:05.439 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "ALERTMANAGER_API_HOST", 2024-03-31T23:37:05.439 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.439 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.439 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.439 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-31T23:37:05.439 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.439 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.439 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.439 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.439 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.439 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.439 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.440 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.440 INFO:teuthology.orchestra.run.smithi046.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-03-31T23:37:05.440 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-03-31T23:37:05.440 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.440 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.440 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.440 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "True", 2024-03-31T23:37:05.440 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.440 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.440 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.440 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.440 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.440 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.440 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.441 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.441 INFO:teuthology.orchestra.run.smithi046.stdout: "AUDIT_API_ENABLED": { 2024-03-31T23:37:05.441 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "AUDIT_API_ENABLED", 2024-03-31T23:37:05.441 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.441 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.441 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.441 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-31T23:37:05.441 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.441 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.441 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.441 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.441 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.441 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.441 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.441 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.442 INFO:teuthology.orchestra.run.smithi046.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-03-31T23:37:05.442 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-03-31T23:37:05.442 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.442 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.442 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.442 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "True", 2024-03-31T23:37:05.442 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.442 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.442 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.442 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.442 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.442 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.442 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.442 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.443 INFO:teuthology.orchestra.run.smithi046.stdout: "ENABLE_BROWSABLE_API": { 2024-03-31T23:37:05.443 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "ENABLE_BROWSABLE_API", 2024-03-31T23:37:05.443 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.443 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.443 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.443 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "True", 2024-03-31T23:37:05.443 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.443 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.443 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.443 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.443 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.443 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.443 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.443 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.444 INFO:teuthology.orchestra.run.smithi046.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-03-31T23:37:05.444 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-03-31T23:37:05.444 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.444 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.444 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.444 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "True", 2024-03-31T23:37:05.444 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.444 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.444 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.444 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.444 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.444 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.444 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.444 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.444 INFO:teuthology.orchestra.run.smithi046.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-03-31T23:37:05.445 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-03-31T23:37:05.445 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.445 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.445 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.445 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "True", 2024-03-31T23:37:05.445 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.445 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.445 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.445 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.445 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.445 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.445 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.445 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.445 INFO:teuthology.orchestra.run.smithi046.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-03-31T23:37:05.446 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-03-31T23:37:05.446 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.446 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.446 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.446 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "True", 2024-03-31T23:37:05.446 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.446 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.446 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.446 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.446 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.446 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.446 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.446 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.447 INFO:teuthology.orchestra.run.smithi046.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-03-31T23:37:05.447 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-03-31T23:37:05.447 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.447 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.447 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.447 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "True", 2024-03-31T23:37:05.447 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.447 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.447 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.447 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.447 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.447 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.447 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.447 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.448 INFO:teuthology.orchestra.run.smithi046.stdout: "FEATURE_TOGGLE_NFS": { 2024-03-31T23:37:05.448 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-03-31T23:37:05.448 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.448 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.448 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.448 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "True", 2024-03-31T23:37:05.448 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.448 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.448 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.448 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.448 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.448 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.448 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.448 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.448 INFO:teuthology.orchestra.run.smithi046.stdout: "FEATURE_TOGGLE_RBD": { 2024-03-31T23:37:05.449 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-03-31T23:37:05.449 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.449 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.449 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.449 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "True", 2024-03-31T23:37:05.449 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.449 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.449 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.449 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.449 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.449 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.449 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.449 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.449 INFO:teuthology.orchestra.run.smithi046.stdout: "FEATURE_TOGGLE_RGW": { 2024-03-31T23:37:05.449 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-03-31T23:37:05.450 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.450 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.450 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.450 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "True", 2024-03-31T23:37:05.450 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.450 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.450 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.450 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.450 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.450 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.450 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.450 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.450 INFO:teuthology.orchestra.run.smithi046.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-03-31T23:37:05.450 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-03-31T23:37:05.451 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.451 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.451 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.451 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-31T23:37:05.451 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.451 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.451 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.451 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.451 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.451 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.451 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.451 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.451 INFO:teuthology.orchestra.run.smithi046.stdout: "GRAFANA_API_PASSWORD": { 2024-03-31T23:37:05.451 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "GRAFANA_API_PASSWORD", 2024-03-31T23:37:05.451 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.452 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.452 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.452 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "admin", 2024-03-31T23:37:05.452 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.452 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.452 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.452 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.452 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.452 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.452 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.452 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.452 INFO:teuthology.orchestra.run.smithi046.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-03-31T23:37:05.452 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-03-31T23:37:05.453 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.453 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.453 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.453 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "True", 2024-03-31T23:37:05.453 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.453 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.453 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.453 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.453 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.453 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.453 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.453 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.453 INFO:teuthology.orchestra.run.smithi046.stdout: "GRAFANA_API_URL": { 2024-03-31T23:37:05.453 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "GRAFANA_API_URL", 2024-03-31T23:37:05.453 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.453 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.454 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.454 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-31T23:37:05.454 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.454 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.454 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.454 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.454 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.454 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.454 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.454 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.454 INFO:teuthology.orchestra.run.smithi046.stdout: "GRAFANA_API_USERNAME": { 2024-03-31T23:37:05.454 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "GRAFANA_API_USERNAME", 2024-03-31T23:37:05.454 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.454 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.455 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.455 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "admin", 2024-03-31T23:37:05.455 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.455 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.455 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.455 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.455 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.455 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.455 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.455 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.455 INFO:teuthology.orchestra.run.smithi046.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-03-31T23:37:05.455 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-03-31T23:37:05.455 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.455 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.456 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.456 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-31T23:37:05.456 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.456 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.456 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.456 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.456 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.456 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.456 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.456 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.456 INFO:teuthology.orchestra.run.smithi046.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-03-31T23:37:05.456 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-03-31T23:37:05.456 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.456 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.456 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.457 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-31T23:37:05.457 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.457 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.457 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.457 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.457 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.457 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.457 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.457 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.457 INFO:teuthology.orchestra.run.smithi046.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-03-31T23:37:05.457 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-03-31T23:37:05.457 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.457 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.457 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.458 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "True", 2024-03-31T23:37:05.458 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.458 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.458 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.458 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.458 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.458 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.458 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.458 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.458 INFO:teuthology.orchestra.run.smithi046.stdout: "ISSUE_TRACKER_API_KEY": { 2024-03-31T23:37:05.458 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-03-31T23:37:05.458 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.458 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.458 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.458 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-31T23:37:05.459 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.459 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.459 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.459 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.459 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.459 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.459 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.459 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.459 INFO:teuthology.orchestra.run.smithi046.stdout: "PROMETHEUS_API_HOST": { 2024-03-31T23:37:05.459 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "PROMETHEUS_API_HOST", 2024-03-31T23:37:05.459 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.459 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.459 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.460 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-31T23:37:05.460 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.460 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.460 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.460 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.460 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.460 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.460 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.460 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.460 INFO:teuthology.orchestra.run.smithi046.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-03-31T23:37:05.460 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-03-31T23:37:05.460 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.460 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.460 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.461 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "True", 2024-03-31T23:37:05.461 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.461 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.461 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.461 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.461 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.461 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.461 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.461 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.461 INFO:teuthology.orchestra.run.smithi046.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-03-31T23:37:05.461 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-03-31T23:37:05.461 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.461 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.461 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.461 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-31T23:37:05.462 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.462 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.462 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.462 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.462 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.462 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.462 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.462 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.462 INFO:teuthology.orchestra.run.smithi046.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-03-31T23:37:05.462 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-03-31T23:37:05.462 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.462 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.462 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.462 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-31T23:37:05.463 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.463 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.463 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.463 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.463 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.463 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.463 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.463 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.463 INFO:teuthology.orchestra.run.smithi046.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-03-31T23:37:05.463 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-03-31T23:37:05.463 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.463 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.463 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.463 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "True", 2024-03-31T23:37:05.464 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.464 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.464 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.464 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.464 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.464 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.464 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.464 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.464 INFO:teuthology.orchestra.run.smithi046.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-03-31T23:37:05.464 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-03-31T23:37:05.464 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.464 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.464 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.465 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "True", 2024-03-31T23:37:05.465 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.465 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.465 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.465 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.465 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.465 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.465 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.465 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.465 INFO:teuthology.orchestra.run.smithi046.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-03-31T23:37:05.465 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-03-31T23:37:05.465 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.465 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.465 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.466 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-31T23:37:05.466 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.466 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.466 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.466 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.466 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.466 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.466 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.466 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.466 INFO:teuthology.orchestra.run.smithi046.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-03-31T23:37:05.466 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-03-31T23:37:05.466 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.466 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.466 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.466 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-31T23:37:05.467 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.467 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.467 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.467 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.467 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.467 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.467 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.467 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.467 INFO:teuthology.orchestra.run.smithi046.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-03-31T23:37:05.467 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-03-31T23:37:05.467 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.467 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.467 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.467 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-31T23:37:05.467 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.468 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.468 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.468 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.468 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.468 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.468 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.468 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.468 INFO:teuthology.orchestra.run.smithi046.stdout: "PWD_POLICY_ENABLED": { 2024-03-31T23:37:05.468 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "PWD_POLICY_ENABLED", 2024-03-31T23:37:05.468 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.468 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.468 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.468 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "True", 2024-03-31T23:37:05.468 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.468 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.469 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.469 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.469 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.469 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.469 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.469 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.469 INFO:teuthology.orchestra.run.smithi046.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-03-31T23:37:05.469 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-03-31T23:37:05.469 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.469 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.469 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.469 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-03-31T23:37:05.469 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.469 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.469 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.470 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.470 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.470 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.470 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.470 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.470 INFO:teuthology.orchestra.run.smithi046.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-03-31T23:37:05.470 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-03-31T23:37:05.470 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "int", 2024-03-31T23:37:05.470 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.470 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.470 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "10", 2024-03-31T23:37:05.470 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.470 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.470 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.470 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.471 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.471 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.471 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.471 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.471 INFO:teuthology.orchestra.run.smithi046.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-03-31T23:37:05.471 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-03-31T23:37:05.471 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "int", 2024-03-31T23:37:05.471 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.471 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.471 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "8", 2024-03-31T23:37:05.471 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.471 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.471 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.471 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.471 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.472 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.472 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.472 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.472 INFO:teuthology.orchestra.run.smithi046.stdout: "REST_REQUESTS_TIMEOUT": { 2024-03-31T23:37:05.472 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-03-31T23:37:05.472 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "int", 2024-03-31T23:37:05.472 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.472 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.472 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "45", 2024-03-31T23:37:05.472 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.472 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.472 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.472 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.472 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.473 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.473 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.473 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.473 INFO:teuthology.orchestra.run.smithi046.stdout: "RGW_API_ACCESS_KEY": { 2024-03-31T23:37:05.473 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "RGW_API_ACCESS_KEY", 2024-03-31T23:37:05.473 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.473 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.473 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.473 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-31T23:37:05.473 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.473 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.473 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.473 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.473 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.473 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.474 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.474 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.474 INFO:teuthology.orchestra.run.smithi046.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-03-31T23:37:05.474 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-03-31T23:37:05.474 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.474 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.474 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.474 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "admin", 2024-03-31T23:37:05.474 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.474 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.474 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.474 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.474 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.474 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.475 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.475 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.475 INFO:teuthology.orchestra.run.smithi046.stdout: "RGW_API_SECRET_KEY": { 2024-03-31T23:37:05.475 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "RGW_API_SECRET_KEY", 2024-03-31T23:37:05.475 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.475 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.475 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.475 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-31T23:37:05.475 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.475 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.475 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.475 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.475 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.475 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.475 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.476 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.476 INFO:teuthology.orchestra.run.smithi046.stdout: "RGW_API_SSL_VERIFY": { 2024-03-31T23:37:05.476 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "RGW_API_SSL_VERIFY", 2024-03-31T23:37:05.476 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.476 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.476 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.476 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "True", 2024-03-31T23:37:05.476 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.476 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.476 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.476 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.476 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.476 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.476 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.476 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.477 INFO:teuthology.orchestra.run.smithi046.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-03-31T23:37:05.477 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-03-31T23:37:05.477 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "int", 2024-03-31T23:37:05.477 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.477 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.477 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "0", 2024-03-31T23:37:05.477 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.477 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.477 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.477 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.477 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.477 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.477 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.477 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.478 INFO:teuthology.orchestra.run.smithi046.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-03-31T23:37:05.478 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-03-31T23:37:05.478 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "int", 2024-03-31T23:37:05.478 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.478 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.478 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "10", 2024-03-31T23:37:05.478 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.478 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.478 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.478 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.478 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.478 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.478 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.478 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.478 INFO:teuthology.orchestra.run.smithi046.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-03-31T23:37:05.479 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-03-31T23:37:05.479 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "int", 2024-03-31T23:37:05.479 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.479 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.479 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "5", 2024-03-31T23:37:05.479 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.479 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.479 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.479 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.479 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.479 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.479 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.479 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.479 INFO:teuthology.orchestra.run.smithi046.stdout: "cross_origin_url": { 2024-03-31T23:37:05.479 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "cross_origin_url", 2024-03-31T23:37:05.480 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.480 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.480 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.480 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-31T23:37:05.480 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.480 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.480 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.480 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.480 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.480 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.480 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.480 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.480 INFO:teuthology.orchestra.run.smithi046.stdout: "crt_file": { 2024-03-31T23:37:05.480 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "crt_file", 2024-03-31T23:37:05.480 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.481 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.481 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.481 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-31T23:37:05.481 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.481 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.481 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.481 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.481 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.481 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.481 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.481 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.481 INFO:teuthology.orchestra.run.smithi046.stdout: "debug": { 2024-03-31T23:37:05.481 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "debug", 2024-03-31T23:37:05.481 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.482 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.482 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.482 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-31T23:37:05.482 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.482 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.482 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.482 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Enable/disable debug options", 2024-03-31T23:37:05.482 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.482 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.482 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.482 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.482 INFO:teuthology.orchestra.run.smithi046.stdout: "jwt_token_ttl": { 2024-03-31T23:37:05.482 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "jwt_token_ttl", 2024-03-31T23:37:05.482 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "int", 2024-03-31T23:37:05.483 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.483 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.483 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "28800", 2024-03-31T23:37:05.483 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.483 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.483 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.483 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.483 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.483 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.483 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.483 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.483 INFO:teuthology.orchestra.run.smithi046.stdout: "key_file": { 2024-03-31T23:37:05.483 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "key_file", 2024-03-31T23:37:05.483 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.483 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.484 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.484 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-31T23:37:05.484 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.484 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.484 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.484 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.484 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.484 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.484 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.484 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.484 INFO:teuthology.orchestra.run.smithi046.stdout: "log_level": { 2024-03-31T23:37:05.484 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_level", 2024-03-31T23:37:05.484 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.484 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.485 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.485 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-31T23:37:05.485 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.485 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.485 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-31T23:37:05.485 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-31T23:37:05.485 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-31T23:37:05.485 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-31T23:37:05.485 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-31T23:37:05.485 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-31T23:37:05.485 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-31T23:37:05.485 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-31T23:37:05.485 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.485 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.486 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.486 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.486 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.486 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster": { 2024-03-31T23:37:05.486 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster", 2024-03-31T23:37:05.486 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.486 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.486 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.486 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-31T23:37:05.486 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.486 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.486 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.486 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.486 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.487 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.487 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.487 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.487 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster_level": { 2024-03-31T23:37:05.487 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster_level", 2024-03-31T23:37:05.487 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.487 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.487 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.487 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "info", 2024-03-31T23:37:05.487 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.487 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.487 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-31T23:37:05.487 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-31T23:37:05.487 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-31T23:37:05.487 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-31T23:37:05.488 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-31T23:37:05.488 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-31T23:37:05.488 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-31T23:37:05.488 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-31T23:37:05.488 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.488 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.488 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.488 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.488 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.488 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_file": { 2024-03-31T23:37:05.488 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_file", 2024-03-31T23:37:05.488 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.488 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.488 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.488 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-31T23:37:05.489 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.489 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.489 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.489 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.489 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.489 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.489 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.489 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.489 INFO:teuthology.orchestra.run.smithi046.stdout: "motd": { 2024-03-31T23:37:05.489 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "motd", 2024-03-31T23:37:05.489 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.489 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.489 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.489 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-31T23:37:05.490 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.490 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.490 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.490 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "The message of the day", 2024-03-31T23:37:05.490 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.490 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.490 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.490 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.490 INFO:teuthology.orchestra.run.smithi046.stdout: "redirect_resolve_ip_addr": { 2024-03-31T23:37:05.490 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "redirect_resolve_ip_addr", 2024-03-31T23:37:05.490 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.490 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.490 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.490 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-31T23:37:05.491 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.491 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.491 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.491 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.491 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.491 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.491 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.491 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.491 INFO:teuthology.orchestra.run.smithi046.stdout: "server_addr": { 2024-03-31T23:37:05.491 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "server_addr", 2024-03-31T23:37:05.491 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.491 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.491 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.491 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "::", 2024-03-31T23:37:05.492 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.492 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.492 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.492 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.492 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.492 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.492 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.492 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.492 INFO:teuthology.orchestra.run.smithi046.stdout: "server_port": { 2024-03-31T23:37:05.492 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "server_port", 2024-03-31T23:37:05.492 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "int", 2024-03-31T23:37:05.492 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.492 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.492 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "8080", 2024-03-31T23:37:05.492 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.492 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.493 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.493 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.493 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.493 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.493 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.493 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.493 INFO:teuthology.orchestra.run.smithi046.stdout: "ssl": { 2024-03-31T23:37:05.493 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "ssl", 2024-03-31T23:37:05.493 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.493 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.493 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.493 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "True", 2024-03-31T23:37:05.493 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.493 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.493 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.494 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.494 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.494 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.494 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.494 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.494 INFO:teuthology.orchestra.run.smithi046.stdout: "ssl_server_port": { 2024-03-31T23:37:05.494 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "ssl_server_port", 2024-03-31T23:37:05.494 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "int", 2024-03-31T23:37:05.494 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.494 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.494 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "8443", 2024-03-31T23:37:05.494 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.494 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.494 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.495 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.495 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.495 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.495 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.495 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.495 INFO:teuthology.orchestra.run.smithi046.stdout: "standby_behaviour": { 2024-03-31T23:37:05.495 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "standby_behaviour", 2024-03-31T23:37:05.495 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.495 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.495 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.495 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "redirect", 2024-03-31T23:37:05.495 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.495 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.495 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-31T23:37:05.496 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-31T23:37:05.496 INFO:teuthology.orchestra.run.smithi046.stdout: "redirect" 2024-03-31T23:37:05.496 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-31T23:37:05.496 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.496 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.496 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.496 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.496 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.496 INFO:teuthology.orchestra.run.smithi046.stdout: "standby_error_status_code": { 2024-03-31T23:37:05.496 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "standby_error_status_code", 2024-03-31T23:37:05.496 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "int", 2024-03-31T23:37:05.496 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.496 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.496 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "500", 2024-03-31T23:37:05.497 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "400", 2024-03-31T23:37:05.497 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "599", 2024-03-31T23:37:05.497 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.497 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.497 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.497 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.497 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.497 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.497 INFO:teuthology.orchestra.run.smithi046.stdout: "url_prefix": { 2024-03-31T23:37:05.497 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "url_prefix", 2024-03-31T23:37:05.497 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.497 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.497 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.497 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-31T23:37:05.497 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.498 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.498 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.498 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.498 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.498 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.498 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.498 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-31T23:37:05.498 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-31T23:37:05.498 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.498 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-03-31T23:37:05.498 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "devicehealth", 2024-03-31T23:37:05.498 INFO:teuthology.orchestra.run.smithi046.stdout: "can_run": true, 2024-03-31T23:37:05.498 INFO:teuthology.orchestra.run.smithi046.stdout: "error_string": "", 2024-03-31T23:37:05.499 INFO:teuthology.orchestra.run.smithi046.stdout: "module_options": { 2024-03-31T23:37:05.499 INFO:teuthology.orchestra.run.smithi046.stdout: "enable_monitoring": { 2024-03-31T23:37:05.499 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "enable_monitoring", 2024-03-31T23:37:05.499 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.499 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.499 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.499 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "True", 2024-03-31T23:37:05.499 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.499 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.499 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.499 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "monitor device health metrics", 2024-03-31T23:37:05.499 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.499 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.499 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.499 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.500 INFO:teuthology.orchestra.run.smithi046.stdout: "log_level": { 2024-03-31T23:37:05.500 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_level", 2024-03-31T23:37:05.500 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.500 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.500 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.500 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-31T23:37:05.500 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.500 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.500 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-31T23:37:05.500 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-31T23:37:05.500 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-31T23:37:05.500 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-31T23:37:05.500 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-31T23:37:05.500 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-31T23:37:05.501 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-31T23:37:05.501 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-31T23:37:05.501 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.501 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.501 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.501 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.501 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.501 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster": { 2024-03-31T23:37:05.501 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster", 2024-03-31T23:37:05.501 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.501 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.501 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.501 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-31T23:37:05.501 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.501 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.502 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.502 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.502 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.502 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.502 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.502 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.502 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster_level": { 2024-03-31T23:37:05.502 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster_level", 2024-03-31T23:37:05.502 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.502 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.502 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.502 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "info", 2024-03-31T23:37:05.502 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.502 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.503 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-31T23:37:05.503 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-31T23:37:05.503 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-31T23:37:05.503 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-31T23:37:05.503 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-31T23:37:05.503 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-31T23:37:05.503 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-31T23:37:05.503 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-31T23:37:05.503 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.503 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.503 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.503 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.503 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.503 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_file": { 2024-03-31T23:37:05.503 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_file", 2024-03-31T23:37:05.504 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.504 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.504 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.504 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-31T23:37:05.504 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.504 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.504 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.504 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.504 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.504 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.504 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.504 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.504 INFO:teuthology.orchestra.run.smithi046.stdout: "mark_out_threshold": { 2024-03-31T23:37:05.504 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "mark_out_threshold", 2024-03-31T23:37:05.504 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "secs", 2024-03-31T23:37:05.505 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.505 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.505 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "2419200", 2024-03-31T23:37:05.505 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.505 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.505 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.505 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-03-31T23:37:05.505 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.505 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.505 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.505 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.505 INFO:teuthology.orchestra.run.smithi046.stdout: "pool_name": { 2024-03-31T23:37:05.505 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "pool_name", 2024-03-31T23:37:05.505 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.505 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.506 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.506 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "device_health_metrics", 2024-03-31T23:37:05.506 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.506 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.506 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.506 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "name of pool in which to store device health metrics", 2024-03-31T23:37:05.506 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.506 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.506 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.506 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.506 INFO:teuthology.orchestra.run.smithi046.stdout: "retention_period": { 2024-03-31T23:37:05.506 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "retention_period", 2024-03-31T23:37:05.506 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "secs", 2024-03-31T23:37:05.506 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.507 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.507 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "15552000", 2024-03-31T23:37:05.507 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.507 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.507 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.507 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "how long to retain device health metrics", 2024-03-31T23:37:05.507 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.507 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.507 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.507 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.507 INFO:teuthology.orchestra.run.smithi046.stdout: "scrape_frequency": { 2024-03-31T23:37:05.507 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "scrape_frequency", 2024-03-31T23:37:05.507 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "secs", 2024-03-31T23:37:05.507 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.507 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.508 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "86400", 2024-03-31T23:37:05.508 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.508 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.508 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.508 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "how frequently to scrape device health metrics", 2024-03-31T23:37:05.508 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.508 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.508 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.508 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.508 INFO:teuthology.orchestra.run.smithi046.stdout: "self_heal": { 2024-03-31T23:37:05.508 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "self_heal", 2024-03-31T23:37:05.508 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.508 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.508 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.509 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "True", 2024-03-31T23:37:05.509 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.509 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.509 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.509 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-03-31T23:37:05.509 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.509 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.509 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.509 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.509 INFO:teuthology.orchestra.run.smithi046.stdout: "sleep_interval": { 2024-03-31T23:37:05.509 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "sleep_interval", 2024-03-31T23:37:05.509 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "secs", 2024-03-31T23:37:05.509 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.509 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.509 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "600", 2024-03-31T23:37:05.510 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.510 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.510 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.510 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "how frequently to wake up and check device health", 2024-03-31T23:37:05.510 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.510 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.510 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.510 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.510 INFO:teuthology.orchestra.run.smithi046.stdout: "warn_threshold": { 2024-03-31T23:37:05.510 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "warn_threshold", 2024-03-31T23:37:05.510 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "secs", 2024-03-31T23:37:05.510 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.510 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.510 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "7257600", 2024-03-31T23:37:05.511 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.511 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.511 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.511 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-03-31T23:37:05.511 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.511 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.511 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.511 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-31T23:37:05.511 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-31T23:37:05.511 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.511 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-03-31T23:37:05.511 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "diskprediction_local", 2024-03-31T23:37:05.511 INFO:teuthology.orchestra.run.smithi046.stdout: "can_run": true, 2024-03-31T23:37:05.511 INFO:teuthology.orchestra.run.smithi046.stdout: "error_string": "", 2024-03-31T23:37:05.512 INFO:teuthology.orchestra.run.smithi046.stdout: "module_options": { 2024-03-31T23:37:05.512 INFO:teuthology.orchestra.run.smithi046.stdout: "log_level": { 2024-03-31T23:37:05.512 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_level", 2024-03-31T23:37:05.512 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.512 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.512 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.512 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-31T23:37:05.512 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.512 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.512 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-31T23:37:05.512 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-31T23:37:05.512 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-31T23:37:05.512 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-31T23:37:05.512 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-31T23:37:05.512 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-31T23:37:05.513 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-31T23:37:05.513 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-31T23:37:05.513 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.513 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.513 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.513 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.513 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.513 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster": { 2024-03-31T23:37:05.513 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster", 2024-03-31T23:37:05.513 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.513 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.513 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.513 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-31T23:37:05.513 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.513 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.514 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.514 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.514 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.514 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.514 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.514 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.514 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster_level": { 2024-03-31T23:37:05.514 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster_level", 2024-03-31T23:37:05.514 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.514 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.514 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.514 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "info", 2024-03-31T23:37:05.514 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.515 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.515 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-31T23:37:05.515 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-31T23:37:05.515 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-31T23:37:05.515 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-31T23:37:05.515 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-31T23:37:05.515 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-31T23:37:05.515 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-31T23:37:05.515 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-31T23:37:05.515 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.515 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.515 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.515 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.515 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.516 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_file": { 2024-03-31T23:37:05.516 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_file", 2024-03-31T23:37:05.516 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.516 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.516 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.516 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-31T23:37:05.516 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.516 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.516 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.516 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.516 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.516 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.516 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.516 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.516 INFO:teuthology.orchestra.run.smithi046.stdout: "predict_interval": { 2024-03-31T23:37:05.517 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "predict_interval", 2024-03-31T23:37:05.517 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.517 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.517 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.517 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "86400", 2024-03-31T23:37:05.517 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.517 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.517 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.517 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.517 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.517 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.517 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.517 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.517 INFO:teuthology.orchestra.run.smithi046.stdout: "predictor_model": { 2024-03-31T23:37:05.518 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "predictor_model", 2024-03-31T23:37:05.518 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.518 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.518 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.518 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "prophetstor", 2024-03-31T23:37:05.518 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.518 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.518 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.518 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.518 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.518 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.518 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.518 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.519 INFO:teuthology.orchestra.run.smithi046.stdout: "sleep_interval": { 2024-03-31T23:37:05.519 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "sleep_interval", 2024-03-31T23:37:05.519 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.519 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.519 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.519 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "600", 2024-03-31T23:37:05.519 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.519 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.519 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.519 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.519 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.519 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.519 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.519 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-31T23:37:05.519 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-31T23:37:05.520 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.520 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-03-31T23:37:05.520 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "influx", 2024-03-31T23:37:05.520 INFO:teuthology.orchestra.run.smithi046.stdout: "can_run": false, 2024-03-31T23:37:05.520 INFO:teuthology.orchestra.run.smithi046.stdout: "error_string": "influxdb python module not found", 2024-03-31T23:37:05.520 INFO:teuthology.orchestra.run.smithi046.stdout: "module_options": { 2024-03-31T23:37:05.520 INFO:teuthology.orchestra.run.smithi046.stdout: "batch_size": { 2024-03-31T23:37:05.520 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "batch_size", 2024-03-31T23:37:05.520 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "int", 2024-03-31T23:37:05.520 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.520 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.520 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "5000", 2024-03-31T23:37:05.520 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.520 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.521 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.521 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-03-31T23:37:05.521 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.521 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.521 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.521 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.521 INFO:teuthology.orchestra.run.smithi046.stdout: "database": { 2024-03-31T23:37:05.521 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "database", 2024-03-31T23:37:05.521 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.521 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.521 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.521 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "ceph", 2024-03-31T23:37:05.521 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.521 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.521 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.522 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-03-31T23:37:05.522 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.522 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.522 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.522 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.522 INFO:teuthology.orchestra.run.smithi046.stdout: "hostname": { 2024-03-31T23:37:05.522 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "hostname", 2024-03-31T23:37:05.522 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.522 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.522 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.522 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-31T23:37:05.522 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.522 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.522 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.523 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "InfluxDB server hostname", 2024-03-31T23:37:05.523 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.523 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.523 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.523 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.523 INFO:teuthology.orchestra.run.smithi046.stdout: "interval": { 2024-03-31T23:37:05.523 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "interval", 2024-03-31T23:37:05.523 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "secs", 2024-03-31T23:37:05.523 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.523 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.523 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "30", 2024-03-31T23:37:05.523 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "5", 2024-03-31T23:37:05.523 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.524 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.524 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-03-31T23:37:05.524 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.524 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.524 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.524 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.524 INFO:teuthology.orchestra.run.smithi046.stdout: "log_level": { 2024-03-31T23:37:05.525 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_level", 2024-03-31T23:37:05.525 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.525 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.525 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.525 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-31T23:37:05.525 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.525 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.526 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-31T23:37:05.526 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-31T23:37:05.526 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-31T23:37:05.526 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-31T23:37:05.526 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-31T23:37:05.526 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-31T23:37:05.526 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-31T23:37:05.527 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-31T23:37:05.527 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.527 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.527 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.527 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.527 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.527 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster": { 2024-03-31T23:37:05.528 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster", 2024-03-31T23:37:05.528 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.528 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.528 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.528 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-31T23:37:05.528 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.528 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.529 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.529 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.529 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.529 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.529 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.529 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.529 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster_level": { 2024-03-31T23:37:05.530 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster_level", 2024-03-31T23:37:05.530 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.530 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.530 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.530 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "info", 2024-03-31T23:37:05.530 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.530 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.531 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-31T23:37:05.531 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-31T23:37:05.531 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-31T23:37:05.531 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-31T23:37:05.531 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-31T23:37:05.531 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-31T23:37:05.532 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-31T23:37:05.532 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-31T23:37:05.532 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.532 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.532 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.532 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.532 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.532 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_file": { 2024-03-31T23:37:05.532 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_file", 2024-03-31T23:37:05.532 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.533 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.533 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.533 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-31T23:37:05.533 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.533 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.533 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.533 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.533 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.533 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.533 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.533 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.533 INFO:teuthology.orchestra.run.smithi046.stdout: "password": { 2024-03-31T23:37:05.533 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "password", 2024-03-31T23:37:05.533 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.533 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.534 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.534 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-31T23:37:05.534 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.534 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.534 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.534 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "password of InfluxDB server user", 2024-03-31T23:37:05.534 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.534 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.534 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.534 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.534 INFO:teuthology.orchestra.run.smithi046.stdout: "port": { 2024-03-31T23:37:05.534 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "port", 2024-03-31T23:37:05.534 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "int", 2024-03-31T23:37:05.534 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.535 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.535 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "8086", 2024-03-31T23:37:05.535 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.535 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.535 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.535 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "InfluxDB server port", 2024-03-31T23:37:05.535 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.535 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.535 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.535 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.535 INFO:teuthology.orchestra.run.smithi046.stdout: "ssl": { 2024-03-31T23:37:05.535 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "ssl", 2024-03-31T23:37:05.535 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.535 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.535 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.536 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "false", 2024-03-31T23:37:05.536 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.536 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.536 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.536 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-03-31T23:37:05.536 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.536 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.536 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.536 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.536 INFO:teuthology.orchestra.run.smithi046.stdout: "threads": { 2024-03-31T23:37:05.536 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "threads", 2024-03-31T23:37:05.536 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "int", 2024-03-31T23:37:05.536 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.536 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.536 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "5", 2024-03-31T23:37:05.537 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "1", 2024-03-31T23:37:05.537 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "32", 2024-03-31T23:37:05.537 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.537 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-03-31T23:37:05.537 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.537 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.537 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.537 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.537 INFO:teuthology.orchestra.run.smithi046.stdout: "username": { 2024-03-31T23:37:05.537 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "username", 2024-03-31T23:37:05.537 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.537 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.537 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.537 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-31T23:37:05.537 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.538 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.538 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.538 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "username of InfluxDB server user", 2024-03-31T23:37:05.538 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.538 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.538 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.538 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.538 INFO:teuthology.orchestra.run.smithi046.stdout: "verify_ssl": { 2024-03-31T23:37:05.538 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "verify_ssl", 2024-03-31T23:37:05.538 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.538 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.538 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.538 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "true", 2024-03-31T23:37:05.538 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.539 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.539 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.539 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-03-31T23:37:05.539 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.539 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.539 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.539 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-31T23:37:05.539 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-31T23:37:05.539 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.539 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-03-31T23:37:05.539 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "insights", 2024-03-31T23:37:05.539 INFO:teuthology.orchestra.run.smithi046.stdout: "can_run": true, 2024-03-31T23:37:05.539 INFO:teuthology.orchestra.run.smithi046.stdout: "error_string": "", 2024-03-31T23:37:05.539 INFO:teuthology.orchestra.run.smithi046.stdout: "module_options": { 2024-03-31T23:37:05.539 INFO:teuthology.orchestra.run.smithi046.stdout: "log_level": { 2024-03-31T23:37:05.540 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_level", 2024-03-31T23:37:05.540 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.540 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.540 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.540 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-31T23:37:05.540 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.540 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.540 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-31T23:37:05.540 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-31T23:37:05.540 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-31T23:37:05.540 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-31T23:37:05.540 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-31T23:37:05.540 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-31T23:37:05.540 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-31T23:37:05.540 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-31T23:37:05.541 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.541 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.541 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.541 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.541 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.541 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster": { 2024-03-31T23:37:05.541 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster", 2024-03-31T23:37:05.541 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.541 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.541 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.541 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-31T23:37:05.541 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.541 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.541 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.542 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.542 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.542 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.542 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.542 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.542 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster_level": { 2024-03-31T23:37:05.542 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster_level", 2024-03-31T23:37:05.542 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.542 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.542 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.542 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "info", 2024-03-31T23:37:05.542 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.542 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.542 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-31T23:37:05.543 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-31T23:37:05.543 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-31T23:37:05.543 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-31T23:37:05.543 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-31T23:37:05.543 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-31T23:37:05.543 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-31T23:37:05.543 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-31T23:37:05.543 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.543 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.543 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.543 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.543 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.543 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_file": { 2024-03-31T23:37:05.543 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_file", 2024-03-31T23:37:05.544 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.544 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.544 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.544 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-31T23:37:05.544 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.544 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.544 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.544 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.544 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.544 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.544 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.544 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-31T23:37:05.544 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-31T23:37:05.544 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.544 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-03-31T23:37:05.545 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "iostat", 2024-03-31T23:37:05.545 INFO:teuthology.orchestra.run.smithi046.stdout: "can_run": true, 2024-03-31T23:37:05.545 INFO:teuthology.orchestra.run.smithi046.stdout: "error_string": "", 2024-03-31T23:37:05.545 INFO:teuthology.orchestra.run.smithi046.stdout: "module_options": { 2024-03-31T23:37:05.545 INFO:teuthology.orchestra.run.smithi046.stdout: "log_level": { 2024-03-31T23:37:05.545 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_level", 2024-03-31T23:37:05.545 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.545 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.545 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.545 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-31T23:37:05.545 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.545 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.545 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-31T23:37:05.545 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-31T23:37:05.546 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-31T23:37:05.546 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-31T23:37:05.546 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-31T23:37:05.546 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-31T23:37:05.546 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-31T23:37:05.546 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-31T23:37:05.546 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.546 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.546 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.546 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.546 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.546 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster": { 2024-03-31T23:37:05.546 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster", 2024-03-31T23:37:05.546 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.547 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.547 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.547 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-31T23:37:05.547 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.547 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.547 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.547 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.547 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.547 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.547 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.547 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.547 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster_level": { 2024-03-31T23:37:05.547 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster_level", 2024-03-31T23:37:05.547 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.548 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.548 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.548 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "info", 2024-03-31T23:37:05.548 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.548 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.548 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-31T23:37:05.548 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-31T23:37:05.548 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-31T23:37:05.548 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-31T23:37:05.548 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-31T23:37:05.548 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-31T23:37:05.548 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-31T23:37:05.548 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-31T23:37:05.548 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.548 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.549 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.549 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.549 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.549 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_file": { 2024-03-31T23:37:05.549 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_file", 2024-03-31T23:37:05.549 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.549 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.549 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.549 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-31T23:37:05.549 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.549 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.549 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.549 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.549 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.549 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.550 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.550 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-31T23:37:05.550 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-31T23:37:05.550 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.550 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-03-31T23:37:05.550 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "k8sevents", 2024-03-31T23:37:05.550 INFO:teuthology.orchestra.run.smithi046.stdout: "can_run": true, 2024-03-31T23:37:05.550 INFO:teuthology.orchestra.run.smithi046.stdout: "error_string": "", 2024-03-31T23:37:05.550 INFO:teuthology.orchestra.run.smithi046.stdout: "module_options": { 2024-03-31T23:37:05.550 INFO:teuthology.orchestra.run.smithi046.stdout: "ceph_event_retention_days": { 2024-03-31T23:37:05.550 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "ceph_event_retention_days", 2024-03-31T23:37:05.550 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "int", 2024-03-31T23:37:05.550 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.550 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.551 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "7", 2024-03-31T23:37:05.551 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.551 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.551 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.551 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Days to hold ceph event information within local cache", 2024-03-31T23:37:05.551 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.551 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.551 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.551 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.551 INFO:teuthology.orchestra.run.smithi046.stdout: "config_check_secs": { 2024-03-31T23:37:05.551 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "config_check_secs", 2024-03-31T23:37:05.551 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "int", 2024-03-31T23:37:05.551 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.551 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.552 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "10", 2024-03-31T23:37:05.552 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "10", 2024-03-31T23:37:05.552 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.552 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.552 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-03-31T23:37:05.552 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.552 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.552 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.552 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.552 INFO:teuthology.orchestra.run.smithi046.stdout: "log_level": { 2024-03-31T23:37:05.552 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_level", 2024-03-31T23:37:05.552 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.552 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.552 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.552 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-31T23:37:05.553 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.553 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.553 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-31T23:37:05.553 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-31T23:37:05.553 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-31T23:37:05.553 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-31T23:37:05.553 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-31T23:37:05.553 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-31T23:37:05.553 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-31T23:37:05.553 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-31T23:37:05.553 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.553 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.553 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.553 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.553 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.554 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster": { 2024-03-31T23:37:05.554 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster", 2024-03-31T23:37:05.554 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.554 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.554 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.554 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-31T23:37:05.554 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.554 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.554 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.554 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.554 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.554 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.554 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.554 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.555 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster_level": { 2024-03-31T23:37:05.555 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster_level", 2024-03-31T23:37:05.555 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.555 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.555 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.555 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "info", 2024-03-31T23:37:05.555 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.555 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.555 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-31T23:37:05.555 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-31T23:37:05.555 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-31T23:37:05.555 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-31T23:37:05.555 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-31T23:37:05.556 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-31T23:37:05.556 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-31T23:37:05.556 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-31T23:37:05.556 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.556 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.556 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.556 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.556 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.556 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_file": { 2024-03-31T23:37:05.556 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_file", 2024-03-31T23:37:05.556 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.556 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.556 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.556 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-31T23:37:05.556 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.557 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.557 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.557 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.557 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.557 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.557 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.557 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-31T23:37:05.557 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-31T23:37:05.557 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.557 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-03-31T23:37:05.557 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "localpool", 2024-03-31T23:37:05.557 INFO:teuthology.orchestra.run.smithi046.stdout: "can_run": true, 2024-03-31T23:37:05.557 INFO:teuthology.orchestra.run.smithi046.stdout: "error_string": "", 2024-03-31T23:37:05.558 INFO:teuthology.orchestra.run.smithi046.stdout: "module_options": { 2024-03-31T23:37:05.558 INFO:teuthology.orchestra.run.smithi046.stdout: "failure_domain": { 2024-03-31T23:37:05.558 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "failure_domain", 2024-03-31T23:37:05.558 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.558 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.558 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.558 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "host", 2024-03-31T23:37:05.558 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.558 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.558 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.558 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "failure domain for any created local pool", 2024-03-31T23:37:05.558 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-03-31T23:37:05.558 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.559 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.559 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.559 INFO:teuthology.orchestra.run.smithi046.stdout: "log_level": { 2024-03-31T23:37:05.559 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_level", 2024-03-31T23:37:05.559 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.559 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.559 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.559 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-31T23:37:05.559 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.559 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.559 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-31T23:37:05.559 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-31T23:37:05.559 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-31T23:37:05.559 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-31T23:37:05.559 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-31T23:37:05.560 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-31T23:37:05.560 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-31T23:37:05.560 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-31T23:37:05.560 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.560 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.560 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.560 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.560 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.560 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster": { 2024-03-31T23:37:05.560 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster", 2024-03-31T23:37:05.560 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.560 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.560 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.560 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-31T23:37:05.561 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.561 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.561 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.561 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.561 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.561 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.561 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.561 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.561 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster_level": { 2024-03-31T23:37:05.561 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster_level", 2024-03-31T23:37:05.561 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.561 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.561 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.561 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "info", 2024-03-31T23:37:05.561 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.562 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.562 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-31T23:37:05.562 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-31T23:37:05.562 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-31T23:37:05.562 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-31T23:37:05.562 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-31T23:37:05.562 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-31T23:37:05.562 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-31T23:37:05.562 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-31T23:37:05.562 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.562 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.562 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.562 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.563 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.563 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_file": { 2024-03-31T23:37:05.563 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_file", 2024-03-31T23:37:05.563 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.563 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.563 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.563 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-31T23:37:05.563 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.563 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.563 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.563 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.563 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.563 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.563 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.564 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.564 INFO:teuthology.orchestra.run.smithi046.stdout: "min_size": { 2024-03-31T23:37:05.564 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "min_size", 2024-03-31T23:37:05.564 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "int", 2024-03-31T23:37:05.564 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.564 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.564 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-31T23:37:05.564 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.564 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.564 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.564 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "default min_size for any created local pool", 2024-03-31T23:37:05.564 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-03-31T23:37:05.564 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.564 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.564 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.565 INFO:teuthology.orchestra.run.smithi046.stdout: "num_rep": { 2024-03-31T23:37:05.565 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "num_rep", 2024-03-31T23:37:05.565 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "int", 2024-03-31T23:37:05.565 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.565 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.565 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "3", 2024-03-31T23:37:05.565 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.565 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.565 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.565 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "default replica count for any created local pool", 2024-03-31T23:37:05.565 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.565 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.565 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.565 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.565 INFO:teuthology.orchestra.run.smithi046.stdout: "pg_num": { 2024-03-31T23:37:05.566 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "pg_num", 2024-03-31T23:37:05.566 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "int", 2024-03-31T23:37:05.566 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.566 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.566 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "128", 2024-03-31T23:37:05.566 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.566 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.566 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.566 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "default pg_num for any created local pool", 2024-03-31T23:37:05.566 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.566 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.566 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.566 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.566 INFO:teuthology.orchestra.run.smithi046.stdout: "prefix": { 2024-03-31T23:37:05.567 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "prefix", 2024-03-31T23:37:05.567 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.567 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.567 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.567 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-31T23:37:05.567 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.567 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.567 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.567 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "name prefix for any created local pool", 2024-03-31T23:37:05.567 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.567 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.567 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.567 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.567 INFO:teuthology.orchestra.run.smithi046.stdout: "subtree": { 2024-03-31T23:37:05.568 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "subtree", 2024-03-31T23:37:05.568 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.568 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.568 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.568 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "rack", 2024-03-31T23:37:05.568 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.568 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.568 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.568 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "CRUSH level for which to create a local pool", 2024-03-31T23:37:05.568 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-03-31T23:37:05.568 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.569 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.569 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-31T23:37:05.569 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-31T23:37:05.569 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.569 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-03-31T23:37:05.569 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "mds_autoscaler", 2024-03-31T23:37:05.569 INFO:teuthology.orchestra.run.smithi046.stdout: "can_run": true, 2024-03-31T23:37:05.569 INFO:teuthology.orchestra.run.smithi046.stdout: "error_string": "", 2024-03-31T23:37:05.569 INFO:teuthology.orchestra.run.smithi046.stdout: "module_options": { 2024-03-31T23:37:05.569 INFO:teuthology.orchestra.run.smithi046.stdout: "log_level": { 2024-03-31T23:37:05.569 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_level", 2024-03-31T23:37:05.569 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.569 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.569 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.570 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-31T23:37:05.570 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.570 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.570 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-31T23:37:05.570 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-31T23:37:05.570 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-31T23:37:05.570 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-31T23:37:05.570 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-31T23:37:05.570 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-31T23:37:05.570 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-31T23:37:05.570 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-31T23:37:05.570 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.570 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.570 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.570 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.571 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.571 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster": { 2024-03-31T23:37:05.571 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster", 2024-03-31T23:37:05.571 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.571 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.571 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.571 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-31T23:37:05.571 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.571 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.571 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.571 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.571 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.571 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.571 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.572 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.572 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster_level": { 2024-03-31T23:37:05.572 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster_level", 2024-03-31T23:37:05.572 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.572 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.572 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.572 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "info", 2024-03-31T23:37:05.572 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.572 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.572 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-31T23:37:05.572 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-31T23:37:05.572 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-31T23:37:05.572 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-31T23:37:05.572 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-31T23:37:05.572 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-31T23:37:05.573 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-31T23:37:05.573 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-31T23:37:05.573 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.573 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.573 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.573 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.573 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.573 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_file": { 2024-03-31T23:37:05.573 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_file", 2024-03-31T23:37:05.573 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.573 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.573 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.573 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-31T23:37:05.573 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.573 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.574 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.574 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.574 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.574 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.574 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.574 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-31T23:37:05.574 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-31T23:37:05.574 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.574 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-03-31T23:37:05.574 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "mirroring", 2024-03-31T23:37:05.574 INFO:teuthology.orchestra.run.smithi046.stdout: "can_run": true, 2024-03-31T23:37:05.574 INFO:teuthology.orchestra.run.smithi046.stdout: "error_string": "", 2024-03-31T23:37:05.574 INFO:teuthology.orchestra.run.smithi046.stdout: "module_options": { 2024-03-31T23:37:05.574 INFO:teuthology.orchestra.run.smithi046.stdout: "log_level": { 2024-03-31T23:37:05.575 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_level", 2024-03-31T23:37:05.575 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.575 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.575 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.575 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-31T23:37:05.575 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.575 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.575 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-31T23:37:05.575 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-31T23:37:05.575 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-31T23:37:05.575 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-31T23:37:05.575 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-31T23:37:05.575 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-31T23:37:05.575 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-31T23:37:05.575 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-31T23:37:05.576 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.576 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.576 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.576 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.576 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.576 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster": { 2024-03-31T23:37:05.576 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster", 2024-03-31T23:37:05.576 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.576 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.576 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.576 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-31T23:37:05.576 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.576 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.576 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.576 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.577 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.577 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.577 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.577 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.577 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster_level": { 2024-03-31T23:37:05.577 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster_level", 2024-03-31T23:37:05.577 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.577 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.577 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.577 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "info", 2024-03-31T23:37:05.577 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.577 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.577 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-31T23:37:05.577 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-31T23:37:05.578 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-31T23:37:05.578 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-31T23:37:05.578 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-31T23:37:05.578 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-31T23:37:05.578 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-31T23:37:05.578 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-31T23:37:05.578 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.578 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.578 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.578 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.578 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.578 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_file": { 2024-03-31T23:37:05.578 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_file", 2024-03-31T23:37:05.578 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.579 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.579 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.579 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-31T23:37:05.579 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.579 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.579 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.579 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.579 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.579 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.579 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.579 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-31T23:37:05.579 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-31T23:37:05.579 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.579 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-03-31T23:37:05.580 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "nfs", 2024-03-31T23:37:05.580 INFO:teuthology.orchestra.run.smithi046.stdout: "can_run": true, 2024-03-31T23:37:05.580 INFO:teuthology.orchestra.run.smithi046.stdout: "error_string": "", 2024-03-31T23:37:05.580 INFO:teuthology.orchestra.run.smithi046.stdout: "module_options": { 2024-03-31T23:37:05.580 INFO:teuthology.orchestra.run.smithi046.stdout: "log_level": { 2024-03-31T23:37:05.580 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_level", 2024-03-31T23:37:05.580 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.580 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.580 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.580 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-31T23:37:05.580 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.580 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.580 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-31T23:37:05.580 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-31T23:37:05.580 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-31T23:37:05.581 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-31T23:37:05.581 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-31T23:37:05.581 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-31T23:37:05.581 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-31T23:37:05.581 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-31T23:37:05.581 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.581 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.581 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.581 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.581 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.581 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster": { 2024-03-31T23:37:05.581 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster", 2024-03-31T23:37:05.581 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.581 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.581 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.582 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-31T23:37:05.582 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.582 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.582 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.582 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.582 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.582 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.582 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.582 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.582 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster_level": { 2024-03-31T23:37:05.582 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster_level", 2024-03-31T23:37:05.582 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.582 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.582 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.583 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "info", 2024-03-31T23:37:05.583 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.583 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.583 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-31T23:37:05.583 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-31T23:37:05.583 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-31T23:37:05.583 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-31T23:37:05.583 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-31T23:37:05.583 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-31T23:37:05.583 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-31T23:37:05.583 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-31T23:37:05.583 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.583 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.583 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.584 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.584 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.584 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_file": { 2024-03-31T23:37:05.584 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_file", 2024-03-31T23:37:05.584 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.584 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.584 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.584 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-31T23:37:05.584 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.584 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.584 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.584 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.584 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.584 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.584 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.585 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-31T23:37:05.585 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-31T23:37:05.585 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.585 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-03-31T23:37:05.585 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "orchestrator", 2024-03-31T23:37:05.585 INFO:teuthology.orchestra.run.smithi046.stdout: "can_run": true, 2024-03-31T23:37:05.585 INFO:teuthology.orchestra.run.smithi046.stdout: "error_string": "", 2024-03-31T23:37:05.585 INFO:teuthology.orchestra.run.smithi046.stdout: "module_options": { 2024-03-31T23:37:05.585 INFO:teuthology.orchestra.run.smithi046.stdout: "fail_fs": { 2024-03-31T23:37:05.585 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "fail_fs", 2024-03-31T23:37:05.585 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.585 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.585 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.585 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-31T23:37:05.586 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.586 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.586 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.586 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-03-31T23:37:05.586 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.586 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.586 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.586 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.586 INFO:teuthology.orchestra.run.smithi046.stdout: "log_level": { 2024-03-31T23:37:05.586 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_level", 2024-03-31T23:37:05.586 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.586 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.586 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.586 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-31T23:37:05.587 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.587 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.587 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-31T23:37:05.587 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-31T23:37:05.587 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-31T23:37:05.587 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-31T23:37:05.587 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-31T23:37:05.587 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-31T23:37:05.587 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-31T23:37:05.587 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-31T23:37:05.587 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.587 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.587 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.587 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.588 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.588 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster": { 2024-03-31T23:37:05.588 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster", 2024-03-31T23:37:05.588 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.588 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.588 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.588 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-31T23:37:05.588 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.588 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.588 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.588 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.588 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.588 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.588 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.588 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.589 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster_level": { 2024-03-31T23:37:05.589 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster_level", 2024-03-31T23:37:05.589 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.589 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.589 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.589 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "info", 2024-03-31T23:37:05.589 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.589 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.589 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-31T23:37:05.589 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-31T23:37:05.589 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-31T23:37:05.589 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-31T23:37:05.589 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-31T23:37:05.589 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-31T23:37:05.590 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-31T23:37:05.590 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-31T23:37:05.590 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.590 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.590 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.590 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.590 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.590 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_file": { 2024-03-31T23:37:05.590 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_file", 2024-03-31T23:37:05.590 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.590 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.590 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.590 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-31T23:37:05.590 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.591 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.591 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.591 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.591 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.591 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.591 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.591 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.591 INFO:teuthology.orchestra.run.smithi046.stdout: "orchestrator": { 2024-03-31T23:37:05.591 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "orchestrator", 2024-03-31T23:37:05.591 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.591 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.591 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.591 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-31T23:37:05.591 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.592 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.592 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-31T23:37:05.592 INFO:teuthology.orchestra.run.smithi046.stdout: "cephadm", 2024-03-31T23:37:05.592 INFO:teuthology.orchestra.run.smithi046.stdout: "rook", 2024-03-31T23:37:05.592 INFO:teuthology.orchestra.run.smithi046.stdout: "test_orchestrator" 2024-03-31T23:37:05.592 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-31T23:37:05.592 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Orchestrator backend", 2024-03-31T23:37:05.592 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.592 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.592 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.592 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-31T23:37:05.592 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-31T23:37:05.592 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.592 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-03-31T23:37:05.592 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "osd_perf_query", 2024-03-31T23:37:05.593 INFO:teuthology.orchestra.run.smithi046.stdout: "can_run": true, 2024-03-31T23:37:05.593 INFO:teuthology.orchestra.run.smithi046.stdout: "error_string": "", 2024-03-31T23:37:05.593 INFO:teuthology.orchestra.run.smithi046.stdout: "module_options": { 2024-03-31T23:37:05.593 INFO:teuthology.orchestra.run.smithi046.stdout: "log_level": { 2024-03-31T23:37:05.593 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_level", 2024-03-31T23:37:05.593 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.593 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.593 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.593 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-31T23:37:05.593 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.593 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.593 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-31T23:37:05.593 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-31T23:37:05.593 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-31T23:37:05.593 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-31T23:37:05.594 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-31T23:37:05.594 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-31T23:37:05.594 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-31T23:37:05.594 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-31T23:37:05.594 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.594 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.594 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.594 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.594 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.594 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster": { 2024-03-31T23:37:05.594 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster", 2024-03-31T23:37:05.594 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.594 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.594 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.595 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-31T23:37:05.595 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.595 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.595 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.595 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.595 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.595 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.595 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.595 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.595 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster_level": { 2024-03-31T23:37:05.595 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster_level", 2024-03-31T23:37:05.595 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.595 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.595 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.596 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "info", 2024-03-31T23:37:05.596 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.596 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.596 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-31T23:37:05.596 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-31T23:37:05.596 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-31T23:37:05.596 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-31T23:37:05.596 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-31T23:37:05.596 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-31T23:37:05.596 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-31T23:37:05.596 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-31T23:37:05.596 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.596 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.596 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.597 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.597 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.597 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_file": { 2024-03-31T23:37:05.597 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_file", 2024-03-31T23:37:05.597 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.597 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.597 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.597 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-31T23:37:05.597 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.597 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.597 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.597 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.597 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.597 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.597 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.598 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-31T23:37:05.598 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-31T23:37:05.598 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.598 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-03-31T23:37:05.598 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "osd_support", 2024-03-31T23:37:05.598 INFO:teuthology.orchestra.run.smithi046.stdout: "can_run": true, 2024-03-31T23:37:05.598 INFO:teuthology.orchestra.run.smithi046.stdout: "error_string": "", 2024-03-31T23:37:05.598 INFO:teuthology.orchestra.run.smithi046.stdout: "module_options": { 2024-03-31T23:37:05.598 INFO:teuthology.orchestra.run.smithi046.stdout: "log_level": { 2024-03-31T23:37:05.598 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_level", 2024-03-31T23:37:05.598 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.598 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.598 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.598 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-31T23:37:05.599 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.599 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.599 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-31T23:37:05.599 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-31T23:37:05.599 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-31T23:37:05.599 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-31T23:37:05.599 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-31T23:37:05.599 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-31T23:37:05.599 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-31T23:37:05.599 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-31T23:37:05.599 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.599 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.599 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.599 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.599 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.600 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster": { 2024-03-31T23:37:05.600 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster", 2024-03-31T23:37:05.600 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.600 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.600 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.600 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-31T23:37:05.600 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.600 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.600 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.600 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.600 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.600 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.600 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.600 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.600 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster_level": { 2024-03-31T23:37:05.601 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster_level", 2024-03-31T23:37:05.601 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.601 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.601 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.601 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "info", 2024-03-31T23:37:05.601 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.601 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.601 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-31T23:37:05.601 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-31T23:37:05.601 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-31T23:37:05.601 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-31T23:37:05.601 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-31T23:37:05.601 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-31T23:37:05.601 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-31T23:37:05.602 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-31T23:37:05.602 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.602 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.602 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.602 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.602 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.602 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_file": { 2024-03-31T23:37:05.602 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_file", 2024-03-31T23:37:05.602 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.602 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.602 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.602 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-31T23:37:05.602 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.602 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.603 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.603 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.603 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.603 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.603 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.603 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-31T23:37:05.603 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-31T23:37:05.603 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.603 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-03-31T23:37:05.603 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "pg_autoscaler", 2024-03-31T23:37:05.603 INFO:teuthology.orchestra.run.smithi046.stdout: "can_run": true, 2024-03-31T23:37:05.603 INFO:teuthology.orchestra.run.smithi046.stdout: "error_string": "", 2024-03-31T23:37:05.603 INFO:teuthology.orchestra.run.smithi046.stdout: "module_options": { 2024-03-31T23:37:05.603 INFO:teuthology.orchestra.run.smithi046.stdout: "log_level": { 2024-03-31T23:37:05.603 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_level", 2024-03-31T23:37:05.604 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.604 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.604 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.604 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-31T23:37:05.604 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.604 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.604 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-31T23:37:05.604 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-31T23:37:05.604 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-31T23:37:05.604 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-31T23:37:05.604 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-31T23:37:05.604 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-31T23:37:05.604 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-31T23:37:05.604 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-31T23:37:05.604 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.605 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.605 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.605 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.605 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.605 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster": { 2024-03-31T23:37:05.605 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster", 2024-03-31T23:37:05.605 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.605 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.605 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.605 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-31T23:37:05.605 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.605 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.605 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.605 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.605 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.606 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.606 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.606 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.606 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster_level": { 2024-03-31T23:37:05.606 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster_level", 2024-03-31T23:37:05.606 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.606 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.606 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.606 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "info", 2024-03-31T23:37:05.606 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.606 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.606 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-31T23:37:05.606 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-31T23:37:05.606 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-31T23:37:05.607 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-31T23:37:05.607 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-31T23:37:05.607 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-31T23:37:05.607 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-31T23:37:05.607 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-31T23:37:05.607 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.607 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.607 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.607 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.607 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.607 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_file": { 2024-03-31T23:37:05.607 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_file", 2024-03-31T23:37:05.607 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.607 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.607 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.608 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-31T23:37:05.608 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.608 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.608 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.608 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.608 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.608 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.608 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.608 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.608 INFO:teuthology.orchestra.run.smithi046.stdout: "sleep_interval": { 2024-03-31T23:37:05.608 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "sleep_interval", 2024-03-31T23:37:05.608 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "secs", 2024-03-31T23:37:05.608 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.608 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.608 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "60", 2024-03-31T23:37:05.608 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.609 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.609 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.609 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.609 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.609 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.609 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.609 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.609 INFO:teuthology.orchestra.run.smithi046.stdout: "threshold": { 2024-03-31T23:37:05.609 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "threshold", 2024-03-31T23:37:05.609 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "float", 2024-03-31T23:37:05.609 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.609 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.609 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "3.0", 2024-03-31T23:37:05.609 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "1.0", 2024-03-31T23:37:05.610 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.610 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.610 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "scaling threshold", 2024-03-31T23:37:05.610 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-03-31T23:37:05.610 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.610 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.610 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-31T23:37:05.610 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-31T23:37:05.610 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.610 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-03-31T23:37:05.610 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "progress", 2024-03-31T23:37:05.610 INFO:teuthology.orchestra.run.smithi046.stdout: "can_run": true, 2024-03-31T23:37:05.610 INFO:teuthology.orchestra.run.smithi046.stdout: "error_string": "", 2024-03-31T23:37:05.610 INFO:teuthology.orchestra.run.smithi046.stdout: "module_options": { 2024-03-31T23:37:05.610 INFO:teuthology.orchestra.run.smithi046.stdout: "allow_pg_recovery_event": { 2024-03-31T23:37:05.611 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "allow_pg_recovery_event", 2024-03-31T23:37:05.611 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.611 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.611 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.611 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-31T23:37:05.611 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.611 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.611 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.611 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "allow the module to show pg recovery progress", 2024-03-31T23:37:05.611 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.611 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.611 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.611 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.611 INFO:teuthology.orchestra.run.smithi046.stdout: "enabled": { 2024-03-31T23:37:05.612 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "enabled", 2024-03-31T23:37:05.612 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.612 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.612 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.612 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "True", 2024-03-31T23:37:05.612 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.612 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.612 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.612 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.612 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.612 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.612 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.612 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.612 INFO:teuthology.orchestra.run.smithi046.stdout: "log_level": { 2024-03-31T23:37:05.612 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_level", 2024-03-31T23:37:05.613 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.613 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.613 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.613 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-31T23:37:05.613 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.613 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.613 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-31T23:37:05.613 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-31T23:37:05.613 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-31T23:37:05.613 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-31T23:37:05.613 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-31T23:37:05.613 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-31T23:37:05.613 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-31T23:37:05.613 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-31T23:37:05.613 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.614 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.614 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.614 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.614 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.614 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster": { 2024-03-31T23:37:05.614 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster", 2024-03-31T23:37:05.614 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.614 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.614 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.614 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-31T23:37:05.614 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.614 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.614 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.614 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.614 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.615 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.615 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.615 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.615 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster_level": { 2024-03-31T23:37:05.615 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster_level", 2024-03-31T23:37:05.615 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.615 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.615 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.615 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "info", 2024-03-31T23:37:05.615 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.615 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.615 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-31T23:37:05.615 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-31T23:37:05.615 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-31T23:37:05.616 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-31T23:37:05.616 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-31T23:37:05.616 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-31T23:37:05.616 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-31T23:37:05.616 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-31T23:37:05.616 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.616 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.616 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.616 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.616 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.616 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_file": { 2024-03-31T23:37:05.616 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_file", 2024-03-31T23:37:05.616 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.616 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.617 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.617 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-31T23:37:05.617 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.617 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.617 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.617 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.617 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.617 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.617 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.617 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.617 INFO:teuthology.orchestra.run.smithi046.stdout: "max_completed_events": { 2024-03-31T23:37:05.617 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "max_completed_events", 2024-03-31T23:37:05.617 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "int", 2024-03-31T23:37:05.617 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.617 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.618 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "50", 2024-03-31T23:37:05.618 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.618 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.618 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.618 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "number of past completed events to remember", 2024-03-31T23:37:05.618 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.618 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.618 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.618 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.618 INFO:teuthology.orchestra.run.smithi046.stdout: "sleep_interval": { 2024-03-31T23:37:05.618 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "sleep_interval", 2024-03-31T23:37:05.618 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "secs", 2024-03-31T23:37:05.618 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.619 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.619 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "5", 2024-03-31T23:37:05.619 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.619 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.619 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.619 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "how long the module is going to sleep", 2024-03-31T23:37:05.619 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.619 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.619 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.619 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-31T23:37:05.619 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-31T23:37:05.619 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.619 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-03-31T23:37:05.619 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "prometheus", 2024-03-31T23:37:05.619 INFO:teuthology.orchestra.run.smithi046.stdout: "can_run": true, 2024-03-31T23:37:05.620 INFO:teuthology.orchestra.run.smithi046.stdout: "error_string": "", 2024-03-31T23:37:05.620 INFO:teuthology.orchestra.run.smithi046.stdout: "module_options": { 2024-03-31T23:37:05.620 INFO:teuthology.orchestra.run.smithi046.stdout: "cache": { 2024-03-31T23:37:05.620 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "cache", 2024-03-31T23:37:05.620 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.620 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.620 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.620 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "True", 2024-03-31T23:37:05.620 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.620 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.620 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.620 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.620 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.620 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.620 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.621 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.621 INFO:teuthology.orchestra.run.smithi046.stdout: "exclude_perf_counters": { 2024-03-31T23:37:05.621 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "exclude_perf_counters", 2024-03-31T23:37:05.621 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.621 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.621 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.621 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "True", 2024-03-31T23:37:05.621 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.621 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.621 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.621 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-03-31T23:37:05.621 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-03-31T23:37:05.621 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.621 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.622 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.622 INFO:teuthology.orchestra.run.smithi046.stdout: "log_level": { 2024-03-31T23:37:05.622 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_level", 2024-03-31T23:37:05.622 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.622 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.622 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.622 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-31T23:37:05.622 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.622 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.622 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-31T23:37:05.622 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-31T23:37:05.622 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-31T23:37:05.622 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-31T23:37:05.622 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-31T23:37:05.623 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-31T23:37:05.623 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-31T23:37:05.623 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-31T23:37:05.623 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.623 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.623 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.623 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.623 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.623 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster": { 2024-03-31T23:37:05.623 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster", 2024-03-31T23:37:05.623 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.623 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.623 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.623 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-31T23:37:05.624 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.624 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.624 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.624 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.624 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.624 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.624 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.624 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.624 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster_level": { 2024-03-31T23:37:05.624 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster_level", 2024-03-31T23:37:05.624 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.624 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.624 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.624 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "info", 2024-03-31T23:37:05.624 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.625 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.625 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-31T23:37:05.625 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-31T23:37:05.625 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-31T23:37:05.625 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-31T23:37:05.625 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-31T23:37:05.625 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-31T23:37:05.625 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-31T23:37:05.625 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-31T23:37:05.625 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.625 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.625 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.625 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.625 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.626 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_file": { 2024-03-31T23:37:05.626 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_file", 2024-03-31T23:37:05.626 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.626 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.626 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.626 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-31T23:37:05.626 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.626 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.626 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.626 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.626 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.626 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.626 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.626 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.627 INFO:teuthology.orchestra.run.smithi046.stdout: "rbd_stats_pools": { 2024-03-31T23:37:05.627 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "rbd_stats_pools", 2024-03-31T23:37:05.627 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.627 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.627 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.627 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-31T23:37:05.627 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.627 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.627 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.627 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.627 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.627 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.627 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.627 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.627 INFO:teuthology.orchestra.run.smithi046.stdout: "rbd_stats_pools_refresh_interval": { 2024-03-31T23:37:05.628 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-03-31T23:37:05.628 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "int", 2024-03-31T23:37:05.628 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.628 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.628 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "300", 2024-03-31T23:37:05.628 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.628 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.628 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.628 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.628 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.628 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.628 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.628 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.628 INFO:teuthology.orchestra.run.smithi046.stdout: "scrape_interval": { 2024-03-31T23:37:05.629 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "scrape_interval", 2024-03-31T23:37:05.629 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "float", 2024-03-31T23:37:05.629 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.629 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.629 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "15.0", 2024-03-31T23:37:05.629 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.629 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.629 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.629 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.629 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.629 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.630 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.630 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.630 INFO:teuthology.orchestra.run.smithi046.stdout: "server_addr": { 2024-03-31T23:37:05.630 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "server_addr", 2024-03-31T23:37:05.630 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.630 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.630 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.631 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "::", 2024-03-31T23:37:05.631 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.631 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.631 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.631 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-03-31T23:37:05.631 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.631 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.631 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.631 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.631 INFO:teuthology.orchestra.run.smithi046.stdout: "server_port": { 2024-03-31T23:37:05.631 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "server_port", 2024-03-31T23:37:05.632 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "int", 2024-03-31T23:37:05.632 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.632 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.632 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "9283", 2024-03-31T23:37:05.632 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.632 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.632 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.632 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-03-31T23:37:05.632 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.632 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.632 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.632 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.632 INFO:teuthology.orchestra.run.smithi046.stdout: "stale_cache_strategy": { 2024-03-31T23:37:05.632 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "stale_cache_strategy", 2024-03-31T23:37:05.633 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.633 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.633 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.633 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "log", 2024-03-31T23:37:05.633 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.633 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.633 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.633 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.633 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.633 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.633 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.633 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.633 INFO:teuthology.orchestra.run.smithi046.stdout: "standby_behaviour": { 2024-03-31T23:37:05.633 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "standby_behaviour", 2024-03-31T23:37:05.634 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.634 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.634 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.634 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "default", 2024-03-31T23:37:05.634 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.634 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.634 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-31T23:37:05.634 INFO:teuthology.orchestra.run.smithi046.stdout: "default", 2024-03-31T23:37:05.634 INFO:teuthology.orchestra.run.smithi046.stdout: "error" 2024-03-31T23:37:05.634 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-31T23:37:05.634 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.634 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.634 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.634 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.635 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.635 INFO:teuthology.orchestra.run.smithi046.stdout: "standby_error_status_code": { 2024-03-31T23:37:05.635 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "standby_error_status_code", 2024-03-31T23:37:05.635 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "int", 2024-03-31T23:37:05.635 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.635 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.635 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "500", 2024-03-31T23:37:05.635 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "400", 2024-03-31T23:37:05.635 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "599", 2024-03-31T23:37:05.635 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.635 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.635 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.636 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.636 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.636 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-31T23:37:05.636 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-31T23:37:05.636 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.636 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-03-31T23:37:05.636 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "rbd_support", 2024-03-31T23:37:05.636 INFO:teuthology.orchestra.run.smithi046.stdout: "can_run": true, 2024-03-31T23:37:05.636 INFO:teuthology.orchestra.run.smithi046.stdout: "error_string": "", 2024-03-31T23:37:05.636 INFO:teuthology.orchestra.run.smithi046.stdout: "module_options": { 2024-03-31T23:37:05.636 INFO:teuthology.orchestra.run.smithi046.stdout: "log_level": { 2024-03-31T23:37:05.636 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_level", 2024-03-31T23:37:05.636 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.637 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.637 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.637 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-31T23:37:05.637 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.637 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.637 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-31T23:37:05.637 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-31T23:37:05.637 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-31T23:37:05.637 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-31T23:37:05.637 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-31T23:37:05.637 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-31T23:37:05.637 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-31T23:37:05.637 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-31T23:37:05.637 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.638 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.638 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.638 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.638 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.638 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster": { 2024-03-31T23:37:05.638 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster", 2024-03-31T23:37:05.638 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.638 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.638 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.638 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-31T23:37:05.638 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.638 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.638 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.638 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.639 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.639 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.639 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.639 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.639 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster_level": { 2024-03-31T23:37:05.639 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster_level", 2024-03-31T23:37:05.639 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.639 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.640 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.640 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "info", 2024-03-31T23:37:05.640 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.640 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.640 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-31T23:37:05.640 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-31T23:37:05.640 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-31T23:37:05.640 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-31T23:37:05.640 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-31T23:37:05.640 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-31T23:37:05.640 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-31T23:37:05.640 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-31T23:37:05.640 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.641 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.641 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.641 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.641 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.641 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_file": { 2024-03-31T23:37:05.641 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_file", 2024-03-31T23:37:05.641 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.641 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.641 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.641 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-31T23:37:05.641 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.641 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.641 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.642 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.642 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.642 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.642 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.642 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.642 INFO:teuthology.orchestra.run.smithi046.stdout: "max_concurrent_snap_create": { 2024-03-31T23:37:05.642 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "max_concurrent_snap_create", 2024-03-31T23:37:05.642 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "int", 2024-03-31T23:37:05.642 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.642 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.642 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "10", 2024-03-31T23:37:05.642 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.642 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.642 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.642 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.643 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.643 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.643 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.643 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.643 INFO:teuthology.orchestra.run.smithi046.stdout: "mirror_snapshot_schedule": { 2024-03-31T23:37:05.643 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "mirror_snapshot_schedule", 2024-03-31T23:37:05.643 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.643 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.643 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.643 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-31T23:37:05.643 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.643 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.643 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.644 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.644 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.644 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.644 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.644 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.644 INFO:teuthology.orchestra.run.smithi046.stdout: "trash_purge_schedule": { 2024-03-31T23:37:05.644 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "trash_purge_schedule", 2024-03-31T23:37:05.644 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.644 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.644 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.644 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-31T23:37:05.644 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.644 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.644 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.645 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.645 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.645 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.645 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.645 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-31T23:37:05.645 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-31T23:37:05.645 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.645 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-03-31T23:37:05.645 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "restful", 2024-03-31T23:37:05.645 INFO:teuthology.orchestra.run.smithi046.stdout: "can_run": true, 2024-03-31T23:37:05.645 INFO:teuthology.orchestra.run.smithi046.stdout: "error_string": "", 2024-03-31T23:37:05.645 INFO:teuthology.orchestra.run.smithi046.stdout: "module_options": { 2024-03-31T23:37:05.645 INFO:teuthology.orchestra.run.smithi046.stdout: "enable_auth": { 2024-03-31T23:37:05.646 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "enable_auth", 2024-03-31T23:37:05.646 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.646 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.646 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.646 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "True", 2024-03-31T23:37:05.646 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.646 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.646 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.646 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.646 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.646 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.646 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.646 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.646 INFO:teuthology.orchestra.run.smithi046.stdout: "key_file": { 2024-03-31T23:37:05.646 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "key_file", 2024-03-31T23:37:05.647 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.647 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.647 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.647 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-31T23:37:05.647 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.647 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.647 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.647 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.647 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.647 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.647 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.647 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.647 INFO:teuthology.orchestra.run.smithi046.stdout: "log_level": { 2024-03-31T23:37:05.647 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_level", 2024-03-31T23:37:05.648 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.648 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.648 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.648 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-31T23:37:05.648 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.648 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.648 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-31T23:37:05.648 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-31T23:37:05.648 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-31T23:37:05.648 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-31T23:37:05.648 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-31T23:37:05.648 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-31T23:37:05.648 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-31T23:37:05.648 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-31T23:37:05.648 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.649 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.649 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.649 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.649 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.649 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster": { 2024-03-31T23:37:05.649 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster", 2024-03-31T23:37:05.649 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.649 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.649 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.649 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-31T23:37:05.649 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.649 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.649 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.649 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.649 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.650 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.650 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.650 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.650 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster_level": { 2024-03-31T23:37:05.650 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster_level", 2024-03-31T23:37:05.650 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.650 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.650 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.650 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "info", 2024-03-31T23:37:05.650 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.650 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.650 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-31T23:37:05.650 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-31T23:37:05.651 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-31T23:37:05.651 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-31T23:37:05.651 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-31T23:37:05.651 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-31T23:37:05.651 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-31T23:37:05.651 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-31T23:37:05.651 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.651 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.651 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.651 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.651 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.651 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_file": { 2024-03-31T23:37:05.651 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_file", 2024-03-31T23:37:05.651 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.651 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.652 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.652 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-31T23:37:05.652 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.652 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.652 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.652 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.652 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.652 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.652 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.652 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.652 INFO:teuthology.orchestra.run.smithi046.stdout: "server_addr": { 2024-03-31T23:37:05.652 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "server_addr", 2024-03-31T23:37:05.652 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.652 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.652 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.653 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-31T23:37:05.653 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.653 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.653 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.653 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.653 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.653 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.653 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.653 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.653 INFO:teuthology.orchestra.run.smithi046.stdout: "server_port": { 2024-03-31T23:37:05.653 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "server_port", 2024-03-31T23:37:05.653 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.653 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.653 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.654 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-31T23:37:05.654 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.654 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.654 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.654 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.654 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.654 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.654 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.654 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-31T23:37:05.654 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-31T23:37:05.654 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.654 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-03-31T23:37:05.654 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "rgw", 2024-03-31T23:37:05.654 INFO:teuthology.orchestra.run.smithi046.stdout: "can_run": true, 2024-03-31T23:37:05.655 INFO:teuthology.orchestra.run.smithi046.stdout: "error_string": "", 2024-03-31T23:37:05.655 INFO:teuthology.orchestra.run.smithi046.stdout: "module_options": { 2024-03-31T23:37:05.655 INFO:teuthology.orchestra.run.smithi046.stdout: "log_level": { 2024-03-31T23:37:05.655 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_level", 2024-03-31T23:37:05.655 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.655 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.655 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.655 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-31T23:37:05.655 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.655 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.655 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-31T23:37:05.655 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-31T23:37:05.655 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-31T23:37:05.655 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-31T23:37:05.655 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-31T23:37:05.656 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-31T23:37:05.656 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-31T23:37:05.656 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-31T23:37:05.656 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.656 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.656 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.656 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.656 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.656 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster": { 2024-03-31T23:37:05.656 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster", 2024-03-31T23:37:05.656 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.656 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.656 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.656 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-31T23:37:05.657 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.657 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.657 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.657 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.657 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.657 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.657 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.657 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.657 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster_level": { 2024-03-31T23:37:05.657 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster_level", 2024-03-31T23:37:05.657 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.657 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.657 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.657 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "info", 2024-03-31T23:37:05.657 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.657 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.658 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-31T23:37:05.658 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-31T23:37:05.658 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-31T23:37:05.658 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-31T23:37:05.658 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-31T23:37:05.658 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-31T23:37:05.658 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-31T23:37:05.658 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-31T23:37:05.658 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.658 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.658 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.658 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.658 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.658 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_file": { 2024-03-31T23:37:05.659 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_file", 2024-03-31T23:37:05.659 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.659 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.659 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.659 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-31T23:37:05.659 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.659 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.659 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.659 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.659 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.659 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.659 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.659 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-31T23:37:05.659 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-31T23:37:05.659 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.660 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-03-31T23:37:05.660 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "rook", 2024-03-31T23:37:05.660 INFO:teuthology.orchestra.run.smithi046.stdout: "can_run": true, 2024-03-31T23:37:05.660 INFO:teuthology.orchestra.run.smithi046.stdout: "error_string": "", 2024-03-31T23:37:05.660 INFO:teuthology.orchestra.run.smithi046.stdout: "module_options": { 2024-03-31T23:37:05.660 INFO:teuthology.orchestra.run.smithi046.stdout: "log_level": { 2024-03-31T23:37:05.660 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_level", 2024-03-31T23:37:05.660 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.660 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.660 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.660 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-31T23:37:05.660 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.660 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.660 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-31T23:37:05.661 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-31T23:37:05.661 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-31T23:37:05.661 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-31T23:37:05.661 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-31T23:37:05.661 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-31T23:37:05.661 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-31T23:37:05.661 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-31T23:37:05.661 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.661 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.661 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.661 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.661 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.661 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster": { 2024-03-31T23:37:05.661 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster", 2024-03-31T23:37:05.661 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.661 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.662 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.662 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-31T23:37:05.662 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.662 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.662 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.662 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.662 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.662 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.662 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.662 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.662 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster_level": { 2024-03-31T23:37:05.662 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster_level", 2024-03-31T23:37:05.662 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.662 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.663 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.663 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "info", 2024-03-31T23:37:05.663 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.663 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.663 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-31T23:37:05.663 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-31T23:37:05.663 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-31T23:37:05.663 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-31T23:37:05.663 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-31T23:37:05.663 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-31T23:37:05.663 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-31T23:37:05.663 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-31T23:37:05.663 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.663 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.663 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.664 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.664 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.664 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_file": { 2024-03-31T23:37:05.664 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_file", 2024-03-31T23:37:05.664 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.664 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.664 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.664 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-31T23:37:05.664 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.664 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.664 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.664 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.664 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.664 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.665 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.665 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.665 INFO:teuthology.orchestra.run.smithi046.stdout: "storage_class": { 2024-03-31T23:37:05.665 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "storage_class", 2024-03-31T23:37:05.665 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.665 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.665 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.665 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "local", 2024-03-31T23:37:05.665 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.665 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.665 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.665 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-03-31T23:37:05.665 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.665 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.665 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.666 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-31T23:37:05.666 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-31T23:37:05.666 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.666 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-03-31T23:37:05.666 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "selftest", 2024-03-31T23:37:05.666 INFO:teuthology.orchestra.run.smithi046.stdout: "can_run": true, 2024-03-31T23:37:05.666 INFO:teuthology.orchestra.run.smithi046.stdout: "error_string": "", 2024-03-31T23:37:05.666 INFO:teuthology.orchestra.run.smithi046.stdout: "module_options": { 2024-03-31T23:37:05.666 INFO:teuthology.orchestra.run.smithi046.stdout: "log_level": { 2024-03-31T23:37:05.666 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_level", 2024-03-31T23:37:05.666 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.666 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.666 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.666 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-31T23:37:05.666 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.667 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.667 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-31T23:37:05.667 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-31T23:37:05.667 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-31T23:37:05.667 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-31T23:37:05.667 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-31T23:37:05.667 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-31T23:37:05.667 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-31T23:37:05.667 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-31T23:37:05.667 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.667 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.667 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.667 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.667 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.668 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster": { 2024-03-31T23:37:05.668 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster", 2024-03-31T23:37:05.668 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.668 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.668 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.668 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-31T23:37:05.668 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.668 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.668 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.668 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.668 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.668 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.668 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.668 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.668 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster_level": { 2024-03-31T23:37:05.669 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster_level", 2024-03-31T23:37:05.669 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.669 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.669 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.669 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "info", 2024-03-31T23:37:05.669 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.669 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.669 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-31T23:37:05.669 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-31T23:37:05.669 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-31T23:37:05.669 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-31T23:37:05.669 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-31T23:37:05.669 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-31T23:37:05.669 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-31T23:37:05.670 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-31T23:37:05.670 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.670 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.670 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.670 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.670 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.670 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_file": { 2024-03-31T23:37:05.670 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_file", 2024-03-31T23:37:05.670 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.670 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.670 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.670 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-31T23:37:05.670 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.670 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.670 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.671 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.671 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.671 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.671 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.671 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.671 INFO:teuthology.orchestra.run.smithi046.stdout: "roption1": { 2024-03-31T23:37:05.671 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "roption1", 2024-03-31T23:37:05.671 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.671 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.671 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.671 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-31T23:37:05.671 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.671 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.671 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.672 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.672 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.672 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.672 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.672 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.672 INFO:teuthology.orchestra.run.smithi046.stdout: "roption2": { 2024-03-31T23:37:05.672 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "roption2", 2024-03-31T23:37:05.672 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.672 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.672 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.672 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "xyz", 2024-03-31T23:37:05.672 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.672 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.672 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.673 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.673 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.673 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.673 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.673 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.673 INFO:teuthology.orchestra.run.smithi046.stdout: "rwoption1": { 2024-03-31T23:37:05.673 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "rwoption1", 2024-03-31T23:37:05.673 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.673 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.673 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.673 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-31T23:37:05.673 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.673 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.673 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.674 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.674 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.674 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.674 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.674 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.674 INFO:teuthology.orchestra.run.smithi046.stdout: "rwoption2": { 2024-03-31T23:37:05.674 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "rwoption2", 2024-03-31T23:37:05.674 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "int", 2024-03-31T23:37:05.674 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.674 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.674 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-31T23:37:05.674 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.674 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.674 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.675 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.675 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.675 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.675 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.675 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.675 INFO:teuthology.orchestra.run.smithi046.stdout: "rwoption3": { 2024-03-31T23:37:05.675 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "rwoption3", 2024-03-31T23:37:05.675 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "float", 2024-03-31T23:37:05.675 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.675 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.675 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-31T23:37:05.675 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.675 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.676 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.676 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.676 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.676 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.676 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.676 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.676 INFO:teuthology.orchestra.run.smithi046.stdout: "rwoption4": { 2024-03-31T23:37:05.676 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "rwoption4", 2024-03-31T23:37:05.676 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.676 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.676 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.676 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-31T23:37:05.676 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.676 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.676 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.677 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.677 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.677 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.677 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.677 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.677 INFO:teuthology.orchestra.run.smithi046.stdout: "rwoption5": { 2024-03-31T23:37:05.677 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "rwoption5", 2024-03-31T23:37:05.677 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.677 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.677 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.677 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-31T23:37:05.677 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.677 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.677 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.678 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.678 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.678 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.678 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.678 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.678 INFO:teuthology.orchestra.run.smithi046.stdout: "rwoption6": { 2024-03-31T23:37:05.678 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "rwoption6", 2024-03-31T23:37:05.678 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.678 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.678 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.678 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "True", 2024-03-31T23:37:05.678 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.678 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.679 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.679 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.679 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.679 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.679 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.679 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.679 INFO:teuthology.orchestra.run.smithi046.stdout: "rwoption7": { 2024-03-31T23:37:05.679 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "rwoption7", 2024-03-31T23:37:05.679 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "int", 2024-03-31T23:37:05.679 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.679 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.679 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-31T23:37:05.679 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "1", 2024-03-31T23:37:05.679 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "42", 2024-03-31T23:37:05.679 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.680 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.680 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.680 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.680 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.680 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.680 INFO:teuthology.orchestra.run.smithi046.stdout: "testkey": { 2024-03-31T23:37:05.680 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "testkey", 2024-03-31T23:37:05.680 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.680 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.680 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.680 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-31T23:37:05.680 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.680 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.680 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.681 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.681 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.681 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.681 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.681 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.681 INFO:teuthology.orchestra.run.smithi046.stdout: "testlkey": { 2024-03-31T23:37:05.681 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "testlkey", 2024-03-31T23:37:05.681 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.681 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.681 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.681 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-31T23:37:05.681 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.681 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.681 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.682 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.682 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.682 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.682 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.682 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.682 INFO:teuthology.orchestra.run.smithi046.stdout: "testnewline": { 2024-03-31T23:37:05.682 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "testnewline", 2024-03-31T23:37:05.682 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.682 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.682 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.682 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-31T23:37:05.682 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.682 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.682 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.682 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.683 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.683 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.683 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.683 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-31T23:37:05.683 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-31T23:37:05.683 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.683 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-03-31T23:37:05.683 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "snap_schedule", 2024-03-31T23:37:05.683 INFO:teuthology.orchestra.run.smithi046.stdout: "can_run": true, 2024-03-31T23:37:05.683 INFO:teuthology.orchestra.run.smithi046.stdout: "error_string": "", 2024-03-31T23:37:05.683 INFO:teuthology.orchestra.run.smithi046.stdout: "module_options": { 2024-03-31T23:37:05.683 INFO:teuthology.orchestra.run.smithi046.stdout: "allow_m_granularity": { 2024-03-31T23:37:05.683 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "allow_m_granularity", 2024-03-31T23:37:05.683 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.684 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.684 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.684 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-31T23:37:05.684 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.684 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.684 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.684 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "allow minute scheduled snapshots", 2024-03-31T23:37:05.684 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.684 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.684 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.684 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.684 INFO:teuthology.orchestra.run.smithi046.stdout: "dump_on_update": { 2024-03-31T23:37:05.684 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "dump_on_update", 2024-03-31T23:37:05.684 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.684 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.685 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.685 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-31T23:37:05.685 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.685 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.685 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.685 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "dump database to debug log on update", 2024-03-31T23:37:05.685 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.685 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.685 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.685 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.685 INFO:teuthology.orchestra.run.smithi046.stdout: "log_level": { 2024-03-31T23:37:05.685 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_level", 2024-03-31T23:37:05.685 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.686 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.686 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.686 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-31T23:37:05.686 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.686 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.686 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-31T23:37:05.686 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-31T23:37:05.686 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-31T23:37:05.686 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-31T23:37:05.686 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-31T23:37:05.686 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-31T23:37:05.686 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-31T23:37:05.686 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-31T23:37:05.687 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.687 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.687 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.687 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.687 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.687 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster": { 2024-03-31T23:37:05.687 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster", 2024-03-31T23:37:05.687 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.687 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.687 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.687 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-31T23:37:05.687 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.687 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.688 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.688 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.688 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.688 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.688 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.688 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.688 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster_level": { 2024-03-31T23:37:05.688 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster_level", 2024-03-31T23:37:05.688 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.688 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.688 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.688 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "info", 2024-03-31T23:37:05.688 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.688 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.689 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-31T23:37:05.689 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-31T23:37:05.689 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-31T23:37:05.689 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-31T23:37:05.689 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-31T23:37:05.689 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-31T23:37:05.689 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-31T23:37:05.689 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-31T23:37:05.689 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.689 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.689 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.689 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.689 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.689 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_file": { 2024-03-31T23:37:05.690 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_file", 2024-03-31T23:37:05.690 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.690 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.690 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.690 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-31T23:37:05.690 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.690 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.690 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.690 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.690 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.690 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.690 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.690 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-31T23:37:05.690 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-31T23:37:05.691 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.691 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-03-31T23:37:05.691 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "stats", 2024-03-31T23:37:05.691 INFO:teuthology.orchestra.run.smithi046.stdout: "can_run": true, 2024-03-31T23:37:05.691 INFO:teuthology.orchestra.run.smithi046.stdout: "error_string": "", 2024-03-31T23:37:05.691 INFO:teuthology.orchestra.run.smithi046.stdout: "module_options": { 2024-03-31T23:37:05.691 INFO:teuthology.orchestra.run.smithi046.stdout: "log_level": { 2024-03-31T23:37:05.691 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_level", 2024-03-31T23:37:05.691 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.691 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.691 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.691 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-31T23:37:05.691 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.691 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.692 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-31T23:37:05.692 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-31T23:37:05.692 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-31T23:37:05.692 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-31T23:37:05.692 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-31T23:37:05.692 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-31T23:37:05.692 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-31T23:37:05.692 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-31T23:37:05.692 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.692 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.692 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.692 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.692 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.692 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster": { 2024-03-31T23:37:05.693 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster", 2024-03-31T23:37:05.693 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.693 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.693 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.693 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-31T23:37:05.693 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.693 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.693 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.693 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.693 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.693 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.693 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.693 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.693 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster_level": { 2024-03-31T23:37:05.694 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster_level", 2024-03-31T23:37:05.694 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.694 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.694 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.694 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "info", 2024-03-31T23:37:05.694 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.694 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.694 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-31T23:37:05.694 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-31T23:37:05.694 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-31T23:37:05.694 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-31T23:37:05.694 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-31T23:37:05.694 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-31T23:37:05.694 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-31T23:37:05.694 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-31T23:37:05.695 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.695 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.695 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.695 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.695 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.695 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_file": { 2024-03-31T23:37:05.695 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_file", 2024-03-31T23:37:05.695 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.695 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.695 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.695 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-31T23:37:05.695 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.695 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.695 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.695 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.696 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.696 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.696 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.696 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-31T23:37:05.696 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-31T23:37:05.696 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.696 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-03-31T23:37:05.696 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "status", 2024-03-31T23:37:05.696 INFO:teuthology.orchestra.run.smithi046.stdout: "can_run": true, 2024-03-31T23:37:05.696 INFO:teuthology.orchestra.run.smithi046.stdout: "error_string": "", 2024-03-31T23:37:05.696 INFO:teuthology.orchestra.run.smithi046.stdout: "module_options": { 2024-03-31T23:37:05.696 INFO:teuthology.orchestra.run.smithi046.stdout: "log_level": { 2024-03-31T23:37:05.696 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_level", 2024-03-31T23:37:05.696 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.697 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.697 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.697 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-31T23:37:05.697 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.697 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.697 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-31T23:37:05.697 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-31T23:37:05.697 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-31T23:37:05.697 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-31T23:37:05.697 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-31T23:37:05.697 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-31T23:37:05.697 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-31T23:37:05.697 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-31T23:37:05.697 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.697 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.698 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.698 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.698 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.698 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster": { 2024-03-31T23:37:05.698 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster", 2024-03-31T23:37:05.698 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.698 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.698 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.698 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-31T23:37:05.698 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.698 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.698 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.698 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.698 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.698 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.699 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.699 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.699 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster_level": { 2024-03-31T23:37:05.699 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster_level", 2024-03-31T23:37:05.699 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.699 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.699 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.699 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "info", 2024-03-31T23:37:05.699 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.699 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.699 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-31T23:37:05.699 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-31T23:37:05.699 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-31T23:37:05.699 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-31T23:37:05.699 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-31T23:37:05.700 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-31T23:37:05.700 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-31T23:37:05.700 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-31T23:37:05.700 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.700 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.700 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.700 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.700 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.700 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_file": { 2024-03-31T23:37:05.700 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_file", 2024-03-31T23:37:05.700 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.700 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.700 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.700 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-31T23:37:05.700 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.701 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.701 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.701 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.701 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.701 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.701 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.701 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-31T23:37:05.701 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-31T23:37:05.701 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.701 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-03-31T23:37:05.701 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "telegraf", 2024-03-31T23:37:05.701 INFO:teuthology.orchestra.run.smithi046.stdout: "can_run": true, 2024-03-31T23:37:05.701 INFO:teuthology.orchestra.run.smithi046.stdout: "error_string": "", 2024-03-31T23:37:05.701 INFO:teuthology.orchestra.run.smithi046.stdout: "module_options": { 2024-03-31T23:37:05.701 INFO:teuthology.orchestra.run.smithi046.stdout: "address": { 2024-03-31T23:37:05.702 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "address", 2024-03-31T23:37:05.702 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.702 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.702 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.702 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-03-31T23:37:05.702 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.702 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.702 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.702 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.702 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.702 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.702 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.702 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.702 INFO:teuthology.orchestra.run.smithi046.stdout: "interval": { 2024-03-31T23:37:05.703 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "interval", 2024-03-31T23:37:05.703 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "secs", 2024-03-31T23:37:05.703 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.703 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.703 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "15", 2024-03-31T23:37:05.703 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.703 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.703 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.703 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.703 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.703 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.703 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.703 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.703 INFO:teuthology.orchestra.run.smithi046.stdout: "log_level": { 2024-03-31T23:37:05.703 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_level", 2024-03-31T23:37:05.704 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.704 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.704 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.704 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-31T23:37:05.704 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.704 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.704 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-31T23:37:05.704 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-31T23:37:05.704 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-31T23:37:05.704 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-31T23:37:05.704 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-31T23:37:05.704 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-31T23:37:05.704 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-31T23:37:05.704 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-31T23:37:05.705 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.705 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.705 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.705 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.705 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.705 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster": { 2024-03-31T23:37:05.705 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster", 2024-03-31T23:37:05.705 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.705 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.705 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.705 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-31T23:37:05.705 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.705 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.705 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.706 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.706 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.706 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.706 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.706 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.706 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster_level": { 2024-03-31T23:37:05.706 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster_level", 2024-03-31T23:37:05.706 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.706 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.706 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.706 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "info", 2024-03-31T23:37:05.706 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.706 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.707 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-31T23:37:05.707 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-31T23:37:05.707 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-31T23:37:05.707 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-31T23:37:05.707 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-31T23:37:05.707 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-31T23:37:05.707 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-31T23:37:05.707 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-31T23:37:05.707 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.707 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.707 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.707 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.707 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.707 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_file": { 2024-03-31T23:37:05.707 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_file", 2024-03-31T23:37:05.708 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.708 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.708 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.708 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-31T23:37:05.708 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.708 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.708 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.708 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.708 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.708 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.708 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.708 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-31T23:37:05.708 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-31T23:37:05.708 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.709 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-03-31T23:37:05.709 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "telemetry", 2024-03-31T23:37:05.709 INFO:teuthology.orchestra.run.smithi046.stdout: "can_run": true, 2024-03-31T23:37:05.709 INFO:teuthology.orchestra.run.smithi046.stdout: "error_string": "", 2024-03-31T23:37:05.709 INFO:teuthology.orchestra.run.smithi046.stdout: "module_options": { 2024-03-31T23:37:05.709 INFO:teuthology.orchestra.run.smithi046.stdout: "channel_basic": { 2024-03-31T23:37:05.709 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "channel_basic", 2024-03-31T23:37:05.709 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.709 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.709 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.709 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "True", 2024-03-31T23:37:05.709 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.709 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.709 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.710 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Share basic cluster information (size, version)", 2024-03-31T23:37:05.710 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.710 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.710 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.710 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.710 INFO:teuthology.orchestra.run.smithi046.stdout: "channel_crash": { 2024-03-31T23:37:05.710 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "channel_crash", 2024-03-31T23:37:05.710 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.710 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.710 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.710 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "True", 2024-03-31T23:37:05.710 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.710 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.710 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.711 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-03-31T23:37:05.711 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.711 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.711 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.711 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.711 INFO:teuthology.orchestra.run.smithi046.stdout: "channel_device": { 2024-03-31T23:37:05.711 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "channel_device", 2024-03-31T23:37:05.711 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.711 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.711 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.711 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "True", 2024-03-31T23:37:05.711 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.711 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.711 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.712 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-03-31T23:37:05.712 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.712 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.712 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.712 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.712 INFO:teuthology.orchestra.run.smithi046.stdout: "channel_ident": { 2024-03-31T23:37:05.712 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "channel_ident", 2024-03-31T23:37:05.712 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.712 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.712 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.712 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-31T23:37:05.712 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.712 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.712 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.713 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-03-31T23:37:05.713 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.713 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.713 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.713 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.713 INFO:teuthology.orchestra.run.smithi046.stdout: "channel_perf": { 2024-03-31T23:37:05.713 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "channel_perf", 2024-03-31T23:37:05.713 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.713 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.713 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.713 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-31T23:37:05.713 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.713 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.713 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.714 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Share various performance metrics of a cluster", 2024-03-31T23:37:05.714 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.714 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.714 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.714 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.714 INFO:teuthology.orchestra.run.smithi046.stdout: "contact": { 2024-03-31T23:37:05.714 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "contact", 2024-03-31T23:37:05.714 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.714 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.714 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.714 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-31T23:37:05.714 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.714 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.714 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.715 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.715 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.715 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.715 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.715 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.715 INFO:teuthology.orchestra.run.smithi046.stdout: "description": { 2024-03-31T23:37:05.715 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "description", 2024-03-31T23:37:05.715 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.715 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.715 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.715 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-31T23:37:05.715 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.716 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.716 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.716 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.716 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.716 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.716 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.716 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.716 INFO:teuthology.orchestra.run.smithi046.stdout: "device_url": { 2024-03-31T23:37:05.716 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "device_url", 2024-03-31T23:37:05.716 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.716 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.716 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.716 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-03-31T23:37:05.716 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.717 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.717 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.717 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.717 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.717 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.717 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.717 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.717 INFO:teuthology.orchestra.run.smithi046.stdout: "enabled": { 2024-03-31T23:37:05.717 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "enabled", 2024-03-31T23:37:05.717 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.717 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.717 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.717 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-31T23:37:05.717 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.717 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.717 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.718 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.718 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.718 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.718 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.718 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.718 INFO:teuthology.orchestra.run.smithi046.stdout: "interval": { 2024-03-31T23:37:05.718 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "interval", 2024-03-31T23:37:05.718 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "int", 2024-03-31T23:37:05.718 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.718 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.718 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "24", 2024-03-31T23:37:05.718 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "8", 2024-03-31T23:37:05.718 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.718 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.719 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.719 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.719 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.719 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.719 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.719 INFO:teuthology.orchestra.run.smithi046.stdout: "last_opt_revision": { 2024-03-31T23:37:05.719 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "last_opt_revision", 2024-03-31T23:37:05.719 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "int", 2024-03-31T23:37:05.719 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.719 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.719 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "1", 2024-03-31T23:37:05.719 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.719 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.719 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.720 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.720 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.720 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.720 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.720 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.720 INFO:teuthology.orchestra.run.smithi046.stdout: "leaderboard": { 2024-03-31T23:37:05.720 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "leaderboard", 2024-03-31T23:37:05.720 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.720 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.720 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.720 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-31T23:37:05.720 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.720 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.720 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.720 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.721 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.721 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.721 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.721 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.721 INFO:teuthology.orchestra.run.smithi046.stdout: "leaderboard_description": { 2024-03-31T23:37:05.721 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "leaderboard_description", 2024-03-31T23:37:05.721 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.721 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.721 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.721 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-31T23:37:05.721 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.721 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.721 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.721 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.722 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.722 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.722 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.722 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.722 INFO:teuthology.orchestra.run.smithi046.stdout: "log_level": { 2024-03-31T23:37:05.722 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_level", 2024-03-31T23:37:05.722 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.722 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.722 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.722 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-31T23:37:05.722 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.722 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.722 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-31T23:37:05.722 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-31T23:37:05.722 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-31T23:37:05.723 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-31T23:37:05.723 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-31T23:37:05.723 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-31T23:37:05.723 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-31T23:37:05.723 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-31T23:37:05.723 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.723 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.723 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.723 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.723 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.723 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster": { 2024-03-31T23:37:05.723 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster", 2024-03-31T23:37:05.723 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.723 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.724 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.724 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-31T23:37:05.724 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.724 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.724 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.724 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.724 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.724 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.724 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.724 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.724 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster_level": { 2024-03-31T23:37:05.724 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster_level", 2024-03-31T23:37:05.724 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.724 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.725 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.725 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "info", 2024-03-31T23:37:05.725 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.725 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.725 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-31T23:37:05.725 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-31T23:37:05.725 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-31T23:37:05.725 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-31T23:37:05.725 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-31T23:37:05.725 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-31T23:37:05.725 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-31T23:37:05.725 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-31T23:37:05.725 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.725 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.725 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.726 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.726 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.726 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_file": { 2024-03-31T23:37:05.726 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_file", 2024-03-31T23:37:05.726 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.726 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.726 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.726 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-31T23:37:05.726 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.726 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.726 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.726 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.726 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.726 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.726 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.727 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.727 INFO:teuthology.orchestra.run.smithi046.stdout: "organization": { 2024-03-31T23:37:05.727 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "organization", 2024-03-31T23:37:05.727 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.727 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.727 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.727 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-31T23:37:05.727 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.727 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.727 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.727 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.727 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.727 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.727 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.728 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.728 INFO:teuthology.orchestra.run.smithi046.stdout: "proxy": { 2024-03-31T23:37:05.728 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "proxy", 2024-03-31T23:37:05.728 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.728 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.728 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.728 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-31T23:37:05.728 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.728 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.728 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.728 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.728 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.728 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.728 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.728 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.729 INFO:teuthology.orchestra.run.smithi046.stdout: "url": { 2024-03-31T23:37:05.729 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "url", 2024-03-31T23:37:05.729 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.729 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.729 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.729 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-03-31T23:37:05.729 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.729 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.729 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.729 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.729 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.729 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.729 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.729 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-31T23:37:05.730 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-31T23:37:05.730 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.730 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-03-31T23:37:05.730 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "test_orchestrator", 2024-03-31T23:37:05.730 INFO:teuthology.orchestra.run.smithi046.stdout: "can_run": true, 2024-03-31T23:37:05.730 INFO:teuthology.orchestra.run.smithi046.stdout: "error_string": "", 2024-03-31T23:37:05.730 INFO:teuthology.orchestra.run.smithi046.stdout: "module_options": { 2024-03-31T23:37:05.730 INFO:teuthology.orchestra.run.smithi046.stdout: "log_level": { 2024-03-31T23:37:05.730 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_level", 2024-03-31T23:37:05.730 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.730 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.730 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.730 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-31T23:37:05.730 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.731 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.731 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-31T23:37:05.731 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-31T23:37:05.731 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-31T23:37:05.731 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-31T23:37:05.731 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-31T23:37:05.731 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-31T23:37:05.731 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-31T23:37:05.731 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-31T23:37:05.731 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.731 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.731 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.731 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.731 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.731 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster": { 2024-03-31T23:37:05.732 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster", 2024-03-31T23:37:05.732 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.732 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.732 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.732 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-31T23:37:05.732 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.732 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.732 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.732 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.732 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.732 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.732 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.732 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.732 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster_level": { 2024-03-31T23:37:05.733 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster_level", 2024-03-31T23:37:05.733 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.733 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.733 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.733 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "info", 2024-03-31T23:37:05.733 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.733 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.733 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-31T23:37:05.733 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-31T23:37:05.733 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-31T23:37:05.733 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-31T23:37:05.733 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-31T23:37:05.733 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-31T23:37:05.734 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-31T23:37:05.734 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-31T23:37:05.734 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.734 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.734 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.734 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.734 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.734 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_file": { 2024-03-31T23:37:05.734 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_file", 2024-03-31T23:37:05.734 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.734 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.734 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.734 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-31T23:37:05.734 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.735 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.735 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.735 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.735 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.735 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.735 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.735 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-31T23:37:05.735 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-31T23:37:05.735 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.735 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-03-31T23:37:05.735 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "volumes", 2024-03-31T23:37:05.735 INFO:teuthology.orchestra.run.smithi046.stdout: "can_run": true, 2024-03-31T23:37:05.735 INFO:teuthology.orchestra.run.smithi046.stdout: "error_string": "", 2024-03-31T23:37:05.735 INFO:teuthology.orchestra.run.smithi046.stdout: "module_options": { 2024-03-31T23:37:05.735 INFO:teuthology.orchestra.run.smithi046.stdout: "log_level": { 2024-03-31T23:37:05.736 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_level", 2024-03-31T23:37:05.736 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.736 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.736 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.736 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-31T23:37:05.736 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.736 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.736 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-31T23:37:05.736 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-31T23:37:05.736 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-31T23:37:05.736 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-31T23:37:05.736 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-31T23:37:05.736 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-31T23:37:05.736 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-31T23:37:05.736 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-31T23:37:05.737 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.737 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.737 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.737 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.737 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.737 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster": { 2024-03-31T23:37:05.737 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster", 2024-03-31T23:37:05.737 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.737 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.737 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.737 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-31T23:37:05.737 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.737 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.737 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.738 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.738 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.738 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.738 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.738 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.738 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster_level": { 2024-03-31T23:37:05.738 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster_level", 2024-03-31T23:37:05.738 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.738 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.738 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.738 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "info", 2024-03-31T23:37:05.738 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.738 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.738 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-31T23:37:05.739 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-31T23:37:05.739 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-31T23:37:05.739 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-31T23:37:05.739 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-31T23:37:05.739 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-31T23:37:05.739 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-31T23:37:05.739 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-31T23:37:05.739 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.739 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.739 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.739 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.739 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.739 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_file": { 2024-03-31T23:37:05.739 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_file", 2024-03-31T23:37:05.739 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.740 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.740 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.740 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-31T23:37:05.740 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.740 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.740 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.740 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.740 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.740 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.740 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.740 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.740 INFO:teuthology.orchestra.run.smithi046.stdout: "max_concurrent_clones": { 2024-03-31T23:37:05.740 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "max_concurrent_clones", 2024-03-31T23:37:05.740 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "int", 2024-03-31T23:37:05.740 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.741 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.741 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "4", 2024-03-31T23:37:05.741 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.741 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.741 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.741 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Number of asynchronous cloner threads", 2024-03-31T23:37:05.741 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.741 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.741 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.741 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.741 INFO:teuthology.orchestra.run.smithi046.stdout: "periodic_async_work": { 2024-03-31T23:37:05.741 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "periodic_async_work", 2024-03-31T23:37:05.741 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.741 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.741 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.742 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-31T23:37:05.742 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.742 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.742 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.742 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Periodically check for async work", 2024-03-31T23:37:05.742 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.742 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.742 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.742 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.742 INFO:teuthology.orchestra.run.smithi046.stdout: "snapshot_clone_delay": { 2024-03-31T23:37:05.742 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "snapshot_clone_delay", 2024-03-31T23:37:05.742 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "int", 2024-03-31T23:37:05.742 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.742 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.743 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "0", 2024-03-31T23:37:05.743 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.743 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.743 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.743 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-03-31T23:37:05.743 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.743 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.743 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.743 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.743 INFO:teuthology.orchestra.run.smithi046.stdout: "snapshot_clone_no_wait": { 2024-03-31T23:37:05.743 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "snapshot_clone_no_wait", 2024-03-31T23:37:05.743 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.743 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.743 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.744 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "True", 2024-03-31T23:37:05.744 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.744 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.744 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.744 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-03-31T23:37:05.744 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.744 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.744 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.744 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-31T23:37:05.744 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-31T23:37:05.744 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.744 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-03-31T23:37:05.744 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "zabbix", 2024-03-31T23:37:05.744 INFO:teuthology.orchestra.run.smithi046.stdout: "can_run": true, 2024-03-31T23:37:05.744 INFO:teuthology.orchestra.run.smithi046.stdout: "error_string": "", 2024-03-31T23:37:05.745 INFO:teuthology.orchestra.run.smithi046.stdout: "module_options": { 2024-03-31T23:37:05.745 INFO:teuthology.orchestra.run.smithi046.stdout: "discovery_interval": { 2024-03-31T23:37:05.745 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "discovery_interval", 2024-03-31T23:37:05.745 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "uint", 2024-03-31T23:37:05.745 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.745 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.745 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "100", 2024-03-31T23:37:05.745 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.745 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.745 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.745 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.745 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.745 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.745 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.745 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.746 INFO:teuthology.orchestra.run.smithi046.stdout: "identifier": { 2024-03-31T23:37:05.746 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "identifier", 2024-03-31T23:37:05.746 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.746 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.746 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.746 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-31T23:37:05.746 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.746 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.746 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.746 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.746 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.746 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.746 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.746 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.747 INFO:teuthology.orchestra.run.smithi046.stdout: "interval": { 2024-03-31T23:37:05.747 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "interval", 2024-03-31T23:37:05.747 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "secs", 2024-03-31T23:37:05.747 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.747 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.747 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "60", 2024-03-31T23:37:05.747 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.747 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.747 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.747 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.747 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.747 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.747 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.747 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.748 INFO:teuthology.orchestra.run.smithi046.stdout: "log_level": { 2024-03-31T23:37:05.748 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_level", 2024-03-31T23:37:05.748 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.748 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.748 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.748 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-31T23:37:05.748 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.748 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.748 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-31T23:37:05.748 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-31T23:37:05.748 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-31T23:37:05.748 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-31T23:37:05.748 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-31T23:37:05.748 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-31T23:37:05.748 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-31T23:37:05.749 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-31T23:37:05.749 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.749 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.749 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.749 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.749 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.749 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster": { 2024-03-31T23:37:05.749 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster", 2024-03-31T23:37:05.749 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.749 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.749 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.749 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-31T23:37:05.749 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.749 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.749 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.750 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.750 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.750 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.750 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.750 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.750 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster_level": { 2024-03-31T23:37:05.750 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster_level", 2024-03-31T23:37:05.750 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.750 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.750 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.750 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "info", 2024-03-31T23:37:05.750 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.750 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.751 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-31T23:37:05.751 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-31T23:37:05.751 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-31T23:37:05.751 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-31T23:37:05.751 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-31T23:37:05.751 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-31T23:37:05.751 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-31T23:37:05.751 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-31T23:37:05.751 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.751 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.751 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.751 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.751 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.751 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_file": { 2024-03-31T23:37:05.751 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_file", 2024-03-31T23:37:05.752 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.752 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.752 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.752 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-31T23:37:05.752 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.752 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.752 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.752 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.752 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.752 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.752 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.752 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.752 INFO:teuthology.orchestra.run.smithi046.stdout: "zabbix_host": { 2024-03-31T23:37:05.752 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "zabbix_host", 2024-03-31T23:37:05.753 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.753 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.753 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.753 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-31T23:37:05.753 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.753 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.753 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.753 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.753 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.753 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.753 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.753 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.753 INFO:teuthology.orchestra.run.smithi046.stdout: "zabbix_port": { 2024-03-31T23:37:05.753 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "zabbix_port", 2024-03-31T23:37:05.753 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "int", 2024-03-31T23:37:05.754 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.754 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.754 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "10051", 2024-03-31T23:37:05.754 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.754 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.754 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.754 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.754 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.754 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.754 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.754 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.754 INFO:teuthology.orchestra.run.smithi046.stdout: "zabbix_sender": { 2024-03-31T23:37:05.754 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "zabbix_sender", 2024-03-31T23:37:05.754 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.755 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.755 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.755 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-03-31T23:37:05.755 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.755 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.755 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.755 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.755 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.755 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.755 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.755 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-31T23:37:05.755 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-31T23:37:05.755 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-31T23:37:05.755 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-03-31T23:37:05.756 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-31T23:37:05.756 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-31T23:37:05.756 INFO:teuthology.orchestra.run.smithi046.stdout: "modules": [ 2024-03-31T23:37:05.756 INFO:teuthology.orchestra.run.smithi046.stdout: "cephadm", 2024-03-31T23:37:05.756 INFO:teuthology.orchestra.run.smithi046.stdout: "dashboard", 2024-03-31T23:37:05.756 INFO:teuthology.orchestra.run.smithi046.stdout: "iostat", 2024-03-31T23:37:05.756 INFO:teuthology.orchestra.run.smithi046.stdout: "nfs", 2024-03-31T23:37:05.756 INFO:teuthology.orchestra.run.smithi046.stdout: "restful", 2024-03-31T23:37:05.756 INFO:teuthology.orchestra.run.smithi046.stdout: "snap_schedule" 2024-03-31T23:37:05.756 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-31T23:37:05.756 INFO:teuthology.orchestra.run.smithi046.stdout: "available_modules": [ 2024-03-31T23:37:05.756 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-03-31T23:37:05.756 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "alerts", 2024-03-31T23:37:05.756 INFO:teuthology.orchestra.run.smithi046.stdout: "can_run": true, 2024-03-31T23:37:05.756 INFO:teuthology.orchestra.run.smithi046.stdout: "error_string": "", 2024-03-31T23:37:05.757 INFO:teuthology.orchestra.run.smithi046.stdout: "module_options": { 2024-03-31T23:37:05.757 INFO:teuthology.orchestra.run.smithi046.stdout: "interval": { 2024-03-31T23:37:05.757 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "interval", 2024-03-31T23:37:05.757 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "secs", 2024-03-31T23:37:05.757 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.757 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.757 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "60", 2024-03-31T23:37:05.757 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.757 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.757 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.757 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "How frequently to reexamine health status", 2024-03-31T23:37:05.757 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.757 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.757 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.757 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.758 INFO:teuthology.orchestra.run.smithi046.stdout: "log_level": { 2024-03-31T23:37:05.758 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_level", 2024-03-31T23:37:05.758 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.758 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.758 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.758 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-31T23:37:05.758 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.758 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.758 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-31T23:37:05.758 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-31T23:37:05.758 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-31T23:37:05.758 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-31T23:37:05.758 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-31T23:37:05.758 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-31T23:37:05.759 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-31T23:37:05.759 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-31T23:37:05.759 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.759 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.759 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.759 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.759 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.759 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster": { 2024-03-31T23:37:05.759 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster", 2024-03-31T23:37:05.759 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.759 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.759 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.759 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-31T23:37:05.759 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.760 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.760 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.760 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.760 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.760 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.760 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.760 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.760 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster_level": { 2024-03-31T23:37:05.760 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster_level", 2024-03-31T23:37:05.760 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.760 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.760 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.760 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "info", 2024-03-31T23:37:05.760 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.760 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.761 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-31T23:37:05.761 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-31T23:37:05.761 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-31T23:37:05.761 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-31T23:37:05.761 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-31T23:37:05.761 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-31T23:37:05.761 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-31T23:37:05.761 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-31T23:37:05.761 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.761 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.761 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.761 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.761 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.762 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_file": { 2024-03-31T23:37:05.762 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_file", 2024-03-31T23:37:05.762 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.762 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.762 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.762 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-31T23:37:05.762 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.762 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.762 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.762 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.762 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.762 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.762 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.762 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.762 INFO:teuthology.orchestra.run.smithi046.stdout: "smtp_destination": { 2024-03-31T23:37:05.763 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "smtp_destination", 2024-03-31T23:37:05.763 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.763 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.763 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.763 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-31T23:37:05.763 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.763 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.763 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.763 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Email address to send alerts to", 2024-03-31T23:37:05.763 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.763 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.763 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.763 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.763 INFO:teuthology.orchestra.run.smithi046.stdout: "smtp_from_name": { 2024-03-31T23:37:05.763 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "smtp_from_name", 2024-03-31T23:37:05.764 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.764 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.764 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.764 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "Ceph", 2024-03-31T23:37:05.764 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.764 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.764 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.764 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Email From: name", 2024-03-31T23:37:05.764 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.764 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.764 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.764 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.764 INFO:teuthology.orchestra.run.smithi046.stdout: "smtp_host": { 2024-03-31T23:37:05.764 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "smtp_host", 2024-03-31T23:37:05.765 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.765 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.765 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.765 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-31T23:37:05.765 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.765 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.765 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.765 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "SMTP server", 2024-03-31T23:37:05.765 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.765 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.765 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.765 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.765 INFO:teuthology.orchestra.run.smithi046.stdout: "smtp_password": { 2024-03-31T23:37:05.765 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "smtp_password", 2024-03-31T23:37:05.766 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.766 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.766 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.766 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-31T23:37:05.766 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.766 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.766 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.766 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Password to authenticate with", 2024-03-31T23:37:05.766 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.766 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.766 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.766 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.766 INFO:teuthology.orchestra.run.smithi046.stdout: "smtp_port": { 2024-03-31T23:37:05.766 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "smtp_port", 2024-03-31T23:37:05.766 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "int", 2024-03-31T23:37:05.767 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.767 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.767 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "465", 2024-03-31T23:37:05.767 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.767 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.767 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.767 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "SMTP port", 2024-03-31T23:37:05.767 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.767 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.767 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.767 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.767 INFO:teuthology.orchestra.run.smithi046.stdout: "smtp_sender": { 2024-03-31T23:37:05.767 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "smtp_sender", 2024-03-31T23:37:05.767 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.768 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.768 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.768 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-31T23:37:05.768 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.768 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.768 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.768 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "SMTP envelope sender", 2024-03-31T23:37:05.768 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.768 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.768 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.768 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.768 INFO:teuthology.orchestra.run.smithi046.stdout: "smtp_ssl": { 2024-03-31T23:37:05.768 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "smtp_ssl", 2024-03-31T23:37:05.768 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.768 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.769 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.769 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "True", 2024-03-31T23:37:05.769 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.769 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.769 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.769 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Use SSL to connect to SMTP server", 2024-03-31T23:37:05.769 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.769 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.769 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.769 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.769 INFO:teuthology.orchestra.run.smithi046.stdout: "smtp_user": { 2024-03-31T23:37:05.769 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "smtp_user", 2024-03-31T23:37:05.769 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.769 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.769 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.770 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-31T23:37:05.770 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.770 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.770 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.770 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "User to authenticate as", 2024-03-31T23:37:05.770 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.770 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.770 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.770 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-31T23:37:05.770 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-31T23:37:05.770 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.770 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-03-31T23:37:05.770 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "balancer", 2024-03-31T23:37:05.770 INFO:teuthology.orchestra.run.smithi046.stdout: "can_run": true, 2024-03-31T23:37:05.771 INFO:teuthology.orchestra.run.smithi046.stdout: "error_string": "", 2024-03-31T23:37:05.771 INFO:teuthology.orchestra.run.smithi046.stdout: "module_options": { 2024-03-31T23:37:05.771 INFO:teuthology.orchestra.run.smithi046.stdout: "active": { 2024-03-31T23:37:05.771 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "active", 2024-03-31T23:37:05.771 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.771 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.771 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.771 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "True", 2024-03-31T23:37:05.771 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.771 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.771 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.771 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "automatically balance PGs across cluster", 2024-03-31T23:37:05.771 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.771 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.771 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.772 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.772 INFO:teuthology.orchestra.run.smithi046.stdout: "begin_time": { 2024-03-31T23:37:05.772 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "begin_time", 2024-03-31T23:37:05.772 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.772 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.772 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.772 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "0000", 2024-03-31T23:37:05.772 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.772 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.772 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.772 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "beginning time of day to automatically balance", 2024-03-31T23:37:05.772 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-31T23:37:05.772 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.772 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.773 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.773 INFO:teuthology.orchestra.run.smithi046.stdout: "begin_weekday": { 2024-03-31T23:37:05.773 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "begin_weekday", 2024-03-31T23:37:05.773 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "uint", 2024-03-31T23:37:05.773 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.773 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.773 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "0", 2024-03-31T23:37:05.773 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "0", 2024-03-31T23:37:05.773 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "6", 2024-03-31T23:37:05.773 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.773 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-03-31T23:37:05.773 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-31T23:37:05.773 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.773 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.773 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.774 INFO:teuthology.orchestra.run.smithi046.stdout: "crush_compat_max_iterations": { 2024-03-31T23:37:05.774 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "crush_compat_max_iterations", 2024-03-31T23:37:05.774 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "uint", 2024-03-31T23:37:05.774 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.774 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.774 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "25", 2024-03-31T23:37:05.774 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "1", 2024-03-31T23:37:05.774 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "250", 2024-03-31T23:37:05.774 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.774 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-03-31T23:37:05.774 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.774 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.774 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.774 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.775 INFO:teuthology.orchestra.run.smithi046.stdout: "crush_compat_metrics": { 2024-03-31T23:37:05.775 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "crush_compat_metrics", 2024-03-31T23:37:05.775 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.775 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.775 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.775 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "pgs,objects,bytes", 2024-03-31T23:37:05.775 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.775 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.775 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.775 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-03-31T23:37:05.775 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-03-31T23:37:05.775 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.775 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.775 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.776 INFO:teuthology.orchestra.run.smithi046.stdout: "crush_compat_step": { 2024-03-31T23:37:05.776 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "crush_compat_step", 2024-03-31T23:37:05.776 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "float", 2024-03-31T23:37:05.776 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.776 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.776 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "0.5", 2024-03-31T23:37:05.776 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "0.001", 2024-03-31T23:37:05.776 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "0.999", 2024-03-31T23:37:05.776 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.776 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "aggressiveness of optimization", 2024-03-31T23:37:05.776 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-03-31T23:37:05.776 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.776 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.776 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.776 INFO:teuthology.orchestra.run.smithi046.stdout: "end_time": { 2024-03-31T23:37:05.777 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "end_time", 2024-03-31T23:37:05.777 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.777 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.777 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.777 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "2359", 2024-03-31T23:37:05.777 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.777 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.777 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.777 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "ending time of day to automatically balance", 2024-03-31T23:37:05.777 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-31T23:37:05.777 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.777 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.777 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.777 INFO:teuthology.orchestra.run.smithi046.stdout: "end_weekday": { 2024-03-31T23:37:05.777 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "end_weekday", 2024-03-31T23:37:05.778 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "uint", 2024-03-31T23:37:05.778 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.778 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.778 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "0", 2024-03-31T23:37:05.778 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "0", 2024-03-31T23:37:05.778 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "6", 2024-03-31T23:37:05.778 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.778 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-03-31T23:37:05.778 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-31T23:37:05.778 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.778 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.778 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.778 INFO:teuthology.orchestra.run.smithi046.stdout: "log_level": { 2024-03-31T23:37:05.778 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_level", 2024-03-31T23:37:05.779 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.779 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.779 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.779 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-31T23:37:05.779 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.779 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.779 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-31T23:37:05.779 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-31T23:37:05.779 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-31T23:37:05.779 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-31T23:37:05.779 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-31T23:37:05.779 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-31T23:37:05.779 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-31T23:37:05.779 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-31T23:37:05.780 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.780 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.780 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.780 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.780 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.780 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster": { 2024-03-31T23:37:05.780 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster", 2024-03-31T23:37:05.780 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.780 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.780 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.780 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-31T23:37:05.780 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.780 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.780 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.780 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.781 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.781 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.781 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.781 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.781 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster_level": { 2024-03-31T23:37:05.781 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster_level", 2024-03-31T23:37:05.781 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.781 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.781 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.781 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "info", 2024-03-31T23:37:05.781 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.781 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.781 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-31T23:37:05.781 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-31T23:37:05.782 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-31T23:37:05.782 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-31T23:37:05.782 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-31T23:37:05.782 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-31T23:37:05.782 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-31T23:37:05.782 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-31T23:37:05.782 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.782 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.782 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.782 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.782 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.782 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_file": { 2024-03-31T23:37:05.782 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_file", 2024-03-31T23:37:05.782 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.782 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.783 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.783 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-31T23:37:05.783 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.783 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.783 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.783 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.783 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.783 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.783 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.783 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.783 INFO:teuthology.orchestra.run.smithi046.stdout: "min_score": { 2024-03-31T23:37:05.783 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "min_score", 2024-03-31T23:37:05.783 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "float", 2024-03-31T23:37:05.783 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.784 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.784 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "0", 2024-03-31T23:37:05.784 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.784 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.784 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.784 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-03-31T23:37:05.784 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.784 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.784 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.784 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.784 INFO:teuthology.orchestra.run.smithi046.stdout: "mode": { 2024-03-31T23:37:05.784 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "mode", 2024-03-31T23:37:05.784 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.784 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.784 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.785 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "upmap", 2024-03-31T23:37:05.785 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.785 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.785 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-31T23:37:05.785 INFO:teuthology.orchestra.run.smithi046.stdout: "crush-compat", 2024-03-31T23:37:05.785 INFO:teuthology.orchestra.run.smithi046.stdout: "none", 2024-03-31T23:37:05.785 INFO:teuthology.orchestra.run.smithi046.stdout: "read", 2024-03-31T23:37:05.785 INFO:teuthology.orchestra.run.smithi046.stdout: "upmap", 2024-03-31T23:37:05.785 INFO:teuthology.orchestra.run.smithi046.stdout: "upmap-read" 2024-03-31T23:37:05.785 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-31T23:37:05.785 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Balancer mode", 2024-03-31T23:37:05.785 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.785 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.786 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.786 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.786 INFO:teuthology.orchestra.run.smithi046.stdout: "pool_ids": { 2024-03-31T23:37:05.786 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "pool_ids", 2024-03-31T23:37:05.786 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.786 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.786 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.786 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-31T23:37:05.786 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.786 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.786 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.786 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-03-31T23:37:05.786 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.786 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.787 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.787 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.787 INFO:teuthology.orchestra.run.smithi046.stdout: "sleep_interval": { 2024-03-31T23:37:05.787 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "sleep_interval", 2024-03-31T23:37:05.787 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "secs", 2024-03-31T23:37:05.787 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.787 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.787 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "60", 2024-03-31T23:37:05.787 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.787 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.787 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.787 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-03-31T23:37:05.787 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.787 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.788 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.788 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.788 INFO:teuthology.orchestra.run.smithi046.stdout: "upmap_max_deviation": { 2024-03-31T23:37:05.788 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "upmap_max_deviation", 2024-03-31T23:37:05.788 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "int", 2024-03-31T23:37:05.788 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.788 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.788 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "5", 2024-03-31T23:37:05.788 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "1", 2024-03-31T23:37:05.788 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.788 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.788 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "deviation below which no optimization is attempted", 2024-03-31T23:37:05.788 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-03-31T23:37:05.788 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.788 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.789 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.789 INFO:teuthology.orchestra.run.smithi046.stdout: "upmap_max_optimizations": { 2024-03-31T23:37:05.789 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "upmap_max_optimizations", 2024-03-31T23:37:05.789 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "uint", 2024-03-31T23:37:05.789 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.789 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.789 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "10", 2024-03-31T23:37:05.789 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.789 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.789 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.789 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-03-31T23:37:05.789 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.789 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.789 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.789 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-31T23:37:05.789 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-31T23:37:05.790 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.790 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-03-31T23:37:05.790 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "cephadm", 2024-03-31T23:37:05.790 INFO:teuthology.orchestra.run.smithi046.stdout: "can_run": true, 2024-03-31T23:37:05.790 INFO:teuthology.orchestra.run.smithi046.stdout: "error_string": "", 2024-03-31T23:37:05.790 INFO:teuthology.orchestra.run.smithi046.stdout: "module_options": { 2024-03-31T23:37:05.790 INFO:teuthology.orchestra.run.smithi046.stdout: "agent_down_multiplier": { 2024-03-31T23:37:05.790 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "agent_down_multiplier", 2024-03-31T23:37:05.790 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "float", 2024-03-31T23:37:05.790 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.790 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.790 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "3.0", 2024-03-31T23:37:05.790 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.791 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.791 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.791 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-03-31T23:37:05.791 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.791 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.791 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.791 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.791 INFO:teuthology.orchestra.run.smithi046.stdout: "agent_refresh_rate": { 2024-03-31T23:37:05.791 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "agent_refresh_rate", 2024-03-31T23:37:05.791 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "secs", 2024-03-31T23:37:05.791 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.791 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.791 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "20", 2024-03-31T23:37:05.791 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.792 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.792 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.792 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-03-31T23:37:05.792 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.792 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.792 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.792 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.792 INFO:teuthology.orchestra.run.smithi046.stdout: "agent_starting_port": { 2024-03-31T23:37:05.792 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "agent_starting_port", 2024-03-31T23:37:05.792 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "int", 2024-03-31T23:37:05.792 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.792 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.792 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "4721", 2024-03-31T23:37:05.792 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.792 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.793 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.793 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-03-31T23:37:05.793 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.793 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.793 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.793 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.793 INFO:teuthology.orchestra.run.smithi046.stdout: "allow_ptrace": { 2024-03-31T23:37:05.793 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "allow_ptrace", 2024-03-31T23:37:05.793 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.793 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.793 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.793 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-31T23:37:05.793 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.793 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.794 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.794 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-03-31T23:37:05.794 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-03-31T23:37:05.794 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.794 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.794 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.794 INFO:teuthology.orchestra.run.smithi046.stdout: "autotune_interval": { 2024-03-31T23:37:05.794 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "autotune_interval", 2024-03-31T23:37:05.794 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "secs", 2024-03-31T23:37:05.794 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.794 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.794 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "600", 2024-03-31T23:37:05.794 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.794 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.795 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.795 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "how frequently to autotune daemon memory", 2024-03-31T23:37:05.795 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.795 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.795 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.795 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.795 INFO:teuthology.orchestra.run.smithi046.stdout: "autotune_memory_target_ratio": { 2024-03-31T23:37:05.795 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "autotune_memory_target_ratio", 2024-03-31T23:37:05.795 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "float", 2024-03-31T23:37:05.795 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.795 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.795 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "0.7", 2024-03-31T23:37:05.795 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.795 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.795 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.796 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-03-31T23:37:05.796 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.796 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.796 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.796 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.796 INFO:teuthology.orchestra.run.smithi046.stdout: "cephadm_log_destination": { 2024-03-31T23:37:05.796 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "cephadm_log_destination", 2024-03-31T23:37:05.796 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.796 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.796 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.796 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-31T23:37:05.796 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.796 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.796 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-31T23:37:05.797 INFO:teuthology.orchestra.run.smithi046.stdout: "file", 2024-03-31T23:37:05.797 INFO:teuthology.orchestra.run.smithi046.stdout: "file,syslog", 2024-03-31T23:37:05.797 INFO:teuthology.orchestra.run.smithi046.stdout: "syslog" 2024-03-31T23:37:05.797 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-31T23:37:05.797 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-03-31T23:37:05.797 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.797 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.797 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.797 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.797 INFO:teuthology.orchestra.run.smithi046.stdout: "cgroups_split": { 2024-03-31T23:37:05.797 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "cgroups_split", 2024-03-31T23:37:05.797 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.797 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.797 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.797 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "True", 2024-03-31T23:37:05.798 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.798 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.798 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.798 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-03-31T23:37:05.798 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.798 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.798 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.798 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.798 INFO:teuthology.orchestra.run.smithi046.stdout: "config_checks_enabled": { 2024-03-31T23:37:05.798 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "config_checks_enabled", 2024-03-31T23:37:05.798 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.798 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.798 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.798 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-31T23:37:05.799 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.799 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.799 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.799 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-03-31T23:37:05.799 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.799 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.799 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.799 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.799 INFO:teuthology.orchestra.run.smithi046.stdout: "config_dashboard": { 2024-03-31T23:37:05.799 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "config_dashboard", 2024-03-31T23:37:05.799 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.799 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.799 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.799 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "True", 2024-03-31T23:37:05.799 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.800 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.800 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.800 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-03-31T23:37:05.800 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.800 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.800 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.800 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.800 INFO:teuthology.orchestra.run.smithi046.stdout: "container_image_alertmanager": { 2024-03-31T23:37:05.800 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "container_image_alertmanager", 2024-03-31T23:37:05.800 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.800 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.800 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.800 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-03-31T23:37:05.800 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.800 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.801 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.801 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Prometheus container image", 2024-03-31T23:37:05.801 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.801 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.801 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.801 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.801 INFO:teuthology.orchestra.run.smithi046.stdout: "container_image_base": { 2024-03-31T23:37:05.801 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "container_image_base", 2024-03-31T23:37:05.801 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.801 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.801 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.801 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "quay.io/ceph/ceph", 2024-03-31T23:37:05.801 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.802 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.802 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.802 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Container image name, without the tag", 2024-03-31T23:37:05.802 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.802 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.802 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.802 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.802 INFO:teuthology.orchestra.run.smithi046.stdout: "container_image_elasticsearch": { 2024-03-31T23:37:05.802 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "container_image_elasticsearch", 2024-03-31T23:37:05.802 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.802 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.802 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.802 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-03-31T23:37:05.802 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.803 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.803 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.803 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "elasticsearch container image", 2024-03-31T23:37:05.803 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.803 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.803 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.803 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.803 INFO:teuthology.orchestra.run.smithi046.stdout: "container_image_grafana": { 2024-03-31T23:37:05.803 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "container_image_grafana", 2024-03-31T23:37:05.803 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.803 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.803 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.803 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "quay.io/ceph/grafana:9.4.12", 2024-03-31T23:37:05.803 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.803 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.804 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.804 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Prometheus container image", 2024-03-31T23:37:05.804 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.804 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.804 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.804 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.804 INFO:teuthology.orchestra.run.smithi046.stdout: "container_image_haproxy": { 2024-03-31T23:37:05.804 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "container_image_haproxy", 2024-03-31T23:37:05.804 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.804 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.804 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.804 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-03-31T23:37:05.804 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.804 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.805 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.805 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "HAproxy container image", 2024-03-31T23:37:05.805 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.805 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.805 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.805 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.805 INFO:teuthology.orchestra.run.smithi046.stdout: "container_image_jaeger_agent": { 2024-03-31T23:37:05.805 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "container_image_jaeger_agent", 2024-03-31T23:37:05.805 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.805 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.805 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.805 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-03-31T23:37:05.805 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.805 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.805 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.806 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Jaeger agent container image", 2024-03-31T23:37:05.806 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.806 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.806 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.806 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.806 INFO:teuthology.orchestra.run.smithi046.stdout: "container_image_jaeger_collector": { 2024-03-31T23:37:05.806 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "container_image_jaeger_collector", 2024-03-31T23:37:05.806 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.806 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.806 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.806 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-03-31T23:37:05.806 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.806 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.807 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.807 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Jaeger collector container image", 2024-03-31T23:37:05.807 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.807 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.807 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.807 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.807 INFO:teuthology.orchestra.run.smithi046.stdout: "container_image_jaeger_query": { 2024-03-31T23:37:05.807 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "container_image_jaeger_query", 2024-03-31T23:37:05.807 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.807 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.807 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.807 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-03-31T23:37:05.807 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.807 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.807 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.808 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Jaeger query container image", 2024-03-31T23:37:05.808 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.808 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.808 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.808 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.808 INFO:teuthology.orchestra.run.smithi046.stdout: "container_image_keepalived": { 2024-03-31T23:37:05.808 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "container_image_keepalived", 2024-03-31T23:37:05.808 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.808 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.808 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.808 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-03-31T23:37:05.808 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.808 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.808 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.809 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Keepalived container image", 2024-03-31T23:37:05.809 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.809 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.809 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.809 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.809 INFO:teuthology.orchestra.run.smithi046.stdout: "container_image_loki": { 2024-03-31T23:37:05.809 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "container_image_loki", 2024-03-31T23:37:05.809 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.809 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.809 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.809 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-03-31T23:37:05.809 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.809 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.809 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.809 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Loki container image", 2024-03-31T23:37:05.810 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.810 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.810 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.810 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.810 INFO:teuthology.orchestra.run.smithi046.stdout: "container_image_node_exporter": { 2024-03-31T23:37:05.810 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "container_image_node_exporter", 2024-03-31T23:37:05.810 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.810 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.810 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.810 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-03-31T23:37:05.810 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.810 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.810 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.810 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Prometheus container image", 2024-03-31T23:37:05.811 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.811 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.811 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.811 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.811 INFO:teuthology.orchestra.run.smithi046.stdout: "container_image_nvmeof": { 2024-03-31T23:37:05.811 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "container_image_nvmeof", 2024-03-31T23:37:05.811 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.811 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.811 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.811 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "quay.io/ceph/nvmeof:latest", 2024-03-31T23:37:05.811 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.811 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.811 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.811 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Nvme-of container image", 2024-03-31T23:37:05.811 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.812 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.812 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.812 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.812 INFO:teuthology.orchestra.run.smithi046.stdout: "container_image_prometheus": { 2024-03-31T23:37:05.812 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "container_image_prometheus", 2024-03-31T23:37:05.812 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.812 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.812 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.812 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-03-31T23:37:05.812 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.812 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.812 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.812 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Prometheus container image", 2024-03-31T23:37:05.812 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.812 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.813 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.813 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.813 INFO:teuthology.orchestra.run.smithi046.stdout: "container_image_promtail": { 2024-03-31T23:37:05.813 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "container_image_promtail", 2024-03-31T23:37:05.813 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.813 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.813 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.813 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-03-31T23:37:05.813 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.813 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.813 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.813 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Promtail container image", 2024-03-31T23:37:05.813 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.813 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.813 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.814 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.814 INFO:teuthology.orchestra.run.smithi046.stdout: "container_image_snmp_gateway": { 2024-03-31T23:37:05.814 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "container_image_snmp_gateway", 2024-03-31T23:37:05.814 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.814 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.814 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.814 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-03-31T23:37:05.814 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.814 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.814 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.814 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "SNMP Gateway container image", 2024-03-31T23:37:05.814 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.814 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.814 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.815 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.815 INFO:teuthology.orchestra.run.smithi046.stdout: "container_init": { 2024-03-31T23:37:05.815 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "container_init", 2024-03-31T23:37:05.815 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.815 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.815 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.815 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "True", 2024-03-31T23:37:05.815 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.815 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.815 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.815 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Run podman/docker with `--init`", 2024-03-31T23:37:05.815 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.815 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.815 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.815 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.816 INFO:teuthology.orchestra.run.smithi046.stdout: "daemon_cache_timeout": { 2024-03-31T23:37:05.816 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "daemon_cache_timeout", 2024-03-31T23:37:05.816 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "secs", 2024-03-31T23:37:05.816 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.816 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.816 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "600", 2024-03-31T23:37:05.816 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.816 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.816 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.816 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-03-31T23:37:05.816 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.816 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.816 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.816 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.816 INFO:teuthology.orchestra.run.smithi046.stdout: "default_cephadm_command_timeout": { 2024-03-31T23:37:05.817 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "default_cephadm_command_timeout", 2024-03-31T23:37:05.817 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "int", 2024-03-31T23:37:05.817 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.817 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.817 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "900", 2024-03-31T23:37:05.817 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.817 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.817 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.817 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-03-31T23:37:05.817 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.817 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.817 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.817 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.817 INFO:teuthology.orchestra.run.smithi046.stdout: "default_registry": { 2024-03-31T23:37:05.817 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "default_registry", 2024-03-31T23:37:05.818 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.818 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.818 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.818 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "docker.io", 2024-03-31T23:37:05.818 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.818 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.818 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.818 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-03-31T23:37:05.818 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.818 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.818 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.818 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.818 INFO:teuthology.orchestra.run.smithi046.stdout: "device_cache_timeout": { 2024-03-31T23:37:05.819 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "device_cache_timeout", 2024-03-31T23:37:05.819 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "secs", 2024-03-31T23:37:05.819 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.819 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.819 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "1800", 2024-03-31T23:37:05.819 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.819 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.819 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.819 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "seconds to cache device inventory", 2024-03-31T23:37:05.819 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.819 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.819 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.819 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.819 INFO:teuthology.orchestra.run.smithi046.stdout: "device_enhanced_scan": { 2024-03-31T23:37:05.819 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "device_enhanced_scan", 2024-03-31T23:37:05.820 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.820 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.820 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.820 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-31T23:37:05.820 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.820 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.820 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.820 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Use libstoragemgmt during device scans", 2024-03-31T23:37:05.820 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.820 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.820 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.820 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.820 INFO:teuthology.orchestra.run.smithi046.stdout: "facts_cache_timeout": { 2024-03-31T23:37:05.820 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "facts_cache_timeout", 2024-03-31T23:37:05.820 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "secs", 2024-03-31T23:37:05.821 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.821 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.821 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "60", 2024-03-31T23:37:05.821 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.821 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.821 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.821 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "seconds to cache host facts data", 2024-03-31T23:37:05.821 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.821 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.821 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.821 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.821 INFO:teuthology.orchestra.run.smithi046.stdout: "grafana_dashboards_path": { 2024-03-31T23:37:05.821 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "grafana_dashboards_path", 2024-03-31T23:37:05.821 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.822 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.822 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.822 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "/etc/grafana/dashboards/ceph-dashboard/", 2024-03-31T23:37:05.822 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.822 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.822 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.822 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "location of dashboards to include in grafana deployments", 2024-03-31T23:37:05.822 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.822 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.822 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.822 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.822 INFO:teuthology.orchestra.run.smithi046.stdout: "host_check_interval": { 2024-03-31T23:37:05.822 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "host_check_interval", 2024-03-31T23:37:05.822 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "secs", 2024-03-31T23:37:05.823 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.823 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.823 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "600", 2024-03-31T23:37:05.823 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.823 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.823 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.823 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "how frequently to perform a host check", 2024-03-31T23:37:05.823 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.823 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.823 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.823 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.823 INFO:teuthology.orchestra.run.smithi046.stdout: "hw_monitoring": { 2024-03-31T23:37:05.823 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "hw_monitoring", 2024-03-31T23:37:05.823 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.823 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.824 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.824 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-31T23:37:05.824 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.824 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.824 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.824 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-03-31T23:37:05.824 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.824 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.824 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.824 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.824 INFO:teuthology.orchestra.run.smithi046.stdout: "inventory_list_all": { 2024-03-31T23:37:05.824 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "inventory_list_all", 2024-03-31T23:37:05.824 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.824 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.825 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.825 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-31T23:37:05.825 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.825 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.825 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.825 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-03-31T23:37:05.825 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.825 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.825 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.825 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.825 INFO:teuthology.orchestra.run.smithi046.stdout: "log_level": { 2024-03-31T23:37:05.825 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_level", 2024-03-31T23:37:05.825 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.825 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.825 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.826 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-31T23:37:05.826 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.826 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.826 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-31T23:37:05.826 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-31T23:37:05.826 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-31T23:37:05.826 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-31T23:37:05.826 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-31T23:37:05.826 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-31T23:37:05.826 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-31T23:37:05.826 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-31T23:37:05.826 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.826 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.826 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.827 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.827 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.827 INFO:teuthology.orchestra.run.smithi046.stdout: "log_refresh_metadata": { 2024-03-31T23:37:05.827 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_refresh_metadata", 2024-03-31T23:37:05.827 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.827 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.827 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.827 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-31T23:37:05.827 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.827 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.827 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.827 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-03-31T23:37:05.827 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.827 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.827 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.828 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.828 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster": { 2024-03-31T23:37:05.828 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster", 2024-03-31T23:37:05.828 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.828 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.828 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.828 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "True", 2024-03-31T23:37:05.828 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.828 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.828 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.828 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-03-31T23:37:05.828 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.828 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.828 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.829 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.829 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster_level": { 2024-03-31T23:37:05.829 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster_level", 2024-03-31T23:37:05.829 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.829 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.829 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.829 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "info", 2024-03-31T23:37:05.829 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.829 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.829 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-31T23:37:05.829 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-31T23:37:05.829 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-31T23:37:05.829 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-31T23:37:05.829 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-31T23:37:05.829 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-31T23:37:05.830 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-31T23:37:05.830 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-31T23:37:05.830 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.830 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.830 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.830 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.830 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.830 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_file": { 2024-03-31T23:37:05.830 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_file", 2024-03-31T23:37:05.830 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.830 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.830 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.830 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-31T23:37:05.830 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.830 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.831 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.831 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.831 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.831 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.831 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.831 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.831 INFO:teuthology.orchestra.run.smithi046.stdout: "manage_etc_ceph_ceph_conf": { 2024-03-31T23:37:05.831 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-03-31T23:37:05.831 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.831 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.831 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.831 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-31T23:37:05.831 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.831 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.832 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.832 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-03-31T23:37:05.832 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.832 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.832 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.832 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.832 INFO:teuthology.orchestra.run.smithi046.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-03-31T23:37:05.832 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-03-31T23:37:05.832 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.832 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.832 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.832 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "*", 2024-03-31T23:37:05.832 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.832 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.832 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.833 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-03-31T23:37:05.833 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.833 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.833 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.833 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.833 INFO:teuthology.orchestra.run.smithi046.stdout: "max_count_per_host": { 2024-03-31T23:37:05.833 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "max_count_per_host", 2024-03-31T23:37:05.833 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "int", 2024-03-31T23:37:05.833 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.833 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.833 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "10", 2024-03-31T23:37:05.833 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.833 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.833 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.834 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "max number of daemons per service per host", 2024-03-31T23:37:05.834 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.834 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.834 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.834 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.834 INFO:teuthology.orchestra.run.smithi046.stdout: "max_osd_draining_count": { 2024-03-31T23:37:05.834 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "max_osd_draining_count", 2024-03-31T23:37:05.834 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "int", 2024-03-31T23:37:05.834 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.834 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.834 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "10", 2024-03-31T23:37:05.834 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.834 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.834 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.835 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-03-31T23:37:05.835 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.835 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.835 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.835 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.835 INFO:teuthology.orchestra.run.smithi046.stdout: "migration_current": { 2024-03-31T23:37:05.835 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "migration_current", 2024-03-31T23:37:05.835 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "int", 2024-03-31T23:37:05.835 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.835 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.835 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-31T23:37:05.835 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.835 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.835 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.836 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "internal - do not modify", 2024-03-31T23:37:05.836 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.836 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.836 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.836 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.836 INFO:teuthology.orchestra.run.smithi046.stdout: "mode": { 2024-03-31T23:37:05.836 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "mode", 2024-03-31T23:37:05.838 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.838 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.838 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.838 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "root", 2024-03-31T23:37:05.838 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.838 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.838 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-31T23:37:05.838 INFO:teuthology.orchestra.run.smithi046.stdout: "cephadm-package", 2024-03-31T23:37:05.838 INFO:teuthology.orchestra.run.smithi046.stdout: "root" 2024-03-31T23:37:05.838 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-31T23:37:05.838 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "mode for remote execution of cephadm", 2024-03-31T23:37:05.839 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.839 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.839 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.839 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.839 INFO:teuthology.orchestra.run.smithi046.stdout: "oob_default_addr": { 2024-03-31T23:37:05.839 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "oob_default_addr", 2024-03-31T23:37:05.839 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.839 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.839 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.839 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "169.254.1.1", 2024-03-31T23:37:05.839 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.839 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.839 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.839 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Default address for RedFish API (oob management).", 2024-03-31T23:37:05.839 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.840 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.840 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.840 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.840 INFO:teuthology.orchestra.run.smithi046.stdout: "prometheus_alerts_path": { 2024-03-31T23:37:05.840 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "prometheus_alerts_path", 2024-03-31T23:37:05.840 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.840 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.840 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.840 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-03-31T23:37:05.840 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.840 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.840 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.840 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-03-31T23:37:05.840 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.841 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.841 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.841 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.841 INFO:teuthology.orchestra.run.smithi046.stdout: "registry_insecure": { 2024-03-31T23:37:05.841 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "registry_insecure", 2024-03-31T23:37:05.841 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.841 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.841 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.841 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-31T23:37:05.841 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.841 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.841 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.841 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-03-31T23:37:05.841 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.842 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.842 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.842 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.842 INFO:teuthology.orchestra.run.smithi046.stdout: "registry_password": { 2024-03-31T23:37:05.842 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "registry_password", 2024-03-31T23:37:05.842 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.842 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.842 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.842 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-31T23:37:05.842 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.842 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.842 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.842 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-03-31T23:37:05.842 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.842 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.843 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.843 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.843 INFO:teuthology.orchestra.run.smithi046.stdout: "registry_url": { 2024-03-31T23:37:05.843 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "registry_url", 2024-03-31T23:37:05.843 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.843 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.843 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.843 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-31T23:37:05.843 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.843 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.843 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.843 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-03-31T23:37:05.843 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.843 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.843 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.844 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.844 INFO:teuthology.orchestra.run.smithi046.stdout: "registry_username": { 2024-03-31T23:37:05.844 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "registry_username", 2024-03-31T23:37:05.844 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.844 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.844 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.844 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-31T23:37:05.844 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.844 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.844 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.844 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-03-31T23:37:05.844 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.844 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.844 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.845 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.845 INFO:teuthology.orchestra.run.smithi046.stdout: "secure_monitoring_stack": { 2024-03-31T23:37:05.845 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "secure_monitoring_stack", 2024-03-31T23:37:05.845 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.845 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.845 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.845 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-31T23:37:05.845 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.845 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.845 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.845 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-03-31T23:37:05.845 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.845 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.845 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.845 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.846 INFO:teuthology.orchestra.run.smithi046.stdout: "service_discovery_port": { 2024-03-31T23:37:05.846 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "service_discovery_port", 2024-03-31T23:37:05.846 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "int", 2024-03-31T23:37:05.846 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.846 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.846 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "8765", 2024-03-31T23:37:05.846 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.846 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.846 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.846 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "cephadm service discovery port", 2024-03-31T23:37:05.846 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.846 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.846 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.846 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.847 INFO:teuthology.orchestra.run.smithi046.stdout: "ssh_config_file": { 2024-03-31T23:37:05.847 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "ssh_config_file", 2024-03-31T23:37:05.847 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.847 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.847 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.847 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-31T23:37:05.847 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.847 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.847 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.847 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-03-31T23:37:05.847 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.847 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.847 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.847 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.848 INFO:teuthology.orchestra.run.smithi046.stdout: "use_agent": { 2024-03-31T23:37:05.848 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "use_agent", 2024-03-31T23:37:05.848 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.848 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.848 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.848 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-31T23:37:05.848 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.848 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.848 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.848 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-03-31T23:37:05.848 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.848 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.848 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.848 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.848 INFO:teuthology.orchestra.run.smithi046.stdout: "use_repo_digest": { 2024-03-31T23:37:05.849 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "use_repo_digest", 2024-03-31T23:37:05.849 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.849 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.849 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.849 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "True", 2024-03-31T23:37:05.849 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.849 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.849 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.849 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-03-31T23:37:05.849 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.849 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.849 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.849 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.849 INFO:teuthology.orchestra.run.smithi046.stdout: "warn_on_failed_host_check": { 2024-03-31T23:37:05.850 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "warn_on_failed_host_check", 2024-03-31T23:37:05.850 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.850 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.850 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.850 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "True", 2024-03-31T23:37:05.850 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.850 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.850 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.850 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "raise a health warning if the host check fails", 2024-03-31T23:37:05.850 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.850 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.850 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.850 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.850 INFO:teuthology.orchestra.run.smithi046.stdout: "warn_on_stray_daemons": { 2024-03-31T23:37:05.850 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "warn_on_stray_daemons", 2024-03-31T23:37:05.851 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.851 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.851 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.851 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "True", 2024-03-31T23:37:05.851 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.851 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.851 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.851 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-03-31T23:37:05.851 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.851 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.851 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.851 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.851 INFO:teuthology.orchestra.run.smithi046.stdout: "warn_on_stray_hosts": { 2024-03-31T23:37:05.851 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "warn_on_stray_hosts", 2024-03-31T23:37:05.852 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.852 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.852 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.852 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "True", 2024-03-31T23:37:05.852 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.852 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.852 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.852 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-03-31T23:37:05.852 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.852 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.852 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.852 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-31T23:37:05.852 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-31T23:37:05.852 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.852 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-03-31T23:37:05.853 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "crash", 2024-03-31T23:37:05.853 INFO:teuthology.orchestra.run.smithi046.stdout: "can_run": true, 2024-03-31T23:37:05.853 INFO:teuthology.orchestra.run.smithi046.stdout: "error_string": "", 2024-03-31T23:37:05.853 INFO:teuthology.orchestra.run.smithi046.stdout: "module_options": { 2024-03-31T23:37:05.853 INFO:teuthology.orchestra.run.smithi046.stdout: "log_level": { 2024-03-31T23:37:05.853 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_level", 2024-03-31T23:37:05.853 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.853 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.853 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.853 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-31T23:37:05.853 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.853 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.853 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-31T23:37:05.853 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-31T23:37:05.853 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-31T23:37:05.854 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-31T23:37:05.854 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-31T23:37:05.854 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-31T23:37:05.854 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-31T23:37:05.854 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-31T23:37:05.854 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.854 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.854 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.854 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.854 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.854 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster": { 2024-03-31T23:37:05.854 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster", 2024-03-31T23:37:05.854 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.855 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.855 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.855 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-31T23:37:05.855 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.855 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.855 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.855 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.855 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.855 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.855 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.855 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.855 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster_level": { 2024-03-31T23:37:05.855 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster_level", 2024-03-31T23:37:05.855 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.855 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.856 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.856 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "info", 2024-03-31T23:37:05.856 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.856 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.856 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-31T23:37:05.856 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-31T23:37:05.856 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-31T23:37:05.856 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-31T23:37:05.856 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-31T23:37:05.856 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-31T23:37:05.856 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-31T23:37:05.856 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-31T23:37:05.856 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.856 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.856 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.857 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.857 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.857 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_file": { 2024-03-31T23:37:05.857 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_file", 2024-03-31T23:37:05.857 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.857 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.857 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.857 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-31T23:37:05.857 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.857 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.857 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.857 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.857 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.857 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.857 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.858 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.858 INFO:teuthology.orchestra.run.smithi046.stdout: "retain_interval": { 2024-03-31T23:37:05.858 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "retain_interval", 2024-03-31T23:37:05.858 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "secs", 2024-03-31T23:37:05.858 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.858 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.858 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "31536000", 2024-03-31T23:37:05.858 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.858 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.858 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.858 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "how long to retain crashes before pruning them", 2024-03-31T23:37:05.858 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.858 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.858 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.859 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.859 INFO:teuthology.orchestra.run.smithi046.stdout: "warn_recent_interval": { 2024-03-31T23:37:05.859 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "warn_recent_interval", 2024-03-31T23:37:05.859 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "secs", 2024-03-31T23:37:05.859 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.859 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.859 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "1209600", 2024-03-31T23:37:05.859 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.859 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.859 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.859 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "time interval in which to warn about recent crashes", 2024-03-31T23:37:05.859 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.859 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.859 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.860 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-31T23:37:05.860 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-31T23:37:05.860 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.860 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-03-31T23:37:05.860 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "dashboard", 2024-03-31T23:37:05.860 INFO:teuthology.orchestra.run.smithi046.stdout: "can_run": true, 2024-03-31T23:37:05.860 INFO:teuthology.orchestra.run.smithi046.stdout: "error_string": "", 2024-03-31T23:37:05.860 INFO:teuthology.orchestra.run.smithi046.stdout: "module_options": { 2024-03-31T23:37:05.860 INFO:teuthology.orchestra.run.smithi046.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-03-31T23:37:05.860 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-03-31T23:37:05.860 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "int", 2024-03-31T23:37:05.860 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.860 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.860 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "10", 2024-03-31T23:37:05.860 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.861 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.861 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.861 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.861 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.861 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.861 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.861 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.861 INFO:teuthology.orchestra.run.smithi046.stdout: "ALERTMANAGER_API_HOST": { 2024-03-31T23:37:05.861 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "ALERTMANAGER_API_HOST", 2024-03-31T23:37:05.861 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.861 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.861 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.861 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-31T23:37:05.861 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.862 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.862 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.862 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.862 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.862 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.862 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.862 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.862 INFO:teuthology.orchestra.run.smithi046.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-03-31T23:37:05.862 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-03-31T23:37:05.862 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.862 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.862 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.862 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "True", 2024-03-31T23:37:05.863 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.863 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.863 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.863 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.863 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.863 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.863 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.863 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.863 INFO:teuthology.orchestra.run.smithi046.stdout: "AUDIT_API_ENABLED": { 2024-03-31T23:37:05.863 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "AUDIT_API_ENABLED", 2024-03-31T23:37:05.863 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.863 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.863 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.864 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-31T23:37:05.864 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.864 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.864 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.864 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.864 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.864 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.864 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.864 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.864 INFO:teuthology.orchestra.run.smithi046.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-03-31T23:37:05.864 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-03-31T23:37:05.864 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.864 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.864 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.865 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "True", 2024-03-31T23:37:05.865 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.865 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.865 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.865 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.865 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.865 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.865 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.865 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.865 INFO:teuthology.orchestra.run.smithi046.stdout: "ENABLE_BROWSABLE_API": { 2024-03-31T23:37:05.865 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "ENABLE_BROWSABLE_API", 2024-03-31T23:37:05.865 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.865 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.865 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.866 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "True", 2024-03-31T23:37:05.866 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.866 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.866 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.866 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.866 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.866 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.866 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.866 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.866 INFO:teuthology.orchestra.run.smithi046.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-03-31T23:37:05.866 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-03-31T23:37:05.866 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.866 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.866 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.867 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "True", 2024-03-31T23:37:05.867 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.867 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.867 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.867 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.867 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.867 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.867 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.867 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.867 INFO:teuthology.orchestra.run.smithi046.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-03-31T23:37:05.867 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-03-31T23:37:05.867 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.867 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.867 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.868 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "True", 2024-03-31T23:37:05.868 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.868 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.868 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.868 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.868 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.868 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.868 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.868 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.868 INFO:teuthology.orchestra.run.smithi046.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-03-31T23:37:05.868 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-03-31T23:37:05.868 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.868 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.868 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.869 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "True", 2024-03-31T23:37:05.869 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.869 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.869 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.869 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.869 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.869 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.869 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.869 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.869 INFO:teuthology.orchestra.run.smithi046.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-03-31T23:37:05.869 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-03-31T23:37:05.869 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.869 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.869 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.870 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "True", 2024-03-31T23:37:05.870 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.870 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.870 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.870 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.870 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.870 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.870 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.870 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.870 INFO:teuthology.orchestra.run.smithi046.stdout: "FEATURE_TOGGLE_NFS": { 2024-03-31T23:37:05.870 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-03-31T23:37:05.870 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.870 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.870 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.871 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "True", 2024-03-31T23:37:05.871 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.871 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.871 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.871 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.871 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.871 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.871 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.871 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.871 INFO:teuthology.orchestra.run.smithi046.stdout: "FEATURE_TOGGLE_RBD": { 2024-03-31T23:37:05.871 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-03-31T23:37:05.871 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.871 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.871 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.872 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "True", 2024-03-31T23:37:05.872 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.872 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.872 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.872 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.872 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.872 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.872 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.872 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.872 INFO:teuthology.orchestra.run.smithi046.stdout: "FEATURE_TOGGLE_RGW": { 2024-03-31T23:37:05.872 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-03-31T23:37:05.872 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.872 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.872 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.872 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "True", 2024-03-31T23:37:05.873 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.873 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.873 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.873 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.873 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.873 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.873 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.873 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.873 INFO:teuthology.orchestra.run.smithi046.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-03-31T23:37:05.873 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-03-31T23:37:05.873 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.873 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.873 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.873 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-31T23:37:05.873 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.874 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.874 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.874 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.874 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.874 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.874 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.874 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.874 INFO:teuthology.orchestra.run.smithi046.stdout: "GRAFANA_API_PASSWORD": { 2024-03-31T23:37:05.874 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "GRAFANA_API_PASSWORD", 2024-03-31T23:37:05.874 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.874 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.874 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.874 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "admin", 2024-03-31T23:37:05.874 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.874 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.875 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.875 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.875 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.875 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.875 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.875 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.875 INFO:teuthology.orchestra.run.smithi046.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-03-31T23:37:05.875 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-03-31T23:37:05.875 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.875 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.875 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.875 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "True", 2024-03-31T23:37:05.875 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.875 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.875 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.876 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.876 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.876 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.876 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.876 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.876 INFO:teuthology.orchestra.run.smithi046.stdout: "GRAFANA_API_URL": { 2024-03-31T23:37:05.876 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "GRAFANA_API_URL", 2024-03-31T23:37:05.876 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.876 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.876 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.876 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-31T23:37:05.876 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.876 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.876 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.876 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.877 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.877 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.877 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.877 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.877 INFO:teuthology.orchestra.run.smithi046.stdout: "GRAFANA_API_USERNAME": { 2024-03-31T23:37:05.877 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "GRAFANA_API_USERNAME", 2024-03-31T23:37:05.877 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.877 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.877 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.877 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "admin", 2024-03-31T23:37:05.877 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.877 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.877 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.877 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.877 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.878 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.878 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.878 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.878 INFO:teuthology.orchestra.run.smithi046.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-03-31T23:37:05.878 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-03-31T23:37:05.878 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.878 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.878 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.878 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-31T23:37:05.878 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.878 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.878 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.878 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.878 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.878 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.879 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.879 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.879 INFO:teuthology.orchestra.run.smithi046.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-03-31T23:37:05.879 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-03-31T23:37:05.879 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.879 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.879 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.879 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-31T23:37:05.879 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.879 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.879 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.879 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.879 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.879 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.879 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.880 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.880 INFO:teuthology.orchestra.run.smithi046.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-03-31T23:37:05.880 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-03-31T23:37:05.880 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.880 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.880 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.880 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "True", 2024-03-31T23:37:05.880 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.880 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.880 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.880 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.880 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.880 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.880 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.881 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.881 INFO:teuthology.orchestra.run.smithi046.stdout: "ISSUE_TRACKER_API_KEY": { 2024-03-31T23:37:05.881 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-03-31T23:37:05.881 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.881 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.881 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.881 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-31T23:37:05.881 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.881 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.881 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.881 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.881 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.881 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.881 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.881 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.882 INFO:teuthology.orchestra.run.smithi046.stdout: "PROMETHEUS_API_HOST": { 2024-03-31T23:37:05.882 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "PROMETHEUS_API_HOST", 2024-03-31T23:37:05.882 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.882 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.882 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.882 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-31T23:37:05.882 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.882 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.882 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.882 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.882 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.882 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.882 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.882 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.882 INFO:teuthology.orchestra.run.smithi046.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-03-31T23:37:05.883 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-03-31T23:37:05.883 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.883 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.883 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.883 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "True", 2024-03-31T23:37:05.883 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.883 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.883 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.883 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.883 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.883 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.883 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.883 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.883 INFO:teuthology.orchestra.run.smithi046.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-03-31T23:37:05.884 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-03-31T23:37:05.884 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.884 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.884 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.884 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-31T23:37:05.884 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.884 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.884 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.884 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.884 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.884 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.884 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.884 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.884 INFO:teuthology.orchestra.run.smithi046.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-03-31T23:37:05.884 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-03-31T23:37:05.885 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.885 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.885 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.885 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-31T23:37:05.885 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.885 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.885 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.885 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.885 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.885 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.885 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.885 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.885 INFO:teuthology.orchestra.run.smithi046.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-03-31T23:37:05.885 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-03-31T23:37:05.886 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.886 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.886 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.886 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "True", 2024-03-31T23:37:05.886 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.886 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.886 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.886 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.886 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.886 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.886 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.886 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.886 INFO:teuthology.orchestra.run.smithi046.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-03-31T23:37:05.886 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-03-31T23:37:05.887 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.887 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.887 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.887 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "True", 2024-03-31T23:37:05.887 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.887 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.887 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.887 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.887 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.887 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.887 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.887 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.887 INFO:teuthology.orchestra.run.smithi046.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-03-31T23:37:05.887 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-03-31T23:37:05.888 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.888 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.888 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.888 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-31T23:37:05.888 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.888 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.888 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.888 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.888 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.888 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.888 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.888 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.888 INFO:teuthology.orchestra.run.smithi046.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-03-31T23:37:05.888 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-03-31T23:37:05.889 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.889 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.889 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.889 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-31T23:37:05.889 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.889 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.889 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.889 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.889 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.889 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.889 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.889 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.889 INFO:teuthology.orchestra.run.smithi046.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-03-31T23:37:05.889 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-03-31T23:37:05.889 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.890 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.890 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.890 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-31T23:37:05.890 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.890 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.890 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.890 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.890 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.890 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.890 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.890 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.890 INFO:teuthology.orchestra.run.smithi046.stdout: "PWD_POLICY_ENABLED": { 2024-03-31T23:37:05.890 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "PWD_POLICY_ENABLED", 2024-03-31T23:37:05.890 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.890 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.891 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.891 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "True", 2024-03-31T23:37:05.891 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.891 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.891 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.891 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.891 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.891 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.891 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.891 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.891 INFO:teuthology.orchestra.run.smithi046.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-03-31T23:37:05.891 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-03-31T23:37:05.891 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.891 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.892 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.892 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-03-31T23:37:05.892 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.892 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.892 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.892 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.892 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.892 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.892 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.892 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.892 INFO:teuthology.orchestra.run.smithi046.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-03-31T23:37:05.892 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-03-31T23:37:05.892 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "int", 2024-03-31T23:37:05.892 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.892 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.893 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "10", 2024-03-31T23:37:05.893 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.893 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.893 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.893 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.893 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.893 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.893 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.893 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.893 INFO:teuthology.orchestra.run.smithi046.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-03-31T23:37:05.893 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-03-31T23:37:05.893 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "int", 2024-03-31T23:37:05.893 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.893 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.893 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "8", 2024-03-31T23:37:05.894 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.894 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.894 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.894 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.894 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.894 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.894 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.894 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.894 INFO:teuthology.orchestra.run.smithi046.stdout: "REST_REQUESTS_TIMEOUT": { 2024-03-31T23:37:05.894 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-03-31T23:37:05.894 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "int", 2024-03-31T23:37:05.894 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.894 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.894 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "45", 2024-03-31T23:37:05.894 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.895 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.895 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.895 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.895 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.895 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.895 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.895 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.895 INFO:teuthology.orchestra.run.smithi046.stdout: "RGW_API_ACCESS_KEY": { 2024-03-31T23:37:05.895 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "RGW_API_ACCESS_KEY", 2024-03-31T23:37:05.895 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.895 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.895 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.895 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-31T23:37:05.895 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.896 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.896 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.896 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.896 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.896 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.896 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.896 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.896 INFO:teuthology.orchestra.run.smithi046.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-03-31T23:37:05.896 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-03-31T23:37:05.896 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.896 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.896 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.896 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "admin", 2024-03-31T23:37:05.896 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.897 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.897 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.897 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.897 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.897 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.897 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.897 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.897 INFO:teuthology.orchestra.run.smithi046.stdout: "RGW_API_SECRET_KEY": { 2024-03-31T23:37:05.897 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "RGW_API_SECRET_KEY", 2024-03-31T23:37:05.897 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.897 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.897 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.897 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-31T23:37:05.897 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.897 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.898 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.898 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.898 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.898 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.898 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.898 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.898 INFO:teuthology.orchestra.run.smithi046.stdout: "RGW_API_SSL_VERIFY": { 2024-03-31T23:37:05.898 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "RGW_API_SSL_VERIFY", 2024-03-31T23:37:05.898 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.898 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.898 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.898 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "True", 2024-03-31T23:37:05.898 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.898 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.899 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.899 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.899 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.899 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.899 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.899 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.899 INFO:teuthology.orchestra.run.smithi046.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-03-31T23:37:05.899 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-03-31T23:37:05.899 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "int", 2024-03-31T23:37:05.899 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.899 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.899 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "0", 2024-03-31T23:37:05.899 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.899 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.899 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.900 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.900 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.900 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.900 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.900 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.900 INFO:teuthology.orchestra.run.smithi046.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-03-31T23:37:05.900 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-03-31T23:37:05.900 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "int", 2024-03-31T23:37:05.900 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.900 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.900 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "10", 2024-03-31T23:37:05.900 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.900 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.900 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.901 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.901 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.901 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.901 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.901 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.901 INFO:teuthology.orchestra.run.smithi046.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-03-31T23:37:05.901 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-03-31T23:37:05.901 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "int", 2024-03-31T23:37:05.901 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.901 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.901 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "5", 2024-03-31T23:37:05.901 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.901 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.901 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.902 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.902 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.902 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.902 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.902 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.902 INFO:teuthology.orchestra.run.smithi046.stdout: "cross_origin_url": { 2024-03-31T23:37:05.902 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "cross_origin_url", 2024-03-31T23:37:05.902 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.902 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.902 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.902 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-31T23:37:05.902 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.902 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.903 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.903 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.903 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.903 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.903 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.903 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.903 INFO:teuthology.orchestra.run.smithi046.stdout: "crt_file": { 2024-03-31T23:37:05.903 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "crt_file", 2024-03-31T23:37:05.903 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.904 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.904 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.904 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-31T23:37:05.904 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.904 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.904 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.904 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.904 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.904 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.904 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.904 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.904 INFO:teuthology.orchestra.run.smithi046.stdout: "debug": { 2024-03-31T23:37:05.904 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "debug", 2024-03-31T23:37:05.904 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.905 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.905 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.905 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-31T23:37:05.905 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.905 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.905 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.905 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Enable/disable debug options", 2024-03-31T23:37:05.905 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.905 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.905 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.905 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.905 INFO:teuthology.orchestra.run.smithi046.stdout: "jwt_token_ttl": { 2024-03-31T23:37:05.905 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "jwt_token_ttl", 2024-03-31T23:37:05.905 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "int", 2024-03-31T23:37:05.905 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.906 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.906 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "28800", 2024-03-31T23:37:05.906 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.906 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.906 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.906 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.906 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.906 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.906 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.906 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.906 INFO:teuthology.orchestra.run.smithi046.stdout: "key_file": { 2024-03-31T23:37:05.906 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "key_file", 2024-03-31T23:37:05.906 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.906 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.907 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.907 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-31T23:37:05.907 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.907 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.907 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.907 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.907 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.907 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.907 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.907 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.907 INFO:teuthology.orchestra.run.smithi046.stdout: "log_level": { 2024-03-31T23:37:05.907 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_level", 2024-03-31T23:37:05.907 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.907 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.908 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.908 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-31T23:37:05.908 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.908 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.908 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-31T23:37:05.908 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-31T23:37:05.908 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-31T23:37:05.908 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-31T23:37:05.908 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-31T23:37:05.908 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-31T23:37:05.908 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-31T23:37:05.908 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-31T23:37:05.908 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.908 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.909 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.909 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.909 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.909 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster": { 2024-03-31T23:37:05.909 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster", 2024-03-31T23:37:05.909 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.909 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.909 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.909 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-31T23:37:05.909 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.909 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.909 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.909 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.909 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.910 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.910 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.910 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.910 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster_level": { 2024-03-31T23:37:05.910 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster_level", 2024-03-31T23:37:05.910 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.910 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.910 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.910 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "info", 2024-03-31T23:37:05.910 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.910 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.910 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-31T23:37:05.910 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-31T23:37:05.910 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-31T23:37:05.910 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-31T23:37:05.911 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-31T23:37:05.911 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-31T23:37:05.911 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-31T23:37:05.911 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-31T23:37:05.911 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.911 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.911 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.911 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.911 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.911 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_file": { 2024-03-31T23:37:05.911 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_file", 2024-03-31T23:37:05.911 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.911 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.911 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.911 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-31T23:37:05.912 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.912 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.912 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.912 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.912 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.912 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.912 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.912 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.912 INFO:teuthology.orchestra.run.smithi046.stdout: "motd": { 2024-03-31T23:37:05.912 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "motd", 2024-03-31T23:37:05.912 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.912 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.912 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.913 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-31T23:37:05.913 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.913 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.913 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.913 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "The message of the day", 2024-03-31T23:37:05.913 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.913 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.913 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.913 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.913 INFO:teuthology.orchestra.run.smithi046.stdout: "redirect_resolve_ip_addr": { 2024-03-31T23:37:05.913 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "redirect_resolve_ip_addr", 2024-03-31T23:37:05.913 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.913 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.913 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.914 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-31T23:37:05.914 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.914 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.914 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.914 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.914 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.914 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.914 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.914 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.914 INFO:teuthology.orchestra.run.smithi046.stdout: "server_addr": { 2024-03-31T23:37:05.914 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "server_addr", 2024-03-31T23:37:05.914 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.914 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.914 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.914 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "::", 2024-03-31T23:37:05.915 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.915 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.915 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.915 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.915 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.915 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.915 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.915 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.915 INFO:teuthology.orchestra.run.smithi046.stdout: "server_port": { 2024-03-31T23:37:05.915 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "server_port", 2024-03-31T23:37:05.915 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "int", 2024-03-31T23:37:05.915 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.915 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.915 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "8080", 2024-03-31T23:37:05.916 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.916 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.916 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.916 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.916 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.916 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.916 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.916 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.916 INFO:teuthology.orchestra.run.smithi046.stdout: "ssl": { 2024-03-31T23:37:05.916 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "ssl", 2024-03-31T23:37:05.916 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.916 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.916 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.916 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "True", 2024-03-31T23:37:05.916 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.917 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.917 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.917 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.917 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.917 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.917 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.917 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.917 INFO:teuthology.orchestra.run.smithi046.stdout: "ssl_server_port": { 2024-03-31T23:37:05.917 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "ssl_server_port", 2024-03-31T23:37:05.917 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "int", 2024-03-31T23:37:05.917 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.917 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.917 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "8443", 2024-03-31T23:37:05.917 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.918 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.918 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.918 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.918 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.918 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.918 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.918 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.918 INFO:teuthology.orchestra.run.smithi046.stdout: "standby_behaviour": { 2024-03-31T23:37:05.918 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "standby_behaviour", 2024-03-31T23:37:05.918 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.918 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.918 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.918 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "redirect", 2024-03-31T23:37:05.918 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.919 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.919 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-31T23:37:05.919 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-31T23:37:05.919 INFO:teuthology.orchestra.run.smithi046.stdout: "redirect" 2024-03-31T23:37:05.919 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-31T23:37:05.919 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.919 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.919 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.919 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.919 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.919 INFO:teuthology.orchestra.run.smithi046.stdout: "standby_error_status_code": { 2024-03-31T23:37:05.919 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "standby_error_status_code", 2024-03-31T23:37:05.919 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "int", 2024-03-31T23:37:05.919 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.920 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.920 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "500", 2024-03-31T23:37:05.920 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "400", 2024-03-31T23:37:05.920 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "599", 2024-03-31T23:37:05.920 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.920 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.920 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.920 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.920 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.920 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.920 INFO:teuthology.orchestra.run.smithi046.stdout: "url_prefix": { 2024-03-31T23:37:05.920 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "url_prefix", 2024-03-31T23:37:05.920 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.920 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.920 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.921 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-31T23:37:05.921 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.921 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.921 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.921 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.921 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.921 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.921 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.921 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-31T23:37:05.921 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-31T23:37:05.921 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.921 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-03-31T23:37:05.921 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "devicehealth", 2024-03-31T23:37:05.921 INFO:teuthology.orchestra.run.smithi046.stdout: "can_run": true, 2024-03-31T23:37:05.922 INFO:teuthology.orchestra.run.smithi046.stdout: "error_string": "", 2024-03-31T23:37:05.922 INFO:teuthology.orchestra.run.smithi046.stdout: "module_options": { 2024-03-31T23:37:05.922 INFO:teuthology.orchestra.run.smithi046.stdout: "enable_monitoring": { 2024-03-31T23:37:05.922 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "enable_monitoring", 2024-03-31T23:37:05.922 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.922 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.922 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.922 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "True", 2024-03-31T23:37:05.922 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.922 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.922 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.922 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "monitor device health metrics", 2024-03-31T23:37:05.922 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.922 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.923 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.923 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.923 INFO:teuthology.orchestra.run.smithi046.stdout: "log_level": { 2024-03-31T23:37:05.923 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_level", 2024-03-31T23:37:05.923 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.923 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.923 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.923 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-31T23:37:05.923 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.923 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.923 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-31T23:37:05.923 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-31T23:37:05.923 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-31T23:37:05.923 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-31T23:37:05.923 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-31T23:37:05.924 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-31T23:37:05.924 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-31T23:37:05.924 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-31T23:37:05.924 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.924 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.924 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.924 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.924 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.924 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster": { 2024-03-31T23:37:05.924 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster", 2024-03-31T23:37:05.924 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.924 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.924 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.924 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-31T23:37:05.925 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.925 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.925 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.925 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.925 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.925 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.925 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.925 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.925 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster_level": { 2024-03-31T23:37:05.925 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster_level", 2024-03-31T23:37:05.925 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.925 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.925 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.925 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "info", 2024-03-31T23:37:05.925 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.926 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.926 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-31T23:37:05.926 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-31T23:37:05.926 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-31T23:37:05.926 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-31T23:37:05.926 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-31T23:37:05.926 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-31T23:37:05.926 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-31T23:37:05.926 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-31T23:37:05.926 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.926 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.926 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.926 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.926 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.926 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_file": { 2024-03-31T23:37:05.927 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_file", 2024-03-31T23:37:05.927 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.927 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.927 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.927 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-31T23:37:05.927 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.927 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.927 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.927 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.927 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.927 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.927 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.927 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.927 INFO:teuthology.orchestra.run.smithi046.stdout: "mark_out_threshold": { 2024-03-31T23:37:05.927 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "mark_out_threshold", 2024-03-31T23:37:05.928 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "secs", 2024-03-31T23:37:05.928 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.928 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.928 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "2419200", 2024-03-31T23:37:05.928 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.928 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.928 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.928 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-03-31T23:37:05.928 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.928 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.928 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.928 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.928 INFO:teuthology.orchestra.run.smithi046.stdout: "pool_name": { 2024-03-31T23:37:05.928 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "pool_name", 2024-03-31T23:37:05.929 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.929 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.929 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.929 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "device_health_metrics", 2024-03-31T23:37:05.929 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.929 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.929 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.929 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "name of pool in which to store device health metrics", 2024-03-31T23:37:05.929 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.929 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.929 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.929 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.929 INFO:teuthology.orchestra.run.smithi046.stdout: "retention_period": { 2024-03-31T23:37:05.929 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "retention_period", 2024-03-31T23:37:05.930 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "secs", 2024-03-31T23:37:05.930 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.930 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.930 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "15552000", 2024-03-31T23:37:05.930 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.930 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.930 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.930 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "how long to retain device health metrics", 2024-03-31T23:37:05.930 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.930 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.930 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.930 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.930 INFO:teuthology.orchestra.run.smithi046.stdout: "scrape_frequency": { 2024-03-31T23:37:05.930 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "scrape_frequency", 2024-03-31T23:37:05.930 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "secs", 2024-03-31T23:37:05.931 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.931 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.931 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "86400", 2024-03-31T23:37:05.931 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.931 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.931 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.931 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "how frequently to scrape device health metrics", 2024-03-31T23:37:05.931 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.931 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.931 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.931 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.931 INFO:teuthology.orchestra.run.smithi046.stdout: "self_heal": { 2024-03-31T23:37:05.931 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "self_heal", 2024-03-31T23:37:05.931 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.931 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.932 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.932 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "True", 2024-03-31T23:37:05.932 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.932 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.932 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.932 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-03-31T23:37:05.932 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.932 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.932 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.932 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.932 INFO:teuthology.orchestra.run.smithi046.stdout: "sleep_interval": { 2024-03-31T23:37:05.932 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "sleep_interval", 2024-03-31T23:37:05.932 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "secs", 2024-03-31T23:37:05.932 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.932 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.933 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "600", 2024-03-31T23:37:05.933 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.933 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.933 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.933 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "how frequently to wake up and check device health", 2024-03-31T23:37:05.933 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.933 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.933 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.933 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.933 INFO:teuthology.orchestra.run.smithi046.stdout: "warn_threshold": { 2024-03-31T23:37:05.933 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "warn_threshold", 2024-03-31T23:37:05.933 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "secs", 2024-03-31T23:37:05.933 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.933 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.933 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "7257600", 2024-03-31T23:37:05.934 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.934 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.934 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.934 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-03-31T23:37:05.934 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.934 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.934 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.934 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-31T23:37:05.934 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-31T23:37:05.934 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.934 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-03-31T23:37:05.934 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "diskprediction_local", 2024-03-31T23:37:05.934 INFO:teuthology.orchestra.run.smithi046.stdout: "can_run": true, 2024-03-31T23:37:05.934 INFO:teuthology.orchestra.run.smithi046.stdout: "error_string": "", 2024-03-31T23:37:05.935 INFO:teuthology.orchestra.run.smithi046.stdout: "module_options": { 2024-03-31T23:37:05.935 INFO:teuthology.orchestra.run.smithi046.stdout: "log_level": { 2024-03-31T23:37:05.935 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_level", 2024-03-31T23:37:05.935 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.935 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.935 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.935 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-31T23:37:05.935 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.935 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.935 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-31T23:37:05.935 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-31T23:37:05.935 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-31T23:37:05.935 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-31T23:37:05.935 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-31T23:37:05.935 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-31T23:37:05.935 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-31T23:37:05.936 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-31T23:37:05.936 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.936 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.936 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.936 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.936 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.936 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster": { 2024-03-31T23:37:05.936 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster", 2024-03-31T23:37:05.936 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.936 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.936 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.936 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-31T23:37:05.936 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.936 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.936 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.937 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.937 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.937 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.937 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.937 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.937 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster_level": { 2024-03-31T23:37:05.937 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster_level", 2024-03-31T23:37:05.937 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.937 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.937 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.937 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "info", 2024-03-31T23:37:05.937 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.937 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.937 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-31T23:37:05.938 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-31T23:37:05.938 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-31T23:37:05.938 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-31T23:37:05.938 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-31T23:37:05.938 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-31T23:37:05.938 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-31T23:37:05.938 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-31T23:37:05.938 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.938 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.938 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.938 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.938 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.938 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_file": { 2024-03-31T23:37:05.938 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_file", 2024-03-31T23:37:05.939 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.939 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.939 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.939 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-31T23:37:05.939 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.939 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.939 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.939 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.939 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.939 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.939 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.939 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.939 INFO:teuthology.orchestra.run.smithi046.stdout: "predict_interval": { 2024-03-31T23:37:05.939 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "predict_interval", 2024-03-31T23:37:05.939 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.940 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.940 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.940 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "86400", 2024-03-31T23:37:05.940 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.940 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.940 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.940 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.940 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.940 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.940 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.940 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.940 INFO:teuthology.orchestra.run.smithi046.stdout: "predictor_model": { 2024-03-31T23:37:05.940 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "predictor_model", 2024-03-31T23:37:05.940 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.940 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.940 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.941 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "prophetstor", 2024-03-31T23:37:05.941 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.941 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.941 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.941 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.941 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.941 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.941 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.941 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.941 INFO:teuthology.orchestra.run.smithi046.stdout: "sleep_interval": { 2024-03-31T23:37:05.941 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "sleep_interval", 2024-03-31T23:37:05.941 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.941 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.941 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.942 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "600", 2024-03-31T23:37:05.942 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.942 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.942 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.942 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.942 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.942 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.942 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.942 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-31T23:37:05.942 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-31T23:37:05.942 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.942 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-03-31T23:37:05.942 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "influx", 2024-03-31T23:37:05.942 INFO:teuthology.orchestra.run.smithi046.stdout: "can_run": false, 2024-03-31T23:37:05.942 INFO:teuthology.orchestra.run.smithi046.stdout: "error_string": "influxdb python module not found", 2024-03-31T23:37:05.943 INFO:teuthology.orchestra.run.smithi046.stdout: "module_options": { 2024-03-31T23:37:05.943 INFO:teuthology.orchestra.run.smithi046.stdout: "batch_size": { 2024-03-31T23:37:05.943 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "batch_size", 2024-03-31T23:37:05.943 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "int", 2024-03-31T23:37:05.943 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.943 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.943 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "5000", 2024-03-31T23:37:05.943 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.943 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.943 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.943 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-03-31T23:37:05.943 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.944 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.944 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.944 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.944 INFO:teuthology.orchestra.run.smithi046.stdout: "database": { 2024-03-31T23:37:05.944 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "database", 2024-03-31T23:37:05.944 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.944 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.944 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.944 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "ceph", 2024-03-31T23:37:05.944 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.944 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.944 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.944 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-03-31T23:37:05.944 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.945 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.945 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.945 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.945 INFO:teuthology.orchestra.run.smithi046.stdout: "hostname": { 2024-03-31T23:37:05.945 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "hostname", 2024-03-31T23:37:05.945 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.945 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.945 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.945 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-31T23:37:05.945 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.945 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.945 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.945 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "InfluxDB server hostname", 2024-03-31T23:37:05.945 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.946 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.946 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.946 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.946 INFO:teuthology.orchestra.run.smithi046.stdout: "interval": { 2024-03-31T23:37:05.946 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "interval", 2024-03-31T23:37:05.946 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "secs", 2024-03-31T23:37:05.946 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.946 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.946 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "30", 2024-03-31T23:37:05.946 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "5", 2024-03-31T23:37:05.946 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.946 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.946 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-03-31T23:37:05.946 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.947 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.947 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.947 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.947 INFO:teuthology.orchestra.run.smithi046.stdout: "log_level": { 2024-03-31T23:37:05.947 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_level", 2024-03-31T23:37:05.947 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.947 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.947 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.947 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-31T23:37:05.947 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.947 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.947 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-31T23:37:05.947 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-31T23:37:05.947 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-31T23:37:05.947 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-31T23:37:05.948 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-31T23:37:05.948 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-31T23:37:05.948 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-31T23:37:05.948 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-31T23:37:05.948 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.948 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.948 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.948 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.948 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.948 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster": { 2024-03-31T23:37:05.948 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster", 2024-03-31T23:37:05.948 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.948 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.948 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.948 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-31T23:37:05.949 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.949 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.949 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.949 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.949 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.949 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.949 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.949 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.949 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster_level": { 2024-03-31T23:37:05.949 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster_level", 2024-03-31T23:37:05.949 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.949 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.949 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.949 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "info", 2024-03-31T23:37:05.949 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.950 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.950 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-31T23:37:05.950 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-31T23:37:05.950 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-31T23:37:05.950 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-31T23:37:05.950 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-31T23:37:05.950 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-31T23:37:05.950 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-31T23:37:05.950 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-31T23:37:05.950 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.950 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.950 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.950 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.950 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.951 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_file": { 2024-03-31T23:37:05.951 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_file", 2024-03-31T23:37:05.951 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.951 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.951 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.951 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-31T23:37:05.951 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.951 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.951 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.951 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.951 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.951 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.951 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.951 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.951 INFO:teuthology.orchestra.run.smithi046.stdout: "password": { 2024-03-31T23:37:05.952 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "password", 2024-03-31T23:37:05.952 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.952 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.952 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.952 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-31T23:37:05.952 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.952 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.952 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.952 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "password of InfluxDB server user", 2024-03-31T23:37:05.952 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.952 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.952 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.952 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.952 INFO:teuthology.orchestra.run.smithi046.stdout: "port": { 2024-03-31T23:37:05.952 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "port", 2024-03-31T23:37:05.952 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "int", 2024-03-31T23:37:05.953 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.953 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.953 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "8086", 2024-03-31T23:37:05.953 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.953 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.953 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.953 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "InfluxDB server port", 2024-03-31T23:37:05.953 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.953 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.953 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.953 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.953 INFO:teuthology.orchestra.run.smithi046.stdout: "ssl": { 2024-03-31T23:37:05.953 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "ssl", 2024-03-31T23:37:05.953 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.954 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.954 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.954 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "false", 2024-03-31T23:37:05.954 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.954 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.954 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.954 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-03-31T23:37:05.954 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.954 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.954 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.954 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.954 INFO:teuthology.orchestra.run.smithi046.stdout: "threads": { 2024-03-31T23:37:05.954 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "threads", 2024-03-31T23:37:05.954 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "int", 2024-03-31T23:37:05.955 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.955 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.955 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "5", 2024-03-31T23:37:05.955 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "1", 2024-03-31T23:37:05.955 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "32", 2024-03-31T23:37:05.955 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.955 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-03-31T23:37:05.955 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.955 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.955 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.955 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.955 INFO:teuthology.orchestra.run.smithi046.stdout: "username": { 2024-03-31T23:37:05.955 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "username", 2024-03-31T23:37:05.955 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.956 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.956 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.956 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-31T23:37:05.956 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.956 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.956 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.956 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "username of InfluxDB server user", 2024-03-31T23:37:05.956 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.956 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.956 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.956 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.956 INFO:teuthology.orchestra.run.smithi046.stdout: "verify_ssl": { 2024-03-31T23:37:05.956 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "verify_ssl", 2024-03-31T23:37:05.956 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.957 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.957 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.957 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "true", 2024-03-31T23:37:05.957 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.957 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.957 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.957 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-03-31T23:37:05.957 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.957 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.957 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.957 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-31T23:37:05.957 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-31T23:37:05.957 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.957 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-03-31T23:37:05.958 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "insights", 2024-03-31T23:37:05.958 INFO:teuthology.orchestra.run.smithi046.stdout: "can_run": true, 2024-03-31T23:37:05.958 INFO:teuthology.orchestra.run.smithi046.stdout: "error_string": "", 2024-03-31T23:37:05.958 INFO:teuthology.orchestra.run.smithi046.stdout: "module_options": { 2024-03-31T23:37:05.958 INFO:teuthology.orchestra.run.smithi046.stdout: "log_level": { 2024-03-31T23:37:05.958 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_level", 2024-03-31T23:37:05.958 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.958 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.958 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.958 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-31T23:37:05.958 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.958 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.958 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-31T23:37:05.958 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-31T23:37:05.959 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-31T23:37:05.959 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-31T23:37:05.959 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-31T23:37:05.959 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-31T23:37:05.959 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-31T23:37:05.959 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-31T23:37:05.959 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.959 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.959 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.959 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.959 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.959 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster": { 2024-03-31T23:37:05.959 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster", 2024-03-31T23:37:05.959 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.960 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.960 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.960 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-31T23:37:05.960 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.960 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.960 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.960 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.960 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.960 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.960 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.960 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.960 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster_level": { 2024-03-31T23:37:05.960 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster_level", 2024-03-31T23:37:05.960 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.961 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.961 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.961 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "info", 2024-03-31T23:37:05.961 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.961 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.961 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-31T23:37:05.961 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-31T23:37:05.961 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-31T23:37:05.961 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-31T23:37:05.961 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-31T23:37:05.961 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-31T23:37:05.961 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-31T23:37:05.961 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-31T23:37:05.961 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.961 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.962 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.962 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.962 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.962 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_file": { 2024-03-31T23:37:05.962 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_file", 2024-03-31T23:37:05.962 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.962 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.962 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.962 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-31T23:37:05.962 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.962 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.962 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.962 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.962 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.963 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.963 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.963 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-31T23:37:05.963 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-31T23:37:05.963 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.963 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-03-31T23:37:05.963 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "iostat", 2024-03-31T23:37:05.963 INFO:teuthology.orchestra.run.smithi046.stdout: "can_run": true, 2024-03-31T23:37:05.963 INFO:teuthology.orchestra.run.smithi046.stdout: "error_string": "", 2024-03-31T23:37:05.963 INFO:teuthology.orchestra.run.smithi046.stdout: "module_options": { 2024-03-31T23:37:05.963 INFO:teuthology.orchestra.run.smithi046.stdout: "log_level": { 2024-03-31T23:37:05.963 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_level", 2024-03-31T23:37:05.963 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.963 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.964 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.964 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-31T23:37:05.964 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.964 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.964 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-31T23:37:05.964 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-31T23:37:05.964 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-31T23:37:05.964 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-31T23:37:05.964 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-31T23:37:05.964 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-31T23:37:05.964 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-31T23:37:05.964 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-31T23:37:05.964 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.964 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.965 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.965 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.965 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.965 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster": { 2024-03-31T23:37:05.965 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster", 2024-03-31T23:37:05.965 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.965 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.965 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.965 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-31T23:37:05.965 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.965 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.965 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.965 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.965 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.966 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.966 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.966 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.966 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster_level": { 2024-03-31T23:37:05.966 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster_level", 2024-03-31T23:37:05.966 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.966 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.966 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.966 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "info", 2024-03-31T23:37:05.966 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.966 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.966 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-31T23:37:05.966 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-31T23:37:05.966 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-31T23:37:05.967 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-31T23:37:05.967 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-31T23:37:05.967 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-31T23:37:05.967 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-31T23:37:05.967 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-31T23:37:05.967 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.967 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.967 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.967 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.967 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.967 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_file": { 2024-03-31T23:37:05.967 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_file", 2024-03-31T23:37:05.967 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.967 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.968 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.968 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-31T23:37:05.968 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.968 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.968 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.968 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.968 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.968 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.968 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.968 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-31T23:37:05.968 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-31T23:37:05.968 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.968 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-03-31T23:37:05.968 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "k8sevents", 2024-03-31T23:37:05.968 INFO:teuthology.orchestra.run.smithi046.stdout: "can_run": true, 2024-03-31T23:37:05.969 INFO:teuthology.orchestra.run.smithi046.stdout: "error_string": "", 2024-03-31T23:37:05.969 INFO:teuthology.orchestra.run.smithi046.stdout: "module_options": { 2024-03-31T23:37:05.969 INFO:teuthology.orchestra.run.smithi046.stdout: "ceph_event_retention_days": { 2024-03-31T23:37:05.969 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "ceph_event_retention_days", 2024-03-31T23:37:05.969 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "int", 2024-03-31T23:37:05.969 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.969 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.969 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "7", 2024-03-31T23:37:05.969 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.969 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.969 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.969 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Days to hold ceph event information within local cache", 2024-03-31T23:37:05.969 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.969 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.969 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.970 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.970 INFO:teuthology.orchestra.run.smithi046.stdout: "config_check_secs": { 2024-03-31T23:37:05.970 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "config_check_secs", 2024-03-31T23:37:05.970 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "int", 2024-03-31T23:37:05.970 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.970 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:05.970 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "10", 2024-03-31T23:37:05.970 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "10", 2024-03-31T23:37:05.970 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.970 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.970 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-03-31T23:37:05.970 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.970 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.970 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.971 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.971 INFO:teuthology.orchestra.run.smithi046.stdout: "log_level": { 2024-03-31T23:37:05.971 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_level", 2024-03-31T23:37:05.971 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.971 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.971 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.971 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-31T23:37:05.971 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.971 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.971 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-31T23:37:05.971 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-31T23:37:05.971 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-31T23:37:05.971 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-31T23:37:05.971 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-31T23:37:05.971 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-31T23:37:05.972 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-31T23:37:05.972 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-31T23:37:05.972 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.972 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.972 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.972 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.972 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.972 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster": { 2024-03-31T23:37:05.972 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster", 2024-03-31T23:37:05.972 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.972 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.972 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.972 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-31T23:37:05.972 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.973 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.973 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.973 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.973 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.973 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.973 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.973 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.973 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster_level": { 2024-03-31T23:37:05.973 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster_level", 2024-03-31T23:37:05.973 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.973 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.973 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.973 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "info", 2024-03-31T23:37:05.973 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.973 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.974 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-31T23:37:05.974 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-31T23:37:05.974 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-31T23:37:05.974 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-31T23:37:05.974 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-31T23:37:05.974 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-31T23:37:05.974 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-31T23:37:05.974 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-31T23:37:05.974 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.974 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.974 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.974 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.974 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.974 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_file": { 2024-03-31T23:37:05.975 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_file", 2024-03-31T23:37:05.975 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.975 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.975 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.975 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-31T23:37:05.975 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.975 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.975 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.975 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.975 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.975 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.975 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.975 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-31T23:37:05.975 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-31T23:37:05.975 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.976 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-03-31T23:37:05.976 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "localpool", 2024-03-31T23:37:05.976 INFO:teuthology.orchestra.run.smithi046.stdout: "can_run": true, 2024-03-31T23:37:05.976 INFO:teuthology.orchestra.run.smithi046.stdout: "error_string": "", 2024-03-31T23:37:05.976 INFO:teuthology.orchestra.run.smithi046.stdout: "module_options": { 2024-03-31T23:37:05.976 INFO:teuthology.orchestra.run.smithi046.stdout: "failure_domain": { 2024-03-31T23:37:05.976 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "failure_domain", 2024-03-31T23:37:05.976 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.976 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.976 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.976 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "host", 2024-03-31T23:37:05.976 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.976 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.976 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.977 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "failure domain for any created local pool", 2024-03-31T23:37:05.977 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-03-31T23:37:05.977 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.977 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.977 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.977 INFO:teuthology.orchestra.run.smithi046.stdout: "log_level": { 2024-03-31T23:37:05.977 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_level", 2024-03-31T23:37:05.977 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.977 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.977 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.977 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-31T23:37:05.977 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.977 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.978 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-31T23:37:05.978 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-31T23:37:05.978 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-31T23:37:05.978 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-31T23:37:05.978 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-31T23:37:05.978 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-31T23:37:05.978 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-31T23:37:05.978 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-31T23:37:05.978 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.978 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.978 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.978 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.978 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.978 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster": { 2024-03-31T23:37:05.979 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster", 2024-03-31T23:37:05.979 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.979 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.979 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.979 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-31T23:37:05.979 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.979 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.979 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.979 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.979 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.979 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.979 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.979 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.979 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster_level": { 2024-03-31T23:37:05.979 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster_level", 2024-03-31T23:37:05.980 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.980 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.980 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.980 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "info", 2024-03-31T23:37:05.980 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.980 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.980 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-31T23:37:05.980 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-31T23:37:05.980 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-31T23:37:05.980 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-31T23:37:05.980 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-31T23:37:05.980 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-31T23:37:05.980 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-31T23:37:05.980 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-31T23:37:05.981 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.981 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.981 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.981 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.981 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.981 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_file": { 2024-03-31T23:37:05.981 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_file", 2024-03-31T23:37:05.981 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.981 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.981 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.981 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-31T23:37:05.981 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.981 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.981 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.982 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.982 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.982 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.982 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.982 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.982 INFO:teuthology.orchestra.run.smithi046.stdout: "min_size": { 2024-03-31T23:37:05.982 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "min_size", 2024-03-31T23:37:05.982 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "int", 2024-03-31T23:37:05.982 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.982 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.982 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-31T23:37:05.982 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.982 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.982 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.983 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "default min_size for any created local pool", 2024-03-31T23:37:05.983 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-03-31T23:37:05.983 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.983 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.983 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.983 INFO:teuthology.orchestra.run.smithi046.stdout: "num_rep": { 2024-03-31T23:37:05.983 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "num_rep", 2024-03-31T23:37:05.983 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "int", 2024-03-31T23:37:05.983 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.983 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.983 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "3", 2024-03-31T23:37:05.983 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.983 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.983 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.983 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "default replica count for any created local pool", 2024-03-31T23:37:05.984 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.984 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.984 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.984 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.984 INFO:teuthology.orchestra.run.smithi046.stdout: "pg_num": { 2024-03-31T23:37:05.984 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "pg_num", 2024-03-31T23:37:05.984 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "int", 2024-03-31T23:37:05.984 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.984 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.984 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "128", 2024-03-31T23:37:05.984 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.984 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.984 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.984 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "default pg_num for any created local pool", 2024-03-31T23:37:05.985 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.985 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.985 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.985 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.985 INFO:teuthology.orchestra.run.smithi046.stdout: "prefix": { 2024-03-31T23:37:05.985 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "prefix", 2024-03-31T23:37:05.985 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.985 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.985 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.985 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-31T23:37:05.985 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.985 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.985 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.985 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "name prefix for any created local pool", 2024-03-31T23:37:05.985 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.986 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.986 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.986 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.986 INFO:teuthology.orchestra.run.smithi046.stdout: "subtree": { 2024-03-31T23:37:05.986 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "subtree", 2024-03-31T23:37:05.986 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.986 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.986 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.986 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "rack", 2024-03-31T23:37:05.986 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.986 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.986 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.986 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "CRUSH level for which to create a local pool", 2024-03-31T23:37:05.986 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-03-31T23:37:05.987 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.987 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.987 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-31T23:37:05.987 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-31T23:37:05.987 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.987 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-03-31T23:37:05.987 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "mds_autoscaler", 2024-03-31T23:37:05.987 INFO:teuthology.orchestra.run.smithi046.stdout: "can_run": true, 2024-03-31T23:37:05.987 INFO:teuthology.orchestra.run.smithi046.stdout: "error_string": "", 2024-03-31T23:37:05.987 INFO:teuthology.orchestra.run.smithi046.stdout: "module_options": { 2024-03-31T23:37:05.987 INFO:teuthology.orchestra.run.smithi046.stdout: "log_level": { 2024-03-31T23:37:05.987 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_level", 2024-03-31T23:37:05.987 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.987 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.988 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.988 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-31T23:37:05.988 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.988 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.988 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-31T23:37:05.988 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-31T23:37:05.988 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-31T23:37:05.988 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-31T23:37:05.988 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-31T23:37:05.988 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-31T23:37:05.988 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-31T23:37:05.988 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-31T23:37:05.988 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.988 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.988 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.989 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.989 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.989 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster": { 2024-03-31T23:37:05.989 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster", 2024-03-31T23:37:05.989 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.989 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.989 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.989 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-31T23:37:05.989 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.989 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.989 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.989 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.989 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.989 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.989 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.990 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.990 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster_level": { 2024-03-31T23:37:05.990 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster_level", 2024-03-31T23:37:05.990 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.990 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.990 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.990 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "info", 2024-03-31T23:37:05.990 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.990 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.990 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-31T23:37:05.990 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-31T23:37:05.990 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-31T23:37:05.990 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-31T23:37:05.990 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-31T23:37:05.991 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-31T23:37:05.991 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-31T23:37:05.991 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-31T23:37:05.991 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.991 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.991 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.991 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.991 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.991 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_file": { 2024-03-31T23:37:05.991 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_file", 2024-03-31T23:37:05.991 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.991 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.991 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.991 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-31T23:37:05.991 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.992 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.992 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.992 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.992 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.992 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.992 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.992 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-31T23:37:05.992 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-31T23:37:05.992 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.992 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-03-31T23:37:05.992 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "mirroring", 2024-03-31T23:37:05.992 INFO:teuthology.orchestra.run.smithi046.stdout: "can_run": true, 2024-03-31T23:37:05.992 INFO:teuthology.orchestra.run.smithi046.stdout: "error_string": "", 2024-03-31T23:37:05.992 INFO:teuthology.orchestra.run.smithi046.stdout: "module_options": { 2024-03-31T23:37:05.993 INFO:teuthology.orchestra.run.smithi046.stdout: "log_level": { 2024-03-31T23:37:05.993 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_level", 2024-03-31T23:37:05.993 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.993 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.993 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.993 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-31T23:37:05.993 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.993 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.993 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-31T23:37:05.993 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-31T23:37:05.993 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-31T23:37:05.993 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-31T23:37:05.993 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-31T23:37:05.993 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-31T23:37:05.993 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-31T23:37:05.994 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-31T23:37:05.994 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.994 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.994 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.994 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.994 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.994 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster": { 2024-03-31T23:37:05.994 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster", 2024-03-31T23:37:05.994 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.994 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.994 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.994 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-31T23:37:05.994 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.994 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.994 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.995 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.995 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.995 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.995 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.995 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.995 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster_level": { 2024-03-31T23:37:05.995 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster_level", 2024-03-31T23:37:05.995 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.995 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.995 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.995 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "info", 2024-03-31T23:37:05.995 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.995 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.995 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-31T23:37:05.996 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-31T23:37:05.996 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-31T23:37:05.996 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-31T23:37:05.996 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-31T23:37:05.996 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-31T23:37:05.996 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-31T23:37:05.996 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-31T23:37:05.996 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.996 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.996 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.996 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.996 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.996 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_file": { 2024-03-31T23:37:05.996 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_file", 2024-03-31T23:37:05.996 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.997 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.997 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.997 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-31T23:37:05.997 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.997 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.997 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:05.997 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.997 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.997 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.997 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.997 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-31T23:37:05.997 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-31T23:37:05.997 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.997 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-03-31T23:37:05.998 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "nfs", 2024-03-31T23:37:05.998 INFO:teuthology.orchestra.run.smithi046.stdout: "can_run": true, 2024-03-31T23:37:05.998 INFO:teuthology.orchestra.run.smithi046.stdout: "error_string": "", 2024-03-31T23:37:05.998 INFO:teuthology.orchestra.run.smithi046.stdout: "module_options": { 2024-03-31T23:37:05.998 INFO:teuthology.orchestra.run.smithi046.stdout: "log_level": { 2024-03-31T23:37:05.998 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_level", 2024-03-31T23:37:05.998 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:05.998 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.998 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:05.998 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-31T23:37:05.998 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:05.998 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:05.998 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-31T23:37:05.998 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-31T23:37:05.998 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-31T23:37:05.999 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-31T23:37:05.999 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-31T23:37:05.999 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-31T23:37:05.999 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-31T23:37:05.999 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-31T23:37:05.999 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:05.999 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:05.999 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:05.999 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:05.999 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:05.999 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster": { 2024-03-31T23:37:05.999 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster", 2024-03-31T23:37:05.999 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:05.999 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:05.999 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:06.000 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-31T23:37:06.000 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:06.000 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:06.000 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:06.000 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:06.000 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:06.000 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:06.000 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:06.000 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:06.000 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster_level": { 2024-03-31T23:37:06.000 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster_level", 2024-03-31T23:37:06.000 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:06.000 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:06.000 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:06.001 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "info", 2024-03-31T23:37:06.001 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:06.001 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:06.001 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-31T23:37:06.001 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-31T23:37:06.001 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-31T23:37:06.001 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-31T23:37:06.001 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-31T23:37:06.001 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-31T23:37:06.001 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-31T23:37:06.001 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-31T23:37:06.001 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:06.001 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:06.001 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:06.002 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:06.002 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:06.002 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_file": { 2024-03-31T23:37:06.002 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_file", 2024-03-31T23:37:06.002 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:06.002 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:06.002 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:06.002 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-31T23:37:06.002 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:06.002 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:06.002 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:06.002 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:06.002 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:06.002 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:06.003 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:06.003 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-31T23:37:06.003 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-31T23:37:06.003 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:06.003 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-03-31T23:37:06.003 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "orchestrator", 2024-03-31T23:37:06.003 INFO:teuthology.orchestra.run.smithi046.stdout: "can_run": true, 2024-03-31T23:37:06.003 INFO:teuthology.orchestra.run.smithi046.stdout: "error_string": "", 2024-03-31T23:37:06.003 INFO:teuthology.orchestra.run.smithi046.stdout: "module_options": { 2024-03-31T23:37:06.003 INFO:teuthology.orchestra.run.smithi046.stdout: "fail_fs": { 2024-03-31T23:37:06.003 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "fail_fs", 2024-03-31T23:37:06.003 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:06.003 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:06.003 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:06.003 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-31T23:37:06.003 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:06.004 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:06.004 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:06.004 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-03-31T23:37:06.004 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:06.004 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:06.004 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:06.004 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:06.004 INFO:teuthology.orchestra.run.smithi046.stdout: "log_level": { 2024-03-31T23:37:06.004 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_level", 2024-03-31T23:37:06.004 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:06.004 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:06.004 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:06.004 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-31T23:37:06.004 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:06.005 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:06.005 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-31T23:37:06.005 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-31T23:37:06.005 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-31T23:37:06.005 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-31T23:37:06.005 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-31T23:37:06.005 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-31T23:37:06.005 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-31T23:37:06.005 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-31T23:37:06.005 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:06.005 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:06.005 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:06.005 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:06.005 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:06.006 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster": { 2024-03-31T23:37:06.006 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster", 2024-03-31T23:37:06.006 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:06.006 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:06.006 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:06.006 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-31T23:37:06.006 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:06.006 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:06.006 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:06.006 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:06.006 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:06.006 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:06.006 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:06.006 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:06.007 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster_level": { 2024-03-31T23:37:06.007 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster_level", 2024-03-31T23:37:06.007 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:06.007 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:06.007 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:06.007 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "info", 2024-03-31T23:37:06.007 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:06.007 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:06.007 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-31T23:37:06.007 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-31T23:37:06.007 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-31T23:37:06.007 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-31T23:37:06.007 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-31T23:37:06.007 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-31T23:37:06.008 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-31T23:37:06.008 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-31T23:37:06.008 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:06.008 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:06.008 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:06.008 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:06.008 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:06.008 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_file": { 2024-03-31T23:37:06.008 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_file", 2024-03-31T23:37:06.008 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:06.008 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:06.008 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:06.008 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-31T23:37:06.008 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:06.008 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:06.009 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:06.009 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:06.009 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:06.009 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:06.009 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:06.009 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:06.009 INFO:teuthology.orchestra.run.smithi046.stdout: "orchestrator": { 2024-03-31T23:37:06.009 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "orchestrator", 2024-03-31T23:37:06.009 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:06.009 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:06.009 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:06.009 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-31T23:37:06.009 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:06.010 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:06.010 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-31T23:37:06.010 INFO:teuthology.orchestra.run.smithi046.stdout: "cephadm", 2024-03-31T23:37:06.010 INFO:teuthology.orchestra.run.smithi046.stdout: "rook", 2024-03-31T23:37:06.010 INFO:teuthology.orchestra.run.smithi046.stdout: "test_orchestrator" 2024-03-31T23:37:06.010 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-31T23:37:06.010 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Orchestrator backend", 2024-03-31T23:37:06.010 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:06.010 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:06.010 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:06.010 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-31T23:37:06.010 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-31T23:37:06.010 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:06.010 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-03-31T23:37:06.010 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "osd_perf_query", 2024-03-31T23:37:06.011 INFO:teuthology.orchestra.run.smithi046.stdout: "can_run": true, 2024-03-31T23:37:06.011 INFO:teuthology.orchestra.run.smithi046.stdout: "error_string": "", 2024-03-31T23:37:06.011 INFO:teuthology.orchestra.run.smithi046.stdout: "module_options": { 2024-03-31T23:37:06.011 INFO:teuthology.orchestra.run.smithi046.stdout: "log_level": { 2024-03-31T23:37:06.011 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_level", 2024-03-31T23:37:06.011 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:06.011 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:06.011 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:06.011 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-31T23:37:06.011 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:06.011 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:06.011 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-31T23:37:06.011 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-31T23:37:06.011 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-31T23:37:06.012 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-31T23:37:06.012 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-31T23:37:06.012 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-31T23:37:06.012 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-31T23:37:06.012 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-31T23:37:06.012 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:06.012 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:06.012 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:06.012 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:06.012 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:06.012 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster": { 2024-03-31T23:37:06.012 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster", 2024-03-31T23:37:06.012 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:06.012 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:06.013 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:06.013 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-31T23:37:06.013 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:06.013 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:06.013 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:06.013 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:06.013 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:06.013 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:06.013 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:06.013 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:06.013 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster_level": { 2024-03-31T23:37:06.013 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster_level", 2024-03-31T23:37:06.013 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:06.014 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:06.014 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:06.014 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "info", 2024-03-31T23:37:06.014 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:06.014 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:06.014 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-31T23:37:06.014 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-31T23:37:06.014 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-31T23:37:06.014 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-31T23:37:06.014 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-31T23:37:06.014 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-31T23:37:06.014 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-31T23:37:06.014 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-31T23:37:06.015 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:06.015 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:06.015 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:06.015 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:06.015 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:06.015 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_file": { 2024-03-31T23:37:06.015 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_file", 2024-03-31T23:37:06.015 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:06.015 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:06.015 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:06.015 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-31T23:37:06.015 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:06.015 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:06.015 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:06.016 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:06.016 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:06.016 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:06.016 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:06.016 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-31T23:37:06.016 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-31T23:37:06.016 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:06.016 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-03-31T23:37:06.016 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "osd_support", 2024-03-31T23:37:06.016 INFO:teuthology.orchestra.run.smithi046.stdout: "can_run": true, 2024-03-31T23:37:06.016 INFO:teuthology.orchestra.run.smithi046.stdout: "error_string": "", 2024-03-31T23:37:06.016 INFO:teuthology.orchestra.run.smithi046.stdout: "module_options": { 2024-03-31T23:37:06.016 INFO:teuthology.orchestra.run.smithi046.stdout: "log_level": { 2024-03-31T23:37:06.016 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_level", 2024-03-31T23:37:06.016 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:06.017 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:06.017 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:06.017 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-31T23:37:06.017 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:06.017 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:06.017 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-31T23:37:06.017 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-31T23:37:06.017 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-31T23:37:06.017 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-31T23:37:06.017 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-31T23:37:06.017 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-31T23:37:06.017 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-31T23:37:06.017 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-31T23:37:06.017 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:06.018 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:06.018 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:06.018 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:06.018 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:06.018 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster": { 2024-03-31T23:37:06.018 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster", 2024-03-31T23:37:06.018 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:06.018 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:06.018 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:06.018 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-31T23:37:06.018 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:06.018 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:06.018 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:06.018 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:06.018 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:06.019 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:06.019 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:06.019 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:06.019 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster_level": { 2024-03-31T23:37:06.019 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster_level", 2024-03-31T23:37:06.019 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:06.019 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:06.019 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:06.019 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "info", 2024-03-31T23:37:06.019 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:06.019 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:06.019 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-31T23:37:06.019 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-31T23:37:06.019 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-31T23:37:06.019 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-31T23:37:06.020 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-31T23:37:06.020 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-31T23:37:06.020 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-31T23:37:06.020 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-31T23:37:06.020 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:06.020 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:06.020 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:06.020 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:06.020 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:06.020 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_file": { 2024-03-31T23:37:06.020 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_file", 2024-03-31T23:37:06.020 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:06.020 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:06.020 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:06.021 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-31T23:37:06.021 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:06.021 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:06.021 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:06.021 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:06.021 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:06.021 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:06.021 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:06.021 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-31T23:37:06.021 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-31T23:37:06.021 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:06.021 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-03-31T23:37:06.021 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "pg_autoscaler", 2024-03-31T23:37:06.021 INFO:teuthology.orchestra.run.smithi046.stdout: "can_run": true, 2024-03-31T23:37:06.021 INFO:teuthology.orchestra.run.smithi046.stdout: "error_string": "", 2024-03-31T23:37:06.022 INFO:teuthology.orchestra.run.smithi046.stdout: "module_options": { 2024-03-31T23:37:06.022 INFO:teuthology.orchestra.run.smithi046.stdout: "log_level": { 2024-03-31T23:37:06.022 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_level", 2024-03-31T23:37:06.022 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:06.022 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:06.022 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:06.022 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-31T23:37:06.022 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:06.022 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:06.022 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-31T23:37:06.022 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-31T23:37:06.022 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-31T23:37:06.022 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-31T23:37:06.022 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-31T23:37:06.023 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-31T23:37:06.023 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-31T23:37:06.023 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-31T23:37:06.023 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:06.023 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:06.023 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:06.023 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:06.023 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:06.023 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster": { 2024-03-31T23:37:06.023 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster", 2024-03-31T23:37:06.023 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:06.023 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:06.023 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:06.023 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-31T23:37:06.024 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:06.024 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:06.024 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:06.024 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:06.024 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:06.024 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:06.024 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:06.024 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:06.024 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster_level": { 2024-03-31T23:37:06.024 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster_level", 2024-03-31T23:37:06.024 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:06.024 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:06.024 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:06.024 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "info", 2024-03-31T23:37:06.025 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:06.025 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:06.025 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-31T23:37:06.025 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-31T23:37:06.025 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-31T23:37:06.025 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-31T23:37:06.025 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-31T23:37:06.025 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-31T23:37:06.025 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-31T23:37:06.025 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-31T23:37:06.025 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:06.025 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:06.025 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:06.025 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:06.025 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:06.026 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_file": { 2024-03-31T23:37:06.026 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_file", 2024-03-31T23:37:06.026 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:06.026 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:06.026 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:06.026 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-31T23:37:06.026 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:06.026 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:06.026 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:06.026 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:06.026 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:06.026 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:06.026 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:06.026 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:06.027 INFO:teuthology.orchestra.run.smithi046.stdout: "sleep_interval": { 2024-03-31T23:37:06.027 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "sleep_interval", 2024-03-31T23:37:06.027 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "secs", 2024-03-31T23:37:06.027 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:06.027 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:06.027 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "60", 2024-03-31T23:37:06.027 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:06.027 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:06.027 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:06.027 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:06.027 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:06.027 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:06.027 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:06.027 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:06.028 INFO:teuthology.orchestra.run.smithi046.stdout: "threshold": { 2024-03-31T23:37:06.028 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "threshold", 2024-03-31T23:37:06.028 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "float", 2024-03-31T23:37:06.028 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:06.028 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:06.028 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "3.0", 2024-03-31T23:37:06.028 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "1.0", 2024-03-31T23:37:06.028 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:06.028 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:06.028 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "scaling threshold", 2024-03-31T23:37:06.028 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-03-31T23:37:06.028 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:06.028 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:06.028 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-31T23:37:06.028 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-31T23:37:06.029 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:06.029 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-03-31T23:37:06.029 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "progress", 2024-03-31T23:37:06.029 INFO:teuthology.orchestra.run.smithi046.stdout: "can_run": true, 2024-03-31T23:37:06.029 INFO:teuthology.orchestra.run.smithi046.stdout: "error_string": "", 2024-03-31T23:37:06.029 INFO:teuthology.orchestra.run.smithi046.stdout: "module_options": { 2024-03-31T23:37:06.029 INFO:teuthology.orchestra.run.smithi046.stdout: "allow_pg_recovery_event": { 2024-03-31T23:37:06.029 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "allow_pg_recovery_event", 2024-03-31T23:37:06.029 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:06.029 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:06.029 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:06.029 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-31T23:37:06.029 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:06.029 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:06.029 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:06.030 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "allow the module to show pg recovery progress", 2024-03-31T23:37:06.030 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:06.030 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:06.030 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:06.030 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:06.030 INFO:teuthology.orchestra.run.smithi046.stdout: "enabled": { 2024-03-31T23:37:06.030 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "enabled", 2024-03-31T23:37:06.030 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:06.030 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:06.030 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:06.030 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "True", 2024-03-31T23:37:06.030 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:06.030 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:06.030 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:06.031 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:06.031 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:06.031 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:06.031 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:06.031 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:06.031 INFO:teuthology.orchestra.run.smithi046.stdout: "log_level": { 2024-03-31T23:37:06.031 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_level", 2024-03-31T23:37:06.031 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:06.031 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:06.031 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:06.031 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-31T23:37:06.031 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:06.031 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:06.031 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-31T23:37:06.031 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-31T23:37:06.032 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-31T23:37:06.032 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-31T23:37:06.032 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-31T23:37:06.032 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-31T23:37:06.032 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-31T23:37:06.032 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-31T23:37:06.032 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:06.032 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:06.032 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:06.032 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:06.032 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:06.032 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster": { 2024-03-31T23:37:06.032 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster", 2024-03-31T23:37:06.032 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:06.032 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:06.032 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:06.033 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-31T23:37:06.033 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:06.033 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:06.033 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:06.033 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:06.033 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:06.033 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:06.033 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:06.033 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:06.033 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster_level": { 2024-03-31T23:37:06.033 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster_level", 2024-03-31T23:37:06.033 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:06.033 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:06.033 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:06.033 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "info", 2024-03-31T23:37:06.034 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:06.034 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:06.034 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-31T23:37:06.034 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-31T23:37:06.034 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-31T23:37:06.034 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-31T23:37:06.034 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-31T23:37:06.034 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-31T23:37:06.034 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-31T23:37:06.034 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-31T23:37:06.034 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:06.034 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:06.034 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:06.034 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:06.035 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:06.035 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_file": { 2024-03-31T23:37:06.035 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_file", 2024-03-31T23:37:06.035 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:06.035 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:06.035 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:06.035 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-31T23:37:06.035 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:06.035 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:06.035 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:06.035 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:06.035 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:06.035 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:06.035 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:06.036 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:06.036 INFO:teuthology.orchestra.run.smithi046.stdout: "max_completed_events": { 2024-03-31T23:37:06.036 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "max_completed_events", 2024-03-31T23:37:06.036 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "int", 2024-03-31T23:37:06.036 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:06.036 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:06.036 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "50", 2024-03-31T23:37:06.036 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:06.036 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:06.036 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:06.036 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "number of past completed events to remember", 2024-03-31T23:37:06.036 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:06.036 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:06.036 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:06.037 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:06.037 INFO:teuthology.orchestra.run.smithi046.stdout: "sleep_interval": { 2024-03-31T23:37:06.037 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "sleep_interval", 2024-03-31T23:37:06.037 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "secs", 2024-03-31T23:37:06.037 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:06.037 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:06.037 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "5", 2024-03-31T23:37:06.037 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:06.037 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:06.037 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:06.037 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "how long the module is going to sleep", 2024-03-31T23:37:06.037 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:06.037 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:06.037 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:06.038 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-31T23:37:06.038 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-31T23:37:06.038 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:06.038 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-03-31T23:37:06.038 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "prometheus", 2024-03-31T23:37:06.038 INFO:teuthology.orchestra.run.smithi046.stdout: "can_run": true, 2024-03-31T23:37:06.038 INFO:teuthology.orchestra.run.smithi046.stdout: "error_string": "", 2024-03-31T23:37:06.038 INFO:teuthology.orchestra.run.smithi046.stdout: "module_options": { 2024-03-31T23:37:06.038 INFO:teuthology.orchestra.run.smithi046.stdout: "cache": { 2024-03-31T23:37:06.038 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "cache", 2024-03-31T23:37:06.038 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:06.038 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:06.038 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:06.038 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "True", 2024-03-31T23:37:06.038 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:06.039 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:06.039 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:06.039 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:06.039 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:06.039 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:06.039 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:06.039 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:06.039 INFO:teuthology.orchestra.run.smithi046.stdout: "exclude_perf_counters": { 2024-03-31T23:37:06.039 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "exclude_perf_counters", 2024-03-31T23:37:06.039 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:06.039 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:06.039 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:06.039 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "True", 2024-03-31T23:37:06.039 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:06.040 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:06.040 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:06.040 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-03-31T23:37:06.040 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-03-31T23:37:06.040 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:06.040 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:06.040 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:06.040 INFO:teuthology.orchestra.run.smithi046.stdout: "log_level": { 2024-03-31T23:37:06.040 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_level", 2024-03-31T23:37:06.040 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:06.040 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:06.040 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:06.040 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-31T23:37:06.040 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:06.041 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:06.041 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-31T23:37:06.041 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-31T23:37:06.041 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-31T23:37:06.041 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-31T23:37:06.041 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-31T23:37:06.041 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-31T23:37:06.041 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-31T23:37:06.041 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-31T23:37:06.041 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:06.041 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:06.041 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:06.041 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:06.041 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:06.041 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster": { 2024-03-31T23:37:06.042 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster", 2024-03-31T23:37:06.042 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:06.042 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:06.042 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:06.042 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-31T23:37:06.042 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:06.042 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:06.042 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:06.042 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:06.042 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:06.042 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:06.042 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:06.042 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:06.042 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster_level": { 2024-03-31T23:37:06.043 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster_level", 2024-03-31T23:37:06.043 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:06.043 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:06.043 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:06.043 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "info", 2024-03-31T23:37:06.043 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:06.043 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:06.043 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-31T23:37:06.043 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-31T23:37:06.043 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-31T23:37:06.043 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-31T23:37:06.043 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-31T23:37:06.043 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-31T23:37:06.043 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-31T23:37:06.044 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-31T23:37:06.044 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:06.044 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:06.044 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:06.044 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:06.044 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:06.044 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_file": { 2024-03-31T23:37:06.044 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_file", 2024-03-31T23:37:06.044 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:06.044 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:06.044 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:06.044 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-31T23:37:06.044 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:06.044 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:06.044 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:06.045 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:06.045 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:06.045 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:06.045 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:06.045 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:06.045 INFO:teuthology.orchestra.run.smithi046.stdout: "rbd_stats_pools": { 2024-03-31T23:37:06.045 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "rbd_stats_pools", 2024-03-31T23:37:06.045 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:06.045 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:06.045 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:06.045 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-31T23:37:06.045 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:06.045 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:06.045 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:06.046 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:06.046 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:06.046 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:06.046 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:06.046 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:06.046 INFO:teuthology.orchestra.run.smithi046.stdout: "rbd_stats_pools_refresh_interval": { 2024-03-31T23:37:06.046 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-03-31T23:37:06.046 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "int", 2024-03-31T23:37:06.046 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:06.046 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:06.046 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "300", 2024-03-31T23:37:06.046 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:06.046 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:06.046 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:06.047 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:06.047 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:06.047 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:06.047 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:06.047 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:06.047 INFO:teuthology.orchestra.run.smithi046.stdout: "scrape_interval": { 2024-03-31T23:37:06.047 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "scrape_interval", 2024-03-31T23:37:06.047 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "float", 2024-03-31T23:37:06.047 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:06.047 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:06.047 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "15.0", 2024-03-31T23:37:06.047 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:06.047 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:06.047 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:06.047 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:06.048 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:06.048 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:06.048 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:06.048 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:06.048 INFO:teuthology.orchestra.run.smithi046.stdout: "server_addr": { 2024-03-31T23:37:06.048 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "server_addr", 2024-03-31T23:37:06.048 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:06.048 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:06.048 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:06.048 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "::", 2024-03-31T23:37:06.048 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:06.048 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:06.048 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:06.048 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-03-31T23:37:06.049 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:06.049 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:06.049 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:06.049 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:06.049 INFO:teuthology.orchestra.run.smithi046.stdout: "server_port": { 2024-03-31T23:37:06.049 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "server_port", 2024-03-31T23:37:06.049 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "int", 2024-03-31T23:37:06.049 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:06.049 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:06.049 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "9283", 2024-03-31T23:37:06.049 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:06.049 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:06.049 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:06.049 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-03-31T23:37:06.050 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:06.050 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:06.050 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:06.050 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:06.050 INFO:teuthology.orchestra.run.smithi046.stdout: "stale_cache_strategy": { 2024-03-31T23:37:06.050 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "stale_cache_strategy", 2024-03-31T23:37:06.050 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:06.050 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:06.050 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:06.050 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "log", 2024-03-31T23:37:06.050 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:06.050 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:06.050 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:06.050 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:06.050 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:06.051 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:06.051 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:06.051 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:06.051 INFO:teuthology.orchestra.run.smithi046.stdout: "standby_behaviour": { 2024-03-31T23:37:06.051 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "standby_behaviour", 2024-03-31T23:37:06.051 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:06.051 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:06.051 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:06.051 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "default", 2024-03-31T23:37:06.051 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:06.051 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:06.051 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-31T23:37:06.051 INFO:teuthology.orchestra.run.smithi046.stdout: "default", 2024-03-31T23:37:06.051 INFO:teuthology.orchestra.run.smithi046.stdout: "error" 2024-03-31T23:37:06.051 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-31T23:37:06.052 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:06.052 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:06.052 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:06.052 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:06.052 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:06.052 INFO:teuthology.orchestra.run.smithi046.stdout: "standby_error_status_code": { 2024-03-31T23:37:06.052 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "standby_error_status_code", 2024-03-31T23:37:06.052 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "int", 2024-03-31T23:37:06.052 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:06.052 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:06.052 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "500", 2024-03-31T23:37:06.052 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "400", 2024-03-31T23:37:06.052 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "599", 2024-03-31T23:37:06.052 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:06.053 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:06.053 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:06.053 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:06.053 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:06.053 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-31T23:37:06.053 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-31T23:37:06.053 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:06.053 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-03-31T23:37:06.053 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "rbd_support", 2024-03-31T23:37:06.053 INFO:teuthology.orchestra.run.smithi046.stdout: "can_run": true, 2024-03-31T23:37:06.053 INFO:teuthology.orchestra.run.smithi046.stdout: "error_string": "", 2024-03-31T23:37:06.053 INFO:teuthology.orchestra.run.smithi046.stdout: "module_options": { 2024-03-31T23:37:06.053 INFO:teuthology.orchestra.run.smithi046.stdout: "log_level": { 2024-03-31T23:37:06.053 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_level", 2024-03-31T23:37:06.053 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:06.054 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:06.054 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:06.054 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-31T23:37:06.054 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:06.054 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:06.054 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-31T23:37:06.054 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-31T23:37:06.054 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-31T23:37:06.054 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-31T23:37:06.054 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-31T23:37:06.054 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-31T23:37:06.054 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-31T23:37:06.054 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-31T23:37:06.054 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:06.055 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:06.055 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:06.055 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:06.055 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:06.055 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster": { 2024-03-31T23:37:06.055 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster", 2024-03-31T23:37:06.055 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:06.055 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:06.055 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:06.055 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-31T23:37:06.055 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:06.055 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:06.055 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:06.055 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:06.056 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:06.056 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:06.056 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:06.056 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:06.056 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster_level": { 2024-03-31T23:37:06.056 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster_level", 2024-03-31T23:37:06.056 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:06.056 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:06.056 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:06.056 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "info", 2024-03-31T23:37:06.056 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:06.056 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:06.056 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-31T23:37:06.056 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-31T23:37:06.057 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-31T23:37:06.057 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-31T23:37:06.057 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-31T23:37:06.057 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-31T23:37:06.057 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-31T23:37:06.057 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-31T23:37:06.057 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:06.057 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:06.057 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:06.057 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:06.057 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:06.057 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_file": { 2024-03-31T23:37:06.057 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_file", 2024-03-31T23:37:06.057 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:06.057 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:06.058 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:06.058 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-31T23:37:06.058 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:06.058 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:06.058 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:06.058 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:06.058 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:06.058 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:06.058 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:06.058 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:06.058 INFO:teuthology.orchestra.run.smithi046.stdout: "max_concurrent_snap_create": { 2024-03-31T23:37:06.058 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "max_concurrent_snap_create", 2024-03-31T23:37:06.058 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "int", 2024-03-31T23:37:06.059 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:06.059 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:06.059 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "10", 2024-03-31T23:37:06.059 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:06.059 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:06.059 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:06.059 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:06.059 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:06.059 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:06.059 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:06.059 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:06.059 INFO:teuthology.orchestra.run.smithi046.stdout: "mirror_snapshot_schedule": { 2024-03-31T23:37:06.059 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "mirror_snapshot_schedule", 2024-03-31T23:37:06.059 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:06.060 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:06.060 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:06.060 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-31T23:37:06.060 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:06.060 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:06.060 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:06.060 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:06.060 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:06.060 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:06.060 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:06.060 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:06.060 INFO:teuthology.orchestra.run.smithi046.stdout: "trash_purge_schedule": { 2024-03-31T23:37:06.060 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "trash_purge_schedule", 2024-03-31T23:37:06.060 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:06.061 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:06.061 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:06.061 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-31T23:37:06.061 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:06.061 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:06.061 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:06.061 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:06.061 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:06.061 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:06.061 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:06.061 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-31T23:37:06.061 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-31T23:37:06.061 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:06.061 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-03-31T23:37:06.061 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "restful", 2024-03-31T23:37:06.062 INFO:teuthology.orchestra.run.smithi046.stdout: "can_run": true, 2024-03-31T23:37:06.062 INFO:teuthology.orchestra.run.smithi046.stdout: "error_string": "", 2024-03-31T23:37:06.062 INFO:teuthology.orchestra.run.smithi046.stdout: "module_options": { 2024-03-31T23:37:06.062 INFO:teuthology.orchestra.run.smithi046.stdout: "enable_auth": { 2024-03-31T23:37:06.062 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "enable_auth", 2024-03-31T23:37:06.062 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:06.062 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:06.062 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:06.062 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "True", 2024-03-31T23:37:06.062 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:06.062 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:06.062 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:06.062 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:06.062 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:06.063 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:06.063 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:06.063 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:06.063 INFO:teuthology.orchestra.run.smithi046.stdout: "key_file": { 2024-03-31T23:37:06.063 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "key_file", 2024-03-31T23:37:06.063 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:06.063 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:06.063 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:06.063 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-31T23:37:06.063 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:06.063 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:06.063 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:06.063 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:06.063 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:06.064 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:06.064 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:06.064 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:06.064 INFO:teuthology.orchestra.run.smithi046.stdout: "log_level": { 2024-03-31T23:37:06.064 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_level", 2024-03-31T23:37:06.064 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:06.064 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:06.064 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:06.064 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-31T23:37:06.064 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:06.064 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:06.064 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-31T23:37:06.064 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-31T23:37:06.064 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-31T23:37:06.065 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-31T23:37:06.065 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-31T23:37:06.065 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-31T23:37:06.065 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-31T23:37:06.065 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-31T23:37:06.065 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:06.065 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:06.065 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:06.065 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:06.065 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:06.065 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster": { 2024-03-31T23:37:06.065 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster", 2024-03-31T23:37:06.065 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:06.065 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:06.066 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:06.066 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-31T23:37:06.066 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:06.066 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:06.066 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:06.066 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:06.066 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:06.066 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:06.066 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:06.066 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:06.066 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster_level": { 2024-03-31T23:37:06.066 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster_level", 2024-03-31T23:37:06.066 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:06.066 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:06.067 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:06.067 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "info", 2024-03-31T23:37:06.067 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:06.067 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:06.067 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-31T23:37:06.067 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-31T23:37:06.067 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-31T23:37:06.067 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-31T23:37:06.067 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-31T23:37:06.067 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-31T23:37:06.067 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-31T23:37:06.067 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-31T23:37:06.067 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:06.067 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:06.068 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:06.068 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:06.068 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:06.068 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_file": { 2024-03-31T23:37:06.068 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_file", 2024-03-31T23:37:06.068 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:06.068 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:06.068 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:06.068 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-31T23:37:06.068 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:06.068 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:06.068 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:06.068 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:06.068 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:06.068 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:06.069 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:06.069 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:06.069 INFO:teuthology.orchestra.run.smithi046.stdout: "server_addr": { 2024-03-31T23:37:06.069 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "server_addr", 2024-03-31T23:37:06.069 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:06.069 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:06.069 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:06.069 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-31T23:37:06.069 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:06.069 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:06.069 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:06.069 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:06.069 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:06.069 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:06.070 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:06.070 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:06.070 INFO:teuthology.orchestra.run.smithi046.stdout: "server_port": { 2024-03-31T23:37:06.070 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "server_port", 2024-03-31T23:37:06.070 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:06.070 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:06.070 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:06.070 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-31T23:37:06.070 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:06.070 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:06.070 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:06.070 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:06.070 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:06.070 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:06.070 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:06.071 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-31T23:37:06.071 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-31T23:37:06.071 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:06.071 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-03-31T23:37:06.071 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "rgw", 2024-03-31T23:37:06.071 INFO:teuthology.orchestra.run.smithi046.stdout: "can_run": true, 2024-03-31T23:37:06.071 INFO:teuthology.orchestra.run.smithi046.stdout: "error_string": "", 2024-03-31T23:37:06.071 INFO:teuthology.orchestra.run.smithi046.stdout: "module_options": { 2024-03-31T23:37:06.071 INFO:teuthology.orchestra.run.smithi046.stdout: "log_level": { 2024-03-31T23:37:06.071 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_level", 2024-03-31T23:37:06.071 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:06.071 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:06.071 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:06.071 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-31T23:37:06.072 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:06.072 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:06.072 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-31T23:37:06.072 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-31T23:37:06.072 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-31T23:37:06.072 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-31T23:37:06.072 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-31T23:37:06.072 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-31T23:37:06.072 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-31T23:37:06.072 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-31T23:37:06.072 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:06.072 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:06.072 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:06.072 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:06.072 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:06.073 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster": { 2024-03-31T23:37:06.073 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster", 2024-03-31T23:37:06.073 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:06.073 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:06.073 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:06.073 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-31T23:37:06.073 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:06.073 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:06.073 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:06.073 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:06.073 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:06.073 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:06.073 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:06.073 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:06.073 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster_level": { 2024-03-31T23:37:06.074 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster_level", 2024-03-31T23:37:06.074 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:06.074 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:06.074 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:06.074 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "info", 2024-03-31T23:37:06.074 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:06.074 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:06.074 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-31T23:37:06.074 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-31T23:37:06.074 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-31T23:37:06.074 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-31T23:37:06.074 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-31T23:37:06.074 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-31T23:37:06.074 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-31T23:37:06.075 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-31T23:37:06.075 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:06.075 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:06.075 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:06.075 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:06.075 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:06.075 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_file": { 2024-03-31T23:37:06.075 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_file", 2024-03-31T23:37:06.075 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:06.075 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:06.075 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:06.075 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-31T23:37:06.075 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:06.076 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:06.076 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:06.076 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:06.076 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:06.076 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:06.076 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:06.076 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-31T23:37:06.076 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-31T23:37:06.076 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:06.076 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-03-31T23:37:06.076 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "rook", 2024-03-31T23:37:06.076 INFO:teuthology.orchestra.run.smithi046.stdout: "can_run": true, 2024-03-31T23:37:06.076 INFO:teuthology.orchestra.run.smithi046.stdout: "error_string": "", 2024-03-31T23:37:06.076 INFO:teuthology.orchestra.run.smithi046.stdout: "module_options": { 2024-03-31T23:37:06.077 INFO:teuthology.orchestra.run.smithi046.stdout: "log_level": { 2024-03-31T23:37:06.077 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_level", 2024-03-31T23:37:06.077 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:06.077 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:06.077 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:06.077 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-31T23:37:06.077 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:06.077 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:06.077 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-31T23:37:06.077 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-31T23:37:06.077 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-31T23:37:06.077 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-31T23:37:06.077 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-31T23:37:06.078 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-31T23:37:06.078 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-31T23:37:06.078 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-31T23:37:06.078 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:06.078 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:06.078 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:06.078 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:06.078 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:06.078 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster": { 2024-03-31T23:37:06.078 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster", 2024-03-31T23:37:06.078 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:06.078 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:06.078 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:06.079 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-31T23:37:06.079 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:06.079 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:06.079 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:06.079 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:06.079 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:06.079 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:06.079 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:06.079 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:06.079 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster_level": { 2024-03-31T23:37:06.079 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster_level", 2024-03-31T23:37:06.079 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:06.079 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:06.079 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:06.079 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "info", 2024-03-31T23:37:06.080 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:06.080 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:06.080 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-31T23:37:06.080 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-31T23:37:06.080 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-31T23:37:06.080 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-31T23:37:06.080 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-31T23:37:06.080 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-31T23:37:06.080 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-31T23:37:06.080 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-31T23:37:06.080 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:06.080 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:06.080 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:06.081 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:06.081 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:06.081 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_file": { 2024-03-31T23:37:06.081 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_file", 2024-03-31T23:37:06.081 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:06.081 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:06.081 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:06.081 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-31T23:37:06.081 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:06.081 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:06.081 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:06.081 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:06.081 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:06.081 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:06.082 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:06.082 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:06.082 INFO:teuthology.orchestra.run.smithi046.stdout: "storage_class": { 2024-03-31T23:37:06.082 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "storage_class", 2024-03-31T23:37:06.082 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:06.082 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:06.082 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:06.082 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "local", 2024-03-31T23:37:06.082 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:06.082 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:06.082 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:06.082 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-03-31T23:37:06.082 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:06.082 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:06.083 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:06.083 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-31T23:37:06.083 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-31T23:37:06.083 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:06.083 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-03-31T23:37:06.083 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "selftest", 2024-03-31T23:37:06.083 INFO:teuthology.orchestra.run.smithi046.stdout: "can_run": true, 2024-03-31T23:37:06.083 INFO:teuthology.orchestra.run.smithi046.stdout: "error_string": "", 2024-03-31T23:37:06.083 INFO:teuthology.orchestra.run.smithi046.stdout: "module_options": { 2024-03-31T23:37:06.083 INFO:teuthology.orchestra.run.smithi046.stdout: "log_level": { 2024-03-31T23:37:06.083 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_level", 2024-03-31T23:37:06.083 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:06.083 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:06.083 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:06.083 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-31T23:37:06.084 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:06.084 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:06.084 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-31T23:37:06.084 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-31T23:37:06.084 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-31T23:37:06.084 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-31T23:37:06.084 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-31T23:37:06.084 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-31T23:37:06.084 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-31T23:37:06.084 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-31T23:37:06.084 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:06.084 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:06.084 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:06.084 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:06.084 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:06.085 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster": { 2024-03-31T23:37:06.085 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster", 2024-03-31T23:37:06.085 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:06.085 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:06.085 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:06.085 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-31T23:37:06.085 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:06.085 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:06.085 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:06.085 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:06.085 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:06.085 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:06.085 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:06.085 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:06.086 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster_level": { 2024-03-31T23:37:06.086 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster_level", 2024-03-31T23:37:06.086 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:06.086 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:06.086 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:06.086 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "info", 2024-03-31T23:37:06.086 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:06.086 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:06.086 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-31T23:37:06.086 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-31T23:37:06.086 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-31T23:37:06.086 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-31T23:37:06.086 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-31T23:37:06.086 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-31T23:37:06.086 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-31T23:37:06.087 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-31T23:37:06.087 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:06.087 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:06.087 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:06.087 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:06.087 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:06.087 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_file": { 2024-03-31T23:37:06.087 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_file", 2024-03-31T23:37:06.087 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:06.087 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:06.087 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:06.087 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-31T23:37:06.087 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:06.087 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:06.088 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:06.088 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:06.088 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:06.088 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:06.088 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:06.088 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:06.088 INFO:teuthology.orchestra.run.smithi046.stdout: "roption1": { 2024-03-31T23:37:06.088 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "roption1", 2024-03-31T23:37:06.088 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:06.088 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:06.088 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:06.088 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-31T23:37:06.088 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:06.088 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:06.088 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:06.089 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:06.089 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:06.089 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:06.089 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:06.089 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:06.089 INFO:teuthology.orchestra.run.smithi046.stdout: "roption2": { 2024-03-31T23:37:06.089 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "roption2", 2024-03-31T23:37:06.089 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:06.089 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:06.089 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:06.089 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "xyz", 2024-03-31T23:37:06.089 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:06.089 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:06.089 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:06.089 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:06.090 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:06.090 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:06.090 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:06.090 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:06.090 INFO:teuthology.orchestra.run.smithi046.stdout: "rwoption1": { 2024-03-31T23:37:06.090 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "rwoption1", 2024-03-31T23:37:06.090 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:06.090 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:06.090 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:06.090 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-31T23:37:06.090 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:06.090 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:06.090 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:06.090 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:06.090 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:06.091 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:06.091 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:06.091 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:06.091 INFO:teuthology.orchestra.run.smithi046.stdout: "rwoption2": { 2024-03-31T23:37:06.091 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "rwoption2", 2024-03-31T23:37:06.091 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "int", 2024-03-31T23:37:06.091 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:06.091 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:06.091 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-31T23:37:06.091 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:06.091 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:06.091 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:06.091 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:06.091 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:06.092 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:06.092 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:06.092 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:06.092 INFO:teuthology.orchestra.run.smithi046.stdout: "rwoption3": { 2024-03-31T23:37:06.092 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "rwoption3", 2024-03-31T23:37:06.092 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "float", 2024-03-31T23:37:06.092 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:06.092 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:06.092 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-31T23:37:06.092 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:06.092 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:06.092 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:06.092 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:06.092 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:06.093 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:06.093 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:06.093 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:06.093 INFO:teuthology.orchestra.run.smithi046.stdout: "rwoption4": { 2024-03-31T23:37:06.093 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "rwoption4", 2024-03-31T23:37:06.093 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:06.093 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:06.093 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:06.093 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-31T23:37:06.093 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:06.093 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:06.093 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:06.093 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:06.093 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:06.093 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:06.094 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:06.094 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:06.094 INFO:teuthology.orchestra.run.smithi046.stdout: "rwoption5": { 2024-03-31T23:37:06.094 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "rwoption5", 2024-03-31T23:37:06.094 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:06.094 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:06.094 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:06.094 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-31T23:37:06.094 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:06.094 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:06.094 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:06.094 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:06.094 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:06.094 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:06.095 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:06.095 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:06.095 INFO:teuthology.orchestra.run.smithi046.stdout: "rwoption6": { 2024-03-31T23:37:06.095 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "rwoption6", 2024-03-31T23:37:06.095 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:06.095 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:06.095 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:06.095 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "True", 2024-03-31T23:37:06.095 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:06.095 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:06.095 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:06.095 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:06.095 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:06.095 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:06.095 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:06.096 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:06.096 INFO:teuthology.orchestra.run.smithi046.stdout: "rwoption7": { 2024-03-31T23:37:06.096 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "rwoption7", 2024-03-31T23:37:06.096 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "int", 2024-03-31T23:37:06.096 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:06.096 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:06.096 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-31T23:37:06.096 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "1", 2024-03-31T23:37:06.096 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "42", 2024-03-31T23:37:06.096 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:06.096 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:06.096 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:06.096 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:06.096 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:06.097 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:06.097 INFO:teuthology.orchestra.run.smithi046.stdout: "testkey": { 2024-03-31T23:37:06.097 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "testkey", 2024-03-31T23:37:06.097 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:06.097 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:06.097 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:06.097 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-31T23:37:06.097 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:06.097 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:06.097 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:06.097 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:06.097 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:06.097 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:06.097 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:06.097 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:06.098 INFO:teuthology.orchestra.run.smithi046.stdout: "testlkey": { 2024-03-31T23:37:06.098 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "testlkey", 2024-03-31T23:37:06.098 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:06.098 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:06.098 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:06.098 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-31T23:37:06.098 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:06.098 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:06.098 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:06.098 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:06.098 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:06.098 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:06.098 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:06.098 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:06.099 INFO:teuthology.orchestra.run.smithi046.stdout: "testnewline": { 2024-03-31T23:37:06.099 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "testnewline", 2024-03-31T23:37:06.099 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:06.099 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:06.099 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:06.099 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-31T23:37:06.099 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:06.099 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:06.099 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:06.099 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:06.099 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:06.099 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:06.099 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:06.099 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-31T23:37:06.100 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-31T23:37:06.100 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:06.100 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-03-31T23:37:06.100 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "snap_schedule", 2024-03-31T23:37:06.100 INFO:teuthology.orchestra.run.smithi046.stdout: "can_run": true, 2024-03-31T23:37:06.100 INFO:teuthology.orchestra.run.smithi046.stdout: "error_string": "", 2024-03-31T23:37:06.100 INFO:teuthology.orchestra.run.smithi046.stdout: "module_options": { 2024-03-31T23:37:06.100 INFO:teuthology.orchestra.run.smithi046.stdout: "allow_m_granularity": { 2024-03-31T23:37:06.100 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "allow_m_granularity", 2024-03-31T23:37:06.100 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:06.100 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:06.100 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:06.100 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-31T23:37:06.100 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:06.101 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:06.101 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:06.101 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "allow minute scheduled snapshots", 2024-03-31T23:37:06.101 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:06.101 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:06.101 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:06.101 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:06.101 INFO:teuthology.orchestra.run.smithi046.stdout: "dump_on_update": { 2024-03-31T23:37:06.101 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "dump_on_update", 2024-03-31T23:37:06.101 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:06.101 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:06.101 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:06.101 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-31T23:37:06.101 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:06.101 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:06.102 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:06.102 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "dump database to debug log on update", 2024-03-31T23:37:06.102 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:06.102 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:06.102 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:06.102 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:06.102 INFO:teuthology.orchestra.run.smithi046.stdout: "log_level": { 2024-03-31T23:37:06.102 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_level", 2024-03-31T23:37:06.102 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:06.102 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:06.102 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:06.102 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-31T23:37:06.102 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:06.102 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:06.103 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-31T23:37:06.103 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-31T23:37:06.103 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-31T23:37:06.103 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-31T23:37:06.103 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-31T23:37:06.103 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-31T23:37:06.103 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-31T23:37:06.103 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-31T23:37:06.103 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:06.103 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:06.103 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:06.103 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:06.103 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:06.103 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster": { 2024-03-31T23:37:06.103 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster", 2024-03-31T23:37:06.104 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:06.104 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:06.104 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:06.104 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-31T23:37:06.104 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:06.104 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:06.104 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:06.104 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:06.104 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:06.104 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:06.104 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:06.104 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:06.104 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster_level": { 2024-03-31T23:37:06.105 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster_level", 2024-03-31T23:37:06.105 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:06.105 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:06.105 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:06.105 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "info", 2024-03-31T23:37:06.105 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:06.105 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:06.105 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-31T23:37:06.105 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-31T23:37:06.105 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-31T23:37:06.105 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-31T23:37:06.105 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-31T23:37:06.105 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-31T23:37:06.105 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-31T23:37:06.106 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-31T23:37:06.106 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:06.106 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:06.106 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:06.106 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:06.106 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:06.106 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_file": { 2024-03-31T23:37:06.106 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_file", 2024-03-31T23:37:06.106 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:06.106 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:06.106 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:06.106 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-31T23:37:06.106 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:06.106 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:06.107 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:06.107 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:06.107 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:06.107 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:06.107 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:06.107 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-31T23:37:06.107 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-31T23:37:06.107 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:06.107 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-03-31T23:37:06.107 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "stats", 2024-03-31T23:37:06.107 INFO:teuthology.orchestra.run.smithi046.stdout: "can_run": true, 2024-03-31T23:37:06.107 INFO:teuthology.orchestra.run.smithi046.stdout: "error_string": "", 2024-03-31T23:37:06.107 INFO:teuthology.orchestra.run.smithi046.stdout: "module_options": { 2024-03-31T23:37:06.107 INFO:teuthology.orchestra.run.smithi046.stdout: "log_level": { 2024-03-31T23:37:06.107 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_level", 2024-03-31T23:37:06.108 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:06.108 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:06.108 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:06.108 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-31T23:37:06.108 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:06.108 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:06.108 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-31T23:37:06.108 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-31T23:37:06.108 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-31T23:37:06.108 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-31T23:37:06.108 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-31T23:37:06.108 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-31T23:37:06.108 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-31T23:37:06.108 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-31T23:37:06.109 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:06.109 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:06.109 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:06.109 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:06.109 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:06.109 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster": { 2024-03-31T23:37:06.109 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster", 2024-03-31T23:37:06.109 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:06.109 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:06.109 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:06.109 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-31T23:37:06.109 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:06.109 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:06.109 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:06.110 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:06.110 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:06.110 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:06.110 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:06.110 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:06.110 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster_level": { 2024-03-31T23:37:06.110 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster_level", 2024-03-31T23:37:06.110 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:06.110 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:06.110 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:06.110 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "info", 2024-03-31T23:37:06.110 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:06.110 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:06.110 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-31T23:37:06.111 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-31T23:37:06.111 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-31T23:37:06.111 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-31T23:37:06.111 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-31T23:37:06.111 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-31T23:37:06.111 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-31T23:37:06.111 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-31T23:37:06.111 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:06.111 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:06.111 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:06.111 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:06.111 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:06.111 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_file": { 2024-03-31T23:37:06.111 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_file", 2024-03-31T23:37:06.112 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:06.112 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:06.112 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:06.112 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-31T23:37:06.112 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:06.112 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:06.112 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:06.112 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:06.112 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:06.112 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:06.112 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:06.112 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-31T23:37:06.112 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-31T23:37:06.112 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:06.112 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-03-31T23:37:06.113 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "status", 2024-03-31T23:37:06.113 INFO:teuthology.orchestra.run.smithi046.stdout: "can_run": true, 2024-03-31T23:37:06.113 INFO:teuthology.orchestra.run.smithi046.stdout: "error_string": "", 2024-03-31T23:37:06.113 INFO:teuthology.orchestra.run.smithi046.stdout: "module_options": { 2024-03-31T23:37:06.113 INFO:teuthology.orchestra.run.smithi046.stdout: "log_level": { 2024-03-31T23:37:06.113 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_level", 2024-03-31T23:37:06.113 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:06.113 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:06.113 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:06.113 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-31T23:37:06.113 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:06.113 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:06.113 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-31T23:37:06.113 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-31T23:37:06.114 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-31T23:37:06.114 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-31T23:37:06.114 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-31T23:37:06.114 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-31T23:37:06.114 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-31T23:37:06.114 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-31T23:37:06.114 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:06.114 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:06.114 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:06.114 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:06.114 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:06.114 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster": { 2024-03-31T23:37:06.114 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster", 2024-03-31T23:37:06.114 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:06.115 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:06.115 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:06.115 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-31T23:37:06.115 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:06.115 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:06.115 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:06.115 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:06.115 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:06.115 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:06.115 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:06.115 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:06.115 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster_level": { 2024-03-31T23:37:06.115 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster_level", 2024-03-31T23:37:06.115 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:06.116 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:06.116 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:06.116 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "info", 2024-03-31T23:37:06.116 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:06.116 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:06.116 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-31T23:37:06.116 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-31T23:37:06.116 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-31T23:37:06.116 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-31T23:37:06.116 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-31T23:37:06.116 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-31T23:37:06.116 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-31T23:37:06.116 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-31T23:37:06.116 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:06.117 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:06.117 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:06.117 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:06.117 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:06.117 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_file": { 2024-03-31T23:37:06.117 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_file", 2024-03-31T23:37:06.117 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:06.117 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:06.117 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:06.117 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-31T23:37:06.117 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:06.117 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:06.117 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:06.117 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:06.118 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:06.118 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:06.118 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:06.118 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-31T23:37:06.118 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-31T23:37:06.118 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:06.118 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-03-31T23:37:06.118 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "telegraf", 2024-03-31T23:37:06.118 INFO:teuthology.orchestra.run.smithi046.stdout: "can_run": true, 2024-03-31T23:37:06.118 INFO:teuthology.orchestra.run.smithi046.stdout: "error_string": "", 2024-03-31T23:37:06.118 INFO:teuthology.orchestra.run.smithi046.stdout: "module_options": { 2024-03-31T23:37:06.118 INFO:teuthology.orchestra.run.smithi046.stdout: "address": { 2024-03-31T23:37:06.118 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "address", 2024-03-31T23:37:06.118 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:06.119 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:06.119 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:06.119 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-03-31T23:37:06.119 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:06.119 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:06.119 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:06.119 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:06.119 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:06.119 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:06.119 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:06.119 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:06.119 INFO:teuthology.orchestra.run.smithi046.stdout: "interval": { 2024-03-31T23:37:06.119 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "interval", 2024-03-31T23:37:06.119 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "secs", 2024-03-31T23:37:06.119 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:06.120 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:06.120 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "15", 2024-03-31T23:37:06.120 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:06.120 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:06.120 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:06.120 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:06.120 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:06.120 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:06.120 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:06.120 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:06.120 INFO:teuthology.orchestra.run.smithi046.stdout: "log_level": { 2024-03-31T23:37:06.120 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_level", 2024-03-31T23:37:06.120 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:06.120 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:06.120 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:06.121 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-31T23:37:06.121 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:06.121 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:06.121 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-31T23:37:06.121 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-31T23:37:06.121 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-31T23:37:06.121 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-31T23:37:06.121 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-31T23:37:06.121 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-31T23:37:06.121 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-31T23:37:06.121 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-31T23:37:06.121 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:06.121 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:06.121 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:06.121 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:06.122 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:06.122 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster": { 2024-03-31T23:37:06.122 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster", 2024-03-31T23:37:06.122 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:06.122 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:06.122 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:06.122 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-31T23:37:06.122 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:06.122 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:06.122 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:06.122 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:06.122 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:06.122 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:06.122 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:06.123 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:06.123 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster_level": { 2024-03-31T23:37:06.123 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster_level", 2024-03-31T23:37:06.123 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:06.123 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:06.123 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:06.123 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "info", 2024-03-31T23:37:06.123 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:06.123 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:06.123 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-31T23:37:06.123 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-31T23:37:06.123 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-31T23:37:06.123 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-31T23:37:06.123 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-31T23:37:06.123 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-31T23:37:06.124 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-31T23:37:06.124 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-31T23:37:06.124 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:06.124 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:06.124 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:06.124 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:06.124 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:06.124 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_file": { 2024-03-31T23:37:06.124 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_file", 2024-03-31T23:37:06.124 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:06.124 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:06.124 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:06.124 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-31T23:37:06.124 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:06.124 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:06.124 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:06.125 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:06.125 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:06.125 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:06.125 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:06.125 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-31T23:37:06.125 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-31T23:37:06.125 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:06.125 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-03-31T23:37:06.125 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "telemetry", 2024-03-31T23:37:06.125 INFO:teuthology.orchestra.run.smithi046.stdout: "can_run": true, 2024-03-31T23:37:06.125 INFO:teuthology.orchestra.run.smithi046.stdout: "error_string": "", 2024-03-31T23:37:06.125 INFO:teuthology.orchestra.run.smithi046.stdout: "module_options": { 2024-03-31T23:37:06.125 INFO:teuthology.orchestra.run.smithi046.stdout: "channel_basic": { 2024-03-31T23:37:06.125 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "channel_basic", 2024-03-31T23:37:06.126 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:06.126 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:06.126 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:06.126 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "True", 2024-03-31T23:37:06.126 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:06.126 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:06.126 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:06.126 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Share basic cluster information (size, version)", 2024-03-31T23:37:06.126 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:06.126 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:06.126 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:06.126 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:06.126 INFO:teuthology.orchestra.run.smithi046.stdout: "channel_crash": { 2024-03-31T23:37:06.126 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "channel_crash", 2024-03-31T23:37:06.127 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:06.127 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:06.127 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:06.127 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "True", 2024-03-31T23:37:06.127 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:06.127 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:06.127 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:06.127 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-03-31T23:37:06.127 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:06.127 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:06.127 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:06.127 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:06.127 INFO:teuthology.orchestra.run.smithi046.stdout: "channel_device": { 2024-03-31T23:37:06.128 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "channel_device", 2024-03-31T23:37:06.128 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:06.128 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:06.128 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:06.128 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "True", 2024-03-31T23:37:06.128 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:06.128 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:06.128 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:06.128 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-03-31T23:37:06.128 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:06.128 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:06.128 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:06.128 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:06.128 INFO:teuthology.orchestra.run.smithi046.stdout: "channel_ident": { 2024-03-31T23:37:06.129 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "channel_ident", 2024-03-31T23:37:06.129 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:06.129 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:06.129 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:06.129 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-31T23:37:06.129 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:06.129 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:06.129 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:06.129 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-03-31T23:37:06.129 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:06.129 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:06.129 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:06.129 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:06.129 INFO:teuthology.orchestra.run.smithi046.stdout: "channel_perf": { 2024-03-31T23:37:06.129 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "channel_perf", 2024-03-31T23:37:06.130 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:06.130 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:06.130 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:06.130 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-31T23:37:06.130 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:06.130 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:06.130 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:06.130 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Share various performance metrics of a cluster", 2024-03-31T23:37:06.130 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:06.130 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:06.130 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:06.130 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:06.130 INFO:teuthology.orchestra.run.smithi046.stdout: "contact": { 2024-03-31T23:37:06.130 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "contact", 2024-03-31T23:37:06.131 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:06.131 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:06.131 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:06.131 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-31T23:37:06.131 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:06.131 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:06.131 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:06.131 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:06.131 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:06.131 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:06.131 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:06.131 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:06.131 INFO:teuthology.orchestra.run.smithi046.stdout: "description": { 2024-03-31T23:37:06.131 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "description", 2024-03-31T23:37:06.132 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:06.132 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:06.132 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:06.132 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-31T23:37:06.132 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:06.132 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:06.132 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:06.132 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:06.132 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:06.132 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:06.132 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:06.132 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:06.132 INFO:teuthology.orchestra.run.smithi046.stdout: "device_url": { 2024-03-31T23:37:06.132 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "device_url", 2024-03-31T23:37:06.133 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:06.133 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:06.133 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:06.133 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-03-31T23:37:06.133 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:06.133 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:06.133 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:06.133 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:06.133 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:06.133 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:06.133 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:06.133 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:06.133 INFO:teuthology.orchestra.run.smithi046.stdout: "enabled": { 2024-03-31T23:37:06.133 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "enabled", 2024-03-31T23:37:06.133 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:06.134 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:06.134 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:06.134 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-31T23:37:06.134 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:06.134 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:06.134 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:06.134 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:06.134 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:06.134 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:06.134 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:06.134 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:06.134 INFO:teuthology.orchestra.run.smithi046.stdout: "interval": { 2024-03-31T23:37:06.134 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "interval", 2024-03-31T23:37:06.134 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "int", 2024-03-31T23:37:06.134 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:06.135 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:06.135 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "24", 2024-03-31T23:37:06.135 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "8", 2024-03-31T23:37:06.135 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:06.135 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:06.135 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:06.135 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:06.135 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:06.135 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:06.135 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:06.135 INFO:teuthology.orchestra.run.smithi046.stdout: "last_opt_revision": { 2024-03-31T23:37:06.135 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "last_opt_revision", 2024-03-31T23:37:06.135 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "int", 2024-03-31T23:37:06.135 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:06.135 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:06.136 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "1", 2024-03-31T23:37:06.136 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:06.136 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:06.136 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:06.136 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:06.136 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:06.136 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:06.136 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:06.136 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:06.136 INFO:teuthology.orchestra.run.smithi046.stdout: "leaderboard": { 2024-03-31T23:37:06.136 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "leaderboard", 2024-03-31T23:37:06.136 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:06.136 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:06.136 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:06.136 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-31T23:37:06.137 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:06.137 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:06.137 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:06.137 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:06.137 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:06.137 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:06.137 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:06.137 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:06.137 INFO:teuthology.orchestra.run.smithi046.stdout: "leaderboard_description": { 2024-03-31T23:37:06.137 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "leaderboard_description", 2024-03-31T23:37:06.137 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:06.137 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:06.137 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:06.137 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-31T23:37:06.138 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:06.138 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:06.138 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:06.138 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:06.138 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:06.138 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:06.138 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:06.138 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:06.138 INFO:teuthology.orchestra.run.smithi046.stdout: "log_level": { 2024-03-31T23:37:06.138 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_level", 2024-03-31T23:37:06.138 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:06.138 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:06.138 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:06.138 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-31T23:37:06.139 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:06.139 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:06.139 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-31T23:37:06.139 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-31T23:37:06.139 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-31T23:37:06.139 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-31T23:37:06.139 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-31T23:37:06.139 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-31T23:37:06.139 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-31T23:37:06.139 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-31T23:37:06.139 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:06.139 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:06.139 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:06.139 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:06.139 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:06.140 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster": { 2024-03-31T23:37:06.140 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster", 2024-03-31T23:37:06.140 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:06.140 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:06.140 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:06.140 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-31T23:37:06.140 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:06.140 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:06.140 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:06.140 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:06.140 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:06.140 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:06.140 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:06.140 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:06.140 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster_level": { 2024-03-31T23:37:06.141 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster_level", 2024-03-31T23:37:06.141 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:06.141 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:06.141 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:06.141 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "info", 2024-03-31T23:37:06.141 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:06.141 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:06.141 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-31T23:37:06.141 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-31T23:37:06.141 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-31T23:37:06.141 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-31T23:37:06.141 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-31T23:37:06.141 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-31T23:37:06.141 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-31T23:37:06.142 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-31T23:37:06.142 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:06.142 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:06.142 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:06.142 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:06.142 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:06.142 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_file": { 2024-03-31T23:37:06.142 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_file", 2024-03-31T23:37:06.142 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:06.142 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:06.142 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:06.142 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-31T23:37:06.142 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:06.142 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:06.142 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:06.143 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:06.143 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:06.143 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:06.143 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:06.143 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:06.143 INFO:teuthology.orchestra.run.smithi046.stdout: "organization": { 2024-03-31T23:37:06.143 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "organization", 2024-03-31T23:37:06.143 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:06.143 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:06.143 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:06.143 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-31T23:37:06.143 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:06.143 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:06.143 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:06.144 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:06.144 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:06.144 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:06.144 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:06.144 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:06.144 INFO:teuthology.orchestra.run.smithi046.stdout: "proxy": { 2024-03-31T23:37:06.144 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "proxy", 2024-03-31T23:37:06.144 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:06.144 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:06.144 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:06.144 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-31T23:37:06.144 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:06.144 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:06.144 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:06.144 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:06.145 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:06.145 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:06.145 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:06.145 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:06.145 INFO:teuthology.orchestra.run.smithi046.stdout: "url": { 2024-03-31T23:37:06.145 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "url", 2024-03-31T23:37:06.145 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:06.145 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:06.145 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:06.145 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-03-31T23:37:06.145 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:06.145 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:06.145 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:06.145 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:06.146 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:06.146 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:06.146 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:06.146 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-31T23:37:06.146 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-31T23:37:06.146 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:06.146 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-03-31T23:37:06.146 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "test_orchestrator", 2024-03-31T23:37:06.146 INFO:teuthology.orchestra.run.smithi046.stdout: "can_run": true, 2024-03-31T23:37:06.146 INFO:teuthology.orchestra.run.smithi046.stdout: "error_string": "", 2024-03-31T23:37:06.146 INFO:teuthology.orchestra.run.smithi046.stdout: "module_options": { 2024-03-31T23:37:06.146 INFO:teuthology.orchestra.run.smithi046.stdout: "log_level": { 2024-03-31T23:37:06.146 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_level", 2024-03-31T23:37:06.146 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:06.146 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:06.147 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:06.147 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-31T23:37:06.147 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:06.147 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:06.147 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-31T23:37:06.147 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-31T23:37:06.147 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-31T23:37:06.147 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-31T23:37:06.147 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-31T23:37:06.147 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-31T23:37:06.147 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-31T23:37:06.147 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-31T23:37:06.147 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:06.147 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:06.148 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:06.148 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:06.148 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:06.148 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster": { 2024-03-31T23:37:06.148 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster", 2024-03-31T23:37:06.148 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:06.148 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:06.148 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:06.148 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-31T23:37:06.148 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:06.148 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:06.148 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:06.148 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:06.148 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:06.148 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:06.149 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:06.149 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:06.149 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster_level": { 2024-03-31T23:37:06.149 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster_level", 2024-03-31T23:37:06.149 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:06.149 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:06.149 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:06.149 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "info", 2024-03-31T23:37:06.149 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:06.149 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:06.149 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-31T23:37:06.149 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-31T23:37:06.149 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-31T23:37:06.149 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-31T23:37:06.150 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-31T23:37:06.150 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-31T23:37:06.150 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-31T23:37:06.150 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-31T23:37:06.150 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:06.150 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:06.150 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:06.150 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:06.150 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:06.150 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_file": { 2024-03-31T23:37:06.150 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_file", 2024-03-31T23:37:06.150 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:06.150 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:06.150 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:06.150 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-31T23:37:06.151 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:06.151 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:06.151 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:06.151 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:06.151 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:06.151 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:06.151 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:06.151 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-31T23:37:06.151 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-31T23:37:06.151 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:06.151 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-03-31T23:37:06.151 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "volumes", 2024-03-31T23:37:06.151 INFO:teuthology.orchestra.run.smithi046.stdout: "can_run": true, 2024-03-31T23:37:06.151 INFO:teuthology.orchestra.run.smithi046.stdout: "error_string": "", 2024-03-31T23:37:06.151 INFO:teuthology.orchestra.run.smithi046.stdout: "module_options": { 2024-03-31T23:37:06.152 INFO:teuthology.orchestra.run.smithi046.stdout: "log_level": { 2024-03-31T23:37:06.152 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_level", 2024-03-31T23:37:06.152 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:06.152 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:06.152 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:06.152 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-31T23:37:06.152 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:06.152 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:06.152 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-31T23:37:06.152 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-31T23:37:06.152 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-31T23:37:06.152 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-31T23:37:06.152 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-31T23:37:06.152 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-31T23:37:06.152 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-31T23:37:06.153 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-31T23:37:06.153 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:06.153 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:06.153 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:06.153 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:06.153 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:06.153 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster": { 2024-03-31T23:37:06.153 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster", 2024-03-31T23:37:06.153 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:06.153 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:06.153 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:06.153 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-31T23:37:06.153 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:06.153 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:06.154 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:06.154 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:06.154 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:06.154 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:06.154 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:06.154 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:06.154 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster_level": { 2024-03-31T23:37:06.154 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster_level", 2024-03-31T23:37:06.154 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:06.154 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:06.154 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:06.154 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "info", 2024-03-31T23:37:06.154 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:06.154 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:06.155 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-31T23:37:06.155 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-31T23:37:06.155 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-31T23:37:06.155 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-31T23:37:06.155 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-31T23:37:06.155 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-31T23:37:06.155 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-31T23:37:06.155 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-31T23:37:06.155 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:06.155 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:06.155 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:06.155 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:06.155 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:06.155 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_file": { 2024-03-31T23:37:06.155 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_file", 2024-03-31T23:37:06.156 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:06.156 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:06.156 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:06.156 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-31T23:37:06.156 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:06.156 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:06.156 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:06.156 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:06.156 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:06.156 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:06.156 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:06.156 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:06.156 INFO:teuthology.orchestra.run.smithi046.stdout: "max_concurrent_clones": { 2024-03-31T23:37:06.156 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "max_concurrent_clones", 2024-03-31T23:37:06.156 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "int", 2024-03-31T23:37:06.157 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:06.157 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:06.157 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "4", 2024-03-31T23:37:06.157 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:06.157 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:06.157 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:06.157 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Number of asynchronous cloner threads", 2024-03-31T23:37:06.157 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:06.157 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:06.157 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:06.157 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:06.157 INFO:teuthology.orchestra.run.smithi046.stdout: "periodic_async_work": { 2024-03-31T23:37:06.157 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "periodic_async_work", 2024-03-31T23:37:06.157 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:06.158 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:06.158 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:06.158 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-31T23:37:06.158 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:06.158 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:06.158 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:06.158 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Periodically check for async work", 2024-03-31T23:37:06.158 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:06.158 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:06.158 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:06.158 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:06.158 INFO:teuthology.orchestra.run.smithi046.stdout: "snapshot_clone_delay": { 2024-03-31T23:37:06.158 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "snapshot_clone_delay", 2024-03-31T23:37:06.158 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "int", 2024-03-31T23:37:06.159 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:06.159 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:06.159 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "0", 2024-03-31T23:37:06.159 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:06.159 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:06.159 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:06.159 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-03-31T23:37:06.159 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:06.159 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:06.159 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:06.159 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:06.159 INFO:teuthology.orchestra.run.smithi046.stdout: "snapshot_clone_no_wait": { 2024-03-31T23:37:06.159 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "snapshot_clone_no_wait", 2024-03-31T23:37:06.159 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:06.159 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:06.160 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:06.160 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "True", 2024-03-31T23:37:06.160 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:06.160 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:06.160 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:06.160 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-03-31T23:37:06.160 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:06.160 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:06.160 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:06.160 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-31T23:37:06.160 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-31T23:37:06.160 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:06.160 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-03-31T23:37:06.160 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "zabbix", 2024-03-31T23:37:06.161 INFO:teuthology.orchestra.run.smithi046.stdout: "can_run": true, 2024-03-31T23:37:06.161 INFO:teuthology.orchestra.run.smithi046.stdout: "error_string": "", 2024-03-31T23:37:06.161 INFO:teuthology.orchestra.run.smithi046.stdout: "module_options": { 2024-03-31T23:37:06.161 INFO:teuthology.orchestra.run.smithi046.stdout: "discovery_interval": { 2024-03-31T23:37:06.161 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "discovery_interval", 2024-03-31T23:37:06.161 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "uint", 2024-03-31T23:37:06.161 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:06.161 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:06.161 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "100", 2024-03-31T23:37:06.161 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:06.161 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:06.161 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:06.161 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:06.161 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:06.162 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:06.162 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:06.162 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:06.162 INFO:teuthology.orchestra.run.smithi046.stdout: "identifier": { 2024-03-31T23:37:06.162 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "identifier", 2024-03-31T23:37:06.162 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:06.162 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:06.162 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:06.162 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-31T23:37:06.162 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:06.162 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:06.162 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:06.162 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:06.162 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:06.162 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:06.163 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:06.163 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:06.163 INFO:teuthology.orchestra.run.smithi046.stdout: "interval": { 2024-03-31T23:37:06.163 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "interval", 2024-03-31T23:37:06.163 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "secs", 2024-03-31T23:37:06.163 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:06.163 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:06.163 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "60", 2024-03-31T23:37:06.163 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:06.163 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:06.163 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:06.163 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:06.163 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:06.163 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:06.163 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:06.164 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:06.164 INFO:teuthology.orchestra.run.smithi046.stdout: "log_level": { 2024-03-31T23:37:06.164 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_level", 2024-03-31T23:37:06.164 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:06.164 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:06.164 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:06.164 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-31T23:37:06.164 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:06.164 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:06.164 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-31T23:37:06.164 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-31T23:37:06.164 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-31T23:37:06.164 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-31T23:37:06.164 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-31T23:37:06.165 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-31T23:37:06.165 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-31T23:37:06.165 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-31T23:37:06.165 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:06.165 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:06.165 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:06.165 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:06.165 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:06.165 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster": { 2024-03-31T23:37:06.165 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster", 2024-03-31T23:37:06.165 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:06.165 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:06.165 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:06.165 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-31T23:37:06.165 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:06.166 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:06.166 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:06.166 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:06.166 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:06.166 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:06.166 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:06.166 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:06.166 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster_level": { 2024-03-31T23:37:06.166 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster_level", 2024-03-31T23:37:06.166 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:06.166 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:06.166 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:06.166 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "info", 2024-03-31T23:37:06.167 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:06.167 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:06.167 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-31T23:37:06.167 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-31T23:37:06.167 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-31T23:37:06.167 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-31T23:37:06.167 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-31T23:37:06.167 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-31T23:37:06.167 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-31T23:37:06.167 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-31T23:37:06.167 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:06.167 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:06.167 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:06.167 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:06.167 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:06.168 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_file": { 2024-03-31T23:37:06.168 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_file", 2024-03-31T23:37:06.168 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-31T23:37:06.168 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:06.168 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-31T23:37:06.168 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-31T23:37:06.168 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:06.168 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:06.168 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:06.168 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:06.168 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:06.168 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:06.168 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:06.168 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:06.168 INFO:teuthology.orchestra.run.smithi046.stdout: "zabbix_host": { 2024-03-31T23:37:06.169 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "zabbix_host", 2024-03-31T23:37:06.169 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:06.169 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:06.169 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:06.169 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-31T23:37:06.169 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:06.169 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:06.169 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:06.169 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:06.169 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:06.169 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:06.169 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:06.169 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:06.169 INFO:teuthology.orchestra.run.smithi046.stdout: "zabbix_port": { 2024-03-31T23:37:06.169 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "zabbix_port", 2024-03-31T23:37:06.170 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "int", 2024-03-31T23:37:06.170 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:06.170 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:06.170 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "10051", 2024-03-31T23:37:06.170 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:06.170 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:06.170 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:06.170 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:06.170 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:06.170 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:06.170 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:06.170 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:06.170 INFO:teuthology.orchestra.run.smithi046.stdout: "zabbix_sender": { 2024-03-31T23:37:06.170 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "zabbix_sender", 2024-03-31T23:37:06.170 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-31T23:37:06.171 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-31T23:37:06.171 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-31T23:37:06.171 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-03-31T23:37:06.171 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-31T23:37:06.171 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-31T23:37:06.171 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-31T23:37:06.171 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-31T23:37:06.171 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-31T23:37:06.171 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-31T23:37:06.171 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-31T23:37:06.171 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-31T23:37:06.171 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-31T23:37:06.171 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-31T23:37:06.171 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-31T23:37:06.172 INFO:teuthology.orchestra.run.smithi046.stdout: "services": { 2024-03-31T23:37:06.172 INFO:teuthology.orchestra.run.smithi046.stdout: "dashboard": "https://172.21.15.46:8443/" 2024-03-31T23:37:06.172 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:06.172 INFO:teuthology.orchestra.run.smithi046.stdout: "always_on_modules": { 2024-03-31T23:37:06.172 INFO:teuthology.orchestra.run.smithi046.stdout: "octopus": [ 2024-03-31T23:37:06.172 INFO:teuthology.orchestra.run.smithi046.stdout: "balancer", 2024-03-31T23:37:06.172 INFO:teuthology.orchestra.run.smithi046.stdout: "crash", 2024-03-31T23:37:06.172 INFO:teuthology.orchestra.run.smithi046.stdout: "devicehealth", 2024-03-31T23:37:06.172 INFO:teuthology.orchestra.run.smithi046.stdout: "orchestrator", 2024-03-31T23:37:06.172 INFO:teuthology.orchestra.run.smithi046.stdout: "pg_autoscaler", 2024-03-31T23:37:06.172 INFO:teuthology.orchestra.run.smithi046.stdout: "progress", 2024-03-31T23:37:06.172 INFO:teuthology.orchestra.run.smithi046.stdout: "rbd_support", 2024-03-31T23:37:06.172 INFO:teuthology.orchestra.run.smithi046.stdout: "status", 2024-03-31T23:37:06.172 INFO:teuthology.orchestra.run.smithi046.stdout: "telemetry", 2024-03-31T23:37:06.172 INFO:teuthology.orchestra.run.smithi046.stdout: "volumes" 2024-03-31T23:37:06.172 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-31T23:37:06.173 INFO:teuthology.orchestra.run.smithi046.stdout: "pacific": [ 2024-03-31T23:37:06.173 INFO:teuthology.orchestra.run.smithi046.stdout: "balancer", 2024-03-31T23:37:06.173 INFO:teuthology.orchestra.run.smithi046.stdout: "crash", 2024-03-31T23:37:06.173 INFO:teuthology.orchestra.run.smithi046.stdout: "devicehealth", 2024-03-31T23:37:06.173 INFO:teuthology.orchestra.run.smithi046.stdout: "orchestrator", 2024-03-31T23:37:06.173 INFO:teuthology.orchestra.run.smithi046.stdout: "pg_autoscaler", 2024-03-31T23:37:06.173 INFO:teuthology.orchestra.run.smithi046.stdout: "progress", 2024-03-31T23:37:06.173 INFO:teuthology.orchestra.run.smithi046.stdout: "rbd_support", 2024-03-31T23:37:06.173 INFO:teuthology.orchestra.run.smithi046.stdout: "status", 2024-03-31T23:37:06.173 INFO:teuthology.orchestra.run.smithi046.stdout: "telemetry", 2024-03-31T23:37:06.173 INFO:teuthology.orchestra.run.smithi046.stdout: "volumes" 2024-03-31T23:37:06.173 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-31T23:37:06.173 INFO:teuthology.orchestra.run.smithi046.stdout: "quincy": [ 2024-03-31T23:37:06.173 INFO:teuthology.orchestra.run.smithi046.stdout: "balancer", 2024-03-31T23:37:06.173 INFO:teuthology.orchestra.run.smithi046.stdout: "crash", 2024-03-31T23:37:06.174 INFO:teuthology.orchestra.run.smithi046.stdout: "devicehealth", 2024-03-31T23:37:06.174 INFO:teuthology.orchestra.run.smithi046.stdout: "orchestrator", 2024-03-31T23:37:06.174 INFO:teuthology.orchestra.run.smithi046.stdout: "pg_autoscaler", 2024-03-31T23:37:06.174 INFO:teuthology.orchestra.run.smithi046.stdout: "progress", 2024-03-31T23:37:06.174 INFO:teuthology.orchestra.run.smithi046.stdout: "rbd_support", 2024-03-31T23:37:06.174 INFO:teuthology.orchestra.run.smithi046.stdout: "status", 2024-03-31T23:37:06.174 INFO:teuthology.orchestra.run.smithi046.stdout: "telemetry", 2024-03-31T23:37:06.174 INFO:teuthology.orchestra.run.smithi046.stdout: "volumes" 2024-03-31T23:37:06.174 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-31T23:37:06.174 INFO:teuthology.orchestra.run.smithi046.stdout: "reef": [ 2024-03-31T23:37:06.174 INFO:teuthology.orchestra.run.smithi046.stdout: "balancer", 2024-03-31T23:37:06.174 INFO:teuthology.orchestra.run.smithi046.stdout: "crash", 2024-03-31T23:37:06.174 INFO:teuthology.orchestra.run.smithi046.stdout: "devicehealth", 2024-03-31T23:37:06.174 INFO:teuthology.orchestra.run.smithi046.stdout: "orchestrator", 2024-03-31T23:37:06.175 INFO:teuthology.orchestra.run.smithi046.stdout: "pg_autoscaler", 2024-03-31T23:37:06.175 INFO:teuthology.orchestra.run.smithi046.stdout: "progress", 2024-03-31T23:37:06.175 INFO:teuthology.orchestra.run.smithi046.stdout: "rbd_support", 2024-03-31T23:37:06.175 INFO:teuthology.orchestra.run.smithi046.stdout: "status", 2024-03-31T23:37:06.175 INFO:teuthology.orchestra.run.smithi046.stdout: "telemetry", 2024-03-31T23:37:06.175 INFO:teuthology.orchestra.run.smithi046.stdout: "volumes" 2024-03-31T23:37:06.175 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-31T23:37:06.175 INFO:teuthology.orchestra.run.smithi046.stdout: "squid": [ 2024-03-31T23:37:06.175 INFO:teuthology.orchestra.run.smithi046.stdout: "balancer", 2024-03-31T23:37:06.175 INFO:teuthology.orchestra.run.smithi046.stdout: "crash", 2024-03-31T23:37:06.175 INFO:teuthology.orchestra.run.smithi046.stdout: "devicehealth", 2024-03-31T23:37:06.175 INFO:teuthology.orchestra.run.smithi046.stdout: "orchestrator", 2024-03-31T23:37:06.175 INFO:teuthology.orchestra.run.smithi046.stdout: "pg_autoscaler", 2024-03-31T23:37:06.175 INFO:teuthology.orchestra.run.smithi046.stdout: "progress", 2024-03-31T23:37:06.175 INFO:teuthology.orchestra.run.smithi046.stdout: "rbd_support", 2024-03-31T23:37:06.176 INFO:teuthology.orchestra.run.smithi046.stdout: "status", 2024-03-31T23:37:06.176 INFO:teuthology.orchestra.run.smithi046.stdout: "telemetry", 2024-03-31T23:37:06.176 INFO:teuthology.orchestra.run.smithi046.stdout: "volumes" 2024-03-31T23:37:06.176 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-03-31T23:37:06.176 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:06.176 INFO:teuthology.orchestra.run.smithi046.stdout: "last_failure_osd_epoch": 96, 2024-03-31T23:37:06.176 INFO:teuthology.orchestra.run.smithi046.stdout: "active_clients": [ 2024-03-31T23:37:06.176 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-03-31T23:37:06.176 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "devicehealth", 2024-03-31T23:37:06.176 INFO:teuthology.orchestra.run.smithi046.stdout: "addrvec": [ 2024-03-31T23:37:06.176 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-03-31T23:37:06.176 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v2", 2024-03-31T23:37:06.176 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:0", 2024-03-31T23:37:06.176 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 620718269 2024-03-31T23:37:06.176 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-31T23:37:06.177 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-03-31T23:37:06.177 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:06.177 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-03-31T23:37:06.177 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "libcephsqlite", 2024-03-31T23:37:06.177 INFO:teuthology.orchestra.run.smithi046.stdout: "addrvec": [ 2024-03-31T23:37:06.177 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-03-31T23:37:06.177 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v2", 2024-03-31T23:37:06.177 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:0", 2024-03-31T23:37:06.177 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 3651055062 2024-03-31T23:37:06.177 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-31T23:37:06.177 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-03-31T23:37:06.177 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:06.177 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-03-31T23:37:06.178 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "rbd_support", 2024-03-31T23:37:06.178 INFO:teuthology.orchestra.run.smithi046.stdout: "addrvec": [ 2024-03-31T23:37:06.178 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-03-31T23:37:06.178 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v2", 2024-03-31T23:37:06.178 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:0", 2024-03-31T23:37:06.178 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 442434065 2024-03-31T23:37:06.178 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-31T23:37:06.178 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-03-31T23:37:06.178 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:06.178 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-03-31T23:37:06.178 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "snap_schedule", 2024-03-31T23:37:06.178 INFO:teuthology.orchestra.run.smithi046.stdout: "addrvec": [ 2024-03-31T23:37:06.178 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-03-31T23:37:06.178 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v2", 2024-03-31T23:37:06.178 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:0", 2024-03-31T23:37:06.179 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 841860464 2024-03-31T23:37:06.179 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-31T23:37:06.179 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-03-31T23:37:06.179 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:06.179 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-03-31T23:37:06.179 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "snap_schedule", 2024-03-31T23:37:06.179 INFO:teuthology.orchestra.run.smithi046.stdout: "addrvec": [ 2024-03-31T23:37:06.179 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-03-31T23:37:06.179 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v2", 2024-03-31T23:37:06.179 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:0", 2024-03-31T23:37:06.179 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 3637599866 2024-03-31T23:37:06.179 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-31T23:37:06.179 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-03-31T23:37:06.179 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:06.180 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-03-31T23:37:06.180 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "volumes", 2024-03-31T23:37:06.180 INFO:teuthology.orchestra.run.smithi046.stdout: "addrvec": [ 2024-03-31T23:37:06.180 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-03-31T23:37:06.180 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v2", 2024-03-31T23:37:06.180 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:0", 2024-03-31T23:37:06.180 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 757544560 2024-03-31T23:37:06.180 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-31T23:37:06.180 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-03-31T23:37:06.180 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-31T23:37:06.180 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-03-31T23:37:06.180 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-03-31T23:37:06.181 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:37:04 smithi162 ceph-mon[36237]: from='client.? 172.21.15.46:0/4104412423' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-31T23:37:06.182 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:37:04 smithi162 ceph-mon[36237]: from='client.? 172.21.15.46:0/2992532885' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-31T23:37:06.182 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:37:04 smithi162 ceph-mon[36237]: from='client.? 172.21.15.46:0/4032938391' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-31T23:37:06.182 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:37:04 smithi142 ceph-mon[35271]: from='client.? 172.21.15.46:0/4104412423' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-31T23:37:06.182 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:37:04 smithi142 ceph-mon[35271]: from='client.? 172.21.15.46:0/2992532885' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-31T23:37:06.182 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:37:04 smithi142 ceph-mon[35271]: from='client.? 172.21.15.46:0/4032938391' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-31T23:37:06.183 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:04.914+0000 7fb5a38d4640 1 -- 172.21.15.46:0/4154080297 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fb5880a2610 msgr2=0x7fb5880a4ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:37:06.183 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:04.914+0000 7fb5a38d4640 1 --2- 172.21.15.46:0/4154080297 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fb5880a2610 0x7fb5880a4ad0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fb598009900 tx=0x7fb598005a20 comp rx=0 tx=0).stop 2024-03-31T23:37:06.183 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:04.914+0000 7fb5a38d4640 1 -- 172.21.15.46:0/4154080297 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb59c1076f0 msgr2=0x7fb59c19c800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:37:06.183 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:04.914+0000 7fb5a38d4640 1 --2- 172.21.15.46:0/4154080297 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb59c1076f0 0x7fb59c19c800 secure :-1 s=READY pgs=1070 cs=0 l=1 rev1=1 crypto rx=0x7fb59400eb10 tx=0x7fb59400c2d0 comp rx=0 tx=0).stop 2024-03-31T23:37:06.183 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:04.914+0000 7fb5a38d4640 1 -- 172.21.15.46:0/4154080297 shutdown_connections 2024-03-31T23:37:06.183 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:04.914+0000 7fb5a38d4640 1 --2- 172.21.15.46:0/4154080297 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fb5880a2610 0x7fb5880a4ad0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:37:06.183 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:04.914+0000 7fb5a38d4640 1 --2- 172.21.15.46:0/4154080297 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb59c1076f0 0x7fb59c19c800 unknown :-1 s=CLOSED pgs=1070 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:37:06.183 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:04.914+0000 7fb5a38d4640 1 --2- 172.21.15.46:0/4154080297 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb59c1020a0 0x7fb59c198480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:37:06.183 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:04.914+0000 7fb5a38d4640 1 --2- 172.21.15.46:0/4154080297 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb59c1016e0 0x7fb59c197f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:37:06.183 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:04.914+0000 7fb5a38d4640 1 -- 172.21.15.46:0/4154080297 >> 172.21.15.46:0/4154080297 conn(0x7fb59c0fd1f0 msgr2=0x7fb59c0fdb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:37:06.183 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:04.914+0000 7fb5a38d4640 1 -- 172.21.15.46:0/4154080297 shutdown_connections 2024-03-31T23:37:06.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:04.914+0000 7fb5a38d4640 1 -- 172.21.15.46:0/4154080297 wait complete. 2024-03-31T23:37:06.189 DEBUG:tasks.check_counter:Getting stats from a 2024-03-31T23:37:06.189 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:384f31f3d3aee7f8ba3f60846b8844c049165583 shell --fsid 6a9d8f08-efb2-11ee-b647-cb9ed24678a4 -- ceph daemon mds.a perf dump 2024-03-31T23:37:06.190 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:37:04 smithi046 ceph-mon[32498]: from='client.? 172.21.15.46:0/4104412423' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-31T23:37:06.190 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:37:04 smithi046 ceph-mon[32498]: from='client.? 172.21.15.46:0/2992532885' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-31T23:37:06.190 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:37:04 smithi046 ceph-mon[32498]: from='client.? 172.21.15.46:0/4032938391' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-31T23:37:06.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:37:05 smithi142 ceph-mon[35271]: from='client.? 172.21.15.46:0/4154080297' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-03-31T23:37:06.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:37:05 smithi142 ceph-mon[35271]: pgmap v806: 129 pgs: 129 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 3.2 MiB/s wr, 318 op/s 2024-03-31T23:37:06.218 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:37:05 smithi046 ceph-mon[32498]: from='client.? 172.21.15.46:0/4154080297' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-03-31T23:37:06.218 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:37:05 smithi046 ceph-mon[32498]: pgmap v806: 129 pgs: 129 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 3.2 MiB/s wr, 318 op/s 2024-03-31T23:37:06.295 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:37:05 smithi162 ceph-mon[36237]: from='client.? 172.21.15.46:0/4154080297' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-03-31T23:37:06.295 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:37:05 smithi162 ceph-mon[36237]: pgmap v806: 129 pgs: 129 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 3.2 MiB/s wr, 318 op/s 2024-03-31T23:37:06.450 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/6a9d8f08-efb2-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-31T23:37:07.147 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:37:06 smithi046 ceph-mon[32498]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-03-31T23:37:07.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:37:06 smithi142 ceph-mon[35271]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-03-31T23:37:07.295 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:37:06 smithi162 ceph-mon[36237]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-03-31T23:37:08.186 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:37:07 smithi046 ceph-mon[32498]: pgmap v807: 129 pgs: 129 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 3.2 MiB/s wr, 318 op/s 2024-03-31T23:37:08.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:37:07 smithi142 ceph-mon[35271]: pgmap v807: 129 pgs: 129 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 3.2 MiB/s wr, 318 op/s 2024-03-31T23:37:08.295 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:37:07 smithi162 ceph-mon[36237]: pgmap v807: 129 pgs: 129 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 3.2 MiB/s wr, 318 op/s 2024-03-31T23:37:08.612 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-03-31T23:37:08.612 INFO:teuthology.orchestra.run.smithi046.stdout: "AsyncMessenger::Worker-0": { 2024-03-31T23:37:08.612 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_recv_messages": 0, 2024-03-31T23:37:08.612 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_send_messages": 2, 2024-03-31T23:37:08.612 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_recv_bytes": 0, 2024-03-31T23:37:08.612 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_send_bytes": 0, 2024-03-31T23:37:08.612 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_created_connections": 4, 2024-03-31T23:37:08.612 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_active_connections": 0, 2024-03-31T23:37:08.613 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_running_total_time": 0.001340181, 2024-03-31T23:37:08.613 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_running_send_time": 0.000000000, 2024-03-31T23:37:08.613 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_running_recv_time": 0.000300427, 2024-03-31T23:37:08.613 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-31T23:37:08.613 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_send_messages_queue_lat": { 2024-03-31T23:37:08.613 INFO:teuthology.orchestra.run.smithi046.stdout: "avgcount": 0, 2024-03-31T23:37:08.613 INFO:teuthology.orchestra.run.smithi046.stdout: "sum": 0.000000000, 2024-03-31T23:37:08.613 INFO:teuthology.orchestra.run.smithi046.stdout: "avgtime": 0.000000000 2024-03-31T23:37:08.613 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:08.613 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_handle_ack_lat": { 2024-03-31T23:37:08.613 INFO:teuthology.orchestra.run.smithi046.stdout: "avgcount": 0, 2024-03-31T23:37:08.613 INFO:teuthology.orchestra.run.smithi046.stdout: "sum": 0.000000000, 2024-03-31T23:37:08.613 INFO:teuthology.orchestra.run.smithi046.stdout: "avgtime": 0.000000000 2024-03-31T23:37:08.613 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:08.614 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-31T23:37:08.614 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_send_encrypted_bytes": 0 2024-03-31T23:37:08.614 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:08.614 INFO:teuthology.orchestra.run.smithi046.stdout: "AsyncMessenger::Worker-1": { 2024-03-31T23:37:08.614 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_recv_messages": 457, 2024-03-31T23:37:08.614 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_send_messages": 447, 2024-03-31T23:37:08.614 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_recv_bytes": 1505392, 2024-03-31T23:37:08.614 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_send_bytes": 276752, 2024-03-31T23:37:08.614 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_created_connections": 2, 2024-03-31T23:37:08.614 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_active_connections": 1, 2024-03-31T23:37:08.614 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_running_total_time": 0.063979021, 2024-03-31T23:37:08.614 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_running_send_time": 0.023288193, 2024-03-31T23:37:08.614 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_running_recv_time": 0.037138765, 2024-03-31T23:37:08.615 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_running_fast_dispatch_time": 0.008702282, 2024-03-31T23:37:08.615 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_send_messages_queue_lat": { 2024-03-31T23:37:08.615 INFO:teuthology.orchestra.run.smithi046.stdout: "avgcount": 446, 2024-03-31T23:37:08.615 INFO:teuthology.orchestra.run.smithi046.stdout: "sum": 0.015477024, 2024-03-31T23:37:08.615 INFO:teuthology.orchestra.run.smithi046.stdout: "avgtime": 0.000034701 2024-03-31T23:37:08.615 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:08.615 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_handle_ack_lat": { 2024-03-31T23:37:08.615 INFO:teuthology.orchestra.run.smithi046.stdout: "avgcount": 0, 2024-03-31T23:37:08.615 INFO:teuthology.orchestra.run.smithi046.stdout: "sum": 0.000000000, 2024-03-31T23:37:08.615 INFO:teuthology.orchestra.run.smithi046.stdout: "avgtime": 0.000000000 2024-03-31T23:37:08.615 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:08.615 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_recv_encrypted_bytes": 1505392, 2024-03-31T23:37:08.615 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_send_encrypted_bytes": 276752 2024-03-31T23:37:08.615 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:08.616 INFO:teuthology.orchestra.run.smithi046.stdout: "AsyncMessenger::Worker-2": { 2024-03-31T23:37:08.616 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_recv_messages": 7, 2024-03-31T23:37:08.616 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_send_messages": 344, 2024-03-31T23:37:08.616 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_recv_bytes": 3472, 2024-03-31T23:37:08.616 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_send_bytes": 293968, 2024-03-31T23:37:08.616 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_created_connections": 6, 2024-03-31T23:37:08.616 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_active_connections": 3, 2024-03-31T23:37:08.616 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_running_total_time": 0.021155875, 2024-03-31T23:37:08.616 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_running_send_time": 0.015291975, 2024-03-31T23:37:08.616 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_running_recv_time": 0.002574522, 2024-03-31T23:37:08.616 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-31T23:37:08.616 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_send_messages_queue_lat": { 2024-03-31T23:37:08.616 INFO:teuthology.orchestra.run.smithi046.stdout: "avgcount": 342, 2024-03-31T23:37:08.616 INFO:teuthology.orchestra.run.smithi046.stdout: "sum": 0.017625103, 2024-03-31T23:37:08.616 INFO:teuthology.orchestra.run.smithi046.stdout: "avgtime": 0.000051535 2024-03-31T23:37:08.617 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:08.617 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_handle_ack_lat": { 2024-03-31T23:37:08.617 INFO:teuthology.orchestra.run.smithi046.stdout: "avgcount": 0, 2024-03-31T23:37:08.617 INFO:teuthology.orchestra.run.smithi046.stdout: "sum": 0.000000000, 2024-03-31T23:37:08.617 INFO:teuthology.orchestra.run.smithi046.stdout: "avgtime": 0.000000000 2024-03-31T23:37:08.617 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:08.617 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_recv_encrypted_bytes": 3472, 2024-03-31T23:37:08.617 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_send_encrypted_bytes": 293968 2024-03-31T23:37:08.617 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:08.617 INFO:teuthology.orchestra.run.smithi046.stdout: "cct": { 2024-03-31T23:37:08.617 INFO:teuthology.orchestra.run.smithi046.stdout: "total_workers": 0, 2024-03-31T23:37:08.617 INFO:teuthology.orchestra.run.smithi046.stdout: "unhealthy_workers": 0 2024-03-31T23:37:08.617 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:08.617 INFO:teuthology.orchestra.run.smithi046.stdout: "mempool": { 2024-03-31T23:37:08.618 INFO:teuthology.orchestra.run.smithi046.stdout: "bloom_filter_bytes": 0, 2024-03-31T23:37:08.618 INFO:teuthology.orchestra.run.smithi046.stdout: "bloom_filter_items": 0, 2024-03-31T23:37:08.618 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_alloc_bytes": 0, 2024-03-31T23:37:08.618 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_alloc_items": 0, 2024-03-31T23:37:08.618 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_cache_data_bytes": 0, 2024-03-31T23:37:08.618 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_cache_data_items": 0, 2024-03-31T23:37:08.618 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-31T23:37:08.618 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_cache_onode_items": 0, 2024-03-31T23:37:08.618 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-31T23:37:08.618 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_cache_meta_items": 0, 2024-03-31T23:37:08.618 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_cache_other_bytes": 0, 2024-03-31T23:37:08.618 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_cache_other_items": 0, 2024-03-31T23:37:08.618 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-31T23:37:08.618 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_cache_buffer_items": 0, 2024-03-31T23:37:08.618 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_extent_bytes": 0, 2024-03-31T23:37:08.619 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_extent_items": 0, 2024-03-31T23:37:08.619 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_blob_bytes": 0, 2024-03-31T23:37:08.619 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_blob_items": 0, 2024-03-31T23:37:08.619 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-31T23:37:08.619 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_shared_blob_items": 0, 2024-03-31T23:37:08.619 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-31T23:37:08.619 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_inline_bl_items": 0, 2024-03-31T23:37:08.619 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_fsck_bytes": 0, 2024-03-31T23:37:08.619 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_fsck_items": 0, 2024-03-31T23:37:08.619 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_txc_bytes": 0, 2024-03-31T23:37:08.619 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_txc_items": 0, 2024-03-31T23:37:08.619 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-31T23:37:08.619 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_writing_deferred_items": 0, 2024-03-31T23:37:08.619 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_writing_bytes": 0, 2024-03-31T23:37:08.620 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_writing_items": 0, 2024-03-31T23:37:08.620 INFO:teuthology.orchestra.run.smithi046.stdout: "bluefs_bytes": 0, 2024-03-31T23:37:08.620 INFO:teuthology.orchestra.run.smithi046.stdout: "bluefs_items": 0, 2024-03-31T23:37:08.620 INFO:teuthology.orchestra.run.smithi046.stdout: "bluefs_file_reader_bytes": 0, 2024-03-31T23:37:08.620 INFO:teuthology.orchestra.run.smithi046.stdout: "bluefs_file_reader_items": 0, 2024-03-31T23:37:08.620 INFO:teuthology.orchestra.run.smithi046.stdout: "bluefs_file_writer_bytes": 0, 2024-03-31T23:37:08.620 INFO:teuthology.orchestra.run.smithi046.stdout: "bluefs_file_writer_items": 0, 2024-03-31T23:37:08.620 INFO:teuthology.orchestra.run.smithi046.stdout: "buffer_anon_bytes": 148714, 2024-03-31T23:37:08.620 INFO:teuthology.orchestra.run.smithi046.stdout: "buffer_anon_items": 30, 2024-03-31T23:37:08.620 INFO:teuthology.orchestra.run.smithi046.stdout: "buffer_meta_bytes": 0, 2024-03-31T23:37:08.620 INFO:teuthology.orchestra.run.smithi046.stdout: "buffer_meta_items": 0, 2024-03-31T23:37:08.620 INFO:teuthology.orchestra.run.smithi046.stdout: "osd_bytes": 0, 2024-03-31T23:37:08.620 INFO:teuthology.orchestra.run.smithi046.stdout: "osd_items": 0, 2024-03-31T23:37:08.620 INFO:teuthology.orchestra.run.smithi046.stdout: "osd_mapbl_bytes": 0, 2024-03-31T23:37:08.620 INFO:teuthology.orchestra.run.smithi046.stdout: "osd_mapbl_items": 0, 2024-03-31T23:37:08.621 INFO:teuthology.orchestra.run.smithi046.stdout: "osd_pglog_bytes": 0, 2024-03-31T23:37:08.621 INFO:teuthology.orchestra.run.smithi046.stdout: "osd_pglog_items": 0, 2024-03-31T23:37:08.621 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap_bytes": 0, 2024-03-31T23:37:08.621 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap_items": 0, 2024-03-31T23:37:08.621 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap_mapping_bytes": 0, 2024-03-31T23:37:08.621 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap_mapping_items": 0, 2024-03-31T23:37:08.621 INFO:teuthology.orchestra.run.smithi046.stdout: "pgmap_bytes": 0, 2024-03-31T23:37:08.621 INFO:teuthology.orchestra.run.smithi046.stdout: "pgmap_items": 0, 2024-03-31T23:37:08.621 INFO:teuthology.orchestra.run.smithi046.stdout: "mds_co_bytes": 928, 2024-03-31T23:37:08.621 INFO:teuthology.orchestra.run.smithi046.stdout: "mds_co_items": 2, 2024-03-31T23:37:08.621 INFO:teuthology.orchestra.run.smithi046.stdout: "unittest_1_bytes": 0, 2024-03-31T23:37:08.621 INFO:teuthology.orchestra.run.smithi046.stdout: "unittest_1_items": 0, 2024-03-31T23:37:08.621 INFO:teuthology.orchestra.run.smithi046.stdout: "unittest_2_bytes": 0, 2024-03-31T23:37:08.621 INFO:teuthology.orchestra.run.smithi046.stdout: "unittest_2_items": 0 2024-03-31T23:37:08.621 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:08.621 INFO:teuthology.orchestra.run.smithi046.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-31T23:37:08.622 INFO:teuthology.orchestra.run.smithi046.stdout: "val": 0, 2024-03-31T23:37:08.622 INFO:teuthology.orchestra.run.smithi046.stdout: "max": 104857600, 2024-03-31T23:37:08.622 INFO:teuthology.orchestra.run.smithi046.stdout: "get_started": 0, 2024-03-31T23:37:08.622 INFO:teuthology.orchestra.run.smithi046.stdout: "get": 460, 2024-03-31T23:37:08.622 INFO:teuthology.orchestra.run.smithi046.stdout: "get_sum": 1440835, 2024-03-31T23:37:08.622 INFO:teuthology.orchestra.run.smithi046.stdout: "get_or_fail_fail": 0, 2024-03-31T23:37:08.622 INFO:teuthology.orchestra.run.smithi046.stdout: "get_or_fail_success": 460, 2024-03-31T23:37:08.622 INFO:teuthology.orchestra.run.smithi046.stdout: "take": 0, 2024-03-31T23:37:08.622 INFO:teuthology.orchestra.run.smithi046.stdout: "take_sum": 0, 2024-03-31T23:37:08.622 INFO:teuthology.orchestra.run.smithi046.stdout: "put": 460, 2024-03-31T23:37:08.622 INFO:teuthology.orchestra.run.smithi046.stdout: "put_sum": 1440835, 2024-03-31T23:37:08.622 INFO:teuthology.orchestra.run.smithi046.stdout: "wait": { 2024-03-31T23:37:08.622 INFO:teuthology.orchestra.run.smithi046.stdout: "avgcount": 0, 2024-03-31T23:37:08.622 INFO:teuthology.orchestra.run.smithi046.stdout: "sum": 0.000000000, 2024-03-31T23:37:08.622 INFO:teuthology.orchestra.run.smithi046.stdout: "avgtime": 0.000000000 2024-03-31T23:37:08.623 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-31T23:37:08.623 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-31T23:37:08.623 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-03-31T23:37:10.443 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:37:10 smithi142 ceph-mon[35271]: pgmap v808: 129 pgs: 129 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 702 KiB/s rd, 620 KiB/s wr, 126 op/s 2024-03-31T23:37:10.545 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:37:10 smithi162 ceph-mon[36237]: pgmap v808: 129 pgs: 129 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 702 KiB/s rd, 620 KiB/s wr, 126 op/s 2024-03-31T23:37:10.611 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:37:10 smithi046 ceph-mon[32498]: pgmap v808: 129 pgs: 129 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 702 KiB/s rd, 620 KiB/s wr, 126 op/s 2024-03-31T23:37:12.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:37:11 smithi162 ceph-mon[36237]: osdmap e139: 12 total, 12 up, 12 in 2024-03-31T23:37:12.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:37:11 smithi046 ceph-mon[32498]: osdmap e139: 12 total, 12 up, 12 in 2024-03-31T23:37:12.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:37:11 smithi142 ceph-mon[35271]: osdmap e139: 12 total, 12 up, 12 in 2024-03-31T23:37:12.829 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.a 2024-03-31T23:37:12.830 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.a 2024-03-31T23:37:12.830 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update' not found on daemon mds.a 2024-03-31T23:37:12.830 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update_receipt' not found on daemon mds.a 2024-03-31T23:37:12.830 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.a 2024-03-31T23:37:12.830 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.a 2024-03-31T23:37:12.830 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.a 2024-03-31T23:37:12.830 DEBUG:tasks.check_counter:Getting stats from d 2024-03-31T23:37:12.830 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:384f31f3d3aee7f8ba3f60846b8844c049165583 shell --fsid 6a9d8f08-efb2-11ee-b647-cb9ed24678a4 -- ceph daemon mds.d perf dump 2024-03-31T23:37:13.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:37:12 smithi162 ceph-mon[36237]: pgmap v810: 129 pgs: 129 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 855 KiB/s rd, 754 KiB/s wr, 154 op/s 2024-03-31T23:37:13.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:37:12 smithi046 ceph-mon[32498]: pgmap v810: 129 pgs: 129 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 855 KiB/s rd, 754 KiB/s wr, 154 op/s 2024-03-31T23:37:13.125 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/6a9d8f08-efb2-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-31T23:37:13.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:37:12 smithi142 ceph-mon[35271]: pgmap v810: 129 pgs: 129 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 855 KiB/s rd, 754 KiB/s wr, 154 op/s 2024-03-31T23:37:14.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:37:14 smithi046 ceph-mon[32498]: pgmap v811: 129 pgs: 129 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 702 KiB/s rd, 620 KiB/s wr, 126 op/s 2024-03-31T23:37:15.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:37:14 smithi162 ceph-mon[36237]: pgmap v811: 129 pgs: 129 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 702 KiB/s rd, 620 KiB/s wr, 126 op/s 2024-03-31T23:37:15.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:37:14 smithi142 ceph-mon[35271]: pgmap v811: 129 pgs: 129 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 702 KiB/s rd, 620 KiB/s wr, 126 op/s 2024-03-31T23:37:17.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:37:16 smithi162 ceph-mon[36237]: pgmap v812: 129 pgs: 129 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 68 KiB/s wr, 20 op/s 2024-03-31T23:37:17.097 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:37:16 smithi046 ceph-mon[32498]: pgmap v812: 129 pgs: 129 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 68 KiB/s wr, 20 op/s 2024-03-31T23:37:17.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:37:16 smithi142 ceph-mon[35271]: pgmap v812: 129 pgs: 129 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 68 KiB/s wr, 20 op/s 2024-03-31T23:37:17.746 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-03-31T23:37:17.746 INFO:teuthology.orchestra.run.smithi046.stdout: "AsyncMessenger::Worker-0": { 2024-03-31T23:37:17.746 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_recv_messages": 0, 2024-03-31T23:37:17.746 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_send_messages": 2, 2024-03-31T23:37:17.746 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_recv_bytes": 0, 2024-03-31T23:37:17.746 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_send_bytes": 0, 2024-03-31T23:37:17.746 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_created_connections": 4, 2024-03-31T23:37:17.746 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_active_connections": 0, 2024-03-31T23:37:17.746 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_running_total_time": 0.001454991, 2024-03-31T23:37:17.747 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_running_send_time": 0.000000000, 2024-03-31T23:37:17.747 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_running_recv_time": 0.000309902, 2024-03-31T23:37:17.747 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-31T23:37:17.747 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_send_messages_queue_lat": { 2024-03-31T23:37:17.747 INFO:teuthology.orchestra.run.smithi046.stdout: "avgcount": 0, 2024-03-31T23:37:17.747 INFO:teuthology.orchestra.run.smithi046.stdout: "sum": 0.000000000, 2024-03-31T23:37:17.747 INFO:teuthology.orchestra.run.smithi046.stdout: "avgtime": 0.000000000 2024-03-31T23:37:17.747 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:17.747 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_handle_ack_lat": { 2024-03-31T23:37:17.747 INFO:teuthology.orchestra.run.smithi046.stdout: "avgcount": 0, 2024-03-31T23:37:17.747 INFO:teuthology.orchestra.run.smithi046.stdout: "sum": 0.000000000, 2024-03-31T23:37:17.748 INFO:teuthology.orchestra.run.smithi046.stdout: "avgtime": 0.000000000 2024-03-31T23:37:17.748 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:17.748 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-31T23:37:17.748 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_send_encrypted_bytes": 0 2024-03-31T23:37:17.748 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:17.748 INFO:teuthology.orchestra.run.smithi046.stdout: "AsyncMessenger::Worker-1": { 2024-03-31T23:37:17.748 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_recv_messages": 468, 2024-03-31T23:37:17.748 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_send_messages": 456, 2024-03-31T23:37:17.748 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_recv_bytes": 1510288, 2024-03-31T23:37:17.748 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_send_bytes": 281680, 2024-03-31T23:37:17.748 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_created_connections": 2, 2024-03-31T23:37:17.748 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_active_connections": 1, 2024-03-31T23:37:17.749 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_running_total_time": 0.065516763, 2024-03-31T23:37:17.749 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_running_send_time": 0.022954442, 2024-03-31T23:37:17.749 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_running_recv_time": 0.040009869, 2024-03-31T23:37:17.749 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_running_fast_dispatch_time": 0.008709992, 2024-03-31T23:37:17.749 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_send_messages_queue_lat": { 2024-03-31T23:37:17.749 INFO:teuthology.orchestra.run.smithi046.stdout: "avgcount": 456, 2024-03-31T23:37:17.749 INFO:teuthology.orchestra.run.smithi046.stdout: "sum": 0.014337875, 2024-03-31T23:37:17.749 INFO:teuthology.orchestra.run.smithi046.stdout: "avgtime": 0.000031442 2024-03-31T23:37:17.749 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:17.749 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_handle_ack_lat": { 2024-03-31T23:37:17.749 INFO:teuthology.orchestra.run.smithi046.stdout: "avgcount": 0, 2024-03-31T23:37:17.749 INFO:teuthology.orchestra.run.smithi046.stdout: "sum": 0.000000000, 2024-03-31T23:37:17.750 INFO:teuthology.orchestra.run.smithi046.stdout: "avgtime": 0.000000000 2024-03-31T23:37:17.750 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:17.750 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_recv_encrypted_bytes": 1510288, 2024-03-31T23:37:17.750 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_send_encrypted_bytes": 281680 2024-03-31T23:37:17.750 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:17.750 INFO:teuthology.orchestra.run.smithi046.stdout: "AsyncMessenger::Worker-2": { 2024-03-31T23:37:17.750 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_recv_messages": 3, 2024-03-31T23:37:17.750 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_send_messages": 348, 2024-03-31T23:37:17.750 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_recv_bytes": 480, 2024-03-31T23:37:17.750 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_send_bytes": 294576, 2024-03-31T23:37:17.750 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_created_connections": 6, 2024-03-31T23:37:17.750 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_active_connections": 3, 2024-03-31T23:37:17.751 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_running_total_time": 0.021089219, 2024-03-31T23:37:17.751 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_running_send_time": 0.015606349, 2024-03-31T23:37:17.751 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_running_recv_time": 0.002348949, 2024-03-31T23:37:17.751 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-31T23:37:17.751 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_send_messages_queue_lat": { 2024-03-31T23:37:17.751 INFO:teuthology.orchestra.run.smithi046.stdout: "avgcount": 345, 2024-03-31T23:37:17.751 INFO:teuthology.orchestra.run.smithi046.stdout: "sum": 0.014800750, 2024-03-31T23:37:17.751 INFO:teuthology.orchestra.run.smithi046.stdout: "avgtime": 0.000042900 2024-03-31T23:37:17.751 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:17.751 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_handle_ack_lat": { 2024-03-31T23:37:17.751 INFO:teuthology.orchestra.run.smithi046.stdout: "avgcount": 0, 2024-03-31T23:37:17.751 INFO:teuthology.orchestra.run.smithi046.stdout: "sum": 0.000000000, 2024-03-31T23:37:17.752 INFO:teuthology.orchestra.run.smithi046.stdout: "avgtime": 0.000000000 2024-03-31T23:37:17.752 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:17.752 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_recv_encrypted_bytes": 480, 2024-03-31T23:37:17.752 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_send_encrypted_bytes": 294576 2024-03-31T23:37:17.752 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:17.752 INFO:teuthology.orchestra.run.smithi046.stdout: "cct": { 2024-03-31T23:37:17.752 INFO:teuthology.orchestra.run.smithi046.stdout: "total_workers": 0, 2024-03-31T23:37:17.752 INFO:teuthology.orchestra.run.smithi046.stdout: "unhealthy_workers": 0 2024-03-31T23:37:17.752 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:17.752 INFO:teuthology.orchestra.run.smithi046.stdout: "mempool": { 2024-03-31T23:37:17.752 INFO:teuthology.orchestra.run.smithi046.stdout: "bloom_filter_bytes": 0, 2024-03-31T23:37:17.752 INFO:teuthology.orchestra.run.smithi046.stdout: "bloom_filter_items": 0, 2024-03-31T23:37:17.753 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_alloc_bytes": 0, 2024-03-31T23:37:17.753 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_alloc_items": 0, 2024-03-31T23:37:17.753 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_cache_data_bytes": 0, 2024-03-31T23:37:17.753 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_cache_data_items": 0, 2024-03-31T23:37:17.753 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-31T23:37:17.753 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_cache_onode_items": 0, 2024-03-31T23:37:17.753 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-31T23:37:17.753 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_cache_meta_items": 0, 2024-03-31T23:37:17.753 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_cache_other_bytes": 0, 2024-03-31T23:37:17.753 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_cache_other_items": 0, 2024-03-31T23:37:17.753 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-31T23:37:17.753 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_cache_buffer_items": 0, 2024-03-31T23:37:17.754 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_extent_bytes": 0, 2024-03-31T23:37:17.754 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_extent_items": 0, 2024-03-31T23:37:17.754 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_blob_bytes": 0, 2024-03-31T23:37:17.754 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_blob_items": 0, 2024-03-31T23:37:17.754 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-31T23:37:17.754 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_shared_blob_items": 0, 2024-03-31T23:37:17.754 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-31T23:37:17.754 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_inline_bl_items": 0, 2024-03-31T23:37:17.754 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_fsck_bytes": 0, 2024-03-31T23:37:17.754 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_fsck_items": 0, 2024-03-31T23:37:17.754 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_txc_bytes": 0, 2024-03-31T23:37:17.754 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_txc_items": 0, 2024-03-31T23:37:17.754 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-31T23:37:17.755 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_writing_deferred_items": 0, 2024-03-31T23:37:17.755 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_writing_bytes": 0, 2024-03-31T23:37:17.755 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_writing_items": 0, 2024-03-31T23:37:17.755 INFO:teuthology.orchestra.run.smithi046.stdout: "bluefs_bytes": 0, 2024-03-31T23:37:17.755 INFO:teuthology.orchestra.run.smithi046.stdout: "bluefs_items": 0, 2024-03-31T23:37:17.755 INFO:teuthology.orchestra.run.smithi046.stdout: "bluefs_file_reader_bytes": 0, 2024-03-31T23:37:17.755 INFO:teuthology.orchestra.run.smithi046.stdout: "bluefs_file_reader_items": 0, 2024-03-31T23:37:17.755 INFO:teuthology.orchestra.run.smithi046.stdout: "bluefs_file_writer_bytes": 0, 2024-03-31T23:37:17.755 INFO:teuthology.orchestra.run.smithi046.stdout: "bluefs_file_writer_items": 0, 2024-03-31T23:37:17.755 INFO:teuthology.orchestra.run.smithi046.stdout: "buffer_anon_bytes": 148794, 2024-03-31T23:37:17.755 INFO:teuthology.orchestra.run.smithi046.stdout: "buffer_anon_items": 32, 2024-03-31T23:37:17.756 INFO:teuthology.orchestra.run.smithi046.stdout: "buffer_meta_bytes": 0, 2024-03-31T23:37:17.756 INFO:teuthology.orchestra.run.smithi046.stdout: "buffer_meta_items": 0, 2024-03-31T23:37:17.756 INFO:teuthology.orchestra.run.smithi046.stdout: "osd_bytes": 0, 2024-03-31T23:37:17.756 INFO:teuthology.orchestra.run.smithi046.stdout: "osd_items": 0, 2024-03-31T23:37:17.756 INFO:teuthology.orchestra.run.smithi046.stdout: "osd_mapbl_bytes": 0, 2024-03-31T23:37:17.756 INFO:teuthology.orchestra.run.smithi046.stdout: "osd_mapbl_items": 0, 2024-03-31T23:37:17.756 INFO:teuthology.orchestra.run.smithi046.stdout: "osd_pglog_bytes": 0, 2024-03-31T23:37:17.756 INFO:teuthology.orchestra.run.smithi046.stdout: "osd_pglog_items": 0, 2024-03-31T23:37:17.756 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap_bytes": 0, 2024-03-31T23:37:17.756 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap_items": 0, 2024-03-31T23:37:17.756 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap_mapping_bytes": 0, 2024-03-31T23:37:17.756 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap_mapping_items": 0, 2024-03-31T23:37:17.757 INFO:teuthology.orchestra.run.smithi046.stdout: "pgmap_bytes": 0, 2024-03-31T23:37:17.757 INFO:teuthology.orchestra.run.smithi046.stdout: "pgmap_items": 0, 2024-03-31T23:37:17.757 INFO:teuthology.orchestra.run.smithi046.stdout: "mds_co_bytes": 928, 2024-03-31T23:37:17.757 INFO:teuthology.orchestra.run.smithi046.stdout: "mds_co_items": 2, 2024-03-31T23:37:17.757 INFO:teuthology.orchestra.run.smithi046.stdout: "unittest_1_bytes": 0, 2024-03-31T23:37:17.757 INFO:teuthology.orchestra.run.smithi046.stdout: "unittest_1_items": 0, 2024-03-31T23:37:17.757 INFO:teuthology.orchestra.run.smithi046.stdout: "unittest_2_bytes": 0, 2024-03-31T23:37:17.757 INFO:teuthology.orchestra.run.smithi046.stdout: "unittest_2_items": 0 2024-03-31T23:37:17.757 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:17.757 INFO:teuthology.orchestra.run.smithi046.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-31T23:37:17.757 INFO:teuthology.orchestra.run.smithi046.stdout: "val": 0, 2024-03-31T23:37:17.758 INFO:teuthology.orchestra.run.smithi046.stdout: "max": 104857600, 2024-03-31T23:37:17.758 INFO:teuthology.orchestra.run.smithi046.stdout: "get_started": 0, 2024-03-31T23:37:17.758 INFO:teuthology.orchestra.run.smithi046.stdout: "get": 467, 2024-03-31T23:37:17.758 INFO:teuthology.orchestra.run.smithi046.stdout: "get_sum": 1441745, 2024-03-31T23:37:17.758 INFO:teuthology.orchestra.run.smithi046.stdout: "get_or_fail_fail": 0, 2024-03-31T23:37:17.758 INFO:teuthology.orchestra.run.smithi046.stdout: "get_or_fail_success": 467, 2024-03-31T23:37:17.758 INFO:teuthology.orchestra.run.smithi046.stdout: "take": 0, 2024-03-31T23:37:17.758 INFO:teuthology.orchestra.run.smithi046.stdout: "take_sum": 0, 2024-03-31T23:37:17.758 INFO:teuthology.orchestra.run.smithi046.stdout: "put": 467, 2024-03-31T23:37:17.758 INFO:teuthology.orchestra.run.smithi046.stdout: "put_sum": 1441745, 2024-03-31T23:37:17.758 INFO:teuthology.orchestra.run.smithi046.stdout: "wait": { 2024-03-31T23:37:17.758 INFO:teuthology.orchestra.run.smithi046.stdout: "avgcount": 0, 2024-03-31T23:37:17.759 INFO:teuthology.orchestra.run.smithi046.stdout: "sum": 0.000000000, 2024-03-31T23:37:17.759 INFO:teuthology.orchestra.run.smithi046.stdout: "avgtime": 0.000000000 2024-03-31T23:37:17.759 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-31T23:37:17.759 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-31T23:37:17.759 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-03-31T23:37:18.518 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.d 2024-03-31T23:37:18.518 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.d 2024-03-31T23:37:18.518 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update' not found on daemon mds.d 2024-03-31T23:37:18.518 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update_receipt' not found on daemon mds.d 2024-03-31T23:37:18.519 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.d 2024-03-31T23:37:18.519 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.d 2024-03-31T23:37:18.519 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.d 2024-03-31T23:37:18.519 DEBUG:tasks.check_counter:Getting stats from g 2024-03-31T23:37:18.519 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:384f31f3d3aee7f8ba3f60846b8844c049165583 shell --fsid 6a9d8f08-efb2-11ee-b647-cb9ed24678a4 -- ceph daemon mds.g perf dump 2024-03-31T23:37:18.770 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/6a9d8f08-efb2-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-31T23:37:18.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:37:18 smithi046 ceph-mon[32498]: pgmap v813: 129 pgs: 129 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 68 KiB/s wr, 20 op/s 2024-03-31T23:37:19.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:37:18 smithi162 ceph-mon[36237]: pgmap v813: 129 pgs: 129 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 68 KiB/s wr, 20 op/s 2024-03-31T23:37:19.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:37:18 smithi142 ceph-mon[35271]: pgmap v813: 129 pgs: 129 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 68 KiB/s wr, 20 op/s 2024-03-31T23:37:20.466 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-03-31T23:37:20.467 INFO:teuthology.orchestra.run.smithi046.stdout: "AsyncMessenger::Worker-0": { 2024-03-31T23:37:20.467 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_recv_messages": 464, 2024-03-31T23:37:20.467 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_send_messages": 452, 2024-03-31T23:37:20.467 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_recv_bytes": 1509200, 2024-03-31T23:37:20.467 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_send_bytes": 279120, 2024-03-31T23:37:20.467 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_created_connections": 4, 2024-03-31T23:37:20.467 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_active_connections": 0, 2024-03-31T23:37:20.467 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_running_total_time": 0.064683901, 2024-03-31T23:37:20.467 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_running_send_time": 0.023235817, 2024-03-31T23:37:20.467 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_running_recv_time": 0.037338275, 2024-03-31T23:37:20.468 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_running_fast_dispatch_time": 0.008958862, 2024-03-31T23:37:20.468 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_send_messages_queue_lat": { 2024-03-31T23:37:20.468 INFO:teuthology.orchestra.run.smithi046.stdout: "avgcount": 452, 2024-03-31T23:37:20.468 INFO:teuthology.orchestra.run.smithi046.stdout: "sum": 0.016078807, 2024-03-31T23:37:20.468 INFO:teuthology.orchestra.run.smithi046.stdout: "avgtime": 0.000035572 2024-03-31T23:37:20.468 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:20.468 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_handle_ack_lat": { 2024-03-31T23:37:20.468 INFO:teuthology.orchestra.run.smithi046.stdout: "avgcount": 0, 2024-03-31T23:37:20.468 INFO:teuthology.orchestra.run.smithi046.stdout: "sum": 0.000000000, 2024-03-31T23:37:20.468 INFO:teuthology.orchestra.run.smithi046.stdout: "avgtime": 0.000000000 2024-03-31T23:37:20.468 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:20.468 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_recv_encrypted_bytes": 1509200, 2024-03-31T23:37:20.468 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_send_encrypted_bytes": 279120 2024-03-31T23:37:20.468 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:20.469 INFO:teuthology.orchestra.run.smithi046.stdout: "AsyncMessenger::Worker-1": { 2024-03-31T23:37:20.469 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_recv_messages": 3, 2024-03-31T23:37:20.469 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_send_messages": 345, 2024-03-31T23:37:20.469 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_recv_bytes": 480, 2024-03-31T23:37:20.469 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_send_bytes": 294048, 2024-03-31T23:37:20.469 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_created_connections": 6, 2024-03-31T23:37:20.469 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_active_connections": 3, 2024-03-31T23:37:20.469 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_running_total_time": 0.021714859, 2024-03-31T23:37:20.469 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_running_send_time": 0.015152464, 2024-03-31T23:37:20.469 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_running_recv_time": 0.003302129, 2024-03-31T23:37:20.469 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-31T23:37:20.469 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_send_messages_queue_lat": { 2024-03-31T23:37:20.469 INFO:teuthology.orchestra.run.smithi046.stdout: "avgcount": 342, 2024-03-31T23:37:20.469 INFO:teuthology.orchestra.run.smithi046.stdout: "sum": 0.014678366, 2024-03-31T23:37:20.469 INFO:teuthology.orchestra.run.smithi046.stdout: "avgtime": 0.000042919 2024-03-31T23:37:20.470 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:20.470 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_handle_ack_lat": { 2024-03-31T23:37:20.470 INFO:teuthology.orchestra.run.smithi046.stdout: "avgcount": 0, 2024-03-31T23:37:20.470 INFO:teuthology.orchestra.run.smithi046.stdout: "sum": 0.000000000, 2024-03-31T23:37:20.470 INFO:teuthology.orchestra.run.smithi046.stdout: "avgtime": 0.000000000 2024-03-31T23:37:20.470 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:20.470 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_recv_encrypted_bytes": 480, 2024-03-31T23:37:20.470 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_send_encrypted_bytes": 294048 2024-03-31T23:37:20.470 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:20.470 INFO:teuthology.orchestra.run.smithi046.stdout: "AsyncMessenger::Worker-2": { 2024-03-31T23:37:20.470 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_recv_messages": 0, 2024-03-31T23:37:20.470 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_send_messages": 2, 2024-03-31T23:37:20.471 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_recv_bytes": 0, 2024-03-31T23:37:20.471 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_send_bytes": 0, 2024-03-31T23:37:20.471 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_created_connections": 2, 2024-03-31T23:37:20.471 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_active_connections": 1, 2024-03-31T23:37:20.471 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_running_total_time": 0.000397300, 2024-03-31T23:37:20.471 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_running_send_time": 0.000000000, 2024-03-31T23:37:20.471 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_running_recv_time": 0.000304716, 2024-03-31T23:37:20.471 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-31T23:37:20.471 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_send_messages_queue_lat": { 2024-03-31T23:37:20.471 INFO:teuthology.orchestra.run.smithi046.stdout: "avgcount": 0, 2024-03-31T23:37:20.471 INFO:teuthology.orchestra.run.smithi046.stdout: "sum": 0.000000000, 2024-03-31T23:37:20.471 INFO:teuthology.orchestra.run.smithi046.stdout: "avgtime": 0.000000000 2024-03-31T23:37:20.471 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:20.472 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_handle_ack_lat": { 2024-03-31T23:37:20.472 INFO:teuthology.orchestra.run.smithi046.stdout: "avgcount": 0, 2024-03-31T23:37:20.472 INFO:teuthology.orchestra.run.smithi046.stdout: "sum": 0.000000000, 2024-03-31T23:37:20.472 INFO:teuthology.orchestra.run.smithi046.stdout: "avgtime": 0.000000000 2024-03-31T23:37:20.472 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:20.472 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-31T23:37:20.472 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_send_encrypted_bytes": 0 2024-03-31T23:37:20.472 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:20.472 INFO:teuthology.orchestra.run.smithi046.stdout: "cct": { 2024-03-31T23:37:20.472 INFO:teuthology.orchestra.run.smithi046.stdout: "total_workers": 0, 2024-03-31T23:37:20.472 INFO:teuthology.orchestra.run.smithi046.stdout: "unhealthy_workers": 0 2024-03-31T23:37:20.472 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:20.472 INFO:teuthology.orchestra.run.smithi046.stdout: "mempool": { 2024-03-31T23:37:20.472 INFO:teuthology.orchestra.run.smithi046.stdout: "bloom_filter_bytes": 0, 2024-03-31T23:37:20.473 INFO:teuthology.orchestra.run.smithi046.stdout: "bloom_filter_items": 0, 2024-03-31T23:37:20.473 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_alloc_bytes": 0, 2024-03-31T23:37:20.473 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_alloc_items": 0, 2024-03-31T23:37:20.473 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_cache_data_bytes": 0, 2024-03-31T23:37:20.473 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_cache_data_items": 0, 2024-03-31T23:37:20.473 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-31T23:37:20.473 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_cache_onode_items": 0, 2024-03-31T23:37:20.473 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-31T23:37:20.473 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_cache_meta_items": 0, 2024-03-31T23:37:20.473 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_cache_other_bytes": 0, 2024-03-31T23:37:20.473 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_cache_other_items": 0, 2024-03-31T23:37:20.473 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-31T23:37:20.473 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_cache_buffer_items": 0, 2024-03-31T23:37:20.473 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_extent_bytes": 0, 2024-03-31T23:37:20.473 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_extent_items": 0, 2024-03-31T23:37:20.474 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_blob_bytes": 0, 2024-03-31T23:37:20.474 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_blob_items": 0, 2024-03-31T23:37:20.474 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-31T23:37:20.474 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_shared_blob_items": 0, 2024-03-31T23:37:20.474 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-31T23:37:20.474 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_inline_bl_items": 0, 2024-03-31T23:37:20.474 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_fsck_bytes": 0, 2024-03-31T23:37:20.474 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_fsck_items": 0, 2024-03-31T23:37:20.474 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_txc_bytes": 0, 2024-03-31T23:37:20.474 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_txc_items": 0, 2024-03-31T23:37:20.474 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-31T23:37:20.475 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_writing_deferred_items": 0, 2024-03-31T23:37:20.475 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_writing_bytes": 0, 2024-03-31T23:37:20.475 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_writing_items": 0, 2024-03-31T23:37:20.475 INFO:teuthology.orchestra.run.smithi046.stdout: "bluefs_bytes": 0, 2024-03-31T23:37:20.475 INFO:teuthology.orchestra.run.smithi046.stdout: "bluefs_items": 0, 2024-03-31T23:37:20.475 INFO:teuthology.orchestra.run.smithi046.stdout: "bluefs_file_reader_bytes": 0, 2024-03-31T23:37:20.475 INFO:teuthology.orchestra.run.smithi046.stdout: "bluefs_file_reader_items": 0, 2024-03-31T23:37:20.475 INFO:teuthology.orchestra.run.smithi046.stdout: "bluefs_file_writer_bytes": 0, 2024-03-31T23:37:20.475 INFO:teuthology.orchestra.run.smithi046.stdout: "bluefs_file_writer_items": 0, 2024-03-31T23:37:20.475 INFO:teuthology.orchestra.run.smithi046.stdout: "buffer_anon_bytes": 148754, 2024-03-31T23:37:20.475 INFO:teuthology.orchestra.run.smithi046.stdout: "buffer_anon_items": 31, 2024-03-31T23:37:20.475 INFO:teuthology.orchestra.run.smithi046.stdout: "buffer_meta_bytes": 0, 2024-03-31T23:37:20.475 INFO:teuthology.orchestra.run.smithi046.stdout: "buffer_meta_items": 0, 2024-03-31T23:37:20.475 INFO:teuthology.orchestra.run.smithi046.stdout: "osd_bytes": 0, 2024-03-31T23:37:20.476 INFO:teuthology.orchestra.run.smithi046.stdout: "osd_items": 0, 2024-03-31T23:37:20.476 INFO:teuthology.orchestra.run.smithi046.stdout: "osd_mapbl_bytes": 0, 2024-03-31T23:37:20.476 INFO:teuthology.orchestra.run.smithi046.stdout: "osd_mapbl_items": 0, 2024-03-31T23:37:20.476 INFO:teuthology.orchestra.run.smithi046.stdout: "osd_pglog_bytes": 0, 2024-03-31T23:37:20.476 INFO:teuthology.orchestra.run.smithi046.stdout: "osd_pglog_items": 0, 2024-03-31T23:37:20.476 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap_bytes": 0, 2024-03-31T23:37:20.476 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap_items": 0, 2024-03-31T23:37:20.476 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap_mapping_bytes": 0, 2024-03-31T23:37:20.476 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap_mapping_items": 0, 2024-03-31T23:37:20.476 INFO:teuthology.orchestra.run.smithi046.stdout: "pgmap_bytes": 0, 2024-03-31T23:37:20.476 INFO:teuthology.orchestra.run.smithi046.stdout: "pgmap_items": 0, 2024-03-31T23:37:20.476 INFO:teuthology.orchestra.run.smithi046.stdout: "mds_co_bytes": 928, 2024-03-31T23:37:20.476 INFO:teuthology.orchestra.run.smithi046.stdout: "mds_co_items": 2, 2024-03-31T23:37:20.476 INFO:teuthology.orchestra.run.smithi046.stdout: "unittest_1_bytes": 0, 2024-03-31T23:37:20.476 INFO:teuthology.orchestra.run.smithi046.stdout: "unittest_1_items": 0, 2024-03-31T23:37:20.477 INFO:teuthology.orchestra.run.smithi046.stdout: "unittest_2_bytes": 0, 2024-03-31T23:37:20.477 INFO:teuthology.orchestra.run.smithi046.stdout: "unittest_2_items": 0 2024-03-31T23:37:20.477 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:20.477 INFO:teuthology.orchestra.run.smithi046.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-31T23:37:20.477 INFO:teuthology.orchestra.run.smithi046.stdout: "val": 0, 2024-03-31T23:37:20.477 INFO:teuthology.orchestra.run.smithi046.stdout: "max": 104857600, 2024-03-31T23:37:20.477 INFO:teuthology.orchestra.run.smithi046.stdout: "get_started": 0, 2024-03-31T23:37:20.477 INFO:teuthology.orchestra.run.smithi046.stdout: "get": 463, 2024-03-31T23:37:20.477 INFO:teuthology.orchestra.run.smithi046.stdout: "get_sum": 1441225, 2024-03-31T23:37:20.477 INFO:teuthology.orchestra.run.smithi046.stdout: "get_or_fail_fail": 0, 2024-03-31T23:37:20.477 INFO:teuthology.orchestra.run.smithi046.stdout: "get_or_fail_success": 463, 2024-03-31T23:37:20.477 INFO:teuthology.orchestra.run.smithi046.stdout: "take": 0, 2024-03-31T23:37:20.477 INFO:teuthology.orchestra.run.smithi046.stdout: "take_sum": 0, 2024-03-31T23:37:20.478 INFO:teuthology.orchestra.run.smithi046.stdout: "put": 463, 2024-03-31T23:37:20.478 INFO:teuthology.orchestra.run.smithi046.stdout: "put_sum": 1441225, 2024-03-31T23:37:20.478 INFO:teuthology.orchestra.run.smithi046.stdout: "wait": { 2024-03-31T23:37:20.478 INFO:teuthology.orchestra.run.smithi046.stdout: "avgcount": 0, 2024-03-31T23:37:20.478 INFO:teuthology.orchestra.run.smithi046.stdout: "sum": 0.000000000, 2024-03-31T23:37:20.478 INFO:teuthology.orchestra.run.smithi046.stdout: "avgtime": 0.000000000 2024-03-31T23:37:20.478 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-31T23:37:20.478 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-31T23:37:20.478 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-03-31T23:37:21.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:37:20 smithi162 ceph-mon[36237]: pgmap v814: 129 pgs: 129 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 11 KiB/s wr, 10 op/s 2024-03-31T23:37:21.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:37:20 smithi046 ceph-mon[32498]: pgmap v814: 129 pgs: 129 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 11 KiB/s wr, 10 op/s 2024-03-31T23:37:21.186 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.g 2024-03-31T23:37:21.186 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.g 2024-03-31T23:37:21.186 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update' not found on daemon mds.g 2024-03-31T23:37:21.186 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update_receipt' not found on daemon mds.g 2024-03-31T23:37:21.186 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.g 2024-03-31T23:37:21.186 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.g 2024-03-31T23:37:21.186 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.g 2024-03-31T23:37:21.186 DEBUG:tasks.check_counter:Getting stats from j 2024-03-31T23:37:21.187 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:384f31f3d3aee7f8ba3f60846b8844c049165583 shell --fsid 6a9d8f08-efb2-11ee-b647-cb9ed24678a4 -- ceph daemon mds.j perf dump 2024-03-31T23:37:21.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:37:20 smithi142 ceph-mon[35271]: pgmap v814: 129 pgs: 129 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 11 KiB/s wr, 10 op/s 2024-03-31T23:37:21.435 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/6a9d8f08-efb2-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-31T23:37:22.994 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:37:22 smithi142 ceph-mon[35271]: pgmap v815: 129 pgs: 129 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 11 KiB/s wr, 9 op/s 2024-03-31T23:37:23.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:37:22 smithi162 ceph-mon[36237]: pgmap v815: 129 pgs: 129 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 11 KiB/s wr, 9 op/s 2024-03-31T23:37:23.085 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:37:22 smithi046 ceph-mon[32498]: pgmap v815: 129 pgs: 129 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 11 KiB/s wr, 9 op/s 2024-03-31T23:37:23.401 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-03-31T23:37:23.401 INFO:teuthology.orchestra.run.smithi046.stdout: "AsyncMessenger::Worker-0": { 2024-03-31T23:37:23.401 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_recv_messages": 0, 2024-03-31T23:37:23.401 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_send_messages": 2, 2024-03-31T23:37:23.401 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_recv_bytes": 0, 2024-03-31T23:37:23.402 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_send_bytes": 0, 2024-03-31T23:37:23.402 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_created_connections": 4, 2024-03-31T23:37:23.402 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_active_connections": 0, 2024-03-31T23:37:23.402 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_running_total_time": 0.001309052, 2024-03-31T23:37:23.402 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_running_send_time": 0.000000000, 2024-03-31T23:37:23.402 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_running_recv_time": 0.000430121, 2024-03-31T23:37:23.402 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-31T23:37:23.402 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_send_messages_queue_lat": { 2024-03-31T23:37:23.402 INFO:teuthology.orchestra.run.smithi046.stdout: "avgcount": 0, 2024-03-31T23:37:23.402 INFO:teuthology.orchestra.run.smithi046.stdout: "sum": 0.000000000, 2024-03-31T23:37:23.402 INFO:teuthology.orchestra.run.smithi046.stdout: "avgtime": 0.000000000 2024-03-31T23:37:23.402 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:23.402 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_handle_ack_lat": { 2024-03-31T23:37:23.403 INFO:teuthology.orchestra.run.smithi046.stdout: "avgcount": 0, 2024-03-31T23:37:23.403 INFO:teuthology.orchestra.run.smithi046.stdout: "sum": 0.000000000, 2024-03-31T23:37:23.403 INFO:teuthology.orchestra.run.smithi046.stdout: "avgtime": 0.000000000 2024-03-31T23:37:23.403 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:23.403 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-31T23:37:23.403 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_send_encrypted_bytes": 0 2024-03-31T23:37:23.403 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:23.403 INFO:teuthology.orchestra.run.smithi046.stdout: "AsyncMessenger::Worker-1": { 2024-03-31T23:37:23.403 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_recv_messages": 7, 2024-03-31T23:37:23.403 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_send_messages": 351, 2024-03-31T23:37:23.403 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_recv_bytes": 3472, 2024-03-31T23:37:23.403 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_send_bytes": 295200, 2024-03-31T23:37:23.403 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_created_connections": 6, 2024-03-31T23:37:23.403 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_active_connections": 3, 2024-03-31T23:37:23.403 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_running_total_time": 0.022982522, 2024-03-31T23:37:23.404 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_running_send_time": 0.017120263, 2024-03-31T23:37:23.404 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_running_recv_time": 0.002624973, 2024-03-31T23:37:23.404 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-31T23:37:23.404 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_send_messages_queue_lat": { 2024-03-31T23:37:23.404 INFO:teuthology.orchestra.run.smithi046.stdout: "avgcount": 349, 2024-03-31T23:37:23.404 INFO:teuthology.orchestra.run.smithi046.stdout: "sum": 0.015535884, 2024-03-31T23:37:23.404 INFO:teuthology.orchestra.run.smithi046.stdout: "avgtime": 0.000044515 2024-03-31T23:37:23.404 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:23.404 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_handle_ack_lat": { 2024-03-31T23:37:23.404 INFO:teuthology.orchestra.run.smithi046.stdout: "avgcount": 0, 2024-03-31T23:37:23.404 INFO:teuthology.orchestra.run.smithi046.stdout: "sum": 0.000000000, 2024-03-31T23:37:23.404 INFO:teuthology.orchestra.run.smithi046.stdout: "avgtime": 0.000000000 2024-03-31T23:37:23.404 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:23.404 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_recv_encrypted_bytes": 3472, 2024-03-31T23:37:23.405 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_send_encrypted_bytes": 295200 2024-03-31T23:37:23.405 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:23.405 INFO:teuthology.orchestra.run.smithi046.stdout: "AsyncMessenger::Worker-2": { 2024-03-31T23:37:23.405 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_recv_messages": 466, 2024-03-31T23:37:23.405 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_send_messages": 456, 2024-03-31T23:37:23.405 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_recv_bytes": 1507840, 2024-03-31T23:37:23.405 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_send_bytes": 282512, 2024-03-31T23:37:23.405 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_created_connections": 2, 2024-03-31T23:37:23.405 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_active_connections": 1, 2024-03-31T23:37:23.405 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_running_total_time": 0.068490478, 2024-03-31T23:37:23.405 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_running_send_time": 0.023946160, 2024-03-31T23:37:23.405 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_running_recv_time": 0.041242333, 2024-03-31T23:37:23.405 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_running_fast_dispatch_time": 0.009007984, 2024-03-31T23:37:23.405 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_send_messages_queue_lat": { 2024-03-31T23:37:23.405 INFO:teuthology.orchestra.run.smithi046.stdout: "avgcount": 455, 2024-03-31T23:37:23.405 INFO:teuthology.orchestra.run.smithi046.stdout: "sum": 0.019393389, 2024-03-31T23:37:23.406 INFO:teuthology.orchestra.run.smithi046.stdout: "avgtime": 0.000042622 2024-03-31T23:37:23.406 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:23.406 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_handle_ack_lat": { 2024-03-31T23:37:23.406 INFO:teuthology.orchestra.run.smithi046.stdout: "avgcount": 0, 2024-03-31T23:37:23.406 INFO:teuthology.orchestra.run.smithi046.stdout: "sum": 0.000000000, 2024-03-31T23:37:23.406 INFO:teuthology.orchestra.run.smithi046.stdout: "avgtime": 0.000000000 2024-03-31T23:37:23.406 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:23.406 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_recv_encrypted_bytes": 1507840, 2024-03-31T23:37:23.406 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_send_encrypted_bytes": 282512 2024-03-31T23:37:23.406 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:23.406 INFO:teuthology.orchestra.run.smithi046.stdout: "cct": { 2024-03-31T23:37:23.406 INFO:teuthology.orchestra.run.smithi046.stdout: "total_workers": 0, 2024-03-31T23:37:23.406 INFO:teuthology.orchestra.run.smithi046.stdout: "unhealthy_workers": 0 2024-03-31T23:37:23.407 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:23.407 INFO:teuthology.orchestra.run.smithi046.stdout: "mempool": { 2024-03-31T23:37:23.407 INFO:teuthology.orchestra.run.smithi046.stdout: "bloom_filter_bytes": 0, 2024-03-31T23:37:23.407 INFO:teuthology.orchestra.run.smithi046.stdout: "bloom_filter_items": 0, 2024-03-31T23:37:23.407 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_alloc_bytes": 0, 2024-03-31T23:37:23.407 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_alloc_items": 0, 2024-03-31T23:37:23.407 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_cache_data_bytes": 0, 2024-03-31T23:37:23.407 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_cache_data_items": 0, 2024-03-31T23:37:23.407 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-31T23:37:23.407 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_cache_onode_items": 0, 2024-03-31T23:37:23.407 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-31T23:37:23.407 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_cache_meta_items": 0, 2024-03-31T23:37:23.407 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_cache_other_bytes": 0, 2024-03-31T23:37:23.407 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_cache_other_items": 0, 2024-03-31T23:37:23.408 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-31T23:37:23.408 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_cache_buffer_items": 0, 2024-03-31T23:37:23.408 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_extent_bytes": 0, 2024-03-31T23:37:23.408 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_extent_items": 0, 2024-03-31T23:37:23.408 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_blob_bytes": 0, 2024-03-31T23:37:23.408 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_blob_items": 0, 2024-03-31T23:37:23.408 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-31T23:37:23.408 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_shared_blob_items": 0, 2024-03-31T23:37:23.408 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-31T23:37:23.408 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_inline_bl_items": 0, 2024-03-31T23:37:23.408 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_fsck_bytes": 0, 2024-03-31T23:37:23.408 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_fsck_items": 0, 2024-03-31T23:37:23.408 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_txc_bytes": 0, 2024-03-31T23:37:23.408 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_txc_items": 0, 2024-03-31T23:37:23.409 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-31T23:37:23.409 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_writing_deferred_items": 0, 2024-03-31T23:37:23.409 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_writing_bytes": 0, 2024-03-31T23:37:23.409 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_writing_items": 0, 2024-03-31T23:37:23.409 INFO:teuthology.orchestra.run.smithi046.stdout: "bluefs_bytes": 0, 2024-03-31T23:37:23.409 INFO:teuthology.orchestra.run.smithi046.stdout: "bluefs_items": 0, 2024-03-31T23:37:23.409 INFO:teuthology.orchestra.run.smithi046.stdout: "bluefs_file_reader_bytes": 0, 2024-03-31T23:37:23.409 INFO:teuthology.orchestra.run.smithi046.stdout: "bluefs_file_reader_items": 0, 2024-03-31T23:37:23.409 INFO:teuthology.orchestra.run.smithi046.stdout: "bluefs_file_writer_bytes": 0, 2024-03-31T23:37:23.409 INFO:teuthology.orchestra.run.smithi046.stdout: "bluefs_file_writer_items": 0, 2024-03-31T23:37:23.409 INFO:teuthology.orchestra.run.smithi046.stdout: "buffer_anon_bytes": 148731, 2024-03-31T23:37:23.409 INFO:teuthology.orchestra.run.smithi046.stdout: "buffer_anon_items": 31, 2024-03-31T23:37:23.409 INFO:teuthology.orchestra.run.smithi046.stdout: "buffer_meta_bytes": 0, 2024-03-31T23:37:23.410 INFO:teuthology.orchestra.run.smithi046.stdout: "buffer_meta_items": 0, 2024-03-31T23:37:23.410 INFO:teuthology.orchestra.run.smithi046.stdout: "osd_bytes": 0, 2024-03-31T23:37:23.410 INFO:teuthology.orchestra.run.smithi046.stdout: "osd_items": 0, 2024-03-31T23:37:23.410 INFO:teuthology.orchestra.run.smithi046.stdout: "osd_mapbl_bytes": 0, 2024-03-31T23:37:23.410 INFO:teuthology.orchestra.run.smithi046.stdout: "osd_mapbl_items": 0, 2024-03-31T23:37:23.410 INFO:teuthology.orchestra.run.smithi046.stdout: "osd_pglog_bytes": 0, 2024-03-31T23:37:23.410 INFO:teuthology.orchestra.run.smithi046.stdout: "osd_pglog_items": 0, 2024-03-31T23:37:23.410 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap_bytes": 0, 2024-03-31T23:37:23.410 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap_items": 0, 2024-03-31T23:37:23.410 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap_mapping_bytes": 0, 2024-03-31T23:37:23.410 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap_mapping_items": 0, 2024-03-31T23:37:23.410 INFO:teuthology.orchestra.run.smithi046.stdout: "pgmap_bytes": 0, 2024-03-31T23:37:23.410 INFO:teuthology.orchestra.run.smithi046.stdout: "pgmap_items": 0, 2024-03-31T23:37:23.411 INFO:teuthology.orchestra.run.smithi046.stdout: "mds_co_bytes": 928, 2024-03-31T23:37:23.411 INFO:teuthology.orchestra.run.smithi046.stdout: "mds_co_items": 2, 2024-03-31T23:37:23.411 INFO:teuthology.orchestra.run.smithi046.stdout: "unittest_1_bytes": 0, 2024-03-31T23:37:23.411 INFO:teuthology.orchestra.run.smithi046.stdout: "unittest_1_items": 0, 2024-03-31T23:37:23.411 INFO:teuthology.orchestra.run.smithi046.stdout: "unittest_2_bytes": 0, 2024-03-31T23:37:23.411 INFO:teuthology.orchestra.run.smithi046.stdout: "unittest_2_items": 0 2024-03-31T23:37:23.411 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-31T23:37:23.411 INFO:teuthology.orchestra.run.smithi046.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-31T23:37:23.411 INFO:teuthology.orchestra.run.smithi046.stdout: "val": 0, 2024-03-31T23:37:23.411 INFO:teuthology.orchestra.run.smithi046.stdout: "max": 104857600, 2024-03-31T23:37:23.411 INFO:teuthology.orchestra.run.smithi046.stdout: "get_started": 0, 2024-03-31T23:37:23.411 INFO:teuthology.orchestra.run.smithi046.stdout: "get": 469, 2024-03-31T23:37:23.411 INFO:teuthology.orchestra.run.smithi046.stdout: "get_sum": 1442005, 2024-03-31T23:37:23.411 INFO:teuthology.orchestra.run.smithi046.stdout: "get_or_fail_fail": 0, 2024-03-31T23:37:23.411 INFO:teuthology.orchestra.run.smithi046.stdout: "get_or_fail_success": 469, 2024-03-31T23:37:23.412 INFO:teuthology.orchestra.run.smithi046.stdout: "take": 0, 2024-03-31T23:37:23.412 INFO:teuthology.orchestra.run.smithi046.stdout: "take_sum": 0, 2024-03-31T23:37:23.412 INFO:teuthology.orchestra.run.smithi046.stdout: "put": 469, 2024-03-31T23:37:23.412 INFO:teuthology.orchestra.run.smithi046.stdout: "put_sum": 1442005, 2024-03-31T23:37:23.412 INFO:teuthology.orchestra.run.smithi046.stdout: "wait": { 2024-03-31T23:37:23.412 INFO:teuthology.orchestra.run.smithi046.stdout: "avgcount": 0, 2024-03-31T23:37:23.412 INFO:teuthology.orchestra.run.smithi046.stdout: "sum": 0.000000000, 2024-03-31T23:37:23.412 INFO:teuthology.orchestra.run.smithi046.stdout: "avgtime": 0.000000000 2024-03-31T23:37:23.412 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-31T23:37:23.412 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-31T23:37:23.412 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-03-31T23:37:24.163 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.j 2024-03-31T23:37:24.163 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.j 2024-03-31T23:37:24.163 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update' not found on daemon mds.j 2024-03-31T23:37:24.163 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update_receipt' not found on daemon mds.j 2024-03-31T23:37:24.163 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.j 2024-03-31T23:37:24.163 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.j 2024-03-31T23:37:24.163 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.j 2024-03-31T23:37:24.163 DEBUG:tasks.check_counter:Getting stats from b 2024-03-31T23:37:24.164 DEBUG:teuthology.orchestra.run.smithi142:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:384f31f3d3aee7f8ba3f60846b8844c049165583 shell --fsid 6a9d8f08-efb2-11ee-b647-cb9ed24678a4 -- ceph daemon mds.b perf dump 2024-03-31T23:37:24.481 INFO:teuthology.orchestra.run.smithi142.stderr:Inferring config /var/lib/ceph/6a9d8f08-efb2-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-31T23:37:24.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:37:24 smithi142 ceph-mon[35271]: pgmap v816: 129 pgs: 129 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 9.2 KiB/s wr, 8 op/s 2024-03-31T23:37:25.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:37:24 smithi162 ceph-mon[36237]: pgmap v816: 129 pgs: 129 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 9.2 KiB/s wr, 8 op/s 2024-03-31T23:37:25.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:37:24 smithi046 ceph-mon[32498]: pgmap v816: 129 pgs: 129 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 9.2 KiB/s wr, 8 op/s 2024-03-31T23:37:26.528 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-03-31T23:37:26.528 INFO:teuthology.orchestra.run.smithi142.stdout: "AsyncMessenger::Worker-0": { 2024-03-31T23:37:26.528 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_recv_messages": 300540, 2024-03-31T23:37:26.528 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_send_messages": 301787, 2024-03-31T23:37:26.528 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_recv_bytes": 113930784, 2024-03-31T23:37:26.528 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_send_bytes": 346570251, 2024-03-31T23:37:26.528 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_created_connections": 266, 2024-03-31T23:37:26.528 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_active_connections": 6, 2024-03-31T23:37:26.529 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_running_total_time": 21.947913710, 2024-03-31T23:37:26.529 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_running_send_time": 10.607845812, 2024-03-31T23:37:26.529 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_running_recv_time": 11.882173747, 2024-03-31T23:37:26.529 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_running_fast_dispatch_time": 1.692063935, 2024-03-31T23:37:26.529 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_send_messages_queue_lat": { 2024-03-31T23:37:26.529 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 301788, 2024-03-31T23:37:26.529 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 24.626989207, 2024-03-31T23:37:26.529 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000081603 2024-03-31T23:37:26.529 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-03-31T23:37:26.529 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_handle_ack_lat": { 2024-03-31T23:37:26.529 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 513070, 2024-03-31T23:37:26.529 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.098029265, 2024-03-31T23:37:26.529 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000000191 2024-03-31T23:37:26.529 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-03-31T23:37:26.530 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_recv_encrypted_bytes": 42224, 2024-03-31T23:37:26.530 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_send_encrypted_bytes": 1216 2024-03-31T23:37:26.530 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-03-31T23:37:26.530 INFO:teuthology.orchestra.run.smithi142.stdout: "AsyncMessenger::Worker-1": { 2024-03-31T23:37:26.530 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_recv_messages": 386656, 2024-03-31T23:37:26.530 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_send_messages": 425592, 2024-03-31T23:37:26.530 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_recv_bytes": 115396127, 2024-03-31T23:37:26.530 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_send_bytes": 280374032, 2024-03-31T23:37:26.530 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_created_connections": 12, 2024-03-31T23:37:26.530 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_active_connections": 5, 2024-03-31T23:37:26.530 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_running_total_time": 21.906985983, 2024-03-31T23:37:26.530 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_running_send_time": 8.657117627, 2024-03-31T23:37:26.530 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_running_recv_time": 14.054690244, 2024-03-31T23:37:26.531 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_running_fast_dispatch_time": 1.406800504, 2024-03-31T23:37:26.531 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_send_messages_queue_lat": { 2024-03-31T23:37:26.531 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 425592, 2024-03-31T23:37:26.531 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 38.829005820, 2024-03-31T23:37:26.531 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000091235 2024-03-31T23:37:26.531 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-03-31T23:37:26.531 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_handle_ack_lat": { 2024-03-31T23:37:26.531 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 732146, 2024-03-31T23:37:26.531 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.138141499, 2024-03-31T23:37:26.531 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000000188 2024-03-31T23:37:26.531 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-03-31T23:37:26.531 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_recv_encrypted_bytes": 42704, 2024-03-31T23:37:26.531 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_send_encrypted_bytes": 874816 2024-03-31T23:37:26.531 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-03-31T23:37:26.531 INFO:teuthology.orchestra.run.smithi142.stdout: "AsyncMessenger::Worker-2": { 2024-03-31T23:37:26.531 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_recv_messages": 1188721, 2024-03-31T23:37:26.532 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_send_messages": 1258320, 2024-03-31T23:37:26.532 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_recv_bytes": 354942818, 2024-03-31T23:37:26.532 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_send_bytes": 915350700, 2024-03-31T23:37:26.532 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_created_connections": 20, 2024-03-31T23:37:26.532 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_active_connections": 9, 2024-03-31T23:37:26.532 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_running_total_time": 58.535118061, 2024-03-31T23:37:26.532 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_running_send_time": 25.028357864, 2024-03-31T23:37:26.532 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_running_recv_time": 39.447151398, 2024-03-31T23:37:26.532 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_running_fast_dispatch_time": 1.641350037, 2024-03-31T23:37:26.532 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_send_messages_queue_lat": { 2024-03-31T23:37:26.532 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 1258314, 2024-03-31T23:37:26.532 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 56.990394978, 2024-03-31T23:37:26.532 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000045291 2024-03-31T23:37:26.533 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-03-31T23:37:26.533 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_handle_ack_lat": { 2024-03-31T23:37:26.533 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 2345623, 2024-03-31T23:37:26.533 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.380105226, 2024-03-31T23:37:26.533 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000000162 2024-03-31T23:37:26.533 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-03-31T23:37:26.533 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_recv_encrypted_bytes": 2610784, 2024-03-31T23:37:26.533 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_send_encrypted_bytes": 610960 2024-03-31T23:37:26.533 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-03-31T23:37:26.533 INFO:teuthology.orchestra.run.smithi142.stdout: "cct": { 2024-03-31T23:37:26.533 INFO:teuthology.orchestra.run.smithi142.stdout: "total_workers": 1, 2024-03-31T23:37:26.533 INFO:teuthology.orchestra.run.smithi142.stdout: "unhealthy_workers": 0 2024-03-31T23:37:26.533 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-03-31T23:37:26.534 INFO:teuthology.orchestra.run.smithi142.stdout: "finisher-MDSRank": { 2024-03-31T23:37:26.534 INFO:teuthology.orchestra.run.smithi142.stdout: "queue_len": 0, 2024-03-31T23:37:26.534 INFO:teuthology.orchestra.run.smithi142.stdout: "complete_latency": { 2024-03-31T23:37:26.534 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 103213, 2024-03-31T23:37:26.534 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 123.673979841, 2024-03-31T23:37:26.534 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.001198240 2024-03-31T23:37:26.534 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-03-31T23:37:26.534 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-03-31T23:37:26.534 INFO:teuthology.orchestra.run.smithi142.stdout: "finisher-PurgeQueue": { 2024-03-31T23:37:26.534 INFO:teuthology.orchestra.run.smithi142.stdout: "queue_len": 0, 2024-03-31T23:37:26.534 INFO:teuthology.orchestra.run.smithi142.stdout: "complete_latency": { 2024-03-31T23:37:26.534 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 26938, 2024-03-31T23:37:26.535 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 5.807673019, 2024-03-31T23:37:26.535 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000215594 2024-03-31T23:37:26.535 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-03-31T23:37:26.535 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-03-31T23:37:26.535 INFO:teuthology.orchestra.run.smithi142.stdout: "mds": { 2024-03-31T23:37:26.535 INFO:teuthology.orchestra.run.smithi142.stdout: "request": 1073605, 2024-03-31T23:37:26.535 INFO:teuthology.orchestra.run.smithi142.stdout: "reply": 1039868, 2024-03-31T23:37:26.535 INFO:teuthology.orchestra.run.smithi142.stdout: "reply_latency": { 2024-03-31T23:37:26.535 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 1039868, 2024-03-31T23:37:26.535 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 1823.726007182, 2024-03-31T23:37:26.535 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.001753805 2024-03-31T23:37:26.535 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-03-31T23:37:26.535 INFO:teuthology.orchestra.run.smithi142.stdout: "slow_reply": 0, 2024-03-31T23:37:26.535 INFO:teuthology.orchestra.run.smithi142.stdout: "forward": 33737, 2024-03-31T23:37:26.535 INFO:teuthology.orchestra.run.smithi142.stdout: "dir_fetch_complete": 30, 2024-03-31T23:37:26.536 INFO:teuthology.orchestra.run.smithi142.stdout: "dir_fetch_keys": 1, 2024-03-31T23:37:26.536 INFO:teuthology.orchestra.run.smithi142.stdout: "dir_commit": 5389, 2024-03-31T23:37:26.536 INFO:teuthology.orchestra.run.smithi142.stdout: "dir_split": 9, 2024-03-31T23:37:26.536 INFO:teuthology.orchestra.run.smithi142.stdout: "dir_merge": 0, 2024-03-31T23:37:26.536 INFO:teuthology.orchestra.run.smithi142.stdout: "inodes": 1797, 2024-03-31T23:37:26.536 INFO:teuthology.orchestra.run.smithi142.stdout: "inodes_top": 921, 2024-03-31T23:37:26.536 INFO:teuthology.orchestra.run.smithi142.stdout: "inodes_bottom": 793, 2024-03-31T23:37:26.536 INFO:teuthology.orchestra.run.smithi142.stdout: "inodes_pin_tail": 83, 2024-03-31T23:37:26.536 INFO:teuthology.orchestra.run.smithi142.stdout: "inodes_pinned": 481, 2024-03-31T23:37:26.536 INFO:teuthology.orchestra.run.smithi142.stdout: "inodes_expired": 4591, 2024-03-31T23:37:26.536 INFO:teuthology.orchestra.run.smithi142.stdout: "inodes_with_caps": 2, 2024-03-31T23:37:26.536 INFO:teuthology.orchestra.run.smithi142.stdout: "caps": 3, 2024-03-31T23:37:26.536 INFO:teuthology.orchestra.run.smithi142.stdout: "subtrees": 4, 2024-03-31T23:37:26.536 INFO:teuthology.orchestra.run.smithi142.stdout: "traverse": 2087831, 2024-03-31T23:37:26.536 INFO:teuthology.orchestra.run.smithi142.stdout: "traverse_hit": 1994387, 2024-03-31T23:37:26.536 INFO:teuthology.orchestra.run.smithi142.stdout: "traverse_forward": 12033, 2024-03-31T23:37:26.537 INFO:teuthology.orchestra.run.smithi142.stdout: "traverse_discover": 272, 2024-03-31T23:37:26.537 INFO:teuthology.orchestra.run.smithi142.stdout: "traverse_dir_fetch": 11, 2024-03-31T23:37:26.537 INFO:teuthology.orchestra.run.smithi142.stdout: "traverse_remote_ino": 0, 2024-03-31T23:37:26.537 INFO:teuthology.orchestra.run.smithi142.stdout: "traverse_lock": 492, 2024-03-31T23:37:26.537 INFO:teuthology.orchestra.run.smithi142.stdout: "load_cent": 57, 2024-03-31T23:37:26.537 INFO:teuthology.orchestra.run.smithi142.stdout: "q": 0, 2024-03-31T23:37:26.537 INFO:teuthology.orchestra.run.smithi142.stdout: "exported": 84, 2024-03-31T23:37:26.537 INFO:teuthology.orchestra.run.smithi142.stdout: "exported_inodes": 3912, 2024-03-31T23:37:26.537 INFO:teuthology.orchestra.run.smithi142.stdout: "imported": 62, 2024-03-31T23:37:26.537 INFO:teuthology.orchestra.run.smithi142.stdout: "imported_inodes": 1394, 2024-03-31T23:37:26.537 INFO:teuthology.orchestra.run.smithi142.stdout: "openino_dir_fetch": 0, 2024-03-31T23:37:26.537 INFO:teuthology.orchestra.run.smithi142.stdout: "openino_backtrace_fetch": 65, 2024-03-31T23:37:26.537 INFO:teuthology.orchestra.run.smithi142.stdout: "openino_peer_discover": 0, 2024-03-31T23:37:26.538 INFO:teuthology.orchestra.run.smithi142.stdout: "root_rfiles": 4, 2024-03-31T23:37:26.538 INFO:teuthology.orchestra.run.smithi142.stdout: "root_rbytes": 234, 2024-03-31T23:37:26.538 INFO:teuthology.orchestra.run.smithi142.stdout: "root_rsnaps": 6, 2024-03-31T23:37:26.538 INFO:teuthology.orchestra.run.smithi142.stdout: "scrub_backtrace_fetch": 65, 2024-03-31T23:37:26.538 INFO:teuthology.orchestra.run.smithi142.stdout: "scrub_set_tag": 0, 2024-03-31T23:37:26.538 INFO:teuthology.orchestra.run.smithi142.stdout: "scrub_backtrace_repaired": 0, 2024-03-31T23:37:26.538 INFO:teuthology.orchestra.run.smithi142.stdout: "scrub_inotable_repaired": 0, 2024-03-31T23:37:26.538 INFO:teuthology.orchestra.run.smithi142.stdout: "scrub_dir_inodes": 44, 2024-03-31T23:37:26.538 INFO:teuthology.orchestra.run.smithi142.stdout: "scrub_dir_base_inodes": 4, 2024-03-31T23:37:26.538 INFO:teuthology.orchestra.run.smithi142.stdout: "scrub_dirfrag_rstats": 40, 2024-03-31T23:37:26.538 INFO:teuthology.orchestra.run.smithi142.stdout: "scrub_file_inodes": 21, 2024-03-31T23:37:26.539 INFO:teuthology.orchestra.run.smithi142.stdout: "handle_inode_file_caps": 46, 2024-03-31T23:37:26.539 INFO:teuthology.orchestra.run.smithi142.stdout: "ceph_cap_op_revoke": 52545, 2024-03-31T23:37:26.539 INFO:teuthology.orchestra.run.smithi142.stdout: "ceph_cap_op_grant": 46014, 2024-03-31T23:37:26.539 INFO:teuthology.orchestra.run.smithi142.stdout: "ceph_cap_op_trunc": 0, 2024-03-31T23:37:26.539 INFO:teuthology.orchestra.run.smithi142.stdout: "ceph_cap_op_flushsnap_ack": 5, 2024-03-31T23:37:26.539 INFO:teuthology.orchestra.run.smithi142.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-31T23:37:26.539 INFO:teuthology.orchestra.run.smithi142.stdout: "handle_client_caps": 81101, 2024-03-31T23:37:26.539 INFO:teuthology.orchestra.run.smithi142.stdout: "handle_client_caps_dirty": 21150, 2024-03-31T23:37:26.539 INFO:teuthology.orchestra.run.smithi142.stdout: "handle_client_cap_release": 833, 2024-03-31T23:37:26.539 INFO:teuthology.orchestra.run.smithi142.stdout: "process_request_cap_release": 56461 2024-03-31T23:37:26.539 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-03-31T23:37:26.539 INFO:teuthology.orchestra.run.smithi142.stdout: "mds_cache": { 2024-03-31T23:37:26.539 INFO:teuthology.orchestra.run.smithi142.stdout: "dir_update": 2774, 2024-03-31T23:37:26.539 INFO:teuthology.orchestra.run.smithi142.stdout: "dir_update_receipt": 1046, 2024-03-31T23:37:26.539 INFO:teuthology.orchestra.run.smithi142.stdout: "dir_try_discover": 558, 2024-03-31T23:37:26.540 INFO:teuthology.orchestra.run.smithi142.stdout: "dir_send_discover": 1209, 2024-03-31T23:37:26.540 INFO:teuthology.orchestra.run.smithi142.stdout: "dir_handle_discover": 4891, 2024-03-31T23:37:26.540 INFO:teuthology.orchestra.run.smithi142.stdout: "num_strays": 33, 2024-03-31T23:37:26.540 INFO:teuthology.orchestra.run.smithi142.stdout: "num_strays_delayed": 0, 2024-03-31T23:37:26.540 INFO:teuthology.orchestra.run.smithi142.stdout: "num_strays_enqueuing": 0, 2024-03-31T23:37:26.540 INFO:teuthology.orchestra.run.smithi142.stdout: "strays_created": 15752, 2024-03-31T23:37:26.540 INFO:teuthology.orchestra.run.smithi142.stdout: "strays_enqueued": 15719, 2024-03-31T23:37:26.540 INFO:teuthology.orchestra.run.smithi142.stdout: "strays_reintegrated": 0, 2024-03-31T23:37:26.540 INFO:teuthology.orchestra.run.smithi142.stdout: "strays_migrated": 0, 2024-03-31T23:37:26.540 INFO:teuthology.orchestra.run.smithi142.stdout: "num_recovering_processing": 0, 2024-03-31T23:37:26.540 INFO:teuthology.orchestra.run.smithi142.stdout: "num_recovering_enqueued": 0, 2024-03-31T23:37:26.540 INFO:teuthology.orchestra.run.smithi142.stdout: "num_recovering_prioritized": 0, 2024-03-31T23:37:26.540 INFO:teuthology.orchestra.run.smithi142.stdout: "recovery_started": 0, 2024-03-31T23:37:26.540 INFO:teuthology.orchestra.run.smithi142.stdout: "recovery_completed": 0, 2024-03-31T23:37:26.540 INFO:teuthology.orchestra.run.smithi142.stdout: "ireq_quiesce_path": 0, 2024-03-31T23:37:26.540 INFO:teuthology.orchestra.run.smithi142.stdout: "ireq_quiesce_inode": 0, 2024-03-31T23:37:26.541 INFO:teuthology.orchestra.run.smithi142.stdout: "ireq_enqueue_scrub": 4, 2024-03-31T23:37:26.541 INFO:teuthology.orchestra.run.smithi142.stdout: "ireq_exportdir": 99, 2024-03-31T23:37:26.541 INFO:teuthology.orchestra.run.smithi142.stdout: "ireq_flush": 0, 2024-03-31T23:37:26.541 INFO:teuthology.orchestra.run.smithi142.stdout: "ireq_fragmentdir": 9, 2024-03-31T23:37:26.541 INFO:teuthology.orchestra.run.smithi142.stdout: "ireq_fragstats": 0, 2024-03-31T23:37:26.541 INFO:teuthology.orchestra.run.smithi142.stdout: "ireq_inodestats": 0 2024-03-31T23:37:26.541 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-03-31T23:37:26.541 INFO:teuthology.orchestra.run.smithi142.stdout: "mds_log": { 2024-03-31T23:37:26.541 INFO:teuthology.orchestra.run.smithi142.stdout: "evlrg": 0, 2024-03-31T23:37:26.541 INFO:teuthology.orchestra.run.smithi142.stdout: "evadd": 161275, 2024-03-31T23:37:26.541 INFO:teuthology.orchestra.run.smithi142.stdout: "evex": 158076, 2024-03-31T23:37:26.541 INFO:teuthology.orchestra.run.smithi142.stdout: "evtrm": 158076, 2024-03-31T23:37:26.541 INFO:teuthology.orchestra.run.smithi142.stdout: "ev": 3199, 2024-03-31T23:37:26.541 INFO:teuthology.orchestra.run.smithi142.stdout: "evexg": 0, 2024-03-31T23:37:26.541 INFO:teuthology.orchestra.run.smithi142.stdout: "evexd": 2394, 2024-03-31T23:37:26.542 INFO:teuthology.orchestra.run.smithi142.stdout: "segadd": 304, 2024-03-31T23:37:26.542 INFO:teuthology.orchestra.run.smithi142.stdout: "segex": 299, 2024-03-31T23:37:26.542 INFO:teuthology.orchestra.run.smithi142.stdout: "segtrm": 299, 2024-03-31T23:37:26.542 INFO:teuthology.orchestra.run.smithi142.stdout: "seg": 5, 2024-03-31T23:37:26.542 INFO:teuthology.orchestra.run.smithi142.stdout: "segmjr": 1, 2024-03-31T23:37:26.542 INFO:teuthology.orchestra.run.smithi142.stdout: "segexg": 0, 2024-03-31T23:37:26.542 INFO:teuthology.orchestra.run.smithi142.stdout: "segexd": 4, 2024-03-31T23:37:26.542 INFO:teuthology.orchestra.run.smithi142.stdout: "expos": 482989945, 2024-03-31T23:37:26.542 INFO:teuthology.orchestra.run.smithi142.stdout: "wrpos": 491098550, 2024-03-31T23:37:26.542 INFO:teuthology.orchestra.run.smithi142.stdout: "rdpos": 0, 2024-03-31T23:37:26.542 INFO:teuthology.orchestra.run.smithi142.stdout: "jlat": { 2024-03-31T23:37:26.542 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 52220, 2024-03-31T23:37:26.542 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 256.404520752, 2024-03-31T23:37:26.542 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.004910082 2024-03-31T23:37:26.542 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-03-31T23:37:26.543 INFO:teuthology.orchestra.run.smithi142.stdout: "replayed": 0 2024-03-31T23:37:26.543 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-03-31T23:37:26.543 INFO:teuthology.orchestra.run.smithi142.stdout: "mds_mem": { 2024-03-31T23:37:26.543 INFO:teuthology.orchestra.run.smithi142.stdout: "ino": 1436, 2024-03-31T23:37:26.543 INFO:teuthology.orchestra.run.smithi142.stdout: "ino+": 21132, 2024-03-31T23:37:26.543 INFO:teuthology.orchestra.run.smithi142.stdout: "ino-": 19696, 2024-03-31T23:37:26.543 INFO:teuthology.orchestra.run.smithi142.stdout: "dir": 143, 2024-03-31T23:37:26.543 INFO:teuthology.orchestra.run.smithi142.stdout: "dir+": 923, 2024-03-31T23:37:26.543 INFO:teuthology.orchestra.run.smithi142.stdout: "dir-": 780, 2024-03-31T23:37:26.543 INFO:teuthology.orchestra.run.smithi142.stdout: "dn": 1972, 2024-03-31T23:37:26.543 INFO:teuthology.orchestra.run.smithi142.stdout: "dn+": 39763, 2024-03-31T23:37:26.543 INFO:teuthology.orchestra.run.smithi142.stdout: "dn-": 37791, 2024-03-31T23:37:26.543 INFO:teuthology.orchestra.run.smithi142.stdout: "cap": 3, 2024-03-31T23:37:26.543 INFO:teuthology.orchestra.run.smithi142.stdout: "cap+": 16538, 2024-03-31T23:37:26.543 INFO:teuthology.orchestra.run.smithi142.stdout: "cap-": 16535, 2024-03-31T23:37:26.544 INFO:teuthology.orchestra.run.smithi142.stdout: "rss": 88728, 2024-03-31T23:37:26.544 INFO:teuthology.orchestra.run.smithi142.stdout: "heap": 231804 2024-03-31T23:37:26.544 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-03-31T23:37:26.544 INFO:teuthology.orchestra.run.smithi142.stdout: "mds_server": { 2024-03-31T23:37:26.544 INFO:teuthology.orchestra.run.smithi142.stdout: "dispatch_client_request": 1238578, 2024-03-31T23:37:26.544 INFO:teuthology.orchestra.run.smithi142.stdout: "dispatch_server_request": 19931, 2024-03-31T23:37:26.544 INFO:teuthology.orchestra.run.smithi142.stdout: "handle_client_request": 1073605, 2024-03-31T23:37:26.544 INFO:teuthology.orchestra.run.smithi142.stdout: "handle_client_session": 1127, 2024-03-31T23:37:26.544 INFO:teuthology.orchestra.run.smithi142.stdout: "handle_peer_request": 72772, 2024-03-31T23:37:26.544 INFO:teuthology.orchestra.run.smithi142.stdout: "req_create_latency": { 2024-03-31T23:37:26.544 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 16104, 2024-03-31T23:37:26.544 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 102.729721804, 2024-03-31T23:37:26.544 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.006379143 2024-03-31T23:37:26.544 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-03-31T23:37:26.545 INFO:teuthology.orchestra.run.smithi142.stdout: "req_getattr_latency": { 2024-03-31T23:37:26.545 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 887104, 2024-03-31T23:37:26.545 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 716.501946151, 2024-03-31T23:37:26.545 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000807686 2024-03-31T23:37:26.545 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-03-31T23:37:26.545 INFO:teuthology.orchestra.run.smithi142.stdout: "req_getfilelock_latency": { 2024-03-31T23:37:26.545 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 0, 2024-03-31T23:37:26.545 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.000000000, 2024-03-31T23:37:26.545 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000000000 2024-03-31T23:37:26.545 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-03-31T23:37:26.545 INFO:teuthology.orchestra.run.smithi142.stdout: "req_link_latency": { 2024-03-31T23:37:26.545 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 0, 2024-03-31T23:37:26.545 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.000000000, 2024-03-31T23:37:26.545 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000000000 2024-03-31T23:37:26.545 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-03-31T23:37:26.546 INFO:teuthology.orchestra.run.smithi142.stdout: "req_lookup_latency": { 2024-03-31T23:37:26.546 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 60675, 2024-03-31T23:37:26.546 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 522.846077635, 2024-03-31T23:37:26.546 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.008617158 2024-03-31T23:37:26.546 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-03-31T23:37:26.546 INFO:teuthology.orchestra.run.smithi142.stdout: "req_lookuphash_latency": { 2024-03-31T23:37:26.546 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 0, 2024-03-31T23:37:26.546 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.000000000, 2024-03-31T23:37:26.546 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000000000 2024-03-31T23:37:26.546 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-03-31T23:37:26.546 INFO:teuthology.orchestra.run.smithi142.stdout: "req_lookupino_latency": { 2024-03-31T23:37:26.546 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 0, 2024-03-31T23:37:26.546 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.000000000, 2024-03-31T23:37:26.546 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000000000 2024-03-31T23:37:26.547 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-03-31T23:37:26.547 INFO:teuthology.orchestra.run.smithi142.stdout: "req_lookupname_latency": { 2024-03-31T23:37:26.547 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 0, 2024-03-31T23:37:26.547 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.000000000, 2024-03-31T23:37:26.547 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000000000 2024-03-31T23:37:26.547 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-03-31T23:37:26.547 INFO:teuthology.orchestra.run.smithi142.stdout: "req_lookupparent_latency": { 2024-03-31T23:37:26.547 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 0, 2024-03-31T23:37:26.547 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.000000000, 2024-03-31T23:37:26.547 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000000000 2024-03-31T23:37:26.547 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-03-31T23:37:26.548 INFO:teuthology.orchestra.run.smithi142.stdout: "req_lookupsnap_latency": { 2024-03-31T23:37:26.548 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 0, 2024-03-31T23:37:26.548 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.000000000, 2024-03-31T23:37:26.548 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000000000 2024-03-31T23:37:26.548 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-03-31T23:37:26.548 INFO:teuthology.orchestra.run.smithi142.stdout: "req_lssnap_latency": { 2024-03-31T23:37:26.548 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 26, 2024-03-31T23:37:26.548 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.015743386, 2024-03-31T23:37:26.548 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000605514 2024-03-31T23:37:26.548 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-03-31T23:37:26.548 INFO:teuthology.orchestra.run.smithi142.stdout: "req_mkdir_latency": { 2024-03-31T23:37:26.548 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 119, 2024-03-31T23:37:26.549 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 1.281601604, 2024-03-31T23:37:26.549 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.010769761 2024-03-31T23:37:26.549 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-03-31T23:37:26.549 INFO:teuthology.orchestra.run.smithi142.stdout: "req_mknod_latency": { 2024-03-31T23:37:26.549 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 0, 2024-03-31T23:37:26.549 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.000000000, 2024-03-31T23:37:26.549 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000000000 2024-03-31T23:37:26.549 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-03-31T23:37:26.549 INFO:teuthology.orchestra.run.smithi142.stdout: "req_mksnap_latency": { 2024-03-31T23:37:26.549 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 26, 2024-03-31T23:37:26.549 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 10.935309345, 2024-03-31T23:37:26.549 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.420588820 2024-03-31T23:37:26.549 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-03-31T23:37:26.549 INFO:teuthology.orchestra.run.smithi142.stdout: "req_open_latency": { 2024-03-31T23:37:26.550 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 10256, 2024-03-31T23:37:26.550 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 19.194829045, 2024-03-31T23:37:26.550 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.001871570 2024-03-31T23:37:26.550 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-03-31T23:37:26.550 INFO:teuthology.orchestra.run.smithi142.stdout: "req_readdir_latency": { 2024-03-31T23:37:26.550 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 46462, 2024-03-31T23:37:26.550 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 170.173449171, 2024-03-31T23:37:26.550 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.003662637 2024-03-31T23:37:26.550 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-03-31T23:37:26.550 INFO:teuthology.orchestra.run.smithi142.stdout: "req_rename_latency": { 2024-03-31T23:37:26.550 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 2927, 2024-03-31T23:37:26.550 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 174.759349507, 2024-03-31T23:37:26.550 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.059705961 2024-03-31T23:37:26.550 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-03-31T23:37:26.550 INFO:teuthology.orchestra.run.smithi142.stdout: "req_renamesnap_latency": { 2024-03-31T23:37:26.551 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 0, 2024-03-31T23:37:26.551 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.000000000, 2024-03-31T23:37:26.551 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000000000 2024-03-31T23:37:26.551 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-03-31T23:37:26.551 INFO:teuthology.orchestra.run.smithi142.stdout: "req_snapdiff_latency": { 2024-03-31T23:37:26.551 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 0, 2024-03-31T23:37:26.551 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.000000000, 2024-03-31T23:37:26.551 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000000000 2024-03-31T23:37:26.551 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-03-31T23:37:26.551 INFO:teuthology.orchestra.run.smithi142.stdout: "req_rmdir_latency": { 2024-03-31T23:37:26.551 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 44, 2024-03-31T23:37:26.551 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.342789444, 2024-03-31T23:37:26.551 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.007790669 2024-03-31T23:37:26.552 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-03-31T23:37:26.552 INFO:teuthology.orchestra.run.smithi142.stdout: "req_rmsnap_latency": { 2024-03-31T23:37:26.552 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 20, 2024-03-31T23:37:26.552 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 5.597291824, 2024-03-31T23:37:26.552 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.279864591 2024-03-31T23:37:26.552 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-03-31T23:37:26.552 INFO:teuthology.orchestra.run.smithi142.stdout: "req_rmxattr_latency": { 2024-03-31T23:37:26.552 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 0, 2024-03-31T23:37:26.552 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.000000000, 2024-03-31T23:37:26.552 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000000000 2024-03-31T23:37:26.552 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-03-31T23:37:26.553 INFO:teuthology.orchestra.run.smithi142.stdout: "req_setattr_latency": { 2024-03-31T23:37:26.553 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 2, 2024-03-31T23:37:26.553 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.001606563, 2024-03-31T23:37:26.553 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000803281 2024-03-31T23:37:26.553 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-03-31T23:37:26.553 INFO:teuthology.orchestra.run.smithi142.stdout: "req_setdirlayout_latency": { 2024-03-31T23:37:26.553 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 0, 2024-03-31T23:37:26.553 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.000000000, 2024-03-31T23:37:26.553 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000000000 2024-03-31T23:37:26.553 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-03-31T23:37:26.553 INFO:teuthology.orchestra.run.smithi142.stdout: "req_setfilelock_latency": { 2024-03-31T23:37:26.553 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 376, 2024-03-31T23:37:26.553 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.487354482, 2024-03-31T23:37:26.553 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.001296155 2024-03-31T23:37:26.553 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-03-31T23:37:26.554 INFO:teuthology.orchestra.run.smithi142.stdout: "req_setlayout_latency": { 2024-03-31T23:37:26.554 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 0, 2024-03-31T23:37:26.554 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.000000000, 2024-03-31T23:37:26.554 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000000000 2024-03-31T23:37:26.554 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-03-31T23:37:26.554 INFO:teuthology.orchestra.run.smithi142.stdout: "req_setxattr_latency": { 2024-03-31T23:37:26.554 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 8, 2024-03-31T23:37:26.554 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.018467426, 2024-03-31T23:37:26.554 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.002308428 2024-03-31T23:37:26.554 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-03-31T23:37:26.554 INFO:teuthology.orchestra.run.smithi142.stdout: "req_symlink_latency": { 2024-03-31T23:37:26.554 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 0, 2024-03-31T23:37:26.554 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.000000000, 2024-03-31T23:37:26.554 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000000000 2024-03-31T23:37:26.555 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-03-31T23:37:26.555 INFO:teuthology.orchestra.run.smithi142.stdout: "req_unlink_latency": { 2024-03-31T23:37:26.555 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 15719, 2024-03-31T23:37:26.555 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 98.840469795, 2024-03-31T23:37:26.555 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.006287961 2024-03-31T23:37:26.555 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-03-31T23:37:26.555 INFO:teuthology.orchestra.run.smithi142.stdout: "cap_revoke_eviction": 0, 2024-03-31T23:37:26.555 INFO:teuthology.orchestra.run.smithi142.stdout: "cap_acquisition_throttle": 0, 2024-03-31T23:37:26.555 INFO:teuthology.orchestra.run.smithi142.stdout: "req_getvxattr_latency": { 2024-03-31T23:37:26.555 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 0, 2024-03-31T23:37:26.555 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.000000000, 2024-03-31T23:37:26.555 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000000000 2024-03-31T23:37:26.555 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-03-31T23:37:26.555 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-03-31T23:37:26.555 INFO:teuthology.orchestra.run.smithi142.stdout: "mds_sessions": { 2024-03-31T23:37:26.556 INFO:teuthology.orchestra.run.smithi142.stdout: "session_count": 2, 2024-03-31T23:37:26.556 INFO:teuthology.orchestra.run.smithi142.stdout: "session_add": 6, 2024-03-31T23:37:26.556 INFO:teuthology.orchestra.run.smithi142.stdout: "session_remove": 4, 2024-03-31T23:37:26.556 INFO:teuthology.orchestra.run.smithi142.stdout: "sessions_open": 2, 2024-03-31T23:37:26.556 INFO:teuthology.orchestra.run.smithi142.stdout: "sessions_stale": 0, 2024-03-31T23:37:26.556 INFO:teuthology.orchestra.run.smithi142.stdout: "total_load": 67240, 2024-03-31T23:37:26.556 INFO:teuthology.orchestra.run.smithi142.stdout: "average_load": 33620, 2024-03-31T23:37:26.556 INFO:teuthology.orchestra.run.smithi142.stdout: "avg_session_uptime": 1538, 2024-03-31T23:37:26.556 INFO:teuthology.orchestra.run.smithi142.stdout: "mdthresh_evicted": 0 2024-03-31T23:37:26.556 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-03-31T23:37:26.556 INFO:teuthology.orchestra.run.smithi142.stdout: "mempool": { 2024-03-31T23:37:26.556 INFO:teuthology.orchestra.run.smithi142.stdout: "bloom_filter_bytes": 240, 2024-03-31T23:37:26.556 INFO:teuthology.orchestra.run.smithi142.stdout: "bloom_filter_items": 240, 2024-03-31T23:37:26.556 INFO:teuthology.orchestra.run.smithi142.stdout: "bluestore_alloc_bytes": 0, 2024-03-31T23:37:26.556 INFO:teuthology.orchestra.run.smithi142.stdout: "bluestore_alloc_items": 0, 2024-03-31T23:37:26.557 INFO:teuthology.orchestra.run.smithi142.stdout: "bluestore_cache_data_bytes": 0, 2024-03-31T23:37:26.557 INFO:teuthology.orchestra.run.smithi142.stdout: "bluestore_cache_data_items": 0, 2024-03-31T23:37:26.557 INFO:teuthology.orchestra.run.smithi142.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-31T23:37:26.557 INFO:teuthology.orchestra.run.smithi142.stdout: "bluestore_cache_onode_items": 0, 2024-03-31T23:37:26.557 INFO:teuthology.orchestra.run.smithi142.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-31T23:37:26.557 INFO:teuthology.orchestra.run.smithi142.stdout: "bluestore_cache_meta_items": 0, 2024-03-31T23:37:26.557 INFO:teuthology.orchestra.run.smithi142.stdout: "bluestore_cache_other_bytes": 0, 2024-03-31T23:37:26.557 INFO:teuthology.orchestra.run.smithi142.stdout: "bluestore_cache_other_items": 0, 2024-03-31T23:37:26.557 INFO:teuthology.orchestra.run.smithi142.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-31T23:37:26.557 INFO:teuthology.orchestra.run.smithi142.stdout: "bluestore_cache_buffer_items": 0, 2024-03-31T23:37:26.557 INFO:teuthology.orchestra.run.smithi142.stdout: "bluestore_extent_bytes": 0, 2024-03-31T23:37:26.557 INFO:teuthology.orchestra.run.smithi142.stdout: "bluestore_extent_items": 0, 2024-03-31T23:37:26.557 INFO:teuthology.orchestra.run.smithi142.stdout: "bluestore_blob_bytes": 0, 2024-03-31T23:37:26.557 INFO:teuthology.orchestra.run.smithi142.stdout: "bluestore_blob_items": 0, 2024-03-31T23:37:26.558 INFO:teuthology.orchestra.run.smithi142.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-31T23:37:26.558 INFO:teuthology.orchestra.run.smithi142.stdout: "bluestore_shared_blob_items": 0, 2024-03-31T23:37:26.558 INFO:teuthology.orchestra.run.smithi142.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-31T23:37:26.558 INFO:teuthology.orchestra.run.smithi142.stdout: "bluestore_inline_bl_items": 0, 2024-03-31T23:37:26.558 INFO:teuthology.orchestra.run.smithi142.stdout: "bluestore_fsck_bytes": 0, 2024-03-31T23:37:26.558 INFO:teuthology.orchestra.run.smithi142.stdout: "bluestore_fsck_items": 0, 2024-03-31T23:37:26.558 INFO:teuthology.orchestra.run.smithi142.stdout: "bluestore_txc_bytes": 0, 2024-03-31T23:37:26.558 INFO:teuthology.orchestra.run.smithi142.stdout: "bluestore_txc_items": 0, 2024-03-31T23:37:26.558 INFO:teuthology.orchestra.run.smithi142.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-31T23:37:26.558 INFO:teuthology.orchestra.run.smithi142.stdout: "bluestore_writing_deferred_items": 0, 2024-03-31T23:37:26.558 INFO:teuthology.orchestra.run.smithi142.stdout: "bluestore_writing_bytes": 0, 2024-03-31T23:37:26.558 INFO:teuthology.orchestra.run.smithi142.stdout: "bluestore_writing_items": 0, 2024-03-31T23:37:26.558 INFO:teuthology.orchestra.run.smithi142.stdout: "bluefs_bytes": 0, 2024-03-31T23:37:26.559 INFO:teuthology.orchestra.run.smithi142.stdout: "bluefs_items": 0, 2024-03-31T23:37:26.559 INFO:teuthology.orchestra.run.smithi142.stdout: "bluefs_file_reader_bytes": 0, 2024-03-31T23:37:26.559 INFO:teuthology.orchestra.run.smithi142.stdout: "bluefs_file_reader_items": 0, 2024-03-31T23:37:26.559 INFO:teuthology.orchestra.run.smithi142.stdout: "bluefs_file_writer_bytes": 0, 2024-03-31T23:37:26.559 INFO:teuthology.orchestra.run.smithi142.stdout: "bluefs_file_writer_items": 0, 2024-03-31T23:37:26.559 INFO:teuthology.orchestra.run.smithi142.stdout: "buffer_anon_bytes": 579630, 2024-03-31T23:37:26.559 INFO:teuthology.orchestra.run.smithi142.stdout: "buffer_anon_items": 189, 2024-03-31T23:37:26.559 INFO:teuthology.orchestra.run.smithi142.stdout: "buffer_meta_bytes": 88, 2024-03-31T23:37:26.559 INFO:teuthology.orchestra.run.smithi142.stdout: "buffer_meta_items": 1, 2024-03-31T23:37:26.559 INFO:teuthology.orchestra.run.smithi142.stdout: "osd_bytes": 0, 2024-03-31T23:37:26.559 INFO:teuthology.orchestra.run.smithi142.stdout: "osd_items": 0, 2024-03-31T23:37:26.559 INFO:teuthology.orchestra.run.smithi142.stdout: "osd_mapbl_bytes": 0, 2024-03-31T23:37:26.559 INFO:teuthology.orchestra.run.smithi142.stdout: "osd_mapbl_items": 0, 2024-03-31T23:37:26.559 INFO:teuthology.orchestra.run.smithi142.stdout: "osd_pglog_bytes": 0, 2024-03-31T23:37:26.560 INFO:teuthology.orchestra.run.smithi142.stdout: "osd_pglog_items": 0, 2024-03-31T23:37:26.560 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap_bytes": 9704, 2024-03-31T23:37:26.560 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap_items": 186, 2024-03-31T23:37:26.560 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap_mapping_bytes": 0, 2024-03-31T23:37:26.560 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap_mapping_items": 0, 2024-03-31T23:37:26.560 INFO:teuthology.orchestra.run.smithi142.stdout: "pgmap_bytes": 0, 2024-03-31T23:37:26.560 INFO:teuthology.orchestra.run.smithi142.stdout: "pgmap_items": 0, 2024-03-31T23:37:26.560 INFO:teuthology.orchestra.run.smithi142.stdout: "mds_co_bytes": 5259729, 2024-03-31T23:37:26.560 INFO:teuthology.orchestra.run.smithi142.stdout: "mds_co_items": 28297, 2024-03-31T23:37:26.560 INFO:teuthology.orchestra.run.smithi142.stdout: "unittest_1_bytes": 0, 2024-03-31T23:37:26.560 INFO:teuthology.orchestra.run.smithi142.stdout: "unittest_1_items": 0, 2024-03-31T23:37:26.560 INFO:teuthology.orchestra.run.smithi142.stdout: "unittest_2_bytes": 0, 2024-03-31T23:37:26.560 INFO:teuthology.orchestra.run.smithi142.stdout: "unittest_2_items": 0 2024-03-31T23:37:26.560 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-03-31T23:37:26.560 INFO:teuthology.orchestra.run.smithi142.stdout: "objecter": { 2024-03-31T23:37:26.561 INFO:teuthology.orchestra.run.smithi142.stdout: "op_active": 0, 2024-03-31T23:37:26.561 INFO:teuthology.orchestra.run.smithi142.stdout: "op_laggy": 0, 2024-03-31T23:37:26.561 INFO:teuthology.orchestra.run.smithi142.stdout: "op_send": 105446, 2024-03-31T23:37:26.561 INFO:teuthology.orchestra.run.smithi142.stdout: "op_send_bytes": 514908137, 2024-03-31T23:37:26.561 INFO:teuthology.orchestra.run.smithi142.stdout: "op_resend": 31, 2024-03-31T23:37:26.561 INFO:teuthology.orchestra.run.smithi142.stdout: "op_reply": 105446, 2024-03-31T23:37:26.561 INFO:teuthology.orchestra.run.smithi142.stdout: "op_latency": { 2024-03-31T23:37:26.561 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 105446, 2024-03-31T23:37:26.561 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 1498.760642547, 2024-03-31T23:37:26.561 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.014213537 2024-03-31T23:37:26.561 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-03-31T23:37:26.561 INFO:teuthology.orchestra.run.smithi142.stdout: "op_inflight": 0, 2024-03-31T23:37:26.561 INFO:teuthology.orchestra.run.smithi142.stdout: "oplen_avg": { 2024-03-31T23:37:26.562 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 105446, 2024-03-31T23:37:26.562 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 135262 2024-03-31T23:37:26.562 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-03-31T23:37:26.562 INFO:teuthology.orchestra.run.smithi142.stdout: "op": 105446, 2024-03-31T23:37:26.562 INFO:teuthology.orchestra.run.smithi142.stdout: "op_r": 1601, 2024-03-31T23:37:26.562 INFO:teuthology.orchestra.run.smithi142.stdout: "op_w": 103845, 2024-03-31T23:37:26.562 INFO:teuthology.orchestra.run.smithi142.stdout: "op_rmw": 0, 2024-03-31T23:37:26.562 INFO:teuthology.orchestra.run.smithi142.stdout: "op_pg": 0, 2024-03-31T23:37:26.562 INFO:teuthology.orchestra.run.smithi142.stdout: "osdop_stat": 5206, 2024-03-31T23:37:26.562 INFO:teuthology.orchestra.run.smithi142.stdout: "osdop_create": 10205, 2024-03-31T23:37:26.562 INFO:teuthology.orchestra.run.smithi142.stdout: "osdop_read": 1456, 2024-03-31T23:37:26.562 INFO:teuthology.orchestra.run.smithi142.stdout: "osdop_write": 53672, 2024-03-31T23:37:26.562 INFO:teuthology.orchestra.run.smithi142.stdout: "osdop_writefull": 1257, 2024-03-31T23:37:26.562 INFO:teuthology.orchestra.run.smithi142.stdout: "osdop_writesame": 0, 2024-03-31T23:37:26.563 INFO:teuthology.orchestra.run.smithi142.stdout: "osdop_append": 0, 2024-03-31T23:37:26.563 INFO:teuthology.orchestra.run.smithi142.stdout: "osdop_zero": 0, 2024-03-31T23:37:26.563 INFO:teuthology.orchestra.run.smithi142.stdout: "osdop_truncate": 9, 2024-03-31T23:37:26.563 INFO:teuthology.orchestra.run.smithi142.stdout: "osdop_delete": 32180, 2024-03-31T23:37:26.563 INFO:teuthology.orchestra.run.smithi142.stdout: "osdop_mapext": 0, 2024-03-31T23:37:26.563 INFO:teuthology.orchestra.run.smithi142.stdout: "osdop_sparse_read": 0, 2024-03-31T23:37:26.563 INFO:teuthology.orchestra.run.smithi142.stdout: "osdop_clonerange": 0, 2024-03-31T23:37:26.563 INFO:teuthology.orchestra.run.smithi142.stdout: "osdop_getxattr": 98, 2024-03-31T23:37:26.563 INFO:teuthology.orchestra.run.smithi142.stdout: "osdop_setxattr": 15458, 2024-03-31T23:37:26.563 INFO:teuthology.orchestra.run.smithi142.stdout: "osdop_cmpxattr": 0, 2024-03-31T23:37:26.563 INFO:teuthology.orchestra.run.smithi142.stdout: "osdop_rmxattr": 0, 2024-03-31T23:37:26.563 INFO:teuthology.orchestra.run.smithi142.stdout: "osdop_resetxattrs": 0, 2024-03-31T23:37:26.563 INFO:teuthology.orchestra.run.smithi142.stdout: "osdop_call": 0, 2024-03-31T23:37:26.563 INFO:teuthology.orchestra.run.smithi142.stdout: "osdop_watch": 0, 2024-03-31T23:37:26.564 INFO:teuthology.orchestra.run.smithi142.stdout: "osdop_notify": 0, 2024-03-31T23:37:26.564 INFO:teuthology.orchestra.run.smithi142.stdout: "osdop_src_cmpxattr": 0, 2024-03-31T23:37:26.564 INFO:teuthology.orchestra.run.smithi142.stdout: "osdop_pgls": 0, 2024-03-31T23:37:26.564 INFO:teuthology.orchestra.run.smithi142.stdout: "osdop_pgls_filter": 0, 2024-03-31T23:37:26.564 INFO:teuthology.orchestra.run.smithi142.stdout: "osdop_other": 0, 2024-03-31T23:37:26.564 INFO:teuthology.orchestra.run.smithi142.stdout: "linger_active": 0, 2024-03-31T23:37:26.564 INFO:teuthology.orchestra.run.smithi142.stdout: "linger_send": 0, 2024-03-31T23:37:26.564 INFO:teuthology.orchestra.run.smithi142.stdout: "linger_resend": 0, 2024-03-31T23:37:26.564 INFO:teuthology.orchestra.run.smithi142.stdout: "linger_ping": 0, 2024-03-31T23:37:26.564 INFO:teuthology.orchestra.run.smithi142.stdout: "poolop_active": 0, 2024-03-31T23:37:26.564 INFO:teuthology.orchestra.run.smithi142.stdout: "poolop_send": 0, 2024-03-31T23:37:26.564 INFO:teuthology.orchestra.run.smithi142.stdout: "poolop_resend": 0, 2024-03-31T23:37:26.564 INFO:teuthology.orchestra.run.smithi142.stdout: "poolstat_active": 0, 2024-03-31T23:37:26.564 INFO:teuthology.orchestra.run.smithi142.stdout: "poolstat_send": 0, 2024-03-31T23:37:26.565 INFO:teuthology.orchestra.run.smithi142.stdout: "poolstat_resend": 0, 2024-03-31T23:37:26.565 INFO:teuthology.orchestra.run.smithi142.stdout: "statfs_active": 0, 2024-03-31T23:37:26.565 INFO:teuthology.orchestra.run.smithi142.stdout: "statfs_send": 0, 2024-03-31T23:37:26.565 INFO:teuthology.orchestra.run.smithi142.stdout: "statfs_resend": 0, 2024-03-31T23:37:26.565 INFO:teuthology.orchestra.run.smithi142.stdout: "command_active": 0, 2024-03-31T23:37:26.565 INFO:teuthology.orchestra.run.smithi142.stdout: "command_send": 0, 2024-03-31T23:37:26.565 INFO:teuthology.orchestra.run.smithi142.stdout: "command_resend": 0, 2024-03-31T23:37:26.565 INFO:teuthology.orchestra.run.smithi142.stdout: "map_epoch": 139, 2024-03-31T23:37:26.565 INFO:teuthology.orchestra.run.smithi142.stdout: "map_full": 0, 2024-03-31T23:37:26.565 INFO:teuthology.orchestra.run.smithi142.stdout: "map_inc": 54, 2024-03-31T23:37:26.565 INFO:teuthology.orchestra.run.smithi142.stdout: "osd_sessions": 12, 2024-03-31T23:37:26.565 INFO:teuthology.orchestra.run.smithi142.stdout: "osd_session_open": 12, 2024-03-31T23:37:26.565 INFO:teuthology.orchestra.run.smithi142.stdout: "osd_session_close": 0, 2024-03-31T23:37:26.565 INFO:teuthology.orchestra.run.smithi142.stdout: "osd_laggy": 0, 2024-03-31T23:37:26.565 INFO:teuthology.orchestra.run.smithi142.stdout: "omap_wr": 12348, 2024-03-31T23:37:26.566 INFO:teuthology.orchestra.run.smithi142.stdout: "omap_rd": 107, 2024-03-31T23:37:26.566 INFO:teuthology.orchestra.run.smithi142.stdout: "omap_del": 3266 2024-03-31T23:37:26.566 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-03-31T23:37:26.566 INFO:teuthology.orchestra.run.smithi142.stdout: "oft": { 2024-03-31T23:37:26.566 INFO:teuthology.orchestra.run.smithi142.stdout: "omap_total_objs": 1, 2024-03-31T23:37:26.566 INFO:teuthology.orchestra.run.smithi142.stdout: "omap_total_kv_pairs": 0, 2024-03-31T23:37:26.566 INFO:teuthology.orchestra.run.smithi142.stdout: "omap_total_updates": 0, 2024-03-31T23:37:26.566 INFO:teuthology.orchestra.run.smithi142.stdout: "omap_total_removes": 0 2024-03-31T23:37:26.566 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-03-31T23:37:26.566 INFO:teuthology.orchestra.run.smithi142.stdout: "osd-slow-ops": { 2024-03-31T23:37:26.566 INFO:teuthology.orchestra.run.smithi142.stdout: "slow_ops_count": 0 2024-03-31T23:37:26.566 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-03-31T23:37:26.566 INFO:teuthology.orchestra.run.smithi142.stdout: "purge_queue": { 2024-03-31T23:37:26.566 INFO:teuthology.orchestra.run.smithi142.stdout: "pq_executing_ops": 0, 2024-03-31T23:37:26.567 INFO:teuthology.orchestra.run.smithi142.stdout: "pq_executing_ops_high_water": 16, 2024-03-31T23:37:26.567 INFO:teuthology.orchestra.run.smithi142.stdout: "pq_executing": 0, 2024-03-31T23:37:26.567 INFO:teuthology.orchestra.run.smithi142.stdout: "pq_executing_high_water": 8, 2024-03-31T23:37:26.567 INFO:teuthology.orchestra.run.smithi142.stdout: "pq_executed_ops": 31438, 2024-03-31T23:37:26.567 INFO:teuthology.orchestra.run.smithi142.stdout: "pq_executed": 15719, 2024-03-31T23:37:26.567 INFO:teuthology.orchestra.run.smithi142.stdout: "pq_item_in_journal": 0 2024-03-31T23:37:26.567 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-03-31T23:37:26.567 INFO:teuthology.orchestra.run.smithi142.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-31T23:37:26.567 INFO:teuthology.orchestra.run.smithi142.stdout: "val": 0, 2024-03-31T23:37:26.567 INFO:teuthology.orchestra.run.smithi142.stdout: "max": 104857600, 2024-03-31T23:37:26.567 INFO:teuthology.orchestra.run.smithi142.stdout: "get_started": 0, 2024-03-31T23:37:26.567 INFO:teuthology.orchestra.run.smithi142.stdout: "get": 1875913, 2024-03-31T23:37:26.567 INFO:teuthology.orchestra.run.smithi142.stdout: "get_sum": 415303727, 2024-03-31T23:37:26.567 INFO:teuthology.orchestra.run.smithi142.stdout: "get_or_fail_fail": 0, 2024-03-31T23:37:26.568 INFO:teuthology.orchestra.run.smithi142.stdout: "get_or_fail_success": 1875913, 2024-03-31T23:37:26.568 INFO:teuthology.orchestra.run.smithi142.stdout: "take": 0, 2024-03-31T23:37:26.568 INFO:teuthology.orchestra.run.smithi142.stdout: "take_sum": 0, 2024-03-31T23:37:26.568 INFO:teuthology.orchestra.run.smithi142.stdout: "put": 1875913, 2024-03-31T23:37:26.568 INFO:teuthology.orchestra.run.smithi142.stdout: "put_sum": 415303727, 2024-03-31T23:37:26.568 INFO:teuthology.orchestra.run.smithi142.stdout: "wait": { 2024-03-31T23:37:26.568 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 0, 2024-03-31T23:37:26.568 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.000000000, 2024-03-31T23:37:26.568 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000000000 2024-03-31T23:37:26.568 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-03-31T23:37:26.568 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-03-31T23:37:26.568 INFO:teuthology.orchestra.run.smithi142.stdout: "throttle-objecter_bytes": { 2024-03-31T23:37:26.568 INFO:teuthology.orchestra.run.smithi142.stdout: "val": 0, 2024-03-31T23:37:26.568 INFO:teuthology.orchestra.run.smithi142.stdout: "max": 104857600, 2024-03-31T23:37:26.568 INFO:teuthology.orchestra.run.smithi142.stdout: "get_started": 0, 2024-03-31T23:37:26.569 INFO:teuthology.orchestra.run.smithi142.stdout: "get": 0, 2024-03-31T23:37:26.569 INFO:teuthology.orchestra.run.smithi142.stdout: "get_sum": 0, 2024-03-31T23:37:26.569 INFO:teuthology.orchestra.run.smithi142.stdout: "get_or_fail_fail": 0, 2024-03-31T23:37:26.569 INFO:teuthology.orchestra.run.smithi142.stdout: "get_or_fail_success": 0, 2024-03-31T23:37:26.569 INFO:teuthology.orchestra.run.smithi142.stdout: "take": 105446, 2024-03-31T23:37:26.569 INFO:teuthology.orchestra.run.smithi142.stdout: "take_sum": 516635537, 2024-03-31T23:37:26.569 INFO:teuthology.orchestra.run.smithi142.stdout: "put": 105446, 2024-03-31T23:37:26.569 INFO:teuthology.orchestra.run.smithi142.stdout: "put_sum": 516635537, 2024-03-31T23:37:26.569 INFO:teuthology.orchestra.run.smithi142.stdout: "wait": { 2024-03-31T23:37:26.569 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 0, 2024-03-31T23:37:26.569 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.000000000, 2024-03-31T23:37:26.569 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000000000 2024-03-31T23:37:26.569 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-03-31T23:37:26.569 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-03-31T23:37:26.569 INFO:teuthology.orchestra.run.smithi142.stdout: "throttle-objecter_ops": { 2024-03-31T23:37:26.570 INFO:teuthology.orchestra.run.smithi142.stdout: "val": 0, 2024-03-31T23:37:26.570 INFO:teuthology.orchestra.run.smithi142.stdout: "max": 1024, 2024-03-31T23:37:26.570 INFO:teuthology.orchestra.run.smithi142.stdout: "get_started": 0, 2024-03-31T23:37:26.570 INFO:teuthology.orchestra.run.smithi142.stdout: "get": 0, 2024-03-31T23:37:26.570 INFO:teuthology.orchestra.run.smithi142.stdout: "get_sum": 0, 2024-03-31T23:37:26.570 INFO:teuthology.orchestra.run.smithi142.stdout: "get_or_fail_fail": 0, 2024-03-31T23:37:26.570 INFO:teuthology.orchestra.run.smithi142.stdout: "get_or_fail_success": 0, 2024-03-31T23:37:26.570 INFO:teuthology.orchestra.run.smithi142.stdout: "take": 105446, 2024-03-31T23:37:26.570 INFO:teuthology.orchestra.run.smithi142.stdout: "take_sum": 105446, 2024-03-31T23:37:26.570 INFO:teuthology.orchestra.run.smithi142.stdout: "put": 105446, 2024-03-31T23:37:26.570 INFO:teuthology.orchestra.run.smithi142.stdout: "put_sum": 105446, 2024-03-31T23:37:26.570 INFO:teuthology.orchestra.run.smithi142.stdout: "wait": { 2024-03-31T23:37:26.570 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 0, 2024-03-31T23:37:26.570 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.000000000, 2024-03-31T23:37:26.571 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000000000 2024-03-31T23:37:26.571 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-03-31T23:37:26.571 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-03-31T23:37:26.571 INFO:teuthology.orchestra.run.smithi142.stdout: "throttle-write_buf_throttle": { 2024-03-31T23:37:26.571 INFO:teuthology.orchestra.run.smithi142.stdout: "val": 0, 2024-03-31T23:37:26.571 INFO:teuthology.orchestra.run.smithi142.stdout: "max": 3758096384, 2024-03-31T23:37:26.571 INFO:teuthology.orchestra.run.smithi142.stdout: "get_started": 0, 2024-03-31T23:37:26.571 INFO:teuthology.orchestra.run.smithi142.stdout: "get": 15719, 2024-03-31T23:37:26.571 INFO:teuthology.orchestra.run.smithi142.stdout: "get_sum": 1729219, 2024-03-31T23:37:26.571 INFO:teuthology.orchestra.run.smithi142.stdout: "get_or_fail_fail": 0, 2024-03-31T23:37:26.571 INFO:teuthology.orchestra.run.smithi142.stdout: "get_or_fail_success": 15719, 2024-03-31T23:37:26.571 INFO:teuthology.orchestra.run.smithi142.stdout: "take": 0, 2024-03-31T23:37:26.571 INFO:teuthology.orchestra.run.smithi142.stdout: "take_sum": 0, 2024-03-31T23:37:26.571 INFO:teuthology.orchestra.run.smithi142.stdout: "put": 1452, 2024-03-31T23:37:26.572 INFO:teuthology.orchestra.run.smithi142.stdout: "put_sum": 1729219, 2024-03-31T23:37:26.572 INFO:teuthology.orchestra.run.smithi142.stdout: "wait": { 2024-03-31T23:37:26.572 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 0, 2024-03-31T23:37:26.572 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.000000000, 2024-03-31T23:37:26.572 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000000000 2024-03-31T23:37:26.572 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-03-31T23:37:26.572 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-03-31T23:37:26.572 INFO:teuthology.orchestra.run.smithi142.stdout: "throttle-write_buf_throttle-0x5624d10a0500": { 2024-03-31T23:37:26.572 INFO:teuthology.orchestra.run.smithi142.stdout: "val": 0, 2024-03-31T23:37:26.572 INFO:teuthology.orchestra.run.smithi142.stdout: "max": 3758096384, 2024-03-31T23:37:26.572 INFO:teuthology.orchestra.run.smithi142.stdout: "get_started": 0, 2024-03-31T23:37:26.572 INFO:teuthology.orchestra.run.smithi142.stdout: "get": 161275, 2024-03-31T23:37:26.572 INFO:teuthology.orchestra.run.smithi142.stdout: "get_sum": 486904246, 2024-03-31T23:37:26.573 INFO:teuthology.orchestra.run.smithi142.stdout: "get_or_fail_fail": 0, 2024-03-31T23:37:26.573 INFO:teuthology.orchestra.run.smithi142.stdout: "get_or_fail_success": 161275, 2024-03-31T23:37:26.573 INFO:teuthology.orchestra.run.smithi142.stdout: "take": 0, 2024-03-31T23:37:26.573 INFO:teuthology.orchestra.run.smithi142.stdout: "take_sum": 0, 2024-03-31T23:37:26.573 INFO:teuthology.orchestra.run.smithi142.stdout: "put": 52220, 2024-03-31T23:37:26.573 INFO:teuthology.orchestra.run.smithi142.stdout: "put_sum": 486904246, 2024-03-31T23:37:26.573 INFO:teuthology.orchestra.run.smithi142.stdout: "wait": { 2024-03-31T23:37:26.573 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 0, 2024-03-31T23:37:26.573 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.000000000, 2024-03-31T23:37:26.573 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000000000 2024-03-31T23:37:26.573 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-03-31T23:37:26.573 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-03-31T23:37:26.573 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-03-31T23:37:27.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:37:26 smithi162 ceph-mon[36237]: pgmap v817: 129 pgs: 129 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 9.2 KiB/s wr, 10 op/s 2024-03-31T23:37:27.113 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:37:26 smithi046 ceph-mon[32498]: pgmap v817: 129 pgs: 129 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 9.2 KiB/s wr, 10 op/s 2024-03-31T23:37:27.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:37:26 smithi142 ceph-mon[35271]: pgmap v817: 129 pgs: 129 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 9.2 KiB/s wr, 10 op/s 2024-03-31T23:37:27.339 INFO:tasks.check_counter:Daemon mds.b mds.exported=84 2024-03-31T23:37:27.339 INFO:tasks.check_counter:Daemon mds.b mds.imported=62 2024-03-31T23:37:27.339 INFO:tasks.check_counter:Daemon mds.b mds_cache.dir_update=2774 2024-03-31T23:37:27.339 INFO:tasks.check_counter:Daemon mds.b mds_cache.dir_update_receipt=1046 2024-03-31T23:37:27.339 INFO:tasks.check_counter:Daemon mds.b mds.root_rsnaps=6 2024-03-31T23:37:27.339 INFO:tasks.check_counter:Daemon mds.b mds_server.req_mksnap_latency.avgcount=26 2024-03-31T23:37:27.339 INFO:tasks.check_counter:Daemon mds.b mds_server.req_rmsnap_latency.avgcount=20 2024-03-31T23:37:27.339 DEBUG:tasks.check_counter:Getting stats from e 2024-03-31T23:37:27.340 DEBUG:teuthology.orchestra.run.smithi142:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:384f31f3d3aee7f8ba3f60846b8844c049165583 shell --fsid 6a9d8f08-efb2-11ee-b647-cb9ed24678a4 -- ceph daemon mds.e perf dump 2024-03-31T23:37:27.648 INFO:teuthology.orchestra.run.smithi142.stderr:Inferring config /var/lib/ceph/6a9d8f08-efb2-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-31T23:37:29.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:37:28 smithi162 ceph-mon[36237]: pgmap v818: 129 pgs: 129 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 6.0 KiB/s wr, 5 op/s 2024-03-31T23:37:29.063 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:37:28 smithi142 ceph-mon[35271]: pgmap v818: 129 pgs: 129 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 6.0 KiB/s wr, 5 op/s 2024-03-31T23:37:29.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:37:28 smithi046 ceph-mon[32498]: pgmap v818: 129 pgs: 129 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 6.0 KiB/s wr, 5 op/s 2024-03-31T23:37:29.675 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-03-31T23:37:29.675 INFO:teuthology.orchestra.run.smithi142.stdout: "AsyncMessenger::Worker-0": { 2024-03-31T23:37:29.675 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_recv_messages": 4092, 2024-03-31T23:37:29.675 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_send_messages": 4092, 2024-03-31T23:37:29.675 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_recv_bytes": 232455498, 2024-03-31T23:37:29.675 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_send_bytes": 1300439, 2024-03-31T23:37:29.675 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_created_connections": 8, 2024-03-31T23:37:29.675 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_active_connections": 4, 2024-03-31T23:37:29.675 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_running_total_time": 0.397851240, 2024-03-31T23:37:29.675 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_running_send_time": 0.083002630, 2024-03-31T23:37:29.675 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_running_recv_time": 0.200819979, 2024-03-31T23:37:29.675 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_running_fast_dispatch_time": 0.092796418, 2024-03-31T23:37:29.676 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_send_messages_queue_lat": { 2024-03-31T23:37:29.676 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 4091, 2024-03-31T23:37:29.676 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.100036161, 2024-03-31T23:37:29.676 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000024452 2024-03-31T23:37:29.676 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-03-31T23:37:29.676 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_handle_ack_lat": { 2024-03-31T23:37:29.676 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 0, 2024-03-31T23:37:29.676 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.000000000, 2024-03-31T23:37:29.676 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000000000 2024-03-31T23:37:29.676 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-03-31T23:37:29.676 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_recv_encrypted_bytes": 2992, 2024-03-31T23:37:29.676 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_send_encrypted_bytes": 448 2024-03-31T23:37:29.676 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-03-31T23:37:29.676 INFO:teuthology.orchestra.run.smithi142.stdout: "AsyncMessenger::Worker-1": { 2024-03-31T23:37:29.676 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_recv_messages": 1919, 2024-03-31T23:37:29.676 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_send_messages": 1872, 2024-03-31T23:37:29.677 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_recv_bytes": 215989317, 2024-03-31T23:37:29.677 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_send_bytes": 725288, 2024-03-31T23:37:29.677 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_created_connections": 6, 2024-03-31T23:37:29.677 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_active_connections": 5, 2024-03-31T23:37:29.677 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_running_total_time": 0.278140691, 2024-03-31T23:37:29.677 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_running_send_time": 0.050016520, 2024-03-31T23:37:29.677 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_running_recv_time": 0.187505151, 2024-03-31T23:37:29.677 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_running_fast_dispatch_time": 0.041504964, 2024-03-31T23:37:29.677 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_send_messages_queue_lat": { 2024-03-31T23:37:29.677 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 1871, 2024-03-31T23:37:29.677 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.055550818, 2024-03-31T23:37:29.677 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000029690 2024-03-31T23:37:29.677 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-03-31T23:37:29.677 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_handle_ack_lat": { 2024-03-31T23:37:29.678 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 0, 2024-03-31T23:37:29.678 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.000000000, 2024-03-31T23:37:29.678 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000000000 2024-03-31T23:37:29.678 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-03-31T23:37:29.678 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_recv_encrypted_bytes": 1720688, 2024-03-31T23:37:29.678 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_send_encrypted_bytes": 291792 2024-03-31T23:37:29.678 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-03-31T23:37:29.678 INFO:teuthology.orchestra.run.smithi142.stdout: "AsyncMessenger::Worker-2": { 2024-03-31T23:37:29.678 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_recv_messages": 1227, 2024-03-31T23:37:29.678 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_send_messages": 1574, 2024-03-31T23:37:29.678 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_recv_bytes": 239394774, 2024-03-31T23:37:29.678 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_send_bytes": 1166270, 2024-03-31T23:37:29.678 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_created_connections": 9, 2024-03-31T23:37:29.678 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_active_connections": 6, 2024-03-31T23:37:29.678 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_running_total_time": 0.231677245, 2024-03-31T23:37:29.679 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_running_send_time": 0.033063867, 2024-03-31T23:37:29.679 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_running_recv_time": 0.160832054, 2024-03-31T23:37:29.679 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_running_fast_dispatch_time": 0.027808302, 2024-03-31T23:37:29.679 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_send_messages_queue_lat": { 2024-03-31T23:37:29.679 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 1571, 2024-03-31T23:37:29.679 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.057118586, 2024-03-31T23:37:29.679 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000036358 2024-03-31T23:37:29.679 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-03-31T23:37:29.679 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_handle_ack_lat": { 2024-03-31T23:37:29.679 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 0, 2024-03-31T23:37:29.679 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.000000000, 2024-03-31T23:37:29.679 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000000000 2024-03-31T23:37:29.679 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-03-31T23:37:29.679 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_recv_encrypted_bytes": 320, 2024-03-31T23:37:29.680 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_send_encrypted_bytes": 776720 2024-03-31T23:37:29.680 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-03-31T23:37:29.680 INFO:teuthology.orchestra.run.smithi142.stdout: "cct": { 2024-03-31T23:37:29.680 INFO:teuthology.orchestra.run.smithi142.stdout: "total_workers": 1, 2024-03-31T23:37:29.680 INFO:teuthology.orchestra.run.smithi142.stdout: "unhealthy_workers": 0 2024-03-31T23:37:29.680 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-03-31T23:37:29.680 INFO:teuthology.orchestra.run.smithi142.stdout: "finisher-MDSRank": { 2024-03-31T23:37:29.680 INFO:teuthology.orchestra.run.smithi142.stdout: "queue_len": 0, 2024-03-31T23:37:29.680 INFO:teuthology.orchestra.run.smithi142.stdout: "complete_latency": { 2024-03-31T23:37:29.680 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 10978, 2024-03-31T23:37:29.680 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.252334520, 2024-03-31T23:37:29.680 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000022985 2024-03-31T23:37:29.680 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-03-31T23:37:29.680 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-03-31T23:37:29.680 INFO:teuthology.orchestra.run.smithi142.stdout: "finisher-PurgeQueue": { 2024-03-31T23:37:29.681 INFO:teuthology.orchestra.run.smithi142.stdout: "queue_len": 0, 2024-03-31T23:37:29.681 INFO:teuthology.orchestra.run.smithi142.stdout: "complete_latency": { 2024-03-31T23:37:29.681 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 0, 2024-03-31T23:37:29.681 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.000000000, 2024-03-31T23:37:29.681 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000000000 2024-03-31T23:37:29.681 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-03-31T23:37:29.681 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-03-31T23:37:29.681 INFO:teuthology.orchestra.run.smithi142.stdout: "mds": { 2024-03-31T23:37:29.681 INFO:teuthology.orchestra.run.smithi142.stdout: "request": 0, 2024-03-31T23:37:29.681 INFO:teuthology.orchestra.run.smithi142.stdout: "reply": 0, 2024-03-31T23:37:29.681 INFO:teuthology.orchestra.run.smithi142.stdout: "reply_latency": { 2024-03-31T23:37:29.681 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 0, 2024-03-31T23:37:29.681 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.000000000, 2024-03-31T23:37:29.681 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000000000 2024-03-31T23:37:29.681 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-03-31T23:37:29.681 INFO:teuthology.orchestra.run.smithi142.stdout: "slow_reply": 0, 2024-03-31T23:37:29.682 INFO:teuthology.orchestra.run.smithi142.stdout: "forward": 0, 2024-03-31T23:37:29.682 INFO:teuthology.orchestra.run.smithi142.stdout: "dir_fetch_complete": 0, 2024-03-31T23:37:29.682 INFO:teuthology.orchestra.run.smithi142.stdout: "dir_fetch_keys": 0, 2024-03-31T23:37:29.682 INFO:teuthology.orchestra.run.smithi142.stdout: "dir_commit": 0, 2024-03-31T23:37:29.682 INFO:teuthology.orchestra.run.smithi142.stdout: "dir_split": 0, 2024-03-31T23:37:29.682 INFO:teuthology.orchestra.run.smithi142.stdout: "dir_merge": 0, 2024-03-31T23:37:29.682 INFO:teuthology.orchestra.run.smithi142.stdout: "inodes": 21616, 2024-03-31T23:37:29.682 INFO:teuthology.orchestra.run.smithi142.stdout: "inodes_top": 5049, 2024-03-31T23:37:29.682 INFO:teuthology.orchestra.run.smithi142.stdout: "inodes_bottom": 16567, 2024-03-31T23:37:29.682 INFO:teuthology.orchestra.run.smithi142.stdout: "inodes_pin_tail": 0, 2024-03-31T23:37:29.682 INFO:teuthology.orchestra.run.smithi142.stdout: "inodes_pinned": 14403, 2024-03-31T23:37:29.682 INFO:teuthology.orchestra.run.smithi142.stdout: "inodes_expired": 0, 2024-03-31T23:37:29.682 INFO:teuthology.orchestra.run.smithi142.stdout: "inodes_with_caps": 0, 2024-03-31T23:37:29.682 INFO:teuthology.orchestra.run.smithi142.stdout: "caps": 0, 2024-03-31T23:37:29.683 INFO:teuthology.orchestra.run.smithi142.stdout: "subtrees": 1, 2024-03-31T23:37:29.683 INFO:teuthology.orchestra.run.smithi142.stdout: "traverse": 0, 2024-03-31T23:37:29.683 INFO:teuthology.orchestra.run.smithi142.stdout: "traverse_hit": 0, 2024-03-31T23:37:29.683 INFO:teuthology.orchestra.run.smithi142.stdout: "traverse_forward": 0, 2024-03-31T23:37:29.683 INFO:teuthology.orchestra.run.smithi142.stdout: "traverse_discover": 0, 2024-03-31T23:37:29.683 INFO:teuthology.orchestra.run.smithi142.stdout: "traverse_dir_fetch": 0, 2024-03-31T23:37:29.683 INFO:teuthology.orchestra.run.smithi142.stdout: "traverse_remote_ino": 0, 2024-03-31T23:37:29.683 INFO:teuthology.orchestra.run.smithi142.stdout: "traverse_lock": 0, 2024-03-31T23:37:29.683 INFO:teuthology.orchestra.run.smithi142.stdout: "load_cent": 0, 2024-03-31T23:37:29.683 INFO:teuthology.orchestra.run.smithi142.stdout: "q": 0, 2024-03-31T23:37:29.683 INFO:teuthology.orchestra.run.smithi142.stdout: "exported": 0, 2024-03-31T23:37:29.683 INFO:teuthology.orchestra.run.smithi142.stdout: "exported_inodes": 0, 2024-03-31T23:37:29.683 INFO:teuthology.orchestra.run.smithi142.stdout: "imported": 0, 2024-03-31T23:37:29.683 INFO:teuthology.orchestra.run.smithi142.stdout: "imported_inodes": 0, 2024-03-31T23:37:29.683 INFO:teuthology.orchestra.run.smithi142.stdout: "openino_dir_fetch": 0, 2024-03-31T23:37:29.684 INFO:teuthology.orchestra.run.smithi142.stdout: "openino_backtrace_fetch": 0, 2024-03-31T23:37:29.684 INFO:teuthology.orchestra.run.smithi142.stdout: "openino_peer_discover": 0, 2024-03-31T23:37:29.684 INFO:teuthology.orchestra.run.smithi142.stdout: "root_rfiles": 0, 2024-03-31T23:37:29.684 INFO:teuthology.orchestra.run.smithi142.stdout: "root_rbytes": 0, 2024-03-31T23:37:29.684 INFO:teuthology.orchestra.run.smithi142.stdout: "root_rsnaps": 0, 2024-03-31T23:37:29.684 INFO:teuthology.orchestra.run.smithi142.stdout: "scrub_backtrace_fetch": 0, 2024-03-31T23:37:29.684 INFO:teuthology.orchestra.run.smithi142.stdout: "scrub_set_tag": 0, 2024-03-31T23:37:29.684 INFO:teuthology.orchestra.run.smithi142.stdout: "scrub_backtrace_repaired": 0, 2024-03-31T23:37:29.684 INFO:teuthology.orchestra.run.smithi142.stdout: "scrub_inotable_repaired": 0, 2024-03-31T23:37:29.684 INFO:teuthology.orchestra.run.smithi142.stdout: "scrub_dir_inodes": 0, 2024-03-31T23:37:29.684 INFO:teuthology.orchestra.run.smithi142.stdout: "scrub_dir_base_inodes": 0, 2024-03-31T23:37:29.684 INFO:teuthology.orchestra.run.smithi142.stdout: "scrub_dirfrag_rstats": 0, 2024-03-31T23:37:29.684 INFO:teuthology.orchestra.run.smithi142.stdout: "scrub_file_inodes": 0, 2024-03-31T23:37:29.684 INFO:teuthology.orchestra.run.smithi142.stdout: "handle_inode_file_caps": 0, 2024-03-31T23:37:29.684 INFO:teuthology.orchestra.run.smithi142.stdout: "ceph_cap_op_revoke": 0, 2024-03-31T23:37:29.684 INFO:teuthology.orchestra.run.smithi142.stdout: "ceph_cap_op_grant": 0, 2024-03-31T23:37:29.685 INFO:teuthology.orchestra.run.smithi142.stdout: "ceph_cap_op_trunc": 0, 2024-03-31T23:37:29.685 INFO:teuthology.orchestra.run.smithi142.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-31T23:37:29.685 INFO:teuthology.orchestra.run.smithi142.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-31T23:37:29.685 INFO:teuthology.orchestra.run.smithi142.stdout: "handle_client_caps": 0, 2024-03-31T23:37:29.685 INFO:teuthology.orchestra.run.smithi142.stdout: "handle_client_caps_dirty": 0, 2024-03-31T23:37:29.685 INFO:teuthology.orchestra.run.smithi142.stdout: "handle_client_cap_release": 0, 2024-03-31T23:37:29.685 INFO:teuthology.orchestra.run.smithi142.stdout: "process_request_cap_release": 0 2024-03-31T23:37:29.685 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-03-31T23:37:29.685 INFO:teuthology.orchestra.run.smithi142.stdout: "mds_cache": { 2024-03-31T23:37:29.685 INFO:teuthology.orchestra.run.smithi142.stdout: "dir_update": 0, 2024-03-31T23:37:29.685 INFO:teuthology.orchestra.run.smithi142.stdout: "dir_update_receipt": 0, 2024-03-31T23:37:29.685 INFO:teuthology.orchestra.run.smithi142.stdout: "dir_try_discover": 0, 2024-03-31T23:37:29.685 INFO:teuthology.orchestra.run.smithi142.stdout: "dir_send_discover": 0, 2024-03-31T23:37:29.685 INFO:teuthology.orchestra.run.smithi142.stdout: "dir_handle_discover": 0, 2024-03-31T23:37:29.685 INFO:teuthology.orchestra.run.smithi142.stdout: "num_strays": 0, 2024-03-31T23:37:29.685 INFO:teuthology.orchestra.run.smithi142.stdout: "num_strays_delayed": 0, 2024-03-31T23:37:29.686 INFO:teuthology.orchestra.run.smithi142.stdout: "num_strays_enqueuing": 0, 2024-03-31T23:37:29.686 INFO:teuthology.orchestra.run.smithi142.stdout: "strays_created": 0, 2024-03-31T23:37:29.686 INFO:teuthology.orchestra.run.smithi142.stdout: "strays_enqueued": 0, 2024-03-31T23:37:29.686 INFO:teuthology.orchestra.run.smithi142.stdout: "strays_reintegrated": 0, 2024-03-31T23:37:29.686 INFO:teuthology.orchestra.run.smithi142.stdout: "strays_migrated": 0, 2024-03-31T23:37:29.686 INFO:teuthology.orchestra.run.smithi142.stdout: "num_recovering_processing": 0, 2024-03-31T23:37:29.686 INFO:teuthology.orchestra.run.smithi142.stdout: "num_recovering_enqueued": 0, 2024-03-31T23:37:29.686 INFO:teuthology.orchestra.run.smithi142.stdout: "num_recovering_prioritized": 0, 2024-03-31T23:37:29.686 INFO:teuthology.orchestra.run.smithi142.stdout: "recovery_started": 0, 2024-03-31T23:37:29.686 INFO:teuthology.orchestra.run.smithi142.stdout: "recovery_completed": 0, 2024-03-31T23:37:29.686 INFO:teuthology.orchestra.run.smithi142.stdout: "ireq_quiesce_path": 0, 2024-03-31T23:37:29.686 INFO:teuthology.orchestra.run.smithi142.stdout: "ireq_quiesce_inode": 0, 2024-03-31T23:37:29.686 INFO:teuthology.orchestra.run.smithi142.stdout: "ireq_enqueue_scrub": 0, 2024-03-31T23:37:29.686 INFO:teuthology.orchestra.run.smithi142.stdout: "ireq_exportdir": 0, 2024-03-31T23:37:29.687 INFO:teuthology.orchestra.run.smithi142.stdout: "ireq_flush": 0, 2024-03-31T23:37:29.687 INFO:teuthology.orchestra.run.smithi142.stdout: "ireq_fragmentdir": 0, 2024-03-31T23:37:29.687 INFO:teuthology.orchestra.run.smithi142.stdout: "ireq_fragstats": 0, 2024-03-31T23:37:29.687 INFO:teuthology.orchestra.run.smithi142.stdout: "ireq_inodestats": 0 2024-03-31T23:37:29.687 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-03-31T23:37:29.687 INFO:teuthology.orchestra.run.smithi142.stdout: "mds_log": { 2024-03-31T23:37:29.687 INFO:teuthology.orchestra.run.smithi142.stdout: "evlrg": 0, 2024-03-31T23:37:29.687 INFO:teuthology.orchestra.run.smithi142.stdout: "evadd": 0, 2024-03-31T23:37:29.687 INFO:teuthology.orchestra.run.smithi142.stdout: "evex": 0, 2024-03-31T23:37:29.687 INFO:teuthology.orchestra.run.smithi142.stdout: "evtrm": 0, 2024-03-31T23:37:29.687 INFO:teuthology.orchestra.run.smithi142.stdout: "ev": 149835, 2024-03-31T23:37:29.687 INFO:teuthology.orchestra.run.smithi142.stdout: "evexg": 0, 2024-03-31T23:37:29.687 INFO:teuthology.orchestra.run.smithi142.stdout: "evexd": 0, 2024-03-31T23:37:29.687 INFO:teuthology.orchestra.run.smithi142.stdout: "segadd": 0, 2024-03-31T23:37:29.687 INFO:teuthology.orchestra.run.smithi142.stdout: "segex": 0, 2024-03-31T23:37:29.687 INFO:teuthology.orchestra.run.smithi142.stdout: "segtrm": 0, 2024-03-31T23:37:29.687 INFO:teuthology.orchestra.run.smithi142.stdout: "seg": 136, 2024-03-31T23:37:29.688 INFO:teuthology.orchestra.run.smithi142.stdout: "segmjr": 14, 2024-03-31T23:37:29.688 INFO:teuthology.orchestra.run.smithi142.stdout: "segexg": 0, 2024-03-31T23:37:29.688 INFO:teuthology.orchestra.run.smithi142.stdout: "segexd": 0, 2024-03-31T23:37:29.688 INFO:teuthology.orchestra.run.smithi142.stdout: "expos": 151235441, 2024-03-31T23:37:29.688 INFO:teuthology.orchestra.run.smithi142.stdout: "wrpos": 688455771, 2024-03-31T23:37:29.688 INFO:teuthology.orchestra.run.smithi142.stdout: "rdpos": 688455172, 2024-03-31T23:37:29.688 INFO:teuthology.orchestra.run.smithi142.stdout: "jlat": { 2024-03-31T23:37:29.688 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 0, 2024-03-31T23:37:29.688 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.000000000, 2024-03-31T23:37:29.688 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000000000 2024-03-31T23:37:29.688 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-03-31T23:37:29.688 INFO:teuthology.orchestra.run.smithi142.stdout: "replayed": 149835 2024-03-31T23:37:29.688 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-03-31T23:37:29.688 INFO:teuthology.orchestra.run.smithi142.stdout: "mds_mem": { 2024-03-31T23:37:29.688 INFO:teuthology.orchestra.run.smithi142.stdout: "ino": 3181, 2024-03-31T23:37:29.688 INFO:teuthology.orchestra.run.smithi142.stdout: "ino+": 22462, 2024-03-31T23:37:29.689 INFO:teuthology.orchestra.run.smithi142.stdout: "ino-": 19281, 2024-03-31T23:37:29.689 INFO:teuthology.orchestra.run.smithi142.stdout: "dir": 120, 2024-03-31T23:37:29.689 INFO:teuthology.orchestra.run.smithi142.stdout: "dir+": 216, 2024-03-31T23:37:29.689 INFO:teuthology.orchestra.run.smithi142.stdout: "dir-": 96, 2024-03-31T23:37:29.689 INFO:teuthology.orchestra.run.smithi142.stdout: "dn": 21616, 2024-03-31T23:37:29.689 INFO:teuthology.orchestra.run.smithi142.stdout: "dn+": 23091, 2024-03-31T23:37:29.689 INFO:teuthology.orchestra.run.smithi142.stdout: "dn-": 1475, 2024-03-31T23:37:29.689 INFO:teuthology.orchestra.run.smithi142.stdout: "cap": 0, 2024-03-31T23:37:29.689 INFO:teuthology.orchestra.run.smithi142.stdout: "cap+": 0, 2024-03-31T23:37:29.689 INFO:teuthology.orchestra.run.smithi142.stdout: "cap-": 0, 2024-03-31T23:37:29.689 INFO:teuthology.orchestra.run.smithi142.stdout: "rss": 85348, 2024-03-31T23:37:29.689 INFO:teuthology.orchestra.run.smithi142.stdout: "heap": 215420 2024-03-31T23:37:29.689 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-03-31T23:37:29.689 INFO:teuthology.orchestra.run.smithi142.stdout: "mds_server": { 2024-03-31T23:37:29.689 INFO:teuthology.orchestra.run.smithi142.stdout: "dispatch_client_request": 0, 2024-03-31T23:37:29.689 INFO:teuthology.orchestra.run.smithi142.stdout: "dispatch_server_request": 0, 2024-03-31T23:37:29.689 INFO:teuthology.orchestra.run.smithi142.stdout: "handle_client_request": 0, 2024-03-31T23:37:29.690 INFO:teuthology.orchestra.run.smithi142.stdout: "handle_client_session": 0, 2024-03-31T23:37:29.690 INFO:teuthology.orchestra.run.smithi142.stdout: "handle_peer_request": 0, 2024-03-31T23:37:29.690 INFO:teuthology.orchestra.run.smithi142.stdout: "req_create_latency": { 2024-03-31T23:37:29.690 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 0, 2024-03-31T23:37:29.690 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.000000000, 2024-03-31T23:37:29.690 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000000000 2024-03-31T23:37:29.690 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-03-31T23:37:29.690 INFO:teuthology.orchestra.run.smithi142.stdout: "req_getattr_latency": { 2024-03-31T23:37:29.690 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 0, 2024-03-31T23:37:29.690 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.000000000, 2024-03-31T23:37:29.690 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000000000 2024-03-31T23:37:29.690 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-03-31T23:37:29.690 INFO:teuthology.orchestra.run.smithi142.stdout: "req_getfilelock_latency": { 2024-03-31T23:37:29.690 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 0, 2024-03-31T23:37:29.691 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.000000000, 2024-03-31T23:37:29.691 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000000000 2024-03-31T23:37:29.691 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-03-31T23:37:29.691 INFO:teuthology.orchestra.run.smithi142.stdout: "req_link_latency": { 2024-03-31T23:37:29.691 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 0, 2024-03-31T23:37:29.691 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.000000000, 2024-03-31T23:37:29.691 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000000000 2024-03-31T23:37:29.691 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-03-31T23:37:29.691 INFO:teuthology.orchestra.run.smithi142.stdout: "req_lookup_latency": { 2024-03-31T23:37:29.691 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 0, 2024-03-31T23:37:29.691 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.000000000, 2024-03-31T23:37:29.691 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000000000 2024-03-31T23:37:29.691 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-03-31T23:37:29.691 INFO:teuthology.orchestra.run.smithi142.stdout: "req_lookuphash_latency": { 2024-03-31T23:37:29.691 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 0, 2024-03-31T23:37:29.692 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.000000000, 2024-03-31T23:37:29.692 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000000000 2024-03-31T23:37:29.692 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-03-31T23:37:29.692 INFO:teuthology.orchestra.run.smithi142.stdout: "req_lookupino_latency": { 2024-03-31T23:37:29.692 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 0, 2024-03-31T23:37:29.692 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.000000000, 2024-03-31T23:37:29.692 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000000000 2024-03-31T23:37:29.692 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-03-31T23:37:29.692 INFO:teuthology.orchestra.run.smithi142.stdout: "req_lookupname_latency": { 2024-03-31T23:37:29.692 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 0, 2024-03-31T23:37:29.692 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.000000000, 2024-03-31T23:37:29.692 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000000000 2024-03-31T23:37:29.692 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-03-31T23:37:29.692 INFO:teuthology.orchestra.run.smithi142.stdout: "req_lookupparent_latency": { 2024-03-31T23:37:29.693 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 0, 2024-03-31T23:37:29.693 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.000000000, 2024-03-31T23:37:29.693 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000000000 2024-03-31T23:37:29.693 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-03-31T23:37:29.693 INFO:teuthology.orchestra.run.smithi142.stdout: "req_lookupsnap_latency": { 2024-03-31T23:37:29.693 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 0, 2024-03-31T23:37:29.693 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.000000000, 2024-03-31T23:37:29.693 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000000000 2024-03-31T23:37:29.693 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-03-31T23:37:29.693 INFO:teuthology.orchestra.run.smithi142.stdout: "req_lssnap_latency": { 2024-03-31T23:37:29.693 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 0, 2024-03-31T23:37:29.693 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.000000000, 2024-03-31T23:37:29.693 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000000000 2024-03-31T23:37:29.694 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-03-31T23:37:29.694 INFO:teuthology.orchestra.run.smithi142.stdout: "req_mkdir_latency": { 2024-03-31T23:37:29.694 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 0, 2024-03-31T23:37:29.694 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.000000000, 2024-03-31T23:37:29.694 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000000000 2024-03-31T23:37:29.694 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-03-31T23:37:29.694 INFO:teuthology.orchestra.run.smithi142.stdout: "req_mknod_latency": { 2024-03-31T23:37:29.694 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 0, 2024-03-31T23:37:29.695 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.000000000, 2024-03-31T23:37:29.695 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000000000 2024-03-31T23:37:29.695 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-03-31T23:37:29.695 INFO:teuthology.orchestra.run.smithi142.stdout: "req_mksnap_latency": { 2024-03-31T23:37:29.695 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 0, 2024-03-31T23:37:29.695 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.000000000, 2024-03-31T23:37:29.695 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000000000 2024-03-31T23:37:29.695 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-03-31T23:37:29.695 INFO:teuthology.orchestra.run.smithi142.stdout: "req_open_latency": { 2024-03-31T23:37:29.695 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 0, 2024-03-31T23:37:29.695 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.000000000, 2024-03-31T23:37:29.695 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000000000 2024-03-31T23:37:29.695 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-03-31T23:37:29.695 INFO:teuthology.orchestra.run.smithi142.stdout: "req_readdir_latency": { 2024-03-31T23:37:29.696 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 0, 2024-03-31T23:37:29.696 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.000000000, 2024-03-31T23:37:29.696 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000000000 2024-03-31T23:37:29.696 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-03-31T23:37:29.696 INFO:teuthology.orchestra.run.smithi142.stdout: "req_rename_latency": { 2024-03-31T23:37:29.696 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 0, 2024-03-31T23:37:29.696 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.000000000, 2024-03-31T23:37:29.696 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000000000 2024-03-31T23:37:29.696 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-03-31T23:37:29.696 INFO:teuthology.orchestra.run.smithi142.stdout: "req_renamesnap_latency": { 2024-03-31T23:37:29.696 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 0, 2024-03-31T23:37:29.696 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.000000000, 2024-03-31T23:37:29.696 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000000000 2024-03-31T23:37:29.696 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-03-31T23:37:29.697 INFO:teuthology.orchestra.run.smithi142.stdout: "req_snapdiff_latency": { 2024-03-31T23:37:29.697 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 0, 2024-03-31T23:37:29.697 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.000000000, 2024-03-31T23:37:29.697 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000000000 2024-03-31T23:37:29.697 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-03-31T23:37:29.697 INFO:teuthology.orchestra.run.smithi142.stdout: "req_rmdir_latency": { 2024-03-31T23:37:29.697 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 0, 2024-03-31T23:37:29.697 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.000000000, 2024-03-31T23:37:29.697 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000000000 2024-03-31T23:37:29.697 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-03-31T23:37:29.697 INFO:teuthology.orchestra.run.smithi142.stdout: "req_rmsnap_latency": { 2024-03-31T23:37:29.697 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 0, 2024-03-31T23:37:29.697 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.000000000, 2024-03-31T23:37:29.697 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000000000 2024-03-31T23:37:29.698 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-03-31T23:37:29.698 INFO:teuthology.orchestra.run.smithi142.stdout: "req_rmxattr_latency": { 2024-03-31T23:37:29.698 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 0, 2024-03-31T23:37:29.698 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.000000000, 2024-03-31T23:37:29.698 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000000000 2024-03-31T23:37:29.698 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-03-31T23:37:29.698 INFO:teuthology.orchestra.run.smithi142.stdout: "req_setattr_latency": { 2024-03-31T23:37:29.698 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 0, 2024-03-31T23:37:29.698 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.000000000, 2024-03-31T23:37:29.698 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000000000 2024-03-31T23:37:29.698 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-03-31T23:37:29.698 INFO:teuthology.orchestra.run.smithi142.stdout: "req_setdirlayout_latency": { 2024-03-31T23:37:29.698 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 0, 2024-03-31T23:37:29.698 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.000000000, 2024-03-31T23:37:29.699 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000000000 2024-03-31T23:37:29.699 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-03-31T23:37:29.699 INFO:teuthology.orchestra.run.smithi142.stdout: "req_setfilelock_latency": { 2024-03-31T23:37:29.699 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 0, 2024-03-31T23:37:29.699 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.000000000, 2024-03-31T23:37:29.699 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000000000 2024-03-31T23:37:29.699 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-03-31T23:37:29.699 INFO:teuthology.orchestra.run.smithi142.stdout: "req_setlayout_latency": { 2024-03-31T23:37:29.699 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 0, 2024-03-31T23:37:29.699 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.000000000, 2024-03-31T23:37:29.699 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000000000 2024-03-31T23:37:29.699 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-03-31T23:37:29.699 INFO:teuthology.orchestra.run.smithi142.stdout: "req_setxattr_latency": { 2024-03-31T23:37:29.699 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 0, 2024-03-31T23:37:29.700 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.000000000, 2024-03-31T23:37:29.700 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000000000 2024-03-31T23:37:29.700 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-03-31T23:37:29.700 INFO:teuthology.orchestra.run.smithi142.stdout: "req_symlink_latency": { 2024-03-31T23:37:29.700 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 0, 2024-03-31T23:37:29.700 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.000000000, 2024-03-31T23:37:29.700 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000000000 2024-03-31T23:37:29.700 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-03-31T23:37:29.700 INFO:teuthology.orchestra.run.smithi142.stdout: "req_unlink_latency": { 2024-03-31T23:37:29.700 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 0, 2024-03-31T23:37:29.700 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.000000000, 2024-03-31T23:37:29.700 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000000000 2024-03-31T23:37:29.700 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-03-31T23:37:29.700 INFO:teuthology.orchestra.run.smithi142.stdout: "cap_revoke_eviction": 0, 2024-03-31T23:37:29.700 INFO:teuthology.orchestra.run.smithi142.stdout: "cap_acquisition_throttle": 0, 2024-03-31T23:37:29.701 INFO:teuthology.orchestra.run.smithi142.stdout: "req_getvxattr_latency": { 2024-03-31T23:37:29.701 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 0, 2024-03-31T23:37:29.701 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.000000000, 2024-03-31T23:37:29.701 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000000000 2024-03-31T23:37:29.701 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-03-31T23:37:29.701 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-03-31T23:37:29.701 INFO:teuthology.orchestra.run.smithi142.stdout: "mds_sessions": { 2024-03-31T23:37:29.701 INFO:teuthology.orchestra.run.smithi142.stdout: "session_count": 2, 2024-03-31T23:37:29.701 INFO:teuthology.orchestra.run.smithi142.stdout: "session_add": 4, 2024-03-31T23:37:29.701 INFO:teuthology.orchestra.run.smithi142.stdout: "session_remove": 2, 2024-03-31T23:37:29.701 INFO:teuthology.orchestra.run.smithi142.stdout: "sessions_open": 3, 2024-03-31T23:37:29.701 INFO:teuthology.orchestra.run.smithi142.stdout: "sessions_stale": 0, 2024-03-31T23:37:29.701 INFO:teuthology.orchestra.run.smithi142.stdout: "total_load": 0, 2024-03-31T23:37:29.701 INFO:teuthology.orchestra.run.smithi142.stdout: "average_load": 0, 2024-03-31T23:37:29.702 INFO:teuthology.orchestra.run.smithi142.stdout: "avg_session_uptime": 3215, 2024-03-31T23:37:29.702 INFO:teuthology.orchestra.run.smithi142.stdout: "mdthresh_evicted": 0 2024-03-31T23:37:29.702 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-03-31T23:37:29.702 INFO:teuthology.orchestra.run.smithi142.stdout: "mempool": { 2024-03-31T23:37:29.702 INFO:teuthology.orchestra.run.smithi142.stdout: "bloom_filter_bytes": 0, 2024-03-31T23:37:29.702 INFO:teuthology.orchestra.run.smithi142.stdout: "bloom_filter_items": 0, 2024-03-31T23:37:29.702 INFO:teuthology.orchestra.run.smithi142.stdout: "bluestore_alloc_bytes": 0, 2024-03-31T23:37:29.702 INFO:teuthology.orchestra.run.smithi142.stdout: "bluestore_alloc_items": 0, 2024-03-31T23:37:29.702 INFO:teuthology.orchestra.run.smithi142.stdout: "bluestore_cache_data_bytes": 0, 2024-03-31T23:37:29.702 INFO:teuthology.orchestra.run.smithi142.stdout: "bluestore_cache_data_items": 0, 2024-03-31T23:37:29.702 INFO:teuthology.orchestra.run.smithi142.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-31T23:37:29.702 INFO:teuthology.orchestra.run.smithi142.stdout: "bluestore_cache_onode_items": 0, 2024-03-31T23:37:29.702 INFO:teuthology.orchestra.run.smithi142.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-31T23:37:29.702 INFO:teuthology.orchestra.run.smithi142.stdout: "bluestore_cache_meta_items": 0, 2024-03-31T23:37:29.702 INFO:teuthology.orchestra.run.smithi142.stdout: "bluestore_cache_other_bytes": 0, 2024-03-31T23:37:29.703 INFO:teuthology.orchestra.run.smithi142.stdout: "bluestore_cache_other_items": 0, 2024-03-31T23:37:29.703 INFO:teuthology.orchestra.run.smithi142.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-31T23:37:29.703 INFO:teuthology.orchestra.run.smithi142.stdout: "bluestore_cache_buffer_items": 0, 2024-03-31T23:37:29.703 INFO:teuthology.orchestra.run.smithi142.stdout: "bluestore_extent_bytes": 0, 2024-03-31T23:37:29.703 INFO:teuthology.orchestra.run.smithi142.stdout: "bluestore_extent_items": 0, 2024-03-31T23:37:29.703 INFO:teuthology.orchestra.run.smithi142.stdout: "bluestore_blob_bytes": 0, 2024-03-31T23:37:29.703 INFO:teuthology.orchestra.run.smithi142.stdout: "bluestore_blob_items": 0, 2024-03-31T23:37:29.703 INFO:teuthology.orchestra.run.smithi142.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-31T23:37:29.703 INFO:teuthology.orchestra.run.smithi142.stdout: "bluestore_shared_blob_items": 0, 2024-03-31T23:37:29.703 INFO:teuthology.orchestra.run.smithi142.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-31T23:37:29.703 INFO:teuthology.orchestra.run.smithi142.stdout: "bluestore_inline_bl_items": 0, 2024-03-31T23:37:29.703 INFO:teuthology.orchestra.run.smithi142.stdout: "bluestore_fsck_bytes": 0, 2024-03-31T23:37:29.703 INFO:teuthology.orchestra.run.smithi142.stdout: "bluestore_fsck_items": 0, 2024-03-31T23:37:29.703 INFO:teuthology.orchestra.run.smithi142.stdout: "bluestore_txc_bytes": 0, 2024-03-31T23:37:29.703 INFO:teuthology.orchestra.run.smithi142.stdout: "bluestore_txc_items": 0, 2024-03-31T23:37:29.704 INFO:teuthology.orchestra.run.smithi142.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-31T23:37:29.704 INFO:teuthology.orchestra.run.smithi142.stdout: "bluestore_writing_deferred_items": 0, 2024-03-31T23:37:29.704 INFO:teuthology.orchestra.run.smithi142.stdout: "bluestore_writing_bytes": 0, 2024-03-31T23:37:29.704 INFO:teuthology.orchestra.run.smithi142.stdout: "bluestore_writing_items": 0, 2024-03-31T23:37:29.704 INFO:teuthology.orchestra.run.smithi142.stdout: "bluefs_bytes": 0, 2024-03-31T23:37:29.704 INFO:teuthology.orchestra.run.smithi142.stdout: "bluefs_items": 0, 2024-03-31T23:37:29.704 INFO:teuthology.orchestra.run.smithi142.stdout: "bluefs_file_reader_bytes": 0, 2024-03-31T23:37:29.704 INFO:teuthology.orchestra.run.smithi142.stdout: "bluefs_file_reader_items": 0, 2024-03-31T23:37:29.704 INFO:teuthology.orchestra.run.smithi142.stdout: "bluefs_file_writer_bytes": 0, 2024-03-31T23:37:29.704 INFO:teuthology.orchestra.run.smithi142.stdout: "bluefs_file_writer_items": 0, 2024-03-31T23:37:29.704 INFO:teuthology.orchestra.run.smithi142.stdout: "buffer_anon_bytes": 241833, 2024-03-31T23:37:29.704 INFO:teuthology.orchestra.run.smithi142.stdout: "buffer_anon_items": 86, 2024-03-31T23:37:29.704 INFO:teuthology.orchestra.run.smithi142.stdout: "buffer_meta_bytes": 0, 2024-03-31T23:37:29.704 INFO:teuthology.orchestra.run.smithi142.stdout: "buffer_meta_items": 0, 2024-03-31T23:37:29.704 INFO:teuthology.orchestra.run.smithi142.stdout: "osd_bytes": 0, 2024-03-31T23:37:29.704 INFO:teuthology.orchestra.run.smithi142.stdout: "osd_items": 0, 2024-03-31T23:37:29.705 INFO:teuthology.orchestra.run.smithi142.stdout: "osd_mapbl_bytes": 0, 2024-03-31T23:37:29.705 INFO:teuthology.orchestra.run.smithi142.stdout: "osd_mapbl_items": 0, 2024-03-31T23:37:29.705 INFO:teuthology.orchestra.run.smithi142.stdout: "osd_pglog_bytes": 0, 2024-03-31T23:37:29.705 INFO:teuthology.orchestra.run.smithi142.stdout: "osd_pglog_items": 0, 2024-03-31T23:37:29.705 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap_bytes": 9704, 2024-03-31T23:37:29.705 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap_items": 186, 2024-03-31T23:37:29.705 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap_mapping_bytes": 0, 2024-03-31T23:37:29.705 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap_mapping_items": 0, 2024-03-31T23:37:29.705 INFO:teuthology.orchestra.run.smithi142.stdout: "pgmap_bytes": 0, 2024-03-31T23:37:29.705 INFO:teuthology.orchestra.run.smithi142.stdout: "pgmap_items": 0, 2024-03-31T23:37:29.705 INFO:teuthology.orchestra.run.smithi142.stdout: "mds_co_bytes": 22618659, 2024-03-31T23:37:29.705 INFO:teuthology.orchestra.run.smithi142.stdout: "mds_co_items": 103047, 2024-03-31T23:37:29.705 INFO:teuthology.orchestra.run.smithi142.stdout: "unittest_1_bytes": 0, 2024-03-31T23:37:29.705 INFO:teuthology.orchestra.run.smithi142.stdout: "unittest_1_items": 0, 2024-03-31T23:37:29.705 INFO:teuthology.orchestra.run.smithi142.stdout: "unittest_2_bytes": 0, 2024-03-31T23:37:29.705 INFO:teuthology.orchestra.run.smithi142.stdout: "unittest_2_items": 0 2024-03-31T23:37:29.706 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-03-31T23:37:29.706 INFO:teuthology.orchestra.run.smithi142.stdout: "objecter": { 2024-03-31T23:37:29.706 INFO:teuthology.orchestra.run.smithi142.stdout: "op_active": 0, 2024-03-31T23:37:29.706 INFO:teuthology.orchestra.run.smithi142.stdout: "op_laggy": 0, 2024-03-31T23:37:29.706 INFO:teuthology.orchestra.run.smithi142.stdout: "op_send": 6676, 2024-03-31T23:37:29.706 INFO:teuthology.orchestra.run.smithi142.stdout: "op_send_bytes": 21, 2024-03-31T23:37:29.706 INFO:teuthology.orchestra.run.smithi142.stdout: "op_resend": 3, 2024-03-31T23:37:29.706 INFO:teuthology.orchestra.run.smithi142.stdout: "op_reply": 6676, 2024-03-31T23:37:29.706 INFO:teuthology.orchestra.run.smithi142.stdout: "op_latency": { 2024-03-31T23:37:29.706 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 6676, 2024-03-31T23:37:29.706 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 5.930887265, 2024-03-31T23:37:29.706 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000888389 2024-03-31T23:37:29.706 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-03-31T23:37:29.706 INFO:teuthology.orchestra.run.smithi142.stdout: "op_inflight": 0, 2024-03-31T23:37:29.707 INFO:teuthology.orchestra.run.smithi142.stdout: "oplen_avg": { 2024-03-31T23:37:29.707 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 6676, 2024-03-31T23:37:29.707 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 6677 2024-03-31T23:37:29.707 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-03-31T23:37:29.707 INFO:teuthology.orchestra.run.smithi142.stdout: "op": 6676, 2024-03-31T23:37:29.707 INFO:teuthology.orchestra.run.smithi142.stdout: "op_r": 6676, 2024-03-31T23:37:29.707 INFO:teuthology.orchestra.run.smithi142.stdout: "op_w": 0, 2024-03-31T23:37:29.707 INFO:teuthology.orchestra.run.smithi142.stdout: "op_rmw": 0, 2024-03-31T23:37:29.707 INFO:teuthology.orchestra.run.smithi142.stdout: "op_pg": 0, 2024-03-31T23:37:29.707 INFO:teuthology.orchestra.run.smithi142.stdout: "osdop_stat": 3749, 2024-03-31T23:37:29.707 INFO:teuthology.orchestra.run.smithi142.stdout: "osdop_create": 0, 2024-03-31T23:37:29.707 INFO:teuthology.orchestra.run.smithi142.stdout: "osdop_read": 2925, 2024-03-31T23:37:29.707 INFO:teuthology.orchestra.run.smithi142.stdout: "osdop_write": 0, 2024-03-31T23:37:29.707 INFO:teuthology.orchestra.run.smithi142.stdout: "osdop_writefull": 0, 2024-03-31T23:37:29.707 INFO:teuthology.orchestra.run.smithi142.stdout: "osdop_writesame": 0, 2024-03-31T23:37:29.707 INFO:teuthology.orchestra.run.smithi142.stdout: "osdop_append": 0, 2024-03-31T23:37:29.707 INFO:teuthology.orchestra.run.smithi142.stdout: "osdop_zero": 0, 2024-03-31T23:37:29.708 INFO:teuthology.orchestra.run.smithi142.stdout: "osdop_truncate": 0, 2024-03-31T23:37:29.708 INFO:teuthology.orchestra.run.smithi142.stdout: "osdop_delete": 0, 2024-03-31T23:37:29.708 INFO:teuthology.orchestra.run.smithi142.stdout: "osdop_mapext": 0, 2024-03-31T23:37:29.708 INFO:teuthology.orchestra.run.smithi142.stdout: "osdop_sparse_read": 0, 2024-03-31T23:37:29.708 INFO:teuthology.orchestra.run.smithi142.stdout: "osdop_clonerange": 0, 2024-03-31T23:37:29.708 INFO:teuthology.orchestra.run.smithi142.stdout: "osdop_getxattr": 1, 2024-03-31T23:37:29.708 INFO:teuthology.orchestra.run.smithi142.stdout: "osdop_setxattr": 0, 2024-03-31T23:37:29.708 INFO:teuthology.orchestra.run.smithi142.stdout: "osdop_cmpxattr": 0, 2024-03-31T23:37:29.708 INFO:teuthology.orchestra.run.smithi142.stdout: "osdop_rmxattr": 0, 2024-03-31T23:37:29.708 INFO:teuthology.orchestra.run.smithi142.stdout: "osdop_resetxattrs": 0, 2024-03-31T23:37:29.708 INFO:teuthology.orchestra.run.smithi142.stdout: "osdop_call": 0, 2024-03-31T23:37:29.708 INFO:teuthology.orchestra.run.smithi142.stdout: "osdop_watch": 0, 2024-03-31T23:37:29.708 INFO:teuthology.orchestra.run.smithi142.stdout: "osdop_notify": 0, 2024-03-31T23:37:29.708 INFO:teuthology.orchestra.run.smithi142.stdout: "osdop_src_cmpxattr": 0, 2024-03-31T23:37:29.708 INFO:teuthology.orchestra.run.smithi142.stdout: "osdop_pgls": 0, 2024-03-31T23:37:29.708 INFO:teuthology.orchestra.run.smithi142.stdout: "osdop_pgls_filter": 0, 2024-03-31T23:37:29.709 INFO:teuthology.orchestra.run.smithi142.stdout: "osdop_other": 0, 2024-03-31T23:37:29.709 INFO:teuthology.orchestra.run.smithi142.stdout: "linger_active": 0, 2024-03-31T23:37:29.709 INFO:teuthology.orchestra.run.smithi142.stdout: "linger_send": 0, 2024-03-31T23:37:29.709 INFO:teuthology.orchestra.run.smithi142.stdout: "linger_resend": 0, 2024-03-31T23:37:29.709 INFO:teuthology.orchestra.run.smithi142.stdout: "linger_ping": 0, 2024-03-31T23:37:29.709 INFO:teuthology.orchestra.run.smithi142.stdout: "poolop_active": 0, 2024-03-31T23:37:29.709 INFO:teuthology.orchestra.run.smithi142.stdout: "poolop_send": 0, 2024-03-31T23:37:29.709 INFO:teuthology.orchestra.run.smithi142.stdout: "poolop_resend": 0, 2024-03-31T23:37:29.709 INFO:teuthology.orchestra.run.smithi142.stdout: "poolstat_active": 0, 2024-03-31T23:37:29.709 INFO:teuthology.orchestra.run.smithi142.stdout: "poolstat_send": 0, 2024-03-31T23:37:29.709 INFO:teuthology.orchestra.run.smithi142.stdout: "poolstat_resend": 0, 2024-03-31T23:37:29.709 INFO:teuthology.orchestra.run.smithi142.stdout: "statfs_active": 0, 2024-03-31T23:37:29.709 INFO:teuthology.orchestra.run.smithi142.stdout: "statfs_send": 0, 2024-03-31T23:37:29.709 INFO:teuthology.orchestra.run.smithi142.stdout: "statfs_resend": 0, 2024-03-31T23:37:29.709 INFO:teuthology.orchestra.run.smithi142.stdout: "command_active": 0, 2024-03-31T23:37:29.709 INFO:teuthology.orchestra.run.smithi142.stdout: "command_send": 0, 2024-03-31T23:37:29.709 INFO:teuthology.orchestra.run.smithi142.stdout: "command_resend": 0, 2024-03-31T23:37:29.710 INFO:teuthology.orchestra.run.smithi142.stdout: "map_epoch": 139, 2024-03-31T23:37:29.710 INFO:teuthology.orchestra.run.smithi142.stdout: "map_full": 0, 2024-03-31T23:37:29.710 INFO:teuthology.orchestra.run.smithi142.stdout: "map_inc": 48, 2024-03-31T23:37:29.710 INFO:teuthology.orchestra.run.smithi142.stdout: "osd_sessions": 11, 2024-03-31T23:37:29.710 INFO:teuthology.orchestra.run.smithi142.stdout: "osd_session_open": 12, 2024-03-31T23:37:29.710 INFO:teuthology.orchestra.run.smithi142.stdout: "osd_session_close": 1, 2024-03-31T23:37:29.710 INFO:teuthology.orchestra.run.smithi142.stdout: "osd_laggy": 0, 2024-03-31T23:37:29.710 INFO:teuthology.orchestra.run.smithi142.stdout: "omap_wr": 0, 2024-03-31T23:37:29.710 INFO:teuthology.orchestra.run.smithi142.stdout: "omap_rd": 2, 2024-03-31T23:37:29.710 INFO:teuthology.orchestra.run.smithi142.stdout: "omap_del": 0 2024-03-31T23:37:29.710 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-03-31T23:37:29.710 INFO:teuthology.orchestra.run.smithi142.stdout: "oft": { 2024-03-31T23:37:29.710 INFO:teuthology.orchestra.run.smithi142.stdout: "omap_total_objs": 0, 2024-03-31T23:37:29.710 INFO:teuthology.orchestra.run.smithi142.stdout: "omap_total_kv_pairs": 0, 2024-03-31T23:37:29.710 INFO:teuthology.orchestra.run.smithi142.stdout: "omap_total_updates": 0, 2024-03-31T23:37:29.711 INFO:teuthology.orchestra.run.smithi142.stdout: "omap_total_removes": 0 2024-03-31T23:37:29.711 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-03-31T23:37:29.711 INFO:teuthology.orchestra.run.smithi142.stdout: "osd-slow-ops": { 2024-03-31T23:37:29.711 INFO:teuthology.orchestra.run.smithi142.stdout: "slow_ops_count": 0 2024-03-31T23:37:29.711 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-03-31T23:37:29.711 INFO:teuthology.orchestra.run.smithi142.stdout: "purge_queue": { 2024-03-31T23:37:29.711 INFO:teuthology.orchestra.run.smithi142.stdout: "pq_executing_ops": 0, 2024-03-31T23:37:29.711 INFO:teuthology.orchestra.run.smithi142.stdout: "pq_executing_ops_high_water": 0, 2024-03-31T23:37:29.711 INFO:teuthology.orchestra.run.smithi142.stdout: "pq_executing": 0, 2024-03-31T23:37:29.711 INFO:teuthology.orchestra.run.smithi142.stdout: "pq_executing_high_water": 0, 2024-03-31T23:37:29.711 INFO:teuthology.orchestra.run.smithi142.stdout: "pq_executed_ops": 0, 2024-03-31T23:37:29.711 INFO:teuthology.orchestra.run.smithi142.stdout: "pq_executed": 0, 2024-03-31T23:37:29.711 INFO:teuthology.orchestra.run.smithi142.stdout: "pq_item_in_journal": 0 2024-03-31T23:37:29.711 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-03-31T23:37:29.711 INFO:teuthology.orchestra.run.smithi142.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-31T23:37:29.712 INFO:teuthology.orchestra.run.smithi142.stdout: "val": 0, 2024-03-31T23:37:29.712 INFO:teuthology.orchestra.run.smithi142.stdout: "max": 104857600, 2024-03-31T23:37:29.712 INFO:teuthology.orchestra.run.smithi142.stdout: "get_started": 0, 2024-03-31T23:37:29.712 INFO:teuthology.orchestra.run.smithi142.stdout: "get": 7234, 2024-03-31T23:37:29.712 INFO:teuthology.orchestra.run.smithi142.stdout: "get_sum": 687157270, 2024-03-31T23:37:29.712 INFO:teuthology.orchestra.run.smithi142.stdout: "get_or_fail_fail": 0, 2024-03-31T23:37:29.712 INFO:teuthology.orchestra.run.smithi142.stdout: "get_or_fail_success": 7234, 2024-03-31T23:37:29.712 INFO:teuthology.orchestra.run.smithi142.stdout: "take": 0, 2024-03-31T23:37:29.712 INFO:teuthology.orchestra.run.smithi142.stdout: "take_sum": 0, 2024-03-31T23:37:29.712 INFO:teuthology.orchestra.run.smithi142.stdout: "put": 7234, 2024-03-31T23:37:29.712 INFO:teuthology.orchestra.run.smithi142.stdout: "put_sum": 687157270, 2024-03-31T23:37:29.712 INFO:teuthology.orchestra.run.smithi142.stdout: "wait": { 2024-03-31T23:37:29.712 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 0, 2024-03-31T23:37:29.712 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.000000000, 2024-03-31T23:37:29.712 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000000000 2024-03-31T23:37:29.712 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-03-31T23:37:29.713 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-03-31T23:37:29.713 INFO:teuthology.orchestra.run.smithi142.stdout: "throttle-objecter_bytes": { 2024-03-31T23:37:29.713 INFO:teuthology.orchestra.run.smithi142.stdout: "val": 0, 2024-03-31T23:37:29.713 INFO:teuthology.orchestra.run.smithi142.stdout: "max": 104857600, 2024-03-31T23:37:29.713 INFO:teuthology.orchestra.run.smithi142.stdout: "get_started": 0, 2024-03-31T23:37:29.713 INFO:teuthology.orchestra.run.smithi142.stdout: "get": 0, 2024-03-31T23:37:29.713 INFO:teuthology.orchestra.run.smithi142.stdout: "get_sum": 0, 2024-03-31T23:37:29.713 INFO:teuthology.orchestra.run.smithi142.stdout: "get_or_fail_fail": 0, 2024-03-31T23:37:29.713 INFO:teuthology.orchestra.run.smithi142.stdout: "get_or_fail_success": 0, 2024-03-31T23:37:29.713 INFO:teuthology.orchestra.run.smithi142.stdout: "take": 6676, 2024-03-31T23:37:29.713 INFO:teuthology.orchestra.run.smithi142.stdout: "take_sum": 684278861, 2024-03-31T23:37:29.713 INFO:teuthology.orchestra.run.smithi142.stdout: "put": 6676, 2024-03-31T23:37:29.713 INFO:teuthology.orchestra.run.smithi142.stdout: "put_sum": 684278861, 2024-03-31T23:37:29.713 INFO:teuthology.orchestra.run.smithi142.stdout: "wait": { 2024-03-31T23:37:29.713 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 0, 2024-03-31T23:37:29.713 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.000000000, 2024-03-31T23:37:29.714 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000000000 2024-03-31T23:37:29.714 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-03-31T23:37:29.714 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-03-31T23:37:29.714 INFO:teuthology.orchestra.run.smithi142.stdout: "throttle-objecter_ops": { 2024-03-31T23:37:29.714 INFO:teuthology.orchestra.run.smithi142.stdout: "val": 0, 2024-03-31T23:37:29.714 INFO:teuthology.orchestra.run.smithi142.stdout: "max": 1024, 2024-03-31T23:37:29.714 INFO:teuthology.orchestra.run.smithi142.stdout: "get_started": 0, 2024-03-31T23:37:29.714 INFO:teuthology.orchestra.run.smithi142.stdout: "get": 0, 2024-03-31T23:37:29.714 INFO:teuthology.orchestra.run.smithi142.stdout: "get_sum": 0, 2024-03-31T23:37:29.714 INFO:teuthology.orchestra.run.smithi142.stdout: "get_or_fail_fail": 0, 2024-03-31T23:37:29.714 INFO:teuthology.orchestra.run.smithi142.stdout: "get_or_fail_success": 0, 2024-03-31T23:37:29.714 INFO:teuthology.orchestra.run.smithi142.stdout: "take": 6676, 2024-03-31T23:37:29.714 INFO:teuthology.orchestra.run.smithi142.stdout: "take_sum": 6676, 2024-03-31T23:37:29.714 INFO:teuthology.orchestra.run.smithi142.stdout: "put": 6676, 2024-03-31T23:37:29.714 INFO:teuthology.orchestra.run.smithi142.stdout: "put_sum": 6676, 2024-03-31T23:37:29.714 INFO:teuthology.orchestra.run.smithi142.stdout: "wait": { 2024-03-31T23:37:29.715 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 0, 2024-03-31T23:37:29.715 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.000000000, 2024-03-31T23:37:29.715 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000000000 2024-03-31T23:37:29.715 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-03-31T23:37:29.715 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-03-31T23:37:29.715 INFO:teuthology.orchestra.run.smithi142.stdout: "throttle-write_buf_throttle": { 2024-03-31T23:37:29.715 INFO:teuthology.orchestra.run.smithi142.stdout: "val": 0, 2024-03-31T23:37:29.715 INFO:teuthology.orchestra.run.smithi142.stdout: "max": 3758096384, 2024-03-31T23:37:29.715 INFO:teuthology.orchestra.run.smithi142.stdout: "get_started": 0, 2024-03-31T23:37:29.715 INFO:teuthology.orchestra.run.smithi142.stdout: "get": 0, 2024-03-31T23:37:29.715 INFO:teuthology.orchestra.run.smithi142.stdout: "get_sum": 0, 2024-03-31T23:37:29.715 INFO:teuthology.orchestra.run.smithi142.stdout: "get_or_fail_fail": 0, 2024-03-31T23:37:29.715 INFO:teuthology.orchestra.run.smithi142.stdout: "get_or_fail_success": 0, 2024-03-31T23:37:29.715 INFO:teuthology.orchestra.run.smithi142.stdout: "take": 0, 2024-03-31T23:37:29.715 INFO:teuthology.orchestra.run.smithi142.stdout: "take_sum": 0, 2024-03-31T23:37:29.715 INFO:teuthology.orchestra.run.smithi142.stdout: "put": 0, 2024-03-31T23:37:29.715 INFO:teuthology.orchestra.run.smithi142.stdout: "put_sum": 0, 2024-03-31T23:37:29.716 INFO:teuthology.orchestra.run.smithi142.stdout: "wait": { 2024-03-31T23:37:29.716 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 0, 2024-03-31T23:37:29.716 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.000000000, 2024-03-31T23:37:29.716 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000000000 2024-03-31T23:37:29.716 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-03-31T23:37:29.716 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-03-31T23:37:29.716 INFO:teuthology.orchestra.run.smithi142.stdout: "throttle-write_buf_throttle-0x564b7278e6e0": { 2024-03-31T23:37:29.716 INFO:teuthology.orchestra.run.smithi142.stdout: "val": 0, 2024-03-31T23:37:29.716 INFO:teuthology.orchestra.run.smithi142.stdout: "max": 3758096384, 2024-03-31T23:37:29.716 INFO:teuthology.orchestra.run.smithi142.stdout: "get_started": 0, 2024-03-31T23:37:29.716 INFO:teuthology.orchestra.run.smithi142.stdout: "get": 0, 2024-03-31T23:37:29.716 INFO:teuthology.orchestra.run.smithi142.stdout: "get_sum": 0, 2024-03-31T23:37:29.716 INFO:teuthology.orchestra.run.smithi142.stdout: "get_or_fail_fail": 0, 2024-03-31T23:37:29.716 INFO:teuthology.orchestra.run.smithi142.stdout: "get_or_fail_success": 0, 2024-03-31T23:37:29.716 INFO:teuthology.orchestra.run.smithi142.stdout: "take": 0, 2024-03-31T23:37:29.716 INFO:teuthology.orchestra.run.smithi142.stdout: "take_sum": 0, 2024-03-31T23:37:29.717 INFO:teuthology.orchestra.run.smithi142.stdout: "put": 0, 2024-03-31T23:37:29.717 INFO:teuthology.orchestra.run.smithi142.stdout: "put_sum": 0, 2024-03-31T23:37:29.717 INFO:teuthology.orchestra.run.smithi142.stdout: "wait": { 2024-03-31T23:37:29.717 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 0, 2024-03-31T23:37:29.717 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.000000000, 2024-03-31T23:37:29.717 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000000000 2024-03-31T23:37:29.717 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-03-31T23:37:29.717 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-03-31T23:37:29.717 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-03-31T23:37:30.678 INFO:tasks.check_counter:Daemon mds.e mds.exported=0 2024-03-31T23:37:30.678 INFO:tasks.check_counter:Daemon mds.e mds.imported=0 2024-03-31T23:37:30.678 INFO:tasks.check_counter:Daemon mds.e mds_cache.dir_update=0 2024-03-31T23:37:30.678 INFO:tasks.check_counter:Daemon mds.e mds_cache.dir_update_receipt=0 2024-03-31T23:37:30.678 INFO:tasks.check_counter:Daemon mds.e mds.root_rsnaps=0 2024-03-31T23:37:30.678 INFO:tasks.check_counter:Daemon mds.e mds_server.req_mksnap_latency.avgcount=0 2024-03-31T23:37:30.678 INFO:tasks.check_counter:Daemon mds.e mds_server.req_rmsnap_latency.avgcount=0 2024-03-31T23:37:30.678 DEBUG:tasks.check_counter:Getting stats from h 2024-03-31T23:37:30.678 DEBUG:teuthology.orchestra.run.smithi142:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:384f31f3d3aee7f8ba3f60846b8844c049165583 shell --fsid 6a9d8f08-efb2-11ee-b647-cb9ed24678a4 -- ceph daemon mds.h perf dump 2024-03-31T23:37:30.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:37:30 smithi142 ceph-mon[35271]: pgmap v819: 129 pgs: 129 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 8.3 KiB/s wr, 8 op/s 2024-03-31T23:37:30.945 INFO:teuthology.orchestra.run.smithi142.stderr:Inferring config /var/lib/ceph/6a9d8f08-efb2-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-31T23:37:31.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:37:30 smithi162 ceph-mon[36237]: pgmap v819: 129 pgs: 129 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 8.3 KiB/s wr, 8 op/s 2024-03-31T23:37:31.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:37:30 smithi046 ceph-mon[32498]: pgmap v819: 129 pgs: 129 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 8.3 KiB/s wr, 8 op/s 2024-03-31T23:37:32.649 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-03-31T23:37:32.649 INFO:teuthology.orchestra.run.smithi142.stdout: "AsyncMessenger::Worker-0": { 2024-03-31T23:37:32.650 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_recv_messages": 285121, 2024-03-31T23:37:32.650 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_send_messages": 283793, 2024-03-31T23:37:32.650 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_recv_bytes": 83926616, 2024-03-31T23:37:32.650 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_send_bytes": 304579879, 2024-03-31T23:37:32.650 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_created_connections": 18, 2024-03-31T23:37:32.650 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_active_connections": 5, 2024-03-31T23:37:32.650 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_running_total_time": 19.443949040, 2024-03-31T23:37:32.650 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_running_send_time": 9.238000861, 2024-03-31T23:37:32.650 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_running_recv_time": 11.576317149, 2024-03-31T23:37:32.650 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_running_fast_dispatch_time": 0.904404222, 2024-03-31T23:37:32.650 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_send_messages_queue_lat": { 2024-03-31T23:37:32.650 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 283789, 2024-03-31T23:37:32.650 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 11.720838292, 2024-03-31T23:37:32.651 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000041301 2024-03-31T23:37:32.651 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-03-31T23:37:32.651 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_handle_ack_lat": { 2024-03-31T23:37:32.651 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 511746, 2024-03-31T23:37:32.651 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.098617501, 2024-03-31T23:37:32.651 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000000192 2024-03-31T23:37:32.651 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-03-31T23:37:32.651 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_recv_encrypted_bytes": 5168, 2024-03-31T23:37:32.651 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_send_encrypted_bytes": 3952 2024-03-31T23:37:32.651 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-03-31T23:37:32.651 INFO:teuthology.orchestra.run.smithi142.stdout: "AsyncMessenger::Worker-1": { 2024-03-31T23:37:32.651 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_recv_messages": 1018864, 2024-03-31T23:37:32.651 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_send_messages": 1119437, 2024-03-31T23:37:32.651 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_recv_bytes": 305078883, 2024-03-31T23:37:32.651 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_send_bytes": 929528742, 2024-03-31T23:37:32.651 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_created_connections": 16, 2024-03-31T23:37:32.652 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_active_connections": 5, 2024-03-31T23:37:32.652 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_running_total_time": 54.236712637, 2024-03-31T23:37:32.652 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_running_send_time": 22.886643240, 2024-03-31T23:37:32.652 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_running_recv_time": 37.315617026, 2024-03-31T23:37:32.652 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_running_fast_dispatch_time": 1.226643729, 2024-03-31T23:37:32.652 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_send_messages_queue_lat": { 2024-03-31T23:37:32.652 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 1119437, 2024-03-31T23:37:32.652 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 51.082648884, 2024-03-31T23:37:32.652 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000045632 2024-03-31T23:37:32.652 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-03-31T23:37:32.652 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_handle_ack_lat": { 2024-03-31T23:37:32.652 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 2038415, 2024-03-31T23:37:32.652 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.394169563, 2024-03-31T23:37:32.652 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000000193 2024-03-31T23:37:32.653 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-03-31T23:37:32.653 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_recv_encrypted_bytes": 2256, 2024-03-31T23:37:32.653 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_send_encrypted_bytes": 777984 2024-03-31T23:37:32.653 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-03-31T23:37:32.653 INFO:teuthology.orchestra.run.smithi142.stdout: "AsyncMessenger::Worker-2": { 2024-03-31T23:37:32.653 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_recv_messages": 32498, 2024-03-31T23:37:32.653 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_send_messages": 32552, 2024-03-31T23:37:32.653 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_recv_bytes": 11796344, 2024-03-31T23:37:32.653 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_send_bytes": 245671184, 2024-03-31T23:37:32.653 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_created_connections": 9, 2024-03-31T23:37:32.653 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_active_connections": 6, 2024-03-31T23:37:32.653 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_running_total_time": 3.775504008, 2024-03-31T23:37:32.653 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_running_send_time": 1.404489176, 2024-03-31T23:37:32.653 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_running_recv_time": 0.653649918, 2024-03-31T23:37:32.654 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_running_fast_dispatch_time": 1.551909279, 2024-03-31T23:37:32.654 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_send_messages_queue_lat": { 2024-03-31T23:37:32.654 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 32551, 2024-03-31T23:37:32.654 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 3.542046035, 2024-03-31T23:37:32.654 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000108815 2024-03-31T23:37:32.654 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-03-31T23:37:32.654 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_handle_ack_lat": { 2024-03-31T23:37:32.654 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 141, 2024-03-31T23:37:32.654 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.000028744, 2024-03-31T23:37:32.654 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000000203 2024-03-31T23:37:32.654 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-03-31T23:37:32.654 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_recv_encrypted_bytes": 1731808, 2024-03-31T23:37:32.654 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_send_encrypted_bytes": 328288 2024-03-31T23:37:32.654 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-03-31T23:37:32.655 INFO:teuthology.orchestra.run.smithi142.stdout: "cct": { 2024-03-31T23:37:32.655 INFO:teuthology.orchestra.run.smithi142.stdout: "total_workers": 1, 2024-03-31T23:37:32.655 INFO:teuthology.orchestra.run.smithi142.stdout: "unhealthy_workers": 0 2024-03-31T23:37:32.655 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-03-31T23:37:32.655 INFO:teuthology.orchestra.run.smithi142.stdout: "finisher-MDSRank": { 2024-03-31T23:37:32.655 INFO:teuthology.orchestra.run.smithi142.stdout: "queue_len": 0, 2024-03-31T23:37:32.655 INFO:teuthology.orchestra.run.smithi142.stdout: "complete_latency": { 2024-03-31T23:37:32.655 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 72119, 2024-03-31T23:37:32.655 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 97.807316463, 2024-03-31T23:37:32.655 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.001356193 2024-03-31T23:37:32.655 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-03-31T23:37:32.655 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-03-31T23:37:32.655 INFO:teuthology.orchestra.run.smithi142.stdout: "finisher-PurgeQueue": { 2024-03-31T23:37:32.655 INFO:teuthology.orchestra.run.smithi142.stdout: "queue_len": 0, 2024-03-31T23:37:32.655 INFO:teuthology.orchestra.run.smithi142.stdout: "complete_latency": { 2024-03-31T23:37:32.656 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 32420, 2024-03-31T23:37:32.656 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 6.660478068, 2024-03-31T23:37:32.656 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000205443 2024-03-31T23:37:32.656 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-03-31T23:37:32.656 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-03-31T23:37:32.656 INFO:teuthology.orchestra.run.smithi142.stdout: "mds": { 2024-03-31T23:37:32.656 INFO:teuthology.orchestra.run.smithi142.stdout: "request": 649299, 2024-03-31T23:37:32.656 INFO:teuthology.orchestra.run.smithi142.stdout: "reply": 645665, 2024-03-31T23:37:32.656 INFO:teuthology.orchestra.run.smithi142.stdout: "reply_latency": { 2024-03-31T23:37:32.656 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 645665, 2024-03-31T23:37:32.656 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 1093.858538488, 2024-03-31T23:37:32.656 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.001694158 2024-03-31T23:37:32.656 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-03-31T23:37:32.656 INFO:teuthology.orchestra.run.smithi142.stdout: "slow_reply": 0, 2024-03-31T23:37:32.656 INFO:teuthology.orchestra.run.smithi142.stdout: "forward": 3634, 2024-03-31T23:37:32.657 INFO:teuthology.orchestra.run.smithi142.stdout: "dir_fetch_complete": 15, 2024-03-31T23:37:32.657 INFO:teuthology.orchestra.run.smithi142.stdout: "dir_fetch_keys": 0, 2024-03-31T23:37:32.657 INFO:teuthology.orchestra.run.smithi142.stdout: "dir_commit": 176, 2024-03-31T23:37:32.657 INFO:teuthology.orchestra.run.smithi142.stdout: "dir_split": 13, 2024-03-31T23:37:32.657 INFO:teuthology.orchestra.run.smithi142.stdout: "dir_merge": 5, 2024-03-31T23:37:32.657 INFO:teuthology.orchestra.run.smithi142.stdout: "inodes": 3492, 2024-03-31T23:37:32.657 INFO:teuthology.orchestra.run.smithi142.stdout: "inodes_top": 1539, 2024-03-31T23:37:32.657 INFO:teuthology.orchestra.run.smithi142.stdout: "inodes_bottom": 1913, 2024-03-31T23:37:32.657 INFO:teuthology.orchestra.run.smithi142.stdout: "inodes_pin_tail": 40, 2024-03-31T23:37:32.657 INFO:teuthology.orchestra.run.smithi142.stdout: "inodes_pinned": 1291, 2024-03-31T23:37:32.657 INFO:teuthology.orchestra.run.smithi142.stdout: "inodes_expired": 1820, 2024-03-31T23:37:32.657 INFO:teuthology.orchestra.run.smithi142.stdout: "inodes_with_caps": 4, 2024-03-31T23:37:32.657 INFO:teuthology.orchestra.run.smithi142.stdout: "caps": 4, 2024-03-31T23:37:32.657 INFO:teuthology.orchestra.run.smithi142.stdout: "subtrees": 4, 2024-03-31T23:37:32.658 INFO:teuthology.orchestra.run.smithi142.stdout: "traverse": 1247738, 2024-03-31T23:37:32.658 INFO:teuthology.orchestra.run.smithi142.stdout: "traverse_hit": 1160078, 2024-03-31T23:37:32.658 INFO:teuthology.orchestra.run.smithi142.stdout: "traverse_forward": 3617, 2024-03-31T23:37:32.658 INFO:teuthology.orchestra.run.smithi142.stdout: "traverse_discover": 561, 2024-03-31T23:37:32.658 INFO:teuthology.orchestra.run.smithi142.stdout: "traverse_dir_fetch": 1, 2024-03-31T23:37:32.658 INFO:teuthology.orchestra.run.smithi142.stdout: "traverse_remote_ino": 0, 2024-03-31T23:37:32.658 INFO:teuthology.orchestra.run.smithi142.stdout: "traverse_lock": 633, 2024-03-31T23:37:32.658 INFO:teuthology.orchestra.run.smithi142.stdout: "load_cent": 0, 2024-03-31T23:37:32.658 INFO:teuthology.orchestra.run.smithi142.stdout: "q": 0, 2024-03-31T23:37:32.658 INFO:teuthology.orchestra.run.smithi142.stdout: "exported": 53, 2024-03-31T23:37:32.658 INFO:teuthology.orchestra.run.smithi142.stdout: "exported_inodes": 1382, 2024-03-31T23:37:32.658 INFO:teuthology.orchestra.run.smithi142.stdout: "imported": 33, 2024-03-31T23:37:32.658 INFO:teuthology.orchestra.run.smithi142.stdout: "imported_inodes": 1439, 2024-03-31T23:37:32.658 INFO:teuthology.orchestra.run.smithi142.stdout: "openino_dir_fetch": 0, 2024-03-31T23:37:32.658 INFO:teuthology.orchestra.run.smithi142.stdout: "openino_backtrace_fetch": 29, 2024-03-31T23:37:32.659 INFO:teuthology.orchestra.run.smithi142.stdout: "openino_peer_discover": 0, 2024-03-31T23:37:32.659 INFO:teuthology.orchestra.run.smithi142.stdout: "root_rfiles": 4, 2024-03-31T23:37:32.659 INFO:teuthology.orchestra.run.smithi142.stdout: "root_rbytes": 234, 2024-03-31T23:37:32.659 INFO:teuthology.orchestra.run.smithi142.stdout: "root_rsnaps": 6, 2024-03-31T23:37:32.659 INFO:teuthology.orchestra.run.smithi142.stdout: "scrub_backtrace_fetch": 29, 2024-03-31T23:37:32.659 INFO:teuthology.orchestra.run.smithi142.stdout: "scrub_set_tag": 0, 2024-03-31T23:37:32.659 INFO:teuthology.orchestra.run.smithi142.stdout: "scrub_backtrace_repaired": 0, 2024-03-31T23:37:32.659 INFO:teuthology.orchestra.run.smithi142.stdout: "scrub_inotable_repaired": 0, 2024-03-31T23:37:32.659 INFO:teuthology.orchestra.run.smithi142.stdout: "scrub_dir_inodes": 0, 2024-03-31T23:37:32.659 INFO:teuthology.orchestra.run.smithi142.stdout: "scrub_dir_base_inodes": 0, 2024-03-31T23:37:32.659 INFO:teuthology.orchestra.run.smithi142.stdout: "scrub_dirfrag_rstats": 0, 2024-03-31T23:37:32.659 INFO:teuthology.orchestra.run.smithi142.stdout: "scrub_file_inodes": 29, 2024-03-31T23:37:32.659 INFO:teuthology.orchestra.run.smithi142.stdout: "handle_inode_file_caps": 362, 2024-03-31T23:37:32.659 INFO:teuthology.orchestra.run.smithi142.stdout: "ceph_cap_op_revoke": 35446, 2024-03-31T23:37:32.660 INFO:teuthology.orchestra.run.smithi142.stdout: "ceph_cap_op_grant": 55050, 2024-03-31T23:37:32.660 INFO:teuthology.orchestra.run.smithi142.stdout: "ceph_cap_op_trunc": 0, 2024-03-31T23:37:32.660 INFO:teuthology.orchestra.run.smithi142.stdout: "ceph_cap_op_flushsnap_ack": 6, 2024-03-31T23:37:32.660 INFO:teuthology.orchestra.run.smithi142.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-31T23:37:32.660 INFO:teuthology.orchestra.run.smithi142.stdout: "handle_client_caps": 70452, 2024-03-31T23:37:32.660 INFO:teuthology.orchestra.run.smithi142.stdout: "handle_client_caps_dirty": 26241, 2024-03-31T23:37:32.660 INFO:teuthology.orchestra.run.smithi142.stdout: "handle_client_cap_release": 1078, 2024-03-31T23:37:32.660 INFO:teuthology.orchestra.run.smithi142.stdout: "process_request_cap_release": 65401 2024-03-31T23:37:32.660 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-03-31T23:37:32.660 INFO:teuthology.orchestra.run.smithi142.stdout: "mds_cache": { 2024-03-31T23:37:32.660 INFO:teuthology.orchestra.run.smithi142.stdout: "dir_update": 1222, 2024-03-31T23:37:32.660 INFO:teuthology.orchestra.run.smithi142.stdout: "dir_update_receipt": 6372, 2024-03-31T23:37:32.660 INFO:teuthology.orchestra.run.smithi142.stdout: "dir_try_discover": 3243, 2024-03-31T23:37:32.660 INFO:teuthology.orchestra.run.smithi142.stdout: "dir_send_discover": 7105, 2024-03-31T23:37:32.661 INFO:teuthology.orchestra.run.smithi142.stdout: "dir_handle_discover": 1543, 2024-03-31T23:37:32.661 INFO:teuthology.orchestra.run.smithi142.stdout: "num_strays": 30, 2024-03-31T23:37:32.661 INFO:teuthology.orchestra.run.smithi142.stdout: "num_strays_delayed": 0, 2024-03-31T23:37:32.661 INFO:teuthology.orchestra.run.smithi142.stdout: "num_strays_enqueuing": 0, 2024-03-31T23:37:32.661 INFO:teuthology.orchestra.run.smithi142.stdout: "strays_created": 18021, 2024-03-31T23:37:32.661 INFO:teuthology.orchestra.run.smithi142.stdout: "strays_enqueued": 17991, 2024-03-31T23:37:32.661 INFO:teuthology.orchestra.run.smithi142.stdout: "strays_reintegrated": 0, 2024-03-31T23:37:32.661 INFO:teuthology.orchestra.run.smithi142.stdout: "strays_migrated": 0, 2024-03-31T23:37:32.661 INFO:teuthology.orchestra.run.smithi142.stdout: "num_recovering_processing": 0, 2024-03-31T23:37:32.661 INFO:teuthology.orchestra.run.smithi142.stdout: "num_recovering_enqueued": 0, 2024-03-31T23:37:32.661 INFO:teuthology.orchestra.run.smithi142.stdout: "num_recovering_prioritized": 0, 2024-03-31T23:37:32.661 INFO:teuthology.orchestra.run.smithi142.stdout: "recovery_started": 0, 2024-03-31T23:37:32.661 INFO:teuthology.orchestra.run.smithi142.stdout: "recovery_completed": 0, 2024-03-31T23:37:32.661 INFO:teuthology.orchestra.run.smithi142.stdout: "ireq_quiesce_path": 0, 2024-03-31T23:37:32.661 INFO:teuthology.orchestra.run.smithi142.stdout: "ireq_quiesce_inode": 0, 2024-03-31T23:37:32.662 INFO:teuthology.orchestra.run.smithi142.stdout: "ireq_enqueue_scrub": 0, 2024-03-31T23:37:32.662 INFO:teuthology.orchestra.run.smithi142.stdout: "ireq_exportdir": 57, 2024-03-31T23:37:32.662 INFO:teuthology.orchestra.run.smithi142.stdout: "ireq_flush": 0, 2024-03-31T23:37:32.662 INFO:teuthology.orchestra.run.smithi142.stdout: "ireq_fragmentdir": 18, 2024-03-31T23:37:32.662 INFO:teuthology.orchestra.run.smithi142.stdout: "ireq_fragstats": 0, 2024-03-31T23:37:32.662 INFO:teuthology.orchestra.run.smithi142.stdout: "ireq_inodestats": 0 2024-03-31T23:37:32.662 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-03-31T23:37:32.662 INFO:teuthology.orchestra.run.smithi142.stdout: "mds_log": { 2024-03-31T23:37:32.662 INFO:teuthology.orchestra.run.smithi142.stdout: "evlrg": 0, 2024-03-31T23:37:32.662 INFO:teuthology.orchestra.run.smithi142.stdout: "evadd": 149835, 2024-03-31T23:37:32.662 INFO:teuthology.orchestra.run.smithi142.stdout: "evex": 68713, 2024-03-31T23:37:32.662 INFO:teuthology.orchestra.run.smithi142.stdout: "evtrm": 61451, 2024-03-31T23:37:32.662 INFO:teuthology.orchestra.run.smithi142.stdout: "ev": 88384, 2024-03-31T23:37:32.663 INFO:teuthology.orchestra.run.smithi142.stdout: "evexg": 0, 2024-03-31T23:37:32.663 INFO:teuthology.orchestra.run.smithi142.stdout: "evexd": 7262, 2024-03-31T23:37:32.663 INFO:teuthology.orchestra.run.smithi142.stdout: "segadd": 220, 2024-03-31T23:37:32.663 INFO:teuthology.orchestra.run.smithi142.stdout: "segex": 92, 2024-03-31T23:37:32.663 INFO:teuthology.orchestra.run.smithi142.stdout: "segtrm": 84, 2024-03-31T23:37:32.663 INFO:teuthology.orchestra.run.smithi142.stdout: "seg": 136, 2024-03-31T23:37:32.663 INFO:teuthology.orchestra.run.smithi142.stdout: "segmjr": 8, 2024-03-31T23:37:32.663 INFO:teuthology.orchestra.run.smithi142.stdout: "segexg": 0, 2024-03-31T23:37:32.663 INFO:teuthology.orchestra.run.smithi142.stdout: "segexd": 8, 2024-03-31T23:37:32.663 INFO:teuthology.orchestra.run.smithi142.stdout: "expos": 151235441, 2024-03-31T23:37:32.663 INFO:teuthology.orchestra.run.smithi142.stdout: "wrpos": 688455771, 2024-03-31T23:37:32.663 INFO:teuthology.orchestra.run.smithi142.stdout: "rdpos": 0, 2024-03-31T23:37:32.663 INFO:teuthology.orchestra.run.smithi142.stdout: "jlat": { 2024-03-31T23:37:32.663 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 36318, 2024-03-31T23:37:32.663 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 135.269555732, 2024-03-31T23:37:32.664 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.003724587 2024-03-31T23:37:32.664 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-03-31T23:37:32.664 INFO:teuthology.orchestra.run.smithi142.stdout: "replayed": 0 2024-03-31T23:37:32.664 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-03-31T23:37:32.664 INFO:teuthology.orchestra.run.smithi142.stdout: "mds_mem": { 2024-03-31T23:37:32.664 INFO:teuthology.orchestra.run.smithi142.stdout: "ino": 3051, 2024-03-31T23:37:32.664 INFO:teuthology.orchestra.run.smithi142.stdout: "ino+": 22554, 2024-03-31T23:37:32.664 INFO:teuthology.orchestra.run.smithi142.stdout: "ino-": 19503, 2024-03-31T23:37:32.664 INFO:teuthology.orchestra.run.smithi142.stdout: "dir": 139, 2024-03-31T23:37:32.664 INFO:teuthology.orchestra.run.smithi142.stdout: "dir+": 3745, 2024-03-31T23:37:32.664 INFO:teuthology.orchestra.run.smithi142.stdout: "dir-": 3606, 2024-03-31T23:37:32.664 INFO:teuthology.orchestra.run.smithi142.stdout: "dn": 3494, 2024-03-31T23:37:32.664 INFO:teuthology.orchestra.run.smithi142.stdout: "dn+": 33796, 2024-03-31T23:37:32.664 INFO:teuthology.orchestra.run.smithi142.stdout: "dn-": 30302, 2024-03-31T23:37:32.665 INFO:teuthology.orchestra.run.smithi142.stdout: "cap": 4, 2024-03-31T23:37:32.665 INFO:teuthology.orchestra.run.smithi142.stdout: "cap+": 18096, 2024-03-31T23:37:32.665 INFO:teuthology.orchestra.run.smithi142.stdout: "cap-": 18092, 2024-03-31T23:37:32.665 INFO:teuthology.orchestra.run.smithi142.stdout: "rss": 104836, 2024-03-31T23:37:32.665 INFO:teuthology.orchestra.run.smithi142.stdout: "heap": 223612 2024-03-31T23:37:32.665 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-03-31T23:37:32.665 INFO:teuthology.orchestra.run.smithi142.stdout: "mds_server": { 2024-03-31T23:37:32.665 INFO:teuthology.orchestra.run.smithi142.stdout: "dispatch_client_request": 774012, 2024-03-31T23:37:32.665 INFO:teuthology.orchestra.run.smithi142.stdout: "dispatch_server_request": 805, 2024-03-31T23:37:32.665 INFO:teuthology.orchestra.run.smithi142.stdout: "handle_client_request": 649299, 2024-03-31T23:37:32.665 INFO:teuthology.orchestra.run.smithi142.stdout: "handle_client_session": 1455, 2024-03-31T23:37:32.665 INFO:teuthology.orchestra.run.smithi142.stdout: "handle_peer_request": 19250, 2024-03-31T23:37:32.665 INFO:teuthology.orchestra.run.smithi142.stdout: "req_create_latency": { 2024-03-31T23:37:32.665 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 17735, 2024-03-31T23:37:32.666 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 89.242000061, 2024-03-31T23:37:32.666 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.005031970 2024-03-31T23:37:32.666 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-03-31T23:37:32.666 INFO:teuthology.orchestra.run.smithi142.stdout: "req_getattr_latency": { 2024-03-31T23:37:32.666 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 494786, 2024-03-31T23:37:32.666 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 492.432473163, 2024-03-31T23:37:32.666 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000995243 2024-03-31T23:37:32.666 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-03-31T23:37:32.666 INFO:teuthology.orchestra.run.smithi142.stdout: "req_getfilelock_latency": { 2024-03-31T23:37:32.666 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 0, 2024-03-31T23:37:32.666 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.000000000, 2024-03-31T23:37:32.666 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000000000 2024-03-31T23:37:32.666 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-03-31T23:37:32.666 INFO:teuthology.orchestra.run.smithi142.stdout: "req_link_latency": { 2024-03-31T23:37:32.667 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 0, 2024-03-31T23:37:32.667 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.000000000, 2024-03-31T23:37:32.667 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000000000 2024-03-31T23:37:32.667 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-03-31T23:37:32.667 INFO:teuthology.orchestra.run.smithi142.stdout: "req_lookup_latency": { 2024-03-31T23:37:32.667 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 31594, 2024-03-31T23:37:32.667 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 127.665480862, 2024-03-31T23:37:32.667 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.004040814 2024-03-31T23:37:32.667 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-03-31T23:37:32.667 INFO:teuthology.orchestra.run.smithi142.stdout: "req_lookuphash_latency": { 2024-03-31T23:37:32.667 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 0, 2024-03-31T23:37:32.667 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.000000000, 2024-03-31T23:37:32.667 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000000000 2024-03-31T23:37:32.667 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-03-31T23:37:32.668 INFO:teuthology.orchestra.run.smithi142.stdout: "req_lookupino_latency": { 2024-03-31T23:37:32.668 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 0, 2024-03-31T23:37:32.668 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.000000000, 2024-03-31T23:37:32.668 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000000000 2024-03-31T23:37:32.668 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-03-31T23:37:32.668 INFO:teuthology.orchestra.run.smithi142.stdout: "req_lookupname_latency": { 2024-03-31T23:37:32.668 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 0, 2024-03-31T23:37:32.668 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.000000000, 2024-03-31T23:37:32.668 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000000000 2024-03-31T23:37:32.668 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-03-31T23:37:32.668 INFO:teuthology.orchestra.run.smithi142.stdout: "req_lookupparent_latency": { 2024-03-31T23:37:32.668 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 0, 2024-03-31T23:37:32.668 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.000000000, 2024-03-31T23:37:32.668 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000000000 2024-03-31T23:37:32.668 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-03-31T23:37:32.669 INFO:teuthology.orchestra.run.smithi142.stdout: "req_lookupsnap_latency": { 2024-03-31T23:37:32.669 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 0, 2024-03-31T23:37:32.669 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.000000000, 2024-03-31T23:37:32.669 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000000000 2024-03-31T23:37:32.669 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-03-31T23:37:32.669 INFO:teuthology.orchestra.run.smithi142.stdout: "req_lssnap_latency": { 2024-03-31T23:37:32.670 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 0, 2024-03-31T23:37:32.670 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.000000000, 2024-03-31T23:37:32.670 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000000000 2024-03-31T23:37:32.670 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-03-31T23:37:32.670 INFO:teuthology.orchestra.run.smithi142.stdout: "req_mkdir_latency": { 2024-03-31T23:37:32.670 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 0, 2024-03-31T23:37:32.670 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.000000000, 2024-03-31T23:37:32.670 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000000000 2024-03-31T23:37:32.670 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-03-31T23:37:32.670 INFO:teuthology.orchestra.run.smithi142.stdout: "req_mknod_latency": { 2024-03-31T23:37:32.670 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 0, 2024-03-31T23:37:32.670 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.000000000, 2024-03-31T23:37:32.670 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000000000 2024-03-31T23:37:32.670 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-03-31T23:37:32.671 INFO:teuthology.orchestra.run.smithi142.stdout: "req_mksnap_latency": { 2024-03-31T23:37:32.671 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 0, 2024-03-31T23:37:32.671 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.000000000, 2024-03-31T23:37:32.671 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000000000 2024-03-31T23:37:32.671 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-03-31T23:37:32.671 INFO:teuthology.orchestra.run.smithi142.stdout: "req_open_latency": { 2024-03-31T23:37:32.671 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 12480, 2024-03-31T23:37:32.671 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 19.138863182, 2024-03-31T23:37:32.671 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.001533562 2024-03-31T23:37:32.671 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-03-31T23:37:32.671 INFO:teuthology.orchestra.run.smithi142.stdout: "req_readdir_latency": { 2024-03-31T23:37:32.671 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 66585, 2024-03-31T23:37:32.671 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 222.736330419, 2024-03-31T23:37:32.671 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.003345142 2024-03-31T23:37:32.672 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-03-31T23:37:32.672 INFO:teuthology.orchestra.run.smithi142.stdout: "req_rename_latency": { 2024-03-31T23:37:32.672 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 3884, 2024-03-31T23:37:32.672 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 45.246445627, 2024-03-31T23:37:32.672 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.011649445 2024-03-31T23:37:32.672 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-03-31T23:37:32.672 INFO:teuthology.orchestra.run.smithi142.stdout: "req_renamesnap_latency": { 2024-03-31T23:37:32.672 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 0, 2024-03-31T23:37:32.672 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.000000000, 2024-03-31T23:37:32.672 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000000000 2024-03-31T23:37:32.672 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-03-31T23:37:32.672 INFO:teuthology.orchestra.run.smithi142.stdout: "req_snapdiff_latency": { 2024-03-31T23:37:32.672 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 0, 2024-03-31T23:37:32.672 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.000000000, 2024-03-31T23:37:32.672 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000000000 2024-03-31T23:37:32.672 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-03-31T23:37:32.673 INFO:teuthology.orchestra.run.smithi142.stdout: "req_rmdir_latency": { 2024-03-31T23:37:32.673 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 30, 2024-03-31T23:37:32.673 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.236500409, 2024-03-31T23:37:32.673 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.007883346 2024-03-31T23:37:32.673 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-03-31T23:37:32.673 INFO:teuthology.orchestra.run.smithi142.stdout: "req_rmsnap_latency": { 2024-03-31T23:37:32.673 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 0, 2024-03-31T23:37:32.673 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.000000000, 2024-03-31T23:37:32.673 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000000000 2024-03-31T23:37:32.673 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-03-31T23:37:32.673 INFO:teuthology.orchestra.run.smithi142.stdout: "req_rmxattr_latency": { 2024-03-31T23:37:32.673 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 0, 2024-03-31T23:37:32.673 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.000000000, 2024-03-31T23:37:32.673 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000000000 2024-03-31T23:37:32.674 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-03-31T23:37:32.674 INFO:teuthology.orchestra.run.smithi142.stdout: "req_setattr_latency": { 2024-03-31T23:37:32.674 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 0, 2024-03-31T23:37:32.674 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.000000000, 2024-03-31T23:37:32.674 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000000000 2024-03-31T23:37:32.674 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-03-31T23:37:32.674 INFO:teuthology.orchestra.run.smithi142.stdout: "req_setdirlayout_latency": { 2024-03-31T23:37:32.674 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 0, 2024-03-31T23:37:32.674 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.000000000, 2024-03-31T23:37:32.674 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000000000 2024-03-31T23:37:32.674 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-03-31T23:37:32.674 INFO:teuthology.orchestra.run.smithi142.stdout: "req_setfilelock_latency": { 2024-03-31T23:37:32.674 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 580, 2024-03-31T23:37:32.674 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.992852747, 2024-03-31T23:37:32.674 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.001711815 2024-03-31T23:37:32.675 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-03-31T23:37:32.675 INFO:teuthology.orchestra.run.smithi142.stdout: "req_setlayout_latency": { 2024-03-31T23:37:32.675 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 0, 2024-03-31T23:37:32.675 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.000000000, 2024-03-31T23:37:32.675 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000000000 2024-03-31T23:37:32.675 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-03-31T23:37:32.675 INFO:teuthology.orchestra.run.smithi142.stdout: "req_setxattr_latency": { 2024-03-31T23:37:32.675 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 0, 2024-03-31T23:37:32.675 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.000000000, 2024-03-31T23:37:32.675 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000000000 2024-03-31T23:37:32.675 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-03-31T23:37:32.675 INFO:teuthology.orchestra.run.smithi142.stdout: "req_symlink_latency": { 2024-03-31T23:37:32.675 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 0, 2024-03-31T23:37:32.675 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.000000000, 2024-03-31T23:37:32.675 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000000000 2024-03-31T23:37:32.676 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-03-31T23:37:32.676 INFO:teuthology.orchestra.run.smithi142.stdout: "req_unlink_latency": { 2024-03-31T23:37:32.676 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 17991, 2024-03-31T23:37:32.676 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 96.167592018, 2024-03-31T23:37:32.676 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.005345316 2024-03-31T23:37:32.676 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-03-31T23:37:32.676 INFO:teuthology.orchestra.run.smithi142.stdout: "cap_revoke_eviction": 0, 2024-03-31T23:37:32.676 INFO:teuthology.orchestra.run.smithi142.stdout: "cap_acquisition_throttle": 0, 2024-03-31T23:37:32.676 INFO:teuthology.orchestra.run.smithi142.stdout: "req_getvxattr_latency": { 2024-03-31T23:37:32.676 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 0, 2024-03-31T23:37:32.676 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.000000000, 2024-03-31T23:37:32.676 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000000000 2024-03-31T23:37:32.676 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-03-31T23:37:32.676 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-03-31T23:37:32.677 INFO:teuthology.orchestra.run.smithi142.stdout: "mds_sessions": { 2024-03-31T23:37:32.677 INFO:teuthology.orchestra.run.smithi142.stdout: "session_count": 2, 2024-03-31T23:37:32.677 INFO:teuthology.orchestra.run.smithi142.stdout: "session_add": 4, 2024-03-31T23:37:32.677 INFO:teuthology.orchestra.run.smithi142.stdout: "session_remove": 2, 2024-03-31T23:37:32.677 INFO:teuthology.orchestra.run.smithi142.stdout: "sessions_open": 2, 2024-03-31T23:37:32.677 INFO:teuthology.orchestra.run.smithi142.stdout: "sessions_stale": 0, 2024-03-31T23:37:32.677 INFO:teuthology.orchestra.run.smithi142.stdout: "total_load": 69387, 2024-03-31T23:37:32.677 INFO:teuthology.orchestra.run.smithi142.stdout: "average_load": 34693, 2024-03-31T23:37:32.677 INFO:teuthology.orchestra.run.smithi142.stdout: "avg_session_uptime": 1547, 2024-03-31T23:37:32.677 INFO:teuthology.orchestra.run.smithi142.stdout: "mdthresh_evicted": 0 2024-03-31T23:37:32.677 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-03-31T23:37:32.677 INFO:teuthology.orchestra.run.smithi142.stdout: "mempool": { 2024-03-31T23:37:32.677 INFO:teuthology.orchestra.run.smithi142.stdout: "bloom_filter_bytes": 0, 2024-03-31T23:37:32.677 INFO:teuthology.orchestra.run.smithi142.stdout: "bloom_filter_items": 0, 2024-03-31T23:37:32.677 INFO:teuthology.orchestra.run.smithi142.stdout: "bluestore_alloc_bytes": 0, 2024-03-31T23:37:32.678 INFO:teuthology.orchestra.run.smithi142.stdout: "bluestore_alloc_items": 0, 2024-03-31T23:37:32.678 INFO:teuthology.orchestra.run.smithi142.stdout: "bluestore_cache_data_bytes": 0, 2024-03-31T23:37:32.678 INFO:teuthology.orchestra.run.smithi142.stdout: "bluestore_cache_data_items": 0, 2024-03-31T23:37:32.678 INFO:teuthology.orchestra.run.smithi142.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-31T23:37:32.678 INFO:teuthology.orchestra.run.smithi142.stdout: "bluestore_cache_onode_items": 0, 2024-03-31T23:37:32.678 INFO:teuthology.orchestra.run.smithi142.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-31T23:37:32.678 INFO:teuthology.orchestra.run.smithi142.stdout: "bluestore_cache_meta_items": 0, 2024-03-31T23:37:32.678 INFO:teuthology.orchestra.run.smithi142.stdout: "bluestore_cache_other_bytes": 0, 2024-03-31T23:37:32.678 INFO:teuthology.orchestra.run.smithi142.stdout: "bluestore_cache_other_items": 0, 2024-03-31T23:37:32.678 INFO:teuthology.orchestra.run.smithi142.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-31T23:37:32.678 INFO:teuthology.orchestra.run.smithi142.stdout: "bluestore_cache_buffer_items": 0, 2024-03-31T23:37:32.678 INFO:teuthology.orchestra.run.smithi142.stdout: "bluestore_extent_bytes": 0, 2024-03-31T23:37:32.678 INFO:teuthology.orchestra.run.smithi142.stdout: "bluestore_extent_items": 0, 2024-03-31T23:37:32.678 INFO:teuthology.orchestra.run.smithi142.stdout: "bluestore_blob_bytes": 0, 2024-03-31T23:37:32.679 INFO:teuthology.orchestra.run.smithi142.stdout: "bluestore_blob_items": 0, 2024-03-31T23:37:32.679 INFO:teuthology.orchestra.run.smithi142.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-31T23:37:32.679 INFO:teuthology.orchestra.run.smithi142.stdout: "bluestore_shared_blob_items": 0, 2024-03-31T23:37:32.679 INFO:teuthology.orchestra.run.smithi142.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-31T23:37:32.679 INFO:teuthology.orchestra.run.smithi142.stdout: "bluestore_inline_bl_items": 0, 2024-03-31T23:37:32.679 INFO:teuthology.orchestra.run.smithi142.stdout: "bluestore_fsck_bytes": 0, 2024-03-31T23:37:32.679 INFO:teuthology.orchestra.run.smithi142.stdout: "bluestore_fsck_items": 0, 2024-03-31T23:37:32.679 INFO:teuthology.orchestra.run.smithi142.stdout: "bluestore_txc_bytes": 0, 2024-03-31T23:37:32.679 INFO:teuthology.orchestra.run.smithi142.stdout: "bluestore_txc_items": 0, 2024-03-31T23:37:32.679 INFO:teuthology.orchestra.run.smithi142.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-31T23:37:32.679 INFO:teuthology.orchestra.run.smithi142.stdout: "bluestore_writing_deferred_items": 0, 2024-03-31T23:37:32.679 INFO:teuthology.orchestra.run.smithi142.stdout: "bluestore_writing_bytes": 0, 2024-03-31T23:37:32.679 INFO:teuthology.orchestra.run.smithi142.stdout: "bluestore_writing_items": 0, 2024-03-31T23:37:32.679 INFO:teuthology.orchestra.run.smithi142.stdout: "bluefs_bytes": 0, 2024-03-31T23:37:32.679 INFO:teuthology.orchestra.run.smithi142.stdout: "bluefs_items": 0, 2024-03-31T23:37:32.679 INFO:teuthology.orchestra.run.smithi142.stdout: "bluefs_file_reader_bytes": 0, 2024-03-31T23:37:32.680 INFO:teuthology.orchestra.run.smithi142.stdout: "bluefs_file_reader_items": 0, 2024-03-31T23:37:32.680 INFO:teuthology.orchestra.run.smithi142.stdout: "bluefs_file_writer_bytes": 0, 2024-03-31T23:37:32.680 INFO:teuthology.orchestra.run.smithi142.stdout: "bluefs_file_writer_items": 0, 2024-03-31T23:37:32.680 INFO:teuthology.orchestra.run.smithi142.stdout: "buffer_anon_bytes": 528038, 2024-03-31T23:37:32.680 INFO:teuthology.orchestra.run.smithi142.stdout: "buffer_anon_items": 159, 2024-03-31T23:37:32.680 INFO:teuthology.orchestra.run.smithi142.stdout: "buffer_meta_bytes": 0, 2024-03-31T23:37:32.680 INFO:teuthology.orchestra.run.smithi142.stdout: "buffer_meta_items": 0, 2024-03-31T23:37:32.680 INFO:teuthology.orchestra.run.smithi142.stdout: "osd_bytes": 0, 2024-03-31T23:37:32.680 INFO:teuthology.orchestra.run.smithi142.stdout: "osd_items": 0, 2024-03-31T23:37:32.680 INFO:teuthology.orchestra.run.smithi142.stdout: "osd_mapbl_bytes": 0, 2024-03-31T23:37:32.680 INFO:teuthology.orchestra.run.smithi142.stdout: "osd_mapbl_items": 0, 2024-03-31T23:37:32.680 INFO:teuthology.orchestra.run.smithi142.stdout: "osd_pglog_bytes": 0, 2024-03-31T23:37:32.680 INFO:teuthology.orchestra.run.smithi142.stdout: "osd_pglog_items": 0, 2024-03-31T23:37:32.680 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap_bytes": 9704, 2024-03-31T23:37:32.680 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap_items": 186, 2024-03-31T23:37:32.680 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap_mapping_bytes": 0, 2024-03-31T23:37:32.681 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap_mapping_items": 0, 2024-03-31T23:37:32.681 INFO:teuthology.orchestra.run.smithi142.stdout: "pgmap_bytes": 0, 2024-03-31T23:37:32.681 INFO:teuthology.orchestra.run.smithi142.stdout: "pgmap_items": 0, 2024-03-31T23:37:32.681 INFO:teuthology.orchestra.run.smithi142.stdout: "mds_co_bytes": 9602928, 2024-03-31T23:37:32.681 INFO:teuthology.orchestra.run.smithi142.stdout: "mds_co_items": 41539, 2024-03-31T23:37:32.681 INFO:teuthology.orchestra.run.smithi142.stdout: "unittest_1_bytes": 0, 2024-03-31T23:37:32.681 INFO:teuthology.orchestra.run.smithi142.stdout: "unittest_1_items": 0, 2024-03-31T23:37:32.681 INFO:teuthology.orchestra.run.smithi142.stdout: "unittest_2_bytes": 0, 2024-03-31T23:37:32.681 INFO:teuthology.orchestra.run.smithi142.stdout: "unittest_2_items": 0 2024-03-31T23:37:32.681 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-03-31T23:37:32.681 INFO:teuthology.orchestra.run.smithi142.stdout: "objecter": { 2024-03-31T23:37:32.681 INFO:teuthology.orchestra.run.smithi142.stdout: "op_active": 0, 2024-03-31T23:37:32.681 INFO:teuthology.orchestra.run.smithi142.stdout: "op_laggy": 0, 2024-03-31T23:37:32.681 INFO:teuthology.orchestra.run.smithi142.stdout: "op_send": 79504, 2024-03-31T23:37:32.681 INFO:teuthology.orchestra.run.smithi142.stdout: "op_send_bytes": 689825132, 2024-03-31T23:37:32.681 INFO:teuthology.orchestra.run.smithi142.stdout: "op_resend": 28, 2024-03-31T23:37:32.681 INFO:teuthology.orchestra.run.smithi142.stdout: "op_reply": 79504, 2024-03-31T23:37:32.682 INFO:teuthology.orchestra.run.smithi142.stdout: "op_latency": { 2024-03-31T23:37:32.682 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 79504, 2024-03-31T23:37:32.682 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 429.789866254, 2024-03-31T23:37:32.682 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.005405889 2024-03-31T23:37:32.682 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-03-31T23:37:32.682 INFO:teuthology.orchestra.run.smithi142.stdout: "op_inflight": 0, 2024-03-31T23:37:32.682 INFO:teuthology.orchestra.run.smithi142.stdout: "oplen_avg": { 2024-03-31T23:37:32.682 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 79504, 2024-03-31T23:37:32.682 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 82466 2024-03-31T23:37:32.682 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-03-31T23:37:32.682 INFO:teuthology.orchestra.run.smithi142.stdout: "op": 79504, 2024-03-31T23:37:32.682 INFO:teuthology.orchestra.run.smithi142.stdout: "op_r": 2053, 2024-03-31T23:37:32.682 INFO:teuthology.orchestra.run.smithi142.stdout: "op_w": 77451, 2024-03-31T23:37:32.682 INFO:teuthology.orchestra.run.smithi142.stdout: "op_rmw": 0, 2024-03-31T23:37:32.683 INFO:teuthology.orchestra.run.smithi142.stdout: "op_pg": 0, 2024-03-31T23:37:32.683 INFO:teuthology.orchestra.run.smithi142.stdout: "osdop_stat": 40, 2024-03-31T23:37:32.683 INFO:teuthology.orchestra.run.smithi142.stdout: "osdop_create": 327, 2024-03-31T23:37:32.683 INFO:teuthology.orchestra.run.smithi142.stdout: "osdop_read": 1995, 2024-03-31T23:37:32.683 INFO:teuthology.orchestra.run.smithi142.stdout: "osdop_write": 38313, 2024-03-31T23:37:32.683 INFO:teuthology.orchestra.run.smithi142.stdout: "osdop_writefull": 1143, 2024-03-31T23:37:32.683 INFO:teuthology.orchestra.run.smithi142.stdout: "osdop_writesame": 0, 2024-03-31T23:37:32.683 INFO:teuthology.orchestra.run.smithi142.stdout: "osdop_append": 0, 2024-03-31T23:37:32.683 INFO:teuthology.orchestra.run.smithi142.stdout: "osdop_zero": 0, 2024-03-31T23:37:32.683 INFO:teuthology.orchestra.run.smithi142.stdout: "osdop_truncate": 11, 2024-03-31T23:37:32.683 INFO:teuthology.orchestra.run.smithi142.stdout: "osdop_delete": 36204, 2024-03-31T23:37:32.683 INFO:teuthology.orchestra.run.smithi142.stdout: "osdop_mapext": 0, 2024-03-31T23:37:32.683 INFO:teuthology.orchestra.run.smithi142.stdout: "osdop_sparse_read": 0, 2024-03-31T23:37:32.683 INFO:teuthology.orchestra.run.smithi142.stdout: "osdop_clonerange": 0, 2024-03-31T23:37:32.683 INFO:teuthology.orchestra.run.smithi142.stdout: "osdop_getxattr": 30, 2024-03-31T23:37:32.684 INFO:teuthology.orchestra.run.smithi142.stdout: "osdop_setxattr": 506, 2024-03-31T23:37:32.684 INFO:teuthology.orchestra.run.smithi142.stdout: "osdop_cmpxattr": 0, 2024-03-31T23:37:32.684 INFO:teuthology.orchestra.run.smithi142.stdout: "osdop_rmxattr": 0, 2024-03-31T23:37:32.684 INFO:teuthology.orchestra.run.smithi142.stdout: "osdop_resetxattrs": 0, 2024-03-31T23:37:32.684 INFO:teuthology.orchestra.run.smithi142.stdout: "osdop_call": 0, 2024-03-31T23:37:32.684 INFO:teuthology.orchestra.run.smithi142.stdout: "osdop_watch": 0, 2024-03-31T23:37:32.684 INFO:teuthology.orchestra.run.smithi142.stdout: "osdop_notify": 0, 2024-03-31T23:37:32.684 INFO:teuthology.orchestra.run.smithi142.stdout: "osdop_src_cmpxattr": 0, 2024-03-31T23:37:32.684 INFO:teuthology.orchestra.run.smithi142.stdout: "osdop_pgls": 0, 2024-03-31T23:37:32.684 INFO:teuthology.orchestra.run.smithi142.stdout: "osdop_pgls_filter": 0, 2024-03-31T23:37:32.684 INFO:teuthology.orchestra.run.smithi142.stdout: "osdop_other": 0, 2024-03-31T23:37:32.684 INFO:teuthology.orchestra.run.smithi142.stdout: "linger_active": 0, 2024-03-31T23:37:32.684 INFO:teuthology.orchestra.run.smithi142.stdout: "linger_send": 0, 2024-03-31T23:37:32.684 INFO:teuthology.orchestra.run.smithi142.stdout: "linger_resend": 0, 2024-03-31T23:37:32.684 INFO:teuthology.orchestra.run.smithi142.stdout: "linger_ping": 0, 2024-03-31T23:37:32.684 INFO:teuthology.orchestra.run.smithi142.stdout: "poolop_active": 0, 2024-03-31T23:37:32.685 INFO:teuthology.orchestra.run.smithi142.stdout: "poolop_send": 0, 2024-03-31T23:37:32.685 INFO:teuthology.orchestra.run.smithi142.stdout: "poolop_resend": 0, 2024-03-31T23:37:32.685 INFO:teuthology.orchestra.run.smithi142.stdout: "poolstat_active": 0, 2024-03-31T23:37:32.685 INFO:teuthology.orchestra.run.smithi142.stdout: "poolstat_send": 0, 2024-03-31T23:37:32.685 INFO:teuthology.orchestra.run.smithi142.stdout: "poolstat_resend": 0, 2024-03-31T23:37:32.685 INFO:teuthology.orchestra.run.smithi142.stdout: "statfs_active": 0, 2024-03-31T23:37:32.685 INFO:teuthology.orchestra.run.smithi142.stdout: "statfs_send": 0, 2024-03-31T23:37:32.685 INFO:teuthology.orchestra.run.smithi142.stdout: "statfs_resend": 0, 2024-03-31T23:37:32.685 INFO:teuthology.orchestra.run.smithi142.stdout: "command_active": 0, 2024-03-31T23:37:32.685 INFO:teuthology.orchestra.run.smithi142.stdout: "command_send": 0, 2024-03-31T23:37:32.685 INFO:teuthology.orchestra.run.smithi142.stdout: "command_resend": 0, 2024-03-31T23:37:32.685 INFO:teuthology.orchestra.run.smithi142.stdout: "map_epoch": 139, 2024-03-31T23:37:32.685 INFO:teuthology.orchestra.run.smithi142.stdout: "map_full": 0, 2024-03-31T23:37:32.685 INFO:teuthology.orchestra.run.smithi142.stdout: "map_inc": 48, 2024-03-31T23:37:32.685 INFO:teuthology.orchestra.run.smithi142.stdout: "osd_sessions": 12, 2024-03-31T23:37:32.685 INFO:teuthology.orchestra.run.smithi142.stdout: "osd_session_open": 12, 2024-03-31T23:37:32.686 INFO:teuthology.orchestra.run.smithi142.stdout: "osd_session_close": 0, 2024-03-31T23:37:32.686 INFO:teuthology.orchestra.run.smithi142.stdout: "osd_laggy": 0, 2024-03-31T23:37:32.686 INFO:teuthology.orchestra.run.smithi142.stdout: "omap_wr": 2636, 2024-03-31T23:37:32.686 INFO:teuthology.orchestra.run.smithi142.stdout: "omap_rd": 44, 2024-03-31T23:37:32.686 INFO:teuthology.orchestra.run.smithi142.stdout: "omap_del": 1217 2024-03-31T23:37:32.686 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-03-31T23:37:32.686 INFO:teuthology.orchestra.run.smithi142.stdout: "oft": { 2024-03-31T23:37:32.686 INFO:teuthology.orchestra.run.smithi142.stdout: "omap_total_objs": 1, 2024-03-31T23:37:32.686 INFO:teuthology.orchestra.run.smithi142.stdout: "omap_total_kv_pairs": 0, 2024-03-31T23:37:32.686 INFO:teuthology.orchestra.run.smithi142.stdout: "omap_total_updates": 0, 2024-03-31T23:37:32.686 INFO:teuthology.orchestra.run.smithi142.stdout: "omap_total_removes": 0 2024-03-31T23:37:32.686 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-03-31T23:37:32.686 INFO:teuthology.orchestra.run.smithi142.stdout: "osd-slow-ops": { 2024-03-31T23:37:32.686 INFO:teuthology.orchestra.run.smithi142.stdout: "slow_ops_count": 1 2024-03-31T23:37:32.687 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-03-31T23:37:32.687 INFO:teuthology.orchestra.run.smithi142.stdout: "purge_queue": { 2024-03-31T23:37:32.687 INFO:teuthology.orchestra.run.smithi142.stdout: "pq_executing_ops": 0, 2024-03-31T23:37:32.687 INFO:teuthology.orchestra.run.smithi142.stdout: "pq_executing_ops_high_water": 16, 2024-03-31T23:37:32.687 INFO:teuthology.orchestra.run.smithi142.stdout: "pq_executing": 0, 2024-03-31T23:37:32.687 INFO:teuthology.orchestra.run.smithi142.stdout: "pq_executing_high_water": 8, 2024-03-31T23:37:32.687 INFO:teuthology.orchestra.run.smithi142.stdout: "pq_executed_ops": 35982, 2024-03-31T23:37:32.687 INFO:teuthology.orchestra.run.smithi142.stdout: "pq_executed": 17991, 2024-03-31T23:37:32.687 INFO:teuthology.orchestra.run.smithi142.stdout: "pq_item_in_journal": 0 2024-03-31T23:37:32.687 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-03-31T23:37:32.687 INFO:teuthology.orchestra.run.smithi142.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-31T23:37:32.687 INFO:teuthology.orchestra.run.smithi142.stdout: "val": 0, 2024-03-31T23:37:32.687 INFO:teuthology.orchestra.run.smithi142.stdout: "max": 104857600, 2024-03-31T23:37:32.687 INFO:teuthology.orchestra.run.smithi142.stdout: "get_started": 0, 2024-03-31T23:37:32.687 INFO:teuthology.orchestra.run.smithi142.stdout: "get": 1336479, 2024-03-31T23:37:32.687 INFO:teuthology.orchestra.run.smithi142.stdout: "get_sum": 280484540, 2024-03-31T23:37:32.688 INFO:teuthology.orchestra.run.smithi142.stdout: "get_or_fail_fail": 0, 2024-03-31T23:37:32.688 INFO:teuthology.orchestra.run.smithi142.stdout: "get_or_fail_success": 1336479, 2024-03-31T23:37:32.688 INFO:teuthology.orchestra.run.smithi142.stdout: "take": 0, 2024-03-31T23:37:32.688 INFO:teuthology.orchestra.run.smithi142.stdout: "take_sum": 0, 2024-03-31T23:37:32.688 INFO:teuthology.orchestra.run.smithi142.stdout: "put": 1336479, 2024-03-31T23:37:32.688 INFO:teuthology.orchestra.run.smithi142.stdout: "put_sum": 280484540, 2024-03-31T23:37:32.688 INFO:teuthology.orchestra.run.smithi142.stdout: "wait": { 2024-03-31T23:37:32.688 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 0, 2024-03-31T23:37:32.688 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.000000000, 2024-03-31T23:37:32.688 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000000000 2024-03-31T23:37:32.688 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-03-31T23:37:32.688 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-03-31T23:37:32.688 INFO:teuthology.orchestra.run.smithi142.stdout: "throttle-objecter_bytes": { 2024-03-31T23:37:32.688 INFO:teuthology.orchestra.run.smithi142.stdout: "val": 0, 2024-03-31T23:37:32.688 INFO:teuthology.orchestra.run.smithi142.stdout: "max": 104857600, 2024-03-31T23:37:32.688 INFO:teuthology.orchestra.run.smithi142.stdout: "get_started": 0, 2024-03-31T23:37:32.688 INFO:teuthology.orchestra.run.smithi142.stdout: "get": 0, 2024-03-31T23:37:32.689 INFO:teuthology.orchestra.run.smithi142.stdout: "get_sum": 0, 2024-03-31T23:37:32.689 INFO:teuthology.orchestra.run.smithi142.stdout: "get_or_fail_fail": 0, 2024-03-31T23:37:32.689 INFO:teuthology.orchestra.run.smithi142.stdout: "get_or_fail_success": 0, 2024-03-31T23:37:32.689 INFO:teuthology.orchestra.run.smithi142.stdout: "take": 79504, 2024-03-31T23:37:32.689 INFO:teuthology.orchestra.run.smithi142.stdout: "take_sum": 691803261, 2024-03-31T23:37:32.689 INFO:teuthology.orchestra.run.smithi142.stdout: "put": 79504, 2024-03-31T23:37:32.689 INFO:teuthology.orchestra.run.smithi142.stdout: "put_sum": 691803261, 2024-03-31T23:37:32.689 INFO:teuthology.orchestra.run.smithi142.stdout: "wait": { 2024-03-31T23:37:32.689 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 0, 2024-03-31T23:37:32.689 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.000000000, 2024-03-31T23:37:32.689 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000000000 2024-03-31T23:37:32.689 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-03-31T23:37:32.689 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-03-31T23:37:32.689 INFO:teuthology.orchestra.run.smithi142.stdout: "throttle-objecter_ops": { 2024-03-31T23:37:32.689 INFO:teuthology.orchestra.run.smithi142.stdout: "val": 0, 2024-03-31T23:37:32.689 INFO:teuthology.orchestra.run.smithi142.stdout: "max": 1024, 2024-03-31T23:37:32.690 INFO:teuthology.orchestra.run.smithi142.stdout: "get_started": 0, 2024-03-31T23:37:32.690 INFO:teuthology.orchestra.run.smithi142.stdout: "get": 0, 2024-03-31T23:37:32.690 INFO:teuthology.orchestra.run.smithi142.stdout: "get_sum": 0, 2024-03-31T23:37:32.690 INFO:teuthology.orchestra.run.smithi142.stdout: "get_or_fail_fail": 0, 2024-03-31T23:37:32.690 INFO:teuthology.orchestra.run.smithi142.stdout: "get_or_fail_success": 0, 2024-03-31T23:37:32.690 INFO:teuthology.orchestra.run.smithi142.stdout: "take": 79504, 2024-03-31T23:37:32.690 INFO:teuthology.orchestra.run.smithi142.stdout: "take_sum": 79504, 2024-03-31T23:37:32.690 INFO:teuthology.orchestra.run.smithi142.stdout: "put": 79504, 2024-03-31T23:37:32.690 INFO:teuthology.orchestra.run.smithi142.stdout: "put_sum": 79504, 2024-03-31T23:37:32.690 INFO:teuthology.orchestra.run.smithi142.stdout: "wait": { 2024-03-31T23:37:32.690 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 0, 2024-03-31T23:37:32.690 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.000000000, 2024-03-31T23:37:32.690 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000000000 2024-03-31T23:37:32.690 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-03-31T23:37:32.690 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-03-31T23:37:32.691 INFO:teuthology.orchestra.run.smithi142.stdout: "throttle-write_buf_throttle": { 2024-03-31T23:37:32.691 INFO:teuthology.orchestra.run.smithi142.stdout: "val": 0, 2024-03-31T23:37:32.691 INFO:teuthology.orchestra.run.smithi142.stdout: "max": 3758096384, 2024-03-31T23:37:32.691 INFO:teuthology.orchestra.run.smithi142.stdout: "get_started": 0, 2024-03-31T23:37:32.691 INFO:teuthology.orchestra.run.smithi142.stdout: "get": 17991, 2024-03-31T23:37:32.691 INFO:teuthology.orchestra.run.smithi142.stdout: "get_sum": 1978795, 2024-03-31T23:37:32.691 INFO:teuthology.orchestra.run.smithi142.stdout: "get_or_fail_fail": 0, 2024-03-31T23:37:32.691 INFO:teuthology.orchestra.run.smithi142.stdout: "get_or_fail_success": 17991, 2024-03-31T23:37:32.691 INFO:teuthology.orchestra.run.smithi142.stdout: "take": 0, 2024-03-31T23:37:32.691 INFO:teuthology.orchestra.run.smithi142.stdout: "take_sum": 0, 2024-03-31T23:37:32.691 INFO:teuthology.orchestra.run.smithi142.stdout: "put": 1995, 2024-03-31T23:37:32.691 INFO:teuthology.orchestra.run.smithi142.stdout: "put_sum": 1978795, 2024-03-31T23:37:32.691 INFO:teuthology.orchestra.run.smithi142.stdout: "wait": { 2024-03-31T23:37:32.691 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 0, 2024-03-31T23:37:32.691 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.000000000, 2024-03-31T23:37:32.692 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000000000 2024-03-31T23:37:32.692 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-03-31T23:37:32.692 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-03-31T23:37:32.692 INFO:teuthology.orchestra.run.smithi142.stdout: "throttle-write_buf_throttle-0x559403186500": { 2024-03-31T23:37:32.692 INFO:teuthology.orchestra.run.smithi142.stdout: "val": 0, 2024-03-31T23:37:32.692 INFO:teuthology.orchestra.run.smithi142.stdout: "max": 3758096384, 2024-03-31T23:37:32.692 INFO:teuthology.orchestra.run.smithi142.stdout: "get_started": 0, 2024-03-31T23:37:32.692 INFO:teuthology.orchestra.run.smithi142.stdout: "get": 149835, 2024-03-31T23:37:32.692 INFO:teuthology.orchestra.run.smithi142.stdout: "get_sum": 684261467, 2024-03-31T23:37:32.692 INFO:teuthology.orchestra.run.smithi142.stdout: "get_or_fail_fail": 0, 2024-03-31T23:37:32.692 INFO:teuthology.orchestra.run.smithi142.stdout: "get_or_fail_success": 149835, 2024-03-31T23:37:32.692 INFO:teuthology.orchestra.run.smithi142.stdout: "take": 0, 2024-03-31T23:37:32.692 INFO:teuthology.orchestra.run.smithi142.stdout: "take_sum": 0, 2024-03-31T23:37:32.692 INFO:teuthology.orchestra.run.smithi142.stdout: "put": 36318, 2024-03-31T23:37:32.692 INFO:teuthology.orchestra.run.smithi142.stdout: "put_sum": 684261467, 2024-03-31T23:37:32.692 INFO:teuthology.orchestra.run.smithi142.stdout: "wait": { 2024-03-31T23:37:32.693 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 0, 2024-03-31T23:37:32.693 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.000000000, 2024-03-31T23:37:32.693 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000000000 2024-03-31T23:37:32.693 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-03-31T23:37:32.693 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-03-31T23:37:32.693 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-03-31T23:37:32.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:37:32 smithi142 ceph-mon[35271]: pgmap v820: 129 pgs: 129 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 2.3 KiB/s wr, 5 op/s 2024-03-31T23:37:33.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:37:32 smithi162 ceph-mon[36237]: pgmap v820: 129 pgs: 129 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 2.3 KiB/s wr, 5 op/s 2024-03-31T23:37:33.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:37:32 smithi046 ceph-mon[32498]: pgmap v820: 129 pgs: 129 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 2.3 KiB/s wr, 5 op/s 2024-03-31T23:37:33.441 INFO:tasks.check_counter:Daemon mds.h mds.exported=53 2024-03-31T23:37:33.441 INFO:tasks.check_counter:Daemon mds.h mds.imported=33 2024-03-31T23:37:33.441 INFO:tasks.check_counter:Daemon mds.h mds_cache.dir_update=1222 2024-03-31T23:37:33.441 INFO:tasks.check_counter:Daemon mds.h mds_cache.dir_update_receipt=6372 2024-03-31T23:37:33.442 INFO:tasks.check_counter:Daemon mds.h mds.root_rsnaps=6 2024-03-31T23:37:33.442 INFO:tasks.check_counter:Daemon mds.h mds_server.req_mksnap_latency.avgcount=0 2024-03-31T23:37:33.442 INFO:tasks.check_counter:Daemon mds.h mds_server.req_rmsnap_latency.avgcount=0 2024-03-31T23:37:33.442 DEBUG:tasks.check_counter:Getting stats from k 2024-03-31T23:37:33.442 DEBUG:teuthology.orchestra.run.smithi142:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:384f31f3d3aee7f8ba3f60846b8844c049165583 shell --fsid 6a9d8f08-efb2-11ee-b647-cb9ed24678a4 -- ceph daemon mds.k perf dump 2024-03-31T23:37:33.732 INFO:teuthology.orchestra.run.smithi142.stderr:Inferring config /var/lib/ceph/6a9d8f08-efb2-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-31T23:37:34.861 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:37:34 smithi142 ceph-mon[35271]: pgmap v821: 129 pgs: 129 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 2.3 KiB/s wr, 5 op/s 2024-03-31T23:37:35.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:37:34 smithi162 ceph-mon[36237]: pgmap v821: 129 pgs: 129 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 2.3 KiB/s wr, 5 op/s 2024-03-31T23:37:35.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:37:34 smithi046 ceph-mon[32498]: pgmap v821: 129 pgs: 129 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 2.3 KiB/s wr, 5 op/s 2024-03-31T23:37:35.479 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-03-31T23:37:35.479 INFO:teuthology.orchestra.run.smithi142.stdout: "AsyncMessenger::Worker-0": { 2024-03-31T23:37:35.479 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_recv_messages": 0, 2024-03-31T23:37:35.479 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_send_messages": 2, 2024-03-31T23:37:35.479 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_recv_bytes": 0, 2024-03-31T23:37:35.479 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_send_bytes": 0, 2024-03-31T23:37:35.479 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_created_connections": 4, 2024-03-31T23:37:35.479 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_active_connections": 0, 2024-03-31T23:37:35.479 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_running_total_time": 0.001157389, 2024-03-31T23:37:35.479 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_running_send_time": 0.000000000, 2024-03-31T23:37:35.479 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_running_recv_time": 0.000374493, 2024-03-31T23:37:35.479 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-31T23:37:35.479 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_send_messages_queue_lat": { 2024-03-31T23:37:35.480 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 0, 2024-03-31T23:37:35.480 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.000000000, 2024-03-31T23:37:35.480 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000000000 2024-03-31T23:37:35.480 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-03-31T23:37:35.480 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_handle_ack_lat": { 2024-03-31T23:37:35.480 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 0, 2024-03-31T23:37:35.480 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.000000000, 2024-03-31T23:37:35.480 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000000000 2024-03-31T23:37:35.480 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-03-31T23:37:35.480 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-31T23:37:35.480 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_send_encrypted_bytes": 0 2024-03-31T23:37:35.480 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-03-31T23:37:35.480 INFO:teuthology.orchestra.run.smithi142.stdout: "AsyncMessenger::Worker-1": { 2024-03-31T23:37:35.480 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_recv_messages": 7, 2024-03-31T23:37:35.480 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_send_messages": 355, 2024-03-31T23:37:35.481 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_recv_bytes": 3472, 2024-03-31T23:37:35.481 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_send_bytes": 295904, 2024-03-31T23:37:35.481 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_created_connections": 6, 2024-03-31T23:37:35.481 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_active_connections": 3, 2024-03-31T23:37:35.481 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_running_total_time": 0.015584400, 2024-03-31T23:37:35.481 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_running_send_time": 0.010479741, 2024-03-31T23:37:35.481 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_running_recv_time": 0.002431334, 2024-03-31T23:37:35.481 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-31T23:37:35.481 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_send_messages_queue_lat": { 2024-03-31T23:37:35.481 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 353, 2024-03-31T23:37:35.481 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.020239375, 2024-03-31T23:37:35.481 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000057335 2024-03-31T23:37:35.481 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-03-31T23:37:35.481 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_handle_ack_lat": { 2024-03-31T23:37:35.481 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 0, 2024-03-31T23:37:35.481 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.000000000, 2024-03-31T23:37:35.481 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000000000 2024-03-31T23:37:35.482 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-03-31T23:37:35.482 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_recv_encrypted_bytes": 3472, 2024-03-31T23:37:35.482 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_send_encrypted_bytes": 295904 2024-03-31T23:37:35.482 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-03-31T23:37:35.482 INFO:teuthology.orchestra.run.smithi142.stdout: "AsyncMessenger::Worker-2": { 2024-03-31T23:37:35.482 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_recv_messages": 470, 2024-03-31T23:37:35.482 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_send_messages": 460, 2024-03-31T23:37:35.482 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_recv_bytes": 1508928, 2024-03-31T23:37:35.482 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_send_bytes": 285072, 2024-03-31T23:37:35.482 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_created_connections": 2, 2024-03-31T23:37:35.482 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_active_connections": 1, 2024-03-31T23:37:35.482 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_running_total_time": 0.061015536, 2024-03-31T23:37:35.482 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_running_send_time": 0.018849533, 2024-03-31T23:37:35.483 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_running_recv_time": 0.039770875, 2024-03-31T23:37:35.483 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_running_fast_dispatch_time": 0.009225242, 2024-03-31T23:37:35.483 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_send_messages_queue_lat": { 2024-03-31T23:37:35.483 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 459, 2024-03-31T23:37:35.483 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.012587996, 2024-03-31T23:37:35.483 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000027424 2024-03-31T23:37:35.483 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-03-31T23:37:35.483 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_handle_ack_lat": { 2024-03-31T23:37:35.483 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 0, 2024-03-31T23:37:35.483 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.000000000, 2024-03-31T23:37:35.483 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000000000 2024-03-31T23:37:35.483 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-03-31T23:37:35.483 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_recv_encrypted_bytes": 1508928, 2024-03-31T23:37:35.483 INFO:teuthology.orchestra.run.smithi142.stdout: "msgr_send_encrypted_bytes": 285072 2024-03-31T23:37:35.483 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-03-31T23:37:35.484 INFO:teuthology.orchestra.run.smithi142.stdout: "cct": { 2024-03-31T23:37:35.484 INFO:teuthology.orchestra.run.smithi142.stdout: "total_workers": 0, 2024-03-31T23:37:35.484 INFO:teuthology.orchestra.run.smithi142.stdout: "unhealthy_workers": 0 2024-03-31T23:37:35.484 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-03-31T23:37:35.484 INFO:teuthology.orchestra.run.smithi142.stdout: "mempool": { 2024-03-31T23:37:35.484 INFO:teuthology.orchestra.run.smithi142.stdout: "bloom_filter_bytes": 0, 2024-03-31T23:37:35.484 INFO:teuthology.orchestra.run.smithi142.stdout: "bloom_filter_items": 0, 2024-03-31T23:37:35.484 INFO:teuthology.orchestra.run.smithi142.stdout: "bluestore_alloc_bytes": 0, 2024-03-31T23:37:35.484 INFO:teuthology.orchestra.run.smithi142.stdout: "bluestore_alloc_items": 0, 2024-03-31T23:37:35.484 INFO:teuthology.orchestra.run.smithi142.stdout: "bluestore_cache_data_bytes": 0, 2024-03-31T23:37:35.484 INFO:teuthology.orchestra.run.smithi142.stdout: "bluestore_cache_data_items": 0, 2024-03-31T23:37:35.484 INFO:teuthology.orchestra.run.smithi142.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-31T23:37:35.484 INFO:teuthology.orchestra.run.smithi142.stdout: "bluestore_cache_onode_items": 0, 2024-03-31T23:37:35.484 INFO:teuthology.orchestra.run.smithi142.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-31T23:37:35.485 INFO:teuthology.orchestra.run.smithi142.stdout: "bluestore_cache_meta_items": 0, 2024-03-31T23:37:35.485 INFO:teuthology.orchestra.run.smithi142.stdout: "bluestore_cache_other_bytes": 0, 2024-03-31T23:37:35.485 INFO:teuthology.orchestra.run.smithi142.stdout: "bluestore_cache_other_items": 0, 2024-03-31T23:37:35.485 INFO:teuthology.orchestra.run.smithi142.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-31T23:37:35.485 INFO:teuthology.orchestra.run.smithi142.stdout: "bluestore_cache_buffer_items": 0, 2024-03-31T23:37:35.485 INFO:teuthology.orchestra.run.smithi142.stdout: "bluestore_extent_bytes": 0, 2024-03-31T23:37:35.485 INFO:teuthology.orchestra.run.smithi142.stdout: "bluestore_extent_items": 0, 2024-03-31T23:37:35.485 INFO:teuthology.orchestra.run.smithi142.stdout: "bluestore_blob_bytes": 0, 2024-03-31T23:37:35.485 INFO:teuthology.orchestra.run.smithi142.stdout: "bluestore_blob_items": 0, 2024-03-31T23:37:35.485 INFO:teuthology.orchestra.run.smithi142.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-31T23:37:35.485 INFO:teuthology.orchestra.run.smithi142.stdout: "bluestore_shared_blob_items": 0, 2024-03-31T23:37:35.485 INFO:teuthology.orchestra.run.smithi142.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-31T23:37:35.485 INFO:teuthology.orchestra.run.smithi142.stdout: "bluestore_inline_bl_items": 0, 2024-03-31T23:37:35.485 INFO:teuthology.orchestra.run.smithi142.stdout: "bluestore_fsck_bytes": 0, 2024-03-31T23:37:35.485 INFO:teuthology.orchestra.run.smithi142.stdout: "bluestore_fsck_items": 0, 2024-03-31T23:37:35.485 INFO:teuthology.orchestra.run.smithi142.stdout: "bluestore_txc_bytes": 0, 2024-03-31T23:37:35.486 INFO:teuthology.orchestra.run.smithi142.stdout: "bluestore_txc_items": 0, 2024-03-31T23:37:35.486 INFO:teuthology.orchestra.run.smithi142.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-31T23:37:35.486 INFO:teuthology.orchestra.run.smithi142.stdout: "bluestore_writing_deferred_items": 0, 2024-03-31T23:37:35.486 INFO:teuthology.orchestra.run.smithi142.stdout: "bluestore_writing_bytes": 0, 2024-03-31T23:37:35.486 INFO:teuthology.orchestra.run.smithi142.stdout: "bluestore_writing_items": 0, 2024-03-31T23:37:35.486 INFO:teuthology.orchestra.run.smithi142.stdout: "bluefs_bytes": 0, 2024-03-31T23:37:35.486 INFO:teuthology.orchestra.run.smithi142.stdout: "bluefs_items": 0, 2024-03-31T23:37:35.486 INFO:teuthology.orchestra.run.smithi142.stdout: "bluefs_file_reader_bytes": 0, 2024-03-31T23:37:35.486 INFO:teuthology.orchestra.run.smithi142.stdout: "bluefs_file_reader_items": 0, 2024-03-31T23:37:35.486 INFO:teuthology.orchestra.run.smithi142.stdout: "bluefs_file_writer_bytes": 0, 2024-03-31T23:37:35.486 INFO:teuthology.orchestra.run.smithi142.stdout: "bluefs_file_writer_items": 0, 2024-03-31T23:37:35.486 INFO:teuthology.orchestra.run.smithi142.stdout: "buffer_anon_bytes": 148771, 2024-03-31T23:37:35.486 INFO:teuthology.orchestra.run.smithi142.stdout: "buffer_anon_items": 32, 2024-03-31T23:37:35.487 INFO:teuthology.orchestra.run.smithi142.stdout: "buffer_meta_bytes": 0, 2024-03-31T23:37:35.487 INFO:teuthology.orchestra.run.smithi142.stdout: "buffer_meta_items": 0, 2024-03-31T23:37:35.487 INFO:teuthology.orchestra.run.smithi142.stdout: "osd_bytes": 0, 2024-03-31T23:37:35.487 INFO:teuthology.orchestra.run.smithi142.stdout: "osd_items": 0, 2024-03-31T23:37:35.487 INFO:teuthology.orchestra.run.smithi142.stdout: "osd_mapbl_bytes": 0, 2024-03-31T23:37:35.487 INFO:teuthology.orchestra.run.smithi142.stdout: "osd_mapbl_items": 0, 2024-03-31T23:37:35.487 INFO:teuthology.orchestra.run.smithi142.stdout: "osd_pglog_bytes": 0, 2024-03-31T23:37:35.487 INFO:teuthology.orchestra.run.smithi142.stdout: "osd_pglog_items": 0, 2024-03-31T23:37:35.487 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap_bytes": 0, 2024-03-31T23:37:35.487 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap_items": 0, 2024-03-31T23:37:35.487 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap_mapping_bytes": 0, 2024-03-31T23:37:35.487 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap_mapping_items": 0, 2024-03-31T23:37:35.487 INFO:teuthology.orchestra.run.smithi142.stdout: "pgmap_bytes": 0, 2024-03-31T23:37:35.487 INFO:teuthology.orchestra.run.smithi142.stdout: "pgmap_items": 0, 2024-03-31T23:37:35.487 INFO:teuthology.orchestra.run.smithi142.stdout: "mds_co_bytes": 928, 2024-03-31T23:37:35.488 INFO:teuthology.orchestra.run.smithi142.stdout: "mds_co_items": 2, 2024-03-31T23:37:35.488 INFO:teuthology.orchestra.run.smithi142.stdout: "unittest_1_bytes": 0, 2024-03-31T23:37:35.488 INFO:teuthology.orchestra.run.smithi142.stdout: "unittest_1_items": 0, 2024-03-31T23:37:35.488 INFO:teuthology.orchestra.run.smithi142.stdout: "unittest_2_bytes": 0, 2024-03-31T23:37:35.488 INFO:teuthology.orchestra.run.smithi142.stdout: "unittest_2_items": 0 2024-03-31T23:37:35.488 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-03-31T23:37:35.488 INFO:teuthology.orchestra.run.smithi142.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-31T23:37:35.488 INFO:teuthology.orchestra.run.smithi142.stdout: "val": 0, 2024-03-31T23:37:35.488 INFO:teuthology.orchestra.run.smithi142.stdout: "max": 104857600, 2024-03-31T23:37:35.488 INFO:teuthology.orchestra.run.smithi142.stdout: "get_started": 0, 2024-03-31T23:37:35.488 INFO:teuthology.orchestra.run.smithi142.stdout: "get": 473, 2024-03-31T23:37:35.488 INFO:teuthology.orchestra.run.smithi142.stdout: "get_sum": 1442525, 2024-03-31T23:37:35.488 INFO:teuthology.orchestra.run.smithi142.stdout: "get_or_fail_fail": 0, 2024-03-31T23:37:35.488 INFO:teuthology.orchestra.run.smithi142.stdout: "get_or_fail_success": 473, 2024-03-31T23:37:35.488 INFO:teuthology.orchestra.run.smithi142.stdout: "take": 0, 2024-03-31T23:37:35.488 INFO:teuthology.orchestra.run.smithi142.stdout: "take_sum": 0, 2024-03-31T23:37:35.489 INFO:teuthology.orchestra.run.smithi142.stdout: "put": 473, 2024-03-31T23:37:35.489 INFO:teuthology.orchestra.run.smithi142.stdout: "put_sum": 1442525, 2024-03-31T23:37:35.489 INFO:teuthology.orchestra.run.smithi142.stdout: "wait": { 2024-03-31T23:37:35.489 INFO:teuthology.orchestra.run.smithi142.stdout: "avgcount": 0, 2024-03-31T23:37:35.489 INFO:teuthology.orchestra.run.smithi142.stdout: "sum": 0.000000000, 2024-03-31T23:37:35.489 INFO:teuthology.orchestra.run.smithi142.stdout: "avgtime": 0.000000000 2024-03-31T23:37:35.489 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-03-31T23:37:35.489 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-03-31T23:37:35.489 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-03-31T23:37:36.208 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.k 2024-03-31T23:37:36.208 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.k 2024-03-31T23:37:36.208 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update' not found on daemon mds.k 2024-03-31T23:37:36.208 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update_receipt' not found on daemon mds.k 2024-03-31T23:37:36.208 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.k 2024-03-31T23:37:36.208 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.k 2024-03-31T23:37:36.208 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.k 2024-03-31T23:37:36.208 DEBUG:tasks.check_counter:Getting stats from c 2024-03-31T23:37:36.209 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:384f31f3d3aee7f8ba3f60846b8844c049165583 shell --fsid 6a9d8f08-efb2-11ee-b647-cb9ed24678a4 -- ceph daemon mds.c perf dump 2024-03-31T23:37:36.523 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/6a9d8f08-efb2-11ee-b647-cb9ed24678a4/mon.c/config 2024-03-31T23:37:36.957 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:37:36 smithi162 ceph-mon[36237]: pgmap v822: 129 pgs: 129 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 2.3 KiB/s wr, 7 op/s 2024-03-31T23:37:36.957 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:37:36 smithi162 ceph-mon[36237]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-31T23:37:37.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:37:36 smithi046 ceph-mon[32498]: pgmap v822: 129 pgs: 129 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 2.3 KiB/s wr, 7 op/s 2024-03-31T23:37:37.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:37:36 smithi046 ceph-mon[32498]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-31T23:37:37.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:37:36 smithi142 ceph-mon[35271]: pgmap v822: 129 pgs: 129 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 2.3 KiB/s wr, 7 op/s 2024-03-31T23:37:37.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:37:36 smithi142 ceph-mon[35271]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-31T23:37:38.577 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-03-31T23:37:38.577 INFO:teuthology.orchestra.run.smithi162.stdout: "AsyncMessenger::Worker-0": { 2024-03-31T23:37:38.577 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_recv_messages": 413036, 2024-03-31T23:37:38.577 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_send_messages": 374353, 2024-03-31T23:37:38.577 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_recv_bytes": 120566066, 2024-03-31T23:37:38.577 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_send_bytes": 276990921, 2024-03-31T23:37:38.577 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_created_connections": 9, 2024-03-31T23:37:38.577 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_active_connections": 5, 2024-03-31T23:37:38.578 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_running_total_time": 18.453654149, 2024-03-31T23:37:38.578 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_running_send_time": 7.653649646, 2024-03-31T23:37:38.578 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_running_recv_time": 12.761713218, 2024-03-31T23:37:38.578 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_running_fast_dispatch_time": 0.664472862, 2024-03-31T23:37:38.578 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_send_messages_queue_lat": { 2024-03-31T23:37:38.578 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 374352, 2024-03-31T23:37:38.578 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 8.751286705, 2024-03-31T23:37:38.578 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000023377 2024-03-31T23:37:38.578 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-31T23:37:38.578 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_handle_ack_lat": { 2024-03-31T23:37:38.578 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 730661, 2024-03-31T23:37:38.578 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.098825855, 2024-03-31T23:37:38.578 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000135 2024-03-31T23:37:38.578 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-31T23:37:38.579 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_recv_encrypted_bytes": 1726368, 2024-03-31T23:37:38.579 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_send_encrypted_bytes": 342320 2024-03-31T23:37:38.579 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-31T23:37:38.579 INFO:teuthology.orchestra.run.smithi162.stdout: "AsyncMessenger::Worker-1": { 2024-03-31T23:37:38.579 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_recv_messages": 41568, 2024-03-31T23:37:38.579 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_send_messages": 41917, 2024-03-31T23:37:38.579 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_recv_bytes": 10581246, 2024-03-31T23:37:38.579 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_send_bytes": 330121313, 2024-03-31T23:37:38.579 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_created_connections": 20, 2024-03-31T23:37:38.579 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_active_connections": 5, 2024-03-31T23:37:38.579 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_running_total_time": 4.063774322, 2024-03-31T23:37:38.579 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_running_send_time": 1.743825307, 2024-03-31T23:37:38.579 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_running_recv_time": 0.732004697, 2024-03-31T23:37:38.579 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_running_fast_dispatch_time": 1.360461559, 2024-03-31T23:37:38.580 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_send_messages_queue_lat": { 2024-03-31T23:37:38.580 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 41915, 2024-03-31T23:37:38.580 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 2.575095062, 2024-03-31T23:37:38.580 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000061436 2024-03-31T23:37:38.580 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-31T23:37:38.580 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_handle_ack_lat": { 2024-03-31T23:37:38.580 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 27, 2024-03-31T23:37:38.580 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000003246, 2024-03-31T23:37:38.580 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000120 2024-03-31T23:37:38.580 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-31T23:37:38.580 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_recv_encrypted_bytes": 5456, 2024-03-31T23:37:38.580 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_send_encrypted_bytes": 782320 2024-03-31T23:37:38.580 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-31T23:37:38.580 INFO:teuthology.orchestra.run.smithi162.stdout: "AsyncMessenger::Worker-2": { 2024-03-31T23:37:38.581 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_recv_messages": 1262928, 2024-03-31T23:37:38.581 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_send_messages": 1309587, 2024-03-31T23:37:38.581 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_recv_bytes": 401975083, 2024-03-31T23:37:38.581 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_send_bytes": 890355372, 2024-03-31T23:37:38.581 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_created_connections": 14, 2024-03-31T23:37:38.581 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_active_connections": 5, 2024-03-31T23:37:38.581 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_running_total_time": 52.460392006, 2024-03-31T23:37:38.581 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_running_send_time": 22.217475486, 2024-03-31T23:37:38.581 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_running_recv_time": 36.777904700, 2024-03-31T23:37:38.581 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_running_fast_dispatch_time": 0.842821408, 2024-03-31T23:37:38.581 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_send_messages_queue_lat": { 2024-03-31T23:37:38.581 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 1309585, 2024-03-31T23:37:38.581 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 28.007283351, 2024-03-31T23:37:38.581 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000021386 2024-03-31T23:37:38.582 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-31T23:37:38.582 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_handle_ack_lat": { 2024-03-31T23:37:38.582 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 2455240, 2024-03-31T23:37:38.582 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.327764923, 2024-03-31T23:37:38.582 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000133 2024-03-31T23:37:38.582 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-31T23:37:38.582 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_recv_encrypted_bytes": 928, 2024-03-31T23:37:38.582 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_send_encrypted_bytes": 736 2024-03-31T23:37:38.582 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-31T23:37:38.582 INFO:teuthology.orchestra.run.smithi162.stdout: "cct": { 2024-03-31T23:37:38.582 INFO:teuthology.orchestra.run.smithi162.stdout: "total_workers": 1, 2024-03-31T23:37:38.582 INFO:teuthology.orchestra.run.smithi162.stdout: "unhealthy_workers": 0 2024-03-31T23:37:38.583 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-31T23:37:38.583 INFO:teuthology.orchestra.run.smithi162.stdout: "finisher-MDSRank": { 2024-03-31T23:37:38.583 INFO:teuthology.orchestra.run.smithi162.stdout: "queue_len": 0, 2024-03-31T23:37:38.583 INFO:teuthology.orchestra.run.smithi162.stdout: "complete_latency": { 2024-03-31T23:37:38.583 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 105722, 2024-03-31T23:37:38.583 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 60.391847695, 2024-03-31T23:37:38.583 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000571232 2024-03-31T23:37:38.583 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-03-31T23:37:38.583 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-31T23:37:38.583 INFO:teuthology.orchestra.run.smithi162.stdout: "finisher-PurgeQueue": { 2024-03-31T23:37:38.583 INFO:teuthology.orchestra.run.smithi162.stdout: "queue_len": 0, 2024-03-31T23:37:38.583 INFO:teuthology.orchestra.run.smithi162.stdout: "complete_latency": { 2024-03-31T23:37:38.583 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 26912, 2024-03-31T23:37:38.583 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 3.100647833, 2024-03-31T23:37:38.584 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000115214 2024-03-31T23:37:38.584 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-03-31T23:37:38.584 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-31T23:37:38.584 INFO:teuthology.orchestra.run.smithi162.stdout: "mds": { 2024-03-31T23:37:38.584 INFO:teuthology.orchestra.run.smithi162.stdout: "request": 884358, 2024-03-31T23:37:38.584 INFO:teuthology.orchestra.run.smithi162.stdout: "reply": 846598, 2024-03-31T23:37:38.584 INFO:teuthology.orchestra.run.smithi162.stdout: "reply_latency": { 2024-03-31T23:37:38.584 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 846598, 2024-03-31T23:37:38.584 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 756.104294512, 2024-03-31T23:37:38.584 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000893109 2024-03-31T23:37:38.584 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-31T23:37:38.584 INFO:teuthology.orchestra.run.smithi162.stdout: "slow_reply": 0, 2024-03-31T23:37:38.584 INFO:teuthology.orchestra.run.smithi162.stdout: "forward": 37760, 2024-03-31T23:37:38.584 INFO:teuthology.orchestra.run.smithi162.stdout: "dir_fetch_complete": 37, 2024-03-31T23:37:38.585 INFO:teuthology.orchestra.run.smithi162.stdout: "dir_fetch_keys": 0, 2024-03-31T23:37:38.585 INFO:teuthology.orchestra.run.smithi162.stdout: "dir_commit": 60, 2024-03-31T23:37:38.585 INFO:teuthology.orchestra.run.smithi162.stdout: "dir_split": 4, 2024-03-31T23:37:38.585 INFO:teuthology.orchestra.run.smithi162.stdout: "dir_merge": 0, 2024-03-31T23:37:38.585 INFO:teuthology.orchestra.run.smithi162.stdout: "inodes": 4604, 2024-03-31T23:37:38.585 INFO:teuthology.orchestra.run.smithi162.stdout: "inodes_top": 2219, 2024-03-31T23:37:38.585 INFO:teuthology.orchestra.run.smithi162.stdout: "inodes_bottom": 2325, 2024-03-31T23:37:38.585 INFO:teuthology.orchestra.run.smithi162.stdout: "inodes_pin_tail": 60, 2024-03-31T23:37:38.585 INFO:teuthology.orchestra.run.smithi162.stdout: "inodes_pinned": 1387, 2024-03-31T23:37:38.585 INFO:teuthology.orchestra.run.smithi162.stdout: "inodes_expired": 2239, 2024-03-31T23:37:38.585 INFO:teuthology.orchestra.run.smithi162.stdout: "inodes_with_caps": 0, 2024-03-31T23:37:38.585 INFO:teuthology.orchestra.run.smithi162.stdout: "caps": 0, 2024-03-31T23:37:38.585 INFO:teuthology.orchestra.run.smithi162.stdout: "subtrees": 4, 2024-03-31T23:37:38.585 INFO:teuthology.orchestra.run.smithi162.stdout: "traverse": 1727496, 2024-03-31T23:37:38.585 INFO:teuthology.orchestra.run.smithi162.stdout: "traverse_hit": 1649959, 2024-03-31T23:37:38.586 INFO:teuthology.orchestra.run.smithi162.stdout: "traverse_forward": 7432, 2024-03-31T23:37:38.586 INFO:teuthology.orchestra.run.smithi162.stdout: "traverse_discover": 522, 2024-03-31T23:37:38.586 INFO:teuthology.orchestra.run.smithi162.stdout: "traverse_dir_fetch": 21, 2024-03-31T23:37:38.586 INFO:teuthology.orchestra.run.smithi162.stdout: "traverse_remote_ino": 0, 2024-03-31T23:37:38.586 INFO:teuthology.orchestra.run.smithi162.stdout: "traverse_lock": 286, 2024-03-31T23:37:38.586 INFO:teuthology.orchestra.run.smithi162.stdout: "load_cent": 0, 2024-03-31T23:37:38.586 INFO:teuthology.orchestra.run.smithi162.stdout: "q": 0, 2024-03-31T23:37:38.586 INFO:teuthology.orchestra.run.smithi162.stdout: "exported": 47, 2024-03-31T23:37:38.586 INFO:teuthology.orchestra.run.smithi162.stdout: "exported_inodes": 1280, 2024-03-31T23:37:38.586 INFO:teuthology.orchestra.run.smithi162.stdout: "imported": 89, 2024-03-31T23:37:38.586 INFO:teuthology.orchestra.run.smithi162.stdout: "imported_inodes": 3741, 2024-03-31T23:37:38.586 INFO:teuthology.orchestra.run.smithi162.stdout: "openino_dir_fetch": 0, 2024-03-31T23:37:38.586 INFO:teuthology.orchestra.run.smithi162.stdout: "openino_backtrace_fetch": 29, 2024-03-31T23:37:38.587 INFO:teuthology.orchestra.run.smithi162.stdout: "openino_peer_discover": 0, 2024-03-31T23:37:38.587 INFO:teuthology.orchestra.run.smithi162.stdout: "root_rfiles": 4, 2024-03-31T23:37:38.587 INFO:teuthology.orchestra.run.smithi162.stdout: "root_rbytes": 234, 2024-03-31T23:37:38.587 INFO:teuthology.orchestra.run.smithi162.stdout: "root_rsnaps": 6, 2024-03-31T23:37:38.587 INFO:teuthology.orchestra.run.smithi162.stdout: "scrub_backtrace_fetch": 29, 2024-03-31T23:37:38.587 INFO:teuthology.orchestra.run.smithi162.stdout: "scrub_set_tag": 0, 2024-03-31T23:37:38.587 INFO:teuthology.orchestra.run.smithi162.stdout: "scrub_backtrace_repaired": 0, 2024-03-31T23:37:38.587 INFO:teuthology.orchestra.run.smithi162.stdout: "scrub_inotable_repaired": 0, 2024-03-31T23:37:38.587 INFO:teuthology.orchestra.run.smithi162.stdout: "scrub_dir_inodes": 0, 2024-03-31T23:37:38.587 INFO:teuthology.orchestra.run.smithi162.stdout: "scrub_dir_base_inodes": 0, 2024-03-31T23:37:38.587 INFO:teuthology.orchestra.run.smithi162.stdout: "scrub_dirfrag_rstats": 0, 2024-03-31T23:37:38.587 INFO:teuthology.orchestra.run.smithi162.stdout: "scrub_file_inodes": 29, 2024-03-31T23:37:38.587 INFO:teuthology.orchestra.run.smithi162.stdout: "handle_inode_file_caps": 32, 2024-03-31T23:37:38.588 INFO:teuthology.orchestra.run.smithi162.stdout: "ceph_cap_op_revoke": 40003, 2024-03-31T23:37:38.588 INFO:teuthology.orchestra.run.smithi162.stdout: "ceph_cap_op_grant": 39007, 2024-03-31T23:37:38.588 INFO:teuthology.orchestra.run.smithi162.stdout: "ceph_cap_op_trunc": 0, 2024-03-31T23:37:38.588 INFO:teuthology.orchestra.run.smithi162.stdout: "ceph_cap_op_flushsnap_ack": 1, 2024-03-31T23:37:38.588 INFO:teuthology.orchestra.run.smithi162.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-31T23:37:38.588 INFO:teuthology.orchestra.run.smithi162.stdout: "handle_client_caps": 59582, 2024-03-31T23:37:38.588 INFO:teuthology.orchestra.run.smithi162.stdout: "handle_client_caps_dirty": 14576, 2024-03-31T23:37:38.588 INFO:teuthology.orchestra.run.smithi162.stdout: "handle_client_cap_release": 1173, 2024-03-31T23:37:38.588 INFO:teuthology.orchestra.run.smithi162.stdout: "process_request_cap_release": 49827 2024-03-31T23:37:38.588 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-31T23:37:38.588 INFO:teuthology.orchestra.run.smithi162.stdout: "mds_cache": { 2024-03-31T23:37:38.588 INFO:teuthology.orchestra.run.smithi162.stdout: "dir_update": 571, 2024-03-31T23:37:38.588 INFO:teuthology.orchestra.run.smithi162.stdout: "dir_update_receipt": 2122, 2024-03-31T23:37:38.588 INFO:teuthology.orchestra.run.smithi162.stdout: "dir_try_discover": 1172, 2024-03-31T23:37:38.588 INFO:teuthology.orchestra.run.smithi162.stdout: "dir_send_discover": 2260, 2024-03-31T23:37:38.589 INFO:teuthology.orchestra.run.smithi162.stdout: "dir_handle_discover": 4140, 2024-03-31T23:37:38.589 INFO:teuthology.orchestra.run.smithi162.stdout: "num_strays": 50, 2024-03-31T23:37:38.589 INFO:teuthology.orchestra.run.smithi162.stdout: "num_strays_delayed": 0, 2024-03-31T23:37:38.589 INFO:teuthology.orchestra.run.smithi162.stdout: "num_strays_enqueuing": 0, 2024-03-31T23:37:38.589 INFO:teuthology.orchestra.run.smithi162.stdout: "strays_created": 13669, 2024-03-31T23:37:38.589 INFO:teuthology.orchestra.run.smithi162.stdout: "strays_enqueued": 13619, 2024-03-31T23:37:38.589 INFO:teuthology.orchestra.run.smithi162.stdout: "strays_reintegrated": 0, 2024-03-31T23:37:38.589 INFO:teuthology.orchestra.run.smithi162.stdout: "strays_migrated": 0, 2024-03-31T23:37:38.589 INFO:teuthology.orchestra.run.smithi162.stdout: "num_recovering_processing": 0, 2024-03-31T23:37:38.589 INFO:teuthology.orchestra.run.smithi162.stdout: "num_recovering_enqueued": 0, 2024-03-31T23:37:38.589 INFO:teuthology.orchestra.run.smithi162.stdout: "num_recovering_prioritized": 0, 2024-03-31T23:37:38.589 INFO:teuthology.orchestra.run.smithi162.stdout: "recovery_started": 0, 2024-03-31T23:37:38.589 INFO:teuthology.orchestra.run.smithi162.stdout: "recovery_completed": 0, 2024-03-31T23:37:38.589 INFO:teuthology.orchestra.run.smithi162.stdout: "ireq_quiesce_path": 0, 2024-03-31T23:37:38.590 INFO:teuthology.orchestra.run.smithi162.stdout: "ireq_quiesce_inode": 0, 2024-03-31T23:37:38.590 INFO:teuthology.orchestra.run.smithi162.stdout: "ireq_enqueue_scrub": 0, 2024-03-31T23:37:38.590 INFO:teuthology.orchestra.run.smithi162.stdout: "ireq_exportdir": 49, 2024-03-31T23:37:38.590 INFO:teuthology.orchestra.run.smithi162.stdout: "ireq_flush": 0, 2024-03-31T23:37:38.590 INFO:teuthology.orchestra.run.smithi162.stdout: "ireq_fragmentdir": 4, 2024-03-31T23:37:38.590 INFO:teuthology.orchestra.run.smithi162.stdout: "ireq_fragstats": 0, 2024-03-31T23:37:38.590 INFO:teuthology.orchestra.run.smithi162.stdout: "ireq_inodestats": 0 2024-03-31T23:37:38.590 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-31T23:37:38.590 INFO:teuthology.orchestra.run.smithi162.stdout: "mds_log": { 2024-03-31T23:37:38.590 INFO:teuthology.orchestra.run.smithi162.stdout: "evlrg": 1, 2024-03-31T23:37:38.590 INFO:teuthology.orchestra.run.smithi162.stdout: "evadd": 127845, 2024-03-31T23:37:38.590 INFO:teuthology.orchestra.run.smithi162.stdout: "evex": 69939, 2024-03-31T23:37:38.590 INFO:teuthology.orchestra.run.smithi162.stdout: "evtrm": 61451, 2024-03-31T23:37:38.590 INFO:teuthology.orchestra.run.smithi162.stdout: "ev": 66394, 2024-03-31T23:37:38.591 INFO:teuthology.orchestra.run.smithi162.stdout: "evexg": 0, 2024-03-31T23:37:38.591 INFO:teuthology.orchestra.run.smithi162.stdout: "evexd": 8488, 2024-03-31T23:37:38.591 INFO:teuthology.orchestra.run.smithi162.stdout: "segadd": 217, 2024-03-31T23:37:38.591 INFO:teuthology.orchestra.run.smithi162.stdout: "segex": 89, 2024-03-31T23:37:38.591 INFO:teuthology.orchestra.run.smithi162.stdout: "segtrm": 76, 2024-03-31T23:37:38.591 INFO:teuthology.orchestra.run.smithi162.stdout: "seg": 141, 2024-03-31T23:37:38.591 INFO:teuthology.orchestra.run.smithi162.stdout: "segmjr": 6, 2024-03-31T23:37:38.591 INFO:teuthology.orchestra.run.smithi162.stdout: "segexg": 0, 2024-03-31T23:37:38.591 INFO:teuthology.orchestra.run.smithi162.stdout: "segexd": 13, 2024-03-31T23:37:38.591 INFO:teuthology.orchestra.run.smithi162.stdout: "expos": 102516747, 2024-03-31T23:37:38.591 INFO:teuthology.orchestra.run.smithi162.stdout: "wrpos": 651970097, 2024-03-31T23:37:38.591 INFO:teuthology.orchestra.run.smithi162.stdout: "rdpos": 0, 2024-03-31T23:37:38.591 INFO:teuthology.orchestra.run.smithi162.stdout: "jlat": { 2024-03-31T23:37:38.591 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 54969, 2024-03-31T23:37:38.592 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 262.665275791, 2024-03-31T23:37:38.592 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.004778425 2024-03-31T23:37:38.592 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-31T23:37:38.592 INFO:teuthology.orchestra.run.smithi162.stdout: "replayed": 0 2024-03-31T23:37:38.592 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-31T23:37:38.592 INFO:teuthology.orchestra.run.smithi162.stdout: "mds_mem": { 2024-03-31T23:37:38.592 INFO:teuthology.orchestra.run.smithi162.stdout: "ino": 3877, 2024-03-31T23:37:38.592 INFO:teuthology.orchestra.run.smithi162.stdout: "ino+": 19130, 2024-03-31T23:37:38.592 INFO:teuthology.orchestra.run.smithi162.stdout: "ino-": 15253, 2024-03-31T23:37:38.592 INFO:teuthology.orchestra.run.smithi162.stdout: "dir": 156, 2024-03-31T23:37:38.592 INFO:teuthology.orchestra.run.smithi162.stdout: "dir+": 1306, 2024-03-31T23:37:38.592 INFO:teuthology.orchestra.run.smithi162.stdout: "dir-": 1150, 2024-03-31T23:37:38.592 INFO:teuthology.orchestra.run.smithi162.stdout: "dn": 4604, 2024-03-31T23:37:38.592 INFO:teuthology.orchestra.run.smithi162.stdout: "dn+": 27900, 2024-03-31T23:37:38.592 INFO:teuthology.orchestra.run.smithi162.stdout: "dn-": 23296, 2024-03-31T23:37:38.593 INFO:teuthology.orchestra.run.smithi162.stdout: "cap": 0, 2024-03-31T23:37:38.593 INFO:teuthology.orchestra.run.smithi162.stdout: "cap+": 14059, 2024-03-31T23:37:38.593 INFO:teuthology.orchestra.run.smithi162.stdout: "cap-": 14059, 2024-03-31T23:37:38.593 INFO:teuthology.orchestra.run.smithi162.stdout: "rss": 99572, 2024-03-31T23:37:38.593 INFO:teuthology.orchestra.run.smithi162.stdout: "heap": 223612 2024-03-31T23:37:38.593 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-31T23:37:38.593 INFO:teuthology.orchestra.run.smithi162.stdout: "mds_server": { 2024-03-31T23:37:38.593 INFO:teuthology.orchestra.run.smithi162.stdout: "dispatch_client_request": 964437, 2024-03-31T23:37:38.593 INFO:teuthology.orchestra.run.smithi162.stdout: "dispatch_server_request": 49910, 2024-03-31T23:37:38.593 INFO:teuthology.orchestra.run.smithi162.stdout: "handle_client_request": 884358, 2024-03-31T23:37:38.593 INFO:teuthology.orchestra.run.smithi162.stdout: "handle_client_session": 1165, 2024-03-31T23:37:38.593 INFO:teuthology.orchestra.run.smithi162.stdout: "handle_peer_request": 112289, 2024-03-31T23:37:38.593 INFO:teuthology.orchestra.run.smithi162.stdout: "req_create_latency": { 2024-03-31T23:37:38.593 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 13494, 2024-03-31T23:37:38.593 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 40.415247228, 2024-03-31T23:37:38.594 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.002995053 2024-03-31T23:37:38.594 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-31T23:37:38.594 INFO:teuthology.orchestra.run.smithi162.stdout: "req_getattr_latency": { 2024-03-31T23:37:38.594 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 759210, 2024-03-31T23:37:38.594 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 533.663224624, 2024-03-31T23:37:38.594 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000702919 2024-03-31T23:37:38.594 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-31T23:37:38.594 INFO:teuthology.orchestra.run.smithi162.stdout: "req_getfilelock_latency": { 2024-03-31T23:37:38.594 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-31T23:37:38.594 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-31T23:37:38.594 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-31T23:37:38.594 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-31T23:37:38.594 INFO:teuthology.orchestra.run.smithi162.stdout: "req_link_latency": { 2024-03-31T23:37:38.594 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-31T23:37:38.595 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-31T23:37:38.595 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-31T23:37:38.595 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-31T23:37:38.595 INFO:teuthology.orchestra.run.smithi162.stdout: "req_lookup_latency": { 2024-03-31T23:37:38.595 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 19359, 2024-03-31T23:37:38.595 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 16.635845299, 2024-03-31T23:37:38.595 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000859333 2024-03-31T23:37:38.595 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-31T23:37:38.595 INFO:teuthology.orchestra.run.smithi162.stdout: "req_lookuphash_latency": { 2024-03-31T23:37:38.595 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-31T23:37:38.595 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-31T23:37:38.595 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-31T23:37:38.595 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-31T23:37:38.595 INFO:teuthology.orchestra.run.smithi162.stdout: "req_lookupino_latency": { 2024-03-31T23:37:38.596 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-31T23:37:38.596 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-31T23:37:38.596 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-31T23:37:38.596 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-31T23:37:38.596 INFO:teuthology.orchestra.run.smithi162.stdout: "req_lookupname_latency": { 2024-03-31T23:37:38.596 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-31T23:37:38.596 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-31T23:37:38.596 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-31T23:37:38.596 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-31T23:37:38.596 INFO:teuthology.orchestra.run.smithi162.stdout: "req_lookupparent_latency": { 2024-03-31T23:37:38.596 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-31T23:37:38.596 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-31T23:37:38.596 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-31T23:37:38.596 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-31T23:37:38.597 INFO:teuthology.orchestra.run.smithi162.stdout: "req_lookupsnap_latency": { 2024-03-31T23:37:38.597 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-31T23:37:38.597 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-31T23:37:38.597 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-31T23:37:38.597 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-31T23:37:38.597 INFO:teuthology.orchestra.run.smithi162.stdout: "req_lssnap_latency": { 2024-03-31T23:37:38.597 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-31T23:37:38.598 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-31T23:37:38.598 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-31T23:37:38.598 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-31T23:37:38.598 INFO:teuthology.orchestra.run.smithi162.stdout: "req_mkdir_latency": { 2024-03-31T23:37:38.598 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-31T23:37:38.598 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-31T23:37:38.598 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-31T23:37:38.598 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-31T23:37:38.598 INFO:teuthology.orchestra.run.smithi162.stdout: "req_mknod_latency": { 2024-03-31T23:37:38.598 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-31T23:37:38.598 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-31T23:37:38.598 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-31T23:37:38.598 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-31T23:37:38.598 INFO:teuthology.orchestra.run.smithi162.stdout: "req_mksnap_latency": { 2024-03-31T23:37:38.599 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-31T23:37:38.599 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-31T23:37:38.599 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-31T23:37:38.599 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-31T23:37:38.599 INFO:teuthology.orchestra.run.smithi162.stdout: "req_open_latency": { 2024-03-31T23:37:38.599 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 8228, 2024-03-31T23:37:38.599 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 7.785249042, 2024-03-31T23:37:38.599 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000946189 2024-03-31T23:37:38.599 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-31T23:37:38.599 INFO:teuthology.orchestra.run.smithi162.stdout: "req_readdir_latency": { 2024-03-31T23:37:38.599 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 29068, 2024-03-31T23:37:38.599 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 86.425257743, 2024-03-31T23:37:38.599 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.002973209 2024-03-31T23:37:38.599 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-31T23:37:38.600 INFO:teuthology.orchestra.run.smithi162.stdout: "req_rename_latency": { 2024-03-31T23:37:38.600 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 3022, 2024-03-31T23:37:38.600 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 25.948110292, 2024-03-31T23:37:38.600 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.008586403 2024-03-31T23:37:38.600 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-31T23:37:38.600 INFO:teuthology.orchestra.run.smithi162.stdout: "req_renamesnap_latency": { 2024-03-31T23:37:38.600 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-31T23:37:38.600 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-31T23:37:38.600 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-31T23:37:38.600 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-31T23:37:38.600 INFO:teuthology.orchestra.run.smithi162.stdout: "req_snapdiff_latency": { 2024-03-31T23:37:38.600 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-31T23:37:38.600 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-31T23:37:38.600 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-31T23:37:38.600 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-31T23:37:38.601 INFO:teuthology.orchestra.run.smithi162.stdout: "req_rmdir_latency": { 2024-03-31T23:37:38.601 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 50, 2024-03-31T23:37:38.601 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.483350039, 2024-03-31T23:37:38.601 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.009667000 2024-03-31T23:37:38.601 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-31T23:37:38.601 INFO:teuthology.orchestra.run.smithi162.stdout: "req_rmsnap_latency": { 2024-03-31T23:37:38.601 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-31T23:37:38.601 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-31T23:37:38.601 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-31T23:37:38.601 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-31T23:37:38.601 INFO:teuthology.orchestra.run.smithi162.stdout: "req_rmxattr_latency": { 2024-03-31T23:37:38.601 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-31T23:37:38.601 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-31T23:37:38.601 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-31T23:37:38.602 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-31T23:37:38.602 INFO:teuthology.orchestra.run.smithi162.stdout: "req_setattr_latency": { 2024-03-31T23:37:38.602 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-31T23:37:38.602 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-31T23:37:38.602 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-31T23:37:38.602 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-31T23:37:38.602 INFO:teuthology.orchestra.run.smithi162.stdout: "req_setdirlayout_latency": { 2024-03-31T23:37:38.602 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-31T23:37:38.602 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-31T23:37:38.602 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-31T23:37:38.602 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-31T23:37:38.602 INFO:teuthology.orchestra.run.smithi162.stdout: "req_setfilelock_latency": { 2024-03-31T23:37:38.602 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 548, 2024-03-31T23:37:38.603 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.320982652, 2024-03-31T23:37:38.603 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000585734 2024-03-31T23:37:38.603 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-31T23:37:38.603 INFO:teuthology.orchestra.run.smithi162.stdout: "req_setlayout_latency": { 2024-03-31T23:37:38.603 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-31T23:37:38.603 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-31T23:37:38.603 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-31T23:37:38.603 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-31T23:37:38.603 INFO:teuthology.orchestra.run.smithi162.stdout: "req_setxattr_latency": { 2024-03-31T23:37:38.603 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-31T23:37:38.603 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-31T23:37:38.603 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-31T23:37:38.603 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-31T23:37:38.603 INFO:teuthology.orchestra.run.smithi162.stdout: "req_symlink_latency": { 2024-03-31T23:37:38.604 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-31T23:37:38.604 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-31T23:37:38.604 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-31T23:37:38.604 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-31T23:37:38.604 INFO:teuthology.orchestra.run.smithi162.stdout: "req_unlink_latency": { 2024-03-31T23:37:38.604 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 13619, 2024-03-31T23:37:38.604 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 44.427027593, 2024-03-31T23:37:38.604 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.003262135 2024-03-31T23:37:38.604 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-31T23:37:38.604 INFO:teuthology.orchestra.run.smithi162.stdout: "cap_revoke_eviction": 0, 2024-03-31T23:37:38.604 INFO:teuthology.orchestra.run.smithi162.stdout: "cap_acquisition_throttle": 0, 2024-03-31T23:37:38.604 INFO:teuthology.orchestra.run.smithi162.stdout: "req_getvxattr_latency": { 2024-03-31T23:37:38.604 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-31T23:37:38.604 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-31T23:37:38.605 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-31T23:37:38.605 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-03-31T23:37:38.605 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-31T23:37:38.605 INFO:teuthology.orchestra.run.smithi162.stdout: "mds_sessions": { 2024-03-31T23:37:38.605 INFO:teuthology.orchestra.run.smithi162.stdout: "session_count": 2, 2024-03-31T23:37:38.605 INFO:teuthology.orchestra.run.smithi162.stdout: "session_add": 4, 2024-03-31T23:37:38.605 INFO:teuthology.orchestra.run.smithi162.stdout: "session_remove": 2, 2024-03-31T23:37:38.605 INFO:teuthology.orchestra.run.smithi162.stdout: "sessions_open": 2, 2024-03-31T23:37:38.605 INFO:teuthology.orchestra.run.smithi162.stdout: "sessions_stale": 0, 2024-03-31T23:37:38.605 INFO:teuthology.orchestra.run.smithi162.stdout: "total_load": 86634, 2024-03-31T23:37:38.605 INFO:teuthology.orchestra.run.smithi162.stdout: "average_load": 43317, 2024-03-31T23:37:38.605 INFO:teuthology.orchestra.run.smithi162.stdout: "avg_session_uptime": 1554, 2024-03-31T23:37:38.605 INFO:teuthology.orchestra.run.smithi162.stdout: "mdthresh_evicted": 0 2024-03-31T23:37:38.605 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-31T23:37:38.605 INFO:teuthology.orchestra.run.smithi162.stdout: "mempool": { 2024-03-31T23:37:38.606 INFO:teuthology.orchestra.run.smithi162.stdout: "bloom_filter_bytes": 0, 2024-03-31T23:37:38.606 INFO:teuthology.orchestra.run.smithi162.stdout: "bloom_filter_items": 0, 2024-03-31T23:37:38.606 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_alloc_bytes": 0, 2024-03-31T23:37:38.606 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_alloc_items": 0, 2024-03-31T23:37:38.606 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_cache_data_bytes": 0, 2024-03-31T23:37:38.606 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_cache_data_items": 0, 2024-03-31T23:37:38.606 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-31T23:37:38.606 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_cache_onode_items": 0, 2024-03-31T23:37:38.606 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-31T23:37:38.606 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_cache_meta_items": 0, 2024-03-31T23:37:38.606 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_cache_other_bytes": 0, 2024-03-31T23:37:38.606 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_cache_other_items": 0, 2024-03-31T23:37:38.606 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-31T23:37:38.606 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_cache_buffer_items": 0, 2024-03-31T23:37:38.607 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_extent_bytes": 0, 2024-03-31T23:37:38.607 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_extent_items": 0, 2024-03-31T23:37:38.607 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_blob_bytes": 0, 2024-03-31T23:37:38.607 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_blob_items": 0, 2024-03-31T23:37:38.607 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-31T23:37:38.607 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_shared_blob_items": 0, 2024-03-31T23:37:38.607 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-31T23:37:38.607 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_inline_bl_items": 0, 2024-03-31T23:37:38.607 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_fsck_bytes": 0, 2024-03-31T23:37:38.607 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_fsck_items": 0, 2024-03-31T23:37:38.607 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_txc_bytes": 0, 2024-03-31T23:37:38.607 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_txc_items": 0, 2024-03-31T23:37:38.607 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-31T23:37:38.607 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_writing_deferred_items": 0, 2024-03-31T23:37:38.608 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_writing_bytes": 0, 2024-03-31T23:37:38.608 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_writing_items": 0, 2024-03-31T23:37:38.608 INFO:teuthology.orchestra.run.smithi162.stdout: "bluefs_bytes": 0, 2024-03-31T23:37:38.608 INFO:teuthology.orchestra.run.smithi162.stdout: "bluefs_items": 0, 2024-03-31T23:37:38.608 INFO:teuthology.orchestra.run.smithi162.stdout: "bluefs_file_reader_bytes": 0, 2024-03-31T23:37:38.608 INFO:teuthology.orchestra.run.smithi162.stdout: "bluefs_file_reader_items": 0, 2024-03-31T23:37:38.608 INFO:teuthology.orchestra.run.smithi162.stdout: "bluefs_file_writer_bytes": 0, 2024-03-31T23:37:38.608 INFO:teuthology.orchestra.run.smithi162.stdout: "bluefs_file_writer_items": 0, 2024-03-31T23:37:38.608 INFO:teuthology.orchestra.run.smithi162.stdout: "buffer_anon_bytes": 709986, 2024-03-31T23:37:38.608 INFO:teuthology.orchestra.run.smithi162.stdout: "buffer_anon_items": 213, 2024-03-31T23:37:38.608 INFO:teuthology.orchestra.run.smithi162.stdout: "buffer_meta_bytes": 0, 2024-03-31T23:37:38.608 INFO:teuthology.orchestra.run.smithi162.stdout: "buffer_meta_items": 0, 2024-03-31T23:37:38.608 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_bytes": 0, 2024-03-31T23:37:38.608 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_items": 0, 2024-03-31T23:37:38.608 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_mapbl_bytes": 0, 2024-03-31T23:37:38.609 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_mapbl_items": 0, 2024-03-31T23:37:38.609 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_pglog_bytes": 0, 2024-03-31T23:37:38.609 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_pglog_items": 0, 2024-03-31T23:37:38.609 INFO:teuthology.orchestra.run.smithi162.stdout: "osdmap_bytes": 9704, 2024-03-31T23:37:38.609 INFO:teuthology.orchestra.run.smithi162.stdout: "osdmap_items": 186, 2024-03-31T23:37:38.609 INFO:teuthology.orchestra.run.smithi162.stdout: "osdmap_mapping_bytes": 0, 2024-03-31T23:37:38.609 INFO:teuthology.orchestra.run.smithi162.stdout: "osdmap_mapping_items": 0, 2024-03-31T23:37:38.609 INFO:teuthology.orchestra.run.smithi162.stdout: "pgmap_bytes": 0, 2024-03-31T23:37:38.609 INFO:teuthology.orchestra.run.smithi162.stdout: "pgmap_items": 0, 2024-03-31T23:37:38.609 INFO:teuthology.orchestra.run.smithi162.stdout: "mds_co_bytes": 12178248, 2024-03-31T23:37:38.609 INFO:teuthology.orchestra.run.smithi162.stdout: "mds_co_items": 47823, 2024-03-31T23:37:38.609 INFO:teuthology.orchestra.run.smithi162.stdout: "unittest_1_bytes": 0, 2024-03-31T23:37:38.609 INFO:teuthology.orchestra.run.smithi162.stdout: "unittest_1_items": 0, 2024-03-31T23:37:38.609 INFO:teuthology.orchestra.run.smithi162.stdout: "unittest_2_bytes": 0, 2024-03-31T23:37:38.610 INFO:teuthology.orchestra.run.smithi162.stdout: "unittest_2_items": 0 2024-03-31T23:37:38.610 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-31T23:37:38.610 INFO:teuthology.orchestra.run.smithi162.stdout: "objecter": { 2024-03-31T23:37:38.610 INFO:teuthology.orchestra.run.smithi162.stdout: "op_active": 0, 2024-03-31T23:37:38.610 INFO:teuthology.orchestra.run.smithi162.stdout: "op_laggy": 0, 2024-03-31T23:37:38.610 INFO:teuthology.orchestra.run.smithi162.stdout: "op_send": 89606, 2024-03-31T23:37:38.610 INFO:teuthology.orchestra.run.smithi162.stdout: "op_send_bytes": 650937016, 2024-03-31T23:37:38.610 INFO:teuthology.orchestra.run.smithi162.stdout: "op_resend": 33, 2024-03-31T23:37:38.610 INFO:teuthology.orchestra.run.smithi162.stdout: "op_reply": 89606, 2024-03-31T23:37:38.610 INFO:teuthology.orchestra.run.smithi162.stdout: "op_latency": { 2024-03-31T23:37:38.610 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 89606, 2024-03-31T23:37:38.610 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 420.422949156, 2024-03-31T23:37:38.610 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.004691906 2024-03-31T23:37:38.610 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-31T23:37:38.611 INFO:teuthology.orchestra.run.smithi162.stdout: "op_inflight": 0, 2024-03-31T23:37:38.611 INFO:teuthology.orchestra.run.smithi162.stdout: "oplen_avg": { 2024-03-31T23:37:38.611 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 89606, 2024-03-31T23:37:38.611 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 92361 2024-03-31T23:37:38.611 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-31T23:37:38.611 INFO:teuthology.orchestra.run.smithi162.stdout: "op": 89606, 2024-03-31T23:37:38.611 INFO:teuthology.orchestra.run.smithi162.stdout: "op_r": 2267, 2024-03-31T23:37:38.611 INFO:teuthology.orchestra.run.smithi162.stdout: "op_w": 87339, 2024-03-31T23:37:38.611 INFO:teuthology.orchestra.run.smithi162.stdout: "op_rmw": 0, 2024-03-31T23:37:38.611 INFO:teuthology.orchestra.run.smithi162.stdout: "op_pg": 0, 2024-03-31T23:37:38.611 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_stat": 13, 2024-03-31T23:37:38.611 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_create": 87, 2024-03-31T23:37:38.611 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_read": 2176, 2024-03-31T23:37:38.611 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_write": 57145, 2024-03-31T23:37:38.611 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_writefull": 1236, 2024-03-31T23:37:38.612 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_writesame": 0, 2024-03-31T23:37:38.612 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_append": 0, 2024-03-31T23:37:38.612 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_zero": 0, 2024-03-31T23:37:38.612 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_truncate": 4, 2024-03-31T23:37:38.612 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_delete": 27427, 2024-03-31T23:37:38.612 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_mapext": 0, 2024-03-31T23:37:38.612 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_sparse_read": 0, 2024-03-31T23:37:38.612 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_clonerange": 0, 2024-03-31T23:37:38.612 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_getxattr": 66, 2024-03-31T23:37:38.612 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_setxattr": 147, 2024-03-31T23:37:38.612 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_cmpxattr": 0, 2024-03-31T23:37:38.613 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_rmxattr": 0, 2024-03-31T23:37:38.613 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_resetxattrs": 0, 2024-03-31T23:37:38.613 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_call": 0, 2024-03-31T23:37:38.613 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_watch": 0, 2024-03-31T23:37:38.613 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_notify": 0, 2024-03-31T23:37:38.613 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_src_cmpxattr": 0, 2024-03-31T23:37:38.613 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_pgls": 0, 2024-03-31T23:37:38.613 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_pgls_filter": 0, 2024-03-31T23:37:38.613 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_other": 0, 2024-03-31T23:37:38.613 INFO:teuthology.orchestra.run.smithi162.stdout: "linger_active": 0, 2024-03-31T23:37:38.613 INFO:teuthology.orchestra.run.smithi162.stdout: "linger_send": 0, 2024-03-31T23:37:38.613 INFO:teuthology.orchestra.run.smithi162.stdout: "linger_resend": 0, 2024-03-31T23:37:38.613 INFO:teuthology.orchestra.run.smithi162.stdout: "linger_ping": 0, 2024-03-31T23:37:38.613 INFO:teuthology.orchestra.run.smithi162.stdout: "poolop_active": 0, 2024-03-31T23:37:38.614 INFO:teuthology.orchestra.run.smithi162.stdout: "poolop_send": 0, 2024-03-31T23:37:38.614 INFO:teuthology.orchestra.run.smithi162.stdout: "poolop_resend": 0, 2024-03-31T23:37:38.614 INFO:teuthology.orchestra.run.smithi162.stdout: "poolstat_active": 0, 2024-03-31T23:37:38.614 INFO:teuthology.orchestra.run.smithi162.stdout: "poolstat_send": 0, 2024-03-31T23:37:38.614 INFO:teuthology.orchestra.run.smithi162.stdout: "poolstat_resend": 0, 2024-03-31T23:37:38.614 INFO:teuthology.orchestra.run.smithi162.stdout: "statfs_active": 0, 2024-03-31T23:37:38.614 INFO:teuthology.orchestra.run.smithi162.stdout: "statfs_send": 0, 2024-03-31T23:37:38.614 INFO:teuthology.orchestra.run.smithi162.stdout: "statfs_resend": 0, 2024-03-31T23:37:38.614 INFO:teuthology.orchestra.run.smithi162.stdout: "command_active": 0, 2024-03-31T23:37:38.614 INFO:teuthology.orchestra.run.smithi162.stdout: "command_send": 0, 2024-03-31T23:37:38.614 INFO:teuthology.orchestra.run.smithi162.stdout: "command_resend": 0, 2024-03-31T23:37:38.614 INFO:teuthology.orchestra.run.smithi162.stdout: "map_epoch": 139, 2024-03-31T23:37:38.614 INFO:teuthology.orchestra.run.smithi162.stdout: "map_full": 0, 2024-03-31T23:37:38.614 INFO:teuthology.orchestra.run.smithi162.stdout: "map_inc": 48, 2024-03-31T23:37:38.615 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_sessions": 12, 2024-03-31T23:37:38.615 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_session_open": 12, 2024-03-31T23:37:38.615 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_session_close": 0, 2024-03-31T23:37:38.615 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_laggy": 0, 2024-03-31T23:37:38.615 INFO:teuthology.orchestra.run.smithi162.stdout: "omap_wr": 2636, 2024-03-31T23:37:38.615 INFO:teuthology.orchestra.run.smithi162.stdout: "omap_rd": 99, 2024-03-31T23:37:38.615 INFO:teuthology.orchestra.run.smithi162.stdout: "omap_del": 1325 2024-03-31T23:37:38.615 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-31T23:37:38.615 INFO:teuthology.orchestra.run.smithi162.stdout: "oft": { 2024-03-31T23:37:38.615 INFO:teuthology.orchestra.run.smithi162.stdout: "omap_total_objs": 1, 2024-03-31T23:37:38.615 INFO:teuthology.orchestra.run.smithi162.stdout: "omap_total_kv_pairs": 0, 2024-03-31T23:37:38.615 INFO:teuthology.orchestra.run.smithi162.stdout: "omap_total_updates": 0, 2024-03-31T23:37:38.615 INFO:teuthology.orchestra.run.smithi162.stdout: "omap_total_removes": 0 2024-03-31T23:37:38.615 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-31T23:37:38.616 INFO:teuthology.orchestra.run.smithi162.stdout: "osd-slow-ops": { 2024-03-31T23:37:38.616 INFO:teuthology.orchestra.run.smithi162.stdout: "slow_ops_count": 1 2024-03-31T23:37:38.616 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-31T23:37:38.616 INFO:teuthology.orchestra.run.smithi162.stdout: "purge_queue": { 2024-03-31T23:37:38.616 INFO:teuthology.orchestra.run.smithi162.stdout: "pq_executing_ops": 0, 2024-03-31T23:37:38.616 INFO:teuthology.orchestra.run.smithi162.stdout: "pq_executing_ops_high_water": 16, 2024-03-31T23:37:38.616 INFO:teuthology.orchestra.run.smithi162.stdout: "pq_executing": 0, 2024-03-31T23:37:38.616 INFO:teuthology.orchestra.run.smithi162.stdout: "pq_executing_high_water": 8, 2024-03-31T23:37:38.616 INFO:teuthology.orchestra.run.smithi162.stdout: "pq_executed_ops": 27238, 2024-03-31T23:37:38.616 INFO:teuthology.orchestra.run.smithi162.stdout: "pq_executed": 13619, 2024-03-31T23:37:38.616 INFO:teuthology.orchestra.run.smithi162.stdout: "pq_item_in_journal": 0 2024-03-31T23:37:38.616 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-31T23:37:38.616 INFO:teuthology.orchestra.run.smithi162.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-31T23:37:38.616 INFO:teuthology.orchestra.run.smithi162.stdout: "val": 0, 2024-03-31T23:37:38.616 INFO:teuthology.orchestra.run.smithi162.stdout: "max": 104857600, 2024-03-31T23:37:38.617 INFO:teuthology.orchestra.run.smithi162.stdout: "get_started": 0, 2024-03-31T23:37:38.617 INFO:teuthology.orchestra.run.smithi162.stdout: "get": 1717528, 2024-03-31T23:37:38.617 INFO:teuthology.orchestra.run.smithi162.stdout: "get_sum": 378510818, 2024-03-31T23:37:38.617 INFO:teuthology.orchestra.run.smithi162.stdout: "get_or_fail_fail": 0, 2024-03-31T23:37:38.617 INFO:teuthology.orchestra.run.smithi162.stdout: "get_or_fail_success": 1717528, 2024-03-31T23:37:38.617 INFO:teuthology.orchestra.run.smithi162.stdout: "take": 0, 2024-03-31T23:37:38.617 INFO:teuthology.orchestra.run.smithi162.stdout: "take_sum": 0, 2024-03-31T23:37:38.617 INFO:teuthology.orchestra.run.smithi162.stdout: "put": 1717528, 2024-03-31T23:37:38.617 INFO:teuthology.orchestra.run.smithi162.stdout: "put_sum": 378510818, 2024-03-31T23:37:38.617 INFO:teuthology.orchestra.run.smithi162.stdout: "wait": { 2024-03-31T23:37:38.617 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-31T23:37:38.617 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-31T23:37:38.617 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-31T23:37:38.617 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-03-31T23:37:38.618 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-31T23:37:38.618 INFO:teuthology.orchestra.run.smithi162.stdout: "throttle-objecter_bytes": { 2024-03-31T23:37:38.618 INFO:teuthology.orchestra.run.smithi162.stdout: "val": 0, 2024-03-31T23:37:38.618 INFO:teuthology.orchestra.run.smithi162.stdout: "max": 104857600, 2024-03-31T23:37:38.618 INFO:teuthology.orchestra.run.smithi162.stdout: "get_started": 0, 2024-03-31T23:37:38.618 INFO:teuthology.orchestra.run.smithi162.stdout: "get": 0, 2024-03-31T23:37:38.618 INFO:teuthology.orchestra.run.smithi162.stdout: "get_sum": 0, 2024-03-31T23:37:38.618 INFO:teuthology.orchestra.run.smithi162.stdout: "get_or_fail_fail": 0, 2024-03-31T23:37:38.618 INFO:teuthology.orchestra.run.smithi162.stdout: "get_or_fail_success": 0, 2024-03-31T23:37:38.618 INFO:teuthology.orchestra.run.smithi162.stdout: "take": 89606, 2024-03-31T23:37:38.618 INFO:teuthology.orchestra.run.smithi162.stdout: "take_sum": 652430165, 2024-03-31T23:37:38.618 INFO:teuthology.orchestra.run.smithi162.stdout: "put": 89606, 2024-03-31T23:37:38.618 INFO:teuthology.orchestra.run.smithi162.stdout: "put_sum": 652430165, 2024-03-31T23:37:38.618 INFO:teuthology.orchestra.run.smithi162.stdout: "wait": { 2024-03-31T23:37:38.619 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-31T23:37:38.619 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-31T23:37:38.619 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-31T23:37:38.619 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-03-31T23:37:38.619 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-31T23:37:38.619 INFO:teuthology.orchestra.run.smithi162.stdout: "throttle-objecter_ops": { 2024-03-31T23:37:38.619 INFO:teuthology.orchestra.run.smithi162.stdout: "val": 0, 2024-03-31T23:37:38.619 INFO:teuthology.orchestra.run.smithi162.stdout: "max": 1024, 2024-03-31T23:37:38.619 INFO:teuthology.orchestra.run.smithi162.stdout: "get_started": 0, 2024-03-31T23:37:38.619 INFO:teuthology.orchestra.run.smithi162.stdout: "get": 0, 2024-03-31T23:37:38.619 INFO:teuthology.orchestra.run.smithi162.stdout: "get_sum": 0, 2024-03-31T23:37:38.619 INFO:teuthology.orchestra.run.smithi162.stdout: "get_or_fail_fail": 0, 2024-03-31T23:37:38.619 INFO:teuthology.orchestra.run.smithi162.stdout: "get_or_fail_success": 0, 2024-03-31T23:37:38.619 INFO:teuthology.orchestra.run.smithi162.stdout: "take": 89606, 2024-03-31T23:37:38.620 INFO:teuthology.orchestra.run.smithi162.stdout: "take_sum": 89606, 2024-03-31T23:37:38.620 INFO:teuthology.orchestra.run.smithi162.stdout: "put": 89606, 2024-03-31T23:37:38.620 INFO:teuthology.orchestra.run.smithi162.stdout: "put_sum": 89606, 2024-03-31T23:37:38.620 INFO:teuthology.orchestra.run.smithi162.stdout: "wait": { 2024-03-31T23:37:38.620 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-31T23:37:38.620 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-31T23:37:38.620 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-31T23:37:38.620 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-03-31T23:37:38.620 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-31T23:37:38.620 INFO:teuthology.orchestra.run.smithi162.stdout: "throttle-write_buf_throttle": { 2024-03-31T23:37:38.620 INFO:teuthology.orchestra.run.smithi162.stdout: "val": 0, 2024-03-31T23:37:38.620 INFO:teuthology.orchestra.run.smithi162.stdout: "max": 3758096384, 2024-03-31T23:37:38.620 INFO:teuthology.orchestra.run.smithi162.stdout: "get_started": 0, 2024-03-31T23:37:38.620 INFO:teuthology.orchestra.run.smithi162.stdout: "get": 13619, 2024-03-31T23:37:38.620 INFO:teuthology.orchestra.run.smithi162.stdout: "get_sum": 1494543, 2024-03-31T23:37:38.621 INFO:teuthology.orchestra.run.smithi162.stdout: "get_or_fail_fail": 0, 2024-03-31T23:37:38.621 INFO:teuthology.orchestra.run.smithi162.stdout: "get_or_fail_success": 13619, 2024-03-31T23:37:38.621 INFO:teuthology.orchestra.run.smithi162.stdout: "take": 0, 2024-03-31T23:37:38.621 INFO:teuthology.orchestra.run.smithi162.stdout: "take_sum": 0, 2024-03-31T23:37:38.621 INFO:teuthology.orchestra.run.smithi162.stdout: "put": 2176, 2024-03-31T23:37:38.621 INFO:teuthology.orchestra.run.smithi162.stdout: "put_sum": 1494543, 2024-03-31T23:37:38.621 INFO:teuthology.orchestra.run.smithi162.stdout: "wait": { 2024-03-31T23:37:38.621 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-31T23:37:38.621 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-31T23:37:38.621 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-31T23:37:38.621 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-03-31T23:37:38.621 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-31T23:37:38.621 INFO:teuthology.orchestra.run.smithi162.stdout: "throttle-write_buf_throttle-0x55c95b716500": { 2024-03-31T23:37:38.621 INFO:teuthology.orchestra.run.smithi162.stdout: "val": 0, 2024-03-31T23:37:38.622 INFO:teuthology.orchestra.run.smithi162.stdout: "max": 3758096384, 2024-03-31T23:37:38.622 INFO:teuthology.orchestra.run.smithi162.stdout: "get_started": 0, 2024-03-31T23:37:38.622 INFO:teuthology.orchestra.run.smithi162.stdout: "get": 127845, 2024-03-31T23:37:38.622 INFO:teuthology.orchestra.run.smithi162.stdout: "get_sum": 647775793, 2024-03-31T23:37:38.622 INFO:teuthology.orchestra.run.smithi162.stdout: "get_or_fail_fail": 0, 2024-03-31T23:37:38.622 INFO:teuthology.orchestra.run.smithi162.stdout: "get_or_fail_success": 127845, 2024-03-31T23:37:38.622 INFO:teuthology.orchestra.run.smithi162.stdout: "take": 0, 2024-03-31T23:37:38.622 INFO:teuthology.orchestra.run.smithi162.stdout: "take_sum": 0, 2024-03-31T23:37:38.622 INFO:teuthology.orchestra.run.smithi162.stdout: "put": 54969, 2024-03-31T23:37:38.622 INFO:teuthology.orchestra.run.smithi162.stdout: "put_sum": 647775793, 2024-03-31T23:37:38.622 INFO:teuthology.orchestra.run.smithi162.stdout: "wait": { 2024-03-31T23:37:38.622 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-31T23:37:38.622 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-31T23:37:38.622 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-31T23:37:38.623 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-03-31T23:37:38.623 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-03-31T23:37:38.623 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-03-31T23:37:39.038 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:37:38 smithi046 ceph-mon[32498]: pgmap v823: 129 pgs: 129 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 2.3 KiB/s wr, 5 op/s 2024-03-31T23:37:39.045 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:37:38 smithi162 ceph-mon[36237]: pgmap v823: 129 pgs: 129 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 2.3 KiB/s wr, 5 op/s 2024-03-31T23:37:39.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:37:38 smithi142 ceph-mon[35271]: pgmap v823: 129 pgs: 129 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 2.3 KiB/s wr, 5 op/s 2024-03-31T23:37:39.267 INFO:tasks.check_counter:Daemon mds.c mds.exported=47 2024-03-31T23:37:39.267 INFO:tasks.check_counter:Daemon mds.c mds.imported=89 2024-03-31T23:37:39.267 INFO:tasks.check_counter:Daemon mds.c mds_cache.dir_update=571 2024-03-31T23:37:39.268 INFO:tasks.check_counter:Daemon mds.c mds_cache.dir_update_receipt=2122 2024-03-31T23:37:39.268 INFO:tasks.check_counter:Daemon mds.c mds.root_rsnaps=6 2024-03-31T23:37:39.268 INFO:tasks.check_counter:Daemon mds.c mds_server.req_mksnap_latency.avgcount=0 2024-03-31T23:37:39.268 INFO:tasks.check_counter:Daemon mds.c mds_server.req_rmsnap_latency.avgcount=0 2024-03-31T23:37:39.268 DEBUG:tasks.check_counter:Getting stats from f 2024-03-31T23:37:39.268 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:384f31f3d3aee7f8ba3f60846b8844c049165583 shell --fsid 6a9d8f08-efb2-11ee-b647-cb9ed24678a4 -- ceph daemon mds.f perf dump 2024-03-31T23:37:39.527 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/6a9d8f08-efb2-11ee-b647-cb9ed24678a4/mon.c/config 2024-03-31T23:37:40.861 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:37:40 smithi046 ceph-mon[32498]: pgmap v824: 129 pgs: 129 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 3.8 KiB/s wr, 8 op/s 2024-03-31T23:37:41.042 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:37:40 smithi162 ceph-mon[36237]: pgmap v824: 129 pgs: 129 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 3.8 KiB/s wr, 8 op/s 2024-03-31T23:37:41.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:37:40 smithi142 ceph-mon[35271]: pgmap v824: 129 pgs: 129 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 3.8 KiB/s wr, 8 op/s 2024-03-31T23:37:41.324 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-03-31T23:37:41.325 INFO:teuthology.orchestra.run.smithi162.stdout: "AsyncMessenger::Worker-0": { 2024-03-31T23:37:41.325 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_recv_messages": 1658, 2024-03-31T23:37:41.325 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_send_messages": 1614, 2024-03-31T23:37:41.325 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_recv_bytes": 173958406, 2024-03-31T23:37:41.325 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_send_bytes": 643832, 2024-03-31T23:37:41.325 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_created_connections": 8, 2024-03-31T23:37:41.325 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_active_connections": 3, 2024-03-31T23:37:41.325 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_running_total_time": 0.224849795, 2024-03-31T23:37:41.325 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_running_send_time": 0.039613055, 2024-03-31T23:37:41.325 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_running_recv_time": 0.154931726, 2024-03-31T23:37:41.325 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_running_fast_dispatch_time": 0.031392846, 2024-03-31T23:37:41.325 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_send_messages_queue_lat": { 2024-03-31T23:37:41.325 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 1613, 2024-03-31T23:37:41.325 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.039981263, 2024-03-31T23:37:41.326 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000024786 2024-03-31T23:37:41.326 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-31T23:37:41.326 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_handle_ack_lat": { 2024-03-31T23:37:41.326 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-31T23:37:41.326 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-31T23:37:41.326 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-31T23:37:41.326 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-31T23:37:41.326 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_recv_encrypted_bytes": 1710544, 2024-03-31T23:37:41.326 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_send_encrypted_bytes": 293072 2024-03-31T23:37:41.326 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-31T23:37:41.326 INFO:teuthology.orchestra.run.smithi162.stdout: "AsyncMessenger::Worker-1": { 2024-03-31T23:37:41.326 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_recv_messages": 2102, 2024-03-31T23:37:41.326 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_send_messages": 2451, 2024-03-31T23:37:41.327 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_recv_bytes": 260583203, 2024-03-31T23:37:41.327 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_send_bytes": 1448109, 2024-03-31T23:37:41.327 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_created_connections": 9, 2024-03-31T23:37:41.327 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_active_connections": 7, 2024-03-31T23:37:41.327 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_running_total_time": 0.258978077, 2024-03-31T23:37:41.327 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_running_send_time": 0.051197446, 2024-03-31T23:37:41.327 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_running_recv_time": 0.150228458, 2024-03-31T23:37:41.327 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_running_fast_dispatch_time": 0.044285208, 2024-03-31T23:37:41.327 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_send_messages_queue_lat": { 2024-03-31T23:37:41.327 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 2448, 2024-03-31T23:37:41.327 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.065704653, 2024-03-31T23:37:41.327 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000026840 2024-03-31T23:37:41.327 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-31T23:37:41.327 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_handle_ack_lat": { 2024-03-31T23:37:41.328 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-31T23:37:41.328 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-31T23:37:41.328 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-31T23:37:41.328 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-31T23:37:41.328 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_recv_encrypted_bytes": 320, 2024-03-31T23:37:41.328 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_send_encrypted_bytes": 780304 2024-03-31T23:37:41.328 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-31T23:37:41.328 INFO:teuthology.orchestra.run.smithi162.stdout: "AsyncMessenger::Worker-2": { 2024-03-31T23:37:41.328 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_recv_messages": 3533, 2024-03-31T23:37:41.328 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_send_messages": 3533, 2024-03-31T23:37:41.328 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_recv_bytes": 216801792, 2024-03-31T23:37:41.328 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_send_bytes": 1122728, 2024-03-31T23:37:41.328 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_created_connections": 6, 2024-03-31T23:37:41.328 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_active_connections": 5, 2024-03-31T23:37:41.328 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_running_total_time": 0.340462141, 2024-03-31T23:37:41.329 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_running_send_time": 0.063454382, 2024-03-31T23:37:41.329 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_running_recv_time": 0.179169141, 2024-03-31T23:37:41.329 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_running_fast_dispatch_time": 0.078426419, 2024-03-31T23:37:41.329 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_send_messages_queue_lat": { 2024-03-31T23:37:41.329 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 3532, 2024-03-31T23:37:41.329 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.083952965, 2024-03-31T23:37:41.329 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000023769 2024-03-31T23:37:41.329 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-31T23:37:41.329 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_handle_ack_lat": { 2024-03-31T23:37:41.329 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-31T23:37:41.329 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-31T23:37:41.329 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-31T23:37:41.329 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-31T23:37:41.330 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_recv_encrypted_bytes": 2992, 2024-03-31T23:37:41.330 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_send_encrypted_bytes": 448 2024-03-31T23:37:41.330 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-31T23:37:41.330 INFO:teuthology.orchestra.run.smithi162.stdout: "cct": { 2024-03-31T23:37:41.330 INFO:teuthology.orchestra.run.smithi162.stdout: "total_workers": 1, 2024-03-31T23:37:41.330 INFO:teuthology.orchestra.run.smithi162.stdout: "unhealthy_workers": 0 2024-03-31T23:37:41.330 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-31T23:37:41.330 INFO:teuthology.orchestra.run.smithi162.stdout: "finisher-MDSRank": { 2024-03-31T23:37:41.330 INFO:teuthology.orchestra.run.smithi162.stdout: "queue_len": 0, 2024-03-31T23:37:41.330 INFO:teuthology.orchestra.run.smithi162.stdout: "complete_latency": { 2024-03-31T23:37:41.330 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 11218, 2024-03-31T23:37:41.330 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.223057450, 2024-03-31T23:37:41.330 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000019883 2024-03-31T23:37:41.330 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-03-31T23:37:41.331 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-31T23:37:41.331 INFO:teuthology.orchestra.run.smithi162.stdout: "finisher-PurgeQueue": { 2024-03-31T23:37:41.331 INFO:teuthology.orchestra.run.smithi162.stdout: "queue_len": 0, 2024-03-31T23:37:41.331 INFO:teuthology.orchestra.run.smithi162.stdout: "complete_latency": { 2024-03-31T23:37:41.331 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-31T23:37:41.331 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-31T23:37:41.331 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-31T23:37:41.331 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-03-31T23:37:41.331 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-31T23:37:41.331 INFO:teuthology.orchestra.run.smithi162.stdout: "mds": { 2024-03-31T23:37:41.331 INFO:teuthology.orchestra.run.smithi162.stdout: "request": 0, 2024-03-31T23:37:41.331 INFO:teuthology.orchestra.run.smithi162.stdout: "reply": 0, 2024-03-31T23:37:41.331 INFO:teuthology.orchestra.run.smithi162.stdout: "reply_latency": { 2024-03-31T23:37:41.331 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-31T23:37:41.332 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-31T23:37:41.332 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-31T23:37:41.332 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-31T23:37:41.332 INFO:teuthology.orchestra.run.smithi162.stdout: "slow_reply": 0, 2024-03-31T23:37:41.332 INFO:teuthology.orchestra.run.smithi162.stdout: "forward": 0, 2024-03-31T23:37:41.332 INFO:teuthology.orchestra.run.smithi162.stdout: "dir_fetch_complete": 0, 2024-03-31T23:37:41.332 INFO:teuthology.orchestra.run.smithi162.stdout: "dir_fetch_keys": 0, 2024-03-31T23:37:41.332 INFO:teuthology.orchestra.run.smithi162.stdout: "dir_commit": 0, 2024-03-31T23:37:41.332 INFO:teuthology.orchestra.run.smithi162.stdout: "dir_split": 0, 2024-03-31T23:37:41.332 INFO:teuthology.orchestra.run.smithi162.stdout: "dir_merge": 0, 2024-03-31T23:37:41.332 INFO:teuthology.orchestra.run.smithi162.stdout: "inodes": 18226, 2024-03-31T23:37:41.332 INFO:teuthology.orchestra.run.smithi162.stdout: "inodes_top": 6913, 2024-03-31T23:37:41.332 INFO:teuthology.orchestra.run.smithi162.stdout: "inodes_bottom": 11313, 2024-03-31T23:37:41.332 INFO:teuthology.orchestra.run.smithi162.stdout: "inodes_pin_tail": 0, 2024-03-31T23:37:41.333 INFO:teuthology.orchestra.run.smithi162.stdout: "inodes_pinned": 8353, 2024-03-31T23:37:41.333 INFO:teuthology.orchestra.run.smithi162.stdout: "inodes_expired": 0, 2024-03-31T23:37:41.333 INFO:teuthology.orchestra.run.smithi162.stdout: "inodes_with_caps": 0, 2024-03-31T23:37:41.333 INFO:teuthology.orchestra.run.smithi162.stdout: "caps": 0, 2024-03-31T23:37:41.333 INFO:teuthology.orchestra.run.smithi162.stdout: "subtrees": 1, 2024-03-31T23:37:41.333 INFO:teuthology.orchestra.run.smithi162.stdout: "traverse": 0, 2024-03-31T23:37:41.333 INFO:teuthology.orchestra.run.smithi162.stdout: "traverse_hit": 0, 2024-03-31T23:37:41.333 INFO:teuthology.orchestra.run.smithi162.stdout: "traverse_forward": 0, 2024-03-31T23:37:41.333 INFO:teuthology.orchestra.run.smithi162.stdout: "traverse_discover": 0, 2024-03-31T23:37:41.333 INFO:teuthology.orchestra.run.smithi162.stdout: "traverse_dir_fetch": 0, 2024-03-31T23:37:41.333 INFO:teuthology.orchestra.run.smithi162.stdout: "traverse_remote_ino": 0, 2024-03-31T23:37:41.333 INFO:teuthology.orchestra.run.smithi162.stdout: "traverse_lock": 0, 2024-03-31T23:37:41.333 INFO:teuthology.orchestra.run.smithi162.stdout: "load_cent": 0, 2024-03-31T23:37:41.333 INFO:teuthology.orchestra.run.smithi162.stdout: "q": 0, 2024-03-31T23:37:41.334 INFO:teuthology.orchestra.run.smithi162.stdout: "exported": 0, 2024-03-31T23:37:41.334 INFO:teuthology.orchestra.run.smithi162.stdout: "exported_inodes": 0, 2024-03-31T23:37:41.334 INFO:teuthology.orchestra.run.smithi162.stdout: "imported": 0, 2024-03-31T23:37:41.334 INFO:teuthology.orchestra.run.smithi162.stdout: "imported_inodes": 0, 2024-03-31T23:37:41.334 INFO:teuthology.orchestra.run.smithi162.stdout: "openino_dir_fetch": 0, 2024-03-31T23:37:41.334 INFO:teuthology.orchestra.run.smithi162.stdout: "openino_backtrace_fetch": 0, 2024-03-31T23:37:41.334 INFO:teuthology.orchestra.run.smithi162.stdout: "openino_peer_discover": 0, 2024-03-31T23:37:41.334 INFO:teuthology.orchestra.run.smithi162.stdout: "root_rfiles": 0, 2024-03-31T23:37:41.334 INFO:teuthology.orchestra.run.smithi162.stdout: "root_rbytes": 0, 2024-03-31T23:37:41.334 INFO:teuthology.orchestra.run.smithi162.stdout: "root_rsnaps": 0, 2024-03-31T23:37:41.334 INFO:teuthology.orchestra.run.smithi162.stdout: "scrub_backtrace_fetch": 0, 2024-03-31T23:37:41.334 INFO:teuthology.orchestra.run.smithi162.stdout: "scrub_set_tag": 0, 2024-03-31T23:37:41.334 INFO:teuthology.orchestra.run.smithi162.stdout: "scrub_backtrace_repaired": 0, 2024-03-31T23:37:41.334 INFO:teuthology.orchestra.run.smithi162.stdout: "scrub_inotable_repaired": 0, 2024-03-31T23:37:41.335 INFO:teuthology.orchestra.run.smithi162.stdout: "scrub_dir_inodes": 0, 2024-03-31T23:37:41.335 INFO:teuthology.orchestra.run.smithi162.stdout: "scrub_dir_base_inodes": 0, 2024-03-31T23:37:41.335 INFO:teuthology.orchestra.run.smithi162.stdout: "scrub_dirfrag_rstats": 0, 2024-03-31T23:37:41.335 INFO:teuthology.orchestra.run.smithi162.stdout: "scrub_file_inodes": 0, 2024-03-31T23:37:41.335 INFO:teuthology.orchestra.run.smithi162.stdout: "handle_inode_file_caps": 0, 2024-03-31T23:37:41.335 INFO:teuthology.orchestra.run.smithi162.stdout: "ceph_cap_op_revoke": 0, 2024-03-31T23:37:41.335 INFO:teuthology.orchestra.run.smithi162.stdout: "ceph_cap_op_grant": 0, 2024-03-31T23:37:41.335 INFO:teuthology.orchestra.run.smithi162.stdout: "ceph_cap_op_trunc": 0, 2024-03-31T23:37:41.335 INFO:teuthology.orchestra.run.smithi162.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-31T23:37:41.335 INFO:teuthology.orchestra.run.smithi162.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-31T23:37:41.335 INFO:teuthology.orchestra.run.smithi162.stdout: "handle_client_caps": 0, 2024-03-31T23:37:41.335 INFO:teuthology.orchestra.run.smithi162.stdout: "handle_client_caps_dirty": 0, 2024-03-31T23:37:41.335 INFO:teuthology.orchestra.run.smithi162.stdout: "handle_client_cap_release": 0, 2024-03-31T23:37:41.335 INFO:teuthology.orchestra.run.smithi162.stdout: "process_request_cap_release": 0 2024-03-31T23:37:41.336 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-31T23:37:41.336 INFO:teuthology.orchestra.run.smithi162.stdout: "mds_cache": { 2024-03-31T23:37:41.336 INFO:teuthology.orchestra.run.smithi162.stdout: "dir_update": 0, 2024-03-31T23:37:41.336 INFO:teuthology.orchestra.run.smithi162.stdout: "dir_update_receipt": 0, 2024-03-31T23:37:41.336 INFO:teuthology.orchestra.run.smithi162.stdout: "dir_try_discover": 0, 2024-03-31T23:37:41.336 INFO:teuthology.orchestra.run.smithi162.stdout: "dir_send_discover": 0, 2024-03-31T23:37:41.336 INFO:teuthology.orchestra.run.smithi162.stdout: "dir_handle_discover": 0, 2024-03-31T23:37:41.336 INFO:teuthology.orchestra.run.smithi162.stdout: "num_strays": 0, 2024-03-31T23:37:41.336 INFO:teuthology.orchestra.run.smithi162.stdout: "num_strays_delayed": 0, 2024-03-31T23:37:41.336 INFO:teuthology.orchestra.run.smithi162.stdout: "num_strays_enqueuing": 0, 2024-03-31T23:37:41.336 INFO:teuthology.orchestra.run.smithi162.stdout: "strays_created": 0, 2024-03-31T23:37:41.336 INFO:teuthology.orchestra.run.smithi162.stdout: "strays_enqueued": 0, 2024-03-31T23:37:41.336 INFO:teuthology.orchestra.run.smithi162.stdout: "strays_reintegrated": 0, 2024-03-31T23:37:41.336 INFO:teuthology.orchestra.run.smithi162.stdout: "strays_migrated": 0, 2024-03-31T23:37:41.336 INFO:teuthology.orchestra.run.smithi162.stdout: "num_recovering_processing": 0, 2024-03-31T23:37:41.337 INFO:teuthology.orchestra.run.smithi162.stdout: "num_recovering_enqueued": 0, 2024-03-31T23:37:41.337 INFO:teuthology.orchestra.run.smithi162.stdout: "num_recovering_prioritized": 0, 2024-03-31T23:37:41.337 INFO:teuthology.orchestra.run.smithi162.stdout: "recovery_started": 0, 2024-03-31T23:37:41.337 INFO:teuthology.orchestra.run.smithi162.stdout: "recovery_completed": 0, 2024-03-31T23:37:41.337 INFO:teuthology.orchestra.run.smithi162.stdout: "ireq_quiesce_path": 0, 2024-03-31T23:37:41.337 INFO:teuthology.orchestra.run.smithi162.stdout: "ireq_quiesce_inode": 0, 2024-03-31T23:37:41.337 INFO:teuthology.orchestra.run.smithi162.stdout: "ireq_enqueue_scrub": 0, 2024-03-31T23:37:41.337 INFO:teuthology.orchestra.run.smithi162.stdout: "ireq_exportdir": 0, 2024-03-31T23:37:41.337 INFO:teuthology.orchestra.run.smithi162.stdout: "ireq_flush": 0, 2024-03-31T23:37:41.337 INFO:teuthology.orchestra.run.smithi162.stdout: "ireq_fragmentdir": 0, 2024-03-31T23:37:41.337 INFO:teuthology.orchestra.run.smithi162.stdout: "ireq_fragstats": 0, 2024-03-31T23:37:41.337 INFO:teuthology.orchestra.run.smithi162.stdout: "ireq_inodestats": 0 2024-03-31T23:37:41.337 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-31T23:37:41.337 INFO:teuthology.orchestra.run.smithi162.stdout: "mds_log": { 2024-03-31T23:37:41.338 INFO:teuthology.orchestra.run.smithi162.stdout: "evlrg": 0, 2024-03-31T23:37:41.338 INFO:teuthology.orchestra.run.smithi162.stdout: "evadd": 0, 2024-03-31T23:37:41.338 INFO:teuthology.orchestra.run.smithi162.stdout: "evex": 0, 2024-03-31T23:37:41.338 INFO:teuthology.orchestra.run.smithi162.stdout: "evtrm": 0, 2024-03-31T23:37:41.338 INFO:teuthology.orchestra.run.smithi162.stdout: "ev": 127845, 2024-03-31T23:37:41.338 INFO:teuthology.orchestra.run.smithi162.stdout: "evexg": 0, 2024-03-31T23:37:41.338 INFO:teuthology.orchestra.run.smithi162.stdout: "evexd": 0, 2024-03-31T23:37:41.338 INFO:teuthology.orchestra.run.smithi162.stdout: "segadd": 0, 2024-03-31T23:37:41.338 INFO:teuthology.orchestra.run.smithi162.stdout: "segex": 0, 2024-03-31T23:37:41.338 INFO:teuthology.orchestra.run.smithi162.stdout: "segtrm": 0, 2024-03-31T23:37:41.338 INFO:teuthology.orchestra.run.smithi162.stdout: "seg": 141, 2024-03-31T23:37:41.338 INFO:teuthology.orchestra.run.smithi162.stdout: "segmjr": 12, 2024-03-31T23:37:41.338 INFO:teuthology.orchestra.run.smithi162.stdout: "segexg": 0, 2024-03-31T23:37:41.338 INFO:teuthology.orchestra.run.smithi162.stdout: "segexd": 0, 2024-03-31T23:37:41.338 INFO:teuthology.orchestra.run.smithi162.stdout: "expos": 102516747, 2024-03-31T23:37:41.339 INFO:teuthology.orchestra.run.smithi162.stdout: "wrpos": 651970097, 2024-03-31T23:37:41.339 INFO:teuthology.orchestra.run.smithi162.stdout: "rdpos": 651934338, 2024-03-31T23:37:41.339 INFO:teuthology.orchestra.run.smithi162.stdout: "jlat": { 2024-03-31T23:37:41.339 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-31T23:37:41.339 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-31T23:37:41.339 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-31T23:37:41.339 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-31T23:37:41.339 INFO:teuthology.orchestra.run.smithi162.stdout: "replayed": 127845 2024-03-31T23:37:41.339 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-31T23:37:41.339 INFO:teuthology.orchestra.run.smithi162.stdout: "mds_mem": { 2024-03-31T23:37:41.339 INFO:teuthology.orchestra.run.smithi162.stdout: "ino": 3878, 2024-03-31T23:37:41.339 INFO:teuthology.orchestra.run.smithi162.stdout: "ino+": 18619, 2024-03-31T23:37:41.339 INFO:teuthology.orchestra.run.smithi162.stdout: "ino-": 14741, 2024-03-31T23:37:41.339 INFO:teuthology.orchestra.run.smithi162.stdout: "dir": 131, 2024-03-31T23:37:41.339 INFO:teuthology.orchestra.run.smithi162.stdout: "dir+": 181, 2024-03-31T23:37:41.340 INFO:teuthology.orchestra.run.smithi162.stdout: "dir-": 50, 2024-03-31T23:37:41.340 INFO:teuthology.orchestra.run.smithi162.stdout: "dn": 18226, 2024-03-31T23:37:41.340 INFO:teuthology.orchestra.run.smithi162.stdout: "dn+": 19539, 2024-03-31T23:37:41.340 INFO:teuthology.orchestra.run.smithi162.stdout: "dn-": 1313, 2024-03-31T23:37:41.340 INFO:teuthology.orchestra.run.smithi162.stdout: "cap": 0, 2024-03-31T23:37:41.340 INFO:teuthology.orchestra.run.smithi162.stdout: "cap+": 0, 2024-03-31T23:37:41.340 INFO:teuthology.orchestra.run.smithi162.stdout: "cap-": 0, 2024-03-31T23:37:41.340 INFO:teuthology.orchestra.run.smithi162.stdout: "rss": 80108, 2024-03-31T23:37:41.340 INFO:teuthology.orchestra.run.smithi162.stdout: "heap": 215420 2024-03-31T23:37:41.340 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-31T23:37:41.340 INFO:teuthology.orchestra.run.smithi162.stdout: "mds_server": { 2024-03-31T23:37:41.340 INFO:teuthology.orchestra.run.smithi162.stdout: "dispatch_client_request": 0, 2024-03-31T23:37:41.340 INFO:teuthology.orchestra.run.smithi162.stdout: "dispatch_server_request": 0, 2024-03-31T23:37:41.340 INFO:teuthology.orchestra.run.smithi162.stdout: "handle_client_request": 0, 2024-03-31T23:37:41.341 INFO:teuthology.orchestra.run.smithi162.stdout: "handle_client_session": 0, 2024-03-31T23:37:41.341 INFO:teuthology.orchestra.run.smithi162.stdout: "handle_peer_request": 0, 2024-03-31T23:37:41.341 INFO:teuthology.orchestra.run.smithi162.stdout: "req_create_latency": { 2024-03-31T23:37:41.341 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-31T23:37:41.341 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-31T23:37:41.341 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-31T23:37:41.341 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-31T23:37:41.341 INFO:teuthology.orchestra.run.smithi162.stdout: "req_getattr_latency": { 2024-03-31T23:37:41.341 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-31T23:37:41.341 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-31T23:37:41.341 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-31T23:37:41.341 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-31T23:37:41.341 INFO:teuthology.orchestra.run.smithi162.stdout: "req_getfilelock_latency": { 2024-03-31T23:37:41.341 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-31T23:37:41.341 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-31T23:37:41.342 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-31T23:37:41.342 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-31T23:37:41.342 INFO:teuthology.orchestra.run.smithi162.stdout: "req_link_latency": { 2024-03-31T23:37:41.342 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-31T23:37:41.342 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-31T23:37:41.342 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-31T23:37:41.342 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-31T23:37:41.342 INFO:teuthology.orchestra.run.smithi162.stdout: "req_lookup_latency": { 2024-03-31T23:37:41.342 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-31T23:37:41.342 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-31T23:37:41.342 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-31T23:37:41.342 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-31T23:37:41.342 INFO:teuthology.orchestra.run.smithi162.stdout: "req_lookuphash_latency": { 2024-03-31T23:37:41.342 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-31T23:37:41.343 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-31T23:37:41.343 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-31T23:37:41.343 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-31T23:37:41.343 INFO:teuthology.orchestra.run.smithi162.stdout: "req_lookupino_latency": { 2024-03-31T23:37:41.343 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-31T23:37:41.343 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-31T23:37:41.343 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-31T23:37:41.343 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-31T23:37:41.343 INFO:teuthology.orchestra.run.smithi162.stdout: "req_lookupname_latency": { 2024-03-31T23:37:41.343 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-31T23:37:41.343 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-31T23:37:41.343 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-31T23:37:41.343 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-31T23:37:41.344 INFO:teuthology.orchestra.run.smithi162.stdout: "req_lookupparent_latency": { 2024-03-31T23:37:41.344 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-31T23:37:41.344 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-31T23:37:41.344 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-31T23:37:41.344 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-31T23:37:41.344 INFO:teuthology.orchestra.run.smithi162.stdout: "req_lookupsnap_latency": { 2024-03-31T23:37:41.344 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-31T23:37:41.344 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-31T23:37:41.344 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-31T23:37:41.344 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-31T23:37:41.344 INFO:teuthology.orchestra.run.smithi162.stdout: "req_lssnap_latency": { 2024-03-31T23:37:41.344 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-31T23:37:41.344 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-31T23:37:41.344 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-31T23:37:41.345 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-31T23:37:41.345 INFO:teuthology.orchestra.run.smithi162.stdout: "req_mkdir_latency": { 2024-03-31T23:37:41.345 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-31T23:37:41.345 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-31T23:37:41.345 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-31T23:37:41.345 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-31T23:37:41.346 INFO:teuthology.orchestra.run.smithi162.stdout: "req_mknod_latency": { 2024-03-31T23:37:41.346 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-31T23:37:41.346 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-31T23:37:41.346 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-31T23:37:41.346 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-31T23:37:41.346 INFO:teuthology.orchestra.run.smithi162.stdout: "req_mksnap_latency": { 2024-03-31T23:37:41.346 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-31T23:37:41.346 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-31T23:37:41.346 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-31T23:37:41.346 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-31T23:37:41.346 INFO:teuthology.orchestra.run.smithi162.stdout: "req_open_latency": { 2024-03-31T23:37:41.346 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-31T23:37:41.346 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-31T23:37:41.346 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-31T23:37:41.347 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-31T23:37:41.347 INFO:teuthology.orchestra.run.smithi162.stdout: "req_readdir_latency": { 2024-03-31T23:37:41.347 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-31T23:37:41.347 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-31T23:37:41.347 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-31T23:37:41.347 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-31T23:37:41.347 INFO:teuthology.orchestra.run.smithi162.stdout: "req_rename_latency": { 2024-03-31T23:37:41.347 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-31T23:37:41.347 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-31T23:37:41.347 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-31T23:37:41.347 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-31T23:37:41.347 INFO:teuthology.orchestra.run.smithi162.stdout: "req_renamesnap_latency": { 2024-03-31T23:37:41.347 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-31T23:37:41.347 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-31T23:37:41.348 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-31T23:37:41.348 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-31T23:37:41.348 INFO:teuthology.orchestra.run.smithi162.stdout: "req_snapdiff_latency": { 2024-03-31T23:37:41.348 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-31T23:37:41.348 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-31T23:37:41.348 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-31T23:37:41.348 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-31T23:37:41.348 INFO:teuthology.orchestra.run.smithi162.stdout: "req_rmdir_latency": { 2024-03-31T23:37:41.348 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-31T23:37:41.348 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-31T23:37:41.348 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-31T23:37:41.348 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-31T23:37:41.348 INFO:teuthology.orchestra.run.smithi162.stdout: "req_rmsnap_latency": { 2024-03-31T23:37:41.348 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-31T23:37:41.349 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-31T23:37:41.349 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-31T23:37:41.349 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-31T23:37:41.349 INFO:teuthology.orchestra.run.smithi162.stdout: "req_rmxattr_latency": { 2024-03-31T23:37:41.349 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-31T23:37:41.349 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-31T23:37:41.349 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-31T23:37:41.349 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-31T23:37:41.349 INFO:teuthology.orchestra.run.smithi162.stdout: "req_setattr_latency": { 2024-03-31T23:37:41.349 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-31T23:37:41.349 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-31T23:37:41.349 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-31T23:37:41.349 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-31T23:37:41.350 INFO:teuthology.orchestra.run.smithi162.stdout: "req_setdirlayout_latency": { 2024-03-31T23:37:41.350 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-31T23:37:41.350 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-31T23:37:41.350 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-31T23:37:41.350 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-31T23:37:41.350 INFO:teuthology.orchestra.run.smithi162.stdout: "req_setfilelock_latency": { 2024-03-31T23:37:41.350 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-31T23:37:41.350 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-31T23:37:41.350 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-31T23:37:41.350 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-31T23:37:41.350 INFO:teuthology.orchestra.run.smithi162.stdout: "req_setlayout_latency": { 2024-03-31T23:37:41.350 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-31T23:37:41.350 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-31T23:37:41.350 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-31T23:37:41.351 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-31T23:37:41.351 INFO:teuthology.orchestra.run.smithi162.stdout: "req_setxattr_latency": { 2024-03-31T23:37:41.351 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-31T23:37:41.351 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-31T23:37:41.351 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-31T23:37:41.351 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-31T23:37:41.351 INFO:teuthology.orchestra.run.smithi162.stdout: "req_symlink_latency": { 2024-03-31T23:37:41.351 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-31T23:37:41.351 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-31T23:37:41.351 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-31T23:37:41.351 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-31T23:37:41.351 INFO:teuthology.orchestra.run.smithi162.stdout: "req_unlink_latency": { 2024-03-31T23:37:41.351 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-31T23:37:41.351 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-31T23:37:41.352 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-31T23:37:41.352 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-31T23:37:41.352 INFO:teuthology.orchestra.run.smithi162.stdout: "cap_revoke_eviction": 0, 2024-03-31T23:37:41.352 INFO:teuthology.orchestra.run.smithi162.stdout: "cap_acquisition_throttle": 0, 2024-03-31T23:37:41.352 INFO:teuthology.orchestra.run.smithi162.stdout: "req_getvxattr_latency": { 2024-03-31T23:37:41.352 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-31T23:37:41.352 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-31T23:37:41.352 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-31T23:37:41.352 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-03-31T23:37:41.352 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-31T23:37:41.352 INFO:teuthology.orchestra.run.smithi162.stdout: "mds_sessions": { 2024-03-31T23:37:41.352 INFO:teuthology.orchestra.run.smithi162.stdout: "session_count": 2, 2024-03-31T23:37:41.352 INFO:teuthology.orchestra.run.smithi162.stdout: "session_add": 4, 2024-03-31T23:37:41.353 INFO:teuthology.orchestra.run.smithi162.stdout: "session_remove": 2, 2024-03-31T23:37:41.353 INFO:teuthology.orchestra.run.smithi162.stdout: "sessions_open": 3, 2024-03-31T23:37:41.353 INFO:teuthology.orchestra.run.smithi162.stdout: "sessions_stale": 0, 2024-03-31T23:37:41.353 INFO:teuthology.orchestra.run.smithi162.stdout: "total_load": 0, 2024-03-31T23:37:41.353 INFO:teuthology.orchestra.run.smithi162.stdout: "average_load": 0, 2024-03-31T23:37:41.353 INFO:teuthology.orchestra.run.smithi162.stdout: "avg_session_uptime": 3219, 2024-03-31T23:37:41.353 INFO:teuthology.orchestra.run.smithi162.stdout: "mdthresh_evicted": 0 2024-03-31T23:37:41.353 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-31T23:37:41.353 INFO:teuthology.orchestra.run.smithi162.stdout: "mempool": { 2024-03-31T23:37:41.353 INFO:teuthology.orchestra.run.smithi162.stdout: "bloom_filter_bytes": 0, 2024-03-31T23:37:41.353 INFO:teuthology.orchestra.run.smithi162.stdout: "bloom_filter_items": 0, 2024-03-31T23:37:41.353 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_alloc_bytes": 0, 2024-03-31T23:37:41.353 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_alloc_items": 0, 2024-03-31T23:37:41.353 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_cache_data_bytes": 0, 2024-03-31T23:37:41.354 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_cache_data_items": 0, 2024-03-31T23:37:41.354 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-31T23:37:41.354 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_cache_onode_items": 0, 2024-03-31T23:37:41.354 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-31T23:37:41.354 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_cache_meta_items": 0, 2024-03-31T23:37:41.354 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_cache_other_bytes": 0, 2024-03-31T23:37:41.354 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_cache_other_items": 0, 2024-03-31T23:37:41.354 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-31T23:37:41.354 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_cache_buffer_items": 0, 2024-03-31T23:37:41.354 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_extent_bytes": 0, 2024-03-31T23:37:41.354 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_extent_items": 0, 2024-03-31T23:37:41.354 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_blob_bytes": 0, 2024-03-31T23:37:41.354 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_blob_items": 0, 2024-03-31T23:37:41.354 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-31T23:37:41.355 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_shared_blob_items": 0, 2024-03-31T23:37:41.355 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-31T23:37:41.355 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_inline_bl_items": 0, 2024-03-31T23:37:41.355 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_fsck_bytes": 0, 2024-03-31T23:37:41.355 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_fsck_items": 0, 2024-03-31T23:37:41.355 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_txc_bytes": 0, 2024-03-31T23:37:41.355 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_txc_items": 0, 2024-03-31T23:37:41.355 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-31T23:37:41.355 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_writing_deferred_items": 0, 2024-03-31T23:37:41.355 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_writing_bytes": 0, 2024-03-31T23:37:41.355 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_writing_items": 0, 2024-03-31T23:37:41.355 INFO:teuthology.orchestra.run.smithi162.stdout: "bluefs_bytes": 0, 2024-03-31T23:37:41.355 INFO:teuthology.orchestra.run.smithi162.stdout: "bluefs_items": 0, 2024-03-31T23:37:41.355 INFO:teuthology.orchestra.run.smithi162.stdout: "bluefs_file_reader_bytes": 0, 2024-03-31T23:37:41.355 INFO:teuthology.orchestra.run.smithi162.stdout: "bluefs_file_reader_items": 0, 2024-03-31T23:37:41.355 INFO:teuthology.orchestra.run.smithi162.stdout: "bluefs_file_writer_bytes": 0, 2024-03-31T23:37:41.356 INFO:teuthology.orchestra.run.smithi162.stdout: "bluefs_file_writer_items": 0, 2024-03-31T23:37:41.356 INFO:teuthology.orchestra.run.smithi162.stdout: "buffer_anon_bytes": 243401, 2024-03-31T23:37:41.356 INFO:teuthology.orchestra.run.smithi162.stdout: "buffer_anon_items": 90, 2024-03-31T23:37:41.356 INFO:teuthology.orchestra.run.smithi162.stdout: "buffer_meta_bytes": 0, 2024-03-31T23:37:41.356 INFO:teuthology.orchestra.run.smithi162.stdout: "buffer_meta_items": 0, 2024-03-31T23:37:41.356 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_bytes": 0, 2024-03-31T23:37:41.356 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_items": 0, 2024-03-31T23:37:41.356 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_mapbl_bytes": 0, 2024-03-31T23:37:41.356 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_mapbl_items": 0, 2024-03-31T23:37:41.356 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_pglog_bytes": 0, 2024-03-31T23:37:41.356 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_pglog_items": 0, 2024-03-31T23:37:41.356 INFO:teuthology.orchestra.run.smithi162.stdout: "osdmap_bytes": 9704, 2024-03-31T23:37:41.356 INFO:teuthology.orchestra.run.smithi162.stdout: "osdmap_items": 186, 2024-03-31T23:37:41.356 INFO:teuthology.orchestra.run.smithi162.stdout: "osdmap_mapping_bytes": 0, 2024-03-31T23:37:41.356 INFO:teuthology.orchestra.run.smithi162.stdout: "osdmap_mapping_items": 0, 2024-03-31T23:37:41.357 INFO:teuthology.orchestra.run.smithi162.stdout: "pgmap_bytes": 0, 2024-03-31T23:37:41.357 INFO:teuthology.orchestra.run.smithi162.stdout: "pgmap_items": 0, 2024-03-31T23:37:41.357 INFO:teuthology.orchestra.run.smithi162.stdout: "mds_co_bytes": 21663407, 2024-03-31T23:37:41.357 INFO:teuthology.orchestra.run.smithi162.stdout: "mds_co_items": 89776, 2024-03-31T23:37:41.357 INFO:teuthology.orchestra.run.smithi162.stdout: "unittest_1_bytes": 0, 2024-03-31T23:37:41.357 INFO:teuthology.orchestra.run.smithi162.stdout: "unittest_1_items": 0, 2024-03-31T23:37:41.357 INFO:teuthology.orchestra.run.smithi162.stdout: "unittest_2_bytes": 0, 2024-03-31T23:37:41.357 INFO:teuthology.orchestra.run.smithi162.stdout: "unittest_2_items": 0 2024-03-31T23:37:41.357 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-31T23:37:41.357 INFO:teuthology.orchestra.run.smithi162.stdout: "objecter": { 2024-03-31T23:37:41.357 INFO:teuthology.orchestra.run.smithi162.stdout: "op_active": 0, 2024-03-31T23:37:41.357 INFO:teuthology.orchestra.run.smithi162.stdout: "op_laggy": 0, 2024-03-31T23:37:41.357 INFO:teuthology.orchestra.run.smithi162.stdout: "op_send": 6732, 2024-03-31T23:37:41.357 INFO:teuthology.orchestra.run.smithi162.stdout: "op_send_bytes": 21, 2024-03-31T23:37:41.358 INFO:teuthology.orchestra.run.smithi162.stdout: "op_resend": 3, 2024-03-31T23:37:41.358 INFO:teuthology.orchestra.run.smithi162.stdout: "op_reply": 6732, 2024-03-31T23:37:41.358 INFO:teuthology.orchestra.run.smithi162.stdout: "op_latency": { 2024-03-31T23:37:41.358 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 6732, 2024-03-31T23:37:41.358 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 5.994885058, 2024-03-31T23:37:41.358 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000890505 2024-03-31T23:37:41.358 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-31T23:37:41.358 INFO:teuthology.orchestra.run.smithi162.stdout: "op_inflight": 0, 2024-03-31T23:37:41.358 INFO:teuthology.orchestra.run.smithi162.stdout: "oplen_avg": { 2024-03-31T23:37:41.358 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 6732, 2024-03-31T23:37:41.358 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 6733 2024-03-31T23:37:41.358 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-31T23:37:41.358 INFO:teuthology.orchestra.run.smithi162.stdout: "op": 6732, 2024-03-31T23:37:41.358 INFO:teuthology.orchestra.run.smithi162.stdout: "op_r": 6732, 2024-03-31T23:37:41.359 INFO:teuthology.orchestra.run.smithi162.stdout: "op_w": 0, 2024-03-31T23:37:41.359 INFO:teuthology.orchestra.run.smithi162.stdout: "op_rmw": 0, 2024-03-31T23:37:41.359 INFO:teuthology.orchestra.run.smithi162.stdout: "op_pg": 0, 2024-03-31T23:37:41.359 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_stat": 3658, 2024-03-31T23:37:41.359 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_create": 0, 2024-03-31T23:37:41.359 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_read": 3072, 2024-03-31T23:37:41.359 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_write": 0, 2024-03-31T23:37:41.359 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_writefull": 0, 2024-03-31T23:37:41.359 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_writesame": 0, 2024-03-31T23:37:41.359 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_append": 0, 2024-03-31T23:37:41.359 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_zero": 0, 2024-03-31T23:37:41.359 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_truncate": 0, 2024-03-31T23:37:41.359 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_delete": 0, 2024-03-31T23:37:41.359 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_mapext": 0, 2024-03-31T23:37:41.360 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_sparse_read": 0, 2024-03-31T23:37:41.360 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_clonerange": 0, 2024-03-31T23:37:41.360 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_getxattr": 1, 2024-03-31T23:37:41.360 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_setxattr": 0, 2024-03-31T23:37:41.360 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_cmpxattr": 0, 2024-03-31T23:37:41.360 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_rmxattr": 0, 2024-03-31T23:37:41.360 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_resetxattrs": 0, 2024-03-31T23:37:41.360 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_call": 0, 2024-03-31T23:37:41.360 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_watch": 0, 2024-03-31T23:37:41.360 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_notify": 0, 2024-03-31T23:37:41.360 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_src_cmpxattr": 0, 2024-03-31T23:37:41.360 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_pgls": 0, 2024-03-31T23:37:41.360 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_pgls_filter": 0, 2024-03-31T23:37:41.360 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_other": 0, 2024-03-31T23:37:41.361 INFO:teuthology.orchestra.run.smithi162.stdout: "linger_active": 0, 2024-03-31T23:37:41.361 INFO:teuthology.orchestra.run.smithi162.stdout: "linger_send": 0, 2024-03-31T23:37:41.361 INFO:teuthology.orchestra.run.smithi162.stdout: "linger_resend": 0, 2024-03-31T23:37:41.361 INFO:teuthology.orchestra.run.smithi162.stdout: "linger_ping": 0, 2024-03-31T23:37:41.361 INFO:teuthology.orchestra.run.smithi162.stdout: "poolop_active": 0, 2024-03-31T23:37:41.361 INFO:teuthology.orchestra.run.smithi162.stdout: "poolop_send": 0, 2024-03-31T23:37:41.361 INFO:teuthology.orchestra.run.smithi162.stdout: "poolop_resend": 0, 2024-03-31T23:37:41.361 INFO:teuthology.orchestra.run.smithi162.stdout: "poolstat_active": 0, 2024-03-31T23:37:41.361 INFO:teuthology.orchestra.run.smithi162.stdout: "poolstat_send": 0, 2024-03-31T23:37:41.361 INFO:teuthology.orchestra.run.smithi162.stdout: "poolstat_resend": 0, 2024-03-31T23:37:41.361 INFO:teuthology.orchestra.run.smithi162.stdout: "statfs_active": 0, 2024-03-31T23:37:41.361 INFO:teuthology.orchestra.run.smithi162.stdout: "statfs_send": 0, 2024-03-31T23:37:41.361 INFO:teuthology.orchestra.run.smithi162.stdout: "statfs_resend": 0, 2024-03-31T23:37:41.361 INFO:teuthology.orchestra.run.smithi162.stdout: "command_active": 0, 2024-03-31T23:37:41.361 INFO:teuthology.orchestra.run.smithi162.stdout: "command_send": 0, 2024-03-31T23:37:41.361 INFO:teuthology.orchestra.run.smithi162.stdout: "command_resend": 0, 2024-03-31T23:37:41.362 INFO:teuthology.orchestra.run.smithi162.stdout: "map_epoch": 139, 2024-03-31T23:37:41.362 INFO:teuthology.orchestra.run.smithi162.stdout: "map_full": 0, 2024-03-31T23:37:41.362 INFO:teuthology.orchestra.run.smithi162.stdout: "map_inc": 48, 2024-03-31T23:37:41.362 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_sessions": 11, 2024-03-31T23:37:41.362 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_session_open": 12, 2024-03-31T23:37:41.362 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_session_close": 1, 2024-03-31T23:37:41.362 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_laggy": 0, 2024-03-31T23:37:41.362 INFO:teuthology.orchestra.run.smithi162.stdout: "omap_wr": 0, 2024-03-31T23:37:41.362 INFO:teuthology.orchestra.run.smithi162.stdout: "omap_rd": 2, 2024-03-31T23:37:41.362 INFO:teuthology.orchestra.run.smithi162.stdout: "omap_del": 0 2024-03-31T23:37:41.362 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-31T23:37:41.362 INFO:teuthology.orchestra.run.smithi162.stdout: "oft": { 2024-03-31T23:37:41.362 INFO:teuthology.orchestra.run.smithi162.stdout: "omap_total_objs": 0, 2024-03-31T23:37:41.362 INFO:teuthology.orchestra.run.smithi162.stdout: "omap_total_kv_pairs": 0, 2024-03-31T23:37:41.363 INFO:teuthology.orchestra.run.smithi162.stdout: "omap_total_updates": 0, 2024-03-31T23:37:41.363 INFO:teuthology.orchestra.run.smithi162.stdout: "omap_total_removes": 0 2024-03-31T23:37:41.363 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-31T23:37:41.363 INFO:teuthology.orchestra.run.smithi162.stdout: "osd-slow-ops": { 2024-03-31T23:37:41.363 INFO:teuthology.orchestra.run.smithi162.stdout: "slow_ops_count": 0 2024-03-31T23:37:41.363 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-31T23:37:41.363 INFO:teuthology.orchestra.run.smithi162.stdout: "purge_queue": { 2024-03-31T23:37:41.363 INFO:teuthology.orchestra.run.smithi162.stdout: "pq_executing_ops": 0, 2024-03-31T23:37:41.363 INFO:teuthology.orchestra.run.smithi162.stdout: "pq_executing_ops_high_water": 0, 2024-03-31T23:37:41.363 INFO:teuthology.orchestra.run.smithi162.stdout: "pq_executing": 0, 2024-03-31T23:37:41.363 INFO:teuthology.orchestra.run.smithi162.stdout: "pq_executing_high_water": 0, 2024-03-31T23:37:41.363 INFO:teuthology.orchestra.run.smithi162.stdout: "pq_executed_ops": 0, 2024-03-31T23:37:41.363 INFO:teuthology.orchestra.run.smithi162.stdout: "pq_executed": 0, 2024-03-31T23:37:41.363 INFO:teuthology.orchestra.run.smithi162.stdout: "pq_item_in_journal": 0 2024-03-31T23:37:41.364 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-31T23:37:41.364 INFO:teuthology.orchestra.run.smithi162.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-31T23:37:41.364 INFO:teuthology.orchestra.run.smithi162.stdout: "val": 0, 2024-03-31T23:37:41.364 INFO:teuthology.orchestra.run.smithi162.stdout: "max": 104857600, 2024-03-31T23:37:41.364 INFO:teuthology.orchestra.run.smithi162.stdout: "get_started": 0, 2024-03-31T23:37:41.364 INFO:teuthology.orchestra.run.smithi162.stdout: "get": 7289, 2024-03-31T23:37:41.364 INFO:teuthology.orchestra.run.smithi162.stdout: "get_sum": 650656159, 2024-03-31T23:37:41.364 INFO:teuthology.orchestra.run.smithi162.stdout: "get_or_fail_fail": 0, 2024-03-31T23:37:41.364 INFO:teuthology.orchestra.run.smithi162.stdout: "get_or_fail_success": 7289, 2024-03-31T23:37:41.364 INFO:teuthology.orchestra.run.smithi162.stdout: "take": 0, 2024-03-31T23:37:41.364 INFO:teuthology.orchestra.run.smithi162.stdout: "take_sum": 0, 2024-03-31T23:37:41.364 INFO:teuthology.orchestra.run.smithi162.stdout: "put": 7289, 2024-03-31T23:37:41.364 INFO:teuthology.orchestra.run.smithi162.stdout: "put_sum": 650656159, 2024-03-31T23:37:41.364 INFO:teuthology.orchestra.run.smithi162.stdout: "wait": { 2024-03-31T23:37:41.364 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-31T23:37:41.365 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-31T23:37:41.365 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-31T23:37:41.365 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-03-31T23:37:41.365 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-31T23:37:41.365 INFO:teuthology.orchestra.run.smithi162.stdout: "throttle-objecter_bytes": { 2024-03-31T23:37:41.365 INFO:teuthology.orchestra.run.smithi162.stdout: "val": 0, 2024-03-31T23:37:41.365 INFO:teuthology.orchestra.run.smithi162.stdout: "max": 104857600, 2024-03-31T23:37:41.365 INFO:teuthology.orchestra.run.smithi162.stdout: "get_started": 0, 2024-03-31T23:37:41.365 INFO:teuthology.orchestra.run.smithi162.stdout: "get": 0, 2024-03-31T23:37:41.365 INFO:teuthology.orchestra.run.smithi162.stdout: "get_sum": 0, 2024-03-31T23:37:41.365 INFO:teuthology.orchestra.run.smithi162.stdout: "get_or_fail_fail": 0, 2024-03-31T23:37:41.365 INFO:teuthology.orchestra.run.smithi162.stdout: "get_or_fail_success": 0, 2024-03-31T23:37:41.365 INFO:teuthology.orchestra.run.smithi162.stdout: "take": 6732, 2024-03-31T23:37:41.366 INFO:teuthology.orchestra.run.smithi162.stdout: "take_sum": 647778363, 2024-03-31T23:37:41.366 INFO:teuthology.orchestra.run.smithi162.stdout: "put": 6732, 2024-03-31T23:37:41.366 INFO:teuthology.orchestra.run.smithi162.stdout: "put_sum": 647778363, 2024-03-31T23:37:41.366 INFO:teuthology.orchestra.run.smithi162.stdout: "wait": { 2024-03-31T23:37:41.366 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-31T23:37:41.366 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-31T23:37:41.366 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-31T23:37:41.366 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-03-31T23:37:41.366 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-31T23:37:41.366 INFO:teuthology.orchestra.run.smithi162.stdout: "throttle-objecter_ops": { 2024-03-31T23:37:41.366 INFO:teuthology.orchestra.run.smithi162.stdout: "val": 0, 2024-03-31T23:37:41.366 INFO:teuthology.orchestra.run.smithi162.stdout: "max": 1024, 2024-03-31T23:37:41.366 INFO:teuthology.orchestra.run.smithi162.stdout: "get_started": 0, 2024-03-31T23:37:41.367 INFO:teuthology.orchestra.run.smithi162.stdout: "get": 0, 2024-03-31T23:37:41.367 INFO:teuthology.orchestra.run.smithi162.stdout: "get_sum": 0, 2024-03-31T23:37:41.367 INFO:teuthology.orchestra.run.smithi162.stdout: "get_or_fail_fail": 0, 2024-03-31T23:37:41.367 INFO:teuthology.orchestra.run.smithi162.stdout: "get_or_fail_success": 0, 2024-03-31T23:37:41.367 INFO:teuthology.orchestra.run.smithi162.stdout: "take": 6732, 2024-03-31T23:37:41.367 INFO:teuthology.orchestra.run.smithi162.stdout: "take_sum": 6732, 2024-03-31T23:37:41.367 INFO:teuthology.orchestra.run.smithi162.stdout: "put": 6732, 2024-03-31T23:37:41.367 INFO:teuthology.orchestra.run.smithi162.stdout: "put_sum": 6732, 2024-03-31T23:37:41.367 INFO:teuthology.orchestra.run.smithi162.stdout: "wait": { 2024-03-31T23:37:41.367 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-31T23:37:41.367 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-31T23:37:41.367 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-31T23:37:41.367 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-03-31T23:37:41.367 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-31T23:37:41.368 INFO:teuthology.orchestra.run.smithi162.stdout: "throttle-write_buf_throttle": { 2024-03-31T23:37:41.368 INFO:teuthology.orchestra.run.smithi162.stdout: "val": 0, 2024-03-31T23:37:41.368 INFO:teuthology.orchestra.run.smithi162.stdout: "max": 3758096384, 2024-03-31T23:37:41.368 INFO:teuthology.orchestra.run.smithi162.stdout: "get_started": 0, 2024-03-31T23:37:41.368 INFO:teuthology.orchestra.run.smithi162.stdout: "get": 0, 2024-03-31T23:37:41.368 INFO:teuthology.orchestra.run.smithi162.stdout: "get_sum": 0, 2024-03-31T23:37:41.368 INFO:teuthology.orchestra.run.smithi162.stdout: "get_or_fail_fail": 0, 2024-03-31T23:37:41.368 INFO:teuthology.orchestra.run.smithi162.stdout: "get_or_fail_success": 0, 2024-03-31T23:37:41.368 INFO:teuthology.orchestra.run.smithi162.stdout: "take": 0, 2024-03-31T23:37:41.368 INFO:teuthology.orchestra.run.smithi162.stdout: "take_sum": 0, 2024-03-31T23:37:41.368 INFO:teuthology.orchestra.run.smithi162.stdout: "put": 0, 2024-03-31T23:37:41.368 INFO:teuthology.orchestra.run.smithi162.stdout: "put_sum": 0, 2024-03-31T23:37:41.369 INFO:teuthology.orchestra.run.smithi162.stdout: "wait": { 2024-03-31T23:37:41.369 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-31T23:37:41.369 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-31T23:37:41.369 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-31T23:37:41.369 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-03-31T23:37:41.369 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-31T23:37:41.369 INFO:teuthology.orchestra.run.smithi162.stdout: "throttle-write_buf_throttle-0x55aaa88306e0": { 2024-03-31T23:37:41.369 INFO:teuthology.orchestra.run.smithi162.stdout: "val": 0, 2024-03-31T23:37:41.369 INFO:teuthology.orchestra.run.smithi162.stdout: "max": 3758096384, 2024-03-31T23:37:41.369 INFO:teuthology.orchestra.run.smithi162.stdout: "get_started": 0, 2024-03-31T23:37:41.369 INFO:teuthology.orchestra.run.smithi162.stdout: "get": 0, 2024-03-31T23:37:41.369 INFO:teuthology.orchestra.run.smithi162.stdout: "get_sum": 0, 2024-03-31T23:37:41.369 INFO:teuthology.orchestra.run.smithi162.stdout: "get_or_fail_fail": 0, 2024-03-31T23:37:41.370 INFO:teuthology.orchestra.run.smithi162.stdout: "get_or_fail_success": 0, 2024-03-31T23:37:41.370 INFO:teuthology.orchestra.run.smithi162.stdout: "take": 0, 2024-03-31T23:37:41.370 INFO:teuthology.orchestra.run.smithi162.stdout: "take_sum": 0, 2024-03-31T23:37:41.370 INFO:teuthology.orchestra.run.smithi162.stdout: "put": 0, 2024-03-31T23:37:41.370 INFO:teuthology.orchestra.run.smithi162.stdout: "put_sum": 0, 2024-03-31T23:37:41.370 INFO:teuthology.orchestra.run.smithi162.stdout: "wait": { 2024-03-31T23:37:41.370 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-31T23:37:41.370 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-31T23:37:41.370 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-31T23:37:41.370 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-03-31T23:37:41.370 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-03-31T23:37:41.370 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-03-31T23:37:42.295 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:37:42 smithi162 ceph-mon[36237]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' 2024-03-31T23:37:42.296 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:37:42 smithi162 ceph-mon[36237]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' 2024-03-31T23:37:42.296 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:37:42 smithi162 ceph-mon[36237]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:37:42.296 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:37:42 smithi162 ceph-mon[36237]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-31T23:37:42.296 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:37:42 smithi162 ceph-mon[36237]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' 2024-03-31T23:37:42.296 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:37:42 smithi162 ceph-mon[36237]: pgmap v825: 129 pgs: 129 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 1.5 KiB/s wr, 5 op/s 2024-03-31T23:37:42.361 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:37:42 smithi046 ceph-mon[32498]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' 2024-03-31T23:37:42.362 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:37:42 smithi046 ceph-mon[32498]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' 2024-03-31T23:37:42.362 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:37:42 smithi046 ceph-mon[32498]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:37:42.362 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:37:42 smithi046 ceph-mon[32498]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-31T23:37:42.362 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:37:42 smithi046 ceph-mon[32498]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' 2024-03-31T23:37:42.362 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:37:42 smithi046 ceph-mon[32498]: pgmap v825: 129 pgs: 129 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 1.5 KiB/s wr, 5 op/s 2024-03-31T23:37:42.412 INFO:tasks.check_counter:Daemon mds.f mds.exported=0 2024-03-31T23:37:42.412 INFO:tasks.check_counter:Daemon mds.f mds.imported=0 2024-03-31T23:37:42.412 INFO:tasks.check_counter:Daemon mds.f mds_cache.dir_update=0 2024-03-31T23:37:42.412 INFO:tasks.check_counter:Daemon mds.f mds_cache.dir_update_receipt=0 2024-03-31T23:37:42.412 INFO:tasks.check_counter:Daemon mds.f mds.root_rsnaps=0 2024-03-31T23:37:42.412 INFO:tasks.check_counter:Daemon mds.f mds_server.req_mksnap_latency.avgcount=0 2024-03-31T23:37:42.412 INFO:tasks.check_counter:Daemon mds.f mds_server.req_rmsnap_latency.avgcount=0 2024-03-31T23:37:42.412 DEBUG:tasks.check_counter:Getting stats from i 2024-03-31T23:37:42.412 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:384f31f3d3aee7f8ba3f60846b8844c049165583 shell --fsid 6a9d8f08-efb2-11ee-b647-cb9ed24678a4 -- ceph daemon mds.i perf dump 2024-03-31T23:37:42.443 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:37:42 smithi142 ceph-mon[35271]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' 2024-03-31T23:37:42.443 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:37:42 smithi142 ceph-mon[35271]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' 2024-03-31T23:37:42.443 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:37:42 smithi142 ceph-mon[35271]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-31T23:37:42.443 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:37:42 smithi142 ceph-mon[35271]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-31T23:37:42.443 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:37:42 smithi142 ceph-mon[35271]: from='mgr.15654 172.21.15.46:0/3611074372' entity='mgr.x' 2024-03-31T23:37:42.443 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:37:42 smithi142 ceph-mon[35271]: pgmap v825: 129 pgs: 129 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 1.5 KiB/s wr, 5 op/s 2024-03-31T23:37:42.671 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/6a9d8f08-efb2-11ee-b647-cb9ed24678a4/mon.c/config 2024-03-31T23:37:44.398 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:37:44 smithi162 ceph-mon[36237]: pgmap v826: 129 pgs: 129 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 1.5 KiB/s wr, 5 op/s 2024-03-31T23:37:44.398 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-03-31T23:37:44.398 INFO:teuthology.orchestra.run.smithi162.stdout: "AsyncMessenger::Worker-0": { 2024-03-31T23:37:44.399 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_recv_messages": 472, 2024-03-31T23:37:44.399 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_send_messages": 465, 2024-03-31T23:37:44.399 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_recv_bytes": 42522166, 2024-03-31T23:37:44.399 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_send_bytes": 167283, 2024-03-31T23:37:44.399 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_created_connections": 7, 2024-03-31T23:37:44.399 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_active_connections": 3, 2024-03-31T23:37:44.399 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_running_total_time": 0.056635621, 2024-03-31T23:37:44.399 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_running_send_time": 0.010185916, 2024-03-31T23:37:44.399 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_running_recv_time": 0.058048587, 2024-03-31T23:37:44.399 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_running_fast_dispatch_time": 0.009623023, 2024-03-31T23:37:44.399 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_send_messages_queue_lat": { 2024-03-31T23:37:44.399 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 464, 2024-03-31T23:37:44.399 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.014267029, 2024-03-31T23:37:44.399 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000030747 2024-03-31T23:37:44.400 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-31T23:37:44.400 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_handle_ack_lat": { 2024-03-31T23:37:44.400 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-31T23:37:44.400 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-31T23:37:44.400 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-31T23:37:44.400 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-31T23:37:44.400 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_recv_encrypted_bytes": 213104, 2024-03-31T23:37:44.400 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_send_encrypted_bytes": 48032 2024-03-31T23:37:44.400 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-31T23:37:44.400 INFO:teuthology.orchestra.run.smithi162.stdout: "AsyncMessenger::Worker-1": { 2024-03-31T23:37:44.400 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_recv_messages": 481, 2024-03-31T23:37:44.400 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_send_messages": 535, 2024-03-31T23:37:44.400 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_recv_bytes": 33705050, 2024-03-31T23:37:44.400 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_send_bytes": 336053, 2024-03-31T23:37:44.401 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_created_connections": 6, 2024-03-31T23:37:44.401 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_active_connections": 5, 2024-03-31T23:37:44.401 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_running_total_time": 0.055653273, 2024-03-31T23:37:44.401 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_running_send_time": 0.011698552, 2024-03-31T23:37:44.401 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_running_recv_time": 0.029021141, 2024-03-31T23:37:44.401 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_running_fast_dispatch_time": 0.011903905, 2024-03-31T23:37:44.401 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_send_messages_queue_lat": { 2024-03-31T23:37:44.401 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 534, 2024-03-31T23:37:44.401 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.022663622, 2024-03-31T23:37:44.401 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000042441 2024-03-31T23:37:44.401 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-31T23:37:44.401 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_handle_ack_lat": { 2024-03-31T23:37:44.401 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-31T23:37:44.401 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-31T23:37:44.402 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-31T23:37:44.402 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-31T23:37:44.402 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_recv_encrypted_bytes": 3152, 2024-03-31T23:37:44.402 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_send_encrypted_bytes": 184624 2024-03-31T23:37:44.402 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-31T23:37:44.402 INFO:teuthology.orchestra.run.smithi162.stdout: "AsyncMessenger::Worker-2": { 2024-03-31T23:37:44.402 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_recv_messages": 228, 2024-03-31T23:37:44.402 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_send_messages": 230, 2024-03-31T23:37:44.402 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_recv_bytes": 37807499, 2024-03-31T23:37:44.402 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_send_bytes": 72519, 2024-03-31T23:37:44.402 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_created_connections": 6, 2024-03-31T23:37:44.402 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_active_connections": 5, 2024-03-31T23:37:44.402 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_running_total_time": 0.034754649, 2024-03-31T23:37:44.402 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_running_send_time": 0.004893791, 2024-03-31T23:37:44.403 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_running_recv_time": 0.023376456, 2024-03-31T23:37:44.403 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_running_fast_dispatch_time": 0.005110834, 2024-03-31T23:37:44.403 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_send_messages_queue_lat": { 2024-03-31T23:37:44.403 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 228, 2024-03-31T23:37:44.403 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.015660878, 2024-03-31T23:37:44.403 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000068688 2024-03-31T23:37:44.403 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-31T23:37:44.403 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_handle_ack_lat": { 2024-03-31T23:37:44.403 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-31T23:37:44.403 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-31T23:37:44.403 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-31T23:37:44.403 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-31T23:37:44.403 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-31T23:37:44.403 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_send_encrypted_bytes": 0 2024-03-31T23:37:44.403 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-31T23:37:44.404 INFO:teuthology.orchestra.run.smithi162.stdout: "cct": { 2024-03-31T23:37:44.404 INFO:teuthology.orchestra.run.smithi162.stdout: "total_workers": 1, 2024-03-31T23:37:44.404 INFO:teuthology.orchestra.run.smithi162.stdout: "unhealthy_workers": 0 2024-03-31T23:37:44.404 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-31T23:37:44.404 INFO:teuthology.orchestra.run.smithi162.stdout: "finisher-MDSRank": { 2024-03-31T23:37:44.404 INFO:teuthology.orchestra.run.smithi162.stdout: "queue_len": 0, 2024-03-31T23:37:44.404 INFO:teuthology.orchestra.run.smithi162.stdout: "complete_latency": { 2024-03-31T23:37:44.404 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 1794, 2024-03-31T23:37:44.404 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.116751792, 2024-03-31T23:37:44.404 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000065079 2024-03-31T23:37:44.404 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-03-31T23:37:44.404 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-31T23:37:44.404 INFO:teuthology.orchestra.run.smithi162.stdout: "finisher-PurgeQueue": { 2024-03-31T23:37:44.404 INFO:teuthology.orchestra.run.smithi162.stdout: "queue_len": 0, 2024-03-31T23:37:44.405 INFO:teuthology.orchestra.run.smithi162.stdout: "complete_latency": { 2024-03-31T23:37:44.405 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-31T23:37:44.405 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-31T23:37:44.405 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-31T23:37:44.405 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-03-31T23:37:44.405 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-31T23:37:44.405 INFO:teuthology.orchestra.run.smithi162.stdout: "mds": { 2024-03-31T23:37:44.405 INFO:teuthology.orchestra.run.smithi162.stdout: "request": 0, 2024-03-31T23:37:44.405 INFO:teuthology.orchestra.run.smithi162.stdout: "reply": 0, 2024-03-31T23:37:44.405 INFO:teuthology.orchestra.run.smithi162.stdout: "reply_latency": { 2024-03-31T23:37:44.405 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-31T23:37:44.405 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-31T23:37:44.405 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-31T23:37:44.405 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-31T23:37:44.405 INFO:teuthology.orchestra.run.smithi162.stdout: "slow_reply": 0, 2024-03-31T23:37:44.406 INFO:teuthology.orchestra.run.smithi162.stdout: "forward": 0, 2024-03-31T23:37:44.406 INFO:teuthology.orchestra.run.smithi162.stdout: "dir_fetch_complete": 0, 2024-03-31T23:37:44.406 INFO:teuthology.orchestra.run.smithi162.stdout: "dir_fetch_keys": 0, 2024-03-31T23:37:44.406 INFO:teuthology.orchestra.run.smithi162.stdout: "dir_commit": 0, 2024-03-31T23:37:44.406 INFO:teuthology.orchestra.run.smithi162.stdout: "dir_split": 0, 2024-03-31T23:37:44.406 INFO:teuthology.orchestra.run.smithi162.stdout: "dir_merge": 0, 2024-03-31T23:37:44.406 INFO:teuthology.orchestra.run.smithi162.stdout: "inodes": 5746, 2024-03-31T23:37:44.406 INFO:teuthology.orchestra.run.smithi162.stdout: "inodes_top": 3374, 2024-03-31T23:37:44.406 INFO:teuthology.orchestra.run.smithi162.stdout: "inodes_bottom": 2372, 2024-03-31T23:37:44.406 INFO:teuthology.orchestra.run.smithi162.stdout: "inodes_pin_tail": 0, 2024-03-31T23:37:44.406 INFO:teuthology.orchestra.run.smithi162.stdout: "inodes_pinned": 926, 2024-03-31T23:37:44.406 INFO:teuthology.orchestra.run.smithi162.stdout: "inodes_expired": 0, 2024-03-31T23:37:44.406 INFO:teuthology.orchestra.run.smithi162.stdout: "inodes_with_caps": 0, 2024-03-31T23:37:44.406 INFO:teuthology.orchestra.run.smithi162.stdout: "caps": 0, 2024-03-31T23:37:44.406 INFO:teuthology.orchestra.run.smithi162.stdout: "subtrees": 2, 2024-03-31T23:37:44.407 INFO:teuthology.orchestra.run.smithi162.stdout: "traverse": 0, 2024-03-31T23:37:44.407 INFO:teuthology.orchestra.run.smithi162.stdout: "traverse_hit": 0, 2024-03-31T23:37:44.407 INFO:teuthology.orchestra.run.smithi162.stdout: "traverse_forward": 0, 2024-03-31T23:37:44.407 INFO:teuthology.orchestra.run.smithi162.stdout: "traverse_discover": 0, 2024-03-31T23:37:44.407 INFO:teuthology.orchestra.run.smithi162.stdout: "traverse_dir_fetch": 0, 2024-03-31T23:37:44.407 INFO:teuthology.orchestra.run.smithi162.stdout: "traverse_remote_ino": 0, 2024-03-31T23:37:44.407 INFO:teuthology.orchestra.run.smithi162.stdout: "traverse_lock": 0, 2024-03-31T23:37:44.407 INFO:teuthology.orchestra.run.smithi162.stdout: "load_cent": 0, 2024-03-31T23:37:44.407 INFO:teuthology.orchestra.run.smithi162.stdout: "q": 0, 2024-03-31T23:37:44.407 INFO:teuthology.orchestra.run.smithi162.stdout: "exported": 0, 2024-03-31T23:37:44.407 INFO:teuthology.orchestra.run.smithi162.stdout: "exported_inodes": 0, 2024-03-31T23:37:44.407 INFO:teuthology.orchestra.run.smithi162.stdout: "imported": 0, 2024-03-31T23:37:44.407 INFO:teuthology.orchestra.run.smithi162.stdout: "imported_inodes": 0, 2024-03-31T23:37:44.407 INFO:teuthology.orchestra.run.smithi162.stdout: "openino_dir_fetch": 0, 2024-03-31T23:37:44.407 INFO:teuthology.orchestra.run.smithi162.stdout: "openino_backtrace_fetch": 0, 2024-03-31T23:37:44.408 INFO:teuthology.orchestra.run.smithi162.stdout: "openino_peer_discover": 0, 2024-03-31T23:37:44.408 INFO:teuthology.orchestra.run.smithi162.stdout: "root_rfiles": 4, 2024-03-31T23:37:44.408 INFO:teuthology.orchestra.run.smithi162.stdout: "root_rbytes": 234, 2024-03-31T23:37:44.408 INFO:teuthology.orchestra.run.smithi162.stdout: "root_rsnaps": 6, 2024-03-31T23:37:44.408 INFO:teuthology.orchestra.run.smithi162.stdout: "scrub_backtrace_fetch": 0, 2024-03-31T23:37:44.408 INFO:teuthology.orchestra.run.smithi162.stdout: "scrub_set_tag": 0, 2024-03-31T23:37:44.408 INFO:teuthology.orchestra.run.smithi162.stdout: "scrub_backtrace_repaired": 0, 2024-03-31T23:37:44.408 INFO:teuthology.orchestra.run.smithi162.stdout: "scrub_inotable_repaired": 0, 2024-03-31T23:37:44.408 INFO:teuthology.orchestra.run.smithi162.stdout: "scrub_dir_inodes": 0, 2024-03-31T23:37:44.408 INFO:teuthology.orchestra.run.smithi162.stdout: "scrub_dir_base_inodes": 0, 2024-03-31T23:37:44.408 INFO:teuthology.orchestra.run.smithi162.stdout: "scrub_dirfrag_rstats": 0, 2024-03-31T23:37:44.408 INFO:teuthology.orchestra.run.smithi162.stdout: "scrub_file_inodes": 0, 2024-03-31T23:37:44.408 INFO:teuthology.orchestra.run.smithi162.stdout: "handle_inode_file_caps": 0, 2024-03-31T23:37:44.408 INFO:teuthology.orchestra.run.smithi162.stdout: "ceph_cap_op_revoke": 0, 2024-03-31T23:37:44.408 INFO:teuthology.orchestra.run.smithi162.stdout: "ceph_cap_op_grant": 0, 2024-03-31T23:37:44.409 INFO:teuthology.orchestra.run.smithi162.stdout: "ceph_cap_op_trunc": 0, 2024-03-31T23:37:44.409 INFO:teuthology.orchestra.run.smithi162.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-31T23:37:44.409 INFO:teuthology.orchestra.run.smithi162.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-31T23:37:44.409 INFO:teuthology.orchestra.run.smithi162.stdout: "handle_client_caps": 0, 2024-03-31T23:37:44.409 INFO:teuthology.orchestra.run.smithi162.stdout: "handle_client_caps_dirty": 0, 2024-03-31T23:37:44.409 INFO:teuthology.orchestra.run.smithi162.stdout: "handle_client_cap_release": 0, 2024-03-31T23:37:44.409 INFO:teuthology.orchestra.run.smithi162.stdout: "process_request_cap_release": 0 2024-03-31T23:37:44.409 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-31T23:37:44.409 INFO:teuthology.orchestra.run.smithi162.stdout: "mds_cache": { 2024-03-31T23:37:44.409 INFO:teuthology.orchestra.run.smithi162.stdout: "dir_update": 0, 2024-03-31T23:37:44.409 INFO:teuthology.orchestra.run.smithi162.stdout: "dir_update_receipt": 0, 2024-03-31T23:37:44.409 INFO:teuthology.orchestra.run.smithi162.stdout: "dir_try_discover": 0, 2024-03-31T23:37:44.409 INFO:teuthology.orchestra.run.smithi162.stdout: "dir_send_discover": 0, 2024-03-31T23:37:44.409 INFO:teuthology.orchestra.run.smithi162.stdout: "dir_handle_discover": 0, 2024-03-31T23:37:44.409 INFO:teuthology.orchestra.run.smithi162.stdout: "num_strays": 0, 2024-03-31T23:37:44.410 INFO:teuthology.orchestra.run.smithi162.stdout: "num_strays_delayed": 0, 2024-03-31T23:37:44.410 INFO:teuthology.orchestra.run.smithi162.stdout: "num_strays_enqueuing": 0, 2024-03-31T23:37:44.410 INFO:teuthology.orchestra.run.smithi162.stdout: "strays_created": 0, 2024-03-31T23:37:44.410 INFO:teuthology.orchestra.run.smithi162.stdout: "strays_enqueued": 0, 2024-03-31T23:37:44.410 INFO:teuthology.orchestra.run.smithi162.stdout: "strays_reintegrated": 0, 2024-03-31T23:37:44.410 INFO:teuthology.orchestra.run.smithi162.stdout: "strays_migrated": 0, 2024-03-31T23:37:44.410 INFO:teuthology.orchestra.run.smithi162.stdout: "num_recovering_processing": 0, 2024-03-31T23:37:44.410 INFO:teuthology.orchestra.run.smithi162.stdout: "num_recovering_enqueued": 0, 2024-03-31T23:37:44.410 INFO:teuthology.orchestra.run.smithi162.stdout: "num_recovering_prioritized": 0, 2024-03-31T23:37:44.410 INFO:teuthology.orchestra.run.smithi162.stdout: "recovery_started": 0, 2024-03-31T23:37:44.410 INFO:teuthology.orchestra.run.smithi162.stdout: "recovery_completed": 0, 2024-03-31T23:37:44.410 INFO:teuthology.orchestra.run.smithi162.stdout: "ireq_quiesce_path": 0, 2024-03-31T23:37:44.410 INFO:teuthology.orchestra.run.smithi162.stdout: "ireq_quiesce_inode": 0, 2024-03-31T23:37:44.410 INFO:teuthology.orchestra.run.smithi162.stdout: "ireq_enqueue_scrub": 0, 2024-03-31T23:37:44.410 INFO:teuthology.orchestra.run.smithi162.stdout: "ireq_exportdir": 0, 2024-03-31T23:37:44.411 INFO:teuthology.orchestra.run.smithi162.stdout: "ireq_flush": 0, 2024-03-31T23:37:44.411 INFO:teuthology.orchestra.run.smithi162.stdout: "ireq_fragmentdir": 0, 2024-03-31T23:37:44.411 INFO:teuthology.orchestra.run.smithi162.stdout: "ireq_fragstats": 0, 2024-03-31T23:37:44.411 INFO:teuthology.orchestra.run.smithi162.stdout: "ireq_inodestats": 0 2024-03-31T23:37:44.411 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-31T23:37:44.411 INFO:teuthology.orchestra.run.smithi162.stdout: "mds_log": { 2024-03-31T23:37:44.411 INFO:teuthology.orchestra.run.smithi162.stdout: "evlrg": 0, 2024-03-31T23:37:44.411 INFO:teuthology.orchestra.run.smithi162.stdout: "evadd": 0, 2024-03-31T23:37:44.411 INFO:teuthology.orchestra.run.smithi162.stdout: "evex": 0, 2024-03-31T23:37:44.411 INFO:teuthology.orchestra.run.smithi162.stdout: "evtrm": 0, 2024-03-31T23:37:44.411 INFO:teuthology.orchestra.run.smithi162.stdout: "ev": 45515, 2024-03-31T23:37:44.411 INFO:teuthology.orchestra.run.smithi162.stdout: "evexg": 0, 2024-03-31T23:37:44.411 INFO:teuthology.orchestra.run.smithi162.stdout: "evexd": 0, 2024-03-31T23:37:44.411 INFO:teuthology.orchestra.run.smithi162.stdout: "segadd": 0, 2024-03-31T23:37:44.411 INFO:teuthology.orchestra.run.smithi162.stdout: "segex": 0, 2024-03-31T23:37:44.412 INFO:teuthology.orchestra.run.smithi162.stdout: "segtrm": 0, 2024-03-31T23:37:44.412 INFO:teuthology.orchestra.run.smithi162.stdout: "seg": 5, 2024-03-31T23:37:44.412 INFO:teuthology.orchestra.run.smithi162.stdout: "segmjr": 17, 2024-03-31T23:37:44.412 INFO:teuthology.orchestra.run.smithi162.stdout: "segexg": 0, 2024-03-31T23:37:44.412 INFO:teuthology.orchestra.run.smithi162.stdout: "segexd": 0, 2024-03-31T23:37:44.412 INFO:teuthology.orchestra.run.smithi162.stdout: "expos": 482989945, 2024-03-31T23:37:44.412 INFO:teuthology.orchestra.run.smithi162.stdout: "wrpos": 491140865, 2024-03-31T23:37:44.412 INFO:teuthology.orchestra.run.smithi162.stdout: "rdpos": 491132323, 2024-03-31T23:37:44.412 INFO:teuthology.orchestra.run.smithi162.stdout: "jlat": { 2024-03-31T23:37:44.412 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-31T23:37:44.412 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-31T23:37:44.412 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-31T23:37:44.412 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-31T23:37:44.412 INFO:teuthology.orchestra.run.smithi162.stdout: "replayed": 45515 2024-03-31T23:37:44.413 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-31T23:37:44.413 INFO:teuthology.orchestra.run.smithi162.stdout: "mds_mem": { 2024-03-31T23:37:44.413 INFO:teuthology.orchestra.run.smithi162.stdout: "ino": 522, 2024-03-31T23:37:44.413 INFO:teuthology.orchestra.run.smithi162.stdout: "ino+": 5884, 2024-03-31T23:37:44.413 INFO:teuthology.orchestra.run.smithi162.stdout: "ino-": 5362, 2024-03-31T23:37:44.413 INFO:teuthology.orchestra.run.smithi162.stdout: "dir": 77, 2024-03-31T23:37:44.413 INFO:teuthology.orchestra.run.smithi162.stdout: "dir+": 130, 2024-03-31T23:37:44.413 INFO:teuthology.orchestra.run.smithi162.stdout: "dir-": 53, 2024-03-31T23:37:44.413 INFO:teuthology.orchestra.run.smithi162.stdout: "dn": 5746, 2024-03-31T23:37:44.413 INFO:teuthology.orchestra.run.smithi162.stdout: "dn+": 6557, 2024-03-31T23:37:44.413 INFO:teuthology.orchestra.run.smithi162.stdout: "dn-": 811, 2024-03-31T23:37:44.413 INFO:teuthology.orchestra.run.smithi162.stdout: "cap": 0, 2024-03-31T23:37:44.413 INFO:teuthology.orchestra.run.smithi162.stdout: "cap+": 0, 2024-03-31T23:37:44.413 INFO:teuthology.orchestra.run.smithi162.stdout: "cap-": 0, 2024-03-31T23:37:44.413 INFO:teuthology.orchestra.run.smithi162.stdout: "rss": 59396, 2024-03-31T23:37:44.414 INFO:teuthology.orchestra.run.smithi162.stdout: "heap": 215396 2024-03-31T23:37:44.414 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-31T23:37:44.414 INFO:teuthology.orchestra.run.smithi162.stdout: "mds_server": { 2024-03-31T23:37:44.414 INFO:teuthology.orchestra.run.smithi162.stdout: "dispatch_client_request": 0, 2024-03-31T23:37:44.414 INFO:teuthology.orchestra.run.smithi162.stdout: "dispatch_server_request": 0, 2024-03-31T23:37:44.414 INFO:teuthology.orchestra.run.smithi162.stdout: "handle_client_request": 0, 2024-03-31T23:37:44.414 INFO:teuthology.orchestra.run.smithi162.stdout: "handle_client_session": 0, 2024-03-31T23:37:44.414 INFO:teuthology.orchestra.run.smithi162.stdout: "handle_peer_request": 0, 2024-03-31T23:37:44.414 INFO:teuthology.orchestra.run.smithi162.stdout: "req_create_latency": { 2024-03-31T23:37:44.414 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-31T23:37:44.414 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-31T23:37:44.414 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-31T23:37:44.414 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-31T23:37:44.414 INFO:teuthology.orchestra.run.smithi162.stdout: "req_getattr_latency": { 2024-03-31T23:37:44.415 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-31T23:37:44.415 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-31T23:37:44.415 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-31T23:37:44.415 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-31T23:37:44.415 INFO:teuthology.orchestra.run.smithi162.stdout: "req_getfilelock_latency": { 2024-03-31T23:37:44.415 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-31T23:37:44.415 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-31T23:37:44.415 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-31T23:37:44.415 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-31T23:37:44.415 INFO:teuthology.orchestra.run.smithi162.stdout: "req_link_latency": { 2024-03-31T23:37:44.415 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-31T23:37:44.415 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-31T23:37:44.415 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-31T23:37:44.415 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-31T23:37:44.415 INFO:teuthology.orchestra.run.smithi162.stdout: "req_lookup_latency": { 2024-03-31T23:37:44.416 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-31T23:37:44.416 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-31T23:37:44.416 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-31T23:37:44.416 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-31T23:37:44.416 INFO:teuthology.orchestra.run.smithi162.stdout: "req_lookuphash_latency": { 2024-03-31T23:37:44.416 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-31T23:37:44.416 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-31T23:37:44.416 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-31T23:37:44.416 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-31T23:37:44.416 INFO:teuthology.orchestra.run.smithi162.stdout: "req_lookupino_latency": { 2024-03-31T23:37:44.416 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-31T23:37:44.416 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-31T23:37:44.416 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-31T23:37:44.416 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-31T23:37:44.417 INFO:teuthology.orchestra.run.smithi162.stdout: "req_lookupname_latency": { 2024-03-31T23:37:44.417 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-31T23:37:44.417 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-31T23:37:44.417 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-31T23:37:44.417 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-31T23:37:44.417 INFO:teuthology.orchestra.run.smithi162.stdout: "req_lookupparent_latency": { 2024-03-31T23:37:44.417 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-31T23:37:44.417 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-31T23:37:44.417 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-31T23:37:44.417 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-31T23:37:44.417 INFO:teuthology.orchestra.run.smithi162.stdout: "req_lookupsnap_latency": { 2024-03-31T23:37:44.417 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-31T23:37:44.417 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-31T23:37:44.417 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-31T23:37:44.417 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-31T23:37:44.418 INFO:teuthology.orchestra.run.smithi162.stdout: "req_lssnap_latency": { 2024-03-31T23:37:44.418 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-31T23:37:44.418 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-31T23:37:44.418 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-31T23:37:44.418 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-31T23:37:44.418 INFO:teuthology.orchestra.run.smithi162.stdout: "req_mkdir_latency": { 2024-03-31T23:37:44.419 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-31T23:37:44.419 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-31T23:37:44.419 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-31T23:37:44.419 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-31T23:37:44.419 INFO:teuthology.orchestra.run.smithi162.stdout: "req_mknod_latency": { 2024-03-31T23:37:44.419 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-31T23:37:44.419 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-31T23:37:44.419 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-31T23:37:44.419 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-31T23:37:44.419 INFO:teuthology.orchestra.run.smithi162.stdout: "req_mksnap_latency": { 2024-03-31T23:37:44.419 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-31T23:37:44.419 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-31T23:37:44.419 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-31T23:37:44.420 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-31T23:37:44.420 INFO:teuthology.orchestra.run.smithi162.stdout: "req_open_latency": { 2024-03-31T23:37:44.420 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-31T23:37:44.420 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-31T23:37:44.420 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-31T23:37:44.420 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-31T23:37:44.420 INFO:teuthology.orchestra.run.smithi162.stdout: "req_readdir_latency": { 2024-03-31T23:37:44.420 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-31T23:37:44.420 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-31T23:37:44.420 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-31T23:37:44.420 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-31T23:37:44.420 INFO:teuthology.orchestra.run.smithi162.stdout: "req_rename_latency": { 2024-03-31T23:37:44.420 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-31T23:37:44.420 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-31T23:37:44.420 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-31T23:37:44.421 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-31T23:37:44.421 INFO:teuthology.orchestra.run.smithi162.stdout: "req_renamesnap_latency": { 2024-03-31T23:37:44.421 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-31T23:37:44.421 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-31T23:37:44.421 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-31T23:37:44.421 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-31T23:37:44.421 INFO:teuthology.orchestra.run.smithi162.stdout: "req_snapdiff_latency": { 2024-03-31T23:37:44.421 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-31T23:37:44.421 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-31T23:37:44.421 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-31T23:37:44.421 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-31T23:37:44.421 INFO:teuthology.orchestra.run.smithi162.stdout: "req_rmdir_latency": { 2024-03-31T23:37:44.421 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-31T23:37:44.421 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-31T23:37:44.422 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-31T23:37:44.422 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-31T23:37:44.422 INFO:teuthology.orchestra.run.smithi162.stdout: "req_rmsnap_latency": { 2024-03-31T23:37:44.422 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-31T23:37:44.422 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-31T23:37:44.422 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-31T23:37:44.422 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-31T23:37:44.422 INFO:teuthology.orchestra.run.smithi162.stdout: "req_rmxattr_latency": { 2024-03-31T23:37:44.422 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-31T23:37:44.422 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-31T23:37:44.422 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-31T23:37:44.422 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-31T23:37:44.422 INFO:teuthology.orchestra.run.smithi162.stdout: "req_setattr_latency": { 2024-03-31T23:37:44.422 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-31T23:37:44.423 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-31T23:37:44.423 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-31T23:37:44.423 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-31T23:37:44.423 INFO:teuthology.orchestra.run.smithi162.stdout: "req_setdirlayout_latency": { 2024-03-31T23:37:44.423 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-31T23:37:44.423 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-31T23:37:44.423 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-31T23:37:44.423 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-31T23:37:44.423 INFO:teuthology.orchestra.run.smithi162.stdout: "req_setfilelock_latency": { 2024-03-31T23:37:44.423 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-31T23:37:44.423 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-31T23:37:44.423 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-31T23:37:44.423 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-31T23:37:44.423 INFO:teuthology.orchestra.run.smithi162.stdout: "req_setlayout_latency": { 2024-03-31T23:37:44.424 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-31T23:37:44.424 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-31T23:37:44.424 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-31T23:37:44.424 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-31T23:37:44.424 INFO:teuthology.orchestra.run.smithi162.stdout: "req_setxattr_latency": { 2024-03-31T23:37:44.424 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-31T23:37:44.424 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-31T23:37:44.424 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-31T23:37:44.424 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-31T23:37:44.424 INFO:teuthology.orchestra.run.smithi162.stdout: "req_symlink_latency": { 2024-03-31T23:37:44.424 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-31T23:37:44.424 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-31T23:37:44.424 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-31T23:37:44.424 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-31T23:37:44.424 INFO:teuthology.orchestra.run.smithi162.stdout: "req_unlink_latency": { 2024-03-31T23:37:44.425 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-31T23:37:44.425 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-31T23:37:44.425 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-31T23:37:44.425 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-31T23:37:44.425 INFO:teuthology.orchestra.run.smithi162.stdout: "cap_revoke_eviction": 0, 2024-03-31T23:37:44.425 INFO:teuthology.orchestra.run.smithi162.stdout: "cap_acquisition_throttle": 0, 2024-03-31T23:37:44.425 INFO:teuthology.orchestra.run.smithi162.stdout: "req_getvxattr_latency": { 2024-03-31T23:37:44.425 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-31T23:37:44.425 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-31T23:37:44.425 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-31T23:37:44.425 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-03-31T23:37:44.425 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-31T23:37:44.425 INFO:teuthology.orchestra.run.smithi162.stdout: "mds_sessions": { 2024-03-31T23:37:44.425 INFO:teuthology.orchestra.run.smithi162.stdout: "session_count": 2, 2024-03-31T23:37:44.425 INFO:teuthology.orchestra.run.smithi162.stdout: "session_add": 2, 2024-03-31T23:37:44.426 INFO:teuthology.orchestra.run.smithi162.stdout: "session_remove": 0, 2024-03-31T23:37:44.426 INFO:teuthology.orchestra.run.smithi162.stdout: "sessions_open": 0, 2024-03-31T23:37:44.426 INFO:teuthology.orchestra.run.smithi162.stdout: "sessions_stale": 0, 2024-03-31T23:37:44.426 INFO:teuthology.orchestra.run.smithi162.stdout: "total_load": 0, 2024-03-31T23:37:44.426 INFO:teuthology.orchestra.run.smithi162.stdout: "average_load": 0, 2024-03-31T23:37:44.426 INFO:teuthology.orchestra.run.smithi162.stdout: "avg_session_uptime": 2665, 2024-03-31T23:37:44.426 INFO:teuthology.orchestra.run.smithi162.stdout: "mdthresh_evicted": 0 2024-03-31T23:37:44.426 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-31T23:37:44.426 INFO:teuthology.orchestra.run.smithi162.stdout: "mempool": { 2024-03-31T23:37:44.426 INFO:teuthology.orchestra.run.smithi162.stdout: "bloom_filter_bytes": 0, 2024-03-31T23:37:44.426 INFO:teuthology.orchestra.run.smithi162.stdout: "bloom_filter_items": 0, 2024-03-31T23:37:44.426 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_alloc_bytes": 0, 2024-03-31T23:37:44.426 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_alloc_items": 0, 2024-03-31T23:37:44.427 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_cache_data_bytes": 0, 2024-03-31T23:37:44.427 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_cache_data_items": 0, 2024-03-31T23:37:44.427 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-31T23:37:44.427 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_cache_onode_items": 0, 2024-03-31T23:37:44.427 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-31T23:37:44.427 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_cache_meta_items": 0, 2024-03-31T23:37:44.427 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_cache_other_bytes": 0, 2024-03-31T23:37:44.427 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_cache_other_items": 0, 2024-03-31T23:37:44.427 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-31T23:37:44.427 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_cache_buffer_items": 0, 2024-03-31T23:37:44.427 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_extent_bytes": 0, 2024-03-31T23:37:44.427 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_extent_items": 0, 2024-03-31T23:37:44.427 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_blob_bytes": 0, 2024-03-31T23:37:44.427 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_blob_items": 0, 2024-03-31T23:37:44.428 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-31T23:37:44.428 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_shared_blob_items": 0, 2024-03-31T23:37:44.428 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-31T23:37:44.428 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_inline_bl_items": 0, 2024-03-31T23:37:44.428 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_fsck_bytes": 0, 2024-03-31T23:37:44.428 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_fsck_items": 0, 2024-03-31T23:37:44.428 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_txc_bytes": 0, 2024-03-31T23:37:44.428 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_txc_items": 0, 2024-03-31T23:37:44.428 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-31T23:37:44.428 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_writing_deferred_items": 0, 2024-03-31T23:37:44.428 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_writing_bytes": 0, 2024-03-31T23:37:44.428 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_writing_items": 0, 2024-03-31T23:37:44.428 INFO:teuthology.orchestra.run.smithi162.stdout: "bluefs_bytes": 0, 2024-03-31T23:37:44.428 INFO:teuthology.orchestra.run.smithi162.stdout: "bluefs_items": 0, 2024-03-31T23:37:44.428 INFO:teuthology.orchestra.run.smithi162.stdout: "bluefs_file_reader_bytes": 0, 2024-03-31T23:37:44.429 INFO:teuthology.orchestra.run.smithi162.stdout: "bluefs_file_reader_items": 0, 2024-03-31T23:37:44.429 INFO:teuthology.orchestra.run.smithi162.stdout: "bluefs_file_writer_bytes": 0, 2024-03-31T23:37:44.429 INFO:teuthology.orchestra.run.smithi162.stdout: "bluefs_file_writer_items": 0, 2024-03-31T23:37:44.429 INFO:teuthology.orchestra.run.smithi162.stdout: "buffer_anon_bytes": 229639, 2024-03-31T23:37:44.429 INFO:teuthology.orchestra.run.smithi162.stdout: "buffer_anon_items": 68, 2024-03-31T23:37:44.429 INFO:teuthology.orchestra.run.smithi162.stdout: "buffer_meta_bytes": 0, 2024-03-31T23:37:44.429 INFO:teuthology.orchestra.run.smithi162.stdout: "buffer_meta_items": 0, 2024-03-31T23:37:44.429 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_bytes": 0, 2024-03-31T23:37:44.429 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_items": 0, 2024-03-31T23:37:44.429 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_mapbl_bytes": 0, 2024-03-31T23:37:44.429 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_mapbl_items": 0, 2024-03-31T23:37:44.429 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_pglog_bytes": 0, 2024-03-31T23:37:44.429 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_pglog_items": 0, 2024-03-31T23:37:44.429 INFO:teuthology.orchestra.run.smithi162.stdout: "osdmap_bytes": 9704, 2024-03-31T23:37:44.429 INFO:teuthology.orchestra.run.smithi162.stdout: "osdmap_items": 186, 2024-03-31T23:37:44.430 INFO:teuthology.orchestra.run.smithi162.stdout: "osdmap_mapping_bytes": 0, 2024-03-31T23:37:44.430 INFO:teuthology.orchestra.run.smithi162.stdout: "osdmap_mapping_items": 0, 2024-03-31T23:37:44.430 INFO:teuthology.orchestra.run.smithi162.stdout: "pgmap_bytes": 0, 2024-03-31T23:37:44.430 INFO:teuthology.orchestra.run.smithi162.stdout: "pgmap_items": 0, 2024-03-31T23:37:44.430 INFO:teuthology.orchestra.run.smithi162.stdout: "mds_co_bytes": 5430285, 2024-03-31T23:37:44.430 INFO:teuthology.orchestra.run.smithi162.stdout: "mds_co_items": 29032, 2024-03-31T23:37:44.430 INFO:teuthology.orchestra.run.smithi162.stdout: "unittest_1_bytes": 0, 2024-03-31T23:37:44.430 INFO:teuthology.orchestra.run.smithi162.stdout: "unittest_1_items": 0, 2024-03-31T23:37:44.430 INFO:teuthology.orchestra.run.smithi162.stdout: "unittest_2_bytes": 0, 2024-03-31T23:37:44.430 INFO:teuthology.orchestra.run.smithi162.stdout: "unittest_2_items": 0 2024-03-31T23:37:44.430 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-31T23:37:44.430 INFO:teuthology.orchestra.run.smithi162.stdout: "objecter": { 2024-03-31T23:37:44.430 INFO:teuthology.orchestra.run.smithi162.stdout: "op_active": 0, 2024-03-31T23:37:44.430 INFO:teuthology.orchestra.run.smithi162.stdout: "op_laggy": 0, 2024-03-31T23:37:44.431 INFO:teuthology.orchestra.run.smithi162.stdout: "op_send": 1079, 2024-03-31T23:37:44.431 INFO:teuthology.orchestra.run.smithi162.stdout: "op_send_bytes": 26, 2024-03-31T23:37:44.431 INFO:teuthology.orchestra.run.smithi162.stdout: "op_resend": 4, 2024-03-31T23:37:44.431 INFO:teuthology.orchestra.run.smithi162.stdout: "op_reply": 1079, 2024-03-31T23:37:44.431 INFO:teuthology.orchestra.run.smithi162.stdout: "op_latency": { 2024-03-31T23:37:44.431 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 1079, 2024-03-31T23:37:44.431 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 1.490971497, 2024-03-31T23:37:44.431 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.001381808 2024-03-31T23:37:44.431 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-31T23:37:44.431 INFO:teuthology.orchestra.run.smithi162.stdout: "op_inflight": 0, 2024-03-31T23:37:44.431 INFO:teuthology.orchestra.run.smithi162.stdout: "oplen_avg": { 2024-03-31T23:37:44.431 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 1079, 2024-03-31T23:37:44.431 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 1080 2024-03-31T23:37:44.431 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-31T23:37:44.431 INFO:teuthology.orchestra.run.smithi162.stdout: "op": 1079, 2024-03-31T23:37:44.432 INFO:teuthology.orchestra.run.smithi162.stdout: "op_r": 1079, 2024-03-31T23:37:44.432 INFO:teuthology.orchestra.run.smithi162.stdout: "op_w": 0, 2024-03-31T23:37:44.432 INFO:teuthology.orchestra.run.smithi162.stdout: "op_rmw": 0, 2024-03-31T23:37:44.432 INFO:teuthology.orchestra.run.smithi162.stdout: "op_pg": 0, 2024-03-31T23:37:44.432 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_stat": 552, 2024-03-31T23:37:44.432 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_create": 0, 2024-03-31T23:37:44.432 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_read": 524, 2024-03-31T23:37:44.432 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_write": 0, 2024-03-31T23:37:44.432 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_writefull": 0, 2024-03-31T23:37:44.432 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_writesame": 0, 2024-03-31T23:37:44.432 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_append": 0, 2024-03-31T23:37:44.432 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_zero": 0, 2024-03-31T23:37:44.432 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_truncate": 0, 2024-03-31T23:37:44.432 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_delete": 0, 2024-03-31T23:37:44.432 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_mapext": 0, 2024-03-31T23:37:44.433 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_sparse_read": 0, 2024-03-31T23:37:44.433 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_clonerange": 0, 2024-03-31T23:37:44.433 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_getxattr": 2, 2024-03-31T23:37:44.433 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_setxattr": 0, 2024-03-31T23:37:44.433 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_cmpxattr": 0, 2024-03-31T23:37:44.433 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_rmxattr": 0, 2024-03-31T23:37:44.433 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_resetxattrs": 0, 2024-03-31T23:37:44.433 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_call": 0, 2024-03-31T23:37:44.433 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_watch": 0, 2024-03-31T23:37:44.433 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_notify": 0, 2024-03-31T23:37:44.433 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_src_cmpxattr": 0, 2024-03-31T23:37:44.433 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_pgls": 0, 2024-03-31T23:37:44.433 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_pgls_filter": 0, 2024-03-31T23:37:44.433 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_other": 0, 2024-03-31T23:37:44.434 INFO:teuthology.orchestra.run.smithi162.stdout: "linger_active": 0, 2024-03-31T23:37:44.434 INFO:teuthology.orchestra.run.smithi162.stdout: "linger_send": 0, 2024-03-31T23:37:44.434 INFO:teuthology.orchestra.run.smithi162.stdout: "linger_resend": 0, 2024-03-31T23:37:44.434 INFO:teuthology.orchestra.run.smithi162.stdout: "linger_ping": 0, 2024-03-31T23:37:44.434 INFO:teuthology.orchestra.run.smithi162.stdout: "poolop_active": 0, 2024-03-31T23:37:44.434 INFO:teuthology.orchestra.run.smithi162.stdout: "poolop_send": 0, 2024-03-31T23:37:44.434 INFO:teuthology.orchestra.run.smithi162.stdout: "poolop_resend": 0, 2024-03-31T23:37:44.434 INFO:teuthology.orchestra.run.smithi162.stdout: "poolstat_active": 0, 2024-03-31T23:37:44.434 INFO:teuthology.orchestra.run.smithi162.stdout: "poolstat_send": 0, 2024-03-31T23:37:44.434 INFO:teuthology.orchestra.run.smithi162.stdout: "poolstat_resend": 0, 2024-03-31T23:37:44.434 INFO:teuthology.orchestra.run.smithi162.stdout: "statfs_active": 0, 2024-03-31T23:37:44.434 INFO:teuthology.orchestra.run.smithi162.stdout: "statfs_send": 0, 2024-03-31T23:37:44.434 INFO:teuthology.orchestra.run.smithi162.stdout: "statfs_resend": 0, 2024-03-31T23:37:44.434 INFO:teuthology.orchestra.run.smithi162.stdout: "command_active": 0, 2024-03-31T23:37:44.435 INFO:teuthology.orchestra.run.smithi162.stdout: "command_send": 0, 2024-03-31T23:37:44.435 INFO:teuthology.orchestra.run.smithi162.stdout: "command_resend": 0, 2024-03-31T23:37:44.435 INFO:teuthology.orchestra.run.smithi162.stdout: "map_epoch": 139, 2024-03-31T23:37:44.435 INFO:teuthology.orchestra.run.smithi162.stdout: "map_full": 0, 2024-03-31T23:37:44.435 INFO:teuthology.orchestra.run.smithi162.stdout: "map_inc": 8, 2024-03-31T23:37:44.435 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_sessions": 10, 2024-03-31T23:37:44.435 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_session_open": 10, 2024-03-31T23:37:44.435 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_session_close": 0, 2024-03-31T23:37:44.435 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_laggy": 0, 2024-03-31T23:37:44.435 INFO:teuthology.orchestra.run.smithi162.stdout: "omap_wr": 0, 2024-03-31T23:37:44.435 INFO:teuthology.orchestra.run.smithi162.stdout: "omap_rd": 2, 2024-03-31T23:37:44.435 INFO:teuthology.orchestra.run.smithi162.stdout: "omap_del": 0 2024-03-31T23:37:44.435 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-31T23:37:44.435 INFO:teuthology.orchestra.run.smithi162.stdout: "oft": { 2024-03-31T23:37:44.435 INFO:teuthology.orchestra.run.smithi162.stdout: "omap_total_objs": 0, 2024-03-31T23:37:44.436 INFO:teuthology.orchestra.run.smithi162.stdout: "omap_total_kv_pairs": 0, 2024-03-31T23:37:44.436 INFO:teuthology.orchestra.run.smithi162.stdout: "omap_total_updates": 0, 2024-03-31T23:37:44.436 INFO:teuthology.orchestra.run.smithi162.stdout: "omap_total_removes": 0 2024-03-31T23:37:44.436 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-31T23:37:44.436 INFO:teuthology.orchestra.run.smithi162.stdout: "osd-slow-ops": { 2024-03-31T23:37:44.436 INFO:teuthology.orchestra.run.smithi162.stdout: "slow_ops_count": 0 2024-03-31T23:37:44.436 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-31T23:37:44.436 INFO:teuthology.orchestra.run.smithi162.stdout: "purge_queue": { 2024-03-31T23:37:44.436 INFO:teuthology.orchestra.run.smithi162.stdout: "pq_executing_ops": 0, 2024-03-31T23:37:44.436 INFO:teuthology.orchestra.run.smithi162.stdout: "pq_executing_ops_high_water": 0, 2024-03-31T23:37:44.436 INFO:teuthology.orchestra.run.smithi162.stdout: "pq_executing": 0, 2024-03-31T23:37:44.436 INFO:teuthology.orchestra.run.smithi162.stdout: "pq_executing_high_water": 0, 2024-03-31T23:37:44.436 INFO:teuthology.orchestra.run.smithi162.stdout: "pq_executed_ops": 0, 2024-03-31T23:37:44.436 INFO:teuthology.orchestra.run.smithi162.stdout: "pq_executed": 0, 2024-03-31T23:37:44.437 INFO:teuthology.orchestra.run.smithi162.stdout: "pq_item_in_journal": 0 2024-03-31T23:37:44.437 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-31T23:37:44.437 INFO:teuthology.orchestra.run.smithi162.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-31T23:37:44.437 INFO:teuthology.orchestra.run.smithi162.stdout: "val": 0, 2024-03-31T23:37:44.437 INFO:teuthology.orchestra.run.smithi162.stdout: "max": 104857600, 2024-03-31T23:37:44.437 INFO:teuthology.orchestra.run.smithi162.stdout: "get_started": 0, 2024-03-31T23:37:44.437 INFO:teuthology.orchestra.run.smithi162.stdout: "get": 1177, 2024-03-31T23:37:44.437 INFO:teuthology.orchestra.run.smithi162.stdout: "get_sum": 113920881, 2024-03-31T23:37:44.437 INFO:teuthology.orchestra.run.smithi162.stdout: "get_or_fail_fail": 0, 2024-03-31T23:37:44.437 INFO:teuthology.orchestra.run.smithi162.stdout: "get_or_fail_success": 1177, 2024-03-31T23:37:44.437 INFO:teuthology.orchestra.run.smithi162.stdout: "take": 0, 2024-03-31T23:37:44.437 INFO:teuthology.orchestra.run.smithi162.stdout: "take_sum": 0, 2024-03-31T23:37:44.437 INFO:teuthology.orchestra.run.smithi162.stdout: "put": 1177, 2024-03-31T23:37:44.437 INFO:teuthology.orchestra.run.smithi162.stdout: "put_sum": 113920881, 2024-03-31T23:37:44.438 INFO:teuthology.orchestra.run.smithi162.stdout: "wait": { 2024-03-31T23:37:44.438 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-31T23:37:44.438 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-31T23:37:44.438 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-31T23:37:44.438 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-03-31T23:37:44.438 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-31T23:37:44.438 INFO:teuthology.orchestra.run.smithi162.stdout: "throttle-objecter_bytes": { 2024-03-31T23:37:44.438 INFO:teuthology.orchestra.run.smithi162.stdout: "val": 0, 2024-03-31T23:37:44.438 INFO:teuthology.orchestra.run.smithi162.stdout: "max": 104857600, 2024-03-31T23:37:44.438 INFO:teuthology.orchestra.run.smithi162.stdout: "get_started": 0, 2024-03-31T23:37:44.438 INFO:teuthology.orchestra.run.smithi162.stdout: "get": 0, 2024-03-31T23:37:44.438 INFO:teuthology.orchestra.run.smithi162.stdout: "get_sum": 0, 2024-03-31T23:37:44.438 INFO:teuthology.orchestra.run.smithi162.stdout: "get_or_fail_fail": 0, 2024-03-31T23:37:44.438 INFO:teuthology.orchestra.run.smithi162.stdout: "get_or_fail_success": 0, 2024-03-31T23:37:44.439 INFO:teuthology.orchestra.run.smithi162.stdout: "take": 1079, 2024-03-31T23:37:44.439 INFO:teuthology.orchestra.run.smithi162.stdout: "take_sum": 113516317, 2024-03-31T23:37:44.439 INFO:teuthology.orchestra.run.smithi162.stdout: "put": 1079, 2024-03-31T23:37:44.439 INFO:teuthology.orchestra.run.smithi162.stdout: "put_sum": 113516317, 2024-03-31T23:37:44.439 INFO:teuthology.orchestra.run.smithi162.stdout: "wait": { 2024-03-31T23:37:44.439 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-31T23:37:44.439 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-31T23:37:44.439 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-31T23:37:44.439 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-03-31T23:37:44.439 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-31T23:37:44.439 INFO:teuthology.orchestra.run.smithi162.stdout: "throttle-objecter_ops": { 2024-03-31T23:37:44.439 INFO:teuthology.orchestra.run.smithi162.stdout: "val": 0, 2024-03-31T23:37:44.439 INFO:teuthology.orchestra.run.smithi162.stdout: "max": 1024, 2024-03-31T23:37:44.439 INFO:teuthology.orchestra.run.smithi162.stdout: "get_started": 0, 2024-03-31T23:37:44.439 INFO:teuthology.orchestra.run.smithi162.stdout: "get": 0, 2024-03-31T23:37:44.440 INFO:teuthology.orchestra.run.smithi162.stdout: "get_sum": 0, 2024-03-31T23:37:44.440 INFO:teuthology.orchestra.run.smithi162.stdout: "get_or_fail_fail": 0, 2024-03-31T23:37:44.440 INFO:teuthology.orchestra.run.smithi162.stdout: "get_or_fail_success": 0, 2024-03-31T23:37:44.440 INFO:teuthology.orchestra.run.smithi162.stdout: "take": 1079, 2024-03-31T23:37:44.440 INFO:teuthology.orchestra.run.smithi162.stdout: "take_sum": 1079, 2024-03-31T23:37:44.440 INFO:teuthology.orchestra.run.smithi162.stdout: "put": 1079, 2024-03-31T23:37:44.440 INFO:teuthology.orchestra.run.smithi162.stdout: "put_sum": 1079, 2024-03-31T23:37:44.440 INFO:teuthology.orchestra.run.smithi162.stdout: "wait": { 2024-03-31T23:37:44.440 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-31T23:37:44.440 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-31T23:37:44.440 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-31T23:37:44.440 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-03-31T23:37:44.440 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-31T23:37:44.440 INFO:teuthology.orchestra.run.smithi162.stdout: "throttle-write_buf_throttle": { 2024-03-31T23:37:44.440 INFO:teuthology.orchestra.run.smithi162.stdout: "val": 0, 2024-03-31T23:37:44.441 INFO:teuthology.orchestra.run.smithi162.stdout: "max": 3758096384, 2024-03-31T23:37:44.441 INFO:teuthology.orchestra.run.smithi162.stdout: "get_started": 0, 2024-03-31T23:37:44.441 INFO:teuthology.orchestra.run.smithi162.stdout: "get": 0, 2024-03-31T23:37:44.441 INFO:teuthology.orchestra.run.smithi162.stdout: "get_sum": 0, 2024-03-31T23:37:44.441 INFO:teuthology.orchestra.run.smithi162.stdout: "get_or_fail_fail": 0, 2024-03-31T23:37:44.441 INFO:teuthology.orchestra.run.smithi162.stdout: "get_or_fail_success": 0, 2024-03-31T23:37:44.441 INFO:teuthology.orchestra.run.smithi162.stdout: "take": 0, 2024-03-31T23:37:44.441 INFO:teuthology.orchestra.run.smithi162.stdout: "take_sum": 0, 2024-03-31T23:37:44.441 INFO:teuthology.orchestra.run.smithi162.stdout: "put": 0, 2024-03-31T23:37:44.441 INFO:teuthology.orchestra.run.smithi162.stdout: "put_sum": 0, 2024-03-31T23:37:44.441 INFO:teuthology.orchestra.run.smithi162.stdout: "wait": { 2024-03-31T23:37:44.441 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-31T23:37:44.441 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-31T23:37:44.441 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-31T23:37:44.442 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-03-31T23:37:44.442 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-31T23:37:44.442 INFO:teuthology.orchestra.run.smithi162.stdout: "throttle-write_buf_throttle-0x55e96b16a640": { 2024-03-31T23:37:44.442 INFO:teuthology.orchestra.run.smithi162.stdout: "val": 0, 2024-03-31T23:37:44.442 INFO:teuthology.orchestra.run.smithi162.stdout: "max": 3758096384, 2024-03-31T23:37:44.442 INFO:teuthology.orchestra.run.smithi162.stdout: "get_started": 0, 2024-03-31T23:37:44.442 INFO:teuthology.orchestra.run.smithi162.stdout: "get": 0, 2024-03-31T23:37:44.442 INFO:teuthology.orchestra.run.smithi162.stdout: "get_sum": 0, 2024-03-31T23:37:44.442 INFO:teuthology.orchestra.run.smithi162.stdout: "get_or_fail_fail": 0, 2024-03-31T23:37:44.442 INFO:teuthology.orchestra.run.smithi162.stdout: "get_or_fail_success": 0, 2024-03-31T23:37:44.442 INFO:teuthology.orchestra.run.smithi162.stdout: "take": 0, 2024-03-31T23:37:44.442 INFO:teuthology.orchestra.run.smithi162.stdout: "take_sum": 0, 2024-03-31T23:37:44.442 INFO:teuthology.orchestra.run.smithi162.stdout: "put": 0, 2024-03-31T23:37:44.442 INFO:teuthology.orchestra.run.smithi162.stdout: "put_sum": 0, 2024-03-31T23:37:44.442 INFO:teuthology.orchestra.run.smithi162.stdout: "wait": { 2024-03-31T23:37:44.443 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-31T23:37:44.443 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-31T23:37:44.443 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-31T23:37:44.443 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-03-31T23:37:44.443 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-03-31T23:37:44.443 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-03-31T23:37:44.443 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:37:44 smithi142 ceph-mon[35271]: pgmap v826: 129 pgs: 129 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 1.5 KiB/s wr, 5 op/s 2024-03-31T23:37:44.611 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:37:44 smithi046 ceph-mon[32498]: pgmap v826: 129 pgs: 129 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 1.5 KiB/s wr, 5 op/s 2024-03-31T23:37:45.196 INFO:tasks.check_counter:Daemon mds.i mds.exported=0 2024-03-31T23:37:45.197 INFO:tasks.check_counter:Daemon mds.i mds.imported=0 2024-03-31T23:37:45.197 INFO:tasks.check_counter:Daemon mds.i mds_cache.dir_update=0 2024-03-31T23:37:45.197 INFO:tasks.check_counter:Daemon mds.i mds_cache.dir_update_receipt=0 2024-03-31T23:37:45.197 INFO:tasks.check_counter:Daemon mds.i mds.root_rsnaps=6 2024-03-31T23:37:45.197 INFO:tasks.check_counter:Daemon mds.i mds_server.req_mksnap_latency.avgcount=0 2024-03-31T23:37:45.197 INFO:tasks.check_counter:Daemon mds.i mds_server.req_rmsnap_latency.avgcount=0 2024-03-31T23:37:45.197 DEBUG:tasks.check_counter:Getting stats from l 2024-03-31T23:37:45.197 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:384f31f3d3aee7f8ba3f60846b8844c049165583 shell --fsid 6a9d8f08-efb2-11ee-b647-cb9ed24678a4 -- ceph daemon mds.l perf dump 2024-03-31T23:37:45.453 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/6a9d8f08-efb2-11ee-b647-cb9ed24678a4/mon.c/config 2024-03-31T23:37:46.443 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:37:46 smithi142 ceph-mon[35271]: pgmap v827: 129 pgs: 129 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 1.5 KiB/s wr, 7 op/s 2024-03-31T23:37:46.546 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:37:46 smithi162 ceph-mon[36237]: pgmap v827: 129 pgs: 129 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 1.5 KiB/s wr, 7 op/s 2024-03-31T23:37:46.611 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:37:46 smithi046 ceph-mon[32498]: pgmap v827: 129 pgs: 129 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 1.5 KiB/s wr, 7 op/s 2024-03-31T23:37:47.167 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-03-31T23:37:47.167 INFO:teuthology.orchestra.run.smithi162.stdout: "AsyncMessenger::Worker-0": { 2024-03-31T23:37:47.167 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_recv_messages": 0, 2024-03-31T23:37:47.168 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_send_messages": 2, 2024-03-31T23:37:47.168 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_recv_bytes": 0, 2024-03-31T23:37:47.168 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_send_bytes": 0, 2024-03-31T23:37:47.168 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_created_connections": 4, 2024-03-31T23:37:47.168 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_active_connections": 0, 2024-03-31T23:37:47.168 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_running_total_time": 0.000992936, 2024-03-31T23:37:47.168 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_running_send_time": 0.000000000, 2024-03-31T23:37:47.168 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_running_recv_time": 0.000187372, 2024-03-31T23:37:47.168 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-31T23:37:47.168 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_send_messages_queue_lat": { 2024-03-31T23:37:47.168 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-31T23:37:47.168 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-31T23:37:47.168 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-31T23:37:47.169 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-31T23:37:47.169 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_handle_ack_lat": { 2024-03-31T23:37:47.169 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-31T23:37:47.169 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-31T23:37:47.169 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-31T23:37:47.169 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-31T23:37:47.169 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-31T23:37:47.169 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_send_encrypted_bytes": 0 2024-03-31T23:37:47.169 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-31T23:37:47.169 INFO:teuthology.orchestra.run.smithi162.stdout: "AsyncMessenger::Worker-1": { 2024-03-31T23:37:47.169 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_recv_messages": 477, 2024-03-31T23:37:47.169 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_send_messages": 465, 2024-03-31T23:37:47.169 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_recv_bytes": 1512736, 2024-03-31T23:37:47.169 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_send_bytes": 287440, 2024-03-31T23:37:47.169 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_created_connections": 2, 2024-03-31T23:37:47.170 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_active_connections": 1, 2024-03-31T23:37:47.170 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_running_total_time": 0.056790913, 2024-03-31T23:37:47.170 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_running_send_time": 0.018393337, 2024-03-31T23:37:47.170 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_running_recv_time": 0.036825155, 2024-03-31T23:37:47.170 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_running_fast_dispatch_time": 0.008530170, 2024-03-31T23:37:47.170 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_send_messages_queue_lat": { 2024-03-31T23:37:47.170 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 465, 2024-03-31T23:37:47.170 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.012996532, 2024-03-31T23:37:47.170 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000027949 2024-03-31T23:37:47.170 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-31T23:37:47.170 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_handle_ack_lat": { 2024-03-31T23:37:47.170 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-31T23:37:47.170 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-31T23:37:47.170 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-31T23:37:47.171 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-31T23:37:47.171 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_recv_encrypted_bytes": 1512736, 2024-03-31T23:37:47.171 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_send_encrypted_bytes": 287440 2024-03-31T23:37:47.171 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-31T23:37:47.171 INFO:teuthology.orchestra.run.smithi162.stdout: "AsyncMessenger::Worker-2": { 2024-03-31T23:37:47.171 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_recv_messages": 3, 2024-03-31T23:37:47.171 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_send_messages": 355, 2024-03-31T23:37:47.171 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_recv_bytes": 480, 2024-03-31T23:37:47.171 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_send_bytes": 295808, 2024-03-31T23:37:47.171 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_created_connections": 6, 2024-03-31T23:37:47.171 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_active_connections": 3, 2024-03-31T23:37:47.171 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_running_total_time": 0.014847616, 2024-03-31T23:37:47.171 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_running_send_time": 0.010322308, 2024-03-31T23:37:47.171 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_running_recv_time": 0.001787822, 2024-03-31T23:37:47.171 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-31T23:37:47.171 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_send_messages_queue_lat": { 2024-03-31T23:37:47.172 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 352, 2024-03-31T23:37:47.172 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.022056435, 2024-03-31T23:37:47.172 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000062660 2024-03-31T23:37:47.172 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-31T23:37:47.172 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_handle_ack_lat": { 2024-03-31T23:37:47.172 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-31T23:37:47.172 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-31T23:37:47.172 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-31T23:37:47.172 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-31T23:37:47.172 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_recv_encrypted_bytes": 480, 2024-03-31T23:37:47.172 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_send_encrypted_bytes": 295808 2024-03-31T23:37:47.172 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-31T23:37:47.172 INFO:teuthology.orchestra.run.smithi162.stdout: "cct": { 2024-03-31T23:37:47.172 INFO:teuthology.orchestra.run.smithi162.stdout: "total_workers": 0, 2024-03-31T23:37:47.172 INFO:teuthology.orchestra.run.smithi162.stdout: "unhealthy_workers": 0 2024-03-31T23:37:47.173 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-31T23:37:47.173 INFO:teuthology.orchestra.run.smithi162.stdout: "mempool": { 2024-03-31T23:37:47.173 INFO:teuthology.orchestra.run.smithi162.stdout: "bloom_filter_bytes": 0, 2024-03-31T23:37:47.173 INFO:teuthology.orchestra.run.smithi162.stdout: "bloom_filter_items": 0, 2024-03-31T23:37:47.173 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_alloc_bytes": 0, 2024-03-31T23:37:47.173 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_alloc_items": 0, 2024-03-31T23:37:47.173 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_cache_data_bytes": 0, 2024-03-31T23:37:47.173 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_cache_data_items": 0, 2024-03-31T23:37:47.173 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-31T23:37:47.173 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_cache_onode_items": 0, 2024-03-31T23:37:47.173 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-31T23:37:47.173 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_cache_meta_items": 0, 2024-03-31T23:37:47.173 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_cache_other_bytes": 0, 2024-03-31T23:37:47.173 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_cache_other_items": 0, 2024-03-31T23:37:47.173 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-31T23:37:47.173 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_cache_buffer_items": 0, 2024-03-31T23:37:47.174 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_extent_bytes": 0, 2024-03-31T23:37:47.174 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_extent_items": 0, 2024-03-31T23:37:47.174 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_blob_bytes": 0, 2024-03-31T23:37:47.174 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_blob_items": 0, 2024-03-31T23:37:47.174 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-31T23:37:47.174 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_shared_blob_items": 0, 2024-03-31T23:37:47.174 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-31T23:37:47.174 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_inline_bl_items": 0, 2024-03-31T23:37:47.174 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_fsck_bytes": 0, 2024-03-31T23:37:47.174 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_fsck_items": 0, 2024-03-31T23:37:47.174 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_txc_bytes": 0, 2024-03-31T23:37:47.174 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_txc_items": 0, 2024-03-31T23:37:47.174 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-31T23:37:47.175 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_writing_deferred_items": 0, 2024-03-31T23:37:47.175 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_writing_bytes": 0, 2024-03-31T23:37:47.175 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_writing_items": 0, 2024-03-31T23:37:47.175 INFO:teuthology.orchestra.run.smithi162.stdout: "bluefs_bytes": 0, 2024-03-31T23:37:47.175 INFO:teuthology.orchestra.run.smithi162.stdout: "bluefs_items": 0, 2024-03-31T23:37:47.175 INFO:teuthology.orchestra.run.smithi162.stdout: "bluefs_file_reader_bytes": 0, 2024-03-31T23:37:47.175 INFO:teuthology.orchestra.run.smithi162.stdout: "bluefs_file_reader_items": 0, 2024-03-31T23:37:47.175 INFO:teuthology.orchestra.run.smithi162.stdout: "bluefs_file_writer_bytes": 0, 2024-03-31T23:37:47.175 INFO:teuthology.orchestra.run.smithi162.stdout: "bluefs_file_writer_items": 0, 2024-03-31T23:37:47.175 INFO:teuthology.orchestra.run.smithi162.stdout: "buffer_anon_bytes": 148714, 2024-03-31T23:37:47.175 INFO:teuthology.orchestra.run.smithi162.stdout: "buffer_anon_items": 30, 2024-03-31T23:37:47.175 INFO:teuthology.orchestra.run.smithi162.stdout: "buffer_meta_bytes": 0, 2024-03-31T23:37:47.175 INFO:teuthology.orchestra.run.smithi162.stdout: "buffer_meta_items": 0, 2024-03-31T23:37:47.175 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_bytes": 0, 2024-03-31T23:37:47.176 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_items": 0, 2024-03-31T23:37:47.176 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_mapbl_bytes": 0, 2024-03-31T23:37:47.176 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_mapbl_items": 0, 2024-03-31T23:37:47.176 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_pglog_bytes": 0, 2024-03-31T23:37:47.176 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_pglog_items": 0, 2024-03-31T23:37:47.176 INFO:teuthology.orchestra.run.smithi162.stdout: "osdmap_bytes": 0, 2024-03-31T23:37:47.176 INFO:teuthology.orchestra.run.smithi162.stdout: "osdmap_items": 0, 2024-03-31T23:37:47.176 INFO:teuthology.orchestra.run.smithi162.stdout: "osdmap_mapping_bytes": 0, 2024-03-31T23:37:47.176 INFO:teuthology.orchestra.run.smithi162.stdout: "osdmap_mapping_items": 0, 2024-03-31T23:37:47.176 INFO:teuthology.orchestra.run.smithi162.stdout: "pgmap_bytes": 0, 2024-03-31T23:37:47.176 INFO:teuthology.orchestra.run.smithi162.stdout: "pgmap_items": 0, 2024-03-31T23:37:47.176 INFO:teuthology.orchestra.run.smithi162.stdout: "mds_co_bytes": 928, 2024-03-31T23:37:47.176 INFO:teuthology.orchestra.run.smithi162.stdout: "mds_co_items": 2, 2024-03-31T23:37:47.176 INFO:teuthology.orchestra.run.smithi162.stdout: "unittest_1_bytes": 0, 2024-03-31T23:37:47.176 INFO:teuthology.orchestra.run.smithi162.stdout: "unittest_1_items": 0, 2024-03-31T23:37:47.177 INFO:teuthology.orchestra.run.smithi162.stdout: "unittest_2_bytes": 0, 2024-03-31T23:37:47.177 INFO:teuthology.orchestra.run.smithi162.stdout: "unittest_2_items": 0 2024-03-31T23:37:47.177 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-31T23:37:47.177 INFO:teuthology.orchestra.run.smithi162.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-31T23:37:47.177 INFO:teuthology.orchestra.run.smithi162.stdout: "val": 0, 2024-03-31T23:37:47.177 INFO:teuthology.orchestra.run.smithi162.stdout: "max": 104857600, 2024-03-31T23:37:47.177 INFO:teuthology.orchestra.run.smithi162.stdout: "get_started": 0, 2024-03-31T23:37:47.177 INFO:teuthology.orchestra.run.smithi162.stdout: "get": 476, 2024-03-31T23:37:47.177 INFO:teuthology.orchestra.run.smithi162.stdout: "get_sum": 1442915, 2024-03-31T23:37:47.177 INFO:teuthology.orchestra.run.smithi162.stdout: "get_or_fail_fail": 0, 2024-03-31T23:37:47.177 INFO:teuthology.orchestra.run.smithi162.stdout: "get_or_fail_success": 476, 2024-03-31T23:37:47.177 INFO:teuthology.orchestra.run.smithi162.stdout: "take": 0, 2024-03-31T23:37:47.177 INFO:teuthology.orchestra.run.smithi162.stdout: "take_sum": 0, 2024-03-31T23:37:47.177 INFO:teuthology.orchestra.run.smithi162.stdout: "put": 476, 2024-03-31T23:37:47.178 INFO:teuthology.orchestra.run.smithi162.stdout: "put_sum": 1442915, 2024-03-31T23:37:47.178 INFO:teuthology.orchestra.run.smithi162.stdout: "wait": { 2024-03-31T23:37:47.178 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-31T23:37:47.178 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-31T23:37:47.178 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-31T23:37:47.178 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-03-31T23:37:47.178 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-03-31T23:37:47.178 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-03-31T23:37:47.898 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.l 2024-03-31T23:37:47.898 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.l 2024-03-31T23:37:47.899 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update' not found on daemon mds.l 2024-03-31T23:37:47.899 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update_receipt' not found on daemon mds.l 2024-03-31T23:37:47.899 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.l 2024-03-31T23:37:47.899 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.l 2024-03-31T23:37:47.899 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.l 2024-03-31T23:37:47.899 DEBUG:teuthology.run_tasks:Unwinding manager ceph-fuse 2024-03-31T23:37:47.910 INFO:tasks.ceph_fuse:Unmounting ceph-fuse clients... 2024-03-31T23:37:47.910 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-03-31T23:37:47.910 DEBUG:teuthology.orchestra.run.smithi046:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-31T23:37:47.941 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-03-31T23:37:47.941 DEBUG:teuthology.orchestra.run.smithi046:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-31T23:37:48.000 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-31T23:37:48.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:48.159+0000 7fb5c678a640 1 -- 172.21.15.46:0/3023267916 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb5c01029c0 msgr2=0x7fb5c0102da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:37:48.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:48.159+0000 7fb5c678a640 1 --2- 172.21.15.46:0/3023267916 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb5c01029c0 0x7fb5c0102da0 secure :-1 s=READY pgs=1071 cs=0 l=1 rev1=1 crypto rx=0x7fb5b0009900 tx=0x7fb5b002bc30 comp rx=0 tx=0).stop 2024-03-31T23:37:48.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:48.159+0000 7fb5c678a640 1 -- 172.21.15.46:0/3023267916 shutdown_connections 2024-03-31T23:37:48.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:48.159+0000 7fb5c678a640 1 --2- 172.21.15.46:0/3023267916 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb5c010bd90 0x7fb5c010e230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:37:48.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:48.159+0000 7fb5c678a640 1 --2- 172.21.15.46:0/3023267916 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb5c01032e0 0x7fb5c010b7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:37:48.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:48.159+0000 7fb5c678a640 1 --2- 172.21.15.46:0/3023267916 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb5c01029c0 0x7fb5c0102da0 unknown :-1 s=CLOSED pgs=1071 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:37:48.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:48.159+0000 7fb5c678a640 1 -- 172.21.15.46:0/3023267916 >> 172.21.15.46:0/3023267916 conn(0x7fb5c0077750 msgr2=0x7fb5c0077b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:37:48.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:48.159+0000 7fb5c678a640 1 -- 172.21.15.46:0/3023267916 shutdown_connections 2024-03-31T23:37:48.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:48.159+0000 7fb5c678a640 1 -- 172.21.15.46:0/3023267916 wait complete. 2024-03-31T23:37:48.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:48.160+0000 7fb5c678a640 1 Processor -- start 2024-03-31T23:37:48.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:48.160+0000 7fb5c678a640 1 -- start start 2024-03-31T23:37:48.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:48.160+0000 7fb5c678a640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb5c01029c0 0x7fb5c0198000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:37:48.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:48.160+0000 7fb5c678a640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb5c01032e0 0x7fb5c0198540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:37:48.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:48.160+0000 7fb5c678a640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb5c010bd90 0x7fb5c019c870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:37:48.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:48.160+0000 7fb5c678a640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fb5c0111a40 con 0x7fb5c01029c0 2024-03-31T23:37:48.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:48.160+0000 7fb5c678a640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fb5c01118c0 con 0x7fb5c010bd90 2024-03-31T23:37:48.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:48.160+0000 7fb5c678a640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fb5c0111bc0 con 0x7fb5c01032e0 2024-03-31T23:37:48.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:48.160+0000 7fb5bffff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb5c01029c0 0x7fb5c0198000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:37:48.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:48.161+0000 7fb5bffff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb5c01029c0 0x7fb5c0198000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:35282/0 (socket says 172.21.15.46:35282) 2024-03-31T23:37:48.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:48.161+0000 7fb5bffff640 1 -- 172.21.15.46:0/2566314748 learned_addr learned my addr 172.21.15.46:0/2566314748 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:37:48.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:48.161+0000 7fb5c4d27640 1 --2- 172.21.15.46:0/2566314748 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb5c010bd90 0x7fb5c019c870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:37:48.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:48.161+0000 7fb5bf7fe640 1 --2- 172.21.15.46:0/2566314748 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb5c01032e0 0x7fb5c0198540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:37:48.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:48.161+0000 7fb5bffff640 1 -- 172.21.15.46:0/2566314748 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb5c01032e0 msgr2=0x7fb5c0198540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:37:48.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:48.161+0000 7fb5bffff640 1 --2- 172.21.15.46:0/2566314748 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb5c01032e0 0x7fb5c0198540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:37:48.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:48.161+0000 7fb5bffff640 1 -- 172.21.15.46:0/2566314748 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb5c010bd90 msgr2=0x7fb5c019c870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:37:48.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:48.161+0000 7fb5bffff640 1 --2- 172.21.15.46:0/2566314748 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb5c010bd90 0x7fb5c019c870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:37:48.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:48.161+0000 7fb5bffff640 1 -- 172.21.15.46:0/2566314748 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb5c019cfa0 con 0x7fb5c01029c0 2024-03-31T23:37:48.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:48.161+0000 7fb5c4d27640 1 --2- 172.21.15.46:0/2566314748 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb5c010bd90 0x7fb5c019c870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:37:48.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:48.161+0000 7fb5bffff640 1 --2- 172.21.15.46:0/2566314748 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb5c01029c0 0x7fb5c0198000 secure :-1 s=READY pgs=1072 cs=0 l=1 rev1=1 crypto rx=0x7fb5b0005940 tx=0x7fb5b002c5f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:37:48.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:48.161+0000 7fb5bd7fa640 1 -- 172.21.15.46:0/2566314748 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5b0043020 con 0x7fb5c01029c0 2024-03-31T23:37:48.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:48.161+0000 7fb5bd7fa640 1 -- 172.21.15.46:0/2566314748 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb5b002ca70 con 0x7fb5c01029c0 2024-03-31T23:37:48.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:48.161+0000 7fb5c678a640 1 -- 172.21.15.46:0/2566314748 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb5c019d230 con 0x7fb5c01029c0 2024-03-31T23:37:48.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:48.161+0000 7fb5bd7fa640 1 -- 172.21.15.46:0/2566314748 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5b0003a80 con 0x7fb5c01029c0 2024-03-31T23:37:48.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:48.161+0000 7fb5c678a640 1 -- 172.21.15.46:0/2566314748 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb5c01a4b20 con 0x7fb5c01029c0 2024-03-31T23:37:48.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:48.162+0000 7fb5c678a640 1 -- 172.21.15.46:0/2566314748 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb5c0068540 con 0x7fb5c01029c0 2024-03-31T23:37:48.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:48.163+0000 7fb5bd7fa640 1 -- 172.21.15.46:0/2566314748 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 25) ==== 137451+0+0 (secure 0 0 0) 0x7fb5b0046020 con 0x7fb5c01029c0 2024-03-31T23:37:48.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:48.164+0000 7fb5bd7fa640 1 --2- 172.21.15.46:0/2566314748 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fb5a80a2610 0x7fb5a80a4ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:37:48.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:48.164+0000 7fb5bf7fe640 1 --2- 172.21.15.46:0/2566314748 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fb5a80a2610 0x7fb5a80a4ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:37:48.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:48.164+0000 7fb5bd7fa640 1 -- 172.21.15.46:0/2566314748 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(139..139 src has 1..139) ==== 9554+0+0 (secure 0 0 0) 0x7fb5b00f0f10 con 0x7fb5c01029c0 2024-03-31T23:37:48.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:48.165+0000 7fb5bf7fe640 1 --2- 172.21.15.46:0/2566314748 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fb5a80a2610 0x7fb5a80a4ad0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fb5c010db00 tx=0x7fb5b4005f90 comp rx=0 tx=0).ready entity=mgr.15654 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:37:48.169 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:48.169+0000 7fb5bd7fa640 1 -- 172.21.15.46:0/2566314748 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fb5b004d020 con 0x7fb5c01029c0 2024-03-31T23:37:48.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:48.352+0000 7fb5c678a640 1 -- 172.21.15.46:0/2566314748 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fb5c0106e80 con 0x7fb5c01029c0 2024-03-31T23:37:48.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:48.354+0000 7fb5bd7fa640 1 -- 172.21.15.46:0/2566314748 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v139) ==== 74+0+25380 (secure 0 0 0) 0x7fb5b002cc20 con 0x7fb5c01029c0 2024-03-31T23:37:48.369 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-31T23:37:48.370 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":139,"fsid":"6a9d8f08-efb2-11ee-b647-cb9ed24678a4","created":"2024-03-31T23:01:15.463711+0000","modified":"2024-03-31T23:37:10.712908+0000","last_up_change":"2024-03-31T23:08:20.199888+0000","last_in_change":"2024-03-31T23:07:59.438035+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-31T23:04:32.823751+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-31T23:09:09.145711+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-31T23:09:10.148579+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"138","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":47,"snap_epoch":138,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-31T23:09:13.419519+0000","flags":8197,"flags_names":"hashpspool,ec_overwrites,selfmanaged_snaps","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"138","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":47,"snap_epoch":138,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"47429095-3f46-4ed7-b7c4-504464719815","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":1824772695},{"type":"v1","addr":"172.21.15.46:6803","nonce":1824772695}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":1824772695},{"type":"v1","addr":"172.21.15.46:6805","nonce":1824772695}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":1824772695},{"type":"v1","addr":"172.21.15.46:6809","nonce":1824772695}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":1824772695},{"type":"v1","addr":"172.21.15.46:6807","nonce":1824772695}]},"public_addr":"172.21.15.46:6803/1824772695","cluster_addr":"172.21.15.46:6805/1824772695","heartbeat_back_addr":"172.21.15.46:6809/1824772695","heartbeat_front_addr":"172.21.15.46:6807/1824772695","state":["exists","up"]},{"osd":1,"uuid":"be56f883-411e-4dbe-b50e-a158134df4a6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6800","nonce":973693922},{"type":"v1","addr":"172.21.15.142:6801","nonce":973693922}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6802","nonce":973693922},{"type":"v1","addr":"172.21.15.142:6803","nonce":973693922}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6806","nonce":973693922},{"type":"v1","addr":"172.21.15.142:6807","nonce":973693922}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6804","nonce":973693922},{"type":"v1","addr":"172.21.15.142:6805","nonce":973693922}]},"public_addr":"172.21.15.142:6801/973693922","cluster_addr":"172.21.15.142:6803/973693922","heartbeat_back_addr":"172.21.15.142:6807/973693922","heartbeat_front_addr":"172.21.15.142:6805/973693922","state":["exists","up"]},{"osd":2,"uuid":"c99a8794-ab19-4260-9a8d-4bc0bd2c8175","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6800","nonce":4084261813},{"type":"v1","addr":"172.21.15.162:6801","nonce":4084261813}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6802","nonce":4084261813},{"type":"v1","addr":"172.21.15.162:6803","nonce":4084261813}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6806","nonce":4084261813},{"type":"v1","addr":"172.21.15.162:6807","nonce":4084261813}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6804","nonce":4084261813},{"type":"v1","addr":"172.21.15.162:6805","nonce":4084261813}]},"public_addr":"172.21.15.162:6801/4084261813","cluster_addr":"172.21.15.162:6803/4084261813","heartbeat_back_addr":"172.21.15.162:6807/4084261813","heartbeat_front_addr":"172.21.15.162:6805/4084261813","state":["exists","up"]},{"osd":3,"uuid":"f877814e-9ef6-4872-a081-5609bb9ebc06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":3496317970},{"type":"v1","addr":"172.21.15.46:6811","nonce":3496317970}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":3496317970},{"type":"v1","addr":"172.21.15.46:6813","nonce":3496317970}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":3496317970},{"type":"v1","addr":"172.21.15.46:6817","nonce":3496317970}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":3496317970},{"type":"v1","addr":"172.21.15.46:6815","nonce":3496317970}]},"public_addr":"172.21.15.46:6811/3496317970","cluster_addr":"172.21.15.46:6813/3496317970","heartbeat_back_addr":"172.21.15.46:6817/3496317970","heartbeat_front_addr":"172.21.15.46:6815/3496317970","state":["exists","up"]},{"osd":4,"uuid":"beecec0a-8280-44fe-8f5f-8498a35eca8f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6808","nonce":2996266929},{"type":"v1","addr":"172.21.15.142:6809","nonce":2996266929}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6810","nonce":2996266929},{"type":"v1","addr":"172.21.15.142:6811","nonce":2996266929}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6814","nonce":2996266929},{"type":"v1","addr":"172.21.15.142:6815","nonce":2996266929}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6812","nonce":2996266929},{"type":"v1","addr":"172.21.15.142:6813","nonce":2996266929}]},"public_addr":"172.21.15.142:6809/2996266929","cluster_addr":"172.21.15.142:6811/2996266929","heartbeat_back_addr":"172.21.15.142:6815/2996266929","heartbeat_front_addr":"172.21.15.142:6813/2996266929","state":["exists","up"]},{"osd":5,"uuid":"9da7ff75-f658-45ea-a02f-1943fdda76b9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6808","nonce":914208293},{"type":"v1","addr":"172.21.15.162:6809","nonce":914208293}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6810","nonce":914208293},{"type":"v1","addr":"172.21.15.162:6811","nonce":914208293}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6814","nonce":914208293},{"type":"v1","addr":"172.21.15.162:6815","nonce":914208293}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6812","nonce":914208293},{"type":"v1","addr":"172.21.15.162:6813","nonce":914208293}]},"public_addr":"172.21.15.162:6809/914208293","cluster_addr":"172.21.15.162:6811/914208293","heartbeat_back_addr":"172.21.15.162:6815/914208293","heartbeat_front_addr":"172.21.15.162:6813/914208293","state":["exists","up"]},{"osd":6,"uuid":"8fa11224-cb26-4d3f-87d1-6a8f31de2a19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":56291966},{"type":"v1","addr":"172.21.15.46:6819","nonce":56291966}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":56291966},{"type":"v1","addr":"172.21.15.46:6821","nonce":56291966}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6824","nonce":56291966},{"type":"v1","addr":"172.21.15.46:6825","nonce":56291966}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":56291966},{"type":"v1","addr":"172.21.15.46:6823","nonce":56291966}]},"public_addr":"172.21.15.46:6819/56291966","cluster_addr":"172.21.15.46:6821/56291966","heartbeat_back_addr":"172.21.15.46:6825/56291966","heartbeat_front_addr":"172.21.15.46:6823/56291966","state":["exists","up"]},{"osd":7,"uuid":"0d0338b7-4d0a-4bb0-b152-8f5224871d9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6816","nonce":2146598446},{"type":"v1","addr":"172.21.15.142:6817","nonce":2146598446}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6818","nonce":2146598446},{"type":"v1","addr":"172.21.15.142:6819","nonce":2146598446}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6822","nonce":2146598446},{"type":"v1","addr":"172.21.15.142:6823","nonce":2146598446}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6820","nonce":2146598446},{"type":"v1","addr":"172.21.15.142:6821","nonce":2146598446}]},"public_addr":"172.21.15.142:6817/2146598446","cluster_addr":"172.21.15.142:6819/2146598446","heartbeat_back_addr":"172.21.15.142:6823/2146598446","heartbeat_front_addr":"172.21.15.142:6821/2146598446","state":["exists","up"]},{"osd":8,"uuid":"8b6f6037-892a-416c-babb-7552648d0b3a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6816","nonce":1173078318},{"type":"v1","addr":"172.21.15.162:6817","nonce":1173078318}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6818","nonce":1173078318},{"type":"v1","addr":"172.21.15.162:6819","nonce":1173078318}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6822","nonce":1173078318},{"type":"v1","addr":"172.21.15.162:6823","nonce":1173078318}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6820","nonce":1173078318},{"type":"v1","addr":"172.21.15.162:6821","nonce":1173078318}]},"public_addr":"172.21.15.162:6817/1173078318","cluster_addr":"172.21.15.162:6819/1173078318","heartbeat_back_addr":"172.21.15.162:6823/1173078318","heartbeat_front_addr":"172.21.15.162:6821/1173078318","state":["exists","up"]},{"osd":9,"uuid":"faccce25-6bf2-4220-8858-cda82b72be68","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":532708881},{"type":"v1","addr":"172.21.15.46:6827","nonce":532708881}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6828","nonce":532708881},{"type":"v1","addr":"172.21.15.46:6829","nonce":532708881}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":532708881},{"type":"v1","addr":"172.21.15.46:6833","nonce":532708881}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":532708881},{"type":"v1","addr":"172.21.15.46:6831","nonce":532708881}]},"public_addr":"172.21.15.46:6827/532708881","cluster_addr":"172.21.15.46:6829/532708881","heartbeat_back_addr":"172.21.15.46:6833/532708881","heartbeat_front_addr":"172.21.15.46:6831/532708881","state":["exists","up"]},{"osd":10,"uuid":"cdd356db-f9eb-4080-95ab-db5d8223e572","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6824","nonce":2845222366},{"type":"v1","addr":"172.21.15.142:6825","nonce":2845222366}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6826","nonce":2845222366},{"type":"v1","addr":"172.21.15.142:6827","nonce":2845222366}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6830","nonce":2845222366},{"type":"v1","addr":"172.21.15.142:6831","nonce":2845222366}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6828","nonce":2845222366},{"type":"v1","addr":"172.21.15.142:6829","nonce":2845222366}]},"public_addr":"172.21.15.142:6825/2845222366","cluster_addr":"172.21.15.142:6827/2845222366","heartbeat_back_addr":"172.21.15.142:6831/2845222366","heartbeat_front_addr":"172.21.15.142:6829/2845222366","state":["exists","up"]},{"osd":11,"uuid":"72104d1c-3ea2-45ad-bc0b-0500f146c06b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6824","nonce":1986323778},{"type":"v1","addr":"172.21.15.162:6825","nonce":1986323778}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6826","nonce":1986323778},{"type":"v1","addr":"172.21.15.162:6827","nonce":1986323778}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6830","nonce":1986323778},{"type":"v1","addr":"172.21.15.162:6831","nonce":1986323778}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6828","nonce":1986323778},{"type":"v1","addr":"172.21.15.162:6829","nonce":1986323778}]},"public_addr":"172.21.15.162:6825/1986323778","cluster_addr":"172.21.15.162:6827/1986323778","heartbeat_back_addr":"172.21.15.162:6831/1986323778","heartbeat_front_addr":"172.21.15.162:6829/1986323778","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:03:44.996331+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:04:07.994738+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:04:30.942255+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:04:55.588884+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:05:19.900600+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:05:43.807442+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:06:09.026976+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:06:34.586784+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:06:59.751605+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:07:26.144495+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:07:52.326909+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:08:17.423042+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.b","mappings":[{"from":8,"to":11}]},{"pgid":"3.6","mappings":[{"from":4,"to":7}]},{"pgid":"3.c","mappings":[{"from":3,"to":9}]},{"pgid":"3.1a","mappings":[{"from":5,"to":6}]},{"pgid":"3.1b","mappings":[{"from":4,"to":8}]},{"pgid":"3.1d","mappings":[{"from":4,"to":8}]},{"pgid":"3.2a","mappings":[{"from":3,"to":6}]},{"pgid":"3.33","mappings":[{"from":5,"to":9}]},{"pgid":"3.36","mappings":[{"from":4,"to":9}]},{"pgid":"4.6","mappings":[{"from":5,"to":8}]},{"pgid":"4.17","mappings":[{"from":9,"to":8}]},{"pgid":"4.18","mappings":[{"from":9,"to":8}]},{"pgid":"4.1b","mappings":[{"from":9,"to":10}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.46:0/249968062":"2024-04-01T23:11:42.678401+0000","172.21.15.46:6801/162289569":"2024-04-01T23:11:42.678401+0000","172.21.15.46:6800/162289569":"2024-04-01T23:11:42.678401+0000","172.21.15.46:0/34745898":"2024-04-01T23:11:42.678401+0000","172.21.15.46:0/3306820105":"2024-04-01T23:11:42.678401+0000","172.21.15.46:0/4021039857":"2024-04-01T23:02:08.493619+0000","172.21.15.46:6801/3681182647":"2024-04-01T23:02:08.493619+0000","172.21.15.46:0/4022214806":"2024-04-01T23:02:08.493619+0000","172.21.15.46:6800/1515528215":"2024-04-01T23:01:41.910162+0000","172.21.15.46:0/1146542737":"2024-04-01T23:11:42.678401+0000","172.21.15.46:6800/3681182647":"2024-04-01T23:02:08.493619+0000","172.21.15.46:0/143885707":"2024-04-01T23:01:41.910162+0000","172.21.15.46:6801/1515528215":"2024-04-01T23:01:41.910162+0000","172.21.15.46:0/1680389778":"2024-04-01T23:01:41.910162+0000","172.21.15.46:0/1782133966":"2024-04-01T23:01:41.910162+0000","172.21.15.46:0/3925209376":"2024-04-01T23:02:08.493619+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":3,"snaps":[{"begin":34,"length":1},{"begin":47,"length":1}]},{"pool":4,"snaps":[{"begin":34,"length":1},{"begin":47,"length":1}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-31T23:37:48.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:48.356+0000 7fb5c678a640 1 -- 172.21.15.46:0/2566314748 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fb5a80a2610 msgr2=0x7fb5a80a4ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:37:48.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:48.356+0000 7fb5c678a640 1 --2- 172.21.15.46:0/2566314748 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fb5a80a2610 0x7fb5a80a4ad0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fb5c010db00 tx=0x7fb5b4005f90 comp rx=0 tx=0).stop 2024-03-31T23:37:48.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:48.356+0000 7fb5c678a640 1 -- 172.21.15.46:0/2566314748 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb5c01029c0 msgr2=0x7fb5c0198000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:37:48.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:48.356+0000 7fb5c678a640 1 --2- 172.21.15.46:0/2566314748 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb5c01029c0 0x7fb5c0198000 secure :-1 s=READY pgs=1072 cs=0 l=1 rev1=1 crypto rx=0x7fb5b0005940 tx=0x7fb5b002c5f0 comp rx=0 tx=0).stop 2024-03-31T23:37:48.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:48.356+0000 7fb5c678a640 1 -- 172.21.15.46:0/2566314748 shutdown_connections 2024-03-31T23:37:48.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:48.356+0000 7fb5c678a640 1 --2- 172.21.15.46:0/2566314748 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fb5a80a2610 0x7fb5a80a4ad0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:37:48.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:48.356+0000 7fb5c678a640 1 --2- 172.21.15.46:0/2566314748 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb5c010bd90 0x7fb5c019c870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:37:48.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:48.356+0000 7fb5c678a640 1 --2- 172.21.15.46:0/2566314748 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb5c01032e0 0x7fb5c0198540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:37:48.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:48.356+0000 7fb5c678a640 1 --2- 172.21.15.46:0/2566314748 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb5c01029c0 0x7fb5c0198000 unknown :-1 s=CLOSED pgs=1072 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:37:48.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:48.356+0000 7fb5c678a640 1 -- 172.21.15.46:0/2566314748 >> 172.21.15.46:0/2566314748 conn(0x7fb5c0077750 msgr2=0x7fb5c00fd7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:37:48.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:48.356+0000 7fb5c678a640 1 -- 172.21.15.46:0/2566314748 shutdown_connections 2024-03-31T23:37:48.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:48.356+0000 7fb5c678a640 1 -- 172.21.15.46:0/2566314748 wait complete. 2024-03-31T23:37:48.372 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:37:48 smithi046 ceph-mon[32498]: pgmap v828: 129 pgs: 129 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 1.5 KiB/s wr, 5 op/s 2024-03-31T23:37:48.373 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-31T23:04:32.823751+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-31T23:09:09.145711+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-31T23:09:10.148579+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '138', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 47, 'snap_epoch': 138, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-31T23:09:13.419519+0000', 'flags': 8197, 'flags_names': 'hashpspool,ec_overwrites,selfmanaged_snaps', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '138', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 47, 'snap_epoch': 138, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-31T23:37:48.373 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-31T23:37:48.443 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:37:48 smithi142 ceph-mon[35271]: pgmap v828: 129 pgs: 129 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 1.5 KiB/s wr, 5 op/s 2024-03-31T23:37:48.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:48.478+0000 7fa023464640 1 -- 172.21.15.46:0/2603895142 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa01c1077b0 msgr2=0x7fa01c107b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:37:48.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:48.478+0000 7fa023464640 1 --2- 172.21.15.46:0/2603895142 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa01c1077b0 0x7fa01c107b90 secure :-1 s=READY pgs=1073 cs=0 l=1 rev1=1 crypto rx=0x7fa018009930 tx=0x7fa01802bc30 comp rx=0 tx=0).stop 2024-03-31T23:37:48.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:48.479+0000 7fa023464640 1 -- 172.21.15.46:0/2603895142 shutdown_connections 2024-03-31T23:37:48.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:48.479+0000 7fa023464640 1 --2- 172.21.15.46:0/2603895142 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa01c06a800 0x7fa01c10e5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:37:48.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:48.479+0000 7fa023464640 1 --2- 172.21.15.46:0/2603895142 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa01c069e40 0x7fa01c06a2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:37:48.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:48.479+0000 7fa023464640 1 --2- 172.21.15.46:0/2603895142 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa01c1077b0 0x7fa01c107b90 unknown :-1 s=CLOSED pgs=1073 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:37:48.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:48.479+0000 7fa023464640 1 -- 172.21.15.46:0/2603895142 >> 172.21.15.46:0/2603895142 conn(0x7fa01c0769d0 msgr2=0x7fa01c076de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:37:48.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:48.479+0000 7fa023464640 1 -- 172.21.15.46:0/2603895142 shutdown_connections 2024-03-31T23:37:48.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:48.479+0000 7fa023464640 1 -- 172.21.15.46:0/2603895142 wait complete. 2024-03-31T23:37:48.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:48.479+0000 7fa023464640 1 Processor -- start 2024-03-31T23:37:48.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:48.479+0000 7fa023464640 1 -- start start 2024-03-31T23:37:48.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:48.479+0000 7fa023464640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa01c069e40 0x7fa01c197ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:37:48.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:48.479+0000 7fa023464640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa01c06a800 0x7fa01c198430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:37:48.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:48.479+0000 7fa023464640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa01c1077b0 0x7fa01c19c7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:37:48.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:48.479+0000 7fa023464640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fa01c1119b0 con 0x7fa01c1077b0 2024-03-31T23:37:48.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:48.479+0000 7fa023464640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fa01c111830 con 0x7fa01c069e40 2024-03-31T23:37:48.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:48.479+0000 7fa023464640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fa01c111b30 con 0x7fa01c06a800 2024-03-31T23:37:48.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:48.480+0000 7fa021a01640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa01c1077b0 0x7fa01c19c7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:37:48.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:48.480+0000 7fa021a01640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa01c1077b0 0x7fa01c19c7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:35304/0 (socket says 172.21.15.46:35304) 2024-03-31T23:37:48.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:48.480+0000 7fa021a01640 1 -- 172.21.15.46:0/476271405 learned_addr learned my addr 172.21.15.46:0/476271405 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:37:48.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:48.480+0000 7fa0209ff640 1 --2- 172.21.15.46:0/476271405 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa01c06a800 0x7fa01c198430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:37:48.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:48.480+0000 7fa021200640 1 --2- 172.21.15.46:0/476271405 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa01c069e40 0x7fa01c197ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:37:48.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:48.480+0000 7fa021a01640 1 -- 172.21.15.46:0/476271405 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa01c06a800 msgr2=0x7fa01c198430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:37:48.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:48.480+0000 7fa021a01640 1 --2- 172.21.15.46:0/476271405 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa01c06a800 0x7fa01c198430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:37:48.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:48.480+0000 7fa021a01640 1 -- 172.21.15.46:0/476271405 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa01c069e40 msgr2=0x7fa01c197ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:37:48.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:48.480+0000 7fa021a01640 1 --2- 172.21.15.46:0/476271405 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa01c069e40 0x7fa01c197ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:37:48.484 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:48.480+0000 7fa021a01640 1 -- 172.21.15.46:0/476271405 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa01c19cee0 con 0x7fa01c1077b0 2024-03-31T23:37:48.484 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:48.480+0000 7fa021a01640 1 --2- 172.21.15.46:0/476271405 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa01c1077b0 0x7fa01c19c7b0 secure :-1 s=READY pgs=1074 cs=0 l=1 rev1=1 crypto rx=0x7fa014009fd0 tx=0x7fa01400b260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:37:48.484 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:48.480+0000 7fa0127fc640 1 -- 172.21.15.46:0/476271405 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa014017070 con 0x7fa01c1077b0 2024-03-31T23:37:48.484 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:48.480+0000 7fa0127fc640 1 -- 172.21.15.46:0/476271405 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa014003e20 con 0x7fa01c1077b0 2024-03-31T23:37:48.484 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:48.480+0000 7fa023464640 1 -- 172.21.15.46:0/476271405 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa01c19d1d0 con 0x7fa01c1077b0 2024-03-31T23:37:48.484 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:48.480+0000 7fa0127fc640 1 -- 172.21.15.46:0/476271405 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa014010460 con 0x7fa01c1077b0 2024-03-31T23:37:48.484 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:48.480+0000 7fa023464640 1 -- 172.21.15.46:0/476271405 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa01c1a4a60 con 0x7fa01c1077b0 2024-03-31T23:37:48.485 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:48.482+0000 7fa023464640 1 -- 172.21.15.46:0/476271405 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa01c102e30 con 0x7fa01c1077b0 2024-03-31T23:37:48.485 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:48.482+0000 7fa0127fc640 1 -- 172.21.15.46:0/476271405 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 25) ==== 137451+0+0 (secure 0 0 0) 0x7fa014003970 con 0x7fa01c1077b0 2024-03-31T23:37:48.485 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:48.483+0000 7fa0127fc640 1 --2- 172.21.15.46:0/476271405 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fa0080a2690 0x7fa0080a4b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:37:48.485 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:48.483+0000 7fa021200640 1 --2- 172.21.15.46:0/476271405 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fa0080a2690 0x7fa0080a4b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:37:48.485 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:48.483+0000 7fa0127fc640 1 -- 172.21.15.46:0/476271405 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(139..139 src has 1..139) ==== 9554+0+0 (secure 0 0 0) 0x7fa0140cec30 con 0x7fa01c1077b0 2024-03-31T23:37:48.485 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:48.484+0000 7fa021200640 1 --2- 172.21.15.46:0/476271405 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fa0080a2690 0x7fa0080a4b50 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fa018005ab0 tx=0x7fa018005fd0 comp rx=0 tx=0).ready entity=mgr.15654 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:37:48.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:48.488+0000 7fa0127fc640 1 -- 172.21.15.46:0/476271405 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fa01409b050 con 0x7fa01c1077b0 2024-03-31T23:37:48.545 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:37:48 smithi162 ceph-mon[36237]: pgmap v828: 129 pgs: 129 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 1.5 KiB/s wr, 5 op/s 2024-03-31T23:37:48.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:48.669+0000 7fa023464640 1 -- 172.21.15.46:0/476271405 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fa01c199240 con 0x7fa01c1077b0 2024-03-31T23:37:48.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:48.669+0000 7fa0127fc640 1 -- 172.21.15.46:0/476271405 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v139) ==== 93+0+10 (secure 0 0 0) 0x7fa01409f0f0 con 0x7fa01c1077b0 2024-03-31T23:37:48.670 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 1 2024-03-31T23:37:48.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:48.671+0000 7fa023464640 1 -- 172.21.15.46:0/476271405 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fa0080a2690 msgr2=0x7fa0080a4b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:37:48.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:48.671+0000 7fa023464640 1 --2- 172.21.15.46:0/476271405 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fa0080a2690 0x7fa0080a4b50 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fa018005ab0 tx=0x7fa018005fd0 comp rx=0 tx=0).stop 2024-03-31T23:37:48.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:48.671+0000 7fa023464640 1 -- 172.21.15.46:0/476271405 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa01c1077b0 msgr2=0x7fa01c19c7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:37:48.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:48.671+0000 7fa023464640 1 --2- 172.21.15.46:0/476271405 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa01c1077b0 0x7fa01c19c7b0 secure :-1 s=READY pgs=1074 cs=0 l=1 rev1=1 crypto rx=0x7fa014009fd0 tx=0x7fa01400b260 comp rx=0 tx=0).stop 2024-03-31T23:37:48.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:48.671+0000 7fa023464640 1 -- 172.21.15.46:0/476271405 shutdown_connections 2024-03-31T23:37:48.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:48.671+0000 7fa023464640 1 --2- 172.21.15.46:0/476271405 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fa0080a2690 0x7fa0080a4b50 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:37:48.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:48.671+0000 7fa023464640 1 --2- 172.21.15.46:0/476271405 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa01c1077b0 0x7fa01c19c7b0 unknown :-1 s=CLOSED pgs=1074 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:37:48.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:48.671+0000 7fa023464640 1 --2- 172.21.15.46:0/476271405 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa01c06a800 0x7fa01c198430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:37:48.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:48.671+0000 7fa023464640 1 --2- 172.21.15.46:0/476271405 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa01c069e40 0x7fa01c197ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:37:48.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:48.671+0000 7fa023464640 1 -- 172.21.15.46:0/476271405 >> 172.21.15.46:0/476271405 conn(0x7fa01c0769d0 msgr2=0x7fa01c0fe070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:37:48.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:48.671+0000 7fa023464640 1 -- 172.21.15.46:0/476271405 shutdown_connections 2024-03-31T23:37:48.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:48.671+0000 7fa023464640 1 -- 172.21.15.46:0/476271405 wait complete. 2024-03-31T23:37:48.683 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-31T23:37:48.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:48.789+0000 7f9ac36af640 1 -- 172.21.15.46:0/2108546934 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9abc1016f0 msgr2=0x7f9abc101b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:37:48.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:48.789+0000 7f9ac36af640 1 --2- 172.21.15.46:0/2108546934 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9abc1016f0 0x7f9abc101b70 secure :-1 s=READY pgs=1075 cs=0 l=1 rev1=1 crypto rx=0x7f9aac009930 tx=0x7f9aac02bc30 comp rx=0 tx=0).stop 2024-03-31T23:37:48.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:48.789+0000 7f9ac36af640 1 -- 172.21.15.46:0/2108546934 shutdown_connections 2024-03-31T23:37:48.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:48.789+0000 7f9ac36af640 1 --2- 172.21.15.46:0/2108546934 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9abc1020b0 0x7f9abc10e5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:37:48.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:48.789+0000 7f9ac36af640 1 --2- 172.21.15.46:0/2108546934 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9abc1016f0 0x7f9abc101b70 unknown :-1 s=CLOSED pgs=1075 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:37:48.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:48.789+0000 7f9ac36af640 1 --2- 172.21.15.46:0/2108546934 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9abc107700 0x7f9abc107ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:37:48.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:48.789+0000 7f9ac36af640 1 -- 172.21.15.46:0/2108546934 >> 172.21.15.46:0/2108546934 conn(0x7f9abc0fd180 msgr2=0x7f9abc0ff5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:37:48.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:48.789+0000 7f9ac36af640 1 -- 172.21.15.46:0/2108546934 shutdown_connections 2024-03-31T23:37:48.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:48.789+0000 7f9ac36af640 1 -- 172.21.15.46:0/2108546934 wait complete. 2024-03-31T23:37:48.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:48.789+0000 7f9ac36af640 1 Processor -- start 2024-03-31T23:37:48.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:48.789+0000 7f9ac36af640 1 -- start start 2024-03-31T23:37:48.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:48.789+0000 7f9ac36af640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9abc1016f0 0x7f9abc197fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:37:48.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:48.789+0000 7f9ac36af640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9abc1020b0 0x7f9abc1984e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:37:48.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:48.789+0000 7f9ac36af640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9abc107700 0x7f9abc19c860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:37:48.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:48.789+0000 7f9ac36af640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f9abc1119c0 con 0x7f9abc1020b0 2024-03-31T23:37:48.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:48.789+0000 7f9ac36af640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f9abc111840 con 0x7f9abc107700 2024-03-31T23:37:48.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:48.789+0000 7f9ac36af640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f9abc111b40 con 0x7f9abc1016f0 2024-03-31T23:37:48.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:48.790+0000 7f9ac0c4a640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9abc1020b0 0x7f9abc1984e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:37:48.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:48.790+0000 7f9ac0c4a640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9abc1020b0 0x7f9abc1984e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:35322/0 (socket says 172.21.15.46:35322) 2024-03-31T23:37:48.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:48.790+0000 7f9ac144b640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9abc1016f0 0x7f9abc197fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:37:48.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:48.790+0000 7f9ac0c4a640 1 -- 172.21.15.46:0/250742174 learned_addr learned my addr 172.21.15.46:0/250742174 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:37:48.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:48.790+0000 7f9ac1c4c640 1 --2- 172.21.15.46:0/250742174 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9abc107700 0x7f9abc19c860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:37:48.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:48.790+0000 7f9ac0c4a640 1 -- 172.21.15.46:0/250742174 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9abc1016f0 msgr2=0x7f9abc197fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:37:48.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:48.790+0000 7f9ac0c4a640 1 --2- 172.21.15.46:0/250742174 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9abc1016f0 0x7f9abc197fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:37:48.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:48.790+0000 7f9ac0c4a640 1 -- 172.21.15.46:0/250742174 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9abc107700 msgr2=0x7f9abc19c860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:37:48.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:48.790+0000 7f9ac0c4a640 1 --2- 172.21.15.46:0/250742174 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9abc107700 0x7f9abc19c860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:37:48.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:48.790+0000 7f9ac0c4a640 1 -- 172.21.15.46:0/250742174 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9abc19cf90 con 0x7f9abc1020b0 2024-03-31T23:37:48.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:48.790+0000 7f9ac144b640 1 --2- 172.21.15.46:0/250742174 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9abc1016f0 0x7f9abc197fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:37:48.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:48.790+0000 7f9ac1c4c640 1 --2- 172.21.15.46:0/250742174 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9abc107700 0x7f9abc19c860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:37:48.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:48.790+0000 7f9ac0c4a640 1 --2- 172.21.15.46:0/250742174 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9abc1020b0 0x7f9abc1984e0 secure :-1 s=READY pgs=1076 cs=0 l=1 rev1=1 crypto rx=0x7f9aac009230 tx=0x7f9aac02c5d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:37:48.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:48.790+0000 7f9ab27fc640 1 -- 172.21.15.46:0/250742174 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9aac043020 con 0x7f9abc1020b0 2024-03-31T23:37:48.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:48.790+0000 7f9ab27fc640 1 -- 172.21.15.46:0/250742174 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9aac02ca70 con 0x7f9abc1020b0 2024-03-31T23:37:48.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:48.790+0000 7f9ac36af640 1 -- 172.21.15.46:0/250742174 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9abc19d220 con 0x7f9abc1020b0 2024-03-31T23:37:48.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:48.790+0000 7f9ab27fc640 1 -- 172.21.15.46:0/250742174 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9aac003a50 con 0x7f9abc1020b0 2024-03-31T23:37:48.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:48.790+0000 7f9ac36af640 1 -- 172.21.15.46:0/250742174 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9abc1a4b10 con 0x7f9abc1020b0 2024-03-31T23:37:48.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:48.791+0000 7f9ac36af640 1 -- 172.21.15.46:0/250742174 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9abc102e30 con 0x7f9abc1020b0 2024-03-31T23:37:48.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:48.792+0000 7f9ab27fc640 1 -- 172.21.15.46:0/250742174 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 25) ==== 137451+0+0 (secure 0 0 0) 0x7f9aac046020 con 0x7f9abc1020b0 2024-03-31T23:37:48.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:48.793+0000 7f9ab27fc640 1 --2- 172.21.15.46:0/250742174 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f9aa80a2610 0x7f9aa80a4ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:37:48.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:48.793+0000 7f9ac144b640 1 --2- 172.21.15.46:0/250742174 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f9aa80a2610 0x7f9aa80a4ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:37:48.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:48.793+0000 7f9ab27fc640 1 -- 172.21.15.46:0/250742174 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(139..139 src has 1..139) ==== 9554+0+0 (secure 0 0 0) 0x7f9aac0f1210 con 0x7f9abc1020b0 2024-03-31T23:37:48.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:48.793+0000 7f9ac144b640 1 --2- 172.21.15.46:0/250742174 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f9aa80a2610 0x7f9aa80a4ad0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f9ab8009f40 tx=0x7f9ab8005f90 comp rx=0 tx=0).ready entity=mgr.15654 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:37:48.797 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:48.796+0000 7f9ab27fc640 1 -- 172.21.15.46:0/250742174 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f9aac04d020 con 0x7f9abc1020b0 2024-03-31T23:37:48.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:48.978+0000 7f9ac36af640 1 -- 172.21.15.46:0/250742174 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f9abc109c60 con 0x7f9abc1020b0 2024-03-31T23:37:48.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:48.978+0000 7f9ab27fc640 1 -- 172.21.15.46:0/250742174 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v139) ==== 104+0+11 (secure 0 0 0) 0x7f9aac0bdd30 con 0x7f9abc1020b0 2024-03-31T23:37:48.979 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 32 2024-03-31T23:37:48.981 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:48.980+0000 7f9ac36af640 1 -- 172.21.15.46:0/250742174 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f9aa80a2610 msgr2=0x7f9aa80a4ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:37:48.981 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:48.980+0000 7f9ac36af640 1 --2- 172.21.15.46:0/250742174 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f9aa80a2610 0x7f9aa80a4ad0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f9ab8009f40 tx=0x7f9ab8005f90 comp rx=0 tx=0).stop 2024-03-31T23:37:48.981 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:48.980+0000 7f9ac36af640 1 -- 172.21.15.46:0/250742174 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9abc1020b0 msgr2=0x7f9abc1984e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:37:48.981 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:48.980+0000 7f9ac36af640 1 --2- 172.21.15.46:0/250742174 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9abc1020b0 0x7f9abc1984e0 secure :-1 s=READY pgs=1076 cs=0 l=1 rev1=1 crypto rx=0x7f9aac009230 tx=0x7f9aac02c5d0 comp rx=0 tx=0).stop 2024-03-31T23:37:48.981 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:48.980+0000 7f9ac36af640 1 -- 172.21.15.46:0/250742174 shutdown_connections 2024-03-31T23:37:48.981 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:48.980+0000 7f9ac36af640 1 --2- 172.21.15.46:0/250742174 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f9aa80a2610 0x7f9aa80a4ad0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:37:48.981 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:48.980+0000 7f9ac36af640 1 --2- 172.21.15.46:0/250742174 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9abc107700 0x7f9abc19c860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:37:48.981 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:48.980+0000 7f9ac36af640 1 --2- 172.21.15.46:0/250742174 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9abc1020b0 0x7f9abc1984e0 unknown :-1 s=CLOSED pgs=1076 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:37:48.981 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:48.980+0000 7f9ac36af640 1 --2- 172.21.15.46:0/250742174 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9abc1016f0 0x7f9abc197fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:37:48.982 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:48.980+0000 7f9ac36af640 1 -- 172.21.15.46:0/250742174 >> 172.21.15.46:0/250742174 conn(0x7f9abc0fd180 msgr2=0x7f9abc0fdb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:37:48.982 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:48.980+0000 7f9ac36af640 1 -- 172.21.15.46:0/250742174 shutdown_connections 2024-03-31T23:37:48.982 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:48.980+0000 7f9ac36af640 1 -- 172.21.15.46:0/250742174 wait complete. 2024-03-31T23:37:48.992 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-31T23:37:49.100 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:49.097+0000 7f0a94556640 1 -- 172.21.15.46:0/1413684873 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0a8c1016c0 msgr2=0x7f0a8c101b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:37:49.100 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:49.097+0000 7f0a94556640 1 --2- 172.21.15.46:0/1413684873 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0a8c1016c0 0x7f0a8c101b40 secure :-1 s=READY pgs=1077 cs=0 l=1 rev1=1 crypto rx=0x7f0a7c009930 tx=0x7f0a7c02bc30 comp rx=0 tx=0).stop 2024-03-31T23:37:49.100 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:49.097+0000 7f0a94556640 1 -- 172.21.15.46:0/1413684873 shutdown_connections 2024-03-31T23:37:49.101 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:49.097+0000 7f0a94556640 1 --2- 172.21.15.46:0/1413684873 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0a8c102080 0x7f0a8c10e590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:37:49.101 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:49.097+0000 7f0a94556640 1 --2- 172.21.15.46:0/1413684873 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0a8c1016c0 0x7f0a8c101b40 unknown :-1 s=CLOSED pgs=1077 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:37:49.101 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:49.097+0000 7f0a94556640 1 --2- 172.21.15.46:0/1413684873 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0a8c1076d0 0x7f0a8c107ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:37:49.101 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:49.097+0000 7f0a94556640 1 -- 172.21.15.46:0/1413684873 >> 172.21.15.46:0/1413684873 conn(0x7f0a8c0fd1f0 msgr2=0x7f0a8c0ff610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:37:49.101 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:49.097+0000 7f0a94556640 1 -- 172.21.15.46:0/1413684873 shutdown_connections 2024-03-31T23:37:49.101 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:49.097+0000 7f0a94556640 1 -- 172.21.15.46:0/1413684873 wait complete. 2024-03-31T23:37:49.101 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:49.097+0000 7f0a94556640 1 Processor -- start 2024-03-31T23:37:49.101 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:49.097+0000 7f0a94556640 1 -- start start 2024-03-31T23:37:49.101 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:49.098+0000 7f0a94556640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0a8c1016c0 0x7f0a8c197f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:37:49.101 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:49.098+0000 7f0a94556640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0a8c102080 0x7f0a8c1984a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:37:49.101 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:49.098+0000 7f0a94556640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0a8c1076d0 0x7f0a8c19c820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:37:49.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:49.098+0000 7f0a94556640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f0a8c111990 con 0x7f0a8c1076d0 2024-03-31T23:37:49.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:49.098+0000 7f0a94556640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f0a8c111810 con 0x7f0a8c1016c0 2024-03-31T23:37:49.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:49.098+0000 7f0a94556640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f0a8c111b10 con 0x7f0a8c102080 2024-03-31T23:37:49.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:49.098+0000 7f0a92af3640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0a8c1076d0 0x7f0a8c19c820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:37:49.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:49.098+0000 7f0a92af3640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0a8c1076d0 0x7f0a8c19c820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:35328/0 (socket says 172.21.15.46:35328) 2024-03-31T23:37:49.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:49.098+0000 7f0a92af3640 1 -- 172.21.15.46:0/932435228 learned_addr learned my addr 172.21.15.46:0/932435228 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:37:49.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:49.098+0000 7f0a922f2640 1 --2- 172.21.15.46:0/932435228 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0a8c1016c0 0x7f0a8c197f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:37:49.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:49.098+0000 7f0a91af1640 1 --2- 172.21.15.46:0/932435228 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0a8c102080 0x7f0a8c1984a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:37:49.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:49.098+0000 7f0a92af3640 1 -- 172.21.15.46:0/932435228 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0a8c102080 msgr2=0x7f0a8c1984a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:37:49.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:49.098+0000 7f0a92af3640 1 --2- 172.21.15.46:0/932435228 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0a8c102080 0x7f0a8c1984a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:37:49.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:49.098+0000 7f0a92af3640 1 -- 172.21.15.46:0/932435228 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0a8c1016c0 msgr2=0x7f0a8c197f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:37:49.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:49.098+0000 7f0a92af3640 1 --2- 172.21.15.46:0/932435228 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0a8c1016c0 0x7f0a8c197f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:37:49.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:49.098+0000 7f0a92af3640 1 -- 172.21.15.46:0/932435228 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a8c19cf50 con 0x7f0a8c1076d0 2024-03-31T23:37:49.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:49.098+0000 7f0a92af3640 1 --2- 172.21.15.46:0/932435228 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0a8c1076d0 0x7f0a8c19c820 secure :-1 s=READY pgs=1078 cs=0 l=1 rev1=1 crypto rx=0x7f0a8400eb10 tx=0x7f0a8400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:37:49.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:49.098+0000 7f0a837fe640 1 -- 172.21.15.46:0/932435228 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a84004b20 con 0x7f0a8c1076d0 2024-03-31T23:37:49.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:49.098+0000 7f0a94556640 1 -- 172.21.15.46:0/932435228 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0a8c19d240 con 0x7f0a8c1076d0 2024-03-31T23:37:49.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:49.098+0000 7f0a837fe640 1 -- 172.21.15.46:0/932435228 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0a84004cc0 con 0x7f0a8c1076d0 2024-03-31T23:37:49.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:49.098+0000 7f0a837fe640 1 -- 172.21.15.46:0/932435228 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a84010690 con 0x7f0a8c1076d0 2024-03-31T23:37:49.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:49.098+0000 7f0a94556640 1 -- 172.21.15.46:0/932435228 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0a8c1a4ad0 con 0x7f0a8c1076d0 2024-03-31T23:37:49.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:49.099+0000 7f0a94556640 1 -- 172.21.15.46:0/932435228 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0a8c068990 con 0x7f0a8c1076d0 2024-03-31T23:37:49.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:49.101+0000 7f0a837fe640 1 -- 172.21.15.46:0/932435228 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 25) ==== 137451+0+0 (secure 0 0 0) 0x7f0a84005000 con 0x7f0a8c1076d0 2024-03-31T23:37:49.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:49.101+0000 7f0a837fe640 1 --2- 172.21.15.46:0/932435228 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f0a780a2610 0x7f0a780a4ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:37:49.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:49.101+0000 7f0a922f2640 1 --2- 172.21.15.46:0/932435228 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f0a780a2610 0x7f0a780a4ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:37:49.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:49.101+0000 7f0a837fe640 1 -- 172.21.15.46:0/932435228 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(139..139 src has 1..139) ==== 9554+0+0 (secure 0 0 0) 0x7f0a840cf6b0 con 0x7f0a8c1076d0 2024-03-31T23:37:49.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:49.102+0000 7f0a922f2640 1 --2- 172.21.15.46:0/932435228 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f0a780a2610 0x7f0a780a4ad0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f0a88009b50 tx=0x7f0a88005f90 comp rx=0 tx=0).ready entity=mgr.15654 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:37:49.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:49.105+0000 7f0a837fe640 1 -- 172.21.15.46:0/932435228 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f0a84053280 con 0x7f0a8c1076d0 2024-03-31T23:37:49.288 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:49.287+0000 7f0a94556640 1 -- 172.21.15.46:0/932435228 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f0a8c109c90 con 0x7f0a8c1076d0 2024-03-31T23:37:49.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:49.288+0000 7f0a837fe640 1 -- 172.21.15.46:0/932435228 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v139) ==== 100+0+11 (secure 0 0 0) 0x7f0a8409c120 con 0x7f0a8c1076d0 2024-03-31T23:37:49.289 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 64 2024-03-31T23:37:49.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:49.290+0000 7f0a94556640 1 -- 172.21.15.46:0/932435228 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f0a780a2610 msgr2=0x7f0a780a4ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:37:49.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:49.290+0000 7f0a94556640 1 --2- 172.21.15.46:0/932435228 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f0a780a2610 0x7f0a780a4ad0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f0a88009b50 tx=0x7f0a88005f90 comp rx=0 tx=0).stop 2024-03-31T23:37:49.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:49.290+0000 7f0a94556640 1 -- 172.21.15.46:0/932435228 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0a8c1076d0 msgr2=0x7f0a8c19c820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:37:49.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:49.290+0000 7f0a94556640 1 --2- 172.21.15.46:0/932435228 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0a8c1076d0 0x7f0a8c19c820 secure :-1 s=READY pgs=1078 cs=0 l=1 rev1=1 crypto rx=0x7f0a8400eb10 tx=0x7f0a8400c2d0 comp rx=0 tx=0).stop 2024-03-31T23:37:49.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:49.290+0000 7f0a94556640 1 -- 172.21.15.46:0/932435228 shutdown_connections 2024-03-31T23:37:49.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:49.290+0000 7f0a94556640 1 --2- 172.21.15.46:0/932435228 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f0a780a2610 0x7f0a780a4ad0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:37:49.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:49.290+0000 7f0a94556640 1 --2- 172.21.15.46:0/932435228 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0a8c1076d0 0x7f0a8c19c820 unknown :-1 s=CLOSED pgs=1078 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:37:49.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:49.290+0000 7f0a94556640 1 --2- 172.21.15.46:0/932435228 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0a8c102080 0x7f0a8c1984a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:37:49.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:49.290+0000 7f0a94556640 1 --2- 172.21.15.46:0/932435228 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0a8c1016c0 0x7f0a8c197f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:37:49.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:49.290+0000 7f0a94556640 1 -- 172.21.15.46:0/932435228 >> 172.21.15.46:0/932435228 conn(0x7f0a8c0fd1f0 msgr2=0x7f0a8c0fdb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:37:49.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:49.290+0000 7f0a94556640 1 -- 172.21.15.46:0/932435228 shutdown_connections 2024-03-31T23:37:49.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:49.290+0000 7f0a94556640 1 -- 172.21.15.46:0/932435228 wait complete. 2024-03-31T23:37:49.300 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:37:49 smithi046 ceph-mon[32498]: from='client.? 172.21.15.46:0/2566314748' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-31T23:37:49.300 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:37:49 smithi046 ceph-mon[32498]: from='client.? 172.21.15.46:0/476271405' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-31T23:37:49.300 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:37:49 smithi046 ceph-mon[32498]: from='client.? 172.21.15.46:0/250742174' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-31T23:37:49.301 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-31T23:37:49.409 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:49.406+0000 7facce8eb640 1 -- 172.21.15.46:0/1151799911 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7facc8100fc0 msgr2=0x7facc810d4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:37:49.409 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:49.406+0000 7facce8eb640 1 --2- 172.21.15.46:0/1151799911 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7facc8100fc0 0x7facc810d4d0 secure :-1 s=READY pgs=1079 cs=0 l=1 rev1=1 crypto rx=0x7facc0009cd0 tx=0x7facc002c0f0 comp rx=0 tx=0).stop 2024-03-31T23:37:49.409 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:49.406+0000 7facce8eb640 1 -- 172.21.15.46:0/1151799911 shutdown_connections 2024-03-31T23:37:49.409 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:49.406+0000 7facce8eb640 1 --2- 172.21.15.46:0/1151799911 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7facc8100fc0 0x7facc810d4d0 unknown :-1 s=CLOSED pgs=1079 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:37:49.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:49.406+0000 7facce8eb640 1 --2- 172.21.15.46:0/1151799911 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7facc8100600 0x7facc8100a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:37:49.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:49.406+0000 7facce8eb640 1 --2- 172.21.15.46:0/1151799911 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7facc81066a0 0x7facc8106a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:37:49.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:49.406+0000 7facce8eb640 1 -- 172.21.15.46:0/1151799911 >> 172.21.15.46:0/1151799911 conn(0x7facc80769d0 msgr2=0x7facc8076de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:37:49.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:49.406+0000 7facce8eb640 1 -- 172.21.15.46:0/1151799911 shutdown_connections 2024-03-31T23:37:49.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:49.406+0000 7facce8eb640 1 -- 172.21.15.46:0/1151799911 wait complete. 2024-03-31T23:37:49.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:49.406+0000 7facce8eb640 1 Processor -- start 2024-03-31T23:37:49.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:49.407+0000 7facce8eb640 1 -- start start 2024-03-31T23:37:49.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:49.407+0000 7facce8eb640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7facc8100600 0x7facc81022a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:37:49.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:49.407+0000 7facce8eb640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7facc8100fc0 0x7facc81027e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:37:49.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:49.407+0000 7facce8eb640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7facc81066a0 0x7facc8102d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:37:49.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:49.407+0000 7facce8eb640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7facc81108d0 con 0x7facc8100fc0 2024-03-31T23:37:49.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:49.407+0000 7facce8eb640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7facc8110750 con 0x7facc8100600 2024-03-31T23:37:49.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:49.407+0000 7facce8eb640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7facc8110a50 con 0x7facc81066a0 2024-03-31T23:37:49.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:49.407+0000 7facc77fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7facc8100fc0 0x7facc81027e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:37:49.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:49.407+0000 7facc7fff640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7facc8100600 0x7facc81022a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:37:49.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:49.407+0000 7facc77fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7facc8100fc0 0x7facc81027e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:35350/0 (socket says 172.21.15.46:35350) 2024-03-31T23:37:49.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:49.407+0000 7facc77fe640 1 -- 172.21.15.46:0/3315085659 learned_addr learned my addr 172.21.15.46:0/3315085659 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:37:49.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:49.407+0000 7faccce88640 1 --2- 172.21.15.46:0/3315085659 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7facc81066a0 0x7facc8102d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:37:49.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:49.407+0000 7facc77fe640 1 -- 172.21.15.46:0/3315085659 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7facc81066a0 msgr2=0x7facc8102d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:37:49.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:49.407+0000 7facc77fe640 1 --2- 172.21.15.46:0/3315085659 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7facc81066a0 0x7facc8102d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:37:49.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:49.407+0000 7facc77fe640 1 -- 172.21.15.46:0/3315085659 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7facc8100600 msgr2=0x7facc81022a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:37:49.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:49.407+0000 7facc77fe640 1 --2- 172.21.15.46:0/3315085659 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7facc8100600 0x7facc81022a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:37:49.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:49.407+0000 7facc77fe640 1 -- 172.21.15.46:0/3315085659 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7facc81a6380 con 0x7facc8100fc0 2024-03-31T23:37:49.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:49.407+0000 7facc7fff640 1 --2- 172.21.15.46:0/3315085659 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7facc8100600 0x7facc81022a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:37:49.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:49.408+0000 7facc77fe640 1 --2- 172.21.15.46:0/3315085659 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7facc8100fc0 0x7facc81027e0 secure :-1 s=READY pgs=1080 cs=0 l=1 rev1=1 crypto rx=0x7facbc00e9f0 tx=0x7facbc00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:37:49.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:49.408+0000 7facc57fa640 1 -- 172.21.15.46:0/3315085659 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7facbc00cd80 con 0x7facc8100fc0 2024-03-31T23:37:49.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:49.408+0000 7facce8eb640 1 -- 172.21.15.46:0/3315085659 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7facc81a6670 con 0x7facc8100fc0 2024-03-31T23:37:49.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:49.408+0000 7facc57fa640 1 -- 172.21.15.46:0/3315085659 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7facbc0054d0 con 0x7facc8100fc0 2024-03-31T23:37:49.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:49.408+0000 7facc57fa640 1 -- 172.21.15.46:0/3315085659 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7facbc010490 con 0x7facc8100fc0 2024-03-31T23:37:49.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:49.408+0000 7facce8eb640 1 -- 172.21.15.46:0/3315085659 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7facc81a6b70 con 0x7facc8100fc0 2024-03-31T23:37:49.414 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:49.409+0000 7facce8eb640 1 -- 172.21.15.46:0/3315085659 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7facc8101d40 con 0x7facc8100fc0 2024-03-31T23:37:49.414 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:49.410+0000 7facc57fa640 1 -- 172.21.15.46:0/3315085659 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 25) ==== 137451+0+0 (secure 0 0 0) 0x7facbc005020 con 0x7facc8100fc0 2024-03-31T23:37:49.414 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:49.410+0000 7facc57fa640 1 --2- 172.21.15.46:0/3315085659 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7facb00a2640 0x7facb00a4b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:37:49.415 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:49.411+0000 7facc7fff640 1 --2- 172.21.15.46:0/3315085659 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7facb00a2640 0x7facb00a4b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:37:49.415 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:49.411+0000 7facc57fa640 1 -- 172.21.15.46:0/3315085659 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(139..139 src has 1..139) ==== 9554+0+0 (secure 0 0 0) 0x7facbc0cf2d0 con 0x7facc8100fc0 2024-03-31T23:37:49.415 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:49.411+0000 7facc7fff640 1 --2- 172.21.15.46:0/3315085659 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7facb00a2640 0x7facb00a4b00 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7facc8101740 tx=0x7facb8005f90 comp rx=0 tx=0).ready entity=mgr.15654 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:37:49.416 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:49.415+0000 7facc57fa640 1 -- 172.21.15.46:0/3315085659 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7facbc09bce0 con 0x7facc8100fc0 2024-03-31T23:37:49.443 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:37:49 smithi142 ceph-mon[35271]: from='client.? 172.21.15.46:0/2566314748' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-31T23:37:49.443 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:37:49 smithi142 ceph-mon[35271]: from='client.? 172.21.15.46:0/476271405' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-31T23:37:49.443 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:37:49 smithi142 ceph-mon[35271]: from='client.? 172.21.15.46:0/250742174' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-31T23:37:49.545 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:37:49 smithi162 ceph-mon[36237]: from='client.? 172.21.15.46:0/2566314748' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-31T23:37:49.545 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:37:49 smithi162 ceph-mon[36237]: from='client.? 172.21.15.46:0/476271405' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-31T23:37:49.546 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:37:49 smithi162 ceph-mon[36237]: from='client.? 172.21.15.46:0/250742174' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-31T23:37:49.601 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:49.600+0000 7facce8eb640 1 -- 172.21.15.46:0/3315085659 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7facc81a6ed0 con 0x7facc8100fc0 2024-03-31T23:37:49.601 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:49.600+0000 7facc57fa640 1 -- 172.21.15.46:0/3315085659 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v139) ==== 103+0+11 (secure 0 0 0) 0x7facbc09b5e0 con 0x7facc8100fc0 2024-03-31T23:37:49.601 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 32 2024-03-31T23:37:49.603 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:49.602+0000 7facce8eb640 1 -- 172.21.15.46:0/3315085659 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7facb00a2640 msgr2=0x7facb00a4b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:37:49.604 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:49.602+0000 7facce8eb640 1 --2- 172.21.15.46:0/3315085659 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7facb00a2640 0x7facb00a4b00 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7facc8101740 tx=0x7facb8005f90 comp rx=0 tx=0).stop 2024-03-31T23:37:49.604 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:49.602+0000 7facce8eb640 1 -- 172.21.15.46:0/3315085659 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7facc8100fc0 msgr2=0x7facc81027e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:37:49.604 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:49.602+0000 7facce8eb640 1 --2- 172.21.15.46:0/3315085659 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7facc8100fc0 0x7facc81027e0 secure :-1 s=READY pgs=1080 cs=0 l=1 rev1=1 crypto rx=0x7facbc00e9f0 tx=0x7facbc00edf0 comp rx=0 tx=0).stop 2024-03-31T23:37:49.604 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:49.603+0000 7facce8eb640 1 -- 172.21.15.46:0/3315085659 shutdown_connections 2024-03-31T23:37:49.604 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:49.603+0000 7facce8eb640 1 --2- 172.21.15.46:0/3315085659 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7facb00a2640 0x7facb00a4b00 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:37:49.604 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:49.603+0000 7facce8eb640 1 --2- 172.21.15.46:0/3315085659 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7facc81066a0 0x7facc8102d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:37:49.604 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:49.603+0000 7facce8eb640 1 --2- 172.21.15.46:0/3315085659 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7facc8100fc0 0x7facc81027e0 unknown :-1 s=CLOSED pgs=1080 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:37:49.604 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:49.603+0000 7facce8eb640 1 --2- 172.21.15.46:0/3315085659 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7facc8100600 0x7facc81022a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:37:49.604 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:49.603+0000 7facce8eb640 1 -- 172.21.15.46:0/3315085659 >> 172.21.15.46:0/3315085659 conn(0x7facc80769d0 msgr2=0x7facc80ff650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:37:49.604 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:49.603+0000 7facce8eb640 1 -- 172.21.15.46:0/3315085659 shutdown_connections 2024-03-31T23:37:49.604 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:49.603+0000 7facce8eb640 1 -- 172.21.15.46:0/3315085659 wait complete. 2024-03-31T23:37:49.615 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-31T23:37:49.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:49.717+0000 7f100f302640 1 -- 172.21.15.46:0/2261984004 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1008106660 msgr2=0x7f1008106a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:37:49.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:49.717+0000 7f100f302640 1 --2- 172.21.15.46:0/2261984004 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1008106660 0x7f1008106a40 secure :-1 s=READY pgs=1081 cs=0 l=1 rev1=1 crypto rx=0x7f0ff8009930 tx=0x7f0ff802bc30 comp rx=0 tx=0).stop 2024-03-31T23:37:49.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:49.717+0000 7f100f302640 1 -- 172.21.15.46:0/2261984004 shutdown_connections 2024-03-31T23:37:49.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:49.717+0000 7f100f302640 1 --2- 172.21.15.46:0/2261984004 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1008100f80 0x7f100810d490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:37:49.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:49.717+0000 7f100f302640 1 --2- 172.21.15.46:0/2261984004 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f10081005c0 0x7f1008100a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:37:49.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:49.717+0000 7f100f302640 1 --2- 172.21.15.46:0/2261984004 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1008106660 0x7f1008106a40 unknown :-1 s=CLOSED pgs=1081 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:37:49.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:49.717+0000 7f100f302640 1 -- 172.21.15.46:0/2261984004 >> 172.21.15.46:0/2261984004 conn(0x7f1008077750 msgr2=0x7f1008077b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:37:49.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:49.717+0000 7f100f302640 1 -- 172.21.15.46:0/2261984004 shutdown_connections 2024-03-31T23:37:49.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:49.717+0000 7f100f302640 1 -- 172.21.15.46:0/2261984004 wait complete. 2024-03-31T23:37:49.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:49.718+0000 7f100f302640 1 Processor -- start 2024-03-31T23:37:49.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:49.718+0000 7f100f302640 1 -- start start 2024-03-31T23:37:49.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:49.718+0000 7f100f302640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f10081005c0 0x7f1008196db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:37:49.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:49.718+0000 7f100f302640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1008100f80 0x7f10081972f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:37:49.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:49.718+0000 7f100f302640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1008106660 0x7f100819b670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:37:49.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:49.718+0000 7f100f302640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f1008110890 con 0x7f1008106660 2024-03-31T23:37:49.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:49.718+0000 7f100f302640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f1008110710 con 0x7f1008100f80 2024-03-31T23:37:49.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:49.718+0000 7f100f302640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f1008110a10 con 0x7f10081005c0 2024-03-31T23:37:49.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:49.718+0000 7f100d89f640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1008106660 0x7f100819b670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:37:49.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:49.718+0000 7f100d89f640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1008106660 0x7f100819b670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:35370/0 (socket says 172.21.15.46:35370) 2024-03-31T23:37:49.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:49.718+0000 7f100d89f640 1 -- 172.21.15.46:0/1784070970 learned_addr learned my addr 172.21.15.46:0/1784070970 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:37:49.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:49.718+0000 7f100c89d640 1 --2- 172.21.15.46:0/1784070970 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1008100f80 0x7f10081972f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:37:49.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:49.718+0000 7f100d09e640 1 --2- 172.21.15.46:0/1784070970 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f10081005c0 0x7f1008196db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:37:49.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:49.718+0000 7f100c89d640 1 -- 172.21.15.46:0/1784070970 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f10081005c0 msgr2=0x7f1008196db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:37:49.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:49.718+0000 7f100c89d640 1 --2- 172.21.15.46:0/1784070970 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f10081005c0 0x7f1008196db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:37:49.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:49.718+0000 7f100c89d640 1 -- 172.21.15.46:0/1784070970 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1008106660 msgr2=0x7f100819b670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:37:49.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:49.718+0000 7f100c89d640 1 --2- 172.21.15.46:0/1784070970 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1008106660 0x7f100819b670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:37:49.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:49.718+0000 7f100c89d640 1 -- 172.21.15.46:0/1784070970 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f100819bda0 con 0x7f1008100f80 2024-03-31T23:37:49.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:49.718+0000 7f100d89f640 1 --2- 172.21.15.46:0/1784070970 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1008106660 0x7f100819b670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:37:49.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:49.718+0000 7f100d09e640 1 --2- 172.21.15.46:0/1784070970 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f10081005c0 0x7f1008196db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:37:49.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:49.718+0000 7f100c89d640 1 --2- 172.21.15.46:0/1784070970 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1008100f80 0x7f10081972f0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f0ffc00e9f0 tx=0x7f0ffc00eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:37:49.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:49.719+0000 7f10067fc640 1 -- 172.21.15.46:0/1784070970 <== mon.2 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ffc00ccc0 con 0x7f1008100f80 2024-03-31T23:37:49.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:49.719+0000 7f100f302640 1 -- 172.21.15.46:0/1784070970 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f100819c090 con 0x7f1008100f80 2024-03-31T23:37:49.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:49.719+0000 7f10067fc640 1 -- 172.21.15.46:0/1784070970 <== mon.2 v2:172.21.15.142:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0ffc005020 con 0x7f1008100f80 2024-03-31T23:37:49.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:49.719+0000 7f10067fc640 1 -- 172.21.15.46:0/1784070970 <== mon.2 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ffc004a00 con 0x7f1008100f80 2024-03-31T23:37:49.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:49.719+0000 7f100f302640 1 -- 172.21.15.46:0/1784070970 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f10081a3920 con 0x7f1008100f80 2024-03-31T23:37:49.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:49.720+0000 7f100f302640 1 -- 172.21.15.46:0/1784070970 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1008068540 con 0x7f1008100f80 2024-03-31T23:37:49.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:49.721+0000 7f10067fc640 1 -- 172.21.15.46:0/1784070970 <== mon.2 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 25) ==== 137451+0+0 (secure 0 0 0) 0x7f0ffc004ba0 con 0x7f1008100f80 2024-03-31T23:37:49.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:49.721+0000 7f10067fc640 1 --2- 172.21.15.46:0/1784070970 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f0ff00a2610 0x7f0ff00a4ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:37:49.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:49.722+0000 7f100d09e640 1 --2- 172.21.15.46:0/1784070970 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f0ff00a2610 0x7f0ff00a4ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:37:49.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:49.722+0000 7f10067fc640 1 -- 172.21.15.46:0/1784070970 <== mon.2 v2:172.21.15.142:3300/0 5 ==== osd_map(139..139 src has 1..139) ==== 9554+0+0 (secure 0 0 0) 0x7f0ffc0cef20 con 0x7f1008100f80 2024-03-31T23:37:49.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:49.722+0000 7f100d09e640 1 --2- 172.21.15.46:0/1784070970 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f0ff00a2610 0x7f0ff00a4ad0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f0ff8000c00 tx=0x7f0ff8005fd0 comp rx=0 tx=0).ready entity=mgr.15654 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:37:49.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:49.729+0000 7f10067fc640 1 -- 172.21.15.46:0/1784070970 <== mon.2 v2:172.21.15.142:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f0ffc09ba20 con 0x7f1008100f80 2024-03-31T23:37:49.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:49.926+0000 7f100f302640 1 -- 172.21.15.46:0/1784070970 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f1008198100 con 0x7f1008100f80 2024-03-31T23:37:49.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:49.926+0000 7f10067fc640 1 -- 172.21.15.46:0/1784070970 <== mon.2 v2:172.21.15.142:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 16 entries v139) ==== 81+0+940 (secure 0 0 0) 0x7f1008198100 con 0x7f1008100f80 2024-03-31T23:37:49.928 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:0/249968062 2024-04-01T23:11:42.678401+0000 2024-03-31T23:37:49.928 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6801/162289569 2024-04-01T23:11:42.678401+0000 2024-03-31T23:37:49.928 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6800/162289569 2024-04-01T23:11:42.678401+0000 2024-03-31T23:37:49.928 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:0/34745898 2024-04-01T23:11:42.678401+0000 2024-03-31T23:37:49.928 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:0/3306820105 2024-04-01T23:11:42.678401+0000 2024-03-31T23:37:49.928 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:0/4021039857 2024-04-01T23:02:08.493619+0000 2024-03-31T23:37:49.928 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6801/3681182647 2024-04-01T23:02:08.493619+0000 2024-03-31T23:37:49.928 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:0/4022214806 2024-04-01T23:02:08.493619+0000 2024-03-31T23:37:49.928 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6800/1515528215 2024-04-01T23:01:41.910162+0000 2024-03-31T23:37:49.928 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:0/1146542737 2024-04-01T23:11:42.678401+0000 2024-03-31T23:37:49.928 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6800/3681182647 2024-04-01T23:02:08.493619+0000 2024-03-31T23:37:49.928 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:0/143885707 2024-04-01T23:01:41.910162+0000 2024-03-31T23:37:49.929 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6801/1515528215 2024-04-01T23:01:41.910162+0000 2024-03-31T23:37:49.929 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:0/1680389778 2024-04-01T23:01:41.910162+0000 2024-03-31T23:37:49.929 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:0/1782133966 2024-04-01T23:01:41.910162+0000 2024-03-31T23:37:49.929 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:0/3925209376 2024-04-01T23:02:08.493619+0000 2024-03-31T23:37:49.930 INFO:teuthology.orchestra.run.smithi046.stderr:listed 16 entries 2024-03-31T23:37:49.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:49.928+0000 7f100f302640 1 -- 172.21.15.46:0/1784070970 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f0ff00a2610 msgr2=0x7f0ff00a4ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:37:49.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:49.928+0000 7f100f302640 1 --2- 172.21.15.46:0/1784070970 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f0ff00a2610 0x7f0ff00a4ad0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f0ff8000c00 tx=0x7f0ff8005fd0 comp rx=0 tx=0).stop 2024-03-31T23:37:49.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:49.928+0000 7f100f302640 1 -- 172.21.15.46:0/1784070970 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1008100f80 msgr2=0x7f10081972f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:37:49.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:49.928+0000 7f100f302640 1 --2- 172.21.15.46:0/1784070970 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1008100f80 0x7f10081972f0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f0ffc00e9f0 tx=0x7f0ffc00eeb0 comp rx=0 tx=0).stop 2024-03-31T23:37:49.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:49.928+0000 7f100f302640 1 -- 172.21.15.46:0/1784070970 shutdown_connections 2024-03-31T23:37:49.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:49.928+0000 7f100f302640 1 --2- 172.21.15.46:0/1784070970 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f0ff00a2610 0x7f0ff00a4ad0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:37:49.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:49.928+0000 7f100f302640 1 --2- 172.21.15.46:0/1784070970 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1008106660 0x7f100819b670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:37:49.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:49.928+0000 7f100f302640 1 --2- 172.21.15.46:0/1784070970 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1008100f80 0x7f10081972f0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:37:49.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:49.928+0000 7f100f302640 1 --2- 172.21.15.46:0/1784070970 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f10081005c0 0x7f1008196db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:37:49.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:49.928+0000 7f100f302640 1 -- 172.21.15.46:0/1784070970 >> 172.21.15.46:0/1784070970 conn(0x7f1008077750 msgr2=0x7f10080ff290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:37:49.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:49.928+0000 7f100f302640 1 -- 172.21.15.46:0/1784070970 shutdown_connections 2024-03-31T23:37:49.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:49.928+0000 7f100f302640 1 -- 172.21.15.46:0/1784070970 wait complete. 2024-03-31T23:37:49.940 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-03-31T23:37:49.940 DEBUG:teuthology.orchestra.run.smithi046:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-31T23:37:49.955 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-31T23:37:50.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:50.097+0000 7f654b9eb640 1 -- 172.21.15.46:0/761634781 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f65441009a0 msgr2=0x7f65441049e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:37:50.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:50.097+0000 7f654b9eb640 1 --2- 172.21.15.46:0/761634781 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f65441009a0 0x7f65441049e0 secure :-1 s=READY pgs=1082 cs=0 l=1 rev1=1 crypto rx=0x7f6534009900 tx=0x7f653402bc30 comp rx=0 tx=0).stop 2024-03-31T23:37:50.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:50.097+0000 7f654b9eb640 1 -- 172.21.15.46:0/761634781 shutdown_connections 2024-03-31T23:37:50.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:50.097+0000 7f654b9eb640 1 --2- 172.21.15.46:0/761634781 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f65441050c0 0x7f654410cc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:37:50.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:50.097+0000 7f654b9eb640 1 --2- 172.21.15.46:0/761634781 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f65441009a0 0x7f65441049e0 unknown :-1 s=CLOSED pgs=1082 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:37:50.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:50.097+0000 7f654b9eb640 1 --2- 172.21.15.46:0/761634781 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f65440ffff0 0x7f65441003d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:37:50.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:50.097+0000 7f654b9eb640 1 -- 172.21.15.46:0/761634781 >> 172.21.15.46:0/761634781 conn(0x7f6544076be0 msgr2=0x7f65440fdf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:37:50.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:50.097+0000 7f654b9eb640 1 -- 172.21.15.46:0/761634781 shutdown_connections 2024-03-31T23:37:50.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:50.097+0000 7f654b9eb640 1 -- 172.21.15.46:0/761634781 wait complete. 2024-03-31T23:37:50.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:50.098+0000 7f654b9eb640 1 Processor -- start 2024-03-31T23:37:50.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:50.098+0000 7f654b9eb640 1 -- start start 2024-03-31T23:37:50.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:50.098+0000 7f654b9eb640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f65440ffff0 0x7f6544197f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:37:50.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:50.098+0000 7f654b9eb640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f65441050c0 0x7f6544198470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:37:50.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:50.098+0000 7f654b9eb640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f654419c7f0 0x7f654419cca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:37:50.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:50.098+0000 7f654b9eb640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f65441119b0 con 0x7f654419c7f0 2024-03-31T23:37:50.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:50.098+0000 7f654b9eb640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f6544111830 con 0x7f65441050c0 2024-03-31T23:37:50.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:50.098+0000 7f654b9eb640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f6544111b30 con 0x7f65440ffff0 2024-03-31T23:37:50.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:50.098+0000 7f6549f88640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f654419c7f0 0x7f654419cca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:37:50.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:50.098+0000 7f6549f88640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f654419c7f0 0x7f654419cca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:35388/0 (socket says 172.21.15.46:35388) 2024-03-31T23:37:50.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:50.098+0000 7f6549f88640 1 -- 172.21.15.46:0/2715317897 learned_addr learned my addr 172.21.15.46:0/2715317897 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:37:50.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:50.098+0000 7f6549787640 1 --2- 172.21.15.46:0/2715317897 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f65440ffff0 0x7f6544197f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:37:50.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:50.098+0000 7f6548f86640 1 --2- 172.21.15.46:0/2715317897 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f65441050c0 0x7f6544198470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:37:50.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:50.098+0000 7f6549f88640 1 -- 172.21.15.46:0/2715317897 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f65440ffff0 msgr2=0x7f6544197f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:37:50.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:50.098+0000 7f6549f88640 1 --2- 172.21.15.46:0/2715317897 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f65440ffff0 0x7f6544197f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:37:50.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:50.098+0000 7f6549f88640 1 -- 172.21.15.46:0/2715317897 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f65441050c0 msgr2=0x7f6544198470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:37:50.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:50.098+0000 7f6549f88640 1 --2- 172.21.15.46:0/2715317897 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f65441050c0 0x7f6544198470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:37:50.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:50.098+0000 7f6549f88640 1 -- 172.21.15.46:0/2715317897 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f654419d2e0 con 0x7f654419c7f0 2024-03-31T23:37:50.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:50.098+0000 7f6549787640 1 --2- 172.21.15.46:0/2715317897 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f65440ffff0 0x7f6544197f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:37:50.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:50.099+0000 7f6549f88640 1 --2- 172.21.15.46:0/2715317897 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f654419c7f0 0x7f654419cca0 secure :-1 s=READY pgs=1083 cs=0 l=1 rev1=1 crypto rx=0x7f653c00eb10 tx=0x7f653c00c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:37:50.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:50.099+0000 7f653a7fc640 1 -- 172.21.15.46:0/2715317897 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f653c004b20 con 0x7f654419c7f0 2024-03-31T23:37:50.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:50.099+0000 7f654b9eb640 1 -- 172.21.15.46:0/2715317897 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f654419d5d0 con 0x7f654419c7f0 2024-03-31T23:37:50.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:50.099+0000 7f653a7fc640 1 -- 172.21.15.46:0/2715317897 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f653c004cc0 con 0x7f654419c7f0 2024-03-31T23:37:50.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:50.099+0000 7f653a7fc640 1 -- 172.21.15.46:0/2715317897 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f653c010690 con 0x7f654419c7f0 2024-03-31T23:37:50.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:50.099+0000 7f654b9eb640 1 -- 172.21.15.46:0/2715317897 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f65441a4d40 con 0x7f654419c7f0 2024-03-31T23:37:50.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:50.100+0000 7f654b9eb640 1 -- 172.21.15.46:0/2715317897 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6544068990 con 0x7f654419c7f0 2024-03-31T23:37:50.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:50.101+0000 7f653a7fc640 1 -- 172.21.15.46:0/2715317897 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 25) ==== 137451+0+0 (secure 0 0 0) 0x7f653c005020 con 0x7f654419c7f0 2024-03-31T23:37:50.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:50.101+0000 7f653a7fc640 1 --2- 172.21.15.46:0/2715317897 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f65300a2610 0x7f65300a4ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:37:50.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:50.101+0000 7f6549787640 1 --2- 172.21.15.46:0/2715317897 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f65300a2610 0x7f65300a4ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:37:50.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:50.101+0000 7f653a7fc640 1 -- 172.21.15.46:0/2715317897 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(139..139 src has 1..139) ==== 9554+0+0 (secure 0 0 0) 0x7f653c0cf460 con 0x7f654419c7f0 2024-03-31T23:37:50.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:50.102+0000 7f6549787640 1 --2- 172.21.15.46:0/2715317897 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f65300a2610 0x7f65300a4ad0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f654000e4a0 tx=0x7f6540010040 comp rx=0 tx=0).ready entity=mgr.15654 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:37:50.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:50.106+0000 7f653a7fc640 1 -- 172.21.15.46:0/2715317897 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f653c09bed0 con 0x7f654419c7f0 2024-03-31T23:37:50.294 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:50.294+0000 7f654b9eb640 1 -- 172.21.15.46:0/2715317897 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f65441009a0 con 0x7f654419c7f0 2024-03-31T23:37:50.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:50.295+0000 7f653a7fc640 1 -- 172.21.15.46:0/2715317897 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v139) ==== 74+0+25380 (secure 0 0 0) 0x7f653c0bb020 con 0x7f654419c7f0 2024-03-31T23:37:50.297 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-31T23:37:50.297 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":139,"fsid":"6a9d8f08-efb2-11ee-b647-cb9ed24678a4","created":"2024-03-31T23:01:15.463711+0000","modified":"2024-03-31T23:37:10.712908+0000","last_up_change":"2024-03-31T23:08:20.199888+0000","last_in_change":"2024-03-31T23:07:59.438035+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-31T23:04:32.823751+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-31T23:09:09.145711+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-31T23:09:10.148579+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"138","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":47,"snap_epoch":138,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-31T23:09:13.419519+0000","flags":8197,"flags_names":"hashpspool,ec_overwrites,selfmanaged_snaps","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"138","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":47,"snap_epoch":138,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"47429095-3f46-4ed7-b7c4-504464719815","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":1824772695},{"type":"v1","addr":"172.21.15.46:6803","nonce":1824772695}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":1824772695},{"type":"v1","addr":"172.21.15.46:6805","nonce":1824772695}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":1824772695},{"type":"v1","addr":"172.21.15.46:6809","nonce":1824772695}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":1824772695},{"type":"v1","addr":"172.21.15.46:6807","nonce":1824772695}]},"public_addr":"172.21.15.46:6803/1824772695","cluster_addr":"172.21.15.46:6805/1824772695","heartbeat_back_addr":"172.21.15.46:6809/1824772695","heartbeat_front_addr":"172.21.15.46:6807/1824772695","state":["exists","up"]},{"osd":1,"uuid":"be56f883-411e-4dbe-b50e-a158134df4a6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6800","nonce":973693922},{"type":"v1","addr":"172.21.15.142:6801","nonce":973693922}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6802","nonce":973693922},{"type":"v1","addr":"172.21.15.142:6803","nonce":973693922}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6806","nonce":973693922},{"type":"v1","addr":"172.21.15.142:6807","nonce":973693922}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6804","nonce":973693922},{"type":"v1","addr":"172.21.15.142:6805","nonce":973693922}]},"public_addr":"172.21.15.142:6801/973693922","cluster_addr":"172.21.15.142:6803/973693922","heartbeat_back_addr":"172.21.15.142:6807/973693922","heartbeat_front_addr":"172.21.15.142:6805/973693922","state":["exists","up"]},{"osd":2,"uuid":"c99a8794-ab19-4260-9a8d-4bc0bd2c8175","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6800","nonce":4084261813},{"type":"v1","addr":"172.21.15.162:6801","nonce":4084261813}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6802","nonce":4084261813},{"type":"v1","addr":"172.21.15.162:6803","nonce":4084261813}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6806","nonce":4084261813},{"type":"v1","addr":"172.21.15.162:6807","nonce":4084261813}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6804","nonce":4084261813},{"type":"v1","addr":"172.21.15.162:6805","nonce":4084261813}]},"public_addr":"172.21.15.162:6801/4084261813","cluster_addr":"172.21.15.162:6803/4084261813","heartbeat_back_addr":"172.21.15.162:6807/4084261813","heartbeat_front_addr":"172.21.15.162:6805/4084261813","state":["exists","up"]},{"osd":3,"uuid":"f877814e-9ef6-4872-a081-5609bb9ebc06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":3496317970},{"type":"v1","addr":"172.21.15.46:6811","nonce":3496317970}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":3496317970},{"type":"v1","addr":"172.21.15.46:6813","nonce":3496317970}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":3496317970},{"type":"v1","addr":"172.21.15.46:6817","nonce":3496317970}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":3496317970},{"type":"v1","addr":"172.21.15.46:6815","nonce":3496317970}]},"public_addr":"172.21.15.46:6811/3496317970","cluster_addr":"172.21.15.46:6813/3496317970","heartbeat_back_addr":"172.21.15.46:6817/3496317970","heartbeat_front_addr":"172.21.15.46:6815/3496317970","state":["exists","up"]},{"osd":4,"uuid":"beecec0a-8280-44fe-8f5f-8498a35eca8f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6808","nonce":2996266929},{"type":"v1","addr":"172.21.15.142:6809","nonce":2996266929}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6810","nonce":2996266929},{"type":"v1","addr":"172.21.15.142:6811","nonce":2996266929}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6814","nonce":2996266929},{"type":"v1","addr":"172.21.15.142:6815","nonce":2996266929}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6812","nonce":2996266929},{"type":"v1","addr":"172.21.15.142:6813","nonce":2996266929}]},"public_addr":"172.21.15.142:6809/2996266929","cluster_addr":"172.21.15.142:6811/2996266929","heartbeat_back_addr":"172.21.15.142:6815/2996266929","heartbeat_front_addr":"172.21.15.142:6813/2996266929","state":["exists","up"]},{"osd":5,"uuid":"9da7ff75-f658-45ea-a02f-1943fdda76b9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6808","nonce":914208293},{"type":"v1","addr":"172.21.15.162:6809","nonce":914208293}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6810","nonce":914208293},{"type":"v1","addr":"172.21.15.162:6811","nonce":914208293}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6814","nonce":914208293},{"type":"v1","addr":"172.21.15.162:6815","nonce":914208293}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6812","nonce":914208293},{"type":"v1","addr":"172.21.15.162:6813","nonce":914208293}]},"public_addr":"172.21.15.162:6809/914208293","cluster_addr":"172.21.15.162:6811/914208293","heartbeat_back_addr":"172.21.15.162:6815/914208293","heartbeat_front_addr":"172.21.15.162:6813/914208293","state":["exists","up"]},{"osd":6,"uuid":"8fa11224-cb26-4d3f-87d1-6a8f31de2a19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":56291966},{"type":"v1","addr":"172.21.15.46:6819","nonce":56291966}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":56291966},{"type":"v1","addr":"172.21.15.46:6821","nonce":56291966}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6824","nonce":56291966},{"type":"v1","addr":"172.21.15.46:6825","nonce":56291966}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":56291966},{"type":"v1","addr":"172.21.15.46:6823","nonce":56291966}]},"public_addr":"172.21.15.46:6819/56291966","cluster_addr":"172.21.15.46:6821/56291966","heartbeat_back_addr":"172.21.15.46:6825/56291966","heartbeat_front_addr":"172.21.15.46:6823/56291966","state":["exists","up"]},{"osd":7,"uuid":"0d0338b7-4d0a-4bb0-b152-8f5224871d9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6816","nonce":2146598446},{"type":"v1","addr":"172.21.15.142:6817","nonce":2146598446}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6818","nonce":2146598446},{"type":"v1","addr":"172.21.15.142:6819","nonce":2146598446}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6822","nonce":2146598446},{"type":"v1","addr":"172.21.15.142:6823","nonce":2146598446}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6820","nonce":2146598446},{"type":"v1","addr":"172.21.15.142:6821","nonce":2146598446}]},"public_addr":"172.21.15.142:6817/2146598446","cluster_addr":"172.21.15.142:6819/2146598446","heartbeat_back_addr":"172.21.15.142:6823/2146598446","heartbeat_front_addr":"172.21.15.142:6821/2146598446","state":["exists","up"]},{"osd":8,"uuid":"8b6f6037-892a-416c-babb-7552648d0b3a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6816","nonce":1173078318},{"type":"v1","addr":"172.21.15.162:6817","nonce":1173078318}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6818","nonce":1173078318},{"type":"v1","addr":"172.21.15.162:6819","nonce":1173078318}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6822","nonce":1173078318},{"type":"v1","addr":"172.21.15.162:6823","nonce":1173078318}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6820","nonce":1173078318},{"type":"v1","addr":"172.21.15.162:6821","nonce":1173078318}]},"public_addr":"172.21.15.162:6817/1173078318","cluster_addr":"172.21.15.162:6819/1173078318","heartbeat_back_addr":"172.21.15.162:6823/1173078318","heartbeat_front_addr":"172.21.15.162:6821/1173078318","state":["exists","up"]},{"osd":9,"uuid":"faccce25-6bf2-4220-8858-cda82b72be68","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":532708881},{"type":"v1","addr":"172.21.15.46:6827","nonce":532708881}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6828","nonce":532708881},{"type":"v1","addr":"172.21.15.46:6829","nonce":532708881}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":532708881},{"type":"v1","addr":"172.21.15.46:6833","nonce":532708881}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":532708881},{"type":"v1","addr":"172.21.15.46:6831","nonce":532708881}]},"public_addr":"172.21.15.46:6827/532708881","cluster_addr":"172.21.15.46:6829/532708881","heartbeat_back_addr":"172.21.15.46:6833/532708881","heartbeat_front_addr":"172.21.15.46:6831/532708881","state":["exists","up"]},{"osd":10,"uuid":"cdd356db-f9eb-4080-95ab-db5d8223e572","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6824","nonce":2845222366},{"type":"v1","addr":"172.21.15.142:6825","nonce":2845222366}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6826","nonce":2845222366},{"type":"v1","addr":"172.21.15.142:6827","nonce":2845222366}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6830","nonce":2845222366},{"type":"v1","addr":"172.21.15.142:6831","nonce":2845222366}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6828","nonce":2845222366},{"type":"v1","addr":"172.21.15.142:6829","nonce":2845222366}]},"public_addr":"172.21.15.142:6825/2845222366","cluster_addr":"172.21.15.142:6827/2845222366","heartbeat_back_addr":"172.21.15.142:6831/2845222366","heartbeat_front_addr":"172.21.15.142:6829/2845222366","state":["exists","up"]},{"osd":11,"uuid":"72104d1c-3ea2-45ad-bc0b-0500f146c06b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6824","nonce":1986323778},{"type":"v1","addr":"172.21.15.162:6825","nonce":1986323778}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6826","nonce":1986323778},{"type":"v1","addr":"172.21.15.162:6827","nonce":1986323778}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6830","nonce":1986323778},{"type":"v1","addr":"172.21.15.162:6831","nonce":1986323778}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6828","nonce":1986323778},{"type":"v1","addr":"172.21.15.162:6829","nonce":1986323778}]},"public_addr":"172.21.15.162:6825/1986323778","cluster_addr":"172.21.15.162:6827/1986323778","heartbeat_back_addr":"172.21.15.162:6831/1986323778","heartbeat_front_addr":"172.21.15.162:6829/1986323778","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:03:44.996331+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:04:07.994738+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:04:30.942255+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:04:55.588884+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:05:19.900600+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:05:43.807442+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:06:09.026976+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:06:34.586784+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:06:59.751605+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:07:26.144495+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:07:52.326909+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T23:08:17.423042+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.b","mappings":[{"from":8,"to":11}]},{"pgid":"3.6","mappings":[{"from":4,"to":7}]},{"pgid":"3.c","mappings":[{"from":3,"to":9}]},{"pgid":"3.1a","mappings":[{"from":5,"to":6}]},{"pgid":"3.1b","mappings":[{"from":4,"to":8}]},{"pgid":"3.1d","mappings":[{"from":4,"to":8}]},{"pgid":"3.2a","mappings":[{"from":3,"to":6}]},{"pgid":"3.33","mappings":[{"from":5,"to":9}]},{"pgid":"3.36","mappings":[{"from":4,"to":9}]},{"pgid":"4.6","mappings":[{"from":5,"to":8}]},{"pgid":"4.17","mappings":[{"from":9,"to":8}]},{"pgid":"4.18","mappings":[{"from":9,"to":8}]},{"pgid":"4.1b","mappings":[{"from":9,"to":10}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.46:0/249968062":"2024-04-01T23:11:42.678401+0000","172.21.15.46:6801/162289569":"2024-04-01T23:11:42.678401+0000","172.21.15.46:6800/162289569":"2024-04-01T23:11:42.678401+0000","172.21.15.46:0/34745898":"2024-04-01T23:11:42.678401+0000","172.21.15.46:0/3306820105":"2024-04-01T23:11:42.678401+0000","172.21.15.46:0/4021039857":"2024-04-01T23:02:08.493619+0000","172.21.15.46:6801/3681182647":"2024-04-01T23:02:08.493619+0000","172.21.15.46:0/4022214806":"2024-04-01T23:02:08.493619+0000","172.21.15.46:6800/1515528215":"2024-04-01T23:01:41.910162+0000","172.21.15.46:0/1146542737":"2024-04-01T23:11:42.678401+0000","172.21.15.46:6800/3681182647":"2024-04-01T23:02:08.493619+0000","172.21.15.46:0/143885707":"2024-04-01T23:01:41.910162+0000","172.21.15.46:6801/1515528215":"2024-04-01T23:01:41.910162+0000","172.21.15.46:0/1680389778":"2024-04-01T23:01:41.910162+0000","172.21.15.46:0/1782133966":"2024-04-01T23:01:41.910162+0000","172.21.15.46:0/3925209376":"2024-04-01T23:02:08.493619+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":3,"snaps":[{"begin":34,"length":1},{"begin":47,"length":1}]},{"pool":4,"snaps":[{"begin":34,"length":1},{"begin":47,"length":1}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-31T23:37:50.298 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:50.297+0000 7f654b9eb640 1 -- 172.21.15.46:0/2715317897 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f65300a2610 msgr2=0x7f65300a4ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:37:50.298 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:50.297+0000 7f654b9eb640 1 --2- 172.21.15.46:0/2715317897 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f65300a2610 0x7f65300a4ad0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f654000e4a0 tx=0x7f6540010040 comp rx=0 tx=0).stop 2024-03-31T23:37:50.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:50.297+0000 7f654b9eb640 1 -- 172.21.15.46:0/2715317897 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f654419c7f0 msgr2=0x7f654419cca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:37:50.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:50.297+0000 7f654b9eb640 1 --2- 172.21.15.46:0/2715317897 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f654419c7f0 0x7f654419cca0 secure :-1 s=READY pgs=1083 cs=0 l=1 rev1=1 crypto rx=0x7f653c00eb10 tx=0x7f653c00c2d0 comp rx=0 tx=0).stop 2024-03-31T23:37:50.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:50.297+0000 7f654b9eb640 1 -- 172.21.15.46:0/2715317897 shutdown_connections 2024-03-31T23:37:50.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:50.297+0000 7f654b9eb640 1 --2- 172.21.15.46:0/2715317897 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f65300a2610 0x7f65300a4ad0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:37:50.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:50.297+0000 7f654b9eb640 1 --2- 172.21.15.46:0/2715317897 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f654419c7f0 0x7f654419cca0 unknown :-1 s=CLOSED pgs=1083 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:37:50.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:50.297+0000 7f654b9eb640 1 --2- 172.21.15.46:0/2715317897 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f65441050c0 0x7f6544198470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:37:50.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:50.297+0000 7f654b9eb640 1 --2- 172.21.15.46:0/2715317897 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f65440ffff0 0x7f6544197f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:37:50.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:50.297+0000 7f654b9eb640 1 -- 172.21.15.46:0/2715317897 >> 172.21.15.46:0/2715317897 conn(0x7f6544076be0 msgr2=0x7f65441033c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:37:50.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:50.297+0000 7f654b9eb640 1 -- 172.21.15.46:0/2715317897 shutdown_connections 2024-03-31T23:37:50.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:50.297+0000 7f654b9eb640 1 -- 172.21.15.46:0/2715317897 wait complete. 2024-03-31T23:37:50.308 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:37:50 smithi046 ceph-mon[32498]: pgmap v829: 129 pgs: 129 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 2.3 KiB/s wr, 7 op/s 2024-03-31T23:37:50.308 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:37:50 smithi046 ceph-mon[32498]: from='client.? 172.21.15.46:0/932435228' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-31T23:37:50.308 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:37:50 smithi046 ceph-mon[32498]: from='client.? 172.21.15.46:0/3315085659' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-31T23:37:50.308 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:37:50 smithi046 ceph-mon[32498]: from='client.? 172.21.15.46:0/1784070970' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-31T23:37:50.309 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-31T23:04:32.823751+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-31T23:09:09.145711+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-31T23:09:10.148579+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '138', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 47, 'snap_epoch': 138, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-31T23:09:13.419519+0000', 'flags': 8197, 'flags_names': 'hashpspool,ec_overwrites,selfmanaged_snaps', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '138', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 47, 'snap_epoch': 138, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-31T23:37:50.310 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-31T23:37:50.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:50.414+0000 7f1c22007640 1 -- 172.21.15.46:0/1288489304 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1c1c1008a0 msgr2=0x7f1c1c100d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:37:50.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:50.414+0000 7f1c22007640 1 --2- 172.21.15.46:0/1288489304 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1c1c1008a0 0x7f1c1c100d00 secure :-1 s=READY pgs=1084 cs=0 l=1 rev1=1 crypto rx=0x7f1c10009900 tx=0x7f1c1002bc30 comp rx=0 tx=0).stop 2024-03-31T23:37:50.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:50.414+0000 7f1c22007640 1 -- 172.21.15.46:0/1288489304 shutdown_connections 2024-03-31T23:37:50.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:50.414+0000 7f1c22007640 1 --2- 172.21.15.46:0/1288489304 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1c1c101240 0x7f1c1c10d430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:37:50.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:50.414+0000 7f1c22007640 1 --2- 172.21.15.46:0/1288489304 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1c1c1008a0 0x7f1c1c100d00 unknown :-1 s=CLOSED pgs=1084 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:37:50.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:50.414+0000 7f1c22007640 1 --2- 172.21.15.46:0/1288489304 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1c1c1066e0 0x7f1c1c106ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:37:50.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:50.414+0000 7f1c22007640 1 -- 172.21.15.46:0/1288489304 >> 172.21.15.46:0/1288489304 conn(0x7f1c1c0773e0 msgr2=0x7f1c1c0777f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:37:50.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:50.414+0000 7f1c22007640 1 -- 172.21.15.46:0/1288489304 shutdown_connections 2024-03-31T23:37:50.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:50.414+0000 7f1c22007640 1 -- 172.21.15.46:0/1288489304 wait complete. 2024-03-31T23:37:50.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:50.415+0000 7f1c22007640 1 Processor -- start 2024-03-31T23:37:50.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:50.415+0000 7f1c22007640 1 -- start start 2024-03-31T23:37:50.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:50.415+0000 7f1c22007640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1c1c1008a0 0x7f1c1c197e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:37:50.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:50.415+0000 7f1c22007640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1c1c101240 0x7f1c1c1983a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:37:50.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:50.415+0000 7f1c22007640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1c1c1066e0 0x7f1c1c19c720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:37:50.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:50.415+0000 7f1c22007640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f1c1c110870 con 0x7f1c1c1066e0 2024-03-31T23:37:50.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:50.415+0000 7f1c22007640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f1c1c1106f0 con 0x7f1c1c1008a0 2024-03-31T23:37:50.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:50.415+0000 7f1c22007640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f1c1c1109f0 con 0x7f1c1c101240 2024-03-31T23:37:50.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:50.415+0000 7f1c1bfff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1c1c1066e0 0x7f1c1c19c720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:37:50.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:50.415+0000 7f1c1b7fe640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1c1c1008a0 0x7f1c1c197e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:37:50.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:50.416+0000 7f1c1b7fe640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1c1c1008a0 0x7f1c1c197e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.46:46610/0 (socket says 172.21.15.46:46610) 2024-03-31T23:37:50.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:50.416+0000 7f1c1b7fe640 1 -- 172.21.15.46:0/2395087672 learned_addr learned my addr 172.21.15.46:0/2395087672 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:37:50.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:50.415+0000 7f1c1bfff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1c1c1066e0 0x7f1c1c19c720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:35406/0 (socket says 172.21.15.46:35406) 2024-03-31T23:37:50.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:50.416+0000 7f1c1affd640 1 --2- 172.21.15.46:0/2395087672 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1c1c101240 0x7f1c1c1983a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:37:50.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:50.416+0000 7f1c1bfff640 1 -- 172.21.15.46:0/2395087672 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1c1c101240 msgr2=0x7f1c1c1983a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:37:50.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:50.416+0000 7f1c1bfff640 1 --2- 172.21.15.46:0/2395087672 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1c1c101240 0x7f1c1c1983a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:37:50.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:50.416+0000 7f1c1bfff640 1 -- 172.21.15.46:0/2395087672 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1c1c1008a0 msgr2=0x7f1c1c197e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:37:50.420 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:50.416+0000 7f1c1bfff640 1 --2- 172.21.15.46:0/2395087672 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1c1c1008a0 0x7f1c1c197e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:37:50.420 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:50.416+0000 7f1c1bfff640 1 -- 172.21.15.46:0/2395087672 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c1c19ce50 con 0x7f1c1c1066e0 2024-03-31T23:37:50.420 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:50.416+0000 7f1c1affd640 1 --2- 172.21.15.46:0/2395087672 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1c1c101240 0x7f1c1c1983a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:37:50.420 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:50.416+0000 7f1c1b7fe640 1 --2- 172.21.15.46:0/2395087672 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1c1c1008a0 0x7f1c1c197e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T23:37:50.420 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:50.416+0000 7f1c1bfff640 1 --2- 172.21.15.46:0/2395087672 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1c1c1066e0 0x7f1c1c19c720 secure :-1 s=READY pgs=1085 cs=0 l=1 rev1=1 crypto rx=0x7f1c14009fd0 tx=0x7f1c1400ef10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:37:50.420 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:50.416+0000 7f1c18ff9640 1 -- 172.21.15.46:0/2395087672 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c1400ce20 con 0x7f1c1c1066e0 2024-03-31T23:37:50.420 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:50.416+0000 7f1c18ff9640 1 -- 172.21.15.46:0/2395087672 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1c14005020 con 0x7f1c1c1066e0 2024-03-31T23:37:50.420 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:50.416+0000 7f1c22007640 1 -- 172.21.15.46:0/2395087672 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1c1c19d140 con 0x7f1c1c1066e0 2024-03-31T23:37:50.420 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:50.416+0000 7f1c18ff9640 1 -- 172.21.15.46:0/2395087672 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c14010690 con 0x7f1c1c1066e0 2024-03-31T23:37:50.420 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:50.416+0000 7f1c22007640 1 -- 172.21.15.46:0/2395087672 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1c1c1a49d0 con 0x7f1c1c1066e0 2024-03-31T23:37:50.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:50.417+0000 7f1c22007640 1 -- 172.21.15.46:0/2395087672 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1c1c068990 con 0x7f1c1c1066e0 2024-03-31T23:37:50.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:50.418+0000 7f1c18ff9640 1 -- 172.21.15.46:0/2395087672 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 25) ==== 137451+0+0 (secure 0 0 0) 0x7f1c140051d0 con 0x7f1c1c1066e0 2024-03-31T23:37:50.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:50.419+0000 7f1c18ff9640 1 --2- 172.21.15.46:0/2395087672 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f1c040a2690 0x7f1c040a4b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:37:50.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:50.419+0000 7f1c18ff9640 1 -- 172.21.15.46:0/2395087672 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(139..139 src has 1..139) ==== 9554+0+0 (secure 0 0 0) 0x7f1c140cf560 con 0x7f1c1c1066e0 2024-03-31T23:37:50.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:50.419+0000 7f1c1b7fe640 1 --2- 172.21.15.46:0/2395087672 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f1c040a2690 0x7f1c040a4b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:37:50.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:50.419+0000 7f1c1b7fe640 1 --2- 172.21.15.46:0/2395087672 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f1c040a2690 0x7f1c040a4b50 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f1c0c011550 tx=0x7f1c0c005f90 comp rx=0 tx=0).ready entity=mgr.15654 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:37:50.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:50.422+0000 7f1c18ff9640 1 -- 172.21.15.46:0/2395087672 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f1c1409bfd0 con 0x7f1c1c1066e0 2024-03-31T23:37:50.443 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:37:50 smithi142 ceph-mon[35271]: pgmap v829: 129 pgs: 129 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 2.3 KiB/s wr, 7 op/s 2024-03-31T23:37:50.443 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:37:50 smithi142 ceph-mon[35271]: from='client.? 172.21.15.46:0/932435228' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-31T23:37:50.443 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:37:50 smithi142 ceph-mon[35271]: from='client.? 172.21.15.46:0/3315085659' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-31T23:37:50.443 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:37:50 smithi142 ceph-mon[35271]: from='client.? 172.21.15.46:0/1784070970' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-31T23:37:50.545 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:37:50 smithi162 ceph-mon[36237]: pgmap v829: 129 pgs: 129 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 2.3 KiB/s wr, 7 op/s 2024-03-31T23:37:50.545 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:37:50 smithi162 ceph-mon[36237]: from='client.? 172.21.15.46:0/932435228' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-31T23:37:50.546 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:37:50 smithi162 ceph-mon[36237]: from='client.? 172.21.15.46:0/3315085659' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-31T23:37:50.546 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:37:50 smithi162 ceph-mon[36237]: from='client.? 172.21.15.46:0/1784070970' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-31T23:37:50.605 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:50.604+0000 7f1c22007640 1 -- 172.21.15.46:0/2395087672 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f1c1c1991b0 con 0x7f1c1c1066e0 2024-03-31T23:37:50.605 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:50.604+0000 7f1c18ff9640 1 -- 172.21.15.46:0/2395087672 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v139) ==== 93+0+10 (secure 0 0 0) 0x7f1c1409b8d0 con 0x7f1c1c1066e0 2024-03-31T23:37:50.605 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 1 2024-03-31T23:37:50.607 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:50.606+0000 7f1c22007640 1 -- 172.21.15.46:0/2395087672 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f1c040a2690 msgr2=0x7f1c040a4b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:37:50.607 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:50.606+0000 7f1c22007640 1 --2- 172.21.15.46:0/2395087672 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f1c040a2690 0x7f1c040a4b50 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f1c0c011550 tx=0x7f1c0c005f90 comp rx=0 tx=0).stop 2024-03-31T23:37:50.607 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:50.606+0000 7f1c22007640 1 -- 172.21.15.46:0/2395087672 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1c1c1066e0 msgr2=0x7f1c1c19c720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:37:50.607 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:50.606+0000 7f1c22007640 1 --2- 172.21.15.46:0/2395087672 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1c1c1066e0 0x7f1c1c19c720 secure :-1 s=READY pgs=1085 cs=0 l=1 rev1=1 crypto rx=0x7f1c14009fd0 tx=0x7f1c1400ef10 comp rx=0 tx=0).stop 2024-03-31T23:37:50.607 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:50.606+0000 7f1c22007640 1 -- 172.21.15.46:0/2395087672 shutdown_connections 2024-03-31T23:37:50.607 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:50.606+0000 7f1c22007640 1 --2- 172.21.15.46:0/2395087672 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f1c040a2690 0x7f1c040a4b50 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:37:50.607 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:50.606+0000 7f1c22007640 1 --2- 172.21.15.46:0/2395087672 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1c1c1066e0 0x7f1c1c19c720 unknown :-1 s=CLOSED pgs=1085 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:37:50.607 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:50.606+0000 7f1c22007640 1 --2- 172.21.15.46:0/2395087672 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1c1c101240 0x7f1c1c1983a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:37:50.607 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:50.606+0000 7f1c22007640 1 --2- 172.21.15.46:0/2395087672 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1c1c1008a0 0x7f1c1c197e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:37:50.608 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:50.606+0000 7f1c22007640 1 -- 172.21.15.46:0/2395087672 >> 172.21.15.46:0/2395087672 conn(0x7f1c1c0773e0 msgr2=0x7f1c1c06a030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:37:50.608 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:50.606+0000 7f1c22007640 1 -- 172.21.15.46:0/2395087672 shutdown_connections 2024-03-31T23:37:50.608 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:50.606+0000 7f1c22007640 1 -- 172.21.15.46:0/2395087672 wait complete. 2024-03-31T23:37:50.618 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-31T23:37:50.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:50.724+0000 7fbebcf7a640 1 -- 172.21.15.46:0/1816032825 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbeb8106660 msgr2=0x7fbeb8106a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:37:50.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:50.724+0000 7fbebcf7a640 1 --2- 172.21.15.46:0/1816032825 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbeb8106660 0x7fbeb8106a40 secure :-1 s=READY pgs=1086 cs=0 l=1 rev1=1 crypto rx=0x7fbea8009930 tx=0x7fbea802bc30 comp rx=0 tx=0).stop 2024-03-31T23:37:50.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:50.724+0000 7fbebcf7a640 1 -- 172.21.15.46:0/1816032825 shutdown_connections 2024-03-31T23:37:50.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:50.724+0000 7fbebcf7a640 1 --2- 172.21.15.46:0/1816032825 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbeb8100f80 0x7fbeb810d490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:37:50.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:50.724+0000 7fbebcf7a640 1 --2- 172.21.15.46:0/1816032825 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbeb81005c0 0x7fbeb8100a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:37:50.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:50.724+0000 7fbebcf7a640 1 --2- 172.21.15.46:0/1816032825 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbeb8106660 0x7fbeb8106a40 unknown :-1 s=CLOSED pgs=1086 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:37:50.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:50.724+0000 7fbebcf7a640 1 -- 172.21.15.46:0/1816032825 >> 172.21.15.46:0/1816032825 conn(0x7fbeb8077750 msgr2=0x7fbeb8077b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:37:50.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:50.724+0000 7fbebcf7a640 1 -- 172.21.15.46:0/1816032825 shutdown_connections 2024-03-31T23:37:50.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:50.724+0000 7fbebcf7a640 1 -- 172.21.15.46:0/1816032825 wait complete. 2024-03-31T23:37:50.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:50.725+0000 7fbebcf7a640 1 Processor -- start 2024-03-31T23:37:50.728 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:50.725+0000 7fbebcf7a640 1 -- start start 2024-03-31T23:37:50.728 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:50.725+0000 7fbebcf7a640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbeb81005c0 0x7fbeb8197f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:37:50.728 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:50.725+0000 7fbebcf7a640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbeb8100f80 0x7fbeb8198480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:37:50.728 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:50.725+0000 7fbebcf7a640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbeb8106660 0x7fbeb819c7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:37:50.728 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:50.725+0000 7fbebcf7a640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fbeb8110890 con 0x7fbeb8106660 2024-03-31T23:37:50.728 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:50.725+0000 7fbebcf7a640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fbeb8110710 con 0x7fbeb81005c0 2024-03-31T23:37:50.728 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:50.725+0000 7fbebcf7a640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fbeb8110a10 con 0x7fbeb8100f80 2024-03-31T23:37:50.728 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:50.725+0000 7fbeb6d9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbeb8106660 0x7fbeb819c7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:37:50.728 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:50.725+0000 7fbeb6d9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbeb8106660 0x7fbeb819c7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:35426/0 (socket says 172.21.15.46:35426) 2024-03-31T23:37:50.728 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:50.725+0000 7fbeb6d9d640 1 -- 172.21.15.46:0/2828771676 learned_addr learned my addr 172.21.15.46:0/2828771676 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:37:50.728 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:50.725+0000 7fbeb5d9b640 1 --2- 172.21.15.46:0/2828771676 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbeb8100f80 0x7fbeb8198480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:37:50.728 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:50.725+0000 7fbeb659c640 1 --2- 172.21.15.46:0/2828771676 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbeb81005c0 0x7fbeb8197f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:37:50.728 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:50.725+0000 7fbeb6d9d640 1 -- 172.21.15.46:0/2828771676 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbeb8100f80 msgr2=0x7fbeb8198480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:37:50.728 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:50.725+0000 7fbeb6d9d640 1 --2- 172.21.15.46:0/2828771676 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbeb8100f80 0x7fbeb8198480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:37:50.728 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:50.725+0000 7fbeb6d9d640 1 -- 172.21.15.46:0/2828771676 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbeb81005c0 msgr2=0x7fbeb8197f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:37:50.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:50.725+0000 7fbeb6d9d640 1 --2- 172.21.15.46:0/2828771676 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbeb81005c0 0x7fbeb8197f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:37:50.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:50.725+0000 7fbeb6d9d640 1 -- 172.21.15.46:0/2828771676 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbeb819cee0 con 0x7fbeb8106660 2024-03-31T23:37:50.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:50.725+0000 7fbeb6d9d640 1 --2- 172.21.15.46:0/2828771676 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbeb8106660 0x7fbeb819c7b0 secure :-1 s=READY pgs=1087 cs=0 l=1 rev1=1 crypto rx=0x7fbeb000eb10 tx=0x7fbeb000c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:37:50.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:50.725+0000 7fbea77fe640 1 -- 172.21.15.46:0/2828771676 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbeb0004b20 con 0x7fbeb8106660 2024-03-31T23:37:50.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:50.726+0000 7fbea77fe640 1 -- 172.21.15.46:0/2828771676 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fbeb0004cc0 con 0x7fbeb8106660 2024-03-31T23:37:50.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:50.726+0000 7fbea77fe640 1 -- 172.21.15.46:0/2828771676 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbeb0010690 con 0x7fbeb8106660 2024-03-31T23:37:50.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:50.726+0000 7fbebcf7a640 1 -- 172.21.15.46:0/2828771676 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbeb819d1d0 con 0x7fbeb8106660 2024-03-31T23:37:50.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:50.726+0000 7fbebcf7a640 1 -- 172.21.15.46:0/2828771676 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbeb81a4a60 con 0x7fbeb8106660 2024-03-31T23:37:50.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:50.727+0000 7fbebcf7a640 1 -- 172.21.15.46:0/2828771676 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbeb8068540 con 0x7fbeb8106660 2024-03-31T23:37:50.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:50.728+0000 7fbea77fe640 1 -- 172.21.15.46:0/2828771676 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 25) ==== 137451+0+0 (secure 0 0 0) 0x7fbeb0005020 con 0x7fbeb8106660 2024-03-31T23:37:50.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:50.728+0000 7fbea77fe640 1 --2- 172.21.15.46:0/2828771676 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fbea00a2610 0x7fbea00a4ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:37:50.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:50.728+0000 7fbeb659c640 1 --2- 172.21.15.46:0/2828771676 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fbea00a2610 0x7fbea00a4ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:37:50.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:50.728+0000 7fbea77fe640 1 -- 172.21.15.46:0/2828771676 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(139..139 src has 1..139) ==== 9554+0+0 (secure 0 0 0) 0x7fbeb00cf530 con 0x7fbeb8106660 2024-03-31T23:37:50.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:50.729+0000 7fbeb659c640 1 --2- 172.21.15.46:0/2828771676 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fbea00a2610 0x7fbea00a4ad0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fbea8000c00 tx=0x7fbea8005fd0 comp rx=0 tx=0).ready entity=mgr.15654 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:37:50.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:50.733+0000 7fbea77fe640 1 -- 172.21.15.46:0/2828771676 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fbeb009bf40 con 0x7fbeb8106660 2024-03-31T23:37:50.917 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:50.916+0000 7fbebcf7a640 1 -- 172.21.15.46:0/2828771676 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fbeb81a4d90 con 0x7fbeb8106660 2024-03-31T23:37:50.917 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:50.916+0000 7fbea77fe640 1 -- 172.21.15.46:0/2828771676 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v139) ==== 104+0+11 (secure 0 0 0) 0x7fbeb009b840 con 0x7fbeb8106660 2024-03-31T23:37:50.917 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 32 2024-03-31T23:37:50.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:50.918+0000 7fbebcf7a640 1 -- 172.21.15.46:0/2828771676 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fbea00a2610 msgr2=0x7fbea00a4ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:37:50.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:50.918+0000 7fbebcf7a640 1 --2- 172.21.15.46:0/2828771676 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fbea00a2610 0x7fbea00a4ad0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fbea8000c00 tx=0x7fbea8005fd0 comp rx=0 tx=0).stop 2024-03-31T23:37:50.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:50.918+0000 7fbebcf7a640 1 -- 172.21.15.46:0/2828771676 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbeb8106660 msgr2=0x7fbeb819c7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:37:50.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:50.918+0000 7fbebcf7a640 1 --2- 172.21.15.46:0/2828771676 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbeb8106660 0x7fbeb819c7b0 secure :-1 s=READY pgs=1087 cs=0 l=1 rev1=1 crypto rx=0x7fbeb000eb10 tx=0x7fbeb000c2d0 comp rx=0 tx=0).stop 2024-03-31T23:37:50.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:50.918+0000 7fbebcf7a640 1 -- 172.21.15.46:0/2828771676 shutdown_connections 2024-03-31T23:37:50.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:50.918+0000 7fbebcf7a640 1 --2- 172.21.15.46:0/2828771676 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fbea00a2610 0x7fbea00a4ad0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:37:50.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:50.918+0000 7fbebcf7a640 1 --2- 172.21.15.46:0/2828771676 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbeb8106660 0x7fbeb819c7b0 unknown :-1 s=CLOSED pgs=1087 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:37:50.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:50.918+0000 7fbebcf7a640 1 --2- 172.21.15.46:0/2828771676 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbeb8100f80 0x7fbeb8198480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:37:50.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:50.918+0000 7fbebcf7a640 1 --2- 172.21.15.46:0/2828771676 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbeb81005c0 0x7fbeb8197f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:37:50.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:50.918+0000 7fbebcf7a640 1 -- 172.21.15.46:0/2828771676 >> 172.21.15.46:0/2828771676 conn(0x7fbeb8077750 msgr2=0x7fbeb80ff930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:37:50.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:50.918+0000 7fbebcf7a640 1 -- 172.21.15.46:0/2828771676 shutdown_connections 2024-03-31T23:37:50.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:50.918+0000 7fbebcf7a640 1 -- 172.21.15.46:0/2828771676 wait complete. 2024-03-31T23:37:50.930 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-31T23:37:51.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:51.034+0000 7f76030ae640 1 -- 172.21.15.46:0/3565107521 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f75fc100f90 msgr2=0x7f75fc10d4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:37:51.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:51.034+0000 7f76030ae640 1 --2- 172.21.15.46:0/3565107521 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f75fc100f90 0x7f75fc10d4a0 secure :-1 s=READY pgs=1088 cs=0 l=1 rev1=1 crypto rx=0x7f75f4009cd0 tx=0x7f75f402c0f0 comp rx=0 tx=0).stop 2024-03-31T23:37:51.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:51.035+0000 7f76030ae640 1 -- 172.21.15.46:0/3565107521 shutdown_connections 2024-03-31T23:37:51.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:51.035+0000 7f76030ae640 1 --2- 172.21.15.46:0/3565107521 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f75fc100f90 0x7f75fc10d4a0 unknown :-1 s=CLOSED pgs=1088 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:37:51.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:51.035+0000 7f76030ae640 1 --2- 172.21.15.46:0/3565107521 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f75fc1005d0 0x7f75fc100a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:37:51.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:51.035+0000 7f76030ae640 1 --2- 172.21.15.46:0/3565107521 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f75fc106670 0x7f75fc106a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:37:51.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:51.035+0000 7f76030ae640 1 -- 172.21.15.46:0/3565107521 >> 172.21.15.46:0/3565107521 conn(0x7f75fc077800 msgr2=0x7f75fc077c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:37:51.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:51.035+0000 7f76030ae640 1 -- 172.21.15.46:0/3565107521 shutdown_connections 2024-03-31T23:37:51.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:51.035+0000 7f76030ae640 1 -- 172.21.15.46:0/3565107521 wait complete. 2024-03-31T23:37:51.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:51.035+0000 7f76030ae640 1 Processor -- start 2024-03-31T23:37:51.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:51.035+0000 7f76030ae640 1 -- start start 2024-03-31T23:37:51.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:51.035+0000 7f76030ae640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f75fc1005d0 0x7f75fc198c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:37:51.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:51.035+0000 7f76030ae640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f75fc100f90 0x7f75fc199180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:37:51.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:51.035+0000 7f76030ae640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f75fc106670 0x7f75fc1a3cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:37:51.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:51.035+0000 7f76030ae640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f75fc110860 con 0x7f75fc1005d0 2024-03-31T23:37:51.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:51.035+0000 7f76030ae640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f75fc1106e0 con 0x7f75fc106670 2024-03-31T23:37:51.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:51.035+0000 7f76030ae640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f75fc1109e0 con 0x7f75fc100f90 2024-03-31T23:37:51.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:51.035+0000 7f7600e4a640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f75fc1005d0 0x7f75fc198c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:37:51.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:51.035+0000 7f7600e4a640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f75fc1005d0 0x7f75fc198c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:35434/0 (socket says 172.21.15.46:35434) 2024-03-31T23:37:51.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:51.035+0000 7f7600e4a640 1 -- 172.21.15.46:0/3819040180 learned_addr learned my addr 172.21.15.46:0/3819040180 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:37:51.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:51.036+0000 7f760164b640 1 --2- 172.21.15.46:0/3819040180 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f75fc106670 0x7f75fc1a3cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:37:51.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:51.036+0000 7f75fbfff640 1 --2- 172.21.15.46:0/3819040180 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f75fc100f90 0x7f75fc199180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:37:51.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:51.036+0000 7f7600e4a640 1 -- 172.21.15.46:0/3819040180 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f75fc100f90 msgr2=0x7f75fc199180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:37:51.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:51.036+0000 7f7600e4a640 1 --2- 172.21.15.46:0/3819040180 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f75fc100f90 0x7f75fc199180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:37:51.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:51.036+0000 7f7600e4a640 1 -- 172.21.15.46:0/3819040180 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f75fc106670 msgr2=0x7f75fc1a3cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:37:51.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:51.036+0000 7f7600e4a640 1 --2- 172.21.15.46:0/3819040180 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f75fc106670 0x7f75fc1a3cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:37:51.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:51.036+0000 7f7600e4a640 1 -- 172.21.15.46:0/3819040180 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f75fc1a4200 con 0x7f75fc1005d0 2024-03-31T23:37:51.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:51.036+0000 7f7600e4a640 1 --2- 172.21.15.46:0/3819040180 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f75fc1005d0 0x7f75fc198c40 secure :-1 s=READY pgs=1089 cs=0 l=1 rev1=1 crypto rx=0x7f75f000e9c0 tx=0x7f75f000ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:37:51.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:51.036+0000 7f75f9ffb640 1 -- 172.21.15.46:0/3819040180 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75f000cd80 con 0x7f75fc1005d0 2024-03-31T23:37:51.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:51.036+0000 7f75f9ffb640 1 -- 172.21.15.46:0/3819040180 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f75f00035f0 con 0x7f75fc1005d0 2024-03-31T23:37:51.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:51.036+0000 7f76030ae640 1 -- 172.21.15.46:0/3819040180 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f75fc1a43d0 con 0x7f75fc1005d0 2024-03-31T23:37:51.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:51.036+0000 7f75f9ffb640 1 -- 172.21.15.46:0/3819040180 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75f0010690 con 0x7f75fc1005d0 2024-03-31T23:37:51.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:51.036+0000 7f76030ae640 1 -- 172.21.15.46:0/3819040180 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f75fc1a4920 con 0x7f75fc1005d0 2024-03-31T23:37:51.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:51.037+0000 7f76030ae640 1 -- 172.21.15.46:0/3819040180 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f75fc068540 con 0x7f75fc1005d0 2024-03-31T23:37:51.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:51.038+0000 7f75f9ffb640 1 -- 172.21.15.46:0/3819040180 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 25) ==== 137451+0+0 (secure 0 0 0) 0x7f75f00037a0 con 0x7f75fc1005d0 2024-03-31T23:37:51.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:51.039+0000 7f75f9ffb640 1 --2- 172.21.15.46:0/3819040180 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f75ec0a2610 0x7f75ec0a4ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:37:51.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:51.039+0000 7f75fbfff640 1 --2- 172.21.15.46:0/3819040180 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f75ec0a2610 0x7f75ec0a4ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:37:51.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:51.039+0000 7f75f9ffb640 1 -- 172.21.15.46:0/3819040180 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(139..139 src has 1..139) ==== 9554+0+0 (secure 0 0 0) 0x7f75f00cf2c0 con 0x7f75fc1005d0 2024-03-31T23:37:51.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:51.039+0000 7f75fbfff640 1 --2- 172.21.15.46:0/3819040180 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f75ec0a2610 0x7f75ec0a4ad0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f75e80095f0 tx=0x7f75e8005f90 comp rx=0 tx=0).ready entity=mgr.15654 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:37:51.044 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:51.043+0000 7f75f9ffb640 1 -- 172.21.15.46:0/3819040180 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f75f009bcd0 con 0x7f75fc1005d0 2024-03-31T23:37:51.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:51.228+0000 7f76030ae640 1 -- 172.21.15.46:0/3819040180 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f75fc108b40 con 0x7f75fc1005d0 2024-03-31T23:37:51.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:51.228+0000 7f75f9ffb640 1 -- 172.21.15.46:0/3819040180 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v139) ==== 100+0+11 (secure 0 0 0) 0x7f75f009b5d0 con 0x7f75fc1005d0 2024-03-31T23:37:51.229 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 64 2024-03-31T23:37:51.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:51.230+0000 7f76030ae640 1 -- 172.21.15.46:0/3819040180 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f75ec0a2610 msgr2=0x7f75ec0a4ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:37:51.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:51.230+0000 7f76030ae640 1 --2- 172.21.15.46:0/3819040180 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f75ec0a2610 0x7f75ec0a4ad0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f75e80095f0 tx=0x7f75e8005f90 comp rx=0 tx=0).stop 2024-03-31T23:37:51.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:51.230+0000 7f76030ae640 1 -- 172.21.15.46:0/3819040180 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f75fc1005d0 msgr2=0x7f75fc198c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:37:51.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:51.230+0000 7f76030ae640 1 --2- 172.21.15.46:0/3819040180 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f75fc1005d0 0x7f75fc198c40 secure :-1 s=READY pgs=1089 cs=0 l=1 rev1=1 crypto rx=0x7f75f000e9c0 tx=0x7f75f000ee80 comp rx=0 tx=0).stop 2024-03-31T23:37:51.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:51.230+0000 7f76030ae640 1 -- 172.21.15.46:0/3819040180 shutdown_connections 2024-03-31T23:37:51.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:51.230+0000 7f76030ae640 1 --2- 172.21.15.46:0/3819040180 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f75ec0a2610 0x7f75ec0a4ad0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:37:51.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:51.230+0000 7f76030ae640 1 --2- 172.21.15.46:0/3819040180 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f75fc106670 0x7f75fc1a3cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:37:51.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:51.230+0000 7f76030ae640 1 --2- 172.21.15.46:0/3819040180 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f75fc100f90 0x7f75fc199180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:37:51.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:51.230+0000 7f76030ae640 1 --2- 172.21.15.46:0/3819040180 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f75fc1005d0 0x7f75fc198c40 unknown :-1 s=CLOSED pgs=1089 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:37:51.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:51.230+0000 7f76030ae640 1 -- 172.21.15.46:0/3819040180 >> 172.21.15.46:0/3819040180 conn(0x7f75fc077800 msgr2=0x7f75fc0ff1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:37:51.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:51.230+0000 7f76030ae640 1 -- 172.21.15.46:0/3819040180 shutdown_connections 2024-03-31T23:37:51.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:51.230+0000 7f76030ae640 1 -- 172.21.15.46:0/3819040180 wait complete. 2024-03-31T23:37:51.241 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:37:51 smithi046 ceph-mon[32498]: from='client.? 172.21.15.46:0/2715317897' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-31T23:37:51.241 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:37:51 smithi046 ceph-mon[32498]: from='client.? 172.21.15.46:0/2395087672' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-31T23:37:51.241 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:37:51 smithi046 ceph-mon[32498]: from='client.? 172.21.15.46:0/2828771676' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-31T23:37:51.242 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-31T23:37:51.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:51.347+0000 7fd8c694b640 1 -- 172.21.15.46:0/1583406668 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd8c0100f80 msgr2=0x7fd8c010d490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:37:51.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:51.347+0000 7fd8c694b640 1 --2- 172.21.15.46:0/1583406668 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd8c0100f80 0x7fd8c010d490 secure :-1 s=READY pgs=1090 cs=0 l=1 rev1=1 crypto rx=0x7fd8b8009cd0 tx=0x7fd8b802c0f0 comp rx=0 tx=0).stop 2024-03-31T23:37:51.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:51.348+0000 7fd8c694b640 1 -- 172.21.15.46:0/1583406668 shutdown_connections 2024-03-31T23:37:51.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:51.348+0000 7fd8c694b640 1 --2- 172.21.15.46:0/1583406668 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd8c0100f80 0x7fd8c010d490 unknown :-1 s=CLOSED pgs=1090 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:37:51.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:51.348+0000 7fd8c694b640 1 --2- 172.21.15.46:0/1583406668 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd8c01005c0 0x7fd8c0100a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:37:51.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:51.348+0000 7fd8c694b640 1 --2- 172.21.15.46:0/1583406668 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd8c0106660 0x7fd8c0106a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:37:51.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:51.348+0000 7fd8c694b640 1 -- 172.21.15.46:0/1583406668 >> 172.21.15.46:0/1583406668 conn(0x7fd8c0077750 msgr2=0x7fd8c0077b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:37:51.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:51.348+0000 7fd8c694b640 1 -- 172.21.15.46:0/1583406668 shutdown_connections 2024-03-31T23:37:51.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:51.348+0000 7fd8c694b640 1 -- 172.21.15.46:0/1583406668 wait complete. 2024-03-31T23:37:51.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:51.348+0000 7fd8c694b640 1 Processor -- start 2024-03-31T23:37:51.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:51.348+0000 7fd8c694b640 1 -- start start 2024-03-31T23:37:51.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:51.348+0000 7fd8c694b640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd8c01005c0 0x7fd8c0196d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:37:51.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:51.348+0000 7fd8c694b640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd8c0100f80 0x7fd8c0197270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:37:51.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:51.348+0000 7fd8c694b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd8c0106660 0x7fd8c019b5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:37:51.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:51.348+0000 7fd8c694b640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fd8c0110850 con 0x7fd8c0106660 2024-03-31T23:37:51.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:51.348+0000 7fd8c694b640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fd8c01106d0 con 0x7fd8c0100f80 2024-03-31T23:37:51.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:51.348+0000 7fd8c694b640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fd8c01109d0 con 0x7fd8c01005c0 2024-03-31T23:37:51.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:51.349+0000 7fd8c4ee8640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd8c0106660 0x7fd8c019b5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:37:51.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:51.349+0000 7fd8c4ee8640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd8c0106660 0x7fd8c019b5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:35450/0 (socket says 172.21.15.46:35450) 2024-03-31T23:37:51.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:51.349+0000 7fd8c4ee8640 1 -- 172.21.15.46:0/2064654131 learned_addr learned my addr 172.21.15.46:0/2064654131 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:37:51.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:51.349+0000 7fd8bf7fe640 1 --2- 172.21.15.46:0/2064654131 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd8c0100f80 0x7fd8c0197270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:37:51.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:51.349+0000 7fd8bffff640 1 --2- 172.21.15.46:0/2064654131 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd8c01005c0 0x7fd8c0196d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:37:51.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:51.349+0000 7fd8c4ee8640 1 -- 172.21.15.46:0/2064654131 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd8c01005c0 msgr2=0x7fd8c0196d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:37:51.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:51.349+0000 7fd8c4ee8640 1 --2- 172.21.15.46:0/2064654131 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd8c01005c0 0x7fd8c0196d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:37:51.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:51.349+0000 7fd8c4ee8640 1 -- 172.21.15.46:0/2064654131 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd8c0100f80 msgr2=0x7fd8c0197270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:37:51.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:51.349+0000 7fd8c4ee8640 1 --2- 172.21.15.46:0/2064654131 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd8c0100f80 0x7fd8c0197270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:37:51.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:51.349+0000 7fd8c4ee8640 1 -- 172.21.15.46:0/2064654131 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd8c019bd20 con 0x7fd8c0106660 2024-03-31T23:37:51.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:51.349+0000 7fd8c4ee8640 1 --2- 172.21.15.46:0/2064654131 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd8c0106660 0x7fd8c019b5f0 secure :-1 s=READY pgs=1091 cs=0 l=1 rev1=1 crypto rx=0x7fd8b8004af0 tx=0x7fd8b8035ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:37:51.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:51.349+0000 7fd8bd7fa640 1 -- 172.21.15.46:0/2064654131 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8b8039040 con 0x7fd8c0106660 2024-03-31T23:37:51.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:51.349+0000 7fd8bd7fa640 1 -- 172.21.15.46:0/2064654131 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd8b8044070 con 0x7fd8c0106660 2024-03-31T23:37:51.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:51.349+0000 7fd8c694b640 1 -- 172.21.15.46:0/2064654131 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd8c019bfb0 con 0x7fd8c0106660 2024-03-31T23:37:51.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:51.349+0000 7fd8bd7fa640 1 -- 172.21.15.46:0/2064654131 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8b80042a0 con 0x7fd8c0106660 2024-03-31T23:37:51.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:51.349+0000 7fd8c694b640 1 -- 172.21.15.46:0/2064654131 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd8c01a38a0 con 0x7fd8c0106660 2024-03-31T23:37:51.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:51.350+0000 7fd8c694b640 1 -- 172.21.15.46:0/2064654131 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd8c0068540 con 0x7fd8c0106660 2024-03-31T23:37:51.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:51.351+0000 7fd8bd7fa640 1 -- 172.21.15.46:0/2064654131 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 25) ==== 137451+0+0 (secure 0 0 0) 0x7fd8b8004440 con 0x7fd8c0106660 2024-03-31T23:37:51.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:51.352+0000 7fd8bd7fa640 1 --2- 172.21.15.46:0/2064654131 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fd8a80a2610 0x7fd8a80a4ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:37:51.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:51.352+0000 7fd8bffff640 1 --2- 172.21.15.46:0/2064654131 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fd8a80a2610 0x7fd8a80a4ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:37:51.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:51.352+0000 7fd8bd7fa640 1 -- 172.21.15.46:0/2064654131 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(139..139 src has 1..139) ==== 9554+0+0 (secure 0 0 0) 0x7fd8b80f1310 con 0x7fd8c0106660 2024-03-31T23:37:51.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:51.352+0000 7fd8bffff640 1 --2- 172.21.15.46:0/2064654131 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fd8a80a2610 0x7fd8a80a4ad0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fd8c0101700 tx=0x7fd8b0005f90 comp rx=0 tx=0).ready entity=mgr.15654 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:37:51.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:51.356+0000 7fd8bd7fa640 1 -- 172.21.15.46:0/2064654131 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fd8b80bde90 con 0x7fd8c0106660 2024-03-31T23:37:51.443 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:37:51 smithi142 ceph-mon[35271]: from='client.? 172.21.15.46:0/2715317897' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-31T23:37:51.443 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:37:51 smithi142 ceph-mon[35271]: from='client.? 172.21.15.46:0/2395087672' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-31T23:37:51.443 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:37:51 smithi142 ceph-mon[35271]: from='client.? 172.21.15.46:0/2828771676' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-31T23:37:51.538 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:51.537+0000 7fd8c694b640 1 -- 172.21.15.46:0/2064654131 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7fd8c0108b30 con 0x7fd8c0106660 2024-03-31T23:37:51.538 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:51.538+0000 7fd8bd7fa640 1 -- 172.21.15.46:0/2064654131 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v139) ==== 103+0+11 (secure 0 0 0) 0x7fd8b80bd790 con 0x7fd8c0106660 2024-03-31T23:37:51.539 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 32 2024-03-31T23:37:51.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:51.540+0000 7fd8c694b640 1 -- 172.21.15.46:0/2064654131 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fd8a80a2610 msgr2=0x7fd8a80a4ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:37:51.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:51.540+0000 7fd8c694b640 1 --2- 172.21.15.46:0/2064654131 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fd8a80a2610 0x7fd8a80a4ad0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fd8c0101700 tx=0x7fd8b0005f90 comp rx=0 tx=0).stop 2024-03-31T23:37:51.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:51.540+0000 7fd8c694b640 1 -- 172.21.15.46:0/2064654131 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd8c0106660 msgr2=0x7fd8c019b5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:37:51.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:51.540+0000 7fd8c694b640 1 --2- 172.21.15.46:0/2064654131 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd8c0106660 0x7fd8c019b5f0 secure :-1 s=READY pgs=1091 cs=0 l=1 rev1=1 crypto rx=0x7fd8b8004af0 tx=0x7fd8b8035ce0 comp rx=0 tx=0).stop 2024-03-31T23:37:51.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:51.540+0000 7fd8c694b640 1 -- 172.21.15.46:0/2064654131 shutdown_connections 2024-03-31T23:37:51.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:51.540+0000 7fd8c694b640 1 --2- 172.21.15.46:0/2064654131 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7fd8a80a2610 0x7fd8a80a4ad0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:37:51.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:51.540+0000 7fd8c694b640 1 --2- 172.21.15.46:0/2064654131 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd8c0106660 0x7fd8c019b5f0 unknown :-1 s=CLOSED pgs=1091 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:37:51.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:51.540+0000 7fd8c694b640 1 --2- 172.21.15.46:0/2064654131 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd8c0100f80 0x7fd8c0197270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:37:51.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:51.540+0000 7fd8c694b640 1 --2- 172.21.15.46:0/2064654131 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd8c01005c0 0x7fd8c0196d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:37:51.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:51.540+0000 7fd8c694b640 1 -- 172.21.15.46:0/2064654131 >> 172.21.15.46:0/2064654131 conn(0x7fd8c0077750 msgr2=0x7fd8c00ff240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:37:51.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:51.540+0000 7fd8c694b640 1 -- 172.21.15.46:0/2064654131 shutdown_connections 2024-03-31T23:37:51.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:51.540+0000 7fd8c694b640 1 -- 172.21.15.46:0/2064654131 wait complete. 2024-03-31T23:37:51.545 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:37:51 smithi162 ceph-mon[36237]: from='client.? 172.21.15.46:0/2715317897' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-31T23:37:51.545 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:37:51 smithi162 ceph-mon[36237]: from='client.? 172.21.15.46:0/2395087672' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-31T23:37:51.545 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:37:51 smithi162 ceph-mon[36237]: from='client.? 172.21.15.46:0/2828771676' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-31T23:37:51.552 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-31T23:37:51.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:51.655+0000 7f32d03e9640 1 -- 172.21.15.46:0/1738313324 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f32c81029c0 msgr2=0x7f32c8102da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:37:51.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:51.655+0000 7f32d03e9640 1 --2- 172.21.15.46:0/1738313324 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f32c81029c0 0x7f32c8102da0 secure :-1 s=READY pgs=1092 cs=0 l=1 rev1=1 crypto rx=0x7f32c4009900 tx=0x7f32c402bcd0 comp rx=0 tx=0).stop 2024-03-31T23:37:51.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:51.655+0000 7f32d03e9640 1 -- 172.21.15.46:0/1738313324 shutdown_connections 2024-03-31T23:37:51.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:51.655+0000 7f32d03e9640 1 --2- 172.21.15.46:0/1738313324 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f32c810bd90 0x7f32c810e200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:37:51.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:51.655+0000 7f32d03e9640 1 --2- 172.21.15.46:0/1738313324 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f32c81032e0 0x7f32c810b7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:37:51.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:51.655+0000 7f32d03e9640 1 --2- 172.21.15.46:0/1738313324 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f32c81029c0 0x7f32c8102da0 unknown :-1 s=CLOSED pgs=1092 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:37:51.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:51.655+0000 7f32d03e9640 1 -- 172.21.15.46:0/1738313324 >> 172.21.15.46:0/1738313324 conn(0x7f32c80777b0 msgr2=0x7f32c8077bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:37:51.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:51.655+0000 7f32d03e9640 1 -- 172.21.15.46:0/1738313324 shutdown_connections 2024-03-31T23:37:51.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:51.655+0000 7f32d03e9640 1 -- 172.21.15.46:0/1738313324 wait complete. 2024-03-31T23:37:51.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:51.655+0000 7f32d03e9640 1 Processor -- start 2024-03-31T23:37:51.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:51.655+0000 7f32d03e9640 1 -- start start 2024-03-31T23:37:51.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:51.655+0000 7f32d03e9640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f32c81029c0 0x7f32c8197fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:37:51.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:51.655+0000 7f32d03e9640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f32c81032e0 0x7f32c8198500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:37:51.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:51.655+0000 7f32d03e9640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f32c810bd90 0x7f32c819c830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:37:51.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:51.656+0000 7f32d03e9640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f32c8111a60 con 0x7f32c810bd90 2024-03-31T23:37:51.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:51.656+0000 7f32d03e9640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f32c81118e0 con 0x7f32c81029c0 2024-03-31T23:37:51.660 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:51.656+0000 7f32d03e9640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f32c8111be0 con 0x7f32c81032e0 2024-03-31T23:37:51.660 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:51.656+0000 7f32ce986640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f32c810bd90 0x7f32c819c830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:37:51.660 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:51.656+0000 7f32ce986640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f32c810bd90 0x7f32c819c830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:46164/0 (socket says 172.21.15.46:46164) 2024-03-31T23:37:51.660 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:51.656+0000 7f32ce986640 1 -- 172.21.15.46:0/4273757076 learned_addr learned my addr 172.21.15.46:0/4273757076 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-31T23:37:51.660 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:51.656+0000 7f32ce185640 1 --2- 172.21.15.46:0/4273757076 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f32c81029c0 0x7f32c8197fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:37:51.660 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:51.656+0000 7f32cd984640 1 --2- 172.21.15.46:0/4273757076 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f32c81032e0 0x7f32c8198500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:37:51.660 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:51.656+0000 7f32ce986640 1 -- 172.21.15.46:0/4273757076 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f32c81032e0 msgr2=0x7f32c8198500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:37:51.660 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:51.656+0000 7f32ce986640 1 --2- 172.21.15.46:0/4273757076 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f32c81032e0 0x7f32c8198500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:37:51.660 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:51.656+0000 7f32ce986640 1 -- 172.21.15.46:0/4273757076 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f32c81029c0 msgr2=0x7f32c8197fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:37:51.660 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:51.656+0000 7f32ce986640 1 --2- 172.21.15.46:0/4273757076 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f32c81029c0 0x7f32c8197fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:37:51.660 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:51.656+0000 7f32ce986640 1 -- 172.21.15.46:0/4273757076 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f32c819cf60 con 0x7f32c810bd90 2024-03-31T23:37:51.660 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:51.656+0000 7f32ce986640 1 --2- 172.21.15.46:0/4273757076 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f32c810bd90 0x7f32c819c830 secure :-1 s=READY pgs=1093 cs=0 l=1 rev1=1 crypto rx=0x7f32c000eb10 tx=0x7f32c000c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:37:51.660 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:51.656+0000 7f32bf7fe640 1 -- 172.21.15.46:0/4273757076 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32c0004b20 con 0x7f32c810bd90 2024-03-31T23:37:51.660 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:51.656+0000 7f32bf7fe640 1 -- 172.21.15.46:0/4273757076 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f32c0004cc0 con 0x7f32c810bd90 2024-03-31T23:37:51.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:51.656+0000 7f32bf7fe640 1 -- 172.21.15.46:0/4273757076 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32c0010690 con 0x7f32c810bd90 2024-03-31T23:37:51.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:51.656+0000 7f32d03e9640 1 -- 172.21.15.46:0/4273757076 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f32c819d250 con 0x7f32c810bd90 2024-03-31T23:37:51.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:51.656+0000 7f32d03e9640 1 -- 172.21.15.46:0/4273757076 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f32c81a4ae0 con 0x7f32c810bd90 2024-03-31T23:37:51.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:51.657+0000 7f32d03e9640 1 -- 172.21.15.46:0/4273757076 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f32c8068990 con 0x7f32c810bd90 2024-03-31T23:37:51.662 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:51.658+0000 7f32bf7fe640 1 -- 172.21.15.46:0/4273757076 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 25) ==== 137451+0+0 (secure 0 0 0) 0x7f32c0005000 con 0x7f32c810bd90 2024-03-31T23:37:51.662 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:51.659+0000 7f32bf7fe640 1 --2- 172.21.15.46:0/4273757076 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f32b40a2690 0x7f32b40a4b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T23:37:51.662 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:51.659+0000 7f32ce185640 1 --2- 172.21.15.46:0/4273757076 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f32b40a2690 0x7f32b40a4b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T23:37:51.662 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:51.659+0000 7f32bf7fe640 1 -- 172.21.15.46:0/4273757076 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(139..139 src has 1..139) ==== 9554+0+0 (secure 0 0 0) 0x7f32c00cf870 con 0x7f32c810bd90 2024-03-31T23:37:51.662 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:51.659+0000 7f32ce185640 1 --2- 172.21.15.46:0/4273757076 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f32b40a2690 0x7f32b40a4b50 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f32c4005ab0 tx=0x7f32c4005a20 comp rx=0 tx=0).ready entity=mgr.15654 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T23:37:51.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:51.662+0000 7f32bf7fe640 1 -- 172.21.15.46:0/4273757076 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f32c009c2e0 con 0x7f32c810bd90 2024-03-31T23:37:51.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:51.856+0000 7f32d03e9640 1 -- 172.21.15.46:0/4273757076 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f32c8106ee0 con 0x7f32c810bd90 2024-03-31T23:37:51.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:51.857+0000 7f32bf7fe640 1 -- 172.21.15.46:0/4273757076 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 16 entries v139) ==== 81+0+940 (secure 0 0 0) 0x7f32c009bbe0 con 0x7f32c810bd90 2024-03-31T23:37:51.859 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:0/249968062 2024-04-01T23:11:42.678401+0000 2024-03-31T23:37:51.859 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6801/162289569 2024-04-01T23:11:42.678401+0000 2024-03-31T23:37:51.859 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6800/162289569 2024-04-01T23:11:42.678401+0000 2024-03-31T23:37:51.859 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:0/34745898 2024-04-01T23:11:42.678401+0000 2024-03-31T23:37:51.859 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:0/3306820105 2024-04-01T23:11:42.678401+0000 2024-03-31T23:37:51.859 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:0/4021039857 2024-04-01T23:02:08.493619+0000 2024-03-31T23:37:51.860 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6801/3681182647 2024-04-01T23:02:08.493619+0000 2024-03-31T23:37:51.860 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:0/4022214806 2024-04-01T23:02:08.493619+0000 2024-03-31T23:37:51.860 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6800/1515528215 2024-04-01T23:01:41.910162+0000 2024-03-31T23:37:51.860 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:0/1146542737 2024-04-01T23:11:42.678401+0000 2024-03-31T23:37:51.860 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6800/3681182647 2024-04-01T23:02:08.493619+0000 2024-03-31T23:37:51.860 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:0/143885707 2024-04-01T23:01:41.910162+0000 2024-03-31T23:37:51.861 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6801/1515528215 2024-04-01T23:01:41.910162+0000 2024-03-31T23:37:51.861 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:0/1680389778 2024-04-01T23:01:41.910162+0000 2024-03-31T23:37:51.861 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:0/1782133966 2024-04-01T23:01:41.910162+0000 2024-03-31T23:37:51.861 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:0/3925209376 2024-04-01T23:02:08.493619+0000 2024-03-31T23:37:51.863 INFO:teuthology.orchestra.run.smithi046.stderr:listed 16 entries 2024-03-31T23:37:51.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:51.859+0000 7f32d03e9640 1 -- 172.21.15.46:0/4273757076 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f32b40a2690 msgr2=0x7f32b40a4b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:37:51.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:51.859+0000 7f32d03e9640 1 --2- 172.21.15.46:0/4273757076 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f32b40a2690 0x7f32b40a4b50 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f32c4005ab0 tx=0x7f32c4005a20 comp rx=0 tx=0).stop 2024-03-31T23:37:51.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:51.859+0000 7f32d03e9640 1 -- 172.21.15.46:0/4273757076 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f32c810bd90 msgr2=0x7f32c819c830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T23:37:51.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:51.859+0000 7f32d03e9640 1 --2- 172.21.15.46:0/4273757076 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f32c810bd90 0x7f32c819c830 secure :-1 s=READY pgs=1093 cs=0 l=1 rev1=1 crypto rx=0x7f32c000eb10 tx=0x7f32c000c2d0 comp rx=0 tx=0).stop 2024-03-31T23:37:51.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:51.859+0000 7f32d03e9640 1 -- 172.21.15.46:0/4273757076 shutdown_connections 2024-03-31T23:37:51.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:51.859+0000 7f32d03e9640 1 --2- 172.21.15.46:0/4273757076 >> [v2:172.21.15.46:6800/685378567,v1:172.21.15.46:6801/685378567] conn(0x7f32b40a2690 0x7f32b40a4b50 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:37:51.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:51.859+0000 7f32d03e9640 1 --2- 172.21.15.46:0/4273757076 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f32c810bd90 0x7f32c819c830 unknown :-1 s=CLOSED pgs=1093 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:37:51.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:51.859+0000 7f32d03e9640 1 --2- 172.21.15.46:0/4273757076 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f32c81032e0 0x7f32c8198500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:37:51.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:51.859+0000 7f32d03e9640 1 --2- 172.21.15.46:0/4273757076 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f32c81029c0 0x7f32c8197fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T23:37:51.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:51.859+0000 7f32d03e9640 1 -- 172.21.15.46:0/4273757076 >> 172.21.15.46:0/4273757076 conn(0x7f32c80777b0 msgr2=0x7f32c80fd7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T23:37:51.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:51.859+0000 7f32d03e9640 1 -- 172.21.15.46:0/4273757076 shutdown_connections 2024-03-31T23:37:51.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-31T23:37:51.859+0000 7f32d03e9640 1 -- 172.21.15.46:0/4273757076 wait complete. 2024-03-31T23:37:51.870 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi046.front.sepia.ceph.com... 2024-03-31T23:37:51.870 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T23:37:51.871 DEBUG:teuthology.orchestra.run.smithi046:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-03-31T23:37:51.931 INFO:teuthology.orchestra.run:waiting for 300 2024-03-31T23:37:51.931 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi046.stderr:ceph-fuse[86040]: fuse finished with error 0 and tester_r 0 2024-03-31T23:37:52.443 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:37:52 smithi142 ceph-mon[35271]: pgmap v830: 129 pgs: 129 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 853 B/s wr, 5 op/s 2024-03-31T23:37:52.443 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:37:52 smithi142 ceph-mon[35271]: from='client.? 172.21.15.46:0/3819040180' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-31T23:37:52.443 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:37:52 smithi142 ceph-mon[35271]: from='client.? 172.21.15.46:0/2064654131' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-31T23:37:52.444 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:37:52 smithi142 ceph-mon[35271]: from='client.? 172.21.15.46:0/4273757076' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-31T23:37:52.545 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:37:52 smithi162 ceph-mon[36237]: pgmap v830: 129 pgs: 129 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 853 B/s wr, 5 op/s 2024-03-31T23:37:52.545 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:37:52 smithi162 ceph-mon[36237]: from='client.? 172.21.15.46:0/3819040180' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-31T23:37:52.546 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:37:52 smithi162 ceph-mon[36237]: from='client.? 172.21.15.46:0/2064654131' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-31T23:37:52.546 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:37:52 smithi162 ceph-mon[36237]: from='client.? 172.21.15.46:0/4273757076' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-31T23:37:52.591 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:37:52 smithi046 ceph-mon[32498]: pgmap v830: 129 pgs: 129 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 853 B/s wr, 5 op/s 2024-03-31T23:37:52.591 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:37:52 smithi046 ceph-mon[32498]: from='client.? 172.21.15.46:0/3819040180' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-31T23:37:52.592 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:37:52 smithi046 ceph-mon[32498]: from='client.? 172.21.15.46:0/2064654131' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-31T23:37:52.592 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:37:52 smithi046 ceph-mon[32498]: from='client.? 172.21.15.46:0/4273757076' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-31T23:37:54.443 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:37:54 smithi142 ceph-mon[35271]: pgmap v831: 129 pgs: 129 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 853 B/s wr, 5 op/s 2024-03-31T23:37:54.546 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:37:54 smithi162 ceph-mon[36237]: pgmap v831: 129 pgs: 129 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 853 B/s wr, 5 op/s 2024-03-31T23:37:54.611 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:37:54 smithi046 ceph-mon[32498]: pgmap v831: 129 pgs: 129 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 853 B/s wr, 5 op/s 2024-03-31T23:37:56.421 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:37:56 smithi162 ceph-mon[36237]: pgmap v832: 129 pgs: 129 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 1.5 KiB/s wr, 9 op/s 2024-03-31T23:37:56.443 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:37:56 smithi142 ceph-mon[35271]: pgmap v832: 129 pgs: 129 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 1.5 KiB/s wr, 9 op/s 2024-03-31T23:37:56.611 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:37:56 smithi046 ceph-mon[32498]: pgmap v832: 129 pgs: 129 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 1.5 KiB/s wr, 9 op/s 2024-03-31T23:37:58.032 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi046.front.sepia.ceph.com 2024-03-31T23:37:58.032 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T23:37:58.032 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-31T23:37:58.077 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T23:37:58.077 DEBUG:teuthology.orchestra.run.smithi046:> ip netns list 2024-03-31T23:37:58.127 INFO:teuthology.orchestra.run.smithi046.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-31T23:37:58.127 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T23:37:58.127 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-31T23:37:58.159 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T23:37:58.159 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip link delete ceph-brx 2024-03-31T23:37:58.260 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-03-31T23:37:58.271 INFO:tasks.cephadm:Teardown begin 2024-03-31T23:37:58.272 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_ceph_384f31f3d3aee7f8ba3f60846b8844c049165583/qa/tasks/cephadm.py", line 1807, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph_384f31f3d3aee7f8ba3f60846b8844c049165583/qa/tasks/fwd_scrub.py", line 164, in task stop_all_fwd_scrubbers(ctx.ceph[config['cluster']].thrashers) File "/home/teuthworker/src/git.ceph.com_ceph_384f31f3d3aee7f8ba3f60846b8844c049165583/qa/tasks/fwd_scrub.py", line 99, in stop_all_fwd_scrubbers raise RuntimeError(f"error during scrub thrashing: {thrasher.exception}") RuntimeError: error during scrub thrashing: rank damage found: {'backtrace'} 2024-03-31T23:37:58.273 DEBUG:teuthology.orchestra.run.smithi046:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-31T23:37:58.312 DEBUG:teuthology.orchestra.run.smithi142:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-31T23:37:58.372 DEBUG:teuthology.orchestra.run.smithi162:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-31T23:37:58.425 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:37:58 smithi162 ceph-mon[36237]: pgmap v833: 129 pgs: 129 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 1.5 KiB/s wr, 6 op/s 2024-03-31T23:37:58.426 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-03-31T23:37:58.427 DEBUG:teuthology.orchestra.run.smithi046:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-31T23:37:58.442 DEBUG:teuthology.orchestra.run.smithi142:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-31T23:37:58.443 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:37:58 smithi142 ceph-mon[35271]: pgmap v833: 129 pgs: 129 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 1.5 KiB/s wr, 6 op/s 2024-03-31T23:37:58.456 DEBUG:teuthology.orchestra.run.smithi162:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-31T23:37:58.471 INFO:tasks.cephadm:Stopping all daemons... 2024-03-31T23:37:58.471 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-03-31T23:37:58.471 DEBUG:teuthology.orchestra.run.smithi046:> sudo systemctl stop ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4@mon.a 2024-03-31T23:37:58.509 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:37:58 smithi046 ceph-mon[32498]: pgmap v833: 129 pgs: 129 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 1.5 KiB/s wr, 6 op/s 2024-03-31T23:37:58.809 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:37:58 smithi046 systemd[1]: Stopping Ceph mon.a for 6a9d8f08-efb2-11ee-b647-cb9ed24678a4... 2024-03-31T23:37:59.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:37:58 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mon-a[32494]: 2024-03-31T23:37:58.807+0000 7fdb0e7a1700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-31T23:37:59.111 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:37:58 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mon-a[32494]: 2024-03-31T23:37:58.807+0000 7fdb0e7a1700 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-03-31T23:37:59.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:37:59 smithi046 podman[107180]: 2024-03-31 23:37:59.002148923 +0000 UTC m=+0.314858055 container died 857af8e9024c6c11705485611e9ddc07a77e0a6606eb97c45f49b8633751af71 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:384f31f3d3aee7f8ba3f60846b8844c049165583, name=ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mon-a, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240326, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, io.buildah.version=1.33.5, RELEASE=squid-384f31f, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-03-31T23:37:59.611 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:37:59 smithi046 podman[107180]: 2024-03-31 23:37:59.409239133 +0000 UTC m=+0.721948253 container cleanup 857af8e9024c6c11705485611e9ddc07a77e0a6606eb97c45f49b8633751af71 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:384f31f3d3aee7f8ba3f60846b8844c049165583, name=ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mon-a, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, RELEASE=squid-384f31f, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240326) 2024-03-31T23:37:59.612 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 31 23:37:59 smithi046 bash[107180]: ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mon-a 2024-03-31T23:37:59.771 DEBUG:teuthology.orchestra.run.smithi046:> sudo pkill -f 'journalctl -f -n 0 -u ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4@mon.a.service' 2024-03-31T23:37:59.878 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-31T23:37:59.878 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-03-31T23:37:59.878 INFO:tasks.cephadm.mon.c:Stopping mon.b... 2024-03-31T23:37:59.878 DEBUG:teuthology.orchestra.run.smithi142:> sudo systemctl stop ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4@mon.b 2024-03-31T23:38:00.193 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:37:59 smithi142 systemd[1]: Stopping Ceph mon.b for 6a9d8f08-efb2-11ee-b647-cb9ed24678a4... 2024-03-31T23:38:01.927 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:38:01 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mon-b[35267]: 2024-03-31T23:38:01.465+0000 7fd807ce4700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-31T23:38:01.927 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:38:01 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mon-b[35267]: 2024-03-31T23:38:01.465+0000 7fd807ce4700 -1 mon.b@2(peon) e3 *** Got Signal Terminated *** 2024-03-31T23:38:01.927 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:38:01 smithi142 podman[74761]: 2024-03-31 23:38:01.628675963 +0000 UTC m=+0.864242714 container died 5a36e49c19c9dad01033713a1e9f1692bf3201d01eedb8c3144be00d5aa0b724 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mon-b, GIT_CLEAN=True, RELEASE=squid-384f31f, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240326, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5) 2024-03-31T23:38:02.179 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:38:02 smithi142 podman[74761]: 2024-03-31 23:38:02.16658802 +0000 UTC m=+1.402154759 container cleanup 5a36e49c19c9dad01033713a1e9f1692bf3201d01eedb8c3144be00d5aa0b724 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mon-b, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240326, org.label-schema.license=GPLv2, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, RELEASE=squid-384f31f, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0) 2024-03-31T23:38:02.179 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:38:02 smithi142 bash[74761]: ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mon-b 2024-03-31T23:38:02.943 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 31 23:38:02 smithi142 podman[74791]: 2024-03-31 23:38:02.451847784 +0000 UTC m=+0.824869906 container remove 5a36e49c19c9dad01033713a1e9f1692bf3201d01eedb8c3144be00d5aa0b724 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mon-b, RELEASE=squid-384f31f, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240326, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True) 2024-03-31T23:38:03.416 DEBUG:teuthology.orchestra.run.smithi142:> sudo pkill -f 'journalctl -f -n 0 -u ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4@mon.b.service' 2024-03-31T23:38:03.470 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-31T23:38:03.470 INFO:tasks.cephadm.mon.c:Stopped mon.b 2024-03-31T23:38:03.470 INFO:tasks.cephadm.mon.c:Stopping mon.c... 2024-03-31T23:38:03.471 DEBUG:teuthology.orchestra.run.smithi162:> sudo systemctl stop ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4@mon.c 2024-03-31T23:38:03.795 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:38:03 smithi162 systemd[1]: Stopping Ceph mon.c for 6a9d8f08-efb2-11ee-b647-cb9ed24678a4... 2024-03-31T23:38:04.234 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:38:03 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mon-c[36206]: 2024-03-31T23:38:03.840+0000 7fd6f7fbf700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-31T23:38:04.235 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:38:03 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mon-c[36206]: 2024-03-31T23:38:03.840+0000 7fd6f7fbf700 -1 mon.c@1(peon) e3 *** Got Signal Terminated *** 2024-03-31T23:38:04.235 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:38:03 smithi162 podman[77850]: 2024-03-31 23:38:03.99376224 +0000 UTC m=+0.277911568 container died ce17b19294be5c9adae1c26e57056358f993406384d7c6c6043bfbc26e4c9d83 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mon-c, org.label-schema.build-date=20240326, GIT_BRANCH=HEAD, RELEASE=squid-384f31f, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2) 2024-03-31T23:38:04.545 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:38:04 smithi162 podman[77850]: 2024-03-31 23:38:04.325688144 +0000 UTC m=+0.609837465 container cleanup ce17b19294be5c9adae1c26e57056358f993406384d7c6c6043bfbc26e4c9d83 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mon-c, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240326, CEPH_POINT_RELEASE=, RELEASE=squid-384f31f, ceph=True, io.buildah.version=1.33.5, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-03-31T23:38:04.545 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:38:04 smithi162 bash[77850]: ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mon-c 2024-03-31T23:38:04.825 DEBUG:teuthology.orchestra.run.smithi162:> sudo pkill -f 'journalctl -f -n 0 -u ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4@mon.c.service' 2024-03-31T23:38:04.849 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:38:04 smithi162 podman[77864]: 2024-03-31 23:38:04.577422161 +0000 UTC m=+0.580177703 container remove ce17b19294be5c9adae1c26e57056358f993406384d7c6c6043bfbc26e4c9d83 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mon-c, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240326, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-384f31f, CEPH_POINT_RELEASE=, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, maintainer=Guillaume Abrioux ) 2024-03-31T23:38:04.849 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:38:04 smithi162 systemd[1]: ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4@mon.c.service: Deactivated successfully. 2024-03-31T23:38:04.849 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:38:04 smithi162 systemd[1]: Stopped Ceph mon.c for 6a9d8f08-efb2-11ee-b647-cb9ed24678a4. 2024-03-31T23:38:04.849 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 31 23:38:04 smithi162 systemd[1]: ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4@mon.c.service: Consumed 17.907s CPU time. 2024-03-31T23:38:04.882 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-31T23:38:04.882 INFO:tasks.cephadm.mon.c:Stopped mon.c 2024-03-31T23:38:04.882 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-03-31T23:38:04.882 DEBUG:teuthology.orchestra.run.smithi046:> sudo systemctl stop ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4@mgr.x 2024-03-31T23:38:05.189 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 31 23:38:04 smithi046 systemd[1]: Stopping Ceph mgr.x for 6a9d8f08-efb2-11ee-b647-cb9ed24678a4... 2024-03-31T23:38:05.490 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 31 23:38:05 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mgr-x[32727]: 2024-03-31T23:38:05.188+0000 7f1d820af700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mgr -n mgr.x -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-31T23:38:05.490 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 31 23:38:05 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mgr-x[32727]: 2024-03-31T23:38:05.188+0000 7f1d820af700 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2024-03-31T23:38:05.491 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 31 23:38:05 smithi046 podman[107304]: 2024-03-31 23:38:05.220020679 +0000 UTC m=+0.140804015 container died 2adfecce4e56bfe409ec19699477f75777c8d2397f49aceac6ba5bd2e14c8c22 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:384f31f3d3aee7f8ba3f60846b8844c049165583, name=ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mgr-x, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240326, RELEASE=squid-384f31f, org.label-schema.license=GPLv2, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-03-31T23:38:05.861 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 31 23:38:05 smithi046 podman[107304]: 2024-03-31 23:38:05.573490975 +0000 UTC m=+0.494274357 container cleanup 2adfecce4e56bfe409ec19699477f75777c8d2397f49aceac6ba5bd2e14c8c22 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:384f31f3d3aee7f8ba3f60846b8844c049165583, name=ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mgr-x, RELEASE=squid-384f31f, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.build-date=20240326) 2024-03-31T23:38:05.862 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 31 23:38:05 smithi046 bash[107304]: ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mgr-x 2024-03-31T23:38:06.051 DEBUG:teuthology.orchestra.run.smithi046:> sudo pkill -f 'journalctl -f -n 0 -u ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4@mgr.x.service' 2024-03-31T23:38:06.089 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-31T23:38:06.089 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-03-31T23:38:06.089 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-03-31T23:38:06.089 DEBUG:teuthology.orchestra.run.smithi142:> sudo systemctl stop ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4@mgr.y 2024-03-31T23:38:06.414 INFO:journalctl@ceph.mgr.y.smithi142.stdout:Mar 31 23:38:06 smithi142 systemd[1]: Stopping Ceph mgr.y for 6a9d8f08-efb2-11ee-b647-cb9ed24678a4... 2024-03-31T23:38:06.693 INFO:journalctl@ceph.mgr.y.smithi142.stdout:Mar 31 23:38:06 smithi142 podman[77591]: 2024-03-31 23:38:06.414093361 +0000 UTC m=+0.138978369 container died 1db2604d0e85c52c4ab3e717432e55e0c0f0fdf99161a4d456c4cd9dcfcc4e29 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mgr-y, org.label-schema.license=GPLv2, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_BRANCH=HEAD, RELEASE=squid-384f31f, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , ceph=True, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240326, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-03-31T23:38:07.095 INFO:journalctl@ceph.mgr.y.smithi142.stdout:Mar 31 23:38:06 smithi142 podman[77591]: 2024-03-31 23:38:06.810850605 +0000 UTC m=+0.535735612 container cleanup 1db2604d0e85c52c4ab3e717432e55e0c0f0fdf99161a4d456c4cd9dcfcc4e29 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mgr-y, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-384f31f, GIT_CLEAN=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.build-date=20240326) 2024-03-31T23:38:07.095 INFO:journalctl@ceph.mgr.y.smithi142.stdout:Mar 31 23:38:06 smithi142 bash[77591]: ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mgr-y 2024-03-31T23:38:07.240 DEBUG:teuthology.orchestra.run.smithi142:> sudo pkill -f 'journalctl -f -n 0 -u ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4@mgr.y.service' 2024-03-31T23:38:07.282 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-31T23:38:07.283 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-03-31T23:38:07.283 INFO:tasks.cephadm.mgr.z:Stopping mgr.z... 2024-03-31T23:38:07.283 DEBUG:teuthology.orchestra.run.smithi162:> sudo systemctl stop ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4@mgr.z 2024-03-31T23:38:07.626 INFO:journalctl@ceph.mgr.z.smithi162.stdout:Mar 31 23:38:07 smithi162 systemd[1]: Stopping Ceph mgr.z for 6a9d8f08-efb2-11ee-b647-cb9ed24678a4... 2024-03-31T23:38:07.924 INFO:journalctl@ceph.mgr.z.smithi162.stdout:Mar 31 23:38:07 smithi162 podman[77971]: 2024-03-31 23:38:07.626289925 +0000 UTC m=+0.145812472 container died e44587aca31bac3080bf444a7f7973106a2a3ffbb3436f17e0af74bade9ca78e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mgr-z, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, RELEASE=squid-384f31f, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240326, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-03-31T23:38:08.295 INFO:journalctl@ceph.mgr.z.smithi162.stdout:Mar 31 23:38:08 smithi162 podman[77971]: 2024-03-31 23:38:08.006211085 +0000 UTC m=+0.525733631 container cleanup e44587aca31bac3080bf444a7f7973106a2a3ffbb3436f17e0af74bade9ca78e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mgr-z, GIT_CLEAN=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=squid-384f31f, GIT_BRANCH=HEAD, org.label-schema.build-date=20240326, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0) 2024-03-31T23:38:08.296 INFO:journalctl@ceph.mgr.z.smithi162.stdout:Mar 31 23:38:08 smithi162 bash[77971]: ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mgr-z 2024-03-31T23:38:08.449 DEBUG:teuthology.orchestra.run.smithi162:> sudo pkill -f 'journalctl -f -n 0 -u ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4@mgr.z.service' 2024-03-31T23:38:08.486 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-31T23:38:08.487 INFO:tasks.cephadm.mgr.z:Stopped mgr.z 2024-03-31T23:38:08.487 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-03-31T23:38:08.487 DEBUG:teuthology.orchestra.run.smithi046:> sudo systemctl stop ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4@osd.0 2024-03-31T23:38:08.787 INFO:journalctl@ceph.osd.0.smithi046.stdout:Mar 31 23:38:08 smithi046 systemd[1]: Stopping Ceph osd.0 for 6a9d8f08-efb2-11ee-b647-cb9ed24678a4... 2024-03-31T23:38:09.111 INFO:journalctl@ceph.osd.0.smithi046.stdout:Mar 31 23:38:08 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-0[47281]: 2024-03-31T23:38:08.785+0000 7f9fc17bf700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-31T23:38:09.112 INFO:journalctl@ceph.osd.0.smithi046.stdout:Mar 31 23:38:08 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-0[47281]: 2024-03-31T23:38:08.785+0000 7f9fc17bf700 -1 osd.0 139 *** Got signal Terminated *** 2024-03-31T23:38:09.112 INFO:journalctl@ceph.osd.0.smithi046.stdout:Mar 31 23:38:08 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-0[47281]: 2024-03-31T23:38:08.785+0000 7f9fc17bf700 -1 osd.0 139 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-31T23:38:14.612 INFO:journalctl@ceph.osd.0.smithi046.stdout:Mar 31 23:38:14 smithi046 podman[107426]: 2024-03-31 23:38:14.320500097 +0000 UTC m=+5.638703576 container died 534c29b008e359e9f9aa200e208149fb2844be52572b5a5ecb063fd0ae2824aa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-0, ceph=True, GIT_CLEAN=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.build-date=20240326, RELEASE=squid-384f31f, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5) 2024-03-31T23:38:15.111 INFO:journalctl@ceph.osd.0.smithi046.stdout:Mar 31 23:38:14 smithi046 podman[107426]: 2024-03-31 23:38:14.937177117 +0000 UTC m=+6.255380598 container cleanup 534c29b008e359e9f9aa200e208149fb2844be52572b5a5ecb063fd0ae2824aa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, RELEASE=squid-384f31f, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240326, ceph=True) 2024-03-31T23:38:15.112 INFO:journalctl@ceph.osd.0.smithi046.stdout:Mar 31 23:38:14 smithi046 bash[107426]: ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-0 2024-03-31T23:38:15.611 INFO:journalctl@ceph.osd.0.smithi046.stdout:Mar 31 23:38:15 smithi046 podman[107441]: 2024-03-31 23:38:15.205630286 +0000 UTC m=+0.879346649 container remove 534c29b008e359e9f9aa200e208149fb2844be52572b5a5ecb063fd0ae2824aa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240326, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, ceph=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=squid-384f31f, io.buildah.version=1.33.5) 2024-03-31T23:38:15.907 INFO:journalctl@ceph.osd.0.smithi046.stdout:Mar 31 23:38:15 smithi046 podman[107544]: 2024-03-31 23:38:15.535423924 +0000 UTC m=+0.019147542 image pull ec18d7baa60e8b1a0b126fbe5306acce11b1ed0ec5e3bcdddbf39b0a3c591791 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175 2024-03-31T23:38:15.907 INFO:journalctl@ceph.osd.0.smithi046.stdout:Mar 31 23:38:15 smithi046 podman[107544]: 2024-03-31 23:38:15.665097075 +0000 UTC m=+0.148820664 container create 0bdc465345f27f1e1454c06781c0b5596092c4e69a7b1aeb4f37973704ce3a69 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-0-deactivate, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240326, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=squid-384f31f, ceph=True) 2024-03-31T23:38:16.339 INFO:journalctl@ceph.osd.0.smithi046.stdout:Mar 31 23:38:16 smithi046 podman[107544]: 2024-03-31 23:38:16.026390019 +0000 UTC m=+0.510113621 container init 0bdc465345f27f1e1454c06781c0b5596092c4e69a7b1aeb4f37973704ce3a69 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-0-deactivate, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.build-date=20240326, RELEASE=squid-384f31f, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-03-31T23:38:16.339 INFO:journalctl@ceph.osd.0.smithi046.stdout:Mar 31 23:38:16 smithi046 podman[107544]: 2024-03-31 23:38:16.029554189 +0000 UTC m=+0.513277780 container start 0bdc465345f27f1e1454c06781c0b5596092c4e69a7b1aeb4f37973704ce3a69 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-0-deactivate, org.label-schema.build-date=20240326, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, RELEASE=squid-384f31f, ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-03-31T23:38:16.339 INFO:journalctl@ceph.osd.0.smithi046.stdout:Mar 31 23:38:16 smithi046 podman[107544]: 2024-03-31 23:38:16.109724506 +0000 UTC m=+0.593448112 container attach 0bdc465345f27f1e1454c06781c0b5596092c4e69a7b1aeb4f37973704ce3a69 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-0-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_CLEAN=True, RELEASE=squid-384f31f, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240326, CEPH_POINT_RELEASE=, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.schema-version=1.0) 2024-03-31T23:38:16.611 INFO:journalctl@ceph.osd.0.smithi046.stdout:Mar 31 23:38:16 smithi046 podman[107571]: 2024-03-31 23:38:16.364457571 +0000 UTC m=+0.019598278 container died 0bdc465345f27f1e1454c06781c0b5596092c4e69a7b1aeb4f37973704ce3a69 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-0-deactivate, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240326, GIT_CLEAN=True, RELEASE=squid-384f31f) 2024-03-31T23:38:16.987 DEBUG:teuthology.orchestra.run.smithi046:> sudo pkill -f 'journalctl -f -n 0 -u ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4@osd.0.service' 2024-03-31T23:38:17.034 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-31T23:38:17.035 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-03-31T23:38:17.035 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-03-31T23:38:17.035 DEBUG:teuthology.orchestra.run.smithi142:> sudo systemctl stop ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4@osd.1 2024-03-31T23:38:17.374 INFO:journalctl@ceph.osd.1.smithi142.stdout:Mar 31 23:38:17 smithi142 systemd[1]: Stopping Ceph osd.1 for 6a9d8f08-efb2-11ee-b647-cb9ed24678a4... 2024-03-31T23:38:17.693 INFO:journalctl@ceph.osd.1.smithi142.stdout:Mar 31 23:38:17 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-1[39861]: 2024-03-31T23:38:17.372+0000 7f23af1c8700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-31T23:38:17.693 INFO:journalctl@ceph.osd.1.smithi142.stdout:Mar 31 23:38:17 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-1[39861]: 2024-03-31T23:38:17.372+0000 7f23af1c8700 -1 osd.1 139 *** Got signal Terminated *** 2024-03-31T23:38:17.693 INFO:journalctl@ceph.osd.1.smithi142.stdout:Mar 31 23:38:17 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-1[39861]: 2024-03-31T23:38:17.372+0000 7f23af1c8700 -1 osd.1 139 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-31T23:38:23.097 INFO:journalctl@ceph.osd.1.smithi142.stdout:Mar 31 23:38:22 smithi142 podman[77716]: 2024-03-31 23:38:22.834812737 +0000 UTC m=+5.592177861 container died d72da928432bdc319655fd3bb4087cbe622a36b0c6f29762a381533fa07e7023 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-1, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, RELEASE=squid-384f31f, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240326, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-03-31T23:38:23.443 INFO:journalctl@ceph.osd.1.smithi142.stdout:Mar 31 23:38:23 smithi142 podman[77716]: 2024-03-31 23:38:23.215600066 +0000 UTC m=+5.972965195 container cleanup d72da928432bdc319655fd3bb4087cbe622a36b0c6f29762a381533fa07e7023 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-1, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-384f31f, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240326, CEPH_POINT_RELEASE=) 2024-03-31T23:38:23.444 INFO:journalctl@ceph.osd.1.smithi142.stdout:Mar 31 23:38:23 smithi142 bash[77716]: ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-1 2024-03-31T23:38:23.815 INFO:journalctl@ceph.osd.1.smithi142.stdout:Mar 31 23:38:23 smithi142 podman[77733]: 2024-03-31 23:38:23.509413733 +0000 UTC m=+0.673105678 container remove d72da928432bdc319655fd3bb4087cbe622a36b0c6f29762a381533fa07e7023 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-1, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240326, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, RELEASE=squid-384f31f, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-03-31T23:38:24.072 INFO:journalctl@ceph.osd.1.smithi142.stdout:Mar 31 23:38:23 smithi142 podman[77823]: 2024-03-31 23:38:23.718707134 +0000 UTC m=+0.017915348 image pull ec18d7baa60e8b1a0b126fbe5306acce11b1ed0ec5e3bcdddbf39b0a3c591791 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175 2024-03-31T23:38:24.072 INFO:journalctl@ceph.osd.1.smithi142.stdout:Mar 31 23:38:23 smithi142 podman[77823]: 2024-03-31 23:38:23.827056294 +0000 UTC m=+0.126264498 container create dba7caaf7984954debc4ae4575ffdc36c7a5c184003e1e2e4c501da8e212f04b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-1-deactivate, CEPH_POINT_RELEASE=, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240326, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , RELEASE=squid-384f31f, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-31T23:38:24.072 INFO:journalctl@ceph.osd.1.smithi142.stdout:Mar 31 23:38:24 smithi142 podman[77823]: 2024-03-31 23:38:24.072128546 +0000 UTC m=+0.371336759 container init dba7caaf7984954debc4ae4575ffdc36c7a5c184003e1e2e4c501da8e212f04b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-1-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.vendor=CentOS, RELEASE=squid-384f31f, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240326) 2024-03-31T23:38:24.407 INFO:journalctl@ceph.osd.1.smithi142.stdout:Mar 31 23:38:24 smithi142 podman[77823]: 2024-03-31 23:38:24.077219005 +0000 UTC m=+0.376427206 container start dba7caaf7984954debc4ae4575ffdc36c7a5c184003e1e2e4c501da8e212f04b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-1-deactivate, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.build-date=20240326, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=squid-384f31f, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) 2024-03-31T23:38:24.407 INFO:journalctl@ceph.osd.1.smithi142.stdout:Mar 31 23:38:24 smithi142 podman[77823]: 2024-03-31 23:38:24.163465736 +0000 UTC m=+0.462673948 container attach dba7caaf7984954debc4ae4575ffdc36c7a5c184003e1e2e4c501da8e212f04b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-1-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240326, RELEASE=squid-384f31f, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2) 2024-03-31T23:38:24.693 INFO:journalctl@ceph.osd.1.smithi142.stdout:Mar 31 23:38:24 smithi142 podman[77823]: 2024-03-31 23:38:24.407937012 +0000 UTC m=+0.707145213 container died dba7caaf7984954debc4ae4575ffdc36c7a5c184003e1e2e4c501da8e212f04b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-1-deactivate, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240326, RELEASE=squid-384f31f, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.33.5) 2024-03-31T23:38:25.078 DEBUG:teuthology.orchestra.run.smithi142:> sudo pkill -f 'journalctl -f -n 0 -u ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4@osd.1.service' 2024-03-31T23:38:25.121 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-31T23:38:25.122 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-03-31T23:38:25.122 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-03-31T23:38:25.122 DEBUG:teuthology.orchestra.run.smithi162:> sudo systemctl stop ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4@osd.2 2024-03-31T23:38:25.485 INFO:journalctl@ceph.osd.2.smithi162.stdout:Mar 31 23:38:25 smithi162 systemd[1]: Stopping Ceph osd.2 for 6a9d8f08-efb2-11ee-b647-cb9ed24678a4... 2024-03-31T23:38:25.755 INFO:journalctl@ceph.osd.2.smithi162.stdout:Mar 31 23:38:25 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-2[41385]: 2024-03-31T23:38:25.484+0000 7f986a26b700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-31T23:38:25.755 INFO:journalctl@ceph.osd.2.smithi162.stdout:Mar 31 23:38:25 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-2[41385]: 2024-03-31T23:38:25.484+0000 7f986a26b700 -1 osd.2 139 *** Got signal Terminated *** 2024-03-31T23:38:25.755 INFO:journalctl@ceph.osd.2.smithi162.stdout:Mar 31 23:38:25 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-2[41385]: 2024-03-31T23:38:25.484+0000 7f986a26b700 -1 osd.2 139 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-31T23:38:31.227 INFO:journalctl@ceph.osd.2.smithi162.stdout:Mar 31 23:38:30 smithi162 podman[78095]: 2024-03-31 23:38:30.917094038 +0000 UTC m=+5.606550311 container died 65c531cf5c18a199ae718034d42e88dd4a8ed37ced4ada14b1dfab786bf2d8cb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-2, GIT_CLEAN=True, RELEASE=squid-384f31f, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240326, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-03-31T23:38:31.545 INFO:journalctl@ceph.osd.2.smithi162.stdout:Mar 31 23:38:31 smithi162 podman[78095]: 2024-03-31 23:38:31.309826081 +0000 UTC m=+5.999282352 container cleanup 65c531cf5c18a199ae718034d42e88dd4a8ed37ced4ada14b1dfab786bf2d8cb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=squid-384f31f, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240326, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, io.buildah.version=1.33.5) 2024-03-31T23:38:31.545 INFO:journalctl@ceph.osd.2.smithi162.stdout:Mar 31 23:38:31 smithi162 bash[78095]: ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-2 2024-03-31T23:38:31.974 INFO:journalctl@ceph.osd.2.smithi162.stdout:Mar 31 23:38:31 smithi162 podman[78125]: 2024-03-31 23:38:31.611759719 +0000 UTC m=+0.688324325 container remove 65c531cf5c18a199ae718034d42e88dd4a8ed37ced4ada14b1dfab786bf2d8cb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-2, org.label-schema.build-date=20240326, maintainer=Guillaume Abrioux , RELEASE=squid-384f31f, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, ceph=True, GIT_CLEAN=True) 2024-03-31T23:38:32.241 INFO:journalctl@ceph.osd.2.smithi162.stdout:Mar 31 23:38:31 smithi162 podman[78216]: 2024-03-31 23:38:31.878000608 +0000 UTC m=+0.018037721 image pull ec18d7baa60e8b1a0b126fbe5306acce11b1ed0ec5e3bcdddbf39b0a3c591791 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175 2024-03-31T23:38:32.241 INFO:journalctl@ceph.osd.2.smithi162.stdout:Mar 31 23:38:31 smithi162 podman[78216]: 2024-03-31 23:38:31.996079536 +0000 UTC m=+0.136116637 container create eeeb97380e8048bca8623818fc3f48c3f6dcf0e49c083c102a798d0bf18de21d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-2-deactivate, RELEASE=squid-384f31f, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240326, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-31T23:38:32.241 INFO:journalctl@ceph.osd.2.smithi162.stdout:Mar 31 23:38:32 smithi162 podman[78216]: 2024-03-31 23:38:32.24105272 +0000 UTC m=+0.381089830 container init eeeb97380e8048bca8623818fc3f48c3f6dcf0e49c083c102a798d0bf18de21d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-2-deactivate, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.build-date=20240326, GIT_CLEAN=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, RELEASE=squid-384f31f, CEPH_POINT_RELEASE=) 2024-03-31T23:38:32.545 INFO:journalctl@ceph.osd.2.smithi162.stdout:Mar 31 23:38:32 smithi162 podman[78216]: 2024-03-31 23:38:32.244529148 +0000 UTC m=+0.384566259 container start eeeb97380e8048bca8623818fc3f48c3f6dcf0e49c083c102a798d0bf18de21d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-2-deactivate, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240326, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , RELEASE=squid-384f31f, GIT_BRANCH=HEAD, ceph=True) 2024-03-31T23:38:32.546 INFO:journalctl@ceph.osd.2.smithi162.stdout:Mar 31 23:38:32 smithi162 podman[78216]: 2024-03-31 23:38:32.332446335 +0000 UTC m=+0.472483442 container attach eeeb97380e8048bca8623818fc3f48c3f6dcf0e49c083c102a798d0bf18de21d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-2-deactivate, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, ceph=True, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=squid-384f31f, org.label-schema.build-date=20240326) 2024-03-31T23:38:33.045 INFO:journalctl@ceph.osd.2.smithi162.stdout:Mar 31 23:38:32 smithi162 podman[78216]: 2024-03-31 23:38:32.77804439 +0000 UTC m=+0.918081490 container died eeeb97380e8048bca8623818fc3f48c3f6dcf0e49c083c102a798d0bf18de21d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-2-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, RELEASE=squid-384f31f, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240326, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-03-31T23:38:33.493 DEBUG:teuthology.orchestra.run.smithi162:> sudo pkill -f 'journalctl -f -n 0 -u ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4@osd.2.service' 2024-03-31T23:38:33.535 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-31T23:38:33.535 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-03-31T23:38:33.535 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-03-31T23:38:33.535 DEBUG:teuthology.orchestra.run.smithi046:> sudo systemctl stop ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4@osd.3 2024-03-31T23:38:33.850 INFO:journalctl@ceph.osd.3.smithi046.stdout:Mar 31 23:38:33 smithi046 systemd[1]: Stopping Ceph osd.3 for 6a9d8f08-efb2-11ee-b647-cb9ed24678a4... 2024-03-31T23:38:34.111 INFO:journalctl@ceph.osd.3.smithi046.stdout:Mar 31 23:38:33 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-3[54227]: 2024-03-31T23:38:33.849+0000 7f3d991a7700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-31T23:38:34.112 INFO:journalctl@ceph.osd.3.smithi046.stdout:Mar 31 23:38:33 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-3[54227]: 2024-03-31T23:38:33.849+0000 7f3d991a7700 -1 osd.3 139 *** Got signal Terminated *** 2024-03-31T23:38:34.112 INFO:journalctl@ceph.osd.3.smithi046.stdout:Mar 31 23:38:33 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-3[54227]: 2024-03-31T23:38:33.849+0000 7f3d991a7700 -1 osd.3 139 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-31T23:38:35.361 INFO:journalctl@ceph.osd.6.smithi046.stdout:Mar 31 23:38:35 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-6[60760]: 2024-03-31T23:38:35.017+0000 7f8b7cce2700 -1 osd.6 139 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-31T23:38:12.034796+0000 front 2024-03-31T23:38:12.034926+0000 (oldest deadline 2024-03-31T23:38:34.335020+0000) 2024-03-31T23:38:36.361 INFO:journalctl@ceph.osd.6.smithi046.stdout:Mar 31 23:38:36 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-6[60760]: 2024-03-31T23:38:36.009+0000 7f8b7cce2700 -1 osd.6 139 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-31T23:38:12.034796+0000 front 2024-03-31T23:38:12.034926+0000 (oldest deadline 2024-03-31T23:38:34.335020+0000) 2024-03-31T23:38:36.362 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 31 23:38:36 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-9[67499]: 2024-03-31T23:38:36.022+0000 7fcc07615700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-31T23:38:10.469941+0000 front 2024-03-31T23:38:10.470388+0000 (oldest deadline 2024-03-31T23:38:35.170101+0000) 2024-03-31T23:38:37.045 INFO:journalctl@ceph.osd.5.smithi162.stdout:Mar 31 23:38:36 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-5[48185]: 2024-03-31T23:38:36.628+0000 7f433d90e700 -1 osd.5 139 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-31T23:38:11.433029+0000 front 2024-03-31T23:38:11.432884+0000 (oldest deadline 2024-03-31T23:38:36.132991+0000) 2024-03-31T23:38:37.361 INFO:journalctl@ceph.osd.6.smithi046.stdout:Mar 31 23:38:37 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-6[60760]: 2024-03-31T23:38:37.024+0000 7f8b7cce2700 -1 osd.6 139 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-31T23:38:12.034796+0000 front 2024-03-31T23:38:12.034926+0000 (oldest deadline 2024-03-31T23:38:34.335020+0000) 2024-03-31T23:38:37.362 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 31 23:38:37 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-9[67499]: 2024-03-31T23:38:37.057+0000 7fcc07615700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-31T23:38:10.469941+0000 front 2024-03-31T23:38:10.470388+0000 (oldest deadline 2024-03-31T23:38:35.170101+0000) 2024-03-31T23:38:37.545 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 31 23:38:37 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-8[55211]: 2024-03-31T23:38:37.223+0000 7fee85469700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-31T23:38:12.286566+0000 front 2024-03-31T23:38:12.286547+0000 (oldest deadline 2024-03-31T23:38:36.986433+0000) 2024-03-31T23:38:38.045 INFO:journalctl@ceph.osd.5.smithi162.stdout:Mar 31 23:38:37 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-5[48185]: 2024-03-31T23:38:37.675+0000 7f433d90e700 -1 osd.5 139 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-31T23:38:11.433029+0000 front 2024-03-31T23:38:11.432884+0000 (oldest deadline 2024-03-31T23:38:36.132991+0000) 2024-03-31T23:38:38.046 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:38:37 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:38:37.688+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-31T23:38:12.221366+0000 front 2024-03-31T23:38:12.221347+0000 (oldest deadline 2024-03-31T23:38:37.521357+0000) 2024-03-31T23:38:38.361 INFO:journalctl@ceph.osd.6.smithi046.stdout:Mar 31 23:38:38 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-6[60760]: 2024-03-31T23:38:38.041+0000 7f8b7cce2700 -1 osd.6 139 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-31T23:38:12.034796+0000 front 2024-03-31T23:38:12.034926+0000 (oldest deadline 2024-03-31T23:38:34.335020+0000) 2024-03-31T23:38:38.362 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 31 23:38:38 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-9[67499]: 2024-03-31T23:38:38.012+0000 7fcc07615700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-31T23:38:10.469941+0000 front 2024-03-31T23:38:10.470388+0000 (oldest deadline 2024-03-31T23:38:35.170101+0000) 2024-03-31T23:38:38.443 INFO:journalctl@ceph.osd.10.smithi142.stdout:Mar 31 23:38:38 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-10[60116]: 2024-03-31T23:38:38.073+0000 7f64c04a8700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-31T23:38:13.357804+0000 front 2024-03-31T23:38:13.357906+0000 (oldest deadline 2024-03-31T23:38:38.057730+0000) 2024-03-31T23:38:38.545 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 31 23:38:38 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-8[55211]: 2024-03-31T23:38:38.199+0000 7fee85469700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-31T23:38:12.286566+0000 front 2024-03-31T23:38:12.286547+0000 (oldest deadline 2024-03-31T23:38:36.986433+0000) 2024-03-31T23:38:39.045 INFO:journalctl@ceph.osd.5.smithi162.stdout:Mar 31 23:38:38 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-5[48185]: 2024-03-31T23:38:38.677+0000 7f433d90e700 -1 osd.5 139 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-31T23:38:11.433029+0000 front 2024-03-31T23:38:11.432884+0000 (oldest deadline 2024-03-31T23:38:36.132991+0000) 2024-03-31T23:38:39.046 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:38:38 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:38:38.652+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-31T23:38:12.221366+0000 front 2024-03-31T23:38:12.221347+0000 (oldest deadline 2024-03-31T23:38:37.521357+0000) 2024-03-31T23:38:39.285 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 31 23:38:39 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-9[67499]: 2024-03-31T23:38:39.006+0000 7fcc07615700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-31T23:38:10.469941+0000 front 2024-03-31T23:38:10.470388+0000 (oldest deadline 2024-03-31T23:38:35.170101+0000) 2024-03-31T23:38:39.285 INFO:journalctl@ceph.osd.6.smithi046.stdout:Mar 31 23:38:39 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-6[60760]: 2024-03-31T23:38:39.067+0000 7f8b7cce2700 -1 osd.6 139 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-31T23:38:12.034796+0000 front 2024-03-31T23:38:12.034926+0000 (oldest deadline 2024-03-31T23:38:34.335020+0000) 2024-03-31T23:38:39.443 INFO:journalctl@ceph.osd.10.smithi142.stdout:Mar 31 23:38:39 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-10[60116]: 2024-03-31T23:38:39.046+0000 7f64c04a8700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-31T23:38:13.357804+0000 front 2024-03-31T23:38:13.357906+0000 (oldest deadline 2024-03-31T23:38:38.057730+0000) 2024-03-31T23:38:39.545 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 31 23:38:39 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-8[55211]: 2024-03-31T23:38:39.230+0000 7fee85469700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-31T23:38:12.286566+0000 front 2024-03-31T23:38:12.286547+0000 (oldest deadline 2024-03-31T23:38:36.986433+0000) 2024-03-31T23:38:39.608 INFO:journalctl@ceph.osd.3.smithi046.stdout:Mar 31 23:38:39 smithi046 podman[107652]: 2024-03-31 23:38:39.285319495 +0000 UTC m=+5.550660668 container died 39269df5e56dd5f439244362348782842f4fa9d9c6c8f342540967e7f4e6039e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-3, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-384f31f, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.build-date=20240326, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-03-31T23:38:39.861 INFO:journalctl@ceph.osd.3.smithi046.stdout:Mar 31 23:38:39 smithi046 podman[107652]: 2024-03-31 23:38:39.725738257 +0000 UTC m=+5.991079416 container cleanup 39269df5e56dd5f439244362348782842f4fa9d9c6c8f342540967e7f4e6039e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-3, RELEASE=squid-384f31f, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240326, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=) 2024-03-31T23:38:39.862 INFO:journalctl@ceph.osd.3.smithi046.stdout:Mar 31 23:38:39 smithi046 bash[107652]: ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-3 2024-03-31T23:38:40.045 INFO:journalctl@ceph.osd.5.smithi162.stdout:Mar 31 23:38:39 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-5[48185]: 2024-03-31T23:38:39.636+0000 7f433d90e700 -1 osd.5 139 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-31T23:38:11.433029+0000 front 2024-03-31T23:38:11.432884+0000 (oldest deadline 2024-03-31T23:38:36.132991+0000) 2024-03-31T23:38:40.046 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:38:39 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:38:39.673+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-31T23:38:12.221366+0000 front 2024-03-31T23:38:12.221347+0000 (oldest deadline 2024-03-31T23:38:37.521357+0000) 2024-03-31T23:38:40.361 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 31 23:38:40 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-9[67499]: 2024-03-31T23:38:40.054+0000 7fcc07615700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-31T23:38:10.469941+0000 front 2024-03-31T23:38:10.470388+0000 (oldest deadline 2024-03-31T23:38:35.170101+0000) 2024-03-31T23:38:40.361 INFO:journalctl@ceph.osd.3.smithi046.stdout:Mar 31 23:38:39 smithi046 podman[107669]: 2024-03-31 23:38:39.94376248 +0000 UTC m=+0.656227072 container remove 39269df5e56dd5f439244362348782842f4fa9d9c6c8f342540967e7f4e6039e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-3, ceph=True, GIT_CLEAN=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=squid-384f31f, org.label-schema.build-date=20240326, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-31T23:38:40.362 INFO:journalctl@ceph.osd.6.smithi046.stdout:Mar 31 23:38:40 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-6[60760]: 2024-03-31T23:38:40.041+0000 7f8b7cce2700 -1 osd.6 139 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-31T23:38:12.034796+0000 front 2024-03-31T23:38:12.034926+0000 (oldest deadline 2024-03-31T23:38:34.335020+0000) 2024-03-31T23:38:40.443 INFO:journalctl@ceph.osd.10.smithi142.stdout:Mar 31 23:38:40 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-10[60116]: 2024-03-31T23:38:40.054+0000 7f64c04a8700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-31T23:38:13.357804+0000 front 2024-03-31T23:38:13.357906+0000 (oldest deadline 2024-03-31T23:38:38.057730+0000) 2024-03-31T23:38:40.545 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 31 23:38:40 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-8[55211]: 2024-03-31T23:38:40.230+0000 7fee85469700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-31T23:38:12.286566+0000 front 2024-03-31T23:38:12.286547+0000 (oldest deadline 2024-03-31T23:38:36.986433+0000) 2024-03-31T23:38:40.772 INFO:journalctl@ceph.osd.3.smithi046.stdout:Mar 31 23:38:40 smithi046 podman[107762]: 2024-03-31 23:38:40.288706527 +0000 UTC m=+0.018357996 image pull ec18d7baa60e8b1a0b126fbe5306acce11b1ed0ec5e3bcdddbf39b0a3c591791 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175 2024-03-31T23:38:40.772 INFO:journalctl@ceph.osd.3.smithi046.stdout:Mar 31 23:38:40 smithi046 podman[107762]: 2024-03-31 23:38:40.41136133 +0000 UTC m=+0.141012780 container create 2ee44605b9541983cddbbaba5d43ae9c82b40faa2fbaeea6fe8b7a0b91c5ef23 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-3-deactivate, org.label-schema.license=GPLv2, RELEASE=squid-384f31f, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240326, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2024-03-31T23:38:41.045 INFO:journalctl@ceph.osd.5.smithi162.stdout:Mar 31 23:38:40 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-5[48185]: 2024-03-31T23:38:40.642+0000 7f433d90e700 -1 osd.5 139 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-31T23:38:11.433029+0000 front 2024-03-31T23:38:11.432884+0000 (oldest deadline 2024-03-31T23:38:36.132991+0000) 2024-03-31T23:38:41.046 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:38:40 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:38:40.723+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-31T23:38:12.221366+0000 front 2024-03-31T23:38:12.221347+0000 (oldest deadline 2024-03-31T23:38:37.521357+0000) 2024-03-31T23:38:41.070 INFO:journalctl@ceph.osd.3.smithi046.stdout:Mar 31 23:38:40 smithi046 podman[107762]: 2024-03-31 23:38:40.77234832 +0000 UTC m=+0.501999770 container init 2ee44605b9541983cddbbaba5d43ae9c82b40faa2fbaeea6fe8b7a0b91c5ef23 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-3-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=squid-384f31f, org.label-schema.build-date=20240326, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.33.5) 2024-03-31T23:38:41.071 INFO:journalctl@ceph.osd.3.smithi046.stdout:Mar 31 23:38:40 smithi046 podman[107762]: 2024-03-31 23:38:40.77547207 +0000 UTC m=+0.505123527 container start 2ee44605b9541983cddbbaba5d43ae9c82b40faa2fbaeea6fe8b7a0b91c5ef23 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-3-deactivate, org.label-schema.build-date=20240326, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_CLEAN=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, RELEASE=squid-384f31f, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-31T23:38:41.071 INFO:journalctl@ceph.osd.3.smithi046.stdout:Mar 31 23:38:40 smithi046 podman[107762]: 2024-03-31 23:38:40.864232009 +0000 UTC m=+0.593883459 container attach 2ee44605b9541983cddbbaba5d43ae9c82b40faa2fbaeea6fe8b7a0b91c5ef23 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-3-deactivate, RELEASE=squid-384f31f, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240326, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-31T23:38:41.361 INFO:journalctl@ceph.osd.6.smithi046.stdout:Mar 31 23:38:41 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-6[60760]: 2024-03-31T23:38:41.074+0000 7f8b7cce2700 -1 osd.6 139 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-31T23:38:12.034796+0000 front 2024-03-31T23:38:12.034926+0000 (oldest deadline 2024-03-31T23:38:34.335020+0000) 2024-03-31T23:38:41.362 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 31 23:38:41 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-9[67499]: 2024-03-31T23:38:41.070+0000 7fcc07615700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-31T23:38:10.469941+0000 front 2024-03-31T23:38:10.470388+0000 (oldest deadline 2024-03-31T23:38:35.170101+0000) 2024-03-31T23:38:41.362 INFO:journalctl@ceph.osd.3.smithi046.stdout:Mar 31 23:38:41 smithi046 podman[107762]: 2024-03-31 23:38:41.083142031 +0000 UTC m=+0.812793473 container died 2ee44605b9541983cddbbaba5d43ae9c82b40faa2fbaeea6fe8b7a0b91c5ef23 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-3-deactivate, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240326, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-384f31f) 2024-03-31T23:38:41.443 INFO:journalctl@ceph.osd.10.smithi142.stdout:Mar 31 23:38:41 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-10[60116]: 2024-03-31T23:38:41.053+0000 7f64c04a8700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-31T23:38:13.357804+0000 front 2024-03-31T23:38:13.357906+0000 (oldest deadline 2024-03-31T23:38:38.057730+0000) 2024-03-31T23:38:41.545 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 31 23:38:41 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-8[55211]: 2024-03-31T23:38:41.183+0000 7fee85469700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-31T23:38:12.286566+0000 front 2024-03-31T23:38:12.286547+0000 (oldest deadline 2024-03-31T23:38:36.986433+0000) 2024-03-31T23:38:41.750 DEBUG:teuthology.orchestra.run.smithi046:> sudo pkill -f 'journalctl -f -n 0 -u ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4@osd.3.service' 2024-03-31T23:38:41.792 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-31T23:38:41.792 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-03-31T23:38:41.793 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-03-31T23:38:41.793 DEBUG:teuthology.orchestra.run.smithi142:> sudo systemctl stop ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4@osd.4 2024-03-31T23:38:42.046 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:38:41 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:38:41.685+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-31T23:38:12.221366+0000 front 2024-03-31T23:38:12.221347+0000 (oldest deadline 2024-03-31T23:38:37.521357+0000) 2024-03-31T23:38:42.046 INFO:journalctl@ceph.osd.5.smithi162.stdout:Mar 31 23:38:41 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-5[48185]: 2024-03-31T23:38:41.626+0000 7f433d90e700 -1 osd.5 139 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-31T23:38:11.433029+0000 front 2024-03-31T23:38:11.432884+0000 (oldest deadline 2024-03-31T23:38:36.132991+0000) 2024-03-31T23:38:42.079 INFO:journalctl@ceph.osd.4.smithi142.stdout:Mar 31 23:38:41 smithi142 systemd[1]: Stopping Ceph osd.4 for 6a9d8f08-efb2-11ee-b647-cb9ed24678a4... 2024-03-31T23:38:42.361 INFO:journalctl@ceph.osd.6.smithi046.stdout:Mar 31 23:38:42 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-6[60760]: 2024-03-31T23:38:42.088+0000 7f8b7cce2700 -1 osd.6 139 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-31T23:38:12.034796+0000 front 2024-03-31T23:38:12.034926+0000 (oldest deadline 2024-03-31T23:38:34.335020+0000) 2024-03-31T23:38:42.361 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 31 23:38:42 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-9[67499]: 2024-03-31T23:38:42.063+0000 7fcc07615700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-31T23:38:10.469941+0000 front 2024-03-31T23:38:10.470388+0000 (oldest deadline 2024-03-31T23:38:35.170101+0000) 2024-03-31T23:38:42.443 INFO:journalctl@ceph.osd.10.smithi142.stdout:Mar 31 23:38:42 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-10[60116]: 2024-03-31T23:38:42.078+0000 7f64c04a8700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-31T23:38:13.357804+0000 front 2024-03-31T23:38:13.357906+0000 (oldest deadline 2024-03-31T23:38:38.057730+0000) 2024-03-31T23:38:42.443 INFO:journalctl@ceph.osd.4.smithi142.stdout:Mar 31 23:38:42 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-4[46503]: 2024-03-31T23:38:42.128+0000 7f635a834700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-31T23:38:42.443 INFO:journalctl@ceph.osd.4.smithi142.stdout:Mar 31 23:38:42 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-4[46503]: 2024-03-31T23:38:42.128+0000 7f635a834700 -1 osd.4 139 *** Got signal Terminated *** 2024-03-31T23:38:42.443 INFO:journalctl@ceph.osd.4.smithi142.stdout:Mar 31 23:38:42 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-4[46503]: 2024-03-31T23:38:42.128+0000 7f635a834700 -1 osd.4 139 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-31T23:38:42.545 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 31 23:38:42 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-8[55211]: 2024-03-31T23:38:42.156+0000 7fee85469700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-31T23:38:12.286566+0000 front 2024-03-31T23:38:12.286547+0000 (oldest deadline 2024-03-31T23:38:36.986433+0000) 2024-03-31T23:38:43.045 INFO:journalctl@ceph.osd.5.smithi162.stdout:Mar 31 23:38:42 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-5[48185]: 2024-03-31T23:38:42.619+0000 7f433d90e700 -1 osd.5 139 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-31T23:38:11.433029+0000 front 2024-03-31T23:38:11.432884+0000 (oldest deadline 2024-03-31T23:38:36.132991+0000) 2024-03-31T23:38:43.046 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:38:42 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:38:42.693+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-31T23:38:12.221366+0000 front 2024-03-31T23:38:12.221347+0000 (oldest deadline 2024-03-31T23:38:37.521357+0000) 2024-03-31T23:38:43.362 INFO:journalctl@ceph.osd.6.smithi046.stdout:Mar 31 23:38:43 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-6[60760]: 2024-03-31T23:38:43.096+0000 7f8b7cce2700 -1 osd.6 139 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-31T23:38:12.034796+0000 front 2024-03-31T23:38:12.034926+0000 (oldest deadline 2024-03-31T23:38:34.335020+0000) 2024-03-31T23:38:43.362 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 31 23:38:43 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-9[67499]: 2024-03-31T23:38:43.058+0000 7fcc07615700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-31T23:38:10.469941+0000 front 2024-03-31T23:38:10.470388+0000 (oldest deadline 2024-03-31T23:38:35.170101+0000) 2024-03-31T23:38:43.443 INFO:journalctl@ceph.osd.10.smithi142.stdout:Mar 31 23:38:43 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-10[60116]: 2024-03-31T23:38:43.061+0000 7f64c04a8700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-31T23:38:13.357804+0000 front 2024-03-31T23:38:13.357906+0000 (oldest deadline 2024-03-31T23:38:38.057730+0000) 2024-03-31T23:38:43.539 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 31 23:38:43 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-8[55211]: 2024-03-31T23:38:43.192+0000 7fee85469700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-31T23:38:12.286566+0000 front 2024-03-31T23:38:12.286547+0000 (oldest deadline 2024-03-31T23:38:36.986433+0000) 2024-03-31T23:38:43.795 INFO:journalctl@ceph.osd.5.smithi162.stdout:Mar 31 23:38:43 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-5[48185]: 2024-03-31T23:38:43.578+0000 7f433d90e700 -1 osd.5 139 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-31T23:38:11.433029+0000 front 2024-03-31T23:38:11.432884+0000 (oldest deadline 2024-03-31T23:38:36.132991+0000) 2024-03-31T23:38:43.796 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:38:43 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:38:43.706+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-31T23:38:12.221366+0000 front 2024-03-31T23:38:12.221347+0000 (oldest deadline 2024-03-31T23:38:37.521357+0000) 2024-03-31T23:38:44.361 INFO:journalctl@ceph.osd.6.smithi046.stdout:Mar 31 23:38:44 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-6[60760]: 2024-03-31T23:38:44.129+0000 7f8b7cce2700 -1 osd.6 139 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-31T23:38:12.034796+0000 front 2024-03-31T23:38:12.034926+0000 (oldest deadline 2024-03-31T23:38:34.335020+0000) 2024-03-31T23:38:44.362 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 31 23:38:44 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-9[67499]: 2024-03-31T23:38:44.049+0000 7fcc07615700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-31T23:38:10.469941+0000 front 2024-03-31T23:38:10.470388+0000 (oldest deadline 2024-03-31T23:38:35.170101+0000) 2024-03-31T23:38:44.362 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 31 23:38:44 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-9[67499]: 2024-03-31T23:38:44.049+0000 7fcc07615700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.142:6804 osd.1 since back 2024-03-31T23:38:21.571770+0000 front 2024-03-31T23:38:21.571801+0000 (oldest deadline 2024-03-31T23:38:43.871904+0000) 2024-03-31T23:38:44.443 INFO:journalctl@ceph.osd.10.smithi142.stdout:Mar 31 23:38:44 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-10[60116]: 2024-03-31T23:38:44.105+0000 7f64c04a8700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-31T23:38:13.357804+0000 front 2024-03-31T23:38:13.357906+0000 (oldest deadline 2024-03-31T23:38:38.057730+0000) 2024-03-31T23:38:44.545 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 31 23:38:44 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-8[55211]: 2024-03-31T23:38:44.205+0000 7fee85469700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-31T23:38:12.286566+0000 front 2024-03-31T23:38:12.286547+0000 (oldest deadline 2024-03-31T23:38:36.986433+0000) 2024-03-31T23:38:45.045 INFO:journalctl@ceph.osd.5.smithi162.stdout:Mar 31 23:38:44 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-5[48185]: 2024-03-31T23:38:44.560+0000 7f433d90e700 -1 osd.5 139 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-31T23:38:11.433029+0000 front 2024-03-31T23:38:11.432884+0000 (oldest deadline 2024-03-31T23:38:36.132991+0000) 2024-03-31T23:38:45.046 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:38:44 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:38:44.714+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-31T23:38:12.221366+0000 front 2024-03-31T23:38:12.221347+0000 (oldest deadline 2024-03-31T23:38:37.521357+0000) 2024-03-31T23:38:45.148 INFO:journalctl@ceph.osd.4.smithi142.stdout:Mar 31 23:38:44 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-4[46503]: 2024-03-31T23:38:44.849+0000 7f6352e20700 -1 osd.4 139 heartbeat_check: no reply from 172.21.15.142:6804 osd.1 since back 2024-03-31T23:38:19.395222+0000 front 2024-03-31T23:38:19.395258+0000 (oldest deadline 2024-03-31T23:38:44.695354+0000) 2024-03-31T23:38:45.361 INFO:journalctl@ceph.osd.6.smithi046.stdout:Mar 31 23:38:45 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-6[60760]: 2024-03-31T23:38:45.141+0000 7f8b7cce2700 -1 osd.6 139 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-31T23:38:12.034796+0000 front 2024-03-31T23:38:12.034926+0000 (oldest deadline 2024-03-31T23:38:34.335020+0000) 2024-03-31T23:38:45.361 INFO:journalctl@ceph.osd.6.smithi046.stdout:Mar 31 23:38:45 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-6[60760]: 2024-03-31T23:38:45.141+0000 7f8b7cce2700 -1 osd.6 139 heartbeat_check: no reply from 172.21.15.142:6804 osd.1 since back 2024-03-31T23:38:19.636274+0000 front 2024-03-31T23:38:19.636207+0000 (oldest deadline 2024-03-31T23:38:44.936378+0000) 2024-03-31T23:38:45.362 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 31 23:38:45 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-9[67499]: 2024-03-31T23:38:45.031+0000 7fcc07615700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-31T23:38:10.469941+0000 front 2024-03-31T23:38:10.470388+0000 (oldest deadline 2024-03-31T23:38:35.170101+0000) 2024-03-31T23:38:45.362 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 31 23:38:45 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-9[67499]: 2024-03-31T23:38:45.031+0000 7fcc07615700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.142:6804 osd.1 since back 2024-03-31T23:38:21.571770+0000 front 2024-03-31T23:38:21.571801+0000 (oldest deadline 2024-03-31T23:38:43.871904+0000) 2024-03-31T23:38:45.443 INFO:journalctl@ceph.osd.10.smithi142.stdout:Mar 31 23:38:45 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-10[60116]: 2024-03-31T23:38:45.147+0000 7f64c04a8700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-31T23:38:13.357804+0000 front 2024-03-31T23:38:13.357906+0000 (oldest deadline 2024-03-31T23:38:38.057730+0000) 2024-03-31T23:38:45.443 INFO:journalctl@ceph.osd.10.smithi142.stdout:Mar 31 23:38:45 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-10[60116]: 2024-03-31T23:38:45.147+0000 7f64c04a8700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.142:6804 osd.1 since back 2024-03-31T23:38:20.958341+0000 front 2024-03-31T23:38:20.958408+0000 (oldest deadline 2024-03-31T23:38:45.058481+0000) 2024-03-31T23:38:45.545 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 31 23:38:45 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-8[55211]: 2024-03-31T23:38:45.216+0000 7fee85469700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-31T23:38:12.286566+0000 front 2024-03-31T23:38:12.286547+0000 (oldest deadline 2024-03-31T23:38:36.986433+0000) 2024-03-31T23:38:46.045 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:38:45 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:38:45.720+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-31T23:38:12.221366+0000 front 2024-03-31T23:38:12.221347+0000 (oldest deadline 2024-03-31T23:38:37.521357+0000) 2024-03-31T23:38:46.046 INFO:journalctl@ceph.osd.5.smithi162.stdout:Mar 31 23:38:45 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-5[48185]: 2024-03-31T23:38:45.584+0000 7f433d90e700 -1 osd.5 139 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-31T23:38:11.433029+0000 front 2024-03-31T23:38:11.432884+0000 (oldest deadline 2024-03-31T23:38:36.132991+0000) 2024-03-31T23:38:46.046 INFO:journalctl@ceph.osd.5.smithi162.stdout:Mar 31 23:38:45 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-5[48185]: 2024-03-31T23:38:45.584+0000 7f433d90e700 -1 osd.5 139 heartbeat_check: no reply from 172.21.15.142:6804 osd.1 since back 2024-03-31T23:38:21.735080+0000 front 2024-03-31T23:38:21.735097+0000 (oldest deadline 2024-03-31T23:38:45.235148+0000) 2024-03-31T23:38:46.158 INFO:journalctl@ceph.osd.4.smithi142.stdout:Mar 31 23:38:45 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-4[46503]: 2024-03-31T23:38:45.837+0000 7f6352e20700 -1 osd.4 139 heartbeat_check: no reply from 172.21.15.142:6804 osd.1 since back 2024-03-31T23:38:19.395222+0000 front 2024-03-31T23:38:19.395258+0000 (oldest deadline 2024-03-31T23:38:44.695354+0000) 2024-03-31T23:38:46.361 INFO:journalctl@ceph.osd.6.smithi046.stdout:Mar 31 23:38:46 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-6[60760]: 2024-03-31T23:38:46.122+0000 7f8b7cce2700 -1 osd.6 139 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-31T23:38:12.034796+0000 front 2024-03-31T23:38:12.034926+0000 (oldest deadline 2024-03-31T23:38:34.335020+0000) 2024-03-31T23:38:46.361 INFO:journalctl@ceph.osd.6.smithi046.stdout:Mar 31 23:38:46 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-6[60760]: 2024-03-31T23:38:46.122+0000 7f8b7cce2700 -1 osd.6 139 heartbeat_check: no reply from 172.21.15.142:6804 osd.1 since back 2024-03-31T23:38:19.636274+0000 front 2024-03-31T23:38:19.636207+0000 (oldest deadline 2024-03-31T23:38:44.936378+0000) 2024-03-31T23:38:46.362 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 31 23:38:46 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-9[67499]: 2024-03-31T23:38:46.011+0000 7fcc07615700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-31T23:38:10.469941+0000 front 2024-03-31T23:38:10.470388+0000 (oldest deadline 2024-03-31T23:38:35.170101+0000) 2024-03-31T23:38:46.362 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 31 23:38:46 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-9[67499]: 2024-03-31T23:38:46.011+0000 7fcc07615700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.142:6804 osd.1 since back 2024-03-31T23:38:21.571770+0000 front 2024-03-31T23:38:21.571801+0000 (oldest deadline 2024-03-31T23:38:43.871904+0000) 2024-03-31T23:38:46.443 INFO:journalctl@ceph.osd.10.smithi142.stdout:Mar 31 23:38:46 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-10[60116]: 2024-03-31T23:38:46.157+0000 7f64c04a8700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-31T23:38:13.357804+0000 front 2024-03-31T23:38:13.357906+0000 (oldest deadline 2024-03-31T23:38:38.057730+0000) 2024-03-31T23:38:46.443 INFO:journalctl@ceph.osd.10.smithi142.stdout:Mar 31 23:38:46 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-10[60116]: 2024-03-31T23:38:46.157+0000 7f64c04a8700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.142:6804 osd.1 since back 2024-03-31T23:38:20.958341+0000 front 2024-03-31T23:38:20.958408+0000 (oldest deadline 2024-03-31T23:38:45.058481+0000) 2024-03-31T23:38:46.545 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 31 23:38:46 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-8[55211]: 2024-03-31T23:38:46.187+0000 7fee85469700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-31T23:38:12.286566+0000 front 2024-03-31T23:38:12.286547+0000 (oldest deadline 2024-03-31T23:38:36.986433+0000) 2024-03-31T23:38:47.045 INFO:journalctl@ceph.osd.5.smithi162.stdout:Mar 31 23:38:46 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-5[48185]: 2024-03-31T23:38:46.605+0000 7f433d90e700 -1 osd.5 139 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-31T23:38:11.433029+0000 front 2024-03-31T23:38:11.432884+0000 (oldest deadline 2024-03-31T23:38:36.132991+0000) 2024-03-31T23:38:47.046 INFO:journalctl@ceph.osd.5.smithi162.stdout:Mar 31 23:38:46 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-5[48185]: 2024-03-31T23:38:46.605+0000 7f433d90e700 -1 osd.5 139 heartbeat_check: no reply from 172.21.15.142:6804 osd.1 since back 2024-03-31T23:38:21.735080+0000 front 2024-03-31T23:38:21.735097+0000 (oldest deadline 2024-03-31T23:38:45.235148+0000) 2024-03-31T23:38:47.046 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:38:46 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:38:46.768+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-31T23:38:12.221366+0000 front 2024-03-31T23:38:12.221347+0000 (oldest deadline 2024-03-31T23:38:37.521357+0000) 2024-03-31T23:38:47.046 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:38:46 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:38:46.768+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.142:6804 osd.1 since back 2024-03-31T23:38:22.222192+0000 front 2024-03-31T23:38:22.222213+0000 (oldest deadline 2024-03-31T23:38:45.722340+0000) 2024-03-31T23:38:47.184 INFO:journalctl@ceph.osd.4.smithi142.stdout:Mar 31 23:38:46 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-4[46503]: 2024-03-31T23:38:46.858+0000 7f6352e20700 -1 osd.4 139 heartbeat_check: no reply from 172.21.15.142:6804 osd.1 since back 2024-03-31T23:38:19.395222+0000 front 2024-03-31T23:38:19.395258+0000 (oldest deadline 2024-03-31T23:38:44.695354+0000) 2024-03-31T23:38:47.361 INFO:journalctl@ceph.osd.6.smithi046.stdout:Mar 31 23:38:47 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-6[60760]: 2024-03-31T23:38:47.075+0000 7f8b7cce2700 -1 osd.6 139 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-31T23:38:12.034796+0000 front 2024-03-31T23:38:12.034926+0000 (oldest deadline 2024-03-31T23:38:34.335020+0000) 2024-03-31T23:38:47.362 INFO:journalctl@ceph.osd.6.smithi046.stdout:Mar 31 23:38:47 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-6[60760]: 2024-03-31T23:38:47.075+0000 7f8b7cce2700 -1 osd.6 139 heartbeat_check: no reply from 172.21.15.142:6804 osd.1 since back 2024-03-31T23:38:19.636274+0000 front 2024-03-31T23:38:19.636207+0000 (oldest deadline 2024-03-31T23:38:44.936378+0000) 2024-03-31T23:38:47.362 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 31 23:38:46 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-9[67499]: 2024-03-31T23:38:46.966+0000 7fcc07615700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-31T23:38:10.469941+0000 front 2024-03-31T23:38:10.470388+0000 (oldest deadline 2024-03-31T23:38:35.170101+0000) 2024-03-31T23:38:47.362 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 31 23:38:46 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-9[67499]: 2024-03-31T23:38:46.966+0000 7fcc07615700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.142:6804 osd.1 since back 2024-03-31T23:38:21.571770+0000 front 2024-03-31T23:38:21.571801+0000 (oldest deadline 2024-03-31T23:38:43.871904+0000) 2024-03-31T23:38:47.443 INFO:journalctl@ceph.osd.10.smithi142.stdout:Mar 31 23:38:47 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-10[60116]: 2024-03-31T23:38:47.183+0000 7f64c04a8700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-31T23:38:13.357804+0000 front 2024-03-31T23:38:13.357906+0000 (oldest deadline 2024-03-31T23:38:38.057730+0000) 2024-03-31T23:38:47.443 INFO:journalctl@ceph.osd.10.smithi142.stdout:Mar 31 23:38:47 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-10[60116]: 2024-03-31T23:38:47.183+0000 7f64c04a8700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.142:6804 osd.1 since back 2024-03-31T23:38:20.958341+0000 front 2024-03-31T23:38:20.958408+0000 (oldest deadline 2024-03-31T23:38:45.058481+0000) 2024-03-31T23:38:47.545 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 31 23:38:47 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-8[55211]: 2024-03-31T23:38:47.186+0000 7fee85469700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-31T23:38:12.286566+0000 front 2024-03-31T23:38:12.286547+0000 (oldest deadline 2024-03-31T23:38:36.986433+0000) 2024-03-31T23:38:47.789 INFO:journalctl@ceph.osd.4.smithi142.stdout:Mar 31 23:38:47 smithi142 podman[77942]: 2024-03-31 23:38:47.521629046 +0000 UTC m=+5.526928751 container died 2f2a509d7641c725d57e93d687709ef81a5ccf251646de28cdcb84e215d2c88c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-4, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240326, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, ceph=True, CEPH_POINT_RELEASE=, RELEASE=squid-384f31f, org.label-schema.license=GPLv2) 2024-03-31T23:38:48.045 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:38:47 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:38:47.723+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-31T23:38:12.221366+0000 front 2024-03-31T23:38:12.221347+0000 (oldest deadline 2024-03-31T23:38:37.521357+0000) 2024-03-31T23:38:48.045 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:38:47 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:38:47.723+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.142:6804 osd.1 since back 2024-03-31T23:38:22.222192+0000 front 2024-03-31T23:38:22.222213+0000 (oldest deadline 2024-03-31T23:38:45.722340+0000) 2024-03-31T23:38:48.046 INFO:journalctl@ceph.osd.5.smithi162.stdout:Mar 31 23:38:47 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-5[48185]: 2024-03-31T23:38:47.636+0000 7f433d90e700 -1 osd.5 139 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-31T23:38:11.433029+0000 front 2024-03-31T23:38:11.432884+0000 (oldest deadline 2024-03-31T23:38:36.132991+0000) 2024-03-31T23:38:48.046 INFO:journalctl@ceph.osd.5.smithi162.stdout:Mar 31 23:38:47 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-5[48185]: 2024-03-31T23:38:47.636+0000 7f433d90e700 -1 osd.5 139 heartbeat_check: no reply from 172.21.15.142:6804 osd.1 since back 2024-03-31T23:38:21.735080+0000 front 2024-03-31T23:38:21.735097+0000 (oldest deadline 2024-03-31T23:38:45.235148+0000) 2024-03-31T23:38:48.182 INFO:journalctl@ceph.osd.4.smithi142.stdout:Mar 31 23:38:47 smithi142 podman[77942]: 2024-03-31 23:38:47.879194274 +0000 UTC m=+5.884493974 container cleanup 2f2a509d7641c725d57e93d687709ef81a5ccf251646de28cdcb84e215d2c88c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-4, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, CEPH_POINT_RELEASE=, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240326, org.label-schema.vendor=CentOS, RELEASE=squid-384f31f, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2) 2024-03-31T23:38:48.182 INFO:journalctl@ceph.osd.4.smithi142.stdout:Mar 31 23:38:47 smithi142 bash[77942]: ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-4 2024-03-31T23:38:48.361 INFO:journalctl@ceph.osd.6.smithi046.stdout:Mar 31 23:38:48 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-6[60760]: 2024-03-31T23:38:48.083+0000 7f8b7cce2700 -1 osd.6 139 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-31T23:38:12.034796+0000 front 2024-03-31T23:38:12.034926+0000 (oldest deadline 2024-03-31T23:38:34.335020+0000) 2024-03-31T23:38:48.362 INFO:journalctl@ceph.osd.6.smithi046.stdout:Mar 31 23:38:48 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-6[60760]: 2024-03-31T23:38:48.083+0000 7f8b7cce2700 -1 osd.6 139 heartbeat_check: no reply from 172.21.15.142:6804 osd.1 since back 2024-03-31T23:38:19.636274+0000 front 2024-03-31T23:38:19.636207+0000 (oldest deadline 2024-03-31T23:38:44.936378+0000) 2024-03-31T23:38:48.362 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 31 23:38:47 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-9[67499]: 2024-03-31T23:38:47.969+0000 7fcc07615700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-31T23:38:10.469941+0000 front 2024-03-31T23:38:10.470388+0000 (oldest deadline 2024-03-31T23:38:35.170101+0000) 2024-03-31T23:38:48.362 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 31 23:38:47 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-9[67499]: 2024-03-31T23:38:47.969+0000 7fcc07615700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.142:6804 osd.1 since back 2024-03-31T23:38:21.571770+0000 front 2024-03-31T23:38:21.571801+0000 (oldest deadline 2024-03-31T23:38:43.871904+0000) 2024-03-31T23:38:48.438 INFO:journalctl@ceph.osd.10.smithi142.stdout:Mar 31 23:38:48 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-10[60116]: 2024-03-31T23:38:48.181+0000 7f64c04a8700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-31T23:38:13.357804+0000 front 2024-03-31T23:38:13.357906+0000 (oldest deadline 2024-03-31T23:38:38.057730+0000) 2024-03-31T23:38:48.438 INFO:journalctl@ceph.osd.10.smithi142.stdout:Mar 31 23:38:48 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-10[60116]: 2024-03-31T23:38:48.181+0000 7f64c04a8700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.142:6804 osd.1 since back 2024-03-31T23:38:20.958341+0000 front 2024-03-31T23:38:20.958408+0000 (oldest deadline 2024-03-31T23:38:45.058481+0000) 2024-03-31T23:38:48.438 INFO:journalctl@ceph.osd.4.smithi142.stdout:Mar 31 23:38:48 smithi142 podman[77959]: 2024-03-31 23:38:48.189171885 +0000 UTC m=+0.661302954 container remove 2f2a509d7641c725d57e93d687709ef81a5ccf251646de28cdcb84e215d2c88c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-4, io.buildah.version=1.33.5, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, RELEASE=squid-384f31f, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240326, org.label-schema.vendor=CentOS, ceph=True) 2024-03-31T23:38:48.545 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 31 23:38:48 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-8[55211]: 2024-03-31T23:38:48.141+0000 7fee85469700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-31T23:38:12.286566+0000 front 2024-03-31T23:38:12.286547+0000 (oldest deadline 2024-03-31T23:38:36.986433+0000) 2024-03-31T23:38:48.545 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 31 23:38:48 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-8[55211]: 2024-03-31T23:38:48.141+0000 7fee85469700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.142:6804 osd.1 since back 2024-03-31T23:38:21.587779+0000 front 2024-03-31T23:38:21.587835+0000 (oldest deadline 2024-03-31T23:38:47.487919+0000) 2024-03-31T23:38:48.693 INFO:journalctl@ceph.osd.4.smithi142.stdout:Mar 31 23:38:48 smithi142 podman[78050]: 2024-03-31 23:38:48.463819149 +0000 UTC m=+0.021339049 image pull ec18d7baa60e8b1a0b126fbe5306acce11b1ed0ec5e3bcdddbf39b0a3c591791 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175 2024-03-31T23:38:48.693 INFO:journalctl@ceph.osd.4.smithi142.stdout:Mar 31 23:38:48 smithi142 podman[78050]: 2024-03-31 23:38:48.590216892 +0000 UTC m=+0.147736772 container create a663e951172c0fef8e84da4f6737ab2a33fbf872f853e3f4b64126c186cf66a5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-4-deactivate, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-384f31f, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240326, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-03-31T23:38:48.694 INFO:journalctl@ceph.osd.7.smithi142.stdout:Mar 31 23:38:48 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-7[53065]: 2024-03-31T23:38:48.437+0000 7fa41768e700 -1 osd.7 139 heartbeat_check: no reply from 172.21.15.142:6804 osd.1 since back 2024-03-31T23:38:18.318617+0000 front 2024-03-31T23:38:18.318635+0000 (oldest deadline 2024-03-31T23:38:48.319131+0000) 2024-03-31T23:38:49.045 INFO:journalctl@ceph.osd.5.smithi162.stdout:Mar 31 23:38:48 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-5[48185]: 2024-03-31T23:38:48.618+0000 7f433d90e700 -1 osd.5 139 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-31T23:38:11.433029+0000 front 2024-03-31T23:38:11.432884+0000 (oldest deadline 2024-03-31T23:38:36.132991+0000) 2024-03-31T23:38:49.046 INFO:journalctl@ceph.osd.5.smithi162.stdout:Mar 31 23:38:48 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-5[48185]: 2024-03-31T23:38:48.618+0000 7f433d90e700 -1 osd.5 139 heartbeat_check: no reply from 172.21.15.142:6804 osd.1 since back 2024-03-31T23:38:21.735080+0000 front 2024-03-31T23:38:21.735097+0000 (oldest deadline 2024-03-31T23:38:45.235148+0000) 2024-03-31T23:38:49.046 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:38:48 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:38:48.677+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-31T23:38:12.221366+0000 front 2024-03-31T23:38:12.221347+0000 (oldest deadline 2024-03-31T23:38:37.521357+0000) 2024-03-31T23:38:49.046 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:38:48 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:38:48.677+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.142:6804 osd.1 since back 2024-03-31T23:38:22.222192+0000 front 2024-03-31T23:38:22.222213+0000 (oldest deadline 2024-03-31T23:38:45.722340+0000) 2024-03-31T23:38:49.133 INFO:journalctl@ceph.osd.4.smithi142.stdout:Mar 31 23:38:48 smithi142 podman[78050]: 2024-03-31 23:38:48.834671132 +0000 UTC m=+0.392191039 container init a663e951172c0fef8e84da4f6737ab2a33fbf872f853e3f4b64126c186cf66a5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-4-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240326, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, RELEASE=squid-384f31f, io.buildah.version=1.33.5) 2024-03-31T23:38:49.133 INFO:journalctl@ceph.osd.4.smithi142.stdout:Mar 31 23:38:48 smithi142 podman[78050]: 2024-03-31 23:38:48.837862659 +0000 UTC m=+0.395382549 container start a663e951172c0fef8e84da4f6737ab2a33fbf872f853e3f4b64126c186cf66a5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-4-deactivate, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, ceph=True, org.label-schema.schema-version=1.0, RELEASE=squid-384f31f, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.build-date=20240326, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-03-31T23:38:49.133 INFO:journalctl@ceph.osd.4.smithi142.stdout:Mar 31 23:38:48 smithi142 podman[78050]: 2024-03-31 23:38:48.926259591 +0000 UTC m=+0.483779473 container attach a663e951172c0fef8e84da4f6737ab2a33fbf872f853e3f4b64126c186cf66a5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-4-deactivate, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240326, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, io.buildah.version=1.33.5, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, RELEASE=squid-384f31f, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD) 2024-03-31T23:38:49.133 INFO:journalctl@ceph.osd.4.smithi142.stdout:Mar 31 23:38:49 smithi142 podman[78050]: 2024-03-31 23:38:49.13338257 +0000 UTC m=+0.690902460 container died a663e951172c0fef8e84da4f6737ab2a33fbf872f853e3f4b64126c186cf66a5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-4-deactivate, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, RELEASE=squid-384f31f, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.build-date=20240326, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2) 2024-03-31T23:38:49.361 INFO:journalctl@ceph.osd.6.smithi046.stdout:Mar 31 23:38:49 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-6[60760]: 2024-03-31T23:38:49.130+0000 7f8b7cce2700 -1 osd.6 139 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-31T23:38:12.034796+0000 front 2024-03-31T23:38:12.034926+0000 (oldest deadline 2024-03-31T23:38:34.335020+0000) 2024-03-31T23:38:49.361 INFO:journalctl@ceph.osd.6.smithi046.stdout:Mar 31 23:38:49 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-6[60760]: 2024-03-31T23:38:49.130+0000 7f8b7cce2700 -1 osd.6 139 heartbeat_check: no reply from 172.21.15.142:6804 osd.1 since back 2024-03-31T23:38:19.636274+0000 front 2024-03-31T23:38:19.636207+0000 (oldest deadline 2024-03-31T23:38:44.936378+0000) 2024-03-31T23:38:49.362 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 31 23:38:48 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-9[67499]: 2024-03-31T23:38:48.982+0000 7fcc07615700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-31T23:38:10.469941+0000 front 2024-03-31T23:38:10.470388+0000 (oldest deadline 2024-03-31T23:38:35.170101+0000) 2024-03-31T23:38:49.362 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 31 23:38:48 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-9[67499]: 2024-03-31T23:38:48.982+0000 7fcc07615700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.142:6804 osd.1 since back 2024-03-31T23:38:21.571770+0000 front 2024-03-31T23:38:21.571801+0000 (oldest deadline 2024-03-31T23:38:43.871904+0000) 2024-03-31T23:38:49.419 INFO:journalctl@ceph.osd.10.smithi142.stdout:Mar 31 23:38:49 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-10[60116]: 2024-03-31T23:38:49.181+0000 7f64c04a8700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-31T23:38:13.357804+0000 front 2024-03-31T23:38:13.357906+0000 (oldest deadline 2024-03-31T23:38:38.057730+0000) 2024-03-31T23:38:49.419 INFO:journalctl@ceph.osd.10.smithi142.stdout:Mar 31 23:38:49 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-10[60116]: 2024-03-31T23:38:49.181+0000 7f64c04a8700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.142:6804 osd.1 since back 2024-03-31T23:38:20.958341+0000 front 2024-03-31T23:38:20.958408+0000 (oldest deadline 2024-03-31T23:38:45.058481+0000) 2024-03-31T23:38:49.545 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 31 23:38:49 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-8[55211]: 2024-03-31T23:38:49.182+0000 7fee85469700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-31T23:38:12.286566+0000 front 2024-03-31T23:38:12.286547+0000 (oldest deadline 2024-03-31T23:38:36.986433+0000) 2024-03-31T23:38:49.545 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 31 23:38:49 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-8[55211]: 2024-03-31T23:38:49.182+0000 7fee85469700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.142:6804 osd.1 since back 2024-03-31T23:38:21.587779+0000 front 2024-03-31T23:38:21.587835+0000 (oldest deadline 2024-03-31T23:38:47.487919+0000) 2024-03-31T23:38:49.693 INFO:journalctl@ceph.osd.7.smithi142.stdout:Mar 31 23:38:49 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-7[53065]: 2024-03-31T23:38:49.421+0000 7fa41768e700 -1 osd.7 139 heartbeat_check: no reply from 172.21.15.142:6804 osd.1 since back 2024-03-31T23:38:18.318617+0000 front 2024-03-31T23:38:18.318635+0000 (oldest deadline 2024-03-31T23:38:48.319131+0000) 2024-03-31T23:38:49.771 DEBUG:teuthology.orchestra.run.smithi142:> sudo pkill -f 'journalctl -f -n 0 -u ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4@osd.4.service' 2024-03-31T23:38:49.851 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-31T23:38:49.852 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-03-31T23:38:49.852 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-03-31T23:38:49.852 DEBUG:teuthology.orchestra.run.smithi162:> sudo systemctl stop ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4@osd.5 2024-03-31T23:38:49.876 INFO:journalctl@ceph.osd.5.smithi162.stdout:Mar 31 23:38:49 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-5[48185]: 2024-03-31T23:38:49.607+0000 7f433d90e700 -1 osd.5 139 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-31T23:38:11.433029+0000 front 2024-03-31T23:38:11.432884+0000 (oldest deadline 2024-03-31T23:38:36.132991+0000) 2024-03-31T23:38:49.876 INFO:journalctl@ceph.osd.5.smithi162.stdout:Mar 31 23:38:49 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-5[48185]: 2024-03-31T23:38:49.607+0000 7f433d90e700 -1 osd.5 139 heartbeat_check: no reply from 172.21.15.142:6804 osd.1 since back 2024-03-31T23:38:21.735080+0000 front 2024-03-31T23:38:21.735097+0000 (oldest deadline 2024-03-31T23:38:45.235148+0000) 2024-03-31T23:38:49.876 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:38:49 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:38:49.686+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-31T23:38:12.221366+0000 front 2024-03-31T23:38:12.221347+0000 (oldest deadline 2024-03-31T23:38:37.521357+0000) 2024-03-31T23:38:49.876 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:38:49 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:38:49.686+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.142:6804 osd.1 since back 2024-03-31T23:38:22.222192+0000 front 2024-03-31T23:38:22.222213+0000 (oldest deadline 2024-03-31T23:38:45.722340+0000) 2024-03-31T23:38:50.146 INFO:journalctl@ceph.osd.5.smithi162.stdout:Mar 31 23:38:49 smithi162 systemd[1]: Stopping Ceph osd.5 for 6a9d8f08-efb2-11ee-b647-cb9ed24678a4... 2024-03-31T23:38:50.361 INFO:journalctl@ceph.osd.6.smithi046.stdout:Mar 31 23:38:50 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-6[60760]: 2024-03-31T23:38:50.172+0000 7f8b7cce2700 -1 osd.6 139 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-31T23:38:12.034796+0000 front 2024-03-31T23:38:12.034926+0000 (oldest deadline 2024-03-31T23:38:34.335020+0000) 2024-03-31T23:38:50.362 INFO:journalctl@ceph.osd.6.smithi046.stdout:Mar 31 23:38:50 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-6[60760]: 2024-03-31T23:38:50.172+0000 7f8b7cce2700 -1 osd.6 139 heartbeat_check: no reply from 172.21.15.142:6804 osd.1 since back 2024-03-31T23:38:19.636274+0000 front 2024-03-31T23:38:19.636207+0000 (oldest deadline 2024-03-31T23:38:44.936378+0000) 2024-03-31T23:38:50.362 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 31 23:38:50 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-9[67499]: 2024-03-31T23:38:50.005+0000 7fcc07615700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-31T23:38:10.469941+0000 front 2024-03-31T23:38:10.470388+0000 (oldest deadline 2024-03-31T23:38:35.170101+0000) 2024-03-31T23:38:50.362 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 31 23:38:50 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-9[67499]: 2024-03-31T23:38:50.005+0000 7fcc07615700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.142:6804 osd.1 since back 2024-03-31T23:38:21.571770+0000 front 2024-03-31T23:38:21.571801+0000 (oldest deadline 2024-03-31T23:38:43.871904+0000) 2024-03-31T23:38:50.545 INFO:journalctl@ceph.osd.5.smithi162.stdout:Mar 31 23:38:50 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-5[48185]: 2024-03-31T23:38:50.154+0000 7f4345322700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-31T23:38:50.546 INFO:journalctl@ceph.osd.5.smithi162.stdout:Mar 31 23:38:50 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-5[48185]: 2024-03-31T23:38:50.154+0000 7f4345322700 -1 osd.5 139 *** Got signal Terminated *** 2024-03-31T23:38:50.546 INFO:journalctl@ceph.osd.5.smithi162.stdout:Mar 31 23:38:50 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-5[48185]: 2024-03-31T23:38:50.154+0000 7f4345322700 -1 osd.5 139 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-31T23:38:50.546 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 31 23:38:50 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-8[55211]: 2024-03-31T23:38:50.145+0000 7fee85469700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-31T23:38:12.286566+0000 front 2024-03-31T23:38:12.286547+0000 (oldest deadline 2024-03-31T23:38:36.986433+0000) 2024-03-31T23:38:50.546 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 31 23:38:50 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-8[55211]: 2024-03-31T23:38:50.145+0000 7fee85469700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.142:6804 osd.1 since back 2024-03-31T23:38:21.587779+0000 front 2024-03-31T23:38:21.587835+0000 (oldest deadline 2024-03-31T23:38:47.487919+0000) 2024-03-31T23:38:50.693 INFO:journalctl@ceph.osd.7.smithi142.stdout:Mar 31 23:38:50 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-7[53065]: 2024-03-31T23:38:50.446+0000 7fa41768e700 -1 osd.7 139 heartbeat_check: no reply from 172.21.15.142:6804 osd.1 since back 2024-03-31T23:38:18.318617+0000 front 2024-03-31T23:38:18.318635+0000 (oldest deadline 2024-03-31T23:38:48.319131+0000) 2024-03-31T23:38:50.693 INFO:journalctl@ceph.osd.10.smithi142.stdout:Mar 31 23:38:50 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-10[60116]: 2024-03-31T23:38:50.215+0000 7f64c04a8700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-31T23:38:13.357804+0000 front 2024-03-31T23:38:13.357906+0000 (oldest deadline 2024-03-31T23:38:38.057730+0000) 2024-03-31T23:38:50.693 INFO:journalctl@ceph.osd.10.smithi142.stdout:Mar 31 23:38:50 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-10[60116]: 2024-03-31T23:38:50.215+0000 7f64c04a8700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.142:6804 osd.1 since back 2024-03-31T23:38:20.958341+0000 front 2024-03-31T23:38:20.958408+0000 (oldest deadline 2024-03-31T23:38:45.058481+0000) 2024-03-31T23:38:51.045 INFO:journalctl@ceph.osd.5.smithi162.stdout:Mar 31 23:38:50 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-5[48185]: 2024-03-31T23:38:50.581+0000 7f433d90e700 -1 osd.5 139 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-31T23:38:11.433029+0000 front 2024-03-31T23:38:11.432884+0000 (oldest deadline 2024-03-31T23:38:36.132991+0000) 2024-03-31T23:38:51.045 INFO:journalctl@ceph.osd.5.smithi162.stdout:Mar 31 23:38:50 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-5[48185]: 2024-03-31T23:38:50.581+0000 7f433d90e700 -1 osd.5 139 heartbeat_check: no reply from 172.21.15.142:6804 osd.1 since back 2024-03-31T23:38:21.735080+0000 front 2024-03-31T23:38:21.735097+0000 (oldest deadline 2024-03-31T23:38:45.235148+0000) 2024-03-31T23:38:51.046 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:38:50 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:38:50.650+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-31T23:38:12.221366+0000 front 2024-03-31T23:38:12.221347+0000 (oldest deadline 2024-03-31T23:38:37.521357+0000) 2024-03-31T23:38:51.046 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:38:50 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:38:50.650+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.142:6804 osd.1 since back 2024-03-31T23:38:22.222192+0000 front 2024-03-31T23:38:22.222213+0000 (oldest deadline 2024-03-31T23:38:45.722340+0000) 2024-03-31T23:38:51.361 INFO:journalctl@ceph.osd.6.smithi046.stdout:Mar 31 23:38:51 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-6[60760]: 2024-03-31T23:38:51.205+0000 7f8b7cce2700 -1 osd.6 139 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-31T23:38:12.034796+0000 front 2024-03-31T23:38:12.034926+0000 (oldest deadline 2024-03-31T23:38:34.335020+0000) 2024-03-31T23:38:51.361 INFO:journalctl@ceph.osd.6.smithi046.stdout:Mar 31 23:38:51 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-6[60760]: 2024-03-31T23:38:51.205+0000 7f8b7cce2700 -1 osd.6 139 heartbeat_check: no reply from 172.21.15.142:6804 osd.1 since back 2024-03-31T23:38:19.636274+0000 front 2024-03-31T23:38:19.636207+0000 (oldest deadline 2024-03-31T23:38:44.936378+0000) 2024-03-31T23:38:51.362 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 31 23:38:51 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-9[67499]: 2024-03-31T23:38:51.024+0000 7fcc07615700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-31T23:38:10.469941+0000 front 2024-03-31T23:38:10.470388+0000 (oldest deadline 2024-03-31T23:38:35.170101+0000) 2024-03-31T23:38:51.362 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 31 23:38:51 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-9[67499]: 2024-03-31T23:38:51.024+0000 7fcc07615700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.142:6804 osd.1 since back 2024-03-31T23:38:21.571770+0000 front 2024-03-31T23:38:21.571801+0000 (oldest deadline 2024-03-31T23:38:43.871904+0000) 2024-03-31T23:38:51.545 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 31 23:38:51 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-8[55211]: 2024-03-31T23:38:51.153+0000 7fee85469700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-31T23:38:12.286566+0000 front 2024-03-31T23:38:12.286547+0000 (oldest deadline 2024-03-31T23:38:36.986433+0000) 2024-03-31T23:38:51.545 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 31 23:38:51 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-8[55211]: 2024-03-31T23:38:51.153+0000 7fee85469700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.142:6804 osd.1 since back 2024-03-31T23:38:21.587779+0000 front 2024-03-31T23:38:21.587835+0000 (oldest deadline 2024-03-31T23:38:47.487919+0000) 2024-03-31T23:38:51.693 INFO:journalctl@ceph.osd.7.smithi142.stdout:Mar 31 23:38:51 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-7[53065]: 2024-03-31T23:38:51.461+0000 7fa41768e700 -1 osd.7 139 heartbeat_check: no reply from 172.21.15.142:6804 osd.1 since back 2024-03-31T23:38:18.318617+0000 front 2024-03-31T23:38:18.318635+0000 (oldest deadline 2024-03-31T23:38:48.319131+0000) 2024-03-31T23:38:51.693 INFO:journalctl@ceph.osd.10.smithi142.stdout:Mar 31 23:38:51 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-10[60116]: 2024-03-31T23:38:51.233+0000 7f64c04a8700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-31T23:38:13.357804+0000 front 2024-03-31T23:38:13.357906+0000 (oldest deadline 2024-03-31T23:38:38.057730+0000) 2024-03-31T23:38:51.693 INFO:journalctl@ceph.osd.10.smithi142.stdout:Mar 31 23:38:51 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-10[60116]: 2024-03-31T23:38:51.233+0000 7f64c04a8700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.142:6804 osd.1 since back 2024-03-31T23:38:20.958341+0000 front 2024-03-31T23:38:20.958408+0000 (oldest deadline 2024-03-31T23:38:45.058481+0000) 2024-03-31T23:38:52.045 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:38:51 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:38:51.671+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-31T23:38:12.221366+0000 front 2024-03-31T23:38:12.221347+0000 (oldest deadline 2024-03-31T23:38:37.521357+0000) 2024-03-31T23:38:52.045 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:38:51 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:38:51.671+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.142:6804 osd.1 since back 2024-03-31T23:38:22.222192+0000 front 2024-03-31T23:38:22.222213+0000 (oldest deadline 2024-03-31T23:38:45.722340+0000) 2024-03-31T23:38:52.046 INFO:journalctl@ceph.osd.5.smithi162.stdout:Mar 31 23:38:51 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-5[48185]: 2024-03-31T23:38:51.601+0000 7f433d90e700 -1 osd.5 139 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-31T23:38:11.433029+0000 front 2024-03-31T23:38:11.432884+0000 (oldest deadline 2024-03-31T23:38:36.132991+0000) 2024-03-31T23:38:52.046 INFO:journalctl@ceph.osd.5.smithi162.stdout:Mar 31 23:38:51 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-5[48185]: 2024-03-31T23:38:51.601+0000 7f433d90e700 -1 osd.5 139 heartbeat_check: no reply from 172.21.15.142:6804 osd.1 since back 2024-03-31T23:38:21.735080+0000 front 2024-03-31T23:38:21.735097+0000 (oldest deadline 2024-03-31T23:38:45.235148+0000) 2024-03-31T23:38:52.361 INFO:journalctl@ceph.osd.6.smithi046.stdout:Mar 31 23:38:52 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-6[60760]: 2024-03-31T23:38:52.184+0000 7f8b7cce2700 -1 osd.6 139 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-31T23:38:12.034796+0000 front 2024-03-31T23:38:12.034926+0000 (oldest deadline 2024-03-31T23:38:34.335020+0000) 2024-03-31T23:38:52.362 INFO:journalctl@ceph.osd.6.smithi046.stdout:Mar 31 23:38:52 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-6[60760]: 2024-03-31T23:38:52.184+0000 7f8b7cce2700 -1 osd.6 139 heartbeat_check: no reply from 172.21.15.142:6804 osd.1 since back 2024-03-31T23:38:19.636274+0000 front 2024-03-31T23:38:19.636207+0000 (oldest deadline 2024-03-31T23:38:44.936378+0000) 2024-03-31T23:38:52.362 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 31 23:38:52 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-9[67499]: 2024-03-31T23:38:52.064+0000 7fcc07615700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-31T23:38:10.469941+0000 front 2024-03-31T23:38:10.470388+0000 (oldest deadline 2024-03-31T23:38:35.170101+0000) 2024-03-31T23:38:52.362 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 31 23:38:52 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-9[67499]: 2024-03-31T23:38:52.064+0000 7fcc07615700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.142:6804 osd.1 since back 2024-03-31T23:38:21.571770+0000 front 2024-03-31T23:38:21.571801+0000 (oldest deadline 2024-03-31T23:38:43.871904+0000) 2024-03-31T23:38:52.545 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 31 23:38:52 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-8[55211]: 2024-03-31T23:38:52.107+0000 7fee85469700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-31T23:38:12.286566+0000 front 2024-03-31T23:38:12.286547+0000 (oldest deadline 2024-03-31T23:38:36.986433+0000) 2024-03-31T23:38:52.546 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 31 23:38:52 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-8[55211]: 2024-03-31T23:38:52.107+0000 7fee85469700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.142:6804 osd.1 since back 2024-03-31T23:38:21.587779+0000 front 2024-03-31T23:38:21.587835+0000 (oldest deadline 2024-03-31T23:38:47.487919+0000) 2024-03-31T23:38:52.693 INFO:journalctl@ceph.osd.7.smithi142.stdout:Mar 31 23:38:52 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-7[53065]: 2024-03-31T23:38:52.422+0000 7fa41768e700 -1 osd.7 139 heartbeat_check: no reply from 172.21.15.142:6804 osd.1 since back 2024-03-31T23:38:18.318617+0000 front 2024-03-31T23:38:18.318635+0000 (oldest deadline 2024-03-31T23:38:48.319131+0000) 2024-03-31T23:38:52.693 INFO:journalctl@ceph.osd.10.smithi142.stdout:Mar 31 23:38:52 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-10[60116]: 2024-03-31T23:38:52.264+0000 7f64c04a8700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-31T23:38:13.357804+0000 front 2024-03-31T23:38:13.357906+0000 (oldest deadline 2024-03-31T23:38:38.057730+0000) 2024-03-31T23:38:52.693 INFO:journalctl@ceph.osd.10.smithi142.stdout:Mar 31 23:38:52 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-10[60116]: 2024-03-31T23:38:52.264+0000 7f64c04a8700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.142:6804 osd.1 since back 2024-03-31T23:38:20.958341+0000 front 2024-03-31T23:38:20.958408+0000 (oldest deadline 2024-03-31T23:38:45.058481+0000) 2024-03-31T23:38:52.693 INFO:journalctl@ceph.osd.10.smithi142.stdout:Mar 31 23:38:52 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-10[60116]: 2024-03-31T23:38:52.264+0000 7f64c04a8700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-31T23:38:25.559393+0000 front 2024-03-31T23:38:25.559313+0000 (oldest deadline 2024-03-31T23:38:51.459398+0000) 2024-03-31T23:38:53.045 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:38:52 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:38:52.686+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-31T23:38:12.221366+0000 front 2024-03-31T23:38:12.221347+0000 (oldest deadline 2024-03-31T23:38:37.521357+0000) 2024-03-31T23:38:53.046 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:38:52 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:38:52.686+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.142:6804 osd.1 since back 2024-03-31T23:38:22.222192+0000 front 2024-03-31T23:38:22.222213+0000 (oldest deadline 2024-03-31T23:38:45.722340+0000) 2024-03-31T23:38:53.046 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:38:52 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:38:52.686+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-31T23:38:29.723616+0000 front 2024-03-31T23:38:29.723593+0000 (oldest deadline 2024-03-31T23:38:52.023779+0000) 2024-03-31T23:38:53.046 INFO:journalctl@ceph.osd.5.smithi162.stdout:Mar 31 23:38:52 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-5[48185]: 2024-03-31T23:38:52.600+0000 7f433d90e700 -1 osd.5 139 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-31T23:38:11.433029+0000 front 2024-03-31T23:38:11.432884+0000 (oldest deadline 2024-03-31T23:38:36.132991+0000) 2024-03-31T23:38:53.046 INFO:journalctl@ceph.osd.5.smithi162.stdout:Mar 31 23:38:52 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-5[48185]: 2024-03-31T23:38:52.600+0000 7f433d90e700 -1 osd.5 139 heartbeat_check: no reply from 172.21.15.142:6804 osd.1 since back 2024-03-31T23:38:21.735080+0000 front 2024-03-31T23:38:21.735097+0000 (oldest deadline 2024-03-31T23:38:45.235148+0000) 2024-03-31T23:38:53.046 INFO:journalctl@ceph.osd.5.smithi162.stdout:Mar 31 23:38:52 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-5[48185]: 2024-03-31T23:38:52.600+0000 7f433d90e700 -1 osd.5 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-31T23:38:26.935874+0000 front 2024-03-31T23:38:26.935891+0000 (oldest deadline 2024-03-31T23:38:51.635998+0000) 2024-03-31T23:38:53.361 INFO:journalctl@ceph.osd.6.smithi046.stdout:Mar 31 23:38:53 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-6[60760]: 2024-03-31T23:38:53.195+0000 7f8b7cce2700 -1 osd.6 139 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-31T23:38:12.034796+0000 front 2024-03-31T23:38:12.034926+0000 (oldest deadline 2024-03-31T23:38:34.335020+0000) 2024-03-31T23:38:53.361 INFO:journalctl@ceph.osd.6.smithi046.stdout:Mar 31 23:38:53 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-6[60760]: 2024-03-31T23:38:53.195+0000 7f8b7cce2700 -1 osd.6 139 heartbeat_check: no reply from 172.21.15.142:6804 osd.1 since back 2024-03-31T23:38:19.636274+0000 front 2024-03-31T23:38:19.636207+0000 (oldest deadline 2024-03-31T23:38:44.936378+0000) 2024-03-31T23:38:53.362 INFO:journalctl@ceph.osd.6.smithi046.stdout:Mar 31 23:38:53 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-6[60760]: 2024-03-31T23:38:53.195+0000 7f8b7cce2700 -1 osd.6 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-31T23:38:26.637553+0000 front 2024-03-31T23:38:26.637578+0000 (oldest deadline 2024-03-31T23:38:52.537783+0000) 2024-03-31T23:38:53.362 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 31 23:38:53 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-9[67499]: 2024-03-31T23:38:53.039+0000 7fcc07615700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-31T23:38:10.469941+0000 front 2024-03-31T23:38:10.470388+0000 (oldest deadline 2024-03-31T23:38:35.170101+0000) 2024-03-31T23:38:53.362 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 31 23:38:53 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-9[67499]: 2024-03-31T23:38:53.039+0000 7fcc07615700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.142:6804 osd.1 since back 2024-03-31T23:38:21.571770+0000 front 2024-03-31T23:38:21.571801+0000 (oldest deadline 2024-03-31T23:38:43.871904+0000) 2024-03-31T23:38:53.362 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 31 23:38:53 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-9[67499]: 2024-03-31T23:38:53.039+0000 7fcc07615700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-31T23:38:27.074768+0000 front 2024-03-31T23:38:27.074793+0000 (oldest deadline 2024-03-31T23:38:52.374931+0000) 2024-03-31T23:38:53.545 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 31 23:38:53 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-8[55211]: 2024-03-31T23:38:53.141+0000 7fee85469700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-31T23:38:12.286566+0000 front 2024-03-31T23:38:12.286547+0000 (oldest deadline 2024-03-31T23:38:36.986433+0000) 2024-03-31T23:38:53.545 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 31 23:38:53 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-8[55211]: 2024-03-31T23:38:53.141+0000 7fee85469700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.142:6804 osd.1 since back 2024-03-31T23:38:21.587779+0000 front 2024-03-31T23:38:21.587835+0000 (oldest deadline 2024-03-31T23:38:47.487919+0000) 2024-03-31T23:38:53.594 INFO:journalctl@ceph.osd.7.smithi142.stdout:Mar 31 23:38:53 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-7[53065]: 2024-03-31T23:38:53.394+0000 7fa41768e700 -1 osd.7 139 heartbeat_check: no reply from 172.21.15.142:6804 osd.1 since back 2024-03-31T23:38:18.318617+0000 front 2024-03-31T23:38:18.318635+0000 (oldest deadline 2024-03-31T23:38:48.319131+0000) 2024-03-31T23:38:53.595 INFO:journalctl@ceph.osd.10.smithi142.stdout:Mar 31 23:38:53 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-10[60116]: 2024-03-31T23:38:53.268+0000 7f64c04a8700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-31T23:38:13.357804+0000 front 2024-03-31T23:38:13.357906+0000 (oldest deadline 2024-03-31T23:38:38.057730+0000) 2024-03-31T23:38:53.595 INFO:journalctl@ceph.osd.10.smithi142.stdout:Mar 31 23:38:53 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-10[60116]: 2024-03-31T23:38:53.268+0000 7f64c04a8700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.142:6804 osd.1 since back 2024-03-31T23:38:20.958341+0000 front 2024-03-31T23:38:20.958408+0000 (oldest deadline 2024-03-31T23:38:45.058481+0000) 2024-03-31T23:38:53.595 INFO:journalctl@ceph.osd.10.smithi142.stdout:Mar 31 23:38:53 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-10[60116]: 2024-03-31T23:38:53.268+0000 7f64c04a8700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-31T23:38:25.559393+0000 front 2024-03-31T23:38:25.559313+0000 (oldest deadline 2024-03-31T23:38:51.459398+0000) 2024-03-31T23:38:54.045 INFO:journalctl@ceph.osd.5.smithi162.stdout:Mar 31 23:38:53 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-5[48185]: 2024-03-31T23:38:53.631+0000 7f433d90e700 -1 osd.5 139 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-31T23:38:11.433029+0000 front 2024-03-31T23:38:11.432884+0000 (oldest deadline 2024-03-31T23:38:36.132991+0000) 2024-03-31T23:38:54.045 INFO:journalctl@ceph.osd.5.smithi162.stdout:Mar 31 23:38:53 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-5[48185]: 2024-03-31T23:38:53.631+0000 7f433d90e700 -1 osd.5 139 heartbeat_check: no reply from 172.21.15.142:6804 osd.1 since back 2024-03-31T23:38:21.735080+0000 front 2024-03-31T23:38:21.735097+0000 (oldest deadline 2024-03-31T23:38:45.235148+0000) 2024-03-31T23:38:54.046 INFO:journalctl@ceph.osd.5.smithi162.stdout:Mar 31 23:38:53 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-5[48185]: 2024-03-31T23:38:53.631+0000 7f433d90e700 -1 osd.5 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-31T23:38:26.935874+0000 front 2024-03-31T23:38:26.935891+0000 (oldest deadline 2024-03-31T23:38:51.635998+0000) 2024-03-31T23:38:54.046 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:38:53 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:38:53.657+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-31T23:38:12.221366+0000 front 2024-03-31T23:38:12.221347+0000 (oldest deadline 2024-03-31T23:38:37.521357+0000) 2024-03-31T23:38:54.046 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:38:53 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:38:53.657+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.142:6804 osd.1 since back 2024-03-31T23:38:22.222192+0000 front 2024-03-31T23:38:22.222213+0000 (oldest deadline 2024-03-31T23:38:45.722340+0000) 2024-03-31T23:38:54.046 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:38:53 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:38:53.657+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-31T23:38:29.723616+0000 front 2024-03-31T23:38:29.723593+0000 (oldest deadline 2024-03-31T23:38:52.023779+0000) 2024-03-31T23:38:54.361 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 31 23:38:54 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-9[67499]: 2024-03-31T23:38:54.015+0000 7fcc07615700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-31T23:38:10.469941+0000 front 2024-03-31T23:38:10.470388+0000 (oldest deadline 2024-03-31T23:38:35.170101+0000) 2024-03-31T23:38:54.362 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 31 23:38:54 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-9[67499]: 2024-03-31T23:38:54.015+0000 7fcc07615700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.142:6804 osd.1 since back 2024-03-31T23:38:21.571770+0000 front 2024-03-31T23:38:21.571801+0000 (oldest deadline 2024-03-31T23:38:43.871904+0000) 2024-03-31T23:38:54.362 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 31 23:38:54 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-9[67499]: 2024-03-31T23:38:54.015+0000 7fcc07615700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-31T23:38:27.074768+0000 front 2024-03-31T23:38:27.074793+0000 (oldest deadline 2024-03-31T23:38:52.374931+0000) 2024-03-31T23:38:54.362 INFO:journalctl@ceph.osd.6.smithi046.stdout:Mar 31 23:38:54 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-6[60760]: 2024-03-31T23:38:54.189+0000 7f8b7cce2700 -1 osd.6 139 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-31T23:38:12.034796+0000 front 2024-03-31T23:38:12.034926+0000 (oldest deadline 2024-03-31T23:38:34.335020+0000) 2024-03-31T23:38:54.362 INFO:journalctl@ceph.osd.6.smithi046.stdout:Mar 31 23:38:54 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-6[60760]: 2024-03-31T23:38:54.189+0000 7f8b7cce2700 -1 osd.6 139 heartbeat_check: no reply from 172.21.15.142:6804 osd.1 since back 2024-03-31T23:38:19.636274+0000 front 2024-03-31T23:38:19.636207+0000 (oldest deadline 2024-03-31T23:38:44.936378+0000) 2024-03-31T23:38:54.362 INFO:journalctl@ceph.osd.6.smithi046.stdout:Mar 31 23:38:54 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-6[60760]: 2024-03-31T23:38:54.189+0000 7f8b7cce2700 -1 osd.6 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-31T23:38:26.637553+0000 front 2024-03-31T23:38:26.637578+0000 (oldest deadline 2024-03-31T23:38:52.537783+0000) 2024-03-31T23:38:54.545 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 31 23:38:54 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-8[55211]: 2024-03-31T23:38:54.173+0000 7fee85469700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-31T23:38:12.286566+0000 front 2024-03-31T23:38:12.286547+0000 (oldest deadline 2024-03-31T23:38:36.986433+0000) 2024-03-31T23:38:54.545 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 31 23:38:54 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-8[55211]: 2024-03-31T23:38:54.173+0000 7fee85469700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.142:6804 osd.1 since back 2024-03-31T23:38:21.587779+0000 front 2024-03-31T23:38:21.587835+0000 (oldest deadline 2024-03-31T23:38:47.487919+0000) 2024-03-31T23:38:54.546 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 31 23:38:54 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-8[55211]: 2024-03-31T23:38:54.173+0000 7fee85469700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-31T23:38:27.488310+0000 front 2024-03-31T23:38:27.488293+0000 (oldest deadline 2024-03-31T23:38:53.388420+0000) 2024-03-31T23:38:54.693 INFO:journalctl@ceph.osd.10.smithi142.stdout:Mar 31 23:38:54 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-10[60116]: 2024-03-31T23:38:54.239+0000 7f64c04a8700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-31T23:38:13.357804+0000 front 2024-03-31T23:38:13.357906+0000 (oldest deadline 2024-03-31T23:38:38.057730+0000) 2024-03-31T23:38:54.693 INFO:journalctl@ceph.osd.10.smithi142.stdout:Mar 31 23:38:54 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-10[60116]: 2024-03-31T23:38:54.239+0000 7f64c04a8700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.142:6804 osd.1 since back 2024-03-31T23:38:20.958341+0000 front 2024-03-31T23:38:20.958408+0000 (oldest deadline 2024-03-31T23:38:45.058481+0000) 2024-03-31T23:38:54.693 INFO:journalctl@ceph.osd.10.smithi142.stdout:Mar 31 23:38:54 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-10[60116]: 2024-03-31T23:38:54.239+0000 7f64c04a8700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-31T23:38:25.559393+0000 front 2024-03-31T23:38:25.559313+0000 (oldest deadline 2024-03-31T23:38:51.459398+0000) 2024-03-31T23:38:54.693 INFO:journalctl@ceph.osd.7.smithi142.stdout:Mar 31 23:38:54 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-7[53065]: 2024-03-31T23:38:54.401+0000 7fa41768e700 -1 osd.7 139 heartbeat_check: no reply from 172.21.15.142:6804 osd.1 since back 2024-03-31T23:38:18.318617+0000 front 2024-03-31T23:38:18.318635+0000 (oldest deadline 2024-03-31T23:38:48.319131+0000) 2024-03-31T23:38:54.693 INFO:journalctl@ceph.osd.7.smithi142.stdout:Mar 31 23:38:54 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-7[53065]: 2024-03-31T23:38:54.401+0000 7fa41768e700 -1 osd.7 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-31T23:38:28.319471+0000 front 2024-03-31T23:38:28.319515+0000 (oldest deadline 2024-03-31T23:38:54.219616+0000) 2024-03-31T23:38:55.045 INFO:journalctl@ceph.osd.5.smithi162.stdout:Mar 31 23:38:54 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-5[48185]: 2024-03-31T23:38:54.657+0000 7f433d90e700 -1 osd.5 139 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-31T23:38:11.433029+0000 front 2024-03-31T23:38:11.432884+0000 (oldest deadline 2024-03-31T23:38:36.132991+0000) 2024-03-31T23:38:55.046 INFO:journalctl@ceph.osd.5.smithi162.stdout:Mar 31 23:38:54 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-5[48185]: 2024-03-31T23:38:54.657+0000 7f433d90e700 -1 osd.5 139 heartbeat_check: no reply from 172.21.15.142:6804 osd.1 since back 2024-03-31T23:38:21.735080+0000 front 2024-03-31T23:38:21.735097+0000 (oldest deadline 2024-03-31T23:38:45.235148+0000) 2024-03-31T23:38:55.046 INFO:journalctl@ceph.osd.5.smithi162.stdout:Mar 31 23:38:54 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-5[48185]: 2024-03-31T23:38:54.657+0000 7f433d90e700 -1 osd.5 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-31T23:38:26.935874+0000 front 2024-03-31T23:38:26.935891+0000 (oldest deadline 2024-03-31T23:38:51.635998+0000) 2024-03-31T23:38:55.046 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:38:54 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:38:54.680+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-31T23:38:12.221366+0000 front 2024-03-31T23:38:12.221347+0000 (oldest deadline 2024-03-31T23:38:37.521357+0000) 2024-03-31T23:38:55.046 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:38:54 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:38:54.680+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.142:6804 osd.1 since back 2024-03-31T23:38:22.222192+0000 front 2024-03-31T23:38:22.222213+0000 (oldest deadline 2024-03-31T23:38:45.722340+0000) 2024-03-31T23:38:55.046 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:38:54 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:38:54.680+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-31T23:38:29.723616+0000 front 2024-03-31T23:38:29.723593+0000 (oldest deadline 2024-03-31T23:38:52.023779+0000) 2024-03-31T23:38:55.323 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 31 23:38:55 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-8[55211]: 2024-03-31T23:38:55.168+0000 7fee85469700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-31T23:38:12.286566+0000 front 2024-03-31T23:38:12.286547+0000 (oldest deadline 2024-03-31T23:38:36.986433+0000) 2024-03-31T23:38:55.323 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 31 23:38:55 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-8[55211]: 2024-03-31T23:38:55.169+0000 7fee85469700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.142:6804 osd.1 since back 2024-03-31T23:38:21.587779+0000 front 2024-03-31T23:38:21.587835+0000 (oldest deadline 2024-03-31T23:38:47.487919+0000) 2024-03-31T23:38:55.323 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 31 23:38:55 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-8[55211]: 2024-03-31T23:38:55.169+0000 7fee85469700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-31T23:38:27.488310+0000 front 2024-03-31T23:38:27.488293+0000 (oldest deadline 2024-03-31T23:38:53.388420+0000) 2024-03-31T23:38:55.361 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 31 23:38:55 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-9[67499]: 2024-03-31T23:38:55.026+0000 7fcc07615700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-31T23:38:10.469941+0000 front 2024-03-31T23:38:10.470388+0000 (oldest deadline 2024-03-31T23:38:35.170101+0000) 2024-03-31T23:38:55.361 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 31 23:38:55 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-9[67499]: 2024-03-31T23:38:55.026+0000 7fcc07615700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.142:6804 osd.1 since back 2024-03-31T23:38:21.571770+0000 front 2024-03-31T23:38:21.571801+0000 (oldest deadline 2024-03-31T23:38:43.871904+0000) 2024-03-31T23:38:55.361 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 31 23:38:55 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-9[67499]: 2024-03-31T23:38:55.026+0000 7fcc07615700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-31T23:38:27.074768+0000 front 2024-03-31T23:38:27.074793+0000 (oldest deadline 2024-03-31T23:38:52.374931+0000) 2024-03-31T23:38:55.362 INFO:journalctl@ceph.osd.6.smithi046.stdout:Mar 31 23:38:55 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-6[60760]: 2024-03-31T23:38:55.202+0000 7f8b7cce2700 -1 osd.6 139 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-31T23:38:12.034796+0000 front 2024-03-31T23:38:12.034926+0000 (oldest deadline 2024-03-31T23:38:34.335020+0000) 2024-03-31T23:38:55.362 INFO:journalctl@ceph.osd.6.smithi046.stdout:Mar 31 23:38:55 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-6[60760]: 2024-03-31T23:38:55.202+0000 7f8b7cce2700 -1 osd.6 139 heartbeat_check: no reply from 172.21.15.142:6804 osd.1 since back 2024-03-31T23:38:19.636274+0000 front 2024-03-31T23:38:19.636207+0000 (oldest deadline 2024-03-31T23:38:44.936378+0000) 2024-03-31T23:38:55.362 INFO:journalctl@ceph.osd.6.smithi046.stdout:Mar 31 23:38:55 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-6[60760]: 2024-03-31T23:38:55.202+0000 7f8b7cce2700 -1 osd.6 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-31T23:38:26.637553+0000 front 2024-03-31T23:38:26.637578+0000 (oldest deadline 2024-03-31T23:38:52.537783+0000) 2024-03-31T23:38:55.693 INFO:journalctl@ceph.osd.7.smithi142.stdout:Mar 31 23:38:55 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-7[53065]: 2024-03-31T23:38:55.413+0000 7fa41768e700 -1 osd.7 139 heartbeat_check: no reply from 172.21.15.142:6804 osd.1 since back 2024-03-31T23:38:18.318617+0000 front 2024-03-31T23:38:18.318635+0000 (oldest deadline 2024-03-31T23:38:48.319131+0000) 2024-03-31T23:38:55.693 INFO:journalctl@ceph.osd.7.smithi142.stdout:Mar 31 23:38:55 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-7[53065]: 2024-03-31T23:38:55.413+0000 7fa41768e700 -1 osd.7 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-31T23:38:28.319471+0000 front 2024-03-31T23:38:28.319515+0000 (oldest deadline 2024-03-31T23:38:54.219616+0000) 2024-03-31T23:38:55.693 INFO:journalctl@ceph.osd.10.smithi142.stdout:Mar 31 23:38:55 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-10[60116]: 2024-03-31T23:38:55.200+0000 7f64c04a8700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-31T23:38:13.357804+0000 front 2024-03-31T23:38:13.357906+0000 (oldest deadline 2024-03-31T23:38:38.057730+0000) 2024-03-31T23:38:55.693 INFO:journalctl@ceph.osd.10.smithi142.stdout:Mar 31 23:38:55 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-10[60116]: 2024-03-31T23:38:55.200+0000 7f64c04a8700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.142:6804 osd.1 since back 2024-03-31T23:38:20.958341+0000 front 2024-03-31T23:38:20.958408+0000 (oldest deadline 2024-03-31T23:38:45.058481+0000) 2024-03-31T23:38:55.693 INFO:journalctl@ceph.osd.10.smithi142.stdout:Mar 31 23:38:55 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-10[60116]: 2024-03-31T23:38:55.200+0000 7f64c04a8700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-31T23:38:25.559393+0000 front 2024-03-31T23:38:25.559313+0000 (oldest deadline 2024-03-31T23:38:51.459398+0000) 2024-03-31T23:38:56.045 INFO:journalctl@ceph.osd.5.smithi162.stdout:Mar 31 23:38:55 smithi162 podman[78322]: 2024-03-31 23:38:55.751032351 +0000 UTC m=+5.689199520 container died 2d982a6028966b4ae35b5ad315848e01782f3d8cc65b99bae45ce0b87924c64c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-5, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_CLEAN=True, RELEASE=squid-384f31f, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240326, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-31T23:38:56.046 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:38:55 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:38:55.696+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-31T23:38:12.221366+0000 front 2024-03-31T23:38:12.221347+0000 (oldest deadline 2024-03-31T23:38:37.521357+0000) 2024-03-31T23:38:56.046 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:38:55 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:38:55.696+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.142:6804 osd.1 since back 2024-03-31T23:38:22.222192+0000 front 2024-03-31T23:38:22.222213+0000 (oldest deadline 2024-03-31T23:38:45.722340+0000) 2024-03-31T23:38:56.046 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:38:55 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:38:55.696+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-31T23:38:29.723616+0000 front 2024-03-31T23:38:29.723593+0000 (oldest deadline 2024-03-31T23:38:52.023779+0000) 2024-03-31T23:38:56.361 INFO:journalctl@ceph.osd.6.smithi046.stdout:Mar 31 23:38:56 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-6[60760]: 2024-03-31T23:38:56.153+0000 7f8b7cce2700 -1 osd.6 139 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-31T23:38:12.034796+0000 front 2024-03-31T23:38:12.034926+0000 (oldest deadline 2024-03-31T23:38:34.335020+0000) 2024-03-31T23:38:56.362 INFO:journalctl@ceph.osd.6.smithi046.stdout:Mar 31 23:38:56 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-6[60760]: 2024-03-31T23:38:56.153+0000 7f8b7cce2700 -1 osd.6 139 heartbeat_check: no reply from 172.21.15.142:6804 osd.1 since back 2024-03-31T23:38:19.636274+0000 front 2024-03-31T23:38:19.636207+0000 (oldest deadline 2024-03-31T23:38:44.936378+0000) 2024-03-31T23:38:56.362 INFO:journalctl@ceph.osd.6.smithi046.stdout:Mar 31 23:38:56 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-6[60760]: 2024-03-31T23:38:56.153+0000 7f8b7cce2700 -1 osd.6 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-31T23:38:26.637553+0000 front 2024-03-31T23:38:26.637578+0000 (oldest deadline 2024-03-31T23:38:52.537783+0000) 2024-03-31T23:38:56.362 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 31 23:38:56 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-9[67499]: 2024-03-31T23:38:56.074+0000 7fcc07615700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-31T23:38:10.469941+0000 front 2024-03-31T23:38:10.470388+0000 (oldest deadline 2024-03-31T23:38:35.170101+0000) 2024-03-31T23:38:56.362 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 31 23:38:56 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-9[67499]: 2024-03-31T23:38:56.074+0000 7fcc07615700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.142:6804 osd.1 since back 2024-03-31T23:38:21.571770+0000 front 2024-03-31T23:38:21.571801+0000 (oldest deadline 2024-03-31T23:38:43.871904+0000) 2024-03-31T23:38:56.362 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 31 23:38:56 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-9[67499]: 2024-03-31T23:38:56.074+0000 7fcc07615700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-31T23:38:27.074768+0000 front 2024-03-31T23:38:27.074793+0000 (oldest deadline 2024-03-31T23:38:52.374931+0000) 2024-03-31T23:38:56.375 INFO:journalctl@ceph.osd.5.smithi162.stdout:Mar 31 23:38:56 smithi162 podman[78322]: 2024-03-31 23:38:56.205930054 +0000 UTC m=+6.144097220 container cleanup 2d982a6028966b4ae35b5ad315848e01782f3d8cc65b99bae45ce0b87924c64c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-5, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, RELEASE=squid-384f31f, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240326, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0) 2024-03-31T23:38:56.375 INFO:journalctl@ceph.osd.5.smithi162.stdout:Mar 31 23:38:56 smithi162 bash[78322]: ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-5 2024-03-31T23:38:56.375 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 31 23:38:56 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-8[55211]: 2024-03-31T23:38:56.127+0000 7fee85469700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-31T23:38:12.286566+0000 front 2024-03-31T23:38:12.286547+0000 (oldest deadline 2024-03-31T23:38:36.986433+0000) 2024-03-31T23:38:56.375 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 31 23:38:56 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-8[55211]: 2024-03-31T23:38:56.127+0000 7fee85469700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.142:6804 osd.1 since back 2024-03-31T23:38:21.587779+0000 front 2024-03-31T23:38:21.587835+0000 (oldest deadline 2024-03-31T23:38:47.487919+0000) 2024-03-31T23:38:56.376 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 31 23:38:56 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-8[55211]: 2024-03-31T23:38:56.127+0000 7fee85469700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-31T23:38:27.488310+0000 front 2024-03-31T23:38:27.488293+0000 (oldest deadline 2024-03-31T23:38:53.388420+0000) 2024-03-31T23:38:56.443 INFO:journalctl@ceph.osd.10.smithi142.stdout:Mar 31 23:38:56 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-10[60116]: 2024-03-31T23:38:56.175+0000 7f64c04a8700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-31T23:38:13.357804+0000 front 2024-03-31T23:38:13.357906+0000 (oldest deadline 2024-03-31T23:38:38.057730+0000) 2024-03-31T23:38:56.443 INFO:journalctl@ceph.osd.10.smithi142.stdout:Mar 31 23:38:56 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-10[60116]: 2024-03-31T23:38:56.175+0000 7f64c04a8700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.142:6804 osd.1 since back 2024-03-31T23:38:20.958341+0000 front 2024-03-31T23:38:20.958408+0000 (oldest deadline 2024-03-31T23:38:45.058481+0000) 2024-03-31T23:38:56.443 INFO:journalctl@ceph.osd.10.smithi142.stdout:Mar 31 23:38:56 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-10[60116]: 2024-03-31T23:38:56.175+0000 7f64c04a8700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-31T23:38:25.559393+0000 front 2024-03-31T23:38:25.559313+0000 (oldest deadline 2024-03-31T23:38:51.459398+0000) 2024-03-31T23:38:56.443 INFO:journalctl@ceph.osd.7.smithi142.stdout:Mar 31 23:38:56 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-7[53065]: 2024-03-31T23:38:56.388+0000 7fa41768e700 -1 osd.7 139 heartbeat_check: no reply from 172.21.15.142:6804 osd.1 since back 2024-03-31T23:38:18.318617+0000 front 2024-03-31T23:38:18.318635+0000 (oldest deadline 2024-03-31T23:38:48.319131+0000) 2024-03-31T23:38:56.443 INFO:journalctl@ceph.osd.7.smithi142.stdout:Mar 31 23:38:56 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-7[53065]: 2024-03-31T23:38:56.388+0000 7fa41768e700 -1 osd.7 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-31T23:38:28.319471+0000 front 2024-03-31T23:38:28.319515+0000 (oldest deadline 2024-03-31T23:38:54.219616+0000) 2024-03-31T23:38:56.713 INFO:journalctl@ceph.osd.5.smithi162.stdout:Mar 31 23:38:56 smithi162 podman[78348]: 2024-03-31 23:38:56.49089227 +0000 UTC m=+0.738954131 container remove 2d982a6028966b4ae35b5ad315848e01782f3d8cc65b99bae45ce0b87924c64c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-5, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-384f31f, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240326, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0) 2024-03-31T23:38:56.979 INFO:journalctl@ceph.osd.5.smithi162.stdout:Mar 31 23:38:56 smithi162 podman[78444]: 2024-03-31 23:38:56.737758582 +0000 UTC m=+0.018220384 image pull ec18d7baa60e8b1a0b126fbe5306acce11b1ed0ec5e3bcdddbf39b0a3c591791 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175 2024-03-31T23:38:56.979 INFO:journalctl@ceph.osd.5.smithi162.stdout:Mar 31 23:38:56 smithi162 podman[78444]: 2024-03-31 23:38:56.858520821 +0000 UTC m=+0.138982598 container create bd7f116326b5b959ffae7bd2a42db79bee3ad6606864d9ae3b226374114dc928 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-5-deactivate, org.label-schema.build-date=20240326, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, RELEASE=squid-384f31f, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, io.buildah.version=1.33.5) 2024-03-31T23:38:56.979 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:38:56 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:38:56.712+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-31T23:38:12.221366+0000 front 2024-03-31T23:38:12.221347+0000 (oldest deadline 2024-03-31T23:38:37.521357+0000) 2024-03-31T23:38:56.979 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:38:56 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:38:56.712+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.142:6804 osd.1 since back 2024-03-31T23:38:22.222192+0000 front 2024-03-31T23:38:22.222213+0000 (oldest deadline 2024-03-31T23:38:45.722340+0000) 2024-03-31T23:38:56.979 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:38:56 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:38:56.712+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-31T23:38:29.723616+0000 front 2024-03-31T23:38:29.723593+0000 (oldest deadline 2024-03-31T23:38:52.023779+0000) 2024-03-31T23:38:57.295 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 31 23:38:57 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-8[55211]: 2024-03-31T23:38:57.116+0000 7fee85469700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-31T23:38:12.286566+0000 front 2024-03-31T23:38:12.286547+0000 (oldest deadline 2024-03-31T23:38:36.986433+0000) 2024-03-31T23:38:57.296 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 31 23:38:57 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-8[55211]: 2024-03-31T23:38:57.116+0000 7fee85469700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.142:6804 osd.1 since back 2024-03-31T23:38:21.587779+0000 front 2024-03-31T23:38:21.587835+0000 (oldest deadline 2024-03-31T23:38:47.487919+0000) 2024-03-31T23:38:57.296 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 31 23:38:57 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-8[55211]: 2024-03-31T23:38:57.116+0000 7fee85469700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-31T23:38:27.488310+0000 front 2024-03-31T23:38:27.488293+0000 (oldest deadline 2024-03-31T23:38:53.388420+0000) 2024-03-31T23:38:57.296 INFO:journalctl@ceph.osd.5.smithi162.stdout:Mar 31 23:38:57 smithi162 podman[78444]: 2024-03-31 23:38:57.094527878 +0000 UTC m=+0.374989668 container init bd7f116326b5b959ffae7bd2a42db79bee3ad6606864d9ae3b226374114dc928 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-5-deactivate, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240326, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-384f31f, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-31T23:38:57.296 INFO:journalctl@ceph.osd.5.smithi162.stdout:Mar 31 23:38:57 smithi162 podman[78444]: 2024-03-31 23:38:57.097833626 +0000 UTC m=+0.378295402 container start bd7f116326b5b959ffae7bd2a42db79bee3ad6606864d9ae3b226374114dc928 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-5-deactivate, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240326, org.label-schema.schema-version=1.0, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, ceph=True, RELEASE=squid-384f31f, maintainer=Guillaume Abrioux ) 2024-03-31T23:38:57.296 INFO:journalctl@ceph.osd.5.smithi162.stdout:Mar 31 23:38:57 smithi162 podman[78444]: 2024-03-31 23:38:57.194948102 +0000 UTC m=+0.475409901 container attach bd7f116326b5b959ffae7bd2a42db79bee3ad6606864d9ae3b226374114dc928 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-5-deactivate, RELEASE=squid-384f31f, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240326) 2024-03-31T23:38:57.611 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 31 23:38:57 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-9[67499]: 2024-03-31T23:38:57.113+0000 7fcc07615700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-31T23:38:10.469941+0000 front 2024-03-31T23:38:10.470388+0000 (oldest deadline 2024-03-31T23:38:35.170101+0000) 2024-03-31T23:38:57.612 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 31 23:38:57 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-9[67499]: 2024-03-31T23:38:57.113+0000 7fcc07615700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.142:6804 osd.1 since back 2024-03-31T23:38:21.571770+0000 front 2024-03-31T23:38:21.571801+0000 (oldest deadline 2024-03-31T23:38:43.871904+0000) 2024-03-31T23:38:57.612 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 31 23:38:57 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-9[67499]: 2024-03-31T23:38:57.113+0000 7fcc07615700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-31T23:38:27.074768+0000 front 2024-03-31T23:38:27.074793+0000 (oldest deadline 2024-03-31T23:38:52.374931+0000) 2024-03-31T23:38:57.612 INFO:journalctl@ceph.osd.6.smithi046.stdout:Mar 31 23:38:57 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-6[60760]: 2024-03-31T23:38:57.182+0000 7f8b7cce2700 -1 osd.6 139 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-31T23:38:12.034796+0000 front 2024-03-31T23:38:12.034926+0000 (oldest deadline 2024-03-31T23:38:34.335020+0000) 2024-03-31T23:38:57.612 INFO:journalctl@ceph.osd.6.smithi046.stdout:Mar 31 23:38:57 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-6[60760]: 2024-03-31T23:38:57.182+0000 7f8b7cce2700 -1 osd.6 139 heartbeat_check: no reply from 172.21.15.142:6804 osd.1 since back 2024-03-31T23:38:19.636274+0000 front 2024-03-31T23:38:19.636207+0000 (oldest deadline 2024-03-31T23:38:44.936378+0000) 2024-03-31T23:38:57.612 INFO:journalctl@ceph.osd.6.smithi046.stdout:Mar 31 23:38:57 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-6[60760]: 2024-03-31T23:38:57.182+0000 7f8b7cce2700 -1 osd.6 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-31T23:38:26.637553+0000 front 2024-03-31T23:38:26.637578+0000 (oldest deadline 2024-03-31T23:38:52.537783+0000) 2024-03-31T23:38:57.666 INFO:journalctl@ceph.osd.5.smithi162.stdout:Mar 31 23:38:57 smithi162 podman[78444]: 2024-03-31 23:38:57.414064071 +0000 UTC m=+0.694525848 container died bd7f116326b5b959ffae7bd2a42db79bee3ad6606864d9ae3b226374114dc928 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-5-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, RELEASE=squid-384f31f, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240326) 2024-03-31T23:38:57.693 INFO:journalctl@ceph.osd.10.smithi142.stdout:Mar 31 23:38:57 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-10[60116]: 2024-03-31T23:38:57.214+0000 7f64c04a8700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-31T23:38:13.357804+0000 front 2024-03-31T23:38:13.357906+0000 (oldest deadline 2024-03-31T23:38:38.057730+0000) 2024-03-31T23:38:57.693 INFO:journalctl@ceph.osd.10.smithi142.stdout:Mar 31 23:38:57 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-10[60116]: 2024-03-31T23:38:57.214+0000 7f64c04a8700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.142:6804 osd.1 since back 2024-03-31T23:38:20.958341+0000 front 2024-03-31T23:38:20.958408+0000 (oldest deadline 2024-03-31T23:38:45.058481+0000) 2024-03-31T23:38:57.693 INFO:journalctl@ceph.osd.10.smithi142.stdout:Mar 31 23:38:57 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-10[60116]: 2024-03-31T23:38:57.214+0000 7f64c04a8700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-31T23:38:25.559393+0000 front 2024-03-31T23:38:25.559313+0000 (oldest deadline 2024-03-31T23:38:51.459398+0000) 2024-03-31T23:38:57.693 INFO:journalctl@ceph.osd.7.smithi142.stdout:Mar 31 23:38:57 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-7[53065]: 2024-03-31T23:38:57.350+0000 7fa41768e700 -1 osd.7 139 heartbeat_check: no reply from 172.21.15.142:6804 osd.1 since back 2024-03-31T23:38:18.318617+0000 front 2024-03-31T23:38:18.318635+0000 (oldest deadline 2024-03-31T23:38:48.319131+0000) 2024-03-31T23:38:57.693 INFO:journalctl@ceph.osd.7.smithi142.stdout:Mar 31 23:38:57 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-7[53065]: 2024-03-31T23:38:57.350+0000 7fa41768e700 -1 osd.7 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-31T23:38:28.319471+0000 front 2024-03-31T23:38:28.319515+0000 (oldest deadline 2024-03-31T23:38:54.219616+0000) 2024-03-31T23:38:58.045 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:38:57 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:38:57.664+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-31T23:38:12.221366+0000 front 2024-03-31T23:38:12.221347+0000 (oldest deadline 2024-03-31T23:38:37.521357+0000) 2024-03-31T23:38:58.046 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:38:57 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:38:57.664+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.142:6804 osd.1 since back 2024-03-31T23:38:22.222192+0000 front 2024-03-31T23:38:22.222213+0000 (oldest deadline 2024-03-31T23:38:45.722340+0000) 2024-03-31T23:38:58.046 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:38:57 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:38:57.664+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-31T23:38:29.723616+0000 front 2024-03-31T23:38:29.723593+0000 (oldest deadline 2024-03-31T23:38:52.023779+0000) 2024-03-31T23:38:58.087 DEBUG:teuthology.orchestra.run.smithi162:> sudo pkill -f 'journalctl -f -n 0 -u ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4@osd.5.service' 2024-03-31T23:38:58.171 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-31T23:38:58.171 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-03-31T23:38:58.172 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-03-31T23:38:58.172 DEBUG:teuthology.orchestra.run.smithi046:> sudo systemctl stop ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4@osd.6 2024-03-31T23:38:58.361 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 31 23:38:58 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-9[67499]: 2024-03-31T23:38:58.064+0000 7fcc07615700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-31T23:38:10.469941+0000 front 2024-03-31T23:38:10.470388+0000 (oldest deadline 2024-03-31T23:38:35.170101+0000) 2024-03-31T23:38:58.362 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 31 23:38:58 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-9[67499]: 2024-03-31T23:38:58.064+0000 7fcc07615700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.142:6804 osd.1 since back 2024-03-31T23:38:21.571770+0000 front 2024-03-31T23:38:21.571801+0000 (oldest deadline 2024-03-31T23:38:43.871904+0000) 2024-03-31T23:38:58.362 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 31 23:38:58 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-9[67499]: 2024-03-31T23:38:58.064+0000 7fcc07615700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-31T23:38:27.074768+0000 front 2024-03-31T23:38:27.074793+0000 (oldest deadline 2024-03-31T23:38:52.374931+0000) 2024-03-31T23:38:58.362 INFO:journalctl@ceph.osd.6.smithi046.stdout:Mar 31 23:38:58 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-6[60760]: 2024-03-31T23:38:58.203+0000 7f8b7cce2700 -1 osd.6 139 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-31T23:38:12.034796+0000 front 2024-03-31T23:38:12.034926+0000 (oldest deadline 2024-03-31T23:38:34.335020+0000) 2024-03-31T23:38:58.362 INFO:journalctl@ceph.osd.6.smithi046.stdout:Mar 31 23:38:58 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-6[60760]: 2024-03-31T23:38:58.203+0000 7f8b7cce2700 -1 osd.6 139 heartbeat_check: no reply from 172.21.15.142:6804 osd.1 since back 2024-03-31T23:38:19.636274+0000 front 2024-03-31T23:38:19.636207+0000 (oldest deadline 2024-03-31T23:38:44.936378+0000) 2024-03-31T23:38:58.362 INFO:journalctl@ceph.osd.6.smithi046.stdout:Mar 31 23:38:58 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-6[60760]: 2024-03-31T23:38:58.203+0000 7f8b7cce2700 -1 osd.6 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-31T23:38:26.637553+0000 front 2024-03-31T23:38:26.637578+0000 (oldest deadline 2024-03-31T23:38:52.537783+0000) 2024-03-31T23:38:58.362 INFO:journalctl@ceph.osd.6.smithi046.stdout:Mar 31 23:38:58 smithi046 systemd[1]: Stopping Ceph osd.6 for 6a9d8f08-efb2-11ee-b647-cb9ed24678a4... 2024-03-31T23:38:58.545 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 31 23:38:58 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-8[55211]: 2024-03-31T23:38:58.161+0000 7fee85469700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-31T23:38:12.286566+0000 front 2024-03-31T23:38:12.286547+0000 (oldest deadline 2024-03-31T23:38:36.986433+0000) 2024-03-31T23:38:58.545 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 31 23:38:58 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-8[55211]: 2024-03-31T23:38:58.161+0000 7fee85469700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.142:6804 osd.1 since back 2024-03-31T23:38:21.587779+0000 front 2024-03-31T23:38:21.587835+0000 (oldest deadline 2024-03-31T23:38:47.487919+0000) 2024-03-31T23:38:58.545 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 31 23:38:58 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-8[55211]: 2024-03-31T23:38:58.161+0000 7fee85469700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-31T23:38:27.488310+0000 front 2024-03-31T23:38:27.488293+0000 (oldest deadline 2024-03-31T23:38:53.388420+0000) 2024-03-31T23:38:58.693 INFO:journalctl@ceph.osd.7.smithi142.stdout:Mar 31 23:38:58 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-7[53065]: 2024-03-31T23:38:58.306+0000 7fa41768e700 -1 osd.7 139 heartbeat_check: no reply from 172.21.15.142:6804 osd.1 since back 2024-03-31T23:38:18.318617+0000 front 2024-03-31T23:38:18.318635+0000 (oldest deadline 2024-03-31T23:38:48.319131+0000) 2024-03-31T23:38:58.693 INFO:journalctl@ceph.osd.7.smithi142.stdout:Mar 31 23:38:58 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-7[53065]: 2024-03-31T23:38:58.306+0000 7fa41768e700 -1 osd.7 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-31T23:38:28.319471+0000 front 2024-03-31T23:38:28.319515+0000 (oldest deadline 2024-03-31T23:38:54.219616+0000) 2024-03-31T23:38:58.693 INFO:journalctl@ceph.osd.10.smithi142.stdout:Mar 31 23:38:58 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-10[60116]: 2024-03-31T23:38:58.261+0000 7f64c04a8700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-31T23:38:13.357804+0000 front 2024-03-31T23:38:13.357906+0000 (oldest deadline 2024-03-31T23:38:38.057730+0000) 2024-03-31T23:38:58.694 INFO:journalctl@ceph.osd.10.smithi142.stdout:Mar 31 23:38:58 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-10[60116]: 2024-03-31T23:38:58.261+0000 7f64c04a8700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.142:6804 osd.1 since back 2024-03-31T23:38:20.958341+0000 front 2024-03-31T23:38:20.958408+0000 (oldest deadline 2024-03-31T23:38:45.058481+0000) 2024-03-31T23:38:58.694 INFO:journalctl@ceph.osd.10.smithi142.stdout:Mar 31 23:38:58 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-10[60116]: 2024-03-31T23:38:58.261+0000 7f64c04a8700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-31T23:38:25.559393+0000 front 2024-03-31T23:38:25.559313+0000 (oldest deadline 2024-03-31T23:38:51.459398+0000) 2024-03-31T23:38:58.861 INFO:journalctl@ceph.osd.6.smithi046.stdout:Mar 31 23:38:58 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-6[60760]: 2024-03-31T23:38:58.537+0000 7f8b806ee700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-31T23:38:58.861 INFO:journalctl@ceph.osd.6.smithi046.stdout:Mar 31 23:38:58 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-6[60760]: 2024-03-31T23:38:58.537+0000 7f8b806ee700 -1 osd.6 139 *** Got signal Terminated *** 2024-03-31T23:38:58.862 INFO:journalctl@ceph.osd.6.smithi046.stdout:Mar 31 23:38:58 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-6[60760]: 2024-03-31T23:38:58.537+0000 7f8b806ee700 -1 osd.6 139 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-31T23:38:59.045 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:38:58 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:38:58.686+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-31T23:38:12.221366+0000 front 2024-03-31T23:38:12.221347+0000 (oldest deadline 2024-03-31T23:38:37.521357+0000) 2024-03-31T23:38:59.045 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:38:58 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:38:58.686+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.142:6804 osd.1 since back 2024-03-31T23:38:22.222192+0000 front 2024-03-31T23:38:22.222213+0000 (oldest deadline 2024-03-31T23:38:45.722340+0000) 2024-03-31T23:38:59.045 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:38:58 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:38:58.686+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-31T23:38:29.723616+0000 front 2024-03-31T23:38:29.723593+0000 (oldest deadline 2024-03-31T23:38:52.023779+0000) 2024-03-31T23:38:59.361 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 31 23:38:59 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-9[67499]: 2024-03-31T23:38:59.020+0000 7fcc07615700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-31T23:38:10.469941+0000 front 2024-03-31T23:38:10.470388+0000 (oldest deadline 2024-03-31T23:38:35.170101+0000) 2024-03-31T23:38:59.362 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 31 23:38:59 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-9[67499]: 2024-03-31T23:38:59.020+0000 7fcc07615700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.142:6804 osd.1 since back 2024-03-31T23:38:21.571770+0000 front 2024-03-31T23:38:21.571801+0000 (oldest deadline 2024-03-31T23:38:43.871904+0000) 2024-03-31T23:38:59.362 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 31 23:38:59 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-9[67499]: 2024-03-31T23:38:59.020+0000 7fcc07615700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-31T23:38:27.074768+0000 front 2024-03-31T23:38:27.074793+0000 (oldest deadline 2024-03-31T23:38:52.374931+0000) 2024-03-31T23:38:59.362 INFO:journalctl@ceph.osd.6.smithi046.stdout:Mar 31 23:38:59 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-6[60760]: 2024-03-31T23:38:59.194+0000 7f8b7cce2700 -1 osd.6 139 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-31T23:38:12.034796+0000 front 2024-03-31T23:38:12.034926+0000 (oldest deadline 2024-03-31T23:38:34.335020+0000) 2024-03-31T23:38:59.362 INFO:journalctl@ceph.osd.6.smithi046.stdout:Mar 31 23:38:59 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-6[60760]: 2024-03-31T23:38:59.194+0000 7f8b7cce2700 -1 osd.6 139 heartbeat_check: no reply from 172.21.15.142:6804 osd.1 since back 2024-03-31T23:38:19.636274+0000 front 2024-03-31T23:38:19.636207+0000 (oldest deadline 2024-03-31T23:38:44.936378+0000) 2024-03-31T23:38:59.362 INFO:journalctl@ceph.osd.6.smithi046.stdout:Mar 31 23:38:59 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-6[60760]: 2024-03-31T23:38:59.194+0000 7f8b7cce2700 -1 osd.6 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-31T23:38:26.637553+0000 front 2024-03-31T23:38:26.637578+0000 (oldest deadline 2024-03-31T23:38:52.537783+0000) 2024-03-31T23:38:59.545 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 31 23:38:59 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-8[55211]: 2024-03-31T23:38:59.198+0000 7fee85469700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-31T23:38:12.286566+0000 front 2024-03-31T23:38:12.286547+0000 (oldest deadline 2024-03-31T23:38:36.986433+0000) 2024-03-31T23:38:59.545 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 31 23:38:59 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-8[55211]: 2024-03-31T23:38:59.198+0000 7fee85469700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.142:6804 osd.1 since back 2024-03-31T23:38:21.587779+0000 front 2024-03-31T23:38:21.587835+0000 (oldest deadline 2024-03-31T23:38:47.487919+0000) 2024-03-31T23:38:59.545 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 31 23:38:59 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-8[55211]: 2024-03-31T23:38:59.198+0000 7fee85469700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-31T23:38:27.488310+0000 front 2024-03-31T23:38:27.488293+0000 (oldest deadline 2024-03-31T23:38:53.388420+0000) 2024-03-31T23:38:59.693 INFO:journalctl@ceph.osd.7.smithi142.stdout:Mar 31 23:38:59 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-7[53065]: 2024-03-31T23:38:59.326+0000 7fa41768e700 -1 osd.7 139 heartbeat_check: no reply from 172.21.15.142:6804 osd.1 since back 2024-03-31T23:38:18.318617+0000 front 2024-03-31T23:38:18.318635+0000 (oldest deadline 2024-03-31T23:38:48.319131+0000) 2024-03-31T23:38:59.693 INFO:journalctl@ceph.osd.7.smithi142.stdout:Mar 31 23:38:59 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-7[53065]: 2024-03-31T23:38:59.326+0000 7fa41768e700 -1 osd.7 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-31T23:38:28.319471+0000 front 2024-03-31T23:38:28.319515+0000 (oldest deadline 2024-03-31T23:38:54.219616+0000) 2024-03-31T23:38:59.693 INFO:journalctl@ceph.osd.10.smithi142.stdout:Mar 31 23:38:59 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-10[60116]: 2024-03-31T23:38:59.307+0000 7f64c04a8700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-31T23:38:13.357804+0000 front 2024-03-31T23:38:13.357906+0000 (oldest deadline 2024-03-31T23:38:38.057730+0000) 2024-03-31T23:38:59.693 INFO:journalctl@ceph.osd.10.smithi142.stdout:Mar 31 23:38:59 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-10[60116]: 2024-03-31T23:38:59.307+0000 7f64c04a8700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.142:6804 osd.1 since back 2024-03-31T23:38:20.958341+0000 front 2024-03-31T23:38:20.958408+0000 (oldest deadline 2024-03-31T23:38:45.058481+0000) 2024-03-31T23:38:59.693 INFO:journalctl@ceph.osd.10.smithi142.stdout:Mar 31 23:38:59 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-10[60116]: 2024-03-31T23:38:59.307+0000 7f64c04a8700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-31T23:38:25.559393+0000 front 2024-03-31T23:38:25.559313+0000 (oldest deadline 2024-03-31T23:38:51.459398+0000) 2024-03-31T23:39:00.045 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:38:59 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:38:59.707+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-31T23:38:12.221366+0000 front 2024-03-31T23:38:12.221347+0000 (oldest deadline 2024-03-31T23:38:37.521357+0000) 2024-03-31T23:39:00.045 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:38:59 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:38:59.707+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.142:6804 osd.1 since back 2024-03-31T23:38:22.222192+0000 front 2024-03-31T23:38:22.222213+0000 (oldest deadline 2024-03-31T23:38:45.722340+0000) 2024-03-31T23:39:00.046 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:38:59 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:38:59.707+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-31T23:38:29.723616+0000 front 2024-03-31T23:38:29.723593+0000 (oldest deadline 2024-03-31T23:38:52.023779+0000) 2024-03-31T23:39:00.361 INFO:journalctl@ceph.osd.6.smithi046.stdout:Mar 31 23:39:00 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-6[60760]: 2024-03-31T23:39:00.233+0000 7f8b7cce2700 -1 osd.6 139 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-31T23:38:12.034796+0000 front 2024-03-31T23:38:12.034926+0000 (oldest deadline 2024-03-31T23:38:34.335020+0000) 2024-03-31T23:39:00.362 INFO:journalctl@ceph.osd.6.smithi046.stdout:Mar 31 23:39:00 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-6[60760]: 2024-03-31T23:39:00.233+0000 7f8b7cce2700 -1 osd.6 139 heartbeat_check: no reply from 172.21.15.142:6804 osd.1 since back 2024-03-31T23:38:19.636274+0000 front 2024-03-31T23:38:19.636207+0000 (oldest deadline 2024-03-31T23:38:44.936378+0000) 2024-03-31T23:39:00.362 INFO:journalctl@ceph.osd.6.smithi046.stdout:Mar 31 23:39:00 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-6[60760]: 2024-03-31T23:39:00.233+0000 7f8b7cce2700 -1 osd.6 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-31T23:38:26.637553+0000 front 2024-03-31T23:38:26.637578+0000 (oldest deadline 2024-03-31T23:38:52.537783+0000) 2024-03-31T23:39:00.362 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 31 23:39:00 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-9[67499]: 2024-03-31T23:39:00.020+0000 7fcc07615700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-31T23:38:10.469941+0000 front 2024-03-31T23:38:10.470388+0000 (oldest deadline 2024-03-31T23:38:35.170101+0000) 2024-03-31T23:39:00.362 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 31 23:39:00 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-9[67499]: 2024-03-31T23:39:00.020+0000 7fcc07615700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.142:6804 osd.1 since back 2024-03-31T23:38:21.571770+0000 front 2024-03-31T23:38:21.571801+0000 (oldest deadline 2024-03-31T23:38:43.871904+0000) 2024-03-31T23:39:00.362 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 31 23:39:00 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-9[67499]: 2024-03-31T23:39:00.020+0000 7fcc07615700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-31T23:38:27.074768+0000 front 2024-03-31T23:38:27.074793+0000 (oldest deadline 2024-03-31T23:38:52.374931+0000) 2024-03-31T23:39:00.362 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 31 23:39:00 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-9[67499]: 2024-03-31T23:39:00.020+0000 7fcc07615700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-03-31T23:38:34.075913+0000 front 2024-03-31T23:38:34.075868+0000 (oldest deadline 2024-03-31T23:38:59.376243+0000) 2024-03-31T23:39:00.545 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 31 23:39:00 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-8[55211]: 2024-03-31T23:39:00.173+0000 7fee85469700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-31T23:38:12.286566+0000 front 2024-03-31T23:38:12.286547+0000 (oldest deadline 2024-03-31T23:38:36.986433+0000) 2024-03-31T23:39:00.546 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 31 23:39:00 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-8[55211]: 2024-03-31T23:39:00.173+0000 7fee85469700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.142:6804 osd.1 since back 2024-03-31T23:38:21.587779+0000 front 2024-03-31T23:38:21.587835+0000 (oldest deadline 2024-03-31T23:38:47.487919+0000) 2024-03-31T23:39:00.546 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 31 23:39:00 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-8[55211]: 2024-03-31T23:39:00.173+0000 7fee85469700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-31T23:38:27.488310+0000 front 2024-03-31T23:38:27.488293+0000 (oldest deadline 2024-03-31T23:38:53.388420+0000) 2024-03-31T23:39:00.546 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 31 23:39:00 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-8[55211]: 2024-03-31T23:39:00.173+0000 7fee85469700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-03-31T23:38:33.389031+0000 front 2024-03-31T23:38:33.388915+0000 (oldest deadline 2024-03-31T23:38:59.288849+0000) 2024-03-31T23:39:00.693 INFO:journalctl@ceph.osd.7.smithi142.stdout:Mar 31 23:39:00 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-7[53065]: 2024-03-31T23:39:00.359+0000 7fa41768e700 -1 osd.7 139 heartbeat_check: no reply from 172.21.15.142:6804 osd.1 since back 2024-03-31T23:38:18.318617+0000 front 2024-03-31T23:38:18.318635+0000 (oldest deadline 2024-03-31T23:38:48.319131+0000) 2024-03-31T23:39:00.693 INFO:journalctl@ceph.osd.7.smithi142.stdout:Mar 31 23:39:00 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-7[53065]: 2024-03-31T23:39:00.359+0000 7fa41768e700 -1 osd.7 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-31T23:38:28.319471+0000 front 2024-03-31T23:38:28.319515+0000 (oldest deadline 2024-03-31T23:38:54.219616+0000) 2024-03-31T23:39:00.693 INFO:journalctl@ceph.osd.7.smithi142.stdout:Mar 31 23:39:00 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-7[53065]: 2024-03-31T23:39:00.359+0000 7fa41768e700 -1 osd.7 139 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-03-31T23:38:36.421116+0000 front 2024-03-31T23:38:36.420912+0000 (oldest deadline 2024-03-31T23:38:59.920889+0000) 2024-03-31T23:39:00.693 INFO:journalctl@ceph.osd.10.smithi142.stdout:Mar 31 23:39:00 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-10[60116]: 2024-03-31T23:39:00.352+0000 7f64c04a8700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-31T23:38:13.357804+0000 front 2024-03-31T23:38:13.357906+0000 (oldest deadline 2024-03-31T23:38:38.057730+0000) 2024-03-31T23:39:00.694 INFO:journalctl@ceph.osd.10.smithi142.stdout:Mar 31 23:39:00 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-10[60116]: 2024-03-31T23:39:00.352+0000 7f64c04a8700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.142:6804 osd.1 since back 2024-03-31T23:38:20.958341+0000 front 2024-03-31T23:38:20.958408+0000 (oldest deadline 2024-03-31T23:38:45.058481+0000) 2024-03-31T23:39:00.694 INFO:journalctl@ceph.osd.10.smithi142.stdout:Mar 31 23:39:00 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-10[60116]: 2024-03-31T23:39:00.352+0000 7f64c04a8700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-31T23:38:25.559393+0000 front 2024-03-31T23:38:25.559313+0000 (oldest deadline 2024-03-31T23:38:51.459398+0000) 2024-03-31T23:39:01.045 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:39:00 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:39:00.673+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-31T23:38:12.221366+0000 front 2024-03-31T23:38:12.221347+0000 (oldest deadline 2024-03-31T23:38:37.521357+0000) 2024-03-31T23:39:01.046 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:39:00 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:39:00.673+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.142:6804 osd.1 since back 2024-03-31T23:38:22.222192+0000 front 2024-03-31T23:38:22.222213+0000 (oldest deadline 2024-03-31T23:38:45.722340+0000) 2024-03-31T23:39:01.046 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:39:00 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:39:00.673+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-31T23:38:29.723616+0000 front 2024-03-31T23:38:29.723593+0000 (oldest deadline 2024-03-31T23:38:52.023779+0000) 2024-03-31T23:39:01.361 INFO:journalctl@ceph.osd.6.smithi046.stdout:Mar 31 23:39:01 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-6[60760]: 2024-03-31T23:39:01.212+0000 7f8b7cce2700 -1 osd.6 139 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-31T23:38:12.034796+0000 front 2024-03-31T23:38:12.034926+0000 (oldest deadline 2024-03-31T23:38:34.335020+0000) 2024-03-31T23:39:01.362 INFO:journalctl@ceph.osd.6.smithi046.stdout:Mar 31 23:39:01 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-6[60760]: 2024-03-31T23:39:01.212+0000 7f8b7cce2700 -1 osd.6 139 heartbeat_check: no reply from 172.21.15.142:6804 osd.1 since back 2024-03-31T23:38:19.636274+0000 front 2024-03-31T23:38:19.636207+0000 (oldest deadline 2024-03-31T23:38:44.936378+0000) 2024-03-31T23:39:01.362 INFO:journalctl@ceph.osd.6.smithi046.stdout:Mar 31 23:39:01 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-6[60760]: 2024-03-31T23:39:01.212+0000 7f8b7cce2700 -1 osd.6 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-31T23:38:26.637553+0000 front 2024-03-31T23:38:26.637578+0000 (oldest deadline 2024-03-31T23:38:52.537783+0000) 2024-03-31T23:39:01.362 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 31 23:39:01 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-9[67499]: 2024-03-31T23:39:01.002+0000 7fcc07615700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-31T23:38:10.469941+0000 front 2024-03-31T23:38:10.470388+0000 (oldest deadline 2024-03-31T23:38:35.170101+0000) 2024-03-31T23:39:01.362 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 31 23:39:01 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-9[67499]: 2024-03-31T23:39:01.002+0000 7fcc07615700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.142:6804 osd.1 since back 2024-03-31T23:38:21.571770+0000 front 2024-03-31T23:38:21.571801+0000 (oldest deadline 2024-03-31T23:38:43.871904+0000) 2024-03-31T23:39:01.362 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 31 23:39:01 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-9[67499]: 2024-03-31T23:39:01.002+0000 7fcc07615700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-31T23:38:27.074768+0000 front 2024-03-31T23:38:27.074793+0000 (oldest deadline 2024-03-31T23:38:52.374931+0000) 2024-03-31T23:39:01.362 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 31 23:39:01 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-9[67499]: 2024-03-31T23:39:01.002+0000 7fcc07615700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-03-31T23:38:34.075913+0000 front 2024-03-31T23:38:34.075868+0000 (oldest deadline 2024-03-31T23:38:59.376243+0000) 2024-03-31T23:39:01.545 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 31 23:39:01 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-8[55211]: 2024-03-31T23:39:01.193+0000 7fee85469700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-31T23:38:12.286566+0000 front 2024-03-31T23:38:12.286547+0000 (oldest deadline 2024-03-31T23:38:36.986433+0000) 2024-03-31T23:39:01.545 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 31 23:39:01 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-8[55211]: 2024-03-31T23:39:01.193+0000 7fee85469700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.142:6804 osd.1 since back 2024-03-31T23:38:21.587779+0000 front 2024-03-31T23:38:21.587835+0000 (oldest deadline 2024-03-31T23:38:47.487919+0000) 2024-03-31T23:39:01.545 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 31 23:39:01 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-8[55211]: 2024-03-31T23:39:01.193+0000 7fee85469700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-31T23:38:27.488310+0000 front 2024-03-31T23:38:27.488293+0000 (oldest deadline 2024-03-31T23:38:53.388420+0000) 2024-03-31T23:39:01.546 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 31 23:39:01 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-8[55211]: 2024-03-31T23:39:01.193+0000 7fee85469700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-03-31T23:38:33.389031+0000 front 2024-03-31T23:38:33.388915+0000 (oldest deadline 2024-03-31T23:38:59.288849+0000) 2024-03-31T23:39:01.693 INFO:journalctl@ceph.osd.10.smithi142.stdout:Mar 31 23:39:01 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-10[60116]: 2024-03-31T23:39:01.372+0000 7f64c04a8700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-31T23:38:13.357804+0000 front 2024-03-31T23:38:13.357906+0000 (oldest deadline 2024-03-31T23:38:38.057730+0000) 2024-03-31T23:39:01.693 INFO:journalctl@ceph.osd.10.smithi142.stdout:Mar 31 23:39:01 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-10[60116]: 2024-03-31T23:39:01.372+0000 7f64c04a8700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.142:6804 osd.1 since back 2024-03-31T23:38:20.958341+0000 front 2024-03-31T23:38:20.958408+0000 (oldest deadline 2024-03-31T23:38:45.058481+0000) 2024-03-31T23:39:01.693 INFO:journalctl@ceph.osd.10.smithi142.stdout:Mar 31 23:39:01 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-10[60116]: 2024-03-31T23:39:01.372+0000 7f64c04a8700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-31T23:38:25.559393+0000 front 2024-03-31T23:38:25.559313+0000 (oldest deadline 2024-03-31T23:38:51.459398+0000) 2024-03-31T23:39:01.693 INFO:journalctl@ceph.osd.10.smithi142.stdout:Mar 31 23:39:01 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-10[60116]: 2024-03-31T23:39:01.372+0000 7f64c04a8700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-03-31T23:38:36.660831+0000 front 2024-03-31T23:38:36.660954+0000 (oldest deadline 2024-03-31T23:39:00.760714+0000) 2024-03-31T23:39:01.694 INFO:journalctl@ceph.osd.7.smithi142.stdout:Mar 31 23:39:01 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-7[53065]: 2024-03-31T23:39:01.379+0000 7fa41768e700 -1 osd.7 139 heartbeat_check: no reply from 172.21.15.142:6804 osd.1 since back 2024-03-31T23:38:18.318617+0000 front 2024-03-31T23:38:18.318635+0000 (oldest deadline 2024-03-31T23:38:48.319131+0000) 2024-03-31T23:39:01.694 INFO:journalctl@ceph.osd.7.smithi142.stdout:Mar 31 23:39:01 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-7[53065]: 2024-03-31T23:39:01.379+0000 7fa41768e700 -1 osd.7 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-31T23:38:28.319471+0000 front 2024-03-31T23:38:28.319515+0000 (oldest deadline 2024-03-31T23:38:54.219616+0000) 2024-03-31T23:39:01.694 INFO:journalctl@ceph.osd.7.smithi142.stdout:Mar 31 23:39:01 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-7[53065]: 2024-03-31T23:39:01.379+0000 7fa41768e700 -1 osd.7 139 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-03-31T23:38:36.421116+0000 front 2024-03-31T23:38:36.420912+0000 (oldest deadline 2024-03-31T23:38:59.920889+0000) 2024-03-31T23:39:02.045 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:39:01 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:39:01.687+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-31T23:38:12.221366+0000 front 2024-03-31T23:38:12.221347+0000 (oldest deadline 2024-03-31T23:38:37.521357+0000) 2024-03-31T23:39:02.045 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:39:01 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:39:01.687+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.142:6804 osd.1 since back 2024-03-31T23:38:22.222192+0000 front 2024-03-31T23:38:22.222213+0000 (oldest deadline 2024-03-31T23:38:45.722340+0000) 2024-03-31T23:39:02.046 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:39:01 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:39:01.687+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-31T23:38:29.723616+0000 front 2024-03-31T23:38:29.723593+0000 (oldest deadline 2024-03-31T23:38:52.023779+0000) 2024-03-31T23:39:02.046 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:39:01 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:39:01.687+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-03-31T23:38:36.124750+0000 front 2024-03-31T23:38:36.124868+0000 (oldest deadline 2024-03-31T23:39:00.824619+0000) 2024-03-31T23:39:02.361 INFO:journalctl@ceph.osd.6.smithi046.stdout:Mar 31 23:39:02 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-6[60760]: 2024-03-31T23:39:02.204+0000 7f8b7cce2700 -1 osd.6 139 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-31T23:38:12.034796+0000 front 2024-03-31T23:38:12.034926+0000 (oldest deadline 2024-03-31T23:38:34.335020+0000) 2024-03-31T23:39:02.362 INFO:journalctl@ceph.osd.6.smithi046.stdout:Mar 31 23:39:02 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-6[60760]: 2024-03-31T23:39:02.204+0000 7f8b7cce2700 -1 osd.6 139 heartbeat_check: no reply from 172.21.15.142:6804 osd.1 since back 2024-03-31T23:38:19.636274+0000 front 2024-03-31T23:38:19.636207+0000 (oldest deadline 2024-03-31T23:38:44.936378+0000) 2024-03-31T23:39:02.362 INFO:journalctl@ceph.osd.6.smithi046.stdout:Mar 31 23:39:02 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-6[60760]: 2024-03-31T23:39:02.204+0000 7f8b7cce2700 -1 osd.6 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-31T23:38:26.637553+0000 front 2024-03-31T23:38:26.637578+0000 (oldest deadline 2024-03-31T23:38:52.537783+0000) 2024-03-31T23:39:02.362 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 31 23:39:02 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-9[67499]: 2024-03-31T23:39:02.003+0000 7fcc07615700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-31T23:38:10.469941+0000 front 2024-03-31T23:38:10.470388+0000 (oldest deadline 2024-03-31T23:38:35.170101+0000) 2024-03-31T23:39:02.362 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 31 23:39:02 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-9[67499]: 2024-03-31T23:39:02.003+0000 7fcc07615700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.142:6804 osd.1 since back 2024-03-31T23:38:21.571770+0000 front 2024-03-31T23:38:21.571801+0000 (oldest deadline 2024-03-31T23:38:43.871904+0000) 2024-03-31T23:39:02.362 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 31 23:39:02 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-9[67499]: 2024-03-31T23:39:02.003+0000 7fcc07615700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-31T23:38:27.074768+0000 front 2024-03-31T23:38:27.074793+0000 (oldest deadline 2024-03-31T23:38:52.374931+0000) 2024-03-31T23:39:02.362 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 31 23:39:02 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-9[67499]: 2024-03-31T23:39:02.003+0000 7fcc07615700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-03-31T23:38:34.075913+0000 front 2024-03-31T23:38:34.075868+0000 (oldest deadline 2024-03-31T23:38:59.376243+0000) 2024-03-31T23:39:02.545 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 31 23:39:02 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-8[55211]: 2024-03-31T23:39:02.173+0000 7fee85469700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-31T23:38:12.286566+0000 front 2024-03-31T23:38:12.286547+0000 (oldest deadline 2024-03-31T23:38:36.986433+0000) 2024-03-31T23:39:02.545 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 31 23:39:02 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-8[55211]: 2024-03-31T23:39:02.173+0000 7fee85469700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.142:6804 osd.1 since back 2024-03-31T23:38:21.587779+0000 front 2024-03-31T23:38:21.587835+0000 (oldest deadline 2024-03-31T23:38:47.487919+0000) 2024-03-31T23:39:02.545 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 31 23:39:02 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-8[55211]: 2024-03-31T23:39:02.173+0000 7fee85469700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-31T23:38:27.488310+0000 front 2024-03-31T23:38:27.488293+0000 (oldest deadline 2024-03-31T23:38:53.388420+0000) 2024-03-31T23:39:02.546 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 31 23:39:02 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-8[55211]: 2024-03-31T23:39:02.173+0000 7fee85469700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-03-31T23:38:33.389031+0000 front 2024-03-31T23:38:33.388915+0000 (oldest deadline 2024-03-31T23:38:59.288849+0000) 2024-03-31T23:39:02.693 INFO:journalctl@ceph.osd.10.smithi142.stdout:Mar 31 23:39:02 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-10[60116]: 2024-03-31T23:39:02.371+0000 7f64c04a8700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-31T23:38:13.357804+0000 front 2024-03-31T23:38:13.357906+0000 (oldest deadline 2024-03-31T23:38:38.057730+0000) 2024-03-31T23:39:02.693 INFO:journalctl@ceph.osd.10.smithi142.stdout:Mar 31 23:39:02 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-10[60116]: 2024-03-31T23:39:02.371+0000 7f64c04a8700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.142:6804 osd.1 since back 2024-03-31T23:38:20.958341+0000 front 2024-03-31T23:38:20.958408+0000 (oldest deadline 2024-03-31T23:38:45.058481+0000) 2024-03-31T23:39:02.693 INFO:journalctl@ceph.osd.10.smithi142.stdout:Mar 31 23:39:02 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-10[60116]: 2024-03-31T23:39:02.372+0000 7f64c04a8700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-31T23:38:25.559393+0000 front 2024-03-31T23:38:25.559313+0000 (oldest deadline 2024-03-31T23:38:51.459398+0000) 2024-03-31T23:39:02.693 INFO:journalctl@ceph.osd.10.smithi142.stdout:Mar 31 23:39:02 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-10[60116]: 2024-03-31T23:39:02.372+0000 7f64c04a8700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-03-31T23:38:36.660831+0000 front 2024-03-31T23:38:36.660954+0000 (oldest deadline 2024-03-31T23:39:00.760714+0000) 2024-03-31T23:39:02.693 INFO:journalctl@ceph.osd.7.smithi142.stdout:Mar 31 23:39:02 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-7[53065]: 2024-03-31T23:39:02.380+0000 7fa41768e700 -1 osd.7 139 heartbeat_check: no reply from 172.21.15.142:6804 osd.1 since back 2024-03-31T23:38:18.318617+0000 front 2024-03-31T23:38:18.318635+0000 (oldest deadline 2024-03-31T23:38:48.319131+0000) 2024-03-31T23:39:02.694 INFO:journalctl@ceph.osd.7.smithi142.stdout:Mar 31 23:39:02 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-7[53065]: 2024-03-31T23:39:02.380+0000 7fa41768e700 -1 osd.7 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-31T23:38:28.319471+0000 front 2024-03-31T23:38:28.319515+0000 (oldest deadline 2024-03-31T23:38:54.219616+0000) 2024-03-31T23:39:02.694 INFO:journalctl@ceph.osd.7.smithi142.stdout:Mar 31 23:39:02 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-7[53065]: 2024-03-31T23:39:02.380+0000 7fa41768e700 -1 osd.7 139 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-03-31T23:38:36.421116+0000 front 2024-03-31T23:38:36.420912+0000 (oldest deadline 2024-03-31T23:38:59.920889+0000) 2024-03-31T23:39:03.045 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:39:02 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:39:02.691+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-31T23:38:12.221366+0000 front 2024-03-31T23:38:12.221347+0000 (oldest deadline 2024-03-31T23:38:37.521357+0000) 2024-03-31T23:39:03.046 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:39:02 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:39:02.691+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.142:6804 osd.1 since back 2024-03-31T23:38:22.222192+0000 front 2024-03-31T23:38:22.222213+0000 (oldest deadline 2024-03-31T23:38:45.722340+0000) 2024-03-31T23:39:03.046 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:39:02 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:39:02.691+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-31T23:38:29.723616+0000 front 2024-03-31T23:38:29.723593+0000 (oldest deadline 2024-03-31T23:38:52.023779+0000) 2024-03-31T23:39:03.046 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:39:02 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:39:02.691+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-03-31T23:38:36.124750+0000 front 2024-03-31T23:38:36.124868+0000 (oldest deadline 2024-03-31T23:39:00.824619+0000) 2024-03-31T23:39:03.361 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 31 23:39:03 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-9[67499]: 2024-03-31T23:39:03.036+0000 7fcc07615700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-31T23:38:10.469941+0000 front 2024-03-31T23:38:10.470388+0000 (oldest deadline 2024-03-31T23:38:35.170101+0000) 2024-03-31T23:39:03.361 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 31 23:39:03 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-9[67499]: 2024-03-31T23:39:03.036+0000 7fcc07615700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.142:6804 osd.1 since back 2024-03-31T23:38:21.571770+0000 front 2024-03-31T23:38:21.571801+0000 (oldest deadline 2024-03-31T23:38:43.871904+0000) 2024-03-31T23:39:03.362 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 31 23:39:03 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-9[67499]: 2024-03-31T23:39:03.036+0000 7fcc07615700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-31T23:38:27.074768+0000 front 2024-03-31T23:38:27.074793+0000 (oldest deadline 2024-03-31T23:38:52.374931+0000) 2024-03-31T23:39:03.362 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 31 23:39:03 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-9[67499]: 2024-03-31T23:39:03.036+0000 7fcc07615700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-03-31T23:38:34.075913+0000 front 2024-03-31T23:38:34.075868+0000 (oldest deadline 2024-03-31T23:38:59.376243+0000) 2024-03-31T23:39:03.362 INFO:journalctl@ceph.osd.6.smithi046.stdout:Mar 31 23:39:03 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-6[60760]: 2024-03-31T23:39:03.168+0000 7f8b7cce2700 -1 osd.6 139 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-31T23:38:12.034796+0000 front 2024-03-31T23:38:12.034926+0000 (oldest deadline 2024-03-31T23:38:34.335020+0000) 2024-03-31T23:39:03.362 INFO:journalctl@ceph.osd.6.smithi046.stdout:Mar 31 23:39:03 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-6[60760]: 2024-03-31T23:39:03.168+0000 7f8b7cce2700 -1 osd.6 139 heartbeat_check: no reply from 172.21.15.142:6804 osd.1 since back 2024-03-31T23:38:19.636274+0000 front 2024-03-31T23:38:19.636207+0000 (oldest deadline 2024-03-31T23:38:44.936378+0000) 2024-03-31T23:39:03.362 INFO:journalctl@ceph.osd.6.smithi046.stdout:Mar 31 23:39:03 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-6[60760]: 2024-03-31T23:39:03.168+0000 7f8b7cce2700 -1 osd.6 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-31T23:38:26.637553+0000 front 2024-03-31T23:38:26.637578+0000 (oldest deadline 2024-03-31T23:38:52.537783+0000) 2024-03-31T23:39:03.362 INFO:journalctl@ceph.osd.6.smithi046.stdout:Mar 31 23:39:03 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-6[60760]: 2024-03-31T23:39:03.168+0000 7f8b7cce2700 -1 osd.6 139 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-03-31T23:38:37.739830+0000 front 2024-03-31T23:38:37.739855+0000 (oldest deadline 2024-03-31T23:39:02.440044+0000) 2024-03-31T23:39:03.545 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 31 23:39:03 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-8[55211]: 2024-03-31T23:39:03.145+0000 7fee85469700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-31T23:38:12.286566+0000 front 2024-03-31T23:38:12.286547+0000 (oldest deadline 2024-03-31T23:38:36.986433+0000) 2024-03-31T23:39:03.546 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 31 23:39:03 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-8[55211]: 2024-03-31T23:39:03.145+0000 7fee85469700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.142:6804 osd.1 since back 2024-03-31T23:38:21.587779+0000 front 2024-03-31T23:38:21.587835+0000 (oldest deadline 2024-03-31T23:38:47.487919+0000) 2024-03-31T23:39:03.546 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 31 23:39:03 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-8[55211]: 2024-03-31T23:39:03.145+0000 7fee85469700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-31T23:38:27.488310+0000 front 2024-03-31T23:38:27.488293+0000 (oldest deadline 2024-03-31T23:38:53.388420+0000) 2024-03-31T23:39:03.546 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 31 23:39:03 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-8[55211]: 2024-03-31T23:39:03.145+0000 7fee85469700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-03-31T23:38:33.389031+0000 front 2024-03-31T23:38:33.388915+0000 (oldest deadline 2024-03-31T23:38:59.288849+0000) 2024-03-31T23:39:03.693 INFO:journalctl@ceph.osd.7.smithi142.stdout:Mar 31 23:39:03 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-7[53065]: 2024-03-31T23:39:03.387+0000 7fa41768e700 -1 osd.7 139 heartbeat_check: no reply from 172.21.15.142:6804 osd.1 since back 2024-03-31T23:38:18.318617+0000 front 2024-03-31T23:38:18.318635+0000 (oldest deadline 2024-03-31T23:38:48.319131+0000) 2024-03-31T23:39:03.693 INFO:journalctl@ceph.osd.7.smithi142.stdout:Mar 31 23:39:03 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-7[53065]: 2024-03-31T23:39:03.387+0000 7fa41768e700 -1 osd.7 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-31T23:38:28.319471+0000 front 2024-03-31T23:38:28.319515+0000 (oldest deadline 2024-03-31T23:38:54.219616+0000) 2024-03-31T23:39:03.693 INFO:journalctl@ceph.osd.7.smithi142.stdout:Mar 31 23:39:03 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-7[53065]: 2024-03-31T23:39:03.387+0000 7fa41768e700 -1 osd.7 139 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-03-31T23:38:36.421116+0000 front 2024-03-31T23:38:36.420912+0000 (oldest deadline 2024-03-31T23:38:59.920889+0000) 2024-03-31T23:39:03.694 INFO:journalctl@ceph.osd.10.smithi142.stdout:Mar 31 23:39:03 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-10[60116]: 2024-03-31T23:39:03.379+0000 7f64c04a8700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-31T23:38:13.357804+0000 front 2024-03-31T23:38:13.357906+0000 (oldest deadline 2024-03-31T23:38:38.057730+0000) 2024-03-31T23:39:03.694 INFO:journalctl@ceph.osd.10.smithi142.stdout:Mar 31 23:39:03 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-10[60116]: 2024-03-31T23:39:03.379+0000 7f64c04a8700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.142:6804 osd.1 since back 2024-03-31T23:38:20.958341+0000 front 2024-03-31T23:38:20.958408+0000 (oldest deadline 2024-03-31T23:38:45.058481+0000) 2024-03-31T23:39:03.694 INFO:journalctl@ceph.osd.10.smithi142.stdout:Mar 31 23:39:03 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-10[60116]: 2024-03-31T23:39:03.379+0000 7f64c04a8700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-31T23:38:25.559393+0000 front 2024-03-31T23:38:25.559313+0000 (oldest deadline 2024-03-31T23:38:51.459398+0000) 2024-03-31T23:39:03.694 INFO:journalctl@ceph.osd.10.smithi142.stdout:Mar 31 23:39:03 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-10[60116]: 2024-03-31T23:39:03.379+0000 7f64c04a8700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-03-31T23:38:36.660831+0000 front 2024-03-31T23:38:36.660954+0000 (oldest deadline 2024-03-31T23:39:00.760714+0000) 2024-03-31T23:39:04.045 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:39:03 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:39:03.661+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-31T23:38:12.221366+0000 front 2024-03-31T23:38:12.221347+0000 (oldest deadline 2024-03-31T23:38:37.521357+0000) 2024-03-31T23:39:04.046 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:39:03 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:39:03.661+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.142:6804 osd.1 since back 2024-03-31T23:38:22.222192+0000 front 2024-03-31T23:38:22.222213+0000 (oldest deadline 2024-03-31T23:38:45.722340+0000) 2024-03-31T23:39:04.046 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:39:03 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:39:03.661+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-31T23:38:29.723616+0000 front 2024-03-31T23:38:29.723593+0000 (oldest deadline 2024-03-31T23:38:52.023779+0000) 2024-03-31T23:39:04.046 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:39:03 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:39:03.661+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-03-31T23:38:36.124750+0000 front 2024-03-31T23:38:36.124868+0000 (oldest deadline 2024-03-31T23:39:00.824619+0000) 2024-03-31T23:39:04.362 INFO:journalctl@ceph.osd.6.smithi046.stdout:Mar 31 23:39:04 smithi046 podman[107879]: 2024-03-31 23:39:04.138979225 +0000 UTC m=+5.751428774 container died f2dca9d6110aec61d1fc6d426c91338e2ff499d2be51ebf6451836fa01c2875d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-6, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=squid-384f31f, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240326, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5) 2024-03-31T23:39:04.362 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 31 23:39:04 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-9[67499]: 2024-03-31T23:39:04.005+0000 7fcc07615700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-31T23:38:10.469941+0000 front 2024-03-31T23:38:10.470388+0000 (oldest deadline 2024-03-31T23:38:35.170101+0000) 2024-03-31T23:39:04.362 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 31 23:39:04 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-9[67499]: 2024-03-31T23:39:04.005+0000 7fcc07615700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.142:6804 osd.1 since back 2024-03-31T23:38:21.571770+0000 front 2024-03-31T23:38:21.571801+0000 (oldest deadline 2024-03-31T23:38:43.871904+0000) 2024-03-31T23:39:04.362 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 31 23:39:04 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-9[67499]: 2024-03-31T23:39:04.005+0000 7fcc07615700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-31T23:38:27.074768+0000 front 2024-03-31T23:38:27.074793+0000 (oldest deadline 2024-03-31T23:38:52.374931+0000) 2024-03-31T23:39:04.362 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 31 23:39:04 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-9[67499]: 2024-03-31T23:39:04.005+0000 7fcc07615700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-03-31T23:38:34.075913+0000 front 2024-03-31T23:38:34.075868+0000 (oldest deadline 2024-03-31T23:38:59.376243+0000) 2024-03-31T23:39:04.545 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 31 23:39:04 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-8[55211]: 2024-03-31T23:39:04.191+0000 7fee85469700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-31T23:38:12.286566+0000 front 2024-03-31T23:38:12.286547+0000 (oldest deadline 2024-03-31T23:38:36.986433+0000) 2024-03-31T23:39:04.545 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 31 23:39:04 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-8[55211]: 2024-03-31T23:39:04.191+0000 7fee85469700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.142:6804 osd.1 since back 2024-03-31T23:38:21.587779+0000 front 2024-03-31T23:38:21.587835+0000 (oldest deadline 2024-03-31T23:38:47.487919+0000) 2024-03-31T23:39:04.546 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 31 23:39:04 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-8[55211]: 2024-03-31T23:39:04.191+0000 7fee85469700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-31T23:38:27.488310+0000 front 2024-03-31T23:38:27.488293+0000 (oldest deadline 2024-03-31T23:38:53.388420+0000) 2024-03-31T23:39:04.546 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 31 23:39:04 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-8[55211]: 2024-03-31T23:39:04.191+0000 7fee85469700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-03-31T23:38:33.389031+0000 front 2024-03-31T23:38:33.388915+0000 (oldest deadline 2024-03-31T23:38:59.288849+0000) 2024-03-31T23:39:04.693 INFO:journalctl@ceph.osd.7.smithi142.stdout:Mar 31 23:39:04 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-7[53065]: 2024-03-31T23:39:04.400+0000 7fa41768e700 -1 osd.7 139 heartbeat_check: no reply from 172.21.15.142:6804 osd.1 since back 2024-03-31T23:38:18.318617+0000 front 2024-03-31T23:38:18.318635+0000 (oldest deadline 2024-03-31T23:38:48.319131+0000) 2024-03-31T23:39:04.693 INFO:journalctl@ceph.osd.7.smithi142.stdout:Mar 31 23:39:04 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-7[53065]: 2024-03-31T23:39:04.400+0000 7fa41768e700 -1 osd.7 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-31T23:38:28.319471+0000 front 2024-03-31T23:38:28.319515+0000 (oldest deadline 2024-03-31T23:38:54.219616+0000) 2024-03-31T23:39:04.693 INFO:journalctl@ceph.osd.7.smithi142.stdout:Mar 31 23:39:04 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-7[53065]: 2024-03-31T23:39:04.400+0000 7fa41768e700 -1 osd.7 139 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-03-31T23:38:36.421116+0000 front 2024-03-31T23:38:36.420912+0000 (oldest deadline 2024-03-31T23:38:59.920889+0000) 2024-03-31T23:39:04.693 INFO:journalctl@ceph.osd.10.smithi142.stdout:Mar 31 23:39:04 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-10[60116]: 2024-03-31T23:39:04.390+0000 7f64c04a8700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-31T23:38:13.357804+0000 front 2024-03-31T23:38:13.357906+0000 (oldest deadline 2024-03-31T23:38:38.057730+0000) 2024-03-31T23:39:04.693 INFO:journalctl@ceph.osd.10.smithi142.stdout:Mar 31 23:39:04 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-10[60116]: 2024-03-31T23:39:04.390+0000 7f64c04a8700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.142:6804 osd.1 since back 2024-03-31T23:38:20.958341+0000 front 2024-03-31T23:38:20.958408+0000 (oldest deadline 2024-03-31T23:38:45.058481+0000) 2024-03-31T23:39:04.694 INFO:journalctl@ceph.osd.10.smithi142.stdout:Mar 31 23:39:04 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-10[60116]: 2024-03-31T23:39:04.390+0000 7f64c04a8700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-31T23:38:25.559393+0000 front 2024-03-31T23:38:25.559313+0000 (oldest deadline 2024-03-31T23:38:51.459398+0000) 2024-03-31T23:39:04.694 INFO:journalctl@ceph.osd.10.smithi142.stdout:Mar 31 23:39:04 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-10[60116]: 2024-03-31T23:39:04.390+0000 7f64c04a8700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-03-31T23:38:36.660831+0000 front 2024-03-31T23:38:36.660954+0000 (oldest deadline 2024-03-31T23:39:00.760714+0000) 2024-03-31T23:39:04.798 INFO:journalctl@ceph.osd.6.smithi046.stdout:Mar 31 23:39:04 smithi046 podman[107879]: 2024-03-31 23:39:04.52999946 +0000 UTC m=+6.142449007 container cleanup f2dca9d6110aec61d1fc6d426c91338e2ff499d2be51ebf6451836fa01c2875d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-6, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.build-date=20240326, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , RELEASE=squid-384f31f, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-31T23:39:04.798 INFO:journalctl@ceph.osd.6.smithi046.stdout:Mar 31 23:39:04 smithi046 bash[107879]: ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-6 2024-03-31T23:39:05.045 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:39:04 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:39:04.639+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-31T23:38:12.221366+0000 front 2024-03-31T23:38:12.221347+0000 (oldest deadline 2024-03-31T23:38:37.521357+0000) 2024-03-31T23:39:05.045 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:39:04 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:39:04.639+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.142:6804 osd.1 since back 2024-03-31T23:38:22.222192+0000 front 2024-03-31T23:38:22.222213+0000 (oldest deadline 2024-03-31T23:38:45.722340+0000) 2024-03-31T23:39:05.046 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:39:04 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:39:04.639+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-31T23:38:29.723616+0000 front 2024-03-31T23:38:29.723593+0000 (oldest deadline 2024-03-31T23:38:52.023779+0000) 2024-03-31T23:39:05.046 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:39:04 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:39:04.639+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-03-31T23:38:36.124750+0000 front 2024-03-31T23:38:36.124868+0000 (oldest deadline 2024-03-31T23:39:00.824619+0000) 2024-03-31T23:39:05.095 INFO:journalctl@ceph.osd.6.smithi046.stdout:Mar 31 23:39:04 smithi046 podman[107895]: 2024-03-31 23:39:04.798382007 +0000 UTC m=+0.656002128 container remove f2dca9d6110aec61d1fc6d426c91338e2ff499d2be51ebf6451836fa01c2875d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-6, org.label-schema.build-date=20240326, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-384f31f, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) 2024-03-31T23:39:05.096 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 31 23:39:04 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-9[67499]: 2024-03-31T23:39:04.961+0000 7fcc07615700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-31T23:38:10.469941+0000 front 2024-03-31T23:38:10.470388+0000 (oldest deadline 2024-03-31T23:38:35.170101+0000) 2024-03-31T23:39:05.096 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 31 23:39:04 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-9[67499]: 2024-03-31T23:39:04.961+0000 7fcc07615700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.142:6804 osd.1 since back 2024-03-31T23:38:21.571770+0000 front 2024-03-31T23:38:21.571801+0000 (oldest deadline 2024-03-31T23:38:43.871904+0000) 2024-03-31T23:39:05.096 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 31 23:39:04 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-9[67499]: 2024-03-31T23:39:04.961+0000 7fcc07615700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-31T23:38:27.074768+0000 front 2024-03-31T23:38:27.074793+0000 (oldest deadline 2024-03-31T23:38:52.374931+0000) 2024-03-31T23:39:05.096 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 31 23:39:04 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-9[67499]: 2024-03-31T23:39:04.961+0000 7fcc07615700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-03-31T23:38:34.075913+0000 front 2024-03-31T23:38:34.075868+0000 (oldest deadline 2024-03-31T23:38:59.376243+0000) 2024-03-31T23:39:05.361 INFO:journalctl@ceph.osd.6.smithi046.stdout:Mar 31 23:39:05 smithi046 podman[107989]: 2024-03-31 23:39:04.998309294 +0000 UTC m=+0.017459415 image pull ec18d7baa60e8b1a0b126fbe5306acce11b1ed0ec5e3bcdddbf39b0a3c591791 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175 2024-03-31T23:39:05.362 INFO:journalctl@ceph.osd.6.smithi046.stdout:Mar 31 23:39:05 smithi046 podman[107989]: 2024-03-31 23:39:05.141295916 +0000 UTC m=+0.160446023 container create 1f825c7c162597fb4048baf2bcb33e7f32e9846b64b033526b0dfb87ed91450f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-6-deactivate, org.label-schema.build-date=20240326, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, RELEASE=squid-384f31f, org.label-schema.schema-version=1.0, ceph=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.33.5) 2024-03-31T23:39:05.545 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 31 23:39:05 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-8[55211]: 2024-03-31T23:39:05.143+0000 7fee85469700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-31T23:38:12.286566+0000 front 2024-03-31T23:38:12.286547+0000 (oldest deadline 2024-03-31T23:38:36.986433+0000) 2024-03-31T23:39:05.546 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 31 23:39:05 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-8[55211]: 2024-03-31T23:39:05.143+0000 7fee85469700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.142:6804 osd.1 since back 2024-03-31T23:38:21.587779+0000 front 2024-03-31T23:38:21.587835+0000 (oldest deadline 2024-03-31T23:38:47.487919+0000) 2024-03-31T23:39:05.546 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 31 23:39:05 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-8[55211]: 2024-03-31T23:39:05.143+0000 7fee85469700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-31T23:38:27.488310+0000 front 2024-03-31T23:38:27.488293+0000 (oldest deadline 2024-03-31T23:38:53.388420+0000) 2024-03-31T23:39:05.546 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 31 23:39:05 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-8[55211]: 2024-03-31T23:39:05.143+0000 7fee85469700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-03-31T23:38:33.389031+0000 front 2024-03-31T23:38:33.388915+0000 (oldest deadline 2024-03-31T23:38:59.288849+0000) 2024-03-31T23:39:05.677 INFO:journalctl@ceph.osd.6.smithi046.stdout:Mar 31 23:39:05 smithi046 podman[107989]: 2024-03-31 23:39:05.377234473 +0000 UTC m=+0.396384598 container init 1f825c7c162597fb4048baf2bcb33e7f32e9846b64b033526b0dfb87ed91450f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-6-deactivate, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=squid-384f31f, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240326) 2024-03-31T23:39:05.678 INFO:journalctl@ceph.osd.6.smithi046.stdout:Mar 31 23:39:05 smithi046 podman[107989]: 2024-03-31 23:39:05.380805981 +0000 UTC m=+0.399956090 container start 1f825c7c162597fb4048baf2bcb33e7f32e9846b64b033526b0dfb87ed91450f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-6-deactivate, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.33.5, RELEASE=squid-384f31f, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240326, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-03-31T23:39:05.678 INFO:journalctl@ceph.osd.6.smithi046.stdout:Mar 31 23:39:05 smithi046 podman[107989]: 2024-03-31 23:39:05.451600309 +0000 UTC m=+0.470750427 container attach 1f825c7c162597fb4048baf2bcb33e7f32e9846b64b033526b0dfb87ed91450f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-6-deactivate, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-384f31f, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240326, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-03-31T23:39:05.693 INFO:journalctl@ceph.osd.10.smithi142.stdout:Mar 31 23:39:05 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-10[60116]: 2024-03-31T23:39:05.374+0000 7f64c04a8700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-31T23:38:13.357804+0000 front 2024-03-31T23:38:13.357906+0000 (oldest deadline 2024-03-31T23:38:38.057730+0000) 2024-03-31T23:39:05.693 INFO:journalctl@ceph.osd.10.smithi142.stdout:Mar 31 23:39:05 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-10[60116]: 2024-03-31T23:39:05.374+0000 7f64c04a8700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.142:6804 osd.1 since back 2024-03-31T23:38:20.958341+0000 front 2024-03-31T23:38:20.958408+0000 (oldest deadline 2024-03-31T23:38:45.058481+0000) 2024-03-31T23:39:05.693 INFO:journalctl@ceph.osd.10.smithi142.stdout:Mar 31 23:39:05 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-10[60116]: 2024-03-31T23:39:05.374+0000 7f64c04a8700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-31T23:38:25.559393+0000 front 2024-03-31T23:38:25.559313+0000 (oldest deadline 2024-03-31T23:38:51.459398+0000) 2024-03-31T23:39:05.693 INFO:journalctl@ceph.osd.10.smithi142.stdout:Mar 31 23:39:05 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-10[60116]: 2024-03-31T23:39:05.374+0000 7f64c04a8700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-03-31T23:38:36.660831+0000 front 2024-03-31T23:38:36.660954+0000 (oldest deadline 2024-03-31T23:39:00.760714+0000) 2024-03-31T23:39:05.693 INFO:journalctl@ceph.osd.7.smithi142.stdout:Mar 31 23:39:05 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-7[53065]: 2024-03-31T23:39:05.400+0000 7fa41768e700 -1 osd.7 139 heartbeat_check: no reply from 172.21.15.142:6804 osd.1 since back 2024-03-31T23:38:18.318617+0000 front 2024-03-31T23:38:18.318635+0000 (oldest deadline 2024-03-31T23:38:48.319131+0000) 2024-03-31T23:39:05.693 INFO:journalctl@ceph.osd.7.smithi142.stdout:Mar 31 23:39:05 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-7[53065]: 2024-03-31T23:39:05.400+0000 7fa41768e700 -1 osd.7 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-31T23:38:28.319471+0000 front 2024-03-31T23:38:28.319515+0000 (oldest deadline 2024-03-31T23:38:54.219616+0000) 2024-03-31T23:39:05.693 INFO:journalctl@ceph.osd.7.smithi142.stdout:Mar 31 23:39:05 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-7[53065]: 2024-03-31T23:39:05.400+0000 7fa41768e700 -1 osd.7 139 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-03-31T23:38:36.421116+0000 front 2024-03-31T23:38:36.420912+0000 (oldest deadline 2024-03-31T23:38:59.920889+0000) 2024-03-31T23:39:05.996 INFO:journalctl@ceph.osd.6.smithi046.stdout:Mar 31 23:39:05 smithi046 podman[107989]: 2024-03-31 23:39:05.677631875 +0000 UTC m=+0.696781991 container died 1f825c7c162597fb4048baf2bcb33e7f32e9846b64b033526b0dfb87ed91450f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-6-deactivate, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=squid-384f31f, ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_CLEAN=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240326) 2024-03-31T23:39:05.997 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 31 23:39:05 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-9[67499]: 2024-03-31T23:39:05.925+0000 7fcc07615700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-31T23:38:10.469941+0000 front 2024-03-31T23:38:10.470388+0000 (oldest deadline 2024-03-31T23:38:35.170101+0000) 2024-03-31T23:39:05.997 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 31 23:39:05 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-9[67499]: 2024-03-31T23:39:05.925+0000 7fcc07615700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.142:6804 osd.1 since back 2024-03-31T23:38:21.571770+0000 front 2024-03-31T23:38:21.571801+0000 (oldest deadline 2024-03-31T23:38:43.871904+0000) 2024-03-31T23:39:05.997 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 31 23:39:05 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-9[67499]: 2024-03-31T23:39:05.925+0000 7fcc07615700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-31T23:38:27.074768+0000 front 2024-03-31T23:38:27.074793+0000 (oldest deadline 2024-03-31T23:38:52.374931+0000) 2024-03-31T23:39:05.997 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 31 23:39:05 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-9[67499]: 2024-03-31T23:39:05.925+0000 7fcc07615700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-03-31T23:38:34.075913+0000 front 2024-03-31T23:38:34.075868+0000 (oldest deadline 2024-03-31T23:38:59.376243+0000) 2024-03-31T23:39:06.045 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:39:05 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:39:05.608+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-31T23:38:12.221366+0000 front 2024-03-31T23:38:12.221347+0000 (oldest deadline 2024-03-31T23:38:37.521357+0000) 2024-03-31T23:39:06.046 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:39:05 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:39:05.608+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.142:6804 osd.1 since back 2024-03-31T23:38:22.222192+0000 front 2024-03-31T23:38:22.222213+0000 (oldest deadline 2024-03-31T23:38:45.722340+0000) 2024-03-31T23:39:06.046 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:39:05 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:39:05.608+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-31T23:38:29.723616+0000 front 2024-03-31T23:38:29.723593+0000 (oldest deadline 2024-03-31T23:38:52.023779+0000) 2024-03-31T23:39:06.046 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:39:05 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:39:05.608+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-03-31T23:38:36.124750+0000 front 2024-03-31T23:38:36.124868+0000 (oldest deadline 2024-03-31T23:39:00.824619+0000) 2024-03-31T23:39:06.248 INFO:journalctl@ceph.osd.6.smithi046.stdout:Mar 31 23:39:06 smithi046 podman[108015]: 2024-03-31 23:39:06.230347625 +0000 UTC m=+0.547200747 container remove 1f825c7c162597fb4048baf2bcb33e7f32e9846b64b033526b0dfb87ed91450f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-6-deactivate, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240326, RELEASE=squid-384f31f, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) 2024-03-31T23:39:06.254 DEBUG:teuthology.orchestra.run.smithi046:> sudo pkill -f 'journalctl -f -n 0 -u ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4@osd.6.service' 2024-03-31T23:39:06.297 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-31T23:39:06.297 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-03-31T23:39:06.297 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-03-31T23:39:06.297 DEBUG:teuthology.orchestra.run.smithi142:> sudo systemctl stop ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4@osd.7 2024-03-31T23:39:06.442 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 31 23:39:06 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-8[55211]: 2024-03-31T23:39:06.097+0000 7fee85469700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-31T23:38:12.286566+0000 front 2024-03-31T23:38:12.286547+0000 (oldest deadline 2024-03-31T23:38:36.986433+0000) 2024-03-31T23:39:06.443 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 31 23:39:06 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-8[55211]: 2024-03-31T23:39:06.097+0000 7fee85469700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.142:6804 osd.1 since back 2024-03-31T23:38:21.587779+0000 front 2024-03-31T23:38:21.587835+0000 (oldest deadline 2024-03-31T23:38:47.487919+0000) 2024-03-31T23:39:06.443 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 31 23:39:06 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-8[55211]: 2024-03-31T23:39:06.097+0000 7fee85469700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-31T23:38:27.488310+0000 front 2024-03-31T23:38:27.488293+0000 (oldest deadline 2024-03-31T23:38:53.388420+0000) 2024-03-31T23:39:06.443 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 31 23:39:06 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-8[55211]: 2024-03-31T23:39:06.097+0000 7fee85469700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-03-31T23:38:33.389031+0000 front 2024-03-31T23:38:33.388915+0000 (oldest deadline 2024-03-31T23:38:59.288849+0000) 2024-03-31T23:39:06.642 INFO:journalctl@ceph.osd.10.smithi142.stdout:Mar 31 23:39:06 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-10[60116]: 2024-03-31T23:39:06.360+0000 7f64c04a8700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-31T23:38:13.357804+0000 front 2024-03-31T23:38:13.357906+0000 (oldest deadline 2024-03-31T23:38:38.057730+0000) 2024-03-31T23:39:06.642 INFO:journalctl@ceph.osd.10.smithi142.stdout:Mar 31 23:39:06 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-10[60116]: 2024-03-31T23:39:06.360+0000 7f64c04a8700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.142:6804 osd.1 since back 2024-03-31T23:38:20.958341+0000 front 2024-03-31T23:38:20.958408+0000 (oldest deadline 2024-03-31T23:38:45.058481+0000) 2024-03-31T23:39:06.642 INFO:journalctl@ceph.osd.10.smithi142.stdout:Mar 31 23:39:06 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-10[60116]: 2024-03-31T23:39:06.360+0000 7f64c04a8700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-31T23:38:25.559393+0000 front 2024-03-31T23:38:25.559313+0000 (oldest deadline 2024-03-31T23:38:51.459398+0000) 2024-03-31T23:39:06.642 INFO:journalctl@ceph.osd.10.smithi142.stdout:Mar 31 23:39:06 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-10[60116]: 2024-03-31T23:39:06.360+0000 7f64c04a8700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-03-31T23:38:36.660831+0000 front 2024-03-31T23:38:36.660954+0000 (oldest deadline 2024-03-31T23:39:00.760714+0000) 2024-03-31T23:39:06.642 INFO:journalctl@ceph.osd.7.smithi142.stdout:Mar 31 23:39:06 smithi142 systemd[1]: Stopping Ceph osd.7 for 6a9d8f08-efb2-11ee-b647-cb9ed24678a4... 2024-03-31T23:39:06.642 INFO:journalctl@ceph.osd.7.smithi142.stdout:Mar 31 23:39:06 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-7[53065]: 2024-03-31T23:39:06.435+0000 7fa41768e700 -1 osd.7 139 heartbeat_check: no reply from 172.21.15.142:6804 osd.1 since back 2024-03-31T23:38:18.318617+0000 front 2024-03-31T23:38:18.318635+0000 (oldest deadline 2024-03-31T23:38:48.319131+0000) 2024-03-31T23:39:06.642 INFO:journalctl@ceph.osd.7.smithi142.stdout:Mar 31 23:39:06 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-7[53065]: 2024-03-31T23:39:06.435+0000 7fa41768e700 -1 osd.7 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-31T23:38:28.319471+0000 front 2024-03-31T23:38:28.319515+0000 (oldest deadline 2024-03-31T23:38:54.219616+0000) 2024-03-31T23:39:06.642 INFO:journalctl@ceph.osd.7.smithi142.stdout:Mar 31 23:39:06 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-7[53065]: 2024-03-31T23:39:06.435+0000 7fa41768e700 -1 osd.7 139 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-03-31T23:38:36.421116+0000 front 2024-03-31T23:38:36.420912+0000 (oldest deadline 2024-03-31T23:38:59.920889+0000) 2024-03-31T23:39:06.643 INFO:journalctl@ceph.osd.7.smithi142.stdout:Mar 31 23:39:06 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-7[53065]: 2024-03-31T23:39:06.641+0000 7fa41b09a700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-31T23:39:06.643 INFO:journalctl@ceph.osd.7.smithi142.stdout:Mar 31 23:39:06 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-7[53065]: 2024-03-31T23:39:06.641+0000 7fa41b09a700 -1 osd.7 139 *** Got signal Terminated *** 2024-03-31T23:39:06.643 INFO:journalctl@ceph.osd.7.smithi142.stdout:Mar 31 23:39:06 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-7[53065]: 2024-03-31T23:39:06.641+0000 7fa41b09a700 -1 osd.7 139 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-31T23:39:06.795 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:39:06 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:39:06.602+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-31T23:38:12.221366+0000 front 2024-03-31T23:38:12.221347+0000 (oldest deadline 2024-03-31T23:38:37.521357+0000) 2024-03-31T23:39:06.795 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:39:06 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:39:06.602+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.142:6804 osd.1 since back 2024-03-31T23:38:22.222192+0000 front 2024-03-31T23:38:22.222213+0000 (oldest deadline 2024-03-31T23:38:45.722340+0000) 2024-03-31T23:39:06.796 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:39:06 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:39:06.602+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-31T23:38:29.723616+0000 front 2024-03-31T23:38:29.723593+0000 (oldest deadline 2024-03-31T23:38:52.023779+0000) 2024-03-31T23:39:06.796 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:39:06 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:39:06.602+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-03-31T23:38:36.124750+0000 front 2024-03-31T23:38:36.124868+0000 (oldest deadline 2024-03-31T23:39:00.824619+0000) 2024-03-31T23:39:07.361 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 31 23:39:06 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-9[67499]: 2024-03-31T23:39:06.965+0000 7fcc07615700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-31T23:38:10.469941+0000 front 2024-03-31T23:38:10.470388+0000 (oldest deadline 2024-03-31T23:38:35.170101+0000) 2024-03-31T23:39:07.361 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 31 23:39:06 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-9[67499]: 2024-03-31T23:39:06.965+0000 7fcc07615700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.142:6804 osd.1 since back 2024-03-31T23:38:21.571770+0000 front 2024-03-31T23:38:21.571801+0000 (oldest deadline 2024-03-31T23:38:43.871904+0000) 2024-03-31T23:39:07.362 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 31 23:39:06 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-9[67499]: 2024-03-31T23:39:06.965+0000 7fcc07615700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-31T23:38:27.074768+0000 front 2024-03-31T23:38:27.074793+0000 (oldest deadline 2024-03-31T23:38:52.374931+0000) 2024-03-31T23:39:07.362 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 31 23:39:06 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-9[67499]: 2024-03-31T23:39:06.965+0000 7fcc07615700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-03-31T23:38:34.075913+0000 front 2024-03-31T23:38:34.075868+0000 (oldest deadline 2024-03-31T23:38:59.376243+0000) 2024-03-31T23:39:07.545 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 31 23:39:07 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-8[55211]: 2024-03-31T23:39:07.084+0000 7fee85469700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-31T23:38:12.286566+0000 front 2024-03-31T23:38:12.286547+0000 (oldest deadline 2024-03-31T23:38:36.986433+0000) 2024-03-31T23:39:07.545 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 31 23:39:07 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-8[55211]: 2024-03-31T23:39:07.084+0000 7fee85469700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.142:6804 osd.1 since back 2024-03-31T23:38:21.587779+0000 front 2024-03-31T23:38:21.587835+0000 (oldest deadline 2024-03-31T23:38:47.487919+0000) 2024-03-31T23:39:07.546 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 31 23:39:07 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-8[55211]: 2024-03-31T23:39:07.084+0000 7fee85469700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-31T23:38:27.488310+0000 front 2024-03-31T23:38:27.488293+0000 (oldest deadline 2024-03-31T23:38:53.388420+0000) 2024-03-31T23:39:07.546 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 31 23:39:07 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-8[55211]: 2024-03-31T23:39:07.084+0000 7fee85469700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-03-31T23:38:33.389031+0000 front 2024-03-31T23:38:33.388915+0000 (oldest deadline 2024-03-31T23:38:59.288849+0000) 2024-03-31T23:39:07.693 INFO:journalctl@ceph.osd.10.smithi142.stdout:Mar 31 23:39:07 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-10[60116]: 2024-03-31T23:39:07.398+0000 7f64c04a8700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-31T23:38:13.357804+0000 front 2024-03-31T23:38:13.357906+0000 (oldest deadline 2024-03-31T23:38:38.057730+0000) 2024-03-31T23:39:07.693 INFO:journalctl@ceph.osd.10.smithi142.stdout:Mar 31 23:39:07 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-10[60116]: 2024-03-31T23:39:07.398+0000 7f64c04a8700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.142:6804 osd.1 since back 2024-03-31T23:38:20.958341+0000 front 2024-03-31T23:38:20.958408+0000 (oldest deadline 2024-03-31T23:38:45.058481+0000) 2024-03-31T23:39:07.693 INFO:journalctl@ceph.osd.10.smithi142.stdout:Mar 31 23:39:07 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-10[60116]: 2024-03-31T23:39:07.398+0000 7f64c04a8700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-31T23:38:25.559393+0000 front 2024-03-31T23:38:25.559313+0000 (oldest deadline 2024-03-31T23:38:51.459398+0000) 2024-03-31T23:39:07.693 INFO:journalctl@ceph.osd.10.smithi142.stdout:Mar 31 23:39:07 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-10[60116]: 2024-03-31T23:39:07.398+0000 7f64c04a8700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-03-31T23:38:36.660831+0000 front 2024-03-31T23:38:36.660954+0000 (oldest deadline 2024-03-31T23:39:00.760714+0000) 2024-03-31T23:39:07.693 INFO:journalctl@ceph.osd.7.smithi142.stdout:Mar 31 23:39:07 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-7[53065]: 2024-03-31T23:39:07.437+0000 7fa41768e700 -1 osd.7 139 heartbeat_check: no reply from 172.21.15.142:6804 osd.1 since back 2024-03-31T23:38:18.318617+0000 front 2024-03-31T23:38:18.318635+0000 (oldest deadline 2024-03-31T23:38:48.319131+0000) 2024-03-31T23:39:07.694 INFO:journalctl@ceph.osd.7.smithi142.stdout:Mar 31 23:39:07 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-7[53065]: 2024-03-31T23:39:07.437+0000 7fa41768e700 -1 osd.7 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-31T23:38:28.319471+0000 front 2024-03-31T23:38:28.319515+0000 (oldest deadline 2024-03-31T23:38:54.219616+0000) 2024-03-31T23:39:07.694 INFO:journalctl@ceph.osd.7.smithi142.stdout:Mar 31 23:39:07 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-7[53065]: 2024-03-31T23:39:07.437+0000 7fa41768e700 -1 osd.7 139 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-03-31T23:38:36.421116+0000 front 2024-03-31T23:38:36.420912+0000 (oldest deadline 2024-03-31T23:38:59.920889+0000) 2024-03-31T23:39:08.045 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:39:07 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:39:07.633+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-31T23:38:12.221366+0000 front 2024-03-31T23:38:12.221347+0000 (oldest deadline 2024-03-31T23:38:37.521357+0000) 2024-03-31T23:39:08.046 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:39:07 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:39:07.633+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.142:6804 osd.1 since back 2024-03-31T23:38:22.222192+0000 front 2024-03-31T23:38:22.222213+0000 (oldest deadline 2024-03-31T23:38:45.722340+0000) 2024-03-31T23:39:08.046 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:39:07 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:39:07.633+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-31T23:38:29.723616+0000 front 2024-03-31T23:38:29.723593+0000 (oldest deadline 2024-03-31T23:38:52.023779+0000) 2024-03-31T23:39:08.046 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:39:07 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:39:07.633+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-03-31T23:38:36.124750+0000 front 2024-03-31T23:38:36.124868+0000 (oldest deadline 2024-03-31T23:39:00.824619+0000) 2024-03-31T23:39:08.361 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 31 23:39:07 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-9[67499]: 2024-03-31T23:39:07.950+0000 7fcc07615700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-31T23:38:10.469941+0000 front 2024-03-31T23:38:10.470388+0000 (oldest deadline 2024-03-31T23:38:35.170101+0000) 2024-03-31T23:39:08.362 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 31 23:39:07 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-9[67499]: 2024-03-31T23:39:07.950+0000 7fcc07615700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.142:6804 osd.1 since back 2024-03-31T23:38:21.571770+0000 front 2024-03-31T23:38:21.571801+0000 (oldest deadline 2024-03-31T23:38:43.871904+0000) 2024-03-31T23:39:08.362 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 31 23:39:07 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-9[67499]: 2024-03-31T23:39:07.950+0000 7fcc07615700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-31T23:38:27.074768+0000 front 2024-03-31T23:38:27.074793+0000 (oldest deadline 2024-03-31T23:38:52.374931+0000) 2024-03-31T23:39:08.362 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 31 23:39:07 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-9[67499]: 2024-03-31T23:39:07.950+0000 7fcc07615700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-03-31T23:38:34.075913+0000 front 2024-03-31T23:38:34.075868+0000 (oldest deadline 2024-03-31T23:38:59.376243+0000) 2024-03-31T23:39:08.545 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 31 23:39:08 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-8[55211]: 2024-03-31T23:39:08.091+0000 7fee85469700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-31T23:38:12.286566+0000 front 2024-03-31T23:38:12.286547+0000 (oldest deadline 2024-03-31T23:38:36.986433+0000) 2024-03-31T23:39:08.545 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 31 23:39:08 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-8[55211]: 2024-03-31T23:39:08.091+0000 7fee85469700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.142:6804 osd.1 since back 2024-03-31T23:38:21.587779+0000 front 2024-03-31T23:38:21.587835+0000 (oldest deadline 2024-03-31T23:38:47.487919+0000) 2024-03-31T23:39:08.546 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 31 23:39:08 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-8[55211]: 2024-03-31T23:39:08.091+0000 7fee85469700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-31T23:38:27.488310+0000 front 2024-03-31T23:38:27.488293+0000 (oldest deadline 2024-03-31T23:38:53.388420+0000) 2024-03-31T23:39:08.546 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 31 23:39:08 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-8[55211]: 2024-03-31T23:39:08.091+0000 7fee85469700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-03-31T23:38:33.389031+0000 front 2024-03-31T23:38:33.388915+0000 (oldest deadline 2024-03-31T23:38:59.288849+0000) 2024-03-31T23:39:08.546 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 31 23:39:08 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-8[55211]: 2024-03-31T23:39:08.091+0000 7fee85469700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.142:6812 osd.4 since back 2024-03-31T23:38:46.290078+0000 front 2024-03-31T23:38:46.290091+0000 (oldest deadline 2024-03-31T23:39:07.990166+0000) 2024-03-31T23:39:08.693 INFO:journalctl@ceph.osd.7.smithi142.stdout:Mar 31 23:39:08 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-7[53065]: 2024-03-31T23:39:08.448+0000 7fa41768e700 -1 osd.7 139 heartbeat_check: no reply from 172.21.15.142:6804 osd.1 since back 2024-03-31T23:38:18.318617+0000 front 2024-03-31T23:38:18.318635+0000 (oldest deadline 2024-03-31T23:38:48.319131+0000) 2024-03-31T23:39:08.693 INFO:journalctl@ceph.osd.7.smithi142.stdout:Mar 31 23:39:08 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-7[53065]: 2024-03-31T23:39:08.448+0000 7fa41768e700 -1 osd.7 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-31T23:38:28.319471+0000 front 2024-03-31T23:38:28.319515+0000 (oldest deadline 2024-03-31T23:38:54.219616+0000) 2024-03-31T23:39:08.693 INFO:journalctl@ceph.osd.7.smithi142.stdout:Mar 31 23:39:08 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-7[53065]: 2024-03-31T23:39:08.449+0000 7fa41768e700 -1 osd.7 139 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-03-31T23:38:36.421116+0000 front 2024-03-31T23:38:36.420912+0000 (oldest deadline 2024-03-31T23:38:59.920889+0000) 2024-03-31T23:39:08.693 INFO:journalctl@ceph.osd.7.smithi142.stdout:Mar 31 23:39:08 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-7[53065]: 2024-03-31T23:39:08.449+0000 7fa41768e700 -1 osd.7 139 heartbeat_check: no reply from 172.21.15.142:6812 osd.4 since back 2024-03-31T23:38:42.821718+0000 front 2024-03-31T23:38:42.821743+0000 (oldest deadline 2024-03-31T23:39:08.121808+0000) 2024-03-31T23:39:08.694 INFO:journalctl@ceph.osd.10.smithi142.stdout:Mar 31 23:39:08 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-10[60116]: 2024-03-31T23:39:08.379+0000 7f64c04a8700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-31T23:38:13.357804+0000 front 2024-03-31T23:38:13.357906+0000 (oldest deadline 2024-03-31T23:38:38.057730+0000) 2024-03-31T23:39:08.694 INFO:journalctl@ceph.osd.10.smithi142.stdout:Mar 31 23:39:08 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-10[60116]: 2024-03-31T23:39:08.379+0000 7f64c04a8700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.142:6804 osd.1 since back 2024-03-31T23:38:20.958341+0000 front 2024-03-31T23:38:20.958408+0000 (oldest deadline 2024-03-31T23:38:45.058481+0000) 2024-03-31T23:39:08.694 INFO:journalctl@ceph.osd.10.smithi142.stdout:Mar 31 23:39:08 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-10[60116]: 2024-03-31T23:39:08.379+0000 7f64c04a8700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-31T23:38:25.559393+0000 front 2024-03-31T23:38:25.559313+0000 (oldest deadline 2024-03-31T23:38:51.459398+0000) 2024-03-31T23:39:08.694 INFO:journalctl@ceph.osd.10.smithi142.stdout:Mar 31 23:39:08 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-10[60116]: 2024-03-31T23:39:08.379+0000 7f64c04a8700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-03-31T23:38:36.660831+0000 front 2024-03-31T23:38:36.660954+0000 (oldest deadline 2024-03-31T23:39:00.760714+0000) 2024-03-31T23:39:08.694 INFO:journalctl@ceph.osd.10.smithi142.stdout:Mar 31 23:39:08 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-10[60116]: 2024-03-31T23:39:08.379+0000 7f64c04a8700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.142:6812 osd.4 since back 2024-03-31T23:38:42.361714+0000 front 2024-03-31T23:38:42.361749+0000 (oldest deadline 2024-03-31T23:39:08.261872+0000) 2024-03-31T23:39:09.045 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:39:08 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:39:08.678+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-31T23:38:12.221366+0000 front 2024-03-31T23:38:12.221347+0000 (oldest deadline 2024-03-31T23:38:37.521357+0000) 2024-03-31T23:39:09.046 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:39:08 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:39:08.678+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.142:6804 osd.1 since back 2024-03-31T23:38:22.222192+0000 front 2024-03-31T23:38:22.222213+0000 (oldest deadline 2024-03-31T23:38:45.722340+0000) 2024-03-31T23:39:09.046 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:39:08 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:39:08.678+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-31T23:38:29.723616+0000 front 2024-03-31T23:38:29.723593+0000 (oldest deadline 2024-03-31T23:38:52.023779+0000) 2024-03-31T23:39:09.046 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:39:08 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:39:08.678+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-03-31T23:38:36.124750+0000 front 2024-03-31T23:38:36.124868+0000 (oldest deadline 2024-03-31T23:39:00.824619+0000) 2024-03-31T23:39:09.361 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 31 23:39:08 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-9[67499]: 2024-03-31T23:39:08.987+0000 7fcc07615700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-31T23:38:10.469941+0000 front 2024-03-31T23:38:10.470388+0000 (oldest deadline 2024-03-31T23:38:35.170101+0000) 2024-03-31T23:39:09.361 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 31 23:39:08 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-9[67499]: 2024-03-31T23:39:08.987+0000 7fcc07615700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.142:6804 osd.1 since back 2024-03-31T23:38:21.571770+0000 front 2024-03-31T23:38:21.571801+0000 (oldest deadline 2024-03-31T23:38:43.871904+0000) 2024-03-31T23:39:09.362 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 31 23:39:08 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-9[67499]: 2024-03-31T23:39:08.987+0000 7fcc07615700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-31T23:38:27.074768+0000 front 2024-03-31T23:38:27.074793+0000 (oldest deadline 2024-03-31T23:38:52.374931+0000) 2024-03-31T23:39:09.362 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 31 23:39:08 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-9[67499]: 2024-03-31T23:39:08.987+0000 7fcc07615700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-03-31T23:38:34.075913+0000 front 2024-03-31T23:38:34.075868+0000 (oldest deadline 2024-03-31T23:38:59.376243+0000) 2024-03-31T23:39:09.545 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 31 23:39:09 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-8[55211]: 2024-03-31T23:39:09.128+0000 7fee85469700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-31T23:38:12.286566+0000 front 2024-03-31T23:38:12.286547+0000 (oldest deadline 2024-03-31T23:38:36.986433+0000) 2024-03-31T23:39:09.545 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 31 23:39:09 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-8[55211]: 2024-03-31T23:39:09.128+0000 7fee85469700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.142:6804 osd.1 since back 2024-03-31T23:38:21.587779+0000 front 2024-03-31T23:38:21.587835+0000 (oldest deadline 2024-03-31T23:38:47.487919+0000) 2024-03-31T23:39:09.546 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 31 23:39:09 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-8[55211]: 2024-03-31T23:39:09.128+0000 7fee85469700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-31T23:38:27.488310+0000 front 2024-03-31T23:38:27.488293+0000 (oldest deadline 2024-03-31T23:38:53.388420+0000) 2024-03-31T23:39:09.546 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 31 23:39:09 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-8[55211]: 2024-03-31T23:39:09.128+0000 7fee85469700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-03-31T23:38:33.389031+0000 front 2024-03-31T23:38:33.388915+0000 (oldest deadline 2024-03-31T23:38:59.288849+0000) 2024-03-31T23:39:09.546 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 31 23:39:09 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-8[55211]: 2024-03-31T23:39:09.128+0000 7fee85469700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.142:6812 osd.4 since back 2024-03-31T23:38:46.290078+0000 front 2024-03-31T23:38:46.290091+0000 (oldest deadline 2024-03-31T23:39:07.990166+0000) 2024-03-31T23:39:09.693 INFO:journalctl@ceph.osd.10.smithi142.stdout:Mar 31 23:39:09 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-10[60116]: 2024-03-31T23:39:09.390+0000 7f64c04a8700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-31T23:38:13.357804+0000 front 2024-03-31T23:38:13.357906+0000 (oldest deadline 2024-03-31T23:38:38.057730+0000) 2024-03-31T23:39:09.693 INFO:journalctl@ceph.osd.10.smithi142.stdout:Mar 31 23:39:09 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-10[60116]: 2024-03-31T23:39:09.390+0000 7f64c04a8700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.142:6804 osd.1 since back 2024-03-31T23:38:20.958341+0000 front 2024-03-31T23:38:20.958408+0000 (oldest deadline 2024-03-31T23:38:45.058481+0000) 2024-03-31T23:39:09.693 INFO:journalctl@ceph.osd.10.smithi142.stdout:Mar 31 23:39:09 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-10[60116]: 2024-03-31T23:39:09.390+0000 7f64c04a8700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-31T23:38:25.559393+0000 front 2024-03-31T23:38:25.559313+0000 (oldest deadline 2024-03-31T23:38:51.459398+0000) 2024-03-31T23:39:09.693 INFO:journalctl@ceph.osd.10.smithi142.stdout:Mar 31 23:39:09 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-10[60116]: 2024-03-31T23:39:09.390+0000 7f64c04a8700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-03-31T23:38:36.660831+0000 front 2024-03-31T23:38:36.660954+0000 (oldest deadline 2024-03-31T23:39:00.760714+0000) 2024-03-31T23:39:09.693 INFO:journalctl@ceph.osd.10.smithi142.stdout:Mar 31 23:39:09 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-10[60116]: 2024-03-31T23:39:09.390+0000 7f64c04a8700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.142:6812 osd.4 since back 2024-03-31T23:38:42.361714+0000 front 2024-03-31T23:38:42.361749+0000 (oldest deadline 2024-03-31T23:39:08.261872+0000) 2024-03-31T23:39:09.694 INFO:journalctl@ceph.osd.7.smithi142.stdout:Mar 31 23:39:09 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-7[53065]: 2024-03-31T23:39:09.488+0000 7fa41768e700 -1 osd.7 139 heartbeat_check: no reply from 172.21.15.142:6804 osd.1 since back 2024-03-31T23:38:18.318617+0000 front 2024-03-31T23:38:18.318635+0000 (oldest deadline 2024-03-31T23:38:48.319131+0000) 2024-03-31T23:39:09.694 INFO:journalctl@ceph.osd.7.smithi142.stdout:Mar 31 23:39:09 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-7[53065]: 2024-03-31T23:39:09.488+0000 7fa41768e700 -1 osd.7 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-31T23:38:28.319471+0000 front 2024-03-31T23:38:28.319515+0000 (oldest deadline 2024-03-31T23:38:54.219616+0000) 2024-03-31T23:39:09.694 INFO:journalctl@ceph.osd.7.smithi142.stdout:Mar 31 23:39:09 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-7[53065]: 2024-03-31T23:39:09.488+0000 7fa41768e700 -1 osd.7 139 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-03-31T23:38:36.421116+0000 front 2024-03-31T23:38:36.420912+0000 (oldest deadline 2024-03-31T23:38:59.920889+0000) 2024-03-31T23:39:09.694 INFO:journalctl@ceph.osd.7.smithi142.stdout:Mar 31 23:39:09 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-7[53065]: 2024-03-31T23:39:09.488+0000 7fa41768e700 -1 osd.7 139 heartbeat_check: no reply from 172.21.15.142:6812 osd.4 since back 2024-03-31T23:38:42.821718+0000 front 2024-03-31T23:38:42.821743+0000 (oldest deadline 2024-03-31T23:39:08.121808+0000) 2024-03-31T23:39:10.045 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:39:09 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:39:09.641+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-31T23:38:12.221366+0000 front 2024-03-31T23:38:12.221347+0000 (oldest deadline 2024-03-31T23:38:37.521357+0000) 2024-03-31T23:39:10.045 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:39:09 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:39:09.641+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.142:6804 osd.1 since back 2024-03-31T23:38:22.222192+0000 front 2024-03-31T23:38:22.222213+0000 (oldest deadline 2024-03-31T23:38:45.722340+0000) 2024-03-31T23:39:10.045 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:39:09 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:39:09.641+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-31T23:38:29.723616+0000 front 2024-03-31T23:38:29.723593+0000 (oldest deadline 2024-03-31T23:38:52.023779+0000) 2024-03-31T23:39:10.046 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:39:09 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:39:09.641+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-03-31T23:38:36.124750+0000 front 2024-03-31T23:38:36.124868+0000 (oldest deadline 2024-03-31T23:39:00.824619+0000) 2024-03-31T23:39:10.361 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 31 23:39:09 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-9[67499]: 2024-03-31T23:39:09.986+0000 7fcc07615700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-31T23:38:10.469941+0000 front 2024-03-31T23:38:10.470388+0000 (oldest deadline 2024-03-31T23:38:35.170101+0000) 2024-03-31T23:39:10.361 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 31 23:39:09 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-9[67499]: 2024-03-31T23:39:09.986+0000 7fcc07615700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.142:6804 osd.1 since back 2024-03-31T23:38:21.571770+0000 front 2024-03-31T23:38:21.571801+0000 (oldest deadline 2024-03-31T23:38:43.871904+0000) 2024-03-31T23:39:10.362 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 31 23:39:09 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-9[67499]: 2024-03-31T23:39:09.986+0000 7fcc07615700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-31T23:38:27.074768+0000 front 2024-03-31T23:38:27.074793+0000 (oldest deadline 2024-03-31T23:38:52.374931+0000) 2024-03-31T23:39:10.362 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 31 23:39:09 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-9[67499]: 2024-03-31T23:39:09.986+0000 7fcc07615700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-03-31T23:38:34.075913+0000 front 2024-03-31T23:38:34.075868+0000 (oldest deadline 2024-03-31T23:38:59.376243+0000) 2024-03-31T23:39:10.545 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 31 23:39:10 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-8[55211]: 2024-03-31T23:39:10.098+0000 7fee85469700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-31T23:38:12.286566+0000 front 2024-03-31T23:38:12.286547+0000 (oldest deadline 2024-03-31T23:38:36.986433+0000) 2024-03-31T23:39:10.546 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 31 23:39:10 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-8[55211]: 2024-03-31T23:39:10.098+0000 7fee85469700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.142:6804 osd.1 since back 2024-03-31T23:38:21.587779+0000 front 2024-03-31T23:38:21.587835+0000 (oldest deadline 2024-03-31T23:38:47.487919+0000) 2024-03-31T23:39:10.546 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 31 23:39:10 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-8[55211]: 2024-03-31T23:39:10.098+0000 7fee85469700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-31T23:38:27.488310+0000 front 2024-03-31T23:38:27.488293+0000 (oldest deadline 2024-03-31T23:38:53.388420+0000) 2024-03-31T23:39:10.546 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 31 23:39:10 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-8[55211]: 2024-03-31T23:39:10.098+0000 7fee85469700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-03-31T23:38:33.389031+0000 front 2024-03-31T23:38:33.388915+0000 (oldest deadline 2024-03-31T23:38:59.288849+0000) 2024-03-31T23:39:10.546 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 31 23:39:10 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-8[55211]: 2024-03-31T23:39:10.098+0000 7fee85469700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.142:6812 osd.4 since back 2024-03-31T23:38:46.290078+0000 front 2024-03-31T23:38:46.290091+0000 (oldest deadline 2024-03-31T23:39:07.990166+0000) 2024-03-31T23:39:10.693 INFO:journalctl@ceph.osd.10.smithi142.stdout:Mar 31 23:39:10 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-10[60116]: 2024-03-31T23:39:10.424+0000 7f64c04a8700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-31T23:38:13.357804+0000 front 2024-03-31T23:38:13.357906+0000 (oldest deadline 2024-03-31T23:38:38.057730+0000) 2024-03-31T23:39:10.693 INFO:journalctl@ceph.osd.10.smithi142.stdout:Mar 31 23:39:10 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-10[60116]: 2024-03-31T23:39:10.424+0000 7f64c04a8700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.142:6804 osd.1 since back 2024-03-31T23:38:20.958341+0000 front 2024-03-31T23:38:20.958408+0000 (oldest deadline 2024-03-31T23:38:45.058481+0000) 2024-03-31T23:39:10.693 INFO:journalctl@ceph.osd.10.smithi142.stdout:Mar 31 23:39:10 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-10[60116]: 2024-03-31T23:39:10.424+0000 7f64c04a8700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-31T23:38:25.559393+0000 front 2024-03-31T23:38:25.559313+0000 (oldest deadline 2024-03-31T23:38:51.459398+0000) 2024-03-31T23:39:10.693 INFO:journalctl@ceph.osd.10.smithi142.stdout:Mar 31 23:39:10 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-10[60116]: 2024-03-31T23:39:10.424+0000 7f64c04a8700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-03-31T23:38:36.660831+0000 front 2024-03-31T23:38:36.660954+0000 (oldest deadline 2024-03-31T23:39:00.760714+0000) 2024-03-31T23:39:10.693 INFO:journalctl@ceph.osd.10.smithi142.stdout:Mar 31 23:39:10 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-10[60116]: 2024-03-31T23:39:10.424+0000 7f64c04a8700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.142:6812 osd.4 since back 2024-03-31T23:38:42.361714+0000 front 2024-03-31T23:38:42.361749+0000 (oldest deadline 2024-03-31T23:39:08.261872+0000) 2024-03-31T23:39:10.694 INFO:journalctl@ceph.osd.7.smithi142.stdout:Mar 31 23:39:10 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-7[53065]: 2024-03-31T23:39:10.503+0000 7fa41768e700 -1 osd.7 139 heartbeat_check: no reply from 172.21.15.142:6804 osd.1 since back 2024-03-31T23:38:18.318617+0000 front 2024-03-31T23:38:18.318635+0000 (oldest deadline 2024-03-31T23:38:48.319131+0000) 2024-03-31T23:39:10.694 INFO:journalctl@ceph.osd.7.smithi142.stdout:Mar 31 23:39:10 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-7[53065]: 2024-03-31T23:39:10.503+0000 7fa41768e700 -1 osd.7 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-31T23:38:28.319471+0000 front 2024-03-31T23:38:28.319515+0000 (oldest deadline 2024-03-31T23:38:54.219616+0000) 2024-03-31T23:39:10.694 INFO:journalctl@ceph.osd.7.smithi142.stdout:Mar 31 23:39:10 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-7[53065]: 2024-03-31T23:39:10.503+0000 7fa41768e700 -1 osd.7 139 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-03-31T23:38:36.421116+0000 front 2024-03-31T23:38:36.420912+0000 (oldest deadline 2024-03-31T23:38:59.920889+0000) 2024-03-31T23:39:10.694 INFO:journalctl@ceph.osd.7.smithi142.stdout:Mar 31 23:39:10 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-7[53065]: 2024-03-31T23:39:10.503+0000 7fa41768e700 -1 osd.7 139 heartbeat_check: no reply from 172.21.15.142:6812 osd.4 since back 2024-03-31T23:38:42.821718+0000 front 2024-03-31T23:38:42.821743+0000 (oldest deadline 2024-03-31T23:39:08.121808+0000) 2024-03-31T23:39:11.045 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:39:10 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:39:10.600+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-31T23:38:12.221366+0000 front 2024-03-31T23:38:12.221347+0000 (oldest deadline 2024-03-31T23:38:37.521357+0000) 2024-03-31T23:39:11.045 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:39:10 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:39:10.600+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.142:6804 osd.1 since back 2024-03-31T23:38:22.222192+0000 front 2024-03-31T23:38:22.222213+0000 (oldest deadline 2024-03-31T23:38:45.722340+0000) 2024-03-31T23:39:11.045 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:39:10 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:39:10.600+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-31T23:38:29.723616+0000 front 2024-03-31T23:38:29.723593+0000 (oldest deadline 2024-03-31T23:38:52.023779+0000) 2024-03-31T23:39:11.045 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:39:10 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:39:10.600+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-03-31T23:38:36.124750+0000 front 2024-03-31T23:38:36.124868+0000 (oldest deadline 2024-03-31T23:39:00.824619+0000) 2024-03-31T23:39:11.046 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:39:10 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:39:10.600+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.142:6812 osd.4 since back 2024-03-31T23:38:44.825661+0000 front 2024-03-31T23:38:44.825679+0000 (oldest deadline 2024-03-31T23:39:10.125827+0000) 2024-03-31T23:39:11.361 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 31 23:39:11 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-9[67499]: 2024-03-31T23:39:11.027+0000 7fcc07615700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-31T23:38:10.469941+0000 front 2024-03-31T23:38:10.470388+0000 (oldest deadline 2024-03-31T23:38:35.170101+0000) 2024-03-31T23:39:11.361 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 31 23:39:11 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-9[67499]: 2024-03-31T23:39:11.027+0000 7fcc07615700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.142:6804 osd.1 since back 2024-03-31T23:38:21.571770+0000 front 2024-03-31T23:38:21.571801+0000 (oldest deadline 2024-03-31T23:38:43.871904+0000) 2024-03-31T23:39:11.361 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 31 23:39:11 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-9[67499]: 2024-03-31T23:39:11.027+0000 7fcc07615700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-31T23:38:27.074768+0000 front 2024-03-31T23:38:27.074793+0000 (oldest deadline 2024-03-31T23:38:52.374931+0000) 2024-03-31T23:39:11.361 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 31 23:39:11 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-9[67499]: 2024-03-31T23:39:11.027+0000 7fcc07615700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-03-31T23:38:34.075913+0000 front 2024-03-31T23:38:34.075868+0000 (oldest deadline 2024-03-31T23:38:59.376243+0000) 2024-03-31T23:39:11.545 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 31 23:39:11 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-8[55211]: 2024-03-31T23:39:11.088+0000 7fee85469700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-31T23:38:12.286566+0000 front 2024-03-31T23:38:12.286547+0000 (oldest deadline 2024-03-31T23:38:36.986433+0000) 2024-03-31T23:39:11.545 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 31 23:39:11 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-8[55211]: 2024-03-31T23:39:11.088+0000 7fee85469700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.142:6804 osd.1 since back 2024-03-31T23:38:21.587779+0000 front 2024-03-31T23:38:21.587835+0000 (oldest deadline 2024-03-31T23:38:47.487919+0000) 2024-03-31T23:39:11.545 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 31 23:39:11 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-8[55211]: 2024-03-31T23:39:11.088+0000 7fee85469700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-31T23:38:27.488310+0000 front 2024-03-31T23:38:27.488293+0000 (oldest deadline 2024-03-31T23:38:53.388420+0000) 2024-03-31T23:39:11.546 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 31 23:39:11 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-8[55211]: 2024-03-31T23:39:11.088+0000 7fee85469700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-03-31T23:38:33.389031+0000 front 2024-03-31T23:38:33.388915+0000 (oldest deadline 2024-03-31T23:38:59.288849+0000) 2024-03-31T23:39:11.546 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 31 23:39:11 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-8[55211]: 2024-03-31T23:39:11.088+0000 7fee85469700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.142:6812 osd.4 since back 2024-03-31T23:38:46.290078+0000 front 2024-03-31T23:38:46.290091+0000 (oldest deadline 2024-03-31T23:39:07.990166+0000) 2024-03-31T23:39:11.693 INFO:journalctl@ceph.osd.7.smithi142.stdout:Mar 31 23:39:11 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-7[53065]: 2024-03-31T23:39:11.522+0000 7fa41768e700 -1 osd.7 139 heartbeat_check: no reply from 172.21.15.142:6804 osd.1 since back 2024-03-31T23:38:18.318617+0000 front 2024-03-31T23:38:18.318635+0000 (oldest deadline 2024-03-31T23:38:48.319131+0000) 2024-03-31T23:39:11.693 INFO:journalctl@ceph.osd.7.smithi142.stdout:Mar 31 23:39:11 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-7[53065]: 2024-03-31T23:39:11.522+0000 7fa41768e700 -1 osd.7 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-31T23:38:28.319471+0000 front 2024-03-31T23:38:28.319515+0000 (oldest deadline 2024-03-31T23:38:54.219616+0000) 2024-03-31T23:39:11.693 INFO:journalctl@ceph.osd.7.smithi142.stdout:Mar 31 23:39:11 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-7[53065]: 2024-03-31T23:39:11.522+0000 7fa41768e700 -1 osd.7 139 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-03-31T23:38:36.421116+0000 front 2024-03-31T23:38:36.420912+0000 (oldest deadline 2024-03-31T23:38:59.920889+0000) 2024-03-31T23:39:11.693 INFO:journalctl@ceph.osd.7.smithi142.stdout:Mar 31 23:39:11 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-7[53065]: 2024-03-31T23:39:11.522+0000 7fa41768e700 -1 osd.7 139 heartbeat_check: no reply from 172.21.15.142:6812 osd.4 since back 2024-03-31T23:38:42.821718+0000 front 2024-03-31T23:38:42.821743+0000 (oldest deadline 2024-03-31T23:39:08.121808+0000) 2024-03-31T23:39:11.694 INFO:journalctl@ceph.osd.10.smithi142.stdout:Mar 31 23:39:11 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-10[60116]: 2024-03-31T23:39:11.436+0000 7f64c04a8700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-31T23:38:13.357804+0000 front 2024-03-31T23:38:13.357906+0000 (oldest deadline 2024-03-31T23:38:38.057730+0000) 2024-03-31T23:39:11.694 INFO:journalctl@ceph.osd.10.smithi142.stdout:Mar 31 23:39:11 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-10[60116]: 2024-03-31T23:39:11.436+0000 7f64c04a8700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.142:6804 osd.1 since back 2024-03-31T23:38:20.958341+0000 front 2024-03-31T23:38:20.958408+0000 (oldest deadline 2024-03-31T23:38:45.058481+0000) 2024-03-31T23:39:11.694 INFO:journalctl@ceph.osd.10.smithi142.stdout:Mar 31 23:39:11 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-10[60116]: 2024-03-31T23:39:11.436+0000 7f64c04a8700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-31T23:38:25.559393+0000 front 2024-03-31T23:38:25.559313+0000 (oldest deadline 2024-03-31T23:38:51.459398+0000) 2024-03-31T23:39:11.694 INFO:journalctl@ceph.osd.10.smithi142.stdout:Mar 31 23:39:11 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-10[60116]: 2024-03-31T23:39:11.436+0000 7f64c04a8700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-03-31T23:38:36.660831+0000 front 2024-03-31T23:38:36.660954+0000 (oldest deadline 2024-03-31T23:39:00.760714+0000) 2024-03-31T23:39:11.694 INFO:journalctl@ceph.osd.10.smithi142.stdout:Mar 31 23:39:11 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-10[60116]: 2024-03-31T23:39:11.436+0000 7f64c04a8700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.142:6812 osd.4 since back 2024-03-31T23:38:42.361714+0000 front 2024-03-31T23:38:42.361749+0000 (oldest deadline 2024-03-31T23:39:08.261872+0000) 2024-03-31T23:39:12.045 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:39:11 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:39:11.613+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-31T23:38:12.221366+0000 front 2024-03-31T23:38:12.221347+0000 (oldest deadline 2024-03-31T23:38:37.521357+0000) 2024-03-31T23:39:12.045 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:39:11 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:39:11.613+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.142:6804 osd.1 since back 2024-03-31T23:38:22.222192+0000 front 2024-03-31T23:38:22.222213+0000 (oldest deadline 2024-03-31T23:38:45.722340+0000) 2024-03-31T23:39:12.046 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:39:11 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:39:11.613+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-31T23:38:29.723616+0000 front 2024-03-31T23:38:29.723593+0000 (oldest deadline 2024-03-31T23:38:52.023779+0000) 2024-03-31T23:39:12.046 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:39:11 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:39:11.613+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-03-31T23:38:36.124750+0000 front 2024-03-31T23:38:36.124868+0000 (oldest deadline 2024-03-31T23:39:00.824619+0000) 2024-03-31T23:39:12.046 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:39:11 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:39:11.613+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.142:6812 osd.4 since back 2024-03-31T23:38:44.825661+0000 front 2024-03-31T23:38:44.825679+0000 (oldest deadline 2024-03-31T23:39:10.125827+0000) 2024-03-31T23:39:12.350 INFO:journalctl@ceph.osd.7.smithi142.stdout:Mar 31 23:39:12 smithi142 podman[78166]: 2024-03-31 23:39:12.065551523 +0000 UTC m=+5.554033415 container died 3d1126147b493e52d63407de56db59a919010739ec4a7f70f08fbff894d037f8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-7, org.label-schema.build-date=20240326, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-384f31f, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-03-31T23:39:12.361 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 31 23:39:12 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-9[67499]: 2024-03-31T23:39:12.014+0000 7fcc07615700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-31T23:38:10.469941+0000 front 2024-03-31T23:38:10.470388+0000 (oldest deadline 2024-03-31T23:38:35.170101+0000) 2024-03-31T23:39:12.361 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 31 23:39:12 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-9[67499]: 2024-03-31T23:39:12.014+0000 7fcc07615700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.142:6804 osd.1 since back 2024-03-31T23:38:21.571770+0000 front 2024-03-31T23:38:21.571801+0000 (oldest deadline 2024-03-31T23:38:43.871904+0000) 2024-03-31T23:39:12.361 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 31 23:39:12 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-9[67499]: 2024-03-31T23:39:12.014+0000 7fcc07615700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-31T23:38:27.074768+0000 front 2024-03-31T23:38:27.074793+0000 (oldest deadline 2024-03-31T23:38:52.374931+0000) 2024-03-31T23:39:12.361 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 31 23:39:12 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-9[67499]: 2024-03-31T23:39:12.014+0000 7fcc07615700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-03-31T23:38:34.075913+0000 front 2024-03-31T23:38:34.075868+0000 (oldest deadline 2024-03-31T23:38:59.376243+0000) 2024-03-31T23:39:12.361 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 31 23:39:12 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-9[67499]: 2024-03-31T23:39:12.014+0000 7fcc07615700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.142:6812 osd.4 since back 2024-03-31T23:38:45.177375+0000 front 2024-03-31T23:38:45.177457+0000 (oldest deadline 2024-03-31T23:39:11.077524+0000) 2024-03-31T23:39:12.545 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 31 23:39:12 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-8[55211]: 2024-03-31T23:39:12.078+0000 7fee85469700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-31T23:38:12.286566+0000 front 2024-03-31T23:38:12.286547+0000 (oldest deadline 2024-03-31T23:38:36.986433+0000) 2024-03-31T23:39:12.546 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 31 23:39:12 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-8[55211]: 2024-03-31T23:39:12.078+0000 7fee85469700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.142:6804 osd.1 since back 2024-03-31T23:38:21.587779+0000 front 2024-03-31T23:38:21.587835+0000 (oldest deadline 2024-03-31T23:38:47.487919+0000) 2024-03-31T23:39:12.546 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 31 23:39:12 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-8[55211]: 2024-03-31T23:39:12.078+0000 7fee85469700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-31T23:38:27.488310+0000 front 2024-03-31T23:38:27.488293+0000 (oldest deadline 2024-03-31T23:38:53.388420+0000) 2024-03-31T23:39:12.546 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 31 23:39:12 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-8[55211]: 2024-03-31T23:39:12.078+0000 7fee85469700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-03-31T23:38:33.389031+0000 front 2024-03-31T23:38:33.388915+0000 (oldest deadline 2024-03-31T23:38:59.288849+0000) 2024-03-31T23:39:12.546 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 31 23:39:12 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-8[55211]: 2024-03-31T23:39:12.078+0000 7fee85469700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.142:6812 osd.4 since back 2024-03-31T23:38:46.290078+0000 front 2024-03-31T23:38:46.290091+0000 (oldest deadline 2024-03-31T23:39:07.990166+0000) 2024-03-31T23:39:12.693 INFO:journalctl@ceph.osd.7.smithi142.stdout:Mar 31 23:39:12 smithi142 podman[78166]: 2024-03-31 23:39:12.441299674 +0000 UTC m=+5.929781565 container cleanup 3d1126147b493e52d63407de56db59a919010739ec4a7f70f08fbff894d037f8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-7, ceph=True, org.label-schema.build-date=20240326, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=squid-384f31f, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.license=GPLv2) 2024-03-31T23:39:12.693 INFO:journalctl@ceph.osd.7.smithi142.stdout:Mar 31 23:39:12 smithi142 bash[78166]: ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-7 2024-03-31T23:39:12.693 INFO:journalctl@ceph.osd.10.smithi142.stdout:Mar 31 23:39:12 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-10[60116]: 2024-03-31T23:39:12.471+0000 7f64c04a8700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-31T23:38:13.357804+0000 front 2024-03-31T23:38:13.357906+0000 (oldest deadline 2024-03-31T23:38:38.057730+0000) 2024-03-31T23:39:12.693 INFO:journalctl@ceph.osd.10.smithi142.stdout:Mar 31 23:39:12 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-10[60116]: 2024-03-31T23:39:12.471+0000 7f64c04a8700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.142:6804 osd.1 since back 2024-03-31T23:38:20.958341+0000 front 2024-03-31T23:38:20.958408+0000 (oldest deadline 2024-03-31T23:38:45.058481+0000) 2024-03-31T23:39:12.694 INFO:journalctl@ceph.osd.10.smithi142.stdout:Mar 31 23:39:12 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-10[60116]: 2024-03-31T23:39:12.471+0000 7f64c04a8700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-31T23:38:25.559393+0000 front 2024-03-31T23:38:25.559313+0000 (oldest deadline 2024-03-31T23:38:51.459398+0000) 2024-03-31T23:39:12.694 INFO:journalctl@ceph.osd.10.smithi142.stdout:Mar 31 23:39:12 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-10[60116]: 2024-03-31T23:39:12.471+0000 7f64c04a8700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-03-31T23:38:36.660831+0000 front 2024-03-31T23:38:36.660954+0000 (oldest deadline 2024-03-31T23:39:00.760714+0000) 2024-03-31T23:39:12.694 INFO:journalctl@ceph.osd.10.smithi142.stdout:Mar 31 23:39:12 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-10[60116]: 2024-03-31T23:39:12.471+0000 7f64c04a8700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.142:6812 osd.4 since back 2024-03-31T23:38:42.361714+0000 front 2024-03-31T23:38:42.361749+0000 (oldest deadline 2024-03-31T23:39:08.261872+0000) 2024-03-31T23:39:13.033 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:39:12 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:39:12.651+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-31T23:38:12.221366+0000 front 2024-03-31T23:38:12.221347+0000 (oldest deadline 2024-03-31T23:38:37.521357+0000) 2024-03-31T23:39:13.033 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:39:12 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:39:12.651+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.142:6804 osd.1 since back 2024-03-31T23:38:22.222192+0000 front 2024-03-31T23:38:22.222213+0000 (oldest deadline 2024-03-31T23:38:45.722340+0000) 2024-03-31T23:39:13.033 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:39:12 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:39:12.651+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-31T23:38:29.723616+0000 front 2024-03-31T23:38:29.723593+0000 (oldest deadline 2024-03-31T23:38:52.023779+0000) 2024-03-31T23:39:13.034 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:39:12 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:39:12.651+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-03-31T23:38:36.124750+0000 front 2024-03-31T23:38:36.124868+0000 (oldest deadline 2024-03-31T23:39:00.824619+0000) 2024-03-31T23:39:13.034 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:39:12 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:39:12.651+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.142:6812 osd.4 since back 2024-03-31T23:38:44.825661+0000 front 2024-03-31T23:38:44.825679+0000 (oldest deadline 2024-03-31T23:39:10.125827+0000) 2024-03-31T23:39:13.094 INFO:journalctl@ceph.osd.7.smithi142.stdout:Mar 31 23:39:12 smithi142 podman[78184]: 2024-03-31 23:39:12.751584651 +0000 UTC m=+0.681311041 container remove 3d1126147b493e52d63407de56db59a919010739ec4a7f70f08fbff894d037f8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-7, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , RELEASE=squid-384f31f, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240326) 2024-03-31T23:39:13.295 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 31 23:39:13 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-8[55211]: 2024-03-31T23:39:13.033+0000 7fee85469700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-31T23:38:12.286566+0000 front 2024-03-31T23:38:12.286547+0000 (oldest deadline 2024-03-31T23:38:36.986433+0000) 2024-03-31T23:39:13.295 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 31 23:39:13 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-8[55211]: 2024-03-31T23:39:13.033+0000 7fee85469700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.142:6804 osd.1 since back 2024-03-31T23:38:21.587779+0000 front 2024-03-31T23:38:21.587835+0000 (oldest deadline 2024-03-31T23:38:47.487919+0000) 2024-03-31T23:39:13.296 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 31 23:39:13 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-8[55211]: 2024-03-31T23:39:13.033+0000 7fee85469700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-31T23:38:27.488310+0000 front 2024-03-31T23:38:27.488293+0000 (oldest deadline 2024-03-31T23:38:53.388420+0000) 2024-03-31T23:39:13.296 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 31 23:39:13 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-8[55211]: 2024-03-31T23:39:13.033+0000 7fee85469700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-03-31T23:38:33.389031+0000 front 2024-03-31T23:38:33.388915+0000 (oldest deadline 2024-03-31T23:38:59.288849+0000) 2024-03-31T23:39:13.296 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 31 23:39:13 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-8[55211]: 2024-03-31T23:39:13.033+0000 7fee85469700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.142:6812 osd.4 since back 2024-03-31T23:38:46.290078+0000 front 2024-03-31T23:38:46.290091+0000 (oldest deadline 2024-03-31T23:39:07.990166+0000) 2024-03-31T23:39:13.361 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 31 23:39:12 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-9[67499]: 2024-03-31T23:39:12.982+0000 7fcc07615700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-31T23:38:10.469941+0000 front 2024-03-31T23:38:10.470388+0000 (oldest deadline 2024-03-31T23:38:35.170101+0000) 2024-03-31T23:39:13.362 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 31 23:39:12 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-9[67499]: 2024-03-31T23:39:12.982+0000 7fcc07615700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.142:6804 osd.1 since back 2024-03-31T23:38:21.571770+0000 front 2024-03-31T23:38:21.571801+0000 (oldest deadline 2024-03-31T23:38:43.871904+0000) 2024-03-31T23:39:13.362 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 31 23:39:12 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-9[67499]: 2024-03-31T23:39:12.982+0000 7fcc07615700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-31T23:38:27.074768+0000 front 2024-03-31T23:38:27.074793+0000 (oldest deadline 2024-03-31T23:38:52.374931+0000) 2024-03-31T23:39:13.362 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 31 23:39:12 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-9[67499]: 2024-03-31T23:39:12.982+0000 7fcc07615700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-03-31T23:38:34.075913+0000 front 2024-03-31T23:38:34.075868+0000 (oldest deadline 2024-03-31T23:38:59.376243+0000) 2024-03-31T23:39:13.362 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 31 23:39:12 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-9[67499]: 2024-03-31T23:39:12.982+0000 7fcc07615700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.142:6812 osd.4 since back 2024-03-31T23:38:45.177375+0000 front 2024-03-31T23:38:45.177457+0000 (oldest deadline 2024-03-31T23:39:11.077524+0000) 2024-03-31T23:39:13.388 INFO:journalctl@ceph.osd.7.smithi142.stdout:Mar 31 23:39:13 smithi142 podman[78271]: 2024-03-31 23:39:12.997726734 +0000 UTC m=+0.018101605 image pull ec18d7baa60e8b1a0b126fbe5306acce11b1ed0ec5e3bcdddbf39b0a3c591791 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175 2024-03-31T23:39:13.388 INFO:journalctl@ceph.osd.7.smithi142.stdout:Mar 31 23:39:13 smithi142 podman[78271]: 2024-03-31 23:39:13.12745472 +0000 UTC m=+0.147829573 container create e21f3990bc6093ea5f55dfa808e2c5b5b64437d4a20470684f417db81a9d5747 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-7-deactivate, org.label-schema.build-date=20240326, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-384f31f, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-03-31T23:39:13.388 INFO:journalctl@ceph.osd.7.smithi142.stdout:Mar 31 23:39:13 smithi142 podman[78271]: 2024-03-31 23:39:13.388211904 +0000 UTC m=+0.408586771 container init e21f3990bc6093ea5f55dfa808e2c5b5b64437d4a20470684f417db81a9d5747 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-7-deactivate, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=squid-384f31f, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240326, CEPH_POINT_RELEASE=, ceph=True) 2024-03-31T23:39:13.687 INFO:journalctl@ceph.osd.10.smithi142.stdout:Mar 31 23:39:13 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-10[60116]: 2024-03-31T23:39:13.455+0000 7f64c04a8700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-31T23:38:13.357804+0000 front 2024-03-31T23:38:13.357906+0000 (oldest deadline 2024-03-31T23:38:38.057730+0000) 2024-03-31T23:39:13.687 INFO:journalctl@ceph.osd.10.smithi142.stdout:Mar 31 23:39:13 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-10[60116]: 2024-03-31T23:39:13.455+0000 7f64c04a8700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.142:6804 osd.1 since back 2024-03-31T23:38:20.958341+0000 front 2024-03-31T23:38:20.958408+0000 (oldest deadline 2024-03-31T23:38:45.058481+0000) 2024-03-31T23:39:13.687 INFO:journalctl@ceph.osd.10.smithi142.stdout:Mar 31 23:39:13 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-10[60116]: 2024-03-31T23:39:13.455+0000 7f64c04a8700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-31T23:38:25.559393+0000 front 2024-03-31T23:38:25.559313+0000 (oldest deadline 2024-03-31T23:38:51.459398+0000) 2024-03-31T23:39:13.687 INFO:journalctl@ceph.osd.10.smithi142.stdout:Mar 31 23:39:13 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-10[60116]: 2024-03-31T23:39:13.455+0000 7f64c04a8700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-03-31T23:38:36.660831+0000 front 2024-03-31T23:38:36.660954+0000 (oldest deadline 2024-03-31T23:39:00.760714+0000) 2024-03-31T23:39:13.687 INFO:journalctl@ceph.osd.10.smithi142.stdout:Mar 31 23:39:13 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-10[60116]: 2024-03-31T23:39:13.455+0000 7f64c04a8700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.142:6812 osd.4 since back 2024-03-31T23:38:42.361714+0000 front 2024-03-31T23:38:42.361749+0000 (oldest deadline 2024-03-31T23:39:08.261872+0000) 2024-03-31T23:39:13.687 INFO:journalctl@ceph.osd.7.smithi142.stdout:Mar 31 23:39:13 smithi142 podman[78271]: 2024-03-31 23:39:13.393049871 +0000 UTC m=+0.413424728 container start e21f3990bc6093ea5f55dfa808e2c5b5b64437d4a20470684f417db81a9d5747 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-7-deactivate, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.build-date=20240326, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, RELEASE=squid-384f31f, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-03-31T23:39:13.687 INFO:journalctl@ceph.osd.7.smithi142.stdout:Mar 31 23:39:13 smithi142 podman[78271]: 2024-03-31 23:39:13.47165197 +0000 UTC m=+0.492026828 container attach e21f3990bc6093ea5f55dfa808e2c5b5b64437d4a20470684f417db81a9d5747 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-7-deactivate, RELEASE=squid-384f31f, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240326, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) 2024-03-31T23:39:13.943 INFO:journalctl@ceph.osd.7.smithi142.stdout:Mar 31 23:39:13 smithi142 podman[78271]: 2024-03-31 23:39:13.68734074 +0000 UTC m=+0.707715594 container died e21f3990bc6093ea5f55dfa808e2c5b5b64437d4a20470684f417db81a9d5747 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-7-deactivate, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , RELEASE=squid-384f31f, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.build-date=20240326, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2) 2024-03-31T23:39:14.045 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:39:13 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:39:13.679+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-31T23:38:12.221366+0000 front 2024-03-31T23:38:12.221347+0000 (oldest deadline 2024-03-31T23:38:37.521357+0000) 2024-03-31T23:39:14.045 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:39:13 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:39:13.679+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.142:6804 osd.1 since back 2024-03-31T23:38:22.222192+0000 front 2024-03-31T23:38:22.222213+0000 (oldest deadline 2024-03-31T23:38:45.722340+0000) 2024-03-31T23:39:14.046 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:39:13 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:39:13.679+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-31T23:38:29.723616+0000 front 2024-03-31T23:38:29.723593+0000 (oldest deadline 2024-03-31T23:38:52.023779+0000) 2024-03-31T23:39:14.046 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:39:13 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:39:13.679+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-03-31T23:38:36.124750+0000 front 2024-03-31T23:38:36.124868+0000 (oldest deadline 2024-03-31T23:39:00.824619+0000) 2024-03-31T23:39:14.046 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:39:13 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:39:13.679+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.142:6812 osd.4 since back 2024-03-31T23:38:44.825661+0000 front 2024-03-31T23:38:44.825679+0000 (oldest deadline 2024-03-31T23:39:10.125827+0000) 2024-03-31T23:39:14.263 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 31 23:39:13 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-9[67499]: 2024-03-31T23:39:13.984+0000 7fcc07615700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-31T23:38:10.469941+0000 front 2024-03-31T23:38:10.470388+0000 (oldest deadline 2024-03-31T23:38:35.170101+0000) 2024-03-31T23:39:14.263 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 31 23:39:13 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-9[67499]: 2024-03-31T23:39:13.984+0000 7fcc07615700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.142:6804 osd.1 since back 2024-03-31T23:38:21.571770+0000 front 2024-03-31T23:38:21.571801+0000 (oldest deadline 2024-03-31T23:38:43.871904+0000) 2024-03-31T23:39:14.263 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 31 23:39:13 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-9[67499]: 2024-03-31T23:39:13.984+0000 7fcc07615700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-31T23:38:27.074768+0000 front 2024-03-31T23:38:27.074793+0000 (oldest deadline 2024-03-31T23:38:52.374931+0000) 2024-03-31T23:39:14.263 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 31 23:39:13 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-9[67499]: 2024-03-31T23:39:13.984+0000 7fcc07615700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-03-31T23:38:34.075913+0000 front 2024-03-31T23:38:34.075868+0000 (oldest deadline 2024-03-31T23:38:59.376243+0000) 2024-03-31T23:39:14.263 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 31 23:39:13 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-9[67499]: 2024-03-31T23:39:13.984+0000 7fcc07615700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.142:6812 osd.4 since back 2024-03-31T23:38:45.177375+0000 front 2024-03-31T23:38:45.177457+0000 (oldest deadline 2024-03-31T23:39:11.077524+0000) 2024-03-31T23:39:14.296 DEBUG:teuthology.orchestra.run.smithi142:> sudo pkill -f 'journalctl -f -n 0 -u ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4@osd.7.service' 2024-03-31T23:39:14.340 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-31T23:39:14.340 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-03-31T23:39:14.340 INFO:tasks.cephadm.osd.8:Stopping osd.8... 2024-03-31T23:39:14.341 DEBUG:teuthology.orchestra.run.smithi162:> sudo systemctl stop ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4@osd.8 2024-03-31T23:39:14.365 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 31 23:39:14 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-8[55211]: 2024-03-31T23:39:14.059+0000 7fee85469700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-31T23:38:12.286566+0000 front 2024-03-31T23:38:12.286547+0000 (oldest deadline 2024-03-31T23:38:36.986433+0000) 2024-03-31T23:39:14.365 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 31 23:39:14 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-8[55211]: 2024-03-31T23:39:14.059+0000 7fee85469700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.142:6804 osd.1 since back 2024-03-31T23:38:21.587779+0000 front 2024-03-31T23:38:21.587835+0000 (oldest deadline 2024-03-31T23:38:47.487919+0000) 2024-03-31T23:39:14.366 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 31 23:39:14 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-8[55211]: 2024-03-31T23:39:14.059+0000 7fee85469700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-31T23:38:27.488310+0000 front 2024-03-31T23:38:27.488293+0000 (oldest deadline 2024-03-31T23:38:53.388420+0000) 2024-03-31T23:39:14.366 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 31 23:39:14 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-8[55211]: 2024-03-31T23:39:14.059+0000 7fee85469700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-03-31T23:38:33.389031+0000 front 2024-03-31T23:38:33.388915+0000 (oldest deadline 2024-03-31T23:38:59.288849+0000) 2024-03-31T23:39:14.366 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 31 23:39:14 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-8[55211]: 2024-03-31T23:39:14.059+0000 7fee85469700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.142:6812 osd.4 since back 2024-03-31T23:38:46.290078+0000 front 2024-03-31T23:38:46.290091+0000 (oldest deadline 2024-03-31T23:39:07.990166+0000) 2024-03-31T23:39:14.676 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 31 23:39:14 smithi162 systemd[1]: Stopping Ceph osd.8 for 6a9d8f08-efb2-11ee-b647-cb9ed24678a4... 2024-03-31T23:39:14.693 INFO:journalctl@ceph.osd.10.smithi142.stdout:Mar 31 23:39:14 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-10[60116]: 2024-03-31T23:39:14.429+0000 7f64c04a8700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-31T23:38:13.357804+0000 front 2024-03-31T23:38:13.357906+0000 (oldest deadline 2024-03-31T23:38:38.057730+0000) 2024-03-31T23:39:14.693 INFO:journalctl@ceph.osd.10.smithi142.stdout:Mar 31 23:39:14 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-10[60116]: 2024-03-31T23:39:14.429+0000 7f64c04a8700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.142:6804 osd.1 since back 2024-03-31T23:38:20.958341+0000 front 2024-03-31T23:38:20.958408+0000 (oldest deadline 2024-03-31T23:38:45.058481+0000) 2024-03-31T23:39:14.693 INFO:journalctl@ceph.osd.10.smithi142.stdout:Mar 31 23:39:14 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-10[60116]: 2024-03-31T23:39:14.429+0000 7f64c04a8700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-31T23:38:25.559393+0000 front 2024-03-31T23:38:25.559313+0000 (oldest deadline 2024-03-31T23:38:51.459398+0000) 2024-03-31T23:39:14.693 INFO:journalctl@ceph.osd.10.smithi142.stdout:Mar 31 23:39:14 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-10[60116]: 2024-03-31T23:39:14.429+0000 7f64c04a8700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-03-31T23:38:36.660831+0000 front 2024-03-31T23:38:36.660954+0000 (oldest deadline 2024-03-31T23:39:00.760714+0000) 2024-03-31T23:39:14.693 INFO:journalctl@ceph.osd.10.smithi142.stdout:Mar 31 23:39:14 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-10[60116]: 2024-03-31T23:39:14.429+0000 7f64c04a8700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.142:6812 osd.4 since back 2024-03-31T23:38:42.361714+0000 front 2024-03-31T23:38:42.361749+0000 (oldest deadline 2024-03-31T23:39:08.261872+0000) 2024-03-31T23:39:15.038 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 31 23:39:14 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-8[55211]: 2024-03-31T23:39:14.676+0000 7fee88e75700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.8 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-31T23:39:15.039 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 31 23:39:14 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-8[55211]: 2024-03-31T23:39:14.676+0000 7fee88e75700 -1 osd.8 139 *** Got signal Terminated *** 2024-03-31T23:39:15.039 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 31 23:39:14 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-8[55211]: 2024-03-31T23:39:14.676+0000 7fee88e75700 -1 osd.8 139 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-31T23:39:15.039 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 31 23:39:15 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-8[55211]: 2024-03-31T23:39:15.038+0000 7fee85469700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-31T23:38:12.286566+0000 front 2024-03-31T23:38:12.286547+0000 (oldest deadline 2024-03-31T23:38:36.986433+0000) 2024-03-31T23:39:15.039 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 31 23:39:15 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-8[55211]: 2024-03-31T23:39:15.038+0000 7fee85469700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.142:6804 osd.1 since back 2024-03-31T23:38:21.587779+0000 front 2024-03-31T23:38:21.587835+0000 (oldest deadline 2024-03-31T23:38:47.487919+0000) 2024-03-31T23:39:15.039 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 31 23:39:15 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-8[55211]: 2024-03-31T23:39:15.038+0000 7fee85469700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-31T23:38:27.488310+0000 front 2024-03-31T23:38:27.488293+0000 (oldest deadline 2024-03-31T23:38:53.388420+0000) 2024-03-31T23:39:15.039 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 31 23:39:15 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-8[55211]: 2024-03-31T23:39:15.038+0000 7fee85469700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-03-31T23:38:33.389031+0000 front 2024-03-31T23:38:33.388915+0000 (oldest deadline 2024-03-31T23:38:59.288849+0000) 2024-03-31T23:39:15.039 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:39:14 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:39:14.680+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-31T23:38:12.221366+0000 front 2024-03-31T23:38:12.221347+0000 (oldest deadline 2024-03-31T23:38:37.521357+0000) 2024-03-31T23:39:15.039 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:39:14 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:39:14.680+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.142:6804 osd.1 since back 2024-03-31T23:38:22.222192+0000 front 2024-03-31T23:38:22.222213+0000 (oldest deadline 2024-03-31T23:38:45.722340+0000) 2024-03-31T23:39:15.039 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:39:14 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:39:14.680+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-31T23:38:29.723616+0000 front 2024-03-31T23:38:29.723593+0000 (oldest deadline 2024-03-31T23:38:52.023779+0000) 2024-03-31T23:39:15.040 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:39:14 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:39:14.680+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-03-31T23:38:36.124750+0000 front 2024-03-31T23:38:36.124868+0000 (oldest deadline 2024-03-31T23:39:00.824619+0000) 2024-03-31T23:39:15.040 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:39:14 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:39:14.680+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.142:6812 osd.4 since back 2024-03-31T23:38:44.825661+0000 front 2024-03-31T23:38:44.825679+0000 (oldest deadline 2024-03-31T23:39:10.125827+0000) 2024-03-31T23:39:15.297 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 31 23:39:15 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-8[55211]: 2024-03-31T23:39:15.038+0000 7fee85469700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.142:6812 osd.4 since back 2024-03-31T23:38:46.290078+0000 front 2024-03-31T23:38:46.290091+0000 (oldest deadline 2024-03-31T23:39:07.990166+0000) 2024-03-31T23:39:15.362 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 31 23:39:14 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-9[67499]: 2024-03-31T23:39:14.998+0000 7fcc07615700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-31T23:38:10.469941+0000 front 2024-03-31T23:38:10.470388+0000 (oldest deadline 2024-03-31T23:38:35.170101+0000) 2024-03-31T23:39:15.396 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 31 23:39:14 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-9[67499]: 2024-03-31T23:39:14.998+0000 7fcc07615700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.142:6804 osd.1 since back 2024-03-31T23:38:21.571770+0000 front 2024-03-31T23:38:21.571801+0000 (oldest deadline 2024-03-31T23:38:43.871904+0000) 2024-03-31T23:39:15.396 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 31 23:39:14 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-9[67499]: 2024-03-31T23:39:14.998+0000 7fcc07615700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-31T23:38:27.074768+0000 front 2024-03-31T23:38:27.074793+0000 (oldest deadline 2024-03-31T23:38:52.374931+0000) 2024-03-31T23:39:15.396 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 31 23:39:14 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-9[67499]: 2024-03-31T23:39:14.998+0000 7fcc07615700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-03-31T23:38:34.075913+0000 front 2024-03-31T23:38:34.075868+0000 (oldest deadline 2024-03-31T23:38:59.376243+0000) 2024-03-31T23:39:15.396 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 31 23:39:14 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-9[67499]: 2024-03-31T23:39:14.998+0000 7fcc07615700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.142:6812 osd.4 since back 2024-03-31T23:38:45.177375+0000 front 2024-03-31T23:38:45.177457+0000 (oldest deadline 2024-03-31T23:39:11.077524+0000) 2024-03-31T23:39:15.693 INFO:journalctl@ceph.osd.10.smithi142.stdout:Mar 31 23:39:15 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-10[60116]: 2024-03-31T23:39:15.397+0000 7f64c04a8700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-31T23:38:13.357804+0000 front 2024-03-31T23:38:13.357906+0000 (oldest deadline 2024-03-31T23:38:38.057730+0000) 2024-03-31T23:39:15.693 INFO:journalctl@ceph.osd.10.smithi142.stdout:Mar 31 23:39:15 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-10[60116]: 2024-03-31T23:39:15.397+0000 7f64c04a8700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.142:6804 osd.1 since back 2024-03-31T23:38:20.958341+0000 front 2024-03-31T23:38:20.958408+0000 (oldest deadline 2024-03-31T23:38:45.058481+0000) 2024-03-31T23:39:15.693 INFO:journalctl@ceph.osd.10.smithi142.stdout:Mar 31 23:39:15 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-10[60116]: 2024-03-31T23:39:15.397+0000 7f64c04a8700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-31T23:38:25.559393+0000 front 2024-03-31T23:38:25.559313+0000 (oldest deadline 2024-03-31T23:38:51.459398+0000) 2024-03-31T23:39:15.693 INFO:journalctl@ceph.osd.10.smithi142.stdout:Mar 31 23:39:15 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-10[60116]: 2024-03-31T23:39:15.397+0000 7f64c04a8700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-03-31T23:38:36.660831+0000 front 2024-03-31T23:38:36.660954+0000 (oldest deadline 2024-03-31T23:39:00.760714+0000) 2024-03-31T23:39:15.693 INFO:journalctl@ceph.osd.10.smithi142.stdout:Mar 31 23:39:15 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-10[60116]: 2024-03-31T23:39:15.397+0000 7f64c04a8700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.142:6812 osd.4 since back 2024-03-31T23:38:42.361714+0000 front 2024-03-31T23:38:42.361749+0000 (oldest deadline 2024-03-31T23:39:08.261872+0000) 2024-03-31T23:39:16.045 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:39:15 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:39:15.711+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-31T23:38:12.221366+0000 front 2024-03-31T23:38:12.221347+0000 (oldest deadline 2024-03-31T23:38:37.521357+0000) 2024-03-31T23:39:16.045 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:39:15 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:39:15.711+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.142:6804 osd.1 since back 2024-03-31T23:38:22.222192+0000 front 2024-03-31T23:38:22.222213+0000 (oldest deadline 2024-03-31T23:38:45.722340+0000) 2024-03-31T23:39:16.046 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:39:15 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:39:15.711+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-31T23:38:29.723616+0000 front 2024-03-31T23:38:29.723593+0000 (oldest deadline 2024-03-31T23:38:52.023779+0000) 2024-03-31T23:39:16.046 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:39:15 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:39:15.711+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-03-31T23:38:36.124750+0000 front 2024-03-31T23:38:36.124868+0000 (oldest deadline 2024-03-31T23:39:00.824619+0000) 2024-03-31T23:39:16.046 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:39:15 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:39:15.711+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.142:6812 osd.4 since back 2024-03-31T23:38:44.825661+0000 front 2024-03-31T23:38:44.825679+0000 (oldest deadline 2024-03-31T23:39:10.125827+0000) 2024-03-31T23:39:16.361 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 31 23:39:16 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-9[67499]: 2024-03-31T23:39:16.011+0000 7fcc07615700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-31T23:38:10.469941+0000 front 2024-03-31T23:38:10.470388+0000 (oldest deadline 2024-03-31T23:38:35.170101+0000) 2024-03-31T23:39:16.361 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 31 23:39:16 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-9[67499]: 2024-03-31T23:39:16.011+0000 7fcc07615700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.142:6804 osd.1 since back 2024-03-31T23:38:21.571770+0000 front 2024-03-31T23:38:21.571801+0000 (oldest deadline 2024-03-31T23:38:43.871904+0000) 2024-03-31T23:39:16.362 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 31 23:39:16 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-9[67499]: 2024-03-31T23:39:16.011+0000 7fcc07615700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-31T23:38:27.074768+0000 front 2024-03-31T23:38:27.074793+0000 (oldest deadline 2024-03-31T23:38:52.374931+0000) 2024-03-31T23:39:16.362 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 31 23:39:16 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-9[67499]: 2024-03-31T23:39:16.011+0000 7fcc07615700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-03-31T23:38:34.075913+0000 front 2024-03-31T23:38:34.075868+0000 (oldest deadline 2024-03-31T23:38:59.376243+0000) 2024-03-31T23:39:16.362 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 31 23:39:16 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-9[67499]: 2024-03-31T23:39:16.011+0000 7fcc07615700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.142:6812 osd.4 since back 2024-03-31T23:38:45.177375+0000 front 2024-03-31T23:38:45.177457+0000 (oldest deadline 2024-03-31T23:39:11.077524+0000) 2024-03-31T23:39:16.545 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 31 23:39:16 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-8[55211]: 2024-03-31T23:39:16.068+0000 7fee85469700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-31T23:38:12.286566+0000 front 2024-03-31T23:38:12.286547+0000 (oldest deadline 2024-03-31T23:38:36.986433+0000) 2024-03-31T23:39:16.545 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 31 23:39:16 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-8[55211]: 2024-03-31T23:39:16.068+0000 7fee85469700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.142:6804 osd.1 since back 2024-03-31T23:38:21.587779+0000 front 2024-03-31T23:38:21.587835+0000 (oldest deadline 2024-03-31T23:38:47.487919+0000) 2024-03-31T23:39:16.546 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 31 23:39:16 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-8[55211]: 2024-03-31T23:39:16.068+0000 7fee85469700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-31T23:38:27.488310+0000 front 2024-03-31T23:38:27.488293+0000 (oldest deadline 2024-03-31T23:38:53.388420+0000) 2024-03-31T23:39:16.546 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 31 23:39:16 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-8[55211]: 2024-03-31T23:39:16.068+0000 7fee85469700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-03-31T23:38:33.389031+0000 front 2024-03-31T23:38:33.388915+0000 (oldest deadline 2024-03-31T23:38:59.288849+0000) 2024-03-31T23:39:16.546 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 31 23:39:16 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-8[55211]: 2024-03-31T23:39:16.068+0000 7fee85469700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.142:6812 osd.4 since back 2024-03-31T23:38:46.290078+0000 front 2024-03-31T23:38:46.290091+0000 (oldest deadline 2024-03-31T23:39:07.990166+0000) 2024-03-31T23:39:16.693 INFO:journalctl@ceph.osd.10.smithi142.stdout:Mar 31 23:39:16 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-10[60116]: 2024-03-31T23:39:16.380+0000 7f64c04a8700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-31T23:38:13.357804+0000 front 2024-03-31T23:38:13.357906+0000 (oldest deadline 2024-03-31T23:38:38.057730+0000) 2024-03-31T23:39:16.693 INFO:journalctl@ceph.osd.10.smithi142.stdout:Mar 31 23:39:16 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-10[60116]: 2024-03-31T23:39:16.380+0000 7f64c04a8700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.142:6804 osd.1 since back 2024-03-31T23:38:20.958341+0000 front 2024-03-31T23:38:20.958408+0000 (oldest deadline 2024-03-31T23:38:45.058481+0000) 2024-03-31T23:39:16.693 INFO:journalctl@ceph.osd.10.smithi142.stdout:Mar 31 23:39:16 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-10[60116]: 2024-03-31T23:39:16.380+0000 7f64c04a8700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-31T23:38:25.559393+0000 front 2024-03-31T23:38:25.559313+0000 (oldest deadline 2024-03-31T23:38:51.459398+0000) 2024-03-31T23:39:16.693 INFO:journalctl@ceph.osd.10.smithi142.stdout:Mar 31 23:39:16 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-10[60116]: 2024-03-31T23:39:16.380+0000 7f64c04a8700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-03-31T23:38:36.660831+0000 front 2024-03-31T23:38:36.660954+0000 (oldest deadline 2024-03-31T23:39:00.760714+0000) 2024-03-31T23:39:16.693 INFO:journalctl@ceph.osd.10.smithi142.stdout:Mar 31 23:39:16 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-10[60116]: 2024-03-31T23:39:16.380+0000 7f64c04a8700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.142:6812 osd.4 since back 2024-03-31T23:38:42.361714+0000 front 2024-03-31T23:38:42.361749+0000 (oldest deadline 2024-03-31T23:39:08.261872+0000) 2024-03-31T23:39:17.045 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:39:16 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:39:16.705+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-31T23:38:12.221366+0000 front 2024-03-31T23:38:12.221347+0000 (oldest deadline 2024-03-31T23:38:37.521357+0000) 2024-03-31T23:39:17.045 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:39:16 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:39:16.705+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.142:6804 osd.1 since back 2024-03-31T23:38:22.222192+0000 front 2024-03-31T23:38:22.222213+0000 (oldest deadline 2024-03-31T23:38:45.722340+0000) 2024-03-31T23:39:17.046 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:39:16 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:39:16.705+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-31T23:38:29.723616+0000 front 2024-03-31T23:38:29.723593+0000 (oldest deadline 2024-03-31T23:38:52.023779+0000) 2024-03-31T23:39:17.046 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:39:16 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:39:16.705+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-03-31T23:38:36.124750+0000 front 2024-03-31T23:38:36.124868+0000 (oldest deadline 2024-03-31T23:39:00.824619+0000) 2024-03-31T23:39:17.046 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:39:16 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:39:16.705+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.142:6812 osd.4 since back 2024-03-31T23:38:44.825661+0000 front 2024-03-31T23:38:44.825679+0000 (oldest deadline 2024-03-31T23:39:10.125827+0000) 2024-03-31T23:39:17.046 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:39:16 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:39:16.705+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-03-31T23:38:51.226778+0000 front 2024-03-31T23:38:51.226798+0000 (oldest deadline 2024-03-31T23:39:15.926915+0000) 2024-03-31T23:39:17.361 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 31 23:39:17 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-9[67499]: 2024-03-31T23:39:17.016+0000 7fcc07615700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-31T23:38:10.469941+0000 front 2024-03-31T23:38:10.470388+0000 (oldest deadline 2024-03-31T23:38:35.170101+0000) 2024-03-31T23:39:17.362 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 31 23:39:17 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-9[67499]: 2024-03-31T23:39:17.016+0000 7fcc07615700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.142:6804 osd.1 since back 2024-03-31T23:38:21.571770+0000 front 2024-03-31T23:38:21.571801+0000 (oldest deadline 2024-03-31T23:38:43.871904+0000) 2024-03-31T23:39:17.362 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 31 23:39:17 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-9[67499]: 2024-03-31T23:39:17.016+0000 7fcc07615700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-31T23:38:27.074768+0000 front 2024-03-31T23:38:27.074793+0000 (oldest deadline 2024-03-31T23:38:52.374931+0000) 2024-03-31T23:39:17.362 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 31 23:39:17 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-9[67499]: 2024-03-31T23:39:17.016+0000 7fcc07615700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-03-31T23:38:34.075913+0000 front 2024-03-31T23:38:34.075868+0000 (oldest deadline 2024-03-31T23:38:59.376243+0000) 2024-03-31T23:39:17.362 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 31 23:39:17 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-9[67499]: 2024-03-31T23:39:17.016+0000 7fcc07615700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.142:6812 osd.4 since back 2024-03-31T23:38:45.177375+0000 front 2024-03-31T23:38:45.177457+0000 (oldest deadline 2024-03-31T23:39:11.077524+0000) 2024-03-31T23:39:17.362 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 31 23:39:17 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-9[67499]: 2024-03-31T23:39:17.016+0000 7fcc07615700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-03-31T23:38:53.978583+0000 front 2024-03-31T23:38:53.978472+0000 (oldest deadline 2024-03-31T23:39:16.878532+0000) 2024-03-31T23:39:17.545 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 31 23:39:17 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-8[55211]: 2024-03-31T23:39:17.092+0000 7fee85469700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-31T23:38:12.286566+0000 front 2024-03-31T23:38:12.286547+0000 (oldest deadline 2024-03-31T23:38:36.986433+0000) 2024-03-31T23:39:17.545 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 31 23:39:17 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-8[55211]: 2024-03-31T23:39:17.092+0000 7fee85469700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.142:6804 osd.1 since back 2024-03-31T23:38:21.587779+0000 front 2024-03-31T23:38:21.587835+0000 (oldest deadline 2024-03-31T23:38:47.487919+0000) 2024-03-31T23:39:17.545 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 31 23:39:17 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-8[55211]: 2024-03-31T23:39:17.092+0000 7fee85469700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-31T23:38:27.488310+0000 front 2024-03-31T23:38:27.488293+0000 (oldest deadline 2024-03-31T23:38:53.388420+0000) 2024-03-31T23:39:17.546 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 31 23:39:17 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-8[55211]: 2024-03-31T23:39:17.092+0000 7fee85469700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-03-31T23:38:33.389031+0000 front 2024-03-31T23:38:33.388915+0000 (oldest deadline 2024-03-31T23:38:59.288849+0000) 2024-03-31T23:39:17.546 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 31 23:39:17 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-8[55211]: 2024-03-31T23:39:17.092+0000 7fee85469700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.142:6812 osd.4 since back 2024-03-31T23:38:46.290078+0000 front 2024-03-31T23:38:46.290091+0000 (oldest deadline 2024-03-31T23:39:07.990166+0000) 2024-03-31T23:39:17.693 INFO:journalctl@ceph.osd.10.smithi142.stdout:Mar 31 23:39:17 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-10[60116]: 2024-03-31T23:39:17.380+0000 7f64c04a8700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-31T23:38:13.357804+0000 front 2024-03-31T23:38:13.357906+0000 (oldest deadline 2024-03-31T23:38:38.057730+0000) 2024-03-31T23:39:17.693 INFO:journalctl@ceph.osd.10.smithi142.stdout:Mar 31 23:39:17 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-10[60116]: 2024-03-31T23:39:17.380+0000 7f64c04a8700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.142:6804 osd.1 since back 2024-03-31T23:38:20.958341+0000 front 2024-03-31T23:38:20.958408+0000 (oldest deadline 2024-03-31T23:38:45.058481+0000) 2024-03-31T23:39:17.693 INFO:journalctl@ceph.osd.10.smithi142.stdout:Mar 31 23:39:17 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-10[60116]: 2024-03-31T23:39:17.380+0000 7f64c04a8700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-31T23:38:25.559393+0000 front 2024-03-31T23:38:25.559313+0000 (oldest deadline 2024-03-31T23:38:51.459398+0000) 2024-03-31T23:39:17.693 INFO:journalctl@ceph.osd.10.smithi142.stdout:Mar 31 23:39:17 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-10[60116]: 2024-03-31T23:39:17.380+0000 7f64c04a8700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-03-31T23:38:36.660831+0000 front 2024-03-31T23:38:36.660954+0000 (oldest deadline 2024-03-31T23:39:00.760714+0000) 2024-03-31T23:39:17.693 INFO:journalctl@ceph.osd.10.smithi142.stdout:Mar 31 23:39:17 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-10[60116]: 2024-03-31T23:39:17.380+0000 7f64c04a8700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.142:6812 osd.4 since back 2024-03-31T23:38:42.361714+0000 front 2024-03-31T23:38:42.361749+0000 (oldest deadline 2024-03-31T23:39:08.261872+0000) 2024-03-31T23:39:18.045 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:39:17 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:39:17.724+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-31T23:38:12.221366+0000 front 2024-03-31T23:38:12.221347+0000 (oldest deadline 2024-03-31T23:38:37.521357+0000) 2024-03-31T23:39:18.046 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:39:17 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:39:17.724+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.142:6804 osd.1 since back 2024-03-31T23:38:22.222192+0000 front 2024-03-31T23:38:22.222213+0000 (oldest deadline 2024-03-31T23:38:45.722340+0000) 2024-03-31T23:39:18.046 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:39:17 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:39:17.724+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-31T23:38:29.723616+0000 front 2024-03-31T23:38:29.723593+0000 (oldest deadline 2024-03-31T23:38:52.023779+0000) 2024-03-31T23:39:18.046 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:39:17 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:39:17.724+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-03-31T23:38:36.124750+0000 front 2024-03-31T23:38:36.124868+0000 (oldest deadline 2024-03-31T23:39:00.824619+0000) 2024-03-31T23:39:18.046 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:39:17 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:39:17.724+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.142:6812 osd.4 since back 2024-03-31T23:38:44.825661+0000 front 2024-03-31T23:38:44.825679+0000 (oldest deadline 2024-03-31T23:39:10.125827+0000) 2024-03-31T23:39:18.046 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:39:17 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:39:17.724+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-03-31T23:38:51.226778+0000 front 2024-03-31T23:38:51.226798+0000 (oldest deadline 2024-03-31T23:39:15.926915+0000) 2024-03-31T23:39:18.361 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 31 23:39:18 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-9[67499]: 2024-03-31T23:39:18.039+0000 7fcc07615700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-31T23:38:10.469941+0000 front 2024-03-31T23:38:10.470388+0000 (oldest deadline 2024-03-31T23:38:35.170101+0000) 2024-03-31T23:39:18.361 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 31 23:39:18 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-9[67499]: 2024-03-31T23:39:18.039+0000 7fcc07615700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.142:6804 osd.1 since back 2024-03-31T23:38:21.571770+0000 front 2024-03-31T23:38:21.571801+0000 (oldest deadline 2024-03-31T23:38:43.871904+0000) 2024-03-31T23:39:18.361 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 31 23:39:18 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-9[67499]: 2024-03-31T23:39:18.039+0000 7fcc07615700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-31T23:38:27.074768+0000 front 2024-03-31T23:38:27.074793+0000 (oldest deadline 2024-03-31T23:38:52.374931+0000) 2024-03-31T23:39:18.362 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 31 23:39:18 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-9[67499]: 2024-03-31T23:39:18.039+0000 7fcc07615700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-03-31T23:38:34.075913+0000 front 2024-03-31T23:38:34.075868+0000 (oldest deadline 2024-03-31T23:38:59.376243+0000) 2024-03-31T23:39:18.362 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 31 23:39:18 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-9[67499]: 2024-03-31T23:39:18.039+0000 7fcc07615700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.142:6812 osd.4 since back 2024-03-31T23:38:45.177375+0000 front 2024-03-31T23:38:45.177457+0000 (oldest deadline 2024-03-31T23:39:11.077524+0000) 2024-03-31T23:39:18.362 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 31 23:39:18 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-9[67499]: 2024-03-31T23:39:18.039+0000 7fcc07615700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-03-31T23:38:53.978583+0000 front 2024-03-31T23:38:53.978472+0000 (oldest deadline 2024-03-31T23:39:16.878532+0000) 2024-03-31T23:39:18.546 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 31 23:39:18 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-8[55211]: 2024-03-31T23:39:18.093+0000 7fee85469700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-31T23:38:12.286566+0000 front 2024-03-31T23:38:12.286547+0000 (oldest deadline 2024-03-31T23:38:36.986433+0000) 2024-03-31T23:39:18.546 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 31 23:39:18 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-8[55211]: 2024-03-31T23:39:18.093+0000 7fee85469700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.142:6804 osd.1 since back 2024-03-31T23:38:21.587779+0000 front 2024-03-31T23:38:21.587835+0000 (oldest deadline 2024-03-31T23:38:47.487919+0000) 2024-03-31T23:39:18.546 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 31 23:39:18 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-8[55211]: 2024-03-31T23:39:18.093+0000 7fee85469700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-31T23:38:27.488310+0000 front 2024-03-31T23:38:27.488293+0000 (oldest deadline 2024-03-31T23:38:53.388420+0000) 2024-03-31T23:39:18.546 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 31 23:39:18 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-8[55211]: 2024-03-31T23:39:18.093+0000 7fee85469700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-03-31T23:38:33.389031+0000 front 2024-03-31T23:38:33.388915+0000 (oldest deadline 2024-03-31T23:38:59.288849+0000) 2024-03-31T23:39:18.546 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 31 23:39:18 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-8[55211]: 2024-03-31T23:39:18.093+0000 7fee85469700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.142:6812 osd.4 since back 2024-03-31T23:38:46.290078+0000 front 2024-03-31T23:38:46.290091+0000 (oldest deadline 2024-03-31T23:39:07.990166+0000) 2024-03-31T23:39:18.693 INFO:journalctl@ceph.osd.10.smithi142.stdout:Mar 31 23:39:18 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-10[60116]: 2024-03-31T23:39:18.375+0000 7f64c04a8700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-31T23:38:13.357804+0000 front 2024-03-31T23:38:13.357906+0000 (oldest deadline 2024-03-31T23:38:38.057730+0000) 2024-03-31T23:39:18.693 INFO:journalctl@ceph.osd.10.smithi142.stdout:Mar 31 23:39:18 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-10[60116]: 2024-03-31T23:39:18.375+0000 7f64c04a8700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.142:6804 osd.1 since back 2024-03-31T23:38:20.958341+0000 front 2024-03-31T23:38:20.958408+0000 (oldest deadline 2024-03-31T23:38:45.058481+0000) 2024-03-31T23:39:18.693 INFO:journalctl@ceph.osd.10.smithi142.stdout:Mar 31 23:39:18 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-10[60116]: 2024-03-31T23:39:18.375+0000 7f64c04a8700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-31T23:38:25.559393+0000 front 2024-03-31T23:38:25.559313+0000 (oldest deadline 2024-03-31T23:38:51.459398+0000) 2024-03-31T23:39:18.693 INFO:journalctl@ceph.osd.10.smithi142.stdout:Mar 31 23:39:18 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-10[60116]: 2024-03-31T23:39:18.375+0000 7f64c04a8700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-03-31T23:38:36.660831+0000 front 2024-03-31T23:38:36.660954+0000 (oldest deadline 2024-03-31T23:39:00.760714+0000) 2024-03-31T23:39:18.693 INFO:journalctl@ceph.osd.10.smithi142.stdout:Mar 31 23:39:18 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-10[60116]: 2024-03-31T23:39:18.375+0000 7f64c04a8700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.142:6812 osd.4 since back 2024-03-31T23:38:42.361714+0000 front 2024-03-31T23:38:42.361749+0000 (oldest deadline 2024-03-31T23:39:08.261872+0000) 2024-03-31T23:39:18.693 INFO:journalctl@ceph.osd.10.smithi142.stdout:Mar 31 23:39:18 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-10[60116]: 2024-03-31T23:39:18.375+0000 7f64c04a8700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-03-31T23:38:52.962747+0000 front 2024-03-31T23:38:52.962731+0000 (oldest deadline 2024-03-31T23:39:17.662721+0000) 2024-03-31T23:39:19.045 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:39:18 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:39:18.767+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-31T23:38:12.221366+0000 front 2024-03-31T23:38:12.221347+0000 (oldest deadline 2024-03-31T23:38:37.521357+0000) 2024-03-31T23:39:19.046 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:39:18 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:39:18.767+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.142:6804 osd.1 since back 2024-03-31T23:38:22.222192+0000 front 2024-03-31T23:38:22.222213+0000 (oldest deadline 2024-03-31T23:38:45.722340+0000) 2024-03-31T23:39:19.046 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:39:18 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:39:18.767+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-31T23:38:29.723616+0000 front 2024-03-31T23:38:29.723593+0000 (oldest deadline 2024-03-31T23:38:52.023779+0000) 2024-03-31T23:39:19.046 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:39:18 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:39:18.767+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-03-31T23:38:36.124750+0000 front 2024-03-31T23:38:36.124868+0000 (oldest deadline 2024-03-31T23:39:00.824619+0000) 2024-03-31T23:39:19.046 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:39:18 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:39:18.767+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.142:6812 osd.4 since back 2024-03-31T23:38:44.825661+0000 front 2024-03-31T23:38:44.825679+0000 (oldest deadline 2024-03-31T23:39:10.125827+0000) 2024-03-31T23:39:19.046 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:39:18 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:39:18.767+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-03-31T23:38:51.226778+0000 front 2024-03-31T23:38:51.226798+0000 (oldest deadline 2024-03-31T23:39:15.926915+0000) 2024-03-31T23:39:19.361 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 31 23:39:19 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-9[67499]: 2024-03-31T23:39:19.086+0000 7fcc07615700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-31T23:38:10.469941+0000 front 2024-03-31T23:38:10.470388+0000 (oldest deadline 2024-03-31T23:38:35.170101+0000) 2024-03-31T23:39:19.362 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 31 23:39:19 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-9[67499]: 2024-03-31T23:39:19.086+0000 7fcc07615700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.142:6804 osd.1 since back 2024-03-31T23:38:21.571770+0000 front 2024-03-31T23:38:21.571801+0000 (oldest deadline 2024-03-31T23:38:43.871904+0000) 2024-03-31T23:39:19.362 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 31 23:39:19 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-9[67499]: 2024-03-31T23:39:19.086+0000 7fcc07615700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-31T23:38:27.074768+0000 front 2024-03-31T23:38:27.074793+0000 (oldest deadline 2024-03-31T23:38:52.374931+0000) 2024-03-31T23:39:19.362 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 31 23:39:19 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-9[67499]: 2024-03-31T23:39:19.086+0000 7fcc07615700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-03-31T23:38:34.075913+0000 front 2024-03-31T23:38:34.075868+0000 (oldest deadline 2024-03-31T23:38:59.376243+0000) 2024-03-31T23:39:19.362 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 31 23:39:19 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-9[67499]: 2024-03-31T23:39:19.086+0000 7fcc07615700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.142:6812 osd.4 since back 2024-03-31T23:38:45.177375+0000 front 2024-03-31T23:38:45.177457+0000 (oldest deadline 2024-03-31T23:39:11.077524+0000) 2024-03-31T23:39:19.362 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 31 23:39:19 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-9[67499]: 2024-03-31T23:39:19.086+0000 7fcc07615700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-03-31T23:38:53.978583+0000 front 2024-03-31T23:38:53.978472+0000 (oldest deadline 2024-03-31T23:39:16.878532+0000) 2024-03-31T23:39:19.546 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 31 23:39:19 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-8[55211]: 2024-03-31T23:39:19.083+0000 7fee85469700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-31T23:38:12.286566+0000 front 2024-03-31T23:38:12.286547+0000 (oldest deadline 2024-03-31T23:38:36.986433+0000) 2024-03-31T23:39:19.546 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 31 23:39:19 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-8[55211]: 2024-03-31T23:39:19.083+0000 7fee85469700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.142:6804 osd.1 since back 2024-03-31T23:38:21.587779+0000 front 2024-03-31T23:38:21.587835+0000 (oldest deadline 2024-03-31T23:38:47.487919+0000) 2024-03-31T23:39:19.546 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 31 23:39:19 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-8[55211]: 2024-03-31T23:39:19.083+0000 7fee85469700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-31T23:38:27.488310+0000 front 2024-03-31T23:38:27.488293+0000 (oldest deadline 2024-03-31T23:38:53.388420+0000) 2024-03-31T23:39:19.546 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 31 23:39:19 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-8[55211]: 2024-03-31T23:39:19.083+0000 7fee85469700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-03-31T23:38:33.389031+0000 front 2024-03-31T23:38:33.388915+0000 (oldest deadline 2024-03-31T23:38:59.288849+0000) 2024-03-31T23:39:19.546 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 31 23:39:19 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-8[55211]: 2024-03-31T23:39:19.083+0000 7fee85469700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.142:6812 osd.4 since back 2024-03-31T23:38:46.290078+0000 front 2024-03-31T23:38:46.290091+0000 (oldest deadline 2024-03-31T23:39:07.990166+0000) 2024-03-31T23:39:19.693 INFO:journalctl@ceph.osd.10.smithi142.stdout:Mar 31 23:39:19 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-10[60116]: 2024-03-31T23:39:19.375+0000 7f64c04a8700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-31T23:38:13.357804+0000 front 2024-03-31T23:38:13.357906+0000 (oldest deadline 2024-03-31T23:38:38.057730+0000) 2024-03-31T23:39:19.693 INFO:journalctl@ceph.osd.10.smithi142.stdout:Mar 31 23:39:19 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-10[60116]: 2024-03-31T23:39:19.375+0000 7f64c04a8700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.142:6804 osd.1 since back 2024-03-31T23:38:20.958341+0000 front 2024-03-31T23:38:20.958408+0000 (oldest deadline 2024-03-31T23:38:45.058481+0000) 2024-03-31T23:39:19.693 INFO:journalctl@ceph.osd.10.smithi142.stdout:Mar 31 23:39:19 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-10[60116]: 2024-03-31T23:39:19.375+0000 7f64c04a8700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-31T23:38:25.559393+0000 front 2024-03-31T23:38:25.559313+0000 (oldest deadline 2024-03-31T23:38:51.459398+0000) 2024-03-31T23:39:19.693 INFO:journalctl@ceph.osd.10.smithi142.stdout:Mar 31 23:39:19 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-10[60116]: 2024-03-31T23:39:19.375+0000 7f64c04a8700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-03-31T23:38:36.660831+0000 front 2024-03-31T23:38:36.660954+0000 (oldest deadline 2024-03-31T23:39:00.760714+0000) 2024-03-31T23:39:19.694 INFO:journalctl@ceph.osd.10.smithi142.stdout:Mar 31 23:39:19 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-10[60116]: 2024-03-31T23:39:19.375+0000 7f64c04a8700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.142:6812 osd.4 since back 2024-03-31T23:38:42.361714+0000 front 2024-03-31T23:38:42.361749+0000 (oldest deadline 2024-03-31T23:39:08.261872+0000) 2024-03-31T23:39:19.694 INFO:journalctl@ceph.osd.10.smithi142.stdout:Mar 31 23:39:19 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-10[60116]: 2024-03-31T23:39:19.375+0000 7f64c04a8700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-03-31T23:38:52.962747+0000 front 2024-03-31T23:38:52.962731+0000 (oldest deadline 2024-03-31T23:39:17.662721+0000) 2024-03-31T23:39:20.045 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:39:19 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:39:19.768+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-31T23:38:12.221366+0000 front 2024-03-31T23:38:12.221347+0000 (oldest deadline 2024-03-31T23:38:37.521357+0000) 2024-03-31T23:39:20.046 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:39:19 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:39:19.768+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.142:6804 osd.1 since back 2024-03-31T23:38:22.222192+0000 front 2024-03-31T23:38:22.222213+0000 (oldest deadline 2024-03-31T23:38:45.722340+0000) 2024-03-31T23:39:20.046 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:39:19 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:39:19.768+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-31T23:38:29.723616+0000 front 2024-03-31T23:38:29.723593+0000 (oldest deadline 2024-03-31T23:38:52.023779+0000) 2024-03-31T23:39:20.046 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:39:19 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:39:19.768+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-03-31T23:38:36.124750+0000 front 2024-03-31T23:38:36.124868+0000 (oldest deadline 2024-03-31T23:39:00.824619+0000) 2024-03-31T23:39:20.046 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:39:19 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:39:19.768+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.142:6812 osd.4 since back 2024-03-31T23:38:44.825661+0000 front 2024-03-31T23:38:44.825679+0000 (oldest deadline 2024-03-31T23:39:10.125827+0000) 2024-03-31T23:39:20.046 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:39:19 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:39:19.768+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-03-31T23:38:51.226778+0000 front 2024-03-31T23:38:51.226798+0000 (oldest deadline 2024-03-31T23:39:15.926915+0000) 2024-03-31T23:39:20.361 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 31 23:39:20 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-9[67499]: 2024-03-31T23:39:20.074+0000 7fcc07615700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-31T23:38:10.469941+0000 front 2024-03-31T23:38:10.470388+0000 (oldest deadline 2024-03-31T23:38:35.170101+0000) 2024-03-31T23:39:20.361 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 31 23:39:20 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-9[67499]: 2024-03-31T23:39:20.074+0000 7fcc07615700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.142:6804 osd.1 since back 2024-03-31T23:38:21.571770+0000 front 2024-03-31T23:38:21.571801+0000 (oldest deadline 2024-03-31T23:38:43.871904+0000) 2024-03-31T23:39:20.362 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 31 23:39:20 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-9[67499]: 2024-03-31T23:39:20.074+0000 7fcc07615700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-31T23:38:27.074768+0000 front 2024-03-31T23:38:27.074793+0000 (oldest deadline 2024-03-31T23:38:52.374931+0000) 2024-03-31T23:39:20.362 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 31 23:39:20 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-9[67499]: 2024-03-31T23:39:20.074+0000 7fcc07615700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-03-31T23:38:34.075913+0000 front 2024-03-31T23:38:34.075868+0000 (oldest deadline 2024-03-31T23:38:59.376243+0000) 2024-03-31T23:39:20.362 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 31 23:39:20 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-9[67499]: 2024-03-31T23:39:20.074+0000 7fcc07615700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.142:6812 osd.4 since back 2024-03-31T23:38:45.177375+0000 front 2024-03-31T23:38:45.177457+0000 (oldest deadline 2024-03-31T23:39:11.077524+0000) 2024-03-31T23:39:20.362 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 31 23:39:20 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-9[67499]: 2024-03-31T23:39:20.075+0000 7fcc07615700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-03-31T23:38:53.978583+0000 front 2024-03-31T23:38:53.978472+0000 (oldest deadline 2024-03-31T23:39:16.878532+0000) 2024-03-31T23:39:20.501 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 31 23:39:20 smithi162 podman[78549]: 2024-03-31 23:39:20.234866733 +0000 UTC m=+5.709174833 container died 665072d8f71214a1fa9d8e5a1cf96d38c3fecf994bcbd32ab3ae4f5b1abf30c8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-8, CEPH_POINT_RELEASE=, RELEASE=squid-384f31f, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240326, GIT_CLEAN=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-31T23:39:20.693 INFO:journalctl@ceph.osd.10.smithi142.stdout:Mar 31 23:39:20 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-10[60116]: 2024-03-31T23:39:20.366+0000 7f64c04a8700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-31T23:38:13.357804+0000 front 2024-03-31T23:38:13.357906+0000 (oldest deadline 2024-03-31T23:38:38.057730+0000) 2024-03-31T23:39:20.694 INFO:journalctl@ceph.osd.10.smithi142.stdout:Mar 31 23:39:20 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-10[60116]: 2024-03-31T23:39:20.366+0000 7f64c04a8700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.142:6804 osd.1 since back 2024-03-31T23:38:20.958341+0000 front 2024-03-31T23:38:20.958408+0000 (oldest deadline 2024-03-31T23:38:45.058481+0000) 2024-03-31T23:39:20.694 INFO:journalctl@ceph.osd.10.smithi142.stdout:Mar 31 23:39:20 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-10[60116]: 2024-03-31T23:39:20.366+0000 7f64c04a8700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-31T23:38:25.559393+0000 front 2024-03-31T23:38:25.559313+0000 (oldest deadline 2024-03-31T23:38:51.459398+0000) 2024-03-31T23:39:20.694 INFO:journalctl@ceph.osd.10.smithi142.stdout:Mar 31 23:39:20 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-10[60116]: 2024-03-31T23:39:20.366+0000 7f64c04a8700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-03-31T23:38:36.660831+0000 front 2024-03-31T23:38:36.660954+0000 (oldest deadline 2024-03-31T23:39:00.760714+0000) 2024-03-31T23:39:20.694 INFO:journalctl@ceph.osd.10.smithi142.stdout:Mar 31 23:39:20 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-10[60116]: 2024-03-31T23:39:20.366+0000 7f64c04a8700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.142:6812 osd.4 since back 2024-03-31T23:38:42.361714+0000 front 2024-03-31T23:38:42.361749+0000 (oldest deadline 2024-03-31T23:39:08.261872+0000) 2024-03-31T23:39:20.694 INFO:journalctl@ceph.osd.10.smithi142.stdout:Mar 31 23:39:20 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-10[60116]: 2024-03-31T23:39:20.366+0000 7f64c04a8700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-03-31T23:38:52.962747+0000 front 2024-03-31T23:38:52.962731+0000 (oldest deadline 2024-03-31T23:39:17.662721+0000) 2024-03-31T23:39:20.760 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 31 23:39:20 smithi162 podman[78549]: 2024-03-31 23:39:20.59238067 +0000 UTC m=+6.066688766 container cleanup 665072d8f71214a1fa9d8e5a1cf96d38c3fecf994bcbd32ab3ae4f5b1abf30c8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-8, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, RELEASE=squid-384f31f, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240326, org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=) 2024-03-31T23:39:20.760 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 31 23:39:20 smithi162 bash[78549]: ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-8 2024-03-31T23:39:21.045 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 31 23:39:20 smithi162 podman[78567]: 2024-03-31 23:39:20.927468249 +0000 UTC m=+0.690605887 container remove 665072d8f71214a1fa9d8e5a1cf96d38c3fecf994bcbd32ab3ae4f5b1abf30c8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-8, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.build-date=20240326, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=squid-384f31f, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-31T23:39:21.046 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:39:20 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:39:20.759+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-31T23:38:12.221366+0000 front 2024-03-31T23:38:12.221347+0000 (oldest deadline 2024-03-31T23:38:37.521357+0000) 2024-03-31T23:39:21.046 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:39:20 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:39:20.759+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.142:6804 osd.1 since back 2024-03-31T23:38:22.222192+0000 front 2024-03-31T23:38:22.222213+0000 (oldest deadline 2024-03-31T23:38:45.722340+0000) 2024-03-31T23:39:21.046 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:39:20 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:39:20.759+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-31T23:38:29.723616+0000 front 2024-03-31T23:38:29.723593+0000 (oldest deadline 2024-03-31T23:38:52.023779+0000) 2024-03-31T23:39:21.046 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:39:20 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:39:20.759+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-03-31T23:38:36.124750+0000 front 2024-03-31T23:38:36.124868+0000 (oldest deadline 2024-03-31T23:39:00.824619+0000) 2024-03-31T23:39:21.046 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:39:20 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:39:20.759+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.142:6812 osd.4 since back 2024-03-31T23:38:44.825661+0000 front 2024-03-31T23:38:44.825679+0000 (oldest deadline 2024-03-31T23:39:10.125827+0000) 2024-03-31T23:39:21.046 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:39:20 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:39:20.759+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-03-31T23:38:51.226778+0000 front 2024-03-31T23:38:51.226798+0000 (oldest deadline 2024-03-31T23:39:15.926915+0000) 2024-03-31T23:39:21.361 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 31 23:39:21 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-9[67499]: 2024-03-31T23:39:21.053+0000 7fcc07615700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-31T23:38:10.469941+0000 front 2024-03-31T23:38:10.470388+0000 (oldest deadline 2024-03-31T23:38:35.170101+0000) 2024-03-31T23:39:21.362 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 31 23:39:21 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-9[67499]: 2024-03-31T23:39:21.053+0000 7fcc07615700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.142:6804 osd.1 since back 2024-03-31T23:38:21.571770+0000 front 2024-03-31T23:38:21.571801+0000 (oldest deadline 2024-03-31T23:38:43.871904+0000) 2024-03-31T23:39:21.362 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 31 23:39:21 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-9[67499]: 2024-03-31T23:39:21.053+0000 7fcc07615700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-31T23:38:27.074768+0000 front 2024-03-31T23:38:27.074793+0000 (oldest deadline 2024-03-31T23:38:52.374931+0000) 2024-03-31T23:39:21.362 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 31 23:39:21 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-9[67499]: 2024-03-31T23:39:21.053+0000 7fcc07615700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-03-31T23:38:34.075913+0000 front 2024-03-31T23:38:34.075868+0000 (oldest deadline 2024-03-31T23:38:59.376243+0000) 2024-03-31T23:39:21.362 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 31 23:39:21 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-9[67499]: 2024-03-31T23:39:21.053+0000 7fcc07615700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.142:6812 osd.4 since back 2024-03-31T23:38:45.177375+0000 front 2024-03-31T23:38:45.177457+0000 (oldest deadline 2024-03-31T23:39:11.077524+0000) 2024-03-31T23:39:21.362 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 31 23:39:21 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-9[67499]: 2024-03-31T23:39:21.053+0000 7fcc07615700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-03-31T23:38:53.978583+0000 front 2024-03-31T23:38:53.978472+0000 (oldest deadline 2024-03-31T23:39:16.878532+0000) 2024-03-31T23:39:21.557 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 31 23:39:21 smithi162 podman[78657]: 2024-03-31 23:39:21.202220904 +0000 UTC m=+0.022281777 image pull ec18d7baa60e8b1a0b126fbe5306acce11b1ed0ec5e3bcdddbf39b0a3c591791 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175 2024-03-31T23:39:21.557 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 31 23:39:21 smithi162 podman[78657]: 2024-03-31 23:39:21.336996518 +0000 UTC m=+0.157057368 container create 4b1706fe7825c21d6543d1458f192fd5cfd63c3efe7169cb6824d31f7d353ea2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-8-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-384f31f, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240326, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2) 2024-03-31T23:39:21.557 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 31 23:39:21 smithi162 podman[78657]: 2024-03-31 23:39:21.556713571 +0000 UTC m=+0.376774433 container init 4b1706fe7825c21d6543d1458f192fd5cfd63c3efe7169cb6824d31f7d353ea2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-8-deactivate, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240326, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_CLEAN=True, RELEASE=squid-384f31f, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-31T23:39:21.693 INFO:journalctl@ceph.osd.10.smithi142.stdout:Mar 31 23:39:21 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-10[60116]: 2024-03-31T23:39:21.332+0000 7f64c04a8700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-31T23:38:13.357804+0000 front 2024-03-31T23:38:13.357906+0000 (oldest deadline 2024-03-31T23:38:38.057730+0000) 2024-03-31T23:39:21.693 INFO:journalctl@ceph.osd.10.smithi142.stdout:Mar 31 23:39:21 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-10[60116]: 2024-03-31T23:39:21.332+0000 7f64c04a8700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.142:6804 osd.1 since back 2024-03-31T23:38:20.958341+0000 front 2024-03-31T23:38:20.958408+0000 (oldest deadline 2024-03-31T23:38:45.058481+0000) 2024-03-31T23:39:21.693 INFO:journalctl@ceph.osd.10.smithi142.stdout:Mar 31 23:39:21 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-10[60116]: 2024-03-31T23:39:21.332+0000 7f64c04a8700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-31T23:38:25.559393+0000 front 2024-03-31T23:38:25.559313+0000 (oldest deadline 2024-03-31T23:38:51.459398+0000) 2024-03-31T23:39:21.693 INFO:journalctl@ceph.osd.10.smithi142.stdout:Mar 31 23:39:21 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-10[60116]: 2024-03-31T23:39:21.332+0000 7f64c04a8700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-03-31T23:38:36.660831+0000 front 2024-03-31T23:38:36.660954+0000 (oldest deadline 2024-03-31T23:39:00.760714+0000) 2024-03-31T23:39:21.694 INFO:journalctl@ceph.osd.10.smithi142.stdout:Mar 31 23:39:21 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-10[60116]: 2024-03-31T23:39:21.332+0000 7f64c04a8700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.142:6812 osd.4 since back 2024-03-31T23:38:42.361714+0000 front 2024-03-31T23:38:42.361749+0000 (oldest deadline 2024-03-31T23:39:08.261872+0000) 2024-03-31T23:39:21.694 INFO:journalctl@ceph.osd.10.smithi142.stdout:Mar 31 23:39:21 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-10[60116]: 2024-03-31T23:39:21.332+0000 7f64c04a8700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-03-31T23:38:52.962747+0000 front 2024-03-31T23:38:52.962731+0000 (oldest deadline 2024-03-31T23:39:17.662721+0000) 2024-03-31T23:39:21.862 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:39:21 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:39:21.779+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-31T23:38:12.221366+0000 front 2024-03-31T23:38:12.221347+0000 (oldest deadline 2024-03-31T23:38:37.521357+0000) 2024-03-31T23:39:21.862 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:39:21 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:39:21.779+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.142:6804 osd.1 since back 2024-03-31T23:38:22.222192+0000 front 2024-03-31T23:38:22.222213+0000 (oldest deadline 2024-03-31T23:38:45.722340+0000) 2024-03-31T23:39:21.862 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:39:21 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:39:21.779+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-31T23:38:29.723616+0000 front 2024-03-31T23:38:29.723593+0000 (oldest deadline 2024-03-31T23:38:52.023779+0000) 2024-03-31T23:39:21.863 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:39:21 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:39:21.779+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-03-31T23:38:36.124750+0000 front 2024-03-31T23:38:36.124868+0000 (oldest deadline 2024-03-31T23:39:00.824619+0000) 2024-03-31T23:39:21.863 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:39:21 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:39:21.779+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.142:6812 osd.4 since back 2024-03-31T23:38:44.825661+0000 front 2024-03-31T23:38:44.825679+0000 (oldest deadline 2024-03-31T23:39:10.125827+0000) 2024-03-31T23:39:21.863 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:39:21 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:39:21.779+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-03-31T23:38:51.226778+0000 front 2024-03-31T23:38:51.226798+0000 (oldest deadline 2024-03-31T23:39:15.926915+0000) 2024-03-31T23:39:21.863 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 31 23:39:21 smithi162 podman[78657]: 2024-03-31 23:39:21.560350083 +0000 UTC m=+0.380410932 container start 4b1706fe7825c21d6543d1458f192fd5cfd63c3efe7169cb6824d31f7d353ea2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-8-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=squid-384f31f, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240326, maintainer=Guillaume Abrioux , ceph=True) 2024-03-31T23:39:21.863 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 31 23:39:21 smithi162 podman[78657]: 2024-03-31 23:39:21.656338594 +0000 UTC m=+0.476399463 container attach 4b1706fe7825c21d6543d1458f192fd5cfd63c3efe7169cb6824d31f7d353ea2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-8-deactivate, org.label-schema.build-date=20240326, maintainer=Guillaume Abrioux , RELEASE=squid-384f31f, ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, io.buildah.version=1.33.5) 2024-03-31T23:39:21.863 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 31 23:39:21 smithi162 podman[78657]: 2024-03-31 23:39:21.862762206 +0000 UTC m=+0.682823079 container died 4b1706fe7825c21d6543d1458f192fd5cfd63c3efe7169cb6824d31f7d353ea2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-8-deactivate, maintainer=Guillaume Abrioux , ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240326, io.buildah.version=1.33.5, RELEASE=squid-384f31f, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0) 2024-03-31T23:39:22.361 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 31 23:39:22 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-9[67499]: 2024-03-31T23:39:22.073+0000 7fcc07615700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-31T23:38:10.469941+0000 front 2024-03-31T23:38:10.470388+0000 (oldest deadline 2024-03-31T23:38:35.170101+0000) 2024-03-31T23:39:22.361 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 31 23:39:22 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-9[67499]: 2024-03-31T23:39:22.073+0000 7fcc07615700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.142:6804 osd.1 since back 2024-03-31T23:38:21.571770+0000 front 2024-03-31T23:38:21.571801+0000 (oldest deadline 2024-03-31T23:38:43.871904+0000) 2024-03-31T23:39:22.361 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 31 23:39:22 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-9[67499]: 2024-03-31T23:39:22.073+0000 7fcc07615700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-31T23:38:27.074768+0000 front 2024-03-31T23:38:27.074793+0000 (oldest deadline 2024-03-31T23:38:52.374931+0000) 2024-03-31T23:39:22.362 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 31 23:39:22 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-9[67499]: 2024-03-31T23:39:22.073+0000 7fcc07615700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-03-31T23:38:34.075913+0000 front 2024-03-31T23:38:34.075868+0000 (oldest deadline 2024-03-31T23:38:59.376243+0000) 2024-03-31T23:39:22.362 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 31 23:39:22 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-9[67499]: 2024-03-31T23:39:22.073+0000 7fcc07615700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.142:6812 osd.4 since back 2024-03-31T23:38:45.177375+0000 front 2024-03-31T23:38:45.177457+0000 (oldest deadline 2024-03-31T23:39:11.077524+0000) 2024-03-31T23:39:22.362 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 31 23:39:22 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-9[67499]: 2024-03-31T23:39:22.073+0000 7fcc07615700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-03-31T23:38:53.978583+0000 front 2024-03-31T23:38:53.978472+0000 (oldest deadline 2024-03-31T23:39:16.878532+0000) 2024-03-31T23:39:22.509 DEBUG:teuthology.orchestra.run.smithi162:> sudo pkill -f 'journalctl -f -n 0 -u ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4@osd.8.service' 2024-03-31T23:39:22.591 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-31T23:39:22.591 INFO:tasks.cephadm.osd.8:Stopped osd.8 2024-03-31T23:39:22.591 INFO:tasks.cephadm.osd.9:Stopping osd.9... 2024-03-31T23:39:22.591 DEBUG:teuthology.orchestra.run.smithi046:> sudo systemctl stop ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4@osd.9 2024-03-31T23:39:22.693 INFO:journalctl@ceph.osd.10.smithi142.stdout:Mar 31 23:39:22 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-10[60116]: 2024-03-31T23:39:22.292+0000 7f64c04a8700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-31T23:38:13.357804+0000 front 2024-03-31T23:38:13.357906+0000 (oldest deadline 2024-03-31T23:38:38.057730+0000) 2024-03-31T23:39:22.693 INFO:journalctl@ceph.osd.10.smithi142.stdout:Mar 31 23:39:22 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-10[60116]: 2024-03-31T23:39:22.292+0000 7f64c04a8700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.142:6804 osd.1 since back 2024-03-31T23:38:20.958341+0000 front 2024-03-31T23:38:20.958408+0000 (oldest deadline 2024-03-31T23:38:45.058481+0000) 2024-03-31T23:39:22.693 INFO:journalctl@ceph.osd.10.smithi142.stdout:Mar 31 23:39:22 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-10[60116]: 2024-03-31T23:39:22.292+0000 7f64c04a8700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-31T23:38:25.559393+0000 front 2024-03-31T23:38:25.559313+0000 (oldest deadline 2024-03-31T23:38:51.459398+0000) 2024-03-31T23:39:22.693 INFO:journalctl@ceph.osd.10.smithi142.stdout:Mar 31 23:39:22 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-10[60116]: 2024-03-31T23:39:22.292+0000 7f64c04a8700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-03-31T23:38:36.660831+0000 front 2024-03-31T23:38:36.660954+0000 (oldest deadline 2024-03-31T23:39:00.760714+0000) 2024-03-31T23:39:22.693 INFO:journalctl@ceph.osd.10.smithi142.stdout:Mar 31 23:39:22 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-10[60116]: 2024-03-31T23:39:22.292+0000 7f64c04a8700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.142:6812 osd.4 since back 2024-03-31T23:38:42.361714+0000 front 2024-03-31T23:38:42.361749+0000 (oldest deadline 2024-03-31T23:39:08.261872+0000) 2024-03-31T23:39:22.693 INFO:journalctl@ceph.osd.10.smithi142.stdout:Mar 31 23:39:22 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-10[60116]: 2024-03-31T23:39:22.292+0000 7f64c04a8700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-03-31T23:38:52.962747+0000 front 2024-03-31T23:38:52.962731+0000 (oldest deadline 2024-03-31T23:39:17.662721+0000) 2024-03-31T23:39:22.909 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 31 23:39:22 smithi046 systemd[1]: Stopping Ceph osd.9 for 6a9d8f08-efb2-11ee-b647-cb9ed24678a4... 2024-03-31T23:39:23.045 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:39:22 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:39:22.775+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-31T23:38:12.221366+0000 front 2024-03-31T23:38:12.221347+0000 (oldest deadline 2024-03-31T23:38:37.521357+0000) 2024-03-31T23:39:23.046 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:39:22 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:39:22.775+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.142:6804 osd.1 since back 2024-03-31T23:38:22.222192+0000 front 2024-03-31T23:38:22.222213+0000 (oldest deadline 2024-03-31T23:38:45.722340+0000) 2024-03-31T23:39:23.046 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:39:22 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:39:22.775+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-31T23:38:29.723616+0000 front 2024-03-31T23:38:29.723593+0000 (oldest deadline 2024-03-31T23:38:52.023779+0000) 2024-03-31T23:39:23.046 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:39:22 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:39:22.775+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-03-31T23:38:36.124750+0000 front 2024-03-31T23:38:36.124868+0000 (oldest deadline 2024-03-31T23:39:00.824619+0000) 2024-03-31T23:39:23.046 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:39:22 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:39:22.775+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.142:6812 osd.4 since back 2024-03-31T23:38:44.825661+0000 front 2024-03-31T23:38:44.825679+0000 (oldest deadline 2024-03-31T23:39:10.125827+0000) 2024-03-31T23:39:23.046 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:39:22 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:39:22.775+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-03-31T23:38:51.226778+0000 front 2024-03-31T23:38:51.226798+0000 (oldest deadline 2024-03-31T23:39:15.926915+0000) 2024-03-31T23:39:23.362 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 31 23:39:22 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-9[67499]: 2024-03-31T23:39:22.908+0000 7fcc0b021700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.9 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-31T23:39:23.362 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 31 23:39:22 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-9[67499]: 2024-03-31T23:39:22.908+0000 7fcc0b021700 -1 osd.9 139 *** Got signal Terminated *** 2024-03-31T23:39:23.362 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 31 23:39:22 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-9[67499]: 2024-03-31T23:39:22.908+0000 7fcc0b021700 -1 osd.9 139 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-31T23:39:23.362 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 31 23:39:23 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-9[67499]: 2024-03-31T23:39:23.074+0000 7fcc07615700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-31T23:38:10.469941+0000 front 2024-03-31T23:38:10.470388+0000 (oldest deadline 2024-03-31T23:38:35.170101+0000) 2024-03-31T23:39:23.363 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 31 23:39:23 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-9[67499]: 2024-03-31T23:39:23.074+0000 7fcc07615700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.142:6804 osd.1 since back 2024-03-31T23:38:21.571770+0000 front 2024-03-31T23:38:21.571801+0000 (oldest deadline 2024-03-31T23:38:43.871904+0000) 2024-03-31T23:39:23.363 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 31 23:39:23 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-9[67499]: 2024-03-31T23:39:23.074+0000 7fcc07615700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-31T23:38:27.074768+0000 front 2024-03-31T23:38:27.074793+0000 (oldest deadline 2024-03-31T23:38:52.374931+0000) 2024-03-31T23:39:23.363 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 31 23:39:23 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-9[67499]: 2024-03-31T23:39:23.074+0000 7fcc07615700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-03-31T23:38:34.075913+0000 front 2024-03-31T23:38:34.075868+0000 (oldest deadline 2024-03-31T23:38:59.376243+0000) 2024-03-31T23:39:23.363 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 31 23:39:23 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-9[67499]: 2024-03-31T23:39:23.074+0000 7fcc07615700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.142:6812 osd.4 since back 2024-03-31T23:38:45.177375+0000 front 2024-03-31T23:38:45.177457+0000 (oldest deadline 2024-03-31T23:39:11.077524+0000) 2024-03-31T23:39:23.363 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 31 23:39:23 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-9[67499]: 2024-03-31T23:39:23.074+0000 7fcc07615700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-03-31T23:38:53.978583+0000 front 2024-03-31T23:38:53.978472+0000 (oldest deadline 2024-03-31T23:39:16.878532+0000) 2024-03-31T23:39:23.594 INFO:journalctl@ceph.osd.10.smithi142.stdout:Mar 31 23:39:23 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-10[60116]: 2024-03-31T23:39:23.301+0000 7f64c04a8700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-31T23:38:13.357804+0000 front 2024-03-31T23:38:13.357906+0000 (oldest deadline 2024-03-31T23:38:38.057730+0000) 2024-03-31T23:39:23.594 INFO:journalctl@ceph.osd.10.smithi142.stdout:Mar 31 23:39:23 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-10[60116]: 2024-03-31T23:39:23.301+0000 7f64c04a8700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.142:6804 osd.1 since back 2024-03-31T23:38:20.958341+0000 front 2024-03-31T23:38:20.958408+0000 (oldest deadline 2024-03-31T23:38:45.058481+0000) 2024-03-31T23:39:23.594 INFO:journalctl@ceph.osd.10.smithi142.stdout:Mar 31 23:39:23 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-10[60116]: 2024-03-31T23:39:23.301+0000 7f64c04a8700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-31T23:38:25.559393+0000 front 2024-03-31T23:38:25.559313+0000 (oldest deadline 2024-03-31T23:38:51.459398+0000) 2024-03-31T23:39:23.594 INFO:journalctl@ceph.osd.10.smithi142.stdout:Mar 31 23:39:23 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-10[60116]: 2024-03-31T23:39:23.301+0000 7f64c04a8700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-03-31T23:38:36.660831+0000 front 2024-03-31T23:38:36.660954+0000 (oldest deadline 2024-03-31T23:39:00.760714+0000) 2024-03-31T23:39:23.594 INFO:journalctl@ceph.osd.10.smithi142.stdout:Mar 31 23:39:23 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-10[60116]: 2024-03-31T23:39:23.301+0000 7f64c04a8700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.142:6812 osd.4 since back 2024-03-31T23:38:42.361714+0000 front 2024-03-31T23:38:42.361749+0000 (oldest deadline 2024-03-31T23:39:08.261872+0000) 2024-03-31T23:39:23.594 INFO:journalctl@ceph.osd.10.smithi142.stdout:Mar 31 23:39:23 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-10[60116]: 2024-03-31T23:39:23.301+0000 7f64c04a8700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-03-31T23:38:52.962747+0000 front 2024-03-31T23:38:52.962731+0000 (oldest deadline 2024-03-31T23:39:17.662721+0000) 2024-03-31T23:39:24.046 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:39:23 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:39:23.801+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-31T23:38:12.221366+0000 front 2024-03-31T23:38:12.221347+0000 (oldest deadline 2024-03-31T23:38:37.521357+0000) 2024-03-31T23:39:24.046 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:39:23 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:39:23.801+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.142:6804 osd.1 since back 2024-03-31T23:38:22.222192+0000 front 2024-03-31T23:38:22.222213+0000 (oldest deadline 2024-03-31T23:38:45.722340+0000) 2024-03-31T23:39:24.046 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:39:23 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:39:23.801+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-31T23:38:29.723616+0000 front 2024-03-31T23:38:29.723593+0000 (oldest deadline 2024-03-31T23:38:52.023779+0000) 2024-03-31T23:39:24.046 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:39:23 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:39:23.801+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-03-31T23:38:36.124750+0000 front 2024-03-31T23:38:36.124868+0000 (oldest deadline 2024-03-31T23:39:00.824619+0000) 2024-03-31T23:39:24.046 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:39:23 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:39:23.801+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.142:6812 osd.4 since back 2024-03-31T23:38:44.825661+0000 front 2024-03-31T23:38:44.825679+0000 (oldest deadline 2024-03-31T23:39:10.125827+0000) 2024-03-31T23:39:24.046 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:39:23 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:39:23.801+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-03-31T23:38:51.226778+0000 front 2024-03-31T23:38:51.226798+0000 (oldest deadline 2024-03-31T23:39:15.926915+0000) 2024-03-31T23:39:24.317 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 31 23:39:24 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-9[67499]: 2024-03-31T23:39:24.026+0000 7fcc07615700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-31T23:38:10.469941+0000 front 2024-03-31T23:38:10.470388+0000 (oldest deadline 2024-03-31T23:38:35.170101+0000) 2024-03-31T23:39:24.317 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 31 23:39:24 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-9[67499]: 2024-03-31T23:39:24.026+0000 7fcc07615700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.142:6804 osd.1 since back 2024-03-31T23:38:21.571770+0000 front 2024-03-31T23:38:21.571801+0000 (oldest deadline 2024-03-31T23:38:43.871904+0000) 2024-03-31T23:39:24.317 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 31 23:39:24 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-9[67499]: 2024-03-31T23:39:24.026+0000 7fcc07615700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-31T23:38:27.074768+0000 front 2024-03-31T23:38:27.074793+0000 (oldest deadline 2024-03-31T23:38:52.374931+0000) 2024-03-31T23:39:24.317 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 31 23:39:24 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-9[67499]: 2024-03-31T23:39:24.026+0000 7fcc07615700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-03-31T23:38:34.075913+0000 front 2024-03-31T23:38:34.075868+0000 (oldest deadline 2024-03-31T23:38:59.376243+0000) 2024-03-31T23:39:24.318 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 31 23:39:24 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-9[67499]: 2024-03-31T23:39:24.026+0000 7fcc07615700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.142:6812 osd.4 since back 2024-03-31T23:38:45.177375+0000 front 2024-03-31T23:38:45.177457+0000 (oldest deadline 2024-03-31T23:39:11.077524+0000) 2024-03-31T23:39:24.318 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 31 23:39:24 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-9[67499]: 2024-03-31T23:39:24.027+0000 7fcc07615700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-03-31T23:38:53.978583+0000 front 2024-03-31T23:38:53.978472+0000 (oldest deadline 2024-03-31T23:39:16.878532+0000) 2024-03-31T23:39:24.606 INFO:journalctl@ceph.osd.10.smithi142.stdout:Mar 31 23:39:24 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-10[60116]: 2024-03-31T23:39:24.263+0000 7f64c04a8700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-31T23:38:13.357804+0000 front 2024-03-31T23:38:13.357906+0000 (oldest deadline 2024-03-31T23:38:38.057730+0000) 2024-03-31T23:39:24.606 INFO:journalctl@ceph.osd.10.smithi142.stdout:Mar 31 23:39:24 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-10[60116]: 2024-03-31T23:39:24.263+0000 7f64c04a8700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.142:6804 osd.1 since back 2024-03-31T23:38:20.958341+0000 front 2024-03-31T23:38:20.958408+0000 (oldest deadline 2024-03-31T23:38:45.058481+0000) 2024-03-31T23:39:24.606 INFO:journalctl@ceph.osd.10.smithi142.stdout:Mar 31 23:39:24 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-10[60116]: 2024-03-31T23:39:24.263+0000 7f64c04a8700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-31T23:38:25.559393+0000 front 2024-03-31T23:38:25.559313+0000 (oldest deadline 2024-03-31T23:38:51.459398+0000) 2024-03-31T23:39:24.606 INFO:journalctl@ceph.osd.10.smithi142.stdout:Mar 31 23:39:24 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-10[60116]: 2024-03-31T23:39:24.263+0000 7f64c04a8700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-03-31T23:38:36.660831+0000 front 2024-03-31T23:38:36.660954+0000 (oldest deadline 2024-03-31T23:39:00.760714+0000) 2024-03-31T23:39:24.606 INFO:journalctl@ceph.osd.10.smithi142.stdout:Mar 31 23:39:24 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-10[60116]: 2024-03-31T23:39:24.263+0000 7f64c04a8700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.142:6812 osd.4 since back 2024-03-31T23:38:42.361714+0000 front 2024-03-31T23:38:42.361749+0000 (oldest deadline 2024-03-31T23:39:08.261872+0000) 2024-03-31T23:39:24.606 INFO:journalctl@ceph.osd.10.smithi142.stdout:Mar 31 23:39:24 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-10[60116]: 2024-03-31T23:39:24.263+0000 7f64c04a8700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-03-31T23:38:52.962747+0000 front 2024-03-31T23:38:52.962731+0000 (oldest deadline 2024-03-31T23:39:17.662721+0000) 2024-03-31T23:39:25.295 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:39:24 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:39:24.848+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-31T23:38:12.221366+0000 front 2024-03-31T23:38:12.221347+0000 (oldest deadline 2024-03-31T23:38:37.521357+0000) 2024-03-31T23:39:25.295 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:39:24 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:39:24.848+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.142:6804 osd.1 since back 2024-03-31T23:38:22.222192+0000 front 2024-03-31T23:38:22.222213+0000 (oldest deadline 2024-03-31T23:38:45.722340+0000) 2024-03-31T23:39:25.295 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:39:24 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:39:24.848+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-31T23:38:29.723616+0000 front 2024-03-31T23:38:29.723593+0000 (oldest deadline 2024-03-31T23:38:52.023779+0000) 2024-03-31T23:39:25.296 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:39:24 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:39:24.848+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-03-31T23:38:36.124750+0000 front 2024-03-31T23:38:36.124868+0000 (oldest deadline 2024-03-31T23:39:00.824619+0000) 2024-03-31T23:39:25.296 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:39:24 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:39:24.848+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.142:6812 osd.4 since back 2024-03-31T23:38:44.825661+0000 front 2024-03-31T23:38:44.825679+0000 (oldest deadline 2024-03-31T23:39:10.125827+0000) 2024-03-31T23:39:25.296 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:39:24 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:39:24.848+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-03-31T23:38:51.226778+0000 front 2024-03-31T23:38:51.226798+0000 (oldest deadline 2024-03-31T23:39:15.926915+0000) 2024-03-31T23:39:25.296 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:39:24 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:39:24.848+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.46:6822 osd.6 since back 2024-03-31T23:39:01.140495+0000 front 2024-03-31T23:39:01.140543+0000 (oldest deadline 2024-03-31T23:39:24.640339+0000) 2024-03-31T23:39:25.361 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 31 23:39:24 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-9[67499]: 2024-03-31T23:39:24.993+0000 7fcc07615700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-31T23:38:10.469941+0000 front 2024-03-31T23:38:10.470388+0000 (oldest deadline 2024-03-31T23:38:35.170101+0000) 2024-03-31T23:39:25.362 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 31 23:39:24 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-9[67499]: 2024-03-31T23:39:24.993+0000 7fcc07615700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.142:6804 osd.1 since back 2024-03-31T23:38:21.571770+0000 front 2024-03-31T23:38:21.571801+0000 (oldest deadline 2024-03-31T23:38:43.871904+0000) 2024-03-31T23:39:25.362 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 31 23:39:24 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-9[67499]: 2024-03-31T23:39:24.993+0000 7fcc07615700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-31T23:38:27.074768+0000 front 2024-03-31T23:38:27.074793+0000 (oldest deadline 2024-03-31T23:38:52.374931+0000) 2024-03-31T23:39:25.362 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 31 23:39:24 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-9[67499]: 2024-03-31T23:39:24.993+0000 7fcc07615700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-03-31T23:38:34.075913+0000 front 2024-03-31T23:38:34.075868+0000 (oldest deadline 2024-03-31T23:38:59.376243+0000) 2024-03-31T23:39:25.362 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 31 23:39:24 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-9[67499]: 2024-03-31T23:39:24.993+0000 7fcc07615700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.142:6812 osd.4 since back 2024-03-31T23:38:45.177375+0000 front 2024-03-31T23:38:45.177457+0000 (oldest deadline 2024-03-31T23:39:11.077524+0000) 2024-03-31T23:39:25.362 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 31 23:39:24 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-9[67499]: 2024-03-31T23:39:24.993+0000 7fcc07615700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-03-31T23:38:53.978583+0000 front 2024-03-31T23:38:53.978472+0000 (oldest deadline 2024-03-31T23:39:16.878532+0000) 2024-03-31T23:39:25.693 INFO:journalctl@ceph.osd.10.smithi142.stdout:Mar 31 23:39:25 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-10[60116]: 2024-03-31T23:39:25.249+0000 7f64c04a8700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-31T23:38:13.357804+0000 front 2024-03-31T23:38:13.357906+0000 (oldest deadline 2024-03-31T23:38:38.057730+0000) 2024-03-31T23:39:25.693 INFO:journalctl@ceph.osd.10.smithi142.stdout:Mar 31 23:39:25 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-10[60116]: 2024-03-31T23:39:25.249+0000 7f64c04a8700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.142:6804 osd.1 since back 2024-03-31T23:38:20.958341+0000 front 2024-03-31T23:38:20.958408+0000 (oldest deadline 2024-03-31T23:38:45.058481+0000) 2024-03-31T23:39:25.693 INFO:journalctl@ceph.osd.10.smithi142.stdout:Mar 31 23:39:25 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-10[60116]: 2024-03-31T23:39:25.249+0000 7f64c04a8700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-31T23:38:25.559393+0000 front 2024-03-31T23:38:25.559313+0000 (oldest deadline 2024-03-31T23:38:51.459398+0000) 2024-03-31T23:39:25.693 INFO:journalctl@ceph.osd.10.smithi142.stdout:Mar 31 23:39:25 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-10[60116]: 2024-03-31T23:39:25.249+0000 7f64c04a8700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-03-31T23:38:36.660831+0000 front 2024-03-31T23:38:36.660954+0000 (oldest deadline 2024-03-31T23:39:00.760714+0000) 2024-03-31T23:39:25.693 INFO:journalctl@ceph.osd.10.smithi142.stdout:Mar 31 23:39:25 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-10[60116]: 2024-03-31T23:39:25.249+0000 7f64c04a8700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.142:6812 osd.4 since back 2024-03-31T23:38:42.361714+0000 front 2024-03-31T23:38:42.361749+0000 (oldest deadline 2024-03-31T23:39:08.261872+0000) 2024-03-31T23:39:25.694 INFO:journalctl@ceph.osd.10.smithi142.stdout:Mar 31 23:39:25 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-10[60116]: 2024-03-31T23:39:25.249+0000 7f64c04a8700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-03-31T23:38:52.962747+0000 front 2024-03-31T23:38:52.962731+0000 (oldest deadline 2024-03-31T23:39:17.662721+0000) 2024-03-31T23:39:26.197 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:39:25 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:39:25.890+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-31T23:38:12.221366+0000 front 2024-03-31T23:38:12.221347+0000 (oldest deadline 2024-03-31T23:38:37.521357+0000) 2024-03-31T23:39:26.197 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:39:25 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:39:25.890+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.142:6804 osd.1 since back 2024-03-31T23:38:22.222192+0000 front 2024-03-31T23:38:22.222213+0000 (oldest deadline 2024-03-31T23:38:45.722340+0000) 2024-03-31T23:39:26.197 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:39:25 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:39:25.890+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-31T23:38:29.723616+0000 front 2024-03-31T23:38:29.723593+0000 (oldest deadline 2024-03-31T23:38:52.023779+0000) 2024-03-31T23:39:26.197 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:39:25 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:39:25.890+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-03-31T23:38:36.124750+0000 front 2024-03-31T23:38:36.124868+0000 (oldest deadline 2024-03-31T23:39:00.824619+0000) 2024-03-31T23:39:26.197 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:39:25 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:39:25.890+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.142:6812 osd.4 since back 2024-03-31T23:38:44.825661+0000 front 2024-03-31T23:38:44.825679+0000 (oldest deadline 2024-03-31T23:39:10.125827+0000) 2024-03-31T23:39:26.197 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:39:25 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:39:25.890+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-03-31T23:38:51.226778+0000 front 2024-03-31T23:38:51.226798+0000 (oldest deadline 2024-03-31T23:39:15.926915+0000) 2024-03-31T23:39:26.197 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:39:25 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:39:25.890+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.46:6822 osd.6 since back 2024-03-31T23:39:01.140495+0000 front 2024-03-31T23:39:01.140543+0000 (oldest deadline 2024-03-31T23:39:24.640339+0000) 2024-03-31T23:39:26.361 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 31 23:39:26 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-9[67499]: 2024-03-31T23:39:26.028+0000 7fcc07615700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-31T23:38:10.469941+0000 front 2024-03-31T23:38:10.470388+0000 (oldest deadline 2024-03-31T23:38:35.170101+0000) 2024-03-31T23:39:26.362 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 31 23:39:26 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-9[67499]: 2024-03-31T23:39:26.028+0000 7fcc07615700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.142:6804 osd.1 since back 2024-03-31T23:38:21.571770+0000 front 2024-03-31T23:38:21.571801+0000 (oldest deadline 2024-03-31T23:38:43.871904+0000) 2024-03-31T23:39:26.362 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 31 23:39:26 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-9[67499]: 2024-03-31T23:39:26.028+0000 7fcc07615700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-31T23:38:27.074768+0000 front 2024-03-31T23:38:27.074793+0000 (oldest deadline 2024-03-31T23:38:52.374931+0000) 2024-03-31T23:39:26.362 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 31 23:39:26 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-9[67499]: 2024-03-31T23:39:26.028+0000 7fcc07615700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-03-31T23:38:34.075913+0000 front 2024-03-31T23:38:34.075868+0000 (oldest deadline 2024-03-31T23:38:59.376243+0000) 2024-03-31T23:39:26.362 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 31 23:39:26 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-9[67499]: 2024-03-31T23:39:26.028+0000 7fcc07615700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.142:6812 osd.4 since back 2024-03-31T23:38:45.177375+0000 front 2024-03-31T23:38:45.177457+0000 (oldest deadline 2024-03-31T23:39:11.077524+0000) 2024-03-31T23:39:26.362 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 31 23:39:26 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-9[67499]: 2024-03-31T23:39:26.028+0000 7fcc07615700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-03-31T23:38:53.978583+0000 front 2024-03-31T23:38:53.978472+0000 (oldest deadline 2024-03-31T23:39:16.878532+0000) 2024-03-31T23:39:26.693 INFO:journalctl@ceph.osd.10.smithi142.stdout:Mar 31 23:39:26 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-10[60116]: 2024-03-31T23:39:26.232+0000 7f64c04a8700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-31T23:38:13.357804+0000 front 2024-03-31T23:38:13.357906+0000 (oldest deadline 2024-03-31T23:38:38.057730+0000) 2024-03-31T23:39:26.693 INFO:journalctl@ceph.osd.10.smithi142.stdout:Mar 31 23:39:26 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-10[60116]: 2024-03-31T23:39:26.232+0000 7f64c04a8700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.142:6804 osd.1 since back 2024-03-31T23:38:20.958341+0000 front 2024-03-31T23:38:20.958408+0000 (oldest deadline 2024-03-31T23:38:45.058481+0000) 2024-03-31T23:39:26.693 INFO:journalctl@ceph.osd.10.smithi142.stdout:Mar 31 23:39:26 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-10[60116]: 2024-03-31T23:39:26.232+0000 7f64c04a8700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-31T23:38:25.559393+0000 front 2024-03-31T23:38:25.559313+0000 (oldest deadline 2024-03-31T23:38:51.459398+0000) 2024-03-31T23:39:26.693 INFO:journalctl@ceph.osd.10.smithi142.stdout:Mar 31 23:39:26 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-10[60116]: 2024-03-31T23:39:26.232+0000 7f64c04a8700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-03-31T23:38:36.660831+0000 front 2024-03-31T23:38:36.660954+0000 (oldest deadline 2024-03-31T23:39:00.760714+0000) 2024-03-31T23:39:26.693 INFO:journalctl@ceph.osd.10.smithi142.stdout:Mar 31 23:39:26 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-10[60116]: 2024-03-31T23:39:26.232+0000 7f64c04a8700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.142:6812 osd.4 since back 2024-03-31T23:38:42.361714+0000 front 2024-03-31T23:38:42.361749+0000 (oldest deadline 2024-03-31T23:39:08.261872+0000) 2024-03-31T23:39:26.693 INFO:journalctl@ceph.osd.10.smithi142.stdout:Mar 31 23:39:26 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-10[60116]: 2024-03-31T23:39:26.232+0000 7f64c04a8700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-03-31T23:38:52.962747+0000 front 2024-03-31T23:38:52.962731+0000 (oldest deadline 2024-03-31T23:39:17.662721+0000) 2024-03-31T23:39:26.693 INFO:journalctl@ceph.osd.10.smithi142.stdout:Mar 31 23:39:26 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-10[60116]: 2024-03-31T23:39:26.232+0000 7f64c04a8700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.46:6822 osd.6 since back 2024-03-31T23:39:00.964464+0000 front 2024-03-31T23:39:00.964397+0000 (oldest deadline 2024-03-31T23:39:25.564525+0000) 2024-03-31T23:39:27.296 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:39:26 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:39:26.914+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-31T23:38:12.221366+0000 front 2024-03-31T23:38:12.221347+0000 (oldest deadline 2024-03-31T23:38:37.521357+0000) 2024-03-31T23:39:27.296 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:39:26 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:39:26.914+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.142:6804 osd.1 since back 2024-03-31T23:38:22.222192+0000 front 2024-03-31T23:38:22.222213+0000 (oldest deadline 2024-03-31T23:38:45.722340+0000) 2024-03-31T23:39:27.296 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:39:26 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:39:26.914+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-31T23:38:29.723616+0000 front 2024-03-31T23:38:29.723593+0000 (oldest deadline 2024-03-31T23:38:52.023779+0000) 2024-03-31T23:39:27.296 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:39:26 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:39:26.914+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-03-31T23:38:36.124750+0000 front 2024-03-31T23:38:36.124868+0000 (oldest deadline 2024-03-31T23:39:00.824619+0000) 2024-03-31T23:39:27.296 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:39:26 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:39:26.914+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.142:6812 osd.4 since back 2024-03-31T23:38:44.825661+0000 front 2024-03-31T23:38:44.825679+0000 (oldest deadline 2024-03-31T23:39:10.125827+0000) 2024-03-31T23:39:27.296 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:39:26 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:39:26.914+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-03-31T23:38:51.226778+0000 front 2024-03-31T23:38:51.226798+0000 (oldest deadline 2024-03-31T23:39:15.926915+0000) 2024-03-31T23:39:27.296 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:39:26 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:39:26.914+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.46:6822 osd.6 since back 2024-03-31T23:39:01.140495+0000 front 2024-03-31T23:39:01.140543+0000 (oldest deadline 2024-03-31T23:39:24.640339+0000) 2024-03-31T23:39:27.362 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 31 23:39:27 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-9[67499]: 2024-03-31T23:39:27.071+0000 7fcc07615700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-31T23:38:10.469941+0000 front 2024-03-31T23:38:10.470388+0000 (oldest deadline 2024-03-31T23:38:35.170101+0000) 2024-03-31T23:39:27.362 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 31 23:39:27 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-9[67499]: 2024-03-31T23:39:27.071+0000 7fcc07615700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.142:6804 osd.1 since back 2024-03-31T23:38:21.571770+0000 front 2024-03-31T23:38:21.571801+0000 (oldest deadline 2024-03-31T23:38:43.871904+0000) 2024-03-31T23:39:27.362 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 31 23:39:27 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-9[67499]: 2024-03-31T23:39:27.071+0000 7fcc07615700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-31T23:38:27.074768+0000 front 2024-03-31T23:38:27.074793+0000 (oldest deadline 2024-03-31T23:38:52.374931+0000) 2024-03-31T23:39:27.362 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 31 23:39:27 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-9[67499]: 2024-03-31T23:39:27.071+0000 7fcc07615700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-03-31T23:38:34.075913+0000 front 2024-03-31T23:38:34.075868+0000 (oldest deadline 2024-03-31T23:38:59.376243+0000) 2024-03-31T23:39:27.362 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 31 23:39:27 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-9[67499]: 2024-03-31T23:39:27.071+0000 7fcc07615700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.142:6812 osd.4 since back 2024-03-31T23:38:45.177375+0000 front 2024-03-31T23:38:45.177457+0000 (oldest deadline 2024-03-31T23:39:11.077524+0000) 2024-03-31T23:39:27.363 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 31 23:39:27 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-9[67499]: 2024-03-31T23:39:27.071+0000 7fcc07615700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-03-31T23:38:53.978583+0000 front 2024-03-31T23:38:53.978472+0000 (oldest deadline 2024-03-31T23:39:16.878532+0000) 2024-03-31T23:39:27.693 INFO:journalctl@ceph.osd.10.smithi142.stdout:Mar 31 23:39:27 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-10[60116]: 2024-03-31T23:39:27.237+0000 7f64c04a8700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-31T23:38:13.357804+0000 front 2024-03-31T23:38:13.357906+0000 (oldest deadline 2024-03-31T23:38:38.057730+0000) 2024-03-31T23:39:27.693 INFO:journalctl@ceph.osd.10.smithi142.stdout:Mar 31 23:39:27 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-10[60116]: 2024-03-31T23:39:27.237+0000 7f64c04a8700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.142:6804 osd.1 since back 2024-03-31T23:38:20.958341+0000 front 2024-03-31T23:38:20.958408+0000 (oldest deadline 2024-03-31T23:38:45.058481+0000) 2024-03-31T23:39:27.693 INFO:journalctl@ceph.osd.10.smithi142.stdout:Mar 31 23:39:27 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-10[60116]: 2024-03-31T23:39:27.237+0000 7f64c04a8700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-31T23:38:25.559393+0000 front 2024-03-31T23:38:25.559313+0000 (oldest deadline 2024-03-31T23:38:51.459398+0000) 2024-03-31T23:39:27.693 INFO:journalctl@ceph.osd.10.smithi142.stdout:Mar 31 23:39:27 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-10[60116]: 2024-03-31T23:39:27.237+0000 7f64c04a8700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-03-31T23:38:36.660831+0000 front 2024-03-31T23:38:36.660954+0000 (oldest deadline 2024-03-31T23:39:00.760714+0000) 2024-03-31T23:39:27.693 INFO:journalctl@ceph.osd.10.smithi142.stdout:Mar 31 23:39:27 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-10[60116]: 2024-03-31T23:39:27.237+0000 7f64c04a8700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.142:6812 osd.4 since back 2024-03-31T23:38:42.361714+0000 front 2024-03-31T23:38:42.361749+0000 (oldest deadline 2024-03-31T23:39:08.261872+0000) 2024-03-31T23:39:27.693 INFO:journalctl@ceph.osd.10.smithi142.stdout:Mar 31 23:39:27 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-10[60116]: 2024-03-31T23:39:27.237+0000 7f64c04a8700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-03-31T23:38:52.962747+0000 front 2024-03-31T23:38:52.962731+0000 (oldest deadline 2024-03-31T23:39:17.662721+0000) 2024-03-31T23:39:27.693 INFO:journalctl@ceph.osd.10.smithi142.stdout:Mar 31 23:39:27 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-10[60116]: 2024-03-31T23:39:27.237+0000 7f64c04a8700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.46:6822 osd.6 since back 2024-03-31T23:39:00.964464+0000 front 2024-03-31T23:39:00.964397+0000 (oldest deadline 2024-03-31T23:39:25.564525+0000) 2024-03-31T23:39:28.295 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:39:27 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:39:27.936+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-31T23:38:12.221366+0000 front 2024-03-31T23:38:12.221347+0000 (oldest deadline 2024-03-31T23:38:37.521357+0000) 2024-03-31T23:39:28.296 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:39:27 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:39:27.936+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.142:6804 osd.1 since back 2024-03-31T23:38:22.222192+0000 front 2024-03-31T23:38:22.222213+0000 (oldest deadline 2024-03-31T23:38:45.722340+0000) 2024-03-31T23:39:28.296 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:39:27 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:39:27.936+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-31T23:38:29.723616+0000 front 2024-03-31T23:38:29.723593+0000 (oldest deadline 2024-03-31T23:38:52.023779+0000) 2024-03-31T23:39:28.296 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:39:27 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:39:27.936+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-03-31T23:38:36.124750+0000 front 2024-03-31T23:38:36.124868+0000 (oldest deadline 2024-03-31T23:39:00.824619+0000) 2024-03-31T23:39:28.296 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:39:27 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:39:27.936+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.142:6812 osd.4 since back 2024-03-31T23:38:44.825661+0000 front 2024-03-31T23:38:44.825679+0000 (oldest deadline 2024-03-31T23:39:10.125827+0000) 2024-03-31T23:39:28.296 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:39:27 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:39:27.936+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-03-31T23:38:51.226778+0000 front 2024-03-31T23:38:51.226798+0000 (oldest deadline 2024-03-31T23:39:15.926915+0000) 2024-03-31T23:39:28.296 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:39:27 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:39:27.936+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.46:6822 osd.6 since back 2024-03-31T23:39:01.140495+0000 front 2024-03-31T23:39:01.140543+0000 (oldest deadline 2024-03-31T23:39:24.640339+0000) 2024-03-31T23:39:28.693 INFO:journalctl@ceph.osd.10.smithi142.stdout:Mar 31 23:39:28 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-10[60116]: 2024-03-31T23:39:28.264+0000 7f64c04a8700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-31T23:38:13.357804+0000 front 2024-03-31T23:38:13.357906+0000 (oldest deadline 2024-03-31T23:38:38.057730+0000) 2024-03-31T23:39:28.693 INFO:journalctl@ceph.osd.10.smithi142.stdout:Mar 31 23:39:28 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-10[60116]: 2024-03-31T23:39:28.264+0000 7f64c04a8700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.142:6804 osd.1 since back 2024-03-31T23:38:20.958341+0000 front 2024-03-31T23:38:20.958408+0000 (oldest deadline 2024-03-31T23:38:45.058481+0000) 2024-03-31T23:39:28.693 INFO:journalctl@ceph.osd.10.smithi142.stdout:Mar 31 23:39:28 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-10[60116]: 2024-03-31T23:39:28.264+0000 7f64c04a8700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-31T23:38:25.559393+0000 front 2024-03-31T23:38:25.559313+0000 (oldest deadline 2024-03-31T23:38:51.459398+0000) 2024-03-31T23:39:28.693 INFO:journalctl@ceph.osd.10.smithi142.stdout:Mar 31 23:39:28 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-10[60116]: 2024-03-31T23:39:28.264+0000 7f64c04a8700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-03-31T23:38:36.660831+0000 front 2024-03-31T23:38:36.660954+0000 (oldest deadline 2024-03-31T23:39:00.760714+0000) 2024-03-31T23:39:28.693 INFO:journalctl@ceph.osd.10.smithi142.stdout:Mar 31 23:39:28 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-10[60116]: 2024-03-31T23:39:28.264+0000 7f64c04a8700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.142:6812 osd.4 since back 2024-03-31T23:38:42.361714+0000 front 2024-03-31T23:38:42.361749+0000 (oldest deadline 2024-03-31T23:39:08.261872+0000) 2024-03-31T23:39:28.693 INFO:journalctl@ceph.osd.10.smithi142.stdout:Mar 31 23:39:28 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-10[60116]: 2024-03-31T23:39:28.264+0000 7f64c04a8700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-03-31T23:38:52.962747+0000 front 2024-03-31T23:38:52.962731+0000 (oldest deadline 2024-03-31T23:39:17.662721+0000) 2024-03-31T23:39:28.693 INFO:journalctl@ceph.osd.10.smithi142.stdout:Mar 31 23:39:28 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-10[60116]: 2024-03-31T23:39:28.264+0000 7f64c04a8700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.46:6822 osd.6 since back 2024-03-31T23:39:00.964464+0000 front 2024-03-31T23:39:00.964397+0000 (oldest deadline 2024-03-31T23:39:25.564525+0000) 2024-03-31T23:39:28.767 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 31 23:39:28 smithi046 podman[108107]: 2024-03-31 23:39:28.511796476 +0000 UTC m=+5.717576675 container died fad960dfb69fa35f50e07e2a2c40ef2ba9d2c0b82739e43804fcbf155df51ef6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-9, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, ceph=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240326, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=squid-384f31f, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-03-31T23:39:29.051 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 31 23:39:28 smithi046 podman[108107]: 2024-03-31 23:39:28.841126087 +0000 UTC m=+6.046906294 container cleanup fad960dfb69fa35f50e07e2a2c40ef2ba9d2c0b82739e43804fcbf155df51ef6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-9, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240326, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-384f31f, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5) 2024-03-31T23:39:29.051 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 31 23:39:28 smithi046 bash[108107]: ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-9 2024-03-31T23:39:29.295 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:39:28 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:39:28.906+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-31T23:38:12.221366+0000 front 2024-03-31T23:38:12.221347+0000 (oldest deadline 2024-03-31T23:38:37.521357+0000) 2024-03-31T23:39:29.295 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:39:28 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:39:28.906+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.142:6804 osd.1 since back 2024-03-31T23:38:22.222192+0000 front 2024-03-31T23:38:22.222213+0000 (oldest deadline 2024-03-31T23:38:45.722340+0000) 2024-03-31T23:39:29.295 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:39:28 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:39:28.906+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-31T23:38:29.723616+0000 front 2024-03-31T23:38:29.723593+0000 (oldest deadline 2024-03-31T23:38:52.023779+0000) 2024-03-31T23:39:29.296 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:39:28 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:39:28.906+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-03-31T23:38:36.124750+0000 front 2024-03-31T23:38:36.124868+0000 (oldest deadline 2024-03-31T23:39:00.824619+0000) 2024-03-31T23:39:29.296 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:39:28 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:39:28.906+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.142:6812 osd.4 since back 2024-03-31T23:38:44.825661+0000 front 2024-03-31T23:38:44.825679+0000 (oldest deadline 2024-03-31T23:39:10.125827+0000) 2024-03-31T23:39:29.296 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:39:28 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:39:28.906+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-03-31T23:38:51.226778+0000 front 2024-03-31T23:38:51.226798+0000 (oldest deadline 2024-03-31T23:39:15.926915+0000) 2024-03-31T23:39:29.296 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:39:28 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:39:28.906+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.46:6822 osd.6 since back 2024-03-31T23:39:01.140495+0000 front 2024-03-31T23:39:01.140543+0000 (oldest deadline 2024-03-31T23:39:24.640339+0000) 2024-03-31T23:39:29.361 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 31 23:39:29 smithi046 podman[108125]: 2024-03-31 23:39:29.050649249 +0000 UTC m=+0.533576629 container remove fad960dfb69fa35f50e07e2a2c40ef2ba9d2c0b82739e43804fcbf155df51ef6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-9, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, RELEASE=squid-384f31f, org.label-schema.build-date=20240326, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-03-31T23:39:29.693 INFO:journalctl@ceph.osd.10.smithi142.stdout:Mar 31 23:39:29 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-10[60116]: 2024-03-31T23:39:29.292+0000 7f64c04a8700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-31T23:38:13.357804+0000 front 2024-03-31T23:38:13.357906+0000 (oldest deadline 2024-03-31T23:38:38.057730+0000) 2024-03-31T23:39:29.693 INFO:journalctl@ceph.osd.10.smithi142.stdout:Mar 31 23:39:29 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-10[60116]: 2024-03-31T23:39:29.292+0000 7f64c04a8700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.142:6804 osd.1 since back 2024-03-31T23:38:20.958341+0000 front 2024-03-31T23:38:20.958408+0000 (oldest deadline 2024-03-31T23:38:45.058481+0000) 2024-03-31T23:39:29.693 INFO:journalctl@ceph.osd.10.smithi142.stdout:Mar 31 23:39:29 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-10[60116]: 2024-03-31T23:39:29.292+0000 7f64c04a8700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-31T23:38:25.559393+0000 front 2024-03-31T23:38:25.559313+0000 (oldest deadline 2024-03-31T23:38:51.459398+0000) 2024-03-31T23:39:29.693 INFO:journalctl@ceph.osd.10.smithi142.stdout:Mar 31 23:39:29 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-10[60116]: 2024-03-31T23:39:29.292+0000 7f64c04a8700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-03-31T23:38:36.660831+0000 front 2024-03-31T23:38:36.660954+0000 (oldest deadline 2024-03-31T23:39:00.760714+0000) 2024-03-31T23:39:29.693 INFO:journalctl@ceph.osd.10.smithi142.stdout:Mar 31 23:39:29 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-10[60116]: 2024-03-31T23:39:29.292+0000 7f64c04a8700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.142:6812 osd.4 since back 2024-03-31T23:38:42.361714+0000 front 2024-03-31T23:38:42.361749+0000 (oldest deadline 2024-03-31T23:39:08.261872+0000) 2024-03-31T23:39:29.693 INFO:journalctl@ceph.osd.10.smithi142.stdout:Mar 31 23:39:29 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-10[60116]: 2024-03-31T23:39:29.292+0000 7f64c04a8700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-03-31T23:38:52.962747+0000 front 2024-03-31T23:38:52.962731+0000 (oldest deadline 2024-03-31T23:39:17.662721+0000) 2024-03-31T23:39:29.693 INFO:journalctl@ceph.osd.10.smithi142.stdout:Mar 31 23:39:29 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-10[60116]: 2024-03-31T23:39:29.292+0000 7f64c04a8700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.46:6822 osd.6 since back 2024-03-31T23:39:00.964464+0000 front 2024-03-31T23:39:00.964397+0000 (oldest deadline 2024-03-31T23:39:25.564525+0000) 2024-03-31T23:39:29.729 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 31 23:39:29 smithi046 podman[108212]: 2024-03-31 23:39:29.367001719 +0000 UTC m=+0.017926641 image pull ec18d7baa60e8b1a0b126fbe5306acce11b1ed0ec5e3bcdddbf39b0a3c591791 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175 2024-03-31T23:39:29.729 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 31 23:39:29 smithi046 podman[108212]: 2024-03-31 23:39:29.476615017 +0000 UTC m=+0.127539916 container create 649ddc20247c04c3fb43c1e429e8e572d7b27489ff64ce579a58bac2c44b283e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-9-deactivate, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240326, RELEASE=squid-384f31f, org.label-schema.schema-version=1.0) 2024-03-31T23:39:29.729 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 31 23:39:29 smithi046 podman[108212]: 2024-03-31 23:39:29.729274139 +0000 UTC m=+0.380199045 container init 649ddc20247c04c3fb43c1e429e8e572d7b27489ff64ce579a58bac2c44b283e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-9-deactivate, org.label-schema.build-date=20240326, RELEASE=squid-384f31f, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, io.buildah.version=1.33.5, ceph=True, CEPH_POINT_RELEASE=) 2024-03-31T23:39:30.033 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 31 23:39:29 smithi046 podman[108212]: 2024-03-31 23:39:29.732356809 +0000 UTC m=+0.383281706 container start 649ddc20247c04c3fb43c1e429e8e572d7b27489ff64ce579a58bac2c44b283e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-9-deactivate, org.label-schema.build-date=20240326, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-384f31f, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux ) 2024-03-31T23:39:30.033 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 31 23:39:29 smithi046 podman[108212]: 2024-03-31 23:39:29.821003942 +0000 UTC m=+0.471928836 container attach 649ddc20247c04c3fb43c1e429e8e572d7b27489ff64ce579a58bac2c44b283e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-9-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, RELEASE=squid-384f31f, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240326, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=) 2024-03-31T23:39:30.295 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:39:29 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:39:29.949+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-31T23:38:12.221366+0000 front 2024-03-31T23:38:12.221347+0000 (oldest deadline 2024-03-31T23:38:37.521357+0000) 2024-03-31T23:39:30.296 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:39:29 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:39:29.949+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.142:6804 osd.1 since back 2024-03-31T23:38:22.222192+0000 front 2024-03-31T23:38:22.222213+0000 (oldest deadline 2024-03-31T23:38:45.722340+0000) 2024-03-31T23:39:30.296 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:39:29 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:39:29.949+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-31T23:38:29.723616+0000 front 2024-03-31T23:38:29.723593+0000 (oldest deadline 2024-03-31T23:38:52.023779+0000) 2024-03-31T23:39:30.296 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:39:29 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:39:29.949+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-03-31T23:38:36.124750+0000 front 2024-03-31T23:38:36.124868+0000 (oldest deadline 2024-03-31T23:39:00.824619+0000) 2024-03-31T23:39:30.296 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:39:29 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:39:29.949+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.142:6812 osd.4 since back 2024-03-31T23:38:44.825661+0000 front 2024-03-31T23:38:44.825679+0000 (oldest deadline 2024-03-31T23:39:10.125827+0000) 2024-03-31T23:39:30.296 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:39:29 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:39:29.949+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-03-31T23:38:51.226778+0000 front 2024-03-31T23:38:51.226798+0000 (oldest deadline 2024-03-31T23:39:15.926915+0000) 2024-03-31T23:39:30.296 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:39:29 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:39:29.949+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.46:6822 osd.6 since back 2024-03-31T23:39:01.140495+0000 front 2024-03-31T23:39:01.140543+0000 (oldest deadline 2024-03-31T23:39:24.640339+0000) 2024-03-31T23:39:30.361 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 31 23:39:30 smithi046 podman[108212]: 2024-03-31 23:39:30.033561406 +0000 UTC m=+0.684486304 container died 649ddc20247c04c3fb43c1e429e8e572d7b27489ff64ce579a58bac2c44b283e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-9-deactivate, RELEASE=squid-384f31f, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, io.buildah.version=1.33.5, org.label-schema.build-date=20240326, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-03-31T23:39:30.693 INFO:journalctl@ceph.osd.10.smithi142.stdout:Mar 31 23:39:30 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-10[60116]: 2024-03-31T23:39:30.277+0000 7f64c04a8700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-31T23:38:13.357804+0000 front 2024-03-31T23:38:13.357906+0000 (oldest deadline 2024-03-31T23:38:38.057730+0000) 2024-03-31T23:39:30.693 INFO:journalctl@ceph.osd.10.smithi142.stdout:Mar 31 23:39:30 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-10[60116]: 2024-03-31T23:39:30.277+0000 7f64c04a8700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.142:6804 osd.1 since back 2024-03-31T23:38:20.958341+0000 front 2024-03-31T23:38:20.958408+0000 (oldest deadline 2024-03-31T23:38:45.058481+0000) 2024-03-31T23:39:30.693 INFO:journalctl@ceph.osd.10.smithi142.stdout:Mar 31 23:39:30 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-10[60116]: 2024-03-31T23:39:30.277+0000 7f64c04a8700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-31T23:38:25.559393+0000 front 2024-03-31T23:38:25.559313+0000 (oldest deadline 2024-03-31T23:38:51.459398+0000) 2024-03-31T23:39:30.694 INFO:journalctl@ceph.osd.10.smithi142.stdout:Mar 31 23:39:30 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-10[60116]: 2024-03-31T23:39:30.277+0000 7f64c04a8700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-03-31T23:38:36.660831+0000 front 2024-03-31T23:38:36.660954+0000 (oldest deadline 2024-03-31T23:39:00.760714+0000) 2024-03-31T23:39:30.694 INFO:journalctl@ceph.osd.10.smithi142.stdout:Mar 31 23:39:30 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-10[60116]: 2024-03-31T23:39:30.277+0000 7f64c04a8700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.142:6812 osd.4 since back 2024-03-31T23:38:42.361714+0000 front 2024-03-31T23:38:42.361749+0000 (oldest deadline 2024-03-31T23:39:08.261872+0000) 2024-03-31T23:39:30.694 INFO:journalctl@ceph.osd.10.smithi142.stdout:Mar 31 23:39:30 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-10[60116]: 2024-03-31T23:39:30.277+0000 7f64c04a8700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-03-31T23:38:52.962747+0000 front 2024-03-31T23:38:52.962731+0000 (oldest deadline 2024-03-31T23:39:17.662721+0000) 2024-03-31T23:39:30.694 INFO:journalctl@ceph.osd.10.smithi142.stdout:Mar 31 23:39:30 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-10[60116]: 2024-03-31T23:39:30.277+0000 7f64c04a8700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.46:6822 osd.6 since back 2024-03-31T23:39:00.964464+0000 front 2024-03-31T23:39:00.964397+0000 (oldest deadline 2024-03-31T23:39:25.564525+0000) 2024-03-31T23:39:30.701 DEBUG:teuthology.orchestra.run.smithi046:> sudo pkill -f 'journalctl -f -n 0 -u ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4@osd.9.service' 2024-03-31T23:39:30.746 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-31T23:39:30.746 INFO:tasks.cephadm.osd.9:Stopped osd.9 2024-03-31T23:39:30.746 INFO:tasks.cephadm.osd.10:Stopping osd.10... 2024-03-31T23:39:30.746 DEBUG:teuthology.orchestra.run.smithi142:> sudo systemctl stop ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4@osd.10 2024-03-31T23:39:31.053 INFO:journalctl@ceph.osd.10.smithi142.stdout:Mar 31 23:39:30 smithi142 systemd[1]: Stopping Ceph osd.10 for 6a9d8f08-efb2-11ee-b647-cb9ed24678a4... 2024-03-31T23:39:31.295 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:39:30 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:39:30.914+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-31T23:38:12.221366+0000 front 2024-03-31T23:38:12.221347+0000 (oldest deadline 2024-03-31T23:38:37.521357+0000) 2024-03-31T23:39:31.296 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:39:30 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:39:30.914+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.142:6804 osd.1 since back 2024-03-31T23:38:22.222192+0000 front 2024-03-31T23:38:22.222213+0000 (oldest deadline 2024-03-31T23:38:45.722340+0000) 2024-03-31T23:39:31.296 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:39:30 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:39:30.914+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-31T23:38:29.723616+0000 front 2024-03-31T23:38:29.723593+0000 (oldest deadline 2024-03-31T23:38:52.023779+0000) 2024-03-31T23:39:31.296 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:39:30 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:39:30.914+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-03-31T23:38:36.124750+0000 front 2024-03-31T23:38:36.124868+0000 (oldest deadline 2024-03-31T23:39:00.824619+0000) 2024-03-31T23:39:31.296 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:39:30 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:39:30.915+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.142:6812 osd.4 since back 2024-03-31T23:38:44.825661+0000 front 2024-03-31T23:38:44.825679+0000 (oldest deadline 2024-03-31T23:39:10.125827+0000) 2024-03-31T23:39:31.296 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:39:30 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:39:30.915+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-03-31T23:38:51.226778+0000 front 2024-03-31T23:38:51.226798+0000 (oldest deadline 2024-03-31T23:39:15.926915+0000) 2024-03-31T23:39:31.296 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:39:30 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:39:30.915+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.46:6822 osd.6 since back 2024-03-31T23:39:01.140495+0000 front 2024-03-31T23:39:01.140543+0000 (oldest deadline 2024-03-31T23:39:24.640339+0000) 2024-03-31T23:39:31.443 INFO:journalctl@ceph.osd.10.smithi142.stdout:Mar 31 23:39:31 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-10[60116]: 2024-03-31T23:39:31.052+0000 7f64c3eb4700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.10 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-31T23:39:31.443 INFO:journalctl@ceph.osd.10.smithi142.stdout:Mar 31 23:39:31 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-10[60116]: 2024-03-31T23:39:31.052+0000 7f64c3eb4700 -1 osd.10 139 *** Got signal Terminated *** 2024-03-31T23:39:31.443 INFO:journalctl@ceph.osd.10.smithi142.stdout:Mar 31 23:39:31 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-10[60116]: 2024-03-31T23:39:31.052+0000 7f64c3eb4700 -1 osd.10 139 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-31T23:39:31.443 INFO:journalctl@ceph.osd.10.smithi142.stdout:Mar 31 23:39:31 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-10[60116]: 2024-03-31T23:39:31.275+0000 7f64c04a8700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-31T23:38:13.357804+0000 front 2024-03-31T23:38:13.357906+0000 (oldest deadline 2024-03-31T23:38:38.057730+0000) 2024-03-31T23:39:31.444 INFO:journalctl@ceph.osd.10.smithi142.stdout:Mar 31 23:39:31 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-10[60116]: 2024-03-31T23:39:31.275+0000 7f64c04a8700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.142:6804 osd.1 since back 2024-03-31T23:38:20.958341+0000 front 2024-03-31T23:38:20.958408+0000 (oldest deadline 2024-03-31T23:38:45.058481+0000) 2024-03-31T23:39:31.444 INFO:journalctl@ceph.osd.10.smithi142.stdout:Mar 31 23:39:31 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-10[60116]: 2024-03-31T23:39:31.275+0000 7f64c04a8700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-31T23:38:25.559393+0000 front 2024-03-31T23:38:25.559313+0000 (oldest deadline 2024-03-31T23:38:51.459398+0000) 2024-03-31T23:39:31.444 INFO:journalctl@ceph.osd.10.smithi142.stdout:Mar 31 23:39:31 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-10[60116]: 2024-03-31T23:39:31.275+0000 7f64c04a8700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-03-31T23:38:36.660831+0000 front 2024-03-31T23:38:36.660954+0000 (oldest deadline 2024-03-31T23:39:00.760714+0000) 2024-03-31T23:39:31.444 INFO:journalctl@ceph.osd.10.smithi142.stdout:Mar 31 23:39:31 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-10[60116]: 2024-03-31T23:39:31.275+0000 7f64c04a8700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.142:6812 osd.4 since back 2024-03-31T23:38:42.361714+0000 front 2024-03-31T23:38:42.361749+0000 (oldest deadline 2024-03-31T23:39:08.261872+0000) 2024-03-31T23:39:31.444 INFO:journalctl@ceph.osd.10.smithi142.stdout:Mar 31 23:39:31 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-10[60116]: 2024-03-31T23:39:31.275+0000 7f64c04a8700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-03-31T23:38:52.962747+0000 front 2024-03-31T23:38:52.962731+0000 (oldest deadline 2024-03-31T23:39:17.662721+0000) 2024-03-31T23:39:31.444 INFO:journalctl@ceph.osd.10.smithi142.stdout:Mar 31 23:39:31 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-10[60116]: 2024-03-31T23:39:31.275+0000 7f64c04a8700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.46:6822 osd.6 since back 2024-03-31T23:39:00.964464+0000 front 2024-03-31T23:39:00.964397+0000 (oldest deadline 2024-03-31T23:39:25.564525+0000) 2024-03-31T23:39:32.296 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:39:31 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:39:31.953+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-31T23:38:12.221366+0000 front 2024-03-31T23:38:12.221347+0000 (oldest deadline 2024-03-31T23:38:37.521357+0000) 2024-03-31T23:39:32.296 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:39:31 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:39:31.953+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.142:6804 osd.1 since back 2024-03-31T23:38:22.222192+0000 front 2024-03-31T23:38:22.222213+0000 (oldest deadline 2024-03-31T23:38:45.722340+0000) 2024-03-31T23:39:32.296 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:39:31 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:39:31.953+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-31T23:38:29.723616+0000 front 2024-03-31T23:38:29.723593+0000 (oldest deadline 2024-03-31T23:38:52.023779+0000) 2024-03-31T23:39:32.296 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:39:31 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:39:31.953+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-03-31T23:38:36.124750+0000 front 2024-03-31T23:38:36.124868+0000 (oldest deadline 2024-03-31T23:39:00.824619+0000) 2024-03-31T23:39:32.296 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:39:31 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:39:31.953+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.142:6812 osd.4 since back 2024-03-31T23:38:44.825661+0000 front 2024-03-31T23:38:44.825679+0000 (oldest deadline 2024-03-31T23:39:10.125827+0000) 2024-03-31T23:39:32.296 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:39:31 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:39:31.953+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-03-31T23:38:51.226778+0000 front 2024-03-31T23:38:51.226798+0000 (oldest deadline 2024-03-31T23:39:15.926915+0000) 2024-03-31T23:39:32.296 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:39:31 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:39:31.953+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.46:6822 osd.6 since back 2024-03-31T23:39:01.140495+0000 front 2024-03-31T23:39:01.140543+0000 (oldest deadline 2024-03-31T23:39:24.640339+0000) 2024-03-31T23:39:32.693 INFO:journalctl@ceph.osd.10.smithi142.stdout:Mar 31 23:39:32 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-10[60116]: 2024-03-31T23:39:32.241+0000 7f64c04a8700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-31T23:38:13.357804+0000 front 2024-03-31T23:38:13.357906+0000 (oldest deadline 2024-03-31T23:38:38.057730+0000) 2024-03-31T23:39:32.693 INFO:journalctl@ceph.osd.10.smithi142.stdout:Mar 31 23:39:32 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-10[60116]: 2024-03-31T23:39:32.241+0000 7f64c04a8700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.142:6804 osd.1 since back 2024-03-31T23:38:20.958341+0000 front 2024-03-31T23:38:20.958408+0000 (oldest deadline 2024-03-31T23:38:45.058481+0000) 2024-03-31T23:39:32.693 INFO:journalctl@ceph.osd.10.smithi142.stdout:Mar 31 23:39:32 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-10[60116]: 2024-03-31T23:39:32.241+0000 7f64c04a8700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-31T23:38:25.559393+0000 front 2024-03-31T23:38:25.559313+0000 (oldest deadline 2024-03-31T23:38:51.459398+0000) 2024-03-31T23:39:32.693 INFO:journalctl@ceph.osd.10.smithi142.stdout:Mar 31 23:39:32 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-10[60116]: 2024-03-31T23:39:32.241+0000 7f64c04a8700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-03-31T23:38:36.660831+0000 front 2024-03-31T23:38:36.660954+0000 (oldest deadline 2024-03-31T23:39:00.760714+0000) 2024-03-31T23:39:32.693 INFO:journalctl@ceph.osd.10.smithi142.stdout:Mar 31 23:39:32 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-10[60116]: 2024-03-31T23:39:32.241+0000 7f64c04a8700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.142:6812 osd.4 since back 2024-03-31T23:38:42.361714+0000 front 2024-03-31T23:38:42.361749+0000 (oldest deadline 2024-03-31T23:39:08.261872+0000) 2024-03-31T23:39:32.693 INFO:journalctl@ceph.osd.10.smithi142.stdout:Mar 31 23:39:32 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-10[60116]: 2024-03-31T23:39:32.241+0000 7f64c04a8700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-03-31T23:38:52.962747+0000 front 2024-03-31T23:38:52.962731+0000 (oldest deadline 2024-03-31T23:39:17.662721+0000) 2024-03-31T23:39:32.693 INFO:journalctl@ceph.osd.10.smithi142.stdout:Mar 31 23:39:32 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-10[60116]: 2024-03-31T23:39:32.241+0000 7f64c04a8700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.46:6822 osd.6 since back 2024-03-31T23:39:00.964464+0000 front 2024-03-31T23:39:00.964397+0000 (oldest deadline 2024-03-31T23:39:25.564525+0000) 2024-03-31T23:39:33.295 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:39:32 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:39:32.950+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-31T23:38:12.221366+0000 front 2024-03-31T23:38:12.221347+0000 (oldest deadline 2024-03-31T23:38:37.521357+0000) 2024-03-31T23:39:33.295 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:39:32 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:39:32.950+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.142:6804 osd.1 since back 2024-03-31T23:38:22.222192+0000 front 2024-03-31T23:38:22.222213+0000 (oldest deadline 2024-03-31T23:38:45.722340+0000) 2024-03-31T23:39:33.296 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:39:32 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:39:32.950+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-31T23:38:29.723616+0000 front 2024-03-31T23:38:29.723593+0000 (oldest deadline 2024-03-31T23:38:52.023779+0000) 2024-03-31T23:39:33.296 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:39:32 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:39:32.950+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-03-31T23:38:36.124750+0000 front 2024-03-31T23:38:36.124868+0000 (oldest deadline 2024-03-31T23:39:00.824619+0000) 2024-03-31T23:39:33.296 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:39:32 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:39:32.950+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.142:6812 osd.4 since back 2024-03-31T23:38:44.825661+0000 front 2024-03-31T23:38:44.825679+0000 (oldest deadline 2024-03-31T23:39:10.125827+0000) 2024-03-31T23:39:33.296 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:39:32 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:39:32.950+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-03-31T23:38:51.226778+0000 front 2024-03-31T23:38:51.226798+0000 (oldest deadline 2024-03-31T23:39:15.926915+0000) 2024-03-31T23:39:33.296 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:39:32 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:39:32.950+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.46:6822 osd.6 since back 2024-03-31T23:39:01.140495+0000 front 2024-03-31T23:39:01.140543+0000 (oldest deadline 2024-03-31T23:39:24.640339+0000) 2024-03-31T23:39:33.296 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:39:32 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:39:32.950+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.142:6820 osd.7 since back 2024-03-31T23:39:08.041477+0000 front 2024-03-31T23:39:08.041456+0000 (oldest deadline 2024-03-31T23:39:32.741460+0000) 2024-03-31T23:39:33.693 INFO:journalctl@ceph.osd.10.smithi142.stdout:Mar 31 23:39:33 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-10[60116]: 2024-03-31T23:39:33.237+0000 7f64c04a8700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-31T23:38:13.357804+0000 front 2024-03-31T23:38:13.357906+0000 (oldest deadline 2024-03-31T23:38:38.057730+0000) 2024-03-31T23:39:33.693 INFO:journalctl@ceph.osd.10.smithi142.stdout:Mar 31 23:39:33 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-10[60116]: 2024-03-31T23:39:33.237+0000 7f64c04a8700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.142:6804 osd.1 since back 2024-03-31T23:38:20.958341+0000 front 2024-03-31T23:38:20.958408+0000 (oldest deadline 2024-03-31T23:38:45.058481+0000) 2024-03-31T23:39:33.693 INFO:journalctl@ceph.osd.10.smithi142.stdout:Mar 31 23:39:33 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-10[60116]: 2024-03-31T23:39:33.237+0000 7f64c04a8700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-31T23:38:25.559393+0000 front 2024-03-31T23:38:25.559313+0000 (oldest deadline 2024-03-31T23:38:51.459398+0000) 2024-03-31T23:39:33.693 INFO:journalctl@ceph.osd.10.smithi142.stdout:Mar 31 23:39:33 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-10[60116]: 2024-03-31T23:39:33.237+0000 7f64c04a8700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-03-31T23:38:36.660831+0000 front 2024-03-31T23:38:36.660954+0000 (oldest deadline 2024-03-31T23:39:00.760714+0000) 2024-03-31T23:39:33.693 INFO:journalctl@ceph.osd.10.smithi142.stdout:Mar 31 23:39:33 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-10[60116]: 2024-03-31T23:39:33.237+0000 7f64c04a8700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.142:6812 osd.4 since back 2024-03-31T23:38:42.361714+0000 front 2024-03-31T23:38:42.361749+0000 (oldest deadline 2024-03-31T23:39:08.261872+0000) 2024-03-31T23:39:33.693 INFO:journalctl@ceph.osd.10.smithi142.stdout:Mar 31 23:39:33 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-10[60116]: 2024-03-31T23:39:33.237+0000 7f64c04a8700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-03-31T23:38:52.962747+0000 front 2024-03-31T23:38:52.962731+0000 (oldest deadline 2024-03-31T23:39:17.662721+0000) 2024-03-31T23:39:33.693 INFO:journalctl@ceph.osd.10.smithi142.stdout:Mar 31 23:39:33 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-10[60116]: 2024-03-31T23:39:33.237+0000 7f64c04a8700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.46:6822 osd.6 since back 2024-03-31T23:39:00.964464+0000 front 2024-03-31T23:39:00.964397+0000 (oldest deadline 2024-03-31T23:39:25.564525+0000) 2024-03-31T23:39:34.295 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:39:33 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:39:33.907+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-31T23:38:12.221366+0000 front 2024-03-31T23:38:12.221347+0000 (oldest deadline 2024-03-31T23:38:37.521357+0000) 2024-03-31T23:39:34.296 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:39:33 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:39:33.907+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.142:6804 osd.1 since back 2024-03-31T23:38:22.222192+0000 front 2024-03-31T23:38:22.222213+0000 (oldest deadline 2024-03-31T23:38:45.722340+0000) 2024-03-31T23:39:34.296 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:39:33 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:39:33.907+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-31T23:38:29.723616+0000 front 2024-03-31T23:38:29.723593+0000 (oldest deadline 2024-03-31T23:38:52.023779+0000) 2024-03-31T23:39:34.296 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:39:33 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:39:33.907+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-03-31T23:38:36.124750+0000 front 2024-03-31T23:38:36.124868+0000 (oldest deadline 2024-03-31T23:39:00.824619+0000) 2024-03-31T23:39:34.296 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:39:33 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:39:33.907+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.142:6812 osd.4 since back 2024-03-31T23:38:44.825661+0000 front 2024-03-31T23:38:44.825679+0000 (oldest deadline 2024-03-31T23:39:10.125827+0000) 2024-03-31T23:39:34.296 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:39:33 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:39:33.907+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-03-31T23:38:51.226778+0000 front 2024-03-31T23:38:51.226798+0000 (oldest deadline 2024-03-31T23:39:15.926915+0000) 2024-03-31T23:39:34.296 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:39:33 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:39:33.907+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.46:6822 osd.6 since back 2024-03-31T23:39:01.140495+0000 front 2024-03-31T23:39:01.140543+0000 (oldest deadline 2024-03-31T23:39:24.640339+0000) 2024-03-31T23:39:34.296 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:39:33 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:39:33.907+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.142:6820 osd.7 since back 2024-03-31T23:39:08.041477+0000 front 2024-03-31T23:39:08.041456+0000 (oldest deadline 2024-03-31T23:39:32.741460+0000) 2024-03-31T23:39:34.693 INFO:journalctl@ceph.osd.10.smithi142.stdout:Mar 31 23:39:34 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-10[60116]: 2024-03-31T23:39:34.229+0000 7f64c04a8700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-31T23:38:13.357804+0000 front 2024-03-31T23:38:13.357906+0000 (oldest deadline 2024-03-31T23:38:38.057730+0000) 2024-03-31T23:39:34.693 INFO:journalctl@ceph.osd.10.smithi142.stdout:Mar 31 23:39:34 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-10[60116]: 2024-03-31T23:39:34.229+0000 7f64c04a8700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.142:6804 osd.1 since back 2024-03-31T23:38:20.958341+0000 front 2024-03-31T23:38:20.958408+0000 (oldest deadline 2024-03-31T23:38:45.058481+0000) 2024-03-31T23:39:34.693 INFO:journalctl@ceph.osd.10.smithi142.stdout:Mar 31 23:39:34 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-10[60116]: 2024-03-31T23:39:34.229+0000 7f64c04a8700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-31T23:38:25.559393+0000 front 2024-03-31T23:38:25.559313+0000 (oldest deadline 2024-03-31T23:38:51.459398+0000) 2024-03-31T23:39:34.693 INFO:journalctl@ceph.osd.10.smithi142.stdout:Mar 31 23:39:34 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-10[60116]: 2024-03-31T23:39:34.229+0000 7f64c04a8700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-03-31T23:38:36.660831+0000 front 2024-03-31T23:38:36.660954+0000 (oldest deadline 2024-03-31T23:39:00.760714+0000) 2024-03-31T23:39:34.693 INFO:journalctl@ceph.osd.10.smithi142.stdout:Mar 31 23:39:34 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-10[60116]: 2024-03-31T23:39:34.229+0000 7f64c04a8700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.142:6812 osd.4 since back 2024-03-31T23:38:42.361714+0000 front 2024-03-31T23:38:42.361749+0000 (oldest deadline 2024-03-31T23:39:08.261872+0000) 2024-03-31T23:39:34.693 INFO:journalctl@ceph.osd.10.smithi142.stdout:Mar 31 23:39:34 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-10[60116]: 2024-03-31T23:39:34.229+0000 7f64c04a8700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-03-31T23:38:52.962747+0000 front 2024-03-31T23:38:52.962731+0000 (oldest deadline 2024-03-31T23:39:17.662721+0000) 2024-03-31T23:39:34.693 INFO:journalctl@ceph.osd.10.smithi142.stdout:Mar 31 23:39:34 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-10[60116]: 2024-03-31T23:39:34.229+0000 7f64c04a8700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.46:6822 osd.6 since back 2024-03-31T23:39:00.964464+0000 front 2024-03-31T23:39:00.964397+0000 (oldest deadline 2024-03-31T23:39:25.564525+0000) 2024-03-31T23:39:35.295 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:39:34 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:39:34.890+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-31T23:38:12.221366+0000 front 2024-03-31T23:38:12.221347+0000 (oldest deadline 2024-03-31T23:38:37.521357+0000) 2024-03-31T23:39:35.295 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:39:34 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:39:34.890+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.142:6804 osd.1 since back 2024-03-31T23:38:22.222192+0000 front 2024-03-31T23:38:22.222213+0000 (oldest deadline 2024-03-31T23:38:45.722340+0000) 2024-03-31T23:39:35.296 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:39:34 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:39:34.890+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-31T23:38:29.723616+0000 front 2024-03-31T23:38:29.723593+0000 (oldest deadline 2024-03-31T23:38:52.023779+0000) 2024-03-31T23:39:35.296 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:39:34 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:39:34.890+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-03-31T23:38:36.124750+0000 front 2024-03-31T23:38:36.124868+0000 (oldest deadline 2024-03-31T23:39:00.824619+0000) 2024-03-31T23:39:35.296 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:39:34 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:39:34.890+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.142:6812 osd.4 since back 2024-03-31T23:38:44.825661+0000 front 2024-03-31T23:38:44.825679+0000 (oldest deadline 2024-03-31T23:39:10.125827+0000) 2024-03-31T23:39:35.296 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:39:34 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:39:34.890+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-03-31T23:38:51.226778+0000 front 2024-03-31T23:38:51.226798+0000 (oldest deadline 2024-03-31T23:39:15.926915+0000) 2024-03-31T23:39:35.296 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:39:34 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:39:34.890+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.46:6822 osd.6 since back 2024-03-31T23:39:01.140495+0000 front 2024-03-31T23:39:01.140543+0000 (oldest deadline 2024-03-31T23:39:24.640339+0000) 2024-03-31T23:39:35.296 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:39:34 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:39:34.890+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.142:6820 osd.7 since back 2024-03-31T23:39:08.041477+0000 front 2024-03-31T23:39:08.041456+0000 (oldest deadline 2024-03-31T23:39:32.741460+0000) 2024-03-31T23:39:35.693 INFO:journalctl@ceph.osd.10.smithi142.stdout:Mar 31 23:39:35 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-10[60116]: 2024-03-31T23:39:35.231+0000 7f64c04a8700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-31T23:38:13.357804+0000 front 2024-03-31T23:38:13.357906+0000 (oldest deadline 2024-03-31T23:38:38.057730+0000) 2024-03-31T23:39:35.693 INFO:journalctl@ceph.osd.10.smithi142.stdout:Mar 31 23:39:35 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-10[60116]: 2024-03-31T23:39:35.231+0000 7f64c04a8700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.142:6804 osd.1 since back 2024-03-31T23:38:20.958341+0000 front 2024-03-31T23:38:20.958408+0000 (oldest deadline 2024-03-31T23:38:45.058481+0000) 2024-03-31T23:39:35.693 INFO:journalctl@ceph.osd.10.smithi142.stdout:Mar 31 23:39:35 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-10[60116]: 2024-03-31T23:39:35.231+0000 7f64c04a8700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-31T23:38:25.559393+0000 front 2024-03-31T23:38:25.559313+0000 (oldest deadline 2024-03-31T23:38:51.459398+0000) 2024-03-31T23:39:35.693 INFO:journalctl@ceph.osd.10.smithi142.stdout:Mar 31 23:39:35 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-10[60116]: 2024-03-31T23:39:35.231+0000 7f64c04a8700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-03-31T23:38:36.660831+0000 front 2024-03-31T23:38:36.660954+0000 (oldest deadline 2024-03-31T23:39:00.760714+0000) 2024-03-31T23:39:35.693 INFO:journalctl@ceph.osd.10.smithi142.stdout:Mar 31 23:39:35 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-10[60116]: 2024-03-31T23:39:35.231+0000 7f64c04a8700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.142:6812 osd.4 since back 2024-03-31T23:38:42.361714+0000 front 2024-03-31T23:38:42.361749+0000 (oldest deadline 2024-03-31T23:39:08.261872+0000) 2024-03-31T23:39:35.694 INFO:journalctl@ceph.osd.10.smithi142.stdout:Mar 31 23:39:35 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-10[60116]: 2024-03-31T23:39:35.231+0000 7f64c04a8700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-03-31T23:38:52.962747+0000 front 2024-03-31T23:38:52.962731+0000 (oldest deadline 2024-03-31T23:39:17.662721+0000) 2024-03-31T23:39:35.694 INFO:journalctl@ceph.osd.10.smithi142.stdout:Mar 31 23:39:35 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-10[60116]: 2024-03-31T23:39:35.231+0000 7f64c04a8700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.46:6822 osd.6 since back 2024-03-31T23:39:00.964464+0000 front 2024-03-31T23:39:00.964397+0000 (oldest deadline 2024-03-31T23:39:25.564525+0000) 2024-03-31T23:39:36.237 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:39:35 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:39:35.887+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-31T23:38:12.221366+0000 front 2024-03-31T23:38:12.221347+0000 (oldest deadline 2024-03-31T23:38:37.521357+0000) 2024-03-31T23:39:36.238 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:39:35 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:39:35.887+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.142:6804 osd.1 since back 2024-03-31T23:38:22.222192+0000 front 2024-03-31T23:38:22.222213+0000 (oldest deadline 2024-03-31T23:38:45.722340+0000) 2024-03-31T23:39:36.238 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:39:35 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:39:35.887+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-31T23:38:29.723616+0000 front 2024-03-31T23:38:29.723593+0000 (oldest deadline 2024-03-31T23:38:52.023779+0000) 2024-03-31T23:39:36.238 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:39:35 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:39:35.887+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-03-31T23:38:36.124750+0000 front 2024-03-31T23:38:36.124868+0000 (oldest deadline 2024-03-31T23:39:00.824619+0000) 2024-03-31T23:39:36.238 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:39:35 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:39:35.887+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.142:6812 osd.4 since back 2024-03-31T23:38:44.825661+0000 front 2024-03-31T23:38:44.825679+0000 (oldest deadline 2024-03-31T23:39:10.125827+0000) 2024-03-31T23:39:36.238 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:39:35 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:39:35.887+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-03-31T23:38:51.226778+0000 front 2024-03-31T23:38:51.226798+0000 (oldest deadline 2024-03-31T23:39:15.926915+0000) 2024-03-31T23:39:36.238 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:39:35 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:39:35.887+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.46:6822 osd.6 since back 2024-03-31T23:39:01.140495+0000 front 2024-03-31T23:39:01.140543+0000 (oldest deadline 2024-03-31T23:39:24.640339+0000) 2024-03-31T23:39:36.238 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:39:35 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:39:35.887+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.142:6820 osd.7 since back 2024-03-31T23:39:08.041477+0000 front 2024-03-31T23:39:08.041456+0000 (oldest deadline 2024-03-31T23:39:32.741460+0000) 2024-03-31T23:39:36.712 INFO:journalctl@ceph.osd.10.smithi142.stdout:Mar 31 23:39:36 smithi142 podman[78388]: 2024-03-31 23:39:36.444283035 +0000 UTC m=+5.497950757 container died 7bf1dd84a3515c53007d3cc2e357a829f30c899e89481bfd315f491de05ea5f6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-10, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.build-date=20240326, RELEASE=squid-384f31f, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=) 2024-03-31T23:39:37.086 INFO:journalctl@ceph.osd.10.smithi142.stdout:Mar 31 23:39:36 smithi142 podman[78388]: 2024-03-31 23:39:36.802528507 +0000 UTC m=+5.856196210 container cleanup 7bf1dd84a3515c53007d3cc2e357a829f30c899e89481bfd315f491de05ea5f6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-10, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, RELEASE=squid-384f31f, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240326, ceph=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.schema-version=1.0) 2024-03-31T23:39:37.087 INFO:journalctl@ceph.osd.10.smithi142.stdout:Mar 31 23:39:36 smithi142 bash[78388]: ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-10 2024-03-31T23:39:37.295 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:39:36 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:39:36.887+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-31T23:38:12.221366+0000 front 2024-03-31T23:38:12.221347+0000 (oldest deadline 2024-03-31T23:38:37.521357+0000) 2024-03-31T23:39:37.295 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:39:36 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:39:36.887+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.142:6804 osd.1 since back 2024-03-31T23:38:22.222192+0000 front 2024-03-31T23:38:22.222213+0000 (oldest deadline 2024-03-31T23:38:45.722340+0000) 2024-03-31T23:39:37.296 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:39:36 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:39:36.887+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-31T23:38:29.723616+0000 front 2024-03-31T23:38:29.723593+0000 (oldest deadline 2024-03-31T23:38:52.023779+0000) 2024-03-31T23:39:37.296 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:39:36 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:39:36.887+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-03-31T23:38:36.124750+0000 front 2024-03-31T23:38:36.124868+0000 (oldest deadline 2024-03-31T23:39:00.824619+0000) 2024-03-31T23:39:37.296 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:39:36 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:39:36.887+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.142:6812 osd.4 since back 2024-03-31T23:38:44.825661+0000 front 2024-03-31T23:38:44.825679+0000 (oldest deadline 2024-03-31T23:39:10.125827+0000) 2024-03-31T23:39:37.296 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:39:36 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:39:36.887+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-03-31T23:38:51.226778+0000 front 2024-03-31T23:38:51.226798+0000 (oldest deadline 2024-03-31T23:39:15.926915+0000) 2024-03-31T23:39:37.296 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:39:36 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:39:36.887+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.46:6822 osd.6 since back 2024-03-31T23:39:01.140495+0000 front 2024-03-31T23:39:01.140543+0000 (oldest deadline 2024-03-31T23:39:24.640339+0000) 2024-03-31T23:39:37.296 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:39:36 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:39:36.887+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.142:6820 osd.7 since back 2024-03-31T23:39:08.041477+0000 front 2024-03-31T23:39:08.041456+0000 (oldest deadline 2024-03-31T23:39:32.741460+0000) 2024-03-31T23:39:37.434 INFO:journalctl@ceph.osd.10.smithi142.stdout:Mar 31 23:39:37 smithi142 podman[78407]: 2024-03-31 23:39:37.087049884 +0000 UTC m=+0.639388182 container remove 7bf1dd84a3515c53007d3cc2e357a829f30c899e89481bfd315f491de05ea5f6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-10, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , RELEASE=squid-384f31f, io.buildah.version=1.33.5, org.label-schema.build-date=20240326, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-03-31T23:39:37.693 INFO:journalctl@ceph.osd.10.smithi142.stdout:Mar 31 23:39:37 smithi142 podman[78494]: 2024-03-31 23:39:37.338818219 +0000 UTC m=+0.019834963 image pull ec18d7baa60e8b1a0b126fbe5306acce11b1ed0ec5e3bcdddbf39b0a3c591791 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175 2024-03-31T23:39:37.693 INFO:journalctl@ceph.osd.10.smithi142.stdout:Mar 31 23:39:37 smithi142 podman[78494]: 2024-03-31 23:39:37.496178564 +0000 UTC m=+0.177195288 container create 279411dfa3f342775775b958cf8fbdbafb07dd2fd1892dc24f1bbdb774c7c748 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-10-deactivate, ceph=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.build-date=20240326, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, RELEASE=squid-384f31f, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0) 2024-03-31T23:39:38.130 INFO:journalctl@ceph.osd.10.smithi142.stdout:Mar 31 23:39:37 smithi142 podman[78494]: 2024-03-31 23:39:37.740699002 +0000 UTC m=+0.421715746 container init 279411dfa3f342775775b958cf8fbdbafb07dd2fd1892dc24f1bbdb774c7c748 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-10-deactivate, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240326, CEPH_POINT_RELEASE=, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_CLEAN=True, RELEASE=squid-384f31f, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux ) 2024-03-31T23:39:38.131 INFO:journalctl@ceph.osd.10.smithi142.stdout:Mar 31 23:39:37 smithi142 podman[78494]: 2024-03-31 23:39:37.743714373 +0000 UTC m=+0.424731105 container start 279411dfa3f342775775b958cf8fbdbafb07dd2fd1892dc24f1bbdb774c7c748 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-10-deactivate, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , RELEASE=squid-384f31f, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.build-date=20240326, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, ceph=True, GIT_CLEAN=True) 2024-03-31T23:39:38.131 INFO:journalctl@ceph.osd.10.smithi142.stdout:Mar 31 23:39:37 smithi142 podman[78494]: 2024-03-31 23:39:37.840987801 +0000 UTC m=+0.522004539 container attach 279411dfa3f342775775b958cf8fbdbafb07dd2fd1892dc24f1bbdb774c7c748 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-10-deactivate, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-384f31f, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240326) 2024-03-31T23:39:38.295 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:39:37 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:39:37.929+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-31T23:38:12.221366+0000 front 2024-03-31T23:38:12.221347+0000 (oldest deadline 2024-03-31T23:38:37.521357+0000) 2024-03-31T23:39:38.296 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:39:37 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:39:37.929+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.142:6804 osd.1 since back 2024-03-31T23:38:22.222192+0000 front 2024-03-31T23:38:22.222213+0000 (oldest deadline 2024-03-31T23:38:45.722340+0000) 2024-03-31T23:39:38.296 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:39:37 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:39:37.929+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-31T23:38:29.723616+0000 front 2024-03-31T23:38:29.723593+0000 (oldest deadline 2024-03-31T23:38:52.023779+0000) 2024-03-31T23:39:38.296 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:39:37 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:39:37.929+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-03-31T23:38:36.124750+0000 front 2024-03-31T23:38:36.124868+0000 (oldest deadline 2024-03-31T23:39:00.824619+0000) 2024-03-31T23:39:38.296 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:39:37 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:39:37.929+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.142:6812 osd.4 since back 2024-03-31T23:38:44.825661+0000 front 2024-03-31T23:38:44.825679+0000 (oldest deadline 2024-03-31T23:39:10.125827+0000) 2024-03-31T23:39:38.296 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:39:37 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:39:37.929+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-03-31T23:38:51.226778+0000 front 2024-03-31T23:38:51.226798+0000 (oldest deadline 2024-03-31T23:39:15.926915+0000) 2024-03-31T23:39:38.296 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:39:37 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:39:37.929+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.46:6822 osd.6 since back 2024-03-31T23:39:01.140495+0000 front 2024-03-31T23:39:01.140543+0000 (oldest deadline 2024-03-31T23:39:24.640339+0000) 2024-03-31T23:39:38.296 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:39:37 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:39:37.929+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.142:6820 osd.7 since back 2024-03-31T23:39:08.041477+0000 front 2024-03-31T23:39:08.041456+0000 (oldest deadline 2024-03-31T23:39:32.741460+0000) 2024-03-31T23:39:38.427 INFO:journalctl@ceph.osd.10.smithi142.stdout:Mar 31 23:39:38 smithi142 podman[78494]: 2024-03-31 23:39:38.13084028 +0000 UTC m=+0.811857008 container died 279411dfa3f342775775b958cf8fbdbafb07dd2fd1892dc24f1bbdb774c7c748 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-10-deactivate, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_CLEAN=True, RELEASE=squid-384f31f, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.build-date=20240326, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_BRANCH=HEAD) 2024-03-31T23:39:38.687 DEBUG:teuthology.orchestra.run.smithi142:> sudo pkill -f 'journalctl -f -n 0 -u ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4@osd.10.service' 2024-03-31T23:39:38.729 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-31T23:39:38.729 INFO:tasks.cephadm.osd.10:Stopped osd.10 2024-03-31T23:39:38.729 INFO:tasks.cephadm.osd.11:Stopping osd.11... 2024-03-31T23:39:38.729 DEBUG:teuthology.orchestra.run.smithi162:> sudo systemctl stop ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4@osd.11 2024-03-31T23:39:39.037 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:39:38 smithi162 systemd[1]: Stopping Ceph osd.11 for 6a9d8f08-efb2-11ee-b647-cb9ed24678a4... 2024-03-31T23:39:39.038 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:39:38 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:39:38.919+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-31T23:38:12.221366+0000 front 2024-03-31T23:38:12.221347+0000 (oldest deadline 2024-03-31T23:38:37.521357+0000) 2024-03-31T23:39:39.038 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:39:38 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:39:38.919+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.142:6804 osd.1 since back 2024-03-31T23:38:22.222192+0000 front 2024-03-31T23:38:22.222213+0000 (oldest deadline 2024-03-31T23:38:45.722340+0000) 2024-03-31T23:39:39.038 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:39:38 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:39:38.919+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-31T23:38:29.723616+0000 front 2024-03-31T23:38:29.723593+0000 (oldest deadline 2024-03-31T23:38:52.023779+0000) 2024-03-31T23:39:39.038 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:39:38 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:39:38.919+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-03-31T23:38:36.124750+0000 front 2024-03-31T23:38:36.124868+0000 (oldest deadline 2024-03-31T23:39:00.824619+0000) 2024-03-31T23:39:39.038 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:39:38 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:39:38.919+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.142:6812 osd.4 since back 2024-03-31T23:38:44.825661+0000 front 2024-03-31T23:38:44.825679+0000 (oldest deadline 2024-03-31T23:39:10.125827+0000) 2024-03-31T23:39:39.038 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:39:38 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:39:38.919+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-03-31T23:38:51.226778+0000 front 2024-03-31T23:38:51.226798+0000 (oldest deadline 2024-03-31T23:39:15.926915+0000) 2024-03-31T23:39:39.038 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:39:38 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:39:38.919+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.46:6822 osd.6 since back 2024-03-31T23:39:01.140495+0000 front 2024-03-31T23:39:01.140543+0000 (oldest deadline 2024-03-31T23:39:24.640339+0000) 2024-03-31T23:39:39.038 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:39:38 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:39:38.919+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.142:6820 osd.7 since back 2024-03-31T23:39:08.041477+0000 front 2024-03-31T23:39:08.041456+0000 (oldest deadline 2024-03-31T23:39:32.741460+0000) 2024-03-31T23:39:39.038 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:39:39 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:39:39.036+0000 7f98d7d4c700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.11 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-31T23:39:39.038 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:39:39 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:39:39.036+0000 7f98d7d4c700 -1 osd.11 139 *** Got signal Terminated *** 2024-03-31T23:39:39.038 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:39:39 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:39:39.036+0000 7f98d7d4c700 -1 osd.11 139 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-31T23:39:40.295 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:39:39 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:39:39.913+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-31T23:38:12.221366+0000 front 2024-03-31T23:38:12.221347+0000 (oldest deadline 2024-03-31T23:38:37.521357+0000) 2024-03-31T23:39:40.296 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:39:39 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:39:39.913+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.142:6804 osd.1 since back 2024-03-31T23:38:22.222192+0000 front 2024-03-31T23:38:22.222213+0000 (oldest deadline 2024-03-31T23:38:45.722340+0000) 2024-03-31T23:39:40.296 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:39:39 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:39:39.913+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-31T23:38:29.723616+0000 front 2024-03-31T23:38:29.723593+0000 (oldest deadline 2024-03-31T23:38:52.023779+0000) 2024-03-31T23:39:40.296 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:39:39 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:39:39.913+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-03-31T23:38:36.124750+0000 front 2024-03-31T23:38:36.124868+0000 (oldest deadline 2024-03-31T23:39:00.824619+0000) 2024-03-31T23:39:40.296 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:39:39 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:39:39.913+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.142:6812 osd.4 since back 2024-03-31T23:38:44.825661+0000 front 2024-03-31T23:38:44.825679+0000 (oldest deadline 2024-03-31T23:39:10.125827+0000) 2024-03-31T23:39:40.296 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:39:39 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:39:39.913+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-03-31T23:38:51.226778+0000 front 2024-03-31T23:38:51.226798+0000 (oldest deadline 2024-03-31T23:39:15.926915+0000) 2024-03-31T23:39:40.296 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:39:39 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:39:39.913+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.46:6822 osd.6 since back 2024-03-31T23:39:01.140495+0000 front 2024-03-31T23:39:01.140543+0000 (oldest deadline 2024-03-31T23:39:24.640339+0000) 2024-03-31T23:39:40.296 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:39:39 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:39:39.913+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.142:6820 osd.7 since back 2024-03-31T23:39:08.041477+0000 front 2024-03-31T23:39:08.041456+0000 (oldest deadline 2024-03-31T23:39:32.741460+0000) 2024-03-31T23:39:41.295 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:39:40 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:39:40.908+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-31T23:38:12.221366+0000 front 2024-03-31T23:38:12.221347+0000 (oldest deadline 2024-03-31T23:38:37.521357+0000) 2024-03-31T23:39:41.296 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:39:40 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:39:40.908+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.142:6804 osd.1 since back 2024-03-31T23:38:22.222192+0000 front 2024-03-31T23:38:22.222213+0000 (oldest deadline 2024-03-31T23:38:45.722340+0000) 2024-03-31T23:39:41.296 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:39:40 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:39:40.908+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-31T23:38:29.723616+0000 front 2024-03-31T23:38:29.723593+0000 (oldest deadline 2024-03-31T23:38:52.023779+0000) 2024-03-31T23:39:41.296 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:39:40 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:39:40.908+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-03-31T23:38:36.124750+0000 front 2024-03-31T23:38:36.124868+0000 (oldest deadline 2024-03-31T23:39:00.824619+0000) 2024-03-31T23:39:41.296 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:39:40 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:39:40.908+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.142:6812 osd.4 since back 2024-03-31T23:38:44.825661+0000 front 2024-03-31T23:38:44.825679+0000 (oldest deadline 2024-03-31T23:39:10.125827+0000) 2024-03-31T23:39:41.296 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:39:40 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:39:40.908+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-03-31T23:38:51.226778+0000 front 2024-03-31T23:38:51.226798+0000 (oldest deadline 2024-03-31T23:39:15.926915+0000) 2024-03-31T23:39:41.296 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:39:40 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:39:40.908+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.46:6822 osd.6 since back 2024-03-31T23:39:01.140495+0000 front 2024-03-31T23:39:01.140543+0000 (oldest deadline 2024-03-31T23:39:24.640339+0000) 2024-03-31T23:39:41.296 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:39:40 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:39:40.908+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.142:6820 osd.7 since back 2024-03-31T23:39:08.041477+0000 front 2024-03-31T23:39:08.041456+0000 (oldest deadline 2024-03-31T23:39:32.741460+0000) 2024-03-31T23:39:42.295 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:39:41 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:39:41.890+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-31T23:38:12.221366+0000 front 2024-03-31T23:38:12.221347+0000 (oldest deadline 2024-03-31T23:38:37.521357+0000) 2024-03-31T23:39:42.295 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:39:41 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:39:41.890+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.142:6804 osd.1 since back 2024-03-31T23:38:22.222192+0000 front 2024-03-31T23:38:22.222213+0000 (oldest deadline 2024-03-31T23:38:45.722340+0000) 2024-03-31T23:39:42.296 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:39:41 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:39:41.890+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-31T23:38:29.723616+0000 front 2024-03-31T23:38:29.723593+0000 (oldest deadline 2024-03-31T23:38:52.023779+0000) 2024-03-31T23:39:42.296 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:39:41 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:39:41.890+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-03-31T23:38:36.124750+0000 front 2024-03-31T23:38:36.124868+0000 (oldest deadline 2024-03-31T23:39:00.824619+0000) 2024-03-31T23:39:42.296 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:39:41 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:39:41.890+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.142:6812 osd.4 since back 2024-03-31T23:38:44.825661+0000 front 2024-03-31T23:38:44.825679+0000 (oldest deadline 2024-03-31T23:39:10.125827+0000) 2024-03-31T23:39:42.296 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:39:41 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:39:41.890+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-03-31T23:38:51.226778+0000 front 2024-03-31T23:38:51.226798+0000 (oldest deadline 2024-03-31T23:39:15.926915+0000) 2024-03-31T23:39:42.296 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:39:41 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:39:41.890+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.46:6822 osd.6 since back 2024-03-31T23:39:01.140495+0000 front 2024-03-31T23:39:01.140543+0000 (oldest deadline 2024-03-31T23:39:24.640339+0000) 2024-03-31T23:39:42.296 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:39:41 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:39:41.890+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.142:6820 osd.7 since back 2024-03-31T23:39:08.041477+0000 front 2024-03-31T23:39:08.041456+0000 (oldest deadline 2024-03-31T23:39:32.741460+0000) 2024-03-31T23:39:43.295 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:39:42 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:39:42.866+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-31T23:38:12.221366+0000 front 2024-03-31T23:38:12.221347+0000 (oldest deadline 2024-03-31T23:38:37.521357+0000) 2024-03-31T23:39:43.296 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:39:42 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:39:42.866+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.142:6804 osd.1 since back 2024-03-31T23:38:22.222192+0000 front 2024-03-31T23:38:22.222213+0000 (oldest deadline 2024-03-31T23:38:45.722340+0000) 2024-03-31T23:39:43.296 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:39:42 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:39:42.866+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-31T23:38:29.723616+0000 front 2024-03-31T23:38:29.723593+0000 (oldest deadline 2024-03-31T23:38:52.023779+0000) 2024-03-31T23:39:43.296 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:39:42 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:39:42.866+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-03-31T23:38:36.124750+0000 front 2024-03-31T23:38:36.124868+0000 (oldest deadline 2024-03-31T23:39:00.824619+0000) 2024-03-31T23:39:43.296 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:39:42 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:39:42.866+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.142:6812 osd.4 since back 2024-03-31T23:38:44.825661+0000 front 2024-03-31T23:38:44.825679+0000 (oldest deadline 2024-03-31T23:39:10.125827+0000) 2024-03-31T23:39:43.296 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:39:42 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:39:42.866+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-03-31T23:38:51.226778+0000 front 2024-03-31T23:38:51.226798+0000 (oldest deadline 2024-03-31T23:39:15.926915+0000) 2024-03-31T23:39:43.296 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:39:42 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:39:42.866+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.46:6822 osd.6 since back 2024-03-31T23:39:01.140495+0000 front 2024-03-31T23:39:01.140543+0000 (oldest deadline 2024-03-31T23:39:24.640339+0000) 2024-03-31T23:39:43.296 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:39:42 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:39:42.866+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.142:6820 osd.7 since back 2024-03-31T23:39:08.041477+0000 front 2024-03-31T23:39:08.041456+0000 (oldest deadline 2024-03-31T23:39:32.741460+0000) 2024-03-31T23:39:44.295 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:39:43 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:39:43.829+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-31T23:38:12.221366+0000 front 2024-03-31T23:38:12.221347+0000 (oldest deadline 2024-03-31T23:38:37.521357+0000) 2024-03-31T23:39:44.296 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:39:43 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:39:43.829+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.142:6804 osd.1 since back 2024-03-31T23:38:22.222192+0000 front 2024-03-31T23:38:22.222213+0000 (oldest deadline 2024-03-31T23:38:45.722340+0000) 2024-03-31T23:39:44.296 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:39:43 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:39:43.829+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-31T23:38:29.723616+0000 front 2024-03-31T23:38:29.723593+0000 (oldest deadline 2024-03-31T23:38:52.023779+0000) 2024-03-31T23:39:44.296 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:39:43 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:39:43.829+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-03-31T23:38:36.124750+0000 front 2024-03-31T23:38:36.124868+0000 (oldest deadline 2024-03-31T23:39:00.824619+0000) 2024-03-31T23:39:44.296 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:39:43 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:39:43.829+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.142:6812 osd.4 since back 2024-03-31T23:38:44.825661+0000 front 2024-03-31T23:38:44.825679+0000 (oldest deadline 2024-03-31T23:39:10.125827+0000) 2024-03-31T23:39:44.296 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:39:43 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:39:43.829+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-03-31T23:38:51.226778+0000 front 2024-03-31T23:38:51.226798+0000 (oldest deadline 2024-03-31T23:39:15.926915+0000) 2024-03-31T23:39:44.296 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:39:43 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:39:43.829+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.46:6822 osd.6 since back 2024-03-31T23:39:01.140495+0000 front 2024-03-31T23:39:01.140543+0000 (oldest deadline 2024-03-31T23:39:24.640339+0000) 2024-03-31T23:39:44.296 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:39:43 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:39:43.829+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.142:6820 osd.7 since back 2024-03-31T23:39:08.041477+0000 front 2024-03-31T23:39:08.041456+0000 (oldest deadline 2024-03-31T23:39:32.741460+0000) 2024-03-31T23:39:44.296 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:39:43 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11[62061]: 2024-03-31T23:39:43.829+0000 7f98d0338700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.162:6820 osd.8 since back 2024-03-31T23:39:19.142498+0000 front 2024-03-31T23:39:19.142514+0000 (oldest deadline 2024-03-31T23:39:43.242528+0000) 2024-03-31T23:39:44.761 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:39:44 smithi162 podman[78774]: 2024-03-31 23:39:44.434365058 +0000 UTC m=+5.508061680 container died 642aa2603b732b94e3405767bf1f541b94230c36672c16bf6eb10bb758e71488 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240326, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, RELEASE=squid-384f31f, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-03-31T23:39:45.045 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:39:44 smithi162 podman[78774]: 2024-03-31 23:39:44.7616756 +0000 UTC m=+5.835372232 container cleanup 642aa2603b732b94e3405767bf1f541b94230c36672c16bf6eb10bb758e71488 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11, ceph=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-384f31f, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240326, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-03-31T23:39:45.046 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:39:44 smithi162 bash[78774]: ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11 2024-03-31T23:39:45.046 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:39:44 smithi162 podman[78791]: 2024-03-31 23:39:44.937939777 +0000 UTC m=+0.498516301 container remove 642aa2603b732b94e3405767bf1f541b94230c36672c16bf6eb10bb758e71488 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=squid-384f31f, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240326, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, ceph=True, GIT_CLEAN=True) 2024-03-31T23:39:45.641 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:39:45 smithi162 podman[78882]: 2024-03-31 23:39:45.355481924 +0000 UTC m=+0.103597828 container create 98f3d4b32eeecdb62fb989e41c98c031529f0aaa166b88580f390b0e2e7574ba (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11-deactivate, CEPH_POINT_RELEASE=, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , RELEASE=squid-384f31f, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.build-date=20240326, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-31T23:39:45.641 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:39:45 smithi162 podman[78882]: 2024-03-31 23:39:45.270698707 +0000 UTC m=+0.018814636 image pull ec18d7baa60e8b1a0b126fbe5306acce11b1ed0ec5e3bcdddbf39b0a3c591791 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175 2024-03-31T23:39:45.641 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:39:45 smithi162 podman[78882]: 2024-03-31 23:39:45.566362471 +0000 UTC m=+0.314478391 container init 98f3d4b32eeecdb62fb989e41c98c031529f0aaa166b88580f390b0e2e7574ba (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11-deactivate, GIT_CLEAN=True, io.buildah.version=1.33.5, RELEASE=squid-384f31f, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240326, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-31T23:39:45.641 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:39:45 smithi162 podman[78882]: 2024-03-31 23:39:45.569541327 +0000 UTC m=+0.317657232 container start 98f3d4b32eeecdb62fb989e41c98c031529f0aaa166b88580f390b0e2e7574ba (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240326, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=squid-384f31f) 2024-03-31T23:39:45.641 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:39:45 smithi162 podman[78882]: 2024-03-31 23:39:45.641083682 +0000 UTC m=+0.389199602 container attach 98f3d4b32eeecdb62fb989e41c98c031529f0aaa166b88580f390b0e2e7574ba (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11-deactivate, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=squid-384f31f, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240326, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS) 2024-03-31T23:39:46.045 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 31 23:39:45 smithi162 podman[78882]: 2024-03-31 23:39:45.856829667 +0000 UTC m=+0.604945586 container died 98f3d4b32eeecdb62fb989e41c98c031529f0aaa166b88580f390b0e2e7574ba (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-osd-11-deactivate, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240326, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, ceph=True, RELEASE=squid-384f31f, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-03-31T23:39:46.423 DEBUG:teuthology.orchestra.run.smithi162:> sudo pkill -f 'journalctl -f -n 0 -u ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4@osd.11.service' 2024-03-31T23:39:46.467 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-31T23:39:46.468 INFO:tasks.cephadm.osd.11:Stopped osd.11 2024-03-31T23:39:46.468 INFO:tasks.cephadm.mds.a:Stopping mds.a... 2024-03-31T23:39:46.468 DEBUG:teuthology.orchestra.run.smithi046:> sudo systemctl stop ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4@mds.a 2024-03-31T23:39:46.768 INFO:journalctl@ceph.mds.a.smithi046.stdout:Mar 31 23:39:46 smithi046 systemd[1]: Stopping Ceph mds.a for 6a9d8f08-efb2-11ee-b647-cb9ed24678a4... 2024-03-31T23:39:47.044 INFO:journalctl@ceph.mds.a.smithi046.stdout:Mar 31 23:39:46 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mds-a[74047]: 2024-03-31T23:39:46.767+0000 7f20fb3f0700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-31T23:39:47.044 INFO:journalctl@ceph.mds.a.smithi046.stdout:Mar 31 23:39:46 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mds-a[74047]: 2024-03-31T23:39:46.767+0000 7f20fb3f0700 -1 mds.a *** got signal Terminated *** 2024-03-31T23:39:48.862 INFO:journalctl@ceph.mds.a.smithi046.stdout:Mar 31 23:39:48 smithi046 podman[108317]: 2024-03-31 23:39:48.428421912 +0000 UTC m=+1.751830667 container died c6deca2be3eca8883012d6efae149642a0986a2f39ea83312bc465081a3e7b7c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mds-a, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240326, GIT_BRANCH=HEAD, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=squid-384f31f, org.label-schema.vendor=CentOS, ceph=True) 2024-03-31T23:39:49.160 INFO:journalctl@ceph.mds.a.smithi046.stdout:Mar 31 23:39:48 smithi046 podman[108317]: 2024-03-31 23:39:48.950804181 +0000 UTC m=+2.274212923 container cleanup c6deca2be3eca8883012d6efae149642a0986a2f39ea83312bc465081a3e7b7c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mds-a, io.buildah.version=1.33.5, RELEASE=squid-384f31f, org.label-schema.schema-version=1.0, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240326, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-03-31T23:39:49.160 INFO:journalctl@ceph.mds.a.smithi046.stdout:Mar 31 23:39:48 smithi046 bash[108317]: ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mds-a 2024-03-31T23:39:49.397 DEBUG:teuthology.orchestra.run.smithi046:> sudo pkill -f 'journalctl -f -n 0 -u ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4@mds.a.service' 2024-03-31T23:39:49.421 INFO:journalctl@ceph.mds.a.smithi046.stdout:Mar 31 23:39:49 smithi046 podman[108344]: 2024-03-31 23:39:49.160624725 +0000 UTC m=+0.726296930 container remove c6deca2be3eca8883012d6efae149642a0986a2f39ea83312bc465081a3e7b7c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mds-a, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=squid-384f31f, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240326) 2024-03-31T23:39:49.422 INFO:journalctl@ceph.mds.a.smithi046.stdout:Mar 31 23:39:49 smithi046 systemd[1]: ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4@mds.a.service: Deactivated successfully. 2024-03-31T23:39:49.422 INFO:journalctl@ceph.mds.a.smithi046.stdout:Mar 31 23:39:49 smithi046 systemd[1]: Stopped Ceph mds.a for 6a9d8f08-efb2-11ee-b647-cb9ed24678a4. 2024-03-31T23:39:49.422 INFO:journalctl@ceph.mds.a.smithi046.stdout:Mar 31 23:39:49 smithi046 systemd[1]: ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4@mds.a.service: Consumed 1.213s CPU time. 2024-03-31T23:39:49.436 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-31T23:39:49.436 INFO:tasks.cephadm.mds.a:Stopped mds.a 2024-03-31T23:39:49.436 INFO:tasks.cephadm.mds.d:Stopping mds.d... 2024-03-31T23:39:49.436 DEBUG:teuthology.orchestra.run.smithi046:> sudo systemctl stop ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4@mds.d 2024-03-31T23:39:49.755 INFO:journalctl@ceph.mds.d.smithi046.stdout:Mar 31 23:39:49 smithi046 systemd[1]: Stopping Ceph mds.d for 6a9d8f08-efb2-11ee-b647-cb9ed24678a4... 2024-03-31T23:39:50.111 INFO:journalctl@ceph.mds.d.smithi046.stdout:Mar 31 23:39:49 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mds-d[73088]: 2024-03-31T23:39:49.754+0000 7f3dafe4f700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.d -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-31T23:39:50.111 INFO:journalctl@ceph.mds.d.smithi046.stdout:Mar 31 23:39:49 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mds-d[73088]: 2024-03-31T23:39:49.754+0000 7f3dafe4f700 -1 mds.d *** got signal Terminated *** 2024-03-31T23:39:52.714 INFO:journalctl@ceph.mds.d.smithi046.stdout:Mar 31 23:39:52 smithi046 podman[108450]: 2024-03-31 23:39:52.463003956 +0000 UTC m=+2.808255436 container died 70ad99574a650e8671f156263068ada2ba0b90982c33115aba4922ef80e522a9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mds-d, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-384f31f, org.label-schema.build-date=20240326, io.buildah.version=1.33.5, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux ) 2024-03-31T23:39:52.997 INFO:journalctl@ceph.mds.d.smithi046.stdout:Mar 31 23:39:52 smithi046 podman[108450]: 2024-03-31 23:39:52.804536484 +0000 UTC m=+3.149787963 container cleanup 70ad99574a650e8671f156263068ada2ba0b90982c33115aba4922ef80e522a9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mds-d, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240326, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, ceph=True, RELEASE=squid-384f31f, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-03-31T23:39:52.998 INFO:journalctl@ceph.mds.d.smithi046.stdout:Mar 31 23:39:52 smithi046 bash[108450]: ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mds-d 2024-03-31T23:39:53.212 DEBUG:teuthology.orchestra.run.smithi046:> sudo pkill -f 'journalctl -f -n 0 -u ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4@mds.d.service' 2024-03-31T23:39:53.248 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-31T23:39:53.248 INFO:tasks.cephadm.mds.d:Stopped mds.d 2024-03-31T23:39:53.248 INFO:tasks.cephadm.mds.g:Stopping mds.g... 2024-03-31T23:39:53.248 DEBUG:teuthology.orchestra.run.smithi046:> sudo systemctl stop ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4@mds.g 2024-03-31T23:39:53.611 INFO:journalctl@ceph.mds.g.smithi046.stdout:Mar 31 23:39:53 smithi046 systemd[1]: Stopping Ceph mds.g for 6a9d8f08-efb2-11ee-b647-cb9ed24678a4... 2024-03-31T23:39:53.611 INFO:journalctl@ceph.mds.g.smithi046.stdout:Mar 31 23:39:53 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mds-g[73684]: 2024-03-31T23:39:53.557+0000 7f284ca53700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.g -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-31T23:39:53.611 INFO:journalctl@ceph.mds.g.smithi046.stdout:Mar 31 23:39:53 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mds-g[73684]: 2024-03-31T23:39:53.557+0000 7f284ca53700 -1 mds.g *** got signal Terminated *** 2024-03-31T23:39:57.750 INFO:journalctl@ceph.mds.g.smithi046.stdout:Mar 31 23:39:57 smithi046 podman[108571]: 2024-03-31 23:39:57.424407884 +0000 UTC m=+3.957525704 container died b4331229633e8ef56dc7c1666bed6b78cabde29bdf8fe3bda0f4fd8f459732cd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mds-g, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-384f31f, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.build-date=20240326, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0) 2024-03-31T23:39:58.082 INFO:journalctl@ceph.mds.g.smithi046.stdout:Mar 31 23:39:57 smithi046 podman[108571]: 2024-03-31 23:39:57.890689404 +0000 UTC m=+4.423807221 container cleanup b4331229633e8ef56dc7c1666bed6b78cabde29bdf8fe3bda0f4fd8f459732cd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mds-g, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240326, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_BRANCH=HEAD, RELEASE=squid-384f31f) 2024-03-31T23:39:58.083 INFO:journalctl@ceph.mds.g.smithi046.stdout:Mar 31 23:39:57 smithi046 bash[108571]: ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mds-g 2024-03-31T23:39:58.322 DEBUG:teuthology.orchestra.run.smithi046:> sudo pkill -f 'journalctl -f -n 0 -u ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4@mds.g.service' 2024-03-31T23:39:58.346 INFO:journalctl@ceph.mds.g.smithi046.stdout:Mar 31 23:39:58 smithi046 podman[108585]: 2024-03-31 23:39:58.193257537 +0000 UTC m=+0.766820138 container remove b4331229633e8ef56dc7c1666bed6b78cabde29bdf8fe3bda0f4fd8f459732cd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mds-g, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240326, org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=, RELEASE=squid-384f31f, GIT_CLEAN=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-03-31T23:39:58.346 INFO:journalctl@ceph.mds.g.smithi046.stdout:Mar 31 23:39:58 smithi046 systemd[1]: ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4@mds.g.service: Deactivated successfully. 2024-03-31T23:39:58.346 INFO:journalctl@ceph.mds.g.smithi046.stdout:Mar 31 23:39:58 smithi046 systemd[1]: Stopped Ceph mds.g for 6a9d8f08-efb2-11ee-b647-cb9ed24678a4. 2024-03-31T23:39:58.346 INFO:journalctl@ceph.mds.g.smithi046.stdout:Mar 31 23:39:58 smithi046 systemd[1]: ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4@mds.g.service: Consumed 1.191s CPU time. 2024-03-31T23:39:58.357 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-31T23:39:58.357 INFO:tasks.cephadm.mds.g:Stopped mds.g 2024-03-31T23:39:58.357 INFO:tasks.cephadm.mds.j:Stopping mds.j... 2024-03-31T23:39:58.357 DEBUG:teuthology.orchestra.run.smithi046:> sudo systemctl stop ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4@mds.j 2024-03-31T23:39:58.611 INFO:journalctl@ceph.mds.j.smithi046.stdout:Mar 31 23:39:58 smithi046 systemd[1]: Stopping Ceph mds.j for 6a9d8f08-efb2-11ee-b647-cb9ed24678a4... 2024-03-31T23:39:59.111 INFO:journalctl@ceph.mds.j.smithi046.stdout:Mar 31 23:39:58 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mds-j[72738]: 2024-03-31T23:39:58.703+0000 7fe084db8700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.j -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-31T23:39:59.111 INFO:journalctl@ceph.mds.j.smithi046.stdout:Mar 31 23:39:58 smithi046 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mds-j[72738]: 2024-03-31T23:39:58.703+0000 7fe084db8700 -1 mds.j *** got signal Terminated *** 2024-03-31T23:40:01.862 INFO:journalctl@ceph.mds.j.smithi046.stdout:Mar 31 23:40:01 smithi046 podman[108693]: 2024-03-31 23:40:01.53055886 +0000 UTC m=+2.956472650 container died 85b09f805600e0c06cc942a5c6e2f23ecb5eb992bc83785e7914324991fb49e4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mds-j, ceph=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240326, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-384f31f) 2024-03-31T23:40:02.246 INFO:journalctl@ceph.mds.j.smithi046.stdout:Mar 31 23:40:02 smithi046 podman[108693]: 2024-03-31 23:40:02.004382979 +0000 UTC m=+3.430296773 container cleanup 85b09f805600e0c06cc942a5c6e2f23ecb5eb992bc83785e7914324991fb49e4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mds-j, GIT_BRANCH=HEAD, RELEASE=squid-384f31f, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240326, maintainer=Guillaume Abrioux , GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0) 2024-03-31T23:40:02.247 INFO:journalctl@ceph.mds.j.smithi046.stdout:Mar 31 23:40:02 smithi046 bash[108693]: ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mds-j 2024-03-31T23:40:02.385 DEBUG:teuthology.orchestra.run.smithi046:> sudo pkill -f 'journalctl -f -n 0 -u ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4@mds.j.service' 2024-03-31T23:40:02.420 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-31T23:40:02.420 INFO:tasks.cephadm.mds.j:Stopped mds.j 2024-03-31T23:40:02.420 INFO:tasks.cephadm.mds.b:Stopping mds.b... 2024-03-31T23:40:02.421 DEBUG:teuthology.orchestra.run.smithi142:> sudo systemctl stop ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4@mds.b 2024-03-31T23:40:02.771 INFO:journalctl@ceph.mds.b.smithi142.stdout:Mar 31 23:40:02 smithi142 systemd[1]: Stopping Ceph mds.b for 6a9d8f08-efb2-11ee-b647-cb9ed24678a4... 2024-03-31T23:40:03.193 INFO:journalctl@ceph.mds.b.smithi142.stdout:Mar 31 23:40:02 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mds-b[65801]: 2024-03-31T23:40:02.770+0000 7fb48aa68700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-31T23:40:03.193 INFO:journalctl@ceph.mds.b.smithi142.stdout:Mar 31 23:40:02 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mds-b[65801]: 2024-03-31T23:40:02.770+0000 7fb48aa68700 -1 mds.b *** got signal Terminated *** 2024-03-31T23:40:13.162 INFO:journalctl@ceph.mds.b.smithi142.stdout:Mar 31 23:40:12 smithi142 bash[78614]: time="2024-03-31T23:40:12Z" level=warning msg="StopSignal SIGTERM failed to stop container ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mds-b in 10 seconds, resorting to SIGKILL" 2024-03-31T23:40:13.163 INFO:journalctl@ceph.mds.b.smithi142.stdout:Mar 31 23:40:12 smithi142 podman[78614]: 2024-03-31 23:40:12.791770611 +0000 UTC m=+10.163051386 container died 162eacae90f31163644e60f2d674f1568d00a26f04ace8df61f9c48322607d7c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mds-b, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, RELEASE=squid-384f31f, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240326, org.label-schema.schema-version=1.0) 2024-03-31T23:40:13.443 INFO:journalctl@ceph.mds.b.smithi142.stdout:Mar 31 23:40:13 smithi142 podman[78614]: 2024-03-31 23:40:13.25331191 +0000 UTC m=+10.624592688 container cleanup 162eacae90f31163644e60f2d674f1568d00a26f04ace8df61f9c48322607d7c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mds-b, RELEASE=squid-384f31f, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240326, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-03-31T23:40:13.443 INFO:journalctl@ceph.mds.b.smithi142.stdout:Mar 31 23:40:13 smithi142 bash[78614]: ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mds-b 2024-03-31T23:40:13.730 DEBUG:teuthology.orchestra.run.smithi142:> sudo pkill -f 'journalctl -f -n 0 -u ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4@mds.b.service' 2024-03-31T23:40:13.769 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-31T23:40:13.770 INFO:tasks.cephadm.mds.b:Stopped mds.b 2024-03-31T23:40:13.770 INFO:tasks.cephadm.mds.e:Stopping mds.e... 2024-03-31T23:40:13.770 DEBUG:teuthology.orchestra.run.smithi142:> sudo systemctl stop ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4@mds.e 2024-03-31T23:40:14.108 INFO:journalctl@ceph.mds.e.smithi142.stdout:Mar 31 23:40:13 smithi142 systemd[1]: Stopping Ceph mds.e for 6a9d8f08-efb2-11ee-b647-cb9ed24678a4... 2024-03-31T23:40:14.443 INFO:journalctl@ceph.mds.e.smithi142.stdout:Mar 31 23:40:14 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mds-e[65175]: 2024-03-31T23:40:14.107+0000 7ff706db1700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.e -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-31T23:40:14.443 INFO:journalctl@ceph.mds.e.smithi142.stdout:Mar 31 23:40:14 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mds-e[65175]: 2024-03-31T23:40:14.107+0000 7ff706db1700 -1 mds.e *** got signal Terminated *** 2024-03-31T23:40:15.674 INFO:journalctl@ceph.mds.e.smithi142.stdout:Mar 31 23:40:15 smithi142 podman[78749]: 2024-03-31 23:40:15.397555543 +0000 UTC m=+1.388880321 container died 42bbbb0a6bd12a82c3f618cd2f886d796c00034b4da56fced57d1e73ad8bb973 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mds-e, RELEASE=squid-384f31f, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240326, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, ceph=True) 2024-03-31T23:40:15.943 INFO:journalctl@ceph.mds.e.smithi142.stdout:Mar 31 23:40:15 smithi142 podman[78749]: 2024-03-31 23:40:15.765401899 +0000 UTC m=+1.756726674 container cleanup 42bbbb0a6bd12a82c3f618cd2f886d796c00034b4da56fced57d1e73ad8bb973 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mds-e, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , RELEASE=squid-384f31f, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240326, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, io.buildah.version=1.33.5) 2024-03-31T23:40:15.943 INFO:journalctl@ceph.mds.e.smithi142.stdout:Mar 31 23:40:15 smithi142 bash[78749]: ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mds-e 2024-03-31T23:40:16.179 DEBUG:teuthology.orchestra.run.smithi142:> sudo pkill -f 'journalctl -f -n 0 -u ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4@mds.e.service' 2024-03-31T23:40:16.215 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-31T23:40:16.216 INFO:tasks.cephadm.mds.e:Stopped mds.e 2024-03-31T23:40:16.216 INFO:tasks.cephadm.mds.h:Stopping mds.h... 2024-03-31T23:40:16.216 DEBUG:teuthology.orchestra.run.smithi142:> sudo systemctl stop ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4@mds.h 2024-03-31T23:40:16.443 INFO:journalctl@ceph.mds.h.smithi142.stdout:Mar 31 23:40:16 smithi142 systemd[1]: Stopping Ceph mds.h for 6a9d8f08-efb2-11ee-b647-cb9ed24678a4... 2024-03-31T23:40:16.943 INFO:journalctl@ceph.mds.h.smithi142.stdout:Mar 31 23:40:16 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mds-h[65485]: 2024-03-31T23:40:16.560+0000 7f4374cf4700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.h -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-31T23:40:16.943 INFO:journalctl@ceph.mds.h.smithi142.stdout:Mar 31 23:40:16 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mds-h[65485]: 2024-03-31T23:40:16.560+0000 7f4374cf4700 -1 mds.h *** got signal Terminated *** 2024-03-31T23:40:20.943 INFO:journalctl@ceph.mds.h.smithi142.stdout:Mar 31 23:40:20 smithi142 podman[78872]: 2024-03-31 23:40:20.674581479 +0000 UTC m=+4.215394232 container died eeb232fe0e9b59716285ca3939bc12186a76ea51e560c71ecc84a77f46f3487c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mds-h, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240326, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-384f31f, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-03-31T23:40:21.443 INFO:journalctl@ceph.mds.h.smithi142.stdout:Mar 31 23:40:21 smithi142 podman[78872]: 2024-03-31 23:40:21.244438807 +0000 UTC m=+4.785251539 container cleanup eeb232fe0e9b59716285ca3939bc12186a76ea51e560c71ecc84a77f46f3487c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mds-h, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-384f31f, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240326, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-03-31T23:40:21.443 INFO:journalctl@ceph.mds.h.smithi142.stdout:Mar 31 23:40:21 smithi142 bash[78872]: ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mds-h 2024-03-31T23:40:21.614 DEBUG:teuthology.orchestra.run.smithi142:> sudo pkill -f 'journalctl -f -n 0 -u ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4@mds.h.service' 2024-03-31T23:40:21.651 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-31T23:40:21.651 INFO:tasks.cephadm.mds.h:Stopped mds.h 2024-03-31T23:40:21.652 INFO:tasks.cephadm.mds.k:Stopping mds.k... 2024-03-31T23:40:21.652 DEBUG:teuthology.orchestra.run.smithi142:> sudo systemctl stop ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4@mds.k 2024-03-31T23:40:22.193 INFO:journalctl@ceph.mds.k.smithi142.stdout:Mar 31 23:40:21 smithi142 systemd[1]: Stopping Ceph mds.k for 6a9d8f08-efb2-11ee-b647-cb9ed24678a4... 2024-03-31T23:40:22.193 INFO:journalctl@ceph.mds.k.smithi142.stdout:Mar 31 23:40:21 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mds-k[64855]: 2024-03-31T23:40:21.955+0000 7f0d0abcc700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.k -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-31T23:40:22.193 INFO:journalctl@ceph.mds.k.smithi142.stdout:Mar 31 23:40:21 smithi142 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mds-k[64855]: 2024-03-31T23:40:21.955+0000 7f0d0abcc700 -1 mds.k *** got signal Terminated *** 2024-03-31T23:40:26.356 INFO:journalctl@ceph.mds.k.smithi142.stdout:Mar 31 23:40:25 smithi142 podman[78992]: 2024-03-31 23:40:25.964474808 +0000 UTC m=+4.099639361 container died 7f5af54f1631112124998a5118513482bd5651538b1a1d1b6996134446f5a123 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mds-k, RELEASE=squid-384f31f, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240326, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-31T23:40:26.632 INFO:journalctl@ceph.mds.k.smithi142.stdout:Mar 31 23:40:26 smithi142 podman[78992]: 2024-03-31 23:40:26.439244919 +0000 UTC m=+4.574409456 container cleanup 7f5af54f1631112124998a5118513482bd5651538b1a1d1b6996134446f5a123 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mds-k, GIT_CLEAN=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=squid-384f31f, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240326) 2024-03-31T23:40:26.632 INFO:journalctl@ceph.mds.k.smithi142.stdout:Mar 31 23:40:26 smithi142 bash[78992]: ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mds-k 2024-03-31T23:40:26.857 DEBUG:teuthology.orchestra.run.smithi142:> sudo pkill -f 'journalctl -f -n 0 -u ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4@mds.k.service' 2024-03-31T23:40:26.882 INFO:journalctl@ceph.mds.k.smithi142.stdout:Mar 31 23:40:26 smithi142 podman[79006]: 2024-03-31 23:40:26.632083358 +0000 UTC m=+0.625711735 container remove 7f5af54f1631112124998a5118513482bd5651538b1a1d1b6996134446f5a123 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mds-k, GIT_BRANCH=HEAD, RELEASE=squid-384f31f, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.build-date=20240326, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, ceph=True) 2024-03-31T23:40:26.882 INFO:journalctl@ceph.mds.k.smithi142.stdout:Mar 31 23:40:26 smithi142 systemd[1]: ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4@mds.k.service: Deactivated successfully. 2024-03-31T23:40:26.882 INFO:journalctl@ceph.mds.k.smithi142.stdout:Mar 31 23:40:26 smithi142 systemd[1]: Stopped Ceph mds.k for 6a9d8f08-efb2-11ee-b647-cb9ed24678a4. 2024-03-31T23:40:26.882 INFO:journalctl@ceph.mds.k.smithi142.stdout:Mar 31 23:40:26 smithi142 systemd[1]: ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4@mds.k.service: Consumed 1.153s CPU time. 2024-03-31T23:40:26.891 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-31T23:40:26.891 INFO:tasks.cephadm.mds.k:Stopped mds.k 2024-03-31T23:40:26.891 INFO:tasks.cephadm.mds.c:Stopping mds.c... 2024-03-31T23:40:26.891 DEBUG:teuthology.orchestra.run.smithi162:> sudo systemctl stop ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4@mds.c 2024-03-31T23:40:27.186 INFO:journalctl@ceph.mds.c.smithi162.stdout:Mar 31 23:40:26 smithi162 systemd[1]: Stopping Ceph mds.c for 6a9d8f08-efb2-11ee-b647-cb9ed24678a4... 2024-03-31T23:40:27.545 INFO:journalctl@ceph.mds.c.smithi162.stdout:Mar 31 23:40:27 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mds-c[67067]: 2024-03-31T23:40:27.185+0000 7f45e6812700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-31T23:40:27.545 INFO:journalctl@ceph.mds.c.smithi162.stdout:Mar 31 23:40:27 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mds-c[67067]: 2024-03-31T23:40:27.185+0000 7f45e6812700 -1 mds.c *** got signal Terminated *** 2024-03-31T23:40:31.793 INFO:journalctl@ceph.mds.c.smithi162.stdout:Mar 31 23:40:31 smithi162 podman[79000]: 2024-03-31 23:40:31.517980701 +0000 UTC m=+4.438732423 container died 6c86ca0bcc43acac46a1a8136c7b9e481e6b83fddedbe03afe393f331be9d4ac (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mds-c, RELEASE=squid-384f31f, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240326, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-31T23:40:32.046 INFO:journalctl@ceph.mds.c.smithi162.stdout:Mar 31 23:40:31 smithi162 podman[79000]: 2024-03-31 23:40:31.893896832 +0000 UTC m=+4.814648543 container cleanup 6c86ca0bcc43acac46a1a8136c7b9e481e6b83fddedbe03afe393f331be9d4ac (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mds-c, maintainer=Guillaume Abrioux , RELEASE=squid-384f31f, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240326, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.33.5, ceph=True) 2024-03-31T23:40:32.046 INFO:journalctl@ceph.mds.c.smithi162.stdout:Mar 31 23:40:31 smithi162 bash[79000]: ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mds-c 2024-03-31T23:40:32.274 DEBUG:teuthology.orchestra.run.smithi162:> sudo pkill -f 'journalctl -f -n 0 -u ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4@mds.c.service' 2024-03-31T23:40:32.312 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-31T23:40:32.312 INFO:tasks.cephadm.mds.c:Stopped mds.c 2024-03-31T23:40:32.313 INFO:tasks.cephadm.mds.f:Stopping mds.f... 2024-03-31T23:40:32.313 DEBUG:teuthology.orchestra.run.smithi162:> sudo systemctl stop ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4@mds.f 2024-03-31T23:40:32.545 INFO:journalctl@ceph.mds.f.smithi162.stdout:Mar 31 23:40:32 smithi162 systemd[1]: Stopping Ceph mds.f for 6a9d8f08-efb2-11ee-b647-cb9ed24678a4... 2024-03-31T23:40:33.045 INFO:journalctl@ceph.mds.f.smithi162.stdout:Mar 31 23:40:32 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mds-f[66752]: 2024-03-31T23:40:32.630+0000 7f08ddf03700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.f -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-31T23:40:33.046 INFO:journalctl@ceph.mds.f.smithi162.stdout:Mar 31 23:40:32 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mds-f[66752]: 2024-03-31T23:40:32.630+0000 7f08ddf03700 -1 mds.f *** got signal Terminated *** 2024-03-31T23:40:34.515 INFO:journalctl@ceph.mds.f.smithi162.stdout:Mar 31 23:40:34 smithi162 podman[79120]: 2024-03-31 23:40:34.10673953 +0000 UTC m=+1.566449427 container died b67d463eb8fee0a0edec3829c2ad68283d3970498bb5c32956d5eece780a6166 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mds-f, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-384f31f, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, io.buildah.version=1.33.5, org.label-schema.build-date=20240326, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-03-31T23:40:34.795 INFO:journalctl@ceph.mds.f.smithi162.stdout:Mar 31 23:40:34 smithi162 podman[79120]: 2024-03-31 23:40:34.5895257 +0000 UTC m=+2.049235606 container cleanup b67d463eb8fee0a0edec3829c2ad68283d3970498bb5c32956d5eece780a6166 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mds-f, org.label-schema.build-date=20240326, org.label-schema.schema-version=1.0, RELEASE=squid-384f31f, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-31T23:40:34.795 INFO:journalctl@ceph.mds.f.smithi162.stdout:Mar 31 23:40:34 smithi162 bash[79120]: ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mds-f 2024-03-31T23:40:35.013 DEBUG:teuthology.orchestra.run.smithi162:> sudo pkill -f 'journalctl -f -n 0 -u ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4@mds.f.service' 2024-03-31T23:40:35.047 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-31T23:40:35.048 INFO:tasks.cephadm.mds.f:Stopped mds.f 2024-03-31T23:40:35.048 INFO:tasks.cephadm.mds.i:Stopping mds.i... 2024-03-31T23:40:35.048 DEBUG:teuthology.orchestra.run.smithi162:> sudo systemctl stop ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4@mds.i 2024-03-31T23:40:35.376 INFO:journalctl@ceph.mds.i.smithi162.stdout:Mar 31 23:40:35 smithi162 systemd[1]: Stopping Ceph mds.i for 6a9d8f08-efb2-11ee-b647-cb9ed24678a4... 2024-03-31T23:40:35.795 INFO:journalctl@ceph.mds.i.smithi162.stdout:Mar 31 23:40:35 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mds-i[67378]: 2024-03-31T23:40:35.375+0000 7fbb61789700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.i -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-31T23:40:35.795 INFO:journalctl@ceph.mds.i.smithi162.stdout:Mar 31 23:40:35 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mds-i[67378]: 2024-03-31T23:40:35.375+0000 7fbb61789700 -1 mds.i *** got signal Terminated *** 2024-03-31T23:40:39.545 INFO:journalctl@ceph.mds.i.smithi162.stdout:Mar 31 23:40:39 smithi162 podman[79241]: 2024-03-31 23:40:39.271631884 +0000 UTC m=+4.005379410 container died 608a1c3cee32562aa8619764769eccfbeaf3a7427e69faed2e6ad97595d517a0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mds-i, io.buildah.version=1.33.5, ceph=True, org.label-schema.build-date=20240326, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , RELEASE=squid-384f31f, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-31T23:40:40.043 INFO:journalctl@ceph.mds.i.smithi162.stdout:Mar 31 23:40:39 smithi162 podman[79241]: 2024-03-31 23:40:39.676758445 +0000 UTC m=+4.410505960 container cleanup 608a1c3cee32562aa8619764769eccfbeaf3a7427e69faed2e6ad97595d517a0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mds-i, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240326, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=squid-384f31f, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-03-31T23:40:40.044 INFO:journalctl@ceph.mds.i.smithi162.stdout:Mar 31 23:40:39 smithi162 bash[79241]: ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mds-i 2024-03-31T23:40:40.044 INFO:journalctl@ceph.mds.i.smithi162.stdout:Mar 31 23:40:39 smithi162 podman[79256]: 2024-03-31 23:40:39.836326448 +0000 UTC m=+0.562325168 container remove 608a1c3cee32562aa8619764769eccfbeaf3a7427e69faed2e6ad97595d517a0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mds-i, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.build-date=20240326, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-384f31f, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_CLEAN=True, ceph=True) 2024-03-31T23:40:40.068 DEBUG:teuthology.orchestra.run.smithi162:> sudo pkill -f 'journalctl -f -n 0 -u ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4@mds.i.service' 2024-03-31T23:40:40.102 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-31T23:40:40.102 INFO:tasks.cephadm.mds.i:Stopped mds.i 2024-03-31T23:40:40.103 INFO:tasks.cephadm.mds.l:Stopping mds.l... 2024-03-31T23:40:40.103 DEBUG:teuthology.orchestra.run.smithi162:> sudo systemctl stop ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4@mds.l 2024-03-31T23:40:40.430 INFO:journalctl@ceph.mds.l.smithi162.stdout:Mar 31 23:40:40 smithi162 systemd[1]: Stopping Ceph mds.l for 6a9d8f08-efb2-11ee-b647-cb9ed24678a4... 2024-03-31T23:40:40.795 INFO:journalctl@ceph.mds.l.smithi162.stdout:Mar 31 23:40:40 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mds-l[66444]: 2024-03-31T23:40:40.429+0000 7f3f14ce0700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.l -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-31T23:40:40.795 INFO:journalctl@ceph.mds.l.smithi162.stdout:Mar 31 23:40:40 smithi162 ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mds-l[66444]: 2024-03-31T23:40:40.429+0000 7f3f14ce0700 -1 mds.l *** got signal Terminated *** 2024-03-31T23:40:44.139 INFO:journalctl@ceph.mds.l.smithi162.stdout:Mar 31 23:40:43 smithi162 podman[79363]: 2024-03-31 23:40:43.885901558 +0000 UTC m=+3.562051994 container died dad2ed6cb60ba505d629e02b8f167a0456ceb3006caeacf01b3343199a32a4f5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mds-l, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , RELEASE=squid-384f31f, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.build-date=20240326, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0) 2024-03-31T23:40:44.439 INFO:journalctl@ceph.mds.l.smithi162.stdout:Mar 31 23:40:44 smithi162 podman[79363]: 2024-03-31 23:40:44.221722449 +0000 UTC m=+3.897872885 container cleanup dad2ed6cb60ba505d629e02b8f167a0456ceb3006caeacf01b3343199a32a4f5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mds-l, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240326, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, RELEASE=squid-384f31f, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_CLEAN=True, org.label-schema.vendor=CentOS) 2024-03-31T23:40:44.439 INFO:journalctl@ceph.mds.l.smithi162.stdout:Mar 31 23:40:44 smithi162 bash[79363]: ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4-mds-l 2024-03-31T23:40:44.661 DEBUG:teuthology.orchestra.run.smithi162:> sudo pkill -f 'journalctl -f -n 0 -u ceph-6a9d8f08-efb2-11ee-b647-cb9ed24678a4@mds.l.service' 2024-03-31T23:40:44.695 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-31T23:40:44.695 INFO:tasks.cephadm.mds.l:Stopped mds.l 2024-03-31T23:40:44.695 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 6a9d8f08-efb2-11ee-b647-cb9ed24678a4 --force --keep-logs 2024-03-31T23:40:44.894 INFO:teuthology.orchestra.run.smithi046.stdout:Deleting cluster with fsid: 6a9d8f08-efb2-11ee-b647-cb9ed24678a4 2024-03-31T23:40:47.440 DEBUG:teuthology.orchestra.run.smithi142:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 6a9d8f08-efb2-11ee-b647-cb9ed24678a4 --force --keep-logs 2024-03-31T23:40:47.632 INFO:teuthology.orchestra.run.smithi142.stdout:Deleting cluster with fsid: 6a9d8f08-efb2-11ee-b647-cb9ed24678a4 2024-03-31T23:40:50.189 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 6a9d8f08-efb2-11ee-b647-cb9ed24678a4 --force --keep-logs 2024-03-31T23:40:50.382 INFO:teuthology.orchestra.run.smithi162.stdout:Deleting cluster with fsid: 6a9d8f08-efb2-11ee-b647-cb9ed24678a4 2024-03-31T23:40:52.954 DEBUG:teuthology.orchestra.run.smithi046:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-31T23:40:52.989 DEBUG:teuthology.orchestra.run.smithi142:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-31T23:40:53.024 DEBUG:teuthology.orchestra.run.smithi162:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-31T23:40:53.053 INFO:tasks.cephadm:Archiving crash dumps... 2024-03-31T23:40:53.055 DEBUG:teuthology.misc:Transferring archived files from smithi046:/var/lib/ceph/6a9d8f08-efb2-11ee-b647-cb9ed24678a4/crash to /home/teuthworker/archive/teuthology-2024-03-31_21:24:02-fs-squid-distro-default-smithi/7633118/remote/smithi046/crash 2024-03-31T23:40:53.056 DEBUG:teuthology.orchestra.run.smithi046:> sudo tar c -f - -C /var/lib/ceph/6a9d8f08-efb2-11ee-b647-cb9ed24678a4/crash -- . 2024-03-31T23:40:53.092 INFO:teuthology.orchestra.run.smithi046.stderr:tar: /var/lib/ceph/6a9d8f08-efb2-11ee-b647-cb9ed24678a4/crash: Cannot open: No such file or directory 2024-03-31T23:40:53.092 INFO:teuthology.orchestra.run.smithi046.stderr:tar: Error is not recoverable: exiting now 2024-03-31T23:40:53.094 DEBUG:teuthology.misc:Transferring archived files from smithi142:/var/lib/ceph/6a9d8f08-efb2-11ee-b647-cb9ed24678a4/crash to /home/teuthworker/archive/teuthology-2024-03-31_21:24:02-fs-squid-distro-default-smithi/7633118/remote/smithi142/crash 2024-03-31T23:40:53.095 DEBUG:teuthology.orchestra.run.smithi142:> sudo tar c -f - -C /var/lib/ceph/6a9d8f08-efb2-11ee-b647-cb9ed24678a4/crash -- . 2024-03-31T23:40:53.137 INFO:teuthology.orchestra.run.smithi142.stderr:tar: /var/lib/ceph/6a9d8f08-efb2-11ee-b647-cb9ed24678a4/crash: Cannot open: No such file or directory 2024-03-31T23:40:53.137 INFO:teuthology.orchestra.run.smithi142.stderr:tar: Error is not recoverable: exiting now 2024-03-31T23:40:53.139 DEBUG:teuthology.misc:Transferring archived files from smithi162:/var/lib/ceph/6a9d8f08-efb2-11ee-b647-cb9ed24678a4/crash to /home/teuthworker/archive/teuthology-2024-03-31_21:24:02-fs-squid-distro-default-smithi/7633118/remote/smithi162/crash 2024-03-31T23:40:53.140 DEBUG:teuthology.orchestra.run.smithi162:> sudo tar c -f - -C /var/lib/ceph/6a9d8f08-efb2-11ee-b647-cb9ed24678a4/crash -- . 2024-03-31T23:40:53.181 INFO:teuthology.orchestra.run.smithi162.stderr:tar: /var/lib/ceph/6a9d8f08-efb2-11ee-b647-cb9ed24678a4/crash: Cannot open: No such file or directory 2024-03-31T23:40:53.181 INFO:teuthology.orchestra.run.smithi162.stderr:tar: Error is not recoverable: exiting now 2024-03-31T23:40:53.182 INFO:tasks.cephadm:Checking cluster log for badness... 2024-03-31T23:40:53.182 DEBUG:teuthology.orchestra.run.smithi046:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/6a9d8f08-efb2-11ee-b647-cb9ed24678a4/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v 'slow metadata IO' | egrep -v SLOW_OPS | egrep -v 'slow request' | head -n 1 2024-03-31T23:40:53.265 INFO:tasks.cephadm:Compressing logs... 2024-03-31T23:40:53.265 DEBUG:teuthology.orchestra.run.smithi046:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-31T23:40:53.267 DEBUG:teuthology.orchestra.run.smithi142:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-31T23:40:53.270 DEBUG:teuthology.orchestra.run.smithi162:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-31T23:40:53.302 INFO:teuthology.orchestra.run.smithi142.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-03-31T23:40:53.303 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-03-31T23:40:53.303 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/6a9d8f08-efb2-11ee-b647-cb9ed24678a4/ceph-osd.10.log 2024-03-31T23:40:53.303 INFO:teuthology.orchestra.run.smithi046.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-03-31T23:40:53.304 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/6a9d8f08-efb2-11ee-b647-cb9ed24678a4/ceph.audit.log 2024-03-31T23:40:53.304 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/6a9d8f08-efb2-11ee-b647-cb9ed24678a4/ceph-osd.10.log: gzip -5 --verbose -- /var/log/ceph/6a9d8f08-efb2-11ee-b647-cb9ed24678a4/ceph-mds.b.log 2024-03-31T23:40:53.304 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/6a9d8f08-efb2-11ee-b647-cb9ed24678a4/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/6a9d8f08-efb2-11ee-b647-cb9ed24678a4/ceph-osd.4.log 2024-03-31T23:40:53.308 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/6a9d8f08-efb2-11ee-b647-cb9ed24678a4/ceph-mds.b.log: gzip -5 --verbose -- /var/log/ceph/6a9d8f08-efb2-11ee-b647-cb9ed24678a4/ceph-osd.1.log 2024-03-31T23:40:53.308 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/6a9d8f08-efb2-11ee-b647-cb9ed24678a4/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/6a9d8f08-efb2-11ee-b647-cb9ed24678a4/ceph-mds.k.log 2024-03-31T23:40:53.308 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/6a9d8f08-efb2-11ee-b647-cb9ed24678a4/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/6a9d8f08-efb2-11ee-b647-cb9ed24678a4/ceph-mds.h.log 2024-03-31T23:40:53.308 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/6a9d8f08-efb2-11ee-b647-cb9ed24678a4/ceph-mds.k.log: gzip -5 --verbose -- /var/log/ceph/6a9d8f08-efb2-11ee-b647-cb9ed24678a4/ceph-osd.7.log 2024-03-31T23:40:53.308 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/6a9d8f08-efb2-11ee-b647-cb9ed24678a4/ceph-mds.h.log: gzip -5 --verbose -- /var/log/ceph/6a9d8f08-efb2-11ee-b647-cb9ed24678a4/ceph-mds.e.log 2024-03-31T23:40:53.308 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/6a9d8f08-efb2-11ee-b647-cb9ed24678a4/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/6a9d8f08-efb2-11ee-b647-cb9ed24678a4/ceph-mgr.y.log 2024-03-31T23:40:53.308 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/6a9d8f08-efb2-11ee-b647-cb9ed24678a4/ceph-mds.e.log: gzip -5 --verbose -- /var/log/ceph/6a9d8f08-efb2-11ee-b647-cb9ed24678a4/ceph.cephadm.log 2024-03-31T23:40:53.308 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/6a9d8f08-efb2-11ee-b647-cb9ed24678a4/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/6a9d8f08-efb2-11ee-b647-cb9ed24678a4/ceph.log 2024-03-31T23:40:53.308 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/6a9d8f08-efb2-11ee-b647-cb9ed24678a4/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/6a9d8f08-efb2-11ee-b647-cb9ed24678a4/ceph-volume.log 2024-03-31T23:40:53.308 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/6a9d8f08-efb2-11ee-b647-cb9ed24678a4/ceph.log: gzip -5 --verbose -- /var/log/ceph/6a9d8f08-efb2-11ee-b647-cb9ed24678a4/ceph-mon.b.log 2024-03-31T23:40:53.309 INFO:teuthology.orchestra.run.smithi162.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-03-31T23:40:53.310 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-03-31T23:40:53.310 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.log 2024-03-31T23:40:53.310 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/6a9d8f08-efb2-11ee-b647-cb9ed24678a4/ceph-osd.6.log 2024-03-31T23:40:53.310 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/cephadm.log: /var/log/ceph/ceph-client.0.log: gzip -5 --verbose -- /var/log/ceph/6a9d8f08-efb2-11ee-b647-cb9ed24678a4/ceph-mon.a.log 2024-03-31T23:40:53.310 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/6a9d8f08-efb2-11ee-b647-cb9ed24678a4/ceph.audit.log 2024-03-31T23:40:53.314 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/6a9d8f08-efb2-11ee-b647-cb9ed24678a4/ceph-osd.6.log: /var/log/ceph/6a9d8f08-efb2-11ee-b647-cb9ed24678a4/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/6a9d8f08-efb2-11ee-b647-cb9ed24678a4/ceph-mds.g.log 2024-03-31T23:40:53.314 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/6a9d8f08-efb2-11ee-b647-cb9ed24678a4/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/6a9d8f08-efb2-11ee-b647-cb9ed24678a4/ceph-mds.j.log 2024-03-31T23:40:53.314 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/6a9d8f08-efb2-11ee-b647-cb9ed24678a4/ceph-mds.g.log: gzip -5 --verbose -- /var/log/ceph/6a9d8f08-efb2-11ee-b647-cb9ed24678a4/ceph-osd.3.log 2024-03-31T23:40:53.314 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/6a9d8f08-efb2-11ee-b647-cb9ed24678a4/ceph-mds.j.log: gzip -5 --verbose -- /var/log/ceph/6a9d8f08-efb2-11ee-b647-cb9ed24678a4/ceph-mgr.x.log 2024-03-31T23:40:53.314 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/6a9d8f08-efb2-11ee-b647-cb9ed24678a4/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/6a9d8f08-efb2-11ee-b647-cb9ed24678a4/ceph-osd.0.log 2024-03-31T23:40:53.315 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/6a9d8f08-efb2-11ee-b647-cb9ed24678a4/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/6a9d8f08-efb2-11ee-b647-cb9ed24678a4/ceph.cephadm.log 2024-03-31T23:40:53.315 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/6a9d8f08-efb2-11ee-b647-cb9ed24678a4/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/6a9d8f08-efb2-11ee-b647-cb9ed24678a4/ceph.log 2024-03-31T23:40:53.315 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/6a9d8f08-efb2-11ee-b647-cb9ed24678a4/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/6a9d8f08-efb2-11ee-b647-cb9ed24678a4/ceph-volume.log 2024-03-31T23:40:53.315 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/6a9d8f08-efb2-11ee-b647-cb9ed24678a4/ceph.log: gzip -5 --verbose -- /var/log/ceph/6a9d8f08-efb2-11ee-b647-cb9ed24678a4/ceph-osd.9.log 2024-03-31T23:40:53.315 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/6a9d8f08-efb2-11ee-b647-cb9ed24678a4/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/6a9d8f08-efb2-11ee-b647-cb9ed24678a4/ceph-mds.d.log 2024-03-31T23:40:53.315 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/6a9d8f08-efb2-11ee-b647-cb9ed24678a4/ceph-osd.9.log: gzip -5 --verbose -- /var/log/ceph/6a9d8f08-efb2-11ee-b647-cb9ed24678a4/ceph-mds.a.log 2024-03-31T23:40:53.315 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/6a9d8f08-efb2-11ee-b647-cb9ed24678a4/ceph-mds.d.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.log 2024-03-31T23:40:53.315 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/6a9d8f08-efb2-11ee-b647-cb9ed24678a4/ceph-mds.a.log: /var/log/ceph/ceph-client.admin.log: 87.4% -- replaced with /var/log/ceph/6a9d8f08-efb2-11ee-b647-cb9ed24678a4/ceph.log.gz 2024-03-31T23:40:53.315 INFO:teuthology.orchestra.run.smithi046.stderr: 94.6% -- replaced with /var/log/ceph/6a9d8f08-efb2-11ee-b647-cb9ed24678a4/ceph-mds.g.log.gz 2024-03-31T23:40:53.316 INFO:teuthology.orchestra.run.smithi162.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-03-31T23:40:53.317 INFO:teuthology.orchestra.run.smithi162.stderr:gzip -5 --verbose -- /var/log/ceph/6a9d8f08-efb2-11ee-b647-cb9ed24678a4/ceph-mds.l.log 2024-03-31T23:40:53.317 INFO:teuthology.orchestra.run.smithi162.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/6a9d8f08-efb2-11ee-b647-cb9ed24678a4/ceph.audit.log 2024-03-31T23:40:53.317 INFO:teuthology.orchestra.run.smithi162.stderr:/var/log/ceph/6a9d8f08-efb2-11ee-b647-cb9ed24678a4/ceph-mds.l.log: gzip -5 --verbose -- /var/log/ceph/6a9d8f08-efb2-11ee-b647-cb9ed24678a4/ceph-mds.i.log 2024-03-31T23:40:53.320 INFO:teuthology.orchestra.run.smithi162.stderr:/var/log/ceph/6a9d8f08-efb2-11ee-b647-cb9ed24678a4/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/6a9d8f08-efb2-11ee-b647-cb9ed24678a4/ceph-osd.11.log 2024-03-31T23:40:53.320 INFO:teuthology.orchestra.run.smithi162.stderr:/var/log/ceph/6a9d8f08-efb2-11ee-b647-cb9ed24678a4/ceph-mds.i.log: gzip -5 --verbose -- /var/log/ceph/6a9d8f08-efb2-11ee-b647-cb9ed24678a4/ceph-mgr.z.log 2024-03-31T23:40:53.320 INFO:teuthology.orchestra.run.smithi162.stderr:/var/log/ceph/6a9d8f08-efb2-11ee-b647-cb9ed24678a4/ceph-osd.11.log: gzip -5 --verbose -- /var/log/ceph/6a9d8f08-efb2-11ee-b647-cb9ed24678a4/ceph-osd.8.log 2024-03-31T23:40:53.320 INFO:teuthology.orchestra.run.smithi162.stderr:/var/log/ceph/6a9d8f08-efb2-11ee-b647-cb9ed24678a4/ceph-mgr.z.log: gzip -5 --verbose -- /var/log/ceph/6a9d8f08-efb2-11ee-b647-cb9ed24678a4/ceph-osd.5.log 2024-03-31T23:40:53.320 INFO:teuthology.orchestra.run.smithi162.stderr:/var/log/ceph/6a9d8f08-efb2-11ee-b647-cb9ed24678a4/ceph-osd.8.log: gzip -5 --verbose -- /var/log/ceph/6a9d8f08-efb2-11ee-b647-cb9ed24678a4/ceph-mds.f.log 2024-03-31T23:40:53.320 INFO:teuthology.orchestra.run.smithi162.stderr:/var/log/ceph/6a9d8f08-efb2-11ee-b647-cb9ed24678a4/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/6a9d8f08-efb2-11ee-b647-cb9ed24678a4/ceph.cephadm.log 2024-03-31T23:40:53.320 INFO:teuthology.orchestra.run.smithi162.stderr:/var/log/ceph/6a9d8f08-efb2-11ee-b647-cb9ed24678a4/ceph-mds.f.log: gzip -5 --verbose -- /var/log/ceph/6a9d8f08-efb2-11ee-b647-cb9ed24678a4/ceph.log 2024-03-31T23:40:53.320 INFO:teuthology.orchestra.run.smithi162.stderr:/var/log/ceph/6a9d8f08-efb2-11ee-b647-cb9ed24678a4/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/6a9d8f08-efb2-11ee-b647-cb9ed24678a4/ceph-osd.2.log 2024-03-31T23:40:53.320 INFO:teuthology.orchestra.run.smithi162.stderr:/var/log/ceph/6a9d8f08-efb2-11ee-b647-cb9ed24678a4/ceph.log: gzip -5 --verbose -- /var/log/ceph/6a9d8f08-efb2-11ee-b647-cb9ed24678a4/ceph-mds.c.log 2024-03-31T23:40:53.321 INFO:teuthology.orchestra.run.smithi162.stderr:/var/log/ceph/6a9d8f08-efb2-11ee-b647-cb9ed24678a4/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/6a9d8f08-efb2-11ee-b647-cb9ed24678a4/ceph-volume.log 2024-03-31T23:40:53.321 INFO:teuthology.orchestra.run.smithi162.stderr:gzip -5 --verbose -- /var/log/ceph/6a9d8f08-efb2-11ee-b647-cb9ed24678a4/ceph-mon.c.log 2024-03-31T23:40:53.374 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/6a9d8f08-efb2-11ee-b647-cb9ed24678a4/ceph-volume.log: /var/log/ceph/6a9d8f08-efb2-11ee-b647-cb9ed24678a4/ceph-mon.b.log: 92.1% -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-31T23:40:53.380 INFO:teuthology.orchestra.run.smithi046.stderr: 92.8% -- replaced with /var/log/ceph/ceph-client.admin.log.gz 2024-03-31T23:40:53.399 INFO:teuthology.orchestra.run.smithi142.stderr: 84.3% -- replaced with /var/log/ceph/6a9d8f08-efb2-11ee-b647-cb9ed24678a4/ceph.cephadm.log.gz 2024-03-31T23:40:53.410 INFO:teuthology.orchestra.run.smithi142.stderr: 90.3% -- replaced with /var/log/ceph/6a9d8f08-efb2-11ee-b647-cb9ed24678a4/ceph.audit.log.gz 2024-03-31T23:40:53.424 INFO:teuthology.orchestra.run.smithi046.stderr: 84.3% -- replaced with /var/log/ceph/6a9d8f08-efb2-11ee-b647-cb9ed24678a4/ceph.cephadm.log.gz 2024-03-31T23:40:53.436 INFO:teuthology.orchestra.run.smithi162.stderr:/var/log/ceph/6a9d8f08-efb2-11ee-b647-cb9ed24678a4/ceph-mds.c.log: /var/log/ceph/6a9d8f08-efb2-11ee-b647-cb9ed24678a4/ceph-volume.log: /var/log/ceph/6a9d8f08-efb2-11ee-b647-cb9ed24678a4/ceph-mon.c.log: 84.1% -- replaced with /var/log/ceph/6a9d8f08-efb2-11ee-b647-cb9ed24678a4/ceph.cephadm.log.gz 2024-03-31T23:40:53.440 INFO:teuthology.orchestra.run.smithi162.stderr: 90.4% -- replaced with /var/log/ceph/6a9d8f08-efb2-11ee-b647-cb9ed24678a4/ceph.audit.log.gz 2024-03-31T23:40:53.491 INFO:teuthology.orchestra.run.smithi046.stderr: 90.2% -- replaced with /var/log/ceph/6a9d8f08-efb2-11ee-b647-cb9ed24678a4/ceph.audit.log.gz 2024-03-31T23:40:53.494 INFO:teuthology.orchestra.run.smithi046.stderr: 93.3% -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-31T23:40:53.528 INFO:teuthology.orchestra.run.smithi162.stderr: 90.9% -- replaced with /var/log/ceph/6a9d8f08-efb2-11ee-b647-cb9ed24678a4/ceph-volume.log.gz 2024-03-31T23:40:53.562 INFO:teuthology.orchestra.run.smithi142.stderr: 92.7% -- replaced with /var/log/ceph/6a9d8f08-efb2-11ee-b647-cb9ed24678a4/ceph-mgr.y.log.gz 2024-03-31T23:40:53.593 INFO:teuthology.orchestra.run.smithi046.stderr: 94.6% -- replaced with /var/log/ceph/6a9d8f08-efb2-11ee-b647-cb9ed24678a4/ceph-mds.a.log.gz 2024-03-31T23:40:53.599 INFO:teuthology.orchestra.run.smithi046.stderr: 94.7% -- replaced with /var/log/ceph/6a9d8f08-efb2-11ee-b647-cb9ed24678a4/ceph-mds.d.log.gz 2024-03-31T23:40:53.608 INFO:teuthology.orchestra.run.smithi162.stderr: 92.8% -- replaced with /var/log/ceph/6a9d8f08-efb2-11ee-b647-cb9ed24678a4/ceph-mgr.z.log.gz 2024-03-31T23:40:53.628 INFO:teuthology.orchestra.run.smithi162.stderr: 92.4% -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-31T23:40:53.631 INFO:teuthology.orchestra.run.smithi142.stderr: 94.7% -- replaced with /var/log/ceph/6a9d8f08-efb2-11ee-b647-cb9ed24678a4/ceph-mds.k.log.gz 2024-03-31T23:40:53.677 INFO:teuthology.orchestra.run.smithi046.stderr: 91.1% -- replaced with /var/log/ceph/6a9d8f08-efb2-11ee-b647-cb9ed24678a4/ceph-volume.log.gz 2024-03-31T23:40:53.680 INFO:teuthology.orchestra.run.smithi142.stderr: 91.0% -- replaced with /var/log/ceph/6a9d8f08-efb2-11ee-b647-cb9ed24678a4/ceph-volume.log.gz 2024-03-31T23:40:53.715 INFO:teuthology.orchestra.run.smithi162.stderr: 94.6% -- replaced with /var/log/ceph/6a9d8f08-efb2-11ee-b647-cb9ed24678a4/ceph-mds.l.log.gz 2024-03-31T23:40:53.783 INFO:teuthology.orchestra.run.smithi162.stderr: 87.5% -- replaced with /var/log/ceph/6a9d8f08-efb2-11ee-b647-cb9ed24678a4/ceph.log.gz 2024-03-31T23:40:53.819 INFO:teuthology.orchestra.run.smithi046.stderr: 94.7% -- replaced with /var/log/ceph/6a9d8f08-efb2-11ee-b647-cb9ed24678a4/ceph-mds.j.log.gz 2024-03-31T23:40:54.973 INFO:teuthology.orchestra.run.smithi142.stderr: 87.4% -- replaced with /var/log/ceph/6a9d8f08-efb2-11ee-b647-cb9ed24678a4/ceph.log.gz 2024-03-31T23:40:57.297 INFO:teuthology.orchestra.run.smithi046.stderr: 90.3% -- replaced with /var/log/ceph/6a9d8f08-efb2-11ee-b647-cb9ed24678a4/ceph-mgr.x.log.gz 2024-03-31T23:41:17.133 INFO:teuthology.orchestra.run.smithi162.stderr: 91.7% -- replaced with /var/log/ceph/6a9d8f08-efb2-11ee-b647-cb9ed24678a4/ceph-mon.c.log.gz 2024-03-31T23:41:18.735 INFO:teuthology.orchestra.run.smithi142.stderr: 92.0% -- replaced with /var/log/ceph/6a9d8f08-efb2-11ee-b647-cb9ed24678a4/ceph-mon.b.log.gz 2024-03-31T23:41:38.676 INFO:teuthology.orchestra.run.smithi046.stderr: 91.0% -- replaced with /var/log/ceph/6a9d8f08-efb2-11ee-b647-cb9ed24678a4/ceph-mon.a.log.gz 2024-03-31T23:43:44.597 INFO:teuthology.orchestra.run.smithi162.stderr: 94.9% -- replaced with /var/log/ceph/6a9d8f08-efb2-11ee-b647-cb9ed24678a4/ceph-mds.f.log.gz 2024-03-31T23:44:30.184 INFO:teuthology.orchestra.run.smithi162.stderr: 94.6% -- replaced with /var/log/ceph/6a9d8f08-efb2-11ee-b647-cb9ed24678a4/ceph-mds.i.log.gz 2024-03-31T23:44:36.591 INFO:teuthology.orchestra.run.smithi142.stderr: 94.7% -- replaced with /var/log/ceph/6a9d8f08-efb2-11ee-b647-cb9ed24678a4/ceph-mds.e.log.gz 2024-03-31T23:47:07.034 INFO:teuthology.orchestra.run.smithi142.stderr: 91.2% -- replaced with /var/log/ceph/6a9d8f08-efb2-11ee-b647-cb9ed24678a4/ceph-osd.10.log.gz 2024-03-31T23:47:10.511 INFO:teuthology.orchestra.run.smithi046.stderr: 91.6% -- replaced with /var/log/ceph/6a9d8f08-efb2-11ee-b647-cb9ed24678a4/ceph-osd.0.log.gz 2024-03-31T23:47:12.964 INFO:teuthology.orchestra.run.smithi142.stderr: 91.2% -- replaced with /var/log/ceph/6a9d8f08-efb2-11ee-b647-cb9ed24678a4/ceph-osd.4.log.gz 2024-03-31T23:47:29.678 INFO:teuthology.orchestra.run.smithi046.stderr: 91.1% -- replaced with /var/log/ceph/6a9d8f08-efb2-11ee-b647-cb9ed24678a4/ceph-osd.6.log.gz 2024-03-31T23:47:39.260 INFO:teuthology.orchestra.run.smithi142.stderr: 91.5% -- replaced with /var/log/ceph/6a9d8f08-efb2-11ee-b647-cb9ed24678a4/ceph-osd.1.log.gz 2024-03-31T23:47:58.269 INFO:teuthology.orchestra.run.smithi162.stderr: 91.2% -- replaced with /var/log/ceph/6a9d8f08-efb2-11ee-b647-cb9ed24678a4/ceph-osd.11.log.gz 2024-03-31T23:48:01.478 INFO:teuthology.orchestra.run.smithi046.stderr: 91.1% -- replaced with /var/log/ceph/6a9d8f08-efb2-11ee-b647-cb9ed24678a4/ceph-osd.9.log.gz 2024-03-31T23:48:37.085 INFO:teuthology.orchestra.run.smithi046.stderr: 91.2% -- replaced with /var/log/ceph/6a9d8f08-efb2-11ee-b647-cb9ed24678a4/ceph-osd.3.log.gz 2024-03-31T23:48:56.951 INFO:teuthology.orchestra.run.smithi162.stderr: 91.3% -- replaced with /var/log/ceph/6a9d8f08-efb2-11ee-b647-cb9ed24678a4/ceph-osd.2.log.gz 2024-03-31T23:49:18.971 INFO:teuthology.orchestra.run.smithi162.stderr: 91.0% -- replaced with /var/log/ceph/6a9d8f08-efb2-11ee-b647-cb9ed24678a4/ceph-osd.8.log.gz 2024-03-31T23:50:01.949 INFO:teuthology.orchestra.run.smithi142.stderr: 91.2% -- replaced with /var/log/ceph/6a9d8f08-efb2-11ee-b647-cb9ed24678a4/ceph-osd.7.log.gz 2024-03-31T23:51:00.210 INFO:teuthology.orchestra.run.smithi162.stderr: 91.0% -- replaced with /var/log/ceph/6a9d8f08-efb2-11ee-b647-cb9ed24678a4/ceph-osd.5.log.gz 2024-03-31T23:54:40.376 INFO:teuthology.orchestra.run.smithi162.stderr: 94.1% -- replaced with /var/log/ceph/6a9d8f08-efb2-11ee-b647-cb9ed24678a4/ceph-mds.c.log.gz 2024-03-31T23:54:40.390 INFO:teuthology.orchestra.run.smithi162.stderr: 2024-03-31T23:54:40.390 INFO:teuthology.orchestra.run.smithi162.stderr:real 13m47.106s 2024-03-31T23:54:40.390 INFO:teuthology.orchestra.run.smithi162.stderr:user 6m35.831s 2024-03-31T23:54:40.390 INFO:teuthology.orchestra.run.smithi162.stderr:sys 0m18.194s 2024-03-31T23:54:58.294 INFO:teuthology.orchestra.run.smithi046.stderr: 92.5% -- replaced with /var/log/ceph/ceph-client.0.log.gz 2024-03-31T23:54:58.296 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-03-31T23:54:58.296 INFO:teuthology.orchestra.run.smithi046.stderr:real 14m5.017s 2024-03-31T23:54:58.297 INFO:teuthology.orchestra.run.smithi046.stderr:user 8m45.736s 2024-03-31T23:54:58.297 INFO:teuthology.orchestra.run.smithi046.stderr:sys 0m21.277s 2024-04-01T00:15:31.249 INFO:teuthology.orchestra.run.smithi142.stderr: 93.6% -- replaced with /var/log/ceph/6a9d8f08-efb2-11ee-b647-cb9ed24678a4/ceph-mds.h.log.gz 2024-04-01T00:17:41.688 INFO:teuthology.orchestra.run.smithi142.stderr: 94.5% -- replaced with /var/log/ceph/6a9d8f08-efb2-11ee-b647-cb9ed24678a4/ceph-mds.b.log.gz 2024-04-01T00:17:41.731 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-04-01T00:17:41.732 INFO:teuthology.orchestra.run.smithi142.stderr:real 36m48.446s 2024-04-01T00:17:41.732 INFO:teuthology.orchestra.run.smithi142.stderr:user 11m23.835s 2024-04-01T00:17:41.732 INFO:teuthology.orchestra.run.smithi142.stderr:sys 0m33.989s 2024-04-01T00:17:41.732 INFO:tasks.cephadm:Archiving logs... 2024-04-01T00:17:41.734 DEBUG:teuthology.misc:Transferring archived files from smithi046:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-03-31_21:24:02-fs-squid-distro-default-smithi/7633118/remote/smithi046/log 2024-04-01T00:17:41.736 DEBUG:teuthology.orchestra.run.smithi046:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-01T00:19:08.505 DEBUG:teuthology.misc:Transferring archived files from smithi142:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-03-31_21:24:02-fs-squid-distro-default-smithi/7633118/remote/smithi142/log 2024-04-01T00:19:08.506 DEBUG:teuthology.orchestra.run.smithi142:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-01T00:20:51.661 DEBUG:teuthology.misc:Transferring archived files from smithi162:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-03-31_21:24:02-fs-squid-distro-default-smithi/7633118/remote/smithi162/log 2024-04-01T00:20:51.663 DEBUG:teuthology.orchestra.run.smithi162:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-01T00:21:54.639 INFO:tasks.cephadm:Removing cluster... 2024-04-01T00:21:54.640 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 6a9d8f08-efb2-11ee-b647-cb9ed24678a4 --force 2024-04-01T00:21:55.036 INFO:teuthology.orchestra.run.smithi046.stdout:Deleting cluster with fsid: 6a9d8f08-efb2-11ee-b647-cb9ed24678a4 2024-04-01T00:21:55.399 DEBUG:teuthology.orchestra.run.smithi142:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 6a9d8f08-efb2-11ee-b647-cb9ed24678a4 --force 2024-04-01T00:21:56.598 INFO:teuthology.orchestra.run.smithi142.stdout:Deleting cluster with fsid: 6a9d8f08-efb2-11ee-b647-cb9ed24678a4 2024-04-01T00:21:57.447 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 6a9d8f08-efb2-11ee-b647-cb9ed24678a4 --force 2024-04-01T00:21:57.975 INFO:teuthology.orchestra.run.smithi162.stdout:Deleting cluster with fsid: 6a9d8f08-efb2-11ee-b647-cb9ed24678a4 2024-04-01T00:21:58.670 INFO:tasks.cephadm:Removing cephadm ... 2024-04-01T00:21:58.671 DEBUG:teuthology.orchestra.run.smithi046:> rm -rf /home/ubuntu/cephtest/cephadm 2024-04-01T00:21:58.702 DEBUG:teuthology.orchestra.run.smithi142:> rm -rf /home/ubuntu/cephtest/cephadm 2024-04-01T00:21:58.722 DEBUG:teuthology.orchestra.run.smithi162:> rm -rf /home/ubuntu/cephtest/cephadm 2024-04-01T00:21:58.737 INFO:tasks.cephadm:Teardown complete 2024-04-01T00:21:58.738 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-04-01T00:21:58.753 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/task/install/__init__.py", line 626, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph_384f31f3d3aee7f8ba3f60846b8844c049165583/qa/tasks/fwd_scrub.py", line 164, in task stop_all_fwd_scrubbers(ctx.ceph[config['cluster']].thrashers) File "/home/teuthworker/src/git.ceph.com_ceph_384f31f3d3aee7f8ba3f60846b8844c049165583/qa/tasks/fwd_scrub.py", line 99, in stop_all_fwd_scrubbers raise RuntimeError(f"error during scrub thrashing: {thrasher.exception}") RuntimeError: error during scrub thrashing: rank damage found: {'backtrace'} 2024-04-01T00:21:58.754 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-04-01T00:21:58.754 DEBUG:teuthology.orchestra.run.smithi046:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-01T00:21:58.757 DEBUG:teuthology.orchestra.run.smithi142:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-01T00:21:58.765 DEBUG:teuthology.orchestra.run.smithi162:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-01T00:21:58.877 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-04-01T00:21:58.888 INFO:teuthology.task.clock:Checking final clock skew... 2024-04-01T00:21:58.888 DEBUG:teuthology.orchestra.run.smithi046:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-01T00:21:58.891 DEBUG:teuthology.orchestra.run.smithi142:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-01T00:21:58.894 DEBUG:teuthology.orchestra.run.smithi162:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-01T00:21:58.905 INFO:teuthology.orchestra.run.smithi046.stderr:bash: line 1: ntpq: command not found 2024-04-01T00:21:58.907 INFO:teuthology.orchestra.run.smithi142.stderr:bash: line 1: ntpq: command not found 2024-04-01T00:21:58.908 INFO:teuthology.orchestra.run.smithi162.stderr:bash: line 1: ntpq: command not found 2024-04-01T00:21:58.934 INFO:teuthology.orchestra.run.smithi046.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-01T00:21:58.934 INFO:teuthology.orchestra.run.smithi046.stdout:=============================================================================== 2024-04-01T00:21:58.934 INFO:teuthology.orchestra.run.smithi046.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 48 +330us[ +330us] +/- 85ms 2024-04-01T00:21:58.934 INFO:teuthology.orchestra.run.smithi046.stdout:^* hv02.front.sepia.ceph.com 3 6 377 50 -285us[ -276us] +/- 60ms 2024-04-01T00:21:58.934 INFO:teuthology.orchestra.run.smithi046.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 51 +50us[ +59us] +/- 71ms 2024-04-01T00:21:58.934 INFO:teuthology.orchestra.run.smithi046.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-04-01T00:21:58.959 INFO:teuthology.orchestra.run.smithi162.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-01T00:21:58.959 INFO:teuthology.orchestra.run.smithi162.stdout:=============================================================================== 2024-04-01T00:21:58.959 INFO:teuthology.orchestra.run.smithi162.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 44 +307us[ +307us] +/- 85ms 2024-04-01T00:21:58.959 INFO:teuthology.orchestra.run.smithi162.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 43 -297us[ -297us] +/- 60ms 2024-04-01T00:21:58.959 INFO:teuthology.orchestra.run.smithi162.stdout:^* hv03.front.sepia.ceph.com 3 6 377 48 +68us[ +64us] +/- 71ms 2024-04-01T00:21:58.959 INFO:teuthology.orchestra.run.smithi162.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-04-01T00:21:58.988 INFO:teuthology.orchestra.run.smithi142.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-01T00:21:58.988 INFO:teuthology.orchestra.run.smithi142.stdout:=============================================================================== 2024-04-01T00:21:58.988 INFO:teuthology.orchestra.run.smithi142.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 51 +306us[ +253us] +/- 84ms 2024-04-01T00:21:58.988 INFO:teuthology.orchestra.run.smithi142.stdout:^* hv02.front.sepia.ceph.com 3 7 377 45 -283us[ -337us] +/- 60ms 2024-04-01T00:21:58.988 INFO:teuthology.orchestra.run.smithi142.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 50 +43us[-9102ns] +/- 71ms 2024-04-01T00:21:58.989 INFO:teuthology.orchestra.run.smithi142.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-04-01T00:21:59.013 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-04-01T00:21:59.025 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-04-01T00:21:59.025 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-04-01T00:21:59.049 DEBUG:teuthology.orchestra.run.smithi046:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-01T00:21:59.242 DEBUG:teuthology.orchestra.run.smithi142:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-01T00:21:59.378 DEBUG:teuthology.orchestra.run.smithi162:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-01T00:21:59.532 DEBUG:teuthology.orchestra.run.smithi046:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-01T00:21:59.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-01T00:21:59.569 DEBUG:teuthology.orchestra.run.smithi142:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-01T00:21:59.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-01T00:21:59.603 DEBUG:teuthology.orchestra.run.smithi162:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-01T00:21:59.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-01T00:21:59.635 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-04-01T00:21:59.647 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-04-01T00:21:59.676 INFO:teuthology.task.internal:Duration was 5298.255652 seconds 2024-04-01T00:21:59.677 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-04-01T00:21:59.688 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-04-01T00:21:59.688 DEBUG:teuthology.orchestra.run.smithi046:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-01T00:21:59.691 DEBUG:teuthology.orchestra.run.smithi142:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-01T00:21:59.694 DEBUG:teuthology.orchestra.run.smithi162:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-01T00:21:59.752 INFO:teuthology.orchestra.run.smithi162.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-01T00:21:59.754 INFO:teuthology.orchestra.run.smithi046.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-01T00:21:59.760 INFO:teuthology.orchestra.run.smithi142.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-01T00:22:00.132 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-04-01T00:22:00.133 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi046.front.sepia.ceph.com 2024-04-01T00:22:00.133 DEBUG:teuthology.orchestra.run.smithi046:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-01T00:22:00.165 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi142.front.sepia.ceph.com 2024-04-01T00:22:00.165 DEBUG:teuthology.orchestra.run.smithi142:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-01T00:22:00.192 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi162.front.sepia.ceph.com 2024-04-01T00:22:00.192 DEBUG:teuthology.orchestra.run.smithi162:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-01T00:22:00.217 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-04-01T00:22:00.218 DEBUG:teuthology.orchestra.run.smithi046:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-01T00:22:00.220 DEBUG:teuthology.orchestra.run.smithi142:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-01T00:22:00.234 DEBUG:teuthology.orchestra.run.smithi162:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-01T00:22:00.293 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-04-01T00:22:00.294 DEBUG:teuthology.orchestra.run.smithi046:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-01T00:22:00.297 DEBUG:teuthology.orchestra.run.smithi142:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-01T00:22:00.299 DEBUG:teuthology.orchestra.run.smithi162:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-01T00:22:00.705 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-04-01T00:22:00.718 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-04-01T00:22:00.718 DEBUG:teuthology.orchestra.run.smithi046:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-01T00:22:00.745 DEBUG:teuthology.orchestra.run.smithi142:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-01T00:22:00.783 DEBUG:teuthology.orchestra.run.smithi162:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-01T00:22:00.812 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-04-01T00:22:00.824 DEBUG:teuthology.orchestra.run.smithi046:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-01T00:22:00.826 DEBUG:teuthology.orchestra.run.smithi142:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-01T00:22:00.828 DEBUG:teuthology.orchestra.run.smithi162:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-01T00:22:00.869 INFO:teuthology.orchestra.run.smithi046.stdout:kernel.core_pattern = core 2024-04-01T00:22:00.872 INFO:teuthology.orchestra.run.smithi142.stdout:kernel.core_pattern = core 2024-04-01T00:22:00.889 INFO:teuthology.orchestra.run.smithi162.stdout:kernel.core_pattern = core 2024-04-01T00:22:00.910 DEBUG:teuthology.orchestra.run.smithi046:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-01T00:22:00.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-01T00:22:00.924 DEBUG:teuthology.orchestra.run.smithi142:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-01T00:22:00.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-01T00:22:00.938 DEBUG:teuthology.orchestra.run.smithi162:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-01T00:22:00.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-01T00:22:00.965 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-04-01T00:22:00.977 INFO:teuthology.task.internal:Transferring archived files... 2024-04-01T00:22:00.979 DEBUG:teuthology.misc:Transferring archived files from smithi046:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-03-31_21:24:02-fs-squid-distro-default-smithi/7633118/remote/smithi046 2024-04-01T00:22:00.979 DEBUG:teuthology.orchestra.run.smithi046:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-01T00:22:01.025 DEBUG:teuthology.misc:Transferring archived files from smithi142:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-03-31_21:24:02-fs-squid-distro-default-smithi/7633118/remote/smithi142 2024-04-01T00:22:01.026 DEBUG:teuthology.orchestra.run.smithi142:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-01T00:22:01.065 DEBUG:teuthology.misc:Transferring archived files from smithi162:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-03-31_21:24:02-fs-squid-distro-default-smithi/7633118/remote/smithi162 2024-04-01T00:22:01.065 DEBUG:teuthology.orchestra.run.smithi162:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-01T00:22:01.104 INFO:teuthology.task.internal:Removing archive directory... 2024-04-01T00:22:01.104 DEBUG:teuthology.orchestra.run.smithi046:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-01T00:22:01.107 DEBUG:teuthology.orchestra.run.smithi142:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-01T00:22:01.109 DEBUG:teuthology.orchestra.run.smithi162:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-01T00:22:01.152 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-04-01T00:22:01.167 INFO:teuthology.task.internal:Not uploading archives. 2024-04-01T00:22:01.167 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-04-01T00:22:01.178 INFO:teuthology.task.internal:Tidying up after the test... 2024-04-01T00:22:01.178 DEBUG:teuthology.orchestra.run.smithi046:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-01T00:22:01.182 DEBUG:teuthology.orchestra.run.smithi142:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-01T00:22:01.184 DEBUG:teuthology.orchestra.run.smithi162:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-01T00:22:01.192 INFO:teuthology.orchestra.run.smithi046.stdout: 265594 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 1 00:22 /home/ubuntu/cephtest 2024-04-01T00:22:01.197 INFO:teuthology.orchestra.run.smithi142.stdout: 265305 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 1 00:22 /home/ubuntu/cephtest 2024-04-01T00:22:01.207 INFO:teuthology.orchestra.run.smithi162.stdout: 265955 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 1 00:22 /home/ubuntu/cephtest 2024-04-01T00:22:01.208 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-04-01T00:22:01.221 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-04-01T00:22:01.233 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: RuntimeError: error during scrub thrashing: rank damage found: {'backtrace'} 2024-04-01T00:22:01.233 INFO:teuthology.nuke:Checking targets against current locks 2024-04-01T00:22:01.257 DEBUG:teuthology.nuke:shortname: smithi046 2024-04-01T00:22:01.257 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-01T00:22:01.281 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi046.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-03-31_21:24:02-fs-squid-distro-default-smithi/7633118', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-31 22:45:23.367893', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKdZO7+tm/7WwtmA5s3xtAa9MJRB8l9mA76h6kRc/T2RfJWtZdopxOO1f0WThP2iPe2LvHj3cp16hOSeZKJr7Z0='} 2024-04-01T00:22:01.284 DEBUG:teuthology.nuke:shortname: smithi142 2024-04-01T00:22:01.285 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-01T00:22:01.304 DEBUG:teuthology.nuke:shortname: smithi162 2024-04-01T00:22:01.304 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-01T00:22:01.307 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi142.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-03-31_21:24:02-fs-squid-distro-default-smithi/7633118', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-31 22:45:23.366388', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPFrXvXXABKnguJpriXZPwIgvyK78LKrpd3e2av2k8DYf2gqPhIk/U4d/SMfnaPwoJlGSEiOk3KuNp3zh5O7w9Q='} 2024-04-01T00:22:01.330 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi162.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-03-31_21:24:02-fs-squid-distro-default-smithi/7633118', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-31 22:45:23.369609', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDlvehU7gK/q6QtQaxtb5B9Y0do8OvRq0Me6oeDQ5w1+zo2iivV6Ow2KFz+VHssbVkbn1zw4ACrcM6uAR5wZAW4='} 2024-04-01T00:22:01.339 INFO:teuthology.orchestra.console.smithi046:Power off 2024-04-01T00:22:01.340 DEBUG:teuthology.orchestra.console.smithi046:pexpect command: ipmitool -H smithi046.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-01T00:22:01.356 INFO:teuthology.orchestra.console.smithi142:Power off 2024-04-01T00:22:01.356 DEBUG:teuthology.orchestra.console.smithi142:pexpect command: ipmitool -H smithi142.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-01T00:22:01.370 DEBUG:teuthology.orchestra.console.smithi046:power off output: Chassis Power Control: Down/Off 2024-04-01T00:22:01.370 DEBUG:teuthology.orchestra.console.smithi046:pexpect command: ipmitool -H smithi046.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-01T00:22:01.381 DEBUG:teuthology.orchestra.console.smithi142:power off output: Chassis Power Control: Down/Off 2024-04-01T00:22:01.381 DEBUG:teuthology.orchestra.console.smithi142:pexpect command: ipmitool -H smithi142.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-01T00:22:01.394 DEBUG:teuthology.orchestra.console.smithi046:check power output: Chassis Power is on 2024-04-01T00:22:01.406 INFO:teuthology.orchestra.console.smithi162:Power off 2024-04-01T00:22:01.406 DEBUG:teuthology.orchestra.console.smithi162:pexpect command: ipmitool -H smithi162.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-01T00:22:01.416 DEBUG:teuthology.orchestra.console.smithi142:check power output: Chassis Power is on 2024-04-01T00:22:01.429 DEBUG:teuthology.orchestra.console.smithi162:power off output: Chassis Power Control: Down/Off 2024-04-01T00:22:01.429 DEBUG:teuthology.orchestra.console.smithi162:pexpect command: ipmitool -H smithi162.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-01T00:22:01.451 DEBUG:teuthology.orchestra.console.smithi162:check power output: Chassis Power is on 2024-04-01T00:22:05.396 DEBUG:teuthology.orchestra.console.smithi046:pexpect command: ipmitool -H smithi046.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-01T00:22:05.416 DEBUG:teuthology.orchestra.console.smithi142:pexpect command: ipmitool -H smithi142.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-01T00:22:05.453 DEBUG:teuthology.orchestra.console.smithi162:pexpect command: ipmitool -H smithi162.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-01T00:22:05.509 DEBUG:teuthology.orchestra.console.smithi046:check power output: Chassis Power is on 2024-04-01T00:22:05.529 DEBUG:teuthology.orchestra.console.smithi142:check power output: Chassis Power is on 2024-04-01T00:22:05.567 DEBUG:teuthology.orchestra.console.smithi162:check power output: Chassis Power is on 2024-04-01T00:22:09.514 DEBUG:teuthology.orchestra.console.smithi046:pexpect command: ipmitool -H smithi046.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-01T00:22:09.530 DEBUG:teuthology.orchestra.console.smithi142:pexpect command: ipmitool -H smithi142.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-01T00:22:09.568 DEBUG:teuthology.orchestra.console.smithi162:pexpect command: ipmitool -H smithi162.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-01T00:22:09.625 DEBUG:teuthology.orchestra.console.smithi046:check power output: Chassis Power is off 2024-04-01T00:22:09.640 DEBUG:teuthology.orchestra.console.smithi142:check power output: Chassis Power is off 2024-04-01T00:22:09.678 DEBUG:teuthology.orchestra.console.smithi162:check power output: Chassis Power is off 2024-04-01T00:22:09.726 INFO:teuthology.orchestra.console.smithi046:Power off completed 2024-04-01T00:22:09.740 INFO:teuthology.orchestra.console.smithi142:Power off completed 2024-04-01T00:22:09.779 INFO:teuthology.orchestra.console.smithi162:Power off completed 2024-04-01T00:22:09.884 INFO:teuthology.run:Summary data: description: fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/fuse objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/suites/dbench}} duration: 5298.255652189255 failure_reason: 'error during scrub thrashing: rank damage found: {''backtrace''}' flavor: default owner: scheduled_teuthology@teuthology status: fail success: false 2024-04-01T00:22:09.884 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-01T00:22:09.984 INFO:teuthology.run:FAIL